<?xml version="1.0" encoding="utf-8"?>
<posts>
  <row Id="5" PostTypeId="1" CreationDate="2014-05-13T23:58:30.457" Score="9" ViewCount="968" Body="&lt;p&gt;I've always been interested in machine learning, but I can't figure out one thing about starting out with a simple &quot;Hello World&quot; example - how can I avoid hard-coding behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I wanted to &quot;teach&quot; a bot how to avoid randomly placed obstacles, I couldn't just use relative motion, because the obstacles move around, but I don't want to hard code, say, distance, because that ruins the whole point of machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, randomly generating code would be impractical, so how could I do this?&lt;/p&gt;&#xA;" OwnerUserId="5" LastActivityDate="2014-05-14T00:36:31.077" Title="How can I do simple machine learning without hard-coding behavior?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2014-05-14T14:40:25.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="7" PostTypeId="1" AcceptedAnswerId="10" CreationDate="2014-05-14T00:11:06.457" Score="4" ViewCount="510" Body="&lt;p&gt;As a researcher and instructor, I'm looking for open-source books (or similar materials) that provide a relatively thorough overview of data science from an applied perspective. To be clear, I'm especially interested in a thorough overview that provides material suitable for a college-level course, not particular pieces or papers.&lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="97" LastEditDate="2014-05-16T13:45:00.237" LastActivityDate="2014-05-16T13:45:00.237" Title="What open-source books (or other materials) provide a relatively thorough overview of data science?" Tags="|education|open-source|" AnswerCount="3" CommentCount="4" ClosedDate="2014-05-14T08:40:54.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="9" PostTypeId="2" ParentId="5" CreationDate="2014-05-14T00:36:31.077" Score="5" Body="&lt;p&gt;Not sure if this fits the scope of this SE, but here's a stab at an answer anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With all AI approaches you have to decide what it is you're modelling and what kind of uncertainty there is. Once you pick a framework that allows modelling of your situation, you then see which elements are &quot;fixed&quot; and which are flexible. For example, the model may allow you to define your own network structure (or even learn it) with certain constraints. You have to decide whether this flexibility is sufficient for your purposes. Then within a particular network structure, you can learn parameters given a specific training dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You rarely hard-code behavior in AI/ML solutions. It's all about modelling the underlying situation and accommodating different situations by tweaking elements of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example, perhaps you might have the robot learn how to detect obstacles (by analyzing elements in the environment), or you might have it keep track of where the obstacles were and which way they were moving.&lt;/p&gt;&#xA;" OwnerUserId="51" LastActivityDate="2014-05-14T00:36:31.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10" PostTypeId="2" ParentId="7" CreationDate="2014-05-14T00:53:43.273" Score="13" Body="&lt;p&gt;One book that's freely available is &quot;The Elements of Statistical Learning&quot; by Hastie, Tibshirani, and Friedman (published by Springer): &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot;&gt;see Tibshirani's website&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another fantastic source, although it isn't a book, is Andrew Ng's Machine Learning course on Coursera. This has a much more applied-focus than the above book, and Prof. Ng does a great job of explaining the thinking behind several different machine learning algorithms/situations.&lt;/p&gt;&#xA;" OwnerUserId="22" LastActivityDate="2014-05-14T00:53:43.273" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14" PostTypeId="1" AcceptedAnswerId="29" CreationDate="2014-05-14T01:25:59.677" Score="26" ViewCount="1931" Body="&lt;p&gt;I am sure data science as will be discussed in this forum has several synonyms or at least related fields where large data is analyzed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My particular question is in regards to Data Mining.  I took a graduate class in Data Mining a few years back.  What are the differences between Data Science and Data Mining and in particular what more would I need to look at to become proficient in Data Mining?&lt;/p&gt;&#xA;" OwnerUserId="66" LastEditorUserId="322" LastEditDate="2014-06-17T16:17:20.473" LastActivityDate="2020-08-16T13:01:33.543" Title="Is Data Science the Same as Data Mining?" Tags="|data-mining|definitions|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15" PostTypeId="1" CreationDate="2014-05-14T01:41:23.110" Score="2" ViewCount="657" Body="&lt;p&gt;In which situations would one system be preferred over the other? What are the relative advantages and disadvantages of relational databases versus non-relational databases?&lt;/p&gt;&#xA;" OwnerUserId="64" LastActivityDate="2014-05-14T01:41:23.110" Title="What are the advantages and disadvantages of SQL versus NoSQL in data science?" Tags="|databases|" AnswerCount="0" CommentCount="1" ClosedDate="2014-05-14T07:41:49.437" ContentLicense="CC BY-SA 3.0" />
  <row Id="16" PostTypeId="1" AcceptedAnswerId="46" CreationDate="2014-05-14T01:57:56.880" Score="17" ViewCount="436" Body="&lt;p&gt;I use &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot;&gt;Libsvm&lt;/a&gt; to train data and predict classification on &lt;strong&gt;semantic analysis&lt;/strong&gt; problem. But it has a &lt;strong&gt;performance&lt;/strong&gt; issue on large-scale data, because semantic analysis concerns &lt;strong&gt;&lt;em&gt;n-dimension&lt;/em&gt;&lt;/strong&gt; problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last year, &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/liblinear/&quot;&gt;Liblinear&lt;/a&gt; was release, and it can solve performance bottleneck.&#xA;But it cost too much &lt;strong&gt;memory&lt;/strong&gt;. Is &lt;strong&gt;MapReduce&lt;/strong&gt; the only way to solve semantic analysis problem on big data? Or are there any other methods that can improve memory bottleneck on &lt;strong&gt;Liblinear&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="63" LastEditorUserId="84" LastEditDate="2014-05-17T16:24:14.523" LastActivityDate="2014-05-17T16:24:14.523" Title="Use liblinear on big data for semantic analysis" Tags="|machine-learning|bigdata|libsvm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17" PostTypeId="5" CreationDate="2014-05-14T02:49:14.580" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow&quot;&gt;LIBSVM&lt;/a&gt; is a library for support vector classification (SVM) and regression.&#xA;It was created by Chih-Chung Chang and Chih-Jen Lin in 2001.&lt;/p&gt;&#xA;" OwnerUserId="63" LastEditorUserId="63" LastEditDate="2014-05-16T13:44:53.470" LastActivityDate="2014-05-16T13:44:53.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18" PostTypeId="4" CreationDate="2014-05-14T02:49:14.580" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2014-05-14T02:49:14.580" LastActivityDate="2014-05-14T02:49:14.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19" PostTypeId="1" AcceptedAnswerId="37" CreationDate="2014-05-14T03:56:20.963" Score="94" ViewCount="20064" Body="&lt;p&gt;Lots of people use the term &lt;em&gt;big data&lt;/em&gt; in a rather &lt;em&gt;commercial&lt;/em&gt; way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good performance. Of course, &lt;em&gt;big data&lt;/em&gt; always carry associated terms, like scalability and efficiency, but what exactly defines a problem as a &lt;em&gt;big data&lt;/em&gt; problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the computation have to be related to some set of specific purposes, like data mining/information retrieval, or could an algorithm for general graph problems be labeled &lt;em&gt;big data&lt;/em&gt; if the dataset was &lt;em&gt;big enough&lt;/em&gt;? Also, how &lt;em&gt;big&lt;/em&gt; is &lt;em&gt;big enough&lt;/em&gt; (if this is possible to define)?&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="10119" LastEditDate="2015-06-11T20:15:28.720" LastActivityDate="2018-05-01T13:04:43.563" Title="How big is big data?" Tags="|bigdata|scalability|efficiency|performance|" AnswerCount="12" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20" PostTypeId="1" AcceptedAnswerId="26" CreationDate="2014-05-14T05:37:46.780" Score="19" ViewCount="454" Body="&lt;p&gt;We created a social network application for eLearning purposes. It's an experimental project that we are researching on in our lab. It has been used in some case studies for a while and the data in our relational DBMS (SQL Server 2008) is getting big. It's a few gigabytes now and the tables are highly connected to each other. The performance is still fine, but when should we consider other options? Is it the matter of performance?  &lt;/p&gt;&#xA;" OwnerUserId="96" LastEditorUserId="29169" LastEditDate="2019-09-07T18:23:57.040" LastActivityDate="2022-07-14T08:30:28.583" Title="The data in our relational DBMS is getting big, is it the time to move to NoSQL?" Tags="|nosql|relational-dbms|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="21" PostTypeId="2" ParentId="19" CreationDate="2014-05-14T05:44:29.340" Score="34" Body="&lt;p&gt;As you rightly note, these days &quot;big data&quot; is something everyone wants to say they've got, which entails a certain looseness in how people define the term.  Generally, though, I'd say you're certainly dealing with big data if the scale is such that it's no longer feasible to manage with more traditional technologies such as RDBMS, at least without complementing them with big data technologies such as Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How big your data has to actually be for that to be the case is debatable.  Here's a (somewhat provocative) &lt;a href=&quot;http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html&quot;&gt;blog post&lt;/a&gt; that claims that it's not really the case for less than 5 TB of data.  (To be clear, it doesn't claim &quot;Less than 5 TB isn't big data&quot;, but just &quot;Less than 5 TB isn't big enough that you need Hadoop&quot;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But even on smaller datasets, big data technologies like Hadoop can have other advantages, including being well suited to batch operations, playing well with unstructured data (as well as data whose structure isn't known in advance or could change), horizontal scalability (scaling by adding more nodes instead of beefing up your existing servers), and (as one of the commenters on the above-linked post notes) the ability to integrate your data processing with external data sets (think of a map-reduce where the mapper makes a call to another server).  Other technologies associated with big data, like NoSql databases, emphasize fast performance and consistent availability while dealing with large sets of data, as well also being able to handle semi-unstructured data and to scale horizontally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, traditional RDBMS have their own advantages including ACID guarantees (Atomicity, Consistency, Isolation, Durability) and better performance for certain operations, as well as being more standardized, more mature, and (for many users) more familiar.  So even for indisputably &quot;big&quot; data, it may make sense to load at least a portion of your data into a traditional SQL database and use that in conjunction with big data technologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a more generous definition would be that you have big data so long as it's big enough that big data technologies provide some added value for you.  But as you can see, that can depend not just on the size of your data but on how you want to work with it and what sort of requirements you have in terms of flexibility, consistency, and performance.  &lt;em&gt;How&lt;/em&gt; you're using your data is more relevant to the question than what you're using it &lt;em&gt;for&lt;/em&gt; (e.g. data mining).  That said, uses like data mining and machine learning are more likely to yield useful results if you have a big enough data set to work with.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2014-05-14T05:44:29.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22" PostTypeId="1" AcceptedAnswerId="24" CreationDate="2014-05-14T05:58:21.927" Score="202" ViewCount="306778" Body="&lt;p&gt;My data set contains a number of numeric attributes and one categorical.&lt;/p&gt;&#xA;&lt;p&gt;Say, &lt;code&gt;NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr&lt;/code&gt;,&lt;/p&gt;&#xA;&lt;p&gt;where &lt;code&gt;CategoricalAttr&lt;/code&gt; takes one of three possible values: &lt;code&gt;CategoricalAttrValue1&lt;/code&gt;, &lt;code&gt;CategoricalAttrValue2&lt;/code&gt; or &lt;code&gt;CategoricalAttrValue3&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I'm using default &lt;a href=&quot;https://blog.west.uni-koblenz.de/2012-07-14/a-working-k-means-code-for-octave/&quot; rel=&quot;noreferrer&quot;&gt;k-means clustering algorithm implementation for Octave&lt;/a&gt;.&#xA;It works with numeric data only.&lt;/p&gt;&#xA;&lt;p&gt;So my question: is it correct to split the categorical attribute &lt;code&gt;CategoricalAttr&lt;/code&gt; into three numeric (binary) variables, like &lt;code&gt;IsCategoricalAttrValue1, IsCategoricalAttrValue2, IsCategoricalAttrValue3&lt;/code&gt; ?&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="98307" LastEditDate="2020-08-07T14:12:08.577" LastActivityDate="2022-10-14T09:40:25.270" Title="K-Means clustering for mixed numeric and categorical data" Tags="|data-mining|clustering|octave|k-means|categorical-data|" AnswerCount="13" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="23" PostTypeId="2" ParentId="7" CreationDate="2014-05-14T06:06:13.603" Score="8" Body="&lt;p&gt;Data Science specialization from Johns Hopkins University at Coursera would be a great start.&#xA;&lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1&quot;&gt;https://www.coursera.org/specialization/jhudatascience/1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2014-05-14T06:06:13.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24" PostTypeId="2" ParentId="22" CreationDate="2014-05-14T06:26:27.163" Score="184" Body="&lt;p&gt;The standard k-means algorithm isn't directly applicable to categorical data, for various reasons.  The sample space for categorical data is discrete, and doesn't have a natural origin.  A Euclidean distance function on such a space isn't really meaningful.  As someone put it, &quot;The fact a snake possesses neither wheels nor legs allows us to say nothing about the relative value of wheels and legs.&quot; (from &lt;a href=&quot;http://www.daylight.com/meetings/mug04/Bradshaw/why_k-modes.html&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a variation of k-means known as k-modes, introduced in &lt;a href=&quot;http://www.cs.ust.hk/~qyang/Teaching/537/Papers/huang98extensions.pdf&quot;&gt;this paper&lt;/a&gt; by Zhexue Huang, which is suitable for categorical data.   Note that the solutions you get are sensitive to initial conditions, as discussed &lt;a href=&quot;http://arxiv.org/ftp/cs/papers/0603/0603120.pdf&quot;&gt;here&lt;/a&gt; (PDF), for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Huang's paper (linked above) also has a section on &quot;k-prototypes&quot; which applies to data with a mix of categorical and numeric features.  It uses a distance measure which mixes the Hamming distance for categorical features and the Euclidean distance for numeric features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Google search for &quot;k-means mix of categorical data&quot; turns up quite a few more recent papers on various algorithms for k-means-like clustering with a mix of categorical and numeric data.  (I haven't yet read them, so I can't comment on their merits.)  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Actually, what you suggest (converting categorical attributes to binary values, and then doing k-means as if these were numeric values) is another approach that has been tried before (predating k-modes).  (See Ralambondrainy, H. 1995. A conceptual version of the k-means algorithm. Pattern Recognition Letters, 16:1147–1157.)  But I believe the k-modes approach is preferred for the reasons I indicated above.&lt;/p&gt;&#xA;" OwnerUserId="14" LastEditorUserId="14" LastEditDate="2016-11-29T20:06:51.543" LastActivityDate="2016-11-29T20:06:51.543" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="25" PostTypeId="2" ParentId="19" CreationDate="2014-05-14T07:26:04.390" Score="7" Body="&lt;p&gt;Big Data is defined by the volume of data, that's right, but not only. The particularity of big data is that you need to store a &lt;strong&gt;lots&lt;/strong&gt; of &lt;strong&gt;various&lt;/strong&gt; and sometimes &lt;strong&gt;unstructured&lt;/strong&gt; stuffs &lt;strong&gt;all the times&lt;/strong&gt; and from a &lt;strong&gt;tons of sensors&lt;/strong&gt;, usually &lt;strong&gt;for years or decade&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore you need something scalable, so that it doesn't take you half a year to find a data back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here's come Big Data, where traditional method won't work anymore. SQL is not scalable. And SQL works with very structured and linked data (with all those Primary and foreign key mess, innerjoin, imbricated request...). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, because storage becomes cheaper and cheaper and data becomes more and more valuable, big manager ask engineer to records everything. Add to this tons of new sensors with all those mobile, social network, embeded stuff ...etc. So as classic methods won't work, they have to find new technologies (storing everything in files, in json format, with big index, what we call noSQL). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So Big Data may be very big but can be not so big but complexe unstructured or various data which has to be store quickly and on-the-run in a raw format. We focus and storing at first, and then we look at how to link everything together.&lt;/p&gt;&#xA;" OwnerUserId="104" LastActivityDate="2014-05-14T07:26:04.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26" PostTypeId="2" ParentId="20" CreationDate="2014-05-14T07:38:31.103" Score="15" Body="&lt;p&gt;A few gigabytes is not very &quot;&lt;strong&gt;big&lt;/strong&gt;&quot;. It's more like the normal size of an enterprise DB. As long as you go over PK when joining tables it should work out really well, even in the future (as long as you don't get TB's of data a day).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most professionals working in a big data environment consider &lt;strong&gt;&gt; ~5TB&lt;/strong&gt; as the &lt;em&gt;beginning&lt;/em&gt; of the term big data. But even then it's not always the best way to just install the next best nosql database. You should always think about the task that you want to archive with the data (aggregate,read,search,mine,..) to find the best tools for you problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. if you do alot of searches in you database it would probably be better to run a solr instance/cluster and denormalize your data from a DBMS like Postgres or your SQL Server from time to time and put it into solr instead of just moving the data from sql to nosql in term of persistence and performance.&lt;/p&gt;&#xA;" OwnerUserId="115" LastEditorUserId="115" LastEditDate="2014-05-14T11:03:51.577" LastActivityDate="2014-05-14T11:03:51.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27" PostTypeId="2" ParentId="20" CreationDate="2014-05-14T07:53:02.560" Score="10" Body="&lt;p&gt;To answer this question you have to answer which kind of compromise you can afford. RDBMs implements &lt;a href=&quot;http://en.wikipedia.org/wiki/ACID&quot;&gt;ACID&lt;/a&gt;. This is expensive in terms of resources. There are no NoSQL solutions which are ACID. See &lt;a href=&quot;http://en.wikipedia.org/wiki/CAP_theorem&quot;&gt;CAP theorem&lt;/a&gt; to dive deep into these ideas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have to understand each compromise given by each solution and choose the one which is the most appropriate for your problem.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="14" LastEditDate="2014-05-14T08:03:37.890" LastActivityDate="2014-05-14T08:03:37.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28" PostTypeId="2" ParentId="7" CreationDate="2014-05-14T07:55:40.133" Score="6" Body="&lt;p&gt;There is free ebook &quot;&lt;a href=&quot;http://jsresearch.net/&quot; rel=&quot;noreferrer&quot;&gt;Introduction to Data Science&lt;/a&gt;&quot; based on &lt;a href=&quot;/questions/tagged/r&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;r&amp;#39;&quot; rel=&quot;tag&quot;&gt;r&lt;/a&gt; language&lt;/p&gt;&#xA;" OwnerUserId="118" LastActivityDate="2014-05-14T07:55:40.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29" PostTypeId="2" ParentId="14" CreationDate="2014-05-14T07:56:34.437" Score="27" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/users/36/statsrus&quot;&gt;@statsRus&lt;/a&gt; starts to lay the groundwork for your answer in another question &lt;a href=&quot;https://datascience.meta.stackexchange.com/q/86/98307&quot;&gt;What characterises the difference between data science and statistics?&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data collection&lt;/strong&gt;: web scraping and online surveys&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data manipulation&lt;/strong&gt;: recoding messy data and extracting meaning from linguistic and social network data&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data scale&lt;/strong&gt;: working with extremely large data sets&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data mining&lt;/strong&gt;: finding patterns in large, complex data sets, with an emphasis on algorithmic techniques&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data communication&lt;/strong&gt;: helping turn &amp;quot;machine-readable&amp;quot; data into &amp;quot;human-readable&amp;quot; information via visualization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Definition&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;/questions/tagged/data-mining&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;data-mining&amp;#39;&quot; rel=&quot;tag&quot;&gt;data-mining&lt;/a&gt; can be seen as one item (or set of skills and applications) in the toolkit of the data scientist.  I like how he separates the definition of mining from collection in a sort of trade-specific jargon.&lt;/p&gt;&#xA;&lt;p&gt;However, I think that &lt;em&gt;data-mining&lt;/em&gt; would be synonymous with &lt;em&gt;data-collection&lt;/em&gt; in a US-English colloquial definition.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;As to where to go to become proficient?&lt;/em&gt;  I think that question is too broad as it is currently stated and would receive answers that are primarily opinion based.  Perhaps if you could refine your question, it might be easier to see what you are asking.&lt;/p&gt;&#xA;" OwnerUserId="53" LastEditorUserId="98307" LastEditDate="2020-08-16T13:01:33.543" LastActivityDate="2020-08-16T13:01:33.543" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30" PostTypeId="2" ParentId="19" CreationDate="2014-05-14T08:03:28.117" Score="22" Body="&lt;p&gt;Total amount of data in the world: 2.8 zetabytes in 2012, estimated to reach 8 zetabytes by 2015 (&lt;a href=&quot;http://siliconangle.com/blog/2012/05/21/when-will-the-world-reach-8-zetabytes-of-stored-data-infographic/&quot;&gt;source&lt;/a&gt;) and with a doubling time of 40 months. Can't get bigger than that :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of a single large organization, Facebook pulls in 500 terabytes per day, into a 100 petabyte warehouse, and runs 70k queries per day on it as of 2012 (&lt;a href=&quot;http://gigaom.com/2012/08/22/facebook-is-collecting-your-data-500-terabytes-a-day/&quot;&gt;source&lt;/a&gt;)  Their current warehouse is &gt;300 petabytes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Big data is probably something that is a good fraction of the Facebook numbers (1/100 probably yes, 1/10000 probably not: it's a spectrum not a single number).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to size, some of the features that make it &quot;big&quot; are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;it is actively analyzed, not just stored  (quote &quot;If you aren’t taking advantage of big data, then you don’t have big data, you have just a pile of data&quot; Jay Parikh @ Facebook)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;building and running a data warehouse is a major infrastructure project&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;it is growing at a significant rate&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;it is unstructured or has irregular structure&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Gartner definition: &quot;Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing&quot; (The 3Vs)  So they also think &quot;bigness&quot; isn't entirely about the size of the dataset, but also about the velocity and structure and the kind of tools needed.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2014-05-14T18:30:59.180" LastActivityDate="2014-05-14T18:30:59.180" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="31" PostTypeId="1" AcceptedAnswerId="72" CreationDate="2014-05-14T08:38:07.007" Score="10" ViewCount="1791" Body="&lt;p&gt;I have a bunch of customer profiles stored in a &lt;a href=&quot;/questions/tagged/elasticsearch&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;elasticsearch&amp;#39;&quot; rel=&quot;tag&quot;&gt;elasticsearch&lt;/a&gt; cluster. These profiles are now used for creation of target groups for our email subscriptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Target groups are now formed manually using elasticsearch faceted search capabilities (like  get all male customers of age 23 with one car and 3 children).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I search for interesting groups &lt;strong&gt;automatically&lt;/strong&gt; - using data science, machine learning, clustering or something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;/questions/tagged/r&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;r&amp;#39;&quot; rel=&quot;tag&quot;&gt;r&lt;/a&gt; programming language seems to be a good tool for this task, but I can't form a methodology of such group search. One solution is to somehow find the largest clusters of customers and use them as target groups, so the question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I automatically choose largest clusters of similar customers (similar by parameters that I don't know at this moment)?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: my program will connect to elasticsearch, offload customer data to CSV and using R language script will find that large portion of customers are male with no children and another large portion of customers have a car and their eye color is brown.&lt;/p&gt;&#xA;" OwnerUserId="118" LastEditorUserId="24" LastEditDate="2014-05-15T05:49:39.140" LastActivityDate="2022-10-21T03:12:52.913" Title="Clustering customer data stored in ElasticSearch" Tags="|data-mining|clustering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="33" PostTypeId="2" ParentId="20" CreationDate="2014-05-14T09:34:15.477" Score="6" Body="&lt;p&gt;Is it the time to move to NoSQL will depends on 2 things: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The nature/structure of your data&lt;/li&gt;&#xA;&lt;li&gt;Your current performance&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;SQL databases excel when the data is well structured (e.g. when it can be modeled as a table, an Excel spreadsheet, or a set of rows with a fixed number of columns). Also good when you need to do a lot of table joins (which it sounds like you do).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NoSQL databases excel when the data is un-structured beyond key-value pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performance wise, you gotta ask yourself one question: &lt;strong&gt;is your current SQL solution slow&lt;/strong&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, go with the &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bert_Lance#.22If_it_ain.27t_broke.2C_don.27t_fix_it..22&quot; rel=&quot;nofollow noreferrer&quot;&gt;IIABDFI&lt;/a&gt;&quot; principle. &lt;/p&gt;&#xA;" OwnerUserId="132" LastEditorUserId="132" LastEditDate="2017-08-29T11:26:37.137" LastActivityDate="2017-08-29T11:26:37.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="35" PostTypeId="1" CreationDate="2014-05-14T09:51:54.753" Score="21" ViewCount="750" Body="&lt;p&gt;In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat.   The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context.  I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can the algorithm development itself be sped up and made more scalable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some specific challenges: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can the number of iterations be reduced?  (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to run on bigger datasets during development?  (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can algorithm parameters be tuned faster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to apply machine learning type tools to algorithm development itself?  (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2014-05-20T03:56:43.147" Title="How to scale up algorithm development?" Tags="|algorithms|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="37" PostTypeId="2" ParentId="19" CreationDate="2014-05-14T10:41:23.823" Score="94" Body="&lt;p&gt;To me (coming from a relational database background), &quot;Big Data&quot; is not primarily about the data size (which is the bulk of what the other answers are so far).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Big Data&quot; and &quot;Bad Data&quot; are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require &quot;Great Data&quot;  and a huge amount of time, money, and accountability is put on to making sure the data is well prepared before loading it in to the database. If the data is in the database, it is 'gospel', and it defines the system understanding of reality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Big Data&quot; tackles this problem from the other direction. The data is poorly defined, much of it may be inaccurate, and much of it may in fact be missing. The structure and layout of the data is linear as opposed to relational.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Big Data has to have enough volume so that the amount of bad data, or missing data becomes statistically insignificant. When the errors in your data are common enough to cancel each other out, when the missing data is proportionally small enough to be negligible and when your data access requirements and algorithms are functional even with incomplete and inaccurate data, then you have &quot;Big Data&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Big Data&quot; is not really about the volume, it is about the characteristics of the data.&lt;/p&gt;&#xA;" OwnerUserId="9" LastEditorUserId="51450" LastEditDate="2018-05-01T13:04:43.563" LastActivityDate="2018-05-01T13:04:43.563" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="38" PostTypeId="1" AcceptedAnswerId="43" CreationDate="2014-05-14T10:44:58.933" Score="15" ViewCount="3626" Body="&lt;p&gt;I heard about many tools / frameworks for helping people to process their data (big data environment). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One is called Hadoop and the other is the noSQL concept. What is the difference in point of processing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are they complementary?&lt;/p&gt;&#xA;" OwnerUserId="134" LastEditorUserId="134" LastEditDate="2014-05-14T22:26:59.453" LastActivityDate="2015-05-18T12:30:19.497" Title="What is the difference between Hadoop and noSQL" Tags="|nosql|tools|processing|apache-hadoop|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="40" PostTypeId="2" ParentId="20" CreationDate="2014-05-14T11:12:03.880" Score="8" Body="&lt;p&gt;Big Data is actually not so about the &quot;how big it is&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, few gigabytes is not big at all, it's almost nothing. So don't bother yourself, your system will continu to work efficiently for some time I think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have to think of how do you use your data. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SQL approach: Every data is precious, well collected and selected, and the focus is put on storing high valuable and well structured data. This can be costly, everything is interlink, and it's good for well stuctured system and functionnal data.&lt;/li&gt;&#xA;&lt;li&gt;Big Data approach: In big data you basically store almost everything, regardless of the value it has, and then do a active analytics process. Things are not linked, they are copied. For example let's say I have a blog entry. In Big Data there will not be a link to its author, but the author will be embedded inside the blog entry. Way more scalable, but require a different and more complex approach.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If your storing &quot;functionnal&quot; data use by your application, I will suggest you to remain on SQL. If your storing data in order to search on them later or to do reporting, and if this amount of data may increase quickly, I will suggest big data.&#xA;In my opinion, big data is useful when you are dealing with real data that have to be collect and analyzed continuously.&lt;/p&gt;&#xA;" OwnerUserId="104" LastActivityDate="2014-05-14T11:12:03.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="41" PostTypeId="1" AcceptedAnswerId="44" CreationDate="2014-05-14T11:15:40.907" Score="55" ViewCount="10574" Body="&lt;p&gt;R has many libraries which are aimed at Data Analysis (e.g. JAGS, BUGS, ARULES etc..), and is mentioned in popular textbooks such as: J.Krusche, Doing Bayesian Data Analysis; B.Lantz, &quot;Machine Learning with R&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen a guideline of 5TB for a dataset to be considered as Big Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Is R suitable for the amount of Data typically seen in Big Data problems? &#xA;Are there strategies to be employed when using R with this size of dataset?&lt;/p&gt;&#xA;" OwnerUserId="136" LastEditorUserId="118" LastEditDate="2014-05-14T13:06:28.407" LastActivityDate="2019-02-23T11:34:41.513" Title="Is the R language suitable for Big Data" Tags="|bigdata|r|" AnswerCount="9" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="42" PostTypeId="2" ParentId="38" CreationDate="2014-05-14T11:21:31.500" Score="5" Body="&lt;p&gt;NoSQL is a way to store data that does not require there to be some sort of relation. The simplicity of its design and horizontal scale-ability, one way they store data is the &lt;code&gt;key : value&lt;/code&gt; pair design. This lends itself to processing that is similar to Hadoop. The use of a NoSQL db really depends on the type of problem that one is after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a good wikipedia link &lt;a href=&quot;https://en.wikipedia.org/wiki/NoSQL&quot; rel=&quot;noreferrer&quot;&gt;NoSQL&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadoop is a system that is meant to store and process huge chunks of data. It is a distributed file system dfs. The reason it does this is that central to its design it makes the assumption that hardware failures are common, thus making multiple copies of the same piece of information and spreading it across multiple machines and racks, so if one goes down, no problem, we have two more copies. Here is a great link for Hadoop from wikipedia as well, you will see that it is, in my opinion more than just storage, but also processing:&#xA; &lt;a href=&quot;https://en.wikipedia.org/wiki/Apache_Hadoop&quot; rel=&quot;noreferrer&quot;&gt;Hadoop&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2014-05-14T11:21:31.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="43" PostTypeId="2" ParentId="38" CreationDate="2014-05-14T11:23:25.913" Score="16" Body="&lt;p&gt;&lt;strong&gt;Hadoop is not a database&lt;/strong&gt;, hadoop is an entire ecosystem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/oOYp7.png&quot; alt=&quot;the hadoop ecosystem&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most people will refer to &lt;a href=&quot;http://de.wikipedia.org/wiki/MapReduce&quot; rel=&quot;nofollow noreferrer&quot;&gt;mapreduce&lt;/a&gt; jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Let's assume you load into hadoop a set of &lt;code&gt;&amp;lt;String, Integer&amp;gt;&lt;/code&gt; with the population of some neighborhoods within a city and you want to get the average population over the whole neighborhoods of each city(figure 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;figure 1&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    [new york, 40394]&#xA;    [new york, 134]&#xA;    [la, 44]&#xA;    [la, 647]&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now hadoop will first map each value by using the keys (figure 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;figure 2&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[new york, [40394,134]]&#xA;[la, [44,647]]&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After the mapping it will reduce the values of each key to a new value (in this example the average over the value set of each key)(figure 3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;figure 3&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[new york, [20264]]&#xA;[la, [346]]&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now hadoop would be done with everything. You can now load the result into the HDFS (hadoop distributed file system) or into any DBMS or file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thats just one &lt;strong&gt;very basic&lt;/strong&gt; and &lt;strong&gt;simple&lt;/strong&gt; example of what hadoop can do. You can run much more complicated tasks in hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you already mentioned in your question, hadoop and noSQL are complementary. I know a few setups where i.e. billions of datasets from sensors are stored in HBase and get then through hadoop to finally be stored in a DBMS.&lt;/p&gt;&#xA;" OwnerUserId="115" LastEditorUserId="115" LastEditDate="2015-05-18T12:30:19.497" LastActivityDate="2015-05-18T12:30:19.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="44" PostTypeId="2" ParentId="41" CreationDate="2014-05-14T11:24:39.530" Score="43" Body="&lt;p&gt;Actually this is coming around. In the book R in a Nutshell there is even a section on using R with Hadoop for big data processing. There are some work arounds that need to be done because R does all it's work in memory, so you are basically limited to the amount of RAM you have available to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mature project for R and Hadoop is &lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop&quot;&gt;RHadoop&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RHadoop has been divided into several sub-projects, rhdfs, rhbase, rmr2, plyrmr, and quickcheck (&lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop/wiki&quot;&gt;wiki&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="59" LastEditorUserId="2522" LastEditDate="2015-01-31T11:34:03.700" LastActivityDate="2015-01-31T11:34:03.700" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="45" PostTypeId="2" ParentId="35" CreationDate="2014-05-14T11:26:40.580" Score="7" Body="&lt;p&gt;First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies according to some criteria, usually provides with a better understanding (more consise and precise) than simply inferring algorithms on a brute-force fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, answering each question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;characterization is definitely a means of reducing the number of &lt;em&gt;iterations&lt;/em&gt; while trying to select proper algorithms for specific data;&lt;/li&gt;&#xA;&lt;li&gt;if you have a discrete set of criterias on which your data varies, it becomes much easier to &lt;em&gt;scale up&lt;/em&gt; solutions, as will know what information you'd gain/lose if simpler/specific solutions were applied;&lt;/li&gt;&#xA;&lt;li&gt;after a characterization, you should be also easier to select parameters, since you'd know what kind of &lt;em&gt;specific data&lt;/em&gt; you'd be dealing with;&lt;/li&gt;&#xA;&lt;li&gt;finally, you may use data mining/machine learning algorithms to support this characterization. This includes using:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;clustering algorithms, to reduce the dimensionality of data;&lt;/li&gt;&#xA;&lt;li&gt;classification algorithms, to help deciding on specific properties the data in function of time/context/... may present;&lt;/li&gt;&#xA;&lt;li&gt;association rules, to predict particular knowledge from the dataset, while also improving/fine-graining the data used for later analysis;&lt;/li&gt;&#xA;&lt;li&gt;and other possible strategies and analyses.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And &lt;a href=&quot;http://www.cs.cmu.edu/Groups/sage/sagedc.html&quot;&gt;here&lt;/a&gt; is a list of some criterias on which to analyse data, which you may find helpful.&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-05-14T11:33:27.770" LastActivityDate="2014-05-14T11:33:27.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="46" PostTypeId="2" ParentId="16" CreationDate="2014-05-14T12:32:29.503" Score="11" Body="&lt;p&gt;Note that there is an early version of LIBLINEAR ported to &lt;a href=&quot;http://spark.apache.org&quot;&gt;Apache Spark&lt;/a&gt;. See &lt;a href=&quot;http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-td5546.html&quot;&gt;mailing list comments&lt;/a&gt; for some early details, and the &lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/distributed-liblinear/&quot;&gt;project site&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="21" LastEditDate="2014-05-14T21:03:05.313" LastActivityDate="2014-05-14T21:03:05.313" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="47" PostTypeId="2" ParentId="41" CreationDate="2014-05-14T12:39:41.197" Score="31" Body="&lt;p&gt;The main problem with using R for large data sets is the RAM constraint. The reason behind keeping all the data in RAM is that it provides much faster access and data manipulations than would storing on HDDs. If you are willing to take a hit on performance, then yes, it is quite practical to work with &lt;a href=&quot;http://statistics.org.il/wp-content/uploads/2010/04/Big_Memory%20V0.pdf&quot;&gt;large datasets in R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RODBC Package: Allows connecting to external DB from R to retrieve and handle data. Hence, the data being &lt;em&gt;manipulated&lt;/em&gt; is restricted to your RAM. The overall data set can go much larger.&lt;/li&gt;&#xA;&lt;li&gt;The ff package allows using larger than RAM data sets by utilising memory-mapped pages.&lt;/li&gt;&#xA;&lt;li&gt;BigLM: It builds generalized linear models on big data. It loads data into memory in chunks.&lt;/li&gt;&#xA;&lt;li&gt;bigmemory : An R package which allows powerful and memory-efficient parallel &#xA;analyses and data mining of massive data sets. It permits storing large objects (matrices etc.) in memory (on the RAM) using external pointer objects to refer to them. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="62" LastActivityDate="2014-05-14T12:39:41.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="48" PostTypeId="5" CreationDate="2014-05-14T13:08:26.647" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-project.org&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&amp;amp;T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of R's strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R was created by &lt;a href=&quot;http://www.stat.auckland.ac.nz/~ihaka/&quot; rel=&quot;nofollow&quot;&gt;Ross Ihaka&lt;/a&gt; and &lt;a href=&quot;http://www.gene.com/scientists/our-scientists/robert-gentleman&quot; rel=&quot;nofollow&quot;&gt;Robert Gentleman&lt;/a&gt; and is now developed by the &lt;a href=&quot;http://www.r-project.org/contributors.html&quot; rel=&quot;nofollow&quot;&gt;R Development Core Team&lt;/a&gt;. The R environment is easily extended through a packaging system on &lt;a href=&quot;http://cran.r-project.org&quot; rel=&quot;nofollow&quot; title=&quot;CRAN The Comprehensive R Archive Network&quot;&gt;CRAN&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is available as Free Software under the terms of the Free Software Foundation's GNU General Public License in source code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows and Mac OS.&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="2961" LastEditDate="2014-08-16T17:29:43.517" LastActivityDate="2014-08-16T17:29:43.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="49" PostTypeId="4" CreationDate="2014-05-14T13:08:26.647" Score="0" Body="R is a free, open-source programming language and software environment for statistical computing, bioinformatics, and graphics." OwnerUserId="2961" LastEditorUserId="2961" LastEditDate="2014-08-15T16:38:27.880" LastActivityDate="2014-08-15T16:38:27.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="50" PostTypeId="1" AcceptedAnswerId="53" CreationDate="2014-05-14T14:26:54.313" Score="5" ViewCount="721" Body="&lt;p&gt;I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning? Or perhaps I want it to re-run once a certain number of new records have been added to the database. How might I go about automating this? I should mention I'm on Windows, but I could easily put this script on my Linux machine if that would simplify the process. &lt;/p&gt;&#xA;" OwnerUserId="151" LastActivityDate="2014-05-14T15:42:02.393" Title="Running an R script programmatically" Tags="|r|databases|efficiency|tools|" AnswerCount="2" CommentCount="1" ClosedDate="2017-08-29T10:27:14.067" ContentLicense="CC BY-SA 3.0" />
  <row Id="51" PostTypeId="2" ParentId="50" CreationDate="2014-05-14T14:48:32.180" Score="9" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How can I ask my computer to run this every night at 4 am so that I have an up to date report waiting for me in the morning?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can set up a cronjob on a Linux system. These are run at the set time, if the computer is on. To do so, open a terminal and type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;crontab -e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And add:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;00 4 * * * r source(/home/FilePath/.../myRscript.R)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://stackoverflow.com/a/14710307/1488917&quot;&gt;Stack Overflow&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="62" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-05-14T14:57:32.243" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="52" PostTypeId="1" CreationDate="2014-05-14T15:25:21.700" Score="37" ViewCount="5051" Body="&lt;p&gt;From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any best practices or processes for cleaning data before processing it? If so, are there any automated or semi-automated tools which implement some of these best practices?&lt;/p&gt;&#xA;" OwnerUserId="157" LastEditorUserId="136" LastEditDate="2014-05-16T13:45:07.447" LastActivityDate="2017-01-23T06:27:10.313" Title="Organized processes to clean data" Tags="|r|data-cleaning|" AnswerCount="7" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="53" PostTypeId="2" ParentId="50" CreationDate="2014-05-14T15:42:02.393" Score="10" Body="&lt;p&gt;For windows, use the task scheduler to set the task to run for example daily at 4:00 AM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It gives you many other options regarding frequency etc.&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Windows_Task_Scheduler&quot;&gt;http://en.wikipedia.org/wiki/Windows_Task_Scheduler&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="116" LastActivityDate="2014-05-14T15:42:02.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="57" PostTypeId="2" ParentId="52" CreationDate="2014-05-14T16:29:39.927" Score="18" Body="&lt;p&gt;From my point of view, this question is suitable for a two-step answer. The first part, let us call it &lt;em&gt;soft preprocessing&lt;/em&gt;, could be taken as the usage of different data mining algorithms to preprocess data in such a way that makes it suitable for further analyses. Notice that this could be the analysis itself, in case the goal is simple enough to be tackled in a single shot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second part, the &lt;em&gt;hard preprocessing&lt;/em&gt;, actually comes prior to any other process, and is may be taken as the usage of simple tools or scripts to clean up data, selecting specific contents to be processed. To this problem, POSIX provides us with a wonderous set of magic tools, which can be used to compose concise -- and very powerful -- preprocessing scripts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for people who deal with data coming from social websites (twitter, facebook, ...), the &lt;em&gt;data retrieval&lt;/em&gt; usually yields files with very specific format -- although not always nicely structure, as they may contain missing fields, and so. For these cases, a simple &lt;code&gt;awk&lt;/code&gt; script could clean up the data, producing a &lt;em&gt;valid&lt;/em&gt; input file for later processing. From the magic set, one may also point out &lt;code&gt;grep&lt;/code&gt;, &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;cut&lt;/code&gt;, &lt;code&gt;join&lt;/code&gt;, &lt;code&gt;paste&lt;/code&gt;, &lt;code&gt;sort&lt;/code&gt;, and a whole multitude of other tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case simple the source file has too many nitty-gritties, it may also be necessary to produce a bundle of methods to clean up data. In such cases, it is usually better to use scripting languages (other than shell ones), such as Python, Ruby, and Perl. This allows for building up &lt;em&gt;API&lt;/em&gt;'s to select specific data in a very straightforward and reusable way. Such &lt;em&gt;API&lt;/em&gt;'s are sometimes made public by their writers, such as &lt;a href=&quot;http://imdbpy.sourceforge.net/&quot;&gt;IMDbPY&lt;/a&gt;, &lt;a href=&quot;https://api.stackexchange.com/docs&quot;&gt;Stack Exchange API&lt;/a&gt;, and many others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, answering the question: are there any best practices? It usually depends on your task. If you will always deal with the same data format, it's commonly best to write an &lt;em&gt;organized&lt;/em&gt; script to preprocess it; whereas, if you just need a simple and fast clean up on some dataset, count on POSIX tools for concise shell scripts that will do the whole job &lt;strong&gt;much&lt;/strong&gt; faster than a Python script, or so. Since the &lt;em&gt;clean up&lt;/em&gt; depends both on the dataset and on your purposes, it's hard to have everything already done. Yet, there are lots of API's that puts you halfway through with the problem.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-05-14T16:29:39.927" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="58" PostTypeId="2" ParentId="16" CreationDate="2014-05-14T17:06:33.337" Score="10" Body="&lt;p&gt;You can check out &lt;a href=&quot;http://hunch.net/~vw/&quot;&gt;vowpal wabbit&lt;/a&gt;. It is quite popular for large-scale learning and includes parallel provisions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From their website:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="119" LastActivityDate="2014-05-14T17:06:33.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="59" PostTypeId="1" AcceptedAnswerId="316" CreationDate="2014-05-14T17:48:21.240" Score="11" ViewCount="1335" Body="&lt;p&gt;In reviewing “&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1461468485&quot; rel=&quot;noreferrer&quot;&gt;Applied Predictive Modeling&lt;/a&gt;&quot; a &lt;a href=&quot;http://www.information-management.com/blogs/applied-predictive-modeling-10024771-1.html&quot; rel=&quot;noreferrer&quot;&gt;reviewer states&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One critique I have of statistical learning (SL) pedagogy is the&#xA;  absence of computation performance considerations in the evaluation of&#xA;  different modeling techniques. With its emphases on bootstrapping and&#xA;  cross-validation to tune/test models, SL is quite compute-intensive.&#xA;  Add to that the re-sampling that's embedded in techniques like bagging&#xA;  and boosting, and you have the specter of computation hell for&#xA;  supervised learning of large data sets. &lt;strong&gt;In fact, R's memory&#xA;  constraints impose pretty severe limits on the size of models that can&#xA;  be fit by top-performing methods like random forests.&lt;/strong&gt; Though SL does a&#xA;  good job calibrating model performance against small data sets, it'd&#xA;  sure be nice to understand performance versus computational cost for&#xA;  larger data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What are R's memory constraints, and do they impose severe limits on the size of models that can be fit by top-performing methods like &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;noreferrer&quot;&gt;random forests&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="62" LastEditDate="2014-07-26T15:10:51.000" LastActivityDate="2014-07-26T15:10:51.000" Title="What are R's memory constraints?" Tags="|apache-hadoop|r|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="60" PostTypeId="2" ParentId="59" CreationDate="2014-05-14T17:58:48.297" Score="8" Body="&lt;p&gt;R performs all computation in-memory so you can't perform operation on a dataset that is larger than available RAM amount. However there are some libraries that allow bigdata processing using R and one of popular libraries for bigdata processing like Hadoop.&lt;/p&gt;&#xA;" OwnerUserId="118" LastActivityDate="2014-05-14T17:58:48.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="61" PostTypeId="1" AcceptedAnswerId="62" CreationDate="2014-05-14T18:09:01.940" Score="57" ViewCount="17511" Body="&lt;p&gt;Logic often states that by overfitting a model, its capacity to generalize is limited, though this might only mean that overfitting stops a model from improving after a certain complexity. Does overfitting cause models to become worse regardless of the complexity of data, and if so, why is this the case?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Related:&lt;/strong&gt; Followup to the question above, &quot;&lt;a href=&quot;https://datascience.stackexchange.com/questions/361/when-is-a-model-underfitted&quot;&gt;When is a Model Underfitted?&lt;/a&gt;&quot;&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-09-17T02:27:31.110" Title="Why Is Overfitting Bad in Machine Learning?" Tags="|machine-learning|predictive-modeling|" AnswerCount="8" CommentCount="16" ContentLicense="CC BY-SA 3.0" />
  <row Id="62" PostTypeId="2" ParentId="61" CreationDate="2014-05-14T18:27:56.043" Score="49" Body="&lt;p&gt;Overfitting is &lt;em&gt;empirically&lt;/em&gt; bad.  Suppose you have a data set which you split in two, test and training.  An overfitted model is one that performs much worse on the test dataset than on training dataset.  It is often observed that models like that also in general perform worse on additional (new) test datasets than models which are not overfitted.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to understand that intuitively is that a model may use some relevant parts of the data (signal) and some irrelevant parts (noise).  An overfitted model uses more of the noise, which increases its performance in the case of known noise (training data) and decreases its performance in the case of novel noise (test data).  The difference in performance between training and test data indicates how much noise the model picks up; and picking up noise directly translates into worse performance on test data (including future data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary: overfitting is bad by definition, this has not much to do with either complexity or ability to generalize, but rather has to do with mistaking noise for signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. On the &quot;ability to generalize&quot; part of the question, it is very possible to have a model which has inherently limited ability to generalize due to the structure of the model (for example linear SVM, ...) but is still prone to overfitting.  In a sense overfitting is just one way that generalization may fail.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2015-02-12T07:08:27.463" LastActivityDate="2015-02-12T07:08:27.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="64" PostTypeId="2" ParentId="61" CreationDate="2014-05-14T18:37:52.333" Score="18" Body="&lt;p&gt;Overfitting, in a nutshell, means take into account &lt;strong&gt;too much&lt;/strong&gt; information from your data and/or prior knowledge, and use it in a model. To make it more straightforward, consider the following example: you're hired by some scientists to provide them with a model to predict the growth of some kind of plants. The scientists have given you information collected from their work &#xA;with such plants throughout a whole year, and they shall continuously give you information on the future development of their plantation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you run through the data received, and build up a model out of it. Now suppose that, in your model, you considered just as many characteristics as possible to always find out the exact behavior of the plants you saw in the initial dataset. Now, as the production continues, you'll always take into account those characteristics, and will produce very &lt;em&gt;fine-grained&lt;/em&gt; results. However, if the plantation eventually suffer from some seasonal change, the results you will receive may fit your model in such a way that your predictions will begin to fail (either saying that the growth will slow down, while it shall actually speed up, or the opposite).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from being unable to detect such small variations, and to usually classify your entries incorrectly, the &lt;em&gt;fine-grain&lt;/em&gt; on the model, i.e., the great amount of variables, may cause the processing to be too costly. Now, imagine that your data is already complex. Overfitting your model to the data not only will make the classification/evaluation very complex, but will most probably make you error the prediction over the slightest variation you may have on the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: &lt;a href=&quot;https://www.youtube.com/watch?v=DQWI1kvmwRg&quot;&gt;This&lt;/a&gt; might as well be of some use, perhaps adding dynamicity to the above explanation :D&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-05-15T23:22:39.427" LastActivityDate="2014-05-15T23:22:39.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="65" PostTypeId="5" CreationDate="2014-05-14T18:45:23.917" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2014-05-14T18:45:23.917" LastActivityDate="2014-05-14T18:45:23.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="66" PostTypeId="4" CreationDate="2014-05-14T18:45:23.917" Score="0" Body="Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization." OwnerUserId="118" LastEditorUserId="118" LastEditDate="2014-05-16T13:45:57.450" LastActivityDate="2014-05-16T13:45:57.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="67" PostTypeId="5" CreationDate="2014-05-14T18:48:42.263" Score="0" Body="&lt;p&gt;&lt;strong&gt;GNU Octave&lt;/strong&gt; is a high-level interpreted scripting language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. Octave has a modular framework that allows it to utilize hardware such as GPUs, parallel compute clusters, and physical sensors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Octave is normally used through its interactive command line interface, but it can also be used to write non-interactive programs. It is quite similar to Matlab, so many programs are easily portable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is extensible and customizable via user-defined functions written in Octave's own language, or via dynamically loaded modules written in C++, C, Fortran, or other languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://octave-online.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Octave Online&lt;/a&gt;&lt;/strong&gt; is a free GNU Octave editor and prompt in your browser. Octave Online is powered by the open-source GNU Octave project, whose syntax is mostly compatible with Matlab.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Resources&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.gnu.org/software/octave/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Official site&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/GNU_Octave&quot; rel=&quot;nofollow noreferrer&quot;&gt;Octave on Wikipedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://octave.org/doc/interpreter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Octave Documentation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://wiki.octave.org/GNU_Octave_Wiki&quot; rel=&quot;nofollow noreferrer&quot;&gt;Octave Wiki&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://octave.1599824.n4.nabble.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Octave Forum on Nabble&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T17:28:07.320" LastActivityDate="2019-04-08T17:28:07.320" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="68" PostTypeId="4" CreationDate="2014-05-14T18:48:42.263" Score="0" Body="GNU Octave is a free and open-source mathematical software package and scripting language. The scripting language is intended to be compatible with MATLAB, but the two packages are not interchangeable. Don’t use both the [matlab] and [octave] tags, unless the question is explicitly about the similarities or differences between the two." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T17:28:15.870" LastActivityDate="2019-04-08T17:28:15.870" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="69" PostTypeId="1" CreationDate="2014-05-14T20:03:15.233" Score="3" ViewCount="93" Body="&lt;p&gt;First, think it's worth me stating what I mean by replication &amp;amp; reproducibility:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Replication of analysis A results in an exact copy of all inputs and processes that are supply and result in incidental outputs in analysis B.&lt;/li&gt;&#xA;&lt;li&gt;Reproducibility of analysis A results in inputs, processes, and outputs that are semantically incidental to analysis A, without access to the exact inputs and processes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Putting aside how easy it might be to replicate a given build, especially an ad-hoc one, to me replication always possible if it's planned for and worth doing. That said, it is unclear to me is how to execute a data science workflow that allows for reproducibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closet comparison I'm able to think of is &lt;a href=&quot;http://en.wikipedia.org/wiki/Documentation_generator&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation generators&lt;/a&gt; that generates software documentation intended for programmers - though the main difference I see is that in theory, if two sets of analysis ran the &quot;reproducibility documentation generators&quot; the documentation should match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another issue, is that while I get the concept of reproducibility documentation, I am having a hard time imagining what it would look like in usable form without just being a guide to replicating the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, whole intent of this is to understand if it's possible to &quot;bake-in&quot; reproducibility documentation as you build out a stack, not after the stack is built.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Is it possible to automate generating reproducibility documentation, and if so how, and what would it look like?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;UPDATE:&lt;/strong&gt; Please note that this is the second draft of this question and that &lt;a href=&quot;https://datascience.stackexchange.com/users/178/christopher-louden&quot;&gt;Christopher Louden&lt;/a&gt; was kind enough to let me edit the question after I realized it was likely the first draft was unclear. Thanks!&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-05-15T02:02:08.010" Title="Is it possible to automate generating reproducibility documentation?" Tags="|processing|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="70" PostTypeId="2" ParentId="69" CreationDate="2014-05-14T22:03:50.597" Score="2" Body="&lt;p&gt;To be reproducible without being just a replication, you would need to redo the experiment with new data, following the same technique as before.  The work flow is not as important as the techniques used.  Sample data in the same way, use the same type of models.  It doesn't matter if you switch from one language to another, so long as the models and the data manipulations are the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This type of replication will show that the results you got in the first experiment are less likely to be a fluke than they were earlier.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-05-14T22:03:50.597" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="71" PostTypeId="1" AcceptedAnswerId="84" CreationDate="2014-05-14T22:12:37.203" Score="14" ViewCount="776" Body="&lt;p&gt;What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance?  Are there specific problem types that fall into this category?&lt;/p&gt;&#xA;" OwnerUserId="179" LastActivityDate="2014-05-15T08:25:47.933" Title="When are p-values deceptive?" Tags="|bigdata|statistics|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="72" PostTypeId="2" ParentId="31" CreationDate="2014-05-14T22:40:40.363" Score="6" Body="&lt;p&gt;One algorithm that can be used for this is the &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means clustering algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Basically:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Randomly choose &lt;code&gt;k&lt;/code&gt; datapoints from your set, &lt;span class=&quot;math-container&quot;&gt;$m_1$&lt;/span&gt;, ..., &lt;span class=&quot;math-container&quot;&gt;$m_k$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Until convergence:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign your data points to k clusters, where cluster i is the set of points for which m_i is the closest of your current means&lt;/li&gt;&#xA;&lt;li&gt;Replace each &lt;span class=&quot;math-container&quot;&gt;$m_i$&lt;/span&gt; by the mean of all points assigned to cluster i.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;It is good practice to repeat this algorithm several times, then choose the outcome that minimizes distances between the points of each cluster i and the center &lt;span class=&quot;math-container&quot;&gt;$m_i$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Of course, you have to know &lt;code&gt;k&lt;/code&gt; to start here; you can use cross-validation to choose this parameter, though.&lt;/p&gt;&#xA;" OwnerUserId="22" LastEditorUserId="141355" LastEditDate="2022-10-21T03:12:52.913" LastActivityDate="2022-10-21T03:12:52.913" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="73" PostTypeId="2" ParentId="71" CreationDate="2014-05-14T22:43:23.587" Score="5" Body="&lt;p&gt;You shouldn't consider the p-value out of context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One rather basic point (as illustrated by &lt;a href=&quot;http://xkcd.com/882/&quot; rel=&quot;noreferrer&quot;&gt;xkcd&lt;/a&gt;) is that you need to consider how many tests you're actually doing.  Obviously, you shouldn't be shocked to see p &amp;lt; 0.05 for one out of 20 tests, even if the null hypothesis is true every time.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more subtle example of this occurs in high-energy physics, and is known as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Look-elsewhere_effect&quot; rel=&quot;noreferrer&quot;&gt;look-elsewhere effect&lt;/a&gt;.  The larger the parameter space you search for a signal that might represent a new particle, the more likely you are to see an apparent signal that's really just due to random fluctuations. &lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2014-05-14T22:43:23.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="74" PostTypeId="2" ParentId="71" CreationDate="2014-05-14T22:58:11.583" Score="2" Body="&lt;p&gt;One thing you should be aware of is the sample size you are using. Very large samples, such as economists using census data, will lead to deflated p-values. This paper &lt;a href=&quot;http://galitshmueli.com/system/files/Print%20Version.pdf&quot; rel=&quot;nofollow&quot; title=&quot;Too Big to Fail: Large Samples and the p-Value Problem&quot;&gt;&quot;Too Big to Fail: Large Samples and the p-Value Problem&quot;&lt;/a&gt; covers some of the issues. &lt;/p&gt;&#xA;" OwnerUserId="64" LastActivityDate="2014-05-14T22:58:11.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="75" PostTypeId="1" AcceptedAnswerId="78" CreationDate="2014-05-15T00:26:11.387" Score="5" ViewCount="176" Body="&lt;p&gt;If small p-values are plentiful in big data, what is a comparable replacement for p-values in data with million of samples?&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="1330" LastEditDate="2019-05-07T04:16:29.673" LastActivityDate="2019-05-07T04:16:29.673" Title="Is there a replacement for small p-values in big data?" Tags="|statistics|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="76" PostTypeId="1" AcceptedAnswerId="139" CreationDate="2014-05-15T00:39:33.433" Score="6" ViewCount="188" Body="&lt;p&gt;&lt;em&gt;(Note: Pulled this question from the &lt;a href=&quot;http://area51.stackexchange.com/proposals/55053/data-science/57398#57398&quot;&gt;list of questions in Area51&lt;/a&gt;, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able to field any questions on the question that might pop-up.)&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="118" LastEditDate="2014-05-18T15:18:08.050" LastActivityDate="2014-05-18T15:18:08.050" Title="Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?" Tags="|bigdata|tools|data-stream-mining|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="77" PostTypeId="1" AcceptedAnswerId="87" CreationDate="2014-05-15T01:22:35.167" Score="10" ViewCount="1240" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; Following is from the book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1449356265&quot;&gt;Graph Databases&lt;/a&gt;, which covers a performance test mentioned in the book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1617290769&quot;&gt;Neo4j in Action&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Relationships in a graph naturally form paths. Querying, or&#xA;  traversing, the graph involves following paths. Because of the&#xA;  fundamentally path-oriented nature of the datamodel, the majority of&#xA;  path-based graph database operations are highly aligned with the way&#xA;  in which the data is laid out, making them extremely efficient. In&#xA;  their book Neo4j in Action, Partner and Vukotic perform an experiment&#xA;  using a relational store and Neo4j.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The comparison shows that the graph database is substantially quicker&#xA;  for connected data than a relational store.Partner and Vukotic’s&#xA;  experiment seeks to find friends-of-friends in a social network, to a&#xA;  maximum depth of five. Given any two persons chosen at random, is&#xA;  there a path that connects them which is at most five relationships&#xA;  long? For a social network containing 1,000,000 people, each with&#xA;  approximately 50 friends, the results strongly suggest that graph&#xA;  databases are the best choice for connected data, as we see in Table&#xA;  2-1.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Table 2-1. Finding extended friends in a relational database versus efficient finding in Neo4j&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Depth   RDBMS Execution time (s)    Neo4j Execution time (s)     Records returned&#xA;2       0.016                       0.01                         ~2500    &#xA;3       30.267                      0.168                        ~110,000 &#xA;4       1543.505                    1.359                        ~600,000 &#xA;5       Unfinished                  2.132                        ~800,000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;At depth two (friends-of-friends) both the relational database and the graph database perform well enough for us to consider using them in an online system. While the Neo4j query runs in two-thirds the time of the relational one, an end-user would barely notice the the difference in milliseconds between the two. By the time we reach depth three (friend-of-friend-of-friend), however, it’s clear that the relational database can no longer deal with the query in a reasonable timeframe: the thirty seconds it takes to complete would be completely unacceptable for an online system. In contrast, Neo4j’s response time remains relatively flat: just a fraction of a second to perform the query—definitely quick enough for an online system.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;At depth four the relational database exhibits crippling latency,&#xA;  making it practically useless for an online system. Neo4j’s timings&#xA;  have deteriorated a little too, but the latency here is at the&#xA;  periphery of being acceptable for a responsive online system. Finally,&#xA;  at depth five, the relational database simply takes too long to&#xA;  complete the query. Neo4j, in contrast, returns a result in around two&#xA;  seconds. At depth five, it transpires almost the entire network is our&#xA;  friend: for many real-world use cases, we’d likely trim the results,&#xA;  and the timings.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions are:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is this a reasonable test to emulate what one might except to find in a social network? &lt;em&gt;(Meaning do real social networks normally have nodes with approximately 50 friends for example; seems like the &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/The_rich_get_richer_%28statistics%29&quot;&gt;rich get richer&lt;/a&gt;&quot; model would be more natural for social networks, though might be wrong.)&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Regardless of the naturalness of the emulation, is there any reason to believe the results are off, or unreproducible? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="158" LastEditorUserId="118" LastEditDate="2014-05-15T13:15:02.727" LastActivityDate="2015-05-10T21:18:01.617" Title="Is this Neo4j comparison to RDBMS execution time correct?" Tags="|databases|nosql|neo4j|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="78" PostTypeId="2" ParentId="75" CreationDate="2014-05-15T01:46:28.467" Score="7" Body="&lt;p&gt;There is no replacement in the strict sense of the word.  Instead you should look at other measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other measures you look at depend on what you type of problem you are solving.  In general, if you have a small p-value, also consider the magnitude of the effect size.  It may be highly statistically significant but in practice meaningless.  It is also helpful to report the confidence interval of the effect size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would consider &lt;a href=&quot;http://galitshmueli.com/system/files/Print%20Version.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; as mentoned in DanC's answer to &lt;a href=&quot;https://datascience.stackexchange.com/questions/71/when-are-p-values-deceptive&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="178" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-05-15T01:46:28.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="79" PostTypeId="5" CreationDate="2014-05-15T03:19:40.360" Score="0" Body="&lt;p&gt;Conceptually speaking, &lt;em&gt;data-mining&lt;/em&gt; can be thought of as one item (or set of skills and applications) in the toolkit of the data scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, data-mining is an activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In US-English colloquial speech, data-mining and data-collection are often used interchangeably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a main difference between these two related activities is &lt;em&gt;intentionality&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Definition inspired mostly by the contributions of &lt;a href=&quot;https://datascience.stackexchange.com/users/36/statsrus&quot;&gt;@statsRus&lt;/a&gt; to Data Science.SE&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="53" LastEditorUserId="3117" LastEditDate="2017-08-27T17:25:18.230" LastActivityDate="2017-08-27T17:25:18.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="80" PostTypeId="4" CreationDate="2014-05-15T03:19:40.360" Score="0" Body="An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal." OwnerUserId="53" LastEditorUserId="53" LastEditDate="2014-05-16T13:46:05.850" LastActivityDate="2014-05-16T13:46:05.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="81" PostTypeId="1" AcceptedAnswerId="82" CreationDate="2014-05-15T04:59:54.317" Score="16" ViewCount="1560" Body="&lt;p&gt;What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of machines, and sometimes it is referred to as a parallel processing, or as distributed processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a certain way, the computation seems to be always parallel, since there are things running concurrently. But is the distributed computation simply related to the use of more than one machine, or are there any further specificities that distinguishes these two kinds of processing? Wouldn't it be redundant to say, for example, that a computation is &lt;em&gt;parallel AND distributed&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="118" LastEditDate="2014-05-15T09:31:51.370" LastActivityDate="2023-04-11T10:41:24.483" Title="Parallel and distributed computing" Tags="|definitions|parallel|distributed|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="82" PostTypeId="2" ParentId="81" CreationDate="2014-05-15T05:19:34.757" Score="17" Body="&lt;p&gt;Simply set, 'parallel' means running concurrently on distinct resources (CPUs), while 'distributed' means running across distinct computers, involving issues related to networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parallel computing using for instance &lt;a href=&quot;http://en.wikipedia.org/wiki/OpenMP&quot;&gt;OpenMP&lt;/a&gt; is not distributed, while parallel computing with &lt;a href=&quot;http://en.wikipedia.org/wiki/Message_Passing_Interface&quot;&gt;Message Passing&lt;/a&gt; is often distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Being in a 'distributed but not parallel' setting would mean under-using resources so it is seldom encountered but it is conceptually possible.&lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="172" LastEditDate="2014-05-15T05:25:39.970" LastActivityDate="2014-05-15T05:25:39.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="83" PostTypeId="2" ParentId="20" CreationDate="2014-05-15T07:47:44.710" Score="8" Body="&lt;p&gt;I posted a pretty detailed answer on stackoverflow about when it is appropriate to use relational vs document (or NoSQL) database, here:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://stackoverflow.com/questions/13528216/motivations-for-using-relational-database-orm-or-document-database-odm/13599767#13599767&quot;&gt;Motivations for using relational database / ORM or document database / ODM&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;for small stuff, go with whatever tools you are familiar with&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;a few gigabytes is definitely small stuff: it doesn't get big until it is too big to fit in a single &lt;a href=&quot;http://www.mysql.com/products/cluster/scalability.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MySQL Cluster&lt;/a&gt; with a reasonable number of nodes (16-32), which means maybe 8-16TB data and a few million transactions per second (or a more conventional hard-drive-based database with up to 100's of TB data and a few thousand transactions per second).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if you're stuck with another database (not MySQL Cluster), get more mileage out of it by throwing in FusionIO hardware.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;once you have data larger than a few TB &lt;em&gt;and&lt;/em&gt; faster than thousands of transactions per second, it is a good time to look at moving to logical sharding in the application code first and then to NoSQL.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cassandra.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cassandra&lt;/a&gt; :)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="26" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-05-15T07:59:05.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="84" PostTypeId="2" ParentId="71" CreationDate="2014-05-15T08:19:40.577" Score="10" Body="&lt;p&gt;You are asking about &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_dredging&quot;&gt;Data Dredging&lt;/a&gt;, which is what happens when testing a very large number of hypotheses against a data set, or testing hypotheses against a data set that were suggested by the same data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, check out &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiple_comparisons&quot;&gt;Multiple hypothesis hazard&lt;/a&gt;, and &lt;a href=&quot;http://en.wikipedia.org/wiki/Testing_hypotheses_suggested_by_the_data&quot;&gt;Testing hypotheses suggested by the data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is to use some kind of correction for &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate&quot;&gt;False discovery rate&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Familywise_error_rate&quot;&gt;Familywise error rate&lt;/a&gt;, such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Scheff%C3%A9%27s_method&quot;&gt;Scheffé's method&lt;/a&gt; or the (very old-school) &lt;a href=&quot;http://en.wikipedia.org/wiki/Bonferroni_correction&quot;&gt;Bonferroni correction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a somewhat less rigorous way, it may help to filter your discoveries by the confidence interval for the odds ratio (OR) for each statistical result.  If the 99% confidence interval for the odds ratio is 10-12, then the OR is &amp;lt;= 1 with some &lt;em&gt;extremely&lt;/em&gt; small probability, especially if the sample size is also large.  If you find something like this, it is probably a strong effect even if it came out of a test of millions of hypotheses.  &lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2014-05-15T08:25:47.933" LastActivityDate="2014-05-15T08:25:47.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="85" PostTypeId="2" ParentId="75" CreationDate="2014-05-15T08:44:47.327" Score="4" Body="&lt;p&gt;See also &lt;a href=&quot;https://datascience.stackexchange.com/questions/71/when-are-p-values-deceptive/84#84&quot;&gt;When are p-values deceptive?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When there are a lot of variables that can be tested for pair-wise correlation (for example), the replacement is to use any of the corrections for &lt;a href=&quot;http://en.wikipedia.org/wiki/False_discovery_rate&quot; rel=&quot;nofollow noreferrer&quot;&gt;False discovery rate&lt;/a&gt; (to limit probability that any given discovery is false) or &lt;a href=&quot;http://en.wikipedia.org/wiki/Familywise_error_rate&quot; rel=&quot;nofollow noreferrer&quot;&gt;Familywise error rate&lt;/a&gt; (to limit probability of one or more false discoveries).  For example, you might use the Holm–Bonferroni method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of a large sample rather than a lot of variables, something else is needed.  As Christopher said, magnitude of effect a way to treat this.  Combining these two ideas, you might use a confidence interval around your magnitude of effect, and apply a false discovery rate correction to the p-value of the confidence interval.  The effects for which even the lowest bound of the corrected confidence interval is high are likely to be strong effects, regardless of huge data set size.  I am not aware of any published paper that combines confidence intervals with false discovery rate correction in this way, but it seems like a straightforward and intuitively understandable approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make this even better, use a non-parametric way to estimate confidence intervals.  Assuming a distribution is likely to give very optimistic estimates here, and even fitting a distribution to the data is likely to be inaccurate.  Since the information about the shape of the distribution past the edges of the confidence interval comes from a relatively small subsample of the data, this is where it really pays to be careful.  You can use bootstrapping to get a non-parametric confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-05-15T20:32:26.923" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="86" PostTypeId="1" AcceptedAnswerId="101" CreationDate="2014-05-15T09:04:09.710" Score="15" ViewCount="3019" Body="&lt;p&gt;Given website access data in the form &lt;code&gt;session_id, ip, user_agent&lt;/code&gt;, and optionally timestamp, following the conditions below, how would you best cluster the sessions into unique visitors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;session_id&lt;/code&gt;: is an id given to every new visitor. It does not expire, however if the user doesn't accept cookies/clears cookies/changes browser/changes device, he will not be recognised anymore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;IP&lt;/code&gt; can be shared between different users (Imagine a free wi-fi cafe, or your ISP reassigning IPs), and they will often have at least 2, home and work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;User_agent&lt;/code&gt; is the browser+OS version, allowing to distinguish between devices. For example a user is likely to use both phone and laptop, but is unlikely to use windows+apple laptops. It is unlikely that the same session id has multiple useragents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data might look as the fiddle here:&#xA;&lt;a href=&quot;http://sqlfiddle.com/#!2/c4de40/1&quot;&gt;http://sqlfiddle.com/#!2/c4de40/1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, we are talking about assumptions, but it's about getting as close to reality as possible. For example, if we encounter the same ip and useragent in a limited time frame with a different session_id, it would be a fair assumption that it's the same user, with some edge case exceptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Language in which the problem is solved is irellevant, it's mostly about logic and not implementation. Pseudocode is fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: due to the slow nature of the fiddle, you can alternatively read/run the mysql:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select session_id, floor(rand()*256*256*256*256) as ip_num , floor(rand()*1000) as user_agent_id&#xA;from &#xA;    (select 1+a.nr+10*b.nr as session_id, ceil(rand()*3) as nr&#xA;    from&#xA;        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5&#xA;        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)a&#xA;    join&#xA;        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5&#xA;        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)b&#xA;        order by 1&#xA;    )d&#xA;inner join&#xA;    (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5&#xA;    union all select 6 union all select 7 union all select 8 union all select 9 )e&#xA;    on d.nr&amp;gt;=e.nr&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="116" LastEditorUserId="116" LastEditDate="2014-05-15T10:06:06.393" LastActivityDate="2014-05-15T21:41:22.703" Title="Clustering unique visitors by useragent, ip, session_id" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="87" PostTypeId="2" ParentId="77" CreationDate="2014-05-15T09:30:36.460" Score="8" Body="&lt;p&gt;Looking at this document called &lt;a href=&quot;https://www.facebook.com/notes/facebook-data-team/anatomy-of-facebook/10150388519243859&quot;&gt;Anatomy of Facebook&lt;/a&gt; I note that the median is 100. Looking at the cumulative function plot I can bet that the average is higher, near 200. So 50 seems to not be the best number here. However I think that this is not the main issue here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main issue is the lack of information on how the database was used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems reasonable that a data storage designed specially for graph structures to be more efficient than traditional RDBMs. However, even if the RDBMs are not in the latest trends as a data storage of choice, these systems evolved continuously in a race with the data set dimensions. There are various types of possible designs, various ways of indexing data, improvements related with concurrency and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To conclude I think that regarding reproducibility, the study lack a proper description of how the database schema was designed. I do not expect that a database to dominate on such king of interrogations, however I would expect that with a well-tuned design the differences to not be such massive.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-05-15T09:30:36.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="89" PostTypeId="1" AcceptedAnswerId="91" CreationDate="2014-05-15T11:22:27.293" Score="12" ViewCount="558" Body="&lt;p&gt;For example, when searching something in Google, results return nigh-instantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that Google sorts and indexes pages with algorithms etc., but I imagine it infeasible for the results of every single possible query to be indexed (and results are personalized, which renders this even more infeasible)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs, I imagine the hardware latency to be huge, given the sheer amount of data to process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does MapReduce help solve this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches? Even for the most obscure search I have conducted, I don't think the search has ever been reported to be larger than 5 seconds. How is this possible?&lt;/p&gt;&#xA;" OwnerUserId="189" LastEditorUserId="189" LastEditDate="2014-05-16T02:46:56.510" LastActivityDate="2014-08-30T18:40:02.403" Title="How does a query into a huge database return with negligible latency?" Tags="|bigdata|google|search|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="90" PostTypeId="2" ParentId="81" CreationDate="2014-05-15T11:46:38.170" Score="4" Body="&lt;p&gt;The terms &quot;parallel computing&quot; and &quot;distributed computing&quot; certainly have a large overlap, but can be differentiated further. Actually, you already did this in your question, by later asking about &quot;parallel processing&quot; and &quot;distributed processing&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could consider &quot;distributed computing&quot; as the more general term that involves &quot;distributed processing&quot; as well as, for example, &quot;distributed storage&quot;. The common term, &quot;distributed&quot;, usually refers to some sort of &lt;a href=&quot;http://en.wikipedia.org/wiki/Message_Passing_Interface&quot; rel=&quot;noreferrer&quot;&gt;Message Passing&lt;/a&gt; over a network, between machines that are physically separated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &quot;parallel computing&quot; is also in the process of being further defined, e.g. by explicitly differentiating between the terms &quot;parallel&quot; and &quot;concurrent&quot;, where - roughly - the first one refers &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_parallelism&quot; rel=&quot;noreferrer&quot;&gt;data parallelism&lt;/a&gt; and the latter to &lt;a href=&quot;http://en.wikipedia.org/wiki/Task_parallelism&quot; rel=&quot;noreferrer&quot;&gt;task parallelism&lt;/a&gt;, although there are hardly really strict and binding defintions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So one could say that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;distributed processing&quot; usually (although not necessarily) means that it also is &quot;parallel processing&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;distributed computing&quot; is more general, and also covers aspects that are not related to parallelism&lt;/li&gt;&#xA;&lt;li&gt;and obviously, &quot;parallel computing&quot;/&quot;parallel processing&quot; does not imply that it is &quot;distributed&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="156" LastActivityDate="2014-05-15T11:46:38.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="91" PostTypeId="2" ParentId="89" CreationDate="2014-05-15T11:56:43.607" Score="13" Body="&lt;p&gt;Well, I'm not sure if it is MapReduce that solves the problem, but it surely wouldn't be MapReduce alone to solve all these questions you raised. But here are important things to take into account, and that make it &lt;em&gt;feasible&lt;/em&gt; to have such low latency on queries from all these TBs of data in different machines:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;distributed computing: by being distributed does not mean that the indexes are simply distributed in different machines, they are actually replicated along different clusters, which allows for lots of users performing different queries with low retrieval time (yes, huge companies can afford for that much of machines);&lt;/li&gt;&#xA;&lt;li&gt;caching: caches tremendously reduce execution time, be it for the crawling step, for the retrieval of pages, or for the ranking and exihibition of results;&lt;/li&gt;&#xA;&lt;li&gt;lots of tweaking: all the above and very efficient algorithms/solutions can only be effective if the implementation is also efficient. There are tons of (hard coded) optimizations, such as locality of reference, compression, caching; all of them usually appliable to different parts of the processing.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Considering that, lets try to address your questions:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;but I imagine it infeasible for the results of every single possible query to be indexed&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, it would be, and actually is infeasible to have results for &lt;em&gt;every single possible query&lt;/em&gt;. There is an infinite number of terms in the world (even if you assume that only terms properly spelled will be entered), and there is an exponential number of queries from these &lt;code&gt;n -&amp;gt; inf&lt;/code&gt; terms (&lt;code&gt;2^n&lt;/code&gt;). So what is done? Caching. But if there are so many queries/results, which ones to cache? Caching policies. The most frequent/popular/relevant-for-the-user queries are the ones cached.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;wouldn't the hardware latency in Google's hardware be huge? Even if the data in Google were all stored in TB/s SSDs&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Nowdays, with such highly developed processors, people tend to think that every possible task that must finish within a second (or less), and that deals with so much data, must be processed by extremely powerful processors with multiple cores and lots of memory. However, the one thing &lt;em&gt;ruling&lt;/em&gt; market is money, and the investors are not interested in wasting it. So what is done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The preference is actually for having lots of machines, each using simple/accessible (in terms of cost) processors, which lowers the price of building up the multitude of clusters there are. And yes, it does work. The main bottleneck always boils down to disk, if you consider simple measurements of &lt;a href=&quot;https://i.stack.imgur.com/Uf6al.gif&quot; rel=&quot;noreferrer&quot;&gt;performance&lt;/a&gt;. But once there are so many machines, one can afford to load things up to main memory, instead of working on hard disks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Memory cards are &lt;em&gt;expensive&lt;/em&gt; for us, mere human beings, but they are very cheap for enterprises that buy lots of such cards at once. Since it's not costly, having much memory as needed to load indexes and keep caches at hand is not a problem. And since there are so many machines, there is no need for super fast processors, as you can direct queries to different places, and have clusters of machines responsible for attending &lt;em&gt;specific geographical regions&lt;/em&gt;, which allows for more &lt;em&gt;specialized&lt;/em&gt; data caching, and even better response times.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does MapReduce help solve this problem?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although I don't think that using or not MapReduce is restricted information inside Google, I'm not conversant about this point. However, Google's implementation of MapReduce (which is surely &lt;em&gt;not&lt;/em&gt; Hadoop) must have lots of optimizations, many involving the aspects discussed above. So, the architecture of MapReduce probably helps guiding how the computations are physically distributed, but there are many other points to be considered to justify such speed in querying time.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Okay, so I understand that popular searches can be cached in memory. But what about unpopular searches?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The graph below presents a curve of how the &lt;em&gt;kinds&lt;/em&gt; of queries occur. You can see that there are three main kinds of searches, each of them holding approximately 1/3 of the volume of queries (area below curve). The plot shows power law, and reinforces the fact that smaller queries are the most popular. The second third of queries are still feasible to process, since they hold few words. But the set of so-called &lt;em&gt;obscure queries&lt;/em&gt;, which usually consist of non-experienced users' queries, are not a negligible part of the queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CpcNf.jpg&quot; alt=&quot;Heavy-tailed distribution&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there lies space for novel solutions. Since it's not just one or two queries (but one third of them), they must have &lt;em&gt;relevant&lt;/em&gt; results. If you type in something &lt;em&gt;much too obscure&lt;/em&gt; in a Google search, it shan't take longer to return a list of results, but will most probably show you something it &lt;em&gt;inferred&lt;/em&gt; you'd like to say. Or it may simply state that there was no document with such terms -- or even cut down your search to 32 words (which just happened to me in a random test here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are dozens of appliable heuristics, which may be either to ignore some words, or to try to break the query into smaller ones, and gather the most &lt;em&gt;popular&lt;/em&gt; results. And all these solutions can be tailored and tweaked to respect &lt;em&gt;feasible waiting times&lt;/em&gt; of, say, less then a second? :D&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-05-16T04:33:52.310" LastActivityDate="2014-05-16T04:33:52.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="92" PostTypeId="2" ParentId="89" CreationDate="2014-05-15T13:18:38.693" Score="10" Body="&lt;p&gt;MapReduce has nothing to do with real-time anything. It is a batch-oriented processing framework suitable for some offline tasks, like ETL and index building. Google has moved off of MapReduce for most jobs now, and even the Hadoop ecosystem is doing the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to low latency is generally to keep precomputed indices in memory. Anything that touches disk is hard to make fast and scale. This is how newer-generation Hadoop-based SQL engines like &lt;a href=&quot;http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html&quot;&gt;Impala&lt;/a&gt; get so much speed compared to MapReduce-based infrastructure like &lt;a href=&quot;http://hive.apache.org/&quot;&gt;Hive&lt;/a&gt;, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search infrastructure can't cache the results of every single query. But it sure can cache intermediate results, or, more complete results for top queries. With a little caching you can serve results for a significant minority of all queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search is also split across servers. So one machine can delegate to 100 to each get a part of the result and then combine them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also get away with some degree of approximation. Google does not literally form a thousand pages of search results; it just has to get the first page about right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that Google has &lt;em&gt;millions&lt;/em&gt; of computers around the globe. Your queries are going to a data center geographically near to you and that is only serving your geography. This cuts out most of the latency, which is network and not processing time in the data center.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-15T13:18:38.693" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="93" PostTypeId="2" ParentId="86" CreationDate="2014-05-15T13:30:04.270" Score="6" Body="&lt;p&gt;There's not much you can do with just this data, but what little you can do does not rely on machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, sessions from the same IP but different User-Agents are almost certainly distinct users. Sessions with the same IP and User-Agent are usually the same user, except in the case of proxies / wi-fi access points. Those you might identify by looking at the distribution of session count per IP to identify likely 'aggregate' IPs. Sessions from the same IP / User-Agent that overlap in time are almost surely distinct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further distinguish users you would need more info. For example, the sites or IP addresses that the user is connecting to would be a very strong basis for differentiating sessions. Then you could get into more sophisticated learning to figure out when sessions are the same or different users.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-15T13:30:04.270" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="94" PostTypeId="1" AcceptedAnswerId="97" CreationDate="2014-05-15T14:41:24.020" Score="21" ViewCount="313" Body="&lt;p&gt;While building a rank, say for a search engine, or a recommendation system, is it valid to rely on click frequency to determine the relevance of an entry?&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2015-11-23T15:36:28.760" Title="Does click frequency account for relevance?" Tags="|recommender-system|information-retrieval|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="95" PostTypeId="2" ParentId="94" CreationDate="2014-05-15T15:06:24.600" Score="5" Body="&lt;p&gt;Is it valid to &lt;em&gt;use&lt;/em&gt; click frequency, then &lt;strong&gt;yes&lt;/strong&gt;. Is it valid to use &lt;strong&gt;only&lt;/strong&gt; the click frequency, then probably &lt;strong&gt;no&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search relevance is much more complicated than just one metric. &lt;a href=&quot;http://www.amazon.ca/s/ref=nb_sb_noss?url=search-alias=aps&amp;amp;field-keywords=search%20ranking&quot; rel=&quot;noreferrer&quot;&gt;There are entire books on the subject&lt;/a&gt;. Extending this answer beyond a simple yes/no would likely make the answer far too broad (and opinionated)&lt;/p&gt;&#xA;" OwnerUserId="9" LastActivityDate="2014-05-15T15:06:24.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="96" PostTypeId="2" ParentId="94" CreationDate="2014-05-15T15:10:30.243" Score="7" Body="&lt;p&gt;For my part I can say that I use click frequency on i.e. eCommerce products. When you combine it with the days of the year it can even bring you great suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e.: We have historical data from 1 year over 2 products (Snowboots[], Sandalettes[])&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Snowboots[1024,1253,652,123,50,12,8,4,50,148,345,896]&#xA;Sandalettes[23,50,73,100,534,701,1053,1503,1125,453,213,25]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where [0] = January&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, snowboots are much more searched in January than sandalettes, so you should suggest snowboots to someone searching shoes on your site or /we on january.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also see if something is &quot;fresh&quot; at this time, like when people often click a unknown product it could be an insight for a new comming trend or something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That are just some examples where you could use click frequency as an insight. I think there are &lt;strong&gt;no rules&lt;/strong&gt; for what you can use or not in recommendations, &lt;strong&gt;as long as it makes sense&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="115" LastEditorUserId="115" LastEditDate="2015-11-23T15:36:28.760" LastActivityDate="2015-11-23T15:36:28.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="97" PostTypeId="2" ParentId="94" CreationDate="2014-05-15T17:14:36.817" Score="15" Body="&lt;p&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/169639/cikm-clickpatterns.pdf&quot; rel=&quot;noreferrer&quot;&gt;Depends on the user's intent&lt;/a&gt;, for starters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.seoresearcher.com/distribution-of-clicks-on-googles-serps-and-eye-tracking-analysis.htm&quot; rel=&quot;noreferrer&quot;&gt;Users normally only view the first set of links&lt;/a&gt;, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive those are the best links, otherwise the clicks are most likely going to reflect placement, not relevance. For example, here's a &lt;a href=&quot;http://www.seoresearcher.com/distribution-of-clicks-on-googles-serps-and-eye-tracking-analysis.htm&quot; rel=&quot;noreferrer&quot;&gt;click and attention distribution heat-map&lt;/a&gt; for  Google search results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/8kO5S.jpg&quot; alt=&quot;Google SEPR Click and Attention distribution ‘heat-map’&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, using click frequency to account for relevance is not a direct measure of the resource's relevance. Also, using clicks is problematic, since issues like click-inflation, click-fraud, etc. will pop-up and are hard to counter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, if you're interested in using user interaction to model relevance, I would suggest you attempt to measure post-click engagement, not how users respond to search results; see &quot;&lt;a href=&quot;http://www.youtube.com/watch?v=BsCeNCVb-d8&quot; rel=&quot;noreferrer&quot;&gt;YouTube's head of engineering speaking about clicks vs engagement&lt;/a&gt;&quot; for more information, though note that the &lt;a href=&quot;http://www.orbitmedia.com/blog/ideal-blog-post-length/&quot; rel=&quot;noreferrer&quot;&gt;size itself of the content is a factor&lt;/a&gt; too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Might be worth noting that historically Google was known for &lt;a href=&quot;http://en.wikipedia.org/wiki/PageRank&quot; rel=&quot;noreferrer&quot;&gt;PageRank algorithm&lt;/a&gt; though it's possible your intent is only to review click-streams, so I won't delve &lt;a href=&quot;https://www.google.com/search?q=google%20ranking%20factors&quot; rel=&quot;noreferrer&quot;&gt;Google ranking factors&lt;/a&gt;; if you are interested in the Google's approach, you might find a review of &lt;a href=&quot;http://static.googleusercontent.com/media/www.google.com/en/us/insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf&quot; rel=&quot;noreferrer&quot;&gt;Google's Search Quality Rating Guidelines&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="158" LastEditDate="2014-05-15T23:08:04.300" LastActivityDate="2014-05-15T23:08:04.300" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="101" PostTypeId="2" ParentId="86" CreationDate="2014-05-15T21:41:22.703" Score="9" Body="&lt;p&gt;One possibility here (and this is really an extension of what Sean Owen posted) is to define a &quot;stable user.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the given info you have you can imagine making a user_id that is a hash of ip and some user agent info (pseudo code):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;uid = MD5Hash(ip + UA.device + UA.model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you flag these ids with &quot;stable&quot; or &quot;unstable&quot; based on usage heuristics you observe for your users.  This can be a threshold of # of visits in a given time window, length of time their cookies persist, some end action on your site (I realize this wasn't stated in your original log), etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is to separate the users that don't drop cookies from those that do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here you can attribute session_ids to stable uids from your logs.  You will then have &quot;left over&quot; session_ids for unstable users that you are relatively unsure about.  You may be over or under counting sessions, attributing behavior to multiple people when there is only one, etc...  But this is at least limited to the users you are now &quot;less certain&quot; about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You then perform analytics on your stable group and project that to the unstable group.  Take a user count for example, you know the total # of sessions, but you are unsure of how many users generated those sessions.  You can find the # sessions / unique stable user and use this to project the &quot;estimated&quot; number of unique users in the unstable group since you know the number of sessions attributed to that group.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;projected_num_unstable_users = num_sess_unstable / num_sess_per_stable_uid&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This doesn't help with per user level investigation on unstable users but you can at least get some mileage out of a cohort of stable users that persist for some time.  You can, by various methods, project behavior and counts into the unstable group.  The above is a simple example of something you might want to know.  The general idea is again to define a set of users you are confident persist, measure what you want to measure, and use certain ground truths (num searches, visits, clicks, etc...) to project into the unknown user space and estimate counts for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a longstanding problem in unique user counting, logging, etc... for services that don't require log in.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2014-05-15T21:41:22.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="102" PostTypeId="1" AcceptedAnswerId="111" CreationDate="2014-05-16T05:09:33.557" Score="6" ViewCount="592" Body="&lt;p&gt;What is the best noSQL backend to use for a mobile game? Users can make a lot of servers requests, it needs also to retrieve users' historical records (like app purchasing) and analytics of usage behavior.&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="229" LastEditDate="2014-05-18T19:41:19.157" LastActivityDate="2016-12-09T21:55:46.000" Title="What is the Best NoSQL backend for a mobile game" Tags="|nosql|performance|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="103" PostTypeId="1" CreationDate="2014-05-16T14:26:12.270" Score="25" ViewCount="10347" Body="&lt;p&gt;Assume that we have a set of elements &lt;em&gt;E&lt;/em&gt; and a similarity (&lt;strong&gt;not distance&lt;/strong&gt;) function &lt;em&gt;sim(ei, ej)&lt;/em&gt; between two elements &lt;em&gt;ei,ej ∈ E&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could we (efficiently) cluster the elements of &lt;em&gt;E&lt;/em&gt;, using &lt;em&gt;sim&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;k&lt;/em&gt;-means, for example, requires a given &lt;em&gt;k&lt;/em&gt;, Canopy Clustering requires two threshold values. What if we don't want such predefined parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that &lt;em&gt;sim&lt;/em&gt; is not neccessarily a metric (i.e. the triangle inequality may, or may not hold). Moreover, it doesn't matter if the clusters are disjoint (partitions of &lt;em&gt;E&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="113" LastActivityDate="2021-06-28T09:13:21.753" Title="Clustering based on similarity scores" Tags="|clustering|algorithms|similarity|" AnswerCount="5" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="104" PostTypeId="5" CreationDate="2014-05-16T15:35:51.420" Score="0" Body="&lt;h2&gt;Use the &lt;a href=&quot;/questions/tagged/definitions&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;definitions&amp;#39;&quot; rel=&quot;tag&quot;&gt;definitions&lt;/a&gt; tag when:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You think we should create an official definition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An existing Tag Wiki needs a more precise definition to avoid confusion and we need to create consensus before an edit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(rough draft - needs filling out)&lt;/p&gt;&#xA;" OwnerUserId="53" LastEditorUserId="53" LastEditDate="2014-05-20T13:50:52.447" LastActivityDate="2014-05-20T13:50:52.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="105" PostTypeId="4" CreationDate="2014-05-16T15:35:51.420" Score="0" Body="a discussion (meta) tag used when there exists *disagreement* or *confusion* about the everyday meaning of a term or phrase." OwnerUserId="53" LastEditorUserId="53" LastEditDate="2014-05-20T13:53:05.697" LastActivityDate="2014-05-20T13:53:05.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="106" PostTypeId="2" ParentId="14" CreationDate="2014-05-16T16:25:58.250" Score="6" Body="&lt;p&gt;There are many overlaps between data mining and datascience. I would say that people with the role of datamining are concerned with data collection and the extraction of features from unfiltered, unorganised and mostly raw/wild datasets. Some very important data may be difficult to extract, not do to the implementation issues but because it may have foreign artifacts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg. if I needed someone to look at financial data from written tax returns in the 70s which were scanned and machine read to find out if people saved more on car insurance; a dataminer would be the person to get. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I needed someone to examine the influence Nike's Twitter profile in the tweets of Brazil and identify key positive features from the profile, I would look for a datascientist.&lt;/p&gt;&#xA;" OwnerUserId="34" LastActivityDate="2014-05-16T16:25:58.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="107" PostTypeId="1" CreationDate="2014-05-16T20:07:50.983" Score="12" ViewCount="147" Body="&lt;p&gt;Consider a stream containing &lt;a href=&quot;http://en.m.wikipedia.org/wiki/Tuple&quot; rel=&quot;noreferrer&quot;&gt;tuples&lt;/a&gt; &lt;code&gt;(user, new_score)&lt;/code&gt; representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The game has 200K to 300K unique players. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to have some standing queries like: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Which players posted more than x scores in a sliding window of one hour&lt;/li&gt;&#xA;&lt;li&gt;Which players gained x% score in a sliding window of one hour&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is which open source tools can I employ to jumpstart this project? I am considering &lt;a href=&quot;http://esper.codehaus.org/&quot; rel=&quot;noreferrer&quot;&gt;Esper&lt;/a&gt; at the moment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I have just completed reading &quot;Mining Data Streams&quot; (chapter 4 of &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds.html&quot; rel=&quot;noreferrer&quot;&gt;Mining of Massive Datasets&lt;/a&gt;) and I am quite new to mining data streams.&lt;/p&gt;&#xA;" OwnerUserId="200" LastEditorUserId="118" LastEditDate="2014-05-19T07:33:50.080" LastActivityDate="2014-05-19T07:33:50.080" Title="Opensource tools for help in mining stream of leader board scores" Tags="|tools|data-stream-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="108" PostTypeId="5" CreationDate="2014-05-16T20:24:38.980" Score="1" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2014-05-16T20:24:38.980" LastActivityDate="2014-05-16T20:24:38.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="109" PostTypeId="4" CreationDate="2014-05-16T20:24:38.980" Score="0" Body="An activity that seeks patterns in a continuous stream of data elements, usually involving summarizing the stream in some way." OwnerUserId="200" LastEditorUserId="200" LastEditDate="2014-05-20T13:52:00.620" LastActivityDate="2014-05-20T13:52:00.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="111" PostTypeId="2" ParentId="102" CreationDate="2014-05-17T03:07:59.707" Score="7" Body="&lt;p&gt;Some factors you might consider:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Developer familiarity: go with whatever you or your developers are familiar with.  Mongo, Couch, Riak, DynamoDB etc all have their strengths but all should do ok here, so rather than going for an unfamiliar solution that might be slightly better go for familiar and save a bunch of development time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ease of cloud deployment:  for example, if you are using Amazon AWS, then DynamoDB is likely an excellent choice.  Sure, you could use Mongo on AWS, but why bother?  Other cloud providers have their own preferred db, for example if you are using Google AppEngine, it makes sense to use BigTable or Cloud Datastore. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your use case seems both well suited to NoSQL and not very challenging since your data has a natural partition by user.  I think you'd be technically ok with anything, which is why I'm mainly covering other factors.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2014-05-18T06:43:52.453" LastActivityDate="2014-05-18T06:43:52.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="112" PostTypeId="2" ParentId="107" CreationDate="2014-05-17T04:18:10.020" Score="8" Body="&lt;p&gt;This isn't a full solution, but you may want to look into &lt;a href=&quot;http://www.orientechnologies.com/&quot;&gt;OrientDB&lt;/a&gt; as part of your stack. Orient is a Graph-Document database server written entirely in Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In graph databases, relationships are considered first class citizens and therefore traversing those relationships can be done pretty quickly. Orient is also a document database which would allow you the kind of schema-free architecture it sounds like you would need. The real reason I suggest Orient, however, is because of its extensiblity. It supports streaming via sockets, and the entire database can be embedded into another application. Finally, it can be scaled efficiently and/or can work entirely through memory. So, with some Java expertise, you can actually run your preset queries against the database in memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are doing something similar. In creating an app/site for social science research collaboration, we found ourselves with immensely complex data models. We ended up writing several of the queries using the Gremlin Traversal Language (a subset of Groovy, which is, of course, Java at its heart), and then exposing those queries through the binary connection server of the OrientDB. So, the client opens a TCP socket, sends a short binary message, and the query is executing in Java directly against the in-memory database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OrientDB also supports writing function queries in Javascript, and you can use Node.js to interact directly with an Orient instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For something of this size, I would want to use Orient in conjunction with Hadoop or something like that. You can also use Orient in conjunction with esper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider:&#xA;An introduction to orient: &lt;a href=&quot;http://www.sitepoint.com/a-look-at-orientdb-the-graph-document-nosql/&quot;&gt;http://www.sitepoint.com/a-look-at-orientdb-the-graph-document-nosql/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Complex, real-time queries: &lt;a href=&quot;http://www.gft-blog.com/business-trends/leveraging-real-time-scoring-through-bigdata-to-detect-insurance-fraud/&quot;&gt;http://www.gft-blog.com/business-trends/leveraging-real-time-scoring-through-bigdata-to-detect-insurance-fraud/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A discussion about streaming options with java and orient: &lt;a href=&quot;https://github.com/orientechnologies/orientdb/issues/1227&quot;&gt;https://github.com/orientechnologies/orientdb/issues/1227&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="70" LastActivityDate="2014-05-17T04:18:10.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="113" PostTypeId="1" AcceptedAnswerId="122" CreationDate="2014-05-17T04:53:03.913" Score="13" ViewCount="266" Body="&lt;p&gt;When a relational database, like MySQL, has better performance than a no relational, like MongoDB?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that their performance is still good.&lt;/p&gt;&#xA;" OwnerUserId="199" LastEditorUserId="31513" LastEditDate="2017-06-05T19:30:23.440" LastActivityDate="2017-06-05T19:30:23.440" Title="When a relational database has better performance than a no relational" Tags="|bigdata|performance|databases|nosql|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="115" PostTypeId="1" AcceptedAnswerId="131" CreationDate="2014-05-17T08:45:08.420" Score="15" ViewCount="4467" Body="&lt;p&gt;If I have a very long list of paper names, how could I get abstract of these papers from internet or any database?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper names are like &quot;Assessment of Utility in Web Mining for the Domain of Public Health&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does any one know any API that can give me a solution? I tried to crawl google scholar, however, google blocked my crawler.&lt;/p&gt;&#xA;" OwnerUserId="212" LastActivityDate="2021-01-25T09:43:02.103" Title="Is there any APIs for crawling abstract of paper?" Tags="|data-mining|machine-learning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="116" PostTypeId="1" AcceptedAnswerId="121" CreationDate="2014-05-17T09:16:18.823" Score="28" ViewCount="3267" Body="&lt;p&gt;I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three crucial characteristics of my database:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;the age distribution in my training set (12k of users in sum) is skewed towards younger users (i.e. I have 1157 users aged 27, and 23 users aged 65);&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;many sites have no more than 5 likers (I filtered out the FB sites with less than 5 likers).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;there's many more features than samples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, my questions are: what strategy would you suggest to prepare the data for further analysis? Should I perform some sort of dimensionality reduction? Which ML method would be most appropriate to use in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mainly use Python, so Python-specific hints would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="173" LastEditorUserId="173" LastEditDate="2014-05-17T19:26:53.783" LastActivityDate="2021-02-09T04:31:08.427" Title="Machine learning techniques for estimating users' age based on Facebook sites they like" Tags="|machine-learning|dimensionality-reduction|python|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="118" PostTypeId="5" CreationDate="2014-05-17T13:41:20.283" Score="0" Body="&lt;p&gt;&lt;strong&gt;NoSQL&lt;/strong&gt; (sometimes expanded to &quot;not only &lt;a href=&quot;/questions/tagged/sql&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;sql&amp;#39;&quot; rel=&quot;tag&quot;&gt;sql&lt;/a&gt;&quot;) is a broad class of database management systems that differ from the classic model of the relational database management system (&lt;a href=&quot;/questions/tagged/rdbms&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;rdbms&amp;#39;&quot; rel=&quot;tag&quot;&gt;rdbms&lt;/a&gt;) in some significant ways.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;NoSQL systems:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Specifically designed for high load&lt;/li&gt;&#xA;&lt;li&gt;Natively support horizontal scalability&lt;/li&gt;&#xA;&lt;li&gt;Fault tolerant&lt;/li&gt;&#xA;&lt;li&gt;Store data in denormalised manner&lt;/li&gt;&#xA;&lt;li&gt;Do not usually enforce strict database schema&lt;/li&gt;&#xA;&lt;li&gt;Do not usually store data in a table&lt;/li&gt;&#xA;&lt;li&gt;Sometimes provide eventual consistency instead of ACID transactions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;In contrast to RDBMS, NoSQL systems:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do not guarantee data consistency&lt;/li&gt;&#xA;&lt;li&gt;Usually support a limited query language (subset of SQL or another custom query language)&lt;/li&gt;&#xA;&lt;li&gt;May not provide support for transactions/distributed transactions&lt;/li&gt;&#xA;&lt;li&gt;Do not usually use some advanced concepts of RDBMS, such as triggers, views, stored procedures&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;NoSQL implementations can be categorised by their manner of implementation:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Column-oriented_DBMS&quot; rel=&quot;nofollow noreferrer&quot;&gt;Column-oriented&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Document-oriented_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;Document store&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graph&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://dba.stackexchange.com/questions/607/what-is-a-key-value-store-database&quot;&gt;Key-value store&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/MultiValue&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multivalue databases&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Object_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;Object databases&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Triplestore&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tripplestore&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/NoSQL#Tuple_store&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tuple store&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Free NoSQL Books&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://books.couchdb.org/relax/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CouchDB: The Definitive Guide&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://openmymind.net/2011/3/28/The-Little-MongoDB-Book&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Little MongoDB Book&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://openmymind.net/2012/1/23/The-Little-Redis-Book/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Little Redis Book&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="201" LastEditorUserId="381" LastEditDate="2017-08-27T17:25:05.257" LastActivityDate="2017-08-27T17:25:05.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="119" PostTypeId="4" CreationDate="2014-05-17T13:41:20.283" Score="0" Body="NoSQL (sometimes expanded to &quot;not only sql&quot;) is a broad class of database management systems that differ from the classic model of the relational database management system (rdbms) in some significant ways." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T17:28:13.327" LastActivityDate="2019-04-08T17:28:13.327" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="120" PostTypeId="2" ParentId="115" CreationDate="2014-05-17T18:15:11.937" Score="5" Body="&lt;p&gt;arXiv has an &lt;a href=&quot;http://arxiv.org/help/bulk_data&quot; rel=&quot;noreferrer&quot;&gt;API and bulk download&lt;/a&gt; but if you want something for paid journals it will be hard to come by without paying an indexer like pubmed or elsevier or the like.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2014-05-17T18:15:11.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="121" PostTypeId="2" ParentId="116" CreationDate="2014-05-17T18:53:30.123" Score="16" Body="&lt;p&gt;One thing to start off with would be k-NN.  The idea here is that you have a user/item matrix and for some of the users you have a reported age.  The age for a person in the user item matrix might be well determined by something like the mean or median age of some nearest neighbors in the item space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have each user expressed as a vector in item space, find the k nearest neighbors and assign the vector in question some summary stat of the nearest neighbor ages.  You can choose k on a distance cutoff or more realistically by iteratively assigning ages to a train hold out and choosing the k that minimizes the error in that assignment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the dimensionality is a problem you can easily perform reduction in this setup by single value decomposition choosing the m vectors that capture the most variance across the group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In all cases since each feature is binary it seems that cosine similarity would be your go to distance metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to think a bit more about other approaches (regression, rf, etc...) given the narrow focus of your feature space (all variants of the same action, liking) I think the user/item approach might be the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One note of caution, if the ages you have for train are self reported you might need to correct some of them.  People on facebook tend to report ages in the decade they were born.  Plot a histogram of the birth dates (derived from ages) and see if you have spikes at decades like 70s, 80s, 90s.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2014-05-17T18:53:30.123" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="122" PostTypeId="2" ParentId="113" CreationDate="2014-05-17T20:56:15.577" Score="10" Body="&lt;p&gt;It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively. &lt;/p&gt;&#xA;" OwnerUserId="180" LastActivityDate="2014-05-17T20:56:15.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="123" PostTypeId="5" CreationDate="2014-05-17T21:10:41.990" Score="0" Body="&lt;p&gt;The most basic relationship to describe is a &lt;strong&gt;linear relationship&lt;/strong&gt; between variables, &lt;em&gt;x&lt;/em&gt; and &lt;em&gt;y&lt;/em&gt;, such that they can be said to be highly-correlated when every increase in &lt;em&gt;x&lt;/em&gt; results in a proportional increase in &lt;em&gt;y&lt;/em&gt;. They can also be said to be &lt;em&gt;inversely proportional&lt;/em&gt; so that when &lt;em&gt;x&lt;/em&gt; increases, &lt;em&gt;y&lt;/em&gt; decreases. And finally, the two variables can be said to be &lt;a href=&quot;http://en.wikipedia.org/wiki/Independence_%28probability_theory%29&quot; rel=&quot;nofollow&quot;&gt;independent&lt;/a&gt; in the event that there is no linear relationship between the two (they are uncorrelated, or have a &lt;strong&gt;Pearson correlation coefficient&lt;/strong&gt; of 0. [LaTeX support would be highly desirable at this point.]&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Different correlation coefficients and their uses:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson correlation coefficient&lt;/a&gt; is useful.....&#xA;[draft]&lt;/p&gt;&#xA;" OwnerUserId="53" LastEditorUserId="53" LastEditDate="2014-05-20T13:50:21.763" LastActivityDate="2014-05-20T13:50:21.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="124" PostTypeId="4" CreationDate="2014-05-17T21:10:41.990" Score="0" Body="A statistics term used to describe a type of dependence between variables (or data sets). Correlations are often used as an indicator of predictability. However, correlation does NOT imply causation. Different methods of calculating correlation exist to capture more complicated relationships between the variables being studied." OwnerUserId="53" LastEditorUserId="53" LastEditDate="2014-05-20T13:50:19.543" LastActivityDate="2014-05-20T13:50:19.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="125" PostTypeId="1" AcceptedAnswerId="126" CreationDate="2014-05-17T21:52:34.563" Score="1" ViewCount="792" Body="&lt;p&gt;I want learn about NoSQL and when is better to use SQL or NoSQL. I know that this question depends on the case, but I'm asking for a good documentation on NoSQL, and some explanation of when is better to use SQL or NoSQL (use cases, etc). Also, your opinions on NoSQL databases, and any recommendations for learning about this topic are welcome.&lt;/p&gt;&#xA;" OwnerDisplayName="user109" LastEditorUserId="84" LastEditDate="2014-05-18T16:10:50.820" LastActivityDate="2021-03-15T21:14:08.800" Title="How to learn noSQL databases and how to know when SQL or noSQL is better" Tags="|nosql|" AnswerCount="2" CommentCount="0" ClosedDate="2014-05-21T14:00:22.100" ContentLicense="CC BY-SA 3.0" />
  <row Id="126" PostTypeId="2" ParentId="125" CreationDate="2014-05-17T23:53:42.700" Score="2" Body="&lt;p&gt;Please have a look at my answer here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://stackoverflow.com/questions/13528216/motivations-for-using-relational-database-orm-or-document-database-odm/13599767#13599767&quot;&gt;Motivations for using relational database / ORM or document database / ODM&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Short version:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use NoSQL when data size and number of transactions per second forces it, which typically happens above a few tens of TB and millions of transactions per second (db in memory, running on cluster), &lt;em&gt;or&lt;/em&gt; at hundreds of TB and thousands of transactions per second (traditional db on disk, transactions per second is &lt;em&gt;highly&lt;/em&gt; dependent on the usage pattern). Traditional SQL scales up to that point just fine.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;NoSQL is well suited for some problems (data has a natural sharding, schema is flexible, eventual consistency is ok).  You can use for those even if scaling doesn't force you to.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Developer familiarity with tools and ops ease of deployment are &lt;em&gt;major&lt;/em&gt; factors, don't overlook them.  A solution may be technically better but you may have a hard time using it, make sure you need it and make sure you budget for the learning curve.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As to how to learn it: fire up a MongoDB image on AWS, or DynamoDB, and have fun! &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB on AWS tutorial&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DynamoDB tutorial&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="26" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-04-11T22:08:14.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="128" PostTypeId="1" AcceptedAnswerId="296" CreationDate="2014-05-18T06:10:52.543" Score="62" ViewCount="32570" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot;&gt;Latent Dirichlet Allocation (LDA)&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_Dirichlet_process&quot;&gt;Hierarchical Dirichlet Process (HDP)&lt;/a&gt; are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP doesn't. Why is that so? And what are the differences, pros, and cons of both topic modelling methods?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="84" LastEditDate="2014-05-20T13:45:59.373" LastActivityDate="2021-02-04T09:10:56.807" Title="Latent Dirichlet Allocation vs Hierarchical Dirichlet Process" Tags="|nlp|topic-model|lda|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="129" PostTypeId="1" AcceptedAnswerId="166" CreationDate="2014-05-18T06:17:37.587" Score="10" ViewCount="1630" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/879432/what-is-the-difference-between-a-generative-and-discriminative-algorithm&quot;&gt;This question&lt;/a&gt; asks about generative vs. discriminative algorithm, but can someone give an example of the difference between these forms when applied to Natural Language Processing? &lt;strong&gt;How are generative and discriminative models used in NLP?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-05-19T11:13:48.067" Title="What is generative and discriminative model? How are they used in Natural Language Processing?" Tags="|nlp|language-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="130" PostTypeId="1" AcceptedAnswerId="132" CreationDate="2014-05-18T06:26:15.673" Score="70" ViewCount="39984" Body="&lt;p&gt;From wikipedia:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;dimensionality reduction or dimension reduction is the process of&#xA;reducing the number of random variables under consideration, and&#xA;can be divided into feature selection and feature extraction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;What is the difference between feature selection and feature extraction?&lt;/p&gt;&#xA;&lt;p&gt;What is an example of dimensionality reduction in a Natural Language Processing task?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="29169" LastEditDate="2021-02-07T20:32:33.683" LastActivityDate="2021-02-07T20:32:33.683" Title="What is dimensionality reduction? What is the difference between feature selection and extraction?" Tags="|feature-selection|feature-extraction|dimensionality-reduction|" AnswerCount="11" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="131" PostTypeId="2" ParentId="115" CreationDate="2014-05-18T06:54:08.560" Score="9" Body="&lt;p&gt;Look it up on:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Google Scholar &lt;a href=&quot;http://scholar.google.com/scholar?hl=en&amp;amp;q=Assessment%20of%20Utility%20in%20Web%20Mining%20for%20the%20Domain%20of%20Public%20Health&amp;amp;btnG=&amp;amp;as_sdt=1,5&amp;amp;as_sdtp=&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Citeseer &lt;a href=&quot;http://citeseerx.ist.psu.edu/search?q=Assessment%20of%20Utility%20in%20Web%20Mining%20for%20the%20Domain%20of%20Public%20Health&amp;amp;submit=Search&amp;amp;sort=rlv&amp;amp;t=doc&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you get a single exact title match then you have probably found the right article, and can fill in the rest of the info from there.  Both give you download links and bibtex-style output.  What you would likely want to do though to get perfect metadata is download and parse the pdf (if any) and look for DOI-style identifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please be nice and rate-limit your requests if you do this.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2014-05-18T06:54:08.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="132" PostTypeId="2" ParentId="130" CreationDate="2014-05-18T07:53:58.203" Score="57" Body="&lt;p&gt;Simply put: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;feature selection: you select a subset of the original feature set; while&lt;/li&gt;&#xA;&lt;li&gt;feature extraction: you build a new set of features from the original feature set. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-05-18T07:53:58.203" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="133" PostTypeId="2" ParentId="103" CreationDate="2014-05-18T09:09:47.780" Score="12" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here).  For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters.  So you can probably use DBSCAN, although you may have to do some kind of nonstandard spatial index to do efficient lookups in your case.  Your version of epsilon-neighborhood will likely be sim &gt; 1/epsilon rather than the other way around.  Same story with k-means and related algorithms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can you construct a metric from your similarity?  One possibility: dist(ei, ej) = min( sim(ei, ek) + sim(ek, ej) ) for all k ...  Alternately, can you provide an upper  bound such that sim(ei, ej) &amp;lt; sim(ei, ek) + sim(ek, ej) + d, for all k and some positive constant d?  Intuitively, large sim values means closer together: is 1/sim metric-like?  What about 1/(sim + constant)?  What about min( 1/sim(ei, ek) + 1/sim(ek, ej) ) for all k? (that last is guaranteed to be a metric, btw)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An alternate construction of a metric is to do an embedding.  As a first step, you can try to map your points ei -&gt; xi, such that xi minimize sum( abs( sim(ei, ej) - f( dist(xi, xj) ) ), for some suitable function f and metric dist.  The function f converts distance in the embedding to a similarity-like value; you'd have to experiment a bit, but 1/dist or exp^-dist are good starting points.  You'd also have to experiment on the best dimension for xi.  From there, you can use conventional clustering on xi.  The idea here is that you can almost (in a best fit sense) convert your distances in the embedding to similarity values, so they would cluster correctly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On the use of predefined parameters, all algorithms have some tuning.  DBSCAN can find the number of clusters, but you still need to give it some parameters.  In general, tuning requires multiple runs of the algorithm with different values for the tunable parameters, together with some function that evaluates goodness-of-clustering (either calculated separately, provided by the clustering algorithm itself, or just eyeballed :)  If the character of your data doesn't change, you can tune once and then use those fixed parameters; if it changes then you have to tune for each run.  You can find that out by tuning for each run and then comparing how well the parameters from one run work on another, compared to the parameters specifically tuned for that.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2014-05-18T09:17:15.557" LastActivityDate="2014-05-18T09:17:15.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="134" PostTypeId="1" AcceptedAnswerId="246" CreationDate="2014-05-18T12:03:21.650" Score="11" ViewCount="708" Body="&lt;p&gt;In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select from for implementing the required analyses:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;One approach is to extract the data from MongoDB to a Hadoop cluster and do the analysis completely in Hadoop platform. However, this requires considerable investment on preparing the platform (software and hardware) and educating the team to work with Hadoop and write map-reduce tasks for it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another approach is to just put our effort on designing the map-reduce algorithms, and run the algorithms on MongoDB map-reduce functionalities. This way, we can create an initial prototype of final system that can generate the reports. I know that the MongoDB's map-reduce functionalities are much slower compared to Hadoop, but currently the data is not that big that makes this a bottleneck yet, at least not for the next six months.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The question is, using the second approach and writing the algorithms for MongoDB, can them be later ported to Hadoop with little needed modification and algorithm redesign? MongoDB just supports JavaScript but programming language differences are easy to handle. However, is there any fundamental differences in the map-reduce model of MongoDB and Hadoop that may force us to redesign algorithms substantially for porting to Hadoop?&lt;/p&gt;&#xA;" OwnerUserId="227" LastActivityDate="2014-06-10T02:42:02.050" Title="Can map-reduce algorithms written for MongoDB be ported to Hadoop later?" Tags="|scalability|apache-hadoop|map-reduce|mongodb|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="135" PostTypeId="2" ParentId="41" CreationDate="2014-05-18T12:30:06.853" Score="9" Body="&lt;p&gt;Considering another criteria, I think that in some cases using Python may be much superior to R for Big Data. I know the wide-spread use of R in data science educational materials and the good data analysis libraries available for it, but sometimes it just depend on the team.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, for people already familiar with programming, using Python provides much more flexibility and productivity boost compared to a language like R, which is not as well-designed and powerful compared to Python in terms of a programming language. As an evidence, in a data mining course in my university, the best final project was written in Python, although the others has access to R's rich data analysis library. That is, sometimes the overall productivity (considering learning materials, documentation, etc.) for Python may be better than R even in the lack of special-purpose data analysis libraries for Python. Also, there are some good articles explaining the fast pace of Python in data science: &lt;a href=&quot;http://readwrite.com/2013/11/25/python-displacing-r-as-the-programming-language-for-data-science&quot;&gt;Python Displacing R&lt;/a&gt; and &lt;a href=&quot;http://wesmckinney.com/blog/?p=77&quot;&gt;Rich Scientific Data Structures in Python&lt;/a&gt; that may soon fill the gap of available libraries for R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important reason for not using R is when working with real world Big Data problems, contrary to academical only problems, there is much need for other tools and techniques, like data parsing, cleaning, visualization, web scrapping, and a lot of others that are much easier using a general purpose programming language. This may be why the default language used in many Hadoop courses (including the Udacity's &lt;a href=&quot;https://www.udacity.com/course/ud617&quot;&gt;online course&lt;/a&gt;) is Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently DARPA has also invested $3 million to help fund Python's data processing and visualization capabilities for big data jobs, which is clearly a sign of Python's future in Big Data. (&lt;a href=&quot;http://www.computerworld.com/s/article/9236558/Python_gets_a_big_data_boost_from_DARPA&quot;&gt;details&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="227" LastEditorUserId="227" LastEditDate="2014-05-19T08:13:05.037" LastActivityDate="2014-05-19T08:13:05.037" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="136" PostTypeId="5" CreationDate="2014-05-18T12:55:39.657" Score="0" Body="&lt;p&gt;MapReduce is a framework for processing parallelizable problems across huge datasets using a large number of computers (nodes), collectively referred to as a cluster (if all nodes are on the same local network and use similar hardware) or a grid (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware). Computational processing can occur on data stored either in a file system (unstructured) or in a database (structured). MapReduce can take advantage of locality of data, processing it on or near the storage assets in order to reduce the distance over which it must be transmitted.&lt;/p&gt;&#xA;" OwnerUserId="227" LastEditorUserId="111611" LastEditDate="2021-02-10T14:59:02.167" LastActivityDate="2021-02-10T14:59:02.167" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="137" PostTypeId="4" CreationDate="2014-05-18T12:55:39.657" Score="0" Body="MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster." OwnerUserId="227" LastEditorUserId="227" LastEditDate="2014-05-20T13:53:39.727" LastActivityDate="2014-05-20T13:53:39.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="138" PostTypeId="1" AcceptedAnswerId="251" CreationDate="2014-05-18T14:02:51.350" Score="10" ViewCount="224" Body="&lt;p&gt;Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend to reach out for C/C++/low-level languages. The possibility of tailoring the code to the needs seems to be what makes these languages so appealing for BigData -- be it concerning memory management, parallelism, disk access, or even low-level optimizations (via assembly constructs at C/C++ level).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course such set of benefits would not come without a cost: writing the code, and sometimes even &lt;em&gt;reinventing the wheel&lt;/em&gt;, can be quite expensive/tiresome. Although there are lots of libraries available, people are inclined to write the code by themselves whenever they need to &lt;em&gt;grant&lt;/em&gt; performance. What &lt;em&gt;disables&lt;/em&gt; performance assertions from using libraries while processing large databases?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider an entreprise that continuously crawls webpages and parses the data collected. For each sliding-window, different data mining algorithms are run upon the data extracted. Why would the developers ditch off using available libraries/frameworks (be it for crawling, text processing, and data mining)? Using stuff already implemented would not only ease the burden of coding the whole process, but also would save a lot of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In a single shot&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;what makes writing the code by oneself a &lt;em&gt;guarantee&lt;/em&gt; of performance?&lt;/li&gt;&#xA;&lt;li&gt;why is it &lt;em&gt;risky&lt;/em&gt; to rely on a frameworks/libraries when you must &lt;strong&gt;assure&lt;/strong&gt; high performance?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-05-29T15:01:17.840" LastActivityDate="2014-06-10T05:57:13.897" Title="Why is it hard to grant efficiency while using libraries?" Tags="|bigdata|efficiency|performance|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="139" PostTypeId="2" ParentId="76" CreationDate="2014-05-18T14:04:37.870" Score="3" Body="&lt;p&gt;I'd suggest &lt;a href=&quot;http://kafka.apache.org&quot; rel=&quot;nofollow&quot;&gt;Apache Kafka&lt;/a&gt; as message store and any stream processing solution of your choice like &lt;a href=&quot;https://camel.apache.org&quot; rel=&quot;nofollow&quot;&gt;Apache Camel&lt;/a&gt; or &lt;a href=&quot;https://github.com/apache/incubator-storm&quot; rel=&quot;nofollow&quot;&gt;Twitter Storm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="118" LastActivityDate="2014-05-18T14:04:37.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="140" PostTypeId="2" ParentId="107" CreationDate="2014-05-18T14:30:10.553" Score="4" Body="&lt;p&gt;I've read very good &lt;a href=&quot;http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; recently that suggests using &lt;a href=&quot;https://github.com/nathanmarz/storm&quot; rel=&quot;nofollow&quot;&gt;Twitter storm&lt;/a&gt; for a task that looks pretty similar to yours.&lt;/p&gt;&#xA;" OwnerUserId="118" LastActivityDate="2014-05-18T14:30:10.553" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="141" PostTypeId="5" CreationDate="2014-05-18T14:36:16.350" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2014-05-18T14:36:16.350" LastActivityDate="2014-05-18T14:36:16.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="142" PostTypeId="4" CreationDate="2014-05-18T14:36:16.350" Score="0" Body="Efficiency, in algorithmic processing, is usually associated to resource usage. The metrics to evaluate the efficiency of a process are commonly account for execution time, memory/disk or storage requirements, network usage and power consumption." OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-05-20T13:51:49.240" LastActivityDate="2014-05-20T13:51:49.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="143" PostTypeId="1" AcceptedAnswerId="165" CreationDate="2014-05-18T14:37:20.477" Score="10" ViewCount="1064" Body="&lt;p&gt;As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented Database (which I try to write/play a little around with C#), which technique you suggest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read about MurMurhash-2 and specially v3 comments say Murmur is very fast. Also Lucene.Net has good comments on it. But what about their memory footprints in general? Is there any efficient solution which uses less footprint (and of course if faster is preferable) than Lucene or Murmur? Or should I write a special index structure to get the best results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try to write my own, then is there any accepted scale for a good indexing, something like 1% of data-node, or 5% of data-node? Any useful hint will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-19T12:05:13.513" LastActivityDate="2014-05-19T12:05:13.513" Title="What is the most efficient data indexing technique" Tags="|nosql|efficiency|indexing|data-indexing-techniques|.net|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="144" PostTypeId="5" CreationDate="2014-05-18T14:58:34.853" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cluster analysis&lt;/a&gt; is the task of grouping objects into subsets (called &lt;strong&gt;clusters&lt;/strong&gt;) so that observations in the same cluster are similar in some sense, while observations in different clusters are dissimilar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;/questions/tagged/machine-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;machine-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;machine-learning&lt;/a&gt; and &lt;a href=&quot;/questions/tagged/data-mining&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;data-mining&amp;#39;&quot; rel=&quot;tag&quot;&gt;data-mining&lt;/a&gt;, clustering is a method of &lt;strong&gt;unsupervised learning&lt;/strong&gt; used to discover hidden structure in unlabeled data, and is commonly used in &lt;em&gt;exploratory data analysis&lt;/em&gt;. Popular algorithms include &lt;a href=&quot;/questions/tagged/k-means&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;k-means&amp;#39;&quot; rel=&quot;tag&quot;&gt;k-means&lt;/a&gt;, expectation maximization (EM), spectral clustering, correlation clustering and hierarchical clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Related topics: &lt;a href=&quot;/questions/tagged/classification&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;classification&amp;#39;&quot; rel=&quot;tag&quot;&gt;classification&lt;/a&gt;, pattern-recognition, knowledge discovery, taxonomy. Not to be confused with &lt;em&gt;cluster computing&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2018-01-01T18:56:53.570" LastActivityDate="2018-01-01T18:56:53.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="145" PostTypeId="4" CreationDate="2014-05-18T14:58:34.853" Score="0" Body="Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). It is a main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval etc." OwnerUserId="118" LastEditorUserId="118" LastEditDate="2014-05-20T13:53:26.907" LastActivityDate="2014-05-20T13:53:26.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="146" PostTypeId="5" CreationDate="2014-05-18T15:01:24.080" Score="0" Body="&lt;p&gt;Natural language processing (NLP) is a subfield of artificial intelligence that involves transforming or extracting useful information from natural language data. Methods include machine-learning and rule-based approaches. It is often regarded as the engineering arm of Computational Linguistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NLP tasks&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Text pre-processing&lt;/li&gt;&#xA;&lt;li&gt;Coreference resolution&lt;/li&gt;&#xA;&lt;li&gt;Dependency parsing&lt;/li&gt;&#xA;&lt;li&gt;Document summarization&lt;/li&gt;&#xA;&lt;li&gt;Named entity recognition (NER) &lt;a href=&quot;/questions/tagged/named-entity-recognition&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;named-entity-recognition&amp;#39;&quot; rel=&quot;tag&quot;&gt;named-entity-recognition&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Information extraction (IE) &lt;a href=&quot;/questions/tagged/information-retrieval&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;information-retrieval&amp;#39;&quot; rel=&quot;tag&quot;&gt;information-retrieval&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;Language modeling&lt;/li&gt;&#xA;&lt;li&gt;Part-of-speech (POS) tagging&lt;/li&gt;&#xA;&lt;li&gt;Morphological analysis and wordform generation&lt;/li&gt;&#xA;&lt;li&gt;Phrase-structure (constituency) parsing &lt;/li&gt;&#xA;&lt;li&gt;Machine translation (MT) &lt;a href=&quot;/questions/tagged/machine-translation&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;machine-translation&amp;#39;&quot; rel=&quot;tag&quot;&gt;machine-translation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Question answering (QA)&lt;/li&gt;&#xA;&lt;li&gt;Sentiment analysis &lt;a href=&quot;/questions/tagged/sentiment-analysis&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;sentiment-analysis&amp;#39;&quot; rel=&quot;tag&quot;&gt;sentiment-analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Semantic parsing &lt;/li&gt;&#xA;&lt;li&gt;Text categorization &lt;/li&gt;&#xA;&lt;li&gt;Textual entailment detection&lt;/li&gt;&#xA;&lt;li&gt;Topic modeling &lt;a href=&quot;/questions/tagged/topic-model&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;topic-model&amp;#39;&quot; rel=&quot;tag&quot;&gt;topic-model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Word Sense Disambiguation (WSD)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Beginner books on Natural Language Processing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/0262133601&quot; rel=&quot;nofollow noreferrer&quot;&gt;Foundations of Statistical Natural Language Processing&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/0131873210&quot; rel=&quot;nofollow noreferrer&quot;&gt;Speech and Language Processing&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.nltk.org/book/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK Book&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="21163" LastEditDate="2018-04-10T01:03:28.253" LastActivityDate="2018-04-10T01:03:28.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="147" PostTypeId="4" CreationDate="2014-05-18T15:01:24.080" Score="0" Body="Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation." OwnerUserId="118" LastEditorUserId="118" LastEditDate="2014-05-20T13:52:59.427" LastActivityDate="2014-05-20T13:52:59.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="148" PostTypeId="5" CreationDate="2014-05-18T15:08:08.913" Score="0" Body="&lt;p&gt;In Computer science and Technologies The Data is the most important part.&#xA;Since to work with data in an efficient manner to store the data in mediums and reuse, we use some techniques which named in general indexing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tag interests with these techniques, efficiency, and anything about using stored data with indexing&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:53:31.717" LastActivityDate="2014-05-20T13:53:31.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="149" PostTypeId="4" CreationDate="2014-05-18T15:08:08.913" Score="0" Body="In Computer science and Technologies The Data is the most important part.&#xD;&#xA;Since to work with data in an efficient manner to store the data in mediums and reuse, we use some techniques which named in general indexing.&#xD;&#xA;&#xD;&#xA;This tag interests with these techniques, efficiency, and anything about using stored data with indexing" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:52:19.333" LastActivityDate="2014-05-20T13:52:19.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="151" PostTypeId="5" CreationDate="2014-05-18T15:34:16.437" Score="0" Body="&lt;p&gt;Indexing is the almost most important part of data to get an efficient, properly storing and retrieval data from mediums&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In different Programming Languages, there are different indexing algorithms and structures can be found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an Example, in Java Language the Apache Foundation's Lucene is very popular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In years there are very efficient, fast, scalable Indexing Algorithms projectioned to the computer-world such as MurMurHashing Algorithm (using by some NoSQL Databases), B+Tree Algorithm (using by versions of Windows OS itself), or Lucene (very popular in web technologies) and its variations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also there can be found problem-specific Indexing within Chemicals or Medical Data Representations in Digital World&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:48:12.163" LastActivityDate="2014-05-20T13:48:12.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="152" PostTypeId="4" CreationDate="2014-05-18T15:34:16.437" Score="0" Body="Indexing is the almost most important part of data to get an efficient, properly storing and retrieval data from mediums" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:53:17.567" LastActivityDate="2014-05-20T13:53:17.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="153" PostTypeId="2" ParentId="81" CreationDate="2014-05-18T17:38:01.383" Score="3" Body="&lt;p&gt;The answers presented so far are very nice, but I was also expecting an emphasis on a particular difference between parallel and distributed processing: the code executed. Considering parallel processes, the code executed is the same, regardless of the level of parallelism (instruction, data, task). You write a &lt;em&gt;single code&lt;/em&gt;, and it will be executed by different threads/processors, e.g., while computing matrices products, or generating permutations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, distributed computing involves the execution of different algorithms/programs at the same time in different processors (from one or more machines). Such computations are later merged into a intermediate/final results by using the available means of data communication/synchronization (shared memory, network). Further, distributed computing is very appealing for BigData processing, as it allows for exploiting disk parallelism (usually the bottleneck for large databases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, for the level of parallelism, it may be taken rather as a constraint on the synchronization. For example, in GPGPU, which is single-instruction multiple-data (SIMD), the parallelism occurs by having different inputs for a single instruction, each pair &lt;em&gt;(data_i, instruction)&lt;/em&gt; being executed by a different thread. Such is the restraint that, in case of divergent branches, it is necessary to discard lots of unnecessary computations, until the threads reconverge. For CPU threads, though, they commonly diverge; yet, one may use synchronization structures to grant concurrent execution of specific sections of the code.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-05-18T17:38:01.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="154" PostTypeId="2" ParentId="125" CreationDate="2014-05-18T17:53:37.750" Score="3" Body="&lt;p&gt;Check &lt;a href=&quot;http://www.martinfowler.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Martin Fowler's Personal website&lt;/a&gt;. He writes good and specially answers one of your questions in his book: &amp;quot;NoSQL Distilled&amp;quot;&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="29169" LastEditDate="2021-03-15T21:14:08.800" LastActivityDate="2021-03-15T21:14:08.800" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="155" PostTypeId="1" AcceptedAnswerId="158" CreationDate="2014-05-18T18:45:38.957" Score="202" ViewCount="33053" Body="&lt;p&gt;One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?&lt;/p&gt;&#xA;" OwnerUserId="227" LastEditorUserId="26596" LastEditDate="2016-12-05T22:33:53.380" LastActivityDate="2022-07-01T05:57:50.363" Title="Publicly Available Datasets" Tags="|open-source|dataset|" AnswerCount="35" CommentCount="9" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="156" PostTypeId="2" ParentId="155" CreationDate="2014-05-18T19:19:44.240" Score="39" Body="&lt;p&gt;&lt;a href=&quot;https://www.freebase.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Freebase&lt;/a&gt; is a free community driven database that spans many interesting topics and contains about 2,5 billion facts in machine readable format. It is also have good API to perform data queries.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.datapure.co/open-data-sets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is another compiled list of open data sets&lt;/p&gt;&#xA;" OwnerUserId="118" LastEditorUserId="120060" LastEditDate="2021-07-08T20:42:25.690" LastActivityDate="2021-07-08T20:42:25.690" CommentCount="1" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 4.0" />
  <row Id="157" PostTypeId="2" ParentId="41" CreationDate="2014-05-18T19:22:05.160" Score="12" Body="&lt;p&gt;R is great for &quot;big data&quot;! However, you need a workflow since R is limited (with some simplification) by the amount of RAM in the operating system. The approach I take is to interact with a relational database (see the &lt;code&gt;RSQLite&lt;/code&gt; package for creating and interacting with a SQLite databse), run SQL-style queries to understand the structure of the data, and then extract particular subsets of the data for computationally-intensive statistical analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This just one approach, however: there are packages that allow you to interact with other databases (e.g., Monet) or run analyses in R with fewer memory limitations (e.g., see &lt;code&gt;pbdR&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="36" LastActivityDate="2014-05-18T19:22:05.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="158" PostTypeId="2" ParentId="155" CreationDate="2014-05-18T19:29:53.530" Score="112" Body="&lt;p&gt;There is, in fact, a very reasonable list of publicly-available datasets, supported by different enterprises/sources. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of them are below:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://aws.amazon.com/publicdatasets/&quot; rel=&quot;noreferrer&quot;&gt;Public Datasets on Amazon WebServices&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://fimi.ua.ac.be/data/&quot; rel=&quot;noreferrer&quot;&gt;Frequent Itemset Mining Implementation Repository&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;noreferrer&quot;&gt;UCI Machine Learning Repository&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdnuggets.com/datasets/index.html&quot; rel=&quot;noreferrer&quot;&gt;KDnuggets&lt;/a&gt; -- a big list of lots of public repositories.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, two considerations on your question. First one, regarding policies of database sharing. From personal experience, there are some databases that can't be made publicly available, either for involving privacy restraints (as for some social network information) or for concerning government information (like health system databases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point concerns the usage/application of the dataset. Although some bases can be reprocessed to suit the needs of the application, it would be great to have some &lt;em&gt;nice organization&lt;/em&gt; of the datasets by purpose. The &lt;em&gt;taxonomy&lt;/em&gt; should involve social graph analysis, itemset mining, classification, and lots of other research areas there may be.&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="26596" LastEditDate="2016-11-29T04:04:46.463" LastActivityDate="2016-11-29T04:04:46.463" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="159" PostTypeId="1" AcceptedAnswerId="160" CreationDate="2014-05-18T19:46:44.653" Score="6" ViewCount="563" Body="&lt;p&gt;I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data science is just a mix of computer science and statistics techniques.&#xA;So I always think about this. If it is just a trend and if in 10 years from now, someone will still mention Data Science as an entire field or just a subject/topic inside CS or stats.&#xA;What do you think?&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2014-05-18T21:05:28.990" Title="Is Data Science just a trend or is a long term concept?" Tags="|bigdata|machine-learning|databases|statistics|education|" AnswerCount="1" CommentCount="3" ClosedDate="2014-05-19T08:54:23.303" ContentLicense="CC BY-SA 3.0" />
  <row Id="160" PostTypeId="2" ParentId="159" CreationDate="2014-05-18T21:05:28.990" Score="12" Body="&lt;p&gt;The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like &quot;Big Data&quot; that some people consider as &quot;hypes&quot; or &quot;buzzwords&quot; don't make it easier to flesh out an appropriate answer here. But I'll try. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, interdisciplinary fields often seem to have the problem of not being taken serious by either of the fields they are spanning. However, the more research is invested into a particular field, the greater is the urge to split this field into several sub-topics. And these sub-topics sonner of later have to be re-combined in new ways, in order to prevent an overspecialization, and to increase and broaden the applicability of techniques that are developed by the (over?)specialized experts in the different fields. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I consider &quot;Data Science&quot; as such an approach to combine the expertise and findings from different fields. You described it as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...a mix of computer science and statistics techniques&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And indeed, several questions here aim at the differentiation between data science and statistics. But a pure statistician will most likely not be able to set up a Hadoop cluster and show the results of his analysis in an interactive HTML5 dashboard. And someone who can implement a nice HTML5 dashboard might not be so familiar with the mathematical background of a Chi-Squared-Test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is reasonable to assume that giving students enough knowledge to &lt;em&gt;apply&lt;/em&gt; the most important techniques from the different fields that are covered by data science will lead to new applications of these techniques, and be beneficial - also for the &quot;purists&quot; in these fields. The combination of these techniques is not straightforward in many cases, and can justify an own branch of research. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also asked whether in 10 years, data science will be considered as &quot;just a topic inside computer science&quot;. Again: Nobody can say for sure. But I wonder at which point people stopped asking the question whether &quot;Computer Science&quot; will one day only be considered only as a mix of (or a subject of) Electrical Engineering and Mathematics...&lt;/p&gt;&#xA;" OwnerUserId="156" LastActivityDate="2014-05-18T21:05:28.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="161" PostTypeId="2" ParentId="138" CreationDate="2014-05-18T21:25:29.040" Score="7" Body="&lt;p&gt;I don't think that everyone reaches for C/C++ when performance is an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data scientist using, for example, scikit-learn on her desktop is already calling heavily optimized native routines to do the number crunching. There is no point in writing new code for speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the distributed &quot;big data&quot; context, you are more typically bottleneck on data movement: network transfer and I/O. Native code does not help. What helps is not writing the same code to run faster, but writing smarter code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Higher-level languages are going to let you implement more sophisticated distributed algorithms in a given amount of developer time than C/C++. At scale, the smarter algorithm with better data movement will beat dumb native code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also usually true that developer time, and bugs, cost loads more than new hardware. A year of a senior developer's time might be $200K fully loaded; over a year that also rents hundreds of servers worth of computation time. It may just not make sense in most cases to bother optimizing over throwing more hardware at it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I don't understand the follow up about &quot;grant&quot; and &quot;disable&quot; and &quot;assert&quot;?&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-18T21:25:29.040" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="162" PostTypeId="2" ParentId="155" CreationDate="2014-05-18T22:16:19.300" Score="45" Body="&lt;p&gt;There are many openly available data sets, one many people often overlook is &lt;a href=&quot;http://www.data.gov/&quot;&gt;data.gov&lt;/a&gt;. As mentioned previously Freebase is great, so are all the examples posted by @Rubens&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2014-05-18T22:16:19.300" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="163" PostTypeId="2" ParentId="138" CreationDate="2014-05-18T23:21:07.220" Score="4" Body="&lt;p&gt;As all we know, in Digital world there are many ways to do the same work / get expected results..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And responsibilities / risks which comes from the code are on developers' shoulders..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is small but i guess a very useful example from .NET world..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So Many .NET developers use the built-in BinaryReader - BinaryWriter on their data serialization for performance / get control over the process..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is CSharp source code of the FrameWork's built in BinaryWriter class' one of the overloaded Write Methods :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;// Writes a boolean to this stream. A single byte is written to the stream&#xA;// with the value 0 representing false or the value 1 representing true.&#xA;// &#xA;public virtual void Write(bool value) &#xA;{&#xA;     //_buffer is a byte array which declared in ctor / init codes of the class&#xA;    _buffer = ((byte) (value? 1:0));&#xA;&#xA;    //OutStream is the stream instance which BinaryWriter Writes the value(s) into it.&#xA;    OutStream.WriteByte(_buffer[0]);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you see, this method could written without the extra assigning to _buffer variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public virtual void Write(bool value) &#xA;{&#xA;    OutStream.WriteByte((byte) (value ? 1 : 0));&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Without assigning we could gain few milliseconds..This few milliseconds can accept as &quot;almost nothing&quot; but what if there are multi-thousands of writing (i.e. in a server process)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets suppose that &quot;few&quot; is 2 (milliseconds) and multi-Thousands instances are only 2.000..&#xA;This means 4 seconds more process time..4 seconds later returning..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we continue to subject from .NET and if you can check the source codes of BCL - .NET Base Class Library- from MSDN you can see a lot of performance losts from the developer decides..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any of the point from BCL source It's normal that you see developer decided to use while() or foreach() loops which could implement a faster for() loop in their code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This small gains give us the total performance..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if we return to the BinaryWriter.Write() Method..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually extra assigning to a _buffer implementation is not a developer fault..This is exactly decide to &quot;stay in safe&quot; !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that we decide to not use _buffer and decided to implement the second method..If we try to send multi-thousands bytes over a wire (i.e. upload / download a BLOB or CLOB data) with the second method, it can fail commonly because of connection lost..Cause we try to send all data without any checks and controlling mechanism.When connection lost, Both the server and Client never know the sent data completed or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the developer decides &quot;stay in safe&quot; then normally it means performance costs depends to implemented &quot;stay in safe&quot; mechanism(s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if the developer decides &quot;get risky, gain performance&quot; this is not a fault also..Till there are some discussions about &quot;risky&quot; coding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as a small note : Commercial library developers always try to stay in safe because they can't know where their code will use.&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-18T23:29:39.227" LastActivityDate="2014-05-18T23:29:39.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="164" PostTypeId="2" ParentId="14" CreationDate="2014-05-19T00:09:17.900" Score="11" Body="&lt;p&gt;What @Clayton posted seems about right to me, for those terms, and for &quot;data mining&quot; being one tool of the data scientist. However, I haven't really used the term &quot;data collection,&quot; and it doesn't strike me as synonymous with &quot;data mining.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own answer to your question: &lt;strong&gt;no&lt;/strong&gt;, the terms aren't the same. Definitions may be loose in this field, but I haven't seen those terms used interchangeably. In my work, we sometimes use them to differentiate between goals, or methodologies. For us, &lt;a href=&quot;/questions/tagged/data-science&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;data-science&amp;#39;&quot; rel=&quot;tag&quot;&gt;data-science&lt;/a&gt; is more about testing a hypothesis, and typically the data have been collected just for that purpose. &lt;a href=&quot;/questions/tagged/data-mining&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;data-mining&amp;#39;&quot; rel=&quot;tag&quot;&gt;data-mining&lt;/a&gt; is more about sifting through existing data, looking for structure, and perhaps generating hypotheses. Data mining can start with a hypothesis, but it's often very weak or general, and can be difficult to resolve with confidence. (Dig long enough and you'll find &lt;em&gt;something&lt;/em&gt;, though it may turn out to be pyrite.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we also have used &quot;data science&quot; as a wider term, to include &quot;data mining.&quot; We also talk about &quot;data modeling,&quot; which for us is about finding a model for a system of interest, based on data as well as other knowledge and objectives. Sometimes that means trying to find the math that explains the real system, and sometimes it means finding a predictive model that is good enough for a purpose.&lt;/p&gt;&#xA;" OwnerUserId="208" LastActivityDate="2014-05-19T00:09:17.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="165" PostTypeId="2" ParentId="143" CreationDate="2014-05-19T08:46:47.683" Score="10" Body="&lt;p&gt;I think you messed up some things in your question. Lucene (I know nothing about Lucene,NET, but I suppose is the same) is a library used to analyze, split in tokens, and store documents in order to be able to query and retrieve them later. Lucene has a pretty old but effective model, it uses inverted trees to find and retrieve documents. Without further details, all documents are split in tokens (terms), and for each term is maintained a data structure, which stores all the documents which contains the given term. As a data structure could be used a BTree, a hash table and in the latest major revisions you can even plug in your own data structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A BTree (see &lt;a href=&quot;http://en.wikipedia.org/wiki/B-tree&quot;&gt;Wikipedia page&lt;/a&gt; for further details), is a kind of a tree data structure, which is appropriate for working with big chunks of data and is often used for storing tree-like ordered structures on disk. For in-memory other trees performs better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Murmur hash (see &lt;a href=&quot;http://en.wikipedia.org/wiki/MurmurHash&quot;&gt;Wikipedia page&lt;/a&gt; for further details), is a family of hash functions used in hash table. The implementation of the hash table is not important, it could be a standard chained implementation or more advanced open hash addressing scheme. The idea is that the hash tables allows one to get fast a key, from an unordered set of keys, and can answer to tasks like: is this key part of this set of keys? which is the value associated with this key? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now back to your main problem. You have one library (Lucene) and to data structures, both data structures are used in Lucene. Now you see that it is not possible to answer your question in these terms since they are not comparable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, regarding you footprint and performance part of the question. First of all you have to know which kind of operations you need to implement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Do you need only get value for key, or do you need to find all elements in a range? In other words do you need order or not?&lt;/em&gt; If you do, than a tree can help. If you do not, than a hash table, which is faster could be used instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Do you have a lot of data which does not fit the memory?&lt;/em&gt; If yes than a disk-based solution would help (like BTree). If your data fit the memory, than use the fastest in-memory solution and use disk only as a storage (with a different structure, much simpler).&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-05-19T08:46:47.683" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="166" PostTypeId="2" ParentId="129" CreationDate="2014-05-19T11:13:48.067" Score="8" Body="&lt;p&gt;Let's say you are predicting the topic of a document given its words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A generative model describes how likely each topic is, and how likely words are given the topic. This is how it says documents are actually &quot;generated&quot; by the world -- a topic arises according to some distribution, words arise because of the topic, you have a document. Classifying documents of words W into topic T is a matter of maximizing the joint likelihood: P(T,W) = P(W|T)P(T)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A discriminative model operates by only describing how likely a topic is given the words. It says nothing about how likely the words or topic are by themselves. The task is to model P(T|W) directly and find the T that maximizes this. These approaches do not care about P(T) or P(W) directly.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-19T11:13:48.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="167" PostTypeId="5" CreationDate="2014-05-19T12:17:45.960" Score="0" Body="&lt;p&gt;&lt;strong&gt;.NET&lt;/strong&gt; is a very popular Object Oriented Programming Language Family.This Family includes members such as &lt;strong&gt;C#&lt;/strong&gt; (pronounced &lt;strong&gt;CSharp&lt;/strong&gt;), &lt;strong&gt;VB.NET&lt;/strong&gt;, &lt;strong&gt;F#&lt;/strong&gt; (pronounced &lt;strong&gt;Fsharp&lt;/strong&gt;), &lt;strong&gt;J#&lt;/strong&gt; (pronounced &lt;strong&gt;JSharp&lt;/strong&gt;) and much more. The &lt;strong&gt;.NET&lt;/strong&gt; Family offers programming with small effort with well-known high speed of compiled languages such as &lt;strong&gt;C&lt;/strong&gt; and &lt;strong&gt;C++&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This Tag &lt;strong&gt;aims&lt;/strong&gt; to group &lt;strong&gt;Data Science Questions and Answers&lt;/strong&gt; which users want to operate their processes &lt;strong&gt;under .NET&lt;/strong&gt; Programming Language Family&lt;/p&gt;&#xA;" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:52:50.373" LastActivityDate="2014-05-20T13:52:50.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="168" PostTypeId="4" CreationDate="2014-05-19T12:17:45.960" Score="0" Body=".NET is a very popular Object Oriented Programming Language Family which includes members such as C# (pronounced CSharp), VB.NET, F# (pronounced FSharp), J# (pronounced JSharp) and much more. The .NET Family offers programming with small effort with well-known high speed of compiled languages such as C and C++&#xD;&#xA;&#xD;&#xA;This Tag aims to group Data Science Questions and Answers which users want to operate their processes under .NET Programming Language Family" OwnerUserId="229" LastEditorUserId="229" LastEditDate="2014-05-20T13:50:32.440" LastActivityDate="2014-05-20T13:50:32.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="169" PostTypeId="1" AcceptedAnswerId="170" CreationDate="2014-05-19T15:55:24.983" Score="15" ViewCount="5667" Body="&lt;p&gt;Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to describe the values (e.g. &quot;nationality&quot; vs &quot;bornIn&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to find some &quot;important&quot; attributes that somehow &quot;define&quot; the entities that they describe. So, when I find the same value for such an attribute, I will know that the two descriptions are most likely about the same entity (e.g. the same person).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the attribute &quot;lastName&quot; is more discriminative than the attribute &quot;nationality&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How could I (statistically) find such attributes that are more important than others?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A naive solution would be to take the average IDF of the values of each attribute and make this the &quot;importance&quot; factor of the attribute. A similar approach would be to count how many distinct values appear for each attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen the term feature, or attribute selection in machine learning, but I don't want to discard the remaining attributes, I just want to put higher weights to the most important ones.&lt;/p&gt;&#xA;" OwnerUserId="113" LastEditorUserId="113" LastEditDate="2015-05-18T13:30:46.940" LastActivityDate="2021-03-11T20:12:24.030" Title="How to specify important attributes?" Tags="|machine-learning|statistics|feature-selection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="170" PostTypeId="2" ParentId="169" CreationDate="2014-05-19T18:08:32.327" Score="16" Body="&lt;p&gt;A possible solution is to calculate the &lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain&quot; rel=&quot;nofollow noreferrer&quot;&gt;information gain&lt;/a&gt; associated to each attribute:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$I_{E}(f) = - \sum \limits_{i = 1}^m f_ilog_2f_i$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Initially you have the whole dataset, and compute the information gain of each item. The item with the best information gain is the one you should use to partition the dataset (considering the item's values). Then, perform the same computations for each item (but the ones selected), and always choose the one which best &lt;em&gt;describes/differentiates&lt;/em&gt; the entries from your dataset.&lt;/p&gt;&#xA;&lt;p&gt;There are implementations available for such computations. &lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision trees&lt;/a&gt; usually base their feature selection on the features with best information gain. You may use the resulting tree structure to find these &lt;em&gt;important&lt;/em&gt; items.&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="29169" LastEditDate="2021-03-11T20:12:24.030" LastActivityDate="2021-03-11T20:12:24.030" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="171" PostTypeId="2" ParentId="35" CreationDate="2014-05-19T19:44:48.500" Score="5" Body="&lt;p&gt;Two things you might find useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Meta_learning_%28computer_science%29&quot; rel=&quot;noreferrer&quot;&gt;meta-learning&lt;/a&gt; to speedup the search for the right model and the optimal parameters.&#xA;Meta learning consists in applying machine learning tools to the problem of finding the right machine learning tool/parameters for the problem at hand. This for instance &lt;a href=&quot;http://link.springer.com/chapter/10.1007/978-3-642-14464-6_11&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; for a practical example;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units&quot; rel=&quot;noreferrer&quot;&gt;gpucomputing&lt;/a&gt; to speedup the algorithm on larger datasets. For instance, &lt;a href=&quot;http://docs.opencv.org/modules/gpu/doc/introduction.html&quot; rel=&quot;noreferrer&quot;&gt;OpenCV can use GPUs&lt;/a&gt;, which are very effective at processing images/videos and can bring 10 to 100 speedups with respect to CPUs. As your computer most probably has a gpucomputing-able GPU, you could gain lots of time using it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-05-19T19:44:48.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="172" PostTypeId="1" CreationDate="2014-05-19T23:59:58.070" Score="27" ViewCount="20442" Body="&lt;p&gt;I have a modeling and scoring program that makes heavy use of the &lt;code&gt;DataFrame.isin&lt;/code&gt; function of pandas, searching through lists of facebook &quot;like&quot; records of individual users for each of a few thousand specific pages. This is the most time-consuming part of the program, more so than the modeling or scoring pieces, simply because it only runs on one core while the rest runs on a few dozen simultaneously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I know I could manually break up the dataframe into chunks and run the operation in parallel, is there any straightforward way to do that automatically? In other words, is there any kind of package out there that will recognize I'm running an easily-delegated operation and automatically distribute it? Perhaps that's asking for too much, but I've been surprised enough in the past by what's already available in Python, so I figure it's worth asking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other suggestions about how this might be accomplished (even if not by some magic unicorn package!) would also be appreciated. Mainly, just trying to find a way to shave off 15-20 minutes per run without spending an equal amount of time coding the solution.&lt;/p&gt;&#xA;" OwnerUserId="250" LastEditorUserId="84" LastEditDate="2014-05-20T04:47:25.207" LastActivityDate="2020-08-02T12:40:19.397" Title="Is there a straightforward way to run pandas.DataFrame.isin in parallel?" Tags="|performance|python|pandas|parallel|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="173" PostTypeId="2" ParentId="35" CreationDate="2014-05-20T03:56:43.147" Score="4" Body="&lt;p&gt;Guessing it's likely you've seen this &lt;a href=&quot;http://www.youtube.com/watch?v=1GhNXHCQGsM&quot; rel=&quot;nofollow&quot;&gt;YouTube demo&lt;/a&gt; and the related &lt;a href=&quot;http://www.youtube.com/watch?v=lmG_FjG4Dy8&quot; rel=&quot;nofollow&quot;&gt;Google Tech Talk&lt;/a&gt;, which is related to these papers: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://eprints.pascal-network.org/archive/00006951/01/cvpr2010.pdf&quot; rel=&quot;nofollow&quot;&gt;P-N Learning: Bootstrapping Binary Classifiers by Structural Constraints&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://epubs.surrey.ac.uk/713800/1/Kalal-PAMI-2011%281%29.pdf&quot; rel=&quot;nofollow&quot;&gt;Tracking-Learning-Detection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And this set of code on GitHub for &lt;a href=&quot;https://github.com/zk00006/OpenTLD/&quot; rel=&quot;nofollow&quot;&gt;OpenTLD&lt;/a&gt;. If you check the &quot;read me&quot; on GitHub here, you'll notice that &lt;a href=&quot;https://github.com/zk00006/OpenTLD/blob/master/README&quot; rel=&quot;nofollow&quot;&gt;author's email (Zdenek Kalal)&lt;/a&gt; is listed, so it might be worth sending him an email about your questions, or even inviting him to reply to this question too.&lt;/p&gt;&#xA;" OwnerUserId="158" LastActivityDate="2014-05-20T03:56:43.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="174" PostTypeId="2" ParentId="116" CreationDate="2014-05-20T09:24:30.697" Score="5" Body="&lt;p&gt;Another suggestion is to test the &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;noreferrer&quot;&gt;logistic regression&lt;/a&gt;. As an added bonus, the  weights (coefficients) of the model will give you an idea of which sites are age-distriminant.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sklearn offers the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;noreferrer&quot;&gt;sklearn.linear_model.LogisticRegression&lt;/a&gt; package that is designed to handle sparse data as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentionned in the comments, in the present case, with more input variables than samples, you need to regularize the model (with &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;noreferrer&quot;&gt;sklearn.linear_model.LogisticRegression&lt;/a&gt; use the &lt;code&gt;penalty='l1'&lt;/code&gt; argument).&lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="172" LastEditDate="2014-05-20T19:36:16.283" LastActivityDate="2014-05-20T19:36:16.283" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="175" PostTypeId="1" CreationDate="2014-05-20T22:14:02.927" Score="3" ViewCount="62" Body="&lt;p&gt;I have data coming from a source system that is pipe delimited. Pipe was selected over comma since it was believed no pipes appeared in field, while it was known that commas do occur. After ingesting this data into Hive however it has been discovered that rarely a field does in fact contain a pipe character.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to a constraint we are unable to regenerate from source to escape the delimiter or change delimiters in the usual way. However we have the metadata used to create the Hive table. Could we use knowledge of the fields around the problem field to reprocess the file on our side to escape it or to change the file delimiter prior to reloading the data into Hive?&lt;/p&gt;&#xA;" OwnerUserId="249" LastActivityDate="2014-05-21T07:19:32.297" Title="Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?" Tags="|metadata|parsing|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="176" PostTypeId="1" AcceptedAnswerId="190" CreationDate="2014-05-21T05:29:36.787" Score="12" ViewCount="4886" Body="&lt;p&gt;I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the existing solutions is &lt;a href=&quot;http://www.stanford.edu/group/sonia/&quot;&gt;SoNIA: Social Network Image Animator&lt;/a&gt;. It let's you make movies like &lt;a href=&quot;https://www.youtube.com/watch?v=yGSNCED6mDc&quot;&gt;this one&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SoNIA's documentation says that it's broken at the moment, and besides this I would prefer JavaScript-based solution instead. So, my question is: are you familiar with any tools or are you able to point me to some libraries which would make this task as easy as possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right after posting this question I'll dig into &lt;a href=&quot;http://sigmajs.org/&quot;&gt;sigma.js&lt;/a&gt;, so please consider this library covered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, my input data would be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time_elapsed; node1; node2&#xA;1; A; B&#xA;2; A; C&#xA;3; B; C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, here we have three points in time (1, 2, 3), three nodes (A, B, C), and three edges, which represent a triadic closure between the three considered nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, every node will have two attributes (age and gender), so I would like to be able to change the shape/colour of the nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, after adding a new node, it would be perfect to have some ForceAtlas2 or similar algorithm to adjust the layout of the graph.&lt;/p&gt;&#xA;" OwnerUserId="173" LastEditorUserId="173" LastEditDate="2014-05-21T05:51:58.330" LastActivityDate="2014-05-22T12:14:49.727" Title="How to animate growth of a social network?" Tags="|social-network-analysis|time-series|javascript|visualization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="177" PostTypeId="5" CreationDate="2014-05-21T07:05:44.780" Score="0" Body="&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.mongodb.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB&lt;/a&gt;&lt;/strong&gt; is a &lt;a href=&quot;https://www.mongodb.com/who-uses-mongodb&quot; rel=&quot;nofollow noreferrer&quot;&gt;widely used&lt;/a&gt;, &lt;a href=&quot;https://www.mongodb.com/use-cases&quot; rel=&quot;nofollow noreferrer&quot;&gt;general-purpose&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Document-oriented_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;document-oriented NoSQL database&lt;/a&gt; with features including high-availability replication and auto-sharding for horizontal scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MongoDB Community Edition database server and tools are &lt;a href=&quot;https://github.com/mongodb/mongo&quot; rel=&quot;nofollow noreferrer&quot;&gt;open-sourced&lt;/a&gt; and available under &lt;a href=&quot;https://www.mongodb.com/licensing/server-side-public-license&quot; rel=&quot;nofollow noreferrer&quot;&gt;Server Side Public License&lt;/a&gt; (all versions released after October 16, 2018) or &lt;a href=&quot;http://www.gnu.org/licenses/agpl-3.0.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AGPL v3.0 license&lt;/a&gt; (versions released prior to October 16, 2018). Commercial Licenses are also available from MongoDB, Inc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MongoDB has strong support for dynamic querying and aggregating data including &lt;a href=&quot;https://docs.mongodb.com/manual/core/map-reduce/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MapReduce&lt;/a&gt; and an &lt;a href=&quot;https://docs.mongodb.com/manual/aggregation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Aggregation Framework&lt;/a&gt;.  MongoDB uses &lt;a href=&quot;http://bsonspec.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;BSON&lt;/a&gt; (Binary JSON) format for storage purposes and the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Wire Protocol&lt;/a&gt; for communication between client drivers and the MongoDB server.  Officially supported &lt;a href=&quot;https://docs.mongodb.com/ecosystem/drivers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Drivers and Client Libraries&lt;/a&gt; are available for most popular programming languages, and there are also &lt;a href=&quot;https://docs.mongodb.com/ecosystem/drivers/community-supported-drivers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Community Supported Drivers&lt;/a&gt; which offer alternative implementations and support for further programming languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latest MongoDB server releases can be &lt;a href=&quot;https://docs.mongodb.com/manual/installation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;installed via common packaging systems&lt;/a&gt; or downloaded as binary archives from &lt;a href=&quot;https://www.mongodb.com/download-center&quot; rel=&quot;nofollow noreferrer&quot;&gt;mongodb.com&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current production release series of &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/4.0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB 4.0&lt;/a&gt;. It is generally recommended to stay current with the latest minor release of a production release series (e.g. 4.0.x) to take advantage of bug fixes and backward-compatible improvements. For more information on the versioning scheme used by the server, see &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/#mongodb-version-numbers&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Version Numbers&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;FAQ&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; rel=&quot;nofollow noreferrer&quot;&gt;mongodb-user discussion forum&lt;/a&gt; is very busy and almost every question on earth has been asked on there, so try searching the archives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tip: searching Google with &lt;code&gt;&quot; site:groups.google.com/group/mongodb-user&quot;&lt;/code&gt; added to your search terms generally works better than using the Google Groups search bar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For help with data modeling (schema design), check out the &lt;a href=&quot;https://docs.mongodb.com/manual/data-modeling/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Modeling Considerations for MongoDB Applications&lt;/a&gt; documentation page or search the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; rel=&quot;nofollow noreferrer&quot;&gt;mongodb-user discussion forum&lt;/a&gt; archives. For information on MongoDB Security, view the &lt;a href=&quot;https://docs.mongodb.com/manual/security/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Security section&lt;/a&gt; of the MongoDB Manual which includes a &lt;a href=&quot;https://docs.mongodb.com/manual/administration/security-checklist/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Security Checklist&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MongoDB, Inc. (the company behind MongoDB) provides archives of many &lt;a href=&quot;https://www.mongodb.com/presentations&quot; rel=&quot;nofollow noreferrer&quot;&gt;presentations&lt;/a&gt; from their &lt;a href=&quot;https://www.mongodb.com/events&quot; rel=&quot;nofollow noreferrer&quot;&gt;events&lt;/a&gt; such as conferences and &lt;a href=&quot;https://www.mongodb.com/webinars&quot; rel=&quot;nofollow noreferrer&quot;&gt;webinars&lt;/a&gt;. They also develop a number of related tools and services including &lt;a href=&quot;https://cloud.mongodb.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Cloud Manager&lt;/a&gt;, &lt;a href=&quot;https://www.mongodb.com/products/ops-manager&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Ops Manager&lt;/a&gt;, &lt;a href=&quot;https://www.mongodb.com/atlas&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Atlas&lt;/a&gt;, and &lt;a href=&quot;https://www.mongodb.com/products/compass&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Compass&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Useful links&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/faq/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Frequently Asked Questions&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/installation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Install MongoDB Server&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/ecosystem/drivers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Client Drivers and Libraries&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/ecosystem/drivers/community-supported-drivers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Community Supported Drivers&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB manual&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB tutorials&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://university.mongodb.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB University courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/user/MongoDB/playlists&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB YouTube Channel&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB JIRA&lt;/a&gt; - bug/feature tracking for MongoDB server and drivers&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/atlas&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Atlas&lt;/a&gt; - hosted MongoDB as a Service&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cloud.mongodb.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Cloud Manager&lt;/a&gt; - freemium monitoring, automation, and backup&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/products/ops-manager&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Ops Manager&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/licensing/server-side-public-license/faq&quot; rel=&quot;nofollow noreferrer&quot;&gt;Server Side Public License (SSPL) FAQ&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bsonspec.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;BSON specification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/MongoDB&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Wikipedia Article&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/presentations&quot; rel=&quot;nofollow noreferrer&quot;&gt;Presentations&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/white-papers&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB White Papers&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://mongoosejs.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mongoose&lt;/a&gt; - MongoDB object modeling for Node.js&lt;/li&gt;&#xA;&lt;li&gt;IRC: freenode.net/#mongodb&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Initial Release:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;February 11, 2009&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Stable Release:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;6.2.0&lt;/code&gt; (Feb 7, 2019)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Document-oriented_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.mongodb.com/manual/release-notes/4.0&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Books&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1617291609&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB in Action: Covers MongoDB version 3.0 2nd Edition&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/mongodb-cookbook-second-edition&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Cookbook - Second Edition&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.apress.com/9781484211830&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Definitive Guide to MongoDB: Third Edition A complete guide to dealing with Big Data using MongoDB&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.apress.com/9781484206485&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical MongoDB Architecting, Developing, and Administering MongoDB&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/148420896X&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Basics&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920028031.do&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB the Definitive Guide: 2nd Edition&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920027041.do&quot; rel=&quot;nofollow noreferrer&quot;&gt;MongoDB Applied Design Patterns Practical Use Cases with the Leading NoSQL Database&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920018308.do&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scaling MongoDB Sharding, Cluster Setup, and Administration&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://openmymind.net/2011/3/28/The-Little-MongoDB-Book/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Little MongoDB Book&lt;/a&gt; - a free book introducing MongoDB (updated for MongoDB 2.6)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T13:45:37.843" LastActivityDate="2019-04-08T13:45:37.843" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="178" PostTypeId="4" CreationDate="2014-05-21T07:05:44.780" Score="0" Body="MongoDB is a scalable, high-performance, open source, document-oriented NoSQL database. It supports a large number of languages and application development platforms. Questions about server administration can be asked on http://dba.stackexchange.com." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T17:28:18.497" LastActivityDate="2019-04-08T17:28:18.497" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="179" PostTypeId="2" ParentId="176" CreationDate="2014-05-21T07:09:20.093" Score="6" Body="&lt;p&gt;My first guess is to &lt;a href=&quot;https://www.google.com/search?q=visualize%20social%20network%20in%20tableau&quot; rel=&quot;noreferrer&quot;&gt;visualize social network in Tableau&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And particularly: &lt;a href=&quot;http://www.clearlyandsimply.com/clearly_and_simply/2012/12/build-network-graphs-in-tableau.html&quot; rel=&quot;noreferrer&quot;&gt;building network graphs in Tableau&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need is to add time dimension to the &quot;Pages&quot; section to be able to see network change dynamics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is screen from the link above.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/Quq8G.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-05-21T07:18:48.453" LastActivityDate="2014-05-21T07:18:48.453" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="180" PostTypeId="5" CreationDate="2014-05-21T07:10:45.827" Score="0" Body="&lt;p&gt;Neo4j is a open-source, transactional, high performance native graph database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neo4j stores its data as a graph: Nodes are connected through Relationships, both with arbitrary properties. Neo4j features a graph-centric declarative query language called &lt;a href=&quot;http://neo4j.com/docs/stable/cypher-refcard&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cypher&lt;/a&gt;. Its drivers support many &lt;a href=&quot;http://neo4j.com/developer/language-guides&quot; rel=&quot;nofollow noreferrer&quot;&gt;programming languages&lt;/a&gt;. More details in the Online Training and the &lt;a href=&quot;http://neo4j.com/docs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neo4j Manual&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For questions that are not suitable for the StackOverflow Q&amp;amp;A Format, please reach out on &lt;a href=&quot;https://neo4j-users-slack-invite.herokuapp.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Slack&lt;/a&gt;. Neo4j also maintains a public &lt;a href=&quot;https://trello.com/b/2zFtvDnV/public-idea-board&quot; rel=&quot;nofollow noreferrer&quot;&gt;Trello Board&lt;/a&gt; for feature ideas and future development directions.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Useful Links&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://neo4j.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Official Site&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/neo4j/neo4j&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub Repo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://twitter.com/neo4j&quot; rel=&quot;nofollow noreferrer&quot;&gt;Twitter&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://youtube.com/neo4j&quot; rel=&quot;nofollow noreferrer&quot;&gt;YouTube&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/neueda/awesome-neo4j&quot; rel=&quot;nofollow noreferrer&quot;&gt;Awesome Neo4j&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T13:45:45.567" LastActivityDate="2019-04-08T13:45:45.567" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="181" PostTypeId="4" CreationDate="2014-05-21T07:10:45.827" Score="0" Body="Neo4j is an open-source graph database (GDB) well suited to connected data. Please mention your exact version of Neo4j when asking questions. You can use it for recommendation engines, fraud detection, graph-based search, network ops/security, and many other user cases.  The database is accessed via official drivers in Java, JavaScript, .Python and .NET, or community-contributed drivers in PHP, Ruby, R, Golang, Elixir, Swift and more." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T13:45:18.490" LastActivityDate="2019-04-08T13:45:18.490" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="182" PostTypeId="2" ParentId="175" CreationDate="2014-05-21T07:19:32.297" Score="4" Body="&lt;p&gt;So, a few of your rows will have too many columns by one or more as a result. That's easy to detect, but harder to infer where the error was -- which two columns are actually one? which delimiter is not a delimiter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases, you can use the metadata, because it helps you know when an interpretation of the columns can't be right. For example, if just the one column can have a text value, and all the others must be numeric, it's unambiguous where the error is. Any additional columns created by this error occur right after the text column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they're all text, this doesn't work of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be able to leverage more than the metadata's column type. For example you may know that some fields are from an enumerated set of values, and use that to determine when a column assignment is wrong.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-21T07:19:32.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="183" PostTypeId="2" ParentId="176" CreationDate="2014-05-21T10:53:26.870" Score="12" Body="&lt;h3&gt;Fancy animations are cool&lt;/h3&gt;&#xA;&lt;p&gt;I was very impressed when I saw &lt;a href=&quot;http://youtu.be/T7Ncmq6scck&quot; rel=&quot;noreferrer&quot;&gt;this animation&lt;/a&gt; of the &lt;a href=&quot;http://www.discourse.org&quot; rel=&quot;noreferrer&quot;&gt;discourse&lt;/a&gt; git repository. They used &lt;a href=&quot;https://code.google.com/p/gource/&quot; rel=&quot;noreferrer&quot;&gt;Gourse&lt;/a&gt; which is specifically for git. But it may give ideas about how to represent the dynamics of growth.&lt;/p&gt;&#xA;&lt;h3&gt;You can create animations with matplotlib&lt;/h3&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/13571425/1083707&quot;&gt;This stackoverflow answer&lt;/a&gt; seems to point at a python/networkx/matplotlib solution.&lt;/p&gt;&#xA;&lt;h3&gt;But D3.js provides interaction&lt;/h3&gt;&#xA;&lt;p&gt;If you're looking for a web-based solution then d3.js is excellent. See &lt;a href=&quot;http://bl.ocks.org/mbostock/4062045&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://mbostock.github.io/d3/talk/20111116/force-collapsible.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://bl.ocks.org/mbostock/929623&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; for example.&#xA;See also &lt;a href=&quot;https://stackoverflow.com/questions/7416659/interactive-graph-visualisation&quot;&gt;this stackoverflow question&lt;/a&gt;, the accepted answer points to D3.js again.&lt;/p&gt;&#xA;&lt;h2&gt;Conclusion&lt;/h2&gt;&#xA;&lt;p&gt;I would be drawn towards the python/networkx options for network analysis (possibly to add attributes to your raw data file for example). Then, for visualisation and dissemination D3.js is perfect. You might be surprised how easy it can be to write d3.js once you get into it. I believe &lt;a href=&quot;https://www.youtube.com/watch?v=8UtoIR2IEkI&quot; rel=&quot;noreferrer&quot;&gt;it even works within an ipython notebook!&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="262" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2014-05-21T15:39:57.380" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="184" PostTypeId="1" AcceptedAnswerId="185" CreationDate="2014-05-21T11:22:34.657" Score="9" ViewCount="520" Body="&lt;p&gt;The details of the Google Prediction API are on this &lt;a href=&quot;https://developers.google.com/prediction/&quot;&gt;page&lt;/a&gt;, but I am not able to find any details about the prediction algorithms running behind the API. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have gathered that they let you provide your preprocessing steps in PMML format.&lt;/p&gt;&#xA;" OwnerUserId="200" LastActivityDate="2014-06-10T04:47:13.040" Title="Google prediction API: What training/prediction methods Google Prediction API employs?" Tags="|tools|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="185" PostTypeId="2" ParentId="184" CreationDate="2014-05-21T14:14:38.797" Score="6" Body="&lt;p&gt;If you take a look over the specifications of PMML which you can find &lt;a href=&quot;http://www.dmg.org/v3-0/GeneralStructure.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; you can see on the left menu what options you have (like ModelTree, NaiveBayes, Neural Nets and so on).&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-05-21T14:14:38.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="186" PostTypeId="1" AcceptedAnswerId="187" CreationDate="2014-05-21T15:12:18.980" Score="9" ViewCount="350" Body="&lt;p&gt;I'm learning &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot;&gt;Support Vector Machines&lt;/a&gt;, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in each dimension of the separating hyperplane?&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="84" LastEditDate="2014-05-21T15:26:02.533" LastActivityDate="2014-05-21T15:39:54.830" Title="Using SVM as a binary classifier, is the label for a data point chosen by consensus?" Tags="|svm|classification|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="187" PostTypeId="2" ParentId="186" CreationDate="2014-05-21T15:39:54.830" Score="9" Body="&lt;p&gt;The term &lt;em&gt;consensus&lt;/em&gt;, as far as I'm concerned, is used rather for cases when you have more a than one source of metric/measure/choice from which to make a decision. And, in order to choose a possible result, you perform some &lt;em&gt;average evaluation/consensus&lt;/em&gt; over the values available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not the case for SVM. The algorithm is based on a &lt;a href=&quot;http://upload.wikimedia.org/wikipedia/commons/2/2a/Svm_max_sep_hyperplane_with_margin.png&quot; rel=&quot;noreferrer&quot;&gt;quadratic optimization&lt;/a&gt;, that maximizes the distance from the closest documents of two different classes, using a hyperplane to make the split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CCO7Z.png&quot; alt=&quot;Hyperplane separating two different classes&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the only &lt;em&gt;consensus&lt;/em&gt; here is the resulting hyperplane, computed from the closest documents of each class. In other words, the classes are attributed to each point by calculating the distance from the point to the hyperplane derived. If the distance is positive, it belongs to a certain class, otherwise, it belongs to the other one.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-05-21T15:39:54.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="188" PostTypeId="2" ParentId="134" CreationDate="2014-05-21T16:13:25.590" Score="5" Body="&lt;p&gt;You can use map reduce algorithms in Hadoop without programming them in Java. It is called streaming and works like Linux piping. If you believe that you can port your functions to read and write to terminal, it should work nicely. &lt;a href=&quot;http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; is example blog post which shows how to use map reduce functions written in Python in Hadoop.&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2014-05-21T16:13:25.590" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="189" PostTypeId="1" CreationDate="2014-05-21T19:41:19.857" Score="4" ViewCount="983" Body="&lt;p&gt;I'm currently using &lt;a href=&quot;http://en.wikipedia.org/wiki/General_Algebraic_Modeling_System&quot; rel=&quot;nofollow&quot;&gt;General Algebraic Modeling System&lt;/a&gt; (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the process over 4 cores (although I have more, CPLEX utilizes a maximum of 4 cores), and it finds an optimal solution in a relatively short amount of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an open source mixed integer programming tool that I could use as an alternative to GAMS and CPLEX? It must be comparable in speed or faster for me to consider it. I have a preference for R based solutions, but I'm open to suggestions of all kinds, and other users may be interested in different solutions.&lt;/p&gt;&#xA;" OwnerUserId="151" LastEditorUserId="84" LastEditDate="2014-06-17T16:18:02.437" LastActivityDate="2014-06-17T16:18:02.437" Title="Open source solver for large mixed integer programming task?" Tags="|r|open-source|parallel|optimization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="190" PostTypeId="2" ParentId="176" CreationDate="2014-05-22T12:14:49.727" Score="7" Body="&lt;p&gt;It turned out that this task was quite easy to accomplish using &lt;a href=&quot;http://visjs.org/&quot; rel=&quot;noreferrer&quot;&gt;vis.js&lt;/a&gt;. &lt;a href=&quot;http://visjs.org/examples/graph/20_navigation.html&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; was the best example code which I have found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example of what I have built upon this is &lt;a href=&quot;http://laboratoriumdanych.pl/jak-powstaje-siec/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; (scroll to the bottom of this post). This graph represents the growth of a subnetwork of Facebook friends. Green dots are females, blue ones are males. The darker the colour, the older the user. By clicking &quot;Dodaj węzły&quot; you can add more nodes and edges to the graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I am still interested in other ways to accomplish this task, so I won't accept any answer as for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your contributions!&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2014-05-22T12:14:49.727" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="191" PostTypeId="1" AcceptedAnswerId="194" CreationDate="2014-05-22T13:36:24.120" Score="8" ViewCount="1183" Body="&lt;p&gt;Can someone explain me, how to classify a data like MNIST with MLBP-Neural network if I make more than one output (e.g 8), I mean if I just use one output I can easily classify the data, but if I use more than one, which output should I choose ?&lt;/p&gt;&#xA;" OwnerUserId="273" LastActivityDate="2014-06-10T08:38:27.093" Title="Multi layer back propagation Neural network for classification" Tags="|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="192" PostTypeId="1" CreationDate="2014-05-22T15:15:41.133" Score="8" ViewCount="524" Body="&lt;p&gt;The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what are other typical data science applications, which may be used in a different verticals?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: customer churn prediction with machine learning, evaluating customer lifetime value, sales forecasting.&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2014-05-23T06:03:40.577" Title="What are the most popular data science application use cases for consumer web companies" Tags="|usecase|consumerweb|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="193" PostTypeId="2" ParentId="192" CreationDate="2014-05-22T15:43:57.160" Score="4" Body="&lt;p&gt;It depends, of course, on the focus of the company: commerce, service, etc.  In adition to the use cases you suggested, some other use cases would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Funnel analysis: Analyzing the way in which consumers use a website and complete a sale may include data science techniques, especially if the company operates at a large scale.&lt;/li&gt;&#xA;&lt;li&gt;Advertising: Companies that place ads use &lt;em&gt;a lot&lt;/em&gt; of machine learning techniques to analyze and predict which ads would be most effective or most remunerative give the user's demographics that would view them.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-05-22T15:43:57.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="194" PostTypeId="2" ParentId="191" CreationDate="2014-05-22T19:20:14.130" Score="5" Body="&lt;p&gt;Suppose that you need to classify something in K classes, where K &gt; 2. In this case the most often setup I use is one hot encoding. You will have K output columns, and in the training set you will set all values to 0, except the one which has the category index, which could have value 1. Thus, for each training data set instance you will have all outputs with values 0 or 1, all outputs sum to 1 for each instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks like a probability, which reminds me of a technique used often to connect some outputs which are modeled as probability. This is called softmax function, more details &lt;a href=&quot;http://en.wikipedia.org/wiki/Softmax_activation_function&quot; rel=&quot;noreferrer&quot;&gt;on Wikipedia&lt;/a&gt;. This will allow you to put some constraints on the output values (it is basically a logistic function generalization) so that the output values will be modeled as probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, with or without softmax you can use the output as a discriminant function to select the proper category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another final thought would be to avoid to encode you variables in a connected way. For example you can have the binary representation of the category index. This would induce to the learner an artificial connection between some outputs which are arbitrary. The one hot encoding has the advantage that is neutral to how labels are indexed.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-05-22T19:20:14.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="195" PostTypeId="2" ParentId="192" CreationDate="2014-05-22T20:48:55.297" Score="5" Body="&lt;p&gt;Satisfaction is a huge one that I run into a lot.  Huge referring to importance/difficulty/complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom line is that for very large services (search engines, facebook, linkedin, etc...) your users are simply a collection of log lines.  You have little ability to solicit feed back from them (not a hard and fast rule necessarily).  So you have to infer their positive or negative feedback most of the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means finding ways, even outside of predictive modelling, to truly tell, from a collection of log lines, whether or not someone actually liked something they experienced.  This simple act is even more fundamental (in my biased opinion) than a/b testing since you're talking about metrics you will eventually track on a test scorecard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a handle on good SAT metrics then you can start making predictive models and experimenting.  But even deciding what piece of log instrumentation can tell you about SAT is non-trivial (and often changes).&lt;/p&gt;&#xA;" OwnerUserId="92" LastEditorUserId="92" LastEditDate="2014-05-23T06:03:40.577" LastActivityDate="2014-05-23T06:03:40.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="196" PostTypeId="1" AcceptedAnswerId="197" CreationDate="2014-05-22T21:47:26.980" Score="13" ViewCount="7447" Body="&lt;p&gt;So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a &quot;bucket&quot; that they end up in. However, rather than create models of probabilities like in Naive Bayes or similar classifiers, we want our output to be a set of roughly human-readable rules that can be reviewed and modified by an end user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Association rule learning looks like the family of algorithms that solves this type of problem, but these algorithms seem to focus on identifying common combinations of features and don't include the concept of a final bucket that those features might point to. For example, our data set looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item A { 4-door, small, steel } =&amp;gt; { sedan }&#xA;Item B { 2-door, big,   steel } =&amp;gt; { truck }&#xA;Item C { 2-door, small, steel } =&amp;gt; { coupe }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I just want the rules that say &quot;if it's big and a 2-door, it's a truck,&quot; not the rules that say &quot;if it's a 4-door it's also small.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One workaround I can think of is to simply use association rule learning algorithms and ignore the rules that don't involve an end bucket, but that seems a bit hacky. Have I missed some family of algorithms out there? Or perhaps I'm approaching the problem incorrectly to begin with?&lt;/p&gt;&#xA;" OwnerUserId="275" LastEditorUserId="84" LastEditDate="2014-05-23T03:27:20.630" LastActivityDate="2020-08-06T11:04:09.857" Title="Algorithm for generating classification rules" Tags="|machine-learning|classification|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="197" PostTypeId="2" ParentId="196" CreationDate="2014-05-22T21:54:05.660" Score="10" Body="&lt;p&gt;C45 made by Quinlan is able to produce rule for prediction. Check this &lt;a href=&quot;http://en.wikipedia.org/wiki/C4.5_algorithm&quot;&gt;Wikipedia&lt;/a&gt; page. I know that in &lt;a href=&quot;http://www.cs.waikato.ac.nz/~ml/weka/&quot;&gt;Weka&lt;/a&gt; its name is J48. I have no idea which are implementations in R or Python. Anyway, from this kind of decision tree  you should be able to infer rules for prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Later edit&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you might be interested in algorithms for directly inferring rules for classification. RIPPER is one, which again in Weka it received a different name JRip. See the original paper for RIPPER: &lt;a href=&quot;http://www.google.ro/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CCYQFjAA&amp;amp;url=http://www.cs.utsa.edu/~bylander/cs6243/cohen95ripper.pdf&amp;amp;ei=-XJ-U-7pGoqtyAOej4Ag&amp;amp;usg=AFQjCNFqLnuJWi3gGXVCrugmv3NTRhHHLA&amp;amp;bvm=bv.67229260,d.bGQ&amp;amp;cad=rja&quot;&gt;Fast Effective Rule Induction, W.W. Cohen 1995&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2014-05-22T21:59:22.117" LastActivityDate="2014-05-22T21:59:22.117" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="198" PostTypeId="2" ParentId="192" CreationDate="2014-05-23T03:05:57.990" Score="5" Body="&lt;p&gt;Also, there seem to be a very comprehensive list of data science use cases by function and by vertical on Kaggle - &lt;a href=&quot;http://www.kaggle.com/wiki/DataScienceUseCases&quot;&gt;&quot;Data Science Use Cases&quot;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2014-05-23T03:05:57.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="199" PostTypeId="1" AcceptedAnswerId="202" CreationDate="2014-05-23T06:25:50.480" Score="25" ViewCount="35289" Body="&lt;p&gt;LDA has two hyperparameters, tuning them changes the induced topics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does the alpha and beta hyperparameters contribute to LDA? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the topic change if one or the other hyperparameters increase or decrease? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are they hyperparamters and not just parameters?&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2022-05-20T16:20:33.417" Title="What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?" Tags="|topic-model|lda|parameter|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="200" PostTypeId="2" ParentId="134" CreationDate="2014-05-23T08:34:38.900" Score="4" Body="&lt;p&gt;You also can create a MongoDB-Hadoop &lt;a href=&quot;http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/&quot; rel=&quot;nofollow&quot;&gt;connection&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="278" LastActivityDate="2014-05-23T08:34:38.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="201" PostTypeId="2" ParentId="155" CreationDate="2014-05-23T09:09:44.490" Score="77" Body="&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Kaggle.com&lt;/strong&gt;, a home of modern data science &amp;amp; machine learning enthusiasts:), opened &lt;strong&gt;&lt;a href=&quot;https://www.kaggle.com/datasets&quot; rel=&quot;noreferrer&quot;&gt;it's own repository of the data sets&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In addition to the listed sources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some social network data sets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://snap.stanford.edu/data/&quot; rel=&quot;noreferrer&quot;&gt;Stanford University large network dataset collection (SNAP)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.infochimps.com/2008/12/29/massive-scrape-of-twitters-friend-graph/&quot; rel=&quot;noreferrer&quot;&gt;A huge twitter dataset that includes followers&lt;/a&gt; + &lt;a href=&quot;http://www.infochimps.com/collections/twitter-census&quot; rel=&quot;noreferrer&quot;&gt;large collection of twitter datasets here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mtg.upf.edu/node/1671&quot; rel=&quot;noreferrer&quot;&gt;LastFM data set&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are plenty of sources listed at Stats SE:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/7/locating-freely-available-data-samples/&quot;&gt;Locating freely available data samples&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/12670/data-apis-feeds-available-as-packages-in-r&quot;&gt;Data APIs/feeds available as packages in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/973/free-data-set-for-very-high-dimensional-classification&quot;&gt;Free data set for very high dimensional classification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-01-21T08:44:01.610" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="202" PostTypeId="2" ParentId="199" CreationDate="2014-05-23T13:47:54.603" Score="23" Body="&lt;p&gt;The Dirichlet distribution is a multivariate distribution. We can denote the parameters of the Dirichlet as a vector of size K of the form ~&lt;span class=&quot;math-container&quot;&gt;$\frac{1}{B(a)} \cdot \prod\limits_{i} x_i^{a_{i-1}}$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; is the vector of size &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt; of the parameters, and &lt;span class=&quot;math-container&quot;&gt;$\sum x_i = 1$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Now the LDA uses some constructs like:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a document can have multiple topics (because of this multiplicity, we need the Dirichlet distribution); and there is a Dirichlet distribution which models this relation&lt;/li&gt;&#xA;&lt;li&gt;words can also belong to multiple topics, when you consider them outside of a document; so here we need another Dirichlet to model this&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The previous two are distributions which you do not really see from data, this is why is called latent, or hidden.&lt;/p&gt;&#xA;&lt;p&gt;Now, in Bayesian inference you use the Bayes rule to infer the posterior probability. For simplicity, let's say you have data &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and you have a model for this data governed by some parameters &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;. In order to infer values for this parameters, in full Bayesian inference you will infer the posterior probability of these parameters using Bayes' rule with &lt;span class=&quot;math-container&quot;&gt;$$p(\theta|x) = \frac{p(x|\theta)p(\theta|\alpha)}{p(x|\alpha)} \iff \text{posterior probability} = \frac{\text{likelihood}\times \text{prior probability}}{\text{marginal likelihood}}$$&lt;/span&gt; Note that here comes an &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt;. This is your initial belief about this distribution, and is the parameter of the prior distribution. Usually this is chosen in such a way that will have a conjugate prior (so the distribution of the posterior is the same with the distribution of the prior) and often to encode some knowledge if you have one or to have maximum entropy if you know nothing.&lt;/p&gt;&#xA;&lt;p&gt;The parameters of the prior are called &lt;em&gt;hyperparameters&lt;/em&gt;. So, in LDA, both topic distributions, over documents and over words have also correspondent priors, which are denoted usually with alpha and beta, and because are the parameters of the prior distributions are called hyperparameters.&lt;/p&gt;&#xA;&lt;p&gt;Now about choosing priors. If you plot some Dirichlet distributions you will note that if the individual parameters &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; have the same value, the pdf is symmetric in the simplex defined by the &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; values, which is the minimum or maximum for pdf is at the center.&lt;/p&gt;&#xA;&lt;p&gt;If all the &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; have values lower than unit the maximum is found at corners&lt;/p&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/5khZE.png&quot; width=&quot;200&quot; height=&quot;200&quot;&gt;&#xA;&lt;p&gt;or can if all values &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; are the same and greater than 1 the maximum will be found in center like&lt;/p&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/qrfm7.png&quot; width=&quot;200&quot; height=&quot;200&quot;&gt;&#xA;&lt;p&gt;It is easy to see that if values for &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; are not equal the symmetry is broken and the maximum will be found near bigger values.&lt;/p&gt;&#xA;&lt;p&gt;Additional, please note that values for priors parameters produce smooth pdfs of the distribution as the values of the parameters are near 1. So if you have great confidence that something is clearly distributed in a way you know, with a high degree of confidence, than values far from 1 in absolute value are to be used, if you do not have such kind of knowledge than values near 1 would be encode this lack of knowledge. It is easy to see why 1 plays such a role in Dirichlet distribution from the formula of the distribution itself.&lt;/p&gt;&#xA;&lt;p&gt;Another way to understand this is to see that prior encode prior-knowledge. In the same time you might think that prior encode some prior seen data. This data was not seen by the algorithm itself, it was seen by you, you learned something, and you can model prior according to what you know (learned). So in the prior parameters (hyperparameters) you encode also how big this data set you apriori saw, because the sum of &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; can be that also as the size of this more or less imaginary data set. So the bigger the prior data set, the bigger is the confidence, the bigger the values of &lt;span class=&quot;math-container&quot;&gt;$\alpha_k$&lt;/span&gt; you can choose, the sharper the surface near maximum value, which means also less doubts.&lt;/p&gt;&#xA;&lt;p&gt;Hope it helped.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="-1" LastEditDate="2022-05-20T16:20:33.417" LastActivityDate="2022-05-20T16:20:33.417" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="203" PostTypeId="2" ParentId="189" CreationDate="2014-05-23T14:28:41.563" Score="2" Body="&lt;p&gt;Never done stuff &lt;strong&gt;on that scale&lt;/strong&gt;, but as no-one else has jumped in yet have you seen these two papers that discuss non-commercial solutions?  Symphony and COIN-OR seem to be the dominant suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linderoth, Jeffrey T., and Andrea Lodi. &quot;MILP software.&quot; Wiley encyclopedia of operations research and management science (2010). &lt;a href=&quot;http://homepages.cae.wisc.edu/~Linderot/papers/Linderoth-Lodi-10.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF version&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linderoth, Jeffrey T., and Ted K. Ralphs. &quot;Noncommercial software for mixed-integer linear programming.&quot; Integer programming: theory and practice 3 (2005): 253-303. &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.147.5872&quot; rel=&quot;nofollow&quot;&gt;Compares performance&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="265" LastActivityDate="2014-05-23T14:28:41.563" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="204" PostTypeId="2" ParentId="116" CreationDate="2014-05-23T19:28:01.903" Score="7" Body="&lt;p&gt;I recently did a similar project in Python (predicting opinions using FB like data), and had good results with the following basic process:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Read in the training set (n = N) by iterating over comma-delimited like records line-by-line and use a counter to identify the most popular pages&lt;/li&gt;&#xA;&lt;li&gt;For each of the K most popular pages (I used about 5000, but you can play around with different values), use pandas.DataFrame.isin to test whether each individual in the training set likes each page, then make a N x K dataframe of the results (I'll call it xdata_train)&lt;/li&gt;&#xA;&lt;li&gt;Create a series (I'll call it ydata_train) containing all of the outcome variables (in my case opinions, in yours age) with the same index as xdata_train&lt;/li&gt;&#xA;&lt;li&gt;Set up a random forest classifier through scikit-learn to predict&#xA;ydata_train based on xdata_train&lt;/li&gt;&#xA;&lt;li&gt;Use scikit-learn's cross-validation testing to tweak parameters and&#xA;refine accuracy (tweaking number of popular pages, number of trees,&#xA;min leaf size, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Output random forest classifier and list of most popular pages with pickle (or keep in memory if you are doing everything at once)&lt;/li&gt;&#xA;&lt;li&gt;Load in the rest of your data, load the list of popular pages (if necessary), and repeat step 2 to produce xdata_new&lt;/li&gt;&#xA;&lt;li&gt;Load the random forest classifier (if necessary) and use it to predict values for the xdata_new data&lt;/li&gt;&#xA;&lt;li&gt;Output the predicted scores to a new CSV or other output format of your choosing&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;In your case, you'd need to swap out the classifier for a regressor (so see &lt;a href=&quot;https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;sklearn.ensemble.RandomForestRegressor&lt;/code&gt;&lt;/a&gt;) but otherwise the same process should work without much trouble.&lt;/p&gt;&#xA;&lt;p&gt;Also, you should be aware of the most amazing feature of random forests in Python: instant parallelization! Those of us who started out doing this in R and then moved over are always amazed, especially when you get to work on a machine with a few dozen cores.&lt;/p&gt;&#xA;&lt;p&gt;Finally, note that this would be a perfect application for network analysis if you have the data on friends as well as the individuals themselves. If you can analyze the ages of a user's friends, the age of the user will almost certainly be within a year or two of the median among his or her friends, particularly if the users are young enough to have built their friend networks while still in school (since most will be classmates). That prediction would likely trump any you would get from modeling---this is a textbook example of a problem where the right data &amp;gt; the right model every time.&lt;/p&gt;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="250" LastEditorUserId="98307" LastEditDate="2020-08-20T18:51:34.423" LastActivityDate="2020-08-20T18:51:34.423" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="205" PostTypeId="1" AcceptedAnswerId="208" CreationDate="2014-05-23T19:45:54.283" Score="12" ViewCount="1869" Body="&lt;p&gt;Working on what could often be called &quot;medium data&quot; projects, I've been able to parallelize my code (mostly for modeling and prediction in Python) on a single system across anywhere from 4 to 32 cores. Now I'm looking at scaling up to clusters on EC2 (probably with StarCluster/IPython, but open to other suggestions as well), and have been puzzled by how to reconcile distributing work across cores on an instance vs. instances on a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it even practical to parallelize across instances as well as across cores on each instance? If so, can anyone give a quick rundown of the pros + cons of running many instances with few cores each vs. a few instances with many cores? Is there a rule of thumb for choosing the right ratio of instances to cores per instance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bandwidth and RAM are non-trivial concerns in my projects, but it's easy to spot when those are the bottlenecks and readjust. It's much harder, I'd imagine, to benchmark the right mix of cores to instances without repeated testing, and my projects vary too much for any single test to apply to all circumstances. Thanks in advance, and if I've just failed to google this one properly, feel free to point me to the right answer somewhere else!&lt;/p&gt;&#xA;" OwnerUserId="250" LastActivityDate="2017-02-19T09:12:49.270" Title="Instances vs. cores when using EC2" Tags="|parallel|clustering|aws|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="206" PostTypeId="2" ParentId="205" CreationDate="2014-05-23T21:01:18.630" Score="6" Body="&lt;p&gt;All things considered equal (cost, CPU perf, etc.) you could choose the smallest instance that can hold all of my dataset in memory and scale out. That way &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you make sure not to induce unnecessary latencies due to network communications, and&lt;/li&gt;&#xA;&lt;li&gt;you tend to maximize the overall available memory bandwidth for your processes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming you are running some sort of &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;noreferrer&quot;&gt;cross-validation scheme&lt;/a&gt; to optimize some &lt;a href=&quot;http://en.wikipedia.org/wiki/Meta-optimization&quot; rel=&quot;noreferrer&quot;&gt;meta parameter&lt;/a&gt; of your model, assign each core a value to test and choose an many instances as needed to cover all the parameter space in as few rounds as you see fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data does not fit in the memory of one system, of course you'll need to distribute across instances. Then it is a matter of balancing memory latency (better with many instances) with network latency (better with fewer instances) but given the nature of EC2 I'd bet you'll often prefer to work with few fat instances.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-05-23T21:01:18.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="207" PostTypeId="2" ParentId="205" CreationDate="2014-05-24T10:36:58.987" Score="11" Body="&lt;p&gt;A general rule of thumb is to not distribute until you have to. It's usually more efficient to have N servers of a certain capacity than 2N servers of half that capacity. More of the data access will be local, and therefore fast in memory versus slow across the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a certain point, scaling up one machine becomes uneconomical because the cost of additional resource scales more than linearly. However this point is still amazingly high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Amazon in particular though, the economics of each instance type can vary a lot if you are using spot market instances. The default pricing more or less means that the same amount of resource costs about the same regardless of the instance type, that can vary a lot; large instances can be cheaper than small ones, or N small instances can be much cheaper than one large machine with equivalent resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One massive consideration here is that the computation paradigm can change quite a lot when you move from one machine to multiple machines. The tradeoffs that the communication overhead induce may force you to, for example, adopt a data-parallel paradigm to scale. That means a different choice of tools and algorithm. For example, SGD looks quite different in-memory and in Python than on MapReduce. So you would have to consider this before parallelizing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may choose to distribute work across a cluster, even if a single node and non-distributed paradigms work for you, for reliability. If a single node fails, you lose all of the computation; a distributed computation can potentially recover and complete just the part of the computation that was lost.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-24T10:36:58.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="208" PostTypeId="2" ParentId="205" CreationDate="2014-05-24T11:18:26.497" Score="11" Body="&lt;p&gt;When using IPython, you very nearly don't have to worry about it (at the expense of some loss of efficiency/greater communication overhead).  The parallel IPython plugin in StarCluster will by default start one engine per physical core on each node (I believe this is configurable but not sure where).  You just run whatever you want across all engines by using the DirectView api (map_sync, apply_sync, ...) or the %px magic commands.   If you are already using IPython in parallel on one machine, using it on a cluster is no different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addressing some of your specific questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;how to reconcile distributing work across cores on an instance vs. instances on a cluster&quot; - You get one engine per core (at least); work is automatically distributed across all cores and across all instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Is it even practical to parallelize across instances as well as across cores on each instance?&quot; - Yes :)  If the code you are running is embarrassingly parallel (exact same algo on multiple data sets) then you can mostly ignore where a particular engine is running.  If the core requires a lot of communication between engines, then of course you need to structure it so that engines primarily communicate with other engines on the same physical machine; but that kind of problem is not ideally suited for IPython, I think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;If so, can anyone give a quick rundown of the pros + cons of running many instances with few cores each vs. a few instances with many cores? Is there a rule of thumb for choosing the right ratio of instances to cores per instance?&quot; - Use the largest c3 instances for compute-bound, and the smallest for memory-bandwidth-bound problems; for message-passing-bound problems, also use the largest instances but try to partition the problem so that each partition runs on one physical machine and most message passing is within the same partition.  Problems which would run significantly slower on N quadruple c3 instances than on 2N double c3 are rare (an artificial example may be running multiple simple filters on a large number of images, where you go through all images for each filter rather than all filters for the same image).  Using largest instances is a good rule of thumb.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2017-02-19T09:12:49.270" LastActivityDate="2017-02-19T09:12:49.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="209" PostTypeId="1" AcceptedAnswerId="212" CreationDate="2014-05-25T13:57:52.657" Score="10" ViewCount="1455" Body="&lt;p&gt;A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_id item_id result&#xA;1       4       1&#xA;1       7       -1&#xA;5       19      1&#xA;5       80      1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 1 means the user accepted the recommendation while -1 means the user did not respond to the recommendation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; If I am going to make recommendations to a bunch of users based on the kind of log described above, and I want to maximize MAP@3 scores, how should I deal with the implicit data (1 or -1)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is to treat 1 and -1 as ratings, and predict the rating using factorization machines-type algorithms. But this does not seem right, given the asymmetry of the implicit data (-1 does not mean the user does not like the recommendation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1&lt;/strong&gt;&#xA;Let us think about it in the context of a matrix factorization approach. If we treat -1 and 1 as ratings, there will be some problem. For example, user 1 likes movie A which scores high in one factor (e.g. having glorious background music) in the latent factor space. The system recommends movie B which also scores high in &quot;glorious background music&quot;, but for some reason user 1 is too busy to look into the recommendation, and we have a -1 rating movie B. If we just treat 1 or -1 equally, then the system might be discouraged to recommend movie with glorious BGM to user 1 while user 1 still loves movie with glorious BGM. I think this situation is to be avoided.&lt;/p&gt;&#xA;" OwnerUserId="71" LastEditorUserId="71" LastEditDate="2014-05-26T05:12:32.653" LastActivityDate="2014-05-27T10:58:00.620" Title="How should one deal with implicit data in recommendation" Tags="|recommender-system|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="210" PostTypeId="2" ParentId="199" CreationDate="2014-05-26T04:07:32.390" Score="13" Body="&lt;p&gt;Assuming symmetric Dirichlet distributions (for simplicity), a low alpha value places more weight on having each document composed of only a few dominant topics (whereas a high value will return many more relatively dominant topics). Similarly, a low beta value places more weight on having each topic composed of only a few dominant words.&lt;/p&gt;&#xA;" OwnerUserId="283" LastActivityDate="2014-05-26T04:07:32.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="211" PostTypeId="1" AcceptedAnswerId="213" CreationDate="2014-05-27T10:41:33.220" Score="9" ViewCount="4614" Body="&lt;p&gt;I'm new to this community and hopefully my question will well fit in here.&#xA;As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using smartphone data sets. As far as I'm concern this topic relates to Machine Learning and Support Vector Machines. I'm not well familiar with this technologies yet so I will need some help.&lt;/p&gt;&#xA;&lt;p&gt;I have decided to follow &lt;a href=&quot;http://www.inf.ed.ac.uk/teaching/courses/dme/2014/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this project idea&lt;/a&gt; (first project on the top)&#xA;The project goal is determine what activity a person is engaging in (e.g., WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) from data recorded by a smartphone (Samsung Galaxy S II) on the subject's waist. Using its embedded accelerometer and gyroscope, the data includes 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz.&lt;/p&gt;&#xA;&lt;p&gt;All the data set is given in one folder with some description and feature labels. The data is divided for 'test' and 'train' files in which data is represented in this format:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;  2.5717778e-001 -2.3285230e-002 -1.4653762e-002 -9.3840400e-001 -9.2009078e-001 -6.6768331e-001 -9.5250112e-001 -9.2524867e-001 -6.7430222e-001 -8.9408755e-001 -5.5457721e-001 -4.6622295e-001  7.1720847e-001  6.3550240e-001  7.8949666e-001 -8.7776423e-001 -9.9776606e-001 -9.9841381e-001 -9.3434525e-001 -9.7566897e-001 -9.4982365e-001 -8.3047780e-001 -1.6808416e-001 -3.7899553e-001  2.4621698e-001  5.2120364e-001 -4.8779311e-001  4.8228047e-001 -4.5462113e-002  2.1195505e-001 -1.3489443e-001  1.3085848e-001 -1.4176313e-002 -1.0597085e-001  7.3544013e-002 -1.7151642e-001  4.0062978e-002  7.6988933e-002 -4.9054573e-001 -7.0900265e-001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And that's only a very small sample of what the file contain.&lt;/p&gt;&#xA;&lt;p&gt;I don't really know what this data represents and how can be interpreted. Also for analyzing, classification and clustering of the data, what tools will I need to use?&#xA;Is there any way I can put this data into excel with labels included and for example use R or python to extract sample data and work on this?&lt;/p&gt;&#xA;&lt;p&gt;Any hints/tips would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="295" LastEditorUserId="98307" LastEditDate="2020-08-16T21:51:47.670" LastActivityDate="2020-08-17T03:25:03.437" Title="Human activity recognition using smartphone data set problem" Tags="|bigdata|machine-learning|databases|clustering|data-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="212" PostTypeId="2" ParentId="209" CreationDate="2014-05-27T10:58:00.620" Score="7" Body="&lt;p&gt;Your system isn't just trained on items that are recommended right? if so you have a big feedback loop here. You want to learn from all clicks/views, I hope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You suggest that not-looking at an item is a negative signal. I strongly suggest you do not treat it that way. Not interacting with something is almost always best treated as no information. If you have an explicit signal that indicates a dislike, like a down vote (or, maybe watched 10 seconds of a video and stopped), maybe that's valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would not construe this input as rating-like data. (Although in your case, you may get away with it.) Instead think of them as weights, which is exactly the treatment in the Hu Koren Volinsky paper on ALS that @Trey mentions in a comment. This lets you record relative strength of positive/negative interactions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally I would note that this paper, while is very likely to be what you're looking for, does not provide for negative weights. It is simple to extend in this way. If you get that far I can point you to the easy extension, which exists already in two implementations that I know of, in &lt;a href=&quot;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala&quot; rel=&quot;noreferrer&quot;&gt;Spark&lt;/a&gt; and &lt;a href=&quot;https://github.com/cloudera/oryx/blob/master/als-common/src/main/java/com/cloudera/oryx/als/common/factorizer/als/AlternatingLeastSquares.java&quot; rel=&quot;noreferrer&quot;&gt;Oryx&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-05-27T10:58:00.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="213" PostTypeId="2" ParentId="211" CreationDate="2014-05-27T12:07:45.920" Score="8" Body="&lt;p&gt;The data set definitions are on the page here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones#&quot;&gt;Attribute Information at the bottom&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or you can see inside the ZIP folder the file named activity_labels, that has your column headings inside of it, make sure you read the README carefully, it has some good info in it. You can easily bring in a &lt;code&gt;.csv&lt;/code&gt; file in R using the &lt;code&gt;read.csv&lt;/code&gt; command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if you name you file &lt;code&gt;samsungdata&lt;/code&gt; you can open R and run this command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv(&quot;directory/where/file/is/located/samsungdata.csv&quot;, header = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or if you are already inside of the working directory in R you can just run the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv(&quot;samsungdata.csv&quot;, header = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where the name &lt;code&gt;data&lt;/code&gt; can be changed to whatever you want to call your data set.&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2014-05-27T12:07:45.920" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="214" PostTypeId="2" ParentId="155" CreationDate="2014-05-27T16:05:02.883" Score="38" Body="&lt;p&gt;The following links are available&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.opensciencedatacloud.org/publicdata/&quot; rel=&quot;noreferrer&quot;&gt;Public Data Sets&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.google.com/publicdata/directory&quot; rel=&quot;noreferrer&quot;&gt;Google Public Data Sets&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://aws.amazon.com/publicdatasets/&quot; rel=&quot;noreferrer&quot;&gt;Amazon Web Services&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.inside-r.org/howto/finding-data-internet&quot; rel=&quot;noreferrer&quot;&gt;Finding Data on the Internet&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="295" LastEditorUserId="28175" LastEditDate="2017-12-17T19:30:24.190" LastActivityDate="2017-12-17T19:30:24.190" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="215" PostTypeId="1" CreationDate="2014-05-27T21:07:48.973" Score="16" ViewCount="1063" Body="&lt;p&gt;I'm building a workflow for creating machine learning models (in my case, using Python's &lt;code&gt;pandas&lt;/code&gt; and &lt;code&gt;sklearn&lt;/code&gt; packages) from data pulled from a very large database (here, Vertica by way of SQL and &lt;code&gt;pyodbc&lt;/code&gt;), and a critical step in that process involves imputing missing values of the predictors. This is straightforward within a single analytics or stats platform---be it Python, R, Stata, etc.---but I'm curious where best to locate this step in a multi-platform workflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's simple enough to do this in Python, either with the &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#imputation-of-missing-values&quot;&gt;&lt;code&gt;sklearn.preprocessing.Imputer&lt;/code&gt;&lt;/a&gt; class, using the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.DataFrame.fillna.html&quot;&gt;&lt;code&gt;pandas.DataFrame.fillna&lt;/code&gt;&lt;/a&gt; method, or by hand (depending upon the complexity of the imputation method used). But since I'm going to be using this for dozens or hundreds of columns across hundreds of millions of records, I wonder if there's a more efficient way to do this directly through SQL ahead of time. Aside from the potential efficiencies of doing this in a distributed platform like Vertica, this would have the added benefit of allowing us to create an automated pipeline for building &quot;complete&quot; versions of tables, so we don't need to fill in a new set of missing values from scratch every time we want to run a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't been able to find much guidance about this, but I imagine that we could:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;create a table of substitute values (e.g., mean/median/mode, either overall or by group) for each incomplete column&lt;/li&gt;&#xA;&lt;li&gt;join the substitute value table with the original table to assign a substitute value for each row and incomplete column&lt;/li&gt;&#xA;&lt;li&gt;use a series of case statements to take the original value if available and the substitute value otherwise&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is this a reasonable thing to do in Vertica/SQL, or is there a good reason not to bother and just handle it in Python instead? And if the latter, is there a strong case for doing this in pandas rather than sklearn or vice-versa? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="250" LastActivityDate="2014-06-11T11:32:13.433" Title="Where in the workflow should we deal with missing data?" Tags="|machine-learning|python|pandas|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="216" PostTypeId="2" ParentId="215" CreationDate="2014-05-28T07:08:05.393" Score="15" Body="&lt;p&gt;My strong opinion regarding automated tasks like imputation (but, here I can include also scaling, centering, feature selection, etc) is to avoid in any way do such things without carefully inspecting your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, after deciding what kind of imputation to apply it can be automated (under the assumption that the new data has the same shape/problems).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, before anything, take a wise decision. I often wasted time trying to automate this things, destroying my data. I will give you some examples:&#xA;- a marketplace encoded as N/A, which I missed and considered to be North/America&#xA;- numbers like -999.0, because the data producer could not find a better replacement for missing data&#xA;- number like 0 for blood pressure or body temperature, instead of missing data (it is hard to imagine a living human with 0 blood pressure)&#xA;- multiple placeholders for missing data, due to the fact that the data was collected from various sources&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that you need to understand what kind of imputation would resemble better the information from your data for a given task. This is often much harder to do it right than it seems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After all those things, my advice is to delay your imputation task to an upper layer where you have tools to reproduce on new data and to inspect if the assumptions for the new data are not violated (if it is possible).&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-05-28T07:08:05.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="217" PostTypeId="2" ParentId="211" CreationDate="2014-05-28T09:43:54.197" Score="5" Body="&lt;p&gt;It looks like this (or very similar data set) is used for Coursera courses. Cleaning this dataset is task for &lt;a href=&quot;https://www.coursera.org/course/getdata&quot; rel=&quot;noreferrer&quot;&gt;Getting and Cleaning Data&lt;/a&gt;, but it is also used for case study for &lt;a href=&quot;https://class.coursera.org/exdata-002&quot; rel=&quot;noreferrer&quot;&gt;Exploratory Data analysis&lt;/a&gt;. Video from this case study is available in videos for week 4 of EDA course-ware. It might help you with starting with this data.&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2014-05-28T09:43:54.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="218" PostTypeId="1" CreationDate="2014-05-29T13:08:09.060" Score="3" ViewCount="1742" Body="&lt;p&gt;So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully &#xA;saved in DataFrame but when I try to find cov() it result an error.&#xA;here is the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  import pandas as pd&#xA;  cov_data=pd.DataFrame(dataset).cov()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;File &quot;/home/syahdeini/Desktop/FP/pca_2.py&quot;, line 44, in find_eagen&#xA;cov_data=pd.DataFrame(data_mat).cov()&#xA;File &quot;/usr/lib/python2.7/dist-packages/pandas/core/frame.py&quot;, line 3716, in cov&#xA;baseCov = np.cov(mat.T)&#xA;File &quot;/usr/lib/python2.7/dist-packages/numpy/lib/function_base.py&quot;, line 1766, in cov&#xA;return (dot(X, X.T.conj()) / fact).squeeze()&#xA;ValueError: array is too big.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="273" LastActivityDate="2014-05-29T14:31:51.913" Title="built-in cov in pandas DataFrame results ValueError array is too big" Tags="|python|pandas|" AnswerCount="2" CommentCount="1" ClosedDate="2014-08-08T12:26:12.553" ContentLicense="CC BY-SA 3.0" />
  <row Id="219" PostTypeId="2" ParentId="218" CreationDate="2014-05-29T13:51:48.820" Score="2" Body="&lt;p&gt;Since you have 39,949 variables, the covariance matrix would have about 1.6 billion elements (39,949 * 39,949 = 1,595,922,601).  That is likely why you are getting that error.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-05-29T13:51:48.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="220" PostTypeId="2" ParentId="218" CreationDate="2014-05-29T14:13:00.317" Score="6" Body="&lt;p&gt;Christopher is right about the size of the array. To be simplistic about it, if this translates to 1.6B floats, at 16 bytes per float (32-bit version; 64-bit is bigger), then you're trying to create an array of about 26 GB. Even if you have the RAM for that, I'd imagine that it's probably going to overload something else along the way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Maybe not, but generally speaking, any operations that are that computationally intensive should first raise the question of whether you are doing the right calculation in the first place. And if you do need to do something of that magnitude, you should then try to break it down into more manageable chunks that can be run in parallel or distributed across machines.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But given that you are describing a very, very wide dataset (~40k columns x 180 rows), I wonder whether you really want to take the covariance matrix of the transposed dataset (so 180x180 = 32,400 covariances)? That would be a far more tractable problem, and it's easier to see how it might be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, you're probably far better off calculating each pairwise covariance (or at least, the vector of cov(x_i,x_k) for all x_k != x_i) at the point where you'll actually use it, rather than calculating a giant matrix initially then referring back to it later. Memory issues aside, it'll make your life much easier if you start running things in parallel, and will help ensure you don't waste resources on unnecessary calculations.&lt;/p&gt;&#xA;" OwnerUserId="250" LastEditorUserId="250" LastEditDate="2014-05-29T14:31:51.913" LastActivityDate="2014-05-29T14:31:51.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="221" PostTypeId="2" ParentId="196" CreationDate="2014-05-29T14:30:21.357" Score="8" Body="&lt;p&gt;It's actually even simpler than that, from what you describe---you're just looking for a basic classification tree algorithm (so no need for slightly more complex variants like C4.5 which are optimized for prediction accuracy). The canonical text is &lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/0412048418&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This is readily implemented in &lt;a href=&quot;http://cran.r-project.org/web/packages/tree/tree.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="250" LastEditorUserId="98307" LastEditDate="2020-08-06T11:03:53.690" LastActivityDate="2020-08-06T11:03:53.690" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="222" PostTypeId="2" ParentId="155" CreationDate="2014-05-29T19:02:13.210" Score="23" Body="&lt;p&gt;&lt;a href=&quot;http://enigma.io&quot;&gt;Enigma&lt;/a&gt; is a repository of public available datasets. Its free plan offers public data search, with 10k API calls per month. Not all public databases are listed, but the list is enough for common cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used it for academic research and it saved me a lot of time.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Another interesting source of data is the &lt;a href=&quot;http://theunitedstates.io/&quot;&gt;@unitedstates project&lt;/a&gt;, containing data and tools to collect them, about the United States (members of Congress, geographic shapes…).&lt;/p&gt;&#xA;" OwnerUserId="43" LastEditorUserId="43" LastEditDate="2014-05-30T21:05:52.147" LastActivityDate="2014-05-30T21:05:52.147" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="223" PostTypeId="1" CreationDate="2014-05-29T20:11:16.327" Score="23" ViewCount="3762" Body="&lt;p&gt;Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I saw the company's manager last day.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To be able to extract information from it, it must be annotated with additional data to be less ambiguous. The process of finding such meta-data is not in question, so assume it is done manually. The question is how to store these data in a way that further analysis on it can be done more conveniently/efficiently?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A possible approach is to use XML tags (see below), but it seems too verbose, and maybe there are better approaches/guidelines for storing such meta-data on text documents.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;Person name=&quot;John&quot;&amp;gt;I&amp;lt;/Person&amp;gt; saw the &amp;lt;Organization name=&quot;ACME&quot;&amp;gt;company&amp;lt;/Organization&amp;gt;'s&#xA;manager &amp;lt;Time value=&quot;2014-5-29&quot;&amp;gt;last day&amp;lt;/Time&amp;gt;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="227" LastActivityDate="2020-07-02T13:04:00.943" Title="How to annotate text documents with meta-data?" Tags="|nlp|metadata|data-cleaning|text-mining|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="224" PostTypeId="1" CreationDate="2014-05-31T14:28:42.317" Score="4" ViewCount="361" Body="&lt;p&gt;The output of my word alignment file looks as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I wish to say with regard to the initiative of the Portuguese Presidency that we support the spirit and the political intention behind it . In bezug auf die Initiative der portugiesischen Präsidentschaft möchte ich zum Ausdruck bringen , daß wir den Geist und die politische Absicht , die dahinter stehen , unterstützen .   0-0 5-1 5-2 2-3 8-4 7-5 11-6 12-7 1-8 0-9 9-10 3-11 10-12 13-13 13-14 14-15 16-16 17-17 18-18 16-19 20-20 21-21 19-22 19-23 22-24 22-25 23-26 15-27 24-28&#xA;It may not be an ideal initiative in terms of its structure but we accept Mr President-in-Office , that it is rooted in idealism and for that reason we are inclined to support it .    Von der Struktur her ist es vielleicht keine ideale Initiative , aber , Herr amtierender Ratspräsident , wir akzeptieren , daß sie auf Idealismus fußt , und sind deshalb geneigt , sie mitzutragen .   0-0 11-2 8-3 0-4 3-5 1-6 2-7 5-8 6-9 12-11 17-12 15-13 16-14 16-15 17-16 13-17 14-18 17-19 18-20 19-21 21-22 23-23 21-24 26-25 24-26 29-27 27-28 30-29 31-30 33-31 32-32 34-33&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I produce the phrase tables that are used by MOSES from this output?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="25180" LastEditDate="2018-05-06T22:11:50.250" LastActivityDate="2018-05-06T22:11:50.250" Title="How to get phrase tables from word alignments?" Tags="|machine-translation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="226" PostTypeId="2" ParentId="155" CreationDate="2014-05-31T19:59:15.563" Score="16" Body="&lt;p&gt;I've found this link in Data Science Central with a list of free datasets: &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free&quot;&gt;Big data sets available for free&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="290" LastActivityDate="2014-05-31T19:59:15.563" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="227" PostTypeId="1" AcceptedAnswerId="242" CreationDate="2014-06-01T10:25:51.163" Score="12" ViewCount="7191" Body="&lt;p&gt;Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing via MapReduce in a Hadoop Cluster) is a batch processing system, and Storm is a real-time processing system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have worked a bit with Hadoop Eco System, but I haven't worked with Storm. After looking through a lot of presentations and articles, I still haven't been able to find a satisfactory and comprehensive answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The term tradeoff here is not meant to compare to similar things. It is meant to represent the consequences of getting results real-time that are absent from a batch processing system. &lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditDate="2014-06-01T21:59:02.347" LastActivityDate="2014-06-10T23:33:28.443" Title="Tradeoffs between Storm and Hadoop (MapReduce)" Tags="|bigdata|efficiency|apache-hadoop|distributed|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="228" PostTypeId="1" AcceptedAnswerId="249" CreationDate="2014-06-01T12:51:25.040" Score="8" ViewCount="77" Body="&lt;p&gt;Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm results in cascading of error. In order to avoid this cascading of error and accumulation of it, Hadoop cluster is used to batch process the data and discard the Storm results after the same data is processed by Hadoop. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the reasons for generation of this accumulation of error? and why is it not present in Hadoop? Since I have not worked with Storm, I do not know the reasons for it. Is it because Storm uses some approximate algorithm to process the data in order to process them in real time? or is the cause something else?&lt;/p&gt;&#xA;" OwnerUserId="339" LastActivityDate="2014-06-10T03:42:51.637" Title="Cascaded Error in Apache Storm" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="229" PostTypeId="2" ParentId="227" CreationDate="2014-06-01T19:48:41.693" Score="9" Body="&lt;p&gt;This is kind of like asking about the tradeoffs between frying pan and your drawer of silverware. They are not two things you compare, really. You might use them together as part of a larger project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadoop itself is not one thing, but a name for a federation of services, like HDFS, Hive, HBase, MapReduce, etc. Storm is something you use with some of these services, like HDFS or HBase. It is a stream-processing framework. There are others within the extended Hadoop ecosystem, like Spark Streaming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When would you choose a stream-processing framework? when you need to react to new data in near-real-time. If you need this kind of tool, you deploy this kind of tool, too.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-06-01T19:48:41.693" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="230" PostTypeId="2" ParentId="223" CreationDate="2014-06-02T15:03:35.940" Score="7" Body="&lt;p&gt;In general, you don't want to use XML tags to tag documents in this way because tags may overlap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://uima.apache.org/&quot; rel=&quot;noreferrer&quot;&gt;UIMA&lt;/a&gt;, &lt;a href=&quot;http://gate.ac.uk/&quot; rel=&quot;noreferrer&quot;&gt;GATE&lt;/a&gt; and similar NLP frameworks denote the tags separate from the text.  Each tag, such as &lt;code&gt;Person&lt;/code&gt;, &lt;code&gt;ACME&lt;/code&gt;, &lt;code&gt;John&lt;/code&gt; etc. is stored as the position that the tag begins and the position that it ends.  So, for the tag &lt;code&gt;ACME&lt;/code&gt;, it would be stored as starting a position 11 and ending at position 17. &lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-06-02T15:03:35.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="231" PostTypeId="1" AcceptedAnswerId="287" CreationDate="2014-06-05T09:00:27.950" Score="10" ViewCount="7208" Body="&lt;p&gt;I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each classification I computed a confusion matrix. Now I want to aggregate these results in order to get the overall confusion matrix. How can I achieve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May I sum all confusion matrices in order to obtain the overall one?&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="84" LastEditDate="2014-06-05T15:21:40.640" LastActivityDate="2014-06-11T09:39:34.373" Title="How to get an aggregate confusion matrix from n different classifications" Tags="|classification|confusion-matrix|accuracy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="232" PostTypeId="2" ParentId="231" CreationDate="2014-06-06T14:55:18.867" Score="4" Body="&lt;p&gt;There are a few ways to achieve your &quot;master confusion matrix&quot;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Sum all the confusion matrices together:  Like you suggested, summing this results in a confusion matrix.  The problem with this is you can not interpret totals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Average the entries.  This method is the same as number one, but you divide each entry by the number of trials (~400 in your case).  This would be my preferred method because then you can actually translate each category to a (mean) +- (an error measurement) and actually see which categories are the most volatile or stable.  Careful with interpreting this 'error measurement' though.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Report a problem specific measurement of the confusion numbers.  For example, if your numbers have outliers, medians would preferred over means.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are other statistics that are possible to report as well.  You can redo the method to keep track of individual classifications.  Then we can say other important stats like '% of classifications that stay the same and are accurate', etc...&lt;/p&gt;&#xA;" OwnerUserId="375" LastActivityDate="2014-06-06T14:55:18.867" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="233" PostTypeId="2" ParentId="130" CreationDate="2014-06-08T07:03:43.710" Score="3" Body="&lt;p&gt;To complete Damien's answer, an example of dimensionality reduction in NLP is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic model&lt;/a&gt;, where you represent the document by a vector indicating the weights of its constituent topics.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-06-08T07:03:43.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="234" PostTypeId="1" CreationDate="2014-06-09T04:43:03.497" Score="5" ViewCount="471" Body="&lt;p&gt;As Yann LeCun &lt;a href=&quot;http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_yann_lecun/chisdw1&quot; rel=&quot;noreferrer&quot;&gt;mentioned&lt;/a&gt;, a number of PhD programs in data science will be popping up in the next few years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://datascience.nyu.edu/academics/programs/&quot; rel=&quot;noreferrer&quot;&gt;NYU&lt;/a&gt; already have one, where Prof.LeCun is at right now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A statistics or cs PhD in machine learning is probably more rigorous than a data science one.  Is data science PhD for the less mathy people like myself? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these cash cow programs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a huge industry demand for big data, but what is the academic value of these programs, as you probably can't be a professor or publish any paper.&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2014-06-10T03:21:56.473" Title="Data science Ph.D. program, what do you think?" Tags="|knowledge-base|" AnswerCount="5" CommentCount="1" ClosedDate="2019-05-07T10:57:39.763" ContentLicense="CC BY-SA 3.0" />
  <row Id="235" PostTypeId="1" AcceptedAnswerId="237" CreationDate="2014-06-09T08:34:29.337" Score="3" ViewCount="1584" Body="&lt;p&gt;Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a Python API to Tableau?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any Python based data visualization toolkits?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="29169" LastEditDate="2019-06-08T03:11:24.957" LastActivityDate="2019-06-08T03:11:24.957" Title="Are there any python based data visualization toolkits?" Tags="|python|visualization|" AnswerCount="5" CommentCount="1" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 4.0" />
  <row Id="236" PostTypeId="2" ParentId="234" CreationDate="2014-06-09T18:02:00.613" Score="2" Body="&lt;p&gt;No-one knows since no-one's completed one of these PhD programs yet! However, I would look at the syllabus and the teachers to base my decision. It all depends on what you want to do; industry or academia?&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-06-09T18:02:00.613" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="237" PostTypeId="2" ParentId="235" CreationDate="2014-06-09T19:52:41.847" Score="11" Body="&lt;p&gt;There is a Tablaeu API and you can use Python to use it, but maybe not in the sense that you think. There is a Data Extract API that you could use to import your data into Python and do your visualizations there, so I do not know if this is going to answer your question entirely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in the first comment you can use Matplotlib from &lt;a href=&quot;http://www.matplotlib.org&quot;&gt;Matplotlib website&lt;/a&gt;, or you could install Canopy from Enthought which has it available, there is also Pandas, which you could also use for data analysis and some visualizations. There is also a package called &lt;code&gt;ggplot&lt;/code&gt; which is used in &lt;code&gt;R&lt;/code&gt; alot, but is also made for Python, which you can find here &lt;a href=&quot;https://pypi.python.org/pypi/ggplot&quot;&gt;ggplot for python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Tableau data extract API and some information about it can be found &lt;a href=&quot;http://www.tableausoftware.com/new-features/data-engine-api-0&quot;&gt;at this link&lt;/a&gt;. There are a few web sources that I found concerning it using duckduckgo &lt;a href=&quot;https://duckduckgo.com/?q=tableau%20PYTHON%20API&amp;amp;kp=1&amp;amp;kd=-1&quot;&gt;at this link&lt;/a&gt;.&#xA;Here are some samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.interworks.com/blogs/bbickell/2012/12/06/introducing-python-tableau-data-extract-api-csv-extract-example&quot;&gt;Link 1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ryrobes.com/python/building-tableau-data-extract-files-with-python-in-tableau-8-sample-usage/&quot;&gt;Link 2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nbviewer.ipython.org/github/Btibert3/tableau-r/blob/master/Python-R-Tableau-Predictive-Modeling.ipynb&quot;&gt;Link 3&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as an API like matplotlib, I cannot say for certain that one exists. Hopefully this gives some sort of reference to help answer your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also to help avoid closure flags and downvotes you should try and show some of what you have tried to do or find, this makes for a better question and helps to illicit responses.&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2014-06-09T19:52:41.847" CommentCount="0" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="238" PostTypeId="2" ParentId="234" CreationDate="2014-06-09T21:36:44.297" Score="1" Body="&lt;p&gt;I think this question assumes a false premise. As a student at NYU, I only know of a Masters in Data Science. You linked to a page that confirms this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's hard to gauge the benefit of a program that doesn't exist yet.&lt;/p&gt;&#xA;" OwnerUserId="395" LastActivityDate="2014-06-09T21:36:44.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="241" PostTypeId="2" ParentId="234" CreationDate="2014-06-09T21:51:53.793" Score="3" Body="&lt;p&gt;It seems to me that the premise of a PhD is to expand knowledge in some little slice of the world. Since a &quot;data scientist&quot; is by nature is somewhat of a jack-of-all-trades it does seem a little odd to me. A masters program seems much more appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you hope to gain from a PhD? If the rigor scares (or bores) you, then what about a more applied area? Signal processing, robotics, applied physics, operations research, etc.&lt;/p&gt;&#xA;" OwnerUserId="403" LastActivityDate="2014-06-09T21:51:53.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="242" PostTypeId="2" ParentId="227" CreationDate="2014-06-09T21:57:30.240" Score="13" Body="&lt;p&gt;&lt;strong&gt;MapReduce&lt;/strong&gt;: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A MapReduce job takes a file (or some data store) as an input and writes a file of results. If you want these results available to an application, it is your responsibility to put this data in a place that is accessible. This is likely slow, and there will be a lag between the values you can display, and the values that represent your system in its current state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important distinction to make when considering using MapReduce in building realtime systems is that of training your model, and applying your model. If you think your model parameters do not change quickly, you can fit them with MapReduce, and then have a mechanism for accessing these pre-fit parameters when you want to apply your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Storm&lt;/strong&gt;: A real-time, streaming computational system. Storm is online framework, meaning, in this sense, a service that interacts with a running application. In contrast to MapReduce, it receives small pieces of data (not a whole file) as they are processed in your application. You define a DAG of operations to perform on the data. A common and simple use case for Storm is tracking counters, and using that information to populate a real-time dashboard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Storm doesn't have anything (necessarily) to do with persisting your data. Here, streaming is another way to say keeping the information you care about and throwing the rest away. In reality, you probably have a persistence layer in your application that has already recorded the data, and so this a good and justified separation of concerns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you want to know more...&lt;/strong&gt;&#xA;If you would like to learn more about real-time systems that that fit parameters with MR and apply the models a different way &lt;a href=&quot;http://www.slideshare.net/cloudera/hbasecon-2013-24063525&quot;&gt;here are slides for a talk I gave on building real-time recommendation engines on HBase.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An excellent paper that marries real-time counting and persistence in an interesting way is &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1242610&quot;&gt;Google News Personalization: Scalable Online Collaborative Filtering&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another interesting marriage of MR and Storm is &lt;a href=&quot;https://github.com/twitter/summingbird/wiki&quot;&gt;SummingBird&lt;/a&gt;. Summingbird allows you to define data analysis operations that can be applied via Storm or MR.&lt;/p&gt;&#xA;" OwnerUserId="406" LastEditorUserId="406" LastEditDate="2014-06-10T23:33:28.443" LastActivityDate="2014-06-10T23:33:28.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="243" PostTypeId="2" ParentId="184" CreationDate="2014-06-10T01:36:27.520" Score="5" Body="&lt;p&gt;A variety of methods are available to the user.   The support documentation gives walkthroughs and tips for when one or another model is most appropriate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://developers.google.com/prediction/docs/pmml-schema&quot; rel=&quot;noreferrer&quot;&gt;This page&lt;/a&gt; shows the following learning methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;AssociationModel&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;ClusteringModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;GeneralRegressionModel&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;MiningModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;NaiveBayesModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;NeuralNetwork&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;RegressionModel&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;RuleSetModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;SequenceModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;SupportVectorMachineModel&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;TextModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;TimeSeriesModel&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;TreeModel&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;EDIT: I don't see any specific information about the algorithms, though.  For example, does the tree model use information gain or gini index for splits? &lt;/p&gt;&#xA;" OwnerUserId="432" LastEditorUserId="432" LastEditDate="2014-06-10T01:43:47.883" LastActivityDate="2014-06-10T01:43:47.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="244" PostTypeId="2" ParentId="138" CreationDate="2014-06-10T01:40:23.263" Score="4" Body="&lt;p&gt;Coming from a programmers perspective, frameworks rarely target performance as the highest priority.  If your library is going to be widely leveraged the things people are likely to value most are ease of use, flexibility, and reliability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performance is generally valued in secondary competitive libraries.  &quot;X library is better because it's faster.&quot;  Even then very frequently those libraries will trade off the most optimal solution for one that can be widely leveraged.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using any framework you are inherently taking a risk that a faster solution exists.  I might go so far as to say that a faster solution almost always exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writing something yourself is not a guarantee of performance, but if you know what you are doing and have a fairly limited set of requirements it can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example might be JSON parsing.  There are a hundred libraries out there for a variety of languages that will turn JSON into a referable object and vice versa.  I know of one implementation that does it all in CPU registers.  It's measurably faster than all other parsers, but it is also very limited and that limitation will vary based on what CPU you are working with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the task of building a high-performant environment specific JSON parser a good idea?  I would leverage a respected library 99 times out of 100.  In that one separate instance a few extra CPU cycles multiplied by a million iterations would make the development time worth it.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T01:40:23.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="245" PostTypeId="2" ParentId="234" CreationDate="2014-06-10T01:54:40.647" Score="3" Body="&lt;p&gt;Computer Science is itself a multi-disciplinary field which has varying requirements among universities.  For example, Stockholm University does not require any math above algebra for its CS programs (some courses may have higher requirements, but not often).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what you mean by a machine learning program being more rigorous.  They are just two different programs.  Data Science would likely take a broader view and focus on application and management (business courses are maybe on offer?).  The research could be rigorous in its own right, but it definitely won't be tailored to someone who wants to optimize new algorithms or solve the low-level problems of machine learning.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't see the Ph.D program listed yet in the link you provided.  Will you please follow up here if you get more specific information?&lt;/p&gt;&#xA;" OwnerUserId="432" LastActivityDate="2014-06-10T01:54:40.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="246" PostTypeId="2" ParentId="134" CreationDate="2014-06-10T02:42:02.050" Score="4" Body="&lt;p&gt;There will definitely be a translation task at the end if you prototype using just mongo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you run a MapReduce task on mongodb, it has the data source and structure built in.  When you eventually convert to hadoop, your data structures might not look the same.  You could leverage the mongodb-hadoop connector to access mongo data directly from within hadoop, but that won't be quite as straightforward as you might think.  The time to figure out how exactly to do the conversion most optimally will be easier to justify once you have a prototype in place, IMO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you will need to translate mapreduce functions, the basic pseudocode should apply well to both systems.  You won't find anything that can be done in MongoDB that can't be done using Java or that is significantly more complex to do with Java.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T02:42:02.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="247" PostTypeId="2" ParentId="235" CreationDate="2014-06-10T02:50:51.153" Score="12" Body="&lt;p&gt;&lt;a href=&quot;http://bokeh.pydata.org/&quot;&gt;Bokeh&lt;/a&gt; is an excellent data visualization library for python.&#xA;&lt;a href=&quot;http://www.cityinabottle.org/nodebox/&quot;&gt;NodeBox&lt;/a&gt; is another that comes to mind.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T02:50:51.153" CommentCount="0" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="248" PostTypeId="2" ParentId="234" CreationDate="2014-06-10T03:21:56.473" Score="3" Body="&lt;p&gt;A cash cow program?  No.  PhD programs are never cash cows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know why you couldn't be a professor with a PhD in data science.  Rarely does a professor of a given course have to have a specific degree in order to teach it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as publishing goes, there are any number of related journals that would accept papers from somebody on topics that would be covered by the topic of Data Science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I went to college, MIS, Computer Engineering, and Computer Science were new subjects.  Most of the people in my graduating class for Computer Science couldn't program anything significant at graduation.  Within a few years, CS programs around the country matured significantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are part of a new program, sometimes it's possible to help define what it is that's required for graduation.  Being a part of that puts you in rare company for that field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as mathematical rigor is concerned, I would expect Data Science to leverage a heavy dose of mathematically based material.  I wouldn't expect anything particularly new - statistics, calculus, etc. should have been covered in undergrad.  Masters and PhD programs should be more about applying that knowledge and not so much about learning it.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T03:21:56.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="249" PostTypeId="2" ParentId="228" CreationDate="2014-06-10T03:42:51.637" Score="4" Body="&lt;p&gt;Twitter uses Storm for real-time processing of data.  Problems can happen with real-time data.  Systems might go down.  Data might be inadvertently processed twice.  Network connections can be lost.  A lot can happen in a real-time system.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They use hadoop to reliably process historical data.  I don't know specifics, but for instance, getting solid information from aggregated logs is probably more reliable than attaching to the stream.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they simply relied on Storm for everything - Storm would have problems due to the nature of providing real-time information at scale.  If they relied on hadoop for everything, there's a good deal of latency involved.  Combining the two with Summingbird is the next logical step.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T03:42:51.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="250" PostTypeId="2" ParentId="184" CreationDate="2014-06-10T04:47:13.040" Score="3" Body="&lt;p&gt;Google does not publish the models they use, but they specifically do not support models from the PMML specification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look closely at the documentation on &lt;a href=&quot;https://developers.google.com/prediction/docs/pmml-schema&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;, you will notice that the model selection within the schema is greyed out indicating that it is an unsupported feature of the schema.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://developers.google.com/prediction/docs/developer-guide#whatisprediction&quot; rel=&quot;nofollow&quot;&gt;documentation does spell out&lt;/a&gt; that by default it will use a regression model for training data that has numeric answers, and an unspecified categorization model for training data that results in text based answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Google Prediction API also supports hosted models (although only a few demo models are currently available), and models specified with a PMML transform.  The documentation does contain an &lt;a href=&quot;https://developers.google.com/prediction/docs/pmml-schema&quot; rel=&quot;nofollow&quot;&gt;example of a model defined by a PMML transform&lt;/a&gt;.  (There is also a note on that page stating that PMML ...Model elements are not supported).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The PMML standard that google partially supports is &lt;a href=&quot;http://www.dmg.org//pmml-v4-0-1.html&quot; rel=&quot;nofollow&quot;&gt;version 4.0.1&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T04:47:13.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="251" PostTypeId="2" ParentId="138" CreationDate="2014-06-10T05:57:13.897" Score="4" Body="&lt;p&gt;Having done the rewriting game over and over myself (and still doing it), my immediate reaction was &lt;em&gt;adaptability&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While frameworks and libraries have a huge arsenal of (possibly intertwinable) routines for standard tasks, their framework property often (always?) disallows shortcuts.  In fact, most frameworks have some sort of core infrastructure around which a core layer of basic functionality is implemented.  More specific functionality makes use of the basic layer and is placed in a second layer around the core.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now by shortcuts I mean going straight from a second layer routine to another second layer routine without using the core.  Typical example (from my domain) would be timestamps: You have a timestamped data source of some kind.  Thus far the job is simply to read the data off the wire and pass it to the core so your other code can feast on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now your industry changes the default timestamp format for a very good reason (in my case they went from unix time to GPS time).  Unless your framework is industry-specific it is very unlikely that they're willing to change the core representation of time, so you end up using a framework that &lt;em&gt;almost&lt;/em&gt; does what you want.  Every time you access your data you have to convert it to industry-time-format first, and every time you want it modified you have to convert it back to whatever the core deems appropriate.  There is no way that you can hand over data straight from the source to a sink without double conversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where your hand-crafted frameworks will shine, it's just a minor change and you're back modelling the real world whereas all other (non-industry-specific) frameworks will now have a performance disadvantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over time, the discrepancy between the real world and the model will add up.  With an off-the-shelf framework you'd soon be facing questions like: How can I represent &lt;code&gt;this&lt;/code&gt; in &lt;code&gt;that&lt;/code&gt; or how do make routine &lt;code&gt;X&lt;/code&gt; accept/produce &lt;code&gt;Y&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far this wasn't about C/C++.  But if, for some reason, you can't change the framework, i.e. you do have to put up with double conversion of data to go from one end to another, then you'd typically employ something that minimises the additional overhead.  In my case, a TAI-&gt;UTC or UTC-&gt;TAI converter is best left to raw C (or an FPGA).  There is no elegance possible, no profound smart data structure that makes the problem trivial.  It's just a boring switch statement, and why not use a language whose compilers are good at optimising exactly that?&lt;/p&gt;&#xA;" OwnerUserId="451" LastActivityDate="2014-06-10T05:57:13.897" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="252" PostTypeId="1" CreationDate="2014-06-10T06:19:46.510" Score="3" ViewCount="880" Body="&lt;p&gt;There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;HBase Thrift API&lt;/li&gt;&#xA;&lt;li&gt;HBase Stargate (REST API) &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Does anyone know which one should be used in which circumstances?&#xA;I.e. what are their main differences, and pros/cons?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2014-06-10T07:56:45.587" Title="HBase connector - Thrift or REST" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="253" PostTypeId="1" AcceptedAnswerId="256" CreationDate="2014-06-10T06:20:20.817" Score="40" ViewCount="21214" Body="&lt;p&gt;An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn Hadoop to be a Data Scientist? &lt;/p&gt;&#xA;" OwnerUserId="456" LastActivityDate="2022-08-26T05:51:09.493" Title="Do I need to learn Hadoop to be a Data Scientist?" Tags="|bigdata|apache-hadoop|" AnswerCount="10" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="254" PostTypeId="2" ParentId="169" CreationDate="2014-06-10T07:10:45.057" Score="7" Body="&lt;p&gt;Actually there are more than one question to answer here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to work on schemaless/loose/missing data&lt;/li&gt;&#xA;&lt;li&gt;How to label a person (from what I understand unsupervised) and create an identifier&lt;/li&gt;&#xA;&lt;li&gt;How to train your system so that it can tell you which attributes you should use in order to identify the person &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As Rubens mentioned, you can use &lt;strong&gt;decision tree&lt;/strong&gt; methods, specifically &lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot;&gt;Random Forests&lt;/a&gt; for calculating the most important attributes based on information gain if you have already found a way to identify how to label a person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you do not have any label information maybe you can use some expert view for preliminary attribute selection. After that you make &lt;strong&gt;unsupervised classification&lt;/strong&gt; in order to retrieve your labels. Lastly, you can select the most important fields using &lt;strong&gt;Random Forest&lt;/strong&gt; or other methods like &lt;strong&gt;Bayesian Belief Networks&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to achieve all that, you also need complete data set. If your data set is loose you have to manually or heuristically find a way to couple attributes indicating same thing with different names. What is more, you can use &lt;em&gt;imputation&lt;/em&gt; techniques such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm&quot;&gt;Expectation Maximization&lt;/a&gt; method and complete your data set. Or you can also work with Bayesian Networks and can leave missing fields as they are.&lt;/p&gt;&#xA;" OwnerUserId="463" LastActivityDate="2014-06-10T07:10:45.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="255" PostTypeId="2" ParentId="253" CreationDate="2014-06-10T07:28:49.127" Score="2" Body="&lt;p&gt;You should learn Hadoop if you want to be work as data scientist, but maybe before starting with Hadoop you should read something about ETL or Big Data... this book could be a good starting point: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1617290343&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Big-Data-Principles-practices-scalable/dp/1617290343&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps and good luck!&lt;/p&gt;&#xA;" OwnerUserId="466" LastActivityDate="2014-06-10T07:28:49.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="256" PostTypeId="2" ParentId="253" CreationDate="2014-06-10T07:30:53.510" Score="49" Body="&lt;p&gt;Different people use different tools for different things.  Terms like Data Science are generic for a reason.  A data scientist could spend an entire career without having to learn a particular tool like hadoop.  Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say that a data scientist should be familiar with concepts like MapReduce, distributed systems, distributed file systems, and the like, but I wouldn't judge someone for not knowing about such things.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a big field.  There is a sea of knowledge and most people are capable of learning and being an expert in a single drop.  The key to being a scientist is having the desire to learn and the motivation to know that which you don't already know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example:  I could hand the right person a hundred structured CSV files containing information about classroom performance in one particular class over a decade.  A data scientist would be able to spend a year gleaning insights from the data without ever needing to spread computation across multiple machines.  You could apply machine learning algorithms, analyze it using visualizations, combine it with external data about the region, ethnic makeup, changes to environment over time, political information, weather patterns, etc.  All of that would be &quot;data science&quot; in my opinion.  It might take something like hadoop to test and apply anything you learned to data comprising an entire country of students rather than just a classroom, but that final step doesn't necessarily make someone a data scientist.  And not taking that final step doesn't necessarily disqualify someone from being a data scientist.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-06-10T08:21:19.197" LastActivityDate="2014-06-10T08:21:19.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="257" PostTypeId="2" ParentId="253" CreationDate="2014-06-10T07:42:57.470" Score="4" Body="&lt;p&gt;Yes, you should learn a platform that is capable of dissecting your problem as a data parallel problem. Hadoop is one. For your simple needs (design patterns like counting, aggregation, filtering etc.) you need Hadoop and for more complex Machine Learning stuff like doing some Bayesian, SVM you need Mahout which in turn needs Hadoop (Now Apache Spark) to solve your problem using a data-parallel approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So Hadoop is a good platform to learn and really important for your batch processing needs. Not only Hadoop but you also need to know Spark (Mahout runs it's algorithms utilizing Spark) &amp;amp; Twitter Storm (for your real time analytics needs). This list will continue and evolve so if you are good with the building blocks (Distributed Computing, Data-Parallel Problems and so on) and know how one such platform (say Hadoop) operates you will fairly quickly be up to speed on others.&lt;/p&gt;&#xA;" OwnerUserId="11" LastActivityDate="2014-06-10T07:42:57.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="258" PostTypeId="2" ParentId="252" CreationDate="2014-06-10T07:56:45.587" Score="2" Body="&lt;p&gt;Thrift is generally faster because the data being exchanged is smaller.  Stargate offers a web service which is an integration method that is widely supported, which is a concern when you are working with commercial products with limited integration possibilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a closed environment where everything is controlled, I would prefer Thrift.  If I'm exposing data to external teams or systems I would prefer Stargate.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T07:56:45.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="259" PostTypeId="2" ParentId="155" CreationDate="2014-06-10T08:04:20.400" Score="21" Body="&lt;p&gt;I would like to point to &lt;a href=&quot;http://national.census.okfn.org/&quot;&gt;The Open Data Census&lt;/a&gt;. It is an initiative of the Open Knowledge Foundation based on contributions from open data advocates and experts around the world. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value of Open data Census is open, community driven, and systematic effort to collect and update the database of open datasets globally on country and, in some cases, &lt;a href=&quot;http://us-city.census.okfn.org/&quot;&gt;like U.S., on city level&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it presents an opportunity to compare different countries and cities on in selected areas of interest.&lt;/p&gt;&#xA;" OwnerUserId="454" LastActivityDate="2014-06-10T08:04:20.400" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="260" PostTypeId="2" ParentId="191" CreationDate="2014-06-10T08:38:27.093" Score="2" Body="&lt;p&gt;The algorithm that is used in this case is called &lt;a href=&quot;https://class.coursera.org/ml-003/lecture/38&quot; rel=&quot;nofollow&quot;&gt;one-vs-all classifier&lt;/a&gt; or multiclass classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you have to take one class, e. g. number 1 , mark it as positive and combine the rest seven classes in one negative class. The neural network will output the probability of this case being class number 1 vs the rest of the classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Afterwords, you have to assign as positive another class, e.g. number 2, assign all other classes as one big negative class and get the predicted probability from the network again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After repeating this procedure for all eight classes, assign each case to the the class that had the maximum probability from all the classes outputted from the neural network. &lt;/p&gt;&#xA;" OwnerUserId="454" LastActivityDate="2014-06-10T08:38:27.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="261" PostTypeId="2" ParentId="196" CreationDate="2014-06-10T09:21:14.013" Score="3" Body="&lt;p&gt;You could take a look at CN2 rule learner in &lt;a href=&quot;http://orange.biolab.si/orange2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange 2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="480" LastEditorUserId="98307" LastEditDate="2020-08-06T11:04:09.857" LastActivityDate="2020-08-06T11:04:09.857" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="262" PostTypeId="1" AcceptedAnswerId="293" CreationDate="2014-06-10T09:26:06.593" Score="40" ViewCount="26290" Body="&lt;p&gt;What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorDisplayName="user10640" LastEditDate="2020-04-14T16:54:08.463" LastActivityDate="2021-07-02T00:43:22.377" Title="What are the advantages of HDF compared to alternative formats?" Tags="|data-formats|hierarchical-data-format|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="263" PostTypeId="2" ParentId="172" CreationDate="2014-06-10T09:41:34.697" Score="8" Body="&lt;p&gt;Unfortunately, parallelization is not yet implemented in pandas. You can join &lt;a href=&quot;http://github.com/pydata/pandas/issues/5751&quot;&gt;this github issue&lt;/a&gt; if you want to participate in the development of this feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know any &quot;magic unicorn package&quot; for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you can try the two methods built into MongoDB (map reduce and agg framework).  See &lt;a href=&quot;http://docs.mongodb.org/manual/core/aggregation/&quot;&gt;mongodb_agg_framework&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-10T09:41:34.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="264" PostTypeId="2" ParentId="22" CreationDate="2014-06-10T10:48:58.457" Score="4" Body="&lt;p&gt;You can also give the Expectation Maximization clustering algorithm a try.  It can work on categorical data and will give you a statistical likelihood of which categorical value (or values) a cluster is most likely to take on.&lt;/p&gt;&#xA;" OwnerUserId="490" LastActivityDate="2014-06-10T10:48:58.457" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="265" PostTypeId="1" AcceptedAnswerId="285" CreationDate="2014-06-10T10:58:58.447" Score="42" ViewCount="45982" Body="&lt;p&gt;I have a variety of NFL datasets that I think might make a good side-project, but I haven't done anything with them just yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to this site made me think of machine learning algorithms and I wondering how good they might be at either predicting the outcome of football games or even the next play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that there would be some trends that could be identified - on 3rd down and 1, a team with a strong running back &lt;em&gt;theoretically should&lt;/em&gt; have a tendency to run the ball in that situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scoring might be more difficult to predict, but the winning team might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether these are good questions to throw at a machine learning algorithm.  It could be that a thousand people have tried it before, but the nature of sports makes it an unreliable topic.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="553" LastEditDate="2015-03-02T12:33:11.007" LastActivityDate="2020-08-20T18:25:42.540" Title="Can machine learning algorithms predict sports scores or plays?" Tags="|machine-learning|sports|" AnswerCount="10" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="266" PostTypeId="1" AcceptedAnswerId="272" CreationDate="2014-06-10T11:05:47.273" Score="12" ViewCount="3011" Body="&lt;p&gt;Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious as to what applications you might recommend that would offer the fastest time from installation to producing a meaningful result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, any recommendations for good getting-started materials on the subject of machine-learning in general would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-12T17:58:21.467" Title="What are some easy to learn machine-learning applications?" Tags="|machine-learning|" AnswerCount="9" CommentCount="1" ClosedDate="2014-08-07T23:41:10.580" ContentLicense="CC BY-SA 3.0" />
  <row Id="268" PostTypeId="2" ParentId="266" CreationDate="2014-06-10T11:36:19.287" Score="5" Body="&lt;p&gt;I think &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;noreferrer&quot;&gt;Weka&lt;/a&gt; is a good starting point. You can do a bunch of stuff like supervised learning or clustering and easily compare a large set of algorithms na methodologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weka's manual is actually a book on machine learning and data mining that can be used as introductory material. &lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2014-06-10T11:36:19.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="269" PostTypeId="2" ParentId="265" CreationDate="2014-06-10T11:37:28.293" Score="9" Body="&lt;p&gt;Definitely they can.&#xA;I can target you to a &lt;strong&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.56.7448&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;noreferrer&quot;&gt;nice paper&lt;/a&gt;&lt;/strong&gt;. Once I used it for soccer league results prediction algorithm implementation, primarily aiming at having some value against bookmakers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From paper's abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a Bayesian dynamic generalized model to estimate the time dependent skills of all teams in a league, and to predict next weekend's soccer matches.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Keywords:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dynamic Models, Generalized Linear Models, Graphical Models, Markov&#xA;  Chain Monte Carlo Methods, Prediction of Soccer Matches&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Citation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Rue, Havard, and Oyvind Salvesen. &quot;Prediction and retrospective analysis of soccer matches in a league.&quot; Journal of the Royal Statistical Society: Series D (The Statistician) 49.3 (2000): 399-418.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="97" LastEditorUserId="322" LastEditDate="2014-06-25T16:03:21.447" LastActivityDate="2014-06-25T16:03:21.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="270" PostTypeId="2" ParentId="265" CreationDate="2014-06-10T11:39:19.603" Score="7" Body="&lt;p&gt;Machine learning and statistical techniques can improve the forecast, but nobody can predict the real result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was a kaggle competition a few month ago about &lt;a href=&quot;https://www.kaggle.com/c/march-machine-learning-mania&quot; rel=&quot;noreferrer&quot;&gt;predicting the 2014 NCAA Tournament&lt;/a&gt;. You can read the Competition Forum to get a better idea on what people did and what results did they achieve.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-10T11:39:19.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="271" PostTypeId="2" ParentId="265" CreationDate="2014-06-10T11:49:23.777" Score="8" Body="&lt;p&gt;It has been shown before that machine learning techniques can be applied for predicting sport results. Simple google search should give you a bunch of results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it has also been showed (for NFL btw) that very complex predictive models, simple predictive models, questioning people, or crowd knowledge by utilising betting info, they all perform more or less the same. Source: &quot;&lt;a href=&quot;http://everythingisobvious.com/&quot; rel=&quot;noreferrer&quot;&gt;Everything is obvious once you know the answer - How common sense Fails&lt;/a&gt;&quot;, Chapter 7, by Duncan Watts. &lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2014-06-10T11:49:23.777" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="272" PostTypeId="2" ParentId="266" CreationDate="2014-06-10T11:53:07.737" Score="13" Body="&lt;p&gt;I would recommend to start with some MOOC on machine learning. For example Andrew Ng's &lt;a href=&quot;https://www.coursera.org/course/ml&quot;&gt;course&lt;/a&gt; at coursera.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also take a look at &lt;a href=&quot;http://orange.biolab.si/&quot;&gt;Orange&lt;/a&gt; application. It has a graphical interface and probably it is easier to understand some ML techniques using it. &lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-10T11:53:07.737" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="273" PostTypeId="2" ParentId="253" CreationDate="2014-06-10T12:10:28.713" Score="2" Body="&lt;p&gt;You can apply data science techniques to data on one machine so the answer to the question as the OP phrased it, is no.&lt;/p&gt;&#xA;" OwnerUserId="498" LastEditorUserId="498" LastEditDate="2014-06-10T15:04:01.177" LastActivityDate="2014-06-10T15:04:01.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="274" PostTypeId="2" ParentId="262" CreationDate="2014-06-10T12:57:04.307" Score="12" Body="&lt;p&gt;One benefit is wide support - C, Java, Perl, Python, and R all have HDF5 bindings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another benefit is speed.  I haven't ever seen it benchmarked, but HDF is supposed to be faster than SQL databases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it is very good when used with both large sets of scientific data and time series data - network monitoring, usage tracking, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't believe there is a size limitation for HDF files (although OS limits would still apply.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T12:57:04.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="275" PostTypeId="2" ParentId="155" CreationDate="2014-06-10T13:17:48.433" Score="27" Body="&lt;p&gt;For time series data in particular, &lt;a href=&quot;http://www.quandl.com/&quot;&gt;Quandl&lt;/a&gt; is an excellent resource -- an easily browsable directory of (mostly) clean time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of their coolest features is &lt;a href=&quot;http://blog.quandl.com/blog/quandl-open-data/&quot;&gt;open-data stock prices&lt;/a&gt; -- i.e. financial data that can be edited wiki-style, and isn't encumbered by licensing.&lt;/p&gt;&#xA;" OwnerUserId="508" LastActivityDate="2014-06-10T13:17:48.433" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="276" PostTypeId="2" ParentId="155" CreationDate="2014-06-10T13:38:31.207" Score="10" Body="&lt;p&gt;Not all government data is listed on data.gov - &lt;a href=&quot;http://sunlightfoundation.com/blog/2014/02/21/open-data-inventories-ready-for-human-consumption/&quot;&gt;Sunlight Foundation&lt;/a&gt; put together a &lt;a href=&quot;https://drive.google.com/folderview?id=0B4QuErjcV2a0WXVDOURwbzh6S2s&amp;amp;usp=sharing&quot;&gt;set of spreadsheets&lt;/a&gt; back in February describing sets of available data.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-10T13:38:31.207" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="277" PostTypeId="2" ParentId="266" CreationDate="2014-06-10T14:25:41.903" Score="11" Body="&lt;p&gt;To be honest, I think that doing some projects will teach you much more than doing a full course. One reason is that doing a project is more motivating and open-ended than doing assignments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A course, if you have the time AND motivation (real motivation), is better than doing a project. The other commentators have made good platform recommendations on tech. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, from a fun project standpoint, you should ask a question and get a computer to learn to answer it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some good classic questions that have good examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Neural Networks for recognizing hand written digits&lt;/li&gt;&#xA;&lt;li&gt;Spam email classification using logistic regression&lt;/li&gt;&#xA;&lt;li&gt;Classification of objects using Gaussian Mixture models&lt;/li&gt;&#xA;&lt;li&gt;Some use of linear regression, perhaps forecasting of grocery prices given neighborhoods&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These projects have the math done, code done, and can be found with Google readily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other cool subjects can be done by you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I research robotics, so for me the most FUN applications are behavioral.&#xA;Examples can include (if you can play with an arduino)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a application, that uses logistic regression perhaps, that learns when to turn the fan off and on given the inner temperature, and the status of the light in the room.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create an application that teaches a robot to move an actuator, perhaps a wheel, based on sensor input (perhaps a button press), using Gaussian Mixture Models (learning from demonstration).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, those are pretty advanced. The point I'm making is that if you pick a project that you (really really) like, and spend a few week on it, you will learn a massive amount, and understand so much more than you will get doing a few assignments.&lt;/p&gt;&#xA;" OwnerUserId="518" LastActivityDate="2014-06-10T14:25:41.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="278" PostTypeId="2" ParentId="266" CreationDate="2014-06-10T14:30:33.667" Score="2" Body="&lt;p&gt;Assuming you're familiar with programming I would recommend looking at &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;. It has especially nice help pages that can serve as mini-tutorials/a quick tour through machine learning. Pick an area you find interesting and work through the examples. &lt;/p&gt;&#xA;" OwnerUserId="524" LastActivityDate="2014-06-10T14:30:33.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="279" PostTypeId="2" ParentId="155" CreationDate="2014-06-10T14:57:47.810" Score="19" Body="&lt;p&gt;There is also another resource provided by The Guardian, the British Daily on their website. The datasets published by the Guardian Datablog are all hosted. Datasets related to Football Premier League Clubs' accounts, Inflation and GDP details of UK, Grammy awards data etc.&#xA;The datasets are available at &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.theguardian.com/news/datablog/interactive/2013/jan/14/all-our-datasets-index&quot;&gt;http://www.theguardian.com/news/datablog/interactive/2013/jan/14/all-our-datasets-index&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some more resources. Some of the datasets are in R format or R commads exist for directly importing data to R.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.inside-r.org/howto/finding-data-internet&quot;&gt;http://www.inside-r.org/howto/finding-data-internet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="514" LastEditorUserId="514" LastEditDate="2014-06-11T16:30:06.930" LastActivityDate="2014-06-11T16:30:06.930" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="280" PostTypeId="1" CreationDate="2014-06-10T15:08:54.073" Score="8" ViewCount="167" Body="&lt;p&gt;I am developing a system that is intended to capture the &quot;context&quot; of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made to the system.  It is hoped that this data can then power ML features such as context aware information retrieval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having trouble deciding on what features to select in addition to these user tags - the URL being requested, approximate time spent with any given resource, estimating the current &quot;activity&quot; within the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested to know if there are good examples of this kind of technology or any prior research on the subject - a cursory search of the ACM DL revealed some related papers but nothing really spot-on.&lt;/p&gt;&#xA;" OwnerUserId="531" LastActivityDate="2020-06-19T08:28:09.320" Title="Feature selection for tracking user activity within an application" Tags="|feature-selection|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="282" PostTypeId="2" ParentId="265" CreationDate="2014-06-10T16:25:24.223" Score="14" Body="&lt;p&gt;Yes. Why not?!&#xA;With so much of data being recorded in each sport in each game, smart use of data could lead us to obtain important insights regarding player performance.&lt;/p&gt;&#xA;&lt;p&gt;Some examples:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Baseball&lt;/strong&gt;: In the movie Moneyball (which is an adaptation of the Moneyball book), Brad Pitt plays a character who analyses player statistics to come up with a team that performs tremendously well! It was a depiction of the real-life story of Oakland Athletics baseball team. For more &lt;a href=&quot;http://www.theatlantic.com/entertainment/archive/2013/09/forget-2002-this-years-oakland-as-are-the-real-em-moneyball-em-team/279927/&quot; rel=&quot;nofollow noreferrer&quot;&gt;info&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Cricket&lt;/strong&gt;: SAP Labs has come up with an auction analytics tool that has given insights about impact players to buy in the 2014 Indian Premier League auction for the Kolkata Knight Riders team, which eventually went on to win the 2014 IPL &lt;strong&gt;Championship&lt;/strong&gt;. For more &lt;a href=&quot;http://scn.sap.com/community/hana-in-memory/blog/2014/06/10/sap-hana-academy-cricket-demo--how-sap-hana-powered-the-kolkata-knight-riders-to-ipl-championship&quot; rel=&quot;nofollow noreferrer&quot;&gt;info&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;So, yes, statistical analysis of the player records can give us insights about &lt;strong&gt;which players are more likely to perform but not which players will perform&lt;/strong&gt;. So, machine learning, a close cousin of statistical analysis will be proving to be a game changer.&lt;/p&gt;&#xA;" OwnerUserId="514" LastEditorUserId="98307" LastEditDate="2020-08-20T18:25:04.213" LastActivityDate="2020-08-20T18:25:04.213" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="284" PostTypeId="2" ParentId="280" CreationDate="2014-06-10T17:06:54.950" Score="5" Body="&lt;p&gt;Well, this may not answer the question thoroughly, but since you're dealing with information retrieval, it may be of some use. &lt;a href=&quot;http://moz.com/search-ranking-factors&quot; rel=&quot;noreferrer&quot;&gt;This page&lt;/a&gt; mantains a set of features and associated &lt;em&gt;correlations&lt;/em&gt; with page-ranking methods of search engines. As a disclaimer from the webpage itself:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that these factors are not &quot;proof&quot; of what search engines use to rank websites, but simply show the characteristics of web pages that tend to rank higher.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The list pointed may give you some insights on which features would be nice to select. For example, considering the second most correlated feature, # of google +1's, it may be possible to add some probability of a user making use of such service if he/she accesses many pages with high # of google +1 (infer &quot;user context&quot;). Thus, you could try to &quot;guess&quot; some other relations that may shed light on interesting features for your tracking app.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-06-10T17:06:54.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="285" PostTypeId="2" ParentId="265" CreationDate="2014-06-10T17:15:52.953" Score="19" Body="&lt;p&gt;There are a lot of good questions about Football (and sports, in general) that would be awesome to throw to an algorithm and see what comes out. The tricky part is to know &lt;em&gt;what&lt;/em&gt; to throw to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A team with a good RB could just pass on 3rd-and-short just because the opponents would probably expect run, for instance. So, in order to actually produce some worthy results, I'd break the problem in smaller pieces and analyse them statistically while throwing them to the machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few (good) websites that try to do the same, you should check'em out and use whatever they found to help you out:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.footballoutsiders.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Football Outsiders&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.advancedfootballanalytics.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Advanced Football Analytics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And if you truly want to explore Sports Data Analysis, you should definitely check the &lt;a href=&quot;http://www.sloansportsconference.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sloan Sports Conference&lt;/a&gt; videos. There's a lot of them spread on Youtube.&lt;/p&gt;&#xA;" OwnerUserId="553" LastEditorUserId="553" LastEditDate="2019-01-23T14:36:40.430" LastActivityDate="2019-01-23T14:36:40.430" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="287" PostTypeId="2" ParentId="231" CreationDate="2014-06-10T17:32:19.120" Score="5" Body="&lt;p&gt;I do not know a standard answer to this, but I thought about it some times ago and I have some ideas to share. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have one confusion matrix, you have more or less a picture of how you classification model confuse (mis-classify) classes. When you repeat classification tests you will end up having multiple confusion matrices. The question is how to get a meaningful aggregate confusion matrix. The answer depends on what is the meaning of meaningful (pun intended). I think there is not a single version of meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way is to follow the rough idea of multiple testing. In general, you test something multiple times in order to get more accurate results. As a general principle one can reason that averaging on the results of the multiple tests reduces the variance of the estimates, so as a consequence, it increases the precision of the estimates. You can proceed in this way, of course, by summing position by position and then dividing by the number of tests. You can go further and instead of estimating only a value for each cell of the confusion matrix, you can also compute some confidence intervals, t-values and so on. This is OK from my point of view. But it tell only one side of the story.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other side of the story which might be investigated is how stable are the results for the same instances. To exemplify that I will take an extreme example. Suppose you have a classification model for 3 classes. Suppose that these classes are in the same proportion. If your model is able to predict one class perfectly and the other 2 classes with random like performance, you will end up having 0.33 + 0.166 + 0.166 = 0.66 misclassification ratio. This might seem good, but even if you take a look on a single confusion matrix you will not know that your performance on the last 2 classes varies wildly. Multiple tests can help. But averaging the confusion matrices would reveal this? My belief is not. The averaging will give the same result more or less, and doing multiple tests will only decrease the variance of the estimation. However it says nothing about the wild instability of prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So another way to do compose the confusion matrices would better involve a prediction density for each instance. One can build this density by counting for each instance, the number of times it was predicted a given class. After normalization, you will have for each instance a prediction density rather a single prediction label. You can see that a single prediction label is similar with a degenerated density where you have probability of 1 for the predicted class and 0 for the other classes for each separate instance. Now having this densities one can build a confusion matrix by adding the probabilities from each instance and predicted class to the corresponding cell of the aggregated confusion matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can argue that this would give similar results like the previous method. However I think that this might be the case sometimes, often when the model has low variance, the second method is less affected by how the samples from the tests are drawn, and thus more stable and closer to the reality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the second method might be altered in order to obtain a third method, where one can assign as prediction the label with highest density from the prediction of a given instance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not implemented those things but I plan to study further because I believe might worth spending some time. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2014-06-11T09:39:34.373" LastActivityDate="2014-06-11T09:39:34.373" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="288" PostTypeId="2" ParentId="280" CreationDate="2014-06-10T17:36:11.580" Score="5" Body="&lt;p&gt;The goal determines the features, so I would initially take as many as possible, then use cross validation to select the optimal subset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My educated guess is that a Markov model would work. If you discretize the action space (e.g., select this menu item, press that button, etc.), you can predict the next action based on the past ones. It's a &lt;em&gt;sequence&lt;/em&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Structured_prediction&quot; rel=&quot;nofollow&quot;&gt;structured prediction&lt;/a&gt; problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For commercial offerings, search &lt;strong&gt;app analytics&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-06-11T01:19:18.183" LastActivityDate="2014-06-11T01:19:18.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="289" PostTypeId="1" AcceptedAnswerId="291" CreationDate="2014-06-10T17:56:34.847" Score="10" ViewCount="595" Body="&lt;p&gt;Yann LeCun mentioned in his &lt;a href=&quot;http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_yann_lecun/&quot; rel=&quot;nofollow&quot;&gt;AMA&lt;/a&gt; that he considers having a PhD very important in order to get a job at a top company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a masters in statistics and my undergrad was in economics and applied math, but I am now looking into ML PhD programs. Most programs say there are no absolutely necessary CS courses; however I tend to think most accepted students have at least a very strong CS background. I am currently working as a data scientist/statistician but my company will pay for courses. Should I take some intro software engineering courses at my local University to make myself a stronger candidate? What other advice you have for someone applying to PhD programs from outside the CS field?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: I have taken a few MOOCs (Machine Learning, Recommender Systems, NLP) and code R/python on a daily basis. I have a lot of coding experience with statistical languages and implement ML algorithms daily. I am more concerned with things that I can put on applications.&lt;/p&gt;&#xA;" OwnerUserId="560" LastEditorUserId="560" LastEditDate="2015-09-10T20:17:35.897" LastActivityDate="2015-09-10T20:17:35.897" Title="Qualifications for PhD Programs" Tags="|education|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="290" PostTypeId="2" ParentId="266" CreationDate="2014-06-10T18:22:32.610" Score="2" Body="&lt;p&gt;I found the pluralsight course &lt;a href=&quot;http://pluralsight.com/training/courses/TableOfContents?courseName=introduction-to-machine-learning-encog&amp;amp;highlight=abhishek-kumar_introduction-to-machine-learning-encog-m2-applications!abhishek-kumar_introduction-to-machine-learning-encog-m3-tasks!abhishek-kumar_introduction-to-machine-learning-encog-m1-intro%2a1#introduction-to-machine-learning-encog-m2-applications&quot; rel=&quot;nofollow&quot;&gt;Introduction to machine learning encog&lt;/a&gt; a great resource so start with. It uses the &lt;a href=&quot;http://www.heatonresearch.com/encog&quot; rel=&quot;nofollow&quot;&gt;Encog library&lt;/a&gt; to quickly explore different ml techniques. &lt;/p&gt;&#xA;" OwnerUserId="571" LastActivityDate="2014-06-10T18:22:32.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="291" PostTypeId="2" ParentId="289" CreationDate="2014-06-10T18:55:39.010" Score="10" Body="&lt;p&gt;If I were you I would take a MOOC or two (e.g., &lt;a href=&quot;https://www.coursera.org/course/algs4partI&quot;&gt;Algorithms, Part I&lt;/a&gt;, &lt;a href=&quot;https://www.coursera.org/course/algs4partII&quot;&gt;Algorithms, Part II&lt;/a&gt;, &lt;a href=&quot;https://www.coursera.org/course/progfun&quot;&gt;Functional Programming Principles in Scala&lt;/a&gt;), a good book on data structures and algorithms, then just code as much as possible. You could implement some statistics or ML algorithms, for example; that would be good practice for you and useful to the community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a PhD program, however, I would also make sure I were familiar with the type of maths they use. If you want to see what it's like at the deep end, browse the papers at the &lt;a href=&quot;http://jmlr.org/papers/&quot;&gt;JMLR&lt;/a&gt;. That will let you calibrate yourself in regards to theory; can you sort of follow the maths?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and you don't need a PhD to work at top companies, unless you want to join research departments like his. But then you'll spend more time doing development, and you'll need good coding skills...&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-06-10T18:55:39.010" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="292" PostTypeId="2" ParentId="289" CreationDate="2014-06-10T19:43:11.860" Score="7" Body="&lt;p&gt;Your time would probably be better spent on Kaggle than in a PhD program. When you read the stories by winners (&lt;a href=&quot;http://blog.kaggle.com/&quot;&gt;Kaggle blog&lt;/a&gt;) you'll see that it takes a large amount of practice and the winners are not just experts of one single method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, being active and having a plan in a PhD program can get you connections that you otherwise would probably not get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the real question is for you - what are the reasons for wanting a job at a top company?&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2014-06-10T19:43:11.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="293" PostTypeId="2" ParentId="262" CreationDate="2014-06-10T20:28:54.613" Score="37" Body="&lt;p&gt;Perhaps a good way to paraphrase the question is, what are the advantages compared to alternative formats?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main alternatives are, I think: a database, text files, or another packed/binary format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The database options to consider are probably a columnar store or NoSQL, or for small self-contained datasets SQLite.  The main advantage of the database is the ability to work with data much larger than memory, to have random or indexed access, and to add/append/modify data quickly.  The main *dis*advantage is that it is much slower than HDF, for problems in which the entire dataset needs to be read in and processed.  Another disadvantage is that, with the exception of embedded-style databases like SQLite, a database is a system (requiring admnistration, setup, maintenance, etc) rather than a simple self-contained data store.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The text file format options are XML/JSON/CSV.  They are cross-platform/language/toolkit, and are a good archival format due to the ability to be self-describing (or obvious :).  If uncompressed, they are huge (10x-100x HDF), but if compressed, they can be fairly space-efficient (compressed XML is about the same as HDF).  The main disadvantage here is again speed: parsing text is much, much slower than HDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other binary formats (npy/npz numpy files, blz blaze files, protocol buffers, Avro, ...) have very similar properties to HDF, except they are less widely supported (may be limited to just one platform: numpy) and may have specific other limitations.  They typically do not offer a compelling advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HDF is a good complement to databases, it may make sense to run a query to produce a roughly memory-sized dataset and then cache it in HDF if the same data would be used more than once.  If you have a dataset which is fixed, and usually processed as a whole, storing it as a collection of appropriately sized HDF files is not a bad option.  If you have a dataset which is updated often, staging some of it as HDF files periodically might still be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, HDF is a good format for data which is read (or written) typically as a whole; it is the lingua franca or common/preferred interchange format for many applications due to wide support and compatibility, decent as an archival format, and very fast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. To give this some practical context, my most recent experience comparing HDF to alternatives, a certain small (much less than memory-sized) dataset took 2 seconds to read as HDF (and most of this is probably overhead from Pandas); ~1 minute to read from JSON; and 1 &lt;em&gt;hour&lt;/em&gt; to write to database.  Certainly the database write could be sped up, but you'd better have a good DBA!  This is how it works out of the box.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2014-06-10T20:28:54.613" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="294" PostTypeId="2" ParentId="253" CreationDate="2014-06-10T20:40:25.623" Score="9" Body="&lt;p&gt;As a former Hadoop engineer, it is not needed but it helps. Hadoop is just one system - the most common system, based on Java, and a ecosystem of products, which apply a particular technique &quot;Map/Reduce&quot; to obtain results in a timely manner. Hadoop is not used at Google, though I assure you they use big data analytics. Google uses their own systems, developed in C++. In fact, Hadoop was created as a result of Google publishing their Map/Reduce and BigTable (HBase in Hadoop) white papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data scientists will interface with hadoop engineers, though at smaller places you may be required to wear both hats. If you are strictly a data scientist, then whatever you use for your analytics, R, Excel, Tableau, etc, will operate only on a small subset, then will need to be converted to run against the full data set involving hadoop. &lt;/p&gt;&#xA;" OwnerUserId="602" LastActivityDate="2014-06-10T20:40:25.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="295" PostTypeId="2" ParentId="289" CreationDate="2014-06-10T20:43:28.533" Score="5" Body="&lt;p&gt;I am glad you also found Yann LeCun's AMA page, it's very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my opinions&lt;br&gt;&#xA;Q: Should I take some intro software engineering courses at my local University to make myself a stronger candidate?&lt;br&gt;&#xA;A: No, you need to take more math courses.  It's not the applied stuff that's hard, it's the theory stuff.  I don't know what your school offers.  Take theoretical math courses, along with some computer science courses.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q:What other advice you have for someone applying to PhD programs from outside the CS field?&lt;br&gt;&#xA;A:  How closely related are you looking for.  Without a specific question, it's hard to give a specific answer.  &lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2014-06-10T20:43:28.533" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="296" PostTypeId="2" ParentId="128" CreationDate="2014-06-10T21:50:51.347" Score="44" Body="&lt;p&gt;HDP is an extension of LDA, designed to address the case where the number of mixture components (the number of &quot;topics&quot; in document-modeling terms) is not known a priori.  So that's the reason why there's a difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using LDA for document modeling, one treats each &quot;topic&quot; as a distribution of words in some known vocabulary.  For each document a mixture of topics is drawn from a Dirichlet distribution, and then each word in the document is an independent draw from that mixture (that is, selecting a topic and then using it to generate a word).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For HDP (applied to document modeling), one also uses a Dirichlet process to capture the uncertainty in the number of topics.  So a common base distribution is selected which represents the countably-infinite set of possible topics for the corpus, and then the finite distribution of topics for each document is sampled from this base distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as pros and cons, HDP has the advantage that the maximum number of topics can be unbounded and learned from the data rather than specified in advance.  I suppose though it is more complicated to implement, and unnecessary in the case where a bounded number of topics is acceptable.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2014-06-10T21:50:51.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="297" PostTypeId="2" ParentId="130" CreationDate="2014-06-10T22:26:53.623" Score="7" Body="&lt;p&gt;As in @damienfrancois answer feature selection is about selecting a subset of features. So in NLP it would be selecting a set of specific words (the typical in NLP is that each word represents a feature with value equal to the frequency of the word or some other weight based on TF/IDF or similar).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dimensionality reduction is the introduction of new feature space where the original features are represented. The new space is of lower dimension that the original space. In case of text an example would be the &lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_hashing&quot; rel=&quot;nofollow&quot;&gt;hashing trick&lt;/a&gt; where a piece of text is reduced to a vector of few bits (say 16 or 32) or bytes. The amazing thing is that the geometry of the space is preserved (given enough bits), so relative distances between documents remain the same as in the original space, so you can deploy standard machine learning techniques without having to deal with unbound (and huge number of) dimensions found in text.&lt;/p&gt;&#xA;" OwnerUserId="418" LastEditorUserId="381" LastEditDate="2015-10-20T03:28:37.247" LastActivityDate="2015-10-20T03:28:37.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="298" PostTypeId="2" ParentId="289" CreationDate="2014-06-10T22:29:52.873" Score="7" Body="&lt;p&gt;You already have a Masters in Statistics, which is great! In general, I'd suggest to people to take as much statistics as they can, especially Bayesian Data Analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on what you want to do with your PhD, you would benefit from foundational courses in the discipline(s) in your application area.  You already have Economics but if you want to do Data Science on social behavior, then courses in Sociology would be valuable.  If you want to work in fraud prevention, then a courses in banking and financial transactions would be good.  If you want to work in information security, then taking a few security courses would be good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are people who argue that it's not valuable for Data Scientists to spend time on courses in sociology or other disciplines.  But consider the recent case of the Google Flu Trends project.  In &lt;a href=&quot;http://www.uvm.edu/~cdanfort/csc-reading-group/lazer-flu-science-2014.pdf&quot;&gt;this article&lt;/a&gt; their methods were strongly criticized for making avoidable mistakes.  The critics call it &quot;Big Data hubris&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's another reason for building strength in social science disciplines: personal competitive advantage.  With the rush of academic degree programs, certificate programs, and MOOCs, there is a mad rush of students into the Data Science field.  Most will come out with capabilities for core Machine Learning methods and tools.  PhD graduates will have more depth and more theoretical knowledge, but they are all competing for the same sorts of jobs, delivering the same sorts of value.  With this flood of graduates, I expect that they won't be able to command premium salaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you can differentiate yourself with a combination of formal education and practical experience in a particular domain and application area, then you should be able to set yourself apart from the crowd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Context: I'm in a PhD program in Computational Social Science, which has a heavy focus on modeling, evolutionary computation, and social science disciplines, and less emphasis on ML and other empirical data analysis topics).&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-10T22:29:52.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="300" PostTypeId="2" ParentId="103" CreationDate="2014-06-10T23:20:16.670" Score="4" Body="&lt;p&gt;Topological Data Analysis is a method explicitly designed for the setting you describe. Rather than a global distance metric, it relies only on a local metric of proximity or neighborhood. See: &lt;a href=&quot;http://www.ams.org/bull/2009-46-02/S0273-0979-09-01249-X/S0273-0979-09-01249-X.pdf&quot; rel=&quot;nofollow&quot;&gt;Topology and data&lt;/a&gt; and &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3566620/&quot; rel=&quot;nofollow&quot;&gt;Extracting insights from the shape of complex data using topology&lt;/a&gt;. You can find additional resources at the website for Ayasdi.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-10T23:20:16.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="301" PostTypeId="2" ParentId="52" CreationDate="2014-06-11T00:32:54.887" Score="11" Body="&lt;p&gt;One reason that data cleaning is rarely fully automated is that there is so much judgment required to define what &quot;clean&quot; means given your particular problem, methods, and goals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be as simple as imputing values for any missing data, or it might be as complex as diagnosing data entry errors or data transformation errors from previous automated processes (e.g. coding, censoring, transforming).  In these last two cases, the data &lt;em&gt;looks good&lt;/em&gt; by outward appearance but it's really erroneous.  Such diagnosis often requires manual analysis and inspection, and also out-of-band information such as information about the data sources and methods they used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, some data analysis methods work better when erroneous or missing data is left blank (or N/A) rather than imputed or given a default value.  This is true when there is explicit representations of uncertainty and ignorance, such as Dempster-Shafer Belief functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, it's useful to have specific diagnostics and metrics for the cleaning process.  Are missing or erroneous values randomly distributed or are they concentrated in any way that might affect the outcome of the analysis.  It's useful to test the effects of alternative cleaning strategies or algorithms to see if they affect the final results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given these concerns, I'm very suspicious of any method or process that treats data cleaning in a superficial, cavalier or full-automated fashion.  There are many devils hiding in those details and it pays to give them serious attention.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-11T00:32:54.887" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="302" PostTypeId="2" ParentId="280" CreationDate="2014-06-11T00:44:20.517" Score="3" Body="&lt;p&gt;I've seen a few similar systems over the years.  I remember a company called ClickTrax which if I'm not mistaken got bought by Google and some of their features are now part of Google Analytics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their purpose was marketing, but the same concept can be applied to user experience analytics.  The beauty of their system was that what was tracked was defined by the webmaster - in your case the application developer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine as an application developer I would want to be able to see statistical data on two things - task accomplishment, and general feature usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of task accomplishment, I might have 3 ways to print a page - Ctrl+P, File-&gt;Print, and a toolbar button.  I would want to be able to compare usage to see if the screenspace utilized by the toolbar button was actually worth it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of general feature usage, I would want to define a set of features within my application and focus my development efforts on expanding the features used most by my end users.  Some features that take maybe 5 clicks and are popular, I might want to provide a hotkey for, or slim down the number of clicks to activate that feature.  There is also event timing.  Depending on the application, I might want to know the average amount of time spent on a particular feature.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing I would want to look at are click streams.  How are people getting from point A to point B in my application?  What are the most popular point B's?  What are the most popular starting points?  &lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-11T00:44:20.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="303" PostTypeId="2" ParentId="224" CreationDate="2014-06-11T01:09:06.100" Score="3" Body="&lt;p&gt;This isn't my area of specialty and I'm not familiar with Moses, but I found this after some searching.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you are looking for GIZA++.  You'll see GIZA++ listed in the &quot;Training&quot; section (left menu) on the Moses home page, as the second step. GIZA++ is briefly described in tutorial fashion &lt;a href=&quot;https://stackoverflow.com/questions/5752043/is-there-a-tutorial-about-giza&quot;&gt;here&lt;/a&gt;. Here are a few tutorial PowerPoint slides: &lt;a href=&quot;http://www.tc.umn.edu/~bthomson/wordalignment/GIZA.ppt&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.tc.umn.edu/~bthomson/wordalignment/GIZA.ppt&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-06-11T01:09:06.100" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="305" PostTypeId="1" AcceptedAnswerId="309" CreationDate="2014-06-11T04:24:04.183" Score="12" ViewCount="3096" Body="&lt;p&gt;There is plenty of hype surrounding Hadoop and its eco-system.  However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use &lt;a href=&quot;http://aws.amazon.com/redshift/&quot;&gt;Amazon RedShift&lt;/a&gt; for querying large data sets, rather than spending time and effort building a Hadoop cluster? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how does Amazon Redshift compare with Hadoop with respect to setup complexity, cost, and performance?&lt;/p&gt;&#xA;" OwnerUserId="534" LastEditorUserId="434" LastEditDate="2014-06-11T15:02:46.890" LastActivityDate="2015-01-28T18:42:06.763" Title="Does Amazon RedShift replace Hadoop for ~1XTB data?" Tags="|apache-hadoop|map-reduce|aws|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="306" PostTypeId="2" ParentId="305" CreationDate="2014-06-11T05:17:12.253" Score="3" Body="&lt;p&gt;Personally, I don't think it's all that difficult to set up a hadoop cluster, but I know that it is sometimes painful when you are getting started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HDFS size limitations well exceed a TB (or did you mean exabyte?).  If I'm not mistaken it scales to yottabytes or some other measurement that I don't even know the word for.  Whatever it is, it's really big.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tools like Redshift have their place, but I always worry about vendor specific solutions.  My main concern is always &quot;what do I do when I am dissatisfied with their service?&quot; - I can go to google and shift my analysis work into their paradigm or I can go to hadoop and shift that same work into that system.  Either way, I'm going to have to learn something new and do a lot of work translating things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, it's nice to be able to upload a dataset and get to work quickly - especially if what I'm doing has a short lifecycle.  Amazon has done a good job of answering the data security problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to avoid hadoop, there will always be an alternative.  But it's not all that difficult to work with once you get going with it.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-11T05:17:12.253" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="307" PostTypeId="1" CreationDate="2014-06-11T06:07:45.767" Score="14" ViewCount="3573" Body="&lt;p&gt;I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What kinda data these companies used. What was the size of the data&lt;/li&gt;&#xA;&lt;li&gt;What kinda of tools technologies they used to process the data&lt;/li&gt;&#xA;&lt;li&gt;What was the problem they were facing and how the insight they got the data helped them to resolve the issue.&lt;/li&gt;&#xA;&lt;li&gt;How they selected the tool\technology to suit their need.&lt;/li&gt;&#xA;&lt;li&gt;What kinda pattern they identified from the data &amp;amp; what kind of patterns they were looking from the data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I wonder if someone can provide me answer to all these questions or a link which at-least answer some of the the questions. I am looking for real world example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great if someone share how finance industry is making use of Big Data Analytic.&lt;/p&gt;&#xA;" OwnerUserId="496" LastEditorUserId="3151" LastEditDate="2016-08-17T10:41:41.383" LastActivityDate="2020-08-16T16:54:32.553" Title="Big data case study or use case example" Tags="|data-mining|bigdata|usecase|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="308" PostTypeId="2" ParentId="307" CreationDate="2014-06-11T06:49:04.070" Score="14" Body="&lt;p&gt;News outlets tend to use &quot;Big Data&quot; pretty loosely.  Vendors usually provide case studies surrounding their specific products.  There aren't a lot out there for open source implementations, but they do get mentioned.  For instance, Apache isn't going to spend a lot of time building a case study on hadoop, but vendors like Cloudera and Hortonworks probably will.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an &lt;a href=&quot;http://www.cloudera.com/content/cloudera/en/resources/library/casestudy/joint-success-story-major-retail-bank-case-study-datameer.html&quot; rel=&quot;noreferrer&quot;&gt;example case study from Cloudera&lt;/a&gt; in the finance sector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting the study:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One major global financial services conglomerate uses Cloudera and Datameer to help&#xA;  identify rogue trading activity. Teams within the firm’s asset management group are&#xA;  performing ad hoc analysis on daily feeds of price, position, and order information. Having&#xA;  ad hoc analysis to all of the detailed data allows the group to detect anomalies across&#xA;  certain asset classes and identify suspicious behavior. Users previously relied solely on&#xA;  desktop spreadsheet tools. Now, with Datameer and Cloudera, users have a powerful&#xA;  platform that allows them to sift through more data more quickly and avert potential&#xA;  losses before they begin.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A leading retail bank is using Cloudera and Datameer to validate data accuracy and quality&#xA;  as required by the Dodd-Frank Act and other regulations. Integrating loan and branch data&#xA;  as well as wealth management data, the bank’s data quality initiative is responsible for&#xA;  ensuring that every record is accurate. The process includes subjectingthe data to over&#xA;  50 data sanity and quality checks. The results of those checks are trended over time to&#xA;  ensure that the tolerances for data corruption and data domains aren’t changing adversely&#xA;  and that the risk profiles being reported to investors and regulatory agencies are prudent&#xA;  and in compliance with regulatory requirements. The results are reported through a data&#xA;  quality dashboard to the Chief Risk Officer and Chief Financial Officer, who are ultimately&#xA;  responsible for ensuring the accuracy of regulatory compliance reporting as well as&#xA;  earnings forecasts to investors&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I didn't see any other finance related studies at Cloudera, but I didn't search very hard.  You can have a look at &lt;a href=&quot;http://www.cloudera.com/content/cloudera/en/resources/library.html?q=bank&quot; rel=&quot;noreferrer&quot;&gt;their library&lt;/a&gt; here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Hortonworks has a &lt;a href=&quot;http://hortonworks.com/blog/building-stock-trading-strategies-20-faster-with-hadoop/&quot; rel=&quot;noreferrer&quot;&gt;case study on Trading Strategies&lt;/a&gt; where they saw a 20% decrease in the time it took to develop a strategy by leveraging K-means, Hadoop, and R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/qX3Qy.png&quot; alt=&quot;Each color indicates a group of strategies with similar probability of a profit and loss&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Sj8U9.png&quot; alt=&quot;how the trading system was improved by using Hadoop (Hortonworks Data Platform), and the k-means algorithm&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These don't answer all of your questions.  I'm pretty sure both of these studies covered most of them. I don't see anything about tool selection specifically.  I imagine sales reps had a lot to do with getting the overall product in the door, but the data scientists themselves leveraged the tools they were most comfortable with.  I don't have a lot of insight into that area in the big data space.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-06-11T06:54:42.593" LastActivityDate="2014-06-11T06:54:42.593" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="309" PostTypeId="2" ParentId="305" CreationDate="2014-06-11T06:51:19.143" Score="12" Body="&lt;p&gt;&lt;strong&gt;tl;dr:&lt;/strong&gt; They markedly differ in many aspects and I can't think Redshift will replace Hadoop.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Function&lt;br&gt;&#xA;You can't run anything other than SQL on Redshift. Perhaps most importantly, you can't run any type of custom functions on Redshift. In Hadoop you can, using many languages (Java, Python, Ruby.. you name it). For example, NLP in Hadoop is easy, while it's more or less impossible in Redshift. I.e. there are lots of things you can do in Hadoop but not on Redshift. This is probably the most important difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Performance Profile&lt;br&gt;&#xA;Query execution on Redshift is in most cases significantly more efficient than on Hadoop. However, this efficiency comes from the indexing that is done when the data is loaded into Redshift (I'm using the term &lt;code&gt;indexing&lt;/code&gt; very loose here). Therefore, it's great if you load your data once and execute multiple queries, but if you want to execute only one query for example, you might actually lose out in performance overall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Cost Profile&lt;br&gt;&#xA;Which solution wins out in cost depends on the situation (like performance), but you probably need quite a lot of queries in order to make it cheaper than Hadoop (more specifically Amazon's Elastic Map Reduce). For example, if you are doing OLAP, it's very likely that Redshift comes out cheaper. If you do daily batch ETLs, Hadoop is more likely to come out cheaper.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, we've replaced part of our ETL that was done in Hive to Redshift, and it was a pretty great experience; mostly for the ease of development. Redshift's Query Engine is based on PostgreSQL and is very mature, compared to Hive's. Its ACID characteristics make it easier to reason about it, and the quicker response time allows more testing to be done. It's a great tool to have, but it won't replace Hadoop.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:  As for setup complexity, I'd even say it's easier with Hadoop if you use AWS's EMR. Their tools are so mature that it's ridiculously easy to have your Hadoop job running. Tools and mechanisms surrounding Redshift's operation aren't that mature yet. For example, Redshift can't handle trickle loading and thus you have to come up with something that turns that into a batched load, which can add some complexity to your ETL. &lt;/p&gt;&#xA;" OwnerUserId="638" LastEditorUserId="638" LastEditDate="2014-06-11T09:07:33.570" LastActivityDate="2014-06-11T09:07:33.570" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="310" PostTypeId="1" CreationDate="2014-06-11T10:11:59.397" Score="17" ViewCount="2913" Body="&lt;p&gt;I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are about 1,150 known &quot;positives&quot;, against a background of about 13 million protein sequences (&quot;Unknown/poorly annotated background&quot;), or about 100,000 reviewed, relevant proteins, annotated with a variety of properties (but very few annotated in an explicitly &quot;negative&quot; way). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My previous implementation looked at this as a binary classification problem: &#xA;Positive set = Proteins marked as Neuropeptides.&#xA;Negative set: Random sampling of 1,300 samples (total) from among the remaining proteins of a roughly similar length-wise distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That worked, but I want to greatly improve the machines discriminatory abilities (Currently, it's at about 83-86% in terms of accuracy, AUC, F1, measured by CV, on multiple randomly sampled negative sets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts were to:&#xA;1) Make this a multiclass problem, choosing 2-3 different classes of protein that will definetly be negatives, by their properties/functional class, along with (maybe) another randomly sampled set.&#xA; (Priority here would be negative sets that are similar in their characteristics/features to the positive set, while still having defining characteristics) . &#xA;2) One class learning - Would be nice, but as I understand it, it's meant just for anomaly detection, and has poorer performance than discriminatory approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*) I've heard of P-U learning, which sounds neat, but I'm a programming N00b, and I don't know of any existing implementations for it. (In Python/sci-kit learn).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, does approach 1 make sense in a theoretical POV? Is there a best way to make multiple negative sets? (I could also simply use a massive [50K] pick of the &quot;negative&quot; proteins, but they're all very very different from each other, so I don't know how well the classifier would handle them as one big , unbalanced mix).&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="555" LastActivityDate="2020-08-16T13:00:45.530" Title="One-Class discriminatory classification with imbalanced, heterogenous Negative background?" Tags="|machine-learning|data-mining|python|classification|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="311" PostTypeId="2" ParentId="310" CreationDate="2014-06-11T11:24:19.963" Score="5" Body="&lt;p&gt;The way I would attack the problem, in general, is to leverage statistical analysis like Principal Component Analysis or Ordinary Least Squares to help determine what attributes within these protein sequences are best suited to classify proteins as Neuropeptide hormone precursors.&lt;/p&gt;&#xA;&lt;p&gt;In order to do that, you'll have to convert the protein sequences into numeric data, but I believe some work has already been done in that regard using formulas leveraged in Amino Acid PCA.&lt;/p&gt;&#xA;&lt;p&gt;See these two links:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/24496727&quot; rel=&quot;nofollow noreferrer&quot;&gt;Principal components analysis of protein sequence clusters&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/16615809&quot; rel=&quot;nofollow noreferrer&quot;&gt;Amino Acid Principal Component Analysis (AAPCA) and its applications in protein structural class prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Once that work has been done, I would attempt to classify using the entire dataset and a reinforcement learning algorithm, like &lt;a href=&quot;http://findingscience.com/ankusa/hbase/hadoop/ruby/2010/12/02/naive-bayes-classification-in-ruby-using-hadoop-and-hbase.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; while slimming down the data into that which PCA has identified as important.&lt;/p&gt;&#xA;&lt;p&gt;The reason I would try to use Bayes is that it has proven to be one of the best methods for determining spam vs. regular email, which has a similarly skewed dataset.&lt;/p&gt;&#xA;&lt;p&gt;Having said all of that...&lt;/p&gt;&#xA;&lt;p&gt;Slimming down the number or type of negative classifications might skew your results a few points one way or the other, but I don't think you'll see the long term effectiveness change substantially until you do the leg work of determining how to best remove the fuzziness from your training data. That will either require a field expert or statistical analysis.&lt;/p&gt;&#xA;&lt;p&gt;I could be completely off base. I am interested in seeing some other answers, but that is my 2 cents.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="50406" LastEditDate="2020-08-16T13:00:45.530" LastActivityDate="2020-08-16T13:00:45.530" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="312" PostTypeId="2" ParentId="215" CreationDate="2014-06-11T11:32:13.433" Score="5" Body="&lt;p&gt;Therriault, really happy to hear you are using Vertica! Full disclosure, I am the chief data scientist there :) . The workflow you describe is exactly what I encounter quite frequently and I am a true believer in preprocessing those very large datasets in the database prior to any pyODBC and pandas work. I'd suggest creating a view or table via a file based query just to ensure reproducible work. Good Luck&lt;/p&gt;&#xA;" OwnerUserId="655" LastActivityDate="2014-06-11T11:32:13.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="313" PostTypeId="1" CreationDate="2014-06-11T13:28:35.980" Score="29" ViewCount="3451" Body="&lt;p&gt;What are the books about the science and mathematics behind data science? It feels like so many &quot;data science&quot; books are programming tutorials and don't touch things like data generating processes and statistical inference. I can already code, what I am weak on is the math/stats/theory behind what I am doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am ready to burn $1000 on books (so around 10 books... sigh), what could I buy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples: Agresti's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470463635&quot;&gt;Categorical Data Analysis&lt;/a&gt;, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1441902996&quot;&gt;Linear Mixed Models for Longitudinal Data&lt;/a&gt;, etc... etc...&lt;/p&gt;&#xA;" OwnerUserId="663" LastEditorUserId="11097" LastEditDate="2016-02-21T04:02:38.847" LastActivityDate="2016-02-21T04:02:38.847" Title="Books about the &quot;Science&quot; in Data Science?" Tags="|statistics|reference-request|" AnswerCount="4" CommentCount="5" ClosedDate="2014-06-26T02:01:00.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="314" PostTypeId="2" ParentId="313" CreationDate="2014-06-11T13:49:08.970" Score="14" Body="&lt;p&gt;If I could only recomend one to you, it would be: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot;&gt;The Elements of Statistical Learning and Prediction&lt;/a&gt; by Hastie, Tibshirani and Friedman.  It provides the math/statistics behind a lot of commonly used techniques in data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Bayesian Techniques, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1439840954&quot;&gt;Bayesian Data Analysis&lt;/a&gt; by Gelman, Carlin, Stern, Dunson, Vehtari and Rubin is excellent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534243126&quot;&gt;Statistical Inference&lt;/a&gt; by Casella and Berger is a good graduate-level textbook on the theoretical foundation of statistics.  This book does require a pretty high level of comfort with math (probability theory is based on measure theory, which is not trivial to understand).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With respect to data generating processes, I don't have a recommendation for a book.  What I can say is that a good understanding of the assumptions of the techniques used and ensuring that the data was collected or generated in a manner that does not violate those assumptions goes a long way towards a good analysis.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-06-11T13:49:08.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="316" PostTypeId="2" ParentId="59" CreationDate="2014-06-11T16:25:34.747" Score="10" Body="&lt;p&gt;As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and performs the operations, thus overcoming the memory constraints.  RHadoop (&lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop/wiki&quot;&gt;https://github.com/RevolutionAnalytics/RHadoop/wiki&lt;/a&gt;) is the connector you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to the impact on algorithms which are computation intensive, Random Forests/Decision Trees/Ensemble methods on a considerable amount of data (minimum 50,000 observations in my experience) take up a lot of memory and are considerably slow. To speed up the process, parallelization is the way to go and parallelization is inherently available in Hadoop! That's where, Hadoop is really efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you are going for ensemble methods which are compute intensive and are slow, you would want to try out on the HDFS system which gives a considerable performance improvement.&lt;/p&gt;&#xA;" OwnerUserId="514" LastActivityDate="2014-06-11T16:25:34.747" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="317" PostTypeId="2" ParentId="266" CreationDate="2014-06-11T16:56:59.660" Score="2" Body="&lt;p&gt;If you already know R Studio, then the caret package is a good place to start. Here are some tutorials: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://class.coursera.org/predmachlearn-002&quot; rel=&quot;nofollow&quot;&gt;https://class.coursera.org/predmachlearn-002&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://caret.r-forge.r-project.org/index.html&quot; rel=&quot;nofollow&quot;&gt;http://caret.r-forge.r-project.org/index.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With R and caret you can easily load and splice data sets, feature reduction, principal component analysis, and train and predict using various algorithms.&lt;/p&gt;&#xA;" OwnerUserId="680" LastActivityDate="2014-06-11T16:56:59.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="318" PostTypeId="2" ParentId="266" CreationDate="2014-06-11T17:24:53.610" Score="2" Body="&lt;p&gt;If you can reproduce the 6x3 grid of graphs from the banner of the &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/&lt;/a&gt; page then you will have learnt some ML and some Python. You didn't mention a language. Python is easy enough to learn very quickly, and scikit-learn has a wide range of algorithms implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then try on your own data!&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-06-11T17:24:53.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="319" PostTypeId="1" CreationDate="2014-06-11T18:22:36.267" Score="10" ViewCount="2191" Body="&lt;p&gt;I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've implemented gradient checking, double checked everything etc and I'm pretty certain it's working correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've run it a few times and it reaches 'Optimization terminated successfully' however when I increase the number of hidden layers, the cost of the hypothesis increases (everything else is kept the same) after it has successfully terminated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively it feels as if the cost should decrease when the number of hidden layers is increased, as it is able to generate a more complex hypothesis which can fit the data better, however this appears not to be the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be interested to understand what's going on here, or if I've implemented neural net incorrectly?&lt;/p&gt;&#xA;" OwnerUserId="691" LastEditorUserId="381" LastEditDate="2014-06-12T16:43:59.513" LastActivityDate="2014-06-12T18:08:07.507" Title="Debugging Neural Networks" Tags="|machine-learning|python|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="320" PostTypeId="2" ParentId="319" CreationDate="2014-06-11T20:34:51.873" Score="9" Body="&lt;p&gt;There are so many ways to go wrong with a neural net that it's going to be difficult to debug. Also, to address your intuition, each additional hidden layer makes learning much harder. With that said, here are some possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You have added weight decay. Adding more layers adds more weights which increases your regularization cost.&lt;/li&gt;&#xA;&lt;li&gt;The problem is simple enough that a model with a single hidden layer is sufficient. Adding more hidden layers makes it harder for the network to learn (harder optimization problem).&lt;/li&gt;&#xA;&lt;li&gt;The optimization method is not doing a great job (I prefer climin to scipy.optimize).&lt;/li&gt;&#xA;&lt;li&gt;You are using the sigmoid/tanh activation function. The sigmoid function causes the vanishing gradient problem which makes learning hard with more layers. Try using the ReLu function.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Training neural nets takes a lot of practice, luck, and patience. Good luck.&lt;/p&gt;&#xA;" OwnerUserId="574" LastActivityDate="2014-06-11T20:34:51.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="323" PostTypeId="1" CreationDate="2014-06-11T23:52:37.823" Score="6" ViewCount="9929" Body="&lt;p&gt;The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a decision threshold over different values and computing the effect of that change on the true positive rate and the false positives rate of predictions. What's that decision threshold in the case of a simple fixed decision tree? &lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2021-02-18T20:33:30.027" Title="How can we calculate AUC for a simple decision tree?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="324" PostTypeId="1" CreationDate="2014-06-12T03:11:00.033" Score="-2" ViewCount="624" Body="&lt;p&gt;I would like to extract news about a company from online news by using the RODBC package in R. I would then like to use the extracted data for sentiment analysis. I want to accomplish this in such a way that the positive news is assigned a value of +1, the negative news is assigned a value of -1, and the neutral news is assigned a value of 0.&lt;/p&gt;&#xA;" OwnerUserId="714" LastEditorUserId="29169" LastEditDate="2019-06-11T14:43:33.343" LastActivityDate="2019-06-11T14:43:33.343" Title="How can I extract news about a particular company from various websites using RODBC package in R? And perform sentiment analysis on the data?" Tags="|r|text-mining|sentiment-analysis|" AnswerCount="1" CommentCount="2" ClosedDate="2014-06-26T01:43:32.487" ContentLicense="CC BY-SA 4.0" />
  <row Id="325" PostTypeId="2" ParentId="324" CreationDate="2014-06-12T05:17:46.453" Score="2" Body="&lt;p&gt;This isn't a question with a simple answer, so all I can really do is point you in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The RODBC package isn't meant to extract data online, it's meant to pull data from a database.  If you will be leveraging that package, it will be after you pull data down from the web.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jeffrey Bean put together a &lt;a href=&quot;http://jeffreybreen.wordpress.com/2011/07/04/twitter-text-mining-r-slides/&quot; rel=&quot;nofollow&quot;&gt;slideshow tutorial&lt;/a&gt; for doing sentiment analysis with Twitter data a few years back.  He used the Twitter stream as well as some data pulled in from web scraping.  It's a good starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also this site that discusses a few different approaches to this problem in detail, including Bean's, the sentiment package, and ViralHeat (which is a commercial sentiment analysis service who's data you can pull into R).  Sentiment has since been removed (&lt;a href=&quot;http://cran.us.r-project.org/src/contrib/Archive/sentiment/&quot; rel=&quot;nofollow&quot;&gt;archived versions here&lt;/a&gt;), but the &lt;a href=&quot;http://cran.us.r-project.org/web/packages/qdap/index.html&quot; rel=&quot;nofollow&quot;&gt;qdap package&lt;/a&gt; is available and is designed for use in transcript analysis.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-12T05:17:46.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="326" PostTypeId="1" CreationDate="2014-06-12T06:04:48.243" Score="127" ViewCount="120129" Body="&lt;p&gt;I'm just starting to develop a &lt;a href=&quot;https://en.wikipedia.org/wiki/Machine_learning&quot; rel=&quot;noreferrer&quot;&gt;machine learning&lt;/a&gt; application for academic purposes. I'm currently using &lt;strong&gt;R&lt;/strong&gt; and training myself in it. However, in a lot of places, I have seen people using &lt;strong&gt;Python&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are people using in academia and industry, and what is the recommendation?&lt;/p&gt;&#xA;" OwnerUserId="721" LastEditorUserId="29169" LastEditDate="2019-06-10T15:56:58.013" LastActivityDate="2022-07-11T21:50:49.357" Title="Python vs R for machine learning" Tags="|machine-learning|r|python|" AnswerCount="14" CommentCount="3" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 4.0" />
  <row Id="327" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T07:05:05.653" Score="26" Body="&lt;p&gt;There is nothing like &quot;python is better&quot; or &quot;R is much better than x&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only fact I know is that in the industry allots of people stick to python because that is what they learned at the university. The python community is really active and have a few great frameworks for ML and data mining etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to be honest, if you get a good c programmer he can do the same as people do in python or r, if you got a good java programmer he can also do (near to) everything in java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So just stick with the language you are comfortable with.&lt;/p&gt;&#xA;" OwnerUserId="115" LastActivityDate="2014-06-12T07:05:05.653" CommentCount="8" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="328" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T08:30:49.757" Score="7" Body="&lt;p&gt;There is no &quot;better&quot; language. I have tried both of them and I am comfortable with Python so I work with Python only. Though I am still learning stuff, but I haven't encounter any roadblock with Python till now. The good thing about Python is community is too good and you can get a lot of help on the Internet easily. Other than that, I would say go with the language you like not the one people recommend. &lt;/p&gt;&#xA;" OwnerUserId="456" LastActivityDate="2014-06-12T08:30:49.757" CommentCount="0" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="330" PostTypeId="2" ParentId="235" CreationDate="2014-06-12T09:57:39.890" Score="4" Body="&lt;p&gt;You can also checkout the seaborn package for statistical charts.&lt;/p&gt;&#xA;" OwnerUserId="729" LastActivityDate="2014-06-12T09:57:39.890" CommentCount="0" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="331" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T10:09:23.887" Score="19" Body="&lt;p&gt;Some additional thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The programming language 'per se' is only a tool. All languages were designed to make some type of constructs more easy to build than others. And the knowledge and mastery of a programming language is more important and effective than the features of that language compared to others.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can see there are two dimensions of this question. The first dimension is the ability to explore, build proof of concepts or models at a fast pace, eventually having at hand enough tools to study what is going on (like statistical tests, graphics, measurement tools, etc). This kind of activity is usually preferred by researchers and data scientists (I always wonder what that means, but I use this term for its loose definition). They tend to rely on well-known and verified instruments, which can be used for proofs or arguments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second dimension is the ability to extend, change, improve or even create tools, algorithms or models. In order to achieve that you need a proper programming language. Roughly all of them are the same. If you work for a company, than you depend a lot on the company's infrastructure, internal culture and your choices diminish significantly. Also, when you want to implement an algorithm for production use, you have to trust the implementation. And implementing in another language which you do not master will not help you much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to favor for the first type of activity the R ecosystem. You have a great community, a huge set of tools, proofs that these tools works as expected. Also, you can consider Python, Octave (to name a few), which are reliable candidates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second task, you have to think before at what you really want. If you want robust production ready tools, then C/C++, Java, C# are great candidates. I consider Python as a second citizen in this category, together with Scala and friends. I do not want to start a flame war, it's my opinion only. But after more than 17 years as a developer, I tend to prefer a strict contract and my knowledge, than the freedom to do whatever you might think of (like it happens with a lot of dynamic languages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I want to learn as much as possible. I decided that I have to choose the hard way, which means to implement everything from scratch myself. I use R as a model and inspiration. It has great treasures in libraries and a lot of experience distilled. However, R as a programming language is a nightmare for me. So I decided to use Java, and use no additional library. That is only because of my experience, and nothing else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have time, the best thing you can do is to spend some time with all these things. In this way you will earn for yourself the best answer possible, fitted for you. Dijkstra said once that the tools influence the way you think, so it is advisable to know your tools before letting them to model how you think. You can read more about that in his famous paper called &lt;a href=&quot;http://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD340.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Humble Programmer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="62609" LastEditDate="2018-11-14T18:58:00.230" LastActivityDate="2018-11-14T18:58:00.230" CommentCount="0" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 4.0" />
  <row Id="332" PostTypeId="2" ParentId="323" CreationDate="2014-06-12T10:27:14.480" Score="4" Body="&lt;p&gt;In order to build the ROC curve and AUC (Area under curve) you have to have a binary classifier which provides you at classification time, the distribution (or at least a score), not the classification label. To give you an example, suppose you have a binary classification model, with classes &lt;span class=&quot;math-container&quot;&gt;$c1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$c2$&lt;/span&gt;. For a given instance, your classifier would have to return a score for &lt;span class=&quot;math-container&quot;&gt;$c1$&lt;/span&gt; and another for &lt;span class=&quot;math-container&quot;&gt;$c2$&lt;/span&gt;. If this score is a probability-like (preferrable), than something like &lt;span class=&quot;math-container&quot;&gt;$p(c1)$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$p(c2)$&lt;/span&gt; would work. In plain English is translated like &amp;quot;I (the model) classify this instance as &lt;span class=&quot;math-container&quot;&gt;$c1$&lt;/span&gt; with probability &lt;span class=&quot;math-container&quot;&gt;$p(c1)$&lt;/span&gt;, and as &lt;span class=&quot;math-container&quot;&gt;$c2$&lt;/span&gt; with probability  &lt;span class=&quot;math-container&quot;&gt;$p(c2)=1-p(c1)$&lt;/span&gt;&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;This applies for all type of classifiers, not only for decision trees. Having these scores you can than compute ROC or AUC by varying a threshold on &lt;span class=&quot;math-container&quot;&gt;$p(c1)$&lt;/span&gt; values, from the smallest to the greatest value.&lt;/p&gt;&#xA;&lt;p&gt;Now, if you have an implementation of a decision tree and you want to change that implementation to return scores instead of labels you have to compute those values. The most used way for decision trees is to use the proportion of classes from the leaf nodes. So, for example you have built a decision tree and when you predict the class for an instance you arrive at a leaf node which have (stored from the learning phase) 10 instances of class &lt;span class=&quot;math-container&quot;&gt;$c1$&lt;/span&gt; and 15 instances of class &lt;span class=&quot;math-container&quot;&gt;$c2$&lt;/span&gt;, you can use the ratios as the scores. So, in our example, you would return &lt;span class=&quot;math-container&quot;&gt;$p(c1) = 10 / (10+15) = 0.4$&lt;/span&gt; probability of class &lt;span class=&quot;math-container&quot;&gt;$c1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$p(c2) = 15/(10+15)=0.6$&lt;/span&gt; probability of being class &lt;span class=&quot;math-container&quot;&gt;$c2$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;For further reading on the ROC curves, the best and inspiring source of information I found to be the Tom Fawcett paper called &lt;a href=&quot;https://www.google.ie/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;ved=0CCsQFjAA&amp;amp;url=https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf&amp;amp;ei=DYCZU96QGPOV7AaAx4CgBA&amp;amp;usg=AFQjCNECLoecin6ieT-0ymLQ--FoMjkMZw&amp;amp;sig2=3bGE2y1N3I2Re_YWn0EDFA&quot; rel=&quot;nofollow noreferrer&quot;&gt;An Introduction to ROC Analysis&lt;/a&gt;, it's solid gold on this topic.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="29169" LastEditDate="2021-02-18T20:33:30.027" LastActivityDate="2021-02-18T20:33:30.027" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="334" PostTypeId="1" AcceptedAnswerId="338" CreationDate="2014-06-12T10:52:03.410" Score="37" ViewCount="19950" Body="&lt;p&gt;I've now seen two data science certification programs - the &lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1?utm_medium=listingPage&quot;&gt;John Hopkins one available at Coursera&lt;/a&gt; and the &lt;a href=&quot;http://cloudera.com/content/cloudera/en/training/certification/ccp-ds.html&quot;&gt;Cloudera one&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure there are others out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The John Hopkins set of classes is focused on R as a toolset, but covers a range of topics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;R Programming&lt;/li&gt;&#xA;&lt;li&gt;cleaning and obtaining data&lt;/li&gt;&#xA;&lt;li&gt;Data Analysis&lt;/li&gt;&#xA;&lt;li&gt;Reproducible Research&lt;/li&gt;&#xA;&lt;li&gt;Statistical Inference&lt;/li&gt;&#xA;&lt;li&gt;Regression Models&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning&lt;/li&gt;&#xA;&lt;li&gt;Developing Data Products&lt;/li&gt;&#xA;&lt;li&gt;And what looks to be a Project based completion task similar to Cloudera's Data Science Challenge&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The Cloudera program looks thin on the surface, but looks to answer the two important questions - &quot;Do you know the tools&quot;, &quot;Can you apply the tools in the real world&quot;.  Their program consists of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Introduction to Data Science&lt;/li&gt;&#xA;&lt;li&gt;Data Science Essentials Exam&lt;/li&gt;&#xA;&lt;li&gt;Data Science Challenge (a real world data science project scenario)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am not looking for a recommendation on a program or a quality comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious about other certifications out there, the topics they cover, and how seriously DS certifications are viewed at this point by the community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: These are all great answers.  I'm choosing the correct answer by votes.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-06-13T11:35:51.697" LastActivityDate="2019-07-11T07:02:42.803" Title="What do you think of Data Science certifications?" Tags="|education|" AnswerCount="13" CommentCount="7" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="335" PostTypeId="2" ParentId="334" CreationDate="2014-06-12T11:11:35.600" Score="12" Body="&lt;p&gt;The certification programs you mentioned are really entry level courses. Personally, I think these certificates show only person's persistence and they can be only useful to those who is applying for internships, not the real data science jobs.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-12T11:11:35.600" CommentCount="1" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="336" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T11:30:20.943" Score="13" Body="&lt;p&gt;There isn't a silver bullet language that can be used to solve each and every data related problem. The language choice depends on the context of the problem, size of data and if you are working at a workplace you have to stick to what they use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I use R more often than Python due to its visualization libraries and interactive style. But if I need more performance or structured code I definitely use Python since it has some of the best libraries as SciKit-Learn, numpy, scipy etc. I use both R and Python in my projects interchangeably. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you are starting on data science work I suggest you to learn both and it's not difficult since Python also provides a similar interface to R with &lt;a href=&quot;http://pandas.pydata.org/&quot;&gt;Pandas&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to deal with much larger datasets, you can't escape eco-systems built with Java(Hadoop, Pig, Hbase etc).&lt;/p&gt;&#xA;" OwnerUserId="733" LastActivityDate="2014-06-12T11:30:20.943" CommentCount="0" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="337" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T11:54:59.140" Score="15" Body="&lt;p&gt;I would add to what others have said till now. There is no single answer that one language is better than other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, R has a better community for data exploration and learning. It has extensive visualization capabilities. Python, on the other hand, has become better at data handling since introduction of pandas. Learning and development time is very less in Python, as compared to R (R being a low level language).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it ultimately boils down to the eco-system you are in and personal preferences. For more details, you can look at this comparison &lt;a href=&quot;http://www.analyticsvidhya.com/blog/2014/03/sas-vs-vs-python-tool-learn/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="735" LastActivityDate="2014-06-12T11:54:59.140" CommentCount="2" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="338" PostTypeId="2" ParentId="334" CreationDate="2014-06-12T12:13:26.940" Score="13" Body="&lt;p&gt;I did the first 2 courses and I'm planning to do all the others too.  If you don't know R, it's a really good program. There are assignments and quizzes every week. Many people find some courses very difficult. You are going to have hard time if you don't have any programming experience (even if they say it's not required). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just remember.. it's not because you can drive a car that you are a F1 pilot ;) &lt;/p&gt;&#xA;" OwnerUserId="737" LastActivityDate="2014-06-12T12:13:26.940" CommentCount="0" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="339" PostTypeId="2" ParentId="326" CreationDate="2014-06-12T12:59:00.663" Score="110" Body="&lt;p&gt;Some real important differences to consider when you are choosing &lt;strong&gt;R&lt;/strong&gt; or &lt;strong&gt;Python&lt;/strong&gt; over one another:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Machine Learning&lt;/strong&gt; has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and &lt;strong&gt;predictions are done realtime&lt;/strong&gt;. The model building process is a compute intensive process while the prediction happens in a jiffy. Therefore, performance of an algorithm in Python or R doesn't really affect the turn-around time of the user. Python 1, R 1.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Production:&lt;/strong&gt; The real difference between Python and R comes in being production ready. Python, as such is a full fledged programming language and many organisations use it in their production systems. R is a statistical programming software favoured by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R. Many of these organisations have their production systems either in Java, C++, C#, Python etc. So, ideally they would like to have the &lt;strong&gt;prediction system&lt;/strong&gt; in the same language to reduce the latency and maintenance issues.&#xA;Python 2, R 1.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Libraries:&lt;/strong&gt; Both the languages have enormous and reliable libraries. R has over 5000 libraries catering to many domains while Python has some incredible packages like &lt;strong&gt;Pandas, NumPy, SciPy, Scikit Learn, Matplotlib&lt;/strong&gt;. Python 3, R 2.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Development:&lt;/strong&gt; Both the language are interpreted languages. Many say that python is easy to learn, it's almost like reading english (to put it on a lighter note) but R requires more initial studying effort. Also, both of them have good IDEs (Spyder etc for Python and RStudio for R). Python 4, R 2.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Speed:&lt;/strong&gt; R software initially had problems with large computations (say, like nxn matrix multiplications). But, this issue is addressed with the introduction of R by Revolution Analytics. They have re-written computation intensive operations in C which is blazingly fast. Python being a high level language is relatively slow. Python 4, R 3.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Visualizations:&lt;/strong&gt; In data science, we frequently tend to plot data to showcase patterns to users. Therefore, visualisations become an important criteria in choosing a software and R completely kills Python in this regard. Thanks to Hadley Wickham for an incredible ggplot2 package. R wins hands down. Python 4, R 4.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Dealing with Big Data:&lt;/strong&gt; One of the constraints of R is it stores the data in system memory (RAM). So, RAM capacity becomes a constraint when you are handling Big Data. Python does well, but I would say, as both R and Python have HDFS connectors, leveraging Hadoop infrastructure would give substantial performance improvement. So, Python 5, R 5.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, both the languages are equally good. Therefore, depending upon your domain and the place you work, you have to smartly choose the right language. The technology world usually prefers using a single language. Business users (marketing analytics, retail analytics) usually go with statistical programming languages like R, since they frequently do quick prototyping and build visualisations (which is faster done in R than Python).&lt;/p&gt;&#xA;" OwnerUserId="514" LastEditorUserId="28021" LastEditDate="2017-01-17T14:14:47.423" LastActivityDate="2017-01-17T14:14:47.423" CommentCount="18" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="340" PostTypeId="2" ParentId="235" CreationDate="2014-06-12T13:38:10.877" Score="5" Body="&lt;p&gt;There's plenty. If you've ever used ggplot2 in R and want to do that in &lt;a href=&quot;https://pypi.python.org/pypi/ggplot/0.5.9&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use a similar visualisation grammar (Vega) and go via &lt;a href=&quot;https://github.com/wrobstory/vincent&quot; rel=&quot;nofollow noreferrer&quot;&gt;D3&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if you want the full-on 3d: &lt;a href=&quot;http://docs.enthought.com/mayavi/mayavi/&quot; rel=&quot;nofollow noreferrer&quot;&gt;shizzle&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="29575" LastEditDate="2019-04-25T18:59:20.977" LastActivityDate="2019-04-25T18:59:20.977" CommentCount="0" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 4.0" />
  <row Id="341" PostTypeId="2" ParentId="253" CreationDate="2014-06-12T13:42:05.383" Score="7" Body="&lt;p&gt;You have to first make it clear what do you mean by &quot;learn Hadoop&quot;. If you mean using Hadoop, such as learning to program in MapReduce, then most probably it is a good idea. But fundamental knowledge (database, machine learning, statistics) may play a bigger role as time goes on.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-06-12T13:42:05.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="343" PostTypeId="2" ParentId="319" CreationDate="2014-06-12T15:22:16.247" Score="3" Body="&lt;p&gt;Increasing the number of hidden layers for a standard neural network actually won't improve results in a majority of cases. Changing the &lt;em&gt;size&lt;/em&gt; of the hidden layer will.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This fact (that the number of hidden layers does very little) has actually was noted historically and is the motivation behind the field of deep learning. Deep learning is effectively clever ways of training multilayer neural networks by, for example, isolating subsets of features when training different layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good introductory video on this topic on &lt;a href=&quot;https://www.youtube.com/watch?v=vXMpKYRhpmI&amp;amp;index=52&amp;amp;list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&quot; rel=&quot;nofollow&quot;&gt;YouTube&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="754" LastEditorUserId="754" LastEditDate="2014-06-12T18:08:07.507" LastActivityDate="2014-06-12T18:08:07.507" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="345" PostTypeId="2" ParentId="313" CreationDate="2014-06-12T16:52:46.557" Score="23" Body="&lt;p&gt;Introductory:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1107422221&quot;&gt;Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1600490069&quot;&gt;Learning From Data (Abu-Mostafa et al.)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1461471370&quot;&gt;Introduction to Statistical Learning (James et al.)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Digging deeper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot;&gt;Elements of Statistical Learning (Hastie et al.)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387310738&quot;&gt;Pattern Recognition and Machine Learning (Bishop)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some special interest examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521833787&quot;&gt;Convex Optimization (Boyd)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucl.ac.uk/staff/d.barber/brml/&quot;&gt;Bayesian Reasoning and Machine Learning (Barber)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262013193&quot;&gt;Probabilistic Graphical Models (Koller)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198538642&quot;&gt;Neural Networks for Pattern Recognition (Bishop)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Broader reference works on machine learning (not really what you asked for, but for completeness):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262018020&quot;&gt;Machine Learning: A Probabilistic Perspective (Murphy)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0136042597&quot;&gt;Artificial Intelligence: A Modern Approach (Russell &amp;amp; Norvig)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Bonus paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://projecteuclid.org/euclid.ss/1009213726&quot;&gt;Statistical Modeling: The Two Cultures (Breiman)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="554" LastActivityDate="2014-06-12T16:52:46.557" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="346" PostTypeId="2" ParentId="266" CreationDate="2014-06-12T17:03:15.733" Score="1" Body="&lt;p&gt;In addition to the courses and tutorials posted, I would suggest something a bit more 'hands on': &lt;a href=&quot;http://www.kaggle.com/competitions&quot; rel=&quot;nofollow&quot;&gt;Kaggle&lt;/a&gt; has some introductory competitions that might pique your interest (most people start with the Titanic competition). And there's a large variety of subjects to explore and compete in when you want to get more experience.&lt;/p&gt;&#xA;" OwnerUserId="554" LastActivityDate="2014-06-12T17:03:15.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="347" PostTypeId="2" ParentId="266" CreationDate="2014-06-12T17:58:21.467" Score="1" Body="&lt;p&gt;As mentioned in above answers grasp the basics of ML by following MOOCs by Prof.Andrew Ng and &lt;a href=&quot;http://work.caltech.edu/telecourse.html&quot; rel=&quot;nofollow&quot;&gt;'Learning From Data'&lt;/a&gt; by Prof. Yaser Abu-Mostafa.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is the &lt;a href=&quot;http://www.kaggle.com/wiki/Software&quot; rel=&quot;nofollow&quot;&gt;clear winner&lt;/a&gt; as the most used tool in Kaggle competitions. (Don't forget to check the resources on Kaggle wiki and forums)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learn basic R and Python. Coursera 'Data Science' track has an &lt;a href=&quot;https://www.coursera.org/course/rprog&quot; rel=&quot;nofollow&quot;&gt;introductory R course&lt;/a&gt;. Almost all the algorithms can be found in Python and R libraries. Feel free to use the algorithms you learned in few kaggle competitions. As a starting point compare the performance of several algorithms on Titanic dataset and Digit recognizer dataset on &lt;a href=&quot;http://www.kaggle.com/&quot; rel=&quot;nofollow&quot;&gt;kaggle&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And do continue practising on various datasets!&lt;/p&gt;&#xA;" OwnerUserId="733" LastActivityDate="2014-06-12T17:58:21.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="348" PostTypeId="2" ParentId="334" CreationDate="2014-06-12T18:33:21.540" Score="6" Body="&lt;p&gt;Not sure about the cloud era one, but one of my friends joined the John Hopkins one and in his words it's &quot;brilliant to get you started&quot;. It has also been recommended by a lot of people. I am planning to join it in few weeks. As far as seriousness is concerned, I don't think these certifications are gonna help you land a job, but they sure will help you learn.  &lt;/p&gt;&#xA;" OwnerUserId="456" LastActivityDate="2014-06-12T18:33:21.540" CommentCount="0" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="349" PostTypeId="2" ParentId="334" CreationDate="2014-06-12T20:51:59.930" Score="33" Body="&lt;p&gt;As a former analytics manager and a current lead data scientist, I am very leery of the need for data science certificates.  The term data scientist is pretty vague and the field of data science is in it's infancy.  A certificates implies some sort of uniform standard which is just lacking in data science, it is still very much the wild west. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While a certificate is probably not going to hurt you, I think your time would be better spent developing the experience to know when to use a certain approach, and depth of understanding to be able to explain that approach to a non-technical audience.&lt;/p&gt;&#xA;" OwnerUserId="780" LastEditorUserId="780" LastEditDate="2014-06-13T15:42:46.987" LastActivityDate="2014-06-13T15:42:46.987" CommentCount="3" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="350" PostTypeId="1" AcceptedAnswerId="359" CreationDate="2014-06-12T22:11:46.607" Score="0" ViewCount="3420" Body="&lt;p&gt;Could you give some examples of typical tasks that a data scientist does in his daily job, and the must-know minimum for each of the levels (like junior, senior, etc. if there are any)? If possible, something like a &lt;a href=&quot;http://www.starling-software.com/employment/programmer-competency-matrix.html&quot; rel=&quot;nofollow&quot;&gt;Programmer competency matrix&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="194" LastEditorUserId="84" LastEditDate="2014-06-12T22:42:11.590" LastActivityDate="2014-06-13T17:36:21.937" Title="Example tasks of a data scientist and the necessary knowledge" Tags="|knowledge-base|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="351" PostTypeId="2" ParentId="334" CreationDate="2014-06-13T03:50:24.610" Score="8" Body="&lt;p&gt;There are multiple certifications going on, but they have different focus area and style of teaching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I prefer The Analytics Edge on eDX lot more over John Hopkins specialization, as it is more intensive and hands on. The expectation in John Hopkins specialization is to put in 3 - 4 hours a week vs. 11 - 12 hours a week on Analytics Edge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From an industry perspective, I take these certifications as a sign of interest and not level of knowledge a person possesses. There are too many dropouts in these MOOCs. I value other experience (like participating in Kaggle competitions) lot more than undergoing XYZ certification on MOOC.&lt;/p&gt;&#xA;" OwnerUserId="735" LastActivityDate="2014-06-13T03:50:24.610" CommentCount="4" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="352" PostTypeId="1" CreationDate="2014-06-13T05:40:39.360" Score="10" ViewCount="294" Body="&lt;p&gt;In some cases, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/20975147&quot;&gt;it may be impossible&lt;/a&gt; to draw Euler diagrams with overlapping circles to represent all the overlapping subsets in the correct proportions. This type of data then requires using polygons or other figures to represent each set. When dealing with data that describes overlapping subsets, how can I figure out whether a simple Euler diagram is possible?&lt;/p&gt;&#xA;" OwnerUserId="62" LastActivityDate="2018-09-29T09:01:00.843" Title="How do I figure out if subsets can be plotted in a normal Euler diagram?" Tags="|visualization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="354" PostTypeId="1" AcceptedAnswerId="355" CreationDate="2014-06-13T07:28:37.763" Score="7" ViewCount="287" Body="&lt;p&gt;as I am very interested in programming and statistics, Data Science seems like a great career path to me - I like both fields and would like to combine them. Unfortunately, I have studied political science with a non-statistical sounding Master. I focused on statistics in this Master, visiting optional courses and writing a statistical thesis on a rather large dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since almost all job adds are requiring  a degree in informatics, physics or some other techy-field, I am wondering if there is a chance to become a data scientist or if I should drop that idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am lacking knowledge in machine learning, sql and hadoop, while having a rather strong informatics and statistics background. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can somebody tell me how feasible my goal of becoming a data scientist is? &lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-06-15T01:29:15.240" Title="Data Science as a Social Scientist?" Tags="|statistics|" AnswerCount="2" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="355" PostTypeId="2" ParentId="354" CreationDate="2014-06-13T10:08:14.087" Score="12" Body="&lt;p&gt;The downvotes are because of the topic, but I'll attempt to answer your question as best I can since it's here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data science is a term that is thrown around as loosely as Big Data.  Everyone has a rough idea of what they mean by the term, but when you look at the actual work tasks, a data scientist's responsibilities will vary greatly from company to company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistical analysis could encompass the entirety of the workload in one job, and not even be a consideration for another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't chase after a job title per se.  If you are interested in the field, network (like you are doing now) and find a good fit.  If you are perusing job ads, just look for the ones that stress statistical and informatics backgrounds.  Hadoop and SQL are both easy to become familiar with given the time and motivation, but I would stick with the areas you are strongest in and go from there.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-13T10:08:14.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="356" PostTypeId="1" CreationDate="2014-06-13T10:57:10.623" Score="6" ViewCount="140" Body="&lt;p&gt;I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break.  I would like some input on how to speed things up.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have multiple datasets of information in a SQL database.  The database is vendor-designed, so I have little control over the structure.  It's a sql representation of a class-based structure.  It looks a little bit like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Main-class table&#xA; -sub-class table 1&#xA; -sub-class table 2&#xA;  -sub-sub-class table&#xA; ...&#xA; -sub-class table n&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each table contains fields for each attribute of the class.  A join exists which contains all of the fields for each of the sub-classes which contains all of the fields in the class table and all of the fields in each parent class' table, joined by a unique identifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are hundreds of classes. which means thousands of views and tens of thousands of columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond that, there are multiple datasets, indicated by a field value in the Main-class table.  There is the production dataset, visible to all end users, and there are several other datasets comprised of the most current version of the same data from various integration sources.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Daily, we run jobs that compare the production dataset to the live datasets and based on a set of rules we merge the data, purge the live datasets, then start all over again.  The rules are in place because we might trust one source of data more than another for a particular value of a particular class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The jobs are essentially a series of SQL statements that go row-by-row through each dataset, and field by field within each row.  The common changes are limited to a handful of fields in each row, but since anything can change we compare each value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 10s of millions of rows of data and in some environments the merge jobs can take longer than 24 hours.  We resolve that problem generally, by throwing more hardware at it, but this isn't a hadoop environment currently so there's a pretty finite limit to what can be done in that regard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you go about scaling a solution to this problem such that there were no limitations?  And how would you go about accomplishing the most efficient data-merge?  (currently it is field by field comparisons... painfully slow).&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="84" LastEditDate="2014-06-13T15:47:16.340" LastActivityDate="2014-06-13T17:14:26.260" Title="How to best accomplish high speed comparison of like data?" Tags="|efficiency|scalability|sql|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="357" PostTypeId="2" ParentId="334" CreationDate="2014-06-13T12:18:53.830" Score="10" Body="&lt;p&gt;I lead data science teams for a major Internet company and I have screened hundreds of profiles and interviewed dozens for our teams around the world. Many candidates have passed the aforementioned courses and programs or bring similar credentials. Personally, I have also taken the courses, some are good, others are disappointing but none of them makes you a &quot;data scientist&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I agree with the others here. A certificate from Coursera or Cloudera just signalizes an interest but it does not move the needle.  There is a lot more to consider and you can have a bigger impact by providing a comprehensive repository of your work (github profile for example) and by networking with other data scientists. Anyone hiring for a data science profile will always prefer to see your previous work and coding style/abilities.&lt;/p&gt;&#xA;" OwnerUserId="797" LastActivityDate="2014-06-13T12:18:53.830" CommentCount="0" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="358" PostTypeId="2" ParentId="356" CreationDate="2014-06-13T12:30:45.290" Score="3" Body="&lt;p&gt;Sounds interesting. Could the solution be to dump the data out, build a fast custom processing thingie to run it through and then import it back to the database? I've seen some blazing fast Java-based text processing tools for topic modeling that handle millions of lines of text per second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's an option then you can build a shell script to first dump the data in as good as format as possible. Then some pre-processing to separate the datasets, then real processing of comparison and changes. Lastly something that writes it back to a good format for input into database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Definately not a one-afternoon project, but you could probably get it to work in a couple of weeks.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2014-06-13T12:30:45.290" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="359" PostTypeId="2" ParentId="350" CreationDate="2014-06-13T13:49:35.777" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://nirvacana.com/thoughts/becoming-a-data-scientist/&quot; rel=&quot;nofollow&quot;&gt;Becoming a Data Scientist – Curriculum via Metromap&lt;/a&gt; is a popular reference for this kind of question.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-06-13T13:49:35.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="360" PostTypeId="2" ParentId="61" CreationDate="2014-06-13T14:46:30.393" Score="5" Body="&lt;p&gt;That's because something called &lt;a href=&quot;http://en.wikipedia.org/wiki/Bias%E2%80%93variance_dilemma&quot; rel=&quot;noreferrer&quot;&gt;bias-variance dilema&lt;/a&gt;. The overfitted model means that we will have more complex decision boundary if we give more variance on model. The thing is, not only too simple models but also complex models are likely to have dis-classified result on unseen data. Consequently, over-fitted model is not good as under-fitted model. That's why overfitting is bad and we need to fit the model somewhere in the middle.&lt;/p&gt;&#xA;" OwnerUserId="801" LastActivityDate="2014-06-13T14:46:30.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="361" PostTypeId="1" CreationDate="2014-06-13T16:44:29.323" Score="65" ViewCount="43125" Body="&lt;p&gt;Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the complexity of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you know when your model has struck the right balance and is not underfitting the data it seeks to model?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This is a followup to my question, &quot;&lt;a href=&quot;https://datascience.stackexchange.com/questions/61/why-is-overfitting-bad/&quot;&gt;Why Is Overfitting Bad?&lt;/a&gt;&quot;&lt;/p&gt;&#xA;" OwnerUserId="158" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2020-04-26T15:03:13.663" Title="When is a Model Underfitted?" Tags="|efficiency|algorithms|parameter|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="362" PostTypeId="2" ParentId="356" CreationDate="2014-06-13T17:14:26.260" Score="4" Body="&lt;p&gt;Can't you create a hash for each classes, and then merge rows by rows, field by field only the classes where the hash changed ? It should be faster if most of the classes don't change..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or a hash of each rows or maybe columns.. depending on how the data normally change.. &lt;/p&gt;&#xA;" OwnerUserId="737" LastActivityDate="2014-06-13T17:14:26.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="363" PostTypeId="2" ParentId="361" CreationDate="2014-06-13T17:14:57.517" Score="8" Body="&lt;p&gt;Models are but abstractions of what is seen in real life. They are designed in order to abstract-away nitty-gritties of the real system in observation, while keeping sufficient information to support desired analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a model is overfit, it takes into account too many details about what is being observed, and small changes on such object may cause the model to lose precision. On the other hand, if a model is underfit, it evaluates so few attributes that noteworthy changes on the object may be ignored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that underfit may be seen as an &lt;em&gt;overfit&lt;/em&gt;, depending on the dataset. If your input can be 99%-correctly-classified with a single attribute, you &lt;em&gt;overfit&lt;/em&gt; the model to the data by simplifying the abstraction to a single characteristic. And, in this case, you'd be generalizing too much the 1% of the base into the 99%-class -- or also specifying the model so much that it can only &lt;em&gt;see&lt;/em&gt; one class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A reasonable way to say that a model is neither over nor underfit is by performing cross-validations. You split your dataset into &lt;em&gt;k&lt;/em&gt; parts, and say, pick one of them to perform your analysis, while using the other &lt;em&gt;k - 1&lt;/em&gt; parts to train your model. Considering that the input itself is not biased, you should be able to have as much variance of data to train and evaluate as you'd have while using the model in real life processing.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-06-13T17:14:57.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="364" PostTypeId="2" ParentId="350" CreationDate="2014-06-13T17:36:21.937" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;http://sijinjoseph.com/programmer-competency-matrix/&quot; rel=&quot;nofollow&quot;&gt;Programmer Competency Matrix&lt;/a&gt; is just a set of skills, which are more likely to occur when being a real programmer than other skills, they are not a checklist to being a programmer, or for that matter, required to be a programmer; most common way to know someone is a programmer is that they're paid to be a programmer, which honestly has nothing to do with programming skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be a data scientist, do data science.&lt;/p&gt;&#xA;" OwnerUserId="158" LastActivityDate="2014-06-13T17:36:21.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="365" PostTypeId="2" ParentId="334" CreationDate="2014-06-13T18:59:11.493" Score="4" Body="&lt;p&gt;@OP: Choosing answers by votes is the WORST idea.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question becomes a popularity contest.  You should seek the right answer, I doubt you know what you are asking, know what you are looking for.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question:&lt;br&gt;&#xA;Q: how seriously DS certifications are viewed at this point by the community.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: What is your goal from taking these courses?  For work, for school, for self-improvement, etc?  Coursera classes are very applied, you will not learn much theory, they are intentionally reserved for classroom setting.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nonetheless, Coursera classes are very useful.  I'd say it is equivalent to one year of stat grad class, out of a two year Master program.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure of its industry recognition yet, because the problem of how did you actually take the course?  How much time did you spend?  It's a lot easier to get A's in these courses than a classroom paper-pencil exam.  So, there is be a huge quality variation from person to person.&lt;/p&gt;&#xA;" OwnerUserId="386" LastEditorUserId="386" LastEditDate="2014-06-15T15:56:06.780" LastActivityDate="2014-06-15T15:56:06.780" CommentCount="3" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="366" PostTypeId="2" ParentId="361" CreationDate="2014-06-13T20:13:01.913" Score="11" Body="&lt;p&gt;To answer your question it is important to understand the frame of reference you are looking for, if you are looking for what philosophically you are trying to achieve in model fitting, check out Rubens answer he does a good job of explaining that context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in practice your question is almost entirely defined by business objectives.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a concrete example, lets say that you are a loan officer, you issued loans that are \$3,000 and when people pay you back you make \$50.  Naturally you are trying to build a model that predicts how if a person defaults on their loan.  Lets keep this simple and say that the outcomes are either full payment, or default.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a business perspective you can sum up a models performance with a contingency matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/RgWr9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the model predicts someone is going to default, do they?  To determining the downsides of over and under fitting I find it helpful to think of it as an optimization problem, because in each cross section of predicted verses actual model performance there is either a cost or profit to be made:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/78kH2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example predicting a default that is a default means avoiding any risk, and predicted a non-default which doesn't default will make \$50 per loan issued.  Where things get dicey is when you are wrong, if you default when you predicted non-default you lose the entire loan principal and if you predict default when a customer actually would not have you suffer \$50 of missed opportunity.  The numbers here are not important, just the approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this framework we can now begin to understand the difficulties associated with over and under fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over fitting in this case would mean that your model works far better on you development/test data then it does in production.  Or to put it another way, your model in production will far underperform what you saw in development, this false confidence will probably cause you to take on far more risky loans then you otherwise would and leaves you very vulnerable to losing money.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, under fitting in this context will leave you with a model that just does a poor job of matching reality.  While the results of this can be wildly unpredictable, (the opposite word you want to describe your predictive models), commonly what happens is standards are tightened up to compensate for this, leading to less overall customers leading to lost good customers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under fitting suffers a kind of opposite difficulty that over fitting does, which is under fitting gives you lower confidence.  Insidiously, the lack of predictability still leads you to take on unexpected risk, all of which is bad news.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience the best way to avoid both of these situations is validating your model on data that is completely outside the scope of your training data, so you can have some confidence that you have a representative sample of what you will see 'in the wild'.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, it is always a good practice to revalidate your models periodically, to determine how quickly your model is degrading, and if it is still accomplishing your objectives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to some things up, your model is under fitted when it does a poor job of predicting both the development and production data.&lt;/p&gt;&#xA;" OwnerUserId="780" LastEditorUserId="325" LastEditDate="2015-04-30T22:18:55.023" LastActivityDate="2015-04-30T22:18:55.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="368" PostTypeId="2" ParentId="235" CreationDate="2014-06-14T07:34:37.643" Score="4" Body="&lt;p&gt;If you know R and it's ggplot library, you could try ggplot for python:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like it, because I do work in R and python, and both are virtually identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you are not familiar you have to deal with a very &quot;unpythonic&quot; syntax. But I think it's an easy library overall.&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-06-14T07:34:37.643" CommentCount="0" CommunityOwnedDate="2016-03-16T10:11:42.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="369" PostTypeId="1" AcceptedAnswerId="465" CreationDate="2014-06-14T18:53:32.243" Score="9" ViewCount="4184" Body="&lt;p&gt;What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would be great as well!&lt;/p&gt;&#xA;" OwnerUserId="838" LastEditorUserId="84" LastEditDate="2014-06-16T19:30:46.940" LastActivityDate="2014-10-09T02:35:24.533" Title="Difference between using RMSE and nDCG to evaluate Recommender Systems" Tags="|machine-learning|recommender-system|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="370" PostTypeId="1" AcceptedAnswerId="372" CreationDate="2014-06-14T19:54:53.193" Score="11" ViewCount="1706" Body="&lt;p&gt;I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;machine learning (rather than traditional statistics);&lt;/li&gt;&#xA;&lt;li&gt;a large enough dataset that you have to run analyses on clusters.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What are some good datasets and problems, accessible to a statistician with some programming background, that I can use to explore the field of data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To keep this as narrow as possible, I'd ideally like links to open, well used datasets and example problems.&lt;/p&gt;&#xA;" OwnerUserId="839" LastEditorUserId="322" LastEditDate="2014-06-18T13:53:25.307" LastActivityDate="2014-06-18T13:53:25.307" Title="Data Science oriented dataset/research question for Statistics MSc thesis" Tags="|statistics|education|knowledge-base|definitions|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="371" PostTypeId="1" AcceptedAnswerId="374" CreationDate="2014-06-14T20:32:06.143" Score="12" ViewCount="10359" Body="&lt;p&gt;I'm curious about natural language querying.  Stanford has what looks to be a strong set of &lt;a href=&quot;http://nlp.stanford.edu/software/index.shtml&quot; rel=&quot;noreferrer&quot;&gt;software for processing natural language&lt;/a&gt;.  I've also seen the &lt;a href=&quot;http://opennlp.apache.org/documentation/1.5.3/manual/opennlp.html&quot; rel=&quot;noreferrer&quot;&gt;Apache OpenNLP library&lt;/a&gt;, and the &lt;a href=&quot;http://gate.ac.uk/science.html&quot; rel=&quot;noreferrer&quot;&gt;General Architecture for Text Engineering&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are an incredible amount of uses for natural language processing and that makes the documentation of these projects difficult to quickly absorb.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you simplify things for me a bit and at a high level outline the tasks necessary for performing a basic translation of simple questions into SQL?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first rectangle on my flow chart is a bit of a mystery.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/wJPx9.png&quot; alt=&quot;enter image description here&quot;&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I might want to know:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;How many books were sold last month?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I'd want that translated into&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Select count(*) &#xA;  from sales &#xA;  where &#xA;   item_type='book' and &#xA;   sales_date &amp;gt;= '5/1/2014' and &#xA;   sales_date &amp;lt;= '5/31/2014'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-06-14T20:39:25.657" LastActivityDate="2021-05-16T12:13:06.787" Title="How to process natural language queries?" Tags="|nlp|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="372" PostTypeId="2" ParentId="370" CreationDate="2014-06-14T20:52:00.873" Score="8" Body="&lt;p&gt;Just head to kaggle.com; it'll keep you busy for a long time. For open data there's the &lt;a href=&quot;http://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow noreferrer&quot;&gt;UC Irvine Machine Learning Repository&lt;/a&gt;. In fact, there's a whole &lt;a href=&quot;https://opendata.stackexchange.com/&quot;&gt;Stackexchange site&lt;/a&gt; devoted to this; look there.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="-1" LastEditDate="2017-04-13T12:57:29.190" LastActivityDate="2014-06-14T20:52:00.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="373" PostTypeId="2" ParentId="370" CreationDate="2014-06-14T21:13:20.110" Score="5" Body="&lt;p&gt;The &lt;a href=&quot;http://sunlightfoundation.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sunlight Foundation&lt;/a&gt; is an organization that is focused on opening up and encouraging non-partisan analysis of government data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a ton of analysis out there in the wild that can be used for comparison, and a wide variety of topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They provide &lt;a href=&quot;http://sunlightfoundation.com/tools/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tools&lt;/a&gt; and &lt;a href=&quot;http://sunlightfoundation.com/api/&quot; rel=&quot;nofollow noreferrer&quot;&gt;apis&lt;/a&gt; for accessing data, and have helped push to make data available in places like &lt;a href=&quot;http://www.data.gov/&quot; rel=&quot;nofollow noreferrer&quot;&gt;data.gov&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One interesting project is &lt;a href=&quot;http://influenceexplorer.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Influence Explorer&lt;/a&gt;.  You can get &lt;a href=&quot;http://data.influenceexplorer.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;source data here&lt;/a&gt; as well as access to real time data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to take a look at one of our more popular questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/155/publicly-available-datasets&quot;&gt;Publicly available datasets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-06-14T21:13:20.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="374" PostTypeId="2" ParentId="371" CreationDate="2014-06-14T21:39:15.833" Score="7" Body="&lt;p&gt;Natural language querying poses very many intricacies which can be very difficult to generalize.  From a high level, I would start with trying to think of things in terms of nouns and verbs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for the sentence: How many books were sold last month?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would start by breaking the sentence down with a parser which will return a tree format similar to this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ogoiY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that there is a subject books, a compound verbal phrase indicating the past action of sell, and then a noun phrase where you have the time focus of a month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can further break down the subject for modifiers: &quot;how many&quot; for books, and &quot;last&quot; for month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have broken the sentence down you need to map those elements to sql language e.g.: how many =&gt; count, books =&gt; book, sold =&gt; sales, month =&gt; sales_date (interval), and so on.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, once you have the elements of the language you just need to come up with a set of rules for how different entities interact with each other, which leaves you with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Select count(*) &#xA;  from sales &#xA;  where &#xA;   item_type='book' and &#xA;   sales_date &gt;= '5/1/2014' and &#xA;   sales_date &amp;lt;= '5/31/2014'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is at a high level how I would begin, while almost every step I have mentioned is non-trivial and really the rabbit hole can be endless, this should give you many of the dots to connect.  &lt;/p&gt;&#xA;" OwnerUserId="780" LastActivityDate="2014-06-14T21:39:15.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="375" PostTypeId="1" AcceptedAnswerId="394" CreationDate="2014-06-14T23:52:10.490" Score="4" ViewCount="2223" Body="&lt;p&gt;The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?&lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2021-07-12T21:20:40.697" Title="Amazon S3 vs Google Drive" Tags="|bigdata|" AnswerCount="2" CommentCount="0" ClosedDate="2014-08-08T12:24:27.027" ContentLicense="CC BY-SA 3.0" />
  <row Id="376" PostTypeId="1" CreationDate="2014-06-15T00:00:51.657" Score="8" ViewCount="178" Body="&lt;p&gt;I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise &lt;em&gt;false/unfair&lt;/em&gt; comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can one honestly compare different experiments to point out efficiency gains? &lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-06-15T00:05:58.457" LastActivityDate="2014-07-18T04:50:58.287" Title="How to compare experiments run over different infrastructures" Tags="|bigdata|efficiency|performance|scalability|distributed|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="377" PostTypeId="2" ParentId="375" CreationDate="2014-06-15T00:11:54.950" Score="4" Body="&lt;p&gt;From our perspective on here, the big benefit of S3 is the ease of accessing the data from within EC2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google Drive is directly accessible from the Google Cloud platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a host of other differences that might matter depending on your usage requirements, but that's the one that would matter most around here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only other difference I can think of that would matter to the DS community is that when you are sharing something, you have no control of the address of a given file on google drive.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-15T00:11:54.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="379" PostTypeId="2" ParentId="307" CreationDate="2014-06-15T01:25:48.563" Score="6" Body="&lt;p&gt;Financial Services is a big user of Big Data, and innovator too.  One example is mortgage bond trading.  To answer your questions for it:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What kinda data these companies used. What was the size of the data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Long histories of each mortgage issued for the past many years, and payments by month against them.  (Billions of rows)&lt;/li&gt;&#xA;&lt;li&gt;Long histories of credit histories.  (Billions of rows)&lt;/li&gt;&#xA;&lt;li&gt;Home price indices.  (Not as big)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What kinda of tools technologies they used to process the data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It varies.  Some use in-house solutions built on databases like Netezza or Teradata.  Others access the data via systems provided by the data providers.  (Corelogic, Experian, etc)  Some banks use columnal database technologies like KDB, or 1010data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What was the problem they were facing and how the insight they got the&#xA;  data helped them to resolve the issue.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The key issue is determining when mortgage bonds (mortgage backed-securities) will prepay or default.  This is especially important for bonds that lack the government guarantee.  By digging into payment histories, credit files, and understanding the current value of the house, it's possible to predict the likelihood of a default.  Adding an interest rate model and prepayment model also helps predict the likelihood of a prepayment.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How they selected the tool\technology to suit their need.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If the project is driven by internal IT, usually it's based off of a large database vendor like Oracle, Teradata or Netezza.  If it's driven by the quants, then they are more likely to go straight to the data vendor, or a 3rd party &quot;All in&quot; system.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What kinda pattern they identified from the data &amp;amp; what kind of&#xA;  patterns they were looking from the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Linking the data gives great insights into who is likely to default on their loans, and prepay them.  When you aggregated the loans into bonds, it can be the difference between a bond issued at $100,000,000 being worth that amount, or as little as $20,000,000.&lt;/p&gt;&#xA;" OwnerUserId="842" LastActivityDate="2014-06-15T01:25:48.563" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="380" PostTypeId="2" ParentId="354" CreationDate="2014-06-15T01:29:15.240" Score="5" Body="&lt;p&gt;I suspect this will get closed since it is very narrow, but my 2 cents...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science requires 3 skills:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Math/Stats&lt;/li&gt;&#xA;&lt;li&gt;Programming&lt;/li&gt;&#xA;&lt;li&gt;Domain Knowledge&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It can be very hard to show all three.  #1 and #2 can be signaled via degrees, but a hiring manager who may not have them doesn't want to trust a liberal arts degree.  If you're looking to get into Data Science, position yourself as a domain expert first.  Publish election predictions.  If you're correct, cite them.  That will get you noticed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're Domain knowledge is A+ level, you don't need A+ level programming skills, but learn programming enough so that you don't need someone else to fetch data for you.&lt;/p&gt;&#xA;" OwnerUserId="842" LastActivityDate="2014-06-15T01:29:15.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="381" PostTypeId="2" ParentId="361" CreationDate="2014-06-15T01:36:51.693" Score="4" Body="&lt;p&gt;CAPM (Capital Asset Pricing Model) in Finance is a classic example of an underfit model.  It was built on the beautiful theory that &quot;Investors only pay for risk they can't diversify away&quot; so expected excess returns are equal to correlation to market returns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a formula [0] Ra = Rf + B (Rm - Rf)&#xA;where Ra is the expected return of the asset, Rf is the risk free rate, Rm is the market rate of return, and Beta is the correlation to the Equity premium (Rm - Rf)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is beautiful, elegant, and wrong.  Investors seem to require a higher rate of small stocks and value (defined by book to market, or dividend yield) stocks.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fama and French [1] presented an update to the model, which adds additional Betas for Size and Value.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do you know in a general sense?  When the predictions you are making are wrong, and another variable with a logical explanation increases the prediction quality.  It's easy to understand why someone might think small stocks are risky, independent of non-diversifiable risk.  It's a good story, backed by the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0]  &lt;a href=&quot;http://www.investopedia.com/terms/c/capm.asp&quot; rel=&quot;nofollow&quot;&gt;http://www.investopedia.com/terms/c/capm.asp&lt;/a&gt;&#xA;[1]  &lt;a href=&quot;http://en.wikipedia.org/wiki/Fama%E2%80%93French_three-factor_model&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Fama%E2%80%93French_three-factor_model&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="842" LastActivityDate="2014-06-15T01:36:51.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="382" PostTypeId="1" CreationDate="2014-06-15T12:26:50.060" Score="10" ViewCount="413" Body="&lt;p&gt;I've came across the following problem, that I recon is rather typical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some large data, say, a few million rows. I run some non-trivial analysis on it, e.g. an SQL query consisting of several sub-queries. I get some result, stating, for example, that property X is increasing over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, there are two possible things that could lead to that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;X is indeed increasing over time&lt;/li&gt;&#xA;&lt;li&gt;I have a bug in my analysis&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How can I test that the first happened, rather than the second? A step-wise debugger, even if one exists, won't help, since intermediate results can still consist of millions of lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing I could think of was to somehow generate a small, synthetic data set with the property that I want to test and run the analysis on it as a unit test. Are there tools to do this? Particularly, but not limited to, SQL.&lt;/p&gt;&#xA;" OwnerUserId="846" LastActivityDate="2014-06-16T13:21:58.777" Title="How to debug data analysis?" Tags="|data-mining|sql|experiments|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="384" PostTypeId="1" AcceptedAnswerId="395" CreationDate="2014-06-15T14:01:38.233" Score="20" ViewCount="28264" Body="&lt;p&gt;I have a binary classification problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Approximately 1000 samples in training set&lt;/li&gt;&#xA;&lt;li&gt;10 attributes, including binary, numeric and categorical&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which algorithm is the best choice for this type of problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default I'm going to start with SVM (preliminary having nominal attributes values converted to binary features), as it is considered the best for relatively clean and  not noisy data. &lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-06-16T14:02:42.467" LastActivityDate="2015-04-12T16:12:39.063" Title="Choose binary classification algorithm" Tags="|classification|binary|svm|random-forest|logistic-regression|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="386" PostTypeId="2" ParentId="384" CreationDate="2014-06-15T14:23:19.793" Score="12" Body="&lt;p&gt;For low parameters, pretty limited sample size,  and a binary classifier logistic regression should be plenty powerful enough.  You can use a more advanced algorithm but it's probably overkill. &lt;/p&gt;&#xA;" OwnerUserId="780" LastActivityDate="2014-06-15T14:23:19.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="387" PostTypeId="1" AcceptedAnswerId="388" CreationDate="2014-06-15T15:11:29.970" Score="8" ViewCount="374" Body="&lt;p&gt;I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. Also it'd become a burden to continuously validate the &lt;em&gt;current state&lt;/em&gt; of each spammer blocked, checking if the site/domain still disseminate spam data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering that any approach must be efficient and scalable, so as to support filtering on very large datasets, what are the strategies available to get rid of spam in a non-biased manner?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: if possible, any example of strategy, even if just the intuition behind it, would be very welcome along with the answer.&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-06-15T15:23:01.007" LastActivityDate="2014-06-16T20:18:50.123" Title="Filtering spam from retrieved data" Tags="|bigdata|efficiency|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="388" PostTypeId="2" ParentId="387" CreationDate="2014-06-15T15:20:31.300" Score="5" Body="&lt;p&gt;Spam filtering, especially in email, has been revolutionized by neural networks, here are a couple papers that provide good reading on the subject:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Neural Networks And The Future Of Spam &#xA;A. C. Cosoi, M. S. Vlad, V. Sgarciu &#xA;&lt;a href=&quot;http://ceai.srait.ro/index.php/ceai/article/viewFile/18/8&quot; rel=&quot;nofollow&quot;&gt;http://ceai.srait.ro/index.php/ceai/article/viewFile/18/8&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intelligent Word-Based Spam Filter Detection Using &#xA;Multi-Neural Networks &#xA;Ann Nosseir, Khaled Nagati and Islam Taj-Eddin&#xA;&lt;a href=&quot;http://www.ijcsi.org/papers/IJCSI-10-2-1-17-21.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ijcsi.org/papers/IJCSI-10-2-1-17-21.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spam Detection using Adaptive Neural Networks: Adaptive Resonance Theory &#xA;David Ndumiyana, Richard Gotora, and Tarisai Mupamombe&#xA;&lt;a href=&quot;http://onlineresearchjournals.org/JPESR/pdf/2013/apr/Ndumiyana%20et%20al.pdf&quot; rel=&quot;nofollow&quot;&gt;http://onlineresearchjournals.org/JPESR/pdf/2013/apr/Ndumiyana%20et%20al.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;The basic intuition behind using a neural network to help with spam filtering is by providing a weight to terms based on how often they are associated with spam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks can be trained most quickly in a supervised -- you explicitly provide the classification of the sentence in the training set -- environment.  Without going into the nitty gritty the basic idea can be illustrated with these sentences:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Text = &quot;How is the loss of the Viagra patent going to affect Pfizer&quot;, Spam = false&#xA;Text = &quot;Cheap Viagra Buy Now&quot;, Spam = true&#xA;Text = &quot;Online pharmacy Viagra Cialis Lipitor&quot;, Spam = true&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a two stage neural network, the first stage will calculate the likelihood of spam based off of if the word exists in the sentence.  So from our example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;viagra =&gt; 66%&#xA;buy =&gt; 100%&#xA;Pfizer =&gt; 0%&#xA;etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for the second stage the results in the first stage are used as variables in the second stage:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;viagra &amp;amp; buy =&gt; 100%&#xA;Pfizer &amp;amp; viagra=&gt; 0%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This basic idea is run for many of the permutations of the all the words in your training data.  The end results once trained is basically just an equation that based of the context of the words in the sentence can assign a probability of being spam.  Set spamminess threshold, and filter out any data higher then said threshold.&lt;/p&gt;&#xA;" OwnerUserId="780" LastEditorUserId="780" LastEditDate="2014-06-15T21:22:10.500" LastActivityDate="2014-06-15T21:22:10.500" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="389" PostTypeId="2" ParentId="382" CreationDate="2014-06-15T15:49:12.907" Score="4" Body="&lt;p&gt;Here is a suggestion:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Code your analysis in such a way that it can be run on sub-samples.&lt;/li&gt;&#xA;&lt;li&gt;Code a complementary routine which can sample, either randomly, or by time, or by region, or ...  This may be domain-specific. This is where your knowledge enters.&lt;/li&gt;&#xA;&lt;li&gt;Combine the two and see if the results are stable across subsamples.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="515" LastActivityDate="2014-06-15T15:49:12.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="390" PostTypeId="2" ParentId="384" CreationDate="2014-06-15T16:07:35.543" Score="5" Body="&lt;p&gt;When categorical variables are in the mix, I reach for Random Decision Forests, as it handles categorical variables directly without the 1-of-n encoding transformation. This loses less information.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-06-15T16:07:35.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="391" PostTypeId="2" ParentId="382" CreationDate="2014-06-15T16:59:50.190" Score="1" Body="&lt;p&gt;This is what I normally do - take up the most important variables (basis your business understanding and hypothesis - you can always revise it later), group by on these attributes to reduce the number of rows, which can then be imported into a Pivot. You should include the sum and count of the relevant metrics on each row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure that you don't put any filters in the previous step. Once you have entire data at a summarized level, you can play around in Pivot tables and see what things are changing / increasing or decreasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data is too big to be summarized even on important parameters, you need to partition it in 3 - 4 subsets and then do this again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="735" LastActivityDate="2014-06-15T16:59:50.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="393" PostTypeId="2" ParentId="384" CreationDate="2014-06-15T22:33:17.670" Score="5" Body="&lt;p&gt;Linear SVM should be a good starting point. Take a look at &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; guide to choose the right estimator.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-15T22:33:17.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="394" PostTypeId="2" ParentId="375" CreationDate="2014-06-16T04:21:36.340" Score="5" Body="&lt;p&gt;Personally, we use S3 on top of GCE and really love it. Depending on how much data you're dealing with, Google Drive just doesn't quite match the 5 TB max that S3 gives you. Also, if you're using python, &lt;code&gt;boto&lt;/code&gt; does a pretty fantastic job of making most aws services pretty accessible regardless of what stack you're dealing with. Even if you're not using python, they've got a pretty straightforward API that generally is more accessible than Google Drive.&lt;/p&gt;&#xA;&lt;p&gt;Instead of google drive, though google did recently release a cloud storage service, apart from drive, that lets you more closely integrate your storage with any gce instance you've got, &lt;a href=&quot;https://cloud.google.com/products/cloud-storage/&quot; rel=&quot;nofollow noreferrer&quot;&gt;google cloud storage&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;They've got an API which seems to be pretty comparable to S3's, but I can't profess to having really played around with it much. Pricing-wise the two are identical, but I think that the large community and experience with aws in general still puts S3 squarely above both google's cloud storage and google drive.&lt;/p&gt;&#xA;" OwnerUserId="548" LastEditorUserId="120060" LastEditDate="2021-07-12T21:20:40.697" LastActivityDate="2021-07-12T21:20:40.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="395" PostTypeId="2" ParentId="384" CreationDate="2014-06-16T04:37:58.817" Score="15" Body="&lt;p&gt;It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extreme random forests are pretty similar to standard random forests with the one exception that instead of optimizing splits on trees, extreme random forest makes splits at random. Initially this would seem like a negative, but it generally means that you have significantly better generalization and speed, though the AUC on your training set is likely to be a little worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is also a pretty solid bet for these kinds of tasks, though with your relatively low dimensionality and small sample size I would be worried about overfitting. You might want to check out using K-Nearest Neighbors since it often performs very will with low dimensionalities, but it doesn't usually handle categorical variables very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to pick one without knowing more about the problem I would certainly place my bets on extreme random forest, as it's very likely to give you good generalization on this kind of dataset, and it also handles a mix of numerical and categorical data better than most other methods.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-16T04:37:58.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="396" PostTypeId="2" ParentId="103" CreationDate="2014-06-16T04:51:47.847" Score="10" Body="&lt;p&gt;Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be &lt;code&gt;O(n2)&lt;/code&gt;, which is far from ideal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, my go-to clustering algorithms are OpenOrd for winner-takes-all clustering and FLAME for fuzzy clustering. Both methods are indifferent to whether the metrics used are similarity or distance (FLAME in particular is nearly identical in both constructions). The implementation of OpenOrd in Gephi is &lt;code&gt;O(nlogn)&lt;/code&gt; and is known to be more scalable than any of the other clustering algorithms present in the Gephi package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FLAME on the other hand is great if you're looking for a fuzzy clustering method. While the complexity of FLAME is a little harder to determine since it's an iterative process, it has been shown to be sub-quadratic, and similar in run-speed to knn.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-16T04:51:47.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="397" PostTypeId="2" ParentId="382" CreationDate="2014-06-16T06:34:53.683" Score="1" Body="&lt;p&gt;First you need to verify that your implementation of the algorithm is accurate. For that use a small sample of data and check whether the result is correct. At this stage the sample doesn't need to be representative of the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the implementation is verified, you need to verify that there is a significant relationship among the variables that you try to predict. To do that define null hypothesis and try to reject the null hypothesis with a significant confidence level. (&lt;a href=&quot;http://stattrek.com/regression/slope-test.aspx&quot; rel=&quot;nofollow&quot;&gt;hypothesis testing for linear regression&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There might be unit test frameworks for your SQL distribution. But using a programming language like R will be more easier to implement.&lt;/p&gt;&#xA;" OwnerUserId="733" LastActivityDate="2014-06-16T06:34:53.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="398" PostTypeId="1" AcceptedAnswerId="405" CreationDate="2014-06-16T07:32:29.137" Score="8" ViewCount="992" Body="&lt;p&gt;I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role in this project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In view of this I face the following dilemma: Should I learn Python to be able to use its extensive scientific libraries (Pandas, Numpy, Scipy, ...), or should I just dive into similar packages of a language I'm already acquainted with (Racket, or to a lesser extent Scala)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Ideally I would learn Python in parallel with using statistical libraries in Racket, but I'm not sure I'll have time for both)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not looking for an answer to this dilemma, but rather for feedback on my different considerations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current position is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In favour of Python:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Extensively used libraries&lt;/li&gt;&#xA;&lt;li&gt;Widely used (may be decisive in case of collaboration with others)&lt;/li&gt;&#xA;&lt;li&gt;A lot of online material to start learning it&lt;/li&gt;&#xA;&lt;li&gt;Conferences that are specifically dedicated to Scientific Computing with Python&lt;/li&gt;&#xA;&lt;li&gt;Learning Python won't be a waste of time anyway&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In favour of a language I already know:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It's a way to deepen my knowledge of one language rather than getting superficial knowledge of one more language (under the motto: you should at least know one language really well)&lt;/li&gt;&#xA;&lt;li&gt;It is feasible. Both Racket and Scala have good mathematics and statistics libraries&lt;/li&gt;&#xA;&lt;li&gt;I can start right away with learning what I need to know rather than first having to learn the basics&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Two concrete questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What am I forgetting?&lt;/li&gt;&#xA;&lt;li&gt;How big of a nuisance could the Python 2 vs 3 issue be?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="872" LastActivityDate="2014-06-16T15:00:04.577" Title="What to consider before learning a new language for data analysis" Tags="|python|visualization|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="399" PostTypeId="2" ParentId="398" CreationDate="2014-06-16T09:52:24.913" Score="1" Body="&lt;p&gt;According to me, all the factors, you have mentioned are superficial in nature. You have not considered the core of tool selection. In this case, there are 2 aspects, you mentioned:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Data analysis - What kind of analysis are you working on? There might be some analysis which are easier in some languages and more difficult in other.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Visualization - R provides similar community and learning material (as Python) and has the best visualizations compared to other languages here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;At this stage, you can be flexible with what language to learn, since you are starting from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="735" LastActivityDate="2014-06-16T09:52:24.913" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="400" PostTypeId="2" ParentId="398" CreationDate="2014-06-16T11:45:05.800" Score="5" Body="&lt;p&gt;From my experience, the points to keep in mind when considering a data analysis platform are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can it handle the size of the data that I need? If your data sets fit in memory, there's usually no big trouble, although AFAIK Python is somewhat more memory-efficient than R. If you need to handle larger-than-memory data sets, the platform need to handle it conveniently. In this case, SQL would cover for basic statistics, Python + Apache Spark is another option.&lt;/li&gt;&#xA;&lt;li&gt;Does the platform covers all of my analysis needs? The greatest annoyance I've encountered in data mining projects is having to juggle between several tools, because tool A handles web connections well, tool B does the statistics and tool C renders nice pictures. You want your weapon-of-choice to cover as many aspects of your projects as possible. When considering this issue, Python is very comprehensive, but R has a lot of build-in statistical tests ready-to-use, if that's what you need. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="846" LastActivityDate="2014-06-16T11:45:05.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="401" PostTypeId="2" ParentId="382" CreationDate="2014-06-16T13:21:58.777" Score="1" Body="&lt;p&gt;I like a multiple step strategy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Write clean easy to understand code, as opposed to short-tricky code. I know statisticians like tricky code, but spotting problems in tricky code is dangerous. &#xA;( I am mentioning this because a supervisor of mine was fond of undocumented 500 lines python scrips - have fun debugging that mess and I have seen that pattern a lot, especially from people who are not from an IT background)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Break down your code in smaller functions, which can be tested and evaluated in smaller stes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Look for connected elements, e.g. the number of cases with condition X is Y - so this query MUST return Y. Most often this is more complex, but doable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When you are running your script the first time, test it with a small subsample and carefully check if everything is in order. While I like unit tests in IT, bugs in statistics scripts are often so pronounced that they are easily visible doing a carefully check. Or they are methodical errors, which are probably never caught by unit tests.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;That should suffice to ensure a clean &quot;one - off &quot; job. But for a time series as you seem to have, I would add that you should check for values out of range, impossible combinations etc. For me, most scripts that have reached step 4 are probably bug free - and they will stay that way unless something changes. And most often, the data are changing - and that is something which should be checked for every run. Writing code for that can be time consuming and annoying, but it beats subtle errors due to data entry errors.&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-06-16T13:21:58.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="402" PostTypeId="2" ParentId="223" CreationDate="2014-06-16T13:25:48.453" Score="7" Body="&lt;p&gt;The &lt;a href=&quot;http://brat.nlplab.org/&quot; rel=&quot;noreferrer&quot;&gt;brat annotation tool&lt;/a&gt; might be useful for you as per my comment.  I have tried many of them and this is the best I have found.  It has a nice user interface and can support a number of different types of annotations.  The annotations are stored in a separate .annot file which contain each annotation as well as its location within the original document.  A word of warning though, if you ultimately want to feed the annotations into a classifier like the Stanford NER tool then you will have to do some manipulation to get the data into a format that it will accept.&lt;/p&gt;&#xA;" OwnerUserId="387" LastActivityDate="2014-06-16T13:25:48.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="403" PostTypeId="1" CreationDate="2014-06-16T13:30:01.320" Score="2" ViewCount="482" Body="&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Ih6o.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to do Logistic Regression using SAS Enterprise Miner. &#xA;My Independent variables are &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CPR/Inc (Categorical 1 to 7)&#xA;OD/Inc (Categorical 1 to 4)&#xA;Insurance (Binary 0 or 1)&#xA;Income Loss (Binary 0 or 1)&#xA;Living Arrangement (Categorical 1 to 7)&#xA;Employment Status (categorical 1 to 8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My Dependent Variable is Default (Binary 0 or 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is the output from running Regression Model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Analysis of Maximum Likelihood Estimates&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                  Standard          Wald&#xA;Parameter       DF    Estimate       Error    Chi-Square    Pr &amp;gt; ChiSq    Exp(Est)&#xA;&#xA;Intercept        1     -0.4148      0.0645         41.30        &amp;lt;.0001       0.660&#xA;CPR___Inc  1     1     -0.8022      0.1051         58.26        &amp;lt;.0001       0.448&#xA;CPR___Inc  2     1     -0.4380      0.0966         20.57        &amp;lt;.0001       0.645&#xA;CPR___Inc  3     1      0.3100      0.0871         12.68        0.0004       1.363&#xA;CPR___Inc  4     1    -0.00304      0.0898          0.00        0.9730       0.997&#xA;CPR___Inc  5     1      0.1331      0.0885          2.26        0.1324       1.142&#xA;CPR___Inc  6     1      0.1694      0.0881          3.70        0.0546       1.185&#xA;Emp_Status 1     1     -0.2289      0.1006          5.18        0.0229       0.795&#xA;Emp_Status 2     1      0.4061      0.0940         18.66        &amp;lt;.0001       1.501&#xA;Emp_Status 3     1     -0.2119      0.1004          4.46        0.0347       0.809&#xA;Emp_Status 4     1      0.1100      0.0963          1.30        0.2534       1.116&#xA;Emp_Status 5     1     -0.2280      0.1007          5.12        0.0236       0.796&#xA;Emp_Status 6     1      0.3761      0.0943         15.91        &amp;lt;.0001       1.457&#xA;Emp_Status 7     1     -0.3337      0.1026         10.59        0.0011       0.716&#xA;Inc_Loss   0     1     -0.1996      0.0449         19.76        &amp;lt;.0001       0.819&#xA;Insurance  0     1      0.1256      0.0559          5.05        0.0246       1.134&#xA;Liv_Arran  1     1     -0.1128      0.0916          1.52        0.2178       0.893&#xA;Liv_Arran  2     1      0.2576      0.0880          8.57        0.0034       1.294&#xA;Liv_Arran  3     1      0.0235      0.0904          0.07        0.7950       1.024&#xA;Liv_Arran  4     1      0.0953      0.0887          1.16        0.2825       1.100&#xA;Liv_Arran  5     1     -0.0493      0.0907          0.29        0.5871       0.952&#xA;Liv_Arran  6     1     -0.3732      0.0966         14.93        0.0001       0.689&#xA;OD___Inc   1     1     -0.2136      0.0557         14.72        0.0001       0.808&#xA;OD___Inc   2     1     -0.0279      0.0792          0.12        0.7248       0.973&#xA;OD___Inc   3     1     -0.0249      0.0793          0.10        0.7534       0.975&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I used this Model to Score a new set of data. An example row of my new data is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CPR - 7&#xA;OD - 4&#xA;Living Arrangement - 4&#xA;Employment Status - 4&#xA;Insurance - 0&#xA;Income Loss - 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this sample row, the model predicted output (Probability of default = 1) as 0.7335 &#xA;To check this manually, I added the estimates&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Intercept + Emp Status 4 + Liv Arran 4 + Insurance 0&#xA;-0.4148   + 0.1100  +   0.0953   +   0.1256    =   -0.0839&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Odds ratio = Exponential(-0.0839) = 0.9195&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence probability = 0.9195 / (1 + 0.9195)  =   0.4790&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am unable to understand why there is such a mismatch between the Model's predicted probability and theoretical probability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be much appreciated .&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="880" LastEditorUserId="368" LastEditDate="2014-06-18T13:53:29.630" LastActivityDate="2014-09-16T16:37:33.357" Title="Why is there such a mismatch between the Model's predicted probability and theoretical probability in logistic regression?" Tags="|categorical-data|logistic-regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="404" PostTypeId="2" ParentId="223" CreationDate="2014-06-16T14:35:20.980" Score="16" Body="&lt;p&gt;Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving beyond this level of metadata. I would especially pick a format that can be used regardless of development environment and one that can keep some basic structure if that becomes relevant (like tokenization)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might seem strange, but I would honestly suggest &lt;code&gt;JSON&lt;/code&gt;. It's extremely well supported, supports a lot of structure, and is flexible enough that you shouldn't have to move from it for not being powerful enough. For your example, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'text': 'I saw the company's manager last day.&quot;, {'Person': [{'name': 'John'}, {'indices': [0:1]}, etc...]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The one big advantage you've got over any NLP-specific formats here is that &lt;code&gt;JSON&lt;/code&gt; can be parsed in any environment, and since you'll probably have to edit your format anyway, JSON lends itself to very simple edits that give you a short distance to other formats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also implicitly store tokenization information if you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;text&quot;: [&quot;I&quot;, &quot;saw&quot;, &quot;the&quot;, &quot;company's&quot;, &quot;manager&quot;, &quot;last&quot;, &quot;day.&quot;]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: To clarify the mapping of metadata is pretty open, but here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'body': '&amp;lt;some_text&amp;gt;',&#xA; 'metadata': &#xA;  {'&amp;lt;entity&amp;gt;':&#xA;    {'&amp;lt;attribute&amp;gt;': '&amp;lt;value&amp;gt;',&#xA;     'location': [&amp;lt;start_index&amp;gt;, &amp;lt;end_index&amp;gt;]&#xA;    }&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope that helps, let me know if you've got any more questions.&lt;/p&gt;&#xA;" OwnerUserId="548" LastEditorUserId="548" LastEditDate="2014-06-16T17:35:41.207" LastActivityDate="2014-06-16T17:35:41.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="405" PostTypeId="2" ParentId="398" CreationDate="2014-06-16T15:00:04.577" Score="13" Body="&lt;p&gt;Personally going to make a strong argument in favor of Python here. There are a large number of reasons for this, but I'm going to build on some of the points that other people have mentioned here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Picking a single language:&lt;/strong&gt; It's definitely possible to mix and match languages, picking &lt;code&gt;d3&lt;/code&gt; for your visualization needs, &lt;code&gt;FORTRAN&lt;/code&gt; for your fast matrix multiplies, and &lt;code&gt;python&lt;/code&gt; for all of your networking and scripting. You can do this down the line, but keeping your stack as simple as possible is a good move, especially early on.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Picking something bigger than you:&lt;/strong&gt; You never want to be pushing up against the barriers of the language you want to use. This is a huge issue when it comes to languages like &lt;code&gt;Julia&lt;/code&gt; and &lt;code&gt;FORTRAN&lt;/code&gt;, which simply don't offer the full functionality of languages like &lt;code&gt;python&lt;/code&gt; or &lt;code&gt;R&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Pick Community&lt;/strong&gt;: The one most difficult thing to find in any language is community. &lt;code&gt;Python&lt;/code&gt; is the clear winner here. If you get stuck, you ask something on SO, and someone will answer in a matter of minutes, which is simply not the case for most other languages. If you're learning something in a vacuum you will simply learn much slower.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In terms of the minus points, I might actually push back on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deepening your knowledge of one language is a decent idea, but knowing &lt;em&gt;only&lt;/em&gt; one language, without having practice generalizing that knowledge to other languages is a good way to shoot yourself in the foot. I have changed my entire favored development stack three time over as many years, moving from &lt;code&gt;MATLAB&lt;/code&gt; to &lt;code&gt;Java&lt;/code&gt; to &lt;code&gt;haskell&lt;/code&gt; to &lt;code&gt;python&lt;/code&gt;. Learning to transfer your knowledge to another language is far more valuable than just knowing one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as feasibility, this is something you're going to see again and again in any programming career. Turing completeness means you could technically do everything with &lt;code&gt;HTML4&lt;/code&gt; and &lt;code&gt;CSS3&lt;/code&gt;, but you want to pick the right tool for the job. If you see the ideal tool and decide to leave it by the roadside you're going to find yourself slowed down wishing you had some of the tools you left behind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A great example of that last point is trying to deploy &lt;code&gt;R&lt;/code&gt; code. 'R''s networking capabilities are hugely lacking compared to &lt;code&gt;python&lt;/code&gt;, and if you want to deploy a service, or use slightly off-the-beaten path packages, the fact that &lt;code&gt;pip&lt;/code&gt; has an order of magnitude more packages than &lt;code&gt;CRAN&lt;/code&gt; is a huge help.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-16T15:00:04.577" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="406" PostTypeId="1" CreationDate="2014-06-16T15:49:55.673" Score="21" ViewCount="12258" Body="&lt;p&gt;If I have a retail store and have a way to measure how many people enter my store every minute, and timestamp that data, how can I predict future foot traffic?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked into machine learning algorithms, but I'm not sure which one to use. In my test data, a year over year trend is more accurate compared to other things I've tried, like KNN (with what I think are sensible parameters and distance function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It almost seems like this could be similar to financial modeling, where you deal with time series data. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="886" LastEditorUserId="84" LastEditDate="2014-06-17T16:17:46.027" LastActivityDate="2021-03-11T19:03:30.347" Title="How can I predict traffic based on previous time series data?" Tags="|machine-learning|time-series|" AnswerCount="7" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="407" PostTypeId="2" ParentId="313" CreationDate="2014-06-16T16:10:34.423" Score="8" Body="&lt;p&gt;Other answers recommended a good set of books about the mathematics behind data science. But as you mentioned, its not just mathematics and activities like data collection and inference from data has their own rules and theories, even if not being as rigorous as mathematical backgrounds (yet).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For theses parts, I suggest the book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0596157118&quot;&gt;Beautiful Data: The Stories Behind Elegant Data Solutions&lt;/a&gt; which contains twenty case-study like chapters written by people really engaged with real world data analysis problems. It does not contain any mathematics, but explores areas like collecting data, finding practical ways of using data in analyses, scaling and selecting the best solutions very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another really interesting book is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1449362931&quot;&gt;Thinking with Data: How to Turn Information into Insights&lt;/a&gt;, which is not technical (=programming tutorial) either, but covers important topics on how to really use the data science power in decision making and real world problems.&lt;/p&gt;&#xA;" OwnerUserId="227" LastActivityDate="2014-06-16T16:10:34.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="408" PostTypeId="2" ParentId="406" CreationDate="2014-06-16T16:34:50.317" Score="17" Body="&lt;p&gt;The problem with models like KNN is that they do not take into account seasonality (time-dependent variations in trend).  To take those into account, you should use Time Series analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For count data, such as yours, you can use generalized linear auto-regressive moving average models (GLARMA).  Fortunately, there is an R package that implements them (&lt;a href=&quot;http://cran.r-project.org/web/packages/glarma/index.html&quot;&gt;glarma&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/packages/glarma/vignettes/glarma.pdf&quot;&gt;vignette&lt;/a&gt; is a good resource for the theory behind the tool.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-06-16T16:34:50.317" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="409" PostTypeId="2" ParentId="406" CreationDate="2014-06-16T17:26:17.697" Score="10" Body="&lt;p&gt;I think Christopher's answers above are entirely sensible.  As an alternate approach (or perhaps just in addition to the advise he's given), I might start by just visualizing the data a bit to try get a rough sense of what's going on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't already done this, you might try adding a date's month and day of week as features -- if you end up sticking with KNN, this will help the model pick up seasonality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a different way of taking this on, you might consider starting with a really, really basic model (like OLS).. these often go a long way in generating reasonable predictions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the more we know about your data, the easier it will be for us to help generate suggestions -- What time frame are you observing?  What are the features you're currently using?  etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps --&lt;/p&gt;&#xA;" OwnerUserId="889" LastActivityDate="2014-06-16T17:26:17.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="410" PostTypeId="1" AcceptedAnswerId="414" CreationDate="2014-06-16T18:08:38.623" Score="114" ViewCount="126192" Body="&lt;p&gt;I'm currently working on implementing Stochastic Gradient Descent, &lt;code&gt;SGD&lt;/code&gt;, for neural nets using back-propagation, and while I understand its purpose I have some questions about how to choose values for the learning rate.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent?&lt;/li&gt;&#xA;&lt;li&gt;If so, how do you use this information to inform your decision about a value?&lt;/li&gt;&#xA;&lt;li&gt;If it's not what sort of values should I choose, and how should I choose them?&lt;/li&gt;&#xA;&lt;li&gt;It seems like you would want small values to avoid overshooting, but how do you choose one such that you don't get stuck in local minima or take to long to descend?&lt;/li&gt;&#xA;&lt;li&gt;Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In short: How do I choose the learning rate for SGD?&lt;/p&gt;&#xA;" OwnerUserId="890" LastEditorUserId="28175" LastEditDate="2018-01-17T14:59:36.183" LastActivityDate="2023-10-24T06:25:47.887" Title="Choosing a learning rate" Tags="|machine-learning|neural-network|deep-learning|optimization|hyperparameter|" AnswerCount="11" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="411" PostTypeId="1" AcceptedAnswerId="415" CreationDate="2014-06-16T19:14:38.553" Score="11" ViewCount="7609" Body="&lt;p&gt;It seems as though most languages have some number of scientific computing libraries available. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python has &lt;code&gt;Scipy&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Rust&lt;/code&gt; has &lt;code&gt;SciRust&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;C++&lt;/code&gt; has several including &lt;code&gt;ViennaCL&lt;/code&gt; and &lt;code&gt;Armadillo&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Java&lt;/code&gt; has &lt;code&gt;Java Numerics&lt;/code&gt; and &lt;code&gt;Colt&lt;/code&gt; as well as several other&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not to mention languages like &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;Julia&lt;/code&gt; designed explicitly for scientific computing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With so many options how do you choose the best language for a task? Additionally which languages will be the most performant? &lt;code&gt;Python&lt;/code&gt; and &lt;code&gt;R&lt;/code&gt; seem to have the most traction in the space, but logically a compiled language seems like it would be a better choice. And will anything ever outperform &lt;code&gt;Fortran&lt;/code&gt;? Additionally compiled languages tend to have GPU acceleration, while interpreted languages like &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;Python&lt;/code&gt; don't. What should I take into account when choosing a language, and which languages provide the best balance of utility and performance? Also are there any languages with significant scientific computing resources that I've missed?&lt;/p&gt;&#xA;" OwnerUserId="890" LastEditorUserId="890" LastEditDate="2014-06-16T19:22:00.133" LastActivityDate="2014-06-20T07:11:40.053" Title="Best languages for scientific computing" Tags="|efficiency|statistics|tools|knowledge-base|" AnswerCount="3" CommentCount="17" ClosedDate="2014-06-26T01:58:15.697" ContentLicense="CC BY-SA 3.0" />
  <row Id="412" PostTypeId="1" AcceptedAnswerId="446" CreationDate="2014-06-16T19:48:31.797" Score="44" ViewCount="6581" Body="&lt;h1&gt;Motivation&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I work with datasets that contain personally identifiable information (PII) and sometimes need to share part of a dataset with third parties, in a way that doesn't expose PII and subject my employer to liability. Our usual approach here is to withhold data entirely, or in some cases to reduce its resolution; e.g., replacing an exact street address with the corresponding county or census tract. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that certain types of analysis and processing must be done in-house, even when a third party has resources and expertise more suited to the task. Since the source data is not disclosed, the way we go about this analysis and processing lacks transparency. As a result, any third party's ability to perform QA/QC, adjust parameters or make refinements may be very limited.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Anonymizing Confidential Data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;One task involves identifying individuals by their names, in user-submitted data, while taking into account errors and inconsistencies. A private individual might be recorded in one place as &quot;Dave&quot; and in another as &quot;David,&quot; commercial entities can have many different abbreviations, and there are always some typos. I've developed scripts based on a number of criteria that determine when two records with non-identical names represent the same individual, and assign them a common ID. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point we can make the dataset anonymous by withholding the names and replacing them with this personal ID number. But this means the recipient has almost no information about e.g. the strength of the match. We would prefer to be able to pass along as much information as possible without divulging identity.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What Doesn't Work&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For instance, it would be great to be able to encrypt strings while preserving edit distance. This way, third parties could do some of their own QA/QC, or choose to do further processing on their own, without ever accessing (or being able to potentially reverse-engineer) PII. Perhaps we match strings in-house with edit distance &amp;lt;= 2, and the recipient wants to look at the implications of tightening that tolerance to edit distance &amp;lt;= 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the only method I am familiar with that does this is &lt;a href=&quot;http://www.techrepublic.com/blog/it-security/cryptographys-running-gag-rot13/&quot; rel=&quot;noreferrer&quot;&gt;ROT13&lt;/a&gt; (more generally, any &lt;a href=&quot;https://en.wikipedia.org/wiki/Caesar_cipher&quot; rel=&quot;noreferrer&quot;&gt;shift cipher&lt;/a&gt;), which hardly even counts as encryption; it's like writing the names upside down and saying, &quot;Promise you won't flip the paper over?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another &lt;strong&gt;bad&lt;/strong&gt; solution would be to abbreviate everything. &quot;Ellen Roberts&quot; becomes &quot;ER&quot; and so forth. This is a poor solution because in some cases the initials, in association with public data, will reveal a person's identity, and in other cases it's too ambiguous; &quot;Benjamin Othello Ames&quot; and &quot;Bank of America&quot; will have the same initials, but their names are otherwise dissimilar. So it doesn't do either of the things we want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An inelegant alternative is to introduce additional fields to track certain attributes of the name, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----+----+-------------------+-----------+--------+&#xA;| Row | ID | Name              | WordChars | Origin |&#xA;+-----+----+-------------------+-----------+--------+&#xA;| 1   | 17 | &quot;AMELIA BEDELIA&quot;  | (6, 7)    | Eng    |&#xA;+-----+----+-------------------+-----------+--------+&#xA;| 2   | 18 | &quot;CHRISTOPH BAUER&quot; | (9, 5)    | Ger    |&#xA;+-----+----+-------------------+-----------+--------+&#xA;| 3   | 18 | &quot;C J BAUER&quot;       | (1, 1, 5) | Ger    |&#xA;+-----+----+-------------------+-----------+--------+&#xA;| 4   | 19 | &quot;FRANZ HELLER&quot;    | (5, 6)    | Ger    |&#xA;+-----+----+-------------------+-----------+--------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I call this &quot;inelegant&quot; because it requires anticipating which qualities might be interesting and it's relatively coarse. If the names are removed, there's not much you can reasonably conclude about the strength of the match between rows 2 &amp;amp; 3, or about the distance between rows 2 &amp;amp; 4 (i.e., how close they are to matching).&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Conclusion&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The goal is to transform strings in such a way that as many useful qualities of the original string are preserved as possible while obscuring the original string. Decryption should be impossible, or so impractical as to be effectively impossible, no matter the size of the data set. In particular, a method that preserves the edit distance between arbitrary strings would be very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found a couple papers that might be relevant, but they're a bit over my head:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.merl.com/publications/docs/TR2010-109.pdf&quot; rel=&quot;noreferrer&quot;&gt;Privacy Preserving String Comparisons Based on Levenshtein Distance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.uni-due.de/~hq0215/documents/2010/Bachteler_2010_An_Empirical_Comparison_Of_Approaches_To_Approximate_String_Matching_In_Private_Record_Linkage.pdf&quot; rel=&quot;noreferrer&quot;&gt;An Empirical Comparison of Approaches to Approximate String &#xA;Matching in Private Record Linkage&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="322" LastEditorUserId="322" LastEditDate="2015-12-07T17:44:55.910" LastActivityDate="2015-12-07T17:44:55.910" Title="How can I transform names in a confidential data set to make it anonymous, but preserve some of the characteristics of the names?" Tags="|data-cleaning|anonymization|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="413" PostTypeId="2" ParentId="130" CreationDate="2014-06-16T19:49:24.377" Score="18" Body="&lt;p&gt;Dimensionality reduction is typically choosing a basis or mathematical representation within which you can describe most but not all of the variance within your data, thereby retaining the relevant information, while reducing the amount of information necessary to represent it. There are a variety of techniques for doing this including but not limited to &lt;code&gt;PCA&lt;/code&gt;, &lt;code&gt;ICA&lt;/code&gt;, and &lt;code&gt;Matrix Feature Factorization&lt;/code&gt;. These will take existing data and reduce it to the most discriminative components.These all allow you to represent most of the information in your dataset with fewer, more discriminative features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature Selection is hand selecting features which are highly discriminative. This has a lot more to do with feature engineering than analysis, and requires significantly more work on the part of the data scientist. It requires an understanding of what aspects of your dataset are important in whatever predictions you're making, and which aren't. Feature extraction usually involves generating new features which are composites of existing features. Both of these techniques fall into the category of feature engineering. Generally feature engineering is important if you want to obtain the best results, as it involves creating information that may not exist in your dataset, and increasing your signal to noise ratio.&lt;/p&gt;&#xA;" OwnerUserId="890" LastEditorUserId="890" LastEditDate="2014-06-16T21:44:32.587" LastActivityDate="2014-06-16T21:44:32.587" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="414" PostTypeId="2" ParentId="410" CreationDate="2014-06-16T19:53:09.957" Score="86" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Is the learning rate related to the shape of the error gradient, as&#xA;it dictates the rate of descent?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;If so, how do you use this information to inform your decision about a value?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Adagrad is the most widely known of these and scales a global learning rate &lt;em&gt;η&lt;/em&gt; on each dimension based on l2 norm of the history of the error gradient &lt;em&gt;gt&lt;/em&gt; on each dimension:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/aP96K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://arxiv.org/pdf/1212.5701v1.pdf&quot;&gt;Adadelta&lt;/a&gt; is another such training algorithm which uses both the error gradient history like adagrad and the weight update history and has the advantage of &lt;em&gt;not having to set a learning rate at all&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;If it's not what sort of values should I choose, and how should I choose them?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Setting learning rates for plain SGD in neural nets is usually a&#xA;process of starting with a sane value such as 0.01 and then doing cross-validation&#xA;to find an optimal value. Typical values range over a few orders of&#xA;magnitude from 0.0001 up to 1.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;It seems like you would want small values to avoid overshooting, but&#xA;how do you choose one such that you don't get stuck in local minima&#xA;or take too long to descend? Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Usually, the value that's best is near the highest stable learning&#xA;rate and learning rate decay/annealing (either linear or&#xA;exponentially) is used over the course of training. The reason behind this is that early on there is a clear learning signal so aggressive updates encourage exploration while later on the smaller learning rates allow for more delicate exploitation of local error surface.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="548" LastEditorUserId="26596" LastEditDate="2016-12-05T19:57:08.293" LastActivityDate="2016-12-05T19:57:08.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="415" PostTypeId="2" ParentId="411" CreationDate="2014-06-16T20:10:12.167" Score="12" Body="&lt;p&gt;This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Performance&lt;/strong&gt;: Basically boils down to how quickly the language does matrix multiplication, as that is more or less the most important task in data science.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Scalability&lt;/strong&gt;: At least for me personally, this comes down to ease of building a distributed system. This is somewhere where languages like &lt;code&gt;Julia&lt;/code&gt; really shine.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Community&lt;/strong&gt;: With any language, you're really looking for an active community that can help you when you get stuck using whichever tool you're using. This is where &lt;code&gt;python&lt;/code&gt; pulls very far ahead of most other languages. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Flexibility&lt;/strong&gt;: Nothing is worse than being limited by the language that you use. It doesn't happen very often, but trying to represent graph structures in &lt;code&gt;haskell&lt;/code&gt; is a notorious pain, and &lt;code&gt;Julia&lt;/code&gt; is filled with a lot of code architectures pains as a result of being such a young language.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Ease of Use&lt;/strong&gt;: If you want to use something in a larger environment, you want to make sure that setup is a straightforward and it can be automated. Nothing is worse than having to set up a finnicky build on half a dozen machines.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are a ton of articles out there about performance and scalability, but in general you're going to be looking at a performance differential of maybe 5-10x between languages, which may or may not matter depending on your specific application. As far as GPU acceleration goes, &lt;code&gt;cudamat&lt;/code&gt; is a really seamless way of getting it working with &lt;code&gt;python&lt;/code&gt;, and the &lt;code&gt;cuda&lt;/code&gt; library in general has made GPU acceleration far more accessible than it used to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two primary metrics I use for both community and flexibility are to look at the language's package manager, and the language questions on a site like SO. If there are a large number of high-quality questions and answers, it's a good sign that the community is active. Number of packages and the general activity on those packages can also be a good proxy for this metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as ease of use goes, I am a firm believer that the only way to actually know is to actually set it up yourself. There's a lot of superstition around a lot of Data Science tools, specifically things like databases and distributed computing architecture, but there's no way to really know if something is easy or hard to setup up and deploy without just building it yourself.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-16T20:10:12.167" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="416" PostTypeId="2" ParentId="412" CreationDate="2014-06-16T20:12:39.130" Score="7" Body="&lt;p&gt;If feasible I would link related records (e.g., Dave, David, etc.) and replace them with a sequence number (1,2,3, etc.) or a &lt;a href=&quot;http://en.wikipedia.org/wiki/Salt_%28cryptography%29&quot; rel=&quot;nofollow&quot;&gt;salted&lt;/a&gt; &lt;a href=&quot;http://www.cse.yorku.ca/~oz/hash.html&quot; rel=&quot;nofollow&quot;&gt;hash of the string&lt;/a&gt; that is used to represent all related records (e.g., David instead of Dave).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that third parties need not have any idea what the real name is, otherwise you might as well give it to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit&lt;/strong&gt;: You need to define and justify what kind of operations the third party needs to be able to do. For example, what is wrong with using initials followed by a number (e.g., BOA-1, BOA-2, etc.) to disambiguate Bank of America from Benjamin Othello Ames? If that's too revealing, you could bin some of the letters or names; e.g., [A-E] -&gt; 1, [F-J] -&gt; 2, etc. so BOA would become 1OA, or [&quot;Bank&quot;, &quot;Barry&quot;, &quot;Bruce&quot;, etc.] -&gt; 1 so Bank of America is again 1OA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information see &lt;a href=&quot;http://en.wikipedia.org/wiki/K-anonymity&quot; rel=&quot;nofollow&quot;&gt;k-anonymity&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-06-24T05:05:21.900" LastActivityDate="2014-06-24T05:05:21.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="417" PostTypeId="2" ParentId="387" CreationDate="2014-06-16T20:18:50.123" Score="1" Body="&lt;p&gt;Blacklists aren't  have value for a number of reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;They're easy to set up and scale - it's just a key/value store, and you can probably just re-use some of your caching logic for the most basic implementation.&lt;/li&gt;&#xA;&lt;li&gt;Depending on the size and type of the spam attack, there will probably be some very specific terms or URLs being used. It's much faster to throw that term into a blacklist than wait for your model to adapt.&lt;/li&gt;&#xA;&lt;li&gt;You can remove items just as quickly as you added them.&lt;/li&gt;&#xA;&lt;li&gt;Everybody understands how they work and any admin can use them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The key to fighting spam is &lt;em&gt;monitoring&lt;/em&gt;. Make sure you have some sort of interface showing which items are on your blacklist, how often they've been hit in the last 10 minutes / hour / day / month, and the ability to easily add and remove items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll want to combine a number of different spam detection models and tactics. Neural nets seem to be a good suggestion, and I'd recommend looking at &lt;em&gt;user behavior patterns&lt;/em&gt; in addition to just content. Normal humans don't do things like send batches of 1,000 emails every 30 seconds for 12 consecutive hours.&lt;/p&gt;&#xA;" OwnerUserId="414" LastActivityDate="2014-06-16T20:18:50.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="418" PostTypeId="1" CreationDate="2014-06-17T00:16:24.287" Score="16" ViewCount="13430" Body="&lt;p&gt;I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary is diverse and data become too sparse. I also tried to classify the types of attributes separately and combine the results through meta-learning techniques, but it did not work well.&lt;/p&gt;&#xA;" OwnerUserId="900" LastActivityDate="2023-04-12T19:18:51.107" Title="Best way to classify datasets with mixed types of attributes" Tags="|machine-learning|classification|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="419" PostTypeId="2" ParentId="411" CreationDate="2014-06-17T00:19:09.773" Score="4" Body="&lt;p&gt;First you need to decide what you want to do, then look for the right tool for that task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very general approach is to use R for first versions and to see if your approach is correct. It lacks a little in speed, but has very powerful commands and addon libraries, that you can try almost anything with it:&#xA;&lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-project.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second idea is if you want to understand the algorithms behind the libraries, you might wanna take a look at the Numerical Recipies. They are available for different languages and free to use for learning. If you want to use them in commercial products, you need to ourchase a licence:&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Numerical_Recipes&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Numerical_Recipes&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the time performance will not be the issue but finding the right algorithms and parameters for them, so it is important to have a fast scripting language instead of a monster program that first needs to compile 10 mins before calculating two numbers and putting out the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a big plus in using R is that it has built-in functions or libraries for almost any kind of diagram you might wanna need to visualize your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you then have a working version, it is almost easy to port it to any other language you think is more performant.&lt;/p&gt;&#xA;" OwnerUserId="901" LastActivityDate="2014-06-17T00:19:09.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="420" PostTypeId="2" ParentId="418" CreationDate="2014-06-17T00:39:15.990" Score="6" Body="&lt;p&gt;It is hard to answer this question without knowing more about the data.  That said, I would offer the following advice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most machine learning techniques can handle mixed-type data.  Tree based methods (such as AdaBoost and Random Forests) do well with this type of data.  The more important issue is actually the dimensionality, about which you are correct to be concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest that you do something to reduce that dimensionality.  For example, look for the words or phrases that separate the data the best and discard the other words (note: tree based methods do this automatically).&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-06-17T00:39:15.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="421" PostTypeId="1" CreationDate="2014-06-17T04:31:34.067" Score="10" ViewCount="1696" Body="&lt;p&gt;Does anyone know some good tutorials on online machine learning technics?&#xA;I.e. how it can be used in real-time environments, what are key differences compared to normal machine learning methods etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPD: Thank you everyone for answers, by &quot;online&quot; I mean methods which can be trained in a real-time mode, based on a new inputs one by one.&lt;/p&gt;&#xA;" OwnerUserId="88" LastEditorUserId="88" LastEditDate="2014-06-18T07:17:45.110" LastActivityDate="2015-06-23T15:12:32.633" Title="Online machine learning tutorial" Tags="|machine-learning|education|beginner|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="422" PostTypeId="1" AcceptedAnswerId="510" CreationDate="2014-06-17T05:29:11.830" Score="28" ViewCount="27276" Body="&lt;p&gt;As an extension to our great list of &lt;a href=&quot;https://datascience.stackexchange.com/questions/155/publicly-available-datasets&quot;&gt;publicly available datasets&lt;/a&gt;, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the name of the social network;&lt;/li&gt;&#xA;&lt;li&gt;what kind of user information it provides (posts, profile, friendship network, ...);&lt;/li&gt;&#xA;&lt;li&gt;whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);&lt;/li&gt;&#xA;&lt;li&gt;whether it simply provides a snapshot of the whole dataset.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any suggestions and further characteristics to be added are very welcome.&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2021-02-09T04:27:20.537" Title="Publicly available social network datasets/APIs" Tags="|open-source|dataset|crawling|" AnswerCount="7" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="423" PostTypeId="1" CreationDate="2014-06-17T05:55:04.710" Score="4" ViewCount="847" Body="&lt;p&gt;I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have limited number of machines nearly ten machines having 200 GB of free space on hard disk on each. On the contrary, I wished to perform experiments on more than available nodes in order to measure scalability, &lt;strong&gt;more precisely&lt;/strong&gt;. Since I don't have any, I thought about using a commodity cluster. However, I'm not sure about the policies of usage, and I need to reliably measure execution times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there commodity services which will grant me that only my application would be running at a given time?&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorDisplayName="user1361" LastEditDate="2014-07-30T22:36:59.310" LastActivityDate="2014-08-02T14:55:37.347" Title="How to measure execution time on distributed system" Tags="|bigdata|scalability|distributed|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="424" PostTypeId="1" AcceptedAnswerId="440" CreationDate="2014-06-17T06:05:39.653" Score="23" ViewCount="5698" Body="&lt;p&gt;I recently saw a cool feature that &lt;a href=&quot;https://support.google.com/docs/answer/3543688?hl=en&quot;&gt;was once available&lt;/a&gt; in Google Sheets: you start by writing a few related keywords in consecutive cells, say: &quot;blue&quot;, &quot;green&quot;, &quot;yellow&quot;, and it automatically generates similar keywords (in this case, other colors). See more examples in &lt;a href=&quot;http://youtu.be/dlslNhfrQmw&quot;&gt;this YouTube video&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to reproduce this in my own program. I'm thinking of using Freebase, and it would work like this intuitively: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Retrieve the list of given words in Freebase;&lt;/li&gt;&#xA;&lt;li&gt;Find their &quot;common denominator(s)&quot; and construct a distance metric based on this;&lt;/li&gt;&#xA;&lt;li&gt;Rank other concepts based on their &quot;distance&quot; to the original keywords;&lt;/li&gt;&#xA;&lt;li&gt;Display the next closest concepts.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As I'm not familiar with this area, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a better way to do this?&lt;/li&gt;&#xA;&lt;li&gt;What tools are available for each step?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="906" LastEditorUserId="322" LastEditDate="2014-06-19T05:48:43.540" LastActivityDate="2020-08-06T16:18:05.960" Title="How to grow a list of related words based on initial keywords?" Tags="|nlp|text-mining|freebase|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="425" PostTypeId="2" ParentId="422" CreationDate="2014-06-17T06:32:22.883" Score="5" Body="&lt;p&gt;An example from germany: Xing a site similar to linkedin but limited to german speaking countries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link to it's developer central: &lt;a href=&quot;https://dev.xing.com/overview&quot;&gt;https://dev.xing.com/overview&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provides access to: User profiles, Conversations between users (limited to the user itself), Job advertisings, Contacts and Contacts of Contacts, news from the network and some geolocation api.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes it has an api, but I did not find information about the rate. But it seems to me, that some information is limited to the consent of the user.&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-06-17T06:32:22.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="426" PostTypeId="2" ParentId="421" CreationDate="2014-06-17T06:53:44.007" Score="0" Body="&lt;p&gt;There are plenty on youtube and here's a famous one by Andrew Ng from coursea: &lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/course/ml&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2014-06-17T06:53:44.007" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="427" PostTypeId="1" AcceptedAnswerId="436" CreationDate="2014-06-17T07:46:39.783" Score="5" ViewCount="1689" Body="&lt;p&gt;Is anyone using &lt;code&gt;Julia&lt;/code&gt; (&lt;a href=&quot;http://julialang.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://julialang.org/&lt;/a&gt;) for professional jobs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or using it instead of R, Matlab, or Mathematica? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a good language?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to predict next 5-10 years: Do you think it grow up enough to became such a standard in data science like R or similar?&lt;/p&gt;&#xA;" OwnerUserId="908" LastEditorUserId="31513" LastEditDate="2017-05-10T03:58:56.370" LastActivityDate="2017-05-10T03:58:56.370" Title="Does anyone use Julia programming language?" Tags="|tools|julia|" AnswerCount="2" CommentCount="1" ClosedDate="2014-06-17T16:26:05.453" ContentLicense="CC BY-SA 3.0" />
  <row Id="428" PostTypeId="2" ParentId="59" CreationDate="2014-06-17T09:33:37.230" Score="3" Body="&lt;p&gt;This critique is no longer justified:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it is true that most of the standard and most respected R libraries were restricted to in-memory computations, there is a growing number of specialized libraries to deal with data that doesn't fit into memory.&lt;br&gt;&#xA;For instance, for random forests on large datasets, you have the library &lt;code&gt;bigrf&lt;/code&gt;. More info here: &lt;a href=&quot;http://cran.r-project.org/web/packages/bigrf/&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/bigrf/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another area of growth is R's connectedness to big data environments like hadoop, which opens another world of possibilities.&lt;/p&gt;&#xA;" OwnerUserId="906" LastActivityDate="2014-06-17T09:33:37.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="429" PostTypeId="2" ParentId="422" CreationDate="2014-06-17T09:56:49.180" Score="8" Body="&lt;p&gt;It's not a social network per se, but Stackexchange publish their entire database dump periodically:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://archive.org/details/stackexchange&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stackexchange data dump hosted on the archive.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://meta.stackoverflow.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede&quot;&gt;Post describing the database dump schema&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can extract some social information by analyzing which users ask and answer to each other. One nice thing is that since posts are tagged, you can analyze sub-communities easily.&lt;/p&gt;&#xA;" OwnerUserId="846" LastEditorUserId="-1" LastEditDate="2017-03-20T09:39:09.267" LastActivityDate="2014-06-17T09:56:49.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="430" PostTypeId="1" AcceptedAnswerId="525" CreationDate="2014-06-17T10:37:22.987" Score="14" ViewCount="1615" Body="&lt;p&gt;I'm trying to understand how all the &quot;big data&quot; components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used for different types, but I'd like to get to know more about their interaction in applications, e.g. thinking machine learning for an app, webapp, online shop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have vistors/session, transaction data etc and store that; but if I want to make recommendations on the fly, I can't run slow map/reduce jobs for that on some big database of logs I have. Where can I learn more about the infrastructure aspects? I think I can use most of the tools on their own, but plugging them into each other seems to be an art of its own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any public examples/use cases etc available? I understand that the individual pipelines strongly depend on the use case and the user, but just examples will probably be very useful to me.&lt;/p&gt;&#xA;" OwnerUserId="913" LastEditorUserId="84" LastEditDate="2014-06-17T13:37:47.400" LastActivityDate="2014-06-23T13:36:51.493" Title="Looking for example infrastructure stacks/workflows/pipelines" Tags="|machine-learning|bigdata|efficiency|scalability|distributed|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="431" PostTypeId="2" ParentId="422" CreationDate="2014-06-17T12:37:17.150" Score="2" Body="&lt;p&gt;A small collection of such links can be found at &lt;a href=&quot;http://lgylym.github.io/big-graph/dataset.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Many of them are social graphs.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-06-17T12:37:17.150" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="432" PostTypeId="2" ParentId="421" CreationDate="2014-06-17T12:41:58.900" Score="5" Body="&lt;ol&gt;&#xA;&lt;li&gt;There is a very nice library of online machine learning algorithms from a group at NTU, called &lt;a href=&quot;http://libol.stevenhoi.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;LIBOL&lt;/code&gt;&lt;/a&gt;. This would be a very good place to start experimenting with the algorithms.&lt;br&gt;&#xA;The &lt;a href=&quot;http://libol.stevenhoi.org/LIBOL_manual.pdf&quot; rel=&quot;nofollow&quot;&gt;accompanying user guide&lt;/a&gt;, and &lt;a href=&quot;http://jmlr.org/papers/v15/hoi14a.html&quot; rel=&quot;nofollow&quot;&gt;associated JMLR publication&lt;/a&gt; are very nice introductions to the basic algorithms in this field.  &lt;/li&gt;&#xA;&lt;li&gt;Avrim Blum has an older and more technical &lt;a href=&quot;http://www.cs.cmu.edu/~avrim/Papers/survey.pdf&quot; rel=&quot;nofollow&quot;&gt;survey paper&lt;/a&gt; on online learning algorithms. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="241" LastEditorUserId="241" LastEditDate="2015-06-23T15:12:32.633" LastActivityDate="2015-06-23T15:12:32.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="433" PostTypeId="2" ParentId="430" CreationDate="2014-06-17T13:31:30.503" Score="8" Body="&lt;p&gt;One of the most detailed and clear explanations of setting up a complex analytics pipeline is from the folks over at &lt;a href=&quot;http://www.twitch.tv/&quot;&gt;Twitch&lt;/a&gt;.&lt;br&gt;&#xA;They give detailed motivations of each of the architecture choices for collection, transportation, coordination, processing, storage, and querying their data.&lt;br&gt;&#xA;Compelling reading! Find it &lt;a href=&quot;http://blog.twitch.tv/2014/04/twitch-data-analysis-part-1-the-twitch-statistics-pipeline/&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://blog.twitch.tv/2014/04/twitch-data-analysis-part-2-architectural-decisions/&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="241" LastActivityDate="2014-06-17T13:31:30.503" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="434" PostTypeId="2" ParentId="427" CreationDate="2014-06-17T13:44:43.807" Score="7" Body="&lt;p&gt;There is really no question here as you ask for pure conjectures but consider at least that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;this week has &lt;a href=&quot;http://juliacon.org/&quot;&gt;Julia Con&lt;/a&gt;, the first Julia conference&lt;/li&gt;&#xA;&lt;li&gt;you could search GitHub and/or the registered Julia modules&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="515" LastActivityDate="2014-06-17T13:44:43.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="435" PostTypeId="1" CreationDate="2014-06-17T13:46:06.367" Score="9" ViewCount="334" Body="&lt;p&gt;I have a huge dataset from a relational database which I need to create a classification model for. Normally for this situation I would use &lt;a href=&quot;http://en.wikipedia.org/wiki/Inductive_logic_programming&quot;&gt;Inductive Logic Programming&lt;/a&gt; (ILP), but due to special circumstances I can't do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other way to tackle this would be just to try to aggregate the values when I have a foreign relation. However, I have thousands of important and distinct rows for some nominal attributes (e.g.: A patient with a relation to several distinct drug prescriptions). So, I just can't do that without creating a new attribute for each distinct row of that nominal attribute, and furthermore most of the new columns would have NULL values if I do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any non-ILP algorithm that allows me to data mine relational databases without resorting to techniques like pivoting, which would create thousands of new columns?&lt;/p&gt;&#xA;" OwnerUserId="917" LastEditorUserId="84" LastEditDate="2014-06-17T14:26:40.380" LastActivityDate="2015-04-26T18:23:32.323" Title="Relational Data Mining without ILP" Tags="|data-mining|classification|relational-dbms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="436" PostTypeId="2" ParentId="427" CreationDate="2014-06-17T14:55:47.103" Score="15" Body="&lt;p&gt;I personally have used &lt;code&gt;Julia&lt;/code&gt; for a good number of professional projects, and while, as Dirk mentioned, this is purely conjecture, I can give some insights on where Julia really stands out. The question of whether or not these reasons will prove enough to have &lt;code&gt;Julia&lt;/code&gt; succeed as a language is anyone's guess.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Distributed Systems&lt;/strong&gt;: Julia is the easiest language I've ever dealt with in terms of building distributed systems. This is becoming more and more relevant in computing, and will potentially become a deciding factor, but the question of whether or not &lt;code&gt;Julia&lt;/code&gt;'a relative ease decides this is up for debate&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;JIT Performance&lt;/strong&gt;: Julia's JIT compiler is extremely fast, and while there is a lot of debate as to how accurate these benchmark numbers are, the &lt;a href=&quot;http://julialang.org/&quot;&gt;Julia Website&lt;/a&gt; shows a series of relevant benchmarks&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Community&lt;/strong&gt;: This is an area where &lt;code&gt;Julia&lt;/code&gt; just isn't quite there. The community that is there is generally supportive, but not quite as knowledgable as the &lt;code&gt;R&lt;/code&gt; or &lt;code&gt;python&lt;/code&gt; communities, which is a definite minus.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Extensibility&lt;/strong&gt;: This is another place where &lt;code&gt;Julia&lt;/code&gt; is currently lacking, there is a large disconnect between the implies code patterns that &lt;code&gt;Julia&lt;/code&gt; steers you toward and what it can actually support. The type system is currently overly bulky and difficult to use effectively.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Again, can't say what this means for the future, but these are just a couple of relevant points when it comes to evaluating &lt;code&gt;Julia&lt;/code&gt; in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-17T14:55:47.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="437" PostTypeId="1" AcceptedAnswerId="444" CreationDate="2014-06-17T18:13:46.230" Score="5" ViewCount="160" Body="&lt;p&gt;I think that Bootstrap can be useful in my work, where we have a lot a variables that we don't know the distribution of it. So, simulations could help.&#xA;What are good sources to learn about Bootstrap/other useful simulation methods?&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2014-06-17T22:29:36.720" Title="What are good sources to learn about Bootstrap?" Tags="|data-mining|statistics|education|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="439" PostTypeId="2" ParentId="412" CreationDate="2014-06-17T18:42:55.423" Score="8" Body="&lt;p&gt;Halfway through reading your question, I realized Levenshtein Distance could be a nice solution to your problem.  Its good to see that you have a link to a paper on the topic, let me see if I can shed some light into what a Levenshtein solution would look like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Levenshtein distance is used across many industries for entity resolution, what makes it useful is that it is a measure of the difference between two sequences.  In the case of string comparison it is just sequences characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could help solve your problem by allowing you to provide one number that gives a measure of how similar the text of another field is.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of a basic way of using Levenshtein with the data you gave:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/vpaAH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This provides an ok solution, the distance of 8 provides some indication of a relationship, and it is very PII compliant.  However, it is still not super useful, let see what happens if we do some text magic to take only the first initial of the first name and the full last name dropping anything in the middle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/JlPu9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see the Levenshtein distance of 0 is pretty indicative of a relationship.  Commonly data providers will combine a bunch of Levenshtein permutations of the first and last names with 1, 2, or all of the characters just to give some dimensionality as to how entities are related while still maintaining anonymity within the data.&lt;/p&gt;&#xA;" OwnerUserId="780" LastActivityDate="2014-06-17T18:42:55.423" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="440" PostTypeId="2" ParentId="424" CreationDate="2014-06-17T18:59:14.947" Score="16" Body="&lt;p&gt;The &lt;a href=&quot;https://code.google.com/p/word2vec/&quot;&gt;word2vec algorithm&lt;/a&gt; may be a good way to retrieve more elements for a list of similar words. It is an unsupervised &quot;deep learning&quot; algorithm that has previously been demonstrated with Wikipedia-based training data (helper scripts are provided on the Google code page).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are currently &lt;a href=&quot;https://code.google.com/p/word2vec/&quot;&gt;C&lt;/a&gt; and &lt;a href=&quot;http://radimrehurek.com/gensim/models/word2vec.html&quot;&gt;Python&lt;/a&gt; implementations. This &lt;a href=&quot;http://radimrehurek.com/2014/02/word2vec-tutorial&quot;&gt;tutorial&lt;/a&gt; by &lt;a href=&quot;http://radimrehurek.com/&quot;&gt;Radim Řehůřek&lt;/a&gt;, the author of the &lt;a href=&quot;http://radimrehurek.com/gensim/&quot;&gt;Gensim topic modelling library&lt;/a&gt;, is an excellent place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://radimrehurek.com/2014/02/word2vec-tutorial#single&quot;&gt;&quot;single topic&quot;&lt;/a&gt; demonstration on the tutorial is a good example of retreiving similar words to a single term (try searching on 'red' or 'yellow'). It should be possible to extend this technique to find the words that have the greatest overall similarity to a set of input words.&lt;/p&gt;&#xA;" OwnerUserId="922" LastActivityDate="2014-06-17T18:59:14.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="441" PostTypeId="1" AcceptedAnswerId="449" CreationDate="2014-06-17T20:48:35.267" Score="34" ViewCount="12987" Body="&lt;p&gt;With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've read through the introduction documentation for Spark, but I'm curious if anyone has encountered a problem that was more efficient and easier to solve with Spark compared to Hadoop.&lt;/p&gt;&#xA;" OwnerUserId="426" LastEditorUserId="94454" LastEditDate="2020-04-23T16:00:54.130" LastActivityDate="2020-04-23T16:00:54.130" Title="What are the use cases for Apache Spark vs Hadoop" Tags="|apache-spark|apache-hadoop|distributed|knowledge-base|cloud-computing|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="442" PostTypeId="2" ParentId="418" CreationDate="2014-06-17T21:17:31.210" Score="11" Body="&lt;p&gt;Christopher's answers seem very reasonable. In particular tree based methods do well with this sort of data because they branch on discriminative features. It's a little hard to say without knowing your specific application, but in general if you think that some of your features might be significantly more discriminative than others, you could try some dimensionality reduction techniques to clean this up a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you use a dimensionality reduction technique you end up getting a slightly more robust format for your feature vector (they generally end up being straight numerical vectors instead of mixed data types), which might let you leverage different methods. You could also look into hand engineering features. With properly hand engineered features &lt;code&gt;Random Forest&lt;/code&gt; will get you very close to state of the art on most tasks.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-17T21:17:31.210" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="443" PostTypeId="2" ParentId="412" CreationDate="2014-06-17T22:28:36.070" Score="6" Body="&lt;p&gt;One option (depending on your dataset size) is to just provide edit distances (or other measures of similarity you're using) as an additional dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate a set of unique names in the dataset&lt;/li&gt;&#xA;&lt;li&gt;For each name, calculate edit distance to each other name&lt;/li&gt;&#xA;&lt;li&gt;Generate an ID or irreversable hash for each name&lt;/li&gt;&#xA;&lt;li&gt;Replace names in the original dataset with this ID&lt;/li&gt;&#xA;&lt;li&gt;Provide matrix of edit distances between ID numbers as new dataset&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Though there's still a lot that could be done to deanonymise the data from these even. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. if &quot;Tim&quot; is known to be the most popular name for a boy, frequency counting of IDs that closely match the known percentage of Tims across the population might give that away.  From there you could then look for names with an edit distance of 1, and conclude that those IDs might refer to &quot;Tom&quot; or &quot;Jim&quot; (when combined with other info).&lt;/p&gt;&#xA;" OwnerUserId="474" LastEditorUserId="474" LastEditDate="2014-06-18T08:08:02.163" LastActivityDate="2014-06-18T08:08:02.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="444" PostTypeId="2" ParentId="437" CreationDate="2014-06-17T22:29:36.720" Score="4" Body="&lt;p&gt;A classic book is by B. Efron who created the technique:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bradley Efron; Robert Tibshirani (1994). An Introduction to the Bootstrap. Chapman &amp;amp; Hall/CRC. ISBN 978-0-412-04231-7.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="418" LastActivityDate="2014-06-17T22:29:36.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="445" PostTypeId="2" ParentId="370" CreationDate="2014-06-17T23:30:45.897" Score="5" Body="&lt;p&gt;Is your Masters in Computer Science?  Statistics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is 'data science' going to be at the center of your thesis?  Or a side topic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll assume your in Statistics and that you want to focus your thesis on a 'data science' problem.  If so, then I'm going to go against the grain and suggest that you &lt;em&gt;should not&lt;/em&gt; start with a data set or an ML method.  Instead, you should seek an interesting research problem that's poorly understood or where ML methods have not yet been proven successful, or where there are many competing ML methods but none seem better than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this data source: &lt;a href=&quot;http://snap.stanford.edu/data/&quot;&gt;Stanford Large Network Dataset Collection&lt;/a&gt;.  While you &lt;em&gt;could&lt;/em&gt; pick one of these data sets, make up a problem statement, and then run some list of ML methods, that approach really doesn't tell you very much about what &lt;em&gt;data science&lt;/em&gt; is all about, and in my opinion doesn't lead to a very good Masters thesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, you might do this: look for all the research papers that use ML on some specific category -- e.g. Collaboration networks (a.k.a. co-authorship). As you read each paper, try to find out what they &lt;em&gt;were&lt;/em&gt; able to accomplish with each ML method and what they weren't able to address.  Especially look for their suggestions for &quot;future research&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe they all use the same method, but never tried competing ML methods.  Or maybe they don't adequately validate their results, or maybe there data sets are small, or maybe their research questions and hypothesis were simplistic or limited.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most important: try to find out where this line of research is going.  Why are they even bothering to do this?  What is significant about it?  Where and why are they encountering difficulties?&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-17T23:30:45.897" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="446" PostTypeId="2" ParentId="412" CreationDate="2014-06-17T23:38:20.133" Score="19" Body="&lt;p&gt;One of the references I mentioned in the OP led me to a potential solution that seems quite powerful, described in &quot;Privacy-preserving record linkage using Bloom filters&quot; (&lt;a href=&quot;http://www.biomedcentral.com/1472-6947/9/41&quot;&gt;doi:10.1186/1472-6947-9-41&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A new protocol for privacy-preserving record linkage with encrypted identifiers allowing for errors in identifiers has been developed. The protocol is based on Bloom filters on q-grams of identifiers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The article goes into detail about the method, which I will summarize here to the best of my ability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bloom filter is a fixed-length series of bits storing the results of a fixed set of independent hash functions, each computed on the same input value. The output of each hash function should be an index value from among the possible indexes in the filter; i.e., if you have a 0-indexed series of 10 bits, hash functions should return (or be mapped to) values from 0 to 9. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The filter starts with each bit set to 0. After hashing the input value with each function from the set of hash functions, each bit corresponding to an index value returned by any hash function is set to 1. If the same index is returned by more than one hash function, the bit at that index is only set once. You could consider the Bloom filter to be a superposition of the set of hashes onto the fixed range of bits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The protocol described in the above-linked article divides strings into n-grams, which are in this case sets of characters. As an example, &lt;code&gt;&quot;hello&quot;&lt;/code&gt; might yield the following set of 2-grams:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&quot;_h&quot;, &quot;he&quot;, &quot;el&quot;, &quot;ll&quot;, &quot;lo&quot;, &quot;o_&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Padding the front and back with spaces seems to be generally optional when constructing n-grams; the examples given in the paper that proposes this method use such padding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each n-gram can be hashed to produce a Bloom filter, and this set of Bloom filters can be superimposed on itself (bitwise OR operation) to produce the Bloom filter for the string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the filter contains many more bits than there are hash functions or n-grams, arbitrary strings are relatively unlikely to produce exactly the same filter. However, the more n-grams two strings have in common, the more bits their filters will ultimately share. You can then compare any two filters &lt;code&gt;A, B&lt;/code&gt; by means of their Dice coefficient:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;D&lt;sub&gt;A, B&lt;/sub&gt; = 2h / (a + b)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;h&lt;/code&gt; is the number of bits that are set to 1 in both filters, &lt;code&gt;a&lt;/code&gt; is the number of bits set to 1 in &lt;em&gt;only&lt;/em&gt; filter A, and &lt;code&gt;b&lt;/code&gt; is the number of bits set to 1 in &lt;em&gt;only&lt;/em&gt; filter B. If the strings are exactly the same, the Dice coefficient will be 1; the more they differ, the closer the coefficient will be to &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the hash functions are mapping an indeterminate number of unique inputs to a small number of possible bit indexes, different inputs may produce the same filter, so the coefficient indicates only a &lt;em&gt;probability&lt;/em&gt; that the strings are the same or similar. The number of different hash functions and the number of bits in the filter are important parameters for determining the likelihood of false positives - pairs of inputs that are much less similar than the Dice coefficient produced by this method predicts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found &lt;a href=&quot;http://billmill.org/bloomfilter-tutorial/&quot;&gt;this tutorial&lt;/a&gt; to be very helpful for understanding the Bloom filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is some flexibility in the implementation of this method; see also &lt;a href=&quot;https://www.uni-due.de/~hq0215/documents/2010/Bachteler_2010_An_Empirical_Comparison_Of_Approaches_To_Approximate_String_Matching_In_Private_Record_Linkage.pdf&quot;&gt;this 2010 paper&lt;/a&gt; (also linked at the end of the question) for some indications of how performant it is in relation to other methods, and with various parameters.&lt;/p&gt;&#xA;" OwnerUserId="322" LastEditorUserId="322" LastEditDate="2014-06-18T15:08:44.507" LastActivityDate="2014-06-18T15:08:44.507" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="447" PostTypeId="1" CreationDate="2014-06-18T03:19:07.557" Score="9" ViewCount="257" Body="&lt;p&gt;I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there is a consistent amount of white noise in the background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have manually decoded these streams, but I was wondering if I could use some sort of machine learning technique to learn the encoding schemes. This would save a great deal of time manually recognizing these schemes. The difficulty is that different signals are encoded differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to build a model which can learn to decode more than one encoding scheme? How robust would such a model be, and what sort of techniques would I want to employ? &lt;a href=&quot;http://en.wikipedia.org/wiki/Independent_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Independent Component Analysis&lt;/a&gt; (ICA) seems like could be useful for isolating the frequency I care about, but how would I learn the encoding scheme?&lt;/p&gt;&#xA;" OwnerUserId="890" LastEditorUserId="84" LastEditDate="2014-06-19T14:55:59.483" LastActivityDate="2014-06-25T06:58:43.670" Title="Learning signal encoding" Tags="|machine-learning|data-mining|scalability|algorithms|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="448" PostTypeId="2" ParentId="441" CreationDate="2014-06-18T05:35:35.700" Score="1" Body="&lt;p&gt;Not sure about the YARN, but I think that Spark makes a real difference compared to Hadoop (advertised as 100 times faster) if data can fit nicely in the memory of the computational nodes. Simply because it avoids hard disk access. If data doesn't fit memory there's still some gain because of buffering.&lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2014-06-18T05:35:35.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="449" PostTypeId="2" ParentId="441" CreationDate="2014-06-18T07:27:22.677" Score="43" Body="&lt;p&gt;Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs &lt;em&gt;MapReduce&lt;/em&gt;? Because Spark runs on/with Hadoop, which is rather the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This advantage is very pronounced for iterative computations, which have tens of stages each of which is touching the same data. This is where things might be &quot;100x&quot; faster. For simple, one-pass ETL-like jobs for which MapReduce was designed, it's not in general faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another reason to use Spark is its nicer high-level language compared to MapReduce. It provides a functional programming-like view that mimics Scala, which is far nicer than writing MapReduce code. (Although you have to either use Scala, or adopt the slightly-less-developed Java or Python APIs for Spark). &lt;a href=&quot;http://crunch.apache.org&quot;&gt;Crunch&lt;/a&gt; and &lt;a href=&quot;http://cascading.org&quot;&gt;Cascading&lt;/a&gt; already provide a similar abstraction on top of MapReduce, but this is still an area where Spark is nice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally Spark has as-yet-young but promising subprojects for ML, graph analysis, and streaming, which expose a similar, coherent API. With MapReduce, you would have to turn to several different other projects for this (Mahout, Giraph, Storm). It's nice to have it in one package, albeit not yet 'baked'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would you not use Spark? &lt;a href=&quot;https://www.quora.com/Apache-Spark/Assuming-you-have-a-system-with-both-Hadoop-and-Spark-installed-say-under-Yarn-is-there-any-reason-to-use-Hadoop-map-reduce-instead-of-the-equivalent-Spark-commands&quot;&gt;paraphrasing&lt;/a&gt; myself:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Spark is primarily Scala, with ported Java APIs; MapReduce might be friendlier and more native for Java-based developers&lt;/li&gt;&#xA;&lt;li&gt;There is more MapReduce expertise out there now than Spark&lt;/li&gt;&#xA;&lt;li&gt;For the data-parallel, one-pass, ETL-like jobs MapReduce was designed for, MapReduce is lighter-weight compared to the Spark equivalent&lt;/li&gt;&#xA;&lt;li&gt;Spark is fairly mature, and so is YARN now, but Spark-on-YARN is still pretty new. The two may not be optimally integrated yet. For example until recently I don't think Spark could ask YARN for allocations based on number of cores? That is: MapReduce might be easier to understand, manage and tune&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-06-18T07:27:22.677" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="450" PostTypeId="2" ParentId="412" CreationDate="2014-06-18T14:15:37.957" Score="5" Body="&lt;p&gt;I'm not quite sure, but maybe  locality-sensitive hashing is a good solution. It does hashing of input data (in your case - names), so original strings would be preserved. On the other side, the main idea of LSH is to maximize hashes likelihood for similar items. There are a lot of different LSH-implementations. I tried &lt;a href=&quot;http://spdp.di.unimi.it/papers/pdcs04.pdf&quot;&gt;Nilsimsa-hash&lt;/a&gt; for comparing tweet texts, and it worked quite well. But I'm not sure, how well it will work in case of short strings (names) - this issue require testing. I tried your examples, and here is the result (name A, name B, &quot;distance&quot; - maximum is 120):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. AMELIA BEDELIA  - CHRISTOPH BAUER - 107&#xA;2. AMELIA BEDELIA  - C J BAUER       - 82&#xA;3. AMELIA BEDELIA  - FRANZ HELLER    - 91&#xA;4. CHRISTOPH BAUER - C J BAUER       - 81&#xA;5. CHRISTOPH BAUER - FRANZ HELLER    - 98&#xA;6. C J BAUER       - FRANZ HELLER    - 83&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you see, CHRISTOPH BAUER and C J BAUER turned up to be the closest pair. But difference is not significant.&#xA;And just for example - hash representation of these names:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AMELIA BEDELIA  6b208299602b5000c3005a048122a43a828020889042240005011c1880864502&#xA;CHRISTOPH BAUER 22226448000ab10102e2860b52062487ff0000928e0822ee106028016cc01237&#xA;C J BAUER       2282204100961060048050004400240006032400148000802000a80130402002&#xA;FRANZ HELLER    58002002400880080b49172044020008030002442631e004009195020ad01158&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-06-18T14:15:37.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="451" PostTypeId="1" AcceptedAnswerId="457" CreationDate="2014-06-18T14:36:07.980" Score="5" ViewCount="922" Body="&lt;p&gt;I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much &lt;em&gt;spread&lt;/em&gt; along the entire process as possible, I'm wondering if there any standard solutions to improve the performance of the message passing layer of my application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What changes/improvements could I apply to my code to reduce the time spent sending messages? For what it's worth, I'm communicating up to 10GB between 9 computing nodes, and the framework I'm using is implemented with OpenMPI.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-06-19T01:35:24.700" Title="How to speedup message passing between computing nodes" Tags="|efficiency|distributed|performance|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="452" PostTypeId="2" ParentId="406" CreationDate="2014-06-18T15:10:22.637" Score="3" Body="&lt;p&gt;As @Christopher Lauden mentioned above, time-series analysis is most appropriate for this sort of thing. If, however, you wished to do a more traditional &quot;machine learning approach&quot;, something that I have done in the past is to block up your data into overlapping windows of time as features, then use it to predict the next days (or weeks) traffic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your feature matrix would be something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t1 | t2 | ... | tN&#xA;t2 | t3 | ... | tN+1&#xA;t3 | t4 | ... | tN+2&#xA;...&#xA;tW | tW+1 | ... |tN+W&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;tI&lt;/code&gt; is the traffic on day &lt;code&gt;I&lt;/code&gt;. The feature you'll be predicting is the traffic on the day after the last column. In essence, use a window of traffic to predict the next day's traffic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any sort of ML model would work for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to the question, &quot;can you elaborate on how you use this feature matrix&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The feature matrix has values indicating past traffic over a period of time (for instance, hourly traffic over 1 week), and we use this to predict traffic for some specified time period in the future. We take our historic data and build a feature matrix of historic traffic and label this with the traffic at some period in the future (e.g. 2 days after the window in the feature). Using some sort of regression machine learning model, we can take historic traffic data, and try and build a model that can predict how traffic moved in our historic data set. The presumption is that future traffic will resemble past traffic.&lt;/p&gt;&#xA;" OwnerUserId="403" LastEditorUserId="403" LastEditDate="2014-06-19T14:32:43.503" LastActivityDate="2014-06-19T14:32:43.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="453" PostTypeId="1" CreationDate="2014-06-18T15:27:23.313" Score="13" ViewCount="542" Body="&lt;p&gt;I understand that compression methods may be split into two main sets: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;global&lt;/li&gt;&#xA;&lt;li&gt;local&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first set works regardless of the data being processed, i.e., they do not rely on any characteristic of the data, and thus need not to perform any preprocessing on any part of the dataset (before the compression itself). On the other hand, local methods analyze the data, extracting information that usually improves the compression rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While reading about some of these methods, I noticed that &lt;a href=&quot;http://en.wikipedia.org/wiki/Universal_code_%28data_compression%29#Universal_and_non-universal_codes&quot; rel=&quot;noreferrer&quot;&gt;the unary method is not universal&lt;/a&gt;, which surprised me since I thought &quot;globality&quot; and &quot;universality&quot; referred to the same thing. The unary method does not rely on characteristics of the data to yield its encoding (i.e., it is a global method), and therefore it should be global/universal, shouldn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My primary questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the difference between universal and global methods? &lt;/li&gt;&#xA;&lt;li&gt;Aren't these classifications synonyms?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="84" LastEditorUserId="26596" LastEditDate="2016-12-02T17:49:10.983" LastActivityDate="2018-07-27T00:11:06.840" Title="What is the difference between global and universal compression methods?" Tags="|classification|algorithms|encoding|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="454" PostTypeId="1" AcceptedAnswerId="620" CreationDate="2014-06-18T15:48:19.497" Score="16" ViewCount="2478" Body="&lt;p&gt;I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on this data but it's difficult to create training datasets that contain enough examples of the positive class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the implications of doing a stratified sampling approach to normalize the number of positive and negative examples? In other words, is it a bad idea to, for instance, artificially inflate (by resampling) the number of positive class examples in the training set?&lt;/p&gt;&#xA;" OwnerUserId="403" LastEditorUserId="13727" LastEditDate="2015-11-22T16:25:06.530" LastActivityDate="2016-07-22T20:19:02.920" Title="What are the implications for training a Tree Ensemble with highly biased datasets?" Tags="|machine-learning|feature-selection|class-imbalance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="455" PostTypeId="1" AcceptedAnswerId="464" CreationDate="2014-06-18T16:21:12.203" Score="9" ViewCount="2920" Body="&lt;p&gt;Which freely available datasets can I use to train a text classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified our content based on a predefined bag of words we can recommend to him engaging content by getting his feedback on random number of posts already classified before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can use this info to recommend for him pulses labeled with those classes. But we found If we used a predefined bag of words not related to our content the feature vector will be full of zeros, also categories may be not relevant to our content. so for those reasons we tried another solution that will be clustering our content not classifying it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="960" LastEditorUserId="553" LastEditDate="2015-05-29T08:59:43.343" LastActivityDate="2016-07-05T08:40:00.757" Title="Suggest text classifier training datasets" Tags="|machine-learning|classification|dataset|clustering|text-mining|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="456" PostTypeId="2" ParentId="451" CreationDate="2014-06-18T17:22:46.207" Score="4" Body="&lt;p&gt;If you expect (or find) that nodes are requesting the same data more than once, perhaps you could benefit from a caching strategy? Especially where some data is used much more often than others, so you can target only the most frequently-used information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data is mutable, you also need a way to confirm that it hasn't changed since the last request that's less expensive than repeating the request.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is further complicated if each node has its own separate cache. Depending on the nature of your system and task(s), you could consider adding a node dedicated to serving information between the processing nodes, and building a single cache on that node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an example of when that &lt;em&gt;might&lt;/em&gt; be a good idea, let's suppose I retrieve some data from a remote data store over a low-bandwidth connection, and I have some task(s) requiring that data, which are distributed exclusively among local nodes. I definitely wouldn't want each node requesting information separately over that low-bandwidth connection, which another node might have previously requested. Since my local I/O is much less expensive than my I/O over the low-bandwidth connection, I might add a node between the processing nodes and the remote source that acts as an intermediate server. This node would take requests from the processing nodes, communicate with the remote data store, and cache frequently-requested data to minimize the use of that low-bandwidth connection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The core concepts here that &lt;em&gt;may&lt;/em&gt; be applicable to your specific case are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Eliminate or reduce redundant I/O;&lt;/li&gt;&#xA;&lt;li&gt;Take advantage of trade-offs between memory use and computation time;&lt;/li&gt;&#xA;&lt;li&gt;Not all I/O is created equal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="322" LastActivityDate="2014-06-18T17:22:46.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="457" PostTypeId="2" ParentId="451" CreationDate="2014-06-18T18:22:11.680" Score="7" Body="&lt;p&gt;Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be relatively marginal. Maybe as high as 2-5x, but I would be very surprised if they were any faster than that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the assumption that pure I/O (writing to/reading from memory) is &lt;em&gt;not&lt;/em&gt; your limiting factor (if it is, you're probably not going to get a lot faster), I would make a strong plug for &lt;a href=&quot;http://zeromq.org/&quot;&gt;zeromq&lt;/a&gt;. In the words of the creators:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We took a normal TCP socket, injected it with a mix of radioactive&#xA;  isotopes stolen from a secret Soviet atomic research project,&#xA;  bombarded it with 1950-era cosmic rays, and put it into the hands of a&#xA;  drug-addled comic book author with a badly-disguised fetish for&#xA;  bulging muscles clad in spandex. Yes, ØMQ sockets are the world-saving&#xA;  superheroes of the networking world.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While that may be a little dramatic, &lt;code&gt;zeromq&lt;/code&gt; sockets in my opinion are one of the most amazing pieces of software that the world of computer networks has put together in several years. I'm not sure what you're using for your message-passing layer right now, but if you're using something traditional like &lt;code&gt;rabbitmq&lt;/code&gt;, you're liable to see speedups of multiple orders of magnitude (personally noticed about 500x, but depends a lot of architecture)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out some basic benchmarks &lt;a href=&quot;http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/&quot;&gt;here.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-18T18:22:11.680" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="458" PostTypeId="1" AcceptedAnswerId="459" CreationDate="2014-06-18T19:48:54.883" Score="18" ViewCount="11352" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot;&gt;K-means&lt;/a&gt; is a well known algorithm for clustering, but there is also an online variation of such algorithm (online K-means). What are the pros and cons of these approaches, and when should each be preferred?&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="31513" LastEditDate="2017-04-26T16:24:21.560" LastActivityDate="2017-04-26T16:24:21.560" Title="K-means vs. online K-means" Tags="|clustering|algorithms|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="459" PostTypeId="2" ParentId="458" CreationDate="2014-06-18T20:07:05.017" Score="14" Body="&lt;p&gt;Online k-means (more commonly known as &lt;a href=&quot;https://stackoverflow.com/questions/3698532/online-k-means-clustering&quot;&gt;sequential k-means&lt;/a&gt;) and traditional k-means are very similar.  The difference is that online k-means allows you to update the model as new data is received.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Online k-means should be used when you expect the data to be received one by one (or maybe in chunks).  This allows you to update your model as you get more information about it.  The drawback of this method is that it is dependent on the order in which the data is received (&lt;a href=&quot;http://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/C/sk_means.htm&quot; rel=&quot;noreferrer&quot;&gt;ref&lt;/a&gt;).  &lt;/p&gt;&#xA;" OwnerUserId="178" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-06-18T20:07:05.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="460" PostTypeId="2" ParentId="454" CreationDate="2014-06-18T21:56:29.147" Score="5" Body="&lt;p&gt;A fast, easy an often effective way to approach this imbalance would be to randomly subsample the bigger class (which in your case is the negative class), run the classification N number of times with members from the two classes (one full and the other subsampled) and report the average metric values, the average being computed over N (say 1000) iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more methodical approach would be to execute the Mapping Convergence (MC) algorithm, which involves identifying a subset of strong negative samples with the help of a one-class classifier, such as OSVM or SVDD, and then iteratively execute binary classification on the set of strong negative and positive samples. More details of the MC algorithm can be found in this &lt;a href=&quot;http://link.springer.com/article/10.1007/s10994-005-1122-7#page-1&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-06-18T21:56:29.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="461" PostTypeId="1" CreationDate="2014-06-18T22:10:58.497" Score="13" ViewCount="2175" Body="&lt;p&gt;There's this side project I'm working on where I need to structure a solution to the following problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two groups of people (clients). Group &lt;code&gt;A&lt;/code&gt; intends to buy and group &lt;code&gt;B&lt;/code&gt; intends to sell a determined product &lt;code&gt;X&lt;/code&gt;. The product has a series of attributes &lt;code&gt;x_i&lt;/code&gt;, and my objective is to facilitate the transaction between &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; by matching their preferences. The main idea is to point out to each member of &lt;code&gt;A&lt;/code&gt; a corresponding in &lt;code&gt;B&lt;/code&gt; whose product better suits his needs, and vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some complicating aspects of the problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design, which is rare among the population and I can't predict. Can't previously list all the attributes;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Attributes might be continuous, binary, or non-quantifiable (ex: price, functionality, design);&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any suggestion on how to approach this problem and solve it in an automated way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also appreciate some references to other similar problems if possible.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Great suggestions! Many similarities in to the way I'm thinking of approaching the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main issue on mapping the attributes is that the level of detail to which the product should be described depends on each buyers. Let’s take an example of a car. The product “car” has lots and lots of attributes that range from its performance, mechanical structure, price etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I just want a cheap car, or an electric car. Ok, that's easy to map because they represent main features of this product. But let’s say, for instance, that I want a car with Dual-Clutch transmission or Xenon headlights. Well there might be many cars on the data base with this attributes but I wouldn't ask the seller to fill in this level of detail to their product prior to the information that there is someone looking them. Such a procedure would require every seller fill a complex, very detailed, form just try to sell his car on the platform. Just wouldn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But still, my challenge is to try to be as detailed as necessary in the search to make a good match. So the way I'm thinking is mapping main aspects of the product, those that are probably relevant to everyone, to narrow down de group of potential sellers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next step would be a “refined search”. In order to avoid creating a too detailed form I could ask buyers and sellers to write a free text of their specification. And then use some word matching algorithm to find possible matches. Although I understand that this is not a proper solution to the problem because the seller cannot “guess” what the buyer needs. But might get me close.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weighting criteria suggested is great. It allows me to quantify the level to which the seller matches the buyer’s needs. The scaling part might be a problem though, because the importance of each attribute varies from client to client. I'm thinking of using some kind of pattern recognition or just asking de buyer to input the level of importance of each attribute.&lt;/p&gt;&#xA;" OwnerUserId="986" LastEditorUserId="29575" LastEditDate="2017-11-06T08:07:09.260" LastActivityDate="2017-11-06T08:07:09.260" Title="Preference Matching Algorithm" Tags="|bigdata|text-mining|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="462" PostTypeId="2" ParentId="454" CreationDate="2014-06-18T22:27:06.503" Score="12" Body="&lt;p&gt;I would recommend training on more balanced subsets of your data. Training random forest on sets of randomly selected positive example with a similar number of negative samples. In particular if the discriminative features exhibit a lot of variance this will be fairly effective and avoid over-fitting. However in stratification it is important to find balance as over-fitting can become a problem regardless. I would suggest seeing how the model does with the whole data set then progressively increasing the ratio of positive to negative samples approaching an even ratio, and selecting for the one that maximizes your performance metric on some representative hold out data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper seems fairly relevant &lt;a href=&quot;http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf&quot;&gt;http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf&lt;/a&gt; it talks about a &lt;code&gt;weighted Random Forest&lt;/code&gt; which more heavily penalizes misclassification of the minority class.  &lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-06-18T22:27:06.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="463" PostTypeId="2" ParentId="461" CreationDate="2014-06-18T22:45:25.677" Score="9" Body="&lt;p&gt;My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, I don't think that you need to assume that the list of attributes is not finite. A standard and intuitive approach is to represent each attribute as an individual dimension in a vector space. Each product is then simply a point in this space. In that case, if you want to dynamically add more attributes you simply have to remap the product vectors into the new feature space (with additional dimensions).    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this representation, a seller is a point in the feature space with product attributes and a buyer is a point in the same feature space with the preference attributes. The task is then to find out the most similar buyer point for a given seller point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your dataset (i.e. the number of buyers/sellers) is not very large, you can solve this with a nearest neighbour approach implemented with the help of k-d trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For very large sized data, you can take an IR approach. Index the set of sellers (i.e. the product attributes) by treating each attribute as a separate term with the term-weight being set to the attribute value. A query in this case is a buyer which is also encoded in the term space as a query vector with appropriate term weights. The retrieval step would return you a list of top K most similar matches.&lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-06-18T22:45:25.677" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="464" PostTypeId="2" ParentId="455" CreationDate="2014-06-18T22:48:53.350" Score="14" Body="&lt;p&gt;Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them &lt;a href=&quot;http://web.ist.utl.pt/~acardoso/datasets/&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-06-18T22:48:53.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="465" PostTypeId="2" ParentId="369" CreationDate="2014-06-18T22:58:35.260" Score="9" Body="&lt;p&gt;nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RMSE (Root Mean Squared Error) is typically used to evaluate regression problems where the output (a predicted scalar value) is compared with the true scalar value output for a given data point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you are simply recommending a score (such as recommending a movie rating), then use RMSE. Whereas, if you are recommending a list of items (such as a list of related movies), then use nDCG.  &lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-06-18T22:58:35.260" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="466" PostTypeId="1" CreationDate="2014-06-18T23:02:54.200" Score="10" ViewCount="794" Body="&lt;p&gt;I tried to detect outliers in the energy gas consumption of some dutch buildings, building a neural network model. I have very bad results, but I can't find the reason. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not an expert so I would like to ask you what I can improve and what I'm doing wrong. This is the complete description: &lt;a href=&quot;https://github.com/denadai2/Gas-consumption-outliers&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/denadai2/Gas-consumption-outliers&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The neural network is a FeedFoward Network with Back Propagation. As described &lt;a href=&quot;http://nbviewer.ipython.org/github/denadai2/Gas-consumption-outliers/blob/master/3-%20Regression_NN.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; I splitted the dataset in a &quot;small&quot; dataset of 41'000 rows, 9 features and I tried to add more features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained the networks but the results have 14.14 RMSE, so it can't predict so well the gas consumptions, consecutively I can't run a good outlier detection mechanism. I see that in some papers that even if they predict daily or hourly consumption in the electric power, they have errors like MSE = 0.01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I improve? What am I doing wrong? Can you have a look of my description?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:55.813" LastActivityDate="2017-05-04T19:57:55.813" Title="Gas consumption outliers detection - Neural network project. Bad results" Tags="|neural-network|outlier|" AnswerCount="4" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="468" PostTypeId="1" CreationDate="2014-06-19T03:43:23.853" Score="11" ViewCount="1567" Body="&lt;p&gt;I'm working on a project and need resources to get me up to speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is around 35000 observations on 30 or so variables.  About half the variables are categorical with some having many different possible values, i.e. if you split the categorical variables into dummy variables you would have a lot more than 30 variables.  But still probably on the order of a couple of hundred max.  (n&gt;p).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The response we want to predict is ordinal with 5 levels (1,2,3,4,5).  Predictors are a mix of continuous and categorical, about half of each.  These are my thoughts/plans so far:&#xA;1.  Treat the response as continuous and run vanilla linear regression.&#xA;2.  Run nominal and ordinal logistic and probit regression&#xA;3.  Use MARS and/or another flavor of non-linear regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm familiar with linear regression.  MARS is well enough described by Hastie and Tibshirani.  But I'm at a loss when it comes to ordinal logit/probit, especially with so many variables and a big data set.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The r package &lt;a href=&quot;http://cran.r-project.org/web/packages/glmnetcr/index.html&quot;&gt;glmnetcr&lt;/a&gt; seems to be my best bet so far, but the documentation hardly suffices to get me where I need to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where can I go to learn more?&lt;/p&gt;&#xA;" OwnerUserId="994" LastEditorUserId="906" LastEditDate="2014-06-26T01:50:53.147" LastActivityDate="2014-06-27T13:22:22.577" Title="Learning ordinal regression in R?" Tags="|r|logistic-regression|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="469" PostTypeId="2" ParentId="468" CreationDate="2014-06-19T05:52:57.527" Score="6" Body="&lt;p&gt;I suggest this tutorial on ordered logit: &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/ologit.htm&quot;&gt;http://www.ats.ucla.edu/stat/r/dae/ologit.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It showcases the use of &lt;code&gt;polr&lt;/code&gt; in the &lt;code&gt;MASS&lt;/code&gt; package, and also explains the assumptions and how to interpret the results.&lt;/p&gt;&#xA;" OwnerUserId="906" LastActivityDate="2014-06-19T05:52:57.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="470" PostTypeId="2" ParentId="466" CreationDate="2014-06-19T06:09:43.963" Score="8" Body="&lt;p&gt;Just an idea - your data is highly seasonal: daily and weekly cycles are quite perceptible. So first of all, try to decompose your variables (gas and electricity consumption, temperature, and solar radiation). &lt;a href=&quot;http://www.r-bloggers.com/time-series-decomposition/&quot;&gt;Here is&lt;/a&gt; a nice tutorial on time series decomposition for R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After obtaining trend and seasonal components, the most interesting part begins. It's just an assumption, but I think, gas and electricity consumption variables would be quite predictable by means of time series analysis (e.g., &lt;a href=&quot;http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/tsa_arma_0.html&quot;&gt;ARIMA model&lt;/a&gt;). From my point of view, the most exiting part here is to try to predict residuals after decomposition, using available data (temperature anomalies, solar radiation, wind speed). I suppose, these residuals would be outliers, you are looking for. Hope, you will find this useful.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-06-19T06:09:43.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="471" PostTypeId="2" ParentId="455" CreationDate="2014-06-19T07:22:38.987" Score="7" Body="&lt;p&gt;One of the most widely used test collection for text categorization research (link below). I've used many times. Enjoy your exploration :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.daviddlewis.com/resources/testcollections/reuters21578/&quot;&gt;http://www.daviddlewis.com/resources/testcollections/reuters21578/&lt;/a&gt;&#xA;or&#xA;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection&quot;&gt;http://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="944" LastActivityDate="2014-06-19T07:22:38.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="473" PostTypeId="1" AcceptedAnswerId="484" CreationDate="2014-06-19T08:56:46.847" Score="23" ViewCount="13306" Body="&lt;p&gt;The usual definition of regression (as far as I am aware) is &lt;em&gt;predicting a continuous output variable from a given set of input variables&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is a binary classification algorithm, so it produces a categorical output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it really a regression algorithm? If so, why?&lt;/p&gt;&#xA;" OwnerUserId="922" LastEditorUserId="922" LastEditDate="2014-06-19T10:55:38.920" LastActivityDate="2021-03-11T20:07:53.137" Title="Is logistic regression actually a regression algorithm?" Tags="|algorithms|logistic-regression|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="474" PostTypeId="1" AcceptedAnswerId="481" CreationDate="2014-06-19T09:42:28.160" Score="4" ViewCount="739" Body="&lt;p&gt;In network structure, what is the difference between &lt;strong&gt;k-cliques&lt;/strong&gt; and &lt;strong&gt;p-cliques&lt;/strong&gt;, can anyone give a brief explaination with examples? Thanks in advanced!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;============================&#xA;&lt;br&gt;EDIT:&#xA;I found an online &lt;a href=&quot;http://open.umich.edu/sites/default/files/SI508-F08-Week7-Lab6.ppt&quot; rel=&quot;nofollow&quot;&gt;ppt&lt;/a&gt; while I am googling, please take a look on &lt;strong&gt;p.37&lt;/strong&gt; and &lt;strong&gt;p.39&lt;/strong&gt;, can you comment on them?&lt;/p&gt;&#xA;" OwnerUserId="957" LastEditorUserId="957" LastEditDate="2014-06-20T02:27:28.790" LastActivityDate="2014-06-20T02:27:28.790" Title="Network structure: k-cliques vs. p-cliques" Tags="|definitions|social-network-analysis|graphs|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="475" PostTypeId="2" ParentId="473" CreationDate="2014-06-19T09:50:53.657" Score="0" Body="&lt;p&gt;As you discuss, the definition of regression is predicting a continuous variable. &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logistic regression&lt;/a&gt; is a binary classifier.  Logistic regression is the application of a logit function on the output of a usual regression approach. Logit function turns &lt;span class=&quot;math-container&quot;&gt;$(-\infty, +\infty)$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$[0,1]$&lt;/span&gt;. I think it is just for historical reasons that it keeps that name.&lt;/p&gt;&#xA;&lt;p&gt;Saying something like &amp;quot;I did some regression to classify images. In particular I used logistic regression.&amp;quot; is wrong.&lt;/p&gt;&#xA;" OwnerUserId="418" LastEditorUserId="29169" LastEditDate="2021-03-11T20:07:53.137" LastActivityDate="2021-03-11T20:07:53.137" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="477" PostTypeId="1" CreationDate="2014-06-18T22:15:43.820" Score="2" ViewCount="793" Body="&lt;p&gt;There's this side project I'm working on where I need to structure a solution to the following problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two groups of people (clients). Group &quot;A&quot; intends to buy and group &quot;B&quot; intends to sell a determined product &quot;X&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The product has a series of attributes x_i and my objective is to facilitate the transaction between &quot;A&quot; e &quot;B&quot; by matching their preferences. The main idea is to point out to each member of &quot;A&quot; a corresponding in &quot;B&quot; who’s product better suits his needs, and vice versa. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some complicating aspects of the problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design which is rare among the population and I can’t predict. Can’t previously list all the attributes;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Attributes might be continuous, binary or non-quantifiable (ex: price, functionality, design).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any suggestion on how to approach this problem and solve it in an automated way?&lt;br&gt;&#xA;The idea is to really think out of the box here so feel free to &quot;go wild&quot; on your suggestions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also appreciate some references to other similar problems if possible. &lt;/p&gt;&#xA;" OwnerUserId="986" OwnerDisplayName="user986" LastActivityDate="2014-06-19T10:30:43.993" Title="Preference Matching Algorithm" Tags="|algorithms|" AnswerCount="1" CommentCount="3" ClosedDate="2014-06-26T05:31:41.193" ContentLicense="CC BY-SA 3.0" />
  <row Id="478" PostTypeId="2" ParentId="477" CreationDate="2014-06-19T03:20:00.847" Score="1" Body="&lt;p&gt;This can be a cross between machine learning and simple matching exercise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think X_i tend to be rather defined and finite, while A_i can be vague and not finite. From a pure algorithm perspective I would search for instances where X_i = A_i and store the results into a container of sort. The more hits for certain X'es where X_i_n = A_i_k the more points X scores. X'es are then presented to A in the order of points from best match to lowest match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Onto the machine learning mechanism, as the algorithm serves a lot of As (by that mean thousands and thousands, even millions) patterns will start to develop and certain combination of A_i's will be more prevalent, or in other words, worth more to other A_i's for a certain category of A. Using these patterns, the weighting of points will be re-balanced for higher chance of hitting the correct offers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind of like how a search engine works.&lt;/p&gt;&#xA;" OwnerDisplayName="pipja" LastActivityDate="2014-06-19T03:58:35.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="480" PostTypeId="2" ParentId="468" CreationDate="2014-06-19T10:35:37.190" Score="6" Body="&lt;p&gt;One fairly powerful R package for regression with an ordinal categorical response is VGAM, on the CRAN. The vignette contains some examples of ordinal regression, but admittedly I have never tried it on such a large dataset, so I cannot estimate how long it may take. You may find some additional material about VGAM on the author's &lt;a href=&quot;https://www.stat.auckland.ac.nz/~yee/VGAM/&quot; rel=&quot;noreferrer&quot;&gt;page&lt;/a&gt;. Alternatively you could take a look at Laura Thompson's &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=3&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0CDUQFjAC&amp;amp;url=http://home.comcast.net/~lthompson221/Splusdiscrete2.pdf&amp;amp;ei=8bmiU4HSFcjA7Abk4YHgDg&amp;amp;usg=AFQjCNHuBp2_nRpaPOFgdkcQWJGuSO9V6A&amp;amp;sig2=hAy4d3mu9WCJZulqxCzraw&quot; rel=&quot;noreferrer&quot;&gt;companion&lt;/a&gt; to Agresti's book &quot;Categorical Data Analysis&quot;. Chapter 7 of Thompson's book describes cumulative logit models, which are frequently used with ordinal responses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="1004" LastActivityDate="2014-06-19T10:35:37.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="481" PostTypeId="2" ParentId="474" CreationDate="2014-06-19T12:22:42.017" Score="2" Body="&lt;p&gt;In graph theory a clique indicates a fully connected set of nodes: &lt;a href=&quot;http://books.google.com/books?id=E3-OSVSPbU0C&amp;amp;pg=PA40&amp;amp;lpg=PA40&amp;amp;dq=%22graph%20theory%22,%20%22p-clique%22&amp;amp;source=bl&amp;amp;ots=smbhcK-9AC&amp;amp;sig=X0v_EbqSqB4WBbudgPbqo_j1pvk&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ei=VNWiU8KaA9WxsQSQhYGIAg&amp;amp;ved=0CDEQ6AEwBA#v=onepage&amp;amp;q=%22graph%20theory%22,%20%22p-clique%22&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;as noted here&lt;/a&gt;, a p-clique simply indicates a clique comoprised of p nodes. A k-clique is an undirected graph and a number k, and the output is a clique of size k if one exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Clique_problem&quot; rel=&quot;nofollow&quot;&gt;Clique Problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="59" LastActivityDate="2014-06-19T12:22:42.017" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="482" PostTypeId="2" ParentId="473" CreationDate="2014-06-19T13:23:53.387" Score="11" Body="&lt;p&gt;&lt;strong&gt;Short Answer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event.  That we use it as a binary classifier is due to the interpretation of the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Detail&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is a type of generalize linear regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an ordinary linear regression model, a continuous outcome, &lt;code&gt;y&lt;/code&gt;, is modeled as the sum of the product of predictors and their effect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = b_0 + b_1 * x_1 + b_2 * x_2 + ... b_n * x_n + e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;e&lt;/code&gt; is the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generalized linear models do not model &lt;code&gt;y&lt;/code&gt; directly.  Instead, they use transformations to expand the domain of &lt;code&gt;y&lt;/code&gt; to all real numbers.  This transformation is called the link function.  For logistic regression the link function is the logit function (usually, see note below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logit function is defined as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ln(y/(1 + y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus the form of logistic regression is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ln(y/(1 + y)) = b_0 + b_1 * x_1 + b_2 * x_2 + ... b_n * x_n + e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;y&lt;/code&gt; is the probability of an event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that we use it as a binary classifier is due to the interpretation of the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: probit is another link function used for logistic regression but logit is the most widely used.&lt;/p&gt;&#xA;" OwnerUserId="178" LastEditorUserId="178" LastEditDate="2014-06-19T13:47:55.017" LastActivityDate="2014-06-19T13:47:55.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="483" PostTypeId="2" ParentId="406" CreationDate="2014-06-19T14:15:07.433" Score="3" Body="&lt;p&gt;You could try Neural Network. You can find 2 great explanations on how to apply NN on time series &lt;a href=&quot;https://stats.stackexchange.com/questions/10162/how-to-apply-neural-network-to-time-series-forecasting&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/questions/18670558/prediction-using-recurrent-neural-network-on-time-series-dataset&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that it is best practice to :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Deseasonalize/detrend the input data (so that the NN will not learn the seasonality).&lt;/li&gt;&#xA;&lt;li&gt;Rescale/Normalize the input data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Because what you are looking for is a regression problem, the activation functions should be &lt;code&gt;linear&lt;/code&gt; and not &lt;code&gt;sigmoid&lt;/code&gt; or &lt;code&gt;tanh&lt;/code&gt; and you aim to minimize the &lt;code&gt;sum-of-squares error&lt;/code&gt; (as opposition to the maximization of the &lt;code&gt;negative log-likelihood&lt;/code&gt; in a classification problem).&lt;/p&gt;&#xA;" OwnerUserId="968" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2014-06-19T14:15:07.433" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="484" PostTypeId="2" ParentId="473" CreationDate="2014-06-19T14:52:59.877" Score="36" Body="&lt;p&gt;Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In logistic regression, we are modeling the log odds, or logit, that an event occurs, which is a continuous quantity. If the probability that event $A$ occurs is $P(A)$, the odds are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{P(A)}{1 - P(A)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The log odds, then, are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log \left( \frac{P(A)}{1 - P(A)}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in linear regression, we model this with a linear combination of coefficients and predictors:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\operatorname{logit} = b_0 + b_1x_1 + b_2x_2 + \cdots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine we are given a model of whether a person has gray hair. Our model uses age as the only predictor. Here, our event A = a person has gray hair:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;log odds of gray hair = -10 + 0.25 * age&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...Regression! Here is some Python code and a plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%matplotlib inline&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;import seaborn as sns&#xA;&#xA;x = np.linspace(0, 100, 100)&#xA;&#xA;def log_odds(x):&#xA;    return -10 + .25 * x&#xA;&#xA;plt.plot(x, log_odds(x))&#xA;plt.xlabel(&quot;age&quot;)&#xA;plt.ylabel(&quot;log odds of gray hair&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/xR0OT.png&quot; alt=&quot;plot of the log odds for our toy example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's make it a classifier. First, we need to transform the log odds to get out our probability $P(A)$. We can use the sigmoid function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(A) = \frac1{1 + \exp(-\text{log odds}))}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot(x, 1 / (1 + np.exp(-log_odds(x))))&#xA;plt.xlabel(&quot;age&quot;)&#xA;plt.ylabel(&quot;probability of gray hair&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/hSpCa.png&quot; alt=&quot;plot of the probability of gray hair for our toy example&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last thing we need to make this a classifier is to add a decision rule. One very common rule is to classify a success whenever $P(A) &amp;gt; 0.5$. We will adopt that rule, which implies that our classifier will predict gray hair whenever a person is older than 40 and will predict non-gray hair whenever a person is under 40.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression works great as a classifier in more realistic examples too, but before it can be a classifier, it must be a regression technique!&lt;/p&gt;&#xA;" OwnerUserId="1011" LastEditorUserId="18364" LastEditDate="2016-05-02T13:15:30.030" LastActivityDate="2016-05-02T13:15:30.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="485" PostTypeId="2" ParentId="14" CreationDate="2014-06-19T16:07:29.907" Score="8" Body="&lt;p&gt;My answer would be no. I consider Data mining to be one of the miscellaneous fields in Data science. Data Mining is mostly considered on yielding questions rather than answering them. It is often termed as &quot;detecting something new&quot;, when compared to Data science, where the data scientist try to solve complex problems to be able to reach their end results. However both terms have many commonalities between them.&#xA;For example..if u have an agricultural land where u aim to find the affected plants..Here spatial data mining plays a key role in doing this job.There are good chances that you may end  up with not only finding out the affected plants in the land but also the extent to which they are affected.......this is something that is not possible with data science.&lt;/p&gt;&#xA;" OwnerUserId="1015" LastEditorUserId="1015" LastEditDate="2014-06-20T17:36:05.023" LastActivityDate="2014-06-20T17:36:05.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="487" PostTypeId="2" ParentId="422" CreationDate="2014-06-19T18:01:36.120" Score="6" Body="&lt;p&gt;A good list of publicly available social network datasets can be found on the Stanford Network Analysis Project website: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://snap.stanford.edu/data/&quot;&gt;SNAP datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The site contains internet social network data (Facebook, Twitter, Google Plus), Citation networks for academic journals, co-purchasing networks from Amazon and several others kinds of networks. They have directed, undirected, and bipartite graphs and all datasets are snapshots that can be downloaded in compressed form.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-19T18:01:36.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="488" PostTypeId="1" AcceptedAnswerId="489" CreationDate="2014-06-19T18:02:24.650" Score="12" ViewCount="16093" Body="&lt;p&gt;I thought that generalized linear model (GLM) would be considered a statistical model, but a friend told me that some papers classify it as a machine learning technique. Which one is true (or more precise)? Any explanation would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1021" LastEditorUserId="21" LastEditDate="2015-07-08T11:37:50.907" LastActivityDate="2016-12-05T11:43:00.267" Title="Is GLM a statistical or machine learning model?" Tags="|machine-learning|statistics|glm|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="489" PostTypeId="2" ParentId="488" CreationDate="2014-06-19T18:05:51.070" Score="22" Body="&lt;p&gt;A GLM is absolutely a statistical model, but statistical models and machine learning techniques are not mutually exclusive. In general, statistics is more concerned with inferring parameters, whereas in machine learning, prediction is the ultimate goal.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-19T18:05:51.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="491" PostTypeId="2" ParentId="461" CreationDate="2014-06-19T18:24:32.023" Score="7" Body="&lt;p&gt;As suggested, &lt;strong&gt;going wild&lt;/strong&gt;. First of all, correct me if I’m wrong:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Just a few features exist for each unique product;&lt;/li&gt;&#xA;&lt;li&gt;There is no ultimate features list, and clients are able to add new features to their products.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If so, constructing full product-feature table could be computational expensive. And final data table would be extremely sparse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first step is narrowing customers (products) list for matching. Let’s build a bipartite graph, where sellers would be type-1 nodes, and buyers would be type-2 nodes. Create an edge between any seller and buyer every time they reference a similar product feature, as in the following sketch:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/q1JTN.png&quot; alt=&quot;graph&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the above graph, for every unique seller’s product you can select only buyers who are interested in features that match the product (it’s possible to filter &lt;strong&gt;at least one&lt;/strong&gt; common feature, match the full set of features, or set a threshold level). But certainly, that’s not enough. The next step is to compare feature values, as described by the seller and buyer. There are a lot of variants (e.g., k-Nearest-Neighbors). But why not try to solve this question using the existing graph? Let’s add weights to the edges:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;for continuous features (e.g., price): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=weight%20%3D%20%5Cfrac%7B%7Cprice_%7Bsell%7D%20-%20price_%7Bbuy%7D%7C%7D%7B%5Cmax%28price%29%20-%20%5Cmin%28price%29%7D&quot; alt=&quot;price_weight&quot;&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;for binary and non-quantifiable features - just logical biconditional:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=weight%20%3D%20feature_%7Bsell%7D%20%5Cleftrightarrow%20feature_%7Bbuy%7D&quot; alt=&quot;feature_weight&quot;&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The main idea here is to &lt;strong&gt;scale&lt;/strong&gt; every feature to the interval &lt;code&gt;[0, 1]&lt;/code&gt;. Additionally, we can use feature coefficients to determine most important features. E.g., assuming price is twice as important as availability of some rare function: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=adj.price.weight%20%3D%202%20%5Ctimes%20price.weight&quot; alt=&quot;adj_w_1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=adj.feature.weight%20%3D%201%20%5Ctimes%20feature.weight&quot; alt=&quot;adj_w_2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the final steps is simplifying the graph structure and reducing many edges to one edge with weight equal to the sum of the previously calculated weights of each feature. With such a reduced structure every pair of customers/products could have only one edge (no parallel edges). So, to find the best deal for exact seller you just need to select connected buyers with max weighted edges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Future challenge: introduce a cheap method for weighting edges on first step :)&lt;/p&gt;&#xA;" OwnerUserId="941" LastEditorUserId="322" LastEditDate="2014-06-20T03:28:55.463" LastActivityDate="2014-06-20T03:28:55.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="492" PostTypeId="1" AcceptedAnswerId="2367" CreationDate="2014-06-19T19:29:57.797" Score="27" ViewCount="14132" Body="&lt;p&gt;I'm looking to use google's word2vec implementation to build a named entity recognition system.  I've heard that recursive neural nets with back propagation through structure are well suited for named entity recognition tasks, but I've been unable to find a decent implementation or a decent tutorial for that type of model. Because I'm working with an atypical corpus, standard NER tools in NLTK and similar have performed very poorly, and it looks like I'll have to train my own system.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, what resources are available for this kind of problem?  Is there a standard recursive neural net implementation available?&lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="21" LastEditDate="2017-05-19T16:11:58.100" LastActivityDate="2020-08-05T08:41:02.810" Title="Word2Vec for Named Entity Recognition" Tags="|machine-learning|python|neural-network|nlp|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="493" PostTypeId="2" ParentId="406" CreationDate="2014-06-19T19:33:12.067" Score="2" Body="&lt;p&gt;Well, firstly, I would not even use things like Machine learning without having in depth knowledge. Simplistic things I would do if I had this time series is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Write sql queries to understand which of the times you have the busiest, average and low foot traffic.&lt;/li&gt;&#xA;&lt;li&gt;Then try to visualize the whole time series, and you could use basic pattern matching algorithms to pick up patterns.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This two things will help you understand what your data set is telling you. Then, with that in hand, you will probably be in a better state to use machine learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I'm currently working in building something on time series, and using time series analysis will help you much more than machine learning. For example, there are pattern recognition algorithms that you can use that uses every day data to show patterns, and ones which use up to as much as 3 to 6 months of data to catch a pattern.&lt;/p&gt;&#xA;" OwnerUserId="1029" LastEditorUserId="84" LastEditDate="2014-06-20T00:34:48.087" LastActivityDate="2014-06-20T00:34:48.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="494" PostTypeId="1" CreationDate="2014-06-19T19:33:40.620" Score="5" ViewCount="253" Body="&lt;p&gt;My data set is formatted like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User-id | Threat_score&#xA;aaa       45&#xA;bbb       32&#xA;ccc       20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The list contains the top 100 users with the highest threat scores. I generate such a list monthly and store each month's list in its own file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three things I would like to get from this data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;1. Users who are consistently showing up in this list&lt;/ul&gt;&#xA;&#xA;&lt;ul&gt;2. Users who are consistently showing up in this list with high threat scores&lt;/ul&gt;&#xA;&#xA;&lt;ul&gt;3. Users whose threat scores are increasing very quickly&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am thinking a visual summary would be nice; each month (somehow) decide which users I want to plot on a graph of historic threat scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any known visualization techniques that deal with similar requirements?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I be transforming my current data to achieve what I am looking for?&lt;/p&gt;&#xA;" OwnerUserId="1028" LastEditorUserId="322" LastEditDate="2014-06-26T01:51:28.873" LastActivityDate="2014-06-26T01:51:28.873" Title="Techniques for trend extraction from unbalanced panel data" Tags="|visualization|dataset|graphs|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="495" PostTypeId="2" ParentId="494" CreationDate="2014-06-19T22:02:44.050" Score="6" Body="&lt;p&gt;I would add a third column called &lt;code&gt;month&lt;/code&gt; and then concatenate each list. So if you have a top 100 list for 5 months you will create one big table with 500 entries:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User-id | Threat_score | month&#xA;aaa       45             1&#xA;bbb       32             1&#xA;ccc       20             1&#xA;...       ...            ...&#xA;bbb       64             2&#xA;ccc       29             2&#xA;...       ...            ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, to answer your first question, you could simply count the occurrences of each user-id. For example, if user &lt;code&gt;bbb&lt;/code&gt; is in your concatenated table five times, then you know that person made your list all five months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer you second question, you could do a &lt;code&gt;group by&lt;/code&gt; operation to compute some aggregate function of the users. A &lt;code&gt;group by&lt;/code&gt; operation with an average function is a little crude and sensitive to outliers, but it would probably get you close to what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility for the third question is to compute the difference in threat score between month &lt;code&gt;n-1&lt;/code&gt; and month &lt;code&gt;n&lt;/code&gt;. That is, for each month (not including the first month) you subtract the user's previous threat score from the current threat score. You can make this a new column so your table would now look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User-id | Threat_score | month | difference&#xA;aaa       45             1       null&#xA;bbb       32             1       null&#xA;ccc       20             1       null&#xA;...       ...            ...     ...&#xA;bbb       64             2       32&#xA;ccc       29             2       9&#xA;...       ...            ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this table, you could again do a &lt;code&gt;group by&lt;/code&gt; operation to find people who consistently have a higher threat score than the previous month or you could simply find people with a large difference between the current month and the previous month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you suggest, visualizing this data is a really good idea. If you care about these threat scores over time (which I think you do), I strongly recommend a simple line chart, with month on the x-axis and threat score on the y-axis. It's not fancy, but it's extremely easy to interpret and should give you useful information about the trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of this stuff (not the visualization) can be done in SQL and all of it can be done in R or Python (and many other languages). Good luck!&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-19T22:02:44.050" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="496" PostTypeId="2" ParentId="494" CreationDate="2014-06-20T03:18:07.653" Score="3" Body="&lt;p&gt;You have three questions to answer and 100 records per month to analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this size, I'd recommend doing analysis in a simple SQL database or a spreadsheet to start off with.  The first two questions are fairly easy to figure out.  The third is a little more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd definitely add a column for month and group all of that data together into a spreadsheet or database table given the questions you want to answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;question 1. Users who are consistently showing up in this list&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In excel, this answer should help you out:  &lt;a href=&quot;https://superuser.com/questions/442653/ms-excel-how-count-occurence-of-item-in-a-list&quot;&gt;https://superuser.com/questions/442653/ms-excel-how-count-occurence-of-item-in-a-list&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a SQL database:  &lt;a href=&quot;https://stackoverflow.com/questions/2516546/select-count-duplicates&quot;&gt;https://stackoverflow.com/questions/2516546/select-count-duplicates&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;question 2. Users who are consistently showing up in this list with high risk score &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just adding a little complexity to the above.  For SQL, you would further qualify your query based on a minimum risk score value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In excel, a straight pivot isn't going to work, you'll have to copy the unique values in one column to another, then drag a CountIf function adjacent to each unique value, qualifying the countif function with a minimum risk score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;question 3. Users who have/reaching the high risk level very fast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A fast rise in risk level could be defined as the difference between two months being larger than a given value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each user record you want to know the previous month's threat value, or assume zero as the previous threat value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that difference is greater than your risk threshold, you want to include it in your report.  If not, they can be filtered from the list.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to do this month after month, I would spend the two hours it might take to automate a report after the first couple of months.  I'd throw all the data in a SQL database and write a quick script in perl or java to iterate through the 100 records, do the calculation, and output the users who crossed the threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I needed it to look pretty, I'd use a reporting tool.  I'm not particularly partial to any of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I needed to trend threshold values over time, I'd output the results for all people into a second table add records to that table each month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I just needed to do it once or twice, figuring out how to do it in excel by adding a new column using VLookUp and some basic math and a filter would probably be the fastest and easiest way to get it done.  I tend to avoid using excel for things I'll need to use with consistency because there are limits that you run into when your data gets sizeable.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-06-20T03:18:07.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="497" PostTypeId="1" AcceptedAnswerId="506" CreationDate="2014-06-20T03:18:59.477" Score="23" ViewCount="749" Body="&lt;p&gt;I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example scenario:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image you own a business that has an average of 100 walk-in customers every day. One day, you decide you want to increase the number of walk-in customers arriving at your store each day, so you pull a crazy stunt outside your store to get attention. Over the next week, you see on average 125 customers a day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over the next few months, you again decide that you want to get some more business, and perhaps sustain it a bit longer, so you try some other random things to get more customers in your store. Unfortunately, you are not the best marketer, and some of your tactics have little or no effect, and others even have a negative impact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methodology could I use to determine the probability that any one individual event positively or negatively impacted the number of walk-in customers? I am fully aware that correlation does not necessarily equal causation, but what methods could I use to determine the likely increase or decrease in your business's daily walk in client's following a specific event?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not interested in analyzing whether or not there is a correlation between your attempts to increase the number of walk-in customers, but rather whether or not any one single event, independent of all others, was impactful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize that this example is rather contrived and simplistic, so I will also give you a brief description of the actual data that I am using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attempting to determine the impact that a particular marketing agency has on their client's website when they publish new content, perform social media campaigns, etc. For any one specific agency, they may have anywhere from 1 to 500 clients. Each client has websites ranging in size from 5 pages to well over 1 million. Over the course of the past 5 year, each agency has annotated all of their work for each client, including the type of work that was done, the number of webpages on a website that were influenced, the number of hours spent, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the above data, which I have assembled into a data warehouse (placed into a bunch of star/snowflake schemas), I need to determine how likely it was that any one piece of work (any one event in time) had an impact on the traffic hitting any/all pages influenced by a specific piece of work. I have created models for 40 different types of content that are found on a website that describes the typical traffic pattern a page with said content type might experience from launch date until present. Normalized relative to the appropriate model, I need to determine the highest and lowest number of increased or decreased visitors a specific page received as the result of a specific piece of work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I have experience with basic data analysis (linear and multiple regression, correlation, etc), I am at a loss for how to approach solving this problem. Whereas in the past I have typically analyzed data with multiple measurements for a given axis (for example temperature vs thirst vs animal and determined the impact on thirst that increased temperate has across animals), I feel that above, I am attempting to analyze the impact of a single event at some point in time for a non-linear, but predictable (or at least model-able), longitudinal dataset. I am stumped :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!&lt;/p&gt;&#xA;" OwnerUserId="1047" LastEditorUserId="134" LastEditDate="2014-10-22T12:07:33.977" LastActivityDate="2019-02-15T11:30:40.717" Title="What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data" Tags="|machine-learning|data-mining|statistics|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="498" PostTypeId="2" ParentId="424" CreationDate="2014-06-20T04:09:04.230" Score="5" Body="&lt;p&gt;Have you considered a frequency-based approach exploiting simple word co-occurence in corpora? At least, that's what I've seen most folks use for this. I think it might be covered briefly in Manning and Schütze's book, and I seem to remember something like this as a homework assignment back in grad school...&lt;/p&gt;&#xA;&lt;p&gt;More background &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/automatic-thesaurus-generation-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;For this step:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Rank other concepts based on their &amp;quot;distance&amp;quot; to the original keywords;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;There are several semantic similarity metrics you could look into. &lt;a href=&quot;http://www.eecis.udel.edu/%7Etrnka/CISC889-11S/lectures/greenbacker-WordNet-Similarity.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s a link to some slides I put together for a class project using a few of these similarity metrics in WordNet.&lt;/p&gt;&#xA;" OwnerUserId="819" LastEditorUserId="98307" LastEditDate="2020-08-06T16:18:05.960" LastActivityDate="2020-08-06T16:18:05.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="499" PostTypeId="2" ParentId="128" CreationDate="2014-06-20T04:27:02.467" Score="30" Body="&lt;p&gt;Anecdotally, I've never been impressed with the output from hierarchical LDA. It just doesn't seem to find an optimal level of granularity for choosing the number of topics. I've gotten much better results by running a few iterations of regular LDA, manually inspecting the topics it produced, deciding whether to increase or decrease the number of topics, and continue iterating until I get the granularity I'm looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember: hierarchical LDA can't read your mind... it doesn't know what you actually intend to use the topic modeling for. Just like with k-means clustering, you should choose the k that makes the most sense for your use case.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-06-20T04:27:02.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="500" PostTypeId="2" ParentId="488" CreationDate="2014-06-20T06:49:25.237" Score="6" Body="&lt;p&gt;In addition to Ben's answer, the subtle distinction between statistical models and machine learning models is that, in statistical models, you explicitly decide the output equation structure prior to building the model. The model is built to compute the parameters/coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take linear model or GLM for example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = a1x1 + a2x2 + a3x3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your independent variables are x1, x2, x3 and the coefficients to be determined are a1,a2,a3. You define your equation structure this way prior to building the model and compute a1,a2,a3. If you believe that y is somehow correlated to x2 in a non-linear way, you could try something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = a1x1 + a2(x2)^2 + a3x3.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus, you put a restriction in terms of the output structure. Inherently statistical models are linear models unless you explicitly apply transformations like sigmoid or kernel to make them nonlinear (GLM and SVM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of machine learning models, you rarely specify output structure and algorithms like decision trees are inherently non-linear and work efficiently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contrary to what Ben pointed out, machine learning models aren't just about prediction, they do classification, regression etc which can be used to make predictions which are also done by various statistical models.&lt;/p&gt;&#xA;" OwnerUserId="514" LastActivityDate="2014-06-20T06:49:25.237" CommentCount="3" CommunityOwnedDate="2014-06-20T06:49:25.237" ContentLicense="CC BY-SA 3.0" />
  <row Id="501" PostTypeId="2" ParentId="411" CreationDate="2014-06-20T07:11:40.053" Score="5" Body="&lt;p&gt;The best language depends on what you want to do. First remark: don't limit yourself to one language. Learning a new language is always a good thing, but at some point you will need to choose. Facilities offered by the language itself are an obvious thing to keep into account &lt;em&gt;but&lt;/em&gt; in my opinion the following are more important:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;available libraries&lt;/strong&gt;: do you have to implement everything from scratch or can you reuse existing stuff? Note that this these libraries need not be in whatever language you are considering, as long as you can interface easily. Working in a language without library access won't help you get things done.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;number of experts&lt;/strong&gt;: if you want external developers or start working in a team, you have to consider how many people actually know the language. As an extreme example: if you decide to work in Brainfuck because you happen to like it, know that you will likely work alone. Many surveys exists that can help assess the popularity of languages, including the number of questions per language on SO.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;toolchain&lt;/strong&gt;: do you have access to &lt;em&gt;good&lt;/em&gt; debuggers, profilers, documentation tools and (if you're into that) IDEs?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am aware that most of my points favor established languages. This is from a 'get-things-done' perspective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, I personally believe it is far better to become proficient in a low level language and a high level language:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;low level: C++, C, Fortran, ... using which you can implement certain profiling hot spots &lt;em&gt;only if you need to&lt;/em&gt; because developing in these languages is typically slower (though this is subject to debate). These languages remain king of the hill in terms of critical performance and are likely to stay on top for a long time.&lt;/li&gt;&#xA;&lt;li&gt;high level: Python, R, Clojure, ... to 'glue' stuff together and do non-performance critical stuff (preprocessing, data handling, ...). I find this to be important simply because it is much easier to do rapid development and prototyping in these languages.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="119" LastActivityDate="2014-06-20T07:11:40.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="502" PostTypeId="2" ParentId="488" CreationDate="2014-06-20T10:01:03.363" Score="17" Body="&lt;p&gt;Regarding prediction, statistics and machine learning sciences started to solve mostly the same problem from different perspectives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically statistics assumes that the data were produced by a given stochastic model. So, from a statistical perspective, a model is assumed and given various assumptions the errors are treated and the model parameters and other questions are inferred. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning comes from a computer science perspective. The models are algorithmic and usually very few assumptions are required regarding the data. We work with hypothesis space and learning bias. The best exposition of machine learning I found is contained in Tom Mitchell's book called &lt;a href=&quot;http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a more exhaustive and complete idea regarding the two cultures you can read the Leo Breiman paper called &lt;a href=&quot;http://www.google.ie/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0CDIQFjAA&amp;amp;url=http://strimmerlab.org/courses/ss09/current-topics/download/breiman2001.pdf&amp;amp;ei=RQGkU_DgH8Le7AaRmID4DA&amp;amp;usg=AFQjCNGNnrlqadmBT2fZMT_NfoUQ1rEuow&amp;amp;sig2=g5qovUTsuuJUGGE64g0nrQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;Statistical Modeling: The Two Cultures&lt;/a&gt;     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However what must be added is that even if the two sciences started with different perspectives, both of them now now share a fair amount of common knowledge and techniques. Why, because the problems were the same, but the tools were different. So now machine learning is mostly treated from a statistical perspective (check the Hastie,Tibshirani, Friedman book &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Elements of Statistical Learning&lt;/a&gt; from a machine learning point of view with a statistical treatement, and perhaps Kevin P. Murphy 's book &lt;a href=&quot;http://mitpress.mit.edu/books/machine-learning-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning: A probabilistic perspective&lt;/a&gt;, to name just a few of the best books available today). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even the history of the development of this field show the benefits of this merge of perspectives. I will describe two events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is the creation of CART trees, which was created by Breiman with a solid statistical background. At approximately the same time, Quinlan developed ID3,C45,See5, and so on, decision tree suite with a more computer science background. Now both this families of trees and the ensemble methods like bagging and forests become quite similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second story is about boosting. Initially they were developed by Freund and Shapire when they discovered AdaBoost. The choices for designing AdaBoost were done mostly from a computational perspective. Even the authors did not understood well why it works. Only 5 years later Breiman (again!) described the adaboost model from a statistical perspective and gave an explanation for why that works. Since then, various eminent scientists, with both type of backgrounds, developed further those ideas leading to a Pleiads of boosting algorithms, like logistic boosting, gradient boosting, gentle boosting ans so on. It is hard now to think at boosting without a solid statistical background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generalized Linear Models is a statistical development. However new Bayesian treatments puts this algorithm also in machine learning playground. So I believe both claims could be right, since the interpretation and treatment of how it works could be different. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="26766" LastEditDate="2016-12-05T08:33:23.513" LastActivityDate="2016-12-05T08:33:23.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="503" PostTypeId="2" ParentId="497" CreationDate="2014-06-20T14:17:27.970" Score="2" Body="&lt;p&gt;&lt;strong&gt;Edit: Warning, i leave my message but my answer seems wrong, please check out the comment below!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not an expert but I guess the main problem is to answer this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has an/any event affected the number of hits on a certain day?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't know how to treat multiple events, so I would try to answer this question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does event X affected the number of hits on a certain day?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which can be answered using hypothesis testing with p-values (what scientist do to evaluate for instance if a medicine affects a disease or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using p-values, you could determinate if the number of hits in a certain day were mere random and acceptable under normal circumstances or that they must correspond to a change in your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read more about p-values in &lt;a href=&quot;http://www.openintro.org/stat/&quot; rel=&quot;nofollow&quot;&gt;Open Intro to Statistics Book&lt;/a&gt;, I've actually learn about them from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the other parts of the problem are how to identify your events and calculate the necessary parameters to answer your question (average/median, variance, etc.) and also how to keep that up-to-date and working.&lt;/p&gt;&#xA;" OwnerUserId="1057" LastEditorUserId="1057" LastEditDate="2014-06-20T19:39:41.533" LastActivityDate="2014-06-20T19:39:41.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="504" PostTypeId="1" CreationDate="2014-06-20T14:58:09.320" Score="7" ViewCount="256" Body="&lt;p&gt;I'm currently working with a dataset with a wide range of document lengths -- anywhere from a single word to a full page of text.  In addition, the grammatical structure and use of punctuation varies wildly from document to document.  The goal is to classify those documents into one of about 10-15 categories.  I'm currently using ridge regression and logistic regression for the task, and CV for the alpha values of ridge.  The feature vectors are tf-idf ngrams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently I've noticed that longer documents are much less likely to be categorized. Why might this be the case, and how can one &quot;normalize&quot; for this kind of variation?  As a more general question, how does one typically deal with diverse data sets?  Should documents be grouped based off of metrics like document length, use of punctuation, grammatical rigor, etc. and then fed through different classifiers?  &lt;/p&gt;&#xA;" OwnerUserId="684" LastActivityDate="2014-06-20T16:56:41.160" Title="Dealing with diverse text data" Tags="|classification|nlp|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="506" PostTypeId="2" ParentId="497" CreationDate="2014-06-20T16:09:41.637" Score="13" Body="&lt;p&gt;For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would encourage you &lt;em&gt;not&lt;/em&gt; to use p-values as they can be pretty misleading (&lt;a href=&quot;http://andrewgelman.com/2013/03/12/misunderstanding-the-p-value/&quot; rel=&quot;noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://occamstypewriter.org/boboh/2008/08/19/why_p_values_are_evil/&quot; rel=&quot;noreferrer&quot;&gt;2&lt;/a&gt;). My approach hinges on you being able to summarize traffic on a given page before and after some intervention. What you care about is the difference in the &lt;em&gt;rate&lt;/em&gt; before and after the intervention. That is, how does the number of hits per day change? Below, I explain a first stab approach with some simulated example data. I will then explain one potential pitfall (and what I would do about it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, let's think about one page before and after an intervention. Pretend the intervention increases hits per day by roughly 15%:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns&#xA;&#xA;def simulate_data(true_diff=0):&#xA;    #First choose a number of days between [1, 1000] before the intervention&#xA;    num_before = np.random.randint(1, 1001)&#xA;&#xA;    #Next choose a number of days between [1, 1000] after the intervention&#xA;    num_after = np.random.randint(1, 1001)&#xA;&#xA;    #Next choose a rate for before the intervention. How many views per day on average?&#xA;    rate_before = np.random.randint(50, 151)&#xA;&#xA;    #The intervention causes a `true_diff` increase on average (but is also random)&#xA;    rate_after = np.random.normal(1 + true_diff, .1) * rate_before&#xA;&#xA;    #Simulate viewers per day:&#xA;    vpd_before = np.random.poisson(rate_before, size=num_before)&#xA;    vpd_after = np.random.poisson(rate_after, size=num_after)&#xA;&#xA;    return vpd_before, vpd_after&#xA;&#xA;vpd_before, vpd_after = simulate_data(.15)&#xA;&#xA;plt.hist(vpd_before, histtype=&quot;step&quot;, bins=20, normed=True, lw=2)&#xA;plt.hist(vpd_after, histtype=&quot;step&quot;, bins=20, normed=True, lw=2)&#xA;plt.legend((&quot;before&quot;, &quot;after&quot;))&#xA;plt.title(&quot;Views per day before and after intervention&quot;)&#xA;plt.xlabel(&quot;Views per day&quot;)&#xA;plt.ylabel(&quot;Frequency&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/FJJqD.png&quot; alt=&quot;Distribution of hits per day before and after the intervention&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can clearly see that the intervention increased the number of hits per day, on average. But in order to quantify the difference in rates, we should use one company's intervention for multiple pages. Since the underlying rate will be different for each page, we should compute the percent change in rate (again, the rate here is hits per day).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's pretend we have data for &lt;code&gt;n = 100&lt;/code&gt; pages, each of which received an intervention from the same company. To get the percent difference we take (mean(hits per day before) - mean(hits per day after)) / mean(hits per day before):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 100&#xA;&#xA;pct_diff = np.zeros(n)&#xA;&#xA;for i in xrange(n):&#xA;    vpd_before, vpd_after = simulate_data(.15)&#xA;    # % difference. Note: this is the thing we want to infer&#xA;    pct_diff[i] = (vpd_after.mean() - vpd_before.mean()) / vpd_before.mean()&#xA;&#xA;plt.hist(pct_diff)&#xA;plt.title(&quot;Distribution of percent change&quot;)&#xA;plt.xlabel(&quot;Percent change&quot;)&#xA;plt.ylabel(&quot;Frequency&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CAitf.png&quot; alt=&quot;Distribution of percent change&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have the distribution of our parameter of interest! We can query this result in different ways. For example, we might want to know the mode, or (approximation of) the most likely value for this percent change:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def mode_continuous(x, num_bins=None):&#xA;    if num_bins is None:&#xA;        counts, bins = np.histogram(x)&#xA;    else:&#xA;        counts, bins = np.histogram(x, bins=num_bins)&#xA;&#xA;    ndx = np.argmax(counts)&#xA;    return bins[ndx:(ndx+1)].mean()&#xA;&#xA;mode_continuous(pct_diff, 20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I ran this I got 0.126, which is not bad, considering our true percent change is 15. We can also see the number of positive changes, which approximates the probability that a given company's intervention improves hits per day:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(pct_diff &amp;gt; 0).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, my result is 0.93, so we could say there's a pretty good chance that this company is effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a potential pitfall: Each page probably has some underlying trend that you should probably account for. That is, even without the intervention, hits per day may increase. To account for this, I would estimate a simple linear regression where the outcome variable is hits per day and the independent variable is day (start at day=0 and simply increment for all the days in your sample). Then subtract the estimate, y_hat, from each number of hits per day to de-trend your data. Then you can do the above procedure and be confident that a positive percent difference is not due to the underlying trend. Of course, the trend may not be linear, so use discretion! Good luck!&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-20T16:09:41.637" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="507" PostTypeId="2" ParentId="497" CreationDate="2014-06-20T16:29:27.673" Score="10" Body="&lt;p&gt;Back in my data analyst days this type of problem was pretty typical.  Basically, everyone in marketing would come up with a crazy idea that the sold to higher ups as the single event that would boost KPI's by 2000%.  The higher ups would approve them and then they would begin their &quot;test&quot;.  Results would come back, and management would dump it on the data analysts to determine what worked and who did it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short answer is you cant really know if it wasn't run as a random A/B style test on like time periods.  But I am very aware of how deficient that answer is, especially if the fact that a pure answer doesn't exist is irrelevant to the urgency of future business decisions.  Here are some of the techniques I would use to salvage the analysis in this situation, bear in mind this is more of an art then a science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Handles&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A handle is something that exists in the data that you can hold onto.  From what you are telling me in your situation you have a lot of info on who the marketing agency is, when they tried a tactic, and to which site they applied it to.  These are your starting point and information like this going to be the corner stone of your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Methodology&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The methodology is going to probably hold the strongest impact on which agencies are given credit for any and all gains so you are going to need to make sure that it is clearly outlines and all stake holders agree that it makes sense.  If you cant do that it is going to be difficult for people to trust your analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of this are conversions. Say the marketing department purchases some leads and they arrive at our landing page, we would track them for 3 days, if they made a purchase within that time we would count them as having been converted.  Why 3 days, why not 5 or 1?  Thats not important as long as everyone agrees, you now have a definition you can build off of.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Comparisons&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an ideal would you would have a nice A/B test to prove a definitive relationship, I am going to assume that you are running short on those, still, you can learn something from a simple comparison of like data.  When companies are trying to determine the efficacy of radio advertising they will often run ads on offset months in the same market, or for several months in one market and compare that with the results in a separate but similar market.  Its doesn't pass for science, but even with all that noise a strong results will almost always be noticeable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would combine these in your case to determine how long an event is given to register an effect.  Once you have the data from that time period run it against your modeled out traffic prediction, week over week growth, month over month etc. Which, can then allow a meaningful comparison between agencies, and across time periods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pragmatism&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aspiration is to be able to provide a deep understanding of cause and effect, but it is probably not realistic.  Because of how messy outside factors make your analysis, you are constantly going to run up against the question over and over again: Did this event raise volume/sales/click throughs, or would doing anything at all have had the same effect?  The best advise I can give for this is set very realistic goals for what you are looking to measure.  A good starting point is, within the methodology you have, which event had the largest impact.  Once you have those open your aperture from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have reasoned out all of these aspects you can go about building a general solution which can then be automated.  The advantage to designing your solution in this manner is that the business logic is already built in.  This will make your results much more approachable and intuitive to non-technical business leaders.&lt;/p&gt;&#xA;" OwnerUserId="780" LastActivityDate="2014-06-20T16:29:27.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="508" PostTypeId="2" ParentId="504" CreationDate="2014-06-20T16:56:41.160" Score="5" Body="&lt;p&gt;I'm not sure how you are applying a regression framework for document classification. The way I'd approach the problem is to apply a standard discriminative classification approach such as SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a discriminative classification approach the notion of similarity or inverse distance between data points (documents in this case) is pivotal. Fortunately for documents, there is a standard way of defining pairwise similarity. This is the standard &lt;a href=&quot;http://en.wikipedia.org/wiki/Cosine_similarity&quot;&gt;cosine similarity&lt;/a&gt; measure which makes use of document length normalization to take different document lengths into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, practically speaking, in cosine similarity you would work with relative term weights normalized by document lengths and hence document length diversity should not be a major issue in the similarity computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One also has to be careful when applying idf in term weights. If the number of documents is not significantly large the idf measure may be statistically imprecise thus adding noise to the term weights. It's also a standard practice to ignore stopwords and punctuations.&lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-06-20T16:56:41.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="509" PostTypeId="2" ParentId="130" CreationDate="2014-06-20T17:30:39.097" Score="-3" Body="&lt;p&gt;For example...if u have an agricultural land then selecting one particular area of that land would be feature selection.If u aim to find the affected plants in that area den u need to observe each plant based on a particular feature that is common in each plant so as to find the abnormalities...for this u would be considering feature extraction.In this example the original agricultural land corresponds to Dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="1015" LastEditorUserId="1015" LastEditDate="2014-07-09T17:18:44.043" LastActivityDate="2014-07-09T17:18:44.043" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="510" PostTypeId="2" ParentId="422" CreationDate="2014-06-20T17:34:51.633" Score="22" Body="&lt;p&gt;A couple of words about social networks APIs. About a year ago I wrote a review of popular social networks’ APIs for researchers. Unfortunately, it is in Russian. Here is a summary:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Twitter&lt;/strong&gt; (&lt;a href=&quot;https://dev.twitter.com/docs/api/1.1&quot;&gt;https://dev.twitter.com/docs/api/1.1&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;almost all data about tweets/texts and users is available;&lt;/li&gt;&#xA;&lt;li&gt;lack of sociodemographic data;&lt;/li&gt;&#xA;&lt;li&gt;great streaming API: useful for real time text processing;&lt;/li&gt;&#xA;&lt;li&gt;a lot of wrappers for programing languages;&lt;/li&gt;&#xA;&lt;li&gt;getting network structure (connections) is possible, but time-expensive (1 request per 1 minute).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Facebook&lt;/strong&gt; (&lt;a href=&quot;https://developers.facebook.com/docs/reference/api/&quot;&gt;https://developers.facebook.com/docs/reference/api/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rate limits: about 1 request per second;&lt;/li&gt;&#xA;&lt;li&gt;well documented, sandbox present;&lt;/li&gt;&#xA;&lt;li&gt;FQL (SQL-like) and «regular Rest» Graph API;&lt;/li&gt;&#xA;&lt;li&gt;friendship data and sociodemographic features present;&lt;/li&gt;&#xA;&lt;li&gt;a lot of data is beyond &lt;em&gt;event horizon&lt;/em&gt;: only friends' and friends' of friends data is more or less complete, almost nothing could be investigated about random user;&lt;/li&gt;&#xA;&lt;li&gt;some strange API bugs, and looks like nobody cares about it (e.g., some features available through FQL, but not through Graph API synonym).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Instagram&lt;/strong&gt; (&lt;a href=&quot;http://instagram.com/developer/&quot;&gt;http://instagram.com/developer/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rate limits: 5000 requests per hour;&lt;/li&gt;&#xA;&lt;li&gt;real-time API (like Streaming API for Twitter, but with photos) - connection to it is a little bit tricky: callbacks are used;&lt;/li&gt;&#xA;&lt;li&gt;lack of sociodemographic data;&lt;/li&gt;&#xA;&lt;li&gt;photos, filters data available;&lt;/li&gt;&#xA;&lt;li&gt;unexpected imperfections (e.g., it’s possible to collect only 150 comments to post/photo).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Foursquare&lt;/strong&gt; (&lt;a href=&quot;https://developer.foursquare.com/overview/&quot;&gt;https://developer.foursquare.com/overview/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rate limits: 5000 requests per hour;&lt;/li&gt;&#xA;&lt;li&gt;kingdom of geosocial data :)&lt;/li&gt;&#xA;&lt;li&gt;quite closed from researches because of privacy issues. To collect checkins data one need to build composite parser working with 4sq, bit.ly, and twitter APIs at once;&lt;/li&gt;&#xA;&lt;li&gt;again: lack of sociodemographic data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Google+&lt;/strong&gt; (&lt;a href=&quot;https://developers.google.com/+/api/latest/&quot;&gt;https://developers.google.com/+/api/latest/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;about 5 requests per second (try to verify);&lt;/li&gt;&#xA;&lt;li&gt;main methods: activities and people;&lt;/li&gt;&#xA;&lt;li&gt;like on Facebook, a lot of personal data for random user is hidden;&lt;/li&gt;&#xA;&lt;li&gt;lack of user connections data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And out-of-competition: I reviewed social networks for Russian readers, and #1 network here is &lt;a href=&quot;http://en.wikipedia.org/wiki/VK_%28social_network%29&quot;&gt;&lt;strong&gt;vk.com&lt;/strong&gt;&lt;/a&gt;. It’s translated to many languages, but popular only in Russia and other CIS countries. API docs link: &lt;a href=&quot;http://vk.com/dev/&quot;&gt;http://vk.com/dev/&lt;/a&gt;. And from my point of view, it’s the best choice for homebrew social media research. At least, in Russia. That’s why:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rate limits: 3 requests per second;&lt;/li&gt;&#xA;&lt;li&gt;public text and media data available;&lt;/li&gt;&#xA;&lt;li&gt;sociodemographic data available: for random user availability level is about 60-70%;&lt;/li&gt;&#xA;&lt;li&gt;connections between users are also available: almost all friendships data for random user is available;&lt;/li&gt;&#xA;&lt;li&gt;some special methods: e.g., there is a method to get online/offline status for exact user in realtime, and one could build schedule for his audience.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-06-20T17:34:51.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="511" PostTypeId="1" AcceptedAnswerId="513" CreationDate="2014-06-20T17:57:46.363" Score="13" ViewCount="11548" Body="&lt;p&gt;I wonder which type of model cross-validation to choose for classification problem: K-fold or random sub-sampling (bootstrap sampling)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My best guess is to use 2/3 of the data set (which is ~1000 items) for training and 1/3 for validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case K-fold gives only three iterations(folds), which is not enough to see stable average error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand I don't like random sub-sampling feature: that some items won't be ever selected for training/validation, and some will be used more than once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classification algorithms used: random forest &amp;amp; logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-06-27T17:36:39.530" LastActivityDate="2014-07-01T08:04:30.127" Title="Cross-validation: K-fold vs Repeated random sub-sampling" Tags="|cross-validation|sampling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="512" PostTypeId="2" ParentId="412" CreationDate="2014-06-20T18:56:52.110" Score="3" Body="&lt;p&gt;Here's an approach I didn't see mentioned: separate the process into two steps: the first step focused on encoding names so that alternative versions of the same name are encoded the same (or nearly the same), and the second step focused on making them anonymous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first step, you could use one of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Phonetic_encoding&quot; rel=&quot;nofollow&quot;&gt;Phonetic Algorithms (Soundex and variants)&lt;/a&gt;, applied to first name, last name, and initials in various orders.  (See &lt;a href=&quot;http://www.stevemorse.org/phonetics/bmpm2.htm&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;, also).  It's in this step where you resolve similarities vs. differences in names to balance false positives from false negatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second step, you can pick any hashing or cryptographic method you like, without concern for how that method affects name matching.  This gives you freedom to use a method that has the best characteristics for both performance, robustness, and anonymity.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-20T18:56:52.110" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="513" PostTypeId="2" ParentId="511" CreationDate="2014-06-20T19:15:52.093" Score="8" Body="&lt;p&gt;If you have an adequate number of samples and want to use all the data, then k-fold cross-validation is the way to go. Having ~1,500 seems like a lot but whether it is adequate for k-fold cross-validation also depends on the dimensionality of the data (number of attributes and number of attribute values). For example, if each observation has 100 attributes, then 1,500 observations is low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another potential downside to k-fold cross-validation is the possibility of a single, extreme outlier skewing the results. For example, if you have one extreme outlier that can heavily bias your classifier, then in a 10-fold cross-validation, 9 of the 10 partitions will be affected (though for random forests, I don't think you would have that problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random subsampling (e.g., bootstrap sampling) is preferable when you are either undersampled or when you have the situation above, where you don't want each observation to appear in k-1 folds.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2014-06-20T19:15:52.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="514" PostTypeId="2" ParentId="313" CreationDate="2014-06-20T20:03:51.247" Score="7" Body="&lt;p&gt;I like Amir Ali Akbari's suggestions, and I'll add a few of my own, focusing on topics and skills that are not adequately covered in most machine learning and data analysis books that focus on math and/or programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Cleaning:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1412988012&quot;&gt;Osborne 2012, &lt;em&gt;Best Practices in Data Cleaning&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1449321887&quot;&gt;McCallom 2012, &lt;em&gt;Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Bayesian Data Analysis (alternative to Fisher-style Null Hypothesis Significance Testing):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0123814855&quot;&gt;Kruschke 2011, &lt;em&gt;Doing Bayesian Data Analysis&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Inference in the face of uncertainty, incompleteness, contradictions, ambiguity, imprecision, ignorance, etc.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0810118211&quot;&gt;Schum &amp;amp; Starace, 2001, &lt;em&gt;The Evidential Foundations of Probabilistic Reasoning&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1844074749&quot;&gt;Bammer &amp;amp; Smithson 2008, &lt;em&gt;Uncertainty and Risk: Multidisciplinary Perspectives&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387969454&quot;&gt;Smithson 1989, &lt;em&gt;Ignorance and Uncertainty&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/Tradecraft%20Primer-apr09.pdf&quot;&gt;CIA 2008, &lt;em&gt;A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis&lt;/em&gt;&lt;/a&gt; (FREE! as PDF)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521671930&quot;&gt;Morgan &amp;amp; Winship 2007, &lt;em&gt;Counterfactuals and Causal Inference: Methods and Principles for Social Research&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Experiments:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691159270&quot;&gt;Glennerster &amp;amp; Takavarasha 2013, &lt;em&gt;Running Randomized Evaluations: A Practical Guide&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1107698006&quot;&gt;Dunning 2012, &lt;em&gt;Natural Experiments in the Social Sciences&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Simulation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0691125473&quot;&gt;Epstein 2006, &lt;em&gt;Generative Social Science: Studies in Agent-Based Computational Modeling&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0486477703&quot;&gt;Nelson 2010, &lt;em&gt;Stochastic Modeling: Analysis and Simulation&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Expert elicitation, probabilistic estimation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1118539273&quot;&gt;Hubbard 2014, &lt;em&gt;How to Measure Anything: Finding the Value of Intangibles in Business&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-20T20:03:51.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="516" PostTypeId="1" CreationDate="2014-06-21T04:59:06.620" Score="5" ViewCount="298" Body="&lt;p&gt;I'm working on multiclass logistic regression model with a large number of features (numFeatures &gt; 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg algorithm solves the problem quickly. However, I'm also experimenting with a different cost function and do not have a gradient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way to speed up the calculation process? E.g., is there a different algorithm or fmincg setting that I can use?&lt;/p&gt;&#xA;" OwnerUserId="985" LastEditorUserId="84" LastEditDate="2014-06-21T06:38:58.260" LastActivityDate="2014-06-22T13:18:38.760" Title="Efficient solution of fmincg without providing gradient?" Tags="|bigdata|data-mining|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="517" PostTypeId="2" ParentId="516" CreationDate="2014-06-21T07:02:10.687" Score="3" Body="&lt;p&gt;If you do not have a gradient available, but the problem is convex, you can use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method&quot; rel=&quot;nofollow&quot;&gt;Nelder-Mead simplex method&lt;/a&gt;. It is available in most optimization packages, for example in &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize&quot; rel=&quot;nofollow&quot;&gt;scipy.optimize&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="119" LastActivityDate="2014-06-21T07:02:10.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="518" PostTypeId="1" AcceptedAnswerId="583" CreationDate="2014-06-21T10:55:41.700" Score="-6" ViewCount="241" Body="&lt;p&gt;Please, could someone recommend a paper or blog post that describes the online k-means algorithm.&lt;/p&gt;&#xA;" OwnerUserId="960" LastEditorUserId="8432" LastEditDate="2017-08-14T13:29:11.063" LastActivityDate="2017-08-14T13:29:11.063" Title="Online k-means explanation" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ClosedDate="2014-06-26T01:41:16.660" ContentLicense="CC BY-SA 3.0" />
  <row Id="519" PostTypeId="2" ParentId="466" CreationDate="2014-06-21T12:59:15.373" Score="2" Body="&lt;p&gt;The main problem here is that even before attempting to apply anomaly detection algorithms, you are not getting good enough predictions of gas consumption using neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the main goal here is to reach the stage when anomaly detection algorithms could be used and you state that you have access to examples of successful application of linear regression for this problem, this approach could be more productive. One of the principles of successful machine learning application is that several different algorithms can be tried out before final selection based on results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It you choose to tune your neural network performance, &lt;a href=&quot;https://stackoverflow.com/questions/4617365/what-is-a-learning-curve-in-machine-learning&quot;&gt;learning curve&lt;/a&gt; plotting the effect of change in different hyperparameters on the error rate can be used. Hyperparameters that can be modified are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;number of features&lt;/li&gt;&#xA;&lt;li&gt;order of the polynomial&lt;/li&gt;&#xA;&lt;li&gt;regularization parameter &lt;/li&gt;&#xA;&lt;li&gt;number of layers in the network&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Best settings can be selected by the performance on cross validation set.&lt;/p&gt;&#xA;" OwnerUserId="454" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-06-21T13:36:29.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="520" PostTypeId="2" ParentId="466" CreationDate="2014-06-21T19:13:47.757" Score="3" Body="&lt;p&gt;In your training notebook you present results for training with 20 epochs. Have you tried varying that parameter, to see if it affects your performance? This is an important parameter for back-propagation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For estimating your model parameters, as user tomaskazemekas pointed out, plotting Learning Curves is a very good approach. In addition to that, you could also create a plot using a model parameter (e.g. training epochs or hidden layer size) vs. Training and Validation error. This will allow you to understand the bias/variance tradeoff, and help you pick a good value for your parameters. &lt;a href=&quot;http://www.astroml.org/sklearn_tutorial/practical.html#cross-validation-and-testing&quot; rel=&quot;nofollow&quot;&gt;Some info can be found here&lt;/a&gt;. Naturally, it is a good idea to keep a small percentage of your data for a (third) Test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, it seems that increasing the number of neurons in your model show no significant improvement for your RMSE. This suggests that you could also try with a simpler model, i.e. with less neurons and see how your model behaves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I would suggest (if you haven't done so already) trying a simple model with few or no parameters first e.g. Linear Regression, and compare your results with the literature, just as a sanity check. &lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2014-06-21T19:13:47.757" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="521" PostTypeId="2" ParentId="516" CreationDate="2014-06-22T12:02:52.773" Score="2" Body="&lt;p&gt;Conjugate Gradient -- the cg in Function MINimization (nonlinear) Conjugate Gradiant -- requires you to have a gradient function (or approximation) since that is a critical part of the algorithm itself: it needs to find the steepest descent direction quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;fminsearch&lt;/code&gt; implements Nelder-Mead, a nonlinear gradient-free method. Its convergence properties are not anywhere near as good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your cost function? Are there approximations that are differentiable (pref. twice so you can use the very powerful quasi-Newton methods)?&lt;/p&gt;&#xA;" OwnerUserId="1061" LastActivityDate="2014-06-22T12:02:52.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="522" PostTypeId="2" ParentId="516" CreationDate="2014-06-22T13:18:38.760" Score="2" Body="&lt;p&gt;I have been able to optimize very strange functions with simulated annealing, and it does not require a gradient. Instead, it uses random numbers in a way very similar to Markov Chain Monte Carlo, which helps it avoid getting stuck in local optima. A decent explanation that gives the intuition behind it can be found in this lecture: &lt;a href=&quot;http://iacs-courses.seas.harvard.edu/courses/am207/blog/lecture-13.html&quot; rel=&quot;nofollow&quot;&gt;Simulated Annealing&lt;/a&gt;. scipy 0.14 includes this algorithm in its optimization module: &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.anneal.html&quot; rel=&quot;nofollow&quot;&gt;scipy.optimize.anneal&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-22T13:18:38.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="523" PostTypeId="2" ParentId="466" CreationDate="2014-06-22T16:03:39.047" Score="2" Body="&lt;p&gt;In your notebooks, I did not see your neural network model, can you point which library is using, how many layers you have and what type of neural network are you using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your notebooks, it seems you are using the noisy and outlier dataset to train the neural network, I think you should train the neural network on the dataset that you do not have any outliers so that you could see the observation distance from the prediction of the neural network to label the observation either outlier or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote &lt;a href=&quot;http://bugra.github.io/work/notes/2014-03-31/outlier-detection-in-time-series-signals-fft-median-filtering/&quot; rel=&quot;nofollow&quot;&gt;couple&lt;/a&gt; of &lt;a href=&quot;http://bugra.github.io/work/notes/2014-05-11/robust-regression-and-outlier-detection-via-gaussian-processes/&quot; rel=&quot;nofollow&quot;&gt;things&lt;/a&gt; on outlier detection in time-series signals, your data is highly seasonal as sobach mentioned and you could use FFT(first link above) to get the overall trend in the signal. After you get the frequency component in the gas consumption, you could look at the high frequency components to get the outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you want to insist on using neural network for seasonal data, you may want to check recurrent neural networks out as they could incorporate the past observations better than a vanilla neural network, and supposedly may provide a better result for the data that you have. &lt;/p&gt;&#xA;" OwnerUserId="1096" LastActivityDate="2014-06-22T16:03:39.047" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="525" PostTypeId="2" ParentId="430" CreationDate="2014-06-22T18:35:22.207" Score="15" Body="&lt;p&gt;In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The common theme that these systems have is the separation of model training from model application. In production systems, model application needs to be fast, on the order of 100s of ms, but there is more freedom in how frequently fitted model parameters (or equivalent) need to be updated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People use a wide range of solutions for model training and deployment:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Build a model, then export and deploy it with PMML&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://nerds.airbnb.com/architecting-machine-learning-system-risk/&quot;&gt;AirBnB describes their model training&lt;/a&gt; in R/Python and deployment of PMML models via OpenScoring.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.cascading.org/projects/pattern/&quot;&gt;Pattern&lt;/a&gt; is project related to &lt;a href=&quot;http://www.cascading.org/&quot;&gt;Cascading&lt;/a&gt; that can consume PMML and deploy predictive models.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Build a model in MapReduce and access values in a custom system&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://codeascraft.com/2014/06/18/conjecture-scalable-machine-learning-in-hadoop-with-scalding/&quot;&gt;Conjecture is an open source project from Etsy&lt;/a&gt; that allows for model training with &lt;a href=&quot;https://github.com/twitter/scalding&quot;&gt;Scalding&lt;/a&gt;, an easier to use scala wrapper around MapReduce, and  deployment via Php.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.kiji.org/&quot;&gt;Kiji is an open source project from WibiData&lt;/a&gt; that allows for real-time model scoring (application) as well as functioanlity for persisting user data and training models on that data via &lt;a href=&quot;https://github.com/twitter/scalding&quot;&gt;Scalding.&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use an online system that allows for continuously updating model parameters.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1242610&quot;&gt;Google released a great paper about an online collaborative filtering&lt;/a&gt; they implemented to deal with recommendations in Google News.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="406" LastEditorUserId="406" LastEditDate="2014-06-22T19:54:28.380" LastActivityDate="2014-06-22T19:54:28.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="526" PostTypeId="2" ParentId="430" CreationDate="2014-06-22T18:50:07.193" Score="2" Body="&lt;p&gt;Chapter 1 of Practical Data Science with R (&lt;a href=&quot;http://www.manning.com/zumel/&quot; rel=&quot;nofollow&quot;&gt;http://www.manning.com/zumel/&lt;/a&gt;) has a great breakdown of the data science process, including team roles and how they relate to specific tasks. The book follows the models laid out in the chapter by referencing which stages/personnel this or that particular task would be performed by.  &lt;/p&gt;&#xA;" OwnerUserId="1103" LastEditorUserId="1103" LastEditDate="2014-06-23T13:36:51.493" LastActivityDate="2014-06-23T13:36:51.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="527" PostTypeId="1" AcceptedAnswerId="535" CreationDate="2014-06-22T21:45:35.800" Score="9" ViewCount="2259" Body="&lt;p&gt;I have a load of documents, which have a load of key value pairs in them. The key might not be unique so there might be multiple keys of the same type with different values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to compare the similarity of the keys between 2 documents. More specifically the string similarity of these values. I am thinking of using something like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm&quot; rel=&quot;noreferrer&quot;&gt;Smith-Waterman Algorithm&lt;/a&gt; to compare the similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I've drawn a picture of how I'm thinking about representing the data - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/VC4em.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values in the cells are the result of the smith-waterman algorithm (or some other string similarity metric).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image that this matrix represents a key type of &quot;things&quot; I then need to add the &quot;things&quot; similarity score into a vector of 0 or 1. Thats ok.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can't figure out is how I determine if the matrix is similar or not similar - ideally I want to convert the matrix to an number between 0 and 1 and then I'll just set a threshold to score it as either 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how I can create a score of the matrix? Does anyone know any algorithms that do this type of thing (obviously things like how smith waterman works is kind of applicable).&lt;/p&gt;&#xA;" OwnerUserId="1107" LastEditorUserId="84" LastEditDate="2014-06-22T22:44:19.880" LastActivityDate="2017-01-18T00:53:31.453" Title="Score matrix string similarity" Tags="|algorithms|similarity|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="529" PostTypeId="2" ParentId="430" CreationDate="2014-06-23T03:44:57.643" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://nerds.airbnb.com/architecting-machine-learning-system-risk/&quot; rel=&quot;nofollow&quot;&gt;Airbnb&lt;/a&gt; and &lt;a href=&quot;http://codeascraft.com/2014/06/18/conjecture-scalable-machine-learning-in-hadoop-with-scalding/&quot; rel=&quot;nofollow&quot;&gt;Etsy&lt;/a&gt; both recently posted detailed information about their workflows.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2014-06-23T03:44:57.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="530" PostTypeId="1" AcceptedAnswerId="532" CreationDate="2014-06-23T04:39:26.623" Score="5" ViewCount="1295" Body="&lt;p&gt;There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far away are the algorithms from each other? What are the best combinations? &lt;/p&gt;&#xA;" OwnerUserId="454" LastActivityDate="2014-06-24T15:44:52.540" Title="How to select algorithms for ensemble methods?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="531" PostTypeId="1" AcceptedAnswerId="533" CreationDate="2014-06-23T07:03:15.643" Score="16" ViewCount="6318" Body="&lt;p&gt;I have a dataset with the following specifications:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training dataset with 193,176 samples with 2,821 positives&lt;/li&gt;&#xA;&lt;li&gt;Test Dataset with 82,887 samples with 673 positives&lt;/li&gt;&#xA;&lt;li&gt;There are 10 features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to perform a binary classification (0 or 1). The issue I am facing is that the data is very unbalanced. After normalization and scaling the data along with some feature engineering and using a couple of different algorithms, these are the best results I could achieve:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean square error : 0.00804710026904&#xA;Confusion matrix : [[82214   667]&#xA;                   [    0     6]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e only 6 correct positive hits. This is using logistic regression. Here are the various things I tried with this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Different algorithms like RandomForest, DecisionTree, SVM&lt;/li&gt;&#xA;&lt;li&gt;Changing parameters value to call the function&lt;/li&gt;&#xA;&lt;li&gt;Some intuition based feature engineering to include compounded features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What can I do to improve the number of positive hits ? &lt;/li&gt;&#xA;&lt;li&gt;How can one determine if there is an overfit in such a case ? ( I have tried plotting etc. )&lt;/li&gt;&#xA;&lt;li&gt;At what point could one conclude if maybe this is the best possible fit I could have? ( which seems sad considering only 6 hits out of 673 )&lt;/li&gt;&#xA;&lt;li&gt;Is there a way I could make the positive sample instances weigh more so the pattern recognition improves leading to more hits ?&lt;/li&gt;&#xA;&lt;li&gt;Which graphical plots could help detect outliers or some intuition about which pattern would fit the best?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am using the scikit-learn library with Python and all implementations are library functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the results with a few other algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest Classifier(n_estimators=100)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[82211   667]&#xA;[    3     6]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Decision Trees:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[78611   635]&#xA;[ 3603    38]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="793" LastEditorUserId="1330" LastEditDate="2019-05-07T04:22:04.987" LastActivityDate="2019-06-28T10:01:52.693" Title="Binary classification model for unbalanced data" Tags="|machine-learning|python|classification|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="532" PostTypeId="2" ParentId="530" CreationDate="2014-06-23T08:03:50.327" Score="8" Body="&lt;p&gt;In general in an ensemble you try to combine the opinions of multiple classifiers. The idea is like asking a bunch of experts on the same thing. You get multiple opinions and you later have to combine their answers (e.g. by a voting scheme). For this trick to work you want the classifiers to be different from each other, that is you don't want to ask the same &quot;expert&quot; twice for the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, the classifiers do not have to be different in the sense of a different algorithm. What you can do is train the same algorithm with different subset of the data or a different subset of features (or both). If you use different training sets you end up with different models and different &quot;independent&quot; classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no golden rule on what works best in general. You have to try to see if there is an improvement for your specific problem. &lt;/p&gt;&#xA;" OwnerUserId="418" LastEditorUserId="1085" LastEditDate="2014-06-23T11:21:51.677" LastActivityDate="2014-06-23T11:21:51.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="533" PostTypeId="2" ParentId="531" CreationDate="2014-06-23T08:13:22.837" Score="11" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Since you are doing binary classification, have you tried adjusting the classification threshold? Since your algorithm seems rather insensitive, I would try lowering it and check if there is an improvement.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can always use &lt;a href=&quot;https://stackoverflow.com/questions/4617365/what-is-a-learning-curve-in-machine-learning&quot;&gt;Learning Curves&lt;/a&gt;, or a plot of one model parameter vs. Training and Validation error to determine whether your model is overfitting. It seems it is under fitting in your case, but that's just intuition.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Well, ultimately it depends on your dataset, and the different models you have tried. At this point, and without further testing, there can not be a definite answer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Without claiming to be an expert on the topic, there are a number of different techniques you may follow (hint: &lt;a href=&quot;http://florianhartl.com/thoughts-on-machine-learning-dealing-with-skewed-classes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;first link on google&lt;/a&gt;), but in my opinion you should first make sure you choose your cost function carefully, so that it represents what you are actually looking for. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Not sure what you mean by pattern intuition, can you elaborate?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;By the way, what were your results with the different algorithms you tried? Were they any different?&lt;/p&gt;&#xA;" OwnerUserId="1085" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-06-23T08:13:22.837" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="534" PostTypeId="2" ParentId="527" CreationDate="2014-06-23T09:23:27.617" Score="3" Body="&lt;p&gt;If your goal is to transform your matrix into a number (your similarity measure), you may want to use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Matrix_norm&quot; rel=&quot;nofollow&quot;&gt;matrix norm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm&quot; rel=&quot;nofollow&quot;&gt;Frobenius norm&lt;/a&gt; on your example would return 1.488086.&lt;/p&gt;&#xA;" OwnerUserId="883" LastActivityDate="2014-06-23T09:23:27.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="535" PostTypeId="2" ParentId="527" CreationDate="2014-06-23T09:42:46.450" Score="3" Body="&lt;p&gt;As I understood, Document 1 and Document 2 may have different number of keys. And you wand to get final similarity evaluation between 0 and 1. If so, I would propose following algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sum of max. vals is equal to 0.&lt;/li&gt;&#xA;&lt;li&gt;Select maximum value from doc-doc matrix and add it to Sum of max. vals.&lt;/li&gt;&#xA;&lt;li&gt;Remove row and column with maximum value from the matrix.&lt;/li&gt;&#xA;&lt;li&gt;Repeat steps 2-3 until rows or columns are ended.&lt;/li&gt;&#xA;&lt;li&gt;Denominate Sum of max. vals by average number of key words in two texts.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Final estimation would be equal to 1, if both documents have identical length, and every word from Doc 1 has equivalent in Doc 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You haven't mentioned software, you are using, but here is &lt;strong&gt;R&lt;/strong&gt; example of function, computing such similarity (it takes object of class matrix as input):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eval.sim &amp;lt;- function(sim.matrix){&#xA;  similarity &amp;lt;- 0&#xA;  denominator &amp;lt;- sum(dim(sim.matrix)) / 2&#xA;  for(i in 1:(min(c(nrow(sim.matrix), ncol(sim.matrix))) - 1)){&#xA;    extract &amp;lt;- which(sim.matrix == max(sim.matrix), arr.ind=T)[1, ]&#xA;    similarity &amp;lt;- similarity + sim.matrix[extract[1], extract[2]]&#xA;    sim.matrix &amp;lt;- sim.matrix[-extract[1], -extract[2]]&#xA;  }&#xA;  similarity &amp;lt;- similarity + max(sm.copy)&#xA;  similarity &amp;lt;- similarity / denominator&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In python - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;def score_matrix(sim_matrix):&#xA;    similarity = 0&#xA;    denominator = sum(sim_matrix.shape) / 2&#xA;    for i in range(min(sim_matrix.shape)):&#xA;        x, y = np.where(sim_matrix == np.max(sim_matrix))[0][0], np.where(sim_matrix == np.max(sim_matrix))[1][0]&#xA;        similarity += sim_matrix[x, y]&#xA;        sim_matrix = np.delete(sim_matrix,(x),axis=0)&#xA;        sim_matrix = np.delete(sim_matrix,(y),axis=1)&#xA;    return similarity / denominator&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="941" LastEditorUserId="941" LastEditDate="2014-06-30T05:35:26.340" LastActivityDate="2014-06-30T05:35:26.340" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="536" PostTypeId="1" CreationDate="2014-06-23T11:03:09.767" Score="6" ViewCount="493" Body="&lt;p&gt;More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a fraction of data needs it, I write a script and incorporate it in data processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what to do if only a few entries needs to be fixed (e.g. misspelled city or zip code)? Let's focus on &quot;small data&quot;,  such as that in CSV files or a relational database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The practical problems I encountered:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Writing a general script trying solve all similar errors may give unintended consequences (e.g. matching cities that are different but happen to have similar names).&lt;/li&gt;&#xA;&lt;li&gt;Copying and modifying data may make a mess, as:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Generating it again will destroy all fixes.&lt;/li&gt;&#xA;&lt;li&gt;When there are more errors of different kinds, too many copies of the same file result, and it is hard to keep track of them all.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Writing a script to modify particular entries seems the best, but there is overhead in comparison to opening CSV and fixing it (but still, &lt;em&gt;seems&lt;/em&gt; to be the best); and either we need to create more copies of data (as in the previous point) or run the script every time we load data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What are the best practices in such a case as this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: The question is on the workflow, not whether to use it or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In my particular case I don't want the end-user to see misspelled cities and, even worse, see two points of data, for the same city but with different spelling; the data is small, ~500 different cities, so manual corrections do make sense.)&lt;/p&gt;&#xA;" OwnerUserId="289" LastEditorUserId="24" LastEditDate="2014-07-02T08:48:41.517" LastActivityDate="2014-10-15T11:37:47.090" Title="Good practices for manual modifications of data" Tags="|data-cleaning|" AnswerCount="7" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="538" PostTypeId="1" CreationDate="2014-06-23T13:31:41.443" Score="5" ViewCount="367" Body="&lt;p&gt;Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools.&lt;/p&gt;&#xA;&lt;p&gt;What sources do you use to get all the information?&lt;/p&gt;&#xA;&lt;p&gt;I use mostly Reddit as my first source and the subreddits &lt;a href=&quot;http://www.reddit.com/r/MachineLearning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning&lt;/a&gt; and &lt;a href=&quot;http://www.reddit.com/r/rstats&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But I would also &lt;a href=&quot;http://www.datatau.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Datatau&lt;/a&gt; and of course the great &lt;a href=&quot;http://www.kdnuggets.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;KDNuggets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1125" LastEditorUserId="85045" LastEditDate="2021-03-30T11:24:36.087" LastActivityDate="2021-04-01T08:22:59.627" Title="What are your favorite sources for news about Machine Learning and Data Science?" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ClosedDate="2014-06-25T20:26:56.000" ContentLicense="CC BY-SA 4.0" />
  <row Id="539" PostTypeId="2" ParentId="538" CreationDate="2014-06-23T13:36:18.273" Score="5" Body="&lt;p&gt;I like following certain tags on &lt;a href=&quot;http://stats.stackexchange.com&quot;&gt;Cross Validated&lt;/a&gt;, as well as the &lt;a href=&quot;https://stackoverflow.com/questions/tagged/r&quot;&gt;&lt;code&gt;r&lt;/code&gt; tag&lt;/a&gt; StackOverflow.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.arxiv.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;arXiv.org&lt;/a&gt; is also pretty cool, it has several subsections devoted to stats and ML.&lt;/p&gt;&#xA;" OwnerUserId="1127" LastEditorUserId="85045" LastEditDate="2021-04-01T08:22:59.627" LastActivityDate="2021-04-01T08:22:59.627" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="540" PostTypeId="1" CreationDate="2014-06-23T15:38:10.920" Score="0" ViewCount="1032" Body="&lt;p&gt;What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely helpful for both beginners and experienced. &lt;/p&gt;&#xA;" OwnerUserId="1131" LastActivityDate="2021-03-30T11:52:05.423" Title="First steps on a new cleaned dataset" Tags="|knowledge-base|dataset|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="541" PostTypeId="2" ParentId="536" CreationDate="2014-06-23T15:48:10.033" Score="2" Body="&lt;p&gt;The #1 most important thing is to &lt;em&gt;explicitly document your process&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different people working on different data make different choices. Any scientists who claim that their work is entirely objective and rational are in denial about the nature of science; every single model we create is informed by our biases and perspectives. The key is that we each have different biases and perspectives, which is one reason that science is the pursuit of a society, not of individuals, and underscores the importance of peer review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have already identified some trade-offs between an algorithmic solution and individual/case-by-case corrections. If we have training as scientists, we may be used to thinking that everything has to be applied as an overall rule, so we may bend over backwards to find correlations and fit curves to data that just won't reasonably accept it. Applying an algorithm to the entire data set can be powerful when it works, but it's not the only tool at your disposal. Sometimes you need to use your judgment as the scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if you use your individual judgment to make exceptions and to manually correct your data, be prepared to defend your judgment and argue your case. Be prepared to show that you considered all the data. If you're going to manually correct 8 observations out of a set of 100,000, you need to do more than justify those 8 manual corrections - you also need to justify &lt;strong&gt;not&lt;/strong&gt; correcting the other 99,992 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can still take a methodical approach, perhaps by classifying your data in a systematic way, and choosing which subset of the data to apply your cleaning algorithm to based on that system of classification. And when you do this, you document it, you make your case, and you respect the judgment of your colleagues in the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, why do all this extra work before you know it's necessary? Plenty of &quot;dirty&quot; data sets will still produce useful results. Perhaps 0.5% of your data is &quot;dirty&quot; in a way that you know will have a bad influence on your end product. Well, any analysis is subject to error, and the more you obsess over that 0.5% the more you will start to think of it and treat it like it was 5%, or 25% - much more significant than it truly is. Try to apply your analysis while admitting you know there is some error, and only do extra work to clean the data once you can show for certain that your analysis fails or is not useful otherwise.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;For example...&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Say you have a set of test results from a survey of wells, showing the concentrations of certain dissolved substances, hourly over the course of a year. And you observe in this set certain spikes, of short duration, and orders of magnitude higher than the surrounding data. If they are few, and obvious, and you know that the sensors used to produce the data set occasionally malfunction, then there's no reason to apply an algorithmic solution to the entire data set. You have a choice between excluding some data or modifying it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend the exclusion route whenever possible, since you are making fewer assumptions when you don't have to additionally choose a &quot;correct&quot; value. But if your analysis will absolutely fail with a discontinuity, there are many options. You could choose to replace &quot;bad&quot; values via linear interpolation. You could hold constant the previous value. If there is some great previous literature to apply, and you really have a strong case that the &quot;bad&quot; data is purely due to equipment malfunction, perhaps there's an established model that you can apply to fill in those regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a great many approaches, which is why the most important thing is to document your process, explicitly and exhaustively. Arguing your case is important too, but not strictly necessary; the community is just more likely to ignore you if you don't make a full effort to engage the review process.&lt;/p&gt;&#xA;" OwnerUserId="322" LastEditorUserId="322" LastEditDate="2014-06-23T16:14:33.900" LastActivityDate="2014-06-23T16:14:33.900" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="543" PostTypeId="2" ParentId="540" CreationDate="2014-06-23T17:50:51.320" Score="1" Body="&lt;p&gt;Well, you mention in your question of the data being 'clean and well structured'. In practice, close to 70% of the time is spent in doing these two steps. Of course the first thing you do is to separate the training and test data. Considering the plethora of libraries and tools available irrespective of which technology/language you prefer to use, the next step would be to understand the data via graph plots and drawing useful intuitions specific to your target goal. This would then be followed by various other problem specific methods. As pointed out, the question is very broad and citing code snippets is simply not feasible.&lt;/p&gt;&#xA;" OwnerUserId="793" LastActivityDate="2014-06-23T17:50:51.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="544" PostTypeId="2" ParentId="538" CreationDate="2014-06-23T17:56:32.917" Score="5" Body="&lt;p&gt;You could also try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quora.com&quot; rel=&quot;noreferrer&quot;&gt;Quora&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://paper.li/MachineCoding/1370791453&quot; rel=&quot;noreferrer&quot;&gt;Machine Learning Daily&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="793" LastActivityDate="2014-06-23T17:56:32.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="545" PostTypeId="2" ParentId="540" CreationDate="2014-06-23T18:38:34.260" Score="3" Body="&lt;p&gt;I think this is a reasonable question. Here is what I do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Peak at the first few rows&lt;/li&gt;&#xA;&lt;li&gt;Visualize the distribution of the features I care about (histograms)&lt;/li&gt;&#xA;&lt;li&gt;Visualize the relationship between pairs of features (scatterplots)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I downloaded the abalone dataset from the UCI Machine Learning repository &lt;a href=&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Let's say I care about how height and diameter can be used to predict whole weight. For completeness, I've included the step of reading the data from file.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns&#xA;&#xA;data = pd.read_csv(&quot;abalone.data&quot;, header=False)&#xA;data.columns = [&quot;sex&quot;, &quot;length&quot;, &quot;diameter&quot;, &quot;height&quot;, &#xA;                &quot;whole_weight&quot;, &quot;shucked_weight&quot;,&#xA;                &quot;viscera_weight&quot;, &quot;shell_weight&quot;, &quot;rings&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we can take a peak at the first few rows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZEFH6.png&quot; alt=&quot;Head of dataset&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I know that the variables I care about are floating point values and they can be treated as continuous. I want to take a look to see how these three variables are distributed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fig = plt.figure(figsize=(20,5))&#xA;plt.subplot(1, 3, 1)&#xA;plt.hist(data['diameter'], normed=True)&#xA;plt.title(&quot;Diameter&quot;)&#xA;plt.subplot(1, 3, 2)&#xA;plt.hist(data['height'], normed=True)&#xA;plt.title(&quot;Height&quot;)&#xA;plt.subplot(1, 3, 3)&#xA;plt.hist(data['whole_weight'], normed=True)&#xA;plt.title(&quot;Whole Weight&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rmq4n.png&quot; alt=&quot;histograms&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Great! Now, I know that diameter and whole weight are skewed left and right (respectively). I also know that there are some outliers in terms of height (which is why matplotlib gives me extra room to the right of the distribution). Finally, I'd like to see if I can find any visual patterns between my predictors and outcome variable. I use a scatter plot for this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(15,5))&#xA;plt.subplot(1, 2, 1)&#xA;plt.plot(data['diameter'], data['whole_weight'], 'o')&#xA;plt.title(&quot;Diameter vs. Whole Weight&quot;)&#xA;plt.ylabel(&quot;Whole Weight&quot;)&#xA;plt.xlabel(&quot;Diameter&quot;)&#xA;plt.subplot(1, 2, 2)&#xA;plt.plot(data['height'], data['whole_weight'], 'o')&#xA;plt.title(&quot;Height vs. Whole Weight&quot;)&#xA;plt.ylabel(&quot;Whole Weight&quot;)&#xA;plt.xlabel(&quot;Height&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/FMmN1.png&quot; alt=&quot;scatterplots&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, I see there is a non-linear relationship between diameter and whole weight and I'm going to have to deal with my height outliers. Now, I'm ready to do some analysis!&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-23T18:38:34.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="546" PostTypeId="2" ParentId="536" CreationDate="2014-06-23T18:56:11.577" Score="12" Body="&lt;p&gt;Even if you are effectively modifying certain records by hand, as in the city name example you give, I would recommend doing it in code. The reason to strongly prefer code over hand-tweaking records is that the code makes a result reproducible. You want to make sure that you can always go from raw data to final result without any human intervention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a quick example. Let's say I have a list of city names in a pandas dataframe and I am certain they should all be &quot;omaha&quot; (you need to be absolutely certain, because changing values by hand is fraught with danger). But instead I have the following strings:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.unique(data.city)&#xA;array(['omaha', 'omahd', 'imaha', 'omaka'], dtype=object)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could make the change like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.city.values[data.city.values == 'omahd'] = 'omaha'&#xA;data.city.values[data.city.values == 'imaha'] = 'omaha'&#xA;data.city.values[data.city.values == 'omaka'] = 'omaha'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That code is ugly, but if you run it on the same raw dataset, you will &lt;em&gt;always&lt;/em&gt; get the same result.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-23T18:56:11.577" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="548" PostTypeId="1" CreationDate="2014-06-23T23:20:36.180" Score="11" ViewCount="4038" Body="&lt;p&gt;I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the best way possible for inclusion as predictors in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a concrete example, say a model is being built to predict if a customer will leave the company in the next 90 days (anytime between t and t+90; thus a binary outcome). One of the predictors available is the level of the customers financial balance for periods t_0 to t-1. Maybe this represents monthly observations for the prior 12 months (i.e. 12 measurements). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for ways to construct features from this series. I use descriptives of each customers series such as the mean, high, low, std dev., fit a OLS regression to get the trend. Are their other methods of calculating features? Other measures of change or volatility? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ADD:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned in a response below, I also considered (but forgot to add here) using Dynamic Time Warping (DTW) and then hierarchical clustering on the resulting distance matrix - creating some number of clusters and then using the cluster membership as a feature. Scoring test data would likely have to follow a process where the DTW was done on new cases and the cluster centroids - matching the new data series to their closest centroids... &lt;/p&gt;&#xA;" OwnerUserId="1138" LastEditorUserId="1138" LastEditDate="2014-06-24T14:03:22.697" LastActivityDate="2016-03-26T14:34:31.200" Title="Feature Extraction Technique - Summarizing a Sequence of Data" Tags="|machine-learning|feature-selection|time-series|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="549" PostTypeId="2" ParentId="548" CreationDate="2014-06-24T00:54:13.110" Score="2" Body="&lt;p&gt;What you're trying to do here is reduce the dimensionality of your features. You can search for dimensionality reduction to get several options, but one very popular technique is principal components analysis (PCA). Principal components are not interpretable like the options you've mentioned, but they do a good job of summarizing all of the information.&lt;/p&gt;&#xA;" OwnerUserId="1011" LastActivityDate="2014-06-24T00:54:13.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="550" PostTypeId="2" ParentId="548" CreationDate="2014-06-24T03:50:41.410" Score="2" Body="&lt;p&gt;Feature extraction is always a challenge and the less addressed topic in literature, since it's widely application dependant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some ideas you can try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Raw data, measured day-by-day. That's kind of obvious with some implications and extra preprocessing (normalisation) in order to make timelines of different length comparable.&lt;/li&gt;&#xA;&lt;li&gt;Higher moments: skewness, kurtosis, etc&lt;/li&gt;&#xA;&lt;li&gt;Derivative(s): speed of evolution&lt;/li&gt;&#xA;&lt;li&gt;Time span is not that large but maybe it is worth trying some time series analysis features like for example autocorrelation. &lt;/li&gt;&#xA;&lt;li&gt;Some customised features like breaking timeline in weeks and measure the quantities you already measure in each week separately. Then a non-linear classifier would be able to combine e.g first-week features with last-week features in order to get insight of evolution in time.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="418" LastActivityDate="2014-06-24T03:50:41.410" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="552" PostTypeId="2" ParentId="548" CreationDate="2014-06-24T07:03:35.567" Score="2" Body="&lt;p&gt;At first glance, you need to extract features from your time series (x - 12) - x. One possible approach is to compute summary metrics: average, dispersion, etc. But doing so, you will loose all time-series related information. But data, extracted from curve shape may be quite useful. I recommend you to look through &lt;a href=&quot;http://cs.stanford.edu/people/jure/pubs/memeshapes-wsdm11.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; article, where authors propose algorithm for time series clustering. Hope, it will be useful. Additionally to such clustering you can add summary statistics to your feature list. &lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-06-24T07:03:35.567" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="554" PostTypeId="1" CreationDate="2014-06-24T07:29:57.787" Score="13" ViewCount="824" Body="&lt;p&gt;I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the best practices to understand a dataset (high dimensional with numerical and nominal attributes)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Practices to make sure the dataset is clean?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Practices to make sure the dataset doesn't have wrong values or so?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="728" LastActivityDate="2021-03-13T23:23:44.427" Title="Datasets understanding best practices" Tags="|statistics|dataset|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="555" PostTypeId="2" ParentId="536" CreationDate="2014-06-24T07:35:16.930" Score="15" Body="&lt;p&gt;&lt;strong&gt;1.  Do not modify the original data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having the original data source intact is important.  You may find that updates you make to the data are not valid.  You may also find a more efficient way to make updates and you will want to regression test those updates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Always work with a copy of the data, and add columns/properties/metadata that includes any processed corrections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, if your data is a .csv file that includes a city name that contains several misspellings:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Copy the .csv to a new file&#xA;2. Add a column for containing &quot;adjusted_city_name&quot;&#xA;3. Copy data from the city_name column to the &#xA;   adjusted_city_name column and make corrections in that column.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Document proposed changes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any changes you want to make to data should be documented so that they can be replicated moving forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Version control and timestamp the document every time you change it.  That will help in troubleshooting at a later date.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be explicit.  Do not simply state &quot;correct capitalization problems&quot;, state &quot;ensure that the first letter of each city name begins with a capital letter and the remaining letters are lower-case.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update the document with references to any automation routines that have been built to manage data cleansing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Decide on a standard data cleansing technology&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you use perl, python, java, a particular utility, a manual process or something else is not the issue.  The issue is that in the future you want to hand the data cleansing process to somebody else.  If they have to know 12 different data cleansing technologies, delegating the cleansing procedure will be very difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. Standardize the workflow&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There should be a standard way to handle new data.  Ideally, it will be as simple as dropping a file in a specific location and a predictable automated process cleanses it and hands off a cleansed set of data to the next processing step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5. Make as few changes as is absolutely necessary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's always better to have a fault tolerant analysis than one that makes assumptions about the data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;6. Avoid manual updates&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's always tempting, but people are error-prone and again it makes delegation difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes on manual processing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To more completely address the original question as to whether there's a &quot;good&quot; way to do manual processing, I would say no, there is not.  My answer is based on experience and is not one that I make arbitrarily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had more than one project lose days of time due to a client insisting that a manual data cleansing process was just fine and could be handled internally.  You do not want your projects to be dependent on a single individual accomplishing a judgement based task of varying scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's much better to have that individual build and document a rule set based on what they would do than to have them manually cleanse data.  (And then automating that rule set)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If automation fails you in the end or is simply not possible, the ability to delegate that rule set to others without domain specific knowledge is vital.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, routines to do something like correct city names can be applied to other data sets.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-24T07:35:16.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="556" PostTypeId="2" ParentId="554" CreationDate="2014-06-24T08:09:49.300" Score="8" Body="&lt;p&gt;There are basic things you can do with any set of data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Validate values (String length tolerance, data type, formatting masks, required field presence, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Range correctness (Does this seemingly correct data fall within expected ranges of values)&lt;/li&gt;&#xA;&lt;li&gt;Preliminary processing (If I attempt to analyze this data, can I perform the basics without running into errors)&lt;/li&gt;&#xA;&lt;li&gt;Preliminary reporting (run a report against a data set and ensure that it passes a sanity test)&lt;/li&gt;&#xA;&lt;li&gt;Defining null vs. empty vs. zero vs. False for any given column of data&lt;/li&gt;&#xA;&lt;li&gt;Identifying data that is out of place (numeric values dramatically different than other values in a data set, string values that look like they might be misspelled, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Eliminating or correcting obviously errant data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Understanding data to identify errors is a whole different ball game, and it is very important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, you can have a rule that says a serial number must be present in a given data set and that serial number must be alphanumeric with a maximum string length of 255 and a minimum string length of 5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the data, you may find one particular serial number value reads &lt;code&gt;&quot;PLEASE ENTER SERIAL&quot;&lt;/code&gt;  It's perfectly valid, but wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's kind of an obvious one, but say you're processing stock data and you had a price range for 1000 stocks that was under a dollar.  A lot of people would not know that a stock price so low is invalid on certain exchanges and perfectly valid on others.  You need knowledge about your data to understand if what you are seeing is problematic or not.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the real world, you don't always have the luxury of understanding your data intimately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I avoid problems is by leveraging the people around me.  For small data sets, I can ask someone to review the data in it's entirety.  For large ones, pulling a set of random samples and asking someone to do a sanity check on the data is more appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, questioning the source of the data and how well that data source can be trusted is imperative.  I often have multiple conflicting sources of data and we create rules to determine the &quot;source of truth&quot;.  Sometimes one data set has great data in a given aspect, but other data sets are stronger in other areas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Manually entered data is usually what I'm most skeptical about, but in some cases it is stronger than anything that can be acquired through automation.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-24T08:09:49.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="559" PostTypeId="1" AcceptedAnswerId="561" CreationDate="2014-06-24T12:28:10.990" Score="17" ViewCount="3160" Body="&lt;p&gt;I have a hobby project which I am contemplating committing to as a way of increasing my so far limited experience of machine learning. I have taken and completed the Coursera MOOC on the topic. My question is with regards to the feasibility of the project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neighboring cats are from time to time visiting my garden, which I dislike since they tend to defecate on my lawn. I would like to have a warning system that alerts me when there's a cat present so that I may go chase it off using my super soaker. For simplicity's sake, say that I only care about a cat with black and white coloring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have setup a raspberry pi with camera module that can capture video and/or pictures of a part of the garden. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/cNqus.jpg&quot; alt=&quot;Sample garden image&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first idea was to train a classifier to identify cat or cat-like objects, but after realizing that I will be unable to obtain a large enough number of positive samples, I have abandoned that in favor of anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I estimate that if I captured a photo every second of the day, I would end up with maybe five photos containing cats (out of about 60,000 with sunlight) per day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this feasible using anomaly detection? If so, what features would you suggest? My ideas so far would be to simply count the number of pixels with that has certain colors; do some kind of blob detection/image segmenting (which I do not know how do to, and would thus like to avoid) and perform the same color analysis on them.&lt;/p&gt;&#xA;" OwnerUserId="1147" LastActivityDate="2014-06-25T03:30:19.787" Title="Detecting cats visually by means of anomaly detection" Tags="|machine-learning|" AnswerCount="5" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="560" PostTypeId="2" ParentId="559" CreationDate="2014-06-24T12:55:26.457" Score="4" Body="&lt;p&gt;This is an interesting and also quite ambitious project :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure anomaly detection (at least in the sense described in the course you followed) would be a very fitting algorithm in this case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would consider a more viable approach to be what has been discussed at the end of that course where a Photo OCR workflow was demonstrated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach would consist of segmenting your image in smaller &quot;blocks&quot;, and going through them one-by-one using a supervised learning algorithm and try to classify each block according to whether it contains a cat or not. If one block contains a cat, the alarm goes off. As a bonus, you get the position of the cat as well, so that you may think of incorporating some &quot;automatic&quot; response as a future step to your project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benefit here is that you will not have to train your algorithm using a dataset specific to your garden (which, as you mention is difficult to create), but you can use images of cats taken off the net (e.g. perhaps you can search for &quot;cat on grass&quot; or something), and perhaps patches of photos from your (or other) gardens. Therefore you don;t have to spend your time collecting photos from your camera, and you avoid the risk of having a very small (comparable) sample of positives (i.e. cats).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, of course how easy it is to build an accurate cat detector is another topic..&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2014-06-24T12:55:26.457" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="561" PostTypeId="2" ParentId="559" CreationDate="2014-06-24T13:34:12.107" Score="8" Body="&lt;p&gt;You could simplify your problem significantly by using a motion/change detection approach. For example, you could compare each image/frame with one from an early time (e.g., a minute earlier), then only consider pixels that have changed since the earlier time. You could then extract the rectangular region of change and use that as the basis for your classification or anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking this type of approach can significantly simplify your classifier and reduce your false target rate because you can ignore anything that is not roughly the size of a cat (e.g., a person or bird). You would then use the extracted change regions that were not filtered out to form the training set for your classifier (or anomaly detector).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just be sure to get your false target rate sufficiently low before mounting a laser turret to your feline intrusion detection system.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2014-06-24T13:34:12.107" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="562" PostTypeId="2" ParentId="530" CreationDate="2014-06-24T15:44:52.540" Score="3" Body="&lt;p&gt;As a rule of thumb I always propose three different options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use a bagging learning technique, similar to that one followed by Random Forest. This technique allows the training of 'small' classifiers which see a small portion of the whole data. Afterwards, a simple voting scheme (as in Random Forest) will lead you to a very interesting and robust classification.&lt;/li&gt;&#xA;&lt;li&gt;Use any technique related to fusioning information or probabilistic fusion. This is a very suitable solution in order to combine different likelihoods from different classifiers.&lt;/li&gt;&#xA;&lt;li&gt;My last suggestion is the use of fuzzy logic, a very adequate tool in order to combine information properly from a probabilistic (belonging) perspective.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The selection of specific methods or strategies will depend enormously on the data.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-24T15:44:52.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="563" PostTypeId="2" ParentId="559" CreationDate="2014-06-24T15:55:37.110" Score="3" Body="&lt;p&gt;The strategy of motion/change detection is certainly adequate, but I would add an extra operation. I would detect those regions that are more likely to be changed, for instance, the ladder seems a place where humans can be (also cats) and grass where dogs, cats or humans can be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would capture a map with size of the object and trajectory and with this I would create a cluster with the aim of detecting an object (with specific size within the image in terms of pixels) that moves with a certain speed and trajectory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can achieve this by using R or I would suggest OpenCV in order to detect movement and follow different objects.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-24T15:55:37.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="564" PostTypeId="1" AcceptedAnswerId="567" CreationDate="2014-06-24T18:59:30.560" Score="5" ViewCount="333" Body="&lt;p&gt;I've written a simple recommender which generates recommendations for users based on what they have clicked. The recommender generates a data file with the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;userid,userid,simmilarity (between 0 and 1 - closer to 0 the more similar the users)&#xA;a,b,.2&#xA;a,c,.3&#xA;a,d,.4&#xA;a,e,.1&#xA;e,b,.3&#xA;e,c,.5&#xA;e,d,.8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've looked at &lt;a href=&quot;https://github.com/mbostock/d3/wiki/Gallery&quot; rel=&quot;nofollow noreferrer&quot;&gt;some graphs&lt;/a&gt;, but I'm not sure which one to use, or if are there other ones that will better display the user similarities from the dataset above. Any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aiming this visualization at business users who are not at all technical. I would just like to show them an easy to understand visual that details how similar some users are and so convince the business that for these users the recommendation system is useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Steve Kallestad do you mean something like this : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4zyQR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="237" LastEditorUserId="237" LastEditDate="2014-06-25T07:56:29.733" LastActivityDate="2014-06-25T07:56:29.733" Title="What visualization technique to best describe a recommendation dataset?" Tags="|visualization|javascript|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="565" PostTypeId="1" CreationDate="2014-06-24T19:43:24.643" Score="11" ViewCount="20244" Body="&lt;p&gt;As I increase the number of trees in &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;noreferrer&quot;&gt;scikit learn&lt;/a&gt;'s &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;GradientBoostingRegressor&lt;/code&gt;&lt;/a&gt;, I get more negative predictions, even though there are no negative values in my training or testing set. I have about 10 features, most of which are binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the parameters that I was tuning were:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the number of trees/iterations;&lt;/li&gt;&#xA;&lt;li&gt;learning depth;&lt;/li&gt;&#xA;&lt;li&gt;and learning rate.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The percentage of negative values seemed to max at ~2%. The learning depth of 1(stumps) seemed to have the largest % of negative values. This percentage also seemed to increase with more trees and a smaller learning rate. The dataset is from one of the kaggle playground competitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import GradientBoostingRegressor&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y)&#xA;&#xA;reg = GradientBoostingRegressor(n_estimators=8000, max_depth=1, loss = 'ls', learning_rate = .01)&#xA;&#xA;reg.fit(X_train, y_train)&#xA;&#xA;ypred = reg.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1162" LastEditorUserId="322" LastEditDate="2014-06-25T16:46:30.323" LastActivityDate="2020-10-04T14:08:47.880" Title="Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?" Tags="|machine-learning|python|algorithms|scikit-learn|kaggle|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="566" PostTypeId="1" AcceptedAnswerId="2456" CreationDate="2014-06-24T17:06:10.310" Score="6" ViewCount="1991" Body="&lt;p&gt;Many times &lt;a href=&quot;http://en.wikipedia.org/wiki/Named-entity_recognition&quot;&gt;Named Entity Recognition&lt;/a&gt; (NER) doesn't tag consecutive NNPs as one NE. I think editing the NER to use RegexpTagger also can improve the NER.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider the following input:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Barack Obama is a great person.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Tree('S', [Tree('PERSON', [('Barack', 'NNP')]), Tree('ORGANIZATION', [('Obama', 'NNP')]),&#xA;    ('is', 'VBZ'), ('a', 'DT'), ('great', 'JJ'), ('person', 'NN'), ('.', '.')])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where as for the input: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'Former Vice President Dick Cheney told conservative radio host Laura Ingraham that he &quot;was honored&quot; to be compared to Darth Vader while in office.'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;the output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Tree('S', [('Former', 'JJ'), ('Vice', 'NNP'), ('President', 'NNP'),&#xA;    Tree('NE', [('Dick', 'NNP'), ('Cheney', 'NNP')]), ('told', 'VBD'), ('conservative', 'JJ'),&#xA;    ('radio', 'NN'), ('host', 'NN'), Tree('NE', [('Laura', 'NNP'), ('Ingraham', 'NNP')]),&#xA;    ('that', 'IN'), ('he', 'PRP'), ('``', '``'), ('was', 'VBD'), ('honored', 'VBN'),&#xA;    (&quot;''&quot;, &quot;''&quot;), ('to', 'TO'), ('be', 'VB'), ('compared', 'VBN'), ('to', 'TO'),&#xA;    Tree('NE', [('Darth', 'NNP'), ('Vader', 'NNP')]), ('while', 'IN'), ('in', 'IN'),&#xA;    ('office', 'NN'), ('.', '.')])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here &lt;code&gt;Vice/NNP, President/NNP, (Dick/NNP, Cheney/NNP)&lt;/code&gt; is correctly extracted. So, I think if &lt;code&gt;nltk.ne_chunk&lt;/code&gt; is used first, and then if two consecutive trees are NNP, there are higher chances that both refer to one entity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been playing with NLTK toolkit, and I came across this problem a lot, but couldn't find a satisfying answer. Any suggestion will be really appreciated. I'm looking for flaws in my approach.&lt;/p&gt;&#xA;" OwnerUserId="1165" OwnerDisplayName="mousecoder" LastEditorUserId="29169" LastEditDate="2021-03-30T22:42:29.823" LastActivityDate="2021-03-30T22:42:29.823" Title="Named Entity Recognition: NLTK using Regular Expression" Tags="|nlp|named-entity-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="567" PostTypeId="2" ParentId="564" CreationDate="2014-06-24T20:28:33.693" Score="5" Body="&lt;p&gt;I think you're looking for a &lt;a href=&quot;http://cgi.mtc.sri.com/Cluster-Lab/&quot;&gt;similarity matrix&lt;/a&gt; (see bottom of the page). If you don't have data on similarity between certain pairs, you can always leave them as grey or white. Also, this will only work for data sets small enough to actually make out what's going on. I'd say 25 rows / columns maximum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a similarity matrix, x, and y coordinates correspond to the two things you're comparing, while a &lt;a href=&quot;http://en.wikipedia.org/wiki/False_color&quot;&gt;colormap&lt;/a&gt; magnitude represents similarity &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:&#xA;One thing you could do to replace the colormap is the insert, say, circles of different sizes according to the similarity metric. Or you could insert the numbers themselves, again, varying the size of the number as the magnitude of that number varies. Size usually works best is business visualizations.&lt;/p&gt;&#xA;" OwnerUserId="1163" LastEditorUserId="1163" LastEditDate="2014-06-24T20:37:02.377" LastActivityDate="2014-06-24T20:37:02.377" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="568" PostTypeId="2" ParentId="447" CreationDate="2014-06-24T20:34:18.473" Score="3" Body="&lt;p&gt;I suggest the use of Hidden Markov Models, with two possible states: (1) high levels and (0) low levels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This technique might be helpful to decode your signal. Probably you would need a specific HMM for each codification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If noise is an issue an FIR filter with a Blackman-Harris window function would allow you to isolate the frequency you're concerned with. &lt;/p&gt;&#xA;" OwnerUserId="1155" LastEditorUserId="890" LastEditDate="2014-06-25T06:58:43.670" LastActivityDate="2014-06-25T06:58:43.670" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="569" PostTypeId="2" ParentId="559" CreationDate="2014-06-24T20:35:44.713" Score="1" Body="&lt;p&gt;Is it a bird? Is it a cat? We have black-and-white cat-sized! magpies here. so that would fail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First thing would be to exclude all areas that are green, cats are seldom green.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then compare the rest to a reference image to remove static things like stones and stairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Detecting objects of a minimum size should be possible, but for a classification the resolution is too low. Could be also your neighbor testing his new remote controlled drone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With two cameras you could do a 3d mapping of the objects and eliminate flying objects.&lt;/p&gt;&#xA;" OwnerUserId="1164" LastEditorUserId="1164" LastEditDate="2014-06-24T20:48:37.837" LastActivityDate="2014-06-24T20:48:37.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="570" PostTypeId="2" ParentId="564" CreationDate="2014-06-25T00:13:50.803" Score="2" Body="&lt;p&gt;Personally, I think Netflix got it right.  Break it down into a confidence rating from 1-5 and show your recommendations based on the number of yellow stars.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't have to be stars, but those icon based graphs are very easy to interpret and get the point across clearly.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-25T00:13:50.803" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="571" PostTypeId="1" AcceptedAnswerId="572" CreationDate="2014-06-25T02:33:22.673" Score="5" ViewCount="762" Body="&lt;p&gt;I'm trying to rank some percentages. I have numerators and denominators for each ratio. To give a concrete example, consider ratio as &lt;code&gt;total graduates / total students&lt;/code&gt; in a school.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the issue is that &lt;code&gt;total students&lt;/code&gt; vary over a long range (1000-20000). Smaller schools seem to have higher percentage of students graduating, but I want to standardize it, and not let the size of the school affect the ranking. Is there a way to do it?&lt;/p&gt;&#xA;" OwnerUserId="1174" LastEditorUserId="84" LastEditDate="2014-06-25T03:59:46.827" LastActivityDate="2014-06-25T03:59:46.827" Title="Standardize numbers for ranking ratios" Tags="|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="572" PostTypeId="2" ParentId="571" CreationDate="2014-06-25T03:11:02.457" Score="5" Body="&lt;p&gt;This is relatively simple to do mathematically.  First, fit a regression line to the scatter plot of &quot;total graduates&quot; (y) vs. &quot;total students&quot; (x).  You will probably see a downward sloping line if your assertion is correct (smaller schools graduate a higher %).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can identify the slope and y-intercept for this line to convert it into an equation y = mx + b, and then do a little algebra to convert the equation  into normalized form: &quot;y / x = m + b / x&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, with all the ratios in your data , you should &lt;em&gt;subtract&lt;/em&gt; this RHS:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;normalized ratio = (total grads / total students) - (m + b / total students)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the result is postive, then the ratio is above normal for that size (i.e. above the regression line) and if it is negative it is below the regression line.  If you want all positive numbers, you can add a positive constant to move all results above zero.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is how to do it mathematically, but I suggest that you consider whether it is wise, from a data analysis point of view, to normalize by school size.  This depends on the purpose of your analysis and specifically how this ratio is being analyzed in relation to other data.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-25T03:11:02.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="573" PostTypeId="2" ParentId="559" CreationDate="2014-06-25T03:30:19.787" Score="2" Body="&lt;p&gt;OpenCV's &lt;a href=&quot;http://docs.opencv.org/trunk/doc/tutorials/video/background_subtraction/background_subtraction.html&quot; rel=&quot;nofollow&quot; title=&quot;background subtraction&quot;&gt;background subtraction&lt;/a&gt; will find objects moving about your harden. After that you could use a classifier or shape analysis to differentiate between cats, people, trees and etc.&lt;/p&gt;&#xA;" OwnerUserId="1176" LastActivityDate="2014-06-25T03:30:19.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="574" PostTypeId="2" ParentId="116" CreationDate="2014-06-25T07:22:33.170" Score="2" Body="&lt;p&gt;This is a very interesting problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I faced a similar one by analyzing the pictures users upload to the social network. I did the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rather than associating data to ages (15 y.o., 27 y.o., ...) what I did is to establish different groups of ages: Less than 18, from 18 to 30 and greater than 30 (this is due to the specific problem we were facing, but you can choose whatever intervals you want). This division helps a lot to solve the problem.&lt;/li&gt;&#xA;&lt;li&gt;Afterwards, I created a hierarchical clustering (divisive or aggregative). Then I choose those branches where I had users with known ages (or group ages) and then for that branch I extended the same age to that group.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This approach is &lt;strong&gt;semi-supervised learning&lt;/strong&gt; and I recommended it in case you only have some data labeled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please, notice that on a social network, people usually lie about the age (just for fun, or sometimes because they want to camuflate themselves on the social net).&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-25T07:22:33.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="575" PostTypeId="2" ParentId="376" CreationDate="2014-06-25T09:19:25.560" Score="3" Body="&lt;p&gt;This is a very good question and a common situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion there are three different factors that must be controlled:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data: There exist already different benchmarks in order to evaluate algorithms and architectures. These data must be publicly available so that everybody can contrast their approaches.&lt;/li&gt;&#xA;&lt;li&gt;Architecture: My suggestion is to test everything on the cloud, so that everyone can contrast their results and also there is no doubt the same machines and software is used.&lt;/li&gt;&#xA;&lt;li&gt;Algorithms: If you have developed a distributed algorithm, it would be great to compare your algorithm on a specific data. In this case, algorithms must not be public.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, answering your question, if you want to compare different experiments and state to what extent your distributed algorithm outperforms others, you should try to replicate as accurate as possible the same environment (data and architecture) where the experiments were carried out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not possible, my suggestion is that you test your algorithm with public data and cloud architecture so that &lt;strong&gt;you become a referent&lt;/strong&gt; as you are facilitating the comparison of future algorithms.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-25T09:19:25.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="576" PostTypeId="2" ParentId="196" CreationDate="2014-06-25T10:15:31.477" Score="1" Body="&lt;p&gt;You should try &lt;em&gt;arules&lt;/em&gt; package in R. It allows you to create not only the association rules but also to specify the length of each rule, the importance of each rule and also you can filter them, which is what you are looking for (try the rhs() command of this package).&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-25T10:15:31.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="577" PostTypeId="2" ParentId="376" CreationDate="2014-06-25T10:59:56.737" Score="2" Body="&lt;p&gt;Though it's easy to say, it's better to treat the environment that changes as variables, describe/estimate your algorithm's performance base on these variables. And hopefully others will do the same. Of interest, &lt;a href=&quot;http://infolab.stanford.edu/~ullman/pub/experiments.pdf&quot; rel=&quot;nofollow&quot;&gt;Experiments as Research Validation -- Have We Gone too Far?&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-06-25T10:59:56.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="580" PostTypeId="2" ParentId="265" CreationDate="2014-06-25T14:30:10.597" Score="5" Body="&lt;p&gt;I have read some about it and I had &lt;a href=&quot;http://fellgernon.tumblr.com/post/46117939292/predicting-who-will-win-a-nfl-match-at-half-time#.UtehM7TWtQg&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog&lt;/a&gt; in mind.&lt;/p&gt;&#xA;&lt;p&gt;This blog deals with the prediction of a NFL match after the half time is already over. The prediction is 80% accurate with simple GLM model.&lt;/p&gt;&#xA;&lt;p&gt;I do not know if that is suitable for soccer.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastEditorUserId="98307" LastEditDate="2020-08-20T18:25:42.540" LastActivityDate="2020-08-20T18:25:42.540" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="581" PostTypeId="2" ParentId="22" CreationDate="2014-06-25T14:36:29.163" Score="34" Body="&lt;p&gt;In my opinion, there are solutions to deal with categorical data in clustering. R comes with a specific distance for categorical data. This distance is called &lt;a href=&quot;http://www.rdocumentation.org/packages/StatMatch/versions/1.2.0/topics/gower.dist&quot; rel=&quot;noreferrer&quot;&gt;Gower&lt;/a&gt; and it works pretty well.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastEditorUserId="98307" LastEditDate="2020-08-07T14:12:05.687" LastActivityDate="2020-08-07T14:12:05.687" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="582" PostTypeId="2" ParentId="116" CreationDate="2014-06-25T14:43:53.680" Score="4" Body="&lt;p&gt;Some research from &lt;a href=&quot;http://www.dongnguyen.nl/publications.html&quot; rel=&quot;nofollow&quot;&gt;D. Nguyen et al.&lt;/a&gt; try to predict twitter user's age based on their tweets. Maybe you find them useful. They use logistic and linear regression.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-06-25T14:43:53.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="583" PostTypeId="2" ParentId="518" CreationDate="2014-06-25T15:42:19.287" Score="4" Body="&lt;p&gt;The original MacQueen k-means publication (the first to use the name &quot;kmeans&quot;) is an online algorithm.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MacQueen, J. B. (1967). &quot;Some Methods for classification and Analysis of Multivariate Observations&quot;. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability 1. University of California Press. pp. 281–297&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;After assigning each point, the mean is incrementally updated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, it was also meant to be a single pass over the data only, although it can be trivially repeated multiple times to reassign points until convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MacQueen usually takes fewer iterations than Lloyds to converge if your data is shuffled. On ordered data, it can have problems. On the downside, it requires more computation for each object, so each iteration takes slightly longer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you implement a parallel version of k-means, make sure to study the update formulas in MacQueens publication. They're useful.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-06-25T15:42:19.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="584" PostTypeId="2" ParentId="458" CreationDate="2014-06-25T15:46:21.267" Score="8" Body="&lt;p&gt;The original MacQueen k-means publication (the first to use the name &quot;kmeans&quot;) is an online algorithm.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MacQueen, J. B. (1967). &quot;Some Methods for classification and Analysis of Multivariate Observations&quot;. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability 1. University of California Press. pp. 281–297&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;After assigning each point, the mean is incrementally updated using a simple weighted-average formula (old mean is weighted with n, the new observation is weighted with 1, if the mean had n observations before).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, it was also meant to be a single pass over the data only, although it can be trivially repeated multiple times to reassign points until convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MacQueen usually takes fewer iterations than Lloyds to converge if your data is shuffled (because it updates the mean faster!). On ordered data, it can have problems. On the downside, it requires more computation for each object, so each iteration takes slightly longer (additional math operations, obviously).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-06-25T15:46:21.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="585" PostTypeId="2" ParentId="103" CreationDate="2014-06-25T15:53:30.723" Score="6" Body="&lt;p&gt;DBSCAN (see also: Generalized DBSCAN) does not require a distance.&#xA;All it needs is a &lt;em&gt;binary decision&lt;/em&gt;. Commonly, one would use &quot;distance &amp;lt; epsilon&quot; but nothing says you cannot use &quot;similarity &gt; epsilon&quot; instead. Triangle inequality etc. are not required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Affinity propagation, as the name says, uses similarities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hierarchical clustering, except for maybe Ward linkage, does not make any assumption. In many implementations you can just use negative distances when you have similarities, and it will work just fine. Because all that is needed is min, max, and &amp;lt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kernel k-means could work IF your similarity is a good kernel function. Think of it as computing k-means in a different vector space, where Euclidean distance corresponds to your similarity function. But then you need to know k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PAM (K-medoids) should work. Assign each object to the most similary medoid, then choose the object with the highest average similarity as new medoid... no triangle inequality needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;... and probably many many more. There are literally hundreds of clustering algorithms. &lt;strong&gt;Most should work&lt;/strong&gt; IMHO. Very few seem to actually require metric properties. K-means has probably the strongest requirements: it minimizes &lt;em&gt;variance&lt;/em&gt; (not distance, or similarity), and you must be able to compute means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-06-25T15:53:30.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="586" PostTypeId="1" AcceptedAnswerId="910" CreationDate="2014-06-25T17:37:23.380" Score="7" ViewCount="2403" Body="&lt;p&gt;I'm new to the world of text mining and have been reading up on annotators at places like the &lt;a href=&quot;http://uima.apache.org/&quot; rel=&quot;noreferrer&quot;&gt;UIMA website&lt;/a&gt;. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, gazetteer, etc. Coming from a layman background, this is all very confusing so can anyone tell me or link to resources that can explain what the main categories of annotators are and what they do?&lt;/p&gt;&#xA;" OwnerUserId="1192" LastActivityDate="2015-10-12T07:20:26.220" Title="What are the main types of NLP annotators?" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="587" PostTypeId="1" CreationDate="2014-06-25T18:06:14.293" Score="14" ViewCount="31183" Body="&lt;p&gt;There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data sets but they're restricted to US stocks. Yahoo Finance provides this data online but there's no option to download it ( or none I am aware of ). &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Where can I download this data for stocks belonging to various top stock exchanges across countries by using their ticker name ?&lt;/li&gt;&#xA;&lt;li&gt;Is there some way to download it via Yahoo Finance or Google Finance ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I need data for the last decade or so and hence need some script or API which would do this.&lt;/p&gt;&#xA;" OwnerUserId="793" LastEditorUserId="97" LastEditDate="2015-07-22T12:34:31.783" LastActivityDate="2021-04-04T12:43:17.083" Title="Where can I download historical market capitalization and daily turnover data for stocks?" Tags="|dataset|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="588" PostTypeId="2" ParentId="587" CreationDate="2014-06-25T18:12:04.010" Score="7" Body="&lt;p&gt;Quant SE is better place for questions related to getting financial data:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://quant.stackexchange.com/questions/141/what-data-sources-are-available-online&quot;&gt;What data sources are available online&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://quant.stackexchange.com/search?q=market+capitalization+data&quot;&gt;Market capitalization data | Quantitative Finance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="85045" LastEditDate="2021-03-30T08:44:47.947" LastActivityDate="2021-03-30T08:44:47.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="591" PostTypeId="2" ParentId="468" CreationDate="2014-06-26T06:51:12.243" Score="3" Body="&lt;p&gt;If you are totally unfamiliar with ordinal regression, I would try to read the Tabachnick / Fidell (&lt;a href=&quot;http://www.pearsonhighered.com/educator/product/Using-Multivariate-Statistics-6E/0205849571.page&quot; rel=&quot;nofollow&quot;&gt;http://www.pearsonhighered.com/educator/product/Using-Multivariate-Statistics-6E/0205849571.page&lt;/a&gt;)  chapter on the topic first - while not written for R, the book is very good at conveying the general logic and the &quot;do's&quot; and &quot;do nots&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a question: What are your response catgeories exactly? If they are some sort of scale, like &quot;good - bad&quot; it would be ok to use a linear regression (market research does it all the time...), but if the items are more disjunct, an ordinal regression might be better.&#xA;I dimly remember that some books about structural equatiotion modelling mentioned that linear regression was superior for good scales than probit - bit I cannot recall the book at the moment, sorry!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most serious problem might be the number of dummy variables - a couple of hundred dummy variables will make the analysis slow, hard to interpret and probably unstable - are there enough cases for each dummy / dummy-combination?&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-06-26T06:51:12.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="592" PostTypeId="2" ParentId="554" CreationDate="2014-06-26T07:41:52.403" Score="4" Body="&lt;p&gt;I usually take a two-step approach &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;compute univariate (variable by variable) summary statistics such as mean, range, variance, number of missing, cardinality, etc. for each variable and look for oddities (e.g. range not plausible given the meaning of the variable). Plot histograms for those odd variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;split the data into manageable subsets (choose a meaningful variable and split the data according to it e.g. all positive examples, and all negative) and explore them visually (e.g. with &lt;a href=&quot;http://www.ggobi.org/&quot; rel=&quot;nofollow&quot;&gt;ggobi&lt;/a&gt; ). Especially use tools like brushing and scatter plots to understand how variables are linked together.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And when you start building models, make sure to plot the residuals, looking for extreme errors that might be due to an outlier, or look at the confusion matrix and make sure it is balanced. Use k-fold cross validation to optimize your models and look at the variance of the training error for each fold, if one fold performs much worse than the others, it may contain outliers.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-06-26T07:41:52.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="593" PostTypeId="2" ParentId="536" CreationDate="2014-06-26T08:01:40.763" Score="3" Body="&lt;p&gt;As many have answered, it is always best to avoid anything done manually as it is less reproducible/documentable. Your point about the overhead of writing a script vs. opening and fixing the file is valid, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best practice is often to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;keep an untouched version of the data&lt;/li&gt;&#xA;&lt;li&gt;build a working copy of the data with errors fixed&lt;/li&gt;&#xA;&lt;li&gt;have a way to re-create a working copy from the original data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The last point can be done with a script. Then make sure to be as specific as needed to modify only the data you want to modify, and to write the script in such a way that adding a fix by modifying the script is as easy as modifying the data directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data lie in files, you can also use diffs/patches to store the original data along with the patches needed to produce the working data. To generate them, duplicate your working copy, perform the change, extract the diff/patch, save it, and delete the previous working copy.&lt;/p&gt;&#xA;" OwnerUserId="172" LastEditorUserId="172" LastEditDate="2014-06-26T12:39:40.443" LastActivityDate="2014-06-26T12:39:40.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="594" PostTypeId="1" CreationDate="2014-06-26T12:11:51.253" Score="5" ViewCount="688" Body="&lt;p&gt;While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and creates categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I couldn't find much about it at all. Is anyone able to shed some light on algorithms they may be using to do this, and what basis these categories are created from?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I search for &quot;animals&quot; I get the categories:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;cute&quot;, &quot;baby&quot;, &quot;wild&quot;, &quot;farm&quot;, &quot;zoo&quot;, &quot;clipart&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I go into &quot;wild&quot;, I then have subcategories:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;forest&quot;, &quot;baby&quot;, &quot;africa&quot;, &quot;clipart&quot;, &quot;rainforest&quot;, &quot;domestic&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1206" LastEditorUserId="84" LastEditDate="2014-06-26T16:19:49.043" LastActivityDate="2016-10-11T12:03:51.227" Title="How does Google categorize results from its image search?" Tags="|machine-learning|classification|google|search|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="595" PostTypeId="1" AcceptedAnswerId="611" CreationDate="2014-06-26T12:25:55.663" Score="2" ViewCount="2279" Body="&lt;p&gt;I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore these parameters to each user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which way do I look for? Which algorithm is suitable to solve this problem? I'm familiar with Neural Networks (supervised learning), but it seems they don't fit.&lt;/p&gt;&#xA;" OwnerUserId="1207" LastEditorUserId="84" LastEditDate="2014-06-26T16:25:31.680" LastActivityDate="2014-06-27T19:18:11.433" Title="How to use neural networks with large and variable number of inputs?" Tags="|machine-learning|data-mining|algorithms|neural-network|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="596" PostTypeId="1" AcceptedAnswerId="5759" CreationDate="2014-06-26T13:32:18.050" Score="10" ViewCount="1015" Body="&lt;p&gt;There are several classic datasets for machine learning classification/regression tasks. The most popular are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html&quot;&gt;Iris Flower Data Set&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kaggle.com/c/titanic-gettingStarted&quot;&gt;Titanic Data Set&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/mtcars.html&quot;&gt;Motor Trend Cars&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But does anyone know similar datasets for networks analysis / graph theory? More concrete - I'm looking for &lt;strong&gt;Gold standard&lt;/strong&gt; datasets for comparing/evaluating/learning:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;centrality measures;&lt;/li&gt;&#xA;&lt;li&gt;network clustering algorithms.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don't need a huge list of publicly available networks/graphs, but a couple of actually must-know datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's quite difficult to provide exact features for &quot;gold standard dataset&quot;, but here are some thoughts. I think, real classic dataset should satisfy these criteria:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Multiple references in articles and textbooks;&lt;/li&gt;&#xA;&lt;li&gt;Inclusion in well-known network analysis software packages;&lt;/li&gt;&#xA;&lt;li&gt;Sufficient time of existence;&lt;/li&gt;&#xA;&lt;li&gt;Usage in a number of courses on graph analysis.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Concerning my field of interest, I also need labeled classes for vertices and/or precomputed (or predefined) &quot;authority scores&quot; (i.e. centrality estimates). After asking this question I continued searching, and here are some suitable examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://networkdata.ics.uci.edu/data.php?id=105&quot;&gt;Zachary's Karate Club&lt;/a&gt;: introduced in 1977, cited more than 1.5k times (according to Google Scholar), vertexes have attribute Faction (which can be used for clustering).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.orgnet.com/Erdos.html&quot;&gt;Erdos Collaboration Network&lt;/a&gt;: unfortunately, I haven't find this network in form of data-file, but it's rather famous, and if someone will enrich network with  mathematicians' specialisations data, it also could be used for testing clustering algorithms.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="941" LastEditorUserId="941" LastEditDate="2014-06-27T05:22:25.863" LastActivityDate="2015-05-11T21:08:34.500" Title="Network analysis classic datasets" Tags="|dataset|graphs|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="597" PostTypeId="2" ParentId="594" CreationDate="2014-06-26T13:45:16.573" Score="3" Body="&lt;p&gt;I am not working at Google, but I think it is some sort of recommendation system based on the words which millions of users searched before. So those people who search for &quot;animals&quot; often search for &quot;wild animals&quot; for example. Like in many online stores they recommend you to buy something in addition to the product you are looking for based on the previous purchases of other users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many approaches how to build such recommendation system using machine learning, no one knows for sure what google uses.&lt;/p&gt;&#xA;" OwnerUserId="478" LastActivityDate="2014-06-26T13:45:16.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="598" PostTypeId="2" ParentId="595" CreationDate="2014-06-26T14:17:44.077" Score="2" Body="&lt;p&gt;There exist many possibilities for populating empty gaps on data.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Most repeated value&lt;/strong&gt;: Fill the gaps with the most common value.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Create a distribution&lt;/strong&gt;: Make the histogram and drop values according to that distribution.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Create a new label&lt;/strong&gt;: Since you do not have information, do not assume any value and create another label/category to indicate that value is empty.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Create a classifier&lt;/strong&gt;: Make a relation among the variable with empty gaps and the rest of the data and create a simple classifier. With this, populate the rest of the data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There exist many others, but these are the most common strategies. My suggestion is not to populate and to keep unknown what is unknown.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-26T14:17:44.077" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="599" PostTypeId="2" ParentId="596" CreationDate="2014-06-26T14:21:10.513" Score="1" Body="&lt;p&gt;The only thing I know about is benchmark data for Graph Databases, such as Neo4j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find links similar to this one:&#xA;&lt;a href=&quot;http://istc-bigdata.org/index.php/benchmarking-graph-databases/&quot; rel=&quot;nofollow&quot;&gt;http://istc-bigdata.org/index.php/benchmarking-graph-databases/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where you can find data to test network analysis and graph theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, you could play with the API of Twitter/Facebook to collect your own data. This is also a suggestion in case you do not find the data you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastActivityDate="2014-06-26T14:21:10.513" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="600" PostTypeId="1" CreationDate="2014-06-26T14:39:06.410" Score="11" ViewCount="2494" Body="&lt;p&gt;I am working on a political campaign where dozens of volunteers will be conducting door-knocking promotions over the next few weeks. Given a list with names, addresses and long/lat coordinates, what algorithms can be used to create an optimized walk list.&lt;/p&gt;&#xA;" OwnerUserId="1208" LastEditorUserId="434" LastEditDate="2014-06-26T18:15:57.907" LastActivityDate="2014-07-07T14:41:59.523" Title="How do you create an optimized walk list given longitude and latitude coordinates?" Tags="|algorithms|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="602" PostTypeId="2" ParentId="594" CreationDate="2014-06-26T16:19:46.480" Score="3" Body="&lt;p&gt;I thought of expanding a bit on the answer by Stanpol. While recommendation system is one approach of suggesting related queries, one more standard information retrieval based approach is the query expansion technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking, query expansion involves selecting additional terms from the top ranked documents retrieved in response to an initial query. Terms are typically selected by a combination of a term scoring function such as tf-idf and a co-occurrence based measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in response to a query term &quot;animal&quot;, a term selection function may choose the term &quot;zoo&quot;, because&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;zoo&quot; may be a dominating term (high tf-idf) in the top (say 10) documents retrieved in response to the query &quot;animal&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;zoo&quot; may co-occur frequently (in close proximity) with the original query term &quot;animal&quot; in these documents&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="984" LastEditorUserId="984" LastEditDate="2014-06-26T23:01:36.187" LastActivityDate="2014-06-26T23:01:36.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="603" PostTypeId="2" ParentId="600" CreationDate="2014-06-26T18:01:06.443" Score="4" Body="&lt;p&gt;People see something closely related to the &lt;a href=&quot;http://en.wikipedia.org/wiki/Travelling_salesman_problem&quot; rel=&quot;nofollow&quot;&gt;Travelling Salesman Problem&lt;/a&gt; and think that it can't be solved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good deal of work has been done on this topic and not all of it indicates that a solution is not available.  Depending on the parameters and the desired solution, you may be able to find something that will work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to give a look at the &lt;a href=&quot;http://openopt.org/TSP&quot; rel=&quot;nofollow&quot;&gt;OpenOpt&lt;/a&gt; python library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another resource to look at would  be the &lt;a href=&quot;http://tspsg.info/&quot; rel=&quot;nofollow&quot;&gt;TSP Solver and Generator&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using R, there is a &lt;a href=&quot;http://tsp.r-forge.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;TSP package available&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually implementing a solution to your problem is a little too much to cover here, but this should provide a good starting point.  Within these packages and at the documentation within the links that I provided for you, you will find that there are a fairly wide variety of algorithmic strategies available.  You have a small geographic region and a small set of &quot;salespeople&quot;, so the computational power needed to calculate a strategy within a reasonable time frame should be available on your desktop.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practical terms, you don't need to find the absolutely most optimal strategy.  You just need a very good one.  Pick a TSP package that looks the least overwhelming and give it a go.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-06-26T18:10:42.907" LastActivityDate="2014-06-26T18:10:42.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="604" PostTypeId="2" ParentId="594" CreationDate="2014-06-26T19:16:31.470" Score="2" Body="&lt;p&gt;Google isn't going to give away their proprietary work, but we can speculate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I can gather from my limited usage:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The recommendations do not seem to be user, geography, or history specific.&lt;/li&gt;&#xA;&lt;li&gt;There is never an empty recommendation (one that returns no results)&lt;/li&gt;&#xA;&lt;li&gt;There is not always a recommendation (some searches just return images)&lt;/li&gt;&#xA;&lt;li&gt;The recommendations are not always the same (consecutive searches sometimes return different recommendations)&lt;/li&gt;&#xA;&lt;li&gt;Result ordering shifts regularly (search for a specific image and it won't always be in the same place)&lt;/li&gt;&#xA;&lt;li&gt;Very popular searches seem to be pre-calculated and more static than unpopular searches.&lt;/li&gt;&#xA;&lt;li&gt;Recommendations are not always one additional word, and recommendations do not always include the base query.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It seems to me that they do this based on the history of the general end user population, they rotate recommendations when there are many popular ones, and they do some additional processing to determine that the result set is of a reasonable size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would postulate that it works as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use consecutive search strings from users (short-to-long-tail searches) as training data for a machine-learning algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Run searches that occur &gt; N amount of times a week against that recommendation algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Validate and clean results.&lt;/li&gt;&#xA;&lt;li&gt;Push them out to the general population in rotation/A-B testing.&lt;/li&gt;&#xA;&lt;li&gt;Track click-throughs.&lt;/li&gt;&#xA;&lt;li&gt;Refine results over time.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-06-26T19:16:31.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="605" PostTypeId="1" CreationDate="2014-06-26T19:23:46.043" Score="6" ViewCount="3320" Body="&lt;p&gt;I'm building a recommender system and using SVD as one of the preprocessing techniques.&#xA;However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (cosine, pearson, euclidean) depend on that assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I take the SVD (A = USV^T), is there a standard way to normalize the matrix 'A' between 0 and 1? Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I want all of my similarity measurements to give results between 0 and 1 and my normalized euclidean distance in particular fails if the input matrix does not have values between 0 and 1.&lt;/p&gt;&#xA;" OwnerUserId="838" LastEditorUserId="838" LastEditDate="2014-06-27T00:22:33.810" LastActivityDate="2014-07-01T18:48:13.757" Title="How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?" Tags="|machine-learning|statistics|recommender-system|feature-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="606" PostTypeId="2" ParentId="554" CreationDate="2014-06-26T20:49:38.470" Score="3" Body="&lt;p&gt;Folks here stated great steps, but I think there are great information at the following link &lt;a href=&quot;http://simplystatistics.org/2014/06/13/what-i-do-when-i-get-a-new-data-set-as-told-through-tweets/&quot; rel=&quot;nofollow&quot;&gt;what I do when I get a new data set as told through tweets&lt;/a&gt;,&#xA;It sums up the steps the folks tweeted answering the great @hmason question &quot;Data people: What is the very first thing you do when you get your hands on a new data set?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it will be useful. &lt;/p&gt;&#xA;" OwnerUserId="1220" LastActivityDate="2014-06-26T20:49:38.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="607" PostTypeId="1" AcceptedAnswerId="629" CreationDate="2014-06-26T22:17:52.570" Score="4" ViewCount="1694" Body="&lt;p&gt;I am brand new to the field of data science, want to break into it, and there are so many tools out there.  These VMs have a lot of software on them, but I haven't been able to find any side-by-side comparison.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a start from my research, but if someone could tell me that one is objectively more rich-featured, with a larger community of support, and useful to get started then that would help greatly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;datasciencetoolKIT.org -&gt; vm is on vagrant cloud (4 GB) and seems to be more &quot;hip&quot; with R, iPython notebook, and other useful command-line tools (html-&gt;txt, json-&gt;xml, etc). There is a book being released in August with detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;datasciencetoolBOX.org -&gt; vm is a vagrant box (24 GB) downloadable from their website. There seems to be more features here, and more literature.&lt;/p&gt;&#xA;" OwnerUserId="1223" LastEditorUserId="1330" LastEditDate="2019-05-07T04:23:55.110" LastActivityDate="2019-05-07T04:23:55.110" Title="Do you need a virtual machine for data science?" Tags="|tools|software-recommendation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="608" PostTypeId="1" AcceptedAnswerId="612" CreationDate="2014-06-26T22:58:32.380" Score="9" ViewCount="636" Body="&lt;p&gt;I have just learned about regularisation as an approach to control over-fitting, and I would like to incorporate the idea into a simple implementation of backpropagation and &lt;a href=&quot;http://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;noreferrer&quot;&gt;Multilayer perceptron&lt;/a&gt; (MLP) that I put together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently to avoid over-fitting, I cross-validate and keep the network with best score so far on the validation set. This works OK, but adding regularisation would benefit me in that correct choice of the regularisation algorithm and parameter would make my network converge on a non-overfit model more systematically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula I have for the update term (from Coursera ML course) is stated as a batch update e.g. for each weight, after summing all the applicable deltas for the whole training set from error propagation, an adjustment of &lt;code&gt;lambda * current_weight&lt;/code&gt; is added as well before the combined delta is subtracted at the end of the batch, where &lt;code&gt;lambda&lt;/code&gt; is the regularisation parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My implementation of backpropagation uses per-item weight updates. I am concerned that I cannot just copy the batch approach, although it looks OK intuitively to me. &lt;em&gt;Does a smaller regularisation term per item work just as well?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance &lt;code&gt;lambda * current_weight / N&lt;/code&gt; where N is size of training set - at first glance this looks reasonable. I could not find anything on the subject though, and I wonder if that is because regularisation does not work as well with a per-item update, or even goes under a different name or altered formula.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2014-06-27T09:52:53.277" LastActivityDate="2014-06-27T12:28:25.467" Title="Any differences in regularisation in MLP between batch and individual updates?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="609" PostTypeId="2" ParentId="410" CreationDate="2014-06-26T23:46:22.487" Score="26" Body="&lt;p&gt;Below is a very good note (page 12) on learning rate in Neural Nets (Back Propagation) by Andrew Ng. You will find details relating to learning rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf&quot;&gt;http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your 4th point, you're right that normally one has to choose a &quot;balanced&quot; learning rate, that should neither overshoot nor converge too slowly. One can plot the learning rate w.r.t. the descent of the cost function to diagnose/fine tune. In practice, Andrew normally uses the L-BFGS algorithm (mentioned in page 12) to get a &quot;good enough&quot; learning rate.&lt;/p&gt;&#xA;" OwnerUserId="1224" LastEditorUserId="322" LastEditDate="2014-06-27T20:17:32.853" LastActivityDate="2014-06-27T20:17:32.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="611" PostTypeId="2" ParentId="595" CreationDate="2014-06-27T07:06:14.297" Score="0" Body="&lt;p&gt;I had almost the same problem: 'restoring' age, gender, location for social network users. But I used users' ego-networks, not visited sites statistics. And I faced with two almost independent tasks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;'Restoring' or 'predicting' data. You can use a bunch of different technics to complete this task, but my vote is for simplest ones (KISS, yes). E.g., in my case, for age prediction, mean of ego-network users' ages gave satisfactory results (for about 70% of users error was less than +/-3 years, in my case it was enough). It's just an idea, but you can try to use for age prediction weighted average, defining weight as similarity measure between visited sites sets of current user and others.&lt;/li&gt;&#xA;&lt;li&gt;Evaluating prediction quality. Algorithm from task-1 will produce prediction almost in all cases. And second task is to determine, if prediction is reliable. E.g., in case of ego network and age prediction: can we trust in prediction, if a user has only one 'friend' in his ego network? This task is more about machine-learning: it's a binary classification problem. You need to compose features set, form training and test samples from your data with both right and wrong predictions. Creating appropriate classifier will help you to filter out unpredictable users. But you need to determine, what are your features set. I used a number of network metrics, and summary statistics on feature of interest distribution among ego-network.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This approach wouldn't populate all the gaps, but only predictable ones.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-06-27T07:06:14.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="612" PostTypeId="2" ParentId="608" CreationDate="2014-06-27T07:19:11.420" Score="2" Body="&lt;p&gt;Regularization is relevant in per-item learning as well. I would suggest to start with a basic validation approach for finding out lambda, whether you are doing batch or per-item learning. This is the easiest and safest approach. Try manually with a number of different values. e.g. 0.001. 0.003, 0.01, 0.03, 0.1 etc. and see how your validation set behaves. Later on you may automate this process by introducing a linear or local search method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, I believe the value of lambda should be considered in relation to the updates of the parameter vector, rather than the training set size. For batch training you have one parameter update &lt;em&gt;per dataset pass&lt;/em&gt;, while for online one update &lt;em&gt;per sample&lt;/em&gt; (regardless of the training set size).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently stumbled upon this &lt;a href=&quot;https://stats.stackexchange.com/questions/64224/regularization-and-feature-scaling-in-online-learning&quot;&gt;Crossvalidated Question&lt;/a&gt;, which seems quite similar to yours. There is a link to a paper about &lt;a href=&quot;http://leon.bottou.org/publications/pdf/jmlr-2009.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a new SGD algorithm&lt;/a&gt;, with some relevant content. It might be useful to take a look (especially pages 1742-1743).&lt;/p&gt;&#xA;" OwnerUserId="1085" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-06-27T09:54:54.977" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="613" PostTypeId="1" AcceptedAnswerId="619" CreationDate="2014-06-27T12:03:53.357" Score="2" ViewCount="861" Body="&lt;p&gt;I understand Hadoop MapReduce and its features but I am confused about R MapReduce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My doubt is:&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;R can do all stats, math and data science related stuff, but why R MapReduce?&lt;/li&gt;&#xA;&lt;li&gt;Is there any new task I can achieve by using R MapReduce instead of Hadoop MapReduce? If yes, please specify.&lt;/li&gt;&#xA;&lt;li&gt;We can achieve the task by using R with Hadoop (directly) but what is the importance of MapReduce in R and how it is different from normal MapReduce?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1235" LastEditorUserId="322" LastEditDate="2014-06-27T19:39:00.247" LastActivityDate="2014-06-27T19:39:00.247" Title="Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce" Tags="|machine-learning|r|apache-hadoop|map-reduce|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="614" PostTypeId="2" ParentId="608" CreationDate="2014-06-27T12:28:25.467" Score="2" Body="&lt;p&gt;To complement what &lt;strong&gt;insys&lt;/strong&gt; said :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regularization is used when computing the backpropagation for all the weights in your MLP.&#xA;Therefore, instead of computing the gradient in regard to all the input of the training set (&lt;code&gt;batch&lt;/code&gt;) you only use some/one item(s) (&lt;code&gt;stochastic or semi-stochastic&lt;/code&gt;).&#xA;You will end up limiting a result of the update in regard to one item instead of all which is also correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if i remember correctly, Andrew NG used &lt;code&gt;L2-regularization&lt;/code&gt;.&#xA;The &lt;code&gt;/N&lt;/code&gt; in &lt;code&gt;lambda * current_weight / N&lt;/code&gt; is not mandatory, it just helps rescaling the input. However if you choose not to use it, you will have (in most of the case) to select another value for &lt;code&gt;lambda&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Hyperparameter_optimization&quot; rel=&quot;nofollow&quot;&gt;Grid-search algorithm&lt;/a&gt; to choose the best value for &lt;code&gt;lambda&lt;/code&gt; (the &lt;em&gt;hyperparameter&lt;/em&gt; =&gt; the one you have to choose).&lt;/p&gt;&#xA;" OwnerUserId="968" LastActivityDate="2014-06-27T12:28:25.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="615" PostTypeId="2" ParentId="468" CreationDate="2014-06-27T13:22:22.577" Score="3" Body="&lt;p&gt;One standard reference written from social science perspective is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0803973748&quot; rel=&quot;nofollow noreferrer&quot;&gt;J Scott Long's Limited Dependent Variables&lt;/a&gt; book. It goes much deeper than say Tabachnik suggested in &lt;a href=&quot;https://datascience.stackexchange.com/a/591/1237&quot;&gt;another answer&lt;/a&gt;: Tabachnik is a cookbook at best, with little to no explanations of the &quot;why&quot;, and it seems like you would benefit from figuring this out in more detail that can be found in Long's book. Ordinal regression should be covered in most introductory econometrics courses (Wooldridge's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262232588&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cross-Section and Panel Data&lt;/a&gt; is a great graduate-level book), as well as quantitative social science courses (sociology, psychology), although I would imagine that the latter will loop back to Long's book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that your number of variables is wa-a-ay lower than the sample size, the R package you should be looking is probably &lt;a href=&quot;http://cran.r-project.org/web/packages/ordinal/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;ordinal&lt;/code&gt;&lt;/a&gt; rather than &lt;code&gt;glmnetcr&lt;/code&gt;. &lt;a href=&quot;https://datascience.stackexchange.com/a/469/1237&quot;&gt;Another answer&lt;/a&gt; mentioned that you can find this functionality in a more mainstream &lt;code&gt;MASS&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="1237" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-06-27T13:22:22.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="616" PostTypeId="1" CreationDate="2014-06-27T15:58:19.340" Score="7" ViewCount="1436" Body="&lt;p&gt;I have a matrix that is populated with discrete elements, and I need to cluster them (using R) into intact groups. So, for example, take this matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[A B B C A]  &#xA;[A A B A A]  &#xA;[A B B C C]  &#xA;[A A A A A]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There would be two separate clusters for A, two separate clusters for C, and one cluster for B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output I'm looking for would ideally assign a unique ID to each cluster, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1 2 2 3 4]  &#xA;[1 1 2 4 4]  &#xA;[1 2 2 5 5]  &#xA;[1 1 1 1 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Right now I wrote a code that does this recursively by just iteratively checking nearest neighbor, but it quickly overflows when the matrix gets large (i.e., 100x100).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a built in function in R that can do this? I looked into raster and image processing, but no luck. I'm convinced it must be out there.&lt;/p&gt;&#xA;" OwnerUserId="1240" LastEditorUserId="322" LastEditDate="2014-06-27T17:06:52.027" LastActivityDate="2014-06-28T14:34:17.487" Title="Identifying “clusters” or “groups” in a matrix" Tags="|r|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="617" PostTypeId="2" ParentId="616" CreationDate="2014-06-27T16:40:17.393" Score="2" Body="&lt;p&gt;What do you think is distance measure in your case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume there are three dimensions here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;RowN&lt;/code&gt; (row number)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;ColN&lt;/code&gt; (column number)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Value&lt;/code&gt; (value: A, B or C)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That means data you get from &lt;code&gt;4x5&lt;/code&gt; matrix looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sample1 -&amp;gt; (1, 1, A)&#xA;Sample2 -&amp;gt; (1, 2, B)&#xA;...&#xA;Sample5 -&amp;gt; (1, 5, A)&#xA;Sample6 -&amp;gt; (2, 1, A)&#xA;...&#xA;Sample15 -&amp;gt; (3, 5, C)&#xA;...&#xA;Sample20 -&amp;gt; (4, 5, A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is &lt;code&gt;value&lt;/code&gt; scaled? In other words, is &lt;code&gt;A &amp;lt; B &amp;lt; C&lt;/code&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, then&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you can replace &lt;code&gt;{A, B, C}&lt;/code&gt; with &lt;code&gt;{0, 1, 2}&lt;/code&gt; (or may be &lt;code&gt;{10, 11, 12}&lt;/code&gt;, if you want this difference be less important than RowN and ColN attributes)&lt;/li&gt;&#xA;&lt;li&gt;normalize your data&lt;/li&gt;&#xA;&lt;li&gt;use, for example, K-Means clustering algorithm (&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html&lt;/a&gt;) from &lt;code&gt;stats&lt;/code&gt; R package&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In that case the distance between two will be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sqrt( (RowN1-RowN2)^2 + (ColN1-ColN2)^2 + (Value1-Value2)^2 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If &lt;code&gt;value&lt;/code&gt; is not scaled (regular categorical variable), use some &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;modifications of K-Means that work with categorical data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in case of 100x100 matrix you have 10000 observations and three variables, which is pretty trivial sample size.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-06-27T16:49:26.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="618" PostTypeId="2" ParentId="595" CreationDate="2014-06-27T19:18:11.433" Score="1" Body="&lt;p&gt;Although adesantos has already given a good answer, I would like to add a little background information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The name for the problem you are looking at is &quot;imputation&quot;. As adesantos already said, one of the possibilities is to fit a distribution. For example, you could fit a multivariate Gaussian to the data. You will get the mean only from the samples you know and you calculate the covariances only from the samples you know. You can then use standard MVG results to impute the missing data linearly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is probably the simplest probabilistic method of imputation and it is already quite involved. If you are a neural networks, a recently proposed method that can do so are deep latent gaussian models by Rezende et al. However, understand the method will require a lot of neural net knowledge, quite some variational Bayes knowledge about Markov chains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method, which I have hear to work well is to train a generative stochastic network (Bengio et al). This is done by training a denoising auto encoder on the data you have (neglecting missing values in the reconstruction loss). Say you have a reconstruction function f and a input x. Then you will reconstruct it via x' = f(x). You then reset the values of x' with those you know from x. (I.e. you only keep the values that were missing before reconstruction.) If you do so many times, you are guaranteed to sample from the distribution given the values you know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in either case, these methods require quite some knowledge about statistics and neural nets.&lt;/p&gt;&#xA;" OwnerUserId="1193" LastActivityDate="2014-06-27T19:18:11.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="619" PostTypeId="2" ParentId="613" CreationDate="2014-06-27T19:37:44.847" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop&quot; rel=&quot;nofollow&quot;&gt;rhadoop&lt;/a&gt; (the part you are interested in is now called &lt;a href=&quot;https://github.com/RevolutionAnalytics/rmr2&quot; rel=&quot;nofollow&quot;&gt;rmr2&lt;/a&gt;) is simply a client API for MapReduce written in R. You invoke MapReduce using R package APIs, and send an R function to the workers, where it is executed by an R interpreter locally. But it is otherwise exactly the same MapReduce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can call anything you like in R this way, but no R functions are themselves parallelized to use MapReduce in this way. The point is simply that you can invoke M/R from R. I don't think it somehow lets you do anything more magical than that.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-06-27T19:37:44.847" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="620" PostTypeId="2" ParentId="454" CreationDate="2014-06-27T21:45:23.237" Score="12" Body="&lt;p&gt;Yes, it's problematic.  If you oversample the minority, you risk overfitting.  If you undersample the majority, you risk missing aspects of the majority class.  Stratified sampling, btw, is the equivalent to assigning non-uniform misclassification costs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Independently sampling several subsets from the majority class and making multiple classifiers by combining each subset with all the minority class data, as suggested in the answer from @Debasis and described in this &lt;a href=&quot;http://cse.seu.edu.cn/people/xyliu/publication/tsmcb09.pdf&quot;&gt;EasyEnsemble paper&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) &lt;a href=&quot;http://arxiv.org/pdf/1106.1813.pdf&quot;&gt;SMOTE (Synthetic Minority Oversampling Technique)&lt;/a&gt; or &lt;a href=&quot;http://www3.nd.edu/~nchawla/papers/ECML03.pdf&quot;&gt;SMOTEBoost, (combining SMOTE with boosting)&lt;/a&gt; to create synthetic instances of the minority class by making nearest neighbors in the feature space.  SMOTE is implemented in R in &lt;a href=&quot;http://cran.r-project.org/web/packages/DMwR/index.html&quot;&gt;the DMwR package&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="953" LastActivityDate="2014-06-27T21:45:23.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="621" PostTypeId="2" ParentId="455" CreationDate="2014-06-28T01:07:27.263" Score="5" Body="&lt;p&gt;There is a bunch of datasets made free by UC Irvine &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html&quot;&gt;to play with here&lt;/a&gt;. Among those datasets, &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html?format=&amp;amp;task=&amp;amp;att=&amp;amp;area=&amp;amp;numAtt=&amp;amp;numIns=&amp;amp;type=text&amp;amp;sort=nameUp&amp;amp;view=table&quot;&gt;there are a few dozen textual datasets&lt;/a&gt; that might help you guys with your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those are kind of generic datasets, so depending on your purpose they should not be used as the only data to train your models, or else your model -- while it might work -- will not produce quality results.&lt;/p&gt;&#xA;" OwnerUserId="553" LastActivityDate="2014-06-28T01:07:27.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="622" PostTypeId="2" ParentId="616" CreationDate="2014-06-28T14:34:17.487" Score="1" Body="&lt;p&gt;I'm not sure if your question classifies as a clustering problem. In clustering you are trying to discover clusters of similar examples using unlabelled data. Here, it seems you wish to enumerate existing &quot;clusters&quot; of nearby nodes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be honest, I have no idea of such a function in R. But, as far as the algorithm is concerned, I believe what you are looking for is &lt;a href=&quot;http://en.wikipedia.org/wiki/Connected-component_labeling&quot; rel=&quot;nofollow&quot;&gt;Connected-Component Labeling&lt;/a&gt;. Kind of a bucket fill, for matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wikipedia article is linked above. One of the algorithms presented there, termed as single-pass algorithm, is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;One-Pass(Image)&#xA;        [M, N]=size(Image);&#xA;        Connected = zeros(M,N);&#xA;        Mark = Value;&#xA;        Difference = Increment;&#xA;        Offsets = [-1; M; 1; -M];&#xA;        Index = [];&#xA;        No_of_Objects = 0; &#xA;&#xA;   for i: 1:M :&#xA;       for j: 1:N:&#xA;            if(Image(i,j)==1)            &#xA;                 No_of_Objects = No_of_Objects +1;            &#xA;                 Index = [((j-1)*M + i)];           &#xA;                 Connected(Index)=Mark;            &#xA;                 while ~isempty(Index)                &#xA;                      Image(Index)=0;                &#xA;                      Neighbors = bsxfun(@plus, Index, Offsets');&#xA;                      Neighbors = unique(Neighbors(:));                &#xA;                      Index = Neighbors(find(Image(Neighbors)));                                &#xA;                      Connected(Index)=Mark;&#xA;                 end            &#xA;                 Mark = Mark + Difference;&#xA;            end&#xA;      end&#xA;  end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess it'd be easy to roll your own using the above.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2014-06-28T14:34:17.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="623" PostTypeId="1" CreationDate="2014-06-28T14:57:18.177" Score="13" ViewCount="366" Body="&lt;p&gt;I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if there's a meaningful way of comparing the performance of these classifiers in a similar way, but which also takes into account the total numbers of each entity in the test data that's being classified?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I'm using precision/recall as a measure of performance, so might have something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                    Precision Recall&#xA;Person classifier   65%       40%&#xA;Company classifier  98%       90%&#xA;Cheese classifier   10%       50%&#xA;Egg classifier      100%      100%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the dataset I'm running these on might contain 100k people, 5k companies, 500 cheeses, and 1 egg.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is there a summary statistic I can add to the above table which also takes into account the total number of each item? Or is there some way of measuring the fact that e.g. 100% prec/rec on the Egg classifier might not be meaningful with only 1 data item?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we had hundreds of such classifiers, I guess I'm looking for a good way to answer questions like &quot;Which classifiers are underperforming? Which classifiers lack sufficient test data to tell whether they're underperforming?&quot;. &lt;/p&gt;&#xA;" OwnerUserId="474" LastEditorUserId="31513" LastEditDate="2017-06-05T19:30:18.057" LastActivityDate="2020-08-06T11:03:56.237" Title="Measuring performance of different classifiers with different sample sizes" Tags="|classification|performance|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="624" PostTypeId="2" ParentId="607" CreationDate="2014-06-29T07:06:00.430" Score="2" Body="&lt;p&gt;In most cases a practicing data scientist creates his own working environment on personal computed installing preferred software packages. Normally it is sufficient and efficient use of computing resources, because to run a virtual machine (VM) on your main machine you have to allocate a significant portion of RAM for it. The software will run noticeably slower on both the main and the virtual machine unless a lot of RAM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to this impact on speed it is not common to use VMs as main working environment but they are a good solution in several cases when there is a need of additional working environment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The VMs be considered when:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There is a need to easily replicate a number of identical computing&#xA;environments when teaching a course or doing a presentation on a&#xA;conference.&lt;/li&gt;&#xA;&lt;li&gt;There is a need to save and recreate an exact environment for an experiment or a calculation.&lt;/li&gt;&#xA;&lt;li&gt;There is a need to run a different OS or to test a solution on a tool that runs on a different OS. &lt;/li&gt;&#xA;&lt;li&gt;One wants to try out a bundle of software tools before installing&#xA;them on the main machine. E.g. there is an opportunity to instal an instance of Hadoop (CDH) on a VM during an &lt;a href=&quot;https://www.udacity.com/course/ud617&quot; rel=&quot;nofollow&quot;&gt;Intro to Hadoop&lt;/a&gt; course on Udacity. &lt;/li&gt;&#xA;&lt;li&gt;VMs are sometimes used for fast deployment in the cloud like AWS EC, Rackspace etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The VMs mentioned in the original question are made as easily installable data science software bundles. There are more than these two. This &lt;a href=&quot;http://jeroenjanssens.com/2013/12/07/lean-mean-data-science-machine.html&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; by Jeroen Janssens gives a comparison of at least four:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Data Science Toolbox &lt;/li&gt;&#xA;&lt;li&gt;Mining the Social Web &lt;/li&gt;&#xA;&lt;li&gt;Data Science Toolkit &lt;/li&gt;&#xA;&lt;li&gt;Data Science Box  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="454" LastEditorUserId="454" LastEditDate="2014-06-29T07:20:51.053" LastActivityDate="2014-06-29T07:20:51.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="625" PostTypeId="2" ParentId="596" CreationDate="2014-06-29T12:48:08.840" Score="4" Body="&lt;p&gt;Maybe you can check here - &lt;a href=&quot;http://snap.stanford.edu/data/&quot; rel=&quot;nofollow&quot;&gt;http://snap.stanford.edu/data/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each data set you will also see references of the works where they have been used&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-06-29T12:48:08.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="626" PostTypeId="2" ParentId="623" CreationDate="2014-06-29T21:11:47.727" Score="2" Body="&lt;p&gt;In my opinion, it is difficult to compare the performance when there is such a big difference of size. On this link, (please check it out &lt;a href=&quot;http://en.wikipedia.org/wiki/Effect_size&quot; rel=&quot;nofollow noreferrer&quot;&gt;here in Wikipedia&lt;/a&gt;), you may see different strategies.&lt;/p&gt;&#xA;&lt;p&gt;The one I suggest is one related to the variance. For instance, consider the performance of the classifier (100%) and the person classifier (65%). The minimum error you commit with the former classifier is 100%. However, the minimum error you can commit with the latter classifier is 10e-5.&lt;/p&gt;&#xA;&lt;p&gt;So one way to compare classifier is to have on mind this &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Rule_of_three_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rule of Three&lt;/a&gt;&lt;/strong&gt; where you can compare the performance and its variability.&lt;/p&gt;&#xA;&lt;p&gt;Other possibility is &lt;strong&gt;F-measure&lt;/strong&gt; which is a combination of Precision and Recall and it is somehow independent to the effect size.&lt;/p&gt;&#xA;" OwnerUserId="1155" LastEditorUserId="98307" LastEditDate="2020-08-06T11:03:56.237" LastActivityDate="2020-08-06T11:03:56.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="627" PostTypeId="2" ParentId="61" CreationDate="2014-06-29T22:44:02.560" Score="18" Body="&lt;p&gt;Roughly speaking, over-fitting typically occurs when the ratio&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/AVVpB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is too high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of over-fitting as a situation where your model learn the training data by heart instead of learning the big pictures which prevent it from being able to generalized to the test data: this happens when the model is too complex with respect to the size of the training data, that is to say when the size of the training data is to small in comparison with the model complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if your data is in two dimensions, you have 10000 points in the training set and the model is a line, you are likely to &lt;em&gt;under&lt;/em&gt;-fit.&lt;/li&gt;&#xA;&lt;li&gt;if your data is in two dimensions, you have 10 points in the training set and the model is 100-degree polynomial, you are likely to &lt;em&gt;over&lt;/em&gt;-fit.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/HH23h.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a theoretical standpoint, the amount of data you need to properly train your model is a crucial yet far-to-be-answered question in machine learning. One such approach to answer this question is the &lt;a href=&quot;https://math.stackexchange.com/a/656167/24265&quot;&gt;VC dimension&lt;/a&gt;. Another is the &lt;a href=&quot;https://www.quora.com/Machine-Learning/What-is-an-intuitive-explanation-for-bias-variance-tradeoff/answer/Franck-Dernoncourt?share=1&quot; rel=&quot;noreferrer&quot;&gt;bias-variance tradeoff&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From an empirical standpoint, people typically plot the training error and the test error on the same plot and make sure that they don't reduce the training error at the expense of the test error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/I7LiT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would advise to watch &lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;noreferrer&quot;&gt;Coursera' Machine Learning course&lt;/a&gt;, section &quot;10: Advice for applying Machine Learning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(PS: please go &lt;a href=&quot;https://datascience.meta.stackexchange.com/q/6/843&quot;&gt;here&lt;/a&gt; to ask for TeX support on this SE.) &lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2014-06-29T22:44:02.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="628" PostTypeId="2" ParentId="361" CreationDate="2014-06-29T23:14:39.513" Score="51" Body="&lt;p&gt;A model underfits when it is too simple with regards to the data it is trying to model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to detect such situation is to use the &lt;a href=&quot;http://en.wikipedia.org/wiki/Bias%E2%80%93variance_dilemma&quot; rel=&quot;noreferrer&quot;&gt;bias–variance approach&lt;/a&gt;, which can represented like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/t0zit.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model is underfitted when you have a high bias.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training and test errors:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;High bias: This learning curve shows high error on both the training and test sets, so the algorithm is suffering from high bias:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/KFjM4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;High variance: This learning curve shows a large gap between training and test set errors, so the algorithm is suffering from high variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ypj9y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If an algorithm is suffering from high variance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;more data will probably help&lt;/li&gt;&#xA;&lt;li&gt;otherwise reduce the model complexity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If an algorithm is suffering from high bias:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;increase the model complexity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would advise to watch &lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;noreferrer&quot;&gt;Coursera' Machine Learning course&lt;/a&gt;, section &quot;10: Advice for applying Machine Learning&quot;, from which I took the above graphs.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2014-07-02T16:55:27.367" LastActivityDate="2014-07-02T16:55:27.367" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="629" PostTypeId="2" ParentId="607" CreationDate="2014-06-30T01:51:11.027" Score="3" Body="&lt;h1&gt;Do you need a VM?&lt;/h1&gt;&#xA;&lt;p&gt;You need to keep in mind that a virtual machine is a software emulation of your own or another machine hardware configuration that can run an operating systems. In most basic terms, it acts as a layer interfacing between the virtual OS, and your own OS which then communicates with the lower level hardware to provide support to the virtual OS. What this means for you is:&lt;/p&gt;&#xA;&lt;h1&gt;Cons&lt;/h1&gt;&#xA;&lt;h3&gt;Hardware Support&lt;/h3&gt;&#xA;&lt;p&gt;A drawback of virtual machine technology is that it supports only the hardware that both the virtual machine hypervisor and the guest operating system support. Even if the guest operating system supports the physical hardware, it sees only the virtual hardware presented by the virtual machine.&#xA;The second aspect of virtual machine hardware support is the hardware presented to the guest operating system. No matter the hardware in the host, the hardware presented to the guest environment is usually the same (with the exception of the CPU, which shows through). For example, VMware GSX Server presents an AMD PCnet32 Fast Ethernet card or an optimized VMware-proprietary network card, depending on which you choose. The network card in the host machine does not matter. VMware GSX Server performs the translation between the guest environment's network card and the host environment's network card. This is great for standardization, but it also means that host hardware that VMware does not understand will not be present in the guest environment.&lt;/p&gt;&#xA;&lt;h3&gt;Performance Penalty&lt;/h3&gt;&#xA;&lt;p&gt;Virtual machine technology imposes a performance penalty from running an additional layer above the physical hardware but beneath the guest operating system. The performance penalty varies based on the virtualization software used and the guest software being run. This is significant.&lt;/p&gt;&#xA;&lt;h1&gt;Pros&lt;/h1&gt;&#xA;&lt;h3&gt;Isolation&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;One of the key reasons to employ virtualization is to isolate applications from each other. Running everything on one machine would be great if it all worked, but many times it results in undesirable interactions or even outright conflicts. The cause often is software problems or business requirements, such as the need for isolated security. Virtual machines allow you to isolate each application (or group of applications) in its own sandbox environment. The virtual machines can run on the same physical machine (simplifying IT hardware management), yet appear as independent machines to the software you are running. For all intents and purposes—except performance, the virtual machines are independent machines. If one virtual machine goes down due to application or operating system error, the others continue running, providing services your business needs to function smoothly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Standardization&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Another key benefit virtual machines provide is standardization. The hardware that is presented to the guest operating system is uniform for the most part, usually with the CPU being the only component that is &amp;quot;pass-through&amp;quot; in the sense that the guest sees what is on the host. A standardized hardware platform reduces support costs and increases the share of IT resources that you can devote to accomplishing goals that give your business a competitive advantage. The host machines can be different (as indeed they often are when hardware is acquired at different times), but the virtual machines will appear to be the same across all of them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Ease of Testing&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Virtual machines let you test scenarios easily. Most virtual machine software today provides snapshot and rollback capabilities. This means you can stop a virtual machine, create a snapshot, perform more operations in the virtual machine, and then roll back again and again until you have finished your testing. This is very handy for software development, but it is also useful for system administration. Admins can snapshot a system and install some software or make some configuration changes that they suspect may destabilize the system. If the software installs or changes work, then the admin can commit the updates. If the updates damage or destroy the system, the admin can roll them back.&#xA;Virtual machines also facilitate scenario testing by enabling virtual networks. In VMware Workstation, for example, you can set up multiple virtual machines on a virtual network with configurable parameters, such as packet loss from congestion and latency. You can thus test timing-sensitive or load-sensitive applications to see how they perform under the stress of a simulated heavy workload.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Mobility&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Virtual machines are easy to move between physical machines. Most of the virtual machine software on the market today stores a whole disk in the guest environment as a single file in the host environment. Snapshot and rollback capabilities are implemented by storing the change in state in a separate file in the host information. Having a single file represent an entire guest environment disk promotes the mobility of virtual machines. Transferring the virtual machine to another physical machine is as easy as moving the virtual disk file and some configuration files to the other physical machine. Deploying another copy of a virtual machine is the same as transferring a virtual machine, except that instead of moving the files, you copy them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h1&gt;Which VM should I use if I am starting out?&lt;/h1&gt;&#xA;&lt;p&gt;The Data Science Box or the Data Science Toolbox are your best bets if you just getting into data science. They have the basic software that you will need, with the primary difference being the virtual environment in which each of these can run. The DSB can run on AWS while the DST can run on Virtual Box (which is the most common tool used for VMs).&lt;/p&gt;&#xA;&lt;h1&gt;Sources&lt;/h1&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.devx.com/vmspecialreport/Article/30383&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.devx.com/vmspecialreport/Article/30383&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jeroenjanssens.com/2013/12/07/lean-mean-data-science-machine.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://jeroenjanssens.com/2013/12/07/lean-mean-data-science-machine.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="62" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2014-06-30T01:51:11.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="630" PostTypeId="2" ParentId="600" CreationDate="2014-06-30T04:00:35.077" Score="1" Body="&lt;p&gt;Here's a crazy idea: talk to the volunteers who know the neighborhoods and who have done door-to-door work before.  Get their advice and ideas.  They will probably have insights that no algorithm will produce, and those modifications will be valuable to any computer-generated route list.  One example: Avoiding crossing heavily traveled streets with slow lights or no lights.  Another example: pairs of volunteers working on opposite sides of the same street will feel safer than a volunteer working that street alone.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-06-30T04:00:35.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="634" PostTypeId="1" AcceptedAnswerId="635" CreationDate="2014-06-30T09:43:01.940" Score="10" ViewCount="189" Body="&lt;p&gt;I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder what is the best way to handle it (from the development process perspective)? Just adding a new feature into the feature vector and re-training the classifier seems to be a naive approach, because too much time will be spent for re-learning of the old features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking along the way of training a classifier for each feature (or a couple of related features), and then combining the results of those classifiers with an overall classifier. Are there any drawbacks of this approach? How can I choose an algorithm for the overall classifier?&lt;/p&gt;&#xA;" OwnerUserId="1271" LastEditorUserId="322" LastEditDate="2014-07-09T00:19:42.423" LastActivityDate="2014-07-11T14:27:01.603" Title="Handling a regularly increasing feature set" Tags="|machine-learning|bigdata|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="635" PostTypeId="2" ParentId="634" CreationDate="2014-06-30T10:47:06.453" Score="4" Body="&lt;p&gt;In an ideal world, you retain all of your historical data, and do indeed run a new model with the new feature extracted retroactively from historical data. I'd argue that the computing resource spent on this is quite useful actually. Is it really a problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, it's a widely accepted technique to build an ensemble of classifiers and combine their results. You can build a new model in parallel just on new features and average in its prediction. This should add value, but, you will never capture interaction between the new and old features this way, since they will never appear together in a classifier. &lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-06-30T10:47:06.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="636" PostTypeId="1" CreationDate="2014-06-30T12:05:38.597" Score="3" ViewCount="1143" Body="&lt;p&gt;I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have been advised to try logit since logit seems to be what MS and Google are using. I have some questions regarding logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Click and non click is a very very unbalanced class and the simple GLM predictions do not look good. How can I make the data work better with the GLM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the variables I have are categorical and things like device and city can be numerous. Also the frequency of occurrence of some devices or some cities can be very very low. How can I deal with this distribution of categorical variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the variables that we get is device ID. This is a very unique feature that can be translated to a user's identity. How can I make use of it in logit, or should it be used in a completely different model based on user identity?&lt;/p&gt;&#xA;" OwnerUserId="1273" LastEditorUserId="322" LastEditDate="2014-07-09T00:19:12.683" LastActivityDate="2014-07-09T00:19:12.683" Title="Data preparation and machine learning algorithm for click prediction" Tags="|machine-learning|bigdata|data-mining|dataset|data-cleaning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="637" PostTypeId="2" ParentId="634" CreationDate="2014-06-30T13:56:10.753" Score="2" Body="&lt;p&gt;Here's an idea that just popped out of the blue – what if you make use of &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_subspace_method&quot; rel=&quot;nofollow&quot;&gt;Random Subspace Sampling&lt;/a&gt; (as in fact Sean Owen already suggested) to train a bunch of new classifiers every time a new feature appears (using a random feature subset, including the new set of features). You could train those models on a subset of samples as well to save some training time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way you can have new classifiers possibly taking on both new and old features, and at the same time keeping your old classifiers. You might even, perhaps using a cross validation technique to measure each classifier's performance, be able to kill-off the worst performing ones after a while, to avoid a bloated model.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2014-06-30T13:56:10.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="640" PostTypeId="1" AcceptedAnswerId="642" CreationDate="2014-06-30T20:51:58.640" Score="5" ViewCount="273" Body="&lt;p&gt;I'm currently working on a project that would benefit from personalized predictions.  Given an input document, a set of output documents, and a history of user behavior, I'd like to predict which of the output documents are clicked.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I'm wondering what the typical approach to this kind of personalization problem is.  Are models trained per user, or does a single global model take in summary statistics of past user behavior to help inform that decision?  Per user models won't be accurate until the user has been active for a while, while most global models have to take in a fixed length feature vector (meaning we more or less have to compress a stream of past events into a smaller number of summary statistics).  &lt;/p&gt;&#xA;" OwnerUserId="684" LastActivityDate="2014-06-30T23:10:53.397" Title="Large Scale Personalization - Per User vs Global Models" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="641" PostTypeId="1" CreationDate="2014-06-30T21:02:05.053" Score="21" ViewCount="9744" Body="&lt;p&gt;I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more &quot;informal&quot; than the news article and journal entries that many of today's state of the art named entity recognition systems are trained on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations? So far I've only been able to locate 50k tokens from twitter published &lt;a href=&quot;https://github.com/aritter/twitter_nlp/blob/master/data/annotated/ner.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="29169" LastEditDate="2019-06-08T03:31:44.327" LastActivityDate="2021-01-11T06:31:25.200" Title="Dataset for Named Entity Recognition on Informal Text" Tags="|dataset|nlp|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="642" PostTypeId="2" ParentId="640" CreationDate="2014-06-30T23:10:53.397" Score="4" Body="&lt;p&gt;The answer to this question is going to vary pretty wildly depending on the size and nature of your data. At a high level, you could think of it as a special case of multilevel models; you have the option of estimating a model with complete pooling (i.e., a universal model that doesn't distinguish between users), models with no pooling (a separate model for each user), and partially pooled models (a mixture of the two). You should really read Andrew Gelman on this topic if you're interested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also think of this as a learning-to-rank problem that either tries to produce point-wise estimates using a single function or instead tries to optimize on some list-wise loss function (e.g., NDCG). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As with most machine learning problems, it all depends on what kind of data you have, the quality of it, the sparseness of it, and what kinds of features you are able to extract from it. If you have reason to believe that each and every user is going to be pretty unique in their behavior, you might want to build a per-user model, but that's going to be unwieldy fast -- and what do you do when you are faced with a new user?&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2014-06-30T23:10:53.397" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="643" PostTypeId="2" ParentId="511" CreationDate="2014-07-01T08:04:30.127" Score="5" Body="&lt;p&gt;I guess you say that you want to use 3-fold cross-validation because you know something about your data (that using k=10 would cause overfitting? I'm curious to your reasoning). I am not sure that you know this, if not then you can simply use a larger k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still think that you cannot use standard k-fold cross-validation, then you could modify the algorithm a bit: say that you split the data into 30 folds and each time use 20 for training and 10 for evaluation (and then shift up one fold and use the first and the last 9 as evaluation and the rest as training). This means that you're able to use all your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use k-fold cross-validation I usually run the process multiple times with a different randomisation to make sure that I have sufficient data, if you don't you will see different performances depending on the randomisation. In such cases I would suggest sampling. The trick then is to do it often enough.&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2014-07-01T08:04:30.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="644" PostTypeId="1" AcceptedAnswerId="649" CreationDate="2014-07-01T13:44:23.290" Score="10" ViewCount="4100" Body="&lt;p&gt;For a recommendation system I'm using cosine similarity to compute similarities between items. However, for items with small amounts of data I'd like to bin them under a general &quot;average&quot; category (in the general not mathematical sense). To accomplish this I'm currently trying to create a synthetic observation to represent that middle of the road point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example if these were my observations (rows are observations, cols are features):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[0, 0, 0, 1, 1, 1, 0, 1, 0],&#xA; [1, 0, 1, 0, 0, 0, 1, 0, 0],&#xA; [1, 1, 1, 1, 0, 1, 0, 1, 1],&#xA; [0, 0, 1, 0, 0, 1, 0, 1, 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A strategy where I'd simply take the actual average of all features across observations would generate a synthetic datapoint such as follows, which I'd then append to the matrix before doing the similarity calculation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.5 ,  0.25,  0.75,  0.5 ,  0.25,  0.75,  0.25,  0.75,  0.25]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While this might work well with certain similarity metrics (e.g. L1 distance) I'm sure there are much better ways for cosine similarity. Though, at the moment, I'm having trouble reasoning my way through angles between lines in high dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="754" LastEditorUserId="754" LastEditDate="2014-07-01T15:33:10.697" LastActivityDate="2023-04-13T05:11:50.153" Title="Create most &quot;average&quot; cosine similarity observation" Tags="|recommender-system|similarity|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="645" PostTypeId="2" ParentId="623" CreationDate="2014-07-01T17:01:12.263" Score="5" Body="&lt;p&gt;You need to look at the confidence interval of the statistic.  This helps measure how much uncertainty in the statistic, which is largely a function of sample size.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-07-01T17:01:12.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="646" PostTypeId="1" CreationDate="2014-07-01T17:03:31.907" Score="2" ViewCount="30" Body="&lt;p&gt;I'm trying to build a nonparametric density function for a fairly large dataset that can be evaluated efficently, and can be updated efficiently when new points are added. There will only ever be a maximum of 4 independent variables, but we can start off with 2. Lets use a gaussian kernel. Let the result be a probability density function, i.e. its volume will be 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each evaluation, we can omit all points for which the evaluation point is outside a certain ellipsoid corresponding to the minimum gaussian value we care about. We can change this threshold for accuracy or performance, and the maximum number of points inside the threshold will depend on the chosen covariance matrix of the kernel. Then, we can evaluate the distribution approximately using the subset of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we use a fixed kernel, then we can use the eigenvalues and eigenvectors we get from the covariance matrix to transform each point so that the threshold ellipsoid is a fixed circle. We can then shove all the transformed points into a spatial index, and efficiently find all points within the required radius of the evaluation point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we would the kernel to be variable for two reasons. (1) to fit the data better, and (2) because adding or modifying points would require the fixed kernel to be updated, which would mean that the entire data set would need to be reindexed. With a variable kernel, we could make new/updated points only affect the closest points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, is there a spatial index that can efficiently find ellipses surrounding a given point from a set of around 10 million ellipses of different shapes and sizes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally though, does my approach look sound? I am open to answers like &quot;give up and precalculate a grid of results&quot;. Answers much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="1283" LastActivityDate="2014-07-01T17:03:31.907" Title="Spatial index for variable kernel nonparametric density" Tags="|data-indexing-techniques|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="648" PostTypeId="2" ParentId="605" CreationDate="2014-07-01T18:48:13.757" Score="3" Body="&lt;p&gt;Singular Value Decomposition is a linear algebraic technique as a result of which the notion of normalization is hard to define. In principle, you can do this normalization by dividing each element A(i,j) of the matrix by the sum (or max) of the elements in that particular (ith) row, i.e. A(i,j) = A(i, j) / \sum_{k=1}^{n} A(i,k)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a more elegant way to achieve this would be to apply a probabilistic dimensionality reduction technique such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Probabilistic_latent_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;PLSA&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt;. These dimensionality reduction techniques ensure that you always end up with probability values strictly between 0 and 1.  &lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-07-01T18:48:13.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="649" PostTypeId="2" ParentId="644" CreationDate="2014-07-01T23:11:35.627" Score="12" Body="&lt;p&gt;You are doing the correct thing. Technically, this averaging leads to computing the &lt;a href=&quot;http://en.wikipedia.org/wiki/Centroid&quot;&gt;centroid&lt;/a&gt; in the Euclidean space of a set of N points. The centroid works pretty well with cosine similarities (cosine of the angles between normalized vectors), e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Rocchio_algorithm&quot;&gt;the Rocchio algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-07-01T23:11:35.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="650" PostTypeId="2" ParentId="644" CreationDate="2014-07-02T08:40:09.773" Score="2" Body="&lt;p&gt;As @Debasis writes, what you are doing is correct. Do not worry about the relative values as the cosine similarity focusses on the angle, not the length of the vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the averages that you compute actually are the proportion of observations that have that feature in your subset. If the subset is sufficiently large, you can interpret them as probabilities.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-07-02T08:40:09.773" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="651" PostTypeId="2" ParentId="623" CreationDate="2014-07-02T08:50:55.980" Score="2" Body="&lt;p&gt;The number of data in the class is sometimes referred to as the &lt;code&gt;support&lt;/code&gt; of the classifier. It tells how much you can trust your result, like a p-value would allow you to trust or distrust some test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach you can use is to compute several classifier performance measures, not only precision and recall, but also true positive rate, false positive rate, specificity, sensitivity, positive likelihood, negative likelihood, etc. and see whether they are consistent with one another. If one of the measure maxes out (100%) and the other do not, it is often, in my experience, indicative of something went wrong (e.g. poor support, trivial classifier, biased classifier, etc.). See &lt;a href=&quot;http://www.damienfrancois.be/blog/files/modelperfcheatsheet.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; for a list of classifier performance measures.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-07-02T08:50:55.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="652" PostTypeId="2" ParentId="361" CreationDate="2014-07-02T08:58:25.413" Score="5" Body="&lt;p&gt;Simply, one common approach is to increase the complexity of the model, making it simple, and most probably underfitting at first, and increasing the complexity of the model  until early signs of overfitting are witnessed using a resampling technique such as cross validation, bootstrap, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You increase the complexity either by adding parameters (number of hidden neurons for artificial neural networks, number of trees in a random forest) or by relaxing the regularization (often named lambda, or C for support vector machines) term in your model.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-07-02T08:58:25.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="653" PostTypeId="1" AcceptedAnswerId="654" CreationDate="2014-07-02T13:40:27.000" Score="4" ViewCount="333" Body="&lt;p&gt;I am trying to find which classification methods, that do not use a training phase, are available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The scenario is gene expression based classification, in which you have a matrix of gene expression of m genes (features) and n samples (observations).&#xA;A signature for each class is also provided (that is a list of the features to consider to define to which class belongs a sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An application (non-training) is the &lt;a href=&quot;http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0015543&quot; rel=&quot;nofollow&quot;&gt;Nearest Template Prediction&lt;/a&gt; method. In this case it is computed the cosine distance between each sample and each signature (on the common set of features). Then each sample is assigned to the nearest class (the sample-class comparison resulting in a smaller distance). No already classified samples are needed in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different application (training) is the &lt;a href=&quot;http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow&quot;&gt;kNN&lt;/a&gt; method, in which we have a set of already labeled samples. Then, each new sample is labeled depending on how are labeled the k nearest samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other non-training methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2015-04-12T16:08:13.467" Title="Which non-training classification methods are available?" Tags="|classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="654" PostTypeId="2" ParentId="653" CreationDate="2014-07-02T14:14:22.593" Score="7" Body="&lt;p&gt;What you are asking about is &lt;a href=&quot;http://en.wikipedia.org/wiki/Instance-based_learning&quot;&gt;Instance-Based Learning&lt;/a&gt;. k-Nearest Neighbors (kNN) appears to be the most popular of these methods and is applicable to a wide variety of problem domains. Another general type of instance-based learning is &lt;a href=&quot;http://en.wikipedia.org/wiki/Analogical_modeling&quot;&gt;Analogical Modeling&lt;/a&gt;, which uses instances as exemplars for comparison with new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You referred to kNN as an application that uses training but that is not correct (the Wikipedia entry you linked is somewhat misleading in that regard). Yes, there are &quot;training examples&quot; (labeled instances) but the classifier doesn't learn/train from these data. Rather, they are only used whenever you actually want to classify a new instance, which is why it is considered a &quot;lazy&quot; learner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the Nearest Template Prediction method you mention effectively is a form of kNN with &lt;code&gt;k=1&lt;/code&gt; and cosine distance as the distance measure.&lt;/p&gt;&#xA;" OwnerUserId="964" LastEditorUserId="964" LastEditDate="2014-07-02T14:46:55.010" LastActivityDate="2014-07-02T14:46:55.010" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="655" PostTypeId="1" CreationDate="2014-07-02T14:40:55.413" Score="10" ViewCount="2256" Body="&lt;p&gt;I would like to use ANN to automate trading currencies, preferably USD/EUR or USD/GBP. I know this is hard and may not be straightforward. I have already read some papers and done some experiments but without much luck. I would like to get advice from EXPERTS to make this work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I did so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I got tick by tick data for the month of july 2013. It has bid/ask/bid volume/ask volume.&lt;/li&gt;&#xA;&lt;li&gt;Extracted all ticks for the time frame 12PM to 14PM for all days.&lt;/li&gt;&#xA;&lt;li&gt;From this data, created a data set where each entry consists of n bid values in sequence. &lt;/li&gt;&#xA;&lt;li&gt;Used that data to train an ANN with n-1 inputs and the output is the forecasted nth bid value.&lt;/li&gt;&#xA;&lt;li&gt;The ANN had n-1 inputs neurons, (n-1)*2 + 1 hidden and 1 output neuron. Input layer had linear TF, hidden had log TF and output had linear TF.&lt;/li&gt;&#xA;&lt;li&gt;Trained the network with back propagation with n-125 first and then 10.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For both n, the MSE did not drop below 0.5 and stayed at that value during full training. Assuming that this could be due to the time series being totally random, I used the R package to find partial autocorrelation on the data set (pacf). This gave non zero values for 2 and 3 lags only. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: What does this mean exactly? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I used hurst exponent to evaluate the randomness. In R, hurst(values) showed values above 0.9.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 2: It is supposed to be nearly random. Should it have a value closer to 0.5?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I repeated the training of the ANN with n=3. The ANN was trained and was able to obtain a pretty low value for MSE. However, the calculated output from this ANN does not differ much from the (n-1)th bid value. It looks like ANN just takes the last bid as the next bid! I tried different network structures (all multilayer perceptions), different training parameters, etc, but results are same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 3: How can I improve the accuracy? Are there any other training methods than backpropagation?&lt;/p&gt;&#xA;" OwnerUserId="1300" LastEditorUserId="322" LastEditDate="2014-07-02T22:18:40.597" LastActivityDate="2014-07-09T00:37:51.167" Title="Foreign exchange market forecasting with neural networks" Tags="|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="656" PostTypeId="2" ParentId="636" CreationDate="2014-07-02T15:52:44.470" Score="2" Body="&lt;p&gt;Some suggestions&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove items appearing too infrequently in the data. That will reduce the dimensionality by several orders of magnitude. If a feature occurs less than say 10 times, it's likely that it's not adding any predictive value, and it may lead to overfitting due to low frequency&lt;/li&gt;&#xA;&lt;li&gt;Try a Linear SVM instead. They handle large dimensional data very well in terms of not overfitting. They also often have the option to assign relative weights to different classes, which may help address your unbalanced problem above. The sklearn svm (which simply wraps some other packages such as libsvm) has this option.&lt;/li&gt;&#xA;&lt;li&gt;Don't use the ID column. Producing a model per user will most probably lead to overfitting. Instead, feed in attributes that describe the user that allows the model to generalize over similar users. You could try fitting a separate model per user, but you need a lot of data per user to do this well.&lt;/li&gt;&#xA;&lt;li&gt;It sounds like you really need to try some feature selection here, to reduce the dimensionality of the problem. But try 1 and 2 first, as they may give you good results sooner (although the end solution may still work better with some good feature selection). Sklearn again has a number of options for feature selection.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2014-07-02T15:52:44.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="657" PostTypeId="1" AcceptedAnswerId="665" CreationDate="2014-07-02T16:41:47.467" Score="4" ViewCount="851" Body="&lt;p&gt;Suppose I want to use CART as classification tree (I want a categorical response). I have the training set, and I split it using observation labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, to build the decision tree (classification tree) how are selected the features to decide which label apply to testing observations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supposing we are working on gene expression matrix, in which each element is a real number, is that done using features that are more distant between classes?&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2014-07-03T17:52:00.963" Title="How are selected the features for a decision tree in CART?" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="658" PostTypeId="1" CreationDate="2014-07-02T20:54:53.767" Score="10" ViewCount="185" Body="&lt;p&gt;I would like to use non-atomic data, as a feature for a prediction. &#xA;Suppose I have a Table with these features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- Column 1: Categorical - House&#xA;- Column 2: Numerical - 23.22&#xA;- Column 3: A Vector - [ 12, 22, 32 ]&#xA;- Column 4: A Tree - [ [ 2323, 2323 ],[2323, 2323] , [ Boolean, Categorical ] ]&#xA;- Column 5: A List [ 122, Boolean ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to predict/classify, for instance, Column 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am making something to automatically respond to questions, any type of question, like &quot;Where was Foo Born?&quot; ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first make a query to a search engine, then I get some text data as a result, then I do all the parsing stuff (tagging, stemming, parsing, splitting ... )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first approach was to make a table, each row with a line of text and a lot of features, like &quot;First Word&quot;, &quot;Tag of First Word&quot;, &quot;Chunks&quot;, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But with this approach I am missing the relationships between the sentences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if there is an algorithm that looks inside the tree structures (or vectors) and makes the relations and extract whatever is relevant for predicting/classifying. I'd prefer to know about a library that does that than an algorithm that I have to implement.&lt;/p&gt;&#xA;" OwnerUserId="1306" LastEditorUserId="1367" LastEditDate="2014-07-16T15:35:31.410" LastActivityDate="2014-07-16T15:35:31.410" Title="Prediction with non-atomic features" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="659" PostTypeId="2" ParentId="600" CreationDate="2014-07-03T08:00:10.823" Score="6" Body="&lt;p&gt;As Steve Kallestad has said, this is a TSP problem, and there are wonderful free solvers to find approximate solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be too much work for what you are looking for, but you may try use one of those solvers in combination with the Google Maps API, to find real walking distances beetwen your coordinates:&#xA;&lt;a href=&quot;https://developers.google.com/maps/documentation/directions/#DirectionsRequests&quot;&gt;https://developers.google.com/maps/documentation/directions/#DirectionsRequests&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I have never used this API, so I don't know how easy or effective it would be)&lt;/p&gt;&#xA;" OwnerUserId="1289" LastActivityDate="2014-07-03T08:00:10.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="660" PostTypeId="1" CreationDate="2014-07-03T10:49:50.993" Score="7" ViewCount="1333" Body="&lt;p&gt;I m new to RHadoop and also to RMR...&#xA;I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.&#xA;Tring to read the file from hdfs&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Error:&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : &#xA;   hadoop streaming failed with error code 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Code :&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sys.setenv(HADOOP_HOME=&quot;/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop&quot;)&#xA;Sys.setenv(HADOOP_CMD=&quot;/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/bin/hadoop&quot;)&#xA;&#xA;Sys.setenv(HADOOP_STREAMING=&quot;/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.7.0.jar&quot;)&#xA;library(rmr2)&#xA;library(rhdfs)&#xA;hdfs.init()&#xA;day_file = hdfs.file(&quot;/hdfs/bikes_LR/day.csv&quot;,&quot;r&quot;)&#xA;day_read = hdfs.read(day_file)&#xA;c = rawToChar(day_read)&#xA;&#xA;XtX =&#xA;  values(from.dfs(&#xA;    mapreduce(&#xA;      input = &quot;/hdfs/bikes_LR/day.csv&quot;,&#xA;      map=&#xA;        function(.,Xi){&#xA;         yi =c[Xi[,1],]&#xA;         Xi = Xi[,-1]&#xA;         keyval(1,list(t(Xi)%*%Xi))&#xA;       },&#xA;  reduce = function(k,v )&#xA;  {&#xA;    vals =as.numeric(v)&#xA;    keyval(k,sum(vals))&#xA;  } ,&#xA;  combine = TRUE)))[[1]]&#xA;&#xA;XtY =&#xA; values(from.dfs(&#xA;    mapreduce(&#xA;     input = &quot;/hdfs/bikes_LR/day.csv&quot;,&#xA;     map=&#xA;       function(.,Xi){&#xA;         yi =c[Xi[,1],]&#xA;         Xi = Xi[,-1]&#xA;        keyval(1,list(t(Xi)%*%yi))&#xA;       },&#xA;     reduce = TRUE ,&#xA;     combine = TRUE)))[[1]]&#xA;solve(XtX,XtY)&#xA;&#xA;&#xA;&#xA;Input:&#xA;------------&#xA;&#xA;instant,dteday,season,yr,mnth,holiday,weekday,workingday,weathersit,temp,atemp,hum,windspeed,casual,registered,cnt&#xA;1,2011-01-01,1,0,1,0,6,0,2,0.344167,0.363625,0.805833,0.160446,331,654,985&#xA;2,2011-01-02,1,0,1,0,0,0,2,0.363478,0.353739,0.696087,0.248539,131,670,801&#xA;3,2011-01-03,1,0,1,0,1,1,1,0.196364,0.189405,0.437273,0.248309,120,1229,1349&#xA;4,2011-01-04,1,0,1,0,2,1,1,0.2,0.212122,0.590435,0.160296,108,1454,1562&#xA;5,2011-01-05,1,0,1,0,3,1,1,0.226957,0.22927,0.436957,0.1869,82,1518,1600&#xA;6,2011-01-06,1,0,1,0,4,1,1,0.204348,0.233209,0.518261,0.0895652,88,1518,1606&#xA;7,2011-01-07,1,0,1,0,5,1,2,0.196522,0.208839,0.498696,0.168726,148,1362,1510&#xA;8,2011-01-08,1,0,1,0,6,0,2,0.165,0.162254,0.535833,0.266804,68,891,959&#xA;9,2011-01-09,1,0,1,0,0,0,1,0.138333,0.116175,0.434167,0.36195,54,768,822&#xA;10,2011-01-10,1,0,1,0,1,1,1,0.150833,0.150888,0.482917,0.223267,41,1280,1321&#xA;&#xA;&#xA;&#xA; Please Suggest me any mistakes.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1235" LastActivityDate="2014-12-31T13:08:04.937" Title="Linear Regression in R Mapreduce(RHadoop)" Tags="|machine-learning|r|apache-hadoop|map-reduce|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="661" PostTypeId="1" CreationDate="2014-07-03T14:18:14.387" Score="2" ViewCount="2452" Body="&lt;p&gt;I download and installed CDH 5 package succesfully on a single linux node in pseudo-distributed Mode on my CentOS 6.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting Hadoop and verifying it is Working Properly as in &lt;a href=&quot;http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_mrv1_pseudo.html&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I succesfully finished the following steps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 1: Format the NameNode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 2: Start HDFS&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 3: Create the /tmp Directory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 4: Create the MapReduce system directories:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 5: Verify the HDFS File Structure&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 6: Start MapReduce&lt;/p&gt;&#xA;&#xA;&lt;p&gt;while following command in step 7 I get the following error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step 7: Create User Directories&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;$ sudo -u hdfs hadoop fs -mkdir -p /user/hadoopuser&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;mkdir: '/user/hadoopuser': No such file or directory&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where hadoopuser is my linux login username)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I create the directory manually as /user/hadoopuser in the filesystem, it is not accepting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to success the step 7:?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide the sloution to procced the remaining installation.&lt;/p&gt;&#xA;" OwnerUserId="1314" LastEditorUserId="434" LastEditDate="2014-07-03T17:45:59.843" LastActivityDate="2015-07-22T06:22:54.263" Title="Cannot make user directory on a new CDH5 installation (Hadoop)" Tags="|bigdata|apache-hadoop|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="662" PostTypeId="1" CreationDate="2014-07-03T16:11:22.637" Score="31" ViewCount="29975" Body="&lt;p&gt;Note that I am doing everything in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem goes as follow: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: based on the text on their CVs, I want to classify them into different job sectors. I am particular in those cases, in which the candidates do not have any experience / is a student, and I want to make a prediction to classify which job sectors this candidate will most likely belongs to after graduation . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: I know machine learning algorithms. However, I have never done NLP before. I came across Latent Dirichlet allocation on the internet. However, I am not sure if this is the best approach to tackle my problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original idea:  &lt;em&gt;make this a supervised learning problem&lt;/em&gt;. &#xA;Suppose we already have large amount of labelled data, meaning that we have correctly labelled the job sectors for a list of candidates. We train the model up using ML algorithms (i.e. nearest neighbor... )and feed in those &lt;em&gt;unlabelled data&lt;/em&gt;, which are candidates that have no work experience / are students, and try to predict which job sector they will belong to. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&#xA;Question 2: Would it be a good idea to create an text file by extracting everything in a resume and print these data out in the text file, so that each resume is associated with a text file,which contains unstructured strings, and then we applied text mining techniques to the text files and make the data become structured or even to create a  frequency matrix of terms used out of the text files ? For example, the text file may look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;I deployed ML algorithm in this project and... Skills: Java, Python, c++ ...&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I meant by 'unstructured', i.e. collapsing everything into a single line string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this approach wrong ? Please correct me if you think my approach is wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 3: The tricky part is: how to &lt;strong&gt;identify and extract the keywords&lt;/strong&gt; ? Using the &lt;code&gt;tm&lt;/code&gt; package in R ? what algorithm is the &lt;code&gt;tm&lt;/code&gt;   package based on ?  Should I use NLP algorithms ? If yes, what algorithms should I look at ? Please point me to some good resources to look at as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas would be great.&lt;/p&gt;&#xA;" OwnerUserId="1315" LastEditorUserId="1352" LastEditDate="2014-07-09T00:19:01.640" LastActivityDate="2019-04-02T02:47:17.803" Title="What algorithms should I use to perform job classification based on resume data?" Tags="|machine-learning|classification|nlp|text-mining|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="663" PostTypeId="2" ParentId="662" CreationDate="2014-07-03T17:06:13.143" Score="14" Body="&lt;p&gt;Check out &lt;a href=&quot;http://www.rdatamining.com/examples/text-mining&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, they will take you through loading unstructured text to creating a wordcloud.  You can adapt this strategy and instead of creating a wordcloud, you can create a frequency matrix of terms used.  The idea is to take the unstructured text and structure it somehow.  You change everything to lowercase (or uppercase), remove stop words, and find frequent terms for each job function, via Document Term Matrices.  You also have the option of stemming the words.  If you stem words you will be able to detect different forms of words as the same word.  For example, 'programmed' and 'programming' could be stemmed to 'program'.  You can possibly add the occurrence of these frequent terms as a weighted feature in your ML model training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also adapt this to frequent phrases, finding common groups of 2-3 words for each job function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Load libraries and build the example data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tm)&#xA;library(SnowballC)&#xA;&#xA;doc1 = &quot;I am highly skilled in Java Programming.  I have spent 5 years developing bug-tracking systems and creating data managing system applications in C.&quot;&#xA;job1 = &quot;Software Engineer&quot;&#xA;doc2 = &quot;Tested new software releases for major program enhancements.  Designed and executed test procedures and worked with relational databases.  I helped organize and lead meetings and work independently and in a group setting.&quot;&#xA;job2 = &quot;Quality Assurance&quot;&#xA;doc3 = &quot;Developed large and complex web applications for client service center. Lead projects for upcoming releases and interact with consumers.  Perform database design and debugging of current releases.&quot;&#xA;job3 = &quot;Software Engineer&quot;&#xA;jobInfo = data.frame(&quot;text&quot; = c(doc1,doc2,doc3),&#xA;                     &quot;job&quot; = c(job1,job2,job3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) Now we do some text structuring. I am positive there are quicker/shorter ways to do the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Convert to lowercase&#xA;jobInfo&lt;span class=&quot;math-container&quot;&gt;$text = sapply(jobInfo$&lt;/span&gt;text,tolower)&#xA;&#xA;# Remove Punctuation&#xA;jobInfo&lt;span class=&quot;math-container&quot;&gt;$text = sapply(jobInfo$&lt;/span&gt;text,function(x) gsub(&quot;[[:punct:]]&quot;,&quot; &quot;,x))&#xA;&#xA;# Remove extra white space&#xA;jobInfo&lt;span class=&quot;math-container&quot;&gt;$text = sapply(jobInfo$&lt;/span&gt;text,function(x) gsub(&quot;[ ]+&quot;,&quot; &quot;,x))&#xA;&#xA;# Remove stop words&#xA;jobInfo&lt;span class=&quot;math-container&quot;&gt;$text = sapply(jobInfo$&lt;/span&gt;text, function(x){&#xA;  paste(setdiff(strsplit(x,&quot; &quot;)[[1]],stopwords()),collapse=&quot; &quot;)&#xA;})&#xA;&#xA;# Stem words (Also try without stemming?)&#xA;jobInfo&lt;span class=&quot;math-container&quot;&gt;$text = sapply(jobInfo$&lt;/span&gt;text, function(x)  {&#xA;  paste(setdiff(wordStem(strsplit(x,&quot; &quot;)[[1]]),&quot;&quot;),collapse=&quot; &quot;)&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;3) Make a corpus source and document term matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Create Corpus Source&#xA;jobCorpus = Corpus(VectorSource(jobInfo$text))&#xA;&#xA;# Create Document Term Matrix&#xA;jobDTM = DocumentTermMatrix(jobCorpus)&#xA;&#xA;# Create Term Frequency Matrix&#xA;jobFreq = as.matrix(jobDTM)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we have the frequency matrix, jobFreq, that is a (3 by x) matrix, 3 entries and X number of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where you go from here is up to you.  You can keep only specific (more common) words and use them as features in your model.  Another way is to keep it simple and have a percentage of words used in each job description, say &quot;java&quot; would have 80% occurrence in 'software engineer' and only 50% occurrence in 'quality assurance'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it's time to go look up why 'assurance' has 1 'r' and 'occurrence' has 2 'r's.&lt;/p&gt;&#xA;" OwnerUserId="375" LastEditorUserId="29575" LastEditDate="2019-04-02T02:47:17.803" LastActivityDate="2019-04-02T02:47:17.803" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="664" PostTypeId="1" CreationDate="2014-07-03T17:11:01.770" Score="11" ViewCount="695" Body="&lt;p&gt;R base function &lt;code&gt;glm()&lt;/code&gt; uses Fishers Scoring for MLE, while the &lt;code&gt;glmnet&lt;/code&gt; appears to use the coordinate descent method to solve the same equation. Coordinate descent is more time-efficient than Fisher Scoring, as Fisher Scoring calculates the second order derivative matrix, in addition to some other matrix operations. which makes expensive to perform, while coordinate descent can do the same task in O(np) time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would R base function use Fisher Scoring? Does this method have an advantage over other optimization methods? How does coordinate descent and Fisher Scoring compare? I am relatively new to do this field so any help or resource will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="1319" LastEditorUserId="13413" LastEditDate="2015-10-26T08:28:38.520" LastActivityDate="2016-02-23T17:34:08.210" Title="Fisher Scoring v/s Coordinate Descent for MLE in R" Tags="|machine-learning|r|algorithms|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="665" PostTypeId="2" ParentId="657" CreationDate="2014-07-03T17:52:00.963" Score="4" Body="&lt;p&gt;At each split point, CART will choose the feature which &quot;best&quot; splits the observations. What qualifies as best varies, but generally the split is done so that the subsequent nodes are more homogenous/pure with respect to the target. There are different ways of measuring homogeneity, for example Gini, Entropy, Chi-square. If you are using software, it may allow you to choose the measure of homogenity that the tree algorithm will use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distance is not a factor with trees - what matters is whether the value is greater than or less than the split point, not the distance from the split point.&lt;/p&gt;&#xA;" OwnerUserId="1111" LastActivityDate="2014-07-03T17:52:00.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="666" PostTypeId="2" ParentId="587" CreationDate="2014-07-03T18:36:36.050" Score="3" Body="&lt;p&gt;As far as gathering data goes, you can check out &lt;strong&gt;Quandl&lt;/strong&gt; (there's a tutorial on using it with &lt;strong&gt;R&lt;/strong&gt; on &lt;a href=&quot;https://www.datacamp.com/courses/how-to-work-with-quandl-in-r&quot; rel=&quot;nofollow&quot;&gt;DataCamp&lt;/a&gt; if you're interested).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, Aswath Damodaran's &lt;a href=&quot;http://people.stern.nyu.edu/adamodar/New_Home_Page/data.html&quot; rel=&quot;nofollow&quot;&gt;site&lt;/a&gt; contains a lot of helpful datasets. Though they aren't updated that frequently, they may still be useful, especially as a benchmark for comparing your own output (from the scripts you will inevitably need to write to calculate the necessary metrics).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, again, &lt;strong&gt;Quant SE&lt;/strong&gt; is probably a better place to be looking...&lt;/p&gt;&#xA;" OwnerUserId="786" LastActivityDate="2014-07-03T18:36:36.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="667" PostTypeId="2" ParentId="662" CreationDate="2014-07-03T20:47:20.147" Score="7" Body="&lt;p&gt;This is a tricky problem. There are many ways to handle it. I guess, resumes can be treated as semi-structured documents. Sometimes, it's beneficial to have some minimal structure in the documents. I believe, in resumes you would see some tabular data. You might want to treat these as attribute value pairs. For example, you would get a list of terms for the attribute &quot;Skill set&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key idea is to manually configure a list of key phrases such as &quot;skill&quot;, &quot;education&quot;, &quot;publication&quot; etc. The next step is to extract terms which pertain to these key phrases either by exploiting the structure in some way (such as tables) or by utilizing the proximity of terms around these key phrases, e.g. the fact that the word &quot;Java&quot; is in close proximity to the term &quot;skill&quot; might indicate that the person is skilled in Java.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you extract these information, the next step could be to build up a feature vector for each of these key phrases. You can then represent a document as a vector with different fields (one each for a key phrase). For example, consider the following two resumes represented with two fields, namely &lt;em&gt;project&lt;/em&gt; and &lt;em&gt;education&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doc1: {project: (java, 3) (c, 4)}, {education: (computer, 2), (physics, 1)}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doc2: {project: (java, 3) (python, 2)}, {education: (maths, 3), (computer, 2)}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example, I show a term with the frequency. Of course, while extracting the terms you need to stem and remove stop-words. It is clear from the examples that the person whose resume is Doc1 is more skilled in C than that of D2. Implementation wise, it's very easy to represent documents as field vectors in Lucene.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the next step is to retrieve a ranked list of resumes given a job specification. In fact, that's fairly straight forward if you represent queries (job specs) as field vectors as well. You just need to retrieve a ranked list of candidates (resumes) using Lucene from a collection of indexed resumes.&lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-07-03T20:47:20.147" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="668" PostTypeId="2" ParentId="265" CreationDate="2014-07-03T20:51:59.510" Score="5" Body="&lt;p&gt;I've done some research in this area. I've found first order Markov chains work well for predicting within game scoring dynamics across a variety of sports.&lt;/p&gt;&#xA;&lt;p&gt;You can read in more detail &lt;a href=&quot;http://www.epjdatascience.com/content/3/1/4&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1329" LastEditorUserId="98307" LastEditDate="2020-08-20T18:25:18.637" LastActivityDate="2020-08-20T18:25:18.637" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="669" PostTypeId="2" ParentId="655" CreationDate="2014-07-03T23:00:42.350" Score="6" Body="&lt;p&gt;The results you're seeing aren't a byproduct of your training product, but rather that &lt;code&gt;neural nets&lt;/code&gt; are not a great choice for this task. &lt;code&gt;Neural nets&lt;/code&gt; are effectively a means to create a high order non-linear function by composing a number of simpler functions. This is often a really good thing, because it allows neural nets to fit very complex patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in a stock exchange any complex pattern, when traded upon will quickly decay. Detecting a complicated pattern will generally not generate useful results, because it is typically complex patterns in the short term. Additionally, depending on the metric you choose, there are a number of ways of performing well that actually won't pay off in investing (such as just predicting the last value in your example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition the stock market is startlingly chaotic which can result in a &lt;code&gt;neural net&lt;/code&gt; overfitting. This means that the patterns it learns will generalize poorly. Something along the lines of just seeing a stock decrease over a day and uniformly deciding that the stock will always decrease just because it was seen on a relatively short term. Instead techniques like &lt;code&gt;ridge&lt;/code&gt; and &lt;code&gt;robust regression&lt;/code&gt;, which will identify more general, less complex patterns, do better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The winner of a similar Kaggle competition used &lt;code&gt;robust regression&lt;/code&gt; for this very reason. You are likely to see better results if you switch to a shallow learning model that will find functions of a lower polynomial order, over the deep complex functions of a neural net.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-03T23:00:42.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="670" PostTypeId="1" AcceptedAnswerId="672" CreationDate="2014-07-04T02:31:42.080" Score="3" ViewCount="122" Body="&lt;p&gt;I was building a model that predicts user churn for a website, where I have data on all users, both past and present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can build a model that only uses those users that have left, but then I'm leaving 2/3 of the total user population unused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way to incorporate data from these users into a model from a conceptual standpoint?&lt;/p&gt;&#xA;" OwnerUserId="1334" LastEditorUserId="1334" LastEditDate="2014-07-04T02:38:12.523" LastActivityDate="2016-08-11T09:33:39.037" Title="Dealing with events that have not yet happened when building a model" Tags="|data-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="671" PostTypeId="1" AcceptedAnswerId="729" CreationDate="2014-07-04T05:12:44.707" Score="8" ViewCount="220" Body="&lt;p&gt;I have a linearly increasing time series dataset of a sensor, with value ranges between 50 and 150. I've implemented a &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_linear_regression&quot; rel=&quot;noreferrer&quot;&gt;Simple Linear Regression&lt;/a&gt; algorithm to fit a regression line on such data, and I'm predicting the date when the series would reach 120.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All works fine when the series move upwards. But, there are cases in which the sensor reaches around 110 or 115, and it is reset; in such cases the values would start over again at, say, 50 or 60.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where I start facing issues with the regression line, as it starts moving downwards, and it starts predicting old date. I think I should be considering only the subset of data from where it was previously reset. However, I'm trying to understand if there are any algorithms available that consider this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new to data science, would appreciate any pointers to move further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: nfmcclure's suggestions applied&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before applying the suggestions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZsyyQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the snapshot of what I've got after splitting the dataset where the reset occurs, and the slope of two set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/OEQCw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;finding the mean of the two slopes and drawing the line from the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/i2qv5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this OK?&lt;/p&gt;&#xA;" OwnerUserId="870" LastEditorUserId="870" LastEditDate="2014-08-27T07:15:52.587" LastActivityDate="2019-01-01T19:50:02.190" Title="Linearly increasing data with manual reset" Tags="|machine-learning|statistics|time-series|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="672" PostTypeId="2" ParentId="670" CreationDate="2014-07-04T06:46:19.180" Score="5" Body="&lt;p&gt;This setting is common in reliability, health care, and mortality. The statistical analysis method is called &lt;a href=&quot;http://en.wikipedia.org/wiki/Survival_analysis&quot;&gt;Survival Analysis&lt;/a&gt;. All users are coded according to their start date (or week or month).  You use the empirical data to estimate the survival function, which is the probability that the time of defection is later than some specified time &lt;strong&gt;&lt;em&gt;t&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your baseline model will estimate survival function for all users.  Then you can do more sophisticated modeling to estimate what factors or behaviors might predict defection (churn), given your baseline survival function.  Basically, any model that is predictive will yield a survival probability that is significantly lower than the baseline.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There's another approach which involves attempting to identify precursor events patterns or user behavior pattern that foreshadow defection. Any given event/behavior pattern might occur for users that defect, or for users that stay. For this analysis, you may need to censor your data to only include users that have been members for some minimum period of time. The minimum time period can be estimated using your estimate of survival function, or even simple histogram analysis of the distribution of membership period for users who have defected. &lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-07-04T06:46:19.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="675" PostTypeId="2" ParentId="662" CreationDate="2014-07-04T22:46:32.130" Score="11" Body="&lt;p&gt;Just extract &lt;strong&gt;keywords&lt;/strong&gt; and train a &lt;strong&gt;classifier&lt;/strong&gt; on them. That's all, really. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the text in CVs is not actually related to skills. E.g. consider sentence &quot;I'm experienced and highly efficient in Java&quot;. Here only 1 out of 7 words is a skill name, the rest is just a noise that's going to put your classification accuracy down. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of CVs are not really structured. Or structured too freely. Or use unusual names for sections. Or file formats that don't preserve structure when translated to text. I have experience extracting dates, times, names, addresses and even people intents from unstructured text, but not a skill (or university or anything) list, not even closely. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So just tokenize (and possibly &lt;a href=&quot;https://stackoverflow.com/questions/16069406/text-mining-with-the-tm-package-word-stemming&quot;&gt;stem&lt;/a&gt;) your CVs, select only words from predefined list (you can use LinkedIn or something similar to grab this list), create a feature vector and try out a couple of classifiers (say, SVM and Naive Bayes). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: I used a similar approach to classify LinkedIn profiles into more than 50 classes with accuracy &gt; 90%, so I'm pretty sure even naive implementation will work well.)&lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-07-05T08:31:38.717" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="676" PostTypeId="2" ParentId="565" CreationDate="2014-07-05T14:48:10.570" Score="7" Body="&lt;p&gt;In general regression models (any) can behave in an arbitrary way beyond the domain spanned by training samples. In particular, they are free to assume linearity of the modeled function, so if you for instance train a regression model with points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X     Y&#xA;10    0&#xA;20    1&#xA;30    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it is reasonable to build a model &lt;code&gt;f(x) = x/10-1&lt;/code&gt;, which for &lt;code&gt;x&amp;lt;10&lt;/code&gt; returns negative values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same applies &quot;in between&quot; your data points, it is always possible that due to the assumed famility of functions (which can be modeled by particular method) you will get values &quot;out of your training samples&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think about this in another way - &quot;what is so special about negative values?&quot;, why do you find existance of negative values weird (if not provided in training set) while you don't get alarmed by existance of lets say... value 2131.23? Unless developed in such a way, no model will treat negative values &quot;different&quot; than positive ones. This is just a natural element of the real values which can be attained as any other value.&lt;/p&gt;&#xA;" OwnerUserId="1355" LastActivityDate="2014-07-05T14:48:10.570" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="677" PostTypeId="1" AcceptedAnswerId="5834" CreationDate="2014-07-05T15:01:43.940" Score="7" ViewCount="4765" Body="&lt;p&gt;I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was working through the following dataset on &lt;a href=&quot;https://www.kaggle.com/c/data-science-london-scikit-learn/data&quot; rel=&quot;noreferrer&quot;&gt;Kaggle&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's essentially an unheadered table (1000 rows, 40 features) with floating point values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pdfrom sklearn import neighbors&#xA;from sklearn_pandas import DataFrameMapper, cross_val_score&#xA;path_train =&quot;../kaggle/scikitlearn/train.csv&quot;&#xA;path_labels =&quot;../kaggle/scikitlearn/trainLabels.csv&quot;&#xA;path_test = &quot;../kaggle/scikitlearn/test.csv&quot;&#xA;&#xA;train = pd.read_csv(path_train, header=None)&#xA;labels = pd.read_csv(path_labels, header=None)&#xA;test = pd.read_csv(path_test, header=None)&#xA;mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])&#xA;mapper_train&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DataFrameMapper(features=[([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',&#xA;       n_neighbors=3, p=2, weights='uniform'))])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far so good. But then I try the fit&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mapper_train.fit_transform(train, labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------------------------&#xA;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-6-e3897d6db1b5&amp;gt; in &amp;lt;module&amp;gt;()&#xA;----&amp;gt; 1 mapper_train.fit_transform(train, labels)&#xA;&#xA;//anaconda/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y,     **fit_params)&#xA;    409         else:&#xA;    410             # fit method of arity 2 (supervised transformation)&#xA;--&amp;gt; 411             return self.fit(X, y, **fit_params).transform(X)&#xA;    412 &#xA;    413 &#xA;&#xA;//anaconda/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)&#xA;    116         for columns, transformer in self.features:&#xA;    117             if transformer is not None:&#xA;--&amp;gt; 118                 transformer.fit(self._get_col_subset(X, columns))&#xA;    119         return self&#xA;    120 &#xA;&#xA;TypeError: fit() takes exactly 3 arguments (2 given)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong? While the data in this case is all the same, I'm planning to work up a workflow for mixtures categorical, nominal and floating point features and sklearn_pandas seemed to be a logical fit.&lt;/p&gt;&#xA;" OwnerUserId="974" LastActivityDate="2015-05-27T03:16:56.403" Title="Struggling to integrate sklearn and pandas in simple Kaggle task" Tags="|python|pandas|scikit-learn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="678" PostTypeId="1" AcceptedAnswerId="689" CreationDate="2014-07-05T16:10:21.580" Score="40" ViewCount="11120" Body="&lt;p&gt;When I say &quot;document&quot;, I have in mind web pages like Wikipedia articles and news stories.  I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics, with stronger preference for the latter.&lt;/p&gt;&#xA;" OwnerUserId="1097" LastEditorUserId="13727" LastEditDate="2015-11-06T09:00:24.573" LastActivityDate="2020-08-06T13:01:07.800" Title="What are some standard ways of computing the distance between documents?" Tags="|machine-learning|data-mining|nlp|text-mining|similarity|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="679" PostTypeId="1" AcceptedAnswerId="699" CreationDate="2014-07-05T16:20:17.963" Score="8" ViewCount="200" Body="&lt;p&gt;I made a similar question asking about distance between &quot;documents&quot; (Wikipedia articles, news stories, etc.).  I made this a separate question because search queries are considerably smaller than documents and are considerably noisier.  I hence don't know (and doubt) if the same distance metrics would be used here.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either vanilla lexical distance metrics or state-of-the-art semantic distance metrics are preferred, with stronger preference for the latter.&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2014-08-07T07:07:03.713" Title="What are some standard ways of computing the distance between individual search queries?" Tags="|machine-learning|nlp|search|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="680" PostTypeId="2" ParentId="678" CreationDate="2014-07-06T00:01:41.327" Score="6" Body="&lt;p&gt;There's a number of semantic distance measures, each with its pros and cons. Here are just a few of them: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cosine_similarity&quot;&gt;&lt;strong&gt;cosine distance&lt;/strong&gt;&lt;/a&gt;, inner product between document feature vectors;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot;&gt;&lt;strong&gt;LSA&lt;/strong&gt;&lt;/a&gt;, another vector-based model, but utilizing SVD for de-noising original term-document matrix;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/WordNet&quot;&gt;&lt;strong&gt;WordNet&lt;/strong&gt;&lt;/a&gt;-based, human verified, though hardly extensible.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Start with a simplest approach and then move further based on issues for your specific case. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-06T00:01:41.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="682" PostTypeId="2" ParentId="265" CreationDate="2014-07-06T14:01:57.467" Score="6" Body="&lt;p&gt;Michael Maouboussin, in his book, &quot;The Success Equation,&quot; looks at differentiating luck from skill in various endeavors, including sports.  He actually ranks sports by the amount of luck that contributes to performance in the different sports (p. 23) and about 2/3 of performance in football is attributable to skill.  By contrast, I used MM's technique to analyze performance in Formula 1 racing, and found that 60% is attributable to skill (less than I was expecting.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, it seems this kind of analysis would imply that a sufficiently detailed and crafted feature set would allow ML algorithms to predict performance of NFL teams, perhaps even to the play level, with the caveat that significant variance will still exist because of the influence of luck in the game.&lt;/p&gt;&#xA;" OwnerUserId="1360" LastActivityDate="2014-07-06T14:01:57.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="683" PostTypeId="1" CreationDate="2014-07-07T09:34:36.950" Score="11" ViewCount="2866" Body="&lt;p&gt;I have 2 datasets, one with positive instances of what I would like to detect, and one with unlabeled instances. What methods can I use ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, suppose we want to understand detect spam email based on a few structured email characteristics. We have one dataset of 10000 spam emails, and one dataset of 100000 emails for which we don't know whether they are spam or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we tackle this problem (without labeling manually any of the unlabeled data) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can we do if we have additional information about the proportion of spam in the unlabeled data (i.e. what if we estimate that between 20-40% of the 100000 unlabeled emails are spam) ?&lt;/p&gt;&#xA;" OwnerUserId="906" LastEditorUserId="11097" LastEditDate="2015-12-07T10:50:13.513" LastActivityDate="2015-12-07T10:50:13.513" Title="Build a binary classifier with only positive and unlabeled data" Tags="|classification|semi-supervised-learning|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="684" PostTypeId="2" ParentId="677" CreationDate="2014-07-07T10:25:55.323" Score="7" Body="&lt;p&gt;I have never used &lt;code&gt;sklearn_pandas&lt;/code&gt;, but from reading their source code, it looks like this is a bug on their side. If you look for &lt;a href=&quot;https://github.com/paulgb/sklearn-pandas/blob/master/sklearn_pandas/__init__.py&quot;&gt;the function that is throwing the exception&lt;/a&gt;, you can notice that they are discarding the &lt;code&gt;y&lt;/code&gt; argument (it does not even survive until the docstring), and the inner &lt;code&gt;fit&lt;/code&gt; function expects one argument more, which is probably &lt;code&gt;y&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fit(self, X, y=None):&#xA;    '''&#xA;    Fit a transformation from the pipeline&#xA;&#xA;    X       the data to fit&#xA;    '''&#xA;    for columns, transformer in self.features:&#xA;        if transformer is not None:&#xA;            transformer.fit(self._get_col_subset(X, columns))&#xA;    return self&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would recommend that you open an issue in &lt;a href=&quot;https://github.com/paulgb/sklearn-pandas/issues&quot;&gt;their bug tracker&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can test this if you run your code from IPython. To summarize, if you use the &lt;code&gt;%pdb on&lt;/code&gt; magic right before you run the problematic call, the exception is captured by the Python debugger, so you can play around a bit and see that calling the &lt;code&gt;fit&lt;/code&gt; function with the label values &lt;code&gt;y[0]&lt;/code&gt; does work  -- see the last line with the &lt;code&gt;pdb&amp;gt;&lt;/code&gt; prompt. (The CSV files are downloaded from Kaggle, except for the largest one which is just a part of the real file).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [1]: import pandas as pd&#xA;&#xA;In [2]: from sklearn import neighbors&#xA;&#xA;In [3]: from sklearn_pandas import DataFrameMapper, cross_val_score&#xA;&#xA;In [4]: path_train =&quot;train.csv&quot;&#xA;&#xA;In [5]: path_labels =&quot;trainLabels.csv&quot;&#xA;&#xA;In [6]: path_test = &quot;test.csv&quot;&#xA;&#xA;In [7]: train = pd.read_csv(path_train, header=None)&#xA;&#xA;In [8]: labels = pd.read_csv(path_labels, header=None)&#xA;&#xA;In [9]: test = pd.read_csv(path_test, header=None)&#xA;&#xA;In [10]: mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])&#xA;&#xA;In [13]: %pdb on&#xA;&#xA;In [14]: mapper_train.fit_transform(train, labels)&#xA;---------------------------------------------------------------------------&#xA;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-14-e3897d6db1b5&amp;gt; in &amp;lt;module&amp;gt;()&#xA;----&amp;gt; 1 mapper_train.fit_transform(train, labels)&#xA;&#xA;/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y, **fit_params)&#xA;    409         else:&#xA;    410             # fit method of arity 2 (supervised transformation)&#xA;--&amp;gt; 411             return self.fit(X, y, **fit_params).transform(X)&#xA;    412&#xA;    413&#xA;&#xA;/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)&#xA;    116         for columns, transformer in self.features:&#xA;    117             if transformer is not None:&#xA;--&amp;gt; 118                 transformer.fit(self._get_col_subset(X, columns))&#xA;    119         return self&#xA;    120&#xA;&#xA;TypeError: fit() takes exactly 3 arguments (2 given)&#xA;&amp;gt; /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.py(118)fit()&#xA;    117             if transformer is not None:&#xA;--&amp;gt; 118                 transformer.fit(self._get_col_subset(X, columns))&#xA;    119         return self&#xA;&#xA;ipdb&amp;gt; l&#xA;    113&#xA;    114         X       the data to fit&#xA;    115         '''&#xA;    116         for columns, transformer in self.features:&#xA;    117             if transformer is not None:&#xA;--&amp;gt; 118                 transformer.fit(self._get_col_subset(X, columns))&#xA;    119         return self&#xA;    120&#xA;    121&#xA;    122     def transform(self, X):&#xA;    123         '''&#xA;ipdb&amp;gt; transformer.fit(self._get_col_subset(X, columns), y[0])&#xA;KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',&#xA;           n_neighbors=3, p=2, weights='uniform')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="1367" LastEditDate="2014-07-07T13:42:31.827" LastActivityDate="2014-07-07T13:42:31.827" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="685" PostTypeId="1" CreationDate="2014-07-07T11:43:48.430" Score="5" ViewCount="4578" Body="&lt;p&gt;I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using logistic regression with R's &lt;code&gt;glmnet&lt;/code&gt; package and alpha = 0 for ridge regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using ridge regression actually since lasso deleted all my variables and gave very low area under curve (0.52) but with ridge there isn't much of a difference (0.61).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dependent variable/output is probability of click, based on if there is a click or not in historical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The independent variables are state, city, device, user age, user gender, IP carrier, keyword, mobile manufacturer, ad template, browser version, browser family, OS version and OS family.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of these, for prediction I'm using state, device, user age, user gender, IP carrier, browser version, browser family, OS version and OS family; I am not using keyword or template since we want to reject a user request before deep diving in our system and selecting a keyword or template. I am not using city because they are too many or mobile manufacturer because they are too few.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is that okay or should I be using the rejected variables?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start, I create a sparse matrix from my variables which are mapped against the column of clicks that have yes or no values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After training the model, I save the coefficients and intercept. These are used for new incoming requests using the formula for logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/AZBRq.png&quot; alt=&quot;1 / (1+e^-1*sum(a+k(ith)*x(ith)))&quot;&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;a&lt;/code&gt; is intercept, &lt;code&gt;k&lt;/code&gt; is the &lt;code&gt;i&lt;/code&gt;th coefficient and &lt;code&gt;x&lt;/code&gt; is the &lt;code&gt;i&lt;/code&gt;th variable value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is my approach correct so far?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple GLM in R (that is where there is no regularized regression, right?) gave me 0.56 AUC. With regularization I get 0.61 but there is no distinct threshold where we could say that above 0.xx its mostly ones and below it most zeros are covered; actually, the max probability that a click didn't happen is almost always greater than the max probability that a click happened.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So basically what should I do?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read how stochastic gradient descent is an effective technique in logit so how do I implement stochastic gradient descent in R? If it's not straightforward, is there a way to implement this system in Python? Is SGD implemented after generating a regularized logistic regression model or is it a different process altogether?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also there is an algorithm called follow the regularized leader (FTRL) that is used in click-through rate prediction. Is there a sample code and use of FTRL that I could go through?&lt;/p&gt;&#xA;" OwnerUserId="1273" LastEditorUserId="322" LastEditDate="2014-07-07T20:02:08.777" LastActivityDate="2021-05-15T21:43:02.720" Title="Stochastic gradient descent in logistic regression" Tags="|machine-learning|data-mining|r|logistic-regression|gradient-descent|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="686" PostTypeId="2" ParentId="678" CreationDate="2014-07-07T13:23:28.220" Score="3" Body="&lt;p&gt;State of the art appears to be &amp;quot;paragraph vectors&amp;quot; introduced in a recent &lt;a href=&quot;http://cs.stanford.edu/%7Equocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. Cosine/Euclidean distance between paragraph vectors would likely work better than any other approach. This probably isn't feasible yet due to lack of open source implementations.&lt;/p&gt;&#xA;&lt;p&gt;Next best thing is cosine distance between LSA vectors or cosine distance between raw BOW vectors. Sometimes it works better to choose different weighting schemes, like TF-IDF.&lt;/p&gt;&#xA;" OwnerUserId="574" LastEditorUserId="98307" LastEditDate="2020-08-06T13:01:07.800" LastActivityDate="2020-08-06T13:01:07.800" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="687" PostTypeId="2" ParentId="683" CreationDate="2014-07-07T14:15:51.117" Score="7" Body="&lt;p&gt;My suggestion would be to attempt to build some kind of clustering on your unlabeled data that somewhat approximates a labelled dataset. The rationale is more or less as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You have some feature vector for representing your documents&lt;/li&gt;&#xA;&lt;li&gt;Based on that feature vector, you can come up with a number of different clusterings, with either fuzzy, rough, or class-based clustering methods&lt;/li&gt;&#xA;&lt;li&gt;Knowing what a positive example looks like, you can quickly evaluate the overall similarity of a cluster to your positive cluster&lt;/li&gt;&#xA;&lt;li&gt;Knowing that there should really only be two clusters, you can adjust the hyperparameters on your clustering method so that the above two metrics are closer and closer to satisfaction&lt;/li&gt;&#xA;&lt;li&gt;With the two clusters, you have what is likely a close approximation of a labelled dataset, which you can then use as a silver-standard corpus of sorts to actually train your model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope that makes sense, if you're specifically looking for clustering algorithms, a few that I personally enjoy that might be good in this scenario are &lt;a href=&quot;http://en.wikipedia.org/wiki/FLAME_clustering&quot;&gt;FLAME&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot;&gt;tsne&lt;/a&gt;. Alternately, looking at the spectacular &lt;a href=&quot;http://radimrehurek.com/gensim/&quot;&gt;gensim&lt;/a&gt; library in python will get you a long way toward the clustering you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps and makes sense, leave a comment if you've got any questions.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-07T14:15:51.117" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="688" PostTypeId="2" ParentId="600" CreationDate="2014-07-07T14:41:59.523" Score="2" Body="&lt;p&gt;As @SpacedMan has noted &lt;a href=&quot;https://datascience.stackexchange.com/questions/600/how-do-you-create-an-optimized-walk-list-given-longitude-and-latitude-coordinate#comment1670_600&quot;&gt;in a comment&lt;/a&gt;, the street layout will have a massive influence on the optimization of the walk list.  You have included only &quot;latitude and longitude&quot; in your question's title; but solving that problem does not lead to a &quot;walk list&quot;, but to a &quot;as-the-crow-flies list&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at your street layout as a graph, with edge weights describing distances, and trying to find the shortest traversal between all required addresses, will lead you to think of your problem as a &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Shortest_path_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Shortest path problem&lt;/a&gt;&quot;. &lt;a href=&quot;http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dijkstra's algorithm&lt;/a&gt; is the best known solution (there are others); in its naive implementation it converges in &lt;em&gt;O(n&lt;sup&gt;2&lt;/sup&gt;)&lt;/em&gt;, which may be acceptable if your lists of addresses are moderate in size.  Otherwise, look for optimized versions in the above links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for libraries and resources to start tackling the problem, since you do not specify languages or platforms, let me point to the &lt;a href=&quot;http://wiki.openstreetmap.org/wiki/Routing#Developers&quot; rel=&quot;nofollow noreferrer&quot;&gt;compilation of routing solvers in the Open Street Maps wiki&lt;/a&gt; and in general &lt;a href=&quot;http://wiki.openstreetmap.org/wiki/Frameworks&quot; rel=&quot;nofollow noreferrer&quot;&gt;their frameworks and libraries page&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-07-07T14:41:59.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="689" PostTypeId="2" ParentId="678" CreationDate="2014-07-07T15:36:40.960" Score="53" Body="&lt;p&gt;There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of them have been mentioned by ffriend, and the paragraph vectors by user1133029 is a really solid one, but I just figured I would go into some more depth about plusses and minuses of different approaches.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Cosine_similarity&quot;&gt;Cosine Distance&lt;/a&gt; - Tried a true, cosine distance is probably the most common distance metric used generically across multiple domains. With that said, there's very little information in cosine distance that can actually be mapped back to anything semantic, which seems to be non-ideal for this situation.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Levenshtein_distance&quot;&gt;Levenshtein Distance&lt;/a&gt; - Also known as &lt;code&gt;edit distance&lt;/code&gt;, this is usually just used on the individual token level (words, bigrams, etc...). In general I wouldn't recommend this metric as it not only discards any semantic information, but also tends to treat very different word alterations very similarly, but it is an extremely common metric for this kind of thing&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot;&gt;LSA&lt;/a&gt; - Is a part of a large arsenal of techniques when it comes to evaluating document similarity called &lt;code&gt;topic modeling&lt;/code&gt;. LSA has gone out of fashion pretty recently, and in my experience, it's not quite the strongest topic modeling approach, but it is relatively straightforward to implement and has a few open source implementations&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot;&gt;LDA&lt;/a&gt; - Is also a technique used for &lt;code&gt;topic modeling&lt;/code&gt;, but it's different from &lt;code&gt;LSA&lt;/code&gt; in that it actually learns internal representations that tend to be more smooth and intuitive. In general, the results you get from &lt;code&gt;LDA&lt;/code&gt; are better for modeling document similarity than &lt;code&gt;LSA&lt;/code&gt;, but not quite as good for learning how to discriminate strongly between topics.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Pachinko_allocation&quot;&gt;Pachinko Allocation&lt;/a&gt; - Is a really neat extension on top of LDA. In general, this is just a significantly improved version of &lt;code&gt;LDA&lt;/code&gt;, with the only downside being that it takes a bit longer to train and open-source implementations are a little harder to come by&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://code.google.com/p/word2vec/&quot;&gt;word2vec&lt;/a&gt; - Google has been working on a series of techniques for intelligently reducing words and documents to more reasonable vectors than the sparse vectors yielded by techniques such as &lt;code&gt;Count Vectorizers&lt;/code&gt; and &lt;code&gt;TF-IDF&lt;/code&gt;. Word2vec is great because it has a number of open source implementations. Once you have the vector, any other similarity metric (like cosine distance) can be used on top of it with significantly more efficacy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot;&gt;doc2vec&lt;/a&gt; - Also known as &lt;code&gt;paragraph vectors&lt;/code&gt;, this is the latest and greatest in a series of papers by Google, looking into dense vector representations of documents. The &lt;code&gt;gensim&lt;/code&gt; library in python has an implementation of &lt;code&gt;word2vec&lt;/code&gt; that is straightforward enough that it can pretty reasonably be leveraged to build &lt;code&gt;doc2vec&lt;/code&gt;, but make sure to keep the license in mind if you want to go down this route&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope that helps, let me know if you've got any questions.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-07T15:36:40.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="690" PostTypeId="2" ParentId="683" CreationDate="2014-07-07T18:06:24.583" Score="2" Body="&lt;p&gt;Train 2 generative models, one for each dataset (spam only, spam plus ham), that will give you the probability that a datapoint is drawn from the same probability distribution of the training data. Assign emails as spam or ham based on which model gives you the highest probability of the document arising from the training data used to train it. Example generative models are RBM's, autoencoders (in that case, which model has the lowest reconstruction error). There are likely some bayesian generative models also that will assign a probability to a data point based on some training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best option though would be to take time to curate a second dataset containing only ham. That will give you higher classification accuracy. Assuming a lower proportion of spam to ham emails, that should not be too hard. You can even use Mechanical Turk if you lack the time or resources (or interns \ graduates students or other cheap labor).&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2014-07-07T18:06:24.583" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="691" PostTypeId="2" ParentId="678" CreationDate="2014-07-07T18:20:36.090" Score="6" Body="&lt;p&gt;Empirically I have found LSA vastly superior to LDA every time and on every dataset I have tried it on. I have talked to other people who have said the same thing. It's also been used to win a number of the SemEval competitions for measuring semantic similarity between documents, often in combinations with a wordnet based measure, so I wouldn't say it's going out of fashion, or is definitely inferior to LDA, which is better for topic modelling and not semantic similarity in my experience, contrary to what some responders have stated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use gensim (a python library), it has LSA, LDA and word2vec, so you can easily compare the 3. doc2vec is a cool idea, but does not scale very well and you will likely have to implement it yourself as I am unaware of any open source implementations. It does not scale well as for each document, a new and separate model has to be built using SGD, a slow machine learning algorithm. But it will probably give you the most accurate results. LSA and LDA also don't scale well (word2vec does however), LDA scales worse in general. Gensim's implementations are very fast however, as it uses iterative SVD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One other note, if you use word2vec, you will still have to determine a way to compose vectors from documents, as it gives you a different vector per word. The simplest way to do this is to normalize each vector and take the mean over all word vectors in the document, or take a weighted mean by idf weighting of each word. So it's not as simple as 'use word2vec', you will need to do something further to compute document similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would personally go with LSA, as I have seen it work well empirically, and gensim's library scales very well. However, there's no free lunch, so preferably try each method and see which works better for your data.&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2014-07-07T18:20:36.090" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="692" PostTypeId="2" ParentId="679" CreationDate="2014-07-07T18:28:50.420" Score="4" Body="&lt;p&gt;The cosine similarity metric does a good (if not perfect) job of controlling for the document length, so comparing the similarity of 2 documents or 2 queries using the cosine metric and tf idf weights for the words should work well in either case. I would also recommend doing LSA first on tf idf weights, and then computing the cosine distance\similarities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to build a search engine, I would recommend using a free open source search engine like solr or elastic search, or just the raw lucene libraries, as they do most of the work for you, and have good built in methods for handling the query to document similarity problem.&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2014-07-07T18:28:50.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="693" PostTypeId="2" ParentId="662" CreationDate="2014-07-07T18:36:23.430" Score="7" Body="&lt;p&gt;I work for an online jobs site and we build solutions to recommend jobs based on resumes. Our approach take's a person's job title (or desired job title if a student and known), along with skills we extract from their resume, and their location (which is very important to most people) and find matches with jobs based on that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;in terms of document classification, I would take a similar approach. I would recommend computing a tf idf matrix for each resume as a standard bag of words model, extracting just the person's job title and skills (for which you will need to define a list of skills to look for), and feed that into a ML algorithm. I would recommend trying knn, and an SVM, the latter works very well with high dimensional text data. Linear SVM's tend to do better than non-linear (e.g. using RBf kernels). If you have that outputting reasonable results, I would then play with extracting features using a natural language parser \ chunker, and also some custom built phrases matched by regex's.&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2014-07-07T18:36:23.430" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="694" PostTypeId="1" AcceptedAnswerId="695" CreationDate="2014-07-07T19:17:04.973" Score="150" ViewCount="126169" Body="&lt;p&gt;I'm using Neural Networks to solve different Machine learning problems. I'm using Python and &lt;a href=&quot;http://pybrain.org/&quot; rel=&quot;noreferrer&quot;&gt;pybrain&lt;/a&gt; but this library is almost discontinued. Are there other good alternatives in Python?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="8432" LastEditDate="2017-05-29T17:43:26.890" LastActivityDate="2018-12-11T23:39:52.667" Title="Best python library for neural networks" Tags="|machine-learning|python|neural-network|" AnswerCount="17" CommentCount="13" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="695" PostTypeId="2" ParentId="694" CreationDate="2014-07-07T19:55:51.057" Score="132" Body="&lt;p&gt;UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space.  In particular, I would recommend checking out:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/mila-udem/blocks&quot; rel=&quot;noreferrer&quot;&gt;Blocks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/Lasagne/Lasagne&quot; rel=&quot;noreferrer&quot;&gt;Lasagne&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/fchollet/keras&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/uaca/deepy&quot; rel=&quot;noreferrer&quot;&gt;Deepy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/dnouri/nolearn&quot; rel=&quot;noreferrer&quot;&gt;Nolearn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/itdxer/neupy&quot; rel=&quot;noreferrer&quot;&gt;NeuPy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They each have their strengths and weaknesses, so give them all a go and see which best suits your use case.  Although I would have recommended using PyLearn2 a year ago, the community is no longer active so I would recommend looking elsewhere.  My original response to the answer is included below but is largely irrelevant at this point.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/software/pylearn2/&quot; rel=&quot;noreferrer&quot;&gt;PyLearn2&lt;/a&gt; is generally considered the library of choice for neural networks and deep learning in python.  It's designed for easy scientific experimentation rather than ease of use, so the learning curve is rather steep, but if you take your time and follow the tutorials I think you'll be happy with the functionality it provides.  Everything from standard Multilayer Perceptrons to Restricted Boltzmann Machines to Convolutional Nets to Autoencoders is provided.  There's great GPU support and everything is built on top of Theano, so performance is typically quite good.  The source for PyLearn2 is available &lt;a href=&quot;https://github.com/lisa-lab/pylearn2&quot; rel=&quot;noreferrer&quot;&gt;on github&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be aware that PyLearn2 has the opposite problem of PyBrain at the moment -- rather than being abandoned, PyLearn2 is under active development and is subject to frequent changes.&lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="9634" LastEditDate="2016-12-12T14:48:39.890" LastActivityDate="2016-12-12T14:48:39.890" CommentCount="9" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="697" PostTypeId="1" AcceptedAnswerId="714" CreationDate="2014-07-07T21:26:36.830" Score="1" ViewCount="1809" Body="&lt;p&gt;I'm trying to run some analysis with some big datasets (eg 400k rows vs. 400 columns) with R (e.g. using neural networks and recommendation systems).&#xA;But, it's taking too long to process the data (with huge matrices, e.g. 400k rows vs. 400k columns).&#xA;What are some free/cheap ways to improve R performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm accepting packages or web services suggestions (other options are welcome).&lt;/p&gt;&#xA;" OwnerUserId="199" LastActivityDate="2017-11-15T03:54:08.703" Title="Running huge datasets with R" Tags="|bigdata|r|optimization|processing|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="698" PostTypeId="2" ParentId="265" CreationDate="2014-07-08T02:08:04.560" Score="5" Body="&lt;p&gt;They can't predict, but they can tell you the most likely result. There's an study about this kind of approach from &lt;strong&gt;Etienne&lt;/strong&gt; - &lt;a href=&quot;http://blog.wolfram.com/2014/06/20/predicting-who-will-win-the-world-cup-with-wolfram-language/&quot; rel=&quot;noreferrer&quot;&gt;Predicting Who Will Win the World Cup with Wolfram Language&lt;/a&gt;. This is a very detailed study, so you can check all the methodology used to get the predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interesting enough, 11 from 15 matches were correct!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As one might expect, Brazil is the favorite, with a probability to win of 42.5%. This striking result is due to the fact that Brazil has both the highest Elo ranking and plays at home.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(Let's go Brazil!)&lt;/p&gt;&#xA;" OwnerUserId="1379" LastActivityDate="2014-07-08T02:08:04.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="699" PostTypeId="2" ParentId="679" CreationDate="2014-07-08T06:40:36.923" Score="4" Body="&lt;p&gt;From my experience only some classes of queries can be classified on lexical features (due to ambiguity of natural language). Instead you can try to use boolean search results (sites or segments of sites, not documents, without ranking) as features for classification (instead on words). This approach works well in classes where there is a big lexical ambiguity in a query but exists a lot of good sites relevant to the query (e.g. movies, music, commercial queries and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, for offline classification you can do LSI on query-site matrix. See &quot;Introduction to Information Retrieval&quot; book for details.&lt;/p&gt;&#xA;" OwnerUserId="1384" LastActivityDate="2014-07-08T06:40:36.923" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="700" PostTypeId="1" AcceptedAnswerId="900" CreationDate="2014-07-08T07:29:34.167" Score="7" ViewCount="2671" Body="&lt;p&gt;I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might change dynamically. I'm looking for an ideal clustering algorithm which can handle these issues efficiently.&lt;/p&gt;&#xA;&lt;p&gt;I know &lt;a href=&quot;https://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/C/sk_means.htm&quot; rel=&quot;noreferrer&quot;&gt;sequential k-means clustering&lt;/a&gt; copes with the addition of new instances, and I suppose with minor modification it can work with dynamic instance values (i.e. first taking the modified instance from the respective cluster, then updating the mean of the cluster and finally giving the modified instance as an input to the algorithm just as the addition of an unseen instance).&lt;/p&gt;&#xA;&lt;p&gt;My concern with using the k-means algorithm is the requirement of supplying the number of clusters as an input. I know that they beat other clustering algorithms (GAs, MSTs, Hierarchical Methods etc.) in time&amp;amp;space complexity. Honestly I'm not sure, but maybe I can get away with using one of the aforementioned algorithms. Even that my datasets are relatively large, the existence of a single dimension makes me wonder.&lt;/p&gt;&#xA;&lt;p&gt;More specifically a typical test case of mine would contain about 10K-200K 1-dimensional datapoints. I would like to complete the clustering preferably under a second. The dynamic changes in the value points are assumed to be smooth, i.e. relatively small. Thus being able to use existing solutions (i.e. being able to continue clustering on the existing one when a value is changed or new one is added) is highly preferred.&lt;/p&gt;&#xA;&lt;p&gt;So all in all:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Can you think of an algorithm which will provide a sweet spot between computational efficiency and the accuracy of clusters wrt. the problem defined above?&lt;/p&gt;&#xA;&lt;p&gt;Are there some nice heuristics for the k-means algorithm to automatically compute the value of K beforehand?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1386" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-21T20:48:53.453" Title="Efficient dynamic clustering" Tags="|machine-learning|algorithms|clustering|k-means|hierarchical-data-format|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="701" PostTypeId="1" AcceptedAnswerId="702" CreationDate="2014-07-08T07:37:57.123" Score="4" ViewCount="1296" Body="&lt;p&gt;I have generated a dataset of pairwise distances as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id_1 id_2 dist_12&#xA;id_2 id_3 dist_23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to cluster this data so as to identify the pattern. I have been looking at Spectral clustering and DBSCAN, but I haven't been able to come to a conclusion and have been ambiguous on how to make use of the existing implementations of these algorithms. I have been looking at Python and Java implementations so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone point me to a tutorial or demo on how to make use of these clustering algorithms to handle the situation in hand?&lt;/p&gt;&#xA;" OwnerUserId="645" LastEditorUserId="13727" LastEditDate="2016-03-02T15:50:44.287" LastActivityDate="2016-03-02T15:50:44.287" Title="Clustering pair-wise distance dataset" Tags="|data-mining|clustering|dbscan|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="702" PostTypeId="2" ParentId="701" CreationDate="2014-07-08T09:18:17.990" Score="2" Body="&lt;p&gt;In the scikit-learn implementation of Spectral clustering and DBSCAN you do not need to precompute the distances, you should input the sample coordinates for all &lt;code&gt;id_1&lt;/code&gt; ... &lt;code&gt;id_n&lt;/code&gt;.  Here is a simplification of the &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html&quot; rel=&quot;nofollow&quot;&gt;documented example comparison of clustering algorithms&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import cluster&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;## Prepare the data&#xA;X = np.random.rand(1500, 2)&#xA;# When reading from a file of the form: `id_n coord_x coord_y`&#xA;# you will need this call instead:&#xA;# X = np.loadtxt('coords.csv', usecols=(1, 2))&#xA;X = StandardScaler().fit_transform(X)&#xA;&#xA;## Instantiate the algorithms&#xA;spectral = cluster.SpectralClustering(n_clusters=2,&#xA;                                      eigen_solver='arpack',&#xA;                                      affinity=&quot;nearest_neighbors&quot;)&#xA;dbscan = cluster.DBSCAN(eps=.2)&#xA;&#xA;## Use the algorithms&#xA;spectral_labels = spectral.fit_predict(X)&#xA;dbscan_labels = dbscan.fit_predict(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2014-07-08T09:18:17.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="703" PostTypeId="2" ParentId="694" CreationDate="2014-07-08T10:36:44.220" Score="23" Body="&lt;p&gt;Pylearn relies on Theano and as mentioned in the other answer to use the library is quite complicated, until you get the hold of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meantime I would suggest using &lt;a href=&quot;https://github.com/lmjohns3/theano-nets/&quot;&gt;Theanets&lt;/a&gt;. It also built on top of Theano, but is much more easier to work with. It might be true, that it doesn't have all the features of Pylearn, but for the basic work it's sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also it's open source, so you can add custom networks on the fly, if you dare. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Dec 2015. Recently I have started using &lt;a href=&quot;http://keras.io/&quot;&gt;Keras&lt;/a&gt;. It is a bit lower level than Theanets, but much more powerful. For basic tests the Theanets is appropriate. But if you want to do some research in field of ANN Keras is much more flexible. Plus the Keras can use &lt;strong&gt;Tensorflow&lt;/strong&gt; as a backend.&lt;/p&gt;&#xA;" OwnerUserId="1390" LastEditorUserId="1390" LastEditDate="2015-12-10T09:35:30.443" LastActivityDate="2015-12-10T09:35:30.443" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="704" PostTypeId="1" CreationDate="2014-07-08T13:45:07.583" Score="11" ViewCount="442" Body="&lt;p&gt;In my university, we have an HPC computing cluster. I use the cluster to train classifiers and so on. So, usually, to send a job to the cluster, (e.g. python scikit-learn script), I need to write a Bash script that contains (among others) a command like &lt;code&gt;qsub script.py&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I find this process very very frustrating. Usually what happens is that I write the python script on my laptop and then I login to the server and update the SVN repository, so I get the same python script there. Then I write that Bash script or edit it, so I can run the bash script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see this is really frustrating since, for every little update for the python script, I need to do many steps to have it executed at the computing cluster. Of course the task gets even more complicated when I have to put the data on the server and use the datasets' path on the server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure many people here are using computing clusters for their data science tasks. I just want to know how you guys manage sending the jobs to the clusters?&lt;/p&gt;&#xA;" OwnerUserId="728" LastEditorUserId="84" LastEditDate="2014-07-09T00:25:47.190" LastActivityDate="2014-07-14T13:06:05.523" Title="Working with HPC clusters" Tags="|bigdata|data-mining|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="705" PostTypeId="2" ParentId="685" CreationDate="2014-07-08T14:32:55.827" Score="4" Body="&lt;p&gt;Stochastic gradient descent is a method of setting the parameters of the regressor; since the objective for logistic regression is convex (has only one maximum), this won't be an issue and SGD is generally only needed to improve convergence speed with masses of training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What your numbers suggest to me is that your features are not adequate to separate the classes. Consider adding extra features if you can think any any that are useful. You might also consider interactions and quadratic features in your original feature space.&lt;/p&gt;&#xA;" OwnerUserId="1399" LastActivityDate="2014-07-08T14:32:55.827" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="707" PostTypeId="2" ParentId="655" CreationDate="2014-07-09T00:37:51.167" Score="5" Body="&lt;p&gt;Try a recurrent neural network, a model well suited for time series data. They're notoriously difficult to train, but seem to perform well when trained properly: &lt;a href=&quot;http://cs229.stanford.edu/proj2012/BernalFokPidaparthi-FinancialMarketTimeSeriesPredictionwithRecurrentNeural.pdf&quot;&gt;http://cs229.stanford.edu/proj2012/BernalFokPidaparthi-FinancialMarketTimeSeriesPredictionwithRecurrentNeural.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="684" LastActivityDate="2014-07-09T00:37:51.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="708" PostTypeId="2" ParentId="704" CreationDate="2014-07-09T00:44:28.207" Score="4" Body="&lt;p&gt;There are many solutions to ease the burden of copying the file from a local machine to the computing nodes in the clusters. A simple approach is to use an interface that allows multi-access to the machines in the cluster, like &lt;a href=&quot;http://sourceforge.net/projects/clusterssh/&quot; rel=&quot;nofollow&quot;&gt;clusterssh&lt;/a&gt; (cssh). It allows you to type commands to multiple machines at once via a set of terminal screens (each one a ssh connection to a different machine in the cluster).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your cluster seem to have &lt;code&gt;qsub&lt;/code&gt; set up, your problem may be rather related to &lt;em&gt;replicating&lt;/em&gt; the data along the machines (other than simply running a command in each node). So, to address this point, you may either write an &lt;code&gt;scp&lt;/code&gt; script, to copy things to and from each node in the cluster (which is surely better addressed with SVN), or you may set up a NFS. This would allow for a simple and transparent access to the data, and also reduce the need for replicating unnecessary data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you could access a node, copy the data to such place, and simply use the data &lt;em&gt;remotely&lt;/em&gt;, via network communication. I'm not acquainted with how to set up a NFS, but you already have access to it (in case your home folder is the same across the machines you access). Then, the scripts and data could be sent to a single place, and later accessed from others. This is akin to the SVN approach, except it's more transparent/straightforward.&lt;/p&gt;&#xA;" OwnerUserId="84" LastActivityDate="2014-07-09T00:44:28.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="709" PostTypeId="2" ParentId="697" CreationDate="2014-07-09T07:08:34.740" Score="2" Body="&lt;p&gt;Since you mention you are building a recommendation system, I believe you have a sparse matrix which you are working on. Check &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/Matrix/html/sparseMatrix.html&quot; rel=&quot;nofollow&quot;&gt;sparseMatrix&lt;/a&gt; from Matrix package. This should be able to help you with storing your large size matrix in memory and train your model. &lt;/p&gt;&#xA;" OwnerUserId="1131" LastActivityDate="2014-07-09T07:08:34.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="710" PostTypeId="1" AcceptedAnswerId="720" CreationDate="2014-07-09T11:05:40.813" Score="1" ViewCount="897" Body="&lt;p&gt;I'm looking for commercial text summarization tools (APIs, Libraries,...) which are able to perform any of the following tasks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Extractive Multi-Document Summarization (Generic or query-based)&lt;/li&gt;&#xA;&lt;li&gt;Extractive Single-Document Summarization (Generic or query-based)&lt;/li&gt;&#xA;&lt;li&gt;Generative Single-Document Summarization (Generic or query-based)&lt;/li&gt;&#xA;&lt;li&gt;Generative Multi-Document Summarization (Generic or query-based)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="979" LastEditorUserId="21" LastEditDate="2014-08-08T00:27:05.340" LastActivityDate="2017-03-11T20:01:01.750" Title="Commercial Text Summarization Tools" Tags="|text-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2018-06-02T00:38:05.497" ContentLicense="CC BY-SA 3.0" />
  <row Id="711" PostTypeId="1" AcceptedAnswerId="712" CreationDate="2014-07-09T12:22:22.400" Score="71" ViewCount="11328" Body="&lt;p&gt;This question is in response to a comment I saw on another question.&lt;/p&gt;&#xA;&lt;p&gt;The comment was regarding the Machine Learning course syllabus on Coursera, and along the lines of &amp;quot;SVMs are not used so much nowadays&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;I have only just finished the relevant lectures myself, and my understanding of SVMs is that they are a robust and efficient learning algorithm for classification, and that when using a kernel, they have a &amp;quot;niche&amp;quot; covering number of features perhaps 10 to 1000 and number of training samples perhaps 100 to 10,000. The limit on training samples is because the core algorithm revolves around optimising results generated from a square matrix with dimensions based on number of training samples, not number of original features.&lt;/p&gt;&#xA;&lt;p&gt;So does the comment I saw refer some real change since the course was made, and if so, what is that change: A new algorithm that covers SVM's &amp;quot;sweet spot&amp;quot; just as well, better CPUs meaning SVM's computational advantages are not worth as much? Or is it perhaps opinion or personal experience of the commenter?&lt;/p&gt;&#xA;&lt;p&gt;I tried a search for e.g. &amp;quot;are support vector machines out of fashion&amp;quot; and found nothing to imply they were being dropped in favour of anything else.&lt;/p&gt;&#xA;&lt;p&gt;And Wikipedia has this: &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine#Issues&quot; rel=&quot;noreferrer&quot;&gt;http://en.wikipedia.org/wiki/Support_vector_machine#Issues&lt;/a&gt; . . . the main sticking point appears to be difficulty of interpreting the model. Which makes SVM fine for a black-box predicting engine, but not so good for generating insights. I don't see that as a major issue, just another minor thing to take into account when picking the right tool for the job (along with nature of the training data and learning task etc).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2020-08-17T07:11:34.777" LastActivityDate="2020-08-17T07:11:34.777" Title="Are Support Vector Machines still considered &quot;state of the art&quot; in their niche?" Tags="|data-mining|svm|state-of-the-art|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="712" PostTypeId="2" ParentId="711" CreationDate="2014-07-09T13:07:13.303" Score="56" Body="&lt;p&gt;SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory.&lt;/li&gt;&#xA;&lt;li&gt;The so-called power of kernels: With appropriate kernels you can transform feature space into a higher dimension so that it becomes linearly separable. The notion of kernels work with arbitrary objects on which you can define some notion of similarity with the help of inner products... and hence SVMs can classify arbitrary objects such as trees, graphs etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are some significant disadvantages as well.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Parameter sensitivity: The performance is highly sensitive to the choice of the regularization parameter C, which allows some variance in the model.&lt;/li&gt;&#xA;&lt;li&gt;Extra parameter for the Gaussian kernel: The radius of the Gaussian kernel can have a significant impact on classifier accuracy. Typically a grid search has to be conducted to find optimal parameters. LibSVM has a support for grid search.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;SVMs generally belong to the class of &quot;Sparse Kernel Machines&quot;. The sparse vectors in the case of SVM are the support vectors which are chosen from the maximum margin criterion. Other sparse vector machines such as the &lt;strong&gt;Relevance Vector Machine&lt;/strong&gt; (RVM) perform better than SVM. The following figure shows a comparative performance of the two. In the figure, the x-axis shows one dimensional data from two classes y={0,1}. The mixture model is defined as P(x|y=0)=Unif(0,1) and P(x|y=1)=Unif(.5,1.5) (Unif denotes uniform distribution). 1000 points were sampled from this mixture and an SVM and an RVM were used to estimate the posterior. The problem of SVM is that the predicted values are far off from the true log odds.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/zNYbt.png&quot; alt=&quot;RVM vs. SVM&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very effective classifier, which is very popular nowadays, is the &lt;strong&gt;Random Forest&lt;/strong&gt;. The main advantages are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Only one parameter to tune (i.e. the number of trees in the forest)&lt;/li&gt;&#xA;&lt;li&gt;Not utterly parameter sensitive&lt;/li&gt;&#xA;&lt;li&gt;Can easily be extended to multiple classes&lt;/li&gt;&#xA;&lt;li&gt;Is based on probabilistic principles (maximizing mutual information gain with the help of decision trees)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="984" LastEditorUserId="984" LastEditDate="2014-07-09T20:23:12.380" LastActivityDate="2014-07-09T20:23:12.380" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="713" PostTypeId="1" AcceptedAnswerId="715" CreationDate="2014-07-09T17:51:40.583" Score="3" ViewCount="1350" Body="&lt;p&gt;I have installed cloudera CDH5 Quick start VM on VM player. When I login through HUE in the first page I am the following error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“Potential misconfiguration detected. Fix and restart Hue.”&lt;img src=&quot;https://i.stack.imgur.com/vnq5P.png&quot; alt=&quot;Potential misconfiguration detected. Fix and restart Hue&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to solve this issue.&lt;/p&gt;&#xA;" OwnerUserId="1314" LastEditorUserId="1352" LastEditDate="2016-04-04T01:08:43.203" LastActivityDate="2016-04-04T01:08:43.203" Title="Cloudera QuickStart VM Error" Tags="|apache-hadoop|" AnswerCount="2" CommentCount="4" ClosedDate="2016-04-04T01:08:54.313" ContentLicense="CC BY-SA 3.0" />
  <row Id="714" PostTypeId="2" ParentId="697" CreationDate="2014-07-10T08:38:43.353" Score="6" Body="&lt;p&gt;Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Check sparseMatrix from Matrix package as mentioned by @Sidhha&lt;/li&gt;&#xA;&lt;li&gt;Try running your model in parallel using packages like snowfall, &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf&quot;&gt;Parallel&lt;/a&gt;. Check this &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot;&gt;list of packages on Cran&lt;/a&gt; which can help you runnning your model in multicore parallel mode.&lt;/li&gt;&#xA;&lt;li&gt;You can also try &lt;a href=&quot;http://datatable.r-forge.r-project.org/&quot;&gt;data.table&lt;/a&gt; package. It is quite phenomenal in speed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Good reads:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://theodi.org/blog/fig-data-11-tips-how-handle-big-data-r-and-1-bad-pun&quot;&gt;11 Tips on How to Handle Big Data in R (and 1 Bad Pun)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://adv-r.had.co.nz/Performance.html&quot;&gt;Why R is slow &amp;amp; how to improve its Performance?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2433" LastActivityDate="2014-07-10T08:38:43.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="715" PostTypeId="2" ParentId="713" CreationDate="2014-07-10T09:16:39.937" Score="2" Body="&lt;p&gt;Go into the other link from home - to the cloudera manager.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From there, you'll see Hue can be restarted, but there is probably an alert that needs to be resolved in there first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I remember right there's some initial configuration that needs to be done on the quickstart VM that's spelled out as soon as you log into the manager application.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-07-10T09:16:39.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="716" PostTypeId="1" AcceptedAnswerId="718" CreationDate="2014-07-10T10:07:13.523" Score="22" ViewCount="7218" Body="&lt;p&gt;I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The &amp;quot;best&amp;quot; way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like &lt;a href=&quot;http://www3.nd.edu/%7Emclark19/learn/CorrelationComparison.pdf&quot; rel=&quot;noreferrer&quot;&gt;correlation measures&lt;/a&gt; etc?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="98307" LastEditDate="2020-08-05T11:06:38.627" LastActivityDate="2021-03-11T19:26:46.897" Title="How to choose the features for a neural network?" Tags="|machine-learning|neural-network|feature-selection|feature-extraction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="717" PostTypeId="1" AcceptedAnswerId="723" CreationDate="2014-07-10T11:55:49.637" Score="6" ViewCount="101" Body="&lt;p&gt;I'm using an experimental design to test the robustness of different classification methods, and now I'm searching for the correct definition of such design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm creating different subsets of the full dataset by cutting away some samples. Each subset is created independently with respect to the others. Then, I run each classification method on every subset. Finally, I estimate the accuracy of each method as how many classifications on subsets are in agreement with the classification on the full dataset. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Classification-full     1    2    3    2    1    1    2&#xA;&#xA;Classification-subset1  1    2         2    3    1   &#xA;Classification-subset2       2    3         1    1    2&#xA;...&#xA;&#xA;Accuracy                1    1    1    1  0.5    1    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a correct name to this methodology? I thought it can fall under &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrapping_(statistics)&quot; rel=&quot;noreferrer&quot;&gt;bootstrapping&lt;/a&gt; but I'm not sure about this.&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="84" LastEditDate="2014-07-10T18:04:59.523" LastActivityDate="2014-07-11T11:32:13.550" Title="How to define a custom resampling methodology" Tags="|classification|definitions|accuracy|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="718" PostTypeId="2" ParentId="716" CreationDate="2014-07-10T15:43:53.177" Score="20" Body="&lt;p&gt;A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information.  A low correlation between the new feature and existing features is likely preferable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the new feature was manually constructed from a combination of existing features, consider leaving it out.  The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers.  Whenever possible, prefer learning features to engineering them.&lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="684" LastEditDate="2014-07-10T19:18:05.697" LastActivityDate="2014-07-10T19:18:05.697" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="719" PostTypeId="2" ParentId="653" CreationDate="2014-07-10T22:42:13.720" Score="4" Body="&lt;p&gt;nsl-&#xA;I'm a beginner at machine learning, so forgive the lay-like description here, but it sounds like you might be able to use topic modelling, like latent dirichlet analysis (LDA). It's an algorithm widely used to classify documents, according to what topics they are about, based on the words found and the relative frequencies of those words in the overall corpus.  I bring it up mainly because, in LDA it's not necessary to define the topics in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the help pages on LDA are mostly written for text analysis, the analogy I would use, in order to apply it to your question, is:&#xA;- Treat each gene expression, or feature, as a 'word' (sometimes called a token in typical LDA text-classification applications)&#xA;- Treat each sample as a document (ie it contains an assortment of words, or gene expressions)&#xA;- Treat the signatures as pre-existing topics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I'm not mistaken, LDA should give weighted probabilities for each topic, as to how strongly it is present in each document.&lt;/p&gt;&#xA;" OwnerUserId="2443" LastActivityDate="2014-07-10T22:42:13.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="720" PostTypeId="2" ParentId="710" CreationDate="2014-07-10T23:38:58.153" Score="2" Body="&lt;p&gt;There are a couple of open source options I know of - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;LibOTS - &lt;a href=&quot;http://libots.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;http://libots.sourceforge.net/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DocSum - &lt;a href=&quot;http://docsum.sourceforge.net/docsum/web/about.php&quot; rel=&quot;nofollow&quot;&gt;http://docsum.sourceforge.net/docsum/web/about.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of commercial solutions - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intellix Summarizer Pro - &lt;a href=&quot;http://summarizer.intellexer.com/order_summarizer_pro.php&quot; rel=&quot;nofollow&quot;&gt;http://summarizer.intellexer.com/order_summarizer_pro.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Copernic Summarizer - &lt;a href=&quot;http://www.copernic.com/en/products/summarizer/&quot; rel=&quot;nofollow&quot;&gt;http://www.copernic.com/en/products/summarizer/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this one is a web service - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;TextTeaser - &lt;a href=&quot;http://www.textteaser.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.textteaser.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure there are plenty of others out there.  I have used Copernic a good deal and it's pretty good, but I was hoping it could be automated easily, which it can't - at least it couldn't when I used it.  &lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-07-10T23:38:58.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="721" PostTypeId="2" ParentId="658" CreationDate="2014-07-11T01:28:24.957" Score="7" Body="&lt;p&gt;When it comes to dealing with many disparate kinds of data, especially when the relationships between them are unclear, I would strongly recommend a technique based on &lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree_learning&quot;&gt;decision trees&lt;/a&gt;, the most popular ones today to the best of my knowledge are &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot;&gt;random forest&lt;/a&gt;, and &lt;a href=&quot;http://www.montefiore.ulg.ac.be/~ernst/uploads/news/id63/extremely-randomized-trees.pdf&quot;&gt;extremely randomized trees&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both have implementations in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.fit_transform&quot;&gt;sklearn&lt;/a&gt;, and they are pretty straightforward to use. At a very high level, the reason that a &lt;code&gt;decision tree&lt;/code&gt;-based approach is advantageous for multiple disparate kinds of data is because decision trees are largely independent from the specific data they are dealing with, just so long as they are capable of understanding your representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll still have to fit your data into a feature vector of some kind, but based on your example that seems like a pretty straightforward task, and if you're willing to go a little deeper on your implementation you could certainly come up with a custom tree-splitting rule without actually having to change anything in the underlying algorithm. The &lt;a href=&quot;http://www.cs.nyu.edu/~roweis/csc2515-2006/readings/quinlan.pdf&quot;&gt;original paper&lt;/a&gt; is a pretty decent place to start if you want to give that a shot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want pseudo-structural data from your text data though, I might suggest looking into &lt;code&gt;doc2vec&lt;/code&gt;, recently developed by Google. I don't think there are any good open-source implementations now, but it's a pretty straightforward improvement on the &lt;code&gt;word2vec&lt;/code&gt; algorithm, which has implementations in at least &lt;code&gt;C&lt;/code&gt; and &lt;code&gt;python&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps! Let me know if you've got any more questions.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-11T01:28:24.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="722" PostTypeId="2" ParentId="52" CreationDate="2014-07-11T09:49:32.933" Score="23" Body="&lt;p&gt;R contains some &lt;em&gt;standard&lt;/em&gt; functions for data manipulation, which can be used for data cleaning, in its &lt;strong&gt;base&lt;/strong&gt; package (&lt;code&gt;gsub&lt;/code&gt;, &lt;code&gt;transform&lt;/code&gt;, etc.), as well as in various third-party packages, such as &lt;strong&gt;stringr&lt;/strong&gt;, &lt;strong&gt;reshape&lt;/strong&gt;/&lt;strong&gt;reshape2&lt;/strong&gt;, and &lt;strong&gt;plyr&lt;/strong&gt;/&lt;strong&gt;dplyr&lt;/strong&gt;. Examples and best practices of usage for these packages and their functions are described in the following paper: &lt;a href=&quot;http://vita.had.co.nz/papers/tidy-data.pdf&quot;&gt;http://vita.had.co.nz/papers/tidy-data.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, R offers some packages specifically &lt;em&gt;focused&lt;/em&gt; on data cleaning and transformation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;editrules&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/editrules/index.html&quot;&gt;http://cran.r-project.org/web/packages/editrules/index.html&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;deducorrect&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/deducorrect/index.html&quot;&gt;http://cran.r-project.org/web/packages/deducorrect/index.html&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;StatMatch&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/StatMatch/index.html&quot;&gt;http://cran.r-project.org/web/packages/StatMatch/index.html&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MatchIt&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/MatchIt/index.html&quot;&gt;http://cran.r-project.org/web/packages/MatchIt/index.html&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;DataCombine&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/DataCombine&quot;&gt;http://cran.r-project.org/web/packages/DataCombine&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;data.table&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/data.table&quot;&gt;http://cran.r-project.org/web/packages/data.table&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A comprehensive and coherent approach to &lt;strong&gt;data cleaning&lt;/strong&gt; in R, including examples and use of &lt;strong&gt;editrules&lt;/strong&gt; and &lt;strong&gt;deducorrect&lt;/strong&gt; packages, as well as a description of &lt;em&gt;workflow&lt;/em&gt; (&lt;em&gt;framework&lt;/em&gt;) of data cleaning in R, is presented in the following paper, which I highly recommend: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&quot;&gt;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2014-08-02T08:16:08.640" LastActivityDate="2014-08-02T08:16:08.640" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="723" PostTypeId="2" ParentId="717" CreationDate="2014-07-11T11:32:13.550" Score="2" Body="&lt;p&gt;Random subsampling seems appropriate, bootstrapping is a bit more generic, but also correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some references and synonyms: &lt;a href=&quot;http://www.frank-dieterle.com/phd/2_4_3.html&quot; rel=&quot;nofollow&quot;&gt;http://www.frank-dieterle.com/phd/2_4_3.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="127" LastActivityDate="2014-07-11T11:32:13.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="724" PostTypeId="2" ParentId="704" CreationDate="2014-07-11T14:13:30.403" Score="4" Body="&lt;p&gt;Your approach of using a source version repository is a good one and it actually allows you also working on the cluster and then copying everything back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find yourself making minor edits to your Python script on your laptop, then updating your SVN directory on the cluster, why not work directly on the cluster frontend, make all needed minor edits, and then, at the end of the day, commit everything there and update on your laptop? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All you need is to get familiar with the environment there (OS, editor, etc.) or install your own environment (I usually install in my home directory the latest version of &lt;a href=&quot;http://www.vim.org/&quot; rel=&quot;nofollow&quot;&gt;Vim&lt;/a&gt;, &lt;a href=&quot;http://tmux.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Tmux&lt;/a&gt;, etc. with the proper dotfiles so I feel at home there.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can version your data, and even your intermediate results if size permits. My repositories often comprise code, data (original and cleaned versions), documentation, and paper sources for publishing (latex)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can script your job submission to avoid modifying scripts manually. &lt;code&gt;qsub&lt;/code&gt; accepts a script from stdin and also accepts all &lt;code&gt;#$&lt;/code&gt; comments as command-line arguments. &lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-07-11T14:13:30.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="725" PostTypeId="2" ParentId="634" CreationDate="2014-07-11T14:27:01.603" Score="0" Body="&lt;p&gt;What you describe falls in the category of &lt;a href=&quot;http://en.wikipedia.org/wiki/Concept_drift&quot; rel=&quot;nofollow&quot;&gt;concept drift&lt;/a&gt; in machine learning.&#xA;You might find interesting and actionable ideas in this &lt;a href=&quot;http://arxiv.org/pdf/1010.4784.pdf&quot; rel=&quot;nofollow&quot;&gt;summary paper&lt;/a&gt; and you'll find a taxonomy of the possible approaches in &lt;a href=&quot;http://www.cs.waikato.ac.nz/~abifet/PAKDD2011/PAKDD11Tutorial_Handling_Concept_Drift.pdf&quot; rel=&quot;nofollow&quot;&gt;these slides&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-07-11T14:27:01.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="726" PostTypeId="1" CreationDate="2014-07-11T21:09:58.873" Score="6" ViewCount="376" Body="&lt;p&gt;I am trying to understand a neuroscience article:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Friston, Karl J., et al. &quot;Action and behavior: a free-energy formulation.&quot; &lt;em&gt;Biological cybernetics&lt;/em&gt; 102.3 (2010): 227-260. (&lt;a href=&quot;http://link.springer.com/article/10.1007%2Fs00422-010-0364-z&quot; rel=&quot;nofollow noreferrer&quot;&gt;DOI 10.1007/s00422-010-0364-z&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this article, Friston gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and Shannon entropy. They appear on page 231 of the article as equation (5):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The resulting expression for free-energy can be expressed in three ways (with the use of the Bayes rules and simple rearrangements):&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;• Energy minus entropy&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;• Divergence plus surprise&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;• Complexity minus accuracy&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Mathematically, these correspond to:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/0eQgg.png&quot; alt=&quot;\begin {aligned} F &amp;amp;= - \langle \ln p(\tilde s, \Psi \vert m) \rangle _q + \langle \ln q(\Psi \vert \mu) \rangle _q \ &amp;amp;= D(q(\Psi \vert \mu) \parallel p(\Psi \vert \tilde s, m)) - \ln p(\tilde s \vert m) &amp;amp; \qquad \qquad &amp;amp; (5) \ &amp;amp;= D(q(\Psi \vert \mu) \parallel p(\Psi \vert m)) - \langle \ln p(\tilde s \vert \Psi, m) \rangle _q \end{aligned}&quot;&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The things I am struggling with at this point are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the meaning of the || in the 2nd and 3rd versions of the equations;&lt;/li&gt;&#xA;&lt;li&gt;and the negative logs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc?&lt;/p&gt;&#xA;" OwnerUserId="2458" LastEditorUserId="84" LastEditDate="2014-07-16T16:18:18.573" LastActivityDate="2014-07-16T16:18:18.573" Title="Trying to understand free-energy equations in a Karl Friston neuroscience article" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="727" PostTypeId="2" ParentId="671" CreationDate="2014-07-11T21:35:22.677" Score="3" Body="&lt;p&gt;Your problem is that the resets aren't part of your linear model. You either have to cut your data into different fragments at the resets, so that no reset occurs within each fragment, and you can fit a linear model to each fragment. Or you can build a more complicated model that allows for resets. In this case, either the time of occurrence of the resets has to be put into the model manually, or the time of resets has to be a free parameter in the model that is determined by fitting the model to the data.&lt;/p&gt;&#xA;" OwnerUserId="2459" LastActivityDate="2014-07-11T21:35:22.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="728" PostTypeId="2" ParentId="536" CreationDate="2014-07-11T22:03:31.950" Score="4" Body="&lt;p&gt;In addition to excellent previous answers, I'd like to recommend two papers on &lt;strong&gt;data cleaning&lt;/strong&gt;. They are not specific to &lt;em&gt;manual&lt;/em&gt; data cleaning, but, considering the benefits and advice (which I completely agree with) of expressing even &lt;strong&gt;manual&lt;/strong&gt; data transformations &lt;strong&gt;in code&lt;/strong&gt;, these resources can be as valuable. Also, despite the fact that following papers are somewhat R-focused, I believe that general &lt;em&gt;ideas&lt;/em&gt; and &lt;em&gt;workflows&lt;/em&gt; for data cleaning can be easily extracted and are equally applicable to non-R environments, as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first paper presents the concept of &lt;em&gt;tidy data&lt;/em&gt;, as well as examples and best practices of use of standard and specific R packages in data cleaning: &lt;a href=&quot;http://vita.had.co.nz/papers/tidy-data.pdf&quot; rel=&quot;nofollow&quot;&gt;http://vita.had.co.nz/papers/tidy-data.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A comprehensive and coherent approach to data cleaning in R, including examples, as well as a description of &lt;em&gt;workflow&lt;/em&gt; (&lt;em&gt;framework&lt;/em&gt;) of data cleaning in R, is presented in the following paper, which I highly recommend: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-11T22:03:31.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="729" PostTypeId="2" ParentId="671" CreationDate="2014-07-11T23:08:16.267" Score="5" Body="&lt;p&gt;I thought this was an interesting problem, so I wrote a sample data set and a linear slope estimator in R.  I hope it helps you with your problem.  I'm going to make some assumptions, the biggest is that you want to estimate a constant slope, given by some segments in your data.  Another assumption to separate the blocks of linear data is that the natural 'reset' will be found by comparing consecutive differences and finding ones that are X-standard deviations below the mean. (I chose 4 sd's, but this can be changed)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the data, and the code to generating it is at the bottom.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/2dC1w.png&quot; alt=&quot;Sample Data&quot;&gt;&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;For starters, we find the breaks and fit each set of y-values and record the slopes.&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-r prettyprint-override&quot;&gt;&lt;code&gt;# Find the differences between adjacent points&#xA;diffs = y_data[-1] - y_data[-length(y_data)]&#xA;# Find the break points (here I use 4 s.d.'s)&#xA;break_points = c(0,which(diffs &amp;lt; (mean(diffs) - 4*sd(diffs))),length(y_data))&#xA;# Create the lists of y-values&#xA;y_lists = sapply(1:(length(break_points)-1),function(x){&#xA;  y_data[(break_points[x]+1):(break_points[x+1])]&#xA;})&#xA;# Create the lists of x-values&#xA;x_lists = lapply(y_lists,function(x) 1:length(x))&#xA;#Find all the slopes for the lists of points&#xA;slopes = unlist(lapply(1:length(y_lists), function(x) lm(y_lists[[x]] ~ x_lists[[x]])$coefficients[2]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are the slopes:&#xA;(3.309110, 4.419178, 3.292029, 4.531126, 3.675178, 4.294389)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we can just take the mean to find the expected slope (3.920168).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Predicting when series reaches 120&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realized I didn't finish predicted when series reaches 120.  If we estimate the slope to be m and we see a reset at time t to a value x (x&amp;lt;120), we can predict how much longer it would take to reach 120 by some simple algebra.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/DixZv.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, t is the time it would take to reach 120 after a reset, x is what it resets to, and m is the estimated slope.  I'm not going to even touch the subject of units here, but it's good practice to work them out and make sure everything makes sense.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Creating The Sample Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sample data will consist of 100 points, random noise with a slope of 4 (Hopefully we will estimate this).  When the y-values reach a cutoff, they reset to 50.  The cutoff is randomly chosen between 115 and 120 for each reset.  Here is the R code to create the data set.&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-r prettyprint-override&quot;&gt;&lt;code&gt;# Create Sample Data&#xA;set.seed(1001)&#xA;x_data = 1:100 # x-data&#xA;y_data = rep(0,length(x_data)) # Initialize y-data&#xA;y_data[1] = 50 &#xA;reset_level = sample(115:120,1) # Select initial cutoff&#xA;for (i in x_data[-1]){ # Loop through rest of x-data&#xA;  if(y_data[i-1]&amp;gt;reset_level){ # check if y-value is above cutoff&#xA;    y_data[i] = 50             # Reset if it is and&#xA;    reset_level = sample(115:120,1) # rechoose cutoff&#xA;  }else {&#xA;    y_data[i] = y_data[i-1] + 4 + (10*runif(1)-5) # Or just increment y with random noise&#xA;  }&#xA;}&#xA;plot(x_data,y_data) # Plot data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="375" LastEditorUserId="61" LastEditDate="2019-01-01T19:50:02.190" LastActivityDate="2019-01-01T19:50:02.190" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="730" PostTypeId="1" AcceptedAnswerId="1065" CreationDate="2014-07-12T17:25:52.907" Score="13" ViewCount="1940" Body="&lt;p&gt;As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Apriori_algorithm&quot;&gt;Apriori&lt;/a&gt; algorithm was proposed in 1993, by &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=170072&quot;&gt;Agrawal et al.&lt;/a&gt;, along with the formalization of the problem. The algorithm was able to &lt;em&gt;strip-off&lt;/em&gt; some sets from the &lt;code&gt;2^n - 1&lt;/code&gt; sets (powerset) by using a lattice to maintain the data. A drawback of the approach was the need to re-read the database to compute the frequency of each set expanded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later, on year 1997, &lt;a href=&quot;http://www.computer.org/csdl/trans/tk/2000/03/k0372-abs.html&quot;&gt;Zaki et al.&lt;/a&gt; proposed the algorithm &lt;a href=&quot;http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Frequent_Pattern_Mining/The_Eclat_Algorithm&quot;&gt;Eclat&lt;/a&gt;, which &lt;em&gt;inserted&lt;/em&gt; the resulting frequency of each set inside the lattice. This was done by adding, at each node of the lattice, the set of transaction-ids that had the items from root to the referred node. The main contribution is that one does not have to re-read the entire dataset to know the frequency of each set, but the memory required to keep such data structure built may exceed the size of the dataset itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 2000, &lt;a href=&quot;http://dl.acm.org/citation.cfm?doid=335191.335372&quot;&gt;Han et al.&lt;/a&gt; proposed an algorithm named &lt;a href=&quot;http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Frequent_Pattern_Mining/The_FP-Growth_Algorithm&quot;&gt;FPGrowth&lt;/a&gt;, along with a prefix-tree data structure named FPTree. The algorithm was able to provide significant data compression, while also granting that only frequent itemsets would be yielded (without candidate itemset generation). This was done mainly by sorting the items of each transaction in decreasing order, so that the most frequent items are the ones with the least repetitions in the tree data structure. Since the frequency only descends while traversing the tree in-depth, the algorithm is able to &lt;em&gt;strip-off&lt;/em&gt; non-frequent itemsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;As far as I know, this may be considered a state-of-the-art algorithm, but I'd like to know about other proposed solutions. What other algorithms for FPM are considered &quot;state-of-the-art&quot;? What is the &lt;em&gt;intuition&lt;/em&gt;/&lt;em&gt;main-contribution&lt;/em&gt; of such algorithms?&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the FPGrowth algorithm still considered &quot;state of the art&quot; in frequent pattern mining? If not, what algorithm(s) may extract frequent itemsets from large datasets more efficiently?&lt;/p&gt;&#xA;" OwnerUserId="84" LastEditorUserId="84" LastEditDate="2014-07-13T03:05:46.660" LastActivityDate="2014-08-30T18:36:07.490" Title="Is FPGrowth still considered &quot;state of the art&quot; in frequent pattern mining?" Tags="|bigdata|data-mining|efficiency|state-of-the-art|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="731" PostTypeId="1" AcceptedAnswerId="732" CreationDate="2014-07-13T09:04:39.703" Score="60" ViewCount="27225" Body="&lt;p&gt;When I started with artificial neural networks (NN) I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even beat my score on random forest!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm seeking some very general or not so general advice on what should one do to make a NN start capturing trends in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For implementing NN I use Theano Stacked Auto Encoder with &lt;a href=&quot;https://github.com/lisa-lab/DeepLearningTutorials/blob/master/code/SdA.py&quot;&gt;the code from tutorial&lt;/a&gt; that works great (less than 5% error rate) for classifying the MNIST dataset. It is a multilayer perceptron, with softmax layer on top with each hidden later being pre-trained as autoencoder (fully described at &lt;a href=&quot;http://deeplearning.net/tutorial/deeplearning.pdf&quot;&gt;tutorial&lt;/a&gt;, chapter 8). There are ~50 input features and ~10 output classes. The NN has sigmoid neurons and all data are normalized to [0,1]. I tried lots of different configurations: number of hidden layers and neurons in them (100-&gt;100-&gt;100, 60-&gt;60-&gt;60, 60-&gt;30-&gt;15, etc.), different learning and pre-train rates, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the best thing I can get is a 20% error rate on the validation set and a 40% error rate on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, when I try to use Random Forest (from scikit-learn) I easily get a 12% error rate on the validation set and 25%(!) on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can it be that my deep NN with pre-training behaves so badly? What should I try? &lt;/p&gt;&#xA;" OwnerUserId="2471" LastEditorUserId="8820" LastEditDate="2017-05-01T07:46:22.633" LastActivityDate="2017-05-01T07:46:22.633" Title="How to fight underfitting in a deep neural net" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="732" PostTypeId="2" ParentId="731" CreationDate="2014-07-13T09:58:16.387" Score="38" Body="&lt;p&gt;The problem with deep networks is that they have lots of hyperparameters to tune and very small solution space. Thus, finding good ones is more like an art rather than engineering task. I would start with working example from tutorial and play around with its parameters to see how results change - this gives a good intuition (though not formal explanation) about dependencies between parameters and results (both - final and intermediate). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I found following papers very useful: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://yosinski.com/media/papers/Yosinski2012VisuallyDebuggingRestrictedBoltzmannMachine.pdf&quot;&gt;Visually Debugging Restricted Boltzmann Machine Training&#xA;with a 3D Example&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot;&gt;A Practical Guide to Training Restricted Boltzmann&#xA;Machines&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They both describe RBMs, but contain some insights on deep networks in general. For example, one of key points is that networks need to be debugged layer-wise - if previous layer doesn't provide good representation of features, further layers have almost no chance to fix it. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="1279" LastEditDate="2014-07-13T16:46:23.883" LastActivityDate="2014-07-13T16:46:23.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="733" PostTypeId="2" ParentId="704" CreationDate="2014-07-13T12:24:36.430" Score="5" Body="&lt;p&gt;Ask your grid administrator to add your local machine as a &quot;submit host&quot;, and install SGE (which we assume you are using, you don't actually say) so then you can &lt;code&gt;qsub&lt;/code&gt; from your machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use emacs, then you can edit on your HPC via emacs's &quot;tramp&quot; ssh-connection facilities, and keep a shell open in another emacs window. You don't say what editor/operating system you like to use. You can even configure emacs to save a file in two places, so you could save to your local machine for running tests and to the HPC file system simultaneously for big jobs.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-13T12:24:36.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="734" PostTypeId="2" ParentId="726" CreationDate="2014-07-14T12:08:13.373" Score="3" Body="&lt;p&gt;I'm not qualified to understand almost all of that paper, but, I might be able to give some intuitions from information theory that help you parse the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;||&lt;/code&gt; denotes the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;. It measures an information gain between two distributions. I suppose you could say it indicates the information in the real distribution of data that a model fails to capture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you see &quot;negative log&quot; think &lt;a href=&quot;http://en.wikipedia.org/wiki/Information_entropy&quot; rel=&quot;nofollow&quot;&gt;&quot;entropy&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first equation, think of it as &quot;-ln(...) - -ln(...)&quot;. This may help think of it as the difference of entropies. Likewise in the second, read it as &quot;D(...) + -ln(...)&quot;. This may help think of it as &quot;plus entropy&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the divergence definition, you'll see it is defined as the log of the ratio of the PDFs. This may help connect it to logs and negative logs. Look at the definition that writes it as cross-entropy minus entropy. Then this is all a question of differences of entropies of things which may be clearer.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-07-14T12:08:13.373" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="735" PostTypeId="2" ParentId="704" CreationDate="2014-07-14T13:06:05.523" Score="3" Body="&lt;p&gt;From your question's wording I assume that you have a local machine and a remote machine where you update two files &amp;mdash; a Python script and a Bash script.  Both files are under SVN control, and both machines have access to the same SVN server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sorry I do not have any advice specific to your grid system, but let me list some general points I have found important for any deployment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Keep production changes limited to configuration changes&lt;/strong&gt;. You write that you have to &quot;use the datasets' path on the server&quot;; this sounds to me like you have the paths hardcoded into your Python script.  This is not a good idea, precisely because you will need to change those paths in every other machine where you move the script to.  If you commit those changes back to SVN, then on your local machine you will have the remote paths, and on and on ...  (What if there are not only paths, but also passwords?  You should not have production passwords in an SVN server.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, keep paths and other setup informations in a &lt;code&gt;.ini&lt;/code&gt; file and use &lt;a href=&quot;https://docs.python.org/2/library/configparser.html&quot; rel=&quot;nofollow&quot;&gt;ConfigParser&lt;/a&gt; to read it, or use a &lt;code&gt;.json&lt;/code&gt; file and use the &lt;a href=&quot;https://docs.python.org/2/library/json.html&quot; rel=&quot;nofollow&quot;&gt;json&lt;/a&gt; module. Keep one copy of the file locally and one remotely, both under the same path, both without SVN control, and just keep the path to that configuration file in the Python script (or get it from the command line if you can't keep both configurations under the same path).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Keep configuration as small as possible&lt;/strong&gt;. Any configuration is a &quot;moving part&quot; of your application, and any system is more robust the less it has moving parts.  A good indicator of something that belongs into configuration is exactly that you have to edit it every time you move the code; things that have not needed editing can remain as constants in the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Automate your deployment&lt;/strong&gt;.  You can do it via a Bash script on your local machine; note that you can &lt;a href=&quot;http://malcontentcomics.com/systemsboy/2006/07/send-remote-commands-via-ssh.html&quot; rel=&quot;nofollow&quot;&gt;run any command on a remote machine&lt;/a&gt; through &lt;code&gt;ssh&lt;/code&gt;.  For instance:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svn export yourprojectpath /tmp/exportedproject&#xA;tar czf /tmp/yourproject.tgz /tmp/exportedproject&#xA;scp /tmp/myproject.tgz youruser@remotemachine:~/dev&#xA;&#xA;## Remote commands are in the right hand side, between ''&#xA;ssh youruser@remotemachine 'tar xzf ~/dev/yourproject.tgz'&#xA;ssh youruser@remotemachine 'qsub ~/dev/yourproject/script.py'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this to work, you need of course to have a &lt;a href=&quot;http://www.linuxproblem.org/art_9.html&quot; rel=&quot;nofollow&quot;&gt;passwordless login&lt;/a&gt;, based on public/private keys, set up between your local and the remote machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more than this, you can think of using Python's &lt;a href=&quot;http://www.fabfile.org&quot; rel=&quot;nofollow&quot;&gt;Fabric&lt;/a&gt; or the higher-level &lt;a href=&quot;https://github.com/sebastien/cuisine&quot; rel=&quot;nofollow&quot;&gt;cuisine&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2014-07-14T13:06:05.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="736" PostTypeId="1" AcceptedAnswerId="740" CreationDate="2014-07-14T13:53:28.437" Score="10" ViewCount="3623" Body="&lt;p&gt;I have a dataset which contains ~100,000 samples of 50 classes. I have been using SVM with an RBF kernel to train and predict new data. The problem though is the dataset is skewed towards different classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, Class 1 - 30 (~3% each), Class 31 - 45 (~0.6% each), Class 46 - 50 (~0.2% each)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see that the model tends to very rarely predict the classes which occur less frequent in the training set, even though the test set has the same class distribution as the training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that there are technique such as 'undersampling' where the majority class is scaled down to the minor class. However, is this applicable here where there are so many different classes? Are there other methods to help handle this case?&lt;/p&gt;&#xA;" OwnerUserId="802" LastActivityDate="2015-10-01T20:31:46.680" Title="Skewed multi-class data" Tags="|classification|svm|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="737" PostTypeId="2" ParentId="736" CreationDate="2014-07-14T15:58:42.517" Score="1" Body="&lt;p&gt;I have faced this problem many times while using SVM with Rbf kernel. Using Linear kernel instead of Rbf kernel solved my problem, but I dealt with lesser number of classes. The results were less skewed and more accurate with the linear kernel. Hope this solves your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: While I wrote original answer I was naive enough to not consider weighting the classes as one of them correctly answered. Also, while using rbf kernel its important to make sure that the penalty parameter or the 'C' value as per sklearn's svm module is too generic. I find that the default value of C=1 is too generic most of the time and I typically end up with a value of C=10000. Hope this helps others who get skewed results with svm(rbf) despite of having good distribution of classes in data.&lt;/p&gt;&#xA;" OwnerUserId="2485" LastEditorUserId="2485" LastEditDate="2015-10-01T20:31:46.680" LastActivityDate="2015-10-01T20:31:46.680" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="739" PostTypeId="1" AcceptedAnswerId="742" CreationDate="2014-07-14T19:02:01.670" Score="24" ViewCount="6502" Body="&lt;p&gt;I am an MSc student at the University of Edinburgh, specialized in machine learning and natural language processing. I had some practical courses focused on data mining, and others dealing with machine learning, bayesian statistics and graphical models. My background is a BSc in Computer Science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some software engineering and I learnt the basic concepts, such as design patterns, but I have never been involved in a large software development project. However, I had a data mining project in my MSc. My question is, if I want to go for a career as Data Scientist, should I apply for a graduate data scientist position first, or should I get a position as graduate software engineer first, maybe something related to data science, such as big data infrastructure or machine learning software development?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My concern is that I might need good software engineering skills for data science, and I am not sure if these can be obtained by working as a graduate data scientist directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, at the moment I like Data Mining, but what if I want to change my career to software engineering in the future? It might be difficult if I specialised so much in data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not been employed yet, so my knowledge is still limited. Any clarification or advice are welcome, as I am about to finish my MSc and I want to start applying for graduate positions in early October.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastEditorUserId="553" LastEditDate="2014-07-18T19:29:05.073" LastActivityDate="2014-07-18T19:29:05.073" Title="Starting my career as Data Scientist, is Software Engineering experience required?" Tags="|education|definitions|career|" AnswerCount="3" CommentCount="2" ClosedDate="2014-08-03T06:18:00.257" ContentLicense="CC BY-SA 3.0" />
  <row Id="740" PostTypeId="2" ParentId="736" CreationDate="2014-07-14T19:06:44.090" Score="5" Body="&lt;p&gt;I would suggest you to use libsvm, which already has adjustable class weights implemented in it. Rather than replicating the training samples, one modifies the C parameter for different classes in the SVM optimization. For example if your data has 2 classes, and the first class is only 10% of the data, you would choose class weights to be 10 and 1 for class 1 and 2 respectively. Therefore, margin violations of the first class would cost 10 times more than the margin violations for second class, and per-class accuracies would be more balanced.&lt;/p&gt;&#xA;" OwnerUserId="1350" LastActivityDate="2014-07-14T19:06:44.090" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="741" PostTypeId="2" ParentId="739" CreationDate="2014-07-15T04:39:11.670" Score="6" Body="&lt;p&gt;Absolutely. Keep your software skills sharp. You can do this in an academic program if you simply implement by yourself all the algorithms you learn about. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good selection of courses, btw. Consider getting an internship too.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-15T04:39:11.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="742" PostTypeId="2" ParentId="739" CreationDate="2014-07-15T06:19:31.820" Score="30" Body="&lt;p&gt;1) I think that there's no need to question whether your background is adequate for a career in data science. CS degree IMHO is &lt;strong&gt;more than enough&lt;/strong&gt; for data scientist from software engineering point of view. Having said that, theoretical knowledge is not very helpful without matching &lt;strong&gt;practical experience&lt;/strong&gt;, so I would definitely try to &lt;strong&gt;enrich&lt;/strong&gt; my experience through participating in &lt;em&gt;additional school projects, internships or open source projects&lt;/em&gt; (maybe ones, focused on data science / machine learning / artificial intelligence).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I believe your concern about &lt;strong&gt;focusing&lt;/strong&gt; on data science &lt;strong&gt;too early&lt;/strong&gt; is unfounded, as long as you will be practicing software engineering either as a part of your data science job, or additionally in your spare time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) I find the following &lt;strong&gt;definition of a data scientist&lt;/strong&gt; rather accurate and hope it will be helpful in your future career success:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A &lt;em&gt;data scientist&lt;/em&gt; is someone who is better at statistics than any&#xA;  software engineer and better at software engineering than any&#xA;  statistician.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;P.S. Today's &lt;strong&gt;enormous&lt;/strong&gt; number of various resources on data science topics is mind-blowing, but this &lt;strong&gt;open source curriculum for learning data science&lt;/strong&gt; might fill some gaps between your BSc/MSc respective curricula and reality of the data science career (or, at least, provide some direction for further research and maybe answer some of your concerns): &lt;a href=&quot;http://datasciencemasters.org&quot;&gt;http://datasciencemasters.org&lt;/a&gt;, or on GitHub: &lt;a href=&quot;https://github.com/datasciencemasters/go&quot;&gt;https://github.com/datasciencemasters/go&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-15T06:19:31.820" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="743" PostTypeId="2" ParentId="739" CreationDate="2014-07-15T09:30:02.183" Score="9" Body="&lt;p&gt;From the job ads I have seen, the answer depends: There are jobs which are more technical in nature (designing big data projects, doing some analysis) or the exact opposite (doing analysis, storage etc. is someone elses job).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would say that SOME software design skills are extremely useful , but you don't need the abillity to build a huge program in C# / Java or whatever.&#xA;Why I like some SW skills is simply that your code probably looks way better than code from someone who never programmed for the sake of programming. Most of the time, the latter code is very hard do understand / debug for outsiders. Also, sometimes your analysis needs to be integrated in a bigger program,an understand of the needs of the programms certainly helps.&lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-07-15T09:30:02.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="744" PostTypeId="1" AcceptedAnswerId="773" CreationDate="2014-07-15T21:30:11.600" Score="84" ViewCount="80324" Body="&lt;p&gt;It looks like the cosine similarity of two features is just their dot product scaled by the product of their magnitudes. When does cosine similarity make a better distance metric than the dot product? I.e. do the dot product and cosine similarity have different strengths or weaknesses in different situations?&lt;/p&gt;&#xA;" OwnerUserId="2507" LastActivityDate="2020-09-04T15:43:15.887" Title="Cosine similarity versus dot product as distance metrics" Tags="|classification|" AnswerCount="6" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="745" PostTypeId="1" CreationDate="2014-07-16T00:06:02.160" Score="3" ViewCount="2035" Body="&lt;p&gt;I'm trying to use ARMA/ARIMA with the &lt;a href=&quot;http://statsmodels.sourceforge.net/devel/tsa.html#descriptive-statistics-and-tests&quot; rel=&quot;nofollow noreferrer&quot;&gt;statsmodel Python package&lt;/a&gt;, in order to predict the gas consumption. I tried with &lt;a href=&quot;https://github.com/denadai2/Gas-consumption-outliers/blob/master/exportWeb.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;a dataset&lt;/a&gt; of this format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/uuEpZ.png&quot; alt=&quot;with this format&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using only the gas column.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas.tseries.offsets import *&#xA;&#xA;arma_mod20 = sm.tsa.ARMA(januaryFeb[['gas [m3]']], (5,3)).fit()&#xA;predict_sunspots = arma_mod20.predict('2012-01-13', '2012-01-14', dynamic=True)&#xA;ax = januaryFeb.ix['2012-01-13 00:00:00':'2012-01-15 22:00:00']['gas [m3]'].plot(figsize=(12,8))&#xA;ax = predict_sunspots.plot(ax=ax, style='r--', label='Dynamic Prediction');&#xA;ax.legend();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/JP3yg.png&quot; alt=&quot;result&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the prediction so bad?&lt;/p&gt;&#xA;" OwnerUserId="989" LastActivityDate="2014-07-16T18:26:54.233" Title="ARMA/ARIMA on energy forecasts timeseries: strange prediction" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="746" PostTypeId="2" ParentId="745" CreationDate="2014-07-16T03:32:16.270" Score="5" Body="&lt;p&gt;I'm not an expert on time series, but I have a &lt;strong&gt;general advice&lt;/strong&gt;: may I suggest you to try other packages (and various parameters) to see, if there are any differences in results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, unless you have to use &lt;code&gt;Python&lt;/code&gt;, I'd recommend to take a look at the &lt;code&gt;R&lt;/code&gt;'s extensive &lt;em&gt;ecosystem&lt;/em&gt; for &lt;strong&gt;time series analysis&lt;/strong&gt;: see &lt;a href=&quot;http://www.statmethods.net/advstats/timeseries.html&quot;&gt;http://www.statmethods.net/advstats/timeseries.html&lt;/a&gt; and &lt;a href=&quot;http://cran.r-project.org/web/views/TimeSeries.html&quot;&gt;http://cran.r-project.org/web/views/TimeSeries.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, you may want to check the standard &lt;code&gt;stats&lt;/code&gt; package (including functions &lt;code&gt;arima()&lt;/code&gt; and &lt;code&gt;arima0&lt;/code&gt;), as well as some other packages: &lt;code&gt;FitARMA&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/FitARMA&quot;&gt;http://cran.r-project.org/web/packages/FitARMA&lt;/a&gt;), &lt;code&gt;forecast&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/forecast&quot;&gt;http://cran.r-project.org/web/packages/forecast&lt;/a&gt;) and education-focused &lt;code&gt;fArma&lt;/code&gt; (cran.r-project.org/web/packages/fArma), to mention just a few. I hope this is helpful.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-16T03:32:16.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="747" PostTypeId="2" ParentId="52" CreationDate="2014-07-16T06:45:36.740" Score="5" Body="&lt;p&gt;About automatic cleaning: You really cannot clean data automatically, because the number of errors and the definition of an error is often dependent on the data. E.g.: Your column &quot;Income&quot; might contain negative values, which are an error - you have to do something about the cases. On the other hand a column &quot;monthly savings&quot; could reasonably contain negative values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such errors are highly domain dependent - so to find them, you must have domain knowledge, something at which humans excel, automated processes not so much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where you can and should automate is repeated projects. E.g. a report which has to produced monthly. If you spot errors, you should place some automated process which can spot  these kinds of errors in subsequent months, freeing your time. &lt;/p&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-07-16T06:45:36.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="748" PostTypeId="1" CreationDate="2014-07-16T07:47:48.603" Score="7" ViewCount="107" Body="&lt;p&gt;I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically, &quot;Why is mean revenue per user the best metric to make your decision on in a split test?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original question is here: &lt;a href=&quot;https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean&quot;&gt;https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since it was not well received/understood I simplified the problem to a discrete set of purchases and phrased it as a classical probability problem. That question is here: &lt;a href=&quot;https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn&quot;&gt;https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean may be the best metric for such a decision but I am not convinced. We often have a lot of prior information so a Bayesian method would likely improve our estimates. I realize that this is a difficult question but Data Scientists are doing such split tests everyday. &lt;/p&gt;&#xA;" OwnerUserId="2511" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-07-27T03:58:22.907" Title="Why use mean revenue in a split test?" Tags="|research|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="749" PostTypeId="1" CreationDate="2014-07-16T09:24:51.780" Score="37" ViewCount="20762" Body="&lt;p&gt;I am learning about matrix factorization for recommender systems and I am seeing the term &lt;code&gt;latent features&lt;/code&gt; occurring too frequently but I am unable to understand what it means. I know what a feature is but I don't understand the idea of latent features. Could please explain it? Or at least point me to a paper/place where I can read about it?&lt;/p&gt;&#xA;" OwnerUserId="728" LastEditorUserId="843" LastEditDate="2015-10-18T12:37:43.577" LastActivityDate="2020-08-16T10:15:48.737" Title="Meaning of latent features?" Tags="|machine-learning|data-mining|recommender-system|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="750" PostTypeId="1" AcceptedAnswerId="769" CreationDate="2014-07-16T09:49:15.933" Score="22" ViewCount="12282" Body="&lt;p&gt;I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and SVM. UCI letter recognition dataset with 20000 instances and 16 features is used, which I split in half for training and testing. I have experience with SVM so I quickly set its recognition error to 3.3%. After some experimentation what I got was:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UCI letter recognition:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RTrees - 5.3% &lt;/li&gt;&#xA;&lt;li&gt;Boost -  13% &lt;/li&gt;&#xA;&lt;li&gt;MLP -    7.9% &lt;/li&gt;&#xA;&lt;li&gt;kNN(k=3) -   6.5%  &lt;/li&gt;&#xA;&lt;li&gt;Bayes -  11.5%  &lt;/li&gt;&#xA;&lt;li&gt;SVM -    3.3%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Parameters used:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;RTrees - max_num_of_trees_in_the_forrest=200, max_depth=20,&#xA;min_sample_count=1&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Boost -  boost_type=REAL, weak_count=200, weight_trim_rate=0.95,&#xA;max_depth=7&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;MLP -    method=BACKPROP, param=0.001, max_iter=300 (default values - too&#xA;slow to experiment) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;kNN(k=3) -   k=3&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Bayes -  none&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;SVM -    RBF kernel, C=10, gamma=0.01&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After that I used same parameters and tested on Digits and MNIST datasets by extracting gradient features first (vector size 200 elements):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Digits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RTrees - 5.1%&lt;/li&gt;&#xA;&lt;li&gt;Boost -  23.4%&lt;/li&gt;&#xA;&lt;li&gt;MLP -    4.3%&lt;/li&gt;&#xA;&lt;li&gt;kNN(k=3) -   7.3%&lt;/li&gt;&#xA;&lt;li&gt;Bayes -  17.7%&lt;/li&gt;&#xA;&lt;li&gt;SVM -    4.2%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;MNIST:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RTrees - 1.4%&lt;/li&gt;&#xA;&lt;li&gt;Boost -  out of memory&lt;/li&gt;&#xA;&lt;li&gt;MLP -    1.0%&lt;/li&gt;&#xA;&lt;li&gt;kNN(k=3) -   1.2%&lt;/li&gt;&#xA;&lt;li&gt;Bayes -  34.33%&lt;/li&gt;&#xA;&lt;li&gt;SVM -    0.6%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am new to all classifiers except SVM and kNN, for these two I can say the results seem fine. What about others? I expected more from random trees, on MNIST kNN gives better accuracy, any ideas how to get it higher? Boost and Bayes give very low accuracy. In the end I'd like to use these classifiers to make a multiple classifier system. Any advice?&lt;/p&gt;&#xA;" OwnerUserId="1387" LastEditorUserId="1387" LastEditDate="2014-07-16T15:09:44.907" LastActivityDate="2020-08-06T09:38:11.873" Title="How to increase accuracy of classifiers?" Tags="|machine-learning|classification|svm|accuracy|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="751" PostTypeId="2" ParentId="744" CreationDate="2014-07-16T10:42:55.793" Score="12" Body="&lt;p&gt;You are right, cosine similarity has a lot of common with dot product of vectors. Indeed, it is a dot product, scaled by magnitude. And because of scaling it is normalized between 0 and 1. CS is preferable because it takes into account variability of data and features' relative frequencies. On the other hand, plain dot product is a little bit &quot;cheaper&quot; (in terms of complexity and implementation).&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-07-16T10:42:55.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="752" PostTypeId="2" ParentId="749" CreationDate="2014-07-16T11:07:56.467" Score="3" Body="&lt;p&gt;It seems to me that &lt;em&gt;latent features&lt;/em&gt; is a term used to describe criteria for &lt;strong&gt;classifying entities&lt;/strong&gt; by their structure, in other words, by &lt;em&gt;features&lt;/em&gt; (traits) they contain, instead of &lt;em&gt;classes&lt;/em&gt; they belong to. Meaning of the word &amp;quot;latent&amp;quot; here is most likely similar to its meaning in social sciences, where very popular term &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_variable&quot; rel=&quot;nofollow noreferrer&quot;&gt;latent variable&lt;/a&gt; means unobservable variable (concept).&lt;/p&gt;&#xA;&lt;p&gt;Section &amp;quot;Introduction&amp;quot; in &lt;a href=&quot;http://papers.nips.cc/paper/3846-nonparametric-latent-feature-models-for-link-prediction.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; provides a good explanation of latent features' &lt;strong&gt;meaning&lt;/strong&gt; and use in &lt;strong&gt;modeling&lt;/strong&gt; of social sciences phenomena.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-16T10:15:48.737" LastActivityDate="2020-08-16T10:15:48.737" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="753" PostTypeId="2" ParentId="224" CreationDate="2014-07-16T13:05:41.123" Score="2" Body="&lt;p&gt;I could try to explain you with words, but these slides explain it very well with pictures. Hope it helps.&#xA;&lt;a href=&quot;http://www.inf.ed.ac.uk/teaching/courses/mt/lectures/phrase-model.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.inf.ed.ac.uk/teaching/courses/mt/lectures/phrase-model.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note this slides correspond to the chapter 5 of &quot;Statistical Machine Translation&quot; by Philipp Koehn, highly recommended if you are working on machine translation, and it is easy to read.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2014-07-16T13:05:41.123" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="754" PostTypeId="2" ParentId="730" CreationDate="2014-07-16T14:22:38.677" Score="2" Body="&lt;p&gt;Most of the recent Frequent Pattern approaches that I've seen in the literature are based on optimizing FPGrowth. I have to admit, I haven't seen many developments within the literature in FPM in many years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Frequent_Pattern_Mining/The_FP-Growth_Algorithm&quot; rel=&quot;nofollow&quot;&gt;This wikibook&lt;/a&gt; highlights many of the variants on FPGrowth that are out there.&lt;/p&gt;&#xA;" OwnerUserId="2513" LastActivityDate="2014-07-16T14:22:38.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="755" PostTypeId="2" ParentId="750" CreationDate="2014-07-16T17:34:24.880" Score="5" Body="&lt;p&gt;&lt;em&gt;I expected more from random trees&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;With random forests, typically for N features, sqrt(N) features are used for each decision tree construction. Since in your case &lt;em&gt;N&lt;/em&gt;=20, you could try setting &lt;em&gt;max_depth&lt;/em&gt; (the number of sub-features to construct each decision tree) to 5.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of decision trees, linear models have been proposed and evaluated as base estimators in random forests, in particular multinomial logistic regression and naive Bayes. This might improve your accuracy.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;On MNIST kNN gives better accuracy, any ideas how to get it higher?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try with a higher value of &lt;em&gt;K&lt;/em&gt; (say 5 or 7). A higher value of K would give you more supportive evidence about the class label of a point.&lt;/li&gt;&#xA;&lt;li&gt;You could run PCA or Fisher's Linear Discriminant Analysis before running k-nearest neighbour. By this you could potentially get rid of correlated features while computing distances between the points, and hence your k neighbours would be more robust.&lt;/li&gt;&#xA;&lt;li&gt;Try different K values for different points based on the variance in the distances between the K neighbours.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-07-16T17:34:24.880" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="756" PostTypeId="2" ParentId="749" CreationDate="2014-07-16T18:15:42.343" Score="34" Body="&lt;p&gt;At the expense of over-simplication, latent features are 'hidden' features to distinguish them from observed features. Latent features are computed from observed features using matrix factorization. An example would be text document analysis. 'words' extracted from the documents are features. If you factorize the data of words you can find 'topics', where 'topic' is a group of words with semantic relevance. Low-rank matrix factorization maps several rows (observed features) to a smaller set of rows (latent features).&#xA;To elaborate, the document could have observed features (words) like [sail-boat, schooner, yatch, steamer, cruiser] which would 'factorize' to latent feature (topic) like 'ship' and 'boat'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[sail-boat, schooner, yatch, steamer, cruiser, ...] -&gt; [ship, boat]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The underlying idea is that latent features are semantically relevant 'aggregates' of observered features. When you have large-scale, high-dimensional, and noisy observered features, it makes sense to build your classifier on latent features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a of course a simplified description to elucidate the concept. You can read the details on Latent Dirichlet Allocation (LDA) or probabilistic Latent Semantic Analysis (pLSA) models for an accurate description.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2014-07-16T18:15:42.343" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="757" PostTypeId="2" ParentId="745" CreationDate="2014-07-16T18:26:54.233" Score="3" Body="&lt;p&gt;Gas usage has a daily cycle but there are also secondary weekly and annual cycles that the ARIMA may not be able to capture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a very noticeable difference between the weekday and Saturday data. Try creating a subset of the data for each day of the week or splitting the data into weekday and weekend and applying the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can obtain temperature data for the same period check if there is a correlation between the temperature and gas usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @Aleksandr Blekh said R does have good packages for ARIMA models&lt;/p&gt;&#xA;" OwnerUserId="325" LastActivityDate="2014-07-16T18:26:54.233" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="758" PostTypeId="1" AcceptedAnswerId="759" CreationDate="2014-07-16T20:09:08.640" Score="63" ViewCount="10181" Body="&lt;p&gt;I am working on a data science project using Python.&#xA;The project has several stages.&#xA;Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and creating another data set.&#xA;I store the code in git, so that part is covered.&#xA;I would like to hear about:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tools for data version control.&lt;/li&gt;&#xA;&lt;li&gt;Tools enabling to reproduce stages and experiments.&lt;/li&gt;&#xA;&lt;li&gt;Protocol and suggested directory structure for such a project.&lt;/li&gt;&#xA;&lt;li&gt;Automated build/run tools.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="895" LastEditorUserId="4647" LastEditDate="2015-08-21T16:37:14.087" LastActivityDate="2023-12-15T14:23:56.947" Title="Tools and protocol for reproducible data science using Python" Tags="|python|tools|version-control|" AnswerCount="9" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="759" PostTypeId="2" ParentId="758" CreationDate="2014-07-17T06:02:04.813" Score="51" Body="&lt;p&gt;The topic of &lt;em&gt;reproducible research&lt;/em&gt; (RR) is &lt;strong&gt;very popular&lt;/strong&gt; today and, consequently, is &lt;strong&gt;huge&lt;/strong&gt;, but I hope that my answer will be &lt;strong&gt;comprehensive enough&lt;/strong&gt; as an answer and will provide enough information for &lt;strong&gt;further research&lt;/strong&gt;, should you decide to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more &lt;strong&gt;universal tools&lt;/strong&gt; (you never know for sure what programming languages and computing environments you will be working with in the future). Having said that, let's take a look what tools are available per your list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Tools for data version control&lt;/strong&gt;. Unless you plan to work with (very) &lt;em&gt;big data&lt;/em&gt;, I guess, it would make sense to use the same &lt;code&gt;git&lt;/code&gt;, which you use for source code version control. The infrastructure is already there. Even if your files are binary and big, this advice might be helpful: &lt;a href=&quot;https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git&quot;&gt;https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Tools for managing RR workflows and experiments&lt;/strong&gt;. Here's a list of most popular tools in this category, to the best of my knowledge (in the descending order of popularity):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Taverna Workflow Management System&lt;/em&gt; (&lt;a href=&quot;http://www.taverna.org.uk&quot; rel=&quot;noreferrer&quot;&gt;http://www.taverna.org.uk&lt;/a&gt;) - very solid, if a little too complex, set of tools. The major tool is a Java-based desktop software. However, it is compatible with online workflow repository portal &lt;em&gt;myExperiment&lt;/em&gt; (&lt;a href=&quot;http://www.myexperiment.org&quot; rel=&quot;noreferrer&quot;&gt;http://www.myexperiment.org&lt;/a&gt;), where user can store and share their RR workflows. Web-based RR portal, fully compatible with &lt;em&gt;Taverna&lt;/em&gt; is called &lt;em&gt;Taverna Online&lt;/em&gt;, but it is being developed and maintained by totally different organization in Russia (referred there to as &lt;em&gt;OnlineHPC&lt;/em&gt;: &lt;a href=&quot;http://onlinehpc.com&quot; rel=&quot;noreferrer&quot;&gt;http://onlinehpc.com&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;The Kepler Project&lt;/em&gt; (&lt;a href=&quot;https://kepler-project.org&quot; rel=&quot;noreferrer&quot;&gt;https://kepler-project.org&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;VisTrails&lt;/em&gt; (&lt;a href=&quot;http://vistrails.org&quot; rel=&quot;noreferrer&quot;&gt;http://vistrails.org&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Madagascar&lt;/em&gt; (&lt;a href=&quot;http://www.reproducibility.org&quot; rel=&quot;noreferrer&quot;&gt;http://www.reproducibility.org&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EXAMPLE&lt;/strong&gt;. Here's an interesting article on scientific workflows with an example of the &lt;strong&gt;real&lt;/strong&gt; workflow design and data analysis, based on using &lt;em&gt;Kepler&lt;/em&gt; and &lt;em&gt;myExperiment&lt;/em&gt; projects: &lt;a href=&quot;http://f1000research.com/articles/3-110/v1&quot; rel=&quot;noreferrer&quot;&gt;http://f1000research.com/articles/3-110/v1&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many RR tools that implement &lt;em&gt;literate programming&lt;/em&gt; paradigm, exemplified by &lt;code&gt;LaTeX&lt;/code&gt; software family. Tools that help in report generation and presentation is also a large category, where &lt;code&gt;Sweave&lt;/code&gt; and &lt;code&gt;knitr&lt;/code&gt; are probably the most well-known ones. &lt;code&gt;Sweave&lt;/code&gt; is a tool, focused on R, but it can be integrated with Python-based projects, albeit with some additional effort (&lt;a href=&quot;https://stackoverflow.com/questions/2161152/sweave-for-python&quot;&gt;https://stackoverflow.com/questions/2161152/sweave-for-python&lt;/a&gt;). I think that &lt;code&gt;knitr&lt;/code&gt; might be a better option, as it's modern, has extensive support by popular tools (such as &lt;code&gt;RStudio&lt;/code&gt;) and is language-neutral (&lt;a href=&quot;http://yihui.name/knitr/demo/engines&quot; rel=&quot;noreferrer&quot;&gt;http://yihui.name/knitr/demo/engines&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;strong&gt;Protocol and suggested directory structure&lt;/strong&gt;. If I understood correctly what you implied by using term &lt;em&gt;protocol&lt;/em&gt; (&lt;em&gt;workflow&lt;/em&gt;), generally I think that standard RR data analysis workflow consists of the following sequential phases: &lt;em&gt;data collection&lt;/em&gt; =&gt; &lt;em&gt;data preparation&lt;/em&gt; (cleaning, transformation, merging, sampling) =&gt; &lt;em&gt;data analysis&lt;/em&gt; =&gt; &lt;em&gt;presentation of results&lt;/em&gt; (generating reports and/or presentations). Nevertheless, every workflow is project-specific and, thus, some specific tasks might require adding additional steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sample directory structure, you may take a look at documentation for R package &lt;code&gt;ProjectTemplate&lt;/code&gt; (&lt;a href=&quot;http://projecttemplate.net&quot; rel=&quot;noreferrer&quot;&gt;http://projecttemplate.net&lt;/a&gt;), as an attempt to automate data analysis workflows and projects:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/0B2vo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) &lt;strong&gt;Automated build/run tools&lt;/strong&gt;. Since my answer is focused on universal (language-neutral) RR tools, the most popular tools is &lt;code&gt;make&lt;/code&gt;. Read the following article for some reasons to use &lt;code&gt;make&lt;/code&gt; as the preferred RR workflow automation tool: &lt;a href=&quot;http://bost.ocks.org/mike/make&quot; rel=&quot;noreferrer&quot;&gt;http://bost.ocks.org/mike/make&lt;/a&gt;. Certainly, there are other &lt;strong&gt;similar&lt;/strong&gt; tools, which either improve some aspects of &lt;code&gt;make&lt;/code&gt;, or add some additional features. For example: &lt;code&gt;ant&lt;/code&gt; (officially, Apache Ant: &lt;a href=&quot;http://ant.apache.org&quot; rel=&quot;noreferrer&quot;&gt;http://ant.apache.org&lt;/a&gt;), &lt;code&gt;Maven&lt;/code&gt; (&quot;next generation &lt;code&gt;ant&lt;/code&gt;&quot;: &lt;a href=&quot;http://maven.apache.org&quot; rel=&quot;noreferrer&quot;&gt;http://maven.apache.org&lt;/a&gt;), &lt;code&gt;rake&lt;/code&gt; (&lt;a href=&quot;https://github.com/ruby/rake&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/ruby/rake&lt;/a&gt;), &lt;code&gt;Makepp&lt;/code&gt; (&lt;a href=&quot;http://makepp.sourceforge.net&quot; rel=&quot;noreferrer&quot;&gt;http://makepp.sourceforge.net&lt;/a&gt;). For a comprehensive list of such tools, see Wikipedia: &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_build_automation_software&quot; rel=&quot;noreferrer&quot;&gt;http://en.wikipedia.org/wiki/List_of_build_automation_software&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-07-17T11:42:25.753" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="760" PostTypeId="1" CreationDate="2014-07-17T09:26:11.833" Score="2" ViewCount="441" Body="&lt;p&gt;I am given a time series data vector (ordered by months and years),which contains only &lt;code&gt;0&lt;/code&gt;s and &lt;code&gt;1&lt;/code&gt;s. &lt;code&gt;1&lt;/code&gt; s represent a person changes his job at a particular a month. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt; What model can i use to determine model how frequently this person change his job ? In addition, this model should be able to predict the probability of this person changing his in the next 6 months.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A poisson process ? (I have studied poisson process before however I have no idea when and how to apply it). Any assumptions that data need to meet before applying the poisson process ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would love to gather more information on how to model something like this. Thanks&lt;/p&gt;&#xA;" OwnerUserId="1315" LastActivityDate="2018-10-16T20:20:17.820" Title="Given time series data, how to model the frequency of someone changes his job?" Tags="|data-mining|time-series|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="761" PostTypeId="1" AcceptedAnswerId="764" CreationDate="2014-07-17T09:50:41.437" Score="67" ViewCount="105087" Body="&lt;p&gt;What is the right approach and clustering algorithm for geolocation clustering?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the following code to cluster geolocation coordinates:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from scipy.cluster.vq import kmeans2, whiten&#xA;&#xA;coordinates= np.array([&#xA;           [lat, long],&#xA;           [lat, long],&#xA;            ...&#xA;           [lat, long]&#xA;           ])&#xA;x, y = kmeans2(whiten(coordinates), 3, iter = 20)  &#xA;plt.scatter(coordinates[:,0], coordinates[:,1], c=y);&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it right to use K-means for geolocation clustering, as it uses Euclidean distance, and not &lt;a href=&quot;https://en.wikipedia.org/wiki/Haversine_formula&quot; rel=&quot;noreferrer&quot;&gt;Haversine formula&lt;/a&gt; as a distance function?&lt;/p&gt;&#xA;" OwnerUserId="2533" LastEditorUserId="31513" LastEditDate="2017-05-12T08:39:00.347" LastActivityDate="2023-01-12T18:50:07.947" Title="Clustering geo location coordinates (lat,long pairs)" Tags="|machine-learning|python|clustering|k-means|geospatial|" AnswerCount="9" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="762" PostTypeId="1" AcceptedAnswerId="766" CreationDate="2014-07-17T10:04:29.797" Score="11" ViewCount="6026" Body="&lt;p&gt;t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met.&lt;br /&gt;&#xA;The creators of t-SNE suggests to use KL divergence as a performance criterion for the visualizations:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;you can compare the Kullback-Leibler divergences that t-SNE reports. It is perfectly fine to run t-SNE ten times, and select the solution with the lowest KL divergence [2]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I tried two implementations of t-SNE:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;python&lt;/strong&gt;: &lt;code&gt;sklearn.manifold.TSNE()&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;R&lt;/strong&gt;: &lt;code&gt;tsne&lt;/code&gt;, from &lt;code&gt;library(tsne)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Both these implementations, when verbosity is set, print the error (Kullback-Leibler divergence) for each iteration. However, they don't allow the user to get this information, which looks a bit strange to me.&lt;/p&gt;&#xA;&lt;p&gt;For example, the code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.manifold import TSNE&#xA;X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])&#xA;model = TSNE(n_components=2, verbose=2, n_iter=200)&#xA;t = model.fit_transform(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;produces:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[t-SNE] Computing pairwise distances...&#xA;[t-SNE] Computed conditional probabilities for sample 4 / 4&#xA;[t-SNE] Mean sigma: 1125899906842624.000000&#xA;[t-SNE] Iteration 10: error = 6.7213750, gradient norm = 0.0012028&#xA;[t-SNE] Iteration 20: error = 6.7192064, gradient norm = 0.0012062&#xA;[t-SNE] Iteration 30: error = 6.7178683, gradient norm = 0.0012114&#xA;...&#xA;[t-SNE] Error after 200 iterations: 0.270186&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now, as far as I understand, &lt;strong&gt;0.270186&lt;/strong&gt; should be the KL divergence. However I cannot get this information, neither from &lt;strong&gt;model&lt;/strong&gt; nor from &lt;strong&gt;t&lt;/strong&gt; (which is a simple &lt;code&gt;numpy.ndarray&lt;/code&gt;).&lt;/p&gt;&#xA;&lt;p&gt;To solve this problem I could:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate KL divergence by my self,&lt;/li&gt;&#xA;&lt;li&gt;Do something nasty in python for capturing and parsing &lt;code&gt;TSNE()&lt;/code&gt; function's output [3].&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;However:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;would be quite stupid to re-calculate KL divergence, when &lt;code&gt;TSNE()&lt;/code&gt; has already computed it,&lt;/li&gt;&#xA;&lt;li&gt;would be a bit unusual in terms of code.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Do you have any other suggestion? Is there a standard way to get this information using this library?&lt;/p&gt;&#xA;&lt;p&gt;I mentioned I tried &lt;em&gt;R&lt;/em&gt;'s tsne library, but I'd prefer the answers to focus on the &lt;em&gt;python&lt;/em&gt; sklearn implementation.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;References&lt;/p&gt;&#xA;&lt;p&gt;[1] &lt;a href=&quot;http://nbviewer.ipython.org/urls/gist.githubusercontent.com/AlexanderFabisch/1a0c648de22eff4a2a3e/raw/59d5bc5ed8f8bfd9ff1f7faa749d1b095aa97d5a/t-SNE.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://nbviewer.ipython.org/urls/gist.githubusercontent.com/AlexanderFabisch/1a0c648de22eff4a2a3e/raw/59d5bc5ed8f8bfd9ff1f7faa749d1b095aa97d5a/t-SNE.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[2] &lt;a href=&quot;http://homepage.tudelft.nl/19j49/t-SNE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://homepage.tudelft.nl/19j49/t-SNE.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[3] &lt;a href=&quot;https://stackoverflow.com/questions/16571150/how-to-capture-stdout-output-from-a-python-function-call&quot;&gt;https://stackoverflow.com/questions/16571150/how-to-capture-stdout-output-from-a-python-function-call&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="131" LastEditorUserId="98307" LastEditDate="2020-08-02T14:02:51.077" LastActivityDate="2020-08-02T14:02:51.077" Title="t-SNE Python implementation: Kullback-Leibler divergence" Tags="|machine-learning|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="763" PostTypeId="2" ParentId="760" CreationDate="2014-07-17T10:26:45.547" Score="2" Body="&lt;p&gt;A simple and perhaps somewhat naive approach would be to assume that a person changes jobs at a constant rate and that previous job changes have no influence on future ones. Under these assumptions you could model the job changes as a Poisson process and estimate the rate parameter using MLE (&lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_process&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Poisson_process&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Poisson_distribution&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course one should explore how well these assumptions hold in the data. To do this, you could study whether or not job changes are independent of one another by computing the correlation between events at various lags (&lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Correlation&lt;/a&gt;). You could also plot the distribution of time between job change events. If the process is Poisson-like then you should observe little to no correlation between events at any number of lags and the distribution of time between job change events should be exponentially distributed (&lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Exponential_distribution&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="1329" LastActivityDate="2014-07-17T10:26:45.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="764" PostTypeId="2" ParentId="761" CreationDate="2014-07-17T12:34:11.397" Score="11" Body="&lt;p&gt;K-means should be right in this case. Since k-means tries to group based solely on euclidean distance between objects you will get back clusters of locations that are close to each other.&lt;/p&gt;&#xA;&lt;p&gt;To find the optimal number of clusters you can try making an 'elbow' plot of the within group sum of square distance. &lt;a href=&quot;http://nbviewer.ipython.org/github/nborwankar/LearnDataScience/blob/master/notebooks/D3.%20K-Means%20Clustering%20Analysis.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; may be helpful&lt;/p&gt;&#xA;" OwnerUserId="802" LastEditorUserId="98307" LastEditDate="2020-08-02T14:02:48.237" LastActivityDate="2020-08-02T14:02:48.237" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="766" PostTypeId="2" ParentId="762" CreationDate="2014-07-17T14:07:12.643" Score="4" Body="&lt;p&gt;The TSNE source in scikit-learn is in pure Python. Fit &lt;code&gt;fit_transform()&lt;/code&gt; method is actually calling a private &lt;code&gt;_fit()&lt;/code&gt; function which then calls a private &lt;code&gt;_tsne()&lt;/code&gt; function. That &lt;code&gt;_tsne()&lt;/code&gt; function has a local variable &lt;code&gt;error&lt;/code&gt; which is printed out at the end of the fit. Seems like you could pretty easily change one or two lines of source code to have that value returned to &lt;code&gt;fit_transform()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="159" LastActivityDate="2014-07-17T14:07:12.643" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="767" PostTypeId="2" ParentId="758" CreationDate="2014-07-17T14:28:38.207" Score="20" Body="&lt;p&gt;The best reproducibility tool is to make a log of your actions, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;experiment/input ; expected ; observation/output ; current hypothesis and if supported or rejected&#xA;exp1 ; expected1 ; obs1 ; some fancy hypothesis, supported&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This can be written down on a paper, but, if your experiments fit in a computational framework, you can use computational tools to partly or completely automate that logging process (particularly by helping you track the input datasets which can be huge, and the output figures).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A great reproducibility tool for Python with a low learning curve is of course &lt;a href=&quot;http://ipython.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;IPython/Jupyter Notebook&lt;/strong&gt;&lt;/a&gt; (don't forget the &lt;a href=&quot;https://damontallen.github.io/IPython-quick-ref-sheets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;%logon and %logstart&lt;/a&gt; magics). Tip: to make sure your notebook is reproducible, restart the kernel and try to run all cells from top to bottom (button Run All Cells): if it works, then save everything in an archive file (&quot;freezing&quot;), else, notably if you need to run cells in a non linear and non sequential and &lt;strong&gt;non obvious&lt;/strong&gt; fashion to avoid errors, you need to rework a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another great tool that is very recent (2015) is &lt;a href=&quot;https://github.com/recipy/recipy&quot; rel=&quot;nofollow noreferrer&quot;&gt;recipy&lt;/a&gt;, which is very like sumatra (see below), but made specifically for Python. I don't know if it works with Jupyter Notebooks, but I know the author frequently uses them so I guess that if it's not currently supported, it will be in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Git_(software)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Git&lt;/a&gt; is also awesome, and it's not tied to Python. It will help you not only to keep a history of all your experiments, code, datasets, figures, etc. but also provide you with tools to maintain (&lt;a href=&quot;http://www.philandstuff.com/2014/02/09/git-pickaxe.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;git pickaxe&lt;/a&gt;), collaborate (&lt;a href=&quot;http://jfire.io/blog/2012/03/07/code-archaeology-with-git/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blame&lt;/a&gt;) and debug (&lt;a href=&quot;http://blogs.perl.org/users/ovid/2014/07/making-git-bisect-more-useful.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;git&lt;/a&gt;-&lt;a href=&quot;https://lwn.net/Articles/317154/&quot; rel=&quot;nofollow noreferrer&quot;&gt;bisect&lt;/a&gt;) using a scientific method of debugging (called &lt;a href=&quot;https://en.wikipedia.org/wiki/Delta_Debugging&quot; rel=&quot;nofollow noreferrer&quot;&gt;delta debugging&lt;/a&gt;). &lt;a href=&quot;https://matthew-brett.github.io/pydagogue/curious_git.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a story&lt;/a&gt; of a fictional researcher trying to make his own experiments logging system, until it ends up being a facsimile of Git.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another general tool working with any language (with a Python API on &lt;a href=&quot;https://pypi.python.org/pypi/Sumatra&quot; rel=&quot;nofollow noreferrer&quot;&gt;pypi&lt;/a&gt;) is &lt;a href=&quot;http://neuralensemble.org/sumatra/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Sumatra&lt;/strong&gt;&lt;/a&gt;, which is specifically designed to help you do &lt;strong&gt;replicable&lt;/strong&gt; research (&lt;em&gt;replicable&lt;/em&gt; aims to produce the same results given the exact same code and softwares, whereas &lt;em&gt;reproducibility&lt;/em&gt; aims to produce the same results given any medium, which is a lot harder and time consuming and not automatable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how Sumatra works: for each experiment that you conduct through Sumatra, this software will act like a &quot;save game state&quot; often found in videogames. More precisely, it will will save:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;all the parameters you provided;&lt;/li&gt;&#xA;&lt;li&gt;the exact sourcecode state of your whole experimental application and config files;&lt;/li&gt;&#xA;&lt;li&gt;the output/plots/results and also any file produced by your experimental application.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It will then construct a database with the timestamp and other metadatas for each of your experiments, that you can later crawl using the webGUI. Since Sumatra saved the full state of your application for a specific experiment at one specific point in time, you can restore the code that produced a specific result at any moment you want, thus you have replicable research at a low cost (except for storage if you work on huge datasets, but you can configure exceptions if you don't want to save everything everytime).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another awesome tool is &lt;a href=&quot;https://en.wikipedia.org/wiki/Zeitgeist_(free_software)&quot; rel=&quot;nofollow noreferrer&quot;&gt;GNOME's Zeitgeist&lt;/a&gt; (previously coded in Python but now ported to Vala), an all-compassing action journaling system, which records everything you do and it can use machine learning to summarize for a time period you want the relationship between items based on similarity and usage patterns, eg answering questions like &lt;em&gt;&quot;What was most relevant to me, while I was working on project X, for a month last year?&quot;&lt;/em&gt;. Interestingly, &lt;a href=&quot;http://zim-wiki.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zim Desktop Wiki&lt;/a&gt;, a note-taking app similar to Evernote, has a plugin to work with Zeitgeist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, you can use either Git or Sumatra or any other software you want, they will provide you with about the same replicability power, but Sumatra is specifically tailored for scientific research so it provides a few fancy tools like a web GUI to crawl your results, while Git is more tailored towards code maintenance (but it has debugging tools like git-bisect so if your experiments involve codes, it may actually be better). Or of course you can use both!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/EDIT: &lt;a href=&quot;https://datascience.stackexchange.com/a/775&quot;&gt;dsign&lt;/a&gt; touched a very important point here: the replicability of your setup is as important as the replicability of your application. In other words, you should at least provide a &lt;strong&gt;full list of the libraries and compilers&lt;/strong&gt; you used along with their exact &lt;strong&gt;versions&lt;/strong&gt; and the details of your &lt;strong&gt;platform&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, in scientific computing with Python, I have found that packaging an application along with the libraries is just too painful, thus I now just use an all-in-one scientific python package such as &lt;a href=&quot;https://store.continuum.io/cshop/anaconda/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anaconda&lt;/a&gt; (with the great package manager &lt;a href=&quot;http://www.continuum.io/blog/conda&quot; rel=&quot;nofollow noreferrer&quot;&gt;conda&lt;/a&gt;), and just advise users to use the same package. Another solution could be to provide a script to automatically generate a &lt;a href=&quot;http://docs.python-guide.org/en/latest/dev/virtualenvs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;virtualenv&lt;/a&gt;, or to package everything using the commercial &lt;a href=&quot;https://datascience.stackexchange.com/a/775&quot;&gt;Docker application as cited by dsign&lt;/a&gt; or the opensource &lt;a href=&quot;http://www.vagrantup.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vagrant&lt;/a&gt; (with for example &lt;a href=&quot;http://deeplearning.net/software/pylearn2/#other-methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;pylearn2-in-a-box&lt;/a&gt; which use Vagrant to produce an easily redistributable virtual environment package).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, to really ensure that you have a fully working environment everytime you need, you can make a virtual machine (see VirtualBox), and you can even save the state of the machine (snapshot) with your experiment ready to run inside. Then you can just share this virtual machine with everything included so that anyone can replicate your experiment with your exact setup. This is probably the best way to replicate a software based experiment. Containers might be a more lightweight alternative, but they do not include the whole environment, so that the replication fidelity will be less robust.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/EDIT2: Here's a &lt;a href=&quot;https://www.udacity.com/course/viewer#!/c-cs259/l-48648797/m-48680672&quot; rel=&quot;nofollow noreferrer&quot;&gt;great video&lt;/a&gt; summarizing (for debugging but this can also be applied to research) what is fundamental to do reproducible research: logging your experiments and each other steps of the scientific method, a sort of &lt;em&gt;&quot;explicit experimenting&quot;&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2544" LastEditorUserId="2544" LastEditDate="2018-10-18T03:48:09.637" LastActivityDate="2018-10-18T03:48:09.637" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="768" PostTypeId="2" ParentId="758" CreationDate="2014-07-17T15:19:32.237" Score="25" Body="&lt;p&gt;Since I started doing research in academia I was constantly looking for a satisfactory workflow.&#xA;I think that I finally found something I am happy with:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Put everything under version control, e.g., Git:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;For hobby research projects I use GitHub, for research at work I use the private GitLab server that is provided by our university. I also keep my datasets there.&lt;/p&gt;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;I do most of my analyses along with the documentation on IPython notebooks. It is very organized (for me) to have the code, the plots, and the discussion/conclusion all in one document&#xA;If I am running larger scripts, I would usually put them into separate script .py files, but I would still execute them from the IPython notebook via the %run magic to add information about the purpose, outcome, and other parameters.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I have written a small cell-magic extension for IPython and IPython notebooks, called &amp;quot;watermark&amp;quot; that I use to conveniently create time stamps and keep track of the different package versions I used and also Git hashs&lt;/p&gt;&#xA;&lt;p&gt;For example&lt;/p&gt;&#xA;&lt;br&gt;&#xA;&lt;pre&gt;&lt;code&gt;%watermark&#xA;&#xA;29/06/2014 01:19:10&#xA;&#xA;CPython 3.4.1&#xA;IPython 2.1.0&#xA;&#xA;compiler   : GCC 4.2.1 (Apple Inc. build 5577)&#xA;system     : Darwin&#xA;release    : 13.2.0&#xA;machine    : x86_64&#xA;processor  : i386&#xA;CPU cores  : 2&#xA;interpreter: 64bit&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;br&gt;&#xA;&lt;pre&gt;&lt;code&gt;%watermark -d -t&#xA;&#xA;29/06/2014 01:19:11 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;br&gt;&#xA;&lt;pre&gt;&lt;code&gt;%watermark -v -m -p numpy,scipy&#xA;&#xA;CPython 3.4.1&#xA;IPython 2.1.0&#xA;&#xA;numpy 1.8.1&#xA;scipy 0.14.0&#xA;&#xA;compiler   : GCC 4.2.1 (Apple Inc. build 5577)&#xA;system     : Darwin&#xA;release    : 13.2.0&#xA;machine    : x86_64&#xA;processor  : i386&#xA;CPU cores  : 2&#xA;interpreter: 64bit&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;For more info, see the &lt;a href=&quot;https://github.com/rasbt/watermark&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github repository here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user2556" LastEditorUserId="156922" LastEditDate="2023-12-15T14:23:56.947" LastActivityDate="2023-12-15T14:23:56.947" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="769" PostTypeId="2" ParentId="750" CreationDate="2014-07-17T15:28:25.940" Score="11" Body="&lt;p&gt;&lt;strong&gt;Dimensionality Reduction&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Another important procedure is to compare the error rates on training and test dataset to see if you are overfitting (due to the &amp;quot;curse of dimensionality&amp;quot;). E.g., if your error rate on the test dataset is much larger than the error on the training data set, this would be one indicator.&lt;br /&gt;&#xA;In this case, you could try dimensionality reduction techniques, such as PCA or LDA.&lt;/p&gt;&#xA;&lt;p&gt;If you are interested, I have written about PCA, LDA and some other techniques &lt;a href=&quot;http://sebastianraschka.com/index.html#machine_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and in my GitHub repo &lt;a href=&quot;https://github.com/rasbt/pattern_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Cross validation&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also you may want to take a look at cross-validation techniques in order to evaluate the performance of your classifiers in a more objective manner&lt;/p&gt;&#xA;" OwnerDisplayName="user2556" LastEditorUserId="98307" LastEditDate="2020-08-06T09:38:11.873" LastActivityDate="2020-08-06T09:38:11.873" CommentCount="14" ContentLicense="CC BY-SA 4.0" />
  <row Id="770" PostTypeId="2" ParentId="736" CreationDate="2014-07-17T15:35:17.487" Score="2" Body="&lt;p&gt;I am not an export in using SVMs, but usually (if you are using a machine learning library like Python's &lt;code&gt;scikit-learn&lt;/code&gt; or R's &lt;code&gt;libsvm&lt;/code&gt;, there is the &lt;code&gt;class_weight&lt;/code&gt; parameter, or &lt;code&gt;class.weights&lt;/code&gt;, respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if you'd use a Bayes classifier, you would take this &quot;skew&quot; into account via the &quot;prior (class) probabilities&quot; P(&amp;omega;&lt;sub&gt;j&lt;/sub&gt;)&lt;/p&gt;&#xA;" OwnerDisplayName="user2556" LastActivityDate="2014-07-17T15:35:17.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="771" PostTypeId="2" ParentId="694" CreationDate="2014-07-17T18:25:44.683" Score="6" Body="&lt;p&gt;From what I heard, Pylearn2 might be currently the library of choice for most people. This reminds me of a recent blog post a few month ago that lists all the different machine learning libraries with a short explanation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cbinsights.com/blog/python-tools-machine-learning&quot;&gt;https://www.cbinsights.com/blog/python-tools-machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The section you might be interested in here would be &quot;Deep Learning&quot;. About Pylearn2, he writes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;PyLearn2&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;There is another library built on top of Theano, called PyLearn2 which&#xA;  brings modularity and configurability to Theano where you could create&#xA;  your neural network through different configuration files so that it&#xA;  would be easier to experiment different parameters. Arguably, it&#xA;  provides more modularity by separating the parameters and properties&#xA;  of neural network to the configuration file.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerDisplayName="user2556" LastActivityDate="2014-07-17T18:25:44.683" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="773" PostTypeId="2" ParentId="744" CreationDate="2014-07-17T20:02:11.227" Score="83" Body="&lt;p&gt;Think geometrically. Cosine similarity only cares about angle difference, while dot product cares about angle and magnitude. If you normalize your data to have the same magnitude, the two are indistinguishable. Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. Note that neither of them is a &quot;distance metric&quot;.&lt;/p&gt;&#xA;" OwnerUserId="154" LastActivityDate="2014-07-17T20:02:11.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="774" PostTypeId="2" ParentId="376" CreationDate="2014-07-18T04:50:58.287" Score="2" Body="&lt;p&gt;The following &lt;strong&gt;general&lt;/strong&gt; answer is my uneducated guess, so take it with grain of salt. Hopefully, it makes sense. I think that the best way to describe or analyze &lt;em&gt;experiments&lt;/em&gt; (as any other &lt;em&gt;systems&lt;/em&gt;, in general) is to build their &lt;strong&gt;statistical (multivariate) models&lt;/strong&gt; and evaluate them. Depending on whether &lt;em&gt;environments&lt;/em&gt; for your set of experiments are represented by the &lt;strong&gt;same model or different&lt;/strong&gt;, I see the following &lt;strong&gt;approaches&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Single model approach.&lt;/strong&gt; &lt;em&gt;Define&lt;/em&gt; experiments' statistical model for all environments (dependent and independent variables, data types, assumptions, constraints). &lt;em&gt;Analyze&lt;/em&gt; it (most likely, using &lt;em&gt;regression analysis&lt;/em&gt;). &lt;em&gt;Compare&lt;/em&gt; results &lt;strong&gt;across variables&lt;/strong&gt;, which determine (influence) different environments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Multiple models approach.&lt;/strong&gt; The same steps as previous case, but &lt;em&gt;compare&lt;/em&gt; results &lt;strong&gt;across models&lt;/strong&gt;, corresponding to different environments.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-18T04:50:58.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="775" PostTypeId="2" ParentId="758" CreationDate="2014-07-18T07:43:56.823" Score="14" Body="&lt;p&gt;Be sure to check out &lt;a href=&quot;https://www.docker.com/&quot;&gt;docker&lt;/a&gt;! And in general, all the other good things that software engineering has created along decades for ensuring isolation and reproductibility. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to stress that it is not enough to have &lt;em&gt;just&lt;/em&gt; reproducible workflows, but also &lt;em&gt;easy&lt;/em&gt; to reproduce workflows. Let me show what I mean. Suppose that your project uses Python, a database X and Scipy. Most surely you will be using a specific library to connect to your database from Python, and Scipy will be in turn using some sparse algebraic routines. This is by all means a very simple setup, but not entirely simple to setup, pun intended. If somebody wants to execute your scripts, she will have to install all the dependencies. Or worse, she might have incompatible versions of it already installed. Fixing those things takes time. It will also take time to you if you at some moment need to move your computations to a cluster, to a different cluster, or to some cloud servers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is where I find docker useful. Docker is a way to formalize and compile recipes for binary environments. You can write the following in a dockerfile (I'm using here plain English instead of the Dockerfile syntax):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Start with a basic binary environment, like Ubuntu's&lt;/li&gt;&#xA;&lt;li&gt;Install libsparse-dev&lt;/li&gt;&#xA;&lt;li&gt;(Pip) Install numpy and scipy&lt;/li&gt;&#xA;&lt;li&gt;Install X&lt;/li&gt;&#xA;&lt;li&gt;Install libX-dev&lt;/li&gt;&#xA;&lt;li&gt;(Pip) Install python-X&lt;/li&gt;&#xA;&lt;li&gt;Install IPython-Notebook&lt;/li&gt;&#xA;&lt;li&gt;Copy my python scripts/notebooks to my binary environment, these datafiles, and these configurations to do other miscellaneous things. To ensure reproductibility, copy them from a named url instead of a local file. &lt;/li&gt;&#xA;&lt;li&gt;Maybe run IPython-Notebook.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some of the lines will be installing things in Python using pip, since pip can do a very clean work in selecting specific package versions. Check it out too!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And that's it. If after you create your Dockerfile it can be built, then it can be built anywhere, by anybody (provided they also have access to your project-specific files, e.g. because you put them in a public url referenced from the Dockerfile). What is best, you can upload the resulting environment (called an &quot;image&quot;) to a public or private server (called a &quot;register&quot;) for other people to use. So, when you publish your workflow, you have both a fully reproducible recipe in the form of a Dockerfile, and an easy way for you or other people to reproduce what you do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;docker run dockerregistery.thewheezylab.org/nowyouwillbelieveme&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or if they want to poke around in your scripts and so forth:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;docker run -i -t dockerregistery.thewheezylab.org/nowyouwillbelieveme /bin/bash&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2575" LastActivityDate="2014-07-18T07:43:56.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="776" PostTypeId="2" ParentId="41" CreationDate="2014-07-18T15:24:38.007" Score="2" Body="&lt;p&gt;I am far from an expert, but my understanding of the subject tells me that R (superb in statistics) and e.g. Python (superb in several of those things where R is lacking) complements each other quite well (as pointed out by previous posts). &lt;/p&gt;&#xA;" OwnerUserId="2583" LastActivityDate="2014-07-18T15:24:38.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="777" PostTypeId="1" AcceptedAnswerId="782" CreationDate="2014-07-18T22:29:05.017" Score="2" ViewCount="161" Body="&lt;p&gt;New to the Data Science forum, and first poster here!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may be kind of a specific question (hopefully not too much so), but one I'd imagine others might be interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a way to basically query GitHub with something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Give me a collection of all of the public repositories that have more than 10 stars, at&#xA;least two forks, and more than three committers.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result could take any viable form: a JSON data dump, a URL to the web page, etc. It more than likely will consist of information from 10,000 repos or something large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this sort of thing possible using the API or some other pre-built way, or am I going to have to build out my own custom solution where I try to scrape every page? If so, how feasible is this and how might I approach it?&lt;/p&gt;&#xA;" OwnerUserId="2599" LastActivityDate="2014-07-19T03:42:34.433" Title="Getting GitHub repository information by different criteria" Tags="|bigdata|data-mining|python|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="778" PostTypeId="1" AcceptedAnswerId="781" CreationDate="2014-07-18T22:34:48.080" Score="14" ViewCount="17206" Body="&lt;p&gt;I read in this post &lt;a href=&quot;https://datascience.stackexchange.com/questions/41/is-the-r-language-suitable-for-big-data&quot;&gt;Is the R language suitable for Big Data&lt;/a&gt; that big data constitutes &lt;code&gt;5TB&lt;/code&gt;, and while it does a good job of providing information about the feasibility of working with this type of data in &lt;code&gt;R&lt;/code&gt; it provides very little information about &lt;code&gt;Python&lt;/code&gt;. I was wondering if &lt;code&gt;Python&lt;/code&gt; can work with this much data as well. &lt;/p&gt;&#xA;" OwnerUserId="890" LastActivityDate="2022-08-18T09:59:32.430" Title="Is Python suitable for big data" Tags="|bigdata|python|" AnswerCount="9" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="779" PostTypeId="2" ParentId="778" CreationDate="2014-07-18T22:59:39.100" Score="2" Body="&lt;p&gt;I believe the language itself has little to do with performance capabilities, when it comes to large data. What matters is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How large is the data actually&lt;/li&gt;&#xA;&lt;li&gt;What processing are you going to perform on it&lt;/li&gt;&#xA;&lt;li&gt;What hardware are you going to use&lt;/li&gt;&#xA;&lt;li&gt;Which are the specific libraries that you plan to use&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Anyway, Python is well adopted in data science communities.&lt;/p&gt;&#xA;" OwnerUserId="2600" LastActivityDate="2014-07-18T22:59:39.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="780" PostTypeId="2" ParentId="41" CreationDate="2014-07-19T02:19:46.530" Score="17" Body="&lt;p&gt;Some good answers here. I would like to join the discussion by adding the following three &lt;strong&gt;notes&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The question's emphasis on the &lt;strong&gt;volume of data&lt;/strong&gt; while referring to &lt;em&gt;Big Data&lt;/em&gt; is certainly understandable and valid, especially considering the &lt;strong&gt;problem&lt;/strong&gt; of data volume growth &lt;strong&gt;outpacing&lt;/strong&gt; technological capacities' exponential growth per &lt;em&gt;Moore's Law&lt;/em&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Moore%27s_law&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://en.wikipedia.org/wiki/Moore%27s_law&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Having said that, it is important to remember about other aspects of big data concept. Based on &lt;em&gt;Gartner&lt;/em&gt;'s definition (emphasis mine - AB): &quot;&lt;em&gt;Big data&lt;/em&gt; is high &lt;strong&gt;volume&lt;/strong&gt;, high &lt;strong&gt;velocity&lt;/strong&gt;, and/or high &lt;strong&gt;variety&lt;/strong&gt; information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.&quot; (usually referred to as the &quot;&lt;strong&gt;3Vs model&lt;/strong&gt;&quot;). I mention this, because it forces data scientists and other analysts to look for and use R packages that focus on &lt;strong&gt;other than volume&lt;/strong&gt; aspects of big data (enabled by the &lt;strong&gt;richness&lt;/strong&gt; of enormous &lt;em&gt;R ecosystem&lt;/em&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;While existing answers mention some R packages, related to big data, for a more &lt;strong&gt;comprehensive coverage&lt;/strong&gt;, I'd recommend to refer to &lt;em&gt;CRAN Task View&lt;/em&gt; &lt;strong&gt;&quot;High-Performance and Parallel Computing with R&quot;&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/views/HighPerformanceComputing.html&lt;/a&gt;), in particular, sections &lt;strong&gt;&quot;Parallel computing: Hadoop&quot;&lt;/strong&gt; and &lt;strong&gt;&quot;Large memory and out-of-memory data&quot;&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="23546" LastEditDate="2019-02-23T11:34:41.513" LastActivityDate="2019-02-23T11:34:41.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="781" PostTypeId="2" ParentId="778" CreationDate="2014-07-19T03:29:02.647" Score="18" Body="&lt;p&gt;To clarify, I feel like the original question references by OP probably isn't be best for a SO-type format, but I will certainly represent &lt;code&gt;python&lt;/code&gt; in this particular case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me just start by saying that regardless of your data size, &lt;code&gt;python&lt;/code&gt; shouldn't be your limiting factor. In fact, there are just a couple main issues that you're going to run into dealing with large datasets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Reading data into memory&lt;/strong&gt; - This is by far the most common issue faced in the world of big data. Basically, you can't read in more data than you have memory (RAM) for. The best way to fix this is by making atomic operations on your data instead of trying to read everything in at once.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Storing data&lt;/strong&gt; - This is actually just another form of the earlier issue, by the time to get up to about &lt;code&gt;1TB&lt;/code&gt;, you start having to look elsewhere for storage. AWS S3 is the most common resource, and &lt;code&gt;python&lt;/code&gt; has the fantastic &lt;code&gt;boto&lt;/code&gt; library to facilitate leading with large pieces of data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Network latency&lt;/strong&gt; - Moving data around between different services is going to be your bottleneck. There's not a huge amount you can do to fix this, other than trying to pick co-located resources and plugging into the wall.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-19T03:29:02.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="782" PostTypeId="2" ParentId="777" CreationDate="2014-07-19T03:42:34.433" Score="1" Body="&lt;p&gt;My limited understanding, based on brief browsing &lt;em&gt;GitHub API&lt;/em&gt; documentation, is that currently there is NO &lt;strong&gt;single API request&lt;/strong&gt; that supports &lt;strong&gt;all&lt;/strong&gt; your listed criteria &lt;strong&gt;at once&lt;/strong&gt;. However, I think that you could use the following sequence in order to achieve the goal from your example (at least, I would use this approach):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Request&lt;/strong&gt; information on all public repositories (API returns &lt;em&gt;summary representations&lt;/em&gt; only): &lt;a href=&quot;https://developer.github.com/v3/repos/#list-all-public-repositories&quot; rel=&quot;nofollow&quot;&gt;https://developer.github.com/v3/repos/#list-all-public-repositories&lt;/a&gt;;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Loop&lt;/strong&gt; through the list of all public repositories retrieved in step 1, requesting individual resources, and save it as new (detailed) list (this returns &lt;em&gt;detailed representations&lt;/em&gt;, in other words, all attributes): &lt;a href=&quot;https://developer.github.com/v3/repos/#get&quot; rel=&quot;nofollow&quot;&gt;https://developer.github.com/v3/repos/#get&lt;/a&gt;;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;strong&gt;Loop&lt;/strong&gt; through the detailed list of all repositories, filtering corresponding fields by your criteria. For your example request, you'd be interested in the following attributes of the &lt;strong&gt;parent&lt;/strong&gt; object: &lt;em&gt;stargazers_count&lt;/em&gt;, &lt;em&gt;forks_count&lt;/em&gt;. In order to filter the repositories by number of committers, you could use a separate API: &lt;a href=&quot;https://developer.github.com/v3/repos/#list-contributors&quot; rel=&quot;nofollow&quot;&gt;https://developer.github.com/v3/repos/#list-contributors&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Updates or comments from people more familiar with GitHub API are welcome!&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-19T03:42:34.433" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="783" PostTypeId="1" AcceptedAnswerId="799" CreationDate="2014-07-19T05:27:22.773" Score="11" ViewCount="1267" Body="&lt;p&gt;I want to plot the bytes from a disk image in order to understand a pattern in them. This is mainly an academic task, since I'm almost sure this pattern was created by a disk testing program, but I'd like to reverse-engineer it anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already know that the pattern is aligned, with a periodicity of 256 characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can envision two ways of visualizing this information: either a 16x16 plane viewed through time (3 dimensions), where each pixel's color is the ASCII code for the character, or a 256 pixel line for each period (2 dimensions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a snapshot of the pattern (you can see more than one), seen through &lt;code&gt;xxd&lt;/code&gt; (32x16):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/zOFSK.gif&quot; alt=&quot;Pattern to analyze&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, I am trying to find a way of visualizing this information. This probably isn't hard for anyone into signal analysis, but I can't seem to find a way using open-source software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to avoid Matlab or Mathematica and I'd prefer an answer in R, since I have been learning it recently, but nonetheless, any language is welcome.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Update, 2014-07-25: given Emre's answer below, this is what the pattern looks like, given the first 30MB of the pattern, aligned at 512 instead of 256 (this alignment looks better):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4tDIA.png&quot; alt=&quot;Graphical pattern&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any further ideas are welcome!&lt;/p&gt;&#xA;" OwnerUserId="2604" LastEditorUserId="2604" LastEditDate="2014-07-25T03:26:52.427" LastActivityDate="2014-07-25T12:59:00.850" Title="Data visualization for pattern analysis (language-independent, but R preferred)" Tags="|r|visualization|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="784" PostTypeId="1" AcceptedAnswerId="821" CreationDate="2014-07-19T18:31:36.573" Score="6" ViewCount="1067" Body="&lt;p&gt;I have a timeseries with hourly gas consumption. I want to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model&quot; rel=&quot;noreferrer&quot;&gt;ARMA&lt;/a&gt;/&lt;a href=&quot;http://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;noreferrer&quot;&gt;ARIMA&lt;/a&gt; to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with &lt;a href=&quot;https://www.otexts.org/fpp/6/5&quot; rel=&quot;noreferrer&quot;&gt;Seasonal and Trend decomposition using Loess&lt;/a&gt; (STL)?)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/hYyH8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="84" LastEditDate="2014-07-20T19:34:08.117" LastActivityDate="2015-07-20T21:51:46.383" Title="Why should I care about seasonal data when I forecast?" Tags="|machine-learning|time-series|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="785" PostTypeId="2" ParentId="783" CreationDate="2014-07-19T18:47:38.787" Score="1" Body="&lt;p&gt;I know almost nothing about signal analysis, but 2-dimensional visualization could be easily done using R. Particularly you will need &lt;code&gt;reshape2&lt;/code&gt; and &lt;code&gt;ggplot2&lt;/code&gt; packages. Assuming your data is &lt;strong&gt;wide&lt;/strong&gt; (e.g. [n X 256] size), first you need to transform it to &lt;a href=&quot;http://www.cookbook-r.com/Manipulating_data/Converting_data_between_wide_and_long_format/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;long&lt;/strong&gt;&lt;/a&gt; format using &lt;code&gt;melt()&lt;/code&gt; function from &lt;code&gt;reshape2&lt;/code&gt; package. Then use &lt;a href=&quot;http://docs.ggplot2.org/current/geom_tile.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;geom_tile&lt;/code&gt;&lt;/a&gt; geometry from &lt;code&gt;ggplot2&lt;/code&gt;. Here is a nice &lt;a href=&quot;http://www.r-bloggers.com/simplest-possible-heatmap-with-ggplot2/&quot; rel=&quot;nofollow&quot;&gt;recipe&lt;/a&gt; with &lt;a href=&quot;https://gist.github.com/dsparks/3710171&quot; rel=&quot;nofollow&quot;&gt;gist&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-07-19T18:47:38.787" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="786" PostTypeId="1" AcceptedAnswerId="789" CreationDate="2014-07-19T20:46:52.740" Score="10" ViewCount="1261" Body="&lt;p&gt;I am trying to find stock data to practice with, is there a good resource for this? I found &lt;a href=&quot;ftp://emi.nasdaq.com/ITCH/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; but it only has the current year.&lt;/p&gt;&#xA;&lt;p&gt;I already have a way of parsing the protocol, but would like to have some more data to compare with. It doesn't have to be in the same format, as long as it has price, trades, and date statistics.&lt;/p&gt;&#xA;" OwnerUserId="2567" LastEditorUserId="98307" LastEditDate="2020-08-16T18:02:33.567" LastActivityDate="2020-08-16T18:02:33.567" Title="NASDAQ Trade Data" Tags="|data-mining|dataset|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="787" PostTypeId="1" AcceptedAnswerId="884" CreationDate="2014-07-19T20:51:58.527" Score="3" ViewCount="12047" Body="&lt;p&gt;I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cleared up some &quot;shared library&quot; problems by adding this to yarn-env.sh and hadoop-env.sh:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;export HADOOP_HOME=&quot;/usr/local/hadoop&quot;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;export HADOOP_OPTS=&quot;-Djava.library.path=$HADOOP_HOME/lib&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I also added this to hadoop-env.sh:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;based on the advice of this post at horton works &lt;a href=&quot;http://hortonworks.com/community/forums/topic/hdfs-tmp-dir-issue/&quot; rel=&quot;nofollow&quot;&gt;http://hortonworks.com/community/forums/topic/hdfs-tmp-dir-issue/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That cleared up all of my error messages; when I run /sbin/start-yarn.sh I get this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;starting yarn daemons&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;starting resourcemanager, &#xA;  logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-HdNode.out&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;localhost: starting nodemanager, &#xA;  logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-HdNode.out&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The only problem is, JPS says that the Resource Manager isn't running. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's going on here?&lt;/p&gt;&#xA;" OwnerUserId="2614" LastActivityDate="2017-01-30T17:07:41.170" Title="Hadoop Resource Manager Won't Start" Tags="|apache-hadoop|" AnswerCount="2" CommentCount="5" ClosedDate="2017-01-31T10:21:06.387" ContentLicense="CC BY-SA 3.0" />
  <row Id="788" PostTypeId="2" ParentId="787" CreationDate="2014-07-20T21:37:27.857" Score="1" Body="&lt;p&gt;Check your version of JPS and make sure it's the same as the version of java that you are running.  Sometimes you start out with an out of the box jdk install, upgrade, set alternatives for the java bin, but still have the original jps binary being referenced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run &lt;code&gt;ps -ef |grep java&lt;/code&gt; and look for the resource manager threads.  Maybe it's actually running.  If it is, try &lt;code&gt;update-alternatives --config jps&lt;/code&gt; to see what binary jps is pointing at and compare it with the java binary that you are using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the resource manager is not running, it's time to do some basic linux troubleshooting.  Check log files and barring that check actual command output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the system I'm looking at now, the log files for resource manager are placed in the &lt;code&gt;hadoop-install/logs&lt;/code&gt; directory in &lt;code&gt;yarn-username-resourcemanager-hostname.log&lt;/code&gt; and &lt;code&gt;yarn-user-resourcemanager-hostname.out&lt;/code&gt;.  Your configuration may place them in /var/logs or what have you.  Also, have a look at the syslog.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the logs don't yield any good information, which can happen, my process is to generally try to figure out the command line from the startup script (usually by prefixing the command line with &lt;code&gt;echo&lt;/code&gt;), and then trying to run the command directly to see the output as it comes out.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I have actually run into this problem before, but I can't remember the specific issue.  I'm sure the same result can manifest itself from a variety of problems.  Considering that you are as far as you are in the process of getting set up, I believe it's likely to be a minor configuration issue.&lt;/p&gt;&#xA;" OwnerUserId="434" LastEditorUserId="434" LastEditDate="2014-07-20T21:48:25.257" LastActivityDate="2014-07-20T21:48:25.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="789" PostTypeId="2" ParentId="786" CreationDate="2014-07-20T23:54:09.070" Score="7" Body="&lt;p&gt;You can pull stock data very easyly in python and R (probably other languages as well) with the following packages:&lt;/p&gt;&#xA;&lt;p&gt;In python with &lt;a href=&quot;https://pypi.python.org/pypi/ystockquote&quot; rel=&quot;nofollow noreferrer&quot;&gt;ystockquote&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://nbviewer.ipython.org/github/twiecki/financial-analysis-python-tutorial/blob/master/1.%20Pandas%20Basics.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is also a really nice tutorial in iPython which shows you how to pull the stock data and play with it&lt;/p&gt;&#xA;&lt;p&gt;In R with &lt;a href=&quot;http://www.quantmod.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;quantmod&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;HTH.&lt;/p&gt;&#xA;" OwnerUserId="802" LastEditorUserId="98307" LastEditDate="2020-08-16T18:02:25.707" LastActivityDate="2020-08-16T18:02:25.707" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="790" PostTypeId="2" ParentId="778" CreationDate="2014-07-21T00:35:13.440" Score="4" Body="&lt;p&gt;Absolutely. When you're working with data at that scale it's common to use a big data framework, in which case python or whatever language you're using is merely an interface. See for example &lt;a href=&quot;http://spark.apache.org/docs/0.9.1/python-programming-guide.html&quot; rel=&quot;nofollow&quot;&gt;Spark's Python Programming Guide&lt;/a&gt;. What kind of data do you have and what do you want to do with it?&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-21T00:35:13.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="791" PostTypeId="1" CreationDate="2014-07-21T09:00:04.917" Score="8" ViewCount="1967" Body="&lt;p&gt;I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the words &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;senior, primary, lead, head, vp, director, stuff, principal, chief, &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or the words &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;analyst, expert, modeler, researcher, scientist, developer, engineer, architect.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I get all such possible words with their distance ?&lt;/p&gt;&#xA;" OwnerUserId="921" LastEditorUserId="921" LastEditDate="2014-07-22T09:00:27.183" LastActivityDate="2014-07-23T15:46:48.977" Title="Job title similarity" Tags="|machine-learning|dataset|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="792" PostTypeId="2" ParentId="778" CreationDate="2014-07-21T11:59:49.203" Score="3" Body="&lt;p&gt;To handle such amount of data, programming language is not the main concern but the programming framework is. Frameworks such as MapReduce or Spark have bindings to many languages including Python. These frameworks certainly have many ready-to-use packages for data analysis tasks. But in the end it all comes to your requirement, i.e., what is your task? People have different definitions of data analysis tasks, some of them can be easily solved with relational databases. In that case, SQL is much better than all other alternatives.&lt;/p&gt;&#xA;" OwnerUserId="743" LastActivityDate="2014-07-21T11:59:49.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="793" PostTypeId="1" AcceptedAnswerId="797" CreationDate="2014-07-21T13:41:13.427" Score="21" ViewCount="8198" Body="&lt;p&gt;How can &lt;a href=&quot;http://en.wikipedia.org/wiki/NoSQL&quot;&gt;NoSQL&lt;/a&gt; databases like &lt;a href=&quot;http://en.wikipedia.org/wiki/MongoDB&quot;&gt;MongoDB&lt;/a&gt; be used for data analysis? What are the features in them that can make data analysis faster and powerful?&lt;/p&gt;&#xA;" OwnerUserId="2643" LastEditorUserId="2523" LastEditDate="2014-07-27T07:36:51.510" LastActivityDate="2015-11-07T07:51:08.303" Title="Uses of NoSQL database in data science" Tags="|bigdata|nosql|mongodb|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="794" PostTypeId="2" ParentId="694" CreationDate="2014-07-21T15:35:59.540" Score="9" Body="&lt;p&gt;Pylearn2 seems to be the library of choice, however I find their YAML configuration files off-putting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python itself was designed to be an easy language for prototyping, why would you &lt;strong&gt;not&lt;/strong&gt; use it to define the network properties themselves? We have great editors with autocompletion that would make your life much easier and Python is not like C++ where you have to wait for long builds to finish before you can run your code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;YAML files on the other hand you have to edit using a standard text editor with no assistance whatsoever and this makes the learning curve even steeper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I may be missing the big picture but I still don't understand what were they thinking, I don't think prototyping in code would be much slower. For that reason I'm considering Theanets or using Theano directly.&lt;/p&gt;&#xA;" OwnerUserId="2648" LastActivityDate="2014-07-21T15:35:59.540" CommentCount="3" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="795" PostTypeId="2" ParentId="791" CreationDate="2014-07-21T16:15:07.010" Score="1" Body="&lt;p&gt;Not sure if this is exactly what you're looking for, but r-base has a function called &quot;adist&quot; which creates a distance matrix of approximate string distances (according to the Levenshtein distance). Type '?adist' for more.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words = c(&quot;senior&quot;, &quot;primary&quot;, &quot;lead&quot;, &quot;head&quot;, &quot;vp&quot;, &quot;director&quot;, &quot;stuff&quot;, &quot;principal&quot;, &quot;chief&quot;)&#xA;adist(words)&#xA;&#xA;      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]&#xA; [1,]    0    6    5    5    6    5    5    7    5&#xA; [2,]    6    0    6    6    7    7    7    6    6&#xA; [3,]    5    6    0    1    4    7    5    8    5&#xA; [4,]    5    6    1    0    4    7    5    8    4&#xA; [5,]    6    7    4    4    0    8    5    8    5&#xA; [6,]    5    7    7    7    8    0    8    8    7&#xA; [7,]    5    7    5    5    5    8    0    9    4&#xA; [8,]    7    6    8    8    8    8    9    0    8&#xA; [9,]    5    6    5    4    5    7    4    8    0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, if R isn't an option, the Levenshtein distance algorithm is implemented in many languages here:&#xA;&lt;a href=&quot;http://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance&quot; rel=&quot;nofollow&quot;&gt;http://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="375" LastActivityDate="2014-07-21T16:15:07.010" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="796" PostTypeId="2" ParentId="778" CreationDate="2014-07-21T16:58:22.660" Score="14" Body="&lt;p&gt;There are couple of things you need to understand when dealing with Big data -&lt;/p&gt;&#xA;&lt;h3&gt;What is Big data?&lt;/h3&gt;&#xA;&lt;p&gt;You might be aware of famous V's of Big data - Volume, Velocity, Variety... So, Python may not be suitable for all. And it goes with all data science tools available. You need to know which tool is good for what purpose.&lt;/p&gt;&#xA;&lt;p&gt;If dealing with large Volume of data:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pig/Hive/Shark - Data cleaning and ETL work&lt;/li&gt;&#xA;&lt;li&gt;Hadoop/Spark - Distributed parallel computing&lt;/li&gt;&#xA;&lt;li&gt;Mahout/ML-Lib - Machine Learning&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Now, you can use R/Python in intermediate stages but you'll realize that they become bottleneck in your entire process.&lt;/p&gt;&#xA;&lt;p&gt;If dealing with Velocity of data:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Kafka/Storm - High throughput system&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;People are trying to R/Python here but again it depends on kind of parallelism you want and your model complexity.&lt;/p&gt;&#xA;&lt;h3&gt;What sort of analysis you wish to do?&lt;/h3&gt;&#xA;&lt;p&gt;If your model demands the entire data to be first brought into memory then your model should not be complex because if the intermediate data is large then the code will break. And if you think of writing it into disk then you'll face additional delay because disk read/write is slow as compared to RAM.&lt;/p&gt;&#xA;&lt;h2&gt;Conclusion&lt;/h2&gt;&#xA;&lt;p&gt;You can definitely use Python in Big data space (Definitely, since people are trying with R, why not Python) but know your data and business requirement first. There may be better tools available for same and always remember:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Your tools shouldn’t determine how you answer questions. Your questions should determine what tools you use.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2433" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-27T15:03:17.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="797" PostTypeId="2" ParentId="793" CreationDate="2014-07-21T19:06:43.223" Score="26" Body="&lt;p&gt;To be perfectly honest, most NoSQL databases are not very well suited to applications in big data. For the vast majority of all big data applications, the performance of &lt;a href=&quot;https://en.wikipedia.org/wiki/MongoDB&quot; rel=&quot;noreferrer&quot;&gt;MongoDB&lt;/a&gt; compared to a relational database like &lt;a href=&quot;http://en.wikipedia.org/wiki/MySQL&quot; rel=&quot;noreferrer&quot;&gt;MySQL&lt;/a&gt; is &lt;a href=&quot;http://www.moredevs.com/mysql-vs-mongodb-performance-benchmark/&quot; rel=&quot;noreferrer&quot;&gt;significantly&lt;/a&gt; is poor enough to warrant staying away from something like MongoDB entirely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, there are a couple of really useful properties of NoSQL databases that certainly work in your favor when you're working with large data sets, though the chance of those benefits outweighing the generally poor performance of NoSQL compared to &lt;a href=&quot;http://en.wikipedia.org/wiki/SQL&quot; rel=&quot;noreferrer&quot;&gt;SQL&lt;/a&gt; for read-intensive operations (most similar to typical big data use cases) is low.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;No Schema&lt;/strong&gt; - If you're working with a lot of unstructured data, it might be hard to actually decide on and rigidly apply a schema. NoSQL databases in general are very supporting of this, and will allow you to insert schema-less documents on the fly, which is certainly not something an SQL database will support.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/JSON&quot; rel=&quot;noreferrer&quot;&gt;JSON&lt;/a&gt;&lt;/strong&gt; - If you happen to be working with JSON-style documents instead of with &lt;a href=&quot;http://en.wikipedia.org/wiki/Comma-separated_values&quot; rel=&quot;noreferrer&quot;&gt;CSV&lt;/a&gt; files, then you'll see a lot of advantage in using something like MongoDB for a database-layer. Generally the workflow savings don't outweigh the increased query-times though.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Ease of Use&lt;/strong&gt; - I'm not saying that SQL databases are always hard to use, or that &lt;a href=&quot;http://en.wikipedia.org/wiki/Apache_Cassandra&quot; rel=&quot;noreferrer&quot;&gt;Cassandra&lt;/a&gt; is the easiest thing in the world to set up, but in general NoSQL databases are easier to set up and use than SQL databases. MongoDB is a particularly strong example of this, known for being one of the easiest database layers to use (outside of &lt;a href=&quot;http://en.wikipedia.org/wiki/SQLite&quot; rel=&quot;noreferrer&quot;&gt;SQLite&lt;/a&gt;). SQL also deals with a lot of normalization and there's a large legacy of SQL best practices that just generally bogs down the development process.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Personally I might suggest you also check out &lt;a href=&quot;http://en.wikipedia.org/wiki/Graph_database&quot; rel=&quot;noreferrer&quot;&gt;graph databases&lt;/a&gt; such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Neo4j&quot; rel=&quot;noreferrer&quot;&gt;Neo4j&lt;/a&gt; that show really good performance for certain types of queries if you're looking into picking out a backend for your data science applications.&lt;/p&gt;&#xA;" OwnerUserId="548" LastEditorUserId="2580" LastEditDate="2015-11-07T07:51:08.303" LastActivityDate="2015-11-07T07:51:08.303" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="798" PostTypeId="2" ParentId="791" CreationDate="2014-07-21T20:42:00.143" Score="2" Body="&lt;p&gt;If I understand your question, you can look at the co-occurrence matrix formed using the terms following the title; e.g., senior FOO, primary BAR, etc. Then you can compute the similarity between any pair of terms, such as &quot;senior&quot; and &quot;primary&quot;, using a suitable metric; e.g., the cosine similarity.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-21T20:42:00.143" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="799" PostTypeId="2" ParentId="783" CreationDate="2014-07-21T21:17:07.303" Score="5" Body="&lt;p&gt;I would use a visual analysis. Since you know there is a repetition every 256 bytes, create an image 256 pixels wide by however many deep, and encode the data using brightness. In (i)python it would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os, numpy, matplotlib.pyplot as plt&#xA;&#xA;%matplotlib inline&#xA;&#xA;def read_in_chunks(infile, chunk_size=256):&#xA;    while True:&#xA;        chunk = infile.read(chunk_size)&#xA;        if chunk:&#xA;            yield chunk&#xA;        else:&#xA;            # The chunk was empty, which means we're at the end&#xA;            # of the file&#xA;            return&#xA;&#xA;fname = 'enter something here'&#xA;srcfile = open(fname, 'rb')&#xA;height = 1 + os.path.getsize(fname)/256&#xA;data = numpy.zeros((height, 256), dtype=numpy.uint8)    &#xA;&#xA;for i, line in enumerate(read_in_chunks(srcfile)):&#xA;    vals = list(map(int, line))&#xA;    data[i,:len(vals)] = vals&#xA;&#xA;plt.imshow(data, aspect=1e-2);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is what a PDF looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/bicgF.png&quot; alt=&quot;A PDF file visualized&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 256 byte periodic pattern would have manifested itself as vertical lines. Except for the header and tail it looks pretty noisy.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-07-22T00:13:21.200" LastActivityDate="2014-07-22T00:13:21.200" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="800" PostTypeId="2" ParentId="793" CreationDate="2014-07-21T21:29:26.270" Score="4" Body="&lt;p&gt;One benefit of the schema-free NoSQL approach is that you don't commit prematurely and you can apply the right schema at query time using an appropriate tool like &lt;a href=&quot;http://incubator.apache.org/drill/&quot; rel=&quot;nofollow&quot;&gt;Apache Drill&lt;/a&gt;. See &lt;a href=&quot;http://wiki.apache.org/incubator/DrillProposal?action=AttachFile&amp;amp;do=get&amp;amp;target=Drill+slides.pdf&quot; rel=&quot;nofollow&quot;&gt;this presentation&lt;/a&gt; for details. MySQL wouldn't be my first choice in a big data setting.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-21T21:29:26.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="801" PostTypeId="2" ParentId="791" CreationDate="2014-07-21T21:32:12.963" Score="4" Body="&lt;p&gt;That's an interesting problem, thanks for bring out here on stack.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this problem is similar to when we apply &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;LSA(Latent Semantic Analysis)&lt;/a&gt; in sentiment analysis to find list of positive and negative words with polarity with respect to some predefined positive and negative words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Good reads:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs.stanford.edu/people/ang/papers/acl11-WordVectorsSentimentAnalysis.pdf&quot; rel=&quot;nofollow&quot;&gt;Learning Word Vectors for Sentiment Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/pdf/cs/0212012.pdf&quot; rel=&quot;nofollow&quot;&gt;Unsupervised Learning of Semantic Orientation from a Hundred-Billion-Word Corpus&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, according to me LSA is your best approach to begin with in this situation as it learns the underlying relation between the words from the corpus and probably that's what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="2433" LastActivityDate="2014-07-21T21:32:12.963" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="802" PostTypeId="1" CreationDate="2014-07-21T23:53:11.120" Score="13" ViewCount="3159" Body="&lt;p&gt;I'm working on an application which requires creating a very large database of n-grams that exist in a large text corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need three efficient operation types: Lookup and insertion indexed by the n-gram itself, and querying for all n-grams that contain a sub-n-gram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sounds to me like the database should be a gigantic document tree, and document databases, e.g. Mongo, should be able to do the job well, but I've never used those at scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing the Stack Exchange question format, I'd like to clarify that I'm not asking for suggestions on specific technologies, but rather a type of database that I should be looking for to implement something like this at scale.&lt;/p&gt;&#xA;" OwnerUserId="1163" LastActivityDate="2014-08-16T10:25:19.687" Title="Efficient database model for storing data indexed by n-grams" Tags="|nlp|databases|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="803" PostTypeId="2" ParentId="802" CreationDate="2014-07-22T00:06:10.500" Score="1" Body="&lt;p&gt;I haven't done this before but it sounds like a job for a graph database given the functionality you want. &lt;a href=&quot;http://www.rene-pickhardt.de/download-google-n-gram-data-set-and-neo4j-source-code-for-storing-it/&quot; rel=&quot;nofollow&quot;&gt;Here's a demo in neo4j&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-22T00:06:10.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="804" PostTypeId="1" AcceptedAnswerId="805" CreationDate="2014-07-22T01:12:03.860" Score="4" ViewCount="88" Body="&lt;p&gt;Basically, both are software systems that are based on data and algorithms.&lt;/p&gt;&#xA;" OwnerUserId="1117" LastActivityDate="2014-07-22T06:06:31.513" Title="What's the difference between data products and intelligent systems?" Tags="|tools|education|definitions|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="805" PostTypeId="2" ParentId="804" CreationDate="2014-07-22T02:27:43.710" Score="1" Body="&lt;p&gt;This is a very vague question. However, I will try to make sense of it. Considering rules of &lt;em&gt;logic&lt;/em&gt; as well as your statement that both &lt;em&gt;entities&lt;/em&gt; are &quot;software systems that are based on data and algorithms&quot;, it appears that &lt;strong&gt;data products&lt;/strong&gt; are &lt;em&gt;intelligent systems&lt;/em&gt; and &lt;strong&gt;intelligent systems&lt;/strong&gt; are, to some degree, &lt;em&gt;data products&lt;/em&gt;. Therefore, it can be argued that the difference between the terms &quot;data products&quot; and &quot;intelligent systems&quot; is purely in the &lt;strong&gt;focus&lt;/strong&gt; (&lt;em&gt;source&lt;/em&gt; of information or &lt;em&gt;purpose&lt;/em&gt; of system dimensions) of each type of systems (&lt;em&gt;data&lt;/em&gt; &lt;strong&gt;vs.&lt;/strong&gt; &lt;em&gt;intelligence/algorithms&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2014-07-22T06:06:31.513" LastActivityDate="2014-07-22T06:06:31.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="806" PostTypeId="1" AcceptedAnswerId="807" CreationDate="2014-07-22T03:43:20.327" Score="98" ViewCount="103529" Body="&lt;p&gt;I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've found that some claim its advantage is mostly marginal in that it is best for catching 'lucky' models with high standard accuracy measurements and low AUC.&lt;/p&gt;&#xA;&lt;p&gt;So should I avoid relying on AUC for validating models or would a combination be best?&lt;/p&gt;&#xA;" OwnerUserId="2653" LastEditorUserId="43000" LastEditDate="2023-05-04T17:16:21.923" LastActivityDate="2023-05-04T17:16:21.923" Title="Advantages of AUC vs standard accuracy" Tags="|machine-learning|accuracy|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="807" PostTypeId="2" ParentId="806" CreationDate="2014-07-22T04:10:18.353" Score="84" Body="&lt;p&gt;Really great question, and one that I find that most people don't really understand on an intuitive level. &lt;code&gt;AUC&lt;/code&gt; is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what &lt;code&gt;AUC&lt;/code&gt; is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to figure out exactly how &lt;code&gt;AUC&lt;/code&gt; works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;AUC&lt;/code&gt; stands for &lt;code&gt;Area Under the Curve&lt;/code&gt;, which curve you ask? Well, that would be the &lt;code&gt;ROC&lt;/code&gt; curve. &lt;code&gt;ROC&lt;/code&gt; stands for &lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;noreferrer&quot;&gt;Receiver Operating Characteristic&lt;/a&gt;, which is actually slightly non-intuitive. The implicit goal of &lt;code&gt;AUC&lt;/code&gt; is to deal with situations where you have a very skewed sample distribution, and don't want to overfit to a single class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A great example is in spam detection. Generally, spam datasets are STRONGLY biased towards ham, or not-spam. If your data set is 90% ham, you can get a pretty damn good accuracy by just saying that every single email is ham, which is obviously something that indicates a non-ideal classifier. Let's start with a couple of metrics that are a little more useful for us, specifically the true positive rate (&lt;code&gt;TPR&lt;/code&gt;) and the false positive rate (&lt;code&gt;FPR&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/hNxTl.png&quot; alt=&quot;ROC axes&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in this graph, &lt;code&gt;TPR&lt;/code&gt; is specifically the ratio of true positive to all positives, and &lt;code&gt;FPR&lt;/code&gt; is the ratio of false positives to all negatives. (Keep in mind, this is only for binary classification.) On a graph like this, it should be pretty straightforward to figure out that a prediction of all 0's or all 1's will result in the points of &lt;code&gt;(0,0)&lt;/code&gt; and &lt;code&gt;(1,1)&lt;/code&gt; respectively. If you draw a line through these lines you get something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/B1WT1.png&quot; alt=&quot;Kind of like a triangle&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which looks basically like a diagonal line (it is), and by some easy geometry, you can see that the &lt;code&gt;AUC&lt;/code&gt; of such a model would be &lt;code&gt;0.5&lt;/code&gt; (height and base are both 1). Similarly, if you predict a random assortment of 0's and 1's, let's say 90% 1's, you could get the point &lt;code&gt;(0.9, 0.9)&lt;/code&gt;, which again falls along that diagonal line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now comes the interesting part. What if we weren't only predicting 0's and 1's? What if instead, we wanted to say that, theoretically we were going to set a cutoff, above which every result was a 1, and below which every result were a 0. This would mean that at the extremes you get the original situation where you have all 0's and all 1's (at a cutoff of 0 and 1 respectively), but also a series of intermediate states that fall within the &lt;code&gt;1x1&lt;/code&gt; graph that contains your &lt;code&gt;ROC&lt;/code&gt;. In practice you get something like this:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/13McM.png&quot; alt=&quot;Courtesy of Wikipedia&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, what you're actually getting when you do an &lt;code&gt;AUC&lt;/code&gt; over accuracy is something that will strongly discourage people going for models that are representative, but not discriminative, as this will only actually select for models that achieve false positive and true positive rates that are significantly above random chance, which is not guaranteed for accuracy.&lt;/p&gt;&#xA;" OwnerUserId="548" LastEditorUserId="65710" LastEditDate="2019-02-08T16:20:53.637" LastActivityDate="2019-02-08T16:20:53.637" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="808" PostTypeId="1" AcceptedAnswerId="809" CreationDate="2014-07-22T08:39:33.810" Score="11" ViewCount="2354" Body="&lt;p&gt;i want to become a &lt;strong&gt;data scientist&lt;/strong&gt;. I studied applied &lt;strong&gt;statistics&lt;/strong&gt; (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But now,  I am going to do a master degree in &lt;strong&gt;Computer Science&lt;/strong&gt; focus in Intelligent Systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my study plan:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Machine learning&lt;/li&gt;&#xA;&lt;li&gt;Advanced machine learning&lt;/li&gt;&#xA;&lt;li&gt;Data mining&lt;/li&gt;&#xA;&lt;li&gt;Fuzzy logic&lt;/li&gt;&#xA;&lt;li&gt;Recommendation Systems&lt;/li&gt;&#xA;&lt;li&gt;Distributed Data Systems&lt;/li&gt;&#xA;&lt;li&gt;Cloud Computing&lt;/li&gt;&#xA;&lt;li&gt;Knowledge discovery&lt;/li&gt;&#xA;&lt;li&gt;Business Intelligence&lt;/li&gt;&#xA;&lt;li&gt;Information retrieval&lt;/li&gt;&#xA;&lt;li&gt;Text mining&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At the end, with all my statistical  and  computer science knowledge, can i call myself a data scientist? , or am i wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the answers.&lt;/p&gt;&#xA;" OwnerUserId="1117" LastEditorUserId="11097" LastEditDate="2016-02-25T13:36:09.703" LastActivityDate="2020-08-01T13:12:43.027" Title="Statistics + Computer Science = Data Science?" Tags="|machine-learning|statistics|career|" AnswerCount="3" CommentCount="3" ClosedDate="2014-08-03T06:18:34.900" ContentLicense="CC BY-SA 3.0" />
  <row Id="809" PostTypeId="2" ParentId="808" CreationDate="2014-07-22T09:03:49.270" Score="1" Body="&lt;p&gt;I think that you're on the right track toward becoming an &lt;strong&gt;expert&lt;/strong&gt; &lt;em&gt;data scientist&lt;/em&gt;. Recently I have answered related question &lt;a href=&quot;https://datascience.stackexchange.com/a/742/2452&quot;&gt;here on Data Science StackExchange&lt;/a&gt; (pay attention to the &lt;em&gt;definition&lt;/em&gt; I mention there, as it essentially answers your question by itself, as well as to aspects of &lt;strong&gt;practicing&lt;/strong&gt; &lt;em&gt;software engineering&lt;/em&gt; and &lt;strong&gt;applying&lt;/strong&gt; knowledge to solving &lt;em&gt;real-world&lt;/em&gt; problems). I hope that you will find all that useful. Good luck in your career!&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-01T13:12:43.027" LastActivityDate="2020-08-01T13:12:43.027" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="810" PostTypeId="1" AcceptedAnswerId="811" CreationDate="2014-07-22T12:29:10.050" Score="59" ViewCount="16015" Body="&lt;p&gt;My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should choose a similar data setup for training my models as well. But I came across a research paper or two (in my area of work) which have used a &quot;class balancing&quot; data approach to training the models, implying an equal number of instances of benign and malicious traffic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if I am building machine learning models, should I go for a dataset which is representative of the real world problem, or is a balanced dataset better suited for building the models (since certain classifiers do not behave well with class imbalance, or due to other reasons not known to me)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone shed more light on the &lt;em&gt;pros&lt;/em&gt; and &lt;em&gt;cons&lt;/em&gt; of both the choices and how to decide which one to go choose?&lt;/p&gt;&#xA;" OwnerUserId="2661" LastEditorUserId="26596" LastEditDate="2016-11-29T10:52:50.630" LastActivityDate="2021-02-20T23:23:35.410" Title="Should I go for a 'balanced' dataset or a 'representative' dataset?" Tags="|machine-learning|dataset|class-imbalance|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="811" PostTypeId="2" ParentId="810" CreationDate="2014-07-22T14:22:49.137" Score="41" Body="&lt;p&gt;I would say the answer depends on your use case. Based on my experience:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data.&lt;/li&gt;&#xA;&lt;li&gt;If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-order and plan to use a basic ML framework (i.e. Decision Tree, SVM, Naive Bayes, etc), then I would suggest you feed the framework a balanced dataset. Much of the literature on class imbalance finds that random undersampling (down sampling the majority class to the size of the minority class) can drive performance gains.&lt;/li&gt;&#xA;&lt;li&gt;If you're building a predictive model, but are using a more advanced framework (i.e. something that determines sampling parameters via wrapper or a modification of a bagging framework that samples to class equivalence), then I would suggest again feeding the representative sample and letting the algorithm take care of balancing the data for training.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2513" LastActivityDate="2014-07-22T14:22:49.137" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="812" PostTypeId="1" AcceptedAnswerId="814" CreationDate="2014-07-22T15:17:04.810" Score="28" ViewCount="26847" Body="&lt;p&gt;What is the best tool to use to visualize (draw the vertices and edges) a graph with 1000000 vertices? There are about 50000 edges in the graph. And I can compute the location of individual vertices and edges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking about writing a program to generate a svg. Any other suggestions?  &lt;/p&gt;&#xA;" OwnerUserId="192" LastEditorUserId="471" LastEditDate="2014-07-22T18:34:44.740" LastActivityDate="2021-10-07T13:16:31.077" Title="Visualizing a graph with a million vertices" Tags="|visualization|graphs|" AnswerCount="8" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="813" PostTypeId="2" ParentId="812" CreationDate="2014-07-22T15:44:04.160" Score="8" Body="&lt;p&gt;&lt;a href=&quot;https://gephi.github.io/&quot;&gt;https://gephi.github.io/&lt;/a&gt; says it can handle a million edges. If your graph has 1000000 vertices and only 50000 edges then most of your vertices won't have any edges anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact the Gephi spec is the dual of your example: &quot;Networks up to 50,000 nodes and 1,000,000 edges&quot;&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-22T15:44:04.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="814" PostTypeId="2" ParentId="812" CreationDate="2014-07-22T16:40:17.733" Score="22" Body="&lt;p&gt;I also suggest &lt;code&gt;Gephi&lt;/code&gt; software (&lt;a href=&quot;https://gephi.github.io&quot;&gt;https://gephi.github.io&lt;/a&gt;), which seems to be quite powerful. Some additional information on using &lt;code&gt;Gephi&lt;/code&gt; with &lt;strong&gt;large networks&lt;/strong&gt; can be found &lt;a href=&quot;https://forum.gephi.org/viewtopic.php?t=1554&quot;&gt;here&lt;/a&gt; and, more generally, &lt;a href=&quot;https://forum.gephi.org/viewforum.php?f=25&quot;&gt;here&lt;/a&gt;. &lt;code&gt;Cytoscape&lt;/code&gt; (&lt;a href=&quot;http://www.cytoscape.org&quot;&gt;http://www.cytoscape.org&lt;/a&gt;) is an alternative to &lt;code&gt;Gephi&lt;/code&gt;, being an another popular platform for complex network analysis and visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you'd like to work with networks &lt;strong&gt;programmatically&lt;/strong&gt; (including visualization) in R, Python or C/C++, you can check &lt;code&gt;igraph&lt;/code&gt; collection of libraries. Speaking of R, you may find interesting the following blog posts: on &lt;strong&gt;using R with Cytoscape&lt;/strong&gt; (&lt;a href=&quot;http://www.vesnam.com/Rblog/viznets1&quot;&gt;http://www.vesnam.com/Rblog/viznets1&lt;/a&gt;) and on &lt;strong&gt;using R with Gephi&lt;/strong&gt; (&lt;a href=&quot;http://www.vesnam.com/Rblog/viznets2&quot;&gt;http://www.vesnam.com/Rblog/viznets2&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;extensive lists&lt;/strong&gt; of &lt;em&gt;network analysis and visualization software&lt;/em&gt;, including some comparison and reviews, you might want to check the following pages: 1) &lt;a href=&quot;http://wiki.cytoscape.org/Network_analysis_links&quot;&gt;http://wiki.cytoscape.org/Network_analysis_links&lt;/a&gt;; 2) &lt;a href=&quot;http://www.kdnuggets.com/software/social-network-analysis.html&quot;&gt;http://www.kdnuggets.com/software/social-network-analysis.html&lt;/a&gt;; 3) &lt;a href=&quot;http://www.activatenetworks.net/social-network-analysis-sna-software-review&quot;&gt;http://www.activatenetworks.net/social-network-analysis-sna-software-review&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-07-22T16:40:17.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="815" PostTypeId="1" AcceptedAnswerId="819" CreationDate="2014-07-22T19:20:47.580" Score="5" ViewCount="175" Body="&lt;p&gt;I'm currently facing a project that I could solve with a relational database in a relatively painful way. Having heard so much about NOSQL, I'm wondering if there is not a more appropriate way of tackling it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we are tracking a group of animals in a forest (n ~ 500) and would like to keep a record of a set of observations (this is a fictional scenario).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We would like to store the following information in a database:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a unique identifier for each animal&lt;/li&gt;&#xA;&lt;li&gt;a description of the animal with structured fields: Species, Genus, Family, ...&lt;/li&gt;&#xA;&lt;li&gt;a free text field with additional information&lt;/li&gt;&#xA;&lt;li&gt;each time-point at which it was detected close to a reference point&lt;/li&gt;&#xA;&lt;li&gt;a picture of the animal&lt;/li&gt;&#xA;&lt;li&gt;an indication whether two given animals are siblings&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;there might be additional features appearing later as more data comes in&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We would like to be able to execute the following types of queries:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;return all the animals spotted between in a given time interval&lt;/li&gt;&#xA;&lt;li&gt;return all the animals of a given Species or Family&lt;/li&gt;&#xA;&lt;li&gt;perform a text search on the free text field&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which particular database system would you recommend ? Is there any tutorial / examples that I could use as a starting point ?&lt;/p&gt;&#xA;" OwnerUserId="906" LastActivityDate="2014-07-23T07:01:01.140" Title="is this a good case for NOSQL?" Tags="|nosql|databases|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="816" PostTypeId="2" ParentId="808" CreationDate="2014-07-22T21:12:08.333" Score="9" Body="&lt;p&gt;Well it depends on what kind of &quot;Data Science&quot; you wish to get in to.  For basic analytics and reporting statistics will certainly help, but for Machine Learning and Artificial Intelligence then you'll want a few more skills&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Probability theory&lt;/strong&gt; - you must have a solid background in pure probability so that you can decompose any problem, whether seen before or not, into probabilistic principles.  Statistics helps a lot for already solved problems, but new and unsolved problems require a deep understanding of probability so that you can design appropriate techniques.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Information Theory&lt;/strong&gt; - this (relative to statistics) is quite a new field (though still decades old), the most important work was by Shannon, but even more important and often neglected note in literature is work by Hobson that proved that Kullback-Leibler Divergence is the only mathematical definition that truly captures the notion of a &lt;em&gt;&quot;measure of information&quot;&lt;/em&gt;. Now fundamental to artificial intellgence is being able to quantify information.  Suggest reading &quot;Concepts in Statistical Mechanics&quot; - Arthur Hobson (very expensive book, only available in academic libraries).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Complexity Theory&lt;/strong&gt; - A big problem many Data Scientists face that do not have a solid complexity theory background is that their algorithms do not scale, or just take an extremely long time to run on large data.  Take PCA for example, many peoples favourite answer to the interview question &quot;how do you reduce the number of features in our dataset&quot;, but even if you tell the candidate &quot;the data set is really really really large&quot; they still propose various forms of PCA that are O(n^3).  If you want to stand out, you want to be able to solve each problem on it's own, NOT throw some text book solution at it designed a long time ago before Big Data was such a hip thing.  For that you need to understand how long things take to run, not only theoretically, but practically - so how to use a cluster of computers to distribute an algorithm, or which data structures take up less memory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Communication Skills&lt;/strong&gt; - A huge part of Data Science is understanding business.  Whether it's inventing a product driven by data science, or giving business insight driven by data science, being able to communicate well with both the Project and Product Managers, the tech teams, and your fellow data scientists is very important.  You can have an amazing idea, say an awesome AI solution, but if you cannot effectively (a) communicate WHY that will make the business money, (b) convince your collegues it will work and (c) explain to tech people how you need their help to build it, then it wont get done.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2668" LastActivityDate="2014-07-22T21:12:08.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="817" PostTypeId="2" ParentId="808" CreationDate="2014-07-22T21:24:33.820" Score="6" Body="&lt;p&gt;Data scientist (to me) a big umbrella term. I would see a data scientist as a person who can proficiently use techniques from the fields of data mining, machine learning, pattern classification, and statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, those terms are intertwined to: machine learning is tied together with pattern classification, and also data mining overlaps when it comes finding patterns in data. And all techniques have their underlying statistical principles. I always picture this as a Venn diagram with a huge intersection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computer sciences is related to all those fields too. I would say that you need &quot;data science&quot; techniques to do computer-scientific research, but computer science knowledge is not necessarily implied in &quot;data science&quot;. However, programming skills - I see programming and computer science as different professions, where programming is more the tool in order solve problems - are also important to work with the data and to conduct data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a really nice study plan, and it all makes sense. But I am not sure if you &quot;want&quot; to call yourself just &quot;data scientist&quot;, I have the impression that &quot;data scientist&quot; is such a ambiguous term that can mean everything or nothing. What I want to convey is that you will end up being something more - more &quot;specialized&quot; - than &quot;just&quot; a data scientist.&lt;/p&gt;&#xA;" OwnerDisplayName="user2556" LastActivityDate="2014-07-22T21:24:33.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="818" PostTypeId="2" ParentId="812" CreationDate="2014-07-23T05:29:06.903" Score="8" Body="&lt;p&gt;I think, that &lt;code&gt;Gephi&lt;/code&gt; could face with lack-of-memory issues, you will need at least 8Gb of RAM. Though number of edges is not extremely huge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly, more appropriate tool in this case would be &lt;a href=&quot;http://www.graphviz.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;GraphViz&lt;/code&gt;&lt;/a&gt;. It's a command line tool for network visualizations, and presumably would be more tolerant to graph size. Moreover, as I remember, in &lt;code&gt;GraphViz&lt;/code&gt; it is possible to use precomputed coordinates to facilitate computations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried to find a real-world examples of using &lt;code&gt;GraphViz&lt;/code&gt; with huge graphs, but didn't succeed. Though I found similar discussion on &lt;a href=&quot;https://scicomp.stackexchange.com/questions/3315/visualizing-very-large-link-graphs&quot;&gt;&lt;strong&gt;Computational Science&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="941" LastEditorUserId="-1" LastEditDate="2017-04-13T12:53:55.957" LastActivityDate="2014-07-23T05:29:06.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="819" PostTypeId="2" ParentId="815" CreationDate="2014-07-23T07:01:01.140" Score="7" Body="&lt;p&gt;Three tables: &lt;strong&gt;animal&lt;/strong&gt;, &lt;strong&gt;observation&lt;/strong&gt;, and &lt;strong&gt;sibling&lt;/strong&gt;. The observation has an &lt;strong&gt;animal_id&lt;/strong&gt; column which links to the animal table, and the sibling table has &lt;strong&gt;animal_1_id&lt;/strong&gt; and &lt;strong&gt;animal_2_id&lt;/strong&gt; columns that indicates two animals are siblings for each row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even with 5000 animals and 100000 observations I don't think query time will be a problem for something like PostgreSQL for most reasonable queries (obviously you can construct unreasonable queries but you can do that in any system). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I don't see how this is &quot;relatively painful&quot;. Relative to what? The only complexity is the sibling table. In NOSQL you might store the full list of siblings in the record for each animal, but then when you add a sibling relationship you have to add it to both sibling's animal records. With the relational table approach I've outlined, it only exists once, but at the expense of having to test against both columns to find an animal's siblings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd use PostgreSQL, and that gives you the option of using PostGIS if you have location data - this is a geospatial extension to PostgreSQL that lets you do spatial queries (point in polygon, points near a point etc) which might be something for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really don't think the properties of NOSQL databases are a problem here for you - you aren't changing your schema every ten minutes, you probably &lt;strong&gt;do&lt;/strong&gt; care that your database is ACID-compliant, and you don't need something web-scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mongodb-is-web-scale.com/&quot;&gt;http://www.mongodb-is-web-scale.com/&lt;/a&gt; [warning: strong language]&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-23T07:01:01.140" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="820" PostTypeId="1" AcceptedAnswerId="822" CreationDate="2014-07-23T08:06:21.417" Score="7" ViewCount="114" Body="&lt;p&gt;Apologies if this is very broad question, what I would like to know is how effective is A/B testing (or other methods) of effectively measuring the effects of a design decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance we can analyse user interactions or click results, purchase/ browse decisions and then modify/tailor the results presented to the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could then test the effectiveness of this design change by subjecting 10% of users to the alternative model randomly but then how objective is this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we avoid influencing the user by the model change, for instance we could decided that search queries for 'David Beckham' are probably about football so search results become biased towards this but we could equally say that his lifestyle is just as relevant but this never makes it into the top 10 results that are returned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious how this is dealt with and how to measure this effectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts are that you could be in danger of pushing a model that you think is correct and the user obliges and this becomes a self-fulfilling prophecy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read an article on this: &lt;a href=&quot;http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/&quot;&gt;http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/&lt;/a&gt; and also the book: &lt;a href=&quot;http://shop.oreilly.com/product/0636920028529.do&quot;&gt;http://shop.oreilly.com/product/0636920028529.do&lt;/a&gt; which discussed this so it piqued my interest.&lt;/p&gt;&#xA;" OwnerUserId="95" LastActivityDate="2014-07-23T13:49:01.423" Title="How can we effectively measure the impact of our data decisions" Tags="|search|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="821" PostTypeId="2" ParentId="784" CreationDate="2014-07-23T12:57:37.073" Score="7" Body="&lt;p&gt;&quot;Because its there&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data has a seasonal pattern. So you model it. The data has a trend. So you model it. Maybe the data is correlated with the number of sunspots. So you model that. Eventually you hope to get nothing left to model than uncorrelated random noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think you've screwed up your STL computation here. Your residuals are clearly not serially uncorrelated. I rather suspect you've not told the function that your &quot;seasonality&quot; is a 24-hour cycle rather than an annual one. But hey you haven't given us any code or data so we don't really have a clue what you've done, do we? What do you think &quot;seasonality&quot; even means here? Do you have any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data seems the have three peaks every 24 hours. Really? Is this 'gas'='gasoline'='petrol' or gas in some heating/electric generating system? Either way if you know a priori there's an 8 hour cycle, or an 8 hour cycle on top of a 24 hour cycle on top of what looks like a very high frequency one or two hour cycle you &lt;strong&gt;put that in your model&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually you don't even say what your x-axis is so maybe its days and then I'd fit a daily cycle, a weekly cycle, and then an annual cycle. But given how it all changes at time=85 or so I'd not expect a model to do well on both sides of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With statistics (which is what this is, sorry to disappoint you but you're not a data scientist yet) you don't just robotically go &quot;And.. Now.. I.. Fit.. An... S TL model....&quot;. You look at your data, try and get some understanding, then propose a model, fit it, test it, and use the parameters it make inferences about the data. Fitting cyclic seasonal patterns is part of that.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-23T12:57:37.073" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="822" PostTypeId="2" ParentId="820" CreationDate="2014-07-23T13:49:01.423" Score="4" Body="&lt;p&gt;In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site.  This creates independent samples drawn from the same population.  Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Slight aside&lt;/em&gt;: Now you might argue that the A group or B group may differ in some demographic.  That commonly happens by random chance.  To a certain degree this can be taken care of by covariate adjusted randomization.  It can also be taken care of by adding covariates to the model that tests the effect of the design decision.  It should be noted that there is still some discussion about the proper way to do this within the statistics community.  Essentially A/B testing is an application of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Randomized_controlled_trial&quot; rel=&quot;nofollow&quot;&gt;Randomized Control Trial&lt;/a&gt; to website design.  Some people disagree with adding covariates to the test.  Others, such as Frank Harrel (see &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387952322&quot; rel=&quot;nofollow&quot;&gt;Regression Modeling Strategies&lt;/a&gt;) argue for the use of covariates in such models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would offer the following suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Design the study in advance so as to take care of as much sources of bias and variation as possible.   &lt;/li&gt;&#xA;&lt;li&gt;Let the data speak for itself.  As you get more data (like about searches for David Beckham), let it dominate your assumptions about how the data should be (as how the posterior dominates the prior in Bayesian analysis when the sample size becomes large).    &lt;/li&gt;&#xA;&lt;li&gt;Make sure your data matches the assumptions of the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="178" LastActivityDate="2014-07-23T13:49:01.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="823" PostTypeId="1" AcceptedAnswerId="1162" CreationDate="2014-07-23T14:04:31.057" Score="18" ViewCount="317" Body="&lt;p&gt;There was a recent furore with &lt;a href=&quot;http://online.wsj.com/articles/furor-erupts-over-facebook-experiment-on-users-1404085840&quot;&gt;facebook experimenting on their users to see if they could alter user's emotions&lt;/a&gt; and now &lt;a href=&quot;http://www.bbc.co.uk/news/technology-28542642&quot;&gt;okcupid&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst I am not a professional data scientist I read about &lt;a href=&quot;http://columbiadatascience.com/2013/11/25/data-science-ethics/&quot;&gt;data science ethics&lt;/a&gt; from &lt;a href=&quot;http://shop.oreilly.com/product/0636920028529.do&quot;&gt;Cathy O'Neill's book 'Doing Data Science'&lt;/a&gt; and would like to know if this is something that professionals are taught at academic level (I would expect so) or something that is ignored or is lightly applied in the professional world. Particularly for those who ended up doing data science &lt;em&gt;accidentally&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst the linked article touched on data integrity, the book also discussed the moral ethics behind understanding the impact of the data models that are created and the impact of those models which can have adverse effects when used inappropriately (sometimes unwittingly) or when the models are inaccurate, again producing adverse results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article discusses a code of practice and mentions the &lt;a href=&quot;http://www.datascienceassn.org/code-of-conduct.html&quot;&gt;Data Science Association's Code of conduct&lt;/a&gt;, is this something that is in use? Rule 7 is of particular interest (quoted from their website):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(a) A person who consults with a data scientist about the possibility&#xA;  of forming a client-data scientist relationship with respect to a&#xA;  matter is a prospective client.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(b) Even when no client-data scientist relationship ensues, a data&#xA;  scientist who has learned information from a prospective client shall&#xA;  not use or reveal that information.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(c) A data scientist subject to paragraph (b) shall not provide&#xA;  professional data science services for a client with interests&#xA;  materially adverse to those of a prospective client in the same or a&#xA;  substantially related industry if the data scientist received&#xA;  information from the prospective client that could be significantly&#xA;  harmful to that person in the matter&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is this something that is practiced professionally? Many users blindly accept that we get some free service (mail, social network, image hosting, blog platform etc..) and agree to an EULA in order to have ads pushed at us. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally how is this regulated, I often read about users being up in arms when the terms of a service change but it seems that it requires some liberty organisation, class action or a &lt;a href=&quot;http://www.cnet.com/news/senator-asks-ftc-to-investigate-facebooks-mood-study/&quot;&gt;senator&lt;/a&gt; to react to such things before something happens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way I am not making any judgements here or saying that all data scientists behave like this, I'm interested in what is taught academically and practiced professionally.&lt;/p&gt;&#xA;" OwnerUserId="95" LastEditorUserId="95" LastEditDate="2014-07-29T12:39:59.630" LastActivityDate="2016-05-27T14:02:34.900" Title="How should ethics be applied in data science" Tags="|social-network-analysis|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="824" PostTypeId="2" ParentId="791" CreationDate="2014-07-23T15:46:48.977" Score="1" Body="&lt;p&gt;(too long for a comment)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, @Emre's answer is correct: simple correlation matrix and cosine distance should work well*. There's one subtlety, though - job titles are too short to carry important context. Let me explain this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine LinkedIn profiles (which is pretty good source for data). Normally, they contain 4-10 sentences describing person's skills and qualifications. It's pretty likely that you find phrases like &quot;lead data scientist&quot; and &quot;professional knowledge of Matlab and R&quot; in a same profile, but it's very unlikely to also see &quot;junior Java developer&quot; in it. So we may say that &quot;lead&quot; and &quot;professional&quot; (as well as &quot;data scientist&quot; and &quot;Matlab&quot; and &quot;R&quot;) often occur in same contexts, but they are rarely found together with &quot;junior&quot; and &quot;Java&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Co-occurrence matrix shows exactly this. The more 2 words occur in same context, the more similar their vectors in the matrix will look like. And cosine distance is just a good way to measure this similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about job titles? Normally they are much shorter and don't actually create enough context to catch similarities. Luckily, you don't need source data to be titles themselves - you need to find similarities between skills in general, not specifically in titles. So you can simply build co-occurrence matrix from (long) profiles and then use it to measure similarity of titles. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;*&lt;/em&gt; - in fact, it's already worked for me on a similar project. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-23T15:46:48.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="825" PostTypeId="1" CreationDate="2014-07-24T00:01:40.760" Score="5" ViewCount="228" Body="&lt;p&gt;Most vehicle license/number plate extractors I've found involve reading a plate from an image (OCR) but I'm interested in something that could tag instances of license plates in a body of text. Are there any such annotators out there?&lt;/p&gt;&#xA;" OwnerUserId="1192" LastActivityDate="2014-07-24T02:23:23.703" Title="Are there any annotators or Named Entity Recognition for license plate numbers?" Tags="|text-mining|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="826" PostTypeId="2" ParentId="825" CreationDate="2014-07-24T00:34:41.130" Score="-1" Body="&lt;p&gt;This can be done using &lt;code&gt;regular expressions&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 letters followed by a number (\d denotes digits) would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[A-Z]{2} \d*&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2 or 3 letters followed by a number is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[A-Z]{2,3} \d*&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="325" LastActivityDate="2014-07-24T00:34:41.130" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="827" PostTypeId="2" ParentId="825" CreationDate="2014-07-24T02:23:23.703" Score="4" Body="&lt;p&gt;There are a lot of pretty decent tools out there for text annotation in general, and given the broad nature of the task you're approaching (license plates are about as general as words), the annotation tools you are looking at should probably come from the more classical tools for annotation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There was actually a pretty good discussion about annotation tools on &lt;a href=&quot;https://datascience.stackexchange.com/questions/223/how-to-annotate-text-documents-with-meta-data/404#404&quot;&gt;this&lt;/a&gt; question, which should actually apply to this problem. The most relied-upon thing in annotation right now is probably &lt;code&gt;brat&lt;/code&gt;. You can learn more about &lt;code&gt;brat&lt;/code&gt; &lt;a href=&quot;http://brat.nlplab.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps! Let me know if you've got any more questions.&lt;/p&gt;&#xA;" OwnerUserId="548" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-07-24T02:23:23.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="828" PostTypeId="2" ParentId="812" CreationDate="2014-07-24T03:31:32.087" Score="4" Body="&lt;p&gt;Reporting back: I ended up coding graphml and using yEd for visualization (just because I am familiar with this combination. I bet gephi or graphviz would work fine and might even be better). Since I computed the location of all nodes, memory was not such big of an issue. Coding graphml is a little easier comparing to coding svg, since I don't have to explicitly specify the placement of edges.&lt;/p&gt;&#xA;" OwnerUserId="192" LastActivityDate="2014-07-24T03:31:32.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="829" PostTypeId="1" AcceptedAnswerId="830" CreationDate="2014-07-24T06:26:07.290" Score="4" ViewCount="98" Body="&lt;p&gt;While running the below pig script I am getting an error in line4: &#xA;If it is GROUP then I am getting error.&#xA;If I change from 'GROUP' TO 'group' in line4, then the script is running.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between group and GROUP?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );&#xA;&#xA;WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;&#xA;&#xA;WORDSGROUPED = GROUP WORDS BY ZIPS;&#xA;&#xA;WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);&#xA;&#xA;WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;&#xA;&#xA;DUMP WORDSSORT;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1314" LastEditorUserId="29169" LastEditDate="2019-06-07T17:09:20.403" LastActivityDate="2019-06-07T17:09:20.403" Title="Pig script code error?" Tags="|bigdata|apache-hadoop|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="830" PostTypeId="2" ParentId="829" CreationDate="2014-07-24T07:00:13.187" Score="2" Body="&lt;p&gt;'group' in strictly lower case in the FOREACH is the thing you are looping/grouping over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://squarecog.wordpress.com/2010/05/11/group-operator-in-apache-pig/&quot; rel=&quot;nofollow&quot;&gt;http://squarecog.wordpress.com/2010/05/11/group-operator-in-apache-pig/&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When you group a relation, the result is a new relation with two&#xA;  columns: “group” and the name of the original relation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Column names are case sensitive, so you have to use lower-case 'group' in your FOREACH.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'GROUP' in upper case is the grouping operator. You can't mix them. So don't do that.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-24T07:00:13.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="831" PostTypeId="1" CreationDate="2014-07-24T07:38:07.163" Score="5" ViewCount="96" Body="&lt;p&gt;Have you heard of the &quot;Data Science Association&quot;? &#xA;&lt;br&gt;URL: &lt;a href=&quot;http://www.datascienceassn.org/&quot; rel=&quot;noreferrer&quot;&gt;http://www.datascienceassn.org/&lt;/a&gt;&#xA;&lt;br&gt;Do you expect it to become a professional body like the Actuaries Institute?&#xA;&lt;br&gt;If yes, then why?&#xA;&lt;br&gt;If no, then why not and do you see anyone else becoming the professional body?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, is this question &quot;on-topic&quot; ?&lt;/p&gt;&#xA;" OwnerUserId="366" LastEditorUserId="366" LastEditDate="2014-07-24T09:32:20.100" LastActivityDate="2014-07-24T09:32:20.100" Title="Data Science Association?" Tags="|knowledge-base|" AnswerCount="0" CommentCount="4" ClosedDate="2014-07-26T15:11:29.480" ContentLicense="CC BY-SA 3.0" />
  <row Id="832" PostTypeId="1" CreationDate="2014-07-24T14:04:09.533" Score="7" ViewCount="1751" Body="&lt;p&gt;My data contains a set of start times and duration for an action. I would like to plot this so that for a given time slice I can see how many actions are active. I'm currently thinking of this as a histogram with time on the x axis and number of active actions on the y axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how should I adjust the data so that this is able to be plotted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The times for an action can be between 2 seconds and a minute. And, at any given time I would estimate there could be about 100 actions taking place. Ideally a single plot would be able to show hours of data. The accuracy of the data is in milliseconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past the way that I have done this is to count for each second how many actions started , ended, or were active. This gave me a count of active actions for each second. The issue I found with this technique was that it made it difficult to adjust the time slice that I was looking at. Looking at a time slice of a minute was difficult to compute and looking at time slices of less than a second was impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm open to any advice on how to think about this issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="2702" LastEditorUserId="2702" LastEditDate="2014-07-25T18:30:26.490" LastActivityDate="2014-07-26T21:03:13.470" Title="How do you plot overlapping durations?" Tags="|visualization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="833" PostTypeId="2" ParentId="832" CreationDate="2014-07-24T21:14:45.793" Score="1" Body="&lt;p&gt;This can be done in &lt;code&gt;R&lt;/code&gt; using &lt;code&gt;ggplot&lt;/code&gt;. Based on &lt;a href=&quot;https://stackoverflow.com/questions/18102224/drawing-gantt-charts-with-r-to-sub-second-accuracy&quot;&gt;this&lt;/a&gt; question, it could be done with this code where &lt;code&gt;limits&lt;/code&gt; is the date range of the plot.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tasks &amp;lt;- c(&quot;Task1&quot;, &quot;Task2&quot;)&#xA;dfr &amp;lt;- data.frame(&#xA;name        = factor(tasks, levels = tasks),&#xA;start.date  = c(&quot;2014-08-07 09:03:25.815&quot;, &quot;2014-08-07 09:03:25.956&quot;),&#xA;end.date    = c(&quot;2014-08-07 09:03:28.300&quot;, &quot;2014-08-07 09:03:30.409&quot;)&#xA;)&#xA;&#xA;mdfr &amp;lt;- melt(dfr, measure.vars = c(&quot;start.date&quot;, &quot;end.date&quot;))&#xA;&#xA;&#xA;mdfr$time&amp;lt;-as.POSIXct(mdfr$value)&#xA;&#xA;ggplot(mdfr, aes(time,name)) + &#xA;geom_line(size = 6) +&#xA;xlab(&quot;&quot;) + ylab(&quot;&quot;) +&#xA;theme_bw()+&#xA;scale_x_datetime(breaks=date_breaks(&quot;2 sec&quot;),&#xA;limits = as.POSIXct(c('2014-08-07 09:03:24','2014-08-07 09:03:29')))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Bwwod.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="325" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-07-24T21:14:45.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="834" PostTypeId="1" CreationDate="2014-07-25T00:58:12.253" Score="17" ViewCount="8791" Body="&lt;p&gt;I am trying to build a recommendation system using collaborative filtering. I have the usual &lt;code&gt;[user, movie, rating]&lt;/code&gt; information. I would like to incorporate an additional feature like 'language' or 'duration of movie'. I am not sure what techniques I could use for such a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest references or packages in python/R. &lt;/p&gt;&#xA;" OwnerUserId="1131" LastEditorUserId="29169" LastEditDate="2019-06-08T03:47:52.223" LastActivityDate="2021-02-18T20:43:25.357" Title="Recommending movies with additional features using collaborative filtering" Tags="|python|r|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="835" PostTypeId="2" ParentId="793" CreationDate="2014-07-25T03:05:38.107" Score="4" Body="&lt;p&gt;Consider, try, and perhaps even use multiple databases. It's not just a &quot;performance&quot; issue at play here. It's really going to come down to your requirements. How much data are you talking about? what kind of data? how fast do you need it? Are you more read heavy or write heavy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one thing you can't do in a SQL database: Calculate sentiment. &lt;a href=&quot;http://www.slideshare.net/shift8/mongodb-machine-learning&quot; rel=&quot;nofollow&quot;&gt;http://www.slideshare.net/shift8/mongodb-machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the speed in that case may not be fast enough for your needs, but it is something that's possible. With some caching of specific aggregate values, it was quite acceptable even. Why would you do this? Convenience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convenience really is something that you're going to be persuaded by. That's exactly why (in my opinion) NoSQL databases were created. Performance too of course, but I'm trying to discount benchmarks and focus more on other concerns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MongoDB (and some other NoSQL) databases have some very powerful features such as built-in map/reduce. This could result in a savings both in cost and time over using something like Hadoop. Or it could provide a prototype or MVP to launch a larger business.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about graph databases? They're &quot;NoSQL&quot; too. Look at databases like OrientDB. If you want to argue performance ...I don't think you're gonna show me a SQL database that's faster there =) ...and graph databases have some really amazing application based on what you need to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rule of technology (and the internet) don't get too comfortable with one thing. You're gonna be limited and set yourself up for failure.&lt;/p&gt;&#xA;" OwnerUserId="2711" LastActivityDate="2014-07-25T03:05:38.107" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="836" PostTypeId="2" ParentId="834" CreationDate="2014-07-25T04:54:07.787" Score="17" Body="&lt;p&gt;Here some resources that might be helpful:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://lyle.smu.edu/IDA/recommenderlab&quot; rel=&quot;nofollow noreferrer&quot;&gt;Recommenderlab&lt;/a&gt;&lt;/strong&gt; - a framework and open source software for developing and testing recommendation algorithms. Corresponding &lt;a href=&quot;http://cran.r-project.org/package=recommenderlab&quot; rel=&quot;nofollow noreferrer&quot;&gt;R package recommenderlab&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The following &lt;a href=&quot;http://scn.sap.com/community/developer-center/hana/blog/2013/11/06/movie-recommendation-by-leveraging-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; illustrates the use of &lt;code&gt;recommenderlab&lt;/code&gt; package (which IMHO can be generalized for any open source recommendation engine) for building &lt;strong&gt;movie recommendation&lt;/strong&gt; application, based on &lt;em&gt;collaborative filtering&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://michael.hahsler.net/research/recommender&quot; rel=&quot;nofollow noreferrer&quot;&gt;Research on recommender systems&lt;/a&gt;&lt;/strong&gt; - a nice webpage with resources on the topic, maintained by Recommenderlab's lead developer Michael Hahsler.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/mortardata/mortar-recsys&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mortar Recommendation Engine&lt;/a&gt;&lt;/strong&gt; - an open source customizable recommendation engine for &lt;code&gt;Hadoop&lt;/code&gt; and &lt;code&gt;Pig&lt;/code&gt;, written in &lt;code&gt;Python&lt;/code&gt; and &lt;code&gt;Java&lt;/code&gt;. Company, sponsoring the development of this project, &lt;strong&gt;Mortar Data&lt;/strong&gt;, offers general commercial &lt;em&gt;cloud platform&lt;/em&gt; for development and hosting &lt;em&gt;data science&lt;/em&gt; software projects, including ones based on the &lt;code&gt;Mortar Recommendation Engine&lt;/code&gt; (development and hosting of public projects are free): &lt;a href=&quot;http://www.mortardata.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.mortardata.com&lt;/a&gt;. Mortar Data provides help in form of &lt;strong&gt;public Q&amp;amp;A forum&lt;/strong&gt; (&lt;a href=&quot;https://answers.mortardata.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://answers.mortardata.com&lt;/a&gt;) as well as a &lt;strong&gt;comprehensive tutorial&lt;/strong&gt; on building recommendation engine using open technologies (&lt;a href=&quot;http://help.mortardata.com/data_apps/recommendation_engine&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://help.mortardata.com/data_apps/recommendation_engine&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.coursera.org/course/recsys&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;Introduction to Recommender Systems&amp;quot;&lt;/a&gt;&lt;/strong&gt; - a relevant Coursera course (MOOC), which content and description provide additional resources on the topic.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://prediction.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;PredictionIO&lt;/a&gt;&lt;/strong&gt; - an open source machine learning server software, which allows building &lt;em&gt;data science applications&lt;/em&gt;, including &lt;em&gt;recommendation systems&lt;/em&gt;  (&lt;a href=&quot;https://github.com/PredictionIO&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt; is available on GitHub). &lt;code&gt;PredictionIO&lt;/code&gt; includes a built-in &lt;em&gt;recommendation engine&lt;/em&gt; (&lt;a href=&quot;http://docs.prediction.io/current/engines/itemrec/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://docs.prediction.io/current/engines/itemrec/index.html&lt;/a&gt;) and supports a wide range of programming languages and frameworks via &lt;code&gt;RESTful&lt;/code&gt; APIs as well as SDKs/plug-ins. PredictionIO maintains an &lt;code&gt;Amazon Machine Image&lt;/code&gt; on &lt;strong&gt;&lt;a href=&quot;https://aws.amazon.com/marketplace/pp/B00ECGJYGE&quot; rel=&quot;nofollow noreferrer&quot;&gt;AWS Marketplace&lt;/a&gt;&lt;/strong&gt; for deploying applications on the AWS infrastructure.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Additional open source software projects&lt;/strong&gt;, relevant to the topic (discovered via &lt;code&gt;MLOSS&lt;/code&gt; website on &lt;em&gt;machine learning&lt;/em&gt; open source software: &lt;a href=&quot;http://www.mloss.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.mloss.org&lt;/a&gt;):&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://jubat.us/en&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jubatus&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://mymedialite.net&quot; rel=&quot;nofollow noreferrer&quot;&gt;MyMediaLite&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/ChrisRackauckas/TBEEF&quot; rel=&quot;nofollow noreferrer&quot;&gt;TBEEF&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://prea.gatech.edu&quot; rel=&quot;nofollow noreferrer&quot;&gt;PREA&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://www.cofirank.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;CofiRank&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The following relevant &lt;code&gt;R&lt;/code&gt; &lt;strong&gt;blog posts&lt;/strong&gt; are also interesting:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.revolutionanalytics.com/2012/04/simple-tools-for-building-a-recommendation-engine.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;Simple tools for building a recommendation engine&amp;quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.yhathq.com/posts/recommender-system-in-r.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;Recommendation System in R&amp;quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="29169" LastEditDate="2021-02-18T20:43:25.357" LastActivityDate="2021-02-18T20:43:25.357" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="837" PostTypeId="2" ParentId="783" CreationDate="2014-07-25T12:59:00.850" Score="1" Body="&lt;p&gt;I would look at the &lt;code&gt;raster&lt;/code&gt; package for this, which can read in raw binary data and present it as NxM grids. It can even extract subsets of large binary grids without having to read in the whole file (the R raster object itself is just a proxy to the data, not the data itself).&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-07-25T12:59:00.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="838" PostTypeId="1" CreationDate="2014-07-24T06:34:50.083" Score="3" ViewCount="113" Body="&lt;p&gt;While running the below pig script I am getting error in line4: If it is &lt;code&gt;GROUP&lt;/code&gt; then I am getting error. If I change from &lt;code&gt;GROUP&lt;/code&gt; TO &lt;code&gt;group&lt;/code&gt; in line4, then the script is running. What is the difference between group and GROUP.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );&#xA;&#xA;WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;&#xA;&#xA;WORDSGROUPED = GROUP WORDS BY ZIPS;&#xA;&#xA;WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);&#xA;&#xA;WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;&#xA;&#xA;DUMP WORDSSORT;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1314" OwnerDisplayName="user2576028" LastActivityDate="2014-07-25T15:02:21.217" Title="Pig latin code error" Tags="|apache-hadoop|bigdata|" AnswerCount="1" CommentCount="0" ClosedDate="2014-07-25T16:26:00.903" ContentLicense="CC BY-SA 3.0" />
  <row Id="839" PostTypeId="2" ParentId="838" CreationDate="2014-07-25T09:05:09.493" Score="0" Body="&lt;p&gt;When we do grouping of the data, Pig creates a new key named &quot;group&quot; and puts all the tuple matching that key into a bag and associates the bag with the key. So after the group operation schema of the grouped data will be something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;raw = load '$input' using PigStorage('\u0001') as (id1:int, name:chararray);&#xA;groupdata1 = group raw by (id1,name);  &#xA;describe groupdata1;&#xA;{group: (id1: int,name: chararray),raw: {(id1: int,name: chararray)}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The 'GROUP' in line 4 you are trying to access is one of the attribute of the schema from the last statement. These attribute name are case sensitive. It will produce the error saying it doesn't exist in the schema. So you need to use 'group' only to access it.&lt;/p&gt;&#xA;" OwnerUserId="63174" OwnerDisplayName="abhishek bafna" LastActivityDate="2014-07-25T09:05:09.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="840" PostTypeId="1" CreationDate="2014-07-25T17:18:21.393" Score="2" ViewCount="555" Body="&lt;p&gt;I've been trying to create a similarity matrix in pandas from with a matrix multiplication operation on a document-term count matrix with 2264 rows and 20475 columns.&lt;br /&gt;&#xA;The calculation completes in IPython but inspection shows the results all come back as &lt;code&gt;NaN&lt;/code&gt;.&lt;br /&gt;&#xA;I've also tried doing the same job in numpy, tried converting the original matrix &lt;code&gt;to_sparse&lt;/code&gt; and even re-casting the values as integers, but still no joy.&lt;br /&gt;&#xA;Can anyone suggest the best approach to tackle the problem?&lt;/p&gt;&#xA;&lt;p&gt;EDIT: Here's my code thus far:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;path = &amp;quot;../../reuters.db&amp;quot;&#xA;%pylab inline&#xA;import pandas as pd&#xA;import numpy as np&#xA;import pandas.io.sql as psql&#xA;import sqlite3 as lite&#xA;con = lite.connect(path)&#xA;with con:&#xA;    sql = &amp;quot;SELECT * FROM Frequency&amp;quot;&#xA;    df = psql.frame_query(sql, con)&#xA;    print df.shape&#xA;df = df.rename(columns={&amp;quot;term&amp;quot;:&amp;quot;term_id&amp;quot;, &amp;quot;count&amp;quot;:&amp;quot;count_id&amp;quot;})&#xA;pivoted = df.pivot('docid', 'term_id', 'count_id')&#xA;pivoted.to_sparse()&#xA;similarity_matrix = pivoted.dot(pivoted.T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="974" LastEditorUserId="98307" LastEditDate="2020-07-31T14:25:36.850" LastActivityDate="2020-07-31T14:25:36.850" Title="How to fix similarity matrix in pandas returning all NaNs?" Tags="|pandas|similarity|" AnswerCount="0" CommentCount="2" ClosedDate="2020-10-19T03:05:44.570" ContentLicense="CC BY-SA 4.0" />
  <row Id="841" PostTypeId="2" ParentId="834" CreationDate="2014-07-25T18:12:49.847" Score="5" Body="&lt;p&gt;Instead of collaborative filtering I would use the matrix factorization approach, wherein users and movies alike a represented by vectors of latent features whose dot products yield the ratings. Normally one merely selects the rank (number of features) without regard to what the features represent, and the algorithm does the rest. Like PCA, the result is not immediately interpretable but it yields good results. What you want to do is extend the movie matrix to include the additional features you mentioned and make sure that they stay fixed as the algorithm estimates the two matrices using regularizastion. The corresponding entries in the user matrix will be initialized randomly, then estimated by the matrix factorization algorithm. It's a versatile and performant approach but it takes some understanding of machine learning, or linear algebra at least.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw a nice ipython notebook a while back but I can't find it right now, so I'll refer you to &lt;a href=&quot;http://nbviewer.ipython.org/github/diktat/CPSC540machinelearning/blob/master/1.4%20Collaborative%20Filtering%20for%20Movie%20Recommendation.ipynb&quot; rel=&quot;noreferrer&quot;&gt;another one&lt;/a&gt; which, while not as nice, still clarifies some of the maths.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-25T18:12:49.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="842" PostTypeId="1" AcceptedAnswerId="843" CreationDate="2014-07-25T18:36:31.340" Score="28" ViewCount="42880" Body="&lt;p&gt;I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have just started with Data Science and Machine learning. I am looking for long term project ideas which I can work on for like 8 months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mix of Data Science and Machine learning would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A project big enough to help me understand the core concepts and also implement them at the same time would be very beneficial.&lt;/p&gt;&#xA;" OwnerUserId="2725" LastEditorUserId="1352" LastEditDate="2014-07-27T03:35:06.853" LastActivityDate="2020-08-20T18:57:12.080" Title="Data Science Project Ideas" Tags="|machine-learning|bigdata|dataset|" AnswerCount="3" CommentCount="0" ClosedDate="2014-07-26T15:09:56.510" ContentLicense="CC BY-SA 3.0" />
  <row Id="843" PostTypeId="2" ParentId="842" CreationDate="2014-07-25T20:50:14.540" Score="29" Body="&lt;p&gt;I would try to analyze and solve one or more of the problems published on &lt;strong&gt;&lt;a href=&quot;https://www.kaggle.com/competitions&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Competitions&lt;/a&gt;&lt;/strong&gt;. Note that the competitions are grouped by their expected &lt;em&gt;complexity&lt;/em&gt;, from &lt;code&gt;101&lt;/code&gt; (bottom of the list) to &lt;code&gt;Research&lt;/code&gt; and &lt;code&gt;Featured&lt;/code&gt; (top of the list). A color-coded vertical band is a &lt;em&gt;visual guideline&lt;/em&gt; for grouping. You can &lt;strong&gt;assess time&lt;/strong&gt; you could spend on a project by &lt;strong&gt;adjusting&lt;/strong&gt; the expected &lt;em&gt;length&lt;/em&gt; of corresponding competition, based on your &lt;em&gt;skills&lt;/em&gt; and &lt;em&gt;experience&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;A number of &lt;strong&gt;data science project ideas&lt;/strong&gt; can be found by browsing &lt;a href=&quot;https://www.coursolve.org/browse-needs?query=Data%20Science&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Coursolve&lt;/code&gt;&lt;/a&gt; webpage.&lt;/p&gt;&#xA;&lt;p&gt;If you have skills and desire to work on a &lt;strong&gt;real data science project&lt;/strong&gt;, focused on &lt;strong&gt;social impacts&lt;/strong&gt;, visit &lt;a href=&quot;http://www.datakind.org/projects&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;DataKind&lt;/code&gt;&lt;/a&gt; projects page. More projects with social impacts focus can be found at &lt;a href=&quot;http://dssg.io/projects&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Data Science for Social Good&lt;/code&gt;&lt;/a&gt; webpage.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Science Project Ideas&lt;/strong&gt; page at &lt;a href=&quot;http://mynasadata.larc.nasa.gov/804-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;My NASA Data&lt;/code&gt;&lt;/a&gt; site looks like another place to visit for inspiration.&lt;/p&gt;&#xA;&lt;p&gt;If you would like to use &lt;strong&gt;open data&lt;/strong&gt;, this long list of applications on &lt;code&gt;Data.gov&lt;/code&gt; can provide you with some interesting &lt;em&gt;&lt;a href=&quot;http://www.data.gov/applications&quot; rel=&quot;nofollow noreferrer&quot;&gt;data science&lt;/a&gt;&lt;/em&gt; project ideas.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-20T18:57:12.080" LastActivityDate="2020-08-20T18:57:12.080" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="844" PostTypeId="1" CreationDate="2014-07-25T21:03:44.663" Score="7" ViewCount="2578" Body="&lt;p&gt;so I'm using Spark to do sentiment analysis, and I keep getting errors with the serializers it uses (I think) to pass python objects around.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PySpark worker failed with exception:&#xA;Traceback (most recent call last):&#xA;  File &quot;/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/worker.py&quot;, line 77, in main&#xA;    serializer.dump_stream(func(split_index, iterator), outfile)&#xA;  File &quot;/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py&quot;, line 191, in dump_stream&#xA;    self.serializer.dump_stream(self._batched(iterator), stream)&#xA;  File &quot;/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py&quot;, line 123, in dump_stream&#xA;    for obj in iterator:&#xA;  File &quot;/Users/abdul/Desktop/RSI/spark-1.0.1-bin-    hadoop1/python/pyspark/serializers.py&quot;, line 180, in _batched&#xA;    for item in iterator:&#xA;TypeError: __init__() takes exactly 3 arguments (2 given)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the code for serializers is available &lt;a href=&quot;https://spark.apache.org/docs/latest/api/python/pyspark.serializers-pysrc.html#PickleSerializer.dumps&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and my code is &lt;a href=&quot;https://github.com/seashark97/Scalable-Sentiment-Analysis/blob/master/spark_test.py&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2726" LastEditorUserId="21" LastEditDate="2016-05-10T13:59:41.870" LastActivityDate="2016-05-10T13:59:41.870" Title="Using Apache Spark to do ML. Keep getting serializing errors" Tags="|apache-spark|pyspark|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="845" PostTypeId="2" ParentId="844" CreationDate="2014-07-26T00:11:03.637" Score="10" Body="&lt;p&gt;Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to &lt;code&gt;map&lt;/code&gt;) has &lt;strong&gt;dependencies&lt;/strong&gt; on &lt;strong&gt;non-serializable data&lt;/strong&gt;. Consider following example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rdd = sc.parallelize(range(5))&#xA;rdd = rdd.map(lambda x: x + 1)&#xA;rdd.collect()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here you have distributed collection and lambda function to send to all workers. Lambda is completely self-containing, so it's easy to copy its binary representation to other nodes without any worries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's make things a bit more interesting: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f = open(&quot;/etc/hosts&quot;)&#xA;rdd = sc.parallelize(range(100))&#xA;rdd = rdd.map(lambda x: f.read())&#xA;rdd.collect()&#xA;f.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Boom! Strange error in serialization module! What just happened is that we had attempted to pass &lt;code&gt;f&lt;/code&gt;, which is a file object, to workers. Obviously, file object is a handle to &lt;em&gt;local&lt;/em&gt; data and thus cannot be sent to other machines. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So what's happening in your specific code? Without actual data and knowing record format, I cannot debug it completely, but I guess that problem goes from this line: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def vectorizer(text, vocab=vocab_dict):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In Python, keyword arguments are initialized when function is called for the first time. When you call &lt;code&gt;sc.parallelize(...).map(vectorizer)&lt;/code&gt; just after its definition, &lt;code&gt;vocab_dict&lt;/code&gt; is available &lt;em&gt;locally&lt;/em&gt;, but &lt;em&gt;remote&lt;/em&gt; workers know absolutely nothing about it. Thus function is called with fewer parameters than it expects which results in &lt;code&gt;__init__() takes exactly 3 arguments (2 given)&lt;/code&gt; error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note, that you follow very bad pattern of  &lt;code&gt;sc.parallelize(...)...collect()&lt;/code&gt; calls. First you spread your collection to entire cluster, do some computations, and then pull the result. But sending data back and forth is pretty pointless here. Instead, you can just do these computations locally, and run Spark's parallel processes only when you work with really big datasets (like you main &lt;code&gt;amazon_dataset&lt;/code&gt;, I guess).&lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-26T00:11:03.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="846" PostTypeId="2" ParentId="842" CreationDate="2014-07-26T01:12:08.167" Score="6" Body="&lt;p&gt;Take something from your everyday life. Create predictor of traffic jams in your region, craft personalised music recommender, analyse car market, etc. Choose &lt;strong&gt;real problem&lt;/strong&gt; that you &lt;strong&gt;want to solve&lt;/strong&gt; - this will not only keep you motivated, but also make you go through the whole development circle from data collection to hypothesis testing. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-26T01:12:08.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="847" PostTypeId="2" ParentId="842" CreationDate="2014-07-26T08:37:52.823" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/course/datasci&quot; rel=&quot;nofollow&quot;&gt;Introduction to Data Science&lt;/a&gt; course that is being run on Coursera now includes real-world project assignment where companies post their problems and students are encouraged to solve them. This is done via &lt;a href=&quot;https://www.coursolve.org/&quot; rel=&quot;nofollow&quot;&gt;coursolve.com&lt;/a&gt; (already mentioned here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information &lt;a href=&quot;https://class.coursera.org/datasci-002/wiki/OptionalRealWorldProject&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (you have to be enrolled in the course to see that link)&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-07-26T08:37:52.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="848" PostTypeId="2" ParentId="761" CreationDate="2014-07-26T16:04:25.363" Score="71" Body="&lt;p&gt;K-means is not the most appropriate algorithm here.&lt;/p&gt;&#xA;&lt;p&gt;The reason is that k-means is designed to &lt;strong&gt;minimize variance&lt;/strong&gt;. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not &amp;quot;linear&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;Since your data is in latitude, longitude format, you should use an algorithm that can handle &lt;em&gt;arbitrary&lt;/em&gt; distance functions, in particular geodetic distance functions. Hierarchical clustering, PAM, CLARA, and DBSCAN are popular examples of this.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=QsGOoWdqaT8&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; recommends OPTICS clustering.&lt;/p&gt;&#xA;&lt;p&gt;The problems of k-means are easy to see when you consider points close to the +-180 degrees wrap-around. Even if you hacked k-means to use Haversine distance, in the update step when it recomputes the &lt;em&gt;mean&lt;/em&gt; the result will be badly screwed. &lt;strong&gt;Worst case is, k-means will never converge!&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="98307" LastEditDate="2020-08-02T14:02:53.883" LastActivityDate="2020-08-02T14:02:53.883" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="849" PostTypeId="2" ParentId="832" CreationDate="2014-07-26T21:03:13.470" Score="2" Body="&lt;p&gt;Since you want to show so much data, I think that your best choice is going interactive. Check out this &lt;a href=&quot;http://demo.zunzun.se/intervals/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;demo&lt;/a&gt;, it is close to what you want but not quite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is very difficult to show a lot of data in a single diagram, together with the finest details and the bird-eyes view. But you can let the user interact and look for the details. To show counts, one option is to use color-coding. Take a look at this image (code &lt;a href=&quot;https://gist.github.com/dsign/4c598cfbfc81f6d491d5&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;): &lt;img src=&quot;https://i.stack.imgur.com/xmgHi.png&quot; alt=&quot;image&quot;&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here rgb channels have been used to encode (the logarithm of) the number of active events (red), events starting (green) and events ending (blue) for windows of different size. The X axis is time, and the Y axis represents window size, that is, duration. Thus, a point with coordinates (10, 4) represents the interval of time that goes from 10 to 14. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a lot of data more detailed, it could be a good idea to make the diagram zoomable (like in the demo before), and to give the user the possibility of visualizing just one channel/magnitude.&lt;/p&gt;&#xA;" OwnerUserId="2575" LastActivityDate="2014-07-26T21:03:13.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="850" PostTypeId="2" ParentId="748" CreationDate="2014-07-27T03:58:22.907" Score="6" Body="&lt;p&gt;If you've got prior information then you should certainly not use simple mean in a split test. I assume you're trying to just predict which group will produce the greatest amount of revenue overall, by trying to emulate the underlying distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, it's worth noting that any metrics you choose will actually reduce to mean in a pretty trivial way. Eventually mean will necessarily work out, though using a standard bayesian method to estimate the mean is probably your best bet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you've got a prior then using a standard bayesian approach to update the prior on your mean revenue is probably the best way to do it. Basically, just take the individual results you get and update a multinomial distribution representing your prior in each case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want some more full background on multinomial distributions as bayesian priors are pretty well, &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/minka/papers/minka-multinomial.pdf&quot;&gt;this&lt;/a&gt; Microsoft paper does a pretty good job of outlining it. In general, I wouldn't care so much about the fact that your distribution is technically discrete, as a multinomial distribution will effectively interpolate across your solution space, giving you a continuous distribution that is a very good approximation of your discrete space.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-07-27T03:58:22.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="851" PostTypeId="1" CreationDate="2014-07-27T21:54:05.003" Score="6" ViewCount="789" Body="&lt;h2&gt;Image Similarity based on Color Palette Distribution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as follows,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Img1: &lt;code&gt;{'Brown': 14, 'White': 13, 'Black': 40, 'Gray': 31}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Img2: &lt;code&gt;{'Pink': 82, 'Brown': 8, 'White': 7}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the numbers denote the % of that color present in the image. What would be the best way to compute similarity on a scale of 0-100 between the two images?&lt;/p&gt;&#xA;" OwnerUserId="2744" LastActivityDate="2020-08-17T20:32:41.403" Title="Computing Image Similarity based on Color Distribution" Tags="|data-mining|clustering|similarity|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="853" PostTypeId="1" AcceptedAnswerId="858" CreationDate="2014-07-28T07:19:49.877" Score="13" ViewCount="2844" Body="&lt;p&gt;I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to make the same NER for different language. The problem here is that I can't use POS tags and lemmas. I started reading articles about deep learning and unsupervised feature learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use methods for unsupervised feature learning with CRF algorithm? Did anyone try this and got any good result? Is there any article or tutorial about this matter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still don't completely understand this way of feature creation so I don't want to spend to much time for something that won't work. So any information would be really helpful. To create whole NER system based on deep learning is a bit to much for now.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="31513" LastEditDate="2017-06-30T14:45:37.653" LastActivityDate="2018-04-19T02:02:47.127" Title="Unsupervised feature learning for NER" Tags="|nlp|text-mining|feature-extraction|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="854" PostTypeId="2" ParentId="554" CreationDate="2014-07-28T08:36:53.633" Score="5" Body="&lt;p&gt;I like @Kallestad answer very much, but I would like to add a meta-step: Make sure that you understand how the data where collected, and what types of constraints there are. &#xA;I think it is very common to think that there where no non-obvious steps when the data where collected, but this is not the case: Most of the time, some process or indivudal did somethink with the data, and these steps can and will influence the shape of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two examples:&#xA;I had a study recently where the data where collected by various con&#xA;tractors worldwide. I was not at the briefing, so that was opaque to me. Unfortunately, the measurements where off for some parts of france: People all liked ice cram, but we expected a random distribution. There was no obvious reason for this uniformity, so I began to hunt the errors. When I queried the contractors, one had misunderstood the briefing and selected only ice-cream lovers from his database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second error was more challenging: When doing some geographic analysis, I found that a lot of people had extremely large movement patterns, which suggested that a lot of them traveled from Munich to Hamburg in minutes. When I spoke with ppeople upstream, they found a subtle bug in their data aggregation software, which was unnoticed before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do not assume that your data was collected by perfect processes /humans.&lt;/li&gt;&#xA;&lt;li&gt;Do try to understand the limits of your data providers.&lt;/li&gt;&#xA;&lt;li&gt;Look at individual patterns / values and try to determine if they are logical (easy for movement / geographic data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="791" LastActivityDate="2014-07-28T08:36:53.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="855" PostTypeId="2" ParentId="554" CreationDate="2014-07-28T12:07:25.573" Score="4" Body="&lt;p&gt;Below you can find a copy of my answer to a related (however, focused on data cleaning aspect) question here on &lt;em&gt;Data Science StackExchange&lt;/em&gt; (&lt;a href=&quot;https://datascience.stackexchange.com/a/722/2452&quot;&gt;https://datascience.stackexchange.com/a/722/2452&lt;/a&gt;), provided in its entirety for readers' convenience. I believe that it partially answers your question as well and hope it is helpful. While the answer is focused on &lt;code&gt;R&lt;/code&gt; ecosystem, similar packages and/or libraries can be found for other &lt;em&gt;data analysis environments&lt;/em&gt;. Moreover, while the two &lt;strong&gt;cited papers&lt;/strong&gt; on data preparation also contain examples in R, these papers present &lt;strong&gt;general&lt;/strong&gt; &lt;strong&gt;workflow (framework)&lt;/strong&gt; and &lt;strong&gt;best practices&lt;/strong&gt; that are applicable to &lt;strong&gt;any&lt;/strong&gt; data analysis environment.&lt;/p&gt;&#xA;&lt;p&gt;R contains some &lt;em&gt;standard&lt;/em&gt; functions for data manipulation, which can be used for data cleaning, in its &lt;strong&gt;base&lt;/strong&gt; package (&lt;code&gt;gsub&lt;/code&gt;, &lt;code&gt;transform&lt;/code&gt;, etc.), as well as in various third-party packages, such as &lt;strong&gt;stringr&lt;/strong&gt;, &lt;strong&gt;reshape&lt;/strong&gt;, &lt;strong&gt;reshape2&lt;/strong&gt;, and &lt;strong&gt;plyr&lt;/strong&gt;. Examples and best practices of usage for these packages and their functions are described in the following paper: &lt;a href=&quot;http://vita.had.co.nz/papers/tidy-data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://vita.had.co.nz/papers/tidy-data.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Additionally, R offers some packages specifically &lt;em&gt;focused&lt;/em&gt; on data cleaning and transformation:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/editrules/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;editrules&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/deducorrect/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;deducorrect&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/StatMatch/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;StatMatch&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/MatchIt/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MatchIt&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/DataCombine&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataCombine&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;A comprehensive and coherent approach to &lt;strong&gt;data cleaning&lt;/strong&gt; in R, including examples and use of &lt;strong&gt;editrules&lt;/strong&gt; and &lt;strong&gt;deducorrect&lt;/strong&gt; packages, as well as a description of &lt;em&gt;workflow&lt;/em&gt; (&lt;em&gt;framework&lt;/em&gt;) of data cleaning in R, is presented in the following paper, which I highly recommend: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="29169" LastEditDate="2021-03-13T23:23:44.427" LastActivityDate="2021-03-13T23:23:44.427" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="858" PostTypeId="2" ParentId="853" CreationDate="2014-07-28T14:48:37.743" Score="6" Body="&lt;p&gt;Yes, it is entirely possible to combine unsupervised learning with the CRF model.  In particular, I would recommend that you explore the possibility of using &lt;a href=&quot;https://code.google.com/p/word2vec/&quot;&gt;word2vec&lt;/a&gt; features as inputs to your CRF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2vec trains a  to distinguish between words that are appropriate for a given context and words that are randomly selected.  Select weights of the model can then be interpreted as a dense vector representation of a given word.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These dense vectors have the appealing property that words that are semantically or syntactically similar have similar vector representations.  Basic vector arithmetic even reveals some interesting learned relationships between words.&lt;br&gt;&#xA;For example, vector(&quot;Paris&quot;) - vector(&quot;France&quot;) + vector(&quot;Italy&quot;) yields a vector that is quite similar to vector(&quot;Rome&quot;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a high level, you can think of word2vec representations as being similar to LDA or LSA representations, in the sense that you can convert a sparse input vector into a dense output vector that contains word similarity information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For that matter, LDA and LSA are also valid options for unsupervised feature learning -- both attempt to represent words as combinations of &quot;topics&quot; and output dense word representations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For English text Google distributes word2vec models pretrained on a huge 100 billion word Google News dataset, but for other languages you'll have to train your own model.&lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="684" LastEditDate="2014-07-29T03:29:42.940" LastActivityDate="2014-07-29T03:29:42.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="859" PostTypeId="1" CreationDate="2014-07-28T16:17:49.823" Score="8" ViewCount="1087" Body="&lt;p&gt;What are the different classes of data science problems that can be solved using mapreduce programming model?&lt;/p&gt;&#xA;" OwnerUserId="2643" LastEditorDisplayName="user1361" LastEditDate="2014-07-30T03:13:17.827" LastActivityDate="2016-08-05T09:30:28.207" Title="Data science and MapReduce programming model of Hadoop" Tags="|apache-hadoop|map-reduce|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="860" PostTypeId="2" ParentId="554" CreationDate="2014-07-28T16:23:39.600" Score="1" Body="&lt;p&gt;I'll add one thing- if possible, do a reasonableness check by comparing you data against some other source. It seems that whenever I fail to do this, I get burnt:(&lt;/p&gt;&#xA;" OwnerUserId="1241" LastActivityDate="2014-07-28T16:23:39.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="861" PostTypeId="2" ParentId="859" CreationDate="2014-07-28T16:39:45.703" Score="9" Body="&lt;p&gt;Data Science has many different sub-areas as described in &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/ingredients-of-data-science-1&quot;&gt;my post&lt;/a&gt;). Nearly for each area, scientists and developer has significant contributions. To learn more about what can be done, please look at following websites:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data Mining Algorithms &amp;amp; Machine Learning -&gt; &lt;a href=&quot;https://mahout.apache.org/&quot;&gt;Apache Mahout&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Statistics -&gt; &lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop/wiki&quot;&gt;RHadoop&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Data Warehousing &amp;amp; Database Querying -&gt; &lt;a href=&quot;http://www.teradata.com.tr/Teradata-Aster-SQL-MapReduce/?LangType=1055&amp;amp;LangSelect=true&quot;&gt;SQL-MapReduce&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Social Network Analysis -&gt; &lt;a href=&quot;http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&amp;amp;arnumber=5636636&quot;&gt;Article&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Bio-informatics -&gt; &lt;a href=&quot;http://abhishek-tiwari.com/post/mapreduce-and-hadoop-algorithms-in-bioinformatics-papers?ModPagespeed=noscript&quot;&gt;Article - 1 &lt;/a&gt;, &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/11/S12/S1&quot;&gt;Article - 2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, there are some work on MapReduce + Excel + Cloud combination but I have not found the link.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are the different classes of Data Science problems ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Each &quot;classes&quot; is not purely homogeneous problem domain, i.e. some problem cannot be solved via map and reduce approach due to its communication cost, or algorithm behavior. What I mean by behavior is that some problem wants to have control on all data sets instead of chunks. Thus, I refuse to list type of problem &quot;classes&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not forget that knowing what MapReduce can do is not enough for Data Science. You should also aware of &lt;a href=&quot;http://www.analyticbridge.com/profiles/blogs/what-mapreduce-can-t-do&quot;&gt;What MapReduce can't do&lt;/a&gt;, too.&lt;/p&gt;&#xA;" OwnerDisplayName="user1361" LastEditorDisplayName="user1361" LastEditDate="2014-07-28T16:49:55.573" LastActivityDate="2014-07-28T16:49:55.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="862" PostTypeId="2" ParentId="52" CreationDate="2014-07-28T17:33:43.280" Score="5" Body="&lt;p&gt;I think that there is no universal technique for &quot;cleaning&quot; data before doing actual research. On the other hand, I'm aiming for doing as much &lt;a href=&quot;https://en.wikipedia.org/wiki/Reproducibility#Reproducible_research&quot; rel=&quot;noreferrer&quot;&gt;reproducible research&lt;/a&gt; as possible. By doing reproducible research, if you used cleaning techniques with bugs or with poor parameters/assumptions it could be spot by others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nice R package &lt;a href=&quot;http://yihui.name/knitr/&quot; rel=&quot;noreferrer&quot;&gt;knitr&lt;/a&gt; which helps a lot in reproducible research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, not all research could be fully reproduced (for example live Twitter data) , but at least you can document cleaning, formating and preprocessing steps easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check my &lt;a href=&quot;http://rpubs.com/QuatnumDamage/storm&quot; rel=&quot;noreferrer&quot;&gt;assessment&lt;/a&gt; prepared for &lt;a href=&quot;https://www.coursera.org/course/repdata&quot; rel=&quot;noreferrer&quot;&gt;Reproducible Research course at Coursera&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2014-07-28T17:33:43.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="863" PostTypeId="2" ParentId="859" CreationDate="2014-07-29T13:43:39.060" Score="13" Body="&lt;p&gt;Let's first split it into parts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Science&lt;/strong&gt; is about making knowledge from raw data. It uses machine learning, statistics and other fields to simplify (or even automate) decision making. Data science techniques may work with any data size, but more data means better predictions and thus more precise decisions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hadoop&lt;/strong&gt; is a common name for a set of tools intended to work with large amounts of data. Two most important components in Hadoop are HDFS and MapReduce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;HDFS&lt;/strong&gt;, or Hadoop Distributed File System, is a special distributed storage capable of holding really large data amounts. Large files on HDFS are split into blocks, and for each block HDFS API exposes its &lt;em&gt;location&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MapReduce&lt;/strong&gt; is framework for running computations on nodes with data. MapReduce heavily uses &lt;em&gt;data locality&lt;/em&gt; exposed by HDFS: when possible, data is not transferred between nodes, but instead code is copied to the nodes with data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically any problem (including data science tasks) that doesn't break data locality principle may be efficiently implemented using MapReduce (and a number of other problems may be solved not that efficiently, but still simply enough).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Let's take some examples. Very often analyst only needs some simple statistics over his tabular data. In this case &lt;a href=&quot;https://hive.apache.org/&quot;&gt;&lt;strong&gt;Hive&lt;/strong&gt;&lt;/a&gt;, which is basically SQL engine over MapReduce, works pretty well (there are also Impala, Shark and others, but they don't use Hadoop's MapReduce, so more on them later). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other cases analyst (or developer) may want to work with previously unstructured data. Pure MapReduce is pretty good for &lt;strong&gt;transforming&lt;/strong&gt; and &lt;strong&gt;standardizing&lt;/strong&gt; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people are used to exploratory statistics and visualization using tools like R. It's possible to apply this approach to big data amounts using &lt;a href=&quot;https://github.com/RevolutionAnalytics/RHadoop/wiki&quot;&gt;&lt;strong&gt;RHadoop&lt;/strong&gt;&lt;/a&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And when it comes to MapReduce-based machine learning &lt;a href=&quot;https://mahout.apache.org/&quot;&gt;&lt;strong&gt;Apache Mahout&lt;/strong&gt;&lt;/a&gt; is the first to mention. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There's, however, one type of algorithms that work pretty slowly on Hadoop even in presence of data locality, namely, iterative algorithms. Iterative algorithms tend to have multiple Map and Reduce stages. Hadoop's MR framework &lt;strong&gt;reads&lt;/strong&gt; and &lt;strong&gt;writes&lt;/strong&gt; data to disk on each stage (and sometimes in between), which makes iterative (as well as any multi-stage) tasks terribly slow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fortunately, there are alternative frameworks that can both - use data locality and keep data in memory between stages. Probably, the most notable of them is &lt;a href=&quot;https://spark.apache.org/&quot;&gt;&lt;strong&gt;Apache Spark&lt;/strong&gt;&lt;/a&gt;. Spark is complete replacement for Hadoop's MapReduce that uses its own runtime and exposes pretty rich API for manipulating your distributed dataset. Spark has several sub-projects, closely related to data science: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shark.cs.berkeley.edu/&quot;&gt;&lt;strong&gt;Shark&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;https://spark.apache.org/sql/&quot;&gt;&lt;strong&gt;Spark SQL&lt;/strong&gt;&lt;/a&gt;  provide alternative SQL-like interfaces to data stored on HDFS&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/streaming/&quot;&gt;&lt;strong&gt;Spark Streaming&lt;/strong&gt;&lt;/a&gt; makes it easy to work with continuous data streams (e.g. Twitter feed)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/mllib/&quot;&gt;&lt;strong&gt;MLlib&lt;/strong&gt;&lt;/a&gt; implements a number of machine learning algorithms with a pretty simple and flexible API&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/graphx/&quot;&gt;&lt;strong&gt;GraphX&lt;/strong&gt;&lt;/a&gt; enables large-scale graph processing&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So there's pretty large set of data science problems that you can solve with Hadoop and related projects. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-29T13:43:39.060" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="864" PostTypeId="2" ParentId="859" CreationDate="2014-07-30T07:49:14.467" Score="2" Body="&lt;p&gt;map/reduce is most appropriate for parallelizable offline computations. To be more precise, it works best when the result can be found from the result of some function of a partition of the input. Averaging is a trivial example; you can do this with map/reduce by summing each partition, returning the sum and the number of elements in the partition, then computing the overall mean using these intermediate results. It is less appropriate when the intermediate steps depend on the state of the other partitions.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-30T07:49:14.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="865" PostTypeId="1" AcceptedAnswerId="875" CreationDate="2014-07-30T10:24:54.180" Score="1" ViewCount="1798" Body="&lt;p&gt;I need to build parse tree for some source code (on Python or any program language that describe by CFG).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have source code on some programming language and BNF this language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody give some advice how can I build parse tree in this case?&#xA;Preferably, with tools for Python.&lt;/p&gt;&#xA;" OwnerUserId="988" LastActivityDate="2014-07-30T23:40:54.700" Title="How to build parse tree with BNF" Tags="|python|parsing|" AnswerCount="1" CommentCount="2" ClosedDate="2014-08-08T12:23:02.223" ContentLicense="CC BY-SA 3.0" />
  <row Id="866" PostTypeId="1" AcceptedAnswerId="881" CreationDate="2014-07-30T11:45:08.313" Score="12" ViewCount="1045" Body="&lt;p&gt;I am currently working with a large set of health insurance claims data that includes some laboratory and pharmacy claims. The most consistent information in the data set, however, is made up of diagnosis (ICD-9CM) and procedure codes (CPT, HCSPCS, ICD-9CM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goals are to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease;&lt;/li&gt;&#xA;&lt;li&gt;Identify the likelihood (or probability) that a patient will develop a medical condition based on the conditions they have had in the past;&lt;/li&gt;&#xA;&lt;li&gt;Do the same as 1 and 2, but with procedures and/or diagnoses.&lt;/li&gt;&#xA;&lt;li&gt;Preferably, the results would be interpretable by a doctor&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have looked at things like the &lt;a href=&quot;https://www.heritagehealthprize.com/c/hhp/details/milestone-winners&quot; rel=&quot;nofollow noreferrer&quot;&gt;Heritage Health Prize Milestone papers&lt;/a&gt; and have learned a lot from them, but they are focused on predicting hospitalizations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here are my questions: What methods do you think work well for problems like this? And, what resources would be most useful for learning about data science applications and methods relevant to healthcare and clinical medicine?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT #2 to add plaintext table:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CKD is the target condition, &quot;chronic kidney disease&quot;, &quot;.any&quot; denotes that they have acquired that condition at any time, &quot;.isbefore.ckd&quot; means they had that condition before their first diagnosis of CKD.  The other abbreviations correspond with other conditions identified by ICD-9CM code groupings.  This grouping occurs in SQL during the import process. Each variable, with the exception of patient_age, is binary.&lt;/p&gt;&#xA;" OwnerUserId="2781" LastEditorUserId="2781" LastEditDate="2017-10-02T13:27:02.367" LastActivityDate="2024-02-04T17:35:10.013" Title="Predicting next medical condition from past conditions in claims data" Tags="|machine-learning|r|" AnswerCount="3" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="867" PostTypeId="1" CreationDate="2014-07-30T13:55:36.187" Score="3" ViewCount="886" Body="&lt;p&gt;Does anyone know what (from your experience) is the best open source natural language generators (NLG) out there? What are the relative merits of each?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking to do sophisticated text summarization and would like to use theme extraction/semantic modeling in conjunction with NLG tools to create accurate, context-aware, and natural-sounding text summaries.&lt;/p&gt;&#xA;" OwnerUserId="2785" LastEditorUserId="84" LastEditDate="2014-08-12T16:52:12.833" LastActivityDate="2015-05-04T18:25:08.213" Title="Relative merits of different open source natural language generators" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="868" PostTypeId="2" ParentId="859" CreationDate="2014-07-30T15:32:44.557" Score="4" Body="&lt;p&gt;There is a paper you should look into:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs.smith.edu/dftwiki/images/6/68/MapReduceDistributedComputingMachineLearning.pdf&quot; rel=&quot;nofollow&quot;&gt;MapReduce: Distributed Computing for Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They distinguish 3 classes of machine-learning problems that are reasonable to address with MapReduce:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Single pass algorithms&lt;/li&gt;&#xA;&lt;li&gt;Iterative algorithms&lt;/li&gt;&#xA;&lt;li&gt;Query based algorithms&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;They also give examples for each class.&lt;/p&gt;&#xA;" OwnerUserId="2787" LastActivityDate="2014-07-30T15:32:44.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="869" PostTypeId="1" AcceptedAnswerId="871" CreationDate="2014-07-30T16:27:45.177" Score="33" ViewCount="25648" Body="&lt;p&gt;So, I'm just starting to learn how a neural network can operate to recognize patterns and categorize inputs, and I've seen how an artificial neural network can parse image data and categorize the images (&lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html&quot;&gt;demo with convnetjs&lt;/a&gt;), and the key there is to downsample the image and each pixel stimulates one input neuron into the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm trying to wrap my head around if this is possible to be done with string inputs? The use-case I've got is a &quot;recommendation engine&quot; for movies a user has watched. Movies have lots of string data (title, plot, tags), and I could imagine &quot;downsampling&quot; the text down to a few key words that describe that movie, but even if I parse out the top five words that describe this movie, I think I'd need input neurons for every english word in order to compare a set of movies? I could limit the input neurons just to the words used in the set, but then could it grow/learn by adding new movies (user watches a new movie, with new words)? Most of the libraries I've seen don't allow adding new neurons after the system has been trained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way to map string/word/character data to inputs into a neural network? Or is a neural network really not the right tool for the job of parsing string data like this (what's a better tool for pattern-matching in string data)?&lt;/p&gt;&#xA;" OwnerUserId="2790" LastActivityDate="2017-08-26T15:57:49.753" Title="Neural Network parse string data?" Tags="|neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="870" PostTypeId="2" ParentId="869" CreationDate="2014-07-30T17:42:09.797" Score="3" Body="&lt;p&gt;This is not a problem about neural networks per se, but about representing textual data in machine learning. You can represent the movies, cast, and theme as categorical variables. The plot is more complicated; you'd probably want a &lt;a href=&quot;http://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic model&lt;/a&gt; for that, but I'd leave that out until you get the hang of things. It does precisely that textual &quot;downsampling&quot; you mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;http://visualstudiomagazine.com/articles/2013/07/01/neural-network-data-normalization-and-encoding.aspx&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; tutorial to learn how to encode categorical variables for neural networks. And good luck!&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-07-30T17:42:09.797" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="871" PostTypeId="2" ParentId="869" CreationDate="2014-07-30T17:53:40.330" Score="18" Body="&lt;p&gt;Using a neural network for prediction on natural language data can be a tricky task, but there are tried and true methods for making it possible.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the Natural Language Processing (NLP) field, text is often represented using the bag of words model.  In other words, you have a vector of length &lt;em&gt;n&lt;/em&gt;, where &lt;em&gt;n&lt;/em&gt; is the number of words in your vocabulary, and each word corresponds to an element in the vector.  In order to convert text to numeric data, you simply count the number of occurrences of each word and place that value at the index of the vector that corresponds to the word. &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia does an excellent job of describing this conversion process.&lt;/a&gt;  Because the length of the vector is fixed, its difficult to deal with new words that don't map to an index, but there are ways to help mitigate this problem (lookup &lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_hashing&quot; rel=&quot;noreferrer&quot;&gt;feature hashing&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method of representation has many disadvantages -- it does not preserve the relationship between adjacent words, and results in very sparse vectors.  Looking at &lt;a href=&quot;http://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;noreferrer&quot;&gt;n-grams&lt;/a&gt; helps to fix the problem of preserving word relationships, but for now let's focus on the second problem, sparsity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's difficult to deal directly with these sparse vectors (many linear algebra libraries do a poor job of handling sparse inputs), so often the next step is dimensionality reduction. For that we can refer to the field of &lt;a href=&quot;http://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;noreferrer&quot;&gt;topic modeling&lt;/a&gt;:  Techniques like &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;noreferrer&quot;&gt;Latent Dirichlet Allocation&lt;/a&gt; (LDA) and &lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot; rel=&quot;noreferrer&quot;&gt;Latent Semantic Analysis&lt;/a&gt; (LSA) allow the compression of these sparse vectors into dense vectors by representing a document as a combination of topics.  You can fix the number of topics used, and in doing so fix the size of the output vector producted by LDA or LSA. This dimensionality reduction process drastically reduces the size of the input vector while attempting to lose a minimal amount of information.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, after all of these conversions, you can feed the outputs of the topic modeling process into the inputs of your neural network.     &lt;/p&gt;&#xA;" OwnerUserId="684" LastEditorUserId="29939" LastEditDate="2017-03-14T01:25:26.937" LastActivityDate="2017-03-14T01:25:26.937" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="872" PostTypeId="1" AcceptedAnswerId="873" CreationDate="2014-07-30T18:45:13.790" Score="8" ViewCount="8311" Body="&lt;p&gt;I am attempting to use the tm package to convert a vector of text strings to a corpus element.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code looks something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Corpus(d1$Yes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;d1$Yes&lt;/code&gt; is a factor with 124 levels, each containing a text string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;code&gt;d1$Yes[246] = &quot;So we can get the boat out!&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm receiving the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Error: inherits(x, &quot;Source&quot;) is not TRUE&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to remedy this.&lt;/p&gt;&#xA;" OwnerUserId="2792" LastEditorUserId="375" LastEditDate="2020-03-11T21:33:18.877" LastActivityDate="2020-03-11T21:33:18.877" Title="R error using package tm (text-mining)" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="873" PostTypeId="2" ParentId="872" CreationDate="2014-07-30T19:15:09.383" Score="9" Body="&lt;p&gt;You have to tell Corpus what kind of source you are using.  Try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Corpus(VectorSource(d1$Yes))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="375" LastActivityDate="2014-07-30T19:15:09.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="874" PostTypeId="2" ParentId="423" CreationDate="2014-07-30T21:53:13.790" Score="4" Body="&lt;p&gt;I have some thoughts about your question. I hope it may help you to solve your problem.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm planning to run some experiments with very large data sets, and I'd like to distribute the computation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/meeting-justice-of-data-science&quot; rel=&quot;nofollow&quot;&gt;one of my posts&lt;/a&gt;, I have done research on topic of &lt;em&gt;evaluation methods of Data Science&lt;/em&gt;. With &lt;strong&gt;Learning Curve&lt;/strong&gt;, you can evaluate your experiments &lt;em&gt;learning ability&lt;/em&gt;. To talk a bit more, you will &lt;strong&gt;fix&lt;/strong&gt; commodity configuration, and then will run same experiment on the same number of machines with different size of data set you have (from starting from small chunk in size, incrementally increase the size until you reach the whole data set). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To point on, you should avoid having &lt;em&gt;power distribution&lt;/em&gt; for result of performance test being run with different size of data sets. To avoid, you should carefully choose &lt;em&gt;step size&lt;/em&gt; (step size = amount of increments).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have about ten machines available, each with 200 GB of free space on hard disk. However, I would like to perform experiments on a greater number of nodes, to measure scalability more precisely.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For this type question, I have intuitively searched and read materials; afterwards, published as a &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/scale-your-vision&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt;. At the end of the post, I have briefly talked about how to test your hypothesis on real complex system. If you let me, I want to briefly talk about;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, &lt;strong&gt;base requirement&lt;/strong&gt; should be formed in order to run data set as a whole. The minimum requirement will build your &lt;strong&gt;baseline evaluation score&lt;/strong&gt; which is calculated with one/combination of evaluation metrics you have chosen, or with one/combination of methods being used to calculate &lt;code&gt;Running Time = Computation complexity + Communication cost + Synchronization cost&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After those steps, with an &lt;em&gt;evaluation strategy&lt;/em&gt;, add new elements, e.g. new node, to the system you have doing scalability test; meanwhile, for each addition, measure performance w.r.t new system configuration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to note, &lt;em&gt;evaluation strategy&lt;/em&gt; must be planned along with considerations of default behavior of parallel and distributed systems. For example, what I mean by behavior is that adding just more cores will, after some point, automatically drop performance of the system not due to your algorithm characteristics. It is because more cores need more RAMs, more hard driver, or etc. In other words, there is a &lt;strong&gt;N-Way&lt;/strong&gt; relationship between hardware components. As a second example, adding more nodes to the distributed system will punished you with more communication and synchronization costs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a last step, you will sketch two different graphs with your evaluation results via data analysis program or language (As a recommendation, use GNU Plot or R programming language). Print out and put those results at your desktop, and start to examine them, carefully. According to your investigation, modify/erase + rebuild &lt;em&gt;evaluation strategy&lt;/em&gt; and re-do the performance test.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there commodity services which would grant me that only my application would be running at a given time? Has anyone used such services yet?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have no much experiment on commodity services, but I can easily say whether it grants or not depends on your configuration of services. If you configured say Hadoop to your node as an only service, Hadoop will grant your code will be only running at any time.&lt;/p&gt;&#xA;" OwnerDisplayName="user1361" LastActivityDate="2014-07-30T21:53:13.790" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="875" PostTypeId="2" ParentId="865" CreationDate="2014-07-30T23:40:54.700" Score="2" Body="&lt;p&gt;I suggest you use &lt;a href=&quot;http://www.antlr.org/&quot; rel=&quot;nofollow&quot;&gt;ANTLR&lt;/a&gt;, which is a very powerful parser generator. It has a good GUI for entering your BNF. It has a &lt;a href=&quot;https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Python+Target&quot; rel=&quot;nofollow&quot;&gt;Python target&lt;/a&gt; capability.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-07-30T23:40:54.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="876" PostTypeId="2" ParentId="866" CreationDate="2014-07-31T16:39:37.127" Score="3" Body="&lt;p&gt;&quot;Identify the most influential precursor conditions (comorbidities) for a medical condition like chronic kidney disease&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure that it's possible to ID &lt;em&gt;the&lt;/em&gt; most influential conditions; I think it will depend on what model you're using. Just yesterday I fit a random forest and a boosted regression tree to the same data, and the order and relative importance each model gave for the variables were quite different.&lt;/p&gt;&#xA;" OwnerUserId="1241" LastActivityDate="2014-07-31T16:39:37.127" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="877" PostTypeId="1" AcceptedAnswerId="878" CreationDate="2014-07-31T18:52:56.307" Score="8" ViewCount="246" Body="&lt;p&gt;Well this looks like the most suited place for this question.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every website collect data of the user, some just for usability and personalization, but the majority like social networks track every move on the web, some free apps on your phone scan text messages, call history and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this data siphoning is just for selling your profile for advertisers?&lt;/p&gt;&#xA;" OwnerUserId="2798" LastActivityDate="2020-08-16T21:51:11.787" Title="What is the use of user data collection besides serving ads?" Tags="|data-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="878" PostTypeId="2" ParentId="877" CreationDate="2014-07-31T23:03:33.767" Score="7" Body="&lt;p&gt;A couple of days ago developers from one product company asked me how they can understand why new users were leaving their website. My first question to them was what these users' profiles looked like and how they were different from those who stayed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advertising is only top of an iceberg. User profiles (either filled by users themselves or computed from users' behaviour) hold information about: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;user categories&lt;/strong&gt;, i.e. what kind of people tend to use your website/product&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;paying client portraits&lt;/strong&gt;, i.e. who is more likely to use your paid services&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;UX component performance&lt;/strong&gt;, e.g. how long it takes people to find the button they need&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;action performance comparison&lt;/strong&gt;, e.g. what was more efficient - lower price for a weekend or propose gifts with each buy, etc. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So it's more about improving product and making better user experience rather than selling this data to advertisers. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-07-31T23:03:33.767" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="879" PostTypeId="2" ParentId="877" CreationDate="2014-08-01T06:03:47.700" Score="6" Body="&lt;p&gt;Most companies won't sell the data, not on any small scale anyways.  Most will use it internally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;User tracking data is important for understanding a lot of things.  There's basic A/B testing where you provide different experiences to see which is more effective.  There is understanding how your UI is utilized.  Categorizing your end users in different ways for a variety of reasons.  Figuring out where your end user base is, and within that group where the end users that matter are.  Correlating user experiences with social network updates.  Figuring out what will draw people to your product and what drives them away.  The list of potential for data mining and analysis projects could go on for days.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data storage is cheap.  If you track everything out of the gate, you can figure out what you want to do with that data later.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scanning text messages is sketchy territory when there isn't a good reason for it.  Even when there is a good reason it's sketchy territory.  I'd love to say that nobody does it, but there have been instances where big companies have done it and there are a lot of cases where no-name apps at least require access to that kind of data for installation.  I generally frown on that kind of thing myself as a consumer, but the data analyst in me would love to see if I could build anything useful from a set of information like that.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2014-08-01T06:03:47.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="880" PostTypeId="2" ParentId="810" CreationDate="2014-08-01T12:27:41.580" Score="6" Body="&lt;p&gt;There always is the solution to try both approaches and keep the one that maximizes the expected performances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I would assume you prefer minimizing false negatives at the cost of some false positive, so you want to bias your classifier against the strong negative prior, and address the imbalance by reducing the number of negative examples in your training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then compute the precision/recall, or sensitivity/specificity, or whatever criterion suits you on the full, imbalanced, dataset to make sure you haven't ignored a significant pattern present in the real data while building the model on the reduced data.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-08-01T12:27:41.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="881" PostTypeId="2" ParentId="866" CreationDate="2014-08-01T14:08:13.043" Score="7" Body="&lt;p&gt;I've never worked with medical data, but from general reasoning I'd say that relations between variables in healthcare are pretty complicated. Different models, such as random forests, regression, etc. could capture only part of relations and ignore others. In such circumstances it makes sense to use general &lt;strong&gt;statistical exploration&lt;/strong&gt; and &lt;strong&gt;modelling&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the very first thing I would do is finding out &lt;strong&gt;correlations&lt;/strong&gt; between possible precursor conditions and diagnoses. E.g. in what percent of cases chronic kidney disease was preceded by long flu? If it is high, it &lt;a href=&quot;http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation&quot;&gt;doesn't always mean causality&lt;/a&gt;, but gives pretty good food for thought and helps to better understand relations between different conditions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important step is data visualisation. Does CKD happens in males more often than in females? What about their place of residence? What is distribution of CKD cases by age? It's hard to grasp large dataset as a set of numbers, plotting them out makes it much easier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have an idea of what's going on, perform &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot;&gt;&lt;strong&gt;hypothesis testing&lt;/strong&gt;&lt;/a&gt; to check your assumption. If you reject null hypothesis (basic assumption) in favour of alternative one, congratulations, you've made &quot;something real&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, when you have a good understanding of your data, try to create complete &lt;strong&gt;model&lt;/strong&gt;. It may be something general like &lt;a href=&quot;https://www.coursera.org/course/pgm&quot;&gt;PGM&lt;/a&gt; (e.g. manually-crafted Bayesian network), or something more specific like linear regression or &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machine&quot;&gt;SVM&lt;/a&gt;, or anything. But in any way you will already know how this model corresponds to your data and how you can measure its efficiency. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a good starting resource for learning statistical approach I would recommend &lt;a href=&quot;https://www.udacity.com/course/st101&quot;&gt;Intro to Statistics&lt;/a&gt; course by Sebastian Thrun. While it's pretty basic and doesn't include advanced topics, it describes most important concepts and gives systematic understanding of probability theory and statistics. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-08-01T14:08:13.043" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="882" PostTypeId="2" ParentId="877" CreationDate="2014-08-01T14:10:38.267" Score="3" Body="&lt;p&gt;Here's a practical example of using web data for something other than advertising. Distil Networks (disclaimer, I work there) uses network traffic to determine whether page accesses are from humans or bots - scrapers, click fraud, form spam, etc.&lt;/p&gt;&#xA;&lt;p&gt;Another example is some of the work that Webtrends is doing. They allow site users to build a model for each visitor to predict whether they'll leave, buy, add to cart, etc. Then based on the probability of each action you can change the users experience (e.g. if they're about to leave, give them a coupon). &lt;a href=&quot;http://www.oscon.com/oscon2014/public/schedule/detail/34809&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s the slides from a talk by them.&lt;/p&gt;&#xA;" OwnerUserId="403" LastEditorUserId="98307" LastEditDate="2020-08-16T21:51:11.787" LastActivityDate="2020-08-16T21:51:11.787" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="883" PostTypeId="1" AcceptedAnswerId="886" CreationDate="2014-08-01T18:13:06.063" Score="3" ViewCount="6069" Body="&lt;p&gt;I am working on a data set that has multiple traffic speed measurements per day. My data is from the city of chicago, and it is taken every minute for about six months. I wanted to consolidate this data into days only, so this is what I did:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;traffic &amp;lt;- read.csv(&quot;path.csv&quot;,header=TRUE)&#xA;traffic2 &amp;lt;- aggregate(SPEED~DATE, data=traffic, FUN=MEAN)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this was perfect because it took all of my data and averaged it by date. For example, my original data looked something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATE        SPEED  &#xA;12/31/2012   22&#xA;12/31/2012   25&#xA;12/31/2012   23&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the final looked like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATE        SPEED &#xA;10/1/2012    22&#xA;10/2/2012    23&#xA;10/3/2012    22&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The only problem, is my data is supposed to start at 9/1/2012. I plotted this data, and it turns out the data goes from 10/1/2012-12/31/2012 and then 9/1/2012-9/30/2012.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What in the world is going on here?&lt;/p&gt;&#xA;" OwnerUserId="2614" LastActivityDate="2014-10-18T03:26:04.953" Title="R aggregate() with dates" Tags="|r|dataset|beginner|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="884" PostTypeId="2" ParentId="787" CreationDate="2014-08-01T18:16:57.543" Score="1" Body="&lt;p&gt;So I was never able to find the error by looking through my logs. I ended up reinstalling it with CDH5 (which was MUCH easier than installing &quot;poor&quot; Hadoop)&#xA;Now everything runs fine! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm still having trouble getting things to save to the hdfs, but thats a question for another day... &lt;/p&gt;&#xA;" OwnerUserId="2614" LastActivityDate="2014-08-01T18:16:57.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="886" PostTypeId="2" ParentId="883" CreationDate="2014-08-01T23:21:34.360" Score="4" Body="&lt;p&gt;I am going to agree with @user1683454's comment. After importing, your DATE column is of either &lt;code&gt;character&lt;/code&gt;, or &lt;code&gt;factor&lt;/code&gt; class (depending on your settings for &lt;code&gt;stringsAsFactors&lt;/code&gt;). Therefore, I think that you can solve this issue in at least several ways, as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Convert data&lt;/strong&gt; to correct type &lt;strong&gt;during import&lt;/strong&gt;. To do this, just use the following options of &lt;code&gt;read.csv()&lt;/code&gt;: &lt;code&gt;stringsAsFactors&lt;/code&gt; (or &lt;code&gt;as.is&lt;/code&gt;) and &lt;code&gt;colClasses&lt;/code&gt;. By default, you can specify conversion to &lt;code&gt;Date&lt;/code&gt; or &lt;code&gt;POSIXct&lt;/code&gt; classes. If you need a non-standard format, you have two options. First, if you have a single Date column, you can use &lt;code&gt;as.Date.character()&lt;/code&gt; to pass the desired format to &lt;code&gt;colClasses&lt;/code&gt;. Second, if you have multiple Date columns, you can write a function for that and pass it to &lt;code&gt;colClasses&lt;/code&gt; via &lt;code&gt;setAs()&lt;/code&gt;. Both options are discussed here: &lt;a href=&quot;https://stackoverflow.com/questions/13022299/specify-date-format-for-colclasses-argument-in-read-table-read-csv&quot;&gt;https://stackoverflow.com/questions/13022299/specify-date-format-for-colclasses-argument-in-read-table-read-csv&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Convert data&lt;/strong&gt; to correct format &lt;strong&gt;after import&lt;/strong&gt;. Thus, after calling &lt;code&gt;read.csv()&lt;/code&gt;, you would have to execute the following code: &lt;code&gt;dateColumn &amp;lt;- as.Date(dateColumn, &quot;%m/%d/%Y&quot;)&lt;/code&gt; or &lt;code&gt;dateColumn &amp;lt;- strptime(dateColumn, &quot;%m/%d/%Y&quot;)&lt;/code&gt; (adjust the format to whatever Date format you need).&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-08-01T23:21:34.360" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="887" PostTypeId="1" AcceptedAnswerId="889" CreationDate="2014-08-02T00:07:09.267" Score="1" ViewCount="107" Body="&lt;p&gt;I am currently on a project that will build a model (train and test) on Client-side Web data, but evaluate this model on Sever-side Web data.  Unfortunately building the model on Server-side data is not an option, nor is it an option to evaluate this model on Client-side data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model will be based on metrics collected on specific visitors.  This is a real time system that will be calculating a likelihood based on metrics collected while visitors browse the website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for approaches to ensure the highest possible accuracy on the model evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have the following ideas,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Clean the Server-side data by removing webpages that are never seen Client-side.&lt;/li&gt;&#xA;&lt;li&gt;Collect additional data Server-side data to make the Server-side data more closely resemble Client-side data.&lt;/li&gt;&#xA;&lt;li&gt;Collect data on the Client and send this data to the Server.  This is possible and may be the best solution, but is currently undesirable. &lt;/li&gt;&#xA;&lt;li&gt;Build one or more models that estimate Client-side Visitor metrics from Server-side Visitor metrics and use these estimates in the Likelihood model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any other thoughts on evaluating over one Population while training (and testing) on another Population?&lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2014-08-02T04:54:14.757" Title="Modelling on one Population and Evaluating on another Population" Tags="|data-cleaning|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="888" PostTypeId="2" ParentId="887" CreationDate="2014-08-02T04:21:49.927" Score="0" Body="&lt;p&gt;I'm not an expert on this, so take my advice with a grain of salt. It's not clear for me what is the relationship between server-side and client-side data. Are they both &lt;strong&gt;representative&lt;/strong&gt; of the same population? If Yes, I think it's OK to use different data sets for testing/training and evaluating your models. If No, I think it might be a good idea to use some &lt;strong&gt;resampling&lt;/strong&gt; technique, such as &lt;em&gt;bootstrapping&lt;/em&gt;, &lt;em&gt;jackknifing&lt;/em&gt; or &lt;em&gt;cross-validation&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-08-02T04:21:49.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="889" PostTypeId="2" ParentId="887" CreationDate="2014-08-02T04:54:14.757" Score="2" Body="&lt;p&gt;If the users who you are getting client-side data from are from the same population of users who you would get server-side data from. If that is true, then you aren't really training on one population and applying to another. The main difference is that the client side data happened in the past (by necessity unless you are constantly refitting your model) and the server side data will come in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's reformulate the question in terms of models rather than web clients and servers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are fitting a model on one dataset and applying it to another. That is the classic use of predictive modeling/machine learning. Models use features from the data to make estimates of some parameter or parameters. Once you have a fitted (and tested) model, all that you need is the same set of features to feed into the model to get your estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just make sure to model on a set of features (aka variables) that are available on the client-side and server-side. If that isn't possible, ask that question separately.&lt;/p&gt;&#xA;" OwnerUserId="2666" LastActivityDate="2014-08-02T04:54:14.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="890" PostTypeId="2" ParentId="423" CreationDate="2014-08-02T14:55:37.347" Score="1" Body="&lt;p&gt;If your work is parallelizable enough for a distributed network of cpus to make a difference, why not try to run it on the gpu instead? That will require rather less investment than a large network of cpus with individual software licenses and still provide parallel processing which you can do runtime tracking on yourself.&lt;/p&gt;&#xA;" OwnerUserId="2809" LastActivityDate="2014-08-02T14:55:37.347" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="893" PostTypeId="1" CreationDate="2014-08-03T13:07:24.143" Score="132" ViewCount="379486" Body="&lt;p&gt;I am building a regression model and I need to calculate the below to check for correlations&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Correlation between 2 Multi level categorical variables&lt;/li&gt;&#xA;&lt;li&gt;Correlation between a Multi level categorical variable and&#xA;continuous variable &lt;/li&gt;&#xA;&lt;li&gt;VIF(variance inflation factor) for a Multi&#xA;level categorical variables&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I believe its wrong to use Pearson correlation coefficient for the above scenarios because Pearson only works for 2 continuous variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please answer the below questions&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Which correlation coefficient works best for the above cases ? &lt;/li&gt;&#xA;&lt;li&gt;VIF calculation only works for continuous data so what is the&#xA;alternative? &lt;/li&gt;&#xA;&lt;li&gt;What are the assumptions I need to check before I use the correlation coefficient you suggest? &lt;/li&gt;&#xA;&lt;li&gt;How to implement them in SAS &amp;amp; R?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1151" LastEditorUserId="97" LastEditDate="2014-08-07T23:38:37.457" LastActivityDate="2017-09-27T14:52:39.337" Title="How to get correlation between two categorical variable and a categorical variable and continuous variable?" Tags="|r|statistics|correlation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="895" PostTypeId="1" AcceptedAnswerId="901" CreationDate="2014-08-03T16:09:35.750" Score="4" ViewCount="188" Body="&lt;p&gt;I assume that each person on Facebook is represented as a node (of a Graph) in Facebook, and relationship/friendship between each person(node) is represented as an edge between the involved nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that there are millions of people on Facebook, how is the Graph stored?&lt;/p&gt;&#xA;" OwnerUserId="2756" LastActivityDate="2014-08-04T15:59:12.573" Title="Facebook's Huge Database" Tags="|graphs|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="896" PostTypeId="1" AcceptedAnswerId="932" CreationDate="2014-08-03T16:38:38.853" Score="4" ViewCount="2003" Body="&lt;p&gt;I am trying to implement the Brown Clustering Algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Paper details: &quot;Class-Based n-gram Models of Natural Language&quot; by Brown et al&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is supposed to in &lt;code&gt;O(|V|k^2)&lt;/code&gt; where &lt;code&gt;|V|&lt;/code&gt; is the size of the vocabulary and k is the number of clusters. I am unable to implement it this efficiently. In fact, the best I can manage is &lt;code&gt;O(|V|k^3)&lt;/code&gt; which is too slow. My current implementation for the main part of the algorithm is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for w = number of clusters + 1 to |V|&#xA;{&#xA;   word = next most frequent word in the corpus&#xA;&#xA;   assign word to a new cluster &#xA;&#xA;   initialize MaxQuality to 0&#xA;&#xA;   initialize ArgMax vector to (0,0)&#xA;&#xA;   for i = 0 to number of clusters - 1 &#xA;   {&#xA;      for j = i to number of clusters&#xA;      {&#xA;         Quality = Mutual Information if we merge cluster i and cluster j&#xA;&#xA;         if Quality &amp;gt; MaxQuality&#xA;         {&#xA;            MaxQuality = Quality &#xA;&#xA;            ArgMax = (i,j) &#xA;         }&#xA;      }&#xA;   }&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I compute quality as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Before entering the second loop compute the pre-merge quality i.e. quality before doing any merges.&#xA;2. Every time a cluster-pair merge step is considered:&#xA;    i. assign quality := pre-merge quality&#xA;   ii. quality = quality - any terms in the mutual information equation that contain cluster i or cluster j (pre-merge)&#xA;  iii. quality = quality + any terms in the mutual information equation that contain (cluster i U cluster j)  (post-merge)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my implementation, the first loop has approx |V| iterations, the second and third loop approx k iterations each. To compute quality at each step requires approx a further k iterations. In total it runs in &lt;code&gt;(|V|k^3)&lt;/code&gt; time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you get it to run in &lt;code&gt;(|V|k^2)&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastEditorUserId="11097" LastEditDate="2017-10-04T12:03:07.733" LastActivityDate="2017-10-04T12:03:07.733" Title="How to implement Brown Clustering Algorithm in O(|V|k^2)" Tags="|nlp|efficiency|clustering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="897" PostTypeId="2" ParentId="895" CreationDate="2014-08-03T20:21:08.150" Score="3" Body="&lt;p&gt;Having worked with Facebook data a bit (harvested from Facebook users) we stored it just as a pair of values: USER_ID, FRIEND_USER_ID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I guess your questions is a bit deeper? You can store it in different ways, depending on your research question. One interesting option is triads for example - &lt;a href=&quot;http://mypersonality.org/wiki/doku.php?id=list_of_variables_available#triads&quot; rel=&quot;nofollow&quot;&gt;http://mypersonality.org/wiki/doku.php?id=list_of_variables_available#triads&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2014-08-03T20:21:08.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="898" PostTypeId="2" ParentId="893" CreationDate="2014-08-04T09:42:03.590" Score="128" Body="&lt;h2&gt;Two Categorical Variables&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Checking if two categorical variables are independent can be done with Chi-Squared test of independence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a typical &lt;a href=&quot;https://en.wikipedia.org/wiki/Chi-square_test&quot; rel=&quot;noreferrer&quot;&gt;Chi-Square test&lt;/a&gt;: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the actual values are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There also exists a &lt;a href=&quot;http://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V&quot; rel=&quot;noreferrer&quot;&gt;Crammer's V&lt;/a&gt; that is a measure of correlation that follows from this test&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Suppose we have two variables&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;gender: male and female&lt;/li&gt;&#xA;&lt;li&gt;city: Blois and Tours&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We observed the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/zcCfV.png&quot; alt=&quot;observed values&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are gender and city independent? Let's perform a Chi-Squred test. Null hypothesis: they are independent, Alternative hypothesis is that they are correlated in some way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the Null hypothesis, we assume uniform distribution. So our expected values are the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/H8bKJ.png&quot; alt=&quot;expected value&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we run the chi-squared test and the resulting p-value here can be seen as a measure of correlation between these two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compute Crammer's V we first find the normalizing factor chi-squared-max which is typically the size of the sample, divide the chi-square by it and take a square root &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/v7HY6.png&quot; alt=&quot;crammers v&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;R&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tbl = matrix(data=c(55, 45, 20, 30), nrow=2, ncol=2, byrow=T)&#xA;dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))&#xA;&#xA;chi2 = chisq.test(tbl, correct=F)&#xA;c(chi2$statistic, chi2$p.value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here the p value is 0.08 - quite small, but still not enough to reject the hypothesis of independence. So we can say that the &quot;correlation&quot; here is 0.08&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also compute V: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sqrt(chi2$statistic / sum(tbl))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And get 0.14 (the smaller v, the lower the correlation) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider another dataset &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Gender&#xA;City  M  F&#xA;   B 51 49&#xA;   T 24 26&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this, it would give the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tbl = matrix(data=c(51, 49, 24, 26), nrow=2, ncol=2, byrow=T)&#xA;dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))&#xA;&#xA;chi2 = chisq.test(tbl, correct=F)&#xA;c(chi2$statistic, chi2$p.value)&#xA;&#xA;sqrt(chi2$statistic / sum(tbl))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The p-value is 0.72 which is far closer to 1, and v is 0.03 - very close to 0&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Categorical vs Numerical Variables&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For this type we typically perform &lt;a href=&quot;http://en.wikipedia.org/wiki/F_test#One-way_ANOVA_example&quot; rel=&quot;noreferrer&quot;&gt;One-way ANOVA test&lt;/a&gt;: we calculate in-group variance and intra-group variance and then compare them.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;We want to study the relationship between absorbed fat from donuts vs the type of fat used to produce donuts (example is taken from &lt;a href=&quot;http://statland.org/R/R/R1way.htm&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/LMOS3.png&quot; alt=&quot;donuts&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any dependence between the variables?&#xA;For that we conduct ANOVA test and see that the p-value is just 0.007 - there's no correlation between these variables. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;R&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t1 = c(164, 172, 168, 177, 156, 195)&#xA;t2 = c(178, 191, 197, 182, 185, 177)&#xA;t3 = c(175, 193, 178, 171, 163, 176)&#xA;t4 = c(155, 166, 149, 164, 170, 168)&#xA;&#xA;val = c(t1, t2, t3, t4)&#xA;fac = gl(n=4, k=6, labels=c('type1', 'type2', 'type3', 'type4'))&#xA;&#xA;aov1 = aov(val ~ fac)&#xA;summary(aov1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            Df Sum Sq Mean Sq F value  Pr(&amp;gt;F)   &#xA;fac          3   1636   545.5   5.406 0.00688 **&#xA;Residuals   20   2018   100.9                   &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we can take the p-value as the measure of correlation here as well.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Chi-square_test&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Chi-square_test&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mlwiki.org/index.php/Chi-square_Test_of_Independence&quot; rel=&quot;noreferrer&quot;&gt;http://mlwiki.org/index.php/Chi-square_Test_of_Independence&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://courses.statistics.com/software/R/R1way.htm&quot; rel=&quot;noreferrer&quot;&gt;http://courses.statistics.com/software/R/R1way.htm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mlwiki.org/index.php/One-Way_ANOVA_F-Test&quot; rel=&quot;noreferrer&quot;&gt;http://mlwiki.org/index.php/One-Way_ANOVA_F-Test&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mlwiki.org/index.php/Cramer%27s_Coefficient&quot; rel=&quot;noreferrer&quot;&gt;http://mlwiki.org/index.php/Cramer%27s_Coefficient&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastEditorUserId="37611" LastEditDate="2017-08-06T19:44:38.687" LastActivityDate="2017-08-06T19:44:38.687" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="899" PostTypeId="2" ParentId="895" CreationDate="2014-08-04T09:48:09.660" Score="1" Body="&lt;p&gt;When I worked with social network data, we stoted the &quot;friendship&quot; relation in a database in the table &lt;code&gt;Friends(friend_a, friend_b, ...)&lt;/code&gt; with a B-Tree index on &lt;code&gt;(friend_a, friend_b)&lt;/code&gt; plus also some partitioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In our case it was a little bit different since the graph was directed, so it wasn't really &quot;friendship&quot;, but rather &quot;following/follower&quot; relationship. But for friendship I would just store two edges: both &lt;code&gt;(friend_a, friend_b)&lt;/code&gt; and &lt;code&gt;(friend_b, friend_a)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We used MySQL to store the data, if it matters, but I guess it shouldn't.&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-08-04T09:48:09.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="900" PostTypeId="2" ParentId="700" CreationDate="2014-08-04T15:48:29.757" Score="4" Body="&lt;p&gt;I think hierarchical clustering would be more time efficient in your case (with a single dimension).&#xA;Depending on your task, you may implement something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having N datapoints d&lt;sub&gt;i&lt;/sub&gt; with their 1-dimension value x&lt;sub&gt;i&lt;/sub&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sort datapoints based on their x&lt;sub&gt;i&lt;/sub&gt; value. &lt;/li&gt;&#xA;&lt;li&gt;Calculate distances between adjacent datapoints (N-1 distances). Each distance must be assigned a pair of original datapoints (d&lt;sub&gt;i&lt;/sub&gt;, d&lt;sub&gt;j&lt;/sub&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Sort distances in descending order to generate list of datapoint pairs (d&lt;sub&gt;i&lt;/sub&gt;, d&lt;sub&gt;j&lt;/sub&gt;), starting from the closest one.&lt;/li&gt;&#xA;&lt;li&gt;Iteratively unite datapoints (d&lt;sub&gt;i&lt;/sub&gt;, d&lt;sub&gt;j&lt;/sub&gt;) into clusters, starting from beginning of the list (the closest pair). (Depending on current state of d&lt;sub&gt;i&lt;/sub&gt; and d&lt;sub&gt;j&lt;/sub&gt;, uniting them means: (a) creating new cluster for two unclustered datapoints, (b) adding a datapoint to existing cluster and (c) uniting two clusters.)&lt;/li&gt;&#xA;&lt;li&gt;Stop uniting, if the distance is over some threshold.&lt;/li&gt;&#xA;&lt;li&gt;Create singleton clusters for datapoints which did not get into clusters.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This algorithm implements &lt;a href=&quot;http://en.wikipedia.org/wiki/Single-linkage_clustering&quot; rel=&quot;nofollow&quot;&gt;single linkage&lt;/a&gt; clustering. It can be tuned easily to implement average linkage. &lt;a href=&quot;http://en.wikipedia.org/wiki/Complete_linkage_clustering&quot; rel=&quot;nofollow&quot;&gt;Complete linkage&lt;/a&gt; will be less efficient, but maybe easier ones will give good results depending on your data and task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe for 200K datapoints it must take under second, if you use proper data structures for above operations.&lt;/p&gt;&#xA;" OwnerUserId="2574" LastActivityDate="2014-08-04T15:48:29.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="901" PostTypeId="2" ParentId="895" CreationDate="2014-08-04T15:59:12.573" Score="6" Body="&lt;p&gt;Strange as it sounds, graphs and graph databases are typically implemented as &lt;a href=&quot;http://docs.oracle.com/javase/7/docs/api/java/util/LinkedList.html&quot;&gt;linked lists&lt;/a&gt;. As alluded to &lt;a href=&quot;http://docs.neo4j.org/chunked/stable/cypher-cookbook-newsfeed.html&quot;&gt;here&lt;/a&gt;, even the most popular/performant graph database out there (neo4j), is secretly using something akin to a doubly-linked list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Representing a graph this way has a number of significant benefits, but also a few drawbacks. Firstly, representing a graph this way means that you can do edge-based insertions in near-constant time. Secondly, this means that traversing the graph can happen extremely quickly, if we're only looking to either step up or down a linked list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The biggest drawback of this though comes from something sometimes called The Justin Bieber Effect, where nodes with a large number of connections tend to be extremely slow to evaluate. Imagine having to traverse a million semi-redundant links every time someone linked to Justin Bieber.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the awesome folks over at Neo4j are working on the second problem, but I'm not sure how they're going about it, or how much success they've had.&lt;/p&gt;&#xA;" OwnerUserId="548" LastActivityDate="2014-08-04T15:59:12.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="902" PostTypeId="1" AcceptedAnswerId="908" CreationDate="2014-08-04T19:10:57.187" Score="4" ViewCount="2401" Body="&lt;p&gt;Are there any general rules that one can use to infer what can be learned/generalized from a particular data set?  Suppose the dataset was taken from a sample of people.  Can these rules be stated as functions of the sample or total population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the above may be vague, so a case scenario: Users participate in a search task, where the data are their queries, clicked results, and the HTML content (text only) of those results.  Each of these are tagged with their user and timestamp.  A user may generate a few pages - for a simple fact-finding task - or hundreds of pages - for a longer-term search task, like for class report.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  In addition to generalizing about a population, given a sample, I'm interested in generalizing about an individual's overall search behavior, given a time slice.  Theory and paper references are a plus!&lt;/p&gt;&#xA;" OwnerUserId="1097" LastEditorUserId="1097" LastEditDate="2014-08-04T19:23:09.483" LastActivityDate="2014-08-07T00:20:05.817" Title="When is there enough data for generalization?" Tags="|machine-learning|data-mining|statistics|search|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="903" PostTypeId="1" AcceptedAnswerId="909" CreationDate="2014-08-04T22:27:10.837" Score="15" ViewCount="5245" Body="&lt;p&gt;I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution. &#xA;In fact the distribution resembles more closely the Landau Distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe the independent t-test requires that the samples be at least approximately normally distributed, which discourages me using the t-test as a valid method of significance testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But my question is: &#xA;&lt;strong&gt;At what point can one say that the t-test is not a good method of significance testing?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or put another way, how can one qualify how reliable the p-values of a t-test are, given only the data set?&lt;/p&gt;&#xA;" OwnerUserId="2830" LastEditorUserId="11097" LastEditDate="2016-01-06T11:48:25.473" LastActivityDate="2016-01-06T11:48:25.473" Title="Analyzing A/B test results which are not normally distributed, using independent t-test" Tags="|dataset|statistics|ab-test|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="904" PostTypeId="1" CreationDate="2014-08-04T19:21:45.067" Score="21" ViewCount="13116" Body="&lt;p&gt;I need to generate periodic (daily, monthly) web analytics dashboard reports. They will be static and don't require interaction, so imagine a PDF file as the target output. The reports will mix tables and charts (mainly sparkline and bullet graphs created with ggplot2). Think Stephen Few/Perceptual Edge style dashboards, such as: &lt;img src=&quot;https://i.stack.imgur.com/Edh2e.png&quot; alt=&quot;sample dashboard&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but applied to web analytics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on what packages to use creating these dashboard reports? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first intuition is to use R markdown and knitr, but perhaps you've found a better solution. I can't seem to find rich examples of dashboards generated from R. &lt;/p&gt;&#xA;" OwnerDisplayName="aiolias" LastEditorUserId="1156" LastEditDate="2014-08-07T23:37:26.890" LastActivityDate="2021-03-11T19:06:05.303" Title="What do you use to generate a dashboard in R?" Tags="|r|visualization|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="905" PostTypeId="2" ParentId="904" CreationDate="2014-08-04T19:28:38.173" Score="9" Body="&lt;p&gt;&lt;a href=&quot;http://shiny.rstudio.com&quot;&gt;Shiny&lt;/a&gt; is a framework for generating HTML-based apps that execute R code dynamically. Shiny apps can stand alone or be built into Markdown documents with &lt;code&gt;knitr&lt;/code&gt;, and Shiny development is fully integrated into RStudio. There's even a free service called &lt;a href=&quot;https://www.shinyapps.io&quot;&gt;shinyapps.io&lt;/a&gt; for hosting Shiny apps, the &lt;code&gt;shiny&lt;/code&gt; package has functions for deploying Shiny apps directly from R, and RStudio has a GUI interface for calling those functions. There's plenty more info in the Tutorial section of the site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since it essentially &quot;compiles&quot; the whole thing to JavaScript and HTML, you can use CSS to freely change the formatting and layout, although Shiny has decent wrapper functionality for this. But it just so happens that their default color scheme is similar to the one in the screenshot you posted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: I just realized you don't need them to be dynamic. Shiny still makes very nice-looking webpages out of the box, with lots of options for rearranging elements. There's also functionality for downloading plots, so you can generate your dashboard every month by just updating your data files in the app, and then saving the resulting image to PDF.&lt;/p&gt;&#xA;" OwnerUserId="1156" OwnerDisplayName="ssdecontrol" LastActivityDate="2014-08-04T19:34:39.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="906" PostTypeId="2" ParentId="812" CreationDate="2014-08-04T23:37:27.877" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://mrvar.fdv.uni-lj.si/pajek/PajekXXL.htm&quot; rel=&quot;nofollow&quot;&gt;PajekXXL&lt;/a&gt; is designed to handle enormous networks. But Pajek is also kind of a bizarre program with an unintuitive interface.&lt;/p&gt;&#xA;" OwnerUserId="1156" LastActivityDate="2014-08-04T23:37:27.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="907" PostTypeId="2" ParentId="904" CreationDate="2014-08-05T07:17:34.750" Score="16" Body="&lt;p&gt;I think that &lt;code&gt;Shiny&lt;/code&gt; is an overkill in this situation and doesn't match your requirement of &lt;em&gt;dashboard reports&lt;/em&gt; to be &lt;strong&gt;static&lt;/strong&gt;. I guess, that your use of the term &quot;dashboard&quot; is a bit confusing, as some people might consider that it has more emphasis of &lt;strong&gt;interactivity&lt;/strong&gt; (&lt;em&gt;real-time dashboards&lt;/em&gt;), rather than &lt;strong&gt;information layout&lt;/strong&gt;, as is my understanding (confirmed by the &quot;static&quot; requirement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation to you is to use &lt;strong&gt;R Markdown&lt;/strong&gt; and &lt;strong&gt;knitr&lt;/strong&gt;, especially since these packages have much lower learning curve than &lt;strong&gt;Shiny&lt;/strong&gt;. Moreover, I have recently run across an R package, which, in my view, ideally suits your requirement of embedding small charts/plots in a report, as presented on your picture above. This package generates static or dynamic &lt;em&gt;graphical tables&lt;/em&gt; and is called &lt;strong&gt;sparkTable&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/sparkTable&quot;&gt;http://cran.r-project.org/web/packages/sparkTable&lt;/a&gt;). Its vignette is available here (there is no link to it on the package's home page): &lt;a href=&quot;http://publik.tuwien.ac.at/files/PubDat_228663.pdf&quot;&gt;http://publik.tuwien.ac.at/files/PubDat_228663.pdf&lt;/a&gt;. Should you ever need some &lt;em&gt;interactivity&lt;/em&gt;, &lt;code&gt;sparkTable&lt;/code&gt; provides some via its simple interface to &lt;code&gt;Shiny&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-08-05T07:17:34.750" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="908" PostTypeId="2" ParentId="902" CreationDate="2014-08-05T08:09:17.240" Score="6" Body="&lt;p&gt;It is my understanding that &lt;em&gt;random sampling&lt;/em&gt; is a &lt;strong&gt;mandatory condition&lt;/strong&gt; for making any &lt;em&gt;generalization&lt;/em&gt; statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, clarifying the @ffriend's comment, I believe that you have to &lt;strong&gt;calculate&lt;/strong&gt; needed &lt;em&gt;sample size&lt;/em&gt;, based on desired values of &lt;em&gt;confidence interval&lt;/em&gt;, &lt;em&gt;effect size&lt;/em&gt;, &lt;em&gt;statistical power&lt;/em&gt; and &lt;em&gt;number of predictors&lt;/em&gt; (this is based on Cohen's work - see References section at the following link). For multiple regression, you can use the following calculator: &lt;a href=&quot;http://www.danielsoper.com/statcalc3/calc.aspx?id=1&quot;&gt;http://www.danielsoper.com/statcalc3/calc.aspx?id=1&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information on &lt;strong&gt;how to select, calculate and interpret effect sizes&lt;/strong&gt; can be found in the following nice and comprehensive paper, which is freely available: &lt;a href=&quot;http://jpepsy.oxfordjournals.org/content/34/9/917.full&quot;&gt;http://jpepsy.oxfordjournals.org/content/34/9/917.full&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using &lt;code&gt;R&lt;/code&gt; (and even, if you don't), you may find the following Web page on &lt;strong&gt;confidence intervals and R&lt;/strong&gt; interesting and useful: &lt;a href=&quot;http://osc.centerforopenscience.org/static/CIs_in_r.html&quot;&gt;http://osc.centerforopenscience.org/static/CIs_in_r.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the following &lt;strong&gt;comprehensive guide&lt;/strong&gt; to survey &lt;strong&gt;sampling&lt;/strong&gt; can be helpful, even if you're not using survey research designs. In my opinion, it contains a wealth of useful information on &lt;em&gt;sampling methods&lt;/em&gt;, &lt;em&gt;sampling size determination&lt;/em&gt; (including calculator) and much more: &lt;a href=&quot;http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm&quot;&gt;http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-08-05T08:09:17.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="909" PostTypeId="2" ParentId="903" CreationDate="2014-08-05T08:12:15.647" Score="10" Body="&lt;p&gt;The distribution of your data doesn't need to be normal, it's the &lt;a href=&quot;https://en.wikipedia.org/wiki/Sampling_distribution&quot; rel=&quot;noreferrer&quot;&gt;Sampling Distribution&lt;/a&gt; that has to be nearly normal. If your sample size is big enough, then the sampling distribution of means from Landau Distribution should to be nearly normal, due to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;noreferrer&quot;&gt;Central Limit Theorem&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it means you should be able to safely use t-test with your data.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let's consider this example: suppose we have a population with &lt;a href=&quot;https://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;noreferrer&quot;&gt;Lognormal distribution&lt;/a&gt; with mu=0 and sd=0.5 (it looks a bit similar to Landau)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Hw5mM.png&quot; alt=&quot;lognormal density&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we sample 30 observations 5000 times from this distribution each time calculating the mean of the sample &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is what we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/wjueS.png&quot; alt=&quot;sampling distribution&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looks quite normal, doesn't it? If we increase the sample size, it's even more apparent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/M0FQS.png&quot; alt=&quot;sampling distribution&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;R code&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = seq(0, 4, 0.05)&#xA;y = dlnorm(x, mean=0, sd=0.5)&#xA;plot(x, y, type='l', bty='n')&#xA;&#xA;&#xA;n = 30&#xA;m = 1000&#xA;&#xA;set.seed(0)&#xA;samp = rep(NA, m)&#xA;&#xA;for (i in 1:m) {&#xA;  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))&#xA;}&#xA;&#xA;hist(samp, col='orange', probability=T, breaks=25, main='sample size = 30')&#xA;x = seq(0.5, 1.5, 0.01)&#xA;lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))&#xA;&#xA;&#xA;n = 300&#xA;samp = rep(NA, m)&#xA;&#xA;for (i in 1:m) {&#xA;  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))&#xA;}&#xA;&#xA;hist(samp, col='orange', probability=T, breaks=25, main='sample size = 300')&#xA;x = seq(1, 1.25, 0.005)&#xA;lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-08-05T08:12:15.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="910" PostTypeId="2" ParentId="586" CreationDate="2014-08-05T09:07:42.393" Score="13" Body="&lt;p&gt;Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Sentence_boundary_disambiguation&quot; rel=&quot;noreferrer&quot;&gt;Sentence breaker&lt;/a&gt; - to split text (usually, text paragraphs) to sentences. Even in English it can be hard for some cases like &quot;Mr. and Mrs. Brown stay in room no. 20.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Tokenization&quot; rel=&quot;noreferrer&quot;&gt;Tokenizer&lt;/a&gt; - to split text or sentences to words or word-level units, including punctuation. This task is not trivial for languages with no spaces and no stable understanding of word boundaries (e.g. Chinese, Japanese)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/POS_tagger&quot; rel=&quot;noreferrer&quot;&gt;Part-of-speech Tagger&lt;/a&gt; - to guess part of speech of each word in the context of sentence; usually each word is assigned a so-called POS-tag from a tagset developed in advance to serve your final task (for example, parsing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Lemmatization&quot; rel=&quot;noreferrer&quot;&gt;Lemmatizer&lt;/a&gt; - to convert a given word into its canonical form (&lt;a href=&quot;http://en.wikipedia.org/wiki/Lemma_(morphology)&quot; rel=&quot;noreferrer&quot;&gt;lemma&lt;/a&gt;). Usually you need to know the word's POS-tag. For example, word &quot;heating&quot; as gerund must be converted to &quot;heat&quot;, but as noun it must be left unchanged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Parser&quot; rel=&quot;noreferrer&quot;&gt;Parser&lt;/a&gt; - to perform syntactic analysis of the sentence and build a syntactic tree or graph. There're two main ways to represent syntactic structure of sentence: via &lt;a href=&quot;http://en.wikipedia.org/wiki/Dependency_grammar#Dependency_vs._constituency&quot; rel=&quot;noreferrer&quot;&gt;constituency or dependency&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Automatic_summarization&quot; rel=&quot;noreferrer&quot;&gt;Summarizer&lt;/a&gt; - to generate a short summary of the text by selecting a set of top informative sentences of the document, representing its main idea. However can be done in more intelligent manner than just selecting the sentences from existing ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;noreferrer&quot;&gt;Named Entity Recognition&lt;/a&gt; - to extract so-called named entities from the text. Named entities are the chunks of words from text, which refer to an entity of certain type. The types may include: geographic locations (countries, cities, rivers, ...), person names, organization names etc. Before going into NER task you must understand what do you want to get and, possible, predefine a taxonomy of named entity types to resolve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Coreference_resolution&quot; rel=&quot;noreferrer&quot;&gt;Coreference Resolution&lt;/a&gt; - to group named entities (or, depending on your task, any other text units) into clusters corresponding to a single real object/meaning. For example, &quot;B. Gates&quot;, &quot;William Gates&quot;, &quot;Founder of Microsoft&quot; etc. in one text may mean the same person, referenced by using different expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There're many other interesting NLP applications/annotators (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Category:Tasks_of_natural_language_processing&quot; rel=&quot;noreferrer&quot;&gt;NLP tasks category&lt;/a&gt;), sentiment analysis, machine translation etc.). There're many books on this, the classical book: &quot;Speech and Language Processing&quot; by Daniel Jurafsky and James H. Martin., but it can be too detailed for you.&lt;/p&gt;&#xA;" OwnerUserId="2574" LastEditorUserId="12363" LastEditDate="2015-10-12T07:20:26.220" LastActivityDate="2015-10-12T07:20:26.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="911" PostTypeId="2" ParentId="903" CreationDate="2014-08-05T10:15:33.713" Score="1" Body="&lt;p&gt;Basically an independent t-test or a 2 sample t-test is used to check if the averages of the two samples are significantly different. Or, to put in another words, if there is a significant difference between the means of the two samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the means of those 2 samples are two statistics, which according with CLT, have a normal distribution, if provided enough samples. Note that CLT works no matter of the distribution from which the mean statistic is built. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally one can use a z-test, but if the variances are estimated from the sample (because it is unknown), some additional uncertainty is introduced, which is incorporated in t distribution. That's why 2-sample t-test applies here.  &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2014-08-05T10:15:33.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="912" PostTypeId="2" ParentId="744" CreationDate="2014-08-05T12:29:05.300" Score="8" Body="&lt;p&gt;There is an excellent comparison of the common inner-product-based similarity metrics &lt;a href=&quot;http://brenocon.com/blog/2012/03/cosine-similarity-pearson-correlation-and-ols-coefficients/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In particular, Cosine Similarity is normalized to lie within &lt;span class=&quot;math-container&quot;&gt;$[-1,1]$&lt;/span&gt;, unlike the dot product which can be any real number. But, as everyone else is saying, that will require ignoring the magnitude of the vectors. Personally, I think that's a good thing. I think of magnitude as an internal (within-vector) structure, and angle between vectors as external (between vector) structure. They are different things and (in my opinion) are often best analyzed separately. I can't imagine a situation where I would rather compute inner products than compute cosine similarities and just compare the magnitudes afterward.&lt;/p&gt;&#xA;" OwnerUserId="1156" LastEditorUserId="1156" LastEditDate="2020-09-04T15:43:15.887" LastActivityDate="2020-09-04T15:43:15.887" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="913" PostTypeId="2" ParentId="902" CreationDate="2014-08-05T12:58:16.000" Score="4" Body="&lt;p&gt;There are two rules for generalizability:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The sample must be &lt;strong&gt;representative&lt;/strong&gt;. In expectation, at least, the distribution of features in your sample must match the distribution of features in the population. When you are fitting a model with a response variable, &lt;em&gt;this includes features that you do not observe, but that affect any response variables in your model&lt;/em&gt;. Since it is, in many cases, impossible to know what you do not observe, &lt;strong&gt;random sampling&lt;/strong&gt; is used.&lt;/p&gt;&#xA;&lt;p&gt;The idea with randomization is that a random sample, up to sampling error, &lt;em&gt;must&lt;/em&gt; accurately reflect the distribution of all features in the population, observed and otherwise. This is why &lt;strong&gt;randomization is the &amp;quot;gold standard,&amp;quot;&lt;/strong&gt; but if sample control is available by some other technique, or it is defensible to argue that there are no omitted features, then it isn't always necessary.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your sample must be &lt;strong&gt;large enough&lt;/strong&gt; that the effect of &lt;strong&gt;sampling error&lt;/strong&gt; on the feature distribution is relatively small. This is, again, to ensure representativeness. But deciding who to sample is different from deciding how many people to sample.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Since it sounds like you're fitting a model, there's the additional consideration that certain important combinations of features could be relatively rare in the population. This is not an issue for generalizability, but it bears heavily on your considerations for sample size. For instance, I'm working on a project now with (non-big) data that was originally collected to understand the experiences of minorities in college. As such, it was critically important to ensure that &lt;strong&gt;statistical power&lt;/strong&gt; was high &lt;em&gt;specifically in the minority subpopulation&lt;/em&gt;. For this reason, blacks and Latinos were deliberately &lt;strong&gt;oversampled&lt;/strong&gt;. However, the proportion by which they were oversampled was also recorded. These are used to compute survey weights. These can be used to  re-weight the sample so as to reflect the estimated population proportions, in the event that a representative sample is required.&lt;/p&gt;&#xA;&lt;p&gt;An additional consideration arises if your model is hierarchical. A canonical use for a hierarchical model is one of children's behavior in schools. Children are &amp;quot;grouped&amp;quot; by school and share school-level traits. Therefore a representative sample of schools is required, and within each school a representative sample of children is required. This leads to &lt;strong&gt;stratified sampling&lt;/strong&gt;. This and some other sampling designs are reviewed in surprising depth on &lt;a href=&quot;http://en.wikipedia.org/wiki/Sampling_(statistics)#Sampling_methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1156" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2014-08-05T12:58:16.000" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="915" PostTypeId="1" CreationDate="2014-08-05T18:36:12.753" Score="10" ViewCount="350" Body="&lt;p&gt;Is there a known general table of statistical techniques that explain how they scale with sample size and dimension? For example, a friend of mine told me the other day that the computation time of simply quick-sorting one dimensional data of size n goes as n*log(n).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, if we regress y against X where X is a d-dimensional variable, does it go as O(n^2*d)? How does it scale if I want to find the solution via exact Gauss-Markov solution vs numerical least squares with Newton method? Or simply getting the solution vs using significance tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I more want a good source of answers (like a paper that summarizes the scaling of various statistical techniques) than a good answer here. Like, say, a list that includes the scaling of multiple regression, logistic regression, PCA, cox proportional hazard regression, K-means clustering, etc.&lt;/p&gt;&#xA;" OwnerUserId="2841" LastEditorUserId="2841" LastEditDate="2014-08-05T18:46:46.157" LastActivityDate="2014-08-14T17:28:40.453" Title="How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?" Tags="|bigdata|statistics|efficiency|scalability|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="916" PostTypeId="2" ParentId="915" CreationDate="2014-08-05T20:24:09.200" Score="6" Body="&lt;p&gt;Most of the efficient (and non trivial) statistic algorithms are iterative in nature so that the worst case analysis &lt;code&gt;O()&lt;/code&gt; is irrelevant as the worst case is 'it fails to converge'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, when you have a lot of data, even the linear algorithms (&lt;code&gt;O(n)&lt;/code&gt;) can be slow and you then need to focus on the constant 'hidden' behind the notation. For instance, computing the variance of a single variate is naively done scanning the data twice (once for computing an estimate of the mean, and then once to estimate the variance). But it also can be done in &lt;a href=&quot;http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm&quot;&gt;one pass&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For iterative algorithms, what is more important is convergence rate and number of parameters as a function of the data dimensionality, an element that greatly influences convergence. Many models/algorithm grow a number of parameters that is exponential with the number of variables (e.g. splines) while some other grow linearly (e.g. support vector machines, random forests, ...) &lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-08-05T20:24:09.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="917" PostTypeId="1" AcceptedAnswerId="918" CreationDate="2014-08-05T20:45:01.383" Score="12" ViewCount="2909" Body="&lt;p&gt;I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to solve for the set of 40 coefficients that minimizes the total sum-of-square error between y and the predicted value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that the matrix is very sparse and I do not know the best way to solve the system of equations with sparse data. An example of the dataset is shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   y    x1  x2 x3 x4 x5 x6 ... x40&#xA;87169   14  0  1  0  0  2  ... 0 &#xA;46449   0   0  4  0  1  4  ... 12&#xA;846449  0   0  0  0  0  3  ... 0&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am currently using a Genetic Algorithm to solve this and the results are coming out &#xA;with roughly a factor of two difference between observed and expected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest different methods or techniques which are capable of solving a set of equations with sparse data.&lt;/p&gt;&#xA;" OwnerUserId="802" LastEditorUserId="20343" LastEditDate="2016-10-18T15:44:16.657" LastActivityDate="2016-10-18T15:44:16.657" Title="Solving a system of equations with sparse data" Tags="|machine-learning|regression|algorithms|genetic|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="918" PostTypeId="2" ParentId="917" CreationDate="2014-08-05T22:34:04.550" Score="11" Body="&lt;p&gt;If I understand you correctly, this is the case of &lt;strong&gt;multiple linear regression with sparse data&lt;/strong&gt; (&lt;em&gt;sparse regression&lt;/em&gt;). Assuming that, I hope you will find the following &lt;strong&gt;resources&lt;/strong&gt; useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) NCSU &lt;strong&gt;lecture slides on sparse regression&lt;/strong&gt; with overview of algorithms, notes, formulas, graphics and references to literature: &lt;a href=&quot;http://www.stat.ncsu.edu/people/zhou/courses/st810/notes/lect23sparse.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.stat.ncsu.edu/people/zhou/courses/st810/notes/lect23sparse.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;code&gt;R&lt;/code&gt; ecosystem offers many &lt;strong&gt;packages&lt;/strong&gt;, useful for sparse regression analysis, including:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Matrix&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/Matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/Matrix&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;SparseM&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/SparseM&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/SparseM&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MatrixModels&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/MatrixModels&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/MatrixModels&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;glmnet&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/glmnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/glmnet&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;flare&lt;/strong&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/flare&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/flare&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;3) A blog post with an &lt;strong&gt;example of sparse regression solution&lt;/strong&gt;, based on &lt;code&gt;SparseM&lt;/code&gt;: &lt;a href=&quot;http://aleph-nought.blogspot.com/2012/03/multiple-linear-regression-with-sparse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://aleph-nought.blogspot.com/2012/03/multiple-linear-regression-with-sparse.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) A blog post on using &lt;strong&gt;sparse matrices in R&lt;/strong&gt;, which includes a &lt;strong&gt;primer&lt;/strong&gt; on using &lt;code&gt;glmnet&lt;/code&gt;: &lt;a href=&quot;http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) &lt;strong&gt;More examples and some discussion&lt;/strong&gt; on the topic can be found on &lt;strong&gt;StackOverflow&lt;/strong&gt;: &lt;a href=&quot;https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix&quot;&gt;https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt; (based on your comment):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying to solve an LP problem with constraints, you may find this &lt;strong&gt;theoretical paper&lt;/strong&gt; useful: &lt;a href=&quot;http://web.stanford.edu/group/SOL/papers/gmsw84.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://web.stanford.edu/group/SOL/papers/gmsw84.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, check R package &lt;strong&gt;limSolve&lt;/strong&gt;: &lt;a href=&quot;http://cran.r-project.org/web/packages/limSolve&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/limSolve&lt;/a&gt;. And, in general, check packages in &lt;em&gt;CRAN Task View&lt;/em&gt; &lt;strong&gt;&quot;Optimization and Mathematical Programming&quot;&lt;/strong&gt;: &lt;a href=&quot;http://cran.r-project.org/web/views/Optimization.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/views/Optimization.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, check the book &lt;strong&gt;&quot;Using R for Numerical Analysis in Science and Engineering&quot;&lt;/strong&gt; (by Victor A. Bloomfield). It has a section on solving systems of equations, represented by &lt;strong&gt;sparse matrices&lt;/strong&gt; (section 5.7, pages 99-104), which includes examples, based on some of the above-mentioned packages: &lt;a href=&quot;http://books.google.com/books?id=9ph_AwAAQBAJ&amp;amp;pg=PA99&amp;amp;lpg=PA99&amp;amp;dq=r+limsolve+sparse+matrix&amp;amp;source=bl&amp;amp;ots=PHDE8nXljQ&amp;amp;sig=sPi4n5Wk0M02ywkubq7R7KD_b04&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ei=FZjiU-ioIcjmsATGkYDAAg&amp;amp;ved=0CDUQ6AEwAw#v=onepage&amp;amp;q=r%20limsolve%20sparse%20matrix&amp;amp;f=false&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://books.google.com/books?id=9ph_AwAAQBAJ&amp;amp;pg=PA99&amp;amp;lpg=PA99&amp;amp;dq=r+limsolve+sparse+matrix&amp;amp;source=bl&amp;amp;ots=PHDE8nXljQ&amp;amp;sig=sPi4n5Wk0M02ywkubq7R7KD_b04&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ei=FZjiU-ioIcjmsATGkYDAAg&amp;amp;ved=0CDUQ6AEwAw#v=onepage&amp;amp;q=r%20limsolve%20sparse%20matrix&amp;amp;f=false&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-08-06T21:32:39.427" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="919" PostTypeId="1" AcceptedAnswerId="939" CreationDate="2014-08-06T08:41:44.967" Score="9" ViewCount="2497" Body="&lt;p&gt;Data set looks like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;25000 observations&lt;/li&gt;&#xA;&lt;li&gt;up to 15 predictors of different types: numeric, multi-class categorical, binary&lt;/li&gt;&#xA;&lt;li&gt;target variable is binary&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which cross validation method is typical for this type of problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default I'm using K-Fold. How many folds is enough in this case? (One of the models I use is random forest, which is time consuming...)&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2017-02-25T19:47:29.907" Title="Which cross-validation type best suits to binary classification problem" Tags="|classification|cross-validation|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="920" PostTypeId="1" AcceptedAnswerId="933" CreationDate="2014-08-06T09:02:01.033" Score="1" ViewCount="278" Body="&lt;p&gt;There is a text summarization project called SUMMARIST. Apparently it is able to perform abstractive text summarization. I want to give it a try but unfortunately the demo links on the website do not work. Does anybody have any information regarding this? How can I test this tool?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.isi.edu/natural-language/projects/SUMMARIST.html&quot; rel=&quot;nofollow&quot;&gt;http://www.isi.edu/natural-language/projects/SUMMARIST.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&#xA;PasMod&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2018-09-07T03:25:01.003" Title="SUMMARIST: Automated Text Summarization" Tags="|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="921" PostTypeId="1" AcceptedAnswerId="924" CreationDate="2014-08-06T09:03:20.857" Score="3" ViewCount="276" Body="&lt;p&gt;I am fitting a model in R.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use &lt;code&gt;createFolds&lt;/code&gt; method to create several &lt;code&gt;k&lt;/code&gt; folds from the data set&lt;/li&gt;&#xA;&lt;li&gt;loop through the folds, repeating the following on each iteration:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;train&lt;/code&gt; the model on k-1 folds&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;predict&lt;/code&gt; the outcomes for the i-th fold&lt;/li&gt;&#xA;&lt;li&gt;calculate prediction accuracy&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;average the accuracy&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does R have a function that makes folds itself, repeats model tuning/predictions and gives the average accuracy back?&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2014-08-06T12:16:22.850" Title="Avoid iterations while calculating average model accuracy" Tags="|r|accuracy|cross-validation|sampling|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="922" PostTypeId="1" AcceptedAnswerId="925" CreationDate="2014-08-06T09:08:08.113" Score="3" ViewCount="60" Body="&lt;p&gt;I have  set of documents and I want classify them to true and false &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these documents or I can take only some words that I interested in then I compare it with the documents. Which one is more efficient in classify document and can work with SVM.       &lt;/p&gt;&#xA;" OwnerUserId="2850" LastActivityDate="2014-08-06T13:15:51.460" Title="Can I classify set of documents using classifying method using limited number of concepts ?" Tags="|machine-learning|classification|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="923" PostTypeId="2" ParentId="919" CreationDate="2014-08-06T09:08:12.117" Score="3" Body="&lt;p&gt;I think in your case a 10-fold CV will be O.K. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is more important to randomize the cross validation process than selecting the ideal value for k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So repeat the CV process several times randomly and compute the variance of your classification result to determine if the results are realiable or not.&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2014-08-06T09:08:12.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="924" PostTypeId="2" ParentId="921" CreationDate="2014-08-06T12:16:22.850" Score="4" Body="&lt;p&gt;Yes, you can do all this using the Caret (&lt;a href=&quot;http://caret.r-forge.r-project.org/training.html&quot; rel=&quot;nofollow&quot;&gt;http://caret.r-forge.r-project.org/training.html&lt;/a&gt;) package in R. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fitControl &amp;lt;- trainControl(## 10-fold CV&#xA;                           method = &quot;repeatedcv&quot;,&#xA;                           number = 10,&#xA;                           ## repeated ten times&#xA;                           repeats = 10)&#xA;&#xA;gbmFit1 &amp;lt;- train(Class ~ ., data = training,&#xA;                 method = &quot;gbm&quot;,&#xA;                 trControl = fitControl,&#xA;                ## This last option is actually one&#xA;                ## for gbm() that passes through&#xA;                verbose = FALSE)&#xA;gbmFit1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which will give the output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Stochastic Gradient Boosting &#xA;&#xA;157 samples&#xA; 60 predictors&#xA;  2 classes: 'M', 'R' &#xA;&#xA;No pre-processing&#xA;Resampling: Cross-Validated (10 fold, repeated 10 times) &#xA;&#xA;Summary of sample sizes: 142, 142, 140, 142, 142, 141, ... &#xA;&#xA;Resampling results across tuning parameters:&#xA;&#xA;  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD&#xA;  1                  50       0.8       0.5    0.1          0.2     &#xA;  1                  100      0.8       0.6    0.1          0.2     &#xA;  1                  200      0.8       0.6    0.09         0.2     &#xA;  2                  50       0.8       0.6    0.1          0.2     &#xA;  2                  100      0.8       0.6    0.09         0.2     &#xA;  2                  200      0.8       0.6    0.1          0.2     &#xA;  3                  50       0.8       0.6    0.09         0.2     &#xA;  3                  100      0.8       0.6    0.09         0.2     &#xA;  3                  200      0.8       0.6    0.08         0.2     &#xA;&#xA;Tuning parameter 'shrinkage' was held constant at a value of 0.1&#xA;Accuracy was used to select the optimal model using  the largest value.&#xA;The final values used for the model were n.trees = 150, interaction.depth = 3     &#xA;and shrinkage = 0.1.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Caret offers many other options as well so should be able to suit your needs. &lt;/p&gt;&#xA;" OwnerUserId="802" LastActivityDate="2014-08-06T12:16:22.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="925" PostTypeId="2" ParentId="922" CreationDate="2014-08-06T12:27:31.277" Score="1" Body="&lt;p&gt;Both methods work. However, if you retain all words in documents you would essentially be working with high dimensional vectors (each term representing one dimension). Consequently, a classifier, e.g. SVM, would take more time to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is thus a standard practice to reduce the term-space dimensionality by pre-processing steps such as stop-word removal, stemming, Principal Component Analysis (PCA) etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach could be to analyze the document corpora by a topic modelling technique such as LDA and then retaining only those words which are representative of the topics, i.e. those which have high membership values in a single topic class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach (inspired by information retrieval) could be to retain the top K tf-idf terms from each document.&lt;/p&gt;&#xA;" OwnerUserId="984" LastEditorUserId="984" LastEditDate="2014-08-06T13:15:51.460" LastActivityDate="2014-08-06T13:15:51.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="926" PostTypeId="2" ParentId="917" CreationDate="2014-08-06T13:42:54.083" Score="7" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/918/2853&quot;&gt;Aleksandr's answer&lt;/a&gt; is completely correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the way the question is posed implies that this is a straightforward ordinary least squares regression question: minimizing the sum of squared residuals between a dependent variable and a linear combination of predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, while there may be many zeros in your design matrix, your system as such is not overly large: 300 observations on 40 predictors is no more than medium-sized. You can run such a regression using R without any special efforts for sparse data. Just use the &lt;code&gt;lm()&lt;/code&gt; command (for &quot;linear model&quot;). Use &lt;code&gt;?lm&lt;/code&gt; to see the help page. And note that &lt;code&gt;lm&lt;/code&gt; will by default silently add a constant column of ones to your design matrix (the intercept) - include a &lt;code&gt;-1&lt;/code&gt; on the right hand side of your formula to suppress this. Overall, assuming all your data (and nothing else) is in a &lt;code&gt;data.frame&lt;/code&gt; called &lt;code&gt;foo&lt;/code&gt;, you can do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- lm(y~.-1,data=foo)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then you can look at parameter estimates etc. like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(model)&#xA;residuals(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If&lt;/em&gt; your system is &lt;em&gt;much&lt;/em&gt; larger, say on the order of 10,000 observations and hundreds of predictors, looking at specialized sparse solvers as per &lt;a href=&quot;https://datascience.stackexchange.com/a/918/2853&quot;&gt;Aleksandr's answer&lt;/a&gt; may start to make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, in your comment to &lt;a href=&quot;https://datascience.stackexchange.com/a/918/2853&quot;&gt;Aleksandr's answer&lt;/a&gt;, you mention constraints on your equation. If that is actually your key issue, there are ways to calculate constrained least squares in R. I personally like &lt;code&gt;pcls()&lt;/code&gt; in the &lt;code&gt;mgcv&lt;/code&gt; package. Perhaps you want to edit your question to include the type of constraints (box constraints, nonnegativity constraints, integrality constraints, linear constraints, ...) you face?&lt;/p&gt;&#xA;" OwnerUserId="2853" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-08-06T13:42:54.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="927" PostTypeId="1" AcceptedAnswerId="929" CreationDate="2014-08-06T15:07:07.457" Score="2" ViewCount="2943" Body="&lt;p&gt;I'm working on the dataset with lots of NA values with sklearn and &lt;code&gt;pandas.DataFrame&lt;/code&gt;. I implemented different imputation strategies for different columns of the dataFrame based column names. For example NAs predictor &lt;code&gt;'var1'&lt;/code&gt; I impute with 0's and for &lt;code&gt;'var2'&lt;/code&gt; with mean.&lt;/p&gt;&#xA;&lt;p&gt;When I try to cross validate my model using &lt;code&gt;train_test_split&lt;/code&gt; it returns me a &lt;code&gt;nparray&lt;/code&gt; which does not have column names. How can I impute missing values in this nparray?&lt;/p&gt;&#xA;&lt;p&gt;P.S. I do not impute missing values in the original data set before splitting on purpose so I keep test and validation sets separately.&lt;/p&gt;&#xA;" OwnerUserId="2854" LastEditorUserId="98307" LastEditDate="2020-07-31T14:25:42.320" LastActivityDate="2020-07-31T14:25:42.320" Title="how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?" Tags="|pandas|cross-validation|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="928" PostTypeId="2" ParentId="919" CreationDate="2014-08-06T15:10:59.600" Score="0" Body="&lt;p&gt;K-Fold should do just fine for binary classification problem. Depending on the time it is taking to train your model and predict the outcome I would use 10-20 folds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However sometimes a single fold takes several minutes, in this case I use 3-5 folds but not less than 3. Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="2854" LastActivityDate="2014-08-06T15:10:59.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="929" PostTypeId="2" ParentId="927" CreationDate="2014-08-06T17:07:17.520" Score="1" Body="&lt;p&gt;Can you just cast your &lt;code&gt;np.array&lt;/code&gt; from &lt;code&gt;train_test_split&lt;/code&gt; back into a &lt;code&gt;pandas.DataFrame&lt;/code&gt; so you can carry out your same strategy. This is very common to what I do when dealing with pandas and scikit. For example,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; a = train_test_split&#xA; new_df = pd.DataFrame(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="802" LastEditorUserId="86339" LastEditDate="2020-07-31T14:03:12.133" LastActivityDate="2020-07-31T14:03:12.133" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="930" PostTypeId="2" ParentId="927" CreationDate="2014-08-06T18:17:29.700" Score="0" Body="&lt;p&gt;From the link you mentioned in the comment, the train and test sets should be in the form of a  dataframe if you followed the first explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, you could do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[variable] = df[variable].fillna(df[variable].median())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have options on what to fill the N/A values with, check out the link.&#xA;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/missing_data.html&quot; rel=&quot;nofollow&quot;&gt;http://pandas.pydata.org/pandas-docs/stable/missing_data.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you followed the second explanation, using sklearn's cross-validation, you could implement  mike1886's suggestion of transforming the arrays into dataframes and then use the fillna option.&lt;/p&gt;&#xA;" OwnerUserId="2838" LastActivityDate="2014-08-06T18:17:29.700" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="931" PostTypeId="2" ParentId="902" CreationDate="2014-08-06T23:37:15.293" Score="2" Body="&lt;p&gt;To answer a simpler, but related question, namely 'How well can my model generalize on the data that I have?' the method of learning curves might be applicable. &lt;a href=&quot;https://www.youtube.com/watch?v=g4XluwGYPaA&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is a lecture given by Andrew Ng about them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea is to plot test set error and training set error vs. the complexity of the model you are using (this can be somewhat complicated). If the model is powerful enough to fully 'understand' your data, at some point the complexity of the model will be high enough that performance on the training set will be close to perfect. However, the variance of a complex model will likely cause the test set performance to increase at some point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This analysis tells you two main things, I think. The first is an upper limit on performance. It's pretty unlikely that you'll do better on data that you haven't seen than on your training data. The other thing it tells you is whether or not getting more data might help. If you can demonstrate that you fully understand your training data by driving training error to zero it might be possible, through the inclusion of more data, to drive your test error further down by getting a more complete sample and then training a powerful model on that. &lt;/p&gt;&#xA;" OwnerUserId="2724" LastEditorUserId="2724" LastEditDate="2014-08-07T00:20:05.817" LastActivityDate="2014-08-07T00:20:05.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="932" PostTypeId="2" ParentId="896" CreationDate="2014-08-07T03:15:09.933" Score="4" Body="&lt;p&gt;I have managed to resolve this. There is an excellent and thorough explanation of the optimization steps in the following thesis: &lt;a href=&quot;http://cs.stanford.edu/~pliang/papers/meng-thesis.pdf&quot; rel=&quot;nofollow&quot;&gt;Semi-Supervised Learning for Natural Language by Percy Liang&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My mistake was trying to update the quality for all potential clusters pairs. Instead, you should initialize a table with the quality changes of doing each merge. Use this table to find the best merge, and the update the relevant terms that make up the table entries.&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2014-08-07T03:15:09.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="933" PostTypeId="2" ParentId="920" CreationDate="2014-08-07T09:33:50.817" Score="1" Body="&lt;p&gt;It dates back to 1998, so most likely has been abandoned, or &quot;acquired&quot; by Microsoft as the creator currently works there and has done since publishing that research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/ists97.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/ists97.pdf&lt;/a&gt;&#xA;and &lt;a href=&quot;http://research.microsoft.com/en-us/people/cyl&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://research.microsoft.com/en-us/people/cyl&lt;/a&gt; for the author. Maybe you could try to contact him.&lt;/p&gt;&#xA;" OwnerUserId="2861" LastEditorUserId="29575" LastEditDate="2018-09-07T03:25:01.003" LastActivityDate="2018-09-07T03:25:01.003" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="934" PostTypeId="1" CreationDate="2014-08-07T10:45:38.557" Score="4" ViewCount="331" Body="&lt;p&gt;I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem is that there are only 725 OLD instances in the train set and 95 in the test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only algorithm which succeeds for me to handle imbalance is NaiveBayes in Weka (0.02 precision for OLD class), others (trees) classify each instance as NEW.&#xA;What is the best approach to handle the imbalance and the appropriate algorithm in such a case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="2533" LastActivityDate="2015-12-20T06:02:38.193" Title="Handling huge dataset imbalance (2 class values) and appropriate ML algorithm" Tags="|machine-learning|dataset|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="935" PostTypeId="2" ParentId="934" CreationDate="2014-08-07T12:46:25.123" Score="5" Body="&lt;p&gt;I'm not allowed to comment, but I have more a suggestion: you could try to implement some &quot;Over-sampling Techniques&quot; like SMOTE:&#xA;&lt;a href=&quot;http://scholar.google.com/scholar?q=oversampling+minority+classes&quot;&gt;http://scholar.google.com/scholar?q=oversampling+minority+classes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2863" LastActivityDate="2014-08-07T12:46:25.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="936" PostTypeId="1" CreationDate="2014-08-07T15:12:48.617" Score="-1" ViewCount="165" Body="&lt;p&gt;I am attempting to compile code using Knitr in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code below is returning the following error, and causes errors in the rest of the document.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;miss&amp;lt;-sample$sensor_glucose[!is.na(sample$sensor_glucose)]&#xA;# Error: &quot;## Warning: is.na() applied to non-(list or vector) of type 'NULL'&quot;&#xA;&#xA;str(miss)&#xA;# int [1:103] 213 113 46 268 186 196 187 153 43 175 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to remedy this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="2792" LastEditorUserId="97" LastEditDate="2014-08-08T12:27:15.360" LastActivityDate="2014-08-08T12:27:15.360" Title="R error using Knitr" Tags="|r|error-handling|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="937" PostTypeId="1" AcceptedAnswerId="998" CreationDate="2014-08-07T15:33:43.793" Score="60" ViewCount="127273" Body="&lt;p&gt;I am working on a problem with too many features and training my models takes way too long. I implemented a forward selection algorithm to choose features.&lt;/p&gt;&#xA;&lt;p&gt;However, I was wondering does scikit-learn have a forward selection/stepwise regression algorithm?&lt;/p&gt;&#xA;" OwnerUserId="2854" LastEditorUserId="29169" LastEditDate="2021-08-15T02:07:30.567" LastActivityDate="2021-08-15T21:08:31.770" Title="Does scikit-learn have a forward selection/stepwise regression algorithm?" Tags="|feature-selection|scikit-learn|" AnswerCount="8" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="938" PostTypeId="2" ParentId="934" CreationDate="2014-08-07T17:49:05.640" Score="3" Body="&lt;p&gt;You can apply a clustering algorithm to the instances in the majority class and train a classifier with the centroids/medoids offered by the cluster algorithm. This is subsampling the majority class, the converse of oversampling the minority class. &lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-08-07T17:49:05.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="939" PostTypeId="2" ParentId="919" CreationDate="2014-08-07T17:53:06.970" Score="5" Body="&lt;p&gt;You will have best results if you care to build the folds so that each variable (and most importantly the target variable) is approximately identically distributed in each fold. This is called, when applied to the target variable, stratified k-fold. One approach is to cluster the inputs and make sure each fold contains the same number of instances from each cluster proportional to their size.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2014-08-07T17:53:06.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="940" PostTypeId="1" CreationDate="2014-08-07T22:30:52.913" Score="4" ViewCount="1771" Body="&lt;p&gt;We are storing the information about our users showing interest in our items. Based on this information, we would like to create a simple recommendation engine that will take the items I1, I2, I3 etc of the current user, search for all other users that had shown interest in those items, and then output the items I4, I5, I6 etc of the other users, sorted by their decreasing popularity. So, basically, the standard &quot;other buyer were also interested in...&quot; functionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking myself what kind of a database is suitable for a realtime recommendation engine like this. My current idea is to build a trie of item IDs, then sort the item IDs of the current user (as the order of items is irrelevant) and to go down the trie; the children of the last trie node will build the needed output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that we have 2 million items so that according to our estimation the trie will have at least 1E12 nodes, so that we probably need a distributed sharded database to store it. Before we reinvent the wheel, are there any ready-to-use databases or generally, non-cloud solutions for recommendation engines out there?&lt;/p&gt;&#xA;" OwnerUserId="2873" LastActivityDate="2014-08-11T16:05:02.197" Title="Database for a trie, or other appropriate structure for recommendation engine" Tags="|bigdata|recommender-system|databases|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="941" PostTypeId="2" ParentId="936" CreationDate="2014-08-08T00:15:05.527" Score="2" Body="&lt;p&gt;I agree with @ssdecontrol that a minimal &lt;em&gt;reproducible example&lt;/em&gt; would be the most helpful. However, looking at your code (pay attention to the sequence &lt;code&gt;Error: ... Warning: ...&lt;/code&gt;), I believe that the issue you are experiencing is due to an inappropriate setting of R's global &lt;code&gt;warn&lt;/code&gt; &lt;strong&gt;option&lt;/strong&gt;. It appears that your current setting is likely &lt;code&gt;2&lt;/code&gt;, which refers to &lt;strong&gt;converting warnings to errors&lt;/strong&gt;, whereas, you, most likely want the setting &lt;code&gt;1&lt;/code&gt;, which is to treat warnings as such, without converting them to errors. If that is the case, you just need to set the option appropriately:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;options(warn=1)  # print warnings as they occur&#xA;options(warn=2)  # treat warnings as errors&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note for moderators/administrators&lt;/strong&gt;: This question seems not to be a &lt;em&gt;data science&lt;/em&gt; question, but purely an &lt;em&gt;R&lt;/em&gt; question. Therefore, I think it should be moved to &lt;em&gt;StackOverflow&lt;/em&gt;, where it belongs.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2014-08-08T03:03:11.517" LastActivityDate="2014-08-08T03:03:11.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="942" PostTypeId="2" ParentId="915" CreationDate="2014-08-08T00:25:04.210" Score="5" Body="&lt;p&gt;You mentioned regression and PCA in the title, and there is a definite answer for each of those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The asymptotic complexity of linear regression reduces to O(P^2 * N) if N &gt; P, where P is the number of features and N is the number of observations. More detail in &lt;a href=&quot;https://math.stackexchange.com/a/84503/117452&quot;&gt;Computational complexity of least square regression operation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vanilla PCA is O(P^2 * N + P ^ 3), as in &lt;a href=&quot;https://scicomp.stackexchange.com/q/3220&quot;&gt;Fastest PCA algorithm for high-dimensional data&lt;/a&gt;. However fast algorithms exist for very large matrices, explained in that answer and &lt;a href=&quot;https://stats.stackexchange.com/q/2806/36229&quot;&gt;Best PCA Algorithm For Huge Number of Features?&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I don't think anyone's compiled a single lit review or reference or  book on the subject. Might not be a bad project for my free time...&lt;/p&gt;&#xA;" OwnerUserId="1156" LastEditorUserId="-1" LastEditDate="2017-04-13T12:53:55.957" LastActivityDate="2014-08-08T00:25:04.210" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="944" PostTypeId="2" ParentId="934" CreationDate="2014-08-08T07:12:52.680" Score="2" Body="&lt;p&gt;In addition to undersampling the majority class (i.e. taking only a few NEW), you may consider oversampling the minority class (in essence, duplicating your OLDs, but there are other smarter ways to do that)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that oversampling may lead to overfitting, so pay special attention to testing your classifiers &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check also this answer on CV: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/108325/49130&quot;&gt;https://stats.stackexchange.com/a/108325/49130&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-08-08T07:12:52.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="945" PostTypeId="1" AcceptedAnswerId="947" CreationDate="2014-08-08T10:01:25.400" Score="5" ViewCount="2157" Body="&lt;p&gt;We have a classification algorithm to categorize Java exceptions in Production.&#xA;This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it determines what kind of exception is (development, availability, configuration, etc.) and the responsible component (the most inner component responsible of the exception). In Java an exception can have several causing exceptions, and the whole must be analyzed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, given the following example exception:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;com.myapp.CustomException: Error printing ...&#xA;... (stack)&#xA;Caused by: com.foo.webservice.RemoteException: Unable to communicate ...&#xA;... (stack)&#xA;Caused by: com.acme.PrintException: PrintServer002: Timeout ....&#xA;... (stack)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First of all, our algorithm splits the whole stack in three isolated exceptions. Afterwards it starts analyzing these exceptions starting from the most inner one. In this case, it determines that this exception (the second caused by) is of type &lt;code&gt;Availability&lt;/code&gt; and that the responsible component is a &quot;print server&quot;. This is because there is a rule that matches containing the word &lt;code&gt;Timeout&lt;/code&gt; associated to the &lt;code&gt;Availability&lt;/code&gt; type. There is also a rule that matches &lt;code&gt;com.acme.PrintException&lt;/code&gt; and determines that the responsible component is a print server. As all the information needed is determined using only the most inner exception, the upper exceptions are ignored, but this is not always the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see this kind of approximation is very complex (and chaotic) as a human have to create new rules as new exceptions appear. Besides, the new rules have to be compatible with the current ones because a new rule for classifying a new exception must not change the classification of any of the already classified exceptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are thinking about using Machine Learning to automate this process. Obviously, I am not asking for a solution here as I know the complexity but I'd really appreciate some advice to achieve our goal.&lt;/p&gt;&#xA;" OwnerUserId="2878" LastActivityDate="2022-07-08T14:46:48.740" Title="Classifying Java exceptions" Tags="|machine-learning|classification|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="946" PostTypeId="1" CreationDate="2014-08-08T13:44:52.803" Score="2" ViewCount="4281" Body="&lt;p&gt;Cross posting this from Cross Validated:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What's the most appropriate statistical test to apply to a small A/B test?&lt;/li&gt;&#xA;&lt;li&gt;What's the R code and interpretation to analyze a small A/B test?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm running a small test to figure out which ads perform better. I have the following results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Position 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;variation,impressions,clicks&#xA;row-1,753,26&#xA;row-3,767 7&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Position 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;variation,impressions,clicks&#xA;row-1,753,16&#xA;row-3,767 13&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Position 3:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;variation,impressions,clicks&#xA;row-1,753,2&#xA;row-3,767 7&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's safe to say these numbers are small and likely to be not normally distributed. Also, it's click data so there's a binary outcome of clicked or not and the trials are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Appropriate test&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In analyzing each position for significance, I think comparison with a binomial or Poisson distribution makes the most sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://www.openintro.org/stat/textbook.php?stat_book=os&quot; rel=&quot;nofollow&quot;&gt;the OpenIntro Stats&lt;/a&gt; (and other sources) book, a variable follows a Poisson distribution &quot;... if the event being considered is rare, the population is large, and the events occur independently of each other.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same source classifies a binomial variable approximately the same way adding that the probability of success is the same and the number of trials is fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate this is not an either/or decision and analysis can be done using both distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given A/B (split) testing is a science that has been practiced for several years, I imagine that there is a canonical test. However, looking around the internet, I mostly come across analysis that uses the standard normal distribution. That just seems wrong :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a canonical test to use for A/B tests with small #'s of clicks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Interpretation and R code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used the following R code to test significance for each position:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Position 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;binom.test(7, 767, p=(26/753))&#xA;&#xA;Exact binomial test&#xA;&#xA;data:  7 and 767&#xA;number of successes = 7, number of trials = 767, p-value = 1.077e-05&#xA;alternative hypothesis: true probability of success is not equal to 0.03452855&#xA;95 percent confidence interval:&#xA; 0.003676962 0.018713125&#xA;sample estimates:&#xA;probability of success &#xA;           0.009126467&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I interpret this result to mean: The probability of success in the test group is indeed different than the control group with a 95% confidence interval that the success probability is between .368% and 1.87%&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ppois(((26-1)/753), lambda=(7/767), lower.tail = F)&#xA;[1] 0.009084947&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I interpret this result to mean: Given a Poisson distribution with a click rate of 7 per 767 trials, there is a 0.9% chance of having a click rate of 26 or more per 753 trials in the same distribution. Contextualized in the ad example,&#xA;there is a .1% chance that the control ad actually performs the same as the test ad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the above interpretation correct? Does the test and interpretation change with the different positions (i.e. are the results of the Poisson test more appropriate for Position 3 given the small numbers)?&lt;/p&gt;&#xA;" OwnerUserId="2883" LastEditorUserId="11097" LastEditDate="2016-01-06T11:48:04.607" LastActivityDate="2016-01-06T11:48:04.607" Title="Analysis of Split (A/B) tests using Poisson and/or Binomial Distribution" Tags="|r|ab-test|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="947" PostTypeId="2" ParentId="945" CreationDate="2014-08-08T13:53:36.847" Score="8" Body="&lt;p&gt;First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any supervised ML algorithm consists of at least 2 components: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dataset to train and test on.&lt;/li&gt;&#xA;&lt;li&gt;Algorithm(s) to handle these data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Training dataset consists of a set of pairs &lt;code&gt;(x, y)&lt;/code&gt;, where &lt;code&gt;x&lt;/code&gt; is a &lt;strong&gt;vector of features&lt;/strong&gt; and &lt;code&gt;y&lt;/code&gt; is &lt;strong&gt;predicted variable&lt;/strong&gt;. Predicted variable is just what you want to know, i.e. in your case it is exception type. Features are more tricky. You cannot just throw raw text into an algorithm, you need to extract meaningful parts of it and organize them as feature vectors first. You've already mentioned a couple of useful features - exception class name (e.g. &lt;code&gt;com.acme.PrintException&lt;/code&gt;) and contained words (&quot;Timeout&quot;). All you need is to translate your row exceptions (and human-categorized exception types) into suitable dataset, e.g.: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ex_class                  contains_timeout  ...   | ex_type&#xA;-----------------------------------------------------------&#xA;[com.acme.PrintException, 1                , ...] | Availability&#xA;[java.lang.Exception    , 0                , ...] | Network&#xA; ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This representation is already much better for ML algorithms. But which one to take? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking into account nature of the task and your current approach natural choice is to use &lt;strong&gt;decision trees&lt;/strong&gt;. This class of algorithms will compute optimal decision criteria for all your exception types and print out resulting tree. This is especially useful, because you will have possibility to manually inspect how decision is made and see how much it corresponds to your manually-crafted rules. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's, however, possibility that some exceptions with exactly the same features will belong to different exception types. In this case probabilistic approach may work well. Despite its name, &lt;strong&gt;Naive Bayes&lt;/strong&gt; classifier works pretty well in most cases. There's one issue with NB and our dataset representation, though: dataset contains &lt;em&gt;categorical&lt;/em&gt; variables, and Naive Bayes can work with &lt;em&gt;numerical&lt;/em&gt; attributes only*. Standard way to overcome this problem is to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Dummy_variable_%28statistics%29&quot;&gt;dummy variables&lt;/a&gt;. In short, dummy variables are binary variables that simply indicate whether specific category presents or not. For example, single variable &lt;code&gt;ex_class&lt;/code&gt; with values &lt;code&gt;{com.acme.PrintException, java.lang.Exception, ...}&lt;/code&gt;, etc. may be split into several variables &lt;code&gt;ex_class_printexception&lt;/code&gt;, &lt;code&gt;ex_class_exception&lt;/code&gt;, etc. with values &lt;code&gt;{0, 1}&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ex_class_printexception  ex_class_exception  contains_timeout | ex_type&#xA;-----------------------------------------------------------------------&#xA;[1,                    , 0                 , 1              ] | Availability&#xA;[0,                    , 1                 , 0              ] | Network&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One last algorithm to try is &lt;strong&gt;Support Vector Machines (SVM)&lt;/strong&gt;. It neither provides helpful visualisation, nor is probabilistic, but often gives superior results. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;*&lt;/em&gt; - in fact, neither Bayes theorem, nor Naive Bayes itself state anything about variable type, but most software packages that come to mind rely on numerical features. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-08-08T13:53:36.847" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="948" PostTypeId="1" CreationDate="2014-08-08T16:49:13.850" Score="1" ViewCount="745" Body="&lt;p&gt;I have found a number of libraries and tools for data science in Scala, I would like to know about which one has more adoption and which one is gaining adoption at a faster pace and to what extent this is the case. Basically, which one should I bet for (if any at this point).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the tools I've found are (in no particular order):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scalding&lt;/li&gt;&#xA;&lt;li&gt;Breeze&lt;/li&gt;&#xA;&lt;li&gt;Spark&lt;/li&gt;&#xA;&lt;li&gt;Saddle&lt;/li&gt;&#xA;&lt;li&gt;H2O&lt;/li&gt;&#xA;&lt;li&gt;Spire&lt;/li&gt;&#xA;&lt;li&gt;Mahout&lt;/li&gt;&#xA;&lt;li&gt;Hadoop&lt;/li&gt;&#xA;&lt;li&gt;MongoDB&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I need to be more specific to make the question answerable: I'm not particularly interested in clusters and Big Data at this moment, but I'm interested in sizable data (up to 100 GB) for information integration and predictive analytics.&lt;/p&gt;&#xA;" OwnerUserId="1281" LastEditorUserId="1281" LastEditDate="2014-08-08T17:00:50.270" LastActivityDate="2014-08-09T21:25:58.250" Title="Any clear winner for Data Science in Scala?" Tags="|tools|" AnswerCount="1" CommentCount="7" ClosedDate="2014-08-09T01:26:26.703" ContentLicense="CC BY-SA 3.0" />
  <row Id="949" PostTypeId="1" CreationDate="2014-08-08T17:32:48.377" Score="2" ViewCount="840" Body="&lt;p&gt;I am facing this bizarre issue while using &lt;code&gt;Apache Pig&lt;/code&gt; &lt;strong&gt;rank&lt;/strong&gt; utility. I am executing the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;email_id_ranked = rank email_id;&#xA;store email_id_ranked into '/tmp/';&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, basically I am trying to get the following result&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1,email1&#xA;2,email2&#xA;3,email3&#xA;... &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Issue is sometime pig dumps the above result but sometimes it dumps only the emails without the rank. Also when I dump the data on screen using &lt;code&gt;dump&lt;/code&gt; function pig returns both the columns. I don't know where the issue is. Kindly advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if you need any more information. Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pig version: Apache Pig version 0.11.0-cdh4.6.0&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2433" LastActivityDate="2014-08-08T17:32:48.377" Title="Pig Rank function not generating rank in output" Tags="|bigdata|apache-hadoop|apache-pig|" AnswerCount="0" CommentCount="0" ClosedDate="2020-10-19T00:54:34.543" ContentLicense="CC BY-SA 3.0" />
  <row Id="950" PostTypeId="2" ParentId="946" CreationDate="2014-08-08T19:23:33.637" Score="0" Body="&lt;p&gt;The approximation Binomial(k,n,p) ~= Poisson(k,s) (where s = n*p) can be shown under the assumptions: &lt;br/&gt;&#xA;1) n &gt;&gt; k (to say that n!/(n-k)! ~= n^k), &lt;br/&gt;&#xA;2) p &amp;lt;&amp;lt;1 (to say that (1-p)^(n-k) ~= (1-p)^n). &lt;br/&gt;&#xA;It's up to you whether those are sufficiently satisfied. If the exact calculation can be done quickly, in my opinion, it's nice to stay with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also since, if the probability of row 3 sample is different from the row 1 sample, it would almost certainly be on the lower side. It would probably best for you to use &lt;br/&gt;&#xA;binom.test(7, 767, p=(26/753), alternative='less')&#xA;the final option indicating that the alternative to your null hypothesis is that the probability is less than 26/753, not equal to. Of course, that's simply just the sum of Binomial probabilities from 0 to 7 (you can check yourself), the interpretation being that &lt;em&gt;this&lt;/em&gt; is the probability of having gotten at most 7 rolls from random chance, if the probability truly was 26/753.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind the interpretation of that last sentence. These kinds of tests are generally used when we &lt;em&gt;know&lt;/em&gt; what the inherent probability is that we're comparing to (e.g. to see if the set of coin flips has a probability significantly different from 1/2 which is what we expect from a fair coin). In this case, we don't &lt;em&gt;know&lt;/em&gt; what the probability is that we're comparing to, we're just making the very crude guess that the 26/753 outcome of row 1 reflects the true probability. It's better than a regular Normal t-test in this case, but don't put &lt;em&gt;too&lt;/em&gt; much stock in it unless you have a much higher sample size for row 1.&lt;/p&gt;&#xA;" OwnerUserId="2841" LastActivityDate="2014-08-08T19:23:33.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="951" PostTypeId="2" ParentId="948" CreationDate="2014-08-08T20:46:58.493" Score="5" Body="&lt;p&gt;Not sure anybody have worked with &lt;em&gt;all&lt;/em&gt; these tools, so I'm going to share my experience with some of them and let others share their experience with the others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MongoDB&lt;/strong&gt; addresses problems that involve heterogeneous and nested objects, while data mining mostly works with simple tabular data. MongoDB is neither fast with this type of data, nor provide any advanced tools for analysis (correct me if you know any). So I can think of a very few applications for Mongo in data mining. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hadoop&lt;/strong&gt; is a large ecosystem, containing dozens of different tools. I will assume that you mean core Hadoop features - HDFS and MapReduce. HDFS provides flexible way to store large amounts of data, while MapReduce gives basis for processing them. It has its clear advantages for processing multi-terabyte datasets, but it also has significant drawbacks. In particular, because of intensive disk IO during MapReduce tasks (that slows down computations a lot) it is terrible for interactive development, iterative algorithms and working with not-that-big datasets. For more details see my &lt;a href=&quot;https://datascience.stackexchange.com/a/863/1279&quot;&gt;earlier answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many algorithms in Hadoop require multiple MapReduce jobs with complicated data flow. This is where &lt;strong&gt;Scalding&lt;/strong&gt; gets shiny. Scalding (and underlying Java's &lt;a href=&quot;http://www.cascading.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cascading&lt;/a&gt;) provides much simpler API, but at the moment uses same MapReduce as its runtime and thus holds all the same issues.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Spark&lt;/strong&gt; addresses exactly these issues. It drops Hadoop's MapReduce and offers completely new computational framework based on distributed in-memory collections and delayed evaluations. Its API is somewhat similar to Scalding's with all MR complexity removed, so it's really easy to get started with it. Spark is also the first in this list that comes with data mining library - &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-guide.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLlib&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But Spark doesn't reinvent things like basic linear algebra. For this purpose it uses &lt;strong&gt;Breeze&lt;/strong&gt;. To my opinion, Breeze is far in quality from scientific packages like SciPy, Octave or Julia, but it is still good enough for most practical use cases. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mahout&lt;/strong&gt; relies on Hadoop's MapReduce and thus is terrible for iterative algorithms. &lt;strong&gt;Spire&lt;/strong&gt; and &lt;strong&gt;Saddle&lt;/strong&gt; look cute and probably have their niche, but seem to be much less well-known than Breeze. I couldn't find much information about &lt;strong&gt;H2O&lt;/strong&gt;, so it doesn't look like a big player here (comments from people who used it are welcome).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Some quick summary. Spark seems to be the most simple, flexible and fast-growing project for large-scale data processing. It facilitates a number of new projects (e.g. Shark or Spark SQL) and penetrates into existing (including &lt;a href=&quot;http://www.concurrentinc.com/2014/05/cascading-now-supports-tez-spark-and-storm-up-next/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cascading&lt;/a&gt; and &lt;a href=&quot;https://mahout.apache.org/users/sparkbindings/home.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mahout&lt;/a&gt;). Spark knows how to utilize HDFS API, and thus scales to terabytes of data easily. And for data mining downshifters who don't want to bother with a cluster setup there's always pure Breeze. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-08-09T21:25:58.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="954" PostTypeId="1" CreationDate="2014-08-11T07:59:22.780" Score="4" ViewCount="680" Body="&lt;p&gt;I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know OpenNLP coreference resolution does not support the German language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which components/data do I need to adapt the code such that it is possible to perform coreference resolution for German texts?&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2015-04-04T18:33:40.263" Title="OpenNLP Coreference Resolution (German)" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="955" PostTypeId="1" AcceptedAnswerId="988" CreationDate="2014-08-11T12:25:47.700" Score="8" ViewCount="853" Body="&lt;p&gt;Does anyone know a libarary for performing coreference resolution on German texts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only tool that I know is &lt;a href=&quot;http://www.cl.uzh.ch/static/news.php?om=view&amp;amp;nid=163&quot; rel=&quot;nofollow noreferrer&quot;&gt;CorZu&lt;/a&gt; which is a python library.&lt;/p&gt;&#xA;" OwnerUserId="979" LastEditorUserId="29169" LastEditDate="2019-06-07T17:10:38.600" LastActivityDate="2019-06-07T17:10:38.600" Title="Coreference Resolution for German Texts" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="956" PostTypeId="2" ParentId="904" CreationDate="2014-08-11T12:48:52.320" Score="1" Body="&lt;p&gt;I had to create a web based dashboard. My main charting tool was d3js. But I needed to use ggplot2 to generate few charts. Through d3js's ggplot2 extension, I could create the same. If your charts can be generated through existing extension, then web has better alternatives. Later you can export them to PDF for distribution. &lt;/p&gt;&#xA;" OwnerUserId="2910" LastActivityDate="2014-08-11T12:48:52.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="957" PostTypeId="1" CreationDate="2014-08-11T15:19:02.047" Score="2" ViewCount="563" Body="&lt;p&gt;I know that ARIMA can't detect multiple seasonality, but it is possible to &lt;a href=&quot;http://robjhyndman.com/hyndsight/dailydata/&quot; rel=&quot;nofollow&quot;&gt;use fourier functions to add a second seasonality&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to forecast gas consumption composed by a daily, weekly (week days-weekend), yearly seasonality. Does it make sense to apply three times the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/stl.html&quot; rel=&quot;nofollow&quot;&gt;STL decomposition by LOESS&lt;/a&gt;? The reason is that I applied the fourier method and I have bad results but I don't know if it is only because I applied it wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in the theoretical explanation, but here you find also the code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ARIMA + 2 STL:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b &amp;lt;- ts(drop(coredata(dat.ts)), deltat=1/12/30/24, start=1)&#xA;fit &amp;lt;- stl(b, s.window=&quot;periodic&quot;)&#xA;b &amp;lt;- seasadj(fit)&#xA;dat.ts &amp;lt;- xts(b, index(dat.ts))&#xA;&#xA;# The weekdays are extracted&#xA;dat.weekdays &amp;lt;- dat.ts[.indexwday(dat.ts) %in% 1:5]&#xA;dat.weekdaysTS &amp;lt;- ts(drop(coredata(dat.weekdays)), frequency=24, start=1)&#xA;fit &amp;lt;- stl(dat.weekdaysTS, s.window=&quot;periodic&quot;)&#xA;dat.weekdaysTS &amp;lt;- seasadj(fit)&#xA;&#xA;arima &amp;lt;- Arima(dat.weekdaysTS, order=c(3,0,5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With fourier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat.weekdays &amp;lt;- dat.ts[.indexwday(dat.ts) %in% 1:5]&#xA;dat.weekdaysTS &amp;lt;- ts(drop(coredata(dat.weekdays)), frequency=24, start=1)&#xA;z &amp;lt;- fourier(ts(dat.weekdaysTS, frequency=365.25), K=5)&#xA;arima &amp;lt;- Arima(dat.weekdaysTS, order=c(3,0,5),xreg=z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="989" LastEditorUserId="1330" LastEditDate="2022-04-09T15:57:32.523" LastActivityDate="2022-04-09T15:57:32.523" Title="Multiple seasonality with ARIMA?" Tags="|time-series|arima|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="958" PostTypeId="2" ParentId="940" CreationDate="2014-08-11T16:05:02.197" Score="1" Body="&lt;p&gt;Have a look at &lt;a href=&quot;https://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Apache Mahout&lt;/a&gt;. Last version features also user-item-based recommenders.&lt;/p&gt;&#xA;" OwnerUserId="2912" LastActivityDate="2014-08-11T16:05:02.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="959" PostTypeId="2" ParentId="866" CreationDate="2014-08-11T18:33:37.023" Score="7" Body="&lt;p&gt;While I am not a data scientist, I am an epidemiologist working in a clinical setting. Your research question did not specify a time period (ie odds of developing CKD in 1 year, 10 years, lifetime?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, I would go through a number of steps before even thinking about modeling (univariate analysis, bivariate analysis, colinearity checks, etc). However, the most commonly used method for trying to predict a binary event (using continuous OR binary variables) is logistic regression. If you wanted to look at CKD as a lab value (urine albumin, eGFR) you would use linear regression (continuous outcome).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the methods used should be informed by your data and questions, clinicians are used to seeing odds ratios and risk ratios as these the most commonly reported measures of association in medical journals such as NEJM and JAMA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are working on this problem from a human health perspective (as opposed to Business Intelligence) this Steyerberg's &lt;a href=&quot;http://www.springer.com/medicine/internal/book/978-0-387-77243-1&quot;&gt;Clinical Prediction Models&lt;/a&gt; is an excellent resource. &lt;/p&gt;&#xA;" OwnerUserId="2913" LastActivityDate="2014-08-11T18:33:37.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="960" PostTypeId="2" ParentId="919" CreationDate="2014-08-11T19:21:50.137" Score="2" Body="&lt;p&gt;I have to agree that k-fold should do &quot;just&quot; fine. However, there is a nice article about the &quot;Bootstrap .632+&quot; method (basically a smoothened cross validation) that is supposed to be superior (however, they did the comparisons on not-binary data as far as I can tell)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you want to check out this article here: &lt;a href=&quot;http://www.jstor.org/stable/2965703&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/2965703&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user2556" LastActivityDate="2014-08-11T19:21:50.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="961" PostTypeId="1" CreationDate="2014-08-11T21:13:36.230" Score="8" ViewCount="974" Body="&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/lBxL5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are 4 different weight matrices that I got after training a &lt;strong&gt;restricted Boltzman machine (RBM)&lt;/strong&gt; with ~4k visible units and only 96 hidden units/weight vectors. As you can see, weights are extremely similar - even black pixels on the face are reproduced. The other 92 vectors are very similar too, though &lt;em&gt;none&lt;/em&gt; of weights are &lt;em&gt;exactly&lt;/em&gt; the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can overcome this by increasing number of weight vectors to 512 or more. But I encountered this problem several times earlier with different RBM types (binary, Gaussian, even convolutional), different number of hidden units (including pretty large), different hyper-parameters, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: what is the &lt;strong&gt;most likely reason&lt;/strong&gt; for weights to get &lt;strong&gt;very similar values&lt;/strong&gt;? Do they all just get to some local minimum? Or is it a sign of overfitting? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I currently use a kind of Gaussian-Bernoulli RBM, code may be found &lt;a href=&quot;https://github.com/faithlessfriend/Milk.jl/blob/master/src/rbm.jl&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPD.&lt;/strong&gt; My dataset is based on &lt;a href=&quot;http://www.pitt.edu/~emotion/ck-spread.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;CK+&lt;/a&gt;, which contains &gt; 10k images of 327 individuals. However I do pretty heavy preprocessing. First, I clip only pixels inside of outer contour of a face. Second, I transform each face (using piecewise affine wrapping) to the same grid (e.g. eyebrows, nose, lips etc. are in the same (x,y) position on all images). After preprocessing images look like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/FQMSp.png&quot; alt=&quot;enter image description here&quot;&gt; &lt;img src=&quot;https://i.stack.imgur.com/Hjcaw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When training RBM, I take only non-zero pixels, so outer black region is ignored. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="9420" LastEditDate="2016-07-17T04:55:26.173" LastActivityDate="2019-05-10T05:02:55.607" Title="Why a restricted Boltzman machine (RBM) tends to learn very similar weights?" Tags="|rbm|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="962" PostTypeId="1" AcceptedAnswerId="966" CreationDate="2014-08-12T03:50:52.303" Score="30" ViewCount="24180" Body="&lt;p&gt;I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic modeling over documents to identify a topic? Can I use classification methods to classify the text inside these documents?  &lt;/p&gt;&#xA;" OwnerUserId="2916" LastEditorUserId="29169" LastEditDate="2021-02-09T01:33:51.537" LastActivityDate="2021-02-09T01:33:51.537" Title="What is difference between text classification and topic models?" Tags="|classification|text-mining|topic-model|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="963" PostTypeId="1" CreationDate="2014-08-12T07:57:03.283" Score="14" ViewCount="5441" Body="&lt;p&gt;For experimenting we'd like to use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Emoji&quot;&gt;Emoji&lt;/a&gt; embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, there are 722 Emoji in Unicode 6.0, and probably another 250 will be added in Unicode 7.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a database (like e.g. SentiWordNet) that contains sentiment annotations for them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that SentiWordNet does allow for &lt;em&gt;ambiguous&lt;/em&gt; meanings, too. Consider e.g. &lt;a href=&quot;http://sentiwordnet.isti.cnr.it/search.php?q=funny&quot;&gt;funny&lt;/a&gt;, which is not just positive: &quot;this tastes funny&quot; is probably not positive... same will hold for &lt;code&gt;;-)&lt;/code&gt; for example. But I don't think this is harder for Emoji than it is for regular words...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you have experience with using them for sentiment analysis, I'd be interested to hear.&lt;/p&gt;&#xA;" OwnerUserId="2920" LastEditorUserId="2920" LastEditDate="2014-08-12T16:06:24.710" LastActivityDate="2020-08-06T11:03:50.720" Title="Sentiment data for Emoji" Tags="|machine-learning|classification|parsing|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="964" PostTypeId="2" ParentId="962" CreationDate="2014-08-12T07:59:13.060" Score="4" Body="&lt;p&gt;Topic models are usually &lt;strong&gt;unsupervised&lt;/strong&gt;. There are &quot;supervised topic models&quot;, too; but even then they try to model &lt;strong&gt;topics within a classes&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. you may have a class &quot;football&quot;, but there may be topics inside this class that relate to particular matches or teams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge with topics is that they change over time; consider the matches example above. Such topics may emerge, and disappear again.&lt;/p&gt;&#xA;" OwnerUserId="2920" LastActivityDate="2014-08-12T07:59:13.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="965" PostTypeId="1" CreationDate="2014-08-12T08:54:34.903" Score="1" ViewCount="74" Body="&lt;p&gt;I'm looking for a product that allows us to take in a collection of datastreams, and then after some event, will find any data that changes or correlates with that event. (For example, having a headache, and identifying that I drank too much beer last night and didn't drink enough water)&lt;/p&gt;&#xA;" OwnerUserId="2923" LastActivityDate="2014-08-12T08:54:34.903" Title="Tool for finding correlations between data after some event" Tags="|correlation|" AnswerCount="0" CommentCount="1" ClosedDate="2020-10-19T04:59:12.237" ContentLicense="CC BY-SA 3.0" />
  <row Id="966" PostTypeId="2" ParentId="962" CreationDate="2014-08-12T09:52:01.153" Score="35" Body="&lt;p&gt;&lt;strong&gt;Text Classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their words. Then I give you new documents and ask what you think the label for each one should be. The labels have meaning to me, not to you necessarily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Topic Modeling&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I give you a bunch of documents, without labels. I ask you to explain why the documents have the words they do by identifying some topics that each is &quot;about&quot;. You tell me the topics, by telling me how much of each is in each document, and I decide what the topics &quot;mean&quot; if anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd have to clarify what you me by &quot;identify one topic&quot; or &quot;classify the text&quot;.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-08-12T09:52:01.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="967" PostTypeId="2" ParentId="963" CreationDate="2014-08-12T12:40:44.907" Score="4" Body="&lt;p&gt;Total of 972 emoji is not really that big not to be able to label them manually, but I doubt that they will work as a good ground truth. Sources like Twitter are full of irony, sarcasm and other tricky settings where emotional symbols (such as emoji or emoticon) mean something different from normal interpretation. For example, someone may write &quot;xxx cheated their clients, and now they are cheated themselves! ha ha ha! :D&quot;. This is definitely negative comment, but author is glad to see xxx company in trouble and thus adds positive emoticon. These cases are not that frequent, but definitely not suitable for ground truth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much more common approach is to use emoticon as a &lt;strong&gt;seed for collecting actual data set&lt;/strong&gt;. For example, in &lt;a href=&quot;http://www.saifmohammad.com/WebDocs/NRC-Sentiment-JAIR-2014.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; authors use emoticon and emotional hash tags to grab lexicon of words useful for further classification. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-08-12T12:40:44.907" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="968" PostTypeId="1" CreationDate="2014-08-12T17:11:45.447" Score="4" ViewCount="1027" Body="&lt;p&gt;I am using the &lt;code&gt;stepAIC&lt;/code&gt; function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function means. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Df     Sum of Sq    RSS       AIC&#xA;&amp;lt;none&amp;gt;                        350.71   -5406.0&#xA;- aaa      1     0.283        350.99   -5405.9&#xA;- bbb      1     0.339        351.05   -5405.4&#xA;- ccc      1     0.982        351.69   -5400.5&#xA;- ddd      1     0.989        351.70   -5400.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Question Are the values listed under &lt;code&gt;Df&lt;/code&gt;, &lt;code&gt;Sum of Sq&lt;/code&gt;, &lt;code&gt;RSS&lt;/code&gt;, and &lt;code&gt;AIC&lt;/code&gt; the values for a model where only one variable would be considered as the independent variable (i.e. y ~&#xA;aaa, y ~ bbb, etc.)? &lt;/p&gt;&#xA;" OwnerUserId="2928" LastEditorUserId="2928" LastEditDate="2014-08-12T18:36:08.880" LastActivityDate="2022-05-31T16:47:57.803" Title="Understanding output stepAIC" Tags="|data-mining|r|feature-selection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="969" PostTypeId="2" ParentId="968" CreationDate="2014-08-13T14:18:58.917" Score="1" Body="&lt;p&gt;After consulting with someone I found out that the &amp;lt;none&amp;gt; corresponds to a model that would include all the variables, in other words none of the variables were removed. So consider the line in the output for the variable aaa. The listed RSS and AIC are the values for a model that would include all variables but aaa and we see an increase in the RSS and AIC. The other listed results can be considered in the same fashion. The best model is then the one where none are removed since this has the smallest AIC.&lt;/p&gt;&#xA;" OwnerUserId="2928" LastEditorUserId="25180" LastEditDate="2022-05-31T16:47:57.803" LastActivityDate="2022-05-31T16:47:57.803" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="972" PostTypeId="1" CreationDate="2014-08-13T20:30:18.143" Score="1" ViewCount="44" Body="&lt;p&gt;While finding frequent subgraphs in single large graph, subgraph isomorphism (test) is not considered because its not anti-monotone. How and why subgraph isomorphism is not anti-monotone ?&lt;/p&gt;&#xA;" OwnerUserId="2948" LastActivityDate="2014-08-13T20:30:18.143" Title="Subgraph isomorphism and Anti-monotone property" Tags="|bigdata|data-mining|dataset|graphs|similarity|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="973" PostTypeId="1" CreationDate="2014-08-13T20:53:15.443" Score="3" ViewCount="241" Body="&lt;p&gt;I like to find the weight vector for input-space features in a &lt;a href=&quot;http://en.wikipedia.org/wiki/Structured_support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;structured SVM&lt;/a&gt;. The idea is to identify the most important set of input-space features (based on the magnitude of their corresponding weights). I know that in a binary SVM the weight vector can be written as a &lt;a href=&quot;http://pyml.sourceforge.net/doc/howto.pdf&quot; rel=&quot;nofollow&quot;&gt;linear combination of examples&lt;/a&gt;, and the magnitude of those weights represents how much they were effective for the prediction problem at hand. But how do you compute the same for an SSVM?&lt;/p&gt;&#xA;" OwnerUserId="2949" LastEditorUserId="2949" LastEditDate="2014-08-14T14:42:08.143" LastActivityDate="2014-08-14T14:42:08.143" Title="feature weights in structured support vector machine" Tags="|svm|" AnswerCount="0" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="974" PostTypeId="1" CreationDate="2014-08-14T00:50:51.103" Score="23" ViewCount="8442" Body="&lt;p&gt;I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. Given the size of my dataset and some initial tests I performed, my assumption is that the method I will use will need to be either parallel or distributed. So I'm considering two classes of possible solutions: one that is either available (or implementable in a reasonably easy way) on a single multicore machine, the other on a Spark cluster, i.e. as a MapReduce program. Here are three broad ideas that I considered:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assuming a cosine similarity metric, perform the full multiplication of the normalized matrix by its transpose (implemented as a sum of outer products)&lt;/li&gt;&#xA;&lt;li&gt;Using locality-sensitive hashing (LSH)&lt;/li&gt;&#xA;&lt;li&gt;Reducing first the dimensionality of the problem with a PCA&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate any thoughts or advices about possible other ways in which I could tackle this problem.&lt;/p&gt;&#xA;" OwnerUserId="1242" LastActivityDate="2017-07-10T12:46:30.723" Title="Nearest neighbors search for very high dimensional data" Tags="|machine-learning|distributed|map-reduce|dimensionality-reduction|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="975" PostTypeId="2" ParentId="974" CreationDate="2014-08-14T04:28:36.933" Score="17" Body="&lt;p&gt;I hope that the following &lt;strong&gt;resources&lt;/strong&gt; might get you additional &lt;strong&gt;ideas&lt;/strong&gt; toward solving the problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Research paper &lt;strong&gt;&quot;Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data&quot;&lt;/strong&gt;: &lt;a href=&quot;http://arxiv.org/abs/1011.2807&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/abs/1011.2807&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Class project paper &lt;strong&gt;&quot;Recommendation System Based on Collaborative Filtering&quot;&lt;/strong&gt; (Stanford University): &lt;a href=&quot;http://cs229.stanford.edu/proj2008/Wen-RecommendationSystemBasedOnCollaborativeFiltering.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://cs229.stanford.edu/proj2008/Wen-RecommendationSystemBasedOnCollaborativeFiltering.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Project for the &lt;strong&gt;Netflix Prize Competition (&lt;em&gt;k-NN&lt;/em&gt;-based)&lt;/strong&gt;: &lt;a href=&quot;http://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/index.html&quot; rel=&quot;noreferrer&quot;&gt;http://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Research paper &lt;strong&gt;&quot;Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data&quot;&lt;/strong&gt; on the &lt;em&gt;curse of dimensionality&lt;/em&gt; phenomenon and its relation to &lt;em&gt;machine learning&lt;/em&gt;, in general, and &lt;em&gt;k-NN algorithm&lt;/em&gt;, in particular: &lt;a href=&quot;http://jmlr.org/papers/volume11/radovanovic10a/radovanovic10a.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://jmlr.org/papers/volume11/radovanovic10a/radovanovic10a.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) &lt;strong&gt;Software for sparse k-NN classification&lt;/strong&gt; (free, but appears not to be open source - might clarify with authors): &lt;a href=&quot;http://www.autonlab.org/autonweb/10408.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.autonlab.org/autonweb/10408.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) Several &lt;strong&gt;discussion threads&lt;/strong&gt; on &lt;em&gt;StackOverflow&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/20333092/knn-with-big-sparse-matrices-in-python&quot;&gt;https://stackoverflow.com/questions/20333092/knn-with-big-sparse-matrices-in-python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/18164348/efficient-nearest-neighbour-search-for-sparse-matrices&quot;&gt;https://stackoverflow.com/questions/18164348/efficient-nearest-neighbour-search-for-sparse-matrices&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/21085990/scipy-sparse-distance-matrix-scikit-or-scipy&quot;&gt;https://stackoverflow.com/questions/21085990/scipy-sparse-distance-matrix-scikit-or-scipy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/10472681/handling-incomplete-data-data-sparsity-in-knn&quot;&gt;https://stackoverflow.com/questions/10472681/handling-incomplete-data-data-sparsity-in-knn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/5560218/computing-sparse-pairwise-distance-matrix-in-r&quot;&gt;https://stackoverflow.com/questions/5560218/computing-sparse-pairwise-distance-matrix-in-r&lt;/a&gt; (unlike all previous discussions, which refer to &lt;code&gt;Python&lt;/code&gt;, this one refers to &lt;code&gt;R&lt;/code&gt; ecosystem)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;7) Pay attention to &lt;em&gt;GraphLab&lt;/em&gt;, an open source &lt;strong&gt;parallel framework for machine learning&lt;/strong&gt; (&lt;a href=&quot;http://select.cs.cmu.edu/code/graphlab&quot; rel=&quot;noreferrer&quot;&gt;http://select.cs.cmu.edu/code/graphlab&lt;/a&gt;), which supports &lt;em&gt;parallel clustering&lt;/em&gt; via &lt;code&gt;MapReduce&lt;/code&gt; model: &lt;a href=&quot;http://select.cs.cmu.edu/code/graphlab/clustering.html&quot; rel=&quot;noreferrer&quot;&gt;http://select.cs.cmu.edu/code/graphlab/clustering.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also check my answer here on Data Science StackExchange on &lt;em&gt;sparse regression&lt;/em&gt; for links to relevant &lt;code&gt;R&lt;/code&gt; packages and &lt;code&gt;CRAN Task View&lt;/code&gt; pages: &lt;a href=&quot;https://datascience.stackexchange.com/a/918/2452&quot;&gt;https://datascience.stackexchange.com/a/918/2452&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-08-14T11:52:35.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="976" PostTypeId="2" ParentId="915" CreationDate="2014-08-14T17:28:40.453" Score="5" Body="&lt;p&gt;I gave a very limited partial answer for the confirmatory factor analysis package that I developed for Stata in this &lt;a href=&quot;http://www.stata-journal.com/article.html?article=st0169&quot; rel=&quot;noreferrer&quot;&gt;Stata Journal article&lt;/a&gt; based on timing the actual simulations. Confirmatory factor analysis was implemented as a maximum likelihood estimation technique, and I could see very easily how the computation time grew with each dimension (sample size &lt;code&gt;n&lt;/code&gt;, number of variables &lt;code&gt;p&lt;/code&gt;, number of factors &lt;code&gt;k&lt;/code&gt;). As it is heavily dependent on how Stata thinks about the data (optimized to compute across columns/observations rather than rows), I found performance to be &lt;code&gt;O(n^{0.68} (k+p)^{2.4})&lt;/code&gt; where 2.4 is the fastest matrix inversion asymptotics (and there's hell of a lot of that in confirmatory factor analysis iterative maximization). I did not give a reference for the latter, but I think I got this from &lt;a href=&quot;http://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that there is also a matrix inversion step in OLS. However, for reasons of numerical accuracy, no one would really brute-force inverse the &lt;code&gt;X'X&lt;/code&gt; matrix, and would rather use sweep operators and identify the dangerously collinear variables to deal with precision issues. If you add up $10^8$ numbers that originally were in &lt;a href=&quot;http://en.wikipedia.org/wiki/Double-precision_floating-point_format&quot; rel=&quot;noreferrer&quot;&gt;double precision&lt;/a&gt;, you will likely end up with a number that only has a single precision. Numerical computing issues may become a forgotten corner of big data calculations as you start optimizing for speed.&lt;/p&gt;&#xA;" OwnerUserId="1237" LastActivityDate="2014-08-14T17:28:40.453" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="977" PostTypeId="1" CreationDate="2014-08-14T19:09:29.523" Score="11" ViewCount="322" Body="&lt;p&gt;Let me show you an example of a hypothetical online clustering application:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZVOQN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At time n+1 a new point a is introduced which is assigned to the blue cluster A but also causes the point b to be assigned to the blue cluster A as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end points 1,2,3,4,a,b belong to A and points 5,6,7 to B. To me this seems reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What seems simple at first glance is actually a bit tricky - to maintain identifiers across time steps. Let me try to make this point clear with a more borderline example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/sQj4h.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The green point will cause two blue and two red points to be merged into one cluster which I arbitrarily decided to color blue - mind this is already my human heuristical thinking at work!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A computer to make this decision will have to use rules. For example when points are merged into a cluster then the identity of the cluster is determined by the majority. In this case we would face a draw - both blue and red might be valid choices for the new (here blue colored) cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine a fifth red point close to the green one. Then the majority would be red (3 red vs 2 blue) so red would be a good choice for the new cluster - but this would contradict the even clearer choice of red for the rightmost cluster as those have been red and probably should stay that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find it fishy to think about this. At the end of the day I guess there are no perfect rules for this - rather heuristics optimizing some stability criterea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This finally leads to my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this &quot;problem&quot; have a name that it can be referred to?&lt;/li&gt;&#xA;&lt;li&gt;Are there &quot;standard&quot; solutions to this and ...&lt;/li&gt;&#xA;&lt;li&gt;... is there maybe even an R package for that?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.joyofdata.de/blog/reasonable-inheritance-of-cluster-identities-in-repetitive-clustering/&quot; rel=&quot;noreferrer&quot;&gt;Reasonable Inheritance of Cluster Identities in Repetitive Clustering&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="725" LastEditorUserId="725" LastEditDate="2014-08-16T10:49:06.163" LastActivityDate="2016-09-29T10:26:23.220" Title="Solutions for Continuous Online Cluster Identification?" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="978" PostTypeId="2" ParentId="962" CreationDate="2014-08-14T21:38:57.630" Score="15" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But I don't know what is difference between text classification and topic models in documents&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Text Classification&lt;/code&gt;&lt;/strong&gt; is a form of supervised learning, hence the set of possible classes are known/defined &lt;em&gt;in advance&lt;/em&gt;, and won't change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Topic Modeling&lt;/code&gt;&lt;/strong&gt; is a form of unsupervised learning (akin to clustering), so the set of possible topics are unknown &lt;em&gt;apriori&lt;/em&gt;. They're defined as part of generating the topic models. With a non-deterministic algorithm like LDA, you'll get different topics each time you run the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Text classification&lt;/code&gt; often involves mutually-exclusive classes -- think of these as buckets.&lt;br&gt;&#xA;But it doesn't have to: given the right kind of labeled input data, you can set of a series of non-mutually-exclusive binary classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Topic modeling&lt;/code&gt; is generally not mutually-exclusive: the same document can have its probability distribution spread across many topics.&#xA;In addition, there are also hierarchical topic modeling methods.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Also can I use topic model for the documents to identify one topic later on can I use the classification to classify the text inside this documents ? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you're asking whether you can take all of the documents assigned to one topic by a topic modeling algorithm and then apply a classifier to that collection, then yes, you certainly can do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure it makes much sense, though: at a minimum, you'd need to pick a threshold for the topic probability distribution above which you'll include documents in your collection (typically 0.05-0.1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you elaborate on your use case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, there's a great tutorial on topic modeling using the MALLET library for Java available here: &lt;a href=&quot;http://programminghistorian.org/lessons/topic-modeling-and-mallet&quot; rel=&quot;noreferrer&quot;&gt;Getting Started with Topic Modeling and MALLET&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="819" LastEditorUserId="29575" LastEditDate="2018-05-04T13:11:47.637" LastActivityDate="2018-05-04T13:11:47.637" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="979" PostTypeId="1" AcceptedAnswerId="980" CreationDate="2014-08-15T13:10:20.937" Score="17" ViewCount="12386" Body="&lt;p&gt;I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings.&lt;/p&gt;&#xA;&lt;p&gt;What algorithms are suggested to do this? I don't know number of clusters in advance (and as more data is coming clusters can change as well), what features are normally used to represent each sentence?&lt;/p&gt;&#xA;&lt;p&gt;I'm trying now the simplest features with just list of words and distance between sentences defined as:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$|A \cup B$&lt;/span&gt; \ &lt;span class=&quot;math-container&quot;&gt;$A \cap B|$&lt;/span&gt;/&lt;span class=&quot;math-container&quot;&gt;$|A \cup B|$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;(A and B are corresponding sets of words in sentence A and B)&lt;/p&gt;&#xA;&lt;p&gt;Does it make sense at all?&lt;/p&gt;&#xA;&lt;p&gt;I'm trying to apply &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_mean_shift.html#example-cluster-plot-mean-shift-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mean-Shift&lt;/a&gt; algorithm from scikit library to this distance, as it does not require number of clusters in advance.&lt;/p&gt;&#xA;&lt;p&gt;If anyone will advise better methods/approaches for the problem - it will be very much appreciated as I'm still new to the topic.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastEditorUserId="29169" LastEditDate="2021-03-13T23:09:28.130" LastActivityDate="2021-06-20T19:21:33.163" Title="Algorithms for text clustering" Tags="|clustering|text-mining|algorithms|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="980" PostTypeId="2" ParentId="979" CreationDate="2014-08-15T14:00:14.927" Score="13" Body="&lt;p&gt;Check the &lt;strong&gt;&lt;a href=&quot;http://www-nlp.stanford.edu/software&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford NLP Group&lt;/a&gt;&lt;/strong&gt;'s open source software, in particular, &lt;strong&gt;&lt;a href=&quot;http://www-nlp.stanford.edu/software/classifier.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Classifier&lt;/a&gt;&lt;/strong&gt;. The software is written in &lt;code&gt;Java&lt;/code&gt;, which will likely delight you, but also has bindings for some other languages. Note, the &lt;em&gt;licensing&lt;/em&gt; - if you plan to use their code in commercial products, you have to acquire commercial license.&lt;/p&gt;&#xA;&lt;p&gt;Another interesting set of open source libraries, IMHO suitable for this task and much more, is &lt;strong&gt;&lt;a href=&quot;http://select.cs.cmu.edu/code/graphlab&quot; rel=&quot;nofollow noreferrer&quot;&gt;parallel framework for machine learning GraphLab&lt;/a&gt;&lt;/strong&gt;, which includes &lt;strong&gt;&lt;a href=&quot;http://select.cs.cmu.edu/code/graphlab/clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;clustering library&lt;/a&gt;&lt;/strong&gt;, implementing various clustering algorithms. It is especially suitable for &lt;strong&gt;very large volume of data&lt;/strong&gt; (like you have), as it implements &lt;code&gt;MapReduce&lt;/code&gt; model and, thus, supports &lt;em&gt;multicore&lt;/em&gt; and &lt;em&gt;multiprocessor&lt;/em&gt; &lt;strong&gt;parallel processing&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You most likely are aware of the following, but I will mention it just in case. &lt;strong&gt;&lt;a href=&quot;http://www.nltk.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Natural Language Toolkit (NLTK)&lt;/a&gt;&lt;/strong&gt; for &lt;code&gt;Python&lt;/code&gt; contains modules for clustering/classifying/categorizing text. Check the relevant chapter in the &lt;a href=&quot;http://www.nltk.org/book/ch06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK Book&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Speaking of &lt;strong&gt;algorithms&lt;/strong&gt;, it seems that you've tried most of the ones from &lt;code&gt;scikit-learn&lt;/code&gt;, such as illustrated in &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; topic extraction example. However, you may find useful other libraries, which implement a wide variety of &lt;em&gt;clustering algorithms&lt;/em&gt;, including &lt;em&gt;Non-Negative Matrix Factorization (NMF)&lt;/em&gt;. One of such libraries is &lt;strong&gt;&lt;a href=&quot;https://code.google.com/p/pymf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Matrix Factorization (PyMF)&lt;/a&gt;&lt;/strong&gt; (&lt;a href=&quot;https://github.com/nils-werner/pymf&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;). Another, even more interesting, library, also Python-based, is &lt;strong&gt;&lt;a href=&quot;http://nimfa.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;NIMFA&lt;/a&gt;&lt;/strong&gt;, which implements various &lt;em&gt;NMF algorithms&lt;/em&gt;. Here's a &lt;a href=&quot;http://jmlr.org/papers/volume13/zitnik12a/zitnik12a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper&lt;/a&gt;, describing &lt;code&gt;NIMFA&lt;/code&gt;. &lt;a href=&quot;http://nimfa.biolab.si/nimfa.examples.documents.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's&lt;/a&gt; an example from its documentation, which presents the solution for very similar text processing problem of &lt;em&gt;topic clustering&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="25180" LastEditDate="2021-06-20T19:21:33.163" LastActivityDate="2021-06-20T19:21:33.163" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="981" PostTypeId="2" ParentId="802" CreationDate="2014-08-15T20:22:11.750" Score="3" Body="&lt;p&gt;Basically for this task you can efficiently use any SQL database with good support of B+tree based indexes (MySQL will suite you needs just perfect). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create 3 tables:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Documents table, columns: id/document&lt;/li&gt;&#xA;&lt;li&gt;N-grams table: n_gram_id/n_gram&lt;/li&gt;&#xA;&lt;li&gt;Mapping between n-grams and documents: document_id/n_gram_id&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Create indexes on N-gram table/n_gram string and Mapping table/n_gram_id, also primary keys will be indexed by default well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your operations will be efficient:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Insertion of document: just extract all n-grams and insert into document table and N-grams table&lt;/li&gt;&#xA;&lt;li&gt;Lookup for in_gram will be quick with support of index&lt;/li&gt;&#xA;&lt;li&gt;Querying for all n-grams that contain a sub-n-gram: in 2 steps - just query based on index all n-grams which contain sub-n-gram from 2nd table. Then - retrieve all corresponding documents for each of these n-grams.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You don't even need to use joins to achieve all these operations so indexes will help a lot. Also if data will not suite in one machine - you can implement sharding scheme, like storing n_grams started from a-n on one server and o-z on another or other suitable scheme. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can use MongoDB, but I'm not sure how exactly you need to implement indexing scheme. For MongoDB you will get sharding scheme for free as it is already built-in.&lt;/p&gt;&#xA;" OwnerUserId="2958" LastActivityDate="2014-08-15T20:22:11.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="982" PostTypeId="2" ParentId="802" CreationDate="2014-08-16T10:25:19.687" Score="9" Body="&lt;p&gt;See &lt;a href=&quot;https://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizer.html&quot;&gt;Lucene NGramTokenizer&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you sure you can't just use lucene or similar indexing techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inverted indexes will store the n-gram only once, then just the document ids that contain the ngram; they don't store this as highly redundant raw text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for finding ngrams that contain your query sub-n-gram, I would build an index on the observed ngrams, e.g. using a second lucene index, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Substring_index&quot;&gt;any other substring index&lt;/a&gt; such as a trie or suffix tree. If your data is dynamic, probably lucene is a reasonable choice, using phrase queries to find your n-grams.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-08-16T10:25:19.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="985" PostTypeId="1" CreationDate="2014-08-16T11:05:43.353" Score="5" ViewCount="10126" Body="&lt;p&gt;I have a question about classifying documents using supervised learning and unsupervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: - I have a bunch of documents talking about football. As we know, football has a different meaning in UK, USA, and Australia. Therefore, it is difficult to classify these documents to three different categorizations (which are soccer, American football and Australian football).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach tries to use cosine similarity terms based on unsupervised learning. After we use the cluster learning, we are able to create a number of clusters based on cosine similarity, where each cluster will contain similar documents terms. After we create the clusters, we can use a semantic feature to identify these clusters depending on a supervised model like SVM to make accurate categorizations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to create more accurate categorizations, because if I would like to test a new document, I would like know if this document can be related to these categorizations or not.  &lt;/p&gt;&#xA;" OwnerUserId="2916" LastEditorUserId="29169" LastEditDate="2019-06-07T23:31:10.447" LastActivityDate="2019-06-08T00:13:10.650" Title="Can I use unsupervised learning followed by supervised learning?" Tags="|machine-learning|classification|clustering|dimensionality-reduction|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="986" PostTypeId="2" ParentId="946" CreationDate="2014-08-17T14:17:41.307" Score="2" Body="&lt;p&gt;OK, so here's your data. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dd &amp;lt;- data.frame(position=rep(1:3, each=2), &#xA;                 variation=rep(c(1,3), 3), &#xA;                 impressions=rep(c(753, 767), 3), &#xA;                 clicks=c(26,7,16,13,2,7))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  position variation impressions clicks&#xA;1        1         1         753     26&#xA;2        1         3         767      7&#xA;3        2         1         753     16&#xA;4        2         3         767     13&#xA;5        3         1         753      2&#xA;6        3         3         767      7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The two model assumptions you're thinking about are Binomial&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod.bin &amp;lt;- glm(cbind(clicks, impressions-clicks) ~ variation + position,&#xA;               family=binomial, data=dd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the dependent variable is constructed to have the count of the event of interest in the first column, and the Poisson&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;md.pois &amp;lt;- glm(clicks ~ variation + position + offset(log(impressions)), &#xA;               family=poisson, data=dd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the &lt;code&gt;log(impressions)&lt;/code&gt; offset is necessary whenever the number of trials differs across observations.  This means coefficients are interpretable in terms of change in rate not change in count, which is what you want.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first model generalises the &lt;code&gt;binom.test&lt;/code&gt; to a setting with covariates, which is what you have.  That gets you a more direct answer to your question, and better (if not perfect) measurement of the relevant uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both models assume no interaction between variation and position ('independent effects').  This may or may not be reasonable.  You'd want more replications to investigate that properly.  Swap the &lt;code&gt;+&lt;/code&gt; for a &lt;code&gt;*&lt;/code&gt; to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this data &lt;code&gt;summary&lt;/code&gt; confirms that the two models give rather similar results, so concerns about Poisson vs Binomial don't seem to matter much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the wild, count data is usually overdispersed, that is: more variable than you'd expect from a Poisson with a constant rate or a Binomial with constant click probability, often due to unmodeled determinants of click rate / probability.  If that's the case then prediction intervals from these models will be too narrow.&lt;/p&gt;&#xA;" OwnerUserId="2978" LastEditorUserId="2978" LastEditDate="2014-08-17T15:04:13.880" LastActivityDate="2014-08-17T15:04:13.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="987" PostTypeId="1" CreationDate="2014-08-17T17:29:44.123" Score="26" ViewCount="17900" Body="&lt;p&gt;The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). This works reasonably well (around 90% macro F-1 on test set, nearly 100% on training set). A big problem are unseen words/n-grams. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to improve the classifier by adding other features, e.g. a fixed sized vector computed using distributional similarities (as computed by word2vec) or other categorical features of the examples. My idea was to just add the features to the sparse input features from the bag of words. However, this results in worse performance on the test and training set. The additional features by themselves give about 80% F-1 on the test set, so they aren't garbage. Scaling the features didn't help as well. My current thinking is that these kind of features don't mix well with the (sparse) bag of words features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is: assuming the additional features provide additional information, what is the best way to incorporate them? Could training separate classifiers and combining them in some kind of ensemble work (this would probably have the drawback that no interaction between the features of the different classifiers could be captured)? Are there other more complex models I should consider?&lt;/p&gt;&#xA;" OwnerUserId="2979" LastActivityDate="2016-01-06T04:20:10.207" Title="Text categorization: combining different kind of features" Tags="|machine-learning|classification|feature-selection|logistic-regression|information-retrieval|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="988" PostTypeId="2" ParentId="955" CreationDate="2014-08-18T08:49:11.933" Score="2" Body="&lt;p&gt;Here is a couple of tools that may be worth a look:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bart, an open source tool that have been used for several languages, including German. &lt;a href=&quot;http://www.bart-coref.org/&quot; rel=&quot;nofollow&quot;&gt;Available from the website&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Sucre is a tool developed at the University of Stuttgart. I don't know if it's available easily. &lt;a href=&quot;http://hnk.ffzg.hr/bibl/acl2011/CoNLL-ST/pdf/CoNLL-ST3.pdf&quot; rel=&quot;nofollow&quot;&gt;You can see this paper about it&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2985" LastActivityDate="2014-08-18T08:49:11.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="989" PostTypeId="1" CreationDate="2014-08-18T10:46:57.360" Score="117" ViewCount="151661" Body="&lt;p&gt;I am trying to run SVR using scikit-learn (python) on a training dataset that has 595605 rows and 5 columns (features) while the test dataset has 397070 rows. The data has been pre-processed and regularized.&lt;/p&gt;&#xA;&lt;p&gt;I am able to successfully run the test examples, but on executing using my dataset and letting it run for over an hour, I could still not see any output or termination of the program. I tried executing using a different IDE and even from the terminal, but that does not seem to be the issue.&#xA;I also tried changing the 'C' parameter value from 1 to 1e3.&lt;/p&gt;&#xA;&lt;p&gt;I am facing similar issues with all SVM implementations using scikit.&lt;/p&gt;&#xA;&lt;p&gt;Am I not waiting long enough for it to complete?&#xA;How much time should this execution take?&lt;/p&gt;&#xA;&lt;p&gt;From my experience, it should not require more than a few minutes.&lt;/p&gt;&#xA;&lt;p&gt;Here is my system configuration:&#xA;Ubuntu 14.04, 8GB RAM, lots of free memory, 4th gen i7 processor&lt;/p&gt;&#xA;" OwnerUserId="793" LastEditorUserId="29169" LastEditDate="2021-02-07T20:22:14.793" LastActivityDate="2021-08-11T10:26:27.617" Title="SVM using scikit learn runs endlessly and never completes execution" Tags="|python|svm|scikit-learn|" AnswerCount="12" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="990" PostTypeId="2" ParentId="989" CreationDate="2014-08-18T11:33:32.617" Score="2" Body="&lt;p&gt;This makes sense. IIUC, the speed of execution of support vector operations is bound by number of samples, not dimensionality. In other words, it is capped by CPU time and not RAM. I'm not sure exactly how much time this should take, but I'm running some benchmarks to find out.&lt;/p&gt;&#xA;" OwnerUserId="2987" LastActivityDate="2014-08-18T11:33:32.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="991" PostTypeId="2" ParentId="987" CreationDate="2014-08-18T13:35:28.000" Score="10" Body="&lt;p&gt;Linear models simply add their features multiplied by corresponding weights. If, for example, you have 1000 sparse features only 3 or 4 of which are active in each instance (and the others are zeros) and 20 dense features that are all non-zeros, then it's pretty likely that dense features will make most of the impact while sparse features will add only a little value. You can check this by looking at feature weights for a few instances and how they influence resulting sum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to fix it is to go away from additive model. Here's a couple of candidate models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SVM&lt;/strong&gt; is based on separating hyperplanes. Though hyperplane is linear model itself, SVM doesn't sum up its parameters, but instead tries to split feature space in an optimal way. Given the number of features, I'd say that linear SVM should work fine while more complicated kernels may tend to overfit the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite its name, &lt;strong&gt;Naive Bayes&lt;/strong&gt; is pretty powerful statistical model that showed good results for text classification. It's also flexible enough to capture imbalance in frequency of sparse and dense features, so you should definitely give it a try. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, &lt;strong&gt;random forests&lt;/strong&gt; may work as a good ensemble method in this case. Randomization will ensure that different kinds of features (sparse/dense) will be used as primary decision nodes in different trees. RF/decision trees are also good for inspecting features themselves, so it's worth to note their structure anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that all of these methods have their drawbacks that may turn them into a garbage in your case. Combing sparse and dense features isn't really well-studied task, so let us know what of these approaches works best for your case. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-08-18T13:35:28.000" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="992" PostTypeId="1" AcceptedAnswerId="999" CreationDate="2014-08-18T14:56:13.800" Score="10" ViewCount="729" Body="&lt;p&gt;I've been analyzing a data set of ~400k records and 9 variables The dependent variable is binary. I've fitted a logistic regression, a regression tree, a random forest, and a gradient boosted tree. All of them give virtual identical goodness of fit numbers when I validate them on another data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this so? I'm guessing that it's because my observations to variable ratio is so high. If this is correct, at what observation to variable ratio will different models start to give different results? &lt;/p&gt;&#xA;" OwnerUserId="1241" LastEditorUserId="97" LastEditDate="2014-08-20T11:06:54.447" LastActivityDate="2014-08-20T11:06:54.447" Title="Why might several types of models give almost identical results?" Tags="|data-mining|classification|binary|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="993" PostTypeId="2" ParentId="992" CreationDate="2014-08-18T16:28:02.147" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm guessing that it's because my observations to variable ratio is so high.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think this explanation makes perfect sense.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If this is correct, at what observation to variable ratio will different models start to give different results? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This will probably depend very much on your specific data (for instance, even whether your nine variables are continuous, factors, ordinary or binary), as well as any tuning decisions you made while fitting your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you can play around with the observation-to-variable ratio - not by increasing the number of variables, but by decreasing the number of observations. Randomly draw 100 observations, fit models and see whether different models yield different results. (I guess they will.) Do this multiple times with different samples drawn from your total number of observations. Then look at subsamples of 1,000 observations... 10,000 observations... and so forth.&lt;/p&gt;&#xA;" OwnerUserId="2853" LastActivityDate="2014-08-18T16:28:02.147" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="994" PostTypeId="2" ParentId="992" CreationDate="2014-08-18T17:05:19.957" Score="4" Body="&lt;p&gt;its worth also looking at the training errors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;basically I disagree with your  analysis. if logistic regression etc are all giving the same results it would suggest that the 'best model' is a very simple one (that all models can fit equally well - eg basically linear).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So then the question might be why is the best model a simple model?:&#xA;It might suggest that your variables are not very predictive. Its of course hard to analyse without knowing the data. &lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-08-18T17:05:19.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="995" PostTypeId="2" ParentId="992" CreationDate="2014-08-18T17:25:27.007" Score="1" Body="&lt;p&gt;As @seanv507 suggested, the similar performance may simply be due to the data being best separated by a linear model. But in general, the statement that it is because the &quot;observations to variable ratio is so high&quot; is incorrect. Even as your ratio of sample size to number of variables goes to infinity, you should not expect different models to perform nearly identically, unless they all provide the same predictive bias.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2014-08-18T17:25:27.007" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="996" PostTypeId="2" ParentId="989" CreationDate="2014-08-19T00:56:40.890" Score="99" Body="&lt;p&gt;Kernelized SVMs require the computation of a distance function between each point in the dataset, which is the dominating cost of $\mathcal{O}(n_\text{features} \times n_\text{observations}^2)$. The storage of the distances is a burden on memory, so they're recomputed on the fly. Thankfully, only the points nearest the decision boundary are needed most of the time. Frequently computed distances are stored in a cache. If the cache is getting thrashed then the running time blows up to $\mathcal{O}(n_\text{features} \times n_\text{observations}^3)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can increase this cache by invoking SVR as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = SVR(cache_size=7000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In general, this is not going to work. But all is not lost. You can subsample the data and use the rest as a validation set, or you can pick a different model. Above the 200,000 observation range, it's wise to choose linear learners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kernel SVM can be approximated, by approximating the kernel matrix and feeding it to a linear SVM. This allows you to trade off between accuracy and performance in linear time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A popular means of achieving this is to use 100 or so cluster centers found by kmeans/kmeans++ as the basis of your kernel function. The new derived features are then fed into a linear model. This works very well in practice. Tools like &lt;a href=&quot;https://code.google.com/p/sofia-ml/&quot;&gt;sophia-ml&lt;/a&gt; and &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot;&gt;vowpal wabbit&lt;/a&gt; are how Google, Yahoo and Microsoft do this. Input/output becomes the dominating cost for simple linear learners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the abundance of data, nonparametric models perform roughly the same for most problems. The exceptions being structured inputs, like text, images, time series, audio.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Further reading&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://fastml.com/the-secret-of-the-big-guys/&quot;&gt;How to implement this.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://fastml.com/go-non-linear-with-vowpal-wabbit/&quot;&gt;How to train an ngram neural network with dropout that scales linearly&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://peekaboo-vision.blogspot.co.uk/2012/12/kernel-approximations-for-efficient.html&quot;&gt;Kernel Approximations&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.144.9009&amp;amp;rep=rep1&amp;amp;type=pdf&quot;&gt;A formal paper on using kmeans to approximate kernel machines&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2997" LastEditorUserId="8820" LastEditDate="2016-01-14T15:20:08.840" LastActivityDate="2016-01-14T15:20:08.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="997" PostTypeId="1" CreationDate="2014-08-19T03:41:24.207" Score="7" ViewCount="5303" Body="&lt;p&gt;Where can I find free spatio-temporal dataset for download so that I can play with it in R ? &lt;/p&gt;&#xA;" OwnerUserId="2972" LastEditorUserId="1352" LastEditDate="2016-03-29T10:55:12.397" LastActivityDate="2019-07-12T12:49:27.840" Title="Where can I find free spatio-temporal dataset for download?" Tags="|dataset|open-source|freebase|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="998" PostTypeId="2" ParentId="937" CreationDate="2014-08-19T13:33:32.453" Score="34" Body="&lt;p&gt;No, scikit-learn does not seem to have a forward selection algorithm. However, it does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2969" LastEditorUserId="29169" LastEditDate="2021-08-15T21:08:31.770" LastActivityDate="2021-08-15T21:08:31.770" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="999" PostTypeId="2" ParentId="992" CreationDate="2014-08-19T14:13:07.147" Score="7" Body="&lt;p&gt;This results means that whatever method you use, you are able to get reasonably close to the optimal decision rule (aka &lt;a href=&quot;http://en.wikipedia.org/wiki/Admissible_decision_rule#Bayes_rules_and_generalized_Bayes_rules&quot;&gt;Bayes rule&lt;/a&gt;). The underlying reasons have been explained in Hastie, Tibshirani and Friedman's &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot;&gt;&quot;Elements of Statistical Learning&quot;&lt;/a&gt;. They demonstrated how the different methods perform by comparing Figs. 2.1, 2.2, 2.3, 5.11 (in my first edition -- in section on multidimensional splines), 12.2, 12.3 (support vector machines), and probably some others. If you have not read that book, you need to drop everything &lt;strong&gt;RIGHT NOW&lt;/strong&gt; and read it up. (I mean, it isn't worth losing your job, but it is worth missing a homework or two if you are a student.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think that observations to variable ratio is the explanation. In light of my rationale offered above, it is the relatively simple form of the boundary separating your classes in the multidimensional space that all of the methods you tried have been able to identify.&lt;/p&gt;&#xA;" OwnerUserId="1237" LastActivityDate="2014-08-19T14:13:07.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1000" PostTypeId="2" ParentId="997" CreationDate="2014-08-19T15:21:57.833" Score="3" Body="&lt;p&gt;You can get some documented, publicly available EEG data from the HeadIT database at UCSD.&#xA;&lt;a href=&quot;http://headit-beta.ucsd.edu/studies&quot; rel=&quot;nofollow&quot;&gt;http://headit-beta.ucsd.edu/studies&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data itself appears to be in Biosemi Data Format (.bdf) files, described here: &lt;a href=&quot;http://www.biosemi.com/faq/file_format.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.biosemi.com/faq/file_format.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Biosemi provides links to several open-source methods to access and import .bdf files on their website, including several functions for importing into Matlab, as well as into Python (BioSig) and C/C++ libraries:&#xA;&lt;a href=&quot;http://www.biosemi.com/download.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.biosemi.com/download.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just as a forewarning, EEG data can be a bit of a bear to work with, due to it's inherently low signal/noise ratio.&lt;/p&gt;&#xA;" OwnerUserId="2969" LastActivityDate="2014-08-19T15:21:57.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1001" PostTypeId="2" ParentId="997" CreationDate="2014-08-19T15:47:29.413" Score="5" Body="&lt;p&gt;First thing that came to mind would be one's personal workout data from running or biking apps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise there is a dataset around NYC's taxi trip data. Quick Googling brought me this: &lt;a href=&quot;http://www.andresmh.com/nyctaxitrips/&quot;&gt;http://www.andresmh.com/nyctaxitrips/&lt;/a&gt;. Variables include time and location for both pickups and dropoffs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another dataset comes from Chicago's bikesharing service. It can be found here: &lt;a href=&quot;https://www.divvybikes.com/datachallenge&quot;&gt;https://www.divvybikes.com/datachallenge&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="525" LastActivityDate="2014-08-19T15:47:29.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1002" PostTypeId="1" AcceptedAnswerId="1006" CreationDate="2014-08-19T17:50:52.583" Score="10" ViewCount="1675" Body="&lt;p&gt;Caveat: I am a complete beginner when it comes to machine learning, but eager to learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data, either with known variables, or variables that are contained in the data but which I haven't yet realised are actually variables / relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing this would be a familiar problem in the world of data analysis, so I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The 'silver bullet' would be to throw this all this data into a stats / data analysis program and for it to crunch the data looking for known / unknown patterns trying to find relations. Is SPSS suitable, or are there other applications which may be better suited.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I learn a language like R, and figure out how to manually process the data. Wouldn't this comprimise finding relations as I would have to manually specify what and how to analyse the data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How would a professional data miner approach this problem and what steps would s/he take?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2861" LastEditorUserId="9420" LastEditDate="2016-07-17T14:45:20.330" LastActivityDate="2020-08-16T18:02:41.913" Title="What initial steps should I use to make sense of large data sets, and what tools should I use?" Tags="|machine-learning|data-mining|tools|beginner|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1003" PostTypeId="1" CreationDate="2014-08-19T18:59:03.013" Score="3" ViewCount="222" Body="&lt;p&gt;I recently read &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/sdumais/ecir07-metzlerdumaismeek-final.pdf&quot; rel=&quot;nofollow&quot;&gt;Similarity Measures for Short Segments of Text&lt;/a&gt; (Metzler et al.).  It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their top results. Results are lists of page urls, page titles, and short page snippets.  In the paper, the authors collect 200 results per query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.  There's a substantial difference between 10 and 200.  Hence, how much data is commonly used in practice to measure query similarity (e.g., how many results per query)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References are a plus!&lt;/p&gt;&#xA;" OwnerUserId="1097" LastEditorUserId="97" LastEditDate="2014-08-20T12:14:01.593" LastActivityDate="2015-05-24T22:46:06.903" Title="Query similarity: how much data is used in practice?" Tags="|machine-learning|dataset|text-mining|search|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1005" PostTypeId="2" ParentId="997" CreationDate="2014-08-20T03:06:01.753" Score="3" Body="&lt;p&gt;Another idea is to combine OpenStreetMap project map data, for example, using corresponding nice R package (&lt;a href=&quot;http://www.r-bloggers.com/the-openstreetmap-package-opens-up&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/the-openstreetmap-package-opens-up&lt;/a&gt;), with census data (population census data, such as the US data: &lt;a href=&quot;http://www.census.gov/data/data-tools.html&quot; rel=&quot;nofollow&quot;&gt;http://www.census.gov/data/data-tools.html&lt;/a&gt;, as well as census data in other categories: &lt;a href=&quot;http://national.census.okfn.org&quot; rel=&quot;nofollow&quot;&gt;http://national.census.okfn.org&lt;/a&gt;) to analyze temporal patterns of geosocial trends. HTH.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-08-20T03:06:01.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1006" PostTypeId="2" ParentId="1002" CreationDate="2014-08-20T05:43:08.610" Score="11" Body="&lt;p&gt;I will try to answer your questions, but before I'd like to note that using term &amp;quot;large dataset&amp;quot; is misleading, as &amp;quot;large&amp;quot; is a &lt;em&gt;relative&lt;/em&gt; concept. You have to provide more details. If you're dealing with &lt;strong&gt;bid data&lt;/strong&gt;, then this fact will most likely affect selection of preferred &lt;em&gt;tools&lt;/em&gt;, &lt;em&gt;approaches&lt;/em&gt; and &lt;em&gt;algorithms&lt;/em&gt; for your &lt;strong&gt;data analysis&lt;/strong&gt;. I hope that the following thoughts of mine on data analysis address your sub-questions. Please note that the numbering of my points does not match the numbering of your sub-questions. However, I believe that it better reflects general &lt;strong&gt;data analysis workflow&lt;/strong&gt;, at least, how I understand it.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Firstly, I think that you need to have at least some kind of &lt;strong&gt;conceptual model&lt;/strong&gt; in mind (or, better, on paper). This model should guide you in your &lt;em&gt;exploratory data analysis (EDA)&lt;/em&gt;. A presence of a &lt;em&gt;dependent variable (DV)&lt;/em&gt; in the model means that in your &lt;em&gt;machine learning (ML)&lt;/em&gt; phase later in the analysis you will deal with so called supervised ML, as opposed to unsupervised ML in the absence of an identified DV.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Secondly, &lt;strong&gt;EDA&lt;/strong&gt; is a crucial part. IMHO, EDA should include &lt;strong&gt;multiple iterations&lt;/strong&gt; of producing &lt;em&gt;descriptive statistics&lt;/em&gt; and &lt;em&gt;data visualization&lt;/em&gt;, as you refine your understanding about the data. Not only this phase will give you valuable insights about your datasets, but it will feed your next important phase - &lt;strong&gt;data cleaning and transformation&lt;/strong&gt;. Just throwing your raw data into a statistical software package won't give much - for any &lt;strong&gt;valid&lt;/strong&gt; statistical analysis, data should be &lt;em&gt;clean, correct and consistent&lt;/em&gt;. This is often the most time- and effort-consuming, but absolutely necessary part. For more details on this topic, read &lt;a href=&quot;http://vita.had.co.nz/papers/tidy-data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; nice paper (by Hadley Wickham) and &lt;a href=&quot;http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (by Edwin de Jonge and Mark van der Loo).&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now, as you're hopefully done with &lt;em&gt;EDA&lt;/em&gt; as well as data cleaning and transformation, your ready to start some more statistically-involved phases. One of such phases is &lt;em&gt;exploratory factor analysis (EFA)&lt;/em&gt;, which will allow you to extract the underlying &lt;strong&gt;structure&lt;/strong&gt; of your data. For datasets with large number of variables, the positive side effect of EFA is &lt;em&gt;dimensionality reduction&lt;/em&gt;. And, while in that sense EFA is similar to &lt;em&gt;principal components analysis (PCA)&lt;/em&gt; and other dimensionality reduction approaches, I think that EFA is more important as it allows to refine your conceptual model of the phenomena that your data &amp;quot;describe&amp;quot;, thus making sense of your datasets. Of course, in addition to EFA, you can/should perform &lt;strong&gt;regression analysis&lt;/strong&gt; as well as apply &lt;strong&gt;machine learning techniques&lt;/strong&gt;, based on your findings in previous phases.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Finally, a note on &lt;strong&gt;software tools&lt;/strong&gt;. In my opinion, current state of statistical software packages is at such point that practically any major software packages have comparable offerings feature-wise. If you study or work in an organization that have certain policies and preferences in term of software tools, then you are &lt;em&gt;constrained&lt;/em&gt; by them. However, if that is not the case, I would heartily recommend &lt;strong&gt;open source&lt;/strong&gt; statistical software, based on your comfort with its specific &lt;em&gt;programming language&lt;/em&gt;, &lt;em&gt;learning curve&lt;/em&gt; and your &lt;em&gt;career perspectives&lt;/em&gt;. My current platform of choice is &lt;strong&gt;R Project&lt;/strong&gt;, which offers mature, powerful, flexible, extensive and open statistical software, along with amazing ecosystem of packages, experts and enthusiasts. Other nice choices include &lt;em&gt;Python&lt;/em&gt;, &lt;em&gt;Julia&lt;/em&gt; and specific open source software for processing &lt;strong&gt;big data&lt;/strong&gt;, such as &lt;em&gt;Hadoop&lt;/em&gt;, &lt;em&gt;Spark&lt;/em&gt;, &lt;em&gt;NoSQL&lt;/em&gt; databases, &lt;em&gt;WEKA&lt;/em&gt;. For more examples of open source software for &lt;strong&gt;data mining&lt;/strong&gt;, which include general and specific statistical and ML software, see this section of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_mining#Free_open-source_data_mining_software_and_applications&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;UPDATE: Forgot to mention &lt;em&gt;&lt;a href=&quot;http://rattle.togaware.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rattle&lt;/a&gt;&lt;/em&gt;, which is also a very popular open source R-oriented GUI software for data mining.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-16T18:02:41.913" LastActivityDate="2020-08-16T18:02:41.913" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="1007" PostTypeId="1" AcceptedAnswerId="1008" CreationDate="2014-08-20T14:12:03.870" Score="-3" ViewCount="1382" Body="&lt;p&gt;I want to scrape some data from a website. &#xA;I have used import.io but still not much satisfied.. can any of you suggest about it.. whats the best tool to get the unstructured data from web&lt;/p&gt;&#xA;" OwnerUserId="867" LastEditorUserId="471" LastEditDate="2014-08-21T11:52:35.660" LastActivityDate="2015-01-06T01:26:41.587" Title="Looking for Web scraping tool for unstructured data" Tags="|tools|crawling|" AnswerCount="4" CommentCount="2" ClosedDate="2015-05-26T06:36:22.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="1008" PostTypeId="2" ParentId="1007" CreationDate="2014-08-20T15:34:00.830" Score="3" Body="&lt;p&gt;Try BeautifulSoup - &lt;a href=&quot;http://www.crummy.com/software/BeautifulSoup/&quot; rel=&quot;nofollow&quot;&gt;http://www.crummy.com/software/BeautifulSoup/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the website &quot;Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping.&quot;&#xA;I have no personally used it, but it often comes up in regards to a nice library for scraping. Here's a blog post on using it to scrape Craigslist &lt;a href=&quot;http://www.gregreda.com/2014/07/27/scraping-craigslist-for-tickets/&quot; rel=&quot;nofollow&quot;&gt;http://www.gregreda.com/2014/07/27/scraping-craigslist-for-tickets/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="403" LastActivityDate="2014-08-20T15:34:00.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1009" PostTypeId="2" ParentId="997" CreationDate="2014-08-20T15:53:29.403" Score="9" Body="&lt;p&gt;If you have R and the &lt;code&gt;spacetime&lt;/code&gt; package then you are only &lt;code&gt;data(package=&quot;spacetime&quot;)&lt;/code&gt; away from a list of space-time data sets bundled with the package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data sets in package ‘spacetime’:&#xA;&#xA;DE_NUTS1 (air)          Air quality data, rural background PM10 in&#xA;                        Germany, daily averages 1998-2009&#xA;fires                   Northern Los Angeles County Fires&#xA;rural (air)             Air quality data, rural background PM10 in&#xA;                        Germany, daily averages 1998-2009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data(fires)&#xA;&amp;gt; str(fires)&#xA;'data.frame':   313 obs. of  3 variables:&#xA; $ Time: int  5863 5870 6017 6018 6034 6060 6176 6364 6366 6372 ...&#xA; $ X   : num  63.9 64.3 64.1 64 64.4 ...&#xA; $ Y   : num  19.4 20.1 19.7 19.8 20.3 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-08-20T15:53:29.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1010" PostTypeId="2" ParentId="1007" CreationDate="2014-08-20T19:08:39.807" Score="2" Body="&lt;p&gt;You don't mention what language you're programming in (please consider adding it as a tag), so general help would be to seek out a HTML parser and use that to pull the data. Some web sites can have simply awful HTML code and can be very difficult to scrape, and just when you think you have it...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A HTML parser will parse all the html and allow you to access it in a structured sort of way, whether that's from an array, an object etc.&lt;/p&gt;&#xA;" OwnerUserId="2861" LastActivityDate="2014-08-20T19:08:39.807" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1013" PostTypeId="1" AcceptedAnswerId="1037" CreationDate="2014-08-20T21:12:49.927" Score="2" ViewCount="188" Body="&lt;p&gt;I've just started reading about AB testing, as it pertains to optimizing website design.  I find it interesting that most of the methods assume that changes to the layout and appearance are independent of each other.  I understand that the most common method of optimization is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Multi-armed_bandit&quot; rel=&quot;nofollow&quot;&gt;'multi-armed bandit'&lt;/a&gt; procedure.  While I grasp the concept of it, it seems to ignore the fact that changes (changes to the website in this case) are not independent to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if company is testing the placement and color of the logo on the website, they find the optimal color first then the optimal placement.  Not that I'm some expert on human psychology, but shouldn't these be related? Can the multi-armed bandit method be efficiently used in this case or more complicated cases?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first instinct is to say no.  On that note, why haven't people used heuristic algorithms to optimize over complicated AB testing sample spaces?  For an example, I thought someone might have used a genetic algorithm to optimize a website layout, but I can find no examples of something like this out there. This leads me to believe that I'm missing something important in my understanding of AB testing as it applies to website optimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why isn't heuristic optimization used on more complicated websites?&lt;/p&gt;&#xA;" OwnerUserId="375" LastActivityDate="2014-08-25T16:52:44.817" Title="Using Heuristic Methods for AB Testing" Tags="|optimization|consumerweb|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1015" PostTypeId="1" CreationDate="2014-08-21T06:31:50.197" Score="2" ViewCount="254" Body="&lt;p&gt;I have installed &lt;a href=&quot;https://github.com/Factual/drake&quot; rel=&quot;nofollow&quot;&gt;Drake&lt;/a&gt; on Windows 7 64-bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using JDK 1.7.0_51.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried both using the pre-compiled jar file and&#xA;compiling from the Clojure source using &lt;a href=&quot;https://github.com/technomancy/leiningen&quot; rel=&quot;nofollow&quot;&gt;leiningen&lt;/a&gt;.&#xA;The resulting Drake version is 0.1.6, the current development version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When running Drake, I get the current version number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I tried to go through &lt;a href=&quot;https://github.com/Factual/drake/wiki/Tutorial&quot; rel=&quot;nofollow&quot;&gt;the tutorial&lt;/a&gt;. The command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;java -jar drake.jar  -w .\workflow.d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;results in the following Exception:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;java.lang.Exception: no input data found in locations: D:\tools\drake\in.c&#xA;sv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even though the file exists and has text inside it. &#xA;The same scenario works in a similar installation on Ubuntu 12.04.&#xA;Am I doing something wrong, or is this a Windows-specific bug?&lt;/p&gt;&#xA;" OwnerUserId="895" LastActivityDate="2014-08-21T06:31:50.197" Title="Making Factual drake work on Windows 7 64-bit" Tags="|tools|" AnswerCount="0" CommentCount="0" ClosedDate="2020-10-19T04:59:24.737" ContentLicense="CC BY-SA 3.0" />
  <row Id="1017" PostTypeId="1" CreationDate="2014-08-21T10:13:54.130" Score="3" ViewCount="128" Body="&lt;p&gt;I'm studying reinforcement learning in order to implement a kind of time series pattern analyzer such as market.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most examples I have seen are based on the maze environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in real market environment, the signal changes endlessly as time passes and I can not guess how can I model environment and states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another question is about buy-sell modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume that the agent randomly buy at time $t$ and sell at time $t + \alpha$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's simple to calculate reward.&#xA;The problem is how can I model $Q$ matrix and how can I model signals between buy and sell actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you share some source code or guidance for similar situation?&lt;/p&gt;&#xA;" OwnerUserId="3030" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:33.730" LastActivityDate="2016-01-18T14:36:33.730" Title="How can I model open environment in reinforcement learning?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="0" CommentCount="2" ClosedDate="2016-03-03T18:00:52.860" ContentLicense="CC BY-SA 3.0" />
  <row Id="1019" PostTypeId="2" ParentId="810" CreationDate="2014-08-22T09:03:13.333" Score="9" Body="&lt;p&gt;I think it always depends on the scenario. Using a representative data set is not always the solution. Assume that your training set has 1000 negative examples and 20 positive examples. Without any modification of the classifier, your algorithm will tend to classify all new examples as negative. In some scenarios this is O.K. But in many cases the costs of missing postive examples is high so you have to find a solution for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such cases you can use a cost sensitive machine learning algorithm. For example in the case of medical diagnosis data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary: Classification errors do not have the same cost! &lt;/p&gt;&#xA;" OwnerUserId="979" LastEditorUserId="10806" LastEditDate="2017-07-25T23:18:26.250" LastActivityDate="2017-07-25T23:18:26.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1020" PostTypeId="1" CreationDate="2014-08-22T14:01:23.640" Score="4" ViewCount="84" Body="&lt;p&gt;I am hoping to model the characteristics of the users of a specific page on Facebook, which has roughly 2 million likes. I have been looking at the Facebook SDK/API, but I can't really see if what I would like to do is possible. It seems that the users share quite different amounts of data so I probably discard a lot of users and only use the ones with a quite open public profile. I would like to have the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) See the individuals that have 'liked' the page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) See the list of friends for each person that have 'liked' the page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) See gender for each person (optional)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) See other pages that each person has liked (optional)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone tell me if it is possible to get this data? As mentioned earlier it is okay if I discard data for users that don't like to share this data.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="21" LastEditDate="2014-08-23T09:22:19.073" LastActivityDate="2014-10-22T09:51:35.993" Title="Available data about 'likers' as a page on Facebook" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1021" PostTypeId="1" CreationDate="2014-08-22T15:59:07.097" Score="10" ViewCount="420" Body="&lt;p&gt;I have thousands of lists of strings, and each list has about 10 strings. Most strings in a given list are very similar, though some strings are (rarely) completely unrelated to the others and some strings contain irrelevant words. They can be considered to be noisy variations of a canonical string. I am looking for an algorithm or a library that will convert each list into this canonical string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is one such list.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Star Wars: Episode IV A New Hope | StarWars.com&lt;/li&gt;&#xA;&lt;li&gt;Star Wars Episode IV - A New Hope (1977)&lt;/li&gt;&#xA;&lt;li&gt;Star Wars: Episode IV - A New Hope - Rotten Tomatoes&lt;/li&gt;&#xA;&lt;li&gt;Watch Star Wars: Episode IV - A New Hope Online Free&lt;/li&gt;&#xA;&lt;li&gt;Star Wars (1977) - Greatest Films&lt;/li&gt;&#xA;&lt;li&gt;[REC] 4 poster promises death by outboard motor - SciFiNow&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For this list, any string matching the regular expression &lt;code&gt;^Star Wars:? Episode IV (- )?A New Hope$&lt;/code&gt; would be acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at Andrew Ng's course on Machine Learning on Coursera, but I was not able to find a similar problem.&lt;/p&gt;&#xA;" OwnerUserId="3047" LastEditorUserId="3047" LastEditDate="2014-08-25T08:11:49.307" LastActivityDate="2014-08-25T08:11:49.307" Title="Extract canonical string from a list of noisy strings" Tags="|nlp|similarity|information-retrieval|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1022" PostTypeId="2" ParentId="1020" CreationDate="2014-08-22T18:27:04.377" Score="3" Body="&lt;p&gt;Want to wish you good luck. Some time ago faced with the same problem, but didn't find any satisfying solution. &#xA;First of all, there is no way to get list of users, who &quot;liked&quot; a particular page. Even, if you are an administrator of this page (I was). One only can get list of last 3 or 5 hundred users. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Friendships data for most of the users is also inaccessible. Looks like gender is the only thing from your list, that you can get. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data about pages, that exact user &quot;likes&quot;, should be available (as it's written in docs), but in reality, through API you can collect something only for friends and FoF. Even though this data is available through web interface. So the only way is to try dirty trick with parsing and scraping (but remember, that I didn't advise it ;) ). &lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2014-08-22T18:27:04.377" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1023" PostTypeId="2" ParentId="1021" CreationDate="2014-08-23T09:19:08.577" Score="4" Body="&lt;p&gt;As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second phrase I would do a majority voting. Assuming the 3 sentences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Star Wars: Episode IV A New Hope | StarWars.com&lt;/li&gt;&#xA;&lt;li&gt;Star Wars Episode IV - A New Hope (1977)&lt;/li&gt;&#xA;&lt;li&gt;Star Wars: Episode IV - A New Hope - Rotten Tomatoes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would go through the tokens one by one. We start by &quot;Star&quot;. It wins as all the string start with it. &quot;Wars&quot; will also win. The next one is &quot;:&quot;. It will also win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the tokens will ein in majority voting till &quot;Hope&quot;. The next token after &quot;Hope&quot; will be either &quot;|&quot;, or &quot;(&quot; or &quot;-&quot;. None of the will win in majority voting so I will stop here!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another solution would be probably to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Longest_common_subsequence_problem&quot; rel=&quot;nofollow&quot;&gt;Longest common subsequence&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said I have not though about it much. So there might be much more better solutions to your problem :-)&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2014-08-23T09:19:08.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1024" PostTypeId="1" AcceptedAnswerId="1030" CreationDate="2014-08-23T13:47:01.907" Score="3" ViewCount="199" Body="&lt;p&gt;I've fit a GLM (Poisson) to a data set where one of the variables is categorical for the year a customer bought a product from my company, ranging from 1999 to 2012. There's a linear trend of the coefficients for the values of the variable as the year of sale increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any problem with trying to improve predictions for 2013 and maybe 2014 by extrapolating to get the coefficients for those years?&lt;/p&gt;&#xA;" OwnerUserId="1241" LastEditorUserId="21" LastEditDate="2014-08-26T11:36:01.420" LastActivityDate="2014-08-26T11:36:01.420" Title="Extrapolating GLM coefficients for year a product was sold into future years?" Tags="|statistics|glm|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1025" PostTypeId="1" CreationDate="2014-08-23T13:56:43.813" Score="10" ViewCount="666" Body="&lt;p&gt;I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, mobility, pawn-structure and trapped pieces etc..... My evaluation function is derived from the &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(p) = w_1 \cdot \text{material} + w_2 \cdot \text{kingsafety} + w_3 \cdot \text{mobility} + w_4 \cdot \text{pawn-structure} + w_5 \cdot \text{trapped pieces}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w$ is the weight assigned to each feature. At this point i want to tune the weights of my evaluation function using temporal difference, where the agent plays against itself and in the process gather training data from its environment (which is a form of reinforcement learning). I have read some books and articles in order to have an insight on how to implement this in Java but they seem to be theoretical rather than practical. I need a detailed explanation and pseudo codes on how to automatically tune the weights of my evaluation function based on previous games.&lt;/p&gt;&#xA;" OwnerUserId="3052" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:50.730" LastActivityDate="2016-03-29T04:52:04.283" Title="implementing temporal difference in chess" Tags="|machine-learning|algorithms|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1026" PostTypeId="2" ParentId="810" CreationDate="2014-08-23T14:15:58.660" Score="3" Body="&lt;p&gt;I think there are two separate issues to consider: Training time, and prediction accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a simple example : consider you have two classes, that have a multivariate normal distribution. Basically, you need to estimate the respective class means and class covariances.  Now the first thing you care about is your estimate of the difference in the class means: but your performance is limited by the accuracy of the worst estimated mean: it's no good estimating one mean to the 100th decimal place - if the other mean is only estimated to 1 decimal place.  So it's a waste of computing resources to use all the data - you can instead undersample the  more common class AND reweight the classes appropriately. ( those computing resources can then be used exploring different input variables etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the second issue is predictive accuracy: different algorithms use different error metrics, which may or may not agree with your own objectives. For example, logistic regression will penalize overall probability error,  so if  most of your data is from one class, then it will tend to try to improve accurate probability estimates ( e.g. 90 vs 95% probability) of that one class rather than trying to identify the rare class. In that case, you would definitely want to try to reweight to emphasize the rare class ( and subsequently adjust the estimate [by adjusting the bias term] to get the probability estimates realigned)&lt;/p&gt;&#xA;" OwnerUserId="1256" LastEditorUserId="1256" LastEditDate="2016-11-29T10:18:41.597" LastActivityDate="2016-11-29T10:18:41.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1027" PostTypeId="2" ParentId="1025" CreationDate="2014-08-23T15:25:44.903" Score="2" Body="&lt;p&gt;A first remark, you should watch 'Wargames' to know what you're getting yourself into.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want is f(p) such that f(p) is as close as possible to strength of position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very simple solution using genetic algo would be to setup 10000 players with different weights and see which wins. Then keep the top 1000 winners' weight, copy them 10 times, alter them slightly to explore weight space, and run the simulation again. That's standard GA, given a functional form, what are the best coefficients for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another solution is to extract the positions, so you have a table '(material, kingsafety, mobility, pawn-structure, trappedpieces) -&gt; goodness of position' where goodness of position is some objective factor (outcome win/lose computed using simulations above or known matches, depth of available tree, number of moves under the tree where one of the 5 factors gets better. You can then try different functional forms for your f(p), regression, svm.&lt;/p&gt;&#xA;" OwnerUserId="3053" LastActivityDate="2014-08-23T15:25:44.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1028" PostTypeId="1" AcceptedAnswerId="2315" CreationDate="2014-08-23T16:54:06.380" Score="38" ViewCount="56916" Body="&lt;p&gt;I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when increasing the number of trees in the forest, but it seems that there is not consensus about this. This is creating me quite some confusion about the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe someone more expert than me can give me a more concrete answer or point me in the right direction to better understand the problem.&lt;/p&gt;&#xA;" OwnerUserId="3054" LastActivityDate="2021-02-08T02:11:50.497" Title="Do Random Forest overfit?" Tags="|machine-learning|random-forest|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1029" PostTypeId="1" CreationDate="2014-08-23T19:34:09.417" Score="-3" ViewCount="1120" Body="&lt;p&gt;Which one will be the dominating programming language for next 5 years for analytics , machine learning . R verses python verses SAS. Advantage and disadvantage.&lt;/p&gt;&#xA;" OwnerUserId="3057" LastActivityDate="2014-08-24T11:03:08.773" Title="Which one will be the dominating programming language for next 5 years for analytics , machine learning . R or python or SAS" Tags="|machine-learning|r|python|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ClosedDate="2014-08-26T11:36:17.830" ContentLicense="CC BY-SA 3.0" />
  <row Id="1030" PostTypeId="2" ParentId="1024" CreationDate="2014-08-23T20:05:38.700" Score="4" Body="&lt;p&gt;I believe that this is a case for applying &lt;em&gt;time series analysis&lt;/em&gt;, in particular &lt;em&gt;time series forecasting&lt;/em&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Time_series&lt;/a&gt;). Consider the following resources on &lt;strong&gt;time series regression&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471363553.html&quot; rel=&quot;nofollow&quot;&gt;http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471363553.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stats.uwo.ca/faculty/aim/tsar/tsar.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stats.uwo.ca/faculty/aim/tsar/tsar.pdf&lt;/a&gt; (especially section&#xA;4.6)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/abs/0802.0219&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/0802.0219&lt;/a&gt; (Bayesian approach)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2014-08-23T20:37:43.057" LastActivityDate="2014-08-23T20:37:43.057" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="1031" PostTypeId="2" ParentId="1029" CreationDate="2014-08-24T00:46:03.813" Score="2" Body="&lt;p&gt;There is a great &lt;a href=&quot;http://blog.revolutionanalytics.com/2014/01/in-data-scientist-survey-r-is-the-most-used-tool-other-than-databases.html&quot; rel=&quot;nofollow&quot;&gt;survey&lt;/a&gt; published by O'Reilly collected at Strata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that SAS is not widely popular, and there is no reason why that should change at this point. One can rule that out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is barely ahead of Python, 43% vs 41%. You can find many blogs expressing the rise of Python in data science. I would go with Python in the near future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But 5 years is a very long time. I think Golang will steal a lot of developers from Python in general. This might spill over to data science usage as well. Code can be written to execute in parallel very easily, which makes it a perfect vehicle for Big Data processing. &lt;a href=&quot;http://julialang.org/&quot; rel=&quot;nofollow&quot;&gt;Julia's&lt;/a&gt; benchmarks for technical computing are even more impressive, and you can have iPython like stuff with iJulia. Hence Python is likely to lose some steam to both. But there are ways to call Julia functions from R and Python, so you can experiment using best sides of each.&lt;/p&gt;&#xA;" OwnerUserId="3051" LastActivityDate="2014-08-24T00:46:03.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1032" PostTypeId="2" ParentId="1028" CreationDate="2014-08-24T08:22:23.497" Score="12" Body="&lt;p&gt;You may want to check &lt;a href=&quot;https://stats.stackexchange.com/&quot;&gt;cross-validated&lt;/a&gt; - a stachexchange website for many things, including machine learning.&lt;/p&gt;&#xA;&lt;p&gt;In particular, this question (with exactly same title) has already been answered multiple times. Check &lt;a href=&quot;https://stats.stackexchange.com/search?q=random%20forest%20overfit&quot;&gt;these links&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But I may give you the short answer to it: yes, it does overfit, and sometimes you need to control the complexity of the trees in your forest, or even prune when they grow too much - but this depends on the library you use for building the forest. E.g. in &lt;code&gt;randomForest&lt;/code&gt; in R you can only control the complexity&lt;/p&gt;&#xA;" OwnerUserId="816" LastEditorUserId="98307" LastEditDate="2020-08-20T18:50:02.613" LastActivityDate="2020-08-20T18:50:02.613" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="1033" PostTypeId="2" ParentId="1029" CreationDate="2014-08-24T11:03:08.773" Score="4" Body="&lt;p&gt;Due to the very Big increase in &lt;strong&gt;Big Data&lt;/strong&gt; (pun intended) and the desire for robust stable scalable applications I actually believe it to be &lt;strong&gt;Scala&lt;/strong&gt;.  &lt;strong&gt;Spark&lt;/strong&gt; will inevitably become the main Big Data Machine Learning tool, and it's main API is in Scala.  Furthermore you simply cannot build a product with scripting languages like Python and R, one can only experiment with these languages.  What Scala brings is a way to BOTH experiment and produce a product.  More reasons&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Think functionally - write faster code and more readable code&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Scala means the end of the two team development cycle. So better product ownership, more agile cross functional teams, and half as many employees required to make a product as we will no longer need both a &quot;research&quot; team and an engineering team, Data Scientists will be able to do both.  This is because Scala is;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A production quality language - static typing, but with the flexibility of dynamic typing due to implicits&lt;/li&gt;&#xA;&lt;li&gt;Interoperable with rest of Java world (so Apache Commons Math, Databases, Cassandra, HBase, HDFS, Akka, Storm, many many databases, more spark components (e.g. graphx, SparkStreaming)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Step into Spark code easily and understand it, also helps with debugging&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Scala is awesome:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Amazing IDE support due to static typing&lt;/li&gt;&#xA;&lt;li&gt;Property based tests with ScalaCheck - insane unit testing&lt;/li&gt;&#xA;&lt;li&gt;Very concise language&lt;/li&gt;&#xA;&lt;li&gt;Suits mathematicians perfectly (especially Pure Mathematicians)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A little more efficient as compiled not interpreted&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Python Spark API sits on Scala API and therefore will always be behind Scala API&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Much easier to do Mathematics in Scala as it's a Scalable Language where one can easily define DSLs and due to being so functional&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Akka - another way other than storm to do High Velocity&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pimp my library pattern makes adding methods to Spark RDDs really easy&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2668" LastActivityDate="2014-08-24T11:03:08.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1034" PostTypeId="1" CreationDate="2014-08-24T17:09:40.510" Score="13" ViewCount="4382" Body="&lt;p&gt;I am exploring different types of parse tree structures. The two widely known parse tree structures are &#xA;a) Constituency based parse tree and &#xA;b) Dependency based parse tree structures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to use generate both types of parse tree structures using Stanford NLP package. However, I am not sure how to use these tree structures for my classification task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For e.g If I want to do sentiment analysis and want to categorize text into positive and negative classes, what features can I derive from parse tree structures for my classification task?&lt;/p&gt;&#xA;" OwnerUserId="3064" LastActivityDate="2017-04-20T09:20:31.550" Title="What features are generally used from Parse trees in classification process in NLP?" Tags="|machine-learning|nlp|feature-selection|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1035" PostTypeId="2" ParentId="1021" CreationDate="2014-08-24T22:02:01.050" Score="3" Body="&lt;p&gt;First compute the edit distance between all pairs of strings.  See &lt;a href=&quot;http://en.wikipedia.org/wiki/Edit_distance&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Edit_distance&lt;/a&gt; and &lt;a href=&quot;http://web.stanford.edu/class/cs124/lec/med.pdf&quot; rel=&quot;nofollow&quot;&gt;http://web.stanford.edu/class/cs124/lec/med.pdf&lt;/a&gt;.  Then exclude any outliers strings based on some distance threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With remaining strings, you can use the distance matrix to identify the most central string.  Depending on the method you use, you might get ambiguous results for some data. No method is perfect for all possibilities.  For your purposes, all you need is some heuristic rules to resolve ambiguities -- i.e. pick two or more candidates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you don't want to pick &quot;most central&quot; from your list of strings, but instead want to generate a regular expression that captures the pattern common to all the non-outlier strings.  One way to do this is to synthesize a string that is equidistant from all the non-outlier strings.  You can work out the required edit distance from the matrix, and then you'd randomly generate regular using those distances as constraints.  Then you'd test candidate regular expressions and accept the first one that fits the constraints and also accepts all the strings in your non-outlier list.  (Start building regular expressions from longest common substring lists, because those are non-wildcard characters.)&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-08-24T22:02:01.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1036" PostTypeId="1" CreationDate="2014-08-25T00:28:09.003" Score="7" ViewCount="1220" Body="&lt;p&gt;I am exploring how to model a data set using normal distributions with both mean and variance defined as linear functions of independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like N ~ (f(x), g(x)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generate a random sample like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def draw(x):&#xA;    return norm(5 * x + 2, 3 *x + 4).rvs(1)[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I want to retrieve 5, 2 and 4 as the parameters for my distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I generate my sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;smp = np.zeros((100,2))&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(0, len(smp)):&#xA;    smp[i][0] = i&#xA;    smp[i][1] = draw(i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The likelihood function is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def lh(p):&#xA;    p_loc_b0 = p[0]&#xA;    p_loc_b1 = p[1]&#xA;    p_scl_b0 = p[2]&#xA;    p_scl_b1 = p[3]&#xA;&#xA;    l = 1&#xA;    for i in range(0, len(smp)):&#xA;        x = smp[i][0]&#xA;        y = smp[i][1]&#xA;        l = l * norm(p_loc_b0 + p_loc_b1 * x, p_scl_b0 + p_scl_b1 * x).pdf(y)&#xA;&#xA;    return -l&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the parameters for the linear functions used in the model are given in the p 4-variable vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using scipy.optimize, I can solve for the MLE parameters using an extremely low xtol, and already giving the solution as the starting point:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fmin(lh, x0=[2,5,3,4], xtol=1e-35)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which does not work to well:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Warning: Maximum number of function evaluations has been exceeded.&#xA;array([ 3.27491346,  4.69237042,  5.70317719,  3.30395462])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Raising the xtol to higher values does no good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i try using a starting solution far from the real solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; fmin(lh, x0=[1,1,1,1], xtol=1e-8)&#xA;Optimization terminated successfully.&#xA;         Current function value: -0.000000&#xA;         Iterations: 24&#xA;         Function evaluations: 143&#xA;array([ 1.,  1.,  1.,  1.])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which makes me think:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PDF are largely clustered around the mean, and have very low gradients only a few standard deviations away from the mean, which must be not too good for numerical methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how does one go about doing these kind of numerical estimation in functions where gradient is very near to zero away from the solution?&lt;/p&gt;&#xA;" OwnerUserId="3068" LastActivityDate="2020-08-02T14:02:59.000" Title="How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?" Tags="|python|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1037" PostTypeId="2" ParentId="1013" CreationDate="2014-08-25T03:01:09.837" Score="3" Body="&lt;p&gt;If I understand you question correctly, there are two reasons why genetic algorithm might not a good idea for optimizing website features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Feedback data is coming in too slow, say once a day, genetic algorithm might take a while to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) In the process of testing genetic algorithm will probably come up with combinations that are 'strange' and that might not be the risk the company wants to take.&lt;/p&gt;&#xA;" OwnerUserId="3070" LastEditorUserId="3070" LastEditDate="2014-08-25T16:52:44.817" LastActivityDate="2014-08-25T16:52:44.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1038" PostTypeId="1" CreationDate="2014-08-25T09:53:13.823" Score="1" ViewCount="532" Body="&lt;p&gt;I have created external table in Hive in the hdfs path 'hdfs://localhost.localdomain:8020/user/hive/training'. If I apply describe command I can find the table path as shown below. But when I browse through the namenode web page, the table name does not showing up in the path.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hive&amp;gt; describe extended testtable4;&#xA;OK&#xA;firstname   string  &#xA;lastname    string  &#xA;address string  &#xA;city    string  &#xA;state   string  &#xA;country string  &#xA;&#xA;    ***Detailed Table Information  Table(tableName:testtable4, dbName:default, owner:cloudera, createTime:1408765301, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:firstname, type:string, comment:null), FieldSchema(name:lastname, type:string, comment:null), FieldSchema(name:address, type:string, comment:null), FieldSchema(name:city, type:string, comment:null), FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, type:string, comment:null)], location:hdfs://localhost.localdomain:8020/user/hive/training, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=,, field.delim=,, line.delim=    &#xA;    }), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1408765301}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)       &#xA;    Time taken: 0.7 seconds***&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1314" LastActivityDate="2014-08-25T09:53:13.823" Title="Hive External table does not showing in Namenode (Cloudera-QuickstartVm)" Tags="|bigdata|" AnswerCount="0" CommentCount="1" ClosedDate="2020-10-19T01:55:08.660" ContentLicense="CC BY-SA 3.0" />
  <row Id="1039" PostTypeId="2" ParentId="1034" CreationDate="2014-08-25T11:29:35.047" Score="0" Body="&lt;p&gt;I think dependencies can be used to improve the accurary of your sentiment classifier. Consider the following examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E1: Bill is not a scientist&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and assume that the token &quot;scientist&quot; has a positive sentiment in a specific domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing the dependency neg(scientist, not) we can see that the example above has a negative sentiment. Without knowing this dependency we would probably classify the sentence as positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another types of dependencies can be used probably in the same way to improve the accuracy of the classifiers.&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2014-08-25T11:29:35.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1041" PostTypeId="2" ParentId="1024" CreationDate="2014-08-26T07:14:54.767" Score="6" Body="&lt;p&gt;If you suspect your response is linear with year, then put year in as a numeric term in your model rather than a categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extrapolation is then perfectly valid based on the usual assumptions of the GLM family. Make sure you correctly get the errors on your extrapolated estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just extrapolating the parameters from a categorical variable is wrong for a number of reasons. The first one I can think of is that there may be more observations in some years than others, so any linear extrapolation needs to weight those year's estimates more. Just eyeballing a line - or even fitting a line to the coefficients - won't do this. &lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2014-08-26T07:20:52.097" LastActivityDate="2014-08-26T07:20:52.097" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1042" PostTypeId="1" CreationDate="2014-08-26T07:33:40.080" Score="4" ViewCount="139" Body="&lt;p&gt;I am seeking a basic list of key data analysis methods used for studying social media platforms online. Are there such key methods, or does this process generally vary according to topic? And is there a standard order in which these methods are applied?(The particular context I'm interested in is how the news is impacting on social media)&lt;/p&gt;&#xA;" OwnerUserId="3058" LastEditorUserId="3058" LastEditDate="2014-08-26T17:25:29.930" LastActivityDate="2014-09-29T17:46:58.617" Title="Studying social media platforms - key data analysis methods?" Tags="|social-network-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1044" PostTypeId="1" CreationDate="2014-08-26T14:53:40.647" Score="-6" ViewCount="304" Body="&lt;p&gt;As what I described in the title, we are especially interested in those for dealing with big data----ts efficiency and stability, and used in industry not in experiment or university. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3097" LastActivityDate="2014-08-28T11:24:38.743" Title="which programming language has a large library that can do machine learning algorithm, R, matlab or python" Tags="|machine-learning|bigdata|data-mining|python|r|" AnswerCount="2" CommentCount="3" ClosedDate="2014-09-16T18:35:02.173" ContentLicense="CC BY-SA 3.0" />
  <row Id="1045" PostTypeId="1" CreationDate="2014-08-26T21:37:12.107" Score="2" ViewCount="103" Body="&lt;p&gt;Have anyone used Shark as repository from resulting datasets from Apache Spark?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm starting some tests with Spark and read about this database tecnology. Have anyone been using it?&lt;/p&gt;&#xA;" OwnerUserId="3050" LastActivityDate="2014-08-26T21:37:12.107" Title="Using Shark with Apache Spark" Tags="|apache-hadoop|" AnswerCount="0" CommentCount="7" ClosedDate="2020-10-19T02:22:23.943" ContentLicense="CC BY-SA 3.0" />
  <row Id="1046" PostTypeId="2" ParentId="492" CreationDate="2014-08-26T21:51:07.247" Score="3" Body="&lt;p&gt;Try &lt;a href=&quot;http://deeplearning4j.org/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. This has an implementation of Word2Vec used instead of Bag of Words for NER and other NLP tasks.&lt;/p&gt;&#xA;" OwnerUserId="3100" LastEditorUserId="98307" LastEditDate="2020-08-02T12:40:00.340" LastActivityDate="2020-08-02T12:40:00.340" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="1047" PostTypeId="2" ParentId="1002" CreationDate="2014-08-26T22:48:16.617" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;SPSS is a great tool, but you can accomplish a great deal with resources that you already have on your computer, like Excel, or that are free, like the R-project. Although these tools are powerful, and can help you identify patterns, you'll need to have a firm grasp of your data before running analyses (I'd recommend running descriptive statistics on your data, and exploring the data with graphs to make sure everything is looking normal). In other words, the tool that you use won't offer a &quot;silver bullet&quot;, because the output will only be as valuable as the input (you know the saying... &quot;garbage in, garbage out&quot;). Much of what I'm saying has already been stated in the reply by Aleksandr - spot on. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R can be challenging for those of us who aren't savvy with coding, but the free resources associated with R and its packages are abundant. If you practice learning the program, you'll quickly gain traction. Again, you'll need to be familiar with your data and the analyses you want to run anyway, and that fact remains regardless of the statistical tools you utilize. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'd begin by getting super familiar with my data (follow the steps outlined in the reply from Aleksandr, for starters). You might consider picking up John Foreman's book called Data Smart. It's a hands-on book, as John provides datasets and you follow along with his examples (using Excel) to learn various ways of navigating and exploring data. For beginners, it's a great resource. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3101" LastActivityDate="2014-08-26T22:48:16.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1048" PostTypeId="2" ParentId="1044" CreationDate="2014-08-27T03:41:20.317" Score="3" Body="&lt;p&gt;I've making some researches last months and I could find more libraries, contente and active community with Python. Actually I'm using it to ETL processes, some minning jobs and to make map/reduce.&lt;/p&gt;&#xA;" OwnerUserId="3050" LastActivityDate="2014-08-27T03:41:20.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1050" PostTypeId="1" AcceptedAnswerId="11966" CreationDate="2014-08-27T13:01:14.643" Score="10" ViewCount="909" Body="&lt;p&gt;&lt;strong&gt;General description of the problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a graph where some vertices are labeled with a type with 3 or 4 possible values. For the other vertices, the type is unknown.&#xA;My goal is to use the graph to predict the type for vertices that are unlabeled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Possible framework&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect this fits into the general framework of label propagation problems, based on my reading of the literature (e.g., see &lt;a href=&quot;http://lvk.cs.msu.su/~bruzz/articles/classification/zhu02learning.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; and &lt;a href=&quot;http://www.csc.ncsu.edu/faculty/samatova/practical-graph-mining-with-R/slides/pdf/Frequent_Subgraph_Mining.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method that is mentioned often is &lt;code&gt;Frequent Subgraph Mining&lt;/code&gt;, which includes algorithms like &lt;code&gt;SUBDUE&lt;/code&gt;,&lt;code&gt;SLEUTH&lt;/code&gt;, and &lt;code&gt;gSpan&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Found in R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only label propagation implementation I managed to find in &lt;code&gt;R&lt;/code&gt; is &lt;code&gt;label.propagation.community()&lt;/code&gt; from the &lt;code&gt;igraph&lt;/code&gt; library. &#xA;However, as the name suggests, it is mostly used to find communities, not for classifying unlabeled vertices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There also seems to be several references to a &lt;code&gt;subgraphMining&lt;/code&gt; library (here for example), but it looks like it is missing from CRAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of a library or framework for the task described?&lt;/p&gt;&#xA;" OwnerUserId="3108" LastEditorUserId="609" LastEditDate="2015-10-25T05:44:40.023" LastActivityDate="2016-05-27T18:36:35.830" Title="Libraries for (label propagation algorithms/frequent subgraph mining) for graphs in R" Tags="|classification|r|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1051" PostTypeId="2" ParentId="1003" CreationDate="2014-08-27T18:33:23.110" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's how to get more than 10 results per query: &lt;a href=&quot;https://support.google.com/customsearch/answer/1361951?hl=en&quot; rel=&quot;nofollow&quot;&gt;https://support.google.com/customsearch/answer/1361951?hl=en&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Google Custom Search and Google Site Search return up to 10 results per query. If you want to display more than 10 results to the user, you can issue multiple requests (using the start=0, start=11 ... parameters) and display the results on a single page. In this case, Google will consider each request as a separate query, and if you are using Google Site Search, each query will count towards your limit.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are other search engine APIs as well (e.g., &lt;a href=&quot;http://datamarket.azure.com/dataset/bing/search&quot; rel=&quot;nofollow&quot;&gt;Bing&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-08-27T18:33:23.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1053" PostTypeId="1" CreationDate="2014-08-28T01:36:40.540" Score="6" ViewCount="1234" Body="&lt;p&gt;I would like to &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/base/html/summary.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;summarize&lt;/a&gt; (as in R) the contents of a CSV (possibly after &lt;a href=&quot;http://www.endmemo.com/program/R/readcsv.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;loading&lt;/a&gt; it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and max of the data in a CSV file for each numeric (integer or real numbers) dimension. The standard deviation would be cool as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also like to generate some plots to visualize the data, for example 3 plots for the 3 pairs of variables that are more correlated (&lt;a href=&quot;http://www.r-tutor.com/elementary-statistics/numerical-measures/correlation-coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;correlation coefficient&lt;/a&gt;) and 3 plots for the 3 pairs of variables that are least correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R requires only a few lines to implement this. Are there any libraries (or tools) that would allow a similarly simple (and efficient if possible) implementation in Java or Scala?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PD: This is a specific use case for a &lt;a href=&quot;https://datascience.stackexchange.com/questions/948/any-clear-winner-for-data-science-in-scala&quot;&gt;previous (too broad) question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1281" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-09-01T08:10:56.967" Title="Summarize and visualize a CSV in Java/Scala?" Tags="|tools|visualization|scala|csv|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1054" PostTypeId="2" ParentId="1036" CreationDate="2014-08-28T06:49:05.210" Score="8" Body="&lt;p&gt;There are several reasons why you are getting erroneous results.&#xA;First, you should consider using log likelihood instead of likelihood. There are numerical issues with multiplying many small numbers(imagine if you had millions of samples you had to multiply millions of small numbers for the lhd). Also taking gradients for optimization methods that require gradients is often easier when you are dealing with the log likelihood. In general, it is good to have an objective which is a sum rather than a product of variables when dealing with optimization problems.&lt;/p&gt;&#xA;&lt;p&gt;Second, fmin is using Nelder-Mead simplex algorithm which has no convergence guarantees according to &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html#scipy.optimize.fmin&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy documentation&lt;/a&gt;. This means the convergence is totally random and you should not expect to find parameters close to the originals. To get around this, I would suggest you to use a gradient based method like stochastic gradient descent or BFGS. Since you know the generative model (rvs are Gaussian distributed) you can write the likelihood and log likelihood as:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/bfsvr.png&quot; alt=&quot;equations&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Where a,b,c and d are your model parameters 5,2,3 and 4 respectively.&#xA;Then take the &lt;a href=&quot;http://en.wikipedia.org/wiki/Gradient&quot; rel=&quot;nofollow noreferrer&quot;&gt;gradient&lt;/a&gt; with respect to [a,b,c,d] and feed that into prime input of fmin_bfgs. Note that due to varying variance what could be solved by just linear regression is now a nastier problem.&lt;/p&gt;&#xA;&lt;p&gt;Finally, you may also want to check Generalized least squares &lt;a href=&quot;http://en.wikipedia.org/wiki/Linear_regression#Least-squares_estimation_and_related_techniques&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Heteroscedasticity&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, which talk about your problem and offer several available solutions.&lt;/p&gt;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="1350" LastEditorUserId="98307" LastEditDate="2020-08-02T14:02:59.000" LastActivityDate="2020-08-02T14:02:59.000" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="1055" PostTypeId="2" ParentId="1044" CreationDate="2014-08-28T11:24:38.743" Score="1" Body="&lt;p&gt;Scala is the only real language that has Big Data at it's core. You have MLLib that sits on Spark, and as Scala is Functional it makes parallel computing really natural.  R, Python and Matlab are not suitable for industry productization, well some would say Python's horrible dynamic typing can be handled a little using special build tools, but really its not type safe and there is no way to solve that problem.&lt;/p&gt;&#xA;" OwnerUserId="2668" LastActivityDate="2014-08-28T11:24:38.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1056" PostTypeId="2" ParentId="985" CreationDate="2014-08-28T16:56:45.577" Score="2" Body="&lt;p&gt;It sounds as if you want to use unsupervized learning to create a training set. Am I right? You use your cluster analysis to determine which docs come from UK, US or Oz -- or which docs are talking about Soccer, Football or Australian football respectively? Then feed those tagged docs into a supervized learning algorithm of some sort?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How well this works will depend entirely on how well you can distinguish UK, US and OZ. I would have thought it would be fairly straightforward to find documents where national origin was known, so that you could build a supervized algorithm for detecting language variant. You wouldn't even need a corpus that talked about football, since dialectical differences show up in other ways that are subject matter independent. (For example, I am clearly from North America, since I just wrote &quot;in ways that are subject matter independent&quot; rather than &quot;Since dialectical differences do not depend on subject matter&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the answer to your question, &quot;can I use unsupervized learning and then supervized learning&quot; is No, if you are looking for supervized learning. If the results of an unsupervized learning algorithm are fed to a supervized learning algorithm, the net result is unsupervized --- there are still no grown-ups in the room. And the classification errors of the resulting process will contain error terms from both stages. You won't get the same performance as you would if you did a SVM with properly tagged training data. This doesn't mean you shouldn't use the method you propose ... it might still work well ... but it won't be a supervized learning algorithm.&lt;/p&gt;&#xA;" OwnerUserId="3077" LastEditorUserId="3077" LastEditDate="2014-08-29T19:26:39.963" LastActivityDate="2014-08-29T19:26:39.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1057" PostTypeId="2" ParentId="1042" CreationDate="2014-08-28T18:53:57.297" Score="1" Body="&lt;p&gt;You might want to try this book &lt;a href=&quot;http://shop.oreilly.com/product/0636920030195.do&quot; rel=&quot;nofollow&quot;&gt;Mining the Social Web&lt;/a&gt; for an overview of different techniques. Obviously, the methods you need will depend on the use case. A lot of people do interesting things with graphs, displaying relationships between users, with respect to certain topics. Or you might simply to a timeline showing how a news topic builds in interest and wanes.&lt;/p&gt;&#xA;" OwnerUserId="3077" LastActivityDate="2014-08-28T18:53:57.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1058" PostTypeId="1" AcceptedAnswerId="1066" CreationDate="2014-08-29T01:36:36.320" Score="-2" ViewCount="242" Body="&lt;p&gt;I was wondering if there is any research or study made to calculate the volume of space is used by all scientific articles. It could be in pdf, txt, compressed, or any other format. Is there even a way to measure it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can some one point me towards realizing this study?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards and thanks.&lt;/p&gt;&#xA;" OwnerUserId="3128" LastEditorUserId="75" LastEditDate="2015-05-26T06:35:34.860" LastActivityDate="2015-05-26T06:35:34.860" Title="How much data space is used by all scientific articles?" Tags="|bigdata|research|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1059" PostTypeId="1" CreationDate="2014-08-29T06:00:53.420" Score="7" ViewCount="1123" Body="&lt;p&gt;I have a question regarding the use of neural network. I am currently working with R (&lt;a href=&quot;http://cran.r-project.org/web/packages/neuralnet/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;neuralnet package&lt;/a&gt;) and I am facing the following issue.&#xA;My testing and validation set are always late with respect to the historical data. Is there a way of correcting the result?&#xA;Maybe something is wrong in my analysis&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I use the daily log return &#xA;&lt;img src=&quot;https://i.stack.imgur.com/A3r16.gif&quot; alt=&quot;r(t) = ln(s(t)/s(t-1))&quot;&gt;&lt;/li&gt;&#xA;&lt;li&gt;I normalise my data with the sigmoid function (sigma and mu computed on my whole set)&lt;/li&gt;&#xA;&lt;li&gt;I train my neural networks with 10 dates and the output is the normalised value that follows these 10 dates.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I tried to add the trend but there is no improvement, I observed 1-2 days late. My process seems ok, what do you think about it?&lt;/p&gt;&#xA;" OwnerUserId="3055" LastEditorUserId="97" LastEditDate="2014-11-07T08:28:10.637" LastActivityDate="2015-04-08T14:15:39.407" Title="Forecasting Foreign Exchange with Neural Network - Lag in Prediction" Tags="|r|neural-network|time-series|forecast|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1060" PostTypeId="2" ParentId="1053" CreationDate="2014-08-29T10:42:14.957" Score="2" Body="&lt;p&gt;Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.&lt;/p&gt;&#xA;" OwnerUserId="2668" LastActivityDate="2014-08-29T10:42:14.957" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1061" PostTypeId="1" CreationDate="2014-08-29T11:40:28.657" Score="1" ViewCount="38" Body="&lt;p&gt;I'm looking for the best solution to manage and host datasets for journalistic pursuits. I am assessing &lt;a href=&quot;https://www.documentcloud.org&quot; rel=&quot;nofollow&quot;&gt;https://www.documentcloud.org&lt;/a&gt; and &lt;a href=&quot;http://datahub.io/&quot; rel=&quot;nofollow&quot;&gt;http://datahub.io/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain the differences between them, or recommend a superior solution?&lt;/p&gt;&#xA;" OwnerUserId="3133" LastActivityDate="2014-08-29T11:40:28.657" Title="Apps to manage/host data sets" Tags="|dataset|optimization|" AnswerCount="0" CommentCount="3" ClosedDate="2020-10-19T04:58:59.613" ContentLicense="CC BY-SA 3.0" />
  <row Id="1062" PostTypeId="2" ParentId="985" CreationDate="2014-08-29T16:19:06.903" Score="5" Body="&lt;p&gt;You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if your data looked like this (in 1D):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AA A AA A A      BBB B B B BB BB BB      AA AA A A AAA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then it may be reasonable to run a clustering algorithm on each class, to obtain &lt;em&gt;two different kinds of A&lt;/em&gt;, and learn two separate classifiers for A1 and A2, and just drop the cluster distinction for the final output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other common unsupervised techniques used include PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your football example, the problem is that the unsupervised algorithm does not know what it should be looking for. Instead of learning to separate american football and soccer, it may just as well decide to cluster on international vs. national games. Or Europe vs. U.S.; which may look like it learned about american football and soccer at first, but it put american soccer into the same cluster as american football, and american football teams in Europe into the Europe cluster... because it does not have &lt;em&gt;guidance&lt;/em&gt; on what structure you are interested in; and the continents &lt;em&gt;are&lt;/em&gt; a valid structure, too!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So usually, I would not blindly assume that unsupervised techniques yield a distrinction that matches your desired result. They can yield &lt;em&gt;any&lt;/em&gt; kind of structure, and you will want to carefully inspect what they found before using it. If you use it blindly, make sure you spend enough time on evaluation (e.g. if the clustering improves your classifier performance, then it probably worked as intended ...)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-08-29T16:19:06.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1063" PostTypeId="2" ParentId="1053" CreationDate="2014-08-30T17:05:23.197" Score="1" Body="&lt;p&gt;If your data is numeric, try loading it into ELKI (Java). With the &lt;code&gt;NullAlgorithm&lt;/code&gt; it will give you scatterplots, histograms and parallel coordinate plots. It's fast in reading the data; only the current Apache Batik-based visualization is slooow because it's using SVG. :-( I'm mostly using it &quot;headless&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also has classes for various statistics (including higher order moments on data streams), but I havn't seen them in the default UI yet.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-08-30T17:05:23.197" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1064" PostTypeId="2" ParentId="89" CreationDate="2014-08-30T18:14:05.577" Score="4" Body="&lt;p&gt;MapReduce is not used in searching. It was used a long time ago to build the index; but it is a batch processing framework, and most of the web does not change all the time, so the newer architectures are all &lt;em&gt;incremental&lt;/em&gt; instead of batch oriented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search in Google will largely work the same it works in Lucene and Elastic Search, except for a lot of fine tuned extra weighting and optimizations. But at the very heart, they will use some form of an &lt;strong&gt;inverted index&lt;/strong&gt;. In other words, they do &lt;em&gt;not&lt;/em&gt; search several terabytes when you enter a search query (even when it is not cached). They likely don't look at the actual documents at all. But they use a lookup table that lists which documents match your query term (with stemming, misspellings, synonyms etc. all preprocessed). They probably retrieve the &lt;em&gt;list&lt;/em&gt; of the top 10000 documents for each word (10k integers - just a few kb!) and compute the best matches from that. Only if there aren't good matches in these lists, they expand to the next such blocks etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Queries for common words can be easily cached; and via preprocessing you can build a list of the top 10k results and then rerank them according to the user profile. There is nothing to be gained by computing an &quot;exact&quot; answer, too. Looking at the top 10k results is likely enough; there is no correct answer; and if a better result somewhere at position 10001 is missed, nobody will know or notice (or care). It likely was already ranked down in preprocessing and would not have made it into the top 10 that is presented to the user at the end (or the top 3, the user actually looks at)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rare terms on the other hand aren't much of a challenge either - one of the lists only contains a few matching documents, and you can immediately discard all others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend reading this article:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;The Anatomy of a Large-Scale Hypertextual Web Search Engine&lt;/strong&gt;&lt;br /&gt;&#xA;  Sergey Brin and Lawrence Page&lt;br /&gt;&#xA;  Computer Science Department, Stanford University, Stanford, CA 94305&lt;br /&gt;&#xA;  &lt;a href=&quot;http://infolab.stanford.edu/~backrub/google.html&quot; rel=&quot;nofollow&quot;&gt;http://infolab.stanford.edu/~backrub/google.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And yes, that's the Google founders who wrote this. It's not the latest state, but it will already work at a pretty large scale.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2014-08-30T18:40:02.403" LastActivityDate="2014-08-30T18:40:02.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1065" PostTypeId="2" ParentId="730" CreationDate="2014-08-30T18:36:07.490" Score="10" Body="&lt;p&gt;State of the art as in: used in practise or worked on in theory?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;APRIORI is used everywhere, except in developing new frequent itemset algorithms. It's easy to implement, and easy to reuse in very different domains. You'll find hundreds of APRIORI implementations of varying quality. And it's easy to get APRIORI wrong, actually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FPgrowth is much harder to implement, but also much more interesting. So from an academic point of view, everybody tries to improve FPgrowth - getting work based on APRIORI accepted will be very hard by now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a good implementation, every algorithm has it's good and it's bad situations in my opinion. A good APRIORI implementation will &lt;em&gt;only&lt;/em&gt; need to scan the database &lt;em&gt;k&lt;/em&gt; times to find all frequent itemsets of length &lt;em&gt;k&lt;/em&gt;. In particular if your data fits into main memory this is cheap. What can kill APRIORI is too many frequent 2-itemsets (in particular when you don't use a Trie and similar acceleration techniques etc.). It works best on large data with a low number of frequent itemsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eclat works on columns; but it needs to read each column much more often. There is some work on diffsets to reduce this work. If your data does not fit into main memory, Eclat suffers probably more than Apriori. By going depth first, it will also be able to return a first interesting result much earlier than Apriori, and you can use these results to adjust parameters; so you need less iterations to find good parameters. But by design, it cannot exploit pruning as neatly as Apriori did.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FPGrowth compresses the data set into the tree. This works best when you have lots of duplicate records. You could probably reap of quite some gains for Apriori and Eclat too if you can presort your data and merge duplicates into weighted vectors. FPGrowth does this at an extreme level. The drawback is that the implementation is much harder; and once this tree does not fit into memory anymore it gets a mess to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for performance results and benchmarks - don't trust them. There are so many things to implement incorrectly. Try 10 different implementations, and you get 10 very different performance results. In particular for APRIORI, I have the impression that most implementations are broken in the sense of missing some of the main contributions of APRIORI... and of those that have these parts right, the quality of optimizations varies a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are actually even papers on how to implement these algorithms efficiently:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Efficient Implementations of Apriori and Eclat.&lt;br /&gt; Christian Borgelt&lt;br /&gt;Workshop of Frequent Item Set Mining Implementations (FIMI 2003, Melbourne, FL, USA).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You may also want to read these surveys on this domain:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Goethals, Bart. &quot;Survey on frequent pattern mining.&quot; Univ. of Helsinki (2003).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ferenc Bodon, A Survey on Frequent Itemset Mining, Technical Report, Budapest University of Technology and Economic, 2006, &lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Frequent Item Set Mining&lt;br /&gt;Christian Borgelt&lt;br /&gt;Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2(6):437-456. 2012&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-08-30T18:36:07.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1066" PostTypeId="2" ParentId="1058" CreationDate="2014-09-01T00:01:58.917" Score="1" Body="&lt;p&gt;Perhaps you are looking to quantify the amount of filespace used by a specific subset of data that we will label as &quot;academic publications.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, to estimate, you could find stats on how many publications are housed at all the leading libraries (JSTOR, EBSCO, AcademicHost, etc) and then get the mean average size of each. Multiply that by the number of articles and whamo, you've got yourself an estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the problem, though: PDF files store the text from string &lt;code&gt;s&lt;/code&gt; differently (in size) than, say, a text document stores that same string. Likewise, a compressed JPEG will store an amount of information &lt;code&gt;i&lt;/code&gt; differently than a non-compressed JPEG. So you see we could have two of the same articles containing the same information &lt;code&gt;i&lt;/code&gt; but taking up different amounts of memory &lt;code&gt;m&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you looking to get a wordcount on the amount of scientific literature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you looking to get an approximation of file system space used to store all academically published content in the world? &lt;/p&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2014-09-01T00:01:58.917" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1067" PostTypeId="2" ParentId="1059" CreationDate="2014-09-01T00:17:56.070" Score="0" Body="&lt;p&gt;It is likely to be very hard to draw any conclusion if you are training with only 10 input samples.  With more data, your diagnosis that the model is predicting lagged values would have more plausibility.  As it stands, it seems pretty likely that your model is just saying that the last observed value is pretty close to correct.  This isn't the same as a real lag model, but it is a very reasonable thing to guess if you haven't seen enough data.&lt;/p&gt;&#xA;" OwnerUserId="3153" LastActivityDate="2014-09-01T00:17:56.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1068" PostTypeId="2" ParentId="1053" CreationDate="2014-09-01T08:10:56.967" Score="0" Body="&lt;p&gt;I'd have a closer look at one of Apache Spark's modules: &lt;a href=&quot;https://spark.apache.org/mllib/&quot; rel=&quot;nofollow&quot; title=&quot;MLlib&quot;&gt;MLlib&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3150" LastActivityDate="2014-09-01T08:10:56.967" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1069" PostTypeId="1" CreationDate="2014-09-02T03:24:12.793" Score="6" ViewCount="239" Body="&lt;p&gt;I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, Decision Tree).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am performing a type of cross validation where I randomly select 67% of the data for use in the training set and 33% of the data for use in the testing set. I perform this for several iterations, say, 20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, from each iteration I am able to generate a confusion matrix and compute a kappa. My question is, what are some ways to aggregate these across the iterations? I am also interested in aggregating accuracy and expected accuracy, among other things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the kappa, accuracy, and expected accuracy, I have just been taking the average up to this point. One of the problems is that when I recompute kappa with the aggregated average and expected average, it is not the same with the aggregated kappa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the confusion matrix, I have been first normalizing the confusion matrix from each iteration and then averaging them, in an attempt to avoid an issue of confusion matrices with different numbers of total cases (which is possible with my cross validation scheme).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I recompute the kappa from this aggregated confusion matrix, it is also different from the previous two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which one is most correct? Is there another way of computing an average kappa that is more correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, and if more concrete examples are needed in order to illustrate my question please let me know.&lt;/p&gt;&#xA;" OwnerUserId="3169" LastEditorUserId="3169" LastEditDate="2016-02-18T05:18:34.363" LastActivityDate="2016-02-18T05:18:34.363" Title="Kappa From Combined Confusion Matrices" Tags="|machine-learning|confusion-matrix|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1070" PostTypeId="1" AcceptedAnswerId="1072" CreationDate="2014-09-02T08:47:38.737" Score="3" ViewCount="1577" Body="&lt;p&gt;I am interested in knowing the differences in &lt;strong&gt;functionality&lt;/strong&gt; between SAP HANA and Exasol. Since this is a bit of an open ended question let me be clear. I am not interested in people debating which is &quot;better&quot; or faster. I am only interested in what each was designed to do so please keep your opinions out of it. I suspect it is a bit like comparing HANA to Oracle Exalytics where there is some overlap but the functionality goals are different. &lt;/p&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2014-09-02T14:01:07.407" Title="SAP HANA vs Exasol" Tags="|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1071" PostTypeId="1" CreationDate="2014-09-02T09:48:08.150" Score="1" ViewCount="63" Body="&lt;p&gt;How is the concept of data different for different disciplines? Obviously, for physicists and sociologists, &quot;data&quot; is something different.&lt;/p&gt;&#xA;" OwnerUserId="3178" LastActivityDate="2014-09-05T16:36:03.227" Title="How is the concept of data different for different disciplines?" Tags="|definitions|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1072" PostTypeId="2" ParentId="1070" CreationDate="2014-09-02T14:01:07.407" Score="1" Body="&lt;p&gt;There's not an enormous difference between what you can do with the two databases, it's more a question of the focus and the way the functionality is implemented and that's where it becomes difficult to explain without using words like &quot;better&quot; and &quot;faster&quot; (and for sure words like &quot;cheaper&quot;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EXASOL was designed for speed and ease of use with Analytical processing and is designed to run on clusters of commodity hardware. SAP is a more complex, aims to do more than &quot;just&quot; Analytical processing and runs only on a range of &quot;approved&quot; hardware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of differences did you have in mind ?&lt;/p&gt;&#xA;" OwnerUserId="3181" LastActivityDate="2014-09-02T14:01:07.407" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1073" PostTypeId="1" CreationDate="2014-09-02T19:17:43.210" Score="10" ViewCount="3829" Body="&lt;p&gt;I am looking for packages (either in python, R, or a standalone package) to perform online learning to predict stock data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found and read about Vowpal Wabbit (&lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot;&gt;https://github.com/JohnLangford/vowpal_wabbit/wiki&lt;/a&gt;),&#xA;which seems to be quite promising but I am wondering if there are any other packages out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="802" LastActivityDate="2022-09-20T18:13:11.997" Title="Libraries for Online Machine Learning" Tags="|machine-learning|online-learning|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1074" PostTypeId="1" CreationDate="2014-09-02T19:29:07.490" Score="6" ViewCount="7717" Body="&lt;p&gt;In SVMs the polynomial kernel is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(scale * crossprod(x, y) + offset)^degree&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do the scale and offset parameters affect the model and what range should they be in? (intuitively please)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the scale and offset for numeric stability only (that's what it looks like to me), or do they influence prediction accuracy as well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can good values for scale and offset be calculated/estimated when the data is known or is a grid search required? The caret package always sets the offset to 1, but it does a grid search for scale. (Why) is an offset of 1 a good value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS.: Wikipedia didn't really help my understanding:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For degree-d polynomials, the polynomial kernel is defined as&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/3stCR.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where x and y are vectors in the input space, i.e. vectors of features&#xA;  computed from training or test samples, &lt;img src=&quot;https://i.stack.imgur.com/34dwi.png&quot; alt=&quot;&quot;&gt; &lt;strong&gt;is a constant trading&lt;/strong&gt;&#xA;  &lt;strong&gt;off the influence of higher-order versus lower-order terms&lt;/strong&gt; in the&#xA;  polynomial. When &lt;img src=&quot;https://i.stack.imgur.com/79cDG.png&quot; alt=&quot;&quot;&gt;, the kernel is called homogeneous.(&lt;strong&gt;A further&lt;/strong&gt;&#xA;  &lt;strong&gt;generalized polykernel divides &lt;img src=&quot;https://i.stack.imgur.com/Ps0qt.png&quot; alt=&quot;&quot;&gt; by a user-specified scalar&lt;/strong&gt;&#xA;  &lt;strong&gt;parameter &lt;img src=&quot;https://i.stack.imgur.com/sjfS1.png&quot; alt=&quot;&quot;&gt;.&lt;/strong&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Neither did ?polydot's explanation in R's help system:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;scale&lt;/strong&gt;: The scaling parameter of the polynomial and tangent kernel is a&#xA;  convenient way of normalizing patterns (&amp;lt;-!?) without the need to modify the&#xA;  data itself&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;offset&lt;/strong&gt;: The offset used in a polynomial or hyperbolic tangent kernel (&amp;lt;- lol thanks)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="676" LastEditorUserId="9667" LastEditDate="2015-05-18T07:34:41.633" LastActivityDate="2015-05-18T07:34:41.633" Title="Polynomial Kernel Parameters in SVMs" Tags="|machine-learning|classification|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1075" PostTypeId="1" CreationDate="2014-09-04T18:13:57.343" Score="4" ViewCount="795" Body="&lt;p&gt;Background:&#xA;I run a product that compares sets of data (data matching and data reconciliation).&#xA;To get the result we need to compare each row in a data set with every N rows on the opposing data set&#xA;Now however we get sets of up to 300 000 rows of data in each set to compare and are getting 90 Billion computations to handle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is this:&#xA;Even though we dont have the data volumes to use Hadoop, we have the computational need for something distributed. Is Hadoop a good choice for us?&lt;/p&gt;&#xA;" OwnerUserId="3203" LastActivityDate="2014-11-15T03:18:13.223" Title="Hadoop for grid computing" Tags="|apache-hadoop|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1076" PostTypeId="2" ParentId="1073" CreationDate="2014-09-05T06:17:28.533" Score="4" Body="&lt;p&gt;You could look at scikit-learn or Orange module in Python.&#xA;Scikit-learn has a SGD classifier and regressor that could do a partial fit data in case of online learning.&#xA;In R, take a look at caret package &lt;/p&gt;&#xA;" OwnerUserId="3211" LastEditorUserId="15527" LastEditDate="2016-01-23T22:19:03.783" LastActivityDate="2016-01-23T22:19:03.783" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1077" PostTypeId="2" ParentId="1075" CreationDate="2014-09-05T11:00:22.053" Score="4" Body="&lt;p&gt;Your job seems like a map-reduce job and hence might be good for Hadoop. Hadoop has a zoo of an ecosystem though. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadoop is a distributed file system. It distributes data on a cluster and because this data is split up it can be analysed in parallel. Out of the box, Hadoop allows you to write map reduce jobs on the platform and this is why it might help with your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following technologies work on Hadoop: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the data can be represented in a table format you might want to check out technologies like &lt;a href=&quot;https://hive.apache.org/&quot; rel=&quot;nofollow&quot;&gt;hive&lt;/a&gt; and &lt;a href=&quot;http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html&quot; rel=&quot;nofollow&quot;&gt;impala&lt;/a&gt;. Impala uses all the distributed memory across a cluster and is very performant while it allows you to still work with a table structure. &lt;/li&gt;&#xA;&lt;li&gt;A more new, but promising alternative might also be &lt;a href=&quot;https://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;spark&lt;/a&gt; which allows for more iterative procedures to be run on the cluster. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Don't underestimate the amount of time setting up and the amount of time needed to understand Hadoop.&lt;/p&gt;&#xA;" OwnerUserId="3213" LastActivityDate="2014-09-05T11:00:22.053" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1078" PostTypeId="1" CreationDate="2014-09-05T12:08:11.347" Score="13" ViewCount="7074" Body="&lt;p&gt;I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than three). Does anyone have a suggested for how I might get started?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the &quot;bag of words&quot; approach feasible here? Later, I could add another classification stage based on document structure (perhaps decision trees).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am somewhat familiar with Mahout and Hadoop, so I prefer Java-based solutions. If needed, I can switch to Scala and/or Spark engine (the ML library).&lt;/p&gt;&#xA;" OwnerUserId="3215" LastEditorUserId="13413" LastEditDate="2015-10-24T00:20:42.453" LastActivityDate="2017-03-25T07:31:56.423" Title="Unstructured text classification" Tags="|machine-learning|classification|text-mining|beginner|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1079" PostTypeId="1" CreationDate="2014-09-05T14:47:52.127" Score="3" ViewCount="3662" Body="&lt;p&gt;I'm searching for data sets for evaluating text retrieval quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TF-IDF is a popular similarity measure, but is it the best choice? And which &lt;em&gt;variant&lt;/em&gt; is the best choice? &lt;a href=&quot;https://lucene.apache.org/core/3_6_1/api/all/org/apache/lucene/search/Similarity.html&quot; rel=&quot;nofollow&quot;&gt;Lucenes Scoring&lt;/a&gt; for example uses IDF^2, and IDF defined as 1+log(numdocs/(docFreq+1)). TF in lucene is defined as sqrt(frequency)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many more variants exist, including &lt;a href=&quot;https://en.wikipedia.org/wiki/Okapi_BM25&quot; rel=&quot;nofollow&quot;&gt;Okapi BM25&lt;/a&gt;, which is used by the &lt;a href=&quot;http://xapian.org/docs/bm25.html&quot; rel=&quot;nofollow&quot;&gt;Xapian search engine&lt;/a&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to study the different variants, and I'm looking for &lt;strong&gt;evaluation data sets&lt;/strong&gt;. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="2920" LastActivityDate="2014-11-12T19:00:05.013" Title="Data sets for evaluating text retrieval quality" Tags="|dataset|text-mining|similarity|information-retrieval|" AnswerCount="1" CommentCount="2" ClosedDate="2014-11-15T14:28:46.633" ContentLicense="CC BY-SA 3.0" />
  <row Id="1080" PostTypeId="1" CreationDate="2014-09-05T16:34:55.170" Score="3" ViewCount="536" Body="&lt;p&gt;I was curious about the ANOVA RBF kernel provided by kernlab package available in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tested it with a numeric dataSet of 34 input variables and one output variable. For each variable I have 700 different values. Comparing with other kernels, I got very bad results with this kernel.&#xA;For example using the simple RBF kernel I could predict with 0,88 R2 however with the anova RBF I could only get 0,33 R2.&#xA;I thought that ANOVA RBF would be a very good kernel. Any thoughts? Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(100) #use the same seed to train different models&#xA;svrFitanovaacv &amp;lt;- train(R ~ .,&#xA;                       data = trainSet,&#xA;                       method = SVManova,&#xA;                       preProc = c(&quot;center&quot;, &quot;scale&quot;),&#xA;                       trControl = ctrl, tuneLength = 10) #By default, RMSE and R2 are computed for regression (in all cases, selects the tunning and cross-val model with best value) , metric = &quot;ROC&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;define custom model in caret package:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(caret)&#xA;#RBF ANOVA KERNEL&#xA;SVManova &amp;lt;- list(type = &quot;Regression&quot;, library = &quot;kernlab&quot;, loop = NULL)&#xA;prmanova &amp;lt;- data.frame(parameter = c(&quot;C&quot;, &quot;sigma&quot;, &quot;degree&quot;, &quot;epsilon&quot;),&#xA;                     class = rep(&quot;numeric&quot;, 4),&#xA;                     label = c(&quot;Cost&quot;, &quot;Sigma&quot;, &quot;Degree&quot;, &quot;Epsilon&quot;))&#xA;SVManova$parameters &amp;lt;- prmanova&#xA;svmGridanova &amp;lt;- function(x, y, len = NULL) {&#xA;library(kernlab)&#xA;sigmas &amp;lt;- sigest(as.matrix(x), na.action = na.omit, scaled = TRUE, frac = 1)&#xA;expand.grid(sigma = mean(sigmas[-2]), epsilon = 0.000001,&#xA;            C = 2^(-40:len), degree = 1:2) # len = tuneLength in train&#xA;}&#xA;SVManova$grid &amp;lt;- svmGridanova&#xA;svmFitanova &amp;lt;- function(x, y, wts, param, lev, last, weights, classProbs, ...) {&#xA;  ksvm(x = as.matrix(x), y = y,&#xA;       kernel = &quot;anovadot&quot;,&#xA;       kpar = list(sigma = param$sigma, degree = param$degree),&#xA;       C = param$C, epsilon = param$epsilon,&#xA;       prob.model = classProbs,&#xA;       ...) #default type = &quot;eps-svr&quot;&#xA;}&#xA;SVManova$fit &amp;lt;- svmFitanova&#xA;svmPredanova &amp;lt;- function(modelFit, newdata, preProc = NULL, submodels = NULL)&#xA;  predict(modelFit, newdata)&#xA;SVManova$predict &amp;lt;- svmPredanova&#xA;svmProb &amp;lt;- function(modelFit, newdata, preProc = NULL, submodels = NULL)&#xA;  predict(modelFit, newdata, type=&quot;probabilities&quot;)&#xA;SVManova$prob &amp;lt;- svmProb&#xA;svmSortanova &amp;lt;- function(x) x[order(x$C), ]&#xA;SVManova$sort &amp;lt;- svmSortanova&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;load data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataA2&amp;lt;-read.csv(&quot;C:/results/A2.txt&quot;,header = TRUE, &#xA;                             blank.lines.skip = TRUE,sep = &quot;,&quot;)&#xA;set.seed(1)&#xA;inTrainSet &amp;lt;- createDataPartition(dataA2$R, p = 0.75, list = FALSE) #[[1]]&#xA;trainSet &amp;lt;- dataA2[inTrainSet,]&#xA;testSet &amp;lt;- dataA2[-inTrainSet,]&#xA;#-----------------------------------------------------------------------------&#xA;#K-folds resampling method for fitting svr&#xA;ctrl &amp;lt;- trainControl(method = &quot;repeatedcv&quot;, number = 10, repeats = 10,&#xA;                     allowParallel = TRUE) #10 separate 10-fold cross-validations&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;link to data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wuala.com/jpcgandre/Documents/Data%20SVR/?key=BOD9NTINzRHG&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3216" LastActivityDate="2014-09-05T16:34:55.170" Title="ANOVA RBF kernel returns very poor results" Tags="|machine-learning|r|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1081" PostTypeId="2" ParentId="1071" CreationDate="2014-09-05T16:36:03.227" Score="1" Body="&lt;p&gt;Data is, at it's most basic reduction, a raw element of something. Data is a raw &quot;thing&quot; that exists in any form from which we can analyze it and construct intelligence. When I was an Intelligence Analyst, we used to define data as &quot;anything and everything that could be used to construct a hypothesis.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, data for any discipline is interchangeable; as a sociologist, I have a vector of discrete variables indicating ethnicity, as an economist I have a vector with housing prices, and as an anthropologist I have a vector of tablet names used in some long-gone civilization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data is data.&lt;/p&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2014-09-05T16:36:03.227" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1082" PostTypeId="2" ParentId="1078" CreationDate="2014-09-05T19:13:39.993" Score="4" Body="&lt;p&gt;Here are a couple of really great open source software packages for text classification that should help get you started:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mallet.cs.umass.edu/&quot; rel=&quot;nofollow&quot;&gt;MALLET&lt;/a&gt; is a CPL-licensed Java-based machine learning toolkit built by UMass for working with text data. It includes implementations of several &lt;a href=&quot;http://mallet.cs.umass.edu/classification.php&quot; rel=&quot;nofollow&quot;&gt;classification&lt;/a&gt; algorithms (e.g., naïve Bayes, maximum entropy, decision trees).&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://nlp.stanford.edu/software/classifier.shtml&quot; rel=&quot;nofollow&quot;&gt;Stanford Classifier&lt;/a&gt; from the Stanford NLP Group is a GPL-licensed Java implementation of a maximum entropy classifier designed to work with text data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-05T19:13:39.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1084" PostTypeId="2" ParentId="1078" CreationDate="2014-09-07T01:08:04.330" Score="14" Body="&lt;p&gt;Let's work it out from the ground up. Classification (also known as categorization) is an example of &lt;strong&gt;supervised learning&lt;/strong&gt;. In supervised learning you have: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;model&lt;/strong&gt; - something that approximates internal structure in your data, enabling you to reason about it and make useful predictions (e.g. predict class of an object); normally model has parameters that you want to &quot;learn&quot;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;training&lt;/strong&gt; and &lt;strong&gt;testing datasets&lt;/strong&gt; - sets of objects that you use for training your model (finding good values for parameters) and further evaluating &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;training&lt;/strong&gt; and &lt;strong&gt;classification algorithms&lt;/strong&gt; - first describes how to learn model from training dataset, second shows how to derive class of a new object given trained model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now let's take a simple case of spam classification. Your training dataset is a corpus of emails + corresponding labels - &quot;spam&quot; or &quot;not spam&quot;. Testing dataset has the same structure, but made from some independent emails (normally one just splits his dataset and makes, say, 9/10 of it to be used for training and 1/10 - for testing). One way to model emails is to represent each of them as a set (bag) of words. If we assume that words are independent of each other, we can use &lt;strong&gt;Naive Bayes classifier&lt;/strong&gt;, that is, calculate prior probabilities for each word and each class (training algorithm) and then apply Bayes theorem to find posterior probability of a new document to belong to particular class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, basically we have: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;raw model + training set + training algorithm -&amp;gt; trained model&#xA;trained model + classification algorithm + new object -&amp;gt; object label&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now note that we represented our objects (documents) as a bag of words. But is the only way? In fact, we can extract much more from raw text. For example, instead of words as is we can use their &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;stems or lemmas&lt;/a&gt;, throw out noisy &lt;a href=&quot;http://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow noreferrer&quot;&gt;stop words&lt;/a&gt;, add &lt;a href=&quot;http://nlp.stanford.edu/software/tagger.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;POS tags&lt;/a&gt; of words, extract &lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;named entities&lt;/a&gt; or even explore HTML structure of the document. In fact, more general representation of a document (and, in general, any object) is a &lt;strong&gt;feature vector&lt;/strong&gt;. E.g. for text: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;actor, analogue, bad, burn, ..., NOUN, VERB, ADJ, ..., in_bold, ... | label&#xA;    0,        0,   1,    1, ...,    5,    7,   2, ...,       2, ... | not spam&#xA;    0,        1,   0,    0, ...,    3,   12,  10, ...,       0, ... | spam&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here first row is a list of possible features and subsequent rows show how many times that feature occurs in a document. E.g. in first document there's no occurrences of word &quot;actor&quot;, 1 occurrence of word &quot;burn&quot;, 5 nouns, 2 adjectives and 2 pieces of text in bold. Last column corresponds to a resulting class label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using feature vector you can incorporate any properties of your texts. Though finding good set of features may take some time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what about model and algorithms? Are we bound to Naive Bayes. Not at all. &lt;strong&gt;logistic regression&lt;/strong&gt;, &lt;strong&gt;SVM&lt;/strong&gt;, &lt;strong&gt;decision trees&lt;/strong&gt; - just to mention few popular classifiers. (Note, that we say &quot;classifier&quot; in most cases we mean model + corresponding algorithms for training and classification). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for implementation, you can divide task into 2 parts: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Features extraction - transforming raw texts into feature vectors. &lt;/li&gt;&#xA;&lt;li&gt;Object classification - building and applying model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;First point is well worked out in many &lt;a href=&quot;https://stackoverflow.com/questions/4115526/natural-language-processing&quot;&gt;NLP libraries&lt;/a&gt;. Second is about machine learning, so, depending on your dataset, you can use either &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Weka&lt;/a&gt;, or &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-guide.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLlib&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-09-07T01:08:04.330" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="1085" PostTypeId="2" ParentId="1075" CreationDate="2014-09-07T08:02:39.670" Score="1" Body="&lt;p&gt;If I understand your description correctly, hadoop seems a huge overhead, for the wrong problem. basically you just need a standard distributed architecture: don't you just have to pass pairs of rows - eg mpi.... ipython.parallel, ...&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-09-07T08:02:39.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1086" PostTypeId="1" CreationDate="2014-09-07T22:51:27.490" Score="3" ViewCount="210" Body="&lt;p&gt;We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it &quot;unique content&quot; because our researchers want to know from which document to start reading. All the documents are of the same topic, in the biomedical world we know that there is always a lot of content overlap. So all we want to do is to arrange the documents in the order of their unique content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most Information Retrieval literature suggest query-based ranking which does not fit our need.&lt;/p&gt;&#xA;" OwnerUserId="3232" LastEditorUserId="10189" LastEditDate="2015-09-01T15:30:20.963" LastActivityDate="2015-09-01T15:30:20.963" Title="non query-based document ranking" Tags="|machine-learning|data-mining|text-mining|information-retrieval|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1087" PostTypeId="2" ParentId="1086" CreationDate="2014-09-08T00:47:21.590" Score="1" Body="&lt;p&gt;Here's a simple initial approach to try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;TF-IDF&lt;/a&gt; score of each word in each document.&lt;/li&gt;&#xA;&lt;li&gt;Sort the documents by the average TF-IDF score of their words.&lt;/li&gt;&#xA;&lt;li&gt;The higher the average TF-IDF score, the more unique a document is with respect to the rest of the collection.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You might also try a clustering-based approach where you look for outliers, or perhaps something with the &lt;a href=&quot;http://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow&quot;&gt;Jaccard index&lt;/a&gt; using a bag-of-words model.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-08T00:47:21.590" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1088" PostTypeId="2" ParentId="1078" CreationDate="2014-09-08T02:45:42.653" Score="5" Body="&lt;p&gt;Topic Modeling would be a very appropriate method for your problem.  Topic Models are a form of unsupervised learning/discovery, where a specified (or discovered) number of topics are defined by a list of words that have a high probability of appearing together. In a separate step, you can label each topic using subject matter experts, but for your purposes this isn't necessary since you are only interested in getting to three clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You treat each document as a bag of words, and pre-process to remove stop words, etc.  With the simplest methods, you pre-specify the number of topics.  In your case, you could either specify &quot;3&quot;, which is your fixed limit on categories, or pick a larger number of topics (between 10 and 100), and then in a separate step, form three clusters for documents with common emphasis on topics. K-means or other clustering methods could be used.  (I'd recommend the latter approach)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't need to code topic modeling software from scratch. Here's a &lt;a href=&quot;https://web.archive.org/web/20161027083945/http://www.cs.princeton.edu/~blei/topicmodeling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;web page with many resources, including software libraries/packages&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;None are in Java, but there are ways to run C++ and Python under Java.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="29575" LastEditDate="2017-03-25T07:31:56.423" LastActivityDate="2017-03-25T07:31:56.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1089" PostTypeId="2" ParentId="1086" CreationDate="2014-09-08T03:07:07.413" Score="2" Body="&lt;p&gt;You could use Topic Modeling as described in this paper:&#xA;&lt;a href=&quot;http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf&quot; rel=&quot;nofollow&quot;&gt;http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They performed Topic Modeling on abstracts of patents (limited to 150 words).  They identified papers as &quot;novel&quot; if they were the first to introduce a topic, and measured degree of novelty by how many papers in the following year used the same topic. (Read the paper for details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you follow their lead and only process paper abstracts.  Processing the body of each paper might reveal some novelty that the abstract does not, but you also run the risk of having much more noise in your topic model (i.e. extraneous topics, extraneous words).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you say that all 500 papers are on the same &quot;topic&quot;, it's probably safer to say that they are all on the same &quot;theme&quot; or in the same &quot;sub-category&quot; of Bio-medicine. Topic modeling permits decomposition of the &quot;theme&quot; into &quot;topics&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good news is that there are plenty of good packages/libraries for Topic Modeling.  You still have to do preprocessing, but you don't have to code the algorithms yourself.  See this page for many resources:&#xA;&lt;a href=&quot;http://www.cs.princeton.edu/~blei/topicmodeling.html&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.princeton.edu/~blei/topicmodeling.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-09-08T03:07:07.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1090" PostTypeId="1" CreationDate="2014-09-08T14:57:11.223" Score="1" ViewCount="38" Body="&lt;p&gt;It may be unlikely that anyone knows this but I have a specific question about Freebase.  Here is the Freebase page from the &lt;a href=&quot;http://www.freebase.com/m/014_d3&quot; rel=&quot;nofollow&quot;&gt;Ford Taurus automotive model&lt;/a&gt; .  It has a property called &quot;Related Models&quot;.  Does anyone know how this list of related models was compiled.  What is the similarity measure that they use?  I don't think it is only about other wikipedia pages that link to or from this page.  Alternatively, it may be that this is user generated.  Does anyone know for sure?&lt;/p&gt;&#xA;" OwnerUserId="387" LastActivityDate="2014-09-08T14:57:11.223" Title="Freebase Related Models" Tags="|dataset|" AnswerCount="0" CommentCount="0" ClosedDate="2020-10-19T04:59:17.330" ContentLicense="CC BY-SA 3.0" />
  <row Id="1091" PostTypeId="1" CreationDate="2014-09-08T19:33:00.253" Score="2" ViewCount="1508" Body="&lt;p&gt;What is the best technology to be used to create my custom bag of words with N-grams to apply to. I want to know a functionality that can be achieved over GUI. I cannot use spot fire as it is not available in the organization. Though i can get SAP Hana or R-hadoop. But R-hadoop is bit challenging, any suggessions.&lt;/p&gt;&#xA;" OwnerUserId="3244" LastActivityDate="2014-11-05T17:07:02.563" Title="Creating Bag of words" Tags="|bigdata|text-mining|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1092" PostTypeId="1" AcceptedAnswerId="1097" CreationDate="2014-09-08T21:25:26.183" Score="14" ViewCount="2062" Body="&lt;p&gt;Are there any machine learning libraries for Ruby that are relatively complete (including a wide variety of algorithms for supervised and unsupervised learning), robustly tested, and well-documented? I love Python's &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; for its incredible documentation, but a client would prefer to write the code in Ruby since that's what they're familiar with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally I am looking for a library or set of libraries which, like &lt;code&gt;scikit&lt;/code&gt; and &lt;code&gt;numpy&lt;/code&gt;, can implement a wide variety of data structures like sparse matrices, as well as learners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples of things we'll need to do are binary classification using SVMs, and  implementing bag of words models which we hope to concatenate with arbitrary numeric data, as described in &lt;a href=&quot;https://stackoverflow.com/q/20106940/1435804&quot;&gt;this StackOverflow post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2487" LastEditorUserId="134" LastEditDate="2018-12-29T02:25:42.790" LastActivityDate="2018-12-29T02:25:42.790" Title="Machine learning libraries for Ruby" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="1094" PostTypeId="1" AcceptedAnswerId="4897" CreationDate="2014-09-09T06:33:00.730" Score="2" ViewCount="251" Body="&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my machine learning task, I create a set of predictors.&#xA;Predictors come in &quot;bundles&quot; - multi-dimensional measurements (3 or 4 - dimensional in my case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hole &quot;bundle&quot; makes sense only if it has been measured, and taken all together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, different 'bundles' of predictors can be measured only for small part of the sample, and those parts don't necessary intersect for different 'bundles'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As parts are small, imputing leads to considerable decrease in accuracy(catastrophical to be more accurate)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Possible solutions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could create dummy variables that would mark whether the measurement has taken place for each variable. The problem is, when random forests draws random variables, it does so individually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there are two basic ways to solve this problem:&#xA;1) Combine each &quot;bundle&quot; into one predictor. That is possible, but it seems information will be lost. &#xA;2) Make random forest draw variables not individually, but by obligatory &quot;bundles&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem for random forest&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As random forest draws variables randomly, it takes features that are useless (or much less useful) without other from their &quot;bundle&quot;. I have a feeling that leads to a loss of accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I have variables &lt;code&gt;a&lt;/code&gt;,&lt;code&gt;a_measure&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt;,&lt;code&gt;b_measure&lt;/code&gt;.&#xA;The problem is, variables &lt;code&gt;a_measure&lt;/code&gt; make sense only if variable &lt;code&gt;a&lt;/code&gt; is present, same for &lt;code&gt;b&lt;/code&gt;. So I either have to combine &lt;code&gt;a&lt;/code&gt;and &lt;code&gt;a_measure&lt;/code&gt; into one variable, or make random forest draw both, in case at least one of them is drawn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the best practice solutions for problems when different sets of predictors are measured for small parts of overall population, and these sets of predictors come in obligatory &quot;bundles&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="3108" LastEditorUserId="3108" LastEditDate="2014-09-09T06:45:25.510" LastActivityDate="2015-04-17T06:30:43.227" Title="Creating obligatory combinations of variables for drawing by random forest" Tags="|machine-learning|r|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1095" PostTypeId="1" CreationDate="2014-09-09T12:44:16.967" Score="34" ViewCount="68492" Body="&lt;p&gt;The problem refers to decision trees building. According to Wikipedia '&lt;a href=&quot;http://en.wikipedia.org/wiki/Gini_coefficient&quot;&gt;Gini coefficient&lt;/a&gt;' should not be confused with '&lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity&quot;&gt;Gini impurity&lt;/a&gt;'. However both measures can be used when building a decision tree - these can support our choices when splitting the set of items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) 'Gini impurity' - it is a standard decision-tree splitting metric (see in the link above);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) 'Gini coefficient' - each splitting can be assessed based on the AUC criterion. For each splitting scenario we can build a ROC curve and compute AUC metric. According to Wikipedia AUC=(GiniCoeff+1)/2;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question is: are both these measures equivalent? On the one hand, I am informed that Gini coefficient should not be confused with Gini impurity. On the other hand, both these measures can be used in doing the same thing - assessing the quality of a decision tree split.&lt;/p&gt;&#xA;" OwnerUserId="3250" LastActivityDate="2023-03-13T18:03:52.697" Title="Gini coefficient vs Gini impurity - decision trees" Tags="|data-mining|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1096" PostTypeId="2" ParentId="1091" CreationDate="2014-09-09T15:05:46.093" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;create my custom bag of words with N-grams to apply to&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My initial recommendation would be to use the &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow&quot;&gt;NLTK library for Python&lt;/a&gt;. NLTK offers methods for &lt;a href=&quot;http://www.nltk.org/book/ch01.html#bigrams_index_term&quot; rel=&quot;nofollow&quot;&gt;easily extracting bigrams from text&lt;/a&gt; or &lt;a href=&quot;https://nltk.googlecode.com/svn/trunk/doc/api/nltk.util-module.html#ngrams&quot; rel=&quot;nofollow&quot;&gt;ngrams of arbitrary length&lt;/a&gt;, as well as methods for analyzing the &lt;a href=&quot;http://www.nltk.org/book/ch01.html#frequency_distribution_index_term&quot; rel=&quot;nofollow&quot;&gt;frequency distribution of those items&lt;/a&gt;. However, all of this requires a bit of programming.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a functionality that can be achieved over GUI&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's tricky. Have you looked into &lt;a href=&quot;https://gate.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;GATE&lt;/a&gt;? I'm not exactly sure if/how GATE does what you want, but it does offer a GUI.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-09T15:05:46.093" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1097" PostTypeId="2" ParentId="1092" CreationDate="2014-09-09T16:58:10.537" Score="8" Body="&lt;p&gt;I'll go ahead and post an answer for now; if someone has something better I'll accept theirs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point the most powerful option appears to be accessing WEKA using jRuby. We spent yesterday scouring the 'net, and this combination was even used by a &lt;a href=&quot;http://www.confreaks.com/videos/867-railsconf2012-practical-machine-learning-and-rails&quot;&gt;talk at RailsConf 2012&lt;/a&gt;, so I would guess if there were a comparable pure ruby package, they would have used it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that if you know exactly what you need, there are plenty of individual libraries that either &lt;a href=&quot;https://github.com/febeling/rb-libsvm&quot;&gt;wrap standalone packages like libsvm&lt;/a&gt; or &lt;a href=&quot;https://github.com/alexandru/stuff-classifier&quot;&gt;re-implement some individual algorithms like Naive Bayes in pure Ruby&lt;/a&gt; and will spare you from using jRuby.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for a general-purpose library, WEKA and jRuby seem to be the best bet at this time.&lt;/p&gt;&#xA;" OwnerUserId="2487" LastActivityDate="2014-09-09T16:58:10.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1098" PostTypeId="2" ParentId="1091" CreationDate="2014-09-10T07:33:06.470" Score="1" Body="&lt;p&gt;You can use SKlearn, It is a python library. It is simplest method which i like with minimal code. You can follow this link &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_extraction.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/feature_extraction.html&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="3259" LastActivityDate="2014-09-10T07:33:06.470" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1099" PostTypeId="2" ParentId="1095" CreationDate="2014-09-10T08:15:17.343" Score="34" Body="&lt;p&gt;No, despite their names they &lt;em&gt;are not&lt;/em&gt; equivalent or even that similar. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Gini impurity&lt;/strong&gt; is a measure of misclassification, which applies in a multiclass classifier context. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Gini coefficient&lt;/strong&gt; applies to binary classification and requires a classifier that can in some way rank examples according to the likelihood of being in a positive class. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both could be applied in some cases, but they are different measures for different things. Impurity is what is commonly used in &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;noreferrer&quot;&gt;decision trees&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="26596" LastEditDate="2016-11-30T23:06:49.983" LastActivityDate="2016-11-30T23:06:49.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1100" PostTypeId="1" CreationDate="2014-09-10T14:50:13.720" Score="11" ViewCount="742" Body="&lt;p&gt;I'm looking at pybrain for taking server monitor alarms and determining the root cause of a problem. I'm happy with training it using supervised learning and curating the training data sets. The data is structured something like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Server Type &lt;strong&gt;A&lt;/strong&gt; #1&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Alarm type 1&lt;/li&gt;&#xA;&lt;li&gt;Alarm type 2&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Server Type &lt;strong&gt;A&lt;/strong&gt; #2&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Alarm type 1&lt;/li&gt;&#xA;&lt;li&gt;Alarm type 2&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Server Type &lt;strong&gt;B&lt;/strong&gt; #1&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Alarm type &lt;strong&gt;99&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Alarm type 2&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So there are &lt;em&gt;n&lt;/em&gt; servers, with &lt;em&gt;x&lt;/em&gt; alarms that can be &lt;code&gt;UP&lt;/code&gt; or &lt;code&gt;DOWN&lt;/code&gt;. Both &lt;code&gt;n&lt;/code&gt; and &lt;code&gt;x&lt;/code&gt; are variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If Server A1 has &lt;em&gt;alarm 1 &amp;amp; 2&lt;/em&gt; as &lt;code&gt;DOWN&lt;/code&gt;, then we can say that &lt;em&gt;service a&lt;/em&gt; is down on that server and is the cause of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;em&gt;alarm 1&lt;/em&gt; is down on all servers, then we can say that &lt;em&gt;service a&lt;/em&gt; is the cause.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There can potentially be multiple options for the cause, so straight classification doesn't seem appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also like to tie later sources of data to the net. Such as just scripts that ping some external service.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the appropriate alarms may not be triggered at once, due to serial service checks, so it can start with one server down and then another server down 5 minutes later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to do some basic stuff at first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pybrain.tools.shortcuts import buildNetwork&#xA;from pybrain.datasets import SupervisedDataSet&#xA;from pybrain.supervised.trainers import BackpropTrainer&#xA;&#xA;&#xA;INPUTS = 2&#xA;OUTPUTS = 1&#xA;&#xA;# Build network&#xA;&#xA;# 2 inputs, 3 hidden, 1 output neurons&#xA;net = buildNetwork(INPUTS, 3, OUTPUTS)&#xA;&#xA;&#xA;# Build dataset&#xA;&#xA;# Dataset with 2 inputs and 1 output&#xA;ds = SupervisedDataSet(INPUTS, OUTPUTS)&#xA;&#xA;&#xA;# Add one sample, iterable of inputs and iterable of outputs&#xA;ds.addSample((0, 0), (0,))&#xA;&#xA;&#xA;&#xA;# Train the network with the dataset&#xA;trainer = BackpropTrainer(net, ds)&#xA;&#xA;# Train 1000 epochs&#xA;for x in xrange(10):&#xA;    trainer.train()&#xA;&#xA;# Train infinite epochs until the error rate is low&#xA;trainer.trainUntilConvergence()&#xA;&#xA;&#xA;# Run an input over the network&#xA;result = net.activate([2, 1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I[m having a hard time mapping variable numbers of alarms to static numbers of inputs. For example, if we add an alarm to a server, or add a server, the whole net needs to be rebuilt. If that is something that needs to be done, I can do it, but want to know if there's a better way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option I'm trying to think of, is have a different net for each type of server, but I don't see how I can draw an environment-wide conclusion, since it will just make evaluations on a single host, instead of all hosts at once. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which type of algorithm should I use and how do I map the dataset to draw environment-wide conclusions as a whole with variable inputs?&lt;/p&gt;&#xA;" OwnerUserId="3263" LastActivityDate="2016-03-07T17:23:56.783" Title="Neural net for server monitoring" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1101" PostTypeId="2" ParentId="1095" CreationDate="2014-09-10T15:52:04.743" Score="0" Body="&lt;p&gt;I think they both represent the same concept.&lt;/p&gt;&#xA;&lt;p&gt;In classification trees, the Gini Index is used to compute the impurity of a data partition. So Assume the data partition D consisiting of 4 classes each with equal probability. Then the Gini Index (Gini Impurity) will be:&#xA;&lt;span class=&quot;math-container&quot;&gt;$Gini(D) = 1 - (0.25^2 + 0.25^2 + 0.25^2 + 0.25^2)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;In CART we perform binary splits. So The gini index will be computed as the weighted sum of the resulting partitions and we select the split with the smallest gini index.&lt;/p&gt;&#xA;&lt;p&gt;So the use of Gini Impurity (Gini Index) is not limited to binary situations.&lt;/p&gt;&#xA;&lt;p&gt;Another term for Gini Impurity is Gini Coefficient which is used normally as a measure of income distribution.&lt;/p&gt;&#xA;" OwnerUserId="979" LastEditorUserId="29169" LastEditDate="2021-02-08T02:17:02.927" LastActivityDate="2021-02-08T02:17:02.927" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="1102" PostTypeId="1" AcceptedAnswerId="1103" CreationDate="2014-09-11T17:09:52.313" Score="4" ViewCount="1313" Body="&lt;p&gt;The CoreNLP parts of speech tagger and name entity recognition tagger are pretty good out of the box, but I'd like to improve the accuracy further so that the overall program runs better. To explain more about accuracy -- there are situations in which the POS/NER is wrongly tagged. For instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;Oversaw car manufacturing&quot; gets tagged as NNP-NN-NN&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Rather than VB* or something similar, since it's a verb-like phrase (I'm not a linguist, so take this with a grain of salt).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what's the best way to accomplish accuracy improvement?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there better models out there for POS/NER that can be incorporated into CoreNLP?&lt;/li&gt;&#xA;&lt;li&gt;Should I switch to other NLP tools?&lt;/li&gt;&#xA;&lt;li&gt;Or create training models with exception rules?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2785" LastActivityDate="2014-09-12T00:40:07.877" Title="Improve CoreNLP POS tagger and NER tagger?" Tags="|nlp|language-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1103" PostTypeId="2" ParentId="1102" CreationDate="2014-09-12T00:40:07.877" Score="1" Body="&lt;p&gt;Your best best is to train your own models on the kind of data you're going to be working with.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-12T00:40:07.877" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1104" PostTypeId="1" CreationDate="2014-09-12T04:22:38.823" Score="1" ViewCount="50" Body="&lt;p&gt;I have been tasked with creating a pipeline chart with the live data and the budgeted numbers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know what probability of each phase of reaching the next.  The problem is I have no Idea what to do about the pipeline budgeting with regards to time.  For instance what period of time should I have closed sales in the chart.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have honestly been working on trying to figure it out.  Each successive revision gets me farther from the answer.&lt;/p&gt;&#xA;" OwnerUserId="3279" LastActivityDate="2014-09-12T04:22:38.823" Title="Modeling Pipeline Budget" Tags="|recommender-system|time-series|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1105" PostTypeId="1" AcceptedAnswerId="1150" CreationDate="2014-09-12T11:06:48.203" Score="2" ViewCount="1264" Body="&lt;p&gt;I am currently trying to implement logistic regression with iteratively reweightes LS, according to &quot;Pattern Recognition and Machine Learning&quot; by C. Bishop. In a first approach I tried to implement it in C#, where I used Gauss' algorithm to solve eq. 4.99. For a single feature it gave very promising (nearly exact) results, but whenever I tried to run it with more than one feature my system matrix became singular, and the weights did not converge. I first thought that it was my implementation, but when I implemented it in SciLab the results sustained. The SciLab (more concise due to matrix operators) code I used is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;phi = [1; 0; 1; 1];&#xA;t = [1; 0; 0; 0];&#xA;w= [1];&#xA;&#xA;w' * phi(1,:)'&#xA;&#xA;for in=1:100&#xA;    y = [];&#xA;    R = zeros(size(phi,1));&#xA;    R_inv = zeros(size(phi,1));&#xA;&#xA;    for i=1:size(phi,1)&#xA;        y(i) = 1/(1+ exp(-(w' * phi(i,:)')));&#xA;        R(i,i) = y(i)*(1 - y(i));&#xA;        R_inv(i,i) = 1/R(i,i);&#xA;    end&#xA;&#xA;    z = phi * w - R_inv*(y - t)&#xA;    w = inv(phi'*R*phi)*phi'*R*z&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With the values for phi (input/features) and t (output/classes), it yields a weight of  -0.6931472, which is pretty much 1/3, which seems fine to me, for there is 1/3 probability of beeing assigned to class 1, if feature 1 is present (please forgive me, if my terms do not comply with ML-language completely, for I am an software developer). If I now added an intercept feature, which would accord to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;phi = [1, 1; 1, 0; 1, 1; 1, 1];&#xA;w = [1; 1];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my R-matrix becomes singular and the last weights value is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w  =&#xA;  - 5.8151677  &#xA;  1.290D+30  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which - to my reading - would mean, that the probability of belonging to class 1 would be close to 1 if feature 1 is present about 3% for the rest. There has got to be any error I made, but I do not get which one. For both implementations yield the same results I suspect that there is some point I've been missing or gotten wrong, but I do not understand which one.&lt;/p&gt;&#xA;" OwnerUserId="3283" LastActivityDate="2014-09-23T02:20:24.670" Title="Logistic Regression implementation does not converge" Tags="|logistic-regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1106" PostTypeId="1" CreationDate="2014-09-12T11:48:21.617" Score="9" ViewCount="516" Body="&lt;p&gt;I am having an HTML string and want to find out if a word I supply is relevant in that string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relevancy could be measured based on frequency in the text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example to illustrate my problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;this is an awesome bike store&#xA;bikes can be purchased online.&#xA;the bikes we own rock.&#xA;check out our bike store now&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to test a few other words:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bike repairs&#xA;dog poo&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;bike repairs&lt;/code&gt; should be marked as relevant whereas &lt;code&gt;dog poo&lt;/code&gt; should not be marked as relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How could this be done?&lt;/li&gt;&#xA;&lt;li&gt;How to I filter out ambiguous words like &lt;code&gt;in&lt;/code&gt; or &lt;code&gt;or&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for your ideas!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess it's something Google does to figure out what keywords are relevant to a website. I am basically trying to reproduce their on-page rankings.&lt;/p&gt;&#xA;" OwnerUserId="3284" LastEditorUserId="381" LastEditDate="2014-09-15T01:19:20.410" LastActivityDate="2020-08-17T01:00:18.380" Title="How to build a textual search engine?" Tags="|machine-learning|data-mining|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1107" PostTypeId="1" AcceptedAnswerId="1112" CreationDate="2014-09-12T15:20:51.767" Score="35" ViewCount="15989" Body="&lt;p&gt;I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test samples as a majority class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some good answers about sub-sampling and weighted random forest are given here: &lt;a href=&quot;https://datascience.stackexchange.com/questions/454/what-are-the-implications-for-training-a-tree-ensemble-with-highly-biased-datase&quot;&gt;What are the implications for training a Tree Ensemble with highly biased datasets?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which classification methods besides RF can handle the problem in the best way?&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-15T22:10:08.333" Title="Quick guide into training highly imbalanced data sets" Tags="|machine-learning|classification|dataset|class-imbalance|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1108" PostTypeId="1" AcceptedAnswerId="1121" CreationDate="2014-09-12T16:26:15.827" Score="6" ViewCount="4618" Body="&lt;p&gt;As mentioned &lt;a href=&quot;https://datascience.stackexchange.com/questions/1107/quick-guide-into-training-highly-imbalanced-data-sets&quot;&gt;before&lt;/a&gt;, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples.&#xA;I have trained a Generalized Boosted Regression model using &lt;code&gt;gbm()&lt;/code&gt; from the &lt;code&gt;gbm&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt; and get the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD&#xA;  1                  50       0.906     0.523  0.00978      0.0512  &#xA;  1                  100      0.91      0.561  0.0108       0.0517  &#xA;  1                  150      0.91      0.572  0.0104       0.0492  &#xA;  2                  50       0.908     0.569  0.0106       0.0484  &#xA;  2                  100      0.91      0.582  0.00965      0.0443  &#xA;  2                  150      0.91      0.584  0.00976      0.0437  &#xA;  3                  50       0.909     0.578  0.00996      0.0469  &#xA;  3                  100      0.91      0.583  0.00975      0.0447  &#xA;  3                  150      0.911     0.586  0.00962      0.0443  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking at the 90% accuracy I assume that model has labeled all the samples as majority class. That's clear.&#xA;And what is not transparent: how Kappa is calculated.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What does this Kappa values (near to 60%) really mean? Is it enough to say that the model is not classifying them just by chance? &lt;/li&gt;&#xA;&lt;li&gt;What do &lt;code&gt;Accuracy SD&lt;/code&gt; and &lt;code&gt;Kappa SD&lt;/code&gt; mean?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2020-04-01T21:11:58.403" Title="Kappa near to 60% in unbalanced (1:10) data set" Tags="|r|class-imbalance|gbm|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1109" PostTypeId="2" ParentId="1107" CreationDate="2014-09-12T20:30:51.740" Score="20" Body="&lt;p&gt;Undersampling the majority class is usually the way to go in such situations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think that you have too few instances of the positive class, you may perform oversampling, for example, sample 5n instances with replacement from the dataset of size n.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caveats:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Some methods may be sensitive to changes in the class distribution, e.g. for Naive Bayes - it affects the prior probabilities. &lt;/li&gt;&#xA;&lt;li&gt;Oversampling may lead to overfitting  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-09-12T20:30:51.740" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1110" PostTypeId="1" AcceptedAnswerId="1118" CreationDate="2014-09-13T06:33:17.360" Score="6" ViewCount="1655" Body="&lt;p&gt;I want to cluster a set of long-tailed / pareto-like data into several bins (actually the bin number is not determined yet). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm or model would anyone recommend?&lt;/p&gt;&#xA;" OwnerUserId="3289" LastEditorUserId="14372" LastEditDate="2017-05-30T14:50:23.443" LastActivityDate="2017-05-30T14:50:23.443" Title="Binning long-tailed / pareto data before clustering" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1112" PostTypeId="2" ParentId="1107" CreationDate="2014-09-13T15:36:19.867" Score="24" Body="&lt;ul&gt;&#xA;&lt;li&gt;Max Kuhn covers this well in Ch16 of &lt;em&gt;Applied Predictive Modeling&lt;/em&gt;.     &lt;/li&gt;&#xA;&lt;li&gt;As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data.&lt;/li&gt;&#xA;&lt;li&gt;There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. Rusboost I think is only available as Matlab code. &lt;/li&gt;&#xA;&lt;li&gt;I don't use Weka, but believe it has a large number of cost sensitive classifiers. &lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Handling imbalanced datasets: A review&lt;/em&gt;: Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas'&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;On the Class Imbalance Problem&lt;/em&gt;: Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, Guangtong Zhou&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3294" LastActivityDate="2014-09-13T15:36:19.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1113" PostTypeId="1" AcceptedAnswerId="1146" CreationDate="2014-09-13T17:13:23.373" Score="2" ViewCount="159" Body="&lt;p&gt;I have a general methodological question. I have two columns of data, with one a column a numeric variable for age and another column a short character variable for text responses to a question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to group the age variable (that is, create cut points for the age variable), based on the text responses. I'm unfamiliar with any general approaches for doing this sort of analysis. What general approaches would you recommend? Ideally I'd like to categorize the age variable based on linguistic similarity of the text responses.&lt;/p&gt;&#xA;" OwnerUserId="36" LastEditorUserId="4647" LastEditDate="2015-08-21T07:33:06.983" LastActivityDate="2015-08-21T07:33:06.983" Title="General approahces for grouping a continuous variable based on text data?" Tags="|bigdata|clustering|text-mining|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1114" PostTypeId="2" ParentId="1107" CreationDate="2014-09-13T18:17:33.253" Score="14" Body="&lt;p&gt;Gradient boosting is also a good choice here.  You can use the gradient boosting classifier in sci-kit learn for example.  Gradient boosting is a principled method of dealing with class imbalance by constructing successive training sets based on incorrectly classified examples.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2014-09-13T18:17:33.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1118" PostTypeId="2" ParentId="1110" CreationDate="2014-09-15T13:03:44.083" Score="5" Body="&lt;p&gt;There are several approaches. You can start from the second one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt; Equal-width (distance) partitioning:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;It divides the range into N intervals of equal size: uniform grid&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if A and B are the lowest and highest values of the attribute, the width of intervals will be: &lt;code&gt;W = (B-A)/N&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The most straightforward&#xA;- Outliers may dominate presentation&#xA;- Skewed data is not handled well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt; Equal-depth (frequency) partitioning:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It divides the range into &#xA;N intervals, each containing approximately &#xA;same number of samples&lt;/li&gt;&#xA;&lt;li&gt;Good data scaling&lt;/li&gt;&#xA;&lt;li&gt;Managing categorical attributes can be tricky.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Methods&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Rank&lt;/code&gt;: The rank of a number is its size relative to other values of a numerical variable. First, we sort the list of values, then we assign the position of a value as its rank. Same values receive the same rank but the presence of duplicate values affects the ranks of subsequent values (e.g., 1,2,3,3,5). Rank is a solid binning method with one major drawback, values can have different ranks in different lists.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Quantiles (median, quartiles, percentiles, ...)&lt;/code&gt;: Quantiles are also very useful binning methods but like Rank, one value can have different quantile if the list of values changes.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Math functions&lt;/code&gt;: For example, logarithmic binning is an effective method for the numerical variables with highly skewed distribution (e.g., income).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Entropy-based Binning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.saedsayad.com/supervised_binning.htm&quot; rel=&quot;noreferrer&quot;&gt;Entropy based method&lt;/a&gt; uses a split approach. The entropy (or the information content) is calculated based on the class label. Intuitively, it finds the best split so that the bins are as pure as possible that is the majority of the values in a bin correspond to have the same class label. Formally, it is characterized by finding the split with the maximal information gain. &lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2014-09-15T13:03:44.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1119" PostTypeId="1" CreationDate="2014-09-15T13:14:40.797" Score="5" ViewCount="3811" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/RFM_(customer_value)&quot; rel=&quot;noreferrer&quot;&gt;RFM&lt;/a&gt; - is a ranking model when all customers are ranked according to their purchasing &lt;strong&gt;F&lt;/strong&gt; requency, &lt;strong&gt;R&lt;/strong&gt; recency and &lt;strong&gt;M&lt;/strong&gt; monetary value. This indicator is highly used by marketing departments of various organizations to segment customers into groups according to customer value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is following: are there any substantial models based on RFM scoring (or related to) which have solid predictive power?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;predicting which customer will most likely spend more&lt;/li&gt;&#xA;&lt;li&gt;who is going to upgrade/renew subscribtion/refund etc&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update2&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I understand, this is simple problem with three independent variable and one classifier. My guess and experience say these pure three factors do not predict future customer value. But they can be used together with another data or can be an additional input into some model.&lt;/li&gt;&#xA;&lt;li&gt;Please share which methodologies worked for you personally and are likely to have high predictive ability. What kind of data you used together with RFM indicators and it worked well?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-09-23T09:11:21.413" LastActivityDate="2017-09-21T14:42:59.557" Title="Predictive modeling based on RFM scoring indicators" Tags="|marketing|predictive-modeling|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1120" PostTypeId="2" ParentId="1108" CreationDate="2014-09-15T18:14:16.180" Score="3" Body="&lt;p&gt;This may provide some answer: &lt;a href=&quot;https://cran.r-project.org/web/packages/caret/vignettes/caret.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/caret/vignettes/caret.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also check out Max Kuhn's &quot;Applied Predictive Modeling&quot; book. He talks about the caret package at length in this book, including the kappa statistics and how to use it. This may be of some help to you.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="41411" LastEditDate="2020-04-01T21:11:58.403" LastActivityDate="2020-04-01T21:11:58.403" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="1121" PostTypeId="2" ParentId="1108" CreationDate="2014-09-15T20:29:16.620" Score="9" Body="&lt;p&gt;The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to the level of agreement that would occur through random chance. There's a very good overview of how to calculate Kappa and use it to evaluate a classifier in this stats.stackexchange.com answer &lt;a href=&quot;https://stats.stackexchange.com/questions/82162/kappa-statistic-in-plain-english&quot;&gt;here&lt;/a&gt;, and a more in-depth explanation of Kappa and how to interpret it in &lt;a href=&quot;http://www1.cs.columbia.edu/~julia/courses/CS6998/Interrater_agreement.Kappa_statistic.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;, entitled &quot;Understanding Interobserver Agreement: The Kappa Statistic&quot; by Viera &amp;amp; Garrett (2005).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benefit of using Kappa, particularly in an unbalanced data set like yours, is that with a 90-10% imbalance between the classes, you can achieve 90% accuracy by simply labeling all of the data points with the label of the more commonly occurring class. The Kappa statistic is describing how well the classifier performs above that baseline level of performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kappa ranges from -1 to 1, with 0 indicating no agreement between the raters, 1 indicating a perfect agreement, and negative numbers indicating systematic disagreement. While interpretation is somewhat arbitrary (and very task-dependent), Landis &amp;amp; Koch (1977) defined the following interpretation system which can work as a general rule of thumb:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Kappa Agreement&#xA;&amp;lt; 0 Less than chance agreement&#xA;0.01–0.20 Slight agreement&#xA;0.21– 0.40 Fair agreement&#xA;0.41–0.60 Moderate agreement&#xA;0.61–0.80 Substantial agreement&#xA;0.81–0.99 Almost perfect agreement&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which would indicate that your algorithm is performing moderately well. Accuracy SD and Kappa SD are the respective Standard Deviations of the Accuracy and Kappa scores. I hope this is helpful!&lt;/p&gt;&#xA;" OwnerUserId="2969" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-09-15T20:29:16.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1122" PostTypeId="2" ParentId="1119" CreationDate="2014-09-15T22:43:37.583" Score="3" Body="&lt;p&gt;A Google search leads to many relevant resources that answer your question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;USING RFM DATA TO OPTIMIZE DIRECT MARKETING CAMPAIGNS: A LINEAR PROGRAMMING APPROACH &lt;a href=&quot;http://www.thefreelibrary.com/Using+RFM+data+to+optimize+direct+marketing+campaigns%3A+a+linear...-a0272246211&quot; rel=&quot;nofollow&quot;&gt;http://www.thefreelibrary.com/Using+RFM+data+to+optimize+direct+marketing+campaigns%3A+a+linear...-a0272246211&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Data Mining using RFM Analysis &lt;a href=&quot;http://cdn.intechweb.org/pdfs/13162.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cdn.intechweb.org/pdfs/13162.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Libby on Recency, Frequency, Monetary Value&lt;/em&gt; (book) &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1882222067&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Recency-Frequency-Monetary-Century-Library/dp/1882222067&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From a data science point of view, there is nothing very special or unique about this problem. You have three independent variables and one dependent variable. Regression, clustering, and classification methods can be applied.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-09-15T22:43:37.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1123" PostTypeId="1" AcceptedAnswerId="1126" CreationDate="2014-09-16T08:01:35.997" Score="3" ViewCount="5065" Body="&lt;p&gt;Suppose I am interested in classifying a set of instances composed by different content types, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a piece of &lt;strong&gt;text&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;an &lt;strong&gt;image&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;as &lt;code&gt;relevant&lt;/code&gt; or &lt;code&gt;non-relevant&lt;/code&gt; for a specific class &lt;code&gt;C&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my classification process I perform the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given a sample, I subdivide it in text and image&lt;/li&gt;&#xA;&lt;li&gt;A first SVM binary classifier (&lt;code&gt;SVM-text&lt;/code&gt;), trained only on text, classifies the text as &lt;code&gt;relevant&lt;/code&gt;/&lt;code&gt;non-relevant&lt;/code&gt; for the class &lt;code&gt;C&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;A second SVM binary classifier (&lt;code&gt;SVM-image&lt;/code&gt;), trained only on images, classifies the image as &lt;code&gt;relevant&lt;/code&gt;/&lt;code&gt;non-relevant&lt;/code&gt; for the class &lt;code&gt;C&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both &lt;code&gt;SVM-text&lt;/code&gt; and &lt;code&gt;SVM-image&lt;/code&gt; produce an estimate of the probability of the analyzed content (text or image) of being relevant for the class &lt;code&gt;C&lt;/code&gt;. Given this, I am able to state whether the text is relevant for &lt;code&gt;C&lt;/code&gt; and the image is relevant for &lt;code&gt;C&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, these estimates are valid for segments of the original sample (either the text or the image), while it is not clear how to obtain a general opinion on the whole original sample (text+image). How can I combine conveniently the opinions of the two classifiers, so as to obtain a classification for the whole original sample?&lt;/p&gt;&#xA;" OwnerUserId="3321" LastActivityDate="2014-09-16T12:32:10.130" Title="Combine multiple classifiers to build a multi-modal classifier" Tags="|classification|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1124" PostTypeId="1" AcceptedAnswerId="1138" CreationDate="2014-09-16T08:14:06.307" Score="9" ViewCount="2639" Body="&lt;p&gt;I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach. &#xA;I am using the basic implementation of Random Forests (the one present in scikit), that creates a binary condition on a single variable at each split. Given this, is there a difference between using simple tf (term frequency) features. where each word has an associated weight that represents the number of occurrences in the document, or tf-idf (term frequency * inverse document frequency), where the term frequency is also multiplied by a value that represents the ratio between the total number of documents and the number of documents containing the word)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, there should not be any difference between these two approaches, because the only difference is a scaling factor on each feature, but since the split is done at the level of single features this should not make a difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right in my reasoning? &lt;/p&gt;&#xA;" OwnerUserId="3054" LastActivityDate="2014-09-18T13:14:37.940" Title="Difference between tf-idf and tf with Random Forests" Tags="|classification|text-mining|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1125" PostTypeId="1" CreationDate="2014-09-16T11:41:48.140" Score="5" ViewCount="227" Body="&lt;p&gt;I am interested in the field of &lt;strong&gt;named entity disambiguation&lt;/strong&gt; and want to learn more about it. I have heard that there are contests organised by various associations on these kind of research topics. These contests are very helpful as they give a practical experience in these fields. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found one such contest organised by Microsoft research &lt;a href=&quot;http://web-ngram.research.microsoft.com/erd2014/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; though the dates have already passed. Can anyone point me to any other such contests ? Also, is there a site which catalogues these contests so that one can just go there and know about all upcoming contests ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.  &lt;/p&gt;&#xA;" OwnerUserId="3325" LastEditorUserId="21" LastEditDate="2017-05-19T16:12:17.180" LastActivityDate="2017-05-19T16:12:17.180" Title="Named entity disambiguation contests" Tags="|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1126" PostTypeId="2" ParentId="1123" CreationDate="2014-09-16T12:32:10.130" Score="2" Body="&lt;p&gt;Basically, you can do one of two things: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Combine features&lt;/strong&gt; from both classifiers. I.e., instead of &lt;code&gt;SVM-text&lt;/code&gt; and &lt;code&gt;SVM-image&lt;/code&gt; you may train single &lt;code&gt;SVM&lt;/code&gt; that uses both - textual and visual features. &lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;ensemble learning&lt;/strong&gt;&lt;/a&gt;. If you already have probabilities from separate classifiers, you can simply use them as weights and compute weighted average. For more sophisticated cases there are Bayesian combiners (each classifier has its prior), boosting algorithms (e.g. see &lt;a href=&quot;http://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow&quot;&gt;AdaBoost&lt;/a&gt;) and others. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note, that ensembles where initially created for combining different learners, not different sets if features. In this later case ensembles have advantage mostly in cases when different kinds of features just can't be combined in a single vector efficiently. But in general, combing features is simpler and more straightforward. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-09-16T12:32:10.130" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1127" PostTypeId="2" ParentId="310" CreationDate="2014-09-16T14:29:41.520" Score="6" Body="&lt;p&gt;&lt;strong&gt;One class learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't be too quick to throw out one-class classification methods (option 2) - the key is to model the positive (minority) class with the one-class model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There has been research demonstrating cases where one-class classification out-performed other approaches like sampling for highly imbalanced data as often seen with protein classification tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find the research I recalled, but I did find some other comparisons, showing using one-class classifiers (typically modeling the minority class) achieved as good or better performance than binary classification typically with sampled &quot;negatives&quot; from the large set of proteins not known to be positive.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally this approach also gives the advantage of much improved run-time - since you only need to train the classifier on the smaller, positive set.  A couple papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://journal.imbio.de/articles/pdf/jib-77.pdf&quot; rel=&quot;noreferrer&quot;&gt;&quot;Prediction of protein-protein interactions using one-class classification methods and integrating diverse biological data&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.inf.u-szeged.hu/~busarobi/PDFs/BaBuKe09.pdf&quot; rel=&quot;noreferrer&quot;&gt;&quot;A One-Class Classification Approach for Protein Sequences and Structures&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the very least I would try some one-class methods and compare the performance using validation with your binary/multi-class classification approaches.  There are also open source implementations for many of these so it shouldn't be too costly to try them out, for example LibSVM has a one-class SVM implementation.  Additionally, it might prove valuable for use in an ensemble with binary classifiers, since there may be more disagreement in their predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Higher level representation embedding / clustering&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along the lines of what you were thinking with (1) and the other post suggesting PCA, approaches like clustering, sparse coding, or even topic modeling - treating each protein as a document string and different protein families as different topics - could yield a representation that might make classifying the proteins straightforward.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e., you could identify which group/cluster a protein belongs to or classify the cluster memberships / embedded representations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., such embedding approaches as sparse coding can yield representations that reveal which cluster a protein belongs too - so that some sets of features are only active (non-zero) for proteins in the same cluster - which can make classifying them much easier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally class labels or known cluster membership can be incorporated in the embedding process for most methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ensemble&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ensembles of multiple classifiers tend to work best - especially when the classifiers are very diverse and can achieve comparable performance individually.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are at least two ways use ensembles for this problem.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can build an ensemble of binary classifiers by sampling multiple different same-size negative sets and training a classifier on each.&lt;/li&gt;&#xA;&lt;li&gt;You can build an ensemble from different approaches, such as binary classifiers with different negative samples, combined with a one-class classification approach, combined with classification models trained on the embedded data. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3326" LastActivityDate="2014-09-16T14:29:41.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1128" PostTypeId="1" CreationDate="2014-09-16T15:47:17.430" Score="6" ViewCount="571" Body="&lt;p&gt;I am looking for information on (formal) algebraic systems that can be used to transform time-series - in either a practical or academic context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that there exists (at least one) small, expressive, set of operators - ranging over (finite) time-series.  I want to compare and contrast different systems  with respect to algebraic completeness, and brevity of representation, of common time-series transformations in various domains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise this question is broad - but hope it is not too vague for datascience.stackexchange.  I welcome any pointers to relevant literature for specific scenarios, or the general subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit... (Attempt to better explain what I meant by an algebraic system...)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about &quot;abstract algebras&quot; as discussed in Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Algebra#Abstract_algebra&quot;&gt;http://en.wikipedia.org/wiki/Algebra#Abstract_algebra&lt;/a&gt;&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Abstract_algebra#Basic_concepts&quot;&gt;http://en.wikipedia.org/wiki/Abstract_algebra#Basic_concepts&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boolean Algebras are (very simple) algebras that range over Boolean values.  A simple example of such an algebra would consist the values True and False and the operations AND, OR and NOT. One might argue this algebra is 'complete' as, from these two constants (free-variables) and three basic operations, arbitrary boolean functions can be constructed/described.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested to discover algebras where the values are (time-domain) time-series.  I'd like it to be possible to construct &quot;arbitrary&quot; functions, that map time-series to time-series, from a few operations which, individually, map time-series to time-series.  I am open to liberal interpretations of &quot;arbitrary&quot;. I would be especially interested in examples of these algebras where the operations consist 'higher-order functions' - where such operations have been developed for a specific domain.&lt;/p&gt;&#xA;" OwnerUserId="3328" LastEditorUserId="3328" LastEditDate="2014-09-17T13:34:51.830" LastActivityDate="2015-08-09T10:44:16.053" Title="Looking for algebras designed to transform time series" Tags="|machine-learning|algorithms|time-series|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="1129" PostTypeId="2" ParentId="403" CreationDate="2014-09-16T16:37:33.357" Score="2" Body="&lt;p&gt;It's probably due to the &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/seminars/sas_logistic/logistic1.htm&quot; rel=&quot;nofollow&quot;&gt;effect coding&lt;/a&gt; of categorical predictors. Eg, the regression coefficient for CPR = 7 is not zero but -(sum of regression coefficients for the other 6 levels). I guess EM should have an option to switch it to reference coding, then your way of computing the predicted probability should work.&lt;/p&gt;&#xA;" OwnerUserId="3315" LastActivityDate="2014-09-16T16:37:33.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1131" PostTypeId="2" ParentId="1128" CreationDate="2014-09-17T00:44:19.967" Score="0" Body="&lt;p&gt;The most direct and obvious transformation is from time domain to &lt;a href=&quot;http://en.wikipedia.org/wiki/Frequency_domain&quot; rel=&quot;nofollow&quot;&gt;frequency domain&lt;/a&gt;. Possible methods include &lt;a href=&quot;http://en.wikipedia.org/wiki/Fourier_transform&quot; rel=&quot;nofollow&quot;&gt;Fourier transform&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Wavelet_transform&quot; rel=&quot;nofollow&quot;&gt;wavelet transform&lt;/a&gt;.  After the transform the signal is represented by a function of frequency-domain elements which can be operated on using ordinary algebra.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible to model a time series as a trajectory of a dynamical system in a state space (see: &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9892.1980.tb00300.x/abstract&quot; rel=&quot;nofollow&quot;&gt;http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9892.1980.tb00300.x/abstract&lt;/a&gt;, and &lt;a href=&quot;http://www3.stat.sinica.edu.tw/statistica/oldpdf/A2n16.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www3.stat.sinica.edu.tw/statistica/oldpdf/A2n16.pdf&lt;/a&gt;). Dynamical systems can be modeled symbolically at a course-grain level (see: &lt;a href=&quot;http://en.wikipedia.org/wiki/Symbolic_dynamics&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Symbolic_dynamics&lt;/a&gt; and &lt;a href=&quot;http://www.math.washington.edu/SymbolicDynamics/&quot; rel=&quot;nofollow&quot;&gt;http://www.math.washington.edu/SymbolicDynamics/&lt;/a&gt;)  Symbolic dynamics draws on linear algebra.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-09-17T00:44:19.967" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1132" PostTypeId="1" CreationDate="2014-09-17T04:49:39.770" Score="1" ViewCount="1636" Body="&lt;p&gt;I am a research scholar in data mining. I'm interested in C# implementation of K-Means clustering algorithm for mixed numeric and categorical data.&lt;/p&gt;&#xA;" OwnerUserId="3336" LastEditorUserId="97" LastEditDate="2014-09-19T12:15:42.527" LastActivityDate="2014-11-18T12:55:21.787" Title="K-Means clustering for mixed numeric and categorical data implementation in C#" Tags="|clustering|k-means|beginner|library|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1133" PostTypeId="2" ParentId="1132" CreationDate="2014-09-17T10:16:07.100" Score="1" Body="&lt;p&gt;The solution is described here: &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;K-Means clustering for mixed numeric and categorical data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C# implementation can be found in ALGLIB library, which I strongly recommend: &lt;a href=&quot;http://www.alglib.net/translator/man/manual.csharp.html#gs_packages&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.alglib.net/translator/man/manual.csharp.html#gs_packages&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-09-17T10:16:07.100" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1135" PostTypeId="1" CreationDate="2014-09-17T13:13:59.147" Score="6" ViewCount="349" Body="&lt;p&gt;I have a rather large commute every day - it ranges between about an hour and about an hour and half of driving.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been tracking my driving times, and want to continue to do so. I am capturing the date, my time of departure, my time of arrival, the route I took (there are two or three possible ones), weather conditions (wet/dry and clear/hazy/foggy), and whether I stopped (and if so, for what reason - fuel/toilet break/food break, and for how long) for every journey to and from work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to create a system to analyse this data and suggest an optimal departure time (for the next journey) based on day of the week, weather conditions, and whether i need to stop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anecdotally, I can see that Tuesday mornings are worse than other mornings, the earlier I leave the more likely I am to take a toilet break or a food break, and obviously that the journey takes longer on rainy or foggy days than on clear and dry days - but I would like the system to empirically tell me that!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this is a machine-learning and statistical analysis problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have absolutely no knowledge of machine-learning, or statistical methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What statistical methods should I use to do this kind of analysis to the point where the data will lead to suggestions like &quot;tomorrow is Tuesday and it is going to rain, so you must leave home between 7.50 and 8.00, and take route XYZ, to get the optimal driving time. Oh and chances are you will need a toilet break - and I have factored that in&quot;? (assume that I manually enter tomorrow’s weather forecast - I’ll look into integrating with a weather service later)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is life-hacking for me, trying to optimise the hell out of a tedious process, and it is very personal - specific to me and my habits, specific to this route, and specific to the morning/evening commute times. Google Maps with Traffic, TomTom with IQ, and Waze do very well in the more open-ended situations of ad-hoc driving-time prediction. Even Apple is happy to tell me on my iPhone notification screen how long it will take me to get home if I leave right now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note, it appears to me that traffic is not a consideration - that is to say, I do not think I need to know the actual traffic conditions - traffic is a function of day of the week and weather. For example, there are more people on the roads on Monday and Tuesday mornings, and people drive more slowly, and more people are in cars (opting to drive instead of cycle or take public transport) when it rains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To what extent can I let the data do all the talking? I have a somewhat ambiguous hidden agenda which may not be apparent from the data;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I should be at work at 9.30 (i.,e. 9.15 +/- 15 minutes) every day, but the occasional 10am arrival is OK&lt;/li&gt;&#xA;&lt;li&gt;I want to leave home as late as possible, and yet arrive at work as early as possible&lt;/li&gt;&#xA;&lt;li&gt;I want to leave work as early as possible, and yet have done at least 8 hours’ work&lt;/li&gt;&#xA;&lt;li&gt;it is OK for me to, say, leave half an hour early on one day but stay late on another to compensate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think I can come up with a procedural formula that can encompass all of these rules, but my gut feeling is that statistical analysis can make it a lot smarter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the methods of analysis, the technology stack is not an issue. Java is my language of choice - I am quite familiar with programming in it, and in creating web applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that it is possible, are there Java libraries that can provide the requisite methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What limitations are there? I want to keep capturing more and more data every day, making the data set bigger, hopefully, making the prediction more accurate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other ways are there to do it? Can I push this data into, say, Wolfram Programming Cloud, or maybe something Google provides to get the desired results?&lt;/p&gt;&#xA;" OwnerUserId="3343" LastActivityDate="2014-10-30T14:59:13.243" Title="Statistical Commute Analysis in Java" Tags="|machine-learning|statistics|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1136" PostTypeId="2" ParentId="1135" CreationDate="2014-09-18T01:39:30.063" Score="2" Body="&lt;p&gt;You &lt;em&gt;do&lt;/em&gt; want to model the traffic, at least over a work day, otherwise it wouldn't matter what time you traveled! Absent any data, I'd assume there isn't much variance over the working week, but that's one thing the data will quickly confirm or refute. If it is varying, you can use a different model for each day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have two variables; the departure times from home and work, respectively. Let's call them t_h and t_w. Let's call the commute time T_c(t), where t is the time of day. You can estimate this function from the data, so I'll assume it is given.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to maximize c t_h - (1-c) t_w subject to the constraints t_h + T_c(t_h) &amp;lt; 9.5 and t_w &gt; t_h + T_c(t_h) + 8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where c is a constant you can set to adjust the relative importance of leaving home early relative to leaving work early. You should be able to solve this numerical optimization problem with Mathematica, MATLAB, or something similar. I would not recommend Java; it's not meant for this. The only tricky part is estimating T_c. You know that it's a non-negative function, so you could use the standard trick of estimating it's logarithm (say, with kernels) and exponentiating. For implementation with Mathematica see &lt;a href=&quot;http://reference.wolfram.com/language/ref/InterpolatingFunction.html&quot; rel=&quot;nofollow&quot;&gt;Smoothing Data, Filling Missing Data, and Nonparametric Fitting&lt;/a&gt; and &lt;a href=&quot;http://reference.wolfram.com/language/tutorial/ConstrainedOptimizationOverview.html&quot; rel=&quot;nofollow&quot;&gt;Constrained Optimization&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-09-19T20:17:35.690" LastActivityDate="2014-09-19T20:17:35.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1137" PostTypeId="1" CreationDate="2014-09-18T06:22:13.940" Score="7" ViewCount="3574" Body="&lt;p&gt;I have time series data from mobile sensors for different motions such as walking, pushups, dumbellifts, rowing and so on. All these motions have different length of time series. For classifying them using &lt;a href=&quot;http://en.wikipedia.org/wiki/Dynamic_time_warping&quot;&gt;Dynamic Time Warping (DTW)&lt;/a&gt;, how do I choose an appropriate window size that will give good results?&lt;/p&gt;&#xA;" OwnerUserId="2475" LastEditorUserId="84" LastEditDate="2014-09-19T12:22:10.130" LastActivityDate="2015-07-18T09:41:58.270" Title="Choosing a window size for DTW" Tags="|classification|time-series|parameter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1138" PostTypeId="2" ParentId="1124" CreationDate="2014-09-18T13:14:37.940" Score="8" Body="&lt;p&gt;Decision trees (and hence Random Forests) are insensitive to monotone transformations of input features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since multiplying by the same factor is a monotone transformation, I'd assume that for Random Forests there indeed is no difference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you eventually may consider using other classifiers that do not have this property, so it may still make sense to use the entire TF * IDF. &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-09-18T13:14:37.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1139" PostTypeId="2" ParentId="1125" CreationDate="2014-09-18T13:51:58.473" Score="1" Body="&lt;p&gt;Some of the GREC shared task challenges included a named entity recognition &amp;amp; coreference resolution component (i.e., disambiguation), but I don't think they've run GREC since 2010...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://sites.google.com/site/genchalrepository/reg-in-context/grec-ner&quot; rel=&quot;nofollow&quot;&gt;https://sites.google.com/site/genchalrepository/reg-in-context/grec-ner&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-18T13:51:58.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1140" PostTypeId="2" ParentId="1106" CreationDate="2014-09-18T19:56:25.827" Score="4" Body="&lt;ul&gt;&#xA;&lt;li&gt;pre-process your documents (some of the steps may be skipped)&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Tokenization&quot; rel=&quot;nofollow&quot;&gt;tokenize&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow&quot;&gt;remove stop words&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Stemming&quot; rel=&quot;nofollow&quot;&gt;stem&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Lemmatisation&quot; rel=&quot;nofollow&quot;&gt;lemmatize&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;do normalization (e.g. U.S.A. -&gt; USA, météo -&gt; meteo, etc) and orthographic correction &lt;/li&gt;&#xA;&lt;li&gt;perform phonetic normalization (e.g. with &lt;a href=&quot;http://en.wikipedia.org/wiki/Soundex&quot; rel=&quot;nofollow&quot;&gt;Soundex&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;find equivalence classes (using thesauri, e.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/WordNet&quot; rel=&quot;nofollow&quot;&gt;WordNet&lt;/a&gt;) &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_space_model&quot; rel=&quot;nofollow&quot;&gt;Vector Space model&lt;/a&gt; to represent documents (you may use TF, aforementioned &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;TF-IDF&lt;/a&gt; or other models)&lt;/li&gt;&#xA;&lt;li&gt;do the same with the query: preprocess and represent it in the vector space&lt;/li&gt;&#xA;&lt;li&gt;find the most similar documents by computing the vector similarity (e.g. using the &lt;a href=&quot;http://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow&quot;&gt;cosine similarity&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That's an outline of the &lt;a href=&quot;http://en.wikipedia.org/wiki/Information_retrieval&quot; rel=&quot;nofollow&quot;&gt;Information Retrieval&lt;/a&gt; process &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nlp.stanford.edu/IR-book/&quot; rel=&quot;nofollow&quot;&gt;Introduction to Information Retrieval&lt;/a&gt; by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze is a very good book to get started in IR.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Or just use &lt;a href=&quot;http://lucene.apache.org/solr/&quot; rel=&quot;nofollow&quot;&gt;Apache Solr&lt;/a&gt; to get everything you need out of the box (or &lt;a href=&quot;http://lucene.apache.org/core/&quot; rel=&quot;nofollow&quot;&gt;Apache Lucene&lt;/a&gt;, that is used by Solr, to build your own application)&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2014-09-18T19:56:25.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1141" PostTypeId="1" CreationDate="2014-09-19T02:34:04.093" Score="8" ViewCount="618" Body="&lt;p&gt;I am looking to choose my career in the area of decision science or predictive modeling and I am aware that this is kind of opinion based but I would like to have some suggestion from experts that I can use it to build my career in correct path. What are the tools should I know like R, SAS or any other. What are the thinks I should know to work in a data science or machine learning or predictive modeling. For me I am having problem in identifying steps that I should follow. Please suggest me some steps to follow.&lt;/p&gt;&#xA;" OwnerUserId="3057" LastEditorUserId="97" LastEditDate="2014-09-19T12:19:25.697" LastActivityDate="2014-10-19T13:24:36.110" Title="Some suggestion for career in data science or predictive modeling" Tags="|education|beginner|career|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1142" PostTypeId="2" ParentId="1141" CreationDate="2014-09-19T07:15:13.803" Score="5" Body="&lt;p&gt;First of all I should say you question probably is an off-topic and will be closed soon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discussed at this SE site&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway I can target you to similar questions discussed at this SE site already:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/808/statistics-computer-science-data-science&quot;&gt;Statistics + Computer Science = Data Science?&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/739/starting-my-career-as-data-scientist-is-software-engineering-experience-require&quot;&gt;Starting my career as Data Scientist, is Software Engineering experience required?&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cross Validated SE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A set of relevant questions at Cross Validated Stack Exchange:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/tagged/careers&quot;&gt;https://stats.stackexchange.com/questions/tagged/careers&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data scientist map&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is good infographics of data science knowledge you might need to start a career (&lt;a href=&quot;http://pennlio.files.wordpress.com/2013/09/roadtodatascientist1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link to image&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/tJJ2x.png&quot; alt=&quot;This is good infographics of data science knowledge you might need to start a career&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Careers SE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, simple &quot;data scientist&quot; querying of Careers SE site &#xA;&lt;a href=&quot;http://careers.stackoverflow.com/jobs?searchTerm=data+scientist&amp;amp;location=&quot;&gt;http://careers.stackoverflow.com/jobs?searchTerm=data+scientist&amp;amp;location=&lt;/a&gt; will lead you to the following knowlege &quot;tags&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/r&quot;&gt;R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/bigdata&quot;&gt;bigdata&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/data-visualization&quot;&gt;data-visualization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/hadoop&quot;&gt;hadoop&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/mapreduce&quot;&gt;mapreduce&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/scala&quot;&gt;scala&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/python&quot;&gt;python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://careers.stackoverflow.com/jobs/tag/matlab&quot;&gt;matlab&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-09-19T09:04:24.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1143" PostTypeId="1" CreationDate="2014-09-19T09:08:55.180" Score="14" ViewCount="768" Body="&lt;p&gt;Recently I was introduced to the field of Data Science (its been 6 months approx), and Ii started the journey with Machine Learning Course by Andrew Ng and post that started working on the Data Science Specialization by JHU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On practical application front, I have been working on building a predictive model that would predict attrition. So far I have used glm, bayesglm, rf in an effort to learn and apply these methods, but I find a lot of gap in my understanding of these algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My basic dilemma is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Whether I should focus more on learning the intricacies of a few algorithms or should I use the approach of knowing a lot of them as and when and as much as required?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please guide me in the right direction, maybe by suggesting books or articles or anything that you think would help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be grateful if you would reply with an idea of guiding someone who has just started his career in the field of Data Science, and wants to be a person who solves practical issues for the business world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would read (as many as possible) resources (books,articles) suggested in this post and would provide a personal feed back on the pros and cons of the same so as to make this a helpful post for people who come across a similar question in future,and i think it would be great if people suggesting these books can do the same.&lt;/p&gt;&#xA;" OwnerUserId="3360" OwnerDisplayName="Vinay Tiwari" LastEditorUserId="8878" LastEditDate="2017-05-24T14:31:40.483" LastActivityDate="2021-01-15T00:09:31.857" Title="Studying machine learning algorithms: depth of understanding vs. number of algorithms" Tags="|machine-learning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1144" PostTypeId="2" ParentId="1143" CreationDate="2014-09-19T11:38:36.150" Score="5" Body="&lt;p&gt;Arguably someone calling themself a data scientist ought to know more about the intricacies of the algorithms they use—e.g. what affects the convergence rate of the Fisher scoring algorithm in GLM—than a common or garden statistician—who might be content just to know that the maximum-likelihood solution will be found (perhaps after they make a cup of coffee). In any case understanding the general concepts of statistics &amp;amp; machine learning is important in addition to familiarity with the methods you do use—the theory behind them, the assumptions they make, what diagnostic checks you should perform, how to interpret the results. Avoid being &lt;a href=&quot;https://stats.stackexchange.com/questions/104500/skills-hard-to-find-in-machine-learners/104507#104507&quot;&gt;this parody&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You'd probably enjoy reading &lt;a href=&quot;http://statweb.stanford.edu/%7Etibs/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hastie et al. (2009), &lt;em&gt;The Elements of Statistical Learning&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3361" OwnerDisplayName="Scortchi" LastEditorUserId="3361" LastEditDate="2021-01-15T00:09:31.857" LastActivityDate="2021-01-15T00:09:31.857" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="1145" PostTypeId="2" ParentId="1143" CreationDate="2014-09-19T12:39:23.103" Score="10" Body="&lt;p&gt;I would recommend limiting yourself to a few tried and trusted algorithms. I would not recommend Elements of statistical learning ( as a first book).  It is too theoretical, aimed  at graduate students, with exercises asking how to prove X or Y...  I think &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot;&gt;ISL&lt;/a&gt; is more appropriate, with more practical advice ( in any case both books are free as pdf downloads).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides statistics, I would make sure you are comfortable with experimental design/AB tests, and with Business Intelligence/Visualisation.&lt;/p&gt;&#xA;" OwnerUserId="1256" OwnerDisplayName="seanv507" LastActivityDate="2014-09-19T12:39:23.103" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1146" PostTypeId="2" ParentId="1113" CreationDate="2014-09-19T22:51:43.883" Score="2" Body="&lt;p&gt;Since it is general methodological question, let's assume we have only one text-based variable - total number of words in a sentence. First of all, it's worth to &lt;strong&gt;visualize&lt;/strong&gt; your data. I will pretend I have following data: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/KciVJ.png&quot; alt=&quot;number of words vs. age&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we see slight dependency between age and number of words in responses. We may assume that young people (approx. between 12 and 25) tend to use 1-4 words, while people of age 25-35 try to give longer answers. But how do we split these points? I would do it something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/uect9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 2D plot it looks pretty straightforward, and this is how it works most of the time in practise. However, you asked for splitting data by a single variable - age. That is, something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/G10GI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a good split? I don't know. In fact, it depends on your actual needs and interpretation of the &quot;cut points&quot;. That's why I asked about &lt;em&gt;concrete&lt;/em&gt; task. Anyway, this interpretation is up to you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practise, you will have much more text-based variables. E.g. you can use every word as a feature (don't forget to &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;stem or lemmatize&lt;/a&gt; it first) with values from zero to a number of occurrences in the response. Visualizing high-dimensional data is not an easy task, so you need a way to discover groups of data without plotting them. &lt;a href=&quot;http://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Clustering&lt;/strong&gt;&lt;/a&gt; is a general approach for this. Though clustering algorithms may work with data of arbitrary dimensionality, we still have only 2D to plot it, so let's come back to our example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With algorithm like &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;k-means&lt;/strong&gt;&lt;/a&gt; you can obtain 2 groups like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Tp4aK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two dots - red and blue - show cluster centres, calculated by k-means. You can use coordinates of these points to split your data by any subset of axes, even if you have 10k dimensions. But again, the most important question here is: &lt;em&gt;what linguistic features will provide reasonable grouping of ages&lt;/em&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-09-19T22:51:43.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1147" PostTypeId="1" AcceptedAnswerId="1175" CreationDate="2014-09-20T02:22:07.510" Score="2" ViewCount="422" Body="&lt;p&gt;One of the discussed nice aspects of the procedure that Vowpal Wabbit uses for updates to sgd &#xA;&lt;a href=&quot;http://lowrank.net/nikos/pubs/liw.pdf&quot; rel=&quot;nofollow&quot;&gt;pdf&lt;/a&gt; is so-called weight invariance, described in the linked as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Among these updates we mainly focus on a novel&#xA;set of updates that satisfies an additional invariance&#xA;property: for all importance weights of h, the update&#xA;is equivalent to two updates with importance weight&#xA;h/2. We call these updates importance invariant.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does this mean and why is it useful?&lt;/p&gt;&#xA;" OwnerUserId="1138" LastEditorUserId="1138" LastEditDate="2014-10-01T18:15:01.893" LastActivityDate="2014-10-01T18:15:01.893" Title="Invariance Property of Vowpal Wabbit Updates - Explaination" Tags="|machine-learning|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1148" PostTypeId="2" ParentId="1137" CreationDate="2014-09-21T06:01:41.097" Score="1" Body="&lt;p&gt;If you have enough data, use cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have a lot of data, use cross validation on a similar dataset, and transfer the window size (the UCR archive has a bunch of similar dataset)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget, that the best warping  window size depends on the amount of training data. As you get more data, you can have a smaller warping window, see fig 6 of &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eamonn&lt;/p&gt;&#xA;" OwnerUserId="3376" LastActivityDate="2014-09-21T06:01:41.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1149" PostTypeId="1" CreationDate="2014-09-22T15:48:32.697" Score="2" ViewCount="1216" Body="&lt;p&gt;I am working on a text classification problem on tweets. At the moment I was only considering the content of the tweets as a source of information, and I was using a simple bag of words approach using term frequencies as features, using Random Forests (this is something I cannot change). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my idea is to try to incorporate information present in the URLs used in tweets. Now, not all the tweets have URLs, and if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs. For this reason, I suppose that having a single set of features containing both the tweet term frequencies and the URL term frequencies could be bad. Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions regarding this issue? &lt;/p&gt;&#xA;" OwnerUserId="3054" LastActivityDate="2014-10-23T14:11:10.700" Title="Text Classification with mixed features in Random Forests" Tags="|text-mining|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1150" PostTypeId="2" ParentId="1105" CreationDate="2014-09-23T02:20:24.670" Score="3" Body="&lt;p&gt;This has been asked, and answered, on CrossValidated. See &lt;a href=&quot;https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression&quot;&gt;https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what&quot;&gt;https://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what&lt;/a&gt; for two related answers (and there are other related questions that you can explore there). That logistic regression can blow up is a known effect in computational statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, in situations where exp(-30) gives you roughly the relative accuracy of the double type, you would want to be extremely careful with accumulation of the round-off errors, as 1+exp(-30)=1: summing the likelihood contributions may run into numerical problems, especially when you start computing numerical derivatives and gradients. For a brief introduction into the issue in application to the typical problems in statistical computing and the specific problems it tends to encounter, see &lt;a href=&quot;http://www.stata.com/meeting/nordic-and-baltic14/abstracts/materials/dk14_gould.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.stata.com/meeting/nordic-and-baltic14/abstracts/materials/dk14_gould.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1237" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-09-23T02:20:24.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1151" PostTypeId="1" CreationDate="2014-09-23T03:34:35.750" Score="3" ViewCount="1999" Body="&lt;p&gt;I'm a Java developer and I want to pursue career in Data Science and machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please advise me where and how to begin? What subjects and mathematical/statistical skills are required and so on?&lt;/p&gt;&#xA;" OwnerUserId="3390" LastEditorUserId="2723" LastEditDate="2016-06-18T07:14:29.727" LastActivityDate="2016-06-18T07:14:29.727" Title="Prerequisites for Data Science" Tags="|machine-learning|bigdata|" AnswerCount="4" CommentCount="10" ClosedDate="2016-05-02T20:13:21.130" ContentLicense="CC BY-SA 3.0" />
  <row Id="1152" PostTypeId="2" ParentId="1149" CreationDate="2014-09-23T13:51:24.650" Score="1" Body="&lt;p&gt;Are you using raw term frequencies, or TF-IDF?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you could simply combine the terms in the tweet with the terms in the URL-linked pages (if any) into a single bag of words, calculate TF-IDF, and normalize to avoid bias towards longer documents (i.e., those tweets containing URL links).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't understand what you mean here. Aren't your features the terms in your bag of words? So the number of features will be the size of your vocabulary, which I imagine won't change much whether you include URLs or not.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't understand this either. Term-document matrices are virtually always a sparse matrix, since most of the terms in your vocabulary won't appear in most of your documents. So, the vast majority of values in your TDM will be 0. I don't know where you're getting -1 from.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-09-23T13:51:24.650" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1153" PostTypeId="2" ParentId="1106" CreationDate="2014-09-23T20:39:29.810" Score="1" Body="&lt;p&gt;I remember a long time ago playing with Elastic Search (the website is very different now from what I remember). There is some stuff about dealing with human language &lt;a href=&quot;http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/languages.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Be warned that Elastic search is like a big bazooka to your problem. If your problem is very simple, maybe you want to go from scratch. There is some docs in the web about it.&lt;/p&gt;&#xA;" OwnerUserId="3400" LastEditorUserId="98307" LastEditDate="2020-08-17T01:00:18.380" LastActivityDate="2020-08-17T01:00:18.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="1154" PostTypeId="1" AcceptedAnswerId="2471" CreationDate="2014-09-23T20:58:01.027" Score="3" ViewCount="108" Body="&lt;p&gt;I have a problem and I'm having trouble representing it - first I thought I should use graph theory (nodes and edges) and now I'm not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is some tanks names and it's volumes, those tanks are connected by pipelines which I have the names and length.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;------(pipeline 1)------.-----(pipeline 2)------.----(pipeline 3)---&#xA;  |                     |    |                           |         |&#xA;[R tank 1]        [S tank 1] [S tank 2]            (pipeline 4) [S tank 3]&#xA;                                                         |&#xA;                                                     [S tank 4]&#xA;&#xA;R tank is sink (receiver) and S tank is source (sender)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem is the pipe names change doesn't occur where there is a tank - they change name because historical reasons, size or connections...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I want to graphically show that S tank 2 is connected to pipeline 2 at point X and pipeline 2 connects to pipeline and the content goes to R tank 1, how should I do this? (I think the point X may not be relevant but if I had some way to get the distance travelled would be great).  &lt;/p&gt;&#xA;" OwnerUserId="3400" LastActivityDate="2015-01-13T17:31:51.833" Title="Trouble representing a problem" Tags="|dataset|recommender-system|graphs|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1155" PostTypeId="1" CreationDate="2014-09-24T16:25:48.803" Score="3" ViewCount="81" Body="&lt;p&gt;I couldn't quite think of how best to title this, so recommendations are welcome. Same goes for the tags (I don't have the reputation to use the tags that I thought were appropriate). The question is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Suppose you have N pairs of observations, (x,y), and you have a model with some unknown parameters, B, that estimates the relationship between x and y, F(x,B) -&gt; y. Now suppose you determine B using the method of least-squares (and, implicitly, that all the assumptions of least-squares are satisfied). The parameters, B, are themselves random variables, each with its own variance. Is there any way to estimate the reduction (or increase) in the variance of B that would result from applying the same method of least-squares to N+1 pairs of observations?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is asked in the context of experimentation. If each data point costs $X, an affirmative answer to the question would go a long way in determining whether or not to continue testing.&lt;/p&gt;&#xA;" OwnerUserId="3411" LastEditorUserId="84" LastEditDate="2014-11-25T18:27:15.063" LastActivityDate="2015-05-24T20:42:45.670" Title="Estimating Variance Reduction Resultant from Additional Data" Tags="|statistics|algorithms|experiments|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1159" PostTypeId="1" AcceptedAnswerId="1169" CreationDate="2014-09-25T08:40:59.467" Score="36" ViewCount="33107" Body="&lt;p&gt;I have a large set of data (about 8GB). I would like to use machine learning to analyze it. So, I think that I should use SVD then PCA to reduce the data dimensionality for efficiency. However, MATLAB and Octave cannot load such a large dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What tools I can use to do SVD with such a large amount of data? &lt;/p&gt;&#xA;" OwnerUserId="3167" LastEditorUserId="29169" LastEditDate="2019-06-09T17:14:32.920" LastActivityDate="2019-06-09T17:14:32.920" Title="How to do SVD and PCA with big data?" Tags="|bigdata|data-mining|dimensionality-reduction|" AnswerCount="6" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="1160" PostTypeId="2" ParentId="1113" CreationDate="2014-09-25T15:02:45.533" Score="1" Body="&lt;p&gt;If I understand you correctly, I would try a few featurization methods to transform the text column to a numeric value. Then you can proceed with analysis as usual. There is a great book on NLP called &lt;a href=&quot;http://tamingtext.com/&quot; rel=&quot;nofollow&quot;&gt;Taming Text&lt;/a&gt; that would give numerous ways to think about your text variables.&lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2014-09-25T15:02:45.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1161" PostTypeId="2" ParentId="1151" CreationDate="2014-09-25T15:20:18.660" Score="2" Body="&lt;p&gt;I would say that trying out the resources online such as the &lt;a href=&quot;https://exploredatascience.boozallen.com&quot; rel=&quot;nofollow&quot;&gt;Explore Data Science course&lt;/a&gt; and Andrew Ng's ML Course on Coursera (mentioned in the notes above) are great. However, nothing beats true data that you understand inside and out. After picking up some skills from Explore Data Science, I propose you collect your fuel usage data continuously. Collect the basics: city, state, gallons, total cost, and your vehicles mileage. When you forget to enter it, then you have to learn how to work with missing data. As you feel more comfortable, enter in kaggle contests. The data science community is growing so quickly that the online resources are vast and will help you identify any areas where a text book is what you need. Good Luck!! Have fun!&lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2014-09-25T15:20:18.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1162" PostTypeId="2" ParentId="823" CreationDate="2014-09-25T15:29:00.877" Score="2" Body="&lt;p&gt;I think ethics in Data Science is important. There is a fundamental difference in using user data to better their experience and show relevant ads and using user data to trick people into clicking on ads for the sake of monetary profit. Personally I like ads that give me relevant information like deals on things I would buy anyway. However, showing me weight loss ads because I got dumped is creepy and unethical. As my friend Peter always says, &quot;don't be creepy with data&quot;. &lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2014-09-25T15:29:00.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1163" PostTypeId="2" ParentId="1159" CreationDate="2014-09-25T17:33:09.617" Score="7" Body="&lt;p&gt;Don't bother.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First rule of programming- which also applies to data science: get everything working on a small test problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so take a random sample of your data of say 100,000 rows. try different algorithms etc.  once you have got everything working to your satisfaction, you can try larger (and larger) data sets - and see how the test error reduces as you add more data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;furthermore you do not want to apply svd to only 8 columns: you apply it when you have a lot of columns.&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-09-25T17:33:09.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1164" PostTypeId="2" ParentId="1159" CreationDate="2014-09-25T18:48:12.303" Score="1" Body="&lt;p&gt;Although you can probably find some tools that will let you do it on a single machine, you're getting into the range where it make sense to consider &quot;big data&quot; tools like Spark, especially if you think your data set might grow. Spark has a component called MLlib which supports PCA and SVD. &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html&quot; rel=&quot;nofollow&quot;&gt;The documentation has examples&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-09-25T18:48:12.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1165" PostTypeId="1" AcceptedAnswerId="5252" CreationDate="2014-09-25T20:18:46.880" Score="6" ViewCount="7390" Body="&lt;p&gt;I'm going to start a Computer Science phd this year and for that I need a research topic. I am interested in Predictive Analytics in the context of Big Data. I am interested by the area of Education (MOOCs, Online courses...). In that field, what are the unexplored areas that can help me choose a strong topic? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3433" LastEditorUserId="3433" LastEditDate="2014-09-27T16:56:14.523" LastActivityDate="2020-08-17T20:32:32.897" Title="Looking for a strong Phd Topic in Predictive Analytics in the context of Big Data" Tags="|machine-learning|bigdata|data-mining|statistics|predictive-modeling|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="1166" PostTypeId="1" CreationDate="2014-09-26T06:56:26.913" Score="2" ViewCount="3061" Body="&lt;p&gt;Matlab is a great tool for some mathematical experiments, Neural Networks, Image Processing ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if there is such a comprehensive and strong tool for data manipulation and NLP tasks? such as tokenization, POS tagging, parsing, training, testing .... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am new to NLP and I need a tool which let me experiment, get familiar and progress&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="3436" LastEditDate="2014-09-26T14:18:10.033" LastActivityDate="2017-12-14T12:37:39.437" Title="A tool like Matlab for NLP?" Tags="|data-mining|nlp|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1167" PostTypeId="2" ParentId="946" CreationDate="2014-09-26T07:47:45.600" Score="1" Body="&lt;p&gt;The correct model is binomial, both poisson and normal are just approximations. The binomial pdf is defined on the integers between zero and number of trials. The poisson is defined on the integers between 0 and infinity, and normal is on all real variables between +/- infinity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words: for a poisson there is a (possibly small) but non zero probability of having more clicks than impressions. For gaussian you can have even negative clicks. Of course, the particular parameters determine how big an impact this has... probably worth plotting the respective pdfs&lt;/p&gt;&#xA;" OwnerUserId="1256" LastEditorUserId="1256" LastEditDate="2014-09-27T08:14:16.690" LastActivityDate="2014-09-27T08:14:16.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1168" PostTypeId="2" ParentId="1166" CreationDate="2014-09-26T08:51:46.727" Score="0" Body="&lt;p&gt;Please take a look at this paper: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Survey on Various Natural Language Processing Toolkits&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this work several NLP tools are introduced. Examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Carabao Language Toolkit&lt;/li&gt;&#xA;&lt;li&gt;GATE&lt;/li&gt;&#xA;&lt;li&gt;Stanford NLP toolkit&lt;/li&gt;&#xA;&lt;li&gt;NLTK&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just google the title of the paper and you will find the .pdf file.&lt;/p&gt;&#xA;" OwnerUserId="979" LastActivityDate="2014-09-26T08:51:46.727" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1169" PostTypeId="2" ParentId="1159" CreationDate="2014-09-26T11:39:07.313" Score="46" Body="&lt;p&gt;First of all, &lt;strong&gt;dimensionality reduction&lt;/strong&gt; is used when you have &lt;strong&gt;many covariated dimensions&lt;/strong&gt; and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with memory size, but may affect dataset quality a lot. In your concrete case it's more promising to take a look at &lt;a href=&quot;http://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;online learning&lt;/strong&gt;&lt;/a&gt; methods. Roughly speaking, instead of working with the whole dataset, these methods take a little part of them (often referred to as &quot;mini-batches&quot;) at a time and build a model incrementally. (I personally like to interpret word &quot;online&quot; as a reference to some infinitely long source of data from Internet like a Twitter feed, where you just can't load the whole dataset at once). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if you really wanted to apply dimensionality reduction technique like PCA to a dataset that doesn't fit into a memory? Normally a dataset is represented as a data matrix &lt;em&gt;X&lt;/em&gt; of size &lt;em&gt;n&lt;/em&gt; x &lt;em&gt;m&lt;/em&gt;, where &lt;em&gt;n&lt;/em&gt; is number of observations (rows) and &lt;em&gt;m&lt;/em&gt; is a number of variables (columns). Typically problems with memory come from only one of these two numbers. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Too many observations (n &gt;&gt; m)&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;When you have &lt;strong&gt;too many observations&lt;/strong&gt;, but the number of variables is from small to moderate, you can &lt;strong&gt;build the covariance matrix incrementally&lt;/strong&gt;. Indeed, typical PCA consists of constructing a covariance matrix of size &lt;em&gt;m&lt;/em&gt; x &lt;em&gt;m&lt;/em&gt; and applying singular value decomposition to it. With &lt;em&gt;m&lt;/em&gt;=1000 variables of type float64, a covariance matrix has size 1000*1000*8 ~ 8Mb, which easily fits into memory and may be used with SVD. So you need only to build the covariance matrix without loading entire dataset into memory - &lt;a href=&quot;http://rebcabin.github.io/blog/2013/01/22/covariance-matrices/&quot; rel=&quot;noreferrer&quot;&gt;pretty tractable task&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can select a small representative sample from your dataset and &lt;strong&gt;approximate the covariance matrix&lt;/strong&gt;. This matrix will have all the same properties as normal, just a little bit less accurate. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Too many variables (n &amp;lt;&amp;lt; m)&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On another hand, sometimes, when you have &lt;strong&gt;too many variables&lt;/strong&gt;, the covariance matrix itself will not fit into memory. E.g. if you work with 640x480 images, every observation has 640*480=307200 variables, which results in a 703Gb covariance matrix! That's definitely not what you would like to keep in memory of your computer, or even in memory of your cluster. So we need to reduce dimensions without building a covariance matrix at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My favourite method for doing it is &lt;a href=&quot;http://web.stanford.edu/~hastie/Papers/Ping/KDD06_rp.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Random Projection&lt;/strong&gt;&lt;/a&gt;. In short, if you have dataset &lt;em&gt;X&lt;/em&gt; of size &lt;em&gt;n&lt;/em&gt; x &lt;em&gt;m&lt;/em&gt;, you can multiply it by some sparse random matrix &lt;em&gt;R&lt;/em&gt; of size &lt;em&gt;m&lt;/em&gt; x &lt;em&gt;k&lt;/em&gt; (with &lt;em&gt;k&lt;/em&gt; &amp;lt;&amp;lt; &lt;em&gt;m&lt;/em&gt;) and obtain new matrix &lt;em&gt;X'&lt;/em&gt; of a much smaller size &lt;em&gt;n&lt;/em&gt; x &lt;em&gt;k&lt;/em&gt; with &lt;em&gt;approximately the same properties&lt;/em&gt; as the original one. Why does it work? Well, you should know that PCA aims to find set of orthogonal axes (principal components) and project your data onto first &lt;em&gt;k&lt;/em&gt; of them. It turns out that sparse random vectors are &lt;em&gt;nearly orthogonal&lt;/em&gt; and thus may also be used as a new basis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, of course, you don't have to multiply the whole dataset &lt;em&gt;X&lt;/em&gt; by &lt;em&gt;R&lt;/em&gt; - you can translate every observation &lt;em&gt;x&lt;/em&gt; into the new basis separately or in mini-batches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also somewhat similar algorithm called &lt;strong&gt;Random SVD&lt;/strong&gt;. I don't have any real experience with it, but you can find example code with explanations &lt;a href=&quot;https://stats.stackexchange.com/a/11934/3305&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a bottom line, here's a short check list for dimensionality reduction of big datasets: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you have not that many dimensions (variables), simply use online learning algorithms. &lt;/li&gt;&#xA;&lt;li&gt;If there are many observations, but a moderate number of variables (covariance matrix fits into memory), construct the matrix incrementally and use normal SVD. &lt;/li&gt;&#xA;&lt;li&gt;If number of variables is too high, use incremental algorithms. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-14T21:24:09.753" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1171" PostTypeId="2" ParentId="1166" CreationDate="2014-09-26T16:15:05.580" Score="0" Body="&lt;p&gt;There's a NLP toolbox for MATLAB called &lt;a href=&quot;https://github.com/faridani/MatlabNLP&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;MatlabNLP&lt;/strong&gt;&lt;/a&gt;. It includes modules for tokenization, preprocessing (stop word removal, text cleaning, stemming), and learning algorithms (linear regression, decision trees, support vector machines and a Naïve Bayes). A module for POS tagging is coming soon.&lt;/p&gt;&#xA;" OwnerUserId="3412" LastActivityDate="2014-09-26T16:15:05.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1172" PostTypeId="1" CreationDate="2014-09-27T13:14:16.710" Score="2" ViewCount="52" Body="&lt;p&gt;Are there any general open-source programs or libraries (e.g., a Python library) for analyzing user search behavior?  By &quot;search behavior&quot;, I mean a user's interaction with a search engine, such as querying, clicking relevant results, and spending time on those results.  I'd like something with the following properties - it doesn't have to be all of them, but the more the merrier:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Models individual user behavior (aggregate and time-based)&lt;/li&gt;&#xA;&lt;li&gt;Models group user behavior&lt;/li&gt;&#xA;&lt;li&gt;Simulates individual user behavior, given a model&lt;/li&gt;&#xA;&lt;li&gt;Is easily extensible (to accept data input formats, user models, document models, etc., that end-users define)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Links are a plus!&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2020-02-07T21:35:08.583" Title="General programs/libraries for studying user search behavior?" Tags="|machine-learning|data-mining|search|library|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1173" PostTypeId="1" AcceptedAnswerId="1182" CreationDate="2014-09-27T21:58:42.477" Score="1" ViewCount="76" Body="&lt;p&gt;Not sure if this is Math, Stats or Data Science, but I figured I would post it here to get the site used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a programmer, when you have a system/component implemented, you might want to allow some performance monitoring. For example to query how often a function call was used, how long it took and so on. So typically you care about count, means/percentile, max/min and similiar statistics. This could be measurements since startup, but also a rolling average or window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if there is a good data structure which can be updated efficiently concurrently which can be used as the source for most of those queries. For example having a ringbuffer of rollup-metrics (count, sum, min, max) over increasing periods of time and a background aggregate process triggered regularly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The focus here (for me) is on in-memory data structures with limited memory consumption. (For other things I would use a RRD type of library).&lt;/p&gt;&#xA;" OwnerUserId="3445" LastActivityDate="2014-09-30T14:29:06.307" Title="Versatile data structure for combined statistics" Tags="|dataset|statistics|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1174" PostTypeId="1" CreationDate="2014-09-28T12:51:43.823" Score="4" ViewCount="12749" Body="&lt;p&gt;I have post already the question few months ago about my project that I'm starting to work on. This post can be see here: &#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/211/human-activity-recognition-using-smartphone-data-set-problem&quot;&gt;Human activity recognition using smartphone data set problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I know this is based around multivariate time series analysis and tasks are to classify and cluster the data. I have gathered some materials (e-books, tutorials etc.) on this but still can't see a more detailed picture of how even I should start. Here's the tutorial that looks like it might be helpful but the thing is my data looks differently and I'm not really sure if this can be applied to my work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://little-book-of-r-for-multivariate-analysis.readthedocs.org/en/latest/src/multivariateanalysis.html#scatterplots-of-the-principal-components&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://little-book-of-r-for-multivariate-analysis.readthedocs.org/en/latest/src/multivariateanalysis.html#scatterplots-of-the-principal-components&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How I can start on some very basic analysis? How to read data so it any meaning for me.&#xA;Any tips and advises will be much appreciated!&#xA;Note: I'm just the beginner in data science.  &lt;/p&gt;&#xA;" OwnerUserId="295" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2021-02-12T12:52:09.447" Title="How to classify and cluster this time series data" Tags="|data-mining|classification|dataset|clustering|time-series|" AnswerCount="2" CommentCount="1" ClosedDate="2021-02-12T14:22:38.760" ContentLicense="CC BY-SA 3.0" />
  <row Id="1175" PostTypeId="2" ParentId="1147" CreationDate="2014-09-29T00:20:24.093" Score="2" Body="&lt;p&gt;Often different data samples have different weighting ( eg the costs of misclassification error  for one group of data  is higher than for other classes).&#xA;Most error metrics are of the form $\sum_i e_i$ where e_i is the loss ( eg squared error) on data point $i$. Therefore weightings of the form $\sum_i w_i e_i$ are equivalent to duplicating the data w_i times (eg for w_i integer). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple case is if you have repeated data - rather than keeping all the duplicated data points, you just &quot;weight&quot; your one repeated sample by the number of instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now whilst this is easy to do in a batch setting, it is hard in vowpal wabbits online big data setting: given that you have a large data set, you do not just want to represent the data n times to deal with the weighting ( because it increases your computational load). Similarly, just multiplying the gradient vector by the weighting - which is correct in batch gradient descent - will cause big problems for stochastic/online gradient descent: essentially you shoot off in one direction ( think of large integer weights) then you shoot off in the other - causing significant instability. SGD essentially relies on all the errors to be of roughly the same order ( so that the learning rate can be set appropriately). So what they propose is to ensure that the update for training sample x_i with weight n is equivalent to presenting training sample x_i n times consecutively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea being that presenting it consecutively reduces the problem because the error gradient (for that single example $x_i$) reduces for each consecutive presentation and update (as you get closer &amp;amp; closer to the minimum for that specific example). In other words the consecutive updates provides a kind of feedback control. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me it sounds like you would still have instabilities (you get to zero error on x_i, then you get to zero error on x_i+1,...). the learning rate will need to be adjusted to take into account the size of the weights.&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-09-29T00:20:24.093" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="1176" PostTypeId="2" ParentId="1159" CreationDate="2014-09-29T00:28:26.413" Score="7" Body="&lt;p&gt;PCA is usually implemented by computing SVD on the covariance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the covariance matrix is an &lt;strong&gt;embarrassingly parallel&lt;/strong&gt; task, so it scales &lt;em&gt;linear&lt;/em&gt; with the number of records, and is trivial to distribute on multiple machines!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just do one pass over your data to compute the means. Then a second pass to compute the covariance matrix. This can be done with map-reduce easily - essentially it's the same as computing the means again. Sum terms as in covariance are trivial to parallelize! You may only need to pay attention to numerics when summing a lot of values of similar magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things get different when you have a huge number of &lt;em&gt;variables&lt;/em&gt;. But on an 8 GB system, you should be able to run PCA on up to 20.000 dimensions in-memory with the BLAS libraries. But then you may run into the problem that PCA isn't all that reliable anymore, because it has too many degrees of freedom. In other words: it overfits easily. I've seen the recommendation of having at least 10*d*d records (or was it d^3). So for 10000 dimensions, you should have at least a billion records (of 10000 dimensions... that is a lot!) for the result to be statistically reliable.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-09-29T00:28:26.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1177" PostTypeId="1" CreationDate="2014-09-29T14:52:48.130" Score="6" ViewCount="330" Body="&lt;p&gt;I'm trying to implement item based collaborative filtering. Do any distance calculations allow for weighting of certain ranges of values within each vector? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I would like to be able to say values 10..22 within each vector are more significant than values within the range 0..10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been experimenting with Pearson, Tanimoto and Euclidean algorithms, but they all seem to assume equal weighting for each value within the vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I approaching this problem in the right way, and if not, how do others deal with this problem? &lt;/p&gt;&#xA;" OwnerUserId="3459" LastActivityDate="2015-11-11T16:50:46.357" Title="Distance calculation/vector range significance" Tags="|machine-learning|classification|algorithms|beginner|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1178" PostTypeId="2" ParentId="1042" CreationDate="2014-09-29T17:46:58.617" Score="2" Body="&lt;p&gt;The kinds of tools you will use will vary based on the problem you are trying to solve. Social media data is rich and therefore many questions can be asked - and many tools can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is a general pattern you might keep in mind. Typically, you will have to use the platform's API to gather data. You will then have to normalized and store the data in a data warehouse. Finally, you will access and analyze the data with the tools you desire. Keeping the end goal in mind, you will have to strategically choose the best technologies for the job you are doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say you wanted to study the relationships between users on a social network - a question like &quot;Who are the mutual friends of the most popular individuals in this social network?&quot; In this case, you would gather data using the social media platform's API, normalize it into a CSV, import it into a Neo4j database, and then use Cypher to make queries. &lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-09-29T17:46:58.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1179" PostTypeId="2" ParentId="1177" CreationDate="2014-09-29T20:31:49.387" Score="1" Body="&lt;p&gt;I created a scoring system (&quot;Thomas Scoring System&quot;) to deal with this problem.  If you treat &quot;distance&quot; as a similarity score, this system should work for you. &lt;a href=&quot;http://exploringpossibilityspace.blogspot.com/2014/02/thomas-scoring-system.html&quot; rel=&quot;nofollow&quot;&gt;http://exploringpossibilityspace.blogspot.com/2014/02/thomas-scoring-system.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here's Thomas Scoring System (TSS) in a nutshell: Treat each metric and their values as evidence that weigh for or against specific values in the performance index. This is an inference process and not an arithmetic calculation, as in the Usual Method. The output of TSS is an estimation of the Weight of Evidence for all index values.  The Weight of Evidence can be expressed as a probability distribution, where all the evidence weights sum to one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To use your example, each range is a condition: A) 10 &amp;lt;= &lt;em&gt;x&lt;/em&gt; &amp;lt;= 22; and B) &lt;em&gt;x&lt;/em&gt; &amp;lt; 10. If A is true, then it weighs in favor of certain similarity (distance) scores.  If B is true, then it weights in favor of other similarity (distance) scores, but would have less weight compared to other evidence.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-09-29T20:31:49.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1180" PostTypeId="2" ParentId="1177" CreationDate="2014-09-29T22:45:38.623" Score="1" Body="&lt;p&gt;You mention distance metrics but Pearson and Tanimoto are not. For Euclidean distance, simply scale the dimensions in question by some factor. For example doubling a dimension's values makes its contribution to distance larger and so makes the feature more important to the distance metric. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a related note you may wish to look at the Mahalanobis distance which kind of accounts for covariance or 'redundancy' across dimensions. &lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-09-29T22:45:38.623" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="1181" PostTypeId="1" CreationDate="2014-09-30T09:43:24.153" Score="-1" ViewCount="551" Body="&lt;p&gt;I want the text-based semantic clustering EMD do.&#xA;Is there a better way of using LDA to detect topics in text, there are so provide better results?&#xA;I'm going to do my EMD on discovery topics.&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3473" LastActivityDate="2018-12-04T09:17:15.800" Title="document clustering by semantic similarity based EMD(earth mover distance)" Tags="|text-mining|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1182" PostTypeId="2" ParentId="1173" CreationDate="2014-09-30T14:29:06.307" Score="1" Body="&lt;p&gt;It sounds like you would like the &lt;a href=&quot;http://www.boost.org/doc/libs/1_56_0/doc/html/accumulators.html&quot; rel=&quot;nofollow&quot;&gt;Boost Accumulators&lt;/a&gt; library:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Boost.Accumulators is both a library for incremental statistical&#xA;  computation as well as an extensible framework for incremental&#xA;  calculation in general. The library deals primarily with the concept&#xA;  of an accumulator, which is a primitive computational entity that&#xA;  accepts data one sample at a time and maintains some internal state.&#xA;  These accumulators may offload some of their computations on other&#xA;  accumulators, on which they depend. Accumulators are grouped within an&#xA;  accumulator set. Boost.Accumulators resolves the inter-dependencies&#xA;  between accumulators in a set and ensures that accumulators are&#xA;  processed in the proper order.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="515" LastActivityDate="2014-09-30T14:29:06.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1183" PostTypeId="1" CreationDate="2014-09-30T19:53:20.163" Score="2" ViewCount="49" Body="&lt;p&gt;I'm interested in discovering some kind of dis-associations between the periods of a time series based on its data, e.g., find some (unknown number of) periods where the data is not similar with the data from another period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I would like to compare the same data but over 2 years (something like DTW?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get my data Excel as a two-column list:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c1=date (one per each day of the year), c2=Data To Analyze&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, what algorithms could I use and in what software?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update/Later edit:&lt;/strong&gt;&#xA;I'm looking for dates as cut-off points from which the DataToAnalyze could be part of another cluster of consecutive dates. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2014-1-1 --&amp;gt; 2014-3-10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;are part of &lt;em&gt;Cluster_1&lt;/em&gt; based on DataToAnalyze. And:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2014-3-11 --&amp;gt; 2014-5-2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;are part of &lt;em&gt;Cluster_2&lt;/em&gt; based on DataToAnalyze, and so on. So, clusters of consecutive dates should be automatically determined based on some algorithms, which is what I'm looking for. Which ones (or which software) would be applicable to this problem?&lt;/p&gt;&#xA;" OwnerUserId="3482" LastEditorUserId="84" LastEditDate="2014-10-02T13:36:58.920" LastActivityDate="2014-10-02T13:36:58.920" Title="Discovering dis-associations between periods of time-series" Tags="|clustering|time-series|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1184" PostTypeId="2" ParentId="1159" CreationDate="2014-09-30T20:12:32.870" Score="0" Body="&lt;p&gt;I would reccomend python if you lazily evaluate the file you will have a miniscule memory footprint, and numpy/scipy give you access to all of the tools Octave/Matlab would. &lt;/p&gt;&#xA;" OwnerUserId="890" LastActivityDate="2014-09-30T20:12:32.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1185" PostTypeId="1" CreationDate="2014-10-01T10:32:32.960" Score="2" ViewCount="68" Body="&lt;p&gt;I'm trying to implement GD for standard task of NN training :) The best papers for practioneer I've founded so far are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &quot;Efficient BackProp&quot; by Yann LeCun et al.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &quot;Stochastic Gradient Descent Tricks&quot; by Leon Bottou&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there some other must read papers on this topic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="2471" LastActivityDate="2015-03-09T00:24:48.157" Title="What are some best papers on gradient descent for NN implementation?" Tags="|neural-network|gradient-descent|" AnswerCount="0" CommentCount="3" ClosedDate="2015-08-30T08:20:16.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="1186" PostTypeId="1" CreationDate="2014-10-01T13:26:52.037" Score="11" ViewCount="319" Body="&lt;p&gt;I am planning to use scikit linear support vector machine (SVM) classifier for text classification on a corpus consisting of 1 million labeled documents. What I am planning to do is, when a user enters some keyword, the classifier will first classify it in a category, and then a subsequent information retrieval query will happen in within the documents of that category catagory. I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I confirm that classification will not take much time? I don't want users to have to spend time waiting for a classification to finish in order to get better results.&lt;/li&gt;&#xA;&lt;li&gt;Is using Python's scikit library for websites/web applications suitable for this?&lt;/li&gt;&#xA;&lt;li&gt;Does anyone know how amazon or flipkart perform classification on user queries, or do they use a completely different logic?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3498" LastEditorUserId="13413" LastEditDate="2015-10-24T00:20:30.280" LastActivityDate="2016-01-21T11:29:32.980" Title="How much time do scikit classifiers take to classify?" Tags="|machine-learning|classification|python|scikit-learn|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1188" PostTypeId="2" ParentId="1174" CreationDate="2014-10-01T14:44:35.467" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How I can start on some very basic analysis?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Take your labeled data and compute histograms of the values for each of the sets. Plot these and visually see if there's any differences. Also compute the mean and variance of each of the different labeled sets and see if there are differences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's timeseries data, take small (overlapping) windows of time and compute various metrics - min, max, variance, mean, for instance - and use that as input to a classifier.&lt;/p&gt;&#xA;" OwnerUserId="403" LastActivityDate="2014-10-01T14:44:35.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1189" PostTypeId="1" AcceptedAnswerId="2472" CreationDate="2014-10-01T18:11:26.220" Score="3" ViewCount="599" Body="&lt;p&gt;Below is the dataset where the response variable is play with two labels (yes, and no):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;No. outlook temperature humidity    windy   play&#xA;1   sunny       hot     high        FALSE   no&#xA;2   sunny       hot     high        TRUE    no&#xA;3   overcast    hot     high        FALSE   yes&#xA;4   rainy       mild    high        FALSE   yes&#xA;5   rainy       cool    normal      FALSE   yes&#xA;6   rainy       cool    normal      TRUE    no&#xA;7   overcast    cool    normal      TRUE    yes&#xA;8   sunny       mild    high        FALSE   no&#xA;9   sunny       cool    normal      FALSE   yes&#xA;10  rainy       mild    normal      FALSE   yes&#xA;11  sunny       mild    normal      TRUE    yes&#xA;12  overcast    mild    high        TRUE    yes&#xA;13  overcast    hot     normal      FALSE   yes&#xA;14  rainy       mild    high        TRUE    no&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are the decisions with their respective classifications: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1: (outlook,overcast) -&amp;gt; (play,yes) &#xA;[Support=0.29 , Confidence=1.00 , Correctly Classify= 3, 7, 12, 13]&#xA;&#xA;2: (humidity,normal), (windy,FALSE) -&amp;gt; (play,yes)&#xA;[Support=0.29 , Confidence=1.00 , Correctly Classify= 5, 9, 10]&#xA;&#xA;3: (outlook,sunny), (humidity,high) -&amp;gt; (play,no) &#xA;[Support=0.21 , Confidence=1.00 , Correctly Classify= 1, 2, 8]&#xA;&#xA;4: (outlook,rainy), (windy,FALSE) -&amp;gt; (play,yes) &#xA;[Support=0.21 , Confidence=1.00 , Correctly Classify= 4]&#xA;&#xA;5: (outlook,sunny), (humidity,normal) -&amp;gt; (play,yes) &#xA;[Support=0.14 , Confidence=1.00 , Correctly Classify= 11]&#xA;&#xA;6: (outlook,rainy), (windy,TRUE) -&amp;gt; (play,no) &#xA;[Support=0.14 , Confidence=1.00 , Correctly Classify= 6, 14]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3503" LastEditorUserId="134" LastEditDate="2018-12-27T14:56:31.230" LastActivityDate="2018-12-27T14:56:31.230" Title="How to define confusion matrix for classification?" Tags="|confusion-matrix|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="1190" PostTypeId="1" CreationDate="2014-10-01T19:51:21.220" Score="4" ViewCount="1570" Body="&lt;p&gt;Statement of problem: An ambulance is at the hospital dropping off a patient. The goal of the paramedic is to get released from the hospital as soon as possible. I am curious, what are the factors in how long an ambulance off loads a patient at the hospital? Can I predict how long an offload will take given certain variables. And how confident can I be in this model? The Dependent Variable is HospitalTime, it is a ratio type of data and is measured in seconds. The Independent Variables are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Hospital, a nominal type of data recoded into integers, 1 would stand&#xA;for Lee Memorial.&lt;/li&gt;&#xA;&lt;li&gt;Ambulance, a nominal type of data recoded into integers, 9 would&#xA;stand for ambulance #9&lt;/li&gt;&#xA;&lt;li&gt;PatientPriority is an ordinal type of data recoded into integers. A 1&#xA;is a high priority, 2 is a medium priority and 3 is low acuity.&lt;/li&gt;&#xA;&lt;li&gt;MonthOfCall is an interval type of data recoded into integers. A 6&#xA;would be June and 12 is December. A 12 (December) is not twice as&#xA;much as a 6 (June) in this case.&lt;/li&gt;&#xA;&lt;li&gt;HourOfCall is an interval type of data recoded into integers. Once&#xA;again, an offload happening at 10:00 pm is not more than something&#xA;happening at 10:00 am.&lt;/li&gt;&#xA;&lt;li&gt;Officer1 and Officer2 are nominal data and are integers representing&#xA;an EMT and a paramedic.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is this: Given this type of data and my goal to predict the off loading time at the hospital, what kind of regression model should I look into?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at my statistics books from university days and they are all using ratio data. My data is mixed with nominal, ordinal, interval and ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have as much data as you could ask for. I have at least 100,000 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please push me in the right direction? What kind of model should I use with this type of data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shown below are observations to give you a tiny peek at my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IncidentID,HospitalTime,Hospital,Ambulance,PatientPriority,MonthOfCall,HourOfCall,Officer1,Officer2&#xA;757620,1849,7,11,2,10,10,234,771,chr(10) 802611,2625,7,11,3,1,18,234,777,chr(10) &#xA;765597,1149,7,12,3,11,2,234,777,chr(10) 770926,1785,7,12,3,11,15,234,777,chr(10) &#xA;771689,3557,7,12,2,11,14,234,777,chr(10) 822758,1073,7,20,3,3,13,777,307,chr(10) &#xA;767249,2570,7,22,2,11,11,560,778,chr(10) 767326,1998,7,22,1,11,18,560,777,chr(10) &#xA;785903,1660,7,22,3,12,12,234,777,chr(10) 787644,2852,7,22,3,12,17,234,777,chr(10) &#xA;760294,1327,7,23,2,10,14,498,735,chr(10) 994677,3653,7,32,2,2,15,181,159,chr(10) &#xA;994677,3653,7,32,2,2,15,181,159,chr(10) 788471,2053,5,9,2,1,3,498,777,chr(10) &#xA;788471,2053,5,9,2,1,3,498,777,chr(10) 759983,1342,5,11,2,10,8,474,777,chr(10)&#xA;791243,1635,5,11,2,1,18,234,777,chr(10) 800796,1381,5,11,3,1,11,234,777,chr(10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;P.S. This question is cross-posted in Stack-Overflow under the same title and author.&lt;/p&gt;&#xA;" OwnerUserId="3504" LastEditorUserId="21" LastEditDate="2014-10-14T08:24:14.660" LastActivityDate="2015-04-16T11:14:29.710" Title="Type of regression with nominal, ordinal, interval and ratio data" Tags="|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1191" PostTypeId="1" CreationDate="2014-10-02T03:30:37.930" Score="4" ViewCount="6913" Body="&lt;p&gt;The dataset that I am experimenting with is in the form of a table with columns userid and itemid. If there is a row for a given user and a given item, that means the user accessed the item (like in an online store). I am trying to cluster similar items based on this data. If a pair of items is accessed together often, then the items are similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because this is a case of a high dimensionality (# of users and items will be in 10,000's) I think I am justified in trying to use SVD as a pre-clustering step and then do some classical clustering. When I tried doing this I got poor clustering results when compared with simple hierarchical clustering. Items that weren't very similar were being bucketed together in one dimension, while there were available dimensions that weren't used. The results weren't completely random, but they were definitely worse than the output from the hierarchical clustering. I attempted the SVD step with Mahaut and Octave and the results were similar. For the hierarchical clustering I used the Jaccard measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point I am starting to doubt the notion of SVD as a way to reduce dimensionality. Do you think that SVD cannot be used effectively in this case (and why?) or do you think that I made some mistake along the way?&lt;/p&gt;&#xA;" OwnerUserId="3506" LastEditorUserId="84" LastEditDate="2014-10-02T12:36:46.010" LastActivityDate="2014-10-02T12:36:46.010" Title="Using SVD for clustering" Tags="|clustering|" AnswerCount="1" CommentCount="14" ContentLicense="CC BY-SA 3.0" />
  <row Id="1192" PostTypeId="1" CreationDate="2014-10-02T06:18:11.397" Score="5" ViewCount="139" Body="&lt;p&gt;I have several datasets with thousands of variables. This different datasets have different variables for the same thing. Is there a way to automatically/semi-automatically check compatible variables and make them consistent?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is such thing, that would save me months of tedious work. The data is stored in SPSS format.&lt;/p&gt;&#xA;" OwnerUserId="3507" LastEditorUserId="84" LastEditDate="2014-10-02T13:25:16.317" LastActivityDate="2017-11-19T21:58:07.593" Title="Amalgamating multiple datasets with different variables coding" Tags="|dataset|data-cleaning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1193" PostTypeId="2" ParentId="1186" CreationDate="2014-10-02T08:17:03.253" Score="3" Body="&lt;p&gt;The only reliable way to see how long it takes is to code it up and give it a shot.  Training will take more time, then you can save your model (pickle) to use later.&lt;/p&gt;&#xA;" OwnerUserId="3457" LastActivityDate="2014-10-02T08:17:03.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1194" PostTypeId="2" ParentId="1191" CreationDate="2014-10-02T08:58:18.340" Score="2" Body="&lt;p&gt;We are using Singular Value Decomposition in the much same manner as you, except rather than clustering similar items, we are using a reduced-rank matrix to power a recommendation engine based on a term-document matrix in Latent Semantic Indexing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your brief description, your approach seems sound enough. However, I highly recommend reading &lt;a href=&quot;http://lsirwww.epfl.ch/courses/dis/2003ws/papers/ut-cs-94-270.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Berry, Dumais &amp;amp; O'Brien's Using Linear Algebra for Intelligent Information Retrieval&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Key to using SVD is selecting an acceptable rank-k approximation to the original sparse matrix. You should carry out some exploratory analysis to see how much variance can be explained, using the singular values in the diagonal matrix Sigma. This question was brought up in &lt;a href=&quot;https://stats.stackexchange.com/questions/49646/for-a-random-matrix-shouldnt-a-svd-explain-nothing-at-all-what-am-i-doing-wro&quot;&gt;this question on CrossValidated&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of the papers I've read suggest anywhere a rank k from 200 to 300 singular values. In a proof-of-concept implementation, we had original sparse matrix of about 10000 rows (unique terms) to about 1000 columns (unique documents), and we were capturing just under 85% of the variance with only 300 singular values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, that really hinges upon the nature of your data, so your mileage may vary.&lt;/p&gt;&#xA;" OwnerUserId="24" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-10-02T08:58:18.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1195" PostTypeId="1" AcceptedAnswerId="1206" CreationDate="2014-10-02T13:06:22.433" Score="2" ViewCount="154" Body="&lt;p&gt;I'm coding a program that tests several classifiers over a database weather.arff, I found rules below, I want classify test objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand how the classification, it is described:&#xA;&quot;In classification, let R be the set of generated rules and T the training data. The basic idea of the proposed method is to choose a set of high confidence rules in R to cover T. In classifying a test object, the first rule in the set of rules that matches the test object condition classifies it. This process ensures that only the highest ranked rules classify test objects. &quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to classify test objects?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;No. outlook temperature humidity    windy   play&#xA;1   sunny       hot     high        FALSE   no&#xA;2   sunny       hot     high        TRUE    no&#xA;3   overcast    hot     high        FALSE   yes&#xA;4   rainy       mild    high        FALSE   yes&#xA;5   rainy       cool    normal      FALSE   yes&#xA;6   rainy       cool    normal      TRUE    no&#xA;7   overcast    cool    normal      TRUE    yes&#xA;8   sunny       mild    high        FALSE   no&#xA;9   sunny       cool    normal      FALSE   yes&#xA;10  rainy       mild    normal      FALSE   yes&#xA;11  sunny       mild    normal      TRUE    yes&#xA;12  overcast    mild    high        TRUE    yes&#xA;13  overcast    hot     normal      FALSE   yes&#xA;14  rainy       mild    high        TRUE    no&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Rule found:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1: (outlook,overcast) -&amp;gt; (play,yes) &#xA;[Support=0.29 , Confidence=1.00 , Correctly Classify= 3, 7, 12, 13]&#xA;&#xA;2: (humidity,normal), (windy,FALSE) -&amp;gt; (play,yes)&#xA;[Support=0.29 , Confidence=1.00 , Correctly Classify= 5, 9, 10]&#xA;&#xA;3: (outlook,sunny), (humidity,high) -&amp;gt; (play,no) &#xA;[Support=0.21 , Confidence=1.00 , Correctly Classify= 1, 2, 8]&#xA;&#xA;4: (outlook,rainy), (windy,FALSE) -&amp;gt; (play,yes) &#xA;[Support=0.21 , Confidence=1.00 , Correctly Classify= 4]&#xA;&#xA;5: (outlook,sunny), (humidity,normal) -&amp;gt; (play,yes) &#xA;[Support=0.14 , Confidence=1.00 , Correctly Classify= 11]&#xA;&#xA;6: (outlook,rainy), (windy,TRUE) -&amp;gt; (play,no) &#xA;[Support=0.14 , Confidence=1.00 , Correctly Classify= 6, 14]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Dung&lt;/p&gt;&#xA;" OwnerUserId="3503" LastEditorUserId="8501" LastEditDate="2016-12-13T09:18:45.437" LastActivityDate="2016-12-13T09:18:45.437" Title="How to classify test objects with this ruleset in order of priority?" Tags="|classification|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1196" PostTypeId="1" CreationDate="2014-10-02T17:32:42.003" Score="2" ViewCount="194" Body="&lt;p&gt;Many discussions of missing data in supervised (and unsupervised) learning deal with various methods of imputation, like mean values or EM. But in some cases the data will be missing as a necessary consequence of the data generation process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, let's say I'm trying to predict students' grades, and one of the inputs I want to analyze is the average grades of the student's siblings. If a particular student is an only child, then that value will be missing, not because we failed to collect the data, but because logically there is no data to collect. This is distinct from cases where the student has siblings, but we can't find their grades. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other examples abound: say we're in college admissions and we want to include students' AP exam results, but not all students took AP exams. Or we're looking at social network data, but not all subjects have facebook and/or twitter accounts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These data are missing, but they're certainly not missing at random. And many algorithms, such as all supervised learning packages in scikit-learn, simply demand that there be no missing values at all in the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How have people dealt with this in the past, and what off-the-shelf solutions are there? For instance, I believe the gradient boosting algorithm in R uses trees with three possible branches: left, right, and missing. Any other alternatives out there?&lt;/p&gt;&#xA;" OwnerUserId="3510" LastActivityDate="2014-10-03T11:22:55.273" Title="Supervised Learning with Necessarily Missing Data" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1197" PostTypeId="1" CreationDate="2014-10-02T23:15:27.050" Score="11" ViewCount="2184" Body="&lt;p&gt;I've made a Naive Bayes classifier that uses the bag-of-words technique to classify spam posts on a message board. It works, but I think I could get much better results if my models considered the word orderings and phrases. (ex: 'girls' and 'live' may not trigger a high spam score, even though 'live girls' is most likely junk). How can I build a model that takes word ordering into account?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've considered storing n-grams (check-out-these, out-these-live, these-live-girls), but this seems to radically increase the size of the dictionary I keep score in and causes inconsistency as phrases with very similar wording but different order will slip through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not tied to Bayesian classification, but I'd like something that someone without a strong background in statistics could grok and implement. &lt;/p&gt;&#xA;" OwnerUserId="3513" LastEditorUserId="29575" LastEditDate="2017-11-13T14:20:37.103" LastActivityDate="2017-11-13T14:20:37.103" Title="How can I classify text considering word order, instead of just using a bag-of-words approach?" Tags="|classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1198" PostTypeId="2" ParentId="155" CreationDate="2014-10-03T00:25:48.490" Score="12" Body="&lt;p&gt;As you mentioned, the API is the hard part, not the data.  &lt;a href=&quot;http://quandl.com&quot;&gt;Quandl&lt;/a&gt; seems to solve this problem by providing over 10 million publicly available data sets under one easy, RESTful API.  If programming isn't your strong suit, there is a free tool to make loading data into Excel very easy.  Additionally, if you &lt;em&gt;do&lt;/em&gt; enjoy programming, there are several native libraries in &lt;a href=&quot;https://www.quandl.com/help/libraries&quot;&gt;R, Python, Java and more&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3514" LastActivityDate="2014-10-03T00:25:48.490" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="1199" PostTypeId="2" ParentId="1196" CreationDate="2014-10-03T11:22:55.273" Score="1" Body="&lt;p&gt;I solved a similar problem not so long ago. Let X be a numerical variable that has missing values. First, we assign the value 0 to those instances in which the value is missing. Then, we add a new categorical variable to our model, X_missing, which domain is {True,False}. We obtain a data model with mixed numerical/categorical variables. You can then apply gradient descent to train a regression model from these variables. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2014-10-03T11:22:55.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1200" PostTypeId="2" ParentId="1190" CreationDate="2014-10-03T11:39:11.240" Score="2" Body="&lt;p&gt;I think I'd try to model the &lt;em&gt;rate&lt;/em&gt; at which the ambulances are released instead of the time it takes. That would let me use Poisson regression, which is the canonical type of GLM for rates (in R's GLM, set family = &quot;poisson&quot;.) However, in order to use PR, the data needs to have its variance equal to its mean, or at least close to it. &lt;/p&gt;&#xA;" OwnerUserId="1241" LastActivityDate="2014-10-03T11:39:11.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1202" PostTypeId="2" ParentId="1197" CreationDate="2014-10-03T16:07:30.327" Score="5" Body="&lt;p&gt;Try out some generative models like HMM. &#xA;Just check the following link:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/91290/how-do-i-train-hmms-for-classification&quot;&gt;https://stats.stackexchange.com/questions/91290/how-do-i-train-hmms-for-classification&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-10-03T16:07:30.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1203" PostTypeId="2" ParentId="1197" CreationDate="2014-10-03T18:05:13.163" Score="7" Body="&lt;p&gt;There is a very simple hack to incorporate word order in an existing bag-of-words model implementation. Treat some of the phrases, such as the frequently occurring bi-grams (e.g. New York) as a unit, i.e. a single word instead of treating them as separate entities. This will ensure that &quot;New York&quot; is different from &quot;York New&quot;. You could also define higher order word shingles such as for n=3,4 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use the Lucene &lt;a href=&quot;http://lucene.apache.org/core/4_6_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilter.html&quot; rel=&quot;noreferrer&quot;&gt;ShingleFilter&lt;/a&gt; to decompose your document text into shingles as a pre-processing step and then apply the classifier on this decomposed text.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import java.io.*;&#xA;import org.apache.lucene.analysis.core.*;&#xA;import org.apache.lucene.analysis.*;&#xA;import org.apache.lucene.analysis.shingle.ShingleFilter;&#xA;import org.apache.lucene.analysis.standard.*;&#xA;import org.apache.lucene.util.*;&#xA;import org.apache.lucene.analysis.util.*;&#xA;import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;&#xA;import org.apache.lucene.analysis.charfilter.*;&#xA;import org.apache.lucene.analysis.core.WhitespaceTokenizer;&#xA;&#xA;class TestAnalyzer extends Analyzer {&#xA;&#xA;    TestAnalyzer() {&#xA;        super();&#xA;    }&#xA;&#xA;    protected TokenStreamComponents createComponents( String fieldName, Reader reader ) {&#xA;        String token;&#xA;        TokenStream result = null;&#xA;&#xA;        Tokenizer source = new WhitespaceTokenizer( Version.LUCENE_CURRENT, reader );&#xA;        result = new ShingleFilter(source, 2, 2);&#xA;&#xA;        return new TokenStreamComponents( source, result );&#xA;&#xA;    }&#xA;}&#xA;&#xA;public class LuceneTest {&#xA;&#xA;    public static void main(String[] args) throws Exception {&#xA;&#xA;        TestAnalyzer analyzer = new TestAnalyzer();&#xA;&#xA;        try {&#xA;            TokenStream stream = analyzer.tokenStream(&quot;field&quot;, new StringReader(&quot;This is a sample sentence.&quot;));&#xA;            CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);&#xA;&#xA;            stream.reset();&#xA;&#xA;            // print all tokens until stream is exhausted&#xA;            while (stream.incrementToken()) {&#xA;                System.out.println(termAtt.toString());&#xA;            }&#xA;&#xA;            stream.end();&#xA;            stream.close();&#xA;         }&#xA;         catch (Exception ex) {&#xA;             ex.printStackTrace();&#xA;         }&#xA;&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="984" LastActivityDate="2014-10-03T18:05:13.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1204" PostTypeId="1" CreationDate="2014-10-03T20:12:24.427" Score="-1" ViewCount="1085" Body="&lt;p&gt;I would like to do some data mining and NLP experiments to do some research&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have decided to use NLTK or related tools and software&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which environment or operating system do you suggest for my purpose? I mean doing research on NLP&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Windows or Linux? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a user of Windows but I thought if Linux has better shell and related software for NLP tasks then I switch to Linux&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your experience and your preferred OS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As NLTK is in Python I thought Python is a good language for my purpose, do you suggest Python too?&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="1330" LastEditDate="2020-06-12T19:43:25.510" LastActivityDate="2020-06-12T19:43:25.510" Title="NLP lab, linux or windows and which programming languages?" Tags="|nlp|tools|linux|windows|" AnswerCount="2" CommentCount="0" ClosedDate="2014-10-29T17:39:01.777" ContentLicense="CC BY-SA 3.0" />
  <row Id="1205" PostTypeId="1" AcceptedAnswerId="1212" CreationDate="2014-10-04T07:50:11.633" Score="1" ViewCount="966" Body="&lt;p&gt;Next week I'm going to begin prototyping a recommendation engine for work. I've implemented/completed the Netflix Challenge in Java before (for college) but have no real idea what to use for a production/enterprise level recommendation engine. Taking into consideration everything from a standalone programming language to things like Apache Mahout and Neo4j, does anyone have any advice on how to proceed?&lt;/p&gt;&#xA;" OwnerUserId="3526" LastActivityDate="2014-10-09T10:50:39.197" Title="Recommended Language/Framework for Building a New Recommendation Engine" Tags="|recommender-system|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1206" PostTypeId="2" ParentId="1195" CreationDate="2014-10-04T15:52:44.370" Score="1" Body="&lt;p&gt;Suppose your test object is &lt;code&gt;(sunny, hot, normal, TRUE)&lt;/code&gt;. Look through the rules top to bottom and see if any of the conditions are matched. The first rule for example tests the &lt;code&gt;outlook&lt;/code&gt; feature. The value doesn't match, so the rule isn't matched. Move on to the next rule. And so on. In this case, rule 5 matches the test case and the classification for the p lay variable is &quot;yes&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, for any test case, look at the values its features take and find the first rule that those values satisfy. The implication of that rule will be its classification.&lt;/p&gt;&#xA;" OwnerUserId="51" LastActivityDate="2014-10-04T15:52:44.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1207" PostTypeId="2" ParentId="1197" CreationDate="2014-10-04T19:37:51.523" Score="3" Body="&lt;p&gt;There are a bunch of techniques. You have already mentioned n-gram, then there is word combination and others. But the main problem (at least from your point of view) is that as the feature becomes more complex (like n-gram) the feature count increases dramatically. This is manageable. Basically before classification you must score your features and then threshold at a certain score. this way the features (or in your case n-grams) that are scored below a certain level are omitted and the feature count becomes manageable.&#xA;as for the scoring. There are numerous ways (which to select is dependent on your application) to score the features. You can begin with &quot;BiNormal separation&quot;, &quot;chi square&quot;, &quot;Information Gain&quot; and etc.&#xA;I don't know if this answer helps you but if you are interested i can elaborate...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I forgot, in word combination you put a window of size m on the text and extract each combination of n words. of course n&#xA;" OwnerUserId="3530" LastActivityDate="2014-10-04T19:37:51.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1208" PostTypeId="1" AcceptedAnswerId="4895" CreationDate="2014-10-04T19:49:53.543" Score="3" ViewCount="74" Body="&lt;p&gt;Okay, here is the background:&#xA;I am doing text mining, and my basic flow is like this:&#xA;extract feature (n-gram), reduce feature count, score (tf-idf) and classify. for my own sake i am doing comparison between SVM and neural network classifiers. here is the weird part (or am i wrong and this is reasonable?), if i use 2gram the classifiers' result (accuracy/precision) is different and the SVM is the better one; but when i use 3-gram the results are exactly the same. what causes this? is there any explanation? is it the case of very separable classes?&lt;/p&gt;&#xA;" OwnerUserId="3530" LastActivityDate="2015-01-17T04:42:47.530" Title="What circumstances causes two different classifiers to classify data exactly like one another" Tags="|text-mining|neural-network|svm|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1209" PostTypeId="2" ParentId="1151" CreationDate="2014-10-04T19:55:31.480" Score="1" Body="&lt;p&gt;coursera should be a good start. i have seen machine learning on their course list if im not mistaken. once you get the hang of it, a machine learning classic text like &quot;machine learning and pattern recognition&quot; by bishop or other texts can familiarize you with different classes of algorithms.&#xA;if you want to go in depth, mathematics is a must, but if you just want to get familiar and use the algorithms there's absolutely no need for it&lt;/p&gt;&#xA;" OwnerUserId="3530" LastActivityDate="2014-10-04T19:55:31.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1210" PostTypeId="2" ParentId="1059" CreationDate="2014-10-04T20:09:17.547" Score="0" Body="&lt;p&gt;i have seen many papers that are basically what you just described. there is nothing wrong with what you are doing, but there are severe limitations in how this can predict the market.&#xA;let me give you an example: suppose that u have some data and you begin to predict. with each set of data, you predict the next datapoint. and then you feed this datapoint back to the system as input and do this on and on .... in most of the times the system would just continue the last trend and the timeseries won't &lt;strong&gt;break&lt;/strong&gt;. this is not prediction, this is line continuation... only when the system sees the break in real data, will the prediction break, and this is the lag that you are talking about (if i understand your question right).&#xA;The first thing that you can do to enhance this is to extract some market indicators from the price. this would really help&lt;/p&gt;&#xA;" OwnerUserId="3530" LastActivityDate="2014-10-04T20:09:17.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1212" PostTypeId="2" ParentId="1205" CreationDate="2014-10-05T05:01:03.437" Score="0" Body="&lt;p&gt;If you merely want to scale up a simple collaborative filter (low rank matrix factorization), I'd suggest looking at &lt;a href=&quot;http://docs.graphlab.org/collaborative_filtering.html&quot; rel=&quot;nofollow&quot;&gt;graphlab&lt;/a&gt;. Another graph-based (or should I say &lt;a href=&quot;http://giraph.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Giraph&lt;/a&gt;?) solution is &lt;a href=&quot;http://grafos.ml/okapi.html#collaborative&quot; rel=&quot;nofollow&quot;&gt;Okapi&lt;/a&gt;. Spark's &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;MLLib&lt;/a&gt; is another option (&lt;a href=&quot;https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html&quot; rel=&quot;nofollow&quot;&gt;details&lt;/a&gt;), and it also supports implicit feedback out of the box. Mahout's behind the curve today; I wouldn't bother with it until it is &lt;a href=&quot;https://www.mapr.com/blog/mahout-spark-what%E2%80%99s-new-recommenders&quot; rel=&quot;nofollow&quot;&gt;migrated to Spark&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do something that the libraries don't do, say with regularization, you'll have to roll your own solution in your general purpose programming language of choice. It's not hard to get a prototype running, but you might run into scale problems in production; that's why I recommended solutions that scale easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also black box recommender systems from commercial vendors but I have no experience with those.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-10-07T07:45:19.727" LastActivityDate="2014-10-07T07:45:19.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1213" PostTypeId="1" CreationDate="2014-10-05T05:12:13.633" Score="4" ViewCount="5190" Body="&lt;p&gt;I'm looking for a supervised learning algorithm that can take 2d data for input and output. As an example of something similar to my data, consider a black image with some sparse white dots. Blur that image using a full range of grayscale. Then create a machine that can take the blurred image as input and produce the original sharp image as output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could make some sample 1D data by taking a region/radius around the original sharp point, but I don't know the exact radius. It would be significant data duplication and a lot of guessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any good algorithm suggestions for this problem? Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="3539" LastEditorUserId="847" LastEditDate="2014-11-14T08:36:25.320" LastActivityDate="2017-10-26T15:12:58.433" Title="machine learning algorithms for 2d data?" Tags="|machine-learning|dimensionality-reduction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1214" PostTypeId="1" AcceptedAnswerId="2262" CreationDate="2014-10-05T06:25:27.303" Score="11" ViewCount="440" Body="&lt;p&gt;I am a hands on researcher and I like testing out viable solutions, so I tend to run a lot of experiments. For example, if I am calculating a similarity score between documents, I might want to try out many measures. In fact, for each measure I might need to make several runs to test the effect of some parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I've been tracking the runs inputs and their results by writing out the results into  files with as much info about the inputs. The problem is that retrieving a specific result becomes a challenge sometimes, even if I try to add the input info to th filename. I tried using a spreadsheet with links to results but this isn't making a huge difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What tools/process do you use for the book keeping of your experiments?&lt;/p&gt;&#xA;" OwnerUserId="3540" LastActivityDate="2014-10-12T19:52:55.430" Title="Book keeping of experiment runs and results" Tags="|tools|experiments|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="1216" PostTypeId="1" AcceptedAnswerId="1220" CreationDate="2014-10-06T12:00:55.303" Score="9" ViewCount="5987" Body="&lt;p&gt;I am a 35 year old IT professional who is purely technical. I am good at programming, learning new technologies, understanding them and implementing. I did not like mathematics at school, so I didn't score well in mathematics. I am very much interested in pursuing a career in Big Data analytics. I am more interested in Analytics rather than Big Data technologies (Hadoop etc.), though I do not dislike it.  However, when I look around in the internet, I see that, people who are good in analytics (Data Scientists) are mainly Mathematics graduates  who have done their PHds and sound like intelligent creatures, who are far far ahead of  me. I get scared sometimes to think whether my decision is correct, because learning advance statistics on your own is very tough and requires a of hard work and time investment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know whether my decision is correct, or should I leave this piece of work to only intellectuals who have spend their life in studying in prestigious colleges and earned their degrees and PHDs.&lt;/p&gt;&#xA;" OwnerUserId="3550" LastEditorUserId="97" LastEditDate="2015-01-10T11:10:53.493" LastActivityDate="2019-07-16T12:35:08.443" Title="Career switch to Big Data Analytics" Tags="|career|" AnswerCount="7" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="1220" PostTypeId="2" ParentId="1216" CreationDate="2014-10-06T19:38:00.207" Score="4" Body="&lt;p&gt;Due to high demand, it is possible to start a career in data science without a formal degree. My experience is that having a degree is often a 'requirement' in job descriptions, but if the employer is desperate enough, then that won't matter. In general, it's harder to get into large corporations with formalized job application processes than smaller companies without them. &quot;Knowing people&quot; can get you a long way, in either case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of your education, no matter how high demand is, you must have the skills to do the job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are correct in noting that advanced statistics and other mathematics are very hard to learn independently. It is a matter of how badly you want to make the career change. While some people do have 'natural talent' in mathematics, everybody does have to do the work to learn. Some may learn more quickly, but everybody has to take the time to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What it comes down to is your ability to show potential employers that you have a genuine interest in the field, and that you will be able to learn quickly on the job. The more knowledge you have, the more projects you can share in a portfolio, and the more work experience under your belt, the higher level jobs that will be available to you. You may have to start  in an entry level position first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could suggest ways to study mathematics independently, but that isn't part of your question. For now, just know that it's hard, but possible if you are determined to make a career change. Strike while the iron is hot (while demand is high).&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-10-06T19:38:00.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1221" PostTypeId="2" ParentId="1204" CreationDate="2014-10-06T19:54:57.197" Score="2" Body="&lt;p&gt;Python is easy to use and manage in Linux. The Python package manager PIP and the Python environment manager Virtualenv are both fully supported in Linux. See: &lt;a href=&quot;http://docs.python-guide.org/en/latest/starting/install/linux/&quot; rel=&quot;nofollow&quot;&gt;http://docs.python-guide.org/en/latest/starting/install/linux/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The PyCharm IDE from JetBrains is also available for free in Linux (community edition), and it runs well. This is a great tool for a Python developer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, if you need to integrate other tools, they will most likely be best supported in Linux. Need a MapReduce in your app? Hadoop will run in Linux. Need to store data in MySQL? Linux. Most data science tools are open source, and most open source technologies are supported best in Linux.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also no cost for running Linux.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need to keep Windows for your primary work computer (like your laptop), then you can install Linux on a server and use Putty to SSH into the Linux machine. You would then perform your work on the Linux machine, though you would still be running Windows locally.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-10-06T19:54:57.197" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1222" PostTypeId="2" ParentId="1216" CreationDate="2014-10-06T23:38:17.523" Score="1" Body="&lt;p&gt;You should look more into the infrastructure side of things if you don't like maths. The lower you go in the software stack, the further away you get from maths (of the data science sort). In other words, you could build the foundation that others will use to create the tools that will serve analysts. Think of companies like Cloudera, MapR, Databricks, etc. Skills that will come in handy are distributed systems and database design. You are not going to be become a data scientist without maths; that's a ridiculous notion!&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-06T23:38:17.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1223" PostTypeId="1" AcceptedAnswerId="1224" CreationDate="2014-10-07T17:27:22.063" Score="4" ViewCount="1927" Body="&lt;p&gt;I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a &lt;code&gt;scipy.sparse.csr.csr_matrix&lt;/code&gt; object, with &lt;code&gt;dtype='numpy.float64'&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using this as an input to sklearn's Logistic Regression classifier. The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; indicates that sparse CSR matrices are acceptable inputs to this classifier. However, when I train the classifier, I get extremely bad memory performance; the memory usage of my process explodes from ~150 MB to fill all the available memory and then everything grinds to a halt as memory swapping to disk takes over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know why this classifier might expand the sparse matrix to a dense matrix? I'm using the default parameters for the classifier at the moment, within an updated anacoda distribution. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scipy.__version__ = '0.14.0'&#xA;sklearn.__version__ = '0.15.2'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3568" LastActivityDate="2014-10-07T18:09:19.433" Title="Scikit Learn Logistic Regression Memory Leak" Tags="|efficiency|performance|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1224" PostTypeId="2" ParentId="1223" CreationDate="2014-10-07T18:09:19.433" Score="4" Body="&lt;p&gt;Ok, this ended up being an RTFM situation, although in this case it was RTF error message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While running this, I kept getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I assumed that, since this had to do with the target vector, and since it was a warning only, that it would just silently change my target vector to 1-D.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I explicitly converted my target vector to 1-D, my memory problems went away. Apparently having the target vector in an incorrect form caused it to convert my input vectors into dense vectors from sparse vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lesson learned: &lt;strong&gt;follow the recommendations&lt;/strong&gt; when sklearn 'suggests' you do something.&lt;/p&gt;&#xA;" OwnerUserId="3568" LastActivityDate="2014-10-07T18:09:19.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1225" PostTypeId="1" AcceptedAnswerId="1235" CreationDate="2014-10-08T10:42:41.833" Score="2" ViewCount="413" Body="&lt;p&gt;I have product purchase count data which looks likes this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user item1 item2&#xA;   a     2     4&#xA;   b     1     3&#xA;   c     5     6&#xA;   ...   ...   ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These data are imported into &lt;code&gt;python&lt;/code&gt; using &lt;code&gt;numpy.genfromtxt&lt;/code&gt;. Now I want to process it to get the correlation between &lt;code&gt;item1&lt;/code&gt; purchase amount and &lt;code&gt;item2&lt;/code&gt; purchase amount -- basically for each value &lt;code&gt;x&lt;/code&gt; of &lt;code&gt;item1&lt;/code&gt; I want to find all the users who bought &lt;code&gt;item1&lt;/code&gt; in &lt;code&gt;x&lt;/code&gt; quantity then average the &lt;code&gt;item2&lt;/code&gt; over the same users. What is the best way to do this? I can do this by using &lt;code&gt;for&lt;/code&gt; loops but I thought there might be something more efficient than that. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3580" LastActivityDate="2014-10-09T10:58:02.557" Title="data processing, correlation calculation" Tags="|python|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1226" PostTypeId="2" ParentId="1204" CreationDate="2014-10-08T13:37:27.357" Score="0" Body="&lt;p&gt;If you want to do NLP stay away from Windows! ( No Offense) Python NLTK stands out in terms of resource and community support for NLP. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover in Linux you can always leverage the benefit of  awk,sed, grep which are pretty useful tools when it comes to pre-processing. And no worries of funny characters getting introduced when you use a file.&lt;/p&gt;&#xA;" OwnerUserId="3232" LastActivityDate="2014-10-08T13:37:27.357" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1227" PostTypeId="1" CreationDate="2014-10-08T16:17:11.663" Score="8" ViewCount="171" Body="&lt;p&gt;I'm trying to figure out a good (and fast) solution to the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two models I'm working with, let's call them players and teams. A player can be on multiple teams and a team can have multiple players). I'm working on creating a UI element on a form that allows a user to select multiple teams (checkboxes). As the user is selecting (or deselecting) teams, I'd like to display the teams grouped by the players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If the selected teams have no players that intersect, each team would have its own section. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the user selects two teams and they have the same players, there would be one section containing the names of the two teams and all the players.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If TEAM_A has players [1, 2, 4, 5] and TEAM_B has players [1, 3, 5, 6]. There would be the following sections: SECTION_X = [TEAM_A, TEAM_B, 1, 5], SECTION_Y = [TEAM_A, 2, 3], SECTION _Z = [TEAM_B, 3, 5]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I hope that's clear. Essentially, I want to find the teams that players have in common and group by that. I was thinking maybe there is a way to do this by navigating a bipartite graph? Not exactly sure how though and I might be overthinking it. I was hoping to do this by creating some type of data structure on the server and using it on the client. I would love to hear your suggestions and I appreciate any help you can give!&lt;/p&gt;&#xA;" OwnerUserId="3583" LastActivityDate="2015-02-25T11:18:26.313" Title="Union grouping in bipartite graphs?" Tags="|graphs|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1229" PostTypeId="1" AcceptedAnswerId="1231" CreationDate="2014-10-09T01:41:11.797" Score="7" ViewCount="8949" Body="&lt;p&gt;Lets say I have a database of users who rate different products on a scale of 1-5. Our recommendation engine recommends products to users based on the preferences of other users who are highly similar. My first approach to finding similar users was to use Cosine Similarity, and just treat user ratings as vector components. The main problem with this approach is that it just measures vector angles and doesn't take rating scale or magnitude into consideration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is this:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there any drawbacks to&lt;/strong&gt; &lt;strong&gt;just using the percentage difference between the vector components of two vectors as a measure of similarity&lt;/strong&gt;? What disadvantages, if any, would I encounter if I used that method, instead of Cosine Similarity or Euclidean Distance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Example, why not just do this:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 5 stars&#xA;a = (1,4,4)&#xA;b = (2,3,4)&#xA;&#xA;similarity(a,b) = 1 - ( (|1-2|/5) + (|4-3|/5) + (|4-4|/5) ) / 3 = .86667&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Instead of Cosine Similarity :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = (1,4,4)&#xA;b = (2,3,4)&#xA;&#xA;CosSimilarity(a,b) = &#xA;(1*2)+(4*3)+(4*4) / sqrt( (1^2)+(4^2)+(4^2) ) * sqrt( (2^2)+(3^2)+(4^2) ) = .9697&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3587" LastActivityDate="2014-10-10T00:03:00.447" Title="Cosine Similarity for Ratings Recommendations? Why use it?" Tags="|machine-learning|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1230" PostTypeId="2" ParentId="1229" CreationDate="2014-10-09T01:57:41.010" Score="5" Body="&lt;p&gt;For ratings, I think you would need to use &lt;a href=&quot;http://mathworld.wolfram.com/SpearmanRankCorrelationCoefficient.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spearman's rank correlation&lt;/a&gt; for your similarity metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity is often used when comparing documents, and perhaps would not be a good fit for rank variables. Euclidean distance is fine for lower dimensions, but comparison of rank variables normally call for Spearman.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a &lt;a href=&quot;https://stats.stackexchange.com/questions/8071/how-to-choose-between-pearson-and-spearman-correlation&quot;&gt;question on CrossValidated regarding Spearman (vs Pearson)&lt;/a&gt;, which might shed more light for you.&lt;/p&gt;&#xA;" OwnerUserId="24" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-10-09T01:57:41.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1231" PostTypeId="2" ParentId="1229" CreationDate="2014-10-09T02:24:07.640" Score="5" Body="&lt;p&gt;Rating bias and scale can easily be accounted for by standardization. The point of using  Euclidean similarity metrics in vector space co-embeddings is that it reduces the recommendation problem to one of finding the nearest neighbors, which can be done efficiently both exactly and approximately. What you don't want to do in real-life settings is to have to compare every item/user pair and sort them according to some expensive metric. That just doesn't scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One trick is to use an approximation to cull the herd to a managable size of tentative recommendations, then to run your expensive ranking on top of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: Microsoft Research is presenting a paper that covers this very topic at RecSys right now: &lt;a href=&quot;http://www.ulrichpaquet.com/Papers/SpeedUp.pdf&quot; rel=&quot;nofollow&quot;&gt;Speeding Up the Xbox Recommender System Using a&#xA;Euclidean Transformation for Inner-Product Spaces&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-10-10T00:03:00.447" LastActivityDate="2014-10-10T00:03:00.447" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1232" PostTypeId="2" ParentId="369" CreationDate="2014-10-09T02:35:24.533" Score="3" Body="&lt;p&gt;nDCG is a ranking metric and RMSE is not. In the context of recommender systems, you would use a ranking metric when your ratings are implicit (e.g., item skipped vs. item consumed) rather than explicit (the user provides an actual number, a la Netflix).&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-09T02:35:24.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1234" PostTypeId="2" ParentId="1205" CreationDate="2014-10-09T10:50:39.197" Score="1" Body="&lt;p&gt;I am developing a recommendation engine for stack overflow (personal project). Check it on &lt;a href=&quot;http://recommender.im&quot; rel=&quot;nofollow&quot;&gt;http://recommender.im&lt;/a&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is still a working in progress, but I have a quite functional website working. I am putting there most of the code I used through python notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I used:&lt;br&gt;&#xA;- Frontend: angularJS&lt;br&gt;&#xA;- Website backend: flask + scikit-learn&lt;br&gt;&#xA;- machine learning and data preparation: python, pandas, scikit-learn&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really like python for data science as the community and libraries are really good.&lt;/p&gt;&#xA;" OwnerUserId="3596" LastActivityDate="2014-10-09T10:50:39.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1235" PostTypeId="2" ParentId="1225" CreationDate="2014-10-09T10:58:02.557" Score="2" Body="&lt;p&gt;Pandas is the best thing since sliced bread (for data science, at least). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pd&#xA;In [22]: df = pd.read_csv('yourexample.csv')&#xA;&#xA;In [23]: df&#xA;Out[23]:&#xA;   user   item1   item2&#xA;0     a        2      4&#xA;1     b        1      3&#xA;2     c        5      6&#xA;&#xA;In [24]: df.columns&#xA;Out[24]: Index([u'user ', u'item1 ', u'item2'], dtype='object')&#xA;&#xA;In [25]: df.corr()&#xA;Out[25]:&#xA;          item1      item2&#xA;item1   1.000000  0.995871&#xA;item2   0.995871  1.000000&#xA;&#xA;In [26]: df.cov()&#xA;Out[26]:&#xA;          item1      item2&#xA;item1   4.333333  3.166667&#xA;item2   3.166667  2.333333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Bingo!&lt;/p&gt;&#xA;" OwnerUserId="3596" LastActivityDate="2014-10-09T10:58:02.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1236" PostTypeId="1" AcceptedAnswerId="1249" CreationDate="2014-10-09T12:26:01.643" Score="1" ViewCount="49" Body="&lt;p&gt;how to get the Polysemes of a word in wordnet or any other api. I am looking for any api. with java any idea is appreciated?&lt;/p&gt;&#xA;" OwnerUserId="3598" LastActivityDate="2014-10-10T16:54:45.047" Title="how to get the Polysemes of a word in wordnet or any other api?" Tags="|nlp|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="1240" PostTypeId="1" AcceptedAnswerId="2360" CreationDate="2014-10-10T00:59:46.703" Score="5" ViewCount="13262" Body="&lt;p&gt;I know there is the normal &lt;em&gt;subtract the mean and divide by the standard deviation&lt;/em&gt; for standardizing your data, but I'm interested to know if there are more appropriate methods for this kind of discrete data. Consider the following case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 5 items that have been ranked by customers. First 2 items were ranked on a 1-10 scale. Others are 1-100 and 1-5. To transform everything to a 1 to 10 scale, is there another method better suited for this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data has a central tendency, then the standard would work fine, but what about when you have more of a halo effect, or some more exponential distribution?&lt;/p&gt;&#xA;" OwnerUserId="3430" LastEditorUserId="84" LastEditDate="2014-12-13T21:47:23.723" LastActivityDate="2015-06-12T10:57:34.290" Title="Methods for standardizing / normalizing different rank scales" Tags="|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1241" PostTypeId="1" CreationDate="2014-10-10T01:12:19.163" Score="4" ViewCount="226" Body="&lt;p&gt;I do movement building work for Effective Altruism (&lt;a href=&quot;http://en.m.wikipedia.org/wiki/Effective_altruism&quot; rel=&quot;nofollow&quot;&gt;http://en.m.wikipedia.org/wiki/Effective_altruism&lt;/a&gt;), and would like to level up our growth strategy. It occurred to me that a social network visualization tool which allowed us to strategically find and recruit new influencers/donors would be mega useful. I'd love to find something (preferably free), similar to InMaps, which would allow us to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Combine all of our social media connections into a single map&lt;/li&gt;&#xA;&lt;li&gt;Easily see who the superconnectors are&lt;/li&gt;&#xA;&lt;li&gt;Weight each person by their degree of social influence (perhaps some function of things like Klout score * amount of social media connections * number of Google mentions, etc) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does such a thing exist? If not, is anyone interested in pro bono work for an amazing cause? =)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: I am a data science noob, so preferably the solution would be one with a nice GUI and minimal involvement of R or Python.&lt;/p&gt;&#xA;" OwnerUserId="3605" LastEditorUserId="21" LastEditDate="2014-10-11T09:45:11.417" LastActivityDate="2016-10-16T06:26:57.983" Title="Collaborative Social Network Visualization" Tags="|social-network-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1242" PostTypeId="2" ParentId="1241" CreationDate="2014-10-10T03:33:45.353" Score="3" Body="&lt;p&gt;I think &lt;a href=&quot;https://gephi.github.io/&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt;, an open-source visualization tool, would help you a lot. Actually, as I know, the InMaps and its community detection algorithm are same as the Gephi's.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2014-10-10T03:33:45.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1243" PostTypeId="1" CreationDate="2014-10-10T03:45:09.343" Score="6" ViewCount="207" Body="&lt;p&gt;Suppose, for example, that the first search result on a page of Google search results is swapped with the second result. How much would this change the click-through probabilities of the two results? How much would its click-through probability drop if the fifth search result was swapped with the sixth? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we say something, with some level of assurance, about how expected click-through probabilities change if we do these types of pairwise swaps within pages of search results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we seek is a measure of the contribution to click-through rates made specifically by position bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likely, how position ranking would affect the sales in Amazon or other online shopping website? If we cast the sales into two parts, the product quality and its ranking effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;sales = alpha*quality + beta*position + epsilon&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we quantify the beta?&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2014-12-17T09:32:53.703" Title="Can we quantify how position within search results is related to click-through probability?" Tags="|recommender-system|search|information-retrieval|regression|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1244" PostTypeId="1" AcceptedAnswerId="1255" CreationDate="2014-10-10T03:48:54.660" Score="12" ViewCount="5215" Body="&lt;p&gt;When ML algorithms, e.g. Vowpal Wabbit or some of the factorization machines winning click through rate competitions (&lt;a href=&quot;https://www.kaggle.com/c/criteo-display-ad-challenge/forums/t/10555/3-idiots-solution/55862#post55862&quot;&gt;Kaggle&lt;/a&gt;), mention that features are 'hashed', what does that actually mean for the model? Lets say there is a variable that represents the ID of an internet add, which takes on values such as '236BG231'. Then I understand that this feature is hashed to a random integer. But, my question is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is the integer now used in the model, as an integer (numeric) OR&lt;/li&gt;&#xA;&lt;li&gt;is the hashed value actually still treated like a categorical variable and one-hot-encoded? Thus the hashing trick is just to save space somehow with large data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2014-10-11T19:48:20.583" Title="Hashing Trick - what actually happens" Tags="|machine-learning|predictive-modeling|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1246" PostTypeId="1" AcceptedAnswerId="2515" CreationDate="2014-10-10T13:34:11.543" Score="10" ViewCount="8902" Body="&lt;p&gt;let's assume that I want to train a stochastic gradient descent regression algorithm using a dataset that has N samples. Since the size of the dataset is fixed, I will reuse the data T times. At each iteration or &quot;epoch&quot;, I use each training sample exactly once after randomly reordering the whole training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My implementation is based on Python and Numpy. Therefore, using vector operations can remarkably decrease computation time. Coming up with a vectorized implementation of batch gradient descent is quite straightforward. However, in the case of stochastic gradient descent I can not figure out how to avoid the outer loop that iterates through all the samples at each epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody know any vectorized implementation of stochastic gradient descent? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I've been asked why would I like to use online gradient descent if the size of my dataset is fixed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From [1], one can see that online gradient descent converges slower than batch gradient descent to the minimum of the empirical cost. However, it converges faster to the minimum of the expected cost, which measures generalization performance. I'd like to test the impact of these theoretical results in my particular problem, by means of cross validation. Without a vectorized implementation, my online gradient descent code is much slower than the batch gradient descent one. That remarkably increases the time it takes to the cross validation process to be completed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I include here the pseudocode of my on-line gradient descent implementation, as requested by ffriend. I am solving a regression problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Method: on-line gradient descent (regression)&#xA;Input: X (nxp matrix; each line contains a training sample, represented as a length-p vector), Y (length-n vector; output of the training samples)&#xA;Output: A (length-p+1 vector of coefficients)&#xA;&#xA;Initialize coefficients (assign value 0 to all coefficients)&#xA;Calculate outputs F&#xA;prev_error = inf&#xA;error = sum((F-Y)^2)/n&#xA;it = 0&#xA;while abs(error - prev_error)&amp;gt;ERROR_THRESHOLD and it&amp;lt;=MAX_ITERATIONS:&#xA;    Randomly shuffle training samples&#xA;    for each training sample i:&#xA;        Compute error for training sample i&#xA;        Update coefficients based on the error above&#xA;    prev_error = error&#xA;    Calculate outputs F&#xA;    error = sum((F-Y)^2)/n&#xA;    it = it + 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[1] &quot;Large Scale Online Learning&quot;, L. Bottou, Y. Le Cunn, NIPS 2003.&lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="2576" LastEditDate="2014-11-21T10:02:39.520" LastActivityDate="2014-11-21T11:50:47.717" Title="Stochastic gradient descent based on vector operations?" Tags="|python|gradient-descent|regression|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="1247" PostTypeId="2" ParentId="155" CreationDate="2014-10-10T14:35:11.323" Score="15" Body="&lt;p&gt;I am new to this forum. Chiming in late on this question. I have been maintaining (I am a co-founder of) a catalog of publicly available data portals. There is over 1000 now listed and cover portals at international, federal, state, municipal and academic levels across the globe. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.opengeocode.org/opendata/&quot;&gt;http://www.opengeocode.org/opendata/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3609" LastActivityDate="2014-10-10T14:35:11.323" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="1248" PostTypeId="2" ParentId="536" CreationDate="2014-10-10T14:53:22.500" Score="1" Body="&lt;p&gt;I am new to this forum. Data cleansing of address data is an area I work in. I agree with the other posters that you should not modify the original data, but add fields for corrected values. I developed a technique in our systems (opengeocode.org) we call 'reduced to common form'. In this method, addresses and geographic names are analyzed for reduction into an unambiguous short form, which is then used for record matching (vs. the original values). For example, the method I use for matching US postal addresses is based on the US Post Office's published method for matching addresses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For geographic names, the method will reduce to short gazetteer form in Romanized script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link below is an article I wrote a couple of years ago that explains how the street address reduction works:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.nwstartups.com/api/doc/middleware.php#streetR&quot; rel=&quot;nofollow&quot;&gt;http://www.nwstartups.com/api/doc/middleware.php#streetR&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3609" LastActivityDate="2014-10-10T14:53:22.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1249" PostTypeId="2" ParentId="1236" CreationDate="2014-10-10T16:54:45.047" Score="2" Body="&lt;p&gt;There are several third-party Java APIs for WordNet listed here: &lt;a href=&quot;http://wordnet.princeton.edu/wordnet/related-projects/#Java&quot; rel=&quot;nofollow&quot;&gt;http://wordnet.princeton.edu/wordnet/related-projects/#Java&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past, I've used JWNL the most: &lt;a href=&quot;http://sourceforge.net/projects/jwordnet/&quot; rel=&quot;nofollow&quot;&gt;http://sourceforge.net/projects/jwordnet/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation for JWNL isn't great, but it should provide the functionality you need.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-10-10T16:54:45.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="1250" PostTypeId="2" ParentId="1214" CreationDate="2014-10-10T18:10:14.507" Score="5" Body="&lt;p&gt;I recently ran into a similar problem: How to manage extracting a variety of features from a large dataset, without knowing up front what all of them would be. (Even calculating mean values repeatedly would be computationally expensive.) Further, how would I manage predictions based on different feature sets? Meaning, if I added a new feature, how would I know which models to train on new features? It could quickly snowball into a huge mess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current solution is to track it all in a local NoSQL database (MongoDB). For example, I might have a collection &lt;code&gt;features&lt;/code&gt;, each entry of which has a name, a description of how the feature was was calculated, the python file that ran the extraction, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, a collection &lt;code&gt;models&lt;/code&gt; includes models run on the data. Each entry might have a name, a list of features that were used to train the model, its eventual parameters, predicted values on a held-out test set, metrics for how the model performed, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my vantage point, this has a number of benefits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;By saving predictions, I can use them later in ensemble predictions.&lt;/li&gt;&#xA;&lt;li&gt;Because I keep track of which features were used, I know which ones need retraining as I extract more features.&lt;/li&gt;&#xA;&lt;li&gt;By saving model descriptions, I ensure that I always know what I've tried. I never have to wonder, &quot;Have I tried LASSO with regularization parameters set by grid-search CV?&quot; I can always look it up, and see how successful it was.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From your question, it sounds like you could adapt this approach to your problem's workflow. Install Mongo or another database of choice, and then save each experimental run, its inputs, its results, and anything else you might wish to track over the course of the project. This should be much easier to query than a spreadsheet, at the least.&lt;/p&gt;&#xA;" OwnerUserId="1154" LastActivityDate="2014-10-10T18:10:14.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="1253" PostTypeId="1" CreationDate="2014-10-11T10:24:01.393" Score="28" ViewCount="2240" Body="&lt;p&gt;I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.&lt;/p&gt;&#xA;" OwnerUserId="3615" LastEditorUserId="13727" LastEditDate="2015-11-11T15:45:57.500" LastActivityDate="2020-08-05T08:48:18.683" Title="Why are NLP and Machine Learning communities interested in deep learning?" Tags="|machine-learning|data-mining|neural-network|nlp|deep-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="1255" PostTypeId="2" ParentId="1244" CreationDate="2014-10-11T19:48:20.583" Score="8" Body="&lt;p&gt;The a second bullet is the value in feature hashing.  Hashing and one hot encoding to sparse data saves space.  Depending on the hash algo you can have varying degrees of collisions which acts as a kind of dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, in the specific case of Kaggle feature hashing and one hot encoding help with feature expansion/engineering by taking all possible tuples (usually just second order but sometimes third) of features that are then hashed with collisions that explicitly create interactions that are often predictive whereas the individual features are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases this technique combined with feature selection and elastic net regularization in LR acts very similar to a one hidden layer NN so it performs quite well in competitions.&lt;/p&gt;&#xA;" OwnerUserId="92" LastActivityDate="2014-10-11T19:48:20.583" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2255" PostTypeId="1" CreationDate="2014-10-11T23:26:53.197" Score="4" ViewCount="3899" Body="&lt;p&gt;I need some help with a single layered perceptron with multiple classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need to do is classify a dataset with three different classes, by now I just learnt how to do it with two classes, so I have no really a good clue how to do it with three.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset have three different classes: Iris-setosa, Iris-versicolor and Iris-versicolor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The url with the dataset and the information is in : &lt;a href=&quot;http://ftp.ics.uci.edu/pub/machine-learning-databases/iris/iris.data&quot; rel=&quot;nofollow&quot;&gt;http://ftp.ics.uci.edu/pub/machine-learning-databases/iris/iris.data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really appreciate any help anyone can give to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot!&lt;/p&gt;&#xA;" OwnerUserId="4618" LastActivityDate="2014-10-13T13:35:09.443" Title="Single Layer Perceptron with three classes" Tags="|dataset|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2256" PostTypeId="2" ParentId="1241" CreationDate="2014-10-12T02:53:51.650" Score="2" Body="&lt;p&gt;I don't think there is a way to build your graph from raw data without using at least basic programming skills. I'm not aware of a drag-and-drop interface for importing and displaying data. Graphs are just a bit too complex. Imagine trying to find the profit of selling a product if all you had was CSVs of receipts dropped into Excel. You'd need labels of the columns, some basic calculations, and so on before you had anything intelligible. Graphs are similar in this regard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thankfully, there are open source solutions, with some elbow grease and a few days of work, you can probably get a nice visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cypher queries are relatively simple to write. Using Neo4j and Cypher, you can create a basic visualization of your graph, which is displayed using D3.js&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GraphAlchemist recently open-sourced their project Alchemy.js which specializes in graph visualization. &lt;a href=&quot;https://github.com/GraphAlchemist/Alchemy&quot; rel=&quot;nofollow&quot;&gt;https://github.com/GraphAlchemist/Alchemy&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-10-12T02:53:51.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2257" PostTypeId="1" CreationDate="2014-10-12T05:27:17.687" Score="6" ViewCount="169" Body="&lt;p&gt;I have some very complicated data about some movie sales online, first for each data entry, I have a key which is a combination of five keys, which are territory, day, etc, and then, for each key I have the sales for a period of time, and other information, like the movie's box office and genre.&lt;/p&gt;&#xA;&lt;p&gt;For each day, there is a delay for the data loading to the database, around ten hours, I try to fill the gap, do some data extrapolations.&lt;/p&gt;&#xA;&lt;p&gt;For each movie we sell, there is some decay of selling since the new release of the movie, i.e. usually for each movie, it follows some sales decay pattern.&lt;/p&gt;&#xA;&lt;p&gt;For a recent day, I pulled some data, and I found that some decay pattern:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/H0lSJ.png&quot; alt=&quot;decay curve 1&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/L5DAf.png&quot; alt=&quot;decay curve 2&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/RJPjK.png&quot; alt=&quot;decay curve 3&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;And for that day, the sales for each key can range from around &lt;span class=&quot;math-container&quot;&gt;$150000 to $&lt;/span&gt;0. The pic is as follow:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/o7tsq.png&quot; alt=&quot;one day sales curve&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;In the picture, the 15000 means there are around 15000 keys for each day.&lt;/p&gt;&#xA;&lt;p&gt;Found &lt;a href=&quot;http://homepage.stat.uiowa.edu/%7Ekcowles/s166_2009/Project_Lee&amp;amp;Pyo.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am trying to predict for each key, the sales amount, like for a movie, territory, day etc combination, the sales amount, how much dollars, means for that movie, that territory, that day, how much money we get from selling online. I tried ARIMA time series model, but there is some concerns for that model, seen from the pics, there is some seasonal thing, and decay thing for the movie, so the sales prediction can not be always flat, there may be a pump after a going down, it may happens on a weekend, since there is seasonal thing, and the decay trend, etc, how to capture these things. Thank you for your reply!&lt;/p&gt;&#xA;&lt;p&gt;I am not sure whether can be applied, and how to be applied here.&lt;/p&gt;&#xA;&lt;p&gt;Thanks a lot in advance.&lt;/p&gt;&#xA;" OwnerUserId="4619" LastEditorUserId="98307" LastEditDate="2020-08-18T04:19:29.863" LastActivityDate="2020-08-18T04:19:29.863" Title="How to do this complicated data extrapolation, prediction modeling?" Tags="|data-mining|time-series|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="2258" PostTypeId="1" AcceptedAnswerId="2261" CreationDate="2014-10-12T11:23:26.493" Score="7" ViewCount="737" Body="&lt;p&gt;First of all I know the question may be not suitable for the website but I'd really appreciate it if you just gave me some pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a 16 years old programmer, I've had experience with many different programming languages, a while ago I started a course at Coursera, titled introduction to machine learning and since that moment i got very motivated to learn about AI, I started reading about neural networks and I made a working perceptron using Java and it was really fun but when i started to do something a little more challenging (building a digit recognition software), I found out that I have to learn a lot of math, I love math but the schools here don't teach us much, now I happen to know someone who is a math teacher do you think learning math (specifically calculus) is necessary for me to learn AI or should I wait until I learn those stuff at school?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also what other things would be helpful in the path of me learning AI and machine learning? do other techniques (like SVM) also require strong math?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if my question is long, I'd really appreciate if you could share with me any experience you have had with learning AI.&lt;/p&gt;&#xA;" OwnerUserId="4620" LastActivityDate="2019-11-26T08:41:17.383" Title="Where to start on neural networks" Tags="|machine-learning|neural-network|svm|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2259" PostTypeId="2" ParentId="2255" CreationDate="2014-10-12T11:49:04.480" Score="2" Body="&lt;p&gt;Well , When it comes to AI I am an absolute beginner but here is my answer to your question based on my understandings :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a perceptron has only one activation function, therefore it can return only the values of true and false (in most cases true=0 and false=1), so because of that, I don't think that you will be able to accomplish your goal using only one perceptron but you can absolutely do it using multiple perceptrons which essentially is a neural networks, of course training the network would be a lot harder than calculating the changes of weights as you do in perceptrons, You are gonna have to take advantage of a training algorithm such as backpropagation and a sigmoid activation function.&#xA;I hope my answer was helpful.&lt;/p&gt;&#xA;" OwnerUserId="4620" LastActivityDate="2014-10-12T11:49:04.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2260" PostTypeId="2" ParentId="2255" CreationDate="2014-10-12T16:35:21.260" Score="5" Body="&lt;p&gt;The standard way to do this is called 'one versus all'... you train three perceptrons. First 1 target = is class a?, 2nd perceptron target = is class b? 3rd = is class c. You just train each perceptron separately, and then take max of the three perceptrons to decide class&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-10-12T16:35:21.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2261" PostTypeId="2" ParentId="2258" CreationDate="2014-10-12T19:36:33.027" Score="5" Body="&lt;p&gt;No, you should go ahead and learn the maths on your own. You will &quot;only&quot; need to learn calculus, statistics, and linear algebra (like the rest of machine learning). The theory of neural networks is pretty primitive at this point -- it more of an art than a science -- so I think you can understand it if you try. Ipso facto, there are a lot of tricks that you need practical experience to learn. There are lot of complicated extensions, but you can worry about them once you get that far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you can understand the Coursera classes on ML and neural networks (Hinton's), I suggest getting some practice. You might like &lt;a href=&quot;http://karpathy.github.io/neuralnets/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; introduction.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-10-13T05:27:54.170" LastActivityDate="2014-10-13T05:27:54.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2262" PostTypeId="2" ParentId="1214" CreationDate="2014-10-12T19:52:55.430" Score="5" Body="&lt;p&gt;you might want to look at &lt;a href=&quot;http://deeplearning.net/software/jobman/intro.html&quot; rel=&quot;noreferrer&quot;&gt;http://deeplearning.net/software/jobman/intro.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it was designed for deep learning (I guess), but it is application agnostic. It is effectively an API version of SeanEasters approach&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-10-12T19:52:55.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2263" PostTypeId="1" CreationDate="2014-10-13T11:23:23.483" Score="1" ViewCount="149" Body="&lt;p&gt;I was wondering if someone could point me to suitable database formats for building up a user database:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;basically I am collecting logs of impressions data, and I want to compile a user database &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which sites user visits, country/gender/..? and other categorisations with the aim of &#xA;a) doing searches: give me all users visiting games sites from france...&#xA;b) machine learning: eg clustering users by the sites they visit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so I am interested in storing info about 100's of millions of users&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with indexes? on user, sites, geo-location&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the idea would be that this data would be continually updated ( eg nightly update to user database of new sites visited etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what are suitable database systems. Can someone suggest suitable reading material? &#xA;I was imagining Hbase might be suitable...&lt;/p&gt;&#xA;" OwnerUserId="1256" LastEditorUserId="1256" LastEditDate="2014-11-14T16:00:29.030" LastActivityDate="2014-11-14T16:00:29.030" Title="&quot;Hadoop&quot; formats for user database: online advertising" Tags="|hbase|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2264" PostTypeId="2" ParentId="2255" CreationDate="2014-10-13T13:35:09.443" Score="2" Body="&lt;p&gt;Perceptrons, strictly speaking, are binary classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a multi-class classifier, you should switch to a standard feed-forward neural net with a &lt;a href=&quot;http://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow&quot;&gt;softmax&lt;/a&gt; output layer. Without any hidden layers this is equivalent to multinomial logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also do the one-vs-all trick as @seanv507 suggests. This often works well in practice but there's no strong basis for it in theory, and the true multi-class version is easier conceptually and practically in this case.&lt;/p&gt;&#xA;" OwnerUserId="1399" LastActivityDate="2014-10-13T13:35:09.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2265" PostTypeId="2" ParentId="1246" CreationDate="2014-10-13T13:50:07.080" Score="1" Body="&lt;p&gt;Check out the partial_fit method of &lt;a href=&quot;http://scikit-learn.org/0.15/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier&quot; rel=&quot;nofollow&quot;&gt;scikit's SGD classifier&lt;/a&gt;. You have control over what you call with it: you can do it &quot;true&quot; online learning by passing an instance at a time, or you can batch up instances into mini-batches if all your data are available in an array. If they are, you can slice the array to provide the minibatches.&lt;/p&gt;&#xA;" OwnerUserId="1399" LastActivityDate="2014-10-13T13:50:07.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2266" PostTypeId="2" ParentId="1243" CreationDate="2014-10-13T13:53:36.027" Score="1" Body="&lt;p&gt;It's just logistic regression. Get a bunch of data about presentations of search results, along with whether an item was clicked on. An instance is a search result item, with possible features being rank, &quot;quality&quot; (not sure what you mean by this) etc. The what you're asking about is a question of inference on the parameter related to rank.&lt;/p&gt;&#xA;" OwnerUserId="1399" LastActivityDate="2014-10-13T13:53:36.027" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2267" PostTypeId="2" ParentId="1243" CreationDate="2014-10-13T14:46:54.403" Score="2" Body="&lt;p&gt;you might want to look at this paper&#xA;&lt;a href=&quot;http://research.microsoft.com/pubs/68148/predictingclicks.pdf&quot; rel=&quot;nofollow&quot;&gt;Predicting Clicks: Estimating the Click-Through Rate for New Ads&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Whenever an ad is displayed on the search results page, it has &#xA;  some chance of being viewed by the user. The farther down the &#xA;  page an ad is displayed, the less likely it is to be viewed. As a &#xA;  simplification, we consider the probability that an ad is clicked on &#xA;  to be dependent on two factors: a) the probability that it is viewed, &#xA;  and b) the probability that it is clicked on, given that it is viewed:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-10-13T14:46:54.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2268" PostTypeId="1" CreationDate="2014-10-13T19:02:24.670" Score="1" ViewCount="3451" Body="&lt;p&gt;I am new to Natural Language Processing, I think NLP is a challenging field, the syntax and semantic ambiguities could cause a lot of problems. For example I think for these problems machine translation is a hard task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore there are probably many approaches and methods that have been applied to this field. But what are the latest and most promising approaches and methods in the field of NLP?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these techniques highly dependent on the target language?&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="24" LastEditDate="2014-10-14T00:27:48.050" LastActivityDate="2014-11-12T18:22:07.253" Title="What is the state of the art in the field of NLP?" Tags="|nlp|" AnswerCount="2" CommentCount="2" ClosedDate="2014-11-13T04:38:15.907" ContentLicense="CC BY-SA 3.0" />
  <row Id="2269" PostTypeId="1" CreationDate="2014-10-13T21:13:48.447" Score="29" ViewCount="28148" Body="&lt;p&gt;I am looking for an online console for the language R. Like I write the code and the server should execute and provide me with the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to the website Datacamp.&lt;/p&gt;&#xA;" OwnerUserId="4637" LastActivityDate="2021-02-09T01:33:22.723" Title="Any Online R console?" Tags="|r|statistics|" AnswerCount="10" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2270" PostTypeId="2" ParentId="2269" CreationDate="2014-10-13T22:07:51.180" Score="5" Body="&lt;p&gt;Yes. I believe &lt;a href=&quot;http://www.compileonline.com/execute_r_online.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; is what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastEditorUserId="29169" LastEditDate="2021-02-09T01:32:29.007" LastActivityDate="2021-02-09T01:32:29.007" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2271" PostTypeId="2" ParentId="2268" CreationDate="2014-10-14T01:19:37.067" Score="2" Body="&lt;p&gt;You are probably aware that deep learning is all the rage these days, and it has touched NLP too. There is a tutorial on it from a recent conference: &lt;a href=&quot;http://nlp.stanford.edu/courses/NAACL2013/&quot; rel=&quot;nofollow&quot;&gt;Deep Learning for Natural Language Processing (without Magic)&lt;/a&gt; by Richard Socher and Christopher Manning, who are from Stanford.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-14T01:19:37.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2272" PostTypeId="2" ParentId="2269" CreationDate="2014-10-14T07:02:47.500" Score="28" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://roncloud.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;R On Cloud&lt;/a&gt; provides a browser-embedded R-console.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ELpbR.png&quot; alt=&quot;screenshot-roncloud&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://jupyter.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jupyter.org&lt;/a&gt; evolved from the &lt;a href=&quot;http://ipython.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;IPython Project&lt;/a&gt; (the language-agnostic parts of IPython); supports Python 3, Julia, R, Haskell, Ruby, etc.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/mhHkx.png&quot; alt=&quot;jupyter screenshot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2961" LastEditorUserId="45572" LastEditDate="2018-02-01T18:11:43.553" LastActivityDate="2018-02-01T18:11:43.553" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2273" PostTypeId="1" AcceptedAnswerId="2274" CreationDate="2014-10-14T08:50:53.907" Score="5" ViewCount="1101" Body="&lt;p&gt;I have a variable whose value I would like to predict, and I would like to use only one variable as predictor. For instance, predict traffic density based on weather.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially, I thought about using &lt;a href=&quot;http://en.wikipedia.org/wiki/Self-organizing_map&quot;&gt;Self-Organizing Maps&lt;/a&gt; (SOM), which performs unsupervised clustering + regression. However, since it has an important component of dimensionality reduction, I see it as more appropriated for a large number of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to use it for a single variable as predictor? Maybe there are more adequate techniques for this &lt;em&gt;simple&lt;/em&gt; case: I used &quot;Data Mining&quot; instead of &quot;machine learning&quot; in the title of my question, because I think maybe a linear regression could do the job...&lt;/p&gt;&#xA;" OwnerUserId="3159" LastEditorUserId="84" LastEditDate="2014-10-18T13:25:13.107" LastActivityDate="2014-10-18T13:25:13.107" Title="What is the best Data Mining algorithm for prediction based on a single variable?" Tags="|clustering|regression|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2274" PostTypeId="2" ParentId="2273" CreationDate="2014-10-14T13:10:39.773" Score="5" Body="&lt;p&gt;Common rule in machine learning is to &lt;strong&gt;try simple things first&lt;/strong&gt;. For predicting continuous variables there's nothing more basic than &lt;strong&gt;simple linear regression&lt;/strong&gt;. &quot;Simple&quot; in the name means that there's only one predictor variable used (+ intercept, of course): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = b0 + x*b1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;b0&lt;/code&gt; is an intercept and &lt;code&gt;b1&lt;/code&gt; is a slope. For example, you may want to predict lemonade consumption in a park based on temperature:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cons = b0 + temp * b1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Temperature is in well-defined &lt;strong&gt;continuous&lt;/strong&gt; variable. But if we talk about something more abstract like &quot;weather&quot;, then it's harder to understand how we measure and encode it. It's ok if we say that the weather takes values &lt;code&gt;{terrible, bad, normal, good, excellent}&lt;/code&gt; and assign values numbers from -2 to +2 (implying that &quot;excellent&quot; weather is twice as good as &quot;good&quot;). But what if the weather is given by words &lt;code&gt;{shiny, rainy, cool, ...}&lt;/code&gt;? We can't give an order to these variables. We call such variables &lt;strong&gt;categorical&lt;/strong&gt;. Since there's no natural order between different categories, we can't encode them as a single numerical variable (and linear regression expects numbers only), but we can use so-called &lt;strong&gt;dummy encoding&lt;/strong&gt;:  instead of a single variable &lt;code&gt;weather&lt;/code&gt; we use 3 variables - &lt;code&gt;[weather_shiny, weather_rainy, weather_cool]&lt;/code&gt;, only one of which can take value 1, and others should take value 0. In fact, we will have to drop one variable because of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multicollinearity&quot;&gt;collinearity&lt;/a&gt;. So model for predicting traffic from weather may look like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;traffic = b0 + weather_shiny * b1 + weather_rainy * b2  # weather_cool dropped&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where either &lt;code&gt;b1&lt;/code&gt; or &lt;code&gt;b2&lt;/code&gt; is 1, or both are 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you can also encounter non-linear dependency between predictor and predicted variables (you can easily check it by plotting &lt;code&gt;(x,y)&lt;/code&gt; pairs). Simplest way to deal with it without refusing linear model is to use polynomial features - simply add polynomials of your feature as new features. E.g. for temperature example (for dummy variables it doesn't make sense, cause &lt;code&gt;1^n&lt;/code&gt; and &lt;code&gt;0^n&lt;/code&gt; are still 1 and 0 for any &lt;code&gt;n&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;traffic = b0 + temp * b1 + temp^2 * b2 [+ temp^3 * b3 + ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-10-14T13:10:39.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2275" PostTypeId="1" CreationDate="2014-10-14T15:26:01.460" Score="3" ViewCount="44" Body="&lt;p&gt;&lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=923635&quot; rel=&quot;nofollow noreferrer&quot;&gt;Automated Time Series Forecasting for Biosurveillance&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in the above paper, page 4, two models, non-adaptive regression model, adaptive regression model, the non-adaptive regression model's parameter estimation method is &quot;least squares&quot;, what is the parameter estimation for the adaptive regression model? is there any package in R to do parameter estimation for this kind of adaptive regression model? If I add more predictors in the adaptive regression model, can R still solve it? and how?&lt;/p&gt;&#xA;" OwnerUserId="4619" LastEditorUserId="8878" LastEditDate="2017-05-25T09:19:38.273" LastActivityDate="2017-05-25T09:19:38.273" Title="What are the parameter estimation methods for the two methods in this paper?" Tags="|r|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2276" PostTypeId="2" ParentId="2263" CreationDate="2014-10-14T17:47:02.663" Score="0" Body="&lt;p&gt;The kind of data you store and analyze is very much dependent upon the kind of data you can gather. So, without knowing what your 'impression data' looks like, it is very hard to suggest how to normalize and store it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the way you store data is also dependent upon how you wish to analyze it. For example, if you want to perform basic analytics like page view counts, how many pages a user visits per session, etc (SQL). . . data needs to be stored differently than if you want to build recommendations based on traffic patterns (graph database).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please edit your question to include more detail. Apologies that I cannot simply leave a comment.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-10-14T17:47:02.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2277" PostTypeId="2" ParentId="2273" CreationDate="2014-10-14T18:24:36.880" Score="1" Body="&lt;p&gt;I am more of an expert on data ETL and combining/aggregating than on the forumulas themselves. I work frequently with weather data. I like to give some suggestions on using weather data in analysis.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Two types of data are reported in US/Canada:&lt;br/&gt;&#xA;A. Measurements&lt;br/&gt;&#xA;B. Weather Type&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As far as weather type (sunny, rainy, severe thunderstorm) they are either going to already be reflected in measurements (e.g., sunny, rainy) and are redundant or they are inclement weather conditions and are not necessarily reflected in the measurements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For inclement weather types, I would have separate formulae.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For measurements, there are 7 standard daily measurements for Weather Station reporting in North America.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Temp Min/Max&lt;br/&gt;&#xA;Precipitation&lt;br/&gt;&#xA;Average Wind Speed&lt;br/&gt;&#xA;Average Cloudiness (percentage)&lt;br/&gt;&#xA;Total sunlight (minutes)&lt;br/&gt;&#xA;Snowfall&lt;br/&gt;&#xA;Snow Depth&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not all stations report all 7 daily measurements. Some report only Temp and Precipitation. So you may want to have one formula for Temp/Precipitation and an expanded formulae when all seven measurements are available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two links below are NOAA/NWS weather terms used in their datasets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This document is the vocabulary for the annual summaries:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/ANNUAL_documentation.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/ANNUAL_documentation.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This document is the vocabulary for the daily summaries &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3609" LastActivityDate="2014-10-14T18:24:36.880" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2278" PostTypeId="1" CreationDate="2014-10-15T02:22:09.983" Score="2" ViewCount="79" Body="&lt;p&gt;How would I do parameter estimation and prediction for the adaptive regression model using R, as in the 4th page of the paper linked below?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=923635&quot; rel=&quot;nofollow&quot;&gt;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=923635&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone clarify this for me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know adaptive regression models very well, share some useful link, or describe the model/parameter estimation/prediction, that would be very helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much!&lt;/p&gt;&#xA;" OwnerUserId="4619" LastEditorUserId="24" LastEditDate="2014-10-21T14:07:51.717" LastActivityDate="2014-11-20T15:28:14.573" Title="Adaptive regression parameter estimation in R" Tags="|r|statistics|regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2279" PostTypeId="2" ParentId="2278" CreationDate="2014-10-15T03:08:59.263" Score="1" Body="&lt;p&gt;Here is some notes for an R package 'earth' for adaptive regression. This may be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/earth/vignettes/earth-notes.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/earth/vignettes/earth-notes.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4619" LastActivityDate="2014-10-15T03:08:59.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2280" PostTypeId="2" ParentId="536" CreationDate="2014-10-15T11:37:47.090" Score="0" Body="&lt;p&gt;In the case you mention, I recommend to keep the changes as a dictionary, for instance in a .csv file. Write a script that replaces the values in the original data based on the translation in your dictionary. That way, you separate the corrections from the script itself.&lt;/p&gt;&#xA;" OwnerUserId="906" LastActivityDate="2014-10-15T11:37:47.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2281" PostTypeId="2" ParentId="2258" CreationDate="2014-10-15T20:37:40.633" Score="2" Body="&lt;p&gt;I would say... it really depends. You may need to: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;use&lt;/em&gt; machine learning algorithms: this will be useful for specific applications you may have. In this situation what you need is some programming skills and the taste for testing (practicing will make you strong). Here maths are not so much required I would say. &lt;/li&gt;&#xA;&lt;li&gt;be able to &lt;em&gt;modify&lt;/em&gt; existing algorithms. Your specific application may be reticent to regular algorithms, so you may need to adapt them to get maximum efficiency. Here maths come into play. &lt;/li&gt;&#xA;&lt;li&gt;understand the theory behind algorithms. Here maths are necessary, and will help you increase your knowledge of the field of machine learning, develop your own algorithms, speak the same langage as your peers... NN theory may be primitive as said by @Emre, but for instance this is not the case for SVM (the theory behind SVM requires e.g. to understand &lt;a href=&quot;http://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space&quot; rel=&quot;nofollow&quot;&gt;reproducing kernel Hilbert spaces&lt;/a&gt;). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the mid term for sure you will need strong maths. But you don't need to wait for them to come to you, you can start right now with linear algebra, which is beautiful and useful for everything. And in case you encounter (possibly temporary) difficulties of any sort with maths, keep on practicing the way you already do (many people can talk about the perceptron but are not able to make a perceptron in Java), this is very valuable. &lt;/p&gt;&#xA;" OwnerUserId="3317" LastActivityDate="2014-10-15T20:37:40.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2284" PostTypeId="1" CreationDate="2014-10-16T05:44:40.183" Score="6" ViewCount="248" Body="&lt;p&gt;I want to identifies different queries in sentences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like - &lt;code&gt;Who is Bill Gates and where he was born?&lt;/code&gt; or &lt;code&gt;Who is Bill Gates, where he was born?&lt;/code&gt; contains two queries &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Who is Bill Gates?&lt;/li&gt;&#xA;&lt;li&gt;Where Bill Gates was born&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I worked on Coreference resolution, so I can identify that &lt;code&gt;he&lt;/code&gt; points to &lt;code&gt;Bill Gates&lt;/code&gt; so resolved sentence is &quot;Who is Bill Gates, where Bill Gates was born&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like wise&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MGandhi is good guys, Where he was born?&#xA;single query&#xA;who is MGandhi and where was he born?&#xA;2 queries&#xA;who is MGandhi, where he was born and died?&#xA;3 quries&#xA;India won world cup against Australia, when?&#xA;1 query (when India won WC against Auz)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can perform Coreference resolution (Identifying and converting &lt;code&gt;he&lt;/code&gt; to &lt;code&gt;Gandhi&lt;/code&gt;) but not getting how can I distinguish queries in it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to do this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I checked various sentence parser, but as this is pure nlp stuff, sentence parser does not identify it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to find &quot;Sentence disambiguation&quot; like &quot;word sense disambiguation&quot;, but nothing exist like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help or suggestion would be much appreciable. &lt;/p&gt;&#xA;" OwnerUserId="4662" LastEditorUserId="4662" LastEditDate="2014-10-16T09:26:18.960" LastActivityDate="2015-02-13T11:08:14.467" Title="Is it possible to identify different queries/questions in sentence?" Tags="|machine-learning|data-mining|nlp|social-network-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2285" PostTypeId="1" CreationDate="2014-10-16T06:37:09.887" Score="2" ViewCount="188" Body="&lt;p&gt;Recently in a data analytic job interview for an e-commerce site, they asked me, do i have some knowledge of buyer classification problem. Unfortunately i heard this term for the first time.&lt;br&gt;&#xA;After interview i tried to search a lot about it over google but didn't find something meaningful. Please any one let me know if you have heard this term before and paste some links explaining this concept. Thanks   &lt;/p&gt;&#xA;" OwnerUserId="713" LastActivityDate="2014-10-16T21:12:09.490" Title="what is buyer classification problem?" Tags="|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2286" PostTypeId="2" ParentId="2284" CreationDate="2014-10-16T08:03:15.593" Score="2" Body="&lt;p&gt;The basic thing, you can do in that situation, is to split your query into N simple sentences each of which should be processed in order to receive YES/NO answer considering if the sentence is a query. That way you will receive following results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: Gandhi is good guys, Where he was born?&#xA;-&amp;gt;&#xA;Gandhi is good guys - not query&#xA;Where he was born?  - query&#xA;===&#xA;1 query&#xA;&#xA;Input: who is MGandhi and where was he born?&#xA;-&amp;gt;&#xA;who is MGandhi     - query&#xA;where was he born? - query&#xA;===&#xA;2 queries&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This approach will require anaphora resolution (in order to convert &lt;code&gt;he&lt;/code&gt; into &lt;code&gt;Gandhi&lt;/code&gt; in first example) and a parser to correctly divide complex sentence into simple ones.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2014-10-16T08:03:15.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2287" PostTypeId="1" AcceptedAnswerId="2288" CreationDate="2014-10-16T12:15:32.017" Score="2" ViewCount="332" Body="&lt;p&gt;I am kind of a newbie on machine learning and I would like to ask some questions based on a problem I have .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have x y z as variable and I have values of these variables as time progresses like :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t0  = x0 y0 z0  &lt;br&gt;&#xA;t1  = x1 y1 z1  &lt;br&gt;&#xA;tn  = xn yn zn  &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want a model that when it's given 3 values of x , y , z I want a prediction of them like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input : x_test y_test z_test &#xA;Output : x_prediction y_prediction z_prediction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These values are float numbers. What is the best model for this kind of problem? &#xA;Thanks in advance for all the answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details:&#xA;Ok so let me give some more details about the problems so as to be more specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have run certain benchmarks and taken values of performance counters from the cores of a system per interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The performance counters are the x , y , z in the above example.They are dependent to each other.Simple example is x = IPC , y  =  Cache misses , z  = Energy at Core.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I got this dataset of all these performance counters per interval .What I want to do is create a model that after learning from the training dataset , it will be given a certain state of the core ( the performance counters) and predict the performance counters that the core will have in the next interval.&lt;/p&gt;&#xA;" OwnerUserId="4668" LastEditorUserId="4668" LastEditDate="2014-10-27T16:04:23.527" LastActivityDate="2014-10-27T16:04:23.527" Title="Regression Model for explained model(Details inside)" Tags="|machine-learning|logistic-regression|predictive-modeling|regression|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2288" PostTypeId="2" ParentId="2287" CreationDate="2014-10-16T14:34:33.180" Score="0" Body="&lt;p&gt;AFAIK if you want to predict the value of one variable, you need to have one or more variables as predictors; i.e.: you assume the behaviour of one variable can be explained by the behaviour of other variables.&#xA;In your case you have three independent variables whose value you want to predict, and since you don't mention any other variables, I assume that each variable depends on the others. In that case you could fit three models (for instance, regression models), each of which would predict the value of one variable, based on the others. As an example, to predict x:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_prediction=int+cy*y_test+cz*z_test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;, where int is the intercept and cy, cz, the coefficients of the linear regression. &#xA;Likewise, in order to predict y and z:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_prediction=int+cx*x_test+cx*z_test&#xA;z_prediction=int+cx*x_test+cy*y_test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3159" LastActivityDate="2014-10-16T14:34:33.180" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2290" PostTypeId="2" ParentId="2285" CreationDate="2014-10-16T21:12:09.490" Score="2" Body="&lt;p&gt;Buyer classification is used to categorize users who purchase groups of items. Buyers are categorized in order to be targeted for advertising. When users buy similar items, they are more likely to buy similar items in the future. This is useful information when pricing items on a website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A clear example, a website selling nutritional supplements may want to target different buyers by category. Men in their 20s are unlikely to purchase menopause supplements, and women in their 50s are also unlikely to buy creatine. So, by analyzing user purchase history and categorizing types of buyers, the site can send intelligent promotions - males in their 20s get ads for creatine while women in their 50s get ads for menopause supplements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if I want to run a sale to attract customers, I don't want to offer lower prices on items which are often purchased together. I'd rathe price one low to attract the buyer, and then hope they buy the complimentary item at full price. Categorizing buyers also helps with this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to read up on shopping cart analysis, which is not a new problem. Department stores have been analyzing shopping carts and classifying buyers long before online shopping was popularized. That's why you have to use membership cards to get the 'special' prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fine tuning these details can increase revenue substantially.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-10-16T21:12:09.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2293" PostTypeId="1" AcceptedAnswerId="2296" CreationDate="2014-10-17T10:47:13.197" Score="10" ViewCount="2263" Body="&lt;p&gt;I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms themselves to be scalable, so this excludes using tools such as Weka, R, or even RHadoop. The &lt;a href=&quot;https://mahout.apache.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Mahout Library&lt;/a&gt; seems to be a good option, and it features &lt;a href=&quot;https://mahout.apache.org/users/basics/algorithms.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;algorithms for regression and clustering tasks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am struggling to find is a solution for anomaly or outlier detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Mahout features Hidden Markov Models and a variety of clustering techniques (including K-Means) I was wondering if it would be possible to build a model to detect outliers in time-series, using any of this. I would be grateful if somebody experienced on this could advice me&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;if it is possible, and in case it is&lt;/li&gt;&#xA;&lt;li&gt;how-to do it, plus&lt;/li&gt;&#xA;&lt;li&gt;an estimation of the effort involved and&lt;/li&gt;&#xA;&lt;li&gt;accuracy/problems of this approach.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3159" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:23.533" LastActivityDate="2017-05-05T11:04:04.157" Title="Scalable Outlier/Anomaly Detection" Tags="|data-mining|bigdata|algorithms|outlier|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2294" PostTypeId="2" ParentId="253" CreationDate="2014-10-17T10:55:04.147" Score="3" Body="&lt;p&gt;It strongly depends on the environment/company you are working with. In my eyes there is a &amp;quot;big data&amp;quot; hype at the moment and a lot of companies try to enter the field with hadoop based solutions - what makes hadoop also a buzzword but its not always the best solution.&lt;/p&gt;&#xA;&lt;p&gt;In my mind, a good Data Scientist should be able to ask the right questions and keep on asking again until its clear whats really needed. Than a good DataScientist - of course - needs to know how to address the problem (or at least know someone who can). Otherwise your stakeholder could be frustrated :-)&lt;/p&gt;&#xA;&lt;p&gt;So, I would say it is not absolutely necessary to learn Hadoop.&lt;/p&gt;&#xA;" OwnerUserId="4676" LastEditorUserId="29169" LastEditDate="2021-02-08T01:59:05.257" LastActivityDate="2021-02-08T01:59:05.257" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2296" PostTypeId="2" ParentId="2293" CreationDate="2014-10-17T12:50:56.243" Score="7" Body="&lt;p&gt;I would take a look at &lt;a href=&quot;https://github.com/tdunning/t-digest&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-digest algorithm&lt;/a&gt;. It's &lt;a href=&quot;https://issues.apache.org/jira/browse/MAHOUT-1361&quot; rel=&quot;nofollow noreferrer&quot;&gt;been merged into mahout&lt;/a&gt; and also a part of &lt;a href=&quot;http://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/quantile/TDigest.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;some other libraries&lt;/a&gt; for big data streaming. You can get more about this algorithm particularly and big data anomaly detection in general in next resources:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://info.mapr.com/rs/mapr/images/Practical_Machine_Learning_Anomaly_Detection.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical machine learning anomaly detection book.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://youtube.com/watch?v=i-mSV63Q9rA#t=757&quot; rel=&quot;nofollow noreferrer&quot;&gt;Webinar: Anomaly Detection When You Don't Know What You Need to Find&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://info.prelert.com/anomaly-detection-in-elasticsearch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anomaly Detection in Elasticsearch&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://oreilly.com/pub/e/3211&quot; rel=&quot;nofollow noreferrer&quot;&gt;Beating Billion Dollar Fraud Using Anomaly Detection: A Signal Processing Approach using Argyle Data on the Hortonworks Data Platform with Accumulo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4679" LastEditorUserId="4679" LastEditDate="2017-05-05T11:04:04.157" LastActivityDate="2017-05-05T11:04:04.157" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2297" PostTypeId="1" CreationDate="2014-10-17T13:34:48.540" Score="1" ViewCount="498" Body="&lt;p&gt;I'm looking for a topic for my masters thesis. Machine learning is my primary domain and I want to work on probabilistic models and applied probability in Machine Learning. Please suggest some exciting new topics that would make for a good masters thesis subject.&#xA;Anything related to Markov chains Monte Carlo, Bayesian methods, Probabilistic graphical models, Markov models and so on in context of machine learning would be great!&lt;/p&gt;&#xA;" OwnerUserId="2475" LastActivityDate="2014-10-17T13:34:48.540" Title="Masters thesis topics in Applied probability and Probabilistic models in Machine Learning" Tags="|machine-learning|" AnswerCount="0" CommentCount="1" ClosedDate="2020-05-26T00:07:43.397" ContentLicense="CC BY-SA 3.0" />
  <row Id="2298" PostTypeId="1" AcceptedAnswerId="2363" CreationDate="2014-10-17T13:58:39.353" Score="2" ViewCount="360" Body="&lt;p&gt;I've built a toy Random Forest model in &lt;code&gt;R&lt;/code&gt; (using the &lt;code&gt;German Credit&lt;/code&gt; dataset from the &lt;code&gt;caret&lt;/code&gt; package), exported it in &lt;code&gt;PMML 4.0&lt;/code&gt; and deployed onto Hadoop, using the &lt;code&gt;Cascading Pattern&lt;/code&gt; library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've run into an issue where &lt;code&gt;Cascading Pattern&lt;/code&gt; scores the same data differently (in a binary classification problem) than the same model in &lt;code&gt;R&lt;/code&gt;. Out of 200 observations, 2 are scored differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this? Could it be due to a difference in the implementation of Random Forests? &lt;/p&gt;&#xA;" OwnerUserId="1127" LastActivityDate="2014-10-29T14:37:00.910" Title="Differences in scoring from PMML model on different platforms" Tags="|machine-learning|r|apache-hadoop|random-forest|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2302" PostTypeId="1" AcceptedAnswerId="2307" CreationDate="2014-10-18T04:58:45.100" Score="3" ViewCount="5200" Body="&lt;p&gt;I want to analyze the effectiveness and efficiency of kernel methods for which I would require 3 different data-set in 2 dimensional space for each of the following cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;BAD_kmeans: The data set for which the kmeans clustering algorithm&#xA;will not perform well.&lt;/li&gt;&#xA;&lt;li&gt;BAD_pca: The data set for which the Principal Component Analysis&#xA;(PCA) dimension reduction method upon projection of the original&#xA;points into 1-dimensional space (i.e., the first eigenvector) will&#xA;not perform well.&lt;/li&gt;&#xA;&lt;li&gt;BAD_svm: The data set for which the linear Support Vector Machine&#xA;(SVM) supervised classification method using two classes of points&#xA;(positive and negative)  will not perform well.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which packages can I use in R to generate the random 2d data-set for each of the above cases ? A sample script in R would help in understanding&lt;/p&gt;&#xA;" OwnerUserId="3577" LastActivityDate="2014-10-19T03:35:17.557" Title="R Script to generate random dataset in 2d space" Tags="|machine-learning|classification|r|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2303" PostTypeId="1" CreationDate="2014-10-18T08:08:25.513" Score="4" ViewCount="2668" Body="&lt;p&gt;I would like to learn both Python and R for usage in data science projects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently unemployed, fresh out of university, scouting around for jobs and thought it would be good if I get some Kaggle projects under my profile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have very little knowledge in either language. Have used Matlab and C/C++ in the past. But I haven't produced production quality code or developed an application or software in either language. It has been dirty coding for academic usage all along.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used a little bit of Python, in a university project, but I dont know the fundamentals like what is a package , etc etc. ie havent read the intricacies of the language using a standard Python Textbook etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have done some amount of coding in C/C++ way back (3-4 years back then switched over to Matlab/Octave).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to get started in Python Numpy Scipy scikit-learn and pandas etc. but just reading up Wikipedia articles or Python textbooks is going to be infeasible for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And same goes with R, except that I have zero knowledge of R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="3223" LastActivityDate="2020-08-02T12:49:46.867" Title="Python and R good tutorials?" Tags="|machine-learning|r|python|" AnswerCount="9" CommentCount="2" ClosedDate="2014-11-13T04:38:24.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="2304" PostTypeId="1" CreationDate="2014-10-18T08:19:00.607" Score="1" ViewCount="1095" Body="&lt;p&gt;Is there a good java library for doing time series energy consumption forecasting based on weather data and other variables?&lt;/p&gt;&#xA;" OwnerUserId="4692" LastEditorUserId="29169" LastEditDate="2021-01-16T00:13:14.563" LastActivityDate="2021-01-16T00:13:14.563" Title="Energy consumption time series forcasting" Tags="|time-series|java|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2305" PostTypeId="2" ParentId="2303" CreationDate="2014-10-18T17:32:52.583" Score="0" Body="&lt;p&gt;I can only recommend &lt;a href=&quot;http://adv-r.had.co.nz/&quot; rel=&quot;nofollow&quot;&gt;Advanced R&lt;/a&gt; by Hadley Wickham. I think it is at the same time incredibly rich in content and easy to read. You say you have zero knowledge in R, but I believe since you already have programming skills in other languages this book can complement very fruitfully any classical &quot;R beginner manual&quot; (for the latter see &lt;a href=&quot;http://cran.r-project.org/other-docs.html#english&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="3317" LastActivityDate="2014-10-18T17:32:52.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2307" PostTypeId="2" ParentId="2302" CreationDate="2014-10-19T03:24:22.877" Score="2" Body="&lt;p&gt;None of the algorithms you mention are good with data that has uniform distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;size &amp;lt;- 20             #length of random number vectors&#xA;set.seed(1) &#xA;x &amp;lt;- runif(size)          # generate samples from uniform distribution (0.0, 1.0)&#xA;y &amp;lt;-runif(size) &#xA;df &amp;lt;-data.frame(x,y)&#xA;&#xA;# other distributions: rpois, rmvnorm, rnbinom, rbinom, rbeta, rchisq, rexp, rgamma, rlogis, rstab, rt, rgeom, rhyper, rwilcox, rweibull.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://statistics.ats.ucla.edu/stat/r/modules/prob_dist.htm&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; for tutorial on generating random samples from distributions.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For specific set of randomized data sets that are 'hard' for these methods (e.r. linearly inseparable &lt;em&gt;n&lt;/em&gt;-classes XOR patterns), see this blog post (incl. R code): &lt;a href=&quot;http://tjo-en.hatenablog.com/entry/2014/01/06/234155&quot; rel=&quot;nofollow&quot;&gt;http://tjo-en.hatenablog.com/entry/2014/01/06/234155&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="609" LastEditDate="2014-10-19T03:35:17.557" LastActivityDate="2014-10-19T03:35:17.557" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2308" PostTypeId="1" AcceptedAnswerId="2332" CreationDate="2014-10-19T11:02:44.397" Score="1" ViewCount="7902" Body="&lt;p&gt;I am looking for a thesis to complete my master M2, I will work on a topic in the big data's field (creation big data applications), using hadoop/mapReduce and Ecosystem ( visualisation, analysis ...), Please suggest some topics or project that would make for a good masters thesis subject.&lt;br&gt;&#xA;I add that I have bases in data warehouses, databases, data mining, good skills in programming, system administration and cryptography ... &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="4705" LastEditorUserId="3151" LastEditDate="2015-12-23T11:02:49.097" LastActivityDate="2015-12-23T11:02:49.097" Title="Masters thesis topics in big data" Tags="|bigdata|apache-hadoop|research|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2311" PostTypeId="2" ParentId="1253" CreationDate="2014-10-19T19:12:32.940" Score="7" Body="&lt;p&gt;First we need to understand why we need Deep learning. To build models ML need Test Data with Labels (supervised or unsupervised). In many domains as the data grows maintaining the data with labels is hard. Deep learning networks doesn't need labeled data. The Deep learning algorithms can figure out the labels. So this obviates the need for domain experts to come out with labels for the data which is very important in the areas of speech recognition, computer vision, and language understanding. Google Cat image recognition is a very interesting experiment. Also it is interesting to know &lt;a href=&quot;http://www.wired.com/2014/01/geoffrey-hinton-deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;Geoffrey Hinton&amp;quot;&lt;/a&gt; the professor who was hired by Google.&lt;/p&gt;&#xA;&lt;p&gt;You may get more insight as you explore in this framework.&lt;/p&gt;&#xA;" OwnerUserId="4711" LastEditorUserId="98307" LastEditDate="2020-08-05T08:41:32.857" LastActivityDate="2020-08-05T08:41:32.857" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2313" PostTypeId="1" CreationDate="2014-10-20T06:38:16.490" Score="6" ViewCount="1617" Body="&lt;p&gt;Where is the difference between one-class, binary-class and multinominal-class classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I like to classify text in lets say four classes and also want the system to be able to tell me that none of these classes matches the unknown/untrained test-data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Couldn't I just use all the methods that I mentioned above to reach my goal?&#xA;e.g. I could describe C1, C2, C3 and C4 as four different trainings-sets for binary-classification and use the trained models to label an unknow data-set ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just by saying, Training-Set for C1 contains class 1 (all good samples for C1) and class 0 (mix of all C2, C3 and C4 as bad samples for C1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is unlabeled-data C1 -&gt; 1 or 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is unlabeled-data C2 -&gt; 1 or 0&#xA;... and so on ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For multinominal classification I could just define a training-set containing all good sample data for C1, C2, C3 and C4 in one training-set and then use the one resulting model for classification ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But where is the difference between this two methods? (except of that I have to use different algorithms)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And how would I define a training-set for the described problem of categorizing data in those four classes using one-class classfication (is that even possible)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excuse me if I'm completely wrong in my thinking. Would appreciate an answer that makes the methodology a little bit clearer to me =)&lt;/p&gt;&#xA;" OwnerUserId="4717" LastEditorUserId="4717" LastEditDate="2014-10-20T06:46:20.017" LastActivityDate="2015-03-03T14:21:51.033" Title="Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?" Tags="|machine-learning|data-mining|classification|categorical-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2314" PostTypeId="1" CreationDate="2014-10-20T08:48:42.167" Score="15" ViewCount="7029" Body="&lt;p&gt;A Random Forest (RF) is created by an ensemble of Decision Trees's (DT). By using bagging, each DT is trained in a different data subset. Hence, &lt;strong&gt;is there any way of implementing an on-line random forest by adding more decision tress on new data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, we have 10K samples and train 10 DT's. Then we get 1K samples, and instead of training again the full RF, we add a new DT. The prediction is done now by the Bayesian average of 10+1 DT's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, if we keep all the previous data, the new DT's can be trained mainly in the new data, where the probability of picking a sample is weighted depending how many times have been already picked.&lt;/p&gt;&#xA;" OwnerUserId="4719" LastActivityDate="2019-03-17T04:10:21.730" Title="On-line random forests by adding more single Decisions Trees" Tags="|random-forest|online-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2315" PostTypeId="2" ParentId="1028" CreationDate="2014-10-20T09:31:18.650" Score="30" Body="&lt;p&gt;Every ML algorithm with high complexity can overfit. However, the OP is asking whether an RF will not overfit when increasing the number of trees in the forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, ensemble methods reduces the prediction variance to almost nothing, improving the accuracy of the ensemble. If we define the variance of the expected generalization error of an individual randomized model as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZUORL.gif&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://arxiv.org/abs/1407.7502&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, the variance of the expected generalization error of an ensemble corresponds to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Zf9e.gif&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;p(x)&lt;/code&gt; is the Pearson’s correlation coefficient between the predictions of two randomized models trained on the same data from two independent seeds. If we increase the number of DT's in the RF, larger &lt;code&gt;M&lt;/code&gt;, the variance of the ensemble decreases when &lt;code&gt;ρ(x)&amp;lt;1&lt;/code&gt;. Therefore, the variance of an ensemble is strictly smaller than the variance of an individual model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell, increasing the number of individual randomized models in an ensemble will never increase the generalization error.&lt;/p&gt;&#xA;" OwnerUserId="4719" LastEditorUserId="13727" LastEditDate="2015-11-17T16:19:49.323" LastActivityDate="2015-11-17T16:19:49.323" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2316" PostTypeId="1" CreationDate="2014-10-19T21:03:35.390" Score="1" ViewCount="48" Body="&lt;p&gt;Where is the difference between one-class, binary-class and multinominal-class classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I like to classify text in lets say four classes and also want the system to be able to tell me that none of these classes matches the unknown/untrained test-data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Couldn't I just use all the methods that I mentioned above to reach my goal?&#xA;e.g. I could describe C1, C2, C3 and C4 as four different trainings-sets for binary-classification and use the trained models to label an unknow data-set ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just by saying, Training-Set for C1 contains class 1 (all good samples for C1) and class 0 (mix of all C2, C3 and C4 as bad samples for C1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is unlabeled-data C1 -&gt; 1 or 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is unlabeled-data C2 -&gt; 1 or 0&#xA;... and so on ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For multinominal classification I could just define a training-set containing all good sample data for C1, C2, C3 and C4 in one training-set and then use the one resulting model for classification ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But where is the difference between this two methods? (except of that I have to use different algorithms)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And how would I define a training-set for the described problem of categorizing data in those four classes using one-class classfication (is that even possible)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excuse me if I'm completely wrong in my thinking. Would appreciate an answer that makes the methodology a little bit clearer to me =)&lt;/p&gt;&#xA;" OwnerUserId="4717" OwnerDisplayName="Lothbrok" LastActivityDate="2014-10-20T10:38:59.437" Title="Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?" Tags="|machine-learning|classification|data-mining|" AnswerCount="0" CommentCount="1" ClosedDate="2014-10-21T14:09:06.617" ContentLicense="CC BY-SA 3.0" />
  <row Id="2317" PostTypeId="2" ParentId="2303" CreationDate="2014-10-20T15:32:58.097" Score="0" Body="&lt;p&gt;&lt;em&gt;The Art of R Programming&lt;/em&gt; by Normal Matloff is a great way to find your way towards being an R user.  I've recommended this book to several people navigating the tutorial / book universe and to my knowledge they've all stuck with it. &lt;/p&gt;&#xA;" OwnerUserId="4724" LastActivityDate="2014-10-20T15:32:58.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2318" PostTypeId="2" ParentId="2303" CreationDate="2014-10-20T16:12:45.427" Score="2" Body="&lt;p&gt;There is an online data science &quot;game&quot; that takes you from learning how to use Python for loading a csv and using scikit to machine learning algorithms such as support vector machines. Here is a &lt;a href=&quot;http://mentally-physically-spiritually-strong.com/2014/10/18/explore-data-science-game/&quot; rel=&quot;nofollow&quot;&gt;blog post&lt;/a&gt; with a demo video and the actual site is &lt;a href=&quot;https://exploredatascience.com&quot; rel=&quot;nofollow&quot;&gt;Explore Data Science&lt;/a&gt;. Personally, I think its genius. &lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2014-10-20T16:12:45.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2320" PostTypeId="2" ParentId="19" CreationDate="2014-10-20T19:40:34.193" Score="13" Body="&lt;p&gt;To me Big Data is primarily about the tools (after all, that's where it started); a &quot;big&quot; dataset is one that's too big to be handled with conventional tools - in particular, big enough to demand storage and processing on a cluster rather than a single machine. This rules out a conventional RDBMS, and demands new techniques for processing; in particular, various Hadoop-like frameworks make it easy to distribute a computation over a cluster, at the cost of restricting the form of this computation. I'll second the reference to &lt;a href=&quot;http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html&quot;&gt;http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html&lt;/a&gt; ; Big Data techniques are a last resort for datasets which are simply too big to handle any other way. I'd say any dataset for any purpose could qualify if it was big enough - though if the shape of the problem is such that existing &quot;big data&quot; tools aren't appropriate, then it would probably be better to come up with a new name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course there is some overlap; when I (briefly) worked at last.fm, we worked on the same 50TB dataset using Hadoop and also in an SQL database on a fairly ridiculous server (I remember it had 1TB RAM, and this is a few years ago). Which in a sense meant it both was and wasn't big data, depending on which job you were working on. But I think that's an accurate characterization; the people who worked on the Hadoop jobs found it useful to go to Big Data conferences and websites, while the people who worked on the SQL jobs didn't.&lt;/p&gt;&#xA;" OwnerUserId="4730" LastActivityDate="2014-10-20T19:40:34.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2321" PostTypeId="1" CreationDate="2014-10-20T22:13:32.420" Score="6" ViewCount="6465" Body="&lt;p&gt;I am drawing samples from two classes in the two-dimensional Cartesian space, each of which has the same covariance matrix $[2, 0; 0, 2]$.  One class has a mean of $[1.5, 1]$ and the other has a mean of $[1, 1.5]$.  If the priors are $4/7$ for the former and $3/7$ for the later, how would I derive the equation for the ideal decision boundary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it turns out that misclassifying the second class is twice as expensive as the first class, and the objective is to minimize the expected cost, what equation would I use for the best decision boundary?&lt;/p&gt;&#xA;" OwnerUserId="4736" LastEditorDisplayName="user10169" LastEditDate="2015-08-12T20:30:28.753" LastActivityDate="2015-08-12T20:30:28.753" Title="Bayes Optimal Decision Boundaries for Gaussian Data with Equal Covariance" Tags="|classification|discriminant-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2322" PostTypeId="1" CreationDate="2014-10-20T22:22:09.660" Score="4" ViewCount="255" Body="&lt;p&gt;Are there any good sources that explain how decision trees can be implemented in a scalable way on a distributed computing system.  Where in a given source is this explained?&lt;/p&gt;&#xA;" OwnerUserId="4736" LastEditorUserId="31513" LastEditDate="2017-05-17T06:16:47.753" LastActivityDate="2017-05-17T06:16:47.753" Title="Distributed Scalable Decision Trees" Tags="|classification|bigdata|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2323" PostTypeId="1" AcceptedAnswerId="39726" CreationDate="2014-10-21T00:09:40.597" Score="11" ViewCount="6107" Body="&lt;p&gt;Can anyone explain how field-aware factorization machines (FFM) compare to standard Factorization Machines (FM)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standard:&#xA;&lt;a href=&quot;http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf&quot;&gt;http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Field Aware&quot;:&#xA;&lt;a href=&quot;http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf&quot;&gt;http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1138" LastEditorUserId="20995" LastEditDate="2016-10-01T16:31:45.827" LastActivityDate="2018-10-17T20:53:25.863" Title="Field Aware Factorization Machines" Tags="|machine-learning|recommender-system|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2324" PostTypeId="2" ParentId="2323" CreationDate="2014-10-21T01:34:13.407" Score="1" Body="&lt;p&gt;Standard factorization machines have fields too. The &quot;novelty&quot; here seems to be the use of GBDT features and the application of the hashing tricks. Not to great effect, it seems: check out the minute range in performance on the last slide.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-21T01:34:13.407" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2325" PostTypeId="2" ParentId="2314" CreationDate="2014-10-21T02:43:26.790" Score="9" Body="&lt;p&gt;There's &lt;a href=&quot;http://www.vision.cs.chubu.ac.jp/CV-R/pdf/AmirICCVW2009.pdf&quot; rel=&quot;noreferrer&quot;&gt;a recent paper on this subject&lt;/a&gt; (&lt;em&gt;On-line Random Forests&lt;/em&gt;), coming from computer vision. &lt;a href=&quot;http://www.ymer.org/amir/software/online-random-forests/&quot; rel=&quot;noreferrer&quot;&gt;Here's an implementation&lt;/a&gt;, and a presentation: &lt;a href=&quot;http://www.slideshare.net/CvilleDataScience/online-random-forest-in-10-minutes&quot; rel=&quot;noreferrer&quot;&gt;Online random forests in 10 minutes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-21T02:43:26.790" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2326" PostTypeId="2" ParentId="2322" CreationDate="2014-10-21T06:12:51.887" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Apache Spark&lt;/a&gt; &lt;a href=&quot;http://spark-summit.org/2014/talk/scalable-distributed-decision-trees-in-spark-mllib&quot; rel=&quot;nofollow&quot;&gt;can do it&lt;/a&gt;, using the new &lt;a href=&quot;http://spark.apache.org/docs/1.1.0/mllib-decision-tree.html&quot; rel=&quot;nofollow&quot;&gt;MLLib&lt;/a&gt; library. &lt;a href=&quot;https://www.youtube.com/watch?v=3rRrcPXHu98&quot; rel=&quot;nofollow&quot;&gt;Here's a presentation&lt;/a&gt;, and &lt;a href=&quot;http://databricks.com/blog/2014/09/29/scalable-decision-trees-in-mllib.html&quot; rel=&quot;nofollow&quot;&gt;here are some benchmarks&lt;/a&gt;. Bindings are available for python, scala, and java.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-10-21T06:12:51.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2327" PostTypeId="2" ParentId="2303" CreationDate="2014-10-21T07:06:21.580" Score="1" Body="&lt;p&gt;I have found the video tutorial/IPython notebook format really helped me get into the python ecosystem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There were two tutorials at SciPy 2013 that cover sklearn (&lt;a href=&quot;https://www.youtube.com/watch?v=r4bRUvvlaBw&quot; rel=&quot;nofollow&quot;&gt;part 1 of 1st tutorial&lt;/a&gt;, &lt;a href=&quot;https://github.com/jakevdp/sklearn_scipy2013&quot; rel=&quot;nofollow&quot;&gt;github repo for notebooks&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar tutorials, from PyCon2012 and PyData2012, are out there for pandas but I don't have the rep to link searching for &lt;code&gt;pandas tutorial&lt;/code&gt; on youtube should allow you to find them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you mention Kaggle, I guess you will have seen their getting started with python tutorial for the titanic passenger dataset (I don't have the rep here to provide a link but searching for &lt;code&gt;Getting Started with Python: Kaggle's Titanic Competition&lt;/code&gt; should get you there).&lt;/p&gt;&#xA;" OwnerUserId="4739" LastActivityDate="2014-10-21T07:06:21.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2328" PostTypeId="1" AcceptedAnswerId="2329" CreationDate="2014-10-21T09:42:51.610" Score="2" ViewCount="424" Body="&lt;p&gt;There're many data points, each of which is associated with two coordinates and a numeral value, or three coordinates. And I wish it is coloured.&#xA;I checked packages &quot;scatterplot3d&quot; and &quot;plot3D&quot; but I couldn't find one like the example I give. It is like it has a fitting surface.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is basically like the following. In this way I think this kind of plot is gonna be perfectly suitble for this data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    ki,kt,Top10AverageF1Score&#xA;    360,41,0.09371256716549396&#xA;    324,41,0.09539634212851525&#xA;    360,123,0.09473510831594467&#xA;    36,164,0.09773486852645874&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I also may have one more additional variable, which makes it like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    NeighborhoodSize,ki,kt,Top10AverageF1Score&#xA;    10,360,41,0.09371256716549396&#xA;    15,324,41,0.09539634212851525&#xA;    15,360,123,0.09473510831594467&#xA;    20,36,164,0.09773486852645874&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do you also have any good idea for visualizing the second case? What kind of plot and which packages and functions, etc.&lt;/strong&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/5whM7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3443" LastEditorUserId="3443" LastEditDate="2014-10-21T12:50:19.847" LastActivityDate="2014-10-21T12:50:19.847" Title="Which packages or functions can I use in R to plot 3D data like this?" Tags="|r|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2329" PostTypeId="2" ParentId="2328" CreationDate="2014-10-21T10:22:21.390" Score="2" Body="&lt;p&gt;You could use the &lt;code&gt;wireframe&lt;/code&gt; function from the &lt;code&gt;lattice&lt;/code&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;lattice&quot;)&#xA;wireframe(volcano[1:30, 1:30], shade=TRUE, zlab=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/BQHGX.png&quot; alt=&quot;wireframe plot example&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2961" LastActivityDate="2014-10-21T10:22:21.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2332" PostTypeId="2" ParentId="2308" CreationDate="2014-10-22T08:30:00.990" Score="3" Body="&lt;p&gt;Since it's a master's thesis, how about writing something regarding decision trees, and their &quot;upgrades&quot;: boosting and Random Forests? And then integrate that with Map/Reduce, together with showing how to scale a Random Forest on Hadoop using M/R?&lt;/p&gt;&#xA;" OwnerUserId="1127" LastActivityDate="2014-10-22T08:30:00.990" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2334" PostTypeId="1" AcceptedAnswerId="2335" CreationDate="2014-10-22T14:53:42.127" Score="4" ViewCount="817" Body="&lt;p&gt;I want to analyze &lt;a href=&quot;http://grouplens.org/datasets/movielens/&quot; rel=&quot;nofollow&quot;&gt;MovieLens data set&lt;/a&gt; and load on my machine the M1 file. I combine actually two data files (ratings.dat and movies.dat) and sort the table according &lt;code&gt;'userId'&lt;/code&gt; and &lt;code&gt;'Time'&lt;/code&gt; columns. The head of my DataFrame looks like here (all columns values are corresponding to the original data sets):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [36]: df.head(10)&#xA;Out[36]: &#xA;        userId  movieId  Rating       Time                         movieName  \&#xA;40034        1      150       5  978301777                  Apollo 13 (1995)   &#xA;77615        1     1028       5  978301777               Mary Poppins (1964)   &#xA;550485       1     2018       4  978301777                      Bambi (1942)   &#xA;400889       1     1962       4  978301753         Driving Miss Daisy (1989)   &#xA;787274       1     1035       5  978301753        Sound of Music, The (1965)   &#xA;128308       1      938       4  978301752                       Gigi (1958)   &#xA;497972       1     3105       5  978301713                 Awakenings (1990)   &#xA;28417        1     2028       5  978301619        Saving Private Ryan (1998)   &#xA;6551         1     1961       5  978301590                   Rain Man (1988)   &#xA;35492        1     2692       4  978301570  Run Lola Run (Lola rennt) (1998)   &#xA;&#xA;                            genre  &#xA;40034                       Drama  &#xA;77615   Children's|Comedy|Musical  &#xA;550485       Animation|Children's  &#xA;400889                      Drama  &#xA;787274                    Musical  &#xA;128308                    Musical  &#xA;497972                      Drama  &#xA;28417            Action|Drama|War  &#xA;6551                        Drama  &#xA;35492        Action|Crime|Romance  &#xA;&#xA;[10 rows x 6 columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can not understand that the same user with user Id 1 see or rated the different movies (Apollo13 (Id:150), Mary Poppins (Id:1028) and Bambi (Id:2018) exactly at the same time (up to the milleseconds). If somebody works already with this data set, please, clear this situation.&lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="3281" LastEditDate="2014-10-22T15:43:44.803" LastActivityDate="2014-10-22T15:50:43.230" Title="MovieLens data set" Tags="|dataset|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2335" PostTypeId="2" ParentId="2334" CreationDate="2014-10-22T15:50:43.230" Score="2" Body="&lt;p&gt;When you enter ratings on movie lens, you get pages with 10 movies or so. You set all the ratings, then submit by clicking &quot;next page&quot; or something. &#xA;So I guess all the ratings for the same page are received at the same time, when you submit the page.&lt;/p&gt;&#xA;" OwnerUserId="4760" LastActivityDate="2014-10-22T15:50:43.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2337" PostTypeId="1" AcceptedAnswerId="2339" CreationDate="2014-10-23T14:51:57.160" Score="6" ViewCount="243" Body="&lt;p&gt;I am not sure whether I formulated the question correctly. Basically, what I want to do is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's suppose I have a list of 1000 strings which look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cvzxcvzx&lt;strong&gt;string&lt;/strong&gt;cvzcxvz&lt;/p&gt;&#xA;&#xA;&lt;p&gt;otortorotr&lt;strong&gt;string&lt;/strong&gt;grptprt&lt;/p&gt;&#xA;&#xA;&lt;p&gt;vmvmvmeop&lt;strong&gt;string2&lt;/strong&gt;vmrprp&lt;/p&gt;&#xA;&#xA;&lt;p&gt;vccermpqp&lt;strong&gt;string2&lt;/strong&gt;rowerm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;proororor&lt;strong&gt;string3&lt;/strong&gt;potrprt&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mprto2435&lt;strong&gt;string3&lt;/strong&gt;famerpaer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to extract these reoccuring strings that occur on the list. What solution should I use? Does anyone know about algorithm that could do this?&lt;/p&gt;&#xA;" OwnerUserId="4774" LastActivityDate="2014-10-24T18:35:41.350" Title="Clustering strings inside strings?" Tags="|nlp|text-mining|feature-extraction|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2338" PostTypeId="2" ParentId="326" CreationDate="2014-10-23T15:50:09.820" Score="7" Body="&lt;p&gt;In my experience, the answer depends on the project at hand. For pure research, I prefer R for two reasons: 1) broad variety of libraries and 2) much of the data science literature includes R samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the project requires an interactive interface to be used by laypersons, I've found R to be too constrained. Shiny is a great start, but it's not flexible enough yet. In these cases, I'll start to look at porting my R work over to Python or js.&lt;/p&gt;&#xA;" OwnerUserId="4776" LastActivityDate="2014-10-23T15:50:09.820" CommentCount="0" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="2339" PostTypeId="2" ParentId="2337" CreationDate="2014-10-23T17:10:40.890" Score="2" Body="&lt;p&gt;Interesting question! I have not encountered it before so here is a solution I just made up, inspired by the approach taken by the word2vec paper:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Define the pair-wise similarity based on the longest common substring (LCS), or the LCS normalized by the products of the string lengths. Cache this in a matrix for any pair of strings considered since it is expensive to calculate. Also consider approximations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find a Euclidean (hyperspherical, perhaps?) embedding that minimizes the error (Euclidean distance if using the ball, and the dot product if using the sphere). Assume random initialization, and use a gradient-based optimization method by taking the Jacobian of the error.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now you have a Hilbert space embedding, so cluster using your algorithm of choice!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Response to deleted comment asking how to cluster multiple substrings&lt;/em&gt;: The bulk of the complexity lies in the first stage; the calculation of the LCS, so it depends on efficiently you do that. I've had luck with genetic algorithms. Anyway, what you'd do in this case is define a similarity &lt;em&gt;vector&lt;/em&gt; rather than a scalar, whose elements are the k-longest pair-wise LCS; see &lt;a href=&quot;https://cstheory.stackexchange.com/questions/8361/algorithm-find-the-first-k-longest-substrings-between-two-similar-strings&quot;&gt;this&lt;/a&gt; discussion for algorithms. Then I would define the error by the sum of the errors corresponding to each substring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something I did not address is how to choose the dimensionality of the embedding. The word2vec paper might provide some heuristics; see &lt;a href=&quot;https://groups.google.com/forum/#!topic/word2vec-toolkit/HRvNPIqe6mM&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; discussion. I recall they used pretty big spaces, on the order of a 1000 dimensions, but they were optimizing something more complicated, so I suggest you start at R^2 and work your way up. Of course, you will want to use a higher dimensionality for the multiple LCS case.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="-1" LastEditDate="2017-04-13T12:32:33.517" LastActivityDate="2014-10-24T18:35:41.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2340" PostTypeId="2" ParentId="2304" CreationDate="2014-10-23T20:11:18.750" Score="1" Body="&lt;p&gt;If you just want to get a forecast, use automatic forecasting software like Autobox, ForecastPro, SCA, etc.  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Comparison of different software: &lt;a href=&quot;https://stats.stackexchange.com/questions/68253/expert-forecasting-software-evaluation&quot;&gt;https://stats.stackexchange.com/questions/68253/expert-forecasting-software-evaluation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Worked example: &lt;a href=&quot;http://people.duke.edu/~rnau/autocomp.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://people.duke.edu/~rnau/autocomp.htm&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are doing research on forecasting techniques, there are some Java packages, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;JMotif &lt;a href=&quot;http://code.google.com/p/jmotif/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://code.google.com/p/jmotif/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;JTSA &lt;a href=&quot;http://sourceforge.net/projects/jtsa/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://sourceforge.net/projects/jtsa/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, Java is not very widely used for this; if you must use Java you will likely have a feeling of being out in the cold.   R and Python would have much more packages, examples, etc.  A tiny sample of what is available in R:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387886974&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.amazon.com/Introductory-Time-Series-Paul-Cowpertwait/dp/0387886974/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-bloggers.com/time-series-analysis-and-mining-with-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.r-bloggers.com/time-series-analysis-and-mining-with-r/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="26" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-10-23T20:11:18.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2341" PostTypeId="2" ParentId="1155" CreationDate="2014-10-23T21:12:27.350" Score="1" Body="&lt;p&gt;I would think about starting with a power analysis: i.e. how many data points do you need to measure the effect (or parameter) that you are interested to a specified level of confidence, &lt;em&gt;ceteris paribus&lt;/em&gt;? Then, you estimate a cost. &lt;/p&gt;&#xA;" OwnerUserId="4779" LastActivityDate="2014-10-23T21:12:27.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2342" PostTypeId="2" ParentId="2322" CreationDate="2014-10-23T21:59:32.713" Score="2" Body="&lt;p&gt;You should read the paper from Google on PLANET, which was their distributed MapReduce-based implementation of random decision forests: &lt;a href=&quot;http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36296.pdf&quot; rel=&quot;nofollow&quot;&gt;http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36296.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may or may not like the architecture but there are a number of interesting ideas about scaling up here.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-10-23T21:59:32.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2343" PostTypeId="1" CreationDate="2014-10-24T23:13:07.803" Score="2" ViewCount="5994" Body="&lt;p&gt;I need some serious help. I am supposed to implement a project (Non-Existing as of now) for my Machine Learning course. I have no basics in AI or Data mining or Machine learning. I have been searching for a while and unable to find something that i can finish implementing in 3-4 weeks time. It carries a huge chunk of my final marks and no matter how much i try i am unable to understand how it works!&#xA;Can the machine learning masters please help me out with this. I need a project suggestion to start with. And i want to know how to proceed after gathering the data set. I am totally blank and running out of time for my graduation :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate your suggestions! Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4794" LastActivityDate="2014-12-19T07:02:47.587" Title="Small project ideas for Machine Learning" Tags="|machine-learning|bigdata|data-mining|clustering|algorithms|" AnswerCount="4" CommentCount="1" ClosedDate="2014-12-19T09:40:10.057" ContentLicense="CC BY-SA 3.0" />
  <row Id="2344" PostTypeId="1" CreationDate="2014-10-25T01:36:27.483" Score="2" ViewCount="872" Body="&lt;p&gt;I have a question about memory usage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to do 4 things:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) make a dataframe from one of several columns from a datasource, say a json string&#xA;2) make the third column of the original dataset the index to the dataframe&#xA;3) change the name of another column&#xA;4) change the series i've created to a dataframe&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is about memory efficiency. It seems that for step 1), I am first loading a whole dataframe, then run a concat command to concatenate the columns I want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For step 2, I again need to resave the new dataframe as another object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For step 3, it seems to stick so nothing there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please advise on a more efficient way to go about this, if that exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   df = pd.DataFrame(jsonobject)&#xA;   df = df.set_index(&quot;columnC&quot;)&#xA;   df.index.names= [&quot;foo&quot;]&#xA;   df1 = df[&quot;foo&quot;].map(lambda x:x[&quot;id&quot;])&#xA;   df2 = pd.DataFrame(df1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1223" LastActivityDate="2014-10-25T01:36:27.483" Title="pandas dataframes memory" Tags="|pandas|" AnswerCount="0" CommentCount="1" ClosedDate="2021-03-13T15:35:34.837" ContentLicense="CC BY-SA 3.0" />
  <row Id="2345" PostTypeId="2" ParentId="2343" CreationDate="2014-10-25T08:40:14.613" Score="3" Body="&lt;p&gt;Gather million twitter user accounts. Then try to guess their gender based on their avatar, tweets and who they follow.&lt;/p&gt;&#xA;" OwnerUserId="4774" LastActivityDate="2014-10-25T08:40:14.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2346" PostTypeId="1" CreationDate="2014-10-25T22:37:18.523" Score="6" ViewCount="226" Body="&lt;p&gt;I work in an analytical role at a a large financial services firm. We do a ton of daily reporting over metrics that rarely change in a meaningful way from day to day. From this daily reporting, our management is required to extract what was important yesterday and what important trends have developed / are developing over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to change this to a model of daily exception reporting and weekly trend reporting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features might include: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;User report consolidation (so there's only one daily email)&lt;/li&gt;&#xA;&lt;li&gt;report ordering based upon level of variance from past performance (see the most important stuff first)&lt;/li&gt;&#xA;&lt;li&gt;HTML email support (with my audience, pretty counts)&lt;/li&gt;&#xA;&lt;li&gt;Web interface to allow preference changes, including LDAP support (make administration easier)&lt;/li&gt;&#xA;&lt;li&gt;Unsubscribe feature at the report level&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here's what I'd like to know:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are the practical problems I might run into?&lt;/li&gt;&#xA;&lt;li&gt;What is the best way to display the new reports?&lt;/li&gt;&#xA;&lt;li&gt;How should  I define an &quot;exception&quot;? How can I know if my definition is a good one?&lt;/li&gt;&#xA;&lt;li&gt;I assume I'd be using a mix of Python, SQL, and powershell. Anything else I should consider, e.g. R? What are some good resources?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4800" LastEditorUserId="29575" LastEditDate="2017-12-14T21:41:01.897" LastActivityDate="2017-12-14T21:41:01.897" Title="Business exception reporting" Tags="|recommender-system|visualization|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2347" PostTypeId="2" ParentId="2346" CreationDate="2014-10-25T23:07:15.843" Score="1" Body="&lt;p&gt;Here's some practical advice from my own experiences-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The first thing to do is to convince management the the change will be for the better. A mock-up of a sample report can be very useful here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Even if management agrees, they'll still want to know why variances have occurred, and hence you'll need to be able to supply more data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Although exception reporting is best, management will want to see everything anyway, as doing so makes them feel as though they are doing something useful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Don't change everything at once- too big a change at once can cause resistance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For how to best present the data, read Edward Tufte's books, at the very least his first one, &quot;The Visual Display of Quantitative Data&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Defining what's an exception can be hard, because the recipients will each have their own  ideas. Using say, a 95% confidence interval is good, but it won't be universally liked. Some people will consider any change above $X significant, and others will want to see everything that's more than X% different from the prior period. Have fun with this part :(&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1241" LastEditorUserId="1241" LastEditDate="2014-10-30T13:40:45.283" LastActivityDate="2014-10-30T13:40:45.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2348" PostTypeId="1" CreationDate="2014-10-25T23:46:14.493" Score="2" ViewCount="2775" Body="&lt;p&gt;I have Train and Test data, how to calculate classification accuracy with confusion matrix ? Thanks&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@attribute outlook {sunny, overcast, rainy}&#xA;@attribute temperature {hot, mild, cool}&#xA;@attribute humidity {high, normal}&#xA;@attribute windy {TRUE, FALSE}&#xA;@attribute play {yes, no}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Train:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1   sunny       hot     high    FALSE   no&#xA;2   sunny       hot     high    TRUE    no&#xA;3   overcast    hot     high    FALSE   yes&#xA;4   rainy       mild    high    FALSE   yes&#xA;5   rainy       cool    normal  FALSE   yes&#xA;6   rainy       cool    normal  TRUE    no&#xA;7   sunny       cool    normal  FALSE   yes&#xA;8   rainy       mild    normal  FALSE   yes&#xA;9   sunny       mild    normal  TRUE    yes&#xA;10  overcast    mild    high    TRUE    yes&#xA;11  overcast    hot     normal  FALSE   yes&#xA;12  rainy       mild    high    TRUE    no&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;overcast    cool    normal  TRUE    yes&#xA;sunny       mild    high    FALSE   no&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Rules found:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(humidity,normal), (windy,FALSE) -&amp;gt; (play,yes) [Support=0.33 , Confidence=1.00 , Correctly Classify= 4, 8, 9, 12]&#xA;(outlook,overcast) -&amp;gt; (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 2, 11]&#xA;(outlook,rainy), (windy,FALSE) -&amp;gt; (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 3]&#xA;(outlook,sunny), (temperature,hot) -&amp;gt; (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 0, 1]&#xA;(outlook,sunny), (humidity,normal) -&amp;gt; (play,yes) [Support=0.17 , Confidence=1.00 , Correctly Classify= 10]&#xA;(outlook,rainy), (windy,TRUE) -&amp;gt; (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 5, 13]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3503" LastEditorUserId="3503" LastEditDate="2014-10-27T11:13:51.847" LastActivityDate="2014-10-27T17:39:18.997" Title="How to calculate classification accuracy with confusion matrix?" Tags="|classification|accuracy|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2349" PostTypeId="1" AcceptedAnswerId="2350" CreationDate="2014-10-26T13:12:23.597" Score="5" ViewCount="199" Body="&lt;p&gt;For my Computational Intelligence class, I'm working on classifying short text. One of the papers that I've found makes a lot of use of &lt;em&gt;granular computing&lt;/em&gt;, but I'm struggling to find a decent explanation of what exactly it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I can gather from the paper, it sounds to me like granular computing is very similar to fuzzy sets. So, what exactly is the difference. I'm asking about rough sets as well, because I'm curious about them and how they relate to fuzzy sets. If at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &lt;a href=&quot;http://ijcai.org/papers11/Papers/IJCAI11-298.pdf&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is the paper I'm referencing.&lt;/p&gt;&#xA;" OwnerUserId="4804" LastEditorUserId="84" LastEditDate="2014-12-13T21:30:55.703" LastActivityDate="2014-12-13T21:30:55.703" Title="Rough vs Fuzzy vs Granular Computing" Tags="|machine-learning|data-mining|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2350" PostTypeId="2" ParentId="2349" CreationDate="2014-10-26T19:36:07.113" Score="4" Body="&lt;p&gt;&quot;&lt;strong&gt;Granularity&lt;/strong&gt;&quot; refers to the &lt;strong&gt;&lt;em&gt;resolution&lt;/em&gt;&lt;/strong&gt; of the variables under analysis.  If you are analyzing &lt;em&gt;height&lt;/em&gt; of people, you could use &lt;em&gt;course-grained variables&lt;/em&gt; that have only a few possible values -- e.g. &quot;above-average, average, below-average&quot; -- or a &lt;em&gt;fine-grained variable&lt;/em&gt;, with many or an infinite number of values -- e.g. integer values or real number values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A measure is &quot;&lt;strong&gt;fuzzy&lt;/strong&gt;&quot; if the distinction between alternative values is not crisp.  In the course-grained variable for &lt;em&gt;height&lt;/em&gt;, a &quot;crisp&quot; measure would mean that any given individual could &lt;em&gt;only&lt;/em&gt; be assigned one value -- e.g. a tall-ish person is either &quot;above-average&quot;, or &quot;average&quot;.  In contrast, a &quot;fuzzy&quot; measure allows for &lt;em&gt;degrees of membership&lt;/em&gt; for each value, with &quot;membership&quot; taking values from 0 to 1.0.  Thus, a tall-ish person could be a value of &quot;0.5 above-average&quot;, &quot;0.5 average&quot;, &quot;0.0 below-average&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a measure is &quot;&lt;strong&gt;rough&lt;/strong&gt;&quot; when two values are given: upper and lower bounds as an estimate of the &quot;crisp&quot; measure.  In our example of a tall-ish person, the rough measure would be {UPPER = above-average, LOWER = average}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why use granular, fuzzy, or rough measures at all, you might ask?  Why not measure everything in nice, precise real numbers?  Because many real-world phenomena don't have a good, reliable intrinsic measure and measurement procedure that results in a real number.  If you ask married couples to rate the quality of their marriage on a scale from 1 to 10, or 1.00 to 10.00, they might give you a number (or range of numbers), but how reliable are those reports? Using a course-grained measure (e.g. &quot;happy&quot;, &quot;neutral/mixed&quot;, &quot;unhappy&quot;), or fuzzy measure, or rough measure can be more reliable and more credible in your analysis.  Generally, it's much better to use rough/crude measures well than to use precise/fine-grained measures poorly.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2014-10-26T19:36:07.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2351" PostTypeId="2" ParentId="2298" CreationDate="2014-10-26T20:10:06.923" Score="1" Body="&lt;p&gt;I think the most likely explanation is that the two libraries do not quite support TreeModel in PMML in the same way. Perhaps one only supports a subset of features, and ignores ones it does not understand. This could cause different scoring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also double check that upstream parsing code is the same in both cases. Maybe a missing value is treated differently upstream.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-10-26T20:10:06.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2352" PostTypeId="1" CreationDate="2014-10-26T20:11:22.317" Score="2" ViewCount="48" Body="&lt;p&gt;I have data collected from a computer simulation of football games which seem to have recurring patterns of the following form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;if madrid plays arsernal and the match ends under 3 goal, then on their next match against each others, madrid will win. if madrid happens to loose and then plays against chelsea next, they will win 90% of the time.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;how do I find such inferences from simulation generated data like this. There are other forms of hidden patterns that I believe exists in the dataset.&lt;/p&gt;&#xA;" OwnerUserId="4811" LastActivityDate="2022-05-26T00:03:40.363" Title="making logical inference from a simuation generated data" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2353" PostTypeId="1" CreationDate="2014-10-26T20:35:39.767" Score="4" ViewCount="386" Body="&lt;p&gt;I cant seem to figure out why I have a high percentage error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to get a perceptron between X1 and X2 which are Gaussian distributed data sets with distinct means and identical co-variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N=200;&#xA;C= [2 1; 1 2]; %Covariance&#xA;m1=[0 2];&#xA;m2=[1.5 0];%mean&#xA;X1 = mvnrnd(m1, C, N/2);&#xA;X2 = mvnrnd(m2, C, N/2);&#xA;&#xA;X = [X1; X2];&#xA;X = [X ones(N,1)]; %bias&#xA;y = [-1*ones(N/2,1); ones(N/2,1)]; %classification&#xA;&#xA;%Split data into training and test &#xA;ii = randperm(N);&#xA;Xtr = X(ii(1:N/2),:);&#xA;ytr = X(ii(1:N/2),:);&#xA;Xts = X(ii(N/2+1:N),:);&#xA;yts = y(ii(N/2+1:N),:);&#xA;Nts = N/2;&#xA;&#xA;w = randn(3,1);&#xA;eta = 0.001;&#xA;%learn from training set&#xA;for iter=1:500 &#xA;j = ceil(rand*N/2);&#xA;if( ytr(j)*Xtr(j,:)*w &amp;lt; 0)&#xA;w = w + eta*Xtr(j,:)'; &#xA;end&#xA;end&#xA;&#xA;%apply what you have learnt to test set&#xA;yhts = Xts * w;&#xA;disp([yts yhts])&#xA;PercentageError = 100*sum(yts .*yhts &amp;lt; 0)/Nts;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong and how can I address this challenge?&lt;/p&gt;&#xA;" OwnerUserId="4813" LastEditorUserId="25652" LastEditDate="2016-12-21T19:51:44.737" LastActivityDate="2016-12-21T19:51:44.737" Title="MATLAB Perceptron" Tags="|machine-learning|matlab|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2354" PostTypeId="2" ParentId="172" CreationDate="2014-10-27T04:15:00.747" Score="6" Body="&lt;p&gt;I think your best bet would be &lt;strong&gt;&lt;a href=&quot;http://pythonhosted.org/rosetta/&quot;&gt;rosetta&lt;/a&gt;&lt;/strong&gt;. I'm finding it extremely useful and easy. Check its &lt;a href=&quot;http://pythonhosted.org/rosetta/#module-rosetta.parallel.pandas_easy&quot;&gt;pandas methods&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get it by &lt;strong&gt;pip&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4815" LastActivityDate="2014-10-27T04:15:00.747" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2355" PostTypeId="1" AcceptedAnswerId="2371" CreationDate="2014-10-27T09:55:36.887" Score="3" ViewCount="3066" Body="&lt;p&gt;Sorry, if this topic is not connected directly to Data Science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to understand how the &lt;a href=&quot;http://graphlab.com/learn/gallery/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graphlab tool&lt;/a&gt; works. Firstly I want to execute the toy examples from the Gallery site. When I try to execute the example code, everything is OK except one command: I can not see the graphlab plot after &lt;code&gt;show()&lt;/code&gt;. The command &lt;code&gt;show()&lt;/code&gt; returns to me some kind of object in IPython and nothing in the IPython Notebook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the example code has the plot, which depends directly on the matplotlib module, I can produce the real plots and save it on my machine. Consequently, I suppose the main error depends on the graphlab (or object from its class). &#xA;If somebody already used this tool and rendered the plot, can he/she tell me, how I can execute the plots command?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [8]: import graphlab&#xA;&#xA;In [9]: from IPython.display import display&#xA;&#xA;        from IPython.display import Image&#xA;&#xA;        graphlab.canvas.set_target('ipynb')&#xA;&#xA;In [10]:import urllib&#xA;&#xA;        url = 'https://s3.amazonaws.com/GraphLab-Datasets/americanMovies   /freebase_performances.csv'&#xA;&#xA;        urllib.urlretrieve(url, filename='freebase_performances.csv')  # downloads an 8MB file to the working directory&#xA;&#xA;Out[10]: ('freebase_performances.csv', &amp;lt;httplib.HTTPMessage instance at 0x7f44e153cf38&amp;gt;)&#xA;&#xA;In [11]: data = graphlab.SFrame.read_csv('remote://freebase_performances.csv', column_type_hints={'year': int})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...&#xA;...&#xA;...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [15]:data.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No plot after this line&#xA;...&#xA;...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [19]:print data.show()&#xA;&#xA;&amp;lt;IPython.core.display.Javascript object at 0x7f44e14c0850&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The object of graphlab (?) after print command&lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="26596" LastEditDate="2016-11-29T10:53:36.083" LastActivityDate="2016-11-29T10:53:36.083" Title="How do I show plots when using Graphlab?" Tags="|python|graphs|ipython|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2356" PostTypeId="2" ParentId="2287" CreationDate="2014-10-27T15:43:00.067" Score="0" Body="&lt;p&gt;OK, so values at time t-1 predict values at time t. That makes sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First you should decide whether you think these values are independent or not. Do the x predict the y or z at all? And, do you think just the previous 1 value is predictive, or the previous n?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way you could model this as a simple regression problem. What technique is best really depends on what you expect the relationship to be, and what these variables are; I am not sure that's given here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if they're sensor values read fairly rapidly, and the sensor changes slowly, you'd expect some simple model like a moving average to do well. For other types of values this would not be predictive at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks like the Markov chain model, so you may look into that, but somehow I think it's over-general for what I think the problem is.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-10-27T15:43:00.067" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2357" PostTypeId="2" ParentId="2348" CreationDate="2014-10-27T16:13:21.233" Score="2" Body="&lt;p&gt;A confusion matrix is a cross tabulation of your predicted values against the true observed values, and (test) accuracy is emperical rate of correct predictions. So in this case you'll need to &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predict the 'play' attribute for your test set.  (Currently you don't have a method to predict your second test case, so for the sake of argument let's assume your model would predict yes for the sunny example. &lt;/li&gt;&#xA;&lt;li&gt;The following method of keeping track of your predictions is reffered to as a confusion matrix.  The top labels are prediced&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;pre&gt;&#xA;               Predicted&#xA;         +----------------+&#xA;         ¦     ¦ yes ¦ no ¦&#xA;Oserved  ¦ yes ¦ 1   ¦ 1  ¦&#xA;         ¦ no  ¦ 0   ¦ 0  ¦&#xA;         +----------------+&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here the first 1 is from your first test case and the second 1 is from the misclassified second test case.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Calculate accuracy,&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Accuracy = (# correct predictions)/(# total predictions) = 1 / 2 = .50.&lt;/p&gt;&#xA;" OwnerUserId="4724" LastEditorUserId="4724" LastEditDate="2014-10-27T16:23:02.663" LastActivityDate="2014-10-27T16:23:02.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2358" PostTypeId="2" ParentId="2348" CreationDate="2014-10-27T17:39:18.997" Score="0" Body="&lt;p&gt;It is classify test objects: &quot;In classification, let R be the set of generated rules and T the training data. The basic idea of the proposed method is to choose a set of high confidence rules in R to cover T. In classifying a test object, the first rule in the set of rules that matches the test object condition classifies it. This process ensures that only the highest ranked rules classify test objects.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose 1 test case is (overcast, cool, normal, TRUE). Look through the rules top to bottom and see if any of the conditions are matched. The first rule for example tests the outlook feature. The value doesn't match, so the rule isn't matched. Move on to the next rule. And so on. In this case, rule 2 matches the test case and the classification for the play variable is &quot;yes&quot;. The second test case is misclassified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3503" LastActivityDate="2014-10-27T17:39:18.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2359" PostTypeId="1" CreationDate="2014-10-27T22:17:31.497" Score="3" ViewCount="100" Body="&lt;p&gt;Our system allows an admin to manage a database of university courses. These courses have multiple fields, like the department, a title, and a description.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am adding the ability to add learning objectives to a course. To simplify the problem, let's say that learning objectives are just tags. Courses can have more than one learning objective associated with them. So a course like CHEM 101 might have &quot;chemistry&quot;, &quot;technology&quot;, &quot;science&quot;, and several others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I can reduce a course to a set of features, (using keywords/stemming/nlp, I suppose?), what kind of problem is this and what algorithm would you suggest? It seems very similar to a classification problem, but I want to provide a sorted list of suggestions with the most relevant at the top.&lt;/p&gt;&#xA;" OwnerUserId="4821" LastEditorUserId="381" LastEditDate="2014-10-29T08:12:37.850" LastActivityDate="2014-10-29T08:12:37.850" Title="Ranked tag recommendation for university courses" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2360" PostTypeId="2" ParentId="1240" CreationDate="2014-10-27T23:24:46.673" Score="5" Body="&lt;p&gt;For item-ratings type of data with the restriction that an item's rating should be between 1 and 10 after transformation, I would suggest using a simple re-scaling, such that the item's transformed rating $x_t$ is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x_t = 9\left(\frac{x_i - x_{min}}{x_{max} - x_{min}}\right) + 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x_{min}$ and $x_{max}$ are the minimum and maximum possible rating in the specific scale for the item, and $x_i$ is the item rating. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of the above scaling, the transformation applied is independent of the data. However, in the normalization, the transformation applied is dependent on the data (through mean and standard deviation), and might change as more data becomes available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Section 4.3 on page 30 of &lt;a href=&quot;http://www.dai-labor.de/fileadmin/files/publications/DiplomaThesisStephanSpiegel.pdf&quot; rel=&quot;nofollow&quot;&gt;this document&lt;/a&gt; shows other ways of normalizing in which your restriction (transforming to the same absolute scale) might not be preserved. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="84" LastEditDate="2014-12-13T21:46:55.380" LastActivityDate="2014-12-13T21:46:55.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2361" PostTypeId="2" ParentId="2359" CreationDate="2014-10-27T23:56:11.853" Score="1" Body="&lt;p&gt;This will most likely turn out to be multiple binary classification problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of building just one classifier, you would have to build as many classifiers as there are tags. Each classifier's task would be to predict whether a certain tag exists or not. A tag specific probability threshold can then be learnt by tuning over the precision-curve where course wise precision recall can be calculated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The percentile of the predicted probability for each tag can be used to obtain a ordered (sorted) list of tag suggestions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way of converting a multi-class problem to multiple binary classification problems is known as binary relevance. While it is very efficient, it assumes that the labels themselves are independent. However, picking the probability threshold for each tag in order to optimize the course wise precision recall curve can lead to modeling dependence in the tags. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-10-27T23:56:11.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2362" PostTypeId="1" CreationDate="2014-10-28T06:00:49.643" Score="5" ViewCount="3496" Body="&lt;p&gt;I'm training a NN with 8 features and 8000 training examples with a single output (0, 1) using the scipy.optimise CG algorithm and the results are somewhat inconsistent. The goal is to get the NN to be as 'precise' as possible (recall doesn't really matter too much) so I've set the threshold for y value quite high (0.75). Most of the time it gets a precision of around 80%, however sometimes it fails (using exactly the same parameters, lambda etc..) to generate any outputs which are above the 0.75 threshold, meaning the precision equals 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've successfully trained NNs before without these unusual results (albeit the goal was a somewhat more conventional multi-class classifier with many more features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if the training NNs with fewer features increases the chances of it getting stuck at a local optima; or getting stuck at local optima has a more significant impact on NNs with fewer features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on what's going on!?&lt;/p&gt;&#xA;" OwnerUserId="4824" LastActivityDate="2014-12-28T06:14:08.787" Title="Neural Networks getting stuck at local optima" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2363" PostTypeId="2" ParentId="2298" CreationDate="2014-10-28T06:27:40.090" Score="3" Body="&lt;p&gt;The difference was, it appears, due to the different implementation of Random Forests in &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;Cascading Pattern&lt;/code&gt; (as well as &lt;code&gt;openscoring&lt;/code&gt; which I tried later) with respect to ties in the tree voting - i.e. when an even number of trees are built (say, 500) and exactly half classify an application as &lt;code&gt;Good&lt;/code&gt;, and the other half as &lt;code&gt;Bad&lt;/code&gt;, the handling of those situations differs. Solved it by growing and odd (501) number of trees.&lt;/p&gt;&#xA;" OwnerUserId="1127" LastActivityDate="2014-10-28T06:27:40.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2364" PostTypeId="2" ParentId="985" CreationDate="2014-10-28T07:00:01.593" Score="0" Body="&lt;p&gt;Using unsupervised learning to reduce the dimensionality and then using supervised learning to obtain an accurate predictive model is commonly used. See for example &lt;a href=&quot;http://faculty.ucmerced.edu/hbhat/BhatZaelit2012.pdf&quot; rel=&quot;nofollow&quot;&gt;Bhat and Zaelit, 2012&lt;/a&gt; where they first use PCA to reduce the dimension of a problem from 87 to 35. Then, they use L1 regression to obtain the best predictive model. This method beats non-linear tree based models built on the entire dataset and also its subset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your goal is to create more accurate classification of data into clusters, then a commonly used technique is to use supervised learning as a method to accurately pick the number of clusters &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume14/pan13a/pan13a.pdf&quot; rel=&quot;nofollow&quot;&gt;see Pan et al, 2013&lt;/a&gt; for a recent example. The basic approach here is to choose the number of clusters such that a supervised multi-class method can learn these clusters and predict the clusters with the highest out of sample accuracy. This is one way to convince yourself that the clusters are both meaningful and predictable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach, if your goal is to classify documents as being from US/ USA/ Australia or for that matter discussing, soccer/ American football/ Australian football could be to solve three binary classification problems that independently predict if the document talks about soccer, American football or Australian football. Combining the results from these three classifiers (known as binary relevance), you could also have the ability of tagging a document as both soccer or American football or any combination of the above three tags. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-10-28T07:00:01.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2365" PostTypeId="2" ParentId="2362" CreationDate="2014-10-28T07:14:06.873" Score="5" Body="&lt;p&gt;Ok all sorted - Bit embarrassing but forgot to normalise the data!&lt;/p&gt;&#xA;" OwnerUserId="4824" LastActivityDate="2014-10-28T07:14:06.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2366" PostTypeId="2" ParentId="2353" CreationDate="2014-10-28T17:07:32.723" Score="1" Body="&lt;p&gt;You are altering weights in the wrong direction for the negative cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The line&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = w + eta*Xtr(j,:)';&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = w + eta*Xtr(j,:)'*ytr(j);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With that change I got 12% error.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2014-10-28T17:07:32.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2367" PostTypeId="2" ParentId="492" CreationDate="2014-10-29T00:52:06.097" Score="6" Body="&lt;p&gt;Instead of &amp;quot;recursive neural nets with back propagation&amp;quot; you might consider the approach used by Frantzi, et. al. at National Centre for Text Mining (NaCTeM) at University of Manchester for &lt;em&gt;Termine&lt;/em&gt; (see: &lt;a href=&quot;http://www.nactem.ac.uk/index.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://personalpages.manchester.ac.uk/staff/sophia.ananiadou/IJODL2000.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;) Instead of deep neural nets, they &amp;quot;combine linguistic and statistical information&amp;quot;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="98307" LastEditDate="2020-08-05T08:41:02.810" LastActivityDate="2020-08-05T08:41:02.810" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2368" PostTypeId="1" AcceptedAnswerId="2370" CreationDate="2014-10-29T05:25:55.603" Score="63" ViewCount="67243" Body="&lt;p&gt;What are the common/best practices to handle time data for machine learning application?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if in data set there is a column with timestamp of event, such as &quot;2014-05-05&quot;, how you can extract useful features from this column if any?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2022-10-14T09:40:15.590" Title="Machine learning - features engineering from date/time data" Tags="|machine-learning|time-series|feature-selection|" AnswerCount="10" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2369" PostTypeId="2" ParentId="2368" CreationDate="2014-10-29T07:52:41.060" Score="7" Body="&lt;p&gt;In several cases, data and events inside a time series are seasonal. In such cases, the month and the year of the event matters a lot. Hence, in such scenarios you can use binary variables to represent if the event is during a given month/year or not.&lt;/p&gt;&#xA;&lt;p&gt;Hope this answers your question. If not, kindly be a little more specific on what exactly are you trying to achieve.&lt;/p&gt;&#xA;" OwnerUserId="4839" LastEditorUserId="25750" LastEditDate="2022-10-14T09:40:15.590" LastActivityDate="2022-10-14T09:40:15.590" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2370" PostTypeId="2" ParentId="2368" CreationDate="2014-10-29T13:54:23.793" Score="57" Body="&lt;p&gt;I would start by graphing the time variable vs other variables and looking for trends.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;For example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/QGYUC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case there is a periodic weekly trend and a long term upwards trend.  So you would want to encode two time variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;day_of_week&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;absolute_time&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;In general&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There are several common time frames that trends occur over:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;absolute_time&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;day_of_year&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;day_of_week&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;month_of_year&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;hour_of_day&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;minute_of_hour&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Look for trends in all of these.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Weird trends&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Look for weird trends too.  For example you may see rare but persistent time based trends:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;is_easter&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;is_superbowl&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;is_national_emergency&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;etc.&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These often require that you cross reference your data against some external source that maps events to time.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Why graph?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There are two reasons that I think graphing is so important.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Weird trends&lt;/em&gt;&lt;br&gt;&#xA;While the general trends can be automated pretty easily (just add them&#xA;every time), weird trends will often require a human eye and knowledge&#xA;of the world to find.  This is one reason that graphing is so&#xA;important.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Data errors&lt;/em&gt;&lt;br&gt;&#xA;All too often data has serious errors in it.  For example, you may find that the dates were encoded in two formats and only one of them has been correctly loaded into your program.  There are a myriad of such problems and they are surprisingly common.  This is the other reason I think graphing is important, not just for time series, but for any data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4808" LastEditorUserId="4808" LastEditDate="2014-11-05T22:02:14.857" LastActivityDate="2014-11-05T22:02:14.857" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2371" PostTypeId="2" ParentId="2355" CreationDate="2014-10-29T13:58:55.653" Score="3" Body="&lt;p&gt;I have found some solution and will post it here, because somebody, who works with graphlab, can have the same question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can look at the example here: &lt;a href=&quot;http://graphlab.com/learn/gallery/notebooks/graph_analytics_movies.html&quot; rel=&quot;nofollow&quot;&gt;Six degrees of Kevin Bacon&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At te beginning of the program execution you need to run next command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;graphlab.canvas.set_target('ipynb')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Exactly this is a key of the whole problem (at least by me:-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the beginning it is important to know, which parameter of &lt;code&gt;set_target()&lt;/code&gt; command you want to execute. You can use two options for argument of this command:  &lt;code&gt;'ipynb'&lt;/code&gt; (which is executed direct in iPython Notebook, like in example) or &lt;code&gt;'browser'&lt;/code&gt; (which open the new window with the plots)&#xA;On my machine 64-bit, Ubuntu, I can not use the command &lt;code&gt;'ipynb'&lt;/code&gt;. Only the &lt;code&gt;'browser'&lt;/code&gt;-command get me the plot back. I don't think, it is necessary to change https to http, but you can do it anyway. We have also the other machine by us (32-bit, Ubuntu) and it executes the other command &lt;code&gt;'ipynb'&lt;/code&gt;, but not &lt;code&gt;'browser'&lt;/code&gt; (without to change https to http)&lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="3281" LastEditDate="2014-10-30T07:57:03.453" LastActivityDate="2014-10-30T07:57:03.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2372" PostTypeId="2" ParentId="2298" CreationDate="2014-10-29T14:37:00.910" Score="-1" Body="&lt;p&gt;Decision Trees are unstable learners and very sensitive to changes in the input parameters. &lt;/p&gt;&#xA;" OwnerUserId="4847" LastActivityDate="2014-10-29T14:37:00.910" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2373" PostTypeId="1" AcceptedAnswerId="2376" CreationDate="2014-10-29T21:57:15.603" Score="4" ViewCount="316" Body="&lt;p&gt;The most online tutorials like to use a simple example to introduce to machine learning by classify unknown text in spam or not spam. They say that this is a binary-class problem. But why is this a binary-class problem? I think it is a one-class problem! I do only need positive samples of my inbox to learn what is not spam. If I do take a bunch of not spam textes as positiv samples and a bunch of spam-mails as negativ samples, then of course it's possible to train a binary-classifier and make predictions from unlabeled data, but where is the difference to the onc-class-approach? There I would just define a training-set of all non spam examples and train some one-class classifier. What do you think?&lt;/p&gt;&#xA;" OwnerUserId="4717" LastActivityDate="2014-10-30T14:48:34.907" Title="Detecting Spam using Machine Learning" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2374" PostTypeId="2" ParentId="2313" CreationDate="2014-10-29T22:10:53.570" Score="0" Body="&lt;p&gt;Your training data needs to be one set of data with samples of all the categories, because you are trying (I think) to create a model that will be fed such data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you given any thoughts as to what model(s) you might be using? I'm asking because pure classification models will achieve a better fit if the amount of data in each class is pretty uniform in the training data. However, regression models need the data type proportions to match the expected input.&lt;/p&gt;&#xA;" OwnerUserId="1241" LastActivityDate="2014-10-29T22:10:53.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2375" PostTypeId="2" ParentId="2373" CreationDate="2014-10-30T07:31:03.903" Score="2" Body="&lt;p&gt;The problem arises if you want to classify a new example as either spam or not spam. A one-class method will only give you some score of how well a new instance fits to this class, but how do you turn this into a binary prediction without knowing how big the score would be for the other class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the Naive Bayes classifier it essentially trains a &quot;one-class&quot; model for each class and arrives at a prediction by choosing the class with the highest score. But this requires you to have training examples for all classes.&lt;/p&gt;&#xA;" OwnerUserId="4852" LastActivityDate="2014-10-30T07:31:03.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2376" PostTypeId="2" ParentId="2373" CreationDate="2014-10-30T14:48:34.907" Score="6" Body="&lt;p&gt;Strictly speaking, &quot;one class classification&quot; does not make sense as an idea. If there is only one possible state of a predicted value, then there is no prediction problem. The answer is always the single class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concretely, if you only have spam examples, you would always achieve 100% accuracy by classifying all email as spam. This is clearly wrong, and the only way to know how it is wrong is to know where the classification is wrong -- where emails are not in the spam class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So-called &lt;a href=&quot;http://en.wikipedia.org/wiki/One-class_classification&quot;&gt;one-class classification&lt;/a&gt; techniques are really anomaly detection approaches. They have an implicit assumption that things unlike the examples are not part of the single class, but, this is just an assumption about data being probably &lt;em&gt;not&lt;/em&gt; within the class. There's a binary classification problem lurking in there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is wrong with a binary classifier?&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-10-30T14:48:34.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2377" PostTypeId="2" ParentId="1135" CreationDate="2014-10-30T14:59:13.243" Score="1" Body="&lt;p&gt;Basically your machine-learning problem is:&#xA;given [day-of-week, weather, departure-time, route], predict arrival time, then obtain travel time. Once your model is solid, you can then predict your travel time for each potential departure time and route, and choose the lowest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't really need to factor in the pit stops, if you think they're a function of the rest, or a random variable altogether.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a tool that can do the machine-learning aspect 'out of the box' you should try Weka.&#xA;You'll have to encode the data in the specific format it expects, but other than that you won't have to do any coding (i.e. you won't need to code any of the actual machine-learning algorithms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would discretize your departure time to make sure you have enough data.&#xA;Weka will let you try out the different algorithms, see which one is best.&#xA;Note that it's a regression problem as opposed to a classification problem, so only a subset of the algorithms apply. &#xA;Once you've got the Weka part figured out, you can also call it programmatically, which will allow you to code the interface you want, and potentially include the automatic weather retrieval. &lt;/p&gt;&#xA;" OwnerUserId="4760" LastActivityDate="2014-10-30T14:59:13.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2378" PostTypeId="1" CreationDate="2014-10-30T18:22:18.907" Score="8" ViewCount="1001" Body="&lt;p&gt;I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. When we are doing, for example, linear regression, we need to calculate some of those properties, and fit them into the equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, my question is about the intuition behind these matrix properties, and their implications in the ML/DM literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone could answer, can you teach me what is the importance of eigenvalue, positive semi-definite matrix, and column independence for ML/DM. And possibly, other important matrix properties you think important in study the dataset, and why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be really appreciated if someone can answer this question.&lt;/p&gt;&#xA;" OwnerUserId="4859" LastActivityDate="2014-10-31T04:44:23.100" Title="Matrix properties and machine learning/data mining" Tags="|machine-learning|data-mining|dataset|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2379" PostTypeId="2" ParentId="2378" CreationDate="2014-10-31T01:38:57.483" Score="2" Body="&lt;p&gt;A few things where the knowledge of Linear Algebra might be helpful in the context of Machine Learning: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dimensionality Reduction: There are lots of problems where PCA (a special case of an SVD) followed by a simple Machine Learning method applied on the reduced dataset produces much better results than a non parametric model on the full (non-reduced dataset). For an example see &lt;a href=&quot;http://faculty.ucmerced.edu/hbhat/BhatZaelit2012.pdf&quot; rel=&quot;nofollow&quot;&gt;Bhat and Zaelit, 2012&lt;/a&gt; where PCA followed by linear regression performs better than more involved non-parametric models. It also suggests reasons why dimensionality reduction performs better in these cases.  &lt;/li&gt;&#xA;&lt;li&gt;Visualizing data: Higher dimensional data is complicated to visualize and one often needs to be able to reduce the dimension of the dataset to view it. This comes very handy when one has to &quot;view&quot; the results of clustering on a higher dimensional dataset.  &lt;/li&gt;&#xA;&lt;li&gt;Numerical Accuracy: Eigenvalues are generally handy in order to understand condition numbers of matrices and hence be able to figure out if results of Linear regression or other methods that require to solve Ax=b would be numerically accurate. Positive Definiteness of a matrix might also be able to guarantee bounds on numerical accuracies. &lt;/li&gt;&#xA;&lt;li&gt;Recommendations: Some methods like collaborative filtering use matrix factorization (SVD) tuned smartly to solve recommendation problems.&lt;/li&gt;&#xA;&lt;li&gt;Regularization: Regularization is commonly used to reduce over-fitting in machine learning problems. Most of these regularization techniques like Lasso, Tikhonov etc. have both optimization and Linear Algebra at their heart. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-10-31T01:38:57.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2380" PostTypeId="2" ParentId="2378" CreationDate="2014-10-31T04:44:23.100" Score="3" Body="&lt;p&gt;The importance of a concept in mathematics depends on the circumstances of its application. Sometimes, its importance relies on the fact that it allows you to carry on with what you are doing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you usually need column independence (independent variables between predictors) because multiple regression will behave badly with highly correlated variables. Even worst, when some of your columns (or rows) are dependent, your matrix is not invertible. Why? Because matrix inversion A^-1 involves the determinant 1/|A|, which is 0 when columns or rows are linearly dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eigenvalues is a common occurrence in calculations related to maximization/minimization in machine learning. Let's say you are interested in principal component analysis. A very important idea there is dimensional reduction (you have a dataset with many variables and want to reduce the number of variables without losing too much explanatory power.) One solution is to project your data onto a lower dimensional space (e.g. taking your data with 50 variables and reducing them to 5 variables.) Turns out a good projection to use is one that includes as much variation as possible and maximization of this variation results in the eigenvalue equation S u  = λ u. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other cases, you explicitly include the eigenvalue equation of some quantity of interest because in doing so, you're changing the coordinate system in which you represent the variables. Take the case of a (multivariate) Gaussian distribution in which the argument of the exponent is given by Δ = (x-μ)^T Σ (x-μ). If you consider the eigenvalue equation of Σ, the exponent can be written as Δ = y_1^2 / λ_1 + y_2^2 / λ_2 (in two dimensions) This is the equation of an ellipse only if λ_1 and λ_2 are positive. Therefore, you obtain the following graphical interpretation (Bishop, PRML, p.81):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/pNrLb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Positive semi-definite matrices are used as a matter of convenience. They are well-behaved and well-understood. For instance, their eigenvalues are non-negative, and if you remember the previous paragraph, the argument Δ required positive eigenvalues. By now, you can see why some concepts are very popular: You need them for your calculations or they need each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can recommend a couple of books:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0534998453&quot; rel=&quot;nofollow noreferrer&quot;&gt;Linear Algebra: A Modern Introduction&lt;/a&gt; by David Poole &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1584888326&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding Complex Datasets: Data Mining with Matrix Decompositions &lt;/a&gt; by David Skillicorn. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The second recommendation is more specialized and requires a decent understanding of the basics, but it is of great help to understand matrix decompositions.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2014-10-31T04:44:23.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2381" PostTypeId="1" CreationDate="2014-10-31T08:50:23.160" Score="3" ViewCount="115" Body="&lt;p&gt;I'm examining the activity of customers over the years which have about one event per year. This results is many short time-series for which I found the distributions (hit/miss over 4 years sorted by probability in the data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0000 : 0.31834&#xA;0001 : 0.17582&#xA;0010 : 0.13605&#xA;0100 : 0.13554&#xA;1000 : 0.12886&#xA;0011 : 0.01717&#xA;1100 : 0.01650&#xA;0110 : 0.01578&#xA;0101 : 0.01220&#xA;1010 : 0.01117&#xA;1001 : 0.00883&#xA;0111 : 0.00571&#xA;1110 : 0.00565&#xA;1111 : 0.00496&#xA;1101 : 0.00384&#xA;1011 : 0.00351&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Apparently a purely uncorrelated binomial model wouldn't do, but one can observe that if both, the number of 1's and 11's coincide, then the probabilities are approximately equal (apart from a small recency effect of 0001).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you see a way to approach such data to deduce a probabilistic model? Basically where I have only a few probability parameters which roughly explain this distribution?&lt;/p&gt;&#xA;" OwnerUserId="723" LastEditorUserId="723" LastEditDate="2014-10-31T10:02:13.603" LastActivityDate="2014-10-31T10:02:13.603" Title="How to find a model for a short discrete time-series?" Tags="|time-series|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2383" PostTypeId="2" ParentId="2343" CreationDate="2014-10-31T13:05:49.263" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Gather app reviews for single app and try to find out the most important features that you can mine for the app developer&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;try predicting the next word in a sentence based on previous words.Eg. Swiftkey does this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Get questions data from stack overflow and try to predict tags for every post on stack overflow.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3211" LastActivityDate="2014-10-31T13:05:49.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2384" PostTypeId="1" AcceptedAnswerId="2393" CreationDate="2014-10-31T17:55:25.723" Score="4" ViewCount="2292" Body="&lt;p&gt;I'm planing to write a classification program that is able to classify unknown text in around 10 different categories and if none of them fits it would be nice to know that. It is also possible that more then one category is right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My predefined categories are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c1 = &quot;politics&quot;&#xA;c2 = &quot;biology&quot;&#xA;c3 = &quot;food&quot;&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm thinking about the right approach in how to represent my training-data or what kind of classification is the right one. The first challenge is about finding the right features. If I only have text (250 words each) what method would you recommend to find the right features? My first approach is to remove all stop-words and use the POS-Tagger (&lt;a href=&quot;http://nlp.stanford.edu/software/tagger.shtml&quot; rel=&quot;nofollow&quot;&gt;Stanford NLP POS-Tagger&lt;/a&gt;) to find nouns, adjective etc. I count them an use all frequently appeared words as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. politics, I've around 2.000 text-entities. With the mentioned POS-Tagger I found:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;law:           841&#xA;capitalism:    412&#xA;president:     397&#xA;democracy:     1007&#xA;executive:     112&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would it be right to use only that as features? The trainings-set would then look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training set for politics:&#xA;feature law         numeric&#xA;feature capitalism  numeric&#xA;feature president   numeric&#xA;feature democracy   numeric&#xA;feature executive   numeric&#xA;class politics,all_others&#xA;&#xA;sample data:&#xA;politics,5,7,1,9,3&#xA;politics,14,4,6,7,9&#xA;politics,9,9,9,4,2,1&#xA;politics,5,8,0,7,6&#xA;...&#xA;all_others,0,2,4,1,0&#xA;all_others,0,0,1,1,1&#xA;all_others,7,4,0,0,0&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would this be a right approach for binary-classification? Or how would I define my sets? Or is multi-class classification the right approach? Then it would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training set for politics:&#xA;feature law         numeric&#xA;feature capitalism  numeric&#xA;feature president   numeric&#xA;feature democracy   numeric&#xA;feature executive   numeric&#xA;feature genetics    numeric&#xA;feature muscle      numeric&#xA;feature blood       numeric&#xA;feature burger      numeric&#xA;feature salad       numeric&#xA;feature cooking     numeric &#xA;class politics,biology,food&#xA;&#xA;sample data:&#xA;politics,5,7,1,9,3,0,0,2,1,0,1&#xA;politics,14,4,6,7,9,0,0,0,0,0,1&#xA;politics,9,9,9,4,2,1,1,1,1,0,3&#xA;politics,5,8,0,7,6,2,2,0,1,0,1&#xA;...&#xA;biology,0,2,4,1,0,4,19,5,0,2,2&#xA;biology,0,0,1,1,1,12,9,9,2,1,1&#xA;biology,7,4,0,0,0,10,10,3,0,0,7&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would you say?&lt;/p&gt;&#xA;" OwnerUserId="4717" LastActivityDate="2020-08-18T16:28:32.957" Title="Text-Classification-Problem, what is the right approach?" Tags="|machine-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2385" PostTypeId="2" ParentId="2343" CreationDate="2014-10-31T22:40:24.513" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://www.kaggle.com/competitions&quot; rel=&quot;nofollow&quot;&gt;Kaggle&lt;/a&gt; has a bunch of good practice datasets and basic tutorials.&lt;/p&gt;&#xA;" OwnerUserId="192" LastActivityDate="2014-10-31T22:40:24.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2386" PostTypeId="2" ParentId="2384" CreationDate="2014-11-01T10:07:09.843" Score="-1" Body="&lt;p&gt;The following great article by Sebastian Raschka on &lt;em&gt;Bayesian approach to text classification&lt;/em&gt; should be very helpful for your task. I also highly recommend his excellent &lt;strong&gt;&lt;a href=&quot;http://sebastianraschka.com/articles.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt;&lt;/strong&gt; on &lt;em&gt;data science&lt;/em&gt; topics, as an additional general reference.&lt;/p&gt;&#xA;&lt;p&gt;You may also check this &lt;em&gt;&lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/my-data-science-apprenticeship-project&quot; rel=&quot;nofollow noreferrer&quot;&gt;educational report&lt;/a&gt;&lt;/em&gt; on text classification. It might provide you with some additional ideas.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-18T16:28:32.957" LastActivityDate="2020-08-18T16:28:32.957" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2387" PostTypeId="2" ParentId="2343" CreationDate="2014-11-01T10:16:55.557" Score="3" Body="&lt;p&gt;Check &lt;strong&gt;my answer&lt;/strong&gt; on the related question here: &lt;a href=&quot;https://datascience.stackexchange.com/a/843/2452&quot;&gt;https://datascience.stackexchange.com/a/843/2452&lt;/a&gt;. However, given your situation, you might need to work on something simpler. For this, it might be a good idea to review &lt;strong&gt;recommended literature&lt;/strong&gt; in the &lt;em&gt;open source data science master's curriculum&lt;/em&gt;: &lt;a href=&quot;http://datasciencemasters.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://datasciencemasters.org&lt;/a&gt;. Many referenced there sources are free and easily available online. I'm pretty sure that many of these sources contain examples of simple projects that you could (re)implement, extend or otherwise use for your task.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-11-01T10:16:55.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2391" PostTypeId="1" CreationDate="2014-11-02T07:20:32.603" Score="4" ViewCount="1009" Body="&lt;p&gt;I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. Currently, I am using Nupic from numenta.org for my analysis and it doesn't seem to be effective. Is there any ML library which can detect anomalies in multiple parameters?&lt;/p&gt;&#xA;" OwnerUserId="4887" LastEditorUserId="21" LastEditDate="2014-11-03T08:27:42.083" LastActivityDate="2015-03-17T20:13:00.767" Title="Anomaly detection in multiple parameters" Tags="|machine-learning|bigdata|statistics|anomaly-detection|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2392" PostTypeId="1" CreationDate="2014-11-02T11:36:28.143" Score="2" ViewCount="141" Body="&lt;p&gt;I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error.&#xA;The machine has 16G RAM, so I don't think this should have happened. I tried with some logging where the error came and  found that the data got loaded completely into numpy arrays. And it was the algorithm itself that caused the error. Can someone help me with this&#xA;or sugggest more memory efficient algorithm for feature selection, or is subsampling the &#xA;data my only option. Are there some deterministic good subsampling techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;Relevant code piece:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n=8;&#xA;y=mydata[:,0];&#xA;X=mydata[:,[1,2,3,4,5,6,7,8]];&#xA;#print y;&#xA;#print X;&#xA;print &quot;here&quot;;&#xA;omp = OrthogonalMatchingPursuit(n_nonzero_coefs=5,copy_X = False, normalize=True);&#xA;omp.fit(X,y);&#xA;coef = omp.coef_;&#xA;print omp.coef_;&#xA;idx_r, = coef.nonzero();&#xA;for id in idx_r:&#xA;        print coef[id], vars[id],&quot;\n&quot;;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;File &quot;/usr/local/lib/python2.7/dist-packages/sklearn/base.py&quot;, line 324, in score&#xA;return r2_score(y, self.predict(X), sample_weight=sample_weight)&#xA;File &quot;/usr/local/lib/python2.7/dist-packages/sklearn/metrics/metrics.py&quot;, line 2332, in r2_score&#xA;numerator = (weight * (y_true - y_pred) ** 2).sum(dtype=np.float64)&#xA;MemoryError&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4889" LastEditorUserId="21" LastEditDate="2015-11-13T11:07:48.370" LastActivityDate="2024-02-02T07:04:55.590" Title="scikit-learn OMP mem error" Tags="|python|bigdata|feature-selection|scikit-learn|scalability|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2393" PostTypeId="2" ParentId="2384" CreationDate="2014-11-03T03:22:30.393" Score="3" Body="&lt;p&gt;I think perhaps the first thing to decide that will help clarify some of your other questions is whether you want to perform binary classification or multi-class classification. If you're interested in classifying each instance in your dataset into more than one class, then this brings up a set of new concerns regarding setting up your data set, the experiments you want to run, and how you plan to evaluate your classifier(s). My hunch is that you could formulate your task as a binary one where you train and test one classifier for each class you want to predict, and simply set up the data matrix so that there are two classes to predict - (1) the one you're interested in classifying and (2) everything else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, instead of your training set looking like this (where each row is a document and columns 1-3 contain features for that document, and the class column is the class to be predicted): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1           2           3           class&#xA;feature1    feature2    feature3    politics&#xA;feature1    feature2    feature3    law&#xA;feature1    feature2    feature3    president&#xA;feature1    feature2    feature3    politics&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it would look like the following in the case where you're interested in detecting the politics class against everything else:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1           2           3           class&#xA;feature1    feature2    feature3    politics&#xA;feature1    feature2    feature3    non-politics&#xA;feature1    feature2    feature3    non-politics&#xA;feature1    feature2    feature3    politics&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You would need to do this process for each class you're interested in predicting, and then train and test one classifier per class and evaluate each classifier according to your chosen metrics (usually accuracy, precision, or recall or some variation thereof).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as choosing features, this requires quite a bit of thinking. Features can be highly dependent on the type of text you're trying to classify, so be sure to explore your dataset and get a sense for how people are writing in each domain. Qualitative investigation isn't enough to decide once and for all what are good features, but it is a good way to get ideas. Also, look into &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;TF-IDF&lt;/a&gt; weighting of terms instead of just using their frequency within each instance of your dataset. This will help you pick up on (a) terms that are prevalent within a document (and possibly a target class) and (b) terms that distinguish a given document from other documents. I hope this helps a little.&lt;/p&gt;&#xA;" OwnerUserId="4897" LastActivityDate="2014-11-03T03:22:30.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2394" PostTypeId="1" AcceptedAnswerId="2397" CreationDate="2014-11-03T12:41:51.737" Score="0" ViewCount="207" Body="&lt;p&gt;In most data acquisition settings it is useful to tag your data with time and location. If I write the data to csv file, what are the best formats that I can use for this two variables if I want to create a heatmap on Google Maps? &lt;/p&gt;&#xA;" OwnerUserId="3106" LastEditorUserId="16284" LastEditDate="2016-04-22T17:04:08.310" LastActivityDate="2016-04-22T17:04:08.310" Title="Best format for recording time stamp and GPS" Tags="|data-formats|geospatial|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2395" PostTypeId="1" CreationDate="2014-11-03T16:51:47.467" Score="15" ViewCount="6359" Body="&lt;p&gt;I have a continuous variable, sampled over a period of a year at irregular intervals. Some days have more than one observation per hour, while other periods have nothing for days. This makes it particularly difficult to detect patterns in the time series, because some months (for instance October) are highly sampled, while others are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/7MEXt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what would be the best approach to model this time series?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I believe most time series analysis techniques (like ARMA) need a fixed frequency. I could aggregate the data, in order to have a constant sample or choose a sub-set of the data that is very detailed. With both options I would be missing some information from the original dataset, that could unveil distinct patterns.&lt;/li&gt;&#xA;&lt;li&gt;Instead of decomposing the series in cycles, I could feed the model&#xA;with the entire dataset and expect it to pick up the patterns. For&#xA;instance, I transformed the hour, weekday and month in categorical&#xA;variables and tried a multiple regression with good results (R2=0.71)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have the idea that machine learning techniques such as ANN can also pick these patterns from uneven time series, but I was wondering if anybody has tried that, and could provide me some advice about the best way of representing time patterns in a Neural network.&lt;/p&gt;&#xA;" OwnerUserId="3159" LastActivityDate="2020-07-11T01:35:27.383" Title="Modelling Unevenly Spaced Time Series" Tags="|neural-network|time-series|regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2397" PostTypeId="2" ParentId="2394" CreationDate="2014-11-03T22:10:13.263" Score="1" Body="&lt;p&gt;As Spacedman put it, &quot;best&quot; is pretty subjective. However, as we have found, a good format for time is &lt;a href=&quot;http://en.wikipedia.org/wiki/Unix_time&quot; rel=&quot;nofollow&quot;&gt;Unix time&lt;/a&gt; (aka POSIX time, aka Epoch time). Most databases support it and it is still pretty human readable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For location, we like &lt;a href=&quot;http://en.wikipedia.org/wiki/Decimal_degrees&quot; rel=&quot;nofollow&quot;&gt;decimal degrees&lt;/a&gt; as it is easy to read and stored and is compatible with Google Maps API. It's also easy to convert to other formats if needed.&lt;/p&gt;&#xA;" OwnerUserId="4907" LastActivityDate="2014-11-03T22:10:13.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2398" PostTypeId="1" AcceptedAnswerId="2401" CreationDate="2014-11-04T00:58:00.137" Score="3" ViewCount="433" Body="&lt;p&gt;I'd like to apply some of the more complex supervised machine learning techniques in python - deep learning, generalized addative models, proper implementation of regularization, other cool stuff I dont even know about, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations how I could find expert ML folks that would like to collaborate on projects?&lt;/p&gt;&#xA;" OwnerUserId="4910" LastActivityDate="2014-11-04T13:41:52.240" Title="Python Machine Learning Experts" Tags="|machine-learning|python|" AnswerCount="1" CommentCount="3" ClosedDate="2014-11-07T08:24:49.463" ContentLicense="CC BY-SA 3.0" />
  <row Id="2399" PostTypeId="2" ParentId="1091" CreationDate="2014-11-04T04:23:37.107" Score="1" Body="&lt;p&gt;Check out &lt;a href=&quot;http://lightsidelabs.com/what/research/&quot; rel=&quot;nofollow&quot;&gt;LightSide&lt;/a&gt; for a GUI introduction to text mining in general, and more specifically for generating features quickly. It was developed (and I believe is still being developed) by researchers at CMU and it's how I got hooked on text mining. There is quite a bit of functionality right out of the box, and you can quickly import CSV data into the application, extract features, and start running experiments. It also makes use of suites of algorithms from several other prominent, well-regarded open source toolkits like Weka and LibLinear so you know you're using something credible under the hood. That being said, both of these last mentioned toolkits are definitely worth checking out for added functionality, even if they have a bit of a steeper learning curve. Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="4897" LastActivityDate="2014-11-04T04:23:37.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2401" PostTypeId="2" ParentId="2398" CreationDate="2014-11-04T13:41:52.240" Score="3" Body="&lt;p&gt;You could try some competitions from &lt;a href=&quot;http://kaggle.com&quot; rel=&quot;nofollow&quot;&gt;kaggle&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science courses from Coursera, edX, etc also provide forums for discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linkedin or freelance sites could be other possibilities.&lt;/p&gt;&#xA;" OwnerUserId="3051" LastActivityDate="2014-11-04T13:41:52.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2403" PostTypeId="1" CreationDate="2014-11-05T09:24:58.003" Score="22" ViewCount="11983" Body="&lt;p&gt;I had a conversation with someone recently and mentioned my interest in data analysis and who I intended to learn the necessary skills and tools. They suggested to me that while it is great to learn the tools and build the skills there is little point in doing so unless i have specialized  knowledge in a specific field. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They basically summed it to that I'd just be like a builder with a pile of tools who could build a few wooden boxes and may be build better things (cabins, cupboards etc), but without knowledge in a specific field I'd never be a builder people would come to for a specific product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone found this or have any input on what to make of this ? It would seem if it was true one would have to learn the data science aspects of things and then learn  a new field just to  become specialized.&lt;/p&gt;&#xA;" OwnerUserId="4836" LastActivityDate="2021-05-28T09:43:34.023" Title="Data science without knowledge of a specific topic, is it worth pursuing as a career?" Tags="|education|beginner|career|" AnswerCount="2" CommentCount="3" ClosedDate="2014-11-08T23:12:05.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="2405" PostTypeId="2" ParentId="2269" CreationDate="2014-11-05T15:16:36.867" Score="1" Body="&lt;p&gt;Try out AirXcell : &lt;a href=&quot;http://airxcell.com&quot; rel=&quot;nofollow&quot;&gt;AirXcell calculation software&lt;/a&gt;.&#xA;See documentation &lt;a href=&quot;http://www.airxcell.com/doc/howTo/useConsole.html&quot; rel=&quot;nofollow&quot;&gt;Use AirXCell as an r Console&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4929" LastActivityDate="2014-11-05T15:16:36.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2406" PostTypeId="2" ParentId="2403" CreationDate="2014-11-05T17:03:40.513" Score="53" Body="&lt;p&gt;Drew Conway published the &lt;a href=&quot;http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram&quot; rel=&quot;noreferrer&quot;&gt;Data Science Venn Diagram&lt;/a&gt;, with which I heartily agree:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/7W0VM.png&quot; alt=&quot;Data Science Venn Diagram&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;On the one hand, you should really read his post. On the other hand, I can offer my own experience: my subject matter expertise (which I like better as a term than &amp;quot;Substantive Expertise&amp;quot;, because you should really also have &amp;quot;Substantive Expertise&amp;quot; in math/stats and hacking) is in the retail business, my math/stats are forecasting and inferential statistics, and my hacking skills lie in R.&lt;/p&gt;&#xA;&lt;p&gt;From this vantage point, I can talk to and understand retailers, and someone who doesn't have at least a passing knowledge of this field will have to face a &lt;em&gt;steep&lt;/em&gt; learning curve in a project with retailers. As a side gig, I do statistics in psychology, and it's exactly the same there. And even with quite some knowledge of the hacking/math/stats part of the diagram, I would have a hard time getting up to speed in, say, credit scoring or some other new subject field.&lt;/p&gt;&#xA;&lt;p&gt;Once you have a certain amount of math/stats and hacking skills, it is &lt;em&gt;much&lt;/em&gt; better to acquire a grounding in one or more subjects than in adding &lt;em&gt;yet&lt;/em&gt; another programming language to your hacking skills, or &lt;em&gt;yet&lt;/em&gt; another machine learning algorithm to your math/stats portfolio. After all, once you have a solid math/stats/hacking grounding, you could if need be learn such new tools from the web or from textbooks in a relative short time period. But the subject matter expertise, on the other hand, you will likely not be able to learn from scratch if you start from zero. And clients will rather work with some data scientist A who understands their specific field than with another data scientist B who first needs to learn the basics - even if B is better in math/stats/hacking.&lt;/p&gt;&#xA;&lt;p&gt;Of course, all this will also mean that you will never become an expert in &lt;em&gt;either&lt;/em&gt; of the three fields. But that's fine, because you are a data scientist, not a programmer or a statistician or a subject matter expert. There will always be people in the three separate circles who you can learn from. Which is part of what I like about data science.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;EDIT: A little while and a few thoughts later, I'd like to update this post with a new version of the diagram. I still think that Hacking Skills, Math &amp;amp; Statistics Knowledge and Substantive Expertise (shortened to &amp;quot;Programming&amp;quot;, &amp;quot;Statistics&amp;quot; and &amp;quot;Business&amp;quot; for legibility) are important... but I think that the role of &lt;em&gt;Communication&lt;/em&gt; is important, too. All the insights you derive by leveraging your hacking, stats and business expertise won't make a bit of a difference unless you can communicate them to people who may &lt;em&gt;not&lt;/em&gt; have that unique blend of knowledge. You may need to explain your statistical insights to a business manager who needs to be convinced to spend money or change processes. Or to a programmer who doesn't think statistically. &lt;a href=&quot;https://doi.org/10.1111/1740-9713.01531&quot; rel=&quot;noreferrer&quot;&gt;Caulcutt (2021, &lt;em&gt;Significance&lt;/em&gt;)&lt;/a&gt; is a short article that says much the same, but gives more detail than I do.&lt;/p&gt;&#xA;&lt;p&gt;So here is the new data science Venn diagram, which also includes communication as one indispensable ingredient. I have labeled the areas in ways that should guarantee maximum flaming, while being easy to remember.&lt;/p&gt;&#xA;&lt;p&gt;Comment away.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aiQeT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aiQeT.png&quot; alt=&quot;new data science Venn diagram&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;R code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;draw.ellipse &amp;lt;- function(center,angle,semimajor,semiminor,radius,h,s,v,...) {&#xA;    shape &amp;lt;- rbind(c(cos(angle),-sin(angle)),c(sin(angle),cos(angle))) %*% diag(c(semimajor,semiminor))&#xA;    tt &amp;lt;- seq(0,2*pi,length.out=1000)&#xA;    foo &amp;lt;- matrix(center,nrow=2,ncol=length(tt),byrow=FALSE) + shape%*%(radius*rbind(cos(tt),sin(tt)))&#xA;    polygon(foo[1,],foo[2,],col=hsv(h,s,v,alpha=0.5),border=&amp;quot;black&amp;quot;,...)&#xA;}&#xA;name &amp;lt;- function(x,y,label,cex=1.2,...) text(x,y,label,cex=cex,...)&#xA;&#xA;png(&amp;quot;Venn.png&amp;quot;,width=600,height=600)&#xA;    opar &amp;lt;- par(mai=c(0,0,0,0),lwd=3,font=2)&#xA;        plot(c(0,100),c(0,90),type=&amp;quot;n&amp;quot;,bty=&amp;quot;n&amp;quot;,xaxt=&amp;quot;n&amp;quot;,yaxt=&amp;quot;n&amp;quot;,xlab=&amp;quot;&amp;quot;,ylab=&amp;quot;&amp;quot;)&#xA;        draw.ellipse(center=c(30,30),angle=0.75*pi,semimajor=2,semiminor=1,radius=20,h=60/360,s=.068,v=.976)&#xA;        draw.ellipse(center=c(70,30),angle=0.25*pi,semimajor=2,semiminor=1,radius=20,h=83/360,s=.482,v=.894)&#xA;        draw.ellipse(center=c(48,40),angle=0.7*pi,semimajor=2,semiminor=1,radius=20,h=174/360,s=.397,v=.8)&#xA;        draw.ellipse(center=c(52,40),angle=0.3*pi,semimajor=2,semiminor=1,radius=20,h=200/360,s=.774,v=.745)&#xA;        &#xA;        name(50,90,&amp;quot;The Data Scientist Venn Diagram&amp;quot;,pos=1,cex=2)&#xA;        name(8,62,&amp;quot;Communi-\ncation&amp;quot;,cex=1.5,pos=3)&#xA;        name(30,78,&amp;quot;Statistics&amp;quot;,cex=1.5)&#xA;        name(70,78,&amp;quot;Programming&amp;quot;,cex=1.5)&#xA;        name(92,62,&amp;quot;Business&amp;quot;,cex=1.5,pos=3)&#xA;        &#xA;        name(10,45,&amp;quot;Hot\nAir&amp;quot;)&#xA;        name(90,45,&amp;quot;The\nAccountant&amp;quot;)&#xA;        name(33,65,&amp;quot;The\nData\nNerd&amp;quot;)&#xA;        name(67,65,&amp;quot;The\nHacker&amp;quot;)&#xA;        name(27,50,&amp;quot;The\nStats\nProf&amp;quot;)&#xA;        name(73,50,&amp;quot;The\nIT\nGuy&amp;quot;)&#xA;        name(50,55,&amp;quot;R\nCore\nTeam&amp;quot;)&#xA;        name(38,38,&amp;quot;The\nGood\nConsultant&amp;quot;)&#xA;        name(62,38,&amp;quot;Drew\nConway's\nData\nScientist&amp;quot;)&#xA;        name(50,24,&amp;quot;The\nperfect\nData\nScientist!&amp;quot;)&#xA;        name(31,18,&amp;quot;Comp\nSci\nProf&amp;quot;)&#xA;        name(69,18,&amp;quot;The\nNumber\nCruncher&amp;quot;)&#xA;        name(42,11,&amp;quot;Head\nof IT&amp;quot;)&#xA;        name(58,11,&amp;quot;Ana-\nlyst&amp;quot;)&#xA;        name(50,5,&amp;quot;The\nSalesperson&amp;quot;)&#xA;    par(opar)&#xA;dev.off()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2853" LastEditorUserId="2853" LastEditDate="2021-05-28T09:43:34.023" LastActivityDate="2021-05-28T09:43:34.023" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2407" PostTypeId="2" ParentId="1091" CreationDate="2014-11-05T17:07:02.563" Score="1" Body="&lt;p&gt;Another library that I use that has not been mentioned yet is gensim.  Its &lt;a href=&quot;http://radimrehurek.com/gensim/corpora/dictionary.html&quot; rel=&quot;nofollow&quot;&gt;Dictionary&lt;/a&gt; module allows you to convert a list of words into a list of (id,count) pairs.  It also has an &lt;code&gt;allow_update&lt;/code&gt; variable that updates the dictionary in the case that new words are encountered at runtime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also has built-in support for TF-IDF, LSI, and LDA, among other models.&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2014-11-05T17:07:02.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2410" PostTypeId="2" ParentId="2395" CreationDate="2014-11-05T17:22:00.213" Score="4" Body="&lt;p&gt;ARIMA, Exponential Smoothing and others indeed require evenly spaced sample points. As you write, you could bucketize your data (say into days), but as you also write, you would lose information. In addition, you may end up with missing values, so you would need to impute, since ARIMA is not very good at handling missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One alternative, as you again write, is to feed time dummies into a regression framework. I personally do not really like categorical dummies, because this implies a sharp cutoff between neighboring categories. This is usually not very natural. So I would rather look at periodic splines with different periodicities. This approach has the advantage of dealing with your uneven sampling and also with missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be &lt;em&gt;very&lt;/em&gt; careful about interpreting $R^2$. In-sample fit is notoriously misleading as a measure of out-of-sample forecast accuracy (&lt;a href=&quot;https://www.otexts.org/fpp/2/5&quot; rel=&quot;nofollow&quot;&gt;see here&lt;/a&gt;). I would argue that this disconnect between in-sample fit and out-of-sample forecast accuracy also means that there is no connection between in-sample fit and how well a model &quot;understood&quot; the data, even if your interest lies not in forecasting, but only in modeling per se. My philosophy is that if you can't forecast a time series well, you haven't understood it in any meaningful sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, don't overdo the modeling. Just from eyeballing your data, it is obvious that &lt;em&gt;something&lt;/em&gt; happened in June, on one day in August and in September/October. I suggest you first find out what this &lt;em&gt;something&lt;/em&gt; was and include this in your model, e.g., as explanatory variables (which you can include in ARIMAX if you want to). What happened there is obviously not seasonality.&lt;/p&gt;&#xA;" OwnerUserId="2853" LastActivityDate="2014-11-05T17:22:00.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2411" PostTypeId="2" ParentId="2403" CreationDate="2014-11-05T17:50:39.573" Score="2" Body="&lt;p&gt;Sure, you can. Companies are clamoring for data scientists. Be careful though that they all interpret the term differently. Depending on the company you might find yourself asked to do anything from statistics to writing production code. Either one is a full-time job in itself and you have to be prepared for both, so asking for deep specialized knowledge on top of that's not reasonable, in my opinion, and the companies I've talked to stressed the other two areas (esp. the programming). However, I found that it helps to be familiar with the types of problems that you might face. Depending on the sector, that could be anomaly detection, recommendation/personalization, prediction, record linkage, etc. These are things you can learn as examples at the same times as maths and programming.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-11-05T17:50:39.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2412" PostTypeId="1" CreationDate="2014-11-05T18:57:55.037" Score="2" ViewCount="238" Body="&lt;h1&gt;My Background&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I am a graduate student in Civil Engineering. For the analyses of road traffic data (vehicle trajectories as time series) I work with big data sets mostly about a million data points or more.&lt;br&gt;&#xA;I started using R language when MS Excel could not open the big data files. Using basic statistics knowledge and R code I developed few algorithms to identify certain patterns in the data which worked for many applications. But I still lack serious programming skills in R.&lt;br&gt;&#xA;Now, I am familiar with basic inferential statistics and R packages (plyr, dplyr, ggplot2, etc). Recently I came to know that Machine Learning algorithms also help in defining patterns in the data through supervised/ unsupervised learning and their application might improve the accuracy of prediction of certain 'behaviors' of drivers using the traffic data.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Having the basic knowledge of Statistics and R, I want to learn about the data science/ machine learning as a beginner. I know that some concepts in Stats. and ML overlap and that might bridge the gap in my learning of ML. Keeping my background in mind, what resources (books/ online courses) would you recommend me to start learning data science and apply it in my field?&lt;/p&gt;&#xA;" OwnerUserId="4933" LastEditorUserId="21" LastEditDate="2017-05-19T16:13:40.213" LastActivityDate="2017-05-19T16:13:40.213" Title="Learning resources for Data Science for applications in road traffic data?" Tags="|self-study|" AnswerCount="1" CommentCount="1" ClosedDate="2017-05-19T16:13:44.630" ContentLicense="CC BY-SA 3.0" />
  <row Id="2413" PostTypeId="2" ParentId="2391" CreationDate="2014-11-05T21:22:09.193" Score="2" Body="&lt;p&gt;One way to use both metric1 and metric2 in order to find anomalies in metric3 is to consider residual analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, this would require, creating a predictive model with metric1 and metric2 as the predictors and metric3 as the response variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, calculate the residuals for metric3 as its predicted value subtracted from its true value. Now, you can report the all members of the lowest decile [or any other percentile] as one kind of an anomaly and all the members of the highest decile [or any other percentile]   as another kind of an anomaly. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-05T21:22:09.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2414" PostTypeId="2" ParentId="2412" CreationDate="2014-11-05T21:55:43.587" Score="1" Body="&lt;p&gt;The best way to learn data science is through problem solving. I suggest you to head over to Kaggle and work through the for-knowledge problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a good start on Machine Learning problems, acclimate yourself with the tree package in R. This will help you understand how decision trees work, and building upon that, how random forests, gradient boosting machines and other sophisticated tree based algorithms work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, there are SVMs and deep learning models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an understanding of unsupervised learning problems, learn k-means and employ it for clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other general concepts/ ideas to understand are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;cross-validation &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;overfitting, regularization &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;bias-variance trade off&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;dimensionality reduction/ variable selection&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;generalization error&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ensemble learning &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For books, the most common recommendation to anyone who is familiar with statistics and wants to get into Machine Learning is &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;&quot;The Elements of Statistical Learning&quot;&lt;/a&gt; by Hastie, Tibshirani, and Friedman.  &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-05T21:55:43.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2416" PostTypeId="1" CreationDate="2014-11-06T07:38:33.490" Score="6" ViewCount="806" Body="&lt;p&gt;I would like to ask your opinion on how to choose a similarity measure. I have a set of vectors of length N, each element of which can contain either 0 or 1. The vectors are actually ordered sequences, so the position of each element is important. Suppose I have three vectors of length 10, x_1 x2, x3: x1 has three 1 at positions 6,7,8 (indexes start from 1. Both x2 and x3 have an additional 1, but x2 has it in position 9 while x3 has it in position 1. I am looking for a metric according to which x1 is more similar to x2 than to x3, in that the additional 1 is closer to the &quot;bulk&quot; of ones. I guess this is a relatively common problem, but I am confused on the best way to approach it.&#xA;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="1004" LastActivityDate="2014-11-07T09:04:17.603" Title="Similarity measure for ordered binary vectors" Tags="|similarity|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2417" PostTypeId="2" ParentId="2416" CreationDate="2014-11-06T08:52:22.617" Score="3" Body="&lt;p&gt;If speed isn't a great concern you could use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;KDE&lt;/a&gt; with a high bandwidth to pick up the similarity between neighboring elements, then an appropriate metric like the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;K-L divergence&lt;/a&gt;. Similarity and divergence are complementary, of course, so as a final step you would have to relate them; e.g., &lt;code&gt;sim(A, B) = exp[- KLD(A, B)]&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-11-06T08:52:22.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2418" PostTypeId="2" ParentId="2303" CreationDate="2014-11-06T13:56:02.497" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikibooks.org/wiki/R_Programming&quot; rel=&quot;nofollow&quot;&gt;R Programming Wikibook&lt;/a&gt; is a nice collaborative handbook for R.&lt;/p&gt;&#xA;" OwnerUserId="187" LastActivityDate="2014-11-06T13:56:02.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2419" PostTypeId="2" ParentId="2416" CreationDate="2014-11-06T15:19:35.017" Score="4" Body="&lt;p&gt;One thing you could do is fuzzify your vectors: replace each 1 by (for example) 0.4 in its position, 0.2 in the neighbouring positions, and 0.1 in the second position over. Then add up what's in each position. &#xA;With these fuzzified vectors, you can apply a similarity metric either based on a distance or one like cosines similarity. &#xA;Your example would produce: (showing only first decimal)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0000011100 -&gt; 0001378731  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0000011110 -&gt; 0001378873&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000011100 -&gt; 4211378731&lt;/p&gt;&#xA;&#xA;&lt;p&gt;cos(x1, x2) = 0.9613,  cos(x1,x3) = 0.9469&lt;/p&gt;&#xA;" OwnerUserId="4760" LastActivityDate="2014-11-06T15:19:35.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2420" PostTypeId="2" ParentId="1216" CreationDate="2014-11-06T17:56:06.257" Score="0" Body="&lt;p&gt;Keep in mind that &quot;big data&quot; is an increasingly trendy thing for a company to say they're involved in.  Higher ups might read an article about it in HBR, and say to themselves, &quot;I've got to get me some of that&quot; (not that they're necessarily wrong). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this means for you is that the advanced analytics isn't as necessary for that company as just getting something up and running might be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Luckily for you, most of the components said companies might need are free.  Moreover, I believe both Hortonworks and Cloudera have free &quot;sandbox&quot; virtual machines, which you can run on your PC, to play around with and get your bearings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advanced analytics on big data platforms are valuable, to be sure, but many companies need to learn to crawl before they can run. &lt;/p&gt;&#xA;" OwnerUserId="4943" LastActivityDate="2014-11-06T17:56:06.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2421" PostTypeId="2" ParentId="1216" CreationDate="2014-11-06T21:36:51.897" Score="0" Body="&lt;p&gt;This is a really strange question in my opinion. Why you're going to move in a new direction if you are not sure that you love this new direction or at least find it very interesting? If you do love Big Data, why do you care about the PhD intelligent creatures that are already in the field? The same amount of PhD creatures are in every area of IT. Please have a quick read at this very nice article &lt;a href=&quot;http://www.forbes.com/sites/louisefron/2013/09/13/why-you-cant-find-a-job-you-love/&quot; rel=&quot;nofollow&quot;&gt;http://www.forbes.com/sites/louisefron/2013/09/13/why-you-cant-find-a-job-you-love/&lt;/a&gt; and then ask yourself if you love Big Data enough and you are ready to add your grain of sand to the mountain of knowledge&lt;/p&gt;&#xA;" OwnerUserId="4944" LastActivityDate="2014-11-06T21:36:51.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2422" PostTypeId="1" AcceptedAnswerId="2423" CreationDate="2014-11-07T04:13:44.583" Score="3" ViewCount="2547" Body="&lt;p&gt;I'm just starting to work on a relatively large dataset after ML course in Coursera.&#xA;Trying to work on &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD&quot; rel=&quot;nofollow&quot;&gt;https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD&lt;/a&gt;.&#xA;Got an accuracy of 5.2 in training and test set with linear regression using gradient descent in octave.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried adding all possible quadratic features (515345 instances and 4275 features), but the code just won't stop executing in my HP Pavilion g6 2320tx, with 4GB RAM in Ubuntu 14.04.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this beyond the data size capacity of Octave ?&lt;/p&gt;&#xA;" OwnerUserId="4947" LastEditorUserId="847" LastEditDate="2014-11-07T08:28:07.227" LastActivityDate="2014-11-07T08:28:07.227" Title="Clarification about Octave data size limit" Tags="|octave|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2423" PostTypeId="2" ParentId="2422" CreationDate="2014-11-07T05:53:25.600" Score="2" Body="&lt;p&gt;You have about 4GB of RAM on your machine and Octave is an in memory application. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to work with 515345 instances and 4275 features, assuming that you are using double precision (i.e. 8 bytes), you would need a memory of 515345*4275*8/1000000/1024 bytes ~ 17.6 GB. Even if you were using 4 bytes for each data point, you would require at least 9 GB for the computation to go through. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This issue might not be the Octave memory restriction in this case. See &lt;a href=&quot;https://stackoverflow.com/questions/565806/is-anyone-using-64-bit-build-of-octave&quot;&gt;here&lt;/a&gt; for further details on Octave's memory usage. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-11-07T06:04:59.140" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2424" PostTypeId="2" ParentId="2303" CreationDate="2014-11-07T08:09:40.260" Score="0" Body="&lt;p&gt;I would recommend those materials:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1449319793&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python for Data Analysis&lt;/a&gt; - book which nicely covers Pandas workflow with IPython.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1491962291&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hands-On Machine Learning with Scikit-Learn and TensorFlow&lt;/a&gt; - slightly more advanced book about using Scikit-Learn and Tensor flow in data science projects&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;R&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1?utm_medium=listingPage&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera Data Science Specialisation&lt;/a&gt; - free nine one month time consuming courses which introduces R step by step from beginning to machine learning.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.datacamp.com/courses&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Camp courses&lt;/a&gt; - at least 4 free courses covering topics from data analysis.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="82" LastEditorUserId="82" LastEditDate="2017-07-10T06:48:41.050" LastActivityDate="2017-07-10T06:48:41.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2425" PostTypeId="2" ParentId="2416" CreationDate="2014-11-07T09:04:17.603" Score="1" Body="&lt;p&gt;Another possibility is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Earth_mover&amp;#39;s_distance&quot; rel=&quot;nofollow&quot;&gt;Earth Mover's Distance&lt;/a&gt;. I applied it to Computer Vision, but I think that it may be adapted to your specific problem. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2014-11-07T09:04:17.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2426" PostTypeId="1" CreationDate="2014-11-07T21:36:00.080" Score="3" ViewCount="89" Body="&lt;p&gt;I have a data set that is pivoted in to the following format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[key] [id] [0] [1] [5] [10] [15] [60] [120] [180],.. [365]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So key could be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Products] [1000] [15,000] [4000]... etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where products is the category of item being reviewed and key is the identifier for the product; the only fields (0, 1,... 180,.. [365]) are individual daily samples identify how many of &quot;x&quot; product were logged as either sold, in-stock etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need to do is perform some kind of analysis on an entire slew of products and their inventory levels. i.e. each import of data I need to make sure the incoming data is accurate or predictably accurate and that some human did not typo a stock level. The problem is, using a simple average or rolling average can introduce significant variance and smoothing out the average renders my analysis less reliable. Ideally this analysis would trigger an alarm that someone would have to investigate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better and more accurate way of performing this analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="4956" LastActivityDate="2014-12-08T20:57:17.583" Title="Validity of data" Tags="|machine-learning|statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2427" PostTypeId="1" CreationDate="2014-11-08T12:22:17.920" Score="7" ViewCount="302" Body="&lt;p&gt;The meaning of multi-class classification rules&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example: I have two classification rules (Refund is a predictor and Cheat is a binary response):&#xA;(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57&#xA;(Refund, No) → (Cheat, Yes) Support = 0.3, Confidence = 0.43&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;=&gt; multi-class classification rules:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(Refund, No) → (Cheat, No) v (Cheat, Yes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When predicted classification for test data, (Cheat, No) will be selected priority so why we need to have (Cheat, Yes) in multi-class classification rules here?&lt;/p&gt;&#xA;" OwnerUserId="3503" LastEditorUserId="847" LastEditDate="2014-11-08T22:52:49.717" LastActivityDate="2014-12-12T20:00:23.620" Title="The meaning of multi-class classification rules" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2428" PostTypeId="2" ParentId="2427" CreationDate="2014-11-08T18:33:37.063" Score="3" Body="&lt;pre&gt;&lt;code&gt;(Cheat, No) will be selected (cheat will be classified as No) for the rule (Refund, No). &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a binary response variable like cheat, all the information can be inferred from just the first rule: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The other rule is redundant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the case of a multi-class response variable, we would like to have all the rules written out so we exactly know the likelihood of the rule implying each of the different classes in the response variable. To keep things consistent, this is also done for the case when the response variable is binary. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-08T18:33:37.063" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2429" PostTypeId="2" ParentId="2426" CreationDate="2014-11-08T20:29:37.377" Score="1" Body="&lt;p&gt;The first thing you should do is to identify how large an error your analysis can handle. That will make your job much easier because you won't have to find everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard way of identifying &quot;suspicious&quot; data is is Benford's Law, which predicts the distribution of the first digit of each number. It can also be generalized for for other digits. &lt;a href=&quot;http://en.wikipedia.org/wiki/Benford&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Benford&lt;/a&gt;'s_law&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for finding outliers, I'd probably use boxplots, particularly because you can achieve high data density with them, reducing the time to manually skim them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that might be useful is to compare the ratio of one variable to another- in my company we use this method all the time.&lt;/p&gt;&#xA;" OwnerUserId="1241" LastActivityDate="2014-11-08T20:29:37.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2431" PostTypeId="2" ParentId="1216" CreationDate="2014-11-09T10:36:51.290" Score="1" Body="&lt;p&gt;In my experience to have a PhD doesn't mean necessarily be good in the enviroment of data science company, I work as data scientist and I'm just an engineer but I've known some universitary teachers who works in collaboration with my company and sometimes I've said them that Their point of view was not right because despite of their ideas and reasonings were right they are not applicables to the company activities, so we had to modify some data models to make them usefull for the company and the results lost their value so we had to seek new models. What I mean is that Data Science is a multidisciplinar area so many different people working together is needed so I think that your skills could be very useful in a data scientist team, you only have to find where you fit ;)&lt;/p&gt;&#xA;" OwnerUserId="4966" LastActivityDate="2014-11-09T10:36:51.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2432" PostTypeId="1" CreationDate="2014-11-09T14:11:18.350" Score="5" ViewCount="583" Body="&lt;p&gt;By &quot;large&quot;, I mean in the range of 100m to 10b rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the data is distributed well for the given query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there other technologies that I should be looking at here? If so, what are the trade offs?&lt;/p&gt;&#xA;" OwnerUserId="4967" LastActivityDate="2015-06-14T08:56:13.250" Title="What technologies are fastest at performing joins on large datasets?" Tags="|bigdata|performance|map-reduce|aws|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2433" PostTypeId="1" CreationDate="2014-11-10T00:10:26.947" Score="2" ViewCount="130" Body="&lt;p&gt;I was going through an IEEE research paper which has used Fuzzy ARTMAP for predicting the price of electricity given some highly correlated data.&lt;/p&gt;&#xA;&lt;p&gt;As per my basic understanding about Fuzzy ARTMAP, it is a classification algorithm, so how will it be able to predict continuous data?&lt;/p&gt;&#xA;&lt;p&gt;The text from research paper is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In the architecture of the FA network, the preprocessing stages take&#xA;the input vector and contribute to produce complement coding, which&#xA;avoids category proliferation, i.e., the creation of a relatively&#xA;large number of categories to represent the training data. A sequence&#xA;of input vectors (price and demand) and their respective target&#xA;vectors are introduced to the FA network in order to classify the&#xA;input pattern correctly. The classiﬁed input patterns are then grouped&#xA;into labels using membership functions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I was using MATLAB to implement the same. So is there a library in MATLAB to approach towards the solution?&lt;/p&gt;&#xA;" OwnerUserId="4973" LastEditorUserId="85045" LastEditDate="2021-03-12T06:57:13.800" LastActivityDate="2021-03-12T06:57:13.800" Title="FUZZY ARTMAP for continuous data" Tags="|classification|regression|matlab|fuzzy-classification|" AnswerCount="0" CommentCount="0" ClosedDate="2021-03-13T15:26:17.523" ContentLicense="CC BY-SA 4.0" />
  <row Id="2434" PostTypeId="2" ParentId="2432" CreationDate="2014-11-10T02:15:27.147" Score="4" Body="&lt;p&gt;More importantly than the technology is the type of join you are using. For instance if the join keys are sorted, you can use sort merge joins and use join orders to get a better performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, you can use in memory solutions for fastest joins if the size of your intermediate results will not blow up your cluster memory. Look at Spark SQL or Mem-SQL for instance.&lt;/p&gt;&#xA;" OwnerUserId="4974" LastActivityDate="2014-11-10T02:15:27.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2435" PostTypeId="2" ParentId="974" CreationDate="2014-11-10T09:40:26.753" Score="4" Body="&lt;p&gt;If you are working on collaborative filtering you should pose the problem as a low-rank matrix approximation, wherein both the users are items are co-embedded into the same low-dimensionality space. Similarity search will be much simpler then. I recommend using LSH, as you suggested. Another fruitful avenue for dimensionality reduction not yet mentioned is &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_projection&quot; rel=&quot;noreferrer&quot;&gt;random projection&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-11-10T09:40:26.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2436" PostTypeId="1" CreationDate="2014-11-10T09:45:13.063" Score="4" ViewCount="10517" Body="&lt;p&gt;I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not find any suitable one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody know of any such dataset, or of any additional dataset repository on the Internet?&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-01-12T07:58:39.780" Title="Large categorical dataset for regression" Tags="|dataset|categorical-data|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2437" PostTypeId="1" CreationDate="2014-11-10T11:13:07.987" Score="3" ViewCount="306" Body="&lt;p&gt;In this paper (&lt;a href=&quot;http://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)  they suppose a &amp;quot;unified architecture for NLP&amp;quot; with deep neural networks with multitask learning&lt;/p&gt;&#xA;&lt;p&gt;My problem is to understand the layered architecture in figure 1, see below:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/RoU0P.png&quot; alt=&quot;unified deep learning architecture for NLP&quot; /&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is someone able to give me a concrete, reproducible example of how this architecture processing 3 sentences through their layers?&lt;/li&gt;&#xA;&lt;li&gt;What are the outputs after each layer?&lt;/li&gt;&#xA;&lt;li&gt;Why they choose which layer?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3615" LastEditorUserId="83275" LastEditDate="2021-03-12T05:35:13.430" LastActivityDate="2021-03-12T05:35:13.430" Title="How does the supposed &quot;Unified Architecture for NLP&quot; from Collobert and Weston 2008 really works?" Tags="|nlp|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2438" PostTypeId="2" ParentId="2269" CreationDate="2014-11-10T11:16:07.537" Score="3" Body="&lt;p&gt;You can easily have an RStudio server installed in Digital Ocean using &lt;a href=&quot;https://github.com/sckott/analogsea&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; package.&#xA;&lt;a href=&quot;https://github.com/sckott/analogsea&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4978" LastEditorUserId="29169" LastEditDate="2021-02-09T01:32:56.940" LastActivityDate="2021-02-09T01:32:56.940" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2439" PostTypeId="1" AcceptedAnswerId="2462" CreationDate="2014-11-10T16:20:22.647" Score="5" ViewCount="183" Body="&lt;p&gt;I have a &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Molecular+Biology+%28Splice-junction+Gene+Sequences%29&quot; rel=&quot;nofollow&quot;&gt;database&lt;/a&gt; of 3190 instances of DNA consisting of 60 sequential DNA nucleotide positions classified according to 3 types: EI, IE, Other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to formulate a supervised classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My present approach is to formulate a 2nd order Markov Transition Matrix for each instance and apply the resulting data to a Neural Network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How best to approach this classification problem, given that the Sequence of the data should be relevant? Is there a better approach than the one I came up with?&lt;/p&gt;&#xA;" OwnerUserId="136" LastEditorUserId="136" LastEditDate="2014-11-14T11:36:06.290" LastActivityDate="2014-11-14T11:36:06.290" Title="Classification of DNA Sequences" Tags="|sequence|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2441" PostTypeId="2" ParentId="2384" CreationDate="2014-11-10T18:46:42.993" Score="0" Body="&lt;p&gt;You should probably start with a very basic approach: bag-of-words representation (vector as long as your vocabulary, 1 if the word is found in the text, 0 if it's not), and a simple classifier like naive bayes. This works surprisingly well to find topics (a little less for sentiment classification).&#xA;For preprocessing you would probably want to do stop-word removal and stemming (in order to reduce the vocabulary) rather than POS tagging. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with the basic approach is that you would have a n-class classifier, and no &quot;this fits multiple categories&quot; or &quot;this fits 0 categories&quot; answers. If you want to include that aspect, then the best is to design n 2-class classifiers, one for each of your classes, where each classifier decides whether the text fits the class or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I would try out-of-the box naive bayes first, just to see how it works. You can use Weka, it's free, open-source, and can be integrated with java. You can also do the preprocessing (stemming) with the Python NLTK. &lt;/p&gt;&#xA;" OwnerUserId="4760" LastActivityDate="2014-11-10T18:46:42.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2442" PostTypeId="2" ParentId="2436" CreationDate="2014-11-10T19:27:03.397" Score="0" Body="&lt;p&gt;All you need are data sets with enough records and enough features for your purposes. You can simply convert any continuous variables into categorical ones by grouping. Some sources for large sets can be found by a search for &quot;large free data sets&quot;. If you are dead set on lots of categorical data, try insurance data (given that I'm an actuary, I should have thought of that earlier). Those tend to be laden with categorical variables, as I well know from first person experience.&lt;/p&gt;&#xA;" OwnerUserId="1241" LastEditorUserId="1241" LastEditDate="2014-11-12T11:45:02.060" LastActivityDate="2014-11-12T11:45:02.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2443" PostTypeId="2" ParentId="1216" CreationDate="2014-11-11T08:34:22.210" Score="1" Body="&lt;p&gt;May be it will be a little offtopic, but I'd like to highly recommend you to go through this MOOC &lt;a href=&quot;https://www.coursera.org/course/statistics&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/course/statistics&lt;/a&gt;. This is a very good and clear introduction to statistics. It give you a base principles about core field in data science. I hope it will be a good start point for beginning friendship between you and statistics.&lt;/p&gt;&#xA;" OwnerUserId="4842" LastActivityDate="2014-11-11T08:34:22.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2445" PostTypeId="1" CreationDate="2014-11-11T15:03:51.680" Score="3" ViewCount="104" Body="&lt;p&gt;I need to analyse a dataset about mobile phone usage (#calls, #sms, #internetConnections) per each cell and hour in the different days.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[date] [CDR/Position] [#calls] [#sms] [#internetConnections]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My purpose is detecting similarities in the data (Monday-Tuesday is similar... or Monday night is different...). After this, I'd like to find the reason they are similar/dissimilar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I apply?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="989" LastEditDate="2014-11-12T09:42:00.080" LastActivityDate="2014-11-12T09:42:00.080" Title="Analyzing mobile usage. What kind of approach should I apply?" Tags="|data-mining|clustering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2446" PostTypeId="1" CreationDate="2014-11-11T18:34:40.213" Score="3" ViewCount="467" Body="&lt;p&gt;I have a biosemi bdf of EEG data which contains 32 channel. &#xA;I've opened it using biosig, everything works great, a first list is channel and inside each list there are eeg data.&#xA;But if I open it using MNE it the first list is eeg data, and the second list (inside the list of eeg data) are two list of eeg data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is how I open the data using MNE&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;raw_file=read_raw_edf(&quot;E:\eegDATA\\256\s02_reduced.bdf&quot;,preload=True,verbose=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Am I missing something here?&lt;/p&gt;&#xA;" OwnerUserId="273" LastEditorUserId="11097" LastEditDate="2016-01-12T12:07:15.503" LastActivityDate="2016-01-12T12:07:15.503" Title="Opening biosemi bdf data using MNE and biosig using python" Tags="|python|" AnswerCount="0" CommentCount="0" ClosedDate="2016-01-12T15:55:40.623" ContentLicense="CC BY-SA 3.0" />
  <row Id="2447" PostTypeId="1" AcceptedAnswerId="2452" CreationDate="2014-11-11T19:22:00.073" Score="3" ViewCount="147" Body="&lt;p&gt;I'm trying to create a logistic regression model in jpmml, then write the PMML to a file. The problem I'm having, is that I can't find any way to create a custom tag, such as &quot;shortForm&quot; and &quot;longForm&quot; in the following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;MapValues outputColumn=&quot;longForm&quot;&amp;gt;&#xA;  &amp;lt;FieldColumnPair field=&quot;gender&quot; column=&quot;shortForm&quot;/&amp;gt;&#xA;  &amp;lt;InlineTable&amp;gt;&#xA;    &amp;lt;row&amp;gt;&amp;lt;shortForm&amp;gt;m&amp;lt;/shortForm&amp;gt;&amp;lt;longForm&amp;gt;male&amp;lt;/longForm&amp;gt;&#xA;    &amp;lt;/row&amp;gt;&#xA;    &amp;lt;row&amp;gt;&amp;lt;shortForm&amp;gt;f&amp;lt;/shortForm&amp;gt;&amp;lt;longForm&amp;gt;female&amp;lt;/longForm&amp;gt;&#xA;    &amp;lt;/row&amp;gt;&#xA;  &amp;lt;/InlineTable&amp;gt;&#xA;&amp;lt;/MapValues&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's what I have so far:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MapValues mv = new MapValues(&quot;output&quot;)&#xA;  .withFieldColumnPairs(&#xA;        new FieldColumnPair( new FieldName(&quot;gender&quot;), &quot;shortForm&quot; )&#xA;  ).withInlineTable(&#xA;        new InlineTable().withRows(&#xA;                new Row().with???( new ??? )&#xA;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In short, I am asking for an API call I can use to instantiate the &quot;shortForm&quot; element in the example, and attach it to the &quot;row&quot; object. I've been all through the API, examples, and Google/SO, and can't find a thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="4999" LastEditorUserId="21" LastEditDate="2014-11-13T04:44:06.957" LastActivityDate="2014-11-13T04:44:06.957" Title="How can I create a custom tag in JPMML?" Tags="|logistic-regression|java|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2448" PostTypeId="1" CreationDate="2014-11-12T01:31:01.377" Score="4" ViewCount="39" Body="&lt;p&gt;We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the consumer (either through spelling, case etc.) &quot;lots of duplicates&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purposes of analytics and a better UX, we're aiming to create an evolving &quot;master product list&quot;, or &quot;whitelist&quot;, if you will, that will have users select from an existing list of products they are uploading, OR request to add a new one. We also plan to enrich each product entry with additional information from the web, that would be tied to the &quot;master product&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some methods we're proposing to solve this problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) Take all the &quot;items&quot; listed in the website (~90,000), de-dupe as much as possible by running select &quot;distinct&quot; queries (while maintaining a key-map back to original data by generating an array of item keys from each distinct listing in a group-by.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;THEN&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A1) Running this data through mechanical turk, and asking each turk user to list data in a uniform format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A2) Running each product entry through the Amazon products API and asking the user to identify a match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A3) A better method?&lt;/p&gt;&#xA;" OwnerUserId="5004" LastActivityDate="2014-11-12T01:31:01.377" Title="Method to create master product database to validate entries, and enrich data set" Tags="|data-cleaning|sql|" AnswerCount="0" CommentCount="3" ClosedDate="2021-03-13T15:24:02.367" ContentLicense="CC BY-SA 3.0" />
  <row Id="2449" PostTypeId="2" ParentId="2445" CreationDate="2014-11-12T03:12:11.550" Score="1" Body="&lt;p&gt;There are two straight forward (vanilla) ways without going for any fancy featurization: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clustering&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run a clustering algorithm. Something like k-means should work well with this kind of a dataset. While doing this, I would not feed the day_of_week information into the clustering algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest running k-means (after normalizing each of the columns). Choose a small number of clusters that is easy to investigate (or you could use the number of clusters that maximizes the BIC). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Investigate the clusters to understand membership by day_of_week in each of these clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multi-class Classification&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treat the day_of_week as the response that you would like to predict. Build a decision tree of a fixed depth to predict the day_of_week given the columns. By examining this tree, you can easily tell, which decisions led to a set of leaves being labeled Sunday vs the set of decisions that led to a set of leaves being labeled Monday. These decisions will also help you understand the similarities between different days. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2014-11-12T06:23:54.947" LastActivityDate="2014-11-12T06:23:54.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2450" PostTypeId="2" ParentId="2436" CreationDate="2014-11-12T06:14:15.090" Score="0" Body="&lt;p&gt;Try the &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/KDD+Cup+1998+Data&quot; rel=&quot;nofollow&quot;&gt;1998 KDD Cup dataset&lt;/a&gt;. Its a regression problem with categorical and integer predictors. For your task, you could either treat integer predictors as categorical or ignore them completely. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-12T06:14:15.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2451" PostTypeId="1" CreationDate="2014-11-12T09:21:19.490" Score="3" ViewCount="213" Body="&lt;p&gt;For problems where the data represents online fraud or insurance (where each row represents a transaction), it is typical for the response variable to denote the value of fraud committed in dollars. Such a response value might have less than 5% non-zero values denoting fraudulent transactions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions regarding such a dataset: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What algorithms can we use to ensure that the model not only predicts the fraudulent transactions accurately, but also predicts the value of fraud associated with these.  &lt;/li&gt;&#xA;&lt;li&gt;Assuming that we can quantify the cost involved in each false positive (tagging a non-fraudulent transaction as fraudulent) and cost incurred due to a false negative (tagging a fraudulent transaction as non-fraudulent), how can we optimize the model to maximize savings (or minimize losses)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2014-11-12T16:39:52.363" LastActivityDate="2014-11-18T23:44:58.257" Title="Modeling when the response variable has too many 0's and few continuous values?" Tags="|classification|regression|predictive-modeling|class-imbalance|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2452" PostTypeId="2" ParentId="2447" CreationDate="2014-11-12T11:08:44.973" Score="1" Body="&lt;p&gt;You can/should use a generic Java Architecture for XML Binding (JAXB) approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply put, call &lt;code&gt;Row#withContent(Object...)&lt;/code&gt; with instances of &lt;code&gt;org.w3c.dom.Element&lt;/code&gt; that represent the desired XML content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document document = documentBuilder.newDocument();&#xA;Element shortForm = document.createElement(&quot;shortForm&quot;);&#xA;shortForm.setTextContent(&quot;m&quot;);&#xA;Element longForm = document.createElement(&quot;longForm&quot;);&#xA;longForm.setTextContent(&quot;male&quot;);&#xA;row = row.withContent(shortForm, longForm);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5012" LastActivityDate="2014-11-12T11:08:44.973" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2453" PostTypeId="2" ParentId="2303" CreationDate="2014-11-12T14:51:42.213" Score="0" Body="&lt;p&gt;If you prefer quick hands-on/interactive tutorials, below are my suggestions - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python - &lt;a href=&quot;http://www.codecademy.com/&quot; rel=&quot;nofollow&quot;&gt;codeacademy&lt;/a&gt;, Google Python Class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R - CodeSchool's &lt;a href=&quot;http://tryr.codeschool.com/&quot; rel=&quot;nofollow&quot;&gt;'Try R'&lt;/a&gt; and DataCamp (suggested above)&lt;/p&gt;&#xA;" OwnerUserId="782" LastActivityDate="2014-11-12T14:51:42.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2454" PostTypeId="1" AcceptedAnswerId="5596" CreationDate="2014-11-12T17:27:57.850" Score="6" ViewCount="1457" Body="&lt;p&gt;We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data involved, ideally I'd want to run the computation in the same location as the data rather than shifting the data around, but this doesn't seem possible with Redshift. I've looked at MADlib, but this is not an option as Redshift does not support UDFs (which MADlib requires). I'm currently looking at shifting the data over to EMR and processing it with the Apache Spark machine learning library (or maybe H20, or Mahout, or whatever). So my questions are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;is there a better way?&lt;/li&gt;&#xA;&lt;li&gt;if not, how should I make the data accessible to Spark? The options I've identified so far include: use Sqoop to load it into HDFS, use DBInputFormat, do a Redshift export to S3 and have Spark grab it from there. What are the pros/cons for these different approaches (and any others) when using Spark?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that this is off-line batch learning, but we'd like to be able to do this as quickly as possible so that we can iterate experiments quickly.&lt;/p&gt;&#xA;" OwnerUserId="5019" LastActivityDate="2015-09-26T09:52:25.717" Title="Processing data stored in Redshift" Tags="|apache-hadoop|aws|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2455" PostTypeId="2" ParentId="2268" CreationDate="2014-11-12T18:22:07.253" Score="2" Body="&lt;p&gt;NLP is very vast and varied. Here are a few basic tools in NLP:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sentence splitting: Identifying sentence boundaries in text&lt;/li&gt;&#xA;&lt;li&gt;Tokenization: Splitting a sentence into individual words&lt;/li&gt;&#xA;&lt;li&gt;Lemmatization: Converting a word to its root form. E.g. says, said,&#xA;saying will all map to root form - say &lt;/li&gt;&#xA;&lt;li&gt;Stemmer: It is similar to a&#xA;lemmatizer, but it stems a word rather than get to the root form.&#xA;e.g. laughed, laughing will stem to laugh. However, said, saying&#xA;will map to sa - which is not particularly enlightening in terms of&#xA;what &quot;sa&quot; means&lt;/li&gt;&#xA;&lt;li&gt;POS tagger: Tags a word with the Part of Speech - what is a noun, verb, preposition etc.&lt;/li&gt;&#xA;&lt;li&gt;Parser: Links words with POS tags to other words with POS tags. E.g. John ate an apple. Here John and apple are nouns linked by the verb - eat. John is the subject of the verb, and apple is the object of the verb.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you are looking for the state of the art for these tools, check out &lt;a href=&quot;http://nlp.stanford.edu/software/corenlp.shtml&quot; rel=&quot;nofollow&quot;&gt;StanfordCoreNLP&lt;/a&gt;, which has most of these tools and a trained model to identify the above from a document. There is also an online demo to check out stanfordCoreNLP before downloading and using it with your application. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NLP has several subfields. Here are a few of them: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Machine Translation: Automatic Translation from one language to another&lt;/li&gt;&#xA;&lt;li&gt;Information Retrieval: Something like a search engine, that retrieves relevant information from a large set of documents based on a search query&lt;/li&gt;&#xA;&lt;li&gt;Information Extraction: Extract concepts and keywords - such as names of people, locations, times, synonyms etc.&lt;/li&gt;&#xA;&lt;li&gt;Deep Learning has lately become a new field of NLP, where a system tries to understand a document like a human understands it. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5021" LastActivityDate="2014-11-12T18:22:07.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2456" PostTypeId="2" ParentId="566" CreationDate="2014-11-12T18:41:33.053" Score="1" Body="&lt;p&gt;You have a great idea going, and it might work for your specific project. However there are a few considerations you should take into account: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In your first sentence, Obama in incorrectly classified as an organization, instead of a person. This is because the training model used my NLTK probably does not have enough data to recognize Obama as a PERSON. So, one way would be to update this model by training a new model with a lot of labeled training data. Generating labeled training data is one of the most expensive tasks in NLP - because of all the man hours it takes to tag sentences with the correct Part of Speech as well as semantic role. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In sentence 2, there are 2 concepts - &quot;Former Vice President&quot;, and &quot;Dick Cheney&quot;. You can use co-reference to identify the relation between the 2 NNPs. Both the NNP are refering to the same entity, and the same entity could be referenced as - &quot;former vice president&quot; as well as &quot;Dick Cheney&quot;. Co-reference is often used to identify the Named entity that pronouns refer to. e.g. &quot;Dick Cheney is the former vice president of USA. He is a Republican&quot;. Here the pronoun &quot;he&quot; refers to &quot;Dick Cheney&quot;, and it should be identified by a co-reference identification tool. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5021" LastActivityDate="2014-11-12T18:41:33.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2457" PostTypeId="2" ParentId="1079" CreationDate="2014-11-12T19:00:05.013" Score="2" Body="&lt;p&gt;TF IDF will give you the degree of measure of how relevant a document is to your query. owever, to evaluate your IR system you need to use metrics such as - Precision, Recall and F score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Precision: Out of all the documents that your system retrieves, which ones are relevant? This measures how much noise there is in the output of your IR system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall: Out of all the documents that are relevant, which ones did your system retrieve? This measures how much coverage does your IR system have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to get 100% recall all the time by basically retrieving ALL documents from a collection for any query. However, the precision in this case will be very low. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to get a very high precision by hand modeling an IR system ti produce very accurate results. However, it would produce a very bad recall as there will not be coverage over all the documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we need to measure F score- which is the harmonic mean between Precision and Recall&#xA;Check out &lt;a href=&quot;http://nlp.stanford.edu/IR-book/pdf/08eval.pdf&quot; rel=&quot;nofollow&quot;&gt;Chapter 8 of the Stanford IR book&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for datasets only here are a few that are relevant: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://zitnik.si/mediawiki/index.php?title=Datasets&quot; rel=&quot;nofollow&quot;&gt;http://zitnik.si/mediawiki/index.php?title=Datasets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.daviddlewis.com/resources/testcollections/&quot; rel=&quot;nofollow&quot;&gt;http://www.daviddlewis.com/resources/testcollections/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://boston.lti.cs.cmu.edu/callan/Data/&quot; rel=&quot;nofollow&quot;&gt;http://boston.lti.cs.cmu.edu/callan/Data/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5021" LastActivityDate="2014-11-12T19:00:05.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2458" PostTypeId="2" ParentId="2303" CreationDate="2014-11-12T19:31:27.170" Score="1" Body="&lt;p&gt;There are really so many good resources now.  If you want to stay away from textbooks, both O'Reilly Media and Packt Publishing offer much lighter but effective reading on a lot of great topics.  These books are much more applied in practice.&lt;/p&gt;&#xA;&lt;p&gt;As far as learning the languages go, Coursera, Udacity, Code Acadmey, and Code School have great tutorials.  I would recommend taking a look at the following:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.coursera.org/courses?orderby=upcoming&amp;amp;lngs=en&amp;amp;cats=stats,cs-ai&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera AI and Stats Courses&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.udacity.com/courses#!/data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;Udacity Data Science courses&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="5022" LastEditorUserId="98307" LastEditDate="2020-08-02T12:49:46.867" LastActivityDate="2020-08-02T12:49:46.867" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2459" PostTypeId="2" ParentId="2303" CreationDate="2014-11-12T19:35:01.550" Score="0" Body="&lt;p&gt;Also, &lt;strong&gt;&lt;a href=&quot;http://www.greenteapress.com/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Green Tea Press&lt;/a&gt;&lt;/strong&gt; offers free books on related topics such as an intro to Python and using python with Probability and Stats.&lt;/p&gt;&#xA;" OwnerUserId="5022" LastEditorUserId="98307" LastEditDate="2020-08-02T12:49:39.407" LastActivityDate="2020-08-02T12:49:39.407" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2462" PostTypeId="2" ParentId="2439" CreationDate="2014-11-13T07:17:18.147" Score="3" Body="&lt;p&gt;One way would be to create 20 features (each feature representing a codon). In this way, you would have a dataset with 3190 instances and 20 categorical features. There is no need to treat the sequence as a Markov chain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the dataset has been featurized as suggested above, any supervised classifier can work well. I would suggest using a gradient boosting machine as it might be better suited to handle categorical features. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-13T07:17:18.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2463" PostTypeId="1" AcceptedAnswerId="2490" CreationDate="2014-11-13T08:58:25.847" Score="2" ViewCount="521" Body="&lt;p&gt;I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What will be the best way to visualize this kind of data set? This visualization should also be able to use the geo coordinates to place this entities in the map to form clusters, or to superimpose them on a real map.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in an algorithm and/or a library that will help me do this. Library should be preferably written in Java, Python, Scala, or NodeJS. The record count can be as big as a thousand or hundreds of thousands.&lt;/p&gt;&#xA;" OwnerUserId="5027" LastEditorUserId="31513" LastEditDate="2017-04-24T17:38:16.470" LastActivityDate="2017-04-24T17:38:16.470" Title="Geospatial Social Network Analysis Visualization" Tags="|visualization|social-network-analysis|scalability|library|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2464" PostTypeId="1" CreationDate="2014-11-13T09:19:22.797" Score="9" ViewCount="14580" Body="&lt;p&gt;I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(Ideally I would like to use a good one for overlapping clusters so any recommendation is welcomed). I managed to get the topics but they are in the form of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.041*Minister + 0.041*Key + 0.041*moments + 0.041*controversial + 0.041*Prime&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to apply a clustering algorithm, and correct me if I'm wrong, I believe I should find a way to represent each word as a number using either tfidf or word2vec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any ideas of how I could &quot;strip&quot; the textual information from e.g. a list, in order to do so and then place them back in order to make the appropriate multiplication?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance the way I see it if the word Minister has a tfidf weight of 0.042 and so on for any other word within the same topic I should be to compute something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.041*0.42 + ... + 0.041*tfidf(Prime) and get a result that will be later on used in order to cluster the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time.&lt;/p&gt;&#xA;" OwnerUserId="5029" LastActivityDate="2020-12-05T00:04:44.510" Title="Clustering of documents using the topics derived from Latent Dirichlet Allocation" Tags="|python|clustering|lda|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2466" PostTypeId="1" CreationDate="2014-11-14T08:53:21.510" Score="3" ViewCount="157" Body="&lt;p&gt;I have measurements of 4 devices at two different points of time. A measurement basically consists of an array of ones and zeros corresponding to a bit value at the corresponding location:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;whos measurement1_dev1_time1&#xA;&#xA;Name                         Size               Bytes  Class      Attributes&#xA;&#xA;measurement1_dev1_time1      4096x8             32768  logical&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I assume that for a specific device the changes between time 1 and 2 of the measurements are unique. However, since I am dealing with 32768 bits at different locations, it is quite hard to visualize if there is some kind of dependency. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As every bit at location &lt;code&gt;x&lt;/code&gt;can be regarded as one dimension of an observation I thought to use PCA to reduce the number of dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, for every of the 5 devices:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I randomly sample &lt;code&gt;n&lt;/code&gt; measurements at point &lt;code&gt;t1&lt;/code&gt;and &lt;code&gt;t2&lt;/code&gt; seperatly&lt;/li&gt;&#xA;&lt;li&gt;I prepare an array as input for &lt;code&gt;pca()&lt;/code&gt; with &lt;code&gt;m&lt;/code&gt;*n columns (&lt;code&gt;m&lt;/code&gt;&amp;lt; 32768; its a subset of all the observed bits, as the original data might be too big for pca) and 4 rows (one row for each device).&lt;/li&gt;&#xA;&lt;li&gt;On this array &lt;code&gt;A&lt;/code&gt; I calculate the pca: ``[coeff score latent] = pca(zscore(A))```&lt;/li&gt;&#xA;&lt;li&gt;Then I try to visualize it using &lt;code&gt;biplot&lt;/code&gt;: &lt;code&gt;biplot(coeff(:,1:2), 'score', score(:,1:2))&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, this gives me really strange results. Maybe PCA is not the right approach for this problem? I also modified the input data to do the PCA not on the logical bit array itself. Instead, I created a vector, which holds the indices where there is a '1' in the original measurement array. Also this produces strange results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I am completely new to PCA I want to ask you if you either see a flaw in the process or if PCA is just not the right approach for my goal and I better look for other dimension reduction approaches or clustering algorithms.&lt;/p&gt;&#xA;" OwnerUserId="5038" LastActivityDate="2017-06-27T20:42:42.587" Title="Dimension reduction for logical arrays" Tags="|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2468" PostTypeId="2" ParentId="2263" CreationDate="2014-11-14T15:15:20.617" Score="3" Body="&lt;h3&gt;Storing user profiles&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If you just want to store all user profiles... just save them into normal &lt;strong&gt;RDBMS&lt;/strong&gt;. Assuming one user profiles takes 10Kb of storage, you need only ~9.5Gb for every million of users, which is pretty little and gives you all advantages of mature relational databases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes sense to use &lt;strong&gt;HBase&lt;/strong&gt; only when you have really many users (say, &gt; 1B) or when data is very sparse (most columns are empty). But don't expect it to be as convenient as good old SQL databases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In advertising, and especially in &lt;a href=&quot;http://en.wikipedia.org/wiki/Real-time_bidding&quot; rel=&quot;nofollow&quot;&gt;real-time bidding&lt;/a&gt;, very fast retrieval of user profiles is needed. &lt;a href=&quot;http://www.aerospike.com/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Aerospike&lt;/strong&gt;&lt;/a&gt; becomes more and more popular for this task.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Analysing data slices&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Common use of business logs is to analyse specific slices of data, e.g. number of users from France that visited sites from &quot;game&quot; category on November 1-14, 2014. Standard way to manage such data efficiently is to organize them into &lt;a href=&quot;http://en.wikipedia.org/wiki/Data_cube&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;data cubes&lt;/strong&gt;&lt;/a&gt;. You won't get individual records (e.g. users), but you'll get aggregated statistics really fast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such cubes may have many different dimensions, but in 99% of cases they have &lt;strong&gt;date field&lt;/strong&gt; that they are &lt;strong&gt;partitioned&lt;/strong&gt; by. It makes great sense, because almost every query includes time period to get data from. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for software, &lt;strong&gt;Vertica&lt;/strong&gt; is great for such aggregations. Cheaper* solution from Hadoop world is &lt;strong&gt;Impala&lt;/strong&gt;, which is also great. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(* - if you count only license price)&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Machine learning&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It really depends on concrete tasks and ML toolkit in use. For real-time bidding you would want blazing fast access to user profile vectors and would probably prefer Aerospike. For online learning &lt;strong&gt;Spark Streaming&lt;/strong&gt; may be used as a data source, and no storage used at all. For offline machine learning there's excellent &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-guide.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;MLlib&lt;/strong&gt;&lt;/a&gt; from the same Spark project, which works with a variety of sources. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-11-14T15:15:20.617" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2469" PostTypeId="1" AcceptedAnswerId="2476" CreationDate="2014-11-14T15:20:44.463" Score="3" ViewCount="349" Body="&lt;p&gt;I did small survey and get such data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|-------------| Yes | No | Dont_Know |  &#xA;|-------------|     |    |           |  &#xA;| Employee    | 60  | 5  | 5         |  &#xA;| Workers     | 17  | 0  | 1         |  &#xA;| Businessmen | 71  | 5  | 10        |  &#xA;| Jobless     | 4   | 30 | 0         |  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt &amp;lt;- data.frame(workers = c(&quot;Employee&quot;,&#xA;                             &quot;Workers&quot;, &#xA;                             &quot;Businessmen&quot;, &#xA;                             &quot;Jobless&quot;), &#xA;                 yes = c(60,17,71,4), &#xA;                 no = c(5,0,5,30), &#xA;                 dont_know = c(5,1,10,0)&#xA;                )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What kind of test I must do, if I want to show, that the Jobless people are often choosing &lt;strong&gt;No&lt;/strong&gt; answer?   &lt;/li&gt;&#xA;&lt;li&gt;Is the difference between Jobless and Businessmen answers significant? &lt;/li&gt;&#xA;&lt;li&gt;And what is about other groups?  &lt;/li&gt;&#xA;&lt;li&gt;What another information I can get from such data or what kind questions I can ask from such data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3377" LastEditorUserId="2961" LastEditDate="2014-11-14T15:48:17.037" LastActivityDate="2014-11-15T01:26:16.583" Title="survey data analysis (discrete data)" Tags="|r|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2470" PostTypeId="1" CreationDate="2014-11-14T16:39:29.663" Score="2" ViewCount="189" Body="&lt;p&gt;I have two classes (A,B) that I would like to classify using a SVM. Say that I have a class C and a function f. Can I do this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;A' =  f(A,C) = |A-C|&#xA;B' =  f(B,C) = |B-C|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;and then perform the classification on A' and B' instead? In the context of my problem A and B are classes where elements are vectors. The f function measures the Mahalanobis distance of each vector with respect to the distribution imposed by C.&lt;/p&gt;&#xA;" OwnerUserId="5042" LastEditorUserId="29169" LastEditDate="2021-03-11T18:24:57.270" LastActivityDate="2021-03-11T18:24:57.270" Title="Non-linear transformations input dataset for support vector machines" Tags="|svm|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="2471" PostTypeId="2" ParentId="1154" CreationDate="2014-11-14T17:11:13.060" Score="2" Body="&lt;p&gt;I would definitely use a graph (Though, this clearly depends on the final application, maybe you could add more information) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the nodes, you should consider as nodes not only tanks but also points were pipelines change name or bifurcate. For instance, following your example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    e1     e2        e3          e4        e5          e7           &#xA;    +---+----------.---------+----------.----------+-----------      &#xA;    |          |         |                     |          |      &#xA;[R tank 1]  [S tank 1] [S tank 2]              |e6     [S tank 3]&#xA;                                               |                 &#xA;                                           [S tank 4]            &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now adding the nodes:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    e1 n2  e2     n4   e3   n6   e4    n8  e5     n9   e7           &#xA;n1 +---+----------.---------+----------.----------+----------+      &#xA;   |          |         |                     |              |      &#xA;  n3         n5        n7                     |e6           n11     &#xA;                                              |                 &#xA;                                             n10                &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, you need some kind of mapping. &#xA;Some of the nodes will map to tanks:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[R tank 1]     n3                                                   &#xA;[S tank 1]     n5                                                   &#xA;[S tank 2]     n7                                                   &#xA;[S tank 4]     n10                                                  &#xA;[S tank 3]     n11                                                  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the pipelines will be represented by paths in the graph&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pipeline 1     e1 e2                                                &#xA;Pipeline 2     e3 e4                                                &#xA;Pipeline 3     e5 e7                                                &#xA;Pipeline 4     e6                                                   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2014-11-14T17:11:13.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2472" PostTypeId="2" ParentId="1189" CreationDate="2014-11-14T17:32:26.050" Score="8" Body="&lt;p&gt;You are just predicting if Play = Yes or Play = No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The confusion matrix would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             Predicted&#xA;          +------+------+&#xA;          |  Yes |  No  |&#xA;    +-------------------+&#xA;A   |     |      |      |&#xA;c   | Yes |  TP  |  FP  |&#xA;t   |     |      |      |&#xA;u   +-------------------+&#xA;a   |     |      |      |&#xA;l   | No  |  FN  |  TN  |&#xA;    |     |      |      |&#xA;    +-----+------+------+&#xA;&#xA;TP: True positives&#xA;FP: False positives &#xA;FN: False negatives &#xA;TN: True negatives&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The accuracy can then be calculated as (TP + TN)/(TP + FP + TN + FN). &lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="847" LastEditDate="2014-11-15T14:27:22.397" LastActivityDate="2014-11-15T14:27:22.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2473" PostTypeId="1" AcceptedAnswerId="2479" CreationDate="2014-11-14T19:03:35.603" Score="2" ViewCount="1472" Body="&lt;p&gt;I am dealing with a lot of categorical data right now and I would like to use an appropriate data mining method in any tool [preferably R] to find the effect of each parameter [categorical parameters] over my target variable. To give a brief notion about the data that am dealing with, my target variable denotes the product type [say, disposables and non-disposables] and I have parameters like root cause,symptom,customer name, product name etc. As my target can be considered as a binary value, I tried to find the combination of values leading to the desired categories using Apriori but, I have more than 2 categories in that attribute and I want to use all of them and find the effect of the mentioned parameters over each category. I really wanted to try SVM and use hyperplanes to separate the content and get n-dimensional view. But, I do not have enough knowledge to validate the technique, functions am using to do the analysis. Currently I have like 9000 records and each of them represents a complaint from the user. There are lot of columns available in the dataset which is what I am trying to use to determine the target variable [ myForumla &amp;lt;- Target~. ] I tried with just 4 categorical columns too. Not getting a proper result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can just the categorical variables be used to develop a SVM model and get visualization with n hyper planes? Is there any appropriate data mining technique available for dealing with just the categorical data?&lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2015-10-21T01:46:30.197" Title="Choosing the right data mining method to find the effect of each parameter over the target" Tags="|data-mining|classification|r|svm|categorical-data|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2474" PostTypeId="1" AcceptedAnswerId="2483" CreationDate="2014-11-14T22:48:10.173" Score="2" ViewCount="144" Body="&lt;p&gt;I think this is something that experienced programmers do all the time. But, given my limited programming experience, please bear with me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an excel file which has particular cell entries that read &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[[{&quot;from&quot;: &quot;4&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;8&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;9&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;3&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}], [{&quot;from&quot;: &quot;14&quot;, &quot;response&quot;: false, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;15&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;17&quot;, &quot;response&quot;: false, &quot;value&quot;: 20}, {&quot;from&quot;: &quot;13&quot;, &quot;response&quot;: true, &quot;value&quot;: 20}]]&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for each such entry I want to take the information in each of the curly brackets and make a row of data out of it. Each such row would have 3 columns. For example, the row formed from the first entry within curly brackets should have the entries &quot;4&quot; &quot;true&quot; and &quot;20&quot; respectively. The part I posted should give me 6 such rows, and for n such repetitions I should end up with a matrix of 6n rows, and 4 columns ( an identifier, plus the 3 columns mentioned).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be most efficient way to do this? By &quot;doing this&quot; I mean learning the trick, and then implementing it. I have access to quite a few software packages(Excel, Stata, Matlab, R) in my laboratory, so that should not be an issue.&lt;/p&gt;&#xA;" OwnerUserId="5044" LastActivityDate="2014-11-27T03:27:46.193" Title="Parsing data from a string" Tags="|parsing|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2476" PostTypeId="2" ParentId="2469" CreationDate="2014-11-15T01:26:16.583" Score="2" Body="&lt;p&gt;Here are some things to try.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Plot a bar graph. The bar graph will clearly show jobless people are often choosing NO. Try an 1-way ANOVA test. If the p &amp;lt; delta (i.e. delta=0.05), try a post-hoc test (i.e. Tukey's HSD) to do a pairwise comparison.&lt;/li&gt;&#xA;&lt;li&gt;Like I said earlier, try a multiple comparison test (1-way ANOVA) first, if there is a statistically significant difference, you can try a pairwise comparison test (post-hoc test).&lt;/li&gt;&#xA;&lt;li&gt;Maybe try a clustering algorithm? Be careful, because the marginal sums (by rows or columns) are not equal. Maybe create a similarity matrix by profession? To me, it seems that Employees and Businessmen are in one group (very similar), while Workers and Jobless are each in their own group. If you turn those frequencies into proportions, then you might just have 2 groups; one for employees + workers + businessmen, and one for jobless.&lt;/li&gt;&#xA;&lt;li&gt;Use contingency table analysis to see if the responses (yes/no/don't know) are associated with profession. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3083" LastActivityDate="2014-11-15T01:26:16.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2477" PostTypeId="2" ParentId="2451" CreationDate="2014-11-15T01:30:00.870" Score="1" Body="&lt;p&gt;How about&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Ordinary Least Square (OLS) regression? Since you have a class imbalance, you might want to combine that with boosting algorithms.&lt;/li&gt;&#xA;&lt;li&gt;If you have a function to quantify the cost involved with FP's and FN's, use any optimization technique you can find. My favorite is genetic algorithms. You may also try linear programming. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3083" LastActivityDate="2014-11-15T01:30:00.870" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2478" PostTypeId="2" ParentId="1075" CreationDate="2014-11-15T03:18:13.223" Score="1" Body="&lt;p&gt;Data volume is not the only criterion for using Hadoop. Big Data is often characterized by the 3 V's: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;volume, &lt;/li&gt;&#xA;&lt;li&gt;velocity, and &lt;/li&gt;&#xA;&lt;li&gt;variety. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;More V's than these 3 have been invented since. I suppose the V's were a catchy way to characterize what is Big Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as hinted, computational intensity is a perfect reason for using Hadoop (if your algorithm is computationally expensive). And, as hinted, the problem you describe is perfect for Hadoop, especially since it is embarrassingly parallel in nature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is Hadoop a good choice for you? I would argue, yes. Why? Because &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Hadoop is open source (compared with proprietary systems which may be expensive and black boxes), &lt;/li&gt;&#xA;&lt;li&gt;your problem lends itself well to the MapReduce paradigm (embarassingly parallel, shared-nothing), &lt;/li&gt;&#xA;&lt;li&gt;Hadoop is easily scalable with commodity hardware (as opposed to specialized hardware, and you should get linear speed-up in your performance with just throwing hardware at the problem, and you can just spin a cluster as needed on cloud service providers),&lt;/li&gt;&#xA;&lt;li&gt;Hadoop allows multiple client languages (Java is only one of many supported languages),&lt;/li&gt;&#xA;&lt;li&gt;there might be a library available already to do your cross-product operation, and&lt;/li&gt;&#xA;&lt;li&gt;you're shipping compute code, not data, around the network (which you should benefit from, and as opposed to other distributed platforms where you are shipping data to compute nodes which is the bottleneck).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please note, Hadoop is not a distributed file system (as mentioned, and corrected already). Hadoop is distributed storage and processing platform. The distributed storage component of Hadoop is called the Hadoop Distributed File System (HDFS), and the distributed processing component is called MapReduce. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadoop has now evolved slightly. They keep the HDFS part for distributed storage. But they have a new component called YARN (Yet Another Resource Negotiator), which serves to appropriate resources (CPU, RAM) for any compute task (including MapReduce). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the &quot;overhead&quot; part, there is noticeable overhead with starting/stopping a Java Virtual Machine (JVM) per tasks (map tasks, reduce tasks). You can specify for your MapReduce Jobs to reuse JVMs to mitigate this issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &quot;overhead&quot; is really an issue, look into Apache Spark, which is part of the Hadoop ecosystem, and they are orders of magnitude faster than MapReduce, especially for iterative algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used Hadoop to compute pairwise comparisons (e.g. correlation matrix, similarity matrix) that are O(N^2) (n choose 2) in worst case running time complexity. Imagine computing the correlations between 16,000 variables (16,000 choose 2); Hadoop can easily process and store the results if you have the commodity resources to support the cluster. I did this using the preeminent cloud service provider (I won't name it, but you can surely guess who it is), and it cost me &amp;lt; $100 and under 18 hours.  &lt;/p&gt;&#xA;" OwnerUserId="3083" LastActivityDate="2014-11-15T03:18:13.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2479" PostTypeId="2" ParentId="2473" CreationDate="2014-11-15T09:34:47.653" Score="1" Body="&lt;p&gt;You can try Bayesian belief networks (BBNs). BBNs can easily handle categorical variables and give you the picture of the multivariable interactions. Furthermore, you may use sensitivity analysis to observe how each variable influences your class variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you learn the structure of the BBN, you can identify the Markov blanket of the class variable. The variables in the Markov blanket of the class variable is a subset of all the variables, and you may use optimization techniques to see which combination of values in this Markov blanket maximizes your class prediction.&lt;/p&gt;&#xA;" OwnerUserId="3083" LastEditorUserId="3083" LastEditDate="2014-11-15T09:40:33.473" LastActivityDate="2014-11-15T09:40:33.473" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2480" PostTypeId="1" CreationDate="2014-11-15T15:49:07.213" Score="2" ViewCount="292" Body="&lt;p&gt;I am new to D3 programming (any programming, for that matter). I have protein-protein interaction data in JSON format and csv format. I would like to use that data for network visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data attributes: Protein Name, Protein Group, Protein type, Protein Source Node, Protein Target Node&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest good network visualizations for such data. How does it work with hive plots?&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-10-16T06:21:39.837" Title="Visualization using D3" Tags="|visualization|javascript|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2482" PostTypeId="2" ParentId="2480" CreationDate="2014-11-16T01:05:02.603" Score="3" Body="&lt;p&gt;Why dont you have a look at the following example?&#xA;&lt;a href=&quot;http://bl.ocks.org/mbostock/2066421&quot; rel=&quot;nofollow&quot;&gt;http://bl.ocks.org/mbostock/2066421&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also find a fiddle here: &lt;a href=&quot;http://jsfiddle.net/boatrokr/rk2s5/&quot; rel=&quot;nofollow&quot;&gt;http://jsfiddle.net/boatrokr/rk2s5/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pay attention to the part where the links are defined.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2014-11-16T01:05:02.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2483" PostTypeId="2" ParentId="2474" CreationDate="2014-11-16T01:13:21.363" Score="2" Body="&lt;p&gt;If you have R is quite simple&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Copy the lines into a file, let's say: &quot;mydata.json&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Be sure you have installed the rjson package&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; install.packages(&quot;rjson&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Import your data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(&quot;rjson&quot;)&#xA; json_data &amp;lt;- fromJSON(file=&quot;mydata.json&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2014-11-16T01:13:21.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2486" PostTypeId="1" AcceptedAnswerId="2496" CreationDate="2014-11-16T21:03:09.037" Score="3" ViewCount="7121" Body="&lt;p&gt;I was wondering if anyone was aware of any methods for visualizing an SVM model where there are more than three continuous explanatory variables. In my particular situation, my response variable is binomial, with 6 continuous explanatory variables (predictors), one categorical explanatory variable (predictor). I have already reduced the number of predictors and I am primarily using R for my analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am unaware if such a task is possible/ worth pursuing.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="5023" LastEditorUserId="847" LastEditDate="2014-11-21T10:39:38.337" LastActivityDate="2014-11-21T18:55:07.860" Title="Visualizing Support Vector Machines (SVM) with Multiple Explanatory Variables" Tags="|machine-learning|classification|r|visualization|svm|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2487" PostTypeId="1" CreationDate="2014-11-17T11:54:39.933" Score="1" ViewCount="95" Body="&lt;p&gt;this is my first ever stack exchange question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to build a tool right now and one of the features of the tool is the ability to break down a product or service into it's associated attributes/properties/classes/keywords/entities. (Choose which word best suits, as I have no idea).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if we had a Camera as the product. I would like to be able to generate a breakdown of everything that is associated to a camera. Such as;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Digital, Film, Optical, LCD, Glass, CCD, CMOS, RGB, Lens, Shutter, Negative, Polaroid, Darkroom, Flash, Resolution, Stabilisation, Batteries, Zoom, Angle, Telephoto, Macro, Filters, Memory, CF, SD&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The list could go on for quite some time, those were jsut a few off the top of my head.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How on earth could I go about retrieving such attributes automatically? Is there a database out there that has such info? Are there any special tricks anyone has up their sleeve to be able to accumulate datasets such as the example above?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very interested in your answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="5066" LastActivityDate="2014-11-18T01:26:51.900" Title="Ontology database" Tags="|data-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2488" PostTypeId="2" ParentId="2487" CreationDate="2014-11-18T01:26:51.900" Score="1" Body="&lt;p&gt;It seems to me as if a good starting point would be to read up on the semantic web, perhaps starting with &lt;a href=&quot;http://dbpedia.org/About&quot; rel=&quot;nofollow&quot;&gt;DBpedia&lt;/a&gt; and maybe LinkedData. You could go from there and build up your own database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of a &lt;a href=&quot;http://dbpedia.org/snorql/?query=%0D%0Aselect%20%3Flabel%20where%20%7B%0D%0A%20%20%23%3ACamera%20dbpedia-owl%3Aabstract%20%3Fabstract.%0D%0A%20%20%23FILTER%20langMatches%28%20lang%28%3Fabstract%29%2C%20%27en%27%29.%0D%0A%20%20%23%3Fprod%20dbpedia2%3Aproducts%20%3Frelated%20.%0D%0A%20%20%3Fprod%20dbpedia-owl%3Aproduct%20%3ACamera%20.%0D%0A%20%20%3Fprod%20dcterms%3Asubject%20%3Fcategories%20.%0D%0A%20%20%3Fentity%20dcterms%3Asubject%20%3Fcategories.%0D%0A%20%20%3Fentity%20rdf%3Atype%20yago%3APhysicalEntity100001930%20.%0D%0A%20%20%3Fentity%20rdfs%3Alabel%20%3Flabel%20.%0D%0A%20%20filter%20langMatches%28%20lang%28%3Flabel%29%2C%20%27en%27%29.%0D%0A%7D&quot; rel=&quot;nofollow&quot;&gt;SPARQL query&lt;/a&gt; starting with the DBpedia page of 'Camera':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select ?label where {&#xA;  ?prod dbpedia-owl:product :Camera .&#xA;  ?prod dcterms:subject ?categories .&#xA;  ?entity dcterms:subject ?categories.&#xA;  ?entity rdf:type yago:PhysicalEntity100001930 .&#xA;  ?entity rdfs:label ?label .&#xA;  filter langMatches( lang(?label), 'en').&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Generating a lot of words somehow related to 'Camera'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;&quot;Shutter button&quot;@en&#xA;&quot;Rangefinder camera&quot;@en&#xA;&quot;Still camera&quot;@en&#xA;&quot;Lomo LC-A&quot;@en&#xA;&quot;Flexaret&quot;@en&#xA;&quot;Land Camera&quot;@en&#xA;&quot;Robot (camera)&quot;@en&#xA;&quot;Speed Graphic&quot;@en&#xA;&quot;Ansco Panda&quot;@en&#xA;&quot;Image trigger&quot;@en&#xA;&quot;Still video camera&quot;@en&#xA;&quot;Hidden camera&quot;@en&#xA;&quot;Mainichi Shimbun&quot;@en&#xA;&quot;Ōhiradai Station&quot;@en&#xA;&quot;Depth-of-field adapter&quot;@en&#xA;&quot;Banquet camera&quot;@en&#xA;&quot;Digital versus film photography&quot;@en&#xA;&quot;Fernseh&quot;@en&#xA;&quot;Remote camera&quot;@en&#xA;&quot;Professional video camera&quot;@en&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The above result is just an excerpt.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="5073" LastActivityDate="2014-11-18T01:26:51.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2489" PostTypeId="1" CreationDate="2014-11-18T09:07:00.867" Score="4" ViewCount="137" Body="&lt;p&gt;The questionnaire for the data is &lt;a href=&quot;http://www.cc.gatech.edu/gvu/user_surveys/survey-1997-10/questions/general.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question takes multiple entry for the same question, I want to reduce this to a single variable. How do I do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The clean data is available &lt;a href=&quot;http://wikisend.com/download/586046/DataRaw.arff&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&#xA;NB: The Column CompuPlat has missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;part of dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;CMFam CMHobb  CMNone  CMOther CMPol   CMProf  CMRel&#xA;0   0   1   0   0   0   0&#xA;0   0   0   0   0   0   0&#xA;1   1   0   0   0   1   0&#xA;0   0   0   1   0   0   0&#xA;0   0   0   0   1   1   0&#xA;1   0   0   0   0   1   1&#xA;Community Membership_Family&#xA;Community Membership_Hobbies&#xA;Community Membership_None&#xA;Community Membership_Other&#xA;Community Membership_Political&#xA;Community Membership_Professional&#xA;Community Membership_Religious&#xA;Community Membership_Support&#xA;&lt;/code&gt;&#xA;I want to club all of them in a variable CM&lt;/p&gt;&#xA;" OwnerUserId="5075" LastEditorUserId="8953" LastEditDate="2015-05-19T14:14:42.537" LastActivityDate="2015-05-19T14:14:42.537" Title="Reduction of multiple answers to single variable" Tags="|dataset|dimensionality-reduction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2490" PostTypeId="2" ParentId="2463" CreationDate="2014-11-18T09:48:19.333" Score="1" Body="&lt;p&gt;Your dataset can be viewed as a directed graph. The party's location (latitude and longitude) can be denoted as a node and the directed edge can denote who referred whom. Once the dataset can be viewed as this, the problem boils down to joining co-ordinates with lines. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-18T09:48:19.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2491" PostTypeId="1" AcceptedAnswerId="2656" CreationDate="2014-11-18T09:56:06.157" Score="1" ViewCount="14913" Body="&lt;p&gt;Data sample contains a single feature: random integer number from 1 to 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possble to change &lt;code&gt;1,2,3,4&lt;/code&gt; representation on the filter card to some custom names, say: &lt;code&gt;Type1,Type2,Type3,Type4&lt;/code&gt;? (not changing data set)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4ZPYM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-11-18T10:02:47.110" LastActivityDate="2014-12-09T08:30:28.000" Title="Change aliases of filter items in Tableau" Tags="|visualization|tableau|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2492" PostTypeId="2" ParentId="2489" CreationDate="2014-11-18T10:19:59.510" Score="2" Body="&lt;p&gt;The variable represents the answer to the first question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One straightforward way is to allow for all possible categories in this variable. For example, if there are 5 options in this answer, you will have to treat it as a categorical variable with 2^5 = 32 categories. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the number of categories increase exponentially with the number of options (check boxes) provided for the answer. In that case, it might be better to restrict the number of categories to, for example, 5. This can be done by leaving the top 4 choices/ options (by count) as they are and treating every other choice as &quot;other&quot;.&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2014-11-18T10:32:32.317" LastActivityDate="2014-11-18T10:32:32.317" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="2493" PostTypeId="1" CreationDate="2014-11-18T11:04:28.837" Score="0" ViewCount="265" Body="&lt;p&gt;Currently we are regularly analyzing sets of paragraphs every month. I would like to automate this and split each paragraphs into chunks of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this I would like to employ a neural network. However, I am not really very familiar with creating neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas or starting point on how to do this using Neuroph or maybe in other framework/approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit for more info as suggested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have very little experience on neural networks though I have some introduction with it in college. However I am very much familar with java&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is around 3 megabytes only and consists of rules and relationships for a single domain. This means that the data is complex but relatively limited though still free-form English language.&lt;/p&gt;&#xA;" OwnerUserId="5077" LastEditorUserId="5077" LastEditDate="2014-11-19T11:34:03.507" LastActivityDate="2014-11-19T11:34:03.507" Title="Analyze paragraphs using Neuroph" Tags="|text-mining|clustering|neural-network|java|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2494" PostTypeId="1" CreationDate="2014-11-18T12:41:54.973" Score="3" ViewCount="467" Body="&lt;p&gt;I have 1-4 gram text data from wikipedia for 14 categories, which I am using for NE classification.&#xA;I feed named entity from sentence to lucene indexer which searches named entity from these 14 categories. &#xA;Issue I am facing is, for single entity I get multiple classes as a result with same score.&#xA;like while search &lt;code&gt;titanic&lt;/code&gt;, indexer gives this result&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Score    - 11.23&#xA;Title    - titanic&#xA;Category - Book&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Score    - 11.23&#xA;Title    - titanic&#xA;Category - Movie&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Score    - 11.23&#xA;Title    - titanic&#xA;Category - Product&lt;/p&gt;&#xA;&#xA;&lt;p&gt;now problem is which class to be considered?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already tried with classifiers (NB,ME in nltk,scikit learn), but as it consider each entity from dataset as feature, it works as indexer only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why lucene?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Tz8Uy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5079" LastActivityDate="2015-05-04T20:25:06.967" Title="What is the best practice to classify category of named entity in sentence" Tags="|machine-learning|data-mining|classification|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2495" PostTypeId="1" AcceptedAnswerId="2511" CreationDate="2014-11-18T14:03:26.760" Score="3" ViewCount="755" Body="&lt;p&gt;I am running SVM algorithm in R.It is taking long time to run the algorithm.I have system with 32GB RAM.How can I use that whole RAM memory to speed my process.&lt;/p&gt;&#xA;" OwnerUserId="3551" LastActivityDate="2016-02-24T15:33:03.407" Title="How to run R programs on multicore using doParallel package?" Tags="|r|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2496" PostTypeId="2" ParentId="2486" CreationDate="2014-11-18T19:52:54.050" Score="3" Body="&lt;p&gt;Does it matter that the model is created in the form of SVM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If no, I have seen a clever 6-D visualization. Its varieties are becoming popular in medical presentations. &#xA;3 dimensions are shown as usual, in orthographic projection.&#xA;Dimension 4 is color (0..255)&#xA;Dimension 5 is thickness of the symbol &#xA;Dimension 6 requires animation. It is a frequency of vibration of a dot on the screen. &#xA;In static, printed versions, one can replace frequency of vibration by blur around the point, for a comparable visual perception.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, and you specifically need to draw separating hyperplanes, and make them look like lines\planes, the previous trick will not produce good results. Multiple 3-D images are better.&lt;/p&gt;&#xA;" OwnerUserId="5083" LastActivityDate="2014-11-18T19:52:54.050" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2499" PostTypeId="1" CreationDate="2014-11-19T01:07:10.283" Score="3" ViewCount="1022" Body="&lt;p&gt;I'm looking for an (ideally free) API that would have time series avg/median housing prices by zip code or city/state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quandl almost fits the bill, but it returns inconsistent results across different zip codes and the data is not as up to date as I'd like (it's mid November, and the last month is August).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also looked at Zillow, but storing their data is against TOS, and at 1,000 calls daily--it would take forever to pull in the necessary data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions (even if they aren't free) would be much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="5086" LastActivityDate="2014-12-01T00:26:39.927" Title="API for historical housing prices" Tags="|dataset|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2500" PostTypeId="1" CreationDate="2014-11-19T09:26:58.700" Score="3" ViewCount="1145" Body="&lt;p&gt;I have a list user data: user name, age, sex, address, location etc., and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a set of product data: Product name, Cost, description etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to build a recommendation engine that will be able to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Figure out similar products&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;name  :   category   :  cost    :   ingredients&lt;/p&gt;&#xA;&#xA;&lt;p&gt;x     :     x1   :        15  :       xx1, xx2, xx3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y     :    y1   :        14   :     yy1, yy2, yy3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;z     :    x1  :          12   :     xx1, xy1 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here x and z are similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Recommend relevant products from the product list to a user&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I implement this using mahout?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2014-11-19T12:46:27.353" LastActivityDate="2018-12-21T22:38:05.487" Title="Recommendation engine with mahout" Tags="|machine-learning|data-mining|recommender-system|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2501" PostTypeId="1" AcceptedAnswerId="5626" CreationDate="2014-11-19T12:38:05.597" Score="5" ViewCount="209" Body="&lt;p&gt;I do at the moment some data experiments with the &lt;a href=&quot;http://graphlab.com/products/create/docs/&quot; rel=&quot;nofollow&quot;&gt;Graphlab toolkit&lt;/a&gt;. I have at the first next SFrame, with the three columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Users Items Rating&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The pair in the same row from every &lt;code&gt;Users&lt;/code&gt; and &lt;code&gt;Items&lt;/code&gt; values build the unique key and the &lt;code&gt;Rating&lt;/code&gt; is the corresponded float value. These values are not normalised. First of all, I do someself next normalisation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Division of every rating value of specific user by the rating maximum from this user (scale between 0 and 1)&lt;/li&gt;&#xA;&lt;li&gt;Take the logarithm by every rating value&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Afterward I create a recommender model and evaluate the basic metrics for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this topic I invite everybody to discuss another interesting normalisation methods. If anybody could tell some good method for data preparation, it would be great. The results could be evaluated because of the metrics and I can publish it here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is comming from some music site, the users rated some tracks. I have approximately 100 000 users and 300 000 tracks. Total number of ratings is over 3 millions (actually the matrix is sparse). This is the most simple data set, which I analyze now. In the future I can (and will) use some additional information about the users and tracks (f.e. duration, year, genre, band etc). At the moment I just interest to collect some methods for rating normalisation without to use additional information (users &amp;amp; items features). My problem is, the data set doesn't have any &lt;code&gt;Rating&lt;/code&gt; at the first. I create someself the column &lt;code&gt;Rating&lt;/code&gt;, based on the number of events for unique &lt;code&gt;User-Item&lt;/code&gt; pair (I have this information). You can of course understand that some users can hear some tracks many times, and another users only one time. Consequently the dispersion is very high and I want to reduce it (normalise the ratings value).&lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="3281" LastEditDate="2014-11-20T14:56:14.813" LastActivityDate="2015-04-27T11:58:50.443" Title="Data scheduling for recommender" Tags="|recommender-system|data-cleaning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2502" PostTypeId="1" CreationDate="2014-11-19T19:25:02.770" Score="4" ViewCount="1360" Body="&lt;p&gt;I'm working through the &lt;a href=&quot;https://www.coursera.org/course/nlp&quot; rel=&quot;nofollow&quot;&gt;Coursera NLP course by Jurafsky &amp;amp; Manning&lt;/a&gt;, and the &lt;a href=&quot;https://class.coursera.org/nlp/lecture/32&quot; rel=&quot;nofollow&quot;&gt;lecture on Good-Turing smoothing&lt;/a&gt; struck me odd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example given was:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You are fishing (a scenario from Josh Goodman), and caught:&lt;br&gt;&#xA;  10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish&lt;br&gt;&#xA;  ...&lt;br&gt;&#xA;  How likely is it that the next species is new (i.e. catfish or bass)&lt;br&gt;&#xA;  Let's use our estimate of things-we-saw-once to estimate the new things.&lt;br&gt;&#xA;  3/18 (because N_1=3)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I get the intuition of using the count of uniquely seen items to estimate the number of unseen item types (N = 3), but the next steps seem counterintuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the denominator left unchanged instead of incremented by the estimate of unseen item types? I.e., I would expect the probabilities to become:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Carp : 10 / 21&lt;br&gt;&#xA;  Perch : 3 / 21&lt;br&gt;&#xA;  Whitefish : 2 / 21&lt;br&gt;&#xA;  Trout : 1 / 21&lt;br&gt;&#xA;  Salmon : 1 / 21&lt;br&gt;&#xA;  Eel : 1 / 21&lt;br&gt;&#xA;  Something new : 3 / 21&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It seems like the Good-Turing count penalizes seen items too much (trout, salmon, &amp;amp; eel are each taken down to 1/27); coupled with the need to adjust the formula for gaps in the counts (e.g., Perch &amp;amp; Carp would be zeroed out otherwise), it just feels like a bad hack.&lt;/p&gt;&#xA;" OwnerUserId="5095" LastActivityDate="2014-11-21T10:44:06.783" Title="Good-Turing Smoothing Intuition" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2503" PostTypeId="1" CreationDate="2014-11-20T05:38:27.303" Score="2" ViewCount="343" Body="&lt;p&gt;I am trying to build an item-item similarity matching recommendation engine with mahout. The data set is as in the following format ( attributes are in text not in numerals format )&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name : category : cost : ingredients&#xA;&#xA;x : xx1 : 15 : xxx1, xxx2, xxx3&#xA;&#xA;y : yy1 : 14 : yyy1, yyy2, yyy3&#xA;&#xA;z : xx1 : 12 : xxx1, xxy1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So in-order to use this data set for mahout to train, what is the right way to convert this in to numeric (as CSV Boolean data set) format accepted by mahout.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="1330" LastEditDate="2022-04-18T18:15:12.270" LastActivityDate="2022-04-18T18:15:12.270" Title="Creating Data model for mahout recommendation engine" Tags="|machine-learning|dataset|data-mining|recommender-system|apache-mahout|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2504" PostTypeId="1" AcceptedAnswerId="5152" CreationDate="2014-11-20T06:49:00.357" Score="51" ViewCount="44138" Body="&lt;p&gt;I have a big data problem with a large dataset (take for example 50 million rows and 200 columns). The dataset consists of about 100 numerical columns and 100 categorical columns and a response column that represents a binary class problem. The cardinality of each of the categorical columns is less than 50. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know a priori whether I should go for deep learning methods or ensemble tree based methods (for example gradient boosting, adaboost, or random forests). Are there some exploratory data analysis or some other techniques that can help me decide for one method over the other? &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2020-08-20T18:33:44.403" Title="Deep Learning vs gradient boosting: When to use what?" Tags="|machine-learning|classification|deep-learning|" AnswerCount="7" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="2505" PostTypeId="2" ParentId="2486" CreationDate="2014-11-20T09:33:20.320" Score="3" Body="&lt;p&gt;Have you looked into tourr package in R. This package does hyperplane reduction. In addition it has an optimizer that tries to find the best reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a very nice video in &lt;a href=&quot;https://www.youtube.com/watch?v=iSXNfZESR5I&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=iSXNfZESR5I&lt;/a&gt; That shows what  R is capable even beyound tourr package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I refer you to &lt;a href=&quot;https://stackoverflow.com/questions/8017427/plotting-data-from-an-svm-fit-hyperplane&quot;&gt;https://stackoverflow.com/questions/8017427/plotting-data-from-an-svm-fit-hyperplane&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5100" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-11-20T09:33:20.320" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2506" PostTypeId="2" ParentId="2463" CreationDate="2014-11-20T13:07:45.903" Score="0" Body="&lt;p&gt;Just go through Neo4j (graph data base and will be useful for social network analysis) also.. may be helpful &lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2014-11-20T13:07:45.903" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2507" PostTypeId="1" CreationDate="2014-11-20T14:26:10.463" Score="10" ViewCount="3607" Body="&lt;p&gt;I'm trying to build a data set on several log files of one of our products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The different log files have their own layout and own content; I successfully grouped them together, only one step remaining...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, the log &quot;messages&quot; are the best information. I don't have the comprehensive list of all those messages, and it's a bad idea to hard code based on those because that list can change every day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is to separate the indentification text from the value text (for example: &quot;Loaded file XXX&quot; becomes (identification: &quot;Loaded file&quot;, value: &quot;XXX&quot;)). Unfortunately, this example is simple, and in real world there are different layouts and sometimes multiple values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about using string kernels, but it is intended for clustering ... and cluseting is not applicable here (I don't know the number of different types of messages and eventhough, it would be too much).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: For those who programs, this can be easier to understand. Let's say that the code contains as logs printf(&quot;blabla %s&quot;, &quot;xxx&quot;) -&gt; I would like to have &quot;blabla&quot; and &quot;xxx&quot; seperatated&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2014-12-29T08:38:44.673" Title="Log file analysis: extracting information part from value part" Tags="|text-mining|clustering|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2508" PostTypeId="1" AcceptedAnswerId="2519" CreationDate="2014-11-20T15:02:17.937" Score="2" ViewCount="1164" Body="&lt;p&gt;I would like to run an R script using a single command (e.g. bat file or shortcut).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This R script asks the user to choose a file and then plots information about that file. All is done via dialog boxes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't want the user to go inside R - because they don't know it at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I was using r cmd and other similar stuffs, but as soon as the plots are displayed, R exits and closes the plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2014-11-21T21:39:29.617" Title="How to run R scripts without closing X11" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2509" PostTypeId="2" ParentId="2507" CreationDate="2014-11-20T16:28:48.343" Score="3" Body="&lt;p&gt;How about considering each string as a process trace and applying alpha-algorithm?&#xA;That would give you a graph and nodes with a big number out-edges will most likely point to values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can mark these nodes and for every new string parse/traverse the graph until you reach those areas.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2014-11-20T16:28:48.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2510" PostTypeId="1" CreationDate="2014-11-20T17:54:11.990" Score="1" ViewCount="1303" Body="&lt;p&gt;I'm currently finishing up a B.S. in mathematics and would like to attend graduate school (a master's degree for starters, with the possibility of a subsequent Ph.D.) with an eye toward entering the field of data science. I'm also particularly interested in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the graduate degree choices that would get me to where I want to go?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a consensus as to whether a graduate degree in applied mathematics, statistics, or computer science would put me in a better position to enter the field of data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you all for the help, this is a big choice for me and any input is very much appreciated. Typically I ask my questions on Mathematics Stack Exchange, but I thought asking here would give me a broader and better rounded perspective.&lt;/p&gt;&#xA;" OwnerDisplayName="user5007" LastActivityDate="2016-07-28T17:08:46.557" Title="Graduate Degree Choices for Data Science" Tags="|career|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2511" PostTypeId="2" ParentId="2495" CreationDate="2014-11-20T21:11:12.187" Score="1" Body="&lt;p&gt;I would add a comment but I do not have enough reputation points. I might suggest Using &quot;R revolution open&quot;. It is a Build of R that includes a lot of native support for multi-core processing. I have not used it much as my computer is very old, but it is defiantly worth looking at. Plus it is free. &lt;/p&gt;&#xA;" OwnerUserId="5023" LastActivityDate="2014-11-20T21:11:12.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2512" PostTypeId="2" ParentId="2510" CreationDate="2014-11-21T10:24:18.623" Score="2" Body="&lt;p&gt;UCL - CSML. It covers computer science, machine learning and statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, reputation of the university.&#xA;Secondly, you are from Mathematics background, hence I assume you don't have sufficient programming knowledge.&#xA;Thirdly, Statistics and Machine Learning dominates this field. Employers would prefer these 2 before Mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, this course provides everything that you are lacking. HOWEVER, they don't teach programming languages like Java, C++,... but Matlab, R, and Mathematica. Hence, it would be essential if you pick up the former from somewhere.&lt;/p&gt;&#xA;" OwnerUserId="5110" LastEditorUserId="5110" LastEditDate="2014-11-21T10:47:20.207" LastActivityDate="2014-11-21T10:47:20.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2513" PostTypeId="1" AcceptedAnswerId="5046" CreationDate="2014-11-21T10:34:31.673" Score="7" ViewCount="199" Body="&lt;p&gt;From &lt;a href=&quot;http://nshorter.com/ResearchPapers/MachineLearning/A_Roadmap_to_SVM_SMO.pdf&quot; rel=&quot;noreferrer&quot;&gt;A_Roadmap_to_SVM_SMO.pdf&lt;/a&gt;, pg 12.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Hzn23.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Hzn23.png&quot; alt=&quot;a busy cat&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://s13.postimg.org/9dx9t4w47/whatwhat.png&quot; rel=&quot;noreferrer&quot;&gt;postimg.org&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;&lt;p&gt;Assume I am using linear kernel, how will I be able to get both the first and second inner product?&lt;/p&gt;&#xA;&lt;p&gt;My guess, inner product of datapoint with datapoint j labelled class A for the first inner product of the equation and inner product of datapoint j with datapoints labelled class B for second inner product?&lt;/p&gt;&#xA;" OwnerUserId="5110" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-02-16T16:57:28.567" Title="Please enlighten me with Platt's SMO algorithm (for SVM)" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2514" PostTypeId="2" ParentId="2502" CreationDate="2014-11-21T10:44:06.783" Score="3" Body="&lt;p&gt;There are no &lt;em&gt;unseen&lt;/em&gt; item types in the given data, by definition. 3 is the count of items seen &lt;em&gt;once&lt;/em&gt;, and they are already included in the denominator 18. If the next item were &lt;em&gt;previously&lt;/em&gt; unseen, it would become seen &lt;em&gt;once&lt;/em&gt; when it appears. Since 3-of-18 examples were seen-once items, this is an estimate of the probability that the next item will be seen-once too on its first appearance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is certainly a heuristic. There is no way to know whether there are 0 or 1000 other types out there.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-11-21T10:44:06.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2515" PostTypeId="2" ParentId="1246" CreationDate="2014-11-21T11:50:47.717" Score="10" Body="&lt;p&gt;First of all, word &quot;sample&quot; is normally used to describe &lt;a href=&quot;http://en.wikipedia.org/wiki/Sample_%28statistics%29&quot;&gt;&lt;em&gt;subset&lt;/em&gt; of population&lt;/a&gt;, so I will refer to the same thing as &quot;example&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your SGD implementation is slow because of this line: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for each training example i:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here you explicitly use exactly one example for each update of model parameters. By definition, vectorization is a technique for converting operations on one element into operations on a vector of such elements. Thus, no, you cannot process examples one by one and still use vectorization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can, however, approximate true SGD by using &lt;strong&gt;mini-batches&lt;/strong&gt;. Mini-batch is a small subset of original dataset (say, 100 examples). You calculate error and parameter updates based on mini-batches, but you still iterate over many of them without global optimization, making the process stochastic. So, to make your implementation much faster it's enough to change previous line to: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batches = split dataset into mini-batches&#xA;for batch in batches: &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and calculate error from batch, not from a single example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though pretty obvious, I should also mention vectorization on per-example level. That is, instead of something like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta = np.array([...])  # parameter vector&#xA;x = np.array([...])      # example&#xA;y = 0                    # predicted response&#xA;for i in range(len(example)):&#xA;    y += x[i] * theta[i]&#xA;error = (true_y - y) ** 2  # true_y - true value of response&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you should definitely do something like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;error = (true_y - sum(np.dot(x, theta))) ** 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which, again, easy to generalize for mini-batches:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;true_y = np.array([...])     # vector of response values&#xA;X = np.array([[...], [...]]) # mini-batch&#xA;errors = true_y - sum(np.dot(X, theta), 1)&#xA;error = sum(e ** 2 for e in errors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2014-11-21T11:50:47.717" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2516" PostTypeId="1" CreationDate="2014-11-21T15:23:37.360" Score="5" ViewCount="81" Body="&lt;p&gt;To all:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have nominal multivariant timeseries that look like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Time Var1 Var2 Var3 Var4 Var5 ... VarN&#xA;             0     A     A   B    C    A   ... H&#xA;             1     A     A   B    D    D   ... H&#xA;             2     B     A   C    D    D   ... H&#xA;             ..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And so on from times 0 to 1,000,000. What I would like to do is search the time series for rules of the type:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given Var3 is in state B in the previous step and Var5 is in state D in the previous step, than Var1 will be in state B. What I want to do is have the rules that include the time interval explicitly. A simpler case of interest would simply be to reduce the time series to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               Time    Var1 Var2 Var3 Var4 Var5 ... VarN&#xA;                0        0    0    0     0   0   ... 0&#xA;                1        0    0    0     1   1   ... 0&#xA;                2        1    0    1     0   0   ... 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where the the variable is 1 if its state is different from the previous step and zero otherwise. Then I just want to have rules that say something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If Var4 and Var5 changed in the previous step than Var1 will change in the current step. Which would be easy for a lag of one, as I could just make the data into something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Var1 Var2 Var3 Var4 Var5 ... VarN Var1_t-1 Var2_t-1 Var3_t-1 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then do sequence mining, but if I want to have rules that aren't just a single lag but could be lags from 1 to 500 than my data set begins to be a little difficult to work with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit to respond to comment:&#xA;Each column could be in one of 7 different states. As far as a target, it is non-specific, any rules between the columns would be of interest. However, predicting columns 30-40 and 62-75 would be particularly interesting.&lt;/p&gt;&#xA;" OwnerUserId="5113" LastEditorUserId="5134" LastEditDate="2014-11-24T13:47:38.860" LastActivityDate="2014-12-25T09:27:16.340" Title="Relation mining of multivariant categorical timeseries without excluding the temporal nature" Tags="|data-mining|statistics|text-mining|time-series|categorical-data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2517" PostTypeId="2" ParentId="2486" CreationDate="2014-11-21T18:55:07.860" Score="2" Body="&lt;p&gt;Dimension reduction (like PCA) is an excellent way to visualize the results of classification on a high-dimensional feature space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest approach is to project the features to some low-d (usually 2-d) space and plot them. Then either project the decision boundary onto the space and plot it as well, or simply color/label the points according to their predicted class. You can even use, say, shape to represent ground-truth class, and color to represent predicted class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is true for any categorical classifier, but here's an SVM-specific example: &lt;a href=&quot;http://www.ece.umn.edu/users/cherkass/predictive_learning/Resources/Visualization%20and%20Interpretation%20of%20SVM%20Classifiers.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ece.umn.edu/users/cherkass/predictive_learning/Resources/Visualization%20and%20Interpretation%20of%20SVM%20Classifiers.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, see figures 1a and 2a.&lt;/p&gt;&#xA;" OwnerUserId="1156" LastActivityDate="2014-11-21T18:55:07.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2518" PostTypeId="2" ParentId="2500" CreationDate="2014-11-21T21:33:06.777" Score="1" Body="&lt;p&gt;Try using the &lt;strong&gt;item-based similarity&lt;/strong&gt; algorithm available under &lt;em&gt;Apache Mahout&lt;/em&gt;. It is easy to implement and you will have a good sense how the recommendation system for your data set will work. You could provide ingredients and category as the major inputs to get the similar products.As a neophyte to this field, I would say that this approach is an easy way for all the neophytes to get a good heads up of what kind of a result one can expect from building a recommendation system of their own.   &lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2014-11-21T21:33:06.777" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2519" PostTypeId="2" ParentId="2508" CreationDate="2014-11-21T21:39:29.617" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/24220676/r-script-using-x11-window-only-opens-for-a-second&quot;&gt;This&lt;/a&gt; looks like a similar kind of a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solutions: (taken from above source)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Just sleep via  Sys.sleep(10) which would wait ten seconds.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Wait for user input via readLines(stdin()) or something like that [untested]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the tcltk package which comes with R and is available on all platforms to pop up a window the user has to click to make the click disappear. That solution has been posted a few times over the years on r-help.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;2nd option is better to use for the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Since I did not come up with the answer myself, I tried to put it in comment but my reputation is too low for that.&lt;/p&gt;&#xA;" OwnerUserId="4622" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-11-21T21:39:29.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2520" PostTypeId="2" ParentId="2510" CreationDate="2014-11-22T08:10:40.480" Score="3" Body="&lt;p&gt;Every field has their own variation of &quot;data science,&quot; so I would suggest choosing a subject that interest you and going from there.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't offer what the go to subject is for your particular interest.  A graduate degree that would &quot;get you where you want to go&quot; is quite a personal understanding, so I can' answer that.  But what I will say is, from my own personal experience, when I graduated with my undergrad degree in economics, I was really interested in data science, and economics allowed me to use data science in a field I'm really interested in.  So I applied to Ph.D programs to further my knowledge and am using data science extensively in many different forms of analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion is to apply to graduate degrees that have interesting subject matter to you and will allow you to use data science as understanding.  You would fit well in an economics degree because of your background :)&lt;/p&gt;&#xA;" OwnerUserId="4697" LastActivityDate="2014-11-22T08:10:40.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2521" PostTypeId="1" AcceptedAnswerId="2526" CreationDate="2014-11-22T11:54:54.780" Score="2" ViewCount="269" Body="&lt;p&gt;I browsed a sample for available data at &lt;a href=&quot;http://dbpedia.org/page/Sachin_Tendulkar&quot; rel=&quot;nofollow&quot;&gt;http://dbpedia.org/page/Sachin_Tendulkar&lt;/a&gt;. I wanted these properties as columns, so I downloaded the CSV files from &lt;a href=&quot;http://wiki.dbpedia.org/DBpediaAsTables&quot; rel=&quot;nofollow&quot;&gt;http://wiki.dbpedia.org/DBpediaAsTables&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, when I browse the data for the same entity &quot;Sachin_Tendulkar&quot;, I find that many of the properties are not available. e.g. the property &quot;dbpprop:bestBowling&quot; is not present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I get all the properties that I can browse through the direct resource page.&lt;/p&gt;&#xA;" OwnerUserId="5126" LastActivityDate="2014-11-23T02:27:31.947" Title="DBPedia as Table not having all the properties" Tags="|dataset|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2522" PostTypeId="2" ParentId="2510" CreationDate="2014-11-22T16:08:49.147" Score="4" Body="&lt;p&gt;Why not do an MSc in ooh... &lt;strong&gt;Data Science&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote a quick review of &lt;a href=&quot;http://barry.rowlingson.com/blog/uk-data-science-masters.html&quot; rel=&quot;noreferrer&quot;&gt;UK Data Science Masters&lt;/a&gt;' offerings recently. That should help you get an idea what is offered. Mostly they are mashups of stats and computing, but there are specialisms (health, finance for example) that might interest you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that list was compiled for courses that have already started, so some of those courses might not be available for starting next October, or have different syllabus contents.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2014-11-22T18:16:40.737" LastActivityDate="2014-11-22T18:16:40.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2523" PostTypeId="1" CreationDate="2014-11-23T01:05:06.473" Score="11" ViewCount="12259" Body="&lt;p&gt;Common model validation statistics like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;noreferrer&quot;&gt;Kolmogorov–Smirnov test&lt;/a&gt; (KS), &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;noreferrer&quot;&gt;AUROC&lt;/a&gt;, and &lt;a href=&quot;https://en.wikipedia.org/wiki/Gini_coefficient&quot; rel=&quot;noreferrer&quot;&gt;Gini coefficient&lt;/a&gt; are all functionally related. However, my question has to do with proving how these are all related. I am curious if anyone can help me prove these relationships. I haven't been able to find anything online, but I am just genuinely interested how the proofs work. For example, I know Gini=2AUROC-1, but my best proof involves pointing at a graph. I am interested in formal proofs. Any help would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="5132" LastEditorUserId="9146" LastEditDate="2015-04-18T15:49:57.947" LastActivityDate="2018-11-25T05:15:13.307" Title="Relationship between KS, AUROC, and Gini" Tags="|data-mining|statistics|predictive-modeling|accuracy|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2524" PostTypeId="2" ParentId="2521" CreationDate="2014-11-23T02:26:42.470" Score="0" Body="&lt;p&gt;The official answer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date: Sun, 23 Nov 2014 03:08:19 +0100&#xA;From: Petar Ristoski &amp;lt;petar.ristoski@informatik.uni-mannheim.de&amp;gt;&#xA;To: 'Barry Carter' &amp;lt;carter.barry@gmail.com&amp;gt;&#xA;Subject: RE: CSV tables don't have all properties?&#xA;&#xA;Hi Carter,&#xA;&#xA;The question was already answered on the DBpedia mailing list, but I will&#xA;try to clarify it again. On the DBpedia as Tables web page says that &quot;For&#xA;each class in the DBpedia ontology (such as Person, Radio Station, Ice&#xA;Hockey Player, or Band) we provide a single CSV/JSON file which contains all&#xA;instances of this class. Each instance is described by its URI, an English&#xA;label and a short abstract, the MAPPING-BASED INFOBOX data describing the&#xA;instance (extracted from the English edition of Wikipedia), geo-coordinates,&#xA;and external links.&quot; As you can see we only provide the mapping-based&#xA;infobox properties (dbpedia-owl namespace), while the properties from the&#xA;dbpprop (raw infobox properties) namespace are completely ignored.&#xA;Therefore, dbpprop:bestBowling is missing from the file.&#xA;&#xA;Also, there is a section &quot;Generating your own Custom Tables&quot; [1], where we&#xA;explain how to generate your own tables that will contain the properties you&#xA;need.&#xA;&#xA;Regards,&#xA;&#xA;Petar&#xA;&#xA;[1] http://wiki.dbpedia.org/DBpediaAsTables#h347-4 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user4710" LastActivityDate="2014-11-23T02:26:42.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2525" PostTypeId="1" AcceptedAnswerId="2589" CreationDate="2014-11-23T02:27:10.670" Score="4" ViewCount="492" Body="&lt;p&gt;I am having some difficulty in seeing connection between PCA on second order moment matrix in estimating parameters of Gaussian Mixture Models. Can anyone connect the above??&lt;/p&gt;&#xA;" OwnerUserId="4686" LastActivityDate="2014-12-03T13:55:16.150" Title="Can some one explain how PCA is relevant in extracting parameters of Gaussian Mixture Models" Tags="|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2526" PostTypeId="2" ParentId="2521" CreationDate="2014-11-23T02:27:31.947" Score="0" Body="&lt;p&gt;The question was already answered on the DBpedia-discussion mailing list, by Daniel:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Hi Abhay,&lt;/p&gt;&#xA;&lt;p&gt;the DBpediaAsTables dataset only contains the properties in the&#xA;dbpedia-owl namespace (mapping-based infobox data) and not those from&#xA;the dbpprop (raw infobox properties) namespace (regarding the&#xA;differences see [1]).&lt;/p&gt;&#xA;&lt;p&gt;However, as you are only interested in the data about specific&#xA;entities, take a look at the CSV link at the bottom of the entity's&#xA;description page, e.g., for your example this link is [2].&lt;/p&gt;&#xA;&lt;p&gt;Cheers,&#xA;Daniel&lt;/p&gt;&#xA;&lt;p&gt;[1] wiki.dbpedia.org/Datasets#h434-10&lt;/p&gt;&#xA;&lt;p&gt;[2] dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbpedia.org&amp;amp;query=DESCRIBE+%3Chttp://dbpedia.org/resource/Sachin_Tendulkar%3E&amp;amp;format=text%2Fcsv&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;On the &lt;a href=&quot;http://wiki.dbpedia.org/DBpediaAsTables&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBpediaAsTables web page&lt;/a&gt;, you can find out which datasets were used to generate the tables: instance_types_en, labels, short_abstracts_en, mappingbased_properties_en, geo_coordinates_en.&#xA;Also, I want to clarify that DBpediaAsTables contains all instances from DBpedia 2014, and with &amp;quot;we provide some of the core DBpedia data&amp;quot; we want to say that not all datasets are included in the tables (but only the 5 I stated before)&lt;/p&gt;&#xA;&lt;p&gt;If you want to generate your own tables that will contain custom properties, please refer to the section &lt;a href=&quot;http://wiki.dbpedia.org/DBpediaAsTables#h347-4&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generate your own Custom Tables&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Cheers,&lt;/p&gt;&#xA;&lt;p&gt;Petar&lt;/p&gt;&#xA;" OwnerUserId="5133" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2014-11-23T02:27:31.947" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2527" PostTypeId="1" AcceptedAnswerId="2562" CreationDate="2014-11-23T14:58:34.127" Score="11" ViewCount="502" Body="&lt;p&gt;Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I first use a simple function to determine the frequency of all the four and above character words. I then use this as the feature of each training sample. Now i want my algorithm to be able to cluster the training sets to according to their features, which here is the frequency of each word in the article. (Note that in this example, each article would have its own unique feature since each article has a different feature, for example an article has 10 &quot;water and 23 &quot;pure&quot; and another has 8 &quot;politics&quot; and 14 &quot;leverage&quot;). Can you suggest the best possible clustering algorithm for this example?&lt;/p&gt;&#xA;" OwnerUserId="5138" LastActivityDate="2017-06-08T00:24:37.560" Title="Using Clustering in text processing" Tags="|text-mining|clustering|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2528" PostTypeId="2" ParentId="2527" CreationDate="2014-11-23T17:49:04.487" Score="6" Body="&lt;p&gt;If you want to proceed on your existing path I suggest normalizing each term's frequency by its popularity in the entire corpus, so rare and hence predictive words are promoted. Then use random projections to reduce the dimensionality of these very long vectors down to size so your clustering algorithm will work better (you don't want to cluster in high dimensional spaces).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are other ways of topic modeling. Read &lt;a href=&quot;http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; tutorial to learn more.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="33169" LastEditDate="2017-06-08T00:24:37.560" LastActivityDate="2017-06-08T00:24:37.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2529" PostTypeId="2" ParentId="1227" CreationDate="2014-11-23T23:00:30.007" Score="1" Body="&lt;p&gt;Store the edges (relations) in your server:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(TeamID, playerID)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you want to find common elements just filter all edges where:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TeamID=&quot;TeamA&quot; or TeamID=&quot;TeamB&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(You could use indexes to speed is up, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then group by playerID and check how many items are in each group. The groups with two items belong to both teams and are shared.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2014-11-23T23:00:30.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2530" PostTypeId="2" ParentId="2500" CreationDate="2014-11-24T09:20:47.543" Score="3" Body="&lt;p&gt;I recommend you to take a look to Oryx (&lt;a href=&quot;https://github.com/OryxProject/oryx&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/OryxProject/oryx&lt;/a&gt;). Oryx is based on Apache Mahout (actually one of the creators of Mahout Sean Owen built it) and provides recommendation using collaborative filtering. Oryx is a very practical tool for implementing recommendation. I have used it in several projects: recommending products in retail stores (small businesses), building an e-commerce recommender and user similarity from mobile app interaction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You just have to represent data in the form:&#xA;UserId ItemId Value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where value is a measure (subjective) of the &lt;em&gt;importance&lt;/em&gt; or &lt;em&gt;influence&lt;/em&gt; of the interaction between that user and the item. User and item can be anything actually, and the same procedure can be used for tagging. For example, for recommending songs, finding similar songs and bands, and finding similar users according to their music tastes you can represent data as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UserId SongId NumberOfPlays&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where NumberOfPlays is the amount of times a song has been played by user (in an online music service for example). This exampl was given in Myrrix the predecessor of Oryx. They also show how to recommend tags to questions in StackOverflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The github site is not that well documented but it will be enough to get it running (and working :))&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="64659" LastEditDate="2018-12-21T22:38:05.487" LastActivityDate="2018-12-21T22:38:05.487" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2534" PostTypeId="1" AcceptedAnswerId="2535" CreationDate="2014-11-25T03:08:34.843" Score="3" ViewCount="355" Body="&lt;p&gt;&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD&quot; rel=&quot;nofollow&quot;&gt;https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the description given in the above link, &#xA;the Attribute information specifies &quot;average and covariance over all 'segments', each segment being described by a 12-dimensional timbre vector&quot;. So the covariance matrix should have 12*12 = 144 elements. But why is the number of timbre covariance features only 78 ?&lt;/p&gt;&#xA;" OwnerUserId="4947" LastActivityDate="2014-11-25T03:38:33.600" Title="Confused about description of YearPrediction Dataset" Tags="|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2535" PostTypeId="2" ParentId="2534" CreationDate="2014-11-25T03:38:33.600" Score="1" Body="&lt;p&gt;You are right, the covariance matrix should have n^2 elements. However, since cov_{i,j} = cov_{j,i}, there is no need to have a repeated feature cov_{j,i} if cov_{i,j} is already accounted for. Hence there will be only n*(n+1)/2 = 12*13/2 = 78 unique covariances and thus only 78 unique covariance based features (n of those will be variances). &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-25T03:38:33.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2536" PostTypeId="2" ParentId="2516" CreationDate="2014-11-25T09:17:45.020" Score="2" Body="&lt;p&gt;This problem is one of estimating the lag. Once that is estimated, you could create additional features representing the lagged values and move forward with &quot;sequence mining&quot; as you have already suggested in the question itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each variable, Var_i, you will have to estimate its lag l_i. This lag can be calculated by estimating the order of a Markov chain with seven symbols (you could use either &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow&quot;&gt;BIC&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt; to estimate this order; both would require calculating likelihood of candidate orders and pick the order that maximizes either of these criteria). Once you are done calculating the order of the Markov chain for each of the variables, then you could represent your dataset such that each row will have the current value of Var_i and its preceding values, all the way back to its estimated lag l_i. While this methodology is laborious, it pays rich dividends as its automated and parsimonious way of representing the necessary information. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-11-25T09:17:45.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2537" PostTypeId="1" AcceptedAnswerId="10952" CreationDate="2014-11-25T09:40:20.040" Score="8" ViewCount="228" Body="&lt;p&gt;GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used with dimension sampling at every split of the tree like is common practice with random forests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there some tests that show that dimensional sampling with GBM would decrease its accuracy because of which this is avoided, either in literature form or in practical experience? &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2016-03-30T06:58:54.947" Title="Why isn't dimension sampling used with gradient boosting machines (GBM)?" Tags="|random-forest|accuracy|gbm|ensemble-modeling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2538" PostTypeId="1" CreationDate="2014-11-25T10:30:17.567" Score="5" ViewCount="720" Body="&lt;p&gt;I have historic error of time series.  I want to analyze error series to improve forecast series. Are there any methods to do this?&lt;/p&gt;&#xA;" OwnerUserId="5099" LastActivityDate="2014-12-28T13:16:47.620" Title="Error analysis for better accuracy" Tags="|time-series|forecast|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2540" PostTypeId="2" ParentId="1166" CreationDate="2014-11-26T02:07:06.460" Score="2" Body="&lt;p&gt;I have just finished my Ph.D. and have used some NLP in it. My university didn't offer any NLP courses. So I ended up teaching myself NLP. I used this &lt;a href=&quot;http://www.nltk.org/book_1ed/&quot; rel=&quot;nofollow noreferrer&quot;&gt;book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which serves as a great introduction to NLP using NLTK (Natural Language Tool Kit). It gives a good introduction into programming with Python. So handy if you've never programmed in Python before too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would highly suggest using nltk from nltk.org (sorry can't post more than two links)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book I used is now out of date as NLTK is now on version 3.0, the book mentioned previously is for NLTK 2.x. But the Authors are working on a new version of the book for NLTK 3.X, you can view the unfinished book &lt;a href=&quot;http://www.nltk.org/book/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would highly suggest using NLTK and if you're new to natural language processing. I would highly suggest you try and get yourself a copy of the following book: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://nlp.stanford.edu/fsnlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Foundations of Statistical Natural Language Processing by Manning and Shutze&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though it doesn't contain any code, it servers a great introduction to natural language processing.&lt;/p&gt;&#xA;" OwnerUserId="5170" LastEditorUserId="29575" LastEditDate="2017-12-14T12:37:39.437" LastActivityDate="2017-12-14T12:37:39.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2541" PostTypeId="1" CreationDate="2014-11-26T04:40:17.840" Score="8" ViewCount="287" Body="&lt;p&gt;What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare with precision/recall/F measure of  different algorithms ? Is this the best way for evaluation ? Or is there any other way to compare results of various recommendation algorithms ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2023-01-28T03:46:59.557" Title="Evaluating Recommendation engines" Tags="|machine-learning|data-mining|dataset|statistics|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2543" PostTypeId="1" CreationDate="2014-11-26T08:47:49.650" Score="6" ViewCount="520" Body="&lt;p&gt;I'm from programming background. I'm now learning Analytics. I'm learning concepts from basic statistics to model building like linear regression, logistic regression, time-series analysis, etc.,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As my previous experience is completely on programming, I would like to do some analysis on the data which programmer has.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, Lets have the details below(I'm using SVN repository)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;personname, code check-in date, file checked-in, number of times checkedin, branch, check-in date and time, build version, Number of defects, defect date, file that has defect, build version, defect fix date, defect fix hours, (please feel free to add/remove how many ever variables needed)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I Just need a trigger/ starting point on what can be done with these data. can I bring any insights with this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or can you provide any links that has information about similar type of work done.&lt;/p&gt;&#xA;" OwnerUserId="5172" LastEditorUserId="21" LastEditDate="2017-05-19T16:12:37.980" LastActivityDate="2017-05-19T16:12:37.980" Title="Can Machine Learning be applied in software developement" Tags="|machine-learning|predictive-modeling|software-development|" AnswerCount="5" CommentCount="2" ClosedDate="2017-05-19T23:15:47.017" ContentLicense="CC BY-SA 3.0" />
  <row Id="2544" PostTypeId="2" ParentId="2543" CreationDate="2014-11-26T09:37:43.970" Score="3" Body="&lt;p&gt;Definately - Yes.&#xA;Good question. &#xA;Was thinking about it myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(1) Collect the data&lt;/strong&gt;.&#xA;The first problem you have: gather enough data. &#xA;All the attributes you mentioned (date, name, check-in title/comment, N of deffects etc) are potentially useful - gather as much as possible.&#xA;As soon as you have a big project, a number of developers, many branches, frequent commits and you have started collecting all the data, you a ready to go further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(2) Ask good questions&lt;/strong&gt;.&#xA;The next question you should ask yourself: what effect are you going to measure, estimate and maybe predict.&#xA;Frequency of possible bugs? Tracking inaccurate &quot;committers&quot;? Risky branches? Want to see some groups of users/bugs/commits according to some metrics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(3) Select the model&lt;/strong&gt;.&#xA;As soon as you have the questions formulated, you should follow the general approach in data science - extract needed features in your data, select appropriate model, train you model and test it, apply it. This is too broad process to discuss it this thread, so please use this site to get right answers.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-11-26T09:58:47.527" LastActivityDate="2014-11-26T09:58:47.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2545" PostTypeId="2" ParentId="2543" CreationDate="2014-11-26T09:57:00.047" Score="1" Body="&lt;p&gt;As you are also looking for examples, then github is a good place to check out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a random repository and went to &quot;Graphs&quot; on the right hand side, which opens up &lt;a href=&quot;https://github.com/mbostock/d3/graphs/contributors&quot; rel=&quot;nofollow&quot;&gt;contribution frequency graph&lt;/a&gt;. There's several tabs next to it that display other aspects of a repository and commit history graphically - commits, code frequency, punch card, etc.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2014-11-26T09:57:00.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2546" PostTypeId="2" ParentId="2507" CreationDate="2014-11-26T11:18:13.227" Score="1" Body="&lt;p&gt;If you're simply trying to separate textual and numeric information then there is a solution based on regular expressions or even just string splitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could even do something like finding the first numeric character and split the text in half right before that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regular expressions you can match all numeric characters that follow eachother. The pattern would be &lt;code&gt;([0-9]+)&lt;/code&gt; with a global flag. It would match all the groups of numbers and you can do whatever you with with them afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.regextester.com/index.html&quot; rel=&quot;nofollow&quot;&gt;Regex Tester&lt;/a&gt; is good for playing around with that stuff.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2014-11-26T11:18:13.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2547" PostTypeId="1" CreationDate="2014-11-26T20:01:23.187" Score="3" ViewCount="94" Body="&lt;p&gt;I am trying to determine whether or not we are 90% confident that the mean of a &lt;em&gt;proposed&lt;/em&gt; &lt;em&gt;population&lt;/em&gt; is at least 2 times that of the mean of the &lt;em&gt;incumbant&lt;/em&gt; &lt;em&gt;population&lt;/em&gt; based on samples from each population which is all the data I have right now. Here are the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;incumbantvalues = (7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;proposedvalues =  (17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea if either population is or will be normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ratio of the &lt;em&gt;sample&lt;/em&gt; means does exceed 2.0 but how does that translate to confidence that the proposed population mean will be at least twice that of the mean of the incumbant population with 90% confidence ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can re-sampling (bootstrapping with replacement) help answer this question ?&lt;/p&gt;&#xA;" OwnerUserId="5180" LastActivityDate="2014-11-27T21:58:15.113" Title="Statistical comparison of 2 small data sets for 2X increase in the population mean" Tags="|statistics|sampling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2549" PostTypeId="2" ParentId="2543" CreationDate="2014-11-26T21:20:52.973" Score="3" Body="&lt;p&gt;Without a doubt you can. The key is to have a set of hypotheses (i.e. assumptions \ scenarios that you want to evaluate) and wrangle the data together to prove \ disprove what you thought is true. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few things to watch-out for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Be ready for Disappointments:&lt;/strong&gt; Often times, once you have invested time and energy in building these models, analysts tend to get biased towards publishing results (publication bias). Treat this as an exploration that with a lot of dead-ends and the goal should be to find the ones that are not.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Know your Data:&lt;/strong&gt; You cannot will your data into doing things magically without truly understanding it. Ensure that you know the different attributes (predictors and dependents) very well. Knowing your data well will allow you to cleanse it and think about appropriate models. All models don't work equally well on all data - data that has a lot of categorical variables might require creative solutions like Dimension reduction before it can be modeled.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Know the &quot;Operational&quot; Processes:&lt;/strong&gt; Knowing how things operate within your firm will help you refine the set of hypothesis that you want to test. For e.g. in your scenario above, knowing how developers work with your change management software and what types of administrative setups have been done will help you figure out why the data is coming in the way it is. Some developers might only be focused on certain modules that are more mature than others, might work only on certain shifts and that might limit how many lines of code are checked in, how many bugs are found etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Having said that here are some scenarios you might want to test:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Developer Effectiveness : &#xA;How different developers working on same modules overtime has resulted into increase or decrease in bugs. &#xA;Does more line of code results in more bugs? Maybe this might be an indicator that the programs need to be split further into smaller components&#xA;Folks might be more productive during certain times of day than others - does time of day affect bug introductions?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Module Maturity: &#xA;Which Modules have the most number of issues? &#xA;Are they worked upon by more developers or less?&#xA;Do defects keep aging for a long time before they are fixed?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, these questions will change depending on what you are working on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="5182" LastActivityDate="2014-11-26T21:20:52.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2550" PostTypeId="1" CreationDate="2014-11-27T03:21:11.110" Score="4" ViewCount="1162" Body="&lt;p&gt;I am doing a text classification task(5000 essays evenly distributed by 10 labels). I explored &lt;code&gt;LinearSVC&lt;/code&gt; and got an accuracy of 80%. Now I guess whether accuracy could be raised by using &lt;code&gt;ensemble&lt;/code&gt; classifier with &lt;code&gt;SVM&lt;/code&gt; as base estimator?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I do not know how to employ an &lt;code&gt;ensemble&lt;/code&gt; classifier incorporating all the features? Please note that I do not want to combine the different features directly in a single vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, &lt;strong&gt;My first question:&lt;/strong&gt; in order to improve the current accuracy, is it possible to use &lt;code&gt;ensemble&lt;/code&gt; classifier with &lt;code&gt;svm&lt;/code&gt; as base estimator?&#xA;&lt;strong&gt;My second question&lt;/strong&gt; How to employ an &lt;code&gt;ensemble&lt;/code&gt; classifier incorporating all features?&lt;/p&gt;&#xA;" OwnerUserId="4950" LastActivityDate="2016-07-01T09:27:40.830" Title="How to ensemble classifier incorporating all features in python?" Tags="|machine-learning|python|nlp|scikit-learn|ensemble-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2551" PostTypeId="2" ParentId="2474" CreationDate="2014-11-27T03:27:46.193" Score="0" Body="&lt;p&gt;If these data are available in the actual excel spreadsheet cells (ie, before you export them to the JSON format provided in your question), you can use the following to get them into R:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;highlight the region of interest within excel&lt;/li&gt;&#xA;&lt;li&gt;copy it to the clipboard (eg. Ctrl-C)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;At an R prompt type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d &amp;lt;- read.delim('clipboard')&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The data will now be available as a data.frame in R. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d&#xA;  from response value&#xA;1    4     TRUE    20&#xA;2    8     TRUE    20&#xA;3    9     TRUE    20&#xA;4    3     TRUE    20&#xA;5   14    FALSE    20&#xA;6   15     TRUE    20&#xA;7   17    FALSE    20&#xA;8   13     TRUE    20  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5153" LastActivityDate="2014-11-27T03:27:46.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2552" PostTypeId="2" ParentId="2541" CreationDate="2014-11-27T03:37:08.010" Score="3" Body="&lt;p&gt;The standard way to evaluate a recommendation engine is by using the &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Root-mean-square_deviation&quot; rel=&quot;nofollow&quot;&gt;RMSE (root mean square error)&lt;/a&gt;&lt;/strong&gt; of the predicted values and the ground truth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is almost a SOP that, after finishing developing a recommendation engine, we will evaluate this engine by comparing its RMSE with other famous, common recommendation algorithms like &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Singular_value_decomposition&quot; rel=&quot;nofollow&quot;&gt;SVD&lt;/a&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;tranditional CF&lt;/a&gt;&lt;/strong&gt;, even &lt;a href=&quot;http://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;RBM&lt;/a&gt;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some terms mentioned above do not seem to be related with recommendation, but you can easily find on the internet how these techniques can be used in this topic.&lt;/p&gt;&#xA;" OwnerUserId="5184" LastActivityDate="2014-11-27T03:37:08.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2553" PostTypeId="2" ParentId="2547" CreationDate="2014-11-27T04:23:31.500" Score="2" Body="&lt;p&gt;Yes, in principle, resampling can help answer this question. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;incumbent &amp;lt;- c(7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)&#xA;proposed  &amp;lt;- c(17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)&#xA;&#xA;set.seed(42)&#xA;&#xA;M  &amp;lt;- 2000&#xA;rs &amp;lt;- double(M)&#xA;&#xA;for (i in 1:M) {&#xA;    rs[i] &amp;lt;- mean(sample(proposed, replace=T)) - 2 * mean(sample(incumbent, replace=T))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To make the assessment, you should choose &lt;em&gt;one&lt;/em&gt; (not both) of the following:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. The (two-tailed) 90% confidence interval for the difference in the (weighted) means using Hall's method is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ci.hall &amp;lt;- 2 * (mean(proposed)-2*mean(incumbent)) - rev(quantile(rs,prob=c(0.05, 0.95)))&#xA;names(ci.hall) &amp;lt;- rev(names(ci.hall))&#xA;ci.hall&#xA;&#xA;   5%   95% &#xA;-0.29  2.95 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is appropriate if you have any concern about missing the possibility that mean(proposed) might actually be less than 2 * mean(incumbent).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B. The proportion of resample means &gt;= 0 provides the (one-tailed) estimate that mean(proposed) is at least twice mean(incumbent):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sum(rs&amp;gt;=0)/M&#xA;&#xA;[1] 0.8915&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that the samples are really rather small and resampling estimates&#xA;can be unstable for small n. The same issue applies if you want to assess normality and go with parametric comparisons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can get to, say, n &gt;= 30, the approach described here should be fine.&lt;/p&gt;&#xA;" OwnerUserId="5153" LastEditorUserId="5153" LastEditDate="2014-11-27T05:03:58.203" LastActivityDate="2014-11-27T05:03:58.203" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2555" PostTypeId="2" ParentId="641" CreationDate="2014-11-27T07:21:56.277" Score="3" Body="&lt;p&gt;Check these :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repository of Test Domains for Information Extraction : &lt;a href=&quot;http://www.isi.edu/info-agents/RISE/repository.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.isi.edu/info-agents/RISE/repository.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DBpedia : &lt;a href=&quot;http://wiki.dbpedia.org/Downloads32&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://wiki.dbpedia.org/Downloads32&lt;/a&gt; (&lt;a href=&quot;https://web.archive.org/web/20150415010508/http://wiki.dbpedia.org:80/Downloads32&quot; rel=&quot;nofollow noreferrer&quot;&gt;mirror&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link Updated :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.isi.edu/integration/RISE/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.isi.edu/integration/RISE/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dbpedia/extraction-framework/wiki/The-DBpedia-Data-Set&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dbpedia/extraction-framework/wiki/The-DBpedia-Data-Set&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="843" LastEditDate="2017-05-23T02:57:59.047" LastActivityDate="2017-05-23T02:57:59.047" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2557" PostTypeId="2" ParentId="2547" CreationDate="2014-11-27T21:58:15.113" Score="0" Body="&lt;p&gt;Here is what I programmed within a loop.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;randomly take 10 values (with replacement) from the incumbant sample, determine its mean&lt;/li&gt;&#xA;&lt;li&gt;randomly take 10 values (with replacement) from the proposed sample, determine its mean&lt;/li&gt;&#xA;&lt;li&gt;form the ratio of the above two means and append it to a master list&lt;/li&gt;&#xA;&lt;li&gt;repeat steps 1 thru 3 many times (I chose 1 million)&lt;/li&gt;&#xA;&lt;li&gt;% Confidence=(number of ratios that equal or exceed 2.0/1000000)*100&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Results:&#xA;Exactly 897450 ratios were found to be greater than or equal to 2.0, producing a confidence of 89.745%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion:&#xA;We are less than 90% confident that the proposed population will have a mean at least twice that of the incumbant population.&lt;/p&gt;&#xA;" OwnerUserId="5180" LastActivityDate="2014-11-27T21:58:15.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2558" PostTypeId="1" AcceptedAnswerId="4896" CreationDate="2014-11-28T03:02:44.323" Score="3" ViewCount="66" Body="&lt;p&gt;What are some possible techniques for smoothing proportions across very large categories, in order to take into account the sample size? The application of interest here is to use the proportions as input into a predictive model, but I am wary of using the raw proportions in cases where there is little evidence and I don't want to overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example, where the ID denotes a customer and impressions and clicks are the number of ads shown and clicks the customer has made, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/3oHzQ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2015-01-17T05:20:32.763" Title="Smoothing Proportions :: Massive User Database" Tags="|machine-learning|predictive-modeling|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2559" PostTypeId="2" ParentId="2543" CreationDate="2014-11-28T06:02:59.210" Score="1" Body="&lt;p&gt;Data analysis is always driven by the request. It could be: &quot;I want to find out this, so I need to collect those data first. Then I would use this model to analyze&quot;. If you just want to practice, by reviewing your data set, there is one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Task&lt;/strong&gt;: Which issue affects the &quot;number of check in &quot; most? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data set&lt;/strong&gt;: what you have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;: Correlation (e.g. Spearman, which is nonparametric measure of statistical dependence between two variables)&lt;/p&gt;&#xA;" OwnerUserId="5198" LastActivityDate="2014-11-28T06:02:59.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2561" PostTypeId="2" ParentId="2527" CreationDate="2014-11-28T06:17:34.480" Score="2" Body="&lt;p&gt;Cannot say it is the best one, but Latent Semantic Analysis could be one option. Basically it is based on co-occurrence, you need to weight it first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Latent_semantic_analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://lsa.colorado.edu/papers/dp1.LSAintro.pdf&quot; rel=&quot;nofollow&quot;&gt;http://lsa.colorado.edu/papers/dp1.LSAintro.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that LSA does not have firm statistic support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have fun&lt;/p&gt;&#xA;" OwnerUserId="5198" LastActivityDate="2014-11-28T06:17:34.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2562" PostTypeId="2" ParentId="2527" CreationDate="2014-11-28T08:55:25.813" Score="5" Body="&lt;p&gt;I don't know if you ever read SenseCluster by Ted Pedersen : &lt;a href=&quot;http://senseclusters.sourceforge.net/&quot; rel=&quot;noreferrer&quot;&gt;http://senseclusters.sourceforge.net/&lt;/a&gt;. Very good paper for sense clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, when you analyze words, think that &quot;computer&quot;, &quot;computers&quot;, &quot;computering&quot;, ... represent one concept, so only one feature. Very important for a correct analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To speak about the clustering algorithm, you could use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;noreferrer&quot;&gt;hierarchical clustering&lt;/a&gt;. At each step of the algo, you merge the 2 most similar texts according to their features (using a measure of dissimilarity, euclidean distance for example). With that measure of dissimilarity, you are able to find the best number of clusters and so, the best clustering for your texts and articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck :)&lt;/p&gt;&#xA;" OwnerUserId="5165" LastActivityDate="2014-11-28T08:55:25.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2563" PostTypeId="1" CreationDate="2014-11-28T09:19:32.867" Score="3" ViewCount="91" Body="&lt;p&gt;I have a visualization problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Creating a comparison report of PR event efficiency. Say, show or exhibition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two dimensions of comparison:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;compare vs the same event performance in the past years &lt;/li&gt;&#xA;&lt;li&gt;compare vs another type of analogical/competitive events &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is also a number of comparison aspects:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Audience&lt;/li&gt;&#xA;&lt;li&gt;Media Coverage&lt;/li&gt;&#xA;&lt;li&gt;Social Buzz&lt;/li&gt;&#xA;&lt;li&gt;ROI&lt;/li&gt;&#xA;&lt;li&gt;.... etc&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Each aspect is a set of some final KPI-s (just numbers, which can be compared vs another &quot;dimensions&quot;), plus maybe some descriptive text and pictures (which couldn't be a metric but should be attached to the report).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So finaly it looks like a three-dimensional coube:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Years&lt;/li&gt;&#xA;&lt;li&gt;Another Events&lt;/li&gt;&#xA;&lt;li&gt;Aspects&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If I put it in plain Word or PPT it will look like a document with dozen of slides/papers and linear structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to compile an elegant user-friendly report?&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2014-11-30T08:44:27.780" LastActivityDate="2014-11-30T08:44:27.780" Title="Visualization of three-dimensional report" Tags="|marketing|infographics|visualization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2564" PostTypeId="1" CreationDate="2014-11-28T11:39:09.537" Score="5" ViewCount="433" Body="&lt;p&gt;A short while ago, I came across this ML framework that has implemented several different algorithms ready for use. The site also provides a handy API that you can access with an API key.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have need of the framework to solve a website classification problem where I basically need to categorize several thousand websites based on their HTML content. As I don't want to be bound to their existing API, I wanted to use the framework to implement my own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, besides some introductory-level data mining courses and associated reading, I know very little as to what exactly I would need to use. Specifically, I'm at a loss as to what exactly I need to do to train the classifier and then model the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The framework already includes some classification algorithms like NaiveBayes, which I know is well suited to the task of text classification, but I'm not exactly sure how to apply it to the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me a rough guidelines as to what exactly I would need to do to accomplish this task?&lt;/p&gt;&#xA;" OwnerUserId="5199" LastActivityDate="2019-12-31T13:02:20.290" Title="Using the Datumbox Machine Learning Framework for website classification - guidelines?" Tags="|machine-learning|classification|java|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2565" PostTypeId="2" ParentId="806" CreationDate="2014-11-28T12:48:14.443" Score="29" Body="&lt;p&gt;AUC and accuracy are fairly different things. AUC applies to binary classifiers that have some notion of a decision threshold internally. For example logistic regression returns positive/negative depending on whether the logistic function is greater/smaller than a threshold, usually 0.5 by default. When you choose your threshold, you have a classifier. You have to choose one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given choice of threshold, you can compute accuracy, which is the proportion of true positives and negatives in the whole data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUC measures how true positive rate (recall) and false positive rate trade off, so in that sense it is already measuring something else. More importantly, AUC is not a function of threshold. It is an evaluation of the classifier as threshold varies over all possible values. It is in a sense a broader metric, testing the quality of the internal value that the classifier generates and then compares to a threshold. It is not testing the quality of a particular choice of threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUC has a different interpretation, and that is that it's also the probability that a randomly chosen positive example is ranked above a randomly chosen negative example, according to the classifier's internal value for the examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUC is computable even if you have an algorithm that only produces a ranking on examples. AUC is not computable if you truly only have a black-box classifier, and not one with an internal threshold. These would usually dictate which of the two is even available to a problem at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUC is, I think, a more comprehensive measure, although applicable in fewer situations. It's not strictly better than accuracy; it's different. It depends in part on whether you care more about true positives, false negatives, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;F-measure is more like accuracy in the sense that it's a function of a classifier and its threshold setting. But it measures precision vs recall (true positive rate), which is not the same as either above.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-11-28T12:48:14.443" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2566" PostTypeId="2" ParentId="2538" CreationDate="2014-11-28T13:16:02.520" Score="3" Body="&lt;p&gt;NARMAX Methodology and Residual analysis both address this issue. Search for the following articles:(Error = Residual = Noise)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Chaotic Time Series Prediction with residual Analysis Method Using Hybrid Elman–NARX Neural Networks, Muhammad Ardalani-Farsa (2010)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Orthogonal Least Squares Methods and their Application to Non-Linear System Identification, S. Chen, S. A. Billings, W. Luo (1989)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any article working on NARMAX, NARMA and Residual Analysis. Remember in NARX and NAR there is no error estimation and analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Notice in general you can follow this steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimate a time series and calculate Error or Residuals using any .&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider errors or residuals as a new time series. Try to estimate Error-Time-Series. Now you can add this estimations to your initial model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can do this residual analysis as many times as you need. In practice 2 or 3 times suffices. Remember in practice, residual time series are noisy and SNR in this time series is so small. So you should use some Noise-Robust methods for residual analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5200" LastActivityDate="2014-11-28T13:16:02.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2567" PostTypeId="1" CreationDate="2014-11-29T16:05:08.810" Score="11" ViewCount="1328" Body="&lt;p&gt;I want to make a prediction for the result of the parliamentary elections. My output will be the % each party receives. There is more than 2 parties so logistic regression is not a viable option. I could make a separate regression for each party but in that case the results would be in some manner independent from each other. It would not ensure that the sum of the results would be 100%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What regression (or other method) should I use? Is it possible to use this method in R or Python via a specific library?&lt;/p&gt;&#xA;" OwnerUserId="5211" LastEditorUserId="847" LastEditDate="2014-12-03T13:50:12.587" LastActivityDate="2017-08-09T15:35:01.097" Title="What regression to use to calculate the result of election in a multiparty system?" Tags="|classification|r|python|regression|predictive-modeling|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2568" PostTypeId="1" CreationDate="2014-11-30T00:42:28.237" Score="2" ViewCount="1791" Body="&lt;p&gt;I have some text files containing movie reviews I need to find out whether the review is good or bad. I tried the following code but it's not working:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import nltk&#xA;with open(&amp;quot;c:/users/user/desktop/datascience/moviesr/movies-1-32.txt&amp;quot;, 'r') as m11:&#xA;    mov_rev = m11.read()&#xA;mov_review1=nltk.word_tokenize(mov_rev)&#xA;bon=&amp;quot;crap aweful horrible terrible bad bland trite sucks unpleasant boring dull moronic dreadful disgusting distasteful flawed ordinary slow senseless unoriginal weak wacky uninteresting unpretentious &amp;quot;&#xA;bag_of_negative_words=nltk.word_tokenize(bon)&#xA;bop=&amp;quot;Absorbing Big-Budget Brilliant Brutal Charismatic Charming Clever Comical Dazzling Dramatic Enjoyable Entertaining Excellent Exciting  Expensive Fascinating Fast-Moving First-Rate Funny Highly-Charged Hilarious Imaginative Insightful Inspirational Intriguing Juvenile Lasting Legendary Pleasant Powerful Ripping Riveting Romantic Sad  Satirical Sensitive  Sentimental Surprising Suspenseful Tender Thought Provoking Tragic Uplifting Uproarious&amp;quot;&#xA;bop.lower()&#xA;bag_of_positive_words=nltk.word_tokenize(bop)&#xA;vec=[]&#xA;for i in bag_of_negative_words:&#xA;    if i in mov_review1:&#xA;        vec.append(1)&#xA;    else:&#xA;        for w in bag_of_positive_words:&#xA;            if w in moview_review1:&#xA;                vec.append(5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So I am trying to check whether the review contains a positive word or a negative word. If it contains a negative word then a value of 1 will be assigned to the vector vec else a value of 5 will be assigned. But the output I am getting is an empty vector.&lt;/p&gt;&#xA;&lt;p&gt;Please help. Also, please suggest others way of solving this problem.&lt;/p&gt;&#xA;" OwnerUserId="5214" LastEditorUserId="85045" LastEditDate="2021-04-01T08:23:10.203" LastActivityDate="2021-04-01T08:23:10.203" Title="Sentiment analysis using python" Tags="|python|nlp|sentiment-analysis|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2569" PostTypeId="2" ParentId="2567" CreationDate="2014-11-30T20:01:23.930" Score="5" Body="&lt;p&gt;Robert is right, multinomial logistic regression is the best tool to use. Although you would need to have a integer value representing the party as the dependent variable, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1= Conservative majority, 2= Labour majority, 3= Liberal majority....(and so on)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can perform this in R using the nnet package. &lt;a href=&quot;https://stats.idre.ucla.edu/r/dae/multinomial-logistic-regression/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a good place to quickly run through how to use it.&lt;/p&gt;&#xA;" OwnerUserId="5219" LastEditorUserId="8878" LastEditDate="2017-08-09T15:15:09.553" LastActivityDate="2017-08-09T15:15:09.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2570" PostTypeId="2" ParentId="2567" CreationDate="2014-11-30T20:14:15.560" Score="3" Body="&lt;p&gt;On what do you want to base your prediction? I've tried to predict multiparty election results for my thesis based on previous years and then using results for some polling stations from this year predict the results in all other polling stations. For this the linear model with which I compared estimated the number of votes each party would obtain by regressing over the votes from previous years. If you have the estimated number of votes for all parties you can calculate the percentage from that. See &lt;a href=&quot;http://amstat.tandfonline.com/doi/abs/10.1198/016214504000001835&quot; rel=&quot;nofollow noreferrer&quot;&gt;Forecasts From Nonrandom Samples&lt;/a&gt; for the relevant paper, which extends the linear model.&lt;/p&gt;&#xA;" OwnerUserId="5220" LastEditorUserId="381" LastEditDate="2017-08-09T15:35:01.097" LastActivityDate="2017-08-09T15:35:01.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2571" PostTypeId="2" ParentId="2499" CreationDate="2014-12-01T00:23:30.580" Score="1" Body="&lt;p&gt;The United States Census Bureau has many free housing datasets (some of which are updated more than once every 10 years). There is an &lt;a href=&quot;http://www.census.gov/data/developers/data-sets/acs-survey-1-year-data.html&quot; rel=&quot;nofollow&quot;&gt;API for American Community Survey 1 Year Data&lt;/a&gt; that includes housing data. There are raw data sets at &lt;a href=&quot;http://factfinder2.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t&quot; rel=&quot;nofollow&quot;&gt;American Fact Finder&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2014-12-01T00:23:30.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2572" PostTypeId="2" ParentId="2499" CreationDate="2014-12-01T00:26:39.927" Score="1" Body="&lt;p&gt;There is real estate data for sale at &lt;a href=&quot;http://www.dataquick.com/&quot; rel=&quot;nofollow&quot;&gt;DataQuick&lt;/a&gt; or &lt;a href=&quot;http://www.realquest.com/&quot; rel=&quot;nofollow&quot;&gt;Real Quest&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2014-12-01T00:26:39.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2573" PostTypeId="2" ParentId="2568" CreationDate="2014-12-01T09:58:00.280" Score="0" Body="&lt;p&gt;try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vec =[]&#xA;&#xA;for word in bag_of_negative_words:&#xA;    if word in mov_review1:&#xA;        vec.append(1)&#xA;&#xA;for word in bag_of_positive_words:&#xA;    if word in moview_review1:&#xA;         vec.append(5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2014-12-01T09:58:00.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2574" PostTypeId="2" ParentId="660" CreationDate="2014-12-01T12:23:03.083" Score="1" Body="&lt;p&gt;I suggest using machine learning libraries with already functional linear regression&#xA;&lt;a href=&quot;https://spark.apache.org/docs/1.1.0/mllib-guide.html&quot; rel=&quot;nofollow&quot;&gt;Spark MLlib&lt;/a&gt; or &lt;a href=&quot;https://github.com/myui/hivemall&quot; rel=&quot;nofollow&quot;&gt;hivemall&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2014-12-01T12:23:03.083" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2575" PostTypeId="1" CreationDate="2014-12-01T14:52:31.827" Score="4" ViewCount="1968" Body="&lt;p&gt;In this &lt;a href=&quot;https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF&quot; rel=&quot;nofollow&quot;&gt;wiki page&lt;/a&gt; there is a function &lt;code&gt;corr()&lt;/code&gt; that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that enables to calculate the &lt;a href=&quot;http://en.wikipedia.org/wiki/Kendall%27s_W&quot; rel=&quot;nofollow&quot;&gt;Kendall coefficient&lt;/a&gt; of correlation of a pair of a numeric columns in the group?&lt;/p&gt;&#xA;" OwnerUserId="5224" LastEditorUserId="21" LastEditDate="2014-12-08T14:22:40.260" LastActivityDate="2015-04-28T21:46:01.880" Title="Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?" Tags="|apache-hadoop|correlation|hive|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2576" PostTypeId="1" CreationDate="2014-12-01T20:10:29.967" Score="5" ViewCount="641" Body="&lt;p&gt;I'm looking for API suggestions for enriching data on companies. Currently I use the Crunchbase API to look up a company's name or domain and I am trying to gather the domain/name (if I don't already have both), contact email (this one is a long shot), and the location of their headquarters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works incredibly well if Crunchbase has the company in their API, but I'd say this only happens about 25% of the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd love to get some suggestions on some free APIs that I could use along with Crunchbase. I'd also love to see if anyone has had positive or negative experiences with paid APIs! &lt;/p&gt;&#xA;" OwnerUserId="5227" LastActivityDate="2016-01-26T11:08:14.953" Title="API for Company Data Enrichment Suggestions" Tags="|dataset|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2577" PostTypeId="2" ParentId="2567" CreationDate="2014-12-01T21:59:47.540" Score="2" Body="&lt;p&gt;This is not a regression but a multi-class classification problem. The output is typically the probabilities of all classes for any given test instance (test row). So in your case, the output for any given test row from the trained model will be of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prob_1, prob_2, prob_3,..., prob_k&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where prob_i denotes the probability of the i-th class (in your case i-th party), assuming  there are k classes in the response variable. Note that the sum of these k probabilities is going to be 1. The class prediction in this case is going to be the class that has the maximum probability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many classifiers in R that do multi-class classification. You could use logistic regression with multi-class support through the &lt;a href=&quot;http://cran.r-project.org/web/packages/nnet/nnet.pdf&quot; rel=&quot;nofollow&quot;&gt;nnet&lt;/a&gt; package in R and invoking the &lt;code&gt;multinom&lt;/code&gt; command. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an alternative, you could also use the &lt;a href=&quot;http://cran.r-project.org/web/packages/gbm/gbm.pdf&quot; rel=&quot;nofollow&quot;&gt;gbm&lt;/a&gt; package in R and invoke the &lt;code&gt;gbm&lt;/code&gt; command. To create a multi-class classifier, just use &lt;code&gt;distribution=&quot;multinomial&quot; while using the&lt;/code&gt;gbm` function. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2014-12-01T22:18:54.130" LastActivityDate="2014-12-01T22:18:54.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2578" PostTypeId="1" CreationDate="2014-12-02T10:58:30.950" Score="2" ViewCount="282" Body="&lt;p&gt;I have some question regarding to the choice of the better implementation. I would know the differences and advantages of &lt;a href=&quot;https://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout Apache&lt;/a&gt; (Java implementation) versus &lt;a href=&quot;http://graphlab.com/index.html&quot; rel=&quot;nofollow&quot;&gt;Graphlab&lt;/a&gt; (Python implementation) in the area of the data sciences. Specially in the area of recommenders and classifiers. Can anybody here get some (qualified) feedback about both possibilities?&lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="3466" LastEditDate="2014-12-03T07:43:18.727" LastActivityDate="2015-01-04T22:29:14.980" Title="Graphlab vs Mahout" Tags="|bigdata|classification|python|recommender-system|java|" AnswerCount="2" CommentCount="1" ClosedDate="2015-01-08T14:01:37.343" ContentLicense="CC BY-SA 3.0" />
  <row Id="2579" PostTypeId="1" AcceptedAnswerId="2594" CreationDate="2014-12-02T11:12:06.103" Score="5" ViewCount="2354" Body="&lt;p&gt;Which of the following is best (or widely used) for calculating item-item similarity measure in mahout and why ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pearson Correlation&#xA;Spearman Correlation&#xA;Euclidean Distance&#xA;Tanimoto Coefficient&#xA;LogLikelihood Similarity&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any thumb-rule to chose from these set of algorithm also how to differentiate each of them ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2014-12-11T06:16:59.327" Title="Mahout Similarity algorithm comparison" Tags="|machine-learning|data-mining|statistics|algorithms|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2580" PostTypeId="2" ParentId="2578" CreationDate="2014-12-02T14:35:33.447" Score="0" Body="&lt;p&gt;The advantage of mahout is that it is scalable, apache license , good community and documentation support. Also Fast-prototyping and evaluation&#xA;to evaluate a different configuration of the same algorithm we just need to update a parameter and run again.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2014-12-02T16:00:07.753" LastActivityDate="2014-12-02T16:00:07.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2581" PostTypeId="1" CreationDate="2014-12-02T16:09:35.333" Score="7" ViewCount="2839" Body="&lt;p&gt;I am trying to match new product description with the existing ones. Product description looks like this: ￼Panasonic DMC-FX07EB digital camera silver. These are steps to be performed:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tokenize description and recognize attributes: Panasonic =&gt; Brand, DMC-FX07EB =&gt; Model, etc.&lt;br/&gt;&lt;/li&gt;&#xA;&lt;li&gt;Get few candidates with similar features&lt;br/&gt;&lt;/li&gt;&#xA;&lt;li&gt;Get the best candidate.&lt;br/&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am having problem with the first step (1). In order to get 'Panasonic =&gt; Brand', DMC-FX07EB =&gt; Model, silver =&gt; color, I need to have index where each token of the product description correspond to certain attribute name (Brand, model, color, etc.) in the existing database. The problem is that in my database product descriptions are presented as one atomic attribute e.g. 'description' (no separated product attributes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I don't have training data, so I am trying to build index of all product attributes so I can build training data. So far I have attributes from bestbuy.com and semantics3.com APIs, but both sources lack most of attributes or contain irrelevant ones. Any suggestions for better APIs to get product attributes? Better approach to do this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. For every product there is a matched product description in the Database, which is as well in a form of one atomic attribute. I have checked this &lt;a href=&quot;https://stackoverflow.com/questions/18496925/how-to-parse-product-titles-unstructured-into-structured-data&quot;&gt;question on SO&lt;/a&gt;, it helped me and it seems we have same approach but I am still trying to get training data. &lt;/p&gt;&#xA;" OwnerUserId="5241" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-12-03T14:35:06.510" Title="Attributes extraction from unstructured product descriptions" Tags="|machine-learning|nlp|feature-extraction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2582" PostTypeId="1" AcceptedAnswerId="2595" CreationDate="2014-12-02T16:19:19.043" Score="11" ViewCount="2157" Body="&lt;p&gt;I am currently using SVM and scaling my training features to the range of [0,1].&#xA;I first fit/transform my training set and then apply the &lt;strong&gt;same&lt;/strong&gt; transformation to my testing set. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    ### Configure transformation and apply to training set&#xA;    min_max_scaler = MinMaxScaler(feature_range=(0, 1))&#xA;    X_train = min_max_scaler.fit_transform(X_train)&#xA;&#xA;    ### Perform transformation on testing set&#xA;    X_test = min_max_scaler.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let assume that a given feature in the training set has a range of [0,100], and that same feature in the testing set has a range of [-10,120]. In the training set that feature will be scaled appropriately to [0,1], while in the testing set that feature will be scaled to a range outside of that first specified, something like [-0.1,1.2].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering what the consequences of the testing set features being out of range of those being used to train the model? Is this a problem?&lt;/p&gt;&#xA;" OwnerUserId="802" LastActivityDate="2014-12-03T18:57:22.773" Title="Consequence of Feature Scaling" Tags="|machine-learning|svm|feature-scaling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2583" PostTypeId="2" ParentId="2568" CreationDate="2014-12-02T20:07:14.887" Score="1" Body="&lt;p&gt;Try to search from the databases of official &amp;quot;bad words&amp;quot; that google publishes in this link &lt;a href=&quot;http://fffff.at/googles-official-list-of-bad-words/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google's official list of bad words&lt;/a&gt;.&#xA;Also, here is the link for the good words &lt;a href=&quot;http://www.enchantedlearning.com/wordlist/positivewords.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Not the official list of good words&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;For the code, I would do it like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;textArray = file('dir_to_your_text','r').read().split()&#xA;&#xA;#Bad words should be listed like this for the split function to work&#xA;# &amp;quot;*** ****** **** ****&amp;quot; the stars are for the cenzuration :P&#xA;badArray = file('dir_to_your_bad_word_file).read().split()&#xA;goodArray = file('dir_to_your_good_word_file).read().split()&#xA;&#xA;# Then you use matching algorithm from difflib on good and bad word for every word in an array of words&#xA;import difflib&#xA;&#xA;goodMachingCouter = 0;&#xA;badMacihngCouter = 0;&#xA;&#xA;&#xA;for iGood in range(0, len(goodArray)):&#xA;    for iWord in range(0, len(textArray)):&#xA;        goodMachingCounter += difflib.SequenceMatcher(None, goodArray[iGood], textArray[iWord]).ratio()&#xA;     &#xA;for iBad in range(0, len(badArray)):&#xA;    for iWord in range(0, len(textArray)):&#xA;        badMachingCounter += difflib.SequenceMatcher(None, badArray[ibad], textArray[iWgoodord]).ratio()&#xA;&#xA;goodMachingCouter *= 100/(len(goodArray)*len(textArray))&#xA;badMacihngCouter *= 100/(len(badArray)*len(textArray))&#xA;&#xA;print('Show the good measurment of the text in %: '+goodMachingCouter)&#xA;print('Show the bad measurment of the text in %: '+badMacihngCouter)&#xA;print('Show the hootnes of the text: ' + len(textArray)*goodMachingCounter)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The code will be slow but accurate :) I didn't run and test it please do it for me and post the correct code :) because I wanna test it too :)&lt;/p&gt;&#xA;" OwnerDisplayName="user5244" LastEditorUserId="85045" LastEditDate="2021-03-30T20:43:19.287" LastActivityDate="2021-03-30T20:43:19.287" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2584" PostTypeId="2" ParentId="2581" CreationDate="2014-12-02T22:23:33.700" Score="5" Body="&lt;p&gt;Left you a quick response on SO. The gist is that you can collect a lot of information from electronics shops and manufacturers' web sites, and lots you can annotate manually. If your goal is to only get training data, that's all you need:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My answer form the cross-post:&#xA;&quot;Having developed a commercial analyzer of this kind, I can tell you that there is no easy solution for this problem. But there are multiple shortcuts, especially if your domain is limited to cameras/electronics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, you should look at more sites. Many have product brand annotated in the page (proper html annotations, bold font, all caps in the beginning of the name). Some sites have entire pages with brand selectors for search purposes. This way you can create a pretty good starter dictionary of brand names. Same with product line names and even with models. Alphanumeric models can be extracted in bulk by regular expressions and filtered pretty quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of other tricks, but I'll try to be brief. Just a piece of advice here: there is always a trade-off between manual work and algorithms. Always keep in mind that both approaches can be mixed and both have return-on-invested-time curves, which people tend to forget. If your goal is not to create an automatic algorithm to extract product brands and models, this problem should have limited time budget in your plan. You can realistically create a dictionary of 1000 brands in a day, and for decent performance on known data source of electronic goods (we are not talking Amazon here or are we?) a dictionary of 4000 brands may be all you need for your work. So do the math before you invest weeks into the latest neural network named entity recognizer.&quot;&lt;/p&gt;&#xA;" OwnerUserId="5249" LastEditorUserId="5249" LastEditDate="2014-12-03T14:35:06.510" LastActivityDate="2014-12-03T14:35:06.510" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2585" PostTypeId="2" ParentId="2582" CreationDate="2014-12-03T05:00:30.500" Score="7" Body="&lt;p&gt;This was meant as a comment but it is too long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that your test set has a different range &lt;strong&gt;might&lt;/strong&gt; be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it won't affect a SVM in any circumstance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the rationale for using MinMaxScalar is (according to the documentation): &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The motivation to use this scaling include robustness to very small&#xA;  standard deviations of features and preserving zero entries in sparse&#xA;  data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Therefore, it is important for you to make sure that your data fits that case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are really concerned about having a difference range, you should use a regular standardization (such as &lt;code&gt;preprocessing.scale&lt;/code&gt;) instead.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2014-12-03T05:00:30.500" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2586" PostTypeId="1" AcceptedAnswerId="2606" CreationDate="2014-12-03T07:46:30.967" Score="2" ViewCount="155" Body="&lt;p&gt;I'm very passionate about how computers can be made able to think intelligently and independently (in our favour, of course!). I'm currently studying Bachelors science of information technology at UTS (University of Technology:Sydney). I have two months before I start my second year, and have not yet been able to decide on which major should I select that can lead myself towards dedicated study of Artificial Intelligence (which I love with my life).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following majors available:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Internetworking and Applications &lt;/li&gt;&#xA;&lt;li&gt;Data Analytics&lt;/li&gt;&#xA;&lt;li&gt;(there are other two as well, but business oriented).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://uts.edu.au&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is the link to my subjects. I believe that being able to play with data is a sign of intelligence (I may be wrong too!). Will one of these subjects form me a good foundation for my further study in A.I.? Or should I jump into Engineering? Or Pure Science?&lt;/p&gt;&#xA;" OwnerUserId="5185" LastEditorUserId="84" LastEditDate="2014-12-12T23:48:02.230" LastActivityDate="2014-12-12T23:48:02.230" Title="Can data analytics be a basis for artificial intelligence?" Tags="|bigdata|career|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2587" PostTypeId="1" CreationDate="2014-12-03T11:05:59.490" Score="3" ViewCount="439" Body="&lt;p&gt;Is there a method/class available in Apache Mahout to perform n-fold cross validation?&#xA;If yes how it can be done?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="21" LastEditDate="2014-12-03T13:50:48.697" LastActivityDate="2016-01-29T06:16:35.847" Title="N - fold cross validation in mahout" Tags="|machine-learning|data-mining|java|apache-mahout|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2588" PostTypeId="2" ParentId="2313" CreationDate="2014-12-03T13:21:34.153" Score="4" Body="&lt;p&gt;The difference between these methods is the assumptions they make about the task.  &lt;a href=&quot;http://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow&quot;&gt;Multi-class classification&lt;/a&gt; assumes that each document has exactly one label.  So a document can either be about sports or weather, not both.&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multi-label_classification&quot; rel=&quot;nofollow&quot;&gt;Multi-label classification&lt;/a&gt; allows a document to have any combination of labels, including none.  So a document can be about only sports, only weather, sports AND weather, or neither.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could train a multi-label classifier with data where each document has exactly one label, but there is no guarantee that the predictions made at test time will have only one label.  Also you are forcing the classifier to do more work (and potentially make more errors) by considering more possible labelings than it needs to.&#xA;Therefore, if the multi-class assumption makes sense for your problem, you are better off with a multi-class classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method that you describe for training individual binary classifiers corresponds to multi-label classification.  The binary classifiers that you use could each be trained from one-class data or two-class data.  However, this is only one of the many ways to do multi-label classification (see the wikipedia page above for more).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the problem that you describe does not cleanly fit into either multi-class or multi-label classification, since you want each document to have &lt;em&gt;at most&lt;/em&gt; one label.&lt;/p&gt;&#xA;" OwnerUserId="5263" LastActivityDate="2014-12-03T13:21:34.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2589" PostTypeId="2" ParentId="2525" CreationDate="2014-12-03T13:55:16.150" Score="5" Body="&lt;p&gt;I believe the claim that you are referring to is that the &lt;em&gt;maximum-likelihood estimate&lt;/em&gt; of the component means in a GMM must lie in the span of the eigenvectors of the second moment matrix.  This follows from two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Each component mean in the maximum-likelihood estimate is a linear combination of the data points.  (You can show this by setting the gradient of the log-likelihood function to zero.)&lt;/li&gt;&#xA;&lt;li&gt;Any linear combination of the data points must lie in the span of the eigenvectors of the second moment matrix.  (You can show this by first showing that any individual data point must lie in the span, and therefore any linear combination must also be in the span.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5263" LastActivityDate="2014-12-03T13:55:16.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2590" PostTypeId="1" CreationDate="2014-12-03T14:18:34.003" Score="2" ViewCount="89" Body="&lt;p&gt;I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Up to now I was only taking into considerations the clients as first holders. &#xA;For example, if a client ‘1’ appears as the first holder of a saving account ‘A’ with a balance at the end of the month of 100€ and as an additional holder of a saving account ‘B’ with a balance at the end of the month of 50€, the saving balance at the end of the month for the client is considered to be just 100€.&#xA;Moreover, if a client only appears as an additional holder (and he/she is not a first account holder of ANY product), he/she is dismissed by the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I have been told to include additional holders in the models (additional holders have the same rights of the first holders).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility is to recalculate all the variables summing up the position as first and additional holder (in the previous example, the balance at the end of the month of client ‘1’ would be 150€). Together with this, I would create some variable that represents the maximum degree of intervention of the client in the account (ex. 'first holder', 'second holder').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility would be to “double” all the variables, considering the client as first and additional holder (in the example, we would create two variables:  the balance at the end of the month as FH =100€, :  the balance at the end of the month as AH =50€).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did any of you encounter a similar problem?It would be very helpful to understand how you solved it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="5265" LastActivityDate="2014-12-03T14:18:34.003" Title="Cross-sell models and additional holders" Tags="|data-mining|dataset|predictive-modeling|logistic-regression|data-cleaning|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2591" PostTypeId="1" CreationDate="2014-12-03T14:23:38.830" Score="6" ViewCount="157" Body="&lt;p&gt;I want to create a model to predict the propensity to buy a certain product. As my proportion of 1's is very low, I decided to apply oversampling (to get a 10% of 1's and a 90% of 0's).&lt;/p&gt;&#xA;&lt;p&gt;Now, I want to discretize some of the variables. To do so I run a tree for each variable against the target.&lt;/p&gt;&#xA;&lt;p&gt;Should I define the prior probabilities when I do this (run the trees), or it doesn't matter and I can use the over-sampled dataset just like that?&lt;/p&gt;&#xA;" OwnerUserId="5265" LastEditorUserId="29575" LastEditDate="2020-06-20T13:41:21.260" LastActivityDate="2020-08-02T17:05:55.560" Title="Decision trees, categorizacion and oversampling" Tags="|classification|predictive-modeling|sampling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2593" PostTypeId="1" AcceptedAnswerId="2635" CreationDate="2014-12-03T15:56:50.687" Score="8" ViewCount="127" Body="&lt;p&gt;I want to write a data-mining service in &lt;a href=&quot;http://golang.org&quot;&gt;Google Go&lt;/a&gt; which collects data through scraping and APIs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However as Go lacks good ML support I would like to do the ML stuff in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example most (web) protocols lack at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;buffering between processes&lt;/li&gt;&#xA;&lt;li&gt;clustering over multiple instances&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So what (type of libraries) do data scientists use to connect different languages/processes?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bodo&lt;/p&gt;&#xA;" OwnerUserId="5266" LastActivityDate="2014-12-07T11:29:14.057" Title="How to connect data-mining with machine learner process" Tags="|machine-learning|data-mining|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2594" PostTypeId="2" ParentId="2579" CreationDate="2014-12-03T18:22:29.333" Score="3" Body="&lt;p&gt;For those not familiar, item-item recommenders calculate similarities between items, as opposed to user-user (or user-based) recommenders, which calculate similarities between users. Although some algorithms can be used for both, this question is in regard to item-item algorithms (thanks for being specific in your question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accuracy or effectiveness of recommenders is evaluated based on comparing recommendations to a previously collected data set (training set). For example, I have shopping cart data from the last six months; I'll use the first 5 months as training data, then run my various algorithms, and compare the quality against what really happened during the 6th month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason Mahout ships with so many algorithms is because different algorithms are more or less effective in each data set you may work with. So, ideally, you do some testing as I described with many algorithms and compare the accuracy, then choose the winner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, you can also take other factors into account, such as the need to minimize the data set (for performance reasons), and run your tests only with a certain portion of the training data available. In such a case, one algorithm may work better with the smaller data set, but another may work with the complete set. Then, you get to weigh performance VS accuracy VS challenge of implementation (such as deploying on a Hadoop cluster).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, different algorithms are suited for different project. However, there are some general rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;All algorithms always do better with unreduced data sets (more data is better).&lt;/li&gt;&#xA;&lt;li&gt;More complex algorithms aren't necessarily better.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I suggest starting with a simple algorithm and ensuring you have high quality data. If you have additional time, you can implement more complex algorithms and create a comparison which is unique to your data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of my info comes from &lt;a href=&quot;http://ai.arizona.edu/intranet/papers/comparative.ieeeis.pdf&quot; rel=&quot;nofollow&quot;&gt;This study&lt;/a&gt;. You'll find lots of detail about implementation there.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-12-03T18:22:29.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2595" PostTypeId="2" ParentId="2582" CreationDate="2014-12-03T18:36:00.880" Score="6" Body="&lt;p&gt;Within each class, you'll have distributions of values for the features. That in itself is not a reason for concern. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a slightly theoretical point of view, you can ask yourself why you should scale your features and why you should scale them in exactly the chosen way.&lt;br&gt;&#xA;One reason may be that your particular training algorithm is known to converge faster (better) with values around 0 - 1 than with features which cover other orders of magnitude. In that case, you're probably fine. My guess is that your SVM is fine: you want to avoid too large numbers because of the inner product, but a max of 1.2 vs. a max of 1.0 won't make much of a difference.&lt;br&gt;&#xA;(OTOH, if you e.g. knew your algorithm to not accept negative values you'd obviously be in trouble. )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The practical question is whether your model performs well for cases that are slightly out of the range covered by training. This I believe can best and possibly only be answered by testing with such cases / inspecting test results for performance drop for cases outside the training domain. It is a valid concern and looking into this would be part of the validation of your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observing differences of the size you describe is IMHO a reason to have a pretty close look at model stability. &lt;/p&gt;&#xA;" OwnerUserId="5268" LastEditorUserId="5268" LastEditDate="2014-12-03T18:57:22.773" LastActivityDate="2014-12-03T18:57:22.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2596" PostTypeId="1" AcceptedAnswerId="2597" CreationDate="2014-12-03T21:47:34.907" Score="5" ViewCount="124" Body="&lt;p&gt;I've been toying with this idea for a while. I think there is probably some method in the text mining literature, but I haven't come across anything just right...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is/are some methods for tackling a problem where the number of variables it its self a variable.  This is not a missing data problem, but one where the nature of the problem fundamentally changes. Consider the following example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I want to predict who will win a race, a simple multinomial classification problem. I have lots of past data on races, plenty to train on.  Lets further suppose I have observed each contestant run multiple races. The problem however is that the number or racers is variable. Sometimes there are only 2 racers, sometimes there are as many as 100 racers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution might be to train a separate model for each number or racers, resulting in 99 models in this case, using any method I choose.  E.g. I could have 100 random forests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another solution might be to include an additional variable called 'number_of_contestants' and have input field for 100 racers and simply leave them blank when no racer is present.  Intuitively, it seems that this method would have difficulties predicting the outcome of a 100 contestant race if the number of racers follows a Poisson distribution (which I didn't originally specify in the problem, but I am saying it here).    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="5247" LastActivityDate="2014-12-04T23:26:15.803" Title="Method for solving problem with variable number of predictors" Tags="|machine-learning|data-mining|classification|statistics|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2597" PostTypeId="2" ParentId="2596" CreationDate="2014-12-04T03:42:48.357" Score="2" Body="&lt;p&gt;I don't see the problem. All you need is a learner to map a bit string as long as the &lt;em&gt;total number of contestants&lt;/em&gt;, representing the subset who are taking part, to another bit string (with only one bit set) representing the winner, or a ranked list, if you want them all (assuming you have the whole list in your training data). In the latter case you would have a learning-to-rank problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the contestant landscape can change it would help to find a vector space embedding for them so you can use the previous embeddings as an initial guess and rank anyone, even hypothetical, given their vector representation. As the number of users increases the embedding should stabilize and retraining should become less costly. The question is how to find the embedding, of course. If you have a lot of training data, you could probably find a randomized one along with the ranking function. If you don't, you would have to generate the embedding by some algorithm and estimate only the ranking function. I have not faced your problem before so I can't direct you to a particular paper, but the recent NLP literature should give you some inspiration, e.g. &lt;a href=&quot;http://jmlr.org/papers/volume13/shalit12a/shalit12a.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;. I still think it is feasible.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-12-04T20:11:17.383" LastActivityDate="2014-12-04T20:11:17.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2598" PostTypeId="1" AcceptedAnswerId="2599" CreationDate="2014-12-04T05:18:03.720" Score="16" ViewCount="31009" Body="&lt;p&gt;I would like to know how exactly mahout user based and item based recommendation differ from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It defines that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://mahout.apache.org/users/recommender/userbased-5-minutes.html&quot;&gt;User-based&lt;/a&gt;: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://mahout.apache.org/users/recommender/intro-itembased-hadoop.html&quot;&gt;Item-based&lt;/a&gt;: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed off line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But though there are two kind of recommendation available, what I understand is that both these will take some data model ( say 1,2 or 1,2,.5 as item1,item2,value or user1,user2,value where value is not mandatory) and will perform all calculation as the similarity measure and recommender build-in function we chose and we can run both user/item based recommendation on the same data ( is this a correct assumption ?? ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would like to know how exactly and in which all aspects these two type of algorithm differ. &lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-11-24T12:20:21.137" LastActivityDate="2020-08-16T13:02:03.173" Title="Item based and user based recommendation difference in Mahout" Tags="|machine-learning|data-mining|algorithms|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2599" PostTypeId="2" ParentId="2598" CreationDate="2014-12-04T06:08:25.000" Score="12" Body="&lt;p&gt;You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings.&lt;/p&gt;&#xA;&lt;p&gt;In the user-based approach the algorithm produces a rating for an item &lt;code&gt;i&lt;/code&gt; by a user &lt;code&gt;u&lt;/code&gt; by combining the ratings of other users &lt;code&gt;u'&lt;/code&gt; that are similar to &lt;code&gt;u&lt;/code&gt;. Similar here means that the two user's ratings have a high Pearson correlation or cosine similarity or something similar.&lt;/p&gt;&#xA;&lt;p&gt;In the item-based approach we produce a rating for &lt;code&gt;i&lt;/code&gt; by &lt;code&gt;u&lt;/code&gt; by looking at the set of items &lt;code&gt;i'&lt;/code&gt; that are similar to &lt;code&gt;i&lt;/code&gt; (in the same sense as above except now we'd be looking at the ratings that items have received from users) that &lt;code&gt;u&lt;/code&gt; has rated and then combines the ratings by &lt;code&gt;u&lt;/code&gt; of &lt;code&gt;i'&lt;/code&gt; into a predicted rating  by &lt;code&gt;u&lt;/code&gt; for &lt;code&gt;i&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The item-based approach was invented at &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=642471&quot; rel=&quot;nofollow noreferrer&quot;&gt;Amazon&lt;/a&gt; to address their scale challenges with user-based filtering. The number of things they sell is much less and much less dynamic than the number of users so the item-item similarities can be computed offline and accessed when needed.&lt;/p&gt;&#xA;" OwnerUserId="2724" LastEditorUserId="98307" LastEditDate="2020-08-16T13:02:03.173" LastActivityDate="2020-08-16T13:02:03.173" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2600" PostTypeId="1" CreationDate="2014-12-04T07:47:32.740" Score="2" ViewCount="206" Body="&lt;p&gt;I am interested in graph problems like 2-color, max-clique, stable sets, etc but the documentation for scipy.optimize.anneal seems to be for ordinary functions. How would one apply this library towards graph formulations?&lt;/p&gt;&#xA;" OwnerUserId="5273" LastEditorUserId="1330" LastEditDate="2022-04-09T15:50:19.863" LastActivityDate="2022-04-09T15:50:19.863" Title="How does one feed graph optimization problems into Python's anneal function in SciPy?" Tags="|python|optimization|graphs|scipy|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2601" PostTypeId="1" CreationDate="2014-12-04T08:54:32.863" Score="4" ViewCount="2183" Body="&lt;p&gt;I came across a package in R which has a function called &lt;code&gt;sann&lt;/code&gt; for simulated annealing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;sann&lt;/code&gt; uses parameters &lt;code&gt;fn&lt;/code&gt; and &lt;code&gt;gr&lt;/code&gt; to optimize and to select new points, respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For something like the max-clique or max-stable set problems, &lt;code&gt;fn&lt;/code&gt; would be a summing function, but it's less clear how one would formulate &lt;code&gt;gr&lt;/code&gt; to fix these graph computations. In these cases, how would &lt;code&gt;gr&lt;/code&gt; &quot;select&quot;?&lt;/p&gt;&#xA;" OwnerUserId="5273" LastEditorUserId="2723" LastEditDate="2014-12-05T23:04:05.373" LastActivityDate="2015-05-05T03:35:17.127" Title="How to use &quot;sann&quot; function in R to solve graph problems?" Tags="|r|graphs|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2602" PostTypeId="2" ParentId="2598" CreationDate="2014-12-04T15:12:57.327" Score="23" Body="&lt;p&gt;&lt;strong&gt;Item Based Algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for every item i that u has no preference for yet&#xA;&#xA;  for every item j that u has a preference for&#xA;&#xA;    compute a similarity s between i and j&#xA;&#xA;    add u's preference for j, weighted by s, to a running average&#xA;&#xA; return the top items, ranked by weighted average&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;User Based Algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for every item i that u has no preference for yet&#xA;&#xA; for every other user v that has a preference for i&#xA;&#xA;   compute a similarity s between u and v&#xA;&#xA;   add v's preference for i, weighted by s, to a running average&#xA;&#xA; return the top items, ranked by weighted average&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Item vs User based:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Recommenders scale with the number of items or users they must deal with, so there are scenarios in which each type can perform better than the other&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Similarity estimates between items are more likely to converge over time than similarities between users&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) We can compute and cache similarities that converge, which can give item based recommenders a performance advantage&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Item based recommenders begin with a list of a user's preferred items and therefore do not need a nearest item neighborhood as user based recommenders do&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="5043" LastEditDate="2014-12-04T19:08:33.323" LastActivityDate="2014-12-04T19:08:33.323" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2604" PostTypeId="2" ParentId="155" CreationDate="2014-12-04T18:36:57.947" Score="18" Body="&lt;p&gt;&lt;strong&gt;Custom Google Search&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the Custom Google Search for datasets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.google.com/cse/publicurl?cx=002720237717066476899:v2wv26idk7m&quot;&gt;Google Custom Search: Datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It includes 230 sources and meta-sources of datasets, including all mentioned in this question. Please, feel free to exclude .gov and any other websites from results by adding &quot; -.gov&quot; or &quot; -site.com&quot; to the search line. Other Google Search Operators work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't hesitate to contact me if you have ideas what websites to add.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IOGDS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following service categorizes more than 1,000,000 public datasets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://logd.tw.rpi.edu/node/9903&quot;&gt;IOGDS: International Open Government Dataset Search&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5279" LastActivityDate="2014-12-04T18:36:57.947" CommentCount="2" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="2606" PostTypeId="2" ParentId="2586" CreationDate="2014-12-04T22:13:30.747" Score="2" Body="&lt;p&gt;Classes related to Artificial Intelligence are typically taught in Computer Science departments. Looking at the &lt;a href=&quot;https://my.feit.uts.edu.au/pages/course/undergraduate/it_project_subjects&quot; rel=&quot;nofollow&quot;&gt;IT Project Subjects offered by your university&lt;/a&gt;, I suspect Data Analytics would indeed be more relevant to AI than Internetworking and Applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the courses offered by your department, the following likely involve aspects of AI:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Image Processing and Pattern Recognition&lt;/li&gt;&#xA;&lt;li&gt;Intelligent Agents&lt;/li&gt;&#xA;&lt;li&gt;Building Intelligent Agents&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For self-directed study in AI, I recommend starting with Russell &amp;amp; Norvig's essential textbook &lt;a href=&quot;http://aima.cs.berkeley.edu/&quot; rel=&quot;nofollow&quot;&gt;Artificial Intelligence: A Modern Approach&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to what it will take to create a human-like strong AI, I recommend this collection of essays: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0198248547&quot; rel=&quot;nofollow&quot;&gt;The Philosophy of Artificial Intelligence&lt;/a&gt;... even though the material is getting a bit out-of-date by now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-04T22:13:30.747" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2607" PostTypeId="2" ParentId="2596" CreationDate="2014-12-04T23:26:15.803" Score="1" Body="&lt;p&gt;Would it be possible to use Approximate Bayesian computation (ABC)? If you assume a distribution for the number of competitors (e.g. Poisson), select a subset of competitors each iteration and simulate your data using multinomial distributions with probabilities based on competitors' features, after discarding parameters that don't match your training data, you should be able to obtain parameters for each competitor (that is, posterior distributions) and generate more races. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might not work if the number of competitors is so important that it affects the coefficients of features for each competitor.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2014-12-04T23:26:15.803" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2608" PostTypeId="1" CreationDate="2014-12-05T00:26:12.983" Score="3" ViewCount="96" Body="&lt;p&gt;My data looks like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/9LgwU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this error showing up?&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2014-12-05T15:31:06.383" Title="Simple Excel Question: VLookup Error" Tags="|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2609" PostTypeId="1" AcceptedAnswerId="2616" CreationDate="2014-12-05T05:50:29.903" Score="8" ViewCount="4016" Body="&lt;p&gt;Can anyone suggest any good books to learn hadoop and map reduce basics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also something for Spark, and Spark Streaming?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="5283" LastActivityDate="2014-12-05T15:38:09.867" Title="Good books for Hadoop, Spark, and Spark Streaming" Tags="|apache-hadoop|" AnswerCount="3" CommentCount="2" ClosedDate="2014-12-06T13:29:54.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="2610" PostTypeId="2" ParentId="2609" CreationDate="2014-12-05T11:05:15.257" Score="1" Body="&lt;p&gt;for hadoop try &lt;a href=&quot;http://www.amazon.in/Hadoop-Action-Chuck-Lam/dp/8177228137?tag=googinhydr18418-21&quot; rel=&quot;nofollow&quot;&gt;hadoop in action&lt;/a&gt;  or &lt;a href=&quot;http://www.amazon.in/Hadoop-The-Definitive-Guide-White/dp/9350237563?tag=googinhydr18418-21&quot; rel=&quot;nofollow&quot;&gt;Hadoop: The Definitive Guide&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2014-12-05T11:05:15.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2611" PostTypeId="2" ParentId="2609" CreationDate="2014-12-05T11:22:01.423" Score="2" Body="&lt;p&gt;Here's a good &lt;a href=&quot;http://databricks.gitbooks.io/databricks-spark-reference-applications/&quot; rel=&quot;nofollow&quot;&gt;open source book for spark&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4679" LastActivityDate="2014-12-05T11:22:01.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2612" PostTypeId="1" CreationDate="2014-12-05T14:03:58.777" Score="6" ViewCount="631" Body="&lt;p&gt;Although I have seen a few good questions asked about data anonymization, I was wondering if there were answers to this more specific variant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am seeking a tool (or to design one) that will anonymize human names from a specific country: particularly first names in unstructured text. Many of the tools that I have seen have considered the wider dimensions of data anonymization; with an equal focus on dates of birth, addresses, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An imperative aspect is that it needs to have near absolute recall. The major pitfalls, as far as I can see, are diminutive variants (&quot;Tommy&quot; instead of &quot;Thomas&quot;, &quot;Ben&quot; instead of &quot;Benjamin&quot;, etc.) and typos. These two factors prevent a simple regex based on a database of names (based on censuses, etc.)&lt;/p&gt;&#xA;" OwnerUserId="5290" LastActivityDate="2015-01-09T00:18:05.037" Title="Name Anonymization Software" Tags="|text-mining|anonymization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2613" PostTypeId="2" ParentId="1007" CreationDate="2014-12-05T15:28:39.640" Score="0" Body="&lt;p&gt;Ruby together with &lt;a href=&quot;http://www.nokogiri.org/&quot; rel=&quot;nofollow&quot;&gt;Nokogiri&lt;/a&gt; allows to access HTML and XML documents via XPath and CSS selectors. Here is a &lt;a href=&quot;http://thaiwood.io/screen-scraping-with-a-saw-a-nokogiri-tutorial-with-examples/&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="223" LastActivityDate="2014-12-05T15:28:39.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2614" PostTypeId="2" ParentId="2612" CreationDate="2014-12-05T15:29:04.743" Score="2" Body="&lt;p&gt;I don't think you really need some special software, but rather to employee existing tools, such as encryption algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not just encrypt the names with any key-based algorithm and store the key securely?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you didn't need to be able to recover the names, but just to identify variation to the level of differences in diminutives, then you could simply use hashing rather than encryption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what environment you want to carry this out it, but any language such as &lt;code&gt;R&lt;/code&gt; or SQL/NoSQL database could easily carry this out programmatically.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T15:29:04.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2615" PostTypeId="2" ParentId="2608" CreationDate="2014-12-05T15:31:06.383" Score="1" Body="&lt;p&gt;This is a common result of imprecise matching, such as with whitespace problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you imagine that the grey shading used in codeblocks on StackExchange represents the whitespaces in your Excel cells, then &#xA;  &lt;code&gt;AK&lt;/code&gt; &#xA;is technically the same as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AK  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can trim and/or use wildcard chars to allow matching to skip spaces. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;=VLOOKUP(CONCATENATE(&quot;*&quot;,TRIM(A2),&quot;*&quot;), $E$2:$F$5, 2,FALSE)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T15:31:06.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2616" PostTypeId="2" ParentId="2609" CreationDate="2014-12-05T15:38:09.867" Score="5" Body="&lt;p&gt;There's such an overwhelming amount of literature that with programming, databases, and Big Data I like to stick to the O'reilly series as my go-to source. O'reilly books are extremely popular in the industry and I've been very satisfied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A current version of &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920021773.do&quot; rel=&quot;noreferrer&quot;&gt;Hadoop: The Definitive Guide&lt;/a&gt;, &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920025122.do&quot; rel=&quot;noreferrer&quot;&gt;MapReduce Design Patterns&lt;/a&gt;, and &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920028512.do&quot; rel=&quot;noreferrer&quot;&gt;Learning Spark&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;might suit your needs by providing high quality, immediately useful information and avoiding information overload -- all are published by O'reilly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark Streaming is covered in Chapter 13 of &quot;Learning Spark&quot;.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T15:38:09.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2617" PostTypeId="2" ParentId="867" CreationDate="2014-12-05T15:43:40.720" Score="3" Body="&lt;p&gt;In terms of open source NLG components, I'm most familiar with Mumble and FUF/SURGE. They've got both similarities and differences, so it's hard to say which is better...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://code.google.com/p/sparser/source/browse/#svn%2Ftrunk%2FMumble&quot; rel=&quot;nofollow&quot;&gt;Mumble&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;written in Lisp&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.eclipse.org/legal/epl-v10.html&quot; rel=&quot;nofollow&quot;&gt;EPL license&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;based on tree-adjoining grammar&lt;/li&gt;&#xA;&lt;li&gt;focuses on linguistic message planning&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.bgu.ac.il/surge/index.html&quot; rel=&quot;nofollow&quot;&gt;FUF/SURGE&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;written in Lisp&lt;/li&gt;&#xA;&lt;li&gt;GPL license&lt;/li&gt;&#xA;&lt;li&gt;based on functional unification grammar&lt;/li&gt;&#xA;&lt;li&gt;focuses on syntactic realization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Since it sounds like you're interested in abstractive summarization (which is much harder than traditional extractive summarization), I'd recommend the following academic papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.nist.gov/tac/publications/2010/participant.papers/Rali.proceedings.pdf&quot; rel=&quot;nofollow&quot;&gt;Text Generation for Abstractive Summarization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.aclweb.org/anthology/W11-1608&quot; rel=&quot;nofollow&quot;&gt;Framework for Abstractive Summarization using Text-to-Text Generation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2000990&quot; rel=&quot;nofollow&quot;&gt;Towards a Framework for Abstractive Summarization of Multimodal Documents&lt;/a&gt; -- full disclosure: I'm the author of this one&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, consider checking out this textbook to get started: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521620368&quot; rel=&quot;nofollow&quot;&gt;Building Natural Language Generation Systems&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-05T15:43:40.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2618" PostTypeId="1" AcceptedAnswerId="2624" CreationDate="2014-12-05T15:51:54.690" Score="13" ViewCount="368" Body="&lt;p&gt;Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, the Data Science Toolkit (or &lt;code&gt;RDSTK&lt;/code&gt; for R programmers) allows me to pull lots of good location-based data using IP's or addresses and the &lt;code&gt;tm.webmining.plugin&lt;/code&gt; for R's &lt;code&gt;tm&lt;/code&gt; package makes scraping financial and news data straightfoward. When going beyond such (semi-) structured data I tend to use &lt;code&gt;XPath&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm constantly getting throttled by limits on the number of queries you're allowed to make. I think Google limits me to about 50,000 requests per 24 hours, which is a problem for Big Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a &lt;em&gt;technical&lt;/em&gt; perspective getting around these limits is easy -- just switch IP addresses and purge other identifiers from your environment. However, this presents both ethical and financial concerns (I think?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a solution that I'm overlooking?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="11097" LastEditDate="2015-11-19T17:42:19.730" LastActivityDate="2017-10-02T06:16:35.300" Title="Ethically and Cost-effectively Scaling Data Scrapes" Tags="|text-mining|scraping|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2619" PostTypeId="2" ParentId="2494" CreationDate="2014-12-05T15:57:07.737" Score="1" Body="&lt;p&gt;I'm not sure I fully understand your question, but it seems to me that you're trying to determine the category of the string/entity &quot;titanic&quot; out of context. Your data tells you that &quot;titanic&quot; could be a book, a movie, or a product, and you want to figure out which one is correct -- is that what you're trying to do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, the problem is that you've dropped the context in which the string/entity &quot;titanic&quot; appears in your original text. For example...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In the sentence &quot;I couldn't stop reading Titanic,&quot; the word &quot;titanic&quot; refers to a book.&lt;/li&gt;&#xA;&lt;li&gt;In the sentence &quot;Titanic was one of the highest-grossing films of all time,&quot; the word &quot;titanic&quot; refers to a movie.&lt;/li&gt;&#xA;&lt;li&gt;In the sentence &quot;The Titanic was the world's largest ocean liner,&quot; the word &quot;titanic&quot; refers to a product.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Without that context, there's no way to know which is the correct category. I'd suggest looking into how named entity recognition tools like &lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow&quot;&gt;Stanford NER&lt;/a&gt; work -- that will help you better understand how to do something like this. You'll see that the input to an NER tool generally needs to be a sentence, in order to take advantage of the context to properly categorize the extracted entities.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-05T15:57:07.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2620" PostTypeId="2" ParentId="954" CreationDate="2014-12-05T16:14:01.097" Score="1" Body="&lt;p&gt;From an older version of the &lt;a href=&quot;http://opennlp.sourceforge.net/README.html&quot; rel=&quot;nofollow&quot;&gt;OpenNLP README&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Training the Tools&lt;/strong&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;There are training tools for all components expect the coref component. Please consult the help message of the tool and the javadoc to figure out how to train the tools.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The tutorials in our wiki might also be helpful.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The following modules currently support training via the WordFreak opennlp.plugin v1.4 (&lt;a href=&quot;http://wordfreak.sourceforge.net/plugins.html&quot; rel=&quot;nofollow&quot;&gt;http://wordfreak.sourceforge.net/plugins.html&lt;/a&gt;).&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;coreference: org.annotation.opennlp.OpenNlpCoreferenceAnnotator (use opennlp 1.4.3 for training, models are compatible)&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; In order to train a model you need all the training data. There is not currently a mechanism to update the models distributed with the project with additional data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you can see, OpenNLP does not provide training tools for the coreference component. However, it seems at one point it was possible to train new models for OpenNLP's coref component using the third-party WordFreak plugin... however, it hasn't been updated in over a decade, so your mileage may vary.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-05T16:14:01.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2621" PostTypeId="2" ParentId="2601" CreationDate="2014-12-05T16:35:42.910" Score="1" Body="&lt;p&gt;First, some clarification on terminology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;em&gt;package&lt;/em&gt; in R is a collection of R functions, data, and compiled code in a well-defined format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SANN (&lt;code&gt;sann&lt;/code&gt;) is not a package. Depending on which package you're using, &lt;code&gt;sann&lt;/code&gt; is either a function or, more often, a &lt;em&gt;method&lt;/em&gt; used within an optimization function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Packages containing &lt;code&gt;sann&lt;/code&gt; include &lt;code&gt;optim&lt;/code&gt;, &lt;code&gt;trustOptim&lt;/code&gt;, &lt;code&gt;consPlan&lt;/code&gt;, and &lt;code&gt;constrOptim&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the package &lt;code&gt;optim&lt;/code&gt;, the &lt;code&gt;sann&lt;/code&gt; method is implemented as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; func &amp;lt;- function(x){&#xA;+   out &amp;lt;- (x[1]-2)^2 + (x[2]-1)^2&#xA;+   return &amp;lt;- out&#xA;+   }&amp;gt; &#xA;&amp;gt; optim(par=c(0,0), fn=func, gr = NULL,&#xA;+       method = c(&quot;SANN&quot;),&#xA;+       lower = -Inf, upper = Inf,&#xA;+       control = list(), hessian = T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you said, for the &quot;SANN&quot; (&lt;code&gt;sann&lt;/code&gt;) method &lt;code&gt;gr&lt;/code&gt; is used to generate a new candidate point. If it is NULL a default Gaussian Markov kernel is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in your use case -- the case of a graph -- what you probably want to do is to &lt;a href=&quot;http://www.inside-r.org/r-doc/graphics/par&quot; rel=&quot;nofollow&quot;&gt;use &lt;code&gt;par&lt;/code&gt; and &lt;code&gt;value&lt;/code&gt; to pass values to &lt;code&gt;fn&lt;/code&gt; and &lt;code&gt;gr&lt;/code&gt;&lt;/a&gt;. This is a nice feature of this implementation of SANN in &lt;code&gt;optim&lt;/code&gt; which is covered a little more than half way  through this &lt;a href=&quot;http://www.inside-r.org/r-doc/stats/optim&quot; rel=&quot;nofollow&quot;&gt;documentation page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;par&lt;/code&gt; is the best set of parameters found and &lt;code&gt;value&lt;/code&gt; is the value of &lt;code&gt;fn&lt;/code&gt; corresponding to &lt;code&gt;par&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T16:35:42.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2622" PostTypeId="2" ParentId="2593" CreationDate="2014-12-05T16:37:54.327" Score="4" Body="&lt;p&gt;The &lt;code&gt;Data Science Toolkit&lt;/code&gt; is &lt;a href=&quot;http://www.datasciencetoolkit.org/&quot; rel=&quot;nofollow&quot;&gt;a powerful library&lt;/a&gt; (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called &lt;code&gt;RDSTK&lt;/code&gt; in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of your preferred language, Google Go, there's &lt;a href=&quot;https://code.google.com/p/go-wiki/wiki/Projects#Web_Libraries&quot; rel=&quot;nofollow&quot;&gt;a list&lt;/a&gt; of web-related libraries here which looks very useful.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T16:37:54.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2623" PostTypeId="1" CreationDate="2014-12-05T16:54:13.967" Score="2" ViewCount="493" Body="&lt;p&gt;The &lt;code&gt;mnlogit&lt;/code&gt; package in R allows for the fast estimation of multinomial logit models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The specification of forumlas is a bit different from most other regression models/packages in R, however. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the &lt;code&gt;Fish&lt;/code&gt; dataset as a reproducible example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; require(mnlogit)&#xA;Loading required package: mnlogit&#xA;&#xA;Package: mnlogit&#xA;Version: 1.1.1&#xA;Multinomial Logit Choice Models.&#xA;Scientific Computing Group, Sentrana Inc, 2013.&#xA;&#xA;&amp;gt; data(Fish, package ='mnlogit')&#xA;&amp;gt; head(Fish)&#xA;           mode   income     alt   price  catch chid&#xA;1.beach   FALSE 7083.332   beach 157.930 0.0678    1&#xA;1.boat    FALSE 7083.332    boat 157.930 0.2601    1&#xA;1.charter  TRUE 7083.332 charter 182.930 0.5391    1&#xA;1.pier    FALSE 7083.332    pier 157.930 0.0503    1&#xA;2.beach   FALSE 1250.000   beach  15.114 0.1049    2&#xA;2.boat    FALSE 1250.000    boat  10.534 0.1574    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to understand the difference between the model specification of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm &amp;lt;- formula(mode ~ 0 + price | income | catch)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm &amp;lt;- formula(mode ~ 0 + price | income + catch)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while the documentation covers the detail of such changes  in the general coeffcient area of the forumla (i.e. where &lt;code&gt;price&lt;/code&gt; is), I don't see an explanation of how operators like &lt;code&gt;+&lt;/code&gt; affect the alternative-specific area of the formula/code, relative to &lt;code&gt;|&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2014-12-05T16:54:13.967" Title="How to Interpret Multinomial Specification in R's `mnlogit` package" Tags="|r|logistic-regression|regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2624" PostTypeId="2" ParentId="2618" CreationDate="2014-12-05T17:13:14.073" Score="6" Body="&lt;p&gt;For many APIs (most I've seen) ratelimiting is a function of your API Key or OAuth credentials. (Google, Twitter, NOAA, Yahoo, Facebook, etc.) The good news is you won't need to spoof your IP, you just need to swap out credentials as they hit there rate limit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bit of shameless self promotion here but I wrote a python package specifically for handling this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/rawkintrevo/angemilner&quot; rel=&quot;nofollow&quot;&gt;https://github.com/rawkintrevo/angemilner&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pypi.python.org/pypi/angemilner/0.2.0&quot; rel=&quot;nofollow&quot;&gt;https://pypi.python.org/pypi/angemilner/0.2.0&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It requires a mongodb daemon and basically you make a page for each one of your keys. So you have 4 email addresses each with a separate key assigned. When you load the key in you specify the maximum calls per day and minimum time between uses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Load keys:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from angemilner import APIKeyLibrarian&#xA;l= APIKeyLibrarian()&#xA;l.new_api_key(&quot;your_assigned_key1&quot;, 'noaa', 1000, .2)&#xA;l.new_api_key(&quot;your_assigned_key2&quot;, 'noaa', 1000, .2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then when you run your scraper for instance the NOAA api:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations' &#xA;payload= {  'limit': 1000,&#xA;        'datasetid':  'GHCND', &#xA;        'startdate': '1999-01-01' }&#xA;&#xA;r = requests.get(url, params=payload, headers= {'token': 'your_assigned_key'})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;becomes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'&#xA;payload= {  'limit': 1000,&#xA;            'datasetid':  'GHCND',&#xA;            'startdate': '1999-01-01' }&#xA;&#xA;r = requests.get(url, params=payload, headers= {'token': l.check_out_api_key('noaa')['key']})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so if you have 5 keys, &lt;code&gt;l.check_out_api_key&lt;/code&gt; returns the key that has the least uses and waits until enough time has elapsed for it to be used again. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally to see how often your keys have been used / remaining useage available:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pprint(l.summary())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I didn't write this for R because most scraping is done in python (most of MY scraping).  It could be easily ported. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thats how you can &lt;em&gt;technically&lt;/em&gt; get around rate limiting.  &lt;em&gt;Ethically&lt;/em&gt; ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt; The example uses Google Places API &lt;a href=&quot;https://github.com/rawkintrevo/angemilner/blob/master/example2.py&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5247" LastEditorUserId="5247" LastEditDate="2014-12-05T18:53:24.863" LastActivityDate="2014-12-05T18:53:24.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2625" PostTypeId="2" ParentId="2593" CreationDate="2014-12-05T18:09:27.670" Score="2" Body="&lt;p&gt;If your only motivation for using Google Go is webscraping, and you want to do you ML in python, I would recommend the following stack:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.python-requests.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Python requests for scraping data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mongodb.org/&quot; rel=&quot;nofollow&quot;&gt;MongoDB for caching data&lt;/a&gt; (MongoDB's page oriented format makes it a natural home for storing JSON objects commonly returned by APIs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://api.mongodb.org/python/current/&quot; rel=&quot;nofollow&quot;&gt;pymongo for interfacing python and mongodb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn for doing your machine learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This all happens in python and you can extend it multiple processors with &#xA;&lt;a href=&quot;https://docs.python.org/2/library/multiprocessing.html&quot; rel=&quot;nofollow&quot;&gt;multiprocessing&lt;/a&gt; or to multiple nodes with django&lt;/p&gt;&#xA;" OwnerUserId="5247" LastEditorUserId="5247" LastEditDate="2014-12-05T18:43:57.927" LastActivityDate="2014-12-05T18:43:57.927" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2626" PostTypeId="2" ParentId="2432" CreationDate="2014-12-05T20:13:17.793" Score="0" Body="&lt;p&gt;If you are willing to pay for a vendor solution, &lt;a href=&quot;http://www.teradata.com/&quot; rel=&quot;nofollow&quot;&gt;Teradata&lt;/a&gt; is designed to solve the problem of large scale joins with low latency.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2014-12-05T20:13:17.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2627" PostTypeId="1" CreationDate="2014-12-06T00:41:24.933" Score="6" ViewCount="381" Body="&lt;p&gt;I'm trying to create 3D bars on this map. Can anyone please advise if this is possible, and how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://leafletjs.com/examples/choropleth.html&quot; rel=&quot;noreferrer&quot;&gt;http://leafletjs.com/examples/choropleth.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data: UFO sightings in the USA (location wise).&#xA;Count of these sightings per location will be the height of the 3D bar.&#xA;Base map is a choropleth with US population density values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't mind integrating Javascript or d3.js into the code to create the 3D bars.&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-07-10T10:39:59.713" Title="3D map using leaflet" Tags="|javascript|visualization|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2628" PostTypeId="1" CreationDate="2014-12-06T01:10:30.817" Score="2" ViewCount="589" Body="&lt;p&gt;Using SAS Studio (online, student version)...&#xA;Need to do a &quot;nested likelihood ratio test&quot; for a logistic regression. &#xA;Entirety of instructions are: &quot;Perform a nested likelihood ratio test comparing your full model (all predictors included)to a reduced model of interest.&quot;&#xA;The two models I have are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Proc Logistic Data=Project_C;&#xA;Model Dem (event='1') = VEP TIF Income NonCit Unemployed Swing;&#xA;Run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Proc Logistic Data=Project_C;&#xA;Model Dem (Event='1') = VEP TIF Income / clodds=Wald clparm=Wald expb rsquare;&#xA;Run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I honestly have no idea where to even start. &#xA;Any suggestions would be appreciated. &#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="5298" LastEditorUserId="1330" LastEditDate="2022-04-09T15:50:01.330" LastActivityDate="2022-04-09T15:50:01.330" Title="SAS Nested Likelihood Ratio Test for a Logistic Model" Tags="|logistic-regression|sas|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2629" PostTypeId="1" AcceptedAnswerId="2630" CreationDate="2014-12-06T06:53:14.617" Score="3" ViewCount="441" Body="&lt;p&gt;In recent years, the term &quot;data&quot; seems to have become a term widely used without specific definition. Everyone seems to use the phrase. Even people as technology-impaired as my grandparents use the term and seem to understand words like &quot;data breach.&quot; But I don't understand what makes &quot;data science&quot; a new discipline. Data has been the foundation of science for centuries. Without data, there would be no Mendel, no Schrödinger, etc. You can't have science without interpreting  and analyzing data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But clearly it means something. Everyone is talking about it.  So what exactly do people mean by data when they use terms like &quot;big data&quot; and why has this become a discipline in itself? Also, if it is an emerging discipline, where can I find more serious/in-depth information so I can better educate myself? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="5301" LastActivityDate="2014-12-06T13:02:19.373" Title="What is &quot;data science&quot;?" Tags="|bigdata|definitions|" AnswerCount="1" CommentCount="3" ClosedDate="2014-12-06T13:29:40.867" ContentLicense="CC BY-SA 3.0" />
  <row Id="2630" PostTypeId="2" ParentId="2629" CreationDate="2014-12-06T13:02:19.373" Score="5" Body="&lt;p&gt;I get asked this question all the time, so earlier this year I wrote an article (&lt;a href=&quot;http://oss4ds.com/what-is-data-science.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;What is Data Science?&lt;/strong&gt;&lt;/a&gt;) based on a presentation I've given a few times. Here's the gist...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, a few definitions of data science offered by others:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://twitter.com/josh_wills/status/198093512149958656&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Josh Wills&lt;/strong&gt; from &lt;strong&gt;Cloudera&lt;/strong&gt; says&lt;/a&gt; a data scientist is someone &quot;who is better at statistics than any software engineer and better at software engineering than any statistician.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://twitter.com/nivertech/status/180109930139893761&quot; rel=&quot;noreferrer&quot;&gt;A frequently-heard &lt;strong&gt;joke&lt;/strong&gt;&lt;/a&gt; is that a &quot;Data Scientist&quot; is a Data Analyst who lives in California.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://twitter.com/BigDataBorat/status/372350993255518208&quot; rel=&quot;noreferrer&quot;&gt;According to &lt;strong&gt;Big Data Borat&lt;/strong&gt;&lt;/a&gt;, Data Science is statistics on a Mac.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Drew Conway's&lt;/strong&gt; famous Data Science Venn Diagram&lt;/a&gt;, it's the intersection of Hacking Skills, Math &amp;amp; Statistics Knowledge, and Substantive Expertise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's another good definition I found on the &lt;a href=&quot;http://www.itproportal.com/2014/02/11/how-to-pick-a-data-scientist-the-right-way/&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;ITProPortal&lt;/strong&gt; blog&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;A data scientist is someone who understands the domains of programming, machine learning, data mining, statistics, and hacking&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's how we define Data Science at &lt;strong&gt;Altamira&lt;/strong&gt; (my current employer):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/YAUZy.png&quot; alt=&quot;data science diagram&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom four rows are the &lt;strong&gt;table stakes&lt;/strong&gt; -- the cost of admission just to play the game. These are foundational skills that all aspiring data scientists must obtain. Every data scientist must be a &lt;strong&gt;competent programmer&lt;/strong&gt;. He or she must also have a solid grasp of math, statistics, and &lt;strong&gt;analytic methodology&lt;/strong&gt;. Data science and &quot;&lt;strong&gt;big data&lt;/strong&gt;&quot; go hand-in-hand, so all data scientists need to be familiar with frameworks for distributed computing. Finally, data scientists must have a basic understanding of the domains in which they operate, as well as excellent communications skills and the ability to &lt;strong&gt;tell a good story with data&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these basics covered, the next step is to develop &lt;strong&gt;deep expertise&lt;/strong&gt; in one or more of the vertical areas. &quot;Data Science&quot; is really an umbrella term for a collection of interrelated techniques and approaches taken from a variety of disciplines, including mathematics, statistics, computer science, and software engineering. The goal of these diverse methods is to &lt;strong&gt;extract actionable intelligence&lt;/strong&gt; from data of all kinds, enabling clients to make better &lt;strong&gt;data-driven decisions&lt;/strong&gt;. No one person can ever possibly master all aspects of data science; doing so would require multiple lifetimes of training and experience. The best data scientists are therefore &quot;&lt;strong&gt;T-shaped&lt;/strong&gt;&quot; individuals -- that is, they possess a breadth of knowledge across all areas of data science, along with deep expertise in at least one. Accordingly, the best &lt;strong&gt;data science teams&lt;/strong&gt; bring together a set of individuals with complementary skillsets spanning the &lt;strong&gt;entire spectrum&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-06T13:02:19.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2631" PostTypeId="1" CreationDate="2014-12-06T15:04:03.823" Score="7" ViewCount="2320" Body="&lt;p&gt;For simplicity let's assume the feature space is the XY plane.&lt;/p&gt;&#xA;" OwnerUserId="5306" LastEditorUserId="13727" LastEditDate="2016-03-02T15:51:27.790" LastActivityDate="2020-08-17T18:55:17.380" Title="What would be a good way to use clustering for outlier detection?" Tags="|machine-learning|data-mining|clustering|anomaly-detection|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2632" PostTypeId="1" CreationDate="2014-12-06T17:56:53.157" Score="3" ViewCount="41" Body="&lt;p&gt;I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution time of a query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I'm inserting a huge amount of data, consider it to be 2-3GB, what are the methods to find out the time for the process to be completed.&lt;/p&gt;&#xA;" OwnerUserId="5310" LastEditorUserId="84" LastEditDate="2014-12-13T00:14:33.460" LastActivityDate="2022-05-02T06:00:45.083" Title="Timing sequence in MapReduce" Tags="|efficiency|map-reduce|performance|experiments|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2633" PostTypeId="1" CreationDate="2014-12-07T04:40:53.677" Score="0" ViewCount="1338" Body="&lt;p&gt;I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS.&#xA;I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm a student, I cannot afford much. Can anyone please suggest me some solution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it would be great if you could provide any links for setting up multi cluster hadoop with spark on AWS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I cannot try in GCE as my trial period is exhausted. &lt;/p&gt;&#xA;" OwnerUserId="5172" LastEditorUserId="84" LastEditDate="2015-02-28T13:28:18.943" LastActivityDate="2015-02-28T13:28:18.943" Title="Can hadoop with Spark be configured with 1GB RAM" Tags="|bigdata|apache-hadoop|nosql|aws|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2634" PostTypeId="1" CreationDate="2014-12-07T10:43:01.697" Score="2" ViewCount="160" Body="&lt;p&gt;Having:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a set of soft fuzzy classifiers (classification onto overlapping sets) $C_i(x) \to [0,1]$;&lt;/li&gt;&#xA;&lt;li&gt;a corresponding set of weak estimators $R_i(z)$ of the form $R_i(z) = \mathit{EX}(y\mid z)$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The estimators $R_i$ are just some kind of regression, Kalman or particle filters. The classifiers $C_i$ are fixed and static. How to make a strong estimator out of a weighted combination of the form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(x, z) = \sum_{i}C_i(x)R_i(z)Q_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words how to choose the weights $Q_i$? Is there some kind of online approach to this problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is brief description of a practical application. When an event $E$ is registered, multiple measurements are made. Based on these measurements, the classifiers $C_i$ make a soft assignment of the event to multiple overlapping categories. What we get is fit ratios for the soft clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now there is some chance that event $E$ may trigger a subsequent event $D$, depending on another variable $z$ -- independent from the event $E$. We know that all the soft cluster &quot;memberships&quot; may influence the probability of event $D$ being triggered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to estimate the probability that $E$ triggers $D$, given the $C_i$ fitness ratios and value of $z$.&lt;/p&gt;&#xA;" OwnerUserId="5314" LastEditorUserId="84" LastEditDate="2014-12-13T00:11:04.017" LastActivityDate="2014-12-13T00:11:04.017" Title="Linear combination of weak estimators over fuzzy classifiers?" Tags="|machine-learning|regression|online-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2635" PostTypeId="2" ParentId="2593" CreationDate="2014-12-07T11:29:14.057" Score="1" Body="&lt;p&gt;I am not 100% if a message queue library will be the right tool for this job but so far it looks to me so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a messaging library like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nsq.io&quot; rel=&quot;nofollow&quot;&gt;nsq&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://zeromq.org&quot; rel=&quot;nofollow&quot;&gt;zeromq&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mqtt.org&quot; rel=&quot;nofollow&quot;&gt;mqtt&lt;/a&gt; (?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can connect different processes operating on different environment through a TCP based protocol. As these systems run distributed it is possible to connect multiple nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;nsq&lt;/strong&gt; we even have a library in Python &lt;strong&gt;and&lt;/strong&gt; Go!&lt;/p&gt;&#xA;" OwnerUserId="5266" LastActivityDate="2014-12-07T11:29:14.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2636" PostTypeId="2" ParentId="2631" CreationDate="2014-12-07T12:16:10.207" Score="1" Body="&lt;p&gt;Perhaps you could cluster the items, then those items with the furthest distance from the midpoint of any cluster would be candidates for outliers.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-07T12:16:10.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2637" PostTypeId="2" ParentId="2633" CreationDate="2014-12-07T13:37:40.373" Score="4" Body="&lt;p&gt;So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much more reasonable is to simply run Spark locally on that instance without Hadoop at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I would question whether Spark is the right choice if you are definitely limited to such a small machine.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2014-12-07T13:37:40.373" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2639" PostTypeId="2" ParentId="2634" CreationDate="2014-12-07T15:39:00.947" Score="1" Body="&lt;p&gt;You might find a solution for this by checking out Viola &amp;amp; Jones face detection algorithm (and object detection in general) &lt;a href=&quot;http://www.cs.ubc.ca/~lowe/425/slides/13-ViolaJones.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.ubc.ca/~lowe/425/slides/13-ViolaJones.pdf&lt;/a&gt;. Particularly the AdaBoost algorithm for building a strong classifier from weak classifiers. &lt;a href=&quot;https://www.cs.princeton.edu/~schapire/papers/explaining-adaboost.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.cs.princeton.edu/~schapire/papers/explaining-adaboost.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this algorithm this is used particularly for feature selection, where it is wanted to select just the features that grouped together classify better, discarding the other as noisy features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach to obtain this strong classifier is having a set of examples X, a set of weights W and a set of expected results (classifications) Y. For a two-class example (e.g. face o no face image) the first weak classifier is selected and then the classification error is found between this classifier and the expected output. For the samples that were misclassified their weights are incremented, so the next weak classifier to find is more biased to neglect this wrong classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm (AdaBoost) converges when the sign of the sum of classifications for each sample outputs the correct classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = +1 -1 +1 +1&#xA;&#xA;WC1=+1 +1 +1 +1&#xA;WC2=-1 -1 -1 -1&#xA;WC3=+1 -1 +1 +1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the strong classifier is WC1+WC2+WC3:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SC=+1 -1 +1 +1 == Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this solves your question.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2014-12-07T15:39:00.947" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2640" PostTypeId="2" ParentId="2631" CreationDate="2014-12-07T15:46:40.670" Score="6" Body="&lt;p&gt;A very robust clustering algorithm against outliers is &lt;a href=&quot;http://www.comp.ita.br/%7Eforster/CC-222/material/fuzzyclust/fuzzy01492404.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PFCM from Bezdek&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy posibilistic clustering. This algorithm is particularly good at detecting outliers and avoiding them to influence the clusterization. So using PFCM you could find which points are identified as outliers and at the same time have a very robust fuzzy clustering of your data.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="98307" LastEditDate="2020-08-17T16:15:44.087" LastActivityDate="2020-08-17T16:15:44.087" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2641" PostTypeId="2" ParentId="2568" CreationDate="2014-12-07T16:17:20.613" Score="1" Body="&lt;p&gt;The following link contains a list of positive and negative polarised emotions on the scale of [-5, 5]. Just try to count up the scores based on the word matches and you can get the overall movie review score.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010&quot; rel=&quot;nofollow noreferrer&quot;&gt;AFINN&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1131" LastEditorUserId="85045" LastEditDate="2021-03-30T20:43:25.393" LastActivityDate="2021-03-30T20:43:25.393" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2642" PostTypeId="1" CreationDate="2014-12-07T16:24:43.997" Score="5" ViewCount="4474" Body="&lt;p&gt;Are there any algorithms which were developed using partial differential equations for tackling some of the machine learning problems? Most works I see online are in the field of computer vision and a few bizarre ones in topic modelling. But just curious if someone has used or seen it being used for some decision making process or classification problems?&lt;/p&gt;&#xA;" OwnerUserId="1131" LastActivityDate="2014-12-13T00:14:11.770" Title="Machine Learning &amp; Partial Differential Equations" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2643" PostTypeId="1" CreationDate="2014-12-07T16:31:57.913" Score="4" ViewCount="1007" Body="&lt;p&gt;I'm new to apache spark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to configure multi cluster spark without hadoop?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, can you please provide the steps. &#xA;I would like to create clusters on Google Compute Engine  (1-master, 1-worker)&lt;/p&gt;&#xA;" OwnerUserId="5172" LastActivityDate="2015-05-16T20:06:16.560" Title="How to set up multi cluster spark without hadoop on Google Compute engine" Tags="|bigdata|apache-hadoop|scala|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="2645" PostTypeId="1" CreationDate="2014-12-08T10:47:52.340" Score="1" ViewCount="771" Body="&lt;p&gt;Say I used spectral clustering to cluster a data-set $D$ of points $X_0 - X_n$ into a number $C$ of clusters. How can I efficiently assign a new single point $X_{n+1}$ to his convenient cluster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to do the classification from the beginning (destroy all the clusters and apply the algorithm to the data-set $X_0 - X_{n+1}$), or is there an optimized way to extend to the point $X_{n+1}$?&lt;/p&gt;&#xA;" OwnerUserId="5332" LastEditorUserId="84" LastEditDate="2014-12-12T21:01:42.370" LastActivityDate="2014-12-12T21:01:42.370" Title="Assign new point to a class using spectral clustering" Tags="|machine-learning|data-mining|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2646" PostTypeId="1" AcceptedAnswerId="2648" CreationDate="2014-12-08T14:51:27.613" Score="20" ViewCount="23917" Body="&lt;p&gt;Are there any articles or discussions about extracting part of text that holds the most of information about current document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have a large corpus of documents from the same domain. There are parts of text that hold the key information what single document talks about. I want to extract some of those parts and use them as kind of a summary of the text. Is there any useful documentation about how to achieve something like this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be really helpful if someone could point me into the right direction what I should search for or read to get some insight in work that might have already been done in this field of Natural language processing.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2019-03-19T15:01:33.430" Title="Extract most informative parts of text from documents" Tags="|nlp|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2648" PostTypeId="2" ParentId="2646" CreationDate="2014-12-08T15:48:25.647" Score="26" Body="&lt;p&gt;What you're describing is often achieved using a simple combination of &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot;&gt;TF-IDF&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Automatic_summarization#Extraction-based_summarization&quot;&gt;extractive summarization&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell, TF-IDF tells you the relative importance of each word in each document, in comparison to the rest of your corpus. At this point, you have a score for each word in each document approximating its &quot;importance.&quot; Then you can use these individual word scores to compute a composite score for each sentence by summing the scores of each word in each sentence. Finally, simply take the top-N scoring sentences from each document as its summary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Earlier this year, I put together an iPython Notebook that culminates with an implementation of this in Python using NLTK and Scikit-learn: &lt;a href=&quot;https://github.com/charlieg/A-Smattering-of-NLP-in-Python&quot;&gt;A Smattering of NLP in Python&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-08T15:48:25.647" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2649" PostTypeId="2" ParentId="2645" CreationDate="2014-12-08T20:55:55.677" Score="1" Body="&lt;p&gt;I think you do not, since I was able to find several papers that proposed algorithms for the same:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ifp.illinois.edu/~hning2/papers/incremental_spectral_clustering.pdf&quot; rel=&quot;nofollow&quot;&gt;Incremental spectral clustering by efficiently updating the eigen-system&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0925231214004433&quot; rel=&quot;nofollow&quot;&gt;Incremental kernel spectral clustering for online learning of non-stationary data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dx.doi.org/10.1109/PDCAT.2011.4&quot; rel=&quot;nofollow&quot;&gt;A Fast Incremental Spectral Clustering for Large Data Sets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Such algorithms are called &lt;em&gt;sequential&lt;/em&gt;, &lt;em&gt;incremental&lt;/em&gt;, &lt;em&gt;streaming&lt;/em&gt;, or &lt;em&gt;online&lt;/em&gt;. Armed with this knowledge, you can find more papers on your own.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2014-12-08T20:55:55.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2650" PostTypeId="2" ParentId="2642" CreationDate="2014-12-08T22:32:05.817" Score="1" Body="&lt;p&gt;Neil is correct. There are partial derivatives evwrywhere in gradient computation for machine learning models training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance you can look at the gradient descent method used in the backpropagation method for a neural network. The course from AndrewNg on coursera describes it very well.&lt;/p&gt;&#xA;" OwnerUserId="5342" LastEditorUserId="5342" LastEditDate="2014-12-13T00:14:11.770" LastActivityDate="2014-12-13T00:14:11.770" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2651" PostTypeId="1" CreationDate="2014-12-08T22:37:32.777" Score="29" ViewCount="10174" Body="&lt;p&gt;I am looking for a paper detailing the very basics of deep learning. Ideally like the Andrew Ng course for deep learning. Do you know where I can find this ?&lt;/p&gt;&#xA;" OwnerUserId="5342" LastActivityDate="2020-08-20T18:59:26.887" Title="Deep learning basics" Tags="|machine-learning|deep-learning|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2652" PostTypeId="1" AcceptedAnswerId="5132" CreationDate="2014-12-09T00:58:03.057" Score="3" ViewCount="3198" Body="&lt;p&gt;&lt;br&gt;&#xA;Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R:&lt;br&gt;&#xA;-starting Hadoop services.,&lt;br&gt;&#xA;-loading rJava and Rhipe packages by library function.&lt;br&gt;&#xA;-Calling rhinit() to initialize Rhipe.&lt;br&gt;&#xA;the problem here is when I call rhinit() funtion, it show this error:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Initializing Rhipe v0.73&lt;br&gt;&#xA;    Error in .jnew(&quot;org/godhuli/rhipe/PersonalServer&quot;) : &#xA;     java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream'&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;please some helps to fixe this problem.&lt;/p&gt;&#xA;" OwnerUserId="4705" LastActivityDate="2015-08-13T00:36:49.403" Title="java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream" Tags="|r|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2653" PostTypeId="2" ParentId="2651" CreationDate="2014-12-09T01:19:32.290" Score="19" Body="&lt;p&gt;The subject is new so most of the wisdom is scattered in papers, but here are two recent books:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.iro.umontreal.ca/%7Ebengioy/dlbook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt;, Yoshua Bengio, Ian J. Goodfellow, Aaron Courville.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/209355/DeepLearning-NowPublishing-Vol7-SIG-039.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning: Methods and Applications&lt;/a&gt;, Li Deng and Dong Yu.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;And &lt;a href=&quot;http://deeplearning.net/tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;some practical material&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ACL 2012 + NAACL 2013 Tutorial: &lt;a href=&quot;http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning for NLP (without Magic)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastEditorUserId="98307" LastEditDate="2020-08-20T18:58:20.740" LastActivityDate="2020-08-20T18:58:20.740" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2654" PostTypeId="1" AcceptedAnswerId="2664" CreationDate="2014-12-09T02:28:51.430" Score="1" ViewCount="136" Body="&lt;p&gt;I'm wondering if there is a web framework well suited for placing recommendations on content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases, a data scientist goes through after the fact and builds (or uses) a completely different tool to create recommendations. This involves analyzing traffic logs, a history of shopping cart data, ratings, and so forth. It usually comes from multiples sources (the web server, the application's database, Google Analytics, etc) and then has to be cleaned up and processed, THEN delivered back to the application in way it understands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a web framework on the market which handles collecting this data up front, as to minimize the retrospective data wrangling?&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-12-10T02:05:33.430" Title="Web Framework Built for Recommendations" Tags="|predictive-modeling|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2655" PostTypeId="2" ParentId="2651" CreationDate="2014-12-09T08:26:57.997" Score="14" Body="&lt;p&gt;Neural Networks and Deep Learning by Michael Nielsen. The book is still in progress, but it looks quite interesting and promising. And it's free! &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s the link.&lt;/p&gt;&#xA;&lt;p&gt;There are only 5 chapters so far, and the most of them talk about usual neural networks, but it's still worth having a look.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; the book has been finished!&lt;/p&gt;&#xA;" OwnerUserId="816" LastEditorUserId="98307" LastEditDate="2020-08-20T18:58:36.587" LastActivityDate="2020-08-20T18:58:36.587" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="2656" PostTypeId="2" ParentId="2491" CreationDate="2014-12-09T08:30:28.000" Score="3" Body="&lt;p&gt;Found it myself.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Go to context menu right clicking to the dimension field.&lt;/li&gt;&#xA;&lt;li&gt;Go to &lt;strong&gt;Aliases...&lt;/strong&gt; and change the labels.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/qKq73.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2014-12-09T08:30:28.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2658" PostTypeId="1" CreationDate="2014-12-09T20:49:37.093" Score="9" ViewCount="2253" Body="&lt;p&gt;I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: I need to figure out the number of complaints on &lt;strong&gt;&lt;em&gt;Software&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;Hardware&lt;/em&gt;&lt;/strong&gt; side of my product from the customer complaints. Currently, I am using excel to do the data analysis which do seek a significant amount of manual work to get a tag name to the complaints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way in NLP to build and train a model to automate this process? I have been reading stuffs about NLP for the past couple of days and it looks like NLP has a lot of good features to get a head start in addressing this issue. Could someone please guide me with the way I should use NLP to address this issue?&lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2020-08-17T01:00:02.987" Title="Using NLP to automate the categorization of user description" Tags="|data-mining|classification|nlp|categorical-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2659" PostTypeId="1" AcceptedAnswerId="2662" CreationDate="2014-12-09T21:13:10.797" Score="1" ViewCount="7530" Body="&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; It was pointed out in the Answers-section that I am confusing k-means and kNN. Indeed I was thinking about kNN but wrote k-means since I'm still new to this topic and confuse the terms quite often. So here is the changed question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was looking at kNN today and something struck me as odd or - to be more precise - something that I was unable to find information about namely the following situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that we pick kNN for some dataset. I want to remain as general as possible, thus $k$ will not be specified here. Further we select, at some point, an observation where the number of neighbors that fulfill the requirement to be in the neighbourhood are actually more than the specified $k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What criterion/criteria should be applied here if we are restricted to use the specific K and thus cannot alter the structure of the neighborhood (number of neighbors). Which observations will be left out and why? Also is this a problem that occurs often, or is it something of an anomaly?&lt;/p&gt;&#xA;" OwnerUserId="5356" LastEditorUserId="84" LastEditDate="2014-12-12T21:05:30.020" LastActivityDate="2014-12-12T21:05:30.020" Title="kNN - what happens if more than K observation have the same distance to the centroid of the cluster" Tags="|machine-learning|classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2660" PostTypeId="2" ParentId="2658" CreationDate="2014-12-09T21:19:23.747" Score="7" Body="&lt;p&gt;One way to handle this is to use 'supervised classification'. In this model, you manually classify a subset of the data and use it to train your algorithm. Then, you feed the remaining data into your software to classify it.&lt;/p&gt;&#xA;&lt;p&gt;This is accomplished with NLTK for Python (nltk.org).&lt;/p&gt;&#xA;&lt;p&gt;If you are simply looking for strings like &amp;quot;hardware&amp;quot; and &amp;quot;software&amp;quot;, this is a simple use case, and you will likely get decent results using a 'feature extractor', which informs your classifier which phrases in the document are relevant.&lt;/p&gt;&#xA;&lt;p&gt;While it's possible to implement an automated method for finding the keywords, it sounds like you have a list in mind already, so you can skip that step and just use the tags you are aware of. (If your results aren't satisfactory the first time, this is something you might try later on).&lt;/p&gt;&#xA;&lt;p&gt;That's an overview for getting started. If you are unhappy with the initial results, you can refine your classifier by introducing more complex methods, such as sentence segmentation, identification of dialogue act types, and decision trees. The sky is the limit (or more likely, your time is the limit)!&lt;/p&gt;&#xA;&lt;p&gt;More info &lt;a href=&quot;http://www.nltk.org/book/ch06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastEditorUserId="98307" LastEditDate="2020-08-17T01:00:02.987" LastActivityDate="2020-08-17T01:00:02.987" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="2661" PostTypeId="2" ParentId="2659" CreationDate="2014-12-09T21:33:31.200" Score="3" Body="&lt;p&gt;K-means does not make an assumption regarding how many observations &lt;strong&gt;should be&lt;/strong&gt; assigned to each cluster. &lt;code&gt;K&lt;/code&gt; is simply the number of clusters one chooses to generate. During each iteration, each observation is assigned to the cluster having the nearest mean. There is no assumption that all clusters should have a comparable number of observations assigned (i.e., for &lt;code&gt;N&lt;/code&gt; observations, there is no expectation that each cluster should have ~ &lt;code&gt;N/K&lt;/code&gt; observations assigned).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is quite possible that the numbers of observations in the various clusters are highly imbalanced. This can be due to the distribution of the data, the number of clusters chosen (&lt;code&gt;K&lt;/code&gt;), or even how the cluster means are initialized.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2014-12-09T21:33:31.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2662" PostTypeId="2" ParentId="2659" CreationDate="2014-12-09T22:32:22.660" Score="4" Body="&lt;p&gt;You are mixing up kNN classification and k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing wrong with having more than k observations near a center in k-means. In fact, this it the usual case; you shouldn't choose k too large. If you have 1 million points, a k of 100 may be okay. &lt;em&gt;K-means does not guarantee clusters of a particular size&lt;/em&gt;. Worst case, clusters in k-means can have only one element (outliers) or even disappear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you probably meant to write, but got mixed up, is &lt;strong&gt;what to do if a point is at the same distance to two centers&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a statistical point of view, it doesn't matter. Both have the same squared error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From an implementation point of view, choose any &lt;em&gt;deterministic&lt;/em&gt; rule, so that your algorithm converges and doesn't go into an infinite loop of reassignment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; with respect to kNN classification:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to resolve this, that will surprisingly often work just as good as the other, without a clear advantage of one over the other:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;randomly choose a winner from the tied objects&lt;/li&gt;&#xA;&lt;li&gt;take all into account with equal weighting&lt;/li&gt;&#xA;&lt;li&gt;if you have m objects at the same distance where you expected only r, then put a weight of r/k on each of them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;E.g. k=5.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distance   label   weight&#xA;    0        A       1&#xA;    1        B       1&#xA;    1        A       1&#xA;    2        A      2/3&#xA;    2        B      2/3&#xA;    2        B      2/3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields A=2.66, B=2.33&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason that randomly choosing works just as good as the others is that usually, the majority decision in kNN will not be changed by contributions with a weight of less than 1; in particular when k is larger than say 10.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2014-12-10T16:21:24.247" LastActivityDate="2014-12-10T16:21:24.247" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="2663" PostTypeId="2" ParentId="2631" CreationDate="2014-12-09T22:33:38.640" Score="2" Body="&lt;p&gt;Gaussian mixture modeling can - if your data is nicely gaussian-like - be used for outlier detection. Points with a low density in &lt;em&gt;every&lt;/em&gt; cluster are likely to be outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Works well in idealistic scenarios.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2014-12-09T22:33:38.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2664" PostTypeId="2" ParentId="2654" CreationDate="2014-12-10T01:59:48.700" Score="1" Body="&lt;p&gt;I haven't seen anything like that and very much doubt that such frameworks exist, at least, as complete frameworks. The reason for this is IMHO the fact that data &lt;em&gt;transformation&lt;/em&gt; and &lt;em&gt;cleaning&lt;/em&gt; is very &lt;strong&gt;domain-&lt;/strong&gt; and &lt;strong&gt;project-specific&lt;/strong&gt;. Having said that, there are multiple tools that can help with these activities in terms of &lt;em&gt;partial automation&lt;/em&gt; and &lt;em&gt;integration&lt;/em&gt; with and between existing statistical and Web frameworks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for &lt;strong&gt;Python&lt;/strong&gt;, the use of &lt;em&gt;data manipulation&lt;/em&gt; library &lt;code&gt;pandas&lt;/code&gt; as well as &lt;em&gt;machine learning&lt;/em&gt; library &lt;code&gt;scikit-learn&lt;/code&gt; can be easily integrated with Web frameworks (especially Python-based, but not necessarily), as these libraries are also Python-based. These and other Python data science tools that might be of interest can be found here: &lt;a href=&quot;http://pydata.org/downloads&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pydata.org/downloads&lt;/a&gt;. Specifically, for cleaning and pre-processing tasks, which you asked about, &lt;code&gt;pandas&lt;/code&gt; seem to be the first tool to explore. Again, for Python, the following discussion on StackOverflow on methods and approaches might be helpful: &lt;a href=&quot;https://stackoverflow.com/q/14262433/2872891&quot;&gt;https://stackoverflow.com/q/14262433/2872891&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider an example of &lt;strong&gt;another platform&lt;/strong&gt;. The use of &lt;code&gt;pandas&lt;/code&gt; for data transformation and cleaning is rather &lt;em&gt;low-level&lt;/em&gt;. The platform that I like very much and currently use as the platform of choice for data science tasks is &lt;code&gt;R&lt;/code&gt;. Rich &lt;em&gt;ecosystem of R packages&lt;/em&gt; especially shines in the area of data transformation and cleaning. This is because, in addition to very flexible low-level methods of performing these tasks, there are some R packages, which take a &lt;em&gt;higher-level approach&lt;/em&gt; to the problem, which may potentially improve developer's productivity and decrease the amount of defects. In particular, I'm talking about two packages, which I find very promising: &lt;code&gt;editrules&lt;/code&gt; and &lt;code&gt;deducorrect&lt;/code&gt;. You can find more detailed information about these and other R packages for data transformation and cleaning in my another answer here on &lt;em&gt;Data Science StackExchange&lt;/em&gt; (paper that I reference in the last link there could be especially useful, as it presents an &lt;em&gt;approach&lt;/em&gt; to data transformation and cleaning that is &lt;em&gt;generic&lt;/em&gt; enough, so that could be used as a &lt;strong&gt;framework&lt;/strong&gt; for this on any decent platform): &lt;a href=&quot;https://datascience.stackexchange.com/a/722/2452&quot;&gt;https://datascience.stackexchange.com/a/722/2452&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;: On the topic of &lt;em&gt;recommender systems&lt;/em&gt; and their integration with data wrangling tools and Web frameworks, you may find my other answer here on DS SE useful: &lt;a href=&quot;https://datascience.stackexchange.com/a/836/2452&quot;&gt;https://datascience.stackexchange.com/a/836/2452&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-12-10T02:05:33.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2667" PostTypeId="1" CreationDate="2014-12-10T04:33:31.350" Score="6" ViewCount="3551" Body="&lt;p&gt;How can I connect to Titan database from Python ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I understand is that Titan (Graph database) provides an interface (Blueprint) to Cassandra (Column Store) and &#xA;bulb is a python interface to graph DB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how can I start programming in python to connect with titan DB?&#xA;Is there any good documentation/tutorial available ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2014-12-11T09:03:16.463" LastActivityDate="2015-05-27T03:38:45.067" Title="Python interface to Titan Database" Tags="|python|databases|nosql|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2668" PostTypeId="1" AcceptedAnswerId="3696" CreationDate="2014-12-10T06:37:17.193" Score="15" ViewCount="607" Body="&lt;p&gt;I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also suited for smaller data sets? &lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2021-03-11T20:14:30.817" Title="Data Science Tools Using Scala" Tags="|scalability|scala|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2669" PostTypeId="2" ParentId="2427" CreationDate="2014-12-10T07:12:33.640" Score="0" Body="&lt;p&gt;It is important to see all the rules if one of the states of target column is more important than others. For example, if you are predicting fraudulent transactions, you might want to flag something as fraud even if is has 5% probability.&lt;/p&gt;&#xA;" OwnerUserId="5366" LastEditorUserId="84" LastEditDate="2014-12-12T20:00:23.620" LastActivityDate="2014-12-12T20:00:23.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2670" PostTypeId="1" AcceptedAnswerId="2679" CreationDate="2014-12-10T10:15:00.940" Score="15" ViewCount="4334" Body="&lt;p&gt;I'm trying to find an equivalent of Hinton Diagrams for multilayer networks to plot the weights during training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trained network is somewhat similar to a Deep SRN, i.e. it has a high number of multiple weight matrices which would make the simultaneous plot of several Hinton Diagrams visually confusing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a good way to visualize the weight update process for recurrent networks with multiple layers? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't found much papers on the topic. I was thinking to display time-related information on the weights per layer instead if I can't come up with something. E.g. the weight-delta over time for each layer (omitting the use of every single connection). PCA is another possibility, though I'd like to not produce much additional computations, since the visualization is done online during training.&lt;/p&gt;&#xA;" OwnerUserId="5316" LastActivityDate="2020-08-05T14:57:38.660" Title="Visualizing deep neural network training" Tags="|machine-learning|neural-network|visualization|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2671" PostTypeId="2" ParentId="2659" CreationDate="2014-12-10T13:54:43.770" Score="2" Body="&lt;p&gt;[Since you've updated your question to refer to a different algorithm (changed k-means to kNN), I'm adding this as a separate answer specifically for kNN.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears you may still be confusing &lt;code&gt;kNN&lt;/code&gt; with &lt;code&gt;k-means&lt;/code&gt;. &lt;code&gt;kNN&lt;/code&gt; does not have multiple stages of refinement, nor does it require computing centroids. It is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Lazy_learning&quot; rel=&quot;nofollow&quot;&gt;lazy learner&lt;/a&gt; that classifies a new observation by examining the &lt;code&gt;k&lt;/code&gt; training observations that are closest to the new observation and picks whichever class is most prevalent among those &lt;code&gt;k&lt;/code&gt; training observations. Note that the distances are &lt;em&gt;not&lt;/em&gt; relative to a cluster centroid. You don't have to worry about there being more than &lt;code&gt;k&lt;/code&gt; training observations in the neighborhood of the new observation because the neighborhood isn't based on a threshold distance - it is defined simply by the &lt;code&gt;k&lt;/code&gt; points nearest to the new observation (i.e., it changes for each new observation being evaluated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A possible pathological case is when multiple training observations lie at &lt;em&gt;exactly&lt;/em&gt; the same distance from the new observation, which would require you to evaluate more than &lt;code&gt;k&lt;/code&gt; neighbors. But you would still simply pick the class that is most prevalent among that group of observations.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2014-12-10T13:54:43.770" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2673" PostTypeId="1" CreationDate="2014-12-10T14:18:36.933" Score="3" ViewCount="1315" Body="&lt;p&gt;I'm currently going into the world of machine learning and Neural Networks, thanks to &lt;a href=&quot;https://github.com/cazala/synaptic&quot; rel=&quot;nofollow&quot;&gt;synaptic (js)&lt;/a&gt; that interests me a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I read a lot, wikipedia links and &lt;a href=&quot;https://github.com/cazala/synaptic/wiki/Neural-Networks-101&quot; rel=&quot;nofollow&quot;&gt;synaptic's NN 101&lt;/a&gt;, but there's a lot of basics questions that I don't understand (but I'd like to) in the use of machine learning (NN) and the point of these technologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, I wan't my network to (kind of) learn (something like) gravity, so to train it I set in input 10 objects with a mass, and a position x, y (and z) and I set output the new x, y (and z) of each objects.&#xA;I guess I should give it several configurations and everything but &lt;strong&gt;here is the question; can it, then, be able to compute the interactions between 10000, 100000 objects?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this stage in my learning, what I don't clearly get is what is the point of teaching/training neurons to compute XOR like it's shown in &lt;a href=&quot;https://github.com/cazala/synaptic#documentation&quot; rel=&quot;nofollow&quot;&gt;synaptic's documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var trainingSet = [&#xA;    {&#xA;        input: [0,0],&#xA;        output: [0]&#xA;    },&#xA;    {&#xA;        input: [0,1],&#xA;        output: [1]&#xA;    },&#xA;    {&#xA;        input: [1,0],&#xA;        output: [1]&#xA;    },&#xA;    {&#xA;        input: [1,1],&#xA;        output: [0]&#xA;    },&#xA;];&#xA;&#xA;var trainer = new Trainer(myNetwork);&#xA;trainer.train(trainingSet);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Were we just give it all the possible inputs and outputs to a XOR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, as I'm all new to the technologies I think my questions are full of non-sense and everything but thanks for reading and help you might bring :)&lt;/p&gt;&#xA;" OwnerUserId="5369" LastActivityDate="2022-04-22T13:15:54.290" Title="Training Neural Networks with unknown length of input" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2675" PostTypeId="2" ParentId="2658" CreationDate="2014-12-10T15:37:41.853" Score="1" Body="&lt;p&gt;Sheldon is correct, this sounds like a fairly typical use case for supervised classification. If all of your customer complaints are either software or hardware (i.e., zero individual complaints cover both categories, and zero are outside these two classes), then all you need is a binary classifier, which makes things simpler than they otherwise could be.&lt;/p&gt;&#xA;&lt;p&gt;If you're looking for a Java-based NLP toolkit that supports something like this, you should check out the &lt;a href=&quot;http://nlp.stanford.edu/software/classifier.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Classifier&lt;/a&gt; -- it's licensed as open source software under the GPL.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-nlp.stanford.edu/wiki/Software/Classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Their wiki page&lt;/a&gt; should help you get started using the classifier -- keep in mind that you'll need to manually annotate a large sample of your data as a training set, as Sheldon mentioned.&lt;/p&gt;&#xA;" OwnerUserId="819" LastEditorUserId="98307" LastEditDate="2020-08-17T00:59:48.127" LastActivityDate="2020-08-17T00:59:48.127" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="2676" PostTypeId="2" ParentId="2673" CreationDate="2014-12-10T15:41:03.987" Score="1" Body="&lt;p&gt;&lt;strong&gt;Gravity&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically you'd like to find a function that maps an input (an object with a mass and a location) to an output (new location). It's not necessary to have one set of input neurons for each different object. It's sufficient to encode the input variables generically for all objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like a function: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f( x, y, z, mass ) \to ( x_n, y_n, z_n )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;XOR&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training the XOR function is special as the early neural networks, the perceptrons, were unable to learn the XOR function (because it is not linear separable). Multi-Layer Perceptrons however are able to learn the XOR function. This is just to demonstrate that the NN implementation of Synaptic is capable of learning problems that are not linear separable&lt;/p&gt;&#xA;" OwnerUserId="3132" LastEditorUserId="84" LastEditDate="2014-12-12T23:53:01.607" LastActivityDate="2014-12-12T23:53:01.607" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="2677" PostTypeId="1" AcceptedAnswerId="2678" CreationDate="2014-12-10T16:08:03.537" Score="8" ViewCount="1526" Body="&lt;p&gt;I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found out that &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;tf-idf&lt;/a&gt; could be very useful for this. However, I was unsure as to &lt;em&gt;when&lt;/em&gt; exactly to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assumming a website that is concerned with a specific topic makes repeated mention of it, this was my current process:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Retrieve site content, parse for plain text &lt;/li&gt;&#xA;&lt;li&gt;Normalize and stem content &lt;/li&gt;&#xA;&lt;li&gt;Tokenize into unigrams (maybe bigrams too)&lt;/li&gt;&#xA;&lt;li&gt;Retrieve a count of each unigram for the given document, filtering low length and low occurrence words&lt;/li&gt;&#xA;&lt;li&gt;Train a classifier such as NaiveBayes on the resulting set&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is the following: &lt;strong&gt;Where would tf-idf fit in here&lt;/strong&gt;? Before normalizing/stemming? After normalizing but before tokenizing? After tokenizing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insight would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon closer inspection, I think I may have run into a misunderstanding at to how TF-IDF operates. At the above step 4 that I describe, would I have to feed the &lt;em&gt;entirety&lt;/em&gt; of my data into TF-IDF at once? If, for example, my data is as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[({tokenized_content_site1}, category_string_site1), &#xA; ({tokenized_content_site2}, category_string_site2), &#xA;...&#xA; ({tokenized_content_siten}, category_string_siten)}]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, the outermost structure is a list, containing tuples, containing a dictionary (or hashmap) and a string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would I have to feed the &lt;em&gt;entirety&lt;/em&gt; of that data into the TF-IDF calculator at once to achieve the desired effect? Specifically, I have been looking at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt; TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse.&lt;/p&gt;&#xA;" OwnerUserId="5199" LastEditorUserId="5199" LastEditDate="2014-12-11T16:38:53.367" LastActivityDate="2014-12-11T16:38:53.367" Title="Document classification: tf-idf prior to or after feature filtering?" Tags="|classification|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="2678" PostTypeId="2" ParentId="2677" CreationDate="2014-12-10T17:36:20.550" Score="8" Body="&lt;p&gt;As you've described it, Step 4 is where you want to use TF-IDF. Essentially, TD-IDF will count each term in each document, and assign a score given the relative frequency across the collection of documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's one big step missing from your process, however: annotating a training set. Before you train your classifier, you'll need to manually annotate a sample of your data with the labels you want to be able to apply automatically using the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make all of this easier, you might want to consider using the &lt;a href=&quot;http://nlp.stanford.edu/software/classifier.shtml&quot;&gt;Stanford Classifier&lt;/a&gt;. It will perform the feature extraction and build the classifier model (supporting several different machine learning algorithms), but you'll still need to annotate the training data by hand.&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-10T17:36:20.550" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="2679" PostTypeId="2" ParentId="2670" CreationDate="2014-12-10T19:50:17.797" Score="10" Body="&lt;p&gt;The closes thing I know is &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/&quot;&gt;ConvNetJS&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ConvNetJS is a Javascript library for training Deep Learning models (mainly Neural Networks) entirely in your browser. Open a tab and you're training. No software requirements, no compilers, no installations, no GPUs, no sweat.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Demos on this site plot weighs and how do they change with time (bear in mind, its many parameters, as practical networks do have a lot of neurons). Moreover, if you are not satisfied with their plotting, there is access to networks parameters and you can plot as you wish (since it is JavaScript).&lt;/p&gt;&#xA;" OwnerUserId="289" LastActivityDate="2014-12-10T19:50:17.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="2682" PostTypeId="1" AcceptedAnswerId="4941" CreationDate="2014-12-10T22:47:49.880" Score="4" ViewCount="5947" Body="&lt;p&gt;I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at scaling to a larger data set, I am not confident No4J + Cypher will be performant. Spark has the GraphX project, which I have not used in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody switched from Neo4J to Spark GraphX? Do the use cases overlap, aside from scalability? Or, does GraphX address a completely different problem set than Neo4J?&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-02-02T07:31:51.050" Title="Use Cases of Neo4J and Spark GraphX" Tags="|scalability|graphs|neo4j|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3685" PostTypeId="2" ParentId="2670" CreationDate="2014-12-11T07:08:00.540" Score="5" Body="&lt;p&gt;Based on my cursory understanding of the topics, associated with your question, I think that &lt;strong&gt;&lt;a href=&quot;https://gephi.github.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gephi&lt;/a&gt;&lt;/strong&gt; (the original gephi.org link redirects there) should be able to handle &lt;em&gt;neural network dynamic visualization&lt;/em&gt;. It seems that, in order to achieve your goal, you need to &lt;strong&gt;&lt;a href=&quot;https://forum.gephi.org/viewtopic.php?t=1875&quot; rel=&quot;nofollow noreferrer&quot;&gt;stream&lt;/a&gt;&lt;/strong&gt; your graph(s) with corresponding weights. For &lt;em&gt;streaming&lt;/em&gt;, you most likely will need this &lt;em&gt;&lt;a href=&quot;https://marketplace.gephi.org/plugin/graph-streaming&quot; rel=&quot;nofollow noreferrer&quot;&gt;plug-in&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;: You may also find useful &lt;a href=&quot;http://web.stanford.edu/group/sonia.&quot; rel=&quot;nofollow noreferrer&quot;&gt;SoNIA software&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="98307" LastEditDate="2020-08-05T14:57:38.660" LastActivityDate="2020-08-05T14:57:38.660" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="3687" PostTypeId="1" CreationDate="2014-12-11T13:37:29.430" Score="1" ViewCount="758" Body="&lt;p&gt;I am trying to connect with cassandra in R using &lt;a href=&quot;http://cran.r-project.org/web/packages/RJDBC/index.html&quot; rel=&quot;nofollow&quot;&gt;RJDBC&lt;/a&gt;. When I execute &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;casscon &amp;lt;- dbConnect(cassdrv, &quot;jdbc:cassandra://ipaddrs:9160/demodb&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in .jcall(drv@jdrv, &quot;Ljava/sql/Connection;&quot;, &quot;connect&quot;, as.character(url)[1],&#xA;: java.lang.StringIndexOutOfBoundsException: String index out of range: -1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can't figure out the problem. I need a solution for this.&lt;/p&gt;&#xA;" OwnerUserId="5161" LastEditorUserId="2961" LastEditDate="2014-12-12T04:13:53.157" LastActivityDate="2014-12-12T04:13:53.157" Title="R-Cassandra connection with RJDBC" Tags="|r|databases|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3688" PostTypeId="1" CreationDate="2014-12-11T19:23:55.907" Score="10" ViewCount="1458" Body="&lt;p&gt;I have a corpus of text with a corresponding topics. For example &lt;code&gt;&quot;A rapper Tupac was shot in LA&quot;&lt;/code&gt; and it was labelled as &lt;code&gt;[&quot;celebrity&quot;, &quot;murder&quot;]&lt;/code&gt;. So basically each vector of features can have many labels (not the same amount. The first feature vector can have 3 labels, second 1, third 5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I would have just one label corresponded to each text, I would try a &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot;&gt;Naive Bayes classifier&lt;/a&gt;, but I do not really know how should I proceed if I can have many labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to transform Naive Bayes into multi label classification problem (if there is a better approach - please let me know)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S.&lt;/strong&gt; few things about the data I have.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;approximately 10.000 elements in the dataset&lt;/li&gt;&#xA;&lt;li&gt;text is approximately 2-3 sentences&lt;/li&gt;&#xA;&lt;li&gt;maximum 7 labels per text&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="6387" LastActivityDate="2014-12-12T00:13:10.680" Title="Multiple labels in supervised learning algorithm" Tags="|machine-learning|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3689" PostTypeId="2" ParentId="2668" CreationDate="2014-12-11T20:27:27.847" Score="4" Body="&lt;p&gt;From listening to presentations by Martin Odersky, the creator of Scala, it is especially well suited for building highly scalable systems by leveraging functional programming constructs in conjuction with object orientation and flelxible syntax. It is also useful for development of small systems and rapid prototyping because it takes less lines of code than some other languages and it has an interactive mode for fast feedback. One notable Scala framework is Akka which uses the actor model of concurrent computation. Many of Odersky's presentations are on YouTube and there is a list of tools implemented with Scala on wiki.scala-lang.org.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An implicit point is that tools and frameworks written in Scala inherently have Scala integration and usually a Scala API. Then other APIs may be added to support other languages beginning with Java since Scala is already integrated and in fact critically depends on Java.  If a tool or framework is not written in Scala, it is unlikely that it offers any support for Scala.  That is why in answer to your question I have pointed towards tools and frameworks written in Scala and Spark is one example.  However, Scala currently has a minor share of the market but its adoption rate is growing and the high growth rate of Spark will enhance that. The reason I use Scala is because Spark's API for Scala is richer than the Java and Python APIs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main reasons I prefer Scala generally is because it is much more expressive than Java because it allows and facilitates the use of functions as objects and values while retaining object oriented modularity, which enables development of complex and correct programs with far less code than Java which I had preferred because of widespread use, clarity and excellent documentation.&lt;/p&gt;&#xA;" OwnerUserId="5280" LastEditorUserId="5280" LastEditDate="2014-12-24T20:10:01.487" LastActivityDate="2014-12-24T20:10:01.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3690" PostTypeId="1" AcceptedAnswerId="3703" CreationDate="2014-12-11T23:56:03.023" Score="0" ViewCount="1432" Body="&lt;p&gt;I'm wonder if it's possible to export a model trained in R, to OpenCV's Machine Learning (ML) library format? The latter appears to save/read models in &lt;a href=&quot;http://docs.opencv.org/modules/ml/doc/statistical_models.html#cvstatmodel-load&quot; rel=&quot;nofollow&quot;&gt;XML/YAML&lt;/a&gt;, whereas the former might be exportable via &lt;a href=&quot;http://cran.r-project.org/web/packages/pmml/index.html&quot; rel=&quot;nofollow&quot;&gt;PMML&lt;/a&gt;. Specifically, I'm working with Random Forests, which are classifiers available both in R and OpenCV's ML library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice on how I can get the two to share models would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="6390" LastActivityDate="2014-12-14T03:42:25.213" Title="Exporting R model to OpenCV's Machine Learning Library" Tags="|machine-learning|r|open-source|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3691" PostTypeId="2" ParentId="3688" CreationDate="2014-12-12T00:13:10.680" Score="6" Body="&lt;p&gt;For starters, Naive Bayes is probably not appropriate here.  It assumes independence among the inputs (hence the &quot;Naive&quot;) and words in a sentence are very dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, assuming you really want to run with NB as an algorithm to start your experimentation, there are two options I'd consider:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Ungraceful: Lots of NB classifiers&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This would be an alternative approach.  Make a corupus of all the words observed as your vector of inputs.  Make a corpus off all the tags that are observed as your vector of outputs.  An NB classifier with multiple outputs is the equivalent of having multiple NB classifiers with one output each (so do whichever is easier to implement in whatever software framework you're using).  Treat each element as a training sample where a given input (a word) is a &lt;code&gt;1&lt;/code&gt; if that word is present and a &lt;code&gt;0&lt;/code&gt; if that word isn't.  Use the same binary scheme for the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This brute forces the application of the NB Classifier to your data, and leaves you to find meaning by still haivng to mine the huge set of classifiers you'll be left with.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;More Graceful: Process your data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the approach I'd recommend if you want to run with one multiple-class NB Classifier.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your goal here is to figure out how to map each set of tags to a single class.  I'm sure there is some sort of clustering scheme or network analysis (perhaps [&quot;celebrity&quot;] linked to [&quot;murder&quot;] could become a the segment [&quot;debauchery&quot;]) that will sensibly map your tags to one single cluster.  If you treat tags as nodes and two given tags together as links, then you'll want to look into community detection algorithms (which is where I'd start).  But, if you just want something working, then some sort of hack on the tags that converts a list of tags to only the tag that's most commonly seen in your dataset would be enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method front-loads the work of cleaning your data and would make the NB Classifier's output easier to understand.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-12T00:13:10.680" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="3692" PostTypeId="2" ParentId="2631" CreationDate="2014-12-12T01:30:26.723" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;Apply your clustering algorithm&lt;/li&gt;&#xA;&lt;li&gt;Calculate distance from all data points to its assigned cluster&lt;/li&gt;&#xA;&lt;li&gt;Label the data points furthest from a center as an outlier&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Randomly generating 100 data points from three gaussians, clustering them with k-means, and marking the 10 'furthest from a center' data points gave the following graph:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/TrSDZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see &lt;a href=&quot;http://nbviewer.ipython.org/github/TheGrimmScientist/Snippets/blob/master/OutlierDetectionWithClustering/OutlierDetectionWithClustering.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this notebook&lt;/a&gt; for the full example&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The burden of solving what &quot;distance&quot; means will already have to be solved for you to run a clustering algorithm.  It will still be up to you to pick off what distance means an outlier.  In this example, I just picked the N most distant data point, though you'll probably want to pick any number of data points over a certain number of standard deviations from a center.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-12T01:30:26.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3693" PostTypeId="1" AcceptedAnswerId="3701" CreationDate="2014-12-12T03:00:57.507" Score="8" ViewCount="3088" Body="&lt;p&gt;I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names with their submissions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Setting aside the better solutions of a url where students submit their work and a backend script inserts the anonymized ID, &lt;strong&gt;What sort of solutions could I direct students to implement on their own to anonymized their own names?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm still a noob in this area. I don't know what are the norms. I was thinking the solution could be a hashing algorithm. That sounds like a better solution than making up a fake name as two people could pick the same fake name.possible people could pick the same fake name. &lt;strong&gt;What are some of the concerns I should be aware of?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="2742" LastActivityDate="2014-12-13T21:47:49.987" Title="What are the best practices to anonymize user names in data?" Tags="|machine-learning|data-cleaning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3694" PostTypeId="2" ParentId="3693" CreationDate="2014-12-12T09:02:16.350" Score="2" Body="&lt;p&gt;A standard practice in psychology (where you want to code participants in order to link different measurements together) is to have participants choose their mother's maiden name initials and birthdate, e.g., in the format XX-YYMMDD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This if course can still run into conflicts. Then again, I don't think there is &lt;em&gt;any&lt;/em&gt; surefire conflict-free anonymization algorithm your students could do &lt;em&gt;without knowing all the other students&lt;/em&gt;. Mothers' names and birthdates could be identical, own birthdates could be identical, shoe sizes could be, favorite superhero characters... The only thing I could think of would be (US) Social Security numbers, but you &lt;a href=&quot;http://www.ssa.gov/pubs/EN-05-10064.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;really&lt;/em&gt; don't want to use them&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottom line: anonymize on the backend. Or, as &lt;a href=&quot;https://datascience.stackexchange.com/questions/3693/what-are-the-best-practices-to-anonymize-user-names-in-data#comment3919_3693&quot;&gt;@Emre suggests&lt;/a&gt;, think about whether you really need an identifier at all. Maybe the DB-generated index is enough?&lt;/p&gt;&#xA;" OwnerUserId="2853" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-12-12T09:02:16.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3696" PostTypeId="2" ParentId="2668" CreationDate="2014-12-12T22:39:58.470" Score="11" Body="&lt;h2&gt;Re: size of data&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;The short answer&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Scala works for both small and large data, but its creation and development is motivated by needing something scalable.  &lt;a href=&quot;http://www.scala-lang.org/what-is-scala.html&quot;&gt;Scala is an acronym for “Scalable Language”&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;The long answer&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Scala is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Functional_programming&quot;&gt;functional programming language&lt;/a&gt; that runs on the &lt;a href=&quot;http://en.wikipedia.org/wiki/Java_virtual_machine&quot;&gt;jvm&lt;/a&gt;.  The 'functional' part of this is a fundamental difference in the language that makes you think differently about programming.  If you like that way of thinking, it lets you quickly work with small data.  Whether you like it or not, functional languages are fundamentally easier to massively scale.  The jvm piece is also important because the jvm is basically everywhere and, thus, Scala code can run basically everywhere.  (Note there are plenty of other &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_JVM_languages&quot;&gt;languages written on the jvm&lt;/a&gt; and plenty of other &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_programming_languages_by_type#Functional_languages&quot;&gt;functional programming languages&lt;/a&gt;, and languages beyond Scala do appear in both lists.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=3jg1AheF4n0&quot;&gt;This talk&lt;/a&gt; give a good overview of the motivation behind Scala.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Re: other tools that have good Scala support:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As you mentioned, Spark (distributable batch processing better at iteratative algorithms than its counterpart) is a big one.  With Spark comes its libraries &lt;a href=&quot;https://spark.apache.org/mllib/&quot;&gt;Mllib&lt;/a&gt; for machine learning and &lt;a href=&quot;https://spark.apache.org/graphx/&quot;&gt;GraphX&lt;/a&gt; for graphs.  As mentioned by  Erik Allik and Tris Nefzger, &lt;a href=&quot;http://akka.io&quot;&gt;Akka&lt;/a&gt; and &lt;a href=&quot;https://github.com/factorie/factorie&quot;&gt;Factorie&lt;/a&gt; exist.  There is also &lt;a href=&quot;https://www.playframework.com/&quot;&gt;Play&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally,  I can't tell if there is a specific use case you're digging for (if so, make that a part of your question), or just want a survey of big data tools and happen to know Scala a bit and want to start there.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-12T22:39:58.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3697" PostTypeId="2" ParentId="19" CreationDate="2014-12-12T22:48:41.970" Score="11" Body="&lt;p&gt;Data becomes &quot;big&quot; when a single &lt;a href=&quot;http://en.wikipedia.org/wiki/Commodity_computing&quot;&gt;commodity computer&lt;/a&gt; can no longer handle the amount of data you have.  It denotes the point at which you need to start thinking about building supercomputers or using clusters to process your data.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-12T22:48:41.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3698" PostTypeId="2" ParentId="2631" CreationDate="2014-12-13T05:28:04.013" Score="4" Body="&lt;p&gt;If your Data points are dense and noise points are away from the dense region, you can try &lt;a href=&quot;http://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBSCAN algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/gdFlG.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Tweak its parameters until u get a best fit.&lt;/p&gt;&#xA;" OwnerUserId="6412" LastEditorUserId="98307" LastEditDate="2020-08-17T18:55:17.380" LastActivityDate="2020-08-17T18:55:17.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3699" PostTypeId="1" CreationDate="2014-12-13T17:42:54.927" Score="10" ViewCount="8777" Body="&lt;p&gt;For an imbalanced data set, is it better to choose an L1 or L2 regularization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a cost function more suitable for imbalanced datasets to improve the model score (&lt;code&gt;log_loss&lt;/code&gt; in particular)? &lt;/p&gt;&#xA;" OwnerUserId="6418" LastEditorUserId="29169" LastEditDate="2019-06-07T17:12:43.867" LastActivityDate="2020-08-13T11:46:33.743" Title="What cost function and penalty are suitable for imbalanced datasets?" Tags="|machine-learning|classification|deep-learning|scikit-learn|logistic-regression|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="3700" PostTypeId="1" AcceptedAnswerId="3725" CreationDate="2014-12-13T19:27:56.213" Score="0" ViewCount="135" Body="&lt;p&gt;In &lt;a href=&quot;http://web.cse.ohio-state.edu/~mbelkin/papers/PLM_UCTHESIS_03.pdf&quot; rel=&quot;nofollow&quot; title=&quot;Belkin&amp;#39;s thesis about Laplacian Eigenmaps&quot;&gt;his thesis&lt;/a&gt; (section 2.3.3) Belkin uses the heat equation to derive an approximation for $\mathcal{L}f$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{L}f(x_i)\approx \frac{1}{t}\Big(f(x_i)-\alpha \sum_{x_j, ||x_i-x_j||&amp;lt;\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j)\Big)$$&#xA;where $$\alpha=\Big(\sum_{x_j, ||x_i-x_j||&amp;lt;\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}\Big)^{-1}$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not sure how these considerations lead to this choice of weights for the weight matrix (which will be used to construct the Laplacian):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_{ij} =&#xA;\begin{cases} &#xA;e^{-\frac{||x_i-x_j||^2}{4t}} &amp;amp; if\ ||x_i-x_j||&amp;lt;\epsilon \\ &#xA;0 &amp;amp; otherwise &#xA;\end{cases}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very vague idea of mine was that the factors $\alpha$ and $\frac{1}{t}$ don't change for a given $x_i$ so if one choses the weights like above the resulting discrete Laplacian would (let aside those two constants) converge to the continuous version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas or tips what I'd have to read up to in order to get a better understanding?&lt;/p&gt;&#xA;" OwnerUserId="6415" LastActivityDate="2014-12-19T01:20:39.907" Title="Choice of weights for the Laplacian Eigenmaps algorithm" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3701" PostTypeId="2" ParentId="3693" CreationDate="2014-12-13T20:37:07.843" Score="5" Body="&lt;p&gt;I suspected you were using the names as identifiers. You shouldn't; they're not unique and they raise this privacy issue. Use instead their student numbers, which you can verify from their IDs, stored in hashed form. Use the student's last name as a salt, for good measure (form the string to be hashed by concatenating the ID number and the last name).&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2014-12-13T21:47:49.987" LastActivityDate="2014-12-13T21:47:49.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3702" PostTypeId="1" AcceptedAnswerId="3713" CreationDate="2014-12-13T23:08:53.930" Score="4" ViewCount="1444" Body="&lt;p&gt;Is the k-nearest neighbour algorithm a discriminative or a generative classifier? My first thought on this was that it was generative, because it actually uses Bayes' theorem to compute the posterior. Searching further, it seems like it is a discriminative model. But I couldn't find the explanation.&lt;/p&gt;&#xA;&lt;p&gt;So is KNN discriminative first of all? And if it is, is that because it doesn't model the priors or the likelihood?&lt;/p&gt;&#xA;" OwnerUserId="6419" LastEditorUserId="132929" LastEditDate="2022-02-28T19:28:12.320" LastActivityDate="2022-02-28T19:28:12.320" Title="K nearest neighbour" Tags="|classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="3703" PostTypeId="2" ParentId="3690" CreationDate="2014-12-14T03:42:25.213" Score="1" Body="&lt;p&gt;Instead of &lt;em&gt;exporting&lt;/em&gt; your models, consider creating an R-based &lt;strong&gt;interoperable environment&lt;/strong&gt; for your &lt;em&gt;modeling&lt;/em&gt; needs. Such environment would consists of &lt;em&gt;R environment&lt;/em&gt; proper as well as &lt;em&gt;integration layers&lt;/em&gt; for your third-party libraries. In particular, for the &lt;em&gt;OpenCV&lt;/em&gt; project, consider either using &lt;code&gt;r-opencv&lt;/code&gt; open source project (&lt;a href=&quot;https://code.google.com/p/r-opencv&quot; rel=&quot;nofollow&quot;&gt;https://code.google.com/p/r-opencv&lt;/a&gt;), or integration via &lt;em&gt;OpenCV&lt;/em&gt; C++ APIs and &lt;em&gt;R&lt;/em&gt; &lt;code&gt;Rcpp&lt;/code&gt; package (&lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.html&quot; rel=&quot;nofollow&quot;&gt;http://dirk.eddelbuettel.com/code/rcpp.html&lt;/a&gt;). Finally, if you want to add &lt;em&gt;PMML&lt;/em&gt; support to the mix and create a &lt;strong&gt;deployable-to-cloud solution&lt;/strong&gt;, take a look at the following excellent blog post with relevant examples: &lt;a href=&quot;http://things-about-r.tumblr.com/post/37861967022/predictive-modeling-using-r-and-the&quot; rel=&quot;nofollow&quot;&gt;http://things-about-r.tumblr.com/post/37861967022/predictive-modeling-using-r-and-the&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-12-14T03:42:25.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3705" PostTypeId="1" CreationDate="2014-12-16T01:19:12.477" Score="3" ViewCount="2465" Body="&lt;p&gt;I'm trying to get to grips with sci-kit learn for some simple machine learning projects but I'm coming unstuck with Pipelines and wonder what I've done wrong...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to work through a &lt;a href=&quot;http://www.kaggle.com/c/data-science-london-scikit-learn/data&quot; rel=&quot;nofollow&quot;&gt;tutorial on Kaggle&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;train = pd.read_csv(local path to training data) train_labels =&#xA;pd.read_csv(local path to labels)&#xA;&#xA;&#xA;from sklearn.decomposition import PCA&#xA;from sklearn.svm import LinearSVC&#xA;from sklearn.grid_search import GridSearchCV&#xA;&#xA;pca = PCA()&#xA;clf = LinearSVC()&#xA;&#xA;n_components = arange(1, 39)&#xA;loss = ['l1','l2']&#xA;penalty = ['l1','l2']&#xA;C = arange(0, 1, .1)&#xA;whiten = [True, False]&#xA;&#xA;from sklearn.pipeline import Pipeline&#xA;&#xA;#set up pipeline&#xA;pipe = Pipeline(steps=[('pca', pca), ('clf', clf)])&#xA;&#xA;#set up GridsearchCV&#xA;estimator = GridSearchCV(pipe, dict(pca__n_components = n_components, pca__whiten = whiten, clf__loss = loss, clf__penalty = penalty, clf__C = C)) &#xA;&#xA;&amp;gt; estimator&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Returns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;GridSearchCV(cv=None,&#xA;       estimator=Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('clf', LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,&#xA;     intercept_scaling=1, loss='l2', multi_class='ovr', penalty='l2',&#xA;     random_state=None, tol=0.0001, verbose=0))]),&#xA;       fit_params={}, iid=True, loss_func=None, n_jobs=1,&#xA;       param_grid={'clf__penalty': ['l1', 'l2'], 'clf__loss': ['l1', 'l2'], 'clf__C': array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9]), 'pca__n_components': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,&#xA;       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,&#xA;       35, 36, 37, 38]), 'pca__whiten': [True, False]},&#xA;       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,&#xA;       verbose=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I try to train data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimator.fit(train, train_labels)&#xA;The error is:&#xA;&#xA;    428         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):&#xA;    429             for label, (_, test_split) in zip(unique_labels, per_label_splits):&#xA;--&amp;gt; 430                 label_test_folds = test_folds[y == label]&#xA;    431                 # the test split can be too big because we used&#xA;    432                 # KFold(max(c, self.n_folds), self.n_folds) instead of&#xA;&#xA;IndexError: too many indices for array&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="974" LastEditorUserId="9433" LastEditDate="2015-05-21T19:28:55.737" LastActivityDate="2015-06-20T19:58:59.403" Title="Sci-kit Pipeline and GridsearchCV returns indexError: too many indices for array" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3706" PostTypeId="2" ParentId="3705" CreationDate="2014-12-16T02:44:28.950" Score="1" Body="&lt;p&gt;It turns out that the Pandas dataframe is the wrong shape.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimator.fit(train.values, train_labels[0].values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;works, although I also had to drop the penalty term.&lt;/p&gt;&#xA;" OwnerUserId="974" LastActivityDate="2014-12-16T02:44:28.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3707" PostTypeId="2" ParentId="421" CreationDate="2014-12-16T06:12:35.423" Score="2" Body="&lt;p&gt;There's a lot of online tutorial. Especially in youtube, but if you will want accurate website you can see from here &lt;a href=&quot;http://ttic.uchicago.edu/~shai/icml08tutorial/&quot; rel=&quot;nofollow&quot;&gt;http://ttic.uchicago.edu/~shai/icml08tutorial/&lt;/a&gt; or &lt;a href=&quot;http://cs229.stanford.edu/materials.html&quot; rel=&quot;nofollow&quot;&gt;http://cs229.stanford.edu/materials.html&lt;/a&gt;. you can visit them now.&lt;/p&gt;&#xA;" OwnerUserId="6444" LastActivityDate="2014-12-16T06:12:35.423" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3708" PostTypeId="2" ParentId="421" CreationDate="2014-12-16T08:51:30.987" Score="1" Body="&lt;p&gt;You can see if you can mix Spark streaming (&lt;a href=&quot;https://spark.apache.org/docs/1.1.0/streaming-programming-guide.html&quot; rel=&quot;nofollow&quot;&gt;https://spark.apache.org/docs/1.1.0/streaming-programming-guide.html&lt;/a&gt;) and Spark ML Library (&lt;a href=&quot;https://spark.apache.org/docs/1.1.0/mllib-guide.html&quot; rel=&quot;nofollow&quot;&gt;https://spark.apache.org/docs/1.1.0/mllib-guide.html&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark Streaming permits to process live data streams and Spark ML Library is a Machine Learning Library for Spark. So maybe you can do something good! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is a very interesting subject, I am working on it. It can be good to create a Google Community on it (&lt;a href=&quot;https://plus.google.com/communities&quot; rel=&quot;nofollow&quot;&gt;https://plus.google.com/communities&lt;/a&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="5165" LastActivityDate="2014-12-16T08:51:30.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3711" PostTypeId="1" AcceptedAnswerId="3717" CreationDate="2014-12-16T13:07:55.063" Score="14" ViewCount="24522" Body="&lt;p&gt;Naive Bayes apparently handles missing data differently, depending on whether they exist in training or testing/classification instances.&lt;/p&gt;&#xA;&lt;p&gt;When classifying instances, the attribute with the missing value is simply not included in the probability calculation (&lt;a href=&quot;http://www.inf.ed.ac.uk/teaching/courses/iaml/slides/naive-2x2.pdf&quot; rel=&quot;noreferrer&quot;&gt;reference&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;In training, &lt;em&gt;the instance&lt;/em&gt; [with the missing data] &lt;em&gt;is not included in frequency count for attribute value-class combination&lt;/em&gt;. (&lt;a href=&quot;http://www.csee.wvu.edu/%7Etimm/cs591o/old/BasicMethods.html&quot; rel=&quot;noreferrer&quot;&gt;reference&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;Does that mean that particular training record simply isn't included in the training phase? Or does it mean something else?&lt;/p&gt;&#xA;" OwnerUserId="6451" LastEditorUserId="98307" LastEditDate="2020-08-17T15:55:05.267" LastActivityDate="2022-12-03T12:19:36.527" Title="How does the naive Bayes classifier handle missing data in training?" Tags="|machine-learning|data-mining|classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="3713" PostTypeId="2" ParentId="3702" CreationDate="2014-12-17T02:58:06.057" Score="4" Body="&lt;p&gt;See a similar answer &lt;a href=&quot;https://stats.stackexchange.com/questions/105979/is-knn-a-discriminative-learning-algorithm&quot;&gt;here&lt;/a&gt;. To clarify, k nearest neighbor is a discriminative classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between a generative and a discriminative classifier is that the former models the joint probability where as the latter models the conditional probability (the posterior) starting from the prior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of nearest neighbors, the conditional probability of a class given a data point is modeled. To do this, one starts with the prior probability on the classes.  &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2014-12-17T09:34:11.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3715" PostTypeId="2" ParentId="1190" CreationDate="2014-12-17T09:07:11.320" Score="2" Body="&lt;p&gt;This seems to be a standard regression problem in which there are two goals:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Obtain a predictive model that can be used for prediction.&lt;/li&gt;&#xA;&lt;li&gt;Which variables seem to be the most important ones to be used. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For both the above problems use an ensemble model. Consider both a random forest and a gradient boosted machine. Both these models will use the independent variables and predict the Hospital time. Additionally, through variable importances, you can obtain which variables are the most important ones and have the most impact in predicting the output. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-12-17T09:07:11.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3716" PostTypeId="2" ParentId="1243" CreationDate="2014-12-17T09:32:53.703" Score="1" Body="&lt;p&gt;In the way that you've defined or set up the problem, i.e.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sales = alpha*quality + beta*position + epsilon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can easily quantify &lt;code&gt;beta&lt;/code&gt; given that your model is correct. You just need to run it through linear regression and it will give you the coefficient for &lt;code&gt;beta&lt;/code&gt;*.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to model click through rates, you would have to train a classifier. So you would have to fit a logistic model that models: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clicks ~ alpha*quality + beta*position + epsilon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;*I believe you would have to restrict the training set to contain results where all impressions were obtained on the first page otherwise your model will not hold (I would guess that &lt;code&gt;beta&lt;/code&gt; is going to be strongly dependent on the page). &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-12-17T09:32:53.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3717" PostTypeId="2" ParentId="3711" CreationDate="2014-12-17T18:15:42.133" Score="18" Body="&lt;p&gt;In general, you have a choice when handling missing values hen training a naive Bayes classifier. You can choose to either&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Omit records with any missing values,&lt;/li&gt;&#xA;&lt;li&gt;Omit only the missing attributes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I'll use the example linked to above to demonstrate these two approaches.  Suppose we add one more training record to that example.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Outlook  Temperature  Humidity   Windy   Play&#xA;-------  -----------  --------   -----   ----&#xA;rainy    cool        normal    TRUE    no&#xA;rainy    mild        high      TRUE    no&#xA;sunny    hot         high      FALSE   no&#xA;sunny    hot         high      TRUE    no&#xA;sunny    mild        high      FALSE   no&#xA;overcast cool        normal    TRUE    yes&#xA;overcast hot         high      FALSE   yes&#xA;overcast hot         normal    FALSE   yes&#xA;overcast mild        high      TRUE    yes&#xA;rainy    cool        normal    FALSE   yes&#xA;rainy    mild        high      FALSE   yes&#xA;rainy    mild        normal    FALSE   yes&#xA;sunny    cool        normal    FALSE   yes&#xA;sunny    mild        normal    TRUE    yes&#xA;NA       hot         normal    FALSE   yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If we decide to omit the last record due to the missing &lt;code&gt;outlook&lt;/code&gt; value, we would have the exact same trained model as discussed in the link.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We could also choose to use all of the information available from this record.  We could choose to simply omit the attribute &lt;code&gt;outlook&lt;/code&gt; from this record. This would yield the following updated table.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;pre&gt;&#xA;           Outlook            Temperature           Humidity   &#xA;====================   =================   =================  &#xA;          Yes    No            Yes   No            Yes    No &#xA;Sunny       2     3     Hot     3     2    High      3     4&#xA;Overcast    4     0     Mild    4     2    Normal    7     1 &#xA;Rainy       3     2     Cool    3     1&#xA;          -----------         ---------            ---------- &#xA;Sunny     2/9   3/5     Hot   3/10   2/5    High    3/10   4/5 &#xA;Overcast  4/9   0/5     Mild  4/10   2/5    Normal  7/10   1/5&#xA;Rainy     3/9   2/5     Cool  3/10   1/5&#xA;&#xA;&#xA;            Windy        Play&#xA;=================    ========&#xA;      Yes     No     Yes   No&#xA;False 7      2       10     5&#xA;True  3      3&#xA;      ----------   ----------&#xA;False  7/10    2/5   10/15  5/15&#xA;True   3/10    3/5&#xA;&lt;/pre&gt;&#xA;&lt;p&gt;Notice there are 15 observations for each attribute &lt;strong&gt;except&lt;/strong&gt; &lt;code&gt;Outlook&lt;/code&gt;, which has only 14. This is since that value was unavailable for the last record. All further development would continue as discussed in the linked article.&lt;/p&gt;&#xA;&lt;p&gt;For example in the R package &lt;code&gt;e1071&lt;/code&gt; naiveBayes implementation has the option &lt;code&gt;na.action&lt;/code&gt; which can be set to na.omit or na.pass.&lt;/p&gt;&#xA;" OwnerUserId="4724" LastEditorUserId="4724" LastEditDate="2022-12-03T12:19:36.527" LastActivityDate="2022-12-03T12:19:36.527" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="3718" PostTypeId="1" AcceptedAnswerId="3723" CreationDate="2014-12-17T21:17:13.340" Score="9" ViewCount="1029" Body="&lt;p&gt;I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to use UI (easily scalable, pannable, easily zoomable/fit to screen), but cannot handle the large sets of data I'm collecting. Does anyone know of any alternatives?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have MATLAB, but don't have enough licenses to simultaneously run this and do development at the same time. I know that LabVIEW would be a great option, but it is currently cost-prohibitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="6469" LastEditorUserId="2452" LastEditDate="2014-12-18T02:39:23.860" LastActivityDate="2016-07-15T23:19:12.977" Title="Interactive Graphing while logging data" Tags="|dataset|visualization|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3719" PostTypeId="1" CreationDate="2014-12-18T04:36:06.107" Score="13" ViewCount="6818" Body="&lt;p&gt;I am working on a data-science project related on social relationship mining and need to store data in some graph databases. Initially I chose Neo4j as the database. But it seams Neo4j doesn't scale well. The alternative I found out are Titan  and oriebtDB. I have gone through &lt;a href=&quot;http://db-engines.com/en/system/Neo4j%3BOrientDB%3BTitan&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; comparison on these three Databases, But I would like to get more details on  these databases. So Could some one help me in choosing the best one. Mainly I would like to compare performance, scaling, on line documentation/tutorials available, Python library support, query language complexity and graph algorithm support of these databases. Also is there any other good database options ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-06-02T17:00:53.513" LastActivityDate="2015-06-26T07:32:27.147" Title="Neo4j vs OrientDB vs Titan" Tags="|data-mining|graphs|databases|social-network-analysis|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="3720" PostTypeId="2" ParentId="3718" CreationDate="2014-12-18T04:41:33.493" Score="1" Body="&lt;p&gt;Visualizing large datasets is a long standing problem. One of the issues is to understand how we can show over a million points on a screen that has only about ~ million pixels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, here are a few tools that can handle big data: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tableau: you could use their free desktop tool. &lt;/li&gt;&#xA;&lt;li&gt;Tabplot: built on top of ggplot2 in R to handle larger datasets. &lt;/li&gt;&#xA;&lt;li&gt;See &lt;a href=&quot;http://www.techrepublic.com/blog/five-apps/five-web-based-apps-to-help-you-visualize-big-data/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; review for 5 other products that can help you do your job. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-12-18T04:41:33.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3721" PostTypeId="1" AcceptedAnswerId="3722" CreationDate="2014-12-18T04:48:49.820" Score="3" ViewCount="1846" Body="&lt;p&gt;With increasingly sophisticated methods that work on large scale datasets, financial applications are obvious. I am aware of machine learning being employed on financial services to detect fraud and flag fraudulent activities but I have a lesser understanding of how it helps to predict the price of the stock the next day and how many stocks of a particular company to buy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do the hedge funds still employ portfolio optimization techniques that are right out of the mathematical finance literature or have they started to use machine learning to hedge their bets? More importantly, what are the features that are used by these hedge funds and what is a representative problem set up? &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2014-12-18T08:25:09.517" Title="Machine Learning for hedging/ portfolio optimization?" Tags="|machine-learning|feature-selection|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3722" PostTypeId="2" ParentId="3721" CreationDate="2014-12-18T08:10:00.630" Score="4" Body="&lt;p&gt;That is a rather broad question, and there is tons of literature about quantitative analysis and stock market prediction using machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most classical example of predicting the stock market is employing neural networks; you can use whatever feature you think might be relevant for your prediction, for example the unemployment rate, the oil price, the gold price, the interest rates, and the timeseries itself, i. e. the volatility, the change in the last 2,3,7,..., days etc. - a more classical approach is the input-output-analysis in econometrics, or the autoregression analysis, but all of it can be modeled using neural networks or any other function approximator / regression in a very natural way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, as said, there are tons of other possibilities to model the market, to name a few: Ant Colony Optimization (ACO), Classical regression analysis, genetic algorithms, decision trees, reinforcement learning etc. you name it, almost EVERYTHING has probably been applied to the stock market prediction problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different fond manager types on the markets. There are still the Quants which are doing a quantitative analysis using classical financial maths and maths borrowed from the physics to describe the market movements. There are still the most conservative ones which do a long-term, fundamental analysis of the corporation, that is, looking in how the corporation earns money and where it spends money. Or the tactical analysts who just look for immediate signals to buy / sell a stock in the short term. And those quantitative guys who employ machine learning amongst other methods.&lt;/p&gt;&#xA;" OwnerUserId="3132" LastEditorUserId="3132" LastEditDate="2014-12-18T08:25:09.517" LastActivityDate="2014-12-18T08:25:09.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3723" PostTypeId="2" ParentId="3718" CreationDate="2014-12-18T13:30:54.210" Score="7" Body="&lt;p&gt;For this answer, I have assumed that you prefer &lt;strong&gt;open source solutions&lt;/strong&gt; to &lt;em&gt;big data visualization&lt;/em&gt;. This assumption is based on budgetary details from your question. However, there is one &lt;em&gt;exclusion&lt;/em&gt; to this - below I will add a reference to one commercial product, which I believe might be beneficial in your case (provided that you could afford that). I also assume that &lt;em&gt;browser-based solutions are acceptable&lt;/em&gt; (I would even prefer them, unless you have specific contradictory requirements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naturally, the first candidate as a solution to your problem I would consider &lt;strong&gt;D3.js JavaScript library&lt;/strong&gt;: &lt;a href=&quot;http://d3js.org&quot;&gt;http://d3js.org&lt;/a&gt;. However, despite &lt;em&gt;flexibility&lt;/em&gt; and other &lt;em&gt;benefits&lt;/em&gt;, I think that this solution is &lt;em&gt;too low-level&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I would recommend you to take a look at the following &lt;strong&gt;open source projects for big data visualization&lt;/strong&gt;, which are &lt;em&gt;powerful&lt;/em&gt; and &lt;em&gt;flexible&lt;/em&gt; enough, but operate at a &lt;em&gt;higher level of abstraction&lt;/em&gt; (some of them are based on D3.js foundation and sometimes are referred to as D3.js &lt;a href=&quot;http://schoolofdata.org/2013/08/12/climbing-the-d3-js-visualisation-stack&quot;&gt;visualization stack&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bokeh&lt;/strong&gt; - Python-based interactive visualization library, which supports big data and streaming data: &lt;a href=&quot;http://bokeh.pydata.org&quot;&gt;http://bokeh.pydata.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Flot&lt;/strong&gt; - JavaScript-based interactive visualization library, focused on jQuery: &lt;a href=&quot;http://www.flotcharts.org&quot;&gt;http://www.flotcharts.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;NodeBox&lt;/strong&gt; - unique rapid data visualization system (not browser-based, but multi-language and multi-platform), based on generative design and visual functional programming: &lt;a href=&quot;https://www.nodebox.net&quot;&gt;https://www.nodebox.net&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Processing&lt;/strong&gt; - complete software development system with its own programming language, libraries, plug-ins, etc., oriented to visual content: &lt;a href=&quot;https://www.processing.org&quot;&gt;https://www.processing.org&lt;/a&gt; (allows executing Processing programs in a browser via &lt;a href=&quot;http://processingjs.org&quot;&gt;http://processingjs.org&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Crossfilter&lt;/strong&gt; - JavaScript-based interactive visualization library for big data by Square (very fast visualization of large multivariate data sets): &lt;a href=&quot;http://square.github.io/crossfilter&quot;&gt;http://square.github.io/crossfilter&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;bigvis&lt;/strong&gt; - an R package for big data exploratory analysis (not a visualization library per se, but could be useful to process large data sets /aggregating, smoothing/ prior to visualization, using various R graphics options): &lt;a href=&quot;https://github.com/hadley/bigvis&quot;&gt;https://github.com/hadley/bigvis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;prefuse&lt;/strong&gt; - Java-based interactive visualization library: &lt;a href=&quot;http://prefuse.org&quot;&gt;http://prefuse.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Lumify&lt;/strong&gt; - big data integration, analysis and visualization platform (interesting feature: supports Semantic Web): &lt;a href=&quot;http://lumify.io&quot;&gt;http://lumify.io&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Separately, I'd like to mention two open source &lt;em&gt;big data analysis and visualization projects&lt;/em&gt;, focused on &lt;strong&gt;graph/network data&lt;/strong&gt; (with some support for &lt;em&gt;streaming data&lt;/em&gt; of that type): &lt;a href=&quot;http://www.cytoscape.org&quot;&gt;Cytoscape&lt;/a&gt; and &lt;a href=&quot;https://gephi.github.io&quot;&gt;Gephi&lt;/a&gt;. If you are interested in some other, &lt;em&gt;more specific&lt;/em&gt; (&lt;em&gt;maps&lt;/em&gt; support, etc.) or &lt;em&gt;commercial&lt;/em&gt; (basic free tiers), projects and products, please see this &lt;strong&gt;awesome compilation&lt;/strong&gt;, which I thoroughly &lt;em&gt;curated&lt;/em&gt; to come up with the main list above and &lt;em&gt;analyzed&lt;/em&gt;: &lt;a href=&quot;http://blog.profitbricks.com/39-data-visualization-tools-for-big-data&quot;&gt;http://blog.profitbricks.com/39-data-visualization-tools-for-big-data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, as I promised in the beginning, &lt;strong&gt;Zoomdata&lt;/strong&gt; - a commercial product, which I thought you might want to take a look at: &lt;a href=&quot;http://www.zoomdata.com&quot;&gt;http://www.zoomdata.com&lt;/a&gt;. The reason I made an exclusion for it from my open source software compilation is due to its &lt;strong&gt;built-in support for big data platforms&lt;/strong&gt;. In particular, Zoomdata provides &lt;em&gt;data connectors&lt;/em&gt; for Cloudera Impala, Amazon Redshift, MongoDB, Spark and Hadoop, plus search engines, major database engines and streaming data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt; I have no affiliation with Zoomdata whatsoever - I was just impressed by their &lt;em&gt;range of connectivity options&lt;/em&gt; (which might &lt;strong&gt;cost&lt;/strong&gt; you dearly, but that's another &lt;em&gt;aspect&lt;/em&gt; of this topic's analysis).&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-12-18T13:30:54.210" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="3724" PostTypeId="2" ParentId="2668" CreationDate="2014-12-18T18:07:18.170" Score="5" Body="&lt;p&gt;&lt;a href=&quot;http://www.scalanlp.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ScalaNLP&lt;/a&gt; is a suite of machine learning and numerical computing libraries with support for common natural language processing tasks. &lt;a href=&quot;http://www.scalanlp.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/top-15-scala-libraries-for-data-science-in-2018-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a newly updated list of scala libraries for data science.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="29169" LastEditDate="2021-03-11T20:14:30.817" LastActivityDate="2021-03-11T20:14:30.817" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3725" PostTypeId="2" ParentId="3700" CreationDate="2014-12-19T01:20:39.907" Score="1" Body="&lt;p&gt;Recall the definition he makes for the graph Laplacian earlier, $L = D -W $. Now consider the map in the RHS parentheses which I'll call $L^*$,&#xA;$$ L^*f(x_i) := f(x_i) - \alpha \sum_{x_j, ||x_i-x_j||&amp;lt;\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j).$$&#xA;The suggested weight matrix definition is natural because it lets us write&#xA;$$ L^* := I - D^{-1}W. $$ Here's &lt;a href=&quot;http://www.cs.yale.edu/homes/singer/publications/laplacian_ACHA.pdf&quot; rel=&quot;nofollow&quot;&gt;a reference&lt;/a&gt; to a related paper with some easy to read exposition. Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="4724" LastActivityDate="2014-12-19T01:20:39.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3726" PostTypeId="2" ParentId="2673" CreationDate="2014-12-19T01:42:36.780" Score="1" Body="&lt;p&gt;This is the fundamental challenge to all data modeling.  We don't just want to memorize the the link between a given input and a given output (otherwise you wouldn't be modeling data, you'd be memorizing 1:1 connections with a dict / hash / relational database table / etc).  We want to capture the underlying pattern in the data from only looking at the training data.&lt;/p&gt;&#xA;&lt;p&gt;Let's expand a little on your gravity example.  You have your 10 training samples showing the start and ending position of an object dropped.  For consistency, let's say the object was dropped the moment the object's location was initially recorded and the ending location was recorded at some precise time interval later (but before the object hit the ground).  Let's also say the model (neural network in this case) managed to precisely learn the expected change in location since it just comes down to subtraction in one axis.  You can show it another 10, 100, 1000 examples that all leverage the connection found and your model will keep performing well.&lt;/p&gt;&#xA;&lt;p&gt;Why not keep going to 10k, 100k, or even more samples?  Theoretically, if you managed to isolate the connection and run the experiment the same way each time, your model will always work.  But realistically, something is going to eventually change in the system.  You hire a new lab assistant who tends to press the 'record location' button well after having dropped the object (giving the object more initial velocity, which you won't notice having only recorded location).  Maybe you lost your initial ball and had to use something else which is lighter and catches the wind more (so it goes slower).  .... the longer you run the experiment, the more small changes will creep into your system.  Eventually these changes will alter the connection enough to make your initial model wrong.&lt;/p&gt;&#xA;&lt;p&gt;When modeling data, we want to capture the underlying patterns and acknowledge that the model only matters as long as those underlying patterns stay relevant.  It's not really about the number of samples.  It's about the connections / the model itself.  The number of samples just happens to be one of the better proxies we have - the more samples you use, the more confident you have some underlying pattern.  'Statistical validity' is one stab at solving this, though it's validity is still up for question in the era of big data.  There is &lt;a href=&quot;https://ieeexplore.ieee.org/document/5726612&quot; rel=&quot;nofollow noreferrer&quot;&gt;plenty of work&lt;/a&gt; done trying to solve for how to gain confidence in good generalization in neural networks specifically, but it's still very much an open question.&lt;/p&gt;&#xA;&lt;p&gt;For a different example, if you're looking at user behavior, you'll see differences between day and night; weekdays and weekends; summer and winter; year of a person's life; culture a person grew up in... even if you prove you found a pattern in your initial sample, the system will eventually change and it's up to luck whether the connection(s) you found are a part of the system that changed or a part of the system that didn't.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastEditorUserId="25180" LastEditDate="2022-04-22T13:15:54.290" LastActivityDate="2022-04-22T13:15:54.290" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3728" PostTypeId="1" AcceptedAnswerId="3729" CreationDate="2014-12-19T11:42:04.547" Score="1" ViewCount="258" Body="&lt;p&gt;I was wondering if anyone knew which piece of software is being used in this video? It is an image recognition system that makes the training process very simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn#t-775098&quot; rel=&quot;nofollow&quot;&gt;http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn#t-775098&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example is with car images, though the video should start at the right spot.&lt;/p&gt;&#xA;" OwnerUserId="5175" LastActivityDate="2014-12-19T16:02:06.330" Title="What software is being used in this image recognition system?" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3729" PostTypeId="2" ParentId="3728" CreationDate="2014-12-19T16:02:06.330" Score="1" Body="&lt;p&gt;I'm pretty sure that the software you're referring to is a some kind of &lt;strong&gt;internal&lt;/strong&gt; &lt;em&gt;research project&lt;/em&gt; software, developed by Enlitic (&lt;a href=&quot;http://www.enlitic.com&quot; rel=&quot;nofollow&quot;&gt;http://www.enlitic.com&lt;/a&gt;), where Jeremy Howard works as a founder and CEO. By &quot;internal research project software&quot; I mean either a &lt;em&gt;proof-of-concept&lt;/em&gt; software, or a &lt;em&gt;prototype&lt;/em&gt; software.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2014-12-19T16:02:06.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3730" PostTypeId="1" CreationDate="2014-12-19T16:02:48.693" Score="9" ViewCount="2782" Body="&lt;p&gt;I am training random forest models in R using &lt;code&gt;randomForest()&lt;/code&gt; with 1000 trees and data frames with about 20 predictors and 600K rows. On my laptop everything works fine, but when I move to amazon ec2, to run the same thing, I get the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error: cannot allocate vector of size 5.4 Gb&#xA;Execution halted&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using the &lt;code&gt;c3.4xlarge&lt;/code&gt; instance type so it's pretty beefy. Does anyone know a workaround for this to get it to run on this instance? I would love to know the memory nuances that causes this problem only on the ec2 instance and not on my laptop (OS X 10.9.5 Processor  2.7 GHz Intel Core i7; Memory  16 GB 1600 MHz DDR3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerDisplayName="SOUser" LastEditorUserId="29169" LastEditDate="2019-06-07T17:27:34.900" LastActivityDate="2019-06-07T17:27:34.900" Title="R random forest on Amazon ec2 Error: cannot allocate vector of size 5.4 Gb" Tags="|r|random-forest|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3731" PostTypeId="2" ParentId="3730" CreationDate="2014-12-19T16:33:44.423" Score="5" Body="&lt;p&gt;Here's some advice (use at your own risk!):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;make sure that your R environment on EC2 is identical to the one on your laptop&lt;/li&gt;&#xA;&lt;li&gt;make sure you're using 64-bit images for your virtual machine instance&lt;/li&gt;&#xA;&lt;li&gt;try to create/enable swap space: &lt;a href=&quot;https://stackoverflow.com/a/22247782/2872891&quot;&gt;https://stackoverflow.com/a/22247782/2872891&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;see this discussion: &lt;a href=&quot;https://stackoverflow.com/q/5171593/2872891&quot;&gt;https://stackoverflow.com/q/5171593/2872891&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;see this discussion: &lt;a href=&quot;https://stackoverflow.com/q/12322959/2872891&quot;&gt;https://stackoverflow.com/q/12322959/2872891&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;see this discussion: &lt;a href=&quot;https://stackoverflow.com/q/1358003/2872891&quot;&gt;https://stackoverflow.com/q/1358003/2872891&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the above-mentioned simpler measures don't help OR you want to achieve more &lt;strong&gt;scalability&lt;/strong&gt; and/or &lt;strong&gt;performance&lt;/strong&gt;, including an ability to &lt;em&gt;parallelize&lt;/em&gt; the process on a single machine or across multiple machines, consider using &lt;code&gt;bigrf&lt;/code&gt; R package: &lt;a href=&quot;http://cran.r-project.org/web/packages/bigrf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/web/packages/bigrf&lt;/a&gt;. Also see this discussion: &lt;a href=&quot;https://stackoverflow.com/q/1358003/2872891&quot;&gt;https://stackoverflow.com/q/1358003/2872891&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" OwnerDisplayName="Aleksandr Blekh" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2014-12-19T16:33:44.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3732" PostTypeId="2" ParentId="3730" CreationDate="2014-12-19T23:24:55.017" Score="2" Body="&lt;p&gt;Additional to other ideas: reduce your data until you figure out what you &lt;strong&gt;can&lt;/strong&gt; run on the Amazon instance. If it can't do 100k rows then something is very wrong, if it fails at 590k rows then its marginal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The c3.4xlarge instance has 30Gb of RAM, so yes it should be enough.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2014-12-19T23:24:55.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3733" PostTypeId="1" AcceptedAnswerId="3736" CreationDate="2014-12-20T03:37:21.820" Score="5" ViewCount="618" Body="&lt;p&gt;I have a non-function (not in closed form) that takes in a few parameters (about 20) and returns a real value. A few of these parameters are discrete while others are continuous. Some of these parameters can only be chosen from a finite space of values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I don't have the function in closed form, I cannot use any gradient based methods. However, the discrete nature and the boxed constraints on a few of those parameters restrict even the number of derivative free optimization techniques at my disposal. I am wondering what are the options in terms of optimization methods that I can use. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2017-03-26T19:26:39.027" Title="Which Optimization method to use?" Tags="|optimization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3734" PostTypeId="1" CreationDate="2014-12-20T07:03:57.837" Score="2" ViewCount="165" Body="&lt;p&gt;My apologies in advance as I am new to this. I have searched the internet and tried various processes and nothing seems to work or address this situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset of 30,000 transactions and 500,000 items. Average item size for a transaction is 50. The dataset is sparse, so the support number must be set quite low. Furthermore, the rules become more valuable the larger the number of items in the rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried running this in arules and the tests fail after exceeding 64 gb of RAM (the limit of the machine). I have tried reducing items and transactions to smaller subsets, but still hit this memory limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, I am looking for ways to cluster large groups of similar accounts by selection of items and generate confidence and lift of various next items selected from those clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: are there alternative, more efficient ways to do this, or other approaches to consider?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="6506" OwnerDisplayName="Chris" LastActivityDate="2015-04-19T14:49:57.930" Title="Approaches to high dimension pattern matching problem" Tags="|r|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3735" PostTypeId="2" ParentId="3734" CreationDate="2014-12-20T08:06:13.077" Score="1" Body="&lt;p&gt;According to the following discussion on &lt;em&gt;StackOverflow&lt;/em&gt;, a situation like that you've described can occur, when one of the variables in a data set is of &lt;strong&gt;unexpected type&lt;/strong&gt; (for example, a &lt;code&gt;factor&lt;/code&gt; instead of a &lt;code&gt;character&lt;/code&gt;): &lt;a href=&quot;https://stackoverflow.com/q/7246412/2872891&quot;&gt;https://stackoverflow.com/q/7246412/2872891&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, consider using package &lt;code&gt;bigmemory&lt;/code&gt;, recommended in the accepted answer, or similar packages for &lt;em&gt;big data&lt;/em&gt; analysis. For the latter, please see section &lt;strong&gt;&quot;Large memory and out-of-memory data&quot;&lt;/strong&gt; in &lt;em&gt;CRAN Task View&lt;/em&gt; &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;High-Performance and Parallel Computing with R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, an additional note. There is an &lt;strong&gt;ecosystem&lt;/strong&gt; of R packages, built around the &lt;code&gt;arules&lt;/code&gt; package, which includes supporting packages for algorithms (&lt;code&gt;arulesNBMiner&lt;/code&gt;), applications (&lt;code&gt;arulesSequences&lt;/code&gt;, &lt;code&gt;arulesClassify&lt;/code&gt;) and visualization (&lt;code&gt;arulesViz&lt;/code&gt;). You are likely aware of &lt;a href=&quot;http://www.jmlr.org/papers/volume12/hahsler11a/hahsler11a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;that&lt;/a&gt;, but I have decided to include this good-to-know fact just in case and for the sake of completeness.&lt;/p&gt;&#xA;" OwnerUserId="2452" OwnerDisplayName="Aleksandr Blekh" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2014-12-20T08:06:13.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3736" PostTypeId="2" ParentId="3733" CreationDate="2014-12-21T00:40:12.337" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Bayesian_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian optimization&lt;/a&gt; is a principled way of sequentially finding the extremum of black-box functions. What's more, there a numerous software packages that make it easy, such as &lt;a href=&quot;https://bitbucket.org/rmcantin/bayesopt&quot; rel=&quot;nofollow noreferrer&quot;&gt;BayesOpt&lt;/a&gt; and &lt;a href=&quot;http://engineeringblog.yelp.com/2014/07/introducing-moe-metric-optimization-engine-a-new-open-source-machine-learning-service-for-optimal-ex.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MOE&lt;/a&gt;. Another flexible Bayesian framework that you can use for optimization is &lt;a href=&quot;http://en.wikipedia.org/wiki/Gaussian_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian processes&lt;/a&gt;: &lt;a href=&quot;http://ml.dcs.shef.ac.uk/gpss/gpss13/talks/Sheffield-GPSS2013-Osborne.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Global Optimisation with Gaussian Processes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="29575" LastEditDate="2017-03-26T19:26:39.027" LastActivityDate="2017-03-26T19:26:39.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3737" PostTypeId="2" ParentId="2510" CreationDate="2014-12-21T11:43:35.100" Score="1" Body="&lt;p&gt;This is a &lt;strong&gt;tricky&lt;/strong&gt; question, because it's easy and difficult at the same time. &lt;em&gt;Easy&lt;/em&gt;, because there is a lot of resources that potentially can help you make a decision on the topic. &lt;em&gt;Difficult&lt;/em&gt;, because the situation is very different for a particular person (not to mention that their interest might change at any time), which makes extremely difficult for other people to give you a &lt;strong&gt;good&lt;/strong&gt; advice and for you to make the &lt;strong&gt;right&lt;/strong&gt; decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for &lt;strong&gt;data science career&lt;/strong&gt; options, you can certainly consider a &lt;strong&gt;degree path&lt;/strong&gt; (MS or MS + PhD), but you need to be aware of &lt;strong&gt;other options&lt;/strong&gt;. For a comprehensive resource, dedicated to &lt;strong&gt;data science and related degree programs&lt;/strong&gt; (both &lt;em&gt;traditional&lt;/em&gt; and &lt;em&gt;online&lt;/em&gt;), please visit this page: &lt;a href=&quot;http://www.kdnuggets.com/education/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/education/index.html&lt;/a&gt;. A &lt;strong&gt;comprehensive&lt;/strong&gt; review of &lt;strong&gt;all&lt;/strong&gt; these offerings is IMHO an enormous task and is &lt;strong&gt;far&lt;/strong&gt; beyond an answer here on Stack Exchange or, even, a lengthy blog post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, nowadays one is not limited to &lt;em&gt;traditional educational options&lt;/em&gt; and I think it's important to be aware of &lt;strong&gt;other educational options&lt;/strong&gt;. One of the other options include &lt;strong&gt;certifications&lt;/strong&gt; (linked at the above-mentioned page, but, in my opinion, the only certification worth considering is the &lt;a href=&quot;https://www.informs.org/Certification-Continuing-Ed/Analytics-Certification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Certified Analytics Professional&lt;/a&gt; as a &lt;em&gt;solid&lt;/em&gt; and &lt;em&gt;vendor-neutral&lt;/em&gt; certification from a reputable INFORMS). Another option is recently booming &lt;strong&gt;data science intensive educational offerings&lt;/strong&gt;, from &lt;em&gt;short-term&lt;/em&gt; (and often too commercial, to put it lightly) &lt;em&gt;bootcamps&lt;/em&gt; to &lt;em&gt;more solid offerings&lt;/em&gt;, including free, but competitive, ones, such as &lt;a href=&quot;http://insightdatascience.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Insight Data Science Fellows Program&lt;/a&gt;, where one needs to be a PhD to apply, or its sister program &lt;a href=&quot;http://www.insightdataengineering.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Insight Data Engineering Fellows Program&lt;/a&gt;, which doesn't have such requirement. Finally, there is yet another option: &lt;strong&gt;self-study&lt;/strong&gt;. It partially intersects with the certificate option, if one uses &lt;em&gt;massive open online courses (MOOC)&lt;/em&gt; (a review of which deserves a separate comprehensive post), but there are &lt;strong&gt;open curricula&lt;/strong&gt; that might suit one better, such as the &lt;em&gt;Open Source Data Science Masters&lt;/em&gt; curriculum, linked in my &lt;a href=&quot;https://datascience.stackexchange.com/a/742/2452&quot;&gt;earlier relevant answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S.&lt;/strong&gt; While your question focuses on data science, I think that it may be wise to at least consider another career path, given your &lt;em&gt;math background&lt;/em&gt;. I'm talking about &lt;strong&gt;operations research&lt;/strong&gt; field, which is not that far away from data science (and even somewhat intersects with it). While similar, data science is IMHO more statistics-focused, whereas operations research is more math-focused, at least that's how I see it. Despite all the popularity and &quot;gold rush&quot; of data science, operations research career is a solid one, just not as hot. Of course, if you're excited about things like artificial intelligence, machine learning and, especially, deep learning, data science career is the way to go. Whatever you will choose, the good thing is that with your math background it will be easy to change focus, should you decide to. Hope this helps. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2014-12-25T10:41:26.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3738" PostTypeId="1" CreationDate="2014-12-22T03:30:45.673" Score="27" ViewCount="12358" Body="&lt;h3&gt;Background&lt;/h3&gt;&#xA;&lt;p&gt;I'm working on a time series data set of energy meter readings. The length of the series varies by meter - for some I have several years, others only a few months, etc. Many display significant seasonality, and often multiple layers - within the day, week, or year.&lt;/p&gt;&#xA;&lt;p&gt;One of the things I've been working on is clustering of these time series. My work is academic for the moment, and while I'm doing other analysis of the data as well, I have a specific goal to carry out some clustering.&lt;/p&gt;&#xA;&lt;p&gt;I did some initial work where I calculated various features (percentage used on weekends vs. weekday, percentage used in different time blocks, etc.). I then moved on to looking at using Dynamic Time Warping (DTW) to obtain the distance between different series, and clustering based on the difference values, and I've found several papers related to this.&lt;/p&gt;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&lt;p&gt;Will the seasonality in a specific series changing cause my clustering to be incorrect? And if so, how do I deal with it?&lt;/p&gt;&#xA;&lt;p&gt;My concern is that the distances obtained by DTW could be misleading in the cases where the pattern in a time series has changed. This could lead to incorrect clustering.&lt;/p&gt;&#xA;&lt;p&gt;In case the above is unclear, consider these examples:&lt;/p&gt;&#xA;&lt;h3&gt;Example 1&lt;/h3&gt;&#xA;&lt;p&gt;A meter has low readings from midnight until 8AM, the readings then increase sharply for the next hour and stay high from 9AM until 5PM, then decrease sharply over the next hour and then stay low from 6PM until midnight. The meter continues this pattern consistently every day for several months, but then changes to a pattern where readings simply stay at a consistent level throughout the day.&lt;/p&gt;&#xA;&lt;h3&gt;Example 2&lt;/h3&gt;&#xA;&lt;p&gt;A meter shows approximately the same amount of energy being consumed each month. After several years, it changes to a pattern where energy usage is higher during the summer months before returning to the usual amount.&lt;/p&gt;&#xA;&lt;h3&gt;Possible Directions&lt;/h3&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've wondered whether I can continue to compare whole time series, but split them and consider them as a separate series if the pattern changes considerably. However, to do this I'd need to be able to detect such changes. Also, I just don't know if this is a suitable way or working with the data.&lt;/li&gt;&#xA;&lt;li&gt;I've also considered splitting the data and considering it as many separate time series. For instance, I could consider every day/meter combination as a separate series. However, I'd then need to do similarly if I wanted to consider the weekly/monthly/yearly patterns. I &lt;em&gt;think&lt;/em&gt; this would work, but it's potentially quite onerous and I'd hate to go down this path if there's a better way that I'm missing.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3&gt;Further Notes&lt;/h3&gt;&#xA;&lt;p&gt;These are things that have come up in comments, or things I've thought of due to comments, which might be relevant. I'm putting them here so people don't have to read through everything to get relevant information.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'm working in Python, but have rpy for those places where R is more suitable. I'm not necessarily looking for a Python answer though - if someone has a practical answer of what should be done I'm happy to figure out implementation details myself.&lt;/li&gt;&#xA;&lt;li&gt;I have a lot of working &amp;quot;rough draft&amp;quot; code - I've done some DTW runs, I've done a couple of different types of clustering, etc. I think I largely understand the direction I'm taking, and what I'm really looking for is related to how I process my data before finding distances, running clustering, etc. Given this, I suspect the answer would be the same whether the distances between series are calculated via DTW or a simpler Euclidean Distance (ED).&lt;/li&gt;&#xA;&lt;li&gt;I have found these papers especially informative on time series and DTW and they may be helpful if some background is needed to the topic area: &lt;a href=&quot;http://www.cs.ucr.edu/%7Eeamonn/selected_publications.htm&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.ucr.edu/~eamonn/selected_publications.htm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5246" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2020-09-03T13:08:20.857" Title="How to deal with time series which change in seasonality or other patterns?" Tags="|data-mining|clustering|time-series|beginner|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="3739" PostTypeId="2" ParentId="3738" CreationDate="2014-12-22T05:14:16.130" Score="4" Body="&lt;p&gt;If you want to just mine for seasonal patterns, then look into &lt;a href=&quot;http://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow&quot;&gt;autocorrelation&lt;/a&gt;.  If you're looking for a model that can learn seasonal patterns and make forecasts from it, then &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-patched/library/stats/html/HoltWinters.html&quot; rel=&quot;nofollow&quot;&gt;Holt-Winters&lt;/a&gt; is a good start, and &lt;a href=&quot;http://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMA&lt;/a&gt; would be a good thing to follow up with.  &lt;a href=&quot;http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt;&lt;a href=&quot;https://media.readthedocs.org/pdf/a-little-book-of-r-for-time-series/latest/a-little-book-of-r-for-time-series.pdf&quot; rel=&quot;nofollow&quot;&gt;[pdf]&lt;/a&gt; is the tutorial that got me off the ground.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-22T05:14:16.130" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="3740" PostTypeId="1" CreationDate="2014-12-22T09:31:31.747" Score="2" ViewCount="3250" Body="&lt;p&gt;I am performing document (text) classification on the category of websites, and use the website content (tokenized, stemmed and lowercased).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that I have an over-represented category which has vastly more data points than any other (roughly 70% or 4000~ of my data points are of his one category, while about 20 other categories make up the last 30%, some of which have fewer than 50 data points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My first question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could I do to improve the accuracy of my classifier in this case of sparse data for some of the labels? Should I simply discard a certain proportion of the data points in the category which is over-represented? Should I use something other than Gaussian Naive Bayes with tf-idf?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My second question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I perform the classification, I save the tfidf vector as well as the classifier to disk. However, when I re-rerun the classification on the same data, I sometimes get different results from what I initially got (for example, if previously a data point was classified as &quot;Entertainment&quot;, it might receive &quot;News&quot; now). Is this indicative of an error in my implementation, or expected?&lt;/p&gt;&#xA;" OwnerUserId="5199" LastEditorUserId="8820" LastEditDate="2016-01-27T06:39:35.897" LastActivityDate="2016-01-27T06:39:35.897" Title="Improving Naive Bayes accuracy for text classification" Tags="|machine-learning|classification|accuracy|naive-bayes-classifier|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3741" PostTypeId="1" CreationDate="2014-12-22T13:04:38.577" Score="2" ViewCount="189" Body="&lt;p&gt;I'm trying to figure out a strange phenomenon, when I use matrix factorization (the Netflix Prize solution) for a rating matrix: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R = P^T * Q + B_u + B_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with ratings ranging from 1 to 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I evaluate the model by each label's absolute mean average error in test set, the first column is origin_score, the second(we don't transform the data, then train and its  prediction error), the third(we transform the data all by dividing 2, train, and when I use this model to make prediction, firstly reconstruct the matrix and then just multiply 2 and make it back to the same scale)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see, in grade 3-4 (most samples are label from 3-4), it's more precise while in high score range(like 9 and 10, just 2% of the whole traiing set), it's worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;+----------------------+--------------------+--------------------+&lt;br&gt;&#xA;| rounded_origin_score | abs_mean_avg_error | abs_mean_avg_error  | &#xA;+----------------------+--------------------+---------------------+&lt;br&gt;&#xA;| 1.0                  | 2.185225396100167  |  2.559125413626183  |&#xA;| 2.0                  | 1.4072212825108161 |  1.5290497332538155 |&#xA;| 3.0                  | 0.7606073396581479 |  0.6285151230269825 |&#xA;| 4.0                  | 0.7823491986435621 |  0.6419077576969795 |&#xA;| 5.0                  | 1.2734369551159568 |  1.256590210555053  |&#xA;| 6.0                  | 1.9546560495715863 |  2.0461809588933835 |&#xA;| 7.0                  | 2.707229888048017  |  2.8866856489147494 |&#xA;| 8.0                  | 3.5084244741417137 |  3.7212155956153796 |&#xA;| 9.0                  | 4.357185793060213  |  4.590550124054919  |&#xA;| 10.0                 | 5.180752400467891  |  5.468600926567884  |&#xA;+----------------------+--------------------+---------------------+&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've re-train the model several times, and got same result, so I think it's not effect by randomness.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastEditorUserId="1131" LastEditDate="2014-12-22T15:41:29.910" LastActivityDate="2015-05-23T23:24:59.583" Title="Stochastic gradient descent in matrix factorization, sensitive to label's scale?" Tags="|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3742" PostTypeId="1" AcceptedAnswerId="3753" CreationDate="2014-12-22T14:06:45.610" Score="0" ViewCount="549" Body="&lt;p&gt;I've been working in SAS for a few years but as my time as a student with a no-cost-to-me license comes to an end, I want to learn R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to transpose a data set so that all the observations for a single ID are on the same line?  (I have 2-8 observations per unique individual but they are currently arranged vertically rather than horizontally.)  In SAS, I had been using PROC SQL and PROC TRANSPOSE depending on my analysis aims.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID    date        timeframe  fruit_amt   veg_amt &amp;lt;br/&amp;gt; &#xA;4352  05/23/2013  before     0.25        0.75 &amp;lt;br/&amp;gt; &#xA;5002  05/24/2014  after      0.06        0.25 &amp;lt;br/&amp;gt; &#xA;4352  04/16/2014  after      0           0 &amp;lt;br/&amp;gt; &#xA;4352  05/23/2013  after      0.06        0.25 &amp;lt;br/&amp;gt; &#xA;5002  05/24/2014  before     0.75        0.25 &amp;lt;br/&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Desired:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID    B_fr05/23/2013   B_veg05/23/2013  A_fr05/23/2013  A_veg05/23/2013   B_fr05/24/2014   B_veg05/24/2014   (etc)  &amp;lt;br/&amp;gt;&#xA;4352  0.25             0.75             0.06            0.25              .                .  &amp;lt;br/&amp;gt;&#xA;5002  .                .                .               .                 0.75             0.25 &amp;lt;br/&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="6491" LastEditorUserId="6491" LastEditDate="2014-12-22T16:23:45.540" LastActivityDate="2014-12-23T20:36:06.047" Title="Data transposition code in R" Tags="|data-mining|r|dataset|beginner|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="3743" PostTypeId="1" AcceptedAnswerId="3746" CreationDate="2014-12-22T15:52:12.133" Score="2" ViewCount="679" Body="&lt;p&gt;I recently read a lot about the &lt;strong&gt;n-armed bandit problem&lt;/strong&gt; and its solution with various algorithms, for example for webscale content optimization. Some discussions were referring to '&lt;strong&gt;contextual bandits&lt;/strong&gt;', I couldn't find a clear definition what the word 'contextual' should mean here. Does anyone know what is meant by that, in contrast to 'usual' bandits?&lt;/p&gt;&#xA;" OwnerUserId="3132" LastActivityDate="2014-12-22T18:44:18.983" Title="What does 'contextual' mean in 'contextual bandits'?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3745" PostTypeId="1" CreationDate="2014-12-22T18:33:35.363" Score="1" ViewCount="1249" Body="&lt;p&gt;I was looking to learn about Bayesian theory in decision tree and how it avoids overfitting but couldn't find any tutorials for someone just starting. Do you know any resources to learn about it?&lt;/p&gt;&#xA;" OwnerUserId="6523" LastActivityDate="2020-05-16T17:36:09.177" Title="Bayesian Decision Tree" Tags="|machine-learning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3746" PostTypeId="2" ParentId="3743" CreationDate="2014-12-22T18:44:18.983" Score="2" Body="&lt;p&gt;A contextual bandit algorithm not only adapts to the user-click feedback as the algorithm progresses, it also utilizes pre-existing information about the user's (and similar users) browsing patterns to select which content to display.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, rather than starting with no prediction (cold start) with what the user will click (traditional bandit and also traditional A/B testing), it takes other data into account (warm start) to help predict which content to display during the bandit test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See: &lt;a href=&quot;http://www.research.rutgers.edu/~lihong/pub/Li10Contextual.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.research.rutgers.edu/~lihong/pub/Li10Contextual.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2014-12-22T18:44:18.983" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3747" PostTypeId="2" ParentId="3740" CreationDate="2014-12-22T19:12:53.943" Score="3" Body="&lt;p&gt;Regarding your first question...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you anticipate the majority category to be similarly over-represented in real-world data as it is in your training data? If so, perhaps you could perform two-step classification:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train a binary classifier (on all your training data) to predict membership (yes/no) in the majority class.&lt;/li&gt;&#xA;&lt;li&gt;Train a multi-class classifier (on the rest of the training data) to predict membership in the remaining minority classes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-22T19:12:53.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3748" PostTypeId="2" ParentId="2651" CreationDate="2014-12-22T19:14:26.313" Score="9" Body="&lt;p&gt;There's also Richard Socher's recent PhD dissertation on intersection of NLP and deep learning: &lt;a href=&quot;http://nlp.stanford.edu/~socherr/thesis.pdf&quot;&gt;Recursive Deep Learning for Natural Language Processing and Computer Vision&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2014-12-22T19:14:26.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3749" PostTypeId="2" ParentId="3742" CreationDate="2014-12-22T23:55:11.037" Score="0" Body="&lt;p&gt;Try 'arrange(Data.frame.name, ID)' function from package 'dplyr'&lt;/p&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2014-12-22T23:55:11.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3751" PostTypeId="1" CreationDate="2014-12-23T07:15:34.500" Score="5" ViewCount="333" Body="&lt;p&gt;I would like to pick up on the topic of deep learning. Should I begin from the topic of AI before working my way into Deep learning?&lt;/p&gt;&#xA;" OwnerUserId="6536" LastEditorUserId="1367" LastEditDate="2015-05-24T10:11:15.927" LastActivityDate="2015-05-24T10:11:15.927" Title="Route to picking up Deep learning" Tags="|beginner|deep-learning|" AnswerCount="1" CommentCount="6" ClosedDate="2015-07-04T08:06:27.297" ContentLicense="CC BY-SA 3.0" />
  <row Id="3752" PostTypeId="2" ParentId="3745" CreationDate="2014-12-23T17:10:10.777" Score="1" Body="&lt;p&gt;Regarding the being new to decision trees and wanting to get off the ground, I wrote &lt;a href=&quot;http://thegrimmscientist.com/2014/10/23/tutorial-decision-trees/&quot; rel=&quot;nofollow&quot;&gt;a tutorial on decision trees&lt;/a&gt; that will help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding methods to avoid overfitting:  The game for any model is to limit its complexity to what is reasonable given the data you have.  Complexity in decision trees is manifested as adding new decision boundaries, so any limit in complexity is a limit in the decision boundaries it can draw.  Two common ways to do this is to place constraints on when a new decision can be created (a minimum of data in a leaf, significant increase in information, etc) or more simply to limit the max depth of the tree.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-23T17:10:10.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3753" PostTypeId="2" ParentId="3742" CreationDate="2014-12-23T18:11:24.117" Score="0" Body="&lt;p&gt;You can use the &lt;code&gt;reshape2&lt;/code&gt; package for this task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, transform the data to the long format with &lt;code&gt;melt&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape2)&#xA;dat_m &amp;lt;- melt(dat, measure.vars = c(&quot;fruit_amt&quot;, &quot;veg_amt&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;dat&lt;/code&gt; is the name of your data frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, cast to the wide format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dcast(dat_m, ID ~ timeframe + variable + date)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    ID after_fruit_amt_04/16/2014 after_fruit_amt_05/23/2013 after_fruit_amt_05/24/2014 after_veg_amt_04/16/2014&#xA;1 4352                          0                       0.06                         NA                        0&#xA;2 5002                         NA                         NA                       0.06                       NA&#xA;  after_veg_amt_05/23/2013 after_veg_amt_05/24/2014 before_fruit_amt_05/23/2013 before_fruit_amt_05/24/2014&#xA;1                     0.25                       NA                        0.25                          NA&#xA;2                       NA                     0.25                          NA                        0.75&#xA;  before_veg_amt_05/23/2013 before_veg_amt_05/24/2014&#xA;1                      0.75                        NA&#xA;2                        NA                      0.25&#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="106" LastEditorUserId="106" LastEditDate="2014-12-23T20:36:06.047" LastActivityDate="2014-12-23T20:36:06.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3754" PostTypeId="1" CreationDate="2014-12-23T19:46:57.267" Score="1" ViewCount="216" Body="&lt;p&gt;I am working on a project with two data sets. A time vs. speed data set (let's call it traffic), and a time vs. weather data set (called weather). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking to find a correlation between these two sets using Pig. However the traffic data set has the time field, D/M/Y hr:min:sec, and the weather data set has the time field, D/M/Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to this I would like to average the speed per day and put it into a single D/M/Y value inside the traffic file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then plan to use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = JOIN speed BY day, JOIN weather BY day with 'merge'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I will then find the correlation using: (I am borrowing this code from elsewhere)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set = LOAD 'data.txt' AS (speed:double, weather:double)&#xA;rel = GROUP set ALL&#xA;cor = FOREACH rel GENERATE COR(set.speed, set.weather)&#xA;dump cor;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is my first experience with Pig (I've never even used SQL), so I would like to know a few things:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. How can I merge the rows of my traffic file (ie. average D/M/Y hr:min:sec into D/M/Y)?&#xA;2. Is there a better way to find a correlation between the fields of different datasets?&#xA;3. Are the JOIN BY and the COR() functions used appropriately in my above code?  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2614" LastActivityDate="2014-12-23T19:46:57.267" Title="Hadoop/Pig Aggregate Data" Tags="|apache-hadoop|beginner|correlation|apache-pig|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3757" PostTypeId="2" ParentId="116" CreationDate="2014-12-24T13:46:47.367" Score="3" Body="&lt;p&gt;Apart from the fancier methods you could try the Bayes formula&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$P(I | p_1 ... p_n) =   {{P(p_1 ... p_n | I) P(I)} \over \sum_i (P(p_1 ... p_n | i) P(i))}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$P(I | p_1 ... p_n)$&lt;/span&gt; is the probability that a user belongs to age group I if he liked &lt;span class=&quot;math-container&quot;&gt;$p_1, .., p_n$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$P(i)$&lt;/span&gt; is the probability that a user belongs to age group &lt;span class=&quot;math-container&quot;&gt;$I$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$P(p_1 .. p_n | i)$&lt;/span&gt; is the probability that a user liked &lt;span class=&quot;math-container&quot;&gt;$p_1, .., p_n$&lt;/span&gt; if he belongs to age group &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You already have the estimates for &lt;span class=&quot;math-container&quot;&gt;$P(i)$&lt;/span&gt; from your data: this is just the proportion of users in age group I.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To estimate &lt;span class=&quot;math-container&quot;&gt;$P(p_1 ... p_n |i)$&lt;/span&gt;, for each age group &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; estimate the probability (frequency) &lt;span class=&quot;math-container&quot;&gt;$p_{ij}$&lt;/span&gt; to like a page &lt;span class=&quot;math-container&quot;&gt;$j$&lt;/span&gt;. To have &lt;span class=&quot;math-container&quot;&gt;$p_{ij}$&lt;/span&gt; non-zero for all j, you can mix in the frequency for the whole population with a small weight.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then &lt;span class=&quot;math-container&quot;&gt;$log P(p_1...p_n| i) = \sum(log p_{ij}, i = p_1, .., p_n)$&lt;/span&gt;, the sum over all pages that a new user likes. This formula would be approximately true assuming that a user likes the pages in his age group independently.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Theoretically, you should also add log &lt;span class=&quot;math-container&quot;&gt;$(1-p_{ij})$&lt;/span&gt; for all &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; that he hasn't liked, but in practice you should find that the sum of &lt;span class=&quot;math-container&quot;&gt;$log (1-p_{ij})$&lt;/span&gt; will be irrelevantly small, so you won't need too much memory.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;If you or someone else has tried this, please comment about the result.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="29169" LastEditDate="2021-02-09T04:31:08.427" LastActivityDate="2021-02-09T04:31:08.427" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3758" PostTypeId="2" ParentId="3741" CreationDate="2014-12-24T20:02:42.213" Score="2" Body="&lt;p&gt;The larger your target scores, the larger latent variables should be (well, it's not only a magnitude that matters, but also a variance, but it still applies to your case). There's no problem with larger coefficients of latent vectors unless you use regularization (and, likely, you do). In case of regularization your optimal solution will tend towards smaller values, and'd sometimes prefer to sacrifice some accuracy for lower regularization penalty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient Descent doesn't suffer from problem of large coefficients (unless you run into some sort of numerical issues): if the learning rate is tuned properly (there are lots of stuff on it, google), it should arrive to equivalent parameters. Otherwise nobody guarantees you convergence :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The common rule of thumb when doing regression (and your instance of matrix factorization is a kind of regression) is to standardize your data: make it having zero mean and unit variance.&lt;/p&gt;&#xA;" OwnerUserId="811" LastActivityDate="2014-12-24T20:02:42.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3759" PostTypeId="1" CreationDate="2014-12-25T05:24:55.590" Score="1" ViewCount="57" Body="&lt;p&gt;I am new to data science/ machine learning world. I know that in Statistics we assume that a certain event/ process has some particular distribution and the samples of that random process are part of some sampling distribution. The findings from the data could then be generalized by using confidence intervals and significance levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we generalize our findings once we &quot;learn&quot; the patterns in the data set? What is the alternative to confidence levels here?&lt;/p&gt;&#xA;" OwnerUserId="4933" LastActivityDate="2014-12-29T09:21:26.780" Title="How are the findings learnt from the data set are generalized compared to Statistics?" Tags="|machine-learning|statistics|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3760" PostTypeId="1" CreationDate="2014-12-25T10:13:16.273" Score="1" ViewCount="511" Body="&lt;p&gt;I'm trying to determine what is the best number of hidden neurons for my MATLAB neural network. I was thinking to adopt the following strategy:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Loop for some values of hidden neurons, e.g. 1 to 40;&lt;/li&gt;&#xA;&lt;li&gt;For each NN with a fixed number of hidden neurons, perform a certain number of   training (e.g. 40, limiting the number of epoch for time reasons: I was thinking to doing this because the network seems to be hard to train, the MSE after some epochs is very high)&lt;/li&gt;&#xA;&lt;li&gt;Store the MSE obtained with all the nets with different number of hidden neurons&lt;/li&gt;&#xA;&lt;li&gt;Perform the previous procedure more than 1 time, e.g. 4, to take into account the initial random weight, and take the average of the MSEs&lt;/li&gt;&#xA;&lt;li&gt;Select and perform the &quot;real&quot; training on a NN with a number of hidden neurons such that the MSE previously calculated is minimized&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The MSE that I'm referring is the validation MSE: my samples splitting in trainining, testing and validation to avoid overfitting is 70%, 15% and 15% respectively)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other informations related to my problem are:&lt;br&gt;&#xA;fitting problem&lt;br&gt;&#xA;9 input neurons&lt;br&gt;&#xA;2 output neurons&lt;br&gt;&#xA;1630 samples  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This strategy could be work? Is there any better criterion to adopt? Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Test done, so the result suggest me to adopt 12 neurons? (low validation MSE and  number of neurons lower than 2*numberOfInputNeurons? but also 18 could be good...&#xA;&lt;img src=&quot;https://i.stack.imgur.com/qxOSV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="6559" LastEditorUserId="6559" LastEditDate="2014-12-26T20:37:37.720" LastActivityDate="2022-04-27T22:30:55.150" Title="Neural Network Hidden Neuron Selection Strategy" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3761" PostTypeId="2" ParentId="3760" CreationDate="2014-12-25T15:40:43.283" Score="2" Body="&lt;p&gt;A rule of thumb approach is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;start with a number of hidden neurons equal (or little higher) that the number of features. &lt;/li&gt;&#xA;&lt;li&gt;In your case it would be 9. My suggestion is to start with 9*2 = 18 to cover a wider range of possibilities.&lt;/li&gt;&#xA;&lt;li&gt;Be sure your test and validation sets are selected &quot;fairly&quot;: a random selection and varying the seed some number of times to test different configurations would be ok.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, a number of neurons equal to the number of features will tend to make each hidden neuron try to learn &lt;em&gt;that special thing&lt;/em&gt; that each feature is adding, so will could say it is &quot;learning each feature&quot; separately. Although this sounds good it might tend to overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your number of inputs and your dataset size is small its ok to start with a hidden layer size of the double (18) and start lowering down. When the training error and test error stabilize in a difference lower than a threshold then you could have found a better generalizing model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks are very good at finding local optima by exploring deeply a solution from a starting point. However, the starting point it is also very important. If you are not getting a good generalization you might try to find good initial starting points with methods of Hybrid Neural Networks. A common one, for example, is using genetic algorithms to find an initial combination of weights and then start the neural from that point. Given that your search space would be better covered (in case your problem actually needs that).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for every problem in machine learning is very important to clean your data before introducing it to the NN. Try to be very detailed in order to avoid the NN to learn things you already know. For example if you know how two features are correlated improve the input data by making this correlation explicit so less workload is given to the NN (that might actually get you in trouble).&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2014-12-25T15:40:43.283" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="3762" PostTypeId="2" ParentId="3760" CreationDate="2014-12-25T18:10:13.447" Score="0" Body="&lt;h2&gt;Top level:&lt;/h2&gt;&#xA;&lt;p&gt;The rule is to chose the most simple network that can perform satisfactorily.  See &lt;a href=&quot;https://ieeexplore.ieee.org/document/5726612&quot; rel=&quot;nofollow noreferrer&quot;&gt;this publication&lt;/a&gt; and its &lt;a href=&quot;https://web.archive.org/web/20181025152731/https://www.pdx.edu/sites/www.pdx.edu.sysc/files/EvaluatingANNs.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PDF&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2&gt;The Methodology:&lt;/h2&gt;&#xA;&lt;p&gt;So do your proposed test (training many networks at each number of hidden nodes) and plot the results.  At the minimum number of nodes, you'll see the worst performance.  As you increase the number of nodes, you'll see an increase in performance (reduction of error).  At some point N, you'll see the performance seems to hit an upper limit and increasing nodes beyond this will stop giving significant performance gains.  Further increases may start to hurt performance a little as training gets more difficult).  That point N is the number of nodes you want.&lt;/p&gt;&#xA;&lt;h1&gt;How it worked for me:&lt;/h1&gt;&#xA;&lt;p&gt;The first time I used this methodology, it created a beautiful almost-sigmoid-looking function with a very clear number of nodes that were needed to achieve good results.  I hope this works for you as well as it worked for me.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastEditorUserId="25180" LastEditDate="2022-04-27T22:30:55.150" LastActivityDate="2022-04-27T22:30:55.150" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3763" PostTypeId="1" CreationDate="2014-12-26T06:53:29.670" Score="4" ViewCount="613" Body="&lt;p&gt;In &lt;a href=&quot;http://www.wired.com/2014/01/how-to-hack-okcupid/all/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; article, Chris McKinlay says he used AdaBoost to choose the proper &quot;importances&quot; of questions he answered on okcupid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't read and don't want to read the article, or are unfamiliar with okcupid and the question system, here's the data and problem he had:&#xA;The goal is to &quot;match&quot; as highly as possible with as many users as possible, each of whom may have answered an arbitrary number of questions. These questions may have between 2 and 4 answers each, and for the sake of simplicity, let's pretend that the formula for a match% $\ M $ between you and another user is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\ M = Q_a/Q_c $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\ Q_c $ is the number of questions you and the other user have in common, and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\ Q_a $ is the number of questions you both answered with the same value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real formula is slightly more complex, but the approach would be the same regarding &quot;picking&quot; a correct answer (he actually used boosting to find the ideal &quot;importance&quot; to place on a given question, rather than the right answer).&#xA;In any case, the point is you want to pick a certain value for each question, such that you maximize your match% with as many users as possible - something you might quantify by the sum of $\ M $ over all users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I've watched the MIT course on AI up to and including the lecture on boosting, but I don't understand how you would apply it to a problem like this. Honestly I don't even know where to begin with choosing rules for the weak learners. I don't have any &quot;rules&quot; about what values to choose for each question (if the user is under 5'5, choose A, etc) - I'm just trying to fit the data I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this not the way boosting is supposed to be used? Is there likely some other optimization left out of how he figured this out?&lt;/p&gt;&#xA;" OwnerUserId="6568" LastActivityDate="2015-05-18T00:09:04.393" Title="How to apply AdaBoost to more &quot;complex&quot; (non-binary) classifications/data fitting?" Tags="|machine-learning|bigdata|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3764" PostTypeId="2" ParentId="3738" CreationDate="2014-12-26T10:01:46.500" Score="20" Body="&lt;p&gt;After reading your question, I became curious about the topic of &lt;em&gt;time series clustering&lt;/em&gt; and &lt;em&gt;dynamic time warping (DTW)&lt;/em&gt;. So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO &lt;em&gt;relevant references&lt;/em&gt; (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped research papers, as I was more interested in &lt;em&gt;practical aspects&lt;/em&gt; of the topic.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Resources:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;UCR Time Series Classification/Clustering&lt;/strong&gt;: &lt;a href=&quot;http://www.cs.ucr.edu/%7Eeamonn/time_series_data&quot; rel=&quot;nofollow noreferrer&quot;&gt;main page&lt;/a&gt;, &lt;a href=&quot;http://www.cs.ucr.edu/%7Eeamonn/UCRsuite.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;software page&lt;/a&gt; and &lt;a href=&quot;http://www.cs.ucr.edu/%7Eeamonn/SIGKDD_trillion.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;corresponding paper&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Time Series Classification and Clustering with Python&lt;/strong&gt;: &lt;a href=&quot;http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;a blog post&lt;/a&gt; and an &lt;a href=&quot;http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;ipython notebook&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Capital Bikeshare: Time Series Clustering&lt;/strong&gt;: &lt;a href=&quot;http://ofdataandscience.blogspot.com/2013/03/capital-bikeshare-time-series-clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;another blog post&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Dynamic Time Warping using rpy and Python&lt;/strong&gt;: &lt;a href=&quot;https://nipunbatra.wordpress.com/2013/06/09/dynamic-time-warping-using-rpy-and-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;another blog post&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale&lt;/strong&gt;: &lt;a href=&quot;http://practicalquant.blogspot.com/2012/10/mining-time-series-with-trillions-of.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;another blog post&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Time Series Analysis and Mining in R&lt;/strong&gt; (to add R to the mix): &lt;a href=&quot;http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;yet another blog post&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;And, finally, two &lt;strong&gt;tools implementing/supporting DTW&lt;/strong&gt;, to top it off: &lt;a href=&quot;http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;R package&lt;/a&gt; and &lt;a href=&quot;http://mlpy.sourceforge.net&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python module&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="86627" LastEditDate="2020-09-03T13:08:20.857" LastActivityDate="2020-09-03T13:08:20.857" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="3766" PostTypeId="1" CreationDate="2014-12-26T16:25:27.373" Score="2" ViewCount="1238" Body="&lt;p&gt;If I execute the following code I have no problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(foreign)&#xA;require(nnet)&#xA;require(ggplot2)&#xA;require(reshape2)&#xA;&#xA;ml &amp;lt;- read.dta(&quot;https://stats.idre.ucla.edu/stat/data/hsbdemo.dta&quot;)&#xA;ml$prog2 &amp;lt;- relevel(ml$prog, ref = &quot;academic&quot;)&#xA;test &amp;lt;- multinom(prog2 ~ ses + write, data = ml)&#xA;predict(test, newdata = dses, &quot;probs&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but if I try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(caret)&#xA;ml &amp;lt;- read.dta(&quot;http://www.ats.ucla.edu/stat/data/hsbdemo.dta&quot;)&#xA;ml$prog2 &amp;lt;- relevel(ml$prog, ref = &quot;academic&quot;)&#xA;test &amp;lt;- train(prog2 ~ ses + write,method=&quot;multinom&quot; ,data = ml)&#xA;predict(test$finalModel, newdata = dses, &quot;probs&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it returns &lt;code&gt;Error in eval(expr, envir, enclos) : object 'sesmiddle' not found&lt;/code&gt;, why?&lt;/p&gt;&#xA;" OwnerUserId="6572" LastEditorUserId="24968" LastEditDate="2018-09-08T22:06:18.120" LastActivityDate="2018-09-08T22:06:18.120" Title="predict with Multinomial Logistic Regression" Tags="|machine-learning|r|logistic-regression|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="3767" PostTypeId="2" ParentId="3766" CreationDate="2014-12-26T17:26:02.033" Score="2" Body="&lt;p&gt;That's not the error I get and I'm thinking that you left out some code. I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;predict(test$finalModel, newdata = dses, &quot;probs&quot;)&#xA;Error in as.data.frame(newdata) : object 'dses' not found&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know why I see this so much, but you should avoid using the &lt;code&gt;finalModel&lt;/code&gt; object for prediction. &lt;code&gt;train&lt;/code&gt; is doing some things that the model from &lt;code&gt;multinom&lt;/code&gt; sin't aware of (such as expanding dummy variables). Try using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;predict(test, newdata = dses, &quot;probs&quot;)&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and never use &lt;code&gt;predict(test$finalModel)&lt;/code&gt; again...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Max&lt;/p&gt;&#xA;" OwnerUserId="810" LastActivityDate="2014-12-26T17:26:02.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3770" PostTypeId="1" AcceptedAnswerId="3801" CreationDate="2014-12-28T11:29:14.240" Score="12" ViewCount="13240" Body="&lt;p&gt;Google Trends returns weekly data so I have to find a way to merge them with my daily/monthly data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have done so far is to break each serie into daily data, for exemple: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;from:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2013-03-03 - 2013-03-09 37&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2013-03-03 37 &#xA;2013-03-04 37&#xA;2013-03-05 37&#xA;2013-03-06 37&#xA;2013-03-07 37&#xA;2013-03-08 37&#xA;2013-03-09 37&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is adding a lot of complexity to my problem. I was trying to predict google searchs from the last 6 months values, or 6 values in monthly data. Daily data would imply a work on 180 past values. (I have 10 years of data so 120 points in monthly data / 500+ in weekly data/ 3500+ in daily data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other approach would be to &quot;merge&quot; daily data in weekly/monthly data. But some questions arise from this process. Some data can be averaged because their sum represent something. Rainfall for example, the amount of rain in a given week will be the sum of the amounts for each days composing the weeks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case I am dealing with prices, financial rates and other things. For the prices it is common in my field to take volume exchanged into account, so the weekly data would be a weighted average. For financial rates it is a bit more complex a some formulas are involved to build weekly rates from daily rates.  For the other things i don't know the underlying properties. I think those properties are important to avoid meaningless indicators (an average of fiancial rates would be a non-sense for example). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So three questions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like breaking weekly/monthly data into daily data like i've done is somewhat wrong because I am introducing quantities that have no sense in real life. So almost the same question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least: &lt;strong&gt;when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?&lt;/strong&gt; I think this is a compromise between the number of data and the complexity of the model but I can't see any strong argument to choose between those options. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: if you know a tool (in R Python even Excel) to do it easily it would be very appreciated.&lt;/p&gt;&#xA;" OwnerUserId="303" LastEditorUserId="303" LastEditDate="2014-12-28T11:50:12.840" LastActivityDate="2015-01-06T01:06:26.827" Title="How to merge monthly, daily and weekly data?" Tags="|time-series|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3771" PostTypeId="1" AcceptedAnswerId="4872" CreationDate="2014-12-28T14:42:52.403" Score="1" ViewCount="183" Body="&lt;p&gt;I am trying to predict clients comportement from market rates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value of the products depends on the actual rate but this is not enough. The comportement of the client also depends on their awareness wich depends on the evolution of rates. I've added this in model using past 6 month rates as features in polynomial regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact media coverage of rate mostly depends on rate variations and I wanted to add that in my model.  The idea would be to add a derivative/variation of rate as a feature. But I anticipated something wrong, example with only two month , my variation will be of the form $x_n - x_{n-1}$ that is a simple linear combination of actual and past rates. So for a 1d polynomial regression i will have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ x_{n+1} = a * x_{n} +  b * x_{n-1} + c * (x_{n} - x_{n-1})$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;instead of: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ x_{n+1} = a_0 * x_{n} +  b_0 * x_{n-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;wich is strictly equivalent with $ a + c = a_0 $ and $b-c= b_0$. Higher polynomial degree results in a more or less equivalent result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking about a way to include derivative information but it seems not possible. So I am wondering if all the information is included in my curve. Is this a general idea ? all information is somewhat directly contained in data and modifications of features will result in higher order objective function ?&lt;/p&gt;&#xA;" OwnerUserId="303" LastEditorUserId="303" LastEditDate="2015-01-09T15:28:36.703" LastActivityDate="2015-01-14T01:08:04.997" Title="Time series: variations as a feature" Tags="|time-series|feature-selection|predictive-modeling|optimization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3772" PostTypeId="1" AcceptedAnswerId="3774" CreationDate="2014-12-28T14:58:45.320" Score="5" ViewCount="1358" Body="&lt;p&gt;I am trying to predict a time serie from another one. My approach is based on a moving windows. I predict the output value of the serie from the following features: the previous value and the 6 past values of the source serie. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it usefull to add the previous value of the time serie ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like I don't use all the information contained in the curve to predict futures values. But I don't see how it would be possible to use all previous data to predict a value (first, the number of features would be growing trough time...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the caveats of a 6 month time-window approach ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any paper about differents method of feature selection for time-series ? &lt;/p&gt;&#xA;" OwnerUserId="303" LastActivityDate="2014-12-29T06:53:32.200" Title="Time series prediction" Tags="|time-series|feature-selection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3773" PostTypeId="2" ParentId="3772" CreationDate="2014-12-28T20:10:11.300" Score="3" Body="&lt;p&gt;I'm fairly new to this myself, but have spent a lot of time recently learning about time series and hope that I can help fellow learners. If I had the reputation to comment I'd ask you a few things first, but I can't. I'll happily do further work and edit this response if you respond or make edits to your question. With those caveats out of the way:&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Is it useful to use the previous value as a feature?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;One of the first things I would say is that the correct aspects to be looking at in your data very much depends on the nature of the data, as well as what you're trying to do with it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It sounds like you have monthly values, but it's not clear how far into the future you're wanting to predict, or how much historic data you have access to.&lt;/li&gt;&#xA;&lt;li&gt;We also don't know what these two series represent, or why one time series is being used to predict the other - and without that, I don't think anyone will be able to tell you whether the previous value of the series to be predicted is valuable information or not.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Any caveats to using a 6 month time window?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;One obvious caveat to only using the last 6 months is that if there's any seasonality over the year-long period then you're going to miss it.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you're not sure: if you have multiple years of information, try plotting the series you want to predict over multiple years. You may well be able to see whether the series generally increases or decreases at certain times of year. If you can share this plot here, it might help people answer your questions in more depth.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As far as caveats about this time-window approach, I'm not too clear from your post what algorithm you're using to predict values. More information on that would be helpful; it's possible that rather than questioning what features to select, you should be questioning what methodology to use for forecasting.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Helpful further reading?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Once you've provided more information I'll be happy to tackle your last question on suitable reading if I'm able to. For now, I will say that there's a &lt;em&gt;lot&lt;/em&gt; of information available, but a lot of it is academic in nature. Quite frequently papers aren't easy to digest, seem to contradict one another, or are only relevant to specific situations. This is a rapidly growing and changing field, so it's sometimes difficult to find a clear best practice or consensus opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, it might be worth looking at some of the free, online courses available to see if any would help you understand the area you're interested in:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.coursera.org/courses?query=data%20science&quot; rel=&quot;nofollow&quot;&gt;Coursera &quot;data science&quot; related courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5246" LastActivityDate="2014-12-28T20:10:11.300" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="3774" PostTypeId="2" ParentId="3772" CreationDate="2014-12-29T06:53:32.200" Score="4" Body="&lt;p&gt;Let me give you a few simple approaches in time series analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first approach consists in using previous values of your time series $Y_{t}$ as in $Y_{t} = \phi_{1}Y_{t-1} + ... + \phi_{n}Y_{t-n}$. In case you don't know, these models are called autoregressive (AR) models. This answers your first question. Of course it is useful to include the previous value of your time series. There is a whole set of models based on that idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second approach is taking a window and extracting some features to describe the time series at each point in time. Then you use a conventional machine learning technique to predict future values as typically done. This is more common in a classification or regression setting but future values can be thought of as classifying future values. This technique has the advantage of dramatically reducing the number of features, although you usually lose characteristics associated with time. This addresses your second concern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another model that could be helpful in your case is the vector autoregressive model (VAR) (using &lt;a href=&quot;http://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;'s notation):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left( \begin{array}{ccc}&#xA;y_{1,t} \\&#xA;y_{2,t} &#xA;\end{array}\right) = \left( \begin{array}{ccc}c_{1} \\ c_{2}\end{array}\right) + \left( \begin{array}{ccc}A_{1,1} &amp;amp; A_{1,2} \\ A_{2,1} &amp;amp; A_{2,2}\end{array}\right)\left( \begin{array}{ccc}&#xA;y_{1,t-1} \\&#xA;y_{2,t-1} &#xA;\end{array}\right) + \left( \begin{array}{ccc}&#xA;e_{1,t} \\&#xA;e_{2,t} &#xA;\end{array}\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can see that $y_{1,t}$ has a contribution from its previous value $t_{1,t-1}$ but also includes the value of the other series $y_{2,t-1}$ in a linear combination. As usual, the purpose is to find the elements of $A_{i,j}$ that minimize some measure of error between observed values and estimated values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A general suggestion: The first thing you need to do is to test the autocorrelation of your first series in order to confirm that an autoregressive approach is suitable and then test the cross correlation between both series to support the idea that using the second series to improve your predictions is appropriate.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2014-12-29T06:53:32.200" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3775" PostTypeId="2" ParentId="2507" CreationDate="2014-12-29T08:38:44.673" Score="2" Body="&lt;p&gt;This does not seem a Data Science problem. However there are very nice tools to do exactly that, checkout: logstash, flume and fluentd. Actually if you want to be able to filter in fast and &quot;smart&quot; way checkout Kibana from the guys of ElastichSearch (&lt;a href=&quot;http://www.elasticsearch.org/overview/kibana&quot; rel=&quot;nofollow&quot;&gt;http://www.elasticsearch.org/overview/kibana&lt;/a&gt;). Those tools are enough to solve your problem in a very efficient way.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2014-12-29T08:38:44.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3776" PostTypeId="2" ParentId="3759" CreationDate="2014-12-29T09:03:08.853" Score="1" Body="&lt;p&gt;Bias-Variance decomposition is one way, and V-C dimension/bound is another...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both of these are metrics you can use to get a feeling of how confident you should be that your training results will generalize to out-of-sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;V-C dimension focuses on the results of &lt;em&gt;this&lt;/em&gt; learning algorithm outcome.&#xA;Bias-Variance focuses on the expected outcome of the algorithm itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pick your poison - I sincerely hope that this was helpful.&lt;/p&gt;&#xA;" OwnerUserId="6597" LastEditorUserId="6597" LastEditDate="2014-12-29T09:08:32.210" LastActivityDate="2014-12-29T09:08:32.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3777" PostTypeId="2" ParentId="3759" CreationDate="2014-12-29T09:21:26.780" Score="0" Body="&lt;p&gt;Essentially machine learning uses non parametric methods...The assumption is that you have enough data and (computation)  time&#xA; You Identify best model by cross validation (rather than eg  assessing significance of coefficients) ,  and estimate prediction error by using test set.  Confidence intervals can also be generated by bootstrapping. &lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2014-12-29T09:21:26.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3779" PostTypeId="1" CreationDate="2014-12-29T17:27:59.763" Score="2" ViewCount="235" Body="&lt;p&gt;Let's assume I'm building a content recommendation engine for online content. I have web log data which I can import into a graph, containing a user ID, the page they viewed, and a timestamp for when they viewed it. Essentially, it's a history of which pages each user has viewed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used Neo4J and Cypher to write a simple traversal algorithm. For each page (node) I want to build recommendations for, I find which pages are most popular amongst other users who have also visited this page. That seems to give decent results. But, I'd like to explore alternatives to see which method gives the most relevant recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to my simple traversal, I'm curious if there are graph-level properties I can utilize to build another set of recommendations with this data set. I've looked at &lt;a href=&quot;http://snap.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;SNAP&lt;/a&gt;, it has a good library for algorithms like Page Rank, Clauset-Newman-Moore community detection, Girvan-Newman community detection, betweenness centrality, K core, and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many algorithms to choose from. Which algorithms have you had success with? Which would you consider trying?&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-05-11T21:06:39.957" Title="Which Graph Properties are Useful for Predictive Analytics?" Tags="|recommender-system|predictive-modeling|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3780" PostTypeId="1" CreationDate="2014-12-30T06:43:24.340" Score="1" ViewCount="51" Body="&lt;p&gt;Recently, I studied a paper called &quot;What Does Your Chair Know About Your Stress Level?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be download at the link below.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://www.barnrich.ch/wiki/data/media/pub/2010_what_does_your_chair_know_about_your_stress_level.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.barnrich.ch/wiki/data/media/pub/2010_what_does_your_chair_know_about_your_stress_level.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;at page 210, Fig.5 (picture 5) mentioned that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Spectra of the norm of the CoP vector during the experiment (stages 3–7) for the same subject used in Fig. 4.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How did they do to transform Fig.4 to Fig.5? And what do x-axis and y-axis mean in Fig.5?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fig4&#xA;&lt;img src=&quot;https://i.stack.imgur.com/Iq7qc.png&quot; alt=&quot;Fig4&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fig5&#xA;&lt;img src=&quot;https://i.stack.imgur.com/7HPxu.png&quot; alt=&quot;Fig5&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="6602" LastActivityDate="2014-12-30T06:43:24.340" Title="How to transform one graph to a spectrum?" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3781" PostTypeId="1" CreationDate="2014-12-30T10:33:08.240" Score="4" ViewCount="5186" Body="&lt;p&gt;I have a &lt;strong&gt;dataset of xyz coordinates with a date component&lt;/strong&gt; in a pandas dataframe&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ex:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;date1: $[x_1,y_1,z_1]$,&lt;/li&gt;&#xA;&lt;li&gt;date2: $[x_2,y_2,z_2]$,&lt;/li&gt;&#xA;&lt;li&gt;date3: $[x_3,y_3,z_3]$,&#xA;..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to &lt;strong&gt;classify a sample of object positions over the period of a week&lt;/strong&gt;&#xA;(using indexes to re-map the classification label back to the date), like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Week 1: $[x_1,y_1,z_1], [x_2, y_2, z_2], [x_3,y_3,z_3], [x_4,y_4,z_4], [x_5,y_5,z_5], [x_6,y_6,z_6], [x_7,y_7,z_7]$,&lt;/li&gt;&#xA;&lt;li&gt;Week 2: $[x_8,y_8,z_8],[x_9,y_9,z_9],[x_{10},y_{10},z_{10}],[x_{11},y_{11},z_{11}],[x_{12},y_{12},z_{12}],[x_{13},y_{13},z_{13}],[x_{14},y_{14},z_{14}]$,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When I try to run KMeans it returns&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k_means = KMeans(n_clusters=cclasses)&#xA;k_means.fit(process_set.hpc)&#xA;date_classes = k_means.labels_&#xA;&#xA;ValueError: Found array with dim 3. Expected &amp;lt;= 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do I have to run it through Principal Component Analysis (PCA) first? if so, how do I maintain date mapping to the classification created?&lt;/li&gt;&#xA;&lt;li&gt;Are there any other methods I could use?&lt;/li&gt;&#xA;&lt;li&gt;Am I doing everything completely backwards and should consider a different approach, any thoughts?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="6606" LastEditorUserId="1367" LastEditDate="2015-01-15T13:41:08.957" LastActivityDate="2015-04-15T15:10:56.650" Title="Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?)" Tags="|python|scikit-learn|k-means|dimensionality-reduction|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="3783" PostTypeId="1" CreationDate="2014-12-30T12:55:06.247" Score="0" ViewCount="161" Body="&lt;p&gt;Imagine modeling the &quot;&lt;em&gt;input(plaintext) - output(ciphertext)&quot;&lt;/em&gt; pairs of an encryption algorithm as a data science problem. Very informally, the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output. This is counter-intuitive to classic regression problems, which are frequently used for prediction. Say, informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we model this as data science problem? Given all pairs of two different encryption schemes, can we determine which is stronger just by using the input-output pairs of both the schemes? Is there any other way apart from regression to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="4695" LastEditorUserId="13413" LastEditDate="2015-10-21T02:42:36.147" LastActivityDate="2015-10-21T18:49:41.357" Title="How to model this &quot;un predicatability&quot; problem?" Tags="|machine-learning|regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="3784" PostTypeId="1" CreationDate="2014-12-30T14:23:58.323" Score="2" ViewCount="59" Body="&lt;p&gt;while comparing two different algorithms to feature selection I stumbled upon the follwing question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given dataset with a discrete class variable we want to train a naive bayes classifier. We decide to conduct feature selection during preprocessing using naive bayes in a wrapper approach. &#xA;Does this method of feature selection consider the size of the used feature subsets? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/2Vh2Q.png&quot; alt=&quot;Naive Bayes Classification&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When considering how NB classifies a given instance, the size of the feature subset being used for classification only influences the number of parts that the product of the conditional dependencies has but that does not make a difference, or does it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It'd be great if someone could offer a solid explanation since for me it's more of a gut feeling at the moment.&lt;/p&gt;&#xA;" OwnerUserId="6608" LastActivityDate="2015-04-30T01:37:30.127" Title="Does a NB wrapper consider feature subset size?" Tags="|classification|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3785" PostTypeId="2" ParentId="3740" CreationDate="2014-12-30T19:45:57.283" Score="3" Body="&lt;p&gt;The correct term for what you're describing here is 'class imbalance' or 'class imbalance problem' . It'd be great if you could include that in the title to have a more meaningful title.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concerning your first question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you plotted a confusion matrix of the resulting classifications? Is the reason why the accuracy is not satisfying really that instances are wrongly classified as the most common class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your context of application it seems that you could use a certain degree of oversampling. To what extent this can be applied should depend on the frequency of each underrepresented class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a reasonably high variation in the value distributions on the underrepresented class instances then one could argue that this should be considered when applying the oversampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the approach Charlie suggested in his answer could be considered, given that the instances of the underrepresented classes would form a dataset that is suitable for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt; I haven't used naive bayes for text classification yet, so I'm not too sure how your attributes look like exactly. Do you just use the frequency of the terms that scored best with tfidf? More general, do you have discrete or continously valued attributes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the latter is the case you should consider using some kind of discretisation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regarding your second question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you splitting the dataset in any way? Normally, given that the classifier has been trained on the same data, the outcome for the same instance should also be the same.&lt;/p&gt;&#xA;" OwnerUserId="6608" LastEditorUserId="6608" LastEditDate="2014-12-30T19:56:36.333" LastActivityDate="2014-12-30T19:56:36.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3786" PostTypeId="1" CreationDate="2014-12-30T21:45:13.760" Score="1" ViewCount="747" Body="&lt;p&gt;I have an excel file containing lot of columns. I want to create a graph database in Neo4J with all these columns as a separate node so that I can establish relationship between them and play around with the cypher query to get to know my data quite well. I did write a cypher query to make each column as a node, but when I run the relationship query it shows me the name of one node and gives a random number for all the related nodes despite changing the &lt;strong&gt;captions in the properties&lt;/strong&gt; section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Create statement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LOAD CSV WITH HEADERS FROM &quot;File Location&quot; AS row CREATE (:NodeName {DATE_OCCURED: row.Date});&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Query to visualize the relationship between nodes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MATCH (a)-[:`REL NAME`]-&amp;gt;(b) RETURN a,b LIMIT 25;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me the values the node &quot;a&quot; and random numbers for all the node &quot;b&quot;. I don't know where I am going wrong. I thought it has something to do with the way the data has been set up in my file. So I created a file for each column of my main file and wrote the create statements to get the nodes. I still don't get the actual values of the related nodes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/xxKVi.png&quot; alt=&quot;Sample output for the relationship query&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="5043" LastEditDate="2014-12-31T14:48:00.310" LastActivityDate="2014-12-31T14:48:00.310" Title="How to create separate nodes for each column in a spreadsheet using Neo4J?" Tags="|neo4j|csv|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3787" PostTypeId="2" ParentId="3784" CreationDate="2014-12-31T00:09:35.943" Score="1" Body="&lt;h1&gt;Short Version&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The difference is in the trade-off between complexity of model and how hard it will be to train.  The more states your variable can take, the much more data you'll need to train it.  It sounds like you're reducing the feature space through some front-end processing.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Example&lt;/h1&gt;&#xA;&#xA;&lt;h3&gt;Context:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Say you're using NBC to model if you go outside given the high temperature that day.  Let's say temperature is given to you as an int in Celsius which just happened to range between 0 and 40 degrees.  Over one year of running the experiment - recording the temperature and if you went outside - you'll have 365 data points.  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Internal Representation:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The internal structure the NBC uses will be a matrix with 82 values (41 values for your one input variable * 2 because you have two states in your output class).  That means each bin will have an average of &lt;code&gt;365/82 = 8.9ish&lt;/code&gt; samples.  In practice, you'll probably see a few bins with lots more samples than this and a many bins with 0 samples.  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;A Pitfall&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Say you saw 8 cases at a temperature of 5 C (all of which you stayed inside) and nothing at a temp of 3 or 4 C.  If, after the model is built, you 'ask it' what class a temp of 4 C should be in.  It won't know.  Intuitively, we'd say &quot;stay inside&quot;, but the model won't.  One way to fix this is to bin the classes 0,1,2,... into larger groups of temperatures (i.e., class 0 for temp 0-3, class 1 for temp 4-7, etc).  The extreme case of this would be two temperature states &quot;high&quot; and &quot;low&quot;.  The actual cut-off should depend more on the data observed, but one such scheme is converting temperatures within &lt;code&gt;0-20&lt;/code&gt; to &quot;low&quot; and &lt;code&gt;21-40&lt;/code&gt; is &quot;high&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Discussion&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This front end processing seems to be what you're talking about as the &quot;wrapper&quot; around the NBC.  The result of our &quot;high&quot; and &quot;low&quot; binning scheme will result in a much smaller model (2 input states and 2 output classes gives you a &lt;code&gt;2*2 = 4&lt;/code&gt; number of classes.  This will be way easier to process and will take way less data to confidently train at the expense of complexity.  One example of a drawback of such a scheme is:  Say the actual pattern is that you love spring and fall weather and only go outside when it's not too hot or not too cold.  So your outside adventures are evenly split across the upper part of the &quot;low&quot; class and the lower part of the &quot;high&quot; class giving you a really useless model.  If this were the case, a 3-class bin scheme of &quot;high&quot;, &quot;medium&quot;, and &quot;low&quot; would be best.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2014-12-31T00:09:35.943" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="3788" PostTypeId="1" CreationDate="2014-12-31T02:50:32.863" Score="2" ViewCount="641" Body="&lt;p&gt;What's the best / easiest way to create a graph from address data?  For example, if I have 100 houses from all across a city is there any easy way to determine the shortest distance between two houses and all that good stuff?  Would this require changing the data into coordinates and using GIS software or can I get away with using Python or R?&lt;/p&gt;&#xA;" OwnerUserId="6611" LastActivityDate="2016-01-28T12:41:51.127" Title="Creating a Graph from Address Data" Tags="|python|graphs|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3789" PostTypeId="5" CreationDate="2014-12-31T15:34:17.397" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2014-12-31T15:34:17.397" LastActivityDate="2014-12-31T15:34:17.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3790" PostTypeId="4" CreationDate="2014-12-31T15:34:17.397" Score="0" Body="Statistics is a scientific approach to inductive inference and prediction based on probabilistic models of the data. By extension, it covers the design of experiments and surveys to gather data for this purpose." OwnerUserId="6622" LastEditorUserId="13526" LastEditDate="2017-06-18T08:05:52.913" LastActivityDate="2017-06-18T08:05:52.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3791" PostTypeId="2" ParentId="3788" CreationDate="2014-12-31T16:18:21.237" Score="0" Body="&lt;p&gt;If you have latitude/longitude coordinate data, there should be no problem accomplishing this using great circle calculations which could indeed be accomplished in Python, R, or essentially any other language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an article on several methods and calculations for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.movable-type.co.uk/scripts/latlong.html&quot; rel=&quot;nofollow&quot;&gt;Calculate distance, bearing and more between Latitude/Longitude points&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main issue with street address data alone is of course the lack of contextual information regarding the physical layout of the streets. Given a sufficiently complete scale map of the relevant area, you could also calculate a distance. That said, the haversine formula discussed in the article above would have a greater accuracy unless the scale map mentioned was also plotted as the surface of a sphere.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2014-12-31T16:18:21.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3792" PostTypeId="1" CreationDate="2014-12-31T17:57:48.243" Score="2" ViewCount="144" Body="&lt;p&gt;I'm wondering if e-commerce companies where products are offered by users, such as EBay, are using Object Recognition to ensure that an uploaded image corresponds to an specific type of object (clothing, shoes, glasses, etc) either to classify automatically or more importantly to filter undesired images (such as non related or even illegal types).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, which algorithms and/or open platforms could be use for doing so? From what I've looked it seems that HOG+Exemplar SVM might be one of the most accurate methods developed so far (&lt;a href=&quot;http://www.cs.cmu.edu/~efros/exemplarsvm-iccv11.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.cmu.edu/~efros/exemplarsvm-iccv11.pdf&lt;/a&gt;), even having couple of public repo's with Matlab implementations (&lt;a href=&quot;https://github.com/quantombone/exemplarsvm&quot; rel=&quot;nofollow&quot;&gt;https://github.com/quantombone/exemplarsvm&lt;/a&gt;), but I'm still wondering if this is being used in industry.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2015-03-03T12:21:48.467" Title="Object Recognition for classification, is it being used in industry?" Tags="|classification|object-recognition|hog|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3793" PostTypeId="2" ParentId="3788" CreationDate="2014-12-31T21:47:37.490" Score="1" Body="&lt;p&gt;Here's a simple solution using the R package &lt;code&gt;ggmap&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;start &amp;lt;- '95 clark st, new haven, ct'&#xA;end &amp;lt;- 'maison mathis, new haven, ct'&#xA;legs &amp;lt;- route(start,end, alternatives = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Will find routes from &lt;code&gt;start&lt;/code&gt; to &lt;code&gt;end&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      m    km     miles seconds   minutes       hours  startLon startLat    endLon   endLat leg route&#xA;1    60 0.060 0.0372840       7 0.1166667 0.001944444 -72.91617 41.31511 -72.91678 41.31541   1     A&#xA;2   718 0.718 0.4461652      95 1.5833333 0.026388889 -72.91678 41.31541 -72.92100 41.30979   2     A&#xA;3   436 0.436 0.2709304      64 1.0666667 0.017777778 -72.92100 41.30979 -72.92555 41.31171   3     A&#xA;4   431 0.431 0.2678234      68 1.1333333 0.018888889 -72.92555 41.31171 -72.92792 41.30829   4     A&#xA;5    60 0.060 0.0372840       7 0.1166667 0.001944444 -72.91617 41.31511 -72.91678 41.31541   1     B&#xA;6  1276 1.276 0.7929064     179 2.9833333 0.049722222 -72.91678 41.31541 -72.92430 41.30543   2     B&#xA;7   421 0.421 0.2616094      62 1.0333333 0.017222222 -72.92430 41.30543 -72.92869 41.30729   3     B&#xA;8   129 0.129 0.0801606      28 0.4666667 0.007777778 -72.92869 41.30729 -72.92792 41.30829   4     B&#xA;9    60 0.060 0.0372840       7 0.1166667 0.001944444 -72.91617 41.31511 -72.91678 41.31541   1     C&#xA;10  421 0.421 0.2616094      58 0.9666667 0.016111111 -72.91678 41.31541 -72.91924 41.31211   2     C&#xA;11  522 0.522 0.3243708     101 1.6833333 0.028055556 -72.91924 41.31211 -72.92502 41.31382   3     C&#xA;12  240 0.240 0.1491360      33 0.5500000 0.009166667 -72.92502 41.31382 -72.92555 41.31171   4     C&#xA;13  431 0.431 0.2678234      68 1.1333333 0.018888889 -72.92555 41.31171 -72.92792 41.30829   5     C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then we can find the length of time estimated to walk there with something like this. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(legs$minutes, legs$route, max)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="4724" LastActivityDate="2014-12-31T21:47:37.490" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3795" PostTypeId="1" CreationDate="2015-01-02T05:36:54.783" Score="0" ViewCount="553" Body="&lt;p&gt;This refers to a system described in a book by Nick Lawrence titled &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0975276107&quot; rel=&quot;nofollow&quot;&gt;Correlithm Object Technology&lt;/a&gt;&quot;. The author coined the term &quot;correlithm&quot; as a combination of &quot;correlation&quot; and &quot;algorithm&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Correlithm Objects (&quot;COs&quot;) described in the book are, in the author's view, &quot;primary data tokens&quot; in biological neural systems and are central to &quot;all high-level data representation, storage, and manipulation&quot;. In addition to this the author describes the COs as &quot;important mathematical objects of the statistics of bounded, high-dimensional spaces&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering these descriptions it seems like these COs could be useful for AI. What I was curious about is whether they are actually used anywhere in the industry, and if so, in what kind of situations?&lt;/p&gt;&#xA;" OwnerUserId="6625" LastActivityDate="2018-10-16T20:25:49.890" Title="Are Correlithm Objects used for anything in the industry?" Tags="|machine-learning|algorithms|correlation|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3796" PostTypeId="2" ParentId="3792" CreationDate="2015-01-02T10:58:38.187" Score="1" Body="&lt;p&gt;As far as I am aware, object recognition is not extensively used in industry yet. Google's image search, for instance, is based on exploiting the text on the web pages rather than the images themselves. I have seen several start-up companies that market prototypes based on object recognition, but these products are either prototypes or not widely used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding state-of-the-art techniques, you should take a look to recent literature on deep learning and deep neural networks applied to object recognition, along with more specific techniques like convolutional networks. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-01-02T10:58:38.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3797" PostTypeId="1" CreationDate="2015-01-02T20:03:59.950" Score="7" ViewCount="3403" Body="&lt;p&gt;I am reading up about lambda architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we have our entire data set. in our system. very good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how will you merge these?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any guidance.&lt;/p&gt;&#xA;" OwnerUserId="6644" LastActivityDate="2015-02-03T19:53:47.233" Title="Lambda Architecture - How to implement the Merge Layer / Query Layer" Tags="|bigdata|apache-hadoop|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3798" PostTypeId="2" ParentId="2346" CreationDate="2015-01-03T01:07:15.080" Score="5" Body="&lt;p&gt;Try exploring the rich field of &quot;Anomaly Detection in Time Series&quot;. Control charts and CUSUMs (or cumulative sum control charts) might help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple &lt;a href=&quot;http://en.wikipedia.org/wiki/Bullet_graph&quot; rel=&quot;nofollow&quot;&gt;Bullet Graphs&lt;/a&gt; might be all you need. Based on historical data and domain knowledge, define normal variance. Then make it clear to stakeholders when the current value is outside of predefined ranges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stephen Few is an expert in business dashboards. Any of his books will help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are open to R, try &lt;a href=&quot;http://shiny.rstudio.com/&quot; rel=&quot;nofollow&quot;&gt;Shiny&lt;/a&gt; to create simple interactive web applications (It is very straightforward). There is also an open source package for &lt;a href=&quot;https://github.com/twitter/AnomalyDetection&quot; rel=&quot;nofollow&quot;&gt;anomaly detection&lt;/a&gt;, both local and global.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create quick prototypes and get feedback!&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2015-01-08T03:12:03.630" LastActivityDate="2015-01-08T03:12:03.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3799" PostTypeId="2" ParentId="1174" CreationDate="2015-01-03T01:17:47.957" Score="4" Body="&lt;p&gt;I have shared a number of resources on &lt;em&gt;time series classification and clustering&lt;/em&gt; in one of my recent answers here on &lt;em&gt;Data Science StackExchange&lt;/em&gt;: &lt;a href=&quot;https://datascience.stackexchange.com/a/3764/2452&quot;&gt;https://datascience.stackexchange.com/a/3764/2452&lt;/a&gt;. Hopefully, you will find them relevant to this question and useful.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-01-03T01:17:47.957" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3800" PostTypeId="2" ParentId="3770" CreationDate="2015-01-03T03:15:08.960" Score="2" Body="&lt;p&gt;This won't be a very satisfying answer, but here's my take...&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Same answer for both: you can't do this for unknown properties, and for known properties it will depend on how the values were computed.&lt;/p&gt;&#xA;&lt;p&gt;As you alluded to:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;(an average of fiancial rates would be a non-sense for example)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;There is no single transformation that will be appropriate in all cases, whether the properties/values are known or unknown. Even with known properties, you'll likely need a unique transformation for each type: mean, median, mode, min, max, boolean, etc.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Whenever possible, try to preserve the full granularity of the smallest possible step. Assuming you know how to transform the values, you can always roll-up the steps (e.g., day to month, month to year)... but you won't necessarily be able to reconstruct smaller steps from larger ones following a lossy conversion.&lt;/p&gt;&#xA;" OwnerUserId="819" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-01-03T03:15:08.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3801" PostTypeId="2" ParentId="3770" CreationDate="2015-01-03T07:58:10.663" Score="10" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For your timeseries analysis you should do both: get to the highest granularity possible with the daily dataset, and also repeat the analysis with the monthly dataset. With the monthly dataset you have 120 data points, which is sufficient to get a timeseries model even with seasonality in your data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To obtain say weekly or monthly data from daily data, you can use smoothing functions. For financial data, you can use moving average or exponential smoothing, but if those do not work for your data, then you can use the spline smoothing function &quot;smooth.spline&quot; in R: &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html&quot; rel=&quot;noreferrer&quot;&gt;https://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model returned will have less noise than the original daily dataset, and you can get values for the desired time points. Finally, these data points can be used in your timeseries analysis. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To obtain daily data when you have monthly or weekly data, you can use interpolation. First, you should find an equation to describe the data. In order to do this you should plot the data (e.g. price over time). When factors are known to you, this equation should be influenced by those factors. When factors are unknown, you can use a best fit equation. The simplest would be a linear function or piecewise linear function, but for financial data this won't work well. In that case, you should consider piecewise cubic spline interpolation. This link goes into more detail on possible interpolation functions: &lt;a href=&quot;http://people.math.gatech.edu/~meyer/MA6635/chap2.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://people.math.gatech.edu/~meyer/MA6635/chap2.pdf&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, there is a method for doing interpolation of timeseries data. Here you would create a vector with say weekly values and NAs in the gaps for the daily values, and then use the &quot;interpNA&quot; function to get the interpolated values for the NAs. However, this function uses the &quot;approx&quot; function to get the interpolated values, which applies either a linear or constant interpolation. To perform cubic spline interpolation in R, you should use the &quot;splinefun&quot; function instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something to be aware of is that timeseries models typically do some sort of averaging to forecast future values whether you are looking at exponential smoothing or Auto-Regressive Integrated Moving Average (ARIMA) methods amongst others. So a timeseries model to forecast daily values may not be the best choice, but the weekly or monthly models may be better. &lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2015-01-03T07:58:10.663" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="3802" PostTypeId="2" ParentId="3770" CreationDate="2015-01-03T08:48:51.070" Score="5" Body="&lt;p&gt;I'm not an expert in this area, but I believe that your question is concerned with &lt;em&gt;time series aggregation and disaggregation&lt;/em&gt;. If that is the case, here are some hopefully relevant resources, which might be helpful in solving your problem (first five items are main, but representative, and last two are supplementary):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www4.ncsu.edu/~jjseater/tempaggecontimeseries.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Temporal Aggregation and Economic Time Series&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://journal.r-project.org/archive/2013-2/sax-steiner.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Temporal Disaggregation of Time Series&lt;/a&gt; (IMHO, an excellent overview paper)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views/TimeSeries.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRAN Task View: Time Series Analysis&lt;/a&gt; (R-focused)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://people.su.se/~lundh/reproduce/introduction_ts.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to R's Time Series Facilities&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://faculty.washington.edu/ezivot/econ424/Working%20with%20Time%20Series%20Data%20in%20R.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Working with Financial Time Series Data in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scausa.com/tsfbook.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;Notes on chapters contents for the book &quot;Time Series Analysis and Forecasting&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/121599/31372&quot;&gt;Discussion on Cross Validated&lt;/a&gt; on daily to monthly data conversion (Python-focused)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-01-03T09:00:28.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3804" PostTypeId="1" CreationDate="2015-01-03T22:38:39.097" Score="2" ViewCount="3679" Body="&lt;p&gt;For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a better format? Or does it not matter?&lt;/p&gt;&#xA;" OwnerUserId="6660" LastEditorUserId="1367" LastEditDate="2015-01-08T23:25:32.430" LastActivityDate="2017-07-18T08:40:39.433" Title="Format for storing textual data" Tags="|text-mining|crawling|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="3806" PostTypeId="2" ParentId="2346" CreationDate="2015-01-04T05:22:48.670" Score="3" Body="&lt;p&gt;The &lt;em&gt;optimal&lt;/em&gt; solution very much depends on multiple factors, including your (current and future) business and IT processes, stakeholders' needs and preferences, and, in general, business and IT architectures. So, IMHO it's difficult to answer this wide and not well-defined question. Having said that, I hope that you will find the following earlier &lt;strong&gt;answers&lt;/strong&gt; of mine here on &lt;em&gt;Data Science StackExchange&lt;/em&gt; relevant and useful.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;On &lt;strong&gt;data analysis workflow&lt;/strong&gt; (introductory): &lt;a href=&quot;https://datascience.stackexchange.com/a/1006/2452&quot;&gt;https://datascience.stackexchange.com/a/1006/2452&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;On &lt;strong&gt;data analysis workflow&lt;/strong&gt; (more detailed): &lt;a href=&quot;https://datascience.stackexchange.com/a/759/2452&quot;&gt;https://datascience.stackexchange.com/a/759/2452&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;On &lt;strong&gt;dashboard visualization&lt;/strong&gt;: &lt;a href=&quot;https://datascience.stackexchange.com/a/907/2452&quot;&gt;https://datascience.stackexchange.com/a/907/2452&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;On &lt;strong&gt;big data visualization&lt;/strong&gt;: &lt;a href=&quot;https://datascience.stackexchange.com/a/3723/2452&quot;&gt;https://datascience.stackexchange.com/a/3723/2452&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-01-04T05:22:48.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3807" PostTypeId="1" CreationDate="2015-01-04T10:59:42.973" Score="1" ViewCount="233" Body="&lt;p&gt;Im doing my academic project. im having the base paper for reference  the paper is IEEE paper &quot;effective and efficient clustering methods for correlated probabilistic graph&quot;. i wish to do this in R tool. in this paper two algorithm are implemented. i like to implement the peedr algorithm in the paper. how can i give the input for that algorithm.? suggest the packgages in R tool&#xA; the paper can be found here&lt;br&gt;&#xA;&lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6570474&quot; rel=&quot;nofollow&quot;&gt;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6570474&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6664" LastEditorUserId="8953" LastEditDate="2015-04-09T01:03:45.010" LastActivityDate="2015-05-09T01:45:07.583" Title="How i can generate the probabilistic graph for my dataset?" Tags="|r|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3808" PostTypeId="1" CreationDate="2015-01-04T11:22:28.817" Score="1" ViewCount="124" Body="&lt;p&gt;I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both).&#xA;Since in bayesian regularization I have not the validation set, how can I compare the generalization capability of the networks obtained with the two methodologies? &#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="6559" LastActivityDate="2015-01-05T22:33:02.010" Title="Compare Neural Network generalization results" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="3810" PostTypeId="2" ParentId="3807" CreationDate="2015-01-04T19:29:48.930" Score="2" Body="&lt;p&gt;Søren Højsgaard has many quality resources for graphical models in R. He has a tutorial &lt;a href=&quot;http://people.math.aau.dk/~sorenh/misc/2012-Oslo-GMwR/GMwR-notes.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Graphical Models with R&quot;&lt;/a&gt; and a &lt;a href=&quot;http://cran.r-project.org/web/views/gR.html&quot; rel=&quot;nofollow&quot;&gt;list of CRAN packages&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, &lt;a href=&quot;http://cran.r-project.org/web/packages/mclust/index.html&quot; rel=&quot;nofollow&quot;&gt;mclust&lt;/a&gt; is one of the best clustering packages in R.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2015-01-04T19:29:48.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3811" PostTypeId="1" AcceptedAnswerId="3818" CreationDate="2015-01-04T21:52:38.977" Score="1" ViewCount="98" Body="&lt;p&gt;I'm building a neural network to analyze a business' sales. I'm normalizing all input values to the range &lt;code&gt;{0,1}&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm struggling with the day of the week column. Business days are identified by a number ranging &lt;code&gt;{1-5}&lt;/code&gt; (1=Monday). Normalizing these values to the range &lt;code&gt;{0,1}&lt;/code&gt; is straightforward, but results in a major bias in the final output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is that the full range of normalized values for the business day column is explored with every week worth of data, whereas other price-related column explore their full range of normalized values infrequently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The business day column ends up being the largest contributor to the final output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I normalize it to make its contribution more in tune with the rest of the inputs?&lt;/p&gt;&#xA;" OwnerUserId="6669" LastActivityDate="2015-01-05T23:09:25.800" Title="normalize identification values properly" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3812" PostTypeId="2" ParentId="2578" CreationDate="2015-01-04T22:29:14.980" Score="0" Body="&lt;p&gt;Don't forget about MLlib (the hot kid on the block), cloudera-ml, cloudera oryx, cloudera oryx2, and a myriad of other efforts to make old algorithms like k-means solve nonexistant problems on wannabe big data... &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;There is no one size fits all.&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Thus, there is no answer to your question, in particular no concise answer. &lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-01-04T22:29:14.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3813" PostTypeId="2" ParentId="3804" CreationDate="2015-01-04T23:57:20.150" Score="1" Body="&lt;p&gt;In general, use a storage method that allows you to quickly query it. If your collection is huge, you might need something Lucene-based, like ElasticSearch. If you are a SQL crack and your favorite DB supports it, a full-text index might do the trick. For small sizes like the 5000 documents, even Linux' LocateDb+grep or OSX' spotlight could be enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The important point is to be able to quickly verify assumptions about the content of your data - how many documents contain X and Y, does any document contain W but not V, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be useful at both the whole set level as well as for analyzing your topic clusters. Finally, a few GNU tools or SQL mastery can also help you profile your document sets more efficiently (n-gram counts/ranks, collocations, concordances, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: that means, for the above reasons and given your collection size, good old plain text (in a file system or a database) might be more efficient than any &quot;fancy&quot; format.&lt;/p&gt;&#xA;" OwnerUserId="6672" LastActivityDate="2015-01-04T23:57:20.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3814" PostTypeId="1" CreationDate="2015-01-05T18:34:55.920" Score="1" ViewCount="84" Body="&lt;p&gt;It is often pointed out that &lt;em&gt;sample&lt;/em&gt; is an overloaded term in statistics and the sciences being supported by statistics.  In my field (geological sciences) as in most other sciences, the process of collecting meaningful data is critical and discussions about the traps and pitfalls in that process talk about &lt;em&gt;sampling&lt;/em&gt;.  Not far down the road from that, particularly when lab results are back, conversations involving statisticians, data scientists, geomathematicians, GIS analysts and even &lt;em&gt;normal&lt;/em&gt; geologists are likely to attempt to include multiple meanings of &lt;em&gt;sample&lt;/em&gt; in the same sentence!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q: Have any data scientists (or statisticians) found practical ways to communicate these different meanings?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way is to always add soil, rock, statistical and so on before &lt;em&gt;sample&lt;/em&gt;.  But I was curious if there are any other approaches to effective communication that are in use.&lt;/p&gt;&#xA;" OwnerUserId="6646" LastActivityDate="2015-04-05T23:59:08.727" Title="Communicating clearly about &quot;samples&quot;" Tags="|statistics|sampling|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="3815" PostTypeId="2" ParentId="3814" CreationDate="2015-01-05T22:16:06.093" Score="1" Body="&lt;p&gt;&quot;Sample&quot; as a noun usually refers to a single data point.  &quot;Sample&quot; as a verb is the act of extracting a subset of data points from some larger body (reality or a larger dataset).  The only way to be less ambiguous is use more specific words than &quot;data&quot; or &quot;sample&quot;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Say you collect 1MM data points from four different sensors in the field giving you four sets of 250k data points.  Say this data is too big for some demo of a model you're testing or an analysis you're running, so you pick 100k data points evenly split across the four sensors (giving four sets of 25k data points).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, you're sampling twice.  First, to gather your 1MM data points sampling from reality.  Second, you sample again to decrease the size of your data set to something more manageable.  'Data' or 'sample' could refer to reality, the 1MM dataset, the 100k dataset, or any of the sensor-specific subsets.  To make it less ambiguous, establish a unique name as soon as possible for each possible definition you'll be working with.  ('reality' for the set of all possibly observed samples.  'the complete dataset', something derived from the source of the dataset, or even X for the full 1MM dataset.  'our trial dataset', or even Y for the small 100k dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you actually do comes down to context and what's appropriate for your intended audience, but the general answer is to use more specific words.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2015-01-05T22:16:06.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3816" PostTypeId="2" ParentId="3808" CreationDate="2015-01-05T22:33:02.010" Score="2" Body="&lt;p&gt;As Neil said in the comments, split out a &lt;a href=&quot;http://en.wikipedia.org/wiki/Test_set&quot; rel=&quot;nofollow&quot;&gt;test set&lt;/a&gt; (data you don't use in training either model) and see how each trained model performs on the test set.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2015-01-05T22:33:02.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3817" PostTypeId="2" ParentId="3770" CreationDate="2015-01-05T22:49:27.293" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Aggregation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you have the number of time people searched for 'widgets' every day.  Add up the daily totals for a month to get monthly totals.  I would need to see more specifics about the actual data collected at each granularity to give you a more complete version.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can't.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In physics, a comparable idea is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Nyquist_frequency&quot; rel=&quot;nofollow&quot;&gt;Nyquist frequency&lt;/a&gt;.  The general idea is that you can't add more information than what you already have present in your data without bringing in more data.  Given only the day someone ran a query, how can you tell what time of day that query was ran?  You may be able to make some inferences, but the only way to answer the question is to directly or indirectly bring in more information to the system.  There are things you can do to make informed guesses at the daily state of monthly variables (as &#xA;gchaks mentioned, interpolation), but your data is still fundamentally monthly data stretched to look daily.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That totally depends on what you're trying to answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The smaller granularity will be more sensitive to noise and other anomalies.  The lager granularity will be able to answer questions more confidently, but loose some of it's usefulness.  For example, if you're trying to see when people start looking up venues to weekend plans to know when to launch marketing campaigns for a new night club, you'll want to be looking at daily data, if not smaller.  If you're looking at the general trending of night clubs to figure out who you want to invest in, then monthly would probably be better.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2015-01-05T22:49:27.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3818" PostTypeId="2" ParentId="3811" CreationDate="2015-01-05T23:09:25.800" Score="1" Body="&lt;p&gt;It is possible that the other variables you're feeding into the NN are simply bad at predicting sales.  Sell prediction is a notoriously hard problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically the addressing of mapping a multi-state categorical variable to the NN's {0,1} input range:  Another idea is to change that one, 5-state variable into five boolean variables.  Rather than {0,0.25,0.5,0.75,1.0} on your one variable, make each of the five boolean variables represent a single day and make [1,0,0,0,0] equal Monday, [0,1,0,0,0] equal Tuesday, etc.  I've personally had more success both with training good networks and introspecting the network itself when spreading out states of classes like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other hacks you can try:&lt;br&gt;&#xA; * Take out the the 'day' column all together and see if any of the other variables get used.&lt;br&gt;&#xA; * Plot the distribution of spend as a function of day.  Even if nothing else comes of this current model, it sounds like you've found one interesting insight already.&lt;br&gt;&#xA; * Consider also trying different models.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2015-01-05T23:09:25.800" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="3819" PostTypeId="2" ParentId="3770" CreationDate="2015-01-06T01:00:54.457" Score="2" Body="&lt;p&gt;Being able to aggregate data based on date segments is a piece of cake using Tableau software. You would simply plug your data into the tool, and then you can drag and drop both the metric and date dimensions onto a report body space. Tableau will instantaneously whip up aggregate visualizations and/or detail data tables, on the fly. You can group/sum by year, quarter, month, week, day, hour, etc. (standard, built in, out of the box functionality offered by the tool)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you wanted to incorporate additional data later on (which I assume is what you meant by 'unknown properties'), you can import another data set and easily append it onto the first one, as long as it also offers dates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend checking out the free version, which I believe is called Tableau Public. &lt;/p&gt;&#xA;" OwnerUserId="6682" LastEditorUserId="6682" LastEditDate="2015-01-06T01:06:26.827" LastActivityDate="2015-01-06T01:06:26.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3820" PostTypeId="2" ParentId="1007" CreationDate="2015-01-06T01:26:41.587" Score="0" Body="&lt;p&gt;You don't need a tool and I don't recommend you use one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convert the html to well-formed XML (XHTML) - I recommend the tagsoup.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've done that the data is just another XML feed and you can write an XSLT transformation (or XQuery) to access and pull out the data you want in the format you want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That might mean learning  XSLT/XQuery if you don't already know it but you will be learning skills that (unlike scraping tools) have multiple rather than just than one useful application.&lt;/p&gt;&#xA;" OwnerUserId="498" LastActivityDate="2015-01-06T01:26:41.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3821" PostTypeId="1" AcceptedAnswerId="3823" CreationDate="2015-01-06T04:34:09.967" Score="7" ViewCount="1773" Body="&lt;p&gt;I am interested in finding a statistic that tracks the unpredictability of a time series. For simplicity sake, assume that each value in the time series is either 1 or 0. So for example, the following two time series are entirely predictable&#xA;TS1: 1 1 1 1 1 1 1 1&#xA;TS2: 0 1 0 1 0 1 0 1 0 1 0 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the following time series is not that predictable:&#xA;TS3: 1 1 0 1 0 0 1 0 0 0 0 0 1 1 0 1 1 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a statistic that given a time series, would return a number between 0 and 1 with 0 indicating that the series is completely predictable and 1 indicating the series in completely unpredictable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked at some entropy measures like Kolmogorov Complexity and Shannon entropy, but neither seem to fit my requirement. In Kolmogorov complexity, the statistic value changes depending on the length of the time series (as in &quot;1 0 1 0 1&quot; and &quot;1 0 1 0&quot; have different complexities, so its not possible to compare predictability of two time series with differing number of observations). In Shannon entropy, the order of observations didn't seem to matter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers on what would be a good statistic for my requirement?&lt;/p&gt;&#xA;" OwnerUserId="6685" LastActivityDate="2015-01-06T07:40:12.223" Title="Finding unpredictability or uncertainty in a time series" Tags="|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="3823" PostTypeId="2" ParentId="3821" CreationDate="2015-01-06T07:40:12.223" Score="7" Body="&lt;p&gt;Since you have looked at Kolmogorov-Smirnov and Shannon entropy measures, I would like to suggest some other hopefully relevant options. First of all, you could take a look at the so-called &lt;a href=&quot;http://en.wikipedia.org/wiki/Approximate_entropy&quot;&gt;approximate entropy $ApEn$&lt;/a&gt;. Other potential statistics include &lt;em&gt;block entropy&lt;/em&gt;, &lt;em&gt;T-complexity&lt;/em&gt; (&lt;em&gt;T-entropy&lt;/em&gt;) as well as &lt;em&gt;Tsallis entropy&lt;/em&gt;: &lt;a href=&quot;http://members.noa.gr/anastasi/papers/B29.pdf&quot;&gt;http://members.noa.gr/anastasi/papers/B29.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the above-mentioned potential measures, I would like to suggest to have a look at available statistics in &lt;em&gt;Bayesian inference-based&lt;/em&gt; model of &lt;em&gt;stochastic volatility&lt;/em&gt; in time series, implemented in &lt;code&gt;R&lt;/code&gt; package &lt;code&gt;stochvol&lt;/code&gt;: &lt;a href=&quot;http://cran.r-project.org/web/packages/stochvol&quot;&gt;http://cran.r-project.org/web/packages/stochvol&lt;/a&gt; (see detailed &lt;a href=&quot;http://cran.r-project.org/web/packages/stochvol/vignettes/article.pdf&quot;&gt;vignette&lt;/a&gt;). Such statistics of uncertainty include &lt;em&gt;overall level of volatility&lt;/em&gt; $\mu$, &lt;em&gt;persistence&lt;/em&gt; $\phi$ and &lt;em&gt;volatility of volatility&lt;/em&gt; $\sigma$: &lt;a href=&quot;http://simpsonm.public.iastate.edu/BlogPosts/btcvol/KastnerFruwhirthSchnatterASISstochvol.pdf&quot;&gt;http://simpsonm.public.iastate.edu/BlogPosts/btcvol/KastnerFruwhirthSchnatterASISstochvol.pdf&lt;/a&gt;. A &lt;strong&gt;comprehensive&lt;/strong&gt; &lt;em&gt;example&lt;/em&gt; of using stochastic volatility model approach and &lt;code&gt;stochvol&lt;/code&gt; package can be found in the excellent blog post &lt;a href=&quot;http://www.themattsimpson.com/2014/02/22/exactly-how-volatile-is-bitcoin&quot;&gt;&quot;Exactly how volatile is bitcoin?&quot;&lt;/a&gt; by Matt Simpson.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-01-06T07:40:12.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3825" PostTypeId="1" CreationDate="2015-01-06T14:08:35.533" Score="1" ViewCount="79" Body="&lt;p&gt;I have a data set, in excel format, with &lt;strong&gt;account names, reported symptoms, a determined root cause and a date in month year format&lt;/strong&gt; for each row. I am trying to implement a mahout like system with a purpose of determining the likelihood symptoms an account can report by doing a user based similarity kind of a thing. Technically, I am just hoping to tweak the recommendation system into a deterministic system to spot out the probable symptoms an account can report on. Instead of ratings, I can get the frequency of symptoms by accounts. Is it possible to use a programming language or any other software to build such system?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Account : &lt;em&gt;X&lt;/em&gt; Symptoms : &lt;em&gt;AB, AD, AB, AB&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Account : &lt;em&gt;Y&lt;/em&gt;  Symptoms : &lt;em&gt;AE, AE, AB, AB, EA&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the sake of this example, let's assume that all the dates are this month. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;O/P: Account : X Symptom: AE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here both of them have reported &lt;em&gt;AB&lt;/em&gt; 2 or more times. I could fix such number as a threshold to look for probable symptoms. &lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2015-11-20T01:08:37.500" Title="Mimic a Mahout like system" Tags="|recommender-system|similarity|apache-mahout|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="3826" PostTypeId="1" AcceptedAnswerId="3827" CreationDate="2015-01-06T20:43:35.480" Score="2" ViewCount="231" Body="&lt;p&gt;I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to consider a variable eligible to be an input for my Neural Network?&lt;/p&gt;&#xA;" OwnerUserId="6559" LastActivityDate="2015-01-07T01:26:17.560" Title="Correlation threshold for Neural Network features selection" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="3827" PostTypeId="2" ParentId="3826" CreationDate="2015-01-07T01:26:17.560" Score="2" Body="&lt;p&gt;Given non-linearity of neural networks, I believe correlation analysis isn't a good way to estimate importance of variables. For example, imagine that you have 2 input variables - &lt;code&gt;x1&lt;/code&gt; and &lt;code&gt;x2&lt;/code&gt; - and following conditions hold: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;cor(x2, y) = 1&lt;/code&gt; if &lt;code&gt;x1 = 1&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;cor(x2, y) = 0&lt;/code&gt; otherwise&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;x1 = 1&lt;/code&gt; in 10% of cases&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That is, &lt;code&gt;x2&lt;/code&gt; is a very good predictor for &lt;code&gt;y&lt;/code&gt;, but only given that &lt;code&gt;x1 = 1&lt;/code&gt;, which is the case only in 10% of data. Taking into account correlations of &lt;code&gt;x1&lt;/code&gt; and &lt;code&gt;x2&lt;/code&gt; separately won't expose this dependency, and you will most likely drop out both variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other ways to perform feature selection, however. Simplest one is to train your model with all possible sets of variables and check the best subset. This is pretty inefficient with many variables, though, so many ways to improve it exist. For a good introduction in best subset selection see chapter 6.1 of &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow&quot;&gt;Introduction to Statistical Learning&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2015-01-07T01:26:17.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4826" PostTypeId="2" ParentId="3804" CreationDate="2015-01-07T07:39:04.027" Score="1" Body="&lt;p&gt;Suppose that a forum post has, on average, 2000 characters, which is more or less the equivalent of a page of text, than the total memory needed is 10MB if text is ASCII. Even if the text is Unicode encoded in an Asian language it will take 40MB. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is far too little for modern computers, so a simple text format is the best since it can be parsed in the fastest way, and loaded into RAM all at once.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-01-07T07:39:04.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4827" PostTypeId="1" CreationDate="2015-01-07T16:50:18.803" Score="1" ViewCount="279" Body="&lt;p&gt;Is there any way to use package &lt;code&gt;dplyr&lt;/code&gt; on RStudio having R base 3.0.2 ?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not interested in &lt;code&gt;plyr&lt;/code&gt; package.&lt;/p&gt;&#xA;" OwnerUserId="7702" LastEditorUserId="134" LastEditDate="2018-12-28T00:50:36.973" LastActivityDate="2018-12-28T00:50:36.973" Title="Can we use package &quot;dplyr&quot; on R base 3.0.2?" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="4828" PostTypeId="2" ParentId="4827" CreationDate="2015-01-07T22:03:37.357" Score="5" Body="&lt;p&gt;dplyr 0.3 requires R 3.1+. If you're stuck on R 3.0.x, you have to use dplyr 0.2.x. &lt;/p&gt;&#xA;" OwnerUserId="404" LastActivityDate="2015-01-07T22:03:37.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4829" PostTypeId="2" ParentId="713" CreationDate="2015-01-08T00:25:05.277" Score="1" Body="&lt;p&gt;We faced this problem and analysed the issue.&#xA;Cloudera, Hue or Hive didn't have any error reported but the users via Beeswax were unable to run queries.&#xA;It was good to analyze below:&#xA;Duplicate PID for Hive (check for all hive servers. If you do have a tool to check the duplicate PID then use that or use the kinit to login to cloudera admin node and analyze it.)&#xA;We killed the duplicate PID&#xA;AND&#xA;restarted hive server. It fixed the issue.&lt;/p&gt;&#xA;" OwnerUserId="7707" LastActivityDate="2015-01-08T00:25:05.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4830" PostTypeId="2" ParentId="3804" CreationDate="2015-01-08T07:55:31.067" Score="0" Body="&lt;p&gt;I think one has to be very careful when storing textual data. If they are user comments then, for security concerns it's better if it is encoded in some format before storage.&#xA;A protobuf object can then be defined to resolve the encoding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the query pattern, and accepted latency in retrieval of the data, DB should be decided. Just a recommendation, if the idea is to store comments over a period of time for each user, consider HBase or Cassandra. They are optimized for time range queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Recommend read:&lt;/strong&gt; &lt;a href=&quot;http://info.mapr.com/rs/mapr/images/Time_Series_Databases.pdf&quot; rel=&quot;nofollow&quot;&gt;http://info.mapr.com/rs/mapr/images/Time_Series_Databases.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5179" LastActivityDate="2015-01-08T07:55:31.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4831" PostTypeId="1" AcceptedAnswerId="4833" CreationDate="2015-01-08T10:09:32.010" Score="9" ViewCount="4298" Body="&lt;p&gt;Could anyone recommend a good similarity measure for objects which have multiple classes, where each class is part of a hierarchy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say the classes look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 Produce&#xA;  1.1 Eggs&#xA;    1.1.1 Duck eggs&#xA;    1.1.2 Chicken eggs&#xA;  1.2 Milk&#xA;    1.2.1 Cow milk&#xA;    1.2.2 Goat milk&#xA;2 Baked goods&#xA;  2.1 Cakes&#xA;    2.1.1 Cheesecake&#xA;    2.1.2 Chocolate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;An object might be tagged with items from the above at any level, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Omelette: eggs, milk (1.1, 1.2)&#xA;Duck egg omelette: duck eggs, milk (1.1.1, 1.2)&#xA;Goat milk chocolate cheesecake: goat milk, cheesecake, chocolate (1.2.2, 2.1.1, 2.1.2)&#xA;Beef: produce (1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the classes weren't part of a hierarchy, I'd probably I'd look at cosine similarity (or equivalent) between classes assigned to an object, but I'd like to use the fact that different classes with the same parents also have some similarity value (e.g. in the example above, beef has some small similarity to omelette, since they both have items from the class '1 produce').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it helps, the hierarchy has ~200k classes, with a maximum depth of 5.&lt;/p&gt;&#xA;" OwnerUserId="474" LastActivityDate="2015-01-08T12:08:43.113" Title="Similarity measure based on multiple classes from a hierarchical taxonomy?" Tags="|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4833" PostTypeId="2" ParentId="4831" CreationDate="2015-01-08T12:08:43.113" Score="11" Body="&lt;p&gt;While I don't have enough expertise to advise you on selection of the best &lt;em&gt;similarity measure&lt;/em&gt;, I've seen a number of them in various papers. The following &lt;strong&gt;collection&lt;/strong&gt; of research papers hopefully will be useful to you in determining the optimal measure for your research. Please note that I intentionally included papers, using both &lt;em&gt;frequentist&lt;/em&gt; and &lt;em&gt;Bayesian&lt;/em&gt; approaches to hierarchical classification, including class information, for the sake of more &lt;em&gt;comprehensive&lt;/em&gt; coverage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Frequentist approach:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/pdf/cmp-lg/9709008.pdf&quot;&gt;Semantic similarity based on corpus statistics and lexical taxonomy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0048733313000115&quot;&gt;Can’t see the forest for the leaves: Similarity and distance measures for hierarchical taxonomies with a patent classification example&lt;/a&gt; (also see &lt;a href=&quot;http://mcnameephd.pbworks.com/w/page/11389167/Research%20Policy%20-%20Similarity%20and%20distance%20measures%20for%20hierarchical%20taxonomies&quot;&gt;additional results and data&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/192155/hier_embd.pdf&quot;&gt;Learning hierarchical similarity metrics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/ftp/arxiv/papers/1211/1211.4709.pdf&quot;&gt;A new similarity measure for taxonomy based on edge counting&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.siam.org/meetings/sdm01/pdf/sdm01_22.pdf&quot;&gt;Hierarchical classification of real life documents&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://users.cis.fiu.edu/~taoli/pub/Li-jiis.pdf&quot;&gt;Hierarchical document classification using automatically generated hierarchy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ucla.edu/~weiwang/paper/SSDBM09.pdf&quot;&gt;Split-Order distance for clustering and classification hierarchies&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ccis2k.org/iajit/PDF/vol.8,no.3/791.pdf&quot;&gt;A hierarchical k-NN classifier for textual data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bayesian approach:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ba.stat.cmu.edu/journal/2007/vol02/issue01/shahbaba.pdf&quot;&gt;Improving classification when a class hierarchy is available using a hierarchy-based prior&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://gfx.cs.princeton.edu/pubs/DeCoro_2007_BAF/bayesgenre.pdf&quot;&gt;Bayesian aggregation for hierarchical genre classification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://oro.open.ac.uk/12991/1/Hierarchical_Classification.pdf&quot;&gt;Hierarchical classification for multiple, distributed web databases&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-01-08T12:08:43.113" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4834" PostTypeId="1" AcceptedAnswerId="4835" CreationDate="2015-01-08T13:44:20.160" Score="1" ViewCount="36" Body="&lt;p&gt;I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/normalization are not that import to my question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases measurement A (the starting measurement) is a count with values greater than 1000.  We expect an increase for measurement B for positive effects and a decrease in measurement B for negative effects.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the issue, some of our starting counts are nearly zero which we believe is an artifact of experimental preparation.  This of course leads to some really high ratios/scaling issues for these data points.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to adjust these values in order to better understand the real role in our experiment?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One suggestion we received was to add 1000 to all counts (from measurement A and B) to scale the values and remove the bias of such a low starting count,  is this a viable option?  Thank you in advance for your assistance, let me know if I am not being clear enough.&lt;/p&gt;&#xA;" OwnerUserId="7713" LastEditorUserId="1367" LastEditDate="2015-01-08T23:25:55.630" LastActivityDate="2015-01-08T23:25:55.630" Title="Correcting Datasets with artificially low starting values" Tags="|statistics|data-cleaning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="4835" PostTypeId="2" ParentId="4834" CreationDate="2015-01-08T14:22:33.347" Score="1" Body="&lt;p&gt;Yes, the general idea is to add a baseline small count to every category. The technical term for this is &lt;a href=&quot;http://en.wikipedia.org/wiki/Additive_smoothing&quot; rel=&quot;nofollow&quot;&gt;Laplace smoothing&lt;/a&gt;. Really it's not so much of a hack, as encoding the idea that you think there is some (uniform?) prior distribution of the events occurring.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="1367" LastEditDate="2015-01-08T23:25:48.103" LastActivityDate="2015-01-08T23:25:48.103" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4836" PostTypeId="1" CreationDate="2015-01-08T15:47:34.103" Score="9" ViewCount="10753" Body="&lt;p&gt;I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with some real examples.&lt;/p&gt;&#xA;" OwnerUserId="7715" LastActivityDate="2018-11-30T15:24:04.310" Title="Tutorials on topic models and LDA" Tags="|topic-model|lda|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4837" PostTypeId="1" CreationDate="2015-01-08T16:13:36.613" Score="0" ViewCount="476" Body="&lt;p&gt;(Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One chapter of my course is about cost function. I have been trying to find any example calculation of it with numbers. Googling only finds the same formula everytime, and also on Octave. But I want to do the same thing first with pen+paper and without it, I cannot understand. Please give me a very simple example of using the formula with numbers. Thanks a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I require a cost function calculation example for following sample dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Rooms = Rent&#xA;1 = 4000&#xA;2 = 10000&#xA;3 = 22000&#xA;4 = 30000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7712" LastEditorUserId="1367" LastEditDate="2015-01-10T11:11:10.110" LastActivityDate="2015-01-10T11:11:10.110" Title="Learning cost function for linear regression" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4838" PostTypeId="2" ParentId="4836" CreationDate="2015-01-08T20:58:44.547" Score="8" Body="&lt;p&gt;I highly recommend this tutorial: &lt;strong&gt;&lt;a href=&quot;http://programminghistorian.org/lessons/topic-modeling-and-mallet&quot;&gt;Getting Started with Topic Modeling and MALLET&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some additional links to help you get started...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good introductory materials (including links to research papers): &lt;a href=&quot;http://www.cs.princeton.edu/~blei/topicmodeling.html&quot;&gt;http://www.cs.princeton.edu/~blei/topicmodeling.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Software:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;MALLET (Java): &lt;a href=&quot;http://mallet.cs.umass.edu/topics.php&quot;&gt;http://mallet.cs.umass.edu/topics.php&lt;/a&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;topic modeling developer's guide: &lt;a href=&quot;http://mallet.cs.umass.edu/topics-devel.php&quot;&gt;http://mallet.cs.umass.edu/topics-devel.php&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;gensim (Python): &lt;a href=&quot;http://radimrehurek.com/gensim/&quot;&gt;http://radimrehurek.com/gensim/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;topicmodels (R): &lt;a href=&quot;http://cran.r-project.org/web/packages/topicmodels/index.html&quot;&gt;http://cran.r-project.org/web/packages/topicmodels/index.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Stanford Topic Modeling Toolbox (designed for use by social scientists): &lt;a href=&quot;http://www-nlp.stanford.edu/software/tmt/tmt-0.4/&quot;&gt;http://www-nlp.stanford.edu/software/tmt/tmt-0.4/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Mr.LDA (scalable topic modeling using MapReduce): &lt;a href=&quot;http://lintool.github.io/Mr.LDA/&quot;&gt;http://lintool.github.io/Mr.LDA/&lt;/a&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you're working with &lt;em&gt;massive&lt;/em&gt; amounts of input text, you might want to consider using Mr.LDA to build your topics models -- its MapReduce-based approach might be more efficient when working with lots of data.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Even more here on the Biased Estimates blog: &lt;a href=&quot;http://www.biasedestimates.com/p/topic-models-reading-list.html&quot;&gt;Topic Models Reading List&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="819" LastActivityDate="2015-01-08T20:58:44.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4839" PostTypeId="2" ParentId="3804" CreationDate="2015-01-08T21:07:57.643" Score="1" Body="&lt;p&gt;Let me assume you intend to use Python libraries to analyze the data, since you are using Scrapy to gather the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is true, then a factor to consider for storage would be compatibility with other Python libraries.  Of course, plain text is compatible with anything.  But e.g. Pandas has a &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/io.html&quot; rel=&quot;nofollow&quot;&gt;host of IO tools&lt;/a&gt; that simplifies reading from certain formats.  If you intend to use &lt;code&gt;scikit-learn&lt;/code&gt; for modeling, then Pandas can still read the data in for you, if you then cast it from a DataFrame to a Numpy array as an intermediate step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These tools allow you to read CSV and JSON, but also HDF5 ... particularly, I would draw your attention to the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/io.html#io-msgpack&quot; rel=&quot;nofollow&quot;&gt;experimental support for msgpack&lt;/a&gt;, which seems to be a &lt;a href=&quot;http://msgpack.org&quot; rel=&quot;nofollow&quot;&gt;binary version of JSON&lt;/a&gt;.  Binary means here that the stored files will be smaller and therefore faster to read and write.  A somewhat similar alternative is &lt;a href=&quot;http://bsonspec.org&quot; rel=&quot;nofollow&quot;&gt;BSON&lt;/a&gt;, which has a Python implementation  &amp;mdash; no Pandas or Numpy involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering these formats only makes sense if you intend to give &lt;em&gt;at least some formatting&lt;/em&gt; to the stored text, e.g. storing the post title separately from the post content, or storing all posts in a thread in order, or storing the timestamp ... If you considered JSON at all, then I suppose this is what you intended.  If you just intend to store the plain post contents, then use plain text. &lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-01-08T21:07:57.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4842" PostTypeId="2" ParentId="2612" CreationDate="2015-01-09T00:18:05.037" Score="3" Body="&lt;p&gt;You have a few problems here.  The first is cleaning your data.  That's a whole separate issue form anonymization and belongs in another question if you're still having problems with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second is your anonymization.  After you have some sort of identifier you're satisfied with (sounds like you're using people's real names), try hashing their names to generate a new id.  This id is useful because you'll always be able to take the original name and figure out what id it is but won't be able to derive the real names from just the hashed id (providing your hashing algorithm is good).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further reading:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://security.stackexchange.com/a/61878&quot;&gt;https://security.stackexchange.com/a/61878&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/a/21563966/4435034&quot;&gt;https://stackoverflow.com/a/21563966/4435034&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="7719" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-01-09T00:18:05.037" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4843" PostTypeId="2" ParentId="3795" CreationDate="2015-01-09T00:25:16.067" Score="2" Body="&lt;p&gt;Looks like lots of marketing around a specific case of what everyone was already doing to sound bigger than it is.  So, are people implementing mathematical frameworks in software to extract and use the structure within data?  Absolutely.  Are people using the word 'Correlithm' when doing it?  Not as far as I've seen.&lt;/p&gt;&#xA;" OwnerUserId="7719" LastActivityDate="2015-01-09T00:25:16.067" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4844" PostTypeId="1" AcceptedAnswerId="4846" CreationDate="2015-01-09T03:46:04.837" Score="1" ViewCount="8293" Body="&lt;p&gt;I want to build a home server/workstation to run my R projects. Based on what I have gathered, it should probably be Linux based. I want to buy the hardware now, but I am confused with the many available options for processors/ram/motherboards. I want to be able to use parallel processing, at least 64GB? of memory and enough storage space (~10TB?). Software wise, Ubuntu?, R, RStudio, PostgreSQL, some NOSQL database, probably Hadoop. I do a lot of text/geospatial/network analytics that are resource intensive. Budget ~$3000US.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Questions:&lt;/strong&gt;&lt;BR&gt;&#xA;What could an ideal configuration look like? (Hardware + Software)&lt;br&gt;&#xA;What type of processor?&lt;BR&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes:&lt;/strong&gt;&lt;BR&gt;&#xA;No, I don't want to use a cloud solution.&lt;BR&gt;&#xA;I know it is a vague question, but any thoughts will help, please?&lt;BR&gt;&#xA;If it is off-topic or too vague, I will gladly delete.&lt;BR&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers B&lt;/p&gt;&#xA;" OwnerUserId="7722" LastActivityDate="2015-01-09T08:22:40.920" Title="Hardware requirements for Linux server to run R &amp; RStudio" Tags="|r|" AnswerCount="1" CommentCount="4" ClosedDate="2015-01-10T11:13:44.367" ContentLicense="CC BY-SA 3.0" />
  <row Id="4845" PostTypeId="1" CreationDate="2015-01-09T05:42:54.587" Score="2" ViewCount="1017" Body="&lt;p&gt;I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by relevance $d_1,...,d_n$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm specifically interested in approaches which require absolutely no linguistic knowledge and rely on no outside linguistic or lexical resources for boosting performance (such as thesauri or prebuilt word-nets and the like).  Thus where a ranking is produced entirely by evaluating query-document similarity and where the problems of synonymy and polysemy are overcome by exploiting inter-document word co-occurence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've spent some amount of time in the literature (reading papers/tutorials), however there is so much information out there it's hard to get a bird's eye view.  The best I can make out is that modern approaches involve some combination of a weighted vector space model (such as a generalized vector space model, LSI, or a topic-based vector space model using LDA), in conjunction with pseudo-relevance feedback (using either Rocchio or some more advanced approach).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these vector space models tend to use cosine-similarity as the similarity function, however I have seen some literature discussing similarity functions which are more exotic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this basically where we currently are in attacking this particular type of problem? &lt;/p&gt;&#xA;" OwnerUserId="7723" LastEditorUserId="7723" LastEditDate="2015-01-09T06:30:31.983" LastActivityDate="2015-01-12T16:39:40.640" Title="Approaches to Bag-Of-Words Information Retrieval" Tags="|information-retrieval|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4846" PostTypeId="2" ParentId="4844" CreationDate="2015-01-09T08:22:40.920" Score="2" Body="&lt;p&gt;There is no ideal configuration, for &lt;code&gt;R&lt;/code&gt; or in general - product selection is always a difficult task and many factors are at play. I think that the solution is rather simple - get the best computer that your budget allows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, since you want to focus on &lt;code&gt;R&lt;/code&gt; development and one of &lt;code&gt;R&lt;/code&gt;'s pressing issues is its &lt;a href=&quot;https://stackoverflow.com/q/5171593/2872891&quot;&gt;critical dependence&lt;/a&gt; on the amount of available &lt;em&gt;physical memory&lt;/em&gt; (RAM), I would suggest favoring more RAM to other parameters. The second most important parameter, in my opinion, would be &lt;em&gt;number of cores&lt;/em&gt; (or &lt;em&gt;processors&lt;/em&gt; - see details below), due to your potential &lt;em&gt;multiprocessing&lt;/em&gt; focus. Finally, the two next most important criteria I'd pay attention to would be &lt;em&gt;compatibility&lt;/em&gt; with Linux and system/manufacturer's &lt;em&gt;quality&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as the &lt;em&gt;storage&lt;/em&gt; goes, I suggest considering &lt;em&gt;solid state drives (SSD)&lt;/em&gt;, if you'd rather prefer to have a bit more more speed than more space (however, if your work will involve intensive disk operations, you might want to investigate the issue of SSD &lt;em&gt;reliability&lt;/em&gt; or consult with people, knowledgeable in this matter). However, I think that for R-focused work, disk operations are much less critical than memory ones, as I've mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When choosing a specific &lt;em&gt;Linux distribution&lt;/em&gt;, I suggest using a well-supported one, such as Debian or, even better, Ubuntu (if you care more about support, choose their LTS version). I'd rather not buy parts and assemble custom box, but some people would definitely prefer that route - for that you really need to know hardware well, but potential compatibility could still be an issue. The next paragraph provides some examples for both &lt;em&gt;commercial-off-the-shelf (COTS)&lt;/em&gt; and &lt;em&gt;custom&lt;/em&gt; solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should you be interested in the &lt;em&gt;custom system&lt;/em&gt; route, &lt;a href=&quot;http://www.tomshardware.com/forum/295552-28-multiprocessor-multicore-system-rendering-workstation&quot; rel=&quot;nofollow noreferrer&quot;&gt;this discussion&lt;/a&gt; might be worth reading, as it contains some interesting pricing numbers (just to get an idea of potential savings) and also sheds some light on &lt;em&gt;multiprocessor vs. multi-core&lt;/em&gt; alternatives (obviously, the &lt;em&gt;context&lt;/em&gt; is different, but nevertheless could be useful). As I said, I would go the COTS route, mainly due to reliability and compatibility issues. In terms of single-processor multi-core systems, your budget is more than enough. However, when we go to multiprocessor workstations (I'm not even talking about servers), even two-processor configurations can go over your budget easily. Some, not far away, such as &lt;a href=&quot;http://www8.hp.com/us/en/campaigns/workstations/z820.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;HP Z820 Workstation&lt;/a&gt;. It starts from 2439 USD, but in minimal configuration. When you upgrade it to match your desired specs (if it's even possible), I'm sure that we'll be talking about 5K USD price range (extrapolating from the series' higher-level models). What I like about HP Z820, though, is the fact that this system is &lt;a href=&quot;http://www.ubuntu.com/certification/hardware/201011-6754&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ubuntu certified&lt;/a&gt;. Considering system compatibility and assuming your desire to run Ubuntu, the best way to approach your problem is to go through &lt;a href=&quot;http://www.ubuntu.com/certification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ubuntu-certified hardware lists&lt;/a&gt; and shortlist systems that you like. Just for the sake of completeness, take a look at this &lt;a href=&quot;http://www.mediaworkstations.net/i-x2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;interesting multiprocessor system&lt;/a&gt;, which in compatible configuration might cost less than from HP or other major vendors. However, it's multimedia-oriented as well as it's reliability and compatibility are unknown, not to mention that it's way over your specified budget.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;R&lt;/code&gt;-focused software, I highly recommend you to use &lt;em&gt;RStudio Server&lt;/em&gt; instead of &lt;em&gt;RStudio&lt;/em&gt;, as that will provide you with an opportunity to be able to work from any Internet-enabled location (provided you computer will be running, obviously). Another advice that I have is to keep an eye on &lt;em&gt;alternative &lt;code&gt;R&lt;/code&gt; distributions&lt;/em&gt;. I'm not talking about commercial expensive ones, but about emerging open source projects, such as &lt;code&gt;pqR&lt;/code&gt;: &lt;a href=&quot;http://www.pqr-project.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.pqr-project.org&lt;/a&gt;. Will update as needed. I hope this is helpful.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-01-09T08:22:40.920" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4847" PostTypeId="2" ParentId="4837" CreationDate="2015-01-09T10:02:08.437" Score="2" Body="&lt;p&gt;There is a worked out example &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_linear_regression#Numerical_example&quot; rel=&quot;nofollow&quot;&gt;in the Wikipedia page for &quot;simple linear regression&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for the sake of it, let me plug in your example into the formulas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fitted model should be a straight line with parameters $\alpha$ (value at $x = 0$) and $\beta$ (the slope):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = \alpha + \beta x$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values for these parameters that minimize the distance between line and data points are called $\hat{\alpha}$ and $\hat{\beta}$.  They can be computed out of the data point values by using these formulae, &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_linear_regression#Fitting_the_regression_line&quot; rel=&quot;nofollow&quot;&gt;derived here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{align}&#xA;\hat{\beta} &amp;amp; = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } , &#xA;\\&#xA;\\&#xA;\hat{\alpha} &amp;amp; = \bar{y} - \hat{\beta}\bar{x}&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where an expression with an overline $\overline{xy}$ means the sample average of that expression: $\overline{xy} = \tfrac{1}{n} \sum_{i=1}^n{x_iy_i}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the values I find for the datapoints you have listed in your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align}&#xA;\overline{xy} &amp;amp;= \frac{1}{4} \sum{&amp;lt;(1 \times 4000), (2 \times 10000), (3 \times 22000), (4 \times 30000)&amp;gt;} \\&#xA;              &amp;amp;= \frac{1}{4}(4000 + 20000 + 66000 + 120000) = 52500, \\&#xA;\overline{x} &amp;amp;= \frac{1}{4} \sum{&amp;lt;1, 2, 3, 4&amp;gt;} = 2.5 ,\\&#xA;\overline{y} &amp;amp;= \frac{1}{4} \sum{&amp;lt;4000, 10000, 22000, 30000&amp;gt;} = 16500 , \\&#xA;\overline{x^2} &amp;amp;= \frac{1}{4} \sum{&amp;lt;1^2, 2^2, 3^2, 4^2&amp;gt;} = 7.5 , \\&#xA;\overline{x}^2 &amp;amp;= 2.5^2 = 6.25&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the fitted line should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align}\&#xA;\hat{\beta} &amp;amp;= \frac{52500 - 2.5 \times 16500}{7.5 - 6.25} = \frac{41250}{1.25} = 33000 , \\&#xA;\hat{\alpha} &amp;amp;= 16500 - 33000 \times 2.5 = - 66000 , \\&#xA;\Rightarrow f(x) &amp;amp;= - 66000 + 33000 x &#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the model would predict, for a house with 10 rooms, a rent of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ f(x) = -66000 + 33000 \times 10 = 264000 $$&lt;/p&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="1367" LastEditDate="2015-01-09T10:55:33.910" LastActivityDate="2015-01-09T10:55:33.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4848" PostTypeId="2" ParentId="3797" CreationDate="2015-01-09T11:25:11.623" Score="3" Body="&lt;p&gt;What you are asking about is, in my view, the main problem of implementing a lambda architecture.  Here are some suggestions on how to solve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The combination of Spark and &lt;a href=&quot;https://spark.apache.org/docs/1.1.1/streaming-programming-guide.html&quot; rel=&quot;nofollow&quot;&gt;Spark Streaming&lt;/a&gt; largely supersedes the original lambda architecture (which usually involved Hadoop and Storm).  &lt;a href=&quot;http://blog.cloudera.com/blog/2014/08/building-lambda-architecture-with-spark-streaming/&quot; rel=&quot;nofollow&quot;&gt;Read here&lt;/a&gt; an example of how to use a &lt;code&gt;SparkContext&lt;/code&gt; and a separate &lt;code&gt;StreamingContext&lt;/code&gt; to produce &lt;em&gt;different&lt;/em&gt; &lt;code&gt;RDD&lt;/code&gt;s, one for batch processed results and another for real-time results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have replicated that in your system, you still have to think about &lt;em&gt;how to query&lt;/em&gt; both kind of &lt;code&gt;RDD&lt;/code&gt;s.  The trivial case would be to just &lt;code&gt;union&lt;/code&gt; both of them:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; rdd1.union(rdd2).collect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or maybe you can create a new &lt;code&gt;DStream&lt;/code&gt;, similar to &lt;code&gt;stateStream&lt;/code&gt; in the linked example, where some keys are kept for real-time results, and others for batch results. &lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-01-09T11:25:11.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4850" PostTypeId="2" ParentId="2346" CreationDate="2015-01-09T18:26:08.933" Score="3" Body="&lt;p&gt;I think this point is the core of your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;How should I define an &quot;exception&quot;? How can I know if my definition is a good one?&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would go about it as follows: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Go back to the managers who receive the current reporting and ask them to give you examples of reported exceptions that were actually exceptions, i.e. they were acted upon and the action yielded some benefit.&lt;/li&gt;&#xA;&lt;li&gt;Ask them also if there were any features in that report that were not plainly viewable, but led them to think it was an exception, e.g. the month-on-month difference was not reported, but it could be computed from the week-on-week difference.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Treat the examples from the first as labels for training; you want to learn &quot;what is an exception&quot; and you need to have an expert answer that question for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treat the suggested features as new features for your classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cannot get the experts to answer the questions you make, then try to infer them from their interaction with your reports: how many of those reports were downloaded, how many times are they mentioned, how are they ever used in decision-making ...  Try to separate the important ones from the uninteresting ones, and there you have your labels.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-01-09T18:26:08.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4851" PostTypeId="2" ParentId="3781" CreationDate="2015-01-09T19:05:38.040" Score="3" Body="&lt;p&gt;I don't have enough reputation to ask questions in comment for clarification before answering it, so I'm going to do both here.&lt;/p&gt;&#xA;&lt;h3&gt;Here are the things that would help answering this question for now:&lt;/h3&gt;&#xA;&lt;p&gt;Can you post part of the &lt;code&gt;process_set.hpc&lt;/code&gt;?&lt;/p&gt;&#xA;&lt;p&gt;What's its format? Is it a numpy array? Is it a Pandas dataframe?&lt;/p&gt;&#xA;&lt;p&gt;What's the value of &lt;code&gt;cclasses&lt;/code&gt;?&lt;/p&gt;&#xA;&lt;h3&gt;And now the answer:&lt;/h3&gt;&#xA;&lt;p&gt;First of all, &lt;code&gt;k-means&lt;/code&gt; algorithm is able to find clusters in any n-dimensional data. If &lt;code&gt;n&lt;/code&gt; is too big, it is better to use PCA but for &lt;code&gt;n=3&lt;/code&gt; that wouldn't necessarily add any value.&lt;/p&gt;&#xA;&lt;p&gt;The second thing that looks suspicious to me is that in the documentation for &lt;code&gt;kmeans&lt;/code&gt; in &lt;code&gt;scikit-learn&lt;/code&gt;, there is no &lt;code&gt;compute_labels&lt;/code&gt; option, as seen &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. However, that option exists for &lt;code&gt;MiniBatchKMeans&lt;/code&gt; as seen &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Also, if you make your data in the form of a pandas dataframe (if it is not already so), things would be much easier to track and you won't have to reattach the timing information to your data afterwards.&lt;/p&gt;&#xA;&lt;p&gt;I may be able to give you a more thorough answer if I know a bit more about the format of the data.&lt;/p&gt;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="7735" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-01-09T19:05:38.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4852" PostTypeId="1" AcceptedAnswerId="4853" CreationDate="2015-01-10T00:15:45.990" Score="0" ViewCount="5564" Body="&lt;p&gt;I have a mysql database with the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id      string&#xA;1        foo1...&#xA;2        foo2...&#xA;..       ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are &gt;100k entries in this db.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is for each string, compare it to each other string and store some metric of the comparison.  Doing this will essentially yield a 2D matrix of size &lt;code&gt;NxN&lt;/code&gt; where &lt;code&gt;N&lt;/code&gt; is the number of row in the db.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought was creating another db where each index corresponds to the string of the index in the first db and each column is the value from comparing the two strings. For example, id 1 column 2 in the second db would be the value outputted from comparing id1 and id2 in the first db.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The format of the second db:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id    col1    col2    col3    ....&#xA;1       1      0.4     0.5    .....&#xA;...    ...     ...      ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This way of creating the second db would result in 100k rows x 100k columns, which is the issue at hand.  What is the best way to handle large data sets like this?  Is storing the data in a text file more efficient (say each text file corresponds to one row in the second db.)&lt;/p&gt;&#xA;" OwnerUserId="7739" LastActivityDate="2015-04-13T05:26:58.717" Title="Storing Big Matrix in DataBase" Tags="|bigdata|relational-dbms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4853" PostTypeId="2" ParentId="4852" CreationDate="2015-01-10T04:33:10.407" Score="2" Body="&lt;p&gt;It is usually more practical to compute the distances on the fly rather than storing $N^2$ values. If possible, you will want to use a heuristic to select only the items could be sufficiently similar so you don't waste time computing distance to irrelevant items. PostgreSQL can create an &lt;a href=&quot;http://www.postgresql.org/docs/9.1/static/pgtrgm.html&quot; rel=&quot;nofollow&quot;&gt;index on character trigrams&lt;/a&gt; in strings that enables efficient querying for other strings that are sufficiently similar (measured by the Jaccard similarity coefficient).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Storing an $N$ column wide table is not possible because the maximum number of columns allowed in MySQL is 4096. Even if that were not the case, using an RDBMS in such a way is considered bad practice and such a table would be difficult to work with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If for some reason you must store a dense distance matrix, a format such as &lt;a href=&quot;http://www.hdfgroup.org/HDF5/&quot; rel=&quot;nofollow&quot;&gt;HDF5&lt;/a&gt; would be more efficient than an RDBMS.&lt;/p&gt;&#xA;" OwnerUserId="7740" LastEditorUserId="7740" LastEditDate="2015-01-13T04:21:38.920" LastActivityDate="2015-01-13T04:21:38.920" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4855" PostTypeId="1" CreationDate="2015-01-10T22:53:14.353" Score="1" ViewCount="33" Body="&lt;p&gt;I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given second, then that second's data point would contain a positive value in the pertinent feature. If there are more seller initiated trades, then there would be a negative value. And if either there is an equal amount of buy vs seller initiated trades OR if there are simply not any trades in a given second, there will be a 0 for the feature in that data point. Along with this feature, there are several other features that are based on what occurred in the preceding seconds (eg if the value discussed above was 12 for the data point immediately preceding the current point, then the second feature for the current data point would be 12 - please let me know if this is not clear) After much troubleshooting, I have concluded that if there are too many data points with too many 0's for features, the classifier simply wont work. When I print out the probabilities of evaluation data points falling into different classes, I simply get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0:NaN,1:NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for all model evaluation points I try to classify. (I am using logistic regression from apache-mahout. In total have 183 features, but over 40million data points. There are three categories to which the data point can be classified)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found that if I set the default value to 1, then I no longer encounter this error, e.g. if there are no trades, the value will be a 1, if there is one seller initiated trade, the value will be 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So with all this in mind, I have two related questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Has anyone else encountered this issue? e.g. if you have a vector with x features, and for a majority of the data points, a majority of the features contain 0's, is this know to give issues?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is shifting all values up by a constant (such as 1) a valid fix to this issue? I assume that if this constant is applied to all values, then it shouldn't skew the data, but I figure it won't hurt to check with the experts.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Also, I'm new to this, so if you believe that my question could use more info please let me know, and if you could give me ideas of what information to include, it would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="6636" LastActivityDate="2015-04-12T11:25:44.157" Title="Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)" Tags="|feature-selection|logistic-regression|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="4856" PostTypeId="2" ParentId="3771" CreationDate="2015-01-11T01:57:49.490" Score="1" Body="&lt;p&gt;The derivative is a linear transform, and you're using a linear model. As you've demonstrated, nothing is gained by adding linear combination of other features (like the derivative) as a new feature in a linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want a derivative coefficient to help interpret the model. You could use the normal model $x_{n+1} = a * x_n + b * n_{n-1}$ and then report the value $a-b$ to the client. A more general approach is to examine the &lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.freqz.html&quot; rel=&quot;nofollow&quot;&gt;frequency response&lt;/a&gt; of the model (the derivative is a high-pass filter, your model is a FIR filter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternately you can rotate the basis of the input such that the derivative is a feature and the dimensions are still independent as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of the independent variables $x_n$ and $x_{n-1}$ as forming a &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_basis&quot; rel=&quot;nofollow&quot;&gt;standard basis&lt;/a&gt; for the two dimensions of your model's input, that is $\langle 1, 0\rangle$ and $\langle 0, 1\rangle$. &#xA;Neither $a\langle 1, 0 \rangle$ or $b \langle 0, 1 \rangle$ are orthogonal to  c$\langle1, -1\rangle$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, &lt;strong&gt;if you want to use $c(x_n - x_{n-1})$ as a feature, I would also use the orthogonal feature $d(x_n + x_{n-1})$&lt;/strong&gt;. That is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ x_{n+1} = c(x_n - x_{n-1}) + d(x_n + x_{n-1}) $$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could think of this model as the derivative plus the integral. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same logic applies to the choice of scaling function for the Haar wavelet.&lt;/p&gt;&#xA;" OwnerUserId="7738" LastEditorUserId="7738" LastEditDate="2015-01-13T22:55:35.117" LastActivityDate="2015-01-13T22:55:35.117" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4859" PostTypeId="1" CreationDate="2015-01-11T11:43:38.253" Score="3" ViewCount="3881" Body="&lt;p&gt;I am very new in machine learning. I have annotated data with category, aspect, opinion word and sentiment. for example, for the bellow text&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The apple was really tasty&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have category-&gt;food, aspect-&gt; apple, opinion word -&gt;tasty and sentiment-&gt;positive. I have training data like this format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I train a SVM classifier using this format of training set?&#xA;How to extract features like n-gram, POS and sentiment word to train the classifier?&#xA;Could you please suggest any beginning step for this aspect based sentiment analysis using machine learning algorithms?&lt;/p&gt;&#xA;" OwnerUserId="7750" LastActivityDate="2015-01-13T17:00:04.647" Title="Aspect based sentiment analysis using machine learning approach" Tags="|machine-learning|svm|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4860" PostTypeId="2" ParentId="4859" CreationDate="2015-01-11T13:45:04.360" Score="1" Body="&lt;p&gt;I would recommend you to start from reading the draft of the introductory book &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1608458849&quot; rel=&quot;nofollow&quot;&gt;&quot;Sentiment analysis and opinion mining&quot;&lt;/a&gt; by Bing Liu. The draft in a PDF document format is available for free &lt;a href=&quot;http://www.cs.uic.edu/~liub/FBS/SentimentAnalysis-and-OpinionMining.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details about the new upcoming book of this author, as well as comprehensive information on the topic of aspect-based sentiment analysis, with references and links to data sets, are available at this page: &lt;a href=&quot;http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another interesting resource is a survey book &lt;a href=&quot;http://www.cs.cornell.edu/home/llee/opinion-mining-sentiment-analysis-survey.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Opinion mining and sentiment analysis&lt;/a&gt; by Bo Pang and Lillian Lee. The book is available in print and as a downloadable PDF e-book in a &lt;a href=&quot;http://www.cs.cornell.edu/home/llee/omsa/omsa-published.pdf&quot; rel=&quot;nofollow&quot;&gt;published version&lt;/a&gt; or an &lt;a href=&quot;http://www.cs.cornell.edu/home/llee/omsa/omsa.pdf&quot; rel=&quot;nofollow&quot;&gt;author-formatted version&lt;/a&gt;, which are almost identical in terms of contents.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-01-11T13:45:04.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4861" PostTypeId="2" ParentId="4845" CreationDate="2015-01-11T17:38:19.187" Score="1" Body="&lt;p&gt;While reading blogs and papers is helpful to identify the latest and greatest, having a solid foundation helps a lot, too. But I assume you already have gone over Manning's great (and free in e-book form) book on IR, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nlp.stanford.edu/IR-book/&quot; rel=&quot;nofollow&quot;&gt;http://nlp.stanford.edu/IR-book/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It contains information on creating your own thesaurus from your document collection to solve synonymy problems, LSA for polysemy, etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for similarity measures, you will see there that Okapi BM25 (Robertson et al.) is considered superior to cosine similarity (but more expensive to implement and run).&#xA;Regarding the current state of the art, there was a small emergence of Bayesian Network-based classifiers in the early nineties (starting with Turtle &amp;amp; Croft), but that went quiet for a while.&#xA;However, right now, using BNs for IR is again finding some revival, particularly in  biomedical IR.&#xA;In that respect, I think most ongoing work is directed towards using Bayesian models incl. topic models and deep learning for word-sense disambiguation (WSD) and semantic similarity.&#xA;Here is a pointer to a recent paper with good references on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1412.6629&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1412.6629&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6672" LastActivityDate="2015-01-11T17:38:19.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4862" PostTypeId="2" ParentId="3781" CreationDate="2015-01-12T05:47:37.127" Score="0" Body="&lt;p&gt;If you can provide more details about the processing you're doing to the data, I think the responses would be a little more helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few things to consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What's the shape of your data going into k-means? Are you aggregating up to the week level? If so, then your data won't look like the example you posted, since that seems like daily data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try using .&lt;a href=&quot;http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html#numpy.ndarray.shape&quot; rel=&quot;nofollow&quot;&gt;shape&lt;/a&gt; on your output data set. Does the # of rows match the data set you put into it? (This ties in to question #1. You'll be unable to join back directly if your original data is daily and the k-means data is weekly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm not sure if this is what you're attempting, but just declaring &lt;code&gt;data_classes&lt;/code&gt; as the labels isn't going to add them back in to your original data. Like oxtay suggested, using pandas is a good choice because it will allow you to &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/merging.html&quot; rel=&quot;nofollow&quot;&gt;join&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="7761" LastActivityDate="2015-01-12T05:47:37.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4863" PostTypeId="2" ParentId="2436" CreationDate="2015-01-12T07:58:39.780" Score="2" Body="&lt;p&gt;I would like to recommend to check the following &lt;strong&gt;open data&lt;/strong&gt; &lt;em&gt;repositories&lt;/em&gt; and &lt;em&gt;meta-repositories&lt;/em&gt; (they are not focused on &lt;em&gt;categorical data&lt;/em&gt;, but I'm sure that many data sets, listed there, contain such data):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdnuggets.com/datasets&quot; rel=&quot;nofollow&quot;&gt;http://www.kdnuggets.com/datasets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.data.gov&quot; rel=&quot;nofollow&quot;&gt;http://www.data.gov&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free&quot; rel=&quot;nofollow&quot;&gt;http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also check built-in data sets in the open source software &lt;em&gt;Parallel Sets&lt;/em&gt;, which is focused on the categorical data visualization: &lt;a href=&quot;https://eagereyes.org/parallel-sets&quot; rel=&quot;nofollow&quot;&gt;https://eagereyes.org/parallel-sets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-01-12T07:58:39.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4864" PostTypeId="1" CreationDate="2015-01-12T09:27:15.250" Score="0" ViewCount="1001" Body="&lt;p&gt;I have 11 lottery tickets (used) and I have discovered that in each ticket, the 3rd digit's value is +1 of the value of the 6th digit. I have 11 tickets, each ticket is composed of 16 digits. Would someone (anyone) help me find the logic used here, in creating the other digits and their interrelation? I am a complete noob in data analysis, and any help would be greatly appreciated. For those who would like to know, I incidentally discovered the certain pattern, and made me realize these numbers are not totally random and if I could find the underlying pattern I'd be able to predict. The excel file is here &lt;a href=&quot;http://s000.tinyupload.com/index.php?file_id=39010250377074241779&quot; rel=&quot;nofollow&quot;&gt;Excel file&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7762" LastEditorUserId="6496" LastEditDate="2015-01-12T10:33:17.913" LastActivityDate="2015-01-14T09:59:34.743" Title="What is the underlying pattern, or algorithm(s) used in these numbers?" Tags="|data-mining|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4866" PostTypeId="2" ParentId="4864" CreationDate="2015-01-12T10:37:27.507" Score="6" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;11 instances of data is insufficient for discovering the behavior (even finding patterns) for such presumably complex systems. You need at least thousands of them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;These numbers are encoded versions of numbers generated by Serial Key Generation Algorithms, which in my experience are usually implemented by Python. So it's not at all strange that there may be a correlation between some digits of them. As it is necessary for the system to recognize its created numbers from the numbers created by other lotteries.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Even if there was a correlation or correlation between all the instances of lottery numbers, &lt;em&gt;correlation doesn't imply causation&lt;/em&gt; and these patterns, if found any, couldn't really give any information about the key generation system itself.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are really serious about it. you could go to this page and enter your data as a sequence it would show you the pattern, if there is any:&#xA;&lt;a href=&quot;http://oeis.org/&quot; rel=&quot;nofollow&quot;&gt;The On-Line Encyclopedia of Integer Sequences&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="6496" LastEditorUserId="2853" LastEditDate="2015-01-14T09:59:34.743" LastActivityDate="2015-01-14T09:59:34.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4867" PostTypeId="2" ParentId="4855" CreationDate="2015-01-12T10:41:57.830" Score="1" Body="&lt;p&gt;From your question I am not sure how you are using logistic regression. The &quot;vanilla&quot; version of logistic regression gives a binary classifier, but your target has three values: a positive value, a negative value, and zero. The Apache site, on the page for logistic regression refers to the thesis of Paul Komarek, in which &quot;logistic regression&quot; refers to the binary classifier. So your problem might be that there your target variable has 3 values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that that is the case, you have many options, for example:&#xA; - a) use &quot;multi-nomial logistic&quot;, &#xA; - b) use ordinary logistic regression 3 times to fit 3 models, one per level, to distinguish it from the other two combined  &lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2015-01-12T10:41:57.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4868" PostTypeId="2" ParentId="4845" CreationDate="2015-01-12T16:39:40.640" Score="0" Body="&lt;p&gt;The similarity function at the core of the method will define all the values for your distances $d_1,d_2, \ldots, d_n$. The initial query should have some words as a reference point to compare to the words in the document. Not knowing whether the query is a sentence or arbitrary list, you are restricted to a method that does some kind of histogram comparison of the frequency of the words matching in the documents. You can perform naive summations of keyword mappings counts, look at keyword likelihoods in the normalized distributions, or give a distribution of weighting based on the strongest matches. More exotic functions will be based on your prior belief of how the words should be compared. Working within a Bayesian Framework you can see your prior assumptions explicitly. Cosine similarity or any other vector based measure will be slightly arbitrary without knowing the desired nature of comparison between query and document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is not much more you can do without looking at some type of features, or attempt to cross compare the documents together, or use the initial query's structure. In short, my answer is to use normalized frequency similarities of the document to the queries and produce a ranking, and with more specific goals in mind to apply measures like cosine similarity on test datasets to search for the best measure. &lt;/p&gt;&#xA;" OwnerUserId="34" LastActivityDate="2015-01-12T16:39:40.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4870" PostTypeId="2" ParentId="4859" CreationDate="2015-01-13T17:00:04.647" Score="3" Body="&lt;p&gt;Aleksandar Blekh has given some really nice links about the big picture of how to do sentiment analysis; I'll try to provide some links to software and talk about the nitty-gritty of how to make it work. I'll point you to the example using scikit-learn (&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/&lt;/a&gt; ), a machine learning library in Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would first want to take your dataset and load it into scikit-learn in a sparse format. This link (&lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&lt;/a&gt; ) gives examples of how to load text in a bag-of-words representation, and the same module (scikit-learn.feature_extraction.text) can also count n-grams. It then describes how to run Naive Bayes and SVM on that dataset. You can take that example and start playing with it. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-13T17:00:04.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4872" PostTypeId="2" ParentId="3771" CreationDate="2015-01-14T01:08:04.997" Score="1" Body="&lt;p&gt;Using derivatives as features is &lt;strong&gt;almost the same&lt;/strong&gt; as using past values, as both reconstruct phase or state space for dynamic system behind the time series. but they differ in some points, like noise amplification and how they carry information.&#xA;(see &lt;strong&gt;Ref&lt;/strong&gt;: State space reconstruction in the presence of noise; Martin Casdagli; Physica D - 1991 - section 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice all information is embedded in time series, but using derivatives is going to reinterpret this information, which may be useful or useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, if you use all parameters and terms, i believe there is no use in it. but in case of using some algorithms like orthogonal forward regression (OFR) it may be beneficial. (see &lt;strong&gt;Ref&lt;/strong&gt;: Orthogonal least squares methods and their application to non-linear&#xA;system identification; S. CHEN, S. A. BILLINGS; INT. J. CONTROL, 1989)&lt;/p&gt;&#xA;" OwnerUserId="5200" LastActivityDate="2015-01-14T01:08:04.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4873" PostTypeId="1" AcceptedAnswerId="4874" CreationDate="2015-01-14T09:08:00.483" Score="3" ViewCount="15451" Body="&lt;p&gt;How can I get information about an entity from DBpedia using Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg: I need to get all DBpedia information about &lt;a href=&quot;http://dbpedia.org/page/United_States&quot; rel=&quot;nofollow noreferrer&quot;&gt;USA&lt;/a&gt;.  So I need to write the query from python (SPARQL) and need to get all attributes on USA as result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PREFIX db: &amp;lt;http://dbpedia.org/resource/&amp;gt;&#xA;SELECT ?p ?o&#xA;WHERE { db:United_States ?p ?o }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But here all DBpedia information is not displaying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this and which all are the possible plugins/api available for python to connect with DBpedia ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also what will be the SPARQL query for generating the above problem result?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="24968" LastEditDate="2019-03-28T16:07:29.957" LastActivityDate="2019-03-28T16:07:29.957" Title="Querying DBpedia from Python" Tags="|python|" AnswerCount="1" CommentCount="3" ClosedDate="2015-01-15T13:41:48.937" ContentLicense="CC BY-SA 4.0" />
  <row Id="4874" PostTypeId="2" ParentId="4873" CreationDate="2015-01-14T12:46:41.357" Score="5" Body="&lt;p&gt;You do not need a wrapper for DBPedia, you need a library that can issue a SPARQL query to its SPARQL endpoint.  Here is &lt;a href=&quot;https://pypi.python.org/pypi/SPARQLWrapper/1.6.4&quot;&gt;an option for the library&lt;/a&gt; and here is the URL to point it to: &lt;a href=&quot;http://dbpedia.org/sparql&quot;&gt;http://dbpedia.org/sparql&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to issue a DESCRIBE query on the United_States resource page:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PREFIX dbres: &amp;lt;http://dbpedia.org/resource/&amp;gt;&#xA;&#xA;DESCRIBE dbres:United_States&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note this is a &lt;strong&gt;huge&lt;/strong&gt; download of resulting triplets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how you would issue the query:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from SPARQLWrapper import SPARQLWrapper, JSON&#xA;&#xA;def get_country_description():&#xA;    sparql = SPARQLWrapper(&quot;http://dbpedia.org/sparql&quot;)&#xA;    sparql.setReturnFormat(JSON)&#xA;&#xA;    sparql.setQuery(query)  # the previous query as a literal string&#xA;&#xA;    return sparql.query().convert()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="1367" LastEditDate="2015-01-14T16:38:14.707" LastActivityDate="2015-01-14T16:38:14.707" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4875" PostTypeId="1" CreationDate="2015-01-14T15:42:48.013" Score="5" ViewCount="1099" Body="&lt;p&gt;Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find combinations/subsets/segments of the IVs that are highly correlated with the DV?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple example: &#xA;DV: college education (0/1), and IVs: age (20 to 120), income (0 to 1 million), race (white, black, hispanic etc), gender (0/1), state, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then finding correlations combining IVs and subsets of IVs (e.g. women between 30 and 50, with incomes over 100k are highly positively correlated with the DV), and then being able to compare the combinations (e.g. to find out women between 30 and 40, with incomes over 100k have a higher correlation than women between 40 and 50, with incomes over 100k)&lt;/p&gt;&#xA;" OwnerUserId="102" LastEditorUserId="102" LastEditDate="2015-01-14T16:15:04.760" LastActivityDate="2015-03-01T03:58:26.763" Title="Determine highly correlated segments" Tags="|machine-learning|r|correlation|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4876" PostTypeId="1" AcceptedAnswerId="4878" CreationDate="2015-01-14T15:52:10.633" Score="1" ViewCount="852" Body="&lt;p&gt;I have data, which looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/EqH9I.jpg&quot; alt=&quot;Data table&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These data are only for one subject. I will have a lot more.&lt;br&gt;&#xA;These data will be analyzed in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm storing them like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;subject &amp;lt;- rep(1, times = 24)&#xA;measurement &amp;lt;- factor(x = rep(x = 1:3, each = 8), &#xA;                      labels = c(&quot;Distance&quot;, &quot;Frequency&quot;, &quot;Energy&quot;))&#xA;speed &amp;lt;- factor(x = rep(x = 1:2, each = 4, times = 3), &#xA;                labels = c(&quot;speed1&quot;, &quot;speed2&quot;))&#xA;condition &amp;lt;- factor(x = rep(x = 1:2, each = 2, times = 6), &#xA;                    labels = c(&quot;Control&quot;, &quot;Experm&quot;))&#xA;Try &amp;lt;- factor(x = rep(x = 1:2, times = 12), &#xA;              labels = c(&quot;Try1&quot;, &quot;Try2&quot;))&#xA;result &amp;lt;- c(1:8, &#xA;            11:18, &#xA;            21:28)&#xA;&#xA;dt &amp;lt;- data.frame(subject, measurement, speed, condition, Try, result)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the appropriate way to store these data in R (in a data frame)? &lt;/p&gt;&#xA;" OwnerUserId="3377" LastEditorUserId="2853" LastEditDate="2015-03-27T11:22:00.087" LastActivityDate="2015-03-27T11:22:00.087" Title="Appropriate way to store data in R" Tags="|r|dataset|data-formats|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4877" PostTypeId="2" ParentId="4875" CreationDate="2015-01-14T17:06:58.750" Score="1" Body="&lt;p&gt;I am no expert in that particular case, but doing a bit of research, it seems that the measure you want to construct is called &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Point-biserial correlation coefficient&lt;/a&gt;&quot;, i.e. the inferred correlation between a continuous variable $X$ and a categorical variable $Y$, e.g. $Y∈\{−1,0,1\}$.  See a related question on &lt;a href=&quot;https://stats.stackexchange.com/questions/74116/pearson-correlation-between-discrete-variable-thats-mostly-0-and-a-standard-nor&quot;&gt;Cross Validated SE&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And yes, &lt;a href=&quot;http://cran.r-project.org/web/packages/polycor/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;there is an R package for that&lt;/a&gt; :)&lt;/p&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-01-14T17:06:58.750" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4878" PostTypeId="2" ParentId="4876" CreationDate="2015-01-14T17:56:58.563" Score="3" Body="&lt;p&gt;Without more information all I can say is that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the say you're storing it is fine in general &lt;/li&gt;&#xA;&lt;li&gt;you can further transform/store your data depending on your use case&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To expand on #2, if I want to study Distance vs Energy across all subjects, then I would format my data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(reshape2)&#xA;&amp;gt; dt2 &amp;lt;- dt[dt$measurement %in% c('Distance','Energy'),]&#xA;&amp;gt; dt_cast &amp;lt;- dcast(dt2, subject+Try~measurement+speed+condition, value.var='result')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The transformed data (dt_cast) would then look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  subject  Try Distance_speed1_Control Distance_speed1_Experm Distance_speed2_Control&#xA;1       1 Try1                       1                      3                       5&#xA;2       1 Try2                       2                      4                       6&#xA;  Distance_speed2_Experm Energy_speed1_Control Energy_speed1_Experm Energy_speed2_Control&#xA;1                      7                    21                   23                    25&#xA;2                      8                    22                   24                    26&#xA;  Energy_speed2_Experm&#xA;1                   27&#xA;2                   28&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Allowing me to, for example, look at the relationship between the Distance_speed1_Control vs Energy_speed1_Control columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically subset/aggregate your data and then use the dcast to get the rows and columns the computer needs.&lt;/p&gt;&#xA;" OwnerUserId="525" LastActivityDate="2015-01-14T17:56:58.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4879" PostTypeId="1" AcceptedAnswerId="4994" CreationDate="2015-01-15T00:48:48.383" Score="0" ViewCount="49" Body="&lt;p&gt;As an example, say the input is an array of numbers representing an audio snippet and the output is a transformed/filtered version of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the proper name for that? Which are examples of algorithms for the job?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&#xA;More specifically, I want to train audio source separation. The input is a mixed sound (spectrogram) and the output is the sound with some energy removed in certain frequencies. The function needs to recognize some pattern in the input and decide what to remove.&lt;/p&gt;&#xA;" OwnerUserId="7806" LastEditorUserId="7806" LastEditDate="2015-01-24T14:09:47.810" LastActivityDate="2015-01-30T21:14:25.563" Title="Training a function that maps n-dim to n-dim" Tags="|algorithms|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4880" PostTypeId="1" CreationDate="2015-01-15T01:57:38.387" Score="2" ViewCount="585" Body="&lt;p&gt;I'm wondering if there's a way to automatically generate a list of tags for live chat transcripts without domain knowledge.  I've tried applying NLP chunking to the chat transcripts and keep only the noun phrases as tag candidates.  However, this approach would generate too many useless noun phrases.  I could use some rules to prune out some of them, but it would be hard to generalize the rules.&lt;/p&gt;&#xA;" OwnerUserId="7807" LastEditorUserId="11097" LastEditDate="2015-12-15T08:15:38.597" LastActivityDate="2020-08-07T17:05:08.823" Title="Generate tags for live chat transcripts" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4881" PostTypeId="1" CreationDate="2015-01-15T11:35:08.843" Score="1" ViewCount="2278" Body="&lt;p&gt;This question might sound silly. But I have been wondering why do we assume that there is&#xA;a hidden probability distribution between input-output pairs in machine learning setup ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we want to learn a function $f: \mathcal{X} \rightarrow \mathcal{Y}$, we generally tend to assume a probability distribution $\rho(x,y)$ on $Z=\mathcal{X} \times \mathcal{Y} $ and try to minimize the error &#xA;$$&#xA;\mathcal{E}(f) = \int (f(x)-y)^2 \ d\rho(x,y)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the probability distribtution $\rho$ inherent to the very nature of $Z$ or depends on $f$ ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please provide a good intuitive explanation for this ?&lt;/p&gt;&#xA;" OwnerUserId="7811" LastEditorUserId="7811" LastEditDate="2015-01-18T10:14:42.397" LastActivityDate="2015-01-19T18:03:17.467" Title="Probability distribution in input-output pairs" Tags="|machine-learning|statistics|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4883" PostTypeId="1" CreationDate="2015-01-15T15:13:54.713" Score="3" ViewCount="127" Body="&lt;p&gt;I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) implemented in Mahout. Since we want to recommend daily news for users, we use the reading behavior of each user collected two days ago as training set, data of the next day as the testing set. The evaluated RMSE is good, the best recommender is SVD+SGD, so we implemented the recommender on our system for several days of trial run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the result, the actually recommended news, seems to be not very attractive for real users (&quot;not attractive&quot; here means, the users feel like &quot;why you recommend this to me?&quot;). So we decided another approach: use the tags and categories and their relationship to do the main job of recommendation, the result from CF is for just supporting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes me wonder if CF if not appropriate for some kind of content. Because I also worked on movie and music recommendation, CF is a good tool. But for news, it seems not the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain why this happening, and also give some guideline about how to choose appropriate recommendation methods? Thanks:)&lt;/p&gt;&#xA;" OwnerUserId="5184" LastEditorUserId="5184" LastEditDate="2015-01-16T05:22:00.583" LastActivityDate="2015-05-03T19:02:51.487" Title="What kind of data is not appropriate using CF to do recommendation?" Tags="|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4884" PostTypeId="1" AcceptedAnswerId="4887" CreationDate="2015-01-15T18:32:36.883" Score="0" ViewCount="205" Body="&lt;p&gt;I was wondering whether we could list machine learning winning methods to apply in many fields of interest: NLP, image, vision, medical, deep package inspection, etc. I mean, if someone will get started a new ML project, what are the ML methods that cannot be forgotten?&lt;/p&gt;&#xA;" OwnerUserId="6560" LastActivityDate="2015-01-16T01:00:45.890" Title="What are the current killing machine learning methods?" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ClosedDate="2015-01-16T17:19:24.150" ContentLicense="CC BY-SA 3.0" />
  <row Id="4885" PostTypeId="2" ParentId="4836" CreationDate="2015-01-15T20:07:51.903" Score="10" Body="&lt;p&gt;If you're working in R, Carson Sievert's tutorial on using LDA to model topics in movie reviews is an excellent starting point:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cpsievert.github.io/LDAvis/reviews/reviews.html&quot;&gt;http://cpsievert.github.io/LDAvis/reviews/reviews.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tutorial makes use of LDAvis, an interactive visualization of topic and word distributions that can really aid intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, although not short, David M. Blei's lectures on topic models are a great resource for understanding the meaning behind the parameters: &lt;a href=&quot;http://videolectures.net/mlss09uk_blei_tm/&quot;&gt;http://videolectures.net/mlss09uk_blei_tm/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="686" LastActivityDate="2015-01-15T20:07:51.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4886" PostTypeId="2" ParentId="853" CreationDate="2015-01-15T23:27:20.597" Score="4" Body="&lt;p&gt;I am just 5 months late but with CRFSuite you can actually use those float features as numbers, not as strings. For this, you just need to invent an unique label for each dimension then add a &quot;:&quot; followed by the value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a word &quot;jungle&quot; is represented in 5 dimensions:&#xA;0.1 0.4 0.8 0.2 0.9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then CRFSuite would take that word + feature as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LABEL f1:0.1 f2:0.4 f3:0.8 f4:0.2 f5:0.9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where of course you replace ``LABEL'' by an actual string and you separate all spaces with tabs (that's the format for CRFSuite).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure though for other packages.&lt;/p&gt;&#xA;" OwnerUserId="7818" LastActivityDate="2015-01-15T23:27:20.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4887" PostTypeId="2" ParentId="4884" CreationDate="2015-01-16T01:00:45.890" Score="0" Body="&lt;p&gt;The question is very general. However, there are some studies being conducted to test which algorithms perform relatively well in a broad range of problems (I'll add link to papers later), concerning regression and classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lately Random Decision Forests, Support Vector Machines and certain variations of Neural Networks are being said to achieve the best results for very broad variety of problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does not mean that these are &quot;the best algorithms&quot; for any problem, that does not exist, and actually is not very realistic to pursue. Also it must be observed that both RDF and SVM are rather-easy methods to initially grasp and obtain good results, so they are becoming really popular. NN have been used intensively since couple of decades (after they revived), so they appear often in implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in learning further you should look for an specific area and deal with a problem that can be solved nicely by machine learning to understand the main idea (and why is impossible to find  &lt;strong&gt;the method&lt;/strong&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will find common the task to try to predict the expected behavior of something given some known or observable characteristics (to learn the function that models the problem given input data), the issues related to dealing with data in high-dimensional spaces, the need for good quality data, the notable improvements that can give data pre-processing, and many others.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2015-01-16T01:00:45.890" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="4888" PostTypeId="2" ParentId="1253" CreationDate="2015-01-16T01:10:21.240" Score="28" Body="&lt;h3&gt;Why to use deep networks?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let's first try to solve very simple classification task. Say, you moderate a web forum which is sometimes flooded with spam messages. These messages are easily identifiable - most often they contain specific words like &quot;buy&quot;, &quot;porn&quot;, etc. and a URL to outer resources. You want to create filter that will alert you about such suspecious messages. It turns to be pretty easy - you get list of features (e.g. list of suspicious words and presence of a URL) and train simple logistic regression (a.k.a. perceptron), i.e. model like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g(w0 + w1*x1 + w2*x2 + ... + wnxn)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;x1..xn&lt;/code&gt; are your features (either presence of specific word or a URL), &lt;code&gt;w0..wn&lt;/code&gt; - learned coefficients and &lt;code&gt;g()&lt;/code&gt; is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;noreferrer&quot;&gt;logistic function&lt;/a&gt; to make result be between 0 and 1. It's very simple classifier, but for this simple task it may give very good results, creating linear decision boundary. Assuming you used only 2 features, this boundary may look something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/opDcs.png&quot; alt=&quot;linear boundary&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here 2 axes represent features (e.g. number of occurrences of specific word in a message, normalized around zero), red points stay for spam and blue points - for normal messages, while black line shows separation line. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But soon you notice that some good messages contain a lot of occurrences of word &quot;buy&quot;, but no URLs, or extended discussion of &lt;a href=&quot;https://stackoverflow.com/questions/713247/what-is-the-best-way-to-programatically-detect-porn-images&quot;&gt;porn detection&lt;/a&gt;, not actually refferring to porn movies. Linear decision boundary simply cannot handle such situations. Instead you need something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/wPF38.png&quot; alt=&quot;non-linear boundary&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This new non-linear decision boundary is much more &lt;strong&gt;flexible&lt;/strong&gt;, i.e. it can fit the data much closer. There are many ways to achieve this non-linearity - you can use polynomial features (e.g. &lt;code&gt;x1^2&lt;/code&gt;) or their combination (e.g. &lt;code&gt;x1*x2&lt;/code&gt;) or project them out to a higher dimension like in &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_method&quot; rel=&quot;noreferrer&quot;&gt;kernel methods&lt;/a&gt;. But in neural networks it's common to solve it by &lt;strong&gt;combining perceptrons&lt;/strong&gt; or, in other words, by building &lt;a href=&quot;http://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;multilayer perceptron&lt;/strong&gt;&lt;/a&gt;. Non-linearity here comes from logistic function between layers. The more layers, the more sophisticated patterns may be covered by MLP. Single layer (perceptron) can handle simple spam detection, network with 2-3 layers can catch tricky combinations of features, and networks of 5-9 layers, used by large research labs and companies like Google, may model the whole language or detect cats on images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essential reason to have &lt;strong&gt;deep architectures&lt;/strong&gt; - they can &lt;em&gt;model more sophisticated patterns&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Why deep networks are hard to train?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;With only one feature and linear decision boundary it's in fact enough to have only 2 training examples - one positive and one negative. With several features and/or non-linear decision boundary you need several orders more examples to cover all possible cases (e.g. you need not only find examples with &lt;code&gt;word1&lt;/code&gt;, &lt;code&gt;word2&lt;/code&gt; and &lt;code&gt;word3&lt;/code&gt;, but also with all possible their combinations). And in real life you need to deal with hundreds and thousands of features (e.g. words in a language or pixels in an image) and at least several layers to have enough non-linearity. Size of a data set, needed to fully train such networks, easily exceeds 10^30 examples, making it totally impossible to get enough data. In other words, with many features and many layers our decision function becomes &lt;strong&gt;too flexible&lt;/strong&gt; to be able to learn it &lt;strong&gt;precisely&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are, however, ways to learn it &lt;em&gt;approximately&lt;/em&gt;. For example, if we were working in probabilistic settings, then instead of learning frequencies of all combinations of all features we could assume that they are independent and learn only individual frequencies, reducing full and unconstrained &lt;a href=&quot;http://en.wikipedia.org/wiki/Bayes_classifier&quot; rel=&quot;noreferrer&quot;&gt;Bayes classifier&lt;/a&gt; to a &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; and thus requiring much, much less data to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In neural networks there were several attempts to (meaningfully) reduce complexity (flexibility) of decision function. For example, convolutional networks, extensively used in image classification, assume only local connections between nearby pixels and thus try only learn combinations of pixels inside small &quot;windows&quot; (say, 16x16 pixels = 256 input neurons) as opposed to full images (say, 100x100 pixels = 10000 input neurons). Other approaches include feature engineering, i.e. searching for specific, human-discovered descriptors of input data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Manually discovered features are very promising actually. In natural language processing, for example, it's sometimes helpful to use special dictionaries (like those containing spam-specific words) or catch negation (e.g. &quot;&lt;em&gt;not&lt;/em&gt; good&quot;). And in computer vision things like &lt;a href=&quot;http://en.wikipedia.org/wiki/SURF&quot; rel=&quot;noreferrer&quot;&gt;SURF descriptors&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Haar-like_features&quot; rel=&quot;noreferrer&quot;&gt;Haar-like features&lt;/a&gt; are almost irreplaceable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the problem with manual feature engineering is that it takes literally years to come up with good descriptors. Moreover, these features are often specific &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Unsupervised pretraining&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;But it turns out that we can &lt;em&gt;obtain good features automatically&lt;/em&gt; right from the data using such algorithms as &lt;strong&gt;autoencoders&lt;/strong&gt; and &lt;strong&gt;restricted Boltzmann machines&lt;/strong&gt;. I described them in detail in my other &lt;a href=&quot;https://stats.stackexchange.com/questions/114385/what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma/117188#117188&quot;&gt;answer&lt;/a&gt;, but in short they allow to &lt;strong&gt;find repeated patterns in the input data&lt;/strong&gt; and transform it into higher-level features. For example, given only row pixel values as an input, these algorithms may identify and pass higher whole edges, then from these edges construct figures and so on, until you get really high-level descriptors like variations in faces. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/oGBRR.jpg&quot; alt=&quot;deep learning&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After such (unsupervised) pretraining network is usually converted into MLP and used for normal supervised training. Note, that pretraining is done layer-wise. This significantly reduces solution space for learning algorithm (and thus number of training examples needed) as it only needs to learn parameters &lt;em&gt;inside&lt;/em&gt; each layer without taking into account other layers. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;And beyond...&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Unsupervised pretraining have been here for some time now, but recently other algorithms were found to improve learning both - together with pretraining and without it. One notable example of such algorithms is &lt;a href=&quot;http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;dropout&lt;/strong&gt;&lt;/a&gt; - simple technique, that randomly &quot;drops out&quot; some neurons during training, creatig some distortion and preventing networks of following data too closely. This is still a hot research topic, so I leave this to a reader. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-04-20T09:20:40.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4889" PostTypeId="2" ParentId="4880" CreationDate="2015-01-16T04:44:23.923" Score="0" Body="&lt;p&gt;If you have existing properly tagged chat transcripts, you can try treating it as a supervised learning problem. If you're starting from a blank slate, that won't work. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-16T04:44:23.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4890" PostTypeId="2" ParentId="4883" CreationDate="2015-01-16T04:54:40.550" Score="6" Body="&lt;p&gt;The key is establishing a proper validation metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I notice you talk about how you tried different recommendation algorithms, but at the end of the day you evaluated them all with RMSE. But there's no particular reason to believe that minimizing RMSE generates a &quot;subjectively better&quot; recommendation experience for the user - it just happens to be convenient, and happens to work well in some industries, but there is no real reason why it must.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RMSE is measuring how well your recommender algorithm is predicting user behavior. But that's not the same as measuring recommendation quality. Maybe users value something else - familiarity, or serendipity, or some other quality of the item being recommended. Users don't really care about being predicted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your results, if you want to understand your users further, I'd focus my efforts in coming up with a mathematical metric that more closely matches the target you care about - user satisfaction - rather than RMSE. Once you know what metric you're trying to optimize, the algorithm to optimize it is much easier to select! &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-16T04:54:40.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4891" PostTypeId="2" ParentId="441" CreationDate="2015-01-16T05:30:35.787" Score="-1" Body="&lt;p&gt;Machine learning is a good example of a problem type where Spark-based solutions are light-years ahead of mapreduce-based solutions, despite the young age of spark-on-yarn. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-16T05:30:35.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4892" PostTypeId="2" ParentId="4880" CreationDate="2015-01-16T05:44:41.907" Score="2" Body="&lt;p&gt;You can try RAKE(Rapid Automatic Keyword Extraction) and there is a python implementation &lt;a href=&quot;https://github.com/aneesha/RAKE&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. RAKE is an document-oriented keyword extraction algorithm and also language-independent(theoretically, since RAKE use a generated stop word list to partition candidate keywords, and considering different languages, we need to find a better way to generated stop word list.). However, about English documents, RAKE can extract keywords(or tags) in a acceptable precision and recall. RAKE is also efficient, because to use it we don't have to training a whole corpus, RAKE can generate a keyword list by calculating the word's degree and frequency then comes up a score for every candidate keyword then pick the top N words.&lt;/p&gt;&#xA;&lt;p&gt;Hope this answer helps you or lead you a direction for your next step investigation.&lt;/p&gt;&#xA;" OwnerUserId="1003" LastEditorUserId="98307" LastEditDate="2020-08-07T17:05:08.823" LastActivityDate="2020-08-07T17:05:08.823" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="4894" PostTypeId="2" ParentId="3763" CreationDate="2015-01-17T04:29:24.393" Score="1" Body="&lt;p&gt;AdaBoost is a supervised learning method; it starts with a table of 'correct' answers and generates a predictive model for a target, which is known. It is then possible to inspect this model to figure out how it works, what it judged was more important. With that in mind, here is my guess for what he did:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, he created a training dataset. The dataset was created, according to the article, like this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now he’d do the same for love. First he’d need data. While his&#xA;  dissertation work continued to run on the side, he set up 12 fake&#xA;  OkCupid accounts and wrote a Python script to manage them. The script&#xA;  would search his target demographic (heterosexual and bisexual women&#xA;  between the ages of 25 and 45), visit their pages, and scrape their&#xA;  profiles for every scrap of available information: ethnicity, height,&#xA;  smoker or nonsmoker, astrological sign—“all that crap,” he says.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;To find the survey answers, he had to do a bit of extra sleuthing.&#xA;  OkCupid lets users see the responses of others, but only to questions&#xA;  they’ve answered themselves. McKinlay set up his bots to simply answer&#xA;  each question randomly—he wasn’t using the dummy profiles to attract&#xA;  any of the women, so the answers didn’t mat­ter—then scooped the&#xA;  women’s answers into a database. &quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So he used python scripts to collect lots of information! At the end of this, he had a table of data, where each row had three pieces of information:&#xA;- A bot's answer to all the questions, and their weights&#xA;- A woman's answer to all the questions&#xA;- Their match percentage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On this, he could use AdaBoost to create a predictive model, predicting the match percentage from the available information. The weak learners were probably decision stumps, greedily choosing one variable at a time to split on, that's the standard that people refer to when talking about AdaBoost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the predictive model, was in place, it could be used for optimization - determining which weights to put on the questions when holding all other variables constant, maximizing the average match percentage to women in his target audience. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this is just a guess. The article doesn't have much detail. But it's a potential way to use AdaBoost for that purpose. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-17T04:29:24.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4895" PostTypeId="2" ParentId="1208" CreationDate="2015-01-17T04:42:47.530" Score="0" Body="&lt;p&gt;Your results are reasonable. Your data brings several ideas to mind: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) It is quite reasonable that as you change the available features, this will change the relative performance of machine learning methods. This happens quite a lot. Which machine learning method performs best often depends on the features, so as you change the features the best method changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) It is reasonable that in some cases, disparate models will reach the exact same results. This is most likely in the case where the number of data points is low enough or the data is separable enough that both models reach the exact same conclusions for all test points. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-17T04:42:47.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4896" PostTypeId="2" ParentId="2558" CreationDate="2015-01-17T05:20:32.763" Score="1" Body="&lt;p&gt;A simple way would be to consider Laplace Smoothing (&lt;a href=&quot;http://en.wikipedia.org/wiki/Additive_smoothing&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Additive_smoothing&lt;/a&gt; ) or something like it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, instead of calculating your response rate as (Clicks)/(Impressions) you calculate (Clicks + X)/(Impressions + Y), with X and Y chosen, for example, so that X/Y is the global average of clicks/impressions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When Clicks and Impressions are both high, this smoothed response rate is basically equal to the true response rate (signal dominates the prior). When Clicks and Impressions are both low, the this smoothed response rate will be close to the global average response rate - a good guess when you have little data and don't want to put much weight on it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The absolute scale of X and Y will determine how many data points you consider &quot;enough data&quot;. It's been argued that the right thing to do is set X to 1, and Y appropriately given that. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-17T05:20:32.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4897" PostTypeId="2" ParentId="1094" CreationDate="2015-01-17T05:30:01.813" Score="3" Body="&lt;p&gt;You may want to consider gradient boosted trees rather than random forests. They're also an ensemble tree-based method, but since this method doesn't sample dimensions, it won't run in to the problem of not having a useful predictor available to split on at any particular time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different implementations of GBDT have different ways of handling missing values, which will make a big difference in your case; I believe R does ternary splits which is likely to work fine. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-17T05:30:01.813" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4898" PostTypeId="2" ParentId="155" CreationDate="2015-01-17T16:53:08.033" Score="17" Body="&lt;p&gt;Late answer, but here is an eclectic list of &lt;strong&gt;&lt;a href=&quot;http://rs.io/100-interesting-data-sets-for-statistics/&quot;&gt;100+ Interesting Data Sets&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blog post is fun and easy to read through (I have no affiliation). It's worth to scan through, and to scrape a few from the top:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Last words of every Texas inmate executed since 1984 &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;10,000 annotated images of cats&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;2.2 million chess matches&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="7842" LastActivityDate="2015-01-17T16:53:08.033" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="4899" PostTypeId="2" ParentId="4881" CreationDate="2015-01-18T10:22:35.777" Score="0" Body="&lt;p&gt;I may still misunderstand what you mean, but the general simple formulation is to minimize sum of loss over all training examples. Converting to your formulation, that 'assumes' the joint distribution of the input and output is just the empirical distribution found in the data. That's the best assumption you can make without additional information. If you had reason to assume something else, you would. &lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-01-18T10:22:35.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4900" PostTypeId="2" ParentId="2489" CreationDate="2015-01-18T19:14:41.420" Score="1" Body="&lt;p&gt;If I've understood your question correctly, one easy solution would be to concatenate the bits together and make a 7-bit binary sequence then convert it to integer. So, for your sample dataset quoted in the question you would get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;CM&#xA;16&#xA;0&#xA;98&#xA;8&#xA;6&#xA;67&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="6496" LastEditorUserId="6496" LastEditDate="2015-01-19T08:13:58.220" LastActivityDate="2015-01-19T08:13:58.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4901" PostTypeId="1" AcceptedAnswerId="4912" CreationDate="2015-01-19T09:37:56.340" Score="1" ViewCount="237" Body="&lt;p&gt;I have a data set of video watching records in a 3G network. In this data set, 2 different kind of features are included: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;user-side information, e.g., age, gender, data plan and etc; &lt;/li&gt;&#xA;&lt;li&gt;Video watching records of these users, each of  which associated with a download ratio and some detailed network condition metrics, say, download speed, RTT, and something similar.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Under the scenario of internet streaming, a video is divided into several chunks and downloaded to end device one by one, so we have download ratio = download bytes / file size in bytes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, Given this data set, I want to predict the download ratio of each video.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since it is a regression problem, so I use &lt;em&gt;gradient boosting regression tree&lt;/em&gt; as model and run 10-fold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have tried different model parameter configurations and even different models (linear regression, decision regress tree), the best root-mean-square error I can get is 0.3790, which is quite high, because if I don't use any complex models and just use the mean value of known labels as prediction values, then I can still have an RMSE of 0.3890. There is not obvious difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this problem, I have some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this high error rate imply that the label in data set is unpredictable? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apart from the feature problem, is there any other possibilities? If yes, how can I validate them?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="7867" LastEditorUserId="7867" LastEditDate="2015-01-27T11:58:33.430" LastActivityDate="2015-01-27T11:58:33.430" Title="Does high error rate in regression imply the data set is unpredictable?" Tags="|feature-selection|regression|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="4902" PostTypeId="2" ParentId="3730" CreationDate="2015-01-19T10:07:11.270" Score="2" Body="&lt;p&gt;It is always helpful to just Google the exact error that you are seeing, excluding specifics like the actual memory of the vector.  For me, the first hit was an interesting documentation called &amp;quot;&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;R: Memory limits of R&lt;/a&gt;&amp;quot;, where, under &amp;quot;Unix&amp;quot;, one can read:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The address-space limit is system-specific: 32-bit OSes imposes a limit of no more than 4Gb: it is often 3Gb. Running 32-bit executables on a 64-bit OS will have similar limits: 64-bit executables will have an essentially infinite system-specific limit (e.g., 128Tb for Linux on x86_64 cpus).&lt;/p&gt;&#xA;&lt;p&gt;See the OS/shell's help on commands such as limit or ulimit for how to impose limitations on the resources available to a single process. For example a bash user could use&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;ulimit -t 600 -v 4000000&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;whereas a csh user might use&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;limit cputime 10m&lt;/code&gt;&#xA;&lt;code&gt;limit vmemoryuse 4096m&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;to limit a process to 10 minutes of CPU time and (around) 4Gb of virtual memory. (There are other options to set the RAM in use, but they are not generally honoured.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So, you should check&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/11565649/how-to-find-if-my-amazon-ec2-instance-is-32-bit-or-64-bit&quot;&gt;What type of OS are you running on your EC2 instance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;What type of R build are you running on that OS, and &lt;a href=&quot;http://randyzwitch.com/r-amazon-ec2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;make sure you run a 64bit version&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;If both are already 64bit, then use &lt;code&gt;ulimit&lt;/code&gt; to set memory to e.g. 8Gb: &lt;code&gt;ulimit -v 8000000&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-01-19T10:07:11.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4903" PostTypeId="1" AcceptedAnswerId="4904" CreationDate="2015-01-19T14:26:57.117" Score="16" ViewCount="18502" Body="&lt;p&gt;Can anybody tell me what the purpose of feature generation is? And why feature space enrichment is needed before classifying an image? Is it a necessary step?&lt;/p&gt;&#xA;&lt;p&gt;Is there any method to enrich feature space?&lt;/p&gt;&#xA;" OwnerUserId="7873" LastEditorUserId="29169" LastEditDate="2021-03-11T20:19:17.107" LastActivityDate="2021-03-11T20:19:17.107" Title="What is the difference between feature generation and feature extraction?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="4904" PostTypeId="2" ParentId="4903" CreationDate="2015-01-19T14:55:25.580" Score="16" Body="&lt;p&gt;&lt;strong&gt;Feature Generation&lt;/strong&gt; -- This is the process of taking raw, unstructured data and defining features (i.e. variables) for potential use in your statistical analysis. For instance, in the case of text mining you may begin with a raw log of thousands of text messages (e.g. SMS, email, social network messages, etc) and generate features by removing low-value words (i.e. stopwords), using certain size blocks of words (i.e. n-grams) or applying other rules.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Feature Extraction&lt;/strong&gt; -- After generating features, it is often necessary to test transformations of the original features and select a subset of this pool of potential original and derived features for use in your model (i.e. feature extraction and selection). Testing derived values is a common step because the data may contain important information which has a non-linear pattern or relationship with your outcome, thus the importance of the data element may only be apparent in its transformed state (e.g. higher order derivatives). Using too many features can result in multicollinearity or otherwise confound statistical models, whereas extracting the minimum number of features to suit the purpose of your analysis follows the principle of parsimony.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Enhancing your feature space&lt;/em&gt; in this way is often a necessary step in classification of images or other data objects because the raw feature space is typically filled with an overwhelming amount of unstructured and irrelevant data that comprises what's often referred to as &amp;quot;noise&amp;quot; in the paradigm of a &amp;quot;signal&amp;quot; and &amp;quot;noise&amp;quot; (which is to say that some data has predictive value and other data does not). By enhancing the feature space you can better identify the important data which has predictive or other value in your analysis (i.e. the &amp;quot;signal&amp;quot;) while removing confounding information (i.e. &amp;quot;noise&amp;quot;).&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="100328" LastEditDate="2020-07-07T05:58:21.463" LastActivityDate="2020-07-07T05:58:21.463" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="4905" PostTypeId="2" ParentId="4881" CreationDate="2015-01-19T18:03:17.467" Score="1" Body="&lt;p&gt;Some notes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In the case of supervised learning you assume there is a function $f:\mathcal{X} \rightarrow \mathcal{Y}$, which means there is a connection somehow between inputs and outputs and you want to use it in order to predict. But how this function might look? Without considering the model itself, usually what happens is that there is some noise. And this noise can be in $Y$ and also can be in $X$. This noise gives use the probabilistic setup of the problem, because without it we would have only to solve some equations. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now is important to understand that the noise defines the distribution. So a random variable can be imagined as a function having something fixed and well defined and something not fixed, but taking values according with a distribution. If the variable part does not exist, that you would not have a random variable, right, it would be a simple formula. But its not. Now the $P(X)$ incorporates what happens in $X$ alone, and $P(Y)$ what is in $Y$ alone. When you predict the decision theory says that you are interested in saying which is the most probable value $y_i$ given some input value $x_i$. So you are interested in finding $P(Y|X)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A joint probability is not always completely described by marginal probabilities. In fact, is completely described only when marginal probabilities are independent. Which means for r.v. $X, Y$ knowing $P(X)$ and $P(Y)$ does not put you in position to know the joint density $P(X, Y)$ (thought for independence you have  $P(X,Y)=P(X)P(Y)$). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From here you can go directly and try to estimate $P(Y|X)$. In fact if your only interest in only in prediction, this might be a fair bet. A lot of supervised learning algorithms tries to estimate directly this probability. They are called discriminant classifiers. The reason is because they are used to classify something to  the class with maximal conditional probability, you discriminate (choose) the maximum value. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now arriving at your question. Notice the obvious $P(X,Y) = P(Y|X)P(X)$ than you see that by trying learning the joint probability, you also learn the conditional (what you need for prediction). This kind of approach is called generative, because knowing the joint probability you can not only predict, but you can generate new data for your problem. More than that, knowing the join probability can give you more insights related with how what are you model works. You can find such kind of information which is not contained only in marginal distributions. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some final notes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Technically you do not minimize the error function, but it's expectation. The error function remains as it is.&lt;/li&gt;&#xA;&lt;li&gt;$\mathcal{Z}$ is only a domain it's impossible to describe a probability only by it's domain. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's late, I hope I was not completely incoherent.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-01-19T18:03:17.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4906" PostTypeId="5" CreationDate="2015-01-19T19:11:09.507" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-01-19T19:11:09.507" LastActivityDate="2015-01-19T19:11:09.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4907" PostTypeId="4" CreationDate="2015-01-19T19:11:09.507" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-01-19T19:11:09.507" LastActivityDate="2015-01-19T19:11:09.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4908" PostTypeId="5" CreationDate="2015-01-19T19:17:36.820" Score="0" Body="&lt;h2&gt;What is Machine Learning?&lt;/h2&gt;&#xA;&lt;p&gt;Machine Learning is a subfield of computer science that draws on elements from algorithmic analysis, computational statistics, mathematics, optimization, etc. It is mainly concerned with the use of data to construct models that have high predictive/forecasting ability. Topics include modeling building, applications, theory, etc. Machine Learning is often used on large datasets to help draw predictive relationships between underlying features within the data.&lt;/p&gt;&#xA;&lt;p&gt;Modern applications of Machine Learning are wide ranging including those in Bioinformatics, Astronomy, Computational Physics, Economics, Natural Language Processing, Image Recognition/Object Detection, Robotics, Recommendation Systems, etc.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;&lt;a href=&quot;/questions/tagged/machine-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;machine-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;machine-learning&lt;/a&gt; Tag usage&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;When posting questions about Machine Learning, please make sure to take the following into consideration:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;All questions should include &lt;em&gt;both&lt;/em&gt; sufficient detail and clarity to be able to solve the problem at hand. This includes links to original data sources, code used for model construction, links to tutorials/other resources used, etc.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Questions should generally be more specific than &amp;quot;which model should I use&amp;quot; or &amp;quot;how can I achieve this&amp;quot; and explain what has been attempted/done so far.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Unless directly related to the problem, all questions regarding where to get data (sources, APIs, datasets, etc.) should &lt;em&gt;not&lt;/em&gt; be posted on Stack Exchange Data Science, but rather on: &lt;a href=&quot;https://opendata.stackexchange.com&quot;&gt;Open Data Stack Exchange&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;Types&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;Please see below for a (non-exhaustive) list of the types of Machine Learning:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Supervised Learning &lt;a href=&quot;/questions/tagged/supervised-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;supervised-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;supervised-learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Unsupervised Learning &lt;a href=&quot;/questions/tagged/unsupervised-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;unsupervised-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;unsupervised-learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Semisupervised Learning &lt;a href=&quot;/questions/tagged/semisupervised-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;semisupervised-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;semisupervised-learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reinforcement Learning &lt;a href=&quot;/questions/tagged/reinforcement-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;reinforcement-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;reinforcement-learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Deep Learning &lt;a href=&quot;/questions/tagged/deep-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;deep-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;deep-learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;External Resources&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;: Machine Learning in Python&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;Machine Learning Journals&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.jmlr.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Journal of Machine Learning Research&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.springer.com/journal/10994&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://cis.ieee.org/publications/t-neural-networks-and-learning-systems&quot; rel=&quot;nofollow noreferrer&quot;&gt;IEEE Neural Networks and Learning Systems&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.jair.org/index.php/jair&quot; rel=&quot;nofollow noreferrer&quot;&gt;Journal of Artificial Intelligence Research&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=34&quot; rel=&quot;nofollow noreferrer&quot;&gt;IEEE Pattern Analysis and Machine Intelligence&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.journals.elsevier.com/neurocomputing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neurocomputing&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="29169" LastEditDate="2021-04-03T23:49:53.930" LastActivityDate="2021-04-03T23:49:53.930" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="4909" PostTypeId="4" CreationDate="2015-01-19T19:17:36.820" Score="0" Body="Machine Learning is a subfield of computer science that draws on elements from algorithmic analysis, computational statistics, mathematics, optimization, etc. It is mainly concerned with the use of data to construct models that have high predictive/forecasting ability. Topics include modeling building, applications, theory, etc.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="29169" LastEditDate="2021-03-23T14:23:11.697" LastActivityDate="2021-03-23T14:23:11.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="4910" PostTypeId="5" CreationDate="2015-01-19T19:22:11.877" Score="0" Body="&lt;p&gt;In machine learning and statistics, &lt;a href=&quot;http://en.wikipedia.org/wiki/Statistical_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification&lt;/a&gt; refers to the problem of predicting category memberships based on a set of pre-labeled examples. It is thus a type of &lt;a href=&quot;http://en.wikipedia.org/wiki/Supervised_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;supervised learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the most important classification algorithms are &lt;a href=&quot;http://en.wikipedia.org/wiki/Support_vector_machines&quot; rel=&quot;nofollow noreferrer&quot;&gt;support vector machines&lt;/a&gt; &lt;a href=&quot;/questions/tagged/svm&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;svm&amp;#39;&quot; rel=&quot;tag&quot;&gt;svm&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;logistic regression&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;naive Bayes&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest&lt;/a&gt; &lt;a href=&quot;/questions/tagged/random-forest&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;random-forest&amp;#39;&quot; rel=&quot;tag&quot;&gt;random-forest&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;artificial neural networks&lt;/a&gt; &lt;a href=&quot;/questions/tagged/neural-network&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;neural-network&amp;#39;&quot; rel=&quot;tag&quot;&gt;neural-network&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we wish to associate inputs with continuous values in a supervised framework, the problem is instead known as &lt;a href=&quot;/questions/tagged/regression&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;regression&amp;#39;&quot; rel=&quot;tag&quot;&gt;regression&lt;/a&gt;. The unsupervised counterpart to classification is known as &lt;a href=&quot;/questions/tagged/clustering&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;clustering&amp;#39;&quot; rel=&quot;tag&quot;&gt;clustering&lt;/a&gt; (or cluster analysis), and involves grouping data into categories based on some measure of inherent similarity. &lt;/p&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-12-26T04:14:54.953" LastActivityDate="2017-12-26T04:14:54.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4911" PostTypeId="4" CreationDate="2015-01-19T19:22:11.877" Score="0" Body="An instance of supervised learning that identifies the category or categories which a new instance of dataset belongs." OwnerUserId="6496" LastEditorUserId="6496" LastEditDate="2015-01-20T23:20:11.470" LastActivityDate="2015-01-20T23:20:11.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4912" PostTypeId="2" ParentId="4901" CreationDate="2015-01-19T21:18:54.347" Score="1" Body="&lt;p&gt;It's a little hasty to make too many conclusions about your data based on what you presented here. At the end of the day, all the information you have right now is that &quot;GBT did not work well for this prediction problem and this metric&quot;, summed up by a single RMSE comparison. This isn't very much information - it could be that this is a bad dataset for GBT and some &#xA;other model would work, it could be that the label can't be predicted from these features with any model, or there could be some error in model setup/validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend checking the following hypotheses: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Maybe, with your dataset size and the features you have, GBT isn't a very high-performance model. Try something completely different - maybe just a simple linear regression! Or a random forest. Or GBDT with very different parameter settings. Or something else. This will help you diagnose whether it's an issue with choice of models or with something else; if a few very different approaches give you roughly similar results, you'll know that it's not the model choice that is causing these results, and if one of those models behaves differently, then that gives you additional information to help diagnose the issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Maybe there's some issue with model setup and validation? I would recommend doing some exploration to get some intuition as to whether the RMSE you're getting is reasonable or whether you should expect better. Your post contained very little detail about what the data actually represents, what you know about the features and labels, etc. Perhaps you know those things but didn't include them here, but if not, you should go back and try to get additional understanding of the data before continuing. Look at some random data points, plot the columns against the target, look at the histograms of your features and labels, that sort of thing. There's no substitute for looking at the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Maybe there just aren't enough data points to justify complex models. When you have low numbers of data points (&amp;lt; 100), a simpler parametric model built with domain expertise and knowledge of what the features are may very well outperform a nonparametric model.  &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-01-19T21:18:54.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4914" PostTypeId="1" CreationDate="2015-01-20T15:27:38.160" Score="40" ViewCount="2668" Body="&lt;p&gt;Recently in a Machine Learning class from professor Oriol Pujol at UPC/Barcelona he described the most common algorithms, principles and concepts to use for a wide range of machine learning related task. Here I share them with you and ask you: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is there any comprehensive framework matching tasks with approaches or methods related to different types of machine learning related problems?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I learn a simple Gaussian?&lt;/strong&gt; &#xA;Probability, random variables, distributions; estimation, convergence and asymptotics, confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I learn a mixture of Gaussians (MoG)?&lt;/strong&gt; Likelihood, Expectation-Maximization (EM); generalization, model selection, cross-validation; k-means, hidden markov models (HMM)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I learn any density?&lt;/strong&gt; Parametric vs. non-Parametric estimation, Sobolev and other functional spaces; l  ́ 2 error; Kernel density estimation (KDE), optimal kernel, KDE theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I predict a continuous variable (regression)?&lt;/strong&gt; Linear regression, regularization, ridge regression, and LASSO; local linear regression; conditional density estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I predict a discrete variable (classification)?&lt;/strong&gt; Bayes classifier, naive Bayes, generative vs. discriminative; perceptron, weight decay, linear support vector machine; nearest neighbor classifier and theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which loss function should I use?&lt;/strong&gt; Maximum likelihood estimation theory; l -2 estimation; Bayessian estimation; minimax and decision theory, Bayesianism vs frequentism&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which model should I use?&lt;/strong&gt; AIC and BIC; Vapnik-Chervonenskis theory; cross-validation theory; bootstrapping; Probably Approximately Correct (PAC) theory; Hoeffding-derived bounds&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I learn fancier (combined) models?&lt;/strong&gt; Ensemble learning theory; boosting; bagging; stacking&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I learn fancier (nonlinear) models?&lt;/strong&gt; Generalized linear models, logistic regression; Kolmogorov theorem, generalized additive models; kernelization, reproducing kernel Hilbert spaces, non-linear SVM, Gaussian process regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I learn fancier (compositional) models?&lt;/strong&gt; Recursive models, decision trees, hierarchical clustering; neural networks, back propagation, deep belief networks; graphical models, mixtures of HMMs, conditional random fields, max-margin Markov networks; log-linear models; grammars&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I reduce or relate features?&lt;/strong&gt; Feature selection vs dimensionality reduction, wrapper methods for feature selection; causality vs correlation, partial correlation, Bayes net structure learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I create new features?&lt;/strong&gt; principal component analysis (PCA), independent component analysis (ICA), multidimensional scaling, manifold learning, supervised dimensionality reduction, metric learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I reduce or relate the data?&lt;/strong&gt; Clustering, bi-clustering, constrained clustering; association rules and market basket analysis; ranking/ordinal regression; link analysis; relational data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I treat time series?&lt;/strong&gt; ARMA; Kalman filter and stat-space models, particle filter; functional data analysis; change-point detection; cross-validation for time series&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I treat non-ideal data?&lt;/strong&gt; covariate shift; class imbalance; missing data, irregularly sampled data, measurement errors; anomaly detection, robustness&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I optimize the parameters?&lt;/strong&gt; Unconstrained vs constrained/Convex optimization, derivative-free methods, first- and second-order methods, backfitting; natural gradient; bound optimization and EM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I optimize linear functions?&lt;/strong&gt; computational linear algebra, matrix inversion for regression, singular value decomposition (SVD) for dimensionality reduction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I optimize with constraints?&lt;/strong&gt; Convexity, Lagrange multipliers, Karush-Kuhn-Tucker conditions, interior point methods, SMO algorithm for SVM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I evaluate deeply-nested sums?&lt;/strong&gt; Exact graphical model inference, variational bounds on sums, approximate graphical model inference, expectation propagation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I evaluate large sums and searches?&lt;/strong&gt; Generalized N-body problems (GNP), hierarchical data structures, nearest neighbor search, fast multiple method; Monte Carlo integration, Markov Chain Monte Carlo, Monte Carlo SVD&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I treat even larger problems?&lt;/strong&gt; Parallel/distributed EM, parallel/distributed GNP; stochastic subgradient methods, online learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I apply all this in the real world?&lt;/strong&gt; Overview of the parts of the ML, choosing between the methods to use for each task, prior knowledge and assumptions; exploratory data analysis and information visualization; evaluation and interpretation, using confidence intervals and hypothesis test, ROC curves; where the research problems in ML are&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="5143" LastEditDate="2015-01-28T14:23:32.613" LastActivityDate="2015-01-28T14:23:32.613" Title="When to use what - Machine Learning" Tags="|machine-learning|algorithms|" AnswerCount="3" CommentCount="7" ClosedDate="2015-01-26T16:43:10.387" ContentLicense="CC BY-SA 3.0" />
  <row Id="4915" PostTypeId="2" ParentId="4914" CreationDate="2015-01-21T03:06:39.243" Score="3" Body="&lt;p&gt;That's a good list covering a lot.  I've used some of these methods since before anything was called machine learning, and I think you will see some of the methods you list coming in and out of use over time.  If a method has been out of favour for too long, it might be time for a revisit.  Some methods can obfuscate behind different names resulting from different fields of study.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the main areas I have used these methods is in mineral potential modelling, which is geospatial and to support that you could add some additional categories relating to spatial and oriented data methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking your broad question to specific fields will probably be where you find more examples of methods not in your comprehensive list.  For example, two methods I've seen in mineral potential have been backward stepwise regression and weights of evidence modelling.  I'm not a statistician; perhaps these would be considered covered in the list under linear regression and Bayesian methods.&lt;/p&gt;&#xA;" OwnerUserId="6646" LastEditorUserId="6646" LastEditDate="2015-01-21T21:52:52.590" LastActivityDate="2015-01-21T21:52:52.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4916" PostTypeId="2" ParentId="4914" CreationDate="2015-01-21T07:50:53.953" Score="6" Body="&lt;p&gt;I agree with @geogaffer. This is a very good list, indeed. However, I see some &lt;em&gt;issues&lt;/em&gt; with this list as it is currently formulated. For example, one issue is that suggested solutions are of different &lt;strong&gt;granularity levels&lt;/strong&gt; - some of them represent &lt;em&gt;approaches&lt;/em&gt;, some - &lt;em&gt;methods&lt;/em&gt;, some - &lt;em&gt;algorithms&lt;/em&gt;, and some other - just &lt;em&gt;concepts&lt;/em&gt; (in other words, &lt;em&gt;terms&lt;/em&gt; within a topic's domain terminology). In addition, - and I believe that this is much more important than the above - I think that it would be much valuable, if all those solutions in the list were arranged within a &lt;strong&gt;unified thematic statistical framework&lt;/strong&gt;. This idea was inspired by reading an excellent book by Lisa Harlow &quot;The essence of multivariate thinking&quot;. Hence, recently I've initiated a corresponding, albeit currently somewhat limited, &lt;a href=&quot;https://stats.stackexchange.com/q/131320/31372&quot;&gt;discussion&lt;/a&gt; on the StackExchange's &lt;em&gt;Cross Validated&lt;/em&gt; site. Don't let the title confuse you - my implied intention and hope is for building a &lt;em&gt;unified framework&lt;/em&gt;, as mentioned above.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-01-21T07:50:53.953" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="4917" PostTypeId="1" CreationDate="2015-01-21T09:15:54.750" Score="1" ViewCount="175" Body="&lt;p&gt;I want to visualize goal achievment progress.&#xA;This is my first idea:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use area chart to show progress in current metric&lt;/li&gt;&#xA;&lt;li&gt;use horizontal band to show the goal value&lt;/li&gt;&#xA;&lt;li&gt;colorize areas under/above the band into &quot;positive&quot; and &quot;negative&quot; colors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/h7fUa.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/EIhmz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this approach informative enough? Are there better choises?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional info:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;charts made in Tableau&lt;/li&gt;&#xA;&lt;li&gt;two data sources: metric progress &amp;amp; goals&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastActivityDate="2015-01-23T23:16:54.220" Title="Visualize performance, % of goal implementation" Tags="|visualization|marketing|tableau|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4920" PostTypeId="2" ParentId="4914" CreationDate="2015-01-21T14:17:17.970" Score="1" Body="&lt;p&gt;I think your approach is a little backwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;What is the mean of a Gaussian distribution fitted to this data?&quot; is never the problem statement, so &quot;how do I fit a Gaussian?&quot; is never the problem you actually want to solve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is more than semantic. Consider the question of &quot;how do I construct new features?&quot; If your goal is to develop an index, you might use some type of factor analysis. If your goal is to simply reduce the feature space before fitting a linear model, you might skip the step entirely and use elastic net regression instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach would be to compile a list of &lt;em&gt;actual data analysis tasks you would like to be able to tackle&lt;/em&gt;. Questions like:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do I predict whether customers will return to my shopping website?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;How do I learn how many &quot;major&quot; consumer shopping patterns there are, and what are they?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;How do I construct an index of &quot;volatility&quot; for different items in my online store?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also your list right now includes an enormous amount of material; far too much to &quot;review&quot; and gain more than a surface-level understanding. Having an actual purpose in mind can help you sort out your priorities.&lt;/p&gt;&#xA;" OwnerUserId="1156" LastActivityDate="2015-01-21T14:17:17.970" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4921" PostTypeId="1" AcceptedAnswerId="4922" CreationDate="2015-01-22T00:52:51.710" Score="2" ViewCount="157" Body="&lt;p&gt;I am COMPLETELY new to the field of Data Science, mainly because every employer I have worked for, simply COULDN'T sell any customers anything that would use techniques learned in this field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of particular interest to me is machine learning/Predictive Analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attempted many &quot;test projects&quot; myself, but I seem to NEED some sort of outside &quot;catalyst&quot; to tell me a specific goal, and a specific set of guidelines, when I am trying to learn something.&#xA;Otherwise, I tend to lose focus, and jump from one interesting topic to the next, without ever gaining any experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!!&lt;/p&gt;&#xA;" OwnerUserId="7909" LastActivityDate="2015-01-22T17:25:40.690" Title="Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to &quot;get my feet wet, so to speak&quot;" Tags="|machine-learning|data-mining|bigdata|predictive-modeling|" AnswerCount="2" CommentCount="2" ClosedDate="2015-01-23T10:23:39.330" ContentLicense="CC BY-SA 3.0" />
  <row Id="4922" PostTypeId="2" ParentId="4921" CreationDate="2015-01-22T10:44:37.240" Score="3" Body="&lt;p&gt;I would suggest Kaggle learning projects - &lt;a href=&quot;http://www.kaggle.com/competitions&quot; rel=&quot;nofollow&quot;&gt;http://www.kaggle.com/competitions&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look for the ones in the 101 section that offer knowledge. There's many pre-made solutions ready, which you can ingest and try variations of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I have bookmarked a &lt;a href=&quot;http://www.analyticsvidhya.com/blog/learning-path-data-science-python/&quot; rel=&quot;nofollow&quot;&gt;Comprehensive learning path – Data Science in Python&lt;/a&gt;, which among other things gives a few answers to your specific question.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-01-22T10:44:37.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4923" PostTypeId="1" CreationDate="2015-01-22T13:36:24.967" Score="2" ViewCount="623" Body="&lt;p&gt;In SIFT feature extraction how the key points will be generated and how the features will be stored in the database. In image will the bag of visual words be images or text words?&lt;/p&gt;&#xA;" OwnerUserId="7873" LastActivityDate="2015-01-23T06:15:40.773" Title="Bag of Words creation in image" Tags="|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4924" PostTypeId="2" ParentId="4921" CreationDate="2015-01-22T17:25:40.690" Score="2" Body="&lt;p&gt;You can use some public data sets to play around with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The iris flower data set is great for classification problems. As an example you can try to classify the flower species with k-nearest neighbor model or a decision tree (CART) model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find this and other sample machine learning datasets here: &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow&quot;&gt;http://archive.ics.uci.edu/ml/datasets.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This list identifies what type of machine learning tasks can be performed with each dataset. &lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2015-01-22T17:25:40.690" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4925" PostTypeId="1" CreationDate="2015-01-22T21:34:57.443" Score="28" ViewCount="3297" Body="&lt;p&gt;As there are numerous tools available for data science tasks, and it's cumbersome to install everything and build up a perfect system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a Linux/Mac OS image with Python, R and other open-source data science tools installed and available for people to use right away? An Ubuntu or a light weight OS with latest version of Python, R (including IDEs), and other open source data visualization tools installed will be ideal. I haven't come across one in my quick search on Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if there are any or if someone of you have created one for yourself? I assume some universities might have their own VM images. Please share such links.&lt;/p&gt;&#xA;" OwnerUserId="7923" LastEditorUserId="31513" LastEditDate="2017-04-24T17:37:47.470" LastActivityDate="2021-02-09T04:28:40.193" Title="VM image for data science projects" Tags="|python|r|tools|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4926" PostTypeId="2" ParentId="4925" CreationDate="2015-01-23T02:08:59.950" Score="13" Body="&lt;p&gt;There is another choice which popular recently: docker(&lt;a href=&quot;https://www.docker.com&quot;&gt;https://www.docker.com&lt;/a&gt;). Docker is a container and let you create/maintain a working environment very easily and fast. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;install essential tools for data science in python &#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://registry.hub.docker.com/u/ceshine/python-datascience/&quot;&gt;https://registry.hub.docker.com/u/ceshine/python-datascience/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;use r language to do data science&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/rocker-org/rocker&quot;&gt;https://github.com/rocker-org/rocker&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope that would help you.&lt;/p&gt;&#xA;" OwnerUserId="1003" LastActivityDate="2015-01-23T02:08:59.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4927" PostTypeId="2" ParentId="4925" CreationDate="2015-01-23T02:26:23.773" Score="8" Body="&lt;p&gt;While &lt;em&gt;Docker&lt;/em&gt; images are now more trendy, I personally find &lt;em&gt;Docker&lt;/em&gt; technology not user-friendly, even for advanced users. If you are OK with using &lt;em&gt;non-local&lt;/em&gt; VM images and can use &lt;em&gt;Amazon Web Services (AWS) EC2&lt;/em&gt;, consider R-focused images for data science projects, pre-built by Louis Aslett. The images contain very recent, if not the latest, versions of &lt;em&gt;Ubuntu LTS&lt;/em&gt;, &lt;em&gt;R&lt;/em&gt; and &lt;em&gt;RStudio Server&lt;/em&gt;. You can access them &lt;a href=&quot;http://www.louisaslett.com/RStudio_AMI&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides main components I've listed above, the images contain many useful data science tools built-in as well. For example, the images support LaTeX, ODBC, OpenGL, Git, optimized numeric libraries and more.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-01-23T02:32:18.343" LastActivityDate="2015-01-23T02:32:18.343" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="4928" PostTypeId="2" ParentId="4923" CreationDate="2015-01-23T06:10:37.417" Score="2" Body="&lt;p&gt;How to calculate key points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Take your image $I(x,y)$ and convolve it using a Gaussian $G(x,y,k\sigma$) at different scales $k \sigma$ to obtain $L(x,y,k\sigma)$. This produces several versions (one for each scale $k\sigma$) of your image $I(x,y)$ with different degrees of blurring.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Separate your blurred images according to octaves (an octave is usually taken as $2\sigma$). Within a given octave, take the difference between adjacent blurred images $L(x,y,k_{i}\sigma)$ and  $L(x,y,k_{j}\sigma)$. This difference is called a Difference of Gaussians (DoG) $D(x,y,\sigma) = L(x,y,k_{i}\sigma) - L(x,y,k_{j}\sigma)$. At this point, an image will be helpful (source: &lt;a href=&quot;http://docs.opencv.org/master/doc/py_tutorials/py_feature2d/py_sift_intro/py_sift_intro.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;opencv&lt;/a&gt;):&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/yLVAi.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, you have the blurred/convolved/filtered images on the left and the differences between adjacent images on the right.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Then, take one pixel on a DoG and compare it to its 8 neighbors in the same DoG, as well as to the 9 equivalent pixels of the DoG located in the next scale  and to the other 9 pixels in the previous DoG. In the drawing, this corresponds to the DoG's located above and below. If this pixel is a local extrema, then you have a candidate keypoint. Keep in mind that candidates can be discarded based on other criteria.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Once you have these keypoints, you proceed to generate descriptors. For each keypoint, calculate an image gradient. As you should know, gradients tell you the direction of maximum rate of change. Therefore, you can construct a grid around each keypoint that is oriented according to the dominant gradient around that point. This grid has subregions (usually 16 subregions) and for each subregion, calculate an 8-bin histogram. Finally, concatenate every histogram you obtained from every subregion in your grid and that is the feature vector for that keypoint (actually, the full feature vector also includes the location and rotation angle.) A helpful illustration (from Solem's book Programming Computer Vision with Python):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ws829.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, you have a feature vector with 132 values for each keypoint. As for your second question, I'm not sure about the best approach to store an array in a database. Maybe others can expand on this point. Of course, there are several options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create an array datatype to store your data&lt;/li&gt;&#xA;&lt;li&gt;Simply use a VARCHAR field&lt;/li&gt;&#xA;&lt;li&gt;Store it as a binary file&lt;/li&gt;&#xA;&lt;li&gt;Use a database specifically designed to handle arrays.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="4621" LastEditDate="2015-01-23T06:15:40.773" LastActivityDate="2015-01-23T06:15:40.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4929" PostTypeId="2" ParentId="4925" CreationDate="2015-01-23T10:22:36.327" Score="12" Body="&lt;p&gt;If you are looking for a VM with a bunch of tools preinstalled, try the &lt;a href=&quot;http://datasciencetoolbox.org/&quot;&gt;Data Science Toolbox&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-01-23T10:22:36.327" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="4930" PostTypeId="2" ParentId="4925" CreationDate="2015-01-23T14:00:52.367" Score="5" Body="&lt;p&gt;Did you try Cloudera's QuickStart VM?:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cloudera.com/content/cloudera/en/downloads/quickstart_vms/cdh-5-3-x.html&quot;&gt;http://www.cloudera.com/content/cloudera/en/downloads/quickstart_vms/cdh-5-3-x.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I found it very easy to run it and it includes open source software such as &lt;a href=&quot;http://mahout.apache.org/&quot;&gt;Mahout&lt;/a&gt; and &lt;a href=&quot;https://spark.apache.org/&quot;&gt;Spark&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="7919" LastActivityDate="2015-01-23T14:00:52.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4931" PostTypeId="2" ParentId="4925" CreationDate="2015-01-23T15:34:07.570" Score="5" Body="&lt;p&gt;Today I used &lt;a href=&quot;https://github.com/sequenceiq/docker-spark&quot; rel=&quot;nofollow noreferrer&quot;&gt;this repository&lt;/a&gt; and built it with docker. It is a docker image building spark based on Hadoop image of the same owner. If you to use spark, it has a python api called &lt;a href=&quot;http://spark.apache.org/docs/latest/api/python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pyspark&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="7926" LastEditorUserId="29169" LastEditDate="2021-02-09T04:28:40.193" LastActivityDate="2021-02-09T04:28:40.193" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="4934" PostTypeId="2" ParentId="4917" CreationDate="2015-01-23T23:16:54.220" Score="2" Body="&lt;p&gt;What information should be shown on the charts are based on what the viewer wants. From your sample charts, the x axis indicates 'date'. For time serial analysis, people usually want to know the trending of the metrics and to predict the performance in the future. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are 2 suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;People may want to know how it will perform in the future. Add a trendline. Trendline could help people to predict the performance in the future.&#xA;&lt;a href=&quot;http://onlinehelp.tableausoftware.com/current/pro/online/mac/en-us/trendlines.html&quot; rel=&quot;nofollow&quot;&gt;http://onlinehelp.tableausoftware.com/current/pro/online/mac/en-us/trendlines.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;People may want to know how far away from goal. When people mouseover on the points on the metric line, you can try to show how far from current metric to the goal.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Try to understand what information people want to know and visualise them. Hope my answer could help you.&lt;/p&gt;&#xA;" OwnerUserId="1003" LastActivityDate="2015-01-23T23:16:54.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4935" PostTypeId="1" CreationDate="2015-01-24T15:15:27.777" Score="0" ViewCount="93" Body="&lt;p&gt;In the big data world there is lot of talk about implementing an &quot;active archive&quot;. I see cloudera talk about it a lot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value prop is that you move the low value (and less used) data from EDW to Hadoop and then save on expensive EDW storage by using Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in my company say we keep 10 years of data on EDW and say we don't use anything below 2 years very actively. So I move 8 years of data from EDW to Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can setup an Impala (or equivalent) product to query the 8 years of data as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the problem is how do you order, sort a query which requires some data form EDW and some data from Impala?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;because this kind of grouping, sorting ordering etc will have to be done in memory and the apps which queried the EDW were not written for operations and don't have the capacity as well to sort, group and process so much of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how are people implementing the Active Archive?&lt;/p&gt;&#xA;" OwnerUserId="6644" LastEditorUserId="6644" LastEditDate="2015-01-24T15:22:51.663" LastActivityDate="2015-01-24T16:51:33.300" Title="Questions on &quot;Active Archive&quot;" Tags="|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4936" PostTypeId="1" CreationDate="2015-01-24T15:52:28.727" Score="13" ViewCount="5859" Body="&lt;p&gt;What happens when we train a basic support vector machine (linear kernel and no soft-margin) on non-linearly separable data? The optimisation problem is not feasible, so what does the minimisation algorithm return?&lt;/p&gt;&#xA;" OwnerUserId="7944" LastEditorUserId="1003" LastEditDate="2015-01-25T09:39:27.340" LastActivityDate="2018-02-02T14:40:20.767" Title="What happens when we train a linear SVM on non-linearly separable data?" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4937" PostTypeId="2" ParentId="4935" CreationDate="2015-01-24T16:51:33.300" Score="1" Body="&lt;p&gt;I think the idea is that you have &lt;em&gt;all&lt;/em&gt; data in HDFS, and query it with Impala, and also keep &lt;em&gt;some&lt;/em&gt; small amount of data in your data warehouse. That is, keep all 10 years in Hadoop, and also 2 years in an EDW. The cost of also having the 2 years in Hadoop is small.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-01-24T16:51:33.300" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4938" PostTypeId="2" ParentId="4936" CreationDate="2015-01-25T01:29:53.470" Score="13" Body="&lt;p&gt;I think basic support vector machine means hard-margin SVM. So, let's review:&lt;/p&gt;&#xA;&lt;h3&gt;What is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;noreferrer&quot;&gt;Hard-Margin SVM&lt;/a&gt;&lt;/h3&gt;&#xA;&lt;p&gt;In short, we want to find a hyperplane with the largest margin which be able to separate all observations correctly in our training sample space.&lt;/p&gt;&#xA;&lt;h3&gt;The optimisation problem in hard-margin SVM&lt;/h3&gt;&#xA;&lt;p&gt;Given the above definition, what is the optimisation problem which we need to solve?&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The largest margin hyperplane: We want &lt;strong&gt;&lt;code&gt;max(margin)&lt;/code&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Be able to separate all observations correctly: We need to optimise &lt;strong&gt;&lt;code&gt;margin&lt;/code&gt;&lt;/strong&gt; and also satisfy the constraint: &lt;strong&gt;No in-sample errors&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h3&gt;What happens when we train a linear SVM on non-linearly separable data?&lt;/h3&gt;&#xA;&lt;p&gt;Back to your question, since you mentioned the training data set is not linearly separable, by using hard-margin SVM without feature transformations, &lt;strong&gt;it's impossible to find any hyperplane which satisfies &amp;quot;No in-sample errors&amp;quot;&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Normally, we solve SVM optimisation problem by Quadratic Programming, because it can do optimisation tasks with constraints. If you use Gradient Descent or other optimisation algorithms which without satisfying the constraints of hard-margin SVM, you should still get a result, but that's not a hard-margin SVM hyperplane.&lt;/p&gt;&#xA;&lt;p&gt;By the way, with non-linearly separable data, usually we choose&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;hard-margin SVM + feature transformations&lt;/li&gt;&#xA;&lt;li&gt;use soft-margin SVM directly (In practical, soft-margin SVM usually get good results)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1003" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-02T14:40:20.767" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4941" PostTypeId="2" ParentId="2682" CreationDate="2015-01-25T14:43:21.033" Score="7" Body="&lt;p&gt;Neo4j and Spark GraphX are meant for solving problem at different level and they are complimentary to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They can be connected by Neo4j's Mazerunner extension:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mazerunner is a Neo4j unmanaged extension and distributed graph&#xA;  processing platform that extends Neo4j to do big data graph processing&#xA;  jobs while persisting the results back to Neo4j.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Mazerunner uses a message broker to distribute graph processing jobs&#xA;  to Apache Spark's GraphX module. When an agent job is dispatched, a&#xA;  subgraph is exported from Neo4j and written to Apache Hadoop HDFS.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;After Neo4j exports a subgraph to HDFS, a separate Mazerunner service&#xA;  for Spark is notified to begin processing that data. The Mazerunner&#xA;  service will then start a distributed graph processing algorithm using&#xA;  Scala and Spark's GraphX module. The GraphX algorithm is serialized&#xA;  and dispatched to Apache Spark for processing.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Once the Apache Spark job completes, the results are written back to&#xA;  HDFS as a Key-Value list of property updates to be applied back to&#xA;  Neo4j.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Neo4j is then notified that a property update list is available from&#xA;  Apache Spark on HDFS. Neo4j batch imports the results and applies the&#xA;  updates back to the original graph.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Check out this tutorial to get an idea on how to combine the two:&#xA;&lt;a href=&quot;http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html&quot; rel=&quot;nofollow&quot;&gt;http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7950" LastEditorUserId="7950" LastEditDate="2015-02-02T07:31:51.050" LastActivityDate="2015-02-02T07:31:51.050" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4942" PostTypeId="1" CreationDate="2015-01-25T22:52:23.437" Score="17" ViewCount="11181" Body="&lt;p&gt;Due to various &lt;a href=&quot;http://en.wikipedia.org/wiki/Curse_of_dimensionality&quot;&gt;curses of dimensionality&lt;/a&gt;, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/heuristics that help deal with high-dimensional data effectively? For example,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do certain statistical/modeling methods perform well on high-dimensional datasets?&lt;/li&gt;&#xA;&lt;li&gt;Can we improve the performance of our predictive models on high-dimensional data by using certain (that define alternative notions of distance) or &lt;a href=&quot;http://en.wikipedia.org/wiki/Kernel_method&quot;&gt;kernels&lt;/a&gt; (that define alternative notions of dot product)?&lt;/li&gt;&#xA;&lt;li&gt;What are the most useful techniques of dimensionality reduction for high-dimensional data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5203" LastEditorUserId="5203" LastEditDate="2015-01-25T22:59:05.897" LastActivityDate="2017-02-14T16:31:51.417" Title="High-dimensional data: What are useful techniques to know?" Tags="|machine-learning|statistics|dimensionality-reduction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4943" PostTypeId="1" CreationDate="2015-01-25T23:03:02.297" Score="14" ViewCount="43031" Body="&lt;p&gt;How does varying the regularization parameter in an SVM change the decision boundary for a non-separable dataset? A visual answer and/or some commentary on the limiting behaviors (for large and small regularization) would be very helpful.&lt;/p&gt;&#xA;" OwnerUserId="5203" LastActivityDate="2021-01-10T16:58:56.070" Title="Intuition for the regularization parameter in SVM" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4944" PostTypeId="1" CreationDate="2015-01-25T23:09:22.863" Score="4" ViewCount="1715" Body="&lt;p&gt;What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the &quot;target&quot; class being much rarer than the &quot;background&quot; class)? For example,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should one first down-sample the majority/background class to reduce its frequency and then readjust the probabilities reported by the learning algorithm? How should one do the readjustment?&lt;/li&gt;&#xA;&lt;li&gt;Should one use different approaches for different learning algorithms, i.e., are there different techniques for dealing with imbalance in SVM, random forests, logistic regression, etc.?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5203" LastEditorUserId="97" LastEditDate="2015-01-26T16:43:42.273" LastActivityDate="2017-06-19T11:24:33.477" Title="How to learn a classifier from a dataset with high imbalance" Tags="|machine-learning|classification|class-imbalance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4945" PostTypeId="2" ParentId="4943" CreationDate="2015-01-26T01:37:05.640" Score="20" Body="&lt;p&gt;The regularization parameter (lambda) serves as a degree of importance that is given to misclassifications. SVM pose a quadratic optimization problem that looks for maximizing the margin between both classes and minimizing the amount of misclassifications. However, for non-separable problems, in order to find a solution, the miclassification constraint must be relaxed, and this is done by setting the mentioned &amp;quot;regularization&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;So, intuitively, as lambda grows larger the less the wrongly classified examples are allowed (or the highest the price the pay in the loss function). Then when lambda tends to infinite the solution tends to the hard-margin (allow no miss-classification). When lambda tends to 0 (without being 0) the more the miss-classifications are allowed.&lt;/p&gt;&#xA;&lt;p&gt;There is definitely a tradeoff between these two and normally smaller lambdas, but not too small, generalize well. Below are three examples for linear SVM classification (binary).&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/NbNCS.png&quot; alt=&quot;Linear SVM Lambda = 0.1&quot; /&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/xanGh.png&quot; alt=&quot;Linear SVM Lambda = 1&quot; /&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/R6fHT.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;For non-linear-kernel SVM the idea is the similar.&#xA;Given this, for higher values of lambda there is a higher possibility of overfitting, while for lower values of lambda there is higher possibilities of underfitting.&lt;/p&gt;&#xA;&lt;p&gt;The images below show the behavior for RBF Kernel, letting the sigma parameter fixed on 1 and trying lambda = 0.01 and lambda = 10&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/8KIpI.png&quot; alt=&quot;RBF Kernel SVM lambda = 0.01&quot; /&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/6Vmqt.png&quot; alt=&quot;RBF Kernel SVM lambda = 10&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;You can say the first figure where lambda is lower is more &amp;quot;relaxed&amp;quot; than the second figure where data is intended to be fitted more precisely.&lt;/p&gt;&#xA;&lt;p&gt;(Slides from Prof. Oriol Pujol. Universitat de Barcelona)&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="90523" LastEditDate="2021-01-10T16:58:56.070" LastActivityDate="2021-01-10T16:58:56.070" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="4946" PostTypeId="2" ParentId="4942" CreationDate="2015-01-26T08:00:14.630" Score="13" Body="&lt;p&gt;This is very &lt;strong&gt;broad&lt;/strong&gt; question, which I think it's impossible to cover &lt;em&gt;comprehensively&lt;/em&gt; in a single answer. Therefore, I think that it would be more beneficial to provide some pointers to relevant answers and/or resources. This is exactly what I will do by providing the following information and thoughts of mine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, I should mention the excellent and comprehensive &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cburges/papers/fnt_dimensionreduction.pdf&quot; rel=&quot;noreferrer&quot;&gt;tutorial on dimensionality reduction&lt;/a&gt; by Burges (2009) from Microsoft Research. He touches on &lt;em&gt;high-dimensional aspects&lt;/em&gt; of data frequently throughout the monograph. This work, referring to &lt;em&gt;dimensionality reduction&lt;/em&gt; as &lt;em&gt;dimension reduction&lt;/em&gt;, presents a theoretical introduction into &lt;strong&gt;the problem&lt;/strong&gt;, suggests a &lt;strong&gt;taxonomy&lt;/strong&gt; of dimensionality reduction methods, consisting of &lt;em&gt;projective methods&lt;/em&gt; and &lt;em&gt;manifold modeling methods&lt;/em&gt;, as well as provides an &lt;strong&gt;overview&lt;/strong&gt; of multiple methods in each category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;&lt;strong&gt;projective&lt;/strong&gt; pursuit&quot; methods reviewed include &lt;em&gt;independent component analysis (ICA)&lt;/em&gt;, &lt;em&gt;principal component analysis (PCA)&lt;/em&gt; and its variations, such as &lt;em&gt;kernel PCA&lt;/em&gt; and &lt;em&gt;probabilistic PCA&lt;/em&gt;, &lt;em&gt;canonical correlation analysis (CCA)&lt;/em&gt; and its &lt;em&gt;kernel CCA&lt;/em&gt; variation, &lt;em&gt;linear discriminant analysis (LDA)&lt;/em&gt;, &lt;em&gt;kernel dimension reduction (KDR)&lt;/em&gt; and some others. The &lt;strong&gt;manifold&lt;/strong&gt; methods reviewed include &lt;em&gt;multidimensional scaling (MDS)&lt;/em&gt; and its &lt;em&gt;landmark MDS&lt;/em&gt; variation, &lt;em&gt;Isomap&lt;/em&gt;, &lt;em&gt;Locally Linear Embedding&lt;/em&gt; and graphical methods, such as &lt;em&gt;Laplacian eigenmaps&lt;/em&gt; and &lt;em&gt;spectral clustering&lt;/em&gt;. I'm listing the most of the reviewed methods here in case, if the original publication is inaccessible for you, either &lt;em&gt;online&lt;/em&gt; (link above), or &lt;em&gt;offline&lt;/em&gt; (References).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;strong&gt;caveat&lt;/strong&gt; for the term &quot;comprehensive&quot; that I've applied to the above-mentioned work. While it is indeed rather comprehensive, this is relative, as some of the approaches to dimensionality reduction are not discussed in the monograph, in particular, the ones, focused on &lt;em&gt;unobservable (latent) variables&lt;/em&gt;. Some of them are mentioned, though, with references to another source - a book on dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I will briefly cover several narrower aspects of the topic in question by referring to my relevant or related answers. In regard to &lt;em&gt;nearest neighbors (NN)-type approaches&lt;/em&gt; to high-dimensional data, please see my answers &lt;a href=&quot;https://datascience.stackexchange.com/a/975/2452&quot;&gt;here&lt;/a&gt; (I especially recommend to check the paper #4 in my list). One of the effects of the &lt;em&gt;curse of dimensionality&lt;/em&gt; is that high-dimensional data is frequently &lt;strong&gt;sparse&lt;/strong&gt;. Considering this fact, I believe that my relevant answers &lt;a href=&quot;https://datascience.stackexchange.com/a/918/2452&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/a/130665/31372&quot;&gt;here&lt;/a&gt; on &lt;strong&gt;regression&lt;/strong&gt; and &lt;strong&gt;PCA&lt;/strong&gt; for &lt;em&gt;sparse and high-dimensional data&lt;/em&gt; might be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Burges, C. J. C. (2010). Dimension reduction: A guided tour. &lt;em&gt;Foundations and Trends® in Machine Learning, 2&lt;/em&gt;(4), 275-365. doi:10.1561/2200000002&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-02-14T16:31:51.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4947" PostTypeId="2" ParentId="4944" CreationDate="2015-01-26T09:39:41.527" Score="4" Body="&lt;p&gt;A common strategy for dealing with imbalance is to penalize harder the missclassifications that select the class with higher frequency. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a binary classification problem you could penalize by dividing 1/n where n is the number of examples of the opposite class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the following from Prof. Jordi Vitriá&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/fTosz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the loss function for structured output SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem you mention is common in object recognition and object classification in images where much more background images are used than images containing the object. A stronger case happens with exemplar SVM's where just a single image of the object is used.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="5143" LastEditDate="2015-01-26T09:52:10.093" LastActivityDate="2015-01-26T09:52:10.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4948" PostTypeId="2" ParentId="4944" CreationDate="2015-01-26T14:45:43.053" Score="2" Body="&lt;p&gt;Some good answers have already been posted at this site:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/1107/quick-guide-into-training-highly-imbalanced-data-sets&quot;&gt;Quick guide into training highly imbalanced data sets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And on Stats SE:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/81111/classification-problem-using-imbalanced-dataset&quot;&gt;https://stats.stackexchange.com/questions/81111/classification-problem-using-imbalanced-dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/16050/how-to-handle-data-imbalance-in-classification&quot;&gt;https://stats.stackexchange.com/questions/16050/how-to-handle-data-imbalance-in-classification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/60180/testing-classification-on-oversampled-imbalance-data?rq=1&quot;&gt;https://stats.stackexchange.com/questions/60180/testing-classification-on-oversampled-imbalance-data?rq=1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-01-26T14:45:43.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4949" PostTypeId="1" AcceptedAnswerId="4954" CreationDate="2015-01-26T15:45:51.317" Score="4" ViewCount="210" Body="&lt;p&gt;I hope this is a question appropriate for SO.&lt;/p&gt;&#xA;&lt;p&gt;The article in question: &lt;a href=&quot;http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;As far as I can tell, the only publicly available data from Google Search is through their Trends API.  The help page states that&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don't represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;However in the article, the author reports (absolute) &amp;quot;average monthly searches&amp;quot;.  The source is stated as:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;All monthly search numbers are approximate and derived from anonymous and aggregate web activity.&lt;/p&gt;&#xA;&lt;p&gt;Source: analysis of Google data by (author)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So, how did he get this &amp;quot;anonymous and aggregate web activity&amp;quot;?&lt;/p&gt;&#xA;" OwnerUserId="7961" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-03-24T15:07:35.520" Title="Where did this NY Times op-ed get his Google Search data?" Tags="|dataset|search|google|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4950" PostTypeId="1" CreationDate="2015-01-26T17:45:02.203" Score="5" ViewCount="2606" Body="&lt;p&gt;I am looking for a method to parse semi-structured textual data, i.e. data poorly formatted but usually having a visual structure of a matrix which may vary a lot in content and number of items in it, which may have headers or not, which may be interpreted sometimes column-wise or row-wise, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read about the WHISK information extraction paper : &lt;a href=&quot;https://homes.cs.washington.edu/~soderlan/soderland_ml99.pdf&quot;&gt;https://homes.cs.washington.edu/~soderlan/soderland_ml99.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but unfortunately, it is not very detailed and I have not been able to find a real-system implementing it, or even snippets of code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody have an idea where I can find such help? Or suggest an alternative approach which may be suited to my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for your reply!&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2016-04-02T15:51:32.827" Title="semi-structured text parsing using machine learning" Tags="|text-mining|information-retrieval|parsing|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4951" PostTypeId="1" AcceptedAnswerId="4993" CreationDate="2015-01-26T18:20:58.203" Score="1" ViewCount="592" Body="&lt;p&gt;I came across an SVM predictive model where the author used the probabilistic distribution value of the target variable as a feature in the feature set. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author built a model for each gesture of each player to guess which gesture would be played next. Calculating over 1000 games played the distribution may look like (20%, 10%, 70%). These numbers were then used as feature variables to predict the target variable for cross-fold validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that legitimate? That seems like cheating. I would think you would have to exclude the target variables from your test set when calculating features in order to not &quot;cheat&quot;.&lt;/p&gt;&#xA;" OwnerUserId="3430" LastEditorUserId="3430" LastEditDate="2015-01-30T12:42:45.187" LastActivityDate="2015-01-30T13:52:46.097" Title="Can distribution values of a target variable be used as features in cross-validation?" Tags="|accuracy|methods|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4952" PostTypeId="2" ParentId="4951" CreationDate="2015-01-26T19:24:51.783" Score="2" Body="&lt;p&gt;There is nothing necessarily wrong with this.  If you have no better information, then using past performance (i.e., prior probabilities) can work pretty well, particularly when your classes are very unevenly distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example methods using class priors are &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;Gaussian Maximum Likelihood&lt;/a&gt; classification and &lt;a href=&quot;http://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;Naïve Bayes&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[UPDATE]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you've added additional details to the question...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you are doing 10-fold cross-validation (holding out 10% of the data for validating each of the 10 subsets). If you use the entire data set to establish the priors (including the 10% of validation data), then yes, it is &quot;cheating&quot; since each of the 10 subset models uses information from the corresponding validation set (i.e., it is not truly a blind test). However, if the priors are recomputed for each fold using only the 90% of data used for that fold, then it is a &quot;fair&quot; validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of the effect of this &quot;cheating&quot; is if you have a single, extreme outlier in your data. Normally, with k-fold cross-validation, there would be one fold where the outlier is in the validation data and not the training data. When applying the corresponding classifier to the outlier during validation, it would likely perform poorly. However, if the training data for that fold included global statistics (from the entire data set), then the outlier would influence the statistics (priors) for that fold, potentially resulting in artificially favorable performance.&lt;/p&gt;&#xA;" OwnerUserId="964" LastEditorUserId="964" LastEditDate="2015-01-30T13:52:46.097" LastActivityDate="2015-01-30T13:52:46.097" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4953" PostTypeId="2" ParentId="4944" CreationDate="2015-01-26T21:09:16.850" Score="0" Body="&lt;p&gt;I would also suggest you to try an idea of 'Anomaly detection' using Gaussian distribution. In some cases it works really good - especially if you have a VERY skewed classes (say, among a million of examples only 10-20 are '1' (in class) and all the rest a 0's).  You may look up it in this video by prof. Andrew Ng.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.youtube.com/watch?v=h5iVXB9mczo&quot; rel=&quot;nofollow&quot;&gt;http://www.youtube.com/watch?v=h5iVXB9mczo&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or in text:&#xA;&lt;a href=&quot;http://www.holehouse.org/mlclass/15_Anomaly_Detection.html&quot; rel=&quot;nofollow&quot;&gt;http://www.holehouse.org/mlclass/15_Anomaly_Detection.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice, that this is not a classification problem, it is not using a classification algorithm.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-01-26T21:09:16.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4954" PostTypeId="2" ParentId="4949" CreationDate="2015-01-27T01:37:13.737" Score="3" Body="&lt;p&gt;Google AdWords.  That has absolute search volumes. &lt;/p&gt;&#xA;" OwnerUserId="7972" LastActivityDate="2015-01-27T01:37:13.737" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4955" PostTypeId="1" CreationDate="2015-01-27T10:31:10.233" Score="8" ViewCount="1802" Body="&lt;p&gt;My company provides managed services to a lot of its clients. Our customers typically uses following monitoring tools to monitor their servers/webapps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;OpsView&lt;/li&gt;&#xA;&lt;li&gt;Nagios&lt;/li&gt;&#xA;&lt;li&gt;Pingdom&lt;/li&gt;&#xA;&lt;li&gt;Custom shell scripts&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Whenever any issue is found, an alert mail comes to our Ops team so that they act upon rectifying the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we manage thousands of servers, our Ops teams' inbox is flooded with email alerts all the time. Even a single issue which has a cascading effect, can trigger 20-30 emails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what I want to do is to implement a system which will be able to extract important features out of an alert email - like server IP address, type of problem, severity of problem etc. and also classify the emails into proper category, like &lt;code&gt;CPU-Load-Customer1-Server2, MySQL-Replication-Customer2-DBServer3&lt;/code&gt; etc. We will then have a pre-defined set of debugging steps for each category, in order to help the Ops team to rectify the problem faster. Also, the feature extractor will provide input data to the team for a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have been able to train &lt;em&gt;NaiveBayesClassifier&lt;/em&gt; with supervised learning techniques i.e. labeled training data(cluster data), and able to classify new unseen emails into its proper cluster/category. As the emails are based on certain templates, the accuracy of the classifier is very high. But we also get alert emails from custom scripts, which may not follow the templates. So, instead of doing supervised learning, I want to try out unsupervised learning for the same. I am looking into &lt;em&gt;KMeans clustering&lt;/em&gt;. But again the problem is, we won't know the number of clusters beforehand. &lt;strong&gt;So, which algorithm will be best for this use case?&lt;/strong&gt; Right now I am using Python's TextBlob library for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, for feature extraction out of an alert email, I am looking into NLTK (&lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow&quot;&gt;http://www.nltk.org/book/ch07.html&lt;/a&gt;) library. I tried it out, but it seems to work on proper English paragraphs/texts well, however, for alert emails, it extracted a lot of unnecessary features. &lt;strong&gt;Is there already any existing solution for the same? If not, what will be the best way to implement the same? Which library, which algorithm?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS: I am not a Data Scientist.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample emails:&lt;/strong&gt;    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.10.0.100  Host Group Hierarchy: Opsview &amp;gt; Customer1  - BIG C &amp;gt; Customer1_PROD  State: CRITICAL  Date &amp;amp; Time: Sat Oct 4 07:02:06 UTC 2014    Additional Information:     CRITICAL - load average: 41.46, 40.69, 37.91&#xA;RECOVERY: OK - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.1.1.100  Host Group Hierarchy: Opsview &amp;gt; Customer1  - BIG C &amp;gt; Customer1_PROD  State: OK  Date &amp;amp; Time: Sat Oct 4 07:52:05 UTC 2014    Additional Information:     OK - load average: 0.36, 0.23, 4.83&#xA;PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.100.10.10  Host Group Hierarchy: Opsview &amp;gt; Customer1  - BIG C &amp;gt; Customer1_PROD  State: CRITICAL  Date &amp;amp; Time: Sat Oct 4 09:29:05 UTC 2014    Additional Information:     CRITICAL - load average: 29.59, 26.50, 18.49&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Classifier code:(format of csv - email, &amp;lt;disk/cpu/memory/mysql&amp;gt;)&lt;/strong&gt;    &lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from textblob import TextBlob&#xA;from textblob.classifiers import NaiveBayesClassifier&#xA;import csv&#xA;train = []&#xA;with open('cpu.txt', 'r') as csvfile:&#xA;    reader = csv.reader(csvfile, delimiter=',', quotechar='&quot;')&#xA;    for row in reader:&#xA;        tup = unicode(row[0], &quot;ISO-8859-1&quot;), row[1]&#xA;        train.append(tup)&#xA;// this can be done in a loop, but for the time being let it be&#xA;with open('memory.txt', 'r') as csvfile:&#xA;    reader = csv.reader(csvfile, delimiter=',', quotechar='&quot;')&#xA;    for row in reader:&#xA;        tup = unicode(row[0], &quot;ISO-8859-1&quot;), row[1]&#xA;        train.append(tup)&#xA;&#xA;with open('disk.txt', 'r') as csvfile:&#xA;    reader = csv.reader(csvfile, delimiter=',', quotechar='&quot;')&#xA;    for row in reader:&#xA;        tup = unicode(row[0], &quot;ISO-8859-1&quot;), row[1]&#xA;        train.append(tup)&#xA;&#xA;with open('mysql.txt', 'r') as csvfile:&#xA;    reader = csv.reader(csvfile, delimiter=',', quotechar='&quot;')&#xA;    for row in reader:&#xA;        tup = unicode(row[0], &quot;ISO-8859-1&quot;), row[1]&#xA;        train.append(tup)&#xA;&#xA;cl = NaiveBayesClassifier(train)&#xA;cl.classify(email)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature extractor code taken from:&lt;/strong&gt; &lt;a href=&quot;https://gist.github.com/shlomibabluki/5539628&quot; rel=&quot;nofollow&quot;&gt;https://gist.github.com/shlomibabluki/5539628&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if any more information is required here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="7979" LastEditorUserId="10337" LastEditDate="2015-06-28T08:31:28.390" LastActivityDate="2015-06-28T08:31:28.390" Title="How to extract features and classify alert emails coming from monitoring tools into proper category?" Tags="|machine-learning|classification|clustering|feature-extraction|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="4956" PostTypeId="2" ParentId="155" CreationDate="2015-01-27T11:15:04.250" Score="12" Body="&lt;p&gt;To add to a possibly never ending list:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as mentioned by cyndd, there is &lt;a href=&quot;http://www.wikidata.org/wiki/Wikidata:Main_Page&quot;&gt;Wikidata&lt;/a&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and for curated structured knowledge, &lt;a href=&quot;http://www.wolframalpha.com/&quot;&gt;Wolfram Alpha&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-01-27T11:15:04.250" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="4957" PostTypeId="1" AcceptedAnswerId="4966" CreationDate="2015-01-27T15:13:09.157" Score="5" ViewCount="19854" Body="&lt;p&gt;Do you know of any machine learning add-ins that I could use within Excel? For example I would like to be able to select a range of data and use that for training purposes and then use another sheet for getting the results of different learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="7982" LastActivityDate="2020-02-19T03:55:45.710" Title="Machine learning toolkit for Excel" Tags="|machine-learning|neural-network|" AnswerCount="9" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="4958" PostTypeId="2" ParentId="4875" CreationDate="2015-01-27T22:01:26.237" Score="0" Body="&lt;p&gt;Recently I am working at a similar analysis. I wrote some functions to test any possible combinations between variables, however it is specifically used for my own data set which definitely is different from your one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a fairly small job so I can not say any package dealing with such tests. And you have already worked out some combinations. Just keep going for a ideal function, maybe will be done in a couple of days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I add a link here, which partially answers your question and code is included: &lt;a href=&quot;https://stats.stackexchange.com/questions/4040/r-compute-correlation-by-group&quot;&gt;https://stats.stackexchange.com/questions/4040/r-compute-correlation-by-group&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="7989" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-01-28T03:16:42.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4959" PostTypeId="2" ParentId="4875" CreationDate="2015-01-28T03:55:58.367" Score="4" Body="&lt;p&gt;The idea you have in mind is called &quot;feature selection&quot; or &quot;attribute selection&quot;. The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant because you're expected to use an algorithm or statistical method that is suitable for your requirements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for feature selection methods, there are several options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Find the subset of features that achieves better performance (usually in cross validation)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find the subset of features that correlates highly with the target variable and low with each other (although other criteria can be used)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use an algorithm that includes a built-in feature selection mechanism (e.g. decision trees, hierarchical bayesian methods)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Furthermore, there are several methods aimed at obtaining a good compromise between a thorough search and a reasonable time execution (e.g. best first, steepest ascent search, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://stats.stackexchange.com/questions/56092/feature-selection-packages-in-r&quot;&gt;question&lt;/a&gt; in particular provides very good suggestions for R packages. &lt;/p&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-01-28T03:55:58.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4960" PostTypeId="2" ParentId="4957" CreationDate="2015-01-28T04:34:30.777" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; can import CSV files, and allows you to choose which columns and rows you want to use in your analysis.  It's not an &quot;add-in&quot; for Excel per-se, but it might work for you.&lt;/p&gt;&#xA;" OwnerUserId="7961" LastActivityDate="2015-01-28T04:34:30.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4961" PostTypeId="2" ParentId="4957" CreationDate="2015-01-28T04:42:19.780" Score="3" Body="&lt;p&gt;First of all, let me tell you that Excel shouldn't be used for machine learning or any data analysis complicated enough that you wouldn't be comfortable doing it on paper. Why? Here is a list of resources to tell you why:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a-spreadsheet-for-important-work-i-mean-it/&quot; rel=&quot;nofollow&quot;&gt;You shouldn’t use a spreadsheet for important work (I mean it)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://randyzwitch.com/excel-destroys-data/&quot; rel=&quot;nofollow&quot;&gt;Destroy Your Data Using Excel With This One Weird Trick!&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://people.umass.edu/evagold/excel.html&quot; rel=&quot;nofollow&quot;&gt;Using Excel for Statistical Data Analysis - Caveats&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/ExcelProblems&quot; rel=&quot;nofollow&quot;&gt;Problems with Excel&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.burns-stat.com/documents/tutorials/spreadsheet-addiction/&quot; rel=&quot;nofollow&quot;&gt;Spreadsheet Addiction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, if you really really want to do heavy calculations without exporting your data, I suggest using &lt;a href=&quot;http://xlwings.org/&quot; rel=&quot;nofollow&quot;&gt;xlwings&lt;/a&gt;. Basically, this allows two-way communication between Excel and Python. Watch the video in the homepage for a quick introduction. In this way, you would be able to use numpy, pandas and scikit-learn (or other machine learning library that you may prefer) without exporting your data first.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="4621" LastEditDate="2015-01-28T17:00:58.453" LastActivityDate="2015-01-28T17:00:58.453" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="4962" PostTypeId="2" ParentId="4957" CreationDate="2015-01-28T05:53:28.083" Score="2" Body="&lt;p&gt;Nobody does serious machine learning in Excel; that's not what it's for. Fortunately, you can directly import Excel files into better platforms like python. In particular, there's a great package called &lt;code&gt;pandas&lt;/code&gt;, which makes work very pleasant. &lt;a href=&quot;https://www.youtube.com/watch?v=_JZFSFR6Yeo&quot; rel=&quot;nofollow&quot;&gt;Here's a demo&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-01-28T05:53:28.083" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="4963" PostTypeId="1" AcceptedAnswerId="4964" CreationDate="2015-01-28T05:57:56.090" Score="2" ViewCount="1017" Body="&lt;p&gt;I found that Apache-Spark very powerful in Big-Data processing. but I want to know about Dryad (Microsoft) benefits. Is there any advantage for this framework than Spark?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why we must use Dryad instead of Spark?&lt;/p&gt;&#xA;" OwnerUserId="7977" LastEditorUserId="11097" LastEditDate="2016-04-11T05:08:41.997" LastActivityDate="2016-04-11T05:08:41.997" Title="What is advantage of using Dryad instead of Spark?" Tags="|bigdata|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4964" PostTypeId="2" ParentId="4963" CreationDate="2015-01-28T06:41:28.650" Score="1" Body="&lt;p&gt;Dryad is an academic project, whereas Spark is widely deployed in production, and now has a company behind it for support. Just focus on Spark.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-01-28T06:41:28.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4965" PostTypeId="2" ParentId="4875" CreationDate="2015-01-28T09:24:14.097" Score="2" Body="&lt;p&gt;I would suggest to consider using &lt;em&gt;latent variable modeling (LVM)&lt;/em&gt; or similar &lt;em&gt;structural equation modeling (SEM)&lt;/em&gt; as an approach to this problem. Using this approach is based on recognizing and analyzing &lt;strong&gt;latent variables&lt;/strong&gt; - constructs (&lt;em&gt;factors&lt;/em&gt;), measured not directly, but through sets of measured variables (&lt;em&gt;indicators&lt;/em&gt;). Note that a closely related term &lt;em&gt;latent feature&lt;/em&gt; is frequently used within the &lt;em&gt;machine learning&lt;/em&gt; domain. It seems to me that latent variables resemble what you call &quot;combinations/subsets/segments of the IVs&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By hypothesizing - usually, based on theory or domain knowledge - the &lt;strong&gt;latent structure&lt;/strong&gt; of factors, LVM or SEM are able to automatically confirm or decline those hypotheses. This is done by using a combination of &lt;em&gt;exploratory factor analysis (EFA)&lt;/em&gt; and &lt;em&gt;confirmatory factor analysis (CFA)&lt;/em&gt; (see my answer ). While EFA is frequently performed independently (and maybe that's enough for your purposes), doing it along with CFA represents a large part of LVM/SEM methodology, which is usually completed by performing &lt;em&gt;path analysis&lt;/em&gt;, which is concerned about relationships between latent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;R&lt;/code&gt; &lt;em&gt;ecosystem&lt;/em&gt; offers a variety of &lt;strong&gt;packages&lt;/strong&gt; for performing LVM/SEM in its entirety or for performing EFA, CFA and path analysis. The most popular ones for EFA are &lt;code&gt;psych&lt;/code&gt;, &lt;code&gt;GPArotation&lt;/code&gt; and &lt;code&gt;Hmisc&lt;/code&gt;. The most popular packages for CFA, path analysis and LVM are &lt;code&gt;sem&lt;/code&gt; (the first R package for SEM), &lt;code&gt;lavaan&lt;/code&gt;, &lt;code&gt;OpenMx&lt;/code&gt;, &lt;code&gt;semPLS&lt;/code&gt;, &lt;code&gt;plspm&lt;/code&gt;. Various supplementary SEM-focused packages are &lt;a href=&quot;http://pairach.com/2011/08/13/r-packages-for-structural-equation-model&quot; rel=&quot;nofollow&quot;&gt;also available&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-01-28T09:24:14.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4966" PostTypeId="2" ParentId="4957" CreationDate="2015-01-28T10:33:38.880" Score="11" Body="&lt;p&gt;As far as I know, currently there are not that many projects and products that allow you to perform serious &lt;em&gt;machine learning (ML)&lt;/em&gt; work from within Excel. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the situation seems to be changing rapidly due to active Microsoft's efforts in popularizing its ML cloud platform &lt;em&gt;Azure ML&lt;/em&gt; (along with &lt;em&gt;ML Studio&lt;/em&gt;). The &lt;a href=&quot;http://blogs.microsoft.com/blog/2015/01/23/microsoft-acquire-revolution-analytics-help-customers-find-big-data-value-advanced-statistical-analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;recent acquisition&lt;/a&gt; of R-focused company Revolution Analytics by Microsoft (which appears to me as more of &lt;em&gt;acqui-hiring&lt;/em&gt; to a large extent) is an example of the company's aggressive data science &lt;strong&gt;market strategy&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regard to &lt;em&gt;ML toolkits for Excel&lt;/em&gt;, as a confirmation that we should expect most Excel-enabled ML projects and products to be &lt;strong&gt;Azure ML-focused&lt;/strong&gt;, consider the following two &lt;em&gt;projects&lt;/em&gt; (the latter is an open source):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Excel DataScope&lt;/strong&gt; (Microsoft Research): &lt;a href=&quot;https://www.microsoft.com/en-us/research/video/excel-datascope-overview/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.microsoft.com/en-us/research/video/excel-datascope-overview/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Azure ML Excel Add-In&lt;/strong&gt; (seems to be Microsoft sponsored): &lt;a href=&quot;https://azuremlexcel.codeplex.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://azuremlexcel.codeplex.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="29575" LastEditDate="2018-09-07T03:24:23.073" LastActivityDate="2018-09-07T03:24:23.073" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="4967" PostTypeId="1" CreationDate="2015-01-28T13:42:15.430" Score="5" ViewCount="1037" Body="&lt;p&gt;Let's say I'm trying to predict a person's electricity consumption, using the time of day as a predictor (hours 00-23), and further assume I have a hefty but finite amount of historical measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm trying to set up a linear model akin to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$power.used = \alpha* hr.of.day + \beta * temperature$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem: using the &lt;span class=&quot;math-container&quot;&gt;$hr.of.day$&lt;/span&gt; as a numerical value is a very bad idea for many reasons, the fact that 23 and 0 are actually quite close values is one problem that can be solved with a simple transformation [1]. The fact that electrical consumption is often bi-modal is another problem which isn't solved by a simple transformation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A possible solution that works rather well is to treat the time of day as a categorical variable. That does the trick, but it suffers from a significant drawback in that there's no information sharing between neighbouring hours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I'm asking is this: does anyone know of a &quot;soft&quot; version of categorical values? I'm suggesting something quite loosely defined: Ideally I would have some parameter alpha that reduces the regression to numerical regression where &lt;span class=&quot;math-container&quot;&gt;$\alpha = 1$&lt;/span&gt; and reduces to categorical regression where &lt;span class=&quot;math-container&quot;&gt;$\alpha = 0$&lt;/span&gt;, and behaves &quot;in between&quot; if it's some other number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now the only answer I can think of is to alter the weights in the regression in such a way that they tend towards zero the further away the quasi-categorical value is from the desired value. Surely there are other approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &#xA;introduce the hour variable as two new variables: &lt;span class=&quot;math-container&quot;&gt;$cos(time.of.day/24)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$sin(time.of.day/24)$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="7999" LastEditorUserId="71218" LastEditDate="2019-04-24T21:18:29.377" LastActivityDate="2019-04-24T21:18:29.377" Title="Quasi-categorical variables - any ideas?" Tags="|time-series|regression|categorical-data|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="4968" PostTypeId="1" AcceptedAnswerId="4976" CreationDate="2015-01-28T17:40:32.450" Score="3" ViewCount="3639" Body="&lt;p&gt;Both Apache-Spark and Apache-Flink projects claim pretty much similar capabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is the difference between these projects. Is there any advantage in either Spark or Flink?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="7977" LastActivityDate="2018-09-27T05:00:10.680" Title="What are the differences between Apache Spark and Apache Flink?" Tags="|bigdata|" AnswerCount="2" CommentCount="0" ClosedDate="2018-09-27T18:28:26.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="4969" PostTypeId="2" ParentId="305" CreationDate="2015-01-28T18:42:06.763" Score="5" Body="&lt;p&gt;Current size limit for Amazon Redshift is 128 nodes or 2 PBs of compressed data. Might be circa 6PB uncompressed though mileage varies for compression. You can always let us know if you need more. anurag@aws (I run Amazon Redshift and Amazon EMR)&lt;/p&gt;&#xA;" OwnerUserId="8003" LastActivityDate="2015-01-28T18:42:06.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4971" PostTypeId="1" CreationDate="2015-01-28T19:24:07.543" Score="4" ViewCount="462" Body="&lt;p&gt;Twitter is a popular source of data for many applications, especially involving sentiment analysis and the like.  I have some things I'm interested in doing with Twitter data, but here's the issue:  To get all Tweets, you have to get special permission from Twitter (which, as I understand it, is never granted) or pay big bucks to Gnip or the like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OTOH, &lt;a href=&quot;https://dev.twitter.com/streaming/firehose&quot; rel=&quot;nofollow&quot;&gt;Twitter's API documentation&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Few applications require this level of access. Creative use of a combination of other resources and various access levels can satisfy nearly every application use case.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the filter api with keyword tracking seems like something that would be a big part of this, but you obviously can't enumerate every keyword.  Using a User stream on many User accounts that follow a lot of people might be an option as well, and I'm not sure if it makes sense to think about using the search API in addition.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here's the question &quot;What combination of other resources and access levels is the best way to get the maximum amount of data from Twitter&quot;?&lt;/p&gt;&#xA;" OwnerUserId="6554" LastEditorUserId="21" LastEditDate="2015-07-31T08:33:37.290" LastActivityDate="2015-09-15T04:38:47.217" Title="How to access maximum volume of tweets using Twitter Streaming API, without firehose access?" Tags="|software-development|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4972" PostTypeId="2" ParentId="4967" CreationDate="2015-01-28T21:56:23.980" Score="6" Body="&lt;p&gt;I would suggest you to use the idea of so-called 'fuzzy clustering', where you put each of your hours of the day value into several clusters at the same time. Details in paper: &lt;a href=&quot;http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/cmeans.html&quot; rel=&quot;noreferrer&quot;&gt;http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/cmeans.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is trivial:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You decide how many clusters you want to have. For example, 4 (so you divide your day hours into 4 cathegories). Instead of computing just 1 number (which defines cluster membership) for each of your day hours you compute 4 numbers which represent the degree of membership to each of 4 clusters. So for example if you 4 clusters will contain periods 12 AM-6 AM, 6 AM- 12 PM, 12 PM - 6 PM and 6 PM - 12 AM then you would replace for example 4 AM hour in original data with vector of 4 numbers, first one is the biggest, second is smaller, third one is the smallest one etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you could use these 4 numbers in your model to fit a regression line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you want you could use 24 clusters and in such case each your day of hour would have a high 'relation' with nearby hours and almost 0 with the distant hours.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastEditorUserId="7969" LastEditDate="2015-01-28T22:02:09.333" LastActivityDate="2015-01-28T22:02:09.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="4973" PostTypeId="2" ParentId="155" CreationDate="2015-01-28T22:27:18.587" Score="16" Body="&lt;p&gt;Did you know about the PUMA Benchmarks and dataset downloads? &#xA;&lt;a href=&quot;https://sites.google.com/site/farazahmad/pumadatasets&quot;&gt;https://sites.google.com/site/farazahmad/pumadatasets&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does include the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;TeraSort&lt;/li&gt;&#xA;&lt;li&gt;Wikipedia&lt;/li&gt;&#xA;&lt;li&gt;List item&lt;/li&gt;&#xA;&lt;li&gt;Self-Join&lt;/li&gt;&#xA;&lt;li&gt;Adjacency-List&lt;/li&gt;&#xA;&lt;li&gt;Movies-database&lt;/li&gt;&#xA;&lt;li&gt;Ranked-Inverted-Index &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2699" LastActivityDate="2015-01-28T22:27:18.587" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="4974" PostTypeId="1" CreationDate="2015-01-28T23:19:37.187" Score="6" ViewCount="3332" Body="&lt;p&gt;I have an array of edges and weights:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[['a', 'b', 4],&#xA; ['a', 'c', 3],&#xA; ['c', 'a', 2], &#xA; ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have about 100,000 edges and weights are between 1 and 700, most around 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of using Markov Cluster Algorithm however wanted to reach out to see if this is the best to use. What about Affinity Propagation? In either case, what is the workflow? Do you typically have a way to measure how well clustered the results. Is there an equivalent to a silhouette score? Is there a way to visualize the clusters?&lt;/p&gt;&#xA;" OwnerUserId="8009" LastActivityDate="2018-01-15T12:17:37.263" Title="Partitioning Weighted Undirected Graph" Tags="|clustering|graphs|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4975" PostTypeId="2" ParentId="4974" CreationDate="2015-01-29T00:57:39.603" Score="3" Body="&lt;p&gt;Even a simple Internet search reveals numerous papers on &lt;em&gt;graph clustering&lt;/em&gt; approaches and algorithms. &lt;a href=&quot;http://dollar.biz.uiowa.edu/~street/graphClustering.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper&lt;/a&gt; is most likely the best starting point, as it presents a rather &lt;strong&gt;comprehensive overview&lt;/strong&gt; of the topic in terms of the problem as well as &lt;em&gt;approaches&lt;/em&gt;, &lt;em&gt;methods&lt;/em&gt; and &lt;em&gt;algorithms&lt;/em&gt; for solutions. The rest you can find easily via online search. In regard to &lt;em&gt;graph clustering visualization&lt;/em&gt;, I recommend you to check &lt;a href=&quot;https://datascience.stackexchange.com/a/814/2452&quot;&gt;my relevant answer&lt;/a&gt; - I'm pretty sure that the tools I reference there are able to visualize graph clusters as well.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-01-29T00:57:39.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4976" PostTypeId="2" ParentId="4968" CreationDate="2015-01-29T04:19:16.007" Score="3" Body="&lt;p&gt;Flink is the Apache renaming of the &lt;a href=&quot;http://stratosphere.eu/&quot; rel=&quot;nofollow&quot;&gt;Stratosphere project from several universities in Berlin&lt;/a&gt;. It doesn't have the same industrial foothold and momentum that the Spark project has, but it seems nice, and more mature than, say, Dryad. I'd say it's worth investigating, at least for personal or academic use, but for industrial deployment I'd still prefer Spark, which at this point is battle tested. For a more technical discussion, see &lt;a href=&quot;https://www.quora.com/Are-Spark-and-Stratosphere-competitors-Do-they-cover-the-same-set-of-use-cases&quot; rel=&quot;nofollow&quot;&gt;this Quora post by committers on both projects&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-01-29T04:19:16.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4977" PostTypeId="1" AcceptedAnswerId="4982" CreationDate="2015-01-29T07:32:19.207" Score="3" ViewCount="3682" Body="&lt;p&gt;I found that Apache-Storm, Apache-Spark, Apache-Flink and TIBCO StreamBase are some powerful frameworks for stream processing. but I don't know which one of them has the best performance and capabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know Apache-Spark and Apache-Flink are two general purpose frameworks that support stream processing. but Apache-Storm and TIBCO StreamBase are built for stream processing specially. Is there any considerable advantage between these frameworks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="7977" LastActivityDate="2018-11-15T01:45:56.017" Title="What is the best Big-Data framework for stream processing?" Tags="|bigdata|" AnswerCount="8" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4978" PostTypeId="1" AcceptedAnswerId="4981" CreationDate="2015-01-29T09:04:56.190" Score="0" ViewCount="1158" Body="&lt;p&gt;When considering Support Vector Machine, in an take in multiple inputs. Can each of these inputs be a vector??&#xA;What i am trying to say is, can the input be a 2 dimensional vector??&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2015-01-29T12:42:05.757" Title="Can 2 dimensional input be applied to SVM?" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="1" ClosedDate="2015-01-31T11:32:43.967" ContentLicense="CC BY-SA 3.0" />
  <row Id="4979" PostTypeId="1" CreationDate="2015-01-29T11:35:07.657" Score="0" ViewCount="76" Body="&lt;p&gt;I need to simulate for an academical project how the traffic fluxes (input/output with respect to a monitored area, measured in number of cars) of a city area evolves in correspondence of an event (i.e. the opening of a restricted traffic area to decongest the traffic).&#xA;I have some simulated sensors that provide the data: I was thinking to use a combination of a fuzzy system (to assign a membership function to each type of data, e.g. PM10 value and CO2 value) and a markov process: I would need to modify the probability to decrement the number of car in the monitored area (simulating that a car is going out the congested area, towards the new opened area) basing on decisions made by means of a fuzzy system.&#xA;So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It is a good way to interpret the problem or there are better ideas that I have not taken into account yet?&lt;/li&gt;&#xA;&lt;li&gt;How to implement such a combination of markov chain and fuzzy systems in matlab?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="6559" LastEditorUserId="97" LastEditDate="2015-02-04T15:46:20.233" LastActivityDate="2015-05-05T17:43:39.597" Title="Matlab simulation through FIS and Markov Process" Tags="|markov-process|matlab|simulation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4980" PostTypeId="1" CreationDate="2015-01-29T12:13:01.677" Score="12" ViewCount="5883" Body="&lt;p&gt;I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ACE corpus. I have searched on the web a lot but I could not find any way that can be used to train NLTK's NER. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can provide me with any link/article/blog etc which can direct me to Training Datasets Format used in training NLTK's NER so I can prepare my Datasets on that particular format. And if I am directed to any link/article/blog etc which can help me TRAIN NLTK's NER for my own data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a question widely searched and least answered. Might be helpful for someone in the future whose working with NER.&lt;/p&gt;&#xA;" OwnerUserId="8016" LastEditorUserId="8016" LastEditDate="2015-01-30T04:30:58.490" LastActivityDate="2017-11-27T13:00:07.393" Title="Help regarding NER in NLTK" Tags="|machine-learning|python|nlp|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4981" PostTypeId="2" ParentId="4978" CreationDate="2015-01-29T12:42:05.757" Score="1" Body="&lt;p&gt;If I understand your question correctly. Yes, SVM can take multiple inputs. My suggestion for handling a vector as a feature would be to expand it out. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x0 = (1,2)       x0 = 1&#xA;x1 = .4  -----&amp;gt;  x1 = 2&#xA;x2 = 0           x2 = .4&#xA;                 x3 = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this does not capture all of the characteristics of the vector that are important, then you may want to add other features (like magnitude of the vector) as well. &lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2015-01-29T12:42:05.757" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4982" PostTypeId="2" ParentId="4977" CreationDate="2015-01-29T12:48:41.620" Score="2" Body="&lt;p&gt;It really depends on what you are looking to do. I love Apache Spark, but Storm has some history. I am sure as the streaming capability in Spark is built out that it will become a competitive solution. However, until Spark has some heavy hitting users (for streaming) there will remain unknown bugs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also consider the community. Spark has a great community. I am not sure the level of the Storm community as I am usually the one receiving the data not handling the ingest. I can say we have used Storm on projects and I have been impressed with the real-time analysis and volumes of streaming data.&lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2015-01-29T12:48:41.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4983" PostTypeId="2" ParentId="4951" CreationDate="2015-01-29T12:51:45.280" Score="0" Body="&lt;p&gt;I agree that there is nothing wrong with using these type of features.  I have used for inter-arrival times for example in modeling work.  I have noticed however that many of these kind of features have &quot;interesting&quot; covariance relationships with each other, so you have to be really careful about using multiple distribution features in a model.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2015-01-29T12:51:45.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4984" PostTypeId="2" ParentId="41" CreationDate="2015-01-29T12:58:28.257" Score="7" Body="&lt;p&gt;R is great for a lot of analysis. As mentioned about, there are newer adaptations for big data like MapR, RHadoop, and scalable versions of RStudio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if your concern is libraries, keep your eye on Spark. Spark was created for big data and is MUCH faster than Hadoop alone. It has vastly growing machine learning, SQL, streaming, and graph libraries. Thus allowing much if not all of the analysis to be done within the framework (with multiple language APIs, I prefer Scala) without having to shuffle between languages/tools.&lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2015-01-29T12:58:28.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4985" PostTypeId="1" AcceptedAnswerId="5011" CreationDate="2015-01-29T13:54:24.940" Score="3" ViewCount="1211" Body="&lt;p&gt;I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few versions of the training set (as a result of re-sampling, removing observations, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to perform pre-processing of the data, that is to scale, center and impute (not much imputation though) values. This is the point where I've started to get confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been taught that you should always pre-process the test set in the same way you've pre-processed the training set, that is (for scaling and centering) to measure the mean and standard deviation on the training set and apply those values to the test set. This seems reasonably to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what to do in case when you have shrinked/resampled training set? Should one focus on characteristics of the data that is actually feeding the model (that is what would 'train' function in R's caret package suggest, as you can put the pre-processing object in there directly) and apply these to the test set, or maybe one should capture the real characteristics of the data (from the whole untouched training set) and apply these? If the second option is better, maybe it would be worth it to capture the characteristics of the data by merging the training and test data together just for pre-processing step to get as accurate estimates as possible (I've actually never heard of anyone doing that though)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I can simply test some of the approaches specified here, and I surely will, but are there any suggestions based on theory or your intuition/experience on how to tackle this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have one additional and optional question. Does it make sense to center but NOT scale the data (or the other way around) in any case? Can anyone present any example where that approach would be reasonable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance.&lt;/p&gt;&#xA;" OwnerUserId="8017" LastActivityDate="2015-05-02T03:55:45.303" Title="Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?" Tags="|machine-learning|data-mining|dataset|processing|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4986" PostTypeId="1" CreationDate="2015-01-29T17:21:07.070" Score="2" ViewCount="657" Body="&lt;p&gt;I have a weekly dataset and I have to normalize this data. &#xA;Data is something like this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. week   50&#xA;2. week   51&#xA;3. week   50&#xA;4. week   54&#xA;5. week   150&#xA;6. week   155&#xA;7. week   ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The important thing is, the difference between week 3 and week 4 (50-54) is not same with week 5 and week 6. And also there is a huge different between week 4 and week 5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can i handle all of this things ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the standard normalization functions(for example scikit normalization) can do it for me and should I normalize this data 0-1 or -1 to 1 ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html&quot; rel=&quot;nofollow&quot;&gt;Sklearn normalization page&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NOTE&lt;/strong&gt; I am working with python and generally scikit-learn library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="16218" LastActivityDate="2015-04-29T19:25:35.863" Title="Normalize weekly data - Python" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4987" PostTypeId="2" ParentId="4986" CreationDate="2015-01-29T17:28:51.800" Score="1" Body="&lt;p&gt;I would find the unit variance of the all the weeks and then divide by that. Scikit can do this for you using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html&quot; rel=&quot;nofollow&quot;&gt;scale&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2015-01-29T17:28:51.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4988" PostTypeId="2" ParentId="4876" CreationDate="2015-01-29T20:36:25.567" Score="3" Body="&lt;p&gt;This looks a well structured dataset. You can read more about database design in &lt;a href=&quot;http://en.wikipedia.org/wiki/Database_design#Normalization&quot; rel=&quot;nofollow&quot;&gt;this section of wikipedia&lt;/a&gt;. Your data are well structured so querying is easy. As Jake C says, you'll want to transform it for specific tasks. Packages like dplr and reshape2 are excellent for this. You could also consider writing your data to a specific database. This is particularly useful if your dataset is so large that R runs out of RAM. I've written an example with SQLite here: &lt;a href=&quot;https://scottishsnow.wordpress.com/2014/08/14/writing-to-a-database-r-and-sqlite/&quot; rel=&quot;nofollow&quot;&gt;https://scottishsnow.wordpress.com/2014/08/14/writing-to-a-database-r-and-sqlite/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8021" LastActivityDate="2015-01-29T20:36:25.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4989" PostTypeId="2" ParentId="41" CreationDate="2015-01-29T21:03:27.097" Score="4" Body="&lt;p&gt;As other answers have noted, R can be used along with Hadoop and other distributed computing platforms to scale it up to the &quot;Big Data&quot; level.  However, if you're not wedded to R specifically, but are willing to use an &quot;R-like&quot; environment, &lt;a href=&quot;http://www.incanter.org&quot; rel=&quot;nofollow&quot;&gt;Incanter&lt;/a&gt; is a project that might work well for you, as it is native to the JVM (based on Clojure) and doesn't have the &quot;impedance mismatch&quot; between itself and Hadop that R has.  That is to say, from Incanter, you can invoke Java native Hadoop / HDFS APIs without needing to go through a JNI bridge or anything.&lt;/p&gt;&#xA;" OwnerUserId="6554" LastActivityDate="2015-01-29T21:03:27.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4990" PostTypeId="2" ParentId="4951" CreationDate="2015-01-29T21:31:50.280" Score="0" Body="&lt;p&gt;As bogatron and Paul already said, there is nothing wrong with using the prediction from one classifier as a feature in another classifier. Actually, so-called &quot;Cascading classifiers&quot; work that way. From &lt;a href=&quot;http://en.wikipedia.org/wiki/Cascading_classifiers&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cascading is a particular case of ensemble learning based on the concatenation of several classifiers, using all information collected from the output from a given classifier as additional information for the next classifier in the cascade.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This can be helpful not only to inform posterior classifiers using new features but also as an optimization measure. In the Viola-Jones object detection framework, a set of weak classifiers is used sequentially in order to reduce the amount of computation in the object recognition task. If one of the weak classifiers fails to recognize an object of interest, others classifiers don't need to be computed.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2015-01-29T21:31:50.280" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="4991" PostTypeId="2" ParentId="4980" CreationDate="2015-01-30T10:44:58.467" Score="3" Body="&lt;p&gt;Is this article good enough?&#xA;&lt;a href=&quot;http://www.succeed-project.eu/wiki/index.php/NLTK#Input_format_for_training&quot; rel=&quot;nofollow&quot;&gt;http://www.succeed-project.eu/wiki/index.php/NLTK#Input_format_for_training&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is explanation about how corpus should look like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data needs to be in IOB format (word tag chunktag) to make it work.&lt;br/&gt;&#xA;Eric NNP B-PERSON&lt;br/&gt;&#xA;is VB O&lt;br/&gt;&#xA;the AT B-NP&lt;br/&gt;&#xA;CEO NN I-NP&lt;br/&gt;&#xA;of IN O&lt;br/&gt;&#xA;Google NNP B-ORGANIZATION &lt;br/&gt;&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditDate="2015-01-30T11:57:37.427" LastActivityDate="2015-01-30T11:57:37.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="4992" PostTypeId="1" AcceptedAnswerId="5060" CreationDate="2015-01-30T11:08:37.280" Score="11" ViewCount="415" Body="&lt;p&gt;I'm trying to build a cosine locality sensitive hash so I can find candidate similar pairs of items without having to compare every possible pair. I have it basically working, but most of the pairs in my data seem to have cosine similarity in the -0.2 to +0.2 range so I'm trying to dice it quite finely and pick things with cosine similarity 0.1 and above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been reading Mining Massive Datasets chapter 3. This talks about increasing the accuracy of candidate pair selection by Amplifying a Locality-Sensitive Family. I think I just about understand the mathematical explanation, but I'm struggling to see how I implement this practically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have so far is as follows&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have say 1000 movies each with ratings from some selection of 1M users. Each movie is represented by a sparse vector of user scores (row number = user ID, value = user's score)&lt;/li&gt;&#xA;&lt;li&gt;I build N random vectors. The vector length matches the length of the movie vectors (i.e. the number of users). The vector values are +1 or -1. I actually encode these vectors as binary to save space, with +1 mapped to 1 and -1 mapped to 0&lt;/li&gt;&#xA;&lt;li&gt;I build sketch vectors for each movie by taking the dot product of the movie and each of the N random vectors (or rather, if I create a matrix R by laying the N random vectors horizontally and layering them on top of each other then the sketch for movie m is R*m), then taking the sign of each element in the resulting vector, so I end with a sketch vector for each movie of +1s and -1s, which again I encode as binary. Each vector is length N bits.&lt;/li&gt;&#xA;&lt;li&gt;Next I look for similar sketches by doing the following&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I split the sketch vector into b bands of r bits&lt;/li&gt;&#xA;&lt;li&gt;Each band of r bits is a number. I combine that number with the band number and add the movie to a hash bucket under that number. Each movie can be added to more than one bucket.&lt;/li&gt;&#xA;&lt;li&gt;I then look in each bucket. Any movies that are in the same bucket are candidate pairs.&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Comparing this to 3.6.3 of mmds, my AND step is when I look at bands of r bits - a pair of movies pass the AND step if the r bits have the same value. My OR step happens in the buckets: movies are candidate pairs if they are both in any of the buckets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book suggests I can &quot;amplify&quot; my results by adding more AND and OR steps, but I'm at a loss for how to do this practically as the explanation of the construction process for further layers is in terms of checking pairwise equality rather than coming up with bucket numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me understand how to do this?&lt;/p&gt;&#xA;" OwnerUserId="8030" LastActivityDate="2016-09-12T03:51:16.957" Title="Amplifying a Locality Sensitive Hash" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4993" PostTypeId="2" ParentId="4951" CreationDate="2015-01-30T12:11:41.293" Score="0" Body="&lt;p&gt;After speaking with some experienced statisticians, this is what I got.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As for technical issues regarding the paper, I'd be worried about data leakage or using future information in the current model. This can also occur in cross validation. You should make sure each model trains only on past data, and predicts on future data. I wasn't sure exactly how they conducted CV, but it definitely matters. It's also non-trivial to prevent all sources of leakage. They do claim unseen examples but it's not explicit exactly what code they wrote here. I'm not saying they are leaking for sure, but I'm saying it could happen.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2015-01-30T12:11:41.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4994" PostTypeId="2" ParentId="4879" CreationDate="2015-01-30T16:33:23.683" Score="0" Body="&lt;p&gt;I would call a mapping between N dimensional input and N dimensional output a regression problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you add more constraints about the relation between the input and output it might be called different names: linear filtering, nonlinear filtering, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;some examples on common techniques for that would be: neural networks, regression trees, regularised regressions...&lt;/p&gt;&#xA;" OwnerUserId="7999" LastActivityDate="2015-01-30T16:33:23.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4995" PostTypeId="1" AcceptedAnswerId="5005" CreationDate="2015-01-30T18:55:46.173" Score="4" ViewCount="4086" Body="&lt;p&gt;We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="7977" LastActivityDate="2015-01-31T12:29:35.733" Title="Can we access HDFS file system and YARN scheduler in Apache Spark?" Tags="|bigdata|apache-hadoop|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4996" PostTypeId="2" ParentId="4977" CreationDate="2015-01-30T20:15:09.003" Score="3" Body="&lt;p&gt;Apache Storm and Apache Spark are more popular than the other ones, there are already many discussions on Quora(&lt;a href=&quot;http://www.quora.com/What-is-the-difference-between-Apache-Storm-and-Apache-Spark&quot; rel=&quot;nofollow&quot;&gt;Storm vs Spark&lt;/a&gt;, &lt;a href=&quot;http://www.quora.com/Are-there-any-use-cases-for-a-comparison-between-Storm-and-Spark-Streaming&quot; rel=&quot;nofollow&quot;&gt;Use cases for comparison&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I think Spark is a better choice.&lt;/p&gt;&#xA;" OwnerUserId="2522" LastActivityDate="2015-01-30T20:15:09.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4997" PostTypeId="2" ParentId="155" CreationDate="2015-01-30T20:26:42.293" Score="16" Body="&lt;h2&gt;Data Sets&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://academictorrents.com/&quot; rel=&quot;noreferrer&quot;&gt;Academic Torrents&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public&quot; rel=&quot;noreferrer&quot;&gt;Quora&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://hadoopilluminated.com/hadoop_illuminated/Public_Bigdata_Sets.html&quot; rel=&quot;noreferrer&quot;&gt;hadoopilluminated.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://catalog.data.gov/dataset&quot; rel=&quot;noreferrer&quot;&gt;data.gov&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quandl.com/&quot; rel=&quot;noreferrer&quot;&gt;Quandl&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.freebase.com/&quot; rel=&quot;noreferrer&quot;&gt;freebase.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://usgovxml.com/&quot; rel=&quot;noreferrer&quot;&gt;usgovxml.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://public.enigma.com/&quot; rel=&quot;noreferrer&quot;&gt;enigma.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://datahub.io/&quot; rel=&quot;noreferrer&quot;&gt;datahub.io&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://aws.amazon.com/datasets&quot; rel=&quot;noreferrer&quot;&gt;aws.amazon.com/datasets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://databib.org/&quot; rel=&quot;noreferrer&quot;&gt;databib.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.datacite.org&quot; rel=&quot;noreferrer&quot;&gt;datacite.org&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quandl.com/&quot; rel=&quot;noreferrer&quot;&gt;quandl.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://figshare.com/&quot; rel=&quot;noreferrer&quot;&gt;figshare.com&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dev.maxmind.com/geoip/legacy/geolite/&quot; rel=&quot;noreferrer&quot;&gt;GeoLite Legacy Downloadable Databases&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public&quot; rel=&quot;noreferrer&quot;&gt;Quora's Big Datasets Answer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://hadoopilluminated.com/hadoop_illuminated/Public_Bigdata_Sets.html&quot; rel=&quot;noreferrer&quot;&gt;Public Big Data Sets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://data.ohouston.org/&quot; rel=&quot;noreferrer&quot;&gt;Houston Data Portal&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/wiki/DataSources&quot; rel=&quot;noreferrer&quot;&gt;Kaggle Data Sources&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.1000genomes.org/data&quot; rel=&quot;noreferrer&quot;&gt;A Deep Catalog of Human Genetic Variation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.freebase.com/&quot; rel=&quot;noreferrer&quot;&gt;A community-curated database of well-known people, places, and things&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.google.com/publicdata/directory&quot; rel=&quot;noreferrer&quot;&gt;Google Public Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://data.worldbank.org/&quot; rel=&quot;noreferrer&quot;&gt;World Bank Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nyctaxi.herokuapp.com/&quot; rel=&quot;noreferrer&quot;&gt;NYC Taxi data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.opendataphilly.org/&quot; rel=&quot;noreferrer&quot;&gt;Open Data Philly&lt;/a&gt; Connecting people with data for Philadelphia&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://networkrepository.com&quot; rel=&quot;noreferrer&quot;&gt;Network Repository&lt;/a&gt; An interactive data repository with over 600+ networks  in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ahmetkurnaz.net/en/statistical-data-sources/&quot; rel=&quot;noreferrer&quot;&gt;A list of useful sources&lt;/a&gt; A blog post includes many data set databases&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/okulbilisim/awesome-datascience#data-sets&quot; rel=&quot;noreferrer&quot;&gt;Data Sets&lt;/a&gt; From &lt;a href=&quot;https://github.com/okulbilisim/awesome-datascience&quot; rel=&quot;noreferrer&quot;&gt;awesome-datascience&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2522" LastEditorUserId="-1" LastEditDate="2017-09-14T19:16:18.913" LastActivityDate="2017-09-14T19:16:18.913" CommentCount="1" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="4998" PostTypeId="1" CreationDate="2015-01-30T21:04:30.173" Score="2" ViewCount="189" Body="&lt;p&gt;I found that Apache-Spark has pretty much simple interface and easy to use. But I want to know about other interfaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me a ranking of Big-Data frameworks in base of simplicity of their interfaces. also this is useful to express most simple and complex interfaces in base of your experiences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Definitely this question is about some frameworks with same tasks. For example a selection between Flink and Spark just in your opinion. Detailed comparison is so lengthy and this is not my purpose. Just a selection or ranking on your opinions is sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="7977" LastEditorUserId="7977" LastEditDate="2015-02-01T07:13:30.427" LastActivityDate="2015-02-06T01:08:32.770" Title="Which Big-Data Frameworks have most simple interfaces?" Tags="|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="4999" PostTypeId="2" ParentId="4879" CreationDate="2015-01-30T21:14:25.563" Score="0" Body="&lt;p&gt;N_dimension input - n_dimension output is a too general description. You could think of it as a regression problem where you predict multi-dimensional output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also it could be the case that you are solving multiclass-classification problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;input: n features&lt;/p&gt;&#xA;&#xA;&lt;p&gt;output: vector which defines class membership - either 0's and 1's or the real value which defines degree of membership to the class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could also think of it as of multilabel classification problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;input: n features&lt;/p&gt;&#xA;&#xA;&lt;p&gt;output: vector of 0 and 1 which define which labels are associated with the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in general multi-dimensional output is not telling anything about the matter of task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try 2 approaches to solve the task which involves multi-dimensional output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) One-vs-rest or one-vs-one strategies (or their variations) where for each 'part' (dimension) of the output you train separate classifier or separate regressor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Neural network with multiple output neurons. I would suggest to try it after trying #1, neural networks are complicated, computing-expensive and maybe somewhat clumsy - so far, I wasn't able to construct neural network which would outperform other models in specific tasks I tried to solve.  But of course, this is my personal opinion about NN. In your case they may really shine.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-01-30T21:14:25.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5000" PostTypeId="1" AcceptedAnswerId="5019" CreationDate="2015-01-30T21:30:18.077" Score="10" ViewCount="9284" Body="&lt;p&gt;Maybe it is a bit general question. I am trying to solve various regression tasks and I try various algorithms for them. For example, multivariate linear regression or an SVR. I know that the output can't be negative and I never have negative output values in my training set, though I could have 0's in it (for example, I predict 'amount of cars on the road' - it can't be negative but can be 0). Rather often I face a problem that I am able to train relatively good algorithm (maybe fit a good regression line to my data) and I have relatively small average squared error on training set. But when I try to run my regression algorithm against new data I sometimes get a negative output. Obviously, I can't accept negative output since it is not a valid value. The question is - what is the proper way of working with such output? Should I think of negative output as a 0 output? Is there any general advice for such cases?&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-02-02T20:20:08.503" Title="Proper way of fighting negative outputs of a regression algorithms where output must be positive all the way" Tags="|machine-learning|regression|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5001" PostTypeId="1" AcceptedAnswerId="6756" CreationDate="2015-01-30T23:39:04.687" Score="4" ViewCount="291" Body="&lt;p&gt;I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking for historical operation/maintenance/fault datasets of any kind of machine in the oil &amp;amp; gas industry (drills, steam injectors etc) or electrical companies (transformators, generators etc).&lt;/p&gt;&#xA;" OwnerUserId="8037" LastEditorUserId="31513" LastEditDate="2017-06-30T14:45:48.930" LastActivityDate="2017-06-30T14:45:48.930" Title="Data available from industry operations" Tags="|dataset|open-source|freebase|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5002" PostTypeId="2" ParentId="4998" CreationDate="2015-01-30T23:44:42.127" Score="5" Body="&lt;p&gt;I think that it is impossible to answer this question &lt;em&gt;comprehensively&lt;/em&gt;, at least for the following &lt;strong&gt;reasons&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;big data frameworks have different &lt;strong&gt;goals&lt;/strong&gt; and target different &lt;strong&gt;knowledge domains&lt;/strong&gt;, so the comparison simply doesn't make much sense;&lt;/li&gt;&#xA;&lt;li&gt;most big data frameworks (and other programming frameworks, for that matter) have &lt;em&gt;multiple interfaces&lt;/em&gt;, and frequently those &lt;strong&gt;sets of interfaces&lt;/strong&gt; are significantly different (the &lt;em&gt;intersection&lt;/em&gt; is small), so there is a risk of comparing apples and oranges;&lt;/li&gt;&#xA;&lt;li&gt;trying to compare anything (in this case, interfaces), using &lt;em&gt;simplicity&lt;/em&gt; as a criterion, involves a significant amount of &lt;strong&gt;subjectivity&lt;/strong&gt; - what one person &lt;em&gt;perceive&lt;/em&gt; as very simple, another person might find quite complex;&lt;/li&gt;&#xA;&lt;li&gt;the &lt;strong&gt;variety&lt;/strong&gt; and the &lt;strong&gt;number&lt;/strong&gt; of &lt;em&gt;big data frameworks&lt;/em&gt; is mind-boggling (for example, see &lt;a href=&quot;https://github.com/onurakpolat/awesome-bigdata&quot; rel=&quot;nofollow&quot;&gt;https://github.com/onurakpolat/awesome-bigdata&lt;/a&gt;); the same applies to a related topic of &lt;em&gt;machine learning frameworks&lt;/em&gt; (for example, see &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;nofollow&quot;&gt;https://github.com/josephmisiti/awesome-machine-learning&lt;/a&gt;);&lt;/li&gt;&#xA;&lt;li&gt;corollary from the points above: a &lt;strong&gt;comprehensive&lt;/strong&gt; comparison (considering all the above-mentioned issues) would go far beyond the &lt;strong&gt;scope&lt;/strong&gt; of a single answer on this site, in volume and effort - it would be more like a long research paper, a book chapter or even a book.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-02-06T01:08:32.770" LastActivityDate="2015-02-06T01:08:32.770" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5004" PostTypeId="2" ParentId="5000" CreationDate="2015-01-31T01:18:10.307" Score="8" Body="&lt;p&gt;A standard trick is to estimate the logarithm of the desired quantity, then take its exponential, which is always positive. The drawback is that the error is optimized for the log, which treats differences in order of magnitude as equal. Another option is to do your regression as usual then project onto the feasible set (use the positive part of the output; $max(0, \cdot)$)&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2015-02-01T19:45:23.910" LastActivityDate="2015-02-01T19:45:23.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5005" PostTypeId="2" ParentId="4995" CreationDate="2015-01-31T02:22:36.000" Score="4" Body="&lt;p&gt;Yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are examples on spark official document: &lt;a href=&quot;https://spark.apache.org/examples.html&quot; rel=&quot;nofollow&quot;&gt;https://spark.apache.org/examples.html&lt;/a&gt;&#xA;Just put your HDFS file uri in your input file path as below (scala syntax).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val file = spark.textFile(&quot;hdfs://train_data&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1003" LastEditorUserId="1279" LastEditDate="2015-01-31T10:49:29.950" LastActivityDate="2015-01-31T10:49:29.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5006" PostTypeId="2" ParentId="4995" CreationDate="2015-01-31T12:29:35.733" Score="6" Body="&lt;p&gt;&lt;strong&gt;HDFS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark was built as an alternative to MapReduce and thus supports most of its functionality. In particular, it means that &quot;Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc.&quot;&lt;a href=&quot;http://spark.apache.org/docs/1.2.0/programming-guide.html#external-datasets&quot;&gt;1&lt;/a&gt;. For most common data sources (like HDFS or S3) Spark automatically recognizes schema, e.g.: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val sc = SparkContext(...)&#xA;val localRDD = sc.textFile(&quot;file://...&quot;)&#xA;val hdfsRDD  = sc.textFile(&quot;hdfs://...&quot;)&#xA;val s3RDD    = sc.textFile(&quot;s3://...&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more complicated cases you may need to work with lower-level functions like &lt;code&gt;newAPIHadoopFile&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], &#xA;      classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],&#xA;      classOf[org.apache.hadoop.hbase.client.Result])&#xA;val customRDD = sc.newAPIHadoopRDD(conf, classOf[MyCustomInputFormat], &#xA;      classOf[MyCustomKeyClass],&#xA;      classOf[MyCustomValueClass]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But general rule is that if some data source is available for MapReduce, it can be easily reused in Spark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;YARN&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently Spark supports 3 cluster managers / modes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Standalone&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Mesos&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;YARN&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Standalone mode uses Spark's own master server and works for Spark only, while YARN and Mesos modes aim to share same set of system resources between several frameworks (e.g. Spark, MapReduce, Impala, etc.). Comparison of YARN and Mesos may be found &lt;a href=&quot;http://www.quora.com/How-does-YARN-compare-to-Mesos&quot;&gt;here&lt;/a&gt;, and detailed description of Spark on YARN &lt;a href=&quot;http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, in best traditions of Spark, you can switch between different modes simply by changing &lt;a href=&quot;https://spark.apache.org/docs/1.2.0/submitting-applications.html&quot;&gt;master URL&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2015-01-31T12:29:35.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5007" PostTypeId="1" CreationDate="2015-01-31T13:28:45.590" Score="0" ViewCount="1531" Body="&lt;p&gt;I am wondering if there is a way to proceed 2 exectuions in 1 step in hive.&#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT * FROM TABLE1&#xA;SELECT * FROM TABLE2&#xA;;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do this in one window, and do not have to open 2 hive windows to execute each line separetly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can it be done on HUE?&lt;/p&gt;&#xA;" OwnerUserId="5224" LastEditorUserId="5224" LastEditDate="2015-02-01T18:37:12.500" LastActivityDate="2015-02-02T17:00:51.307" Title="How to proceed 2 executions in 1 step in hive?" Tags="|hive|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5008" PostTypeId="2" ParentId="5001" CreationDate="2015-01-31T14:27:15.657" Score="3" Body="&lt;p&gt;A huge list of open data sets is listed here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/155/publicly-available-datasets&quot;&gt;Publicly Available Datasets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Including Amazon, KDnuggets, Stanford, Twitter, Freebase, Google Public and more.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-01-31T14:27:15.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5009" PostTypeId="2" ParentId="5007" CreationDate="2015-01-31T16:55:53.350" Score="2" Body="&lt;p&gt;You can use HiveCLI Tool to run HiveQL with a given sql file.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$HIVE_HOME/bin/hive -f /home/my/hive-script.sql&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please see official document: &lt;a href=&quot;https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli&quot; rel=&quot;nofollow&quot;&gt;https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need to do is to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Put your HiveQLs in a file as below&lt;br&gt;&#xA;&lt;code&gt;SELECT * FROM TABLE1;&#xA;SELECT * FROM TABLE2;&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use HiveCLI and run with above file&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1003" LastActivityDate="2015-01-31T16:55:53.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5010" PostTypeId="2" ParentId="5007" CreationDate="2015-01-31T17:31:09.967" Score="1" Body="&lt;p&gt;You can separate each query with a semi colon (;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select column1 from table1;&#xA;select column2 from table2;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This command can be executed from command line via inline queries or a file. Usage of Hive CLI is not recommended. You must use beeline to execute queries configured via hive server 2 so that all/any underlying security control measures are honored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you may invoke beeline with the command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;beeline&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7809" LastActivityDate="2015-01-31T17:31:09.967" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5011" PostTypeId="2" ParentId="4985" CreationDate="2015-02-01T01:34:13.597" Score="2" Body="&lt;p&gt;I thought about it this way: the training and test sets are both a sample of the unknown population. We assume that the training set is representative of the population we're studying. That is, whatever transformations we make to the training set are what we would make to the overall population. In addition, whatever subset of the training data we use, we assume that this subset represents the training set, which represents the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in response to your first question, it's fine to use that shrinked/resmpled training as long as you feel it's still representative of that population. That's assuming your untouched training set captures the &quot;real characteristics&quot; in the first place :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your second question, don't merge the training and testing set. The testing set is there to act as future unknown observations. If you build these into the model then you won't know if the model wrong or not, because you used up the data you were going to test it with.&lt;/p&gt;&#xA;" OwnerUserId="525" LastActivityDate="2015-02-01T01:34:13.597" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5013" PostTypeId="1" AcceptedAnswerId="5016" CreationDate="2015-02-01T09:35:16.113" Score="0" ViewCount="2188" Body="&lt;p&gt;Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel 47XX 4th Gen core i7 and GPU: Nvidia quadro k1100M 2GB. Kindly help me with recommendation for architecture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking in the lines of 27000 input neurons on basis of 30x30x30 voxel grid but can't tell in advance if this is a good option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional Note: Dataset has 4500 points on average per view per point cloud&lt;/p&gt;&#xA;" OwnerUserId="8051" LastEditorUserId="21" LastEditDate="2017-05-19T16:12:46.690" LastActivityDate="2017-05-19T16:12:46.690" Title="Machine learning for Point Clouds Lidar data" Tags="|machine-learning|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5014" PostTypeId="1" AcceptedAnswerId="5015" CreationDate="2015-02-02T14:53:51.810" Score="3" ViewCount="356" Body="&lt;p&gt;Recently I read about path ranking algorithm in a paper (source: &lt;a href=&quot;https://www.cs.cmu.edu/~nlao/publication/2014.kdd.pdf&quot; rel=&quot;nofollow&quot;&gt;Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this paper was a table (Table 3) with facts and I tried to understand how they were calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;F1 (harmonic mean of precision and recall) = 0.04&lt;br/&gt;&#xA;P (precision) = 0.03&lt;br/&gt;&#xA;R (recall) = 0.33&lt;br/&gt;&#xA;W (weight given to this feature by logistic regression)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a formula for F1 via Google which is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F1 = 2 * \frac{precision * recall}{precision + recall}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I get the result of 0.055 with this formula, but not the expected result of 0.04.&#xA;Can someone help me to get this part?&#xA;Also, does someone know how 'W' can be calculated?&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="8063" LastEditorUserId="8063" LastEditDate="2015-02-02T14:59:43.687" LastActivityDate="2018-06-21T18:24:00.583" Title="How to compute F1 score?" Tags="|machine-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5015" PostTypeId="2" ParentId="5014" CreationDate="2015-02-02T15:55:49.363" Score="2" Body="&lt;p&gt;First you need to learn about Logistic Regression, it is an algorithm that will assign weights to different features given some training data. Read the wiki intro, is quite helpful, basically the Betas there are the same as the Ws in the paper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula you have is correct, and those value do seem off. It also depends on the number of significant figures you have, perhaps they are making their calculations with more than the ones they are reporting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But honestly, you can't understand much of the paper unless you understand LR&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-02T15:55:49.363" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5016" PostTypeId="2" ParentId="5013" CreationDate="2015-02-02T15:59:40.643" Score="1" Body="&lt;p&gt;First, CNNs are great for image recognition, where you usually take sub sampled windows of about 80 by 80 pixels, 27,000 input neurons is too large and it will take you forever to train a CNN on that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, why did you choose CNN? Why don't you try some more down to earth algorithms fisrst? Like SVMs, or Logistic regressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4500 Data points and 27000 features seems unrealistic to me, and very prone to over fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/tutorial/machine_learning_map/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-02T15:59:40.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5017" PostTypeId="2" ParentId="4979" CreationDate="2015-02-02T16:08:50.580" Score="1" Body="&lt;p&gt;I don't really get why would you mix Fuzziness and Probabilities. HMMs already can give you probabilities without the need of adding Fuzzy systems into the mix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would just do a random walk with probabilities of transitions defined by the state of the lights.&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-02T16:08:50.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5018" PostTypeId="2" ParentId="5007" CreationDate="2015-02-02T17:00:51.307" Score="2" Body="&lt;p&gt;Yes, you can execute multiple HQL's using Hue as long as each individual HQL is separated by a semi colon (;)&lt;/p&gt;&#xA;" OwnerUserId="7809" LastActivityDate="2015-02-02T17:00:51.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5019" PostTypeId="2" ParentId="5000" CreationDate="2015-02-02T20:20:08.503" Score="7" Body="&lt;p&gt;The problem is your model choice, as you seem to recognize. In the case of linear regression, there is no restriction on your outputs.  Often this is fine when predictions need to be non-negative so long as they are far enough away from zero.  However, since many of your training examples are zero-valued, this isn't the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data is non-negative and discrete (as in the case with number of cars on the road), you could model using a generalized linear model (GLM) with a log link function.  This is known as Poisson regression and is helpful for modeling discrete non-negative counts such as the problem you described. The Poisson distribution is parameterized by a single value $\lambda$, which describes both the expected value and the variance of the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This results in an approach similar to the one described by Emre in that you are attempting to fit a linear model to the log of your observations.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-02-02T20:20:08.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5020" PostTypeId="2" ParentId="4955" CreationDate="2015-02-02T20:58:47.970" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to try out unsupervised learning for the same. I am looking into KMeans clustering. But again the problem is, we won't know the number of clusters beforehand. So, which algorithm will be best for this use case?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When you don't know the number of clusters beforehand, it is still possible to do unsupervised learning using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirichlet_process&quot; rel=&quot;nofollow&quot;&gt;Dirichlet process&lt;/a&gt; to sample parameters associated to clusters/groups and then cluster your tokens according to those parameters. The general idea is to use a Dirichlet distribution to generate probabilities over words for each cluster and a Dirichlet process uses these probabilities to assign a cluster to each word in your vocabulary. If you want to share clusters between emails, then you use Hierarchical Dirichlet Processes. &lt;a href=&quot;http://blog.echen.me/2012/03/20/infinite-mixture-models-with-nonparametric-bayes-and-the-dirichlet-process/&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; you can find a nice blog post of how this works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most popular library for clustering is &lt;a href=&quot;http://radimrehurek.com/gensim/tut2.html&quot; rel=&quot;nofollow&quot;&gt;gensim&lt;/a&gt;, but notice their warning regarding the Hierarchical Dirichlet Process implementation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;gensim uses a fast, online implementation based on [3]. The HDP model is a new addition to gensim, and still rough around its academic edges – use with care.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As for feature extraction, your question doesn't say exactly what kind of unnecessary features you are getting, but if that's the case, you need to filter your tokens before or after processing them with NLTK. In general, you can't expect excellent results for very specific applications.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2015-02-02T20:58:47.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5021" PostTypeId="1" CreationDate="2015-02-03T04:23:37.463" Score="4" ViewCount="1184" Body="&lt;p&gt;I've recently become interested in possibly of developing some sort of method for ranking athletes of sports such as American football and determining which players are better than others in terms of specific statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts are that there are two ways to go about doing this. The first would be some sort of mathematical formula which would take in the statistics of a given player and provide some sort of standardized score which could be compared with other players to determine which is better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My other idea would be to have some machine learning algorithm go through historical data and determine the patterns which indicate how well a certain combination of statistics would perform in the following week of play by using the patterns it recognizes over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure which approach would be more effective and so I'm hoping that someone has an idea or any advice as to which would be best to look into. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="8074" LastEditorUserId="97" LastEditDate="2015-02-03T23:58:36.380" LastActivityDate="2015-02-03T23:58:36.380" Title="Rank players of any given sport" Tags="|predictive-modeling|scoring|ranking|sports|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5022" PostTypeId="2" ParentId="5021" CreationDate="2015-02-03T08:09:44.953" Score="2" Body="&lt;p&gt;&lt;strong&gt;Prediction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the main goal is predicting anything, say, the statisitcs of the player in the next game, game result, then I would not recommend to do any scoring. Better way to go is using the pure statistics data as an input to the model. Any scoring/rankning - is information loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ranking&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the goal is ranking itself, than you still need to have some target variable to predict. As you may want to check real predictive value of those ranks. That could be, again, playser stats in the &lt;em&gt;next&lt;/em&gt; game or game result itself.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;References&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/265/can-machine-learning-algorithms-predict-sports-scores-or-plays/269#269&quot;&gt;Sport scores prediction&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/1119/predictive-modeling-based-on-rfm-scoring-indicators&quot;&gt;RFM scoring&lt;/a&gt; are probably the next directions for you to look at.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-02-03T08:09:44.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5023" PostTypeId="1" AcceptedAnswerId="5032" CreationDate="2015-02-03T08:52:42.133" Score="4" ViewCount="8835" Body="&lt;p&gt;With respect to &lt;a href=&quot;http://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;ROC&lt;/a&gt; can anyone please tell me what the phrase &quot;discrimination threshold of binary classifier system&quot; means? I know what a binary classifier is.&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="85045" LastEditDate="2021-02-10T17:49:46.840" LastActivityDate="2021-02-10T17:49:46.840" Title="What is a discrimination threshold of binary classifier?" Tags="|classification|graphs|classifier|roc|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5024" PostTypeId="1" CreationDate="2015-02-03T10:14:53.543" Score="6" ViewCount="9840" Body="&lt;p&gt;I am learning about Data Science and I love the Healthcare part. That's why I have started a blog and my third entry is about using Genetic Algorithm for solving NP-Problems. &#xA;This post is &lt;a href=&quot;https://datasciencecgp.wordpress.com/2015/01/31/the-amazing-genetic-algorithms/&quot; rel=&quot;noreferrer&quot;&gt;https://datasciencecgp.wordpress.com/2015/01/31/the-amazing-genetic-algorithms/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some expertise with GA package solving problems like the TSP, but do you know any most powerful R package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks so much!&lt;/p&gt;&#xA;" OwnerUserId="8076" LastEditorUserId="8076" LastEditDate="2015-02-04T16:48:43.630" LastActivityDate="2015-09-15T15:34:15.223" Title="Is the GA R package the best Genetic Algorithm package?" Tags="|r|algorithms|genetic|" AnswerCount="2" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5025" PostTypeId="2" ParentId="5024" CreationDate="2015-02-03T11:37:06.760" Score="5" Body="&lt;p&gt;For such questions, I like to go to the &lt;a href=&quot;http://cran.r-project.org/web/views/&quot;&gt;Task Views&lt;/a&gt; on CRAN, since the packages noted there are, to a degree, pre-vetted by the R community. I'd trust those a tiny bit more than just googling myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://cran.r-project.org/web/views/MachineLearning.html&quot;&gt;Machine Learning Task View at CRAN&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Packages &lt;a href=&quot;http://cran.r-project.org/web/packages/rgp/index.html&quot;&gt;rgp&lt;/a&gt;&#xA;  and&#xA;  &lt;a href=&quot;http://cran.r-project.org/web/packages/rgenoud/index.html&quot;&gt;rgenoud&lt;/a&gt;&#xA;  offer optimization routines based on genetic algorithms. The package&#xA;  &lt;a href=&quot;http://cran.r-project.org/web/packages/Rmalschains/index.html&quot;&gt;Rmalschains&lt;/a&gt;&#xA;  implements memetic algorithms with local search chains, which are a&#xA;  special type of evolutionary algorithms, combining a steady state&#xA;  genetic algorithm with local search for real-valued parameter&#xA;  optimization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2853" LastActivityDate="2015-02-03T11:37:06.760" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5026" PostTypeId="2" ParentId="5023" CreationDate="2015-02-03T16:00:05.673" Score="4" Body="&lt;p&gt;Classifiers often return probabilities of belonging to a class. For example in logistic regression the predicted values are the predicted probability of belonging to the non-reference class or $\text{Pr}(Y = 1)$. The discrimination threshold is just the cutoff imposed on the predicted probabilities for assigning observations to each class.&lt;/p&gt;&#xA;" OwnerUserId="5103" LastActivityDate="2015-02-03T16:00:05.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5027" PostTypeId="2" ParentId="4977" CreationDate="2015-02-03T19:01:59.827" Score="3" Body="&lt;p&gt;Amazon Kinesis might be another choice for stream processing, if you don't want to set up the clusters by yourself.&lt;/p&gt;&#xA;" OwnerUserId="8084" LastActivityDate="2015-02-03T19:01:59.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5028" PostTypeId="2" ParentId="3797" CreationDate="2015-02-03T19:53:47.233" Score="3" Body="&lt;p&gt;From what I understand of the objectives of the lambda architecture your point:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Your batch layer is probably a map reduce job or a HIVE query. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is not what was intended. The batch layer is not meant to be directly queried against, but rather feeds a serving layer, possibly a simple key-value store, for low latency queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/jtQ1b.png&quot; alt=&quot;lambda architecture diagram&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://lambda-architecture.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://lambda-architecture.net/&lt;/a&gt; for a more full explanation.&lt;/p&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2015-02-03T19:53:47.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5029" PostTypeId="2" ParentId="155" CreationDate="2015-02-03T20:05:59.917" Score="9" Body="&lt;p&gt;One other data source I didn't see listed is &lt;a href=&quot;http://gdeltproject.org/&quot;&gt;The GDELT Project&lt;/a&gt;. From the site:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;GDELT Project monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, counts, themes, sources, and events driving our global society every second of every day, creating a free open platform for computing on the entire world.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2015-02-03T20:05:59.917" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="5031" PostTypeId="2" ParentId="3699" CreationDate="2015-02-03T22:51:57.083" Score="3" Body="&lt;p&gt;If you have an imbalanced dataset you usually want to make it balanced to begin with, since that will artificially affect your scores.&lt;/p&gt;&#xA;&lt;p&gt;Now, you want to be measuring precision and recall, since those can capture a bit better the imbalanced dataset biases.&lt;/p&gt;&#xA;&lt;p&gt;L1 or L2 won't perform particularly better in a balanced or unbalanced dataset, what you want to do is call elastic nets (which is a combination of the two) and do cross validation over the coefficients of each of the regularizers.&lt;/p&gt;&#xA;&lt;p&gt;Also, doing grid search is very odd, you are better using just cross validation and see what parameters work better.&lt;/p&gt;&#xA;&lt;p&gt;They even have &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html#sklearn.linear_model.ElasticNetCV&quot; rel=&quot;nofollow noreferrer&quot;&gt;ElasticNetCV&lt;/a&gt;, which does that part for you&lt;/p&gt;&#xA;" OwnerUserId="8065" LastEditorUserId="98307" LastEditDate="2020-08-13T11:46:33.743" LastActivityDate="2020-08-13T11:46:33.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5032" PostTypeId="2" ParentId="5023" CreationDate="2015-02-03T22:56:04.747" Score="4" Body="&lt;p&gt;Just to add a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like it was mentioned before, if you have a classifier (probabilistic) your output is a probability (a number between 0 and 1), ideally you want to say that everything larger than 0.5 is part of one class and anything less than 0.5 is the other class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you are classifying cancer rates, you are deeply concerned with false negatives (telling some he does not have cancer, when he does) while a false positive (telling someone he does have cancer when he doesn't) is not as critical (IDK - being told you've cancer coudl be psychologically very costly). So you might artificially move that threshold from 0.5 to higher or lower values, to change the sensitivity of the model in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By doing this, you can generate the ROC plot for different thresholds.&lt;/p&gt;&#xA;" OwnerUserId="8065" LastEditorUserId="-1" LastEditDate="2016-08-22T17:48:36.400" LastActivityDate="2016-08-22T17:48:36.400" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5033" PostTypeId="1" CreationDate="2015-02-03T23:49:30.013" Score="-1" ViewCount="215" Body="&lt;p&gt;I'am trying to create a regression based prediction (like booking website): predict the number of clicks for each hotel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to generate a .csv file containing two columns: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;hotel_id&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;predicted_number_of_clicks&lt;/code&gt; for all hotel_ids.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My first question question is: should I put the &lt;code&gt;hotel_id&lt;/code&gt; as feature in the predictive model? I think that I must to drop it... right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second question is: how can I write in the csv file only this 2 columns if I drop &lt;code&gt;hotel_id&lt;/code&gt; from the model features?&lt;/p&gt;&#xA;" OwnerUserId="8088" LastEditorUserId="29575" LastEditDate="2018-12-16T16:42:17.090" LastActivityDate="2018-12-16T16:42:17.090" Title="Format of CSV file" Tags="|regression|predictive-modeling|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5034" PostTypeId="2" ParentId="5033" CreationDate="2015-02-04T00:16:15.323" Score="0" Body="&lt;p&gt;The hotel_id should not be a feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's see if I understand you correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At testing time you give your model a whole set of feature values for a particular hotel you are interested in. This hotel has an id, which is known to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model should be able to take both an id and a set of feature values as input, so it can print both to an output file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that answer your question?&lt;/p&gt;&#xA;" OwnerUserId="8075" LastActivityDate="2015-02-04T00:16:15.323" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5037" PostTypeId="2" ParentId="5001" CreationDate="2015-02-04T02:54:52.773" Score="0" Body="&lt;p&gt;I've found an interesting project With tons of data available. It's a real data benchmark executed over an industrial valve. This is the website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://diag.mchtr.pw.edu.pl/damadics/&quot; rel=&quot;nofollow&quot;&gt;Industrial Actuator Real Data Benchmark Study.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8037" LastActivityDate="2015-02-04T02:54:52.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5038" PostTypeId="1" AcceptedAnswerId="5071" CreationDate="2015-02-04T04:51:28.063" Score="0" ViewCount="592" Body="&lt;p&gt;I'm trying to use a particular cost function (based on doubling rate of wealth) for a classification problem, and the solution works well in MATLAB. See &lt;a href=&quot;https://github.com/acmyers/compareCostFXs&quot; rel=&quot;nofollow&quot;&gt;https://github.com/acmyers/compareCostFXs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I try to do this in Python 2.7.6 I don't get any errors, but it only returns zeros for the theta values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the cost function and optimization method I've used in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def costFunctionDRW(theta, X, y):&#xA;&#xA;    # Initialize useful values&#xA;    m = len(y)&#xA;    # Marginal probability of acceptance&#xA;    marg_pA = sum(y)/m&#xA;    # Marginal probability of rejection&#xA;    marg_pR = 1 - marg_pA&#xA;&#xA;    # =============================================================&#xA;    pred = sigmoid(np.dot(X,theta))&#xA;    final_wealth_individual = (pred/marg_pA)*y + ((1-pred)/marg_pR)*(1-y)&#xA;    final_wealth = np.prod(final_wealth_individual)&#xA;    final_wealth = -final_wealth&#xA;&#xA;    return final_wealth&#xA;&#xA;result = scipy.optimize.fmin(costFunctionDRW, x0=initial_theta, \&#xA;                   args=(X_array, y_array), maxiter=1000, disp=False, full_output=True )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any advice would be much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="985" LastActivityDate="2015-02-06T09:41:54.033" Title="minimization with a negative cost function: works in MATLAB, not in Python" Tags="|classification|python|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5039" PostTypeId="2" ParentId="5038" CreationDate="2015-02-04T05:33:59.907" Score="0" Body="&lt;p&gt;If your number of cases is large, you may be running into a problem of numerical underflow in the line &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;final_wealth = np.prod(final_wealth_individual)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If each value of &lt;code&gt;final_wealth_individual&lt;/code&gt; is between 0 and 1, multiplying them all together can lead to a result that is too small to represent as a floating point number, resulting in a value of 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address this issue, take the log of &lt;code&gt;final_wealth_individual&lt;/code&gt; and add them together instead of multiplying.  Note that this will cause &lt;code&gt;final_wealth&lt;/code&gt; to be negative, so you will not need to multiply it by -1 as you are currently.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-02-04T05:33:59.907" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="5040" PostTypeId="1" AcceptedAnswerId="5043" CreationDate="2015-02-04T06:10:34.893" Score="1" ViewCount="1390" Body="&lt;p&gt;I'm using a set of features, says $X_1, X_2, ..., X_m $, to predict a target value $Y$, which is a continuous value from zero to one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first, I try to use a linear regression model to do the prediction, but it does not perform well. The root-mean-squared error is about 0.35, which is quite high for prediction of a value from 0 to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I have tried different models, &lt;em&gt;e.g.,&lt;/em&gt; decision-tree-based regression, random-forest-based regression, gradient boosting tree regression and &lt;em&gt;etc.&lt;/em&gt; However, all of these models also do not perform well. (RMSE $\approx $0.35, there is not significant difference with linear regression)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand there are many possible reasons for this problem, such as: feature selection or choice of model, but maybe more fundamentally, the quality of data set is not good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;strong&gt;how can I examine whether it is caused by bad data quality?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, for the size of data set, there are more than 10K data points, each of which associated with 105 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried to investigate importance of each feature by using decision-tree-based regression, it turns out that, only one feature (which should not be the most outstanding feature in my knowledge to this problem) have an importance of 0.2, while the rest of them only have an importance less than 0.1.&lt;/p&gt;&#xA;" OwnerUserId="7867" LastEditorUserId="7867" LastEditDate="2015-02-04T07:37:14.677" LastActivityDate="2015-02-04T07:37:14.677" Title="How to determine whether a bad performance is caused by data quality?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5041" PostTypeId="2" ParentId="5040" CreationDate="2015-02-04T06:15:52.873" Score="1" Body="&lt;p&gt;How many features do you have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is quite unlikely that ALL the features are bad. So you could regress with a different number of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, do one pass with all the features, then take one out (usually X_m) so you have m-1 features. Keep doing this so you can take out uninformative features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would recommend you calculating P-Values to see whether your regessors are significative are informative.&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-04T06:15:52.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5042" PostTypeId="2" ParentId="5033" CreationDate="2015-02-04T06:18:04.207" Score="1" Body="&lt;p&gt;Trying to classify using hotel ID is the same as trying to determine if a student is going to perform well on a test based on their last name. You should get additional things, like number of rooms, amenities, location, staff, etc. Those are informative features that you can use in a classifier&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-04T06:18:04.207" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5043" PostTypeId="2" ParentId="5040" CreationDate="2015-02-04T06:22:23.377" Score="3" Body="&lt;p&gt;First, it sounds like your choice of model selection is a problem here.  Your outputs are binary-valued, not continuous. Specifically you may have a classification problem on your hands rather than a traditional regression problem.  My first recommendation would be to try a simple classification approach such as logistic regression or linear discriminant analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your suspicions of &lt;em&gt;bad data&lt;/em&gt;, what would bad data look like in this situation? Do you have reason to suspect that your $X$ values are noisy or that your $y$ values are mislabeled? It is also possible that there is not a strong relationship between any of your features and your targets.  Since your targets are binary, you should look at histograms of each of your features to get a rough sense of the class conditional distributions, i.e. $p(X_1|y=1)$ vs $p(X_1|y=0)$. In general though, you will need to be more specific about what &quot;bad data&quot; means to you.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-02-04T06:22:23.377" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5044" PostTypeId="2" ParentId="5033" CreationDate="2015-02-04T09:28:16.910" Score="1" Body="&lt;p&gt;When training the model you don't need to use the hotel id. The model needs to learn from examples. It only needs feature values and number of clicks, so it can learn the relationship between these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've trained your model, you can use it for unseen examples.&#xA;These would be hotels for which you have both an id and a set of feature values.&#xA;Your model should take the id and the feature values as input, but it should only use the feature values for the prediction. The id should just be kept on the side, so it can print it together with the prediction to the output csv file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="8075" LastActivityDate="2015-02-04T09:28:16.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5045" PostTypeId="2" ParentId="5033" CreationDate="2015-02-04T11:22:53.063" Score="1" Body="&lt;p&gt;Supervised learning should try to 'understand' what makes a hotel to have more clicks than other. As a consequence learning tries to define which are the characteristics of some given hotels which make them attractive or not. So it uses some kind of similarities, because it is supposed that similar hotels behaves in a similar way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you restrict the similarity to identity than you learn nothing new because hotels are unique. In fact such kind of learner exists and is called Rote learner, and it consists of one-to-one mapping from inputs to outputs. It is also called memoisation. And this happens if you will add hotel_id in the features. However I think you hope to use that to predict the number of clicks for new hotels (which does have a different hotel_id than any from training set). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, in order to use hotel_id to store prediction you only have to save a copy of the original data set. At learning time you have a train data set from which you remove hotel_id, and use that for learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At prediction time you make a copy of the data set for later use. From the original data set remove order_id, use that for prediction and get the results. Now the predicted results have the same order of instances as the copied data set. This happens for sure in python (scikit learn), java (weka), R. In fact I am not aware of a system which does not preserve positions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now using positions from the copy of the original and prediction you can associate each hotel_id to each prediction with no problem. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-02-04T11:22:53.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5046" PostTypeId="2" ParentId="2513" CreationDate="2015-02-04T12:51:31.943" Score="3" Body="&lt;p&gt;Your understanding is correct. The point is that equation (8)&#xA;$$y_i(&amp;lt;\textbf{w}, \phi_i&amp;gt; + b) - 1 = 0$$&#xA;is not exactly an equation, but a system of equations, one for each $i$ index of the support vectors (those for each $0&amp;lt;\alpha_i&amp;lt;C$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that you cannot compute $b$ during the optimization of the dual problem since it does not matter for optimization, you have to go back and compute $b$ from all the other equations you have (one possible way is (8)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vapnick suggestion is to not use only one of those equations, but two of them, specifically one support vector for a negative observation and one for a positive observation. In other words two support vectors which have opposite signs for $y_i$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's name $A$ the index of one support vector and $B$ the index of a suport vector of opposite side, baiscally you select from the system of equations at (8) only two of them. Evaluate both of them and take the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From:&#xA;$$y_A(&amp;lt;\textbf{w},\phi_A&amp;gt;+b)=1$$&#xA;$$y_B(&amp;lt;\textbf{w}, \phi_B&amp;gt;+b)=1$$&#xA;We get:&#xA;$$b_A=\frac{1}{y_A}-&amp;lt;\textbf{w},\phi_A&amp;gt;$$&#xA;$$b_B=\frac{1}{y_B}-&amp;lt;\textbf{w},\phi_B&amp;gt;$$&#xA;Where $b_A$ and $b_B$ are two estimations, then the mean is&#xA;$$b = (b_A+b_B)/2 = -\frac{1}{2}(&amp;lt;\textbf{w},\phi_A&amp;gt;+&amp;lt;\textbf{w},\phi_B&amp;gt;)=-\frac{1}{2}\sum_{i=1}^{n}y_i\alpha_i(&amp;lt;\phi(x_i),\phi(x_A)&amp;gt;+&amp;lt;\phi(x_i),\phi(x_B)&amp;gt;)$$&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-02-04T12:51:31.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5047" PostTypeId="1" CreationDate="2015-02-04T22:03:48.507" Score="2" ViewCount="1070" Body="&lt;p&gt;I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. There are two sources of data here, each presenting their own challenges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Army and Air Force data.&lt;/strong&gt; The National Archives hosts lists of Army and Air Force servicemen killed in action by state and county. There are .gif images of the report available online. &lt;a href=&quot;http://media.nara.gov/media/images/29/19/29-1891a.gif&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a sample for several counties in Texas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I DO NOT need to recover the names or any other information. I simply need to count the number of names (each on its own line, and listed in groups of five) under each County. There are hundreds of these images (50 states - 30-100 for each state). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been unable to find an OCR program that can tackle this problem adequately. How would you suggest I approach this challenge? (I have some programming expertise in Python and Java, but would prefer to use any off-the-shelf solutions that may exist).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Navy and Marine Core data.&lt;/strong&gt; This data is organized differently. Each state has long lists of casualties with the address of their next of kin. &lt;a href=&quot;http://media.nara.gov/media/images/27/31/27-3023a.gif&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a sample for Texas again. For these images, I need to BOTH count the number of dead and recover their hometown, which is typically the last word in each entry. I can then match these hometowns to counties and merge with database 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, the usual OCR programs have proved inadequate. Any help on this (admittedly more difficult) problem would be very much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance experts!&lt;/p&gt;&#xA;" OwnerUserId="8102" LastActivityDate="2015-02-13T05:57:36.840" Title="OCR / Text Recognition and Recovery Problem" Tags="|dataset|text-mining|data-cleaning|processing|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5048" PostTypeId="2" ParentId="5047" CreationDate="2015-02-04T23:58:53.323" Score="0" Body="&lt;p&gt;In what way have the usual OCR programs proved inadequate?&#xA;Do you have some example output that you find you can't work with?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see how the columns complicate things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say for data set 1:&#xA;OCR the images, then read the files line by line and match on for instance a sequence of at least five numbers. So you get 0, 1, 2 or 3 per row.&#xA;You may miss a couple due to the OCR accidentally recognizing a number as a letter for instance, but I expect this to work reasonably well.&#xA;How precise do you have to be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data set 2 seems more difficult.&#xA;Maybe counting can be done by matching on capitalized sequences followed by a comma.&#xA;Placename... very tricky. Once again, do you have some OCR output we can look at?&lt;/p&gt;&#xA;" OwnerUserId="8075" LastActivityDate="2015-02-04T23:58:53.323" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5049" PostTypeId="2" ParentId="155" CreationDate="2015-02-05T00:49:11.713" Score="15" Body="&lt;p&gt;I'm surprised one has not mentioned this, as it seems fairly obvious: &lt;a href=&quot;http://www.kaggle.com&quot;&gt;http://www.kaggle.com&lt;/a&gt; consistently has new and very interesting datasets. Information is considered an asset, so often companies don't want to release that data (plus privacy concerns). Kaggle gives you data and they hope you solve business problems with it in exchange.&lt;/p&gt;&#xA;" OwnerUserId="9768" LastEditorUserId="84" LastEditDate="2015-02-28T13:36:02.240" LastActivityDate="2015-02-28T13:36:02.240" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="5050" PostTypeId="1" CreationDate="2015-02-05T01:29:00.997" Score="3" ViewCount="984" Body="&lt;p&gt;I'm doing some work trying to extract commonly occurring words from a set of human classified documents and had a couple questions for anyone who might know something about NLP or statistical analysis of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a set of a bunch of documents, and users have classified them as either good or bad. What I'd like to do is figure out what words are common to the good documents, but not necessarily the other ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could, for example, use the (frequency within good documents / total frequency) which would essentially normalize the effect of a word being generally common. This, unfortunately, gives very high precedence to words that occur in only a few good documents &amp;amp; not at all in the other documents. I could add some kind of minimum threshold for # of occurrences in good docs before evaluating the total frequency, but it seems kind of hacky.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know what the best practice equation or model to use in this case is? I've done a lot of searching and found a lot of references to TF-IDF but that seems more applicable for assessing the value of a term on a single document against the whole set of docs. Here I'm dealing with a set of docs that is a subset of the larger collection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, I'd like to identify which words are uniquely or more important to the class of good documents.&lt;/p&gt;&#xA;" OwnerUserId="8105" LastActivityDate="2015-02-06T00:47:49.187" Title="Word Frequency Analysis of Document Sets" Tags="|data-mining|classification|statistics|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5051" PostTypeId="2" ParentId="5050" CreationDate="2015-02-05T02:25:51.627" Score="1" Body="&lt;p&gt;Your (frequency within good documents / total frequency) seems reasonable to me.&#xA;It could be that most words that occur in many good documents, simply also occur in many bad documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about you make a list of all the words appearing in the good documents.&#xA;Then you count their appearance in the good documents and their appearance in the bad documents and compare those two numbers.&#xA;The words that appear more often in the good ones, with a difference higher than a certain threshold are the ones of interest to you (if they exist).&lt;/p&gt;&#xA;" OwnerUserId="8075" LastActivityDate="2015-02-05T02:25:51.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5053" PostTypeId="2" ParentId="5050" CreationDate="2015-02-05T06:54:16.023" Score="2" Body="&lt;p&gt;I guess what you are looking for is Differential Word Usage. This method takes two text corpora as input and you can get the list of words which are being used more in one text corpus over the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically what you need to do is to build a common Term Document Matrix for the corpora you are using and then divide this TDM into two TDMs such that all the document columns from corpus 1 fall in one TDM and all the documents columns form corpus 2 fall in the second TDM.  For example, you have 2 corpora, the first one containing 10 documents and the second one containing 15 documents. You first, combine both these corpora and form 25 document corpus and then form the TDM, where terms become the rows (let's say there are 300 terms) and the 25 documents become the 25 columns. Here the first 10 columns represent the documents of first corpus and the remaining 15 belong to the second corpus. So, you divide this TDM of dimensions &lt;code&gt;300 x 25&lt;/code&gt; to two TDMs of dimensions &lt;code&gt;300 x 10&lt;/code&gt; and &lt;code&gt;300 x 15&lt;/code&gt;. Then you can use &lt;code&gt;Chi-square difference&lt;/code&gt; over these TDMs to determine which words are occurring more in one corpus than the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A wonderful example has been given regarding this approach by Vik in his blog using Wikileaks corpus and &lt;code&gt;R&lt;/code&gt; here: &lt;a href=&quot;http://www.vikparuchuri.com/blog/finding-word-use-patterns-in-wikileaks/&quot; rel=&quot;nofollow&quot;&gt;http://www.vikparuchuri.com/blog/finding-word-use-patterns-in-wikileaks/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8059" LastActivityDate="2015-02-05T06:54:16.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5054" PostTypeId="2" ParentId="5050" CreationDate="2015-02-05T07:26:04.347" Score="4" Body="&lt;p&gt;There are many algorithm to do classification: Naïve Bayes, logistic regression, SVM, decision tree..etc. My suggestion is to try Naïve Bayes first by calculating below probabilities which a new document belongs to $class_{good}$ or $class_{bad}$. (&lt;a href=&quot;https://web.stanford.edu/class/cs124/lec/naivebayes.pdf&quot; rel=&quot;nofollow&quot;&gt;https://web.stanford.edu/class/cs124/lec/naivebayes.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;P(Class_{good} \vert document_{new}) = \frac{P(document_{new} \vert Class_{good}) \cdot P(Class_{good}) }{P(document_{new})}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;P(Class_{bad} \vert document_{new}) = \frac{P(document_{new} \vert Class_{bad}) \cdot P(Class_{bad}) }{P(document_{new})}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And generally, when we are doing text mining questions, we will do several preprocess on one document: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tokenization(1-gram/bigram/...etc)&lt;/li&gt;&#xA;&lt;li&gt;remove stop words ('a', 'the' 'at', ... etc)&lt;/li&gt;&#xA;&lt;li&gt;Stemming: transforms a word into its root form. (studied =&gt; study)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My suggestion is to do above preprocesses and try more features not just the words in one document, if there are some metadataes.&lt;/p&gt;&#xA;" OwnerUserId="1003" LastEditorUserId="1003" LastEditDate="2015-02-05T07:32:45.207" LastActivityDate="2015-02-05T07:32:45.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5055" PostTypeId="1" CreationDate="2015-02-05T07:41:10.443" Score="4" ViewCount="793" Body="&lt;p&gt;&lt;strong&gt;Example Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset (in &lt;code&gt;R&lt;/code&gt; as a &lt;code&gt;data frame&lt;/code&gt;) of race results for athletes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;athlete racedistance time    location tracktype       date    coach&#xA;A          100       10.0       UK     typeA       2014-01-01  carlos&#xA;A          200       20.0       US     typeB       2014-02-01  carla&#xA;A          100        9.5      AUS     typeC       2014-03-01  chris&#xA;B          100       11.0       UK     typeA       2014-01-01  carla&#xA;B          200       21.0       US     typeB       2014-02-01  carlos&#xA;B          400       61.0      AUS     typeC       2014-03-01  carla&#xA;B          100       10.5      GER     typeA       2014-04-01  clive&#xA;C          100        9.5       UK     typeA       2014-01-01  clive&#xA;C          200       21.5       US     typeB       2014-02-01  chris&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an appropriate machine learning algorithm or method that can use the previous results of each athlete as a feature, when trying to predict the &lt;code&gt;time&lt;/code&gt; for an athlete in a future race? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;code&gt;athlete A&lt;/code&gt; has three races, with one month rest between them. In the third race he performs slightly better than the first race over the same distance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can an algorithm learn that the second race had an effect on the athlete, which meant he performed better in the third race?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I've read on the subject and the training examples I've completed it would appear that each 'row' of data should be independent, is this the case for all ML algorithms? Is there another prediction technique I should be considering to solve this type of problem?&lt;/p&gt;&#xA;" OwnerUserId="6683" LastActivityDate="2015-02-10T16:26:19.457" Title="machine learning on athlete performances to predict the time in a future race" Tags="|machine-learning|r|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5056" PostTypeId="2" ParentId="5055" CreationDate="2015-02-05T10:02:08.827" Score="3" Body="&lt;p&gt;By working with your features you could make the ML algorithm (maybe regression or SVR or whatever) to learn this fact (that sequental races are increasing the performance of athlete). To do this you may want to drop out the date column and introduce some new column, maybe 'race number' with 0 for first race, 1 for second, 2 for third etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such case regression model will be able to learn what you say 'that the second race had an effect on the athlete, which meant he performed better in the third race'. It is all about feature selection.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-02-05T10:02:08.827" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5058" PostTypeId="2" ParentId="5055" CreationDate="2015-02-05T17:22:04.123" Score="3" Body="&lt;p&gt;Ok, in this case, time is your dependent variables, and all the other ones are your features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should use linear regression (since more complex stuff needs more expertise), any machine learning library has that implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not use the date as a feature, those are usually lousy estimators.&lt;/p&gt;&#xA;" OwnerUserId="8065" LastActivityDate="2015-02-05T17:22:04.123" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5060" PostTypeId="2" ParentId="4992" CreationDate="2015-02-05T20:40:51.363" Score="4" Body="&lt;p&gt;I think I've worked something out. Basically I'm looking for an approach that works in a map/reduce type environment and I think this approach does it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;suppose I have b bands of r rows and I want to add another AND stage, say another c ANDs.&lt;/li&gt;&#xA;&lt;li&gt;so instead of b * r bits I need hashes of b * r * c bits&lt;/li&gt;&#xA;&lt;li&gt;and I run my previous procedure c times, each time on b * r bits&lt;/li&gt;&#xA;&lt;li&gt;If x and y are found to be a candidate pair by any of these procedures it emits a key value pair ((x, y), 1), with the tuple of IDs (x,y) as the key and the value 1&lt;/li&gt;&#xA;&lt;li&gt;At the end of the c procedures I group these pairs by key and sum&lt;/li&gt;&#xA;&lt;li&gt;Any pair (x,y) with a sum equal to c was a candidate pair in each of the c rounds, and so is a candidate pair of the entire procedure.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So now I have a workable solution, and all I need to do is work out whether using 3 steps like this will actually help me get a better result with fewer overall hash bits or better overall performance...&lt;/p&gt;&#xA;" OwnerUserId="8030" LastActivityDate="2015-02-05T20:40:51.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5061" PostTypeId="1" CreationDate="2015-02-05T21:18:01.527" Score="3" ViewCount="8006" Body="&lt;p&gt;I once knew some Java, but that was close to 10 years ago.  Assuming I can learn a language to get into data analytics.... what language do you recommend?  &lt;/p&gt;&#xA;" OwnerUserId="8123" LastEditorUserId="41" LastEditDate="2015-02-06T19:45:21.850" LastActivityDate="2021-04-15T17:09:01.203" Title="How much of a background in programming is necessary to become a data scientist?" Tags="|bigdata|beginner|" AnswerCount="6" CommentCount="0" ClosedDate="2015-02-07T00:38:20.700" ContentLicense="CC BY-SA 3.0" />
  <row Id="5062" PostTypeId="2" ParentId="5061" CreationDate="2015-02-05T23:22:08.800" Score="1" Body="&lt;p&gt;Based on this infographic and other things I've read, it sounds like you need to know some coding to be a true data scientist. &lt;a href=&quot;http://blog.datacamp.com/how-to-become-a-data-scientist/&quot; rel=&quot;nofollow&quot;&gt;http://blog.datacamp.com/how-to-become-a-data-scientist/&lt;/a&gt; But you could still be a data &lt;strong&gt;analyst&lt;/strong&gt; without compsci - basically a statistician. &lt;/p&gt;&#xA;" OwnerUserId="8126" LastActivityDate="2015-02-05T23:22:08.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5063" PostTypeId="2" ParentId="5061" CreationDate="2015-02-05T23:23:04.180" Score="0" Body="&lt;p&gt;The R language is the best place to start. Grab some open datasets, and start programming with r. R has many different analytical functions that you can learn a lot with it.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2015-02-05T23:23:04.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5064" PostTypeId="1" CreationDate="2015-02-05T23:57:05.757" Score="0" ViewCount="52" Body="&lt;p&gt;I am not certain, if this is the right place to ask the following question. I am looking for some practical scenarios in social networks where the following information propagation model can arise:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/mERQh.jpg&quot; alt=&quot;A toy example of the desired information propagation model&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have a source node and some information propagating radially from it and each recipient receives the information from a single sender.&lt;/p&gt;&#xA;" OwnerUserId="8127" LastActivityDate="2015-02-08T16:05:58.517" Title="Where in practice can we see following information propagation model" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5065" PostTypeId="2" ParentId="5061" CreationDate="2015-02-06T00:05:02.313" Score="2" Body="&lt;p&gt;Data Scientists code every day. However, just because you don't have background doesn't mean you can't pick it up! The level of programming you need to know to start doing Data Science isn't very high, but you will at least need:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the logical mindset to phrase the solution to your problem in&#xA;procedural code&lt;/li&gt;&#xA;&lt;li&gt;to know the programming language, functions, and libraries needed in this field.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;1st point is the most difficult of the two. Hopefully, you have taken enough math and physics by now to wire your mind to think programmatically. If so then yes, you absolutely can learn a language! There are guides out there that teach out the syntax and functions. For example:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;R - &lt;a href=&quot;http://tryr.codeschool.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pluralsight&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;General Python - &lt;a href=&quot;http://www.codecademy.com/en/tracks/python&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.codecademy.com/en/tracks/python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;DataSci Python - &lt;a href=&quot;https://www.kaggle.com/c/titanic-gettingStarted/details/getting-started-with-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/titanic-gettingStarted/details/getting-started-with-python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Personally, I would recommend Python first. To me the language places more emphasis on readability and cleanliness, making it a great first language. It's also a general-purpose language so it's good to know. I did start with R though and it's also good but is more function-over-form IMO. Try both out and see which feels best first, since you'll likely have to pick up both if you delve into this field anyway.&lt;/p&gt;&#xA;" OwnerUserId="525" LastEditorUserId="85045" LastEditDate="2021-04-15T17:09:01.203" LastActivityDate="2021-04-15T17:09:01.203" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5066" PostTypeId="2" ParentId="5050" CreationDate="2015-02-06T00:47:49.187" Score="2" Body="&lt;p&gt;It sounds to me like you have a binary classification problem (classifying good vs. bad documents for some definitions of good and bad) and the words are being used as features or &quot;signals&quot; for what predicts good vs. bad documents. One thing you might try is to measure some type of correlation statistic between unigrams and each class you're interested in. This preserves your requirement of measuring occurrences of words given a target class over &lt;em&gt;groups&lt;/em&gt; of documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to be a bit more concrete, you could split your documents into two sets (good and bad), and then tokenize your documents to obtain individual terms. From here you could really choose whichever term weighting scheme you like (TF, TF normalized against the length of the document, TF-IDF) and measure your correlation statistic between &lt;em&gt;all these unigrams&lt;/em&gt; and the class of interest. You could then produce a ranking based on the correlation coefficients for each term, and take the top-&lt;em&gt;k&lt;/em&gt; terms. Some correlation statistics you might try could be &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test&quot; rel=&quot;nofollow&quot;&gt;Chi-squared&lt;/a&gt; (which would measure &quot;lack of independence&quot; between terms and a class). There's also a nice implementation of Chi-squared test for feature selection in &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection&quot; rel=&quot;nofollow&quot;&gt;Python's Scikit-Learn&lt;/a&gt; machine learning library that may be a place to start for this task. Hopefully, that helps!&lt;/p&gt;&#xA;" OwnerUserId="4897" LastActivityDate="2015-02-06T00:47:49.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5067" PostTypeId="2" ParentId="5061" CreationDate="2015-02-06T05:51:56.090" Score="0" Body="&lt;p&gt;Johns Hopkins University as a set of course on Coursea that is gear on Data Science. Here is the link to the classes &lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1?utm_medium=courseDescripTop&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specialization/jhudatascience/1?utm_medium=courseDescripTop&lt;/a&gt;. You can also take the classes for free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a set of 9 classes that would give you a good foundation to build-on and to start a career.&lt;/p&gt;&#xA;" OwnerDisplayName="user3588" LastActivityDate="2015-02-06T05:51:56.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5068" PostTypeId="2" ParentId="5061" CreationDate="2015-02-06T06:11:35.397" Score="0" Body="&lt;p&gt;Data science, being a new term, covers a broad spectrum of jobs. At one end you are expected to write production code. At the other end you do statistics in packaged software. They also call such people statisticians or analysts. So decide what you enjoy doing before you leap. If you just want to analyze data, you could definitely get by with R or python as long as you're mathematically proficient. I find that in these kind of jobs, your communication and social skills matter too, since you have to explain the data to executives and the like.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-02-06T06:11:35.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5070" PostTypeId="2" ParentId="5061" CreationDate="2015-02-06T09:06:02.607" Score="7" Body="&lt;p&gt;First of all, the fact that you have known some Java, even ten years ago, already means that you don't &quot;know nothing about programming&quot; (I suggest you update the title of your question to reflect that - change &quot;nothing&quot; to &quot;a little&quot;). I'd like to make several points, which I hope will be useful to you.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In terms of the level of &lt;em&gt;programming proficiency&lt;/em&gt;, which is expected (needed) for a data scientist, the following popular &lt;strong&gt;definition&lt;/strong&gt; says it all:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A &lt;em&gt;data scientist&lt;/em&gt; is someone who is better at statistics than any&#xA;  software engineer and better at software engineering than any&#xA;  statistician.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Another perspective on the role of programming abilities in a data scientist's skill set can be found in a popular visual representation of data science, using Venn diagrams. The original &lt;strong&gt;data science Venn diagram&lt;/strong&gt; was presented by data scientist Drew Conway (see &lt;a href=&quot;http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram&quot; rel=&quot;noreferrer&quot;&gt;this blog post&lt;/a&gt;):&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZwWt2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Since its original introduction, the original diagram was modified by various people for various reasons. The two interesting &lt;strong&gt;adaptations&lt;/strong&gt; are for data science in the social sciences domain (&lt;a href=&quot;http://www.datascienceassn.org/content/fourth-bubble-data-science-venn-diagram-social-sciences&quot; rel=&quot;noreferrer&quot;&gt;http://www.datascienceassn.org/content/fourth-bubble-data-science-venn-diagram-social-sciences&lt;/a&gt;), as well as data science Venn diagram V2.0, where data science is represented not as an intersection of knowledge domains, but as their union (&lt;a href=&quot;http://www.anlytcs.com/2014/01/data-science-venn-diagram-v20.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.anlytcs.com/2014/01/data-science-venn-diagram-v20.html&lt;/a&gt;). Another very interesting and useful visual perspective of data science skill set, also based on Venn diagram, is the following Gartner's diagram, &lt;strong&gt;mapping&lt;/strong&gt; specific &lt;em&gt;skills&lt;/em&gt; to business intelligence (BI) or business analytics &lt;em&gt;knowledge domains&lt;/em&gt;:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/XBbDM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;An alternative perspective for a data scientist's skill set and domain knowledge is a &lt;strong&gt;taxonomy of data scientists&lt;/strong&gt;, such as &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/six-categories-of-data-scientists&quot; rel=&quot;noreferrer&quot;&gt;this taxonomy&lt;/a&gt;, which classifies data scientists, according to their &lt;em&gt;focus&lt;/em&gt; (or the &lt;em&gt;strongest skill set&lt;/em&gt;): mathematics, data engineering, machine learning, business, software engineering, visualization, spacial data (GIS) or others.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you're curious about the meaning of the &lt;strong&gt;&quot;Danger Zone&quot;&lt;/strong&gt; in the original data science Venn diagram, &lt;a href=&quot;http://www.quora.com/In-the-data-science-venn-diagram-why-is-the-common-region-of-Hacking-Skills-and-Substantive-Expertise-considered-as-danger-zone&quot; rel=&quot;noreferrer&quot;&gt;this Quora discussion&lt;/a&gt;, containing, among other nice answers, also an answer by the original diagram's author, can be very helpful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you're interested in learning about a range of &lt;strong&gt;skills&lt;/strong&gt; and &lt;strong&gt;knowledge domains&lt;/strong&gt;, useful for a data scientist, check this open source &lt;em&gt;curriculum&lt;/em&gt; for learning data science: &lt;a href=&quot;http://datasciencemasters.org&quot; rel=&quot;noreferrer&quot;&gt;http://datasciencemasters.org&lt;/a&gt;, or on GitHub: &lt;a href=&quot;https://github.com/datasciencemasters/go&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/datasciencemasters/go&lt;/a&gt;. Of course, popular and research papers, lectures on YouTube, MOOC courses, online and offline bootcamps as well as a wealth of other resources is only an Internet search away.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, a note on &lt;strong&gt;programming languages&lt;/strong&gt; for data science. I think that it is important to understand that this aspect is really of secondary importance. The &lt;strong&gt;focus&lt;/strong&gt; should be on two words, which the term &quot;data science&quot; consists of: &lt;em&gt;data&lt;/em&gt; and &lt;em&gt;science&lt;/em&gt;. &lt;strong&gt;Focus on data&lt;/strong&gt; means that it is important to think about data science (or BI, or analytics) tasks in terms of the corresponding &lt;em&gt;domain knowledge&lt;/em&gt; as well as to pay attention to &lt;em&gt;data quality&lt;/em&gt; and &lt;em&gt;representativeness&lt;/em&gt;. &lt;strong&gt;Focus on science&lt;/strong&gt; means adhering to &lt;em&gt;scientific approaches&lt;/em&gt; to data collection and analysis, of which &lt;em&gt;reproducibility&lt;/em&gt; plays an important role. A &lt;em&gt;programming language&lt;/em&gt; for data science is just a &lt;strong&gt;tool&lt;/strong&gt; and, therefore, should be chosen to match the task at hand. Python and R represent very good and the most popular programming languages and environments for a data scientist, however, you should be aware of other options (&lt;em&gt;tool set&lt;/em&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-02-06T09:22:15.570" LastActivityDate="2015-02-06T09:22:15.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5071" PostTypeId="2" ParentId="5038" CreationDate="2015-02-06T09:41:54.033" Score="3" Body="&lt;p&gt;The problem is with division on python 2.x.  In python 2.x, division involving two integers produces an integer result.  So &lt;code&gt;1/2==0&lt;/code&gt;.  Python 3.x does not have this problem, &lt;code&gt;1/2==.5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two ways to avoid this.  First, you can always convert one value you a float.  So &lt;code&gt;1./2==0.5&lt;/code&gt; and &lt;code&gt;1/2.==0.5&lt;/code&gt;.  However, you have to remember to do this everywhere, and if you forget it can lead to hard-to-find errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more reliable method is to always put this at the top of your code: &lt;code&gt;from __future__ import division&lt;/code&gt;.  This will switch python 2.x to the python 3 behavior, so &lt;code&gt;1/2==.5&lt;/code&gt;.  In python 3.x it does nothing, so it also makes your code python 3.x compatible in this regard.&lt;/p&gt;&#xA;" OwnerUserId="8110" LastActivityDate="2015-02-06T09:41:54.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5073" PostTypeId="1" CreationDate="2015-02-06T17:52:05.363" Score="7" ViewCount="1840" Body="&lt;p&gt;I want to build a chatbot that serves as a first line customer support on a  retail website. I have a large log of chat sessions between customers and support professionals that I can use. I am wondering what is the best way to go about building this chatbot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some ideas I have looked into;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first thing I did was to breakdown the chat logs into Q&amp;amp;A pairs. Treat each question as a document, compute a term-document matrix and use it to retrieve the most similar question when a customer asks sometihng. The idea was to then simply pick the answer given to the question by the human (with some modifications). However, this gives really abysmal results and does not match to previous questions very well. Even if I get this approach to work well, we would be limited to the questions that already been asked. &lt;/li&gt;&#xA;&lt;li&gt;Another thing I thought about was to modify something like ALICE and write some AIML for customer support related QA. However, that would take a lot of very precise AIML writint to work well. This solution would not be able to scale to other languages, which is something I want to do.&lt;/li&gt;&#xA;&lt;li&gt;Another idea I have is to try and cluster the answers given by the customer support staff (they are more likely to be aligned compared to the questions asked by customers). This would give me a sense of what questions are similar. Then I can use something like LSA on the questions and fall back to the first approach.&lt;/li&gt;&#xA;&lt;li&gt;Yet another way is to build an ontology of the domain specific knowledge and given a question decide which category the question falls into. The questions from the chat logs can then be mapped to the ontology and I could train classifiers for mapping questions to different knowledge areas. Once I reach the leaf nodes, I can give back pre-defined answers. However, my reservation is that mapping chat logs to the ontology would be very tedious. Is there a good way to map the existing QA to the ontology?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8145" LastActivityDate="2018-09-16T14:20:41.747" Title="Question and Answer Chatbot for Customer Support" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5074" PostTypeId="1" CreationDate="2015-02-06T18:13:16.210" Score="0" ViewCount="215" Body="&lt;p&gt;I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you help me to choose an appropriate one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please ask if the description is not clear.&lt;/p&gt;&#xA;" OwnerUserId="8146" LastActivityDate="2015-04-08T04:21:19.443" Title="MPI, MapReduce, or Spark for complex datasets and processing" Tags="|map-reduce|parallel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5077" PostTypeId="2" ParentId="5074" CreationDate="2015-02-07T02:41:06.783" Score="1" Body="&lt;p&gt;If I understand your question correctly. What you want to do is &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Write a sql query against to a database to get the processed/aggregated results first.&lt;/li&gt;&#xA;&lt;li&gt;Execute several queries in parallel.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first thing first, if you want to run MapReduce programs on your data, you need to distribute your data. In addition, MapReduce doesn't run your queries to different nodes separately. If your $query_{A}$ runs on $data_{A}$, what we do here is to divide $data_{A}$ to different nodes and run $query_{A}$ on these nodes on $PartOfData_{A}$ in parallel. By the way, you can use Sqoop to migrate the data from your DB to HDFS(it is used to store data separately).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are steps for you:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;migrate all/part of your data to HDFS&lt;/li&gt;&#xA;&lt;li&gt;transform your queries to MapReduce version program&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1003" LastActivityDate="2015-02-07T02:41:06.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5078" PostTypeId="2" ParentId="5047" CreationDate="2015-02-07T07:37:14.040" Score="1" Body="&lt;p&gt;The reason your OCR program is not recognizing anything is that the font size is very small. If you resize your image to increase its size, you will obtain better results. For example, using ImageMagick to apply a fixed threshold to remove the background on your first image and increase its size:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;convert -density 500 -threshold 40% 29-1891a.gif -resize 250% output.tiff&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this, tesseract does a reasonable job:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tesseract output.tiff output test.config&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I included a configuration file &lt;code&gt;test.config&lt;/code&gt; to tesseract in order to restrict the permitted characters. In this way, we don't obtain mistaken unicode characters in our text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;test.config&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tessedit_char_whitelist ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;HOWARD COUNTY 10011607&#xA;&#xA;NALL JOHN E 0-686196 1 LT F05 DALE MARVIN E 30037267 PVT DON SANSJNG DAVID 10505969 PFC&#xA;NIXSON EONOND R 10077072 CPL KIA DANIEL NILBORN N 30637064 PVT FOB SAUNDER6 NEWTON L 0-401709 CAPT&#xA;NOBLES STEPNEN E 30502791 PVT 0N0 DAVIS IR D 30433764 PVT 0N0 SNELTON CHARLES R 14001742 VT&#xA;OSBURN SILLY F 02008518 2 LT DNB BAVIS JINKIE L 18214208 5 SC KIA SHELTON CHARLIE 33423953 7335&#xA;PELTON J L 30105473 5 SC RNA 0 DEAN CHARLES 02066314 3 LT 0N0 SMERAN CARLTON A JR 0-510441 2 LT&#xA;GUEZADA FRANK 5 10014939 PFC KIA IDIXON JOHN N 6202114 PVT 00R SIBLEV JESSE 30037241 PFC&#xA;RAMIREZ JOSE T 30347914 PPC KIA DREADIN RAYMOND 6950964 PVT KIA SLATER JERONE E JR 0-605266 1 LT&#xA;OSE THO S 0 30345100 PPC NTA - DUNN GARLAND J 30037202 PVT 0N0 SMITH 00 SE 30509104 PPC&#xA;R035 ORREN C 16015376 PVT ONE 1 ELDER GERALD P 0-387706 2 LT KIA SNITN JASPER 7 33037355 7 SC&#xA;RUTLEDCE CARL R 38107116 PVTELPNAA JEARHER MARVIN R 30279097 CPL DON SMITN JERALO D 10005000 CPL&#xA;SCUDDAY BERNIE L 0-603906 1 LT KIA - FINDLEY PAUL A 50536010 PFC OMB SMITH NELDON A 5550735 PFC&#xA;SNITH ROBERT L 10015524 507 ONE- ELINC ROY T I 0-724730 CAPT F05 S THEY 4295431 5 5&#xA;SNITH TRAVIS L 30670530 PVT IDUN FORD HERRELL E 0-665672 CAPT E00 STEPN NSON JESSIE P 10006200 PVT&#xA;SNEED ROY A 13076927 S 80 ONE GARRETT TOMMY 30220694 PFC KIA SULLIVAN 31 01525125 2 L&#xA;SOUTN CARL 0 JR 30343379 PVT OMB CATLOR R T 20012665 SOT KIA SNINDELL VERL O 30037252 7005&#xA;STEVENS JANESCO 10015557 PVT DNR CLASSCOCN CNARLES J 6571000 5 SO ONO TAYLOR ARTHUR V 33531479 PFC&#xA;STENA D JO 0-723036 2 LT POL 7 00 A 10007027 PVT DOM ITOHN E N A JR 30035251 5 SO&#xA;UTTON ARVI 6360431 AV C 0N6 COSSETT JANES N 30436055 CPL KIAI ALKER J NEE H 30117597 SOT&#xA;IALBOTT CHARLES 9 38342083 PFC KIA GRESN J 18126923 5 SO KIA NALKER RALPH L 203126071PPC&#xA;TUCKER JAMES 30848763 PVT MIA CRIFFIS WILLIAM J 6270119 PFC 0N0 NALLACE JESSE A 30043713 T SC&#xA;-TUCKER STERLING P 33341143 RPCI KIA CROSS ELERY C 0-390713 3 LT DNR NARREN CHARLES D 30110253 5 50&#xA;NAOSNORTN PAUL P 30345079 5 SC ONE HAKNONBS ROBERT M 38433940 3 SC KIA MASHINCTON HENRY 30299100 PFC&#xA;NALNER JAN S H JR 0-696023 2 LT P0 HANDLSY JAMES J JR 0-417955 1 LT ONE EEMS NINPRED E 20017930 CPL&#xA;NEBB GLEN 30343104 PVT -K1 7 NANET FARRELL 8 30012597 SCT KIA NNITE DE NIS 10124415 PFC&#xA;NRAT JAMES H 30067743 TE05 KIA NARCIE PRANCNARD 6370053 1 SC NIA NMITE MARVIN J 30430974 PFC&#xA;NRIOHT NAILAND 0 30609570 PFC KIA NARKEY VEWCEN 30424190 PVT KIA NOODARD BILLY E 510217472 T 5&#xA;HARRIS PRBS 38111537 PFC OKIA NRICNT EILL 0-725355 CAPT&#xA;I HARRISO DUKE N JR 18055432 AV 0 ONE YOST TNURNAN R 6273153 PVT&#xA;HENDRIX JOHN N JR 10217563 PVT 0N8&#xA;HICKERSDN JACK 04431540 1 LT 9N8&#xA;HUDSPETH COUNTY 3536305354 A CHEW 55555 W&#xA;9 0776 A -&#xA;- JOHNSTON LONNIE O 0-754906 3 LT KIA H INSON COUNTY&#xA;ONES GEORBE N 01173315 1 LT ONE UTCH&#xA;JUMPER ISAAC H 30605960 PVT KIA I&#xA;A9515 HAN C -7 7 3 LT N5 LONG OSCAR D 30812573 5 SC KIA ALEXANDER BOYD A 0-690739 2 LT&#xA;EARDNERNE3SEPN H 3L5E3355 3 L7 335 LVTLE JOHN E 30433946 PVT KIA EALDNIN JANES H 0-407114 CAPT&#xA;NORALES ALFREDO L 18015539 3 CG P05 MACK HULET 00057345 1 LT NIA EICCERSTAPP C N 34530001 PC&#xA;54N1952 LOCAS N 39441440 PFC K14 NAJORS TRUSTT J 00410046 2 LT KIA BRITTON JAMES H 18036279 8 SC&#xA;ROBLES VICENTE 30570733 PFC 0N5 MASON DICE 30203437 CPL NIA EULLARD CAR 5 30607623 PVT&#xA;SANCNE2 ANGEL R 38310341 PVT ONE MASON HALTER P 0-671673 3 LT NIA C IN ROBE T E 33105799 TECS&#xA;VALLES RENICIO 0 30441430 PIC DON NASSEV JESSE D 30435424 PVT KIA COHA K LL03 N 10077256 PVT&#xA;NC CLENDON J H 30117530 PPC ONE EVANS L 0 37392406 TECS&#xA;NC HNORTER C R 30300391 PVT KIA FIRLEY MARVIN L 38572402 PVT&#xA;4 HILLNAN ODEAN R T-000345 FL 0 DNB IPIELDS JAKES 35711546 PVT&#xA;HIL10N JACKSON 7 13030591 PFC 933 ORADDY ROY L 39112920 PFC&#xA;HUNT COUNTY HOORE ARLON D 6295620 9 SC KIA GRANT 30572401 PVT&#xA;MORRISON DURHARD D 0-519411 3 LT KIA HANNA EVERETT T 10104243 S SC&#xA;- LNULLINS GERALD D 6295622 T 50 NIA HANSARD SAMUEL 2 04754619 1 LT&#xA;NEAL HOMER M 30609200 PVT KIA THARVEY JOHN B 0-602116 2 LT&#xA;ALANIS VICENTE 30894642 VT KIA MEAL RAVN 39 9 1759550 3 NECOAL MARTIN J 6396940 PFC&#xA;ALLEN TRUHAN L 0-123936 1 LT KIA NELSON 1 0 30431073 PPET ONE HILL RAYMONO 8 30050142 PVT&#xA;ALLEY NILBUR K 01703993 2 LT KIA NELSON TRAVIS C 38204649 PVT DOH HOP JACK H D 30342393 PFC&#xA;BENCH CLARENCE A 30003600 PFC KIA INEWLAND OTIS T T-000346 FL 0 F051 HDFF ROBERT C 30345133 T 50&#xA;BENNETT EVERETT N 0-692130 LT 0N5 NICNOLSON DALE- 10005042 PVT KIA JONES JANES 0 30343505 PVT&#xA;I I&#xA;A NIXON LOYD 5041674 TECS KIA KAPPELMN MC 0-360687 CAPT&#xA;BLACKHELL E C 38357321 307 KIA PARKER N 33130N V 52959524 3 L7 KIA KECANS TIM JR 38342898 FC&#xA;ERITT BASIL JR 36002307 PFC K14 PATTERSON THOMAS H 18136913 SCT 9N5 KENNINER EARL 0 30401408 SGT&#xA;BROHN SHOE 8 38634396 PFC KIA PERRI JA 5 0-562333 1 LT KIA LANTRON EDWARD L 19190302 5 SO&#xA;BURNS VIRCIL P 38035701 PRC KIA PETTICREH FRED 0 16215700 S 50 NIA LESNER ROLLIE H 30050735 AV C&#xA;BUTLER GEORGE A 37259433 1 SC KIA PHILLIPS C L JR 0-431139 CAPT KIA NC CARTY TOURNAN P 30711042 PFC&#xA;CAMERON VANDSLL C 30634570 PPCI KIA PILCRTN CLYDE 30115026 V NIA NC GLENDON JACK 30330395 PFC&#xA;CARTER LEO RD K 30119959 AV 0 ONE PO D EUCENE N 4 30431074 PRC KIA NC NINNET K E 30340151 PVT&#xA;CARTER OILLIAH F 38685535 PFC 0N8 PRESLEY NILLIAN H 30037547 PVT 0N5 NC QUEEN JAMES Y 02055059 2 LT&#xA;CREEK LOTD 6379416 A SC DNB PRICE PREDRICK P 30409331 CPL 100W NOTEN GEORGE N 30304695 PFC&#xA;CLARK JOHN 18317907 CPL KIA PURCELL SAMUEL N 30017905 PVT 0N5 PANNELL NOLENIA 6227623 1 SC&#xA;COLLINS RAYMOND N 30137040 PFC OOH RAILIPF NARDEN A 30037006 CPL 00H PIERCE FELIX 0 30335390 5 SC&#xA;CREAKER SAH BL N 18006101 PVT KIA HAYNES NILLIAN T 7-954739 PL 0 9N5 PIETZSCH JAI E 0-426961 2 LT&#xA;CRIDER NARLAN 0 10154794 SCT KIA REED JDRN L 30529414 PVT POL RTER NALLACE N JR 38193560 CPL&#xA;DALE CHARLEY 8 I 38049500 3 SC KIA ROI NILLIS 5 01393956 1 LT KIA PRESCOTT DAVID L JR 0-707191 2 LT&#xA;05 231&#xA;&#xA;&#xA;&#xA;D KZXXN D&#xA;NNNNH 2&#xA;XXLA-A- Q&#xA;&#xA;2&#xA;33&#xA;&#xA;UR R&#xA;2055&#xA;35&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, there are many mistakes, but at least we are obtaining reasonable results. In any case, you don't want to do this because even if you were able to get perfect recognition, some of your lines would be placed in the wrong county because tesseract is looking at your document as a big paragraph. I recommend you to use the vertical lines to segment your image into 3 parts and then preprocess each one of them. You can even try to concatenate the parts vertically and perform OCR in a single page. This also applies to the second image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the resolution of your image is not great, so if you can get a better image, that is going to make a big difference. &lt;/p&gt;&#xA;" OwnerUserId="4621" LastActivityDate="2015-02-07T07:37:14.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5079" PostTypeId="1" CreationDate="2015-02-07T14:21:42.543" Score="5" ViewCount="932" Body="&lt;p&gt;I have an item-item similarity matrix. e.g. (the matrix is symmetric, and much bigger):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.00 0.88 0.96 0.99 &#xA;0.88 1.00 0.99 0.96 &#xA;0.96 0.99 1.00 0.86 &#xA;0.99 0.96 0.86 1.00 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to implement recommender which, for a set of items, recommends a new set of items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about using SVD to reduce the items to n-dimensional space, let's say 50-dimensional space, so each item is represented with a vector 50 numbers, and similarity between two items is calculated by cosine similarity between two 50-dimensional vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a base set of items (which can get quite big), I hope I could calculate an average of their vectors, and use it for search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a good idea? What is this procedure called? And can it be done in Mahout?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my code so far:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ItemSimilarity similarity = new LogLikelihoodSimilarity(model);&#xA;Matrix m = new DenseMatrix(NUM_ITEMS, NUM_ITEMS);&#xA;// copy similarities to a matrix&#xA;for (int i = 0; i &amp;lt; NUM_ITEMS; i++) {&#xA;        double[] similar = similarity.itemSimilarities(i, range(NUM_ITEMS));&#xA;        for (int j = 0; j &amp;lt; NUM_ITEMS; j++) {&#xA;            m.setQuick(i, j, similar[j]);&#xA;    }&#xA;}&#xA;Matrix v = new SingularValueDecomposition(m).getV();&#xA;Matrix reduced = v.viewPart(0, NUM_ITEMS, 0, 50);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is, SVD is taking forever for NUM_ITEMS &gt; 30. I don't know if there is an issue with data, or with SVD implementation I'm using. The matrix m is symmetrical, could that be an issue? I tried googling &quot;demean matrix mahout&quot; with no results. How should I preprocess it for SVD to work faster? I will need NUM_ITEMS to be about 20.000 - 40.000 in the future. Is this reasonable size for SVD?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem was the matrix contained a few NaN values, that's why SVD was taking infinite time. After replacing these with 0.0 it works fine for 1000 x 1000 matrix. And my recommendations are working like a charm. I'll still need compute SVD of 20x more rows and columns. If anyone knows what's the easiest way to compute (approximate) SVD of 20.000 x 20.000 dense matrix, probably through some cloud parallel service (?), please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. Thanks for help!&lt;/p&gt;&#xA;" OwnerUserId="8158" LastEditorUserId="8158" LastEditDate="2015-02-08T23:14:53.843" LastActivityDate="2015-03-26T03:39:33.163" Title="Item based recommender using SVD" Tags="|recommender-system|apache-mahout|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5080" PostTypeId="1" CreationDate="2015-02-07T15:37:47.327" Score="3" ViewCount="6500" Body="&lt;p&gt;Hello I work as data scientist for a private company.&#xA;I am interested in working for a nonprofit company, such as a research institute (public or private) or a company that takes care of  issues such as  environment, public health,  social improvements. Even an internet company like Wikipedia can be interesting.&#xA;Does anybody know if nonprofit companies hire data scientists?&lt;/p&gt;&#xA;" OwnerUserId="3346" LastEditorUserId="21" LastEditDate="2015-02-07T18:37:01.737" LastActivityDate="2015-02-10T21:13:37.107" Title="Working as data scientist for a nonprofit company" Tags="|career|" AnswerCount="2" CommentCount="3" ClosedDate="2015-02-11T02:06:50.447" ContentLicense="CC BY-SA 3.0" />
  <row Id="5081" PostTypeId="2" ParentId="5080" CreationDate="2015-02-08T01:47:57.963" Score="6" Body="&lt;p&gt;I see at least five ways to approach this problem of finding a data scientist position/work specifically at non-profit, non-governmental or similar organizations, as I describe below. I hope that this is helpful.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;First, and the most obvious, way is to search &lt;strong&gt;major job portals&lt;/strong&gt;, such as indeed.com, dice.com, monster.com, CareerBuilder, Glassdoor and others, for &lt;em&gt;data scientist&lt;/em&gt; or similar positions, such as &lt;em&gt;data analyst&lt;/em&gt;, &lt;em&gt;data engineer&lt;/em&gt;, &lt;em&gt;quantitative analyst&lt;/em&gt;, &lt;em&gt;statistical modeler&lt;/em&gt;, or even &lt;em&gt;market researcher&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The second and also obvious way is to tap your &lt;strong&gt;professional social networks&lt;/strong&gt; and research or inquire your contacts for any potential data science work opportunities in the areas of your interests.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The third way is to search web sites, &lt;strong&gt;focused&lt;/strong&gt; on the non-profit and related topics (they usually have a job listings or similar section) as well as &lt;strong&gt;specialized non-profit job portals&lt;/strong&gt;. For example: &lt;a href=&quot;http://encore.org&quot; rel=&quot;noreferrer&quot;&gt;http://encore.org&lt;/a&gt;, &lt;a href=&quot;http://idealist.org&quot; rel=&quot;noreferrer&quot;&gt;http://idealist.org&lt;/a&gt;, &lt;a href=&quot;http://bridgespan.org&quot; rel=&quot;noreferrer&quot;&gt;http://bridgespan.org&lt;/a&gt;, &lt;a href=&quot;http://cgcareers.org&quot; rel=&quot;noreferrer&quot;&gt;http://cgcareers.org&lt;/a&gt;, &lt;a href=&quot;http://opportunityknocks.org&quot; rel=&quot;noreferrer&quot;&gt;http://opportunityknocks.org&lt;/a&gt;, &lt;a href=&quot;http://foundationcenter.org&quot; rel=&quot;noreferrer&quot;&gt;http://foundationcenter.org&lt;/a&gt;, &lt;a href=&quot;http://thenonprofittimes.com&quot; rel=&quot;noreferrer&quot;&gt;http://thenonprofittimes.com&lt;/a&gt;, &lt;a href=&quot;http://philanthropy.com&quot; rel=&quot;noreferrer&quot;&gt;http://philanthropy.com&lt;/a&gt;, &lt;a href=&quot;http://ynpn.org&quot; rel=&quot;noreferrer&quot;&gt;http://ynpn.org&lt;/a&gt;, &lt;a href=&quot;http://philanthropyjournal.org&quot; rel=&quot;noreferrer&quot;&gt;http://philanthropyjournal.org&lt;/a&gt;, &lt;a href=&quot;http://nonprofit-jobs.org&quot; rel=&quot;noreferrer&quot;&gt;http://nonprofit-jobs.org&lt;/a&gt;, &lt;a href=&quot;http://jobs.cof.org/home/index.cfm?site_id=11690&quot; rel=&quot;noreferrer&quot;&gt;CoF Jobs&lt;/a&gt; (section isn't easy to find), &lt;a href=&quot;http://careers.councilofnonprofits.org&quot; rel=&quot;noreferrer&quot;&gt;http://careers.councilofnonprofits.org&lt;/a&gt;, &lt;a href=&quot;http://nonprofittalentmatch.com&quot; rel=&quot;noreferrer&quot;&gt;http://nonprofittalentmatch.com&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The fourth way is to perform &lt;strong&gt;Internet search and research&lt;/strong&gt; on non-profit or similar organizations that you might be interested in working with (criteria might vary from organization's size to industry focus or geographical locations). Based on the information presented  on their websites, make notes and then approach those organizations, &lt;strong&gt;directly&lt;/strong&gt; inquiring or applying for positions of interest.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The fifth way is to consider various non-profit, for-profit and otherwise social good-themed &lt;em&gt;data science-focused&lt;/em&gt; &lt;strong&gt;organizations, initiatives and Kaggle-like competitions&lt;/strong&gt;, such as &lt;a href=&quot;http://www.datakind.org&quot; rel=&quot;noreferrer&quot;&gt;DataKind&lt;/a&gt;, &lt;a href=&quot;http://www.drivendata.org&quot; rel=&quot;noreferrer&quot;&gt;DrivenData&lt;/a&gt;, &lt;a href=&quot;http://datalook.io&quot; rel=&quot;noreferrer&quot;&gt;DataLook&lt;/a&gt; and &lt;a href=&quot;http://dssg.io&quot; rel=&quot;noreferrer&quot;&gt;Data Science for Social Good&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-02-08T01:47:57.963" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5082" PostTypeId="2" ParentId="5079" CreationDate="2015-02-08T03:29:06.623" Score="3" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;I need to implement recommender which, for a set of items, recommends a new set of items.&lt;/p&gt;&#xA;&lt;p&gt;Is this a good idea?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Have you looked into &lt;a href=&quot;http://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Association Rules Mining&lt;/a&gt;? If you're open to other procedures this one is first one that came to my mind for recommenders based on sets of items. For those not familiar, this is simple method for retail shops to determine &amp;quot;75% of customers who bought A, B also bought C&amp;quot;. Within these algos, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apriori algo&lt;/a&gt; is straight-forward, easy to implement, and may get you what you need.&lt;/p&gt;&#xA;" OwnerUserId="525" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-02-08T03:29:06.623" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5084" PostTypeId="2" ParentId="5064" CreationDate="2015-02-08T16:05:58.517" Score="1" Body="&lt;p&gt;Assuming the central node is at level 0 (root), this graph becomes a tree. Now it is easy reason about the different types of trees that can be formed and different types of social networks that can be modelled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order for the graph to be a tree, communication between nodes has to be restricted, such that the tree structure is not violated (e.g. two nodes on the same level cannot communicate). I can't think of any social network where communication is naturally restricted in this way, but such a social network can be created artificially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples of such artificial social networks would commonly follow a chain-of-command structure. For example in a military setting, the official information flows would have a tree structure.   &lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2015-02-08T16:05:58.517" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5085" PostTypeId="1" CreationDate="2015-02-08T18:05:34.020" Score="0" ViewCount="96" Body="&lt;p&gt;If I have virtually endless training data (it's synthesized) is there still purpose in having epochs? I.e. training on the same samples multiple times?&lt;/p&gt;&#xA;" OwnerUserId="7806" LastActivityDate="2015-02-11T07:56:48.717" Title="Should I use epochs &gt; 1 when training data is unlimited?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5086" PostTypeId="2" ParentId="5085" CreationDate="2015-02-08T21:03:52.657" Score="1" Body="&lt;p&gt;If the data is unlimited, how would you have an epoch to begin with? For example, if you are analyzing tweets, you could never finish an epoch will &lt;em&gt;all&lt;/em&gt; the tweets, since there will be an endless supply of new tweets. A much better approach will be to do some online or streaming learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it make sense to create a subset by ignoring new incoming tweets or data in general?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That really depends on your problem. For example if you have a datastream of tweets, or any other where new data is being produced in real time, you would miss out any trends or patterns that emerged after you sampled from the stream. Are those missed patterns relevant to your problem? They may or may not be.    &lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2015-02-08T21:03:52.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5087" PostTypeId="2" ParentId="5073" CreationDate="2015-02-08T21:17:50.990" Score="5" Body="&lt;p&gt;This was intended as a comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this question is not receiving enough attention because it is too difficult to answer without trying several approaches first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your third idea is a nice one, but LSA is most likely not going to be able to help you choose clusters with the necessary granularity. For example, you may get a cluster of &quot;broken things&quot; but not to the specificity that is required to be helpful to the customer. A similar approach would be using Hierarchical Dirichlet Processes, but again, you may not get the clusters that you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't be so eager to try your fourth option because it requires a lot of work and I'm not sure you're going to get great results unless your customer support deals with very well-defined questions and in that case, you're going to have to build your own ontology from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your situation, I would be curious if I can get information about the inner workings of Watson (obviously, the purpose is only to have a general idea of some steps that are sensible enough to try.) Here is a descriptive image that could be helpful from &lt;a href=&quot;http://en.wikipedia.org/wiki/Watson_%28computer%29&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Anu0t.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main idea is to segment questions into its components (using a regular chunking), then come up with several possibilities (hypothesis generation) of what the question might refer to (maybe this step can be implemented as a clustering). The third stage is rating the previous hypothesis based on available information. Finally, you choose the more likely hypothesis. There is a feedback involved in this process using sample questions and their answers. It would be interesting to see if the rating stage can be modeled in terms of a Bayesian or an incremental approach to allow for a feedback.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="4621" LastEditDate="2015-02-09T07:42:44.700" LastActivityDate="2015-02-09T07:42:44.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5088" PostTypeId="1" CreationDate="2015-02-09T00:18:46.427" Score="2" ViewCount="1524" Body="&lt;p&gt;I am trying to calculate maximum values for different groups in a relation in Pig. The relation has three columns patientid, featureid and featurevalue (all int). &#xA;I group the relation based on featureid and want to calculate the max feature value of each group, heres the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grpd = GROUP features BY featureid;&#xA;DUMP grpd;&#xA;temp = FOREACH grpd GENERATE $0 as featureid, MAX($1.featurevalue) as val;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Its  giving me  &lt;strong&gt;Invalid scalar projection: grpd&lt;/strong&gt; Exception. I read on different forums that MAX takes in a &quot;bag&quot; format for such functions, but when I take the dump of grpd, it shows me a bag format. Here's a small part of the output from the dump:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(5662,{(22579,5662,1)})&#xA;(5663,{(28331,5663,1),(2624,5663,1)})&#xA;(5664,{(27591,5664,1)})&#xA;(5665,{(30217,5665,1),(31526,5665,1)})&#xA;(5666,{(27783,5666,1),(30983,5666,1),(32424,5666,1),(28064,5666,1),(28932,5666,1)})&#xA;(5667,{(31257,5667,1),(27281,5667,1)})&#xA;(5669,{(31041,5669,1)})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Whats the issue ?&lt;/p&gt;&#xA;" OwnerUserId="8174" LastActivityDate="2018-04-27T08:42:56.647" Title="Error when using MAX in Apache Pig (Hadoop)" Tags="|apache-hadoop|apache-pig|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5089" PostTypeId="2" ParentId="5088" CreationDate="2015-02-09T00:34:26.520" Score="1" Body="&lt;p&gt;The issue was with column addressing, heres the correct working code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grpd = GROUP features BY featureid;&#xA;temp = FOREACH grpd GENERATE group as featureid, MAX(features.featurevalue) as val;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8174" LastActivityDate="2015-02-09T00:34:26.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5091" PostTypeId="1" CreationDate="2015-02-09T17:28:54.390" Score="0" ViewCount="5068" Body="&lt;p&gt;I have a complete Hadoop platform with &lt;em&gt;HDFS&lt;/em&gt;, &lt;em&gt;MR&lt;/em&gt;, &lt;em&gt;Hive&lt;/em&gt;, &lt;em&gt;PIG&lt;/em&gt;, &lt;em&gt;Hbase&lt;/em&gt;, etc., &lt;em&gt;Python&lt;/em&gt;, &lt;em&gt;R&lt;/em&gt;, &lt;em&gt;Java&lt;/em&gt;. All data sets have a large size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data set A, describing the jobs of people working in a company, is composed of the following fields:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Id Person: a unique alphanumeric identifier per person.&lt;/li&gt;&#xA;&lt;li&gt;Start Date: a date format iso entry in the post  &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;End Date: iso size release date of the position. If the date is not given, it is the current position&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Job Title: a text field containing the title and the name of the company. The text is free, non-standardized, French and / or English and can contain typos. Ex: Director Big Data Analytics with &lt;strong&gt;Google&lt;/strong&gt;, Commercial Manager at [missing text] , Manager at &lt;strong&gt;googole&lt;/strong&gt; ...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is; how can I create a feature to easily process the name of company of the job (jobtitle)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="8088" LastEditorUserId="471" LastEditDate="2015-02-11T02:06:56.973" LastActivityDate="2015-02-11T02:06:56.973" Title="Extract company names/job titles from free text" Tags="|apache-hadoop|apache-pig|hbase|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5092" PostTypeId="1" CreationDate="2015-02-09T23:22:05.547" Score="1" ViewCount="166" Body="&lt;p&gt;I am looking for a design pattern that is relevant to a module that extracts features.&#xA;I want to define a certain number of features over my data points, and then according to the performance and the feature selection, I may want to remove some of them and add others, and also I may want to consider any subsets of them to test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a good design pattern to do that? Did I miss something obvious? I am neither an engineer nor a developer, so I never study such things but I understand that it could help me a lot!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help,&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2015-02-11T08:06:24.537" Title="design pattern for extracting features" Tags="|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5093" PostTypeId="1" AcceptedAnswerId="5765" CreationDate="2015-02-10T01:04:15.963" Score="2" ViewCount="1338" Body="&lt;p&gt;I'm very new to this community, so please overlook my noobness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the number of 'YES' instances in ExtremeValues and Outliers takes up to 2947 and 2946 respectively. In other words, all my data are considered outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does this say about my data set? Or am I not meant to perform IQR on this data, if so, is there other algorithms to identify outliers other than IQR?  And how would one perform regression on such a data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="4803" LastActivityDate="2015-05-12T10:59:50.547" Title="InterquartileRange takes up most instances in data set" Tags="|dataset|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5094" PostTypeId="2" ParentId="5091" CreationDate="2015-02-10T02:06:19.957" Score="3" Body="&lt;p&gt;I think what you want is to extract company names from &quot;Job Title&quot;. In natural language process, we call this kind of research as &quot;Name Entity Recognition(NER)&quot;. You can try to use  Stanford Named Entity Recognizer (NER)[&lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml]&quot; rel=&quot;nofollow&quot;&gt;http://nlp.stanford.edu/software/CRF-NER.shtml]&lt;/a&gt;. Stanford NER performs very well on English contents and there are lots packages for many programming language:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;UIMA: Florian Laws made a Stanford NER UIMA annotator using a modified version of Stanford NER, which is available on his homepage. [Old version.]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Perl: Kieren Diment has written Text-NLP-Stanford-EntityExtract, a Perl module that provides an interface to Stanford NER running as a server.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Ruby: tiendung has written a Ruby Binding for the Stanford POS tagger and Named Entity Recognizer.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Python: Dat Hoang wrote pyner, a Python interface to Stanford NER. [Old version.] NLTK (2.0+) contains an interface to Stanford NER written by Nitin Madnani: documentation (note: set the character encoding or you get ASCII by default!), code, on Github.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;F#/C#/.NET: Sergey Tihon has ported Stanford NER to F# (and other .NET languages, such as C#), using IKVM. See also pages on: GitHub and NuGet.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;PHP: PHP-Stanford-NLP. Supports POS Tagger, NER, Parser. By Anthony Gentile (agentile).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are not satisfied with the performance of Stanford NER, you can also train you own models to extract company names by crawl company names from several popular sites with company names, such as Linkedin/Facebook/Glassdoor...etc&lt;/p&gt;&#xA;" OwnerUserId="1003" LastEditorUserId="1003" LastEditDate="2015-02-10T09:44:27.880" LastActivityDate="2015-02-10T09:44:27.880" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5095" PostTypeId="2" ParentId="540" CreationDate="2015-02-10T05:36:05.493" Score="1" Body="&lt;p&gt;While @Ben's answer is nice and partially introduces what should be done first with a newly cleaned data set, I feel that the approach is important enough to have its name presented loud and clear: &lt;em&gt;exploratory data analysis (EDA)&lt;/em&gt;. Therefore, the short answer to your question is that the &lt;strong&gt;first step should be EDA&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This suggestion is supported by most researchers, regardless of their knowledge domain or type of study. Here is how the father of EDA presents his thoughts on the subject (Tukey, 1977, p. 1-3; emphasis mine):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Exploratory Data Analysis (EDA) is detective work – numerical&#xA;  detective work – or counting detective work – or graphical detective&#xA;  work ... unless exploratory data analysis uncovers indications,&#xA;  usually quantitative ones, there is likely to be nothing for&#xA;  confirmatory data analysis to consider ... [it] can never be the whole&#xA;  story, but &lt;strong&gt;nothing else can serve as the foundation stone - as the&#xA;  first step&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is an enormous amount of information on approaches, guidelines and procedures for performing EDA. Potential starting points might include &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/eda.htm&quot; rel=&quot;nofollow&quot;&gt;EDA page&lt;/a&gt; on the NIST's Engineering Statistics Handbook website, &lt;a href=&quot;http://www.epa.gov/caddis/da_exploratory_0.html&quot; rel=&quot;nofollow&quot;&gt;EDA pages&lt;/a&gt; on the EPA's website, &lt;a href=&quot;http://www.stat.cmu.edu/~hseltman/309/Book/chapter4.pdf&quot; rel=&quot;nofollow&quot;&gt;corresponding chapter&lt;/a&gt;  from the book &quot;Experimental Design for Behavioral and Social Sciences&quot; and a &lt;a href=&quot;http://cll.stanford.edu/~langley/cogsys/behrens97pm.pdf&quot; rel=&quot;nofollow&quot;&gt;survey research paper on EDA&lt;/a&gt; by Begrens (1997), among many others. It is interesting to note that some sources include less traditional methods into EDA toolset, such as dimensionality reduction and clustering (for example, see the description of &lt;a href=&quot;http://www.ischool.berkeley.edu/courses/i290-eda&quot; rel=&quot;nofollow&quot;&gt;this research seminar&lt;/a&gt;). While some of the EDA approaches and methods are relatively simple, overall EDA is both &lt;strong&gt;art and science&lt;/strong&gt;, as it &lt;a href=&quot;http://community.lithium.com/t5/Science-of-Social-blog/Exploratory-Data-Analysis-Playing-with-Big-Data/ba-p/71632&quot; rel=&quot;nofollow&quot;&gt;combines unstructured (creative) and structured approaches&lt;/a&gt;. This aspect is especially important to recognize, as &lt;em&gt;big data&lt;/em&gt; exponentially increases complexity of data analyses, including EDA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. &lt;em&gt;Psychological Methods, 2&lt;/em&gt;(2), 131-160.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tukey, J. W. (1977). &lt;em&gt;Exploratory Data Analysis.&lt;/em&gt; Addison-Wesley.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NOTES:&lt;/strong&gt; For those interested in the Tukey's classic, it is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201076160&quot; rel=&quot;nofollow&quot;&gt;available on Amazon&lt;/a&gt;. Various MOOCs on EDA are also available, for example &lt;a href=&quot;https://www.coursera.org/course/exdata&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt; and &lt;a href=&quot;https://www.udacity.com/course/ud651&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt; (both are R-focused).&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-02-10T05:36:05.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5096" PostTypeId="2" ParentId="5092" CreationDate="2015-02-10T07:09:28.237" Score="2" Body="&lt;p&gt;When I think about how to implement a certain ML or data mining process in an OOP (like Java), I usually go to see how smarter people than me designed their system.&#xA;In this case, I'd see how Weka, RapidMiner, JAVA ML or others decided to tackle this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case of feature/attribute selection, i'm adding a link to Weka's API.&#xA;If you want to better understand how it's done, you should download Weka and play with the source code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/AttributeSelectedClassifier.html&quot; rel=&quot;nofollow&quot;&gt;http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/AttributeSelectedClassifier.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without getting into specific implementations, I think that a linked list of any sort would do the trick, since you want to select a subset of all features.&lt;/p&gt;&#xA;" OwnerUserId="8113" LastEditorUserId="8113" LastEditDate="2015-02-11T08:06:24.537" LastActivityDate="2015-02-11T08:06:24.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5097" PostTypeId="1" AcceptedAnswerId="5098" CreationDate="2015-02-10T12:36:48.250" Score="0" ViewCount="255" Body="&lt;p&gt;What are some data analytic package &amp;amp; feature in python which helps do data analytic?&lt;/p&gt;&#xA;" OwnerUserId="8195" LastEditorUserId="8195" LastEditDate="2015-02-11T06:46:22.097" LastActivityDate="2015-02-11T06:46:22.097" Title="Python for data analytics" Tags="|data-mining|python|" AnswerCount="1" CommentCount="1" ClosedDate="2015-02-11T02:07:46.803" ContentLicense="CC BY-SA 3.0" />
  <row Id="5098" PostTypeId="2" ParentId="5097" CreationDate="2015-02-10T15:59:19.010" Score="1" Body="&lt;p&gt;You're looking for this answer: &lt;a href=&quot;https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists&quot; rel=&quot;nofollow&quot;&gt;https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="525" LastActivityDate="2015-02-10T15:59:19.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5099" PostTypeId="2" ParentId="5055" CreationDate="2015-02-10T16:26:19.457" Score="5" Body="&lt;p&gt;I think the key here is what you believe to be dependent variables. You mentioned, for instance, a three month rest. Encoding the rest between races is likely to be a better idea than simply encoding the date of the rest as a lot of the date is redundant to what is actually being said. As with any machine learning algorithm, representation of the data is key and in many ways, the actual algorithm applied is less important.  &lt;/p&gt;&#xA;" OwnerUserId="8200" LastActivityDate="2015-02-10T16:26:19.457" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5101" PostTypeId="2" ParentId="5080" CreationDate="2015-02-10T21:13:37.107" Score="1" Body="&lt;p&gt;It might be a good idea to start with voluntary work and see if that leads to payed position. &lt;a href=&quot;http://www.datakind.org/howitworks/datachapters/datakind-uk/&quot; rel=&quot;nofollow&quot;&gt;DataKind&lt;/a&gt;, that has been mentioned above, is where I would start, especially if you live in England, since you can register your interest online and even do work with them as a data scientist for a weekend in a meetup event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also various &lt;a href=&quot;https://www.google.com/search?client=safari&amp;amp;rls=en&amp;amp;q=data%20science%20site:meetup.com&amp;amp;ie=UTF-8&amp;amp;oe=UTF-8&amp;amp;gfe_rd=cr&amp;amp;oq=data%20science&amp;amp;gs_l=navquery.3..0l4j0i5.1459.4969.0.5279.12.12.0.0.0.0.82.718.12.12.0....0....1.61.navquery..0.12.717.-odm8NeyBww&amp;amp;ved=0CCkQ2wE&amp;amp;bav=on.2,or.r_cp.r_qf.&amp;amp;bvm=bv.85464276,d.d2s&amp;amp;biw=1240&amp;amp;bih=658&amp;amp;ech=1&amp;amp;psi=yHDaVNOsH9LnasblgZAI.1423601865135.3&amp;amp;ei=yHDaVNOsH9LnasblgZAI&amp;amp;emsg=NCSR&amp;amp;noj=1&quot; rel=&quot;nofollow&quot;&gt;meetups&lt;/a&gt; in many cities where it's likely to find people working on nonprofit companies and expand your network. &lt;/p&gt;&#xA;" OwnerUserId="7857" LastActivityDate="2015-02-10T21:13:37.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5103" PostTypeId="2" ParentId="4977" CreationDate="2015-02-10T23:18:18.657" Score="2" Body="&lt;p&gt;Both Storm and Spark are great tools. It depends on your use-case. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you want to quickly parse huge stream of data and store it into a database? Use Storm (e.g. counting tweets).&lt;/li&gt;&#xA;&lt;li&gt;Training a classifier on a stream of data would be a task suitable for Spark. There's a data window on which you're working and it will take a while.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I haven't tried Flink, but it looks more similar to Spark. Spark has general concept of RDD (Resilient Distributed Datasets) that can be used also for graphs, huge matrices etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to write a word count, you any of them. But who wants a word count (except from hello world tutorials)? &lt;/p&gt;&#xA;" OwnerUserId="8207" LastEditorUserId="21" LastEditDate="2015-02-11T02:08:02.163" LastActivityDate="2015-02-11T02:08:02.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5104" PostTypeId="2" ParentId="4950" CreationDate="2015-02-11T06:13:12.223" Score="5" Body="&lt;p&gt;Without a sample of your data, it's unclear what's the structure of your data and what tool is suitable to process it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some blind recommendations based on my experience:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you just need some flexibilty parsing the text record, such as variable repeat number of certain field, or conditional parsing of fields, then you should check out this python library: &lt;a href=&quot;http://construct.readthedocs.org/en/latest/&quot;&gt;http://construct.readthedocs.org/en/latest/&lt;/a&gt;&#xA;it allows you to first define a hirachcal structure of your data, and then apply this structure to parse information from a text file. It's especial useful when parsing binary files.&lt;/li&gt;&#xA;&lt;li&gt;If you're looking for an algorithm that can actually &quot;understand&quot; your text data and &quot;infer&quot; the structure in a smart way. Then you might want to try graph based approach: &#xA;&lt;a href=&quot;http://kavita-ganesan.com/opinosis&quot;&gt;http://kavita-ganesan.com/opinosis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="7950" LastActivityDate="2015-02-11T06:13:12.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5107" PostTypeId="2" ParentId="5085" CreationDate="2015-02-11T07:56:48.717" Score="0" Body="&lt;p&gt;No, no purpose other than saving data. A fresh sample is always better than a used one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only situation I can think of in which having epochs would make sense would be if the synthesis process would be really time consuming per example.&lt;/p&gt;&#xA;" OwnerUserId="8210" LastActivityDate="2015-02-11T07:56:48.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5108" PostTypeId="1" CreationDate="2015-02-11T08:00:52.377" Score="2" ViewCount="238" Body="&lt;p&gt;I was not sure about posting this question with mentioning the name of the company, which I quite respect and admire. However, I've figured that a wider exposure might help the team to fix this and similar problems faster as well as increase the quality of the &lt;em&gt;machine learning (ML)&lt;/em&gt; engine of their website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem exposes itself by too many occurrences of a quite trivial &lt;em&gt;misclassification error&lt;/em&gt; on Amazon's book categories &lt;strong&gt;classification&lt;/strong&gt; (which I'm a frequent visitor of). In the following example, the underlying reason of such behavior is quite clear, but in other cases the reasons might be different. I am curious about what could be other potential &lt;em&gt;reasons&lt;/em&gt; for misclassification and what are the &lt;em&gt;strategies/approaches&lt;/em&gt; to avoiding such problems. Without much further ado, here's how the problem appears in real life.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was reviewing some books, related to transitioning from graduate programs (Ph.D., in particular) to work environment in academia. Among several other books, I ran across the following one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4LSzV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, so good. However, let's scroll down a bit further to see the the books ratings in relevant categories. We should expect Amazon to figure out &lt;em&gt;categories&lt;/em&gt;, &lt;strong&gt;relevant&lt;/strong&gt; to the book's &lt;em&gt;discipline, topic and contents&lt;/em&gt;. How surprised was I (and that's an understatement!) to see the following result of Amazon.com's sophisticated ML engine and algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/KXrFh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, the only fuzzy fact that connects this book with the subject &quot;Audiology and Speech Pathology&quot; (!) is IMHO the author's last name (Boice), which, is &lt;strong&gt;close to&lt;/strong&gt; the word &quot;voice&quot;. If my guess is correct, Amazon's ML engine, for some reason, decided to take into account the book's lexicographical attribute instead of the book's most important and &lt;strong&gt;most relevant attributes&lt;/strong&gt;, such as title, topic and contents. I've seen multiple occurrences of similar absolutely incorrect ML-based decision making on Amazon.com and some other websites. So, hopefully my question makes sense as well as interesting and important enough to spark a discussion: &lt;strong&gt;What could be other potential reasons for misclassification and what are the strategies/approaches to avoiding such problems?&lt;/strong&gt; (Any related thoughts will also be appreciated.)&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-02-11T13:25:28.473" Title="Reasons and prevention of trivial (and less trivial) misclassification errors?" Tags="|machine-learning|classification|topic-model|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5109" PostTypeId="1" AcceptedAnswerId="5111" CreationDate="2015-02-11T10:48:51.903" Score="4" ViewCount="3452" Body="&lt;p&gt;&lt;em&gt;Disclaimer:&lt;/em&gt; although I know some things about big data and am currently learning some other things about machine learning, the specific area that I wish to study is vague, or at least appears vague to me now. I'll do my best to describe it, but this question could still be categorised as too vague or not really a question. Hopefully, I'll be able to reword it more precisely once I get a reaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some experience with Hadoop and the Hadoop stack (gained via using CDH), and I'm reading a book about Mahout, which is a collection of machine learning libraries. I also think I know enough statistics to be able to comprehend the math behind the machine learning algorithms, and I have some experience with R.&#xA;My ultimate goal is making a setup that would make trading predictions and deal with financial data in real time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if there're any materials that I can further read to help me understand ways of managing that problem; books, video tutorials and exercises with example datasets are all welcome.&lt;/p&gt;&#xA;" OwnerUserId="8214" LastEditorUserId="97" LastEditDate="2015-02-11T16:12:47.457" LastActivityDate="2015-02-16T03:03:43.883" Title="Machine Learning on financial big data" Tags="|machine-learning|bigdata|finance|" AnswerCount="2" CommentCount="2" ClosedDate="2015-02-19T10:06:20.863" ContentLicense="CC BY-SA 3.0" />
  <row Id="5110" PostTypeId="1" CreationDate="2015-02-11T11:53:30.890" Score="1" ViewCount="163" Body="&lt;p&gt;I am looking for a suitable graph representation where the nodes/vertices are molecules with 2 main variables: structural characteristic and a real number property (for example biological potency). The network is build up by molecules by matching structural properties and it can be rather complex with many clusters of high degree vertices (ie molecules that are structurally similar to each other). However the representation should give a clear picture of potencies, although high detail accuracy is not required: some sort of classification is adequate. I played with JUNG with a graph having edge weights as the change in potency and the ISOM layout thinking that it reserves (as much as possible) the edge weights but with not good results. Something like radial layout could be ideal, as it gives a clear picture of potency distribution, but this obviously would suffer from crowding of vertices that are closer to the center.  As, obviously, my knowledge is restricted, is there some layout that you would suggest and is practical for a non expert to implement? &lt;/p&gt;&#xA;" OwnerUserId="8218" LastActivityDate="2015-05-12T12:01:48.797" Title="Graph layout for a network of molecules" Tags="|visualization|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5111" PostTypeId="2" ParentId="5109" CreationDate="2015-02-11T12:25:31.077" Score="11" Body="&lt;p&gt;There are tons of materials on financial (big) data analysis that you can read and peruse. I'm not an expert in finance, but am curious about the field, especially in the context of data science and R. Therefore, the following are selected relevant resource suggestions that I have for you. I hope that they will be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Books: Financial analysis (general / non-R)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/statistics/business,+economics+%26+finance/book/978-0-387-20270-9&quot; rel=&quot;noreferrer&quot;&gt;Statistics and Finance: An Introduction&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/economics/public+finance/book/978-0-387-77826-6&quot; rel=&quot;noreferrer&quot;&gt;Statistical Models and Methods for Financial Markets&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Books: Machine Learning in Finance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.worldscientific.com/worldscibooks/10.1142/p818&quot; rel=&quot;noreferrer&quot;&gt;Machine Learning for Financial Engineering&lt;/a&gt; (!) - seems to be an edited collection of papers;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.elsevier.com/books/neural-networks-in-finance/mcnelis/978-0-12-485967-8#description&quot; rel=&quot;noreferrer&quot;&gt;Neural Networks in Finance: Gaining Predictive Edge in the Market&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Books: Financial analysis with R&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/statistics/business,+economics+%26+finance/book/978-1-4614-8787-6&quot; rel=&quot;noreferrer&quot;&gt;Statistical Analysis of Financial Data in R&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/statistics/business,+economics+%26+finance/book/978-1-4419-7786-1&quot; rel=&quot;noreferrer&quot;&gt;Statistics and Data Analysis for Financial Engineering&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0470978708&quot; rel=&quot;noreferrer&quot;&gt;Financial Risk Modelling and Portfolio Optimization with R&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/statistics/business%2C+economics+%26+finance/book/978-3-642-16520-7&quot; rel=&quot;noreferrer&quot;&gt;Statistics of Financial Markets: An Introduction&lt;/a&gt; (code in R and MATLAB).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Academic Journals&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://algorithmicfinance.org&quot; rel=&quot;noreferrer&quot;&gt;Algorithmic Finance&lt;/a&gt; (open access)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Web sites&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.rmetrics.org&quot; rel=&quot;noreferrer&quot;&gt;RMetrics&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://quant.stackexchange.com&quot;&gt;Quantitative Finance on StackExchange&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R Packages&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;the above-mentioned &lt;em&gt;RMetrics&lt;/em&gt; site (see &lt;a href=&quot;https://www.rmetrics.org/about&quot; rel=&quot;noreferrer&quot;&gt;this page&lt;/a&gt; for general description);&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;CRAN Task Views&lt;/em&gt;, including &lt;a href=&quot;http://cran.r-project.org/web/views/Finance.html&quot; rel=&quot;noreferrer&quot;&gt;Finance&lt;/a&gt;, &lt;a href=&quot;http://cran.r-project.org/web/views/Econometrics.html&quot; rel=&quot;noreferrer&quot;&gt;Econometrics&lt;/a&gt; and several other Task Views.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Competitions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.modeloff.com&quot; rel=&quot;noreferrer&quot;&gt;MODELOFF (The Financial Modeling World Championships)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Educational Programs&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ieor.columbia.edu/ms-financial-engineering&quot; rel=&quot;noreferrer&quot;&gt;MS in Financial Engineering - Columbia University&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www3.cs.stonybrook.edu/~skiena/691&quot; rel=&quot;noreferrer&quot;&gt;Computational Finance - Hong Kong University&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Blogs (Finance/R)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://timelyportfolio.blogspot.com&quot; rel=&quot;noreferrer&quot;&gt;Timely Portfolio&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://systematicinvestor.wordpress.com&quot; rel=&quot;noreferrer&quot;&gt;Systematic Investor&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://moneymakingmankind.wordpress.com&quot; rel=&quot;noreferrer&quot;&gt;Money-making Mankind&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-02-12T03:33:47.147" LastActivityDate="2015-02-12T03:33:47.147" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5112" PostTypeId="2" ParentId="5108" CreationDate="2015-02-11T13:25:28.473" Score="2" Body="&lt;p&gt;By looking at the ratings of the book, it looks like they are doing some sort of hierarchical clustering.&#xA;First of all it's a book, then a text book, then in medicine &amp;amp; health science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a pretty difficult problem because:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You don't know how many and which topics you have, and the topics are constantly changing.&lt;/li&gt;&#xA;&lt;li&gt;Since this is a clustering problem (or a semi-supervised problem in the better case), a closed feedback loop for these errors cannot be applied. Clustering is always more difficult than a supervised learning problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Apparently they're doing a pretty good job in general. This book in particular is a tough one...&lt;/p&gt;&#xA;" OwnerUserId="8113" LastActivityDate="2015-02-11T13:25:28.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5113" PostTypeId="1" AcceptedAnswerId="5114" CreationDate="2015-02-11T13:26:27.753" Score="1" ViewCount="2575" Body="&lt;p&gt;A book I'm now reading, &quot;Apache Mahout Cookbook&quot; by Pierro Giacomelli, states that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To avoid [this], you need to divide the vector files into two sets called the 80-20 split &amp;lt;...&gt;&#xA;  A good dividing percentage is shown to be 80% and 20%.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is there a strict statistical proof of this being the best percentage, or is it a euristic result?&lt;/p&gt;&#xA;" OwnerUserId="8214" LastActivityDate="2015-02-11T13:56:30.847" Title="Dividing percentage" Tags="|machine-learning|statistics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5114" PostTypeId="2" ParentId="5113" CreationDate="2015-02-11T13:56:30.847" Score="5" Body="&lt;p&gt;If this is about splitting your data into training and testing data, then 80/20 is a common rule of thumb. An &quot;optimal&quot; split (which would need to be operationalized) would likely depend on your sample size, distributions and relationships between your variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also common to split your data &lt;em&gt;three&lt;/em&gt; ways (e.g., 60/20/20 - again rules of thumb), into a training set that you train your models on and a test set which you test your model on. You will iterate training and testing until you like the result. Then, &lt;em&gt;and only then&lt;/em&gt; you apply the final model (trained on both the training and test set) on the third validation set. This avoids &quot;overfitting on the test set&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29&quot; rel=&quot;noreferrer&quot;&gt;cross-validation&lt;/a&gt; is much better than a simple data split. Your textbook should also cover cross-validation. If it doesn't, get a better textbook.&lt;/p&gt;&#xA;" OwnerUserId="2853" LastActivityDate="2015-02-11T13:56:30.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5115" PostTypeId="1" CreationDate="2015-02-11T16:46:38.800" Score="4" ViewCount="9593" Body="&lt;p&gt;Can you please show the step by step calculation of Entropy(Ssun)?  I do not understand how 0.918 is arrived at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried but I get the values as 0.521089678, 0.528771238, 0.521089678 for Sunny, Windy, Rainy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to calculate the target entropy (Decision) correctly as = -(6/10)*log(6/10) + -(2/10)log(2/10) + -(1/10)log(1/10) + -(1/10)log(1/10) = 1.570950594&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am totally stuck at the next step.  Request your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://www.doc.ic.ac.uk/~sgc/teaching/pre2012/v231/lecture11.html&quot; rel=&quot;nofollow&quot;&gt;http://www.doc.ic.ac.uk/~sgc/teaching/pre2012/v231/lecture11.html&lt;/a&gt;&#xA;Please search for &quot;The first thing we need to do is work out which attribute will be put into the node at the top of our tree:&quot; to reach the line I am referring to.&lt;/p&gt;&#xA;" OwnerUserId="8221" LastActivityDate="2015-02-12T14:58:17.253" Title="Calculating entropies of attributes" Tags="|data-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5117" PostTypeId="2" ParentId="5115" CreationDate="2015-02-11T18:32:14.230" Score="6" Body="&lt;p&gt;Consider the formula for Entropy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(S) = \sum_\limits{i=1}^{n}-p_{i}\log_2p_{i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expanding that summation for the four concept decision attributes for your problem gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E(S) = -p_{cinema}\log_2\left(p_{cinema}\right) - p_{tennis}\log_2\left(p_{tennis}\right) - p_{stayin}\log_2\left(p_{stayin}\right) - p_{shopping}\log_2\left(p_{shopping}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three observations where &lt;code&gt;Weather=Sunny&lt;/code&gt;. Of those three, one has &lt;code&gt;Decision=Cinema&lt;/code&gt; and two have &lt;code&gt;Decision=Tennis&lt;/code&gt;. So for &lt;code&gt;Weather=Sunny&lt;/code&gt; you have $p_{cinema}=\frac{1}{3}$, $p_{tennis}=\frac{2}{3}$, and other decision probabilities are zero. Plugging those values into the equation above gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;\begin{align}&#xA;E(S_{Sun}) &amp;amp;= -\frac{1}{3}\log_2\left(\frac{1}{3}\right)-\frac{2}{3}\log_2\left(\frac{2}{3}\right) - \left(0\right)\cdot\log_2\left(0\right) - \left(0\right)\cdot\log_2\left(0\right) \\&#xA;           &amp;amp;= 0.38998 + 0.52832 + 0 + 0 \\&#xA;           &amp;amp;= 0.918&#xA;\end{align}$&lt;/p&gt;&#xA;" OwnerUserId="964" LastEditorUserId="964" LastEditDate="2015-02-12T14:58:17.253" LastActivityDate="2015-02-12T14:58:17.253" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5118" PostTypeId="2" ParentId="5115" CreationDate="2015-02-11T23:31:20.127" Score="4" Body="&lt;p&gt;&lt;strong&gt;Let me first explain the concept of entropy for decision trees:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Entropy is a so called impurity measure for a set of elements. Impurity - being the opposite of purity - is referring to the distribution of the decision categories (or class labels) within the set. Initially, each row in the table is one element and your set of elements are all rows of your table. This set is called pure if it just consists of the same class label and it is called impure if it contains all different class labels in the same proportion. Translating into the values of entropy (remember being an impurity measure, the set has in the latter case the highest possible value and in the former case (just the same class label) the lowest value. The lowest possible value for entropy is 0.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let's choose a visual respresentation to illustrate what really counts and what does not matter for the entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) You start off with the initial table which contains the following class labels:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CbjbT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that for computing the entropy none of the remaining attributes, attribute values matter, also the order of the rows don't matter (entropy is a measure for a set of elements)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) You convert the different class labels to different colors&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/A4mux.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) So instead of this table with the colored class labeled, you just draw a circle (&quot;ball&quot;) for each class label (=element)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ufvDS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you throw all the balls in a bag. There you go! Your elements of a set are represented as balls in a bag.  If you open the bag and look into it, you might see &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;just balls of the same color --&gt; one pure color. The entropy says, it's not impure at all, I give it a zero! entropy = 0&lt;/li&gt;&#xA;&lt;li&gt;the same amount of balls in different colors --&gt; that doesn't look pure at all, it's impure! The entropy assigns the highest possible value to this set (bag). The highest possible entropy value depends on the number of class labels. If you have just two, the maximum entropy is 1. In your case, you have 4 different class labels, so the maximum entropy would be e.g. if you had 12 balls and 3 balls of each color. The maximum entropy of a 4 class set is 2.&lt;/li&gt;&#xA;&lt;li&gt;none of the above. Then your entropy is between the two values. If one color is dominant then the entropy will be close to 0, if the colors are very mixed up, then it is close to the maximum (2 in your case).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How does a decision tree use the entropy?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, first you calculate the entropy of the whole set. That impurity is your reference. What your decision tree tries to achieve is to reduce the impurity of the whole set. So the information Gain for a given attribute is computed by taking the entropy of the whole set and subtracting it with the entropies of sets that are obtained by breaking the whole set into one piece per attribute category. To make sure a set with just a few elements don't get the same weight as a set with many elements the entropies of sets are multiplied by their relative frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming back to the example, for computing the Gain (S, parents) you need to compute the weighted entropies of the two sets: set 1 = all rows with category(parent) = yes and set 2 = all rows with category(parent) = no).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set 1 refers to five rows, all having the decision category = cinema(just 5 orange balls!)&#xA;So the weight is 5 rows out of 10 rows = 5/10 = 0.5. And the entropy is 0 (totally pure) or by applying the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/dDMBf.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a sum over all class labels so n = 4. So you assign a class label to each of the i's. E.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 1 being cinema: - (5/5 * log(3/3) = - (1 * log(1)) = - (1*0) = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 2 being tennis: - (0/5 * log(0/3) = - (0 * log(0)) = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 3 being stay in: - (0/5 * log(0/3) = - (0 * log(0)) = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 4 being shopping: - (0/5 * log(0/3) = - (0 * log(0)) = 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the sum of all four is 0. 0 times the weight 0.5 equals 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set 2 also refers to five rows, having the decision categories = 1*cinema, 2*tennis, 1*stay in, 1*shopping&#xA;So the weight is also 5 rows out of 10 rows = 5/10 = 0.5. Applying the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 1 being cinema: - (1/5 * log(1/5) = - (0.2 * -2,32) = 0.46&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 2 being tennis: - (2/5 * log(2/5) = - (0.4 * -1,32) = 0.53&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 3 being stay in: - (1/5 * log(1/5) = - (0.2 * -2,32) = 0.46&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For i = 4 being shopping: - (1/5 * log(1/5) = - (0.2 * -2,32) = 0.46&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the sum of all four is 1.92. 1.92 times the weight 0.5 = 0.96.&#xA;Thus, the Gain for this attribute is: &#xA;Initial entropy - weighted entropies for all attribute categories:&#xA;1.57 - 0 - 0.96 = 0.61.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the other attributes you do the same calculation.&lt;/p&gt;&#xA;" OwnerUserId="8191" LastEditorUserId="8191" LastEditDate="2015-02-11T23:38:53.957" LastActivityDate="2015-02-11T23:38:53.957" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5119" PostTypeId="1" AcceptedAnswerId="5135" CreationDate="2015-02-12T07:00:38.160" Score="1" ViewCount="1743" Body="&lt;p&gt;is the survival table classification method on the Kaggle Titanic dataset an example of an implementation of Naive Bayes ? I am asking because I am reading up on Naive Bayes and the basic idea is as follows:&#xA;&quot;Find out the probability of the previously unseen instance&#xA;belonging to each class, then simply pick the most probable class&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The survival table &#xA;(&lt;a href=&quot;http://www.markhneedham.com/blog/tag/kaggle/&quot; rel=&quot;nofollow&quot;&gt;http://www.markhneedham.com/blog/tag/kaggle/&lt;/a&gt;)&#xA;seems like an evaluation of the possibilities of survival given possible combinations of values of the chosen features and I'm wondering if it could be an example of Naive Bayes in another name. Can someone shed light on this ?&lt;/p&gt;&#xA;" OwnerUserId="8234" LastActivityDate="2015-02-13T09:06:33.070" Title="Kaggle Titanic Survival Table an example of Naive Bayes?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5120" PostTypeId="1" CreationDate="2015-02-12T07:07:56.257" Score="1" ViewCount="832" Body="&lt;p&gt;I am working on a problem(non competition) from hacker rank &lt;a href=&quot;https://www.hackerrank.com/challenges/predict-missing-grade&quot; rel=&quot;nofollow&quot;&gt;https://www.hackerrank.com/challenges/predict-missing-grade&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically you're given test data of a bunch of students of their scores in other subjects but math and you are to predict their score in math based off all their other test scores. Say you were passed data of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{&quot;SerialNumber&quot;:1,&quot;English&quot;:1,&quot;Physics&quot;:2,&quot;Chemistry&quot;:3,&quot;ComputerScience&quot;:2}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you go about generating that student's score in mathematics or coming up with a prediction engine to generate the math score? I know that's the whole point of this question but can someone give me a hint or a resource to go to so I can have a chance of figuring this out and actually get started? I really want to learn.&lt;/p&gt;&#xA;" OwnerUserId="8235" LastActivityDate="2015-03-30T18:18:06.550" Title="How to start the process of coming up with the predicted math score?" Tags="|machine-learning|data-mining|statistics|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5121" PostTypeId="1" AcceptedAnswerId="5128" CreationDate="2015-02-12T07:08:16.537" Score="39" ViewCount="56723" Body="&lt;p&gt;&lt;strong&gt;Jaccard similarity&lt;/strong&gt; and &lt;strong&gt;cosine similarity&lt;/strong&gt; are two very common measurements while comparing item similarities. However, I am not very clear in what situation which one should be preferable than another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody help clarify the differences of these two measurements (the difference in concept or principle, not the definition or computation) and their preferable applications?&lt;/p&gt;&#xA;" OwnerUserId="5184" LastActivityDate="2021-08-10T16:45:47.390" Title="Applications and differences for Jaccard similarity and Cosine Similarity" Tags="|similarity|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5122" PostTypeId="1" AcceptedAnswerId="5125" CreationDate="2015-02-12T10:46:41.280" Score="5" ViewCount="2776" Body="&lt;p&gt;I'm curious if anyone has Python library suggestions for inferential statistics. I'm currently reading &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;noreferrer&quot;&gt;An Introduction to Statistical Learning&lt;/a&gt;, which uses R for the example code, but ideally I'd like to use Python as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of my data experience is with Pandas, Matplotlib, and Sklearn doing predictive modeling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've found &lt;a href=&quot;https://pypi.python.org/pypi/statsmodels&quot; rel=&quot;noreferrer&quot;&gt;statsmodels&lt;/a&gt;. Is this what is recommended or is there something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="8236" LastActivityDate="2015-02-12T11:23:23.487" Title="Best Python library for statistical inference" Tags="|python|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5124" PostTypeId="2" ParentId="5120" CreationDate="2015-02-12T11:10:53.967" Score="1" Body="&lt;p&gt;What you are looking for is a machine learning algorithm. Although the easiest way would be to take the average scores and use that, there are much more accurate ways to make predictive models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was the first data science tutorial I did. It's perfect for getting started. Here is it in &lt;a href=&quot;https://www.kaggle.com/c/titanic-gettingStarted/details/new-getting-started-with-r&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; and in &lt;a href=&quot;https://www.kaggle.com/c/titanic-gettingStarted/details/getting-started-with-python-ii&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for a short answer, something you can just look up how to implement, I'd check out Random Forests.&lt;/p&gt;&#xA;" OwnerUserId="8236" LastActivityDate="2015-02-12T11:10:53.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5125" PostTypeId="2" ParentId="5122" CreationDate="2015-02-12T11:23:23.487" Score="8" Body="&lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/devel/&quot; rel=&quot;noreferrer&quot;&gt;statsmodels&lt;/a&gt; is a good, and fairly standard, package to statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Bayesian interference you can go with &lt;a href=&quot;http://pymc-devs.github.io/pymc/&quot; rel=&quot;noreferrer&quot;&gt;PyMC&lt;/a&gt; - see as in &lt;a href=&quot;http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/&quot; rel=&quot;noreferrer&quot;&gt;Cam Davidson-Pilon, Probabilistic Programming &amp;amp; Bayesian Methods for Hackers&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="289" LastActivityDate="2015-02-12T11:23:23.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5128" PostTypeId="2" ParentId="5121" CreationDate="2015-02-12T15:47:15.277" Score="20" Body="&lt;p&gt;Jaccard Similarity is given by&#xA;&lt;span class=&quot;math-container&quot;&gt;$s_{ij} = \frac{p}{p+q+r}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&lt;p&gt;p = # of attributes positive for both objects &lt;br&gt;&#xA;q = # of attributes 1 for i and 0 for j &lt;br&gt;&#xA;r = # of attributes 0 for i and 1 for j &lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;Whereas, cosine similarity = &lt;span class=&quot;math-container&quot;&gt;$\frac{A \cdot B}{\|A\|\|B\|}$&lt;/span&gt; where A and B are object vectors.&lt;/p&gt;&#xA;&lt;p&gt;Simply put, in cases where the vectors A and B are comprised 0s and 1s only, cosine similarity divides the number of common attributes by the product of A and B's distance from zero. Whereas in Jaccard Similarity, the number of common attributes is divided by the number of attributes that exists in at least one of the two objects.&lt;/p&gt;&#xA;&lt;p&gt;And there are many other measures of similarity, each with its own eccentricities. When deciding which one to use, try to think of a few representative cases and work out which index would give the most usable results to achieve your objective.&lt;/p&gt;&#xA;&lt;p&gt;The Cosine index could be used to identify plagiarism, but will not be a good index to identify mirror sites on the internet. Whereas the Jaccard index, will be a good index to identify mirror sites, but not so great at catching copy pasta plagiarism (within a larger document).&lt;/p&gt;&#xA;&lt;p&gt;When applying these indices, you must think about your problem thoroughly and figure out how to define similarity. Once you have a definition in mind, you can go about shopping for an index.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;Earlier, I had an example included in this answer, which was ultimately incorrect. Thanks to the several users who have pointed that out, I have removed the erroneous example.&lt;/p&gt;&#xA;" OwnerUserId="90" LastEditorUserId="90" LastEditDate="2021-08-10T16:45:47.390" LastActivityDate="2021-08-10T16:45:47.390" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="5129" PostTypeId="1" AcceptedAnswerId="5136" CreationDate="2015-02-12T16:11:30.087" Score="0" ViewCount="188" Body="&lt;p&gt;Hi I am new to Data analytics.I am planning to learn R by doing some real time projects. How should I stream line (set goals) my time in learning R and also I have not learnt statistics till data. I am planning to learn both side by side. I am mid level data warehouse engineer who has experience in DBMS Data-Integration. I am planning to learn R so that I can bring out useful analysis from the Integrated data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I be specific, I am beginning in R, so what are the basic statistical concepts I should know and implement it in R. If I want to be an expert or above average person in R how should I plan strategically to become one. Say if I can spend 2 hrs a day for 1 year what level I should reach. FYI am working for a SaaS company. What are the way s in which I can utilize R knowledge in a SaaS environment &lt;/p&gt;&#xA;" OwnerUserId="8244" LastActivityDate="2015-02-13T15:04:24.963" Title="R Programing beginner" Tags="|r|statistics|" AnswerCount="3" CommentCount="1" ClosedDate="2015-02-14T22:02:46.520" ContentLicense="CC BY-SA 3.0" />
  <row Id="5130" PostTypeId="2" ParentId="5129" CreationDate="2015-02-12T16:46:29.453" Score="1" Body="&lt;p&gt;I would start off with &lt;a href=&quot;http://www.amazon.ca/All-Statistics-Concise-Statistical-Inference/dp/0387402721&quot; rel=&quot;nofollow&quot;&gt;All Of Statistics by Larry Wasserman&lt;/a&gt;. This quickly gets you upto speed with statistics assuming you have some mathematical background. I think all it needs is introductory calculus and linear algebra. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is pretty straightforward to pick up and there are a number of resources you can use. The &lt;a href=&quot;https://www.coursera.org/course/rprog&quot; rel=&quot;nofollow&quot;&gt;R Programming&lt;/a&gt; course on coursera is an excellent short course to get you familiarized with R. Besides that there are a number of books and tutorials on the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you start working through All of Statistics and start playing around in R along with that book.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are at a SaaS company, there are a number of data sciency roles and responsibilities that should be available. Most SaaS companies will have analytical tools to provide basic insight. Once you start learning statistics and data science, you will be able to identify the gaps in the pre baked tools and ways to improve them. I would also read &lt;a href=&quot;http://www.boozallen.com/insights/2013/11/data-science-field-guide&quot; rel=&quot;nofollow&quot;&gt;The Field Guide to Data Science&lt;/a&gt;, it's a short book that will give you a high level idea of data science and its utility. &lt;/p&gt;&#xA;" OwnerUserId="90" LastEditorUserId="90" LastEditDate="2015-02-12T16:51:47.660" LastActivityDate="2015-02-12T16:51:47.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5131" PostTypeId="2" ParentId="61" CreationDate="2015-02-12T19:09:05.900" Score="9" Body="&lt;p&gt;No one seems to have posted the XKCD overfitting comic yet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/wSN2q.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3347" LastActivityDate="2015-02-12T19:09:05.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5132" PostTypeId="2" ParentId="2652" CreationDate="2015-02-13T00:33:59.857" Score="1" Body="&lt;p&gt;This issue is caused because the following environment variables are not set correctly&#xA;PKG_CONFIG_PATH and LD_LIBRARY_PATH&#xA;for my configuration, i should execute this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;export PKG_CONFIG_PATH = /usr/local/lib &lt;br&gt;&#xA;export LD_LIBRARY_PATH = /usr/local/lib&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also set this commands on R console: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sys.setenv(HADOOP_HOME=&quot;/usr/local/hadoop/&quot;)&lt;br&gt;&#xA;Sys.setenv(HADOOP_BIN=&quot;/usr/local/hadoop/bin&quot;)&lt;br&gt;&#xA;Sys.setenv(HADOOP_CONF_DIR=&quot;/usr/local/hadoop/conf&quot;)&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="4705" LastEditorUserId="4705" LastEditDate="2015-02-13T21:58:41.240" LastActivityDate="2015-02-13T21:58:41.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5133" PostTypeId="2" ParentId="5047" CreationDate="2015-02-13T05:57:36.840" Score="3" Body="&lt;p&gt;The answer to both data sets is an OCR application with some post-processing, but a more specialized program than a generic low-quality or an open source OCR. Essentially the harder the problem, the more capable and advanced tools need to be used to solve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There will be two major stages in this task: digitizing the data (image to text, i.e. OCR) and processing the data (performing the actual count). Look at them separately in order to select the best method for each stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main challenges in these images and generic OCR are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) images have low resolution. For example the # 1 image has resolution of about 72 dpi. Suggested resolution for such text quality is to scan at 300 to 400 dpi, but it is clear that re-scanning or controlling scan resolution is not applicable now. That’s why one option is to clean and increase the size using image pre-processing tools. This is what the original #1 image snippet looks like after adaptive binarization and zoomed at 300%. It is clear that each character has too few pixels and characters can be easily misread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/oNqPE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) GIF format in #1 is not supported by many OCR applications. Images need to be batch-converted to a different format, such as PNG or TIF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c) in these scans the backgrounds and bleed-through (shadow from the text on the other side of the paper) is visible. Good binarization needs to be used to remove background and bleed-through, but not remove vital parts of actual characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After implementing specific pre-processing solutions for the items listed above, and then using a high quality OCR system, such as www.ocr-it.com API, highest possible results can be achieved.  Result is far from perfect, but it is as high accuracy as it could be achieved with a modern OCR engine on these images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/v5biL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Luckily for this project, the data needs to be counted, so the second stage has all necessary data for reliable data post-processing analysis. Contrary to other basic OCR engines, the OCR provided by &lt;a href=&quot;http://www.ocr-it.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;www.ocr-it.com&lt;/a&gt; API, which I used to produce the above recognition.  OCR-IT API is free to develop with, and very inexpensive to use per page, so that may be a very economical but powerful solution for this project.  It returns formatted text layout, including preserving line breaks and overall format structure.  This makes text post-processing easier.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple algorithm can be run to count the number of lines, resulting in the necessary for the research count.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above describes a two-stage approach: getting best possible OCR result, and using an applicable method to process data for the required task&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bat wait, there is more…&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a second option to use an even more specialized OCR application called &lt;a href=&quot;http://www.wisetrend.com/abbyy-flexicapture/&quot; rel=&quot;nofollow noreferrer&quot;&gt;FlexiCapture&lt;/a&gt; with FlexiLayout technology.  This powerful and intelligent data capture technology has built-in high-accuracy OCR, and it has a powerful rules and data analytics engine to perform very specialized user-defined chains of actions and tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The implementation of this method using FlexiCapture with FlexiLayout takes the following logical steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, full page OCR is performed and all objects are extracted, including characters, noise, black horizontal and vertical lines, white gaps, and objects (which could be pictures, logos, handwriting, etc.).  This produces objects upon which we can apply our search criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, the following constraints are applied to the post-OCR data analysis and search criteria: separate image into three vertical columns and run the following logic per column, use line-start as individual count, skip header/footer/indented lines (county names), assume each name to have at least three characters, find recursively every name starting from top to bottom in every column, exclude previously found lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the above logic sounds complex to setup, the actual setup takes just a few minutes and requires minimal work through user interface (UI) environment.  No coding or programming is necessary.   The following search elements and criteria have been created.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/MOPHu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RepeatingGroup consisting of a CharacterString search object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This setup produces the following search result for the first column of data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/KzpKF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the last step, FlexiCapture is instructed to return the number of total found elements that fit our search criteria, effectively producing the necessary data for the research task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other logic alternatives that can be setup in FlexiCapture, such as finding the number of white spaces between lines, or searching for the fixed-length fixed-placement 3-letter combinations at the end of every column linen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, there are several options (which is always nice) how this task can be achieved with relative ease in effort and high quality, but the success depends on the quality of tools used and necessary knowledge how to use them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you believe some of these tools and processes can be beneficial to your project, please contact my directly.  I specialize in these workflows.  Ilya @ WiseTREND.  My company may be able to help with the setup or guidance.  We have participated in various research initiatives, some through donations to a good cause.&lt;/p&gt;&#xA;" OwnerUserId="8255" LastActivityDate="2015-02-13T05:57:36.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5134" PostTypeId="2" ParentId="5129" CreationDate="2015-02-13T06:03:41.233" Score="1" Body="&lt;p&gt;The MOOCs like UDACITY, COURSERA, UDEMY and EDX  are a great place to learn high quality R programming courses free. You can also search on the MOOCTIVITY siter which is a MOOC aggregator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have personally found great courses on R programming at  UDACITY, COURSERA and UDEMY and also EDX.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck !!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gopinath SUbbegowda&lt;/p&gt;&#xA;" OwnerUserId="8257" LastActivityDate="2015-02-13T06:03:41.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5135" PostTypeId="2" ParentId="5119" CreationDate="2015-02-13T09:06:33.070" Score="2" Body="&lt;p&gt;Naive Bayes is just one of the several approaches that you may apply in order to solve the Titanic's problem. The aim of the Kaggle's Titanic problem is to build a &lt;strong&gt;classification&lt;/strong&gt; system that is able to predict one outcome (whether one person survived or not) given some input data. The survival table is a &lt;strong&gt;training dataset&lt;/strong&gt;, that is, a table containing a set of examples to train your system with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I mentioned before, you could apply Naive Bayes to build your classification system to solve the Titanic problem. Naive Bayes is one of the simplest classification algorithms out there. It assumes that the data in your dataset has a very specific structure. Sometimes Naive Bayes can provide you with results that are good enough. Even if that is not the case, Naive Bayes may be useful as a first step; the information you obtain by analyzing Naive Bayes' results, and by further data analysis, will help you to choose which classification algorithm you could try next. Other examples of classification methods are k-nearest neighbours, neural networks, and logistic regression, but this is just a short list. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are new to Machine Learning, I recommend you to take a look to this course from Stanford: &lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/course/ml&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-02-13T09:06:33.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5136" PostTypeId="2" ParentId="5129" CreationDate="2015-02-13T15:04:24.963" Score="1" Body="&lt;p&gt;I second saq7 and Gopinath, the R courses on Coursera are excellent. I really rate the Johns Hopkins ones: &lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1/courses&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specialization/jhudatascience/1/courses&lt;/a&gt;. You should also keep an eye on the &lt;a href=&quot;http://software-carpentry.org/&quot; rel=&quot;nofollow&quot;&gt;software carpentry&lt;/a&gt; site for courses they run in your area. If you can't wait, all the software carpentry &lt;a href=&quot;http://swcarpentry.github.io/r-novice-inflammation/&quot; rel=&quot;nofollow&quot;&gt;learning material is online&lt;/a&gt; so you can follow it yourself.&lt;/p&gt;&#xA;" OwnerUserId="8021" LastActivityDate="2015-02-13T15:04:24.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5137" PostTypeId="1" CreationDate="2015-02-14T07:09:01.527" Score="-5" ViewCount="337" Body="&lt;p&gt;I am pursing Data Analyst course at Udacity. I came across this video : &lt;a href=&quot;https://www.youtube.com/watch?v=3Z73Wd2T1xE&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=3Z73Wd2T1xE&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Watching it lead me to wonder if &lt;a href=&quot;http://www.ayasdi.com/&quot; rel=&quot;nofollow&quot;&gt;Ayasdi&lt;/a&gt; products would reduce the demand for data scientists. I wish to compete in Kaggle contests but after watching the video, I feel that many of those problems can be easily solved using their platform and that I will be at disadvantage since I do not have access to their tools. Also I feel such tools would reduce the need of experts in data sciences. Now I am worried if I should continue with Nano Degree in Data Analyst at Udacity.&lt;/p&gt;&#xA;" OwnerUserId="8268" LastEditorUserId="75" LastEditDate="2015-05-26T06:36:02.183" LastActivityDate="2015-05-26T06:36:02.183" Title="Will demand for data scientist decrease because of AYASDI?" Tags="|data-mining|bigdata|" AnswerCount="1" CommentCount="4" ClosedDate="2016-02-21T10:55:17.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="5138" PostTypeId="1" CreationDate="2015-02-14T17:13:31.473" Score="2" ViewCount="838" Body="&lt;p&gt;I am confused by the definition of the likelihood function in different contexts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of linear and logistic regression, it is defined as y given x&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case naive bayes and LDA, it is defined over x and y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of the EM algorithm (with observed variable x and unobserved variable z), it is defined over x&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we know over what it should be defined?&lt;/p&gt;&#xA;" OwnerUserId="8273" LastEditorUserId="8273" LastEditDate="2015-02-15T13:20:23.303" LastActivityDate="2015-02-16T20:03:48.880" Title="Definition of likelihood function" Tags="|machine-learning|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5140" PostTypeId="2" ParentId="5138" CreationDate="2015-02-14T21:45:05.987" Score="3" Body="&lt;p&gt;I also found confusing the definition of likelihood the first time I encountered it. The only thing you need to remember is that likelihood refers to the probability of your data given a parameter. That is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(\theta| X) = P(X|\theta)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the likelihood focuses on the parameter, so you're asking &quot;What is the likelihood of this parameter given some observations?&quot; In this case, the phrase &quot;given some observation&quot; assumes the observations $X$ are fixed and the parameter $\theta$ is not fixed. On the other hand, if you describe the same in terms of probability, you're trying to measure the probability of some observations $X$ given a fixed parameter $\theta$. It is only a matter of usage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of logistic regression, the likelihood function is described as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(\beta| y) = \prod_{i}^{N} \pi_{i}^{y_{i}}(1-\pi_{i})^{n_{i} - y_{i}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\beta$ is the vector of parameters $\{\pi_{i}| i=1,...,N \}$and $y$ is the vector of Bernoulli variables. As you can see, this is the same (by definition) as $P(y|\beta)$ and that is what you have in the equation above: The product of the probabilities for each observation $y_{i}$ given its parameter $\pi_{i}$. As usual, each probability $\pi_{i}$ is equal to the inverse of the logit function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Naive Bayes, you have the same basic idea. The likelihood is defined as $L(p_{k}|x) = \prod_{i}^{N}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}$ where $p_{k}$ is a vector of parameters describing the probabilities of a class $k$ generating a word $x_{i}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the EM algorithm, the likelihood is again $L(\theta|X,Z)$, where $\theta$ are your parameters, $X$ your observations and $Z$ are latent variables. The fact that you have a latent variables shouldn't confuse you, because this variable will be &quot;integrated out&quot;  in the expectation step of the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the previous equations, the calculation you need to do is self-evident if you think about it starting from your observations: First, I have some observations. Then I want to calculate the probability of obtaining these observation using some model. Most models require parameters to be fitted. So I want to calculate the probability of these observations given those parameters. That is, $p(X|\theta)$. Now, if you need to talk about likeliihood, then you simply write $L(\theta|X)$.&lt;/p&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="4621" LastEditDate="2015-02-16T20:03:48.880" LastActivityDate="2015-02-16T20:03:48.880" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5141" PostTypeId="2" ParentId="61" CreationDate="2015-02-14T22:45:50.003" Score="5" Body="&lt;p&gt;What got me to understand the problem about overfitting was by imagining what the most overfit model possible would be. Essentially, it would be a simple look-up table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You tell the model what attributes each piece of data has and it simply remembers it and does nothing more with it. If you give it a piece of data that it's seen before, it looks it up and simply regurgitates what you told it earlier. If you give it data it &lt;em&gt;hasn't&lt;/em&gt; seen before, the outcome is unpredictable or random. But the point of machine learning isn't to tell you what happened, it's to understand the patterns and use those patterns to predict what's going on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So think of a decision tree. If you keep growing your decision tree bigger and bigger, eventually you'll wind up with a tree in which every leaf node is based on exactly one data point. You've just found a backdoor way of creating a look-up table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to generalize your results to figure out what might happen in the future, you must create a model that generalizes what's going on in your training set. Overfit models do a great job of describing the data you already have, but descriptive models are not necessarily predictive models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The No Free Lunch Theorem says that no model can outperform any other model on the set of all &lt;em&gt;possible&lt;/em&gt; instances. If you want to predict what will come next in the sequence of numbers &quot;2, 4, 16, 32&quot; you can't build a model more accurate than any other if you don't make the assumption that there's an underlying pattern. A model that's overfit isn't really evaluating the patterns - it's simply modeling what it knows is possible and giving you the observations. You get predictive power by assuming that there is some underlying function and that if you can determine what that function is, you can predict the outcome of events. But if there really is no pattern, then you're out of luck and all you can hope for is a look-up table to tell you what you know is possible.&lt;/p&gt;&#xA;" OwnerUserId="9768" LastActivityDate="2015-02-14T22:45:50.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5142" PostTypeId="1" AcceptedAnswerId="5631" CreationDate="2015-02-15T04:51:21.167" Score="5" ViewCount="1413" Body="&lt;p&gt;I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH into a cloud environment, such as a cluster on AWS? Perhaps there are many tasks where a single-node cluster is adequate, and can be run locally more easily.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-10-23T07:42:20.233" Title="Local Development for Apache Spark" Tags="|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5145" PostTypeId="1" CreationDate="2015-02-15T14:00:31.457" Score="1" ViewCount="2483" Body="&lt;p&gt;Hi I would appreciate it if someone can point me in the right direction.  I'm looking for an algorithm or mathematical theory which I would use to compute the similarity between two ordered lists, where each list element can have n sub-elements.  I will explain with an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I go to a baseball game and I record the sequence of strikes and balls for each of the first 30 players at bat.  My list looks like this, where P is a player, S is a strike and B is a ball. Order matters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;L1: {P1=(S,S,S)}, {P2=(B,B,S)}, {P3=(B,B,S,S)}, ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My friend goes to a baseball game and does the same thing.  Later, we meet up and compare our lists.  We find that our lists are almost identical except that I recorded a strike for player 16 where my friend recorded a ball.  What are the chances we were at the same game and one of us made a mistake at player 16?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance...&lt;/p&gt;&#xA;" OwnerUserId="8281" LastActivityDate="2015-04-16T18:30:00.477" Title="Comparing two ordered lists" Tags="|data-mining|statistics|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5146" PostTypeId="2" ParentId="5145" CreationDate="2015-02-15T14:14:47.110" Score="2" Body="&lt;p&gt;What about a model stating that you have two vectors of size N (where N is the total number of players, maybe unknown to us) where each element belongs to a space of {B,S} sequences, maybe empty. If you then define a distance function between two arbitrary sequences (say, normalized Levenshtein distance for two non-empty ones and some fixed cost when one is missing), you can define cosine similarity between the vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Obviously, you just consider your sequence a compact representation of sparse vector in this case.)&lt;/p&gt;&#xA;" OwnerUserId="5249" LastEditorUserId="5249" LastEditDate="2015-02-15T14:20:40.370" LastActivityDate="2015-02-15T14:20:40.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5147" PostTypeId="1" AcceptedAnswerId="5752" CreationDate="2015-02-15T17:01:50.720" Score="2" ViewCount="719" Body="&lt;p&gt;What are the best known Data Science Methodologies today? By methodology I mean a step-by-step phased process that can be used for framing guidance, although I will be grateful for something close too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To help clarify, there are methodologies in the programming world, like Extreme Programming, Feature Driven Development, Unified Process, and many more. I am looking for their equivalents, if they exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A google search did not turn up much, but I find it hard to believe there is nothing out there. Any ideas? &lt;/p&gt;&#xA;" OwnerUserId="7720" LastEditorUserId="7720" LastEditDate="2015-02-15T18:03:06.890" LastActivityDate="2023-12-01T17:39:11.387" Title="Data Science Methodologies" Tags="|methods|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5148" PostTypeId="2" ParentId="5147" CreationDate="2015-02-15T17:35:11.930" Score="2" Body="&lt;p&gt;Can you elaborate what you mean by 'methodologies'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meantime, take a look at &lt;a href=&quot;http://www.boozallen.com/insights/2013/11/data-science-field-guide&quot; rel=&quot;nofollow&quot;&gt;The Field Guide To Data Science&lt;/a&gt; by Booz Allen Hamilton. This guide talks about data science processes and frameworks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mosaicdatascience.com/resources/data-science-design-patterns/&quot; rel=&quot;nofollow&quot;&gt;Data Science Design Patterns&lt;/a&gt; by Mosaic talks about, you guessed it, data science design patterns. This is quite useful to get a sense of common design patterns. They are also working on releasing a book on the same subject. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then there are several resources out there that will come up as results to more targeted searches, such as machine learning paradigms, recommender systems paradigms, etc. Data Science is a large and varied field, and you'll find many resources out there for each subsection of it. As far as I know, there isn't one book that covers it all. &lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2015-02-15T17:35:11.930" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5149" PostTypeId="2" ParentId="5145" CreationDate="2015-02-15T17:46:34.887" Score="1" Body="&lt;p&gt;Alex's answer is spot on in how you would go about figuring out similarity. One more step required, to answer your question, is to come up with a threshold of similarity. I.e. some similarity threshold beyond which you can say that the discrepancies are probably errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for resources to learn more about this, I'd recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0123814790&quot; rel=&quot;nofollow&quot;&gt;Data Mining Concepts and Techniques&lt;/a&gt; by Han et all&lt;/p&gt;&#xA;" OwnerUserId="90" LastActivityDate="2015-02-15T17:46:34.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5150" PostTypeId="2" ParentId="5147" CreationDate="2015-02-15T23:07:23.753" Score="0" Body="&lt;p&gt;I'm currently writing a book about Data Science in Higher Education, and the following methodologies are the ones I am including:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;em&gt;regression&lt;/em&gt;, we have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Simple Linear Regression&lt;/li&gt;&#xA;&lt;li&gt;Multiple Linear Regression&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For &lt;em&gt;classification&lt;/em&gt;, we have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Naive Bayes Classifier&lt;/li&gt;&#xA;&lt;li&gt;Decision Tree Induction&lt;/li&gt;&#xA;&lt;li&gt;K-Nearest Neighbor&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are some of the more elementary topics in statistical analysis (which you could argue is predictive analytics which you could argue is data science), and thus I would suspect they are also the more common. &lt;/p&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2015-02-15T23:07:23.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5151" PostTypeId="2" ParentId="5137" CreationDate="2015-02-15T23:16:47.210" Score="6" Body="&lt;p&gt;I will assume that you have not been working in statistics that long since you are talking about jumping ship because of a new software that's coming out. Let me give you a bit of reassurance: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the past fifty or so years computers have been solving complex mathematical problems for statisticians and mathematicians, and now with the type of computing power we have today we are able to process terabytes of data per second and develop extremely sophisticated predictive models in a matter of seconds. Software has constantly evolved and constantly made people's lives easier. Has that changed the fact that we still need people to interpret the data? The model? The results? No, it hasn't. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you should know that when software comes out that eliminates a lot of the grunt work, well, you no longer need to employ undergrads/post-baccs to complete those grunt tasks. Now you can use a software to do it, and higher people of a stronger caliber to perform more complex sets of analyses. The more software evolves, the more training a data scientist will require before breaking into the field. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you see, it's not that completing the nano degree alone will or will not give you a competitive edge in the data science job market, it's that this nano degree is &lt;em&gt;the first step&lt;/em&gt; in your journey toward becoming a data scientist. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anything, software like this should encourage all of us to hit the books and ensure that we're still up to date on our game. No amount of bangs and whistles will teach a computer judgement. Prediction? Yes. But not judgement. We don't have Skynet yet. &lt;/p&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2015-02-15T23:16:47.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5152" PostTypeId="2" ParentId="2504" CreationDate="2015-02-16T02:32:21.373" Score="57" Body="&lt;p&gt;Why restrict yourself to those two approaches? Because they're cool? I would always start with a simple linear classifier \ regressor. So in this case a Linear SVM or Logistic Regression, preferably with an algorithm implementation that can take advantage of sparsity due to the size of the data. It will take a long time to run a DL algorithm on that dataset, and I would only normally try deep learning on specialist problems where there's some hierarchical structure in the data, such as images or text. It's overkill for a lot of simpler learning problems, and takes a lot of time and expertise to learn and also DL algorithms are very slow to train. Additionally, just because you have 50M rows, doesn't mean you need to use the entire dataset to get good results. Depending on the data, you may get good results with a sample of a few 100,000 rows or a few million. I would start simple, with a small sample and a linear classifier, and get more complicated from there if the results are not satisfactory. At least that way you'll get a baseline. We've often found simple linear models to out perform more sophisticated models on most tasks, so you want to always start there.&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2015-02-16T02:32:21.373" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5153" PostTypeId="2" ParentId="5109" CreationDate="2015-02-16T03:03:43.883" Score="3" Body="&lt;p&gt;I'm doing some similar research, and have found PluralSight, &lt;a href=&quot;http://pluralsight.com&quot; rel=&quot;nofollow&quot;&gt;http://pluralsight.com&lt;/a&gt;, to be an invaluable resource.  They have video courses on Machine Learning, AWS, Azure, Hadoop, Big Data, etc.  Personally, I find that these video courses allow me to learn the material much faster and more easily than books.&lt;/p&gt;&#xA;" OwnerUserId="8284" LastActivityDate="2015-02-16T03:03:43.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5155" PostTypeId="1" CreationDate="2015-02-16T12:16:26.923" Score="1" ViewCount="168" Body="&lt;p&gt;Given some clusters created from similarity measures between items, is there a recommended way to assign a new item to an existing cluster based on similarity alone? (i.e. avoiding re-clustering)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Measuring the similarity of a new item to all other items is fairly cheap, so I'm looking for a way of using this to assign it to the cluster it's most likely to belong to. It's also important for it to take cluster size into account (i.e. doesn't unfairly weight towards or against larger clusters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'm trying to sacrifice some clustering accuracy in exchange for avoiding a complete re-clustering when the occasional new item is added.&lt;/p&gt;&#xA;" OwnerUserId="474" LastActivityDate="2015-02-16T13:50:32.200" Title="Assigning new items to existing similarity based clustering" Tags="|clustering|similarity|online-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5156" PostTypeId="2" ParentId="5155" CreationDate="2015-02-16T13:50:32.200" Score="2" Body="&lt;p&gt;I suggest you think about this in terms of &quot;data set&quot; and &quot;training set&quot; (technically, it is also recommended to have a separate test set). Once you have your clusters defined on the training set, your can start using them to classify any amount of new data without recalculating, by simply measuring similarity to cluster centroids, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This doesn't prevent you from deciding to enlarge your training set and data set later, just try to not do that selectively to avoiding overfitting.)&lt;/p&gt;&#xA;" OwnerUserId="5249" LastActivityDate="2015-02-16T13:50:32.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5158" PostTypeId="2" ParentId="2269" CreationDate="2015-02-16T20:14:41.513" Score="9" Body="&lt;p&gt;While I have only had a brief look at it, I think &lt;a href=&quot;https://cocalc.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CoCalc&lt;/a&gt; (formerly &lt;a href=&quot;https://cloud.sagemath.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SageMathCloud&lt;/a&gt;) looks quite promising. I have recommended it to at least one person previously, and they seemed to be quite happy with it. Beyond R support, you also get access to Python, SAGE (as the name indicates), and a few other things. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Make sure to check the &lt;a href=&quot;https://github.com/sagemath/cloud/wiki/FAQ#useR&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; on how to get an R (as opposed to a Python) session in a worksheet.&lt;/p&gt;&#xA;" OwnerUserId="536" LastEditorUserId="96238" LastEditDate="2020-05-02T11:04:54.257" LastActivityDate="2020-05-02T11:04:54.257" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="5160" PostTypeId="1" AcceptedAnswerId="5181" CreationDate="2015-02-17T02:39:20.920" Score="0" ViewCount="40" Body="&lt;p&gt;This is in continuation to a previos question. I would like to know where in practice will the following M-ary tree-structured social network propagation model arise. I am looking for some concrete examples in social media (twitter and youtube etc.). Eventually, I want to do some statistical analysis on such social networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/x14St.jpg&quot; alt=&quot;Example of a binary tree-structured social network&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="8127" LastActivityDate="2015-02-18T23:14:02.563" Title="Where the following propagation model arise in practice" Tags="|statistics|social-network-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5161" PostTypeId="2" ParentId="2269" CreationDate="2015-02-17T06:43:03.820" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://www.rstudio.com/products/rstudio/#Server&quot; rel=&quot;nofollow&quot;&gt;RStudio Server&lt;/a&gt; is definately one of the options, meant exactly for this. I've thought about using it with a cloud virtual machine, but haven't had the need yet. But when I (probably) need to prepare an intro data analysis class for the fall semester, then Rstudio Server is the first option I'll be trying out.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-02-17T06:43:03.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5162" PostTypeId="2" ParentId="2504" CreationDate="2015-02-17T07:02:21.497" Score="4" Body="&lt;p&gt;In addition to other answers (and there's some good link in the comments) it depends on what the problem is or what kinds of questions you want to answer. As I can only suggest based on my own experience, then in case of a classification task, the possible methods can be severely limited based on class balance in dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you go to a larger than around 1:10 class imbalance, then most classification methods just stop working. You'll be left with methods based on random forest and maybe neural nets (haven't tried yet). I work with the class balance in the range of 1:500 to 1:1000 and have found that neither down- or upsampling works. Luckily my dataset is &quot;only&quot; 6mln observations by 200 variables and I'm able to run boosted trees on the whole set in reasonable time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to directly answer your question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;you should come up with a bunch of questions you would want to answer and in case of classification then check the class balances of the target variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;you should check the distribution (not in mathematical sense) of missing values in all of your data and document what you find. Some ML methods are fine with missing values while others are not and you need to look into data imputation (which has its own set of rules and guidelines and problems).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-02-17T07:02:21.497" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5164" PostTypeId="1" CreationDate="2015-02-17T18:07:28.333" Score="0" ViewCount="1175" Body="&lt;p&gt;I'd like to see the top N results for a RandomForestClassifier prediction, ordered by descending probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer may be predict_proba, but I have no idea how to interpret the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help appreciated!&lt;/p&gt;&#xA;" OwnerUserId="5233" LastActivityDate="2015-04-19T15:50:01.387" Title="Getting Scikit-Learn RandomForestClassifier to output Top N results" Tags="|random-forest|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5165" PostTypeId="1" AcceptedAnswerId="5167" CreationDate="2015-02-17T19:10:05.547" Score="0" ViewCount="312" Body="&lt;p&gt;&lt;em&gt;Hadoop&lt;/em&gt; is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: &quot;Better to solve it without Hadoop&quot;?&lt;/p&gt;&#xA;" OwnerUserId="5255" LastActivityDate="2015-02-17T22:50:19.067" Title="When it is time to use Hadoop?" Tags="|bigdata|apache-hadoop|efficiency|scalability|performance|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ClosedDate="2015-02-18T11:13:49.617" ContentLicense="CC BY-SA 3.0" />
  <row Id="5166" PostTypeId="1" AcceptedAnswerId="5223" CreationDate="2015-02-17T21:50:10.210" Score="1" ViewCount="1019" Body="&lt;p&gt;I have the following data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($x^1_i$, $y^1_i$) for $i=1,2,...N_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($x^2_i$, $y^2_i$) for $i=1,2,...N_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;($x^m_i$, $y^m_i$) for $i=1,2,...N_m$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to train a neural net to produce some $y_k$ where $k&amp;lt;=min(N)$ given a input ${x_1, x_2, ..., x_{k-1}}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so any suggestion of documentation/ library I can look at (preferably python)?&lt;/p&gt;&#xA;" OwnerUserId="8302" LastActivityDate="2020-12-17T17:26:07.017" Title="training neural net with multiple sets of time-series data" Tags="|machine-learning|dataset|neural-network|time-series|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5167" PostTypeId="2" ParentId="5165" CreationDate="2015-02-17T22:50:19.067" Score="1" Body="&lt;p&gt;It's an economic calculation, really.  When you have a computing &quot;problem&quot; (in the most general possible sense) that you can't solve with one computer, it makes sense to use a cluster of commodity machines when doing so A. allows you to solve the problem, and B. is cheaper than a forklift upgrade to a bigger computer, or upgrading to specialized hardware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When those things are true, and you are going the &quot;commodity cluster&quot; route, Hadoop makes a lot of sense, especially if the nature of the problem maps (no pun intended) well to MapReduce.  If it doesn't, one shouldn't be scared to consider &quot;older&quot; cluster approaches like a Beowulf cluster using MPI or OpenMP.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, the newer YARN based Hadoop does support a form of MPI, so those worlds are starting to move closer together.&lt;/p&gt;&#xA;" OwnerUserId="6554" LastActivityDate="2015-02-17T22:50:19.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5168" PostTypeId="1" CreationDate="2015-02-17T22:55:26.300" Score="3" ViewCount="108" Body="&lt;p&gt;I have 22 classes of objects but they have very skewed distributions where max class has 100.000 images and the min class has 1600 images. In that setting I would like to hear some possible solutions to this balance problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried followings so far;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;multiply number of instances in the lower classes up to the max class by replicating the instances, possibly adding some noise as well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;changing the learning regarding the class distribution in the given minibatch of the next epoch. (no implemented but in my mind)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What are your suggestions?&lt;/p&gt;&#xA;" OwnerUserId="464" LastActivityDate="2015-08-13T10:42:30.263" Title="What are the possible ways to handle class unbalance in a large scale image recognition problem with Deep Neural Nets?" Tags="|classification|class-imbalance|deep-learning|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5169" PostTypeId="2" ParentId="2504" CreationDate="2015-02-17T23:04:07.277" Score="6" Body="&lt;p&gt;From my perspective, for 5 million instances you need lots of trees to get a good generalization bound (a good model in the layman term). If this is not a problem then go for it,even the exact answer is relying on the nature of your problem. GBT is a good method especially if you have mixed feature types like categorical, numerical and such. In addition, compared to Neural Networks it has lower number of hyperparameters to be tuned. Therefore, it is faster to have a best setting model. One more thing is the alternative of parallel training. You can train multiple trees at the same time with a good CPU. If you are not satisfied with the results then go for Neural Nets since that means your model should be more extensive and should learn higher order information through your data. That is the due of NNs compared to other learning algorithms. &lt;/p&gt;&#xA;" OwnerUserId="464" LastActivityDate="2015-02-17T23:04:07.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5170" PostTypeId="1" CreationDate="2015-02-18T00:08:32.800" Score="1" ViewCount="298" Body="&lt;p&gt;I have used the same methods/parameters to create two decision trees. The trees classify the presence or absence of a medical condition using the presence or absence of various symptoms. There is a tree for Medical Condition #1 and another tree for Medical Condition #2. Both trees are based on the same set of symptoms, rated by patients. If Medical Condition #1 resulted in a much simpler tree than Medical Condition #2, can that suggest Medical Condition #2 is a more complex disease? If so, can anyone point me to a reference that states the complexity/depth of a tree can be representative of a complex condition?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The goal of my analysis is separate from this question. I am trying to figure out all the conclusions I can draw or suggest from my analysis. Yes, I am interested in saying something about the complexity of condition A to B. &#xA;When is a condition complex ? When there are many symptoms as the disease is diagnosed by the symptoms.&#xA;If it's hard to diagnose ? YES&#xA;if the symptoms are severe ? NO&#xA;can s.o. have both conditions ? NO&lt;/p&gt;&#xA;" OwnerUserId="8313" LastEditorUserId="21" LastEditDate="2017-05-18T09:23:48.443" LastActivityDate="2017-05-18T09:23:48.443" Title="depth/complexity of decision trees" Tags="|classification|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5171" PostTypeId="1" CreationDate="2015-02-18T00:33:51.823" Score="1" ViewCount="5409" Body="&lt;p&gt;I'm currently trying to predict a continuous variable using KNN. Instead of treating each neighbor equally I would like to use the weights to create a weighted average. The weights by themselves are not ideal, as the closer a neighbor the more I would like that neighbor to influence the final results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This lead me to consider the inverse of each of the distances, but this doesn't handle the case where an instance is the exact same -&gt; with a distance of 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations on how to properly set the weights of each neighbor relative to their distance? Similar to how the inverse would handle this, but one that allows for 0 values.&lt;/p&gt;&#xA;" OwnerUserId="8314" LastActivityDate="2015-02-19T14:59:05.007" Title="How to appropriately set weights for weighted KNN" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5172" PostTypeId="1" CreationDate="2015-02-18T04:24:51.153" Score="1" ViewCount="110" Body="&lt;p&gt;I have code to generate feature vectors in the style of the bag-of-words model, you can see it on my &lt;a href=&quot;https://github.com/h1395010/perceptron/blob/master/src/file_dict_createur/FileDictCreateur.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;github page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It renders output of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/data/train/politics/p_0.txt, [0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]&#xA;/data/train/science/s_0.txt, [1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0]&#xA;/data/train/atheism/a_0.txt, [0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]&#xA;/data/train/sports/s_1.txt, [0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to move on to the next step of the perceptron but I'm confused about what to do next. I guess the next thing I need to do is generate a feature vector for the 'test' data, isn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, in &lt;a href=&quot;http://en.literateprograms.org/index.php?title=Special:DownloadCode/Perceptron_(Java)&amp;amp;oldid=19184&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt; what is 'teachingOutput' supposed to be, training data vectors? Such as the ones I have above?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried poking around in &lt;a href=&quot;https://stackoverflow.com/questions/21738277/perceptron-learning-most-important-feature&quot;&gt;this implementation&lt;/a&gt; in an attempt to figure out what to do but, alas, to no avail. &lt;/p&gt;&#xA;" OwnerUserId="8285" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-02-18T04:24:51.153" Title="perceptron implementation, preocedure for post-feature vecor: bag-of-words model" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5173" PostTypeId="2" ParentId="5171" CreationDate="2015-02-18T09:04:27.770" Score="1" Body="&lt;p&gt;Technically speaking the inverse of the distance should not pose any problem. Considering the case when you have to classify an observation (instance) whih is identical with one observation from the 'learned' data set, than the inverse distance is not defined by a direct formula since you have to divide by zero. But considering the math behind, inverse of the zero distance is positive infinity. Which means that, the weight for the identical point would dominate all other weights of the data instances. Thus the classification for that specific point could be taken as the class of the identical observation from the data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand the inverse of the euclidean distance is only one type of distance. You can always apply a kernel function on that to weight non-uniformly the contributions. For example you can choose a normal density with mean equals zero and variance equals 1. Than, your weight would be the value of density function for the euclidean distance. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-02-18T09:04:27.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5175" PostTypeId="1" AcceptedAnswerId="5184" CreationDate="2015-02-18T10:03:13.913" Score="4" ViewCount="16462" Body="&lt;p&gt;I often get the problem when this or that alias name is already used somewhere, and I can't easily find that variable or aggregation to release the name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/kK9mM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some place in Tableau where I can view/edit/reset full list of aliases?&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-02-18T13:07:25.567" LastActivityDate="2018-02-21T04:24:18.393" Title="List of used aliases in Tableau" Tags="|visualization|tools|tableau|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5178" PostTypeId="1" AcceptedAnswerId="6771" CreationDate="2015-02-13T10:09:25.177" Score="61" ViewCount="30038" Body="&lt;p&gt;I am a PhD student of Geophysics and work with large amounts of image data  (hundreds of GB, tens of thousands of files). I know &lt;code&gt;svn&lt;/code&gt; and &lt;code&gt;git&lt;/code&gt; fairly well and come to value a project history, combined with the ability to easily work together and have protection against disk corruption. I find &lt;code&gt;git&lt;/code&gt; also extremely helpful for having consistent backups but I know that git cannot handle large amounts of binary data efficiently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my masters studies I worked on data sets of similar size (also images) and had a lot of problems keeping track of different version on different servers/devices. Diffing 100GB over the network really isn't fun, and cost me a lot of time and effort.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that others in science seem to have similar problems, yet I couldn't find a good solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use the storage facilities of my institute, so I need something that can use a &quot;dumb&quot; server. I also would like to have an additional backup on a portable hard disk, because I would like to avoid transferring hundreds of GB over the network wherever possible. So, I need a tool that can handle more than one remote location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I really need something that other researcher can use, so it does not need to be super simple, but should be learnable in a few hours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have evaluated a lot of different solutions, but none seem to fit the bill:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://subversion.apache.org/&quot;&gt;svn&lt;/a&gt; is somewhat inefficient and needs a smart server&lt;/li&gt;&#xA;&lt;li&gt;hg &lt;a href=&quot;http://mercurial.selenic.com/wiki/BigfilesExtension&quot;&gt;bigfile&lt;/a&gt;/&lt;a href=&quot;http://mercurial.selenic.com/wiki/LargefilesExtension&quot;&gt;largefile&lt;/a&gt; can only use one remote&lt;/li&gt;&#xA;&lt;li&gt;git &lt;a href=&quot;https://github.com/beenje/git-bigfile&quot;&gt;bigfile&lt;/a&gt;/&lt;a href=&quot;https://github.com/alebedev/git-media&quot;&gt;media&lt;/a&gt; can also use only one remote, but is also not very efficient&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://attic-backup.org/&quot;&gt;attic&lt;/a&gt; doesn't seem to have a log, or diffing capabilities&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://bup.github.io/&quot;&gt;bup&lt;/a&gt; looks really good, but needs a &quot;smart&quot; server to work&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've tried &lt;code&gt;git-annex&lt;/code&gt;, which does everything I need it to do (and much more), but it is very difficult to use and not well documented. I've used it for several days and couldn't get my head around it, so I doubt any other coworker would be interested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do researchers deal with large datasets, and what are other research groups using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be clear, I am primarily interested in how other researchers deal with this situation, not just this specific dataset. It seems to me that almost everyone should have this problem, yet I don't know anyone who has solved it. Should I just keep a backup of the original data and forget all this version control stuff? Is that what everyone else is doing?&lt;/p&gt;&#xA;" OwnerUserId="8320" OwnerDisplayName="Johann" LastEditorUserId="2452" LastEditDate="2015-02-19T05:31:32.323" LastActivityDate="2023-09-03T17:20:37.403" Title="How to deal with version control of large amounts of (binary) data" Tags="|bigdata|databases|binary|version-control|" AnswerCount="10" CommentCount="17" ContentLicense="CC BY-SA 3.0" />
  <row Id="5179" PostTypeId="2" ParentId="5178" CreationDate="2015-02-13T13:38:56.797" Score="9" Body="&lt;p&gt;I have dealt with similar problems with very large synthetic biology datasets, where we have many, many GB of &lt;a href=&quot;http://en.wikipedia.org/wiki/Flow_cytometry&quot;&gt;flow cytometry&lt;/a&gt; data spread across many, many thousands of files, and need to maintain them consistently between collaborating groups at (multiple) different institutions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical version control like svn and git is not practical for this circumstance, because it's just not designed for this type of dataset.  Instead, we have fallen to using &quot;cloud storage&quot; solutions, particularly &lt;a href=&quot;https://www.dropbox.com/&quot;&gt;DropBox&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/BitTorrent_Sync&quot;&gt;Bittorrent Sync&lt;/a&gt;.  DropBox has the advantage that it does do at least some primitive logging and version control and manages the servers for you, but the disadvantage that it's a commercial service, you have to pay for large storage, and you're putting your unpublished data on a commercial storage; you don't have to pay much, though, so it's a viable option.  Bittorrent Sync has a very similar interface, but you run it yourself on your own storage servers and it doesn't have any version control.  Both of them hurt my programmer soul, but they're the best solutions my collaborators and I have found so far.&lt;/p&gt;&#xA;" OwnerDisplayName="jakebeal" LastActivityDate="2015-02-13T13:38:56.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5180" PostTypeId="2" ParentId="5164" CreationDate="2015-02-18T14:07:39.487" Score="1" Body="&lt;p&gt;As there isn't too much code/context you're giving us I assume that you're working with a binary classification problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;X = ... # input for classification, shape (n_samples, n_features)&#xA;y_pred = rf.predict_proba(X)[:, 1] # index slicing to retrieve a 1d-array of probabilities&#xA;y_pred&#xA;# array([  0.18, 0.21, 0.1, 0.2, 0.3])&#xA;&#xA;# Now lets see what the 2 biggest probabilities are&#xA;np.argsort(y_pred)&#xA;# [2, 0, 3, 1, 4] =&amp;gt; lowest probability is at index 2 (the 3rd probability as classified by the RandomForestClassifier)&#xA;# Now lets see what the top 5 samples look like (remember, argsort sorts in ascending order, so we take the last 5 indices of argsort):&#xA;X[np.argsort(y_pred)[-5:]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you should see the rows in X with the 5 most confident predictions.&lt;/p&gt;&#xA;" OwnerUserId="8319" LastActivityDate="2015-02-18T14:07:39.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5181" PostTypeId="2" ParentId="5160" CreationDate="2015-02-18T23:14:02.563" Score="1" Body="&lt;p&gt;Imagine each node in your graph as a user, and each edge as an action such as 'share'. It may also be a bidirectional relationship 'share' and 'view'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some social scientists and engineers estimate the probability that a message is 'shared' and 'viewed' given that a particular user decides to share it. This process is called &quot;information diffusion&quot;, &quot;information propagation&quot;, &quot;cascading&quot;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in the details of how these calculations are performed, check out these papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs.stanford.edu/people/jure/pubs/netrate-netsci14.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cs.stanford.edu/people/jure/pubs/netrate-netsci14.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs.stanford.edu/people/jure/pubs/cascades-www14.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cs.stanford.edu/people/jure/pubs/cascades-www14.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs.stanford.edu/people/jure/pubs/infopath-wsdm13.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cs.stanford.edu/people/jure/pubs/infopath-wsdm13.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar topics:&#xA;&lt;a href=&quot;http://snap.stanford.edu/papers.html&quot; rel=&quot;nofollow&quot;&gt;http://snap.stanford.edu/papers.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-02-18T23:14:02.563" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5182" PostTypeId="2" ParentId="5178" CreationDate="2015-02-19T06:30:45.190" Score="9" Body="&lt;p&gt;I have used &lt;a href=&quot;http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html&quot;&gt;Versioning on Amazon S3 buckets&lt;/a&gt; to manage 10-100GB in 10-100 files. Transfer can be slow, so it has helped to compress and transfer in parallel, or just run computations on EC2. The &lt;a href=&quot;https://github.com/boto/boto&quot;&gt;boto&lt;/a&gt; library provides a nice python interface.&lt;/p&gt;&#xA;" OwnerUserId="8335" LastActivityDate="2015-02-19T06:30:45.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5184" PostTypeId="2" ParentId="5175" CreationDate="2015-02-19T12:06:08.173" Score="7" Body="&lt;p&gt;Finally solved the issue.&#xA;You should go to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data –&amp;gt; your_data_source –&amp;gt; Edit Aliases –&amp;gt; Measure Names&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And see full mapping between variables/aggregations and aliases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/uSoHC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2015-02-19T12:06:08.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5185" PostTypeId="2" ParentId="5171" CreationDate="2015-02-19T14:59:05.007" Score="1" Body="&lt;p&gt;A decaying exponential of the form $e^{-\alpha x}$ (where $x$ is the distance from the observation) is convenient in this situation. It has the nice feature that the weight is equal to $1$ when the observation lies exactly at one of your training points and decays to zero as $x\rightarrow \infty$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need to decide is the scaling factor, $\alpha$, which can greatly affect your results, since the decaying exponential is nonlinear. If $\alpha$ is too small or large, then all $k$ points will be weighted nearly equally (assuming none of the $k$ points have a distance very close to zero). One approach is to pick a global value of $\alpha$ that is suitable for your data set. Another approach is to compute a new value of $\alpha$ for each set of $k$ neighbors under consideration, which will guarantee variability in the weights (as long as all $k$ neighbors are not equidistant).&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-02-19T14:59:05.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5186" PostTypeId="1" CreationDate="2015-02-19T16:23:15.093" Score="1" ViewCount="194" Body="&lt;p&gt;I need to draw a decision tree about this subject :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The research and development manager in an old oil company, which is&#xA;  considering making some changes, lists the following courses of action&#xA;  for the company:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(i) Adopt a process developed by another oil company. This would cost&#xA;  €7 million in royalties and yield a net €20 million profit (before&#xA;  paying the royalty).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(ii) Carry out one or two (not simultaneously) alternative research&#xA;  projects :&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(R1) the more expensive one has a 0.8 chance of success; net profit&#xA;  €16 million and a further €6 million in royalties. If it fails there&#xA;  will be a net loss of €10 million. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(R2) the alternative research programme is less expensive but only has&#xA;  a 0.7 chance of success with a net profit of €15 million and a further&#xA;  €5 million in royalties. If it fails a net loss of €6 million will be&#xA;  incurred.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(iii) Make no changes. After meeting current operating costs the&#xA;  company expects to make a net €15 million profit from its existing&#xA;  process. Failure of one research program would still leave open all&#xA;  remaining courses of action (including the other research programme).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I need also to indicate the different payoffs. This is what I've done so far :&#xA;&lt;img src=&quot;https://i.stack.imgur.com/BLTJ6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be sure that I'm going in the right direction since I'm a beginner with decision trees. And then I need to decide the best course of action using Bayes, Maximax and Maximin rules.&lt;/p&gt;&#xA;" OwnerUserId="8343" LastEditorUserId="8878" LastEditDate="2017-05-11T07:49:57.700" LastActivityDate="2017-05-11T07:49:57.700" Title="Decision Tree Bayes rules / Maximax / Maximin" Tags="|data-mining|statistics|visualization|decision-trees|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5188" PostTypeId="2" ParentId="5170" CreationDate="2015-02-19T22:17:45.773" Score="2" Body="&lt;p&gt;No, you can't infer that.&#xA;I assume you have the same training set with the same predictors (symptoms). The only difference in the training set is the binary class label for each patient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The smaller tree just means:&#xA;- with the given symptoms, it might be easier to distinguish between the people who have the condition 1 and the ones not having it. (since you have the same certainty with fewer information)&#xA;This distinction seems harder with medical condition 2, that's way you have to consider more symptoms to be pretty sure about your classification. So if condition 2 is a very mild condition where it is hard to diagnose even for an expert if someone has it then it would result in a deep tree.&lt;/p&gt;&#xA;" OwnerUserId="8191" LastActivityDate="2015-02-19T22:17:45.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5192" PostTypeId="1" AcceptedAnswerId="5234" CreationDate="2015-02-20T12:21:31.560" Score="5" ViewCount="266" Body="&lt;p&gt;I'm quite new to Data Science, but I would like to do a project to learn more about it.&#xA;My subject will be Data Understanding in Public Health.&#xA;So I want to do some introductory research to public health.&#xA;I would like to visualize some data with the use of a tool like Tableau.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which path would you take to develop a good understanding of Data Science? I imagine taking some online courses, eg. Udacity courses on data science, but which courses would you recommend?&#xA;Where can I get real data (secondary Dummy Data) to work with?&#xA;And are there any good resources on research papers done in Data Science area with the subject of Public Health?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions and comments are welcome.&lt;/p&gt;&#xA;" OwnerUserId="7887" LastEditorUserId="2452" LastEditDate="2015-02-25T21:49:01.210" LastActivityDate="2016-08-17T20:22:51.343" Title="Where can I find resources and papers regarding Data Science in the area of Public Health" Tags="|visualization|tableau|research|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5193" PostTypeId="1" CreationDate="2015-02-20T14:09:23.483" Score="6" ViewCount="13536" Body="&lt;p&gt;I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction frauds.&#xA;Thank you very much for your answer.&lt;/p&gt;&#xA;" OwnerUserId="8357" LastActivityDate="2015-07-20T21:15:48.320" Title="Python or R for implementing machine learning algorithms for fraud detection" Tags="|machine-learning|r|python|" AnswerCount="2" CommentCount="0" ClosedDate="2015-02-21T23:22:54.040" ContentLicense="CC BY-SA 3.0" />
  <row Id="5196" PostTypeId="1" CreationDate="2015-02-20T21:05:16.223" Score="0" ViewCount="107" Body="&lt;p&gt;Hello and thanks in advance!  I'd like some advice on a scalability issue and the best way to resolve.  I'm writing an algorithm in R to produce forecasts for several thousand entities.  One entity takes about 43 seconds to generate a forecast and upload the data to my database.  That equates to about 80+ hours for the entire set of entities and that's much too long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities.  Though that would work, is there a better way?  Can Hadoop help at all?  I have little experience with Hadoop so don't really know if it can apply.  Thanks again!&lt;/p&gt;&#xA;" OwnerUserId="8358" LastActivityDate="2015-02-28T21:42:31.383" Title="Scaling thousands of automated forecasts in R" Tags="|r|scalability|forecast|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5197" PostTypeId="1" CreationDate="2015-02-20T21:56:45.690" Score="1" ViewCount="628" Body="&lt;p&gt;A usual way to find association rules in R is the &quot;arules&quot; package, which easily let's use calculate some rules based on the apriori algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for the data i'm using, I have a lot of NULL cases (baskets where no product A o B is present). This means that I need to calculate some null-invariant measures (kulcynski, for instance). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of any package or workable code that let's me implement this as opposed to writing from scratch the entire algorithm?&lt;/p&gt;&#xA;" OwnerUserId="8296" LastActivityDate="2018-03-05T06:30:12.613" Title="Null-invariant measures of association in R" Tags="|data-mining|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5198" PostTypeId="1" AcceptedAnswerId="5210" CreationDate="2015-02-20T23:16:18.767" Score="3" ViewCount="610" Body="&lt;p&gt;Are there commonly accepted ways to visualize the results of a multivariate regression for a non-quantitative audience? In particular, I'm asking how one should present data on coefficients and T statistics (or p-values) for a regression with around 5 independent variables.&lt;/p&gt;&#xA;" OwnerUserId="6403" LastActivityDate="2015-02-23T08:17:30.120" Title="How to visualize multivariate regression results" Tags="|visualization|regression|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5199" PostTypeId="1" AcceptedAnswerId="5201" CreationDate="2015-02-21T01:16:34.387" Score="1" ViewCount="1041" Body="&lt;p&gt;This question is likely somewhat naive.  I know I (and my colleagues) can install and use Python on local machines. But is that really a best practice?  I have no idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there value in setting up a Python &quot;server&quot;?  A box on the network where we develop our data science related Python code.  If so, what are the hardware requirements for such a box?  Do I need to be concerned about any specific packages or conflicts between projects?&lt;/p&gt;&#xA;" OwnerUserId="8368" LastActivityDate="2015-02-21T11:59:31.860" Title="What is a good hardware setup for using Python across multiple users" Tags="|python|" AnswerCount="1" CommentCount="2" ClosedDate="2015-02-21T23:22:32.677" ContentLicense="CC BY-SA 3.0" />
  <row Id="5200" PostTypeId="2" ParentId="5193" CreationDate="2015-02-21T10:08:39.980" Score="9" Body="&lt;p&gt;I would say that it is your call and purely depends on your comfort with (or desire to learn) the language. Both languages have extensive ecosystems of packages/libraries, including some, which could be used for fraud detection. I would consider &lt;strong&gt;anomaly detection&lt;/strong&gt; as the main theme for the topic. Therefore, the following resources illustrate the variety of approaches, methods and tools for the task in each ecosystem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python Ecosystem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;scikit-learn&lt;/code&gt; library: for example, see &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;LSAnomaly&lt;/code&gt;, a Python module, improving &lt;code&gt;OneClassSVM&lt;/code&gt; (a drop-in replacement): see &lt;a href=&quot;http://www.cit.mak.ac.ug/staff/jquinn/software/lsanomaly.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Skyline&lt;/code&gt;: an open source example of implementation, see &lt;a href=&quot;https://github.com/etsy/skyline&quot; rel=&quot;nofollow noreferrer&quot;&gt;its GitHub repo&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;A &lt;a href=&quot;https://stackoverflow.com/q/18970171/2872891&quot;&gt;relevant discussion&lt;/a&gt; on &lt;em&gt;StackOverflow&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/nicolasmiller/pyculiarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;pyculiarity&lt;/a&gt;, a Python port of Twitter's AnomalyDetection R Package (as mentioned in 2nd bullet of R Ecosystem below &quot;Twitter's Anomaly Detection package&quot;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R Ecosystem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/views&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRAN Task Views&lt;/a&gt;, in particular, &lt;em&gt;Time Series&lt;/em&gt;, &lt;em&gt;Multivariate&lt;/em&gt; and &lt;em&gt;Machine Learning&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;Twitter's &lt;a href=&quot;https://github.com/twitter/AnomalyDetection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anomaly Detection package&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.early-warning-signals.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Early Warning Signals (EWS) Toolbox&lt;/a&gt;, which includes &lt;code&gt;earlywarnings&lt;/code&gt; &lt;a href=&quot;http://cran.r-project.org/web/packages/earlywarnings&quot; rel=&quot;nofollow noreferrer&quot;&gt;package&lt;/a&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://h2o.ai&quot; rel=&quot;nofollow noreferrer&quot;&gt;h2o&lt;/a&gt; machine learning platform (interfaces with R) uses &lt;a href=&quot;http://learn.h2o.ai/content/hands-on_training/anomaly_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep learning for anomaly detection&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional General Information&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A &lt;a href=&quot;http://www.autonlab.org/tutorials/biosurv01.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;set of slides&lt;/a&gt;, mentioning a variety of methods for anomaly detection (AD);&lt;/li&gt;&#xA;&lt;li&gt;A &lt;a href=&quot;https://stats.stackexchange.com/q/26688/31372&quot;&gt;relevant discussion&lt;/a&gt; on &lt;em&gt;Cross Validated&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://elki.dbs.ifi.lmu.de&quot; rel=&quot;nofollow noreferrer&quot;&gt;ELKI Data Mining Framework&lt;/a&gt;, implementing a large number of AD (and other) algorithms.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-07-20T21:15:48.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5201" PostTypeId="2" ParentId="5199" CreationDate="2015-02-21T11:59:31.860" Score="0" Body="&lt;p&gt;&lt;strong&gt;Is installing Python locally a good practice?&lt;/strong&gt;  Yes, if you are going to develop in Python, it is always a good idea to have a local environment where you can break things safely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there value in setting up a Python &quot;server&quot;?&lt;/strong&gt;  Yes, but before doing so, be sure to be able to share your code with your colleagues using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Revision_control&quot; rel=&quot;nofollow&quot;&gt;version control system&lt;/a&gt;.  My reasoning would be that, before you move things to a server, you can move a great deal forward by being able to test several different versions in the local environment mentioned above.  Examples of VCS are &lt;a href=&quot;http://git-scm.com&quot; rel=&quot;nofollow&quot;&gt;git&lt;/a&gt;, &lt;a href=&quot;https://subversion.apache.org&quot; rel=&quot;nofollow&quot;&gt;svn&lt;/a&gt;, and for the deep nerds, &lt;a href=&quot;http://darcs.net&quot; rel=&quot;nofollow&quot;&gt;darcs&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, a &quot;Python server&quot; where you can deploy your software once it is integrated into a releasable version is something usually called &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Staging_site&quot; rel=&quot;nofollow&quot;&gt;staging server&lt;/a&gt;&quot;.  There is a whole philosophy in software engineering &amp;mdash; &lt;a href=&quot;http://en.wikipedia.org/wiki/Continuous_integration&quot; rel=&quot;nofollow&quot;&gt;Continuous Integration&lt;/a&gt; &amp;mdash; that advocates staging whatever you have in VCS daily or even on each change.  In the end, this means that some automated program, running on the staging server, checks out your code, sees that it compiles, runs all defined tests and maybe outputs a package with a version number.  Examples of such programs are &lt;a href=&quot;http://jenkins-ci.org&quot; rel=&quot;nofollow&quot;&gt;Jenkins&lt;/a&gt;, &lt;a href=&quot;http://buildbot.net&quot; rel=&quot;nofollow&quot;&gt;Buildbot&lt;/a&gt; (this one is Python-specific), and &lt;a href=&quot;https://travis-ci.org/recent&quot; rel=&quot;nofollow&quot;&gt;Travis&lt;/a&gt; (for cloud-hosted projects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What are the hardware requirements for such a box?&lt;/strong&gt; None, as far as I can tell.  Whenever it runs out of disk space, you will have to clean up.  Having more CPU speed and memory will make concurrent builds easier, but there is no real minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do I need to be concerned about any specific packages or conflicts between projects?&lt;/strong&gt;  Yes, this has been identified as a problem, not only in Python, but in many other systems (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Dependency_hell&quot; rel=&quot;nofollow&quot;&gt;Dependency hell&lt;/a&gt;).  The established practice is to keep projects isolated from each other as far as their dependencies are concerned.  This means, avoid installing dependencies on the system Python interpreter, even locally; always define a &lt;a href=&quot;http://virtualenv.readthedocs.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;virtual environment&lt;/a&gt; and install dependencies there.  Many of the aforementioned CI servers will do that for you anyway.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-02-21T11:59:31.860" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5202" PostTypeId="2" ParentId="5193" CreationDate="2015-02-21T12:19:34.107" Score="2" Body="&lt;p&gt;This is largely a subjective question.  Trying to list some criteria that seem objective to me:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;the important advantage of Python is that it is a general-purpose language.  If you will need to do anything else than statistics with your program (generate a web interface, integrate it with a reporting system, pass it on to other developers for maintenance) you are far better off with Python.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the important advantage of R is that it is a specialized language.  If you already know that there is a technique you want to use, and it is not a usual suspect (like NN), then you probably will find a library in CRAN that makes life easier for you.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And here is another, more subjective advice:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;both languages are not really performance-oriented.  If you need to process large quantities of data, or process very fast, or process in parallel, you will run into trouble ... but it is far easier to run into such trouble with R than with Python.  In my experience, you find the limits of R within some weeks, and the way to push them is quite arcane; while you can use Python for years, never really missing the speed that C developers always mention as the Holy Grail, and even when you do, you can use Cython to make up for the difference.  The only real trouble is concurrency, but for statistics, it is hardly ever an issue.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-02-21T12:19:34.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5203" PostTypeId="2" ParentId="5196" CreationDate="2015-02-21T12:33:53.923" Score="2" Body="&lt;p&gt;If you're working with R language, I would suggest first to try use &lt;strong&gt;R ecosystem&lt;/strong&gt;'s abilities to &lt;em&gt;parallelize&lt;/em&gt; the processing, if possible. For example, take a look at packages, mentioned in &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;this CRAN Task View&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Alternatively&lt;/strong&gt;, if you're not comfortable or satisfied with the approaches, implemented by the above-referred packages, you can try some other approaches, such as Hadoop or something else. I think that a &lt;em&gt;Hadoop&lt;/em&gt; solution would be an &lt;strong&gt;overkill&lt;/strong&gt; for such problem, considering the &lt;em&gt;learning curve&lt;/em&gt;, associated with it, as well as the fact that, as far as I understand, Hadoop or other MapReduce frameworks/architectures target &lt;em&gt;long-running processes&lt;/em&gt; (an average task is ~ 2 hours, I read somewhere recently). Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-02-21T12:33:53.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5204" PostTypeId="1" AcceptedAnswerId="5237" CreationDate="2015-02-21T14:02:39.170" Score="1" ViewCount="373" Body="&lt;p&gt;Most literature focus on either explicit rating data or implicit (like/unknown) data. Are there any good publications to handle like/dislike/unknown data? That is, in the data matrix there are three values, and I'd like to recommend from unknown entries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And are there any good open source implementations on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="1376" LastActivityDate="2015-04-27T12:48:00.957" Title="Matrix factorization for like/dislike/unknown data" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5205" PostTypeId="1" CreationDate="2015-02-22T11:57:57.840" Score="1" ViewCount="1878" Body="&lt;p&gt;I have a pandas dataframe with a salary column which contains values like:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;£36,000 - £40,000 per year plus excellent bene...,&#xA;  £26,658 to £32,547 etc&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I isolated this column and split it with the view to recombining into the data frame later via a column bind in pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now have an object with columns like the below. The columns I split the original data frame column I think are blank because I didn't specify them (I called &lt;code&gt;df['salary']=df['salary'].astype(str).str.split()&lt;/code&gt;&#xA;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my new object contains this type of information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[£26,658, to, £32,547],&#xA;[Competitive, with, Excellent, Benefits]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create three columns called minvalue and maxvalue and realvalue&lt;/li&gt;&#xA;&lt;li&gt;List items starting with £ (something to do with &lt;code&gt;&quot;^£&quot;&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Take till the end of the items found ignoring the £ (get the number out) (something to do with &lt;code&gt;(substr(x,2,nchar(x)))&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;If there are two such items found, call the first number &quot;minvalue&quot; and call the second number &quot;maxvalue&quot; and put it below the right column. If there is only one value in the row, put it below the realvalue column.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am very new to pandas and programming in general, but keen on learning, your help would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="8375" LastEditorUserId="8375" LastEditDate="2015-02-22T12:57:21.040" LastActivityDate="2020-07-31T15:36:10.983" Title="Split columns by finding &quot;£&quot; and then converting to minvalue, maxvalue" Tags="|dataset|statistics|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5206" PostTypeId="2" ParentId="5205" CreationDate="2015-02-22T13:14:36.200" Score="4" Body="&lt;p&gt;This is more of a general regex question, rather than pandas specific.&lt;/p&gt;&#xA;&lt;p&gt;I would first create a function that extracts the numbers you need from strings, and then use the &lt;code&gt;pandas.DataFrame.apply&lt;/code&gt; function to apply it on the pandas column containing the strings. Here is what I would do:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;def parseNumbers(salary_txt):&#xA;    return [int(item.replace(',','')) for item in re.findall('£([\d,]+)',salary_txt)]&#xA;&#xA;#testing if this works&#xA;testcases = ['£23,000 to £100,000','£34,000','£10000']&#xA;for testcase in testcases:&#xA;    print testcase,parseNumbers(testcase)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Here, I just used &lt;code&gt;re.findall&lt;/code&gt;, which finds all patterns that look like &lt;code&gt;£([\d,]+)&lt;/code&gt;. This is anything that starts with £ and is followed by an arbitrary sequence of digits and commas. The parenthesis tells python to extract only the bit after the £ sign. The last thing I do is I remove commas, and parse the remaining string into an integer. You could be more elegant about this I guess, but it works.&lt;/p&gt;&#xA;&lt;h1&gt;using this function in pandas&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;df['salary_list'] = df['salary'].apply(parseNumbers)&#xA;df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)&#xA;df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Checking if this all works:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas&#xA;df = pandas.DataFrame(testcases,columns = ['salary'])&#xA;df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)&#xA;df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)&#xA;df&#xA;&#xA;    salary  minsalary   maxsalary&#xA;0   £23,000 to £100,000 23000   100000&#xA;1   £34,000 34000   34000&#xA;2   £10000  10000   10000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The advantages of moving the parsing logic to a separate function is that:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;it may be reusable in other code&lt;/li&gt;&#xA;&lt;li&gt;it is easier to read for others, even if they aren't pandas experts&lt;/li&gt;&#xA;&lt;li&gt;it's easier to develop and test the parsing functionality in isolation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8376" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-02-22T13:14:36.200" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5208" PostTypeId="1" AcceptedAnswerId="5211" CreationDate="2015-02-22T20:30:30.630" Score="1" ViewCount="218" Body="&lt;p&gt;As an example. If you are tying to classify humans from dogs. Is it possible to approach this problem by classifying different kinds of animals (birds, fish, reptiles, mammals, ...) or even smaller subsets (dogs, cats, whales, lions, ...) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then when you try to classify a new data set, anything that did not fall into one of those classes can be considered a human.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is possible, are there any benefits into breaking a binary class problem into several classes (or perhaps labels)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Benefits I am looking into are: accuracy/precision of the classifier, parallel learning.&lt;/p&gt;&#xA;" OwnerUserId="8381" LastEditorUserId="8381" LastEditDate="2015-02-23T00:19:21.277" LastActivityDate="2015-02-23T11:43:11.833" Title="Splitting binary classification into smaller susbsets" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5209" PostTypeId="1" AcceptedAnswerId="5998" CreationDate="2015-02-23T08:00:03.360" Score="4" ViewCount="3546" Body="&lt;p&gt;I am performing Named Entity Recognition using Stanford NER. I have successfully trained and tested my model. Now I want to know:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What is the general way of measuring accuracy of NER model ?? For example what techniques or approaches are used ??&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is there any built-in method in STANFORD NER for evaluating the accuracy ??&lt;/p&gt;&#xA;" OwnerUserId="8016" LastActivityDate="2015-10-05T20:39:30.453" Title="Accuracy of Stanford NER" Tags="|nlp|performance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5210" PostTypeId="2" ParentId="5198" CreationDate="2015-02-23T08:17:30.120" Score="2" Body="&lt;p&gt;I personally like dotcharts of standardized regression coefficients, possibly with standard error bars to denote uncertainty. Make sure to standardize coefficients (and SEs!) appropriately so they &quot;mean&quot; something to your non-quantitative audience: &quot;As you see, an increase of 1 unit in Z is associated with an increase of 0.3 units in X.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R (without standardization):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;foo &amp;lt;- data.frame(X=rnorm(30),Y=rnorm(30),Z=rnorm(30))&#xA;model &amp;lt;- lm(X~Y+Z,foo)&#xA;&#xA;coefs &amp;lt;- coefficients(model)&#xA;std.errs &amp;lt;- summary(model)$coefficients[,2]&#xA;&#xA;dotchart(coefs,pch=19,xlim=range(c(coefs+std.errs,coefs-std.errs)))&#xA;lines(rbind(coefs+std.errs,coefs-std.errs,NA),rbind(1:3,1:3,NA))&#xA;abline(v=0,lty=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/0AwtB.png&quot; alt=&quot;regression visualization&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2853" LastActivityDate="2015-02-23T08:17:30.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5211" PostTypeId="2" ParentId="5208" CreationDate="2015-02-23T11:43:11.833" Score="1" Body="&lt;p&gt;If you try to get the best accuracy, etc... for a given question you should always learn on a training set that is labeled exactly according to your questions. You shouldn't expect to get better results if you are using more granular class labels. The classifier then would then try to pick up the differences in the classes and try to separate them apart. Since in practice your variables in the training set will not perfectly explain the more granular classification question you shouldn't expect to get a better answer for your less granular classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not happy with the accuracy of your model you try the following instead:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;review the explanatory variables. Think about what might influence the classification problem. Maybe there us a clever way to construct new variables (from your existing ones) that helps. It's nowpossible to give a general advise on that since you have to consider the properties of your classifier&lt;/li&gt;&#xA;&lt;li&gt;if your class distribution is very skewed you might consider over/undersampling &lt;/li&gt;&#xA;&lt;li&gt;you might run more different classifiers and then classify based on the majority vote. Note that you will most likely sacrifice explainability of your model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also you seem to have some missunderstanding, when you write 'you would assign it to human if it doesn't fall into any of the granular classes'. Note that you always try to pick class labels covering the whole universe (all possible classes). This can be always defined as the complement of the other classes. Also you will have to have instances for each class in your training set.&lt;/p&gt;&#xA;" OwnerUserId="8191" LastActivityDate="2015-02-23T11:43:11.833" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5212" PostTypeId="1" AcceptedAnswerId="5220" CreationDate="2015-02-23T12:03:47.773" Score="1" ViewCount="525" Body="&lt;p&gt;We are currently developing a customer relationship management software for SME's. What I'd like to structure for our future CRM is developing CRM with a social-based approach (Social CRM). Therefore we will provide our users (SME's) to integrate their CRM into their social network accounts. Also CRM will be enhance intercorporate communication of owner company.&lt;/p&gt;&#xA;&lt;p&gt;All these processes I've just indicated above will certainly generate lots of unstructured data.&lt;/p&gt;&#xA;&lt;p&gt;I am wondering how can we integrate &lt;i&gt;big data&lt;/i&gt; and &lt;i&gt;data-mining&lt;/i&gt; contepts for our project; especially for the datas generated by social network? I am not the expert of these topics but I really want to start from somewhere.&lt;/p&gt;&#xA;&lt;h3&gt;Basic capabilities of CRM (Modules)&lt;/h3&gt;&#xA;&lt;p&gt;-Contacts: People who you have a business relationship.&lt;/p&gt;&#xA;&lt;p&gt;-Accounts: Clients who you've done a business before.&lt;/p&gt;&#xA;&lt;p&gt;-Leads: Accounts who are your potential customers.&lt;/p&gt;&#xA;&lt;p&gt;-Oppurtunites: Any business opportunity for an account or a lead.&lt;/p&gt;&#xA;&lt;p&gt;-Sales Orders&lt;/p&gt;&#xA;&lt;p&gt;-Calendar&lt;/p&gt;&#xA;&lt;p&gt;-Tasks&lt;/p&gt;&#xA;&lt;p&gt;What kind of unstructured data or the ways (ideas) could be useful for the modules I've just wrote above? If you need more specific information please write in comments.&lt;/p&gt;&#xA;" OwnerUserId="8386" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-07-31T08:33:51.563" Title="Big data and data mining for CRM?" Tags="|data-mining|bigdata|software-development|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5214" PostTypeId="1" AcceptedAnswerId="5216" CreationDate="2015-02-23T15:45:21.700" Score="0" ViewCount="81" Body="&lt;p&gt;I am starting to play around in datamining / machine learning and I am stuck on a problem that's probably easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a report that lists the url and the number of visits a person did. So a combination of ip and url result in an amount of visits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to run the k-means clustering algorithm on this so I thought I could approach it like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;url      ip    visits&#xA;&#xA;abc.be   123   5&#xA;abc.be/a 123   2&#xA;abc.be/b 123   2&#xA;abc.be/b 321   4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I would turn in into a feature vector/matrix like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abc.be  abc.be/a   abc.be/b   impressions&#xA;   1       0          0          5&#xA;   0       1          0          2&#xA;   0       0          1          2&#xA;   0       0          1          4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I am stuck on how to transform my data set to a feature matrix. Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="8389" LastActivityDate="2015-02-23T22:39:53.290" Title="Going from report to feature matrix" Tags="|feature-extraction|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5215" PostTypeId="2" ParentId="5205" CreationDate="2015-02-23T21:57:39.677" Score="0" Body="&lt;p&gt;You can check the data types of your columns by doing &lt;code&gt;df.dtypes&lt;/code&gt;, and if &lt;code&gt;'salary'&lt;/code&gt; isn't a string, you can convert it using &lt;code&gt;df['salary'] = df['salary'].astype(str)&lt;/code&gt;. This is what you were already doing before splitting. From there, Ferenc's method should work!&lt;/p&gt;&#xA;" OwnerUserId="8391" LastEditorUserId="98307" LastEditDate="2020-07-31T15:36:10.983" LastActivityDate="2020-07-31T15:36:10.983" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5216" PostTypeId="2" ParentId="5214" CreationDate="2015-02-23T22:39:53.290" Score="1" Body="&lt;p&gt;I don't understand what you mean by &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So I have a report that lists the url and the number of visits a person did. &lt;strong&gt;So a combination of ip and url result in an amount of visits.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Assuming that you equate an IP with a user, and you wish to cluster users by their URL visitation frequencies, your matrix, &lt;code&gt;M&lt;/code&gt;, would have &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One row per IP (user)&lt;/li&gt;&#xA;&lt;li&gt;One column for each URL that you are tracking (your features)&lt;/li&gt;&#xA;&lt;li&gt;and the entries in &lt;code&gt;M&lt;/code&gt; would be &quot;visits&quot; of a given URL by a particular IP&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Given these assumptions, and your report, &lt;code&gt;M&lt;/code&gt; would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    abc.be  abc.be/a  abc.be/b&#xA;123   5        2         2&#xA;321   0        0         4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8392" LastActivityDate="2015-02-23T22:39:53.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5217" PostTypeId="2" ParentId="2269" CreationDate="2015-02-24T02:11:45.787" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://www.codeschool.com/&quot; rel=&quot;nofollow&quot;&gt;https://www.codeschool.com/&lt;/a&gt; is very similar to &lt;a href=&quot;https://www.datacamp.com/&quot; rel=&quot;nofollow&quot;&gt;https://www.datacamp.com/&lt;/a&gt;&#xA;when I tried it I fell in love with R and then found datacamp.&#xA;www.codecademy.com  is also console-based but R is not yet available.&lt;/p&gt;&#xA;" OwnerUserId="8394" LastActivityDate="2015-02-24T02:11:45.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5220" PostTypeId="2" ParentId="5212" CreationDate="2015-02-24T03:51:15.133" Score="2" Body="&lt;p&gt;The two modules where you can really harness data mining and big data techniques are probably Leads and Opportunities. The reason is that, as you've written yourself, both contain 'potential' information that you can harness (through predictive algorithms) to get more customers. Taking Leads as an example, you can use a variety of machine learning algorithms to assign a probability to each account, based on that account's potential for becoming your customer in the near future. Since you already have an Accounts module which gives you information about your current customers, you can use this information to train your machine learning algorithms. This is all at a very high level but hopefully, you're getting the gist of what I'm saying.&lt;/p&gt;&#xA;" OwnerUserId="8395" LastActivityDate="2015-02-24T03:51:15.133" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5221" PostTypeId="2" ParentId="1151" CreationDate="2015-02-24T04:07:29.860" Score="1" Body="&lt;p&gt;My advice is to begin with a thorough grounding in Statistics, for which a lot of classic and unambiguous material is available online. Topics that you should have a firm grasp on include regression, correlation, hypothesis testing and the bias-variance tradeoff. You don't have to go into too much theoretical depth into any of these topics but you should know what these are before you start studying machine learning. Your study in machine learning should include developing an understanding of hypothesis spaces, Bayesian analysis (can you explain the difference between MLE, MAP and optimal Bayes?), Expectation Maximization, logistic regression, clustering (especially k-means), max-margin classifiers (SVMs), overfitting (can you explain what it is in terms of bias and variance?) and feature selection. Linear algebra is helpful but can be done without in many cases.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As someone who does research in data mining and has worked closely with several companies, I can tell you that if you seriously want to go into machine learning, it is NOT enough to simply know how to code something using Weka or R. Those are easy enough to use once you know the concepts. When companies hire data scientists, they want someone who can take the raw data and do something useful with it. A good grasp of fundamentals is obviously essential, since each company's data has its own quirks (and will typically be too big for you to try 'everything'). Good luck!&lt;/p&gt;&#xA;" OwnerUserId="8395" LastActivityDate="2015-02-24T04:07:29.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5222" PostTypeId="2" ParentId="1151" CreationDate="2015-02-24T08:40:05.083" Score="2" Body="&lt;p&gt;So far the answers have focused on learning particular methods. They are fine, but they won't make you a Data Scientist. Being a Data Scientist is not solely or even primarily about having mastery of particular data analysis methods (ML or others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most fundamental is &lt;em&gt;problem solving&lt;/em&gt; and &lt;em&gt;decision support&lt;/em&gt;. What ever data you collect, what ever analysis methods you apply, and however you improve those methods over time, these must support the over-arching goals of solving problems or making better decisions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to start getting first-hand experience with data in your field.  I don't mean Kaggle data (i.e. already cleaned).  I mean raw data or nearly raw. A good 50% of a data scientist's time is spent wrangling raw data and cleaning it to the point where it's usable in analysis.  You need to learn how to deal with missing data, erroneous data, ambiguous data, misformatted data, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also get some experience with decisions that do not map neatly on to the data.  Recommender systems are easy in this regard.  For example, you might take on the challenge of evaluating software vulnerabilities to guide vulnerability management decisions.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-02-24T08:40:05.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5223" PostTypeId="2" ParentId="5166" CreationDate="2015-02-24T11:52:40.977" Score="2" Body="&lt;p&gt;Yes, this is a straightforward application for neural networks.  In this case yk are the outputs of the last layer (&quot;classifier&quot;); xk is a feature vector and yk is what it gets classified into.  For simplicity prepare your data so that N is the same for all.  The problem you have is perhaps that in the case of time series you won't have &lt;em&gt;enough&lt;/em&gt; data: you need (ideally) many 1000's of examples to train a network, which in this case means time series, not points.  Look at the specialized literature on neural networks for time series prediction for ideas on network architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Library: try Pylearn2 at &lt;a href=&quot;http://deeplearning.net/software/pylearn2/&quot; rel=&quot;nofollow&quot;&gt;http://deeplearning.net/software/pylearn2/&lt;/a&gt;  It's not the only good option but it should serve you well.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2015-02-24T11:52:40.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5224" PostTypeId="1" AcceptedAnswerId="9227" CreationDate="2015-02-24T11:59:36.033" Score="41" ViewCount="41795" Body="&lt;p&gt;I would like to use a neural network for image classification.  I'll start with pre-trained CaffeNet and train it for my application.  &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How should I prepare the input images?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In this case, all the images are of the same object but with variations (think: quality control).  They are at somewhat different scales/resolutions/distances/lighting conditions (and in many cases I don't know the scale).  Also, in each image there is an area (known) around the object of interest that should be ignored by the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could (for example) crop the center of each image, which is guaranteed to contain a portion of the object of interest and none of the ignored area; but that seems like it would throw away information, and also the results wouldn't be really the same scale (maybe 1.5x variation).&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Dataset augmentation&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I've heard of creating more training data by random crop/mirror/etc, is there a standard method for this?  Any results on how much improvement it produces to classifier accuracy?&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="29169" LastEditDate="2022-06-16T19:08:44.980" LastActivityDate="2022-06-16T19:08:44.980" Title="How to prepare/augment images for neural network?" Tags="|neural-network|image-classification|convolutional-neural-network|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5225" PostTypeId="2" ParentId="5192" CreationDate="2015-02-24T22:13:46.327" Score="1" Body="&lt;p&gt;Data science is a new somewhat vague terminology.  You will find much more information by using keywords such as: epidemiology, population health, public health surveillance, public health, statistics, evidence based medicine, biostatistics, statistical epidemiology, clinical decision making, interventions etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question as posted is overly broad. On the other hand you're in luck since there is a substantial amount of information on this topic.  Many of the R/Data Science courses on Coursera from John Hopkins have a public health flavor. Also, R and S-PLUS before it were primarily used in health settings.&lt;/p&gt;&#xA;" OwnerUserId="6467" LastActivityDate="2015-02-24T22:13:46.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5226" PostTypeId="1" AcceptedAnswerId="5229" CreationDate="2015-02-25T01:07:14.717" Score="87" ViewCount="151018" Body="&lt;p&gt;I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, &lt;a href=&quot;http://scikit-learn.org&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt; takes only numbers as parameters, but I want to inject the strings as well as they carry a significant amount of knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I handle such a scenario?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can convert a string to numbers by some mechanism such as hashing in Python. But I would like to know the best practice on how strings are handled in decision tree problems.&lt;/p&gt;&#xA;" OwnerUserId="8409" LastEditorUserId="26686" LastEditDate="2019-10-02T14:32:28.693" LastActivityDate="2020-10-29T06:16:43.570" Title="strings as features in decision tree/random forest" Tags="|machine-learning|python|scikit-learn|random-forest|decision-trees|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="5227" PostTypeId="1" AcceptedAnswerId="5235" CreationDate="2015-02-25T03:24:20.450" Score="1" ViewCount="128" Body="&lt;p&gt;I am struggling to choose a right data prediction method for the following problem.&#xA;Essentially I am trying to model a scheduler operation, trying to predict its scheduling without knowing the scheduling mechanism and having incomplete data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) There are M available resource blocks that can carry data, N data channels that must be scheduled every time instance i&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Inputs into the scheduler:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Matrix $X_i$ size M by N, consisting of N column vectors from each data source.    Each of M elements is index from 1 to 32 carrying information about quality of data channel for particular resource block. 1 - really bad quality, 32 - excellent quality.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data which contains type of data to be carried (voice/internet etc)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Scheduler prioritizes number of resource blocks occupied by each channel every time instant i.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I CAN see resource allocation map every time instant&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I DO have access to matrix $X_i$  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I DON'T know the algorithm of scheduler and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I dont have access to the type of data to be scheduled. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to have a best guess (prediction) how the data will be scheduled based on this incomplete information i.e, which resource block will be occupied by which data channel. What is the best choice of prediction/modelling algorithm?&#xA;Any help appreciated!&lt;/p&gt;&#xA;" OwnerUserId="8410" LastActivityDate="2015-02-26T05:08:31.993" Title="problem of choosing right statistical method for scheduler prediction" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5228" PostTypeId="1" CreationDate="2015-02-25T10:07:24.563" Score="3" ViewCount="669" Body="&lt;p&gt;There are several metrics for the quality of a graph clustering, e.g. Newman modularity. These enable you to compare two candidate clusterings of the same graph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know a metric that will answer the question &quot;how modular is this graph&quot;? For example the first of these two graphs is more modular than the second:&#xA;    o===o-----o====o    o----o===o-----o&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be possible to choose a clustering algorithm, run it, and compute your preferred modularity metric for the best clustering found. But this is only a lower bound, so it doesn't seem very satisfactory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question matters. For example, the work of life scientists will be easier if the molecular organisation of life is modular than if it is not. It would be good to have a robust test - some of the discussion so far seems to involve wishful thinking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My best attempt at this is:&#xA;- a tree is more modular if the edges near leaves are higher weight&#xA;- the modularity of a graph is the modularity of its min cut spanning tree&#xA;Does anyone know of an established answer to this question?&lt;/p&gt;&#xA;" OwnerUserId="8417" LastActivityDate="2018-01-25T07:43:29.207" Title="Graph modularity measure" Tags="|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5229" PostTypeId="2" ParentId="5226" CreationDate="2015-02-25T10:10:55.683" Score="78" Body="&lt;p&gt;In most of the well-established machine learning systems, categorical variables are handled naturally. For example in R you would use factors, in WEKA you would use nominal variables. This is not the case in scikit-learn. The decision trees implemented in scikit-learn uses only numerical features and these features are interpreted always as &lt;strong&gt;continuous numeric variables&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, simply replacing the strings with a hash code should be avoided, because being considered as a continuous numerical feature any coding you will use will induce an order which simply does not exist in your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example is to code ['red','green','blue'] with [1,2,3], would produce weird things like 'red' is lower than 'blue', and if you average a 'red' and a 'blue' you will get a 'green'. Another more subtle example might happen when you code ['low', 'medium', 'high'] with [1,2,3]. In the latter case it might happen to have an ordering which makes sense, however, some subtle inconsistencies might happen when 'medium' in not in the middle of 'low' and 'high'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the answer to your question lies in coding the categorical feature into &lt;strong&gt;multiple binary features&lt;/strong&gt;. For example, you might code ['red','green','blue'] with 3 columns, one for each category, having 1 when the category match and 0 otherwise. This is called &lt;strong&gt;one-hot-encoding&lt;/strong&gt;, binary encoding, one-of-k-encoding or whatever. You can check documentation here for &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html&quot; rel=&quot;noreferrer&quot;&gt;encoding categorical features&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_extraction.html#dict-feature-extraction&quot; rel=&quot;noreferrer&quot;&gt;feature extraction - hashing and dicts&lt;/a&gt;. Obviously one-hot-encoding will expand your space requirements and sometimes it hurts the performance as well. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="26596" LastEditDate="2016-12-04T15:18:47.823" LastActivityDate="2016-12-04T15:18:47.823" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="5230" PostTypeId="2" ParentId="1227" CreationDate="2015-02-25T11:18:26.313" Score="2" Body="&lt;p&gt;Have you heard about the &lt;a href=&quot;http://en.wikipedia.org/wiki/Intersection_graph&quot; rel=&quot;nofollow&quot;&gt;intersection graph&lt;/a&gt;? You can try to draw players as points, connections (team mates) as edges and teams as transparent coloured blobs on top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your original question, I cannot understand your goal. I think your formulation is not well defined / incomplete. Suppose you have teams A [1,2] B [2,3] and C [1,3]. What do you want to display? Do you want to list the parts of the Venn diagram? I think that for more than 3 sets this can become more cumbersome than the bipartite graph itself = simple listing of team compositions.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2015-02-25T11:18:26.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5231" PostTypeId="1" AcceptedAnswerId="5239" CreationDate="2015-02-25T14:10:04.777" Score="1" ViewCount="568" Body="&lt;p&gt;I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that these association rules are true? How can I validate them? What are common practice to test it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about cross validation (with training data and test data) as I read that it's not impossible to use it on unsupervised learning methods..but I'm not sure if it makes sense since I don't use labeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone has a clue, even if it's not specifically about association rules (but testing other unsupervised learning methods), that would also be helpful to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I uploaded an example of the data that I use here in case it's relevant: &lt;a href=&quot;https://www.mediafire.com/?4b1zqpkbjf15iuy&quot; rel=&quot;nofollow&quot;&gt;https://www.mediafire.com/?4b1zqpkbjf15iuy&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8422" LastActivityDate="2015-02-27T00:45:39.740" Title="How to test/validate unlabeled data in association rules in R?" Tags="|machine-learning|r|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5232" PostTypeId="2" ParentId="435" CreationDate="2015-02-25T17:08:09.830" Score="1" Body="&lt;p&gt;&lt;strong&gt;&lt;em&gt;First, some caveats&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure why you can't use your preferred programming (sub-)paradigm*, &lt;em&gt;Inductive Logic Programming (ILP)&lt;/em&gt;, or what it is that you're trying to classify. Giving more detail would probably lead to a much better answer; especially as it's a little unusual to approach selection of classification algorithms on the basis of the programming paradigm with which they're associated. If your real world example is confidential, then simply make up a fictional-but-analogous example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Big Data Classification without ILP&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, after ruling out ILP we have 4 other logic programming paradigms in our consideration set:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Abductive&lt;/li&gt;&#xA;&lt;li&gt;Answer Set&lt;/li&gt;&#xA;&lt;li&gt;Constraint&lt;/li&gt;&#xA;&lt;li&gt;Functional&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;in addition to the dozens of paradigms and sub-paradigms outside of logic programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within &lt;em&gt;Functional Logic Programming&lt;/em&gt; for instance,  there exists extensions of ILP called &lt;em&gt;Inductive Functional Logic Programming&lt;/em&gt;, which is based on inversion narrowing (i.e. inversion of the narrowing mechanism). This approach overcomes several limitations of ILP and (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.225.4755&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;according to some scholars, at least&lt;/a&gt;) is as suitable for application in terms of representation and has the benefit of allowing problems to be expressed in a more natural way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing more about the specifics of your database and the barriers you face to using ILP, I can't know if this solves your problem or suffers from the same problems. As such, I'll throw out a completely different approach as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://books.google.com/books?hl=en&amp;amp;lr=&amp;amp;id=xzVD8C2YpnQC&amp;amp;oi=fnd&amp;amp;pg=PR11&amp;amp;dq=Predictive%20Data-Mining.%20A%20Practical%20Guide.%20&amp;amp;ots=IN48sFFcSQ&amp;amp;sig=d6TOKBx_6qmitB8BSTk4_nLO9ak&quot; rel=&quot;nofollow&quot;&gt;ILP is contrasted with &quot;classical&quot; or &quot;propositional&quot; approaches to data mining&lt;/a&gt;. Those approaches include the meat and bones of Machine Learning like decision trees, neural networks, regression, bagging and other statistical methods. Rather than give up on these approaches due to the size of your data, you can join the ranks of many Data Scientists, Big Data engineers and statisticians who utilize High Performance Computing (HPC) to employ these methods on with massive data sets (there are also sampling and other statistical techniques you may choose to utilize to reduce the computational resources and time required to analyze the Big Data in your relational database). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HPC includes things like utilizing multiple CPU cores, scaling up your analysis with elastic use of servers with high memory and large numbers of fast CPU cores, using high-performance data warehouse appliances, employing clusters or other forms of parallel computing, etc. I'm not sure what language or statistical suite you're analyzing your data with, but as an example this &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;CRAN Task View&lt;/a&gt; lists many HPC resources for the R language which would allow you to scale up a propositional algorithm.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2015-02-25T17:08:09.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5233" PostTypeId="1" CreationDate="2015-02-25T18:28:27.903" Score="2" ViewCount="1718" Body="&lt;p&gt;Please I want to know if there is any SVM R package that can handle more than one response variable (y) at a time. that is to train one model for predicting more than one response variable. it could be regression or multi class classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help&lt;/p&gt;&#xA;" OwnerUserId="8425" LastActivityDate="2019-03-20T17:21:16.947" Title="training one SVM model for predicting more than one response variable" Tags="|machine-learning|r|algorithms|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5234" PostTypeId="2" ParentId="5192" CreationDate="2015-02-25T22:41:45.663" Score="3" Body="&lt;p&gt;I don't think that you will learn much about &lt;em&gt;data science&lt;/em&gt; (meaning, acquire understanding and skills) by using software tools like Tableau. Such tools are targeting mainly &lt;em&gt;advanced users&lt;/em&gt; (not &lt;em&gt;data scientists&lt;/em&gt;), for example analysts and other subject matter experts, who use &lt;em&gt;graphical user interface (GUI)&lt;/em&gt; to &lt;em&gt;analyze&lt;/em&gt; and (mostly) &lt;em&gt;visualize&lt;/em&gt; data. Having said that, software tools like Tableau might be good enough to perform initial phase of data science &lt;strong&gt;workflow&lt;/strong&gt;: &lt;em&gt;exploratory data analysis (EDA)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of data science &lt;strong&gt;self-education&lt;/strong&gt;, there are several popular online courses (MOOCs) that you can choose from (most come in both free and paid versions). In addition to the one on Udacity that you've mentioned (&lt;a href=&quot;https://www.udacity.com/course/ud359&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.udacity.com/course/ud359&lt;/a&gt;), there are two data science courses on Coursera: &lt;em&gt;Introduction to Data Science&lt;/em&gt; by University of Washington (&lt;a href=&quot;https://www.coursera.org/course/datasci&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.coursera.org/course/datasci&lt;/a&gt;) and a set of courses from &lt;em&gt;Data Science specialization&lt;/em&gt; by Johns Hopkins University (&lt;a href=&quot;https://www.coursera.org/specialization/jhudatascience/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.coursera.org/specialization/jhudatascience/1&lt;/a&gt;). Note that you can take specialization's individual courses for free at your convenience. There are several other, albeit less popular, data science MOOCs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of &lt;strong&gt;data sources&lt;/strong&gt;, I'm not sure what do you mean by &quot;Dummy Data&quot;, but there is a wealth of &lt;em&gt;open data sets&lt;/em&gt;, including many in the area of &lt;em&gt;public health&lt;/em&gt;. You can review corresponding resources, listed on KDnuggets (&lt;a href=&quot;http://www.kdnuggets.com/datasets/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/datasets/index.html&lt;/a&gt;) and choose ones that you're interested in. For a country-level analysis, the fastest way to obtain data is finding and visiting corresponding &lt;em&gt;open data&lt;/em&gt; government websites. For example, for public health data in US, I would go to &lt;a href=&quot;http://www.healthdata.gov&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.healthdata.gov&lt;/a&gt; and &lt;a href=&quot;http://www.data.gov&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.data.gov&lt;/a&gt; (the latter - for corresponding non-medical data that you might want to include in your analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regard to &lt;strong&gt;research papers&lt;/strong&gt; in the area of public health, I have two comments: 1) most &lt;em&gt;empirical research&lt;/em&gt; in that (or any other) area IMHO can be considered a data science study/project; 2) you need to perform a &lt;em&gt;literature review&lt;/em&gt; in the area or on the topic of your interest, so you're on your own in that sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a note on &lt;strong&gt;software tools&lt;/strong&gt;. If you're serious about data science, I would suggest to invest some time in learning either R, or Python (if you don't know them already), as those are two most popular open source tools among data scientists nowadays. Both have a variety of feature-rich &lt;em&gt;development environments&lt;/em&gt; as well as large &lt;em&gt;ecosystems&lt;/em&gt; of packages/libraries and users/developers all over the world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also find useful some of my other &lt;strong&gt;related answers&lt;/strong&gt; here on Data Science StackExchange site. For example, I recommend you to read &lt;a href=&quot;https://datascience.stackexchange.com/a/742/2452&quot;&gt;this answer&lt;/a&gt;, &lt;a href=&quot;https://datascience.stackexchange.com/a/843/2452&quot;&gt;this answer&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/759/2452&quot;&gt;this answer&lt;/a&gt;. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-02-25T22:41:45.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5235" PostTypeId="2" ParentId="5227" CreationDate="2015-02-26T05:08:31.993" Score="2" Body="&lt;p&gt;Do you know if the scheduler has a memory?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume for a moment that the scheduler has no memory.  This is a straightforward classification (supervised learning) problem: the inputs are X, the outputs are the schedules (N-&gt;M maps).  Actually, if &lt;em&gt;every&lt;/em&gt; N gets scheduled and the only question is which M it gets, the outputs are lists which channel (or none) is scheduled to each block, and there is only a certain possible number of those, so you can model them as discrete outputs (classes) with their own probabilities.  Use whatever you like (AdaBoost, Naive Bayes, RBF SVM, Random Forest...) as a classifier.  I think you will quickly learn about the general behavior of the scheduler.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the scheduler has a memory, then things get complicated.  I think you might approach that as a hidden Markov model: but the number of individual states may be quite large, and so it may be essentially impossible to build a complete map of transition probabilities.&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2015-02-26T05:08:31.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5236" PostTypeId="2" ParentId="5228" CreationDate="2015-02-26T11:19:35.900" Score="1" Body="&lt;p&gt;I am not sure there is a clear answer to this, especially as the problem does not seem to be well-defined right now - your &quot;figure&quot; seems to indicate edge weights but you then mention node weights, something significantly different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the question is whether you can find a way to split a graph into two smaller modules, then you might want to look into applying &lt;a href=&quot;http://en.wikipedia.org/wiki/Cut_%28graph_theory%29#Sparsest_cut&quot; rel=&quot;nofollow&quot;&gt;Sparsest Cut&lt;/a&gt; techniques - a cut with low cost would imply (?) high modularity. I believe these can be easily modified to account for either unlabeled, edge-labeled or node-labeled graphs.&lt;/p&gt;&#xA;" OwnerUserId="8433" LastActivityDate="2015-02-26T11:19:35.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5237" PostTypeId="2" ParentId="5204" CreationDate="2015-02-26T11:31:04.870" Score="4" Body="&lt;p&gt;This is very similar to the netflix problem, most matrix factorization methods can be adapted so that the error function is only evaluated at known points. For instance, you can take the gradient descent approach to SVD (minimizing the frobenius norm) but only evaluate the error and calculate the gradient at known points. I believe you can easily find code for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be exploiting the binary nature of your matrix and adapting binary matrix factorization tools in order to enforce binary factors (if you require them). I'm sure you can adapt one of the methods described &lt;a href=&quot;http://www.hongliangjie.com/2011/03/15/reviews-on-binary-matrix-decomposition/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; to work with unknown data using a similar trick as the one above.&lt;/p&gt;&#xA;" OwnerUserId="8433" LastActivityDate="2015-02-26T11:31:04.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5239" PostTypeId="2" ParentId="5231" CreationDate="2015-02-27T00:45:39.740" Score="0" Body="&lt;p&gt;You may want to consider using your own &lt;code&gt;APparameter&lt;/code&gt; object to put &quot;significance&quot; constraints on the rules learned by Apriori. See page 13 of the &lt;a href=&quot;http://cran.r-project.org/web/packages/arules/arules.pdf&quot; rel=&quot;nofollow&quot;&gt;arules documentation&lt;/a&gt;. This could reduce the number of uninteresting rules returned in your run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In lieu of gold standard data for your domain, consider bootstrap resampling as a form of validation, as described &lt;a href=&quot;http://eprints.pascal-network.org/archive/00003198/01/lal.pdf&quot; rel=&quot;nofollow&quot;&gt;in this article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8392" LastActivityDate="2015-02-27T00:45:39.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5241" PostTypeId="2" ParentId="2353" CreationDate="2015-02-27T05:28:25.457" Score="0" Body="&lt;p&gt;I think this line is wrong:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ytr = X(ii(1:N/2),:);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ytr should be the label of the training data. In this case, it should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ytr = y(ii(1:N/2),:);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8415" LastEditorUserId="84" LastEditDate="2015-02-28T13:38:20.850" LastActivityDate="2015-02-28T13:38:20.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5242" PostTypeId="2" ParentId="5233" CreationDate="2015-02-27T08:13:53.847" Score="0" Body="&lt;p&gt;Check out the e1071 package, here is the manual &lt;a href=&quot;http://cran.r-project.org/web/packages/e1071/e1071.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/e1071/e1071.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8310" LastActivityDate="2015-02-27T08:13:53.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5243" PostTypeId="1" AcceptedAnswerId="5286" CreationDate="2015-02-27T10:22:59.577" Score="0" ViewCount="284" Body="&lt;p&gt;I have been working in the last years with statistics and have gone pretty deep in programming with R. I have however always felt that I wasn't completely grasping what I was doing, still understanding all passages and procedures conceptually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to get a bit deeper into the math behind it all. I've been looking online for texts and tips, but all texts start with a very high level. Any suggestions on where to start?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more precise, I'm not looking for an exaustive list of statistical models and how they work, I kind of get those. I was looking for something like &quot;Basics of statistical modelling&quot;&lt;/p&gt;&#xA;" OwnerDisplayName="user8452" LastActivityDate="2015-03-06T03:46:41.693" Title="Interested in Mathematical Statistics... where to start from?" Tags="|statistics|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5244" PostTypeId="1" AcceptedAnswerId="5281" CreationDate="2015-02-27T10:42:35.407" Score="3" ViewCount="1207" Body="&lt;p&gt;Many of us are very familiar with using R in reproducible, but very much targeted, ad-hoc analysis. Given that R is currently the best collection of cutting-edge scientific methods from world-class experts in each particular field, and given that plenty of libraries exist for data io in R, it seems very natural to extend its applications into production environments for live decision making.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;did someone of you go into production with pure R (I know of shiny, yhat etc, but would be very interesting to hear of pure R);&lt;/li&gt;&#xA;&lt;li&gt;is there a good book/guide/article on the topic of building R into some serious live decision-making pipelines (such as e.g. credit scoring);&lt;/li&gt;&#xA;&lt;li&gt;I would like to hear also if you think it's not a good idea at all;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8310" LastActivityDate="2022-03-16T17:41:50.737" Title="R in production" Tags="|r|predictive-modeling|scoring|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5248" PostTypeId="2" ParentId="2575" CreationDate="2015-02-27T19:38:22.287" Score="3" Body="&lt;p&gt;In Hive itself? Unfortunately, the answer is simply no -- as the language definition manual shows, that statistic is simply not built in. In addition to the language manual, you can get more information on statistics in development in Hive &lt;a href=&quot;https://cwiki.apache.org/confluence/display/Hive/StatsDev&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://cwiki.apache.org/confluence/display/Hive/StatisticsAndDataMining&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, there are plenty of ways to calculate Kendall's W on data that's in Hive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could write out the data to a file or query it into R or a statistical package such as  SAS, Stat, MATLAB, Excel, etc then run your calculation and, if necessary, write your results back to Hive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, for instance, you could do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;RODBC&quot;)&#xA;require(RODBC)&#xA;db   &amp;lt;- odbcConnect(&quot;Hive_DB&quot;)&#xA;hql  &amp;lt;- &quot;select * from table A&quot;&#xA;data &amp;lt;- sqlQuery(db , hql)&#xA;kenw &amp;lt;- cor(x = data$a, y = data$b, method=&quot;kendall&quot;)&#xA;sqlSave(db, kenw, tablename = &quot;new_table_of_kendall_coef&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or (if using Linux or Unix) then you could use &lt;code&gt;RHive&lt;/code&gt; without needing to use an ODBC name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to go about it would be to take the functions that do exist in Hive (which you linked to) and calculate Kendall's coefficient yourself with a custom function. As to how to specifically implement that, well you'd probably want to post on Cross Validated (stats.stackexchange.com).&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2015-02-27T20:00:37.363" LastActivityDate="2015-02-27T20:00:37.363" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5249" PostTypeId="1" AcceptedAnswerId="5259" CreationDate="2015-02-28T00:44:04.353" Score="1" ViewCount="637" Body="&lt;p&gt;I start with a data.frame (or a data_frame) containing my dependent Y variable for analysis, my independent X variables, and some &quot;Z&quot; variables -- extra columns that I don't need for my modeling exercise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create an analysis data set without the Z variables;&lt;/li&gt;&#xA;&lt;li&gt;Break this data set into random training and test sets;&lt;/li&gt;&#xA;&lt;li&gt;Find my best model;&lt;/li&gt;&#xA;&lt;li&gt;Predict on both the training and test sets using this model;&lt;/li&gt;&#xA;&lt;li&gt;Recombine the training and test sets by rows; and finally&lt;/li&gt;&#xA;&lt;li&gt;Recombine these data with the Z variables, by column.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It's the last step, of course, that presents the problem -- how do I make sure that the rows in the recombined training and test sets match the rows in the original data set? We might try to use the row.names variable from the original set, but I agree with Hadley that this is an error-prone kludge (my words, not his) -- why have a special column that's treated differently from all other data columns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One alternative is to create an ID column that uniquely identifies each row, and then keep this column around when dividing into the train and test sets (but excluding it from all modeling formulas, of course). This seems clumsy as well, and would make all my formulas harder to read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This must be a solved problem -- could people tell me how they deal with this? Especially using the plyr/dplyr/tidyr package framework?&lt;/p&gt;&#xA;" OwnerUserId="3510" LastActivityDate="2015-03-01T07:16:51.440" Title="Combining data sets without using row.name" Tags="|machine-learning|r|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5251" PostTypeId="2" ParentId="5244" CreationDate="2015-02-28T17:10:41.147" Score="2" Body="&lt;p&gt;R and most of its CRAN modules are licensed using the GPL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many companies, legal departments go &lt;em&gt;crazy&lt;/em&gt; if you propose to use anything that is GPL in production... It's not reasonable, but you'll see they love Apache, and hate GPL. &lt;strong&gt;Before going into production, make sure it's okay with the legal department.&lt;/strong&gt; (IMHO you are safe to &lt;em&gt;use&lt;/em&gt; your modified code for &lt;em&gt;internal&lt;/em&gt; products. Integrating R into your commercial product and handing this out to others is very different. But unfortunately, many legal departments try to ban all use of GPL whatsoever.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that, R is often really slooow unless calling Fortran code hidden inside. It's nice when you are still trying to figure out what to do. But for production, you may want maximum performance, and full integration with your services. &lt;em&gt;Benchmark&lt;/em&gt; yourself, if R is the best choice for your use case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the performance issues with R (I know R advocates are going to downvote me for saying so ...):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Morandat, F., Hill, B., Osvald, L., &amp;amp; Vitek, J. (2012). Evaluating the design of the R language. In ECOOP 2012–Object-Oriented Programming (pp. 104-131). Springer Berlin Heidelberg.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(by the TraceR/ProfileR/ReactoR people from purdue, who are now working on fastR which tries to execute R code on the JVM?) states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;On those benchmarks, &lt;strong&gt;R is on average 501 slower than C and 43 times slower Python.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Observations. &lt;strong&gt;R is clearly slow and memory inefficient.&lt;/strong&gt; Much more so than other dynamic languages. This is largely due to the combination of language features (call-by-value, extreme dynamism, lazy evaluation) and the lack of efficient built-in types. We believe that with some effort it should be possible to improve both time and space usage, but this would likely &lt;strong&gt;require a full rewrite of the implementation&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sorry to break the news. It's now my research, but it aligns with my observations.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2015-02-28T19:56:59.157" LastActivityDate="2015-02-28T19:56:59.157" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="5252" PostTypeId="2" ParentId="1165" CreationDate="2015-02-28T17:15:39.423" Score="5" Body="&lt;p&gt;As a fellow CS Ph.D. defending my dissertation in a Big Data-esque topic this year (I started in 2012), the best piece of material I can give you is in a &lt;a href=&quot;http://www.rpajournal.com/dev/wp-content/uploads/2014/10/A3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This is an article written by two Ph.D.s from MIT who have talked about Big Data and MOOCs. Probably, you will find this a good starting point. BTW, along this note, if you really want to come up with a valid topic (that a committee and your adviser will let you propose, research and defend) you need to read LOTS and LOTS of papers. The majority of Ph.D. students make the fatal error of thinking that some 'idea' they have is new, when it's not and has already been done. You'll have to do something truly original to earn your Ph.D. Rather than actually focus on forming an idea right now, you should do a good literature survey and the ideas will 'suggest themselves'. Good luck! It's an exciting time for you.&lt;/p&gt;&#xA;" OwnerUserId="8395" LastEditorUserId="98307" LastEditDate="2020-08-17T20:32:32.897" LastActivityDate="2020-08-17T20:32:32.897" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5253" PostTypeId="2" ParentId="5120" CreationDate="2015-02-28T17:21:46.300" Score="1" Body="&lt;p&gt;I would actually try out regression. Also, don't make the mistake of using the serial number in your machine learning algorithms! The reason why I'm suggesting regression as opposed to 'better' machine learning algorithms is because you said you wanted to learn, and it's important to understand the algorithms (for the long run, and to truly be good at this stuff) that you're using. Regression is the easiest tool in the book that works quite well! Weka is so easy to use that you'll be able to plug and play different machine learning algorithms just for the sake of it. Another pointer that's won me several competitions is to do some feature selection before using regression/machine learning. For example, in your case, it is reasonable to assume that a student who scores high in Physics probably has a better chance of scoring high in Math as opposed to someone who scores high in English (but not necessarily Physics). If you have enough data, the algorithm itself will be able to deduce these positive/negative correlations and train the model accordingly. Sometimes, there isn't enough data, and you have to do some feature selection. Good luck! I'm a regular participant on Kaggle myself, and I think it's great that you're taking the 'hacker' route to learn more. It's the best way to get your hands dirty on real data and engineering problems. &lt;/p&gt;&#xA;" OwnerUserId="8395" LastActivityDate="2015-02-28T17:21:46.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5254" PostTypeId="2" ParentId="5196" CreationDate="2015-02-28T21:42:31.383" Score="0" Body="&lt;p&gt;If you know your approach is working, you can try to implement it more efficiently. Identify the crucial points, try to &lt;em&gt;vectorize&lt;/em&gt; them better, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R interpreter isn't the fastest. There are some efforts underway, but they are not yet ready. Vectorization (which means less interpreter, more low-level code) often yields a factor of 2x-5x, but you can sometimes get a factor of 100x by implementing it e.g. in C. (And the R advocates are going to hate me for saying this...)&#xA;Once you know that your approach is working, this may be worth the effort.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-02-28T21:42:31.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5255" PostTypeId="1" AcceptedAnswerId="5256" CreationDate="2015-02-28T22:10:58.473" Score="12" ViewCount="1364" Body="&lt;p&gt;Terms like 'data science' and 'data scientist' are increasingly used these days.&#xA;Many companies are hiring 'data scientist'. But I don't think it's a completely new job.&#xA;Data have existed from the past and someone had to deal with data. &#xA;I guess the term 'data scientist' becomes more popular because it sounds more fancy and 'sexy'&#xA;How were data scientists called in the past?&lt;/p&gt;&#xA;" OwnerUserId="8478" LastActivityDate="2016-01-29T01:33:57.403" Title="What is an 'old name' of data scientist?" Tags="|bigdata|" AnswerCount="7" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5256" PostTypeId="2" ParentId="5255" CreationDate="2015-02-28T23:21:40.430" Score="12" Body="&lt;p&gt;In reverse chronological order: data miner, statistician, (applied) mathematician.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-02-28T23:21:40.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5257" PostTypeId="1" CreationDate="2015-03-01T02:56:05.590" Score="1" ViewCount="100" Body="&lt;p&gt;I want to turn reviews of up to 5 stars and the number of reviews into upvotes. What's a good algorithm for doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A venue with 10 reviews total with a 5-star average rating should obviously get more upvotes than a venue with 10 reviews total with a 3-star average rating. Also, a venue with 60 ratings and a 4-star rating should probably get more upvotes than the one with 10 reviews and a 5-star rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need this rating to be based off of the total number of reviews and the average star rating, but I would also like the number to stay below a variable number (for example, say upvotes stay below 100, but I can also plug in 200 and it would stay below 200).&lt;/p&gt;&#xA;" OwnerUserId="8482" LastEditorUserId="13413" LastEditDate="2015-10-21T08:05:21.093" LastActivityDate="2015-11-20T09:15:55.650" Title="Good formula for turning star reviews into upvotes" Tags="|algorithms|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5259" PostTypeId="2" ParentId="5249" CreationDate="2015-03-01T07:16:51.440" Score="1" Body="&lt;p&gt;You need neither to use the row names or to create an additonal ID column. Here is an approach based on the indices of the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;dat &amp;lt;- data.frame(Y = rnorm(10),&#xA;                  X1 = rnorm(10),&#xA;                  X2 = rnorm(10),&#xA;                  Z1 = rnorm(10),&#xA;                  Z2 = rnorm(10))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, your steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Create an analysis data set without the &lt;code&gt;Z&lt;/code&gt; variables&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat2 &amp;lt;- dat[grep(&quot;Z&quot;, names(dat), invert = TRUE)]&#xA;dat2&#xA;#             Y          X1          X2&#xA;# 1  -0.6264538  1.51178117  0.91897737&#xA;# 2   0.1836433  0.38984324  0.78213630&#xA;# 3  -0.8356286 -0.62124058  0.07456498&#xA;# 4   1.5952808 -2.21469989 -1.98935170&#xA;# 5   0.3295078  1.12493092  0.61982575&#xA;# 6  -0.8204684 -0.04493361 -0.05612874&#xA;# 7   0.4874291 -0.01619026 -0.15579551&#xA;# 8   0.7383247  0.94383621 -1.47075238&#xA;# 9   0.5757814  0.82122120 -0.47815006&#xA;# 10 -0.3053884  0.59390132  0.41794156&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Break this data set into random training and test sets&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_idx &amp;lt;- sample(nrow(dat2), 0.8 * nrow(dat2))&#xA;train_idx&#xA;# [1]  7  4  3 10  9  2  1  5&#xA;&#xA;train &amp;lt;- dat2[train_idx, ]&#xA;train&#xA;#             Y          X1          X2&#xA;# 7   0.4874291 -0.01619026 -0.15579551&#xA;# 4   1.5952808 -2.21469989 -1.98935170&#xA;# 3  -0.8356286 -0.62124058  0.07456498&#xA;# 10 -0.3053884  0.59390132  0.41794156&#xA;# 9   0.5757814  0.82122120 -0.47815006&#xA;# 2   0.1836433  0.38984324  0.78213630&#xA;# 1  -0.6264538  1.51178117  0.91897737&#xA;# 5   0.3295078  1.12493092  0.61982575&#xA;&#xA;test_idx &amp;lt;- setdiff(seq(nrow(dat2)), train_idx)&#xA;test_idx&#xA;# [1] 6 8&#xA;&#xA;test &amp;lt;- dat2[test_idx, ]&#xA;test&#xA;#            Y          X1          X2&#xA;# 6 -0.8204684 -0.04493361 -0.05612874&#xA;# 8  0.7383247  0.94383621 -1.47075238&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find my best model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Predict on both the training and test sets using this model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recombine the training and test sets by rows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;idx &amp;lt;- order(c(train_idx, test_idx))&#xA;dat3 &amp;lt;- rbind(train, test)[idx, ]&#xA;identical(dat3, dat2)&#xA;# [1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recombine these data with the &lt;code&gt;Z&lt;/code&gt; variables, by column&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat4 &amp;lt;- cbind(dat3, dat[grep(&quot;Z&quot;, names(dat))])&#xA;identical(dat, dat4)&#xA;# [1] TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In summary, we can use the indices of the training and test data to combine the data in the rows in the original order.&lt;/p&gt;&#xA;" OwnerUserId="106" LastActivityDate="2015-03-01T07:16:51.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5261" PostTypeId="1" CreationDate="2015-03-01T14:28:22.123" Score="4" ViewCount="766" Body="&lt;p&gt;I'm struggling with for loop in R. I have a following data frame with sentences and two dictionaries with pos and neg words:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(stringr)&#xA;library(plyr)&#xA;library(dplyr)&#xA;library(stringi)&#xA;library(qdap)&#xA;library(qdapRegex)&#xA;library(reshape2)&#xA;library(zoo)&#xA;&#xA;# Create data.frame with sentences&#xA;sent &amp;lt;- data.frame(words = c(&quot;great just great right size and i love this notebook&quot;, &quot;benefits great laptop at the top&quot;,&#xA;                         &quot;wouldnt bad notebook and very good&quot;, &quot;very good quality&quot;, &quot;bad orgtop but great&quot;,&#xA;                         &quot;great improvement for that great improvement bad product but overall is not good&quot;, &quot;notebook is not good but i love batterytop&quot;), user = c(1,2,3,4,5,6,7),&#xA;               number = c(1,1,1,1,1,1,1), stringsAsFactors=F)&#xA;&#xA;# Create pos/negWords&#xA;posWords &amp;lt;- c(&quot;great&quot;,&quot;improvement&quot;,&quot;love&quot;,&quot;great improvement&quot;,&quot;very good&quot;,&quot;good&quot;,&quot;right&quot;,&quot;very&quot;,&quot;benefits&quot;,&#xA;          &quot;extra&quot;,&quot;benefit&quot;,&quot;top&quot;,&quot;extraordinarily&quot;,&quot;extraordinary&quot;,&quot;super&quot;,&quot;benefits super&quot;,&quot;good&quot;,&quot;benefits great&quot;,&#xA;          &quot;wouldnt bad&quot;)&#xA;negWords &amp;lt;- c(&quot;hate&quot;,&quot;bad&quot;,&quot;not good&quot;,&quot;horrible&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And now I'm gonna to create replication of origin data frame for big data simulation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Replicate original data.frame - big data simulation (700.000 rows of sentences)&#xA;df.expanded &amp;lt;- as.data.frame(replicate(100000,sent$words))&#xA;    sent &amp;lt;- coredata(sent)[rep(seq(nrow(sent)),100000),]&#xA;    sent$words &amp;lt;- paste(c(&quot;&quot;), sent$words, c(&quot;&quot;), collapse = NULL)&#xA;rownames(sent) &amp;lt;- NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For my further approach, I'll have to do descending ordering of words in dictionaries with their sentiment score (pos word = 1 and neg word = -1).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Ordering words in pos/negWords&#xA;wordsDF &amp;lt;- data.frame(words = posWords, value = 1,stringsAsFactors=F)&#xA;wordsDF &amp;lt;- rbind(wordsDF,data.frame(words = negWords, value = -1))&#xA;wordsDF$lengths &amp;lt;- unlist(lapply(wordsDF$words, nchar))&#xA;wordsDF &amp;lt;- wordsDF[order(-wordsDF[,3]),]&#xA;wordsDF$words &amp;lt;- paste(c(&quot;&quot;), wordsDF$words, c(&quot;&quot;), collapse = NULL)&#xA;rownames(wordsDF) &amp;lt;- NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I have a following function with for loop. 1) matching exact words 2) count them 3) compute score 4) remove matched words from sentence for another iteration:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scoreSentence_new &amp;lt;- function(sentence){&#xA;  score &amp;lt;- 0&#xA;  for(x in 1:nrow(wordsDF)){&#xA;    sd &amp;lt;- function(text) {stri_count(text, regex=wordsDF[x,1])} # count matched words&#xA;    results &amp;lt;- sapply(sentence, sd, USE.NAMES=F) # count matched words&#xA;    score &amp;lt;- (score + (results * wordsDF[x,2])) # compute score&#xA;    sentence &amp;lt;- str_replace_all(sentence, wordsDF[x,1], &quot; &quot;) # remove matched words from sentence for next iteration&#xA;  }&#xA;  score&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I call that function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SentimentScore_new &amp;lt;- scoreSentence_new(sent$words)&#xA;    sent_new &amp;lt;- cbind(sent, SentimentScore_new)&#xA;    sent_new$words &amp;lt;- str_trim(sent_new$words, side = &quot;both&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it resulted into desired output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                                                             words user     SentimentScore_new&#xA;                             great just great right size and i love this notebook    1                  4&#xA;                                                 benefits great laptop at the top    2                  2&#xA;                                               wouldnt bad notebook and very good    3                  2&#xA;                                                                very good quality    4                  1&#xA;                                                             bad orgtop but great    5                  0&#xA; great improvement for that great improvement bad product but overall is not good    6                  0&#xA;                                       notebook is not good but i love batterytop    7                  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In real I'm using dictionaries with pos/neg words about 7.000 words and I have 200.000 sentences. When I used my approach for 1.000 sentences it takes 45 mins. Please, could you anyone help me with some faster approach using of vectorization or parallel solution. Because of my beginner R programming skills I'm in the end of my efforts :-( Thank you very much in advance for any of your advice or solution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering about something like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 1:nrow(wordsDF)&#xA;score &amp;lt;- 0&#xA;&#xA;try_1 &amp;lt;- function(ttt) {&#xA;sd &amp;lt;- function(text) {stri_count(text, regex=wordsDF[ttt,1])}&#xA;results &amp;lt;- sapply(sent$words, sd, USE.NAMES=F)&#xA;    score &amp;lt;- (score + (results * wordsDF[ttt,2])) # compute score (count * sentValue)&#xA;    sent$words &amp;lt;- str_replace_all(sent$words, wordsDF[ttt,1], &quot; &quot;)&#xA;score&#xA;}&#xA;&#xA;a &amp;lt;- unlist(sapply(n, try_1))&#xA;apply(a,1,sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But doesn't work :-(&lt;/p&gt;&#xA;" OwnerUserId="8488" LastEditorUserId="8488" LastEditDate="2015-03-02T08:20:08.023" LastActivityDate="2015-04-06T22:19:59.417" Title="Vectorization of for loop in sentiment analysis" Tags="|r|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5263" PostTypeId="2" ParentId="851" CreationDate="2015-03-02T05:05:55.497" Score="3" Body="&lt;p&gt;Try using &lt;a href=&quot;http://ai.stanford.edu/%7Erubner/papers/rubnerIjcv00.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Earth Mover's Distance&lt;/a&gt;. It measures how much it &amp;quot;costs&amp;quot; to optimally reshape one histogram into another, where an elementary transform (using the terminology of edit distance-like distance measures) is moving a unit of mass from a bin to a bin. If you have an implementation of EMD, then your distance is dist = EMD(H1, H2, D), where&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; H1 = [ 14, 13, 40, 31 ]; -- histogram #1 (first image)&#xA; H2 = [ 82, 8, 7 ]; -- histogram #2 (second image)&#xA; D  = [ D11, D12, D13; ... ; D41, D42, D43 ]; -- (cross-bin) ground distance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Dij is a distance from the i'th bin of the first histogram to the j'th bin of the second histogram. For example, D21 is a distance between 'White' and 'Pink'. It may be defined differently, based on which color space you want to use, and further, based on how you define what it means for two colors to be similar. If I use RGB, and I do not care much as to how human eye perceives colors, then D21 can be the \ell_2 norm of the difference [1, 1, 1] - [1, 0.05, 0.7]. (As far as I remember, the paper I mentioned uses LAB instead of RGB.)&lt;/p&gt;&#xA;&lt;p&gt;P.S. Besides color, you may include spatial information into comparison. EMD can handle it well, as long as you properly combine similarity of colors with spatial similarity in the definition of ground distance D.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastEditorUserId="98307" LastEditDate="2020-08-17T20:32:41.403" LastActivityDate="2020-08-17T20:32:41.403" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5264" PostTypeId="1" AcceptedAnswerId="5269" CreationDate="2015-03-02T05:10:01.897" Score="1" ViewCount="890" Body="&lt;p&gt;Does reinforcement learning always need a grid world problem to be applied to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me any other example of how reinforcement learning can be applied to something which does not have a grid world scenario?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="52298" LastEditDate="2019-12-03T06:39:57.647" LastActivityDate="2019-12-03T06:39:57.647" Title="Does reinforcement learning only work on grid world?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="5265" PostTypeId="2" ParentId="5264" CreationDate="2015-03-02T10:06:27.483" Score="3" Body="&lt;p&gt;Reinforcement learning does not depend on a grid world.  It can be applied to any space of possibilities where there is a &quot;fitness function&quot; that maps between points in the space to a fitness metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topological spaces have a formally-defined &quot;neighborhoods&quot; but do not necessarily conform to a grid or any dimensional representation.  In a topological space, the only way to get from &quot;here&quot; to &quot;there&quot; is via some &quot;paths&quot; which are sets of contiguous neighborhoods. Continuous fitness functions can be defined over topological spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For what it is worth, reinforcement learning is not the be-all-end-all (family of) learning algorithms in fitness landscapes.  In a sufficiently rugged fitness landscape, other learning algorithms can perform better.  Also, if there are regions of the space where there are no well-defined fitness function at a given point in time, it may be indeterminate as to what learning algorithms are optimal, if any.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-03-02T10:06:27.483" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5267" PostTypeId="1" CreationDate="2015-03-02T21:15:34.480" Score="4" ViewCount="1277" Body="&lt;p&gt;I took on a project to predict the outcome of soccer matches but it turned out to be a very challenging task.&#xA;I tried out different models but I only got 50-54% accuracy on my test dataset. Some of the models were created in such&#xA;a way that a certain model would predict if a team will win, draw, or lose a match. That same model would also predict if&#xA;the opponent of that team will win, draw, or lose the match. Each model predicting with an accuracy of about 50% on each team distinctively. The second set of models I tried, takes the combination of data from both teams and predicts which class the match belongs to (home win, away win, draw). In the system,&#xA;only 10 matches are given everyday to be predicted. Meaning, if I predict the 10 matches using the second model, I have a chance of predicting 5 &#xA;correctly. In this project, I only need to predict 3 matches correctly out of the 10 matches given in a day. Is there a system&#xA;of knowing the 3 matches which my models have the best chance of predicting correctly? I only need to get 3 correct predictions, I usually&#xA;get 5 correctly but I don't know how to select my 3 best matches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The first type of models uses about 50 features for prediction while the second uses 101. I've tried ensembles, they still give &#xA;me ~50% accuracy. I'm still about to set up a system that selects matches where the prediction for the home team does not &#xA;contradict the prediction for the away team using the first type of models.&lt;/p&gt;&#xA;" OwnerUserId="4811" LastEditorUserId="69822" LastEditDate="2019-03-19T17:47:38.893" LastActivityDate="2019-03-19T17:47:38.893" Title="Predicting Soccer: guessing which matches a model will predict correctly" Tags="|machine-learning|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5268" PostTypeId="1" AcceptedAnswerId="5674" CreationDate="2015-03-02T23:02:45.583" Score="7" ViewCount="825" Body="&lt;p&gt;The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria.  Stock Screeners are then &quot;backtested&quot; by simulating the results of applying in over the last 10 years using Point-in-Time data. I get back the list of trades and overall graph of performance.&#xA;(If that is unclear, I have an overview &lt;a href=&quot;https://www.equitieslab.com/features/stock-screener/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.equitieslab.com/wiki/QuickStart/StockScreener&quot; rel=&quot;noreferrer&quot;&gt;there&lt;/a&gt; with more details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now a common problem is that users create overfitted stock screeners. I would love to give them a warning when the screen is likely to be over-fitted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fields I have to work with&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All trades made by the Stock Screener&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Stock, Start Date, Start Price, End Date, End Price&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;S&amp;amp;P 500 performance for the same time frame&lt;/li&gt;&#xA;&lt;li&gt;Market Cap, Sector, and Industry of each Stock&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8344" LastActivityDate="2015-05-03T00:30:18.570" Title="How to detect overfitting of a stock screener" Tags="|machine-learning|data-mining|classification|bigdata|statistics|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5269" PostTypeId="2" ParentId="5264" CreationDate="2015-03-03T01:35:31.727" Score="6" Body="&lt;p&gt;The short answer is no! Reinforcement Learning is not limited to discrete spaces. But most of the introductory literature does deal with discrete spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you might know by now that there are three important components in any Reinforcement Learning problem: Rewards, States and Actions. The first is a scalar quantity and theoretically the latter two can either be discrete or continuous. The convergence proofs and analyses of the various algorithms are easier to understand for the discrete case and also the corresponding algorithms are easier to code. That is one of the reasons, most introductory material focuses on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, it should be interesting to note that the early research on Reinforcement Learning actually focussed on continuous state representations. It was only in the the 90s since the literature started representing all the standard algorithms for discrete spaces as we had a lot of proofs for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you noticed carefully, I said continuous states only. Mapping continuous states and continuous actions is hard. Nevertheless, we do have some solutions for now. But it is an active area of Research in RL. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://webdocs.cs.ualberta.ca/~sutton/papers/SSR-98.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper by Sutton&lt;/a&gt; from '98 should be a good start for your exploration!&lt;/p&gt;&#xA;" OwnerUserId="8491" LastActivityDate="2015-03-03T01:35:31.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5270" PostTypeId="2" ParentId="4980" CreationDate="2015-03-03T05:59:08.820" Score="4" Body="&lt;p&gt;Training a model, related to &lt;em&gt;information extraction&lt;/em&gt;, in general, and &lt;em&gt;named entity recognition/resolution (NER)&lt;/em&gt;, in particular, is described in detail in &lt;strong&gt;Chapter 7&lt;/strong&gt; of the &lt;a href=&quot;http://www.nltk.org/book&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK Book&lt;/a&gt;, available online at this URL: &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nltk.org/book/ch07.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, I think that you might find useful my &lt;a href=&quot;https://stats.stackexchange.com/a/136760/31372&quot;&gt;related answer&lt;/a&gt; on &lt;em&gt;Cross Validated&lt;/em&gt; site. It has a lot of references to relevant &lt;em&gt;sources&lt;/em&gt; on NER and related topics as well as to various related &lt;em&gt;software&lt;/em&gt; tools.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-03T05:59:08.820" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5273" PostTypeId="1" CreationDate="2015-03-03T16:12:13.860" Score="6" ViewCount="253" Body="&lt;p&gt;The training set contains &lt;span class=&quot;math-container&quot;&gt;$p$&lt;/span&gt; papers. Each paper is annotated as &lt;em&gt;research&lt;/em&gt; or &lt;em&gt;non-research&lt;/em&gt;. To develop the research paper filter, we consider the &lt;span class=&quot;math-container&quot;&gt;$W$&lt;/span&gt; most frequent phrases in a paper. The research paper filter will use the presence/absence of these &lt;span class=&quot;math-container&quot;&gt;$W$&lt;/span&gt; phrases to decide if the paper is indeed a research paper or not.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How many possible hypothesis are there?&lt;/p&gt;&#xA;&lt;p&gt;I have &lt;span class=&quot;math-container&quot;&gt;$2^W$&lt;/span&gt; possible hypothesis, but if we are including conjunctive hypothesis symbols, then I would say &lt;span class=&quot;math-container&quot;&gt;$4^W$&lt;/span&gt;. Is my logic correct?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How many of them are consistent?&lt;/p&gt;&#xA;&lt;p&gt;For this question, I have no idea where to begin. Any insight?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8505" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-12-22T08:29:16.733" Title="How many possible hypotheses" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5275" PostTypeId="2" ParentId="5255" CreationDate="2015-03-04T04:39:35.380" Score="4" Body="&lt;p&gt;I do think it is new job, basically data scientist has to apply mathematical algorithms on data with considerable constraint in terms 1) Run time of the application 2) Resource use of the application. If these constraints are not present, I would not call the job data science. Moreover, these algorithms are often need to be ran on distributed systems, which is another dimension of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this has been done before, in some combination of statistics, mathematics and programming, but it was not wide spread to give rise to the new term. The real rise of data science is from the ability to gather large amounts of data, thus need to need to process it. &lt;/p&gt;&#xA;" OwnerUserId="3070" LastActivityDate="2015-03-04T04:39:35.380" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5276" PostTypeId="2" ParentId="5255" CreationDate="2015-03-04T06:05:05.320" Score="5" Body="&lt;p&gt;Terms that covered more or less the same topics that Data Science covers today:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pattern Recognition&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning &lt;/li&gt;&#xA;&lt;li&gt;Data Mining &lt;/li&gt;&#xA;&lt;li&gt;Quantitative methods&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2015-03-04T06:05:05.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5277" PostTypeId="1" AcceptedAnswerId="5278" CreationDate="2015-03-04T08:05:45.003" Score="16" ViewCount="30299" Body="&lt;p&gt;So, our data set this week has 14 attributes and each column has very different values. One column has values below 1 while another column has values that go from three to four whole digits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We learned normalization last week and it seems like you're supposed to normalize data when they have very different values. For decision trees, is the case the same?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure about this but would normalization affect the resulting decision tree from the same data set? It doesn't seem like it should but...&lt;/p&gt;&#xA;" OwnerUserId="8318" LastEditorUserId="8318" LastEditDate="2015-03-04T08:15:57.703" LastActivityDate="2015-03-04T12:15:10.117" Title="Do you have to normalize data when building decision trees using R?" Tags="|r|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5278" PostTypeId="2" ParentId="5277" CreationDate="2015-03-04T12:15:10.117" Score="24" Body="&lt;p&gt;Most common types of decision trees you encounter are not affected by any monotonic transformation. So, as long as you preserve orde, the decision trees are the same (obviously by the same tree here I understand the same decision structure, not the same values for each test in each node of the tree). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why it happens is because how usual impurity functions works. In order to find the best split it searches on each dimension (attribute) a split point which is basically an if clause which groups target values corresponding to instances which has test value less than split value, and on the right the values greater than equal. This happens for numerical attributes (which I think is your case because I do not know how to normalize a nominal attribute). Now you might note that the criteria is less than or greater than. Which means that the real information from the attributes in order to find the split (and the whole tree) is only the order of the values. Which means that as long as you transform your attributes in such a way that the original ordering is reserved, you will get the same tree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not all models are insensitive to such kind of transformation. For example linear regression models give the same results if you multiply an attribute with something different than zero. You will get different regression coefficients, but the predicted value will be the same. This is not the case when you take a log of that transformation. So for linear regression, for example, normalizing is useless since it will provide the same result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this is not the case with a penalized linear regression, like ridge regression. In penalized linear regressions a constraint is applied to coefficients. The idea is that the constraint is applied to the sum  of a function of coefficients. Now if you inflate an attribute, the coefficient will be deflated, which means that in the end the penalization for that coefficient it will be artificially modified. In such kind of situation, you normalize attributes in order that each coefficient to be constraint 'fairly'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-03-04T12:15:10.117" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5279" PostTypeId="1" CreationDate="2015-03-04T14:19:00.713" Score="4" ViewCount="1074" Body="&lt;p&gt;I was given a target function to design neural network and train: (y = (x1 ∧ x2) ∨ (x3 ∧ x4))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of input and number of output seems obvious (4 and 1). And the training data can use truth table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in order to train as a multilayer artificial neural network, I need to choose number of hidden units. May I know where can I find some general guideline for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="8524" LastActivityDate="2015-04-04T03:28:34.233" Title="How to select topology for neural network?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5280" PostTypeId="1" CreationDate="2015-03-04T14:34:51.013" Score="2" ViewCount="126" Body="&lt;p&gt;I want to generate an $N\times N$ matrix $A$ so as to target an $N$ vector of row sums and simultaneously all column sums should sum to 1. In addition to this, I have a prefixed number of elements which are set to zero. &#xA;For example, beginning with:&#xA;$$&#xA;  \left[\begin{array}{rrr}&#xA;    0 &amp;amp; 1 &amp;amp; 1 \\&#xA;    1 &amp;amp; 0 &amp;amp; 0 \\&#xA;    1 &amp;amp; 1 &amp;amp; 0&#xA;  \end{array}\right]&#xA;$$&#xA;and the row sum vector $[1.5, 0.25, 1]^{T}$, I want to end up with&#xA;$$&#xA;  \left[\begin{array}{rrr}&#xA;    0 &amp;amp; a_{12} &amp;amp; a_{13} \\&#xA;    a_{21} &amp;amp; 0 &amp;amp; 0 \\&#xA;    a_{31} &amp;amp; a_{32} &amp;amp; 0&#xA;  \end{array}\right]&#xA;$$&#xA;under the following conditions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_{12} + a_{13}  = 1.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ a_{21} = 0.25$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_{31}+a_{32} = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_{21}+a_{31} = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_{12}+a_{32} = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_{13} = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this is simplistic, in general, I have $2N$ equations in $N^{2}-Z$ unknowns, where $Z$ is the number of elements fixed to zero. So, this system of equations could be overdetermined or underdetermined, but I would like to be able to generate matrices like this such that all nonzero elements lie in $(0,1]$.&lt;/p&gt;&#xA;" OwnerUserId="8526" LastEditorUserId="8462" LastEditDate="2015-03-06T09:35:47.037" LastActivityDate="2015-05-05T10:37:22.430" Title="Computation of a column-stochastic matrix with target row sums" Tags="|statistics|algorithms|graphs|social-network-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5281" PostTypeId="2" ParentId="5244" CreationDate="2015-03-04T15:53:46.530" Score="1" Body="&lt;p&gt;Speed of code execution is rarely an issue.  The important speed in business is almost always the speed of designing, deploying, and maintaining the application.  An experienced programmer can optimize where necessary to get code execution fast enough.  In these cases, R can make a lot of sense in production. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In cases where speed of execution IS an issue, you are already going to find an optimized C++ or some such real-time decision engine.  So your choices are integrate an R process, or add the bits you need to the engine.  The latter is probably the only option, not because of the speed of R, but because you don't have the time to incorporate any external process.  If the company has nothing to start with, I can't imagine everyone saying &quot;let's build our time critical real-time engine in R because of the great statistical libraries&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll give a few examples from my corporate experiences, where I use R in production:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Delivering Shiny applications dealing with data that is not/ not yet institutionalized.  I will generally load already-processed data frames and use Shiny to display different graphs and charts.  Computation is minimal.   &lt;/li&gt;&#xA;&lt;li&gt;Decision making analysis that requires heavy use of advanced libraries (mcclust, machine learning) but done on a daily or longer time-scale.  In this case there is no reason to use any other language.  I've already done the prototyping in R, so my fastest and best option is to keep things there.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I did not use R for production when integrating with a real-time C++ decision engine.  Issues:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;An additional layer of complication to spawn R processes and integrate the results&lt;/li&gt;&#xA;&lt;li&gt;A suitable machine-learning library (Waffles) was available in C++&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The caveat in the latter case: I still use R to generate the training files.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2015-03-04T15:53:46.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5282" PostTypeId="2" ParentId="5279" CreationDate="2015-03-04T21:47:35.453" Score="1" Body="&lt;p&gt;You problem is linearly separable so you can use a single layer perceptron. &#xA;Hidden units are necessary only for non linear problems (xor is a classic example). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for general guidelines, I don't think anyone has ever needed more than one hidden layer. Also the number of neurons in the hidden layer doesn't need to exceed the number of inputs. &lt;/p&gt;&#xA;" OwnerUserId="8517" LastEditorUserId="8517" LastEditDate="2015-03-05T03:17:42.347" LastActivityDate="2015-03-05T03:17:42.347" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="5284" PostTypeId="1" CreationDate="2015-03-05T18:28:43.367" Score="2" ViewCount="1021" Body="&lt;p&gt;I'm a Newbie to Hadoop!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a new instance temporarily and append the new data to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.would like to know whether the &lt;em&gt;new file&lt;/em&gt; or &lt;em&gt;temp file&lt;/em&gt; is created in the &lt;em&gt;same block&lt;/em&gt; or in a &lt;em&gt;different block&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.What will happen if the revised file &lt;strong&gt;exceeds&lt;/strong&gt; the previous  allocated block size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="8540" LastEditorUserId="8540" LastEditDate="2015-03-06T16:51:02.073" LastActivityDate="2015-03-06T16:51:02.073" Title="How append works in hdfs? Where the newly created instance of file is placed?" Tags="|data-mining|bigdata|apache-hadoop|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5285" PostTypeId="2" ParentId="2543" CreationDate="2015-03-06T02:43:38.727" Score="2" Body="&lt;p&gt;Here are two ideas that I was thinking about.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bug Prediction&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Based on the previous activity future bugs can be predicted. There are a number of papers available on the net. I remember there was a paper about bug prediction from SVN data. ( Just google for software bug prediction )&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Error position from software traces&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Errors may be like rare events or anomalies in the trace output. May be you can classify the error and pinpoint the procedure that caused the error. I am currently thinking about a system like that now ( Not sure about its success though ). I asked a question here: &lt;a href=&quot;https://stats.stackexchange.com/questions/140232/error-position-in-software-trace-file&quot;&gt;https://stats.stackexchange.com/questions/140232/error-position-in-software-trace-file&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8516" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-06T02:43:38.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5286" PostTypeId="2" ParentId="5243" CreationDate="2015-03-06T03:46:41.693" Score="1" Body="&lt;p&gt;When looking for texts to learn advanced topics, I start with a web search for relevant grad courses and textbooks, or background tech/math books like those from Dover.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wit, Theoretical Statistics by Keener looks relevant:&#xA;&lt;a href=&quot;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-93838-7&quot; rel=&quot;nofollow&quot;&gt;http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-93838-7&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this:&#xA;&quot;Looking for a good Mathematical Statistics self-study book (I'm a physics student and my class &amp;amp; current book are useless to me)&quot;&#xA;&lt;a href=&quot;http://www.reddit.com/r/statistics/comments/1n6o19/looking_for_a_good_mathematical_statistics/&quot; rel=&quot;nofollow&quot;&gt;http://www.reddit.com/r/statistics/comments/1n6o19/looking_for_a_good_mathematical_statistics/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3556" LastActivityDate="2015-03-06T03:46:41.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5287" PostTypeId="1" CreationDate="2015-03-06T04:09:56.030" Score="4" ViewCount="99" Body="&lt;p&gt;Any suggestion on what kind of dataset lets say $n \times d$ ($n$ rows, $d$ columns) would give me same eigenvectors?.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe it should be the one with same absolute values in each cell. Like alternating +1 and -1. But it seems to work otherwise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers?&lt;/p&gt;&#xA;" OwnerUserId="8547" LastEditorUserId="84" LastEditDate="2015-03-06T09:38:33.380" LastActivityDate="2015-03-10T09:49:55.490" Title="Dataset to give same eigenvectors?" Tags="|machine-learning|data-mining|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5288" PostTypeId="2" ParentId="5280" CreationDate="2015-03-06T05:45:03.720" Score="1" Body="&lt;p&gt;This problem, being more of a combinatorial rather than statistical nature, can be though of as a &lt;em&gt;max-flow problem in a bipartite network&lt;/em&gt;. Each column of your matrix corresponds to a &quot;source node&quot; in the left part of the bipartite graph $BP(L, R)$; each row of the matrix corresponds to a &quot;destination node&quot; in the right part $R$. Capacity of each source node is the corresponding column's sum; similarly, capacities are defined for the destinations via row sums. Value $a_{ij}$ in the matrix describes flow of mass from $j$'th source to $i$'th destination along an existing edge $(j \to i)$ of $BP$. Your initially given 0-1 matrix describes which edges are present (e.g., the matrix in your example prescribes the edge $(3, 2)$ to be absent from the network, along with all self-loops). If you denote $F$ to be the sum of all column sums (or, alternatively, the total capacity of all sources), then your problem narrows down to finding a max-flow from $L$ to $R$ in $BP$ and checking whether the volume of the discovered flow equals $F$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note 1 (Feasibility):&lt;/em&gt; Before plunging into finding best flows, you may want to check your row sums: a necessary condition for your problem to have a solution is that the sum of all column sums (which is $N$ in your particular case of a column-stochastic matrix) should be equal the sum of row sums, as both numbers describe the same -- the sum of all elements of the target matrix. (For example, for $N = 3$ and the vector of row sums from your example, the problem is unfeasible.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note 2 (Upper bound):&lt;/em&gt; Since $a_{ij}$ describes the flow from $j$'th source, and the capacities of all sources are 1 in your setting, then each source cannot send more than 1 unit of mass to a single destination. Thus, each $a_{ij}$ will not exceed 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note 3 (Lower bound):&lt;/em&gt; To make $a_{ij}$ for each existing edge $(j \to i)$ strictly positive, you can augment the max-flow problem with lower bounds on edge capacities. In a general setting, edge capacity bounds are $0$ and $+\infty$. You may want to replace the lower bound with some positive number, small enough not to interfere with the discovery of the best flow and large enough to satisfy your edge saturation needs.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastActivityDate="2015-03-06T05:45:03.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5289" PostTypeId="2" ParentId="5287" CreationDate="2015-03-06T06:11:10.997" Score="2" Body="&lt;p&gt;The only such existing &quot;dataset&quot; is a 1-by-1 matrix with an arbitrary value of its sole item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the eigenvectors, being the columns of matrix $V$, are &quot;the same&quot; (precisely up to a scalar constant factor), then matrix $V$ is singular (its columns are linearly dependent; its determinant is zero), and, hence, non-invertible $\Rightarrow$ such an eigendecomposition $V \cdot \Lambda \cdot V^{-1}$ cannot exist.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastEditorUserId="8462" LastEditDate="2015-03-06T09:45:35.087" LastActivityDate="2015-03-06T09:45:35.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5290" PostTypeId="2" ParentId="5287" CreationDate="2015-03-06T07:56:19.903" Score="4" Body="&lt;p&gt;I will restrict myself to the finite-dimensional situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, if we are talking about &lt;em&gt;normal&lt;/em&gt; eigenvalues and not generalized ones, we need a square matrix. If $M\in\mathbb{R}^{n\times n}$ (or $\mathbb{C}^{n\times n}$) is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Diagonalizable_matrix&quot; rel=&quot;nofollow&quot;&gt;diagonizable matrix&lt;/a&gt;, it has an &lt;a href=&quot;http://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix#Eigendecomposition_of_a_matrix&quot; rel=&quot;nofollow&quot;&gt;eigendecomposition&lt;/a&gt;&#xA;\begin{equation}&#xA;M = V\Lambda V^{-1}&#xA;\textrm{,}&#xA;\end{equation}&#xA;with $V:=\begin{bmatrix}v_{1},v_{2},\ldots,v_{n}\end{bmatrix}$ the matrix with its $n$ eigenvectors $v_{i}$ as columns and $\Lambda :=\operatorname{diag}\left(\lambda_{1},\lambda_{2},\ldots,\lambda_{n}\right)$ the diagonal matrix with the corresponding eigenvalues on its main diagonal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see from this equation if you want to get matrices with the same eigenvectors, i.e. $V$ fixed, you can only change the eigenvalues $\lambda_{i}$. In general the relationship between the eigenvalues and the entries of the original matrix $M$ is non-trivial. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest situation arises if the eigenvectors $v_{i}$ form the standard basis, i.e. only $1$ at the $i$th position, $0$ otherwise. Then $M=I\Lambda I^{-1}=\Lambda $ ($I$ the identity matrix) and you can change every entry on the main diagonal without changing any eigenvector. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Edit&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I understood the question differently than victor. I thought the question asks what kind of different datasets can be described by the same eigenvectors. &lt;/p&gt;&#xA;" OwnerUserId="8249" LastEditorUserId="8249" LastEditDate="2015-03-10T09:49:55.490" LastActivityDate="2015-03-10T09:49:55.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5291" PostTypeId="1" AcceptedAnswerId="5763" CreationDate="2015-03-06T10:11:02.133" Score="1" ViewCount="185" Body="&lt;p&gt;I am using twitteR package to retrievie timeline data. My request looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tweets &amp;lt;- try(userTimeline(user , n=50),silent=TRUE)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this worked quite well for a time, but now I receive this error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in function (type, msg, asError = TRUE)  : easy handle already used in multi handle&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a related question on Stackoverflow one answer is to use Rcurl directly but this does not seem to work with twitteR package. Anybody got an idea on this?&lt;/p&gt;&#xA;" OwnerUserId="8549" LastActivityDate="2015-05-12T09:30:09.273" Title="Error using twitter R package's userTimlien" Tags="|data-mining|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5292" PostTypeId="1" CreationDate="2015-03-06T10:28:04.483" Score="2" ViewCount="226" Body="&lt;p&gt;I have a huge collection of objects from which only a tiny fraction are in a class of interest. The collection is initially unlabelled, but labels can be added using an expensive operation (for example, by human).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I use the simple generic machine learning strategy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use hand-crafted rules to select a smaller subset of objects (thus leaving out a fraction of interesting ones).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Label part of the smaller subset, and use these for training and choosing a classification algorithm and its parameters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Classify the remaining objects in the smaller set (and also perhaps in the big set).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This has two drawbacks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The labeller still needs to see a huge number of uninteresting objects, and therefore is able to label only a very small fraction of interesting ones. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The objects not in the smaller set are completely ignored in the learning phase, resulting in a loss of some information (the classification algorithm might not work well on this complement).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It seems that it would be better to use online learning: i.e., select the objects to show to the labeller based on the previous labels. But then it becomes no longer obvious that the result of classification algorithm retains the nice theoretical properties (i.e., statistical consistency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general framework for active object detection which works either theoretically or practically (or both)? I could not get the complete picture from the Wikipedia article &lt;a href=&quot;http://en.wikipedia.org/wiki/Active_learning_%28machine_learning%29&quot; rel=&quot;nofollow&quot;&gt;active learning&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="13727" LastEditDate="2017-02-28T12:38:05.317" LastActivityDate="2017-03-30T14:06:53.130" Title="Generic strategy for object detection" Tags="|machine-learning|classification|object-recognition|semi-supervised-learning|active-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5294" PostTypeId="1" AcceptedAnswerId="5298" CreationDate="2015-03-09T16:53:10.930" Score="4" ViewCount="898" Body="&lt;p&gt;This is an interview question&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How are neural nets related to Fourier transforms?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I could find papers that talk about methods to process the Discrete Fourier Transform (DFT) by a single-layer neural network with a linear transfer function. Is there some other correlation that I'm missing?&lt;/p&gt;&#xA;" OwnerUserId="7960" LastActivityDate="2015-03-09T21:56:00.827" Title="How are neural nets related to Fourier transforms?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5295" PostTypeId="2" ParentId="778" CreationDate="2015-03-09T18:21:32.663" Score="8" Body="&lt;p&gt;Python has some very good tools for working with big data:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;numpy&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Numpy's memmory-mapped arrays let you access a file saved on disk as though it were an array.  Only the parts of the array you are actively working with need to be loaded into memory.  It can be used pretty much the same as an ordinary array.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;h5py and pytables&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;These two libraries provide access to HDF5 files.  These files allow access to just part of the data. Further, thanks to the underlying libraries used to access the data, many mathematical operations and other manipulations of the data can be done without loading it into a python data structure.  Massive, highly structured files are possible, much bigger than 5 TB.  It also allows seamless, lossless compression.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;databases&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There are various types of databases that allow you to store big data sets and load just the parts you need.  Many databases allow you to do manipulations without loading the data into a python data structure at all.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;pandas&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This allows higher-level access to various types of data, including HDF5 data, csv files, databases, even websites.  For big data, it provides wrappers around HDF5 file access that makes it easier to do analysis on big data sets.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;mpi4py&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is a tool for running your python code in a distributed way across multiple processors or even multiple computers.  This allows you to work on parts of your data simultaneously.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;dask&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It provides a version of the normal numpy array that supports many of the normal numpy operations in a multi-core manner that can work on data too large to fit into memory. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;blaze&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A tool specifically designed for big data.  It is basically a wrapper around the above libraries, providing consistent interfaces to a variety of different methods of storing large amounts of data (such as HDF5 or databases) and tools to make it easy to manipulate, do mathematical operations on, and analyze data that is too big to fit into memory.&lt;/p&gt;&#xA;" OwnerUserId="8110" LastActivityDate="2015-03-09T18:21:32.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5298" PostTypeId="2" ParentId="5294" CreationDate="2015-03-09T21:54:19.297" Score="4" Body="&lt;p&gt;They are not related in any meaningful sense. Sure, you can use them both to extract features, or do any number of things, but the same can be said about a many techniques. I would have asked &quot;what kind of neural network?&quot; to see if the interviewer had something specific in mind.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-03-09T21:54:19.297" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5299" PostTypeId="2" ParentId="5294" CreationDate="2015-03-09T21:56:00.827" Score="3" Body="&lt;p&gt;The similarity is regression.&#xA;NNs can be used for regression and the fourier transform is in its heart just a curve fit of multiple sin and cos functions to some data.&lt;/p&gt;&#xA;" OwnerUserId="5316" LastActivityDate="2015-03-09T21:56:00.827" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5300" PostTypeId="2" ParentId="5255" CreationDate="2015-03-10T09:29:06.893" Score="2" Body="&lt;p&gt;Also: &quot;Business Intelligence developer&quot;&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2015-03-10T09:29:06.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5301" PostTypeId="1" AcceptedAnswerId="5353" CreationDate="2015-03-10T10:20:22.510" Score="4" ViewCount="505" Body="&lt;p&gt;I need to implement an algorithm for multiple extended string matching in text. Algorithms to match regular expression would be perhaps too slow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extended&lt;/strong&gt; means the presence of wildcards (any number of characters instead of a star), for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abc*def //matches abcdef, abcpppppdef etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multiple&lt;/strong&gt; means that the search is going on simultaneously for multiple string patterns (not a separate search for each pattern), for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abc*def&#xA;abc&#xA;whatever&#xA;some*string&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUESTION:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the fast algorithm that can do multiple extended string matching?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preferably, optimized for SIMD instructions and multicore implementation. Open source implementation (C/C++/Python) would be great as well. I'm interested in 10 Gbps+ performance on a single core of a modern CPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="8597" LastEditorUserId="-1" LastEditDate="2015-03-10T19:08:52.490" LastActivityDate="2015-03-19T07:38:57.917" Title="Algorithm for multiple extended string matching" Tags="|algorithms|search|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5302" PostTypeId="1" CreationDate="2015-03-10T18:03:32.227" Score="1" ViewCount="2057" Body="&lt;p&gt;I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a &quot;long&quot; .txt file as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'cat1/: Topic1_Variable1'&#xA;'cat2/: Topic1_Variable2'&#xA;'cat3/: Topic1_Variable3'&#xA;'cat4/: Topic1_Variable4'&#xA;&#xA;'cat1/: Topic2_Variable1'&#xA;'cat2/: Topic2_Variable2'&#xA;'cat3/: Topic2_Variable3'&#xA;'cat4/: Topic2_Variable4'&#xA;&#xA;'cat1/: Topic3_Variable1'&#xA;'cat2/: Topic3_Variable2'&#xA;'cat3/: Topic3_Variable3'&#xA;'cat4/: Topic3_Variable4'&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fro analysis and sharing with others, I really would like to see it formatted as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat1              cat2              cat3              cat4&#xA;Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4&#xA;Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4&#xA;Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think that this may be easier in R, but I honestly am drawing a complete blank in SAS. I've even played with MS Access to try to get it to look the way I want but the program crashes every time (due to the size?). At any rate, I have looked into some of the statements in PROC TRANSPOSE and PROC SQL but it seems that most functions within those procedures are utilized to combine duplicate 'Topics'. In the data I have been provided, each &quot;group&quot; represents an individual response to a question with several thousand individuals repeated, I want to retain the independence of each occurrence and not perform a UNION as defined in PROC SQL. At this point, I feel like I am over-thinking this but I just can't get around the mental block and actually do what I am working toward. &#xA;Any help or guidance is much appreciated. I'm open to trying all suggestions or ideas, I think I have access to most statistical computing programs. &lt;/p&gt;&#xA;" OwnerUserId="8605" LastActivityDate="2015-12-08T04:58:43.387" Title="Transposing Every nth row to column in a large dataset" Tags="|bigdata|dataset|data-formats|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5303" PostTypeId="1" CreationDate="2015-03-10T19:51:59.277" Score="5" ViewCount="1000" Body="&lt;p&gt;I am trying my first 'project' concerning machine learning and I am a bit stuck.&#xA;However, I am not sure if it's even possible but here goes my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to achieve is clustering user groups based on the amount of visits a user does on a certain website.&#xA;So I started out with this feature matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;USER    abc.be  abc.be/a    abc.be/b    xyz.be  xyz.be/a&#xA;123      0        0           0            0      1&#xA;456      1        0           1            0      0&#xA;789      2        3           1            0      0&#xA;321      1        0           1            0      1&#xA;654      1        1           1            1      1&#xA;987      0        1           0            3      0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I got in this example 5 features (my 5 different websites).&#xA;So then I used PCA to come to 2 dimensions, so I could plot it and see how it went.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My feature matrix (in my example) is 5 columns * 6 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My PCA matrix is 2 columns * 6 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came to this plot (please note that this plot uses different data then the example but the idea is the same)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Tl0Qv.png&quot; alt=&quot;PCA points and k-means centroids&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The green points are my PCA points&#xA;The red circles are my K-Means centroids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the part I am struggling with is this: so I got my clusters (red circles) but how can I use this to say:&quot;Looks like most users go to either site A or site B)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how can I couple my clusters to a feature label from my feature matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or how does one approach this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated :)&lt;/p&gt;&#xA;" OwnerUserId="8389" LastActivityDate="2015-08-15T19:12:31.760" Title="Couple PCA plot and clusters to labels" Tags="|machine-learning|feature-extraction|k-means|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5304" PostTypeId="1" CreationDate="2015-03-11T04:18:21.610" Score="2" ViewCount="247" Body="&lt;p&gt;Problem: I want to know methods to perform an effective sampling from a database.&#xA;The size of the database is about &lt;code&gt;250K&lt;/code&gt; text documents and in this case each text document is related to some majors (Electrical Engineering, Medicine and so on). So far I have seen some simple techniques like &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_random_sample&quot; rel=&quot;nofollow&quot;&gt;Simple random sample&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;Stratified sampling&lt;/a&gt;; however, I don't think it's a good idea to apply them for the following reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In the case of simple random sample, for instance, there are a few documents in the database that talk about majors like Naval Engineering or Arts. Therefore I think they are less likely to be sampled with this method but I would like to have some samples of every major as possible.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the case of stratified sampling, most of the documents talk about more than one major so I cannot divide the database in subgroups because they will not be mutually exclusive, at least for the case in which each major is a subgroup.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, I cannot use the whole database due to expensive computational cost processing. So I would really appreciate any suggestions on other sampling methods. Thanks for any help in advance.&lt;/p&gt;&#xA;" OwnerUserId="8609" LastActivityDate="2015-03-13T15:41:35.577" Title="How to make an effective sampling from a database of text documents?" Tags="|dataset|databases|sampling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5305" PostTypeId="1" CreationDate="2015-03-11T05:25:47.053" Score="1" ViewCount="1311" Body="&lt;p&gt;I am a newbie. I would like to do visualization on twitter data : top trends based on country (over map) and time variations (for each months in 1 year or for each year) . Can someone tell me where can I get the twitter data set and any advice on how to start proceeding would be really help full.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="8610" LastActivityDate="2015-03-13T12:16:11.643" Title="How do I get Twitter Dataset for Visualization" Tags="|data-mining|clustering|data-cleaning|visualization|" AnswerCount="2" CommentCount="1" ClosedDate="2015-03-14T22:06:09.250" ContentLicense="CC BY-SA 3.0" />
  <row Id="5306" PostTypeId="2" ParentId="5305" CreationDate="2015-03-11T06:52:39.440" Score="3" Body="&lt;p&gt;(Fresh and complete) Twitter data is not easy to get nowadays, as Twitter is actively monetizing it (through GNIP). There is still a large Twitter graph collected by Kwak et al. for their &quot;What is Twitter, a social network or a news media?&quot; WWW'10 &lt;a href=&quot;http://an.kaist.ac.kr/traces/WWW2010.html&quot; rel=&quot;nofollow&quot;&gt;publicly available&lt;/a&gt;, but as to tweets, most nice datasets have disappeared from the Internet due to complaints from Twitter. There are still tiny datasets of tweets available online (&lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/&quot; rel=&quot;nofollow&quot;&gt;e.g.&lt;/a&gt;), but they are small and stale. Thus, I see two &quot;legal&quot; ways to go about getting tweets now: 1) crawl Twitter using the new version of Twitter API (the obvious limitation of this approach is the limit on the number of requests you can send; this approach is unlikely to give you a representative picture of what is trending if you collect raw tweets, just because you won't have enough tweets); 2) buy data from Twitter (GNIP), which will cost an awful lot of money. Alternatively, you can 3) study Twitter API for other means of getting &quot;summarized&quot; information from it, not tweets directly. For example, they may provide a method to retrieve, say, 10 most popular hash tags at the moment, maybe, even filtered by geo-location -- you cannot do much data science with the result, but at least you can use it in your visualization application.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastActivityDate="2015-03-11T06:52:39.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5308" PostTypeId="2" ParentId="5304" CreationDate="2015-03-11T11:44:01.873" Score="1" Body="&lt;p&gt;Off the top of my head, I would consider, at least, two &lt;strong&gt;approaches&lt;/strong&gt;, as follows.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Graph sampling.&lt;/strong&gt; If you can (and it makes sense to) model your &lt;em&gt;population&lt;/em&gt; (database of text documents) as a graph, then consider &lt;em&gt;graph sampling&lt;/em&gt;. Check &lt;a href=&quot;https://stats.stackexchange.com/a/128661/31372&quot;&gt;this relevant answer&lt;/a&gt; of mine on the topic on &lt;em&gt;Cross Validated&lt;/em&gt; site (contains many references - I would start with &lt;a href=&quot;http://arxiv.org/abs/1308.5865&quot; rel=&quot;nofollow noreferrer&quot;&gt;this nice overview paper&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Topic modeling.&lt;/strong&gt; Another alternative is topic modeling (also referred to as topic models). See &lt;a href=&quot;http://www.cs.princeton.edu/~blei/papers/Blei2012.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this introductory paper&lt;/a&gt; as well as &lt;a href=&quot;http://www.cs.princeton.edu/~blei/papers/BleiLafferty2009.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this more detailed paper&lt;/a&gt; on the subject. Also, take a look at &lt;a href=&quot;http://people.cs.umass.edu/~lmyao/papers/fast-topic-model10.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this interesting paper&lt;/a&gt; on topic modeling of &lt;em&gt;streaming&lt;/em&gt; documents (or other situations, such as large databases, when unsupervised solutions make more sense). Finally, speaking about &lt;strong&gt;software&lt;/strong&gt; for topic modeling, while many options do exist, I suggest investigating &lt;a href=&quot;http://mallet.cs.umass.edu&quot; rel=&quot;nofollow noreferrer&quot;&gt;MALLET&lt;/a&gt;, which is an interesting Java-based software package for &lt;em&gt;&quot;statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text&quot;&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-11T11:44:01.873" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5309" PostTypeId="1" AcceptedAnswerId="5310" CreationDate="2015-03-12T11:08:43.067" Score="1" ViewCount="766" Body="&lt;p&gt;I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow initialization and the biases are initialized to 1. I am using MATLAB as a platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running the network a number of times without changing any parameters, I am getting results (mean squared error) which range from 0.5 to 0.5*10^-6. I cannot understand how the results can even vary that much, I'd imagine there would at least be a narrower and more consistent window of errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be causing such a large variance?&lt;/p&gt;&#xA;" OwnerUserId="8626" LastActivityDate="2015-03-14T13:02:21.530" Title="How can the performance of a neural network vary considerably without changing any parameters?" Tags="|neural-network|parameter|matlab|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5310" PostTypeId="2" ParentId="5309" CreationDate="2015-03-12T13:18:03.737" Score="2" Body="&lt;p&gt;In general, there is no guarantee that ANNs such as a multi-layer Perceptron network will converge to the global minimum squared error (MSE) solution. The final state of the network can be heavily dependent on how the network weights are initialized. Since most initialization schemes (including Nguyen-Widrow) use random numbers to generate the initial weights, it is quite possible that some initial states will result in convergence on local minima, whereas others will converge on the MSE solution.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-03-12T13:18:03.737" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5312" PostTypeId="2" ParentId="5305" CreationDate="2015-03-13T12:16:11.643" Score="1" Body="&lt;p&gt;There's a tweet collection (comprising of 16 million tweets) that you can get from NIST. This collection is used for the ad hoc search task on micro-blogs. &#xA;&lt;a href=&quot;http://trec.nist.gov/data/tweets/&quot; rel=&quot;nofollow&quot;&gt;TREC 2011 Microblog collection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="984" LastActivityDate="2015-03-13T12:16:11.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5313" PostTypeId="1" CreationDate="2015-03-13T14:10:11.143" Score="0" ViewCount="2792" Body="&lt;p&gt;In a assignment we are given macro economic indicators like GDP, Consumer price index, Producer Price index and Industrial production index. Also we are given Crude oil, Sugar prices and FM-CG Sales. We are required to forecast future quarter sales and give a model. As I'm new to this subject, I don't know where to start with it, or what to read. Can anyone provide me with some examples of what to do, or any PDFs which might be helpful. &lt;/p&gt;&#xA;" OwnerUserId="8641" LastActivityDate="2015-03-13T15:24:57.223" Title="Forecasting sales and creating model" Tags="|predictive-modeling|forecast|" AnswerCount="1" CommentCount="1" ClosedDate="2015-03-14T22:09:31.197" ContentLicense="CC BY-SA 3.0" />
  <row Id="5314" PostTypeId="2" ParentId="5313" CreationDate="2015-03-13T15:24:57.223" Score="1" Body="&lt;p&gt;If you are new to forecasting, I would recommend starting with a very simple model for your sales, e.g., Exponential Smoothing, which will use &lt;em&gt;only&lt;/em&gt; the sales data and no external information, but will capture trend and seasonality. &lt;a href=&quot;https://www.otexts.org/fpp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is an extremely good free online textbook.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you are comfortable with your simple model, you can look at more complex ones that do include external causal variables. For instance, you could run ARIMAX models (the X stands for eXternal or eXplanatory variables). For this, look at &lt;a href=&quot;https://www.otexts.org/fpp/8&quot; rel=&quot;nofollow noreferrer&quot;&gt;chapter 8 in the online textbook&lt;/a&gt; and check the &lt;code&gt;auto.arima()&lt;/code&gt; function in the &lt;code&gt;forecast&lt;/code&gt; package for R, where you can specify external variables using the &lt;code&gt;xreg&lt;/code&gt; parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could look at other methods like Neural Networks, or possibly even Vector Autoregression. To learn more, you could look into standard econometrics textbooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whatever you do, keep two things in mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you include explanatory variables in your forecast, you will need to forecast those variables themselves. In an assignment-type situation, you may be able to work with &lt;em&gt;actual&lt;/em&gt; future macroeconomic variables, but of course in a &quot;real&quot; forecasting situation, you don't know next month's sugar prices yet. Forecasting these to feed the sugar price forecast into your sales forecast model adds an additional bit of uncertainty.&lt;/li&gt;&#xA;&lt;li&gt;You may just find out that your fancy model with lots of additional explanatory variables does not do a much better job at forecasting than your original simple model. This would not at all be surprising. &lt;a href=&quot;https://stats.stackexchange.com/questions/124955/is-it-unusual-for-mean-to-outperform-arima/124956#124956&quot;&gt;Simple methods very often outperform complex ones in forecasting.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2853" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-13T15:24:57.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5315" PostTypeId="2" ParentId="5304" CreationDate="2015-03-13T15:41:35.577" Score="0" Body="&lt;p&gt;You're going to need to do simple random sampling, but maintain counts of the labels you've seen thus far. When you reach quota for a label, you'll need to cull all documents with that label from the pool, and sample from the remaining documents without that label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;think&lt;/em&gt; this will be a fair way of getting a stratified sample when your labels are overlapping.&lt;/p&gt;&#xA;" OwnerUserId="1399" LastActivityDate="2015-03-13T15:41:35.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5316" PostTypeId="1" CreationDate="2015-03-13T16:41:29.280" Score="31" ViewCount="33387" Body="&lt;p&gt;Given a sentence like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Complimentary gym access for two for the length of stay ($12 value per person per day)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What general approach can I take to identify the word gym or gym access?&lt;/p&gt;&#xA;" OwnerUserId="8643" LastActivityDate="2020-08-20T18:39:54.567" Title="General approach to extract key text from sentence (nlp)" Tags="|machine-learning|nlp|text-mining|data-cleaning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5318" PostTypeId="1" CreationDate="2015-03-13T22:43:42.760" Score="2" ViewCount="1924" Body="&lt;p&gt;For example, there are 4 objects. Each object has some elements. There are 3 clusters. Each element of each object belongs to some cluster. Clustering was done before, and we know only how many elements of each object belong to some cluster.&#xA;How we can calculate the porbability of similarity of two objects?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;         Cluster1  Cluster2  Cluster3&#xA;Object1    500       300        200&#xA;Object2    200       200        100&#xA;Object3    250       2500       1250       &#xA;Object4    190       210        300&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="8645" LastEditorUserId="8878" LastEditDate="2015-05-18T07:34:46.413" LastActivityDate="2015-05-18T09:20:51.007" Title="Probability of similarity of two clusters" Tags="|clustering|similarity|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5319" PostTypeId="2" ParentId="5309" CreationDate="2015-03-14T13:02:21.530" Score="0" Body="&lt;p&gt;I asked a similar question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/140406/70282&quot;&gt;https://stats.stackexchange.com/q/140406/70282&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I ended up doing is having a while loop repeatably create NNs of a small size looking for the best root mean square error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially:&#xA;For hidden layer size of 1 to 10 do&#xA;For trials of 1 to 10:&#xA;Create NN and test for rmse with test data.&#xA;Remember NN and best rmse for particular hidden layer size.&#xA;Done for.&#xA;Done for.&#xA;Review results choosing the smallest hidden layer size which had a small rmse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By separating your data into training and tedtinh data, you don't pick an over fit NN.&lt;/p&gt;&#xA;" OwnerUserId="8650" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-14T13:02:21.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5320" PostTypeId="1" AcceptedAnswerId="5327" CreationDate="2015-03-14T14:48:12.233" Score="0" ViewCount="114" Body="&lt;p&gt;I am new to python programming. As part of a data analysis project, I am trying to get a scatter plot of Salary vs Wins for each year of 4 consecutive years (so I am trying to get 4 different scatter plots, one for each year). I am using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;teamName = 'OAK'&#xA;years = np.arange(2000,2004)&#xA;for yr in years:&#xA;    df = joined[joined['yearID']==yr]&#xA;    plt.scatter(df['salary']/1e6,df['W'])&#xA;    plt.title('Wins vs Salaries in year' + str(yr))&#xA;    plt.xlabel('Total Salary (in millions)')&#xA;    plt.ylabel('Wins')&#xA;    plt.xlim(0,180)&#xA;    plt.ylim(30,130)&#xA;    plt.grid()&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I am only getting one plot corresponding to 2003. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point out the mistake ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2015-03-15T19:42:16.513" Title="Need help with python code as part of a data analysis project" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5322" PostTypeId="1" CreationDate="2015-03-14T19:18:11.747" Score="3" ViewCount="2256" Body="&lt;p&gt;The optimisation problem for support vector regression is (see &lt;a href=&quot;http://alex.smola.org/papers/2003/SmoSch03b.pdf&quot; rel=&quot;nofollow&quot;&gt;http://alex.smola.org/papers/2003/SmoSch03b.pdf&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;minimise:&#xA;\begin{align*}&#xA;C\sum_{i=0}^{l}(\xi_{i} +\xi^{*}_{i})+ \frac{1}{2}\lVert w \rVert^{2}&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;subject to the constraints:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;&amp;amp; y_{i} - &amp;lt;w,x_{i}&amp;gt; - b \leq \epsilon + \xi_{i} \\&#xA;&amp;amp; &amp;lt;w,x_{i}&amp;gt; + b - y_{i} \leq \epsilon + \xi^{*}_{i} \\&#xA;&amp;amp; \xi_{i}, \xi^{*}_{i} \geq 0&#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand where the $\lVert w \rVert^{2}$ comes from. I understand how the $\lVert w \rVert^{2}$ is derived in the case of support vector classification (by maximizing the margin), but not in the case of regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper says that the &quot;goal is to find a function $f(x)$ that has at most $\epsilon$ deviation from the actually obtained targets $y_{i}$ for all the training data, and at the same time is as flat as possible. In other words, we do not care about errors as long as they are less than $\epsilon$, but will not accept any deviation larger than this.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am not sure what &quot;flat&quot; means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does someone know?&lt;/p&gt;&#xA;" OwnerUserId="8653" LastEditorUserId="8653" LastEditDate="2015-03-15T11:47:05.957" LastActivityDate="2015-04-14T18:56:07.833" Title="Cost function for support vector regression" Tags="|machine-learning|regression|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5323" PostTypeId="1" AcceptedAnswerId="5324" CreationDate="2015-03-15T11:47:55.003" Score="0" ViewCount="1635" Body="&lt;p&gt;I have implemented an interactive visualization with d3.js, javascript to explore the frequency and various combinations of co-occurring item sets. I want to complement the interactive exploration with some automated options.&lt;/p&gt;&#xA;&lt;p&gt;Does someone know an efficient javascript implementation of the association rules mining ?&lt;/p&gt;&#xA;&lt;p&gt;My typical scenario will have just up to 30 different items.&#xA;There are some good web site with implementations of frequent item set mining (improvements from the initial apriori algorithm): &lt;a href=&quot;http://www.borgelt.net/apriori.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.borgelt.net/apriori.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Any help is greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="8191" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-03-15T18:37:55.840" Title="Implementation of Association Rules in Javascript" Tags="|efficiency|javascript|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5324" PostTypeId="2" ParentId="5323" CreationDate="2015-03-15T18:37:55.840" Score="0" Body="&lt;p&gt;I don't know how efficient they are, but I did find some implementations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/dmarges/apriori&quot; rel=&quot;nofollow&quot;&gt;https://github.com/dmarges/apriori&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/seratch/apriori.js&quot; rel=&quot;nofollow&quot;&gt;https://github.com/seratch/apriori.js&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-03-15T18:37:55.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5325" PostTypeId="2" ParentId="5322" CreationDate="2015-03-15T18:53:08.857" Score="4" Body="&lt;p&gt;Flat means parallel to the x axis; having a small slope. The smaller w is, the closer f(x) is to b; recall that $f(x) \equiv \left&amp;lt; w, x \right&amp;gt; + b$. One way to think about this is as a form of regularization; the flatter the function, the simpler or more parsimonious it is. This applies to both classification and regression.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-03-15T18:53:08.857" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5326" PostTypeId="1" CreationDate="2015-03-15T18:54:51.473" Score="3" ViewCount="17533" Body="&lt;p&gt;I want to read a csv file as input from user in Shiny and assign it to a variable in global.r file.The code I have in ui.R is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fileInput('file1', 'Choose CSV File',&#xA;                     accept=c('text/csv', 'text/comma-separated-values,text/plain', .csv'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code I have in main Panel of server.R is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;textOutput('contents')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code I have currently in server.R is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output$contents &amp;lt;- renderText({    &#xA;      if(is.null(input$file1))return()&#xA;      inFile &amp;lt;- input$file1&#xA;      data&amp;lt;-read.csv(inFile$datapath)&#xA;  print(summary(data))&#xA;&#xA; })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to assign this input$file1 to a variable called 'data' in global.r. Please let me know if this is possible. &#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="8662" LastEditorUserId="8953" LastEditDate="2015-05-02T13:40:37.703" LastActivityDate="2015-05-02T13:40:37.703" Title="How to read file from user in Shiny and assign it to a variable in global.r?" Tags="|r|csv|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5327" PostTypeId="2" ParentId="5320" CreationDate="2015-03-15T19:42:16.513" Score="0" Body="&lt;p&gt;All of your plots are appearing on top of each other.  You need to invoke plt.subplot(xxx) before you create each plot.  For info on how the xxx command works, go to &lt;a href=&quot;http://www.mathworks.com/help/matlab/ref/subplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the MATLAB documentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might end up with multiple figures - see &lt;a href=&quot;https://stackoverflow.com/questions/21321764/matplotlib-multiple-plots-on-one-figure&quot;&gt;this page&lt;/a&gt; for info about that.&lt;/p&gt;&#xA;" OwnerUserId="8663" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-03-15T19:42:16.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5328" PostTypeId="1" CreationDate="2015-03-16T11:20:08.417" Score="2" ViewCount="97" Body="&lt;p&gt;I am from information security field, and have some introductory level understanding of machine learning field. My problem is to identify behavioural patterns from network traffic. If I use supervised learning ( classification ) the results are very promising but I encounter of a problem of missing out a pattern with slight change in behaviour. And If I use clustering then i need to do a much work on manual tuning of centroids position to get better clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking to ensemble classification and clustering and use classification output to tune cluster centroids. Is this is a good idea to address the problem?&lt;/p&gt;&#xA;" OwnerUserId="4993" LastActivityDate="2015-03-16T11:20:08.417" Title="use classification to improve clustering" Tags="|machine-learning|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="5329" PostTypeId="1" AcceptedAnswerId="5337" CreationDate="2015-03-16T11:33:30.593" Score="1" ViewCount="1107" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow&quot;&gt;Jaccard coefficient&lt;/a&gt; measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had 100 sets all of same cardinality. By mistake I calculated the similarity measures as the ratio of intersection with total elements in a set (i.e 100).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives different similarity values than the original Jaccard formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if the original formula has considered the union of two sets to handle cases where there might be sets with different cardinalities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think though numerically my values are different, they repersent the same idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anybody could verify/disverify what I am trying to do ?&lt;/p&gt;&#xA;" OwnerUserId="6576" LastEditorUserId="6576" LastEditDate="2015-03-16T11:44:56.193" LastActivityDate="2015-03-17T14:57:23.023" Title="Measuring similarity for sets with same cardinality" Tags="|dataset|clustering|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5330" PostTypeId="1" CreationDate="2015-03-16T19:53:31.303" Score="0" ViewCount="186" Body="&lt;p&gt;I am subsetting some data frames and am sure there is a better way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, I have two data frames.  The first is the actual data.  The second has some meta data and importantly a flag on whether or not the row is in the subset I am interested in.  All I would like to do is pull out the subset and write a file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, here is my subset data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     head(temp[,c(1,4:8)])&#xA;     ensembl_gene_id   FGF Secreted  SigP Receptors    TF&#xA;1 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;2 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;3 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;4 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;5 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;6 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is my actual data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Expt1_C1_EXONS_dxd_res_wDesc[1:5,1:5]&#xA;                                   groupID featureID exonBaseMean dispersion         stat&#xA;ENSMUSG00000000001:E001 ENSMUSG00000000001      E001    624.80240 0.04271781  1.255734504&#xA;ENSMUSG00000000001:E002 ENSMUSG00000000001      E002     30.92281 0.02036015  0.514038911&#xA;ENSMUSG00000000001:E003 ENSMUSG00000000001      E003     41.61413 0.01546023 10.105615831&#xA;ENSMUSG00000000001:E004 ENSMUSG00000000001      E004    137.47209 0.03975305  0.281105120&#xA;ENSMUSG00000000001:E005 ENSMUSG00000000001      E005     85.97116 0.05753662  0.005482149`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I was doing is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;write.table(Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,],&#xA;            &quot;Expt1_C1_EXONS_dxd_res_wDesc_FGF.tab&quot;,col.names=T,row.names=F,quote=F)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is taking 20+ min per subset just to subset and write the data.  The data frame is 370,000 rows by 27 variables, so large but not huge.  I have about 30 of these to do.  Is there a more efficient way?  Note, the groupID does NOT equal the first column in my subset data frame.  In some instances the groupID contains a concatenated set of ensembl ids.  So I have preprocessed to get the temp data frame to have what I want in the same row order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Bob&lt;/p&gt;&#xA;" OwnerUserId="8677" LastEditorUserId="8679" LastEditDate="2015-03-17T11:45:44.380" LastActivityDate="2015-03-18T12:47:38.797" Title="subsetting R data frame" Tags="|r|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5331" PostTypeId="1" CreationDate="2015-03-17T02:44:16.263" Score="1" ViewCount="210" Body="&lt;p&gt;I am trying to write an ANN in python for handwriting recognition by mouse movements. ( like identify characters we draw in paint app n convert it to text) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question might seem that I haven't done my homework, but I am *not * looking for image datasets for handwritten characters.( at least that's what I've figured out so far)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any database for such (array or co-ordinates corresponding to each character) inputs? If not, how can I create this database from existing ones like MNIST?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I have written a script to generate my own database because I could not find any. But it seems a very tedious task to generate a good database. &#xA;I am a beginner in ML. Could somebody point me in &lt;em&gt;right direction&lt;/em&gt; upon how else can we do it?&lt;/p&gt;&#xA;" OwnerUserId="8681" LastActivityDate="2015-03-17T02:44:16.263" Title="Better approach for handwriting recognition?" Tags="|machine-learning|python|neural-network|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="5332" PostTypeId="1" AcceptedAnswerId="5333" CreationDate="2015-03-17T04:25:34.463" Score="1" ViewCount="782" Body="&lt;p&gt;If I prune a decision tree, does that make the resulting decision tree always more general than the original decision tree?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there examples where this is not the case?&lt;/p&gt;&#xA;" OwnerUserId="8505" LastEditorUserId="25218" LastEditDate="2016-11-19T12:08:42.233" LastActivityDate="2016-11-19T12:08:42.233" Title="Does pruning a decision tree always make it more general?" Tags="|machine-learning|classification|decision-trees|supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5333" PostTypeId="2" ParentId="5332" CreationDate="2015-03-17T07:21:54.110" Score="3" Body="&lt;p&gt;If you filter something out by choosing one branch over another branch in the tree, the observations you did not choose are forever lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to directly answer your question - no, it does not always make it more general. If you construct a tree where all the decision are exactly the same, then pruning does not make it more general.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-03-17T07:21:54.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5334" PostTypeId="2" ParentId="5330" CreationDate="2015-03-17T07:30:40.047" Score="1" Body="&lt;p&gt;Only reason for such a long subset time is that you're running out of memory and the OS starts to swap. AFAIK base data.frame will create a copy of the data in memory before writing it to file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is use either dplyr or data.table packages as they manage some of these steps without copying the data.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-03-17T07:30:40.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5336" PostTypeId="1" AcceptedAnswerId="5338" CreationDate="2015-03-17T12:24:24.303" Score="1" ViewCount="2048" Body="&lt;p&gt;I'm struggling to find a solution to produce a line chart which shows a trend from a Time Series metric. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I try to make my line chart look similar to the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/hZaa2.png&quot; alt=&quot;Time series&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I want to make use of relative change rates, but I struggle to find a calculation which makes a visualization as seen above possible. The graph should always be relative to a specific time window, meaning that I can query my data dynamically with a start and end timestamp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My (incomplete) sample data set is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp,value,change_rate,change_rate_absolute_sum,change_rate_delta,change_rate_sum&#xA;1426587900000,778;0.00129,0.00129;0.00129,0.00129&#xA;1426588200000,778;0,0.00129;-0.00129,0&#xA;1426588500000,1189;0.52828,0.52957;0.52828,0.52828&#xA;1426588800000,1195;0.00505,0.53462;-0.52323,0.00505&#xA;1426589100000,1195;0,0.53462;-0.00505,0&#xA;1426589400000,1196;0.00084,0.53546;0.00084,0.00084&#xA;1426589700000,1286;0.07525,0.61071;0.07441,0.07525&#xA;1426590000000,1290;0.00311,0.61382;-0.07214,0.00311&#xA;1426590300000,1294;0.0031,0.61692;-0.00001,0.0031&#xA;1426590600000,1296;0.00155,0.61847;-0.00155,0.00155&#xA;1426590900000,1356;0.0463,0.66477;0.04475,0.0463&#xA;1426591200000,1358;0.00147,0.66624;-0.04483,0.00147&#xA;1426591500000,1358;0,0.66624;-0.00147,0&#xA;1426591800000,1360;0.00147,0.66771;0.00147,0.00147&#xA;1426592100000,1408;0.03529,0.703;0.03382,0.03529&#xA;1426592400000,1390;-0.01278,0.69022;-0.04807,-0.01278&#xA;1426592700000,1391;0.00072,0.69094;0.0135,0.00072&#xA;1426593000000,1410;0.01366,0.7046;0.01294,0.01366&#xA;1426593300000,1414;0.00284,0.70744;-0.01082,0.00284&#xA;1426593600000,1410;-0.00283,0.70461;-0.00567,-0.00283&#xA;1426593900000,1414;0.00284,0.70745;0.00567,0.00284&#xA;1426594200000,1420;0.00424,0.71169;0.0014,0.00424&#xA;1426594500000,1417;-0.00211,0.70958;-0.00635,-0.00211&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any ideas warmly appreciated. Thanks a lot in advance!&lt;/p&gt;&#xA;" OwnerUserId="8686" LastActivityDate="2015-03-17T14:53:39.517" Title="Time series change rate calculations for displaying trend line chart" Tags="|dataset|time-series|infographics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5337" PostTypeId="2" ParentId="5329" CreationDate="2015-03-17T14:44:22.403" Score="1" Body="&lt;p&gt;Yes, the Jaccard similarity score is normalized by the union to deal with sets of different cardinality. Without this normalization (if you used just the intersection), very small sets would always have very low scores. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the cardinalities of all your sets are the same, the union of any two sets will be a straightforward function of the intersection (this is easy to visualize - as the two sets intersect more and more, their unions get smaller and smaller). The formula is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;union = 2 * cardinality - intersection&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the Jaccard score in your case would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;intersection / (200 - intersection)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you plot this, you'll see it's monotonically the same as what you did. &lt;/p&gt;&#xA;" OwnerUserId="8689" LastEditorUserId="8689" LastEditDate="2015-03-17T14:57:23.023" LastActivityDate="2015-03-17T14:57:23.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5338" PostTypeId="2" ParentId="5336" CreationDate="2015-03-17T14:48:29.080" Score="1" Body="&lt;p&gt;While it is not very clear to me what specific relationships between your data you want to display, I can give some general advice. I think that this time series visualization calls for so-called &lt;a href=&quot;http://geog.uoregon.edu/datagraphics/color_scales.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;diverging color scheme&lt;/a&gt; (as opposed to &lt;em&gt;sequential&lt;/em&gt;, &lt;em&gt;categorical&lt;/em&gt; and other ones). &lt;a href=&quot;http://colorbrewer2.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;ColorBrewer&lt;/a&gt; is nice online tool for selecting an appropriate &lt;em&gt;color scheme&lt;/em&gt; and other parameters (note that &lt;em&gt;ColorBrewer&lt;/em&gt;'s color schemes have built-in support in major data analysis and visualization software, such as R, Python, d3.js, plot.ly and others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would use &lt;code&gt;R&lt;/code&gt; environment, it would be quite easy to produce the time series &lt;em&gt;trend line chart&lt;/em&gt; that you want by using &lt;code&gt;ggplot2&lt;/code&gt; package and its &lt;code&gt;scale_color_gradient2()&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;... + scale_color_gradient2(midpoint=midValue, low=&quot;blue&quot;, mid=&quot;white&quot;, high=&quot;orange&quot; )&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick glance at your profile suggested me that you would prefer a JavaScript solution. In that case, I would advise to use &lt;code&gt;d3.js&lt;/code&gt; library or one of multiple other &lt;em&gt;JavaScript visualization&lt;/em&gt; libraries or packages. For more details, check &lt;a href=&quot;https://datascience.stackexchange.com/a/3723/2452&quot;&gt;this relevant answer&lt;/a&gt; of mine.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-03-17T14:53:39.517" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="5342" PostTypeId="2" ParentId="2391" CreationDate="2015-03-17T20:13:00.767" Score="0" Body="&lt;p&gt;If you label metric 3 as $x_3 = \{1,0\}$, where $1$ means it is an anomaly, this becomes a  &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt; problem where $\mathbb{P}(X_3 = 1) = logit(\beta_0 + \beta_1 x_1 + \beta_2 x_2)$.&lt;/p&gt;&#xA;" OwnerUserId="8696" LastActivityDate="2015-03-17T20:13:00.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5343" PostTypeId="2" ParentId="5301" CreationDate="2015-03-18T01:34:33.133" Score="2" Body="&lt;p&gt;Do you change the patterns often? If not, then you can use Aho-Corasick method, whose idea is, first, to build a finite automaton based on your patterns and, then, to make a single pass over the text with this automaton to find matches (if the automaton visits a &quot;matching&quot; state, then there is a match). The complexity of the automata building should be linear in the length of the patterns (# of patters * max pattern length), and the matching phase should be linear in the size of the text you search in.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastActivityDate="2015-03-18T01:34:33.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5344" PostTypeId="1" CreationDate="2015-03-18T08:10:08.943" Score="1" ViewCount="168" Body="&lt;p&gt;I'm going to describe the procedure for gradient descent using the language of English:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1)&lt;/strong&gt; For all of the files (e.g. documents) in our training corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&gt; commence, now we are considering our first document&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2)&lt;/strong&gt; For all the features in the feature vector associated with that training example (document)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3)&lt;/strong&gt; Take the dot product of the vector of weight values * and the vector of features specific to the document under consideration&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* &lt;em&gt;vector of weight values is applicable to all features, though all features will not be present in all documents, indeed only some features will be present in each individual document&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4)&lt;/strong&gt; Take the value resulting from step 3 and apply this to our loss function. I'm using the Hinge Loss function, i.e. the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/tVvmW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that equation saying that the hinge loss objective function should be the sum of the misclassification error for every file in our training set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is that equivalent to the java code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Math.max(0, 1 - y * value_from_step_three)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;y = {-1, 1}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;^ THIS IS MY FIRST POINT OF CONFUSION&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess that is inaccurate but I'm not sure, according to my mental model it should be right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5)&lt;/strong&gt; For every weight, update it as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;new weight value&lt;sub&gt;j&lt;/sub&gt; = old weight value&lt;sub&gt;j&lt;/sub&gt; - learning rate * &lt;em&gt;value from step 4&lt;/em&gt; * value of feature&lt;sub&gt;j&lt;/sub&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a bit confused about what &lt;em&gt;value of feature&lt;sub&gt;j&lt;/sub&gt;&lt;/em&gt; should be, since I guess this would correspond to a vector, that has values for every weight under consideration, i.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document 1 = [&quot;I&quot;, &quot;am&quot;, &quot;awesome&quot;]&#xA;Document 2 = [&quot;I&quot;, &quot;am&quot;, &quot;great&quot;, &quot;great&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dictionary is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&quot;I&quot;, &quot;am&quot;, &quot;awesome&quot;, &quot;great&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the documents as a vector would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document 1 = [1, 1, 1, 0]&#xA;Document 2 = [1, 1, 0, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There will be weights:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Weight&lt;sub&gt;&quot;I&quot;&lt;/sub&gt;, Weight&lt;sub&gt;&quot;am&quot;&lt;/sub&gt;, Weight&lt;sub&gt;&quot;awesome&quot;&lt;/sub&gt;, Weight&lt;sub&gt;&quot;great&quot;&lt;/sub&gt;]&lt;/p&gt;&#xA;" OwnerUserId="8285" LastEditorUserId="8285" LastEditDate="2015-03-18T08:22:04.997" LastActivityDate="2015-03-18T08:22:04.997" Title="procedure for gradient descent" Tags="|gradient-descent|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5345" PostTypeId="1" AcceptedAnswerId="28853" CreationDate="2015-03-18T11:39:40.353" Score="61" ViewCount="110096" Body="&lt;p&gt;I use RStudio for R programming. I remember about solid IDE-s from other technology stacks, like Visual Studio or Eclipse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What other IDE-s than RStudio are used (please consider providing some brief description on them).&lt;/li&gt;&#xA;&lt;li&gt;Does any of them have noticeable advantages over RStudio?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I mostly mean debug/build/deploy features, besides coding itself (so text editors are probably not a solution).&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-06-16T07:51:34.547" LastActivityDate="2022-01-18T04:57:26.037" Title="IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Studio)" Tags="|r|tools|rstudio|programming|" AnswerCount="10" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5347" PostTypeId="2" ParentId="5330" CreationDate="2015-03-18T12:47:38.797" Score="0" Body="&lt;p&gt;Without looking into it too deeply (i don't have access to R on this computer), it seems...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;temp&lt;/code&gt; is a data frame with x rows, &lt;code&gt;Expt1_C1_EXONS_dxd_res_wDesc&lt;/code&gt; is a data frame with y rows. &lt;code&gt;temp$FGF&lt;/code&gt; is therefore a vector of length x. Since temp is a subset, x &amp;lt; y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you write &lt;code&gt;Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,]&lt;/code&gt; you might be getting errors (NA's) due to the mismatch in length. It may also be because you are not specifying which rows you want, the vector &lt;code&gt;temp$FGF&lt;/code&gt; contains TRUE and FALSE. It doesn't contain row indices. &lt;code&gt;temp$FGF==TRUE&lt;/code&gt; returns (or at least selects based on) indices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(As i say, i don't have R in front of me to check.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as far as i remember, you need to specify whether you want TRUE or FALSE entries to be included. R does not automatically assume you want TRUE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Again, i don't have R in front of me to check.)&lt;/p&gt;&#xA;" OwnerUserId="8708" LastActivityDate="2015-03-18T12:47:38.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5349" PostTypeId="1" CreationDate="2015-03-18T17:25:32.460" Score="4" ViewCount="843" Body="&lt;p&gt;I'm not sure which word to use to differentiate a self-organizing map (SOM) training procedure in which updates for the entire data set are aggregated before they are applied to the network from a training procedure in which the network is updated with each data point individually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of other algorithms I'd say &lt;code&gt;stochastic gradient descent&lt;/code&gt;, but I'm not sure &lt;code&gt;gradient descent&lt;/code&gt; is correct for SOM learning.  To my knowledge, SOM learning does not follow any energy function exactly, so that should mean it doesn't do gradient descent exactly, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another term would be &lt;code&gt;online learning,&lt;/code&gt; but `online' sort of implies I train my SOM in the real world with data points streaming in, eg. from a set of sensors.  It may also imply that I use every data point only once, which is not what I want to say.&lt;/p&gt;&#xA;" OwnerUserId="8714" LastEditorUserId="8714" LastEditDate="2015-03-19T08:25:28.603" LastActivityDate="2015-12-06T12:17:40.473" Title="Terminology: SOMs, batch learning, online learning, and stochastic gradient descent" Tags="|machine-learning|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5350" PostTypeId="2" ParentId="3763" CreationDate="2015-03-18T22:58:40.487" Score="1" Body="&lt;p&gt;My best guess:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;After three weeks he’d harvested 6 million questions and answers from 20,000 women all over the country. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assume that we could collect both what the women answered, what answers they would accept and the importance they attribute to the question. This is the &lt;strong&gt;training set&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;He picked out the 500 questions that were most popular with both clusters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;strong&gt;feature selection&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;loss function&lt;/strong&gt; to be minimized is could be something like the negative sum of (logarithms of) the &lt;a href=&quot;http://www.okcupid.com/help/match-percentages&quot; rel=&quot;nofollow&quot;&gt;match percentages&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m(x,s,w, y_i, t_i, v_i)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;between his answers $x$ (a vector of length 500), the answers he accepts $s$, the importances $w$ he attributes to each questions, the $i$'th woman's answers $y_i$, the answers $t_i$ she would accept and the importances she assigns to them $v_i$. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;He’d already decided he would fill out his answers honestly&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;meaning that $x$ and $s$ are fixed, only $w$ is adjusted to minimize the goal function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the match percentage were something simple like 'number of questions which both answered the same way times your importance assigned to the question', a simple 'give most importance to the question you answered the same way as women in your preferred cluster and zero importance to other questions' should maximize this. However, the match percentage function is more complicated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One would start with equal importance assigned to all questions. &#xA;Then calculate the contribution of each question to the overall loss function: questions $j$ for which his answers do not match many of the women's answers will get a reduced weight in the next round and the weight of well matching questions will be increased (I would have to think about how the weight update would look in detail though).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the question relevances in the vector $w$ can only take on a discrete set of values (0, 1, 10, 50, 250) making it a discrete optimization problem and thus potentially NP-hard. In practice, I would allow the weights $w$ take on any real value between 0 and 250 and then round to the nearest allowed value in the end.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Disclaimer: I have never used OkCupid)&lt;/p&gt;&#xA;" OwnerUserId="462" LastActivityDate="2015-03-18T22:58:40.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5351" PostTypeId="2" ParentId="5345" CreationDate="2015-03-19T00:21:43.773" Score="8" Body="&lt;p&gt;Here's &lt;a href=&quot;https://plugins.jetbrains.com/idea/plugin/6632-r4intellij&quot; rel=&quot;nofollow noreferrer&quot;&gt;R Language Support for IntelliJ IDEA&lt;/a&gt;. However, keep in mind that this support is not in the form of built-in functionality or official plug-in, but rather a &lt;strong&gt;third-party&lt;/strong&gt; plug-in. I haven't tried it, so my opinion on it is limited to the point above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, a better option would be &lt;em&gt;Eclipse&lt;/em&gt;, which offers R support via &lt;em&gt;StatET&lt;/em&gt; IDE: &lt;a href=&quot;http://www.walware.de/goto/statet&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.walware.de/goto/statet&lt;/a&gt;. However, I find &lt;em&gt;Eclipse&lt;/em&gt; IDE too heavyweight. Therefore, my preferred option is &lt;strong&gt;RStudio&lt;/strong&gt; IDE - I don't know why one would prefer other options. I especially like &lt;em&gt;RStudio&lt;/em&gt;'s ability of online access to the full development environment via &lt;strong&gt;RStudio Server&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2017-01-24T01:09:28.290" LastActivityDate="2017-01-24T01:09:28.290" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5352" PostTypeId="2" ParentId="5345" CreationDate="2015-03-19T07:34:00.850" Score="11" Body="&lt;p&gt;VisualStudio added syntax highlighting for R a few days ago: &lt;a href=&quot;https://www.visualstudio.com/news/2015-mar-10-vso&quot;&gt;https://www.visualstudio.com/news/2015-mar-10-vso&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current RStudio preview is pretty cool as well - you can switch to a dark theme, code completion is working well, you can filter in the viewer, etc.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-03-19T07:34:00.850" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5353" PostTypeId="2" ParentId="5301" CreationDate="2015-03-19T07:38:57.917" Score="2" Body="&lt;p&gt;I would recommend regexp pattern matching. I know that usual implementations are slow but you have to study Thompson's construction algorithm for nondeterministic automaton. See the wikipedia dedicated &lt;a href=&quot;http://en.wikipedia.org/wiki/Thompson%27s_construction_algorithm&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt;. However here the wikipedia fails to present this treasure properly. I would strongly recommend to study carefuly this blog article: &lt;a href=&quot;http://swtch.com/~rsc/regexp/regexp1.html&quot; rel=&quot;nofollow&quot;&gt;Regular expressions can be simple and fast&lt;/a&gt;. For implementations you have pointers in the given article (for example awk and grep uses this implementation). &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-03-19T07:38:57.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5354" PostTypeId="2" ParentId="5326" CreationDate="2015-03-19T09:51:34.480" Score="2" Body="&lt;p&gt;I did not use shiny but I tried in my gui code which I wrote using gwidgets to make a ariable global I used &lt;code&gt;data&amp;lt;&amp;lt;-read.csv(inFile$datapath)&lt;/code&gt;&#xA;you can try this.&lt;/p&gt;&#xA;" OwnerUserId="8729" LastActivityDate="2015-03-19T09:51:34.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5355" PostTypeId="2" ParentId="5345" CreationDate="2015-03-19T16:35:56.013" Score="11" Body="&lt;p&gt;What about &lt;a href=&quot;http://ess.r-project.org&quot; rel=&quot;noreferrer&quot;&gt;ESS&lt;/a&gt;, the R (and other stats languages) package for the Emacs editor?&#xA;It's not formally an IDE, though it has many, if not more of the features of RStudio, just in a different UI (code completion, inline help, object-aware autocomplete, debugging etc.).&lt;/p&gt;&#xA;" OwnerUserId="8734" LastActivityDate="2015-03-19T16:35:56.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5357" PostTypeId="1" AcceptedAnswerId="5370" CreationDate="2015-03-20T14:56:23.420" Score="46" ViewCount="54315" Body="&lt;p&gt;I'm an &lt;code&gt;R&lt;/code&gt; language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works out well in my role as a Data Scientist, however, by starting my career in &lt;code&gt;R&lt;/code&gt; and only having basic knowledge of other scripting/web languages, I've felt somewhat inadequate in 2 key areas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Lack of a solid knowledge of programming theory.&lt;/li&gt;&#xA;&lt;li&gt;Lack of a competitive level of skill in faster and more widely used languages like &lt;code&gt;C&lt;/code&gt;, &lt;code&gt;C++&lt;/code&gt; and &lt;code&gt;Java&lt;/code&gt;, which could be utilized to increase the speed of the pipeline and Big Data computations as well as to create DS/data products which can be more readily developed into fast back-end scripts or standalone applications.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The solution is simple of course -- go learn about programming, which is what I've been doing by enrolling in some classes (currently C programming). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, now that I'm starting to address problems #1 and #2 above, I'm left asking myself &quot;&lt;em&gt;Just how viable are languages like &lt;code&gt;C&lt;/code&gt; and &lt;code&gt;C++&lt;/code&gt; for Data Science?&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, I can move data around very quickly and interact with users just fine, but what about advanced regression, Machine Learning, text mining and other more advanced statistical operations? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So. can &lt;code&gt;C&lt;/code&gt; do the job -- what tools are available for advanced statistics, ML, AI, and other areas of Data Science?&lt;/strong&gt; Or must I lose most of the efficiency gained by programming in &lt;code&gt;C&lt;/code&gt; by calling on &lt;code&gt;R&lt;/code&gt; scripts or other languages?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best resource I've found thus far in C is a library called &lt;a href=&quot;http://image.diku.dk/shark/sphinx_pages/build/html/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Shark&lt;/a&gt;, which gives &lt;code&gt;C&lt;/code&gt;/&lt;code&gt;C++&lt;/code&gt; the ability to use Support Vector Machines, linear regression (not non-linear and other advanced regression like multinomial probit, etc) and a shortlist of other (great but) statistical functions.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2020-04-24T23:54:24.993" LastActivityDate="2022-12-05T16:10:12.417" Title="Data Science in C (or C++)" Tags="|machine-learning|bigdata|statistics|programming|c|" AnswerCount="12" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="5358" PostTypeId="1" CreationDate="2015-03-20T15:29:12.540" Score="0" ViewCount="59" Body="&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What approaches are there to not display a search result that a user is not supposed to see?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have the following situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;High-level view&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an enterprise, there are different repositories like websites, databases, filesystem folder etc. And there is a search engine that crawls all those repositories and creates an index. The user can then use the search engine UI to perform searches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say the search engine was &lt;a href=&quot;https://lucene.apache.org/core/&quot; rel=&quot;nofollow&quot;&gt;Apache Lucene&lt;/a&gt; and the encompassing indexing and retrieval system &lt;a href=&quot;http://lucene.apache.org/solr/&quot; rel=&quot;nofollow&quot;&gt;Apache Solr&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;User permissions are managed via &lt;a href=&quot;http://en.wikipedia.org/wiki/Active_Directory&quot; rel=&quot;nofollow&quot;&gt;Microsoft Active Directory&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The content in the different repositories are managed with all kinds of applications like MS Word, custom-made applications, database management tools etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Search-index and user permissions&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The index contains the documents A1, A2, B1 and B2.&lt;/li&gt;&#xA;&lt;li&gt;User A has only permission to see documents of type A*.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Performing a search&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Now, user A performs a search for &lt;code&gt;A* AND B*&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Displaying the results&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A system administrator with permission to see everything should see all four documents in the search results: A1, A2, B1 and B2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;User A however, who only has permissions to see documents of type A*, should/will only see two documents in the search results: A1 and A2. So user A shouldn't even know that documents B1 and B2 exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Elaboration on the question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What ways are there to implement that requirement that users only see documents they are allowed to see?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose it would be a bad idea to add the information to the index whether a user may see a document. I think that because of this use case: suppose all of a sudden user A may only see document A1 and not A2 anymore; those permissions are now set for example on the filesystem. Now it takes a while until the permissions in the index are updated and during that time user A would still be able to see document A2 in the search results even if he can't click on the result and access it anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the above approach probably would break down when there are hundreds of millions of documents and thousands of employees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where and how would the security aspect be implemented in a content indexing and retrieval system in the above scenario?&lt;/p&gt;&#xA;" OwnerDisplayName="user8743" LastEditorDisplayName="user8743" LastEditDate="2015-03-20T18:13:34.507" LastActivityDate="2015-03-20T18:13:34.507" Title="What approaches are there to not display a search result that a user has no permission to see?" Tags="|search|information-retrieval|" AnswerCount="1" CommentCount="5" ClosedDate="2015-03-31T08:10:38.127" ContentLicense="CC BY-SA 3.0" />
  <row Id="5359" PostTypeId="2" ParentId="5326" CreationDate="2015-03-20T17:08:05.330" Score="3" Body="&lt;p&gt;I have added the following code to global.R file&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- reactiveValues()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used assign function in server.R to assign values to data in global.r&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output$contents &amp;lt;- renderText({    &#xA;      if(is.null(input$file1))return()&#xA;      inFile &amp;lt;- input$file1&#xA;      data2&amp;lt;-read.csv(inFile$datapath)&#xA;  assign('data',data2,envir=.GlobalEnv)&#xA;  print(summary(data))&#xA; })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8662" LastActivityDate="2015-03-20T17:08:05.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5360" PostTypeId="2" ParentId="5358" CreationDate="2015-03-20T17:41:43.970" Score="1" Body="&lt;p&gt;Since you aren't sure which software you would like to use, I can explain this concept only in general, by example. The clearest example is a content management system like Drupal or Wordpress (not necessarily a data science tool but the idea is the same).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a content management system, there are different types of content which can be accessed. In your case, there would be two types - document type A and document type B. There can be an unlimited number of documents, each assigned one or more types. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The software has a permission system that gives access to each document type. This is built with roles. In your example, there would be a system administrator role, and a general user role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each role has permissions associated with it. You would configure system administrator to be allowed to view document types A and B, but you would configure general users to only be able to see document type A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each user registers an account in the software system. A site administrator assigns each account one or more roles. So a user with the system administrator role would be able to see all documents of type A and B. However, if a user only has the general user role, they would only be able to see documents of type A. Also noteworthy is the fact that each user can have multiple roles and will get all permissions associated with every role they have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;USER -has-&gt; ROLE(s)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ROLE -has-&gt; PERMISSION(s)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CONTENT TYPE -requires-&gt; PERMISSION(s)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every time the content is requested, the software system ensures that the user making the request has the permission to access the content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a search is run, the software system has to validate every potential result individually before displaying the search results to the user. The user must have the permission to see each piece of content that is be delivered back from the search. &lt;/p&gt;&#xA;" OwnerUserId="3466" LastEditorUserId="3466" LastEditDate="2015-03-20T17:59:00.130" LastActivityDate="2015-03-20T17:59:00.130" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5361" PostTypeId="2" ParentId="5357" CreationDate="2015-03-20T21:24:40.410" Score="5" Body="&lt;p&gt;R is one of the key tool for data scientist, what ever you do don't stop using it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now talking about C, C++ or even Java.  They are good popular languages. Wether you need them or will need them depend on the type of job or projects you have.  From personal experience, there are so many tools out there for data scientist that you will always feel like you constantly need to be learning.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can add Python or Matlab to things to learn if you want and keep adding.  The best way to learn is to take on a work project using other tools that you are not comfortable with.  If I were you, I would learn Python before C.  It is more used in the community than C.  But learning C is not a waste of your time. &lt;/p&gt;&#xA;" OwnerUserId="8751" LastActivityDate="2015-03-20T21:24:40.410" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5362" PostTypeId="2" ParentId="5345" CreationDate="2015-03-21T01:33:11.087" Score="6" Body="&lt;p&gt;The vim-r-plugin is surprisingly good. You can send lines and paragraphs of code from vim into a tmux session running R in a similar manner to R-Studio. It has &lt;a href=&quot;http://www.lepem.ufc.br/jaa/r-plugin.html#r-plugin-use&quot;&gt;these commands&lt;/a&gt; if you want to check out what functionality it adds to vim. Of course I use all my other normal vim plugins - auto-complete, folding, etc.&lt;/p&gt;&#xA;" OwnerUserId="9768" LastActivityDate="2015-03-21T01:33:11.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5365" PostTypeId="2" ParentId="5357" CreationDate="2015-03-21T13:04:43.810" Score="2" Body="&lt;p&gt;I would be keen to understand why you would need another language (apart form Python) if your goal is &quot; but what about advanced regression, Machine Learning, text mining and other more advanced statistical operations&quot;.&lt;br&gt;&#xA;For that kind of thing, C is a waste of time. It's a good tool to have but in the ~20 years since Java came out, I've rarely coded C.&lt;br&gt;&#xA;If you prefer the more functional-programming side of R, learn Scala before you get into too many procedural bad habits coding with C.&lt;br&gt;&#xA;Lastly learn to use Hadley Wickham's libraries - they'll save you a lot of time doing data manipulation.&lt;/p&gt;&#xA;" OwnerUserId="8762" LastActivityDate="2015-03-21T13:04:43.810" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5366" PostTypeId="2" ParentId="5316" CreationDate="2015-03-21T13:23:35.003" Score="0" Body="&lt;p&gt;If you're a R user, there is a lot of good practical information &lt;a href=&quot;http://www.rdatamining.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Look at their text mining examples.&lt;br /&gt;&#xA;Also, take a look at the tm package.&lt;br /&gt;&#xA;&lt;a href=&quot;http://www.tapor.ca/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is also a good aggregation site.&lt;/p&gt;&#xA;" OwnerUserId="8762" LastEditorUserId="98307" LastEditDate="2020-08-06T13:01:01.780" LastActivityDate="2020-08-06T13:01:01.780" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5367" PostTypeId="1" AcceptedAnswerId="5384" CreationDate="2015-03-21T17:05:07.510" Score="0" ViewCount="564" Body="&lt;p&gt;I'm fairly new to machine learning, but I'm doing my best to learn as much as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious about how predicting athlete performance (runners in particular) in a race of a specific starting lineup. For instance, if RunnerA, RunnerB, RunnerC, and RunnerD are all racing a 400 meter race, I want to best predict whether &lt;strong&gt;RunnerA&lt;/strong&gt; will beat &lt;strong&gt;RunnerB&lt;/strong&gt; based on past race result information (which I have at my disposal). However, I have many cases where &lt;strong&gt;RunnerA&lt;/strong&gt; has never raced against &lt;strong&gt;RunnerB&lt;/strong&gt;; yet I do have data showing &lt;strong&gt;RunnerA&lt;/strong&gt; has beat &lt;strong&gt;RunnerC&lt;/strong&gt; in the past, and &lt;strong&gt;RunnerC&lt;/strong&gt; has beat &lt;strong&gt;RunnerB&lt;/strong&gt; in the past. This logic extends deeper as well. So, it would seem that &lt;strong&gt;RunnerA&lt;/strong&gt; &lt;em&gt;should&lt;/em&gt; beat &lt;strong&gt;RunnerB&lt;/strong&gt;, given this information. My real concern is when it gets more complicated than this as I add more features (multiple runners, different distances, etc), and so I'm turing to ML algorithms to help my predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am having difficulty figuring out how to include this in my row data that I can train (after all, correctly formatting data is 99% of proper machine learning), and I am hoping that someone here might have thought along the same lines in the past and might be able to shed some light. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently trying to include &lt;strong&gt;RunnerX&lt;/strong&gt;-&lt;strong&gt;RunnerY&lt;/strong&gt; past race data by counting all the races that &lt;strong&gt;RunnerX&lt;/strong&gt; and &lt;strong&gt;RunnerY&lt;/strong&gt; have run together and normalizing them on a scale from &lt;code&gt;-1&lt;/code&gt; to &lt;code&gt;1&lt;/code&gt;; &lt;code&gt;-1&lt;/code&gt; indicating &lt;strong&gt;RunnerX&lt;/strong&gt; lost all past races against RunnerY; and &lt;code&gt;+1&lt;/code&gt; indicating that RunnerX has  won all past races against RunnerY; and &lt;code&gt;+1&lt;/code&gt; indicating. And &lt;code&gt;0&lt;/code&gt; indicating an equal number of wins and losses (or no past races against each other).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if &lt;strong&gt;RunnerA&lt;/strong&gt; is racing &lt;strong&gt;RunnerB&lt;/strong&gt;, and &lt;strong&gt;RunnerA&lt;/strong&gt; has beat &lt;strong&gt;RunnerB&lt;/strong&gt; in the past, then I want the algorithm to know that (denoted by a &lt;code&gt;+1&lt;/code&gt; on the &lt;strong&gt;RunnerB&lt;/strong&gt; column of row &lt;strong&gt;RunnerA&lt;/strong&gt;); same for vice versa. Taking it another step further, If &lt;strong&gt;RunnerA&lt;/strong&gt; is racing &lt;strong&gt;RunnerC&lt;/strong&gt; (but the two have never raced each other in the past), and &lt;strong&gt;RunnerA&lt;/strong&gt; has beat &lt;strong&gt;RunnerD&lt;/strong&gt; in a past race, and &lt;strong&gt;RunnerD&lt;/strong&gt; has beat &lt;strong&gt;RunnerC&lt;/strong&gt; in a past race, then I want the algorithm to learn that &lt;strong&gt;RunnerA&lt;/strong&gt; should beat &lt;strong&gt;RunnerC&lt;/strong&gt;. I say &lt;em&gt;beat&lt;/em&gt; here, but I mean an &quot;average beat&quot; for any &lt;strong&gt;RunnerX&lt;/strong&gt;-&lt;strong&gt;RunnerY&lt;/strong&gt; combinations when data for more than 1 past race is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have set my data up as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name     track   surface  distance  age    RunnerA   RunnerB   RunnerC   RunnerD&#xA;RunnerA  Home    2        400       11     0         1         0         1&#xA;RunnerC  Away    2        400       12     0         0         0         -1&#xA;RunnerD  Home    2        400       10     0         0         1         0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which shows that RunnerA has beat &lt;strong&gt;RunnerB&lt;/strong&gt; and &lt;strong&gt;RunnerD&lt;/strong&gt; in the past. RunnerC has lost to &lt;strong&gt;RunnerD&lt;/strong&gt;. And &lt;strong&gt;RunnerD&lt;/strong&gt; has beat &lt;strong&gt;RunnerC&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I don't really think this is a correct display of the information for an ML algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;From what I understand, ML data should be row independent. And this data isn't because row 1 (&lt;strong&gt;RunnerA&lt;/strong&gt;) has beat &lt;strong&gt;RunnerD&lt;/strong&gt;, yet the data indicating &lt;strong&gt;RunnerD&lt;/strong&gt; has beat &lt;strong&gt;RunnerC&lt;/strong&gt; is in row 3.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any ideas how I might be able to incorporate this past win-percentage-for-runner-pair-combination data??? I'm totally stuck here. I've read a lot about some algorithms that estimate the win loss by simply totaling win statistics, but those don't say anything about the actual probability of a particular runner to beat another particular runner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be super helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!!!&lt;/p&gt;&#xA;" OwnerUserId="8692" LastActivityDate="2015-03-24T15:35:16.227" Title="Best way to format data for supervised machine learning ranking predictions" Tags="|machine-learning|predictive-modeling|ranking|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5368" PostTypeId="2" ParentId="5357" CreationDate="2015-03-21T20:09:53.157" Score="6" Body="&lt;p&gt;In my opinion, ideally, to be a more &lt;strong&gt;well-rounded&lt;/strong&gt; professional, it would be nice to know &lt;strong&gt;at least&lt;/strong&gt; one programming language for the most popular &lt;strong&gt;programming paradigms&lt;/strong&gt; (&lt;em&gt;procedural&lt;/em&gt;, &lt;em&gt;object-oriented&lt;/em&gt;, &lt;em&gt;functional&lt;/em&gt;). Certainly, I consider &lt;em&gt;R&lt;/em&gt; and &lt;em&gt;Python&lt;/em&gt; as the two most popular programming languages and environments for &lt;em&gt;data science&lt;/em&gt; and, therefore, &lt;strong&gt;primary&lt;/strong&gt; data science tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Julia&lt;/em&gt; is impressive in certain aspects, but it tries to catch up with those two and establish itself as a major data science tool. However, I don't see this happening any time soon, simply due to &lt;em&gt;R/Python&lt;/em&gt;'s &lt;em&gt;popularity&lt;/em&gt;, very large &lt;em&gt;communities&lt;/em&gt; as well as enormous &lt;em&gt;ecosystems&lt;/em&gt; of existing and newly developed &lt;em&gt;packages/libraries&lt;/em&gt;, covering an very wide range of domains / fields of study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, many packages and libraries, focused on data science, ML and AI areas, are &lt;strong&gt;implemented&lt;/strong&gt; and/or provide &lt;strong&gt;APIs&lt;/strong&gt; in languages other than R or Python (for the proof, see &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;this curated list&lt;/a&gt; and &lt;a href=&quot;https://github.com/onurakpolat/awesome-bigdata&quot; rel=&quot;nofollow noreferrer&quot;&gt;this curated list&lt;/a&gt;, both of which are excellent and give a solid perspective about the variety in the field). This is especially true for &lt;em&gt;performance-oriented&lt;/em&gt; or &lt;em&gt;specialized&lt;/em&gt; software. For that software, I've seen projects with implementation and/or APIs mostly in Java, C and C++ (Java is especially popular in the &lt;strong&gt;big data&lt;/strong&gt; segment of data science - due to its closeness to &lt;em&gt;Hadoop&lt;/em&gt; and its ecosystem - and in the &lt;strong&gt;NLP&lt;/strong&gt; segment), but other options are available, albeit to a much more limited, domain-based, extent. Neither of these languages is a waste of time, however you have to &lt;strong&gt;prioritize&lt;/strong&gt; mastering any or all of them with your current work situation, projects and interests. So, to answer your question about viability of C/C++ (and Java), I would say that they are all &lt;strong&gt;viable&lt;/strong&gt;, however not as &lt;em&gt;primary&lt;/em&gt; data science tools, but as &lt;strong&gt;secondary&lt;/strong&gt; ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answering your questions on 1) &lt;em&gt;C&lt;/em&gt; as a &lt;em&gt;potential data science tool&lt;/em&gt; and 2) its &lt;em&gt;efficiency&lt;/em&gt;, I would say that: 1) while it's possible to use &lt;em&gt;C&lt;/em&gt; for data science, I &lt;strong&gt;would recommend against&lt;/strong&gt; doing it, because you'd have a very hard time finding corresponding libraries or, even more so, trying to implement corresponding algorithms by yourself; 2) you shouldn't worry about efficiency, as many performance-critical segments of code are implemented in low-level languages like C, plus, there are options to interface popular data science languages with, say, C (for example, &lt;code&gt;Rcpp&lt;/code&gt; package for integration R with C/C++: &lt;a href=&quot;http://dirk.eddelbuettel.com/code/rcpp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dirk.eddelbuettel.com/code/rcpp.html&lt;/a&gt;). This is in addition to simpler, but often rather effective, approaches to performance, such as consistent use of vectorization in R as well as using various parallel programming frameworks, packages and libraries. For R ecosystem examples, see &lt;a href=&quot;http://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRAN Task View &quot;High-Performance and Parallel Computing with R&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Speaking about &lt;em&gt;data science&lt;/em&gt;, I think that it makes quite a lot of sense to mention the importance of &lt;strong&gt;reproducible research&lt;/strong&gt; approach as well as the availability of various &lt;strong&gt;tools&lt;/strong&gt;, supporting this concept (for more details, please see &lt;a href=&quot;https://datascience.stackexchange.com/a/759/2452&quot;&gt;my relevant answer&lt;/a&gt;). I hope that my answer is helpful.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-03-21T20:09:53.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5369" PostTypeId="2" ParentId="5316" CreationDate="2015-03-21T20:58:48.757" Score="3" Body="&lt;p&gt;You need to &lt;strong&gt;analyze sentence structure&lt;/strong&gt; and extract corresponding &lt;em&gt;syntactic categories&lt;/em&gt; of interest (in this case, I think it would be &lt;strong&gt;noun phrase&lt;/strong&gt;, which is a &lt;em&gt;phrasal category&lt;/em&gt;). For details, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Syntactic_category&quot; rel=&quot;nofollow&quot;&gt;corresponding Wikipedia article&lt;/a&gt; and &lt;a href=&quot;http://www.nltk.org/book/ch08.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Analyzing Sentence Structure&quot;&lt;/a&gt; chapter of NLTK book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regard to available &lt;strong&gt;software&lt;/strong&gt; tools for implementing the above-mentioned approach and beyond, I would suggest to consider either &lt;a href=&quot;http://www.nltk.org&quot; rel=&quot;nofollow&quot;&gt;NLTK&lt;/a&gt; (if you prefer Python), or &lt;a href=&quot;http://www-nlp.stanford.edu/software&quot; rel=&quot;nofollow&quot;&gt;StanfordNLP software&lt;/a&gt; (if you prefer Java). For many other NLP frameworks, libraries and programming various languages support, see corresponding (NLP) sections in &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;nofollow&quot;&gt;this excellent curated list&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-03-21T20:58:48.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5370" PostTypeId="2" ParentId="5357" CreationDate="2015-03-21T21:58:34.687" Score="41" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;Or must I lose most of the efficiency gained by programming in C by calling on R scripts or other languages?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance-critical sections of your new algorithms, use R to build your analysis, import data, make plots, etc.&lt;/p&gt;&#xA;&lt;p&gt;If you want to go beyond R, I'd recommend learning Python. There are many libraries available such as &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; for machine learning algorithms or &lt;a href=&quot;http://pybrain.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyBrain&lt;/a&gt; for building Neural Networks etc. (and use pylab/&lt;a href=&quot;http://matplotlib.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;matplotlib&lt;/a&gt; for plotting and &lt;a href=&quot;http://ipython.org/notebook.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;iPython notebooks&lt;/a&gt; to develop your analyses). Again, C/C++ is useful to implement time critical algorithms as Python extensions.&lt;/p&gt;&#xA;" OwnerUserId="462" LastEditorUserId="-1" LastEditDate="2022-12-05T16:10:12.417" LastActivityDate="2022-12-05T16:10:12.417" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="5371" PostTypeId="1" CreationDate="2015-03-22T18:53:39.820" Score="5" ViewCount="5022" Body="&lt;p&gt;I think there are numerous posts regarding which one to use: R or Python. However, I'm curious about how their architecture differences yield differences in speed performance, &lt;strong&gt;not&lt;/strong&gt; which one to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://alstatr.blogspot.com/2014/01/python-and-r-is-python-really-faster.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; performs a small test between R and python to show that the (optimized) python code was 2x faster than R code.* And I've read in &lt;a href=&quot;https://datascience.stackexchange.com/questions/59/what-are-rs-memory-constraints&quot;&gt;this post&lt;/a&gt; that R tends to put everything in memory, which is why computations on large datasets is generally slow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what makes python's low level memory management so much different than R, which helps it yield these benchmarks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Though python was 2x faster in this test than R, I'm not saying that python is generally 2x faster than R.&lt;/p&gt;&#xA;" OwnerUserId="8774" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2023-12-09T18:31:33.003" Title="R and Python, memory differences yielding performance differences" Tags="|r|python|performance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5373" PostTypeId="1" CreationDate="2015-03-23T14:00:02.720" Score="3" ViewCount="2761" Body="&lt;p&gt;I have a large dataset with characters and 90000 intances and I have the error &lt;em&gt;ValueError: array is too big&lt;/em&gt; when I have the following code before the plot_kmeans_digits.py code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data2=list(csv.DictReader(open('C:\diabeticdata.csv', 'rU')))&#xA;vec = DictVectorizer()&#xA;data = vec.fit_transform(data2).toarray()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know how I can solve this error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="8710" LastActivityDate="2015-05-03T21:36:27.670" Title="sk-learn - ValueError: array is too big." Tags="|bigdata|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5374" PostTypeId="1" CreationDate="2015-03-23T15:48:18.683" Score="2" ViewCount="437" Body="&lt;p&gt;I am struggling with a conceptual problem related to feature scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume I am building a classifier (e.g., a NN) and let's assume I rely on future scaling for the input features of my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this context I will normalise the training set using its mean and its std and I would do the same with the testing set using the testing mean and std.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us also assume I succeed in building my classifier and I move to production where I try to classify new inputs.  However for such new inputs the mean and std are unknown! How can I scale them appropriately before processing with my model? May be I could use the mean and std from training+testing.....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really don't know which is the correct practice here....any hint?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help!&lt;/p&gt;&#xA;" OwnerUserId="8784" LastActivityDate="2015-03-23T21:48:30.313" Title="Feature scaling" Tags="|machine-learning|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5375" PostTypeId="1" CreationDate="2015-03-23T17:36:03.680" Score="2" ViewCount="197" Body="&lt;p&gt;I am investigating whether to use theano or caffe for convnets. I would like to know which one provides a better debug environment. In caffe, it seems you don't write any code just a config in protobuf - how do you debug the convnet then?&lt;/p&gt;&#xA;" OwnerUserId="8777" LastActivityDate="2015-03-23T17:36:03.680" Title="Which one is easier to debug between theano and caffe?" Tags="|deep-learning|" AnswerCount="0" CommentCount="1" ClosedDate="2015-03-24T01:43:40.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="5376" PostTypeId="2" ParentId="5374" CreationDate="2015-03-23T21:48:30.313" Score="5" Body="&lt;p&gt;You should apply the normalization only on your training dataset. Your test set should be kept completely separate and should be used only when your final model has been chosen. If you use include the testing set in the normalization, it can be seen as using the testing set in the training procedure. This is called data snooping.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should pre-process training dataset and use the obtained mean and std  when processing the testing set afterwards. Note that the testing dataset transformations will likely be imperfect (it will not have zero mean or unity standard deviation) but this testing dataset can safely be used since it has not affected any step of the learning process.&lt;/p&gt;&#xA;" OwnerUserId="8792" LastActivityDate="2015-03-23T21:48:30.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5379" PostTypeId="1" CreationDate="2015-03-24T06:01:50.617" Score="1" ViewCount="853" Body="&lt;p&gt;I have a complex dataset with more than 16M rows coming from pharmaceutical industry. Regarding the data, it is saved in a sql server with multiple (more than 400) relational tables. Data got several levels of hierachies like province, city, postal code, person, and antigens measures, etc. I would like to create many dashboards in order to observe the changes &amp;amp; trends happening. I can use Pentaho, R (shiny) or Tableau for this purpose. But the problem is data is so huge, and it take so long to process it with dashboard softwares. I have a choice of making cube and connect it to dashboard. My question here is whether there are  any other solutions that I can use instead of making a cube? I don't want to go through the hassle of making &amp;amp; maintaining a cube. I would like to use a software where I specify relationships between tables, so the aggregation/amalgamation happens smoothly and output processed tables that can connect  to dashboards. I hear Alteryx is one software that can do it for you (I haven't tried it myself, and it is an expensive one!). I understand this task needs two or more softwares/tools. Please share your input &amp;amp; experience. Please mention what tools do you use, size of your data, and how fast/efficient is the entire system, and other necessary details. &lt;/p&gt;&#xA;" OwnerUserId="7923" LastActivityDate="2015-11-04T11:15:29.960" Title="Tools to preprocess a big data for dashboards?" Tags="|bigdata|open-source|tableau|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5380" PostTypeId="1" AcceptedAnswerId="5418" CreationDate="2015-03-24T08:12:53.140" Score="2" ViewCount="194" Body="&lt;p&gt;I'm testing a SVD-based collaborative filter on my data set, in which the label, $r_{ij}$, is a real value from 0 to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like the many papers suggested, to have a better performance, instead of using $ \hat{R} = U \cdot V^T $ directly, I use $\hat{R} = \mu + B_u + B_v + U \cdot V^T $, where $\mu$ is the average rating, $B_u$ is the bias of user, and $B_v$ is the bias of item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, this model corresponds to a loss function:&#xA;$\min_{B_u, B_v, U, V}  = ||I\circ(R-\mu-Bu-B_v-U\cdot V^T)||_F^2 + \lambda (||B_u||_F^2 + ||B_v||_F^2 + ||U||_F^2 + ||V||_F^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where I is the masking matrix in which $I_{ij} = 1$ if $R_{ij}$ is known, and $||\cdot||_F$ is the frobenius norm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I solve this by gradient descent, it seems to work fine, and the test RMSE is 0.25. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I investigate the contribution of each part in predict function $\hat{R} = \mu + B_u + B_v + U \cdot V^T $, I notice that, $\mu$ is about 0.5, $b_u$ and $b_i$ are about $\pm0.3$, but the part of $ U \cdot V^T $ is quite small, normally about $\pm 0.01$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does this part contribute so small? Since this part is the actual part where the collaborative filter works, I expect it to contribute more in prediction.&lt;/p&gt;&#xA;" OwnerUserId="7867" LastEditorUserId="7867" LastEditDate="2015-03-28T07:26:13.863" LastActivityDate="2015-03-30T04:46:43.637" Title="In a SVD with user/video bias, why is the UV contribute so small?" Tags="|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5382" PostTypeId="2" ParentId="4949" CreationDate="2015-03-24T15:07:07.623" Score="0" Body="&lt;p&gt;Good to hear from the author. As he said, it appears to be possible through the Google Adwords Keyword Planner. Specifically, the &quot;Search for new keyword and ad group ideas&quot; option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am unable to get Google to actually generate his list, but I imagine it is just a matter of picking the correct seed words and settings. I get the same order-of-magnitude results when I directly search his terms (&quot;sexless marriage&quot;, &quot;unhappy marriage&quot;, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option for generating his list would be to search against a dictionary of negative terms (&quot; marriage&quot;) and then do some sorting and filtering. You could probably automate this through Google's API.&lt;/p&gt;&#xA;" OwnerUserId="7961" LastActivityDate="2015-03-24T15:07:07.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5383" PostTypeId="1" CreationDate="2015-03-24T15:25:02.270" Score="5" ViewCount="1059" Body="&lt;p&gt;I have one clarification - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First the definitions- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;User-based&lt;/strong&gt;: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Item-based&lt;/strong&gt;: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed offline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the question - in item based CF the similarity between items are tracked via user behavior - since user behavior is changing will it not impact the similarity between items?&lt;/p&gt;&#xA;" OwnerUserId="8800" LastEditorUserId="97" LastEditDate="2015-03-27T11:22:03.790" LastActivityDate="2015-03-27T11:22:03.790" Title="Recommendation - item based vs user based" Tags="|recommender-system|beginner|" AnswerCount="3" CommentCount="2" ClosedDate="2015-03-27T11:27:09.397" ContentLicense="CC BY-SA 3.0" />
  <row Id="5384" PostTypeId="2" ParentId="5367" CreationDate="2015-03-24T15:35:16.227" Score="1" Body="&lt;p&gt;This problem looks like a lot with the problem of ranking college football teams. I have never worked on this ranking problem, but I believe you can borrow some tools used there to build your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here goes a couple of references:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.colleyrankings.com/matrate.pdf&quot; rel=&quot;nofollow&quot;&gt;Colley Matrix Rankings&lt;/a&gt; - This was one of computer rankings used by the BCS. It is also the only one that shared their methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://public.gettysburg.edu/~cwessell/RankingPage/colley.pdf&quot; rel=&quot;nofollow&quot;&gt;An example of the Colley Matrix Rankings&lt;/a&gt; - An easy example to follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://umdrive.memphis.edu/ccrousse/public/MATH%207375/PERRON.pdf&quot; rel=&quot;nofollow&quot;&gt;The Perron-Frobenius Theorem and the Ranking of Football Teams&lt;/a&gt; - A well known reference that presents some ranking methods. This paper also shows how to assess the probability of winning a game based on the rankings of the teams.&lt;/p&gt;&#xA;" OwnerUserId="8792" LastActivityDate="2015-03-24T15:35:16.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5385" PostTypeId="2" ParentId="5357" CreationDate="2015-03-24T20:47:36.317" Score="5" Body="&lt;p&gt;As a data scientist the other languages (C++/Java) come in handy when you need incorporate machine learning into an existing production engine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://waffles.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Waffles&lt;/a&gt; is both a well-maintained C++ class library and command-line analysis package.  It's got supervised and unsupervised learning, tons of data manipulation tools, sparse data tools, and other things such as audio processing.  Since it's also a class library, you can extend it as you need.  Even if you are not the one developing the C++ engine (chances are you won't be), this will allow you to prototype, test, and hand something over to the developers.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most importantly, I believe my knowledge of C++ &amp;amp; Java really help me understand how Python and R work.  Any language is only used properly when you understand a little about what is going on underneath.  By learning the differences between languages you can learn to exploit the strengths of your main language.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For commercial applications with large data sets, Apache Spark - MLLib is important.  Here you can use Scala, Java, or Python.  &lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2015-11-09T15:27:41.420" LastActivityDate="2015-11-09T15:27:41.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5386" PostTypeId="1" CreationDate="2015-03-25T09:51:57.847" Score="0" ViewCount="558" Body="&lt;p&gt;Why would you want to decorrelated data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it to remove interaction between variables, thus allow simpler models to express interactions without having to compute x1*x2?&lt;/p&gt;&#xA;" OwnerUserId="8814" LastActivityDate="2015-04-24T22:50:59.517" Title="Identity covariance matrix, decorrelated data?" Tags="|machine-learning|dataset|statistics|correlation|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5387" PostTypeId="2" ParentId="5379" CreationDate="2015-03-25T10:16:41.007" Score="1" Body="&lt;p&gt;We have dashboards that show information about some processes, which have billions of rows in the database. It's not queried directly though, but instead from pre-aggregated data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have automated scripts running in database that populate the aggregated data tables specifically for dashboards. In most extreme cases the raw data is so large and comes in at such a high pace that there's a two-tier aggregation in place. First tier will de-normalize the data and 2nd tier will do the actual sum/count type of aggregation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you don't need two tools since you can do aggregation purely in database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit (answering question from comment): Our dashboards are in Tableau. Data is all in PostgreSQL databases. Automation is done with unix cronjob executing a database function. The function in turn queries which tasks it has to run and runs them. There is a few moving pieces to the whole system, but the architecture isn't too complicated. Write down your ideas and go talk to some architects / engineers - they will know the best way to approach this.&lt;/p&gt;&#xA;" OwnerUserId="587" LastEditorUserId="587" LastEditDate="2015-03-30T11:18:40.480" LastActivityDate="2015-03-30T11:18:40.480" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5388" PostTypeId="2" ParentId="5316" CreationDate="2015-03-25T12:52:43.580" Score="32" Body="&lt;p&gt;Shallow &lt;strong&gt;N&lt;/strong&gt;atural &lt;strong&gt;L&lt;/strong&gt;anguage &lt;strong&gt;P&lt;/strong&gt;rocessing technique can be used to extract concepts from sentence.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;-------------------------------------------&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Shallow NLP technique steps:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Convert the sentence to lowercase&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text (simply increasing n, model can be used to store more context)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assign a syntactic label (noun, verb etc.)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;-------------------------------------------&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Lets examine the results of applying the above steps to your given sentence &lt;code&gt;Complimentary gym access for two for the length of stay ($12 value per person per day)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;1-gram Results:&lt;/strong&gt; gym, access, length, stay, value, person, day&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Summary of step 1 through 4 of shallow NLP:&#xA;&#xA;1-gram          PoS_Tag   Stopword (Yes/No)?    PoS Tag Description&#xA;-------------------------------------------------------------------    &#xA;Complimentary   NNP                             Proper noun, singular&#xA;gym             NN                              Noun, singular or mass&#xA;access          NN                              Noun, singular or mass&#xA;for             IN         Yes                  Preposition or subordinating conjunction&#xA;two             CD                              Cardinal number&#xA;for             IN         Yes                  Preposition or subordinating conjunction&#xA;the             DT         Yes                  Determiner&#xA;length          NN                              Noun, singular or mass&#xA;of              IN         Yes                  Preposition or subordinating conjunction&#xA;stay            NN                              Noun, singular or mass&#xA;($12            CD                              Cardinal number&#xA;value           NN                              Noun, singular or mass&#xA;per             IN                              Preposition or subordinating conjunction&#xA;person          NN                              Noun, singular or mass&#xA;per             IN                              Preposition or subordinating conjunction&#xA;day)            NN                              Noun, singular or mass&#xA;&#xA;Step 4: Retaining only the Noun/Verbs we end up with gym, access, length, stay, value, person, day&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Lets increase n to store more context and remove stopwords.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;2-gram Results:&lt;/strong&gt; complimentary gym, gym access, length stay, stay value&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Summary of step 1 through 4 of shallow NLP:&#xA;&#xA;2-gram              Pos Tag&#xA;---------------------------&#xA;access two          NN CD&#xA;complimentary gym   NNP NN&#xA;gym access          NN NN&#xA;length stay         NN NN&#xA;per day             IN NN&#xA;per person          IN NN&#xA;person per          NN IN&#xA;stay value          NN NN&#xA;two length          CD NN&#xA;value per           NN IN&#xA;&#xA;Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym, gym access, length stay, stay value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;3-gram Results:&lt;/strong&gt; complimentary gym access, length stay value, person per day&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Summary of step 1 through 4 of shallow NLP:&#xA;&#xA;3-gram                      Pos Tag&#xA;-------------------------------------&#xA;access two length           NN CD NN&#xA;complimentary gym access    NNP NN NN&#xA;gym access two              NN NN CD&#xA;length stay value           NN NN NN&#xA;per person per              IN NN IN&#xA;person per day              NN IN NN&#xA;stay value per              NN NN IN&#xA;two length stay             CD NN NN&#xA;value per person            NN IN NN&#xA;&#xA;&#xA;Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym access, length stay value, person per day&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Things to remember:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Refer the &lt;a href=&quot;https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Penn tree bank&lt;/a&gt; to understand PoS tag description&lt;/li&gt;&#xA;&lt;li&gt;Depending on your &lt;em&gt;data and the business context&lt;/em&gt; you can decide the n value to extract n-grams from sentence&lt;/li&gt;&#xA;&lt;li&gt;Adding domain specific stop words would increase the quality of concept/theme extraction&lt;/li&gt;&#xA;&lt;li&gt;Deep NLP technique will give better results i.e., rather than n-gram, detect relationships within the sentences and represent/express as complex construction to retain the context. For additional info, see &lt;a href=&quot;https://stats.stackexchange.com/a/133680&quot;&gt;this&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;Tools:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort. Below is the sample R code I used for PoS tagging.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Sample R code:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Sys.setenv(JAVA_HOME='C:\\Program Files\\Java\\jre7') # for 32-bit version&#xA;library(rJava)&#xA;require(&amp;quot;openNLP&amp;quot;)&#xA;require(&amp;quot;NLP&amp;quot;)&#xA;&#xA;s &amp;lt;- paste(&amp;quot;Complimentary gym access for two for the length of stay $12 value per person per day&amp;quot;)&#xA;&#xA;tagPOS &amp;lt;-  function(x, ...) {&#xA;  s &amp;lt;- as.String(x)&#xA;    word_token_annotator &amp;lt;- Maxent_Word_Token_Annotator()&#xA;    a2 &amp;lt;- Annotation(1L, &amp;quot;sentence&amp;quot;, 1L, nchar(s))&#xA;    a2 &amp;lt;- annotate(s, word_token_annotator, a2)&#xA;    a3 &amp;lt;- annotate(s, Maxent_POS_Tag_Annotator(), a2)&#xA;    a3w &amp;lt;- a3[a3&lt;span class=&quot;math-container&quot;&gt;$type == &quot;word&quot;]&#xA;    POStags &amp;lt;- unlist(lapply(a3w$&lt;/span&gt;features, `[[`, &amp;quot;POS&amp;quot;))&#xA;    POStagged &amp;lt;- paste(sprintf(&amp;quot;%s/%s&amp;quot;, s[a3w], POStags), collapse = &amp;quot; &amp;quot;)&#xA;    list(POStagged = POStagged, POStags = POStags)&#xA;  }&#xA;  &#xA;  tagged_str &amp;lt;-  tagPOS(s)&#xA;  tagged_str&#xA;&#xA;#&lt;span class=&quot;math-container&quot;&gt;$POStagged&#xA;#[1] &quot;Complimentary/NNP gym/NN access/NN for/IN two/CD for/IN the/DT length/NN of/IN stay/NN $&lt;/span&gt;/&lt;span class=&quot;math-container&quot;&gt;$ 12/CD value/NN per/IN     person/NN per/IN day/NN&quot;&#xA;#&#xA;#$&lt;/span&gt;POStags&#xA;#[1] &amp;quot;NNP&amp;quot; &amp;quot;NN&amp;quot;  &amp;quot;NN&amp;quot;  &amp;quot;IN&amp;quot;  &amp;quot;CD&amp;quot;  &amp;quot;IN&amp;quot;  &amp;quot;DT&amp;quot;  &amp;quot;NN&amp;quot;  &amp;quot;IN&amp;quot;  &amp;quot;NN&amp;quot;  &amp;quot;$&amp;quot;   &amp;quot;CD&amp;quot; &#xA;#[13] &amp;quot;NN&amp;quot;  &amp;quot;IN&amp;quot;  &amp;quot;NN&amp;quot;  &amp;quot;IN&amp;quot;  &amp;quot;NN&amp;quot; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Additional readings on Shallow &amp;amp; Deep NLP:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.dfki.de/%7Efeiyu/WHIES.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Integrating Shallow and Deep NLP for Information Extraction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8465" LastEditorUserId="98307" LastEditDate="2020-08-20T18:39:54.567" LastActivityDate="2020-08-20T18:39:54.567" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="5389" PostTypeId="2" ParentId="694" CreationDate="2015-03-25T19:45:31.320" Score="41" Body="&lt;p&gt;&lt;a href=&quot;http://tensorflow.org/&quot;&gt;Tensor Flow&lt;/a&gt; (&lt;a href=&quot;http://www.tensorflow.org/api_docs/python/nn.html#neural-network&quot;&gt;docs&lt;/a&gt;) by Google is another nice framework which has automatic differentiation. I've written down some &lt;a href=&quot;http://martin-thoma.com/tensor-flow-quick/&quot;&gt;quick thoughts about Google Tensor Flow&lt;/a&gt; on my blog, together with the MNIST example which they have in their tutorial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also: My &lt;a href=&quot;https://martin-thoma.com/tf-xor-tutorial/&quot;&gt;Tensorflow XOR tutorial&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/benanne/Lasagne&quot;&gt;Lasagne&lt;/a&gt; (&lt;a href=&quot;http://lasagne.readthedocs.org/en/latest/&quot;&gt;docs&lt;/a&gt;) is very nice, as it uses theano (→ you can use the GPU) and makes it simpler to use. The author of lasagne won the Kaggle Galaxy challenge, as far as I know. It is nice with &lt;a href=&quot;http://pythonhosted.org/nolearn/&quot;&gt;nolearn&lt;/a&gt;. Here is an MNIST example network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;import lasagne&#xA;from lasagne import layers&#xA;from lasagne.updates import nesterov_momentum&#xA;from nolearn.lasagne import NeuralNet&#xA;&#xA;import sys&#xA;import os&#xA;import gzip&#xA;import pickle&#xA;import numpy&#xA;&#xA;&#xA;PY2 = sys.version_info[0] == 2&#xA;&#xA;if PY2:&#xA;    from urllib import urlretrieve&#xA;&#xA;    def pickle_load(f, encoding):&#xA;        return pickle.load(f)&#xA;else:&#xA;    from urllib.request import urlretrieve&#xA;&#xA;    def pickle_load(f, encoding):&#xA;        return pickle.load(f, encoding=encoding)&#xA;&#xA;DATA_URL = 'http://deeplearning.net/data/mnist/mnist.pkl.gz'&#xA;DATA_FILENAME = 'mnist.pkl.gz'&#xA;&#xA;&#xA;def _load_data(url=DATA_URL, filename=DATA_FILENAME):&#xA;    &quot;&quot;&quot;Load data from `url` and store the result in `filename`.&quot;&quot;&quot;&#xA;    if not os.path.exists(filename):&#xA;        print(&quot;Downloading MNIST dataset&quot;)&#xA;        urlretrieve(url, filename)&#xA;&#xA;    with gzip.open(filename, 'rb') as f:&#xA;        return pickle_load(f, encoding='latin-1')&#xA;&#xA;&#xA;def load_data():&#xA;    &quot;&quot;&quot;Get data with labels, split into training, validation and test set.&quot;&quot;&quot;&#xA;    data = _load_data()&#xA;    X_train, y_train = data[0]&#xA;    X_valid, y_valid = data[1]&#xA;    X_test, y_test = data[2]&#xA;    y_train = numpy.asarray(y_train, dtype=numpy.int32)&#xA;    y_valid = numpy.asarray(y_valid, dtype=numpy.int32)&#xA;    y_test = numpy.asarray(y_test, dtype=numpy.int32)&#xA;&#xA;    return dict(&#xA;        X_train=X_train,&#xA;        y_train=y_train,&#xA;        X_valid=X_valid,&#xA;        y_valid=y_valid,&#xA;        X_test=X_test,&#xA;        y_test=y_test,&#xA;        num_examples_train=X_train.shape[0],&#xA;        num_examples_valid=X_valid.shape[0],&#xA;        num_examples_test=X_test.shape[0],&#xA;        input_dim=X_train.shape[1],&#xA;        output_dim=10,&#xA;    )&#xA;&#xA;&#xA;def nn_example(data):&#xA;    net1 = NeuralNet(&#xA;        layers=[('input', layers.InputLayer),&#xA;                ('hidden', layers.DenseLayer),&#xA;                ('output', layers.DenseLayer),&#xA;                ],&#xA;        # layer parameters:&#xA;        input_shape=(None, 28*28),&#xA;        hidden_num_units=100,  # number of units in 'hidden' layer&#xA;        output_nonlinearity=lasagne.nonlinearities.softmax,&#xA;        output_num_units=10,  # 10 target values for the digits 0, 1, 2, ..., 9&#xA;&#xA;        # optimization method:&#xA;        update=nesterov_momentum,&#xA;        update_learning_rate=0.01,&#xA;        update_momentum=0.9,&#xA;&#xA;        max_epochs=10,&#xA;        verbose=1,&#xA;        )&#xA;&#xA;    # Train the network&#xA;    net1.fit(data['X_train'], data['y_train'])&#xA;&#xA;    # Try the network on new data&#xA;    print(&quot;Feature vector (100-110): %s&quot; % data['X_test'][0][100:110])&#xA;    print(&quot;Label: %s&quot; % str(data['y_test'][0]))&#xA;    print(&quot;Predicted: %s&quot; % str(net1.predict([data['X_test'][0]])))&#xA;&#xA;&#xA;def main():&#xA;    data = load_data()&#xA;    print(&quot;Got %i testing datasets.&quot; % len(data['X_train']))&#xA;    nn_example(data)&#xA;&#xA;if __name__ == '__main__':&#xA;    main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://caffe.berkeleyvision.org/&quot;&gt;Caffe&lt;/a&gt; is a C++ library, but has Python bindings. You can do most stuff by configuration files (prototxt). It has a lot of options and can also make use of the GPU.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-08-08T17:11:56.413" LastActivityDate="2016-08-08T17:11:56.413" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="5391" PostTypeId="2" ParentId="5386" CreationDate="2015-03-25T21:33:45.957" Score="1" Body="&lt;p&gt;Having only taken a few courses on this stuff, I'm going to offer my understanding on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main reason seems to be simplifying, as you assume. It manifests in different areas though. One is mathematical - if your co-variance matrix is identity matrix then the math becomes much easier. The one directly following from this is computational - inverting a matrix is very costly, but identity matrix is its own inverse, so that's perfect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another part of the simplicity is definitely the ease of interpretation. Independent variables are easy to explain to a non-technical person, but as soon as you get into covariances, that goes out the window.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-03-25T21:33:45.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5394" PostTypeId="2" ParentId="5380" CreationDate="2015-03-26T03:15:05.417" Score="1" Body="&lt;p&gt;I think that depends on the pattern of your data, its skewness and sparsity. &#xA;Because it would press these two bias term, which is designed to detect the difference with global mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you can plot mean and std of the user's bias: ($R_u$ - global_mean) then make a histogram of std. Another way, just make up a rating matrix $R'$ according to the $\mu + B_u + B_v$ you've got. And run the solver again and see what happens. These parts play its own role, even your change on global_mean would make a difference to the final RMSE, I think the role of that latent factors are born to capture the &lt;strong&gt;&lt;em&gt;REST&lt;/em&gt;&lt;/strong&gt; non-linear pattern in user &amp;amp; item's interaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as for your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why does this part contributes so small?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Again, you can simply using $R = UV^T$, in this time, I think it contributes quite a lot, right? The contribution of each part is hard to say.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2015-03-26T03:15:05.417" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5395" PostTypeId="2" ParentId="5383" CreationDate="2015-03-26T03:20:39.833" Score="0" Body="&lt;p&gt;Sure, as @LaruiK said. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've notice the factor which would change the similarity measure of the 'co-view' data, a simple way it to integrate a time decay function into your model.&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2015-03-26T03:20:39.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5396" PostTypeId="2" ParentId="5079" CreationDate="2015-03-26T03:39:33.163" Score="1" Body="&lt;p&gt;Now you face the problem of scalability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you ever heard of random projection? Here is my suggestion:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let's say $S$ is your item-item relation matrix, its dimension is too large to decompose, now you can multiply an smaller matrix $U$(randomly generated) and get $R = SU$. And $R$ is small also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then just do SVD of $R$, if $U$ is invertible, and the difference (l2-norm or any norm you like) between $R$ and $S$ is small, it might helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second issue:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NaN value&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If it's dense, try to complete the matrix(using mean?).&lt;/p&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2015-03-26T03:39:33.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5397" PostTypeId="2" ParentId="641" CreationDate="2015-03-26T12:44:44.903" Score="6" Body="&lt;p&gt;As I understand it, these are the properties that you're seeking in a sample dataset:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Text data&lt;/li&gt;&#xA;&lt;li&gt;It should be informal, i.e. have typos, slang, and basically something not professionally edited &lt;/li&gt;&#xA;&lt;li&gt;Something other than Twitter (I don't blame you, Twitter is a useful yet way overused example datasource in text mining)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here are some recommendations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Emails from the &lt;a href=&quot;https://spamassassin.apache.org/publiccorpus/&quot; rel=&quot;noreferrer&quot;&gt;SpamAssassin corpus&lt;/a&gt; -- note that both &quot;ham&quot; (non-spam) and spam datasets are available&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/microblogPCU&quot; rel=&quot;noreferrer&quot;&gt;microblogPCU data set&lt;/a&gt; from UCI, which is data scraped from the microblogs of Sina Weibo users -- note, the raw text data is a mix of Chinese and English (you could perform machine translation of the Chinese, filter to only English, or use it as-is)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Amazon+Commerce+reviews+set&quot; rel=&quot;noreferrer&quot;&gt;Amazon Commerce reviews&lt;/a&gt; dataset from UCI&lt;/li&gt;&#xA;&lt;li&gt;Within the &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Bag+of+Words&quot; rel=&quot;noreferrer&quot;&gt;bag-o-words dataset&lt;/a&gt;, try using the Enron emails&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups&quot; rel=&quot;noreferrer&quot;&gt;Twenty Newsgroups&lt;/a&gt; dataset&lt;/li&gt;&#xA;&lt;li&gt;This nice collection of &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection&quot; rel=&quot;noreferrer&quot;&gt;SMS spam&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;You can always scrape (extract) your own text data from the Internet; I'm not sure which language or statistical package you're using, but XPath-based packages are available in R (&lt;a href=&quot;https://github.com/hadley/rvest&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;rvest&lt;/code&gt;&lt;/a&gt;, &lt;code&gt;scrapeR&lt;/code&gt;, etc) and Python to accomplish this&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2015-03-26T12:44:44.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5399" PostTypeId="2" ParentId="5326" CreationDate="2015-03-26T18:12:19.117" Score="2" Body="&lt;p&gt;Sorry for the short answer, but if you use shiny with rmarkdown, it can be done as shown in this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/29253481/data-specific-selectinput-choices-in-rmd-shiny/29255723#29255723&quot;&gt;https://stackoverflow.com/questions/29253481/data-specific-selectinput-choices-in-rmd-shiny/29255723#29255723&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a reactive function that reads the file in!&lt;/p&gt;&#xA;" OwnerUserId="8830" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-03-26T18:12:19.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5400" PostTypeId="2" ParentId="5383" CreationDate="2015-03-26T20:00:57.663" Score="4" Body="&lt;p&gt;It depends on what data you are using to calculate similarity between your items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using data from a user interaction with an item - like viewing a web page or buying an item - then each time there is a user interaction with an item, the similarity between it and other items will change. If you are doing this calculation offline, then your model won't take that change into account until the next time the data set is updated and your recommendations are recalculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are using item meta-data to calculate similarity, then user behavior won't make any difference. For example, if you took the number of shared tags between two items, or the number of shared inbound links, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have always used user browsing data to build content recommendation engines, but it is certainly possible to calculate similarity in many different ways. Have fun!&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-03-26T20:00:57.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5401" PostTypeId="1" AcceptedAnswerId="5402" CreationDate="2015-03-27T07:20:28.740" Score="0" ViewCount="1223" Body="&lt;p&gt;I am a beginner studying &lt;strong&gt;social network analysis&lt;/strong&gt;.&#xA;I installed python 3 just 2 weeks ago.&#xA;There are a lot of books for python and social network analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't choose one of them.&#xA;I found one book named &quot;Mining the Social Web (Analyzing Data from Facebook, Twitter, Linkedln, and Other Social Media Sites)&quot; written by Matthhew A. Russell.&#xA;This book looks very interesting and fits in my purpose, but it is based on python 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any good books with &lt;strong&gt;python 3&lt;/strong&gt;? I usually use &lt;strong&gt;Twitter, Facebook, or Blog data&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, could you recommend any good book for &lt;strong&gt;nodeXL and UCINET?&lt;/strong&gt;  &lt;/p&gt;&#xA;" OwnerUserId="8839" LastActivityDate="2015-03-27T08:00:09.137" Title="Which book is the best for introduction to analysis social network using python3" Tags="|data-mining|python|social-network-analysis|" AnswerCount="1" CommentCount="3" ClosedDate="2015-03-27T11:28:19.117" ContentLicense="CC BY-SA 3.0" />
  <row Id="5402" PostTypeId="2" ParentId="5401" CreationDate="2015-03-27T08:00:09.137" Score="3" Body="&lt;p&gt;Russell's book is fine. You might also like &lt;a href=&quot;http://shop.oreilly.com/product/0636920020424.do&quot; rel=&quot;nofollow&quot;&gt;Social Network Analysis for Startups&lt;/a&gt;. All the examples are in python. You can do all your analysis in that using packages like &lt;code&gt;networkx&lt;/code&gt;. NodeXL is for the Excel crowd. Definitely not the ideal tool for the job; I would shy away from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious book for NodeXL is &lt;a href=&quot;http://www.elsevier.com/books/analyzing-social-media-networks-with-nodexl/hansen/978-0-12-382229-1&quot; rel=&quot;nofollow&quot;&gt;Analyzing Social Media Networks with NodeXL&lt;/a&gt;, which is written by the authors of NodeXL.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-03-27T08:00:09.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5403" PostTypeId="2" ParentId="5383" CreationDate="2015-03-27T08:02:29.943" Score="1" Body="&lt;p&gt;What you are asking for is usually called &lt;strong&gt;basket analysis&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you should get maximum value from the data you have by using both of them: user &amp;amp; items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you said about &quot;item-based&quot; approach means something like recommendation based on item tags or categories. It isn't a recommendation system in the full sense, because it uses &lt;strong&gt;your&lt;/strong&gt; categorization/tagging. In other words, you will never place diapers and beer into a single category, but it is still legendary buying pattern :). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Item-based&lt;/strong&gt; data is used typically in basket analysis algorithms, such as frequent pattern mining. In couple words: you search for most frequent item sets (items bought together or coherently) and make suggestions based on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;User-based&lt;/strong&gt; data shouldn't be ignored either. Clustering approach works here: you can find some groups of customers which buying attitude to item sets (found above) is different from the average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find more at statistics stack exchage site. &#xA;Questions like &lt;strong&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/95351/what-is-the-best-way-to-analyze-additional-attributes-in-a-market-basket-analysi&quot;&gt;this&lt;/a&gt;&lt;/strong&gt;.&#xA;And &lt;strong&gt;&lt;a href=&quot;https://stats.stackexchange.com/search?q=basket%20analysis&quot;&gt;more&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-03-27T08:02:29.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5404" PostTypeId="1" AcceptedAnswerId="5407" CreationDate="2015-03-27T08:41:13.680" Score="9" ViewCount="7543" Body="&lt;p&gt;This may be too broad of a question with heavy opinions, but I really am finding it hard to seek information about running various algorithms using SQL Server Analysis Service Data Mining projects versus using R. This is mainly because all the data science guys I work with don't have any idea about SSAS because no one seems to use it. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Database Guy&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I start, let me clarify. I am a database guy and not a data scientist. I work with people who are data scientist who mainly use R. I assist these guys with creating large data sets where they can analyze and crunch data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My objective here is to leverage a tool that came with SQL Server that no one is really leveraging because no one seems to have a clue about how it works in comparison to other methods and tools such as R, SAS, SSPS and so forth in my camp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SSAS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have never really used SQL Server Analysis Services (SSAS) outside of creating OLAP cubes. Those who know SSAS, you can also perform data mining tasks on cubes or directly on the data in SQL Server. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SSAS Data Mining comes with a range of algorithm types:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Classification algorithms&lt;/strong&gt; predict one or more discrete variables,&#xA;based on the other attributes in the dataset.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Regression algorithms&lt;/strong&gt; predict one or more continuous variables, such&#xA;as profit or loss, based on other attributes in the dataset.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Segmentation algorithms&lt;/strong&gt; divide data into groups, or clusters, of&#xA;items that have similar properties.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Association algorithms&lt;/strong&gt; find correlations between different attributes&#xA;in a dataset. The most common application of this kind of algorithm&#xA;is for creating association rules, which can be used in a market&#xA;basket analysis.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Sequence analysis algorithms&lt;/strong&gt; summarize frequent sequences or episodes&#xA;in data, such as a Web path flow.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Predicting Discrete Columns&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these different algorithm options, I can start making general predictions from the data such as finding out simply who is going to buy a bike based on a predictable column, Bike Buyers, against an input column, Age. The histogram shows that the age of a person helps distinguish whether that person will purchase a bicycle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/X64Y1.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Predicting Continuous Columns&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the Microsoft Decision Trees algorithm builds a tree based on a continuous predictable column, each node contains a regression formula. A split occurs at a point of non-linearity in the regression formula. For example, consider the following diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/6MkTt.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Comparison&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With some of that said, it seems I can run a range of algorithms on the data and also have various functions available to me in SSAS to run against the data. It also seems I can develop my own algorithms in Visual Studio and deploy them to SSAS (if I'm not mistaken).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what am I missing here in regards to languages and tools from R? Is it just that they have more flexibility to deploy and edit complex algorithms versus SSAS etc?&lt;/p&gt;&#xA;" OwnerUserId="8842" LastEditorUserId="8842" LastEditDate="2015-03-27T08:47:18.077" LastActivityDate="2015-03-27T11:22:23.797" Title="How does SQL Server Analysis Services compare to R?" Tags="|data-mining|r|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5407" PostTypeId="2" ParentId="5404" CreationDate="2015-03-27T11:22:23.797" Score="2" Body="&lt;p&gt;In my opinion, it seems that SSAS makes &lt;strong&gt;more sense&lt;/strong&gt; for someone who:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;has significantly invested in Microsoft's &lt;em&gt;technology stack&lt;/em&gt; and &lt;em&gt;platform&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;prefer point-and-click &lt;em&gt;interface&lt;/em&gt; (GUI) to command line;&lt;/li&gt;&#xA;&lt;li&gt;focus on &lt;em&gt;data warehousing&lt;/em&gt; (OLAP cubes, etc.);&lt;/li&gt;&#xA;&lt;li&gt;has limited needs in terms of &lt;em&gt;statistical methods and algorithms variety&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;has limited needs in &lt;em&gt;cross-language integration&lt;/em&gt;;&lt;/li&gt;&#xA;&lt;li&gt;doesn't care much about &lt;em&gt;openness&lt;/em&gt;, &lt;em&gt;cross-platform integration&lt;/em&gt; and &lt;em&gt;vendor lock-in&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find useful &lt;a href=&quot;http://blog.samibadawi.com/2010/04/r-rapidminer-statistica-ssas-or-weka.html&quot; rel=&quot;nofollow&quot;&gt;this blog post&lt;/a&gt; by &lt;a href=&quot;http://twitter.com/Sami_Badawi&quot; rel=&quot;nofollow&quot;&gt;Sami Badawi&lt;/a&gt;. However, note that the post is &lt;em&gt;not recent&lt;/em&gt;, so some information might be outdated. Plus, the post contains an &lt;em&gt;initial&lt;/em&gt; review, which might be not very accurate or comprehensive. If you're thinking about data science, while considering staying within Microsoft ecosystem, I suggest you to take a look at Microsoft's own machine learning platform Azure ML. &lt;a href=&quot;https://dbolab.com/microsoft-azure-ml-vs-ssas-mining-structures&quot; rel=&quot;nofollow&quot;&gt;This blog post&lt;/a&gt; presents a brief comparison of (early) Azure ML and SSAS.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-03-27T11:22:23.797" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="5408" PostTypeId="1" AcceptedAnswerId="5636" CreationDate="2015-03-27T12:36:11.577" Score="2" ViewCount="325" Body="&lt;p&gt;I'm using k-means clustering to processes running on machines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset sample : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;machine name, process&#xA;m1,java&#xA;m2,tomcat&#xA;m1,word&#xA;m3,excel&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Build a matrix of associated counts : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   java,tomcat,word,excel&#xA;m1,1,0,1,0&#xA;m2,0,1,0,0&#xA;m3,0,0,0,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I then run k-means against this dataset (have tried Euclidean and Manhattan distance functions) &#xA;The dataset is extremely sparse which I think is causing the generated clusters to not make much sense as many machines get grouped into the same cluster(as they are very similar) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to achieve clusters where each cluster contains approx equal number of points ? Or perhaps this is not possible due to the sparseness of the data and instead I should try to cluster on a different attributes of dataset ?&lt;/p&gt;&#xA;" OwnerUserId="237" LastEditorUserId="237" LastEditDate="2015-03-27T14:16:20.760" LastActivityDate="2015-04-28T15:29:24.093" Title="Clustering not producing even clusters" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5409" PostTypeId="1" CreationDate="2015-03-27T15:31:49.090" Score="1" ViewCount="211" Body="&lt;p&gt;I am doing some research on Logistic regression and SVM using different parameters using HOG features. I am facing a bit of problem while understanding each classifier with combination of different parameters and different HOG features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My findings and confusions are given below, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;For Hog:    orientations=18, pixelsPerCell=(6,6), cellsPerBlock=(1,1)&#xA;Classifier: SVC(C=1000.0,  gamma=0.1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=210,Wrong Prediction=6&#xA;Classifier: SVC(C=100.0,  gamma=0.1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=210,Wrong Prediction=6&#xA;Classifier: SVC(C=1.0,  gamma=0.1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=209,Wrong Prediction=7&#xA;    , &#xA;&#xA;For Hog:    orientations=9, pixelsPerCell=(6,6), cellsPerBlock=(1,1)&#xA;Classifier: SVC(C=1000.0,  gamma=0.1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=211,Wrong Prediction=5&#xA;Classifier: SVC(C=100.0,  gamma=0.1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=211,Wrong Prediction=5&#xA;Classifier: SVC(C=1.0,  gamma=0.1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=210,Wrong Prediction=6            &#xA;&#xA;-------------------------------------------------------------------------            &#xA;&#xA;For Hog:    orientations=9, pixelsPerCell=(9,9), cellsPerBlock=(3,3)&#xA;Classifier: SVC(C=1000.0,  gamma=1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=100.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=1.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=213,Wrong Prediction=3&#xA;&#xA;&#xA;&#xA;For Hog:    orientations=9, pixelsPerCell=(6,6), cellsPerBlock=(3,3)&#xA;Classifier: SVC(C=1000.0,  gamma=1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=100.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=210,Wrong Prediction=6&#xA;Classifier: SVC(C=1.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;&#xA;&#xA;For Hog:    orientations=18, pixelsPerCell=(9,9), cellsPerBlock=(3,3)&#xA;Classifier: SVC(C=1000.0,  gamma=1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=100.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=1.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=3&#xA;&#xA;&#xA;For Hog:    orientations=18, pixelsPerCell=(6,6), cellsPerBlock=(3,3)&#xA;Classifier: SVC(C=1000.0,  gamma=1, kernel='rbf'),&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=100.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;Classifier: SVC(C=1.0,  gamma=1, kernel='rbf')&#xA;Output:     Total dataset=216, Correct prediction=212,Wrong Prediction=4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the above results we can conclude that &quot;C&quot; of SVM and orientation of HOG doesn't matter much for the combinations, so we do not have the issue of highbias or variance and bin size. What matters is the value of gamma and combination of HOG with &quot;cellsPerBlock&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;From my analysis, I found&#xA;1. Using &quot;rbf&quot; SVM with HOG &quot;cellsPerBlock=(1,1)&quot; doesn't give proper    outcome.&#xA;2. The &quot;rbf&quot; SVM with HOG &quot;cellsPerBlock=(1,1)&quot; works best with gamma=0.1&#xA;3. The &quot;rbf&quot; SVM with HOG &quot;cellsPerBlock=(3,3)&quot; and different combination of &quot;pixelspercells&quot; works best with gamma=1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've also tried pixelspercell = (3,3) but all the different combinations of &quot;rbf&quot; kernels fails to classify properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&quot; The problem I face is understanding this discrepancy&quot;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Confusion even eggravates when I use Logistic regression for the Combinations of HOG surprisingly Regularized Logistic Regression works better with cellsPerBlock=(1,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below are my findings with 2nd order Regularized Logistic Regression: I used 0.9 as my threshold level, i.e below 0.9 would be classifies as 0 and above 0.9 would be classified as 1&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Q:   Why did I put 0.9 as threshold- &#xA;Sol: Because the test data that had the class(labels)=1, 99% of them output probability &amp;gt;0.9&#xA;     However the test data that had the class(labels)=0, many instances outputprobability &amp;gt;0.5&#xA;   So inorder to filter out many test data instances that gave probability &amp;gt;0.5 A threshold of 0.9 was used.&#xA;&#xA;1.  orientations=9,    &#xA;pixelsPerCell=(6,6)&#xA;cellsPerBlock=(3,3)&#xA;Output:     Total dataset=216, Correct prediction=187, Wrong Prediction=29&#xA;&#xA;2.  orientations=9,    &#xA;pixelsPerCell=(9,9)&#xA;    cellsPerBlock=(3,3)&#xA;    Output:     Total dataset=216, Correct prediction=202, Wrong Prediction=14&#xA;&#xA;3.  orientations=9,    &#xA;pixelsPerCell=(9,9)&#xA;    cellsPerBlock=(1,1)&#xA;    Output:     Total dataset=216, Correct prediction=211, Wrong Prediction=5&#xA;&#xA;4.  orientations=9,    &#xA;pixelsPerCell=(6,6)&#xA;    cellsPerBlock=(1,1)&#xA;    Output:     Total dataset=216, Correct prediction=206, Wrong Prediction=10&#xA;&#xA;5.  orientations=9,    &#xA;pixelsPerCell=(3,3)&#xA;    cellsPerBlock=(1,1)&#xA;    Output:     Total dataset=216, Correct prediction=203, Wrong Prediction=13&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8848" LastActivityDate="2015-03-27T15:31:49.090" Title="Huge discrepancies in Logistic Regression and SVM using HOG features to identify an Object" Tags="|machine-learning|r|svm|logistic-regression|hog|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5410" PostTypeId="2" ParentId="1059" CreationDate="2015-03-27T15:41:42.213" Score="2" Body="&lt;p&gt;A couple of ways to improve your design:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Consider a different normalization: The sigmoid function will attenuate large moves.  It is likely precisely these large non-linear moves that attracted you to using neural networks in the first place.  Why remove them?  A simple &lt;code&gt;whitening&lt;/code&gt; of the data may be better&lt;/li&gt;&#xA;&lt;li&gt;As pointed out by Nima, your model can only predict what is knowable from the data you give it.  If you are only fitting on data using historical prices, it will only give you what is predictable from them.  Things such as news events / earnings surprises / holidays / option market flows will not be fed into your model.  Consider adding these series as well.&lt;/li&gt;&#xA;&lt;li&gt;Lots more historical data.  Neural networks often require very large sample sizes because they are trying to estimate a very large parameter space.  Although, more data doesn't always imply more information, this will still likely help.&lt;/li&gt;&#xA;&lt;li&gt;Experiment with different network architectures.  The number of layers / size of the layers / different gradient decent algorithms / different activation functions / dropout etc..&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8849" LastEditorUserId="8849" LastEditDate="2015-04-08T14:15:39.407" LastActivityDate="2015-04-08T14:15:39.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5411" PostTypeId="2" ParentId="2667" CreationDate="2015-03-27T15:45:50.857" Score="1" Body="&lt;p&gt;I have had success using &lt;a href=&quot;https://bitbucket.org/wellaware/mogwai&quot; rel=&quot;nofollow&quot;&gt;Mogwai Python Library&lt;/a&gt;  which is more actively maintained than Bulbs at this point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I agree that this belongs on stack overflow.  &lt;/p&gt;&#xA;" OwnerUserId="8849" LastActivityDate="2015-03-27T15:45:50.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5413" PostTypeId="1" AcceptedAnswerId="5414" CreationDate="2015-03-28T22:35:01.270" Score="3" ViewCount="2795" Body="&lt;p&gt;I've only worked with ML with .csv formats. I've worked with image formats too but only premade imagesets (MNIST,etc). If I were to create an imageset from scratch, how are the class labels typically formated? Would I have to manually title the image of a jpeg? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;Jeremy&lt;/p&gt;&#xA;" OwnerUserId="8860" LastActivityDate="2015-03-29T11:33:26.277" Title="How to create and format an image dataset from scratch for machine learning?" Tags="|machine-learning|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5414" PostTypeId="2" ParentId="5413" CreationDate="2015-03-29T11:33:26.277" Score="0" Body="&lt;p&gt;Looking at existing challenges around and their data format (for example &lt;a href=&quot;http://www.kaggle.com/c/datasciencebowl/data&quot; rel=&quot;nofollow&quot;&gt;http://www.kaggle.com/c/datasciencebowl/data&lt;/a&gt;) I would say put the images in a folder per class. You can use the file names for the index.&lt;/p&gt;&#xA;" OwnerUserId="8865" LastActivityDate="2015-03-29T11:33:26.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5417" PostTypeId="1" CreationDate="2015-03-30T04:35:58.667" Score="4" ViewCount="1895" Body="&lt;p&gt;I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Storm alone and implement streaming naive bayes on my own in storm topology.&lt;/li&gt;&#xA;&lt;li&gt;Storm + TridentML&lt;/li&gt;&#xA;&lt;li&gt;Storm + SAMOA&lt;/li&gt;&#xA;&lt;li&gt;Spark Streaming + MLlib&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What is the best framework set to choose and start working on. Any suggestion will be of great help.&lt;/p&gt;&#xA;" OwnerUserId="8869" LastActivityDate="2015-11-13T07:17:47.380" Title="Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib" Tags="|machine-learning|classification|distributed|data-stream-mining|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5418" PostTypeId="2" ParentId="5380" CreationDate="2015-03-30T04:46:43.637" Score="1" Body="&lt;p&gt;It means the average predictions of the user across all items, and the average of each item across all users accurately predicts the ratings -- you have an easy data set.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-03-30T04:46:43.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5419" PostTypeId="2" ParentId="5379" CreationDate="2015-03-30T14:10:48.077" Score="1" Body="&lt;p&gt;I can't comment due to reputation, but you really need to tell us what version of SQL Server you are running, maybe some more information about how the data is structured and how you're pulling the data into these dashboard. Maybe even how long it's taking and what resources you have available that actually know what they are doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;OLTP&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, it sounds like you have a OLTP database with lots of tables. As I don't know the relationships of these tables or how you are pulling the data from these tables, I can only assume you are pulling data from many of these tables. If optimizing the tables is not helping due to how many joins and records you are pulling, then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SQL Server Analysis Services&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you need to create a multidimensional database that can be used for reporting. SQL Server Analysis Services helps you do that by allowing you to define OLAP Cubes in many different structures from MOLAP to ROLAP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multidimensional Database (Data Warehousing)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way is creating a new database that is going to be the foundation for your multidimensional data. Therefore, you would need to create a complex ETL System within SQL Server that converts those 400 tables into facts and dimensions automatically on a daily basis and pushes it into your new database. This is a similar process to what SSAS is going to do for you when you define cubes in SQL Server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Preaggregate Tables or Views&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can't do that yourself, then another way is just building new tables in your database that are just preaggregates of the 400 tables that will be used for reporting. Basically established how you are reading the data for your dashboards and find ways to preaggregate that data into fewer tables before you actually pull it into a report.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Automation &amp;amp; Tools&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This (as well building multidimensional data) is accomplished simply by creating stored procedures or SSIS packages and automating the process every day. Then Tableau, SSRS or whatever queries the new tables rather than the previous 400 tables that may be slowing the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hire Someone, You Already Have The Tools&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last and final way is finding a tool that does the ETL for you. There are plenty of ETL vendors out there that can possibly address this problem. But keep in mind, you likely have all the tools you need to do this. You just need to hire the talent to do it either temporarily on contract or full-time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had no knowledge of what I was doing in SQL, I would contract a ETL Developer, SQL Developer or BI Developer to help me. Because why buy another tool box when you already have a good tool box available to you?&lt;/p&gt;&#xA;" OwnerUserId="8842" LastEditorUserId="8842" LastEditDate="2015-03-30T14:28:23.920" LastActivityDate="2015-03-30T14:28:23.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5427" PostTypeId="1" AcceptedAnswerId="5429" CreationDate="2015-04-01T15:23:17.997" Score="22" ViewCount="13753" Body="&lt;p&gt;Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset characteristics?&lt;/p&gt;&#xA;&lt;p&gt;[original data --&amp;gt; build machine learning model --&amp;gt; use ml model to generate synthetic data....!!!]&lt;/p&gt;&#xA;&lt;p&gt;Is it possible? Please point me to related resource if possible.&lt;/p&gt;&#xA;" OwnerUserId="6465" LastEditorUserId="29169" LastEditDate="2021-02-09T20:21:00.633" LastActivityDate="2021-02-09T20:21:00.633" Title="How to generate synthetic dataset using machine learning model learnt with original dataset?" Tags="|machine-learning|dataset|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5428" PostTypeId="1" CreationDate="2015-04-01T20:56:22.337" Score="0" ViewCount="265" Body="&lt;p&gt;I'm having a having a hard time understanding the difference between an isomorphism in graphs and canonical graphs. I have read through the Wikipedia articles, but it still isn't clicking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody explain the difference, perhaps with an example?&lt;/p&gt;&#xA;" OwnerUserId="3466" LastEditorUserId="8953" LastEditDate="2015-05-22T19:35:40.290" LastActivityDate="2015-05-22T19:35:40.290" Title="Canonical VS Graph Isomorphism" Tags="|graphs|terminology|" AnswerCount="1" CommentCount="5" ClosedDate="2015-06-02T09:34:30.853" ContentLicense="CC BY-SA 3.0" />
  <row Id="5429" PostTypeId="2" ParentId="5427" CreationDate="2015-04-01T22:37:39.300" Score="31" Body="&lt;p&gt;The general approach is to do traditional statistical analysis on your data set to define a multidimensional random process that will generate data with the same statistical characteristics.  The virtue of this approach is that your synthetic data is independent of your ML model, but statistically &amp;quot;close&amp;quot; to your data. (see below for discussion of your alternative)&lt;/p&gt;&#xA;&lt;p&gt;In essence, you are estimating the multivariate probability distribution associated with the process.  Once you have estimated the distribution, you can generate synthetic data through the Monte Carlo method or similar repeated sampling methods.  If your data resembles some parametric distribution (e.g. lognormal) then this approach is straightforward and reliable.  The tricky part is to estimate the dependence between variables. See: &lt;a href=&quot;https://www.encyclopediaofmath.org/index.php/Multi-dimensional_statistical_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.encyclopediaofmath.org/index.php/Multi-dimensional_statistical_analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;If your data is irregular, then non-parametric methods are easier and probably more robust.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Multivariate_kernel_density_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multivariate kernal density estimation&lt;/a&gt; is a method that is accessible and appealing to people with ML background. For a general introduction and links to specific methods, see: &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonparametric_statistics&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Nonparametric_statistics&lt;/a&gt; .&lt;/p&gt;&#xA;&lt;p&gt;To validate that this process worked for you, you go through the machine learning process again with the synthesized data, and you should end up with a model that is fairly close to your original.  Likewise, if you put the synthesized data into your ML model, you should get outputs that have similar distribution as your original outputs.&lt;/p&gt;&#xA;&lt;p&gt;In contrast, you are proposing this:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;[original data --&amp;gt; build machine learning model --&amp;gt; use ml model to generate synthetic data....!!!]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This accomplishes something different that the method I just described.  This would solve the &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Inverse_problem#Conceptual_understanding&quot; rel=&quot;nofollow noreferrer&quot;&gt;inverse problem&lt;/a&gt;&lt;/em&gt;: &amp;quot;what inputs could generate any given set of model outputs&amp;quot;. Unless your ML model is over-fitted to your original data, this synthesized data &lt;em&gt;will not&lt;/em&gt; look like your original data in every respect, or even most.&lt;/p&gt;&#xA;&lt;p&gt;Consider a linear regression model. The same linear regression model can have identical fit to data that have very different characteristics. A famous demonstration of this is through &lt;a href=&quot;http://en.wikipedia.org/wiki/Anscombe%27s_quartet&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anscombe's quartet&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/P9s4w.png&quot; alt=&quot;All four sets are identical when examined using simple summary statistics, but vary considerably when graphed&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Thought I don't have references, I believe this problem can also arise in logistic regression, generalized linear models, SVM, and K-means clustering.&lt;/p&gt;&#xA;&lt;p&gt;There are some ML model types (e.g. decision tree) where it's possible to inverse them to generate synthetic data, though it takes some work.  See: &lt;a href=&quot;http://library.cust.edu.pk/IEEEMagazineTransactions_2008/IEEE%20Internet%20Computing/May-June/Generating%20Synthetic%20Data%20to%20Match%20Data%20Mining%20Patterns%20%20Josh%20Eno,%20Pp.%2078-82.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generating Synthetic Data to Match Data Mining Patterns&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="16605" LastEditDate="2021-01-20T08:41:33.757" LastActivityDate="2021-01-20T08:41:33.757" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5430" PostTypeId="2" ParentId="5417" CreationDate="2015-04-02T01:33:37.173" Score="6" Body="&lt;p&gt;If I were you, I would pick anyone of the frameworks I am comfortable with and implement the use-case. &#xA;Spark-Streaming + MLlib should work and would be my choice since its user base is on the rise and it is one of the most popular project under the Apache Umbrella with good enterprise business plan. Both Cloudera and Hortonworks provide enterprise level support. &#xA;Now, in theory Spark-Streaming lacks behind Storm in stream processing, but the framework is cool in a way that it provides you the option to do streaming, common map and reduce, graph processing and SQL under the same framework. So once you have the pipeline to convert your data to RDD you are good for most of the common jobs related to Data Analysis. It's written from scratch in Scala which is a very powerful language and provides huge scalability in a distributed setup when handling concurrency. &#xA;Hope this helps, feel free to reach out to me with any questions you have. &lt;/p&gt;&#xA;" OwnerUserId="5179" LastEditorUserId="847" LastEditDate="2015-06-04T04:05:32.770" LastActivityDate="2015-06-04T04:05:32.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5431" PostTypeId="2" ParentId="661" CreationDate="2015-04-02T10:50:12.953" Score="0" Body="&lt;p&gt;Problem is resolved. I created the folders in terminal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6b.  hdfs dfs -mkdir /user&#xA;hdfs dfs -mkdir /user/hduser&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;7&quot;&gt;&#xA;&lt;li&gt;Put this file to HDFS:&#xA;hadoop fs -put file1.txt /user/hduser/file1.txt&#xA;hadoop fs -put file.txt file.txt&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8923" LastActivityDate="2015-04-02T10:50:12.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5432" PostTypeId="2" ParentId="2269" CreationDate="2015-04-02T14:08:05.547" Score="2" Body="&lt;p&gt;I am using this one, so far so good.&lt;br&gt;&#xA;Online terminals: &lt;a href=&quot;http://www.tutorialspoint.com/codingground.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.tutorialspoint.com/codingground.htm&lt;/a&gt; &lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also, &lt;a href=&quot;http://www.r-fiddle.org/#/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R-Fiddle&lt;/a&gt; is an option.&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="8930" LastEditorUserId="29169" LastEditDate="2021-02-09T01:33:22.723" LastActivityDate="2021-02-09T01:33:22.723" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5436" PostTypeId="1" AcceptedAnswerId="5437" CreationDate="2015-04-02T23:00:20.687" Score="1" ViewCount="98" Body="&lt;p&gt;I am working on a data set where the categorical variables have lots of empty spaces (not &quot;NA&quot; but &quot;&quot;). For example, one variable has 14587 empty spaces out of 14644 observations. There are many such variables where most of the observations are empty.In fact it is a survey dataset where the participant just chose to ignore a particular question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have never handled similar dataset. I am looking for advise as to how best to handle such datasets before any modeling is done. Deleting the rows or the variables with lots of empty spaces doesn't seem feasible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2015-04-02T23:38:56.127" Title="how to modify sparse survey dataset with empty data points?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5437" PostTypeId="2" ParentId="5436" CreationDate="2015-04-02T23:38:56.127" Score="0" Body="&lt;p&gt;I would consider approaching this situation from the following two &lt;strong&gt;perspectives&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Missing data analysis&lt;/strong&gt;. Despite formally the values in question are empty and not NA, I think that effectively &lt;em&gt;incomplete&lt;/em&gt; data can (and should) be considered as &lt;em&gt;missing&lt;/em&gt;. If that is the case, you need to automatically recode those values and then apply standard &lt;em&gt;missing data handling&lt;/em&gt; approaches, such as &lt;em&gt;multiple imputation&lt;/em&gt;. If you use &lt;code&gt;R&lt;/code&gt;, you can use packages &lt;code&gt;Amelia&lt;/code&gt; (if the data is multivariate normal), &lt;code&gt;mice&lt;/code&gt; (supports non-normal data) or some others. For a nice &lt;strong&gt;overview&lt;/strong&gt; of &lt;em&gt;approaches&lt;/em&gt;, &lt;em&gt;methods&lt;/em&gt; and &lt;em&gt;software&lt;/em&gt; for multiple imputation of data with missing values, see the 2007 excellent article by Nicholas Horton and Ken Kleinman &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839993&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models&quot;&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Sparse data analysis&lt;/strong&gt;, such as &lt;em&gt;sparse regression&lt;/em&gt;. I'm not too sure how well this approach would work for variables with high levels of sparsity, but you can find a lot of corresponding information in &lt;a href=&quot;https://datascience.stackexchange.com/a/918/2452&quot;&gt;my relevant answer&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-04-02T23:38:56.127" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5438" PostTypeId="1" AcceptedAnswerId="5441" CreationDate="2015-04-02T23:52:02.953" Score="8" ViewCount="294" Body="&lt;p&gt;I'm trying to see if there is a conventional term for this concept to help me in my literature research and writing.  When a machine learning model causes an action to be taken in the real world that affects future instances, what is that called?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking about something like a recommender system that recommends one given product and doesn't recommend another given product.  Then, you've increased the likelihood that someone is going to buy the first product and decreased the likelihood that someone is going to buy the second product.  So then those sales numbers will eventually become training instances, creating a sort of feedback loop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a term for this?&lt;/p&gt;&#xA;" OwnerUserId="8938" LastEditorUserId="21" LastEditDate="2015-04-05T13:54:36.420" LastActivityDate="2015-11-30T15:59:29.730" Title="What is the term for when a model acts on the thing being modeled and thus changes the concept?" Tags="|machine-learning|terminology|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5440" PostTypeId="2" ParentId="5438" CreationDate="2015-04-03T00:51:29.800" Score="3" Body="&lt;p&gt;Though it is not specifically a term, focused on &lt;em&gt;machine learning&lt;/em&gt;, but I would refer to such behavior of a statistical model, using a general term &lt;strong&gt;side effect&lt;/strong&gt; (while adding some &lt;em&gt;clarifying adjectives&lt;/em&gt;, such as expected or unexpected, desired or undesired, and similar). &lt;strong&gt;Modeling outcome&lt;/strong&gt; or &lt;strong&gt;transitive feedback loop outcome&lt;/strong&gt; might be some of the &lt;em&gt;alternative&lt;/em&gt; terms.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-04-03T00:51:29.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5441" PostTypeId="2" ParentId="5438" CreationDate="2015-04-03T01:44:18.450" Score="6" Body="&lt;p&gt;There are three terms from social science that apply to your situation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Reflexivity_%28social_theory%29&quot;&gt;Reflexivity&lt;/a&gt; - refers to circular relationships between cause and effect.  In particular, you could use the definition of the term adopted by George Soros to refer to reverse causal loop between share prices (i.e. present value of fundamentals) and business fundamentals.  In a way, the share price is a &quot;model&quot; of the fundamental business processes.  Usually, people assume that causality is one-way, from fundamentals to share price.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Performativity&quot;&gt;Performativity&lt;/a&gt; - As used by Donald MacKenzie (e.g. &lt;a href=&quot;http://www.lse.ac.uk/accounting/CARR/pdf/MacKenzie.pdf&quot;&gt;here&lt;/a&gt;), many economic models are not &quot;cameras&quot; -- taking pictures of economic reality -- but in fact are &quot;engines&quot; -- an integral part of the construction of economic reality. He has a book of that title: &lt;em&gt;&lt;a href=&quot;http://mitpress.mit.edu/books/engine-not-camera&quot;&gt;An Engine, Not a Camera&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Self-fulfilling_prophecy&quot;&gt;Self-fulfilling Prophecy&lt;/a&gt; - a prediction that directly or indirectly causes itself to become true, by the very terms of the prophecy itself, due to positive feedback between belief and behavior. This is the broadest term, and least specific to the situation you describe.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Of the three terms, I suggest that MacKenzie's &quot;performativity&quot; is the best fit to your situation.  He claims, among other things, that the validity of the economic models (e.g. Black-Scholes option pricing) has been improved by its very use by market participants, and therefore how it reflects in options pricing and trading patterns.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="609" LastEditDate="2015-04-03T01:49:38.230" LastActivityDate="2015-04-03T01:49:38.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5442" PostTypeId="1" AcceptedAnswerId="5446" CreationDate="2015-04-03T03:07:50.493" Score="4" ViewCount="2164" Body="&lt;p&gt;Does anyone know, where I can learn about applying data science to win a political campaign? I know the Obama campaign had 12 data scientists in 2008 and 165 data scientists in 2012. In 2012, they ran over 65,000 simulations every night, for 14 months. They correctly predicted every state within 0.5% and Florida within 0.05%. How did they do this? And where can I find the data they used?&lt;/p&gt;&#xA;" OwnerUserId="8860" LastEditorUserId="2452" LastEditDate="2015-04-04T07:20:53.957" LastActivityDate="2016-04-14T12:56:43.170" Title="Learning resources for data science to win political campaigns?" Tags="|bigdata|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5443" PostTypeId="1" AcceptedAnswerId="5450" CreationDate="2015-04-03T20:28:36.710" Score="43" ViewCount="32584" Body="&lt;p&gt;I would consider myself a journeyman data scientist.  Like most (I think), I made my first charts and did my first aggregations in high school and college, using Excel.  As I went through college, grad school and ~7 years of work experience, I quickly picked up what I consider to be more advanced tools, like SQL, R, Python, Hadoop, LaTeX, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are interviewing for a data scientist position and one candidate advertises himself as a &quot;senior data scientist&quot; (a very buzzy term these days) with 15+ years experience.  When asked what his preferred toolset was, he responded that it was Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took this as evidence that he was not as experienced as his resume would claim, but wasn't sure.  After all, just because it's not my preferred tool, doesn't mean it's not other people's. &lt;strong&gt;Do experienced data scientists use Excel?  Can you assume a lack of experience from someone who does primarily use Excel?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="8944" LastEditorUserId="3151" LastEditDate="2015-12-20T08:18:38.047" LastActivityDate="2019-06-28T22:41:11.520" Title="Do data scientists use Excel?" Tags="|tools|career|excel|" AnswerCount="10" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="5444" PostTypeId="2" ParentId="5443" CreationDate="2015-04-03T22:37:22.080" Score="17" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Do experienced data scientists use Excel?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've seen some experienced data scientists, who use Excel - either due to their preference, or due to their workplace's business and IT environment specifics (for example, many financial institutions use Excel as their major tool, at least, for modeling). However, I think that most experienced data scientists recognize the need to use tools, which are optimal for particular tasks, and adhere to this approach.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can you assume a lack of experience from someone who does primarily&#xA;  use Excel?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, you cannot. This is the corollary from my above-mentioned thoughts. Data science does not automatically imply big data - there is plenty of data science work that Excel can handle quite well. Having said that, if a data scientist (even experienced one) does not have knowledge (at least, basic) of modern data science tools, including big data-focused ones, it is somewhat disturbing. This is because experimentation is deeply ingrained into the nature of data science due to exploratory data analysis being a essential and, even, a crucial part of it. Therefore, a person, who does not have an urge to explore other tools within their domain, could rank lower among candidates in the overall fit for a data science position (of course, this is quite fuzzy, as some people are very quick in learning new material, plus, people might have not had an opportunity to satisfy their interest in other tools due to various personal or workplace reasons).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, in conclusion, I think that the best answer an experienced data scientist might have to a question in regard to their preferred tool is the following: &lt;strong&gt;My preferred tool is the optimal one, that is the one that best fits the task at hand.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-04-03T22:37:22.080" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5445" PostTypeId="1" CreationDate="2015-04-03T23:19:27.283" Score="1" ViewCount="271" Body="&lt;p&gt;with regards to the Logistic Regression cost function of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/lBaI2.png&quot; alt=&quot;Logistic Regression Cost Function&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And hypothesis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ISUJW.png&quot; alt=&quot;Hypothesis&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to tell the +/- of the error for how &quot;confident&quot; the hypothesis is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. if the +/- of the error was 0.1, I would know that if my hypothesis predicted 0.4 it could be 0.1 greater (0.5) or 0.1 less (0.3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is for binary classification&lt;/p&gt;&#xA;" OwnerUserId="8269" LastEditorUserId="15620" LastEditDate="2017-04-20T07:13:17.803" LastActivityDate="2017-04-20T07:13:17.803" Title="Logistic Regression Cost Function Error" Tags="|classification|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5446" PostTypeId="2" ParentId="5442" CreationDate="2015-04-04T06:57:47.793" Score="5" Body="&lt;p&gt;This is an interesting and &lt;a href=&quot;http://www.datanami.com/2014/11/04/data-analytics-shaped-election-day-2014&quot; rel=&quot;noreferrer&quot;&gt;relevant question&lt;/a&gt;. I think that from &lt;em&gt;data science perspective&lt;/em&gt;, it should not be, in principle, any different from any other similar data science tasks, such as prediction, forecasting or other analyses. Similarly to any data science work, the quality of applying data science to politics very much depends on understanding not only data science &lt;strong&gt;approaches, methods and tools&lt;/strong&gt;, but, first and foremost, the &lt;strong&gt;domain&lt;/strong&gt; being analyzed, that is politics domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rapidly rising popularity of data science and machine learning (ML), in general, certainly has a significant impact on particular verticals and politics is not an exception. This impact can be seen not only in increased &lt;strong&gt;research interest&lt;/strong&gt; in applying data science and ML to political science (for example, see &lt;a href=&quot;http://imai.princeton.edu/talk/files/SML11.pdf&quot; rel=&quot;noreferrer&quot;&gt;this presentation&lt;/a&gt;, &lt;a href=&quot;http://www.stat.columbia.edu/~jakulin/Politics&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;, &lt;a href=&quot;http://stanford.edu/~jgrimmer/bd_2.pdf&quot; rel=&quot;noreferrer&quot;&gt;this overview paper&lt;/a&gt; and &lt;a href=&quot;http://www.oxfordjournals.org/our_journals/polana/virtualissue4.html&quot; rel=&quot;noreferrer&quot;&gt;this whole virtual/open issue in a prominent Oxford journal&lt;/a&gt;), but in &lt;strong&gt;practical applications&lt;/strong&gt;. Moreover, a new term - &lt;strong&gt;political informatics&lt;/strong&gt; or &lt;em&gt;poliInformatics&lt;/em&gt; or &lt;em&gt;poli-informatics&lt;/em&gt; - has been coined to name an &lt;a href=&quot;http://poliinformatics.org&quot; rel=&quot;noreferrer&quot;&gt;interdisciplinary field&lt;/a&gt;, which stated goal is to study and use data science, big data and ML in the government and politics domains. As I've said earlier, the interest in applying data science to politics goes beyond research and often results in politics-focused startups, such as &lt;a href=&quot;https://gigaom.com/2012/06/25/can-a-big-data-product-level-the-playing-field-in-politics&quot; rel=&quot;noreferrer&quot;&gt;PoliticIt&lt;/a&gt; or &lt;a href=&quot;http://www.parabellumlabs.com&quot; rel=&quot;noreferrer&quot;&gt;Para Bellum Labs&lt;/a&gt;. Following the unfortunate, but established trend in startup ecosystem, many of those ventures fail. For example, read &lt;a href=&quot;http://www.poliana.com&quot; rel=&quot;noreferrer&quot;&gt;the story&lt;/a&gt; of one of such startups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty sure that you will be able to find neither &lt;em&gt;proprietary algorithms&lt;/em&gt; that political startups or election data science teams used and use, nor the their &lt;strong&gt;data sets&lt;/strong&gt;. However, I am rather positive that you can get some understanding about typical data sets as well as data collection and analysis methods via the resources that I have referenced above. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-04-04T07:18:46.990" LastActivityDate="2015-04-04T07:18:46.990" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5447" PostTypeId="2" ParentId="5445" CreationDate="2015-04-04T07:32:19.127" Score="3" Body="&lt;p&gt;Logistic regression is just a generalized linear model, so there is a linear regression lurking in here whose errors are supposed to be normally distributed, and from which you can get confidence intervals in the usual way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is you have a linear predictor (the -theta' * x part), transformed by the inverse of a link function (the logistic function). You can transform prediction +/- confidence interval using this inverse link to get your +/- in probability terms. It won't be symmetric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nice writeup of how to do this in R: &lt;a href=&quot;https://stackoverflow.com/a/14424417/64174&quot;&gt;https://stackoverflow.com/a/14424417/64174&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-04-04T07:32:19.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5448" PostTypeId="2" ParentId="5443" CreationDate="2015-04-04T08:30:27.967" Score="4" Body="&lt;p&gt;Excel allows only very small data and doesn't have anything that is sufficiently useful and flexible for machine learning or even just plotting. All I would do in Excel, is stare at a subset of the data for a first glance over the values to make sure I don't miss anything visible by eye.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if his favourite tool is Excel, this might suggest he rarely deals with machine learning, statistics, larger data sizes or any advanced plotting. Someone like this I wouldn't call a Data Scientist. Of course titles don't matter and it depends a lot on your requirements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, don't make a judgement by statements of experience or CV. I've seen CVs and known the people behind it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't assume. &lt;strong&gt;Test him!&lt;/strong&gt; You should be good enough to set up a test. It has been shown that interviews alone are close to useless to determine skills (they only show personality). Set up a very simple supervised learning test and let him use any tool he wants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you want to screen people at an interview first, then ask him about very basic but important insights about statistics or machine learning. Something that every single of your current employees knows.&lt;/p&gt;&#xA;" OwnerUserId="723" LastEditorUserId="723" LastEditDate="2015-04-04T17:46:27.390" LastActivityDate="2015-04-04T17:46:27.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5449" PostTypeId="2" ParentId="5443" CreationDate="2015-04-04T12:53:13.217" Score="4" Body="&lt;p&gt;Let me first clarify that I am starting my journey into data science from a programmer and database developer standpoint. I am not a 10-year data science expert nor a statistical god. However, I do work data scientist and large datasets for a company that works with rather large clients worldwide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my experience, &lt;strong&gt;data scientist use whatever tools they need to get the job done.&lt;/strong&gt; Excel, R, SAS, Python and more are all tools in a toolbox for good data scientist. The best can use a wide variety of tools to analyze and crunch data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Therefore, if you find yourself comparing R to Python, then you're likely doing it all wrong in the data science world.&lt;/strong&gt; Good data scientist use both when it makes sense to use one over the other. This also applies to Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that it's rather hard to find anyone that is going to have experience in so many different tools and languages while been great at everything. I also think it's going to be hard to find data scientist specifically that can not only program complex algorithms but also know how to use them from a statistical standpoint too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the data scientist I've worked with come in about 2 flavors. Those that can program and those that can't. I rarely work with data scientist that can pull data in Python, manipulate it with something like Pandas, fit a model to the data in R and then present it to management at the end of the week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, I know they exist. I've read many data science blogs from guys developing web scrappers, pushing it into Hadoop, pulling it back out in Python, programming complex things and running it through R to boot. They exist. They're out there. I just haven't ran into too many that can do all of that. Maybe it's just my area though?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, does that mean only specializing in one thing bad? No. Plenty of my friends specialize in just one main language and kill it. I know plenty of data guys who only know R and kill it. &lt;strong&gt;I also know plenty of people who just use Excel to analyze data because that's the only thing most non-data scientist can open and use (especially in B2B companies).&lt;/strong&gt; The question you really need to answer is if this one thing is the ONE thing you need for this position? And most importantly, can they learn new things?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science is not just restricted to &quot;BIG DATA&quot; or NoSQL.&lt;/p&gt;&#xA;" OwnerUserId="3417" LastEditorUserId="3417" LastEditDate="2015-04-04T13:26:45.183" LastActivityDate="2015-04-04T13:26:45.183" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5450" PostTypeId="2" ParentId="5443" CreationDate="2015-04-04T20:37:55.330" Score="35" Body="&lt;p&gt;Most non-technical people often use Excel as a database replacement. I think that's wrong but tolerable. However, someone who is supposedly experienced in data analysis simply can not use Excel as his main tool (excluding the obvious task of looking at the data for the first time). That's because Excel was never intended for that kind of analysis and as a consequence of this, it is incredibly easy to make mistakes in Excel (that's not to say that it is not incredibly easy to make another type of mistakes when using other tools, but Excel aggravates the situation even more.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize what Excel doesn't have and is a must for any analysis:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Reproducibility. A data analysis needs to be reproducible.&lt;/li&gt;&#xA;&lt;li&gt;Version control. Good for collaboration and also good for reproducibility. Instead of using xls, use csv (still very complex and has lots of edge cases, but csv parsers are fairly good nowadays.) &lt;/li&gt;&#xA;&lt;li&gt;Testing. If you don't have tests, your code is broken. If your code is broken, your analysis is worse than useless. &lt;/li&gt;&#xA;&lt;li&gt;Maintainability.&lt;/li&gt;&#xA;&lt;li&gt;Accuracy. Numerical accuracy, accurate date parsing, among others are really lacking in Excel.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;More resources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.eusprig.org/horror-stories.htm&quot; rel=&quot;noreferrer&quot;&gt;European Spreadsheet Risks Interest Group - Horror Stories&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a-spreadsheet-for-important-work-i-mean-it/&quot; rel=&quot;noreferrer&quot;&gt;You shouldn’t use a spreadsheet for important work (I mean it)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.forbes.com/sites/timworstall/2013/02/13/microsofts-excel-might-be-the-most-dangerous-software-on-the-planet/&quot; rel=&quot;noreferrer&quot;&gt;Microsoft's Excel Might Be The Most Dangerous Software On The Planet&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://randyzwitch.com/excel-destroys-data/&quot; rel=&quot;noreferrer&quot;&gt;Destroy Your Data Using Excel With This One Weird Trick!&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.win-vector.com/blog/2014/11/excel-spreadsheets-are-hard-to-get-right/&quot; rel=&quot;noreferrer&quot;&gt;Excel spreadsheets are hard to get right&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4621" LastEditorUserId="4621" LastEditDate="2015-12-20T00:56:25.960" LastActivityDate="2015-12-20T00:56:25.960" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="5452" PostTypeId="2" ParentId="5443" CreationDate="2015-04-04T23:43:50.157" Score="15" Body="&lt;p&gt;I think most people are answering without having a good knowledge of excel. Excel (since 2010) has an in memory columnar [multi table] database , called power pivot (which allows input from csv/databases etc), allowing it to store millions of rows (it doesn't have to be loaded on a spreadsheet).&#xA;It also has an ETL tool called power query allowing you to read the data from a variety of sources (including hadoop). And it has a visualisation tool (power view &amp;amp; power map). A lot of Data Science is doing aggregation and top-n analysis at which power pivot excels.  Add to this the interactive nature of these tools - any user can easily drag and drop a dimension on which to break up the results adn I hope you can see the benefits. So yes you can't do machine learning, but I would question how much machine learning is done by data scientists day to day: eg when I want to analyse the prediction errors made in machine learning program I find it easiest to slice and dice the errors with excel. &lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2015-04-04T23:43:50.157" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5457" PostTypeId="2" ParentId="5438" CreationDate="2015-04-05T07:57:05.907" Score="0" Body="&lt;p&gt;I think in the literature the overarching term often used is  active learning  &lt;a href=&quot;http://en.wikipedia.org/wiki/Active_learning_%28machine_learning%29&quot; rel=&quot;nofollow&quot;&gt;Active Learning&lt;/a&gt; ...in particular &#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Multi-armed_bandit#Contextual_Bandit&quot; rel=&quot;nofollow&quot;&gt;Multi-Armed Contextual Bandit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2015-04-05T07:57:05.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5458" PostTypeId="1" CreationDate="2015-04-05T10:00:43.167" Score="1" ViewCount="1162" Body="&lt;p&gt;I have around 1,000 job ads in the filed of IT (in excel file). I want to find the skills which are mentioned in each of ads. and then find the similar jobs based on skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My method: I created 12 categories Such as programming skills,  testing skills,  communication skills, network skills, ... . Each advertisement may belong to 3-4 categories. In this case, some said multi-variate classification or Multi label classification is useful. But I don't know how to do this kind of classification in RapidMiner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- Does anyone know how to do multi-variate classification or Multi label classification in RapidMiner? or is there another way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Do you recommend &quot;classification&quot; in order to analysis required job skills? or another technique? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- Is there any better way to classify the skills which are stated in job ads?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new in the field of text mining. Please let me know if you have any idea. Thanks&lt;/p&gt;&#xA;" OwnerUserId="8969" LastEditorUserId="8969" LastEditDate="2015-04-05T11:00:14.943" LastActivityDate="2016-02-18T13:42:13.210" Title="Classification of skills based on job ads" Tags="|machine-learning|classification|nlp|text-mining|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5459" PostTypeId="1" AcceptedAnswerId="5460" CreationDate="2015-04-05T14:37:45.787" Score="0" ViewCount="164" Body="&lt;p&gt;How to write &quot;OR&quot; in this example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DATA a1;&#xA;set a;&#xA;if var1=1 OR 2;&#xA;run;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. va1 is the categorial (with categories: 1, 2, 3)&lt;/p&gt;&#xA;" OwnerUserId="8951" LastEditorUserId="21" LastEditDate="2015-04-09T01:06:36.417" LastActivityDate="2015-04-09T01:06:36.417" Title="SAS. How to write &quot;OR&quot;" Tags="|sampling|sas|" AnswerCount="1" CommentCount="1" ClosedDate="2015-04-09T01:06:27.490" ContentLicense="CC BY-SA 3.0" />
  <row Id="5460" PostTypeId="2" ParentId="5459" CreationDate="2015-04-05T17:42:10.457" Score="0" Body="&lt;p&gt;i'm assuming this will be migrated to stack overflow, but instead of trying to do &lt;code&gt;if var1=1 or 2&lt;/code&gt; wouldn't it be better to use &lt;code&gt;if var1 in (1, 2)&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...and somebody with enough reputation should probably create an sas tag (and an spss tag while you're at it) unless data scientists only use open source languages like &lt;code&gt;r&lt;/code&gt; and &lt;code&gt;python&lt;/code&gt; now...&lt;/p&gt;&#xA;" OwnerUserId="8953" LastActivityDate="2015-04-05T17:42:10.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5461" PostTypeId="2" ParentId="5442" CreationDate="2015-04-06T01:07:35.447" Score="0" Body="&lt;p&gt;Erin Hartmann who just finished up a PhD in Political Science at UC-Berkley has a paper that uses Obama for America data from 2012.  She is using the paper on the job-market for an American Politics sub-field position  Might want to check out her work, though I'm not sure it will answer your question.  She has a public site here: &lt;a href=&quot;http://www.erinhartman.com&quot; rel=&quot;nofollow&quot;&gt;http://www.erinhartman.com&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8977" LastActivityDate="2015-04-06T01:07:35.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5462" PostTypeId="1" CreationDate="2015-04-06T04:18:57.317" Score="1" ViewCount="93" Body="&lt;p&gt;If I parse following sentence:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;He is playing cricket in ground with grass.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with stanford parser, the result is: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(ROOT&#xA;  (S&#xA;    (NP (PRP He))&#xA;    (VP (VBZ is)&#xA;      (VP (VBG playing)&#xA;        (NP&#xA;          (NP (NN cricket))&#xA;          (PP (IN in)&#xA;            (NP (NN ground))))&#xA;        (PP (IN with)&#xA;          (NP (NN grass)))))&#xA;    (. .)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any parser who can correct the result on the basis of probability as the probability of appearing grass with ground is higher than cricket?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or parser which do chunking before parsing like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;He is playing cricket |in ground| |with grass|.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and calculate probability with all combinations before generating the parse tree.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;He is playing cricket |in ground| |with grass|.&lt;/li&gt;&#xA;&lt;li&gt;He is playing cricket |with grass| |in ground|.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8948" LastEditorUserId="8953" LastEditDate="2015-04-06T13:31:49.307" LastActivityDate="2015-04-06T13:31:49.307" Title="Correct tagging but wrong parse tree" Tags="|nlp|parsing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5464" PostTypeId="1" CreationDate="2015-04-06T17:36:22.860" Score="1" ViewCount="156" Body="&lt;p&gt;Obviously most employers, when hiring a data scientist, would prefer experience with big data and/or data science.  But what can one safely assume they will acknowledge as experience?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say someone often launches software on a computing cluster that typically generates an amount of data.  I'm not sure what the best measure of this data is for data science.  I'll call it one or two thousand rows, 200k or 300k points per row...certainly under 500k.  Then for each point, let's call it 25 or 30 values.  This amounts to 30 or 40 gig of data.  300 or 400 times of this and you can call it a study - maybe one or two studies per year.  I'm under the impression that this is much smaller than a data scientist at Google or Facebook would be used to, but it's certainly too big for my home computing systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone's been working with this for years (some people at this company have been doing it since before &lt;em&gt;data science&lt;/em&gt; was coined/before social media existed), is it fair for them to claim big data experience?  According to &lt;a href=&quot;https://datascience.stackexchange.com/a/37/8953&quot;&gt;this answer&lt;/a&gt;, it's not the amount of data but what needs to be done with the data that matters - is that a universally accepted opinion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For what it's worth, working with this data entails manipulation/cleaning of the data with some proprietary languages, shell scripts, and a lot of Python.  A little bit of R but that's a more recent thing.  It involves tons of data visualization, drawing conclusions, and presenting to management/convincing decision makers.  Some of it involves trend determination, extrapolation, and comparisons made between data sets that aren't related in a straightforward way, so it sounds data science-ish to me.  But I will be the first to admit that I have a limited understanding of what data science currently is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...bonus points if you can tell me whether or not this is an Easter egg or is the actual answer total for this site at the moment:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/w6haD.png&quot; alt=&quot;1337&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll try to clarify.  What do data science employers acknowledge as experience with big data/data science?  Does the size of the data above qualify experience with it?  Or is it universally accepted among those in the field that it's not at all the size of the data, but what you need to do with the data?&lt;/p&gt;&#xA;" OwnerUserId="8953" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-08T02:37:25.507" Title="Is it ethical to claim experience with big data when the data isn't a part of the new advertising/social media/retail fad?" Tags="|bigdata|terminology|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5466" PostTypeId="2" ParentId="778" CreationDate="2015-04-06T19:20:18.873" Score="2" Body="&lt;p&gt;I've been using Anaconda Python 3.4 and Pandas to search 10M row database to match 20K of login credentials. Takes about a minute. The pandas internals make great use of memory. That said, truly big data requires a processing architecture matched to the problem. Pandas is just the glue (logic) in this equation, and other tools can do this as well. R, Scala, Haskell, SAS, etc. can replicate some of the logic - perhaps just enough to answer questions faster. But python makes a good (best?) general-purpose tool. You can run R code in python, as well as most other languages. Although interpretive, there are high performance techniques and tools such as pypy that can make python run almost as fast as benchmark tools with only slightly more effort. And python has many libraries that do just about everything - see above list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are asking if you should learn and use python, my answer is yes Articles indicate that python is used more than R among people who use both. But few data science problems are solved by a single tool. It may become your go-to tool, but its only that - a tool. And just as no sane person builds a house with just a hammer, no sane Data Scientist uses just one tool.&lt;/p&gt;&#xA;" OwnerUserId="8986" LastActivityDate="2015-04-06T19:20:18.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5469" PostTypeId="2" ParentId="5261" CreationDate="2015-04-06T22:07:18.737" Score="1" Body="&lt;h1&gt;looooops&lt;/h1&gt;&#xA;&lt;p&gt;I can imagine it will run slow :)&#xA;avoid when possible very long for-loops in R. If so, keep the iterations as simple as possible, be careful using slow search functions and do not subset or edit large data structures.&lt;/p&gt;&#xA;&lt;h1&gt;restructure your data&lt;/h1&gt;&#xA;&lt;p&gt;I was talking to a guy, at a party, doing similar analysis. He said they arranged the data in a matrix of nrow sentences(200.000) and ncol(7000) words. Any element count how many times a given word occoured in a given sentence. If you need to do more analysis on the data set later, any operations can be done really fast. The result of your sentiment analysis would be the inner product of any row to a scoring vector(+1 +2 -1, score for word). The matrix would take some ~2Gb space if you use the bigmemory package and choose shorts(max 255 counts) instead of integers. bigmemory goes smoothly with multi-core(foreach, doMC, etc.) and elements can be modified individually much faster than for R matrices. The matrix can be saved to shared memort or HDD. bigmemory is using a C++ data structure which makes it possible to do some really fast operations with Rcpp. Steap laerning curve though.&lt;/p&gt;&#xA;&lt;p&gt;Sparse matrix would also be a good idea.&#xA;bigmemory package do not support windows&lt;/p&gt;&#xA;&lt;p&gt;Maybe get some inspiration here:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/10233087/sentiment-analysis-using-r&quot;&gt;https://stackoverflow.com/questions/10233087/sentiment-analysis-using-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8987" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-04-06T22:19:59.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5471" PostTypeId="1" AcceptedAnswerId="5477" CreationDate="2015-04-07T14:05:28.010" Score="2" ViewCount="971" Body="&lt;p&gt;I am trying to understand the procedure of building a static local website using &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; and &lt;code&gt;Rmarkdown&lt;/code&gt;. I am aware of a &lt;a href=&quot;http://rmarkdown.rstudio.com/html_document_format.html#creating-a-website&quot; rel=&quot;nofollow&quot;&gt;Rmarkdown&lt;/a&gt; website where the procedure is outlined, but unfortunately I do not understand the steps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody here have some experience in building a static local website and would be so kind as to describe the procedure in more detail?   &lt;/p&gt;&#xA;" OwnerUserId="8644" LastActivityDate="2015-04-08T03:47:26.333" Title="Building a static local website using Rmarkdown: step by step procedure" Tags="|r|" AnswerCount="1" CommentCount="5" ClosedDate="2015-04-09T01:00:26.303" ContentLicense="CC BY-SA 3.0" />
  <row Id="5473" PostTypeId="2" ParentId="5464" CreationDate="2015-04-07T14:46:19.027" Score="3" Body="&lt;p&gt;Data can be big in various ways. It can be large N (observations) and small P (variables per observation), like access logs for example. It can be large P and small N, like in biostatistics where you have tens of thousands of gene expressions from tens or hundreds of people. Or it can be both N and P, like in Facebook data. Data can also be made big, by normalizing every single detail of it, meaning that you'll be able to make a massive relational database schema that actually represents simple data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All kinds of big data needs different approaches and companies mostly care about their own version of big data. That's why the answer isn't black and white.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the question you posed in title then I believe it's ethical to claim experience with big data as long as you can put it into context of the business or domain where you encountered it and can describe some situations where a different set of tools or skills was needed than is used for smaller toy examples.&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-04-07T14:46:19.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5474" PostTypeId="2" ParentId="5428" CreationDate="2015-04-08T00:00:33.730" Score="2" Body="&lt;p&gt;this might take a few iterations, but perhaps it can be moved to chat if it gets too long.  are you wondering about a specific application or just the general concept?  as far as i know, the canonical form of something is the unique/simplest arrangement/representation of that something.  as for isomorphism (one to one relationship, a function gets you from one set to the other), you can oversimplify it and think about factors of numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for a moment consider an analogy - instead of graphs, using sets of numbers.  this is randomly off the top of my head so hopefully there aren't too many holes in it.  we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A = 6, 10, 14$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$B = 18, 30, 42$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let's call $A$ and $B$ isomorphic because to get between any value in $A$ to the corresponding value in $B$, you multiply or divide by 3.  now let's introduce another isomorphic set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = 3, 5, 7$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it looks like $A$, $B$, and $C$ are all isomorphic because there is a one to one relationship between any of their initial values.  the same can be said for any of their middle values, or any of their ending values.  while it's great that they're all isomorphic, we'll consider something additional about $C$.  i'm going to call $C$ canonical because if you remove the greatest common factor from $A$, or you remove the greatest common factor from $B$, you get $C$.  there is no common factor we can remove to reduce $C$ itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;extending this, you can see that a canonical representation of a graph &lt;em&gt;is&lt;/em&gt; an isomorphic version of that graph, but not necessarily the other way around.  all isomorphic versions of said graph will reduce to the same canonical form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that being said, the definition of canonical is more complicated for graphs.  first, for graph isomorphism, &lt;a href=&quot;http://www.tcs.hut.fi/Software/bliss/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page about a code named bliss&lt;/a&gt; presents some small graphs as an example.  they define $G_{1}$ as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/xWHFw.png&quot; alt=&quot;g1&quot;&gt;&#xA;and $G_{2}$ as&#xA;&lt;img src=&quot;https://i.stack.imgur.com/gKwj1.png&quot; alt=&quot;g2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and offer two transformations for $G_{1}$ that will make $G_{1}$ and $G_{2}$ isomorphic.  first, they consider $\{\langle 1\to 2, 2\to 4, 3\to 1, 4\to 3\rangle\}$, and then $\{\langle 1\to 2, 2\to 4, 3\to 3, 4\to 1\rangle\}$ - both of these transformations swap vertices.  the transformations result in the following isomorphic graphs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ovd06.png&quot; alt=&quot;transformationResults&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;each pair of connected vertices has a corresponding pair in $G_{2}$ ($2\to 1,3,4; 1\to 4; 3\to 4$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;now, to discuss canonical graphs, we must identify a cost function with which we can measure what is the 'most canonical' form.  &lt;a href=&quot;http://www.csie.ntu.edu.tw/~r94054/paper/draw_plane_graph.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; mentions metrics such as area, minimum angle, and total number of bends.  for simplicity, i'll try to minimize bends and favor angles that are closer to $90^{\circ}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you use a coordinate transformation, you can see that this simple graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Atqpw.png&quot; alt=&quot;firstIsomorphic&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/kG7mg.png&quot; alt=&quot;otherIsomorphic&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;are isomorphic.  the same edges connect the same pairs of vertices, and if you move vertices from one around, you can make the other.  this graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CtUNC.png&quot; alt=&quot;canonical&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is also isomorphic with the first two, but since it has the optimal amount of bent edges (zero) and optimal angles (four right angles), it is also the canonical representation for these graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for further reading, you might consider &lt;a href=&quot;http://www.borgelt.net/papers/mgts_05.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, which discusses several algorithms used to find canonical representations for web mining, or &lt;a href=&quot;http://www.ecs.umass.edu/ece/labs/vlsicad/papers/dd-chapter.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, which discusses canonical representations of binary decision diagrams, word-level decision diagrams, binary moment diagrams, taylor expansion diagrams, and finite field decision diagrams.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;tl;dr:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;just have a look at the &lt;strong&gt;minion with bananas&lt;/strong&gt; &lt;a href=&quot;http://webhome.cs.uvic.ca/~wendym/courses/422/14/notes/422_14_group_theory.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;on slide 10 of this presentation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8953" LastEditorUserId="8953" LastEditDate="2015-05-22T17:05:05.333" LastActivityDate="2015-05-22T17:05:05.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5477" PostTypeId="2" ParentId="5471" CreationDate="2015-04-08T03:47:26.333" Score="5" Body="&lt;p&gt;In most things, related to &lt;strong&gt;R&lt;/strong&gt;, there are &lt;strong&gt;many approaches&lt;/strong&gt; to solve a problem, sometimes too many, I would say. The task of building a static website, using &lt;em&gt;RMarkdown&lt;/em&gt;, is not an exception.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the best, albeit somewhat brief, sets of workflows on the topic include the following one by Daniel Wollschlaeger, which includes &lt;a href=&quot;http://www.uni-kiel.de/psychologie/rexrepos/posts/rerWorkflowJN.html&quot;&gt;this workflow&lt;/a&gt;, based on &lt;em&gt;R&lt;/em&gt;, &lt;em&gt;nanoc&lt;/em&gt; and &lt;em&gt;Jekyll&lt;/em&gt;, as well as &lt;a href=&quot;http://www.uni-kiel.de/psychologie/rexrepos/posts/rerWorkflowWP.html&quot;&gt;this workflow&lt;/a&gt;, based on &lt;em&gt;R&lt;/em&gt; and &lt;em&gt;WordPress&lt;/em&gt;. Another good workflow is &lt;a href=&quot;http://jason.bryer.org/posts/2012-12-10/Markdown_Jekyll_R_for_Blogging.html&quot;&gt;this one&lt;/a&gt; by Jason Bryer, which is focused on &lt;em&gt;R(Markdown)&lt;/em&gt;, &lt;em&gt;Jekyll&lt;/em&gt; and &lt;em&gt;GitHub Pages&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not everyone likes &lt;em&gt;GitHub Pages&lt;/em&gt;, &lt;em&gt;Jekyll&lt;/em&gt;, &lt;em&gt;Octopress&lt;/em&gt; and &lt;em&gt;Ruby&lt;/em&gt;, so some people came up with &lt;em&gt;alternative solutions&lt;/em&gt;. For example, &lt;a href=&quot;http://www.znmeb.mobi/stories/blogging-with-rstudio-and-nikola.html&quot;&gt;this workflow&lt;/a&gt; by Edward Borasky is based on &lt;em&gt;R&lt;/em&gt; and, for a static website generator, on &lt;em&gt;Python&lt;/em&gt;-based &lt;em&gt;Nicola&lt;/em&gt; (instead of &lt;em&gt;Ruby&lt;/em&gt;-based &lt;em&gt;Jekyll&lt;/em&gt; or &lt;em&gt;nanoc&lt;/em&gt;). Speaking about &lt;strong&gt;static website generators&lt;/strong&gt;, there are tons of them, in various programming languages, so, if you want to experiment, check &lt;a href=&quot;http://staticsitegenerators.net&quot;&gt;this amazing website&lt;/a&gt;, listing almost all of them. Almost, because some are missing - for example, &lt;em&gt;Samantha&lt;/em&gt; and &lt;em&gt;Ghost&lt;/em&gt;, listed &lt;a href=&quot;http://ropensci.github.io/reproducibility-guide/sections/tools&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other interesting workflows include &lt;a href=&quot;http://joshualande.com/jekyll-github-pages-poole&quot;&gt;this one&lt;/a&gt; by Joshua Lande, which is based on Jekyll and GitHub Pages, but includes some nice examples of customization for integrating a website with Disqus, Google Analytics and Twitter as well as getting custom URL for the site and more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those who want a &lt;strong&gt;pure R-based static site solution&lt;/strong&gt;, now have some options, including &lt;code&gt;rsmith&lt;/code&gt; (&lt;a href=&quot;https://github.com/hadley/rsmith&quot;&gt;https://github.com/hadley/rsmith&lt;/a&gt;), a static site generator by Hadley Wickham, and &lt;code&gt;Poirot&lt;/code&gt; (&lt;a href=&quot;https://github.com/ramnathv/poirot&quot;&gt;https://github.com/ramnathv/poirot&lt;/a&gt;), a static site generator by Ramnath Vaidyanathan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I would like to mention an &lt;a href=&quot;http://notebook.madsenlab.org/labnotebook.html&quot;&gt;interesting project&lt;/a&gt; (from an &lt;strong&gt;open science&lt;/strong&gt; perspective) that I recently ran across - an open source software by Mark Madsen for a &lt;em&gt;lab notebook static site&lt;/em&gt;, which is based on &lt;em&gt;GitHub Pages&lt;/em&gt; and &lt;em&gt;Jekyll&lt;/em&gt;, but also supports &lt;em&gt;pandoc&lt;/em&gt;, &lt;em&gt;R&lt;/em&gt;, &lt;em&gt;RMarkdown&lt;/em&gt; and &lt;em&gt;knitr&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-04-08T03:47:26.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5479" PostTypeId="1" CreationDate="2015-04-08T07:17:40.713" Score="1" ViewCount="772" Body="&lt;p&gt;In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper &lt;a href=&quot;http://www.cise.ufl.edu/~mythai/files/mobicom_CS.pdf&quot; rel=&quot;nofollow&quot;&gt;Overlapping Communities in Dynamic Networks: Their Detection and Mobile Applications&lt;/a&gt;, LFR is used. But I don't know how this dataset is generated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pages.towson.edu/npnguyen/Databases/AFOCS.zip&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is the code of this paper and ground truth included is only for one snapshot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question: How is the dataset generated?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="8240" LastEditorUserId="5177" LastEditDate="2015-10-18T06:55:46.880" LastActivityDate="2024-01-04T04:07:15.840" Title="LFR as synthesized data for dynamic community detection methods" Tags="|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5480" PostTypeId="2" ParentId="5443" CreationDate="2015-04-08T07:20:29.317" Score="8" Body="&lt;p&gt;In his book Data Smart, John Foreman solves common data science problems (clustering, naive bayes, ensemble methods,...) using Excel. Indeed it's always good to have some knowledge of Python or R but I guess Excel can still get most of the job done !&lt;/p&gt;&#xA;" OwnerUserId="8498" LastActivityDate="2015-04-08T07:20:29.317" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5481" PostTypeId="1" CreationDate="2015-04-08T16:51:01.880" Score="4" ViewCount="93" Body="&lt;p&gt;Is there a recommended approach for storing processed data for testing new data products?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product to present to users, do the data processing to create it, and then put it in a data store that our application can then access easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm not sure about is what kind of data store would be good for this type of &quot;testing&quot; use case.  Since it would need to be flexible enough to handle different types of data products, like aggregates, windowed data, etc.  And ideally it wouldn't require a huge instrumentation process to try out new things.&lt;/p&gt;&#xA;" OwnerUserId="9015" LastActivityDate="2015-05-13T01:18:48.150" Title="Data store for testing data products?" Tags="|nosql|sql|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5483" PostTypeId="2" ParentId="5178" CreationDate="2015-04-08T19:21:29.893" Score="11" Body="&lt;p&gt;Try looking at &lt;a href=&quot;https://git-lfs.github.com/&quot; rel=&quot;noreferrer&quot;&gt;Git Large File Storage (LFS)&lt;/a&gt;. It is new, but might be the thing worth looking at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I see, &lt;a href=&quot;https://news.ycombinator.com/item?id=9343021&quot; rel=&quot;noreferrer&quot;&gt;a discussion on Hacker News&lt;/a&gt; mentions a few other ways to deal with large files:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://git-annex.branchable.com/git-annex/&quot; rel=&quot;noreferrer&quot;&gt;git-annex&lt;/a&gt; (and e.g. &lt;a href=&quot;http://git-annex.branchable.com/tips/using_Amazon_S3/&quot; rel=&quot;noreferrer&quot;&gt;using it with Amazon S3&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mercurial.selenic.com/wiki/LargefilesExtension&quot; rel=&quot;noreferrer&quot;&gt;Mercurual Largefiles extension&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="289" LastEditorUserId="289" LastEditDate="2015-04-08T20:24:07.617" LastActivityDate="2015-04-08T20:24:07.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5485" PostTypeId="1" AcceptedAnswerId="5507" CreationDate="2015-04-09T08:24:04.600" Score="1" ViewCount="129" Body="&lt;p&gt;I have :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;a matrix X with N lines&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;a vector Y&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've computed the Euclidean distance with Y for each line of X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I get is a vector of distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want is a vector of scores between 0 and 1, 1 meaning &quot;very&quot; high correlation, 0 meaning &quot;no&quot; correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here what I did :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I divided the vector of distances by the max distance inside it.&#xA;I get vector D. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1 - D&lt;/em&gt; is the final result with values between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I get many values (75%) too close to 1.&#xA;Do you think what I did is correct ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you get a better result ?&#xA;(Between 0 and 1 but not everything too close to 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I tried to take the square of the result. (To stay between 0 and 1 but to minimize the values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here a picture of the distance values I want to turn in a score&#xA;&lt;img src=&quot;https://i.stack.imgur.com/36LDh.png&quot; alt=&quot;distance values I want to turn in a score&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="2586" LastEditorUserId="2586" LastEditDate="2015-04-13T09:59:45.020" LastActivityDate="2015-04-27T07:29:06.110" Title="Correlations - Get values in the way we want" Tags="|machine-learning|data-mining|statistics|python|correlation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5488" PostTypeId="2" ParentId="5357" CreationDate="2015-04-09T12:21:08.427" Score="11" Body="&lt;p&gt;As Andre Holzner has said, extending R with C/C++ extension is a very good way to take advantage of the best of both sides. Also you can try the inverse , working with C++ and ocasionally calling function of R with the RInside package  o R. Here you can find how&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/RInside/index.html&quot;&gt;http://cran.r-project.org/web/packages/RInside/index.html&lt;/a&gt;&#xA;&lt;a href=&quot;http://dirk.eddelbuettel.com/code/rinside.html&quot;&gt;http://dirk.eddelbuettel.com/code/rinside.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you're working in C++ you have many libraries , many of them built up for specific problems, other more general&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.shogun-toolbox.org/page/features/&quot;&gt;http://www.shogun-toolbox.org/page/features/&lt;/a&gt;&#xA;&lt;a href=&quot;http://image.diku.dk/shark/sphinx_pages/build/html/index.html&quot;&gt;http://image.diku.dk/shark/sphinx_pages/build/html/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mlpack.org/&quot;&gt;http://mlpack.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8843" LastActivityDate="2015-04-09T12:21:08.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5489" PostTypeId="1" CreationDate="2015-04-09T14:27:36.663" Score="2" ViewCount="11579" Body="&lt;p&gt;I've made a logistic regression to combine two independent variables in &lt;code&gt;R&lt;/code&gt;, using &lt;code&gt;pROC&lt;/code&gt; package and I obtain this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; summary(fit)&#xA;&#xA;Call: glm(formula = Case ~ X + Y, family = &quot;binomial&quot;, data = data)&#xA;&#xA;Deviance Residuals: &#xA;  Min       1Q     Median     3Q      Max  &#xA;-1.5751  -0.8277  -0.6095   1.0701   2.3080  &#xA;&#xA;Coefficients:&#xA;             Estimate  Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept) -0.153731   0.538511  -0.285 0.775281    &#xA;X           -0.048843   0.012856  -3.799 0.000145 ***&#xA;Y            0.028364   0.009077   3.125 0.001780 ** &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;Null deviance: 287.44  on 241  degrees of freedom&#xA;Residual deviance: 260.34  on 239  degrees of freedom&#xA;AIC: 266.34&#xA;&#xA;Number of Fisher Scoring iterations: 4&#xA;&#xA;&amp;gt;     fit&#xA;&#xA;Call:  glm(formula = Case ~ X + Y, family = &quot;binomial&quot;, data = data)&#xA;&#xA;Coefficients:&#xA;  (Intercept)       X            Y  &#xA;   -0.15373     -0.04884      0.02836  &#xA;&#xA;Degrees of Freedom: 241 Total (i.e. Null);  239 Residual&#xA;Null Deviance:      287.4 &#xA;Residual Deviance:  260.3        AIC: 266.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I need to extract some information from this data and I'm not sure about how to do it. First, I need the model equation: suppose that fit is a combined predictor called &lt;code&gt;CP&lt;/code&gt;; could it be &lt;code&gt;CP=-0.15-0.05X+0.03Y&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the resulting combined predictor from the regression should present a median value, so that I can compare median from the two groups &lt;code&gt;Case&lt;/code&gt; and &lt;code&gt;Controls&lt;/code&gt; which I used to make the regression (in other words, my &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; variables are N-dimensional with &lt;code&gt;N = N1+N2&lt;/code&gt;, where &lt;code&gt;N1 = Number of Controls&lt;/code&gt;, for which &lt;code&gt;Case=0&lt;/code&gt;, and &lt;code&gt;N2 = Number of Cases&lt;/code&gt;, for which &lt;code&gt;Case=1&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="9033" LastEditorUserId="2452" LastEditDate="2015-04-10T01:55:02.780" LastActivityDate="2015-04-10T01:55:02.780" Title="Extracting model equation and other data from 'glm' function in R" Tags="|r|logistic-regression|glm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5490" PostTypeId="2" ParentId="5489" CreationDate="2015-04-09T15:27:53.033" Score="3" Body="&lt;p&gt;In order to extract some data from the fitted &lt;code&gt;glm&lt;/code&gt; model object, you need to figure out where that data resides (use documentation and &lt;code&gt;str()&lt;/code&gt; for that). Some data might be available from the &lt;code&gt;summary.glm&lt;/code&gt; object, while more detailed data is available from the &lt;code&gt;glm&lt;/code&gt; object itself. For extracting model parameters, you can use &lt;code&gt;coef()&lt;/code&gt; function or direct access to the structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Princeton's* introduction to R course's website, &lt;a href=&quot;http://data.princeton.edu/R/glms.html&quot; rel=&quot;nofollow&quot;&gt;GLM section&lt;/a&gt; - see for details &amp;amp; examples:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The functions that can be used to extract results from the fit&#xA;  include&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- 'residuals' or 'resid', for the deviance residuals&#xA;- 'fitted' or 'fitted.values', for the fitted values (estimated probabilities)&#xA;- 'predict', for the linear predictor (estimated logits)&#xA;- 'coef' or 'coefficients', for the coefficients, and&#xA;- 'deviance', for the deviance. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;Some of these functions have optional arguments; for example, you can&#xA;  extract five different types of residuals, called &quot;deviance&quot;,&#xA;  &quot;pearson&quot;, &quot;response&quot; (response - fitted value), &quot;working&quot; (the&#xA;  working dependent variable in the IRLS algorithm - linear predictor),&#xA;  and &quot;partial&quot; (a matrix of working residuals formed by omitting each&#xA;  term in the model). You specify the one you want using the type&#xA;  argument, for example residuals(lrfit,type=&quot;pearson&quot;).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;*) More accurately, this website is by Germán Rodríguez from Princeton University.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-04-10T00:31:03.853" LastActivityDate="2015-04-10T00:31:03.853" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="5491" PostTypeId="2" ParentId="5178" CreationDate="2015-04-10T08:37:18.263" Score="2" Body="&lt;p&gt;I haven't used them but there was a similar discussion in a finance group&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.nuclearphynance.com/Show%20Post.aspx?PostIDKey=175835&quot; rel=&quot;nofollow&quot;&gt;data repository software&lt;/a&gt;&#xA;suggestions &#xA;&lt;a href=&quot;http://paradigm4.com/forum/viewtopic.php?f=16&amp;amp;t=1596&quot; rel=&quot;nofollow&quot;&gt;scidb&lt;/a&gt;, zfs, &lt;a href=&quot;http://www.urbackup.org/&quot; rel=&quot;nofollow&quot;&gt;http://www.urbackup.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1256" LastActivityDate="2015-04-10T08:37:18.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5492" PostTypeId="1" CreationDate="2015-04-10T12:12:55.500" Score="1" ViewCount="1046" Body="&lt;p&gt;I am working for a logistics firm and there are approx. 750+ customers who avail our services. I am in the process of building and generating some insight for the business based on the payments made by these customers for the last 1 year. Some make payment on time whereas some are late &amp;amp; some are extremely late. Could you please advice which modeling technique or statistical approach would be best in this case. &#xA;I can think of creating clusters based on the payment history and highlights &amp;amp; placing all defaulters in one cluster where our company can focus. &#xA;PLease suggest. &lt;/p&gt;&#xA;" OwnerUserId="9052" LastActivityDate="2015-04-10T17:31:30.327" Title="Predictive Model for Customer Payment Pattern" Tags="|data-mining|statistics|clustering|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5493" PostTypeId="1" CreationDate="2015-04-10T12:42:13.517" Score="2" ViewCount="452" Body="&lt;p&gt;I understand that by filtering out the instances with labels that Random Forest trees are uncertain upon with their decisions, and model these with another classifier could give a better overall result. My question is, how can I &quot;combine&quot; two (or more) classifiers' classification on a single unlabeled dataset?&lt;/p&gt;&#xA;" OwnerUserId="9011" LastActivityDate="2015-04-12T21:54:11.407" Title="Post processing with Random Forest" Tags="|classification|random-forest|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5496" PostTypeId="2" ParentId="5492" CreationDate="2015-04-10T17:31:30.327" Score="1" Body="&lt;p&gt;Here are some things that come to mind: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) ARMA/ARIMA models: These may show if there is any seasonal trends in their payment history.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;a href=&quot;http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf&quot; rel=&quot;nofollow&quot;&gt;Logistic Regression&lt;/a&gt;: Will allow you model the conditional probability of a missed payments based off of number of previous missed payments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here you need to determine what the threshold for risk is since that's a little more subjective.&lt;/p&gt;&#xA;" OwnerUserId="9054" LastActivityDate="2015-04-10T17:31:30.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5502" PostTypeId="1" AcceptedAnswerId="5505" CreationDate="2015-04-11T17:45:26.600" Score="0" ViewCount="81" Body="&lt;p&gt;I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R&#xA;&lt;a href=&quot;http://www.kddcup2012.org/c/kddcup2012-track2&quot; rel=&quot;nofollow&quot;&gt;http://www.kddcup2012.org/c/kddcup2012-track2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems not possible to work with this 10GB training data on my local system with 4GB RAM.&#xA;Can anyone work on this data using this kind of a local system ? Or is using a cluster the norm ?&lt;br&gt;&#xA;It would be great if anyone could provide me with any guidance on how to get started with working on a cluster and the normally used type of cluster for such tasks&lt;/p&gt;&#xA;" OwnerUserId="4947" LastActivityDate="2015-04-12T05:28:59.763" Title="Possibility of working on KDDCup data in local system" Tags="|distributed|state-of-the-art|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5503" PostTypeId="1" CreationDate="2015-04-11T21:35:14.517" Score="7" ViewCount="1191" Body="&lt;p&gt;I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance.  By small I mean &lt;em&gt;few parameters&lt;/em&gt;, not &lt;em&gt;few arithmetic operations&lt;/em&gt; (=fast).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested primarily in convolutional neural networks for vision applications, using something simple like CIFAR-10 without augmentation as the benchmark.  Top-performing networks on CIFAR in recent years have had anywhere between 100 million and 0.7 million parameters (!!), so clearly small size is not (always) a bad thing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Small networks are also in general faster to train and overfit less.  Moreover, recent work on &lt;a href=&quot;http://arxiv.org/abs/1503.02531&quot; rel=&quot;nofollow&quot;&gt;Knowledge Distillation&lt;/a&gt;, &lt;a href=&quot;http://arxiv.org/abs/1412.6550&quot; rel=&quot;nofollow&quot;&gt;FitNets&lt;/a&gt;, etc show ways of making smaller networks from large networks while preserving most of the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another question is, what is the best performance achievable with a network no larger than a fixed size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples of especially small networks that get good performance (100k parameters with 10% on CIFAR, anyone?) or systematic studies of the size vs performance tradeoff would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="29169" LastEditDate="2022-06-16T19:08:55.603" LastActivityDate="2022-06-16T19:08:55.603" Title="Neural Network Golf: smallest network for a certain level of performance" Tags="|neural-network|convolutional-neural-network|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5504" PostTypeId="1" CreationDate="2015-04-12T02:30:05.370" Score="2" ViewCount="584" Body="&lt;p&gt;I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have finished applying my clustering techniques on my data set and the output of the clusters were the clusters of the states for each year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I can know from my results; which states are close to each other with the birth names and which were not. By looking &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to have interesting results to make my project report interesting, Which states are similar in birth names are not enough to make the read be excited about my project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;any ideas of what can be learned from my project?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I show the comparison between the clustering techniques? anyway other than seeing how Homogeneous the clusters are?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9079" LastEditorUserId="21" LastEditDate="2015-04-12T10:33:34.953" LastActivityDate="2016-08-11T09:39:56.553" Title="Data mining- Clustering techniques" Tags="|data-mining|clustering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5505" PostTypeId="2" ParentId="5502" CreationDate="2015-04-12T05:23:13.703" Score="1" Body="&lt;p&gt;I think that you have, at least, the following &lt;strong&gt;major options&lt;/strong&gt; for your &lt;em&gt;data analysis scenario&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;em&gt;big data-enabling&lt;/em&gt; &lt;code&gt;R&lt;/code&gt; packages on your &lt;strong&gt;local system&lt;/strong&gt;. You can find most of them via the corresponding &lt;em&gt;CRAN Task View&lt;/em&gt; that I reference in &lt;a href=&quot;https://datascience.stackexchange.com/a/780/2452&quot;&gt;this answer&lt;/a&gt; (see point #3).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the same packages on a &lt;strong&gt;public cloud&lt;/strong&gt; infrastructure, such as &lt;em&gt;Amazon Web Services (AWS) EC2&lt;/em&gt;. If your analysis is non-critical and tolerant to potential restarts, consider using &lt;em&gt;AWS Spot Instances&lt;/em&gt;, as their pricing allows for significant financial savings.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the above mention &lt;strong&gt;public cloud&lt;/strong&gt; option with &lt;code&gt;R&lt;/code&gt; &lt;em&gt;standard platform&lt;/em&gt;, but on &lt;strong&gt;more powerful instances&lt;/strong&gt; (for example, on AWS you can opt for memory-optimized EC2 instances or general purpose on-demand instances with more memory).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In some cases, it is possible to &lt;strong&gt;tune&lt;/strong&gt; a &lt;em&gt;local system&lt;/em&gt; (or a &lt;em&gt;cloud on-demand instance&lt;/em&gt;) to enable &lt;code&gt;R&lt;/code&gt; to work with big(ger) data sets. For some help in this regard, see &lt;a href=&quot;https://datascience.stackexchange.com/a/3731/2452&quot;&gt;my relevant answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For both above-mentioned cloud (AWS) options, you can find more convenient to use &lt;code&gt;R&lt;/code&gt;-focused &lt;strong&gt;pre-built VM images&lt;/strong&gt;. See &lt;a href=&quot;https://datascience.stackexchange.com/a/4927/2452&quot;&gt;my relevant answer&lt;/a&gt; for details. You may also find useful &lt;a href=&quot;https://github.com/onurakpolat/awesome-bigdata&quot; rel=&quot;nofollow noreferrer&quot;&gt;this excellent comprehensive list of big data frameworks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-04-12T05:28:59.763" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5506" PostTypeId="2" ParentId="5504" CreationDate="2015-04-12T15:36:25.557" Score="2" Body="&lt;p&gt;It's common to perform clustering on certain features, and then use characteristics such as mean differences between other features (not used in clustering) to show how clusters differ.  It's not always meaningful to say that clusters differ merely by lower distances between cluster centers and objects assigned to them (as in k-means).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, you might acquire other features (variables) for the states and then average each feature among objects assigned to each cluster.  Then show bar charts of averages for those variables among all the clusters to assist in explaining how the clusters differ.  You might also test for significantly different means of the supplemental features across clusters. &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-04-12T15:36:25.557" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5507" PostTypeId="2" ParentId="5485" CreationDate="2015-04-12T16:01:40.447" Score="2" Body="&lt;p&gt;Several kernel functions can serve as similarity functions (=scores). See a list, for example, &lt;a href=&quot;https://www.otexts.org/1560&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. You can try several of them and see which suits you the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need something that drops fast at low distances. You can try&#xA;$$ score = 1/(1+distance)^2$$&#xA;and adjust coefficient in front of distance so that the score fits between 0 and 1&#xA;&lt;img src=&quot;https://i.stack.imgur.com/oufxf.png&quot; alt=&quot;enter image description here&quot;&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About your picture: what are axis labels? and what are x-ticks?&lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2015-04-26T06:35:34.637" LastActivityDate="2015-04-26T06:35:34.637" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5508" PostTypeId="2" ParentId="736" CreationDate="2015-04-12T16:03:49.887" Score="1" Body="&lt;p&gt;Regarding the approach, SVM with an RBF kernel does a good job, but SVMs can be slowed down by large object sizes, unless you are employing CV with e.g. one tenth of the data randomly assigned to each fold.  However, did you ask yourself why you are employing SVMs in the first place?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you tried multivariate linear regression, $\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}$, where each record of $\mathbf{Y}$ is coded $y_{ij}=+1$ if the $i$th object is in class $j$, and $y_{ij}=-1$ otherwise?  If the classification accuracy is appreciably high using linear regression, then your data are linearly separable, and more complex methods such as SVMs and ANNs aren't needed.  Step 2 would be to show that k-nearest neighbor, naive Bayes, linear (Fisher) discriminant analysis, polytomous logistic regression, etc., break down and fail. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For terminology, you might couch the issue of having more class weights in the context of &quot;lower proportions of objects in certain classes,&quot; or &quot;near-zero class size.&quot;  Skew tends to be used for describing the distribution of a feature's values, as in skewness, fat tails, etc.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many features do you have?  Did you try unsupervised clustering (class discovery) on the 100,000 objects before trying supervised classification (class prediction) with SVM?  Maybe the 100,000 objects can be grouped into fewer classes than 50, for which the new class membership could be used as the target class during classification analysis.  This may alleviate the problem of having near-zero class size.&lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-04-12T16:03:49.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5509" PostTypeId="2" ParentId="653" CreationDate="2015-04-12T16:08:13.467" Score="1" Body="&lt;p&gt;Probably kNN and naive Bayes classifier.  kNN is very fast, but NBC can break down a lot.   Linear regression is also one-step solution that does not involve gradient-based learning, so that might help.  LDA is your next choice. &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-04-12T16:08:13.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5510" PostTypeId="2" ParentId="384" CreationDate="2015-04-12T16:12:39.063" Score="2" Body="&lt;p&gt;Wouldn't recommend use of complex methods first.  Use faster simple approaches initially (kNN, NBC, etc.), then progress through linear regression, logistic regression, LDA, CART(RF), KREG, and then to least squares SVM, gradient ascent SVM, ANNs, and then metaheurustics (greedy heuristic hill climbing with GAs, swarm intelligence, ant colony optimization, etc.)&lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-04-12T16:12:39.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5511" PostTypeId="2" ParentId="310" CreationDate="2015-04-12T16:24:28.790" Score="2" Body="&lt;p&gt;There are three approaches you could take: APP, OAA, AAO.  APP is discrimination between objects based on all possible pairs of classes.  OAA is use of one-against all (remaining classes), and AAO is all at once such as use of an F-test for multiple classes simultaneously (or Hotelling's test for MVN).  APP and AAO are actually multiple binary classification runs but with more than two of your original classes.  Each of these approaches yields different results for various classifiers employed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random sampling is a good technique.   You might also try to cluster all of the objects into centers using k-means, and then use the centers as new objects.  Either way, linear and non-linear dimension reduction methods might help get away from the large sample size.     &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-04-12T16:24:28.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5512" PostTypeId="2" ParentId="5493" CreationDate="2015-04-12T21:54:11.407" Score="2" Body="&lt;p&gt;Here's a solution using the caret package for R. A Random Forest is first trained on the data. All observations for which the probability (from the voting) is less than 99% are then passed to model 2, linear discriminant analysis. Only the probabilities from unseen resampling observations are used, since the Random Forest will otherwise fit the training data perfectly. That is what caret is needed for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy is a little higher for the uncertain cases, but this is probably overfitting since I have tried several different models and the data set is small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to know if this really improves the performance in your application for the out of sample test data. Are there any papers that recommend this approach? This approach seems to resemble boosting. I tried it on some of my data but could not improve the out of sample performance I got from model 1 (a Random Forest).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(iris)&#xA;library(caret)&#xA;myTrainControl &amp;lt;- trainControl(method = &quot;cv&quot;, number = 10,&#xA;                               savePredictions = T,&#xA;                               classProbs = TRUE)&#xA;set.seed(4213) # To get the same resamples every time&#xA;M1 &amp;lt;- train(y = iris$Species,&#xA;            x = iris[, !(names(iris) == &quot;Species&quot;)],&#xA;            method = &quot;rf&quot;,&#xA;            trControl = myTrainControl)&#xA;&#xA;M1pred &amp;lt;- M1$pred[M1$pred$mtry == 2, ]&#xA;confusionMatrix(M1pred$pred, M1pred$obs)&#xA;# Accuracy 96%&#xA;&#xA;# Inspect the predicted class probabilities:&#xA;probs &amp;lt;- cbind(M1pred$setosa,&#xA;                   M1pred$versicolor,&#xA;               M1pred$virginica)&#xA;colnames(probs) &amp;lt;- c(&quot;setosa&quot;, &quot;versicolor&quot;, &quot;virginica&quot;)&#xA;maxCol &amp;lt;- max.col(probs)&#xA;probs &amp;lt;- cbind(probs, maxCol)&#xA;maxProbs &amp;lt;- apply(probs, 1, function(x) x[x[&quot;maxCol&quot;]])&#xA;summary(maxProbs)&#xA;# Min. 1st Qu.  Median    Mean 3rd Qu.    Max.&#xA;# 0.5120  0.9805  0.9980  0.9645  1.0000  1.0000&#xA;&#xA;# Let's define anything below 99% as uncertain:&#xA;uncertain &amp;lt;- which(maxProbs &amp;lt; 0.99)&#xA;length(uncertain) # 45&#xA;# Find rows in iris data that belong to 'uncertain' cases:&#xA;uncertainIndex &amp;lt;- M1pred$rowIndex[uncertain]&#xA;    confusionMatrix(M1pred$pred[uncertain],&#xA;                reference = M1pred$obs[uncertain])&#xA;# M1 Performance in uncertain cases:&#xA;#              Reference&#xA;# Prediction   setosa versicolor virginica&#xA;# setosa          5          0         0&#xA;# versicolor      0         17         3&#xA;# virginica       0          3        17&#xA;#&#xA;# Overall Statistics&#xA;# Accuracy : 0.8667&#xA;&#xA;# Train new model on uncertain data only:&#xA;irisUncertain &amp;lt;- iris[uncertainIndex, ]&#xA;set.seed(4213) # To get the same resamples every time&#xA;M2 &amp;lt;- train(y = irisUncertain$Species,&#xA;            x = irisUncertain[, !(names(irisUncertain) == &quot;Species&quot;)],&#xA;            method = &quot;lda&quot;,&#xA;            trControl = myTrainControl)&#xA;&#xA;M2pred &amp;lt;- M2$pred&#xA;&#xA;confusionMatrix(M2pred$pred, reference = M2pred$obs)&#xA;#              Reference&#xA;# Prediction   setosa versicolor virginica&#xA;# setosa          4          0         0&#xA;# versicolor      1         16         0&#xA;# virginica       0          4        20&#xA;# &#xA;# Overall Statistics&#xA;# Accuracy : 0.8889 &#xA;# (Small discrepancy: Why does caret report an accuracy of 88,5% for M2?)&#xA;&#xA;# For new data predictions can be made as follows:&#xA;# (just as an example from the original data again)&#xA;# Some 'uncertain' cases are 'certain' now using the full M2 model&#xA;newdat &amp;lt;- irisUncertain[30:34, -5]&#xA;M1maxProbs &amp;lt;- apply(predict(M1, newdat, type = &quot;prob&quot;), 1, max)&#xA;ifelse(M1maxProbs &amp;lt; 0.99,&#xA;       paste(&quot;M2:&quot;, predict(M2, newdat)),&#xA;       paste(&quot;M1:&quot;, predict(M1, newdat)))&#xA;# 85               94              107              126 &#xA;# &quot;M2: versicolor&quot; &quot;M1: versicolor&quot;  &quot;M2: virginica&quot;  &quot;M1: virginica&quot; &#xA;# 71 &#xA;# &quot;M2: virginica&quot;  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5377" LastActivityDate="2015-04-12T21:54:11.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5513" PostTypeId="2" ParentId="5481" CreationDate="2015-04-13T00:22:32.793" Score="2" Body="&lt;p&gt;You might try Azure Table Storage.  Since you can't lock yourself down to a specific schema (since one data product might be aggregates whereas another might be time series or something else), Azure Table storage would give you the flexibility of storing data from multiple sources, each having their own format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would also lend itself to making a system highly scalable, as you could use Azure Service Bus in conjunction with Azure Table Storage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might check out this tutorial at Pluralsight, &lt;a href=&quot;http://www.pluralsight.com/courses/applied-windows-azure&quot; rel=&quot;nofollow&quot;&gt;Applied Windows Azure&lt;/a&gt;, as it shows a number of examples, one using Table Storage and Service Bus, another using Hadoop, and I suspect that some of these might match the extensibility you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="8284" LastActivityDate="2015-04-13T00:22:32.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5514" PostTypeId="1" AcceptedAnswerId="5515" CreationDate="2015-04-13T08:45:07.773" Score="0" ViewCount="109" Body="&lt;p&gt;PROC means data=d mean; &#xA;var a;&#xA;class b; var a;&#xA;run; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to perform the &quot;PROC means&quot; for continuous &quot;var a&quot;:&#xA;1) in general and&#xA;2) by classes.&#xA;But it performed by the classes only.&#xA;How to make procedure for &quot;var a&quot; here in general too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. SAS WARNING: Analysis variable &quot;a&quot; was defined in a previous statement, duplicate definition will be ignored.&lt;/p&gt;&#xA;" OwnerUserId="8951" LastEditorUserId="8951" LastEditDate="2015-04-13T08:54:34.413" LastActivityDate="2015-04-13T10:52:02.360" Title="SAS PROC means (two variants together)" Tags="|processing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5515" PostTypeId="2" ParentId="5514" CreationDate="2015-04-13T08:59:00.397" Score="0" Body="&lt;p&gt;Just run &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PROC means data=d mean; var a; run;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for overall data&lt;/p&gt;&#xA;" OwnerUserId="735" LastActivityDate="2015-04-13T08:59:00.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5517" PostTypeId="2" ParentId="5514" CreationDate="2015-04-13T10:52:02.360" Score="0" Body="&lt;p&gt;PROC means data=d mean; var a; run;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PROC means data=d mean; class b; var a; run;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I knew that can be write so separately!&#xA;But I thought, it is can be write somehow together!&lt;/p&gt;&#xA;" OwnerUserId="8951" LastActivityDate="2015-04-13T10:52:02.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5519" PostTypeId="1" AcceptedAnswerId="5527" CreationDate="2015-04-13T16:23:14.553" Score="0" ViewCount="389" Body="&lt;p&gt;I have a new data point and want to classify it into the existing classes.I can calculate pairwise distance for the new point to all existing points(in the existing classes). I know using KNN would be a straightforward to classify this point. Is there a way I could randomly sampling existing classes and then correlated the new point to a potential classes without calculating all pairwise distances? &lt;/p&gt;&#xA;" OwnerUserId="9097" LastActivityDate="2018-11-14T09:53:08.260" Title="about predict the class for a new datapoint" Tags="|data-mining|classification|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5521" PostTypeId="2" ParentId="5519" CreationDate="2015-04-13T23:54:08.857" Score="0" Body="&lt;p&gt;Since you have your clusters set up already you should be able to calculate cluster centroids and then determine the distance between a each cluster centroid and the new data point. You could follow the same process with random samples of each cluster. &lt;/p&gt;&#xA;" OwnerUserId="9054" LastActivityDate="2015-04-13T23:54:08.857" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5522" PostTypeId="5" CreationDate="2015-04-14T01:54:02.727" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt; is a general-purpose, dynamic, strongly typed language with many 3rd-party libraries for data science applications. There are two versions currently in wide use: 2 and 3. Python 2 is the &quot;old&quot; version, with no new versions being released beyond 2.7, save bugfixes. Python 3 is the &quot;new&quot; version, with active development. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python syntax is relatively easy to comprehend compared to other languages. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;numbers = [1, 2, 5, 8, 9]&#xA;for number in numbers:&#xA;    print(&quot;Hello world #&quot;, number)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Python has a clean look due to its regulatory approach to &lt;a href=&quot;http://en.wikipedia.org/wiki/Whitespace_%28programming_language%29&quot; rel=&quot;nofollow&quot;&gt;whitespace&lt;/a&gt;. While seemingly restrictive, it allows all Python code to look similar, which makes inspecting code much more predictable. All loops and conditionals (&lt;code&gt;for&lt;/code&gt;, &lt;code&gt;while&lt;/code&gt;, &lt;code&gt;if&lt;/code&gt;, etc) must be indented for the code block that follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Popular scientific and data science packages include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;Numpy&lt;/a&gt; - A fast, N-dimensional array library; the foundation for&#xA;all things scientific Python.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;Scipy&lt;/a&gt; - Numerical analysis built on Numpy. Allows for optimization, linear algebra, Fourier Transforms and much else. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Pandas&lt;/a&gt; (PANel DAta) - A fast and extremely flexible package that is very useful for data exploration. It handles NaN data well as well as fast indexing. Handles a wide variety of external data types and file formats.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9102" LastEditorUserId="9102" LastEditDate="2015-04-14T15:22:52.737" LastActivityDate="2015-04-14T15:22:52.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5523" PostTypeId="4" CreationDate="2015-04-14T01:54:02.727" Score="0" Body="Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow). " OwnerUserId="9102" LastEditorUserId="138209" LastEditDate="2022-07-18T06:35:48.613" LastActivityDate="2022-07-18T06:35:48.613" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5524" PostTypeId="1" CreationDate="2015-04-14T07:17:08.857" Score="-2" ViewCount="1648" Body="&lt;p&gt;Assume the training data is &lt;code&gt;fruit&lt;/code&gt;, which I am going to use for prediction in a CART model in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fruit = data.frame(color=c(&quot;red&quot;, &quot;red&quot;, &quot;red&quot;, &quot;yellow&quot;, &quot;yellow&quot;,&#xA;                           &quot;orange&quot;, &quot;green&quot;, &quot;pink&quot;, &quot;red&quot;, &quot;red&quot;),&#xA;                   isApple=c(TRUE, TRUE, FALSE, FALSE, FALSE,&#xA;                             FALSE, FALSE, FALSE, TRUE, FALSE))&#xA;mod = rpart(isApple ~ color, data=fruit, method=&quot;class&quot;, minbucket=1)&#xA;prp(mod)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could anyone explain what is exactly the role of &lt;code&gt;minbucket&lt;/code&gt; in plotting CART tree for this example if we are going to use &lt;code&gt;minbucket = 2, 3, 4, 5&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say &lt;code&gt;fruit&lt;/code&gt; is my data frame, I'm finding whether the outcome is apple or not? I have 5 red apples (4 TRUE, 1 FALSE), one FALSE value is tomato here, so what ever is red need not be an apple. But if I give &lt;code&gt;minbucket&lt;/code&gt;=5 or 4 here, there is no split at all. Only for &lt;code&gt;minbucket&lt;/code&gt; 1 to 3 there is a split beyond 3 there is no split. But I have more than 3 observation in my leaf node.&lt;/p&gt;&#xA;" OwnerUserId="9105" LastEditorUserId="2961" LastEditDate="2015-04-14T15:22:19.143" LastActivityDate="2015-04-14T15:50:49.573" Title="Understanding of minbucket function in CART model using R" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5525" PostTypeId="2" ParentId="5212" CreationDate="2015-04-14T08:13:07.223" Score="0" Body="&lt;p&gt;Geop Solutions write blogs related to bigdata and CRM on weekly basis. You can read their blog at &lt;a href=&quot;https://geopsolutions.wordpress.com/&quot; rel=&quot;nofollow&quot;&gt;https://geopsolutions.wordpress.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9106" LastActivityDate="2015-04-14T08:13:07.223" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5526" PostTypeId="2" ParentId="5524" CreationDate="2015-04-14T15:50:49.573" Score="0" Body="&lt;p&gt;Reading the documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minbucket: the minimum number of observations in any terminal ‘&amp;lt;leaf&amp;gt;’&#xA;          node.  If only one of ‘minbucket’ or ‘minsplit’ is specified,&#xA;          the code either sets ‘minsplit’ to ‘minbucket*3’ or&#xA;          ‘minbucket’ to ‘minsplit/3’, as appropriate.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So if you set &lt;code&gt;minbucket&lt;/code&gt; to 4, then &lt;code&gt;minsplit&lt;/code&gt; is 12, and you've only got 10 observations so it refuses to split. Because:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minsplit: the minimum number of observations that must exist in a node&#xA;          in order for a split to be attempted.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Its all there in the documentation.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2015-04-14T15:50:49.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5527" PostTypeId="2" ParentId="5519" CreationDate="2015-04-14T17:17:57.647" Score="0" Body="&lt;p&gt;I think you need to take a step back and figure out what you're trying to do at a higher level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How were the existing classes built? If they were built by clustering unlabeled data, then with this new data point you're continuing with the clustering process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the existing classes are labeled data, then k-NN is one possible classification method, and there are plenty more (decision trees, naive bayes, neural networks, etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're doing clustering, then there are several ways of assigning a point to a cluster, among measuring the distances from the point to cluster centroids is one. There's also single-linkage (distance is min of distances from point to points of cluster), complete-linkage (max of distances). These different methods will give clusters with different shapes and there's no universally best approach. You could test them with points that are already in clusters... but then if you're certain of what classes they re in, then you have a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So... if it's classification, then you can use k-NN, that's similar to the idea of assigning a point to a cluster according to distance. But it's not defined as finding the nearest cluster, it's defined as finding the classes of the k nearest points, then applying a vote or something. 1-NN is basically like single-linkage clustering. kNN does require finding the most similar (training) data points to your new data point. Sampling is definitely sub-optimal, but it may be good enough if you classes are well separated. If the cost of calculating distances is high, then one way of reducing the cost of calculation is the idea of skyline clustering: use a cheap distance metric to determine a subset of points that are likely to be among the k nearest neighbours, then compute these neighbours using the more expensive distance metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you will be classifying many points and not updating your model, it may be worth training a model (e.g. a decision tree) on the existing classes.&lt;/p&gt;&#xA;" OwnerUserId="9114" LastActivityDate="2015-04-14T17:17:57.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5528" PostTypeId="1" CreationDate="2015-04-14T20:44:56.423" Score="1" ViewCount="369" Body="&lt;p&gt;I'm conjecturing that with &lt;a href=&quot;http://en.wikipedia.org/wiki/Complete-linkage_clustering&quot; rel=&quot;nofollow&quot;&gt;Complete-linkage clustering&lt;/a&gt; two elements from the same cluster will always be closer to each other some other element from another cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more formal terms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $C$ be a clustering.&#xA; $\not\exists z \in C_j$ s.t. $\bigtriangleup(x, z) &amp;lt; \bigtriangleup(x, y)$ where $x,y \in C_i$, $C_i \neq C_j$ and $C_i, C_j \in C$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't been able to prove the conjecture yet, thus I'm wondering whether I'm right or wrong. If this is indeed the case, I would much appreciate a sketch a proof. I'm pretty sure I can work my way from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note (not that I think it makes a difference), I'll be applying the clustering algorithm on a one-dimensinal dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your input is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="1386" LastActivityDate="2015-05-23T09:12:42.480" Title="Complete link clustering" Tags="|machine-learning|clustering|algorithms|hierarchical-data-format|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5529" PostTypeId="1" AcceptedAnswerId="5531" CreationDate="2015-04-15T11:47:44.013" Score="6" ViewCount="7494" Body="&lt;p&gt;I have a pandas DataFrame containing a time series column. The years are shifted in the past, so that I have to add a constant number of years to every element of that column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best way I found is to iterate through all the records and use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x.replace(year=x.year + years)  # x = current element, years = years to add&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is cythonized as below, but still very slow (proofing)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cdef list _addYearsToTimestamps(list elts, int years):&#xA;    cdef cpdatetime x&#xA;    cdef int i&#xA;    for (i, x) in enumerate(elts):&#xA;        try:&#xA;            elts[i] = x.replace(year=x.year + years)&#xA;        except Exception as e:&#xA;            logError(None, &quot;Cannot replace year of %s - leaving value as this: %s&quot; % (str(x), repr(e)))&#xA;    return elts&#xA;&#xA;def fixYear(data):&#xA;    data.loc[:, 'timestamp'] = _addYearsToTimestamps(list(data.loc[:, 'timestamp']), REAL_YEAR-(list(data[-1:]['timestamp'])[0].year))&#xA;    return data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that there is a way to change the year without iterating, by using Pandas's Timestamp features. Unfortunately, I don't find how. Could someone elaborate?&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorDisplayName="user10014" LastEditDate="2015-06-06T16:09:58.373" LastActivityDate="2019-10-22T17:07:24.760" Title="Pandas time series optimization problem: add year" Tags="|python|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5530" PostTypeId="1" CreationDate="2015-04-15T13:10:08.470" Score="1" ViewCount="166" Body="&lt;p&gt;I don't know, if I can ask this here, but I'm interested to know what kind of &lt;em&gt;abstract data type (ADT)&lt;/em&gt; does Twitter use to &lt;strong&gt;model relations&lt;/strong&gt; between profiles and why. I'm just starting to learn ADTs, so I would like to learn how they work in the real world as well as their applications.&lt;/p&gt;&#xA;" OwnerUserId="9125" LastEditorUserId="2452" LastEditDate="2015-04-19T21:33:32.713" LastActivityDate="2015-04-19T21:33:32.713" Title="Abstract data type?" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5531" PostTypeId="2" ParentId="5529" CreationDate="2015-04-15T13:37:52.470" Score="2" Body="&lt;p&gt;Make a pandas Timedelta object then add with the += operator:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = pandas.Timedelta(days=365)&#xA;mydataframe.timestampcolumn += x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the key is to store your time series as timestamps.  To do that, use the pandas &lt;code&gt;to_datetime&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mydataframe['timestampcolumn'] = pandas.to_datetime(x['epoch'], unit='s')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;assuming you have your timestamps as epoch seconds in the dataframe &lt;code&gt;x&lt;/code&gt;.  That's not a requirement of course; see the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/dev/generated/pandas.tseries.tools.to_datetime.html&quot; rel=&quot;nofollow&quot;&gt;to_datetime&lt;/a&gt; documentation for converting other formats.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2015-04-15T17:47:53.533" LastActivityDate="2015-04-15T17:47:53.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5533" PostTypeId="2" ParentId="5530" CreationDate="2015-04-15T18:04:28.930" Score="2" Body="&lt;p&gt;&lt;strong&gt;Any&lt;/strong&gt; platform, focused on &lt;em&gt;social networking&lt;/em&gt; (not necessarily Twitter), at its &lt;em&gt;core&lt;/em&gt; uses the most appropriate and natural &lt;em&gt;abstract data type (ADT)&lt;/em&gt; for such domain - a &lt;a href=&quot;http://en.wikipedia.org/wiki/Graph_%28abstract_data_type%29&quot; rel=&quot;nofollow&quot;&gt;graph data structure&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use Python, you can check nice &lt;a href=&quot;http://networkx.github.io&quot; rel=&quot;nofollow&quot;&gt;NetworkX package&lt;/a&gt;, used for &quot;the creation, manipulation, and study of the structure, dynamics, and functions of complex networks&quot;. Of course, there are many other software tools for various programming languages for building, using and analyzing &lt;em&gt;network structures&lt;/em&gt;. You might also find useful the relevant book &lt;a href=&quot;http://shop.oreilly.com/product/0636920020424.do&quot; rel=&quot;nofollow&quot;&gt;&quot;Social Network Analysis for Startups: Finding connections on the social web&quot;&lt;/a&gt;, which provides a nice introduction into the &lt;em&gt;social network analysis (SNA)&lt;/em&gt; and uses the above-mentioned &lt;em&gt;NetworkX&lt;/em&gt; software for SNA examples. P.S. I have no affiliation whatsoever with &lt;em&gt;NetworkX&lt;/em&gt; open source project or the book's authors.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-04-15T18:11:36.323" LastActivityDate="2015-04-15T18:11:36.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5534" PostTypeId="1" CreationDate="2015-04-15T23:53:13.957" Score="12" ViewCount="31403" Body="&lt;p&gt;I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape &lt;a href=&quot;http://www.imdb.com/search/title?sort=num_votes,desc&amp;amp;start=1&amp;amp;title_type=feature&amp;amp;year=1950,2012&quot; rel=&quot;nofollow noreferrer&quot;&gt;imdb webpage&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am using BeautifulSoup module. Following is the code I am using:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;r = requests.get(url) # where url is the above url    &#xA;bs = BeautifulSoup(r.text)&#xA;for movie in bs.findAll('td','title'):&#xA;    title = movie.find('a').contents[0]&#xA;    genres = movie.find('span','genre').findAll('a')&#xA;    genres = [g.contents[0] for g in genres]&#xA;    runtime = movie.find('span','runtime').contents[0]&#xA;    year = movie.find('span','year_type').contents[0]&#xA;    print title, genres,runtime, rating, year&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am getting the following outputs:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.&lt;/p&gt;&#xA;&lt;p&gt;Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="98307" LastEditDate="2020-08-03T01:00:18.697" LastActivityDate="2021-01-19T02:24:57.500" Title="How to scrape imdb webpage?" Tags="|data-mining|python|scraping|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="5535" PostTypeId="2" ParentId="5529" CreationDate="2015-04-16T07:38:38.360" Score="1" Body="&lt;p&gt;Adapted from Pete's answer, here's an implementation of the solution, and the demonstration.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python3&#xA;&#xA;import random&#xA;import pandas&#xA;import time&#xA;import datetime&#xA;&#xA;def getRandomDates(n):&#xA;    tsMin = time.mktime(time.strptime(&quot;1980-01-01 00:00:00&quot;, &quot;%Y-%m-%d %H:%M:%S&quot;))&#xA;    tsMax = time.mktime(time.strptime(&quot;2005-12-31 23:59:59&quot;, &quot;%Y-%m-%d %H:%M:%S&quot;))&#xA;    return pandas.Series([datetime.datetime.fromtimestamp(tsMin + random.random() * (tsMax - tsMin)) for x in range(0, n)])&#xA;&#xA;def setMaxYear(tss, target):&#xA;    maxYearBefore = tss.max().to_datetime().year&#xA;    # timedelta cannot be given in years, so we compute the number of days to add in the next line&#xA;    deltaDays = (datetime.date(target, 1, 1) - datetime.date(maxYearBefore, 1, 1)).days&#xA;    return tss + pandas.Timedelta(days=deltaDays)&#xA;&#xA;data = pandas.DataFrame({'t1': getRandomDates(1000)})&#xA;data['t2'] = setMaxYear(data['t1'], 2015)&#xA;data['delta'] = data['t2'] - data['t1']&#xA;print(data)&#xA;print(&quot;delta min: %s&quot; % str(min(data['delta'])))&#xA;print(&quot;delta max: %s&quot; % str(max(data['delta'])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2015-04-16T07:38:38.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5536" PostTypeId="2" ParentId="5534" CreationDate="2015-04-16T10:07:48.187" Score="3" Body="&lt;p&gt;You can get everything from div with &lt;code&gt;class=&amp;quot;rating rating-list&amp;quot;&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;All you need to do is retrive attribute id: &lt;code&gt;[id=&amp;quot;tt1345836|imdb|8.5|8.5|advsearch&amp;quot;]&lt;/code&gt;&#xA;When you have this content, you split this string by &lt;code&gt;'|'&lt;/code&gt;, and you get:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;parameter: movie id&lt;/li&gt;&#xA;&lt;li&gt;parameter: movie score&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="98307" LastEditDate="2020-08-16T16:54:01.390" LastActivityDate="2020-08-16T16:54:01.390" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5537" PostTypeId="1" AcceptedAnswerId="5538" CreationDate="2015-04-16T10:12:56.220" Score="0" ViewCount="437" Body="&lt;p&gt;I'm using the &lt;a href=&quot;http://cran.r-project.org/web/packages/biglm/biglm.pdf&quot; rel=&quot;nofollow&quot;&gt;biglm&lt;/a&gt; R package for linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;click/impression is the output required. But the test data does not contain click and impression. &#xA;The &lt;code&gt;predict&lt;/code&gt; function of &lt;code&gt;biglm&lt;/code&gt; gives the error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in eval(expr, envir, enclos) : object 'click' not found&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assume this is because &lt;code&gt;predict&lt;/code&gt; tries to compute the standard errors also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a method to just obtain the predictions ? I tried assigning values to &lt;code&gt;se.fit&lt;/code&gt; and &lt;code&gt;type&lt;/code&gt; attributes, but I get the same error. &lt;/p&gt;&#xA;" OwnerUserId="4947" LastEditorUserId="2961" LastEditDate="2015-04-16T15:22:44.657" LastActivityDate="2015-04-16T15:22:44.657" Title="Finding predictions using biglm without finding errors" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5538" PostTypeId="2" ParentId="5537" CreationDate="2015-04-16T14:34:23.550" Score="2" Body="&lt;p&gt;biglm calls model.frame which &quot;all the variables in the formula are included in the data frame&quot; see documentation for model.frame. This is the issue that comes up when predict is called on the biglm class. It looks for those values in the predict function. To get around this you can just create a variable and encode it with 0. See below...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(trees)&#xA;ff&amp;lt;-log(Volume)~log(Girth)+log(Height)&#xA;&#xA;chunk1&amp;lt;-trees[1:10,]&#xA;chunk2&amp;lt;-trees[11:20,]&#xA;chunk3&amp;lt;-trees[21:31,]&#xA;&#xA;a &amp;lt;- biglm(ff,chunk1)&#xA;summary(a)&#xA;&#xA;#produces same error&#xA;chunk2 &amp;lt;- select(chunk2, -Volume)&#xA;predict(a, chunk2)&#xA;&#xA;#Fixed&#xA;chunk2$Volume &amp;lt;- 0&#xA;predict(a, chunk2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9054" LastActivityDate="2015-04-16T14:34:23.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5539" PostTypeId="2" ParentId="5534" CreationDate="2015-04-16T15:57:01.270" Score="9" Body="&lt;p&gt;I have been able to figure out a solution. I thought of posting just in case it is of any help to anyone or if somebody wants to suggest something different. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bs = BeautifulSoup(r.text)&#xA;for movie in bs.findAll('td','title'):&#xA;    title = movie.find('a').contents[0]&#xA;    genres = movie.find('span','genre').findAll('a')&#xA;    genres = [g.contents[0] for g in genres]&#xA;    runtime = movie.find('span','runtime').contents[0]&#xA;    rating = movie.find('span','value').contents[0]&#xA;    year = movie.find('span','year_type').contents[0]&#xA;    imdbID = movie.find('span','rating-cancel').a['href'].split('/')[2]&#xA;    print title, genres,runtime, rating, year, imdbID&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. 9.3 (1994) tt0111161&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="29575" LastEditDate="2018-01-14T01:47:19.083" LastActivityDate="2018-01-14T01:47:19.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5541" PostTypeId="2" ParentId="2523" CreationDate="2015-04-16T19:02:45.040" Score="1" Body="&lt;p&gt;The Wikipedia entry for &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;Receiver operating characteristic&lt;/a&gt; references this paper for the Gini=2AUROC-1 result: Hand, David J.; and Till, Robert J. (2001); A simple generalization of the area under the ROC curve for multiple class classification problems, Machine Learning, 45, 171–186.  But I'm afraid I don't have easy access to it to see how close it comes to what you want.&lt;/p&gt;&#xA;" OwnerUserId="9146" LastActivityDate="2015-04-16T19:02:45.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5542" PostTypeId="2" ParentId="2643" CreationDate="2015-04-16T19:24:55.447" Score="1" Body="&lt;p&gt;As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: &lt;a href=&quot;https://greenido.wordpress.com/2014/05/13/spark-cluster-on-google-compute-engine/&quot; rel=&quot;nofollow&quot;&gt;Spark Cluster on Google Compute Engine | Ido Green&lt;/a&gt;.  It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that it uses gcutil, which is deprecated in favor of &lt;a href=&quot;https://cloud.google.com/compute/docs/gcloud-compute/&quot; rel=&quot;nofollow&quot;&gt;gcloud compute&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9146" LastActivityDate="2015-04-16T19:24:55.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5543" PostTypeId="1" CreationDate="2015-04-16T21:11:14.103" Score="1" ViewCount="719" Body="&lt;p&gt;I noticed a difference between R and Excel when it comes to trend lines.  Basically I am automating a simple process that someone does.  He/she takes web traffic and finds the best trend line by looking at the R squared to predict what the web traffic will be a few months from now to make sure we will have the servers to handle it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, one can do: lm(log(y)~x,data) but you must &quot;remember&quot; to do &lt;b&gt;exp&lt;/b&gt;(predict(...)).  Is there a better predict function that remembers the left hand operations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the R-squared reported for log(y)~x is the r squared for x vs that y's log.  The R-squared against the original y was better.  Is there a nifty function that calculates that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see from this output, what summary reports as the rsquared is not &quot;correct&quot; while my rsq data accurately shows the best rmse. (See source code below.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;[1] &quot;rmse is 305221.416535481 rsq is 0.749525713697497 summary$r.squared 0.705953722113025&quot;&#xA;[1] &quot;rmse is 304961.752311906 rsq is 0.749815047530537 summary$r.squared 0.706163612870978&quot;&#xA;[1] &quot;rmse is 318083.254498832 rsq is 0.723564406971294 summary$r.squared 0.723564406971402&quot;&#xA;[1] &quot;rmse is 317352.614485029 rsq is 0.724832898371528 summary$r.squared 0.724832898371531&quot;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to learn the best way to do this in R...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&#xA;rmse=function(x,y){&#xA;  return( sqrt(sum((x-y)^2)/length(x)));&#xA;}&#xA;rsq=function(actual,pred){    &#xA;  return( cor(actual,pred,method=c(&quot;pearson&quot;))^2);&#xA;}&#xA;&#xA;&#xA;findBestFormula=function(data,yvar,xvar){&#xA;  #This figures out the best Excel-like trend function based on lowest rmse&#xA;  data$y=data[,yvar];&#xA;  data$x=data[,xvar];  &#xA;  models=list(lm(log(y)~log(x),data),&#xA;              lm(log(y)~x,data),&#xA;             lm(y~log(x),data),&#xA;              lm(y~x,data));&#xA;  preds=list(exp(predict(models[[1]],data)),&#xA;    exp(predict(models[[2]],data)),&#xA;    (predict(models[[3]],data)),&#xA;    (predict(models[[4]],data)));&#xA;  rs=c();&#xA;  for(i in (1:4)){&#xA;    #say(names(models[[i]]))&#xA;    rs[i]=rmse(data$y,preds[[i]]);&#xA;    print(paste(&quot;rmse is&quot;,rs[i],&quot;rsq is&quot;,rsq(data$y,preds[[i]]),&#xA;                &quot;summary$r.squared&quot;,summary(models[[i]])$r.squared));&#xA;  }&#xA;  best=min(rs)&#xA;  #say(best);&#xA;  for(i in (1:4)){&#xA;    if(best==rs[i]){&#xA;      return(list(model=models[[i]],pred=preds[[i]],modelNum=i,rmse=best));  &#xA;    }&#xA;  }&#xA;&#xA;}&#xA;&#xA;&#xA;summary(findBestFormula(weekDayDf,'daily','d')$model)&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any tips on niftier code to write the above?&lt;/p&gt;&#xA;" OwnerUserId="8650" LastActivityDate="2015-04-16T21:11:14.103" Title="R lm(log(y)~x,data) models and predict, need to remember the exp. R2 differences" Tags="|r|forecast|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5544" PostTypeId="1" CreationDate="2015-04-16T23:09:45.397" Score="1" ViewCount="774" Body="&lt;p&gt;I have a question about two different methods from different libraries which seems doing same job. I am trying to make linear regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code which I using statsmodel library with OLS :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = cross_validation.train_test_split(x, y, test_size=0.3, random_state=1)&#xA;&#xA;x_train = sm.add_constant(X_train)&#xA;model = sm.OLS(y_train, x_train)&#xA;results = model.fit()&#xA;&#xA;print &quot;GFT + Wiki / GT  R-squared&quot;, results.rsquared&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This print out &lt;strong&gt;GFT + Wiki / GT  R-squared 0.981434611923&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the second one is scikit learn library Linear model method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = LinearRegression()&#xA;model.fit(X_train, y_train)&#xA;&#xA;predictions = model.predict(X_test)&#xA;&#xA;print 'GFT + Wiki / GT R-squared: %.4f' % model.score(X_test, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This print out &lt;strong&gt;GFT + Wiki / GT R-squared: 0.8543&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my understanding, OLS works with training dataset. So my questions, &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a way that work with test data set with OLS ?&lt;/li&gt;&#xA;&lt;li&gt;Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.&lt;/li&gt;&#xA;&lt;li&gt;What is the difference between OLS and scikit linear regression. Which one we use for calculating the score of the model ? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help.&lt;/p&gt;&#xA;" OwnerUserId="16218" LastActivityDate="2015-06-19T21:44:44.067" Title="Difference between OLS(statsmodel) and Scikit Linear Regression" Tags="|machine-learning|scikit-learn|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5547" PostTypeId="1" CreationDate="2015-04-17T18:56:36.563" Score="1" ViewCount="3016" Body="&lt;p&gt;I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype.&#xA;All standard examples like MNIST numbers classification run well, but problems appear when I try to work with my own images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use 3-channel images, not grayscale.&#xA;And there is the code where I'm trying to get arrays from images:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; img = Image.open(item)&#xA; img = ImageOps.fit(img, (256, 256), Image.ANTIALIAS)&#xA; img = np.asarray(img, dtype = 'float64') / 255.&#xA; img = img.transpose(2,0,1).reshape(3, 256, 256)   &#xA; X.append(img)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the code of NN and its fitting:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X, y = simple_load(&quot;new&quot;)&#xA;&#xA;X = np.array(X)&#xA;y = np.array(y)&#xA;&#xA;&#xA;net1 = NeuralNet(&#xA;    layers=[  # three layers: one hidden layer&#xA;        ('input', layers.InputLayer),&#xA;        ('hidden', layers.DenseLayer),&#xA;        ('output', layers.DenseLayer),&#xA;        ],&#xA;    # layer parameters:&#xA;    input_shape=(None, 65536),  # 96x96 input pixels per batch&#xA;    hidden_num_units=100,  # number of units in hidden layer&#xA;    output_nonlinearity=None,  # output layer uses identity function&#xA;    output_num_units=len(y),  # 30 target values&#xA;&#xA;    # optimization method:&#xA;    update=nesterov_momentum,&#xA;    update_learning_rate=0.01,&#xA;    update_momentum=0.9,&#xA;&#xA;    regression=True,  # flag to indicate we're dealing with regression problem&#xA;&#xA;&#xA;       max_epochs=400,  # we want to train this many epochs&#xA;        verbose=1,&#xA;        )&#xA;&#xA;  net1.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I recieve exceptions like this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;las_mnist.py&quot;, line 39, in &amp;lt;module&amp;gt;&#xA;    net1.fit(X[i], y[i])&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py&quot;, line 266, in fit&#xA;    self.train_loop(X, y)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py&quot;, line 273, in train_loop&#xA;    X, y, self.eval_size)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py&quot;, line 377, in train_test_split&#xA;    kf = KFold(y.shape[0], round(1. / eval_size))&#xA;IndexError: tuple index out of range&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So, in which format do you &quot;feed&quot; your networks with image data?&lt;/strong&gt;&#xA;Thanks for answers or any tips!&lt;/p&gt;&#xA;" OwnerUserId="9167" LastActivityDate="2015-09-30T16:49:14.207" Title="Theano/Lasagne/Nolearn Neural Network Image Input" Tags="|classification|python|neural-network|image-classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5548" PostTypeId="1" CreationDate="2015-04-17T19:03:34.930" Score="3" ViewCount="141" Body="&lt;p&gt;I'm having trouble figuring out a way to analyze some simple data. When graphed, the data I have make a somewhat sinusoidal curve. What I want to do is to find the x-values of the maximum peaks of the sinusoidal curve. I then want to subtract each of these x-values from the last peak found and average these differences to obtain an average distance between peaks. Is there an easy way to do this with Excel, Mathematica, or MatLab (the programs I have available to me?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!!!&lt;/p&gt;&#xA;" OwnerUserId="9168" LastActivityDate="2015-05-20T15:47:13.337" Title="Algorithms to find average distance between data maxima" Tags="|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5549" PostTypeId="1" AcceptedAnswerId="5550" CreationDate="2015-04-17T19:14:49.843" Score="3" ViewCount="1289" Body="&lt;p&gt;I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2016-10-14T16:25:27.033" Title="What are some good sources to learn fraud/anomaly detection in normal/time-series data?" Tags="|data-mining|time-series|anomaly-detection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5550" PostTypeId="2" ParentId="5549" CreationDate="2015-04-18T08:23:52.907" Score="2" Body="&lt;p&gt;Usually this is done as outliers analysis (fraud is an outlier vs normal usage). For this aspect, you can find more info in the &lt;a href=&quot;http://www.sciencedirect.com/science/book/9780123814791&quot; rel=&quot;nofollow&quot;&gt;data mining: concepts and techniques&lt;/a&gt; book, even if general purpose book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am convinced that learning this kind of basis is needed to understanding the domain specific methods.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2015-04-18T08:23:52.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5551" PostTypeId="1" CreationDate="2015-04-18T16:40:30.900" Score="1" ViewCount="49" Body="&lt;p&gt;I have an application that tracks people making mentions of various topics. We've used a Bayes algorithm to do some simple classification (users give a thumbs up/thumbs down) to pick the people that they believe are the best fit for their project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our intention was to use this data to help us sort and order the influencers based on &quot;fit&quot; to the customer's needs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we've got some trained data, and now all we can do is say are they similar to the &quot;thumbs up&quot; group, or the &quot;thumbs down&quot; group.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithm should we have used for this instead? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the ideal is to have a score.. and the biggest, smallest based on the trained data is the one that gets shown first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even better if its in Ruby. &lt;/p&gt;&#xA;" OwnerUserId="9176" LastActivityDate="2015-05-21T19:59:40.563" Title="What algorithmic approach for selecting similar, relevancy based documents" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5552" PostTypeId="2" ParentId="5267" CreationDate="2015-04-18T19:45:50.017" Score="1" Body="&lt;p&gt;This sounds like an interesting project. I recently worked on an almost case study. In order to get only 3 most accurate predictions, I think you may wanna sort those correctly predicted 5 matches by probability of event (win, draw or loose) and then select the first three matches.I hope your models are able to give you probabilities of events. &#xA;I hope this helps :-)&lt;/p&gt;&#xA;" OwnerUserId="9179" LastActivityDate="2015-04-18T19:45:50.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5553" PostTypeId="2" ParentId="5443" CreationDate="2015-04-19T02:46:50.280" Score="1" Body="&lt;p&gt;I teach a Business Analytics course that includes SQL and Excel. I teach in a business school so my students aren't the most technically capable, which is why I didn't use something like R, Pandas, or Weka. That being said, Excel is a powerful enough tool to use for some data analysis. It gets most of this power from its ability to act as a front end to SQL Server Analysis Services (a component in SQL Server for data analysis) using the Data Mining Add-In.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SSAS lets you construct decision trees, perform linear and logistic regressions, and even make bayesian or neural networks. I've found that using Excel as a front-end is a less threatening approach to doing these kinds of analyses since they've all used Excel before. The way to use SSAS without Excel is through a specialized version of Visual Studio and that isn't the most user friendly tool out there. When you combine it with a few other Excel tools like Power Query and Power Pivot, you're able to do some fairly sophisticated analysis of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Full Disclosure, I'm probably not going to use it again when I teach the new version of the course next year (we're splitting it into two courses so one can focus more heavily on data analysis). But that's just because the university was able to get enough licenses for Alteryx which is even easier to use and more powerful but is $4-85k/user/year if you can't get it free somehow. Say what you will about Excel, but it beats that price point.&lt;/p&gt;&#xA;" OwnerUserId="9182" LastActivityDate="2015-04-19T02:46:50.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5557" PostTypeId="1" AcceptedAnswerId="5560" CreationDate="2015-04-20T11:44:39.870" Score="0" ViewCount="38" Body="&lt;p&gt;I am looking towards a solution where classification algorithms produce output with some confidence value. but I am confused whether classification algorithms are able to produce results with percentage of confidence? &#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="4993" LastActivityDate="2015-04-20T16:09:13.293" Title="percentage of confidance on desion trees results" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5558" PostTypeId="1" AcceptedAnswerId="5562" CreationDate="2015-04-20T15:18:38.840" Score="0" ViewCount="141" Body="&lt;p&gt;I'm working on a problem where frequency analysis applies (decomposition of a signal into frequencies, that is), but it's noisy and the samples are unevenly spaced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically: given a list of items purchased at a bar/restaurant, try to estimate the number of guests on the check based on distinct &quot;frequencies&quot; of purchase. The logic is that if there are N guests on a check, then it's reasonable to see N frequencies of drinks being purchased, one person buying every 10 minutes, another every 15, etc. (Plenty of other properties of the check should be included, but here I'm focusing specifically on estimating distinct frequencies).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So more formally: given a noisy, unevenly spaced time series, find the smallest number of frequencies which reproduce the signal while minimizing the error (... for some sensible definition of how to minimize both the error and the number of frequencies simultaneously).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is more a machine learning problem than signal processing. I realize it's also an open question, but can anyone point me in the right direction? Is there a particular method or algorithm that applies here?&lt;/p&gt;&#xA;" OwnerUserId="9210" LastActivityDate="2015-04-20T17:25:20.867" Title="Finding frequencies in a noisy, &quot;uneven&quot; dataset" Tags="|machine-learning|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5559" PostTypeId="2" ParentId="5548" CreationDate="2015-04-20T15:23:01.027" Score="2" Body="&lt;p&gt;The first technique that comes to mind is &lt;a href=&quot;http://en.wikipedia.org/wiki/Spline_interpolation&quot; rel=&quot;nofollow&quot;&gt;cubic spline interpolation&lt;/a&gt;, which is good at approximating sinusoidal functions without suffering from &lt;a href=&quot;http://en.wikipedia.org/wiki/Runge%27s_phenomenon&quot; rel=&quot;nofollow&quot;&gt;Runge's phenomenon&lt;/a&gt;. Here's a visual of cubic splines applied to a similar problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTAmsixio8YnpQTT2zL_i602MkIa72JM3b9E1iw-YlNsrahv9w_&quot; alt=&quot;Example of cubic spline interpolation&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matlab has an implementation documented &lt;a href=&quot;http://www.mathworks.com/help/matlab/ref/spline.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a representation for the underlying function you should be able to find the local maxima easily and take the differences you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2015-04-20T15:23:01.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5560" PostTypeId="2" ParentId="5557" CreationDate="2015-04-20T16:09:13.293" Score="0" Body="&lt;p&gt;Some classification algorithms can indeed return a probability distribution over the considered classes (see Wikipedia on &lt;a href=&quot;http://en.wikipedia.org/wiki/Probabilistic_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;probabilistic classification&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the topic of your question you're asking about Decision Trees. Well, these have their limitations in terms of providing probability estimates (see this &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.324.132&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper on probability estimates from decision trees&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you would like to play around with this, it's very easy to start with scikit-learn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;&#xA;dtc = DecisionTreeClassifier()&#xA;&#xA;# training samples&#xA;X = np.array([[1, 1, 0],&#xA;              [1, 1, 0],&#xA;              [0, 0, 1],&#xA;              [0, 0, 1]])&#xA;&#xA;# target values for training samples&#xA;y = np.array([0, 0, 0, 1])&#xA;&#xA;dtc.fit(X, y)&#xA;&#xA;print 'Class probabilities for training samples:'&#xA;print dtc.predict_proba(X)&#xA;print 'Probabilities for previously unseen samples:'&#xA;for sample in ((1, 0, 1), (0, 0, 1), (1, 1, 1), (0, 0, 0)):&#xA;     print 'Sample {}. Result: {}'.format(sample, dtc.predict_proba(sample))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code returns following results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Class probabilities for training samples:&#xA;[[ 1.   0. ]&#xA; [ 1.   0. ]&#xA; [ 0.5  0.5]&#xA; [ 0.5  0.5]]&#xA;Probabilities for previously unseen samples:&#xA;Sample (1, 0, 1). Result: [[ 0.5  0.5]]&#xA;Sample (0, 0, 1). Result: [[ 0.5  0.5]]&#xA;Sample (1, 1, 1). Result: [[ 0.5  0.5]]&#xA;Sample (0, 0, 0). Result: [[ 1.  0.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this scale the results are easily interpretable:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a sample of features (1, 1, 0) is classified as 0 with 100% probability&lt;/li&gt;&#xA;&lt;li&gt;a sample of features (0, 0, 1) is classified 50/50 as 0 or 1. etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This brings us to an important question of &lt;a href=&quot;https://stats.stackexchange.com/a/34936/41317&quot;&gt;how accurate is your model&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="173" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-04-20T16:09:13.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5561" PostTypeId="1" AcceptedAnswerId="5571" CreationDate="2015-04-20T17:04:42.813" Score="1" ViewCount="1876" Body="&lt;p&gt;I am analyzing a dataset in Python for strictly learning purpose.&#xA;In the code below that I wrote, I am getting some errors which I cannot get rid off. Here is the code first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')&#xA;plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)&#xA;plt.xlabel('Year')&#xA;plt.ylabel('Rating')&#xA;remove_border()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the following errors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. TypeError: 'str' object is not callable&#xA;2. NameError: name 'remove_border' is not defined&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, the label='Decade Average' is not showing up in the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What confuses me most is the fact that in a separate  code snippet for plots (see below), I didn't get the 1st error above, although &lt;code&gt;remove_border&lt;/code&gt; was still a problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.hist(movieDF.rating, bins = 5, color = 'blue', alpha = 0.3)&#xA;plt.xlabel('Rating')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any explanations of all or some of the errors would be greatly appreciated. Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the comments, I am posting the data and the traceback below:&#xA;decade_mean is given below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;year&#xA;1970    8.925000&#xA;1980    8.650000&#xA;1990    8.615789&#xA;2000    8.378947&#xA;2010    8.233333&#xA;Name: rating, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;traceback:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-361-a6efc7e46c45&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')&#xA;      2 plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)&#xA;----&amp;gt; 3 plt.xlabel('Year')&#xA;      4 plt.ylabel('Rating')&#xA;      5 remove_border()&#xA;&#xA;TypeError: 'str' object is not callable&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have solved remove_border problem. It was a stupid mistake I made. But I couldn't figure out the problem with the 'str'.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="3314" LastEditDate="2015-04-21T13:59:30.500" LastActivityDate="2023-01-09T22:04:46.273" Title="Can someone explain the following error in my python code?" Tags="|data-mining|python|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5562" PostTypeId="2" ParentId="5558" CreationDate="2015-04-20T17:25:20.867" Score="0" Body="&lt;p&gt;Just to point you in one possible direction: you could treat this problem as one of probabilistic mixture modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that each person's drink ordering is governed by a probability distribution. That distribution may be characterized by the time since their last drink order.  As time passes, the probability that they will order another drink increases until eventually they order another drink and the time resets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible model for the time between drink orders for a single person is the &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution&quot; rel=&quot;nofollow&quot;&gt;exponential distribution&lt;/a&gt;.  If you consider many people ordering, the resultant times would likely be a mixture of exponentials. The problem then comes down to fitting a mixture of exponentials to the table's drink data.  You would likely have to supply some additional prior data as well to get a meaningful model (otherwise it's difficult to tell the difference between 5 people ordering 2 drinks a piece or 1 person ordering 10 drinks).&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-04-20T17:25:20.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5563" PostTypeId="2" ParentId="5544" CreationDate="2015-04-20T20:26:15.337" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is a lot of information missing here, but it looks like the first example could be finding the correlation between the predicted and actual values of data from your training set and the second example is finding the correlation between the predicted and actual values of your testing set. This would also explain why the correlation in the second result is lower. The performance of a model on novel (testing) data should nearly always be poorer than on the training dataset that was used to fit the model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a way that work with test data set with OLS ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/0.5.0/generated/statsmodels.regression.linear_model.OLS.html&quot; rel=&quot;nofollow&quot;&gt;The statsmodels documentation&lt;/a&gt; describes a &lt;code&gt;predict()&lt;/code&gt; method that can be used to make predictions on inputs from a design matrix.  You should be able to use this method on your test data once you have fit the model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Evaluation on training data can often be used as a quick check for &quot;best case&quot; generalization performance.  However, this is often only used as a diagnostic to ensure your model has fit properly and should not be taken to be a true indicator of generalization.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-04-20T20:26:15.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5564" PostTypeId="2" ParentId="5561" CreationDate="2015-04-21T01:47:50.237" Score="0" Body="&lt;p&gt;Seems to me that &lt;code&gt;remove_border()&lt;/code&gt; is not being recognized as a function. Seeing how you haven't listed a package it belongs to, did you implement this yourself? Python is guessing that &lt;code&gt;remove_border&lt;/code&gt; is a string and you're trying to &quot;call it&quot; - treat it like a function. The 2nd error is saying the same thing, the function &lt;code&gt;remove_border&lt;/code&gt; isn't defined anywhere that Python can find it.&lt;/p&gt;&#xA;" OwnerUserId="403" LastActivityDate="2015-04-21T01:47:50.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5565" PostTypeId="1" CreationDate="2015-04-21T05:38:55.830" Score="3" ViewCount="1910" Body="&lt;p&gt;Is it possible to learn the weights for a logistic regression classifier using EM (Expectation Maximization)algorithm? Is there any instance reference?&lt;/p&gt;&#xA;" OwnerUserId="9218" LastEditorUserId="843" LastEditDate="2015-10-12T13:39:25.247" LastActivityDate="2015-10-12T13:39:25.247" Title="Using EM (Expectation Maximization) algorithm for Training Logistic Regression" Tags="|machine-learning|classification|statistics|logistic-regression|expectation-maximization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5566" PostTypeId="1" CreationDate="2015-04-21T05:55:07.347" Score="3" ViewCount="384" Body="&lt;p&gt;Undergraduate researcher here. I worked at many traditional scientific research labs, ranging from cancer biology, to radiation medicine, and to supercapacitors. I'm thinking of switching to Statistics and Computer Science from Computational Biology to join the exciting field of Data Science. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What ways can I contribute to Humanity without a technical background in basic science? I don't know how I will feel about making better clickbaits. I've thought about continuing in scientific research but I will lack the science background. I've thought about working for the United Nations. What other ways can I contribute? Inspire me!&lt;/p&gt;&#xA;" OwnerUserId="9219" LastEditorUserId="-1" LastEditDate="2015-04-23T05:43:16.857" LastActivityDate="2015-04-23T05:43:16.857" Title="How can I use Data Science to profoundly contribute to Humanity?" Tags="|career|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5567" PostTypeId="1" CreationDate="2015-04-21T07:00:39.540" Score="1" ViewCount="88" Body="&lt;p&gt;I understand how recurrent neural networks work, however I'm trying to build a deep intuitive understanding of their behavior which is difficult for me because they exhibit such complex behaviors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most difficult thing for me to understand is how a Reccurrent Neural Network can exhibit an oscillatory behavior along with the notion of exploding weights. For one, I'm guessing that oscillatory behaviors are only possible for certain activation functions and configurations. Here are my following questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Can sigmoid activation functions exhibit oscillatory behavior? I've convinced myself that they don't since they have a positive range between 0 and 1 which doesn't allow for negative derivatives, but maybe I'm wrong. Is there a formal proof out there for whether it can or cannot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Which activation functions can exhibit oscillatory behaviors and are there proofs out there for them? I believe tanh might have this behavior but I'm not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) What are exploding weight derivatives and how do they occur? There is polar opposite of exploding weight's which seems to be where the weights do not learn and stay stagnate. What causes these issues? I imagine this is dependent upon the activation function as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I began reading this article:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;ved=0CCsQFjAB&amp;amp;url=http%3A%2F%2Fwww.researchgate.net%2Fprofile%2FChristian_OReilly%2Fpublication%2F52004955_Permanent_oscillations_in_a_3-node_recurrent_neural_network_model%2Flinks%2F0c96052464be171d68000000.pdf&amp;amp;ei=X_E1Vf7ULNGsogSo9YHIBQ&amp;amp;usg=AFQjCNE0fex0s7hY2w_upMNkmaxIseUKww&amp;amp;bvm=bv.91386359,d.cGU&quot; rel=&quot;nofollow&quot;&gt;http://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;ved=0CCsQFjAB&amp;amp;url=http%3A%2F%2Fwww.researchgate.net%2Fprofile%2FChristian_OReilly%2Fpublication%2F52004955_Permanent_oscillations_in_a_3-node_recurrent_neural_network_model%2Flinks%2F0c96052464be171d68000000.pdf&amp;amp;ei=X_E1Vf7ULNGsogSo9YHIBQ&amp;amp;usg=AFQjCNE0fex0s7hY2w_upMNkmaxIseUKww&amp;amp;bvm=bv.91386359,d.cGU&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but got confused around page 10 because the author omit's explaining opaque assumptions such as why we are defining the matrix measure as is and so forth(it's a good paper nonetheless). Is there a more transparent paper or lecture out there that can shed light on this subject, or maybe simple explanation for what the author is trying to get at?&lt;/p&gt;&#xA;" OwnerUserId="9220" LastActivityDate="2015-04-21T07:00:39.540" Title="Prove Reccurrent Neural Network can exhibit oscillatory behavior" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5571" PostTypeId="2" ParentId="5561" CreationDate="2015-04-21T09:30:38.253" Score="1" Body="&lt;p&gt;Seems that &lt;code&gt;remove border&lt;/code&gt; is not defined. You have to define the function before used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know where the string error comes, is not clear to me. If you post the full traceback it will be clearer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally your label is not show because you have to call the method &lt;code&gt;plt.legend()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="9223" LastActivityDate="2015-04-21T09:30:38.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5572" PostTypeId="1" AcceptedAnswerId="5573" CreationDate="2015-04-21T09:47:05.917" Score="2" ViewCount="469" Body="&lt;p&gt;I am currently working with the forest cover type prediction from Kaggle, using classification models with scikit-learn. My main purpose is learning about the different models, so I don't pretend to discuss about which one is better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When working with logistic regression, I wonder if I need the 'penalty' parameter (where I can choose L1 or L2 regularization). Based on what I found, these regularization terms are useful to avoid over-fitting, specially when the parameter values are extreme (by extreme I understand the range of some parameter values are very large compared to other parameters, Correct me if I am wrong. In this case, wouldn't it be enough to apply log-scale or normalization to these values?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main questions are: as the number of parameters is large, are there visualization techniques and tools in scikit-learn which can help me to find parameters with extreme values? is there any statistical function/tool which returns how extreme the values of parameters are?&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2015-04-21T12:28:13.413" Title="Finding parameters with extreme values (classification with scikit-learn)" Tags="|classification|logistic-regression|scikit-learn|parameter|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5573" PostTypeId="2" ParentId="5572" CreationDate="2015-04-21T12:28:13.413" Score="1" Body="&lt;p&gt;If by &quot;parameters&quot; you mean features (called &quot;Data Fields&quot; at Kaggle), then, yes, you can log-scale those. To visualize them you can just use histograms.&#xA;To do it for all features in python, for example, you can put your data in pandas DataFrame (let us call it &quot;data&quot;) and then use data.hist()&#xA;This has nothing to do with the regularization in any model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If by &quot;parameters&quot; you mean the &lt;em&gt;coefficients&lt;/em&gt; obtained after fitting the logistic regression, then one uses regularization. This has, however, is not directly related to log-transform. How you list/visualize your coefficients depends on the programming tool you use for logistic regression (or other model)&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-04-21T12:28:13.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5574" PostTypeId="1" CreationDate="2015-04-21T13:18:17.537" Score="1" ViewCount="431" Body="&lt;p&gt;Has there been a successful implementation of Nash-Equilibrium in big data problems like suggesting a best buy in a stock market, in traffic monitoring systems or crowd control systems.&#xA;All the above mentioned scenarios have competitive environments and one needs to get the best possible solution in them, which should suit well for Nash-Equilibrium cases.&lt;/p&gt;&#xA;" OwnerUserId="9227" LastActivityDate="2015-05-08T06:57:00.863" Title="Use of Nash-Equilibrium in big data environments" Tags="|bigdata|social-network-analysis|computer-vision|finance|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5576" PostTypeId="2" ParentId="5551" CreationDate="2015-04-21T19:31:39.673" Score="1" Body="&lt;p&gt;Check out term frequency-inverse document frequency (TF-IDF), if you're parsing large chunks of text for mentioned topics. The metric measures importance of topics bounded by how often they appear across a corpus of documents, which helps to weed out topics that may be very common, but are very common to all discussions, and therefore not very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a simple calculation, but there is a ruby gem that calculates it directly: &lt;code&gt;tf-idf-similarity&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="9234" LastActivityDate="2015-04-21T19:31:39.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5577" PostTypeId="2" ParentId="5566" CreationDate="2015-04-21T20:36:48.103" Score="1" Body="&lt;p&gt;There are almost too many possible uses for data science in the nonprofit sector to count. You might want to look at the question from the other way around--find a cause that interests you and see how data science would apply. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disaster relief efforts need better modeling and forecasting in everything from anticipating possible disasters to optimizing relief efforts. How can we identify the population most at risk from natural or human-created disasters, what are the most effective ways to mitigate the risks, and during a disaster, what are the most effective ways to deliver relief?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In education, what methods work best? In a country or region with low female literacy, what would be the economic impact of increasing female literacy by a certain percentage? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For poverty relief, based on a particular area's demographics and economy, what is the most affordable way to reduce poverty? How could a funding organization recognize sooner whether an ongoing project is likely to succeed or fail?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if you're interested as much or more in the human aspects, you could look at ways for communities or countries to build reliable data banks, getting all of the government, nonprofit, educational, and other organizations to share and use data effectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9236" LastActivityDate="2015-04-21T20:36:48.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5578" PostTypeId="2" ParentId="5566" CreationDate="2015-04-21T22:30:04.510" Score="1" Body="&lt;p&gt;I'll offer a slightly different perspective. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you can help many needy &lt;em&gt;humans&lt;/em&gt; -- e.g. individual refugees, poor people, sick people, war-torn people -- to help &lt;strong&gt;&lt;em&gt;humanity&lt;/em&gt;&lt;/strong&gt; it's necessary to address &lt;strong&gt;&lt;em&gt;root causes&lt;/em&gt;&lt;/strong&gt;: i.e. &lt;em&gt;why&lt;/em&gt; are there so many refugees, persistent poverty, persistent (preventable) illness, and chronic violence. The root causes of so many of these are in the domain of the social sciences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, data science has only contributed to root cause analysis when it operates within a solid research program, integrating with other types of research and findings. (This research may or may not take place in an academic environment, or maybe in a mixture of environments.) It also places premium value on domain knowledge on the part of the data scientists involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A related theme is to empower &quot;change agents&quot; (a.k.a. social entrepreneurs) with data science and related services.  Nearly all of them are under-resourced and often face severe or menacing opposition.  To do this, you will need to make &lt;em&gt;partnerships&lt;/em&gt; with some of them to understand their needs and their world.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-04-21T22:30:04.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5579" PostTypeId="2" ParentId="5566" CreationDate="2015-04-21T22:56:21.440" Score="2" Body="&lt;p&gt;Since I have already answered a similar question on &lt;em&gt;Data Science StackExchange&lt;/em&gt; site, plus some related ones, I will mention all of them here and let you decide, if you find them helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/843/2452&quot;&gt;on data science project ideas, including those with social impact&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/5081/2452&quot;&gt;on working as a data scientist for a non-profit company&lt;/a&gt;;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/5446/2452&quot;&gt;on using data science in political campaigns&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-04-21T22:56:21.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5581" PostTypeId="1" AcceptedAnswerId="5586" CreationDate="2015-04-22T03:44:02.667" Score="1" ViewCount="148" Body="&lt;p&gt;I have scraped the rating that a customer gave in the following categories namely overall rating score, value for money, seat comfort,staff service, catering and entertainment from an airline forum. I would like to know what quality information I can deduce from such ratings about an airline. I have rating of nearly 300 customers who traveled in an airline.  &lt;/p&gt;&#xA;" OwnerUserId="9174" LastEditorUserId="7980" LastEditDate="2015-04-22T12:37:19.053" LastActivityDate="2015-04-22T13:35:06.473" Title="what can be done with the following data?" Tags="|data-mining|" AnswerCount="1" CommentCount="4" ClosedDate="2015-04-25T19:56:18.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="5582" PostTypeId="1" CreationDate="2015-04-22T06:05:29.730" Score="1" ViewCount="782" Body="&lt;p&gt;I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a training data of 1000 items as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Item_name, Attrib_1, Attrib_2, Attrib_3,....Attrib_N, Cost&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my aim is to create a model to predict cost for a new item given the attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So where should I start and what are the different ways to prediction and solve this problem ? Also how to evaluate different methods ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-04-22T16:48:21.760" Title="Basic ML based Prediction model" Tags="|python|regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5585" PostTypeId="1" AcceptedAnswerId="5607" CreationDate="2015-04-22T12:32:10.243" Score="2" ViewCount="1550" Body="&lt;p&gt;First of all, I hope I'm in the right StackExchange here. If not, apologies!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@attribute 'feature#value' numeric&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And per row, the value is either 1 or it is absent (so note it's a &lt;em&gt;sparse&lt;/em&gt; ARFF file). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot. If you need more information, please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-04-25T05:50:05.257" Title="Optimizing Weka for large data sets" Tags="|machine-learning|classification|bigdata|clustering|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5586" PostTypeId="2" ParentId="5581" CreationDate="2015-04-22T13:35:06.473" Score="2" Body="&lt;p&gt;I think you're looking at it the wrong way. First, you need to have a question and then you may find out whether the data could potentially hold the answer to that question. Right now, you're asking us to what questions this data may hold the answer. And that is a very difficult question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to provide you with at least some ideas:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Find out what people deem important (the correlation between overall rating and the other ratings in (linear) regression analyses)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find out the relation between certain characteristics of people (age/gender/nationality, if available) and their overall rating (or whatever rating available)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Basically, the relation between certain variables and their outcomes. But make sure you have a clear question before you start looking for answers.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-04-22T13:35:06.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5588" PostTypeId="2" ParentId="5582" CreationDate="2015-04-22T16:48:21.760" Score="4" Body="&lt;p&gt;The type of ML problem you are trying to solve is a regression problem.  Essentially, given your attributes, predict a continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are multiple ways to solve your problem, but (assuming attributes are numerical) they all boil down to &quot;plot your data points, then draw a line through them&quot;.  The different methods calculate the placement of this predictor based on various different measures of error; the most basic probably being least squares, which aims to minimize the sum of squared errors.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Metacademy is a great website for learning ML related stuff online, because they provide roadmaps that tell you what you need to learn before learning ______, as well as how to learn those prerequisites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.metacademy.org/graphs/concepts/linear_regression&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a link to the metacademy page for linear regression, which is where you're going to want to start.  I'd recommend watching the coursera videos from Andrew Ng.  If you're just getting started with ML, his course is a wonderful introduction.  However, a common complaint for the coursera class is it's too shallow, and is to be considered as an overview of machine learning.  If you consider yourself a little more mathematically savvy, I hear the Stanford course (also taught by him) may be a more in depth resource for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should implement your own basic linear regression at first, just to get a deeper understanding of how it works.  Afterwords, you can just use a python ML library called scikit learn to perform the regression (&lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares&quot; rel=&quot;nofollow&quot;&gt;link here&lt;/a&gt;).  I strongly recommend you take this approach for all ML algorithms; strive to understand at least how the algorithms work at a basic level.  It is far too easy for someone just starting to learn about ML to see all the implemented algorithms in scikit learn go on to use the algorithms as black boxes without understanding what the algorithm is actually doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, in order to evaluate the different methods, you should split your dataset into two different parts; a training set with which to train your models on, then a testing set with which to test how accurate the predictions are.  The idea here being that your data is representative of the &quot;wild&quot; data that you'll see, so by purposely hiding some data, you can effectively simulate how the algorithm will perform in &quot;real world&quot; conditions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, people go with something along the lines of 70% of your data being used to train, and 30% of it being used to test.  Later on, as you get into complicated models with multiple parameters, it will help to split your data into 3 parts: training, used to train your model, validation, used to adjust model parameters, and finally, a test set, used to measure how accurate the model is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="9246" LastActivityDate="2015-04-22T16:48:21.760" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5589" PostTypeId="1" AcceptedAnswerId="5641" CreationDate="2015-04-22T18:00:27.083" Score="18" ViewCount="25456" Body="&lt;p&gt;Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I just want to supply the download URL to S3 and wait for them to download it to their filesystem. It seems like an easy thing to do, but I just can't find the documentation on it.&lt;/p&gt;&#xA;" OwnerUserId="9249" LastEditorUserId="29575" LastEditDate="2018-01-15T15:42:41.653" LastActivityDate="2020-06-01T17:02:28.840" Title="Downloading a large dataset on the web directly into AWS S3" Tags="|dataset|aws|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5590" PostTypeId="1" AcceptedAnswerId="5591" CreationDate="2015-04-22T21:26:38.700" Score="2" ViewCount="5770" Body="&lt;p&gt;I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can do the regularisation job with this formula : (value-min)/(max-min)  where min and max are respectively the minimum and maximum value in the domain of val attribute.&#xA;The question is that one, does it exist other ways ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="97" LastEditDate="2015-04-23T07:44:25.927" LastActivityDate="2015-04-23T07:44:25.927" Title="What is the best way to scale a numerical dataset" Tags="|data-mining|dataset|beginner|feature-scaling|" AnswerCount="1" CommentCount="1" ClosedDate="2015-04-25T19:57:05.790" ContentLicense="CC BY-SA 3.0" />
  <row Id="5591" PostTypeId="2" ParentId="5590" CreationDate="2015-04-23T07:34:34.667" Score="3" Body="&lt;p&gt;There is pretty much mess in terminology in your question :).&#xA;&lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Regularization_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Regularization&lt;/a&gt;&lt;/strong&gt; is used for model selection, it is not about data processing. &lt;a href=&quot;https://stats.stackexchange.com/questions/4961/what-is-regularization-in-plain-english&quot;&gt;Here&lt;/a&gt; it is described in more friendly manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you mean is &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Scaling&lt;/a&gt;&lt;/strong&gt;. It can be done in &lt;strong&gt;several&lt;/strong&gt; ways including &lt;strong&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_scaling#Rescaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rescaling&lt;/a&gt;&lt;/strong&gt;, the method you described. You may also use Standardization (normalization) and Scaling to unit length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These answers may be helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/35591/normalization-vs-scaling&quot;&gt;Normalization vs Scaling&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/10289/whats-the-difference-between-normalization-and-standardization&quot;&gt;Normalization vs Standardization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-04-23T07:42:54.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5592" PostTypeId="2" ParentId="5528" CreationDate="2015-04-23T08:10:27.503" Score="1" Body="&lt;p&gt;Complete-linkage clustering tends to find compact clusters with equal diameters. It is define the inter-group distances as the largest distance of objects between two clusters. Beside, it is may produce clusters that has the same number of objects and the distance between objects is contiguous.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/7vyjS.png&quot; alt=&quot;maximum distance of objects in two clusters&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I can say that your conjecture is correct. I cannot wait for any experts to explain this in detail.&lt;/p&gt;&#xA;" OwnerUserId="9267" LastActivityDate="2015-04-23T08:10:27.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5593" PostTypeId="2" ParentId="5528" CreationDate="2015-04-23T09:12:02.230" Score="1" Body="&lt;p&gt;I may have missed something in the definition in Wikipedia, but wouldn't the set {1,2,4,5,7,8} with $\Delta(x,y) = |x-y|$ be a counterexample? After the 4th step there are two clusters, {{1,2}, {4,5}}, {7,8} and x=5, y=1, z=7 seems to violate your claim.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2015-04-23T09:12:02.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5594" PostTypeId="1" CreationDate="2015-04-23T12:50:52.173" Score="1" ViewCount="239" Body="&lt;p&gt;Using scikit-learn, why would you use bfgs optimization which is non-linear for a linear classifier as logistic regression? I am confused. Does the optimization method finds the optimum of the chosen score function? if so, which one? I can't choose it when defining the estimator. does the linearity or non-linearity of the score function depend on the model (whether it is linear or non-linear)?&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2015-04-23T12:50:52.173" Title="non-linear optimization for a linear classifier? (scikit-learn)" Tags="|machine-learning|logistic-regression|scikit-learn|optimization|kaggle|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5595" PostTypeId="2" ParentId="5589" CreationDate="2015-04-23T13:02:21.580" Score="3" Body="&lt;p&gt;Refer Aws documentation : &lt;a href=&quot;http://aws.amazon.com/code&quot; rel=&quot;nofollow&quot;&gt;http://aws.amazon.com/code&lt;/a&gt; there are libraries available for most of the programing languages.&#xA;So you can create a bucket and configure in your code to fetch data from url and write to this bucket in s3 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for eg in python :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from boto.s3.key import Key&#xA;k = Key(bucket)&#xA;k.key = 'foobar'&#xA;k.set_contents_from_string(url_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ref : &lt;a href=&quot;https://boto.readthedocs.org/en/latest/s3_tut.html&quot; rel=&quot;nofollow&quot;&gt;https://boto.readthedocs.org/en/latest/s3_tut.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-04-23T13:02:21.580" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5596" PostTypeId="2" ParentId="2454" CreationDate="2015-04-23T15:10:58.970" Score="3" Body="&lt;p&gt;The new Amazon Machine Learning Service may work for you. It works directly with Redshift and might be a good way to start. &lt;a href=&quot;http://aws.amazon.com/machine-learning/&quot; rel=&quot;nofollow&quot;&gt;http://aws.amazon.com/machine-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking to process using EMR, then you can use Redshift's UNLOAD command to land data on S3. Spark on EMR can then access it directly without you having to pull it into HDFS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark on EMR: &lt;a href=&quot;https://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923&quot; rel=&quot;nofollow&quot;&gt;https://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9272" LastActivityDate="2015-04-23T15:10:58.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5600" PostTypeId="1" CreationDate="2015-04-23T21:59:23.603" Score="3" ViewCount="961" Body="&lt;p&gt;I used clustering on my dataset. Now when I'm trying to use a LASSO with cv to predict a response, one of the variables it takes into consideration is which cluster a new point is classified into.(I included the cluster variable as a predictor to see if being in a particular group affects the response) &#xA;Since the information on all variables is already captured by the cluster variable,using it again in the Lasso model with some other variables,does it become redundant/biased?&lt;/p&gt;&#xA;" OwnerUserId="9278" LastActivityDate="2016-12-30T20:27:17.357" Title="Using clustering and Lasso with cv" Tags="|machine-learning|clustering|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5601" PostTypeId="1" AcceptedAnswerId="5603" CreationDate="2015-04-24T07:50:31.950" Score="1" ViewCount="1418" Body="&lt;p&gt;I have a text classification problem in which i need to classify an answer to a message as either relevant or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first phase of my calculations, I have already used a SVM to determine if the original message was relevant or not, deciding whether a message contains a hint or question if somebody's twitter account has been hacked.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;example:&#xA;&quot;Hey @foobar, have you been hacked?&quot;   &amp;lt;-- relevant&#xA;&quot;My bank account has just been hacked&quot; &amp;lt;-- not relevant&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, when I want to classify whether the answer is relevant, I would want to have both the original message and the answer as input, right? An answer is relevant in my case if it, in any way, responds to the original message. Is this approach possible using a SVM or any other machine learning tool? I'm using python with the scikit-learn library.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;example:&#xA;&quot;Hey @foobar, have you been hacked?&quot;&#xA;&quot;@barfoo it seems so, thx for suggesting&quot; &amp;lt;-- relevant&#xA;&#xA;&quot;Hey @foobar, have you been hacked?&quot;&#xA;&quot;Lose 20 pounds quickly! http://blabla.com&quot; &amp;lt;-- not relevant&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not very experienced in this field, so any input would be very appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9281" LastEditorUserId="836" LastEditDate="2015-04-25T19:34:52.587" LastActivityDate="2015-04-25T19:34:52.587" Title="How to classify whether text answer is relevant to an initial text question" Tags="|classification|python|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5602" PostTypeId="1" AcceptedAnswerId="5623" CreationDate="2015-04-24T14:50:51.290" Score="2" ViewCount="2203" Body="&lt;p&gt;I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the four categories : Science and technology, Sports, politics, Entertainment. Can anybody please help me find a dataset for this. I've been stuck on this problem for quite some time now. Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9287" LastActivityDate="2020-08-02T17:46:29.627" Title="Training Dataset required for Classifier" Tags="|classification|python|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5603" PostTypeId="2" ParentId="5601" CreationDate="2015-04-24T15:15:06.430" Score="2" Body="&lt;p&gt;Both message and answer are your input, so your feature vector should contain information about both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a simple structure of a possible solution using scikit-learn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.svm import SVC&#xA;from sklearn.feature_extraction import DictVectorizer&#xA;&#xA;dataset = ((&quot;Hey @foobar, have you been hacked?&quot;,&#xA;            &quot;@barfoo it seems so, thx for suggesting&quot;,&#xA;            True), # True for relevant, False for not relevant&#xA;           (&quot;Hey @foobar, have you been hacked?&quot;,&#xA;            &quot;Lose 20 pounds quickly! http://blabla.com&quot;,&#xA;            False))&#xA;&#xA;def extractMessageFeatures(message):&#xA;    # here comes your real feature extraction algorithm&#xA;    return { 'message_predicted_spam': False,&#xA;             'message_contains_valid_username': True }&#xA;&#xA;def extractAnswerFeatures(answer):&#xA;    # here comes your real feature extraction algorithm&#xA;    return { 'answer_predicted_spam': False,&#xA;             'answer_contains_valid_username': True }&#xA;&#xA;def extractFeatures(data):&#xA;    features = []&#xA;    for instance in data:&#xA;        instanceFeatures = extractMessageFeatures(data[0])&#xA;        instanceFeatures.update(extractAnswerFeatures(data[1]))&#xA;        features.append(instanceFeatures)&#xA;    return features&#xA;&#xA;def trainClassifier(data):&#xA;    features = extractFeatures(data)&#xA;    vec = DictVectorizer()&#xA;    featureVector = vec.fit_transform(features)&#xA;&#xA;    print vec.get_feature_names()&#xA;    print featureVector.toarray()&#xA;&#xA;    svc = SVC()&#xA;    svc.fit(featureVector, np.array([i[2] for i in data]))&#xA;    return svc&#xA;&#xA;clf = trainClassifier(dataset)&#xA;&#xA;# now, you can clf.predict(...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, the hardest part is to decide which features to extract from both messages and answers. It's up to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the simplest solutions would be to use n-gram features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other approach would be to use some spam detection to decide whether answer is spam or not and treat this information as a feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use Twitter-specific information (for example, whether users are mentioning each other in their tweets, using the same hashtags etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can combine these features in whatever fashion you like, of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Except of creating feature extraction functionality you need a labeled dataset of messages, answers and relevant/non-relevant labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But once you have both (feature extraction functionality and a proper dataset), you're good to go with a task which clearly matches standard machine learning approaches.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-24T15:15:06.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5605" PostTypeId="2" ParentId="5602" CreationDate="2015-04-24T21:17:21.963" Score="4" Body="&lt;p&gt;You have at least three options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use some of many available datasets (for example: &lt;a href=&quot;http://mlg.ucd.ie/datasets/bbc.html&quot; rel=&quot;nofollow&quot;&gt;BBC documents&lt;/a&gt;); if you need more, simply go to Google Scholar or any similar service and search for &lt;em&gt;classification news politics sports&lt;/em&gt;. In the articles you may find many references to available datasets;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Crawl any news service, and use clustering techniques to group articles into clusters (this often separates the articles along their domain, such as politics, sports etc.), and label the articles by their affiliation to a cluster;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Crawl politics-specific, sports-specific news services and use them as labeled datasets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-24T21:17:21.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5606" PostTypeId="1" AcceptedAnswerId="5616" CreationDate="2015-04-25T01:48:19.270" Score="2" ViewCount="300" Body="&lt;p&gt;I'm currently studying Chapter 7 (&quot;Modeling with Decision Trees&quot;) of the book &quot;Programming Collective intelligence&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find the output of the function &lt;code&gt;mdclassify()&lt;/code&gt; p.157 confusing. The function deals with missing data. The explanation provided is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the basic decision tree, everything has an implied weight of 1,&#xA;  meaning that the observations count fully for the probability that an&#xA;  item fits into a certain category. If you are following multiple&#xA;  branches instead, you can give each branch a weight equal to the&#xA;  fraction of all the other rows that are on that side.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From what I understand, an instance is then split between branches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I simply don't understand how we can obtain:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'None': 0.125, 'Premium': 2.25, 'Basic': 0.125}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as &lt;code&gt;0.125+0.125+2.25&lt;/code&gt; does not sum to 1 nor even an integer. How was the new observation split?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/arthur-e/Programming-Collective-Intelligence/blob/master/chapter7/treepredict.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/arthur-e/Programming-Collective-Intelligence/blob/master/chapter7/treepredict.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the original dataset, I obtain the tree shown here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zT39z.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zT39z.jpg&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please explain me precisely what the numbers precisely mean and how they were exactly obtained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS : The 1st example of the book is wrong as described on their errata page but just explaining the second example (mentioned above) would be nice.&lt;/p&gt;&#xA;" OwnerUserId="9296" LastEditorUserId="29575" LastEditDate="2018-06-26T04:28:19.580" LastActivityDate="2018-06-26T04:28:19.580" Title="Question on decision tree in the book Programming Collective Intelligence" Tags="|python|decision-trees|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="5607" PostTypeId="2" ParentId="5585" CreationDate="2015-04-25T05:50:05.257" Score="2" Body="&lt;p&gt;I would go for dimensionality reduction. You can start with SVD (should be available in Weka). If SVD is too slow / too memory consuming, then there are still some options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;CUR-decomposition: a variant of singular-value decomposition that keeps the matrices of the decomposition sparse if the original matrix is sparse (see: this &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch11.pdf&quot; rel=&quot;nofollow&quot;&gt;chapter&lt;/a&gt; of Mining Massive Datasets book)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Random projections: projecting the data onto a random lower-dimensional subspace (see: the &lt;a href=&quot;http://people.inf.elte.hu/fekete/algoritmusok_msc/dimenzio_csokkentes/randon_projection_kdd.pdf&quot; rel=&quot;nofollow&quot;&gt;Random projection in dimensionality reduction: Applications to image and text data&lt;/a&gt; paper)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Coresets: Given a matrix A, a coreset C is defined as a weighted subset of rows of A such that the sum of squared distances from any given k-dimensional subspace to the rows of A is approximately the same as the sum of squared weighted distances to the rows in C &lt;a href=&quot;http://arxiv.org/pdf/1503.01663v1.pdf&quot; rel=&quot;nofollow&quot;&gt;see the Dimensionality Reduction of Massive Sparse Datasets Using Coresets&lt;/a&gt; paper)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That's the tip of an iceberg. More approaches are there in the wild. The problem is that I doubt any of these solutions come with Weka (please, correct me if I am wrong on this). I would search for a usable Java implementation of any of these algorithms and try to port it to work with Weka's arff files.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-25T05:50:05.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5609" PostTypeId="1" CreationDate="2015-04-25T17:35:30.843" Score="6" ViewCount="349" Body="&lt;p&gt;Assume that &quot;1,2,3&quot; are the ids of users, active means that person visited the stackoverflow in last one month (0=passive, 1=active), and there are positive and negative votes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id  question       votes                 active&#xA; 1     1        -1, +1, -1, -1, -1         0&#xA; 1     2        -1, +1, -1, -1, +1         0&#xA; 2     1        +1, +1, -1, -1             0&#xA; 3     1        +1, +1, +1, -1, +1         1&#xA; 3     2        +1, +1, -1, +1, +1, +1     1&#xA; 3     3        -1, +1                     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to know what makes the users stop using stackoverflow. Think that, I have already calculate the how many times did they get negative votes, total vote, average vote for each question...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder what kind of information could I get from these sequences. I want to find something like this: these users who are passive have two negative votes sequentially. For example, one positive vote after two negative votes in the second question of user 1, doesn't prevent the user churn. User 3 doesn't have any 2 negative votes sequentially in any of his questions. Hence he is still active.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for something like PrefixSpan Algorithm but order is important for me. I mean, I can't write the sequences like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;(-1 +1 -1 -1 -1) (-1 +1 -1 -1 +1 )&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;(-1) (+1) (-1) (-1) (-1) (-1) (+1) (-1) (-1) (+1 )&amp;gt;. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because the first one loses the order, and the second one jumbled the questions together. Is there any algorithm to find these sequences which is common for churners?&lt;/p&gt;&#xA;" OwnerUserId="9323" LastEditorUserId="9323" LastEditDate="2015-04-26T09:35:11.590" LastActivityDate="2015-04-27T20:50:57.117" Title="Algorithm to find common sequence" Tags="|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5610" PostTypeId="1" AcceptedAnswerId="5638" CreationDate="2015-04-25T18:29:33.000" Score="1" ViewCount="209" Body="&lt;p&gt;What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive Bayes? &lt;/p&gt;&#xA;" OwnerUserId="9174" LastActivityDate="2015-04-28T19:34:42.863" Title="What are Hybrid Classifiers used in Sentiment Analysis?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5611" PostTypeId="1" CreationDate="2015-04-25T19:43:00.023" Score="0" ViewCount="394" Body="&lt;p&gt;I'm doing some cluster analysis on the &lt;code&gt;MLTobs&lt;/code&gt; from the &lt;code&gt;LifeTables&lt;/code&gt; package and have come across a tricky problem plotting frequency of the &lt;code&gt;Year&lt;/code&gt; variable in the &lt;code&gt;mlt.mx.info&lt;/code&gt; dataframe. &lt;code&gt;Year&lt;/code&gt; contains the period that the life table was measured over, in intervals. Here's a table of the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    1751-1754 1755-1759 1760-1764 1765-1769 1770-1774 1775-1779 1780-1784 1785-1789 1790-1794 &#xA;        1         1         1         1         1         1         1         1         1 &#xA;1795-1799 1800-1804 1805-1809 1810-1814 1815-1819 1816-1819 1820-1824 1825-1829 1830-1834 &#xA;        1         1         1         1         1         2         3         3         3 &#xA;1835-1839 1838-1839 1840-1844 1841-1844 1845-1849 1846-1849 1850-1854 1855-1859 1860-1864 &#xA;        4         1         5         3         8         1        10        11        11 &#xA;1865-1869 1870-1874 1872-1874 1875-1879 1876-1879 1878-1879 1880-1884 1885-1889 1890-1894 &#xA;       11        11         1        12         2         1        15        15        15 &#xA;1895-1899 1900-1904 1905-1909 1908-1909 1910-1914 1915-1919 1920-1924 1921-1924 1922-1924 &#xA;       15        15        15         1        16        16        16         2         1 &#xA;1925-1929 1930-1934 1933-1934 1935-1939 1937-1939 1940-1944 1945-1949 1947-1949 1948-1949 &#xA;       19        19         1        20         1        22        22         3         1 &#xA;1950-1954 1955-1959 1956-1959 1958-1959 1960-1964 1965-1969 1970-1974 1975-1979 1980-1984 &#xA;       30        30         2         1        40        40        41        41        41 &#xA;1983-1984 1985-1989 1990-1994 1991-1994 1992-1994 1995-1999 2000-2003 2000-2004 2005-2006 &#xA;        1        42        42         1         1        44         3        41        22 &#xA;2005-2007 &#xA;       14 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, some of the intervals sit within other intervals. Thankfully none of them overlap. I want to simplify the intervals so intervals such as &lt;code&gt;1992-1994&lt;/code&gt; and &lt;code&gt;1991-1994&lt;/code&gt; all go into &lt;code&gt;1990-1994&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An idea might be to get the modulo of each interval and sort them into their new intervals that way but I'm unsure how to do this with the interval data type. If anyone has any ideas I'd really appreciate the help. Ultimately I want to create a histogram or barplot to illustrate the nicely.&lt;/p&gt;&#xA;" OwnerUserId="9070" LastActivityDate="2015-04-27T07:24:13.680" Title="Tidying Time Intervals for Plotting a Histogram in R" Tags="|data-mining|r|clustering|data-cleaning|visualization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5612" PostTypeId="2" ParentId="5602" CreationDate="2015-04-25T19:48:05.637" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow&quot;&gt;UCI Machine Learning repo&lt;/a&gt; is my go to place to find data sets to work on across a really broad range of topics. The data sets are already cleaned and tagged with the task they were originally purposed for. If you google the data set you'll often find papers citing them or forums with questions that you may have yourself. There's a number of text based classification tasked data sets in there.&lt;/p&gt;&#xA;" OwnerUserId="9070" LastEditorUserId="9070" LastEditDate="2015-04-25T20:15:36.903" LastActivityDate="2015-04-25T20:15:36.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5614" PostTypeId="2" ParentId="5611" CreationDate="2015-04-25T23:46:19.727" Score="1" Body="&lt;p&gt;This was a x-post. There is a great solution over &lt;a href=&quot;https://stackoverflow.com/questions/29869565/tidying-time-intervals-for-plotting-histogram-in-r/29870642#29870642&quot;&gt;here&lt;/a&gt;, where I originally asked the question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I understand your problem, you'll want something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bottom &amp;lt;- seq(1750, 2010, 5)&#xA;library(dplyr)&#xA;new_df &amp;lt;- mlt.mx.info %&amp;gt;%&#xA;  arrange(Year) %&amp;gt;%&#xA;  mutate(year2 = as.numeric(substr(Year, 6, 9))) %&amp;gt;%&#xA;  mutate(new_year = paste0(bottom[findInterval(year2, bottom)], &quot;-&quot;,(bottom[findInterval(year2, bottom) + 1] - 1)))&#xA;View(new_df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;So what this does, it creates bins, and outputs a new column&#xA;  (new_year) that is the bottom of the bin. So everything from 1750-1754&#xA;  will correspond to a new value of 1750-1754 (in string form; the&#xA;  original is an integer type, not sure how to fix that). Does this do&#xA;  what you want? Double check the results, but it looks right to me.&#xA;  @goodtimeslim&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9070" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-04-25T23:46:19.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5615" PostTypeId="1" AcceptedAnswerId="5637" CreationDate="2015-04-26T01:38:52.180" Score="3" ViewCount="4809" Body="&lt;p&gt;For word2vec with negative sampling, the cost function for a single word is the following according to &lt;a href=&quot;http://www-personal.umich.edu/%7Eronxin/pdf/w2vexp.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt;:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}.u_{w_{I}}))&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$v_{w_{O}}^{'}$&lt;/span&gt; = hidden-&amp;gt;output word vector of the output word&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$u_{w_{I}}$&lt;/span&gt; = input-&amp;gt;hidden word vector of the output word&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$v_{w_{k}}^{'}$&lt;/span&gt; = hidden-&amp;gt;output word vector of the negative sampled word&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; is the sigmoid function&lt;/p&gt;&#xA;&lt;p&gt;And taking the derivative with respect to &lt;span class=&quot;math-container&quot;&gt;$v_{w_{O}}^{'}.u_{w_{j}}$&lt;/span&gt; is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$&#xA;\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * (\sigma(v_{w_{j}}^{'}.u_{w_{I}}) - 1)&#xA;$&lt;/span&gt; &lt;span class=&quot;math-container&quot;&gt;$ \text{if } w_j = w_O $&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$&#xA;\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * \sigma(-v_{w_{j}}^{'}.u_{w_{I}})&#xA;$&lt;/span&gt; &lt;span class=&quot;math-container&quot;&gt;$ \text{if } w_j = w_k \text{ for } k = 1\ldots K$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Then we can use chain rule to get&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$ \frac{\partial E}{\partial v_{w_{j}}^{'}} = \frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} * \frac{\partial v_{w_{j}}^{'}.u_{w_{I}}}{\partial v_{w_{j}}^{'}} $&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Is my reasoning and derivative correct? I am still new to ML so any help would be great!&lt;/p&gt;&#xA;" OwnerUserId="6523" LastEditorUserId="-1" LastEditDate="2023-03-15T16:21:25.067" LastActivityDate="2023-03-15T16:21:25.067" Title="Gradient Descent Step for word2vec negative sampling" Tags="|gradient-descent|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5616" PostTypeId="2" ParentId="5606" CreationDate="2015-04-26T06:04:19.777" Score="1" Body="&lt;p&gt;There are four features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;referer,&lt;/li&gt;&#xA;&lt;li&gt;location,&lt;/li&gt;&#xA;&lt;li&gt;FAQ,&lt;/li&gt;&#xA;&lt;li&gt;pages.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In your case, you're trying to classify an instance where &lt;code&gt;FAQ&lt;/code&gt; and &lt;code&gt;pages&lt;/code&gt; are unknown: &lt;code&gt;mdclassify(['google','France',None,None], tree)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the first known attribute is &lt;code&gt;google&lt;/code&gt;, in your decision tree you're only interested in the edge that comes out of &lt;code&gt;google&lt;/code&gt; node on the right-hand side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are five instances: three labeled &lt;code&gt;Premium&lt;/code&gt;, one labeled &lt;code&gt;Basic&lt;/code&gt; and one labeled &lt;code&gt;None&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instances with labels &lt;code&gt;Basic&lt;/code&gt; and &lt;code&gt;None&lt;/code&gt; split on the &lt;code&gt;FAQ&lt;/code&gt; attribute. There's two of them, so the weight for both of them is &lt;code&gt;0.5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we split on the &lt;code&gt;pages&lt;/code&gt; attribute. There are 3 instances with &lt;code&gt;pages&lt;/code&gt; value larger than 20, and two with &lt;code&gt;pages&lt;/code&gt; value no larger than 20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the trick: we already know that the weights for two of these were altered from &lt;code&gt;1&lt;/code&gt; to &lt;code&gt;0.5&lt;/code&gt; each. So, now we have three instances weighted &lt;code&gt;1&lt;/code&gt; each, and 2 instances weighted &lt;code&gt;0.5&lt;/code&gt; each. So the total value is &lt;code&gt;4&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we can count the weights for &lt;code&gt;pages&lt;/code&gt; attribute:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;pages_larger_than_20 = 3/4&lt;/li&gt;&#xA;&lt;li&gt;pages_not_larger_than_20 = 1/4 # the 1 is: 0.5 + 0.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All weights are ascribed. Now we can multiply the weights by the &quot;frequencies&quot; of instances (remembering that for &lt;code&gt;Basic&lt;/code&gt; and &lt;code&gt;None&lt;/code&gt; the &quot;frequency&quot; is now &lt;code&gt;0.5&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Premium&lt;/code&gt;: &lt;code&gt;3 * 3/4 = 2.25&lt;/code&gt; # because there are three &lt;code&gt;Premium&lt;/code&gt; instances, each weighting &lt;code&gt;0.75&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Basic&lt;/code&gt;: &lt;code&gt;0.5 * 1/4 = 0.125&lt;/code&gt; # because &lt;code&gt;Basic&lt;/code&gt; is now &lt;code&gt;0.5&lt;/code&gt;, and the split on&#xA;&lt;code&gt;pages_not_larger_than_20&lt;/code&gt; is &lt;code&gt;1/4&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;None&lt;/code&gt;: &lt;code&gt;0.5 * 1/4 = 0.125&lt;/code&gt; # analogously&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That's at least where the numbers come from. I share your doubts about the maximum value of this metric, and whether it should sum to 1, but now that you know where these numbers come from you can think how to normalize them.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-26T06:04:19.777" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5618" PostTypeId="1" CreationDate="2015-04-26T11:09:49.133" Score="2" ViewCount="1262" Body="&lt;p&gt;I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient is available to measure the strength of these two variables?&lt;/p&gt;&#xA;" OwnerUserId="9312" LastEditorDisplayName="user9380" LastEditDate="2015-05-02T13:40:34.810" LastActivityDate="2015-05-02T13:40:34.810" Title="Correlation between time to event data and continuous data" Tags="|statistics|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5620" PostTypeId="2" ParentId="5611" CreationDate="2015-04-27T07:24:13.680" Score="-2" Body="&lt;p&gt;Use library(reshape) for most of the pivot &amp;amp; filter operations. Should resolve your problem.&lt;/p&gt;&#xA;" OwnerUserId="9324" LastActivityDate="2015-04-27T07:24:13.680" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5621" PostTypeId="2" ParentId="5618" CreationDate="2015-04-27T07:26:25.893" Score="1" Body="&lt;p&gt;Use Event as Row Label, Time as Column label and Activity as Value in a Pivot.&#xA;If you're doing this in R, use library(reshape). Using melt and cast functions, you can get a solution to your problem i guess. Let me know if you need any further clarifications.&lt;/p&gt;&#xA;" OwnerUserId="9324" LastActivityDate="2015-04-27T07:26:25.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5622" PostTypeId="2" ParentId="5485" CreationDate="2015-04-27T07:29:06.110" Score="1" Body="&lt;p&gt;Use sigmoidal functions to get the best correlational value. Use Octave/MATLAB for processing your matrix using the function.&lt;/p&gt;&#xA;" OwnerUserId="9324" LastActivityDate="2015-04-27T07:29:06.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5623" PostTypeId="2" ParentId="5602" CreationDate="2015-04-27T07:33:24.683" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.kdnuggets.com/datasets/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; should get you maximum datasets for your classification exercise.&lt;/p&gt;&#xA;" OwnerUserId="9324" LastEditorUserId="98307" LastEditDate="2020-08-02T17:46:29.627" LastActivityDate="2020-08-02T17:46:29.627" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5624" PostTypeId="1" CreationDate="2015-04-27T09:09:55.170" Score="1" ViewCount="63" Body="&lt;p&gt;Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naive bayes provides and random class as a result here. Which classifier deals best in such scenario?&lt;/p&gt;&#xA;" OwnerUserId="8840" LastActivityDate="2015-04-27T13:42:48.057" Title="Which classifier is efficient in dealing with test query which belongs to no trained class?" Tags="|machine-learning|classification|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5626" PostTypeId="2" ParentId="2501" CreationDate="2015-04-27T11:58:50.443" Score="1" Body="&lt;p&gt;Another good normalization is &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow&quot;&gt;zScore&lt;/a&gt; normalization. Was already implemented in python in the module &lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mstats.zscore.html&quot; rel=&quot;nofollow&quot;&gt;scipy.stats&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3281" LastActivityDate="2015-04-27T11:58:50.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5627" PostTypeId="2" ParentId="5624" CreationDate="2015-04-27T13:42:48.057" Score="2" Body="&lt;p&gt;You have trained a model to recognize or discriminate between several particular classes. So when having a new test sample (that according to your knowledge) does not belong to any of this class, the model will fit it to the most similar class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is, off course, a very general way of putting it. The behavior and things to consider wether you are using a probabilistic graphical model (like Naive Bayes), linear or non linear classifiers, etc, will vary. However, the principle is the same: the model has learnt the relation among the features of the training data you have used to match particular classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, if you are using a probabilistic approach you can  use the probability of belonging to a class and define a threshold. So if a probability higher than certain value (let's say 65%) is not given, then your confidence on the result will be low, and you might say &quot;I cannot say wether it belongs to any of this 5 classes with certainty&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other non probabilistic methods have interesting approaches for scoring a new example in terms of probabilities. Check this link for SVM (&lt;a href=&quot;https://stats.stackexchange.com/questions/55072/svm-confidence-according-to-distance-from-hyperline&quot;&gt;https://stats.stackexchange.com/questions/55072/svm-confidence-according-to-distance-from-hyperline&lt;/a&gt;), which is actually the approach used in Python scikit-learn: &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use Random Forests you could use the voting of each tree to define a rate of confidence in a similar way (&lt;a href=&quot;https://stats.stackexchange.com/questions/94845/how-to-estimate-confidence-level-for-svm-or-random-forest&quot;&gt;https://stats.stackexchange.com/questions/94845/how-to-estimate-confidence-level-for-svm-or-random-forest&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar thing can be done with K-Nearest-Neighbors by analyzing the distance of the K or J (where J &amp;lt; K) nearest neighbors and just trust the result if they are close to your test sample in some ratio (very generally said, this can be done in many ways).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, methods based on deep learning are trying to learn features in an unsupervised way, so your problem could solved in a more interesting way. However, it would not be strict classification anymore, nor you would have the amount of data and servers required to even try it (but just saying :) )&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-04-27T13:42:48.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5628" PostTypeId="2" ParentId="5609" CreationDate="2015-04-27T17:46:32.680" Score="3" Body="&lt;p&gt;Not a typical problem formulation indeed. I suggest two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use a simple bayesian model where the states or nodes are each of your features. Connect the nodes directly when they haven a direct time dependency (for example vote1 -&gt; vote2 -&gt; vote3, but not vote3 -&gt; vote1) and calculate the probability of a final (output) node to be something (for example user type 1, user type 2, etc). You could also use a hidden markov model that naturally models transitions between states. In your case it would output just 1 state (the prediction). I am not very fan of this model for your problem, but it could be nice to try. You will need a lot of data though.&lt;/li&gt;&#xA;&lt;li&gt;You could use Fuzzy Inductive Reasoning (FIR). It is definitely not a simple algorithm, but works quite well for classification having time series or time-related features. In FIR you want to find an optimal mask of a given depth. &#xA;If you have, say, 5 features per sample, and for instance you want to know which features to select you will learn an optimal mask for your data that address this (and creates fuzzy rules, etc...). Also if you want to relate two or more examples you will define a depth higher than 1 for the mask. This will allow you to find a mask that for instance uses features 1,2 and 4 of sample t, features 2,3 ,4 and 5 of sample t+1 and features 1 and 4 of sample t+2. Moving t from 1 to N number of samples. So, interesting combinations of features from samples that have a time-relation (or some sort of sequence) are created.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2015-04-27T17:46:32.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5629" PostTypeId="1" AcceptedAnswerId="5645" CreationDate="2015-04-27T18:29:45.513" Score="1" ViewCount="785" Body="&lt;p&gt;I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to my mounted EBS volume and why do I not see the contents of my EBS volume in the r-studio files area (bottom right side? ) . Also, I had a file in an s3 bucket. I passed this command to bring it from s3 to my ebs volume: &lt;code&gt;s3cmd get s3://data-analysis/input-data/filename.csv&lt;/code&gt; . I assume this command downloads the file from s3 into the ebs volume. But I can't find it in RStudio Server! I have scoured the internet looking for help on this but not able to solve my problem.&lt;/p&gt;&#xA;" OwnerUserId="8553" LastActivityDate="2015-04-29T17:05:22.277" Title="How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance" Tags="|aws|rstudio|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5630" PostTypeId="2" ParentId="5609" CreationDate="2015-04-27T20:50:57.117" Score="3" Body="&lt;p&gt;One thing you could do is use an associations rule approach, in the sense of finding the most frequent patterns associated with being passive, or &lt;em&gt;over-associated&lt;/em&gt; with being passive:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;set a frequency threshold&lt;/li&gt;&#xA;&lt;li&gt;figure out what kind of patterns you're looking for, for example frequent sequences. It's good if you can partially order them, similarly to set inclusion in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;a priori algorithm&lt;/a&gt;. For example, look for sequences, and order them by &quot;is subsequence of&quot;.&lt;/li&gt;&#xA;&lt;li&gt;Starting with a reasonably small list of minimal patterns that you're looking for, go through your data counting the occurrences of these patterns.  &lt;/li&gt;&#xA;&lt;li&gt;Keep the most frequent patterns (above your threshold), this is your step-1 set of frequent patterns.&lt;/li&gt;&#xA;&lt;li&gt;Generate candidates for step-&lt;em&gt;i+1&lt;/em&gt; patterns by adding something to your step-i patterns.&lt;/li&gt;&#xA;&lt;li&gt;Count how many times the candidates occur. Go to step 4 until you have no more frequent patterns for step &lt;em&gt;i&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;li&gt;You then have a bunch of rule candidates, and you want to use association rules metrics to find the patterns that best predict being passive. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9114" LastActivityDate="2015-04-27T20:50:57.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5631" PostTypeId="2" ParentId="5142" CreationDate="2015-04-27T21:46:21.373" Score="7" Body="&lt;p&gt;Spark is intended to be pointed at large distributed data sets, so as you suggest, the most typical use cases will involve connecting to some sort of Cloud system like AWS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, if the data set you aim to analyze can fit on your local system, you'll usually find that you can analyze it just as simply using pure python.  If you're trying to leverage a series of local VMs, you're going to run out of memory pretty quickly and jobs will either fail or grind to a halt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, a local instance of spark is very useful for the purpose of development. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way that I've found that works is if I have a directory in HDFS with many files, I'll pull over a single file, develop locally, then port my spark script to my cloud system for execution.  If you're using AWS, this is really helpful for avoiding big fees while you're developing.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="5177" LastEditDate="2015-10-23T07:42:20.233" LastActivityDate="2015-10-23T07:42:20.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5632" PostTypeId="1" AcceptedAnswerId="5634" CreationDate="2015-04-28T06:48:25.867" Score="3" ViewCount="73" Body="&lt;p&gt;It is said that before an earthquake happens, a viewer experiences disturbances in DTH TV transmission in the form of distorted images on the screen which automatically correct after a few seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to identify patterns of such disturbances by continuously monitoring TV images so that earthquakes can potentially be predicted at least few minutes in advance and many lives could be saved?&lt;/p&gt;&#xA;" OwnerUserId="9342" LastActivityDate="2015-04-28T08:44:09.613" Title="Predicting earthquakes using disturbances in DTH TV transmission" Tags="|predictive-modeling|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5634" PostTypeId="2" ParentId="5632" CreationDate="2015-04-28T08:44:09.613" Score="4" Body="&lt;p&gt;I think that if there are disturbances caused by something preceding earthquake, they might be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that you need to find which cause produces that effect and measure it. For example you can team up with geologists, geophysicists or someone similar and try to build as complete hypothesis as possible and then design experiment to gather data. If your experiment would bring some non noise data then you can start to thinking about machine learning algorithms which will work in this situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you might also try to do it on &quot;brute force&quot; way by actually recording DTH TV images in seismically unstable regions and try to correlate those videos with seismic data. Then after some manual examination and categorization you could define (or not if this cause and effect hypothesis is wrong) some possible glitches that are observed and try to develop software that tries to detect them (&lt;a href=&quot;http://opencv.org/&quot; rel=&quot;nofollow&quot;&gt;OpenCV&lt;/a&gt; might be useful for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO in either way, some domain knowledge related to earthquakes would be more useful on beginning stage, and experience related to machine learning would be more useful on latest stages of such ambitious project.&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2015-04-28T08:44:09.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5635" PostTypeId="1" CreationDate="2015-04-28T12:58:33.920" Score="1" ViewCount="31" Body="&lt;p&gt;I have one small list of entities, such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Russia&#xA;Vladimir&#xA;Moscow&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I have a massive database of JSON indices. For each entry there are multiple alpha-numeric identifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for instance, for &lt;code&gt;Russia&lt;/code&gt; there might only be one. But for &lt;code&gt;Vladimir&lt;/code&gt; maybe there will be 100. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They're stored in JSON but I read them into my java program like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        // GET JSON DATA&#xA;        File f = new File(&quot;/home/matthias/Workbench/SUTD/nytimes_corpus/wdtk-parent/wdtk-examples/JSON_Output/user666.json&quot;);&#xA;        String jsonTxt = null;&#xA;&#xA;        if (f.exists())&#xA;        {&#xA;            InputStream is = new FileInputStream(&quot;/home/matthias/Workbench/SUTD/nytimes_corpus/wdtk-parent/wdtk-examples/JSON_Output/user666.json&quot;);&#xA;            jsonTxt = IOUtils.toString(is);&#xA;        }&#xA;        //reformat&#xA;        jsonTxt = ( jsonTxt.substring(1, jsonTxt.length()-1) ).replace(&quot;\\&quot;,&quot;&quot;);&#xA;&#xA;        Gson json = new Gson();&#xA;        Map&amp;lt;String, HashSet&amp;lt;String&amp;gt;&amp;gt; mast_Q_storage_map = new HashMap&amp;lt;String, HashSet&amp;lt;String&amp;gt;&amp;gt;();&#xA;        mast_Q_storage_map = (Map&amp;lt;String, HashSet&amp;lt;String&amp;gt;&amp;gt;) json.fromJson(jsonTxt, mast_Q_storage_map.getClass());&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to get all of the values associated with the entities from the small list. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I need to search the big list and retrieve their values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I want to try to determine if there is a relationship between any of the entities in the sentence, as in I want to check if &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Russia   X Vladimir&#xA;Russia   X Moscow&#xA;Moscow   X Vladimir&#xA;Moscow   X Russia&#xA;Vladimir X Russia&#xA;Vladimir X Moscow&lt;/p&gt;&#xA;&#xA;&lt;p&gt;will result in a relationship, I've been trying to do it like this, but I'm running into big problems:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        // Read in all the sentences, that are in files, in this folder&#xA;        final File folder = new File(&quot;/home/matthias/Workbench/SUTD/nytimes_corpus/wdtk-parent/wdtk-examples/JSON_Output/&quot;);&#xA;&#xA;        for (final File fileEntry : folder.listFiles()) &#xA;        {&#xA;            BufferedReader br = new BufferedReader(new FileReader(fileEntry));&#xA;            try &#xA;            {&#xA;                //Store the filename&#xA;                //System.out.println(fileEntry.getName());&#xA;                StringBuilder sb = new StringBuilder();&#xA;                String line = br.readLine();&#xA;&#xA;                while (line != null) &#xA;                {   &#xA;                    sb.append(line);&#xA;                    sb.append(System.lineSeparator());&#xA;                    line = br.readLine();&#xA;                }&#xA;                String everything = sb.toString();&#xA;                //System.out.println(everything);&#xA;&#xA;                Document doc = Jsoup.parse(everything);&#xA;&#xA;                Elements contents = doc.getElementsByTag(&quot;sentence&quot;);&#xA;                for (Element content : contents) &#xA;                {&#xA;                    //store the sentence number&#xA;                    String number = content.select(&quot;sentence&quot;).text();&#xA;                    number = number.substring(0, number.indexOf(&quot; &quot;)); &#xA;                    System.out.println(number);&#xA;&#xA;                    //get all the entities in this sentence&#xA;                    Elements pers = content.select(&quot;PERSON&quot;);&#xA;                    Elements locs = content.select(&quot;LOCATION&quot;);&#xA;                    Elements orgs = content.select(&quot;ORGANIZATION&quot;);&#xA;&#xA;                    //collect all the elements to a list, all the elements of one sentence&#xA;                    List&amp;lt;String&amp;gt; list = new ArrayList&amp;lt;String&amp;gt;();&#xA;&#xA;                    for (Element per : pers) &#xA;                    {&#xA;                        list.add(per.text().trim());&#xA;                    }&#xA;                    for (Element loc : locs) &#xA;                    {&#xA;                        list.add(loc.text().trim());&#xA;                    }&#xA;                    for (Element org : orgs) &#xA;                    {&#xA;                        list.add(org.text().trim());&#xA;                    }&#xA;&#xA;&#xA;                    System.out.println();&#xA;                    System.out.println();&#xA;                    System.out.println();&#xA;                    System.out.println(&quot;This is list of sentence elements:&quot;);&#xA;                    for (String s : list)&#xA;                        System.out.println(s);&#xA;                    System.out.println();&#xA;                    System.out.println();&#xA;                    System.out.println();&#xA;&#xA;&#xA;                    List&amp;lt;String&amp;gt; Q_value_list = new ArrayList&amp;lt;String&amp;gt;();&#xA;&#xA;                    // for the list of Q values to keys&#xA;                    for (Entry&amp;lt;String, HashSet&amp;lt;String&amp;gt;&amp;gt; e : mast_Q_storage_map.entrySet()) &#xA;                    {&#xA;                        for (String s : list)&#xA;                        {&#xA;                            if (e.getKey().contains(s)) &#xA;                            {&#xA;                                //List&amp;lt;String&amp;gt; Q_value_list = new ArrayList&amp;lt;String&amp;gt;(e.getValue());&#xA;&#xA;                                System.out.println(e.getKey() + &quot; :: &quot; + Q_value_list.toString());&#xA;                            }&#xA;                        }&#xA;                    }&#xA;&#xA;&#xA;&#xA;                                //czeher&#xA;                                for (String home:Q_value_list) &#xA;                                {&#xA;                                  for (String away:Q_value_list) &#xA;                                  {&#xA;&#xA;&#xA;                                    String URL_czech = &quot;http://milenio.dcc.uchile.cl/sparql?default-graph-uri=&amp;amp;query=PREFIX+%3A+%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3E%0D%0ASELECT+*+WHERE+%7B%0D%0A+++%3A&quot; &#xA;                                                       + home + &quot;+%3FsimpleProperty+%3A&quot; &#xA;                                                       + away + &quot;%0D%0A%7D%0D%0A&amp;amp;format=text%2Fhtml&amp;amp;timeout=0&amp;amp;debug=on&quot;;&#xA;&#xA;&#xA;                                    URL wikidata_page = new URL(URL_czech);&#xA;                                    HttpURLConnection wiki_connection = (HttpURLConnection)wikidata_page.openConnection();&#xA;                                    InputStream wikiInputStream = null;&#xA;&#xA;&#xA;                                        try &#xA;                                        {&#xA;                                            // try to connect and use the input stream&#xA;                                            wiki_connection.connect();&#xA;                                            wikiInputStream = wiki_connection.getInputStream();&#xA;                                        } &#xA;                                        catch(IOException error) &#xA;                                        {&#xA;                                            // failed, try using the error stream&#xA;                                            wikiInputStream = wiki_connection.getErrorStream();&#xA;                                        }&#xA;                                    // parse the input stream using Jsoup&#xA;                                    Document docx = Jsoup.parse(wikiInputStream, null, wikidata_page.getProtocol()+&quot;://&quot;+wikidata_page.getHost()+&quot;/&quot;);&#xA;&#xA;&#xA;&#xA;                                    Elements link_text = docx.select(&quot;table.sparql &amp;gt; tbody &amp;gt; tr:nth-child(2) &amp;gt; td &amp;gt; a&quot;);&#xA;                                    //link_text.text();&#xA;                                    for (Element l : link_text) &#xA;                                    {&#xA;                                        String output = l.text();&#xA;                                        System.out.println( output );&#xA;                                    }&#xA;&#xA;&#xA;                                  }&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;                    }&#xA;&#xA;                }&#xA;            }&#xA;            finally &#xA;            {&#xA;                br.close();&#xA;            }&#xA;&#xA;        }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8285" LastActivityDate="2015-04-28T12:58:33.920" Title="large database operation. check for relatedness between entities" Tags="|search|parsing|java|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5636" PostTypeId="2" ParentId="5408" CreationDate="2015-04-28T15:29:24.093" Score="4" Body="&lt;p&gt;Cluster analysis is not &lt;em&gt;supposed&lt;/em&gt; to produce paritions of equal size. It is meant to discover structure in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the majority of objects is highly similar, then this majority is supposed to be in the majority cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider all your data is identical. Any clustering algorithm producing more than one cluster has failed, in my opinion...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you may be using the wrong class of algorithms for your problem.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-04-28T15:29:24.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5637" PostTypeId="2" ParentId="5615" CreationDate="2015-04-28T19:06:07.887" Score="0" Body="&lt;p&gt;Looks good to me. This derivative is also presented in the paper (equations 56-58).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper you're linking to is the most advanced attempt - at least to best of my knowledge - to explain how word2vec works, but there is also a lot of other resources on the topic (just search for word2vec on &lt;a href=&quot;http://arxiv.org/find/all/1/all:+word2vec/0/1/0/all/0/1&quot; rel=&quot;nofollow&quot;&gt;arxiv.org&lt;/a&gt;). If you're interested in word2vec, you may find GloVe interesting too (see: &lt;a href=&quot;http://arxiv.org/pdf/1411.5595v2.pdf&quot; rel=&quot;nofollow&quot;&gt;Linking GloVe with word2vec&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-28T19:06:07.887" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5638" PostTypeId="2" ParentId="5610" CreationDate="2015-04-28T19:34:42.863" Score="1" Body="&lt;p&gt;In sentiment analysis you may want to combine a number of classifiers. Let's say: a separate classifier for emoticons, another one for emotionally loaded terms, another one for some special linguistic patterns and - let's say - yet another one to detect and filter out spam messages. It's all up to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use either SVM, Naive Bayes, or anything else that best suits your problem. You may use majority voting, weights (for example based on cross validation results), or any other more advanced technique to decide which class is the most appropriate one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, googling for &lt;em&gt;hybrid sentiment&lt;/em&gt; returns tons of papers containing answers to the questions that you have stated. Please, don't ask us to rewrite this papers here.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-28T19:34:42.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5639" PostTypeId="1" CreationDate="2015-04-29T07:27:33.967" Score="6" ViewCount="8631" Body="&lt;p&gt;I would like to find different patterns recognition algorithm to detect different type of fraud. I have 1 million unstructured text documents about the clients' information with metadata about the client name, viewers, location in the cloud.&lt;/p&gt;&#xA;&lt;p&gt;Here are the patterns I was thinking of and the related fraud that i would like to detect:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Numeric Patterns - fictitious invoice numbers, fictitiously-generated transaction amounts.&lt;/p&gt;&#xA;&lt;p&gt;Time Patterns - transactions occurring too regularly, activity at unusual times or dates.&lt;/p&gt;&#xA;&lt;p&gt;Name Patterns - similar and alerted name and addresses.&lt;/p&gt;&#xA;&lt;p&gt;Geographic Patterns - Proximity relationships between apparently unrelated entities.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;What technique can I use? any keywords?&lt;/p&gt;&#xA;" OwnerUserId="9362" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-05-02T14:21:23.940" Title="Fraud detection use text mining" Tags="|text-mining|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5641" PostTypeId="2" ParentId="5589" CreationDate="2015-04-29T12:07:56.710" Score="13" Body="&lt;p&gt;Since you obviously posses an AWS account I'd recommend the following:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create an EC2 instance (any size)  &lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://www.gnu.org/software/wget/&quot; rel=&quot;noreferrer&quot;&gt;wget&lt;/a&gt;(or curl) to fetch the file(s) to that EC2 instance. For example: &lt;code&gt;wget http://example.com/my_large_file.csv&lt;/code&gt;.  &lt;/li&gt;&#xA;&lt;li&gt;Install &lt;a href=&quot;http://s3tools.org/&quot; rel=&quot;noreferrer&quot;&gt;s3cmd&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;s3cmd&lt;/code&gt; to upload the file to S3. For example: &lt;code&gt;s3cmd cp my_large_file.csv s3://my.bucket/my_large_file.csv&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Since connections made between various AWS services leverage AWS's internal network, uploading from an EC2 instance to S3 is pretty fast. Much faster than uploading it from your own computer. This way allows you to avoid downloading the file to your computer and saving potentially significant time uploading it through the web interface.&lt;/p&gt;&#xA;" OwnerUserId="8373" LastActivityDate="2015-04-29T12:07:56.710" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5642" PostTypeId="2" ParentId="155" CreationDate="2015-04-29T13:53:09.180" Score="16" Body="&lt;p&gt;The UK Government provide an excellent source of non-personal data collected throughout government departments: &lt;a href=&quot;http://data.gov.uk&quot;&gt;http://data.gov.uk&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9370" LastActivityDate="2015-04-29T13:53:09.180" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="5643" PostTypeId="1" CreationDate="2015-04-29T14:51:59.177" Score="0" ViewCount="743" Body="&lt;p&gt;Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking (which can have very many candidates in relation to the correct spelling of the word in question); rather I am looking for a similar list as used by Microsoft Word (for instance correcting &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;teh&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;becaise&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;because&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not only can the manner in which these sort of errors are fixed be hard-coded, their frequent occurrence in text provides significant dividends in textual mining (provided that such a list of errors and corrections can be obtained, of course).&lt;/p&gt;&#xA;" OwnerUserId="5290" LastActivityDate="2015-07-28T19:30:03.753" Title="Automomatic typographical error correction" Tags="|text-mining|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5644" PostTypeId="1" CreationDate="2015-04-29T16:43:34.397" Score="-1" ViewCount="1565" Body="&lt;p&gt;What is the difference between data mining approaches: frequent itemsets and item-based collaborative filtering in the area of recommender systems?&lt;/p&gt;&#xA;" OwnerUserId="9371" LastActivityDate="2015-04-29T18:44:29.593" Title="How is frequent itemsets compared with item-based collaborative filtering in recommender systems?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5645" PostTypeId="2" ParentId="5629" CreationDate="2015-04-29T17:05:22.277" Score="1" Body="&lt;p&gt;I see this question got up-voted so someone else is looking for the answer. Here it is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was logged in as the default user on Ubuntu instance when i did the &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$s3cmd get s3://data-analysis/input-data/filename.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data got saved from the s3 bucket to the home directory of the user named ubuntu. this can be verified with : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$ df -h /home/ubuntu/&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I had followed &lt;a href=&quot;http://randyzwitch.com/r-amazon-ec2/&quot; rel=&quot;nofollow&quot; title=&quot;randy zwitch&amp;#39;s tutorial&quot;&gt;randy zwitch's tutorial&lt;/a&gt; on installing R on aws - i had created a user named rstudio and that was the user name I was using to log-in to rstudio server. Hence I had to move the file from the default 'ubuntu' user's home directory to the user 'rstudio'. this can be done with: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$ sudo mv /home/ubuntu/filename.csv /home/rstudio&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy to answer follow up questions. &lt;/p&gt;&#xA;" OwnerUserId="8553" LastActivityDate="2015-04-29T17:05:22.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5646" PostTypeId="2" ParentId="5643" CreationDate="2015-04-29T17:07:14.157" Score="1" Body="&lt;p&gt;I will reference two sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Wikipedia has its own lists: &lt;a href=&quot;http://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings&quot; rel=&quot;nofollow&quot;&gt;Lists of common misspellings&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://aspell.net/&quot; rel=&quot;nofollow&quot;&gt;Aspell&lt;/a&gt; is certainly a place to visit. You can find a list of misspellings and corrections &lt;a href=&quot;http://aspell.net/test/common-all/batch0.tab&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Still, it is a good idea to think of using a spell checker, such as Aspell, since it gives you the ability to deal with no-so-common misspellings (of course, at the cost of false positives). A good starting point is Peter Norvig's &lt;a href=&quot;http://norvig.com/spell-correct.html&quot; rel=&quot;nofollow&quot;&gt;essay&lt;/a&gt; on creating a spelling corrector.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-04-29T17:07:14.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5647" PostTypeId="2" ParentId="5644" CreationDate="2015-04-29T18:44:29.593" Score="1" Body="&lt;p&gt;I think by &quot;frequent itemsets&quot; you mean the association rules that you derive from counting frequent itemsets. They tell us:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;people who do (like, buy, etc.) X tend to do (...) Y.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this you only need &quot;positive&quot; information, i.e. people who liked/bought/did.&#xA;Say this is about movies, you only know who liked the movie, among the others you don't know whether they disliked it or didn't see it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;item-based CF, and other CF techniques are typically used when you have both positive and negative feedback, such as ratings 0 to 5. You then use this information to &lt;em&gt;predict unknown ratings&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For item-based CF, you're trying to predict a person's rating on item Y, and you look how this person has rated other &lt;em&gt;similar&lt;/em&gt; items, where &quot;X is similar to Y&quot; means that other people who have rated both X and Y have given them very similar ratings. The information you get is then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Your have previously liked similar items to Y, therefore we predict you'll like Y.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;User-based CF is perhaps closer to association rules, and you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;People similar to you have liked Y, therefore we predict you will like Y.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="9114" LastActivityDate="2015-04-29T18:44:29.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5648" PostTypeId="1" CreationDate="2015-04-29T20:39:45.520" Score="2" ViewCount="3795" Body="&lt;p&gt;I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me.&#xA;I am trying to generate a matrix of small figures which can show me any hidden pattern etc. Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fig, axes = plt.subplots(nrows=4, ncols=6, figsize=(12, 8), &#xA;                         tight_layout=True)&#xA;&#xA;bins = np.arange(1950,2012,3)&#xA;for ax, genre in zip(axes.ravel(), movieGenre):&#xA;    ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')&#xA;    ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')&#xA;    ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)&#xA;    ax.xaxis.set_ticks(np.arange(1950, 2013, 30))&#xA;    ax.set_yticks([])&#xA;    ax.set_xlabel('Year')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first &lt;code&gt;hist&lt;/code&gt; isn't working, but the second one is working when I am commenting out the first one. Here is the traceback:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; KeyError                                  Traceback (most recent call last)&#xA;&amp;lt;ipython-input-158-c2e7c2737372&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      4 bins = np.arange(1950,2012,3)&#xA;      5 for ax, genre in zip(axes.ravel(), movieGenre):&#xA;----&amp;gt; 6     ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')&#xA;      7     ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')&#xA;      8     ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)&#xA;&#xA;/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/matplotlib/axes.pyc in hist(self, x, bins, range, normed, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, **kwargs)&#xA;   8247         # Massage 'x' for processing.&#xA;   8248         # NOTE: Be sure any changes here is also done below to 'weights'&#xA;-&amp;gt; 8249         if isinstance(x, np.ndarray) or not iterable(x[0]):&#xA;   8250             # TODO: support masked arrays;&#xA;   8251             x = np.asarray(x)&#xA;&#xA;/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/series.pyc in __getitem__(self, key)&#xA;    477     def __getitem__(self, key):&#xA;    478         try:&#xA;--&amp;gt; 479             result = self.index.get_value(self, key)&#xA;    480 &#xA;    481             if not np.isscalar(result):&#xA;&#xA;/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/index.pyc in get_value(self, series, key)&#xA;   1169 &#xA;   1170         try:&#xA;-&amp;gt; 1171             return self._engine.get_value(s, k)&#xA;   1172         except KeyError as e1:&#xA;   1173             if len(self) &amp;gt; 0 and self.inferred_type == 'integer':&#xA;&#xA;&#xA;&#xA;KeyError: 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the first few columns of the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     imdbID     title                      rating    vote      runtime  year    genre&#xA;0   tt0111161   The Shawshank Redemption    9.3      1,439,277  142    1994 [Crime, Drama]&#xA;1   tt0468569   The Dark Knight             9.0      1,410,124  152    2008 [Action, Crime, Drama]&#xA;2   tt1375666   Inception                   8.8      1,209,159  148    2010 [Action, Mystery, Sci-Fi, Thriller]&#xA;3   tt0137523   Fight Club                  8.9      1,123,462  139    1999 [Drama]&#xA;4   tt0110912   Pulp Fiction                8.9      1,117,193  154    1994 [Crime, Drama]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;movieGenre is basically collecting all the different genres from 'genre' column with duplicates removed: &lt;code&gt;movieGenre = set(movieDF.genre.sum())&lt;/code&gt; . I then added a single column to movieDF data frame for each genre such that if a particular movie belong to that genre, then that cell is &lt;code&gt;True&lt;/code&gt; otherwise it is &lt;code&gt;False&lt;/code&gt;.  So for example, for the movie Inception, the &lt;code&gt;Action&lt;/code&gt; column is marked &lt;code&gt;True&lt;/code&gt; but &lt;code&gt;Crime&lt;/code&gt; column is marked &lt;code&gt;False&lt;/code&gt; and so forth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="3314" LastEditDate="2015-04-30T19:05:34.900" LastActivityDate="2015-04-30T19:32:25.803" Title="Where is the error in the following code?" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5649" PostTypeId="2" ParentId="5618" CreationDate="2015-04-30T06:19:06.690" Score="3" Body="&lt;p&gt;There is nothing wrong with using Pearson's correlation coefficient (frequently called THE correlation coefficient) with these variables. I would also have a look at Spearman's rank correlation coefficient. It is less sensitive to extreme values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, both coefficients have the drawback that they only measure linear relationships. Hence it is worthwile to have a look at the scatterplot of the variables. Adding a smoother to the graph can be worthwile too, especially  if the plot is clutterd and you have trouble to spot a relationship with the naked eye.&lt;/p&gt;&#xA;" OwnerDisplayName="user9380" LastActivityDate="2015-04-30T06:19:06.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5650" PostTypeId="1" CreationDate="2015-04-30T08:01:39.763" Score="2" ViewCount="68" Body="&lt;p&gt;I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this?&#xA;&lt;img src=&quot;https://i.stack.imgur.com/bGExV.png&quot; alt=&quot;equation&quot;&gt;&#xA;The link for paper is &lt;a href=&quot;http://arxiv.org/pdf/1411.5595v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6523" LastActivityDate="2015-05-24T10:04:25.767" Title="Pound notation in Summation" Tags="|machine-learning|notation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5651" PostTypeId="2" ParentId="5650" CreationDate="2015-04-30T08:18:14.520" Score="-1" Body="&lt;p&gt;That just means count. #(w) is the number of times w occurs in the corpus.&lt;/p&gt;&#xA;" OwnerUserId="9383" LastActivityDate="2015-04-30T08:18:14.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5652" PostTypeId="1" AcceptedAnswerId="5660" CreationDate="2015-04-30T10:06:11.107" Score="1" ViewCount="193" Body="&lt;p&gt;I am looking for a package that does gradient descent parameter estimation in R, maybe with some bootstrapping to get confidence intervals. I wonder if people call it something different here as I get almost nothing on my searches, and the one article I found was from someone who rolled their own. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not that hard to implement, but I would prefer to use something standard.&lt;/p&gt;&#xA;" OwnerUserId="7720" LastActivityDate="2015-05-01T08:47:31.613" Title="Gradient descent parameter estimation Package for R" Tags="|r|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5653" PostTypeId="1" CreationDate="2015-04-30T11:50:44.760" Score="-1" ViewCount="50" Body="&lt;p&gt;I would like to post a paper in International Conference on Soft Computing. I want to know whether the journal is a reputed journal. &lt;/p&gt;&#xA;" OwnerUserId="9322" LastActivityDate="2020-08-14T14:34:56.443" Title="International Journal" Tags="|data-mining|nlp|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-02T13:38:52.657" ContentLicense="CC BY-SA 3.0" />
  <row Id="5654" PostTypeId="1" AcceptedAnswerId="5686" CreationDate="2015-04-30T12:26:47.637" Score="4" ViewCount="923" Body="&lt;p&gt;Something that I often see in papers (&lt;a href=&quot;http://papers.nips.cc/paper/5333-large-scale-l-bfgs-using-mapreduce.pdf&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;) about large-scale learning is that click-through rate (CTR) problems can have up to a billion of features for each example. In &lt;a href=&quot;http://research.google.com/pubs/pub41159.html&quot; rel=&quot;nofollow&quot;&gt;this Google paper&lt;/a&gt; the authors mention:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The features used in our system are drawn from a variety of sources,&#xA;  including the query, the text of the ad creative, and various&#xA;  ad-related metadata.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I can imagine a few thousands of features coming from this type of source, I guess through some form of feature hashing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how does one get to a billion features? How do companies translate user behavior into features in order to reach that scale of features?&lt;/p&gt;&#xA;" OwnerUserId="9385" LastEditorUserId="9385" LastEditDate="2015-05-06T08:48:01.893" LastActivityDate="2015-05-06T08:48:01.893" Title="What types of features are used in a large-scale click-through rate prediction problem?" Tags="|machine-learning|classification|bigdata|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5655" PostTypeId="2" ParentId="5648" CreationDate="2015-04-30T13:38:55.750" Score="3" Body="&lt;p&gt;First of all, since you only seem to want to know about the Python error, your question is probably better suited for Stack Overflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, this is not fun to answer with the information you've provided.  This is certainly not a minimal working example (MWE) - I can't copy your code as is, and run it myself.  I'm left to guess many things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are you importing &lt;code&gt;matplotlib&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Did you rename &lt;code&gt;matplotlib.pyplot&lt;/code&gt; as &lt;code&gt;plt&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Are you importing &lt;code&gt;numpy&lt;/code&gt; and did you rename it &lt;code&gt;np&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Does your traceback literally end in &lt;code&gt;KeyError is 0&lt;/code&gt; or is it &lt;code&gt;KeyError: 0&lt;/code&gt; or &lt;code&gt;KeyError: False&lt;/code&gt;?  Can you show the full traceback?  The traceback for a MWE is typically not long.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the positional argument (&lt;code&gt;movieDF[movieDF['%s'%genre]==1].year&lt;/code&gt;) on your problematic line, you ask for a key (&lt;code&gt;movieDF['%s'%genre]==1&lt;/code&gt;) that looks like it should evaluate to a Boolean.  Is this never 0?  I'm forced to grasp at straws because you haven't provided a sample dataset; I shouldn't have to download from IMDb myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to troubleshoot, can you place &lt;code&gt;x = movieDF[movieDF['%s'%genre]==1].year&lt;/code&gt; on a line by itself and tell me what &lt;code&gt;x[0]&lt;/code&gt; is?  What type would that &lt;code&gt;x&lt;/code&gt; be?  Is it a dictionary, where the &lt;code&gt;.hist()&lt;/code&gt; method is expecting a list or other array?  &lt;a href=&quot;http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.hist&quot; rel=&quot;nofollow&quot;&gt;This documentation&lt;/a&gt; makes it sound like the method will not accept a dictionary...what type is &lt;code&gt;movieDF.year&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="8953" LastEditorUserId="8953" LastEditDate="2015-04-30T19:32:25.803" LastActivityDate="2015-04-30T19:32:25.803" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5656" PostTypeId="1" AcceptedAnswerId="5662" CreationDate="2015-04-30T13:42:05.343" Score="20" ViewCount="23333" Body="&lt;p&gt;When a random initialization of centroids is used, different runs of K-means produce different total SSEs. And it is crucial in the performance of the algorithm. &#xA;What are some effective approaches toward solving this problem? Recent approaches are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9388" LastEditorUserId="8953" LastEditDate="2015-05-02T13:40:32.223" LastActivityDate="2020-08-16T12:59:00.477" Title="K-means: What are some good ways to choose an efficient set of initial centroids?" Tags="|data-mining|clustering|k-means|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5657" PostTypeId="2" ParentId="5656" CreationDate="2015-04-30T18:26:46.633" Score="5" Body="&lt;p&gt;I may be misunderstanding your question, but usually k-means chooses your centroids randomly for you depending on the number of clusters you set (i.e. k). Choosing the number for k tends to be a subjective exercise. A good place to start is an Elbow/Scree plot which can be found &lt;a href=&quot;http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="525" LastEditorUserId="98307" LastEditDate="2020-08-16T12:59:00.477" LastActivityDate="2020-08-16T12:59:00.477" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5658" PostTypeId="2" ParentId="5653" CreationDate="2015-04-30T19:17:04.320" Score="1" Body="&lt;p&gt;You can check it &lt;a href=&quot;http://www.scimagojr.com/journalrank.php?category=1712&amp;amp;area=0&amp;amp;year=2013&amp;amp;country=US&amp;amp;order=sjr&amp;amp;min=0&amp;amp;min_type=cd&amp;amp;page=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. At least that one is in the list.&lt;/p&gt;&#xA;" OwnerUserId="8394" LastEditorUserId="98307" LastEditDate="2020-08-14T14:34:56.443" LastActivityDate="2020-08-14T14:34:56.443" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5659" PostTypeId="2" ParentId="5534" CreationDate="2015-04-30T19:23:00.970" Score="14" Body="&lt;p&gt;Instead of scraping,  you might try to get the data directly &lt;a href=&quot;http://www.imdb.com/interfaces&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.  It looks like they have data available via ftp for movies, actors, etc.&lt;/p&gt;&#xA;" OwnerUserId="8284" LastEditorUserId="98307" LastEditDate="2020-08-02T18:20:21.780" LastActivityDate="2020-08-02T18:20:21.780" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5660" PostTypeId="2" ParentId="5652" CreationDate="2015-05-01T08:47:31.613" Score="1" Body="&lt;p&gt;Ok, after a lot of looking I found the &quot;optim&quot; routine which is in &quot;stats&quot;, one of the packages that is always loaded. It has quite a few methods including conjugate gradients and BGGS and a few others and worked well on the first few examples I tried. It doesn't seem to get a lot of attention strangely. I guess optimization people tend to use Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I knew there had to be something.&lt;/p&gt;&#xA;" OwnerUserId="7720" LastActivityDate="2015-05-01T08:47:31.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5661" PostTypeId="2" ParentId="5656" CreationDate="2015-05-01T12:54:39.857" Score="4" Body="&lt;p&gt;The usual approach to this problem is to re-run your K-means algorithm several times, with different random initializations of the centroids, and to keep the best solution. You can do that by evaluating the results on your training data or by means of cross validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many other ways to initialize the centroids, but none of them is going to perform the best for every single problem. You could evaluate these approaches together with random initialization for your particular problem. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-05-01T12:54:39.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5662" PostTypeId="2" ParentId="5656" CreationDate="2015-05-01T14:06:08.007" Score="14" Body="&lt;p&gt;An approach that yields more consistent results is &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means%2B%2B&quot;&gt;K-means++&lt;/a&gt;.  This approach acknowledges that there is probably a better choice of initial centroid locations than simple random assignment. Specifically, K-means tends to perform better when centroids are seeded in such a way that doesn't clump them together in space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, the method is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose one of your data points at random as an initial centroid.&lt;/li&gt;&#xA;&lt;li&gt;Calculate $D(x)$, the distance between your initial centroid and all other data points, $x$.&lt;/li&gt;&#xA;&lt;li&gt;Choose your next centroid from the remaining datapoints with probability proportional to $D(x)^2$&lt;/li&gt;&#xA;&lt;li&gt;Repeat until all centroids have been assigned.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: $D(x)$ should be updated as more centroids are added.  It should be set to be the distance between a data point and the nearest centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also be interested to read &lt;a href=&quot;http://ilpubs.stanford.edu:8090/778/1/2006-13.pdf&quot;&gt;this paper&lt;/a&gt; that proposes the method and describes its overall expected performance.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-05-01T14:06:08.007" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5663" PostTypeId="1" AcceptedAnswerId="5669" CreationDate="2015-05-01T15:19:17.943" Score="4" ViewCount="564" Body="&lt;p&gt;I am not in the data science field, but I would like to examine in depth this field and, particularly, I would like to start from the analysis of the social networks data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to find some good references, both paper, websites and books, in order to start learning about the topic. Browsing on the internet, one can find a lot of sites, forum, papers about the topic, but I'm not able to discriminate among good and bad readings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am an R, Matlab, SAS user and I know a little bit of python language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you suggest any references from which I could start studying and deepen the industry?&lt;/p&gt;&#xA;" OwnerUserId="9225" LastEditorUserId="843" LastEditDate="2015-10-17T12:29:19.000" LastActivityDate="2015-10-17T12:29:19.000" Title="Reference about social network data-mining" Tags="|bigdata|beginner|social-network-analysis|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5664" PostTypeId="2" ParentId="5639" CreationDate="2015-05-01T15:29:48.200" Score="2" Body="&lt;p&gt;Without more detail, it's difficult to give detailed advice, but at first glance, this seems like a textbook case for machine learning. See if you can get a similar data set with known fraudulent activity and, depending on the data, pick a machine learning approach. &lt;/p&gt;&#xA;" OwnerUserId="9236" LastActivityDate="2015-05-01T15:29:48.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5665" PostTypeId="2" ParentId="5534" CreationDate="2015-05-01T16:52:15.717" Score="4" Body="&lt;p&gt;As a bit of general feedback, I think you would do well to improve your output format.  The problem with the format as it stands is there is not a transparent way to programmatically get the data. Consider instead trying:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print &quot;\t&quot;.join([title, genres,runtime, rating, year])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The nice thing about a tab delimited file is that if you end up scaling up, it can easily be read into something like impala (or at smaller scales, simple mySql tables).  Additionally, you can then programatically read in the data in python using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; line.split(&quot;\t&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second bit of advice, is I would suggest getting more information than you think you need on your initial scrape.  Disk space is cheaper than processing time, so rerunning the scraper every time you expand your analytic will not be fun.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-01T16:52:15.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5666" PostTypeId="1" AcceptedAnswerId="5668" CreationDate="2015-05-01T19:31:34.570" Score="2" ViewCount="1458" Body="&lt;p&gt;I've got survey data that resembles:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|-------------| Q1a | Q1b | Q1c | Q2a | Q2b | Q2c | Classification&#xA;| Respondent  | 1   | 0   | 0   | 1   | 0   | 0   | Red&#xA;| Respondent  | 0   | 0   | 1   | 1   | 0   | 0   | Green&#xA;| Respondent  | 0   | 1   | 0   | 0   | 0   | 1   | Yellow&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict the classification for new respondents. Currently I'm using a Naive Bayes, and getting pretty bad accuracy (~20%). I don't have much training data, and the training data is hand scraped from non-standard sources (internal company procedures are a mess here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for other ways to predict the classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking about assigning weights to each question, and magically predicting the result based on those, somehow. Although I don't really know where to start learning about how to do that, and whether it's appropriate for this data. I have very little background in this :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas or tips on predicting the classification column with no training data?&lt;/p&gt;&#xA;" OwnerUserId="9410" LastActivityDate="2015-05-01T20:57:34.133" Title="Analyzing survey data for predictions" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5667" PostTypeId="1" AcceptedAnswerId="6752" CreationDate="2015-05-01T20:32:51.900" Score="10" ViewCount="14782" Body="&lt;p&gt;I have a large dataset that I need to split into groups according to specific parameters.  I want the job to process as efficiently as possible.  I can envision two ways of doing so&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 1&lt;/strong&gt; - Create map from original RDD and filter&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def customMapper(record):&#xA;    if passesSomeTest(record):&#xA;        return (1,record)&#xA;    else:&#xA;        return (0,record)&#xA;&#xA;mappedRdd = rddIn.map(lambda x: customMapper(x))&#xA;rdd0 = mappedRdd.filter(lambda x: x[0]==0).cache()&#xA;rdd1 = mappedRdd.filter(lambda x: x[1]==1).cache()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 2&lt;/strong&gt; - Filter original RDD directly&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def customFilter(record):&#xA;    return passesSomeTest(record)&#xA;&#xA;rdd0 = rddIn.filter(lambda x: customFilter(x)==False).cache()&#xA;rdd1 = rddIn.filter(customFilter).cache()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fist method has to itterate over all the records of the original data set 3 times, where the second only has to do so twice, under normal circumstances, however, spark does some behind the scenes graph building, so I could imagine that they are effectively done in the same way.  My questions are:&#xA;a.) Is one method more efficient than the other, or does the spark graph building make them equivalent&#xA;b.) Is it possible to do this split in a single pass&lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="17116" LastEditDate="2016-05-11T20:14:31.130" LastActivityDate="2016-05-11T20:14:31.130" Title="Spark, optimally splitting a single RDD into two" Tags="|apache-spark|pyspark|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5668" PostTypeId="2" ParentId="5666" CreationDate="2015-05-01T20:57:34.133" Score="2" Body="&lt;p&gt;Can you give a bit more information on the size of the data you're training on (and if it's really 6 parameters you're basing the predictions off of)?  If it's really 6 questions with binary answers (1, 0 as you suggest), then there are 2^6 (i.e. 64) unique answer combinations, and to determine a probability for them you'll want a multiple entries per combination.  Standard error scales like 1/sqrt(n) so for 10% accuracy you'll need roughly 6,400 inputs which given your description, sounds like more data than you may have.  You may want to invest time into automating data collection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If on the other hand, you have a reasonably large data set and are hoping for some alternative models, both boosted decision trees and random forest models sound like good candidates for this problem.  &lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-01T20:57:34.133" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5669" PostTypeId="2" ParentId="5663" CreationDate="2015-05-01T21:42:55.350" Score="2" Body="&lt;p&gt;My favorite place to find information about social network analysis is from SNAP, the Stanford Network Analysis Project. Led by Jure Leskovec, this team of students and professors has built software tools, gathered data sets, and published papers on social network analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://snap.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;http://snap.stanford.edu/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The collection of research papers there is outstanding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They also have a Python tool you could try. &lt;a href=&quot;http://snap.stanford.edu/snappy/index.html&quot; rel=&quot;nofollow&quot;&gt;http://snap.stanford.edu/snappy/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The focus is on graph analysis, because social networks fit this model well. If you are new to graph analysis, I suggest you take a undergraduate level discrete mathematics course, or check out my favorite book on the topic &quot;Graph Theory with Algorithms and its Applications&quot; by Santanu Ray.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a hands-on approach to social network analysis, check out &quot;Mining the Social Web&quot; by Matthew A Russell. It has examples which cover how to collect and analyze data from the major social networks like Twitter, Facebook, and LinkedIn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was Jure Leskovec who initially excited me about this field. He has many great talks on YouTube, for example: &lt;a href=&quot;https://www.youtube.com/watch?v=LmQ_3nijMCs&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=LmQ_3nijMCs&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-05-01T21:42:55.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5670" PostTypeId="2" ParentId="2258" CreationDate="2015-05-02T01:16:38.717" Score="1" Body="&lt;p&gt;Neural Networks are not a great introductory model, simply because of the complexity that you describe.  If you're trying to get your feet wet, boosted decision trees tend to perform well by comparison, and are a bit more intuitive.  If you want a description on this method, and are already familiar with Coursera, The University of Washington has an introductory course on data science which explains it quite well.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-02T01:16:38.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5671" PostTypeId="2" ParentId="685" CreationDate="2015-05-02T01:26:46.707" Score="1" Body="&lt;p&gt;A lot of the features you mentioned are categorical, and with so many levels of each, the dimensions of your problem will expand.  Rather than initially focusing on Lasso and Ridge regression, why don't you first look for clusters among the samples (records) to learn about the data set?  Under you current approach, you're throwing everything into a model and expecting a high AUC(?).  You may find several clusters where frequencies of the category levels predominate in one or more potential clusters.  If you don't know what the cluster structure is of the samples (records), try k-means clustering based on centroids of feature values to see if there are unique clusters.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get a handle on the cluster structure, then address regression issues.  Your regression models may be breaking down, in part, because of large inhomogeneities in your data, along with the previously suggested issues.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning is all about performing unsupervised class discovery followed by class prediction (your output binary variable).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, it's not clear that you studied the data to learn about its cluster structure, and rather threw it into a supervised model expecting to attain high AUC values.   &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastEditorDisplayName="user9086" LastEditDate="2015-05-03T21:00:08.863" LastActivityDate="2015-05-03T21:00:08.863" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5672" PostTypeId="2" ParentId="5663" CreationDate="2015-05-02T02:55:09.197" Score="3" Body="&lt;p&gt;I'm going to pursue the following series of online courses on the Coursera: &#xA;&lt;a href=&quot;https://www.coursera.org/specialization/socialscience/25?utm_medium=listingPage&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Become a Social Scientist: Methods and Statistics&lt;/strong&gt;&lt;/a&gt; by &lt;em&gt;University of Amsterdam&lt;/em&gt;. The good news - it is free, or you can get a nice-looking certificate for $49 or so. The bad news - the nearest enrollment is Aug 31st 2015. You will have opportunity to get a lot of information in condensed way during a short time frame and you will be enforced to actually apply the knowledge in exercises, quizzes and project assignments. You will also have opportunity to discuss lessons/projects on the forum with many other students and lecturers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[update] I apologize, I just remembered there is another course &lt;a href=&quot;https://www.coursera.org/course/statistics101&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Introduction to Statistics for the Social Sciences&lt;/strong&gt;&lt;/a&gt; by &lt;em&gt;University of Zurich&lt;/em&gt; - just started April 28th, 2015. If you want to pursue it - do not forget about deadlines for quizzes and homeworks. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="8394" LastEditorUserId="8394" LastEditDate="2015-05-04T01:10:20.467" LastActivityDate="2015-05-04T01:10:20.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5674" PostTypeId="2" ParentId="5268" CreationDate="2015-05-03T00:30:18.570" Score="9" Body="&lt;p&gt;Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting.  Separate your data (in your case the &quot;back data&quot;) into 60% training data and 40% testing data.  Fit the model on the training data as you usually would and see how well it is working with the test data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, when you think you have the model that you want, split each of the training and test sets into 10-100 subsets and retrain and test with incrementally larger sets.  Apply your favorite performance metric and plot the results of performance vs. the number of cases used for testing and training.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The curves will never come together if the model is overfit (high variance).  The curves will come together but the performance will be lower than desired if the model is underfit (high bias) and the lines will come together at an acceptable performance for a well performing model that is not overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of overfitting and underfitting with root mean square error as the performance metric:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/m3a90.png&quot; alt=&quot;Bias-Variance decomposition via learning curves&quot;&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://followthedata.wordpress.com/2012/06/02/practical-advice-for-machine-learning-bias-variance/&quot; rel=&quot;noreferrer&quot;&gt;Here is a pretty good link&lt;/a&gt; on the process and &lt;a href=&quot;http://www.astroml.org/sklearn_tutorial/practical.html&quot; rel=&quot;noreferrer&quot;&gt;here is another one&lt;/a&gt;.  Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-05-03T00:30:18.570" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5675" PostTypeId="2" ParentId="5639" CreationDate="2015-05-03T00:58:54.410" Score="1" Body="&lt;p&gt;I would suggest playing around with this in either &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; or &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt; using &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;Scikit Learn&lt;/a&gt;.  The text will take a bit of playing with, but is pretty straightforward to normalize and turn into a feature set.  I suggest applying &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow&quot;&gt;TFIDF&lt;/a&gt; to normalize each text document against the entire corpus.  You will then have lots of text features that you can join with your metadata features to do novelty detection.  Scikit-Learn and it's excellent &lt;a href=&quot;http://scikit-learn.org/stable/user_guide.html#&quot; rel=&quot;nofollow&quot;&gt;User Guide&lt;/a&gt; are a great resource for getting started in this arena as are &lt;a href=&quot;http://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout&lt;/a&gt; (Java based) and the book &lt;a href=&quot;http://www.manning.com/owen/&quot; rel=&quot;nofollow&quot;&gt;Mahout In Action&lt;/a&gt;.  There are also countless packages in R to accomplish this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing to note is that you seem to suggest specific ways that you want to detect fraud.  This type of hunch based machine learning is known as applying heuristics and tends to perform worse than pure machine learning methods.  You should just focus on using novelty detection algorithms or possibly anomaly detection algorithms and let the statistics find the fraud rather than trying to apply your own intuition.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-05-03T00:58:54.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5676" PostTypeId="2" ParentId="5639" CreationDate="2015-05-03T04:22:43.507" Score="6" Body="&lt;p&gt;I have worked on building a fraud detection solution using text mining, so I understand the scenario that has led to this question. I'll talk about the approach/techniques that is to be followed to build the fraud detection solution.I'll divide it into 4 sections&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Build line of business fraud dictionary&lt;/li&gt;&#xA;&lt;li&gt;Fraud detection &amp;amp; scoring&lt;/li&gt;&#xA;&lt;li&gt;Tools&lt;/li&gt;&#xA;&lt;li&gt;Additional notes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Building Fraud Dictionary:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;As a first step you'll have to identify fraud concepts with help from subject matter expert for the line of business in concern. Fraud concept is a person, characteristic, entity, or event that represents a suspicious scenario and similar concepts can be grouped together to form a concept category. Each concept is further represented by words, phrases, entities etc. The presence of these words/phrases in the document implies occurrence of fraud concept in that transaction. The end result of this exercise will be a fraud dictionary that is a repository of concepts and suspicious key words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example considering the data set mentioned in the question, the &lt;code&gt;numeric patterns&lt;/code&gt; become the concept category, and &lt;code&gt;fictitious invoice numbers&lt;/code&gt; and &lt;code&gt;fictitiously-generated transaction amounts&lt;/code&gt; are 2 different concepts. There will be a keywords/phrases that is associated to this concept which should be captured onto the fraud dictionary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use NLP technique to build the dictionary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) Convert the sentence to lowercase&#xA;&#xA;2) Remove stopwords, these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words&#xA;&#xA;3) Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text. simply increasing n, model can be used to store more context&#xA;&#xA;4) Assign a syntactic label (noun, verb etc.)&#xA;&#xA;5) Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Fraud detection &amp;amp; scoring:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Phrases/Keywords to be semantically matched with Phrases/Keywords from Fraud Dictionary and Fraud concepts in transactions can be identified. At this point, care needs to be taken to maintain the context (negation, positive sense) in which a keyword/phrase has been used to avoid false positives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each identified occurrence of Fraud concept in the transaction, a weight of 1 can be assigned. This is done for calculating a suspicious score for the transaction. Higher the score high suspecious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that using a combination of larger data set with high textual content, and extensive fraud dictionary would result in higher number potentially identified.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Tools:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Additional notes:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You can refer &lt;a href=&quot;https://datascience.stackexchange.com/a/5388/8465&quot;&gt;here&lt;/a&gt; to get an idea of extracting concepts from sentences&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fruther read my blog on &lt;a href=&quot;https://manoharswamynathan.wordpress.com/category/text-mining/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text mining 101&lt;/a&gt; to learn more about &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;TM process overview&lt;/li&gt;&#xA;&lt;li&gt;Calculate term weight (TF-IDF)&lt;/li&gt;&#xA;&lt;li&gt;Similarity distance measure (Cosine)&lt;/li&gt;&#xA;&lt;li&gt;Overview of key text mining techniques&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="8465" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-03T08:51:33.677" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5677" PostTypeId="1" AcceptedAnswerId="5683" CreationDate="2015-05-03T07:11:50.737" Score="2" ViewCount="411" Body="&lt;p&gt;I have two series of values, a and b as inputs and I want to create a score, c, which reflects both of them equally.  The distribution of a and b are below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/TSKfA.png&quot; alt=&quot;Distribution of a&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/WivxT.png&quot; alt=&quot;Distribution of b&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases, the x-axis is just an index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I go about creating an equation c = f(a,b) such that a and b are (on average) represented equally in c?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: c = (a+b)/2 or c = ab will not work because c will be too heavily weighted by a or b.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need a function, f, where c = f(a,b) and c' = f(a + stdev(a),b) = f(a, b + stdev(b))&lt;/p&gt;&#xA;" OwnerUserId="9394" LastEditorDisplayName="user9380" LastEditDate="2015-05-04T17:25:53.173" LastActivityDate="2015-05-04T17:25:53.173" Title="How should I create a single score with two values as input?" Tags="|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5678" PostTypeId="2" ParentId="4883" CreationDate="2015-05-03T09:54:34.857" Score="2" Body="&lt;p&gt;Different domains might require different approaches, the news domain being one of the more elusive ones. News items are extremely short-lived (one, maybe two days): the very items the user viewed two days ago are just not interesting anymore. Also, CF works well if a large amount of items have &quot;enough&quot; events (that is, lots of users have viewed/rated them), and depending on the size of your data, most news items just don't collect enough views during their lifetime for classical CF methods to be effective. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, news recommender systems tend to use some combination of content-based filtering (the tags and categories you mention), popularity-based recommendation (this often discarded approach might work well for news), and recency scoring. There is some literature around it, try googling for &quot;news recommender system&quot;.&lt;/p&gt;&#xA;" OwnerUserId="7921" LastEditorUserId="7921" LastEditDate="2015-05-03T19:02:51.487" LastActivityDate="2015-05-03T19:02:51.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5679" PostTypeId="1" CreationDate="2015-05-03T14:12:09.000" Score="0" ViewCount="581" Body="&lt;p&gt;In Support Vector Machines, when used for sentiment analysis, text gets converted into a set of data points. How does this happen, usually?&lt;/p&gt;&#xA;" OwnerUserId="9322" LastEditorUserId="1367" LastEditDate="2015-05-24T10:10:00.543" LastActivityDate="2015-05-24T10:10:00.543" Title="Understanding text conversion into SVM input" Tags="|machine-learning|nlp|svm|libsvm|" AnswerCount="2" CommentCount="1" ClosedDate="2015-06-02T09:33:51.120" ContentLicense="CC BY-SA 3.0" />
  <row Id="5680" PostTypeId="2" ParentId="5677" CreationDate="2015-05-03T18:35:31.813" Score="1" Body="&lt;p&gt;using R syntax:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; c &amp;lt;- a/2+b/2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or you can create a function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c &amp;lt;- function(a,b) { return(a/2+b/2) }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8394" LastActivityDate="2015-05-03T18:35:31.813" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5681" PostTypeId="2" ParentId="5373" CreationDate="2015-05-03T21:36:27.670" Score="2" Body="&lt;p&gt;The problem is that you're converting a large sparse dataset into a dense array via &lt;code&gt;toarray()&lt;/code&gt;. Don't do that :) Why do you need it in dense array format? Try posting more of your code to see if we can figure out better ways to solve the problem. With any large dataset, you want to stay sparse as long as possible and only work with smaller sets of dense data (if dense is needed at all).&lt;/p&gt;&#xA;" OwnerUserId="9433" LastActivityDate="2015-05-03T21:36:27.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5682" PostTypeId="2" ParentId="5679" CreationDate="2015-05-04T00:22:47.427" Score="2" Body="&lt;p&gt;Text can be converted to data via the use of concept clusters (after stemming and stopping), or to count (frequencies) via use of n-grams.  N-grams are basically tabulations of the 1-gram count (frequency) of alphabet characters (a though z) in each document, and counts of 2-grams (aa to zz), 3-grams (aaa through zzz), up to about 5-grams (aaaaa through zzzzz).  Beyond 5-grams, the data will be sparse and less informative.  Thus, a dataset can be constructed for which rows represent documents, and columns represent n-grams. The data values themselves are the total number of occurrences of each gram found in each document.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYI - n-grams have proven to be the best technique for identifying different languages based on characters.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding SVMs, focus on the SVM literature.           &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2015-05-04T00:22:47.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5683" PostTypeId="2" ParentId="5677" CreationDate="2015-05-04T04:29:06.830" Score="4" Body="&lt;p&gt;If you're looking for something where A and B are equally represented, consider trying something like Z score normalization (or &lt;a href=&quot;http://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow&quot;&gt;standard score&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c = (a-u_a)/sigma_a + (b-u_b)/sigma_b&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That score equally represents the two, but would be on a smaller scale.  It really shouldn't matter since the numbers are arbitrary, however, if you need to scale it up, you could do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c2 = (sigma_a+sigma_b)*(c) + u_a + u_b&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="8041" LastEditDate="2015-05-04T14:40:13.423" LastActivityDate="2015-05-04T14:40:13.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5684" PostTypeId="1" CreationDate="2015-05-04T13:26:04.767" Score="4" ViewCount="4580" Body="&lt;p&gt;I have a question about MLlib in Spark.(with Scala)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but I now have to do it on Spark to be able to analyze Big Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is the variable selection done? Is there any try of different variable combinations in LogisticRegressionWithLBFGS or LogisticRegressionWithSGD? Something like a test of significance of variable one by one? Or a correlation calculation with the variable of interest? Is there any calculation of BIC, AIC to choose the best model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the model only returns weights and intercept...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I understand those Spark functions and compare to what I'm used to with SAS or R ?&lt;/p&gt;&#xA;" OwnerUserId="9445" LastActivityDate="2016-12-18T00:20:48.310" Title="How Mllib in Spark select variables in logistic regression" Tags="|machine-learning|bigdata|logistic-regression|scala|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5685" PostTypeId="2" ParentId="5684" CreationDate="2015-05-04T14:59:36.430" Score="4" Body="&lt;p&gt;First, the spark programming guide for &lt;a href=&quot;http://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithSGD&quot; rel=&quot;nofollow&quot;&gt;LogisticRegressionWithSGD&lt;/a&gt; recommends using &lt;a href=&quot;http://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS&quot; rel=&quot;nofollow&quot;&gt;L-BFGS&lt;/a&gt; instead, so perhaps focus on the one.  As for variable selection, the model description on the &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-linear-methods.html#logistic-regression&quot; rel=&quot;nofollow&quot;&gt;MLLib page for regressions&lt;/a&gt; has a nice explanation of how models are constructed and selected, but it does not address variable selection.  This leads me to believe that it considers all variables, and simply chooses the model with the best fit.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-04T14:59:36.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5686" PostTypeId="2" ParentId="5654" CreationDate="2015-05-04T19:55:24.797" Score="2" Body="&lt;p&gt;That really is a nice question, although once you're Facebook or Google etc., you have the opposite problem: how to reduce the number of features from many billions, to let's say, a billion or so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There really are billions of features out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine, that in your feature vector you have billions of possible phrases that the user could type in into search engine. Or, that you have billions of web sites a user could visit. Or millions of locations from which a user could log in to the system. Or billions of mail accounts a user could send mails to or receive mails from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, to swich a bit to social networking site-like problem. Imagine that in your feature vector you have billions of users which a particular user could either know or be in some degree of separation from. You can add billions of links that user could post in his SNS feed, or millions of pages a user could 'like' (or do whatever the SNS allows him to do).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar problems may be found in many domains from voice and image recognition, to various branches of biology, chemistry etc. I like your question, because it's a good starting point to dive into the problems of dealing with the abundance of features. Good luck in exploring this area!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE due to your comment:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using features other than binary is just one step further in imagining things. You could somehow cluster the searches, and count frequencies of searches for a particular cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a SNS setting you could build a vector of relations between users defined as degree of separation instead of a mere binary feature of being or not being friends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine logs that global corporations are holding on millions of their users. There's a whole lot of stuff that can be measured in a more detailed way than binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things become even more complicated once we're considering an online setting. In such a case you do not have time for complicated computations and you're often left with binary features since they are cheaper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And no, I am not saying, that the problem becomes tractable once it's reduced to a magical number of billion features. I am only saying that a billion of features is something you may end up after a lot of effort in reducing the number of dimensions.&lt;/p&gt;&#xA;" OwnerUserId="173" LastEditorUserId="173" LastEditDate="2015-05-06T05:30:33.507" LastActivityDate="2015-05-06T05:30:33.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5687" PostTypeId="1" AcceptedAnswerId="5688" CreationDate="2015-05-05T07:12:46.797" Score="3" ViewCount="214" Body="&lt;p&gt;What are the tools, practices and algorithms used in automated text writing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, lets assume that I have access to wikipedia/wikinews and similar websites API and I would like to produce article about &quot;Data Science with Python&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that this task should be divided into two segments. First would be &lt;strong&gt;text mining&lt;/strong&gt; and second would be &lt;strong&gt;text building&lt;/strong&gt;. I'm more or less aware how text mining is performed and there are lots of materials about it in Internet. However, amount of materials related to automated text building seems to be lower. There are plenty of articles which says that some companies are using it, but there is lack of details. Are there any common ideas about such text building?&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2015-05-05T07:47:45.473" Title="How to approach automated text writing?" Tags="|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5688" PostTypeId="2" ParentId="5687" CreationDate="2015-05-05T07:47:45.473" Score="6" Body="&lt;p&gt;You should probably do some reading in the field of &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Natural_language_generation&quot;&gt;Natural Language Generation&lt;/a&gt;&quot;, since this seems to relate most directly to your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the way you have described the process -- &quot;text mining...text building&quot; -- leads me to wonder if you are aiming for something much more ambitious.  It seems as though you aim to automate the process of 1) reading natural language texts, 2) &lt;strong&gt;&lt;em&gt;understanding the meaning&lt;/em&gt;&lt;/strong&gt;, and then 3) generate new texts based on that semantic knowledge.  I'm not aware of any general-purpose end-to-end systems that can do that, not even specialized systems by the likes of &lt;a href=&quot;http://www.palantir.com/&quot;&gt;Palantir&lt;/a&gt;.  What you are aiming for would probably pass the Turing Test for fully capable Artificial Intelligence.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-05-05T07:47:45.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5689" PostTypeId="1" CreationDate="2015-05-05T08:14:29.573" Score="0" ViewCount="1792" Body="&lt;p&gt;I've used &lt;code&gt;scikit-learn&lt;/code&gt; in Python to compare results of naive Bayes and SVM. I've found that naive Bayes is quicker than SVM. Could anyone shed some light on reasons for such finding?&lt;/p&gt;&#xA;" OwnerUserId="9436" LastEditorUserId="8820" LastEditDate="2016-01-27T10:17:10.730" LastActivityDate="2016-01-27T10:17:10.730" Title="Why is the Naive Bayes classifier of sklearn faster than sklearns SVM?" Tags="|machine-learning|svm|scikit-learn|performance|naive-bayes-classifier|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5690" PostTypeId="1" CreationDate="2015-05-05T10:10:19.463" Score="1" ViewCount="64" Body="&lt;p&gt;I'd like to classify the data on coordinate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are 2 example data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data1 = [(1,1), (2,2), &lt;strong&gt;(3, 3), (4, 2), (5, 3)&lt;/strong&gt;, (6, 0)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data2 = [(1,1), (2,2), &lt;strong&gt;(3, 10), (4, 9), (5, 10)&lt;/strong&gt;, (6, 0)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bold part have the same wave in above data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The length are all different in my data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way that I can find the similar wave in many data like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="9457" LastActivityDate="2015-05-05T10:46:09.693" Title="Classify the coordinate data" Tags="|data-mining|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5691" PostTypeId="2" ParentId="5690" CreationDate="2015-05-05T10:32:10.873" Score="1" Body="&lt;p&gt;Calcualte distance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distance = data2 - data1 = [(1,&lt;strong&gt;0&lt;/strong&gt;), (2,&lt;strong&gt;0&lt;/strong&gt;), &lt;em&gt;(3, &lt;strong&gt;7&lt;/strong&gt;), (4, &lt;strong&gt;7&lt;/strong&gt;), (5, &lt;strong&gt;7&lt;/strong&gt;)&lt;/em&gt;, (6, &lt;strong&gt;0&lt;/strong&gt;)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what will you treat as &lt;em&gt;the same wave&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any three (or more) nearby points that have the same distance... &lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-05-05T10:46:09.693" LastActivityDate="2015-05-05T10:46:09.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5693" PostTypeId="1" CreationDate="2015-05-05T14:37:25.377" Score="4" ViewCount="762" Body="&lt;p&gt;First of all, I am new in this field we call &lt;em&gt;big data&lt;/em&gt;, so my questions may be naive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to build an application, which deals with geolocation data, which could be : &lt;em&gt;latitude and longitude coordinates&lt;/em&gt; and &lt;a href=&quot;https://msdn.microsoft.com/en-us/library/cc280766.aspx&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Geography SQL Server&lt;/em&gt;&lt;/a&gt; column types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to have the following elements made easy:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scalability : be prepared to receive huge amount of data, adding servers to the system have to be easy&lt;/li&gt;&#xA;&lt;li&gt;Proximity requests : in example, how much points are in a circle (at meter scale). &lt;/li&gt;&#xA;&lt;li&gt;Data must be accessible rapidly after being written. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've been looking around for existing solutions, which are &quot;Hadoop friendly&quot; (Hortonworks, Cloudera) and available DBMS, like Cassandra. &#xA;I have found some interesting information, but I still think it's hard to decide, which one to choose. &#xA;It also need drivers for &lt;em&gt;NodeJS&lt;/em&gt; &amp;amp; &lt;em&gt;.NET&lt;/em&gt; (Hadoop with Cassandra seem to be OK with that). &#xA;I've also looked around the MongoDB ecosystem, but, again, I feel that it is hard to know where to look at. By (little) experience with Mongoose, MongoDB can be disqualified by the third point because data writes are slow. But my model could certainly be improved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do any of you have some recent experiences, manipulating massive amount of &lt;em&gt;geolocation data&lt;/em&gt;? I would appreciate sharing them here as well as any quality and recent literature on the subject.&lt;/p&gt;&#xA;" OwnerUserId="9461" LastEditorUserId="9461" LastEditDate="2015-05-26T09:29:22.297" LastActivityDate="2015-05-26T11:26:14.063" Title="NoSQL engine/service recommendation for geolocation data" Tags="|bigdata|clustering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5694" PostTypeId="1" AcceptedAnswerId="5698" CreationDate="2015-05-05T17:48:35.433" Score="23" ViewCount="16814" Body="&lt;p&gt;A commonly heard sentence in unsupervised Machine learning is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;High dimensional inputs typically live on or near a low dimensional&#xA;  manifold&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is a dimension? What is a manifold? What is the difference?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can you give an example to describe both?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Manifold&lt;/strong&gt; from Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In mathematics, a manifold is a topological space that resembles&#xA;  Euclidean space near each point. More precisely, each point of an&#xA;  n-dimensional manifold has a neighbourhood that is homeomorphic to the&#xA;  Euclidean space of dimension n.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dimension&lt;/strong&gt; from Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In physics and mathematics, the dimension of a mathematical space (or&#xA;  object) is informally defined as the minimum number of coordinates&#xA;  needed to specify any point within it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does the Wikipedia even mean in layman terms? It sounds like some bizarre definition like most machine learning definition?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;They are both spaces, so what's the difference between a Euclidean space (i.e. Manifold) and a dimension space (i.e. feature-based)?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="57946" LastEditDate="2020-04-21T18:51:02.880" LastActivityDate="2022-09-27T06:48:12.030" Title="Dimensionality and Manifold" Tags="|machine-learning|dimensionality-reduction|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="5695" PostTypeId="1" CreationDate="2015-05-06T10:26:56.647" Score="1" ViewCount="4378" Body="&lt;p&gt;Could you tell me, are there any techniques for building neural networks with non-negative weights?&lt;/p&gt;&#xA;" OwnerUserId="9215" LastActivityDate="2015-05-06T13:33:46.497" Title="Neural networks with non-negative weights" Tags="|neural-network|" AnswerCount="2" CommentCount="1" ClosedDate="2015-05-11T20:44:21.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="5696" PostTypeId="2" ParentId="5689" CreationDate="2015-05-06T11:48:06.327" Score="6" Body="&lt;p&gt;The differences in speed between Naive Bayes and SVM simply boils down to the formulation and the assumptions of each model, and has little to do with the particular library or implementation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not only is naive bayes a simple probabilistic classifier, it also makes an additional assumption of independence between its features, so that parameter estimates can be calculated independently and thus possibly very quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In comparison, with SVM, every single record in the data base will require a computation of the distance function to determine the optimal decision boundary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can observe that for SVM, we have a complexity of $O(\text{#features} \times \text{#observations}^2)$ or more depending on the choice of SVM and kernel. On the other hand, for naive bayes we would have a complexity of $O(\text{#features} \times \text{#observations})$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence in general, we can say that SVM will take longer than Naive Bayes. &lt;/p&gt;&#xA;" OwnerUserId="9472" LastEditorUserId="8820" LastEditDate="2016-01-26T21:41:14.130" LastActivityDate="2016-01-26T21:41:14.130" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5697" PostTypeId="2" ParentId="5695" CreationDate="2015-05-06T12:25:24.843" Score="1" Body="&lt;p&gt;One possible method to approach neural networks with non-negative weights is using &lt;a href=&quot;http://en.wikipedia.org/wiki/Feedforward_neural_network&quot; rel=&quot;nofollow&quot;&gt;Feedforward neural network&lt;/a&gt;. We can construct the neural network using optimization techniques with the constraint of non-negative weights, as opposed to the normal back propagation method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A modified &lt;a href=&quot;https://github.com/pushkar/ml/tree/master/nn/hill&quot; rel=&quot;nofollow&quot;&gt;Matlab example taken for here is as follows&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;load iris_dataset&#xA;&#xA;% Number of neurons&#xA;n = 4;&#xA;&#xA;% Number of attributes and number of classifications&#xA;[n_attr, ~]  = size(irisInputs);&#xA;[n_class, ~] = size(irisTargets);&#xA;&#xA;% Initialize neural network&#xA;net = feedforwardnet(n);&#xA;&#xA;% Configure the neural network for this dataset&#xA;net = configure(net, irisInputs, irisTargets); %view(net);&#xA;&#xA;fun = @(w) mse_test(w, net, irisInputs, irisTargets);&#xA;&#xA;% Add 'Display' option to display result of iterations&#xA;ps_opts = psoptimset ( 'CompletePoll', 'off', 'Display', 'iter', 'MaxIter', 100); %, 'TimeLimit', 120 );&#xA;&#xA;% There is n_attr attributes in dataset, and there are n neurons so there &#xA;% are total of n_attr*n input weights (uniform weight)&#xA;initial_il_weights = ones(1, n_attr*n)/(n_attr*n);&#xA;% There are n bias values, one for each neuron (random)&#xA;initial_il_bias    = rand(1, n);&#xA;% There is n_class output, so there are total of n_class*n output weights &#xA;% (uniform weight)&#xA;initial_ol_weights = ones(1, n_class*n)/(n_class*n);&#xA;% There are n_class bias values, one for each output neuron (random)&#xA;initial_ol_bias    = rand(1, n_class);&#xA;% starting values&#xA;starting_values = [initial_il_weights, initial_il_bias, ...&#xA;                   initial_ol_weights, initial_ol_bias];&#xA;&#xA;% alter the patternsearch function with the appropriate constraints, in your case we would change it so that the lower bounds of the weights are zero &#xA;[x, fval, flag, output] = patternsearch(fun, starting_values, [], [],[],[], zeros(size(starting_values)), 1e10, ps_opts); &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the &lt;code&gt;mse_test.m&lt;/code&gt; function is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function mse_calc = mse_test(x, net, inputs, targets)&#xA;net = setwb(net, x');&#xA;y = net(inputs);&#xA;[row col] = size(y);&#xA;mse_calc = sum(sum((y - targets).^2))/(row * col);&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9472" LastActivityDate="2015-05-06T12:25:24.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5698" PostTypeId="2" ParentId="5694" CreationDate="2015-05-06T12:56:09.333" Score="36" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What is a dimension?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To put it simply, if you have a tabular data set with m rows and n columns, then the dimensionality of your data is n:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is a manifold? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The simplest example is our planet Earth. For us it looks flat, but it really is a sphere. So it's sort of a 2d manifold embedded in the 3d space. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the difference? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To answer this question, consider another example of a manifold: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/FrBXu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is so-called &quot;swiss roll&quot;. The data points are in 3d, but they all lie on 2d manifold, so the dimensionality of the manifold is 2, while the dimensionality of the input space is 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many techniques to &quot;unwrap&quot; these manifolds. One of them is called &lt;a href=&quot;https://www.google.com/search?q=locally+linear+embedding&quot; rel=&quot;noreferrer&quot;&gt;Locally Linear Embedding&lt;/a&gt;, and this is how it would do that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/pagFb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a scikit-learn snippet for doing that:&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-python prettyprint-override&quot;&gt;&lt;code&gt;from sklearn.manifold import LocallyLinearEmbedding&#xA;&#xA;lle = LocallyLinearEmbedding(n_neighbors=k, n_components=2)&#xA;X_lle = lle.fit_transform(data)&#xA;plt.scatter(X_lle[:, 0], X_lle[:, 1], c=color)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="816" LastEditorUserId="33245" LastEditDate="2019-08-09T16:14:21.023" LastActivityDate="2019-08-09T16:14:21.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5699" PostTypeId="2" ParentId="5695" CreationDate="2015-05-06T13:33:46.497" Score="1" Body="&lt;p&gt;You can easily take an existing &lt;a href=&quot;http://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;nofollow&quot;&gt;Multilayer Perceptron&lt;/a&gt; implementation and modify the &lt;a href=&quot;http://en.wikipedia.org/wiki/Backpropagation&quot; rel=&quot;nofollow&quot;&gt;backpropagation&lt;/a&gt; algorithm to prevent weights from becoming negative. Of course, you would also need to make sure you initialize weights with only non-negative values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you need to be aware that you are limiting the decision surfaces that the neural network can learn. So you should consider whether a neural network with only non-negative weights can provide an acceptable solution for your given data set.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-05-06T13:33:46.497" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5700" PostTypeId="2" ParentId="5694" CreationDate="2015-05-06T18:27:39.693" Score="8" Body="&lt;p&gt;The dimensionality of a dataset is the number of variables used to represent it. For example, if we were interested in describing people in terms of their height and weight, our &quot;people&quot; dataset would have 2 dimensions. If instead we had a dataset of images, and each image is a million pixels, then the dimensionality of the dataset would be a million. In fact, in many modern machine learning applications, the dimensionality of a dataset could be massive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When dimensionality is very large (larger than the number of the samples in the dataset), we could run into some serious problems. Consider a simple classification algorithm that seeks to find a set of weights w such that when dotted with a sample x, gives a negative number for one class and a positive number for another. w will have a length equal to the dimensionality of the data, so it will have more parameters than there are samples in the entire dataset. This means that a learner will be able to overfit the data, and consequently won't generalize well to other samples unseen during training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A manifold is an object of dimensionality d that is embedded in some higher dimensional space. Imagine a set of points on a sheet of paper. If we crinkle up the paper, the points are now in 3 dimensions. Many manifold learning algorithms seek to &quot;uncrinkle&quot; the sheet of paper to put the data back into 2 dimensions. Even if we aren't concerned with overfitting our model, a non-linear manifold learner can produce a space that makes classification and regression problems easier.&lt;/p&gt;&#xA;" OwnerUserId="9483" LastActivityDate="2015-05-06T18:27:39.693" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5701" PostTypeId="1" AcceptedAnswerId="5702" CreationDate="2015-05-06T21:52:55.810" Score="7" ViewCount="768" Body="&lt;p&gt;One of the problems I often encounter is that of poor data provenance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've recently stumbled upon &lt;a href=&quot;https://pythonhosted.org/Sumatra&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sumatra&lt;/a&gt;, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2019: The Sumatra project seems inactive and other, more mature project have come along. &lt;a href=&quot;https://dvc.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;DVC&lt;/a&gt; looks very promising and I've been in contact some of its authors, who have proven very helpful and supportive.&lt;/p&gt;&#xA;" OwnerUserId="9486" LastEditorUserId="9486" LastEditDate="2019-12-03T12:24:48.370" LastActivityDate="2019-12-03T12:24:48.370" Title="Lightweight data provenance tool" Tags="|tools|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5702" PostTypeId="2" ParentId="5701" CreationDate="2015-05-06T23:01:53.847" Score="4" Body="&lt;p&gt;Yes, you should save result files before you make major mods to the code.  Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific.  I would suggest storing old results sets with folder names that include a time stamp of when they were generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-06T23:01:53.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5703" PostTypeId="1" CreationDate="2015-05-07T00:16:08.680" Score="13" ViewCount="3642" Body="&lt;p&gt;Is the Semantic Web dead? Are ontologies dead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am developing a work plan for my thesis about &lt;em&gt;&quot;A knowledge base through a set ontology for interest groups around wetlands&quot;&lt;/em&gt;. I have been researching and developing ontologies for it but I am still unclear about many things. What is the modeling language for ontologies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which methodology for ontologies is better? &lt;a href=&quot;http://semanticweb.org/wiki/OTK_methodology&quot; rel=&quot;noreferrer&quot;&gt;OTK&lt;/a&gt; or &lt;a href=&quot;http://semanticweb.org/wiki/METHONTOLOGY&quot; rel=&quot;noreferrer&quot;&gt;METHONTOLOGY&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any program that does  as does &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.hd.uib.no/AcoHum/abs/Mejia.htm&quot; rel=&quot;noreferrer&quot;&gt;Cratilo&lt;/a&gt; is a software for analyzing of textual corpora and for extraction of specific terms of the domain of study (it is developed by professors Jorge Antonio Mejia, Francisco Javier Alvarez and John Albeiro Sánchez, Institute of Philosophy the University of Antioquia). It enables lexical analysis of texts, identifying the words that appear their frequency and location in the text. Through a process of recognition, Cratylus identifies all the words in the text and builds a database becomes the draft analysis of the work. Are there other similar tools?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the terms found by Cratilo be used to create a knowledge base?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the existing open semantic frameworks that can be used for such things? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there software that automatically creates RDF, OWL, and XML? How does Tails work? Jena? Sesame? &lt;/p&gt;&#xA;" OwnerUserId="9488" LastEditorUserId="9508" LastEditDate="2017-07-04T12:10:32.253" LastActivityDate="2017-07-04T12:10:32.253" Title="Are ontologies and the Semantic Web dead?" Tags="|knowledge-base|" AnswerCount="3" CommentCount="2" ClosedDate="2017-05-07T20:41:06.980" ContentLicense="CC BY-SA 3.0" />
  <row Id="5704" PostTypeId="2" ParentId="5694" CreationDate="2015-05-07T03:52:08.877" Score="1" Body="&lt;p&gt;One way of doing dimensional reduction is to do feature hashing.  This was known about in the 1960's.  So for example if your data is a sparse set of points in 3 dimensions (x,y,z) you create a (good) hash function h(x,y,z).  You can use that of course for a hash table or a Bloom filter lookup. This is a good form of data compression.  I don't know why the AI community doesn't use it.  It is much more to the point than a neural net. &lt;/p&gt;&#xA;" OwnerUserId="9492" LastActivityDate="2015-05-07T03:52:08.877" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5705" PostTypeId="2" ParentId="2258" CreationDate="2015-05-07T04:05:13.373" Score="0" Body="&lt;p&gt;For sure you need to learn some maths.  However you should also make an effort to gain some broader engineering and science skills.  There are far too many people going into computer science and all they know is a few programming languages and math. The end result is a very boring person with little in the way of creativity to do anything new. Take a year out when you are 18 or 19 to travel the world.   &lt;/p&gt;&#xA;" OwnerUserId="9492" LastActivityDate="2015-05-07T04:05:13.373" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5706" PostTypeId="1" AcceptedAnswerId="5734" CreationDate="2015-05-07T04:11:56.600" Score="198" ViewCount="151500" Body="&lt;p&gt;Referring to the Stanford course notes on &lt;a href=&quot;http://cs231n.github.io/neural-networks-1/#actfun&quot; rel=&quot;noreferrer&quot;&gt;Convolutional Neural Networks for Visual Recognition&lt;/a&gt;, a paragraph says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Unfortunately, ReLU units can be fragile during training and can&#xA;  &quot;die&quot;. For example, a large gradient flowing through a ReLU neuron&#xA;  could cause the weights to update in such a way that the neuron will&#xA;  never activate on any datapoint again. If this happens, then the&#xA;  gradient flowing through the unit will forever be zero from that point&#xA;  on. That is, the ReLU units can irreversibly die during training since&#xA;  they can get knocked off the data manifold. For example, you may find&#xA;  that as much as 40% of your network can be &quot;dead&quot; (i.e. neurons that&#xA;  never activate across the entire training dataset) if the learning&#xA;  rate is set too high. With a proper setting of the learning rate this&#xA;  is less frequently an issue.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does dying of neurons here mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please provide an intuitive explanation in simpler terms.&lt;/p&gt;&#xA;" OwnerUserId="793" LastEditorUserId="43000" LastEditDate="2017-12-27T04:38:16.587" LastActivityDate="2018-09-22T10:46:25.477" Title="What is the &quot;dying ReLU&quot; problem in neural networks?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5707" PostTypeId="1" CreationDate="2015-05-07T06:22:51.470" Score="2" ViewCount="262" Body="&lt;p&gt;I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, hypothetically, a simple average calculation of n (=1 billion) numbers is O(n) + C operation using simple for loop, or O(log) I am assuming division to be a constant time operation for the sake for simplicity. If i break this massively parallelizable algorithm for MapReduce, by dividing data over k nodes, my time complexity would simply become O(n/k) + C + C'. Here, C' can be assumed as the job planning time overhead. Note that there was no shuffling involved, and reducer's job was nearly trivial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in a more complete analysis of algorithm with iterative loops over data and involve heavy shuffling and reducer operations. I want to incorporate, if possible, the I/O operations and network transfers of data.&lt;/p&gt;&#xA;" OwnerUserId="9495" LastActivityDate="2024-02-06T03:09:11.443" Title="Time Complexity notation in Big Data platforms" Tags="|bigdata|algorithms|map-reduce|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5708" PostTypeId="1" CreationDate="2015-05-07T08:52:40.293" Score="8" ViewCount="2797" Body="&lt;p&gt;I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category.&#xA;I want to classify that into 300 categories. Some categories are independent while some are somewhat related. Distribution of data among categories is not uniform either i.e some of the categories(around 40 of them) have less data about 2-3 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attaching log probablity of each class/categories. (OR distribution of classes) here.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/nko61.png&quot; alt=&quot;Class prior logarithm of probabilities (log class distribution of data)&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="9497" LastEditorUserId="9497" LastEditDate="2015-05-18T19:55:40.893" LastActivityDate="2015-05-18T19:55:40.893" Title="Which classification algorithms to try for classifying text data into 300 categories" Tags="|machine-learning|classification|nlp|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5709" PostTypeId="2" ParentId="5663" CreationDate="2015-05-07T09:09:39.333" Score="2" Body="&lt;p&gt;I think Social Media Mining: An Introduction by Zafarani et. al. is an excellent starting point. You can find more about it &lt;a href=&quot;http://dmml.asu.edu/smm/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Also a free PDF version is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It first goes through the essentials in graph theory and data mining. It covers some more advanced topics in graph mining, social network analysis, recommendation systems, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, I have seen some online courses in coursera (&lt;a href=&quot;https://www.coursera.org/course/sna&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;). I am not sure about their quality though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, note that social network analysis is data mining for the social media data like Facebook. It is not social science at all; it is computer science. While you may end up borrowing some ideas from them, what you will end up doing is far from what social science guys are doing. So, going through social science courses and books is likely not a good idea at this point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. The book is the text book for the social media mining course offered in my university.&lt;/p&gt;&#xA;" OwnerUserId="9496" LastEditorUserId="9496" LastEditDate="2015-05-07T09:25:34.973" LastActivityDate="2015-05-07T09:25:34.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5710" PostTypeId="1" AcceptedAnswerId="5712" CreationDate="2015-05-07T13:23:47.440" Score="7" ViewCount="3660" Body="&lt;p&gt;I've compared the logistic regression models on R (&lt;code&gt;glm&lt;/code&gt;) and on Spark (&lt;code&gt;LogisticRegressionWithLBFGS&lt;/code&gt;) on a dataset of 390 obs. of 14 variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results are completely different in the intercept and the weights.&#xA;How to explain this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the results of Spark (LogisticRegressionWithLBFGS) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.intercept  : &#xA; 1.119830027739959&#xA;model.weights :&#xA; GEST    0.30798496002530473&#xA; DILATE  0.28121771009716895&#xA; EFFACE  0.01780105068588628&#xA; CONSIS -0.22782058111362183&#xA; CONTR  -0.8094592237248102&#xA; MEMBRAN-1.788173534959893&#xA; AGE    -0.05285751197750732&#xA; STRAT  -1.6650305527536942&#xA; GRAVID  0.38324952943210994&#xA; PARIT  -0.9463956993328745&#xA; DIAB   0.18151162744507293&#xA; TRANSF -0.7413500749909346&#xA; GEMEL  1.5953124037323745&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the result of R :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             Estimate Std. Error z value Pr(&amp;gt;|z|)   &#xA;(Intercept)  3.0682091  3.3944407   0.904 0.366052    &#xA;GEST         0.0086545  0.1494487   0.058 0.953821    &#xA;DILATE       0.4898586  0.2049361   2.390 0.016835 *  &#xA;EFFACE       0.0131834  0.0059331   2.222 0.026283 *  &#xA;CONSIS       0.1598426  0.2332670   0.685 0.493196    &#xA;CONTR        0.0008504  0.5788959   0.001 0.998828    &#xA;MEMBRAN     -1.5497870  0.4215416  -3.676 0.000236 ***   &#xA;AGE         -0.0420145  0.0326184  -1.288 0.197725    &#xA;STRAT       -0.3781365  0.5860476  -0.645 0.518777    &#xA;GRAVID       0.1866430  0.1522925   1.226 0.220366    &#xA;PARIT       -0.6493312  0.2357530  -2.754 0.005882 **  &#xA;DIAB         0.0335458  0.2163165   0.155 0.876760    &#xA;TRANSF      -0.6239330  0.3396592  -1.837 0.066219 .  &#xA;GEMEL        2.2767331  1.0995245   2.071 0.038391 *  &#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9445" LastEditorUserId="182" LastEditDate="2015-05-11T20:49:51.223" LastActivityDate="2016-04-23T17:47:52.937" Title="Why does logistic regression in Spark and R return different models for the same data?" Tags="|machine-learning|r|logistic-regression|apache-spark|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5712" PostTypeId="2" ParentId="5710" CreationDate="2015-05-07T15:34:57.470" Score="6" Body="&lt;p&gt;A quick glance at the &lt;a href=&quot;https://spark.apache.org/docs/1.2.1/api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS&quot;&gt;docs for LogisticRegressionWithLBFGS&lt;/a&gt; indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's &lt;code&gt;glm&lt;/code&gt; is returning a maximum likelihood estimate of the model while Spark's &lt;code&gt;LogisticRegressionWithLBFGS&lt;/code&gt; is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all smaller in magnitude than those in the R model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure whether or not &lt;code&gt;glm&lt;/code&gt; in R is implementing feature scaling, but this would also contribute to different model values.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-05-07T15:34:57.470" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5713" PostTypeId="2" ParentId="5703" CreationDate="2015-05-07T17:31:42.180" Score="2" Body="&lt;p&gt;There are too many too general problems in your post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're definitely in an AI summer era right now (as opposed to &lt;a href=&quot;http://en.wikipedia.org/wiki/AI_winter&quot; rel=&quot;nofollow&quot;&gt;AI winter&lt;/a&gt;), and the research on Semantic Web receives less attention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, there are many projects related to building ontologies. Google has &lt;a href=&quot;http://en.wikipedia.org/wiki/Knowledge_Graph&quot; rel=&quot;nofollow&quot;&gt;Knowledge Graph&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Knowledge_Vault&quot; rel=&quot;nofollow&quot;&gt;Knowledge Vault&lt;/a&gt;. Both of these are using &lt;a href=&quot;https://www.freebase.com/&quot; rel=&quot;nofollow&quot;&gt;Freebase&lt;/a&gt; (among other sources).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are dozens of links I can give you to answer some of your questions, but the best thing you can do is to browse &lt;a href=&quot;https://www.w3.org/2001/sw/wiki/Main_Page&quot; rel=&quot;nofollow&quot;&gt;W3C Semantic Web&lt;/a&gt; pages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at RDF, SPARQL, OWL, Virtuoso, Protege - these are de facto standards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of extracting ontologies from textual corpora - there are various tools out there. Neither of them is perfect, so you really have to do some research and find something that suits your needs. For example, there's the &lt;a href=&quot;http://wwwusers.di.uniroma1.it/~navigli/pubs/CL_2013_Velardi_Faralli_Navigli.pdf&quot; rel=&quot;nofollow&quot;&gt;OntoLearn Reloaded&lt;/a&gt; (this paper is relatively new, so you can check out the bibliography to seek out for other approaches).&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-05-07T17:31:42.180" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5715" PostTypeId="1" CreationDate="2015-05-07T20:08:05.440" Score="4" ViewCount="1170" Body="&lt;p&gt;Beyond the immediate suspects defined in the &lt;a href=&quot;https://spark.apache.org/docs/latest/tuning.html#memory-tuning&quot; rel=&quot;nofollow&quot;&gt;spark documentation&lt;/a&gt;, what are some ways to profile, tune and boost performance of an Apache Spark application? &lt;/p&gt;&#xA;" OwnerUserId="9510" LastActivityDate="2015-05-07T20:08:05.440" Title="Performance profiling and tuning in Apache Spark" Tags="|apache-spark|" AnswerCount="0" CommentCount="3" ClosedDate="2016-02-17T20:18:13.250" ContentLicense="CC BY-SA 3.0" />
  <row Id="5716" PostTypeId="2" ParentId="5703" CreationDate="2015-05-07T20:36:54.033" Score="19" Body="&lt;p&gt;&lt;em&gt;(While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Is the Semantic Web dead? Are ontologies dead?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Many people moved away from the Semantic Web. On the other hand, there are still many using it. There's always been a lot of confusion about where its value lies. There are use cases that genuinely benefit from the semantic web technologies but they are quite few and far between.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Linked Open Data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;That's partially why the subfield of Linked Open Data was created. You can see Linked Data as a more pragmatic stripped down version of the Semantic Web (or as a necessary enabler of the grander semantic web vision).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linked Data, while more pragmatic, still uses ontologies. It is just not so uptight about using OWL and designing your ontologies first and in a very formal way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A knowledge base such as Freebase (now being replaced by &lt;a href=&quot;http://www.wikidata.org/wiki/Wikidata:Main_Page&quot; rel=&quot;noreferrer&quot;&gt;Wikidata&lt;/a&gt;) doesn't even build on Semantic Web technologies (it's always been available also in the form of RDF dumps though). On the other hand, even Freebase builds on concepts similar to those in the Semantic Web and it does have a form of ontologies.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Software to generate ontologies&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There is no software that would create high-quality ontologies automatically for you. At least not in the sense of OWL/first-order logic ontologies. On the other hand, many kinds of software, including Cratilo mentioned by you, can help you to build a lower step in the &lt;a href=&quot;http://en.wikipedia.org/wiki/Semantic_spectrum&quot; rel=&quot;noreferrer&quot;&gt;semantic spectrum&lt;/a&gt; such as a glossary or maybe even a folksonomy which can later be transformed into more of a taxonomy or ontology by other algorithms. There might be even software to create fullblown OWL ontologies but I think that's still rather an active area of research.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Ontologies and methodologies&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As always, it depends on what exactly your needs and goals are. In fields such as biomedicine and life sciences in general, they create complex formal ontologies because they have data with highly varied structure and ontologies help them keep things organized and usable. In contrast, an eshop might be better off sticking with normal relational database modelling and adding only global identifiers (URIs) in the spirit of Linked Data if they want to build a proper knowledge graph later. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you need ontologies you might want to skip the methodologies first. Especially if you are really new to the field and have little idea what your needs are.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Jena, Sesame, ...&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Jena, Sesame, Virtuoso, etc. are &lt;a href=&quot;http://en.wikipedia.org/wiki/Triplestore&quot; rel=&quot;noreferrer&quot;&gt;triplestores&lt;/a&gt; - they are used to store and query RDF. Most ontologies can be represented in RDF. Even those written in OWL. RDF has the semantic part, RDF/S, which allows you to formulate some basic ontologies. That might well be enough for start. Even RDF/S can get confusing and convoluted when you start to think about blank nodes, named graphs (are they fixed? are they dynamic?), etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.&lt;/p&gt;&#xA;" OwnerUserId="9508" LastActivityDate="2015-05-07T20:36:54.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5717" PostTypeId="1" CreationDate="2015-05-07T21:41:25.300" Score="5" ViewCount="5455" Body="&lt;p&gt;RBF kernel using SVM depends on two parameters C and gamma. If the equation of the kernel RBF as the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K(X,X')= \exp(\gamma||X-X'||^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the equation I can see where can I use gamma, but I can't find the C parameter.&#xA;So, can enybody tell me please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&lt;/p&gt;&#xA;" OwnerUserId="9514" LastEditorUserId="182" LastEditDate="2015-05-11T20:50:28.820" LastActivityDate="2015-05-11T20:50:28.820" Title="Where is the cost parameter C in the RBF kernel in SVM?" Tags="|machine-learning|classification|svm|search|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5718" PostTypeId="1" AcceptedAnswerId="5738" CreationDate="2015-05-07T21:53:32.540" Score="5" ViewCount="4422" Body="&lt;p&gt;I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to construct such a system. However, it is not clear how one would go about adding custom labels (e.g. actor, director, movie title).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;noreferrer&quot;&gt;Chapter 7 of the NLTK Book&lt;/a&gt; discusses information extraction using a named entity recognizer, but it glosses over labeling details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I add custom labels?&#xA;If I have bare lists of relevant named entities (e.g. movies, actors, etc.), how can I include them as features? It appears that I would need to use IOB format, but I am unsure about how to do this when I only have lists of named entities.&lt;/p&gt;&#xA;" OwnerUserId="9511" LastActivityDate="2020-08-03T00:59:29.263" Title="Add Custom Labels to NLTK Information Extractor" Tags="|python|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5720" PostTypeId="2" ParentId="5717" CreationDate="2015-05-08T00:32:02.250" Score="5" Body="&lt;p&gt;The C parameter in SVMs doesn't have to do anything with the kernel function. C is the penalty associated to the instances which are either misclassified or violates the maximal margin. As you may know already, SVM returns the maximum margin for the linearly separable datasets (in the kernel space).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be the case that the dataset is not linearly separable. In this case the corresponding SVM quadratic program is unsolvable. To make it solvable, we redefine the quadratic program to return the maximum margin with respect to some error cost C for the points violating the maximum margin (misclassified samples also violate the maximum margin). Below you can see a simple example I found on the Internet. This uses linear kernel but the story is the same for other non-linear kernels as well.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/SIath.jpg&quot; alt=&quot;Non-Linear dataset&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect C has on the classification result is shown in the picture below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/8va0T.png&quot; alt=&quot;C effect&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, if you set the error penalty to be very high, you will end up with less misclassifications and fewer support vectors (the highlighted points are SVs). However, if you set it too high when you are using non-linear kernels (specially Gaussian) you might overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in the theory behind it, here is what we are trying to optimize:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/DmCSV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we write its dual, we'll get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/e9yqg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See how convenient it is! No C in the kernel function!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, you are mapping the samples to a higher dimensional space using the kernel function, then solving it using some penalty term C.&lt;/p&gt;&#xA;" OwnerUserId="9496" LastEditorUserId="9496" LastEditDate="2015-05-08T00:45:28.993" LastActivityDate="2015-05-08T00:45:28.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5722" PostTypeId="2" ParentId="5574" CreationDate="2015-05-08T03:41:54.230" Score="3" Body="&lt;p&gt;This question isn't terribly clear.  Data analysis and strategic modeling (game theory) are different tasks.  Nash equilibrium is a way of understanding the incentives they have by assuming a set of players with assumed utility function and making &lt;strong&gt;deductive&lt;/strong&gt; inferences about what they ought to do to maximize those utility functions given their interaction.  Data analysis is an &lt;strong&gt;inductive&lt;/strong&gt; process.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of ways game theory and data analysis might interact, here are the easy top two: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Someone might use data to infer players' utility functions (I'm sure this exists in econometrics-land somewhere; also, political scientists have a technique called &quot;ideal point estimation,&quot; to infer political preferences from voting behavior---which you can easily google to learn more);&lt;/li&gt;&#xA;&lt;li&gt;Someone might use game theory to generate behavioral predictions which are testable by data.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thinking about the specific kinds of cases you mention, the obvious application would be in the stock market one.  Suppose you have a ML model that can reliably predict the market behavior of other people at time T from a given feature set.  Then the consumer of the ML model might have an optimal purchase at T-1, and finding that optimal purchase is going to be strategic.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But combining the two approaches might just break the ML. This is really interesting to think about... musing out loud...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the simple case of a two-player market in one stock. Player 1 wants to buy at T-1 if player 2 will be buying at T (because the price will go up); player 1 wants to sell at T-1 if player 2 will sell at T (because the price will go down).  The naive approach for player 1 is &quot;use my ML model to predict what player 2 will do, then do it first at T-1.&quot; But, of course, P1's behavior at T-1 is itself observable by P2, and changes P2's behavior (the price has gone up); moreover, by definition P1's behavior at T-1 can't be a feature of the ML model used to predict P2's behavior at T, because it's behavior that is chosen on the basis of the ML prediction.  All sorts of fun puzzles begin here, but none of them look real good...&lt;/p&gt;&#xA;" OwnerUserId="9519" LastActivityDate="2015-05-08T03:41:54.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5723" PostTypeId="1" AcceptedAnswerId="5749" CreationDate="2015-05-08T03:47:00.857" Score="3" ViewCount="2070" Body="&lt;p&gt;This is just for finding overfitting gap.&lt;br&gt;&#xA;After initial research, I can only find method to draw learning curve using evaluation of test set.  However, I could not evaluate on training set and over the two learning curves.&lt;/p&gt;&#xA;" OwnerUserId="8524" LastEditorUserId="31513" LastEditDate="2017-04-28T09:18:54.287" LastActivityDate="2022-02-08T02:44:49.290" Title="In Weka, how to draw learning curve evaluated on both test and training set?" Tags="|machine-learning|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5724" PostTypeId="2" ParentId="2321" CreationDate="2015-05-08T04:27:17.680" Score="2" Body="&lt;p&gt;When class-conditional distributions are gaussian with equal covariance matrices, the optimal decision boundary is a hyperplane.  This is the core concept behind Linear Discriminant Analysis (LDA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any data point $x$, the probability that $x$ comes from class $\omega_1$ is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|\omega_1) \sim N(\mu_1,\Sigma) = (2\pi)^{-1}|\Sigma|^{-1/2}\exp\left\{ (-1/2) (x-\mu_1)'\Sigma^{-1}(x-\mu_1)&#xA;\right\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, $P(x|\omega_2) \sim N(\mu_2,\Sigma)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's denote the prior probability of class 1 and 2 as $P(\omega_1)$ and $P(\omega_2)$, respectively.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Equal Misclassification Cost:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If the cost of misclassification is equal, we want to assign new data points such that the probability of misclassification is minimized. This decision rule assigns points to class 1 when $x$ satisfies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|\omega_1)P(\omega_1) &amp;gt; P(\omega_2)(P(x|\omega_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can establish a similar rule for assigning points to class 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;o find the decision boundary, we need to find the values of $x$ that satisfy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|\omega_1)P(\omega_1) = P(x|\omega_2)P(\omega_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values of $x$ that satisfy this equality here lie on a line (or hyperplane for higher dimensions).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Unequal Misclassification Cost:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For the second question, you will need an additional term.  Let's call $C(\omega_1)$ the cost of making an error when the data point was actually from class 1 and $C(\omega_2)$ the cost of making an error when the data point was from class 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To minimize the cost of an error, your new decision rule would be to assign $x$ to class 1 when $x$ satisfies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|\omega_1)P(\omega_1)C(\omega_1) &amp;gt; P(x|\omega_2)P(\omega_2)C(\omega_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As before, to find the decision boundary, you need to solve for $x$ when:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|\omega_1)P(\omega_1)C(\omega_1) = P(x|\omega_2)P(\omega_2)C(\omega_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This decision boundary will still be a line / hyperplane, however it may have a different offset or orientation from the solution in the case of equal misclassification cost.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-05-08T04:27:17.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5725" PostTypeId="1" CreationDate="2015-05-08T05:27:14.263" Score="3" ViewCount="181" Body="&lt;p&gt;I am trying to work using Amazon machine learning, but the data set that I have is small. The model I want to build is for regression based predictions and the domain I am aiming for the data set to belong is financial, say product price prediction, price and demand prediction based on macro/micro economic factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a data set that contains factors that lead to variations in value of a product or commodity. For example, I would like to predict the value of 1 unit of polyester yarn after 1 year.  The factors which influence the yarn price are say - prices of crude oil, GDP of country,figures of IIP, inflation etc. So I would like a data set that contains the quotes of these factors on which the final price depends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find difficulty assembling this data myself because I don't know all the factors that contribute to a certain predictive price. Does anyone know of a dataset I can start with that sounds like it might contain these factors?&lt;/p&gt;&#xA;" OwnerUserId="9521" LastEditorUserId="9521" LastEditDate="2015-05-11T03:28:37.877" LastActivityDate="2017-12-17T17:08:06.353" Title="Datasource for regression model prediction : Machine Learning" Tags="|machine-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5726" PostTypeId="2" ParentId="5574" CreationDate="2015-05-08T06:57:00.863" Score="1" Body="&lt;p&gt;I have a very limited knowledge of game theory, but hope to learn more. However, I think that potential applications of Nash equilibrium in the context of big data environments, implies the need of analyzing a large number of &lt;strong&gt;features&lt;/strong&gt; (representing various strategic pathways or traits) as well as large number of &lt;strong&gt;cases&lt;/strong&gt; (representing significant number of actors). Considering these points, I would think that &lt;em&gt;complexity&lt;/em&gt; and, consequently, &lt;em&gt;performance requirements&lt;/em&gt; for Nash equilibrium in big data applications grow &lt;strong&gt;exponentially&lt;/strong&gt;. For some examples from the &lt;em&gt;Internet load-balancing&lt;/em&gt; domain, see paper by Even-Dar, Kesselman and Mansour (n.d.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above-mentioned points touch only the &lt;strong&gt;volume&lt;/strong&gt; aspect of &lt;a href=&quot;http://www.ibmbigdatahub.com/infographic/four-vs-big-data&quot; rel=&quot;nofollow&quot;&gt;4V big data model&lt;/a&gt; (an update of Gartner's original &lt;em&gt;3V model&lt;/em&gt;). If you add to that other aspects (&lt;em&gt;variety&lt;/em&gt;, &lt;em&gt;velocity&lt;/em&gt; and &lt;em&gt;veracity&lt;/em&gt;), the situation seems to become even more complex. Perhaps, people with &lt;em&gt;econometrics&lt;/em&gt; background and experience will have some of the most comprehensive opinions on this interesting question. A lot of such people are active on &lt;em&gt;Cross Validated&lt;/em&gt;, so I will let them know about this question - hopefully, some of them will be interested to share their view by answering this question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even-Dar, E., Kesselman, A., &amp;amp; Mansour, Y. (n.d.). &lt;em&gt;Convergence time to Nash equilibria.&lt;/em&gt; Retrieved from &lt;a href=&quot;http://www.tau.ac.il/~mansour/course_games/nash-load.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.tau.ac.il/~mansour/course_games/nash-load.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-05-08T06:57:00.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5729" PostTypeId="2" ParentId="5708" CreationDate="2015-05-08T17:32:38.147" Score="8" Body="&lt;p&gt;In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. &lt;a href=&quot;https://web.stanford.edu/class/cs124/lec/naivebayes.pdf&quot;&gt;Here are some slides describing NB as applied to natural language processing&lt;/a&gt;. There's nothing especially fancy about this approach, but it's pretty easy to implement and will give you a starting point to expand from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've found some initial results assuming independence among your features and your output labels, you'll probably have a better sense of where the model is weak.  From that point forward you can apply some feature engineering (maybe &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot;&gt;TF-IDF&lt;/a&gt;) as well as some post processing to deal with samples that get assigned to related categories.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-05-08T17:32:38.147" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5730" PostTypeId="1" CreationDate="2015-05-09T05:44:50.633" Score="3" ViewCount="461" Body="&lt;p&gt;I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as &lt;a href=&quot;http://en.wikipedia.org/wiki/Enron_Corpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://en.wikipedia.org/wiki/Enron_Corpus&lt;/a&gt;. After doing a bit of research, I also came across &lt;a href=&quot;https://datascience.stackexchange.com/questions/641/dataset-for-named-entity-recognition-on-informal-text/5397#5397&quot;&gt;Dataset for Named Entity Recognition on Informal Text&lt;/a&gt;. So I'm not short of data or a goal, I need to understand high-level techniques to get there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One valuable comment noted that this question appears too broad. What I was hoping to find with this question was the breadth of techniques I should focus research on, not answers that are immediately implementable. Please consider vague answers as entirely appropriate!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expanding on the goal, I am hoping to discover which authors might have affinity toward each other, or conversely do not care much for each other. So I will definitely need to start with Named Entity Recognition and build a means to organize the documents against those entities. Beyond that, I am not so sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What high level concepts should I be looking at? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="9534" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-06-05T05:10:10.423" Title="Finding user similarities within informal data sets" Tags="|machine-learning|classification|clustering|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5731" PostTypeId="2" ParentId="5725" CreationDate="2015-05-09T08:01:05.720" Score="6" Body="&lt;p&gt;There are &lt;a href=&quot;http://quant.caltech.edu/historical-stock-data.html&quot; rel=&quot;noreferrer&quot;&gt;multiple datasets&lt;/a&gt; available in the internet but you need to pay for most of them.&#xA;One of the best datasets for starting (and it is free) is the &lt;em&gt;Quantquote Free dataset&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can download it from &lt;a href=&quot;http://quantquote.com/files/quantquote_daily_sp500_83986.zip&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the description (borrowed from their &lt;a href=&quot;https://quantquote.com/historical-stock-data&quot; rel=&quot;noreferrer&quot;&gt;website&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This collection of daily resolution data goes back to 1998 for all symbols currently active in the S&amp;amp;P500. It is updated quarterly, the last update was 07/31/2013.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="278" LastActivityDate="2015-05-09T08:01:05.720" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5732" PostTypeId="1" AcceptedAnswerId="5733" CreationDate="2015-05-09T10:52:56.290" Score="0" ViewCount="22" Body="&lt;p&gt;What are some Python libraries which can convert a (X,Y) tuple to strings? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1.23,4.56) yields strings “1_4”, “12_45”, “123_456”.&lt;/p&gt;&#xA;" OwnerUserId="9544" LastEditorUserId="9544" LastEditDate="2015-05-09T14:26:48.140" LastActivityDate="2015-05-09T14:26:48.140" Title="Creating Strings corresponding to Location Co-ordinates" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-11T20:49:40.993" ContentLicense="CC BY-SA 3.0" />
  <row Id="5733" PostTypeId="2" ParentId="5732" CreationDate="2015-05-09T11:21:34.607" Score="1" Body="&lt;p&gt;Your question has nothing to do with NLP or text-mining (as you claim in the attached tags), or data science in general. It's a pure programming question best suitable for &lt;a href=&quot;http://stackoverflow.com&quot;&gt;StackOverflow&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, you don't really need any libraries to do, what you want to do. A simple function will do. NOTE: I am using &lt;code&gt;map&lt;/code&gt; and &lt;code&gt;reduce&lt;/code&gt; functions on purpose to include at least a little bit of data science-related stuff (WINK):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coord = (1.23, 4.56)&#xA;&#xA;def get_coord(coord):&#xA;   coord_str = map(lambda x: str(x).replace('.', ''), coord)&#xA;   for level in range(1, len(coord_str[0])+1):&#xA;      yield reduce(lambda x, y: (x[:level]+'_'+y[:level]), coord_str)&#xA;&#xA;print list(get_coord(coord))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running this code will result in printing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['1_4', '12_45', '123_456']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You're not discussing any corner cases in your question, so I assume there are none.&lt;/p&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-05-09T11:21:34.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5734" PostTypeId="2" ParentId="5706" CreationDate="2015-05-09T14:25:59.330" Score="211" Body="&lt;p&gt;A &quot;dead&quot; ReLU always outputs the same value (zero as it happens, but that is not important) for any input. Probably this is arrived at by learning a large negative bias term for its weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In turn, that means that it takes no role in discriminating between inputs. For classification, you could visualise this as a decision plane &lt;em&gt;outside&lt;/em&gt; of all possible input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once a ReLU ends up in this state, it is unlikely to recover, because the function gradient at 0 is also 0, so gradient descent learning will not alter the weights. &quot;Leaky&quot; ReLUs with a small positive gradient for negative inputs (&lt;code&gt;y=0.01x&lt;/code&gt; when x &amp;lt; 0 say) are one attempt to address this issue and give a chance to recover.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sigmoid and tanh neurons can suffer from similar problems as their values saturate, but there is always at least a small gradient allowing them to recover in the long term.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-05-09T14:25:59.330" CommentCount="18" ContentLicense="CC BY-SA 3.0" />
  <row Id="5735" PostTypeId="1" AcceptedAnswerId="5736" CreationDate="2015-05-09T21:57:21.073" Score="2" ViewCount="80" Body="&lt;p&gt;I am looking for a good free existing tool which visualizes geographical data (let's say in the form of coordinates) by plotting them on a map. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be a library (see &lt;a href=&quot;https://stackoverflow.com/questions/9018607/library-for-map-visualization&quot;&gt;this question on StackOverflow&lt;/a&gt;, which suggests a Python library called basemap, which is interesting but not dynamic enough, namely it does not allow for interactivity) or a complete toolkit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Existing things I found are oriented towards realizing web pages, which are not my ultimate goal (see &lt;a href=&quot;http://www.simile-widgets.org/exhibit/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Exhibit&lt;/a&gt; or &lt;a href=&quot;http://modestmaps.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modest Maps&lt;/a&gt;). I'd like something to feed with data which spits out an interactive map where you can click on places and it displays the related data. &lt;/p&gt;&#xA;" OwnerUserId="982" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-05-18T17:36:21.813" Title="Library/package/tool for geographical data visualizations" Tags="|visualization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5736" PostTypeId="2" ParentId="5735" CreationDate="2015-05-10T01:42:25.457" Score="4" Body="&lt;p&gt;Assuming you want your visualizations to be available on the internet, there are many options for this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://developers.google.com/maps/&quot; rel=&quot;nofollow&quot;&gt;Google Maps&lt;/a&gt; (and &lt;a href=&quot;http://tables.googlelabs.com/&quot; rel=&quot;nofollow&quot;&gt;Google Fusion Tables&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.mapbox.com/&quot; rel=&quot;nofollow&quot;&gt;Mapbox&lt;/a&gt; (and the related &lt;a href=&quot;http://leafletjs.com/&quot; rel=&quot;nofollow&quot;&gt;LeafletJS&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cartodb.com/&quot; rel=&quot;nofollow&quot;&gt;CartoDB&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bost.ocks.org/mike/map/&quot; rel=&quot;nofollow&quot;&gt;D3&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It really comes down to exactly what you want to do and how much control you want to have over the map.  Additionally you should consider what sort of interactivity you need.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google Maps is probably the most user friendly for very basic map functions but is somewhat limited in what you can do stylistically.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mapbox and CartoDB are both user friendly and offer good options for styling and displaying different varieties of data.  However, they also both tend to require subscription fees for anything other than small maps. Also, the last time I checked, CartoDB explicitly handles animation and time-series data where Mapbox does not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;D3 will probably give you the most control over display, animation, and interactivity but also has a long learning curve. Even if you don't want the map to be available on the internet, this is still a very good tool for making interactive visualizations that run in the browser.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't care as much about the visualizations being online, you can get a lot of work done in open GIS software like &lt;a href=&quot;http://www.qgis.org/en/site/&quot; rel=&quot;nofollow&quot;&gt;QGIS&lt;/a&gt; or &lt;a href=&quot;http://grass.osgeo.org/&quot; rel=&quot;nofollow&quot;&gt;GrassGIS&lt;/a&gt;, though I don't know if user interactivity is really an option there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said though, it really comes down to the specifics of exactly what you're trying to do and how comfortable you are with various aspects of mapping and coding.&lt;/p&gt;&#xA;" OwnerUserId="182" LastEditorUserId="182" LastEditDate="2015-05-10T02:10:41.533" LastActivityDate="2015-05-10T02:10:41.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5737" PostTypeId="1" AcceptedAnswerId="5764" CreationDate="2015-05-10T09:58:56.670" Score="5" ViewCount="1802" Body="&lt;p&gt;I found a data set called &lt;a href=&quot;http://www.cs.cmu.edu/~./enron/&quot;&gt;Enron Email Dataset&lt;/a&gt;. It is possibly the only substantial collection of &quot;real&quot; email that is public. I found some prior analysis of this work:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A paper describing the Enron data was presented at the 2004 CEAS conference.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some experiments associated with this data are described on Ron Bekkerman's home page&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.parakweet.com/&quot;&gt;Parakweet&lt;/a&gt; has released an open source set of Enron sentence data, labeled for speech acts.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Work at the University of Pennsylvania includes a query dataset for email search as well as a tool for generating spelling errors based on the Enron corpus. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm looking for some interesting current trend topics to work with.please give me some suggestions.&lt;/p&gt;&#xA;" OwnerUserId="9035" LastEditorUserId="182" LastEditDate="2015-05-11T02:49:45.817" LastActivityDate="2017-12-01T11:14:41.023" Title="What kind of research can be done with an email data set?" Tags="|data-mining|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5738" PostTypeId="2" ParentId="5718" CreationDate="2015-05-10T10:03:20.360" Score="4" Body="&lt;p&gt;Once you have your own lists of named entities, and you're only interested in extracting the relations, I believe there are simpler solutions (although I never tried relation extraction in NLTK, so I might be wrong):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://reverb.cs.washington.edu/&quot; rel=&quot;nofollow&quot;&gt;ReVerb&lt;/a&gt; - a tool written in Java. Once it produces the results, you can simply keep the rows, where your labels are present as objects of the relation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://knowitall.github.io/openie/&quot; rel=&quot;nofollow&quot;&gt;OpenIE&lt;/a&gt; - the successor of ReVerb (also written in Java). The authors claim better perfomance, and the output might be more informative.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://iepy.readthedocs.org/en/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;IEPY&lt;/a&gt; - a relation extraction tool in Python. You should be able to provide your own labels/named entities using &lt;a href=&quot;http://iepy.readthedocs.org/en/latest/gazettes.html&quot; rel=&quot;nofollow&quot;&gt;gazetees&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/mit-nlp/MITIE&quot; rel=&quot;nofollow&quot;&gt;MITIE&lt;/a&gt; - this library has bindings in Python, and it offers &lt;a href=&quot;https://github.com/mit-nlp/MITIE/blob/master/examples/python/ner.py&quot; rel=&quot;nofollow&quot;&gt;relation extraction&lt;/a&gt; functionality.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="173" LastActivityDate="2015-05-10T10:03:20.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5741" PostTypeId="1" CreationDate="2015-05-10T11:49:07.233" Score="3" ViewCount="82" Body="&lt;p&gt;When do we feel need to go through non-linear transformation like kernel PCA ? Please share an example&lt;/p&gt;&#xA;" OwnerUserId="7702" LastEditorUserId="182" LastEditDate="2015-05-11T20:49:54.007" LastActivityDate="2015-05-12T20:08:09.497" Title="When to Perform non-linear dimension reduction" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5742" PostTypeId="2" ParentId="307" CreationDate="2015-05-10T16:14:00.560" Score="6" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.kaggle.com/wiki/DataScienceUseCases&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;&lt;/strong&gt; has a short summary of applications&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.revolutionanalytics.com/resources/case-studies?type%5B0%5D=case_study&amp;amp;keys=&quot; rel=&quot;nofollow noreferrer&quot;&gt;Revolution Analytics&lt;/a&gt;&lt;/strong&gt; published many general case studies, datasheets, and white papers&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For applications in sciences and engineering, you can consult &lt;strong&gt;&lt;a href=&quot;http://www.nutonian.com/industries/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nutonian&lt;/a&gt;&lt;/strong&gt; case studies&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.analyx.com/en/company/references/cases.aspx&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analyx&lt;/a&gt;&lt;/strong&gt; told potential clients about applications in commerce&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;http://im.ft-static.com/content/images/e91a32d0-2bac-11e3-bfe2-00144feab7de.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Financial Times&lt;/a&gt;&lt;/strong&gt; published a collection of stories about business applications of big data&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation&quot; rel=&quot;nofollow noreferrer&quot;&gt;McKinsey&lt;/a&gt;&lt;/strong&gt; outlined applications back in 2011&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Other consulting firms made similar reports.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Gartner&lt;/strong&gt; created Hype Cycle for Big data:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/YvQuk.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Not to mention the case studies and white papers by other companies that want to promote their products.&lt;/p&gt;&#xA;" OwnerUserId="5279" LastEditorUserId="98307" LastEditDate="2020-08-16T16:54:32.553" LastActivityDate="2020-08-16T16:54:32.553" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5743" PostTypeId="2" ParentId="1216" CreationDate="2015-05-10T18:49:29.037" Score="1" Body="&lt;p&gt;I haven't seen this mentioned, but it's important to keep in mind that you may see a decrease in salary. I say this without knowing how much you make, but moving from (I assume) an experienced IT professional to an entry level data scientist level may not earn you as much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a link to the a portion of the 2015 Burtch Works study on Data Science salaries:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.burtchworks.com/files/2015/05/DS-2015_Changes-in-Base-Salaries.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.burtchworks.com/files/2015/05/DS-2015_Changes-in-Base-Salaries.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the median salary for level 1 individual contributors is 90k (across the nation). The full report has the breakdown based on region but again, assuming you're an experienced IT professional, you're probably making more than that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anecdotal story with n=1: One of my classmates in my DS masters program was an experienced Java developer with a house, family, etc. Although he was very interested in data analytics (paid for the program out of pocket) his potential salary doing data analytics wouldn't be able to support the lifestyle he currently had as a Java developer. As a result he essentially &quot;wasted&quot; his degree and went back to development. I would really hate to see that happen to more people.&lt;/p&gt;&#xA;" OwnerUserId="525" LastActivityDate="2015-05-10T18:49:29.037" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5744" PostTypeId="2" ParentId="77" CreationDate="2015-05-10T21:12:49.860" Score="4" Body="&lt;p&gt;There are good/fast ways to model graphs in RDBMS, and dumb/slow ways. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Some use clever indexing and Stored Procs, trading CPU load and tuned temp tables on RAM disks for faster graph retrieval speed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some use precomputed graph paths (this may be less feasible in social network scenario, but in a tree with majority of nodes being leaf nodes, it's a pretty good tradeoff space-for-time&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some simply compute in a loop, using un-tuned in-indexed temp table. From the #s thrown in the article, that smells like what they did (30 second- performance on fairly smallish data-set)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have my own tree computation.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;It is encapsulated in a highly-tuned stored proc&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;While it's running in an enterprise-sized-hardware Sybase ASE15 dataserver, that server is shared with a couple terabytes of data from &lt;strong&gt;all other&lt;/strong&gt; enterprise apps, some much more data hungry than mine; and isn't dedicated solely to executing my queries.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I did &lt;strong&gt;not&lt;/strong&gt; have access to the main speedup tool, a temp table on a RAM disk.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A representative set of data I was retrieving that seems to somewhat match theirs was getting a 150,000 node subtree out of 2.5M node full forest dataset (unlimited depth of tree, which varies between 5 and 15, but smaller &lt;em&gt;average&lt;/em&gt; arity of a given node than the 50 friends listed in the experiment) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I tuned it to the point that this query ~30-45 seconds. It most certainly does NOT exhibit the exponential slowdown that the figures in the question seem to indicate on their RDBMS performance, which is extra double strange given there is no exponential growth in the result set (which to me reeks of un-tuned index on a temp table from personal experience).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, this comparison is &lt;strong&gt;most likely&lt;/strong&gt; incorrect and based on poor RDBMS side design, although as the previous answer noted, it is impossible to ascertain without them open sourcing 100% of their code and table definitions.&lt;/p&gt;&#xA;" OwnerUserId="132" LastEditorUserId="132" LastEditDate="2015-05-10T21:18:01.617" LastActivityDate="2015-05-10T21:18:01.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5745" PostTypeId="2" ParentId="5737" CreationDate="2015-05-11T04:21:20.767" Score="4" Body="&lt;p&gt;The following are some research that can done on e-mail dataset:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;linguistic analysis to abbreviate an email message&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Categorize e-mail as spam/ham using machine learning techniques.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;identifying concepts expressed in a collection of email messages, and&#xA;organizing them into an ontology or taxonomy for browsing&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9174" LastActivityDate="2015-05-11T04:21:20.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5746" PostTypeId="1" AcceptedAnswerId="5780" CreationDate="2015-05-11T06:53:52.973" Score="14" ViewCount="2684" Body="&lt;p&gt;I am looking to do k-means clustering on a set of 10-dimensional points.  The catch: &lt;strong&gt;there are &lt;span class=&quot;math-container&quot;&gt;$10^{10}$&lt;/span&gt; points&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am looking for just the center and size of the largest clusters (let's say 10 to 100 clusters); I don't care about what cluster each point ends up in.  Using k-means specifically is not important; I am just looking for a similar effect, any approximate k-means or related algorithm would be great (minibatch-SGD means, ...).  Since GMM is in a sense the same problem as k-means, doing GMM on the same size data is also interesting.&lt;/p&gt;&#xA;&lt;p&gt;At this scale, subsampling the data probably doesn't change the result significantly: the odds of finding the same top 10 clusters using a 1/10000th sample of the data are very good.  But even then, that is a &lt;span class=&quot;math-container&quot;&gt;$10^6$&lt;/span&gt; point problem which is on/beyond the edge of tractable.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="29169" LastEditDate="2021-03-13T20:04:57.623" LastActivityDate="2021-03-13T20:04:57.623" Title="Fast k-means like algorithm for $10^{10}$ points?" Tags="|clustering|k-means|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5749" PostTypeId="2" ParentId="5723" CreationDate="2015-05-11T12:49:42.757" Score="6" Body="&lt;p&gt;This is only possible with KnowledgeFlow. In WekaManual.pdf (which is included in Weka package) for version 3.7.12 there is an example in Chapter 7.4.2 &quot;Plotting multiple ROC curves&quot; with picture and step-by-step instructions. For other Weka versions it is the same, just find the appropriate chapter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give an impression on how it goes, I extracted the picture from the manual. It will draw two curves for two classifiers. For your question it is very similar. You use one classifier and then connect trainingSet to one ClassifierPerformanceEvaluator and testSet to another.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/ZzEKV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2015-05-14T21:21:01.763" LastActivityDate="2015-05-14T21:21:01.763" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5750" PostTypeId="1" CreationDate="2015-05-11T13:07:53.560" Score="1" ViewCount="67" Body="&lt;p&gt;I have an Excel sheet with &lt;code&gt;n&lt;/code&gt; columns, these columns contain info about the students. For admissions we have the score of test scores in multiple subject areas, scores from an interview, and scores of a written test and comprehension test. There is a column which contains student's academic level (High, M.High, Middle, M.Low, Low). I want to compare the last column with the others variables and see whether there are common features that passing students have in common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there software for this? If this can be done with excel, how can I do it? Does SPSS provide this kind of analysis?&lt;/p&gt;&#xA;" OwnerUserId="9127" LastEditorUserId="9127" LastEditDate="2015-05-11T14:54:13.473" LastActivityDate="2016-08-11T09:36:21.693" Title="Determine what data, does certain column have in common" Tags="|data-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5751" PostTypeId="1" CreationDate="2015-05-11T15:07:47.143" Score="5" ViewCount="1696" Body="&lt;p&gt;I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's imagine it's the term &quot;Christmas&quot;). Unfortunately, there are also far less news articles in the third quarter than in the second (due to the lack of news in the summer). So how do we accurately compare the counts in each quarter? We assume that there will be a greater number of occurrences in the fourth quarter, but how much does the magnitude of this difference depend on the change in size of the underlying text?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Heaps%27_law&quot; rel=&quot;noreferrer&quot;&gt;Heap's law&lt;/a&gt; shows the relationship between text size and number of unique terms. It's non-linearity implies that the rate of new, unique words introduced by the text decreases as you increase the size of the text, and the proportion of the text taken up by each existing word therefore increases. This applies given documents taken from the same 'distribution' of text, in other words the underlying zipfian distribution of word ranks is identical (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Zipf%27s_law&quot; rel=&quot;noreferrer&quot;&gt;wiki&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my example above this is obviously not the case, since the underlying topics, and resultant term distribution, will change between summer and winter, especially with regards to the term &quot;Christmas&quot;. But take the same term count but over the whole of 2013 and 2014; you would reasonably expect the general underlying term distribution to be the same in each period, so Heap's law applies, but what if the volume of text has changed? Simply normalising by the size of the text, or the number of documents, does not, as far as I can tell, account for the relative change in expected value of the term count.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a hunch that this might be a simple application of Heap's or Zipf's laws, but I can't see how to apply them to this particular question. Appreciate any insight.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2018-10-25T11:05:32.660" Title="How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?" Tags="|statistics|nlp|text-mining|time-series|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5752" PostTypeId="2" ParentId="5147" CreationDate="2015-05-11T15:30:10.893" Score="0" Body="&lt;p&gt;Okay, I eventually found what I was looking for in the Data Mining Community. There seem to be two candidates, CRISP-DM which comes from SPSS originally but is &quot;Cross-Industry&quot;, and SEMMA which comes from SAS. They are both pretty much what I was looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CRISP-DM &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SEMMA &lt;a href=&quot;http://en.wikipedia.org/wiki/SEMMA&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/SEMMA&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7720" LastActivityDate="2015-05-11T15:30:10.893" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5753" PostTypeId="2" ParentId="5737" CreationDate="2015-05-11T16:14:02.463" Score="3" Body="&lt;p&gt;Wonderful dataset with many opportunities to brush up on text analysis skills!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought would be to try some Topic Modelling on the dataset. If you are using Python there is a library I've used called &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;gensim&lt;/a&gt; which has some fairly thorough tutorials to get you started. A friend of mine &lt;a href=&quot;https://github.com/rpetchler/Enron_Topic_Modeling&quot; rel=&quot;nofollow&quot;&gt;did something similar&lt;/a&gt; with the Enron dataset, using parallelized preprocessing and distributed latent Dirichlet allocation to infer topics over the email corpus. &lt;/p&gt;&#xA;" OwnerUserId="9569" LastActivityDate="2015-05-11T16:14:02.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5754" PostTypeId="2" ParentId="5746" CreationDate="2015-05-11T17:01:05.930" Score="6" Body="&lt;p&gt;As a side comment note that using K-means for 10D data &lt;strong&gt;might&lt;/strong&gt; end up in nowhere according to the curse of dimensionality. Of course it varies a bit according to the nature of the data but once I tried to determine the threshold in which K-Means starts behaving strange regarding the dimensionalty, I got something like 7D. After 7 dimensions it started to miss correct clusters (my data was manually generated according to 4 well-separated Gaussian distributions and I used MATLAB &lt;em&gt;kmeans&lt;/em&gt; function for my little experiment).&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2015-05-11T18:53:53.803" LastActivityDate="2015-05-11T18:53:53.803" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5755" PostTypeId="2" ParentId="5750" CreationDate="2015-05-11T18:14:43.463" Score="2" Body="&lt;p&gt;A good place to start would be to look at the &lt;a href=&quot;http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Spearman's Rank Correlation&lt;/a&gt; between your dependent variable (academic level) and your individual independent variables (other columns). This should give you a basic indication of whether any of your columns are correlated with academic level.  This should be straightforward to implement in both Excel and SPSS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to go a step further you could set the problem up as one of &lt;a href=&quot;http://en.wikipedia.org/wiki/Multinomial_logistic_regression&quot; rel=&quot;nofollow&quot;&gt;Multinomial Logistic Regression&lt;/a&gt;. This would allow you to build a model to directly attempt to predict academic level from your other variables. I'm certain there's probably &lt;em&gt;some way&lt;/em&gt; to do this in Excel, but SPSS can definitely handle it.&lt;/p&gt;&#xA;" OwnerUserId="182" LastActivityDate="2015-05-11T18:14:43.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5756" PostTypeId="2" ParentId="5303" CreationDate="2015-05-11T20:06:17.680" Score="2" Body="&lt;p&gt;&lt;strong&gt;So how can I couple my clusters to a feature label from my feature matrix?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal Components are not intuitive features. What seems common here is to cluster users based on PCs and investigate clusters based on original features afterward i.e. extract different clusters and plot data based on different subsets of features and use different colors for different clusters. It might give you some intution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two paths can be seen in many PCA results where the information varies with e.g time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance in your case a beginner user visits less than an old one and the number of new users are most probably higher than old ones so data will be more dense around origin and far it gets from the origin less dense it becomes. Such a phenomenon affects your PCs as well so you'll see some paths along different lines in PC space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-11T20:06:17.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5757" PostTypeId="2" ParentId="5110" CreationDate="2015-05-11T20:22:42.187" Score="1" Body="&lt;p&gt;I personally use &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;NetworkX&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;http://igraph.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;igraph&lt;/strong&gt;&lt;/a&gt; where the first one is for programming in Python and the second one has interfaces for Python, R and C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not into programming much, &lt;a href=&quot;http://gephi.github.io/&quot; rel=&quot;nofollow&quot;&gt;Gephi&lt;/a&gt; is what you are looking for!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2015-05-12T12:01:48.797" LastActivityDate="2015-05-12T12:01:48.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5758" PostTypeId="2" ParentId="4974" CreationDate="2015-05-11T20:37:43.223" Score="5" Body="&lt;p&gt;So what you need is &lt;a href=&quot;http://supernet.isenberg.umass.edu/fulbright-catania/workshop-talks/nicosia-nagurney-daniele-workshop.pdf&quot; rel=&quot;noreferrer&quot;&gt;Modularity score&lt;/a&gt;. Speaking as a Graph Clustering guy (my master thesis topic, PhD research and my main research direction during last 2.5 years) I recommend you to go through what physicists did in &lt;strong&gt;Complex Networks&lt;/strong&gt; field under the name of &lt;strong&gt;Community Detection&lt;/strong&gt;. If you search &lt;em&gt;Prof. Mark Newman&lt;/em&gt; who first proposed Modularity score you'll find a bunch of interesting papers in this field. &lt;em&gt;Infomap&lt;/em&gt; algorithm by &lt;em&gt;Martin Rosvall&lt;/em&gt;, &lt;em&gt;Louvain&lt;/em&gt; algorithm by &lt;em&gt;Vincent Blondel&lt;/em&gt; and &lt;em&gt;CNM&lt;/em&gt; algorithm by &lt;em&gt;Aaron Clauset&lt;/em&gt; are some of the most known algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most commonly used algorithm for graph clustering nowadays is the one by &lt;a href=&quot;http://perso.crans.org/aynaud/communities/&quot; rel=&quot;noreferrer&quot;&gt;Vincent Blondel&lt;/a&gt; which has implementations for both NetworkX and igraph (if you are a python guy!). This algorithm is originally for weighted graphs and probably answers your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps, Good luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-11T20:37:43.223" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5759" PostTypeId="2" ParentId="596" CreationDate="2015-05-11T20:57:50.870" Score="5" Body="&lt;p&gt;What you are looking for can be found in &lt;a href=&quot;http://konect.uni-koblenz.de/networks&quot; rel=&quot;noreferrer&quot;&gt;KONECT&lt;/a&gt; (the website is down as I'm writing this but it should be fixed soon!). It's almost the most comprehensive data collection for network analysis. But the question is which one is more &lt;strong&gt;standard&lt;/strong&gt; to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, there is no clear answer except of Zachary's Karate Club!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do a literature review in Community Detection algorithms you'll see that almost all shining papers use different networks. My suggestion is going through what Andrea Lancichinetti and Santo Fortunato did for benchmarking graphs. They proposed some benchmark graph generation algorithms e.g. &lt;a href=&quot;http://arxiv.org/abs/0904.3940&quot; rel=&quot;noreferrer&quot;&gt;this one&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :) &lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2015-05-11T21:08:34.500" LastActivityDate="2015-05-11T21:08:34.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5760" PostTypeId="2" ParentId="3779" CreationDate="2015-05-11T21:06:39.957" Score="3" Body="&lt;p&gt;Using Community detection you can build a recommendation system. The most commonly used algorithm in this field is &lt;a href=&quot;http://arxiv.org/pdf/0803.0476.pdf&quot; rel=&quot;nofollow&quot;&gt;Blondel Algorithm&lt;/a&gt; which u have probably seen in SNAP. Blondel is almost the fastest Community Detection algorithm among widely accepted ones and its result is pretty acceptable (at least according to modularity score). As a side comment, you may also want to have a look at NetworkX and igraph libraries for more graph algorithms and visualizations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps, good luck.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-11T21:06:39.957" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5761" PostTypeId="1" CreationDate="2015-05-11T21:40:07.770" Score="1" ViewCount="80" Body="&lt;p&gt;I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical tasks. Now I am striving to find appropriate references about their performance when the data becames &quot;Big&quot;. In other words, how is their performance when the data becames really big. Does the training time increase more than linear? Is there any comparative benchmark between computational time and precision when the data becames high (for the various predictive models).&lt;/p&gt;&#xA;" OwnerUserId="9575" LastActivityDate="2015-05-15T10:00:06.307" Title="computational time and precision trade off" Tags="|data-mining|bigdata|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5762" PostTypeId="2" ParentId="1213" CreationDate="2015-05-11T22:48:36.383" Score="4" Body="&lt;p&gt;You can simply convert a &lt;strong&gt;M&lt;/strong&gt;x&lt;strong&gt;N&lt;/strong&gt; matrix (your image) into a &lt;strong&gt;MN&lt;/strong&gt; dimensional vector. For instance if you have 10 images each of which 10x10 pixels, you can convert them to 10 vectors each of them containing 100 element (you can do it row-wise or column-wise, does not matter). Then putting all these vectors on each other constructs a &lt;strong&gt;n&lt;/strong&gt;x&lt;strong&gt;d&lt;/strong&gt; matrix where &lt;strong&gt;n&lt;/strong&gt; is the number of images and &lt;strong&gt;d&lt;/strong&gt; is the number of pixels in each image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far we just prepared data for analysis. Now let's go to the question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;I'm looking for a supervised learning algorithm that can take 2d data&#xA;for input and output.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What does that mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First- The dimension of input-output is not a property of a machine learning algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second- By &lt;strong&gt;2d&lt;/strong&gt;, you mean a matrix? I'm afraid your data is not really 2d! If you mean a matrix, a &lt;strong&gt;M&lt;/strong&gt;x&lt;strong&gt;N&lt;/strong&gt; matrix is a data of M samples in &lt;strong&gt;N&lt;/strong&gt; dimensional space. also those vectors you may produce from your images are in &lt;strong&gt;MN&lt;/strong&gt; dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third- A supervised algorithm takes some teaching samples (called training) to learn a task. What are those &lt;em&gt;teachers&lt;/em&gt; here? If you have a training data which contains some blurred and corresponding original images then you can try to learn the place of white pixels by exploring blurred area. My suggestion:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If a smoothing filter has been applied to your image, the blurred pixels most probably look like a gaussian. So first extract the gaussians.&lt;/li&gt;&#xA;&lt;li&gt;As you already have the place of sharp whites correspond to those gaussians you can use the mean, location of the mean and also variance as features (inputs) and the place of sharp white pixels as the target (&lt;em&gt;output&lt;/em&gt;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The classifier could be a simple neural network (a MLP should work here) or a simple SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer above can not be evaluated without more information about your data and a more specific description of the question (e.g. What are your images exactly? if you can post one of them here I can probably help more)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2015-05-12T11:53:40.030" LastActivityDate="2015-05-12T11:53:40.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5763" PostTypeId="2" ParentId="5291" CreationDate="2015-05-12T09:30:09.273" Score="2" Body="&lt;p&gt;It seems to be working well on my configuration:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ubuntu Vivid and R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sessionInfo()&#xA;R version 3.1.2 (2014-10-31)&#xA;Platform: x86_64-pc-linux-gnu (64-bit)&#xA;&#xA;locale:&#xA; [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              &#xA; [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    &#xA; [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   &#xA; [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 &#xA; [9] LC_ADDRESS=C               LC_TELEPHONE=C            &#xA;[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       &#xA;&#xA;attached base packages:&#xA;[1] stats     graphics  grDevices utils     datasets  methods   base     &#xA;&#xA;other attached packages:&#xA;[1] twitteR_1.1.8&#xA;&#xA;loaded via a namespace (and not attached):&#xA; [1] bit_1.1-12     bit64_0.9-4    bitops_1.0-6   DBI_0.3.1      httr_0.6.1    &#xA; [6] magrittr_1.5   RCurl_1.95-4.6 rjson_0.2.15   stringi_0.4-1  stringr_1.0.0 &#xA;[11] tools_3.1.2  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Maybe you should update packages versions?&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2015-05-12T09:30:09.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5764" PostTypeId="2" ParentId="5737" CreationDate="2015-05-12T10:10:58.977" Score="2" Body="&lt;p&gt;You're learning, are you? Try to find something easy and interesting to start. Why don't you start off something easy like building a Bayesian model to predict which email will get deleted. You should glance over those deleted emails, are they spams? are they just garbage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, you have a simply supervised model where the data-set already labels the emails for you (deleted or not). Think of something easy like words, titles, length of the email etc, see if you can build a model that predicts email deletion.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2015-05-12T10:10:58.977" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5765" PostTypeId="2" ParentId="5093" CreationDate="2015-05-12T10:59:50.547" Score="1" Body="&lt;p&gt;The InterQuartileRangeFilter from weka library uses an IQR formula to designate some values as outliers/extreme values. Any value outside this range $[Q_1 - k(Q_3-Q_1), Q_3 + k(Q_3-Q_1)]$ is considered some sort of an outlier, where $k$ is some constant, and $ IQR = Q_3 - Q_1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default weka uses $k=3$ to define something as outlier, and $k=3*2$ to define something as extreme value (extreme outlier). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula guarantees that at least 50% value are considered non-outliers. Having a single variable (univariate sample of values), it's practically impossible to reproduce your result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note however that this filter can be applied to a data frame. When applied like this, it will consider as an outlier any instance of the data frame which has at least one value of the instance considered as outlier for that variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, supposing that you have a data frame with 2 variables, totally uncorrelated (independent). Considering again that only 10% of the values from each variable are considered outliers, due to independence, one can expect that $(1-0.9)^2$ values will not be outliers. If you have $p$ variables like that in your data frame, you might expect to have only $(1-0.9)^k$ normal values, and is not very hard to arrive in that situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two things which you will have to consider. One is to increase the factors for outliers if in general too many values are considered outliers (ideally you would like to take a look at each variable graphically and if possible to get some idea about the distribution beneath). The second one is to check if you have many values which are totally independent. The second hint does not solve your problem but might give you a reason why it happens. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-05-12T10:59:50.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5766" PostTypeId="1" CreationDate="2015-05-12T14:28:14.390" Score="0" ViewCount="94" Body="&lt;p&gt;I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in s&lt;a href=&quot;https://archive.org/details/stackexchange&quot; rel=&quot;nofollow&quot;&gt;tack exchange  data dump&lt;/a&gt;. however i browse for researches i found a lot .And whatever i thought about a idea based on this dataset , its already done by someone else . &#xA;please help me out with a new and useful idea for my research .&lt;/p&gt;&#xA;" OwnerUserId="9586" LastActivityDate="2015-05-12T15:17:37.060" Title="new and fresh ideas with stack exchange dataset" Tags="|data-mining|bigdata|dataset|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-12T21:58:17.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="5767" PostTypeId="2" ParentId="5766" CreationDate="2015-05-12T15:17:37.060" Score="2" Body="&lt;p&gt;It might help to have a bit more information on what you want to do, but have a look at &lt;a href=&quot;https://www.kaggle.com&quot; rel=&quot;nofollow&quot;&gt;Kaggle&lt;/a&gt; - it's a regular big data science competition, all of the data sets are huge and &lt;a href=&quot;https://www.kaggle.com&quot; rel=&quot;nofollow&quot;&gt;available for download&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also some &lt;a href=&quot;https://public.tableau.com/s/resources&quot; rel=&quot;nofollow&quot;&gt;smaller datasets made available by Tableau&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that give you some inspiration!&lt;/p&gt;&#xA;" OwnerUserId="9587" LastActivityDate="2015-05-12T15:17:37.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5769" PostTypeId="2" ParentId="326" CreationDate="2015-05-12T15:51:41.727" Score="6" Body="&lt;p&gt;Not much to add to the provided comments. Only thing is maybe this infographic comparing R vs Python for data science purposes &lt;a href=&quot;http://blog.datacamp.com/r-or-python-for-data-analysis/&quot; rel=&quot;noreferrer&quot;&gt;http://blog.datacamp.com/r-or-python-for-data-analysis/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9588" LastActivityDate="2015-05-12T15:51:41.727" CommentCount="0" CommunityOwnedDate="2015-07-19T09:16:15.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="5770" PostTypeId="2" ParentId="5504" CreationDate="2015-05-12T17:23:52.323" Score="2" Body="&lt;p&gt;For question 1, I feel there are a great deal of things you can look into with this.  The basic conclusion is which states are culturally similar to one another (physical proximity would lead to such similarity).  Having a 'cultural similarity score' from year to year, you could determine the connectedness of states over time, see if major events effect how connected the country is..... there is a lot of potential here, and reenforces the notion that you need to know what question you're trying to answer before you get too far into an analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for two, this is a bit trickier.  The visual I see is each state getting a node at it's center, and like states having color coded connectors.  The visual I'm seeing is similar to the graph they use for the scikit learn site for affinity propagation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_affinity_propagation.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/auto_examples/cluster/plot_affinity_propagation.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-05-12T17:23:52.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5771" PostTypeId="1" AcceptedAnswerId="5772" CreationDate="2015-05-12T18:34:44.010" Score="3" ViewCount="313" Body="&lt;p&gt;Users tend to click on results ranked highly by search engines much more often than those ranked lower. How do you train a search engine using click data / search logs without this bias? I.e. you don't want to teach the search engine that the results that are currently ranked highly should necessarily continue to be ranked highly just because they were frequently clicked.&lt;/p&gt;&#xA;" OwnerUserId="9579" LastActivityDate="2015-05-12T20:08:04.033" Title="Ranking Bias in Learning to Rank" Tags="|machine-learning|search|ranking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5772" PostTypeId="2" ParentId="5771" CreationDate="2015-05-12T20:08:04.033" Score="2" Body="&lt;p&gt;Have you seen this paper?&#xA;&lt;a href=&quot;http://www.cs.cornell.edu/people/tj/publications/joachims_02c.pdf&quot; rel=&quot;nofollow&quot;&gt;Optimizing Search Engines using Clickthrough Data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I stumbled upon this the other day, and I'm still reading through it, but the author attempts to deal with the problem you describe. You may also find &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/sdumais/sigir2006-fp345-ranking-agichtein.pdf&quot; rel=&quot;nofollow&quot;&gt;Improving Web Search Ranking by Incorporating User Behavior Information&lt;/a&gt; useful.&lt;/p&gt;&#xA;" OwnerUserId="9576" LastActivityDate="2015-05-12T20:08:04.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5773" PostTypeId="2" ParentId="5741" CreationDate="2015-05-12T20:08:09.497" Score="3" Body="&lt;p&gt;The question is too general to have a specific recipe as an answer. Depends on the data and the relation between features but to the best of my knowledge:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the relation between your features are non-linear i.e. when a non-linear interpretation of correlation between features makes more sense than the linear interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A detailed explanation of what I mean can be found in &lt;a href=&quot;https://stats.stackexchange.com/questions/101344/is-kernel-pca-with-linear-kernel-equivalent-to-standard-pca&quot;&gt;this&lt;/a&gt; nice answer. Please note that the $X^TX$ in the 4th line of the first paragraph should be $XX^T$, because in data mining, we conventionally use $N\times D$ format for data matrices. So in this example, we are looking for the gram matrix of the rows and not columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-05-12T20:08:09.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5775" PostTypeId="1" CreationDate="2015-05-12T22:33:38.013" Score="2" ViewCount="2203" Body="&lt;p&gt;Building out a system that tries to apply zero or more predefined labels to text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each label, we've:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;built out a reasonably good vocabulary of high-value words/features&lt;/li&gt;&#xA;&lt;li&gt;developed a corpus containing thousands of labeled entries&lt;/li&gt;&#xA;&lt;li&gt;trained a NaiveBayesClassifier for each topic that does a good job of classifying valid vs noisy content&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The problem seems to be that the individual classifier is great at differentiating between valid &amp;amp; noisy content &lt;strong&gt;WITHIN&lt;/strong&gt; a topic:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;the green energy bill will revolutionize...&quot; (green = &quot;green energy&quot;)&lt;/li&gt;&#xA;&lt;li&gt;&quot;the green bay packers went on to lose their...&quot; (green != &quot;green energy&quot;) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;...but when classifying content that shouldn't match ANY topic it has a very high rate of false positives. There's no &quot;everything else&quot; label!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;tl;dr it's good at subtle, in-topic differentiation, but terrible at broad topic labeling&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any algorithms that help you classify into N categories, but allow for &quot;everything else&quot; which might not fit into ANY of the categories?&lt;/p&gt;&#xA;" OwnerUserId="9591" LastActivityDate="2015-05-13T16:20:31.883" Title="Multi-label text classification with minimum confidence threshold" Tags="|classification|python|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5776" PostTypeId="1" AcceptedAnswerId="5778" CreationDate="2015-05-13T01:27:32.653" Score="5" ViewCount="9201" Body="&lt;p&gt;Could somebody please recommend a good R package for doing logit and probit regression? I have tried to find an answer by searching on Google but all the links I find go into lengthy explanations about what logit regression is, which I already know, but nobody seems to recommend an R package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jerome Smith&lt;/p&gt;&#xA;" OwnerUserId="9595" LastEditorUserId="8953" LastEditDate="2015-05-14T09:19:27.987" LastActivityDate="2017-11-08T18:53:49.830" Title="Best or recommended R package for logit and probit regression" Tags="|r|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5777" PostTypeId="1" CreationDate="2015-05-13T02:24:54.520" Score="4" ViewCount="1797" Body="&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.ensemble import RandomForestClassifier&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; clf = RandomForestClassifier(n_estimators=10, random_state=1)&#xA;&amp;gt;&amp;gt;&amp;gt; Y=[0,1]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X = [[3,2,1,0], [7,6,5,4]]&#xA;&amp;gt;&amp;gt;&amp;gt; clf = clf.fit(X, Y)&#xA;&amp;gt;&amp;gt;&amp;gt; print clf.feature_importances_&#xA;[ 0.2  0.1  0.1  0. ]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X = [[0, 1, 2,3], [4,5,6,7]]&#xA;&amp;gt;&amp;gt;&amp;gt; clf = clf.fit(X, Y)&#xA;&amp;gt;&amp;gt;&amp;gt; print clf.feature_importances_&#xA;[ 0.2  0.1  0.1  0. ]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X = [[3,2,1,0], [7,6,5,4]]&#xA;&amp;gt;&amp;gt;&amp;gt; clf = clf.fit(X, Y)&#xA;&amp;gt;&amp;gt;&amp;gt; print clf.feature_importances_&#xA;[ 0.2  0.1  0.1  0. ]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X = [[3,1,2,0], [7,5,6,4]]&#xA;&amp;gt;&amp;gt;&amp;gt; clf = clf.fit(X, Y)&#xA;&amp;gt;&amp;gt;&amp;gt; print clf.feature_importances_&#xA;[ 0.2  0.1  0.1  0. ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Assume the features have names.&#xA;When I shuffle/change the listed order of features specified in the training data set, the importance for each feature changes.&lt;br&gt;&#xA;That means the resulted random forest classifier also changes.&#xA;Note that I have rule out the effect of randomness, by fixing the random seed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does the listed order of features specified in the data set matter to the random forest classifier, given that the random seed is fixed? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="562" LastActivityDate="2017-06-02T08:32:44.960" Title="Why does the listed order of features specified in the data set matter to the random forest classifier" Tags="|random-forest|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5778" PostTypeId="2" ParentId="5776" CreationDate="2015-05-13T02:47:46.463" Score="8" Body="&lt;p&gt;Unless you have some very specific or exotic requirements, in order to perform &lt;em&gt;logistic&lt;/em&gt; (&lt;em&gt;logit&lt;/em&gt; and &lt;em&gt;probit&lt;/em&gt;) regression analysis in &lt;code&gt;R&lt;/code&gt;, you can use standard (built-in and loaded by default) &lt;code&gt;stats&lt;/code&gt; package. In particular, you can use &lt;code&gt;glm()&lt;/code&gt; function, as shown in the following nice &lt;em&gt;tutorials&lt;/em&gt; from UCLA: &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/logit.htm&quot;&gt;logit in R tutorial&lt;/a&gt; and &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/probit.htm&quot;&gt;probit in R tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in &lt;em&gt;multinomial logistic regression&lt;/em&gt;, &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/mlogit.htm&quot;&gt;this UCLA tutorial&lt;/a&gt; might be helpful (you can use &lt;code&gt;glm()&lt;/code&gt; or packages, such as &lt;code&gt;glmnet&lt;/code&gt; or &lt;code&gt;mlogit&lt;/code&gt;). For the above-mentioned very specific or exotic requirements, many other R packages are available, for example &lt;code&gt;logistf&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/logistf&quot;&gt;http://cran.r-project.org/web/packages/logistf&lt;/a&gt;) or &lt;code&gt;elrm&lt;/code&gt; (&lt;a href=&quot;http://cran.r-project.org/web/packages/elrm&quot;&gt;http://cran.r-project.org/web/packages/elrm&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also recommend another nice &lt;a href=&quot;http://data.princeton.edu/R/glms.html&quot;&gt;tutorial on GLMs&lt;/a&gt; from Princeton University (by Germán Rodríguez), which discusses some modeling aspects, not addressed in the UCLA materials, in particular &lt;em&gt;updating models&lt;/em&gt; and &lt;em&gt;model selection&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-05-13T02:47:46.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5780" PostTypeId="2" ParentId="5746" CreationDate="2015-05-13T07:59:15.927" Score="12" Body="&lt;p&gt;k-means is based on &lt;strong&gt;averages&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It models clusters using means, and thus &lt;strong&gt;the improvement by adding more data is marginal.&lt;/strong&gt; The error of the average estimation reduces with 1/sqrt(n); so adding more data pays off less and less...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strategies for such large data always revolve around sampling:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If you want sublinear runtime, you have to do sampling!&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, Mini-Batch-Kmeans etc. do exactly this: repeatedly sample from the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, sampling (in particular unbiased sampling) isn't exactly free either... usually, you will have to read your data linearly to sample, because you don't get random access to individual records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd go with MacQueen's algorithm. It's online; by default it does a single pass over your data (although it is popular to iterate this). It's not easy to distribute, but I guess you can afford to linearly read your data say 10 times from a SSD?&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-05-13T07:59:15.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5782" PostTypeId="2" ParentId="5777" CreationDate="2015-05-13T12:51:59.373" Score="5" Body="&lt;p&gt;Since your fixed the seed, with default option &lt;code&gt;bootstrap=True&lt;/code&gt; each tree is build on a fixed subsample of your data. With the default option &lt;code&gt;max_features='auto'&lt;/code&gt; each tree uses 2 features. Since your fixed the seed, it is always the same, let us say, &quot;1st and 4th&quot; features for the first tree. And they are different in each of your shuffles. So, the tree is different. The same applies to each tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, with 4 features in total, only 6 (4*3/2=6) various trees are possible. With n_estimators=10 some of these trees are necessarily repeated.&#xA;And in each shuffle those are different trees.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2015-05-13T13:30:00.440" LastActivityDate="2015-05-13T13:30:00.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5783" PostTypeId="1" AcceptedAnswerId="5784" CreationDate="2015-05-13T15:06:49.200" Score="4" ViewCount="304" Body="&lt;p&gt;I dropped out of college but am interested in a career in data analysis. Now I am self-studying approximately 10 hours per day.  Browsing through job postings on Linkedin has allowed me to compose a rough curriculum.  It would be of great help to me if you would either add a subject I have omitted or eliminate a subject that is not necessary for success in the market place.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Curriculum (in 3-subject groupings):&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Group 1&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Single-variable calculus&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Intro to python&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;SQL&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Group 2&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Multi-variable calculus/linear algebra&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Discrete math&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data structures and algorithms&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Group 3&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculus-based statistics and probability&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hadoop stack&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Differential equations&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Group 4&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Statistical learning/predictive modelling&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Python data analysis techniques/Statistical programming in R&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fundamentals of machine learning&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;All the while I plan to practice using any data sets I can find online.  Will this be sufficient to land a job in data analysis?  Of course I plan to learn far more than just this, but is this foundation solid enough to land an entry level data engineering/science position?&lt;/p&gt;&#xA;" OwnerUserId="9608" LastEditorUserId="182" LastEditDate="2015-05-14T05:45:48.493" LastActivityDate="2015-05-14T05:45:48.493" Title="Preparation for Career in Data Analysis Without College" Tags="|career|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5784" PostTypeId="2" ParentId="5783" CreationDate="2015-05-13T15:33:22.677" Score="3" Body="&lt;p&gt;At least based on what I and other data analysts/scientists do in my company, your technical topics list seems sufficient. But I would also add:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Visualization (ggplot2 in R, matplotlib in Python, d3.js for really cool stuff)&lt;/li&gt;&#xA;&lt;li&gt;Design of experiments&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Communication skills are also quite important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more inspiration, here's a good &quot;curriculum&quot; represented as a metro map: &lt;a href=&quot;http://nirvacana.com/thoughts/becoming-a-data-scientist/&quot; rel=&quot;nofollow&quot;&gt;http://nirvacana.com/thoughts/becoming-a-data-scientist/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me also recommend building up a portfolio of data science projects. This could consist of your analyses of data sets you find online (e.g. on &lt;a href=&quot;https://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow&quot;&gt;UCI&lt;/a&gt;), Kaggle competitions, or class projects (e.g. via Udacity or Coursera). That way, you can give direct proof of your technical skills, your communication in the form of reports or graphics, and your ability to extract insight.&lt;/p&gt;&#xA;" OwnerUserId="9576" LastActivityDate="2015-05-13T15:33:22.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5785" PostTypeId="2" ParentId="5775" CreationDate="2015-05-13T16:20:31.883" Score="1" Body="&lt;p&gt;Looks like the common approaches to multi-class classification actually solve this challenge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Building individual Naive Bayes Classifiers with only the training data for a single label is insufficient - we must also the include data from other labels as &quot;everything else&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://datascience.stackexchange.com/questions/2384/text-classification-problem-what-is-the-right-approach&quot;&gt;Text-Classification-Problem, what is the right approach?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9591" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-13T16:20:31.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5788" PostTypeId="2" ParentId="5751" CreationDate="2015-05-13T18:40:46.997" Score="3" Body="&lt;p&gt;If you are looking to calculate the relative importance of a word in this scenario, you might consider doing an inverse document frequency score on a quarterly basis:&#xA; &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have inverse frequency scores for all the words in the corpus from quarter to quarter, you can normalize by the range of all scores, and then do a quarter to quarter comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying to do a prediction, historical data from previous years would be very useful.  You can find trends over the course of a year by doing a regression on historical data and normalize to the predicted document output for that year.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="609" LastEditDate="2015-05-14T05:29:08.063" LastActivityDate="2015-05-14T05:29:08.063" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5789" PostTypeId="1" AcceptedAnswerId="5790" CreationDate="2015-05-13T21:01:03.070" Score="11" ViewCount="21237" Body="&lt;p&gt;I recently discovered a &lt;a href=&quot;http://www.r-bloggers.com/analyze-linkedin-with-r/&quot;&gt;new R package&lt;/a&gt; for connecting to the LinkedIn API. Unfortunately the LinkedIn API seems pretty limited to begin with; for example, you can only get basic data on companies, and this is detached from data on individuals. I'd like to get data on all employees of a given company, which you can do &lt;a href=&quot;https://www.linkedin.com/vsearch/p?keywords=stack%20exchange&amp;amp;f_CC=974353&amp;amp;sb=People%20who%20work%20at%20Stack%20Exchange&amp;amp;trk=tyah&amp;amp;trkInfo=clickedVertical%3Asuggestion%2Cidx%3A1-1-1%2CtarId%3A1431584515143%2Ctas%3Astack%20exchange&quot;&gt;manually on the site&lt;/a&gt; but is not possible through the API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://import.io/&quot;&gt;import.io&lt;/a&gt; would be perfect if it &lt;a href=&quot;http://blog.import.io/post/tips-tricks&quot;&gt;recognised the LinkedIn pagination&lt;/a&gt; (see end of page).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know any web scraping tools or techniques applicable to the current format of the LinkedIn site, or ways of bending the API to carry out more flexible analysis? Preferably in R or web based, but certainly open to other approaches.&lt;/p&gt;&#xA;" OwnerUserId="378" LastEditorUserId="11097" LastEditDate="2015-11-19T11:17:35.267" LastActivityDate="2020-08-16T16:53:45.230" Title="LinkedIn web scraping" Tags="|data-mining|social-network-analysis|crawling|scraping|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5790" PostTypeId="2" ParentId="5789" CreationDate="2015-05-13T23:52:57.347" Score="10" Body="&lt;p&gt;&lt;a href=&quot;http://www.crummy.com/software/BeautifulSoup/bs4/doc/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Beautiful Soup&lt;/a&gt; is specifically designed for web crawling and scraping, but is written for python and not R&lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="98307" LastEditDate="2020-08-16T16:53:45.230" LastActivityDate="2020-08-16T16:53:45.230" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5791" PostTypeId="1" CreationDate="2015-05-14T01:57:48.797" Score="2" ViewCount="107" Body="&lt;p&gt;I have a data set of tweets regarding vaccines.  They have been collected from an API because they have keywords like &quot;flu, measles, MMR, vaccine&quot; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to find tweets specifically about measles and the outbreak that occurred in California this past February.  It isn't enough to search the data set for words like &quot;California&quot; and &quot;Measles&quot; because tweets like &quot;MMR vaccination rates in Palo Alto on the rise&quot; are about measles and California, but wont be captured by a naive search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any unsupervised algorithms that could help me out?&lt;/p&gt;&#xA;" OwnerUserId="9618" LastActivityDate="2015-05-15T00:32:05.850" Title="What kind of machine learning algorithm can I use?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5792" PostTypeId="2" ParentId="5751" CreationDate="2015-05-14T05:33:12.030" Score="6" Body="&lt;p&gt;The approach suggested by &lt;a href=&quot;https://datascience.stackexchange.com/users/8041/j-a-gartner&quot;&gt;j.a.gartner&lt;/a&gt; is good if you want to analyze the quarter-to-quarter change in frequency for a given term, considering &lt;strong&gt;&lt;em&gt;only that term in isolation&lt;/em&gt;&lt;/strong&gt;.  What it doesn't do is evaluate the relative frequency compared to all other terms in the corpus.  For this you could compare frequency rank for all terms quarter-to-quarter. This is the analysis used in Zipf's Law.  Another advantage of this analysis is that you can test whether the generating process is or is not stationary (e.g. governed by same distribution quarter to quarter).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of comparing frequency rank is that it doesn't depend on the relative size of the corpus for each quarter (as long as they are all &quot;large&quot;).  In simple language, the frequency rank of the word &quot;Christmas&quot; in the 4th quarter will probably be the same, regardless of whether the (US English news) corpus is 10,000 articles or 1,000,000 articles.&lt;/p&gt;&#xA;" OwnerUserId="609" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-14T05:33:12.030" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5793" PostTypeId="1" CreationDate="2015-05-14T06:46:54.377" Score="4" ViewCount="1266" Body="&lt;p&gt;I'm not quite sure what &quot;latent&quot; refers to in this context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://ijcai.org/papers07/Papers/IJCAI07-259.pdf&quot; rel=&quot;nofollow&quot;&gt;Computing Semantic Relatedness using&#xA;Wikipedia-based Explicit Semantic Analysis&lt;/a&gt; they say &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;''Our semantic analysis is explicit in the sense that we manipulate &#xA; manifest concepts grounded in human cognition, rather than &#xA; 'latent concepts' used by Latent Semantic Analysis''.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does that mean in simple terms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a look at the related Wikipedia articles, ( &lt;code&gt;Latent&lt;/code&gt; (wikipedia.org/wiki/Latent_semantic_analysis), &lt;code&gt;Explicit&lt;/code&gt; wikipedia.org/wiki/Explicit_semantic_analysis) ), and I wasn't able to make heads or tails of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps someone with a better appreciate of the nuance involved here might be able to provide me with a clear indication of the similarities and differences, the pros and cons between these two methods for accessing document/text fragment relatedness to a particular concept.  &lt;/p&gt;&#xA;" OwnerUserId="8285" LastActivityDate="2015-05-14T13:41:48.987" Title="What is the difference between Latent and Explicit Semantic Analysis" Tags="|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5794" PostTypeId="2" ParentId="5793" CreationDate="2015-05-14T13:41:48.987" Score="4" Body="&lt;p&gt;The difference is that with ESA, the concepts are already known and labeled (hence, &quot;manifest concepts&quot;), whereas in LSA the concepts are latent (they are undefined and need to be discovered).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the following statement from the &lt;a href=&quot;http://en.wikipedia.org/wiki/Explicit_semantic_analysis&quot; rel=&quot;nofollow&quot;&gt;ESA Wikipedia page&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;The name &quot;explicit semantic analysis&quot; contrasts with latent semantic analysis (LSA), because the use of a knowledge base makes it possible to assign human-readable labels to the concepts that make up the vector space.[3][1]&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, ESA uses prior knowledge of relationships between words and concepts (as well as labels for those concepts), whereas LSA does not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References cited by Wikipedia for the quote above:&#xA;&lt;code&gt;&#xA;[1] Ofer Egozi, Shaul Markovitch and Evgeniy Gabrilovich (2011). &quot;Concept-Based Information Retrieval using Explicit Semantic Analysis&quot; (PDF). ACM Transactions on Information Systems 29 (2). Retrieved January 3, 2015.&#xA;&lt;/code&gt;&#xA;&lt;code&gt;&#xA;[3] Gabrilovich, Evgeniy; Markovitch, Shaul (2007). Computing semantic relatedness using Wikipedia-based Explicit Semantic Analysis (PDF). Proc. 20th Int'l Joint Conf. on Artificial Intelligence (IJCAI). pp. 1606–1611.&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-05-14T13:41:48.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5795" PostTypeId="2" ParentId="2368" CreationDate="2015-05-14T17:30:10.477" Score="8" Body="&lt;p&gt;Divide the data into windows and find features for those windows like autocorrelation coefficients, wavelets, etc. and use those features for learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have temperature and pressure data, break it down to individual parameters and calculate features like number of local minima in that window and others, and use these features for your model. &lt;/p&gt;&#xA;" OwnerUserId="9628" LastEditorUserId="816" LastEditDate="2015-05-18T07:34:37.760" LastActivityDate="2015-05-18T07:34:37.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5796" PostTypeId="2" ParentId="5791" CreationDate="2015-05-15T00:32:05.850" Score="2" Body="&lt;p&gt;A couple ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you have a large data set of tweets, you could try using &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_semantic_indexing&quot; rel=&quot;nofollow&quot;&gt;latent semantic indexing&lt;/a&gt; to find out which terms are semantically related based on their usage and co-occurrence. Then after transformation, you can apply some document similarity metric, e.g. cosine similarity, to find tweets most relevant to your queries, e.g. &quot;California measles&quot;.&lt;/li&gt;&#xA;&lt;li&gt;Use knowledge or lexical databases like &lt;a href=&quot;http://wiki.dbpedia.org/&quot; rel=&quot;nofollow&quot;&gt;DBPedia&lt;/a&gt; and &lt;a href=&quot;https://wordnet.princeton.edu/&quot; rel=&quot;nofollow&quot;&gt;WordNet&lt;/a&gt; to calculate semantic relatedness between your query and the text of your tweets, or to identify tweets with related concepts. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9576" LastActivityDate="2015-05-15T00:32:05.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5797" PostTypeId="1" CreationDate="2015-05-15T08:02:41.523" Score="2" ViewCount="213" Body="&lt;p&gt;I have a training data set distributed in two files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;File 1&lt;/strong&gt;: This contains actual classification for each X1. X1 is unique in this file. X1 has one-to-one relationship with X2, i.e. X2 is also unique. Y is binary.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| X1 | X2 | Y  | &#xA;| 1  | 4  | 0  | &#xA;| 3  | 5  | 1  | &#xA;...&#xA;| 8  | 9  | 1  | &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;File 2&lt;/strong&gt;: This contains the real 'observations' of the experiment. X1 can appear multiple times. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| X1 | X3 | X4 | &#xA;| 3  | 4  | 5  | &#xA;| 3  | 1  | 2  | &#xA;...&#xA;| 1  | 4  | 8  | &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I can combine the two tables to have a structure like below and use them as observations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| X1 | X2 | X3 | X4 | Y |&#xA;| 3  | 5  | 4  | 5  | 1 |&#xA;| 3  | 5  | 1  | 2  | 1 |&#xA;...&#xA;| 1  | 4  | 4  | 8  | 0 |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For test data I have similar structure, just the Y column is missing in File 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have multiple concerns here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;X1 and X2 has one-to-one dependency in the data, i.e. X1 = f(X2) and X2 = f(X1)&lt;/li&gt;&#xA;&lt;li&gt;Y = f'(X1) or f'(X2)&lt;/li&gt;&#xA;&lt;li&gt;Frequency distribution of X1,X2 and Y changes dramatically in the new joined data set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this kind of transformation of data leads to any insights?&lt;/li&gt;&#xA;&lt;li&gt;Does regression and ensemble learning techniques are capable of capturing these internal relationships?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9495" LastEditorUserId="9495" LastEditDate="2015-05-15T08:54:19.723" LastActivityDate="2015-05-15T08:54:19.723" Title="Data cleaning: Relationships between columns" Tags="|classification|dataset|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5798" PostTypeId="2" ParentId="5797" CreationDate="2015-05-15T08:42:14.577" Score="1" Body="&lt;p&gt;I see several issues in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, if there is a one-to-one relationship between X1 and X2, you should remove one of the two columns, because they are redundant. Redundant data may have a negative impact on your classification performance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, the fields X3 and X4 also seem to be totally redundant, since the value of the class label Y only depends on X1/X2. So unless the columns X3 and X4 may be interesting on their own, I don't see the point of including them into the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having dealt with these issues, and in order to obtain Y from X1/X2, there are two possibilities. If file1 contains the value of Y for any possible value of X1 in your domain, you don't need any machine learning technique. You have a perfect mapping. Otherwise, you will need to apply machine learning to find a function that &quot;fills the gaps&quot;. Depending on the nature of the Y variable, you will need to use a regression (if Y is a real number) or classification (if Y is a discrete variable). &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-05-15T08:42:14.577" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5799" PostTypeId="1" AcceptedAnswerId="6127" CreationDate="2015-05-15T08:48:02.190" Score="4" ViewCount="181" Body="&lt;p&gt;I found in many sources that Hidden Markov Models are linear-chain networks(e.g. in Predicting Structured Data book by MIT). However, as I understand it, HMMs can have any edges in its graph. Even simple example of HMM in wikipedia has non-linear graph: &lt;img src=&quot;https://i.stack.imgur.com/JU1YV.png&quot; alt=&quot;enter image description here&quot;&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the question is: what is the formal definition linear-chain structure and in which case forward-backward and Viterbi algorithms can give precise results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also taken into consideration this picture, taken from CRF tutorial, which says, that linear-chain CRF is &quot;generative-discriminative pair&quot; to HMM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/9CA8H.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="9637" LastEditorUserId="9637" LastEditDate="2015-05-15T13:58:00.220" LastActivityDate="2015-06-15T20:36:40.303" Title="Why are HMMs called linear-chain?" Tags="|machine-learning|classification|algorithms|graphs|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5800" PostTypeId="2" ParentId="5761" CreationDate="2015-05-15T10:00:06.307" Score="-1" Body="&lt;p&gt;A good reference can be found there &lt;a href=&quot;http://scikit-learn.org/stable/modules/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="9575" LastActivityDate="2015-05-15T10:00:06.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5801" PostTypeId="1" CreationDate="2015-05-15T13:51:50.087" Score="3" ViewCount="76" Body="&lt;p&gt;HI I am currently trying to apply various algorithms to a classification problem to assess which could be better and then try to fine tune the bests of the first approach. I am a beginner so I use Weka for now. I have basic ML concept understanding but am not in the details of algorithms yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I observed that on my problem, RBF networks performed vastly worse than IBK and other K methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I read about RBF networks, &quot;it implements a normalized Gaussian radial basisbasis function network. It uses the k-means clustering algorithm to provide the basis functions and learns either a logistic regression (discrete class problems) or linear regression (numeric class problems) on top of that. Symmetric multivariate Gaussians are fit to the data from each cluster. If the class is nominal it uses the given number of clusters per class.It standardizes all numeric attributes to zero mean and unit variance.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, it also use k means to classify at first. But for some reason, I get the worst results with it using my metrics (ROC), while K methods are among the bests. Can I deduce from that fact something important about my data, like the fact that it has not a gaussian distribution, or is not fitted for logistic regression, or whatever I can't figure out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also observed that random forests get similar results to K methods, and that adding a filter to reduce dimensionality improved these RF, random projection being better than PCA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this last point means that there is much randomness in my data so random dimension reduction is better than &quot;ruled&quot; dimension reduction like PCA? What can I deduce from the fact that RF perform equally to K methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel there is some signification here, but I am not skilled enough to understand what, and I would be very glad for any insights. Thanks by advance.&lt;/p&gt;&#xA;" OwnerUserId="9644" LastEditorUserId="1330" LastEditDate="2022-02-03T14:55:52.957" LastActivityDate="2022-02-03T14:55:52.957" Title="Various algorithms performance in a problem and what can be deduced about data and problem?" Tags="|machine-learning|random-forest|k-means|dimensionality-reduction|weka|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5802" PostTypeId="1" AcceptedAnswerId="5805" CreationDate="2015-05-15T14:14:20.280" Score="5" ViewCount="3225" Body="&lt;p&gt;I have location data of taxis moving around the city sourced from: &lt;a href=&quot;http://research.microsoft.com/apps/pubs/?id=152883&quot; rel=&quot;nofollow noreferrer&quot;&gt;Microsoft Research&lt;/a&gt;&lt;br&gt;&#xA;Overall it has around 17million data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have converted the data to JSON and filled up mongo. A sample looks like this:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'lat': 39.84349, 'timestamp': '2008-02-08 17:38:10', 'lon': 116.33986, 'ID': 1131}&#xA;{'lat': 39.84441, 'timestamp': '2008-02-08 17:38:15', 'lon': 116.33995, 'ID': 1131}&#xA;{'lat': 39.8453, 'timestamp': '2008-02-08 17:38:20', 'lon': 116.34004, 'ID': 1131}&#xA;{'lat': 39.84615, 'timestamp': '2008-02-08 17:38:25', 'lon': 116.34012, 'ID': 1131}&#xA;{'lat': 39.84705, 'timestamp': '2008-02-08 17:38:30', 'lon': 116.34022, 'ID': 1131}&#xA;`{'lat': 39.84891, 'timestamp': '2008-02-08 17:38:40', 'lon': 116.34039, 'ID': 1131}&#xA;{'lat': 39.85083, 'timestamp': '2008-02-08 17:38:50', 'lon': 116.3406, 'ID': 1131}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;&lt;br&gt;&#xA;It consists of a taxiID - ID field, timestamp of its latitude and longitude combination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: I want to use this data to calculate estimated time of arrival(ETA)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I am doing it a crude way by querying mongoDB with aggregation. It is totally inefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking at some sort of learning algorithm where the historical data can be used to train it. In the end, given two points, the algorithm should traverse the possible route by referring historical data and give an estimate of time.&#xA;Calculating time estimate is not a problem at all if I get the array of JSON documents between the points. But, getting those right arrays is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers in this direction will be very helpful. &lt;/p&gt;&#xA;" OwnerUserId="9003" LastEditorUserId="29169" LastEditDate="2019-06-08T03:22:35.850" LastActivityDate="2019-06-08T03:22:35.850" Title="Learning time of arrival (ETA) from historical location data of vehicle" Tags="|machine-learning|data-mining|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5803" PostTypeId="1" AcceptedAnswerId="5806" CreationDate="2015-05-15T22:18:30.280" Score="3" ViewCount="73" Body="&lt;p&gt;Apologies if this isn't the correct place to ask - I'm not sure if this fits best with Stats or Data Science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using analytics to help marketers identify attributes of their users correspond to successful conversions (such as someone buying a product, signing up for a newsletter, or subscribing to a service). Attributes could be things like which site they came from (referrer), their location, time/day of week, device type, browser, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to say (although I'm not certain it's possible) is to isolate differences in conversion rate to an individual attribute, something like, '11% of your users from Facebook converted whereas only 3% of non-Facebook users converted', which would mean that the attribute 'referrer' and the level of the attribute 'Facebook' are responsible for driving conversions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that I may have 100s of quasi-independent variables, is it even possible to isolate the effect to one variable and one level of that variable? As opposed to a combination of them that is more likely to be driving the difference? If so, what technique or conceptual paradigm do I use to identify which variable-level is responsible for the greatest lift in my dependent variable, conversion rate?&lt;/p&gt;&#xA;" OwnerUserId="7922" LastActivityDate="2015-05-16T02:36:52.970" Title="Attributing causality to single quasi-independent variable" Tags="|experiments|marketing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5804" PostTypeId="1" AcceptedAnswerId="10024" CreationDate="2015-05-16T00:15:37.807" Score="8" ViewCount="4186" Body="&lt;p&gt;I am trying to figure out how to use NLTK's cascading chunker as per &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chapter 7 of the NLTK book&lt;/a&gt;. Unfortunately, I'm running into a few issues when performing non-trivial chunking measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's start with this phrase:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&quot;adventure movies between 2000 and 2015 featuring performances by daniel craig&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to find all the relevant NPs when I use the following grammar:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;grammar = &quot;NP: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am not sure how to build nested structures with NLTK. The book gives the following format, but there are clearly a few things missing (e.g. How does one actually specify multiple rules?):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grammar = r&quot;&quot;&quot;&#xA;  NP: {&amp;lt;DT|JJ|NN.*&amp;gt;+}          # Chunk sequences of DT, JJ, NN&#xA;  PP: {&amp;lt;IN&amp;gt;&amp;lt;NP&amp;gt;}               # Chunk prepositions followed by NP&#xA;  VP: {&amp;lt;VB.*&amp;gt;&amp;lt;NP|PP|CLAUSE&amp;gt;+$} # Chunk verbs and their arguments&#xA;  CLAUSE: {&amp;lt;NP&amp;gt;&amp;lt;VP&amp;gt;}           # Chunk NP, VP&#xA;  &quot;&quot;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my case, I'd like to do something like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grammar = &quot;MEDIA: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;           RELATION: {&amp;lt;V.*&amp;gt;}{&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;           ENTITY: {&amp;lt;NN.*&amp;gt;}&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It occurs to me that a CFG might be a better fit for this, but I only became aware of NLTK's support for this function about 5 minutes ago (from &lt;a href=&quot;https://stackoverflow.com/questions/14692489/chunking-with-nltk&quot;&gt;this question&lt;/a&gt;), and it does not appear that much documentation for the feature exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, assuming that I'd like to use a cascaded chunker for my task, what syntax would I need to use? Additionally, is it possible for me to specify specific words (e.g. &quot;directed&quot; or &quot;acted&quot;) when using a chunker?&lt;/p&gt;&#xA;" OwnerUserId="9511" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-29T18:37:18.537" Title="Complex Chunking with NLTK" Tags="|python|nlp|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5805" PostTypeId="2" ParentId="5802" CreationDate="2015-05-16T00:28:32.713" Score="4" Body="&lt;p&gt;Based on what I figured out from your problem:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;1&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can easily convert your data to a graph using &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;Networkx&lt;/a&gt;, &lt;a href=&quot;http://igraph.org/&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; or any other tool/library/software. Then what you need is a &lt;a href=&quot;http://en.wikipedia.org/wiki/Shortest_path_problem&quot; rel=&quot;nofollow&quot;&gt;Shortest Path Algorithm&lt;/a&gt; (&lt;a href=&quot;http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm&quot; rel=&quot;nofollow&quot;&gt;Dijkstra&lt;/a&gt; is widely used and implemented in all graph/network analysis softwares). Once you created the graph you can simply calculate the average estimated time.&lt;/li&gt;&#xA;&lt;li&gt;For turning the problem into a &lt;em&gt;Learning Problem&lt;/em&gt;, you can use historical time estimations for different paths and assign a weight to an edge proportional to the property of that edge (e.g. traffic jam probability, time conditions) and try to predict the ETA for a new query. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;2&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can also turn it into a &lt;em&gt;Network Science Problem&lt;/em&gt; and use Graph Theoretc approaches to approach the question. You can start with statistical analysis of nodes and edges attributes e.g. passing time distribution, shortest path length distribution, probabilistic modeling of traffic jam and so on to see if some meaningful insight leads you the next step. &lt;/li&gt;&#xA;&lt;li&gt;Another idea is to use graph clustering algorithms to extract most connected parts of the town and go through the analysis of them i.e. calculate the ETA for different parts instead of whole the data and assign the estimated time to the members of corresponding cluster and reduce the computational complexty if your algorithm.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;3&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The last but not least is having a look at &lt;a href=&quot;https://www.arangodb.com/&quot; rel=&quot;nofollow&quot;&gt;ArangoDB&lt;/a&gt;. It's a new database model which is based on graphs and you can run queries on millions of edges in an amazing speed! all what you need is a bit of javascript knowledge and even if you don't have it you can use AQL language designed for arangoDB. The interesting point is that it uses JSON files as the standard data format so you are already half way through ;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope i could help :)&#xA;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-16T00:28:32.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5806" PostTypeId="2" ParentId="5803" CreationDate="2015-05-16T02:36:52.970" Score="1" Body="&lt;p&gt;I would suggest you to consider either direct &lt;em&gt;dimensionality reduction&lt;/em&gt; approach. Check &lt;a href=&quot;https://datascience.stackexchange.com/a/4946/2452&quot;&gt;my relevant answer&lt;/a&gt; on this site. Another valid option is to use &lt;em&gt;latent variable modeling&lt;/em&gt;, for example, &lt;em&gt;structural equation modeling&lt;/em&gt;. You can start with relevant articles on Wikipedia (&lt;a href=&quot;http://en.wikipedia.org/wiki/Latent_variable_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Structural_equation_modeling&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, correspondingly) and then, as needed, read more specialized or more practical articles, papers and books.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-16T02:36:52.970" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5807" PostTypeId="2" ParentId="744" CreationDate="2015-05-16T05:36:33.213" Score="9" Body="&lt;p&gt;I would like to add one more dimension to the answers given above. Usually we use cosine similarity with large text, because using distance matrix on paragraphs of data is not recommended. And also if you intend your cluster to be broad you tend to go with cosine similarity as it captures similarity overall. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if you have texts which are two or three words long at max I feel using cosine similarity does not achieve the precision as achieved by distance metric. &lt;/p&gt;&#xA;" OwnerUserId="9654" LastActivityDate="2015-05-16T05:36:33.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5808" PostTypeId="1" CreationDate="2015-05-16T16:58:08.090" Score="3" ViewCount="70" Body="&lt;p&gt;I prefer this model in R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are capturing sales data by time series (Month by month). Some of items have commissions and some have Discounts and others have both commissions and discounts. Is it Commissions or Discounts or commissions + Discounts have impact on my sales growth? Or is it my sales are growing because of those commissions or discounts or discounts +commissions Can you suggest me best model to solve my use case? I am thinking multiple regression. But I want to double check with experts like you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your all your help&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Data set: (5 variables)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year-Month -Product -Sales  -Commission -Discounts&#xA;2013-01 Milk    300 No  Yes&#xA;2013-02 Milk    400 No  Yes&#xA;2013-03 Milk    200 No  Yes&#xA;2013-04 Milk    150 No  Yes&#xA;2013-05 Milk    500 No  Yes&#xA;2013-01 Bread   800 Yes No&#xA;2013-02 Bread   879 Yes No&#xA;2013-03 Bread   790 Yes No&#xA;2013-04 Bread   459 Yes No&#xA;2013-05 Bread   600 Yes No&#xA;2013-01 Cheese  400 Yes Yes&#xA;2013-02 Cheese  350 Yes Yes&#xA;2013-03 Cheese  600 Yes Yes&#xA;2013-04 Cheese  590 Yes Yes&#xA;2013-05 Cheese  720 Yes Yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9663" LastActivityDate="2015-05-16T19:01:43.953" Title="Is it my Sales growth is depend on Commisions/Discounts?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5809" PostTypeId="1" AcceptedAnswerId="5814" CreationDate="2015-05-16T17:37:53.163" Score="5" ViewCount="2128" Body="&lt;p&gt;I have a data frame (a csv file) with dimensions 100x6 and I need only the columns c(&quot;X1&quot;, &quot;X2&quot;, &quot;X4&quot;) and the rows in which the value of &quot;X1&quot; is greater than 30. So I did:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  data_frame &amp;lt;- read.csv (&quot;data_frame&quot;)&#xA;  data_frame [c(&quot;X1&quot;, &quot;X2&quot;, &quot;X4&quot;)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The column subset problem is solved but now I need to subset rows from data_frame [c(&quot;X1&quot;, &quot;X2&quot;, &quot;X4&quot;)] where the values of &quot;X1&quot; is greater than 30. I tried:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  data_frame [c(&quot;X1&quot; &amp;gt; 30), c(&quot;X1&quot;, &quot;X2&quot;, &quot;X4&quot;)] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it returned the same data frame as data_frame [c(&quot;X1&quot;, &quot;X2&quot;, &quot;X4&quot;)].&#xA;Also tried using the function subset() with the same approach but got the same results.&lt;/p&gt;&#xA;" OwnerUserId="9662" LastEditorUserId="471" LastEditDate="2015-05-17T22:13:24.213" LastActivityDate="2018-11-10T14:38:37.487" Title="How to subset rows from a data frame with comparison operators in R" Tags="|r|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5810" PostTypeId="1" CreationDate="2015-05-16T18:09:15.630" Score="3" ViewCount="1343" Body="&lt;p&gt;I have recently completed an MSc in Control Systems from a top university. It seems to me that control theory must have an application within data science. I would like to apply my degree within this domain, but I want to be sure that it is relevant to the role of a data scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The topics which I have particular interest and experience in are State Space Control, Systems Identification, Model Predictive Control and Optimal Control. I imagine that effective management of any large dataset must involve modelling of the system in terms of transfer functions/state space models (based on large sets of historical input/output data). These models could then be used to predict the evolution of a market/variable over time, and therefore optimise a given cost function such as profit, risk etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this kind of role exists within data science/ other areas, can you please give me more information/ ideas of job roles/ industries to research.&lt;/p&gt;&#xA;" OwnerUserId="9665" LastActivityDate="2015-05-17T08:30:38.917" Title="Application of Control Theory in Data Science" Tags="|bigdata|optimization|finance|" AnswerCount="1" CommentCount="1" ClosedDate="2015-05-18T07:34:21.380" ContentLicense="CC BY-SA 3.0" />
  <row Id="5811" PostTypeId="2" ParentId="5503" CreationDate="2015-05-16T18:52:10.650" Score="3" Body="&lt;p&gt;You might be interested in this paper that explores a few of the questions you are asking: &lt;a href=&quot;http://arxiv.org/pdf/1312.6184.pdf&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/pdf/1312.6184.pdf&lt;/a&gt;. It is aptly titled: &quot;Do Deep Networks Really Need to Be Deep?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crux of the matter is that deep networks allow for a LOT of non-linearity in the data that is being described. For CIFAR-10, I suspect that something similar to what works well for MNIST will work well. Yan LeCunn's page about MNIST results gives a lot of clues about models with few parameters: &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot; rel=&quot;nofollow&quot;&gt;http://yann.lecun.com/exdb/mnist/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The double-problem of image classification is that in addition to the classification you need to extract features somehow. Often the parameters in deep networks go towards feature extraction rather than the classification itself, which is usually a softmax multi-class logistic regression classifier. &lt;/p&gt;&#xA;" OwnerUserId="9667" LastActivityDate="2015-05-16T18:52:10.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5812" PostTypeId="2" ParentId="5808" CreationDate="2015-05-16T19:01:43.953" Score="3" Body="&lt;p&gt;Multiple regression sounds appropriate in this case. Real question is &lt;strong&gt;what variables to use&lt;/strong&gt;. Definitely, you should include indicator variables &lt;code&gt;commisions&lt;/code&gt; and &lt;code&gt;discounts&lt;/code&gt; (and possibly their combination - &lt;code&gt;commisions_and_discounts&lt;/code&gt; - as a separate variable). Date and time information may play important role as well, though treating them is a little bit harder. At the very least, it's worth to include &lt;code&gt;year&lt;/code&gt; (because there may be a global trend), &lt;code&gt;month&lt;/code&gt; and &lt;code&gt;day_of_week&lt;/code&gt; (many patterns are repeated periodically). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Product type is a little bit more complicated. Obviously, some products will always have higher sales than others, so first idea is to include a set of &lt;code&gt;product_type_x&lt;/code&gt; dummy variables. But product type may affect sales not additively, but instead multiplicatively, i.e. not as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sales ~ beta_0 + beta_1*product_type_x + ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but instead as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sales ~ beta_1 * prodcut_type_x * (beta_0 + ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case better solution would be to create &lt;strong&gt;separate regression models&lt;/strong&gt; for each product type. &lt;/p&gt;&#xA;" OwnerUserId="1279" LastActivityDate="2015-05-16T19:01:43.953" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5813" PostTypeId="2" ParentId="1074" CreationDate="2015-05-16T19:16:26.623" Score="5" Body="&lt;p&gt;The offset parameter is sometimes called &quot;bias&quot; in classification tasks, and its intuitive understanding doesn't have to do with what kind of kernel is used. It is basically used to compensate for feature vectors that are not centered around 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will try to intuitively explain what the bias does with a toy example. Let's say you have some feature vector &lt;code&gt;x&lt;/code&gt; who's parameters are always negative. The set of weights &lt;code&gt;w&lt;/code&gt; you use in your SVM (let's say linear, just for clarity) will perhaps transform the features into the range [0, 1] - so they will always be negative. But, those elements that belong to class 1 fall in the range [0, 0.5], and the ones that are from class 2 fall into the range [0.5, 1]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To classify into a class, the SVM uses 0 as the threshold breakpoint - if greater than 0, it is an element of class 1, if less than 0, it is an element of class -1. But in this case, all the elements will be classified into class 2. However, with a bias of 0.5 (in this linear case), they will be classified correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This geometric interpretation doesn't quite work for more complex kernels, but the idea is the same: the bias term attempts to compensate for features that are not centered around zero. In practice, if the features are centered around zero the bias term isn't always needed. To get around the bias issue, you can either: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;center your features and forget about the bias term (doesn't always work)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;augment your feature data to include a 1 at the beginning. i.e.: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[feat1 feat2 feat3] --&amp;gt; [feat1 feat2 feat3 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for all of your feature vectors. Then the bias term will just be learned as another SVM weight parameter. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Learn your weights, then calculate the bias based on the optimal prediction rule:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/0rOre.png&quot; alt=&quot;&amp;quot;prediction rule&amp;quot; for SVMs&quot;&gt;&#xA;That is, given all the support vectors $S$, the learned weights $\alpha_i$, the training class $y^{(i)}$, training features $x^{(i)}$, and features-to-use-for-prediction $x$, the predicted class will be $y(x)$. The optimal $w_0$ or bias will then be calculated as the average distance between the correct and calculated predicted class: &#xA;&lt;img src=&quot;https://i.stack.imgur.com/kyukg.png&quot; alt=&quot;optimal bias&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps. I think the easiest thing is to center your features ;)&lt;/p&gt;&#xA;" OwnerUserId="9667" LastActivityDate="2015-05-16T19:16:26.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5814" PostTypeId="2" ParentId="5809" CreationDate="2015-05-16T20:07:57.017" Score="5" Body="&lt;p&gt;You want&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_frame[data_frame$X1 &amp;gt; 30, c(&quot;X1&quot;,&quot;X2&quot;,&quot;X4&quot;)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that will just print it, you probably want to update &lt;code&gt;data_frame&lt;/code&gt; or store it in something else:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_frame = data_frame[data_frame$X1 &amp;gt; 30, c(&quot;X1&quot;,&quot;X2&quot;,&quot;X4&quot;)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;also you probably want to try asking this on StackOverflow, or reading a bit more basic R documentation because it should be well covered. Its a bit simple to be &quot;data science&quot;.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2015-05-17T16:12:45.037" LastActivityDate="2015-05-17T16:12:45.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5815" PostTypeId="2" ParentId="5810" CreationDate="2015-05-17T08:30:38.917" Score="2" Body="&lt;p&gt;Have you tried the &lt;strong&gt;Internet search&lt;/strong&gt;? The results should be able to answer most, if not all, of your questions. The topics of your interest sound like rather general or high-level. I'm sure that they can, in one form or another, be applied in the &lt;em&gt;data science context&lt;/em&gt;. In my opinion, those topics are more related to &lt;em&gt;operations research (OR)&lt;/em&gt;, therefore, I would recommend you to perform some research on the Internet on the &lt;em&gt;intersections&lt;/em&gt; between control systems (theory) and data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, first thing that comes to my mind is that the most likely candidate for use of control theory concepts and methods in data science context would be &lt;strong&gt;distributed systems&lt;/strong&gt; and &lt;strong&gt;algorithms&lt;/strong&gt; for &lt;em&gt;data analysis&lt;/em&gt;, such as &lt;em&gt;MapReduce&lt;/em&gt; (&lt;em&gt;Hadoop&lt;/em&gt;, etc.), as well as other &lt;strong&gt;parallel processing systems&lt;/strong&gt;. If there exist an intersection between OR's area of optimization and control theory, then it very well could be used for &lt;strong&gt;big data algorithms optimization&lt;/strong&gt;, among other tasks.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-05-17T08:30:38.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5816" PostTypeId="2" ParentId="5318" CreationDate="2015-05-17T18:32:15.197" Score="1" Body="&lt;p&gt;I'd go for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S(object_i,object_j) = 1 - \frac{\sum_{k=1}^{N_c} |n_{object_i,cluster_k}-n_{object_j,cluster_k}|}{n_{object_i}+n_{object_j}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $N_c$ is the number of clusters (here 3), $n_{object_i,cluster_k}$ is the number of elements of object $i$ that belong to cluster $k$ (here $n_{object_2,cluster_2}=200$) and $n_{object_i}$ is the total number of elements in object $i$ (sum of $i^{th}$ row).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-17T18:32:15.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5817" PostTypeId="1" CreationDate="2015-05-17T20:12:48.760" Score="12" ViewCount="1561" Body="&lt;p&gt;I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I would say my knowledge about more complex analysis is quite limited I've done mostly all basic methods to gather a overall view of the data. This includes simple graphs which already help to identify similar patterns. I am also using SAS Enterprise 9.4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am looking for a more number based approach.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Data Set&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The (self) collected data set I am using contain around ~54.000 fares.&#xA;All fares were collected within a 60 day time window, on a daily basis (every night at 00:00).&lt;img src=&quot;https://i.stack.imgur.com/q5bcT.png&quot; alt=&quot;Collection Method&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, every fare within that time window occurs &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; times subject to the availability of the fare as well as the departure date of the flight, when it is passed by the collection date of the fare.&#xA;&lt;em&gt;(You can't collect a fare for a flight when the departure date of the flight is in the past)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The unformatted that looks basically like this: (fake data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| requestDate        | price| tripStartDeparture | tripDestinationDeparture | flightCarrier |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 14APR2015:00:00:00 | 725.32    | 16APR2015:10:50:02 | 23APR2015:21:55:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 14APR2015:00:00:00 | 966.32    | 16APR2015:13:20:02 | 23APR2015:19:00:04       | XY            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 14APR2015:00:00:00 | 915.32    | 16APR2015:13:20:02 | 23APR2015:21:55:04       | XH            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&quot;DaysBeforeDeparture&quot; is calculated via &lt;span class=&quot;math-container&quot;&gt;$I=s-c$&lt;/span&gt; where&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I &amp;amp; interval (days before departure)&lt;/li&gt;&#xA;&lt;li&gt;s &amp;amp; date of the fare (flight departure)&lt;/li&gt;&#xA;&lt;li&gt;c &amp;amp; date of which the fare was collected&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is a example of grouped data set by I (DaysBeforeDep.) (fake data!):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----------------+------------------+------------------+------------------+------------------+&#xA;| DaysBefDeparture | AVG_of_sale | MIN_of_sale | MAX_of_sale | operatingCarrier |&#xA;+-----------------+------------------+------------------+------------------+------------------+&#xA;| 0               | 880.68           | 477.99           | 2,245.23         | DL           |&#xA;+-----------------+------------------+------------------+------------------+------------------+&#xA;| 0               | 904.89           | 477.99           | 2,534.55         | DL           |&#xA;+-----------------+------------------+------------------+------------------+------------------+&#xA;| 0               | 1,044.39         | 920.99           | 2,119.09         | LH               |&#xA;+-----------------+------------------+------------------+------------------+------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;What I came up with so far&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Looking at the line graphs I can already estimate that several lines will have a high correlation factor. Hence, I tried to use correlation analysis first on the grouped data. But is that the correct way? Basically I try now to make correlations on the averages rather then on the individual prices?&#xA;Is there an other way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am unsure which regression model fits here, as the prices do not move in any linear form and appear non-linear. Would I need to fit a model to each of price developments of an airline&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: This is a long text-wall. If I need to clarify anything let me know. I am new to this sub.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone a clue? :-)&lt;/p&gt;&#xA;" OwnerUserId="9684" LastEditorUserId="25180" LastEditDate="2019-03-31T19:49:16.047" LastActivityDate="2022-08-23T03:56:51.240" Title="Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?" Tags="|data-mining|dataset|regression|correlation|visualization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5818" PostTypeId="2" ParentId="5817" CreationDate="2015-05-18T02:32:19.643" Score="4" Body="&lt;p&gt;In addition to exploratory data analysis (EDA), both descriptive and visual, I would try to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow noreferrer&quot;&gt;time series analysis&lt;/a&gt; as a more &lt;em&gt;comprehensive&lt;/em&gt; and &lt;em&gt;sophisticated&lt;/em&gt; analysis. Specifically, I would perform &lt;em&gt;time series regression analysis&lt;/em&gt;. Time series analysis is a huge research and practice domain, so, if you're not familiar with the fundamentals, I suggest starting with the above-linked Wikipedia article, gradually searching for more specific topics and reading corresponding articles, papers and books.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since time series analysis is a &lt;em&gt;very popular&lt;/em&gt; approach, it is supported by most open source and closed source commercial &lt;em&gt;data science&lt;/em&gt; and &lt;em&gt;statistical&lt;/em&gt; &lt;strong&gt;environments (software)&lt;/strong&gt;, such as &lt;em&gt;R&lt;/em&gt;, &lt;em&gt;Python&lt;/em&gt;, &lt;em&gt;SAS&lt;/em&gt;, &lt;em&gt;SPSS&lt;/em&gt; and many others. If you want to use &lt;em&gt;R&lt;/em&gt; for this, check my answers &lt;a href=&quot;https://stats.stackexchange.com/a/131133/31372&quot;&gt;on general time series analysis&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/a/131284/31372&quot;&gt;on time series classification and clustering&lt;/a&gt;. I hope that this is helpful.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-05-18T02:32:19.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5819" PostTypeId="1" CreationDate="2015-05-18T05:59:41.207" Score="4" ViewCount="153" Body="&lt;p&gt;I have several sets of text files on hdfs that are exports from relations.  Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row uniquely.  I know through domain knowledge that the key is multi-part (e.g. reporting-date, and item number) and I can identify some columns that are clearly not in the key (e.g. revenue from sale).  What is an effective way to identify potential sets of columns that are natural keys, because they are not duplicated in the observed data?  I can get several days of logs in a few Gig, so python or sql could work.  This seems like an great application for a dictionary, but I am not sure how to approach this.     &lt;/p&gt;&#xA;" OwnerUserId="9689" LastActivityDate="2020-08-03T00:59:48.943" Title="Looking for smallest set of rows that form a natural key in a data set" Tags="|python|dataset|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5820" PostTypeId="2" ParentId="5318" CreationDate="2015-05-18T09:20:51.007" Score="3" Body="&lt;p&gt;You can consider that the clusters are defined by a multinomial distribution with 4 classes. Thus you would identify the cluster as a vector of 4 probabilities, one for each class, and the sum of probabilities being 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this form there are some tests and distance functions which can prove to be useful for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can ask questions like &quot;is the cluster 1 is different than the cluster 2?&quot;. In order to do that you can employ hypothesis testing, and specifically chi square independence test (wikipedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Test_of_independence&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). You select from your table only the first 2 columns, compute the marginal totals, compute the estimated counts as $E_{ij} = rowsum_i * colsum_j / grandtotal$. Denoting with $O_{ij}$ the cell counts you can compute a test statistic with &#xA;formula: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$chi = \sum_{i=1}^4\sum_{j=1}^2\frac{(E_{ij}-O_{ij})^2}{E_{ij}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is known to has a chi square distribution with $(rows-1)(cols-1)=3$ degrees of freedom. Thus using the cumulative distribution of chi square you can compute the p-value as $1-cdf_{\chi_{3}^2}(chi)$. Now if p-value is smaller than any critical value you choose before test (like 0.05, 0.01 or so), you have a statistical proof that the clusters have a different distribution (this follows from the fact that your procedure tests for the independence of cluster and object distributions, which with short derivation is the same as saying the clusters have the same distribution). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer to the same type of question you can also employ G-test (wikipedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/G-test&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) or Fisher exact test (wikipedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/Fisher%27s_exact_test&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that all those tests can't prove that two clusters are the same. The logic behind the tests can be formulated as &quot;we consider the distributions the same, as a natural state, accepted without other proofs, and if we see some exceptionally obvious differences, we can bring that as a proof that they are different indeed&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different approach would be to build a distance between those two distributions. The advantage of a distance function would be that it could be used for comparisons and eventually some further algorithms based on distances. The drawback of a distance function is that it gives you no hint to answer the first type of questions like &quot;are the differences between 2 distributions small enough in order to be explained by the natural noise in data?&quot;. Thus I have two candidates in mind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First distance is the chi square distance. This type of distance gives a measure to the distance between two normalized histograms (or, put it simply, between two vectors of probabilities of the same size). Thus, if we denote with $cluster_i = (n_{i1}/n_i, n_{i2}/n_i, .., n_{ik}/n_i] = (p_{i1},p_{i2},..p_{ik})$, the distance between cluster i and j would be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d(i,j) = \frac{1}{2}\sum_{k=1}^{4}\frac{(p_{ik}-p_{jk})^2}{p_{ik}-p_{jk}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This distance function has the advantage that it is symmetric regarding $i$ and $j$. To be honest I rarely used it since I did not fully understand it's meaning. In general I use another function, described below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second distance function is mutual information (see wikipedia page &lt;a href=&quot;http://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). The idea is similar with Kulback-Leibler divergence, however the KL distance is an oriented measure (measures how a distribution can be expressed through another one). The MI distance is a measure of mutual independence between two distributions. In your case the two distributions are two clusters, and the MI measures how dependent they are. I give formula just to illustrate the idea behind it. &#xA;$$MI(x,y) = \sum_{x,y}p(x,y)log(\frac{p(x,y)}{p(x)p(y)})$$&#xA;You can see from the formula that when the two variables are independent, the quantity inside the logarithm is $1$, and as a consequence, the MI is $0$. As the difference grows, the function also grows. I will not derive what p means, see what I explained already for chi square test, since is very similar. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-05-18T09:20:51.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5821" PostTypeId="1" AcceptedAnswerId="5825" CreationDate="2015-05-18T11:50:35.383" Score="6" ViewCount="4900" Body="&lt;p&gt;I have some basic features which I encoded in a &lt;strong&gt;one-hot vector&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Length of the feature vector equals to 400.&#xA;It is &lt;strong&gt;sparse&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw that conv nets is applied to a dense feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any problems to apply conv nets to a sparse feature vectors?&lt;/p&gt;&#xA;" OwnerUserId="9690" LastEditorUserId="29169" LastEditDate="2022-06-17T23:17:27.610" LastActivityDate="2022-06-17T23:17:27.610" Title="Convolutional neural network for sparse one-hot representation" Tags="|deep-learning|neural-network|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5822" PostTypeId="1" CreationDate="2015-05-18T13:07:17.983" Score="3" ViewCount="65" Body="&lt;p&gt;I'm wondering if anyone might have some novel insights as to the best way to analyze the following data.  It's a problem I've been thinking about in the back of my mind for a while, so I thought that I'd ask here. I have data that look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     day event actor recipient&#xA;995    8   128     G         J&#xA;996    8   129     G         K&#xA;997    8   130     G         B&#xA;998    8   131     B         G&#xA;999    8   132     H         G&#xA;1000   8   133     G         H&#xA;1001   8   134     E         G&#xA;1002   8   135     G         J&#xA;1003   8   136     B         H&#xA;1004   8   137     G         H&#xA;1005   8   138     G         H&#xA;1006   8   139     B         J&#xA;1007   9     1     D         J&#xA;1008   9     2     A         J&#xA;1009   9     3     A         J&#xA;1010   9     4     H         J&#xA;1011   9     5     A         J&#xA;1012   9     6     D         H&#xA;1013   9     7     A         F&#xA;1014   9     8     D         J&#xA;1015   9     9     A         H&#xA;1016   9    10     D         J&#xA;1017   9    11     A         J&#xA;1018   9    12     F         J&#xA;1019   9    13     F         J&#xA;1020   9    14     F         H&#xA;1021   9    15     F         G&#xA;1022   9    16     F         H&#xA;1023   9    17     C         F&#xA;1024   9    18     C         G&#xA;1025   9    19     D         H&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What you see here is an extract of a R dataframe.  The first column being the rownumber of the df, then the four variables.    These data start at day1 and end at day 22.  There are between 13 and 215 'events' on each day - each event is a separate behavioral event. Higher number events occur later in time than earlier numbered events.  Individuals are in the 'actor' and 'recipient' variables.   The data are available in csv format &lt;a href=&quot;https://github.com/jalapic/exampledata/blob/master/demodf.csv&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 11 individuals (A - K). One thing you'll notice is that recipients tend to be lower down the alphabet, and actors tend to be higher up the alphabet.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A key question I'm interested in working out a methodology to address is to see if the likelihood of becoming an actor increases if an individual has recently been a recipient. You can see this on line 997 that G-B and then B-G occur followed by H-G and G-H.    An individual recipient doesn't have to appear on the next line to count as having an increased likelihood of appearing as an actor - I'm interested in the decay in the probability of this occurring over events (but not continuing to the next day).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, I don't think this will be true for all individuals, so I am keen to test which individuals it is true for.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I am interested to know if an individual who has recently been a recipient but is now an actor is paired with more often an individual of a higher or lower letter than themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that these questions are clear and make sense.   I obviously don't expect a full analysis.  But I would be keen to hear of ideas for this type of analysis.  I believe that examining some Markov processes may be useful but I am interested in hearing about other ideas.&lt;/p&gt;&#xA;" OwnerUserId="9693" LastEditorUserId="8953" LastEditDate="2015-05-18T13:36:53.970" LastActivityDate="2015-05-18T13:36:53.970" Title="Identifying sequences of behavioral interactions between multiple individuals" Tags="|r|markov-process|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5823" PostTypeId="1" CreationDate="2015-05-18T15:10:05.623" Score="3" ViewCount="741" Body="&lt;p&gt;I have a set of strings, each also has soem categorical information associated with it. The categorical information isn't always great though, so I need to cluster the messages based on the text content &amp;amp; the categories. What is the best way to do this? &lt;/p&gt;&#xA;" OwnerUserId="9695" LastActivityDate="2015-05-18T19:01:52.377" Title="How do I cluster data that is a mix of text &amp; categorical data?" Tags="|clustering|text-mining|k-means|categorical-data|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-19T14:14:22.020" ContentLicense="CC BY-SA 3.0" />
  <row Id="5824" PostTypeId="2" ParentId="5735" CreationDate="2015-05-18T17:36:21.813" Score="2" Body="&lt;p&gt;Depending on your technical ability and what you're trying to do, &lt;a href=&quot;http://www.qlik.com/uk/explore/products/sense&quot; rel=&quot;nofollow&quot;&gt;Qlik Sense&lt;/a&gt; might be a good fit as it is fairly easy to use and yet powerful, and &lt;a href=&quot;http://www.qlik.com/uk/videos/how-to/sense/working-with-maps&quot; rel=&quot;nofollow&quot;&gt;comes with mapping functionality&lt;/a&gt;. If it's a commercial project though you will need to pay for (expensive) licenses, otherwise it's a free download for personal non-commercial use.&lt;/p&gt;&#xA;" OwnerUserId="9587" LastActivityDate="2015-05-18T17:36:21.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5825" PostTypeId="2" ParentId="5821" CreationDate="2015-05-18T17:50:19.717" Score="11" Body="&lt;p&gt;I would not apply convolutional neural networks to your problem (at least from what I can gather from the description).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern &lt;code&gt;1 0 1&lt;/code&gt; seen in features 9,10,11 is similar in some way to the same pattern seen in features 15,16,17. Having this assumption built in to the model allows you to train a network with far fewer free parameters when dealing with e.g. image data, where this is a key property of data captured by scanners and cameras. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With one-hot encoding of features, you assign a feature vector index from a value or category essentially at random (via some hashing function). There is no meaning to translations between indices of the feature vectors. The patterns &lt;code&gt;0 0 1 0 1 0 0&lt;/code&gt;  and &lt;code&gt;0 0 0 1 0 1 0&lt;/code&gt; can represent entirely different things, and any associations between them are purely by chance. You can treat a sparse one-hot encoding as an image if you wish, but there is no good reason to do so, and models that assume translations can be made whilst preserving meaning will not do well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For such a small sparse feature vector, assuming you want to try a neural network model, use a simple fully-connected network.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-05-20T06:07:11.297" LastActivityDate="2015-05-20T06:07:11.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5826" PostTypeId="1" CreationDate="2015-05-18T18:55:59.763" Score="1" ViewCount="539" Body="&lt;p&gt;I am looking to change careers and would appreciate some advice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an undergraduate degree in English Literature and a JD. Needless to say, these were not the best decisions and I would like to change my career. I have always enjoyed math and science, and after months of research and self study, I have decided that I would like to pursue statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is basically this : would it be better to get another Bach. Degree in statistics or should I take calculus 1-3, linear algebra, probably and statistics and some computer science courses at a community college then try to get into a grad program instead? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am afraid that without the bachelors degree I will not have the required knowledge for a masters even after taking the courses at my local c.c. However, another bachelors degree may be a worthless waste of time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please share your thoughts. Also, sorry if this is not the right place for this question. &lt;/p&gt;&#xA;" OwnerUserId="9699" LastEditorUserId="75" LastEditDate="2015-05-26T06:35:58.427" LastActivityDate="2015-05-26T06:35:58.427" Title="Master degree in Data Science" Tags="|education|career|" AnswerCount="2" CommentCount="1" ClosedDate="2015-05-19T14:14:10.683" ContentLicense="CC BY-SA 3.0" />
  <row Id="5827" PostTypeId="2" ParentId="5823" CreationDate="2015-05-18T19:01:52.377" Score="6" Body="&lt;p&gt;First and foremost, there is no &lt;em&gt;best&lt;/em&gt; way to do this task in general. It will require some creativity and problem-solving&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is one potential idea. Clustering involves some kind of similarity metric. You can compute a similarity between text documents (after some term weighting like TF-IDF you could use the cosine similarity for example). You can also compute a similarity between objects with categorical data. Matching coefficients are good option for this. For example, &lt;a href=&quot;http://en.wikipedia.org/wiki/Simple_matching_coefficient&quot; rel=&quot;noreferrer&quot;&gt;http://en.wikipedia.org/wiki/Simple_matching_coefficient&lt;/a&gt;  is about as basic as it gets, but there are many other measures of similarity for categorical data that you can find with a simple google search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I have one similarity matrix &lt;strong&gt;A&lt;/strong&gt; that gives the similarity based on the text, and one similarity matrix &lt;strong&gt;B&lt;/strong&gt; that gives the similarity based on the categorical attributes, then I could take some weighted combination of &lt;strong&gt;A&lt;/strong&gt; and &lt;strong&gt;B&lt;/strong&gt; and use it as an overall similarity matrix for clustering. Perhaps a convex combination would work well and give you some flexibility to choose which measure of similarity you want to weight more heavily (since it sounds like the categorical information might not be as good):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;choose some parameter $0\leq \alpha\leq 1$ and compute an overall similarity matrix &lt;strong&gt;S&lt;/strong&gt; as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\textbf{S} = \alpha \textbf{A} + (1-\alpha)\textbf{B}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you could use any number of clustering algorithms using the similarity information. For example, $k$-means should work just fine on a similarity matrix as input. Any type of graph/spectral clustering algorithm is also well suited for this type of input.&lt;/p&gt;&#xA;" OwnerUserId="9700" LastActivityDate="2015-05-18T19:01:52.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5828" PostTypeId="2" ParentId="5826" CreationDate="2015-05-18T19:07:45.150" Score="1" Body="&lt;p&gt;Graduate degrees in data science or business analytics provide a fast-track toward a job these days, even more so than a traditional statistics degree. Check out the graduate programs and what they require in terms of prerequisites. Some may not even require Calculus sequence, only a good foundation of stat and linear algebra. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bachelor's degree is necessarily going to waste your days with core courses like British Literature (no offense to english majors, but given your goals that time would be wasted).&lt;/p&gt;&#xA;" OwnerUserId="9700" LastActivityDate="2015-05-18T19:07:45.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5830" PostTypeId="1" CreationDate="2015-05-19T02:09:13.700" Score="2" ViewCount="518" Body="&lt;p&gt;I was informed of 5 java NLP libraries.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Apache cTAKES™&lt;/li&gt;&#xA;&lt;li&gt;MetaMap&lt;/li&gt;&#xA;&lt;li&gt;LexEVS (&lt;a href=&quot;http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/current/web/download.html&quot; rel=&quot;nofollow&quot;&gt;http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/current/web/download.html&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Apache OpenNLP&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also plan to parallelize an NLP library via map-reduce with hadoop.&#xA;However, I'm new to natural language processing, so I don't know how to approach the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to download a set of clinical trials on cancer from www.clinicaltrials.gov and parse eligibility criteria (both inclusion and exclusion criteria), identify the ECOG scores and annotate each CTA with the allowed ECOG scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in the following document, if ECOG is specified in inclusion criteria, it is not negated. If ECOG is specified in exclusion criteria, it is negated. If both exclusion criteria and inclusion criteria are not mentioned, then ECOG is not negated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Eligibility criteria of CTA &quot;NCT01572038&quot;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Inclusion Criteria:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Patients must have histological proof of a primary non-small cell lung cancer&#xA;  (bronchoalveolar carcinomas presenting as a discrete solitary radiological mass or&#xA;  nodule are eligible)&lt;/li&gt;&#xA;  &lt;li&gt;Patients must be classified post-operatively as stage IB, II or IIIA on the basis of&#xA;  pathologic criteria&lt;/li&gt;&#xA;  &lt;li&gt;At the time of resection a complete mediastinal lymph node resection or at least&#xA;  lymph node sampling should have been attempted; if a complete mediastinal lymph node&#xA;  resection or lymph node sampling was not undertaken, any mediastinal lymph node which&#xA;  measured 1.5 cm or more on the pre-surgical computed tomography (CT)/magnetic&#xA;  resonance imaging (MRI) scan or any area of increased uptake in the mediastinum on a&#xA;  pre-surgical positron emission tomography (PET) scan must have been biopsied; note: a&#xA;  pre-surgical PET scan is not mandatory&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;The nodal tissue must be labelled according to the recommendations of the&#xA;  American Thoracic Society; surgeons are encouraged to dissect or sample all&#xA;  accessible nodal levels; the desirable levels for biopsy are:&lt;/li&gt;&#xA;  &lt;li&gt;Right upper lobe: 4, 7, 10&lt;/li&gt;&#xA;  &lt;li&gt;Right middle lobe: 4, 7, 10&lt;/li&gt;&#xA;  &lt;li&gt;Right lower lobe: 4, 7, 9, 10&lt;/li&gt;&#xA;  &lt;li&gt;Left upper lobe: 5, 6, 7, 10&lt;/li&gt;&#xA;  &lt;li&gt;Left lower lobe: 7, 9, 10&lt;/li&gt;&#xA;  &lt;/ul&gt;&lt;/li&gt;&#xA;  &lt;li&gt;Surgery may consist of lobectomy, sleeve resection, bilobectomy or pneumonectomy as&#xA;  determined by the attending surgeon based on the intraoperative findings; patients&#xA;  who have had only segmentectomies or wedge resections are not eligible for this&#xA;  study; all gross disease must have been removed at the end of surgery; all surgical&#xA;  margins of resection must be negative for tumor&lt;/li&gt;&#xA;  &lt;li&gt;No more than 16 weeks may have elapsed between surgery and randomization; for&#xA;  patients who received post-operative adjuvant platinum-based chemotherapy, no more&#xA;  than 26 weeks may have elapsed between surgery and randomization&lt;/li&gt;&#xA;  &lt;li&gt;Patient must consent to provision of and investigator(s) must agree to submit a&#xA;  representative formalin fixed paraffin block of tumor tissue at the request of the&#xA;  Central Tumor Bank in order that the specific EGFR correlative marker assays may be&#xA;  conducted&lt;/li&gt;&#xA;  &lt;li&gt;The patient must have an Eastern Cooperative Oncology Group (ECOG) performance status&#xA;  of 0, 1 or 2&lt;/li&gt;&#xA;  &lt;li&gt;Leukocytes &amp;gt;= 3.0 x 10^9/L or &amp;gt;= 3000/ul&lt;/li&gt;&#xA;  &lt;li&gt;Absolute granulocyte count &amp;gt;= 1.5 x 10^9/L or &amp;gt;= 1,500/ul&lt;/li&gt;&#xA;  &lt;li&gt;Platelets &amp;gt;= 100 x 10^9/L or &amp;gt;= 100,000/ul&lt;/li&gt;&#xA;  &lt;li&gt;Total bilirubin within normal institutional limits&lt;/li&gt;&#xA;  &lt;li&gt;Alkaline phosphatase =&amp;lt; 2.5 x institutional upper limit of normal; if alkaline&#xA;  phosphatase is greater than the institutional upper limit of normal (UNL) but less&#xA;  than the maximum allowed, an abdominal (including liver) ultrasound, CT or MRI scan&#xA;  and a radionuclide bone scan must be performed prior to randomization to rule out&#xA;  metastatic disease; if the values are greater than the maximum allowed, patients will&#xA;  not be considered eligible regardless of findings on any supplementary imaging&lt;/li&gt;&#xA;  &lt;li&gt;Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase [SGOT])&#xA;  and/or alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT])&#xA;  =&amp;lt; 2.5 x institutional upper limit of normal; if AST (SGOT) or ALT (SGPT) are greater&#xA;  than the institutional upper limit of normal (UNL) but less than the maximum allowed,&#xA;  an abdominal (including liver) ultrasound, CT or MRI scan must be performed prior to&#xA;  randomization to rule out metastatic disease; if the values are greater than the&#xA;  maximum allowed, patients will not be considered eligible regardless of findings on&#xA;  any supplementary imaging&lt;/li&gt;&#xA;  &lt;li&gt;Patient must have a chest x-ray done within 14 days prior to randomization; patient&#xA;  must have a CT or MRI scan of the chest done within 90 days prior to surgical&#xA;  resection if at least one of the following was undertaken:&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;A complete mediastinal lymph node resection; or&lt;/li&gt;&#xA;  &lt;li&gt;Biopsy of all desired levels of lymph nodes - as specified above; or&lt;/li&gt;&#xA;  &lt;li&gt;A pre-surgical PET scan within 60 days prior to surgical resection If none of&#xA;  the above was undertaken then the CT or MRI scan of the chest must have been&#xA;  performed within 60 days prior to surgical resection Note: a pre-surgical PET&#xA;  scan is not mandatory&lt;/li&gt;&#xA;  &lt;/ul&gt;&lt;/li&gt;&#xA;  &lt;li&gt;Patient must have an electrocardiogram (EKG) done within 14 days prior to&#xA;  randomization&lt;/li&gt;&#xA;  &lt;li&gt;Women of childbearing age and men must agree to use adequate contraception (hormonal&#xA;  or barrier method of birth control) prior to study entry and while taking study&#xA;  medication and for a period of three months after final dose; should a woman become&#xA;  pregnant or suspect she is pregnant while she or her male partner are participating&#xA;  in this study, she should inform her treating physician immediately&lt;/li&gt;&#xA;  &lt;li&gt;Patients may receive post-operative radiation therapy; patients must have completed&#xA;  radiation at least 3 weeks prior to randomization and have recovered from all&#xA;  radiation-induced toxicity; patients who have received radiation therapy should also&#xA;  be randomized within 16 weeks of surgery&lt;/li&gt;&#xA;  &lt;li&gt;Patient consent must be obtained according to local institutional and/or University&#xA;  Human Experimentation Committee requirements; it will be the responsibility of the&#xA;  local participating investigators to obtain the necessary local clearance, and to&#xA;  indicate in writing to either the National Cancer Institute of Canada (NCIC) Clinical&#xA;  Trials Group (CTG) study coordinator (for NCIC CTG centers) or the Cancer Trials&#xA;  Support Unit (CTSU) (for all other investigators), that such clearance has been&#xA;  obtained, before the trial can commence in that center; a standard consent form for&#xA;  the trial will not be provided, but a sample form is given; this sample consent form&#xA;  has been approved by the National Cancer Institute (NCI) Central Institutional Review&#xA;  Board (IRB) and must be used unaltered by those CTSI centers which operate under CIRB&#xA;  authority; for NCIC CTG centers, a copy of the initial full board Research Ethics&#xA;  Board (REB) approval and approved consent form must be sent to the NCIC CTG central&#xA;  office; please note that the consent form for this study must contain a statement&#xA;  which gives permission for the government agencies, NCI, NCIC CTG and monitoring&#xA;  agencies to review patient records&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;NCIC-CTG Centers: the patient must have the ability to understand and the&#xA;  willingness to sign a written informed consent document; the patient must sign&#xA;  the consent form prior to randomization&lt;/li&gt;&#xA;  &lt;li&gt;CTSU Centers: the patient, or in the case of a mentally incompetent patient his&#xA;  or her legally authorized and qualified representative, must have the ability to&#xA;  understand and the willingness to sign a written informed consent document; the&#xA;  consent form must be signed prior to randomization&lt;/li&gt;&#xA;  &lt;/ul&gt;&lt;/li&gt;&#xA;  &lt;li&gt;Patients must be accessible for treatment and follow-up; investigators must assure&#xA;  themselves that patients registered on this trial will be available for complete&#xA;  documentation of the treatment administered, toxicity and follow-up&lt;/li&gt;&#xA;  &lt;li&gt;Initiation of protocol treatment must begin within 10 working days of patient&#xA;  randomization&lt;/li&gt;&#xA;  &lt;li&gt;Patients may have received post-operative adjuvant platinum-based chemotherapy;&#xA;  patients must have completed chemotherapy at least 3 weeks prior to randomization and&#xA;  have recovered from all chemotherapy-induced toxicity; patients who have received&#xA;  adjuvant chemotherapy should also be randomized within 26 weeks of surgery&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;Exclusion Criteria:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Prior or concurrent malignancies; patients who have had a previous diagnosis of&#xA;  cancer, if they remain free of recurrence and metastases five years or more following&#xA;  the end of treatment and, in the opinion of the treating physician do not have a&#xA;  substantial risk of recurrence of the prior malignancy, are eligible for the study;&#xA;  patients who have been adequately treated for non-melanomatous skin cancer or&#xA;  carcinoma in situ of the cervix are eligible irrespective of when that treatment was&#xA;  given&lt;/li&gt;&#xA;  &lt;li&gt;A combination of small cell and non-small cell carcinomas or a pulmonary carcinoid&#xA;  tumor&lt;/li&gt;&#xA;  &lt;li&gt;More than one discrete area of apparent primary cancer (even if within the same lobe,&#xA;  T4, IIIB)&lt;/li&gt;&#xA;  &lt;li&gt;Clinically significant or untreated ophthalmologic (e.g. Sjogren's etc.) or&#xA;  gastrointestinal conditions (e.g. Crohn's disease, ulcerative colitis)&lt;/li&gt;&#xA;  &lt;li&gt;Any active pathological condition that would render the protocol treatment dangerous&#xA;  such as: uncontrolled congestive heart failure, angina, or arrhythmias, active&#xA;  uncontrolled infection, or others&lt;/li&gt;&#xA;  &lt;li&gt;A history of psychiatric or neurological disorder that would make the obtainment of&#xA;  informed consent problematic or that would limit compliance with study requirements&lt;/li&gt;&#xA;  &lt;li&gt;Patient, if female, is pregnant or breast-feeding&lt;/li&gt;&#xA;  &lt;li&gt;Neoadjuvant chemotherapy or immunotherapy for NSCLC; however, patients may have&#xA;  received pre-operative limited field, low dose (less than 1000 cGy) external beam&#xA;  radiation therapy or endobronchial brachytherapy or laser therapy for short term&#xA;  control of hemoptysis or lobar obstruction; full dose pre-operative radiotherapy of&#xA;  curative intent is a cause for exclusion; patients may have received post-operative&#xA;  adjuvant platinum-based chemotherapy however non-platinum-based chemotherapy is a&#xA;  cause for exclusion&lt;/li&gt;&#xA;  &lt;li&gt;History of allergic reactions attributed to compounds of similar chemical or biologic&#xA;  composition to the agents used on this trial; patients with ongoing use of phenytoin,&#xA;  carbamazepine, barbiturates, rifampicin, or St John's Wort are excluded&lt;/li&gt;&#xA;  &lt;li&gt;Incomplete healing from previous oncologic or other major surgery&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I want to find documents with a query as below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'Patients with Eastern cooperative Oncology Group (ECOG) performance status &gt; 2' will be excluded&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Output &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CTA: NCT01572038 &#xA;Labels: ECOG 0, ECOG 1, ECOG 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the simplest way to approach this problem?&lt;/p&gt;&#xA;" OwnerDisplayName="user9706" LastEditorUserId="816" LastEditDate="2015-05-22T19:36:02.187" LastActivityDate="2015-05-22T19:36:02.187" Title="How do I best parse ECOG scores in cancer clinical trial results in java?" Tags="|nlp|java|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5832" PostTypeId="2" ParentId="5826" CreationDate="2015-05-19T09:29:02.810" Score="4" Body="&lt;p&gt;Data science is a multi discipline subject.&#xA;In addition to math you need some programming skills and ideally - understanding of the domain where you are going to solve problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can start from online-courses and some introduction literature.&#xA;Look at the answers at current site:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/334/what-do-you-think-of-data-science-certifications&quot;&gt;Data Science Sertifications&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/313/books-about-the-science-in-data-science&quot;&gt;Books about Data Science&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/14/is-data-science-the-same-as-data-mining&quot;&gt;Data Science vs Data Mining&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/159/is-data-science-just-a-trend-or-is-a-long-term-concept&quot;&gt;Data Science is a trend or a long term concept&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/234/data-science-ph-d-program-what-do-you-think&quot;&gt;Ph.D. in Data Science?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/808/statistics-computer-science-data-science&quot;&gt;Statistics + Computer Science = Data Science?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/739/starting-my-career-as-data-scientist-is-software-engineering-experience-require&quot;&gt;Starting career as Data Scientist, is Software Engineering experience required?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/1216/career-switch-to-big-data-analytics&quot;&gt;Career switch to Big Data Analytics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/tagged/career&quot;&gt;And more about career...&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I can't resist from placing this illustration for the millionth time:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/wpkyD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-19T09:36:52.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5833" PostTypeId="1" CreationDate="2015-05-19T13:10:37.463" Score="4" ViewCount="506" Body="&lt;p&gt;I have a problem I would like to solve using machine learning. I would like to use some sort of classification to know if a just added change in a tree data structure is &quot;good&quot; or is &quot;bad&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have this tree:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        (A) &#xA;        / \&#xA;       /   \&#xA;     (B)   (C)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I apply a change to it (a &quot;good&quot; change, so the algorithm should associate this change with the &quot;good&quot; changes). The updated tree would be like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       (A)&#xA;       / \&#xA;      /   \&#xA;    (D)   (C)&#xA;    /&#xA;   /&#xA; (B)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Added a certain node (D) above another node (B) would be classified as a &quot;good&quot; change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when I have the learner with the correct data, the algorithm should be able to know that if I add a node of type D above a node of type B, it is a &quot;good&quot; change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to work with XML files that keeps the tree structure, a simple classifier like a naive bayes would not work, because it wouldn't be able to recognise if a node is added above another one, it only would be able to know that a node has been added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know how which algorithm/technique I should use and I don't know how I should pass the data to the learner, because the context in this scenario is important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best technique/algorithm to compare trees changes?&lt;/p&gt;&#xA;" OwnerUserId="9714" LastEditorUserId="29575" LastEditDate="2017-12-14T21:39:50.513" LastActivityDate="2017-12-14T21:39:50.513" Title="What is the best technique/algorithm to compare trees changes?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5834" PostTypeId="2" ParentId="677" CreationDate="2015-05-19T15:46:55.280" Score="6" Body="&lt;p&gt;Here is an example of how to get pandas and sklearn to play nice &lt;/p&gt;&#xA;&#xA;&lt;p&gt;say you have 2 columns that are both strings and you wish to vectorize - but you have no idea which vectorization params will result in the best downstream performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;create the vectorizer &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;to_vect = Pipeline([('vect',CountVectorizer(min_df =1,max_df=.9,ngram_range=(1,2),max_features=1000)),&#xA;                    ('tfidf', TfidfTransformer())])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;create the DataFrameMapper obj.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;full_mapper = DataFrameMapper([&#xA;        ('col_name1', to_vect),&#xA;        ('col_name2',to_vect)&#xA;    ])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this is the full pipeline&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;full_pipeline  = Pipeline([('mapper',full_mapper),('clf', SGDClassifier(n_iter=15, warm_start=True))])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;define the params you want the scan to consider&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;full_params = {'clf__alpha': [1e-2,1e-3,1e-4],&#xA;                   'clf__loss':['modified_huber','hinge'],&#xA;                   'clf__penalty':['l2','l1'],&#xA;                   'mapper__features':[[('col_name1',deepcopy(to_vect)),&#xA;                                        ('col_name2',deepcopy(to_vect))],&#xA;                                       [('col_name1',deepcopy(to_vect).set_params(vect__analyzer= 'char_wb')),&#xA;                                        ('col_name2',deepcopy(to_vect))]]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thats it! - note however that mapper_features are a single item in this dictionary - so use a for loop or itertools.product to generate a FLAT list of all to_vect options you wish to consider - but that is a separate task outside the scope of the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go on to create the optimal classifier or whatever else your pipeline ends with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gs_clf = GridSearchCV(full_pipe, full_params, n_jobs=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9718" LastEditorUserId="9718" LastEditDate="2015-05-27T03:16:56.403" LastActivityDate="2015-05-27T03:16:56.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5835" PostTypeId="2" ParentId="5819" CreationDate="2015-05-19T16:59:34.567" Score="1" Body="&lt;p&gt;You could do this a few ways...&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;write a script that does the following.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;pick a table,&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;get number of rows in the table, can do this as you go. (+=1 in loop)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;select several fields(columns) in table that you think may form the key,&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;create empty &lt;code&gt;Set()&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;burn through the file row by row, for each row grab the target fields and construct a string key &lt;code&gt;str(field1)+'_'+str(field2)&lt;/code&gt;..etc.  Add this key to your set.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;when you're done going through all rows of a table, check &lt;code&gt;len(set)&lt;/code&gt;.  If the key you picked is good then &lt;code&gt;len(set)&lt;/code&gt; will equal # rows.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another way, and this will depend on what you want to do with your data and how often you will access it...is to import it into a sql database.  I usually use mysql. Once your data is in:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;select count(*) from table_name;  will give you rows in table&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;select count(distinct field1, field2, field3) from table;  gives number of distinct combos of field1+field2+field3&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if the two selects give the same number then you have a key that works. It may not guarantee a join-key but it will be unique to that table and will help the linking process.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;many people highly advocate the use of pandas. If you get the data imported into arrays you can create a dataframe and do some queries similar to SQL.  I'm not super familiar with pandas so I can't give an example.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;If it's a one-off, I'd make a python script.  But if you think you'll be exploring the data further, spending the time to put the data in a database could be useful.&lt;/p&gt;&#xA;" OwnerUserId="3457" LastEditorUserId="98307" LastEditDate="2020-08-03T00:59:48.943" LastActivityDate="2020-08-03T00:59:48.943" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5836" PostTypeId="2" ParentId="5821" CreationDate="2015-05-19T17:13:18.667" Score="3" Body="&lt;p&gt;Although I agree with Neil Slater's response, you should keep a couple of things in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &quot;you never know!&quot; In data exploration, you never know what you may find.  If you have a ton of data, perhaps playing around with a 20x20 conv net will give you some decent results.  Of course, it would be helpful if there are more than just a few features for it to learn...if your 400 length vector is the result of one-hotting 4 different features then it's probably safe to say that a conv net won't give you much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)  If you're looking for a reason to implement a conv net, then go for it. Even if your accuracy metrics are terrible you at least get to learn how to create your net, train, and predict using your own data...one cannot underestimate this learning experience!  So much more valuable that running yet another mnist example out of the box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3)  Comparison. Make a regular net and a conv net...then you get to compare the two. Not only that, compare it to a Random Forest, logistic regression, etc. etc.. Do this enough times and you start to develop intuition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I say do it! (unless somebody is paying you...in which case try the regular NN first)&lt;/p&gt;&#xA;" OwnerUserId="3457" LastActivityDate="2015-05-19T17:13:18.667" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5837" PostTypeId="2" ParentId="5789" CreationDate="2015-05-19T20:32:50.033" Score="1" Body="&lt;p&gt;I like &lt;a href=&quot;http://blog.rstudio.org/2014/11/24/rvest-easy-web-scraping-with-r/&quot; rel=&quot;nofollow&quot;&gt;rvest&lt;/a&gt; in combination with the SelectorGadget chrome plug-in for selecting relevant sections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used rvest and built small scripts to paginate through forums by:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Look for the &quot;Page n Of m&quot; object &lt;/li&gt;&#xA;&lt;li&gt;Extract m &lt;/li&gt;&#xA;&lt;li&gt;Based on the page structure,  build a list of links from 1 to m (e.g.&#xA;    www.sample.com/page1)  &lt;/li&gt;&#xA;&lt;li&gt;Iterate the scraper through the full list of&#xA;    links&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4776" LastActivityDate="2015-05-19T20:32:50.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5838" PostTypeId="1" AcceptedAnswerId="5857" CreationDate="2015-05-19T23:34:25.213" Score="15" ViewCount="11080" Body="&lt;p&gt;I have a dataset of users purchasing products from a website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The attributes I have are user id, region(state) of the user, the categories id of product, keywords id of product, keywords id of website, and sales amount spent of the product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to use the information of a product and website to identity who the users are, such as &quot;male young gamer&quot; or &quot;stay at home mom&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I attached a sample picture as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/S7OSa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are all together 1940 unique categories and 13845 unique keywords for products. For the website, there are 13063 unique keywords. The whole dataset is huge as that's the daily logging data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of clustering, as those are unsupervised, but those id are ordered number having no numeric meaning. Then I don't know how to apply the algorithm. I am also thinking of classification. If I add a column of class based on the sales amount of product purchased. I think clustering is more preferred. I don't know what algorithm I should use for this case as the dimensions of the keywords id could be more than 10000 (each product could have many keywords, so does website). I need to use Spark for this project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me out with some ideas or suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much!&lt;/p&gt;&#xA;" OwnerUserId="9724" LastEditorUserId="29169" LastEditDate="2019-06-08T03:44:19.133" LastActivityDate="2019-06-08T03:44:19.133" Title="Using attributes to classify/cluster user profiles" Tags="|machine-learning|data-mining|classification|clustering|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="5839" PostTypeId="5" CreationDate="2015-05-20T06:53:23.057" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://hadoop.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Apache™ Hadoop™&lt;/a&gt; project develops open-source software for reliable, scalable, distributed computing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Apache_Hadoop&quot; rel=&quot;nofollow&quot;&gt;Hadoop&lt;/a&gt;&quot; typically refers to the software in the project that implements the &lt;a href=&quot;https://en.wikipedia.org/wiki/MapReduce&quot; rel=&quot;nofollow&quot;&gt;map-reduce&lt;/a&gt; data analysis framework, plus the distributed file system (HDFS) that underlies it.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since version 0.23, Hadoop disposes of a standalone resource manager : &lt;a href=&quot;https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html&quot; rel=&quot;nofollow&quot;&gt;yarn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This resource manager makes it easier to use other modules alongside with the MapReduce engine, such as :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://ambari.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Ambari&lt;/a&gt;,  A web-based tool for provisioning, managing, and&lt;br&gt;&#xA;monitoring Apache Hadoop clusters which includes support for Hadoop&lt;br&gt;&#xA;HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig &#xA;and Sqoop. Ambari also provides a dashboard for viewing cluster&lt;br&gt;&#xA;health such as heatmaps and ability to view MapReduce, Pig and Hive&lt;br&gt;&#xA;applications visually along with features to diagnose their&lt;br&gt;&#xA;performance characteristics in a user-friendly manner.     &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://avro.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Avro&lt;/a&gt;, a data serialization system based on JSON schemas.   &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cassandra.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Cassandra&lt;/a&gt;, a replicated, fault-tolerant, decentralized and&#xA;scalable database system.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://chukwa.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Chukwa&lt;/a&gt;:  A data collection system for managing large distributed&#xA;systems.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://hbase.apache.org/&quot; rel=&quot;nofollow&quot;&gt;HBase&lt;/a&gt;, A scalable, distributed database that supports structured&#xA;data storage for large tables. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://hive.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Hive&lt;/a&gt;, A data warehouse infrastructure that provides data&#xA;summarization and ad hoc querying.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout&lt;/a&gt;, a library of machine learning algorithms compatible&#xA;with M/R paradigm.   &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pig.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Pig&lt;/a&gt;, a platform/programming language for authoring parallelizable&#xA;jobs&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://storm.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Storm&lt;/a&gt;, a system for real-time and stream processing&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://zookeeper.apache.org/&quot; rel=&quot;nofollow&quot;&gt;ZooKeeper&lt;/a&gt;, a system for coordinating distributed nodes, similar to Google's Chubby&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://oozie.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Oozie&lt;/a&gt;, a workflow scheduler system to manage Apache Hadoop jobs.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Spark&lt;/a&gt;, a fast and general engine for large-scale data processing.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://flink.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Flink&lt;/a&gt;, a fast and reliable large-scale data processing engine. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Recommended reference sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://wiki.apache.org/hadoop/Hive/LanguageManual/DDL&quot; rel=&quot;nofollow&quot;&gt;Hive Language Reference&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Commercial support is available from &lt;a href=&quot;https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support&quot; rel=&quot;nofollow&quot;&gt;a variety of companies&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:43.897" LastActivityDate="2015-12-03T22:02:43.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5840" PostTypeId="4" CreationDate="2015-05-20T06:53:23.057" Score="0" Body="Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:35.553" LastActivityDate="2015-12-03T22:02:35.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5841" PostTypeId="2" ParentId="5833" CreationDate="2015-05-20T09:59:47.257" Score="5" Body="&lt;p&gt;Most of the machine learning algorithms are designed to work with data in a tabular format. That mean, each data instance is contained in a single row, and the values from each column are the observed values for a specific instance for a given variable. There are few reasons why the most ML algorithms are designed to work on this kind of data. An important factor is that the structure is very simple and various operations can be done with ease. A second reason is that even if looks like a inflexible structure, some sort of additional structure in your data still can be represented on tabular format (using redundancy). Another reason would be that an algorithm designed to work for a specific structure of the data will be constraint to work on a much smaller set of problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the main point is that &lt;em&gt;&quot;If the mountain won't come to Muhammad then Muhammad must go to the mountain&quot;&lt;/em&gt; (note there's nothing religion related here). So what you have to do would be to fabricate the features yourself in a tabular format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will give you an example on how I see a starting point. Consider an instance a row in a table. Each row will be a &lt;strong&gt;change&lt;/strong&gt;. A &lt;strong&gt;change&lt;/strong&gt; has a label, it is &lt;em&gt;good&lt;/em&gt; or &lt;em&gt;bad&lt;/em&gt;. So, you can add a feature used as target feature called &lt;strong&gt;class&lt;/strong&gt;. We go further by noting that a change is an insertion of a node. If your changes are of multiple type you can add a feature called &lt;strong&gt;operation-type&lt;/strong&gt; having values: &lt;strong&gt;insert&lt;/strong&gt;, &lt;strong&gt;delete&lt;/strong&gt;, &lt;strong&gt;change&lt;/strong&gt;, etc. Now, a node has a type also. You can add a new feature called &lt;strong&gt;node-type&lt;/strong&gt;, which could be &lt;strong&gt;A&lt;/strong&gt;, &lt;strong&gt;B&lt;/strong&gt;, etc. What you have to do would be to invent those features by noting what is important for you or your business and eventually select only those features which would be relevant enough. I really hope it was clear enough.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-05-20T09:59:47.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5842" PostTypeId="1" CreationDate="2015-05-20T11:04:18.223" Score="1" ViewCount="40" Body="&lt;p&gt;I'm looking for a spatial index that can efficiently find the most extreme n points in a certain direction, i.e. for a given w, find x[0:n] in the dataset where x0 gives the largest value of w.x and x1 the second largest value of w.x, etc... . Is there a name for this type of query? What would be an efficient data structure to use? x might have around 20 dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thankyou!&lt;/p&gt;&#xA;" OwnerUserId="1283" LastActivityDate="2015-05-20T14:11:38.367" Title="Index for efficient argmax(w.x) query ~ 20d" Tags="|data-indexing-techniques|indexing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5843" PostTypeId="2" ParentId="5789" CreationDate="2015-05-20T13:13:05.653" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://scrapy.org/&quot; rel=&quot;nofollow&quot;&gt;Scrapy&lt;/a&gt; is a great Python library which can help you scrape different sites faster and make your code structure better. Not all sites can be parsed with classic tools, because they can use dynamic JS content building. For this task it is better to use &lt;a href=&quot;http://www.seleniumhq.org/&quot; rel=&quot;nofollow&quot;&gt;Selenium&lt;/a&gt; (This is a test framework for web sites, but it also a great web scraping tool). There's also a &lt;a href=&quot;http://selenium-python.readthedocs.org/&quot; rel=&quot;nofollow&quot;&gt;Python wrapper&lt;/a&gt; available for this library. In Google you can find a few tricks which can help you use Selenium inside &lt;a href=&quot;http://scrapy.org/&quot; rel=&quot;nofollow&quot;&gt;Scrapy&lt;/a&gt; and make your code clear, organized, and you can use some great tools for &lt;a href=&quot;http://scrapy.org/&quot; rel=&quot;nofollow&quot;&gt;Scrapy&lt;/a&gt; library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that Selenium would be a better scraper for Linkedin than classic tools. There is a lot of javascript and dynamic content. Also, if you want to make authentication in your account and scrape all available content, you will get a lot of problems with classic authentication using simple libraries like &lt;a href=&quot;http://docs.python-requests.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;requests&lt;/a&gt; or &lt;a href=&quot;https://docs.python.org/2/library/urllib.html&quot; rel=&quot;nofollow&quot;&gt;urllib&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9634" LastEditorUserId="378" LastEditDate="2015-05-21T08:51:08.577" LastActivityDate="2015-05-21T08:51:08.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5844" PostTypeId="1" CreationDate="2015-05-20T13:24:33.733" Score="4" ViewCount="310" Body="&lt;p&gt;I am interested in modeling startup companies failure and success rates to describe what is the representative startup. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 40 companies in a dataset. Each company is represented as a list of all the investment financing rounds it has gone through. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Company 1: Seed Round, Series A Round&#xA;Company 2: Seed Round, Series A Round, Series B Round&#xA;Company 3: Seed Round, Series A Round, Series B Round&#xA;Company 4: Seed Round, Series A Round, Series B Round, Series C Round&#xA;Company 5: Seed Round&#xA;Company 6: Series A Round, Series B Round&#xA;Company 6: Series A Round&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see that each round can be represented as a state in a Markov Chain, and transitions are only allowed from earlier stages to later stages. I can go from Seed to Series A, and from Series A to Series B, but not from Series B to Series A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we have N-order markov chains (production data has N &amp;lt;= 4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output I'm looking for is a binary tree chart showing each stage as a node and each node can either transition to the next node or to a final state meaning the company has failed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem can also be seen as a real options model...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to implement this model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can code in Python or Ruby (but I am no expert).&lt;/p&gt;&#xA;" OwnerUserId="10171" LastEditorUserId="1367" LastEditDate="2015-05-21T19:28:50.147" LastActivityDate="2015-05-21T19:28:50.147" Title="What are available methods for modeling startup survival rates?" Tags="|python|survival-analysis|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5845" PostTypeId="2" ParentId="5842" CreationDate="2015-05-20T14:11:38.367" Score="2" Body="&lt;p&gt;The query is called a 'top-k' query, and you can answer it quickly using the ranking cube approach. &lt;a href=&quot;http://www1.se.cuhk.edu.hk/~hcheng/paper/vldb06_rankcube.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www1.se.cuhk.edu.hk/~hcheng/paper/vldb06_rankcube.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper does not derive the time complexity.&lt;/p&gt;&#xA;" OwnerUserId="1283" LastActivityDate="2015-05-20T14:11:38.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5846" PostTypeId="1" CreationDate="2015-05-20T17:41:15.180" Score="0" ViewCount="154" Body="&lt;p&gt;I have completely followed the machine learning course on &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;coursera Machine Learning by professor Andrew Ng&lt;/a&gt; &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to put my knowledge to action. Some ideas that I have include : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Voice synthesis&#xA; -Voice recognition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But since the course did not focus specifically on application of machine learning in these domains, could some one point me to some other course or books that can get me started.&lt;/p&gt;&#xA;" OwnerUserId="9747" LastEditorUserId="75" LastEditDate="2015-05-26T06:35:29.197" LastActivityDate="2015-05-26T06:35:29.197" Title="Ideas for next step of Machine Learning" Tags="|machine-learning|algorithms|beginner|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-21T19:30:19.237" ContentLicense="CC BY-SA 3.0" />
  <row Id="5848" PostTypeId="2" ParentId="5846" CreationDate="2015-05-21T05:33:57.283" Score="3" Body="&lt;p&gt;I would suggest you to check &lt;a href=&quot;http://research.microsoft.com/en-US/people/deng/keynote-deeplearning-interspeech2014-sept18-removebackups.pdf&quot; rel=&quot;nofollow&quot;&gt;this excellent presentation&lt;/a&gt; by Li Deng (Microsoft Research). Many of the slides contain references to relevant &lt;em&gt;research papers&lt;/em&gt; and even several &lt;strong&gt;interesting books&lt;/strong&gt; on the topics of interest (it should be pretty easy to find). It might be also helpful to check references, listed in &lt;a href=&quot;http://arxiv.org/abs/1412.5567&quot; rel=&quot;nofollow&quot;&gt;this research paper&lt;/a&gt; by Prof. Andrew Ng and his colleagues at Baidu Research. Finally, a focused Internet search will provide you with &lt;em&gt;comprehensive&lt;/em&gt; list of resources for further research.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-05-21T05:33:57.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5849" PostTypeId="2" ParentId="5751" CreationDate="2015-05-21T08:17:15.573" Score="1" Body="&lt;p&gt;Both previous answers take an interesting approach, but neither really tackles exactly what I was asking; how to compare a given terms &lt;em&gt;frequency&lt;/em&gt; over time, and compare with &lt;em&gt;other terms&lt;/em&gt; over the same period, in a statistically rigorous way that accounts for the power law distribution of term frequencies with collection size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer below is my work so far on the problem. &lt;strong&gt;It is not, by any means, a complete answer&lt;/strong&gt;, however I post it here to elicit feedback and suggestions for improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the definition of Zipf's law,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$f(r) \sim z_{max} r^{-\alpha}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here, &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; is the rank of a given term, &lt;span class=&quot;math-container&quot;&gt;$z_{max}$&lt;/span&gt; is the frequency of the most frequent term, &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is Zipf's exponent, and &lt;span class=&quot;math-container&quot;&gt;$f(r)$&lt;/span&gt; is the frequency of the &lt;em&gt;r-th&lt;/em&gt; ranked term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the log of both sides, you get a linear relationship between &lt;span class=&quot;math-container&quot;&gt;$log(f)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$log(r)$&lt;/span&gt;, with gradient &lt;span class=&quot;math-container&quot;&gt;$-\alpha$&lt;/span&gt; and intercept &lt;span class=&quot;math-container&quot;&gt;$z_{max}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$log(f) \sim log(z_{max}) - \alpha log(r)$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/xzlwB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to find the expected frequency of a single term over a given period. First, split the period into equal sized windows. For the first window, &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;, we know the frequency of our term &lt;span class=&quot;math-container&quot;&gt;$f_i$&lt;/span&gt;, and the frequency of the largest term &lt;span class=&quot;math-container&quot;&gt;$z_{max,i}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt;. We can use these to find the rank of our term,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$r \sim \left(\frac{f_i(r)}{z_{max,i}}\right)^{-\alpha}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This rank can then be used to calculate the 'expected' frequency for all subsequent time windows, which I'll indicate with &lt;span class=&quot;math-container&quot;&gt;$\hat{f}$&lt;/span&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\hat{f}_{i+1}(r) \sim z_{max,i+1} r^{\alpha(i+1)} $&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ratio &lt;span class=&quot;math-container&quot;&gt;$\frac{\hat{f}}{f}$&lt;/span&gt; represents the normalised frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some observations:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For a stationary underlying distribution, this approach identifies minor deviations from the expected frequency of a term. A moving window could be used if the underlying text distribution changes dramatically, or exhibits seasonal behaviour.&lt;/li&gt;&#xA;&lt;li&gt;The initial choice of &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; could be calculated for the entire period, rather than the first window.&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; could be calculated from the raw term and total frequency counts using heap's law (see &lt;a href=&quot;http://arxiv.org/abs/1002.3861&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lu et al. 2010&lt;/a&gt; for a derivation of Heap's law from Zipf's).&lt;/li&gt;&#xA;&lt;li&gt;This approach assumes a linear relationship between &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; in log space.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="378" LastEditorUserId="378" LastEditDate="2018-10-25T11:05:32.660" LastActivityDate="2018-10-25T11:05:32.660" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5851" PostTypeId="2" ParentId="5267" CreationDate="2015-05-21T08:26:42.463" Score="1" Body="&lt;p&gt;Sounds like you could use a regression model to estimate the probabilities that a team wins/draws/loses vs. another team. Basically, for any outcome (win, draw and lose), you want this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(A|B) = ...&#xA;P(B|A) = ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which means: the probability of &lt;strong&gt;outcome&lt;/strong&gt; for team A given that it is matched against team B (and vice versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimations could be represented like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(A &amp;gt; B) = 0.75 % A wins&#xA;P(A = B) = 0.10 % A draws&#xA;P(A &amp;lt; B) = 0.15 % A loses&#xA;&#xA;P(B &amp;gt; A) = 0.20 % B wins&#xA;P(B = A) = 0.10 % B draws&#xA;P(B &amp;gt; A) = 0.70 % B loses&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think a logical step is to measure the bias towards a certain outcome. That would represent the &lt;em&gt;confidence ratio&lt;/em&gt; of your algorithm. The more the probabilities of any outcome looks alike (i.e. P(B &gt;/=/&amp;lt; A) = 0.33), the less confidence it has.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-05-21T08:26:42.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5852" PostTypeId="2" ParentId="5838" CreationDate="2015-05-21T08:46:49.783" Score="-2" Body="&lt;p&gt;For orientation and exploration, I can recommend &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;WeKa&lt;/a&gt;, which is a very nice toolkit for machine learning. It does take a certain input format (.ARFF) so you might need to look into that as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the keyword dilemma, I recommend performing some feature selection in order to eliminate redundant or non-indicative keywords.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-05-21T08:46:49.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5853" PostTypeId="2" ParentId="5817" CreationDate="2015-05-21T12:51:35.033" Score="9" Body="&lt;p&gt;Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Airline fares have two components to them: all the actual fares (complete with fare rules and what have you) that an airline has available for a certain route, most of which are published the Airline Tariff Publishing Company (a few special-use ones are not, but those are the exception rather than the rule) and the actual inventory management performed by the airline on a day-to-day basis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fares can be submitted to ATPCO four times a day, at set intervals, and when airlines do so, it will usually consist of a mixture of additions, deletions, and modifications of existing fares. When an airline initiates a pricing action (assuming their competitors aren't trying to make their own moves here), they usually have to wait until the next update to see if their competitors follow/respond. The converse goes when a competitor initiates a pricing action, as the airline has to wait until the next update before they can respond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is all well and good with respect to fares, but the problem is that, because this is all getting published in ATPCO, fares are the next best thing to public information... all your competitors get to see what you've got in your arsenal, so attempts to obfuscate are not unheard of, such as publishing fares that will never actually be assigned any inventory, listing all the fares as day-of-departure, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many ways, the secret sauce comes down to the actual inventory allocation, i.e. how many seats on each flight will you be willing to sell for a given fare, and this information is not publicly available. You can get some glimpses by scraping web info, but the potential combinations of departure time/date and fare rules are quite numerous and may quickly escalate beyond your ability to easily keep track of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically an airline will only be willing to sell a handful of seats for a very low fare and the people who snag those have to book quite far in advance lest the fare rules lock them out, or other travelers simply beat them to the punch. The airline will be willing to sell a few more seats for a higher fare, and so on and so forth. They will be quite happy to sell all of the seats for the highest fare they've got published, but this is not usually feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you're seeing with fares getting higher the closer you get to the day of departure is simply the natural process of having the cheap seats get booked farther out, while the remaining inventory gradually gets more expensive. Of course, there are some caveats here. The RM process is actively managed and human intervention is quite common as the RM team generally strives to meet its revenue goals and maximize revenue on each flight. As such, flights that fill up quickly may be &quot;tightened up&quot; by closing out low fares. Flights that are booking slowly may be &quot;loosened up&quot; by allocating more seats to lower fares.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a constant interplay and competition between airlines in this area, but you are not very likely to capture the actual dynamics just from scraping fares. Don't get me wrong, we had such tools at our disposal, and, despite their limitations, they were quite valuable, but they were just one data source that fed into the decision-making process. You'd need access to the hundreds, if not thousands of operational decisions made by RM teams on a daily basis, as well as state-of-the-world information as they see it at the time. If you cannot find an airline partner to work with in order to get this data, you might need to consider alternate data sources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend looking into getting access to O&amp;amp;D fare data from the Official Airline Guide (or one of their competitors) and try to use that for your analysis. It's sample-based (about 10% of all tickets sold) and aggregated at a higher level than would be ideal so careful route selection is imperative (I'd recommend something with plenty of airlines, flying non-stop multiple times a day, with large aircraft), but you may be able to get a better picture of what was actually sold (average fare) and how much of it was sold (load factor), vs. merely what is available for sale at a given point in time. Using that information you might be in better position to at least explore the outcomes of the airlines' pricing strategy, and make your inferences from there.&lt;/p&gt;&#xA;" OwnerUserId="9448" LastActivityDate="2015-05-21T12:51:35.033" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5854" PostTypeId="2" ParentId="5844" CreationDate="2015-05-21T15:23:44.270" Score="4" Body="&lt;p&gt;If the data you show are the only data that you have, then the Markov Chain is really boring: it is a linear chain, going from Round A to Round B to Round C, and all those states are connected to a base state (which is Death, or something).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can directly calculate the transition probabilities from the data you have, since the number of companies that reached round N are &lt;strong&gt;all companies that could have reached round N&lt;/strong&gt; (there is no alternative path).  The death probability at the previous stage is (1 - $P_{reaching N}$)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [1]: raw_data = &quot;&quot;&quot;&#xA;   ...: Company 1: Seed Round, Series A Round&#xA;   ...: Company 2: Seed Round, Series A Round, Series B Round&#xA;   ...: Company 3: Seed Round, Series A Round, Series B Round&#xA;   ...: Company 4: Seed Round, Series A Round, Series B Round, Series C Round&#xA;   ...: Company 5: Seed Round&#xA;   ...: Company 6: Series A Round, Series B Round&#xA;   ...: Company 6: Series A Round&#xA;   ...: &quot;&quot;&quot;&#xA;In [2]: data_lines = raw_data.splitlines()[1:]&#xA;In [6]: key_vals = {}&#xA;In [12]: for line in data_lines:&#xA;             key, val = line.split(':')&#xA;             key = key.strip()&#xA;             vals = [v.strip() for v in val.split(',')]&#xA;             key_vals[key] = vals&#xA;&#xA;In [13]: key_vals&#xA;Out[13]:&#xA;{'Company 1': ['Seed Round', 'Series A Round'],&#xA; 'Company 2': ['Seed Round', 'Series A Round', 'Series B Round'],&#xA; 'Company 3': ['Seed Round', 'Series A Round', 'Series B Round'],&#xA; 'Company 4': ['Seed Round',&#xA;  'Series A Round',&#xA;  'Series B Round',&#xA;  'Series C Round'],&#xA; 'Company 5': ['Seed Round'],&#xA; 'Company 6': ['Series A Round']}&#xA;&#xA;In [14]: transitions = ['Seed Round', 'Series A Round', 'Series B Round', 'Series C Round']&#xA;&#xA;In [19]: for transition in transitions:&#xA;    summed = 0&#xA;    for company, rounds in key_vals.iteritems():&#xA;        if transition in rounds:&#xA;            summed += 1&#xA;    prob = float(summed) / float(len(key_vals.keys()))&#xA;    death_prob = 1 - prob&#xA;    print &quot;From previous to %s: probability %s&quot; % (transition, prob)&#xA;    print &quot;Death rate at %s: probability %s&quot; % (transition, death_prob)&#xA;&#xA;From previous to Seed Round: probability 0.833333333333&#xA;Death rate at Seed Round: probability 0.166666666667&#xA;From previous to Series A Round: probability 0.833333333333&#xA;Death rate at Series A Round: probability 0.166666666667&#xA;From previous to Series B Round: probability 0.5&#xA;Death rate at Series B Round: probability 0.5&#xA;From previous to Series C Round: probability 0.166666666667&#xA;Death rate at Series C Round: probability 0.833333333333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if you had some more features of each company, like the amount of money they received at each stage, or the profit they were making, then you could train a &lt;a href=&quot;http://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow&quot;&gt;decision tree&lt;/a&gt;, e.g. with &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow&quot;&gt;this implementation in sklearn&lt;/a&gt;, that told you, in simple words, &quot;if a company arrived at round X with at least Y dollars raised and at least Z dollars of profit, then they are passing to the next round with 0.XX probability&quot;.  Which is, I think, what you are aiming at.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastActivityDate="2015-05-21T15:23:44.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5855" PostTypeId="1" CreationDate="2015-05-21T15:25:35.377" Score="6" ViewCount="323" Body="&lt;p&gt;I am looking for a practical guide/tutorial preferably in R to show how to do gerrymandering. (I was looking for it also in CRAN but didn't find such package) Gerrymandering is the manipulation of the boundaries of electorial districts in order to gain political advantage for one party. If there is an analogy/similar process in another area it would be also interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have detailed historical  election results. I can see the results from each &quot;voting office&quot; and also the geographical area they cover. These results are aggregated to electorial district level. I'd like to see how moving voting offices to different electorial districts can influence the results of the election and how this process can be optimized for one party. Certainly there should be some constraints not to create really weird shaped districts at the end.&lt;/p&gt;&#xA;" OwnerUserId="5211" LastActivityDate="2016-03-22T15:27:30.863" Title="Gerrymandering - Geospatial optimization to maximize votes in R" Tags="|r|optimization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5856" PostTypeId="1" CreationDate="2015-05-21T17:57:35.003" Score="7" ViewCount="3890" Body="&lt;p&gt;I went through &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/17-analytic-disciplines-compared&quot; rel=&quot;nofollow&quot;&gt;this comparison of analytic disciplines&lt;/a&gt; and &lt;a href=&quot;http://www.win-vector.com/blog/2010/10/a-personal-perspective-on-machine-learning/&quot; rel=&quot;nofollow&quot;&gt;this perspective of machine learning&lt;/a&gt;, but I am not finding any answers on the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How is Data Science related to Machine learning? &lt;/li&gt;&#xA;&lt;li&gt;How is it &lt;strong&gt;not&lt;/strong&gt; related to Machine Learning? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9521" LastEditorUserId="1367" LastEditDate="2015-05-24T08:14:19.263" LastActivityDate="2022-01-08T08:20:11.440" Title="How is Data Science related to Machine learning?" Tags="|machine-learning|definitions|knowledge-base|" AnswerCount="5" CommentCount="1" CommunityOwnedDate="2015-05-25T18:09:23.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="5857" PostTypeId="2" ParentId="5838" CreationDate="2015-05-21T19:00:55.473" Score="12" Body="&lt;p&gt;Right now, I only have time for a very brief answer, but I'll try to expand on it later on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want to do is a &lt;strong&gt;clustering&lt;/strong&gt;, since you want to discover some labels for your data.  (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to perform a clustering on your users, you need to have them as some kind of points in an abstract space.  Then you will measure distances between points, and say that points that are &quot;near&quot; are &quot;similar&quot;, and label them according to their place in that space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to transform your data into something that looks like a user profile, i.e.: a user ID, followed by a vector of numbers that represent the features of this user.  In your case, each feature could be a &quot;category of website&quot; or a &quot;category of product&quot;, and the number could be the amount of dollars spent in that feature.  Or a feature could be a combination of web and product, of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, let us imagine the user profile with just three features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;dollars spent in &quot;techy&quot; webs, &lt;/li&gt;&#xA;&lt;li&gt;dollars spent on &quot;fashion&quot; products, &lt;/li&gt;&#xA;&lt;li&gt;and dollars spent on &quot;aggressive&quot; video games on &quot;family-oriented&quot; webs (who knows).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In order to build those profiles, you need to map the &quot;categories&quot; and &quot;keywords&quot; that you have, which are too plentiful, into the features you think are relevant.  Look into &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf.html&quot; rel=&quot;noreferrer&quot;&gt;topic modeling&lt;/a&gt; or &lt;a href=&quot;http://en.wikipedia.org/wiki/Semantic_similarity&quot; rel=&quot;noreferrer&quot;&gt;semantic similarity&lt;/a&gt; to do so.  Once that map is built, it will state that all dollars spent on webs with keywords &quot;gadget&quot;, &quot;electronics&quot;, &quot;programming&quot;, and X others, should all be aggregated into our first feature; and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not be afraid of &quot;imposing&quot; the features!  You will need to refine them and maybe completely change them once you have clustered the users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have user profiles, proceed to cluster them using &lt;a href=&quot;http://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;noreferrer&quot;&gt;k-means&lt;/a&gt; or whatever else you think is interesting.  Whatever technique you use, you will be interested in getting the &quot;representative&quot; point for each cluster.  This is usually the geometric &quot;center&quot; of the points in that cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot those &quot;representative&quot; points, and also plot how they compare to other clusters.  Using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Radar_chart&quot; rel=&quot;noreferrer&quot;&gt;radar chart&lt;/a&gt; is very useful here.  Wherever there is a salient feature (something in the representative that is very marked, and is also very prominent in its comparison to other clusters) is a good candidate to help you label the cluster with some catchy phrase (&quot;nerds&quot;, &quot;fashionistas&quot;, &quot;aggressive moms&quot; ...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that a clustering problem is an open problem, so there is no &quot;right&quot; solution!  And I think my answer is quite long already; check also about normalization of the profiles and filtering outliers.&lt;/p&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="1367" LastEditDate="2015-05-21T19:06:40.980" LastActivityDate="2015-05-21T19:06:40.980" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5858" PostTypeId="2" ParentId="5856" CreationDate="2015-05-21T19:29:24.063" Score="3" Body="&lt;p&gt;Data science is much more broad. It's sort of a catch-all term that right now doesn't honestly have a very clear definition. But data-science includes all of the skills and techniques required to make sense of data which has high velocity (it's coming at you quickly), volume (there's a lot of it), or variability (it's messy, like natural language processing). This means that it certainly includes machine learning and AI, but that it's also about the tools one might use in a real-world situation such as SQL, Hadoop or Spark (and related information such as a knowledge of parallel programming). Additionally, data science may or may not include the communication aspect such as making good graphs and using Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, Data Science is ML+.&lt;/p&gt;&#xA;" OwnerUserId="9768" LastActivityDate="2015-05-21T19:29:24.063" CommentCount="0" CommunityOwnedDate="2015-05-25T18:09:23.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="5859" PostTypeId="2" ParentId="5856" CreationDate="2015-05-21T19:46:02.663" Score="4" Body="&lt;p&gt;Machine Learning tries to create systems that can learn from data. As such it can be used in a wide variety of settings, for example to &lt;a href=&quot;https://www.youtube.com/watch?v=iNL5-0_T1D0&quot; rel=&quot;nofollow&quot;&gt;make robots learn to walk&lt;/a&gt; or &lt;a href=&quot;https://www.youtube.com/watch?v=m0tJLTXNT0A&quot; rel=&quot;nofollow&quot;&gt;train virtual agents to play video games&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data science concerns itself with the extraction of knowledge from data. In order to do so it uses a bunch of different techniques from different disciplines. Machine learning includes some techniques that can be very useful for a data scientist such as deep learning, decision trees and different clustering algorithms. However, machine learning has more to offer than Data Science uses and Data Science does not solely rely on Machine Learning.&lt;/p&gt;&#xA;" OwnerUserId="9769" LastActivityDate="2015-05-21T19:46:02.663" CommentCount="0" CommunityOwnedDate="2015-05-25T18:09:23.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="5860" PostTypeId="1" CreationDate="2015-05-21T22:26:59.077" Score="3" ViewCount="1464" Body="&lt;p&gt;What is a good book or resource to start learning Artificial Intelligence ? &lt;/p&gt;&#xA;" OwnerUserId="9771" LastEditorUserId="9771" LastEditDate="2019-05-18T17:56:51.010" LastActivityDate="2019-05-18T17:56:51.010" Title="Where to start, which books" Tags="|beginner|books|" AnswerCount="6" CommentCount="3" ClosedDate="2015-10-30T14:50:36.757" CommunityOwnedDate="2015-05-22T19:35:19.367" ContentLicense="CC BY-SA 4.0" />
  <row Id="5861" PostTypeId="2" ParentId="5856" CreationDate="2015-05-22T09:22:12.163" Score="6" Body="&lt;p&gt;Data science is much broader concept than machine learning.&#xA;It starts from simple data visualization and descriptive statistics to get insights, manipulations like cleansing to prepare data. Before you can use some ML algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically such huge stacks as bigdata, visualization and data preprocessing are out of machine learning scope. And they are all integral parts of &quot;Data Science&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://pbs.twimg.com/media/B6sZGVfIQAAMZvH.png:large&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Large resolution image:&#xA;&lt;a href=&quot;https://whatsthebigdata.files.wordpress.com/2013/07/datascientistmap.png&quot; rel=&quot;nofollow&quot;&gt;https://whatsthebigdata.files.wordpress.com/2013/07/datascientistmap.png&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-05-24T14:33:02.237" LastActivityDate="2015-05-24T14:33:02.237" CommentCount="2" CommunityOwnedDate="2015-05-25T18:09:23.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="5862" PostTypeId="2" ParentId="5789" CreationDate="2015-05-22T11:28:01.467" Score="0" Body="&lt;p&gt;I would also go with beautifulsoup, if you know python. In case you rather code javascript/JQuery (and you are familiar with node.js), you may want to checkout &lt;a href=&quot;http://coffeescript.org/&quot; rel=&quot;nofollow&quot;&gt;CoffeeScript&lt;/a&gt; (Check out the &lt;a href=&quot;http://blog.teamtreehouse.com/the-absolute-beginners-guide-to-coffeescript&quot; rel=&quot;nofollow&quot;&gt;Tutorial&lt;/a&gt;) I already used it successfully on several occasions for scraping web pages.&lt;/p&gt;&#xA;" OwnerUserId="9776" LastActivityDate="2015-05-22T11:28:01.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5863" PostTypeId="1" CreationDate="2015-05-22T13:21:44.143" Score="3" ViewCount="6133" Body="&lt;p&gt;Training a basic multilayer perceptron neural network boils down to minimizing some kind of error function. Often the sum of squared errors is chosen as a this error function, but where does this function come from?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always thought this function was chosen because it makes sense intuitively. However, recently I learned that this is only partly true and there is more behind it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bishop wrote in one of his &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/cmbishop/downloads/bishop%20-%20nna%20-%20rsi94.pdf&quot; rel=&quot;nofollow&quot;&gt;papers&lt;/a&gt; that the sum of squared errors function can be derived from the principle of &lt;a href=&quot;http://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;maximum likelihood&lt;/a&gt;. Furthermore he wrote that the squared error therefore makes the assumption that the noise on the target value has a Gaussian distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what he means with that. How does the sum of squared errors relate to the maximum likelihood principle in the context of neural networks?&lt;/p&gt;&#xA;" OwnerUserId="9769" LastActivityDate="2023-12-31T18:34:07.170" Title="Where does the sum of squared errors function in neural networks come from?" Tags="|machine-learning|neural-network|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5864" PostTypeId="2" ParentId="5860" CreationDate="2015-05-22T13:38:27.207" Score="4" Body="&lt;p&gt;The standard textbooks that covers AI is &quot;Artificial Intelligence: A Modern Approach&quot; by Russel &amp;amp; Norvig. The book's website can be found &lt;a href=&quot;http://aima.cs.berkeley.edu/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also recommend &quot;Artificial Intelligence: Foundations of Computational Agents&quot; by Poole and Mackworth. The book can be read &lt;a href=&quot;http://artint.info/&quot; rel=&quot;noreferrer&quot;&gt;online&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9769" LastActivityDate="2015-05-22T13:38:27.207" CommentCount="1" CommunityOwnedDate="2015-05-22T19:35:19.367" ContentLicense="CC BY-SA 3.0" />
  <row Id="5865" PostTypeId="2" ParentId="5863" CreationDate="2015-05-22T14:01:28.910" Score="2" Body="&lt;p&gt;Your reference of Bishop is not entirely accurate. What he states in the paper you linked is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;It should be noted that the standard sum-of-squares error, introduced here from a heuristic viewpoint, can be derived from the principle of maximum likelihood on the assumption that the noise on the target data has a Gaussian distribution [references cited]. Even when this assumption is not satisfied, however, the sum-of-squares error function remains of great practical importance.&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The important point with regard to your question is that there is no inherent assumption that there is Gaussian noise when training a Multilayer Perceptron (MLP). Therefore, for an MLP, the sum-of-squares error function is not derived from the principle of maximum likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider training an MLP to learn the &lt;code&gt;XOR&lt;/code&gt; function. There are four pairs of inputs with corresponding outputs but there is no noise in the data. Yet the sum-of-squares error is still applicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevance of using sum-of-squares for neural networks (and many other situations) is that the error function is differentiable and since the errors are squared, it can be used to reduce or minimize the magnitudes of both positive and negative errors.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-05-22T14:01:28.910" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5867" PostTypeId="1" CreationDate="2015-05-22T21:22:15.133" Score="3" ViewCount="1133" Body="&lt;p&gt;All distributions in the &lt;code&gt;gbm&lt;/code&gt; package in R are associated with a loss function. For example, when we set &lt;code&gt;distribution = 'binomial'&lt;/code&gt;, the loss function chosen internally is the logistic loss function. Can anyone explain how multi-class classification works with &lt;code&gt;gbm&lt;/code&gt; and the loss function that is being used for it i.e. when we set &lt;code&gt;distribution='multinomial'&lt;/code&gt;? Is it using one-vs-all or all-vs-all under the hood for doing its multi-class classification? &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="8953" LastEditDate="2015-05-23T09:23:26.930" LastActivityDate="2015-05-29T01:09:27.367" Title="What loss function does the 'multinomial' distribution with the gbm package in R use?" Tags="|r|gbm|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5868" PostTypeId="2" ParentId="5856" CreationDate="2015-05-22T22:22:34.720" Score="3" Body="&lt;p&gt;Data Science is, as others have noted, a much broader term than machine learning. Applying Machine learning techniques is one aspect of data science. Data Science,  more generally, is the science of deriving knowledge from data. The term was coined back in 1960 and kept evolving to describe the flow and interplay of problem definition, data collection, data transformation, data modeling/ analysis,  and decision making. So to answer your question specifically: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Machine learning aids data science by providing a suit of algorithms for data modeling/ analysis (through training of machine learning algorithms), decision making (through streaming, online learning, real-time testing that are all topics that come under machine learning), and even data preparation (machine learning algorithms automatically detect anomalies in the data). &lt;/li&gt;&#xA;&lt;li&gt;Data Science stitches together a bunch of ideas/ algorithms drawn from machine learning to create a solution and in doing so borrows a lot of ideas from traditional statistics, domain expertise and basic mathematics. In this way, data science is the process of solving a use case, providing a solution as opposed to machine learning that is an important cog in that solution. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="847" LastActivityDate="2015-05-22T22:22:34.720" CommentCount="0" CommunityOwnedDate="2015-05-25T18:09:23.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="5869" PostTypeId="2" ParentId="169" CreationDate="2015-05-22T22:29:22.153" Score="2" Body="&lt;p&gt;Lots of techniques out there. If your information system has a decision attribute or labels attached, the best way I found is to use rough set based attribute reduction. Check out the quick reduction algorithm by Qiang Shen and Richard Jensen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have unlabeled data, check &lt;a href=&quot;http://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Component Analysis (PCA)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9787" LastEditorUserId="84" LastEditDate="2015-05-24T10:07:50.763" LastActivityDate="2015-05-24T10:07:50.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5870" PostTypeId="2" ParentId="5679" CreationDate="2015-05-22T22:39:53.793" Score="2" Body="&lt;p&gt;Well the text doesn't get converted into data points ...&#xA;Let's say we are doing sentence level opinion mining..&#xA;Features are extracted from a sentence .&#xA;Now it depends on case to case as to what features to use...&#xA;A common one is bag of words models in which features become distinct words in sentence and the value of features are the frequency it is repeated in a sentence. Those frequencies are your data points. &lt;/p&gt;&#xA;" OwnerUserId="9787" LastActivityDate="2015-05-22T22:39:53.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5872" PostTypeId="2" ParentId="5267" CreationDate="2015-05-22T22:48:36.107" Score="0" Body="&lt;p&gt;Model is as a stochastic process . &#xA;Markov chains is the way to go. &#xA;Create a stochastic matrix where states could be&#xA;  A team T  ...get all the combinations possible and use past data to get initial probabilities of winning ...and then use the beautiful property of Xn= XiP^n&#xA;Where Xn is probability vector of nth stage from now and Xi is initial vector and P is probability transition matrix .&lt;/p&gt;&#xA;" OwnerUserId="9787" LastActivityDate="2015-05-22T22:48:36.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5874" PostTypeId="1" AcceptedAnswerId="5892" CreationDate="2015-05-23T02:20:24.870" Score="1" ViewCount="65" Body="&lt;p&gt;I have customer data since 2013 and there is a file which has the customer unique id, a timestamp, and the reason for the call (a drop down from the person who handled the call).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some cumulative counts based on customer ID and the timestamp and I saw that one customer called in over 1000 times alone. What's the best way to make sense of the call driver data when I'm looking at millions of rows and around 200 categories of call types?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a broader topic which looks into 'downstream'  issues or predicting the probability of future calls or events?  The end goal would be to visualize these calling patterns and focus on reducing the call backs. This is a specific problem but it seems like it should be common and I can learn about addressing it in a bigger picture manner. &lt;/p&gt;&#xA;" OwnerUserId="9788" LastActivityDate="2015-05-24T16:11:35.860" Title="What techniques are used to understand call patterns?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5875" PostTypeId="1" CreationDate="2015-05-23T11:45:23.043" Score="4" ViewCount="1422" Body="&lt;p&gt;I am working on a  classification problem. I have 1000+ features in this dataset. I don't know how to select the right variables/ features that can actually contribute to predicting the output. What are the different methods through which I can identify the important variables that can be used out of these 1000+ variables.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="847" LastEditDate="2015-05-28T12:18:04.847" LastActivityDate="2015-05-29T00:41:17.407" Title="How to find the input variables for a classification problem?" Tags="|classification|algorithms|feature-selection|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5877" PostTypeId="2" ParentId="5875" CreationDate="2015-05-23T13:09:29.803" Score="5" Body="&lt;p&gt;Two approaches come to mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use all of them and perform feature selection to identify indicative features. Algorithms include Information Gain and Mutual Information.&lt;/li&gt;&#xA;&lt;li&gt;Hand-pick some features you deem indicative intuitively, or delete the ones you deem non-indicative.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Generally, you have some sort of hypothesis that you are testing where you attest the capability of (a subset of) your features (and the algorithm you'll use) to uniquely identify the label/class they belong to. The features you use are one of the most important things in the classification task. So it is good to spend time to intelligently find out which features contribute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this task, you do have a lot of possible features so the automatic feature selection approach might be better.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-05-23T13:09:29.803" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5878" PostTypeId="2" ParentId="5860" CreationDate="2015-05-23T20:45:02.473" Score="4" Body="&lt;p&gt;A somewhat popular introduction is Andrew Ng's Stanford &lt;a href=&quot;https://www.youtube.com/playlist?list=PLA89DCFA6ADACE599&quot; rel=&quot;nofollow&quot;&gt; machine learning lectures&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8354" LastActivityDate="2015-05-23T20:45:02.473" CommentCount="1" CommunityOwnedDate="2015-05-23T20:45:02.473" ContentLicense="CC BY-SA 3.0" />
  <row Id="5879" PostTypeId="2" ParentId="3751" CreationDate="2015-05-23T20:56:03.840" Score="4" Body="&lt;p&gt;Start with neural nets. They are the basic building blocks of Deep Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=g-BJSl4zV_g&quot; rel=&quot;nofollow&quot;&gt;Neural Nets for Newbies video&lt;/a&gt; is a well-explained introduction. &lt;a href=&quot;http://www.willamette.edu/~gorr/classes/cs449/intro.html&quot; rel=&quot;nofollow&quot;&gt;CS-449&lt;/a&gt; from Willamette University is an free and approachable class in neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should lay the foundation for more advanced Deep Learning material. &lt;a href=&quot;http://www-labs.iro.umontreal.ca/~bengioy/dlbook/&quot; rel=&quot;nofollow&quot;&gt;Deep Learning&lt;/a&gt; by Bengio, Goodfellow, and Courville is a quality book on the subject.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2015-05-23T20:56:03.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5880" PostTypeId="2" ParentId="5860" CreationDate="2015-05-23T21:13:32.363" Score="2" Body="&lt;p&gt;Take a look at this free class &lt;a href=&quot;https://www.udacity.com/course/intro-to-artificial-intelligence--cs271&quot; rel=&quot;nofollow&quot;&gt;Intro to Artificial Intelligence&lt;/a&gt; from Udacity. One of the instructors is Norvig. The class is more suitable for beginners than Norvig's book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though this class doesn't have programming exercise, it explains concepts so well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its follow up class &lt;a href=&quot;https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373&quot; rel=&quot;nofollow&quot;&gt;Artificial Intelligence for Robotics&lt;/a&gt; has programming exercises and does a fantastic job in explaining Partical Filters.&lt;/p&gt;&#xA;" OwnerUserId="9593" LastActivityDate="2015-05-23T21:13:32.363" CommentCount="2" CommunityOwnedDate="2015-05-23T21:13:32.363" ContentLicense="CC BY-SA 3.0" />
  <row Id="5881" PostTypeId="1" CreationDate="2015-05-23T21:49:17.603" Score="0" ViewCount="105" Body="&lt;p&gt;While I was studying, few years ago , one of the most interesting topic was evolution, genetic algorithms and neural networks. Many of the problems I faced could be solved by using that knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that since that time world has found more interesting algorithms, do you recommend some books worthy of reading in that domain ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mainly I am looking a way to find patterns in huge amount of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex. Having energy consumption for few years for one building.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets have an algorithm that is trying to find all possible repeatings in many variations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously at the beginning it should find that at the weekend energy consumption is less then average between Mon and Fri, but is that possible that an algorithm would tell me sth like this ? :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every Friday at third week in even months user sleeps for 3 hours and in uneven months 5 hours ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or the algorithm finds it self that user like to save energy so if previous month he sees that he spent more then usual , next month he is trying to spent less, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or lets assume that user eats breakfast at work but if he eats it at home then he will stay whole day at home, then lets assume that he has off day, then check energy usage after breakfast if is high, which means user is preparing to leave or is small which means that he wants to sleep and basically stays home. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was wondering if this possible to auto detect this kind of patterns ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am enthusiast of c# and interested in R.&lt;/p&gt;&#xA;" OwnerUserId="9799" LastEditorUserId="9799" LastEditDate="2015-06-02T21:46:13.043" LastActivityDate="2015-06-02T21:46:13.043" Title="Data prediction book" Tags="|algorithms|neural-network|genetic-algorithms|" AnswerCount="2" CommentCount="2" ClosedDate="2015-06-01T22:21:05.903" ContentLicense="CC BY-SA 3.0" />
  <row Id="5882" PostTypeId="2" ParentId="5863" CreationDate="2015-05-24T00:07:19.097" Score="4" Body="&lt;p&gt;you can trace the squared error in statistics through multivariate calculus all the way to Pythagorus. You are basically calculating the 'effective length' of the error, the hypotenuse, among errors from multiple variables $(X_1 - X_2)^2 + (Y_1 - Y_2)^2 + ...$ like in a triangle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But where did the square root go?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somebody realized that calculating roots of multiple variables over multiple iterations is computationally very expensive. So they decided to drop it. Checkout the squared Euclidean distance &lt;a href=&quot;http://en.wikipedia.org/wiki/Euclidean_distance#Squared_Euclidean_distance&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for more details&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would a cubic error or a logarithmic error affect the outcome?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It just takes more time to converge because they are not as accurate. But we do see logarithmic errors over squares such as logistic regression where it is more optimal&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all it is a simple case of optimization&lt;/p&gt;&#xA;" OwnerUserId="9801" LastEditorUserId="9801" LastEditDate="2015-06-07T19:45:06.300" LastActivityDate="2015-06-07T19:45:06.300" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5883" PostTypeId="2" ParentId="5860" CreationDate="2015-05-24T00:42:07.003" Score="1" Body="&lt;p&gt;If you are looking specifically for books, the leaders in the industry have a variety of recommendations as posted on reddit AMA's of machine learning. One specific book I recommend, which is in preparation at the time, is Deep Learning by Yoshua Bengio et al &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of resources outside books to learn from as posted &lt;a href=&quot;http://zhengrui.github.io/zerryland/ML-CV-Resource.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://digitalmind.io/post/deep-learning&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depends on how much time you have. If you are in it for the long haul, start by brushing up your knowledge on math, specifically Linear Algebra, Calculus, Probability and Statistics&lt;/p&gt;&#xA;" OwnerUserId="9801" LastActivityDate="2015-05-24T00:42:07.003" CommentCount="1" CommunityOwnedDate="2015-05-24T00:42:07.003" ContentLicense="CC BY-SA 3.0" />
  <row Id="5884" PostTypeId="2" ParentId="5881" CreationDate="2015-05-24T00:51:22.027" Score="2" Body="&lt;p&gt;please refer &lt;a href=&quot;https://datascience.stackexchange.com/questions/5860/where-to-start-which-books&quot;&gt;this&lt;/a&gt; question &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Books recommended by Yann Lecun can be found &lt;a href=&quot;http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_yann_lecun&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and by G. Hinton &lt;a href=&quot;http://www.reddit.com/r/MachineLearning/comments/2lmo0l/ama_geoffrey_hinton&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9801" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-05-24T00:51:22.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5885" PostTypeId="1" AcceptedAnswerId="5888" CreationDate="2015-05-24T02:11:36.640" Score="16" ViewCount="48747" Body="&lt;p&gt;I'm using &lt;a href=&quot;https://www.npmjs.com/package/brain&quot; rel=&quot;noreferrer&quot;&gt;Brain&lt;/a&gt; to train a neural network on a feature set that includes both positive and negative values. But Brain requires input values between 0 and 1. What's the best way to normalize my data?&lt;/p&gt;&#xA;" OwnerUserId="9802" LastEditorUserId="9802" LastEditDate="2018-11-06T21:14:33.163" LastActivityDate="2021-03-11T20:08:46.913" Title="How to scale an array of signed integers to range from 0 to 1?" Tags="|machine-learning|neural-network|feature-scaling|normalization|javascript|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5886" PostTypeId="2" ParentId="5885" CreationDate="2015-05-24T03:12:06.020" Score="4" Body="&lt;p&gt;Find the largest positive number and the smallest (most negative) number in the array.&#xA;Add the absolute value of the smallest (most negative) number to every value in the array.&#xA;Divide each result by the difference between the largest and the smallest number.&lt;/p&gt;&#xA;" OwnerUserId="9804" LastActivityDate="2015-05-24T03:12:06.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5887" PostTypeId="2" ParentId="5885" CreationDate="2015-05-24T03:48:42.427" Score="1" Body="&lt;p&gt;Say you have a vector/array of values &lt;code&gt;v = [1, -2, 3]&lt;/code&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;minV = Math.min.apply(Math, v);;&#xA;for(var i=0; i&amp;lt;v.length; i++) {v[i] -= minV;}&#xA;maxV = Math.max.apply(Math, v);;&#xA;for(var i=0; i&amp;lt;v.length; i++) {v[i] /= ( maxV - minV );}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output at the end will be &lt;code&gt;v = [0.6, 0, 1]&lt;/code&gt;. Explanation:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Pushing the entire range of values to start from 0, so that we have no negatives&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Dividing the values by (max - min) of range, so that max will be 1&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9801" LastEditorUserId="29169" LastEditDate="2021-03-11T20:08:46.913" LastActivityDate="2021-03-11T20:08:46.913" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5888" PostTypeId="2" ParentId="5885" CreationDate="2015-05-24T05:50:50.530" Score="26" Body="&lt;p&gt;This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$ &lt;/p&gt;&#xA;" OwnerUserId="9766" LastActivityDate="2015-05-24T05:50:50.530" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5889" PostTypeId="2" ParentId="5650" CreationDate="2015-05-24T10:04:25.767" Score="2" Body="&lt;p&gt;In the third paragraph of the first section (page 1), they define $\#(w)$, $\#(w; c)$, and $\#(c)$. Quoting such paragraph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Word-context pairs are denoted as $(w; c)$, and $\#(w; c)$ counts all observations of $(w; c)$ from the corpus. We use $\#( w ) =  \sum_c \#(w; c)$ and $\#(c) = \sum_w \#(w; c)$ to refer to the count of occurrences of a word (context) in all word-context pairs. Either $\sum_c \#(c)$ or  $\sum_w \#(w)$ may represent the count of all word-context pairs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, considering the equation you cited:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$\#(w_i, c_j)$ counts the occurrences of a specific word $w_i$ over the context $c_j$;&lt;/li&gt;&#xA;&lt;li&gt;and $\frac{\#(c_)}{\sum_w \#(w)}$ accounts for the the probability of having context $c_j$ in the dataset.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="84" LastActivityDate="2015-05-24T10:04:25.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5890" PostTypeId="1" CreationDate="2015-05-24T11:54:43.347" Score="5" ViewCount="352" Body="&lt;p&gt;Consider a talent pool in which each member has some set of skills. Some of these talent are submitted to orders as potential candidates of which one is selected. It is reasonable to assume that the submitted talent have some dominant thing in common in their skill sets (let's call it a segment) that qualifies them for the order. Example segments are &quot;front end web-designer&quot; or &quot;brochure / sprint designer&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the total set of skills over all of the talent submitted to an order (like 2-5 with say 10 skills each, so 20 - 50 skills total), I am looking for the dominant segment. Then, I am looking for the dominant segment for each individual talent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My plan is to use latent Dirichlet allocation (LDA) such that the skills of all the talent submitted for an order are a &quot;document&quot; that contains some segments or &quot;topics&quot; with some probability. Likely, there will be one or two dominant topics depending on the total topic number. I will then use this model to predict the dominant segment for each talent where the individual talent skill set is a &quot;document&quot; with some segments or &quot;topics&quot; within.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious if anyone has feedback about my use of LDA or other ideas about how I might go about discovering these segments?&lt;/p&gt;&#xA;" OwnerUserId="9807" LastEditorUserId="182" LastEditDate="2015-05-24T17:52:45.137" LastActivityDate="2015-05-25T18:06:22.790" Title="Different methods for clustering skills in text" Tags="|machine-learning|data-mining|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5892" PostTypeId="2" ParentId="5874" CreationDate="2015-05-24T16:11:35.860" Score="0" Body="&lt;p&gt;When you said &quot;a broader topic&quot;, did you mean what algorithms to use to examine the event log with the goal of reducing future calls from the same customer on the same topic? In other words, a customer may call for help for a different topic. If a customer calls in for the same topic repetitively, something can be improved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may get ideas from a Coursera class &lt;a href=&quot;https://www.coursera.org/course/procmin&quot; rel=&quot;nofollow&quot;&gt;Processing Mining&lt;/a&gt; since the issue you're solving is similar to the example of a spaghetti process in lecture 6.7:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Spaghetti process describing the diagnosis and treatment of 2765 patients in a Dutch hospital. The process model was constructed based on an event log containing 114,592 events. There are 619 different activities (taking event types into account) executed by 266 different individuals (doctors, nurses, etc.).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, you can click the drop down menu under &quot;Sessions&quot;, choose &quot;April 1, 2015 to May 19, 2015&quot; then you can register and view the lectures. On the right of each lecture, there are some icons. The first icon is to download slides for the lecture. You may find reading the slides is faster than listening to the lecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose a customer called for installation of software release 1.5 then called a day later for running the new features of software release 1.5. Are these two issues logged as two of the 200 call categories? If so, we can use a time period (say one week) to judge whether it is about the same topic. Within a short time period, a customer is likely to work on the same topic, especially with the same key words such as &quot;software release 1.5&quot;. We can coach the call center employees on solving possible follow up questions by saying something like &quot;now that we finished installation, let me show you a couple of new features. It'll save you time.&quot; This will reduce the number of calls on the same topic from the same customer.&lt;/p&gt;&#xA;" OwnerUserId="9593" LastActivityDate="2015-05-24T16:11:35.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5893" PostTypeId="1" AcceptedAnswerId="5902" CreationDate="2015-05-24T21:45:02.207" Score="10" ViewCount="18699" Body="&lt;p&gt;I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and filtering?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a huge amount of human text input of variable length (searchterms and whole sentences (up to 200 characters) ) over several years. The text contains a lot of spam (like machine input from bots, single words, stupid searches, product searches ... ) and only a few % of seems to be useful. I realised that sometimes (only very rarely) people search my side by asking really cool questions. These questions are so cool, that i think it is worth to have a deeper look into them to see how people search over time and what topics people have been interested in using my website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is that i am really struggling with the preprocessing (i.e. dropping the spam). I already tried some stopword list from the web (NLTK etc.), but these don't really help my needs regarding this dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your ideas and discussion folks!&lt;/p&gt;&#xA;" OwnerUserId="4676" LastEditorUserId="172" LastEditDate="2015-05-25T11:19:37.453" LastActivityDate="2017-09-20T23:41:42.930" Title="How to create a good list of stopwords" Tags="|data-mining|nlp|information-retrieval|language-model|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5894" PostTypeId="2" ParentId="155" CreationDate="2015-05-25T06:32:29.107" Score="12" Body="&lt;p&gt;I came across this collection on Github. The collection is categorised as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/caesar0301/awesome-public-datasets&quot;&gt;https://github.com/caesar0301/awesome-public-datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for the part regarding &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can not a open-source model for sharing data sets devised for data scientists?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;you can refer &lt;a href=&quot;https://github.com/jtleek/datasharing&quot;&gt;The Leek group guide to data sharing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="75" LastEditDate="2015-05-25T06:41:34.577" LastActivityDate="2015-05-25T06:41:34.577" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="5895" PostTypeId="5" CreationDate="2015-05-25T07:22:43.650" Score="0" Body="&lt;h2&gt;What is scikit-learn?&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; is a popular machine learning package for Python that has simple and efficient tools for predictive data analysis. Topics include classification, regression, clustering, dimensionality reduction, model selection, and preprocessing. It is built upon &lt;a href=&quot;https://numpy.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;NumPy&lt;/a&gt;, &lt;a href=&quot;https://www.scipy.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;SciPy&lt;/a&gt;, and &lt;a href=&quot;https://matplotlib.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;matplotlib&lt;/a&gt; and is open-sourced under the BSD License. It is part of the scientific computation ecosystem and useful for both individual and commercial use.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;New to scikit-learn?&lt;/h2&gt;&#xA;&lt;p&gt;There are various resources including books, tutorials/workshops, etc. for those looking to learn how to use scikit-learn.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A popular introductory tutorial is:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;SciPy 2018 Conference Tutorial:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=4PXAztQtoTg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning with scikit-learn - Part 1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=gK43gtGh49o&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning with scikit-learn - Part 2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;A popular introductory book is:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1449369413&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Machine Learning with Python&lt;/a&gt;&lt;/em&gt;, by Andreas C. Müller and Sarah Guido.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;&lt;a href=&quot;/questions/tagged/scikit-learn&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; aria-label=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; rel=&quot;tag&quot; aria-labelledby=&quot;scikit-learn-container&quot;&gt;scikit-learn&lt;/a&gt; Tag usage&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;When posting questions about scikit-learn, please take the following into consideration:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When tagging questions with the &lt;a href=&quot;/questions/tagged/scikit-learn&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; aria-label=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; rel=&quot;tag&quot; aria-labelledby=&quot;scikit-learn-container&quot;&gt;scikit-learn&lt;/a&gt; tag, users should &lt;em&gt;not&lt;/em&gt; use the tag sklearn, despite semantic similarity, as the latter is marked as a synonym and will automatically be retagged.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explicit programming related questions are more suitable for Stack Overflow and should &lt;em&gt;not&lt;/em&gt; be posted on Stack Exchange Data Science.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Questions should include sufficient details and clarity to be able to provide support for the problem at hand. This includes linking to underlying data used, providing code used for the model's construction, highlighting relevant outputs, etc.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;External Resources&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;: Documentation page&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;: GitHub page&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;Important links&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;HTML documentation (development version): &lt;a href=&quot;http://scikit-learn.org/dev/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/dev/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Download releases: &lt;a href=&quot;http://sourceforge.net/projects/scikit-learn/files/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://sourceforge.net/projects/scikit-learn/files/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Issue tracker: &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/issues&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Mailing list: &lt;a href=&quot;https://lists.sourceforge.net/lists/listinfo/scikit-learn-general&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://lists.sourceforge.net/lists/listinfo/scikit-learn-general&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="75" LastEditorUserId="29169" LastEditDate="2022-11-05T00:33:34.657" LastActivityDate="2022-11-05T00:33:34.657" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5896" PostTypeId="4" CreationDate="2015-05-25T07:22:43.650" Score="0" Body="scikit-learn is a popular machine learning package for Python that has simple and efficient tools for predictive data analysis. Topics include classification, regression, clustering, dimensionality reduction, model selection, and preprocessing." OwnerUserId="75" LastEditorUserId="29169" LastEditDate="2021-03-23T17:10:48.920" LastActivityDate="2021-03-23T17:10:48.920" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5897" PostTypeId="1" AcceptedAnswerId="5898" CreationDate="2015-05-25T07:54:09.283" Score="2" ViewCount="139" Body="&lt;p&gt;How important is linear algebra to being a data scientist? Are we talking college postgraduate level?&lt;/p&gt;&#xA;" OwnerUserId="4836" LastEditorUserId="182" LastEditDate="2015-05-26T06:35:14.043" LastActivityDate="2015-05-26T06:35:14.043" Title="How deep should ones linear algebra knowledge be before starting data science?" Tags="|education|" AnswerCount="1" CommentCount="1" ClosedDate="2015-05-26T06:37:00.573" ContentLicense="CC BY-SA 3.0" />
  <row Id="5898" PostTypeId="2" ParentId="5897" CreationDate="2015-05-25T08:57:19.877" Score="2" Body="&lt;p&gt;I think it truly depends on what you decide to specialize in. Data science is a very broad field, and you can actually work with data without knowing what eigenvalues and eigenvectors are. However, if you want to acquire a intermediate/advanced understanding of statistics or machine learning, you need at least an intermediate/advanced knowledge of linear algebra.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest to take an introductory class on linear algebra on MOOC - just to have a more precise idea of what linear algebra is - and then study some other topics that you are interested in. Linear algebra is a useful tool, but it can be very boring, especially if you are an &quot;applied&quot; kind of guy. Moreover, I think that some concepts like eigenvalues or eigenvectors are easier to understand when seen in an applied context, e.g. principal component analysis.&lt;/p&gt;&#xA;" OwnerUserId="9766" LastActivityDate="2015-05-25T08:57:19.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5899" PostTypeId="1" CreationDate="2015-05-25T12:37:20.030" Score="1" ViewCount="66" Body="&lt;p&gt;I have data of the form :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Id1      A_Id2      B_Id2       C_Id2       D_Id2      E_Id2      F_Id2&#xA;1         6           3           9           23         20         5&#xA;1         4           7           8           9          11         56                                                   &#xA;1         2           36          98          73         2          4     &#xA;1         9           5           2           7          32         24           &#xA;1         14          7           5           9          12         5                                                   &#xA;2         34          4           7           10          7         12                                                       &#xA;2         5           57          23          91          4         6                                                    &#xA;2         7           .           .           .           .         .&#xA;2         3           .           .           .           .         .&#xA;2         .           .           .           .           .         .&#xA;.&#xA;.&#xA;100      .            .           .           .           .         .            &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, I want to build a model such that I can get a best match of Id1, given top 5 Id2 matches of each attribute(A_Id2, B_Id2,...,F_Id2). Now every match should be computed keeping in mind A has highest priority, followed by B and C, followed by D and least priority to E and F. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the output will look like this :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Id1    Match_Id2&#xA;1        9&#xA;2        7&#xA;3        .&#xA;4        .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope the problem is clear, if not please ask.&#xA;How should I go about building a Machine Learning model for this? I was wondering if ranking algorithm will help ?&lt;/p&gt;&#xA;" OwnerUserId="8657" LastEditorUserId="8657" LastEditDate="2015-05-25T13:13:35.463" LastActivityDate="2015-05-25T13:13:35.463" Title="Do I need to apply a Ranking Algorithm for this?" Tags="|machine-learning|algorithms|predictive-modeling|ranking|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5901" PostTypeId="2" ParentId="5737" CreationDate="2015-05-25T15:13:50.583" Score="1" Body="&lt;p&gt;The first idea in my mind is : view it forming a social graph where nodes are the email-ids (people) and 2 nodes are connected if they communicate with each other. You can also view it as a weighted graph where weight comes from the frequency of conversation and you can add the sense of direction as well using the sender-receiver information. Now you can apply all sort of social network analysis on this.&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="75" LastEditDate="2015-05-26T07:54:15.237" LastActivityDate="2015-05-26T07:54:15.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5902" PostTypeId="2" ParentId="5893" CreationDate="2015-05-25T15:19:21.227" Score="5" Body="&lt;p&gt;One approach would be to use &lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;tf-idf&lt;/a&gt; score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries from the bad ones.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2015-05-25T15:19:21.227" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5903" PostTypeId="2" ParentId="5893" CreationDate="2015-05-25T15:19:34.493" Score="1" Body="&lt;p&gt;Using TFIDF (term frequency inverse document frequency) will solve your purpose. Get the TFIDF score for each word in your document and sort the words by their scores by which you can select the important words in your data.&lt;/p&gt;&#xA;" OwnerUserId="9828" LastActivityDate="2015-05-25T15:19:34.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5904" PostTypeId="2" ParentId="5663" CreationDate="2015-05-25T15:27:09.517" Score="2" Body="&lt;p&gt;A good introductory course for social network analysis : &lt;a href=&quot;https://www.coursera.org/course/sna&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/course/sna&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2015-05-25T15:27:09.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5907" PostTypeId="2" ParentId="5890" CreationDate="2015-05-25T18:06:22.790" Score="2" Body="&lt;p&gt;You may want to consider preprocessing - convert different wording for the same type of talent to same wording. For example, a talent in machine learning is called data scientist at Coursera job site, data engineer at Udacity job site, or data analyst. This preprocessing is similar to &lt;a href=&quot;http://en.wikipedia.org/wiki/Stemming&quot; rel=&quot;nofollow&quot;&gt;stemming&lt;/a&gt; in concept.&lt;/p&gt;&#xA;" OwnerUserId="9593" LastActivityDate="2015-05-25T18:06:22.790" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5908" PostTypeId="2" ParentId="5875" CreationDate="2015-05-26T05:11:36.160" Score="2" Body="&lt;p&gt;The general approach in feature selection is to get a score of each feature in the data set and select top features. We can run algorithms like GBM or Random Forest on all the variables simply to get a ranking of variable importance. We can also use χ² (chi-squared) statistic with cross validation to select a user-specified percentile of features with the highest scoring. But, the disadvantage with these approaches is not detecting correlations between features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can also use backward elimination: features are tested one by one and the ones that are not statically significant are deleted from data set. In forward selection that starts with out any variable in dataset and then adds variables that are statically significant. Hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="9828" LastActivityDate="2015-05-26T05:11:36.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5909" PostTypeId="2" ParentId="5693" CreationDate="2015-05-26T11:26:14.063" Score="2" Body="&lt;p&gt;One approach can be to use a no-SQL database running on top of a distributed store (like Cassandra or Hbase). Add an external index which support spatial indexing (say elastic) for fast search. This makes your solution scalable (due to distributed store) and fast enough for spatial search queries.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2015-05-26T11:26:14.063" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5910" PostTypeId="1" CreationDate="2015-05-26T12:36:54.197" Score="4" ViewCount="359" Body="&lt;p&gt;I am not sure if &quot;minimize correlation&quot; is the right title for this issue but I could not find a better sentence to describe what I would like to achieve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I have a black box with multiple inputs and a single output. I know one of the inputs and the output and I have multiple example recordings of both. This known input modifies the output in a way that it is not desired, therefore, I would like to get rid of this &quot;noise&quot; caused by the known input. The transfer function for this input can be safely assumed as linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am doing right now, it is to loop through the example recordings, creating a linear regression model to predict the unwanted outcome and subtracting it from the real measured output signal, for each example. Afterwards, I compute the average of all the fixed output signals to reveal meaningful data beyond noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This strategy seems to work according to the following plot:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/oAdyb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X axis is the known input signal, Y axis is the output signal, blue and green dots represent the averaged data before and after applying the linear regression algorithm, respectively. Lines are the best fit for each data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the green line (&quot;cleaned&quot; dataset) has the smallest slope, meaning that the output variable is considerably less linearly correlated with the input than it was previously. Therefore, I assume that the regression technique explained before is working as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question, looking at the plot, is there any mathematical procedure to directly &quot;project&quot; the original dataset in a way that the correlation between the input and output variables is minimized? Is there any math trick to avoid the use of the regression technique on all the example datasets to obtain a similar result? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My written expression is not the best so please feel free to comment the question if you need further explanations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any code is welcomed but python (pandas, numpy, etc.) and Matlab are preferred. Theoretical explanations are also very welcomed. &lt;/p&gt;&#xA;" OwnerUserId="9838" LastEditorUserId="8953" LastEditDate="2015-05-26T14:09:21.700" LastActivityDate="2020-08-02T17:44:56.477" Title="Minimize correlation between input and output of black box system" Tags="|python|regression|correlation|matlab|linear-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5913" PostTypeId="1" CreationDate="2015-05-26T22:11:54.650" Score="2" ViewCount="173" Body="&lt;p&gt;How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such reason is we need to meet deadlines for reports &amp;amp; publications. I am wondering if anyone has adopted or come across a method/guidelines that help to find issues within the data (time-saving tips), so we can analyze the data efficiently. Please share your experience, tips, etc. &lt;/p&gt;&#xA;" OwnerUserId="7923" LastEditorUserId="21" LastEditDate="2017-05-19T16:12:04.230" LastActivityDate="2017-05-19T16:12:04.230" Title="How do you ascertain the quality of your data?" Tags="|dataset|data-cleaning|" AnswerCount="2" CommentCount="1" ClosedDate="2015-05-28T12:17:32.320" ContentLicense="CC BY-SA 3.0" />
  <row Id="5915" PostTypeId="2" ParentId="2667" CreationDate="2015-05-27T03:38:45.067" Score="2" Body="&lt;p&gt;You can use &lt;a href=&quot;https://github.com/tinkerpop/rexster/wiki/Basic-REST-API&quot; rel=&quot;nofollow&quot;&gt;Rexster&lt;/a&gt; to expose a REST api and then you can use this api via &lt;a href=&quot;http://docs.python-requests.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Requests&lt;/a&gt; module (there are many other modules to do this task - though Requests is what I prefer)&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2015-05-27T03:38:45.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5916" PostTypeId="1" AcceptedAnswerId="5919" CreationDate="2015-05-27T05:41:21.753" Score="5" ViewCount="180" Body="&lt;p&gt;I'm using Google Analytics on my mobile app to see how different users use the app. I draw a path based on the pages they move to. Given a list of paths for say a 100 users, how do I go about clustering the users. Which algorithm to use? By the way, I'm thinking of using sckit learn package for the implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset (in csv) would look like this :  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DeviceID,Pageid,Time_spent_on_Page,Transition.&amp;lt;br&amp;gt; &#xA;ABC,Page1, 3s, 1-&amp;gt;2.&amp;lt;br&amp;gt;&#xA;ABC,Page2, 2s, 2-&amp;gt;4.&amp;lt;br&amp;gt;&#xA;ABC,Page4,1s,4-&amp;gt;1.&amp;lt;br&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the path, here is 1-&gt;2-&gt;4-&gt;1, where 1,2,4 are Pageids.&lt;/p&gt;&#xA;" OwnerUserId="9852" LastEditorUserId="1367" LastEditDate="2015-05-28T12:17:55.100" LastActivityDate="2015-05-28T12:17:55.100" Title="How to cluster a link traversal dataset" Tags="|machine-learning|clustering|scikit-learn|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="5917" PostTypeId="2" ParentId="5916" CreationDate="2015-05-27T06:12:44.633" Score="1" Body="&lt;p&gt;I have not worked with such a dataset myself but I think you can model this problem as a graph where the pages form the node and then you have directed edges based on transition. Add weights to nodes based on time spent on them and then use graph clustering algorithms. If you choose to use this, you can use the &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;networkx&lt;/a&gt; library in python for graph based analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : We can use the information about different possible paths and how frequently they are use to classify the users. Let's take example of Google search app. Suppose I want to search for images. One option is that I use the image search option, make the query and reach the results page. Other is that I make the query first and then switch to image option after getting the results. In both the cases I did up at the same page. I can use this information to classify my users. Now there can be quite a lot of paths possible so which all do I consider? The graph can be used to leveraged here along with the information about how you want to classify your users. Modeling it as a graph looks very intuitive to me as it lends itself to the concept of path.&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="75" LastEditDate="2015-05-27T09:23:39.450" LastActivityDate="2015-05-27T09:23:39.450" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5918" PostTypeId="1" AcceptedAnswerId="5920" CreationDate="2015-05-27T07:53:53.170" Score="1" ViewCount="275" Body="&lt;p&gt;While running rattle in my system I am getting this error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rattle()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error: attempt to apply non-function&#xA;In addition: Warning message:&#xA;In method(obj, ...) : Unknown internal child: selection&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R version 3.1.0 (2014-04-10)&lt;/p&gt;&#xA;" OwnerUserId="8729" LastEditorUserId="843" LastEditDate="2017-03-08T15:21:21.153" LastActivityDate="2017-03-08T15:21:21.153" Title="How to install rattle in centos" Tags="|r|rattle|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5919" PostTypeId="2" ParentId="5916" CreationDate="2015-05-27T08:02:10.797" Score="4" Body="&lt;p&gt;@Shagun's answer is right actually. I just expand it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 different approaches to your problem:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Graph Approach&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;As stated in @Shagun's answer you have a weighted directed graph and you want to cluster the paths. I mention again because it's important to know that your problem is not a &lt;em&gt;Graph Clustering&lt;/em&gt; or &lt;em&gt;Community Detection&lt;/em&gt; problem where vertices are clustered!&lt;/li&gt;&#xA;&lt;li&gt;Cunstructing a Graph in networkx using the last two column of the data, you can add time spent as weight and users who passed that link as an edge attribute. After all you'll have different features for clustering: the set of all vertices an individual ever met in the graph, total, mean and std of time spent, shortest path distribution parameters, ... which can be used for clustering the user behaviors.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Standard Data&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All above can be done by reading data efficiently in a matrix. If you consider each edge for a specified user as a single row (i.e. you'll have &lt;strong&gt;M&lt;/strong&gt;x&lt;strong&gt;N&lt;/strong&gt; rows where &lt;strong&gt;M&lt;/strong&gt; is the number of users and &lt;strong&gt;N&lt;/strong&gt; the number of edges in case you stick with 100 case!) and add properties as columns you'll probably able to cluster behaviors. if a user passed an edge &lt;em&gt;n&lt;/em&gt; times, in the row corresponding to that user and that edge add a count column with value &lt;em&gt;n&lt;/em&gt; and same for time spend, etc. Starting and ending edges are also informative. Be careful that node names are categorical variables.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Regarding clustering algorithms you can find enough if you have a quick look at SKlearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helped.&#xA;Good Luck :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-27T08:02:10.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5920" PostTypeId="2" ParentId="5918" CreationDate="2015-05-27T09:41:38.917" Score="2" Body="&lt;p&gt;I got the answer.&#xA;I have to install some of the packages in terminal . I installed it and it works.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sudo yum install gtk+-devel gtk2-devel&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8729" LastActivityDate="2015-05-27T09:41:38.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5921" PostTypeId="1" AcceptedAnswerId="5939" CreationDate="2015-05-27T10:02:03.657" Score="5" ViewCount="901" Body="&lt;p&gt;I am searching for pointers to algorithms for feature detection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: all the answers helped me a lot, I cannot decide which one I should accept. THX guys!&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What I did:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For discrete variables (i.e. $D_i, E$ are finite sets) $X_i : \Omega \to D_i$ and a given data table &#xA;$$ &#xA;\begin{pmatrix}{}&#xA;  X_1 &amp;amp; ... &amp;amp; X_n &amp;amp; X_{n+1} \\&#xA;  x_1^{(1)} &amp;amp; ... &amp;amp; x_n^{(1)} &amp;amp; x_{n+1}^{(1)} \\&#xA;  ... \\&#xA;  x_1^{(m)} &amp;amp; ... &amp;amp; x_n^{(m)} &amp;amp; x_{n+1}^{(m)} \\&#xA;\end{pmatrix}&#xA;$$&#xA;(the last variable will be the 'outcome', thats why I stress it with a special index) and $X, Y$ being some of the $X_1, ..., X_{n+1}$ (so if $X=X_a, Y=X_b$ then $D=D_a, E=D_b$) compute&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H(X) = - \sum_{d \in D} P[X=d] * log(P[X=d])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H(Y|X) = - \sum_{d \in D} {&#xA;             P[X=d] * \sum_{e \in E} {&#xA;               P[Y=e|X=d] * log(P[Y=e|X=d])&#xA;             }&#xA;           }$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where we estimate&#xA;  $$P[X_a=d] = |\{j \in \{1, ..., m\} : x_a^{(j)} = d\}|$$&#xA;and analogously&#xA;  $$P[X_a=d \cap X_b=e] = |\{j \in \{1, ..., m\} : x_a^{(j)} = d ~\text{and}~ x_b^{(j)}=e\}|$$&#xA;and then&#xA;  $$I(Y;X) = \frac{H(Y) - H(Y|X)}{\text{log}(\text{min}(|D|, |E|))}$$&#xA; which is to be interpreted as the influence of $Y$ on $X$ (or vice versa, its symmetric).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: A little late now but still:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is wrong:&#xA;&lt;s&gt;Exercise for you: show that if $X=Y$ then $I(X,Y)=1$.&lt;/s&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is correct:&#xA;Exercise for you: show that if $X=Y$ then $I(X,X)=H(X)/log(|D|)$ and if $X$ is additionally equally distributed then $I(X,X)=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For selecting features start with the available set $\{X_1, ..., X_n\}$ and a set 'already selected'$ = ()$ [this is an ordered list!]. We select them step by step, always taking the one that maximizes   $$\text{goodness}(X) = I(X, X_{n+1}) - \beta \sum_{X_i ~\text{already selected}} I(X, X_i)$$ for a value $\beta$ to be determined (authors suggest $\beta = 0.5$). I.e. goodness = influence on outcome - redundancy introduced by selecting this variable. After doing this procedure, take the first 'few' of them and throw away the ones with lower rank (whatever that means, I have to play with it a little bit). This is what is described in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74.7629&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For computing the $I$ for continuous variables one needs to bin them in some way. More concretely, the inventors of 'I' suggest to take the maximal value over binning $X$ into $n_x$ bins, $Y$ into $n_y$ bins and $n_x \cdot n_y &amp;lt;= m^{0.6}$, i.e.  compute&#xA;  $$ \text{MIC}(X;Y) = \text{max}_{n_X \cdot n_Y \leq m^{0.6}} \left( \frac{I_{n_X, n_Y}(X;Y)}{log(\text{min}(n_X, n_Y)} \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $I_{n_X, n_Y}(X;Y)$ means: compute the $I$ precisely as you did for discrete variables by treating $X$ as a discrete random variable after binning it into $n_X$ bins and analogously with $Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;===&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;ORIGINAL QUESTION&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;More precisely: I have a classification problem for one boolean variable, let's call this variable &lt;code&gt;outcome&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have lots of data and lots of features (~150 or so) but these features are not totally 'meaningless' as in image prediction (where every x and y coordinate is a feature) but they are of the form gender, age, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did until now: from these 150 features, I guessed the ones that 'seem' to have some importance for the outcome. Still, I am unsure which features to select and also how to measure their importance before starting the actual learning algorithm (that involves yet more selection like PCA and stuff). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for a feature &lt;code&gt;f&lt;/code&gt; taking only finitely many values &lt;code&gt;x_1, ..., x_n&lt;/code&gt; my very naive approach would be to compute some relation between&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;P(outcome==TRUE | f==x_1)&lt;/code&gt;, ..., &lt;code&gt;P(outcome==TRUE | f==x_n)&lt;/code&gt; and &lt;code&gt;P(outcome==TRUE)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i.e. the feature is important when I can deduce more information about the coutcome from it than without any knowledge about the feature).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;em&gt;Concrete question(s): Is that a good idea? Which relation to take? What to do with continuous variables?&lt;/em&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'm sure that I'm not the first one ever wondering about this. I've read about (parts of) algorithms that do this selection in a sort-of automated way. Can somebody point me into the right direction (references, names of algorithms to look for, ...)?&lt;/p&gt;&#xA;" OwnerUserId="9856" LastEditorUserId="9856" LastEditDate="2015-07-07T17:50:38.483" LastActivityDate="2015-07-07T17:50:38.483" Title="Is automatic feature detection feasible?" Tags="|feature-selection|featurization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5922" PostTypeId="2" ParentId="5921" CreationDate="2015-05-27T10:08:56.653" Score="3" Body="&lt;p&gt;What you are describing makes sense and relates to the naive Bayesian classifier. A useful tool in that respect is &lt;a href=&quot;http://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;mutual information (MI)&lt;/a&gt; (and all its derivatives). MI was first proposed for feature selection by &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74.7629&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Battiti&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;MI is based on marginal and joint density estimations so you will need some pdf estimators for your continuous variables, like histograms, or kernel-based density estimators.&lt;/p&gt;&#xA;" OwnerUserId="172" LastActivityDate="2015-05-27T10:08:56.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5923" PostTypeId="2" ParentId="5913" CreationDate="2015-05-27T10:17:19.210" Score="2" Body="&lt;p&gt;This entirely depends on what you wish or aim to do with your data, and what you mean by your tag &lt;code&gt;data-cleaning&lt;/code&gt;. This can involve the technical process of sanitising data i.e. fixing broken &lt;code&gt;XML&lt;/code&gt;, but also the manual deletion of outliers etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you going to perform statistics to measure the correlation between certain variables? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or:&#xA;Are you going to include the data into a machine-learning solution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, you can plot descriptive statistics to get an impression of the quality of your data, such as:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Box-plots to detect outliers&lt;/li&gt;&#xA;&lt;li&gt;Standard deviation, mean, mode, median to assess if it's normally distributed or not&lt;/li&gt;&#xA;&lt;li&gt;More, don't know from the top of my head&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Remember that those measures give you an &lt;em&gt;impression&lt;/em&gt; of the quality of your data, but even good measurements can come from bad data, and possibly vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, the best way of assessing the quality of your data is to assess the quality of your results, which are acquired generally &lt;strong&gt;after&lt;/strong&gt; several experiments. Only then will you see how well your data is suited for the problem you are trying to solve. That - or you need to get your hands dirty and examine the data qualitatively before you use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data (in assumingly large quantities) are generally used to identify certain patterns within the phenomena described by the data. It is very important to describe your task in terms of a &lt;strong&gt;(research) question&lt;/strong&gt; and a &lt;strong&gt;proposed solution&lt;/strong&gt; or &lt;strong&gt;hypothesis&lt;/strong&gt;. Then, you can assess whether the data is suitable for your particular (research) project. Keep in mind that, theoretically, you can try to answer numerous questions on the same data, or answer the same question on different data sets. The goal is the right balance.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-05-27T10:17:19.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5924" PostTypeId="1" AcceptedAnswerId="5925" CreationDate="2015-05-27T10:26:11.687" Score="0" ViewCount="62" Body="&lt;p&gt;I have a multi year dataset. Each time frame  of the data has different predictor importance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say for example, I am slicing the data into two partions as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a dataset for the year 2014 (whole year)&lt;/li&gt;&#xA;&lt;li&gt;a 2015 Jan. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When i look for the predictor importance, the predictor variables are different for both the partions. (1) Hence i am not able to arrive at one  unique decision tree which can explain the model better. (2). I am not able to train a model which can predict the new data correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything I am going wrong here.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-05-27T11:07:38.020" Title="change in variable importance" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5925" PostTypeId="2" ParentId="5924" CreationDate="2015-05-27T11:07:38.020" Score="2" Body="&lt;p&gt;Probably you can use multiple decision trees - each for slice of dataset. Then combine the results from each decision tree. You can also weigh the results from each decision tree. Eg the dataset for 2015 can be given more weight than dataset for 2014. I do not see much harm in spliting dataset and training different trees as it helps to account for the predictor values in a more accurate manner. If you could more details about the data or the &quot;variable&quot; you mentioned, people would be able to help more.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2015-05-27T11:07:38.020" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5926" PostTypeId="2" ParentId="5860" CreationDate="2015-05-27T13:27:44.253" Score="1" Body="&lt;p&gt;If you want to have some data science programming experience in R try &lt;a href=&quot;https://www.datacamp.com/&quot; rel=&quot;nofollow&quot;&gt;https://www.datacamp.com/&lt;/a&gt; Is' an interactive online coding environment for learning R programming. &lt;/p&gt;&#xA;" OwnerUserId="9588" LastActivityDate="2015-05-27T13:27:44.253" CommentCount="0" CommunityOwnedDate="2015-05-27T13:27:44.253" ContentLicense="CC BY-SA 3.0" />
  <row Id="5927" PostTypeId="1" AcceptedAnswerId="5928" CreationDate="2015-05-27T15:20:16.180" Score="1" ViewCount="345" Body="&lt;p&gt;I tried finding about exception lists in wordnet lemmatizers. &quot;Morphy() uses inflectional ending rules and exception lists to handle different possibilities&quot; which I read from &lt;a href=&quot;http://www.nltk.org/howto/wordnet.html&quot; rel=&quot;nofollow&quot;&gt;http://www.nltk.org/howto/wordnet.html&lt;/a&gt; . Can you explain what is an exception list. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="9322" LastActivityDate="2015-05-27T15:54:02.617" Title="Wordnet Lemmatization" Tags="|machine-learning|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5928" PostTypeId="2" ParentId="5927" CreationDate="2015-05-27T15:43:31.750" Score="2" Body="&lt;p&gt;The exception list files are used to help the processor find base forms from 'irregular inflections' according to the &lt;a href=&quot;http://wordnet.princeton.edu/wordnet/man/wndb.5WN.html&quot; rel=&quot;nofollow&quot;&gt;man page&lt;/a&gt;.  They mean that some words, when plural or a different tense, can't be algorithmically processed to find the base/root word.  More details can be found in the &lt;a href=&quot;https://wordnet.princeton.edu/man/morphy.7WN.html#sect3&quot; rel=&quot;nofollow&quot;&gt;morphy man&lt;/a&gt;.  I'm not a language processing expert, but this is likely a result of English words that 'break the rules'.  If you think about the code like a human trying to learn English: the student learns rules to use (algorithm) and then has to memorize exceptions for the rules (exception lists).  An over-simplified analogy that does not involve endings/conjugation would a spell checking program.  An algorithm might check for 'i before e, except after c' but would first have to check the word against an exception list to make sure it isn't 'weird' or 'caffeine' - please don't start a linguistics fight about this rule, I am not commenting on the validity of it/that's not the point I'd like to make.&lt;/p&gt;&#xA;" OwnerUserId="8953" LastEditorUserId="8953" LastEditDate="2015-05-27T15:54:02.617" LastActivityDate="2015-05-27T15:54:02.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5929" PostTypeId="2" ParentId="5913" CreationDate="2015-05-27T22:39:35.300" Score="0" Body="&lt;p&gt;Your question is very broad, and therefore I won't offer very specific answers.  You asked for &quot;time-saving tips&quot; but there are many and they depend on context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I'll offer a &lt;em&gt;general&lt;/em&gt; set of heuristics that are useful most of the time.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;Start&lt;/em&gt; with a specification or definition of what &quot;quality&quot; means for your analysis, and therefore for your data.&lt;/li&gt;&#xA;&lt;li&gt;Use definition/specification in 1) to enumerate the types of errors, omissions, missteps, modifications, etc. that &lt;em&gt;might&lt;/em&gt; arise in the course of data collection and recording.  &lt;em&gt;This enumeration will always be provisional&lt;/em&gt; because there could be ways that quality could be degraded that you didn't think of originally.&lt;/li&gt;&#xA;&lt;li&gt;Use 2) to define &lt;em&gt;inspection and test methods&lt;/em&gt; that might reveal the existence of any of these types of errors, omissions, etc.  The default method is &lt;em&gt;&quot;eyeballing&quot;&lt;/em&gt; -- having a knowledgeable, experienced person looking at the data to see if it looks right. &lt;/li&gt;&#xA;&lt;li&gt;When you uncover errors, omissions, etc. try to determine the &lt;em&gt;root causes and generating processes&lt;/em&gt;. E.g. erroneous data in dozens of columns resulted from an &quot;off-by-one&quot; bug in the ETL job that generated/translated the data from source files.&lt;/li&gt;&#xA;&lt;li&gt;Using 2), 3) and 4), define processes that might either &lt;em&gt;correct erroneous data&lt;/em&gt; or &lt;em&gt;mitigate&lt;/em&gt; the effects of data quality problems.  E.g. &lt;a href=&quot;http://en.wikipedia.org/wiki/Soundex&quot; rel=&quot;nofollow&quot;&gt;Soundex&lt;/a&gt; transformation of names can mitigate effects of misspelling, but can't mitigate people misusing the &quot;name&quot; field to enter notes: e.g. &quot;Nelson - DO NOT CALL&quot;.  Be aware that any process you might use to correct or mitigate errors or problems might &lt;em&gt;create&lt;/em&gt; new errors or problems (e.g. truncation of numerical data).&lt;/li&gt;&#xA;&lt;li&gt;After you perform your analysis, look back at the data and ask &quot;Could these results be the result of data quality problems rather than true/accurate/appropriate data?&quot;  In other words, &lt;em&gt;double check&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-05-27T22:39:35.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5930" PostTypeId="2" ParentId="5910" CreationDate="2015-05-27T23:25:19.413" Score="2" Body="&lt;p&gt;Please allow me to paraphrase your question to make sure I get your question right.&lt;/p&gt;&#xA;&lt;p&gt;Suppose you have 1 million data entries. Each entry consists of three inputs &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$X_2$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$X_3$&lt;/span&gt; and one output &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt;. One of the three inputs, &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt;, is a noise. You want to find the relation between &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt;. So you can remove the impact of &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; on &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Let's use a simplified example, &lt;span class=&quot;math-container&quot;&gt;$Y = 2X_1 + 3X_2 + 4X_3$&lt;/span&gt;&#xA;If you update each of the 1 million entries with:&#xA;&lt;span class=&quot;math-container&quot;&gt;$Y_{new} = Y - 2X_1$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now you can examine the updated 1 million entries to find the relation between input &lt;span class=&quot;math-container&quot;&gt;$X_2$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$X_3$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y_{new}$&lt;/span&gt;:&#xA;&lt;span class=&quot;math-container&quot;&gt;$Y_{new} = 3X_2 + 4X_3$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The relation between &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt; can be computed with different algorithms, such as regression, decision tree. How to measure which algorithm generates the best result, i.e. truly reflects the relation between &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt; hence removes the impact of &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; on &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt; as much as possible?&lt;/p&gt;&#xA;&lt;p&gt;The problem you're solving is an interesting one. I would use a free tool such as Knime to find out which algorithm generates the best result in modeling the relation between &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt;. It's faster than coding in Python or Matlab.&lt;/p&gt;&#xA;&lt;p&gt;Details: to put the problem intuitively, suppose the price of a house is determined by location, size of the house, condition of the house. Assume these three factors are independent from each other. You want to remove the impact of &amp;quot;condition of the house&amp;quot; on &amp;quot;price&amp;quot; as much as possible. In essence, you want to find the relation between &amp;quot;condition of the house&amp;quot; and &amp;quot;price&amp;quot; which may not be linear.&lt;/p&gt;&#xA;&lt;p&gt;You can let Knime read the 1 million entries of &amp;quot;condition of the house&amp;quot; and &amp;quot;price&amp;quot;, try Decision Tree, SVM, or even Ensemble Learning, and see which algorithm generates the best model of &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Y$&lt;/span&gt;. In other words, try and error :)&lt;/p&gt;&#xA;" OwnerUserId="9593" LastEditorUserId="98307" LastEditDate="2020-08-02T17:44:56.477" LastActivityDate="2020-08-02T17:44:56.477" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="5932" PostTypeId="1" AcceptedAnswerId="5934" CreationDate="2015-05-28T07:38:38.790" Score="-2" ViewCount="4022" Body="&lt;p&gt;I am trying to connect to hive from java but getting error. I searched in google but not got any helpfull solution. I have added all jars also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;package mypackage;&#xA;import java.sql.SQLException;&#xA;import java.sql.Connection;&#xA;import java.sql.ResultSet;&#xA;import java.sql.Statement;&#xA;import java.sql.DriverManager;&#xA;&#xA;&#xA;public class HiveJdbcClient {&#xA;&#xA;&#xA;private static String driver = &quot;org.apache.hadoop.hive.jdbc.HiveDriver&quot;;&#xA;&#xA;public static void main(String[] args) throws SQLException, &#xA;       ClassNotFoundException {&#xA; Class.forName(&quot;org.apache.hadoop.hive.jdbc.HiveDriver&quot;);&#xA; try {&#xA; Class.forName(driver);&#xA; } catch (ClassNotFoundException e) {&#xA; e.printStackTrace();&#xA;  System.exit(1);&#xA;}&#xA;&#xA;Connection connect = DriverManager.getConnection(&quot;jdbc:hive://master:10000 /default&quot;, &quot;&quot;, &quot;&quot;);&#xA;Statement state = connect.createStatement();&#xA;String tableName = &quot;mytable&quot;;&#xA;state.executeQuery(&quot;drop table &quot; + tableName);&#xA;ResultSet res=state.executeQuery(&quot;ADD JAR /home/hadoop_home/hive/lib /hive-serdes-1.0-SNAPSHOT.jar&quot;);&#xA;res = state.executeQuery(&quot;create table tweets (id BIGINT,created_at     STRING,source STRING,favorited BOOLEAN,retweet_count INT,retweeted_status STRUCT&amp;lt;text:STRING,user:STRUCT&amp;lt;screen_name:STRING,name:STRING&amp;gt;&amp;gt;,entities STRUCT&amp;lt;urls:ARRAY&amp;lt;STRUCT&amp;lt;expanded_url:STRING&amp;gt;&amp;gt;,user_mentions:ARRAY&amp;lt;STRUCT&amp;lt;screen_name:STRING,name:STRING&amp;gt;&amp;gt;,hashtags:ARRAY&amp;lt;STRUCT&amp;lt;text:STRING&amp;gt;&amp;gt;&amp;gt;,text STRING,user  STRUCT&amp;lt;screen_name:STRING,name:STRING,friends_count:INT,followers_count:INT,statuses_count:INT,verified:BOOLEAN,utc_offset:INT,time_zone:STRING&amp;gt;,in_reply_to_screen_name STRING) ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe' LOCATION '/user/flume/tweets'&quot;);&#xA;&#xA;&#xA;String show = &quot;show tables&quot;;&#xA;System.out.println(&quot;Running show&quot;);&#xA;res = state.executeQuery(show);&#xA;if (res.next()) {&#xA;  System.out.println(res.getString(1));&#xA;}&#xA;&#xA;&#xA;String describe = &quot;describe &quot; + tableName;&#xA;System.out.println(&quot;Running describe&quot;);&#xA;res = state.executeQuery(describe);&#xA;while (res.next()) {&#xA;  System.out.println(res.getString(1) + &quot;\t&quot; + res.getString(2));&#xA;}&#xA;&#xA;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting these errors:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;run:&#xA;SLF4J: Class path contains multiple SLF4J bindings.&#xA;SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]&#xA;SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]&#xA;SLF4J: Found binding in [jar:file:/home/GlassFish_Server/glassfish/modules/weld-osgi-bundle.jar!/org/slf4j/impl/StaticLoggerBinder.class]&#xA;SLF4J: See &lt;a href=&quot;http://www.slf4j.org/codes.html#multiple_bindings&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.slf4j.org/codes.html#multiple_bindings&lt;/a&gt; for an explanation.&#xA;Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable&#xA;    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:198)&#xA;    at org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:132)&#xA;    at org.apache.hadoop.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:133)&#xA;    at org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:122)&#xA;    at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:106)&#xA;    at java.sql.DriverManager.getConnection(DriverManager.java:571)&#xA;    at java.sql.DriverManager.getConnection(DriverManager.java:215)&#xA;    at dp.HiveJdbcClient.main(HiveJdbcClient.java:35)&#xA;Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable&#xA;    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)&#xA;	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)&#xA;    at java.security.AccessController.doPrivileged(Native Method)&#xA;    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)&#xA;    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)&#xA;    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)&#xA;    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)&#xA;    ... 8 more&#xA;Java Result: 1&lt;/p&gt;&#xA;" OwnerUserId="8729" LastEditorUserId="8729" LastEditDate="2016-11-29T12:39:57.930" LastActivityDate="2016-11-29T12:39:57.930" Title="getting error:-Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable" Tags="|apache-hadoop|java|hive|" AnswerCount="1" CommentCount="2" ClosedDate="2015-05-28T12:16:50.673" ContentLicense="CC BY-SA 3.0" />
  <row Id="5933" PostTypeId="1" CreationDate="2015-05-28T08:44:13.343" Score="2" ViewCount="165" Body="&lt;p&gt;I have a dataset which contains information about when do people enter and leave a premise. I have the following information in the dataset : &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Person Id&lt;/li&gt;&#xA;&lt;li&gt;Time of Entry&lt;/li&gt;&#xA;&lt;li&gt;Time of Leaving&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The dataset has around 50 unique persons. Each person will have multiple entries corresponding to multiple visits. The data spans over a year so I have quite a lot of entries (around 1 million).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These people can be classified on the basis of the department they work under (2 departments - mutually exclusive) or on basis of  their role (4 possible roles - all mutually exclusive)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering what kind of data analysis can be done with this kind of dataset. I am not looking for straight-forward things like &quot;who spent the most time in building&quot;. However things like finding correlation between visits of 2 people would be interesting. So if person A visits the premise, what is the probability that person B would also visit. Since I have only around 50 unique visitors, I think such an analysis is feasible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another line of thought was to apply some interval-pattern mining techniques but I am not much familiar with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone give me some pointers/ideas about what kind of data products can be build using this or what kind of techniques can be used with such data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : As discussed in comments, I call it a product in the sense that I do not want some simple or trivial analysis. And I am not looking for any commercially viable idea - just some cool fun idea :)&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="75" LastEditDate="2015-05-28T12:15:27.920" LastActivityDate="2015-05-31T11:22:46.940" Title="What possible data products can be built using this dataset" Tags="|machine-learning|data-mining|dataset|" AnswerCount="3" CommentCount="6" ClosedDate="2015-06-01T22:20:54.520" ContentLicense="CC BY-SA 3.0" />
  <row Id="5934" PostTypeId="2" ParentId="5932" CreationDate="2015-05-28T12:00:55.070" Score="1" Body="&lt;p&gt;I got the answer. One jar file was missing now it is solved. &#xA;This file was missing.&#xA;hadoop-common-2.1.0-beta.jar&lt;/p&gt;&#xA;" OwnerUserId="8729" LastActivityDate="2015-05-28T12:00:55.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5935" PostTypeId="2" ParentId="5881" CreationDate="2015-05-28T12:49:25.243" Score="1" Body="&lt;p&gt;Artificial Intelligence: A Modern Approach by Stuart Russel and Peter Norvig.&lt;/p&gt;&#xA;" OwnerUserId="9322" LastActivityDate="2015-05-28T12:49:25.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5936" PostTypeId="1" AcceptedAnswerId="5937" CreationDate="2015-05-28T15:23:27.847" Score="0" ViewCount="492" Body="&lt;p&gt;I am doing load forecasting using SVR(kernel='rbf').How can I understand which is the best value for parameters C, epsilon and gamma?Thanks.&lt;/p&gt;&#xA;" OwnerUserId="9501" LastActivityDate="2015-05-29T03:28:47.073" Title="Support vector regression and paremeters" Tags="|python|svm|forecast|" AnswerCount="1" CommentCount="0" ClosedDate="2015-05-29T09:00:55.730" ContentLicense="CC BY-SA 3.0" />
  <row Id="5937" PostTypeId="2" ParentId="5936" CreationDate="2015-05-28T15:51:27.623" Score="4" Body="&lt;p&gt;It looks like that you are using scikit-learn. In this case use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html&quot; rel=&quot;nofollow&quot;&gt;Grid Search Cross Validation&lt;/a&gt;  or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.RandomizedSearchCV.html#sklearn.grid_search.RandomizedSearchCV&quot; rel=&quot;nofollow&quot;&gt;Randomized Search Cross Validation&lt;/a&gt; to find the best parameters. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sklearn.grid_search.GridSearchCV(estimator, param_grid, scoring=None, cv=None, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In these approaches you basically loop over possible sets of your parameters, specified via &lt;code&gt;param_grid&lt;/code&gt;. For each set you perform a cross-validation (default is 3-fold, but you can specify it via &lt;code&gt;cv&lt;/code&gt; parameter). Cross-validation gives you the mean value and deviation of your 'scoring' parameter (e.g. 'accuracy' for classification, 'r2' for regression). The set of parameters with the best 'scoring' is the winner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See example &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html#example-model-selection-randomized-search-py&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. It also show how to output not only &lt;em&gt;the&lt;/em&gt; best set of parameters, but also &quot;top-N&quot;. I find it very useful in building intuition about my model. &lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2015-05-29T03:28:47.073" LastActivityDate="2015-05-29T03:28:47.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5938" PostTypeId="2" ParentId="5933" CreationDate="2015-05-28T15:59:26.593" Score="2" Body="&lt;p&gt;I don't know if this is the answer you are looking for, but generally start by asking &quot;what would I do differently&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if I am running a business (to apply some context), at how often do I not have coverage for certain positions. How can I schedule shifts (for example) to ensure that all critical positions are covered but I have as few people in the building as possible (minimising cost)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple time series, you could apply any sort of forecasting to this data set - predicting who and how long they will be in the building, when they will arrive next, for example.  Look for seasonality in the arrival patterns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also look to predict when a given person will next be in the building. And can you detect outliers? Who has been leaving early or breaking from their usual pattern of behaviour?&lt;/p&gt;&#xA;" OwnerUserId="9876" LastActivityDate="2015-05-28T15:59:26.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5939" PostTypeId="2" ParentId="5921" CreationDate="2015-05-28T16:49:51.293" Score="5" Body="&lt;p&gt;It is a good idea to do automatic feature vector selection for classification, and is widely done for decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to find the relative importance of variables in classification schemes, a popular way of selecting variables is by performing entropy minimization.  This a technique that is often used for constructing decision trees.  Entropy is simply a way of numerically quantifying disorder of a system:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=H=%5CSigma_x-p_x*log_2%28p_x%29&quot; alt=&quot;Entropy&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For binary classification, it's simple to calculate entropy for every variable, moving the variables which minimize entropy to the top, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, consider a simple case where a game exists when one must choose between a right and left door, one of which is red, one of which is blue.    The color of the door changes from instance to instance (sometime left is red, sometimes right is red), but the prize is always behind the left door.  Suppose also that over many trials, both left-right, and red-blue have equal number of people choosing those options.  The entropy of the original system (P(prize)=.5, P(!prize)=.5) is 1.  Supposing that we select a door based on color, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=H=p_%7Bred%7D*(H_%7Bred%7D)%2Bp_%7Bblue%7D*(H_%7Bblue%7D)=0.5*(-.5*log_2(.5)-.5*log_2(.5))%2B.5*(-.5*log_2(.5)-.5*log_2(.5))&quot; alt=&quot;RBEntropy&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the probability of red and blue are both .5, and the entropy of each system is 1, so the Entropy of the new system is still 1.  This means that no information was gained from choosing red or blue.  Suppose instead we select on left right:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=H=p_%7Bleft%7D*%28H_%7Bleft%7D%29%2Bp_%7Bright%7D*%28H_%7Bright%7D%29=0.5*%28-1*log_2%281%29%29%2B.5*%28-1*log_2%281%29%29&quot; alt=&quot;LREntropy&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And since log_2(1)=0, there is no disorder (i.e. a perfect predictor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For continuous variables, it becomes a bit more complicated because you then have to select based on a floating threshold (or a few for a single variable).  While the basic entropy calculations hold fast, tuning many thresholds on many variables becomes resource intensive, so at that point you'll want to look at something like spark's mllib. &lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="8041" LastEditDate="2015-05-28T16:55:01.230" LastActivityDate="2015-05-28T16:55:01.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5940" PostTypeId="2" ParentId="5933" CreationDate="2015-05-28T18:30:23.893" Score="3" Body="&lt;p&gt;You might be able to find individuals that act as anchors.  That if Abe is here, then everybody else is more likely to stay longer. Interesting to know if it was always the same people who stayed or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There might also be some interesting patterns with groups that arrive, are present, or leave together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a business owner you could look for individuals who not just work shorter hours, but arrive at their own time, and leave with a group.&lt;/p&gt;&#xA;" OwnerUserId="8895" LastActivityDate="2015-05-28T18:30:23.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5941" PostTypeId="1" CreationDate="2015-05-28T22:07:35.367" Score="5" ViewCount="2229" Body="&lt;p&gt;I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. &#xA;Here is the sample of my dataset after being organized into list of keywords for each product id.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/H6Fhh.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;I consider each id as a &quot;document&quot; and each keywords as the &quot;word&quot; in a &quot;document&quot; for the case of LDA.&#xA;It didn't work out as I expected as each topic have many identical keywords with different weight. I removed 100 most common keywords but there are still some identical keywords in the topics. Here is the sample output:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/kOR8W.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I deal with the identical keywords in my topics? and also how to deal with the 100 most common keywords I removed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the graph showing the frequency of the words in all &quot;documents&quot;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/gbMUC.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;Each word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the words&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much. Any suggestion is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9724" LastEditorUserId="9724" LastEditDate="2015-06-03T18:46:52.203" LastActivityDate="2016-09-28T12:26:50.450" Title="Need help with LDA for selecting features" Tags="|machine-learning|bigdata|topic-model|lda|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5942" PostTypeId="2" ParentId="5875" CreationDate="2015-05-29T00:41:17.407" Score="2" Body="&lt;p&gt;You can perform logistic regression (if your dependent variable has two classes) that penalizes based on the L1 norm. You can choose the correct sparsity parameter (typically $\lambda$) that chooses how strongly to seek sparsity based on cross-validation. The model will force many non-informative features to be 0. This is a form of feature selection. See here: &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="9880" LastActivityDate="2015-05-29T00:41:17.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5943" PostTypeId="2" ParentId="5867" CreationDate="2015-05-29T01:09:27.367" Score="1" Body="&lt;p&gt;Here is the relevant source code. I'm pretty sure it's using cross-entropy for multiclass: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    //  GBM by Greg Ridgeway  Copyright (C) 2003&#xA;&#xA;#include &quot;multinomial.h&quot;&#xA;&#xA;CMultinomial::CMultinomial(int cNumClasses, int cRows)&#xA;{&#xA;   mcNumClasses = cNumClasses;&#xA;   mcRows = cRows;&#xA;&#xA;   madProb = new double[cNumClasses * cRows];&#xA;}&#xA;&#xA;CMultinomial::~CMultinomial()&#xA;{&#xA;   if(madProb != NULL)&#xA;   {&#xA;      delete [] madProb;&#xA;   }&#xA;}&#xA;&#xA;&#xA;GBMRESULT CMultinomial::UpdateParams&#xA;(&#xA;   double *adF,&#xA;   double *adOffset,&#xA;   double *adWeight,&#xA;   unsigned long cLength&#xA;)&#xA;{&#xA;   // Local variables&#xA;   unsigned long ii=0;&#xA;   unsigned long kk=0;&#xA;&#xA;   // Set the probabilities for each observation in each class&#xA;   for (ii = 0; ii &amp;lt; mcRows; ii++)&#xA;   {&#xA;      double dClassSum = 0.0;&#xA;      for (kk = 0; kk &amp;lt; mcNumClasses; kk++)&#xA;      {&#xA;         int iIdx = ii + kk * mcRows;&#xA;         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];&#xA;         madProb[iIdx] = adWeight[iIdx] * exp(dF);&#xA;         dClassSum += adWeight[iIdx] * exp(dF);&#xA;      }&#xA;&#xA;      dClassSum = (dClassSum &amp;gt; 0) ? dClassSum : 1e-8;&#xA;&#xA;      for (kk = 0; kk &amp;lt; mcNumClasses; kk++)&#xA;      {&#xA;         madProb[ii + kk * mcRows] /= dClassSum;&#xA;      }&#xA;   }&#xA;&#xA;   return GBM_OK; &#xA;}&#xA;&#xA;&#xA;GBMRESULT CMultinomial::ComputeWorkingResponse&#xA;(&#xA;    double *adY,&#xA;    double *adMisc,&#xA;    double *adOffset,&#xA;    double *adF, &#xA;    double *adZ, &#xA;    double *adWeight,&#xA;    bool *afInBag,&#xA;    unsigned long nTrain,&#xA;    int cIdxOff&#xA;)&#xA;{&#xA;    unsigned long i = 0;&#xA;&#xA;    for(i=cIdxOff; i&amp;lt;nTrain+cIdxOff; i++)&#xA;    {&#xA;       adZ[i] = adY[i] - madProb[i];&#xA;    }&#xA;&#xA;    return GBM_OK;&#xA;}&#xA;&#xA;&#xA;GBMRESULT CMultinomial::InitF&#xA;(&#xA;    double *adY,&#xA;    double *adMisc,&#xA;    double *adOffset, &#xA;    double *adWeight,&#xA;    double &amp;amp;dInitF, &#xA;    unsigned long cLength&#xA;)&#xA;{&#xA;    dInitF = 0.0;&#xA;    return GBM_OK;&#xA;}&#xA;&#xA;double CMultinomial::Deviance&#xA;(&#xA;    double *adY,&#xA;    double *adMisc,&#xA;    double *adOffset, &#xA;    double *adWeight,&#xA;    double *adF,&#xA;    unsigned long cLength,&#xA;    int cIdxOff&#xA;)&#xA;{&#xA;    unsigned long ii=0;&#xA;    double dL = 0.0;&#xA;    double dW = 0.0;&#xA;&#xA;    for(ii=cIdxOff; ii&amp;lt;cLength+cIdxOff; ii++)&#xA;    {&#xA;        dL += -adWeight[ii] * adY[ii] * log(madProb[ii]);&#xA;        dW += adWeight[ii];&#xA;    }&#xA;&#xA;    return dL/dW;&#xA;}&#xA;&#xA;&#xA;GBMRESULT CMultinomial::FitBestConstant&#xA;(&#xA;    double *adY,&#xA;    double *adMisc,&#xA;    double *adOffset,&#xA;    double *adW,&#xA;    double *adF,&#xA;    double *adZ,&#xA;    unsigned long *aiNodeAssign,&#xA;    unsigned long nTrain,&#xA;    VEC_P_NODETERMINAL vecpTermNodes,&#xA;    unsigned long cTermNodes,&#xA;    unsigned long cMinObsInNode,&#xA;    bool *afInBag,&#xA;    double *adFadj,&#xA;   int cIdxOff&#xA;)&#xA;{&#xA;      // Local variables&#xA;    GBMRESULT hr = GBM_OK;&#xA;    unsigned long iNode = 0;&#xA;    unsigned long iObs = 0;&#xA;&#xA;   // Call LocM for the array of values on each node&#xA;    for(iNode=0; iNode&amp;lt;cTermNodes; iNode++)&#xA;    {&#xA;        if(vecpTermNodes[iNode]-&amp;gt;cN &amp;gt;= cMinObsInNode)&#xA;        {&#xA;         // Get the number of nodes here&#xA;         double dNum = 0.0;&#xA;         double dDenom = 0.0;&#xA;         for (iObs = 0; iObs &amp;lt; nTrain; iObs++)&#xA;         {&#xA;            if(afInBag[iObs] &amp;amp;&amp;amp; (aiNodeAssign[iObs] == iNode))&#xA;                {&#xA;               int iIdx = iObs + cIdxOff;&#xA;                    dNum += adW[iIdx] * adZ[iIdx];&#xA;               dDenom += adW[iIdx] * fabs(adZ[iIdx]) * (1 - fabs(adZ[iIdx]));&#xA;                }&#xA;         }&#xA;&#xA;         dDenom = (dDenom &amp;gt; 0) ? dDenom : 1e-8;&#xA;&#xA;         vecpTermNodes[iNode]-&amp;gt;dPrediction = dNum / dDenom;&#xA;        }&#xA;    }&#xA;&#xA;    return hr;&#xA;}&#xA;&#xA;double CMultinomial::BagImprovement&#xA;(&#xA;    double *adY,&#xA;    double *adMisc,&#xA;    double *adOffset,&#xA;    double *adWeight,&#xA;    double *adF,&#xA;    double *adFadj,&#xA;    bool *afInBag,&#xA;    double dStepSize,&#xA;    unsigned long nTrain&#xA;)&#xA;{&#xA;    double dReturnValue = 0.0;&#xA;    double dW = 0.0;&#xA;&#xA;   unsigned long ii;&#xA;   unsigned long kk;&#xA;&#xA;   // Calculate the probabilities after the step&#xA;   double *adStepProb = new double[mcNumClasses * mcRows];&#xA;&#xA;   // Assume that this is last class - calculate new prob as in updateParams but&#xA;   // using (F_ik + ss*Fadj_ik) instead of F_ik. Then calculate OOB improve&#xA;   for (ii = 0; ii &amp;lt; mcRows; ii++)&#xA;   {&#xA;      double dClassSum = 0.0;&#xA;      for (kk = 0; kk &amp;lt; mcNumClasses; kk++)&#xA;      {&#xA;         int iIdx = ii + kk * mcRows;&#xA;         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];&#xA;         dF += dStepSize * adFadj[iIdx];&#xA;         adStepProb[iIdx] = adWeight[iIdx] * exp(dF);&#xA;         dClassSum += adWeight[iIdx] * exp(dF);&#xA;      }&#xA;&#xA;      dClassSum = (dClassSum &amp;gt; 0) ? dClassSum : 1e-8;&#xA;&#xA;      for (kk = 0; kk &amp;lt; mcNumClasses; kk++)&#xA;      {&#xA;         adStepProb[ii + kk * mcRows] /= dClassSum;&#xA;      }&#xA;   }&#xA;&#xA;   // Calculate the improvement&#xA;    for(ii=0; ii&amp;lt;nTrain; ii++)&#xA;    {&#xA;        if(!afInBag[ii])&#xA;      {&#xA;         for (kk = 0; kk &amp;lt; mcNumClasses; kk++)&#xA;         {&#xA;            int iIdx = ii + kk * mcRows;&#xA;                dReturnValue += adWeight[iIdx] * adY[iIdx] * &#xA;                               (log(adStepProb[iIdx]) - log(madProb[iIdx]));&#xA;&#xA;            dW += adWeight[iIdx] * adY[iIdx];&#xA;         }&#xA;      }&#xA;    }&#xA;&#xA;    return dReturnValue/dW;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9880" LastActivityDate="2015-05-29T01:09:27.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5944" PostTypeId="1" CreationDate="2015-05-29T02:55:18.203" Score="1" ViewCount="47" Body="&lt;p&gt;Problem:&#xA;Lets say we have an irreducible Markov chain.&#xA;Given a failure state or non desirable state F and a current state S&#xA;Is it possible to find how far we are from the failure state F.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: if we try to model an engine failure using Markov process and using historic data we have a state transition matrix P, one of which is a state where the engine was failed .&#xA;Now , in a dynamic system, if the current state S and a state transition matrix are given, can we calculate how far we are from failure state F.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;x(n+1) = (x)P^n&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right hand side of the equation gives us a vector v which consists of probabilities of transitioning to different states at stage n.&#xA;With different value of n, the values in this vector v will change&#xA;one of the values in v will be the probability of going to state F.&#xA;I want to find for what n , transitioning probability to F is maximum.&lt;/p&gt;&#xA;" OwnerUserId="9787" LastActivityDate="2015-05-29T02:55:18.203" Title="How far from a specific state in a Markov chain" Tags="|statistics|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5945" PostTypeId="1" CreationDate="2015-05-29T05:13:40.113" Score="1" ViewCount="36" Body="&lt;p&gt;I have no idea whether this is the right StackExchange flavor to post this question in, but here goes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an agent (an IRC bot that listens to an event stream) that I would like to add &quot;intelligent notifications&quot; to. Let me give a specific example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bot monitors the results from our continuous integration system, so it basically sees a stream of test results associated with changes (and the changes are associated with users.) If I ask it to notify me of interesting events, then for each event it sees, it decides whether to tell me about it (or rather, about the current results so far.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like it to decide based on how interesting something is for me. Something is interesting if (1) it is a result of a change that I pushed to the continuous integration system, (2) it is a test failure, and (3) the information conveyed by that failure is a significant indicator of whether my change was bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tests have a baseline probability of failing for no good reason (&quot;intermittent failures&quot;). If one test fails, it does not necessarily mean my change was bad. It might just be a flaky test. (Any given push tends to result in a couple bogus failures, so this isn't some obscure edge case.) But the baseline probability of an intermittent failure varies according to the test suite running (we have several dozen different test suites that fire off for every change pushed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I don't want to be bothered with reports of failing tests that are probably just noise. And if my push &lt;em&gt;is&lt;/em&gt; bad, I don't want to be flooded with notifications for every failed test. (If I break something, it'll probably show up in multiple test suites. So if the agent sees two failures and 20 successes, maybe it won't bother me, but if it then sees another failure or two it should.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am imagining I could look at the change between the prior and posterior probabilities and use that to estimate the entropy of that new test result, and perhaps also compute my personal entropy (if that makes any sense -- as in, the bot should assume I don't know what any of the results are until it notifies me, at which point it should assume I am aware of the current full set of results and not bother me again until enough additional results have come in to substantially change the probability estimate of my change being bad.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or something like that. I'm really looking for the right mathematical framework to compute things like this. I know next to nothing about machine learning or... well, mathematics in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The above is simplified; I would additionally like to do crazy things like take into account &quot;labeling&quot;, where a third party looks at the test failures and decides whether they are real failures or not. But the time between the failure coming in and when that person labels it for me can also be modeled by a distribution, and I'd like to hold off notifications for interesting results if there's a good chance that this other person may tell me that the result is not interesting after all. But only if that's going to happen &quot;soon&quot;. Also, we have an automated system for guessing whether something &lt;em&gt;might&lt;/em&gt; be an intermittent vs real failure, and it'd be nice to take its opinion into account as well.)&lt;/p&gt;&#xA;" OwnerUserId="9881" LastActivityDate="2015-06-01T11:55:49.633" Title="Compute utility of a notification" Tags="|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5946" PostTypeId="1" AcceptedAnswerId="5966" CreationDate="2015-05-29T07:39:35.713" Score="4" ViewCount="217" Body="&lt;p&gt;Suppose I have a set of data (with  2-dimensional feature space), and I want to obtain clusters from them. But I do not know how many clusters will be formed.&lt;br&gt;&lt;br&gt; Yet, I want separate clusters (The number of clusters is more than 2). &lt;br&gt;&lt;br&gt;I figured that k means of k medoid cannot be used in this case. Nor can I use hierarchical clustering. Also since there is no training set hence cannot use KNN classifier to any others (supervised learning cannot be used as no training set). I cannot use OPTICS algorithm as I do not want to specify the radius (I don't know the radius)&lt;br&gt;&lt;br&gt; Is there any machine learning technique that would give me multiple clusters (distance based clustering) that deals well with outlier points too?&lt;/p&gt;&#xA;&lt;p&gt;This should be the output:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/dAimG.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="85045" LastEditDate="2021-02-10T18:11:45.347" LastActivityDate="2021-02-10T18:11:45.347" Title="Algorithm for deriving multiple clusters" Tags="|machine-learning|clustering|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="5947" PostTypeId="2" ParentId="5941" CreationDate="2015-05-29T08:32:16.483" Score="3" Body="&lt;p&gt;I think, even before doing LDA, you should remove words which appear in more than &quot;x&quot; percent of your documents. Try different &quot;x&quot; starting from 80% and then going down. The logic is that if the word is common for many documents, it  does not distinguished those and should be neglected.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-05-29T08:32:16.483" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5949" PostTypeId="2" ParentId="5921" CreationDate="2015-05-29T09:12:01.650" Score="3" Body="&lt;p&gt;Feature selection is a very well established field in Machine Learning. The objective of feature selection algorithms is to select a subset of your feature set in order to maximize your system's prediction performance. There are two kind of feature selection approaches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Filter methods&lt;/strong&gt;: filter methods select features without taking into consideration any specific prediction algorithm. They are based in applying different kinds of measures (like mutual information or correlation) in order to evaluate the relative information that a feature or a set of features can provide about the output variable. Filter methods are faster than wrapping methods, but do not perform as well as wrapping methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Wrapper methods&lt;/strong&gt;: these methods evaluate features or sets of features by using a specific classification/regression algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd suggest to read one of the seminal papers in the feature selection field as a starting point in order to get to know the basics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Guyon, Isabelle; Elisseeff, André (2003). &quot;An Introduction to Variable and Feature Selection&quot;. JMLR&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-05-29T09:12:01.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5950" PostTypeId="2" ParentId="5946" CreationDate="2015-05-29T09:22:53.750" Score="4" Body="&lt;p&gt;The fact is that you could use any of the algorithms you mentioned, and in general any algorithm that requires to set the number of clusters as a parameter (or any other parameter that indirectly sets the final number of clusters, like the threshold in a hierarchical clustering algorithm.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution to your problem is &lt;strong&gt;model selection&lt;/strong&gt;. Model selection methods evaluate different clustering solutions, and select the one that optimizes a given criterion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, in the case of K-means, you could find a solution for a range of k values, and keep the one that maximizes any cluster validation measure (see the Wikipedia entry for cluster analysis to read about some examples of cluster validation measures). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are automatic and more complex approaches (one specific example is &quot;Automatic Cluster Number Selection Using a Split and Merge K-Means Approach&quot; by Muhr, M. and Granitzer, M., but this is just an example). These methods use cluster validation measures to automatically split or merge clusters, but the idea is basically the same. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-05-29T09:22:53.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5957" PostTypeId="2" ParentId="5946" CreationDate="2015-05-29T20:40:51.467" Score="2" Body="&lt;p&gt;If the data are suitable, you can use Gaussian mixture modeling, fit via an EM algorithm to estimate various separate Gaussian clusters. When determining the number of clusters, you can use something like BIC (or other penalized likelihood criterion) to penalize based on the number of parameters that you are estimating. Then simply search over different numbers of clusters and choose the number with the lowest BIC. This is a form of model-based clustering.&lt;/p&gt;&#xA;&lt;p&gt;You should be able to use the &lt;code&gt;mclust&lt;/code&gt; package in R to do this: &lt;a href=&quot;http://cran.r-project.org/web/packages/mclust/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;mclust: Gaussian Mixture Modelling for Model-Based Clustering, Classification, and Density Estimation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9880" LastEditorUserId="85045" LastEditDate="2021-02-10T17:49:53.853" LastActivityDate="2021-02-10T17:49:53.853" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5958" PostTypeId="1" AcceptedAnswerId="5959" CreationDate="2015-05-29T23:04:54.580" Score="9" ViewCount="9747" Body="&lt;p&gt;Each observation in my data was collected with a difference of 0.1 seconds. I don't call it a time series because it don't have a date and time stamp. In the examples of clustering algorithms (I found online) and PCA the sample data have 1 observation per case and are not timed. But my data have hundreds of observations collected every 0.1 seconds per vehicle and there are many vehicles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: I have asked this question on quora as well.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="4933" LastActivityDate="2020-09-03T13:08:18.213" Title="Are there any unsupervised learning algorithms for time sequenced data?" Tags="|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5959" PostTypeId="2" ParentId="5958" CreationDate="2015-05-30T13:50:32.703" Score="7" Body="&lt;p&gt;What you have is a sequence of events according to time so do not hesitate to call it Time Series!&lt;/p&gt;&#xA;&lt;p&gt;Clustering in time series has 2 different meanings:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Segmentation of time series&lt;/strong&gt; i.e. you want to segment an individual time series into different time intervals according to internal similarities.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Time series clustering&lt;/strong&gt; i.e. you have several time series and you want to find different clusters according to similarities between them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I assume you mean the second one and here is my suggestion:&lt;/p&gt;&#xA;&lt;p&gt;You have many vehicles and many observations per vehicle i.e you have many vehicles. So you have several matrices (each vehicle is a matrix) and each matrix contains &lt;strong&gt;N&lt;/strong&gt; rows (Nr of observations) and &lt;strong&gt;T&lt;/strong&gt; columns (time points). One suggestion could be applying PCA to each matrix to reduce the dimenssionality and observing data in PC space and see if there is meaningful relations between &lt;em&gt;different observations within a matrix (vehicle)&lt;/em&gt;. Then you can put each observation for all vehicles on each other and make a matrix and apply PCA to that to see relations of a single observation between different vehicles.&lt;/p&gt;&#xA;&lt;p&gt;If you do not have negative values &lt;strong&gt;Matrix Factorization&lt;/strong&gt; is strongly recommended for dimension reduction of matrix form data.&lt;/p&gt;&#xA;&lt;p&gt;Another suggestion could be putin all matrices on top of each other and build a &lt;strong&gt;N&lt;/strong&gt;x&lt;strong&gt;M&lt;/strong&gt;x&lt;strong&gt;T&lt;/strong&gt; tensor where &lt;em&gt;N&lt;/em&gt; is the number of vehicles, &lt;em&gt;M&lt;/em&gt; is the number of observations and &lt;em&gt;T&lt;/em&gt; is the time sequence and apply &lt;strong&gt;Tensor Decomposition&lt;/strong&gt; to see relations globally.&lt;/p&gt;&#xA;&lt;p&gt;A very nice approach to Time Series Clustering is shown in &lt;a href=&quot;https://www.researchgate.net/profile/Christian_Bauckhage/publication/262898400_Kernel_Archetypal_Analysis_for_Clustering_Web_Search_Frequency_Time_Series/links/5416e0770cf2bb7347db7a99.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper where the implementation is quiet straight forward.&lt;/p&gt;&#xA;&lt;p&gt;I hope it helped!&lt;/p&gt;&#xA;&lt;p&gt;Good Luck :)&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;EDIT&lt;/h2&gt;&#xA;&lt;p&gt;As you mentioned you mean Time Series Segmentation I add this to the answer.&lt;/p&gt;&#xA;&lt;p&gt;Time series segmentation is the only clustering problem that has a ground truth for evaluation. Indeed you consider the generating distribution behind the time series and analyze it I strongly recommend &lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/papers/icml2014c1_khaleghi14&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;https://tel.archives-ouvertes.fr/tel-00920184/document&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://128.84.21.199/pdf/1203.1515.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://arxiv.org/pdf/1203.1515.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://papers.nips.cc/paper/4623-locating-changes-in-highly-dependent-data-with-unknown-number-of-change-points&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://proceedings.mlr.press/v22/khaleghi12/khaleghi12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; where your problem is comprehensively studied. Specially the last one and the PhD thesis.&lt;/p&gt;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="86627" LastEditDate="2020-09-03T13:08:18.213" LastActivityDate="2020-09-03T13:08:18.213" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="5960" PostTypeId="1" CreationDate="2015-05-30T08:33:06.713" Score="9" ViewCount="451" Body="&lt;p&gt;How widely is Theano used in deep learning research? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is Theano a good start to learn the implementation of machine learning algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will learning the implementation of something like a feed forward network really help? Do graduate students implement neural networks or other algorithms at least once during their college days?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a reasonable idea about feed forward and recurrent networks, backpropagation, the general pipeline for a machine learning problem and the necessary mathematics. &lt;/p&gt;&#xA;" OwnerDisplayName="Karthik Thiagarajan" LastActivityDate="2020-08-03T08:06:32.840" Title="Theano in deep learning research" Tags="|machine-learning|python|deep-learning|library|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5961" PostTypeId="2" ParentId="5960" CreationDate="2015-05-30T12:48:52.713" Score="9" Body="&lt;p&gt;Theano used to be very popular in the last few years. However, from seeing what the top research labs in the world currently use instead of theano (Facebook AI and Google Deepmind), I would either go for caffe or Torch7, with a stronger preference on the latter.&lt;/p&gt;&#xA;&lt;p&gt;The reason is that you can implement from Recurrent Neural Nets to Convolutional Neural Networks very easily and there is a wide range of &lt;a href=&quot;https://github.com/torch/demos&quot; rel=&quot;nofollow noreferrer&quot;&gt;examples&lt;/a&gt; on github.&lt;/p&gt;&#xA;&lt;p&gt;Graduate students in NYU, Stanford, Oxford definitely implement advanced models in their studies; your background will help you a lot, a great place to start and familiarise with such models, and implement them in Torch7, is the &lt;a href=&quot;https://www.youtube.com/user/ProfNandoDF&quot; rel=&quot;nofollow noreferrer&quot;&gt;lessons&lt;/a&gt; of Prof. Nando de Freitas.&lt;/p&gt;&#xA;" OwnerUserId="552" OwnerDisplayName="iassael" LastEditorUserId="98307" LastEditDate="2020-08-03T08:06:32.840" LastActivityDate="2020-08-03T08:06:32.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="5962" PostTypeId="2" ParentId="5946" CreationDate="2015-05-31T09:47:23.370" Score="3" Body="&lt;p&gt;The radius in OPTICS is a &lt;strong&gt;maximum&lt;/strong&gt; value, and it can be set to infinity!&#xA;So you don't need to know it, and you should give OPTICS as well as DBSCAN a try. There are heuristics to choose their parameters, if you know your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, try hierarchical clustering. There are good heuristics on how to extract flat partitions out of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want something that handles noise well - this calls for DBSCAN, OPTICS and HAC.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-05-31T09:47:23.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5963" PostTypeId="2" ParentId="5933" CreationDate="2015-05-31T11:22:46.940" Score="1" Body="&lt;p&gt;I am assuming the context is office setting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Study of an Impact of a systemic event. e.g. how an organizational event impacted the attendance, or reverse of it (what happened which caused a shift in pattern). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Build employee profile based on localized events e.g. school vacations, time-off patterns, response to extra workload. May not necessarily be &quot;fun&quot; (your question), but profiling can also be done on demographics (age, gender, ethnicity, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Department or team analysis. e.g. generate networking index of a person, team or department (how correlated are their activities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possible impact on internal org design (e.g. scale/reduce support staff  services, resource waste reduction, etc. ), use occupancy rate of property to decide on hotel office model, or relocation. &lt;/p&gt;&#xA;" OwnerUserId="9901" LastActivityDate="2015-05-31T11:22:46.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5964" PostTypeId="1" CreationDate="2015-05-31T20:35:54.150" Score="3" ViewCount="229" Body="&lt;p&gt;I need to do some experimenting with Brown clustering, graph partitioning, agglomerative clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Are there Python/Matlab libraries for that? I know sklearn.cluster but it doesn't have algorithms I need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is it possible to install graphical interface for Cluto on Mac OS? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Overall, are there useful tutorials on using Cluto?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Other software for clustering that I could learn within a couple of hours?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="9907" LastActivityDate="2015-05-31T21:52:47.957" Title="Brown clustering, graph partitioning, agglomerative clustering - libraries/software" Tags="|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5965" PostTypeId="2" ParentId="5964" CreationDate="2015-05-31T21:52:47.957" Score="3" Body="&lt;p&gt;For graphs I definitely suggest &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;Networkx&lt;/a&gt; and &lt;a href=&quot;http://igraph.org/&quot; rel=&quot;nofollow&quot;&gt;igraph&lt;/a&gt; libraries. They support many graph partitioning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-05-31T21:52:47.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5966" PostTypeId="2" ParentId="5946" CreationDate="2015-06-01T06:03:05.510" Score="3" Body="&lt;p&gt;I don't think that EM clustering algorithms like k-means and Gaussian mixture models are quite what you're looking for. There are definitely other algorithms that don't require one to pick a number of clusters. My personal favorite (most of the time) is called mean-shift-clustering. You could find a great little blog post about it &lt;a href=&quot;http://spin.atomicobject.com/2015/05/26/mean-shift-clustering/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and it has a good implementation in python's scikit-learn library.&lt;/p&gt;&#xA;" OwnerUserId="9483" LastActivityDate="2015-06-01T06:03:05.510" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5967" PostTypeId="2" ParentId="661" CreationDate="2015-06-01T06:37:06.680" Score="0" Body="&lt;p&gt;You should create /user folder at first.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sudo -u hdfs hadoop fs -mkdir -p /user&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9914" LastActivityDate="2015-06-01T06:37:06.680" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5968" PostTypeId="1" AcceptedAnswerId="5974" CreationDate="2015-06-01T09:58:29.580" Score="1" ViewCount="3134" Body="&lt;p&gt;I am trying to paste a &lt;code&gt;string&lt;/code&gt; and &lt;code&gt;int&lt;/code&gt; from &lt;code&gt;map&lt;/code&gt; in &lt;code&gt;Hive&lt;/code&gt; to an &lt;code&gt;array&lt;/code&gt;.&#xA;For now, record looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;string1&quot;:1,&quot;string2&quot;:1,&quot;string3&quot;:15}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to convert it to an array like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&quot;string1:1&quot;,&quot;string2:1&quot;,&quot;string3:15&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2015-06-01T14:37:08.643" Title="How to paste string and int from map to an array in hive?" Tags="|hive|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5969" PostTypeId="1" AcceptedAnswerId="5979" CreationDate="2015-06-01T11:25:59.933" Score="3" ViewCount="924" Body="&lt;p&gt;We are developing a classification system, where the categories are fixed, but many of them are inter-related. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, we have a category called, &quot;&lt;code&gt;roads&lt;/code&gt;&quot; and another one called &quot;&lt;code&gt;traffic&lt;/code&gt;&quot;. We believe that the model will be confused by the text samples, which could be in &lt;code&gt;roads&lt;/code&gt; category and also in &lt;code&gt;traffic&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of our text samples are suitable for multi class labelling too.  For example, &quot;There is a garbage dump near the footpath. The footpath is broken completely&quot;. This text could be categorized into &lt;code&gt;garbage&lt;/code&gt; bucket or &lt;code&gt;footpath&lt;/code&gt; bucket. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are going to build a training set for this classifier, by manually annotating the text. So, can we put multiple labels for one issue? How should we deal with  text with multiple labels for it? Should they be added into all categories to which it is tagged to, as training sample ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &quot;There is a garbage dump near the footpath. The footpath is broken completely&quot;. This text could be categorized into &lt;code&gt;garbage&lt;/code&gt; bucket or &lt;code&gt;footpath&lt;/code&gt; bucket. So, should this text be added as a training sample for &lt;code&gt;garbage&lt;/code&gt; and &lt;code&gt;footpath&lt;/code&gt;? How should we consider the labels?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please give your insights?&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="847" LastEditDate="2015-06-04T08:47:20.650" LastActivityDate="2015-06-04T08:47:20.650" Title="Making a labelled training data set" Tags="|text-mining|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5970" PostTypeId="1" CreationDate="2015-06-01T11:51:45.187" Score="2" ViewCount="1850" Body="&lt;p&gt;I am solving a rare event cum classification problem. I have come across  a package called &lt;strong&gt;SMOTEBoost&lt;/strong&gt; which oversamples the rare event and boost the results. But I'm not sure is that supported in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please help me how can I use SMOTEBoost in R? Any examples?&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="40853" LastEditDate="2018-01-23T07:02:01.043" LastActivityDate="2018-01-23T07:02:01.043" Title="Package for SMOTEBoost in R" Tags="|machine-learning|classification|r|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5971" PostTypeId="2" ParentId="5945" CreationDate="2015-06-01T11:55:49.633" Score="1" Body="&lt;p&gt;Good ideas there. But start with simple things first and then ramp up to a more complex system once you have something already working and some human feedback on the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simpler solution is just a velocity check - if there is X number of failures per Y number of seconds, then send the notification. From your post it seems like 3 failures over 20 second time window should be a good start. It takes a fair bit of tinkering to get this kind of system running - work out the windowing functions, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck :)&lt;/p&gt;&#xA;" OwnerUserId="587" LastActivityDate="2015-06-01T11:55:49.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5972" PostTypeId="1" AcceptedAnswerId="5976" CreationDate="2015-06-01T12:36:23.203" Score="8" ViewCount="3630" Body="&lt;p&gt;I want to learn how a spam email detector is done. I'm not trying to build a commercial product, it'll be a serious learning exercise for me. Therefore, I'm looking for resources, such as existing projects, source code, articles, papers etc that I can follow. I want to learn by examples, I don't think I am good enough to do it from scratch. Ideally, I'd like to get my hand dirty in Bayesian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything like that that? Programming language isn't a problem for me.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="847" LastEditDate="2015-06-01T22:21:47.310" LastActivityDate="2015-06-01T22:21:47.310" Title="How to learn spam email detection?" Tags="|machine-learning|classification|text-mining|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5973" PostTypeId="1" CreationDate="2015-06-01T13:10:41.363" Score="6" ViewCount="995" Body="&lt;p&gt;Given some dataset for prediction, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for eg say I have different housing price prediction dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataset  1 : 100 training and 100 testing sample, 50 feature&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataset  2 : 100 training and 100 testing sample, 120 feature &lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataset  3 : 1000 training and 1000 testing sample, 50 feature&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataset  4 : 1000 training and 1000 testing sample, 5000 feature&lt;/p&gt;&#xA;&#xA;&lt;p&gt;how should I choose the best methods for estimating the unknown parameters ( predict price) in a linear regression model from the following for each of these dataset?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Ordinary least squares&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Stepwise regression&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Principal component regression &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Partial least squares regression&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Should I experiment with each of these one by one and compare the results or is there any rule of thump on when to use each of them based on the dataset ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="29169" LastEditDate="2020-12-18T03:53:32.853" LastActivityDate="2020-12-18T03:53:32.853" Title="Choosing best methods for estimating the unknown parameters in a linear regression model" Tags="|machine-learning|regression|partial-least-squares|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5974" PostTypeId="2" ParentId="5968" CreationDate="2015-06-01T14:37:08.643" Score="2" Body="&lt;p&gt;Assuming your map is called &quot;M&quot; and you want your array field to be called &quot;A&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT &#xA;...&#xA;array(concat_ws(&quot;:&quot;,&quot;string1&quot;,M[1]),&#xA;      concat_ws(&quot;:&quot;,&quot;string2&quot;,M[2]),&#xA;      concat_ws(&quot;:&quot;,&quot;string3&quot;,M[3]) as A&#xA;....&#xA;FROM table;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-06-01T14:37:08.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5976" PostTypeId="2" ParentId="5972" CreationDate="2015-06-01T15:14:53.047" Score="6" Body="&lt;p&gt;First of all check &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Spambase&quot;&gt;this&lt;/a&gt; carefully. You'll find a simple dataset and some papers to review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BUT&lt;/strong&gt; as you want to start a simple learning project I recommend to not going through papers (which are obviously not &lt;em&gt;basic&lt;/em&gt;) but try to build your own bayesian learner which is not so difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally suggest &lt;a href=&quot;http://www.cs.cmu.edu/~guestrin/Class/10708-F08/schedule.html&quot;&gt;Andrew Moore&lt;/a&gt;'s lecture slides on Probabilistic Graphical Models which are freely available and you can learn from them simply and step by step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more detailed help just comment on this answer and I'll be glad to help :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Enjoy baysian learning!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-06-01T15:14:53.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5977" PostTypeId="2" ParentId="5972" CreationDate="2015-06-01T16:03:47.110" Score="6" Body="&lt;p&gt;In Andrew Ng's Machine Learning Course on Coursera (in someways the flagship course for Coursera) the programmers exercise for Support Vector Machines was an example doing a spam classifier. The lectures are great, famous even, and well worth watching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also this posted course from him:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;amp;doc=exercises/ex6/ex6.html&quot;&gt;http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;amp;doc=exercises/ex6/ex6.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7720" LastActivityDate="2015-06-01T16:03:47.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5978" PostTypeId="2" ParentId="5972" CreationDate="2015-06-01T16:32:03.140" Score="2" Body="&lt;p&gt;There is a basic introduction to the Bayesian method for spam detection in the book &quot;Doing Data Science - Straight Talk from the Frontline&quot; by Cathy O'Neil, Rachel Schutt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chapter is good, because it explains why other common data science models don't work for spam classifiers. The whole book uses R throughout, so only pick it up if you are interested in working with R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It uses the Enron email set as training data, since it has emails divided into spam/not spam already. &lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-06-01T16:32:03.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5979" PostTypeId="2" ParentId="5969" CreationDate="2015-06-01T17:08:46.350" Score="3" Body="&lt;p&gt;Generally with multiple classes you have to make a distinction between exclusive and inclusive groups. The simplest cases are &quot;all classes are exclusive&quot; (predict only one class), and &quot;all classes are compatible&quot; (predict list of classes that apply).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, label the classes as you would want your trained model to predict them. If you expect your classifier to predict an example is in both &lt;code&gt;garbage&lt;/code&gt; and &lt;code&gt;footpath&lt;/code&gt;, then you should label such an example with both. If you want it to disambiguate between them, then label with a single correct class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train a classifier to predict multiple target classes at once, it is usually just a matter of picking the correct objective function and a classifier with architecture that can support it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with a neural network, you would avoid using a &quot;softmax&quot; output which is geared towards predicting a single class - instead you might use a regular &quot;sigmoid&quot; function and predict class membership on a simple threshold on each output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get also more sophisticated perhaps with a pipeline model if your data can be split into several exclusive groups - predict the group in the first stage, and have multiple group-specific models predicting the combination of classes in each group in a second stage. This may be overkill for your problem, although it may still be handy if it keeps your individual models simple (e.g. they could all be logistic regression, and the first stage may gain some accuracy if the groups are easier to separate).&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-06-01T17:08:46.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5980" PostTypeId="2" ParentId="5565" CreationDate="2015-06-01T18:27:41.960" Score="2" Body="&lt;p&gt;Use &lt;a href=&quot;http://arxiv.org/pdf/1306.0040.pdf&quot; rel=&quot;nofollow&quot;&gt;Algorithm 1&lt;/a&gt; from this paper. The rest of paper is more about the analysis of algorithm and theoretical background which is very nice but not necessary for your purpose. Implementation of the algorithm in Python or R is pretty straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-06-01T18:27:41.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5981" PostTypeId="1" CreationDate="2015-06-01T18:54:14.077" Score="1" ViewCount="69" Body="&lt;p&gt;sorry if this question is out of place. I'm a begginer to machine learning, and I have use for a technique, and I don't even know where to look. The problem is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have 5 features which are real valued (Parameters in a deterministic simulation). &lt;/li&gt;&#xA;&lt;li&gt;This features determine two aspects of the instance (model solution). Its feasibility (binary) and some measure of likelihood given certain experimental data (only for the instances that achieved feasibility). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Since I want to avoid generating &quot;infeasible&quot; combination of features, what I devised was an algorithm that iteratively does the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Generate Nc candidate feature vectors&lt;/li&gt;&#xA;&lt;li&gt;Evaluate Feasibility and Likelihood for each&lt;/li&gt;&#xA;&lt;li&gt;Find linear combination of features that involves a compromise between least amount of features / holds largest cluster of feasibility. Add this as constraints to the feature vector generation. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In short, it detects and iteratively refines &quot;simple&quot; constraints that once added to the feature vector generation &quot;guarantee&quot; its feasibility to save computational time evaluating combination of parameters that lead to infeasible models. Afterwards, they could be tested by inverting them and looking for other &quot;regions&quot; (if any) of the feature vector where the model is feasible.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any name of techniques and references I might look for ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="9928" LastActivityDate="2015-06-05T02:49:48.877" Title="Iteratively construct Linear Classifier" Tags="|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5982" PostTypeId="2" ParentId="5972" CreationDate="2015-06-01T19:06:51.007" Score="2" Body="&lt;p&gt;This is actually a really in-depth problem that many people and companies have worked on. Here are some basics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, we need to represent the data well. This involves somehow representing each document as a vector in $d$-dimensional space. Ideally, in this space, we want samples that have the same label to be nearby in euclidean distance, and samples that are different labels to be far away in euclidean distance. This step can be really hard, but one tried-and-true representation is called Term Frequency-Inverse Document Frequency (tf-idf). Here, each dimension in the space represents a particular word, and the value in that dimension for a particular sample basically represents the normalized number of times that word occurs in the document. You could read more about that &lt;a href=&quot;http://www.tfidf.com/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. There's a pretty good scikit-learn implementation of this representation if you want to try it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the data is in a useful space, but a really high-dimensional space. I'd recommend reducing this dimensionality somehow, but that's a whole subject for another thread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you could train some algorithm to classify the samples (which is what the other answers are about). There are lots of great choices - neural networks, adaboost, SVMs, Naive Bayes, and graphical classification models will all give you good results. Many of these also have implementations in scikit-learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the best algorithms leverage the fact that this problem is actually a transfer learning one. That is, the distributions from which the training and testing data come might not be exactly the same - because the sorts of things one person thinks are spam might be different than the sorts of things another person thinks are spam.&lt;/p&gt;&#xA;" OwnerUserId="9483" LastEditorUserId="9483" LastEditDate="2015-06-01T19:21:15.973" LastActivityDate="2015-06-01T19:21:15.973" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="5983" PostTypeId="1" AcceptedAnswerId="5984" CreationDate="2015-06-02T08:39:41.997" Score="2" ViewCount="1712" Body="&lt;p&gt;I am trying to use a Random Forest Model (Regression Type) as a substitute of logistic regression model. I am using R - randomForest Package. I want to understand the meaning of Importance of Variables (%IncMSE and IncNodePurity) by example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a population of 100 employees out of which 30 left the company. &#xA;Suppose in a particular decision tree, population is split by an attribute (say location) into two nodes. One node contains 50 employees out of which 10 left the company and other contains 50 employees from which 20 left the company. Can someone demonstrate me a calculation of %IncMSE and IncNodePurity. (if Required for averages etc., please consider another decision tree)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may look like a repeated question but I could not find a worked out example.&lt;/p&gt;&#xA;" OwnerUserId="9941" LastEditorUserId="847" LastEditDate="2015-06-02T22:36:53.293" LastActivityDate="2015-06-04T16:37:32.417" Title="Random Forest, Type - Regression, Calculation of Importance Example" Tags="|machine-learning|r|logistic-regression|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5984" PostTypeId="2" ParentId="5983" CreationDate="2015-06-02T13:13:55.013" Score="3" Body="&lt;p&gt;MSE is measure of error of the overall regression model, $\frac{1}{n}\sum\|y_i-\hat y_i\|^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an important variable, if it is replaced with random noise, you would imagine MSE with the faulty data to increase. &lt;strong&gt;IncMSE&lt;/strong&gt; (Incremental MSE) for a particular variable is how much the MSE will increase if the variable is completely randomized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is usually computed on the out-of-bag data.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Node purity is a measure of how homogeneous a node is. An example of node purity is information entropy, i.e. $-p_1\log p_1-p_0\log p_0$ if there are two classes.&#xA;For regression models, node &lt;em&gt;im&lt;/em&gt;purity is usually taken as the variance in a node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everytime you split a node, you do it to make the new nodes homogeneous, hence the purity increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IncPurity&lt;/strong&gt; of a variable is weighted average of incremental purity because of each split by this variable was used to split, with the node population as the weight.&lt;/p&gt;&#xA;" OwnerUserId="9946" LastEditorUserId="9946" LastEditDate="2015-06-04T16:37:32.417" LastActivityDate="2015-06-04T16:37:32.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5985" PostTypeId="1" AcceptedAnswerId="5988" CreationDate="2015-06-02T13:45:36.703" Score="1" ViewCount="40" Body="&lt;p&gt;I am trying to determine which site in our organization is in greater need of upgrades to SEP 12, so when I run a query to count, I get these type of numbers&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Group       Windows_SEP_11  Mac_SEP_11  Windows_SEP_12  Mac_SEP_12&#xA;Arizona\A   417                  29              219         6&#xA;Arizona\B   380                  20              282        15&#xA;Arizona\C   340                  30              383        507&#xA;Arizona\D   310                  104             186        857&#xA;Arizona\E   307                  74              403        243&#xA;Arizona\F   285                  171             522        14&#xA;Arizona\G   269                  1               559        41&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, when I find percentages, I get these numbers&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Group          Win_Sep_11_%   Mac_SEP_11_%  Windows_SEP_12_%    Mac_SEP_12_%&#xA;Boston/Site 1   100               0                0               0&#xA;Boston/Site 2   100               0                0               0&#xA;Boston/Site 3   94                0                0               5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And obviously, percentage isn't good indicator because Boston/Site 1 has only 3 computers, Boston/Site 2 only has 4 computers, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to analyze data? I ultimately need a visual of sites that have&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;many computers, and&lt;/li&gt;&#xA;&lt;li&gt;a great need for upgrades to SEP 12, i.e. if there are more computers with SEP 11 than SEP 12.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please point me in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="9947" LastEditorUserId="847" LastEditDate="2015-06-02T18:49:30.440" LastActivityDate="2015-06-02T22:14:07.250" Title="How to analyze which site has most numbers" Tags="|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5986" PostTypeId="2" ParentId="5941" CreationDate="2015-06-02T16:31:29.623" Score="4" Body="&lt;p&gt;Try before doing LDA look at the data - like doing TF, IDF and TFIDF analysis to identify such words which happen in all subject. If You have some taxonomy of Your product definition - consider using it. In my case it was really helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I experimented with LDA topic modeling for recommendation systems purposes. I've few runs for offers in our marketplace service. It is in some aspect similar to Your problem with products and a list of keywords. But in our case, we do not have a product definition and an offer description is created by a seller. So You can imagine what tokens soup You will get :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key point for me wast to analyze TF, IDF and TFIDF inside categories where I've tested LDA. E.g. for Toys category (&lt;a href=&quot;http://allegro.pl/zabawki-11818?ref=simplified-category-tree&quot; rel=&quot;nofollow&quot;&gt;http://allegro.pl/zabawki-11818?ref=simplified-category-tree&lt;/a&gt;) ~ .5M items, LDA based only on its names results in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TOPIC:&#xA;lego    0.02282500803373657&#xA;puzzle  0.016679246682968853&#xA;gra     0.01092812347216676&#xA;klocki  0.010650330607355207&#xA;trefl   0.006797740719581059&#xA;zabawka 0.006365688767908592&#xA;lalka   0.0063546887775893105&#xA;maskotka        0.006111593588558912&#xA;dzieci  0.005888846995546501&#xA;24h     0.005736876716620104&#xA;&#xA;TOPIC:&#xA;lego    0.02296167186908205&#xA;puzzle  0.016613064812376045&#xA;gra     0.010928234683331952&#xA;klocki  0.010607432835126579&#xA;trefl   0.006689061156804526&#xA;lalka   0.006317220992078405&#xA;zabawka 0.0062838718987015575&#xA;maskotka        0.006121026383268814&#xA;dzieci  0.0058569443925760144&#xA;24h     0.00578544965708585&#xA;&#xA;TOPIC:&#xA;lego    0.02285507528449409&#xA;puzzle  0.016539723998111246&#xA;gra     0.010892673154789407&#xA;klocki  0.010624244094063881&#xA;trefl   0.006683424205358961&#xA;lalka   0.006350949499009139&#xA;zabawka 0.006324510823409019&#xA;maskotka        0.006148070280704085&#xA;dzieci  0.0058401580163702565&#xA;24h     0.005782372531889951&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the rest looks almost the same. Before running LDA I just did basic text preprocessing - our dictionary based item-name stop words removal and text tokenization (but without steaming). Even giving more terms for topic descriptions results doesn't look promising. Then, after exporting word frequencies with its offer categories by looking at data and plots I've decided to remove terms with TF-IDF above some threshold. Yes, above - I've used the calculation provided by Spark 1.3.1 implementation (HasingTF + IDF from mllib, no ml). After doing this I received:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TOPIC:&#xA;interaktywny    0.0026985965530064884&#xA;pony    0.0026721637823727343&#xA;dmuchany        0.0015777211309733249&#xA;baterie 0.0012447186564456534&#xA;pojazdy 0.0011171017074143481&#xA;thomas  9.476877418459519E-4&#xA;dinozaury       8.823212274401862E-4&#xA;monsters        7.426822230409613E-4&#xA;heroes  7.365256561824247E-4&#xA;ninjago 7.344344320326593E-4&#xA;&#xA;TOPIC:&#xA;pony    0.0026867880330479327&#xA;interaktywny    0.0026803244251279693&#xA;dmuchany        0.0015797940843537916&#xA;baterie 0.001263424692131187&#xA;pojazdy 0.0010685733429671523&#xA;thomas  9.87888159025782E-4&#xA;heroes  8.26066846602493E-4&#xA;ptaszek 7.952454816383102E-4&#xA;dinozaury       7.849838129629457E-4&#xA;batman  7.408185413186286E-4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where results started to differ from each other. Taking like last time only 10-terms for every subject description. Still, it's not perfect but better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for me doing the experiments which few categories, the way to overcome problem was to remove it based on TF-IDF value. But, for every category the threshold was calculated separately. Mainly: mean(tf-idf) + 3*sd(tf-idf). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that Idf multiplication factor in Tf-Idf should resolve this problem by itself and the term should be punished for occurring in every document. But, using Spark implementation (idf = log((m + 1) / (d(t) + 1))) and our data it was simpler during a quick and dirty experiment to filter it that way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I find few sec. I will get back on this and place results with code and online to share this.&lt;/p&gt;&#xA;" OwnerUserId="9314" LastEditorUserId="9314" LastEditDate="2015-06-05T12:47:11.153" LastActivityDate="2015-06-05T12:47:11.153" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5987" PostTypeId="1" CreationDate="2015-06-02T20:16:43.627" Score="10" ViewCount="3702" Body="&lt;p&gt;I am trying to train an artificial neural network with two convolutional layers (c1, c2) and two hidden layers (c1, c2). I am using the standard backpropagation approach. In the backward pass I calculate the error term of a layer (delta) based on the error of the previous layer, the weights of the previous layer and the gradient of the activation in respect to the activation function of the current layer. More specifically the delta of layer l looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;delta(l) = (w(l+1)' * delta(l+1)) * grad_f_a(l)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am able to compute the gradient of c2, which connects into a regular layer. I just multiply the weights of h1 with it's delta. Then I reshape that matrix into the form of the output of c2, multiply it with the gradient of the activation function and am done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have a the delta term of c2 - Which is a 4D matrix of size (featureMapSize, featureMapSize, filterNum, patternNum). Furthermore I have the weights of c2, which are a 3D matrix of size (filterSize, filterSize, filterNum). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these two terms and the gradient of the activation of c1 I want to calculate the delta of c1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long story short:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the delta term of a previous convolutional layer and the weights of that layer, how do I compute the delta term of a convolutional layer?&lt;/p&gt;&#xA;" OwnerUserId="9769" LastActivityDate="2020-08-16T21:51:06.280" Title="How do I calculate the delta term of a Convolutional Layer, given the delta terms and weights of the previous Convolutional Layer?" Tags="|machine-learning|data-mining|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5988" PostTypeId="2" ParentId="5985" CreationDate="2015-06-02T22:14:07.250" Score="1" Body="&lt;p&gt;The most obvious way of visualizing this is to have the number of computers on the Y-axis and the size of the dots representing the percentages. The categories (or sites in your case) can be represented on the X-axis. The image below shows an example where the Y-axis represents a continuous value (can be mapped to number of computers in your case), the X-axis represents a discrete value (can be mapped to sites in your case), and the size of the dots represents another attribute (like percentage in your case). I have used the &lt;code&gt;R&lt;/code&gt; package &lt;code&gt;ggplot2&lt;/code&gt; for this. &lt;img src=&quot;https://i.stack.imgur.com/bXH9m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2015-06-02T22:14:07.250" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="5989" PostTypeId="2" ParentId="5970" CreationDate="2015-06-03T00:58:36.530" Score="1" Body="&lt;p&gt;There are two packages in &lt;code&gt;R&lt;/code&gt; that should be able to use &lt;code&gt;SMOTE&lt;/code&gt; to up sample the minority class: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;unbalanced&lt;/code&gt; &lt;a href=&quot;http://cran.r-project.org/web/packages/unbalanced/unbalanced.pdf&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt;, and &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;DMwR&lt;/code&gt; &lt;a href=&quot;http://cran.r-project.org/web/packages/DMwR/DMwR.pdf&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It doesn't seem like they provide the boosting that is described in the &lt;a href=&quot;http://www-users.cs.umn.edu/~aleks/papers/pkdd03.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; that you might be referring to. But it should be straightforward to sew in &lt;code&gt;SMOTE&lt;/code&gt; with the idea of boosting using some base tree packages in R like &lt;code&gt;rpart&lt;/code&gt; (for example). &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2015-06-03T00:58:36.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5990" PostTypeId="1" AcceptedAnswerId="6335" CreationDate="2015-06-03T05:56:50.243" Score="36" ViewCount="19717" Body="&lt;p&gt;I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I could think is to do transformation: &lt;code&gt;min(h, 23-h)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]&#xA;&#xA;Output: [0 1 2 3 4 5 6 7 8 9 10 11 11 10 9 8 7 6 5 4 3 2 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any standard to handle such attributes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: I will be using superviseed learning, to train random forest classifier!&lt;/p&gt;&#xA;" OwnerUserId="8338" LastEditorUserId="8338" LastEditDate="2015-06-03T06:59:13.933" LastActivityDate="2024-02-06T14:59:42.853" Title="What is a good way to transform Cyclic Ordinal attributes?" Tags="|feature-extraction|feature-scaling|featurization|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="5991" PostTypeId="1" CreationDate="2015-06-03T07:24:32.630" Score="2" ViewCount="516" Body="&lt;p&gt;I want to create and train a model which classifies a new text content into finance, programming, analytics, design etc. Where can I get a relevant dataset to train my models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;TIA. &lt;/p&gt;&#xA;" OwnerUserId="9958" LastEditorUserId="847" LastEditDate="2015-06-04T08:31:16.843" LastActivityDate="2015-06-05T04:24:14.807" Title="Where can I download a tagged dataset of text related to finance, programming, analytics etc.?" Tags="|machine-learning|dataset|multiclass-classification|" AnswerCount="0" CommentCount="2" ClosedDate="2015-09-10T07:09:40.287" ContentLicense="CC BY-SA 3.0" />
  <row Id="5992" PostTypeId="1" CreationDate="2015-06-03T12:19:59.147" Score="4" ViewCount="263" Body="&lt;p&gt;I am new to practicing NLP and most topics related, but I want to make a program that can gather and extract data for me on its own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more specific, I want to tell the program &quot;I want more information on this topic(i.e heart attacks)&quot;, and then the program shall find, gather and extract meaningful texts on the topic from around the www.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I happen to live in Norway, which means that most interesting data will be in English, but I also want to fetch interesting data found in Norwegian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One challenge is the differences in stop words. For instance, &quot;are&quot; and &quot;and&quot; are both stop words in English and subjects in Norwegian.&#xA;Other challenges are also likely to occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So my question is: Would I need to create separate algorithms for every natural language to be interpreted?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9960" LastActivityDate="2015-06-04T08:27:43.573" Title="Could one algorithm fetch keywords from texts of different natural languages?" Tags="|data-mining|nlp|algorithms|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5993" PostTypeId="2" ParentId="5990" CreationDate="2015-06-03T16:55:37.517" Score="3" Body="&lt;p&gt;The question is very interesting and I do not remember to read about interesting answers. Because of that I dare to give you one possible solution even if it looks crazy enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually one avoids having the same information in multiple features, since many algorithms can't handle that. But this is not the case of random forest. Contrasting linear regression (and all models based on similar ideas), random forests test all the features, by taking into consideration each feature one at a time. This way it is possible to code the same information in multiple ways without affecting learning performance, onyly space and running time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my suggestion would be to create 24 features, each of the form $(h+offset)%24$. It's like when you encode the time in local time zones. Thus you give the occasion to rf to detect using the same units some interesting agglomerations around some hours, because each possible hour has the chance to be encoded properly in at least 1 of 24 features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It waste some space and time, but I would giv it a try to see how that works.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-06-03T16:55:37.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5995" PostTypeId="2" ParentId="5992" CreationDate="2015-06-03T20:05:43.563" Score="5" Body="&lt;p&gt;There are a few ways to deal with this issue.  Python has a package called NLTK which contains stop word lists for several languages (including English and Norwegian).  You can simply use this package, it's usage is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from nltk.corpus import stopwords&#xA;&amp;gt;&amp;gt;&amp;gt; stop = stopwords.words('english')&#xA;&amp;gt;&amp;gt;&amp;gt; sentence = &quot;this is a foo bar sentence&quot;&#xA;&amp;gt;&amp;gt;&amp;gt; print [i for i in sentence.split() if i not in stop]&#xA;  ['foo', 'bar', 'sentence']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, a method for automatically suppressing stop words is called tf-idf;  tf-idf is commonly used in search engines so that the most important words are promoted to the forefront.  In your case, I would suspect you'd want to have IDF scores for both English and Norwegian and apply only the appropriate one on a language to language basis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot;&gt;http://en.wikipedia.org/wiki/Tf%E2%80%93idf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-06-03T20:05:43.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5996" PostTypeId="1" CreationDate="2015-06-03T20:29:40.490" Score="11" ViewCount="12297" Body="&lt;p&gt;I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They could very well be relational records in a table, except for the fact that they have one array field with IDs pointing to child records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What implementation looks more solid?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;JSON objects on a PostgreSQL database (just one large table with one column, namely the JSON object)&lt;/li&gt;&#xA;&lt;li&gt;JSON objects on a MongoDB&lt;/li&gt;&#xA;&lt;li&gt;Explode the JSON objects into columns and use arrays on PostgreSQL&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to maximize performance in joins, so I can massage the data and explore it until I find interesting analyses, at which point I think it will be better to transform the data into a form specific to each analysis.&lt;/p&gt;&#xA;" OwnerUserId="10171" LastActivityDate="2020-08-16T18:02:30.027" Title="Which is faster: PostgreSQL vs MongoDB on large JSON datasets?" Tags="|data-mining|bigdata|databases|sql|mongodb|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="5997" PostTypeId="2" ParentId="5996" CreationDate="2015-06-03T20:59:43.400" Score="12" Body="&lt;p&gt;For data load, Postgre outperforms MongoDB.&#xA;MongoDB is almost always faster when returning query counts.&#xA;PostgreSQL is almost always faster for queries using indexes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out this &lt;a href=&quot;http://tiborsimko.org/postgresql-mongodb-json-select-speed.html&quot;&gt;website&lt;/a&gt;  and &lt;a href=&quot;http://www.enterprisedb.com/postgres-plus-edb-blog/marc-linster/postgres-outperforms-mongodb-and-ushers-new-developer-reality&quot;&gt;this&lt;/a&gt; one too for more info. They have very detailed explanations. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2015-06-03T20:59:43.400" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="5998" PostTypeId="2" ParentId="5209" CreationDate="2015-06-03T23:19:42.263" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Named-entity_recognition#Formal_evaluation&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Named-entity_recognition#Formal_evaluation&lt;/a&gt; :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To evaluate the quality of a NER system's output, several measures&#xA;  have been defined. While accuracy on the token level is one&#xA;  possibility, it suffers from two problems: the vast majority of tokens&#xA;  in real-world text are not part of entity names as usually defined, so&#xA;  the baseline accuracy (always predict &quot;not an entity&quot;) is&#xA;  extravagantly high, typically &gt;90%; and mispredicting the full span of&#xA;  an entity name is not properly penalized (finding only a person's&#xA;  first name when their last name follows is scored as ½ accuracy).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In academic conferences such as CoNLL, a variant of the F1 score has&#xA;  been defined as follows:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Precision is the number of predicted entity name spans that line up exactly with spans in the gold standard evaluation data. I.e. when&#xA;  [Person Hans] [Person Blick] is predicted but [Person Hans Blick] was&#xA;  required, precision for the predicted name is zero. Precision is then&#xA;  averaged over all predicted entity names.&lt;/li&gt;&#xA;  &lt;li&gt;Recall is similarly the number of names in the gold standard that appear at exactly the same location in the predictions.&lt;/li&gt;&#xA;  &lt;li&gt;F1 score is the harmonic mean of these two.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;It follows from the above definition that any prediction that misses a&#xA;  single token, includes a spurious token, or has the wrong class,&#xA;  &quot;scores no points&quot;, i.e. does not contribute to either precision or&#xA;  recall.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-06-03T23:19:42.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="5999" PostTypeId="2" ParentId="2473" CreationDate="2015-06-04T03:31:21.937" Score="0" Body="&lt;p&gt;Have you tried Random Forest to do feature selection for categorical features. &#xA;Random Forest uses proximity calculation(information gain) on deciding to split on a particular feature.&#xA;Random Forest:&#xA;&quot;After each tree is built, all of the data are run down the tree, and proximities are computed for each pair of cases. If two cases occupy the same terminal node, their proximity is increased by one. At the end of the run, the proximities are normalized by dividing by the number of trees. Proximities are used in replacing missing data, locating outliers, and producing illuminating low-dimensional views of the data&quot;&#xA;For more check this link out:&#xA;&lt;a href=&quot;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5179" LastActivityDate="2015-06-04T03:31:21.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6000" PostTypeId="2" ParentId="5992" CreationDate="2015-06-04T07:32:02.597" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;So my question is: Would I need to create separate algorithms for every natural language to be interpreted?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, I believe so. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But building a model for detecting the used language is not hard: usually taking n-grams (n-shingles) and then doing classification on them works quite well in practice. By the way, for the start you can use stop words to detect a language, e.g. like it's described &lt;a href=&quot;http://blog.alejandronolla.com/2013/05/15/detecting-text-language-with-python-and-nltk/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then once the language is detected, I'd do the NLP stuff for each language separately. &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2015-06-04T07:32:02.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6001" PostTypeId="2" ParentId="5992" CreationDate="2015-06-04T08:27:43.573" Score="1" Body="&lt;p&gt;If I understood your question correctly, you want to be able to extract keywords from texts in different languages. One thing for sure, you will need a list of stopwords for each language. As someone else mentioned, that can be also obtained with TF-IDF. Algorithm for extracting keywords that I came across and seems promising is ToPMine - &lt;a href=&quot;http://web.engr.illinois.edu/~elkishk2/papers/ToPMine.pdf&quot; rel=&quot;nofollow&quot;&gt;http://web.engr.illinois.edu/~elkishk2/papers/ToPMine.pdf&lt;/a&gt; It is fairly well explained in this text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2015-06-04T08:27:43.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6004" PostTypeId="1" AcceptedAnswerId="6008" CreationDate="2015-06-04T15:30:18.877" Score="2" ViewCount="528" Body="&lt;p&gt;I have a data set of questions belonging to ten different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The categories are briefly described as follows: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Definition – A question that contains a definition of the answer. &lt;/li&gt;&#xA;&lt;li&gt;Category Relation – The answer has a semantic relation to the question where the relation is specified in the category.&lt;/li&gt;&#xA;&lt;li&gt;FITB – These are generic fill in the blank questions – some of them ask for the completion of a phrase. &lt;/li&gt;&#xA;&lt;li&gt;Abbreviation – The answer is an expansion of an abbreviation in the question. &lt;/li&gt;&#xA;&lt;li&gt;Puzzle – These require derivation or synthesis for the answer. &lt;/li&gt;&#xA;&lt;li&gt;Etymology – The answer is an English word derived from a foreign word. &lt;/li&gt;&#xA;&lt;li&gt;Verb – The answer is a verb. &lt;/li&gt;&#xA;&lt;li&gt;Number – The answer is a numeral. &lt;/li&gt;&#xA;&lt;li&gt;Date – The question asks for a date or a year. &lt;/li&gt;&#xA;&lt;li&gt;Factoid – A question is a factoid if its answer can be found on Wikipedia. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I used the Stanford core NLP package called shiftreducer to find out the Part-Of-Speech (POS) values for each question in a category. I thought of using this POS pattern as a discriminant among the classes but it turned out to be generalized since: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;All the classes follow a similar pattern&lt;/li&gt;&#xA;&lt;li&gt;Nouns top the POS count followed by Determinants, Prepositions, Adjectives, Plural nouns and finally verbs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What could be the other ways in which I could differentiate among the question categories? Or as my question was in its first place, &quot;What kind of features do I select for efficient categorization?&quot;&lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="9966" LastEditDate="2015-06-25T18:05:37.163" LastActivityDate="2015-06-25T18:05:37.163" Title="How to select features from text data?" Tags="|data-mining|text-mining|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6005" PostTypeId="1" CreationDate="2015-06-04T18:37:59.000" Score="3" ViewCount="2268" Body="&lt;p&gt;I want to make a 3D scatter plot of multiple data selections on a single plot (i.e same axes).&lt;/p&gt;&#xA;&lt;p&gt;I know that in 2D this is possible by using par() function like so:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;plot(6:25,rnorm(20),type=&amp;quot;b&amp;quot;,xlim=c(1,30),ylim=c(-2.5,2.5),col=2)&lt;/p&gt;&#xA;&lt;p&gt;par(new=T)&lt;/p&gt;&#xA;&lt;p&gt;plot(rnorm(30),type=&amp;quot;b&amp;quot;,axes=F,col=3)&lt;/p&gt;&#xA;&lt;p&gt;par(new=F)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;(source: &lt;a href=&quot;http://cran.r-project.org/doc/contrib/Lemon-kickstart/kr_addat.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cran.r-project.org/doc/contrib/Lemon-kickstart/kr_addat.html&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;Can I do something like that on a 3D plot, preferably an interactive 3D plot, like the ones created using plot3D from 'rgl' package?&lt;/p&gt;&#xA;" OwnerUserId="9982" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-02-09T12:42:41.447" Title="Plotting multiple datasets on a 3D plot" Tags="|r|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6006" PostTypeId="1" AcceptedAnswerId="6007" CreationDate="2015-06-04T19:22:50.990" Score="0" ViewCount="62" Body="&lt;p&gt;I am new to R Programming and just learned basics through codeschool.com&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our network spans the city, and it is divided into districts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to create a map that assigns a value (based on ratio of outdated software and new software) to each district.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This website has sample of 3-D maps that were created by R Programming, and I see one I am very interested in replicating, but for our city only. &lt;img src=&quot;https://i.stack.imgur.com/Q1HTK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.rforscience.com/portfolio/koppen-geiger/&quot; rel=&quot;nofollow noreferrer&quot;&gt;But when I see the source code&lt;/a&gt;, I don't see any mention of latitude or longitude. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My head is spinning, trying to figure out how I will input this, i.e latitude and longitude of a district in our city, versus an assigned ratio, which I believe will be read from a spreadsheet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any guidance.&lt;/p&gt;&#xA;" OwnerUserId="9947" LastActivityDate="2015-06-04T19:38:40.603" Title="R Programming, how to replicate for districts in a city" Tags="|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6007" PostTypeId="2" ParentId="6006" CreationDate="2015-06-04T19:38:40.603" Score="1" Body="&lt;p&gt;The latitude and longitude are stored in a &lt;a href=&quot;http://www.rforscience.com/wpmain/wp-content/uploads/2014/06/Koeppen-Geiger-ASCII.txt&quot; rel=&quot;nofollow&quot;&gt;text file&lt;/a&gt; that is read into a table, &lt;code&gt;Dat&lt;/code&gt;.  It has the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lat long climate-group&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your particular case, you would calculate &lt;code&gt;Dat[, 3]&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2015-06-04T19:38:40.603" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6008" PostTypeId="2" ParentId="6004" CreationDate="2015-06-04T21:53:54.960" Score="1" Body="&lt;p&gt;If I understand you correctly, you're looking to take the text of these questions and train classifiers to identify which of 10 categories they belong to. And you'd like to come up with a decent feature representation in order to do this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think your finding about part-of-speech is intuitive. It makes sense that in grammatical English (assuming your question data is written in English), most questions would follow similar part-of-speech sequences since grammatically correct questions follow a particular syntactic form (at least when posed interrogatively as in the case of &quot;When was George Washington born?&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you've ruled something out - which you should actually view as progress. If you haven't tried it already, one simple thing you might do is use the actual words within the questions as features. You could use any order n-gram you like, but unigrams stick out as an immediate linguistic feature to try. It seems likely to me that while the POS-tags are similar across classes, making them difficult to distinguish between, the actual &lt;i&gt;words&lt;/i&gt; being used in the questions may vary from class to class, giving your model a better shot of differentiating between classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, maybe words like &quot;time&quot;, &quot;year&quot;, and &quot;when&quot; co-occur more highly with the &lt;code&gt;Date&lt;/code&gt; class while words like &quot;numerical&quot; and &quot;quantity&quot; co-occur more with the &lt;code&gt;Number&lt;/code&gt; class (obviously, this is speculation - I haven't seen your data). You might also look at bigrams, trigrams, or any other number n-gram for this feature set as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, there may be other features you could generate using NLP methods that may be useful. I'm not familiar with the Shiftreducer software, but &lt;a href=&quot;http://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow&quot;&gt;Named-Entity Recognition&lt;/a&gt; could be helpful in generating features for the &lt;code&gt;Factoid&lt;/code&gt; class if there are many questions about proper nouns. Other really simple features such as length of the question (counted in number of tokens). A final thought would be to use only the tagged &lt;i&gt;verbs&lt;/i&gt; from your POS-tagger, tab these up and to see whether they differ between classes. This may be a useful feature for identifying questions present in your &lt;code&gt;Verb&lt;/code&gt; class. Hopefully, those are some ideas to get you started.&lt;/p&gt;&#xA;" OwnerUserId="4897" LastActivityDate="2015-06-04T21:53:54.960" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6009" PostTypeId="2" ParentId="5981" CreationDate="2015-06-05T02:49:48.877" Score="1" Body="&lt;p&gt;Your algorithm is very similar to a &lt;a href=&quot;http://en.wikipedia.org/wiki/Genetic_algorithm&quot; rel=&quot;nofollow&quot;&gt;Genetic Algorithm&lt;/a&gt;, which has almost the same 3 iterative steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would be missing a Termination condition to end the search for an answer, once your reach an acceptable one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can probably adapt your algorithm to fit under a formal GA search heuristic.&lt;/p&gt;&#xA;" OwnerUserId="9986" LastActivityDate="2015-06-05T02:49:48.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6011" PostTypeId="1" AcceptedAnswerId="6017" CreationDate="2015-06-05T04:21:09.853" Score="5" ViewCount="616" Body="&lt;p&gt;I am looking into creating a model to predict whether an item is &quot;Very Good&quot;, &quot;Good&quot;, &quot;Bad&quot; or &quot;Very Bad&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I fit the training data to the models, comparing the accuracy of the models during test stump me: should it matter if a model misclassified a G to VG while the other G to VB? What about a model that has two misclassifications of one level away versus another model with only one misclassification but three levels away (eg VG to VB)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any guideline on what is the common approach? Also, my thinking at the moment is that this should be a regression problem, but I'm happy to be corrected if I should approach this labeling of datasets more as a classification problem.&lt;/p&gt;&#xA;" OwnerUserId="1133" LastEditorUserId="1133" LastEditDate="2015-06-05T08:36:07.253" LastActivityDate="2015-06-05T09:47:34.947" Title="Comparing accuracy of models in ordinal regression / classification" Tags="|machine-learning|classification|regression|scoring|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6014" PostTypeId="2" ParentId="5730" CreationDate="2015-06-05T05:10:10.423" Score="1" Body="&lt;p&gt;As you accept vague answers:&#xA;Sranford NLP tools are strong for this kind of stuff. NER, POS Tagger maybe, Parsers, etc. Now for machine learning itself, I would try looking up at WEKA, it's got a lot of filtering, classifiers and clustering methods, including StringToWordVector filters, that are, in my opinion, fundamental to text classification. Mostly, the tags you should be looking for are Text Categorization, Natural Language Processing and even Sentiment Analysis if you will. &lt;/p&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2015-06-05T05:10:10.423" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6015" PostTypeId="1" AcceptedAnswerId="8874" CreationDate="2015-06-05T05:18:26.460" Score="7" ViewCount="14638" Body="&lt;p&gt;What are the general assumptions of a Random Forest Model? I could not find by searching online. For example, in a linear regression model, limitations/assumptions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It may not work well when there are non-linear relationship between dependent and independent variables. &lt;/li&gt;&#xA;&lt;li&gt;It may not work if the dependent variables considered in the model are linearly related. Therefore one has to remove correlated variable by some other technique. &lt;/li&gt;&#xA;&lt;li&gt;It assumes that model errors are uncorrelated and uniform (No hetroscedasticity). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Are there any assumptions/limitations on similar lines.&lt;/p&gt;&#xA;" OwnerUserId="9941" LastEditorUserId="9941" LastEditDate="2015-06-07T06:27:19.720" LastActivityDate="2016-02-15T19:21:44.170" Title="Assumptions/Limitations of Random Forest Models" Tags="|random-forest|ensemble-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6016" PostTypeId="1" CreationDate="2015-06-05T05:19:23.083" Score="0" ViewCount="963" Body="&lt;p&gt;I just started using machine learning and was wondering if anybody have cool ideas for a Startup project, I've seen this website &lt;a href=&quot;http://treato.com&quot; rel=&quot;nofollow&quot;&gt;Treato&lt;/a&gt;  and was amazed by it. &#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="9988" LastEditorUserId="9988" LastEditDate="2015-06-09T11:42:12.563" LastActivityDate="2015-06-09T11:42:12.563" Title="Machine Learning Startup Ideas" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2015-06-06T08:27:05.113" ContentLicense="CC BY-SA 3.0" />
  <row Id="6017" PostTypeId="2" ParentId="6011" CreationDate="2015-06-05T09:47:34.947" Score="3" Body="&lt;p&gt;Your classes express a certain order. You can classify apples to, say, &quot;green&quot;, &quot;red&quot; or &quot;yellow&quot;, and then every disagreement with a reference set is equal. After all, colours express no order. So as you already suggested, I would certainly use regression. Assume that the classes could be distributed as something like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Very bad = 0 - 0.25&lt;/li&gt;&#xA;&lt;li&gt;Bad = 0.25 - 0.50&lt;/li&gt;&#xA;&lt;li&gt;Good = 0.50 - 0.75&lt;/li&gt;&#xA;&lt;li&gt;Very good = 0.75 - 1.00&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, the mismatch of &lt;em&gt;Very good&lt;/em&gt; vs. &lt;em&gt;Bad&lt;/em&gt; is at least 0.25, where is must be at least 0.50 with &lt;em&gt;Very good&lt;/em&gt; vs. &lt;em&gt;Very bad&lt;/em&gt;, which gives a better and more honest impression of the performance of your model.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-06-05T09:47:34.947" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6018" PostTypeId="1" AcceptedAnswerId="6023" CreationDate="2015-06-05T11:02:43.003" Score="1" ViewCount="1539" Body="&lt;p&gt;I have an algorithm which have as an input about 20-25 numbers. Then in every step it uses some of these numbers with a random function to calculate the local result which will lead to the final output of A, B or C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since every step has a random function, the formula is not deterministic. This means that with the same input, I could have either A, B or C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought was to take step by step the algorithm and calculating mathematically the probability of each output. However, it is really difficult due to the size of the core.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My next thought was to use machine learning with supervised algorithm. I can have as many labeled entries as I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How many labeled inputs should I need for a decent approach of the probabilities? Yes, I can have as many as I want, but it needs time to run the algorithm and I want to estimate the cost of the simulations to gather the labeled data.&lt;/li&gt;&#xA;&lt;li&gt;Which technique do you suggest that works with so many inputs that can give the probability of the three possible outputs?&lt;/li&gt;&#xA;&lt;li&gt;As an extra question, the algorithm run in 10 steps and there is a possibility that some of the inputs will change in one of the steps. My simple approach is to not include this option on the prediction formula, since I have to set different inputs for some of the steps. If I try the advanced methods, is there any other technique I could use?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-06-05T15:23:58.380" Title="Create a prediction formula from data input" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6019" PostTypeId="1" CreationDate="2015-06-05T11:37:57.630" Score="1" ViewCount="237" Body="&lt;p&gt;&quot;Knowledge&quot; is crucial within several fields like Knowledge Discovery, Knowledge Distraction, Natural Language Processing, Data Mining, Big Data, etc etc etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is the definition of knowledge within these fields?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there 1 common definition, or does it depend on the exact context?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9960" LastActivityDate="2015-06-07T05:38:32.923" Title="What is the definition of knowledge within data science?" Tags="|definitions|knowledge-base|" AnswerCount="2" CommentCount="3" ClosedDate="2015-06-07T18:44:26.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="6021" PostTypeId="1" CreationDate="2015-06-05T15:03:41.860" Score="2" ViewCount="393" Body="&lt;p&gt;I am working on a stock market decision system. I have currently centered on gradient boosting as the likely best machine learning solution for the problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have 2 fundamental issues with my data owing to it being from the stock market having to do with it not being IID. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, because of the duration of average in some indicators use, some data-points are highly correlated. For example, the 2-year trailing return of a stock is not very different if measured a month ago. My understanding is that this requires a sampling (for ensembles) where I choose datapoints that are &quot;far away&quot; in time to make trees more independent. From what I can tell so far, Matlab does not have functionality to pick a random subspace with this criteria. When I was previously thinking of using simple bagging, I figured I would just build the trees myself from custom subspaces and aggregate them into an ensemble, but this won’t work if I want to do gradient boosting. Now, on this point I am not totally sure that it is so critical to have samples “far away.” My intuition is that it is better if they are, but even if they are not perhaps by right-sizing the percent of data sampled and having enough trees it gives the same result. I would love any insight on that issue and how I might be able to use LSboost in matlab on custom samples.  (One idea I have is to just to create a small number, like 5-10 custom samples, use LS-Boost on each, and then average them.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second fundamental problem is that data from a given stock is correlated/related to itself. I realized after thinking about it that this is of critical importance. Consider, it would likely be better, if there is enough data, to make a prediction for stock A from training data only or mostly from stock A than to use the entire market. Thus, I had been thinking of a “system” where I train on stock-specific data, stock-group data (where I use a special algorithm to group stocks), and the entire market, and then use a calculation (I can elaborate if interested) that determines which of these models is more likely to give the better result. If the input looks very different from the stock-specific training data, for example, then it will use the group or entire market. I am pretty convicted that some form of taking into account which stock the system is looking at is important to optimizing performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, on the second issue the question is what is the best way to organize this. Thinking naively, it would be great to simply feed categories to the predictor that indicate what stock it is looking at. However, my belief here from what I know about these algorithms is that this will have poor results on new data, because this predictor will assume that it has seen the full universe of potential outcomes for each stock, when many times this isn’t the case. (Say there is a stock with only a one year history with a big rally – the system will think the rally will continue regardless of how different the new data looks). So I feel like I have to do something like in the previous paragraph. I don’t know if there is some way for the system to “automatically” recognize when new data is sufficiently similar to stock-specific data to focus on a stock-specific prediction vs. when it is different and it should go to the default system with multiple stocks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have any insights on these issues and/or how to address them in Matlab or otherwise, I would very much appreciate. Thanks in advance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&#xA;Mike&lt;/p&gt;&#xA;" OwnerUserId="9995" LastActivityDate="2024-01-11T09:01:05.147" Title="How do I deal with non-IID data in gradient boosted random forest (for stock market)?" Tags="|random-forest|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6023" PostTypeId="2" ParentId="6018" CreationDate="2015-06-05T15:23:58.380" Score="2" Body="&lt;p&gt;I'm not sure if I understood your question! Probably better to plot a scheme at least. But according to what I guess from your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q2-&lt;/strong&gt; You probably need a simple &lt;a href=&quot;http://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;nofollow&quot;&gt;MLP&lt;/a&gt; (Multilayer Perceptron)! it's a traditional architecture for Neural Networks where you have $n$ input neurons (here 20-25), one or more hidden layers with several neurons and 3 neurons as output layer. If you use a &lt;a href=&quot;http://en.wikipedia.org/wiki/Sigmoid_function&quot; rel=&quot;nofollow&quot;&gt;sigmoid&lt;/a&gt; activation function ranged from 0 to 1, the output for each class will be $P(Y=1|X=x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q1-&lt;/strong&gt; So your question probably is: how many training data you need for learning a model? and to the best of my knowledge the answer is &lt;em&gt;as many as possible&lt;/em&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and about the last question, I really could not figure out what you mean. You apparently have a very specific task so I suggest to share more insight for sake of clarification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I could help a little!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-06-05T15:23:58.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6024" PostTypeId="2" ParentId="2651" CreationDate="2015-06-05T15:32:19.307" Score="45" Body="&lt;p&gt;This &lt;a href=&quot;https://github.com/ChristosChristofidis/awesome-deep-learning&quot;&gt;link&lt;/a&gt; contains an amazing amount of deep learning literature.&#xA;Summarizing it here(going in the order a beginner ideally should)-&#xA;NOTE: All these resources mainly use python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Andrew Ng's Coursera course is pretty good too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) For Neural networks, nobody explains it better than &lt;a href=&quot;http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/lecture-12-learning-neural-nets-back-propagation/&quot;&gt;Dr.Patrick Winston&lt;/a&gt;.&#xA;The assignments should be tried out for better understanding. They are in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) For a better understanding of Neural Networks, &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot;&gt;Michael Nielsen&lt;/a&gt;'s course should be done(as suggested by Alexey). It is pretty basic but it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) For deep neural networks, and implementing them faster on GPUs, there are multiple frameworks available, such as &lt;a href=&quot;http://deeplearning.net/software/theano/&quot;&gt;Theano&lt;/a&gt;, &lt;a href=&quot;http://caffe.berkeleyvision.org/&quot;&gt;Caffe&lt;/a&gt;, &lt;a href=&quot;http://pybrain.org/&quot;&gt;Pybrain&lt;/a&gt;, &lt;a href=&quot;http://torch.ch/&quot;&gt;Torch&lt;/a&gt;,etc.&#xA;Out of these Theano provides a better low level functionality that allows its user to create custom NNs. It is a python library, so being able to use numpy,scikit-learn, matplotlib, scipy along with it is a big plus.&#xA;The &lt;a href=&quot;http://deeplearning.net/tutorial/&quot;&gt;deep learning tutorial&lt;/a&gt; written by Lisa Lab should be tried out for a better understanding of theano.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) For Convolutional Neural Networks, follow &lt;a href=&quot;https://cs231n.github.io/&quot;&gt;andrej karpathy's tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) For unsupervised learning, follow &lt;a href=&quot;http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7) For an intersection of deep learning and NLP, follow &lt;a href=&quot;http://cs224d.stanford.edu/&quot;&gt;Richard Socher's class&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;8) For LSTMs, read &lt;a href=&quot;http://Hochreiter,%20S.,%20&amp;amp;%20Schmidhuber,%20J.%20(1997).%20Long%20short-term%20memory.%20Neural%20computation,%209(8),%201735-1780&quot;&gt;Hochreiter, S., &amp;amp; Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780&lt;/a&gt; and &lt;a href=&quot;http://Graves,%20Alex.%20Supervised%20sequence%20labelling%20with%20recurrent%20neural%20networks.%20Vol.%20385.%20Springer,%202012.&quot;&gt;Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is LSTM's &lt;a href=&quot;http://deeplearning.net/tutorial/code/lstm.py&quot;&gt;Theano code&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9943" LastEditorUserId="9943" LastEditDate="2015-06-05T15:45:19.417" LastActivityDate="2015-06-05T15:45:19.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6025" PostTypeId="2" ParentId="6016" CreationDate="2015-06-05T19:59:43.367" Score="2" Body="&lt;p&gt;First of all, I would get acquainted with some core concepts of Machine Learning before building websites:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Common algorithms, differences between them (Naive Bayes, KMeans, Regression, Support Vector Machines, Decision Trees, you name it...)&lt;/li&gt;&#xA;&lt;li&gt;Supervised vs. unsupervised&lt;/li&gt;&#xA;&lt;li&gt;Features and feature selection &lt;/li&gt;&#xA;&lt;li&gt;Classification evaluation (precision, recall, accuracy)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Machine learning is a huge field which is applied in many different areas. I don't know how proficient you are at this stage, but I would certainly play around with these things first in order to get a solid understanding of what you're dealing with. After that, the options are virtually limitless in a world of Big Data (forgive me for using the annoying &lt;em&gt;buzz-word&lt;/em&gt;). What is truly valuable is someone knows &lt;em&gt;how&lt;/em&gt; to do machine-learning rather than &lt;em&gt;what&lt;/em&gt; (because there's enough to go about).&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-06-05T19:59:43.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6027" PostTypeId="1" CreationDate="2015-06-05T21:52:05.920" Score="7" ViewCount="141" Body="&lt;p&gt;I have few points in $S^n$, i.e., the $n$-dimensional unit sphere embedded in $\mathbb{R}^{n+1}$, and I would like to project them down to $S^2$, i.e., the 2-dimensional unit sphere (embedded in $\mathbb{R}^3$) to visualize it with the constraint that neighboring points should be close by. I spent some time playing with t-sne but of course, the points no longer lie on $S^2$. I normalized the projections but that introduces weird distortions, for instance, if the variance of one dataset is very small in $S^n$ as compared to other, I expect the same to hold in their $S^2$ projections; that is not the case upon normalizing t-sne. Any ideas? I would really like something that makes the previous statement hold.&lt;/p&gt;&#xA;" OwnerUserId="10001" LastEditorUserId="31513" LastEditDate="2017-04-24T17:37:50.093" LastActivityDate="2017-04-24T17:37:50.093" Title="Projecting data from $S^n$ to $S^2$" Tags="|machine-learning|dimensionality-reduction|visualization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6029" PostTypeId="2" ParentId="5996" CreationDate="2015-06-06T06:51:08.527" Score="2" Body="&lt;p&gt;You may benefit more from the schemaless design of Mongodb. This means its very easy to modify data structures on the fly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no such thing as a join in Mongodb. So how one thinks about data and how to use it needs to be modified to account for document based and schemaless db environments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe speed becomes less important as perspective and priorities change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Todd&lt;/p&gt;&#xA;" OwnerUserId="10009" LastActivityDate="2015-06-06T06:51:08.527" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6030" PostTypeId="1" CreationDate="2015-06-06T07:00:13.363" Score="9" ViewCount="837" Body="&lt;p&gt;The Vowpal Wabbit (VW) apparently supports sequence tagging functionality via &lt;a href=&quot;http://www.umiacs.umd.edu/~hal/searn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SEARN&lt;/a&gt;. The problem is that I cannot find anywhere detailed parameter list with explanations and with some examples. The best I could find is &lt;a href=&quot;http://zinkov.com/posts/2013-08-13-vowpal-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zinkov's blog entry&lt;/a&gt; with a very short example. The &lt;a href=&quot;https://en.wikipedia.org/wiki/Vowpal_Wabbit&quot; rel=&quot;nofollow noreferrer&quot;&gt;main wiki page&lt;/a&gt; barely mentions SEARN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the checked out source code I found demo folder with some NER sample data. Unfortunately, the script running all the tests does not show how to run on this data. At least it was informative enough to see what is the expected format: almost the same as standard VW data format, except that entries are separated by blank lines (this is important).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current understanding is to run the following command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat train.txt | vw -c --passes 10 --searn 25 --searn_task sequence \&#xA;--searn_passes_per_policy 2 -b 30 -f twpos.vw&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;--searn 25&lt;/code&gt; - the total number of NER labels (?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;--searn_task sequence&lt;/code&gt; - sequence tagging task (?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;--searn_passes_per_policy 2&lt;/code&gt; - not clear what it does&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other parameters are standard to VW and need no additional explanation. Perhaps there are more parameters specific to SEARN? What is their importance and impact? How to tune them? Any rules of thumb?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers to examples will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="1330" LastEditDate="2017-11-26T03:05:54.260" LastActivityDate="2019-04-21T03:03:30.627" Title="Using Vowpal Wabbit for NER" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6031" PostTypeId="1" AcceptedAnswerId="6396" CreationDate="2015-06-06T11:59:55.387" Score="3" ViewCount="38" Body="&lt;p&gt;I have a number of weather stations, and I know their positions. I would like to interpolate measurements from them for various other positions as a weighted average of these stations, but of course I need weights for this. I am thinking the most logical choice here from a physics point of view will be weighting by the inverse of the distance to the station, but I haven't quite convinced myself that that is right, and am not sure if I should maybe be using the distance squared, or maybe something in between.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comments? Are there any other reasonable alternatives I should be considering?&lt;/p&gt;&#xA;" OwnerUserId="7720" LastEditorUserId="8820" LastEditDate="2016-02-08T05:47:50.460" LastActivityDate="2016-02-08T05:47:50.460" Title="Geometric weighting of temperature" Tags="|interpolation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6033" PostTypeId="2" ParentId="6019" CreationDate="2015-06-06T21:49:54.407" Score="3" Body="&lt;p&gt;The definition of &lt;em&gt;knowledge&lt;/em&gt; varies based on the context, but can be broadly defined as &lt;strong&gt;actionable&lt;/strong&gt; &lt;em&gt;information&lt;/em&gt; based on further analysis (statistical, heuristic, or otherwise) of those patterns, associations, or relationships identified in the &lt;em&gt;raw data&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Put another way, &lt;em&gt;knowledge&lt;/em&gt; is defined by what &lt;em&gt;information&lt;/em&gt; will most prove &lt;strong&gt;actionable&lt;/strong&gt; based on the needs of the client or user (sometimes called the semantics). &lt;/p&gt;&#xA;" OwnerUserId="10019" LastEditorUserId="10019" LastEditDate="2015-06-07T00:43:06.230" LastActivityDate="2015-06-07T00:43:06.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6034" PostTypeId="2" ParentId="6019" CreationDate="2015-06-07T05:38:32.923" Score="3" Body="&lt;p&gt;Knowledge is a &lt;em&gt;general&lt;/em&gt; term and I don't think that there exist definitions of knowledge for specific disciplines, domains and areas of study. Therefore, in my opinion, knowledge, for a particular subject domain, can be defined just as a &lt;em&gt;domain-specific&lt;/em&gt; (or &lt;em&gt;context-specific&lt;/em&gt;, as mentioned by @JGreenwell +1) &lt;strong&gt;perspective&lt;/strong&gt; (projection) of a general concept of knowledge.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-06-07T05:38:32.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6037" PostTypeId="1" AcceptedAnswerId="6073" CreationDate="2015-06-07T23:26:19.183" Score="0" ViewCount="4412" Body="&lt;p&gt;I'm not sure if this is more appropriate for SO or DS in Stack Exchange since technically it's not about coding: in &lt;code&gt;caret&lt;/code&gt; package for training in R, it's possible to train the model using &lt;code&gt;rpart&lt;/code&gt; or &lt;code&gt;rpart2&lt;/code&gt; as the method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that &lt;code&gt;rpart&lt;/code&gt; is an implementation of CART. What is &lt;code&gt;rpart2&lt;/code&gt; and how is it different from &lt;code&gt;rpart&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My eventual aim is actually to compare the difference between the tree generated by &lt;code&gt;rpart&lt;/code&gt; and &lt;code&gt;rpart2&lt;/code&gt;, because my result seems to imply &lt;code&gt;rpart2&lt;/code&gt; has better accuracy for my dataset, but I have no clue how to view the &lt;code&gt;rpart2&lt;/code&gt; tree.&lt;/p&gt;&#xA;" OwnerUserId="1133" LastActivityDate="2015-06-10T19:06:56.943" Title="rpart and rpart2" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6040" PostTypeId="1" CreationDate="2015-06-08T08:16:38.667" Score="2" ViewCount="204" Body="&lt;p&gt;I have a dataset of clients (their city, name, age, gender, number of children) and another dataset about the products that they have bought. i have been asked to do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;extract knowledge about client profiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what knowledge should I extract.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we are studying clustering and classification so they should connect to the question. what i thought about is to make clusters of clients. but i don't know what criterisas should I depend on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;should I just use a clustering algorithm like k means and let it give me the clusters ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any suggestion would be appreciated&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Update&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;These are the dataset that I have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Card: CardID, City, Region, PostalCode, CardStartDate, Gender, DateOfBirth, MaritalStatus, HasChildren, NumChildren, YoungestChild&lt;/li&gt;&#xA;&lt;li&gt;Item: ItemCode, ItemDescription, CategoryCode, SubCategoryCode, BrandCode, UpmarketFlag&lt;/li&gt;&#xA;&lt;li&gt;Transaction: Store, Date, Time, TransactionID, CardID, PaymentMethod&lt;/li&gt;&#xA;&lt;li&gt;Category: CategoryCode, CategDescription&lt;/li&gt;&#xA;&lt;li&gt;Transaction_Item: Store, Date, Time, TransactionID, ItemNumber, ItemCode, Amount&lt;/li&gt;&#xA;&lt;li&gt;SubCategory: SubCategoryCode, SubCategDescrip&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The teacher said that we &lt;strong&gt;should&lt;/strong&gt; categories the clients and then indicate which products best suit for each customer.&lt;/p&gt;&#xA;" OwnerUserId="10036" LastEditorUserId="10036" LastEditDate="2015-06-08T16:51:39.310" LastActivityDate="2015-06-08T17:19:23.270" Title="What kind of analysis can I do in this dataset?" Tags="|data-mining|classification|clustering|" AnswerCount="1" CommentCount="0" ClosedDate="2015-06-08T16:19:17.777" ContentLicense="CC BY-SA 3.0" />
  <row Id="6042" PostTypeId="1" AcceptedAnswerId="6062" CreationDate="2015-06-08T10:52:39.433" Score="1" ViewCount="926" Body="&lt;p&gt;I am working on a Rare event classification problem. I Have 95% of the data as a majority class and 5% of the data as the minority class. I use classification trees algorithm. I am measuring the goodness of the model using confusion matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the i have the minority class just 5% of the total data, even though my prediction performance of minority class is close to 70%, the total number of errors are high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here is my confusion matrix.&#xA;             0           1&#xA;     0     213812      7008&#xA;     1     29083       16877&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though the Minority class(class 1) has predicted 16877 times correctly(70% and the misclassifcation is just 30%, but the absolute value of the misclassifcation is very high(29083) comparing to the correctly predicted minotriy class (16877). Which makes the solution less usable for the business. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any idea on handling these kind of  issues in such rare event modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind note: I have  balanced the target variable using the SMOTE algorithm before applying Classification tree.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-06-10T02:19:16.067" Title="Poor performance shown on Rare event modeling" Tags="|classification|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6043" PostTypeId="2" ParentId="6040" CreationDate="2015-06-08T11:10:44.397" Score="1" Body="&lt;p&gt;You may go through two different approaches:&lt;/p&gt;&#xA;&lt;h2&gt;Unsupervised Learning (Clustering)&lt;/h2&gt;&#xA;&lt;p&gt;You can choose attributes who make someone's profile and try to cluster it (e.g. using k-means). If you look at clusters according to different attributes it may give you some insight about the data. Do not forget to exclude names as they are meaningless for analysis.&lt;/p&gt;&#xA;&lt;h2&gt;Supervised Learning (Classification)&lt;/h2&gt;&#xA;&lt;p&gt;You can use relevant attributes which may affect an output e.g. age and city may affect gender or Nr of children so if u put these as targets you can classify different users according to their age and city.&lt;/p&gt;&#xA;&lt;p&gt;The point is that your data is not suitable for more sophisticated analysis as your features are not many and too complicated (i.e. answering the questions like age distribution among a certain gender or the relation between cities, age and number of children or something like them are the only questions can be answered)&lt;/p&gt;&#xA;&lt;p&gt;I hope it helped and if there are more questions please comment here.&lt;/p&gt;&#xA;&lt;h2&gt;UPDATE 1&lt;/h2&gt;&#xA;&lt;p&gt;You can look at your data from different point of view so first try to choose one. For example clients can be clustered via their transactions (when they bought, what they bought, how expensive they bought, etc) or by their personal info (gender, age, payment method, etc). It gives you a first impression of your client data. Then go through clustering using all those information i.e. put personal and transactional features together. It gives you an overview of customers in general and might result in some significant clusters (categories).&lt;/p&gt;&#xA;&lt;p&gt;After that you can look at the problem from product point of view meaning you look at the transactional features and try to see different distributions and histograms to get an overview of what's happening to the product e.g. you can set time as the x axis and try to extract different time series like how many product are sold at this point of time?, what kind of people (age, gender, etc) bought at this point of time? and so on.&lt;/p&gt;&#xA;&lt;p&gt;For general analysis a dimensionality reduction (e.g. PCA) might reveal some information and give you insight. Please note that for any kind of analysis ONLY use relevant features e.g. IDs are not informative but categories.&lt;/p&gt;&#xA;&lt;p&gt;The most important point for recommending product to a customer (or predicting what they buy) is to use your input/output pair properly so the customer info (age, city, when does he/she usually buy something?, etc) are inputs and products (their categories, their type, whatever u know about them) are outputs.&lt;/p&gt;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;&lt;p&gt;PS1: From experience of doing an industrial version of exactly the same project I would say payment method is not much informative.&lt;/p&gt;&#xA;&lt;p&gt;PS2: StackExchange provides an upvote and an accept button for good answers!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-06-08T17:19:23.270" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6044" PostTypeId="1" AcceptedAnswerId="6045" CreationDate="2015-06-08T19:50:16.510" Score="1" ViewCount="390" Body="&lt;p&gt;What are xml datasets? Is it possible to convert them to csv files?&#xA;I'm working on a Java program and I sometimes download datasets wich are in a binary format, are those xml? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2015-06-08T21:40:20.450" Title="What is a XML dataset?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6045" PostTypeId="2" ParentId="6044" CreationDate="2015-06-08T21:40:20.450" Score="3" Body="&lt;p&gt;XML is a markup language similar to html. One uses tags with attributes to build data structures. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;sampleXML&amp;gt;&#xA;  &amp;lt;Menu&amp;gt;&#xA;    &amp;lt;Food&amp;gt;&#xA;      &amp;lt;item1&amp;gt;Spaghetti Bolognese&amp;lt;/item1&amp;gt;&#xA;      &amp;lt;item2&amp;gt;Spaghetti Carbonara&amp;lt;/item2&amp;gt;&#xA;    &amp;lt;/Food&amp;gt;&#xA;    &amp;lt;Drinks&amp;gt;&#xA;      &amp;lt;item1 class = 'drinks'&amp;gt;Sprite&amp;lt;/item1&amp;gt;&#xA;    &amp;lt;/Drinks&amp;gt;&#xA;  &amp;lt;/Menu&amp;gt;&#xA;&amp;lt;/sampleXML&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see XML employs tags such as &lt;code&gt;&amp;lt;Food&amp;gt;&amp;lt;/Food&amp;gt;&lt;/code&gt; and attributes such as &lt;code&gt;class = 'drinks'&lt;/code&gt; which is exactly what HTML has. To access the XML data in java you got couple choices. &lt;a href=&quot;http://www.mkyong.com/java/how-to-read-xml-file-in-java-dom-parser/&quot; rel=&quot;nofollow&quot;&gt;You can read it in as a string and parse it using built in DOM parser&lt;/a&gt;. Or you can use JAXB to &lt;a href=&quot;https://docs.oracle.com/javase/tutorial/jaxb/intro/&quot; rel=&quot;nofollow&quot;&gt;map XML directly to Java objects&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surely you can convert XML to a csv file. There are free websites online for that. Just google &quot;XML to csv converter.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binary files are not XML, but can be. This needs little explanation. The letters and words you see here are ASCII characters. This is human readable text. Each ASCII character has a binary representation. For example, j in binary is 1101010. A binary file is any file in computer language (0 an 1). A binary file can also be a combination of text and binary. You can convert binary to ASCII and those files you download may indeed be XML I described above. To convert binary to ASCII just google it.&lt;/p&gt;&#xA;" OwnerUserId="52" LastActivityDate="2015-06-08T21:40:20.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6048" PostTypeId="1" AcceptedAnswerId="6059" CreationDate="2015-06-09T09:37:11.537" Score="17" ViewCount="50692" Body="&lt;p&gt;I am working on a classification problem. I have a dataset containing equal numbers of categorical variables and continuous variables. How do I decide which technique to use, between a decision tree and logistic regression?&lt;/p&gt;&#xA;&lt;p&gt;Is it right to assume that logistic regression will be more suitable for continuous variables and that decision trees will be more suitable for both continuous and categorical variables?&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="100562" LastEditDate="2020-07-11T16:23:13.107" LastActivityDate="2022-09-12T02:11:31.463" Title="Should I use a decision tree or logistic regression for classification?" Tags="|classification|logistic-regression|decision-trees|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="6050" PostTypeId="1" CreationDate="2015-06-09T10:34:14.970" Score="0" ViewCount="205" Body="&lt;p&gt;I have a qualitative variable, e.g. &lt;code&gt;userId&lt;/code&gt;, which could take around 30,000 different coded values ($k$). I would like to represent this variable as a dummy variable. Coding this into a vector of size $k$ doesn't seem to be a good approach. Is there a more compact method for coding for this variable?&lt;/p&gt;&#xA;" OwnerUserId="8338" LastEditorUserId="609" LastEditDate="2015-06-11T06:19:51.683" LastActivityDate="2015-06-11T06:19:51.683" Title="Encoding for k-level qualitatative variable" Tags="|feature-extraction|" AnswerCount="2" CommentCount="7" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6051" PostTypeId="1" AcceptedAnswerId="6063" CreationDate="2015-06-09T11:27:38.940" Score="2" ViewCount="1332" Body="&lt;p&gt;I'm wondering if programmers tend to use AI APIs. And if so, what are they like? And where can I find a nice one for Java?&lt;/p&gt;&#xA;" OwnerUserId="9988" LastEditorUserId="29575" LastEditDate="2017-12-17T20:08:11.900" LastActivityDate="2017-12-17T20:08:11.900" Title="Do I need an Artificial Intelligence API?" Tags="|machine-learning|java|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6052" PostTypeId="1" AcceptedAnswerId="6061" CreationDate="2015-06-09T13:49:57.683" Score="4" ViewCount="1997" Body="&lt;p&gt;I'm not a NLP guy and I have this question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a text dataset containing terms which go like, &quot;big data&quot; and &quot;bigdata&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my purpose both of them are the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I detect them in NLTK (Python)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or any other NLP module in Python?&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2015-06-09T16:28:19.630" LastActivityDate="2017-04-20T21:43:35.100" Title="How to recognize a two part term when the space is removed? (&quot;bigdata&quot; and &quot;big data&quot;)" Tags="|python|nlp|text-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6053" PostTypeId="2" ParentId="6051" CreationDate="2015-06-09T13:51:35.770" Score="1" Body="&lt;p&gt;I don't use Java much but have you looked into machine learning with the Weka Java API?  You might also try Encog, mentioned &lt;a href=&quot;http://www.jeffheaton.com/ai/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, and if you search for &lt;em&gt;Practical Artificial Intelligence Programming With Java&lt;/em&gt; by Mark Watson, you might still be able to find a PDF version that's licensed for free non-commercial use.&lt;/p&gt;&#xA;" OwnerUserId="8953" LastActivityDate="2015-06-09T13:51:35.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6054" PostTypeId="1" CreationDate="2015-06-09T14:39:04.210" Score="23" ViewCount="17835" Body="&lt;p&gt;I am a beginner on Machine Learning.&#xA;In SVM, the separating hyperplane is defined as $y = w^T x + b$.&#xA;Why we say vector $w$ orthogonal to the separating hyperplane?&lt;/p&gt;&#xA;" OwnerUserId="8073" LastEditorUserId="847" LastEditDate="2015-06-10T11:45:57.750" LastActivityDate="2022-08-21T16:12:05.033" Title="In SVM Algorithm, why vector w is orthogonal to the separating hyperplane?" Tags="|machine-learning|svm|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6055" PostTypeId="2" ParentId="6054" CreationDate="2015-06-09T15:12:19.093" Score="13" Body="&lt;p&gt;Geometrically, the vector w is directed orthogonal to the line defined by $w^{T} x = b$. This can be understood as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First take $b = 0$. Now it is clear that all vectors, $x$, with vanishing inner product with $w$ satisfy this equation, i.e. all vectors orthogonal to w satisfy this equation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now translate the hyperplane away from the origin over a vector a. The equation for the plane now becomes: $(x − a)^{T} w = 0$, i.e. we find that for the offset $b = a^{T} w$, which is the projection of the vector $a$ onto the vector $w$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without loss of generality we may thus choose a perpendicular to the plane, in which case the length $\vert\vert a \vert\vert = \vert b \vert /\vert\vert w\vert\vert$ which represents the shortest, orthogonal distance between the origin and the hyperplane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence the vector $w$ is said to be orthogonal to the separating hyperplane. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="-1" LastEditDate="2018-06-18T21:32:18.617" LastActivityDate="2018-06-18T21:32:18.617" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6056" PostTypeId="2" ParentId="6052" CreationDate="2015-06-09T15:16:41.170" Score="1" Body="&lt;p&gt;Your question seems vague without the sample data you are using. How does your data-set look like? If there are delimiters within your data, you could get rid of only spaces between all words and then 'big data' &amp;amp; 'bigdata' would be the same, if that is what you want to do. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2015-06-09T15:16:41.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6057" PostTypeId="2" ParentId="6048" CreationDate="2015-06-09T15:21:27.990" Score="7" Body="&lt;p&gt;Try using both regression and decision trees. Compare the efficiency of each technique by using a 10 fold cross validation. Stick to the one with higher efficiency. It would be difficult to judge which method would be a better fit just by knowing that your dataset is continuous and, or categorical.  &lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="9966" LastEditDate="2015-06-09T17:23:36.583" LastActivityDate="2015-06-09T17:23:36.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6058" PostTypeId="2" ParentId="6048" CreationDate="2015-06-09T15:34:12.437" Score="0" Body="&lt;p&gt;To use Decision Tree, you should transform the continuous variable into categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more thing, Logistic Regression is usually used to predict result according to the probability.&lt;/p&gt;&#xA;" OwnerUserId="8073" LastActivityDate="2015-06-09T15:34:12.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6059" PostTypeId="2" ParentId="6048" CreationDate="2015-06-09T16:58:44.353" Score="26" Body="&lt;p&gt;&lt;strong&gt;Long story short&lt;/strong&gt;: &lt;em&gt;do what @untitledprogrammer said, try both models and cross-validate to help pick one.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression should be able to handle continuous and categorical data just fine. For logistic regression, you'll want to &lt;a href=&quot;https://stats.stackexchange.com/questions/115049/why-do-we-need-to-dummy-code-categorical-variables&quot;&gt;dummy code your categorical variables&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @untitledprogrammer mentioned, it's difficult to know a priori which technique will be better based simply on the types of features you have, continuous or otherwise. It really depends on your specific problem and the data you have. (See &lt;a href=&quot;http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=585893&quot; rel=&quot;noreferrer&quot;&gt;No Free Lunch Theorem&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll want to keep in mind though that a logistic regression model is searching for a single linear decision boundary in your feature space, whereas a decision tree is essentially partitioning your feature space into half-spaces using &lt;em&gt;axis-aligned&lt;/em&gt; linear decision boundaries. The net effect is that you have a non-linear decision boundary, possibly more than one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is nice when your data points aren't easily separated by a single hyperplane, but on the other hand, decisions trees are so flexible that they can be prone to overfitting. To combat this, you can try pruning. Logistic regression tends to be less susceptible (but not immune!) to overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, another thing to consider is that decision trees can automatically take into account interactions between variables, e.g. $xy$ if you have two independent features $x$ and $y$. With logistic regression, you'll have to manually add those interaction terms yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have to ask yourself: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;what kind of decision boundary makes more sense in your particular problem?&lt;/li&gt;&#xA;&lt;li&gt;how do you want to balance bias and variance?&lt;/li&gt;&#xA;&lt;li&gt;are there interactions between my features?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, it's always a good idea to just try both models and do cross-validation. This will help you find out which one is more likely to have better generalization error.&lt;/p&gt;&#xA;" OwnerUserId="9576" LastEditorUserId="43124" LastEditDate="2017-12-11T21:08:38.943" LastActivityDate="2017-12-11T21:08:38.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6060" PostTypeId="2" ParentId="6050" CreationDate="2015-06-09T17:46:48.943" Score="1" Body="&lt;p&gt;There is always clustering techniques you could do that would help you determine how to group userId into a dummy variable.&lt;/p&gt;&#xA;" OwnerUserId="9698" LastActivityDate="2015-06-09T17:46:48.943" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6061" PostTypeId="2" ParentId="6052" CreationDate="2015-06-10T02:10:50.280" Score="9" Body="&lt;p&gt;There is a nice implementation of this in gensim: &lt;a href=&quot;http://radimrehurek.com/gensim/models/phrases.html&quot;&gt;http://radimrehurek.com/gensim/models/phrases.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase &quot;big data&quot; comes up a lot, then the class will learn to combine &quot;big data&quot; into a single token &quot;big_data&quot;. There is a more complete tutorial-style blog post about it here: &lt;a href=&quot;http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/&quot;&gt;http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8275" LastActivityDate="2015-06-10T02:10:50.280" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="6062" PostTypeId="2" ParentId="6042" CreationDate="2015-06-10T02:19:16.067" Score="3" Body="&lt;p&gt;If you are willing to use the caret package in R and use random forests, you can use the method in the following blog post for downsampling with unbalanced datasets: &lt;a href=&quot;http://appliedpredictivemodeling.com/blog/2013/12/8/28rmc2lv96h8fw8700zm4nl50busep&quot; rel=&quot;nofollow&quot;&gt;http://appliedpredictivemodeling.com/blog/2013/12/8/28rmc2lv96h8fw8700zm4nl50busep&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you just add a single line to your &lt;em&gt;train&lt;/em&gt; call. Here is the relevant part: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rfDownsampled &amp;lt;- train(Class ~ ., data = training,&#xA;+                        method = &quot;rf&quot;,&#xA;+                        ntree = 1500,&#xA;+                        tuneLength = 5,&#xA;+                        metric = &quot;ROC&quot;,&#xA;+                        trControl = ctrl,&#xA;+                        ## Tell randomForest to sample by strata. Here, &#xA;+                        ## that means within each class&#xA;+                        strata = training$Class,&#xA;+                        ## Now specify that the number of samples selected&#xA;+                        ## within each class should be the same&#xA;+                        sampsize = rep(nmin, 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have had some success with this approach in your type of situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some more context, here is an in-depth post about experiments with unbalanced datasets: &lt;a href=&quot;http://www.win-vector.com/blog/2015/02/does-balancing-classes-improve-classifier-performance/&quot; rel=&quot;nofollow&quot;&gt;http://www.win-vector.com/blog/2015/02/does-balancing-classes-improve-classifier-performance/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8275" LastActivityDate="2015-06-10T02:19:16.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6063" PostTypeId="2" ParentId="6051" CreationDate="2015-06-10T03:53:36.337" Score="3" Body="&lt;p&gt;One needs to use an &lt;em&gt;artificial intelligence (AI)&lt;/em&gt; API, if there is a need to add &lt;em&gt;AI functionality&lt;/em&gt; to a software application - this is pretty obvious. Traditionally, my advice on &lt;em&gt;machine learning (ML)&lt;/em&gt; software includes the following two excellent &lt;strong&gt;curated lists&lt;/strong&gt; of resources: &lt;a href=&quot;https://github.com/onurakpolat/awesome-bigdata&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt; and &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, keep in mind that ML is just a &lt;strong&gt;subset&lt;/strong&gt; of AI domain, so if your tasks involve AI areas &lt;em&gt;beyond&lt;/em&gt; ML, you need more &lt;em&gt;AI-focused&lt;/em&gt; tools or platforms. For example, you can take a look at &lt;a href=&quot;http://www.ai-one.com&quot; rel=&quot;nofollow&quot;&gt;ai-one&lt;/a&gt;'s AI platforms and APIs as well as interesting &lt;em&gt;general AI&lt;/em&gt; open source project &lt;a href=&quot;http://opencog.org&quot; rel=&quot;nofollow&quot;&gt;OpenCog&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the above-mentioned AI-focused platforms, IBM's &lt;a href=&quot;http://www.ibm.com/smarterplanet/us/en/ibmwatson&quot; rel=&quot;nofollow&quot;&gt;Watson&lt;/a&gt; AI system deserves a separate mention, as quite cool and promising. It offers its own &lt;em&gt;ecosystem&lt;/em&gt; for developers, called &lt;a href=&quot;http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud&quot; rel=&quot;nofollow&quot;&gt;IBM Watson Developer Cloud&lt;/a&gt;, based on IBM's &lt;a href=&quot;https://ace.ng.bluemix.net&quot; rel=&quot;nofollow&quot;&gt;BlueMix&lt;/a&gt; cloud computing &lt;em&gt;platform-as-a-service (PaaS)&lt;/em&gt;. However, at the present time, I find this offering to be quite expensive as well as limiting, especially for individual developers, small startups and other small businesses, due to its tight integration with and reliance only on a single PaaS (Blue Mix). It will be interesting to watch this space as competition in AI domain and marketplace IMHO will surely intensify in the future.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-06-10T04:12:53.907" LastActivityDate="2015-06-10T04:12:53.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6064" PostTypeId="1" CreationDate="2015-06-10T04:57:02.477" Score="5" ViewCount="312" Body="&lt;p&gt;New to R. In my example, my customers have restricted allocation of budget for Milk. I have more than 5 brands of milk in my store. Here my objective is how I know my customer is shifting from one brand to other? (Example: Customer is replacing Brand 1 with Brand 2 in my time series data). I would like to compute that shifting pattern every quarter and observe the trend quarter by quarter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Quarter Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date    Milk-Brand1 Milk-Brand2 Milk-Brand3 Milk-Brand4 Milk-Brand5&#xA;&#xA;1/1/2015    200 140 190 220 150&#xA;1/2/2015    204 138 195 226 144&#xA;1/3/2015    208 136 200 232 126&#xA;1/4/2015    212 134 205 238 108&#xA;2/2/2015    216 132 210 244 90&#xA;1/6/2015    220 130 215 250 72&#xA;1/7/2015    224 128 220 256 54&#xA;1/8/2015    228 126 225 262 36&#xA;1/9/2015    232 124 230 268 18&#xA;3/1/2015    236 122 235 274 0&#xA;3/2/2015    240 120 240 280 13&#xA;3/3/2015    244 118 245 286 33&#xA;3/4/2015    248 116 250 292 15&#xA;20/3/2015   252 114 255 298 33&#xA;20/3/2015   256 112 260 304 15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do you suggest compute correlation between each 'brand of milk' and compare those correlations from one quarter to other quarter? Or Cross-correlation? Or others? I am open.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the data via &lt;code&gt;dput&lt;/code&gt; for anyone wanting it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(Date = structure(c(1420070400, 1422748800, 1425168000, &#xA;1427846400, 1422835200, 1433116800, 1435708800, 1438387200, 1441065600, &#xA;1420243200, 1422921600, 1425340800, 1428019200, 1426809600, 1426809600&#xA;), tzone = &quot;UTC&quot;, class = c(&quot;POSIXct&quot;, &quot;POSIXt&quot;)), Milk.Brand1 = c(200L, &#xA;204L, 208L, 212L, 216L, 220L, 224L, 228L, 232L, 236L, 240L, 244L, &#xA;248L, 252L, 256L), Milk.Brand2 = c(140L, 138L, 136L, 134L, 132L, &#xA;130L, 128L, 126L, 124L, 122L, 120L, 118L, 116L, 114L, 112L), &#xA;    Milk.Brand3 = c(190L, 195L, 200L, 205L, 210L, 215L, 220L, &#xA;    225L, 230L, 235L, 240L, 245L, 250L, 255L, 260L), Milk.Brand4 = c(220L, &#xA;    226L, 232L, 238L, 244L, 250L, 256L, 262L, 268L, 274L, 280L, &#xA;    286L, 292L, 298L, 304L), Milk.Brand5 = c(150L, 144L, 126L, &#xA;    108L, 90L, 72L, 54L, 36L, 18L, 0L, 13L, 33L, 15L, 33L, 15L&#xA;    )), .Names = c(&quot;Date&quot;, &quot;Milk.Brand1&quot;, &quot;Milk.Brand2&quot;, &quot;Milk.Brand3&quot;, &#xA;&quot;Milk.Brand4&quot;, &quot;Milk.Brand5&quot;), row.names = c(NA, -15L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9663" LastEditorUserId="10663" LastEditDate="2015-10-21T08:05:25.770" LastActivityDate="2016-06-17T12:40:38.873" Title="Multivariate - Time series data pattern changes" Tags="|r|time-series|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6065" PostTypeId="2" ParentId="1034" CreationDate="2015-06-10T11:13:37.083" Score="9" Body="&lt;p&gt;By using a parse tree, you divide your sentence into parts. Suppose, in the example of sentiment analysis, you can use those parts to assign a positive/negative sentiment to each part and then take the cumulative effect of those parts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/vZhkR.png&quot; alt=&quot;sentiment analysis&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This image will help you understand more. The first half has a negative sentiment(mainly because of the word &quot;dry&quot;) but because of the word &quot;but&quot; and the usage of the word &quot;enjoyed&quot;, the negative sentiment is turned into a positive sentiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for using them, you can simply generate a &lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;word vector representation&lt;/a&gt; of the individual words in the sentence and use neurons in place of the parent nodes. Each neuron should be connected to another neuron through weights. All the leaf nodes will be the word vector representations of words of the sentence. The top parent neuron(in this case the top blue + symbol) should generate a positive/negative sentiment according to the sentence. This tree structure can be trained in a supervised manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read this &lt;a href=&quot;http://arxiv.org/abs/1503.00075&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; for a more through understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image credits: cs224.stanford.edu&lt;/p&gt;&#xA;" OwnerUserId="9943" LastEditorUserId="15620" LastEditDate="2017-04-20T09:20:31.550" LastActivityDate="2017-04-20T09:20:31.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6066" PostTypeId="1" CreationDate="2015-06-10T11:33:17.250" Score="3" ViewCount="782" Body="&lt;p&gt;I am very new to machine learning.&#xA;I have a text classification problem in hand. I have a tagged dataset of around 750 documents( short texts), categorized manually into 16 buckets. I want to train a classifier on this data. I know that there should be a training set and a test set (an option could be 80-20 ). In my understanding, this should be for the complete set( 80% of my 750 documents- training, 20% of 750 documents - testing ). &#xA;1. They should be randomly generated or is there some condition for category? ie. if category A constitutes 60%,category B 5%, C 7% etc. how to choose the training set?&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="6514" LastEditDate="2015-06-11T10:12:14.230" LastActivityDate="2015-06-11T10:12:14.230" Title="Text classification: Choosing training and testing set for each category" Tags="|machine-learning|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6067" PostTypeId="2" ParentId="6066" CreationDate="2015-06-10T13:00:47.707" Score="2" Body="&lt;p&gt;The most commonly used option is 2/3rd of the data as training and 1/3rd as testing sets respectively. What kind of software or tool are you using for this categorization process? For your classifier to work efficiently at the end of all the hard work, have you considered cross validating it using a multiple fold cross validation before using and assigning the training and testing sets? As in for the skewed data categories, try finding instances for the classes/ categories which have fewer samples (~10%).&lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="9966" LastEditDate="2015-06-10T13:23:42.853" LastActivityDate="2015-06-10T13:23:42.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6068" PostTypeId="1" CreationDate="2015-06-10T13:16:17.530" Score="-1" ViewCount="499" Body="&lt;p&gt;I am an undergraduate who needs to submit a thesis for graduation. I am fairly interested in deep learning, and am working on a project that uses deep learning methods extensively (rCNNs to be precise). The caveat here is that I am working alone, with little help from my advisor. There are hardly any experts on deep learning in my university. How do I go about finishing my project. It is specially intimidating to see that most papers in deep learning are published by multiple accomplished scientists with very little involvement of undergraduate level students. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am really interested in this topic, so please try not to advise me to switch to an easier topic.&lt;/p&gt;&#xA;" OwnerUserId="10085" OwnerDisplayName="Rohit" LastEditorUserId="10085" LastEditDate="2015-06-10T15:00:50.387" LastActivityDate="2019-06-08T13:50:04.293" Title="How can an undergraduate learn more about deep learning?" Tags="|machine-learning|deep-learning|research|" AnswerCount="2" CommentCount="3" ClosedDate="2019-06-08T13:50:12.130" ContentLicense="CC BY-SA 3.0" />
  <row Id="6069" PostTypeId="1" CreationDate="2015-06-10T14:25:05.840" Score="1" ViewCount="36" Body="&lt;p&gt;I have some data of form showed below to squezze through some neural network.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Customer Step Var0  ... VarN&#xA;0        0    some  ... stuff&#xA;0        1    again ... stuff&#xA;0        2    foo   ... bar&#xA;0        3    bla   ... blub&#xA;1        0    other ... stuff&#xA;1        1    and   ... ongoing&#xA;2        0    and   ... so on&#xA;.        .    .     ... .&#xA;.        .    .     ... .&#xA;.        .    .     ... .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the neural network has a fixed input layer size, but the customers step count can differ at each customer, I have following question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best practise (if exists) to rearange the given data for using as input of a NN? &lt;/p&gt;&#xA;" OwnerUserId="10083" LastActivityDate="2015-06-10T14:25:05.840" Title="Rearange input data - unknown size of input" Tags="|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6070" PostTypeId="1" CreationDate="2015-06-10T17:27:05.703" Score="5" ViewCount="1134" Body="&lt;p&gt;I am interested in clustering $N$ time series of $T$ 'values' each. These values are distributions (which can be represented by their cumulative distribution functions (cdf), or their probability density functions (pdf), or more convenient forms &lt;a href=&quot;http://projects.csail.mit.edu/atemuri/wiki/images/f/fe/SrivastavaJermynJoshiCVPR2007.pdf&quot; rel=&quot;nofollow&quot;&gt;such as square-root pdfs yielding a simple spheric geometry&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For comparing given distributions, there is an extensive literature on statistical distances (KL, Hellinger, Wasserstein, and so on), but for comparing given time series of distributions, I am not sure whether there is any literature at all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such distances should somehow take into account dynamics information besides the distribution proximity at time t. Ideally, I wish I could have a kind of &lt;a href=&quot;http://arxiv.org/pdf/1506.00976v1.pdf&quot; rel=&quot;nofollow&quot;&gt;information factorization similar to this result&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if such distances already exist and whether this kind of problem has already been formulated in the literature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&#xA;edit for further precisions and answer to comments:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your answer, but dynamic time warping does not suit to my need. This dp technique only captures a rough similarity of shapes by allowing non-linear time distortion. But, it does not amount for the whole information in these time series, e.g. what about the distribution of distortions? Do the distributions of a given time series vary smoothly through time or violently? DTW is not always the solution, for instance, when working with random walks, it does not make sense to use a DTW since there are no time patterns! In this case, the only information is &quot;correlation&quot; and &quot;distribution&quot; (cf. Sklar's theorem in Copula Theory), and the paper cited above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- edit 2 Here are the papers that are somehow related to my question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pub.ist.ac.at/~chl/erc/papers/lampert-cvpr2015.pdf&quot; rel=&quot;nofollow&quot;&gt;Predicting the Future Behavior of a Time-Varying Probability Distribution&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/paper_files/BanerjeeDGS05.pdf&quot; rel=&quot;nofollow&quot;&gt;Clustering on the unit hypersphere using von Mises-Fisher distributions&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/p636.pdf&quot; rel=&quot;nofollow&quot;&gt;Unsupervised clustering of multidimensional distributions using earth mover distance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.machinelearning.org/archive/icml2009/papers/538.pdf&quot; rel=&quot;nofollow&quot;&gt;Hilbert space embeddings of conditional distributions with applications to dynamical systems&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="7966" LastEditorUserId="11097" LastEditDate="2015-10-18T12:37:21.007" LastActivityDate="2015-11-19T12:32:59.207" Title="Statistical distances for time series of distributions" Tags="|clustering|time-series|distance|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6071" PostTypeId="2" ParentId="6068" CreationDate="2015-06-10T18:47:11.447" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://www.coursera.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera&lt;/a&gt; and &lt;a href=&quot;https://www.edx.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;EdX&lt;/a&gt; have a courses on Neural Networks (which are the basis of Deep Learning).  Stanford also has a &lt;a href=&quot;http://deeplearning.stanford.edu/tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;good series&lt;/a&gt; on Deep Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/3751/route-to-picking-up-deep-learning&quot;&gt;This question&lt;/a&gt; also has some good tips. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, what is most helpful is to program the algorithms to help me understand them step by step.  Python is good for that purpose.&lt;/p&gt;&#xA;" OwnerUserId="178" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-06-10T18:47:11.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6073" PostTypeId="2" ParentId="6037" CreationDate="2015-06-10T19:06:56.943" Score="3" Body="&lt;p&gt;Both &lt;code&gt;rpart&lt;/code&gt; and &lt;code&gt;rpart2&lt;/code&gt; implement a CART and wrap the &lt;code&gt;rpart&lt;/code&gt; function from the &lt;strong&gt;rpart&lt;/strong&gt; library.  The difference is the constraints on the model each enforces.  &lt;code&gt;rpart&lt;/code&gt; uses the complexity parameter, &lt;code&gt;cp&lt;/code&gt;, while &lt;code&gt;rpart2&lt;/code&gt; uses the max tree depth, &lt;code&gt;maxdepth&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See: &lt;code&gt;train_model_list&lt;/code&gt; section of &lt;a href=&quot;http://cran.r-project.org/web/packages/caret/caret.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/caret/caret.pdf&lt;/a&gt;  and &lt;code&gt;rpart.control&lt;/code&gt; section of &lt;a href=&quot;http://cran.r-project.org/web/packages/rpart/rpart.pdf&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/rpart/rpart.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="178" LastActivityDate="2015-06-10T19:06:56.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6074" PostTypeId="2" ParentId="6050" CreationDate="2015-06-10T19:43:17.277" Score="-1" Body="&lt;p&gt;If you are using tree based methods, you can replace the qualitative variables with random numbers or with the counts of each class.&lt;/p&gt;&#xA;" OwnerUserId="10086" LastActivityDate="2015-06-10T19:43:17.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6076" PostTypeId="1" AcceptedAnswerId="6434" CreationDate="2015-06-10T20:39:15.903" Score="3" ViewCount="621" Body="&lt;p&gt;I am thinking of preprocessing techniques for the input data to a convolutional neural network (CNN) using sparse datasets and trained with SGD. In Andrew Ng's coursera course, &lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;nofollow&quot;&gt;Machine Learning&lt;/a&gt;, he states that it is important to preprocess the data so it fits into the interval $ \left[ 3, 3 \right] $ when using SGD. However, the most common preprocessing technique is to standardize each feature so $ \mu = 0 $ and $ \sigma = 1 $. When standardizing a highly sparse dataset many of the values will not end up in the interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am therefore curious - would it be better to aim for e.g. $ \mu = 0 $ and $ \sigma = 0.5 $ in order for the values be closer to the interval $ \left[ 3, 3 \right] $? Could anyone argue based on a knowledge of SGD on whether it is most important to aim for $ \mu = 0 $ and $ \sigma = 1 $ or $ \left[ 3, 3 \right] $?&lt;/p&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="3044" LastEditDate="2015-07-11T22:19:12.347" LastActivityDate="2015-07-11T22:19:12.347" Title="Most important part of feature standardization and how is standardization affected by sparsity?" Tags="|machine-learning|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6078" PostTypeId="1" CreationDate="2015-06-11T06:20:43.703" Score="4" ViewCount="401" Body="&lt;p&gt;Suppose I have data in the form of  Query/Document Pairs, along with corresponding relevance scores (or class labels). Is there a way to use topic modeling to devise a model so that later given a query and a document, we can predict its relevance score?&lt;/p&gt;&#xA;" OwnerUserId="10100" LastEditorUserId="13413" LastEditDate="2015-10-26T19:41:47.293" LastActivityDate="2016-02-24T05:03:29.527" Title="Topic models for Relevance Prediction" Tags="|text-mining|topic-model|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6079" PostTypeId="2" ParentId="744" CreationDate="2015-06-11T07:42:17.480" Score="3" Body="&lt;p&gt;From a geometric point of view, if all your data are unitary, $\forall x, ||x||^2 = \langle x,x \rangle = 1$, then the scalar product of two vectors defines an angle $\phi$, $\langle x,y \rangle = \cos \phi$, and you have a distance $\phi = \arccos \langle x,y \rangle$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually, all your data live on a unit sphere. Using a dot product as a distance will give you a chordal distance, but if you use this cosine distance, it corresponds to the length of the path between the two points on the sphere. That means, if you want an average of the two points, you should take the point in-between on this path (geodesic) rather than the mid-point obtained from the 'arithmetic average/dot product/euclidean geometry' since this point does not live on the sphere (hence essentially not the same object)!&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2015-06-11T07:42:17.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6080" PostTypeId="1" CreationDate="2015-06-11T07:53:26.120" Score="4" ViewCount="111" Body="&lt;p&gt;I am interested in parsing semi-structured text.&#xA;Assume that I have a text with labels of the kind: year_field, year_value, identity_field, identity_value, ..., address_field, address_value, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These fields and their associated values can be everywhere in the text, but usually they are near to each other, and more generally the text in organized in a (very) rough matrix, but rather often the value is just after the associated field with eventually some non-interesting information in between. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of different format can be up to several dozens, and is not that rigid (do not count on spacing, moreover some information can be added and removed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking toward machine learning techniques to extract all those (field,value) of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think metric learning and/or conditional random fields (CRF) could be of a great help, but I have not practical experience with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have already encounter a similar problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion or literature on this topic?&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2015-06-19T07:30:04.703" Title="metric learning and information retrieval" Tags="|text-mining|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6081" PostTypeId="2" ParentId="5303" CreationDate="2015-06-11T12:14:38.347" Score="1" Body="&lt;p&gt;I am not sure about what you are trying to do. But generally speaking, you can look at &lt;a href=&quot;http://ranger.uta.edu/~chqding/papers/KmeansPCA1.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; for the relation between k-means clustering and PCA, especially Theorem 3.3.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2015-06-11T12:14:38.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6082" PostTypeId="1" AcceptedAnswerId="6087" CreationDate="2015-06-11T15:40:45.990" Score="8" ViewCount="2625" Body="&lt;p&gt;I have a dataset like this. The data has been collected through a questionnaire and I am going to do some exploratory data analysis.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;windows &amp;lt;- c(&quot;yes&quot;, &quot;no&quot;,&quot;yes&quot;,&quot;yes&quot;,&quot;no&quot;)&#xA;sql     &amp;lt;- c(&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;no&quot;,&quot;no&quot;)&#xA;excel  &amp;lt;- c(&quot;yes&quot;,&quot;yes&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)&#xA;salary &amp;lt;- c(100,200,300,400,500 )&#xA;&#xA;test&amp;lt;- as.data.frame (cbind(windows,sql,excel,salary),stringsAsFactors=TRUE)&#xA;test[,&quot;salary&quot;] &amp;lt;- as.numeric(as.character(test[,&quot;salary&quot;] ))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have an outcome variable (salary) in my dataset and a couple of input variables (tools). How can I visualize a horizontal box plot like this:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/CNLC1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-06-16T18:41:10.750" LastActivityDate="2015-06-16T18:41:10.750" Title="visualize a horizontal box plot in R" Tags="|r|visualization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6083" PostTypeId="2" ParentId="6082" CreationDate="2015-06-11T16:48:26.687" Score="2" Body="&lt;p&gt;You are going to need to make a column that contains software info-- for example name it software and the salary column has the corresponding salary so something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Software   Salary&#xA; Microsoft  100&#xA; Microsoft  300&#xA; Microsoft  400&#xA; SQL        200&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on...then you can plot with the code below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- ggplot(test, aes(factor(software), salary))&#xA;p + geom_boxplot() + coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9698" LastActivityDate="2015-06-11T16:48:26.687" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6084" PostTypeId="1" AcceptedAnswerId="6593" CreationDate="2015-06-11T16:58:53.380" Score="22" ViewCount="30726" Body="&lt;p&gt;So, I want to create a Player Profile &lt;strong&gt;Radar Chart&lt;/strong&gt; something like this:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/neeVU.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/neeVU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Not only the scale of each variable different, but also I want a reversed scale for some statistics like the 'dispossessed' stat, where less actually means good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution for the variable scale for each statistic maybe is setting a benchmark and then calculating a score on a scale of 100? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, How do I display the actual numbers on the chart then? Also, how do I get the reversed scale for some of the statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently working in Excel. What is the most powerful tool to create a complex chart like this?&lt;/p&gt;&#xA;" OwnerUserId="10114" LastEditorUserId="31513" LastEditDate="2017-04-24T17:37:32.513" LastActivityDate="2020-05-15T21:38:53.983" Title="How do I create a complex Radar Chart?" Tags="|visualization|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6086" PostTypeId="1" CreationDate="2015-06-11T21:21:55.473" Score="12" ViewCount="2433" Body="&lt;p&gt;Suppose I have five sets I'd like to cluster. I understand that the SimHashing technique described here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://moultano.wordpress.com/2010/01/21/simple-simhashing-3kbzhsxyg4467-6/&quot;&gt;https://moultano.wordpress.com/2010/01/21/simple-simhashing-3kbzhsxyg4467-6/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;could yield three clusters (&lt;code&gt;{A}&lt;/code&gt;, &lt;code&gt;{B,C,D}&lt;/code&gt; and &lt;code&gt;{E}&lt;/code&gt;), for instance, if its results were:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A -&amp;gt; h01&#xA;B -&amp;gt; h02&#xA;C -&amp;gt; h02&#xA;D -&amp;gt; h02&#xA;E -&amp;gt; h03&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly, the MinHashing technique described in the Chapter 3 of the MMDS book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot;&gt;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;could also yield the same three clusters if its results were:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A -&amp;gt; h01 - h02 - h03&#xA;&#xA;B -&amp;gt; h04 - h05 - h06&#xA;      |&#xA;C -&amp;gt; h04 - h07 - h08&#xA;                  |&#xA;D -&amp;gt; h09 - h10 - h08&#xA;&#xA;E -&amp;gt; h11 - h12 - h13&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Each set corresponds to a MH signature composed of three &quot;bands&quot;, and two sets are grouped if at least one of their signature bands is matching. More bands would mean more matching chances.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I have several questions related to these:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Can SH be understood as a &lt;em&gt;single band&lt;/em&gt; version of MH?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Does MH necessarily imply the use of a data structure like Union-Find to build the clusters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Am I right in thinking that the clusters, in both techniques, are actually &quot;pre-clusters&quot;, in the sense that they are just sets of &quot;candidate pairs&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) If (3) is true, does it imply that I still have to do an $O(n^2)$ search inside each &quot;pre-cluster&quot;, to partition them further into &quot;real&quot; clusters? (which might be reasonable if I have a lot of small and fairly balanced pre-clusters, not so much otherwise)&lt;/p&gt;&#xA;" OwnerUserId="1242" LastEditorUserId="1242" LastEditDate="2015-06-12T17:24:21.580" LastActivityDate="2016-02-01T14:29:25.683" Title="MinHashing vs SimHashing" Tags="|clustering|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6087" PostTypeId="2" ParentId="6082" CreationDate="2015-06-11T22:34:19.643" Score="3" Body="&lt;p&gt;Let's start by creating some fake dataset. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;software = sample(c(&quot;Windows&quot;,&quot;Linux&quot;,&quot;Mac&quot;), n=100, replace=T) &#xA;salary = runif(n=100,min=1,max=100) &#xA;test = data.frame(software, salary)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should create a dataframe &lt;code&gt;test&lt;/code&gt; that will look like somewhat like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    software    salary&#xA;1    Windows 96.697217&#xA;2      Linux 29.770905&#xA;3    Windows 94.249612&#xA;4        Mac 71.188701&#xA;5      Linux 94.028326&#xA;6      Linux  7.482632&#xA;7        Mac 98.841689&#xA;8        Mac 81.152623&#xA;9    Windows 54.073761&#xA;10   Windows  1.707829&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT based on comment&lt;/strong&gt; Note that if the data does not already exist in the above format, it can be changed to this format. Let's take a data frame provided in the original question and lets assume the dataframe is called &lt;code&gt;raw_test&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    windows sql excel salary&#xA;1     yes  no   yes    100&#xA;2      no  yes  yes    200&#xA;3     yes  no   yes    300&#xA;4     yes  no    no    400&#xA;5      no  no   yes    500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, using the &lt;code&gt;melt&lt;/code&gt; function/ method from the &lt;code&gt;reshape&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt;, first create the dataframe &lt;code&gt;test&lt;/code&gt; (that will be used for final plotting) as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# use melt to convert from wide to long format &#xA;test = melt(raw_test,id.vars=c(&quot;salary&quot;))&#xA;# subset to only select where value is &quot;yes&quot;&#xA;test = subset(test, value == 'yes')&#xA;# replace column name from &quot;variable&quot; to &quot;software&quot; &#xA;names(test)[2] = &quot;software&quot;   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, you will get a datframe &lt;code&gt;test&lt;/code&gt; that looks like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  salary software value&#xA;1     100  windows   yes&#xA;3     300  windows   yes&#xA;4     400  windows   yes&#xA;7     200      sql   yes&#xA;11    100    excel   yes&#xA;12    200    excel   yes&#xA;13    300    excel   yes&#xA;15    500    excel   yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Having created the dataset. We will now generate the plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, create the bar plot on the left based on the counts of software that represents usage rate. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p1 &amp;lt;- ggplot(test, aes(factor(software))) + geom_bar() + coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, create the boxplot on the right. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p2 &amp;lt;- ggplot(test, aes(factor(software), salary)) + geom_boxplot() + coord_flip()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, place both these plots next to each other. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require('gridExtra')&#xA;grid.arrange(p1,p2,nrow=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should create a plot like: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/AbyXC.png&quot; alt=&quot;On the left pane shows the counts of how different software is being used represented through bar plots and on the right pane shows the distribution of salaries grouped by software used represented through box plots.&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2015-06-12T05:19:57.553" LastActivityDate="2015-06-12T05:19:57.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6088" PostTypeId="1" CreationDate="2015-06-12T00:35:25.937" Score="7" ViewCount="1284" Body="&lt;p&gt;I have a dataset that I want to classify as fraud/not fraud and I have many weak learners. My concern is that there is much more fraud than not fraud, so my weak learners perform better than average, but none perform better than 50% accuracy in the complete set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether I should set up testing and training sets that are half fraud and half not fraud or if I should just use a representative sample.&lt;/p&gt;&#xA;" OwnerUserId="10123" LastEditorUserId="847" LastEditDate="2015-06-13T07:23:11.003" LastActivityDate="2015-12-09T16:03:33.357" Title="Training and testing AdaBoost for low probability classification" Tags="|classification|class-imbalance|" AnswerCount="4" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6089" PostTypeId="1" AcceptedAnswerId="6090" CreationDate="2015-06-12T02:11:51.483" Score="3" ViewCount="73" Body="&lt;p&gt;Let's say there is a function $f$ such that $y = f(x)$. However, if $f$ is a piecewise function such that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = \begin{cases} 0 \quad x \leq 0 \\ 1 \quad x &amp;gt;0\end{cases} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I fit $f$ in that case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks, guys.&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="847" LastEditDate="2015-06-12T06:46:05.163" LastActivityDate="2015-06-12T06:46:05.163" Title="How to fit an odd relationship with a function?" Tags="|statistics|predictive-modeling|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6090" PostTypeId="2" ParentId="6089" CreationDate="2015-06-12T02:35:04.070" Score="2" Body="&lt;p&gt;The definition you gave &lt;em&gt;is&lt;/em&gt; the definition of the function. This is called the &lt;a href=&quot;http://en.wikipedia.org/wiki/Heaviside_step_function&quot; rel=&quot;nofollow&quot;&gt;Heaviside Step Function&lt;/a&gt;. There is not a simple &lt;em&gt;analytic&lt;/em&gt; way to express it (like as a ratio, product, or composition  of trigonometric functions, exponentials, or polynomials). Note that it is neither continuous nor differentiable at x = 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of cool ways to represent it. The coolest and most intuitive way is as an integral of a &lt;a href=&quot;http://en.wikipedia.org/wiki/Dirac_delta_function&quot; rel=&quot;nofollow&quot;&gt;Dirac Delta Function&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xA;H(x) = \int_{-\infty}^x { \delta(s)} \, \mathrm{d}s &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, though, that a Dirac Delta Function is itself not an &quot;official&quot; function, since it is not well-defined at x = 0. Check out &lt;a href=&quot;http://en.wikipedia.org/wiki/Distribution_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;Distribution Theory&lt;/a&gt; for some cool info on weird &quot;functions&quot; like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I think you may be trying to approximate this function, because you asked how to &quot;fit&quot; it. Taken straight from Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;For a smooth approximation to the step function, one can use the logistic   function&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;H(x) \approx \frac{1}{2} + \frac{1}{2}\tanh(kx) = \frac{1}{1+\mathrm{e}^{-2kx}},&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;where a larger k corresponds to a sharper transition at x = 0.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8275" LastActivityDate="2015-06-12T02:35:04.070" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6092" PostTypeId="1" CreationDate="2015-06-12T10:25:12.457" Score="0" ViewCount="1887" Body="&lt;p&gt;I am working on generating restaurant ratings automatically and I &lt;strong&gt;have various feature values&lt;/strong&gt; like delivery time, cost estimate, etc. I want to generate a rating for each restaurant between 0 to 5. But &lt;strong&gt;I don't have any training data or ground truth to validate&lt;/strong&gt;. This rating might vary with user. Most of the related work, mostly related to the Yelp data challenge, have some relevance score as training data. I though of using &lt;em&gt;reinforcement learning&lt;/em&gt; to learn the rating with user feedback, but not sure how to do that. Can anyone please suggest a relevant technique or algorithm for this problem?&lt;/p&gt;&#xA;" OwnerUserId="10127" LastEditorUserId="8820" LastEditDate="2016-01-18T13:05:34.027" LastActivityDate="2017-12-07T17:01:59.647" Title="How to generate ratings without training data?" Tags="|machine-learning|dataset|cross-validation|reinforcement-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6093" PostTypeId="2" ParentId="1240" CreationDate="2015-06-12T10:57:34.290" Score="1" Body="&lt;p&gt;There is another interesting technique in &lt;a href=&quot;https://www.cs.purdue.edu/homes/lsi/sigir04-cf-norm.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper called the decoupling normalization method. I've used it and found that the results are good. This finds the affinity of user to a particular item and then you can scale it to a scale of 5 or 10, whichever you want. Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="10127" LastActivityDate="2015-06-12T10:57:34.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6094" PostTypeId="2" ParentId="6092" CreationDate="2015-06-12T11:25:02.010" Score="2" Body="&lt;p&gt;If you model your system by means of reinforcement learning, you will make your system learn from the user's feedback. The system will provide a rating based on the input features (it may be just a random rating during the first stages, since you will not have any prior information), and then the user will tell the system how well it predicted such rating. Based on the difference between the suggested and the actual user's ratings, the reinforcement learning algorithm will refine the recommendation system's model in order to provide more accurate ratings in the future.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sutton's book on reinforcement learning (&lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://incompleteideas.net/book/the-book-2nd.html&lt;/a&gt;) is a good introduction to the reinforcement learning field. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="836" LastEditDate="2017-12-07T17:01:59.647" LastActivityDate="2017-12-07T17:01:59.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6095" PostTypeId="2" ParentId="6088" CreationDate="2015-06-12T12:44:06.463" Score="0" Body="&lt;p&gt;Training set must represent the dataset your application/algorithm is actually going to face. I suggest you to take a representative sample instead of dividing the training and test set with exactly half fraud an half non-fraud. But please make sure that the training set contains both positive and negative example for fraud for your classifier to perform better.&lt;/p&gt;&#xA;" OwnerUserId="10127" LastActivityDate="2015-06-12T12:44:06.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6096" PostTypeId="2" ParentId="6068" CreationDate="2015-06-12T12:57:10.240" Score="1" Body="&lt;p&gt;There are many resources online where you can learn more about deep learning. Specially &lt;a href=&quot;https://github.com/ChristosChristofidis/awesome-deep-learning&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt; has a great collection. I recently forked this repository. You should go through the resources that interest you. Implementing is the best way to learn, where you get to see the results. As @Christopher mentioned, Pyhton is the best to learn for a beginner. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10127" LastActivityDate="2015-06-12T12:57:10.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6097" PostTypeId="1" CreationDate="2015-06-12T13:20:15.063" Score="1" ViewCount="38" Body="&lt;p&gt;I want to find the variables (and its values) used to build a  classification model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came to know that the following code does the same&lt;/p&gt;&#xA;&#xA;&lt;p&gt;get_all_vars(model_built, dataset_used_for_building_a_model)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I would not be able to use the above code as I am going to use the &quot;model_built&quot; alone in the R code and i will not be having the &quot;dataset_used_for_building_a_mode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply put i want to use this &quot;Model_built&quot; alone and fetch the variables used along with the values. &lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-06-12T13:20:15.063" Title="How to extract the values used in columns by a Classification model in R?" Tags="|machine-learning|classification|r|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6098" PostTypeId="1" CreationDate="2015-06-12T13:39:15.880" Score="5" ViewCount="212" Body="&lt;p&gt;I am trying to write the code of a Bernoulli block mixture model in matlab, but am facing an error every time I run the function. In particular, I'm having a problem with how to relate the distribution parameter $\alpha$ to the latent variables $Z$ and $W$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, \mathbf{Z} follows a multinomial distribution of parameter $\pi$, such that $\dim(\pi)=(1, g)$, where $g$ is the number of clusters of rows and $\sum \pi=1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W$ follows a multinomial distribution of parameter $\rho$ such that $\dim(\rho)=(1,m)$, where $m$ is the number of column clusters and $\sum \rho=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{Z}$ is a matrix of $\dim(N,g)$, and $\mathbf{W}$ is a matrix of $\dim(d,m)$, where $N$ is the number of observations, $p$ is the number of variables, $g$ is the number of row clusters, and $m$ is the number of column clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X$ follows a bernoulli distribution of parameter $\alpha_{ZW}$, where the notation $\alpha_{ZW}$ denotes the values of $\alpha$ depends on $\mathbf{Z}$ and $\mathbf{W}$, and $\alpha$ is a $\dim(g,m)$ matrix. Please find attached a graphical representation of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I write the code of $\alpha_{ZW}$ in order to get generate the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/btXXp.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/btXXp.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10131" LastEditorUserId="11097" LastEditDate="2015-11-18T12:26:47.673" LastActivityDate="2016-04-16T15:54:10.337" Title="how can I generate a Bernoulli block mixture model in matlab?" Tags="|machine-learning|clustering|matlab|binary|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6099" PostTypeId="1" AcceptedAnswerId="6130" CreationDate="2015-06-12T14:31:49.337" Score="0" ViewCount="882" Body="&lt;p&gt;I am looking for a simple way to sample from a &lt;a href=&quot;https://en.wikipedia.org/wiki/Von_Mises%E2%80%93Fisher_distribution&quot; rel=&quot;nofollow&quot;&gt;multivariate von Mises-Fisher&lt;/a&gt; distribution in Python. I have looked in &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.vonmises.html&quot; rel=&quot;nofollow&quot;&gt;the stats module in scipy&lt;/a&gt; and the &lt;a href=&quot;http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.vonmises.html#numpy.random.vonmises&quot; rel=&quot;nofollow&quot;&gt;numpy module&lt;/a&gt; but only found the univariate von Mises distribution. Is there any code available? I have not found yet.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- edit. Apparently, Wood (1994) has designed an algorithm for sampling from the vMF distribution according to &lt;a href=&quot;http://idg.pl/mirrors/CRAN/web/packages/movMF/movMF.pdf&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt;, but I can't find the paper.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastEditorUserId="7966" LastEditDate="2015-06-12T15:02:45.577" LastActivityDate="2015-06-16T09:28:02.837" Title="Sampling from a multivariate von Mises-Fisher distribution in Python" Tags="|python|distributed|sampling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6100" PostTypeId="1" CreationDate="2015-06-12T15:11:33.597" Score="3" ViewCount="517" Body="&lt;p&gt;I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I scrap articles from Hungarian newsportals and want to make a wordcloud out of it to show what are the current trending news topics. First I filter out stopwords and then stem the remaining words. (nltk has Hungarian stemmer) So I'm able to make a frequency table which can be the base of the wordcloud. My problem comes afterwards because stems are usually meaningless chunks (and not lemmas) of real words. I want to somehow complete the stem to a real word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first idea was to assign the most common word or the shortest one (or some combination of this 2 rules) to the stem and represent that in the wordcloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better solution for stem completion or should I follow a different workflow?&lt;/p&gt;&#xA;" OwnerUserId="5211" LastActivityDate="2024-03-22T06:04:15.530" Title="Complete a Hungarian stem to a real word" Tags="|python|nlp|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6101" PostTypeId="1" AcceptedAnswerId="6102" CreationDate="2015-06-12T18:33:57.383" Score="4" ViewCount="229" Body="&lt;p&gt;Assuming I can collect the demand of the purchase of a certain product that are of different market tiers. Example: Product A is low end goods. Product B is another low end goods. Product C and D are middle-tier goods and product E and F are high-tier goods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have collected data the last year on the following&#xA;1. Which time period (season - festive? non-festive?) does the different tier product reacts based on the price set? Reacts refer to how many % of the product is sold at certain price range&#xA;2. How fast the reaction from the market after marketing is done? Marketing is done on 10 June and the products are all sold by 18 June for festive season that slated to happened in July (took 8 days at that price to finish selling)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can data science benefit in terms of recommending&#xA;1. If we should push the marketing earlier or later?&#xA;2. If we can higher or lower the price? (Based on demand and sealing rate?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I understanding it right that data science can help a marketer in this aspect? Which direction should I be looking into if I am interested to learn about it.&lt;/p&gt;&#xA;" OwnerUserId="10138" LastEditorUserId="3466" LastEditDate="2015-06-13T07:23:04.537" LastActivityDate="2018-12-22T01:07:12.610" Title="Price optimization for tiered and seasonal products" Tags="|recommender-system|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6102" PostTypeId="2" ParentId="6101" CreationDate="2015-06-12T20:01:55.923" Score="4" Body="&lt;p&gt;You should be able to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;linear regression&lt;/a&gt; to find correlation between the factors which cause your products to sell better (or worse).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many correlations you can test against in this data set. Some examples are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If a product has been marketed aggressively, does it sell more quickly?&lt;/li&gt;&#xA;&lt;li&gt;If a low tier item is available, do fewer high-tier items sell?&lt;/li&gt;&#xA;&lt;li&gt;If multiple high-tier items are available, are fewer sold of each item?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that correlation does not necessarily imply causation. Always think about other factors which may cause sales to go up and down. For example, you may sell more high tier items in a season one year than another year. But, this could be due to changes in the overall economy, rather than changes in your pricing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second thing you can do is perform &lt;a href=&quot;https://en.wikipedia.org/wiki/A/B_testing&quot; rel=&quot;nofollow&quot;&gt;A/B tests&lt;/a&gt; on your product sales pages. This gives you clear feedback right away. Some example tests could be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Show the user one high-tier product and one low-tier product (A). Show the user two high-tier products and no low-tier products(B). Which page generates more revenue?&lt;/li&gt;&#xA;&lt;li&gt;Send out marketing emails for a seasonal sale 5 days in advance to one group of users (A). Send the same email to a different set of users 1 day in advance (B).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are many possibilities. Use your intuition and think about previous knowledge you have about your products.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-06-12T20:01:55.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6103" PostTypeId="1" CreationDate="2015-06-12T20:55:33.353" Score="4" ViewCount="627" Body="&lt;p&gt;I've trained an AWS Machine Learning model with the training data from here : &lt;a href=&quot;https://www.kaggle.com/c/titanic/data&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/titanic/data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm now trying to run a batch prediction with the test data from the same source but I get the following error when I try to load the data : &quot;  The schema in this data file must match the datasource used to create the ML model ml-xxxxxxxxx. Ensure that the data file you are using matches the schema structure.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The schema, as far as I can see, is identical. I have tried it with and without the 'survived' column which is the value I'm trying to predict. I even tried it with the same training set which obviously has an identical schema and got the same error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="10141" LastActivityDate="2019-12-26T15:01:40.220" Title="AWS machine learning prediction schema problems" Tags="|machine-learning|aws|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6105" PostTypeId="2" ParentId="6099" CreationDate="2015-06-13T02:06:13.920" Score="4" Body="&lt;p&gt;It looks like you can sample the von Mises-Fisher distribution with that R package. Have you thought about calling R from within Python using the &lt;a href=&quot;https://sites.google.com/site/aslugsguidetopython/data-analysis/pandas/calling-r-from-python&quot; rel=&quot;nofollow&quot;&gt;rpy2&lt;/a&gt; package? I haven't tried this for myself, but could you try the following?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from numpy import *&#xA;import scipy as sp&#xA;from pandas import *&#xA;from rpy2.robjects.packages import importr&#xA;import rpy2.robjects as ro&#xA;import pandas.rpy.common as com&#xA;from rpy2.robjects.packages import importr&#xA;&#xA;# import the movMF R package&#xA;movMF = importr('movMF')&#xA;# call the rmovMF sampling function from the R package&#xA;print(movMF.rmovMF(10, 3 * c(1, -1) / sqrt(2)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8275" LastActivityDate="2015-06-13T02:06:13.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6107" PostTypeId="1" AcceptedAnswerId="12110" CreationDate="2015-06-13T09:56:45.397" Score="283" ViewCount="276235" Body="&lt;p&gt;I recently read &lt;a href=&quot;http://arxiv.org/abs/1411.4038&quot;&gt;Fully Convolutional Networks for Semantic Segmentation&lt;/a&gt; by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what &quot;deconvolutional layers&quot; do / how they work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevant part is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;3.3. Upsampling is backwards strided convolution&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Another way to connect coarse outputs to dense pixels&#xA;  is interpolation. For instance, simple bilinear interpolation&#xA;  computes each output $y_{ij}$ from the nearest four inputs by a&#xA;  linear map that depends only on the relative positions of the&#xA;  input and output cells.&lt;br/&gt;&#xA;  In a sense, upsampling with factor $f$ is convolution with&#xA;  a fractional input stride of 1/f. So long as $f$ is integral, a&#xA;  natural way to upsample is therefore backwards convolution&#xA;  (sometimes called deconvolution) with an output stride of&#xA;  $f$. Such an operation is trivial to implement, since it simply&#xA;  reverses the forward and backward passes of convolution.&lt;br/&gt;&#xA;  Thus upsampling is performed in-network for end-to-end&#xA;  learning by backpropagation from the pixelwise loss.&lt;br/&gt;&#xA;  Note that the deconvolution filter in such a layer need not&#xA;  be fixed (e.g., to bilinear upsampling), but can be learned.&#xA;  A stack of deconvolution layers and activation functions can&#xA;  even learn a nonlinear upsampling.&lt;br/&gt;&#xA;  In our experiments, we find that in-network upsampling&#xA;  is fast and effective for learning dense prediction. Our best&#xA;  segmentation architecture uses these layers to learn to upsample&#xA;  for refined prediction in Section 4.2.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't think I really understood how convolutional layers are trained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I think I've understood is that convolutional layers with a kernel size $k$ learn filters of size $k \times k$. The output of a convolutional layer with kernel size $k$, stride $s \in \mathbb{N}$ and $n$ filters is of dimension $\frac{\text{Input dim}}{s^2} \cdot n$. However, I don't know how the learning of convolutional layers works. (I understand how simple MLPs learn with gradient descent, if that helps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if my understanding of convolutional layers is correct, I have no clue how this can be reversed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody please help me to understand deconvolutional layers?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2022-06-17T23:17:17.073" LastActivityDate="2022-06-17T23:17:17.073" Title="What are deconvolutional layers?" Tags="|neural-network|convolutional-neural-network|convolution|" AnswerCount="12" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6108" PostTypeId="1" CreationDate="2015-06-13T10:19:56.420" Score="2" ViewCount="1187" Body="&lt;p&gt;I used pretrained GoogleNet from &lt;a href=&quot;https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet&quot; rel=&quot;nofollow&quot;&gt;https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet&lt;/a&gt; and finetuned it with my own data (~ 100k images, 101 classes). After one day training I achieved 62% in top-1 and 85% in top-5 classification and try to use this network to predict several images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just followed example from &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/examples/classification.ipynb&quot; rel=&quot;nofollow&quot;&gt;https://github.com/BVLC/caffe/blob/master/examples/classification.ipynb&lt;/a&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my Python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import caffe&#xA;import numpy as np&#xA;&#xA;&#xA;caffe_root = './caffe'&#xA;&#xA;&#xA;MODEL_FILE = 'caffe/models/bvlc_googlenet/deploy.prototxt'&#xA;PRETRAINED = 'caffe/models/bvlc_googlenet/bvlc_googlenet_iter_200000.caffemodel'&#xA;&#xA;caffe.set_mode_gpu()&#xA;&#xA;net = caffe.Classifier(MODEL_FILE, PRETRAINED,&#xA;               mean=np.load('ilsvrc_2012_mean.npy').mean(1).mean(1),&#xA;               channel_swap=(2,1,0),&#xA;               raw_scale=255,&#xA;               image_dims=(224, 224))&#xA;&#xA;def caffe_predict(path):&#xA;        input_image = caffe.io.load_image(path)&#xA;        print path&#xA;        print input_image&#xA;        prediction = net.predict([input_image])&#xA;&#xA;&#xA;        print prediction&#xA;        print &quot;----------&quot;&#xA;&#xA;        print 'prediction shape:', prediction[0].shape&#xA;        print 'predicted class:', prediction[0].argmax()&#xA;&#xA;&#xA;        proba = prediction[0][prediction[0].argmax()]&#xA;        ind = prediction[0].argsort()[-5:][::-1] # top-5 predictions&#xA;&#xA;&#xA;        return prediction[0].argmax(), proba, ind&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my deploy.prototxt I changed the last layer only to predict my 101 classes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer {&#xA;  name: &quot;loss3/classifier&quot;&#xA;  type: &quot;InnerProduct&quot;&#xA;  bottom: &quot;pool5/7x7_s1&quot;&#xA;  top: &quot;loss3/classifier&quot;&#xA;  param {&#xA;    lr_mult: 1&#xA;    decay_mult: 1&#xA;  }&#xA;  param {&#xA;    lr_mult: 2&#xA;    decay_mult: 0&#xA;  }&#xA;  inner_product_param {&#xA;    num_output: 101&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;      value: 0&#xA;    }&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;prob&quot;&#xA;  type: &quot;Softmax&quot;&#xA;  bottom: &quot;loss3/classifier&quot;&#xA;  top: &quot;prob&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the distribution of softmax output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 0.01106235  0.00343131  0.00807581  0.01530041  0.01077161  0.0081002&#xA;   0.00989228  0.00972753  0.00429183  0.01377776  0.02028225  0.01209726&#xA;   0.01318955  0.00669979  0.00720005  0.00838189  0.00335461  0.01461464&#xA;   0.01485041  0.00543212  0.00400191  0.0084842   0.02134697  0.02500303&#xA;   0.00561895  0.00776423  0.02176422  0.00752334  0.0116104   0.01328687&#xA;   0.00517187  0.02234021  0.00727272  0.02380056  0.01210031  0.00582192&#xA;   0.00729601  0.00832637  0.00819836  0.00520551  0.00625274  0.00426603&#xA;   0.01210176  0.00571806  0.00646495  0.01589645  0.00642173  0.00805364&#xA;   0.00364388  0.01553882  0.01549598  0.01824486  0.00483241  0.01231962&#xA;   0.00545738  0.0101487   0.0040346   0.01066607  0.01328133  0.01027429&#xA;   0.01581303  0.01199994  0.00371804  0.01241552  0.00831448  0.00789811&#xA;   0.00456275  0.00504562  0.00424598  0.01309276  0.0079432   0.0140427&#xA;   0.00487625  0.02614347  0.00603372  0.00892296  0.00924052  0.00712763&#xA;   0.01101298  0.00716757  0.01019373  0.01234141  0.00905332  0.0040798&#xA;   0.00846442  0.00924353  0.00709366  0.01535406  0.00653238  0.01083806&#xA;   0.01168014  0.02076091  0.00542234  0.01246306  0.00704035  0.00529556&#xA;   0.00751443  0.00797437  0.00408798  0.00891858  0.00444583]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems just like random distribution with no sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any help or hint and best regards, Alex&lt;/p&gt;&#xA;" OwnerUserId="9167" LastActivityDate="2015-06-13T10:19:56.420" Title="Caffe net.predict() outputs random results (GoogleNet)" Tags="|machine-learning|python|deep-learning|image-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6109" PostTypeId="2" ParentId="6048" CreationDate="2015-06-13T18:09:22.033" Score="2" Body="&lt;p&gt;It really depends on the structure of the underlying distribution of your data. If you have strong reason to believe that the data approximate a Bernoulli distribution, multinomial logistic regression will perform well and give you interpretable results. However if there exist nonlinear structures in the underlying distribution, you should seriously consider a nonparametric method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While you could use a decision tree as your nonparametric method, you might also consider looking into generating a random forest- this essentially generates a large number of individual decision trees from subsets of the data and the end classification is the agglomerated vote of all the trees. A random forest helps give you an idea of the share each predictor variable contributes to the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another factor to keep in mind is interpretability. If you are just trying to classify data, then you probably don't care about the underlying relationships between explanatory and response variables. However, if you are interested at all in interpretability a multinomial logistic regression is much easier to interpret, parametric methods in general, because they make assumptions about the underlying distribution, tell you more intuitively interpretable relationships.&lt;/p&gt;&#xA;" OwnerUserId="10148" LastActivityDate="2015-06-13T18:09:22.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6110" PostTypeId="1" AcceptedAnswerId="6114" CreationDate="2015-06-13T18:36:49.023" Score="9" ViewCount="634" Body="&lt;p&gt;I'm studying machine learning and I feel there is a strong relationship between the concept of VC dimension and the more classical (statistical) concept of degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain such a connection?&lt;/p&gt;&#xA;" OwnerUserId="9766" LastActivityDate="2015-06-15T06:23:32.277" Title="Relationship between VC dimension and degrees of freedom" Tags="|machine-learning|self-study|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6111" PostTypeId="1" CreationDate="2015-06-13T18:45:53.480" Score="2" ViewCount="132" Body="&lt;p&gt;Similar questions have been asked in this context but this one seems a bit different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[Aim]&lt;/strong&gt;&lt;br&gt;&#xA;We would like to find out what the probability is of a person purchasing a 4USD column D (see below) price given that he has purchased C-2, B-2, A&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/mLlaR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[Context]&lt;/strong&gt;&lt;br&gt;&#xA;We have a sizeable dataset of about 500,000 observations of this pattern. E.g. customer decides to buy A), bought B-1, then C-2, D-3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[Issue]&lt;/strong&gt;&lt;br&gt;&#xA;We don't know how deep the events reach. E.g. it can be to column E, F, G. That's why we would like to auto-generate a probability tree in preferably SPSS but are open to other solutions as well.&#xA;How does this look like mathmatically? Can we just extend bayes conditional probability? If so, how does it look? Any potential issues?&#xA;Any other comments/ ideas are always welcome :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="10150" LastActivityDate="2015-11-25T03:14:00.273" Title="Probability tree in SPSS/ R/ others?" Tags="|bigdata|r|python|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6112" PostTypeId="2" ParentId="2432" CreationDate="2015-06-14T08:56:13.250" Score="0" Body="&lt;p&gt;I would say Teradata should be good choice. However, when you make join you have to take careful consideration of choosing right joining keys.&lt;/p&gt;&#xA;" OwnerUserId="5084" LastActivityDate="2015-06-14T08:56:13.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6113" PostTypeId="1" AcceptedAnswerId="6117" CreationDate="2015-06-14T10:19:31.923" Score="8" ViewCount="1686" Body="&lt;p&gt;I have a large sequence of vectors of length N. I need some unsupervised learning algorithm to divide these vectors into M segments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/O1hyC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-means is not suitable, because it puts similar elements from different locations into a single cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/VyVCU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, I see 3 clusters: &lt;code&gt;[0..50], [50..200], [200..250]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used modified k-means and got this acceptable result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/WBlZn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Borders of clusters: &lt;code&gt;[0, 38, 195, 246]&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="9637" LastEditorUserId="29169" LastEditDate="2019-06-08T03:21:30.757" LastActivityDate="2022-03-09T23:17:06.710" Title="Algorithm for segmentation of sequence data" Tags="|machine-learning|clustering|sequence|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="6114" PostTypeId="2" ParentId="6110" CreationDate="2015-06-14T10:32:51.277" Score="8" Body="&lt;p&gt;As stated by Prof Yaser Abu-Mostafa-&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Degrees of freedom are an abstraction of the effective number of parameters. The effective number is based on how many &lt;a href=&quot;https://en.wikipedia.org/wiki/Dichotomy&quot;&gt;dichotomies&lt;/a&gt; one can get, rather than how many real-valued parameters are used. In the case of 2-dimensional perceptron, one can think of slope and intercept (plus a binary degree of freedom for which region goes to +1), or one can think of 3 parameters w_0,w_1,w_2 (though the weights can be simultaneously scaled up or down without affecting the resulting hypothesis). The degrees of freedom, however, are 3 because we have the flexibility to shatter 3 points, not because of one way or another of counting the number of parameters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://qph.is.quoracdn.net/main-qimg-c943643a81e326633b68e3c40387e793?convert_to_webp=true&quot; alt=&quot;2-d perceptron&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="9943" LastActivityDate="2015-06-14T10:32:51.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6115" PostTypeId="1" AcceptedAnswerId="6118" CreationDate="2015-06-14T11:28:46.043" Score="2" ViewCount="8634" Body="&lt;p&gt;I am using the below R code to convert text to lower case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;movie_Clean &amp;lt;- tm_map(movie_Clean, content_transformer(tolower))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I end up getting the below error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in FUN(content(x), ...) :    invalid input 'I just wanna watch&#xA;  Jurassic World í ½í¸«' in 'utf8towcs'.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please help how to overcome this error.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="2961" LastEditDate="2015-06-14T19:55:03.203" LastActivityDate="2018-03-19T16:16:44.813" Title="How to convert a text to lower case using tm package?" Tags="|machine-learning|r|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6116" PostTypeId="1" CreationDate="2015-06-14T14:27:36.443" Score="25" ViewCount="526" Body="&lt;p&gt;With all the hoopla around Data Science, Machine Learning, and all the success stories around, there are a lot of both justified, as well as overinflated, expectations from Data Scientists and their predictive models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question to practicing Statisticians, Machine Learning experts, and Data Scientists is - how do you manage expectations from the businesspeople in you company, particularly with regards to predictive accuracy of models? To put it trivially, if your best model can only achieve 90% accuracy, and upper management expects nothing less than 99%, how do you handle situations like these?&lt;/p&gt;&#xA;" OwnerUserId="1127" LastActivityDate="2015-06-16T13:42:47.060" Title="How do you manage expectations at work?" Tags="|performance|accuracy|" AnswerCount="3" CommentCount="2" CommunityOwnedDate="2015-06-16T12:51:51.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="6117" PostTypeId="2" ParentId="6113" CreationDate="2015-06-14T19:07:42.017" Score="8" Body="&lt;p&gt;Please see my comment above and this is my answer according to what I understood from your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you correctly stated you do not need &lt;em&gt;Clustering&lt;/em&gt; but &lt;em&gt;Segmentation&lt;/em&gt;. Indeed you are looking for &lt;em&gt;Change Points&lt;/em&gt; in your time series. The answer really depends on the complexity of your data. If the data is as simple as above example you can use the difference of vectors which overshoots at changing points and set a threshold detecting those points like bellow:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/6BKAl.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;As you see for instance a threshold of 20 (i.e. &lt;span class=&quot;math-container&quot;&gt;$dx&amp;lt;-20$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$dx&amp;gt;20$&lt;/span&gt;) will detect the points. Of course for real data you need to investigate more to find the thresholds.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Pre-processing&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Please note that there is a trade-off between accurate location of the change point and the accurate number of segments i.e. if you use the original data you'll find the exact change points but the whole method is to sensitive to noise but if you smooth your signals first you may not find the exact changes but the noise effect will be much less as shown in figures bellow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ceDxi.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/PLIpr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Conclusion&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;My suggestion is to smooth your signals first and go for a simple clustering mthod (e.g. using &lt;a href=&quot;http://www.autonlab.org/tutorials/gmm14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;GMM&lt;/a&gt;s) to find an accurate estimation of the number of segments in signals. Given this information you can start finding changing points constrained by the number of segments you found from previous part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it all helped :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;UPDATE&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Luckily your data is pretty straightforward and clean. I strongly recommend dimensionality reduction algorithms (e.g. simple &lt;a href=&quot;http://sebastianraschka.com/Articles/2014_pca_step_by_step.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt;). I guess it reveals the internal structure of your clusters. Once you apply PCA to the data you can use k-means much much easier and more accurate.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;A Serious(!) Solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;According to your data I see the generative distribution of different segments are different which is a great chance for you to segment your time series. See &lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2012_KhaleghiRMP12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this (original&lt;/a&gt;, &lt;a href=&quot;https://web.archive.org/web/20170709232827/https://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2012_KhaleghiRMP12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;archive&lt;/a&gt;, &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v22/khaleghi12/khaleghi12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;other source&lt;/a&gt;) which is probably the best and most state-of-the-art solution to your problem. The main idea behind this paper is that if different segments of a time series are generated by different underlying distributions you can find those distributions, set tham as ground truth for your clustering approach and find clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example assume a long video in which the first 10 minutes somebody is biking, in the second 10 mins he is running and in the third he is sitting. you can cluster these three different segments (activities) using this approach.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="86439" LastEditDate="2019-12-05T12:58:32.643" LastActivityDate="2019-12-05T12:58:32.643" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="6118" PostTypeId="2" ParentId="6115" CreationDate="2015-06-14T20:18:40.730" Score="2" Body="&lt;p&gt;This seems like an encoding error. Try adding the line &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Encoding(movie_Clean)  &amp;lt;- &quot;UTF-8&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;before you lowercase the data. Check out this answer for a little context: &lt;a href=&quot;https://stackoverflow.com/a/28340080/4539807&quot;&gt;https://stackoverflow.com/a/28340080/4539807&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8275" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-06-14T20:18:40.730" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6119" PostTypeId="2" ParentId="6110" CreationDate="2015-06-15T06:23:32.277" Score="0" Body="&lt;p&gt;The VC dimension is very well explained in &lt;a href=&quot;http://research.microsoft.com/pubs/67119/svmtutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; in Section 2.1 and further, with the basic lemmas and proofs given. You can go through this.&lt;/p&gt;&#xA;" OwnerUserId="10127" LastActivityDate="2015-06-15T06:23:32.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6120" PostTypeId="1" CreationDate="2015-06-15T12:24:03.440" Score="1" ViewCount="357" Body="&lt;p&gt;[Apologies if this post sounds naive, I'm fairly new to the world of data science/big data and very unsure where I'm heading career-wise]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently an undergraduate MMath [integrated master's] Mathematics student in the UK who has finished the third year of the course [out of four years].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have been considering the possibility of doing further research in Mathematics/Statistics/Operational Research/Data Science, I have decided to stay on and complete the Master's component of the course [as it is the only Master's course I can get funding for at this stage]. After the Master's I may continue on and do a PhD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are currently two projects that appeal to me that seem to have relevant applications. The first one is on improved MCMC [Markov Chain Monte Carlo] methods, in particular MCMC using Hamiltonian Dynamics. There is scope for some big data applications here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other project that I could take part in is one on the centrality/communities detection of networks within network science. This could possibly be useful with applications in Operational Research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have an idea as to which project will be more relevant to data science/analytics?&lt;/p&gt;&#xA;" OwnerUserId="10170" LastActivityDate="2015-06-15T19:02:12.293" Title="MCMC vs Network Theory for Big Data and Data Science applications" Tags="|career|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6123" PostTypeId="1" CreationDate="2015-06-15T15:07:43.607" Score="3" ViewCount="2375" Body="&lt;p&gt;I have an excel file that contains details related to determining the quality of a wine and I want to implement the linear model concept using the function &lt;strong&gt;sklearn.linear_model.SGDClassifier(SVM =&gt; Hinge loss) and (Logarithmic regression =&gt;log loss)&lt;/strong&gt; using python. I learned the basics about these function through the &lt;em&gt;scikit learn&lt;/em&gt; website and I am not able to implement the model using excel file. I am very new to python and machine learning and I finding it hard to implement the model. I opened the excel file in python and tried to take two columns [randomly] from the file and use that as an input to call the &lt;strong&gt;fit&lt;/strong&gt; function available in the model. But, I got an error stating &lt;strong&gt;Unknown label type: array&lt;/strong&gt;. I tried a couple of other methods too, but, nothing worked. Can someone guide me with the implementation process? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from xlrd import open_workbook&#xA;from sklearn import linear_model&#xA;i = 0&#xA;fa = []&#xA;ph = []&#xA;&#xA;book = open_workbook('F:/BIG DATA/winequality.xlsx')&#xA;sheet = book.sheet_by_name('Sheet1')&#xA;num_rows = sheet.nrows - 1&#xA;num_cols = sheet.ncols - 1&#xA;curr_row = 0&#xA;while curr_row &amp;lt;num_rows:&#xA;    curr_row += 1&#xA;    cell_val = sheet.cell_value(curr_row,0)&#xA;    cell_val1 = sheet.cell_value(curr_row,10)&#xA;&#xA;    fa.append([float(cell_val),float(cell_val1)])&#xA;    cell_val2 = sheet.cell_value(curr_row,8)&#xA;    ph.append(float(cell_val2))&#xA;&#xA;model = linear_model.SGDClassifier()&#xA;print(model.fit(fa,ph))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/I9J8u.png&quot; alt=&quot;Screenshot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error message screenshot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/lCJTu.png&quot; alt=&quot;ERROR&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="5043" LastEditDate="2015-06-15T17:49:55.647" LastActivityDate="2020-08-03T08:06:38.270" Title="Implementing sklearn.linear_model.SGDClassifier using python" Tags="|machine-learning|classification|python|svm|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6124" PostTypeId="2" ParentId="6123" CreationDate="2015-06-15T15:43:22.097" Score="1" Body="&lt;p&gt;I think that this is the same issue as in this &lt;a href=&quot;https://stackoverflow.com/questions/24923143/x-and-y-have-incompatible-shapes&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The shape of &lt;span class=&quot;math-container&quot;&gt;$X$&lt;/span&gt; must be &lt;code&gt;(n_samples, n_features)&lt;/code&gt; as explained in the&#xA;&lt;code&gt;SVC.fit&lt;/code&gt; docstring. A 1-d array is interpreted as a single sample (for&#xA;convenience when doing predictions on single samples). Reshape your &lt;span class=&quot;math-container&quot;&gt;$X$&lt;/span&gt;&#xA;to &lt;code&gt;(n_samples, 1)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;That means you should use numpy.reshape to reshape the &lt;span class=&quot;math-container&quot;&gt;$X$&lt;/span&gt; column. If the data frame has &lt;code&gt;n&lt;/code&gt; rows, you should use&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;X_new = X.reshape(n, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Then use the fit method with &lt;span class=&quot;math-container&quot;&gt;$X_{new}$&lt;/span&gt;. Note: you probably don't need to do this if you use two or more &lt;span class=&quot;math-container&quot;&gt;$X$&lt;/span&gt; columns for your model fitting.&lt;/p&gt;&#xA;" OwnerUserId="8275" LastEditorUserId="98307" LastEditDate="2020-08-03T08:06:38.270" LastActivityDate="2020-08-03T08:06:38.270" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="6125" PostTypeId="2" ParentId="6116" CreationDate="2015-06-15T16:12:59.267" Score="10" Body="&lt;p&gt;Gather competitive counterparts. Try and determine a &lt;em&gt;state-of-the-art&lt;/em&gt; and see how your models compare with that. It also heavily depends on how long your team has been working on it. Science-driven models are not created statically, they develop dynamically because a good scientist will always try to find ways to improve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upper management personnel should know that a data &lt;strong&gt;scientist&lt;/strong&gt; explores &lt;strong&gt;new&lt;/strong&gt; methods, sometimes/often without knowing their quality. They should know that machine learning techniques do not produce perfect models right away. If they did, it wouldn't be challenging anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A data scientist ought to be evaluated by how he justifies and discusses his results and how he plans the future. A way for the management personnel to handle with their expectations is to not have unrealistically high ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, if reasonable results are expected in the field of context, think about these questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Did/will results get better over time?&lt;/li&gt;&#xA;&lt;li&gt;Are future expectations positive?&lt;/li&gt;&#xA;&lt;li&gt;How well are results compared to similar systems (from competitors)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-06-15T16:12:59.267" CommentCount="0" CommunityOwnedDate="2015-06-16T12:51:51.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="6126" PostTypeId="2" ParentId="6120" CreationDate="2015-06-15T19:02:12.293" Score="4" Body="&lt;p&gt;Both MCMC methods and network analysis play an important role in data science. I think you should go for the project you like more. However, in my experience, community detection in networks is a niche (applied math/graph theory) of network analysis, while MCMC methods involve lots of statistical and computational concepts. Personally, being a statistician, I would go for the MCMC methods.&lt;/p&gt;&#xA;" OwnerUserId="9766" LastActivityDate="2015-06-15T19:02:12.293" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6127" PostTypeId="2" ParentId="5799" CreationDate="2015-06-15T20:36:40.303" Score="3" Body="&lt;p&gt;It is so called because it classifies the linear sequence, and not because the structure of the graph.&lt;/p&gt;&#xA;" OwnerUserId="9637" LastActivityDate="2015-06-15T20:36:40.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6129" PostTypeId="2" ParentId="2651" CreationDate="2015-06-16T03:48:30.780" Score="11" Body="&lt;p&gt;Main references:&lt;/p&gt;&#xA;&lt;p&gt;Courses on deep learning:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.coursera.org/course/ml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng's course on machine learning&lt;/a&gt; has a nice introductory section on neural networks.&lt;/li&gt;&#xA;&lt;li&gt;Geoffrey Hinton's course: &lt;a href=&quot;https://www.coursera.org/course/neuralnets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera Neural Networks for Machine Learning (fall 2012)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Michael Nielsen's free book &lt;em&gt;Neural Networks and Deep Learning&lt;/em&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Yoshua Bengio, Ian Goodfellow and Aaron Courville wrote a &lt;a href=&quot;http://www.iro.umontreal.ca/%7Ebengioy/dlbook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;book on deep learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hugo Larochelle's course (videos + slides) at Université de Sherbrooke&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Main_Page&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford's tutorial (Andrew Ng et al.) on Unsupervised Feature Learning and Deep Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Oxford's ML 2014-2015 course&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://developer.nvidia.com/deep-learning-courses&quot; rel=&quot;nofollow noreferrer&quot;&gt;NVIDIA Deep learning course (summer 2015)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/course/deep-learning--ud730&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google's Deep Learning course on Udacity (January 2016)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;NLP-oriented:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs224d.stanford.edu/syllabus.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS224d: Deep Learning for Natural Language Processing (spring 2015) by Richard Socher&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/courses/NAACL2013/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tutorial given at NAACL HLT 2013: Deep Learning for Natural Language Processing (without Magic) (videos + slides)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Vision-oriented:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs231n.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n Convolutional Neural Networks for Visual Recognition&lt;/a&gt; by Andrej Karpathy (a previous version, shorter and less polished: &lt;a href=&quot;http://karpathy.github.io/neuralnets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hacker's guide to Neural Networks&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Toolkit-specific tutorials:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning4j.org/documentation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DL4J (Java)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano  (Python, Y. Bengio)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.madbits.com/wiki/doku.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning with Torch7 (Lua, LeCun)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://0xdata.com/product/deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;H2O Deep Learning (Java)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://caffe.berkeleyvision.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffee (C++, UCB)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.nervanasys.com/deep-learning-tutorials/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nervana’s Deep Learning Course&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="98307" LastEditDate="2020-08-20T18:59:26.887" LastActivityDate="2020-08-20T18:59:26.887" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6130" PostTypeId="2" ParentId="6099" CreationDate="2015-06-16T09:28:02.837" Score="1" Body="&lt;p&gt;Thanks to your help. I finally got my code working, plus some bibliography.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I put my hands on &lt;em&gt;Directional Statistics&lt;/em&gt; (Mardia and Jupp, 1999) and on the Ulrich-Wood's algorithm for sampling. I post here what I understood from it, i.e. my code (in Python), with a 'movMF' flavour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rejection sampling scheme:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def rW(n,kappa,m):&#xA;dim = m-1&#xA;b = dim / (np.sqrt(4*kappa*kappa + dim*dim) + 2*kappa)&#xA;x = (1-b) / (1+b)&#xA;c = kappa*x + dim*np.log(1-x*x)&#xA;&#xA;y = []&#xA;for i in range(0,n):&#xA;    done = False&#xA;    while not done:&#xA;        z = sc.stats.beta.rvs(dim/2,dim/2)&#xA;        w = (1 - (1+b)*z) / (1 - (1-b)*z)&#xA;        u = sc.stats.uniform.rvs()&#xA;        if kappa*w + dim*np.log(1-x*w) - c &amp;gt;= np.log(u):&#xA;            done = True&#xA;    y.append(w)&#xA;return y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, the desired sampling is $v \sqrt{1-w^2} + w  \mu$, where $w$ is the result from the rejection sampling scheme, and $v$ is uniformly sampled over the hypersphere.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def rvMF(n,theta):&#xA;dim = len(theta)&#xA;kappa = np.linalg.norm(theta)&#xA;mu = theta / kappa&#xA;&#xA;result = []&#xA;for sample in range(0,n):&#xA;    w = rW(kappa,dim)&#xA;    v = np.random.randn(dim)&#xA;    v = v / np.linalg.norm(v)&#xA;&#xA;    result.append(np.sqrt(1-w**2)*v + w*mu)&#xA;&#xA;return result&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And, for effectively sampling with this code, here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import scipy as sc&#xA;import scipy.stats&#xA;&#xA;n = 10&#xA;kappa = 100000&#xA;direction = np.array([1,-1,1])&#xA;direction = direction / np.linalg.norm(direction)&#xA;&#xA;res_sampling = rvMF(n, kappa * direction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2015-06-16T09:28:02.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6131" PostTypeId="2" ParentId="6116" CreationDate="2015-06-16T11:39:51.880" Score="5" Body="&lt;p&gt;I like this question because it gets at the politics that exist in every organization. In my view and to a significant degree, expectations about model performance are a function of the org culture and degree to which an organization is &quot;technically literate.&quot; One way to make clear what I mean is to consider the differences between the 4 big &quot;data science&quot; entities -- Google, FB, Amazon and Yahoo -- versus the 4 big agency holding entities -- WPP, Omnicon, Interbrand and Publicis. Google, et al, are very technically literate. The agencies, on the other hand, are known to lean towards tech phobia. What's the evidence for this? First off, the technically literate group was founded or are run by engineers, computer scientists, geeks and people with strong tech backgrounds. Who runs the tech illiterate companies? Marketers who have risen to prominence by virtue of their soft communication and people skills. And not only that, having worked in some of these shops in NYC, I can testify that these organizations systematically punish and/or push out the highly technically literate types as not a &quot;fit&quot; with the culture.&#xA;Next, consider their aggregate (stock) market caps, The tech literate group adds up to about 800 billion dollars while the tech illiterate group amounts to 80 billion. Tech literate entities are 10x bigger than the others in market cap. This is a clear statement of the market's expectations and it's not high for the illiterates. So, by extrapolation, what kind of hope can you have for challenging the &quot;predictive accuracy&quot; expectations of bozos like these?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, given that cultural breakout and depending on where you fall, you should have more or less realistic expectations. Of course, different &quot;tech illiterate&quot; entities will have managers who know what they're doing, but for the most part, these entities are dominated by the idiocy of the lowest common denominator in tech skills, i.e., people who are at best technical semi-literates (and dangerous) or, more commonly, totally innumerate but don't know it. Case in point, I worked for a guy who wanted words like &quot;correlation&quot; scrubbed from c-suite decks. This is an extreme case: after all, every secretary knows what a &quot;correlation&quot; is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This raises the issue of how one deals with the maddeningly naive and innumerate when they ask a really dumb question like, &quot;Why aren't you getting 99% predictive accuracy?&quot; One good response is to reply with a question like, &quot;Why would you assume such an unrealistically high PA is even possible?&quot; Another might be, &quot;Because if I actually got 99% PA, I would have assumed that I was doing something wrong.&quot; Which is highly likely to be true, even with 90% PA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The there's the more fundamental question of the insistence on PA as the sole criterion for model value. The late Leo Breiman left many footprints on the statistical and predictive modeling community of which PA is one. His primary concern with PA was to address the many criticisms being made in the 90s regarding the instability and error inherent in running a single CART tree. His solution was to motivate “random forests” as an approximate and provisional method that would maximize accuracy and reduce instability by eliminating tree structure.  He benchmarked the lower MSE from ~1,000 iterative RF “mini-models” against the error from a single logistic regression model. The only problem was that he never bothered to mention the glaring apples to oranges comparison: what might the logistic regression MSE have been if it had also been performed 1,000 times? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 2008 Netflix Prize offered a sizeable monetary reward to any statistician or team able to improve upon the MSE of their recommender system. At the time, Netflix was spending $150 million a year on this system, convinced that the costs were more than recovered in customer loyalty and purchase of movies that would otherwise never have been chosen. The eventual winners used a complex ensemble of 107 different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Netflix learned however, the real problem was that, from a fully loaded cost perspective, the actual improvement in error over their current model was a mere 0.005% reduction in the 5 point ratings. Not to mention that the IT costs in time, heavy-lifting and maintenance of the winning ensemble of 107 models more than nullified any gains from the error reduction. Given this, Netflix eventually abandoned the pursuit of MSE and no more Netflix Prizes have been awarded&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the point: minimizing predictive error can be easily gamed or p-hacked and is prone to analyst fraud (i.e., finding a solution that glorifies the analyst’s modeling skills, positively impacting his potential end-of-year bonus). Moreover, it is a completely statistical solution and goal set in an economic and business vacuum. The metric provides little or no consideration of ancillary, collateral costs -- the very real operational consequences evaluated from A to Z that should be an integral part of any fully-loaded, trade-off based decision-making process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has become one of those issues that is embedded in organizations and is very, very difficult to change. In other words, I am fully aware that I am tilting at windmills with this rant about the caveats with the use of PA.&lt;/p&gt;&#xA;" OwnerUserId="9974" LastActivityDate="2015-06-16T11:39:51.880" CommentCount="0" CommunityOwnedDate="2015-06-16T12:51:51.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="6132" PostTypeId="2" ParentId="6116" CreationDate="2015-06-16T13:42:47.060" Score="13" Body="&lt;p&gt;Rational business people don't pay for accuracy, they pay to either &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;save money on a profitable process (thereby making it more profitable), or by&lt;/li&gt;&#xA;&lt;li&gt;creating new money (creating new profitable processes). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So any project that is undertaken has to be couched in terms that&#xA;   reflect this. The first step is always understanding which of the two&#xA;   processes you are working on, and you should have a clear idea of how&#xA;   that might be achieved, while keeping in mind that as you make&#xA;   progress the details of how you do that might change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can improve the accuracy of a process, you can probably make money for the firm and the business people will invest in your progress. The only rational reason a business person could have for insisting on 99 percent accuracy and reject 90 percent is if they already had a way of doing it that was better than the 90 percent.  If that is the case they are of course justified in their position.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Understanding and presenting the business case of the projects you are working on in terms that the Business People understand is part of the maturing process of any Engineer. It is not unique to Data Science at all, though Data Science has some unique aspects (like lower maturity but higher probability of fortuous serendipity discovery - at least in today's environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A relevant process that is close to Data Science that makes this step explicit can be found here:&lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining&quot;&gt;https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But most &lt;em&gt;Enterprise Architecture Frameworks&lt;/em&gt; are similarly applicable. &lt;/p&gt;&#xA;" OwnerUserId="7720" LastActivityDate="2015-06-16T13:42:47.060" CommentCount="2" CommunityOwnedDate="2015-06-16T13:42:47.060" ContentLicense="CC BY-SA 3.0" />
  <row Id="6133" PostTypeId="1" CreationDate="2015-06-17T09:45:49.593" Score="3" ViewCount="2576" Body="&lt;p&gt;I am working on a rare event (unbalanced target variable) classification problem using decision trees. My dataset comprises of 95% non-event and 5% minority (events) class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used decision tree over logistic regression because I had many categorical variables comparing to continuous variables. I get a  good performance for training data with the decision tree C5.0. However I get poor results for the new data. I use the confusion matrix as a measure of performance. Training model is over-fitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did pruning to reduce the over-fitting caused by the decision tree.  I used the following code to build the model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Classifi_C5.0 &amp;lt;- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,&#xA;                      control = C5.0Control(minCases = mincases_count,&#xA;                                            noGlobalPruning = FALSE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I balanced the minority and majority class using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_data_SMOTED &amp;lt;- SMOTE(TARGET ~ ., training_data,&#xA;                              perc.over = 100, k = 5, perc.under = 200)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any sort of advice will be helpful. &lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="9793" LastEditDate="2015-06-17T11:12:27.923" LastActivityDate="2015-08-21T07:33:02.843" Title="Over-fitting issue in a classification problem (unbalanced data)" Tags="|machine-learning|data-mining|classification|r|decision-trees|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="6134" PostTypeId="2" ParentId="6070" CreationDate="2015-06-17T11:25:19.117" Score="0" Body="&lt;p&gt;Why would you need a distance when you have the pdf? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series can be assigned to clusters based on their fit to the cluster's pdf. This also means the method works for ragged timeseries, for which distance methods would break down.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-06-17T11:25:19.117" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6137" PostTypeId="1" CreationDate="2015-06-17T19:36:26.493" Score="4" ViewCount="204" Body="&lt;p&gt;I have several samples (C2, C4, C5) and want to check if they are at a certain stage. I included some known samples (D0 - D77) which were generated at different stages by another lab. In the PCA plot, my samples cluster together on the left and the known samples are dispersed on the right. I think the major difference among all the samples is different experimental protocols (PC1) and the second is different stages (PC2). So my samples are at the same stage. Is that right? And can we say my samples are at a stage between D12 and D19 (when projected to known samples, my samples are located between D12 and D19)? I have no strong mathematical background. Hope someone with math background can give some explanation. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ &lt;strong&gt;UPDATE1&lt;/strong&gt; ]&#xA;I did this analysis using &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/prcomp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;prcomp&lt;/a&gt; in R. The input is a 2D numeric matrix with 17436 rows and 42 columns. Each row represents a gene and each column represents a sample. The number is the gene expression level for a gene in a sample. The gene expression level is normalized using DESeq2 and thus the numbers are comparable across genes (rows) and samples (columns). For the 42 columns, 18 are from my experiments and the rest from published datasets. Besides different protocols, it is possible there are other differences. Generally, the table is a combination of two sources of data. In my data , C2, C4 and C5 are three cell lines which we processed in parallel in experiments. In published datasets, they sampled the cells at different time points (Day 0 to Day 77).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ &lt;strong&gt;UPDATE2&lt;/strong&gt; ] R Code for PCA and plotting&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# normCounts: normalized count from DESeq2, with 17436 rows and 42 columns&#xA;normCounts0 &amp;lt;- normCounts[ rowSums(normCounts) &amp;gt; 0, ]&#xA;tab &amp;lt;- t(normCounts0)&#xA;pca &amp;lt;- prcomp(tab, scale = TRUE)&#xA;tmp.x &amp;lt;- as.data.frame(pca$x)&#xA;tmp.x$sample&amp;lt;-c(rep(&quot;C2_Con&quot;,3),rep(&quot;C2_KD&quot;,3),rep(&quot;C5_Con&quot;,3),rep(&quot;C5_KD&quot;,3),rep(&quot;C4_Con&quot;,3),rep(&quot;C4_KD&quot;,3), rep(&quot;D0&quot;,4),rep(&quot;D7&quot;,4),rep(&quot;D12&quot;,2),rep(&quot;D19&quot;,4),rep(&quot;D26&quot;,2),rep(&quot;D33&quot;,2),rep(&quot;D49&quot;,2),rep(&quot;D63&quot;,2),rep(&quot;D77&quot;,2))&#xA;&#xA;require(&quot;ggplot2&quot;)&#xA;p &amp;lt;- ggplot(tmp.x, aes(x=PC1, y=PC2, color=sample))&#xA;p + geom_point() + scale_color_discrete(breaks=c(&quot;C2_Con&quot;,&quot;C2_KD&quot;,&quot;C5_Con&quot;,&quot;C5_KD&quot;,&quot;C4_Con&quot;,&quot;C4_KD&quot;, &quot;D0&quot;,&quot;D7&quot;, &quot;D12&quot;,&quot;D19&quot;,&quot;D26&quot;,&quot;D33&quot;,&quot;D49&quot;,&quot;D63&quot;,&quot;D77&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/LuU0y.jpg&quot; alt=&quot;PCA plot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="10201" LastEditorUserId="10201" LastEditDate="2015-06-19T02:28:39.227" LastActivityDate="2015-06-21T10:57:07.127" Title="Is it ok to interprete PCA plot this way?" Tags="|machine-learning|dimensionality-reduction|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6139" PostTypeId="1" AcceptedAnswerId="6366" CreationDate="2015-06-18T07:09:08.140" Score="1" ViewCount="145" Body="&lt;p&gt;Detailed Question Explanation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose say our application X is processing huge logs (size varying from MBs to GBs) and giving insight results in these logs(NOT A Social Data logs or Security Logs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;now this logs are in format say log.y with different variety, using C++ as Engine to process these huge logs.(It generates imp. insights about data but need to be processed using our application X only and we don't want to change core way processing of application X)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this processing happens on some server it under or over utilizes resources (That I already know).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we use cloud computing for this processing we get that processing power with optimum usage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do we see help of BIG data analytics in this particular sort of usage?&#xA;Any help or suggestion is very deeply appreciated  &lt;/p&gt;&#xA;" OwnerUserId="10210" LastEditorUserId="924" LastEditDate="2015-06-21T09:37:21.210" LastActivityDate="2015-07-07T01:15:04.840" Title="Can we use HDFS and big data Analytics for processing huge log files being processed through some application on some central server?" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6140" PostTypeId="1" CreationDate="2015-06-18T09:04:22.453" Score="2" ViewCount="2533" Body="&lt;p&gt;I have a csv file with characters in Persian and I cannot view them in R correctly. Also, I cannot subset based on Persian characters values. Here is a sample code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;list1 &amp;lt;- c(&quot;x&quot;,&quot;y&quot;)&#xA;list2 &amp;lt;- c(&quot;ب&quot;,&quot;الف&quot;)&#xA;&#xA;list1 &#xA;list2 ##OK-readable&#xA;&#xA;writedf &amp;lt;- as.data.frame(cbind(list1,list2),encoding=&quot;UTF-8&quot;)&#xA;write.csv(writedf,&quot;test.csv&quot;)&#xA;testdf &amp;lt;- read.csv(&quot;test.csv&quot;,encoding=&quot;UTF-8&quot;)&#xA;&#xA;testdf  &#xA;## not readable&#xA;&#xA;testdf[,testdf$list2==&quot;ب&quot;]&#xA;## data frame with 0 columns and 2 rows  ???&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-06-22T13:00:17.397" Title="Reading Persian Characters in R" Tags="|r|dataset|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6141" PostTypeId="1" AcceptedAnswerId="6145" CreationDate="2015-06-18T10:43:22.037" Score="2" ViewCount="116" Body="&lt;p&gt;I have an excel file containing a long text in column A. I am looking for the words starting by &quot;popul&quot; such as popular and populate . I can find these cells by the formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    =SEARCH(&quot;popul&quot;,A1,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want a function that returns the whole words starting by popul such as popular and populate.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-12-03T15:28:56.900" LastActivityDate="2015-12-03T15:28:56.900" Title="Working with text files in Excel" Tags="|dataset|excel|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6142" PostTypeId="2" ParentId="6137" CreationDate="2015-06-18T12:41:01.350" Score="2" Body="&lt;p&gt;All what I'd say is that you are &lt;strong&gt;most probably&lt;/strong&gt; right! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your plot is compatible to the explanation you have for your data so most probably that is the case. I assume you know what features to use (see example) but still better to post some info about input data -e.g. the dimensionality of the data- to be sure everything is right there. The only reason which my ruin the explanation is presence of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Confounding&quot; rel=&quot;nofollow&quot;&gt;confounder&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Example&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm not an expert of your field but just imagine there is a difference between materials you and the other lab used to get the data. Then I'll ask you how would you know that the difference on PC1 is not  according to the difference of materials but difference of protocoles? you can &lt;strong&gt;confidently&lt;/strong&gt; say that PC1 is capturing the difference of protocols if you are sure there is nothing else affecting it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helped!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-06-18T12:41:01.350" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6143" PostTypeId="2" ParentId="6098" CreationDate="2015-06-18T15:47:18.920" Score="1" Body="&lt;p&gt;First, $x_{ij}$ is not a bernoulli random variable. In the text, $x_{ij}$ is described as real valued. Also, the dimensions of $z$ and $w$ are incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The text indicates that we're drawing a matrix of data call it $X (n\times d)$ such that $X_{i,j}=x_{ij}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A particular entry has distribution $x_{ij} \sim f(\alpha_{z_i, w_j})$. Where $\alpha$ is some given matrix of parameters. The maximum row index of $\alpha$ is $g$. The maximum column index of $\alpha$ is $m$. So $\alpha$ is $g \times m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that remains is to pin down the indices $z$ and $w$. These are vectors rather than matrices since they have a categorical distribution, for example $z_i$ should be an integer between $1$ and $g$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting this together we need to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Draw $z$, a vector of length $n$, by $z_i \sim \text{Cat}(\pi), i=1,...,n$&lt;/li&gt;&#xA;&lt;li&gt;Draw $w$, a vector of length $d$, by $w_j \sim \text{Cat}(\rho), i=1,...,d$&lt;/li&gt;&#xA;&lt;li&gt;Draw $X$, an $(n\times d)$ matrix, by $x_{ij} \sim f(\alpha_{z_i, w_j})$ &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-06-18T15:47:18.920" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6144" PostTypeId="1" AcceptedAnswerId="6524" CreationDate="2015-06-18T16:41:43.320" Score="7" ViewCount="429" Body="&lt;p&gt;I'm using the HMMLearn python package for hidden markov models.  That implementation is build on multivariate gaussian distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a string of features.  How sensitive are gaussians to vastly different feature scales?  Will it be really skewed if one feature is scaled between 0 and 1, and another is scaled between 0 and 1e8?&lt;/p&gt;&#xA;" OwnerUserId="10187" LastActivityDate="2015-07-21T03:40:23.157" Title="Sensitivity to scaling of features in a multivariate gaussians" Tags="|python|statistics|feature-scaling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6145" PostTypeId="2" ParentId="6141" CreationDate="2015-06-18T17:38:19.883" Score="5" Body="&lt;p&gt;I'm no Excel expert, as I generally use Python or R instead, but this might get you started until an Excel expert comes along.  In the meantime, it would help if you clarified your question.  And you should be aware that &lt;code&gt;search&lt;/code&gt; will only find you the index of the &lt;strong&gt;first&lt;/strong&gt; match, not all matches in the string.  If you only need the first hit, you can use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=MID(A1,SEARCH(&quot;popul&quot;,A1,1),IFERROR(FIND(&quot; &quot;,A1,SEARCH(&quot;popul&quot;,A1,1)),LEN(A1)+1)-SEARCH(&quot;popul&quot;,A1,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;although I cannot claim this is the &lt;em&gt;best&lt;/em&gt; way to do this.  You really didn't specify where you want the results to appear, how they should look, or if you only have one cell you need to search in.  It would also help to know the version of Excel you have.  I'll also present a crude way to return all the hits in the string:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/MrYrb.png&quot; alt=&quot;cells&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cell A1 contains the string, B1 has no formula, and if you run out of &quot;n/a&quot;s you can extend columns B, C, and D by filling down.  The formulas are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;B3 and below use&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=IF(C2+1&amp;lt;LEN($A$1),C2+1,&quot;n/a&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;C2 and below use&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=IFERROR(FIND(&quot; &quot;,$A$1,SEARCH(&quot;popul&quot;,$A$1,B2)),LEN($A$1)+1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;D2 and below use&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=IFERROR(MID($A$1,SEARCH(&quot;popul&quot;,$A$1,B2),C2-SEARCH(&quot;popul&quot;,$A$1,B2)),&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, there's little to no error checking except to deal with the match at the end of the string.  In the end though, if you're going to use Excel for this you should probably create a user defined function or utilize VBA instead of in-cell formulas.&lt;/p&gt;&#xA;" OwnerUserId="8953" LastActivityDate="2015-06-18T17:38:19.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6146" PostTypeId="1" CreationDate="2015-06-18T18:23:07.763" Score="8" ViewCount="842" Body="&lt;p&gt;I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R.  have difficulty storing tweets because my machine has only 8 GB of memory. It ran out of memory even before I set it to retrieve for one day. Is there a way where I can store the tweets straight to my disk without storing into RAM? I am not using the streaming API as I need to get old tweets. &lt;/p&gt;&#xA;" OwnerUserId="10225" LastEditorUserId="434" LastEditDate="2015-06-18T23:29:51.727" LastActivityDate="2015-06-18T23:29:51.727" Title="Best way to store large data set using R from Twitter?" Tags="|r|dataset|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6147" PostTypeId="2" ParentId="6146" CreationDate="2015-06-18T18:57:08.203" Score="2" Body="&lt;p&gt;I worked on a Twitter data project last Fall wherein we used Java libraries to pull in tweet data from the streaming and the rest API's. We used &lt;a href=&quot;http://twitter4j.org/en/index.html&quot; rel=&quot;nofollow&quot;&gt;Twitter4J&lt;/a&gt; (an unofficial Java library) for the  &lt;a href=&quot;https://dev.twitter.com/overview/documentation&quot; rel=&quot;nofollow&quot;&gt;Twitter API&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tweet data was fetched and directly written onto text files on our hard drives. Yes, we did increase the memory and heap. I believe R studio will have a similar option. An alternative would be to pull in lesser amounts of tweet data with more number of repetitions. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2015-06-18T18:57:08.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6148" PostTypeId="2" ParentId="6146" CreationDate="2015-06-18T20:36:47.597" Score="5" Body="&lt;p&gt;Find a way to make your program write to disk periodically. Keep count of the number of tweets you grab and save after that number is high. I don't write R but psuedocode might look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$tweets = get_tweets();&#xA;$count = 0;&#xA;$tweet_array = array();&#xA;for each ($tweets as $tweet) {&#xA;  $tweet_array += $tweet;&#xA;  $count++;&#xA;  if ($count &amp;gt; 10000) {&#xA;    append_to_file($tweet_array, 'file_name.txt');&#xA;    clear_array($tweet_array);&#xA;  }&#xA;}&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-06-18T20:36:47.597" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6149" PostTypeId="2" ParentId="6140" CreationDate="2015-06-18T21:50:08.290" Score="0" Body="&lt;p&gt;Use &lt;code&gt;subset&lt;/code&gt; instead i.e. &lt;code&gt;subset(testdf, testdf$list2==&quot;ب&quot;)&lt;/code&gt; results in what you want.  &lt;/p&gt;&#xA;" OwnerUserId="5055" LastActivityDate="2015-06-18T21:50:08.290" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6150" PostTypeId="1" CreationDate="2015-06-19T05:41:26.783" Score="1" ViewCount="5958" Body="&lt;p&gt;I am really new to data science. Please don't mark me down as this website is my only hope of progress.&#xA;I have set of data I obtained from NASA website. When I saved it, it saved as &quot;tsv' file. (Tab separated values). I want to open it on Matlab as a Matrix as I have a code to run on that matrix. &#xA;Basically I want to import that file to matlab and start running the code on it. &#xA;Can someone please help me or guide me in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried various things such as import data, tdfread but so far nothing has worked for me. I was first trying to export the tsv file to MS Excel and then go from Excel to Matlab. That too I don't know how to do. I will give you the link of my data which I want to import on to Matlab. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link for my data is the following.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please take a look at &lt;a href=&quot;http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J%2FApJS%2F209%2F31&quot; rel=&quot;nofollow&quot;&gt;http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J%2FApJS%2F209%2F31&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just click on submit at the lower right corner, one will see the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a = importdata('J_ApJS_209_31_table3-150618.tsv') [This the command I used].&#xA;Error message on matlab is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    ??? Error using ==&amp;gt; importdata at 136&#xA;    Unable to open file.&#xA;    Error in ==&amp;gt; data at 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I wrote a script.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    tdfread(J_ApJS_209_31_table3-150618,'\t')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error message I get is &#xA;        ??? Undefined function or variable 'J_ApJS_209_31_table3'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Error in ==&amp;gt; data at 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9039" LastEditorUserId="9039" LastEditDate="2015-06-24T17:41:26.543" LastActivityDate="2015-06-24T20:21:01.833" Title="How to import a tsv file to Matlab" Tags="|bigdata|matlab|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6151" PostTypeId="1" CreationDate="2015-06-19T07:12:07.153" Score="4" ViewCount="1303" Body="&lt;p&gt;What are the best method/library/data available to extract named entities [Names and Location] from Twitter data ? [Other than dictionary lookup]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried with Python-Stanford NER, But it seems to fail when named entities is not capitalized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried to predict NER after converting text to upper case &#xA;eg : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; text = &quot;david beckham played for england&quot;&#xA;&#xA; stanford.NERTagger.tag(text)&#xA; [(u'david', u'PERSON'), (u'beckham', u'PERSON'), (u'played', u'O'), (u'for', u'O'), (u'england', u'O')]&#xA;&#xA; stanford.NERTagger.tag(text.upper())&#xA; output : [(u'DAVID', u'PERSON'), (u'BECKHAM', u'PERSON'), (u'PLAYED', u'O'), (u'FOR', u'O'), (u'ENGLAND', u'LOCATION')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-06-19T10:11:29.680" LastActivityDate="2015-06-20T06:12:08.537" Title="NER on Twitter data" Tags="|python|named-entity-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6152" PostTypeId="2" ParentId="6080" CreationDate="2015-06-19T07:30:04.703" Score="1" Body="&lt;p&gt;Conditional Random Fields (CRFs) can be used for segmenting/labeling sequential problems. Try CRF++: Yet Another CRF toolkit,  a simple, customizable, and open source implementation of Conditional Random Fields (CRFs) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can label and create a tagged training corpus and use CRF++ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need to create a feature template&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer : &lt;a href=&quot;http://taku910.github.io/crfpp/&quot; rel=&quot;nofollow&quot;&gt;http://taku910.github.io/crfpp/&lt;/a&gt; for more details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check the example from data for CoNLL shared task (PoS tagging).&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-06-19T07:30:04.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6153" PostTypeId="1" CreationDate="2015-06-19T07:37:44.407" Score="2" ViewCount="36" Body="&lt;p&gt;A question crossed my mind not so long ago: I am doing experiments on Language Model with RNN (always with the same network topology: 50 hidden units, and 10M &amp;quot;directs connections&amp;quot; that are emulating N_grams models) and different fraction of corpus (10,25,50,75,100%) (9M words).&lt;/p&gt;&#xA;&lt;p&gt;I noticed that while perplexity seems to decrease when the training data become more abundant, certain times it does not.&lt;/p&gt;&#xA;&lt;p&gt;Last example : 143 118 109 106 112&lt;/p&gt;&#xA;&lt;p&gt;My first thought was network initialization, so I began testing with a smaller corpus and 20 hidden units (for technical reasons. Even with 10% corpus, learning can take up to 30h, which is problematic for me), and I found after 50 tries that all nets converged on values within 3% of each other.&lt;/p&gt;&#xA;&lt;p&gt;But, I thought that maybe the importance of this initialization is a function of the number of hidden units? I mean the more hidden units the more parameters to tune.&lt;/p&gt;&#xA;&lt;p&gt;Also, maybe my stop criterion is too sensitive (It stops if evolution of perplexity between two iterations is inferior to a certain number).&#xA;Do you think it would make an impact to allow it to run one of two iterations after the criterion was met to see if it was just a local thing?&lt;/p&gt;&#xA;" OwnerUserId="10232" LastEditorUserId="1330" LastEditDate="2022-04-18T18:14:31.213" LastActivityDate="2022-04-18T18:14:31.213" Title="Importance of Random initialisation VS number of hidden units" Tags="|neural-network|rnn|language-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6154" PostTypeId="2" ParentId="6107" CreationDate="2015-06-19T10:17:17.027" Score="30" Body="&lt;p&gt;The &lt;a href=&quot;https://cs231n.github.io/&quot; rel=&quot;noreferrer&quot;&gt;notes that accompany Stanford CS class CS231n&lt;/a&gt;: &lt;em&gt;Convolutional Neural Networks for Visual Recognition, by Andrej Karpathy&lt;/em&gt;, do an excellent job of explaining convolutional neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reading this paper should give you a rough idea about:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;&lt;a href=&quot;http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deconvolutional Networks&lt;/a&gt;&#xA;Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor and Rob Fergus&#xA;Dept. of Computer Science, Courant Institute, New York University&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These &lt;a href=&quot;http://cs.nyu.edu/~fergus/drafts/utexas2.pdf&quot; rel=&quot;noreferrer&quot;&gt;slides&lt;/a&gt; are great for Deconvolutional Networks.&lt;/p&gt;&#xA;" OwnerUserId="9943" LastEditorUserId="29575" LastEditDate="2018-06-21T13:21:33.533" LastActivityDate="2018-06-21T13:21:33.533" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="6156" PostTypeId="2" ParentId="6140" CreationDate="2015-06-19T15:14:58.097" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;make sure the files are saved in UTF&lt;/li&gt;&#xA;&lt;li&gt;try &lt;code&gt;Sys.setlocale(&quot;LC_ALL&quot;, locale_code)&lt;/code&gt; and have a look at the &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/locales.html&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; of this function &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastEditorUserId="816" LastEditDate="2015-06-19T17:44:42.983" LastActivityDate="2015-06-19T17:44:42.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6157" PostTypeId="2" ParentId="6151" CreationDate="2015-06-19T15:28:51.003" Score="2" Body="&lt;p&gt;Yes it is a challenging task to extract named entities in tweets. Give a go at NLTK NER and also Alan Ritter's Twitter specific NER and evaluate on their performance and compare to Stanford NER and which one fits in your use. Maybe you want to use more than one to get more named entities if you don't mind so much of false NEs..&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2015-06-19T15:28:51.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6158" PostTypeId="2" ParentId="6137" CreationDate="2015-06-19T16:39:48.010" Score="4" Body="&lt;p&gt;Given the information provided, I do not think it's enough to say that your samples are between stages D12 and D19. I admit knowing nothing about your domain, but with what you're showing it is very conceivable that other factors are causing the spread of your samples from the others on PC1 &lt;em&gt;and&lt;/em&gt; it's too much of a stretch to say that your samples &quot;project&quot; into the others in between D12 and D19. Why does it have to be a projection orthogonal to the axis described by PC1? Remeber that PC1 is a linear combination of all your inputs. That is, it describes an arbitrary line through your input space that is probably not aligned with any one thing in particular. For example, all three of the following projections could be envisioned for your problem:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/se2FV.png&quot; alt=&quot;PCA plot showing possible projections&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another red flag for me is that you say PC1 is related to differences in experimental protocols. You haven't posted a scree plot or listed the eigenvalues for the different PCs but obviously PC1 soaks up the most variance so it's a pretty big deal. If you're saying that PC1 is correlated with experimental protocol, then that would seem good because hopefully the impact of experimental protocol is minimized in other PCs. However, that's never completely the case, really.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, not knowing your domain it's hard to tell what the impact of experimental protocol is on the distribution of your data in the PC plot, but I'd be VERY wary of comparing these data in this way if your data was produced using a different experimental design and, as you noted, &quot;there may be other differences.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My hunch is that you would need to continue minimizing differences in the protocols between your experiments and the published experiments before you could really use this type of analysis. Others might disagree but if you posted the sources of your data, the eigenvectors, and possibly a subset of the data, experts in your field may be able to chime in on this methodology. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short answer is: you &lt;em&gt;could&lt;/em&gt; be right in your interpretation but at this point you don't have enough evidence to suggest your samples are &quot;in between&quot; D12 and D19 in terms of the stage. And I worry about differences in methodology completely swamping your ability to interpret things this way. &lt;/p&gt;&#xA;" OwnerUserId="9956" LastActivityDate="2015-06-19T16:39:48.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6159" PostTypeId="2" ParentId="6150" CreationDate="2015-06-19T16:46:00.387" Score="1" Body="&lt;p&gt;&lt;code&gt;tdfread&lt;/code&gt; displays the File Open dialog box for interactive selection of a data file, then reads data from the file. The file should have variable names separated by tabs in the first row, and data values separated by tabs in the remaining rows. &lt;code&gt;tdfread&lt;/code&gt; creates variables in the workspace, one for each column of the file. The variable names are taken from the first row of the file. If a column of the file contains only numeric data in the second and following rows, tdfread creates a double variable. Otherwise, tdfread creates a char variable. After all values are imported, &lt;code&gt;tdfread&lt;/code&gt; displays information about the imported values using the format of the &lt;code&gt;tdfread&lt;/code&gt; command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tdfread(filename)&lt;/code&gt; allows command line specification of the name of a file in the current folder, or the complete path name of any file, using the string filename.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tdfread(filename,delimiter)&lt;/code&gt; indicates that the character specified by delimiter separates columns in the file. Accepted values for delimiter are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;' ' or 'space'&#xA;'\t' or 'tab'&#xA;',' or 'comma'&#xA;';' or 'semi'&#xA;'|' or 'bar'&#xA;The default delimiter is 'tab'.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;s = tdfread(filename,...)&lt;/code&gt; returns a scalar structure s whose fields each contain a variable.&lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorDisplayName="user8022" LastEditDate="2015-06-24T20:21:01.833" LastActivityDate="2015-06-24T20:21:01.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6162" PostTypeId="2" ParentId="6151" CreationDate="2015-06-20T06:12:08.537" Score="1" Body="&lt;p&gt;I think you are in a better position to train your own NER model. You can start with CRFSuite as a package.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastActivityDate="2015-06-20T06:12:08.537" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6165" PostTypeId="1" AcceptedAnswerId="6166" CreationDate="2015-06-21T08:22:39.630" Score="11" ViewCount="1535" Body="&lt;h3&gt;Problem&lt;/h3&gt;&#xA;&lt;p&gt;I have tried using Naive bayes on a labeled data set of crime data but got really poor results (7% accuracy). Naive Bayes runs much faster than other alogorithms I've been using so I wanted to try finding out why the score was so low.&lt;/p&gt;&#xA;&lt;h3&gt;Research&lt;/h3&gt;&#xA;&lt;p&gt;After reading I found that Naive bayes should be used with balanced datasets because it has a bias for classes with higher frequency. Since my data is unbalanced I wanted to try using the Complementary Naive Bayes since it is specifically made for dealing with data skews. In the paper that describes the process, the application is for text classification but I don't see why the technique wouldn't work in other situations. You can find the paper I'm referring to &lt;a href=&quot;http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. In short the idea is to use weights based on the occurences where a class doesn't show up.&lt;/p&gt;&#xA;&lt;p&gt;After doing some research I was able to find an implementation in Java but unfortunately I don't know any Java and I just don't understand the algorithm well enough to implement myself.&lt;/p&gt;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&lt;p&gt;where I can find an implementation in python? If that doesn't exist how should I go about implementing it myself?&lt;/p&gt;&#xA;" OwnerUserId="10255" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2023-08-31T11:49:37.587" Title="Implementing Complementary Naive Bayes in python?" Tags="|machine-learning|classification|python|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6166" PostTypeId="2" ParentId="6165" CreationDate="2015-06-21T09:03:26.883" Score="5" Body="&lt;p&gt;Naive Bayes should be able to handle imbalanced datasets. Recall that the Bayes formula is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(y \mid x) = \cfrac{P(x \mid y) \, P(y)}{P(x)} \propto P(x \mid y) \, P(y)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $P(x \mid y) \, P(y)$ takes the prior $P(y)$ into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case maybe you overfit and need some smoothing? You can start with +1 smoothing and see if it gives any improvements. In python, when using numpy, I'd implement the smoothing this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;table = # counts for each feature &#xA;PT = (table + 1) / (table + 1).sum(axis=1, keepdims=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this is gives you Multinomial Naive Bayes - which applies only to categorical data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can also suggest the following link: &lt;a href=&quot;http://www.itshared.org/2015/03/naive-bayes-on-apache-flink.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.itshared.org/2015/03/naive-bayes-on-apache-flink.html&lt;/a&gt;. It's about implementing Naive Bayes on Apache Flink. While it's Java, maybe it'll give you some theory you need to understand the algorithm better.&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2015-06-21T09:03:26.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6167" PostTypeId="2" ParentId="6137" CreationDate="2015-06-21T10:57:07.127" Score="3" Body="&lt;h3&gt;Don't just mix the data of two different sources.&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It looks as if the two sources may simply be using different scales. But that makes the whole PCA analysis meaningless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All your plot may be visualizing is that the other samples use a different way of measuring things. It seems &lt;strong&gt;all the variance is in the other samples&lt;/strong&gt;. In particular this means that &lt;strong&gt;you cannot draw any conclusions about how your samples relate to the reference samples&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-06-21T10:57:07.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6168" PostTypeId="2" ParentId="5371" CreationDate="2015-06-21T17:11:39.947" Score="3" Body="&lt;p&gt;It depends on the usage of various packages (Numpy/Scipy etc), which are written in C and be incredibly fast also Python can be complied using JIT.  Here is an excellent comparison between R and Python : &lt;a href=&quot;https://learnanalyticshere.wordpress.com/2015/05/14/clash-of-the-titans-r-vs-python/&quot; rel=&quot;nofollow&quot;&gt;https://learnanalyticshere.wordpress.com/2015/05/14/clash-of-the-titans-r-vs-python/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-06-21T17:11:39.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6169" PostTypeId="1" AcceptedAnswerId="6263" CreationDate="2015-06-21T17:31:05.457" Score="9" ViewCount="8960" Body="&lt;p&gt;I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] as an environment variable and called &lt;strong&gt;spark-submit&lt;/strong&gt; command in the cmd prompt. I believe that it is finding the spark context, but it produces a really big error. Can someone please help me with this issue? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Environment variable path: C:/Users/Name/Spark-1.4;C:/Users/Name/Spark-1.4/bin&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, in cmd prompt: spark-submit script.py&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Run3y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="1330" LastEditDate="2020-06-12T19:43:07.100" LastActivityDate="2020-08-03T13:33:58.107" Title="How to run a pyspark application in windows 8 command prompt" Tags="|python|apache-spark|pyspark|ipython|windows|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6170" PostTypeId="2" ParentId="5345" CreationDate="2015-06-21T18:52:37.563" Score="20" Body="&lt;p&gt;&lt;strong&gt;IntelliJ&lt;/strong&gt; and &lt;strong&gt;PyCharm&lt;/strong&gt; supports R via this &lt;a href=&quot;https://plugins.jetbrains.com/plugin/6632&quot; rel=&quot;nofollow noreferrer&quot;&gt;plugin&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It's a recent project, so RStudio is still more powerful, including its focus on a data-friendly environment (plots and data are always in sight).&lt;/p&gt;&#xA;" OwnerUserId="5279" LastEditorUserId="85045" LastEditDate="2022-01-18T04:57:26.037" LastActivityDate="2022-01-18T04:57:26.037" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="6172" PostTypeId="1" AcceptedAnswerId="6177" CreationDate="2015-06-22T02:13:40.850" Score="4" ViewCount="188" Body="&lt;p&gt;It is well known that science has given us large amounts of free accessible data, such as &lt;a href=&quot;https://www.1000genomes.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.1000genomes.org&lt;/a&gt; and &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/genbank&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.ncbi.nlm.nih.gov/genbank&lt;/a&gt;. How can we play around with the data and apply data science/machine learning to it? What could be some ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own ideas:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Biological data visualisation&lt;/li&gt;&#xA;&lt;li&gt;Gene prediction using hidden-markov-model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any more?&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="13808" LastEditDate="2020-04-16T05:39:09.600" LastActivityDate="2020-04-16T07:36:46.257" Title="What kind of research can be done with genomic data?" Tags="|machine-learning|classification|bioinformatics|data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6173" PostTypeId="2" ParentId="6140" CreationDate="2015-06-22T12:00:29.197" Score="1" Body="&lt;p&gt;First of all I should mention that this is an OS specific question. You will see this problem in Windows but not in Linux (I'm not sure about OS X, but maybe no problem there too.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with your code is not with the reading part.&#xA;If you open your &lt;code&gt;test.csv&lt;/code&gt; in Notepad++ you will see something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/IhqXP.png&quot; alt=&quot;file written from R&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you edit it in Notepad++ so that it looks like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/909pp.png&quot; alt=&quot;file edited in Notepad++&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then you run this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testdf &amp;lt;- read.csv(&quot;test.csv&quot;,encoding=&quot;UTF-8&quot;)&#xA;testdf[1,3]&#xA;testdf$list2&#xA;plot(1:10,main=testdf2[2,3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You will see that there is no problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in fact there are two problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;When writing Unicode text, R changes it back to system locale and then again to unicode, that's why writing data to csv file does not work correctly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R has problems showing Unicode data in &lt;code&gt;data.frame&lt;/code&gt; so even after reading it correctly, &lt;code&gt;testdf&lt;/code&gt; would print those codes there not the Unicode characters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;see this question on &lt;a href=&quot;http://r.789695.n4.nabble.com/Writing-Unicode-Text-into-Text-File-from-R-in-Windows-td4684693.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Writing Unicode Text into Text File from R (in Windows)&lt;/a&gt; in R mailing list archives,&#xA;and also this question on &lt;a href=&quot;https://stackoverflow.com/questions/10675360/utf-8-file-output-in-r&quot;&gt;UTF-8 file output in R&lt;/a&gt; on stackoverflow.&lt;/p&gt;&#xA;" OwnerUserId="10270" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-06-22T13:00:17.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6174" PostTypeId="1" CreationDate="2015-06-22T13:22:57.950" Score="1" ViewCount="7933" Body="&lt;p&gt;I am working on Rstudio on a server which has 250GB ram. But its taking too much time to handle a 2GB data file. how should i speed up my work?&lt;/p&gt;&#xA;" OwnerUserId="10272" LastActivityDate="2015-06-22T20:44:59.480" Title="Rstudio using 2.5% of 250GB RAM. how to Increase it" Tags="|r|rstudio|" AnswerCount="1" CommentCount="1" ClosedDate="2015-06-23T09:34:43.147" ContentLicense="CC BY-SA 3.0" />
  <row Id="6175" PostTypeId="1" CreationDate="2015-06-22T14:41:12.013" Score="2" ViewCount="115" Body="&lt;p&gt;I have a 7 giga confidential dataset which I want to use for a machine learning application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every package recommanded for efficient dataset management in R like :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data.table, &lt;/li&gt;&#xA;&lt;li&gt;ff &lt;/li&gt;&#xA;&lt;li&gt;and sqldf with no success. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Data.table needs to load all the data in the memory from what I read, so it's obvious that it will not work since my computer has only 4g RAM. Ff leads to a memory error too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I decided to turn to sgdb and I tried :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mysql which managed to load my dataset in 2 hours and 21'. Then I began my requests (I have a few requests to do to prepare my data before I export a smaller set in R for machine learning application), and then I had to wait for hours before I got the following message &quot;The total number of locks exceeds the lock table size&quot; (my request was just an update to extract the month from a date for each tuple). &lt;/li&gt;&#xA;&lt;li&gt;I read that postgre was similar to mysql in performance so I didn't try&lt;/li&gt;&#xA;&lt;li&gt;I read that redis was really performant but not at all adapted to massive importation like I want to do here so I didn't try&lt;/li&gt;&#xA;&lt;li&gt;I tried mongoDb, the nosql upraising solution that I heard everywhere about. Not only I find rather disturbing that mongoimport is so limited in options (I had to change all semi-colon in commas using sed before I can import the data), but It seems to be less performant that mysql since I launched the loading yesterday and it is still running. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I can't try : data are confidential so I don't really want to rent some space on Azure or Amazon clouding solution. I am not sure that it is that big that I have to turn to Hadoop solution but maybe I am wrong about that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an open-source performant solution that I didn't try that you would recommend to perform some sql-like requests on a biggish dataset ?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit : &#xA;Some more details about what I want to do with these data for you to visualize. These are events with a timestamp and a geolocalisation. I have 8 billions of lines. One example of what I want to do : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;standardize series identified by geolocalisation (I need to compute mean grouping by geolocalisation for example), &lt;/li&gt;&#xA;&lt;li&gt;compute average count of events by type of season, day... (usual group by sql request)... &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a beginning of answer for those who have limited hardware like me, rSQLite seems to be a possibility. I am still interested in other people experiences. &lt;/p&gt;&#xA;" OwnerUserId="10275" LastEditorUserId="10275" LastEditDate="2015-06-24T16:23:56.347" LastActivityDate="2015-06-24T16:23:56.347" Title="Which open-source sgdb for kind of large data" Tags="|r|performance|mongodb|" AnswerCount="1" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="6176" PostTypeId="1" CreationDate="2015-06-22T15:26:36.693" Score="2" ViewCount="174" Body="&lt;p&gt;I've been coding a Neural Network for recognizing and tagging parts of speech in English (written in Java). The code itself has no 'errors' or apparent flaws. Nevertheless, it is not learning -- the more I train it does not change its ability to predict the testing data. The following is information about what I've done, please ask me to update this post if I left something important out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote the neural network and tested it on several different problems to make sure that the network itself worked. I trained it to learn how to double numbers, XOR, cube numbers, and learn the sin function to a decent accuracy. So, I'm fairly confident that the actual algorithm is working.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network using using the sigmoid activation function. The learning rate is .3, Momentum is .6. The weights are initialized to rng.nextFloat() -.5) * 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then got the Brown Corpus data-set and simplified the tagset to 'universal' with NLTK. I used NLTK for generating and saving all the corpus and dictionary data. I cut the last 15,000 sentences out of the corpus for testing purposes. I used the rest of the corpus (about 40,000 sentences of tagged words) for training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The neural network layout is as follows: There is an input neuron for each Tag. Output Layer: There is one output neuron for each tag. The network is taking inputs for 3 words: first: the word coming before the word we want to tag, second: the word that needs to be tagged, third: the word that follows the second word. So, total number of inputs are 3x(total number of possible tags). The input values are numbers between 0 and 1. Each of the 3 words being fed into the input layer is searched for in a dictionary (made up by the 40,000 corpus, the same corpus that is used for training). The dictionary holds the number of times that each word has been tagged in the corpus as what part of speech. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For instance, the word 'cover' is tagged as a noun 1 time and a verb 3&#xA;  times.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Percentages of being tagged are computed for each part of speech that the word is associated as, and this is what is fed into the network for that particular word. So, the input neuron designated as NOUN would receive .33 and VERB would receive .66. The other input neurons that hold tags for that word receive an input of 0.0. This is done for each of the 3 words to be inputted. If a word is the first word of a sentence, the first group of tags are all 0. If a word is the last word of a sentence, the final group of input neurons that hold the tag probabilities for the following word are left as 0s.&#xA;I've been using 10 hidden nodes (I've read a number of papers and this seems to be a good place to start testing with)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;None of the 15,000 testing sentences were used to make the 'dictionary.' So, when testing the network with this partial corpus there will be some words the network has never seen. Words that are not recognized have their suffix stripped, and their suffix is searched for in another 'dictionary.' Whatever is most probable for that word is then used as inputs for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my set-up, and I started trying to train the network. I've been training the network with all 40,000 sentences. 1 epoch = 1 forward and backpropagation of every word in each sentence of the 40,000 training-set. So, just doing 1 epoch takes quite a few seconds. Just by knowing the word probabilities the network did pretty well, but the more I train it, nothing happens. The numbers that follow the epochs are the number of correctly tagged words divided by the total number of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First run 50 epochs: 0.928218786&lt;/p&gt;&#xA;&#xA;&lt;p&gt;100 epochs:        0.933130661&lt;/p&gt;&#xA;&#xA;&lt;p&gt;500 epochs:       0.928614499 took around 30 minutes to train this                   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tried 10 epochs:         0.928953683 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using only 1 epoch had results that pretty much varied between .92 and .93&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it doesn't appear to be working...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then took 55 sentences from the corpus and used the same dictionary that had probabilities for all 40,000 words. For this one, I trained it in the same way I trained my XOR -- I only used those 55 sentences and I only tested the trained network weights on those 55 sentences. The network was able to learn those 55 sentences quite easily. With 120 epochs (taking a couple seconds) the network went from tagging 3768 incorrectly and 56 correctly (on the first few epochs) to tagging 3772 correctly and 52 incorrectly on the 120th epoch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where I'm at, I've been trying to debug this for over a day now, and haven't figured anything out.&lt;/p&gt;&#xA;" OwnerUserId="10265" LastActivityDate="2015-06-22T15:26:36.693" Title="Debugging Neural Network for (Natural Language) Tagging" Tags="|machine-learning|neural-network|knowledge-base|java|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6177" PostTypeId="2" ParentId="6172" CreationDate="2015-06-22T19:21:53.897" Score="4" Body="&lt;ul&gt;&#xA;&lt;li&gt;Determine the function of genes and the elements that regulate genes&#xA;throughout the genome.&lt;/li&gt;&#xA;&lt;li&gt;Find variations in the DNA sequence among people and determine their&#xA;significance. The most common type of genetic variation is known as a&#xA;single nucleotide polymorphism or SNP (pronounced “snip”).  These&#xA;small differences may help predict a person’s risk of particular&#xA;diseases and response to certain medications.&lt;/li&gt;&#xA;&lt;li&gt;Discover the 3-dimensional structures of proteins and identify their&#xA;functions.&lt;/li&gt;&#xA;&lt;li&gt;Explore how DNA and proteins interact with one another and with the&#xA;environment to create complex living systems.&lt;/li&gt;&#xA;&lt;li&gt;Develop and apply genome-based strategies for the early detection,&#xA;diagnosis, and treatment of disease.&lt;/li&gt;&#xA;&lt;li&gt;Sequence the genomes of other organisms, such as the rat, cow, and&#xA;chimpanzee, in order to compare similar genes between species.&lt;/li&gt;&#xA;&lt;li&gt;Develop new technologies to study genes and DNA on a large scale and&#xA;store genomic data efficiently.&lt;/li&gt;&#xA;&lt;li&gt;Continue to explore the ethical, legal, and social issues raised by&#xA;genomic research.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://ghr.nlm.nih.gov/handbook/genomicresearch?show=all&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source: https://ghr.nlm.nih.gov/handbook/genomicresearch?show=all&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="13808" LastEditDate="2020-04-16T07:36:46.257" LastActivityDate="2020-04-16T07:36:46.257" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6178" PostTypeId="2" ParentId="6174" CreationDate="2015-06-22T20:44:59.480" Score="3" Body="&lt;p&gt;You may or may not already know this, but here are a few basics about how R and Rstudio work and use resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rstudio is a &lt;em&gt;graphical user interface&lt;/em&gt; to R, not the interpeter/runtime environment.  There is a separate &quot;R session&quot; that actually executes your R programs and returns results for Rstudio to display.  Therefore, having &lt;em&gt;Rstudio&lt;/em&gt; use more memory won't make any difference in the execution speed of your program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, memory allocation and cleanup (a.k.a. &quot;garbage collection&quot;) is handled automatically by the R runtime environment.  &quot;&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/base/html/Memory-limits.html&quot; rel=&quot;nofollow&quot;&gt;R holds objects it is using in virtual memory&lt;/a&gt;&quot;. Virtual memory is a combination of physical main memory and secondary storage, the mixture determined by the server OS configuration.  You can use command line to change the amount of virtual memory allocated to processes. (Consult with your favorite Linux expert on this.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third, your speed of your program may or may not be limited by memory speed. You may be compute-bound.  Do some testing to find out what is constraining performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourth, you should first ask yourself whether your program implements an efficient algorithm.  Even if you aren't programming with loops and branches, the functions you call may use them, and maybe not efficiently for your application.  For example, there are dramatic performance gains to be had by switching to &lt;code&gt;data.table&lt;/code&gt; from &lt;code&gt;data.frame&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fifth, once you have chosen an efficient algorithm, you can put effort into parallel execution.  The simplest way to do this is by using functions that automatically vectorize operations.  A bit more complicated is to recode your program use the packages &lt;code&gt;doParallel&lt;/code&gt; and &lt;code&gt;foreach&lt;/code&gt;.  With &lt;code&gt;doParallel&lt;/code&gt;, you can specify the number of CPU cores to use, which on a server may range from 32 to 64 or more.  Finally, if your server has a Graphics Processing Unit (GPU), it's possible for some algorithms to reprogram using the GPU commands and get massive parallelism.  This option takes the most effort and has the most constraints.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-06-22T20:44:59.480" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6179" PostTypeId="2" ParentId="6175" CreationDate="2015-06-23T02:37:33.750" Score="1" Body="&lt;p&gt;analyzing 8 billion lines on a 4gb computer is pretty silly, but you can try&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.asdfree.com/2013/03/column-store-r-or-how-i-learned-to-stop.html&quot; rel=&quot;nofollow&quot;&gt;http://www.asdfree.com/2013/03/column-store-r-or-how-i-learned-to-stop.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9924" LastActivityDate="2015-06-23T02:37:33.750" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6181" PostTypeId="1" CreationDate="2015-06-23T08:36:13.723" Score="7" ViewCount="343" Body="&lt;p&gt;I'm trying to think of a data set that is essentially topologically spherical. It's easier to think of cylindrical datasets (two dimensions, one periodic) or toroidal datasets (two dimensions, both periodic). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obvious candidates are geographical and astronomical, ground and sky; but I think the only thing spherical about the sky is its projection onto the ground, so it really just comes back to Earth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find it helpful to think about in terms of a circle's &lt;a href=&quot;https://en.wikipedia.org/wiki/Fundamental_polygon#Examples&quot; rel=&quot;noreferrer&quot;&gt;fundamental polygon&lt;/a&gt;:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/vxubU.png&quot; alt=&quot;Fundamental polygons&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="10293" LastActivityDate="2020-05-17T15:16:25.623" Title="What kinds of data other than geographical are topologically spherical?" Tags="|dataset|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6182" PostTypeId="1" CreationDate="2015-06-23T09:20:27.740" Score="3" ViewCount="104" Body="&lt;p&gt;I am using rattle in R for predictive models and am trying to see whether there is a difference in different sampling methods. The split function at the start of rattle for splitting into training and testing (optional validation) I take it is split validation. Is there a way to do bootstrap validation and cross validation directly in rattle?&lt;/p&gt;&#xA;" OwnerUserId="9444" LastEditorUserId="843" LastEditDate="2017-03-08T15:21:35.483" LastActivityDate="2017-03-08T15:21:35.483" Title="Is there an interface for bootstrap sampling in rattle in R?" Tags="|machine-learning|r|rattle|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6183" PostTypeId="1" AcceptedAnswerId="6196" CreationDate="2015-06-23T12:59:13.607" Score="6" ViewCount="282" Body="&lt;p&gt;I'm writing my thesis at the moment, and for some time - due to a lack of a proper alternative - I've stuck with &quot;unstructured data&quot; for referring to natural, free flowing text, e.g. Wikipedia articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This nomenclature has bothered me from the very beginning, since it opens a debate that I don't want to get into. Namely, that &quot;unstructured&quot; implies that natural language lacks structure, which it does not - the most obvious being syntax. It also gives a negative impression, since it is the opposite of &quot;structured&quot;, which is accepted as being positive. This is not the focus of my thesis, though the &quot;unstructured&quot; part itself plays an important role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I completely agree with the writer of &lt;a href=&quot;https://gigaom.com/2013/03/17/can-we-please-stop-saying-unstructured-data/&quot;&gt;this article&lt;/a&gt;, but he proposes no alternative except for &quot;rich data&quot;, which doesn't cover my point. The point I'm trying to make that the text lacks a traditional database-like (e.g. tabular) structure of the data, with every piece of data having a clear data type and semantics that is easy to interpret using computer programs. Of course I'd like to condense this definition into a term, but so far I've been unsuccessful coming up with, or discovering an acceptable taxonomy in literature.&lt;/p&gt;&#xA;" OwnerUserId="9486" LastActivityDate="2018-07-05T13:57:45.227" Title="What is an alternative name for &quot;Unstructured Data&quot;?" Tags="|text-mining|definitions|terminology|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6184" PostTypeId="1" CreationDate="2015-06-23T13:54:03.990" Score="5" ViewCount="102" Body="&lt;p&gt;I am new to data science. I have a dataset of around 200,000 records, having 5 columns. There is a field called, &lt;code&gt;class&lt;/code&gt;. For each &lt;code&gt;class&lt;/code&gt;, there are one or many &lt;code&gt;divisions&lt;/code&gt;. I have to do this:&#xA;1. Filter the dataset, such that only those &lt;code&gt;classes&lt;/code&gt; with at least 5 divisions turn up.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;For each division, I have to calculate &lt;code&gt;attendance&lt;/code&gt; from another column.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is a minimum &lt;code&gt;attendance&lt;/code&gt; value for each &lt;code&gt;class&lt;/code&gt;. I have to find the &lt;code&gt;percentage of divisions in each class with the minimum attendance&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I started with importing the data in python using Pandas and started writing loops for processing this. But I am sure this is not the right way to do. Can you please give some idea.Can I do this in Excel pivot table?&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="21" LastEditDate="2015-06-24T06:48:40.240" LastActivityDate="2020-08-03T13:29:12.567" Title="Finding aggregated information of data" Tags="|r|python|data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6186" PostTypeId="1" AcceptedAnswerId="6313" CreationDate="2015-06-23T14:40:47.173" Score="11" ViewCount="6877" Body="&lt;p&gt;What I mean is the following: Instead of processing all the training data at once and calculating a model, we process one data point at a time and update the model directly afterwards. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen the terms &quot;on-line (or online) learning&quot; and &quot;incremental learning&quot; for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: The Bishop book (Pattern Recognition and Machine Learning) uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning. &lt;/p&gt;&#xA;" OwnerUserId="2673" LastEditorUserId="2673" LastEditDate="2015-06-29T16:03:01.080" LastActivityDate="2021-10-15T16:20:37.883" Title="Is there a difference between on-line learning, incremental learning and sequential learning?" Tags="|machine-learning|online-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6187" PostTypeId="2" ParentId="5273" CreationDate="2015-06-23T15:38:09.920" Score="1" Body="&lt;p&gt;I'm not seeing where the relationship between hypotheses and research paper is. I know what the relationship is in reality, but your question is a bit unorganized. Perhaps we could think about it in a different way... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would first create a definition of possible hypotheses (i.e., $H_{0}$ and $H_{1}$ are my two possible hypotheses, and every hypothesis will have both a null and an alternative), and then associate a $H_{0}$ with a $H_{1}$ such that a combination of both null and alternative hypotheses equate to one hypothesis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this, it's a matter of iteratively counting.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With $p \in P$, where $p$ is a paper in the set of papers $P$,&lt;/li&gt;&#xA;&lt;li&gt;With $w \in W$, where $w$ is a word in a set of words $W$ that indicate whether $p$ is a research paper or not,&lt;/li&gt;&#xA;&lt;li&gt;With $i$ as the number of times a $w$ occurred in a $p$, &lt;/li&gt;&#xA;&lt;li&gt;count the number of times a $w$ occurs in each $p$ and iterate $i$ accordingly. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2015-06-23T15:38:09.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6188" PostTypeId="1" AcceptedAnswerId="6283" CreationDate="2015-06-23T16:29:23.320" Score="6" ViewCount="304" Body="&lt;p&gt;The goal of this question is to be able to propose a user further choices based on his past experiences: like Amazon's book advices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a set of mp3 files, I assume that a set of mp3 tags data is already filled, based on the music he/she has alredy listened to  : what is the easiest way to  implement a machine learning that is able to propose a list of music choices  based on the user's set ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;NB : I'm a Machine learning novice I'd appreciate if the answer could be based on Orange, Weka or these kind of tools.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: removed the &lt;em&gt;classification&lt;/em&gt; tag as recommanded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For new comers as me:&lt;br&gt;&#xA;- the book &lt;a href=&quot;http://amzn.com/1118728963&quot; rel=&quot;nofollow&quot;&gt;Predictive Analytics For Dummies&lt;/a&gt; is a nice general introduction about this subject&lt;br&gt;&#xA;- a next step would be the the paper &lt;a href=&quot;http://www.csc.kth.se/utbildning/kth/kurser/DD143X/dkand12/Group89Michael/report/Nordstrom+Hakansson.pdf&quot; rel=&quot;nofollow&quot;&gt;Finding Clusters of Similar Artists&lt;/a&gt; which is really interesting especially for the K-means approach&lt;br&gt;&#xA;- &lt;a href=&quot;http://labrosa.ee.columbia.edu/millionsong/pages/tutorial&quot; rel=&quot;nofollow&quot;&gt;The millions songs dataset&lt;/a&gt; which is a gold mine with its huge dataset as well as tutorials with Python codes to use with it&lt;br&gt;&#xA;- special thanks sheldonkreger for his answer with neo4J graph usage idea wich is really interesting &lt;/p&gt;&#xA;" OwnerUserId="10102" LastEditorUserId="10102" LastEditDate="2015-07-06T12:50:27.080" LastActivityDate="2020-08-17T18:55:06.123" Title="What is the best way to propose an item from a set based on previous choices?" Tags="|machine-learning|data-mining|python|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6189" PostTypeId="1" AcceptedAnswerId="6401" CreationDate="2015-06-23T16:36:41.873" Score="5" ViewCount="125" Body="&lt;p&gt;I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The reason I am confused is &#xA;1)The scikit learn python codes don't scale in big data.&#xA;2)The spark mlib is written in scala and not in python.&#xA;3)Although h2o is open sourced I don't think it is written in python.&lt;/p&gt;&#xA;" OwnerUserId="10327" OwnerDisplayName="xof6" LastActivityDate="2015-07-14T06:09:27.803" Title="Bigdata cluster compatible distributed predictive model" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6190" PostTypeId="1" AcceptedAnswerId="6194" CreationDate="2015-06-23T19:15:08.960" Score="4" ViewCount="1251" Body="&lt;p&gt;To be concrete, given 2D numerical data as is shown as line plots below. There are peaks on a background average movement (with small vibrations). We want to find the values of pairs (x1, x2) if those peaks drops down to average; or (x1) only if the line doesn't back to the average.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/2TeLq.png&quot; alt=&quot;how to find x1 and x2 from data without plotting?&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/eVnwb.png&quot; alt=&quot;how to find x1 from data without plotting?&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are thousands of such 2D data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the right statistic or machine learning algorithm to find x1 and x2 above without plotting?&lt;/p&gt;&#xA;" OwnerUserId="10300" LastActivityDate="2015-06-29T04:27:24.677" Title="What is the right algorithm to detect segmentations of a line chart?" Tags="|machine-learning|logistic-regression|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6191" PostTypeId="2" ParentId="6188" CreationDate="2015-06-23T19:22:12.353" Score="1" Body="&lt;p&gt;I'm not sure what you are saying in the second paragraph, but &lt;em&gt;if&lt;/em&gt; your assumptions are that all the mp3s are already tagged, then the &lt;em&gt;easiest&lt;/em&gt; way to suggest further is to suggest mp3s with the highest overlap in tags.&lt;/p&gt;&#xA;" OwnerUserId="10303" LastActivityDate="2015-06-23T19:22:12.353" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6194" PostTypeId="2" ParentId="6190" CreationDate="2015-06-23T21:07:20.510" Score="5" Body="&lt;p&gt;One way to do what you're talking about is called &quot;change point analysis.&quot; There is an R package for this called &lt;a href=&quot;http://www.jstatsoft.org/v58/i03/paper&quot;&gt;changepoint&lt;/a&gt; that you might want to check out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python, you could try &lt;a href=&quot;https://pypi.python.org/pypi/changefinder/0.03&quot;&gt;changefinder&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8275" LastActivityDate="2015-06-23T21:07:20.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6195" PostTypeId="2" ParentId="6183" CreationDate="2015-06-23T22:52:02.133" Score="7" Body="&lt;p&gt;&quot;Raw data&quot; is what we say in NLP.&lt;/p&gt;&#xA;" OwnerUserId="10303" LastActivityDate="2015-06-23T22:52:02.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6196" PostTypeId="2" ParentId="6183" CreationDate="2015-06-24T01:22:17.710" Score="4" Body="&lt;p&gt;It is a bad idea to counterpose &quot;unstructure data&quot; to, say, tabular data (as in &quot;non-tabular data&quot;), as you will have to elliminate other alternatives as well (e.g., &quot;non-tabular and non-graph and ... data&quot;). &quot;Plain text&quot; (-- my choice) or &quot;raw text&quot; or &quot;raw data&quot; sound fine.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastActivityDate="2015-06-24T01:22:17.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6197" PostTypeId="2" ParentId="6190" CreationDate="2015-06-24T01:29:39.120" Score="2" Body="&lt;p&gt;Partition the time axis (the horizontal axis) into regions, and for each region compute the variation of the function value (or its deviation from the mean) as well as its finite differences (e.g., 1st and 2nd). The time points corresponding to &quot;anomalies&quot; in each of the obtained series are what you are looking for. If the number of points does not satisfy you, repeat the process hierarchically, altering the time granularity.&lt;/p&gt;&#xA;" OwnerUserId="8462" LastActivityDate="2015-06-24T01:29:39.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6198" PostTypeId="2" ParentId="6181" CreationDate="2015-06-24T02:17:05.160" Score="4" Body="&lt;p&gt;Any data consisting of a three element vector of numbers  $v =\{x,y,z\}$ could be viewed as topologically spherical if, after converting to polar coordinates, there is only one datum for every value of the 3D angle $\theta$. Thus, every point in the data represents an exterior of a topological sphere. This is taking the term &quot;topological&quot; literally and formally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also graph structures (acyclic) that have neighborhood structures that allows the graph to be mapped topologically to a sphere. For example, all graphs that can be mapped to Platonic Solids (i.e. vertexes to vertexes, edges to edges, 1:1) are topological spheres. This can include irregular neighborhood structures, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as practical examples of these, none come to mind immediately.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-06-24T02:17:05.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6200" PostTypeId="1" CreationDate="2015-06-24T11:23:03.077" Score="4" ViewCount="2815" Body="&lt;p&gt;Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?&lt;/p&gt;&#xA;" OwnerUserId="2643" LastEditorUserId="29169" LastEditDate="2019-06-07T23:25:05.770" LastActivityDate="2021-01-12T21:43:38.373" Title="What are the basic approaches for balancing a dataset for machine learning?" Tags="|machine-learning|dataset|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="6201" PostTypeId="1" CreationDate="2015-06-24T12:22:46.497" Score="3" ViewCount="6482" Body="&lt;p&gt;I am searching for resources on data science projects which used football/soccer data.&lt;/p&gt;&#xA;" OwnerUserId="10314" LastEditorUserId="75" LastEditDate="2016-02-26T12:35:26.777" LastActivityDate="2016-04-15T12:18:48.987" Title="Resources on Data Science for Football / Soccer?" Tags="|data-mining|reference-request|sports|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6202" PostTypeId="1" CreationDate="2015-06-24T12:24:03.540" Score="1" ViewCount="98" Body="&lt;p&gt;Is there a way to draw custom tables like in SPSS where you put some attributes to rows and some to column to visualize clustering results?&lt;/p&gt;&#xA;" OwnerUserId="8999" LastEditorUserId="15527" LastEditDate="2016-01-23T22:18:56.273" LastActivityDate="2016-01-23T22:18:56.273" Title="SPSS style custom tables in Orange" Tags="|clustering|orange|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6203" PostTypeId="2" ParentId="6188" CreationDate="2015-06-24T15:12:37.963" Score="2" Body="&lt;p&gt;What you are describing is not a classification task. You need to use a recommender engine to do this - that's what they are actually developed for. I suggest you to google 'Mahout in action' book which is brilliantly covers the development of recommender engines. After reading the theory you will be able to find a tool suitable for your task.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-06-24T15:12:37.963" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6204" PostTypeId="2" ParentId="6200" CreationDate="2015-06-24T15:17:38.227" Score="5" Body="&lt;p&gt;Well, you just train it on unbalanced dataset, it is not a problem. I don't think you need to apply some special techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only case where you may want to do something special is if your classes are skewed by their nature (if the skewness is the property of class itself, not of just your specific dataset). For example, if you build a classifier which will tell if someone has a malignant tumor in X-ray image or not. By nature of the task absolute majority of patients will not have a malignant tumor. In such case you may want to tune your classifier algorithm a bit, for example introduce a weighting into your SVM or smth like it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A special case is if you have EXTREMELY unbalanced classes. For example 100000 of positive examples and 20 negatives. In such case you will want to go away from classification task to the approach called 'anomaly detection'. &lt;/p&gt;&#xA;" OwnerUserId="7969" LastEditorUserId="1279" LastEditDate="2015-06-29T00:50:08.320" LastActivityDate="2015-06-29T00:50:08.320" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6205" PostTypeId="1" CreationDate="2015-06-24T15:21:29.590" Score="3" ViewCount="260" Body="&lt;p&gt;I query the &lt;a href=&quot;https://developers.facebook.com/docs/graph-api&quot; rel=&quot;nofollow&quot;&gt;Facebook graph API&lt;/a&gt; to get some users' likes, which come with a &quot;category&quot; field, which can be, for instance, &lt;em&gt;Italian Restaurant&lt;/em&gt; or &lt;em&gt;Health &amp;amp; Wellness website&lt;/em&gt; and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to draw a profile of the user so I was thinking of retrieving the top level root in a taxonomy graph of the aforementioned categories in such a way that the first example falls into something like &lt;em&gt;Food&lt;/em&gt; and the second one into something like &lt;em&gt;Health&lt;/em&gt;. This way I'd know, by a majority rule, who is a sporty person, who likes fashion etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a free public API I could use to get word taxonomies? I found &lt;a href=&quot;http://www.alchemyapi.com&quot; rel=&quot;nofollow&quot;&gt;Alchemy&lt;/a&gt;, an IBM project which is pretty cool but unfortunately limited to 1000 API requests per day and I'd need way more than that given the big amount of users and categories I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I'm probably asking for too much (a free and unlimited tool!) so the question might become &quot;do you have better ideas about how to solve this&quot;?&lt;/p&gt;&#xA;" OwnerUserId="982" LastActivityDate="2015-06-26T10:17:42.997" Title="Word taxonomies for Facebook likes categories" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6206" PostTypeId="1" AcceptedAnswerId="6211" CreationDate="2015-06-24T15:46:52.680" Score="4" ViewCount="1885" Body="&lt;p&gt;I started to study and programming in neural networks for a little while now, but I never read about the minimum number of observations one must collect in a dataset to get robust results. Of course, more observations better results, but, Does exist an empirical or theoretical relationship between variables and observations number?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, neither in econometrics you can compute the minimum number of observations, but it does exist some rule of thumbs that relies the number of exogenous variables to the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if there is something similar to that in neural networks too, but, till now, browsing on the internet, I did not find anything of useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas, advises or hint will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9225" LastEditorUserId="9225" LastEditDate="2015-06-24T21:06:10.770" LastActivityDate="2015-06-25T10:16:58.653" Title="How many observations in a neural networks dataset?" Tags="|dataset|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6207" PostTypeId="2" ParentId="6201" CreationDate="2015-06-24T16:04:32.073" Score="5" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There's this website called &lt;a href=&quot;http://sportsdatamart.com/category/football/&quot; rel=&quot;noreferrer&quot;&gt;Sportsdatamart&lt;/a&gt; which allows you to&#xA;download CSV and XLS file formats for a number of Football series&#xA;like the England Premier League, the Italian Serie A, the German&#xA;Bundesliga and the Spanish La Liga. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On &lt;a href=&quot;http://www.football-data.co.uk/downloadm.php&quot; rel=&quot;noreferrer&quot;&gt;Football Data UK&lt;/a&gt;, you can find EXCEL and CSV data files to&#xA;use for quantitative testing of betting systems in spreadsheet&#xA;applications. League tables, head2head statistics and information on&#xA;goalscrores, first scorers and top scorers could be accessed through&#xA;the &lt;a href=&quot;http://livescore.football-data.co.uk/&quot; rel=&quot;noreferrer&quot;&gt;Livescore&lt;/a&gt; service.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://en.soccerwiki.org/wiki.php&quot; rel=&quot;noreferrer&quot;&gt;Soccer Wiki&lt;/a&gt; is a collaborative database and anyone can create&#xA;and edit data. This community driven database contains information&#xA;on players, clubs, stadiums, managers, referees, leagues and other&#xA;data related to the world of soccer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are looking for data sets centered on European Football,&#xA;check &lt;a href=&quot;http://www.quora.com/Where-can-I-download-a-database-with-european-football-matches-data&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; out.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2015-06-24T16:04:32.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6208" PostTypeId="1" CreationDate="2015-06-24T16:36:32.730" Score="6" ViewCount="1338" Body="&lt;p&gt;This is my problem description:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;According to the Survey on Household Income and Wealth, we need to find out the top 10% households with the most income and expenditures. However, we know that these collected data is not reliable due to many misstatements. Despite these misstatements, we have some features in the dataset which are certainly reliable. But these certain features are just a little part of information for each household wealth.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unreliable data means that households tell lies to government. These households misstate their income and wealth in order to unfairly get more governmental services. Therefore, these fraudulent statements in original data will lead to incorrect results and patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have below questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should we deal with unreliable data in data science?&#xA;Is there any way to figure out these misstatements and then report the top 10% rich people with better accuracy using Machine Learning algorithms? -How can we evaluate our errors in this study? Since we have unlabeled dataset, should I look for labeling techniques? Or, should I use unsupervised methods? Or, should I work with semi-supervised learning methods?&#xA;Is there any idea or application in Machine Learning which tries to improve the quality of collected data?&#xA;Please introduce me any ideas or references which can help me in this issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="10318" LastActivityDate="2019-11-25T11:32:30.690" Title="Working with inaccurate (incorrect) dataset" Tags="|machine-learning|data-mining|dataset|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6209" PostTypeId="1" CreationDate="2015-06-24T20:05:17.320" Score="3" ViewCount="277" Body="&lt;p&gt;I've used optical character recognition (OCR) on a historic directory, and am trying to clean up the text with Microsoft Word. Specifically, I need some help writing a Regular Expression to combine two lines together. For example something that is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;John Smith, 87 Bank&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bldg&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;should actually be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;John Smith, 87 Bank Bldg&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried several approaches, but haven't been successful at all. Can anyone help me with this?&lt;/p&gt;&#xA;" OwnerUserId="10322" LastEditorUserId="13413" LastEditDate="2015-10-21T08:05:11.837" LastActivityDate="2015-10-21T08:05:11.837" Title="Regular Expressions in Word" Tags="|text-mining|" AnswerCount="3" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="6210" PostTypeId="2" ParentId="6209" CreationDate="2015-06-24T20:26:24.657" Score="1" Body="&lt;p&gt;I have a solution that might not be very standardized but will suffice your need. Copy all of your data into any advanced text editor like, Notepad++ or Sublime Text. Next, Use CTRL+H to toggle the find and replace feature. Find : '\n' and Replace with ''. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2015-06-24T20:26:24.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6211" PostTypeId="2" ParentId="6206" CreationDate="2015-06-24T20:52:38.970" Score="3" Body="&lt;p&gt;A neural network is nothing but a set of equations. And the basic rule of any set of equations is that you must have as many data points as the number of parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters of any neural network are its weights and biases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So that means that as the neural network gets deeper and wider, the number of parameters increase a lot, and so must the data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This being said, the more proper and detailed way to know whether the model is &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;overfitting&lt;/a&gt; is to check if the validation error is close to the training error. If yes, then the model is working fine. If no, then the model is most likely overfitting and that means that you need to reduce the size of your model or introduce regularization techniques.&lt;/p&gt;&#xA;" OwnerUserId="9943" LastEditorUserId="9943" LastEditDate="2015-06-25T10:16:58.653" LastActivityDate="2015-06-25T10:16:58.653" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="6212" PostTypeId="1" CreationDate="2015-06-25T12:37:48.030" Score="4" ViewCount="3513" Body="&lt;p&gt;I am currently using caret which optimises for accuracy. Is it possible to optimise for sensitivity. I see documentation mentioning that metric = 'roc' has been used, but sensitivity does not appear to be one of the options.&lt;/p&gt;&#xA;" OwnerUserId="9444" LastActivityDate="2020-01-15T02:36:24.910" Title="How to optimising caret for sensitivity" Tags="|machine-learning|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6213" PostTypeId="1" AcceptedAnswerId="6219" CreationDate="2015-06-25T12:50:03.597" Score="2" ViewCount="53" Body="&lt;p&gt;I'm trying to do a project about email marketing. I'm working on a tourism company and I want to make a best destination suggestion for the clients. But I need to see the relations between destinations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: How many people visited Dublin and then visited London?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: How can I best analyse this relation between the cities, given data about traveler itineraries?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to send email offers to clients who went to London and didn't go to Dublin (assuming a strong relation between London and Dublin).&lt;/p&gt;&#xA;" OwnerUserId="10335" LastEditorUserId="609" LastEditDate="2015-06-26T02:57:08.803" LastActivityDate="2015-06-26T03:49:58.613" Title="How can I show the relations between travel destinations?" Tags="|statistics|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6216" PostTypeId="1" CreationDate="2015-06-25T19:12:04.583" Score="3" ViewCount="979" Body="&lt;p&gt;Does anyone know how to get the learning rate from participant data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm computing all the expected values for all trials (=200)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$V_t(S) = V_{t-1}(S)+ \alpha \cdot \text{error }_t $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(\text{error }_t = R_t − V_{t-1}(S))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then computing alpha at each trial using&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha= (V_t(S)  - V_{t-1}(S) )/ \text{error }_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but it seems not the right way!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestions?&lt;/p&gt;&#xA;" OwnerUserId="10344" LastEditorUserId="5041" LastEditDate="2016-02-26T12:35:08.947" LastActivityDate="2016-08-17T20:19:25.913" Title="learning rate in reinforcement learning" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6217" PostTypeId="1" CreationDate="2015-06-25T21:24:37.237" Score="10" ViewCount="2224" Body="&lt;p&gt;I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When plotting the data, it is clear to the &lt;em&gt;trained&lt;/em&gt; eye that there are three different linear trends, the three color lines show the directions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/rigJH.png&quot; alt=&quot;data plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT, Additional info:&lt;/strong&gt; I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.&lt;/p&gt;&#xA;" OwnerUserId="10346" LastEditorUserId="10346" LastEditDate="2015-07-13T18:47:25.247" LastActivityDate="2019-07-06T01:58:17.090" Title="Is there a method that is opposite of dimensionality reduction?" Tags="|machine-learning|scikit-learn|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6218" PostTypeId="2" ParentId="6201" CreationDate="2015-06-26T03:10:04.370" Score="3" Body="&lt;p&gt;Check out the Github repositories for Christopher Long (octonion):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Football: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/octonion/football-public&quot; rel=&quot;nofollow&quot;&gt;https://github.com/octonion/football-public&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/octonion/football&quot; rel=&quot;nofollow&quot;&gt;https://github.com/octonion/football&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Soccer: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/octonion/soccer-m&quot; rel=&quot;nofollow&quot;&gt;https://github.com/octonion/soccer-m&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/octonion/soccer&quot; rel=&quot;nofollow&quot;&gt;https://github.com/octonion/soccer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Many others:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/octonion?tab=repositories&quot; rel=&quot;nofollow&quot;&gt;https://github.com/octonion?tab=repositories&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-06-26T03:10:04.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6219" PostTypeId="2" ParentId="6213" CreationDate="2015-06-26T03:49:58.613" Score="0" Body="&lt;p&gt;You can try Graph databases (Neo4j/Orient DB etc). Store location and connection between the location as nodes and edges. Then do analysis over graph data. Based on your need, you can use additional attributes (like count) and assign weights for edges etc. Neo4j supports collaborative filtering also.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-06-26T03:49:58.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6220" PostTypeId="2" ParentId="3719" CreationDate="2015-06-26T07:32:27.147" Score="1" Body="&lt;p&gt;I think you might have to keep overall Data pipelines and Machine learning pipelines in mind. For which you need a robust framework to move data between table like and graph like storage apart from powerful distributed processing. From my understanding Spark GraphX is promising to build these pipelines. Joseph Gonzalez's (one of the creator of GraphLab from CMU) talk on GraphX on youtube is worth watching.&lt;/p&gt;&#xA;" OwnerUserId="10352" LastActivityDate="2015-06-26T07:32:27.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6221" PostTypeId="2" ParentId="441" CreationDate="2015-06-26T07:41:55.480" Score="0" Body="&lt;p&gt;Good info @Sean Owen. Would like to add one additional. Spark may help to build Unified data pipelines in Lambda architecture addressing both Batch and Streaming layers with an ability to write to common serving layer. It is huge advantage to reuse the logic between batch and Streaming. Also Streaming K-Means algorithms in Spark1.3 is an added plus to ML apart from excellent job monitoring and process visualizations in 1.4.&lt;/p&gt;&#xA;" OwnerUserId="10352" LastActivityDate="2015-06-26T07:41:55.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6222" PostTypeId="2" ParentId="2651" CreationDate="2015-06-26T07:54:24.640" Score="1" Body="&lt;p&gt;For comprehending the derivation of Back propagation algorithm, I suggest &lt;a href=&quot;https://www.youtube.com/watch?v=aVId8KMsdUU&quot; rel=&quot;nofollow&quot; title=&quot;Ryan Harris&quot;&gt;Ryan Harris&lt;/a&gt; youtube video which is less daunting. You may find second video as well.&lt;/p&gt;&#xA;" OwnerUserId="10352" LastActivityDate="2015-06-26T07:54:24.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6223" PostTypeId="2" ParentId="6172" CreationDate="2015-06-26T08:07:36.270" Score="1" Body="&lt;p&gt;You may build models to classify genomes by population.&#xA;Run unsupervised learning (clustering) to see if populations are reconstructed in the model.&#xA;Build models to infer missing genotypes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do a Scalable DNA analysis you may check &lt;a href=&quot;http://bdgenomics.org/&quot; rel=&quot;nofollow&quot;&gt;Adam&lt;/a&gt; software based on Apache Spark&lt;/p&gt;&#xA;" OwnerUserId="10352" LastActivityDate="2015-06-26T08:07:36.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6224" PostTypeId="1" CreationDate="2015-06-26T08:07:59.093" Score="3" ViewCount="882" Body="&lt;p&gt;What range of values of the relative sum of sqaures error is acceptable for a good neural network? I am getting around 0.9 of the relative error for 1 model and around 0.4 for another. Are both the networks significant?&lt;/p&gt;&#xA;" OwnerUserId="10353" LastActivityDate="2016-02-03T07:56:37.303" Title="What does the relative sum of squares error signify in neural network analysis in SPSS?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6225" PostTypeId="2" ParentId="6205" CreationDate="2015-06-26T10:17:42.997" Score="4" Body="&lt;p&gt;I think total number of default categories in Facebook is somewhat static. There is around 140-160 categories. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ref: &lt;a href=&quot;https://stackoverflow.com/questions/4216648/facebook-pages-authoritative-list-of-categories/8576572#8576572&quot;&gt;https://stackoverflow.com/questions/4216648/facebook-pages-authoritative-list-of-categories/8576572#8576572&lt;/a&gt; and &lt;a href=&quot;http://www.marketinggum.com/types-of-facebook-pages-for-business/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.marketinggum.com/types-of-facebook-pages-for-business/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can cluster (manually) them in to 10-12 generic groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg: Sports Event, &#xA;    Sports League, &#xA;    Amateur sports team,&#xA;    Sports Venue,&#xA;    Athlete,&#xA;    Sports/Recreation/Activities etc&#xA;all belongs to Sports category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly cluster and form generic groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then you can assign a user to any of these generic group based on the like category.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-06-26T10:17:42.997" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6226" PostTypeId="1" AcceptedAnswerId="6229" CreationDate="2015-06-26T12:19:16.267" Score="2" ViewCount="3913" Body="&lt;p&gt;I have two different dataset with same variables. I have built training models separately using random forest. Now i want to combine both these models. Could anyone tell me how can i this be achieved? Do we have something called combine() function in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards,&#xA;Arun&lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-06-26T13:33:43.820" Title="How to combine two different random forest models into one in R?" Tags="|machine-learning|r|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6227" PostTypeId="1" CreationDate="2015-06-26T12:56:55.410" Score="4" ViewCount="146" Body="&lt;p&gt;I have a table representing posts on a message board. Posts may or may not have parents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the most common way to get all posts starting from a given post, or to find the root post of any given post? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of using JOINs to join parents and children, but how can I know how many joins I need before I stop?&lt;/p&gt;&#xA;" OwnerUserId="10171" LastActivityDate="2015-11-17T11:12:56.723" Title="Traversing trees in SQL: JOINs vs imperative algorithm" Tags="|graphs|sql|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6229" PostTypeId="2" ParentId="6226" CreationDate="2015-06-26T13:33:43.820" Score="1" Body="&lt;p&gt;This question has already been answered &lt;a href=&quot;https://stackoverflow.com/questions/19170130/combining-random-forests-built-with-different-training-sets-in-r&quot;&gt;here&lt;/a&gt;. It addresses the problem in hand. &lt;/p&gt;&#xA;" OwnerUserId="9966" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-06-26T13:33:43.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6230" PostTypeId="2" ParentId="6217" CreationDate="2015-06-26T13:58:42.547" Score="8" Body="&lt;p&gt;The short answer is yes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \  j=1,...,K$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of ways, better and worse, of doing this. You may want to look up terms like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Splines and generalised additive models.&lt;/li&gt;&#xA;&lt;li&gt;The kernel trick (how to make a model where $K\rightarrow \infty$).&lt;/li&gt;&#xA;&lt;li&gt;Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).&lt;/li&gt;&#xA;&lt;li&gt;Deep learning, representation learning&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper &lt;a href=&quot;http://arxiv.org/pdf/1206.5538.pdf&quot; rel=&quot;nofollow&quot;&gt;Representation Learning: A Review and New Perspectives&lt;/a&gt; deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastEditorUserId="5303" LastEditDate="2015-07-11T10:50:34.797" LastActivityDate="2015-07-11T10:50:34.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6232" PostTypeId="1" AcceptedAnswerId="6234" CreationDate="2015-06-26T20:48:12.417" Score="8" ViewCount="3544" Body="&lt;h1&gt;Introduction&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a dataset of different observation of different people and I want to group people together to know which person is closest to the other one. I also want to have a measure to know how close they are to each others and know the statistical significance.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Data&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       eat_rate drink_rate   sleep_rate    play_rate  name   game&#xA;1  0.0542192259 0.13041721 5.013682e-03 1.023533e-06  Paul Rayman&#xA;4  0.0688171511 0.01050611 6.178833e-03 3.238838e-07  Paul  Mario&#xA;6  0.0928997660 0.01828468 9.321211e-03 3.525951e-07  Jenn  Mario&#xA;7  0.0001631273 0.02212345 7.061524e-05 1.531270e-07  Jean   FIFA&#xA;8  0.0028735509 0.05414688 1.341689e-03 4.533366e-07  Mark   FIFA&#xA;10 0.0034844717 0.09152440 4.589990e-04 5.802708e-07  Mark Rayman&#xA;11 0.0340738956 0.03384180 1.636508e-02 1.354973e-07  Mark   FIFA&#xA;12 0.0266112679 0.20002020 3.380704e-02 4.533366e-07  Mark  Sonic&#xA;14 0.0046597056 0.01848672 5.472681e-04 4.034696e-07  Paul   FIFA&#xA;15 0.0202715299 0.16365289 2.994086e-02 4.044770e-07 Lucas   SSBM&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Reproduce it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(eat_rate = c(0.0542192259374624, 0.0688171511010916, &#xA;0.0928997659570807, 0.000163127341146237, 0.00287355085557602, &#xA;0.00348447171120939, 0.0340738956099744, 0.0266112679045701, &#xA;0.00465970561072008, 0.0202715299408583), drink_rate = c(0.130417213859986, &#xA;0.0105061117284574, 0.0182846752197192, 0.0221234468128094, 0.0541468835235882, &#xA;0.0915243964036772, 0.0338418022022427, 0.200020204061016, 0.0184867158298818, &#xA;0.163652894231741), sleep_rate = c(0.00501368170182717, 0.00617883308323771, &#xA;0.00932121105128431, 7.06152352370024e-05, 0.00134168946950305, &#xA;0.000458999029040516, 0.0163650807661753, 0.0338070438697149, &#xA;0.000547268073086768, 0.029940859740489), play_rate = c(1.02353325645595e-06, &#xA;3.23883801132467e-07, 3.52595117873603e-07, 1.53127022619393e-07, &#xA;4.53336580123204e-07, 5.80270822557701e-07, 1.35497266725713e-07, &#xA;4.53336580123204e-07, 4.03469556309652e-07, 4.04476970932148e-07&#xA;), name = structure(c(5L, 5L, 2L, 1L, 4L, 4L, 4L, 4L, 5L, 3L), .Label = c(&quot;Jean&quot;, &#xA;&quot;Jenn&quot;, &quot;Lucas&quot;, &quot;Mark&quot;, &quot;Paul&quot;), class = &quot;factor&quot;), game = structure(c(3L, &#xA;2L, 2L, 1L, 1L, 3L, 1L, 4L, 1L, 5L), .Label = c(&quot;FIFA&quot;, &quot;Mario&quot;, &#xA;&quot;Rayman&quot;, &quot;Sonic&quot;, &quot;SSBM&quot;), class = &quot;factor&quot;)), .Names = c(&quot;eat_rate&quot;, &#xA;&quot;drink_rate&quot;, &quot;sleep_rate&quot;, &quot;play_rate&quot;, &quot;name&quot;, &quot;game&quot;), row.names = c(1L, &#xA;4L, 6L, 7L, 8L, 10L, 11L, 12L, 14L, 15L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Given a dataset as fellow (with continuous and categorical feature), how can I know if a person (a categorical answer) identified by a name is more correlated to another person?&lt;/p&gt;&#xA;" OwnerUserId="6527" LastActivityDate="2017-01-18T00:44:44.920" Title="How to find similarity between different factors in a dataset" Tags="|machine-learning|r|similarity|correlation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6234" PostTypeId="2" ParentId="6232" CreationDate="2015-06-27T02:28:24.020" Score="6" Body="&lt;p&gt;One way is to normalize your quantitative  values (play, eat, drink, sleep rates) so they all have the same range (say, 0 -&gt; 1), then assign each game to its own &quot;dimension&quot;, that takes value 0 or 1. Turn each row into a vector and normalize the length to 1. Now, you can compare the inner product of any two people's normalized vectors as a measure of similarity. Something like this is used in text mining quite often&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;R Code for Similarity Matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Assumes you've saved your dataframe to the variable &quot;D&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Get normalization factors for quantitative measures&#xA;maxvect&amp;lt;-apply(D[,1:4],MARGIN=2,FUN=max)&#xA;minvect&amp;lt;-apply(D[,1:4],MARGIN=2,FUN=min)&#xA;rangevect&amp;lt;-maxvect-minvect&#xA;#Normalize quantative factors&#xA;D_matrix &amp;lt;- as.matrix(D[,1:4])&#xA;NormDMatrix&amp;lt;-matrix(nrow=10,ncol=4)&#xA;colnames(NormDMatrix)&amp;lt;-colnames(D_matrix)&#xA;for (i in 1:4) NormDMatrix[,i]&amp;lt;-(D_matrix[,i]-minvect[i]*rep(1,10))/rangevect[i]&#xA;gamenames&amp;lt;-unique(D[,&quot;game&quot;])&#xA;#Create dimension matrix for games&#xA;Ngames&amp;lt;-length(gamenames)&#xA;GameMatrix&amp;lt;-matrix(nrow=10,ncol=Ngames)&#xA;for (i in 1:Ngames) GameMatrix[,i]&amp;lt;-as.numeric(D[,&quot;game&quot;]==gamenames[i])&#xA;colnames(GameMatrix)&amp;lt;-gamenames&#xA;#combine game matrix with normalized quantative matrix&#xA;People&amp;lt;-D[,&quot;name&quot;]&#xA;RowVectors&amp;lt;-cbind(GameMatrix,NormDMatrix)&#xA;#normalize each row vector to length of 1 and then store as a data frame with person names&#xA;NormRowVectors&amp;lt;-t(apply(RowVectors,MARGIN=1,FUN=function(x) x/sqrt(sum(x*x))))&#xA;dfNorm&amp;lt;-data.frame(People,NormRowVectors)&#xA;&#xA;#create person vectors via addition of appropriate row vectors&#xA;PersonMatrix&amp;lt;-array(dim=c(length(unique(People)),ncol(RowVectors)))&#xA;rownames(PersonMatrix)&amp;lt;-unique(People)&#xA;for (p in unique(People)){&#xA;  print(p)&#xA;  MatchIndex&amp;lt;-(dfNorm[,1]==p)*seq(1,nrow(NormRowVectors))&#xA;  MatchIndex&amp;lt;-MatchIndex[MatchIndex&amp;gt;0]&#xA;  nclm&amp;lt;-length(MatchIndex)&#xA;  SubMatrix&amp;lt;-matrix(NormRowVectors[MatchIndex,],nrow=length(MatchIndex),ncol=dim(NormRowVectors)[2])&#xA;  CSUMS&amp;lt;-colSums(SubMatrix)&#xA;  NormSum&amp;lt;-sqrt(sum(CSUMS*CSUMS))&#xA;  PersonMatrix[p,]&amp;lt;-CSUMS/NormSum&#xA;}&#xA;colnames(PersonMatrix)&amp;lt;-colnames(NormRowVectors)&#xA;#Calculate matrix of dot products&#xA;Similarity&amp;lt;-(PersonMatrix)%*%t(PersonMatrix)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user9424" LastEditorDisplayName="user9424" LastEditDate="2015-06-28T04:45:23.703" LastActivityDate="2015-06-28T04:45:23.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6235" PostTypeId="1" AcceptedAnswerId="6239" CreationDate="2015-06-27T03:41:16.967" Score="2" ViewCount="172" Body="&lt;p&gt;Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2021-02-11T04:52:15.083" Title="Data repositories like UCI" Tags="|machine-learning|data-mining|dataset|databases|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6236" PostTypeId="1" CreationDate="2015-06-27T06:34:46.957" Score="3" ViewCount="83" Body="&lt;p&gt;On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dumbo start matrix2seqfile.py -input hdfs://hm1/user/trainf1.csv -output hdfs://hm1/user/train_hdfs5.mseq -numreducetasks 25 -hadoop $HADOOP_INSTALL &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I inspect my file system, I find that all the files produced are accumulated only in the hvs node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like the files to get distributed throughout the cluster--my data is not getting balanced throughout the cluster. Can anyone advise me on how to fix this?&lt;/p&gt;&#xA;" OwnerUserId="10365" LastEditorUserId="13413" LastEditDate="2015-12-29T22:58:44.283" LastActivityDate="2016-01-28T23:10:22.517" Title="Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster" Tags="|bigdata|python|apache-hadoop|map-reduce|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6237" PostTypeId="2" ParentId="6235" CreationDate="2015-06-27T08:46:32.010" Score="2" Body="&lt;p&gt;I do biological network analysis and &lt;a href=&quot;http://biit.cs.ut.ee/graphweb/welcome.cgi?t=examples&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; are some networks.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-06-27T08:46:32.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6239" PostTypeId="2" ParentId="6235" CreationDate="2015-06-27T15:40:36.653" Score="2" Body="&lt;p&gt;There're lots and tons of data sets for biological data.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/genbank/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GenBank Overview&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.1000genomes.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;IGSR: The International Genome Sample Resource&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ddbj.nig.ac.jp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bioinformation and DDBJ Center&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_biological_databases&quot; rel=&quot;nofollow noreferrer&quot;&gt;List of biological databases&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="85045" LastEditDate="2021-02-11T04:52:15.083" LastActivityDate="2021-02-11T04:52:15.083" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6240" PostTypeId="2" ParentId="6232" CreationDate="2015-06-27T17:14:44.760" Score="2" Body="&lt;p&gt;Despite normalized euclidean distance you can also have a look at the pearson distance as a similarity measure. Here is a neat description : &lt;a href=&quot;http://mines.humanoriented.com/classes/2010/fall/csci568/portfolio_exports/sphilip/pear.html&quot; rel=&quot;nofollow&quot;&gt;http://mines.humanoriented.com/classes/2010/fall/csci568/portfolio_exports/sphilip/pear.html&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="10314" LastActivityDate="2015-06-27T17:14:44.760" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6242" PostTypeId="1" CreationDate="2015-06-28T06:44:01.293" Score="4" ViewCount="1332" Body="&lt;p&gt;I am working on named entity recognition project and find &lt;strong&gt;&lt;a href=&quot;https://gate.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;GATE&lt;/a&gt;&lt;/strong&gt; very useful. But I couldn't find a Python connector for GATE yet. Is it possible or is there a way to integrate GATE work flows into Python ? Or how can I write &lt;strong&gt;JAPE&lt;/strong&gt; rules for extracting entities in python ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-07-23T13:49:02.203" LastActivityDate="2015-08-22T14:35:44.383" Title="GATE with Python" Tags="|python|nlp|java|" AnswerCount="0" CommentCount="1" ClosedDate="2021-03-17T03:28:24.110" ContentLicense="CC BY-SA 3.0" />
  <row Id="6243" PostTypeId="1" CreationDate="2015-06-28T14:26:12.743" Score="1" ViewCount="107" Body="&lt;p&gt;A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_{i},   i=1,...,n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example &lt;strong&gt;to maximize the Shannon entropy of the sequences of $\epsilon$&lt;/strong&gt; (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="10381" LastActivityDate="2019-08-12T15:44:44.750" Title="Self-designed objective for linear regression learning" Tags="|regression|linear-regression|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="6244" PostTypeId="1" CreationDate="2015-06-28T15:55:34.763" Score="0" ViewCount="34" Body="&lt;p&gt;I'm learning the linear regression now. I used R to build linear model upon a set of train model and try to predict() data based on the test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: I understand how train data gets collected. But what's with the test data? How did it get collected? Is the test data built, or is it collected, or is it predicted or what?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I'm learning data science by method of self-learning, so I lack the structural in my knowledge. I might know something on one place while lack the knowledge at another place. Please forgive and guide. Thanks. :)&lt;/p&gt;&#xA;" OwnerUserId="10382" LastActivityDate="2015-06-28T16:14:24.527" Title="How does the test data gets collected?" Tags="|beginner|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6245" PostTypeId="2" ParentId="6244" CreationDate="2015-06-28T16:14:24.527" Score="1" Body="&lt;p&gt;In general, you randomly splits the available data into following 3 sets. Train, validation and test. [Different ratios could be used depending on total amount of data at hand and the difficulty of the problem.] You can start with a simple 80%/10%/10% split.&#xA;You use the first two sets to build your model. So you would use the 80% split of your data to lets say build a logistic regression model. Now you use the 10% validation set to see how good your model is. You can iterate on this process until you are satisfied with the validation data set performance of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you use the last set (the &quot;test&quot; split) to see how your model would generalize on an &quot;unseen&quot; data set. [Here unseen means, your model never had access to &quot;see&quot; this data while in the learning phase]. You never use use this 3rd set to glean knowledge about how your model could be improved. Think of this set as a set your customer has, and you don't have access to, and to whom you are going to deliver your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get a hang of this concept also learn more about how you can do &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;cross-validation&lt;/a&gt; if you have limited data and to improve confidence on your model.&lt;/p&gt;&#xA;" OwnerUserId="10383" LastActivityDate="2015-06-28T16:14:24.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6247" PostTypeId="1" AcceptedAnswerId="6251" CreationDate="2015-06-28T16:56:58.467" Score="5" ViewCount="349" Body="&lt;p&gt;As my first project into data science, I would like to pick out the main clusters in noisy data. I think a good example would be trying to pick out certain links on a given StackExchange question that has a number of answers. The most common type of link is a link to a question on the SE network. The next common is either tag links, or links to user profiles. The remaining links might be random links included in posts, which is considered noise.  Ideally, I'm looking for a solution where I don't know how many clusters of links there will be ahead of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've implemented my first attempt using &lt;code&gt;scikit-learn&lt;/code&gt; and KMeans. However, it's not ideal because I appear to have to specify the number of clusters ahead of time, and I think the random, noisy links get grouped improperly. I also think it's more effective on a larger corpus compared to the relatively small one of URL tokens (though that's just a guess).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do this type of clustering, where the number of clusters is unknown or where one of the clusters is a sort of miscellaneous cluster containing objects that don't closely match the other clusters?&lt;/p&gt;&#xA;" OwnerUserId="10337" LastActivityDate="2015-06-29T04:11:05.683" Title="Can you use clustering to pick out signals in noisy data?" Tags="|python|clustering|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6249" PostTypeId="2" ParentId="6200" CreationDate="2015-06-28T22:47:23.027" Score="2" Body="&lt;p&gt;There are a few of other approaches you can take to try to balance your class distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Subsample Majority Class&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can balance the class distributions by subsampling the majority class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Oversample Minority Class&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sampling with replacement can be used to increase your minority class proportion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Adding Noise&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more sophisticated scheme is to add Gaussian, or other suitable, noise to the existing instances of the minority class in an effort to create a greater number of representative, but diverse,  instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;SMOTE&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A popular method to synthesise&#xA;minority class instances with greater complexity than pure noise addition is &lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;nofollow&quot;&gt;SMOTE&lt;/a&gt; ( Synthetic Minority Oversampling TEchnique ). This uses a K-member neighbourhood in feature space to impute new instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;WEKA has a filter for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though there is some evidence that this technique is not overly beneficial with high dimensional data &lt;a href=&quot;http://www.biomedcentral.com/1471-2105/14/106&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-06-28T22:47:23.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6251" PostTypeId="2" ParentId="6247" CreationDate="2015-06-29T04:11:05.683" Score="3" Body="&lt;p&gt;Have you looked at &lt;a href=&quot;https://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow&quot;&gt;DBSCAN&lt;/a&gt;? It is a density-based spatial clustering of data with noise that can define non-linear clusters (unlike k-means).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't require knowing the number of clusters. However, it does require two parameters (minimum cluster size and neighborhood size) that measure density. But you may be able to estimate them in your particular domain.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2015-06-29T04:11:05.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6252" PostTypeId="2" ParentId="6190" CreationDate="2015-06-29T04:27:24.677" Score="1" Body="&lt;p&gt;One method I've used successfully is the &lt;a href=&quot;http://www.mif.vu.lt/atpazinimas/dip/FIP/fip-Segmenta.html&quot; rel=&quot;nofollow&quot;&gt;Triangle Method&lt;/a&gt; used for thresholding in image processing. Basically is looks for the &quot;elbow&quot; in the data by drawing a line between the data peak and the minimum at one end and defining the &quot;elbow&quot; point as being the farthest from that line. Checkout the link and the picture there will clearly explain what the algorithm does.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2015-06-29T04:27:24.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6253" PostTypeId="1" CreationDate="2015-06-29T04:30:30.007" Score="1" ViewCount="263" Body="&lt;p&gt;I have a bunch of OCR produced text files with numerous spelling errors. I want to automatically correct these spelling based upon the prevalence of the closest correctly spelled words. For example, if the text reads &quot;junp&quot; and the options are &quot;jump&quot; or &quot;june&quot;, then it would pick between the two based upon both context and the frequency of the two words. Is there any software like this?&lt;/p&gt;&#xA;" OwnerUserId="10393" LastActivityDate="2015-06-29T04:30:30.007" Title="OCR Spell Check based upon previously found words" Tags="|data-mining|text-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6254" PostTypeId="1" CreationDate="2015-06-29T09:36:45.270" Score="1" ViewCount="48" Body="&lt;p&gt;Can anyone tell me where do I find any documentation for parameters like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-stepOffset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-alpha&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-decayExponent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in an &lt;code&gt;OnlineLogisticRegression&lt;/code&gt; function in &lt;code&gt;Mahout&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in what do they change in calls like this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;int FEATURES = 10000;       &#xA;OnlineLogisticRegression learningAlgorithm = new OnlineLogisticRegression(20, FEATURES, new L1())&#xA;                        .alpha(1).stepOffset(1000).decayExponent(0.9).lambda(3.0e-5).learningRate(20);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5224" LastEditorUserId="5224" LastEditDate="2015-07-09T09:50:30.590" LastActivityDate="2015-07-09T09:50:30.590" Title="Parameters for OnlineLogisticRegression function in Mahout" Tags="|logistic-regression|online-learning|apache-mahout|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6255" PostTypeId="1" AcceptedAnswerId="6261" CreationDate="2015-06-29T13:09:52.317" Score="3" ViewCount="4235" Body="&lt;p&gt;What is the significance of the square root in root-mean-square-error? Essentially, my question is: what is the difference between (rms error) and (rms error)&lt;span class=&quot;math-container&quot;&gt;$^2$&lt;/span&gt;?&lt;/p&gt;&#xA;" OwnerUserId="10291" LastEditorUserId="29169" LastEditDate="2019-06-07T23:37:53.217" LastActivityDate="2019-06-07T23:37:53.217" Title="Root Mean Squared Error (RMSE) - significance of square root" Tags="|machine-learning|statistics|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="6257" PostTypeId="1" AcceptedAnswerId="6258" CreationDate="2015-06-29T14:33:18.370" Score="3" ViewCount="242" Body="&lt;p&gt;Sorry if this has been answered before but could someone help me with solving the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each symbol in a dataset has a set of labels. Given a set of labels how can we predict more labels for that set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or to attempt a more formal wording: let a set of sets $S = \{ s_1, s_2, s_3,  ... \} $ where $ s_i \subseteq L $ and $ L = \{ l_1, l_2, l_3,...,l_m\} $ where $L$ is the finite set of labels. Given a set of labels $Y = \{ y_1 ,..., y_n \} $ where $Y \subseteq X $ what is the probability $ P(r = l_i) \forall l_i \in L$ so that $Y \cup \{r\} \in S $.&lt;/p&gt;&#xA;" OwnerUserId="9202" LastEditorUserId="9202" LastEditDate="2015-06-29T14:40:49.993" LastActivityDate="2015-06-29T16:02:18.727" Title="Predict set elements based on other elements" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6258" PostTypeId="2" ParentId="6257" CreationDate="2015-06-29T16:02:18.727" Score="2" Body="&lt;p&gt;Maybe you could use some algorithm that solves &quot;Market Basket Analysis&quot;. The problem is explained here: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Market Basket Analysis is a modelling technique based upon the theory&#xA;  that if you buy a certain group of items, you are more (or less)&#xA;  likely to buy another group of items. For example, if you are in an&#xA;  English pub and you buy a pint of beer and don't buy a bar meal, you&#xA;  are more likely to buy crisps (US. chips) at the same time than&#xA;  somebody who didn't buy beer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://albionresearch.com/data_mining/market_basket.php&quot; rel=&quot;nofollow&quot;&gt;http://albionresearch.com/data_mining/market_basket.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example of such an algorithm is: &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Association_rule_learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10404" LastActivityDate="2015-06-29T16:02:18.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6259" PostTypeId="1" CreationDate="2015-06-29T16:05:19.520" Score="4" ViewCount="1531" Body="&lt;p&gt;I am searching a scientific database for abstracts of papers containing the words project management. Here is the &lt;a href=&quot;http://en.journals.sid.ir/SearchPaper.aspx?str=project%20management&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For getting abstracts, I need to click on any paper and open a new page. How can I do that for 68 papers? I program in R and bash.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="11097" LastEditDate="2015-11-19T18:12:44.500" LastActivityDate="2022-07-13T14:09:11.727" Title="Web Scraping - a scientific database" Tags="|r|crawling|scraping|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6261" PostTypeId="2" ParentId="6255" CreationDate="2015-06-29T16:32:44.140" Score="0" Body="&lt;p&gt;It depends on what you are using the RMSE for. If you are merely trying to compare two models/estimators, then there is no significance to the square root. However, if you are trying to plot the error in terms of the same units as you made the measurements/estimates, then you need to take the square root to transform the squared units to the original units (much like variance vs standard deviation)&lt;/p&gt;&#xA;" OwnerDisplayName="user9424" LastActivityDate="2015-06-29T16:32:44.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6263" PostTypeId="2" ParentId="6169" CreationDate="2015-06-29T17:15:10.683" Score="0" Body="&lt;p&gt;Finally, I resolved the issue. I had to set the pyspark location in PATH variable and py4j-0.8.2.1-src.zip location in PYTHONPATH variable. &lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2015-06-29T17:15:10.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6264" PostTypeId="1" CreationDate="2015-06-29T17:32:31.397" Score="3" ViewCount="183" Body="&lt;p&gt;I'm trying to cluster and classify users with Mahout. At the moment I am at the planning phase, my mind is completely mixed with ideas, and since I'm relatively new to the area I'm stuck at the data formatting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have two data table (big enough). In the first table there are users and their actions. Every user has at least one action and they can have too many actions, too. About 10000 different user_actions and millions of records are in the table.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user        - user_action&#xA;u1          - a&#xA;u2          - b&#xA;u3          - a&#xA;u1          - c&#xA;u2          - c&#xA;u2          - c&#xA;u1          - b&#xA;u4          - f&#xA;u4          - e&#xA;u1          - e&#xA;u1          - d&#xA;u5          - d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the other table, there're action categories.  Every action may have none or multiple categories. There are 60 categories.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_action - category&#xA;a           - cat1&#xA;b           - cat2&#xA;c           - cat1&#xA;d           - NULL&#xA;e           - cat1, cat3&#xA;f           - cat4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm going to try to build a user classification model with &lt;strong&gt;Mahout&lt;/strong&gt; but I've no idea what I should do. &lt;strong&gt;What type of user vectors should I create? Or do I really need user vectors?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I need to create something like;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u1 (a, c, b, e, d)&#xA;u2 (b, c, c)&#xA;u3 (a)&#xA;u4 (f, e)&#xA;u5 ()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem in here, some users performed more than 100000 actions (some of them are same actions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So; this is more useful, I think;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u1 (cat1, cat1, cat2, cat1, cat3)&#xA;u2 (cat2, cat1, cat1)&#xA;u3 (cat1)&#xA;u4 (cat4, cat1, cat3)&#xA;u5 ()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The things I also worry about are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I weight categories for users? For example u1 has at least three action that related with cat1, while u3 has only 1. These one should be different?&lt;/li&gt;&#xA;&lt;li&gt;How can I decrease the difference between active users and passive ones? Like u1 has too many actions and so categories, u3 has only 1.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any guidance are welcome.&lt;/p&gt;&#xA;" OwnerUserId="10395" LastEditorUserId="10395" LastEditDate="2015-06-29T17:41:22.110" LastActivityDate="2015-07-01T21:43:52.883" Title="User profiling with Mahout from categorized user behavior" Tags="|classification|clustering|apache-mahout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6267" PostTypeId="1" AcceptedAnswerId="6293" CreationDate="2015-06-29T20:26:54.153" Score="1" ViewCount="760" Body="&lt;p&gt;My application is high frequency trading.  My data are time series of the bid and ask prices of a stock recorded on every tick (change in price). For each data point I also have a certain indicators that predict the future movement of the price. The indicators have different horizons of the predictions, some being optimal at few second intervals and others few minutes. I need to assign these predictors weights and based on whether the linear combination crosses a threshold, the decision will be taken to buy of sell the stock. So far I have tried the Differential Evolution (DE) method to figure out the weights. I use a black box model with the weights vector $w_i$ and threshold as inputs. For each data point I have a vector of indicators $\alpha _i$. &#xA;$$&#xA;total\_alpha = \sum\alpha _i*w_i&#xA;$$&#xA;If&#xA;$$&#xA;total\_alpha &amp;gt; threshold, BUY&#xA;$$&#xA;Else If&#xA;$$&#xA;total\_alpha &amp;lt; -threshold, SELL&#xA;$$&#xA;The output of the model is the sum of difference between each between the price of each consecutive buy and sell. This output is being optimised by the DE algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having trouble with the computational aspects. My data is very large (~$7 \times 10^8$ rows by 20 columns), and thus the execution time for the DE algorithm is unacceptable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: Is there a better and a faster way to solve this problem? &lt;/p&gt;&#xA;" OwnerUserId="10409" LastEditorUserId="609" LastEditDate="2015-07-02T06:09:04.920" LastActivityDate="2015-07-02T06:09:04.920" Title="How to speed up optimization using Differential Evolution?" Tags="|machine-learning|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6268" PostTypeId="1" CreationDate="2015-06-29T22:55:28.100" Score="15" ViewCount="32278" Body="&lt;p&gt;I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to perform a classification algorithm taking all the inputs to determine the income range. I need to build a dictionary of variables to mapped variables and use a map function to map the variables to numbers for processing. Essentially, I would my dataset to be in a numerical format so that I can work on implementing the models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the data set, there are categorical columns like education, marital status, working class etc. Can someone tell me how to convert them into numerical columns in pyspark?  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;workclass = {'?':0,'Federal-gov':1,'Local-gov':2,'Never-  worked':3,'Private':4,'Self-emp-inc':5,'Self-emp-not-inc':6,'State-gov':7,'Without-pay':8}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I created a sample dictionary with key value pairs for work class. But, I don't know how to use this in a map function and replace the categorical data in the CSV file with the corresponding value. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wc = pd.read_csv('PATH', usecols = ['Workclass'])&#xA;&#xA;df = pd.DataFrame(wc)&#xA;wcdict = {' ?':0,' Federal-gov':1,' Local-gov':2,' Never-worked':3,' Private':4,' Self-emp-inc':5,' Self-emp-n-inc':6,' State-gov':7,' Without-pay':8}&#xA;df_new = df.applymap(lambda s: wcdict.get(s) if s in wcdict else s)&#xA;print(df_new)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code I have written in normal python to convert the categorical data into numerical data. It works fine. I want to do the conversion in spark context. And, there are 9 categorical columns in the data source. Is there a way to automate the dictionary update process to have a KV pair for all 9 columns?&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="21" LastEditDate="2016-05-10T13:56:29.900" LastActivityDate="2021-04-21T05:18:10.133" Title="How to convert categorical data to numerical data in Pyspark" Tags="|python|apache-spark|categorical-data|pyspark|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6270" PostTypeId="1" CreationDate="2015-06-30T03:22:59.463" Score="0" ViewCount="1087" Body="&lt;p&gt;I want to extract name from CV. &#xA;I need high level if accuracy more than 95 %. I have started with taking assumptions that it is highly likely to be found in 10% top lines or if not there then in some section similar to Personal details.&#xA;Can u plz  provide any help/guidance ?&lt;/p&gt;&#xA;" OwnerUserId="10413" LastActivityDate="2015-06-30T04:34:50.527" Title="I want to extract name from CV" Tags="|text-mining|" AnswerCount="1" CommentCount="1" ClosedDate="2015-06-30T13:13:16.053" ContentLicense="CC BY-SA 3.0" />
  <row Id="6271" PostTypeId="2" ParentId="6270" CreationDate="2015-06-30T04:34:50.527" Score="1" Body="&lt;p&gt;Since CV data is mostly structured you can use &lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow&quot;&gt;Stanford NER&lt;/a&gt; for named entity recognition or use &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow&quot;&gt;CRF&lt;/a&gt; to train and model your own named entity recognizer.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-06-30T04:34:50.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6272" PostTypeId="2" ParentId="6268" CreationDate="2015-06-30T04:46:02.770" Score="2" Body="&lt;pre&gt;&lt;code&gt;workclass = {'?':0,'Federal-gov':1,'Local-gov':2,'Never-  worked':3,'Private':4,'Self-emp-inc':5,'Self-emp-not-inc':6,'State-gov':7,'Without-pay':8}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;try defining a mapper fuction which return key :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def mapr(dict_key):&#xA;    return workclass[dict_key]&#xA;&#xA;print list(map(mapr,workclass))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-06-30T05:10:01.897" LastActivityDate="2015-06-30T05:10:01.897" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6273" PostTypeId="2" ParentId="2576" CreationDate="2015-06-30T04:53:38.793" Score="1" Body="&lt;p&gt;Check out Clearbit.com -- &lt;a href=&quot;https://dashboard.clearbit.com/docs#company-api&quot; rel=&quot;nofollow&quot;&gt;https://dashboard.clearbit.com/docs#company-api&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Super comprehensive company lookup. Not free, but 50 free calls a month. &lt;/p&gt;&#xA;" OwnerUserId="10415" LastActivityDate="2015-06-30T04:53:38.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6274" PostTypeId="1" CreationDate="2015-06-30T05:28:31.537" Score="2" ViewCount="353" Body="&lt;p&gt;My retail dataset contains 3 numeric attributes and two categorical attributes  Time and ID with 50,000 records. Both categorical attributes have more than 20 thousand levels and their format is 1/11/2011 11:54 and 1TD10051 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do kmeans on these dataset? Converting categorical to binary will give very sparse dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to proceed?&lt;/p&gt;&#xA;" OwnerUserId="10416" LastEditorUserId="97" LastEditDate="2015-06-30T06:45:58.747" LastActivityDate="2015-08-29T08:55:58.290" Title="Kmeans on mixed dataset with high level for categ" Tags="|clustering|k-means|categorical-data|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6275" PostTypeId="2" ParentId="6274" CreationDate="2015-06-30T06:44:20.367" Score="2" Body="&lt;p&gt;There are plenty of methods, variations of k-means for the case of mixed dataset: k-modes, k-protoypes etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;It has been discussed already.&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-06-30T06:44:20.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6276" PostTypeId="2" ParentId="2576" CreationDate="2015-06-30T08:05:31.830" Score="1" Body="&lt;p&gt;Check out PySemantic - &lt;a href=&quot;http://github.com/motherbox/pysemantic&quot; rel=&quot;nofollow&quot;&gt;http://github.com/motherbox/pysemantic&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a video too - &lt;a href=&quot;https://www.youtube.com/watch?v=6z-18zP4hOA&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=6z-18zP4hOA&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2987" LastActivityDate="2015-06-30T08:05:31.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6277" PostTypeId="2" ParentId="6255" CreationDate="2015-06-30T10:52:18.617" Score="1" Body="&lt;p&gt;The square in RMSE is used because it always gives a positive value for error, so avoiding errors cancelling each other out, and affords greater weight to values further from the target function, so emphasising points for which the estimator is poor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The square root is used to remove the effects of the squaring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could look at using the Mean Absolute Error ( MAE ) which does not have the distance weighting effect of the RMSE and just takes the average of the absolute value of the errors. &lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-06-30T10:52:18.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6279" PostTypeId="1" CreationDate="2015-06-30T13:39:03.970" Score="1" ViewCount="88" Body="&lt;p&gt;I am currently experimenting with the idea of including the class of a feature vector as a separate feature. My work is about preposition selection in learner language use. I want to train a classifier that, given a certain position of a sentence, uses relevant context features around that position to build vectors. The classifier detects if there should be a preposition or if it should remain empty. Some features include:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Neighbouring N-grams, where N={2,3}, at Part-Of-Speech and token-level&lt;/li&gt;&#xA;&lt;li&gt;Neighbouring nouns and verbs&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The class expresses a binary distribution: 1 for preposition &lt;strong&gt;presence&lt;/strong&gt; and 0 for preposition &lt;strong&gt;absence&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider that if a language learner opted to select a preposition in a given sentence, but the classifier thinks there should &lt;strong&gt;NOT&lt;/strong&gt; be one, it can be interpreted as the classifier marking a language error: The learner inserted a preposition in a prohibited context. Conversely, if a learner did &lt;strong&gt;NOT&lt;/strong&gt; use a preposition at a given position in a sentence, but the classifier does expect one, it is also an error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is trained on correct negative (no preposition) and positive (&quot;yes&quot; preposition) &lt;em&gt;events&lt;/em&gt; (context features around a certain position: a whitespace or a preposition). This means that the assigned labels are the gold standard, similarly for the test data, because they are trained on *native *language use which is assumed to be error-free. The same assumption cannot be made when running the classifier on learner's language - after all, learners make mistakes, which is what I am trying to detect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to include the decision (yes/no) of the learner into the classifier rather than just as the assigned label. Because in practice, the chance on either error is very small, so the classifier should be very confident before marking it as such. But when I do this, the classifier always selects the value of that &lt;strong&gt;decision feature&lt;/strong&gt; as the class. After all, it is trained so that there is a 100% correlation between said feature and the assigned class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I turn this learner's decision - which is NOT an absolute indication of the actual answer - into an &lt;strong&gt;indication&lt;/strong&gt; instead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm experimenting with Linear Support Vector Classifier and Logistic/Linear Regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a bunch, if more details are needed, please ask.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2015-06-30T13:39:03.970" Title="How to include class as a feature" Tags="|machine-learning|feature-selection|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6280" PostTypeId="1" AcceptedAnswerId="6287" CreationDate="2015-06-30T14:40:03.720" Score="-2" ViewCount="21329" Body="&lt;p&gt;I am getting the error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Error: Unexpected '}' in &quot;   }&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have put closing brackets for every open bracket, but am not sure why I am getting this error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error arises when I run the following code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#find number of Columns &amp;amp; Rows&#xA;&#xA;numcol &amp;lt;- ncol(training_main)  &#xA;numrow &amp;lt;- nrow(training_main)&#xA;&#xA;&#xA;attach(training_main)  &#xA;NA_count &amp;lt;- as.data.frame(sapply(training_main, function(x) sum(is.na(x))))  &#xA;for(i in 1:numcol)    &#xA;{  &#xA;     if (NA_count &amp;gt; 0)  &#xA;  {  &#xA;  for(j in 1:numrow)  &#xA;  {    &#xA;    if(is.na(training_main[j,i])    &#xA;      {   &#xA;         training_main[j,i] &amp;lt;- as.character(training_main[j,i])    &#xA;         training_main[j,i] &amp;lt;- &quot;Empty&quot;    &#xA;         training_main[j,i] &amp;lt;- as.factor(training_main[j,i])     &#xA;         print(&quot;empty printed&quot;)    &#xA;&#xA;       }     &#xA;  }     &#xA;}    &#xA;}   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is the nested for -if - for - if as mentioned below allowed in R? That is the logic that I have to use. Is there any other way I can run this code?&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="29169" LastEditDate="2019-06-10T15:51:33.953" LastActivityDate="2019-06-10T15:51:33.953" Title="Error in R - Unexpected '}' in &quot; }&quot;" Tags="|r|" AnswerCount="1" CommentCount="2" ClosedDate="2015-07-01T15:29:56.203" ContentLicense="CC BY-SA 4.0" />
  <row Id="6282" PostTypeId="2" ParentId="6259" CreationDate="2015-06-30T15:04:10.570" Score="3" Body="&lt;p&gt;try RSelenium. with phantomjs since the date is requested and filled in by ajax calls. so any static web scraping tools wont work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I managed to get the list on the first page. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-headless.html&quot; rel=&quot;nofollow&quot;&gt;http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-headless.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample of what i managed to pull.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;remove( mopub, m, run , rx, x , first1)&#xA;library(RSelenium)&#xA;pjs&amp;lt;- phantom( pjs_cmd=&quot;C:/Users/bhavin.patel/Downloads/phantomjs-2.0.0-    windows/bin/phantomjs.exe&quot;)&#xA;Sys.sleep(5)&#xA;remDr &amp;lt;- remoteDriver( browserName = 'PhantomJS')&#xA;dsurl &amp;lt;- &quot;http://en.journals.sid.ir/SearchPaper.aspxstr=project%20management&quot;&#xA;remDr$open()&#xA;remDr$navigate(dsurl)&#xA;allt3 &amp;lt;-remDr$findElements('id', 'Table3')&#xA;lapply( allt3 , FUN=function(dst){ dst$getElementText(); })&#xA;&#xA;[[1]]&#xA;[[1]][[1]]&#xA;[1] &quot; 1 :   EFFECTIVE FACTORS ON RURAL PEOPLE’S NON-PARTICIPATION OF     MAHABAD’S DAM CATCHMENT IN WATERSHED MANAGEMENT PROJECTS\nAuthor(s): RASOULIAZAR SOLEIMAN*,FEALY SAEID\nJournal: INTERNATIONAL JOURNAL OF AGRICULTURAL MANAGEMENT AND DEVELOPMENT (IJAMAD)\nNumber: MARCH 2015 , Volume  5 , Number  1 ; Page(s) 19 To 26.\nKeyword(s): NON-PARTICIPATION, CATCHMENT, WATERSHED MANAGEMENT, MAHABAD TOWNSHIP, IRAN\nReference(s):  (0)      Citation(s):  (0) FullText:&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10391" LastActivityDate="2015-06-30T15:04:10.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6283" PostTypeId="2" ParentId="6188" CreationDate="2015-06-30T17:45:52.297" Score="3" Body="&lt;p&gt;There are many great ways to handle this problem. It is a recommendation problem, not a classification problem, as pointed out by others. There are many ways to do recommendation with a data set like this. I'll point out a few methods and you can choose one or try them all.&lt;/p&gt;&#xA;&lt;p&gt;The first method is called user-based collaborative filtering. The basic idea is to give users recommendations based on the tastes of like-minded users. So, you'd be trying to recommend music based on the listening history of users who have listened to the same songs. Such data can be modeled as a graph or sparse matrix. Then, you choose the exact algorithm depending on how you want to model your data.&lt;/p&gt;&#xA;&lt;p&gt;The second method is called item-based collaborative filtering. Rather than associating users together, this strategy looks at the set of items a user has 'rated' (the songs a user has listened to) and calculates how similar they are to a specific target item (song), or even to all the songs in your data set. It grabs the set of most-similar items and uses various methods to predict how much a user will like the song.&lt;/p&gt;&#xA;&lt;p&gt;In this case, you only have binary data (user listened to it or they did not). These calculations tend to work best with actual rating scores (like a 5 star system) because this gives more detailed variation amongst items in the data set.&lt;/p&gt;&#xA;&lt;p&gt;The third option is to model your data in a graph database like Neo4J and write graph traversal queries in order to find similar items. If you like graph theory, this can be a lot of fun. The sky is the limit in regards to what kinds of traversals will return good results. To get started, think of the users and songs as nodes in the graph, and 'listened' as the edge. &lt;span class=&quot;math-container&quot;&gt;$user-&amp;gt;listened-&amp;gt;$&lt;/span&gt;song&lt;/p&gt;&#xA;&lt;p&gt;Because of ratings and item-based filtering, and because there are probably many songs in your data set, and each user only listens to a very small portion of them, I'd first try a user-based collaborative filtering method which uses sparse matrix operations to calculate recommendations. If your data set is large, these computations scale horizontally so you can leverage parallel processing if you run into performance issues.&lt;/p&gt;&#xA;&lt;p&gt;You can find more detail about collaborative filtering in this &lt;a href=&quot;http://files.grouplens.org/papers/www10_sarwar.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastEditorUserId="98307" LastEditDate="2020-08-03T08:30:55.527" LastActivityDate="2020-08-03T08:30:55.527" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6287" PostTypeId="2" ParentId="6280" CreationDate="2015-06-30T20:09:09.820" Score="1" Body="&lt;p&gt;Error is due to missing &lt;code&gt;')'&lt;/code&gt; in the second &lt;code&gt;if&lt;/code&gt; statement in your code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(is.na(training_main[j,i]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;change to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if(is.na(training_main[j,i]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10299" LastActivityDate="2015-06-30T20:09:09.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6288" PostTypeId="2" ParentId="6212" CreationDate="2015-06-30T20:15:50.307" Score="2" Body="&lt;p&gt;Probably.  It depends on what you are doing exactly, and in particular which summaryFunction you are using.  If it's twoClassSummary for common binary classification then you should be able to specify &lt;code&gt;metric='Sensitivity'&lt;/code&gt;.  If that's not the case you also have the option of overriding the default performance summaryFunction with your own implementation that calculates the metrics you desire.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend reading through the documentation at &lt;a href=&quot;http://topepo.github.io/caret/training.html#control&quot; rel=&quot;nofollow&quot;&gt;http://topepo.github.io/caret/training.html#control&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="916" LastActivityDate="2015-06-30T20:15:50.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6289" PostTypeId="2" ParentId="6259" CreationDate="2015-06-30T21:52:02.733" Score="0" Body="&lt;p&gt;Another workaround is to get the listing by &lt;code&gt;POST&lt;/code&gt; requests using curl in bash.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get the curl post statement from Firebug ( Firefox F12 ) under Network , filter for XHR requests and copy the last statement which requests &lt;code&gt;SearchPaper.aspx?str=project+management&lt;/code&gt; (right-click -&gt; copy curl-adress).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this post request statement you have to increase the parameter &lt;code&gt;ctl00$ContentPlaceHolder1$txtPageNo&lt;/code&gt;  to a desired pagination number (1-6 in this case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then parse the output to a static xml parsing tool to get your data.&lt;/p&gt;&#xA;" OwnerUserId="10314" LastEditorUserId="10314" LastEditDate="2015-07-01T07:56:44.510" LastActivityDate="2015-07-01T07:56:44.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6290" PostTypeId="1" AcceptedAnswerId="6653" CreationDate="2015-06-30T22:01:36.153" Score="5" ViewCount="844" Body="&lt;p&gt;I wrote a Neural Networks prediction model in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data has a few inputs and two outputs. In order to make it work, I have to normalise every column on data for good prediction results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I had an issue. I run several times the prediction model with the same inputs to get the average and standard deviation of it. But obviously, those are also normalised to 0-1. On the test data I knew the min and max, so I could denormalised them. On the prediction values, I cannot the min and max real value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you solve this kind of problem and if you cannot, are there any other decent prediction techniques without the need of normalisation?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-07-31T21:03:23.593" Title="Denormalise data in Neural Networks" Tags="|python|neural-network|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6291" PostTypeId="2" ParentId="6290" CreationDate="2015-07-01T01:45:12.260" Score="2" Body="&lt;p&gt;Why not use the same statistics to denormalize the results? If the test data is good enough to train a neural network then why would its statistics not be good enough to denormalize the results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would suggest not using min and max as the scale factors. The min and max can be very sensitive. I suggest using &lt;a href=&quot;https://en.wikipedia.org/wiki/Robust_statistics&quot; rel=&quot;nofollow&quot;&gt;robust estimates&lt;/a&gt; of the mean and standard deviation to normalize. &lt;/p&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2015-07-01T01:45:12.260" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6292" PostTypeId="1" CreationDate="2015-07-01T06:13:41.820" Score="3" ViewCount="3179" Body="&lt;p&gt;I'd like to download my friends data as a graph/network from Facebook. In Feb 2015, Facebook banned the awesome application Netvizz. Do you know any alternative application for my purpose?&lt;/p&gt;&#xA;" OwnerUserId="10437" LastActivityDate="2015-07-01T10:56:58.213" Title="Is there any working alternatives to Netvizz?" Tags="|graphs|social-network-analysis|" AnswerCount="1" CommentCount="0" ClosedDate="2015-07-01T15:30:13.720" ContentLicense="CC BY-SA 3.0" />
  <row Id="6293" PostTypeId="2" ParentId="6267" CreationDate="2015-07-01T06:16:19.477" Score="0" Body="&lt;p&gt;Do you run your analysis algos in batch or live? Which programing language under which environment do you use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a first naive look, i would recommend to parallelize your code as each indicator calculation seems to be independent of the other's.&lt;/p&gt;&#xA;" OwnerUserId="10314" LastActivityDate="2015-07-01T06:16:19.477" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6294" PostTypeId="1" CreationDate="2015-07-01T06:18:06.550" Score="0" ViewCount="64" Body="&lt;p&gt;I am looking for techniques which utilise information in an incremental manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: A day with inclement climate is likely to be followed by another day with inclement climate. Or when an entire dataset sorted by date is available, an algorithm able to identify that a person is likely to call in sick when the previous day had bad weather.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any effective analysis which utilises this prior information where not all events are completely independent?&lt;/p&gt;&#xA;" OwnerUserId="10125" LastActivityDate="2015-07-01T20:50:14.187" Title="Algorithm/Analysis that utilises incremental information?" Tags="|algorithms|knowledge-base|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6295" PostTypeId="1" CreationDate="2015-07-01T08:44:01.663" Score="3" ViewCount="19630" Body="&lt;p&gt;I have a data table of 5 columns. I want to do a group by on one column  and want to get the count of another column, &lt;code&gt;id&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do this in this way:  &lt;code&gt;dt[,length(id),by=dt$cid]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I also want to get &lt;code&gt;sum of id&lt;/code&gt;, how do I proceed?&#xA;I tried this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt[,(sum(id) length(id)),by=dt$cid]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but it did not work. Could you please help out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="9253" LastEditDate="2015-08-03T07:15:45.487" LastActivityDate="2015-09-03T15:24:43.963" Title="Finding both count and average of a column in R data.table, after group by" Tags="|data-mining|r|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6298" PostTypeId="2" ParentId="6292" CreationDate="2015-07-01T10:56:58.213" Score="0" Body="&lt;p&gt;Go for NodeXL &lt;a href=&quot;https://www.youtube.com/watch?v=GyvwQSTO8Gc&quot; rel=&quot;nofollow&quot;&gt;SocialNetImporter&lt;/a&gt; pluggin&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-07-01T10:56:58.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6300" PostTypeId="2" ParentId="6295" CreationDate="2015-07-01T14:44:13.520" Score="1" Body="&lt;p&gt;Use tapply,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(dt,tapply(id, cid, FUN=sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9698" LastActivityDate="2015-07-01T14:44:13.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6301" PostTypeId="1" AcceptedAnswerId="6305" CreationDate="2015-07-01T15:52:15.467" Score="4" ViewCount="301" Body="&lt;p&gt;I need to extract fields like the document number, date, and invoice amount from a bunch of .csv files, which I believe are referred to as &quot;unstructured text.&quot; I have some labeled input files and will use the NLTK and Python to design a data extraction algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first round of classification, I plan to use tf-idf weighting with a classifier to identify the document type - there are multiple files that use the same format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, I need I way to extract the field from the document, given that it is X type of document. I thought about using features like the &quot;most common numbers&quot; or &quot;largest number with a comma&quot; to find the invoice amount, for example, but since the invoice amount can any numerical value I believe the sample size would be smaller than the number of possible features? (I have no training here, bear with me.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to do the second part? I think the first part should be okay, but I'm not sure that second part will work or if I even really understand the problem. How is my approach in general? I'm new to this kind of thing and this was the best I could come up with.&lt;/p&gt;&#xA;" OwnerUserId="10429" LastActivityDate="2015-07-01T20:34:48.350" Title="python - Will this data mining approach work? Is it a good idea?" Tags="|python|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6303" PostTypeId="2" ParentId="6295" CreationDate="2015-07-01T20:15:53.563" Score="1" Body="&lt;p&gt;You can use tapply or aggregate along with a custom function.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;aggregate(Id~cid, data=dt, FUN=function(x) c(mn=mean(x),ln=length(x)))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You may like to see my blog where I discuss few other scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://r4sqlminded.blogspot.com/2015/06/selecting-data-from-data-frame.html&quot; rel=&quot;nofollow&quot;&gt;http://r4sqlminded.blogspot.com/2015/06/selecting-data-from-data-frame.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10445" LastEditorUserId="10445" LastEditDate="2015-07-01T20:30:23.713" LastActivityDate="2015-07-01T20:30:23.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6304" PostTypeId="1" AcceptedAnswerId="6323" CreationDate="2015-07-01T20:18:20.783" Score="0" ViewCount="1081" Body="&lt;ol&gt;&#xA;&lt;li&gt;I want to try K-fold cross validation in R for C5.0 algorithm,&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The following is the code i use. Can someone suggest me how can i include k-fold as well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classifi_C5.0 &amp;lt;- C5.0(TARGET ~., , data = training_data_SMOTED, trails = 500,&#xA;                      control = C5.0Control(minCases = mincases_count,&#xA;                                            noGlobalPruning = FALSE))&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Is it required to do k-fold cross validation for Random forest? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-07-02T16:58:29.833" Title="Cross validation for C5.0 algorithm" Tags="|machine-learning|data-mining|classification|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6305" PostTypeId="2" ParentId="6301" CreationDate="2015-07-01T20:34:48.350" Score="1" Body="&lt;p&gt;I am not sure if using a classifier is the best way to approach this problem.  If it is something which can be easily extracted using regex, then that is the best way to do it. If however, you want to use classifiers, here are two questions you need to ask yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One, what does the unlabelled data look like and can you design good features from it? Depending on the kind of feature vector you design, the complexity of the classification task may range from very easy, to impossible. (A perceptron cannot solve XOR usually, except when you provide it with specific linear combinations of the input variable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two, what does the labelled data look like? Is it representative of the entire dataset or does it only contain very specific types of format? If it is the former, then your classifier will not work well on files which are not represented in the labelled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to test run a classifier first, you can solve the problem of having more features than training samples by using Regularization. Regularization forces the training algorithm of the classifier to accept the simplest possible solution (think occam's razor). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost all Machine Learning related packages in Python will have regularization options you can use, so enjoy. &lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-01T20:34:48.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6306" PostTypeId="2" ParentId="6294" CreationDate="2015-07-01T20:50:14.187" Score="1" Body="&lt;p&gt;It looks like you are looking for time series based Machine Learning (or Sequential Machine Learning as Emre pointed out). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that you have an entire dataset available to you beforehand. You train on this dataset and then use it to predict next day's weather given the weather of the last n days. Please note that you do not learn incrementally here, i.e. you do not learn from the new data that is rolling in every day. The learning is limited to the dataset that you had beforehand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Widely used algorithms for this kind of learning are Conditional Random Fields(CRF), Hidden Markov Models etc. Statistical methods like Autoregressive Models can also be used sometimes. A good starting point would be CRFs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If on the other hand you want to learn incrementally, read about online learning.  &lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-01T20:50:14.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6308" PostTypeId="1" AcceptedAnswerId="8159" CreationDate="2015-07-01T21:12:52.027" Score="1" ViewCount="24416" Body="&lt;p&gt;I have a SQLContext data frame derived from pandas data frame consisting of several numerical columns. I want to perform &lt;strong&gt;multivariate statistical analysis&lt;/strong&gt; using the &lt;strong&gt;pyspark.mllib.stats&lt;/strong&gt; package. The statistics function expects a RDD of vectors. I could not convert this data frame into RDD of vectors. Is there a way to convert the data frame? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rdd = sqlCtx.createDataFrame(df_new)&#xA; summary = Statistics.colStats(rdd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting df_new from &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; df_new = df.applymap(lambda s: dic.get(s) if s in dic else s) #df is a pandas dataframe&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting a PY4JJava error at the summary line. The issue is with the format of rdd. &lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2015-09-23T11:36:55.943" Title="How to convert a SQLContext Dataframe to RDD of vectors in Python?" Tags="|statistics|apache-spark|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6309" PostTypeId="2" ParentId="6264" CreationDate="2015-07-01T21:38:16.940" Score="3" Body="&lt;p&gt;I think one solution for this might be using Matrix factorization, to perform some kind of collaborative Filtering. This way you will have not have to deal with the two concern points that you have, explicitly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this, create a [User X Action] Matrix W, where rows are users and columns are actions. W(i,j) i.e. the Matrix entry on i row and j column would be the number of times the user i has performed action j. The matrix entries where no data is given are treated as missing data values by most collaborative filtering algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you factorize this Matrix, you factorize it into two parts W= A*B. The A matrix will have rows are users and columns as latent dimensions. From this A, you get a feature vector for each User. You can then run clustering on these feature vectors to cluster the users. For more intuition about the factorization approach, look up &quot;Matrix Factorization for collaborative Filtering&quot; on google. Mahout has matrix factorization implementations that you can use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the action categories is another matter altogether. It depends on what a category means. Does it mean that every action inside a category is very similar to each other or that they just have one or two aspect of similarity. A very easy approach would be to factorize a [User X (Action + Category)] Matrix. This can be solved using the same implementation mentioned earlier. You can also have more fancy schemes, like fixing part of the Matrix B to be very similar to the [Action X Category] matrix. But these schemes will require you to write optimization code yourself.&lt;/p&gt;&#xA;" OwnerUserId="10446" LastEditorUserId="10446" LastEditDate="2015-07-01T21:43:52.883" LastActivityDate="2015-07-01T21:43:52.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6310" PostTypeId="2" ParentId="6267" CreationDate="2015-07-02T01:07:56.313" Score="0" Body="&lt;p&gt;I cannot comment directly on how to accelerate the DE algorithm you are using, because I have not personally used it. I would however, like to suggest breaking up your problem into two different parts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part is to calculate the weights, by trying to predict the price on the next data-point given prices of &quot;n&quot; previous data-points. The second part of the problem is given the weights that you have determined, get the threshold which will allow you to get the best output ( i.e. sum of earnings). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two problems are somewhat decoupled (in my opinion), i.e. solving each optimally or suboptimally should provide you with an good overall solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part is just standard least squares, you can find many references to accelerate it or parallelize different versions of it. In the second part, you can just run a grid search, to find upper-threshold (to sell) and lower-threshold(to buy). Each run in the grid search with a different set of parameters should have O(n) complexity. Grid search itself, can be parallelized very easily.&lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-02T01:07:56.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6313" PostTypeId="2" ParentId="6186" CreationDate="2015-07-02T09:55:32.537" Score="7" Body="&lt;p&gt;So, the following answer is just based on different opinions of collegues and professors from the field. I want to try to summarize it briefly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sequential and online learning is mostly associated with Bayesian updating. The sequential learning is used widely for an order in time of the data, meaning that $x_1$ is coming always first, then $x_2$, then $x_3$ and so on. The Dataset each has a certain order in that sense. &#xA;In contrast to that incremental may be a whole block of data at time x and another block of data at time y. While the block internally may be randomly ordered. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning online learning, the people mostly referred to a data stream, hence a online learning is always incremental learning but incremental learning does not have to be online. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are quite fuzzy definitions, and in my opinion there is not clear definition though. I still hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="10451" LastActivityDate="2015-07-02T09:55:32.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6315" PostTypeId="2" ParentId="6236" CreationDate="2015-07-02T13:14:48.217" Score="1" Body="&lt;p&gt;In hadoop etc folder,hdfs-site.xml was having number of replication units as 1.That is why,all the files were getting saved on one single node.I changed it and the problem resolved.&lt;/p&gt;&#xA;" OwnerUserId="10365" LastActivityDate="2015-07-02T13:14:48.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6316" PostTypeId="1" CreationDate="2015-07-02T13:16:43.333" Score="0" ViewCount="206" Body="&lt;p&gt;We are trying to run J48 on a classified data set. Our class attribute has two possible values ( 0,1) when running J48 the tree terminates at the very first node and doesnt process any further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of considering (0- false) as the starting point of J48. How can we consider running J48 by selecting (1-true) as the starting point of the tree?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion will be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="10456" LastActivityDate="2019-12-31T03:00:30.737" Title="Weka class attribute suggestion" Tags="|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6318" PostTypeId="1" CreationDate="2015-07-02T14:36:59.107" Score="10" ViewCount="2841" Body="&lt;p&gt;I am looking for a website or book where several practical examples are given step by step, explaining how they choose the relevant features, the model selection procedure, etc...&lt;/p&gt;&#xA;" OwnerUserId="2489" LastActivityDate="2017-01-11T20:54:27.213" Title="Data science projects explained step by step?" Tags="|machine-learning|feature-selection|" AnswerCount="7" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6321" PostTypeId="2" ParentId="6318" CreationDate="2015-07-02T15:46:37.280" Score="1" Body="&lt;p&gt;I can recommend this collection of Ipython Notebooks which includes Data Science, Statistics and Machine Learning commented notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks&quot; rel=&quot;nofollow&quot;&gt;https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10314" LastEditorUserId="1352" LastEditDate="2016-04-17T18:32:15.550" LastActivityDate="2016-04-17T18:32:15.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6322" PostTypeId="1" CreationDate="2015-07-02T16:45:35.833" Score="1" ViewCount="260" Body="&lt;p&gt;I have $n$ objects located in a $d$ dimensional space, however I do not know their exact coordinates. For each object and each dimension, I have a set of noisy measurements of the coordinate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to cluster this data in the &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, with $n = 3$ and $d = 2$, I could have access to the following data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;object $a$, dim 1: 0.8 0.7 0.6&lt;/li&gt;&#xA;&lt;li&gt;object $a$, dim 2: 1.0 1.0 1.0 0.9&lt;/li&gt;&#xA;&lt;li&gt;object $b$, dim 1: 0.4 0.3&lt;/li&gt;&#xA;&lt;li&gt;object $b$, dim 2: 0.2 0.1&lt;/li&gt;&#xA;&lt;li&gt;object $c$, dim 1: 0.9 0.6&lt;/li&gt;&#xA;&lt;li&gt;object $c$, dim 2: $\emptyset$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In my current approach, I take the for each (object, dimension) pair that has data the average value, and I imput the average value of the dimension for the missing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would get&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$a$ [0.7 0.975]&lt;/li&gt;&#xA;&lt;li&gt;$b$ [0.35 0.15]&lt;/li&gt;&#xA;&lt;li&gt;$c$ [0.75 0.6] (where $0.6 = \frac{1+1+1+0.9+0.2+0.1}{6}$)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then I use the scikit-learn python library to run the mean-shift algorithm and obtain clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not completely satisfied with this methods for two main reasons (maybe they're the same):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;an (object, dimension) pair with a single observation is treated as as reliable as one with numerous observations&lt;/li&gt;&#xA;&lt;li&gt;there is a discontinuity between how a pair with zero observations and a pair with some observations are treated. In the second case, the value of the other objects does not influence the attribute at all.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My questions are: What would be a more principled approach to this problem? If I need to use another algorithm, are there open source decent quality libraries available that would implement it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I currently use $d = 256$ and $n = 10$ for my testing, but I aim to use $d = 16000$ and $n = 1000$ (but maybe a smaller $n$ if it's unrealistic) in my target application.&lt;/p&gt;&#xA;" OwnerUserId="10460" LastActivityDate="2015-12-30T11:56:27.567" Title="Clustering uncertain data with independent uncertainty per dimension" Tags="|clustering|data-cleaning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6323" PostTypeId="2" ParentId="6304" CreationDate="2015-07-02T16:58:29.833" Score="1" Body="&lt;p&gt;I would say cross validation is unnecessary here since the multiple partitioning of the data and variables is already implicit in Random Forests.   But it's still a good practice to hold out a testing set that is distinct from the training set.  This is mostly because you may introduce changes in your random forest to improve the performance on the test sets overall, thereby introducing the bias that the random forests are trying to overcome.   So if you withheld a portion of your data and judged the final performance of the RF on that withheld set only in the predict step, then it's fine.&lt;/p&gt;&#xA;" OwnerUserId="8947" LastActivityDate="2015-07-02T16:58:29.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6324" PostTypeId="2" ParentId="6318" CreationDate="2015-07-02T18:51:51.377" Score="1" Body="&lt;p&gt;One place you might find some interesting step-by-step explainations is the &lt;a href=&quot;http://blog.kaggle.com/category/winners-interviews/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle tutorial and winner's interviews&lt;/a&gt;. Often people will post a detailed summary of their approach.&lt;/p&gt;&#xA;" OwnerUserId="10462" LastEditorUserId="10462" LastEditDate="2017-01-11T20:54:27.213" LastActivityDate="2017-01-11T20:54:27.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6325" PostTypeId="1" CreationDate="2015-07-02T20:41:18.900" Score="2" ViewCount="99" Body="&lt;p&gt;I want to jointly cluster datapoints coming from different datasets (50 datasets with around 2000 points each). I would like to then extract information associated to the datapoints belonging to the different clusters to compare aspects of the datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my problem is that if I just place the datapoints originating from the different datasets into the same feature space, too few clusters comprise datapoints coming from all datasets simultaneously, so that after filtering I'm left with too few clusters (I tried with kmeans and similar methods).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Which is the best way to jointly cluster my points while imposing the condition that a given cluster should contain points from all datasets? The ideal solution would also allow some outliers to not fulfill this condition. The first thing I could think about is to define distances between points and clusters which are updated depending on whether a point belonging to the same dataset is already present in the cluster? Seems too far fetched though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any ideas, thanks a lot in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Three of my six features are spatial coordinates and ideally I would also want the clusters to be connected within a given dataset.&lt;/p&gt;&#xA;" OwnerUserId="10463" LastEditorUserId="10463" LastEditDate="2015-07-03T12:27:31.093" LastActivityDate="2016-02-02T00:42:05.867" Title="Joint spatial clustering: How to force clusters to minimally contain datapoints from all datasets" Tags="|clustering|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6326" PostTypeId="2" ParentId="6322" CreationDate="2015-07-02T22:13:46.977" Score="0" Body="&lt;p&gt;There are two separate issues and you should be able to combine the two following ideas to solve your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multiple measurements&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if you treat all the measurements as separate data points during the clustering and just cluster all the data together. Run your favorite clustering algorithm. Then, if one of your objects has measurements that get clustered in different clusters that will provide you with a measure of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Fuzzy_clustering&quot; rel=&quot;nofollow&quot;&gt;fuzziness&lt;/a&gt; of the object. Assign the object to the cluster with the most measurements assigned to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Incomplete measurements&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This one is trickier. You don't just have missing data points, but you have missing dimensions. &lt;a href=&quot;http://web.eecs.umich.edu/~girasole/pubs/balzanophdthesis.pdf&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a dissertation that starts to address some of these issues. The literature review may point you in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial idea would be to cluster the data sequentially in order of subspace rank. That is, first cluster all data points with similar rank &lt;em&gt;d&lt;/em&gt; together, then use those cluster labels as the initial clustering to rank those with rank &lt;em&gt;d-1&lt;/em&gt;, then rank &lt;em&gt;d-2&lt;/em&gt;, etc.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastEditorUserId="2728" LastEditDate="2015-07-03T01:27:07.817" LastActivityDate="2015-07-03T01:27:07.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6327" PostTypeId="2" ParentId="6318" CreationDate="2015-07-03T01:57:51.177" Score="1" Body="&lt;p&gt;I had the same question a few weeks ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally found O'Reilly's &lt;em&gt;Python for Data Analysis&lt;/em&gt; very useful in learning the basics. The book assumes you have some python programming experience, but it also has an appendix in the back to go through the basics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author gives you a wide variety of real world (not Monty Python) examples in the beginning you can create within the first few chapters, then goes into detail about each thing as the book goes on, building your knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found the instructions very easy and step by step. My professor who is my guide in all this was impressed how quickly I learned. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have heard good things about Kaggle.&lt;/p&gt;&#xA;" OwnerUserId="10465" LastActivityDate="2015-07-03T01:57:51.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6328" PostTypeId="1" CreationDate="2015-07-03T02:27:59.847" Score="2" ViewCount="1033" Body="&lt;p&gt;I am working on an assignment that is teaching how to plot and label using matplotlib using Python. Science or math is not my background. I have been given the formula for calculating the geostrophic wind and we are to plot it (on the y-axis) versus the latitude on the x-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to plot give an x and a y. Beyond that, the formula is not making sense to me given my lack of background in the area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/mgjFC.jpg&quot; alt=&quot;Formula for geostrophic wind&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, there is a key to translate the symbols to the correct python code, but I don't know the English names to the symbols, making it very difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I am given &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r'$x^{10}$'&#xA;r'$R_^{final}$'&#xA;r'$alpha^{\eta}$' &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first two are superscript and subscript. That I understand. But how this helps with the formula calculations I do not know. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am given the values to put into the formula as well. An explanation of the order of operations would help.&#xA;If,&#xA;g0=9.81 ms-­‐2;&#xA;ΔZ=60m;&#xA;Δn=2x10^5m;&#xA;and&#xA;f=2Ωsin(φ) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do I put the values into the formula and then plot them in matplotlib? is it as easy as x and y?&lt;/p&gt;&#xA;" OwnerUserId="10465" LastActivityDate="2015-11-01T08:45:39.457" Title="matplotlib geostrophic wind plot" Tags="|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6329" PostTypeId="2" ParentId="6255" CreationDate="2015-07-03T05:38:17.023" Score="0" Body="&lt;p&gt;If the set that you are using the RMSE on is a linear space, a good reason to use the square root is that you turn the set into a metric space. The square root ensures the right scaling property. Essentially, the RMSE is equivalent to the Euclidean norm. As a benefit, it is possible to use results of the general theory of metric spaces.&lt;/p&gt;&#xA;" OwnerUserId="10397" LastActivityDate="2015-07-03T05:38:17.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6330" PostTypeId="2" ParentId="6322" CreationDate="2015-07-03T06:47:43.017" Score="1" Body="&lt;p&gt;&lt;strong&gt;Model the measurement process&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each dimension you could attempt to model the distribution of the measurement process variance. Perhaps every dimension has the same measurement process or perhaps it is distinct. Combine all measurements from all dimensions if the process is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have enough data then you can build an empirical model of the distribution of errors using perhaps a kernel based estimator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise you might choose to use a Gaussian distribution formed from the mean value of each measurement and an overall measure of the variance of the measurement process. Then you replace observations you have with a sample drawn from a  distribution with the observation mean and overall measurement process variance.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-07-03T07:14:07.263" LastActivityDate="2015-07-03T07:14:07.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6331" PostTypeId="1" AcceptedAnswerId="6336" CreationDate="2015-07-03T08:52:36.247" Score="5" ViewCount="1231" Body="&lt;p&gt;I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked on data analysis before, I would like some guidance on how to store this data (which database design to use).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we're looking at currently are around 50 sensors that only send data when an event of interest occurs: if the temperature changes by 0.5 degrees or if the light turns on/off or if a room becomes occupied/vacant. So, the data will only be updated every few seconds. Also, in the future, I'd like to analyse some of the data. Hence, the data must be persistent in the database. What kind of technologies would you suggest to carry out this task?&lt;/p&gt;&#xA;" OwnerUserId="10470" LastActivityDate="2015-11-20T03:16:58.080" Title="Storing Sensor Data for Analysis of the Office" Tags="|apache-hadoop|databases|sql|mongodb|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6333" PostTypeId="2" ParentId="6328" CreationDate="2015-07-03T12:08:21.970" Score="0" Body="&lt;p&gt;Let me start with: I don't know how appropriate this is for Data Science StackExchange. But I can't comment yet and think I can provide an answer, so here goes.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Putting the values into the formula. The quantities are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$g_0$: Standard gravity (9.81 m/s2);&lt;/li&gt;&#xA;&lt;li&gt;$\Delta Z$: Change in geopotential height over some distance;&lt;/li&gt;&#xA;&lt;li&gt;$\Delta n$: The distance over which you calculated $\Delta Z$; and&lt;/li&gt;&#xA;&lt;li&gt;$f$: The Coriolis parameter, measuring the rate of revolution at&lt;/li&gt;&#xA;&lt;li&gt;$\phi$: The chosen latitude.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You wrote the equation that links the Coriolis parameter $f$ to the latitude and to $\Omega=7.29\times 10^{-5}$ s$^{-1}$. Let's calculate this part first. I'm assuming that a) you can use numpy along with matplotlib, and b) you were asked to plot the geostrophic wind as a function of latitude. One way to enter this into Python is by writing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; import numpy as np&#xA;&amp;gt; phi = np.arange(10, 91)&#xA;&amp;gt; Omega = 7.29e-5&#xA;&amp;gt; coriolis = 2 * Omega * np.sin(phi * np.pi/180)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second line gives you a numpy array of latitudes, in degrees, from 10N to 90N. I left out the tropics, since $f=0$ at the equator; the geostrophic wind isn't well-defined there. The third line defines Omega, and the last line gives you a numpy array of values for $f$. Note that the argument of np.sin should be in radians, not degrees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After doing this, the rest of the values are easy to input. For the order of operations, make sure everything in the numerator is grouped together and same for the denominator. You get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; grav, dz, dn = 9.81, 60.0, 2.0e5&#xA;&amp;gt; VGeo = (grav * dz) / (coriolis * dn)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first line uses a sequence of values to set grav ($g_0$), dz, and dn. The second line defines a numpy array for $V_g$, with the numerator and denominator grouped.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Plotting out the result. I don't know how familiar you are with the workings of matplotlib. Let me know if this doesn't make sense and I can expand on it. Many plots of something that varies in latitude have latitude on the y-axis, since that's the way globes and maps are oriented. So, when you make a plot of $(x,y)$ values, $y$ should be $\phi$ and $x$ should be $V_g$. A working example with some labels, using pyplot, might be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; import matplotlib.pyplot as plt&#xA;&amp;gt; fig, ax = plt.subplots()&#xA;&amp;gt; ax.set_ylim(0,90)&#xA;&amp;gt; ax.set_xlim(0,125)&#xA;&amp;gt; ax.plot(VGeo, phi, linestyle='-', color='k')&#xA;&amp;gt; ax.set_ylabel('Latitude $\phi$')&#xA;&amp;gt; ax.set_xlabel('Wind $V_g$')&#xA;&amp;gt; ax.set_title('Geostrophic wind $V_g = \frac{g_0 \Delta z}{f \Delta n}$')&#xA;&amp;gt; fig.savefig('test.png', format='png')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm doing several things here. The second line creates a figure and axes using plt.subplots(); without any arguments, this returns a single axis. Next, I'm setting the x- and y-limits so that we see all the data and so that the y-axis starts at the equator (even though we don't plot values south of 10N, I think it looks better). Then I'm just plotting $(x,y)=(V_g,\phi)$, with a solid black line. The next three lines involve creating labels using TeX formatting. The last line is saving the figure to a local PNG file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know how it works out!&lt;/p&gt;&#xA;" OwnerUserId="10427" LastActivityDate="2015-07-03T12:08:21.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6335" PostTypeId="2" ParentId="5990" CreationDate="2015-07-04T02:27:33.023" Score="45" Body="&lt;p&gt;The most logical way to transform hour is into two variables that swing back and forth out of sync. Imagine the position of the end of the hour hand of a 24-hour clock. The &lt;code&gt;x&lt;/code&gt; position swings back and forth out of sync with the &lt;code&gt;y&lt;/code&gt; position. For a 24-hour clock you can accomplish this with &lt;code&gt;x=sin(2pi*hour/24)&lt;/code&gt;,&lt;code&gt;y=cos(2pi*hour/24)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You need both variables or the proper movement through time is lost.  This is due to the fact that the derivative of either sin or cos changes in time where as the &lt;code&gt;(x,y)&lt;/code&gt; position varies smoothly as it travels around the unit circle.&lt;/p&gt;&#xA;&lt;p&gt;Finally, consider whether it is worthwhile to add a third feature to trace linear time, which can be constructed my hours (or minutes or seconds) from the start of the first record or a Unix time stamp or something similar.  These three features then provide proxies for both the cyclic and linear progression of time e.g. you can pull out cyclic phenomenon like sleep cycles in people's movement and also linear growth like population vs. time.&lt;/p&gt;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Adding some relevant example code that I generated for another answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Example of if being accomplished:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Enable inline plotting&#xA;%matplotlib inline&#xA;&#xA;#Import everything I need...&#xA;&#xA;import numpy as np&#xA;import matplotlib as mp&#xA;&#xA;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;&#xA;# Grab some random times from here: https://www.random.org/clock-times/&#xA;# put them into a csv.&#xA;from pandas import DataFrame, read_csv&#xA;df = read_csv('/Users/angus/Machine_Learning/ipython_notebooks/times.csv',delimiter=':')&#xA;df['hourfloat']=df.hour+df.minute/60.0&#xA;df['x']=np.sin(2.*np.pi*df.hourfloat/24.)&#xA;df['y']=np.cos(2.*np.pi*df.hourfloat/24.)&#xA;&#xA;df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1UTYj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1UTYj.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def kmeansshow(k,X):&#xA;&#xA;    from sklearn import cluster&#xA;    from matplotlib import pyplot&#xA;    import numpy as np&#xA;&#xA;    kmeans = cluster.KMeans(n_clusters=k)&#xA;    kmeans.fit(X)&#xA;&#xA;    labels = kmeans.labels_&#xA;    centroids = kmeans.cluster_centers_&#xA;    #print centroids&#xA;&#xA;    for i in range(k):&#xA;        # select only data observations with cluster label == i&#xA;        ds = X[np.where(labels==i)]&#xA;        # plot the data observations&#xA;        pyplot.plot(ds[:,0],ds[:,1],'o')&#xA;        # plot the centroids&#xA;        lines = pyplot.plot(centroids[i,0],centroids[i,1],'kx')&#xA;        # make the centroid x's bigger&#xA;        pyplot.setp(lines,ms=15.0)&#xA;        pyplot.setp(lines,mew=2.0)&#xA;    pyplot.show()&#xA;    return centroids&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now lets try it out:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;kmeansshow(6,df[['x', 'y']].values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MkVNg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MkVNg.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;You can just barely see that there are some after midnight times included with the before midnight green cluster.  Now lets reduce the number of clusters and show that before and after midnight can be connected in a single cluster in more detail:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;kmeansshow(3,df[['x', 'y']].values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bWg2B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bWg2B.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;See how the blue cluster contains times that are from before and after midnight that are clustered together in the same cluster...&lt;/p&gt;&#xA;&lt;p&gt;QED!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="100225" LastEditDate="2021-09-15T03:32:53.783" LastActivityDate="2021-09-15T03:32:53.783" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="6336" PostTypeId="2" ParentId="6331" CreationDate="2015-07-04T05:44:08.447" Score="2" Body="&lt;p&gt;I have been doing similar project in my college. I have classroom and I'm supposed to collect data like temp, humidity, light, occupancy, etc. Assuming that you have worked with sensors and motes to use, I'm going to explain rest of the structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need sensor network setup and like you said you have done it. These sensor networks generally do not send data directly over internet so you need a Gateway that can collect data from sensors and send it over internet to local server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On server side you need REST API and you could use any language to develop it and I use PHP. I find it very easy to use and develop using PHP. This REST API shall receive data from Gateway and store it into database. I use mysql database because amount of data is not so big for us. But if your data is big enough you can use big data Nosql tool like mongoDB or so. Whatever type of database you use structure remains same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sending data from Gateway to server you can use protocols like HTTP or MQTT whichever you feel comfortable. What I do is I have WSN controller that sends data over USB to Gateway then Gateway sends data to server over Ethernet. So I had to develop USB to Ethernet Gateway. If you can just take two UART terminals out of your controller you can build UART to Ethernet Gateway using any microcontroller or even Arduino Ethernet shield would work in that case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case data is sensed periodically but as you said you are sensing data when event of interest occurs then you can use poisson distribution method over periodically collected data to predict what is average number of events per day and then you can decide if your data is big or not.&lt;/p&gt;&#xA;" OwnerUserId="10479" LastActivityDate="2015-07-04T05:44:08.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6338" PostTypeId="2" ParentId="6328" CreationDate="2015-07-04T06:26:36.840" Score="1" Body="&lt;p&gt;Your answer was incorrect because the axes were flipped and it didn't plot the entire line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was the correct answer I was looking for.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt &#xA;import numpy as np&#xA;&#xA;# define the parameters&#xA;g = -9.81 # m/s^2&#xA;dZ = 60 # m&#xA;dx = 2e5 # m&#xA;omega = 7.2921e-5 # rad/s&#xA;phi = np.linspace(10,90) # deg&#xA;f = 2 * omega * np.sin(np.radians(phi)) # coriolis frequency, s^-1&#xA;&#xA;# compute geostrophic wind, x-component&#xA;u_g = -1. * g/f * dZ/dx&#xA;&#xA;# plot phi vs V_g&#xA;fig, ax = plt.subplots()&#xA;ax.plot(phi, u_g, color='#FF33CC')&#xA;ax.set_xlabel('$\phi$')&#xA;ax.set_ylabel('$V_{g}$ (m $s^-1$)')&#xA;plt.title('Geostrophic Wind vs. Latitude')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10465" LastActivityDate="2015-07-04T06:26:36.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6339" PostTypeId="1" CreationDate="2015-07-04T11:08:18.030" Score="1" ViewCount="70" Body="&lt;p&gt;I have Wireless Sensor Network setup in a classroom. I'm sensing data like temp, humidity, light, occupancy, etc. Classroom can accommodate 150 students and has 6 AC with light. From one side the classroom has big windows and that side is towards west and other 3 sides have thick walls and doors always closed. I have been looking for some questions that can be answered by analyzing data collected. Some of the questions I can think of are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Predicting optimum time to turn on AC before class starts so that class is attains required temp with least energy consumed.&lt;/li&gt;&#xA;&lt;li&gt;As of now I have 9 sensors on all walls and ceiling too. So to decide best position for sensor such that I can co-relate and predict other sensor values from only one or two sensor values. &lt;/li&gt;&#xA;&lt;li&gt;Decide optimum amount of light needed by occupants to be comfortable in classroom.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please suggest to me more questions that can asked about this situation. Also suggest if I can use somehow machine learning in WSN I have deployed.&lt;/p&gt;&#xA;" OwnerUserId="10479" LastEditorUserId="31513" LastEditDate="2017-04-26T16:24:37.390" LastActivityDate="2017-04-26T16:24:37.390" Title="Questions for data analysis of a class room" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6340" PostTypeId="1" AcceptedAnswerId="6348" CreationDate="2015-07-04T12:18:51.560" Score="0" ViewCount="4195" Body="&lt;p&gt;I constructed a neural networks in R using &lt;code&gt;neuralnet&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to test that using cross-validation, that is a technique based on using 4/5 of the dataset to train the network and the fifth one as the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder about what measures I should use to measure the neural networks performance in terms of predictability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you suggest what measures are commonly used in the field and explain me why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint and ideas about that will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9225" LastActivityDate="2015-07-06T06:49:46.487" Title="What are the performance measures in the neural networks field?" Tags="|machine-learning|r|neural-network|predictive-modeling|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6341" PostTypeId="1" AcceptedAnswerId="6344" CreationDate="2015-07-04T21:55:51.210" Score="5" ViewCount="1005" Body="&lt;p&gt;I've been using clustering in my bag of ML techniques for quite some time now, and  I've never found a satisfying answer to this question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In DBSCAN, we define a maximum radius with which to form clusters.  The algorithm will scan the space and group together points that are ALL reachable from one another.  However, we can sometimes end up with a non-convex cluster.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My confusion is around how the notion of a &quot;radius&quot;, which describes a convex object, can be an input to an algorithm which results in a non-convex object?&lt;/p&gt;&#xA;" OwnerUserId="10512" LastEditorUserId="13727" LastEditDate="2016-03-02T14:49:02.360" LastActivityDate="2016-03-02T14:49:02.360" Title="How are clusters from DBSCAN sometimes non-convex?" Tags="|machine-learning|clustering|dbscan|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6342" PostTypeId="1" CreationDate="2015-07-05T01:07:42.470" Score="3" ViewCount="99" Body="&lt;p&gt;Stats version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: $f_i(a,b,c;x,y)$, with $f$ being a vector of several quantities. I would like to cluster the points in $(x,y)$-space into regions that have a similar response to the input variables $(a,b,c)$. If I treat the points independently, applying a clustering algorithm is fairly easy. (I can elaborate on which one I'm using if you think it's relevant, but I'm open to suggestions on your favorites.) But there's no guarantee that the clusters are near each other spatially. Having contiguous clusters of roughly equal size is definitely a plus. What are some ways to modify the approach to add in that constraint?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Science version: I'm running an atmospheric chemistry model and varying emissions over the US. In that setup, $(a,b,c)$ are the national-total emissions of NOx, SO2, and NH3; $(x,y)$ are grid points over the US; and $f_i$ is a vector of the various chemical species I'm tracking. I've varied $(a,b,c)$ in a number of simulations. Now I want to divide the country into regions that have broadly similar responses to the emissions changes, and create simplified models for a manageable number of regions. I'm particularly interested in nonlinearities/mixed-effects/second-order derivatives in the chemistry response. It's possible that different regions of the country do respond similarly to emissions even if they're not near each other, but the different regions should have some sort of spatial agglomeration. Thoughts?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Side question: I originally played around with this a few years ago with Matlab code but have since migrated to Python. Suggestions on how to efficiently use Python stats packages for this sort of problem?&lt;/p&gt;&#xA;" OwnerUserId="10427" LastActivityDate="2016-02-02T18:53:28.527" Title="Spatial clustering based on response to inputs and building a reduced model" Tags="|python|clustering|data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6343" PostTypeId="2" ParentId="6341" CreationDate="2015-07-05T08:44:07.217" Score="2" Body="&lt;p&gt;I think it's non-convex because the particular cluster assignment you get when applying DBSCAN depends on the order you traverse the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's try to illustrate it with an example. Consider this dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ScA8N.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to run DBSCAN with radius $r = 3$ and $\text{min_pts} = 4$, so you get this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/OK6rC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point in the center is not a core point because it has only 3 points, not 4, and we have only two core points. And depending on how you traverse the data points, you may get different cluster assignments: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/YdMK9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The top picture shows the result we'd get by traversing left-to-right, and the bottom picture - by traversing right-to-left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently both these results would correspond to the same value of the cost function, thus the cost function has several minima and it is not convex. &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2015-07-05T08:44:07.217" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6344" PostTypeId="2" ParentId="6341" CreationDate="2015-07-05T13:22:46.627" Score="4" Body="&lt;p&gt;A cluster in DBSCAN consists of &lt;em&gt;multiple&lt;/em&gt; core points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The radius is the area covered by a single core point, but together with neighbor core points the shape will be much more complex. In particular, they can be much larger than epsilon, so you should choose a small value, and rely on this &quot;cover&quot; functionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/DBSCAN#/media/File:DBSCAN-density-data.svg&quot; rel=&quot;nofollow&quot;&gt;Wikipedia has an example of a non-convex cluster&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-07-05T13:22:46.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6345" PostTypeId="1" AcceptedAnswerId="6356" CreationDate="2015-07-05T15:10:56.790" Score="8" ViewCount="18196" Body="&lt;p&gt;How can I transform the following DataFrame into one with cities as rows and each cuisine as a column, and 1 or 0 as values (1 if the city has that kind of cuisine)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this turns out to be a very common problem in transforming data into features for machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of the Pandas pivot_table functionality, but it asks for a value column, and in this case we don't have any.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;data = {&#xA;    'city': ['NY','NY', 'SF','SF','SF'],&#xA;    'cuisine': ['Japanese', 'Chinese', 'French', 'Japanse', 'German']&#xA;}&#xA;&#xA;df = pd.DataFrame(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10171" LastEditorUserId="10171" LastEditDate="2015-07-05T15:33:22.707" LastActivityDate="2022-09-23T07:20:39.367" Title="Pivoting a two-column feature table in Pandas" Tags="|data-mining|feature-extraction|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6346" PostTypeId="2" ParentId="6318" CreationDate="2015-07-05T15:50:02.373" Score="2" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you want an application-oriented book, consider Christopher Bishop's &lt;a href=&quot;http://www.mbmlbook.com/&quot; rel=&quot;nofollow&quot;&gt;Model-Based Machine Learning&lt;/a&gt;. He has more technical books that are well regarded.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are looking for lots of code, &lt;a href=&quot;http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/&quot; rel=&quot;nofollow&quot;&gt;Probabilistic Programming &amp;amp; Bayesian Methods for Hackers&lt;/a&gt; is an option.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another introductory book with a more statistical bent is &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow&quot;&gt;An Introduction to Statistical Learning with Applications in R&lt;/a&gt;. Again, the authors have a well-regarded technical version of the book.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-07-05T15:50:02.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6347" PostTypeId="2" ParentId="6331" CreationDate="2015-07-05T17:10:14.417" Score="2" Body="&lt;p&gt;The best choice for storage technologies will depend largely on how much data (in terms of bytes) you expect to accumulate over the lifetime of your project, so the first thing i would do is try to get some sample data, or make some educated guesses (e.g. how many bytes does 1 temperature recording take up X how many change events am I expecting per day X how many temperature sensors X how many days worth of data you want to store and analyse over time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a rough idea of how much data you need to store and analyse, you can use that to start narrowing down your choices. There's no right answer, and others may disagree, but I would suggest that if you're dealing with anything less than terabytes of data, you don't need hadoop (I noticed that's a tag in your question) - hadoop is not really a data storage solution (although it does have it's own file system called HDFS or just DFS), it's more of a framework for processing and transforming huge quantities of data. Also if you don't have thousands of events per second to record, you probably don't need NoSQL solutions either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For storage of structured data, given that you've never really done data analysis before, SQL databases are probably the way to go if you have gigabytes or less, and SQL will be easier and more useful to learn - it's mature, been around for ages and is still the go-to standard in most industries, so there are &lt;a href=&quot;http://www.w3schools.com/sql/&quot; rel=&quot;nofollow&quot;&gt;plenty of learning resources&lt;/a&gt;. Maybe try out MySQL Community Edition server (free, open source) as a start, I would also recommend the MySQL Workbench to help you get started (a bunch of GUI tools you can use to mess around with SQL when learning)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I don't know anything about capturing signals from sensors, so maybe there are more appropriate technologies which I'm not aware of!&lt;/p&gt;&#xA;" OwnerUserId="9587" LastEditorUserId="9587" LastEditDate="2015-07-06T10:22:37.040" LastActivityDate="2015-07-06T10:22:37.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6348" PostTypeId="2" ParentId="6340" CreationDate="2015-07-06T06:49:46.487" Score="1" Body="&lt;p&gt;Typical predictive performance measures used to compare accuracy of ANN models are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RMSE - (root mean squared error) measures the distance between estimated and actual outcomes. Other metrics that measure the same concept are MSE, MAE or MPE.&lt;/li&gt;&#xA;&lt;li&gt;R square - (R^2 or coefficient of determination) measures the reduction of variance when using the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When comparing two different ANN models for performance, metrics that take into account the complexity of the model may be used, such as AIC or BIC.&lt;/p&gt;&#xA;" OwnerUserId="9986" LastActivityDate="2015-07-06T06:49:46.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6349" PostTypeId="1" AcceptedAnswerId="6412" CreationDate="2015-07-06T09:40:59.947" Score="2" ViewCount="408" Body="&lt;p&gt;I'm using SMO, Logistic Regression, Bayesian Network and Simple CART algorithms for classification. Results form WEKA: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Algorithm               Sensitivity (%)       Specificity (%)         Overall accuracy (%)&#xA;Bayesian Network            57.49                 76.09                    65.24&#xA;Logistic Regression         64.73                 69.86                    66.87&#xA;SMO                         54.32                 79.20                    64.69&#xA;Simple CART                 71.88                 61.51                    67.56&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;SMO gives the best result for my classification problem, since it correctly classify the 79.20% of the class which is important for me. I want to increase this accuracy by stacking. I tried to combine some of them. In most of the cases I couldn't increase the accuracy but stacking SMO with Logistic Regression made a little increment in accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I explain why stacking SMO with Logistic Regression is better than others? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any generalization such as combining tree classifiers gives good result in stacking? What should I care about while stacking? &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;EDIT:&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                                Bayesian Network    Logistic Reg.   SMO         CART &#xA;Kappa statistic                   0.3196             0.3367         0.3158      0.3335 &#xA;Mean absolute error               0.3517             0.4164         0.3531      0.4107 &#xA;Root mean squared error           0.5488             0.4548         0.5942      0.4547 &#xA;Relative absolute error (%)      72.3389              85.65        72.6299      84.477 &#xA;Root relative squared error (%) 111.3076            92.2452       120.5239     92.2318 &#xA;Weighted Avg. of F-Measure        0.653               0.671          0.676     92.2318 &#xA;ROC Area                          0.725               0.727          0.668       0.721&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Total number of instance is 25106. 14641 of them is class a, and 10465 of them belong to class b.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=== Confusion Matrix of Simple CART ===&#xA;     a     b   &amp;lt;-- classified as&#xA; 10524  4117 |     a = 0&#xA;  4028  6437 |     b = 1&#xA;&#xA;=== Confusion Matrix of SMO ===&#xA;&#xA;    a    b   &amp;lt;-- classified as&#xA; 7953 6688 |    a = 0&#xA; 2177 8288 |    b = 1&#xA;&#xA;=== Confusion Matrix of Logistic Regression ===&#xA;&#xA;    a    b   &amp;lt;-- classified as&#xA; 9477 5164 |    a = 0&#xA; 3154 7311 |    b = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since SMO is successful at class b and CART is successful at class a, I tried to ensemble these two algorithms. But I couldn't increase the accuracy. Then I tried to combine SMO with Logistic Regression, the accuracy is increased a little bit. Why ensembling SMO with Logistic Regression is better than ensebling SMO with CART, is there any explanation?&lt;/p&gt;&#xA;" OwnerUserId="9323" LastEditorUserId="9323" LastEditDate="2015-07-10T19:58:53.147" LastActivityDate="2015-07-10T19:58:53.147" Title="What should I care about while stacking as an ensemble method?" Tags="|machine-learning|data-mining|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6350" PostTypeId="1" CreationDate="2015-07-06T10:10:34.600" Score="3" ViewCount="181" Body="&lt;p&gt;After conducting a cluster analysis using K-means, I have new data coming online that I need to detect anomalies with.  Anomalies are assumed to not be within the clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how is one to define &quot;inside a cluster&quot; in K-means?&lt;/p&gt;&#xA;" OwnerUserId="10512" LastEditorUserId="21" LastEditDate="2015-07-08T11:40:55.623" LastActivityDate="2015-08-26T15:24:44.017" Title="Definition of &quot;inside&quot; in K-means?" Tags="|machine-learning|clustering|k-means|anomaly-detection|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6354" PostTypeId="2" ParentId="6350" CreationDate="2015-07-06T13:13:24.633" Score="1" Body="&lt;p&gt;It's inside a cluster if it is part of the partition of the respective Voronoi diagram. This is a visual explanation that translates to &quot;a point is inside cluster A if it is closest to the centroid of A (compared to all other centroids).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your clusters don't have infinite boundaries and outliers shouldn't be in any cluster at all, you might need to refine your approach to something else than k-means that detects outliers.&lt;/p&gt;&#xA;" OwnerUserId="5316" LastActivityDate="2015-07-06T13:13:24.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6355" PostTypeId="1" AcceptedAnswerId="16007" CreationDate="2015-07-06T13:42:12.230" Score="7" ViewCount="1749" Body="&lt;p&gt;The Bayes error rate is a theoretical bound that determines the lowest possible error rate for a classification problem, given some data. I was wondering whether an equivalent concept exists for the case of regression algorithms. My aim is to determine how far my regression algorithm's error is from that theoretical bound, as a way to assess how far am I from the best possible solution. Is there any way to obtain a bound of the lowest regression error for a given dataset?&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-02-01T07:48:03.423" Title="Theoretical bound - regression error" Tags="|regression|linear-regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6356" PostTypeId="2" ParentId="6345" CreationDate="2015-07-06T17:11:49.963" Score="8" Body="&lt;p&gt;If there is no value column - introduce it yourself!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[&quot;value&quot;]=1&#xA;pd.pivot_table(df, values=&quot;value&quot;, index=[&quot;city&quot;], columns=&quot;cuisine&quot;, fill_value=0) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your example I got (after fixing the misprint in 'Japanse' to 'Japanese')&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cuisine  Chinese  French  German  Japanese&#xA;city                                      &#xA;NY             1       0       0         1&#xA;SF             0       1       1         1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-07-06T17:11:49.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6357" PostTypeId="2" ParentId="6350" CreationDate="2015-07-06T20:12:07.840" Score="0" Body="&lt;p&gt;With the Voronoi cells that runDosrun brings, it is true that they are likely unbounded, so your new data that is &quot;outside of the cluster&quot; isn't actually outside the cluster.  You should probably look at a different clustering algorithm if that is indeed what you are looking to do.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are intent on using k-means, you could attempt to classify anomalous data into its own cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are libraries out there that you can use to constrain the boundaries of the k-means during cluster analysis to the initial domain.  Without knowing what environment you are trying to do this in or what the context of the data is, it would be really difficult to define &quot;inside a cluster&quot; with out you specifically choosing certain boundaries.&lt;/p&gt;&#xA;" OwnerUserId="10506" LastActivityDate="2015-07-06T20:12:07.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6358" PostTypeId="2" ParentId="6339" CreationDate="2015-07-06T20:24:27.203" Score="2" Body="&lt;p&gt;Very difficult question for someone to answer as ultimately your examples and many questions contain certain constraints you won't get from your sensors.&#xA;For example:&#xA;Your first question is predicting &quot;optimum&quot; time to turn on AC.  who defines optimum?  Optimum for the classroom at full capcity, half capacity?&#xA;Your second question looks like it one, requires a map of the classroom and where the sensors are, two you won't be able to predict 7 sensor values from 2 sensor values, I don't know why you would want to do this.&#xA;Your third questions is a social sciences problem.  Define &quot;light needed to be comfortable&quot;.  I imagine you would get 150 answers if you had 150 students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your scenario lacks context/comparison.  Let's say you had sensors outside the classroom for your various data points.  Then you could do things like, can I predict the temp inside the room based on my other room and external data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, yes machine learning can be used with your wireless sensors.&lt;/p&gt;&#xA;" OwnerUserId="10506" LastActivityDate="2015-07-06T20:24:27.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6359" PostTypeId="2" ParentId="6325" CreationDate="2015-07-06T20:42:57.077" Score="1" Body="&lt;p&gt;I don't understand the purpose of imposing a condition that requires any cluster to contain at least some (let's say even 1) point from every dataset but at the same time find a solution that also allows that condition to be broken by outliers (I assume these are specific cluster outliers or data set outliers?).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you considered implementing an overlapping cluster algorithm?  You don't get the full distinctness of a traditional single membership but your might better fit your data to an algorithm that achieves the desired condition.  It will probably require some exploration of the data and testing for cluster membership after the fact.&lt;/p&gt;&#xA;" OwnerUserId="10506" LastActivityDate="2015-07-06T20:42:57.077" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6361" PostTypeId="1" CreationDate="2015-07-06T21:16:52.867" Score="1" ViewCount="86" Body="&lt;p&gt;I would really appreciate it if someone could tell me where I would start in tackling the following tasks in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not a data-science expert and am trying to teach myself data-analysis from a background in physics. I have introductory statistics and have just finished Udacity's course in R programming and exploratory data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tasks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 )Simulate customer walk-ins for a given period. The simulations should take into account peak volumes at different times of the days and differences in week days and weekends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Simulate customer purchases based on the customer walk-ins. Make assumptions on the average dollar price and variations for different times of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="10510" LastActivityDate="2015-07-06T21:26:01.297" Title="How to simulate customer walk-ins for a given period in a fast food chain in R" Tags="|r|statistics|simulation|" AnswerCount="1" CommentCount="4" ClosedDate="2015-07-06T21:27:38.337" ContentLicense="CC BY-SA 3.0" />
  <row Id="6363" PostTypeId="2" ParentId="6361" CreationDate="2015-07-06T21:26:01.297" Score="0" Body="&lt;p&gt;You need either an initial data set or a large set of constraints to &quot;fake&quot; a data set.  if you have an initial data set you could make somewhat valid assumptions regarding traffic flow, purchasing behaviors and the likes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, you are going to need to research these different metrics from existing companies that are like what you want data for and then I assume put together a little program with those metrics as constraints and simulate your data.  Once you have those constraints you could use simple averaging or weighted options for developing a constraint per variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure they are semi accurate.  For example, if you were modeling company growth for a &quot;really good idea&quot; research growth trends for like companies and for companies you want to emulate and then scale BACK from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also always purchase a dataset:&#xA;&lt;a href=&quot;http://datamarket.azure.com/browse/data&quot; rel=&quot;nofollow&quot;&gt;http://datamarket.azure.com/browse/data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10506" LastActivityDate="2015-07-06T21:26:01.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6364" PostTypeId="2" ParentId="6350" CreationDate="2015-07-06T22:37:50.287" Score="3" Body="&lt;p&gt;Strictly speaking, the k-means algorithm does not have a definition for &quot;inside the cluster&quot; and is therefore not a great candidate for anomaly detection.  In k-means, every point is assigned to one of k clusters and then a new cluster centroid is calculated.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as previous uses have pointed out, you could construct some sort of ad-hoc system where you process a set of data and then define new data as anomalous when its extends beyond 2 standard deviations of the centroid location. DON'T DO THIS! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-Means will not work well for this ad-hoc method.  If k is poorly chosen, then the distribution within a cluster will not be normally distributed.  You very, very frequently see natural distributions of points which are split between two clusters.  For instance, take a look at the ad-hoc segmentation of points in this location data for a cell phone user's location over a month:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/2vLSn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2vLSn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you use another clustering method.  The first option that comes to mind is DBSCAN.  This allows one to set a threshold for noise and the cluster numbers are not set a-priori.  DBSCAN is therefor much more likely to return normal distributions within a cluster.  Here is a single DBSCAN cluster of the same data:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/w6kJm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w6kJm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I'll point out that the method you are proposing is not as good as other novelty and anomaly detection methods.  You should consider doing novelty detection using a single (or possibly even multiple) class support vector machine (SVM) with a nonlinear kernel. The nonlinear kernel will allow you to recover multiple &quot;clusters&quot; while the SVM will do much better at predicting which points are inside the class.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-26T15:24:44.017" LastActivityDate="2015-08-26T15:24:44.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6365" PostTypeId="1" CreationDate="2015-07-07T00:21:53.520" Score="4" ViewCount="3139" Body="&lt;p&gt;I'm trying to wrap my head around &lt;a href=&quot;http://orange.biolab.si/docs/latest/reference/rst/Orange.associate.html&quot; rel=&quot;nofollow&quot;&gt;Association rules and frequent itemsets&lt;/a&gt;.&#xA;So I threw my data in, instead of the samples one and sometime it works, sometimes it doesn't.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rules = Orange.associate.AssociationRulesSparseInducer(data, support = 0.3)&#xA;&#xA;print &quot;%5s   %5s&quot; % (&quot;supp&quot;, &quot;conf&quot;)&#xA;for r in rules:&#xA;    print &quot;%5.3f   %5.3f   %s&quot; % (r.support, r.confidence, r)&#xA;&#xA;inducer = Orange.associate.AssociationRulesSparseInducer(support = 0.2, store_examples = True)&#xA;itemsets = inducer.get_itemsets(data)&#xA;print itemsets&#xA;print data.domain&#xA;print [data.domain[i].name for i in itemsets[4][0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More often than not, itemsets[4][0] shows an &lt;code&gt;IndexError: list index out of range&lt;/code&gt; error.&#xA;So I start playing around with support = 0.3, support = 0.5, support = 0.2 and itemsets[2][0] or itemsets[3][0].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the docs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;support&lt;br&gt;&#xA;  Minimal support for the rule. Depending on the data set it should be set to sufficiently high value to avoid running out of working memory (default: 0.3).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;True - I tried 0.2 and it quickly blasted my memory on a 800 rows data file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea what I should do best there or which are viable values for a shopping cart analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;800 rows of data (800 orders)&lt;br&gt;&#xA;1 to x item(-categories) per order&lt;br&gt;&#xA;15 different item-categories in the file, so my data looks like:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ItemCat1&#xA;ItemCat2, ItemCat2, ItemCat2, ItemCat2, ItemCat7, ItemCat7, ItemCat7,     ItemCat7, ItemCat7&#xA;ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat2&#xA;ItemCat4, ItemCat4&#xA;ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat1, ItemCat2&#xA;ItemCat5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10513" LastEditorUserId="15527" LastEditDate="2016-01-23T16:31:56.920" LastActivityDate="2016-06-28T17:01:53.030" Title="Question about (Python/Orange) Apriori associative algorithm" Tags="|data-mining|python|orange|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6366" PostTypeId="2" ParentId="6139" CreationDate="2015-07-07T01:15:04.840" Score="0" Body="&lt;p&gt;If the size of your logs are still growing then a distributed data system is definitely the right way to go. I have been using &lt;a href=&quot;http://mesos.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mesos&lt;/a&gt; in production for almost a year now and it solved the problem of &lt;code&gt;if this processing happens on some server it under or over utilizes resources&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I probably would look into some stacks like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mesos as your fault-tolerant and elastic distributed systems &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Spark&lt;/a&gt; or some Hadoop-based solutions for log processing and store the output in a DFS like HDFS&lt;/li&gt;&#xA;&lt;li&gt;Have your applications consume data stored in HDFS as final steps&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="708" LastActivityDate="2015-07-07T01:15:04.840" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6367" PostTypeId="1" CreationDate="2015-07-07T03:23:12.333" Score="0" ViewCount="54" Body="&lt;p&gt;I've just embarked on a project to recommend similar candidates to employers for a website that offers employers the ability to search its CV database. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My background is in engineering research on biomedical images - from simple predefined feature extraction to CNNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm about to start with a simple bag of words approach standardised by the frequency of appearance and see where that gets me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone done anything similar before and has suggestions for which algorithms might give good results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is very well structured, separated into seperate sections for each previous employment etc.&lt;/p&gt;&#xA;" OwnerUserId="10515" LastActivityDate="2019-04-18T11:37:36.573" Title="Similar Candidate Recommendations" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6368" PostTypeId="2" ParentId="6318" CreationDate="2015-07-07T03:46:55.903" Score="1" Body="&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://www.oreilly.com/data/free/data-science-in-the-cloud.csp&quot; rel=&quot;nofollow&quot;&gt;Data Science in the Cloud with Microsoft Azure Machine Learning and R&lt;/a&gt;&lt;/em&gt; is a free textbook which works through an example in great detail. Don't be put off by the particular tools used as you don't need them to get some benefit out of the book. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another one which I enjoyed is &lt;a href=&quot;http://shop.oreilly.com/product/9780596529321.do&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Programming Collective Intelligence&lt;/em&gt;&lt;/a&gt; which also goes through a number of projects in detail, including the web scraping part which most books gloss over.&lt;/p&gt;&#xA;" OwnerUserId="10516" LastActivityDate="2015-07-07T03:46:55.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6369" PostTypeId="2" ParentId="6349" CreationDate="2015-07-07T06:50:32.313" Score="0" Body="&lt;p&gt;Read the following by MLWave:&#xA;&lt;a href=&quot;http://mlwave.com/kaggle-ensembling-guide/&quot; rel=&quot;nofollow&quot;&gt;http://mlwave.com/kaggle-ensembling-guide/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very good starting point to stacking / ensembles.&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-07-07T06:50:32.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6370" PostTypeId="1" CreationDate="2015-07-07T09:23:45.250" Score="1" ViewCount="575" Body="&lt;p&gt;Let's say I have one Text File (Size 1 GB). I want to search particular word from file and if it is found the Line number should be returned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can Execute my java program using command line in linux. But what I want is some Interface using REST API. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I submit that word in GUI(My Interface) and submit request it should be able to call HDFS , commands should be provided by itself and it should return me the result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How Can you approach this problem ? &#xA;(I have very primitive knowledge about REST API)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Referred &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#ParameterDictionary[]&quot; rel=&quot;nofollow&quot;&gt;https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#ParameterDictionary[]&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=7YcW25PHnAA&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=7YcW25PHnAA&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10210" LastActivityDate="2015-07-07T09:23:45.250" Title="How to use REST API to execute Map-Reduce Task?" Tags="|apache-hadoop|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6372" PostTypeId="2" ParentId="6367" CreationDate="2015-07-07T12:37:14.897" Score="1" Body="&lt;p&gt;Try content based filtering, you can cluster candidates based on similarity of keywords and by assigning similarity scores. Check tf-idf/cosine similarity measures. Additionally you can customize your algorithm by giving extra weight for section wise similarities since your data is very well structured.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check collaborative filtering on &lt;a href=&quot;https://mahout.apache.org/users/recommender/recommender-documentation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mahout&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="29575" LastEditDate="2019-04-18T11:37:36.573" LastActivityDate="2019-04-18T11:37:36.573" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6374" PostTypeId="2" ParentId="6365" CreationDate="2015-07-07T14:45:11.840" Score="-1" Body="&lt;p&gt;It is not available in Orange, but i would try FP-Growth. Its way faster in combination with low support values than apriori!&lt;/p&gt;&#xA;" OwnerUserId="10521" LastActivityDate="2015-07-07T14:45:11.840" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6375" PostTypeId="1" CreationDate="2015-07-07T15:33:41.323" Score="1" ViewCount="48" Body="&lt;p&gt;I managed to create an emotion recognition system that uses dense optical flow on each entire frame. While the accuracy range is within 80-90% with cross-validation, I am aiming to improve the accuracy of the program. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are four emotions: Neutral, happy, surprised, and angry. So far my classifier works pretty well, though it tends to over guess 'neutral' when  the answer is 'happy' or 'surprised'. This tends to happen when the mouth is only slightly opened, but the subject is visible smiling or have mouth opened in shock while the classifier still thinks the mouth is closed.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Confusion Matrix for Dense Optical Flow:   &#xA;[[27 22  0  0]  &#xA; [ 0 57  1  0]  &#xA; [ 0 12 60  0]  &#xA; [ 0  9  3 68]]  &#xA;Accuracy: 80-90% range&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is something I want to try in order to solve this though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the ability to get the position of facial landmarks, though I don't know what I can do to turn this information into an effective additional feature I can use that would increase the accuracy. I was thinking of just simply getting the face landmark coordinates at the end of each video, but I feel like that it would not be the solution to differentiate between a closed mouth and slightly opened one (the difference in coordinate values will be small I think and am guessing that machine learning won't notice the difference).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I considered the possibility of just simply taking a still image of the subject's mouth and just analyzing that, but rejected it as it vulnerable to factors like lighting and people's appearance, and inconsistent matrix sizes. Plus I want my additional feature to take advantage of facial movement tracking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a smart way to implement facial landmark tracking into a feature that would increase the accuracy of my classifier by dealing with my classifier's tendency to over predict the emotion 'neutral'. Any ways I can accomplish that?&lt;/p&gt;&#xA;" OwnerUserId="10523" LastActivityDate="2015-07-07T15:33:41.323" Title="Trying to come up with a feature to improve emotion classifier based on facial movement using facial landmarks" Tags="|machine-learning|classification|feature-selection|confusion-matrix|computer-vision|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6376" PostTypeId="2" ParentId="6350" CreationDate="2015-07-07T16:06:07.573" Score="0" Body="&lt;p&gt;I'm going to agree with AN6U5 in general. However, in practice what I've done in the past is used k-means with cross validation and and Silhouetting to find a 'good' clustering of historical data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I then did was take the clusters and calculated their covariance matrix so I could use the Mahalanobis distance. This allows me to take into account the actual shape of the distribution along each dimension in determining whether a point lies to far outside my previous examples. Since the Mahalanobis distance is in standard deviations, if its say 3 sigma outside each cluster its usually flagged for further study.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to get away with this since I know the general process for whats generating my data from testing and I have a rather large amount of data for known operating conditions, so I have pretty good faith in my historical data clusters.&lt;/p&gt;&#xA;" OwnerUserId="10524" LastActivityDate="2015-07-07T16:06:07.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6377" PostTypeId="2" ParentId="6342" CreationDate="2015-07-07T16:10:09.023" Score="1" Body="&lt;p&gt;I'm not sure I'm following your question completely, but for spatial type clustering maybe a Self-Organizing Map would meet your needs. Basically it maps out the space based on nearby activations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side answer: I haven't used python&lt;/p&gt;&#xA;" OwnerUserId="10524" LastActivityDate="2015-07-07T16:10:09.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6378" PostTypeId="1" AcceptedAnswerId="11527" CreationDate="2015-07-07T16:26:18.230" Score="4" ViewCount="1316" Body="&lt;p&gt;I am learning the multiple linear regression model.&#xA;I've built a &lt;code&gt;model&lt;/code&gt; and using R command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got this result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 253.2 on 44 degrees of freedom Multiple&#xA;R-squared:  0.3336, Adjusted R-squared:  0.2579  F-statistic: 4.405 on&#xA;5 and 44 DF,  p-value: 0.002444&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I interpret this result in order to have a decision regarding the goodness of the model? In specific, what is the &lt;code&gt;44 degrees of freedom&lt;/code&gt; means for this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, why do we have adjusted and multiple r squared parameters?&lt;/p&gt;&#xA;" OwnerUserId="4968" LastEditorUserId="18328" LastEditDate="2016-05-01T17:41:28.090" LastActivityDate="2016-05-01T17:41:28.090" Title="Interpreting the evaluation result of multiple linear regression" Tags="|r|regression|linear-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6380" PostTypeId="1" AcceptedAnswerId="6430" CreationDate="2015-07-07T18:05:23.903" Score="11" ViewCount="28568" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I want to avoid overfitting in random forest. In this regard, I intend to use mtry, nodesize, and maxnodes etc. Could you please help me choose values for these parameters? I am using R.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, if possible, please tell me how I can use k-fold cross validation for random forest (in R).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="29169" LastEditDate="2019-06-08T03:19:20.417" LastActivityDate="2020-08-16T16:54:10.717" Title="How to avoid overfitting in random forest?" Tags="|machine-learning|data-mining|r|predictive-modeling|random-forest|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6381" PostTypeId="2" ParentId="1128" CreationDate="2015-07-07T18:07:39.507" Score="2" Body="&lt;p&gt;It seems like you are overthinking this and getting caught up in your own head a little.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can define a restricted set of time series as those which are well behaved (non-asymptotic) and single valued.  These time series then form a time series algebra which have additive, multiplicative, commutative, and associative properties.  There is an identity element (straight line with value 1) and a zero element (straight line with value 0).  We even have a well defined inverse member for all time series not crossing zero.  I believe this even forms an Abelian group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, previous posters are correct that this group can be perfectly represented by various orthogonal vector spaces such as polynomials and Fourier space.  This is true whether we are talking about Lagrange polynomials, Fourier series, or Gauss-Lobatto-Legendre polynomials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point then becomes... where does this get you? You got caught up in the suggestion to employ Fourier series in that you didn't want to remain in frequency space, but this is entirely unnecessary.  You can, for instance, Fourier transform into frequency space, apply a high frequency filter, and then transform back into the time domain.  This is incredibly effective in removing noise (although band pass filters tend to work better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stepping back from the question a little, it seems like you might really be asking:  Given typical time series and typical time series operations (smoothing, de-seasoning, de-trending, etc) do the time series and their typical operations constitute a group with a well defined algebra?  The answer is, yes.  All of these operations just involve finding other time series which can be added, subtracted, multiplied, or divided by the original time series in order to yield a more physically tractable (interpretable) representation.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-07T18:07:39.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6384" PostTypeId="1" AcceptedAnswerId="6390" CreationDate="2015-07-07T23:15:34.657" Score="5" ViewCount="7553" Body="&lt;p&gt;I'm trying to understand PCA, but I don't have a machine learning background.  I come from software engineering, but the literature I've tried to read so far is hard for me to digest.&lt;/p&gt;&#xA;&lt;p&gt;As far as I understand PCA, it will take a set of datapoints from an N dimensional space and translate them to an M dimensional space, where N &amp;gt; M.  &lt;strong&gt;I don't yet understand what the actual output of PCA is.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;For example, take this 5 dimensional input data with values in the range [0,10):&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;// dimensions:&#xA;// a  b  c  d  e&#xA;&#xA;[[ 4, 1, 2, 8, 8],      // component 1&#xA; [ 3, 0, 2, 9, 8],&#xA; [ 4, 0, 0, 9, 1],&#xA; ...&#xA; [ 7, 9, 1, 2, 3],      // component 2&#xA; [ 9, 9, 0, 2, 7],&#xA; [ 7, 8, 1, 0, 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;My assumption is that PCA could be used to reduce the data from 5 dimensions to, say, 1 dimension.&lt;/p&gt;&#xA;&lt;h3&gt;Data details:&lt;/h3&gt;&#xA;&lt;p&gt;There are two &amp;quot;components&amp;quot; in the data.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One component has mid &lt;code&gt;a&lt;/code&gt; levels, low &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; levels, high &lt;code&gt;d&lt;/code&gt;, and nondeterministic &lt;code&gt;e&lt;/code&gt; levels.&lt;/li&gt;&#xA;&lt;li&gt;The other component has high &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; levels, low &lt;code&gt;c&lt;/code&gt; and &lt;code&gt;d&lt;/code&gt; levels, and nondeterministic &lt;code&gt;e&lt;/code&gt; levels.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;This means that the two components are most differentiated by &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;d&lt;/code&gt;, somewhat differentiated by &lt;code&gt;a&lt;/code&gt;, and negligibly differentiated by &lt;code&gt;c&lt;/code&gt; and &lt;code&gt;e&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3&gt;Outputs?&lt;/h3&gt;&#xA;&lt;p&gt;&lt;em&gt;I'm making this up&lt;/em&gt;, but &lt;em&gt;say&lt;/em&gt; the (non-normalized) linear combination with the highest differentiating power is something like&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;5*a + 10*b + 0*c + 10*d + 0*e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The above input data translated along that single axis is:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[[110],&#xA; [105],&#xA; [110],&#xA; ...etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Is that linear combination (or a vector describing it) the output of PCA?  Or is the output the actual reduced dataset?  Or something else entirely?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="10531" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-07-08T16:22:11.793" Title="What is the actual output of Principal Component Analysis?" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6385" PostTypeId="2" ParentId="6384" CreationDate="2015-07-08T00:50:49.643" Score="4" Body="&lt;p&gt;Have you tried reading the &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis#Intuition&quot; rel=&quot;nofollow&quot;&gt;Intuition&lt;/a&gt; section of the PCA page on wiki? Also, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis#Further_considerations&quot; rel=&quot;nofollow&quot;&gt;Further Considerations&lt;/a&gt; section, I think, explains what the various components represent and addresses your questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A short summary of what those sections describe is that the original components (i.e., axes) of the data do not properly represent the relationship inherit in the data. Instead, the components can be combined to yield a new set of components, or axes, that better describe how the data is distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wiki link states that &quot;PCA can be thought of as fitting an n-dimensional ellipsoid to the data&quot;. In that line of thinking, the output of PCA is the eigenvectors and eigenvalues that define the orientation and length of the ellipsoid that best fits the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you do with that ellipsoid is up to you and your application of PCA.&lt;/p&gt;&#xA;" OwnerUserId="2728" LastActivityDate="2015-07-08T00:50:49.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6386" PostTypeId="2" ParentId="253" CreationDate="2015-07-08T05:29:27.883" Score="1" Body="&lt;p&gt;Data Science is a field demanding a variety of skills. Having knowledge of Hadoop is one of them. The main tasks of a Data Scientist include:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Gathering data from different resources.&lt;/li&gt;&#xA;&lt;li&gt;Cleaning and pre-processing the data.&lt;/li&gt;&#xA;&lt;li&gt;Studying statistical properties of the data.&lt;/li&gt;&#xA;&lt;li&gt;Using Machine Learning techniques to do forecasting and derive insights from the data.&lt;/li&gt;&#xA;&lt;li&gt;Communicating the results to decision makers in an easy to understand way.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Out of the above points knowledge of Hadoop is useful for points 1,2 and 3, but you also need to have strong mathematical/statistical background and strong knowledge of Computational techniques to work in data science field. Also Hadoop is not the only framework that is being used in Data Science. Big Data ecosystem has a range of frameworks, each specific to a particular use case. This article gives introductory material regarding major Big Data frameworks that could be used in Data Science:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.codophile.com/big-data-frameworks-every-programmer-should-know/&quot; rel=&quot;nofollow&quot;&gt;http://www.codophile.com/big-data-frameworks-every-programmer-should-know/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6534" LastActivityDate="2015-07-08T05:29:27.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6387" PostTypeId="1" CreationDate="2015-07-08T07:35:31.750" Score="2" ViewCount="525" Body="&lt;p&gt;I am new to NLP, I found a format named ConLL which seems a tab-separated file,&#xA;like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I found a dependency Treebank in this format for Persian. I would like to do some experiment on it. I am familiar with C# and C# has great libraries for XML document files. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just wonder why such a corpus is not in XML format! I don't say a sophisticated one but they could put each line in a node of XML. Then at least, I would know what is the tag for each element.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I can convert it to XML.&lt;/p&gt;&#xA;" OwnerUserId="3436" LastActivityDate="2015-07-08T07:51:17.783" Title="Why ConLL is not in XML format" Tags="|nlp|data-formats|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6388" PostTypeId="2" ParentId="6387" CreationDate="2015-07-08T07:51:00.077" Score="1" Body="&lt;p&gt;One reason for sure is, that you can easialy open it in spreadsheet viewer.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2015-07-08T07:51:00.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6389" PostTypeId="2" ParentId="6387" CreationDate="2015-07-08T07:51:17.783" Score="1" Body="&lt;p&gt;ConLL is a format optimised for processing efficiency (both speed and memory usage).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XML formats for NLP often have the annotation layers one after the other; therefore you have to hold the whole file in memory for processing it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A drawback of ConLL is that you have to put in fillers (e.g., &lt;code&gt;0&lt;/code&gt;) for sparse annotations.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-07-08T07:51:17.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6390" PostTypeId="2" ParentId="6384" CreationDate="2015-07-08T08:47:16.220" Score="5" Body="&lt;p&gt;I agree with dpmcmlxxvi's answer that the common &quot;output&quot; of PCA is computing and finding the eigenvectors for the principal components and the eigenvalues for the variances, but I can't add comments yet and would still like to contribute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you hit this step of calculating the eigenvectors and eigenvalues of the principal components, you can do many types of analyses depending on your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe the &quot;output&quot; you are specifically asking about in your question is the resultant data set of applying a transformation or projection of the original data set into the desired linear subspace (of n-dimensions). This is taking the output of PCA and applying it on your original data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://sebastianraschka.com/Articles/2014_pca_step_by_step.html&quot; rel=&quot;nofollow&quot; title=&quot;PCA step by step example&quot;&gt;PCA step by step example&lt;/a&gt; may help. The ultimate output of this 6 step analysis was the projection of a 3 dimensional data set into 2 dimensions. Here are the high level steps:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Taking the whole dataset ignoring the class labels&lt;/li&gt;&#xA;  &lt;li&gt;Compute the d-dimensional mean vector&lt;/li&gt;&#xA;  &lt;li&gt;Computing the scatter matrix (alternatively, the covariance matrix)&lt;/li&gt;&#xA;  &lt;li&gt;Computing eigenvectors and corresponding eigenvalues&lt;/li&gt;&#xA;  &lt;li&gt;Ranking and choosing k eigenvectors&lt;/li&gt;&#xA;  &lt;li&gt;Transforming the samples onto the new subspace&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ultimately, step 4 is the &quot;output&quot; since that is where the common requirements for performing PCA are fulfilled. We can make different decisions at steps 5 and 6 and produce alternative output there.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few more possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could decide to project the observations &lt;em&gt;with outliers removed&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Another possible outcome here would be to calculate the proportion of&#xA;variance explained by one or any combination of principal components. For example, the proportion of variance explained by the first two principal components of K components is &lt;code&gt;(λ1+λ2)/(λ1+λ2+. . .+λK)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;After plotting the projected observations into the first two principal components (as in the given example), you can impose a plot of the loadings of each of the original dimensions into the subspace (scaled by the standard deviation of the principal components). This way, we can see the contribution of the original dimensions (in your case a - e) to principal component 1 and 2. The biplot is another common product of PCA.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10536" LastEditorUserId="10536" LastEditDate="2015-07-08T16:22:11.793" LastActivityDate="2015-07-08T16:22:11.793" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6391" PostTypeId="1" AcceptedAnswerId="9993" CreationDate="2015-07-08T12:05:49.663" Score="9" ViewCount="9344" Body="&lt;p&gt;I know there are similar question on stats.SE, but I didn't find one that fulfills my request; please, before mark the question as a duplicate, ping me in the comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run a neural network based on &lt;code&gt;neuralnet&lt;/code&gt; to forecast SP500 index time series and I want to understand how I can interpret the plot posted below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/EFQwW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly, I'm interested to understand what is the interpretation of the hidden layer weight and the input weight; could someone explain me how to interpret that number, please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9225" LastEditorUserId="8953" LastEditDate="2015-07-08T18:28:23.360" LastActivityDate="2018-10-31T19:29:49.427" Title="R - Interpreting neural networks plot" Tags="|machine-learning|r|neural-network|predictive-modeling|forecast|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6393" PostTypeId="1" CreationDate="2015-07-08T15:22:09.047" Score="2" ViewCount="890" Body="&lt;p&gt;Can someone please recommend me some Python packages which I can use for missing word prediction in a paragraph of text. The missing words are a part of existing semantic list of words.( List of missing words are a common set of words like was ,were etc). If there are better packages in R which can help me with this  it would be very useful if you post it here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="10532" LastActivityDate="2020-08-03T08:31:00.747" Title="Modules on Python which are useful for missing Word/Letter prediction in text paragraphs from a coprpus" Tags="|data-mining|python|text-mining|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6394" PostTypeId="1" CreationDate="2015-07-08T18:58:25.897" Score="1" ViewCount="789" Body="&lt;p&gt;I am setting up for a couple self study projects to explore machine learning techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I will be creating a bunch of calendar and weather features for the data, then trying to forecast using a variety of machine learning techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2nd is some 13 million rows of text data (several paragraphs for each row)  for classification. (currently in solr database)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My compute rig is 6-core, 32g ram, gforce GPU. I plan to install Ubuntu 14.2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect to be using python for file processing, scilearn, pylearn2 and word2vec for general exploration and training. R for getting a taste of the language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly data set 1 will require joining weather and calendar data to date/time and aggregation across time and location. I know how to stuff it all into a MySQL database and do the aggregations and joins there, but I have been reading about spark and wondering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;......&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I take the time to simulate a cluster using virtual box/hadoop/spark (for my learning experience, not performance), can/should I do the aggregations there and write the results to the distributed data store? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since deep learning can not be run on spark, does that mean I would need to copy the aggregated data back out to the local file system to use some of those techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For data set 2, I want to run the word2vec algorithm as found in the kaggle tutorial &lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors&lt;/a&gt;. In that example, that is a deeplearning method, so I should just leave the data in the solr.. right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general I am looking for appropriate applications and insight into data flow from app to app to help me get to the part where I start trying various ML techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for helping me along&lt;/p&gt;&#xA;" OwnerUserId="10545" LastActivityDate="2015-08-15T20:46:37.993" Title="any reason for this project to use hadoop/spark?" Tags="|data-cleaning|databases|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6395" PostTypeId="1" AcceptedAnswerId="13456" CreationDate="2015-07-08T20:04:16.703" Score="10" ViewCount="464" Body="&lt;p&gt;It seems standard in many neural network packages to pair up the objective  function to be minimised with the activation function in the output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, for a linear output layer used for regression it is standard (and often only choice) to have a squared error objective function. Another usual pairing is logistic output and log loss (or cross-entropy). And yet another is softmax and multi log loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using notation, $z$ for pre-activation value (sum of weights times activations from previous layer), $a$ for activation, $y$ for ground truth used for training, $i$ for index of output neuron.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Linear activation $a_i=z_i$ goes with squared error $\frac{1}{2} \sum\limits_{\forall i} (y_i-a_i)^2$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Sigmoid activation $a_i = \frac{1}{1+e^{-z_i}}$ goes with logloss/cross-entropy objective $-\sum\limits_{\forall i} (y_i*log(a_i) + (1-y_i)*log(1-a_i))$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Softmax activation $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ goes with multiclass logloss objective $-\sum\limits_{\forall i} (y_i*log(a_i))$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those are the ones I know, and I expect there are many that I still haven't heard of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that log loss would only work and be numerically stable when the output and targets are in range [0,1]. So it may not make sense to try linear output layer with a logloss objective function. Unless there is a more general logloss function that can cope with values of $y$ that are outside of the range?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it doesn't seem quite so bad to try sigmoid output with a squared error objective. It should be stable and converge at least.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that some of the design behind these pairings is that it makes the formula for $\frac{\delta E}{\delta z}$ - where $E$ is the value of the objective function - easy for back propagation. But it should still be possible to find that derivative using other pairings. Also, there are many other activation functions that are not commonly seen in output layers, but feasibly could be, such as &lt;code&gt;tanh&lt;/code&gt;, and where it is not clear what objective function could be applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any situations when designing the architecture of a neural network, that you would or should use &quot;non-standard&quot; pairings of output activation and objective functions?&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-07-10T18:03:47.263" LastActivityDate="2016-08-16T13:05:06.970" Title="How flexible is the link between objective function and output layer activation function?" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6396" PostTypeId="2" ParentId="6031" CreationDate="2015-07-08T20:18:27.417" Score="3" Body="&lt;p&gt;It sounds like you're describing optimal interpolation (AKA Gauss-Markov analysis, objective analysis, probably others too). &lt;a href=&quot;http://modb.oce.ulg.ac.be/wiki/upload/diva_intro.pdf&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is a solid intro, and &lt;a href=&quot;http://www.atmos.berkeley.edu/~inez/MSRI-NCAR_CarbonDA/lectures/Kalnay2_CH5OptimalInterpolation.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; is a powerpoint on the subject. It's a hard process to summarize quickly, but roughly speaking, you're on the right track.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optimal interpolation is fairly common in meteorology and other environmental sciences, but I'm not convinced it's really the best tool for the job. Any statisticians want to take a crack at it?&lt;/p&gt;&#xA;" OwnerUserId="10427" LastActivityDate="2015-07-08T20:18:27.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6398" PostTypeId="1" CreationDate="2015-07-08T20:57:09.627" Score="2" ViewCount="58" Body="&lt;p&gt;I'm trying to learn Linear Regression and I'd like someone to look at my notebook to see if I miss out anything at all. How how else I can tune my model better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my notebook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get the dataset from here which is really simple. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/slr/frames/frame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/slr/frames/frame.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%matplotlib inline&#xA;import pandas as pd&#xA;import numpy as np&#xA;from scipy import stats&#xA;import matplotlib.pyplot as plt&#xA;&#xA;df = pd.read_csv('slr12.csv', names=['annual', 'cost'], header=0)&#xA;df.describe()&#xA;&#xA;df.plot(kind='scatter', x='X', y='Y');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/jApbl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;slope, intercept, r_value, p_value, std_err = stats.linregress(df['X'], df['Y'])&#xA;&#xA;plt.plot(df['X'], df['Y'], 'o', label='Original data', markersize=2)&#xA;plt.plot(df['X'], slope*df['X'] + intercept, 'r', label='Fitted line')&#xA;plt.legend()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/0sBy7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So from this trend we can predict that if you annual fee is high then you need your startup cost will be high as well.&lt;/p&gt;&#xA;" OwnerUserId="10549" LastActivityDate="2015-07-08T20:57:09.627" Title="Can I conclude my finding with just one linear regression result?" Tags="|python|pandas|linear-regression|" AnswerCount="0" CommentCount="1" ClosedDate="2015-07-09T05:06:11.923" ContentLicense="CC BY-SA 3.0" />
  <row Id="6399" PostTypeId="2" ParentId="694" CreationDate="2015-07-09T00:15:00.987" Score="9" Body="&lt;p&gt;I like &lt;a href=&quot;http://blocks.readthedocs.org/&quot; rel=&quot;noreferrer&quot;&gt;Blocks&lt;/a&gt;, which is also built on top of Theano. Way more approachable than PyLearn2, and more feature rich than Lasagne. Neatly written, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Updated Jan 2016:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the time of writing, &lt;a href=&quot;http://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt; has by far the most momentum. It is highly modular and can run on both Theano and Tensorflow, giving it great opportunities.&lt;/p&gt;&#xA;" OwnerUserId="554" LastEditorUserId="554" LastEditDate="2016-01-08T18:58:56.573" LastActivityDate="2016-01-08T18:58:56.573" CommentCount="2" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="6400" PostTypeId="2" ParentId="6393" CreationDate="2015-07-09T00:59:20.503" Score="1" Body="&lt;p&gt;This problem seems to be exactly (or at least very similar to) that in the &lt;a href=&quot;https://www.kaggle.com/c/billion-word-imputation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Billion Word Imputation&lt;/a&gt; challenge at Kaggle. Some of the approaches discussed in the forums might help you out. My initial thoughts for useful tools/methods would be word2vec, the stanford NLP tools and sklearn.&lt;/p&gt;&#xA;" OwnerUserId="8865" LastEditorUserId="98307" LastEditDate="2020-08-03T08:31:00.747" LastActivityDate="2020-08-03T08:31:00.747" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6401" PostTypeId="2" ParentId="6189" CreationDate="2015-07-09T01:42:27.827" Score="1" Body="&lt;p&gt;Spark does have a pretty good Python API, check out this &lt;a href=&quot;https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For traditional Hadoop stack, take a look at &lt;a href=&quot;https://github.com/Yelp/mrjob&quot; rel=&quot;nofollow&quot;&gt;mrjob&lt;/a&gt;, it lets you write MapReduce jobs in Python and run them on several platforms.&lt;/p&gt;&#xA;" OwnerUserId="708" LastActivityDate="2015-07-09T01:42:27.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6403" PostTypeId="2" ParentId="5302" CreationDate="2015-07-09T02:21:55.417" Score="1" Body="&lt;p&gt;Hey is Python or other tools an option here? Since you mentioned it is a large dataset, you might want to iterate over it instead of loading all of them at once. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a solution in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from collections import defaultdict&#xA;&#xA;&#xA;inputs = [&#xA;'cat1/: Topic1_Variable1',&#xA;'cat2/: Topic1_Variable2',&#xA;'cat3/: Topic1_Variable3',&#xA;'cat4/: Topic1_Variable4',&#xA;'cat1/: Topic2_Variable1',&#xA;'cat2/: Topic2_Variable2',&#xA;'cat3/: Topic2_Variable3',&#xA;'cat4/: Topic2_Variable4',&#xA;'cat1/: Topic3_Variable1',&#xA;'cat2/: Topic3_Variable2',&#xA;'cat3/: Topic3_Variable3',&#xA;'cat4/: Topic3_Variable4',]&#xA;&#xA;&#xA;outputs = defaultdict(list)&#xA;&#xA;for item in inputs:&#xA;    cat, topic = item.split('/: ')&#xA;    outputs[cat].append(topic)&#xA;&#xA;print pd.DataFrame(outputs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               cat1              cat2              cat3              cat4&#xA;0  Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4&#xA;1  Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4&#xA;2  Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="708" LastActivityDate="2015-07-09T02:21:55.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6404" PostTypeId="1" CreationDate="2015-07-09T02:29:10.683" Score="2" ViewCount="54" Body="&lt;p&gt;I have a problem with an extremely large dataset (who doesn't?) which is stored in chunks such that there is low variance across chunks (i.e., the chunks are &lt;em&gt;sort of&lt;/em&gt; representative). I wanted to play around with algorithms to do some classification in an asynchronous fashion but I wanted to code it up myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample code would look like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;start a master&#xA;distribute 10 chunks on 10 slaves&#xA;while some criterion is not met &#xA; for each s in slave:&#xA;  classify the data inexactly using some kind of iterative algorithm and return to master&#xA; master waits for any 2 slaves to report the classifier, averages the classifier and sends it back for the slaves to continue &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What framework do I use? Hadoop, Spark, Other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I was doing this in pure-C, I would use pthreads and have a very fine control over threads, locks and mutexes. Is there any analogous framework in this distributed data-science environment?&lt;/p&gt;&#xA;" OwnerUserId="10559" LastEditorUserId="10559" LastEditDate="2015-07-09T02:38:22.633" LastActivityDate="2023-03-03T12:05:05.603" Title="What Framework To Use for Asynchronous Algorithms?" Tags="|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6405" PostTypeId="1" CreationDate="2015-07-09T03:10:08.480" Score="2" ViewCount="123" Body="&lt;p&gt;I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. &lt;a href=&quot;https://i.stack.imgur.com/Jl3Q7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; are the steps I'm trying to follow.&lt;/p&gt;&#xA;" OwnerUserId="10560" LastEditorUserId="10560" LastEditDate="2015-07-09T12:49:43.920" LastActivityDate="2015-08-08T13:53:28.273" Title="Temporal Difference Learning Getting Stuck" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6406" PostTypeId="1" CreationDate="2015-07-09T04:44:42.500" Score="7" ViewCount="4680" Body="&lt;p&gt;The algorithm in question is Kohonen's SOM. But the question could also apply to PCA and some others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the umatrix (or the codebook?) is examined, is there a way to tell how successful clustering was?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And would it be a good idea to apply GA's to optimize size, lattice structure, learning rate, and the learning degradation functions as well as the epoch count for clustering, or is there a danger of overfitting in this instance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume that the SOM data is coming from a demonstrably weak PRNG and that the very first attempt shows a distinct structure. Is there some statistical property or algorithm that can evaluate the presence and degree of structure to be used for a GA fitness function?&lt;/p&gt;&#xA;" OwnerUserId="10561" LastActivityDate="2018-01-11T22:12:33.503" Title="How to evaluate clustering success in a completely unsupervised system?" Tags="|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6408" PostTypeId="2" ParentId="6405" CreationDate="2015-07-09T10:50:15.607" Score="1" Body="&lt;p&gt;Sounds like an interesting application. To debug RL applications, I like to perform rollouts. First identify a state where the current policy appears to be clearly wrong; (maybe sample some states and look at them individually). Then switch off the learning and run a sample from that state for both the current RL policy and the action you think is correct. This should give you the true action values. Hopefully, that will give you a hint as to what's going wrong. If it's an exploration problem, then the current policy won't be trying any actions similar to the action you think is correct. If the form of the value function is wrong, then it won't fit the values coming from your samples.&lt;/p&gt;&#xA;" OwnerUserId="10568" LastActivityDate="2015-07-09T10:50:15.607" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6409" PostTypeId="2" ParentId="6406" CreationDate="2015-07-09T13:25:24.823" Score="10" Body="&lt;p&gt;First, PCA is not a clustering method. It is a dimensionality reduction scheme.  You can assess the performance of PCA through analysis of the percent of variance in the dataset that is retained as you decrease the number of dimensions.  Retaining 99%,95%, or 90% is usually ideal, depending on your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to clustering, you probably want to start with the &lt;a href=&quot;https://en.wikipedia.org/wiki/Silhouette_(clustering)&quot;&gt;Silhouette Coefficient&lt;/a&gt;.  This combines assessments of both the cohesion (how tight a cluster is) and separation (how well separated each cluster is from other clusters).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Calculate a = average distance of i to the points in its cluster&lt;/li&gt;&#xA;&lt;li&gt;Calculate b = min (average distance of i to points in another cluster) &lt;/li&gt;&#xA;&lt;li&gt;The silhouette coefficient for a point is then given by:&#xA;&lt;code&gt;s =1 – a/b, if a &amp;lt; b&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is typically between 0 and 1 with larger numbers being &quot;better&quot;.  You can average the coefficients over a cluster or the entire region to get an assessment of the cluster or the entire clustering procedure for your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, try googling &quot;assessing clustering&quot; or &quot;cluster validity&quot; to read about all of the other ways that you can score your clustering algorithm.  &lt;a href=&quot;http://www.cs.kent.edu/~jin/DM08/ClusterValidation.pdf&quot;&gt;Here is a very complete treatment of the topic&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-09T13:25:24.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6410" PostTypeId="1" AcceptedAnswerId="6413" CreationDate="2015-07-09T16:12:00.940" Score="2" ViewCount="133" Body="&lt;p&gt;In Matlab, if you build a simple network and train it: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;OP = feedforwardnet(5, 'traingdm');&#xA;inputsVals = [0,1,2,3,4];&#xA;targetVals = [3,2,5,1,9];&#xA;OP = train(OP,inputsVals,targetVals);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then you train it again so another &lt;code&gt;OP = train(OP,inputsVals,targetVals);&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is happens to the network? Does it train again based on what it learned the first time you did &lt;code&gt;OP = train(OP,inputsVals,targetVals);&lt;/code&gt; or does it train as if it were the first time training the network. &lt;/p&gt;&#xA;" OwnerUserId="10584" LastActivityDate="2016-04-27T10:25:52.970" Title="Does the network learn based on previous training or does it restart? Matlab, neuralnetworks" Tags="|neural-network|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6412" PostTypeId="2" ParentId="6349" CreationDate="2015-07-09T17:47:34.793" Score="1" Body="&lt;p&gt;To directly answer your question about stacking: you should care about minimizing 1) bias, and 2) variance. This is obvious, but in practice this often comes down to simply having models which are &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=640232&quot; rel=&quot;nofollow&quot;&gt;&quot;diverse&quot;&lt;/a&gt;. (I apologize that link is behind a paywall, but there are a few others like it and you may well find it other ways)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't want ensembles of like-minded models - they will make the same mistakes and reinforce each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of stacking, what is happening? You are letting the outputs of the probabilistic classifiers on the actual feature input become the new features. A diverse set of classifiers which can in any way give signals about edge cases is desirable. If classifier 1 is terrible at classes A, B, and C but fantastic at class D, or a certain edge case, it is still a good contribution to the ensemble. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why neural nets are so good at what they do in image recognition - deep nets are in fact recursive logistic regression stacking ensembles! Nowadays people don't always use the sigmoid activation and there are many layer architectures, but it's the same general idea. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would recommend is trying to maximize the diversity of your ensemble by using some of the similarity metrics on the classifiers' prediction output vectors (ie, Diettrich's Kappa statistic) in training. &lt;a href=&quot;http://www.sandia.gov/~wpk/pubs/publications/infoFusion.pdf&quot; rel=&quot;nofollow&quot;&gt;Here is another good reference&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="10587" LastEditorUserId="10587" LastEditDate="2015-07-09T17:56:15.970" LastActivityDate="2015-07-09T17:56:15.970" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6413" PostTypeId="2" ParentId="6410" CreationDate="2015-07-09T18:15:42.020" Score="0" Body="&lt;p&gt;It trains again based on what it learned the first time you did &lt;code&gt;OP = train(OP,inputsVals,targetVals)&lt;/code&gt;. More generally, &lt;code&gt;train&lt;/code&gt; uses your network's weights, i.e. it does not initialize the weights. The weight initialization happens in &lt;code&gt;feedforwardnet&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% To generate reproducible results &#xA;% http://stackoverflow.com/a/7797635/395857&#xA;rng(1234,'twister')&#xA;&#xA;% Prepare input and target vectors&#xA;[x,t] = simplefit_dataset;&#xA;&#xA;% Create ANN&#xA;net = feedforwardnet(10);&#xA;&#xA;% Loop to see where train() initializes the weights&#xA;for i = 1:10&#xA;&#xA;    % Learn&#xA;    net.trainParam.epochs = 1;&#xA;    net = train(net,x,t);&#xA;&#xA;    % Score&#xA;    y = net(x);&#xA;    perf = perform(net,y,t)&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;perf =&#xA;&#xA;    0.4825&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0093&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0034&#xA;&#xA;&#xA;perf =&#xA;&#xA;    0.0028&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-07-09T18:15:42.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6414" PostTypeId="1" AcceptedAnswerId="6421" CreationDate="2015-07-09T20:38:22.933" Score="2" ViewCount="436" Body="&lt;p&gt;I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being built in python (available in github or so) so that I can contribute to that.Is anyone aware of such effort.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-09T23:46:29.367" Title="Scalable open source machine learning library written in python" Tags="|machine-learning|scalability|scikit-learn|apache-spark|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6415" PostTypeId="2" ParentId="2257" CreationDate="2015-07-09T20:44:47.437" Score="2" Body="&lt;p&gt;I am not sure I understood the problem,however if you are trying to predict sales amount my guess is ARIMA might not be the right choice as it will not consider external variables.My suggestion is to gather related features such as how did the other movie from same genre did at that time of the year in that region,weather,star presence,presence of other popular matches or tournament at that time and etc.Let me know your thoughts. &lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-09T20:44:47.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6416" PostTypeId="2" ParentId="6380" CreationDate="2015-07-09T21:00:32.840" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/111968&quot;&gt;Here&lt;/a&gt; is a nice link on that on stackexchange, however my general experience is the more depth the model has the more it tends to overfit.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastEditorUserId="98307" LastEditDate="2020-08-16T16:54:10.717" LastActivityDate="2020-08-16T16:54:10.717" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="6417" PostTypeId="1" AcceptedAnswerId="6432" CreationDate="2015-07-09T21:09:15.107" Score="4" ViewCount="484" Body="&lt;p&gt;I am reading &lt;em&gt;Applied Predictive Modeling&lt;/em&gt; by Max Khun. I chapter 16 he discusses using alternate cutoffs as a remedy for class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose our model predicts the most likely outcome of 2 events, e1 and e2. We have e1 occurring with a predicted probability 0.52 and e2 with a predicted probability 0.48. Using the standard 0.5 for e1 cutoff we would predict e1, but using an alternative cutoff of 0.56 for e1 we would predict e2 because we only predict e1 when p(e1) &gt; 0.56.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, does it make sense to also readjust the probabilities when using alternate cutoffs. For example, in my previous example using 0.56 cutoff of e1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(e1) = 0.52;&#xA;p(e2) = 0.48&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we apply an adjustment of 0.56 - 0.5 = 0.06.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;/p&gt;&#xA;&#xA;&lt;p&gt;p_adj(e1) =  0.52 - 0.06 = 0.46;&#xA;p_adj(e2) = 0.48 + 0.06 = 0.54&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically we shift the probabilities so that they predict e1 when p_adj(e1) &gt; 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize if there is something obviously flawed with my logic but it feels intuitively wrong to me to predict e2 when p(e1) &gt; p(e2). Which probabilities would be more in line with the real-world probabilities?&lt;/p&gt;&#xA;" OwnerUserId="2817" LastActivityDate="2015-07-10T22:42:22.630" Title="Adjusting Probabilities When Using Alternative Cutoffs For Classification" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6419" PostTypeId="2" ParentId="6394" CreationDate="2015-07-09T21:23:51.790" Score="0" Body="&lt;p&gt;Actually deep learning can be run in spark using h2o sparkling water feature.Also you can use h2o.deeplearning to run deeplearning on your data in cluster or single node.Spark is good for munging the data in cluster as it does so distributedly but in memory,otherwise h2o has limited functions for data munging and that too it can't distribute data munging.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-09T21:23:51.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6420" PostTypeId="1" CreationDate="2015-07-09T21:29:17.050" Score="5" ViewCount="952" Body="&lt;p&gt;I am not sure whether the Theano library can be used to write parallelized code in map reduce or in spark. Any expert opinion is welcome. A discussion was on at:&#xA;&lt;a href=&quot;https://groups.google.com/forum/#!searchin/theano-dev/theano$20spark/theano-dev/QDN94MW3LkE/k817aa75xZoJ&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano-dev&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10327" LastEditorUserId="29169" LastEditDate="2021-02-18T20:54:09.870" LastActivityDate="2021-02-18T20:54:09.870" Title="Can theano work on mapreduce or on spark?" Tags="|machine-learning|deep-learning|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6421" PostTypeId="2" ParentId="6414" CreationDate="2015-07-09T23:46:29.367" Score="3" Body="&lt;p&gt;Is there a specific reason beside the fact that you would like to contribute? I am asking because there is always &lt;a href=&quot;https://github.com/apache/spark/tree/master/python/pyspark/mllib&quot; rel=&quot;nofollow&quot;&gt;pyspark&lt;/a&gt; that you can use, the Spark python API that exposes the Spark programming model to Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For deep learning specifically, there are a lot of frameworks built on top of &lt;a href=&quot;https://github.com/Theano/Theano&quot; rel=&quot;nofollow&quot;&gt;Theano&lt;/a&gt; -which is a python library for mathematical expressions involving multi-dimensional arrays-, like Lasagne, so they are able to use GPU for intense training. Getting an EC2 instance with GPU on AWS is always an option.&lt;/p&gt;&#xA;" OwnerUserId="10592" LastActivityDate="2015-07-09T23:46:29.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6422" PostTypeId="2" ParentId="6380" CreationDate="2015-07-10T00:23:22.263" Score="5" Body="&lt;p&gt;@xof6 is correct in the sense that the more depth the model has the more it tends to overfit, but I wanted to add some more parameters that might be useful to you. I do not know which package you are using with R and I am not familiar with R at all, but I think there must be counterparts of these parameters implemented there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of trees - The bigger this number, the less likely the forest is to overfit. This means that as each decision tree is learning some aspect of the training data, you are getting more options to choose from, so to speak.&#xA;Number of features - This number constitutes how many features each individual tree learns. As this number grows, the trees get more and more complicated, hence they are learning patterns that might not be there in the test data. It will take some experimenting to find the right value, but such is machine learning.&#xA;Experiment with the general depth as well, as we mentioned!&lt;/p&gt;&#xA;" OwnerUserId="10592" LastActivityDate="2015-07-10T00:23:22.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6423" PostTypeId="1" AcceptedAnswerId="6424" CreationDate="2015-07-10T03:47:16.397" Score="1" ViewCount="114" Body="&lt;p&gt;Given a sample of hexadecimal data, I would like to identify UNKNOWN sequences of bytes that are repeated throughout the sample. (Not searching for a known string or value) I am attempting to reverse engineer a network protocol, and I am working on determining data structures within the packet. As an example of what I'm trying to do (albeit on a smaller scale):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(af:b6:ea:3d:83:02:00:00):{21:03:00:00}:[b3:49:96:23:01]&#xA;&#xA;{21:03:00:00}:(af:b6:ea:3d:83:02:00:00):01:42:00:00:00:00:01:57&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(38:64:88:6e:83:02:00:00):{26:03:00:00}:[b3:49:96:23:01]&#xA;&#xA;{26:03:00:00}:(38:64:88:6e:83:02:00:00):01:42:00:00:00:00:00:01&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously, these are easy to spot by eye, but patterns that are hundreds of chars into the data are not. I'm not expecting a magic bullet for the solution, just a nudge in the right direction, or even better, a premade tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently needing this for a C# project, but I am open to any and all tools.&lt;/p&gt;&#xA;" OwnerUserId="10596" LastActivityDate="2015-07-10T05:34:10.930" Title="Identifying repeating sequences of data in byte array" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6424" PostTypeId="2" ParentId="6423" CreationDate="2015-07-10T05:34:10.930" Score="3" Body="&lt;p&gt;I believe the problem that you are referring to, is that of &quot;Motif Discovery in Time Series Data&quot;. An appreciable amount of research literature already exists in this domain, so you can look through those. If the data that you handle is not very large, you can find some relatively easy to implement algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data is large , you can look at more recent publications in this domain. As a starting point I would recommend taking a look at how Motif Discovery is done in SAX. SAX takes continuous signals as inputs and discretizes them. These discrete levels are then stored as alphabets. This resulting data looks very much like yours in my opinion. Take a look at what they do in &quot;Mining Motifs in Massive Time Series Databases&quot;.&lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-10T05:34:10.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6425" PostTypeId="2" ParentId="6406" CreationDate="2015-07-10T10:47:50.590" Score="4" Body="&lt;p&gt;It is very hard to evaluate the correctness of the results produced by an unsupervised algorithm. In many cases, such evaluation will be totally subjective, and it will require some knowledge about the domain of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we focus on clustering algorithms (as stated in a previous answer, PCA is not a clustering algorithm), many cluster validation measures may be applied, like the ones enumerated in the &quot;Evaluation and assessment&quot; section in the &quot;Cluster analysis&quot; Wikipedia page. These measures return a number that you can use to compare different clustering solutions, in terms of cluster compactness (how close to each other are the elements within each cluster) and separation (how far to each other are the elements from different clusters). You can, of course, use these measures in order to perform hyperparameter selection (lattice size, structure, learning rate, etc.) by means of cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it must be noted that different cluster validation measures may produce different validation results, and as a consequence, your best clustering solution may vary depending on the chosen measure. Therefore, even the choice of a validation measure is subjective. Once again, knowledge about your data is very important to take this decision.&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-07-10T10:47:50.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6427" PostTypeId="1" AcceptedAnswerId="6428" CreationDate="2015-07-10T14:05:44.183" Score="1" ViewCount="1523" Body="&lt;p&gt;I have a categorical variable in my dataset. I want to replace the levels which are present in test dataset and not present in training set with a value called &quot;others&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how it looks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;levels(training$var1)&lt;/code&gt; has levels as &quot;1&quot;  &quot;2&quot;  &quot;3&quot;  &quot;Others&quot;       &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;levels(testing$var1)&lt;/code&gt; has levels as &quot;1&quot;  &quot;2&quot;  &quot;3&quot;  &quot;4&quot;  &quot;5&quot;  &quot;6&quot;  &quot;7&quot;  &quot;8&quot;  &quot;9&quot;  &quot;10&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to replace all the levels in testing data which are not training data. To achieve this I take the difference between the levels first.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a &amp;lt;- setdiff(levels(levels(testing$var1),training$var1)).  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I get output as  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;levels(a)&lt;/code&gt; as &quot;4&quot;  &quot;5&quot;  &quot;6&quot;  &quot;7&quot;  &quot;8&quot;  &quot;9&quot;  &quot;10&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to replace all the above difference values with &quot;Others&quot;. Kindly note that I do not want to drop out these levels but I want to replace with &quot;Others&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this I tried  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testing$var1[testing$var1 == &quot;4&quot; &amp;lt;- 'Others&quot;  &#xA;testing$var1[testing$var1 == &quot;5&quot; &amp;lt;- 'Others&quot; ==&amp;gt; these works.  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I want to make it in run time, something like this:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testing$var1[testing$var1 == a[1,] &amp;lt;- 'Others&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this is not working.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="29575" LastEditDate="2017-12-15T16:09:41.380" LastActivityDate="2017-12-15T16:09:41.380" Title="How to replace levels with certain value?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6428" PostTypeId="2" ParentId="6427" CreationDate="2015-07-10T14:32:53.177" Score="3" Body="&lt;p&gt;I could be able to achieve this with the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testing$var1 &amp;lt;- as.character(testing$var1)&#xA;a &amp;lt;- data.frame(a)&#xA;testing$var1 [testing$var1 %in% a[1,] &amp;lt;- &quot;Others&quot;&#xA;testing$var1 &amp;lt;- as.factor(testing$var1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In case if there is any other better/effective solution/function to achieve this, please let me know. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks all.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="10327" LastEditDate="2015-07-12T19:16:43.227" LastActivityDate="2015-07-12T19:16:43.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6429" PostTypeId="2" ParentId="6417" CreationDate="2015-07-10T15:16:06.270" Score="0" Body="&lt;p&gt;I believe you can try with range of cut offs to see which one gives highest accuracy or F-score in hold out set.I've seen winners in kaggle stating to do experiment with cut-offs to get best result.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-10T15:16:06.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6430" PostTypeId="2" ParentId="6380" CreationDate="2015-07-10T15:18:36.517" Score="7" Body="&lt;p&gt;Relative to other models, Random Forests are less likely to overfit but it is still something that you want to make an explicit effort to avoid. Tuning model parameters is definitely one element of avoiding overfitting but it isn't the only one. In fact I would say that your training features are more likely to lead to overfitting than model parameters, especially with a Random Forests. So I think the key is really having a reliable method to evaluate your model to check for overfitting more than anything else, which brings us to your second question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As alluded to above, running cross validation will allow to you avoid overfitting. Choosing your best model based on CV results will lead to a model that hasn't overfit, which isn't necessarily the case for something like out of the bag error. The easiest way to run CV in R is with the &lt;code&gt;caret&lt;/code&gt; package. A simple example is below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(caret)&#xA;&amp;gt; &#xA;&amp;gt; data(iris)&#xA;&amp;gt; &#xA;&amp;gt; tr &amp;lt;- trainControl(method = &quot;cv&quot;, number = 5)&#xA;&amp;gt; &#xA;&amp;gt; train(Species ~ .,data=iris,method=&quot;rf&quot;,trControl= tr)&#xA;Random Forest &#xA;&#xA;150 samples&#xA;  4 predictor&#xA;  3 classes: 'setosa', 'versicolor', 'virginica' &#xA;&#xA;No pre-processing&#xA;Resampling: Cross-Validated (5 fold) &#xA;&#xA;Summary of sample sizes: 120, 120, 120, 120, 120 &#xA;&#xA;Resampling results across tuning parameters:&#xA;&#xA;  mtry  Accuracy  Kappa  Accuracy SD  Kappa SD  &#xA;  2     0.96      0.94   0.04346135   0.06519202&#xA;  3     0.96      0.94   0.04346135   0.06519202&#xA;  4     0.96      0.94   0.04346135   0.06519202&#xA;&#xA;Accuracy was used to select the optimal model using  the largest value.&#xA;The final value used for the model was mtry = 2. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="947" LastEditorUserId="947" LastEditDate="2015-07-10T15:56:49.237" LastActivityDate="2015-07-10T15:56:49.237" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6431" PostTypeId="2" ParentId="6208" CreationDate="2015-07-10T18:33:46.307" Score="1" Body="&lt;p&gt;I have read @MaximHaytovich's answer and it is a good one. I would just like to suggest some further options, which were generalized in that answer as feature engineering. I would suggest trying to do the obvious first and analyze the data before transforming it to become ready for any machine learning algorithm. First look at the data and learn yourself and hypothesize about what patterns could be indicators of fraud. What is the nature of the data that you are 100% sure is reliable (it is mentioned in the question you posted that there is available data that is reliable) and how informative are they for the goal? The more familiar you are with the data, the better interpretation you can provide to any output you get from the methods you use. Try doing some outlier analysis on the incomes or any features that may be fraudulent that may make sense to do outlier detection on. Are there any suspicious missing values? Try doing clustering (one of the unsupervised algorithms you were talking about) to see if there are any in-betweens or irregularities. &lt;/p&gt;&#xA;" OwnerUserId="10592" LastActivityDate="2015-07-10T18:33:46.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6432" PostTypeId="2" ParentId="6417" CreationDate="2015-07-10T22:42:22.630" Score="1" Body="&lt;p&gt;First of all, you cannot always consider what a machine learning algorithm outputs as a &quot;probability&quot;. Logistic regression outputs a sigmoid activation on a &lt;code&gt;(0, 1)&lt;/code&gt; scale, but that doesn't magically make it so! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We simply often scale things to a &lt;code&gt;(0, 1)&lt;/code&gt; scale in ML as a measure of &lt;strong&gt;confidence&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in your example, if the events are mutually exclusive (like classification), just think of them as &quot;event 1&quot; and &quot;NOT event 1&quot;. Something like &lt;code&gt;p(e1) + p(~e1) = 1&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when your book tells you to lower the threshold, it is simply saying that you require a smaller level of &lt;strong&gt;confidence&lt;/strong&gt; to choose e1 over e2. This doesn't mean you are choosing one with smaller likelihood, you are simply making a conscious choice to adjust your &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow&quot;&gt;precision-recall curve&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other ways to combat class imbalance, but changing the threshold to be more sensitive to any indication of &lt;strong&gt;confidence&lt;/strong&gt; of one class over another is certainly a way to do that. &lt;/p&gt;&#xA;" OwnerUserId="10587" LastActivityDate="2015-07-10T22:42:22.630" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6433" PostTypeId="1" AcceptedAnswerId="6437" CreationDate="2015-07-11T01:55:26.303" Score="2" ViewCount="796" Body="&lt;p&gt;If I have 3 separate feedforward neural networks in Matlab, is it possible to connect them so that, given input data and target data the 3 work in parallel to produce output? If so, how do I do this? &lt;/p&gt;&#xA;" OwnerUserId="10584" LastActivityDate="2015-07-18T17:26:04.860" Title="Is it possible to connect three neural networks in Matlab?" Tags="|neural-network|matlab|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6434" PostTypeId="2" ParentId="6076" CreationDate="2015-07-11T17:12:17.687" Score="3" Body="&lt;p&gt;No, you are misinterpreting his comments.  If you have data that has some outliers in it then the outliers will extend beyond 3 standard deviations.  Then if you standardize the data some will extend beyond the [-3,3] region.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He is simply saying that you need to remove your outliers so the outliers don't reap havoc on your stochastic gradient descent algorithm. He is NOT saying that you need to use some weird scaling algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should standardize your data by subtracting the mean and dividing by the standard deviation, and then remove any points that extend beyond [-3,3], which are the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In stochastic gradient descent, the presence of outliers could increase the instability of the minimization and make it thrash around excessively, so its best to remove them.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the sparseness of the data prevents removal then... Do you need to use stochastic gradient descent, or can you just use gradient descent?  Gradient descent (GD) might help to alleviate some of the problems relating to convergence.  Finally, if GD is having trouble converging, you could always do an direct solve (e.g. direct matrix inversion) rather than an iterative solve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-11T17:12:17.687" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6435" PostTypeId="1" CreationDate="2015-07-11T18:04:05.710" Score="11" ViewCount="353" Body="&lt;p&gt;I already have a functioning $Q(\lambda)$ implementation for a single agent working on a dynamic pricing problem with the goal of maximizing revenue. The problem that I'm working with, however, involves several different products that are replacements for each other, so dynamically pricing them all with independent learners seems incorrect, because the price of one influences the reward of the other. The goal would be to dynamically price them all so as to maximize the sum of each individual revenue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been doing some research to try to find something that applies reinforcement learning in this way, but many multi-agent implementations I have found focus more on competitive games than cooperative, or they assume incomplete knowledge of other agents (I would have complete knowledge of each agent in this scenario). Are there any well-researched/documented applications of cooperative learning in this way?&lt;/p&gt;&#xA;" OwnerUserId="10560" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:39.053" LastActivityDate="2017-03-16T16:17:54.993" Title="Cooperative Reinforcement Learning" Tags="|machine-learning|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6437" PostTypeId="2" ParentId="6433" CreationDate="2015-07-11T21:31:45.743" Score="3" Body="&lt;p&gt;If you want to combine the results from three different Neural Networks to &quot;boost&quot; the performance :) , you might want to look at the different Ensemble Learning Methods as I mentioned earlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which method you should use, depends on how you share or divide the training data between the three NNs. For example if the NNs are trained on same data but have different parameters, you can look at simple voting ( if you are doing a classification task) or averaging ( if you are using them for regression).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more advanced methods like AdaBoost divide the training data between the classifiers. You can read about it in &lt;a href=&quot;http://www.iro.umontreal.ca/~lisa/pointeurs/ada-nc.pdf&quot; rel=&quot;nofollow&quot;&gt;Boosting Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-11T21:31:45.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6438" PostTypeId="1" AcceptedAnswerId="6458" CreationDate="2015-07-11T23:08:14.300" Score="2" ViewCount="121" Body="&lt;p&gt;If I provide:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A list of possible transforms, and,&lt;/li&gt;&#xA;&lt;li&gt;A list of input states, and,&lt;/li&gt;&#xA;&lt;li&gt;A corresponding list of output states for each input state, and,&lt;/li&gt;&#xA;&lt;li&gt;A fitness function to score each output state&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which subset of machine learning can direct me towards an optimization algorithm that can map each input state to a dictionary of input states, and, failing to find a match, apply the necessary transforms to get me to the closest-related output state?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example involving polygon legalization:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Any given &quot;window&quot; can contain N different polygons, where each polygon has lower-left and upper-right co-ordinates, as well as a polygon &quot;type&quot;.&lt;/li&gt;&#xA;&lt;li&gt;The input state of the polygons may or may not be &quot;illegal&quot;.&lt;/li&gt;&#xA;&lt;li&gt;A list of transforms includes: move, copy, rotate, resize&lt;/li&gt;&#xA;&lt;li&gt;If the input state maps directly to any output state, the input state is decided to be legal. Nothing more to be done; move on the next window.&lt;/li&gt;&#xA;&lt;li&gt;If the input state matches any previously seen input state, transform to the matching (known-legal) output state. Nothing more to be done; move on the next window.&lt;/li&gt;&#xA;&lt;li&gt;Attempt transforms in different sequences until a state is reached that satisfies a fitness function. Store this input:output state combination. Move on to the next window.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Would this imply some combination of neural networking (for classification) and genetic/evolutionary algorithms? Or, does the presence of a fitness function negate the need to store combinations of input:output states?&lt;/p&gt;&#xA;" OwnerUserId="10627" LastActivityDate="2015-07-14T21:30:00.377" Title="Machine learning for state-based transforms?" Tags="|machine-learning|optimization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6439" PostTypeId="2" ParentId="5303" CreationDate="2015-07-12T15:57:38.307" Score="1" Body="&lt;p&gt;I think you are trying to see which original (not PCA) features contribute to which cluster users fall into, yes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, be aware that you just reduced the dimensionality of a 5d dataset down to 2d. You need to be careful of how much variance of the data you just threw away by projecting it into 2d. You can &lt;a href=&quot;https://stats.stackexchange.com/a/31911/7362&quot;&gt;easily calculate this&lt;/a&gt;. If your 2d features account for 95% of the variance of the data, then great! You've got some valid insights about which features are important. If lower, like say 40%, then not so much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(From a qualitative standpoint, just plot the clusters with different colors. That will give you some idea of how much variance you threw away - if the clusters look contiguous and not a lot of mixing, then qualitatively, you didn't throw away too much variance.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, realize that in order to intuit on which original feature values contribute to which clusters, you'll need to use the original features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real answer to your question is that after that you should 1) cluster in 5d, 2) use a classifier, and 3) see the important features using 5d feature vectors. This is critical because in 2d, just because you might have kept most of the variance doesn't mean that last bit might have contained very discriminatory (read: important) information that might improve your underlying clustering error and give you a much more accurate answer. Most importantly, the feature importance will be computed on the actual features, not the PCA linear-combination features which don't translate directly to single site visit histories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="10587" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-07-12T19:37:47.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6440" PostTypeId="1" CreationDate="2015-07-12T19:08:38.467" Score="5" ViewCount="1611" Body="&lt;p&gt;I wonder how to do multitask learning using &lt;a href=&quot;https://github.com/BVLC/caffe&quot; rel=&quot;noreferrer&quot;&gt;Caffe&lt;/a&gt;. Should I simply use the output layer &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/docs/tutorial/layers.md#sigmoid-cross-entropy&quot; rel=&quot;noreferrer&quot;&gt;SigmoidCrossEntropyLoss&lt;/a&gt; or &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/docs/tutorial/layers.md#sum-of-squares--euclidean&quot; rel=&quot;noreferrer&quot;&gt;EuclideanLoss&lt;/a&gt;, and define more than one outputs?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;E.g. is the following architecture valid (3 outputs, i.e. 3 tasks concurrently learnt)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Yc4Sf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Corresponding prototxt file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name: &quot;IrisNet&quot;&#xA;layer {&#xA;  name: &quot;iris&quot;&#xA;  type: &quot;HDF5Data&quot;&#xA;  top: &quot;data&quot;&#xA;  top: &quot;label&quot;&#xA;  include {&#xA;    phase: TRAIN&#xA;  }&#xA;  hdf5_data_param {&#xA;    source: &quot;iris_train_data.txt&quot;&#xA;    batch_size: 1&#xA;&#xA;  }&#xA;}&#xA;&#xA;layer {&#xA;  name: &quot;iris&quot;&#xA;  type: &quot;HDF5Data&quot;&#xA;  top: &quot;data&quot;&#xA;  top: &quot;label&quot;&#xA;  include {&#xA;    phase: TEST&#xA;  }&#xA;  hdf5_data_param {&#xA;    source: &quot;iris_test_data.txt&quot;&#xA;    batch_size: 1&#xA;&#xA;  }&#xA;}&#xA;&#xA;layer {&#xA;  name: &quot;ip1&quot;&#xA;  type: &quot;InnerProduct&quot;&#xA;  bottom: &quot;data&quot;&#xA;  top: &quot;ip1&quot;&#xA;  param {&#xA;    lr_mult: 1&#xA;  }&#xA;  param {&#xA;    lr_mult: 2&#xA;  }&#xA;  inner_product_param {&#xA;    num_output: 50&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;&#xA;&#xA;layer {&#xA;  name: &quot;ip2&quot;&#xA;  type: &quot;InnerProduct&quot;&#xA;  bottom: &quot;ip1&quot;&#xA;  top: &quot;ip2&quot;&#xA;  param {&#xA;    lr_mult: 1&#xA;  }&#xA;  param {&#xA;    lr_mult: 2&#xA;  }&#xA;  inner_product_param {&#xA;    num_output: 3&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;&#xA;&#xA;&#xA;&#xA;layer {&#xA;  name: &quot;loss&quot;&#xA;  type: &quot;SigmoidCrossEntropyLoss&quot; &#xA;  # type: &quot;EuclideanLoss&quot; &#xA;  # type: &quot;HingeLoss&quot;  &#xA;  bottom: &quot;ip2&quot;&#xA;  bottom: &quot;label&quot;&#xA;  top: &quot;loss&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-07-12T19:17:17.377" LastActivityDate="2016-03-03T23:39:18.740" Title="How to do multitask learning using Caffe?" Tags="|deep-learning|multitask-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6441" PostTypeId="5" CreationDate="2015-07-12T19:59:56.350" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-07-12T19:59:56.350" LastActivityDate="2015-07-12T19:59:56.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6442" PostTypeId="4" CreationDate="2015-07-12T19:59:56.350" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-07-12T19:59:56.350" LastActivityDate="2015-07-12T19:59:56.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6444" PostTypeId="2" ParentId="6217" CreationDate="2015-07-13T03:02:48.130" Score="-2" Body="&lt;p&gt;I guess you are looking for features which extract out new features. A feature which best represents the dataset. If that is the case then we call such method &quot;feature extraction&quot;.&lt;/p&gt;&#xA;" OwnerUserId="10361" LastEditorUserId="38516" LastEditDate="2019-07-06T01:58:17.090" LastActivityDate="2019-07-06T01:58:17.090" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6446" PostTypeId="1" CreationDate="2015-07-13T13:34:51.983" Score="1" ViewCount="407" Body="&lt;p&gt;While i am using the &lt;code&gt;confusionMatrix&lt;/code&gt; in R, I am getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error in &lt;code&gt;binom.test(sum(diag(x)), sum(x))&lt;/code&gt; : &#xA;  'n' must be a positive integer &gt;= 'x'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code i use as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;rf.pred &amp;lt;- predict(rf, testing, type = &quot;class&quot;)&lt;br&gt;&#xA;confusionMatrix(rf.pred, testing$target)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="10253" LastEditDate="2015-07-14T07:17:22.880" LastActivityDate="2016-07-23T03:00:03.427" Title="Errror in confusionMatrix in R" Tags="|machine-learning|data-mining|r|classification|predictive-modeling|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6447" PostTypeId="1" CreationDate="2015-07-13T15:39:13.290" Score="1" ViewCount="156" Body="&lt;p&gt;I want to  build a model using R, save it and reloading it when required. I am able to save the model and reload it without any problem, however when i use the model along with predict statement, I am getting errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what i do:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;set.seed(345)&#xA;  df &amp;lt;- data.frame(x = rnorm(20))&#xA;  df &amp;lt;- transform(df, y = 5 + (2.3 * x) + rnorm(20))&lt;/p&gt;&#xA;  &#xA;  &lt;h2&gt;model&lt;/h2&gt;&#xA;  &#xA;  &lt;p&gt;m1 &amp;lt;- lm(y ~ x, data = df)&lt;/p&gt;&#xA;  &#xA;  &lt;h2&gt;save this model&lt;/h2&gt;&#xA;  &#xA;  &lt;p&gt;save(m1, file = &quot;my_model1.rda&quot;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;a month later, new observations are available:&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;newdf &amp;lt;- data.frame(x = rnorm(20))&lt;/p&gt;&#xA;  &#xA;  &lt;h2&gt;load the model&lt;/h2&gt;&#xA;  &#xA;  &lt;p&gt;load(&quot;my_model1.rda&quot;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When i use the above model to predict for new values, I am getting output values as &quot;NA&quot;  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;predict(m1, newdata = newdf)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&quot;NA&quot;  &quot;NA&quot;  &quot;NA&quot;  &quot;NA&quot;  &quot;NA&quot;  &quot;NA&quot; &quot;NA&quot;  &quot;NA&quot;  &quot;NA&quot;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could able to see that the model has been loaded correctly by giving a print command&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;print(m1)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Call:&#xA;lm(formula = y ~ x, data = df)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coefficients:&#xA;(Intercept)            x&lt;br&gt;&#xA;      4.792        2.119  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything that I miss here?&lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-07-13T15:39:13.290" Title="Problem while reusing model for new observations" Tags="|machine-learning|data-mining|r|linear-regression|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6448" PostTypeId="2" ParentId="6208" CreationDate="2015-07-13T20:19:39.093" Score="1" Body="&lt;p&gt;I can't see any way to magically identify unreliable data in this context. I'm no expert but 2 ideas spring to mind:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Domain knowledge: an expert on household income may be able to describe some general rules for identifying suspect data based on experience, and you could try to convert these into rules&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Benford%27s_law&quot; rel=&quot;nofollow&quot;&gt;Benford's Law&lt;/a&gt; - I don't know if it would be suitable for this scenario but Benfords Law has applications in detecting accounting fraud, based on the expected frequency of numbers appearing in numeric values. Probably of no use especially if incomes are reported in rounded figures (e.g. nearest 1000) but might be worth a look for inspiration.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9587" LastActivityDate="2015-07-13T20:19:39.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6449" PostTypeId="1" CreationDate="2015-07-13T20:37:28.867" Score="6" ViewCount="159" Body="&lt;p&gt;My data is a set of about 1 million training examples, where each example is represented by about 2000 boolean features. Examples aren't labelled - it wouldn't really have any meaning in this domain. The features fully define the example. My goal is to learn a generative model that I can use to generate more examples. I would like to capture some of the dependencies between the features (e.g. if feature 10 is on, feature 14, 15, and 16 are usually on). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have much background in ML, so I'm not sure what a simple way to do this would be (both algorithms and software packages). My initial thought was to find a structure learning package to learn a Bayesian network and then sample from it, but I figured I'd ask here to see if anyone had any recommendations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="10659" LastEditorUserId="10659" LastEditDate="2015-07-14T00:37:10.937" LastActivityDate="2015-07-14T08:18:37.257" Title="Learn and sample from a generative model with 2000 boolean features" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6450" PostTypeId="2" ParentId="6088" CreationDate="2015-07-14T04:24:44.473" Score="0" Body="&lt;p&gt;In situations where a particular class is really a minority, I suggest using rare category detection. In this particular case of fraud/non-fraud, fraud is a rare category. Its an active field of research - Refere to &lt;a href=&quot;https://scholar.google.com/scholar?q=Rare%20category%20detection&amp;amp;hl=en&amp;amp;as_sdt=0&amp;amp;as_vis=1&amp;amp;oi=scholart&amp;amp;sa=X&amp;amp;ved=0CBsQgQMwAGoVChMI3P2tg-LZxgIV0DCICh0jVAtS&quot; rel=&quot;nofollow&quot; title=&quot;Rare Category Detection&quot;&gt;Rare Category Detection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8513" LastActivityDate="2015-07-14T04:24:44.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6451" PostTypeId="2" ParentId="6189" CreationDate="2015-07-14T06:09:27.803" Score="1" Body="&lt;p&gt;While checking on python &lt;a href=&quot;https://github.com/dmlc/xgboost&quot; rel=&quot;nofollow&quot;&gt;xgboost&lt;/a&gt; I found the existence of this &lt;a href=&quot;https://github.com/dmlc/rabit&quot; rel=&quot;nofollow&quot;&gt;open source project&lt;/a&gt; that helps create scalable machine learning program.Should be worth exploring.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-14T06:09:27.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6452" PostTypeId="2" ParentId="6449" CreationDate="2015-07-14T08:18:37.257" Score="2" Body="&lt;p&gt;You may try an &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;Apriori algorithm&lt;/a&gt; or more advanced &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning#FP-growth_algorithm&quot; rel=&quot;nofollow&quot;&gt;FPGrowth algorithm&lt;/a&gt;. It will give you a set of association rules, which, I think, you required:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;e.g. if feature 10 is on, feature 14, 15, and 16 are usually on&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9637" LastActivityDate="2015-07-14T08:18:37.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6453" PostTypeId="1" CreationDate="2015-07-14T08:45:36.813" Score="1" ViewCount="209" Body="&lt;p&gt;I have a method that calculates a certain variable. This variable is biased by an error which should be related to the observables that were used to calculated the variable. I don't have an exact formula of the calculation and also no errors of the observables, therefore I can't just do error propagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do have training data (the true error). Now I want to use machine learning to find the correlations between the input observables and the error of the calculated output. For now I tried to gather as many observables and used their statistics (mean, variance etc.) as &quot;feature vectors&quot; for machine learning. So far I tried approaches like SVR, linear Regression, SGD Regression etc. but all showed the same bad result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new in this field. So some things are still a little bit unclear to me. Do you have any ideas how to find the relations? Or do you think there is no relation at all? What if there are observables that are totally random; do they destroy the learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="10669" LastActivityDate="2015-07-14T08:45:36.813" Title="Find correlation in observed data" Tags="|machine-learning|regression|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6454" PostTypeId="2" ParentId="6208" CreationDate="2015-07-14T09:02:54.843" Score="1" Body="&lt;p&gt;I'm not sure it is possible to generalise as to how people may mis-state financial information they report to governments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This probably varies by country and hence culture and if the information is used directly to affect the welfare benefits received by that individual, it may not be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is data on household income from a more trusted source, tax receipts for instance, that might be used to normalise the self reported data.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-07-14T09:02:54.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6455" PostTypeId="2" ParentId="5168" CreationDate="2015-07-14T10:02:18.477" Score="1" Body="&lt;p&gt;You can also modify your cost function so that instances in the smaller class have more weight. This makes training computationally less demanding, when compared to your option 1.&lt;/p&gt;&#xA;" OwnerUserId="10667" LastActivityDate="2015-07-14T10:02:18.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6458" PostTypeId="2" ParentId="6438" CreationDate="2015-07-14T21:30:00.377" Score="1" Body="&lt;p&gt;If i get it correctly:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You have an input polygon&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As a first step you want to &quot;match&quot; that against a list of previously seen templates. If this is successful, you pick it's corresponding output and move on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If not, you wish to find some optimal transformation, in order for it to satisfy some constraints that you have (your &quot;objective function&quot;). Then add the original+transformed shape to the templates list and move on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is this correct? I'll risk an answer anyways:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first part, I believe that there is a &lt;a href=&quot;https://www.google.com.tr/search?rls=en&amp;amp;q=shape%20matching%20algorithm&amp;amp;ie=UTF-8&amp;amp;oe=UTF-8&quot; rel=&quot;nofollow&quot;&gt;slew of literature out there&lt;/a&gt;. It's not my expertise, but first thing that comes to mind is measuring the distance in feature space between your shape and each template, and picking the closest one, it the distance is below a threshold that you set. &quot;Feature&quot; here would be either some low-level polygon property, e.g. x and y coordinates of vertices, or an abstraction, e.g. perimeter, area, no. vertices, mean side length/side length variance, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second part, it really depends on the nature of your constraints/objective functions. Are they convex? Uni- or multi-modal? Single or multi-objective? Do you want to incorporate some domain knowledge (i.e. knowledge about what &quot;good&quot; transformations would be?)? One can really not tell without further details. Evolutionary algorithms are quite versatile but expensive methods (although some argue on that). If you can spare the possibly large amount of function evaluations, you could try EAs as a first step, and then refine your approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, while not exactly related to what you describe in your process, I believe you may benefit by taking a look into auto-associative networks (and models in general); these are models that are able to perform constraint-satisfaction on their input, effectively enforcing learned relationships on input values. I could see this being used in your case by inputing a shape, and having a transformed shape as an output, which would be &quot;legal&quot;, i.e. satisfying the constraints learned by the auto associative model. Thus, you would eliminate the need for a template matching + optimization altogether.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2015-07-14T21:30:00.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6459" PostTypeId="1" AcceptedAnswerId="6485" CreationDate="2015-07-14T22:39:37.943" Score="4" ViewCount="2770" Body="&lt;p&gt;I am planning on making an AI song composer that would take in a bunch of songs of one instrument, extract musical notes (like ABCDEFG) and certain features from the sound wave, preform machine learning (most likely through recurrent neural networks), and output a sequence of ABCDEFG notes (aka generate its own songs / music). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that this would be an unsupervised learning problem, but I am not really sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I figured that I would use recurrent neural networks, but I have a few questions on how to approach this:&lt;br&gt;&#xA;- What features from the sound wave I should extract so that the output music is melodious?&lt;br&gt;&#xA;- Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?&lt;br&gt;&#xA;- Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes? &lt;/p&gt;&#xA;" OwnerUserId="10523" LastActivityDate="2017-07-20T13:31:20.510" Title="What features from sound waves to use for an AI song composer?" Tags="|machine-learning|neural-network|feature-selection|feature-extraction|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6460" PostTypeId="1" CreationDate="2015-07-14T23:37:01.940" Score="2" ViewCount="550" Body="&lt;p&gt;I am trying to use &lt;a href=&quot;https://github.com/BVLC/caffe&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe&lt;/a&gt; on the usual &lt;a href=&quot;https://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fisher's Iris data set&lt;/a&gt; (150 flowers, each having 4 features, and split into 3 classes):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if a flower belong to class 1 (setosa), the network output should be [1, 0, 0]&lt;/li&gt;&#xA;&lt;li&gt;if a flower belong to class 2 (versicolor), the network output should be [0, 1, 0]&lt;/li&gt;&#xA;&lt;li&gt;if a flower belong to class 3 (virginica), the network output should be [0, 0, 1]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I use the &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/docs/tutorial/layers.md#sigmoid-cross-entropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;SigmoidCrossEntropyLoss&lt;/a&gt; as it &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network#Loss_layer&quot; rel=&quot;nofollow noreferrer&quot;&gt;used for predicting K independent probability values in [0,1]&lt;/a&gt;. Let ip3 be the layer connected to SigmoidCrossEntropyLoss (in addition to the label layer): looking at ip3's output, I interpret negative values as class absent (0), and positive values as class present (1). E.g. if ip3's output is [-0.3; 0.9; -0.4], then I interpret it as [0; 1; 0], i.e. class 2 is present but classes 1 and 3 are absent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network does a good job classifying classes 1 and 3 (accuracy over 90%), but consistently fails to predict class 2: the network always predicts that class 2 is absent, i.e. the network's second output is always 0, i.e. that ip3's second output is always negative, no matter what the input is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the architecture I use. Am I doing something wrong? Is the architecture not suited for the task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/UL0cr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="31513" LastEditDate="2017-05-15T15:48:46.473" LastActivityDate="2017-05-15T15:48:46.473" Title="Fisher's Iris data set with Caffe" Tags="|deep-learning|multiclass-classification|caffe|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6461" PostTypeId="2" ParentId="6459" CreationDate="2015-07-15T00:15:11.733" Score="3" Body="&lt;p&gt;I believe the question is, you want to learn from musical pieces and try to generate a tune from the trained instance. Lets see if I can set up a simple model to do this, and then you can extrapolate from there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;a href=&quot;https://en.wikipedia.org/wiki/Mel-frequency_cepstrum&quot; rel=&quot;nofollow&quot;&gt;MFCC&lt;/a&gt; is a good feature when working with sound. You can use that to extract the features from lets say 1-2 second windows of your song. You now have a fingerprint for the audio file. Take a look at &lt;a href=&quot;http://www.uoguelph.ca/~gwtaylor/thesis/4/&quot; rel=&quot;nofollow&quot;&gt;Conditional Restricted Boltzmann Machines&lt;/a&gt;. They are Neural Networks which use multiple binary states to encode time series information. As you can see in the webpage, they trained on human-gait data and can now generate their own human gait. This is essentially what you want but for music files. So you can train CRBMs on the Audio MFCC vectors that you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the training is done, to generate an audio file you can either &quot;seed&quot; the CRBM with a few seconds of some melody or just randomly initialize it. Then just allow the CRBM to go nuts and record whatever it produces. This is your new audio file. To produce another sample use a different seed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This solves the question of how you can implement a &quot;melody&quot; generation scheme. There are of course variations. You can add other features to you vector apart from MFCC. You can also use other time series predictors like &lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;nofollow&quot;&gt;LSTM&lt;/a&gt; or Markov models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of this being said, the problem of generating music might be much more nuanced than it looks at first glance. Machine Learning algorithms just apply previously learned patterns in the data. How does that correspond to &quot;creating&quot; new music , is a philosophical question. If we analyze the aforementioned algorithm, essentially the CRBM will generate a next output based on the probability distribution that it has learnt. It would be very interesting to see what kind of output it generates when the said distribution is that of musical notes.  &lt;/p&gt;&#xA;" OwnerUserId="10446" LastEditorUserId="10446" LastEditDate="2015-07-15T00:21:17.483" LastActivityDate="2015-07-15T00:21:17.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6464" PostTypeId="1" CreationDate="2015-07-15T14:15:17.773" Score="2" ViewCount="266" Body="&lt;p&gt;I want to train a binary classification algorithm for spam detection using labeled data set of mentions from social media. These mentions have the following features:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;URL, text message, posting date, is deleted, &lt;em&gt;author&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where the &lt;em&gt;author&lt;/em&gt; has its own fields:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;URL, nick, type (&lt;em&gt;person or community&lt;/em&gt;), number of subscribers, registration date, last activity date&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have $\approx 13.5k$ observations with $\approx 450$ spam messages among them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;transform data features in real numbers&lt;/li&gt;&#xA;&lt;li&gt;binarize URLs by media sources (e.g. facebook, instagram, etc.)&lt;/li&gt;&#xA;&lt;li&gt;use SVM with Gaussian kernel for all data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Questions: &lt;strong&gt;Am I on the right way? What else can I do? How can I process text data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any suggestions!&lt;/p&gt;&#xA;" OwnerUserId="10694" LastActivityDate="2015-07-15T18:47:02.090" Title="Spam detection in social media" Tags="|machine-learning|classification|social-network-analysis|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6466" PostTypeId="2" ParentId="6464" CreationDate="2015-07-15T17:56:33.763" Score="1" Body="&lt;p&gt;The approach seems correct, don't worry. But I'd suggest to use Naive Bayes Classifier instead of SVM - it perfectly works for this task (and actually popular email services use this algorithm for spam detection).&lt;/p&gt;&#xA;" OwnerUserId="7969" LastActivityDate="2015-07-15T17:56:33.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6467" PostTypeId="1" AcceptedAnswerId="6471" CreationDate="2015-07-15T17:59:31.063" Score="4" ViewCount="174" Body="&lt;p&gt;I run into this problem from time to time and have always felt like there should be an obvious answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have probabilities for potential classes (from some classifier).  I will offer the prediction of the class with the highest probability, however, I would also like to attach a confidence for that prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: If I have Classes &lt;code&gt;[C1, C2, C3, C4, C5]&lt;/code&gt; and my Probabilities are &lt;code&gt;{C1: 50, C2: 12, C3: 13, C4: 12, C5:13}&lt;/code&gt; my confidence in predicting C1 should be higher than if I had Probabilities &lt;code&gt;{C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reporting that I predict class C1 with 60% probability isn't the whole story.  I should be able to derive a confidence from the distribution of probabilities as well.  I am certain there is a known method for solving this but I do not know what it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Taking this to the extreme for clarification: &#xA;If I had a class C1 with 100% probability (and assuming the classifier had an accurate representation of each class) then I would be extremely confident that C1 was the correct classification.  On the other hand if all 5 classes had almost equal probability (Say they are all roughly 20%) than I would be very uncertain claiming that any one was the correct classification. These two extreme cases are more obvious, the challenge is derive a confidence for intermediate examples like the one above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions or references would be of great help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="10701" LastEditorUserId="10701" LastEditDate="2015-07-15T19:01:17.677" LastActivityDate="2015-07-15T19:58:35.510" Title="Deriving Confidences from Distribution of Class Probabilities for a Prediction" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6468" PostTypeId="2" ParentId="6467" CreationDate="2015-07-15T18:11:33.163" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1:&#xA;  50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1&#xA;  should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2,&#xA;  C4: 1, C5: 2}.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Assuming that those probabilities are accurate, this isn't true. In your second case you can be a lot more confident that the ground truth is one of C1 or C2, but in terms of absolute confidence about C1 the probability is the same across both examples. To illustrate this with a more clear example, if you had a 100 sided die that had 50 sides labeled with &quot;C1&quot; then the labels on the the other 50 sides are irrelevant to the likelihood that you would roll a &quot;C1&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now with that said, your probabilities from your model are most certainly not perfect so there may be a way to use the intra-class correlations to improve them. Can you provide some more details about your specific problem and modeling workflow that you have used to get your probabilities?&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-07-15T18:11:33.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6470" PostTypeId="2" ParentId="6464" CreationDate="2015-07-15T18:47:02.090" Score="1" Body="&lt;p&gt;Yes, you are essentially on the right track. You can normalize the data , if your text samples have varied length. Also, you can use categorical data but not in the same feature. For example, having one feature which is labelled &quot;1&quot; for facebook and &quot;2&quot; for instagram is not good. Have separate features for these kind of categorical data with {0,1} labels. There are two additional inputs that I would like to give here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, use a systematic grid search instead of just guessing the parameters. This is a very good practice which I picked up very late. If you are using a soft margin SVM ( which you should) with the gaussian kernel, you will have just two parameters to search over, C and gamma. This will make sure that you are extracting the best parameters from your classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, you have 13.5K samples and only 450 are spam. Which means you have 13 K desired mails. So you have 450 positive samples and 13K negative samples. If this is indeed the data distribution, you have an unbalanced data. You will have to weight the data according to the proportion otherwise SVM will end up classifying most of the samples as negative. To read about this in more detail look at Section 7 of this &lt;a href=&quot;http://www.cs.colostate.edu/~asa/pdfs/howto.pdf&quot; rel=&quot;nofollow&quot;&gt;document&lt;/a&gt; . Weighted SVMs are implemented in LibSVM, so you can directly use that. &lt;/p&gt;&#xA;" OwnerUserId="10446" LastActivityDate="2015-07-15T18:47:02.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6471" PostTypeId="2" ParentId="6467" CreationDate="2015-07-15T19:58:35.510" Score="1" Body="&lt;p&gt;As @David says, in your initial example, your confidence about C1 is the same in both cases. In your second example, you most certainly are less confident about the most-probable class in the second case, since the most-probable class is far less probable!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may have to unpack what you're getting at when you say 'confidence' then, since here you're not using it as a term of art but an English word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect you may be looking for the idea of &lt;a href=&quot;https://en.wikipedia.org/wiki/Entropy_(information_theory)&quot; rel=&quot;nofollow&quot;&gt;entropy&lt;/a&gt;, or uncertainty present in the distribution of all class probabilities. In your first example, it is indeed lower in the second case than the first. I don't think what you're getting at is just a function of the most-probable class, that is.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-07-15T19:58:35.510" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6472" PostTypeId="1" CreationDate="2015-07-15T21:06:00.360" Score="4" ViewCount="872" Body="&lt;p&gt;I am currious if there is a shortcut for runing the  selected cell in IPython notebook such as alt+F5 or ctrl+f5 or something simmiliar  that is present in Visual Studio (I don't recall exactly now ) , cause I found it annoing to go with  the mouse on  the run  button every  time I need to run a cell?&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2015-07-16T23:05:15.970" Title="IPython notebook shortcut to run" Tags="|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6473" PostTypeId="2" ParentId="6472" CreationDate="2015-07-16T03:12:41.317" Score="0" Body="&lt;p&gt;IPython notebook Keyboard shortcuts&lt;img src=&quot;https://i.stack.imgur.com/VGcXs.png&quot; alt=&quot;IPython notebook&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;![Option the options [2:4] i think is the answer to your question.] &lt;/p&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="8752" LastEditDate="2015-07-16T23:05:15.970" LastActivityDate="2015-07-16T23:05:15.970" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6476" PostTypeId="1" CreationDate="2015-07-16T08:08:45.270" Score="0" ViewCount="839" Body="&lt;p&gt;In the &lt;a href=&quot;http://www.win.tue.nl/~aserebre/ks/dmkd97episodes.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; &quot;Discovery of frequent episodes in event sequences&quot; by Mannila et al., a method for finding frequent episodes in an event sequence if a class of episodes and a sequence of events is given. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear to me how to find episodes in the first place. I would like to know if there are any tools, algorithms, or published work that can be used to this end, or which describes a method to find episodes in a given stream.&lt;/p&gt;&#xA;" OwnerUserId="10717" LastEditorUserId="13413" LastEditDate="2015-10-18T12:37:50.477" LastActivityDate="2015-10-18T12:37:50.477" Title="Finding Episodes in event sequence" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6478" PostTypeId="1" CreationDate="2015-07-16T11:06:49.757" Score="5" ViewCount="1341" Body="&lt;p&gt;Are there any free tools or libraries that can understand a plot image file automatically? Things like type detection (line, bar, scatter), as well as label and axis scaling detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the tools I have found so far, one has to do manual configuration. I'm asking if it is possible in our AI driven world to let a machine do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are none, I will start a project for myself as it seems like a challenging but instructive AI task. Is this effort doomed from start? Are there any helpful libraries in C++ or Java and theoretical algorithms one could consider?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;br&gt;&#xA;Tomas&lt;/p&gt;&#xA;" OwnerUserId="10314" LastEditorUserId="8878" LastEditDate="2015-07-16T13:59:44.197" LastActivityDate="2022-09-04T04:47:00.460" Title="How to extract information from plot images?" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6481" PostTypeId="2" ParentId="6478" CreationDate="2015-07-16T13:52:17.240" Score="4" Body="&lt;p&gt;In the general sense this is most definitely a non-trivial problem and any full featured solution will require a considerable degree of work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Success will probably be dependent on how much you constrain your problem.&#xA;Monochrome will be easier than color.&#xA;Limited font, font size, line and marker styles will help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more constrained, the more chance of success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For line detection, such as axes, you might want to consider the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hough_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hough&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Radon_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;Radon&lt;/a&gt; transforms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some stackexchange questions covering some of the topics you might want to address, sadly no bargraphs, some have a manual component which you might think how to automate. They are a good starting point to review some of the difficulties involved in the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Selecting a curve:&#xA;&lt;a href=&quot;https://mathematica.stackexchange.com/questions/44355/how-to-make-a-curve-selectable-from-a-scaned-image-and-convert-it-to-a-list-of-c&quot;&gt;https://mathematica.stackexchange.com/questions/44355/how-to-make-a-curve-selectable-from-a-scaned-image-and-convert-it-to-a-list-of-c&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Datapoints and Axes:&#xA;&lt;a href=&quot;https://mathematica.stackexchange.com/questions/1524/recovering-data-points-from-an-image&quot;&gt;https://mathematica.stackexchange.com/questions/1524/recovering-data-points-from-an-image&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scatter:&#xA;&lt;a href=&quot;https://mathematica.stackexchange.com/questions/26356/how-do-i-find-the-coordinates-of-points-in-this-image&quot;&gt;https://mathematica.stackexchange.com/questions/26356/how-do-i-find-the-coordinates-of-points-in-this-image&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Datapoints:&#xA;&lt;a href=&quot;https://mathematica.stackexchange.com/questions/3831/how-can-i-extract-data-points-from-a-black-and-white-image&quot;&gt;https://mathematica.stackexchange.com/questions/3831/how-can-i-extract-data-points-from-a-black-and-white-image&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://mathematica.stackexchange.com/questions/14444/obtain-data-points-from-a-graph-on-an-image-without-axes&quot;&gt;https://mathematica.stackexchange.com/questions/14444/obtain-data-points-from-a-graph-on-an-image-without-axes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="-1" LastEditDate="2017-04-13T12:56:10.853" LastActivityDate="2015-07-16T13:57:20.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6482" PostTypeId="2" ParentId="6472" CreationDate="2015-07-16T14:50:34.690" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://ipython.org/ipython-doc/1/interactive/notebook.html#keyboard-shortcuts&quot; rel=&quot;nofollow noreferrer&quot;&gt;IPython notebook Keyboard shortcuts&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Shift-Enter&lt;/code&gt;: run cell&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Ctrl-Enter&lt;/code&gt;: run cell in-place&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Alt-Enter&lt;/code&gt;: run cell, insert below&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Ctrl-m&lt;/code&gt;: This is the prefix for all other shortcuts, which consist of Ctrl-m followed by a single letter or character. For example, if you type Ctrl-m h (that is, the sole letter h after Ctrl-m), IPython will show you all the available keyboard shortcuts.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/WlnUP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-16T14:50:34.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6484" PostTypeId="2" ParentId="6476" CreationDate="2015-07-16T23:06:47.217" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;https://www.cs.helsinki.fi/u/htoivone/pubs/dmkd97episodes.pdf&quot; rel=&quot;nofollow&quot;&gt;paper that you cited&lt;/a&gt; references two other papers that explain the method you are looking for &lt;a href=&quot;https://www.aaai.org/Papers/KDD/1996/KDD96-024.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://link.springer.com/article/10.1023%2FA%3A1009748302351&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.  The second one is behind a pay wall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But... these look like deterministic methods to me, and I suggest you use a machine learning method instead.  You basically need to separate events from each other using some sort of clustering algorithm.  I suggest you use &lt;code&gt;DBSCAN&lt;/code&gt; and adjust the parameters based on whether you want every event to be assigned to an episode or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another algorithm that might work well is simple &lt;code&gt;k-means&lt;/code&gt; clustering with the addition of an &lt;code&gt;elbow method&lt;/code&gt; or &lt;code&gt;silhouette score&lt;/code&gt; to determine the optimal number of clusters. &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-16T23:06:47.217" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6485" PostTypeId="2" ParentId="6459" CreationDate="2015-07-16T23:55:59.507" Score="9" Body="&lt;p&gt;First off, ignore the haters.  I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The areas of study you are inquiring about are Music Information Retrieval (&lt;a href=&quot;https://en.wikipedia.org/wiki/Music_information_retrieval&quot;&gt;Wiki Link&lt;/a&gt;) and Computer Music (&lt;a href=&quot;https://en.wikipedia.org/wiki/Computer_music&quot;&gt;Wiki Link&lt;/a&gt;) .  You have made a good choice in narrowing your problem to a single instrument (monophonic music) as polyphonic music increases the difficulty greatly.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're trying to solve two problems really:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Automatic Transcription of Monophonic Music (&lt;a href=&quot;https://scholar.google.com/scholar?q=Automatic+Transcription+of+Monophonic+Music&quot;&gt;More Readings&lt;/a&gt;) which is the problem of extracting the notes from a single instrument musical piece.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Algorithmic Composition (&lt;a href=&quot;https://scholar.google.com/scholar?q=Algorithmic%20Composition&quot;&gt;More Readings&lt;/a&gt;) which is the problem of generating new music using a corpus of transcribed music.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your questions directly:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I think that this would be an unsupervised learning problem, but I am&#xA;  not really sure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since there are two learning problems here there are two answers. For the Automatic Transcription you will probably want to follow a supervised learning approach, where your classification are the notes you are trying to extract.  For the Algorithmic Composition problem it can actually go either way. Some reading in both areas will clear this up a lot.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What features from the sound wave I should extract so that the output music is melodious?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are a lot of features used commonly in MIR. @abhnj listed MFCC's in his answer but there are a lot more.  Feature analysis in MIR takes place in several domains and there are features for each.  Some Domains are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The Frequency Domain (these are the values we hear played through a speaker)&lt;/li&gt;&#xA;&lt;li&gt;The Spectral Domain (This domain is calculated via the Fourier function (&lt;a href=&quot;https://en.wikipedia.org/wiki/Fast_Fourier_transform&quot;&gt;Read about the Fast Fourier Transform&lt;/a&gt;) and can be transformed using several functions (Magnitude, Power, Log Magnitude, Log Power)&lt;/li&gt;&#xA;&lt;li&gt;The Peak Domain (A domain of amplitude and spectral peaks over the spectral domain)&lt;/li&gt;&#xA;&lt;li&gt;The Harmonic Domain&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One of the first problems you will face is how to segment or &quot;cut up&quot; your music signal so that you can extract features.  This is the problem of Segmentation (&lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;q=music%20segmentation&quot;&gt;Some Readings&lt;/a&gt;) which is complex in itself.  Once you have cut your sound source up you can apply various functions to your segments before extracting features from them. Some of these functions (called window functions) are the: Rectangular, Hamming, Hann, Bartlett, Triangular, Bartlett_hann, Blackman, and Blackman_harris.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have your segments cut from your domain you can then extract features to represent those segments.  Some of these will depend on the domain you selected.  A few example of features are: Your normal statistical features (Mean, Variance, Skewness, etc.), ZCR, RMS, Spectral Centroid, Spectral Irregularity, Spectral Flatness, Spectral Tonality, Spectral Crest, Spectral Slope, Spectral Rolloff, Spectral Loudness, Spectral Pitch, Harmonic Odd Even Ratio, MFCC's and Bark Scale. There are many more but these are some good basics.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes it is. There have been several works to do this already. (&lt;a href=&quot;https://scholar.google.com/scholar?q=Algorithmic%20Composition%20with%20Neural%20Networks&quot;&gt;Here are several readings&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The standard method is to use the explanation I made above (Domain, Segment, Feature Extract) etc.  To save yourself some work I highly recommend starting with a MIR framework such as MARSYAS (&lt;a href=&quot;http://marsyas.info/&quot;&gt;Marsyas&lt;/a&gt;).  They will provide you with all the basics of feature extraction.  There are many frameworks so just find one that uses a language you are comfortable in.&lt;/p&gt;&#xA;" OwnerUserId="10701" LastEditorUserId="10701" LastEditDate="2015-07-19T21:18:54.867" LastActivityDate="2015-07-19T21:18:54.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6486" PostTypeId="1" CreationDate="2015-07-17T08:58:09.017" Score="2" ViewCount="1797" Body="&lt;p&gt;Data.table package is claimed to be faster than Data.frame. what are the implementation changes that made this possible? How can one leverage the power of this package for data analysis?&lt;/p&gt;&#xA;" OwnerUserId="2643" LastActivityDate="2015-07-17T10:13:29.843" Title="Data.frame vs Data.table in R?" Tags="|data-mining|r|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6487" PostTypeId="2" ParentId="6486" CreationDate="2015-07-17T10:13:29.843" Score="3" Body="&lt;p&gt;You have a nice free course about it on DataCamp&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://www.datacamp.com/courses/data-table-data-manipulation-r-tutorial&quot; rel=&quot;nofollow&quot;&gt;Data Analysis in R, the data.table Way&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The author of the course is Matt Dowle, the guy that actually wrote the data.table package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="10727" LastActivityDate="2015-07-17T10:13:29.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6490" PostTypeId="1" CreationDate="2015-07-17T14:17:23.207" Score="4" ViewCount="68" Body="&lt;p&gt;I have a two-class prediction model; it has &lt;code&gt;n&lt;/code&gt; configurable (numeric) parameters. The model can work pretty well if you tune those parameters properly, but the specific values for those parameters are hard to find. I used grid search for that (providing, say, &lt;code&gt;m&lt;/code&gt; values for each parameter). This yields &lt;code&gt;m ^ n&lt;/code&gt; times to learn, and it is &lt;em&gt;very&lt;/em&gt; time-consuming even when run in parallel on a machine with 24 cores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried fixing all parameters but one and changing this only one parameter (which yields &lt;code&gt;m × n&lt;/code&gt; times), but it's not obvious for me what to do with the results I got. This is a sample plot of precision (triangles) and recall (dots) for negative (red) and positive (blue) samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NTMtd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NTMtd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply taking the &quot;winner&quot; values for each parameter obtained this way and combining them doesn't lead to best (or even good) prediction results. I thought about building regression on parameter sets with precision/recall as dependent variable, but I don't think that regression with more than 5 independent variables will be much faster than grid search scenario. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would you propose to find good parameter values, but with reasonable estimation time?  &lt;/p&gt;&#xA;" OwnerUserId="10751" LastEditorUserId="29575" LastEditDate="2017-12-15T16:10:09.610" LastActivityDate="2017-12-15T16:10:09.610" Title="Parameter estimation: reduce time" Tags="|parameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6491" PostTypeId="2" ParentId="6490" CreationDate="2015-07-17T17:39:28.300" Score="2" Body="&lt;p&gt;If an exhaustive nonlinear scan is too expensive and a linear scan doesn't yield the best results then I suggest you try a stochastic nonlinear search i.e. a random search for hyperparameter optimization.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow&quot;&gt;Scikit learn&lt;/a&gt; has a user friendly description in its user guide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf&quot; rel=&quot;nofollow&quot;&gt;paper on random search for hyperparameter optimization&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-17T17:39:28.300" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6492" PostTypeId="1" AcceptedAnswerId="6493" CreationDate="2015-07-17T18:50:39.330" Score="3" ViewCount="5088" Body="&lt;p&gt;I have the following CSV data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shot_id,round_id,hole,shotType,clubType,desiredShape,lineDirection,shotQuality,note&#xA;48,2,1,tee,driver,straight,straight,good,&#xA;49,2,1,approach,iron,straight,right,bad,&#xA;50,2,1,approach,wedge,straight,straight,bad,&#xA;51,2,1,approach,wedge,straight,straight,bad,&#xA;52,2,1,putt,putter,straight,straight,good,&#xA;53,2,1,putt,putter,straight,straight,good,&#xA;54,2,2,tee,driver,draw,straight,good,&#xA;55,2,2,approach,iron,draw,straight,good,&#xA;56,2,2,putt,putter,straight,straight,good,&#xA;57,2,2,putt,putter,straight,straight,good,&#xA;58,2,3,tee,driver,draw,straight,good,&#xA;59,2,3,approach,iron,straight,right,good,&#xA;60,2,3,chip,wedge,straight,straight,good,&#xA;61,2,3,putt,putter,straight,straight,good,&#xA;62,2,4,tee,iron,straight,straight,good,&#xA;63,2,4,putt,putter,straight,straight,good,&#xA;64,2,4,putt,putter,straight,straight,good,&#xA;65,2,5,tee,driver,straight,left,good,&#xA;66,2,5,approach,wedge,straight,straight,good,&#xA;67,2,5,putt,putter,straight,straight,bad,&#xA;68,2,5,putt,putter,straight,straight,good,&#xA;69,2,6,tee,driver,draw,straight,bad,&#xA;70,2,6,approach,hybrid,draw,straight,good,&#xA;71,2,6,putt,putter,straight,straight,good,&#xA;72,2,6,putt,putter,straight,straight,good,&#xA;73,2,7,tee,driver,straight,straight,good,&#xA;74,2,7,approach,wood,fade,straight,good,&#xA;75,2,7,approach,wedge,straight,straight,bad,long&#xA;76,2,7,putt,putter,straight,straight,good,&#xA;77,2,7,putt,putter,straight,straight,good,&#xA;78,2,8,tee,iron,straight,right,bad,&#xA;79,2,8,approach,wedge,straight,straight,good,&#xA;80,2,8,putt,putter,straight,straight,bad,&#xA;81,2,9,tee,driver,straight,straight,good,&#xA;82,2,9,approach,iron,straight,straight,good,&#xA;83,2,9,approach,wedge,straight,straight,bad,&#xA;84,2,9,putt,putter,straight,straight,good,&#xA;85,2,9,putt,putter,straight,straight,good,&#xA;86,2,10,tee,driver,straight,left,good,&#xA;87,2,10,approach,iron,straight,left,good,&#xA;88,2,10,chip,wedge,straight,straight,good,&#xA;89,2,10,putt,putter,straight,straight,good,&#xA;90,2,10,putt,putter,straight,straight,good,&#xA;91,2,11,tee,driver,draw,straight,good,&#xA;92,2,11,approach,iron,draw,straight,good,&#xA;93,2,11,putt,putter,straight,straight,good,&#xA;94,2,11,putt,putter,straight,straight,good,&#xA;95,2,12,tee,iron,draw,straight,good,&#xA;96,2,12,putt,putter,straight,straight,good,&#xA;97,2,12,putt,putter,straight,straight,good,&#xA;98,2,13,tee,driver,draw,straight,good,&#xA;99,2,13,approach,wood,straight,straight,bad,topped&#xA;100,2,13,putt,putter,straight,straight,good,&#xA;101,2,13,putt,putter,straight,straight,good,&#xA;102,2,14,tee,driver,draw,straight,good,&#xA;103,2,14,approach,wood,straight,straight,bad,&#xA;104,2,14,approach,iron,draw,straight,good,&#xA;105,2,14,approach,wedge,straight,straight,bad,&#xA;106,2,14,putt,putter,straight,straight,bad,&#xA;107,2,14,putt,putter,straight,straight,good,&#xA;108,2,15,tee,iron,draw,right,bad,&#xA;109,2,15,approach,wedge,straight,straight,good,&#xA;110,2,15,putt,putter,straight,straight,good,&#xA;111,2,15,putt,putter,straight,straight,good,&#xA;112,2,16,tee,driver,draw,right,good,&#xA;113,2,16,approach,iron,straight,left,bad,&#xA;114,2,16,approach,wedge,straight,left,bad,&#xA;115,2,16,putt,putter,straight,straight,good,&#xA;116,2,17,tee,driver,straight,straight,good,&#xA;117,2,17,approach,wood,straight,right,bad,&#xA;118,2,17,approach,wedge,straight,straight,good,&#xA;119,2,17,putt,putter,straight,straight,good,&#xA;120,2,17,putt,putter,straight,straight,good,&#xA;121,2,18,tee,driver,fade,right,bad,&#xA;122,2,18,approach,wedge,straight,straight,good,&#xA;123,2,18,approach,wedge,straight,straight,good,&#xA;124,2,18,putt,putter,straight,straight,good,&#xA;125,2,18,putt,putter,straight,straight,good,&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I would like to be able to identify which combinations of values are the most frequently occurring.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;club types:&lt;/strong&gt; driver, wood, iron, wedge, putter&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Shot types:&lt;/strong&gt; tee, approach, chip, putt&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;line directions:&lt;/strong&gt; left, center, right&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;shot qualities:&lt;/strong&gt; good, bad, neutral&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Where ideally I'd be able to identify a sweet spot (no pun intended) combination: &quot;driver&quot; + &quot;tee&quot; + &quot;straight&quot; + &quot;good&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I intend only to measure this for a static dataset, not for any future values or prediction. So, my thought is that this is probably a clustering / k-means problem. Is that correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, how would I begin doing a K-Mean analysis with these types of values in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it isn't a kmeans problem, then what is it?&lt;/p&gt;&#xA;" OwnerUserId="10761" LastActivityDate="2015-07-17T19:57:02.793" Title="How to calculate most frequent value combinations" Tags="|r|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6493" PostTypeId="2" ParentId="6492" CreationDate="2015-07-17T19:30:02.820" Score="2" Body="&lt;p&gt;If I understand your question you want to know which combination is most frequent or how frequent a combination is relative to others. This is a static method that will determine the unique combinations in total (i.e., combinations of all five columns). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;plyr&lt;/code&gt; package has a nifty utility for grouping unique combinations of columns in a &lt;code&gt;data.frame&lt;/code&gt;. We can specify the names of the columns we want to group by, and then specify a function to perform for each of those combinations. In this case, we specify the columns associated with your golf shot qualities and use the function &lt;code&gt;nrow&lt;/code&gt; which will count the number of rows in every subset of the large data.frame for which the columns are the identical.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# You need this library for the ddply() function&#xA;require(plyr)&#xA;&#xA;# These are the columns that determine a unique situation (change this if you need)&#xA;qualities &amp;lt;- c(&quot;shotType&quot;,&quot;clubType&quot;,&quot;desiredShape&quot;,&quot;lineDirection&quot;,&quot;shotQuality&quot;)&#xA;&#xA;# The call to ddply() actually gives us what we want, which is the number &#xA;# of times that combination is present in the dataset&#xA;countedCombos &amp;lt;- ddply(golf,qualities,nrow)&#xA;&#xA;# To be nice, let's give that newly added column a meaningful name&#xA;names(countedCombos) &amp;lt;- c(qualities,&quot;count&quot;)&#xA;&#xA;# Finally, you probably want to order it (decreasing, in this case)&#xA;countedCombos &amp;lt;- countedCombos[with(countedCombos, order(-count)),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now check out your product. The final column has the count associated with each unique combination of columns you provided to &lt;code&gt;ddply&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(countedCombos)&#xA;   shotType clubType desiredShape lineDirection shotQuality count&#xA;16     putt   putter     straight      straight        good    30&#xA;10 approach    wedge     straight      straight        good     6&#xA;9  approach    wedge     straight      straight         bad     5&#xA;19      tee   driver         draw      straight        good     5&#xA;22      tee   driver     straight      straight        good     4&#xA;2  approach     iron         draw      straight        good     3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To see the results for a particular cross-section (say, for example, the driver &lt;code&gt;clubType&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;countedCombos[which(countedCombos$clubType==&quot;driver&quot;),]&#xA;   shotType clubType desiredShape lineDirection shotQuality count&#xA;19      tee   driver         draw      straight        good     5&#xA;22      tee   driver     straight      straight        good     4&#xA;21      tee   driver     straight          left        good     2&#xA;17      tee   driver         draw         right        good     1&#xA;18      tee   driver         draw      straight         bad     1&#xA;20      tee   driver         fade         right         bad     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As a bonus, you can dig into these results with &lt;code&gt;ddply&lt;/code&gt; again. For example, if you wanted to look at the ratio of &quot;good&quot; to &quot;bad&quot; shotQuality based on &lt;code&gt;shotType&lt;/code&gt; and &lt;code&gt;clubType&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shotPerformance &amp;lt;- ddply(countedCombos,c(&quot;shotType&quot;,&quot;clubType&quot;),&#xA;    function(x){&#xA;        total&amp;lt;- length(x$shotQuality)&#xA;            good &amp;lt;- length(which(x$shotQuality==&quot;good&quot;))&#xA;        bad &amp;lt;- length(which(x$shotQuality==&quot;bad&quot;))&#xA;        c(total,good,bad,good/(good+bad))&#xA;    }&#xA; )&#xA;names(shotPerformance)&amp;lt;-c(&quot;count&quot;,&quot;shotType&quot;,&quot;clubType&quot;,&quot;good&quot;,&quot;bad&quot;,&quot;goodPct&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives you a new breakdown of some math performed on the counts of a character field (&lt;code&gt;shotQuality&lt;/code&gt;) and shows you how you can build custom functions for &lt;code&gt;ddply&lt;/code&gt;. Of course, you can still order these whichever way you want, too.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(shotPerformance)&#xA;  shotType clubType total good bad   goodPct&#xA;1 approach   hybrid    1  1   0 1.0000000&#xA;2 approach     iron    6  4   2 0.6666667&#xA;3 approach    wedge    3  1   2 0.3333333&#xA;4 approach     wood    3  1   2 0.3333333&#xA;5     chip    wedge    1  1   0 1.0000000&#xA;6     putt   putter    2  1   1 0.5000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9956" LastEditorUserId="9956" LastEditDate="2015-07-17T19:57:02.793" LastActivityDate="2015-07-17T19:57:02.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6494" PostTypeId="1" AcceptedAnswerId="6496" CreationDate="2015-07-17T21:04:15.433" Score="4" ViewCount="211" Body="&lt;p&gt;I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states &lt;code&gt;[a b c]&lt;/code&gt;. For now I am collection data and calculating transitional probabilities:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;T[a][b] = #transitions from a to b / #total transitions to a&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I am stuck at determining the correct Transition Matrix. As I am getting more data, the matrix is changing drastically. So when do I finalize Transition Matrix? Does that mean that my data is too random and cannot be modelled or I am doing some mistake here?&lt;/p&gt;&#xA;" OwnerUserId="8338" LastActivityDate="2015-10-16T07:40:49.880" Title="Markov Chains: How much steps to conclude a Transition Matrix" Tags="|machine-learning|markov-process|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6496" PostTypeId="2" ParentId="6494" CreationDate="2015-07-18T06:29:29.183" Score="3" Body="&lt;p&gt;I expect you have, or can make, a matrix of transition counts. Consider the data in each row to be draws from a multinomial distribution. Then you should be able to use &lt;a href=&quot;https://stats.stackexchange.com/questions/19120/sample-size-for-a-variable-number-of-answers&quot;&gt;sample size calculations for the multinomial&lt;/a&gt; to get off the ground. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible that your data is not well described by a simple Markov chain. There are some available techniques for this, e.g. &lt;a href=&quot;https://cran.r-project.org/web/packages/msm/vignettes/msm-manual.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;multistate modelling&lt;/a&gt;, but which may or may not fit your particular problem. &lt;/p&gt;&#xA;" OwnerUserId="5303" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-07-18T06:29:29.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6497" PostTypeId="2" ParentId="6433" CreationDate="2015-07-18T16:58:37.283" Score="2" Body="&lt;p&gt;In gereral, there are four ways one can &quot;connect&quot; neural networks (depending on you application at hand) as described in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.2383&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Combining Artificial Neural Networks, Sharkey et al.&lt;/a&gt;:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/d2RrP.png&quot; alt=&quot;Four ways to connect &quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the cooperative mode, there are various ways in which one can combine the decisions made by different models. One common way is to take the average of the predictions. Other ways are taking the median, or some weighed average of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the competitive mode, it is winner-take-all, where the prediction from the best model for that test point is selected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sequential mode, you classify on coarse classes (i.e. natural objects vs. man made objects), the prediction from this stage would be used to decide what to do next..(i.e. if the model predicts, man-made objects, one can have a classifier to do fine grained classification).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks have lot of &lt;a href=&quot;http://colinraffel.com/wiki/neural_network_hyperparameters&quot; rel=&quot;nofollow noreferrer&quot;&gt;hyper-parameters&lt;/a&gt; to tune. In the supervisory mode, one can train an neuralnet just to select these hyper-parameters of a &quot;real&quot; neuralnet that is actually trained on your problem at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please dig deeper into the above mentioned article for more details.&lt;/p&gt;&#xA;" OwnerUserId="10383" LastEditorUserId="10383" LastEditDate="2015-07-18T17:26:04.860" LastActivityDate="2015-07-18T17:26:04.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6498" PostTypeId="1" CreationDate="2015-07-18T19:39:18.780" Score="2" ViewCount="92" Body="&lt;p&gt;Which treebanks are based on an XML format? &#xA;What is the advantage of XML format for a treebank? I think it may have effects on annotation and querying the treebank. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example &lt;a href=&quot;http://www.let.rug.nl/vannoord/Lassy/&quot; rel=&quot;nofollow&quot;&gt;LASSY&lt;/a&gt; and Alpino or TIGER are in xml format.&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="3436" LastEditDate="2015-07-21T07:45:03.710" LastActivityDate="2021-07-13T01:00:14.783" Title="Advantage of a treebank in XML format" Tags="|nlp|data-formats|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6499" PostTypeId="2" ParentId="716" CreationDate="2015-07-18T22:10:05.050" Score="1" Body="&lt;p&gt;If you are using scikit-learn, there is a good function available called &lt;code&gt;model.feature_importances_&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Give it a try with your model/new feature and see if it helps. Also look &lt;a href=&quot;http://machinelearningmastery.com/feature-selection-in-python-with-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://blog.datadive.net/selecting-good-features-part-iii-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for examples.&lt;/p&gt;&#xA;" OwnerUserId="10383" LastEditorUserId="29169" LastEditDate="2021-03-11T19:26:46.897" LastActivityDate="2021-03-11T19:26:46.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6500" PostTypeId="1" AcceptedAnswerId="6877" CreationDate="2015-07-19T02:40:32.413" Score="2" ViewCount="3893" Body="&lt;p&gt;I am new to R and hence my question is likely to be basic. I tried researching the answer before posting here however didn't get the answer I was looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to order a data frame on the basis of certain columns. The column is dynamic and depends on a certain &quot;outcome&quot; which would be from a vector of &quot;validoutcomes&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;validoutcomes &amp;lt;- c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if outcome is &quot;A&quot; then I am supposed to read data from column 11. For &quot;B&quot; its columns 17 and for &quot;C&quot; it is 23.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I renamed the columns so that it is easier for my readability and alignment with  validoutcomes. &quot;input&quot; is my source data from the CSV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;colnames(input)[11] &amp;lt;- validoutcomes[1]&#xA; colnames(input)[17] &amp;lt;- validoutcomes[2]&#xA; colnames(input)[23] &amp;lt;- validoutcomes[3]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then tried to sort using&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;statelist &amp;lt;- statelist[order(statelist$outcome)]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2015-08-23T17:30:57.120" Title="Sort a data frame when column name is assigned in a variable" Tags="|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6503" PostTypeId="2" ParentId="326" CreationDate="2015-07-19T14:54:16.683" Score="4" Body="&lt;p&gt;I haven't tried R (well, a bit, but not enough to make a good comparison). However, here are some of Pythons strengths:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Very intuitive syntax&lt;/strong&gt;: tuple unpacking, &lt;code&gt;element in a_list&lt;/code&gt;, &lt;code&gt;for element in sequence&lt;/code&gt;, &lt;code&gt;matrix_a * matrix_b&lt;/code&gt; (for matrix multiplication), ...&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Many libraries&lt;/strong&gt;:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scipy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy&lt;/a&gt;: Scientific computations; many parts of it are only wrappers for pretty fast Fortran code&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning.net/software/theano/&quot; rel=&quot;nofollow noreferrer&quot;&gt;theano&lt;/a&gt; &gt; &lt;a href=&quot;http://lasagne.readthedocs.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lasagne&lt;/a&gt; &gt; &lt;a href=&quot;https://github.com/dnouri/nolearn&quot; rel=&quot;nofollow noreferrer&quot;&gt;nolearn&lt;/a&gt;: Libraries for neural networks - they can be trained on GPU (nvidia, CUDA is required) without any adjustment&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;: General learning algorithms&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Good community&lt;/strong&gt;:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python has &lt;a href=&quot;https://stackoverflow.com/questions/tagged/python&quot;&gt;448,000+ questions on SO&lt;/a&gt; (R has 99,000+)&lt;/li&gt;&#xA;&lt;li&gt;Python has &lt;a href=&quot;https://datascience.stackexchange.com/questions/tagged/python&quot;&gt;103 questions on datascience.SE&lt;/a&gt; (R has 140)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.python.org/dev/peps/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PEPs&lt;/a&gt;, 63,000+ packages on &lt;a href=&quot;https://pypi.python.org/pypi&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyPI&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ipython.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;IPython notebooks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Misc&lt;/strong&gt;:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0-indexed arrays ... I made that error all the time with R.&lt;/li&gt;&#xA;&lt;li&gt;Established package structures&lt;/li&gt;&#xA;&lt;li&gt;Good support for testing your code&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-07-19T14:54:16.683" CommentCount="0" CommunityOwnedDate="2015-07-19T14:54:16.683" ContentLicense="CC BY-SA 3.0" />
  <row Id="6505" PostTypeId="2" ParentId="6500" CreationDate="2015-07-19T21:10:09.263" Score="0" Body="&lt;p&gt;Ok, I have found the likely answer. I am not sure if this is the best answer but it worked for me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since &lt;code&gt;$&lt;/code&gt; is used to call the colnames, R was looking for a column name as &lt;code&gt;outcome&lt;/code&gt;. However all it needed was square brackets&lt;code&gt;[]&lt;/code&gt;. I replaced it and so far seems to be working for me.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;statelist &amp;lt;- statelist[order(statelist[outcome])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am still very interested in knowing if there is a better solution or if someone sees issue with my observation.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2015-07-19T21:10:09.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6506" PostTypeId="1" AcceptedAnswerId="6521" CreationDate="2015-07-20T04:48:17.547" Score="12" ViewCount="10303" Body="&lt;p&gt;I want to compute the semantic similarity of two words using their vector representations (obtained using e.g. word2vec, GloVe, etc.). Shall I use the Euclidean Distance or the Cosine Similarity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;noreferrer&quot;&gt;GloVe website&lt;/a&gt; mentions both measures without telling the pros and cons of each:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Euclidean distance (or cosine similarity) between two word vectors provides an effective method for measuring the linguistic or semantic similarity of the corresponding words.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-07-28T17:55:16.293" LastActivityDate="2021-04-25T14:47:10.563" Title="Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words?" Tags="|nlp|word-embeddings|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6507" PostTypeId="1" CreationDate="2015-07-20T06:15:34.527" Score="1" ViewCount="635" Body="&lt;pre&gt;&lt;code&gt;custid  tot_prod       tot_rev          Last_tran   total_trans&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;1   1002          13       1465.25         11/23/2011       4&#xA;2   1003          2         353.25          1/1/2011        1&#xA;3   1004          12        2466.5          11/25/2011      12&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Above is the Retail dataset of customer. I need to calculate recency(a new attribute when the customer is last seen) from last transaction(last_tran)  in terms of month.i.e. he purchased in last 3 months or last 6 months or may be one year in R. please suggest.    &lt;/p&gt;&#xA;" OwnerUserId="10416" LastActivityDate="2015-08-19T16:50:26.183" Title="how to calculate recency(RFM) from a given date attribute in R" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6508" PostTypeId="1" CreationDate="2015-07-20T08:03:21.013" Score="25" ViewCount="19175" Body="&lt;p&gt;I'm trying to cluster some vectors with 90 features with K-means. Since this algorithm asks me the number of clusters, I want to validate my choice with some nice math.&#xA;I expect to have from 8 to 10 clusters. The features are Z-score scaled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Elbow method and variance explained&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.spatial.distance import cdist, pdist&#xA;from sklearn.cluster import KMeans&#xA;&#xA;K = range(1,50)&#xA;KM = [KMeans(n_clusters=k).fit(dt_trans) for k in K]&#xA;centroids = [k.cluster_centers_ for k in KM]&#xA;&#xA;D_k = [cdist(dt_trans, cent, 'euclidean') for cent in centroids]&#xA;cIdx = [np.argmin(D,axis=1) for D in D_k]&#xA;dist = [np.min(D,axis=1) for D in D_k]&#xA;avgWithinSS = [sum(d)/dt_trans.shape[0] for d in dist]&#xA;&#xA;# Total with-in sum of square&#xA;wcss = [sum(d**2) for d in dist]&#xA;tss = sum(pdist(dt_trans)**2)/dt_trans.shape[0]&#xA;bss = tss-wcss&#xA;&#xA;kIdx = 10-1&#xA;&#xA;# elbow curve&#xA;fig = plt.figure()&#xA;ax = fig.add_subplot(111)&#xA;ax.plot(K, avgWithinSS, 'b*-')&#xA;ax.plot(K[kIdx], avgWithinSS[kIdx], marker='o', markersize=12, &#xA;markeredgewidth=2, markeredgecolor='r', markerfacecolor='None')&#xA;plt.grid(True)&#xA;plt.xlabel('Number of clusters')&#xA;plt.ylabel('Average within-cluster sum of squares')&#xA;plt.title('Elbow for KMeans clustering')&#xA;&#xA;fig = plt.figure()&#xA;ax = fig.add_subplot(111)&#xA;ax.plot(K, bss/tss*100, 'b*-')&#xA;plt.grid(True)&#xA;plt.xlabel('Number of clusters')&#xA;plt.ylabel('Percentage of variance explained')&#xA;plt.title('Elbow for KMeans clustering')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/VZZgt.png&quot; alt=&quot;Elbow method&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/NIyI2.png&quot; alt=&quot;Variance&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From these two pictures, it seems that the number of clusters never stops :D. Strange! Where is the elbow? How can I choose K?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bayesian information criterion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This methods comes directly from X-means and uses the &lt;a href=&quot;http://Bayesian%20information%20criterion&quot; rel=&quot;noreferrer&quot;&gt;BIC&lt;/a&gt; to choose the number of clusters. &lt;a href=&quot;https://stackoverflow.com/questions/1793532/how-do-i-determine-k-when-using-k-means-clustering&quot;&gt;another ref&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    from sklearn.metrics import euclidean_distances&#xA;from sklearn.cluster import KMeans&#xA;&#xA;def bic(clusters, centroids):&#xA;    num_points = sum(len(cluster) for cluster in clusters)&#xA;    num_dims = clusters[0][0].shape[0]&#xA;    log_likelihood = _loglikelihood(num_points, num_dims, clusters, centroids)&#xA;    num_params = _free_params(len(clusters), num_dims)&#xA;    return log_likelihood - num_params / 2.0 * np.log(num_points)&#xA;&#xA;&#xA;def _free_params(num_clusters, num_dims):&#xA;    return num_clusters * (num_dims + 1)&#xA;&#xA;&#xA;def _loglikelihood(num_points, num_dims, clusters, centroids):&#xA;    ll = 0&#xA;    for cluster in clusters:&#xA;        fRn = len(cluster)&#xA;        t1 = fRn * np.log(fRn)&#xA;        t2 = fRn * np.log(num_points)&#xA;        variance = _cluster_variance(num_points, clusters, centroids) or np.nextafter(0, 1)&#xA;        t3 = ((fRn * num_dims) / 2.0) * np.log((2.0 * np.pi) * variance)&#xA;        t4 = (fRn - 1.0) / 2.0&#xA;        ll += t1 - t2 - t3 - t4&#xA;    return ll&#xA;&#xA;def _cluster_variance(num_points, clusters, centroids):&#xA;    s = 0&#xA;    denom = float(num_points - len(centroids))&#xA;    for cluster, centroid in zip(clusters, centroids):&#xA;        distances = euclidean_distances(cluster, centroid)&#xA;        s += (distances*distances).sum()&#xA;    return s / denom&#xA;&#xA;from scipy.spatial import distance&#xA;def compute_bic(kmeans,X):&#xA;    &quot;&quot;&quot;&#xA;    Computes the BIC metric for a given clusters&#xA;&#xA;    Parameters:&#xA;    -----------------------------------------&#xA;    kmeans:  List of clustering object from scikit learn&#xA;&#xA;    X     :  multidimension np array of data points&#xA;&#xA;    Returns:&#xA;    -----------------------------------------&#xA;    BIC value&#xA;    &quot;&quot;&quot;&#xA;    # assign centers and labels&#xA;    centers = [kmeans.cluster_centers_]&#xA;    labels  = kmeans.labels_&#xA;    #number of clusters&#xA;    m = kmeans.n_clusters&#xA;    # size of the clusters&#xA;    n = np.bincount(labels)&#xA;    #size of data set&#xA;    N, d = X.shape&#xA;&#xA;    #compute variance for all clusters beforehand&#xA;    cl_var = (1.0 / (N - m) / d) * sum([sum(distance.cdist(X[np.where(labels == i)], [centers[0][i]], 'euclidean')**2) for i in range(m)])&#xA;&#xA;    const_term = 0.5 * m * np.log(N) * (d+1)&#xA;&#xA;    BIC = np.sum([n[i] * np.log(n[i]) -&#xA;               n[i] * np.log(N) -&#xA;             ((n[i] * d) / 2) * np.log(2*np.pi*cl_var) -&#xA;             ((n[i] - 1) * d/ 2) for i in range(m)]) - const_term&#xA;&#xA;    return(BIC)&#xA;&#xA;&#xA;&#xA;sns.set_style(&quot;ticks&quot;)&#xA;sns.set_palette(sns.color_palette(&quot;Blues_r&quot;))&#xA;bics = []&#xA;for n_clusters in range(2,50):&#xA;    kmeans = KMeans(n_clusters=n_clusters)&#xA;    kmeans.fit(dt_trans)&#xA;&#xA;    labels = kmeans.labels_&#xA;    centroids = kmeans.cluster_centers_&#xA;&#xA;    clusters = {}&#xA;    for i,d in enumerate(kmeans.labels_):&#xA;        if d not in clusters:&#xA;            clusters[d] = []&#xA;        clusters[d].append(dt_trans[i])&#xA;&#xA;    bics.append(compute_bic(kmeans,dt_trans))#-bic(clusters.values(), centroids))&#xA;&#xA;plt.plot(bics)&#xA;plt.ylabel(&quot;BIC score&quot;)&#xA;plt.xlabel(&quot;k&quot;)&#xA;plt.title(&quot;BIC scoring for K-means cell's behaviour&quot;)&#xA;sns.despine()&#xA;#plt.savefig('figures/K-means-BIC.pdf', format='pdf', dpi=330,bbox_inches='tight')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/lon6L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same problem here... What is K?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Silhouette&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    from sklearn.metrics import silhouette_score&#xA;&#xA;s = []&#xA;for n_clusters in range(2,30):&#xA;    kmeans = KMeans(n_clusters=n_clusters)&#xA;    kmeans.fit(dt_trans)&#xA;&#xA;    labels = kmeans.labels_&#xA;    centroids = kmeans.cluster_centers_&#xA;&#xA;    s.append(silhouette_score(dt_trans, labels, metric='euclidean'))&#xA;&#xA;plt.plot(s)&#xA;plt.ylabel(&quot;Silouette&quot;)&#xA;plt.xlabel(&quot;k&quot;)&#xA;plt.title(&quot;Silouette for K-means cell's behaviour&quot;)&#xA;sns.despine()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Znt5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alleluja! Here it seems to make sense and this is what I expect. But why is this different from the others?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2019-03-14T17:09:41.220" Title="K-means incoherent behaviour choosing K with Elbow method, BIC, variance explained and silhouette" Tags="|clustering|k-means|" AnswerCount="3" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="6509" PostTypeId="1" CreationDate="2015-07-20T10:52:59.633" Score="2" ViewCount="154" Body="&lt;p&gt;What are the techniques to infer missing time-stamps in the unevenly spaced time series data that has missing time-stamps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequency of data recording is about 1-3 times a day without any fixed time of measurement. In some cases the time stamp of the measurement is unavailable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to predict at which time $t_0$ the new sample $x_0$ was recorded knowing the history of previous measurements $(t_n,x_n)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;(t_{-n},x_{-n}), ... ,(t_{-3},x_{-3}), (t_{-2},x_{-2}), (t_{-1},x_{-1}), (t_0,x_0)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or predict multiple missing time-stamps e.g. $t_{-2},t_{-6},t_{-7}$ from recorded sequence $t_n,x_n$?&lt;/p&gt;&#xA;" OwnerUserId="10790" LastActivityDate="2015-07-20T10:52:59.633" Title="Techniques for dealing with unevenly spaced time series data that have missing time-stamps?" Tags="|predictive-modeling|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6510" PostTypeId="1" CreationDate="2015-07-20T13:42:21.283" Score="35" ViewCount="60302" Body="&lt;p&gt;As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the other hand, the original source states that the fact OOB error is calculated during model training is enough of an indicator of test set performance. Even Trevor Hastie, in a relatively recent talks says that &quot;Random Forests provide free cross-validation&quot;. Intuitively, this makes sense to me, if training and trying to improve a RF-based model on one dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please lay out the arguments for and against the need for cross-validation with random forests?&lt;/p&gt;&#xA;" OwnerUserId="1127" LastEditorUserId="10327" LastEditDate="2018-08-23T18:39:53.467" LastActivityDate="2018-08-23T18:39:53.467" Title="Does modeling with Random Forests require cross-validation?" Tags="|random-forest|cross-validation|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="6511" PostTypeId="1" AcceptedAnswerId="6518" CreationDate="2015-07-20T15:54:31.440" Score="1" ViewCount="3905" Body="&lt;p&gt;Linear regression is a widely used ML algorithm. So far I have only encountered 'boring' applications of it. (e.g predict sales for next quarter, predict housing prices for next year , predict population of a country by 2020 etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some interesting/cool application of linear regression? Finding cool applications greatly helps in motivating myself to learn, hence the quest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cool as in: Application to stock trading, application to video games, application to astronomy, application to sports betting, application to airfare prediction&lt;/p&gt;&#xA;" OwnerUserId="10522" LastEditorUserId="10522" LastEditDate="2015-07-20T16:04:38.073" LastActivityDate="2015-07-20T19:08:46.603" Title="Are there any interesting application of linear regression" Tags="|linear-regression|" AnswerCount="2" CommentCount="1" ClosedDate="2015-07-20T21:16:43.640" ContentLicense="CC BY-SA 3.0" />
  <row Id="6512" PostTypeId="2" ParentId="6511" CreationDate="2015-07-20T15:59:25.220" Score="0" Body="&lt;p&gt;I think those were the coolest.&#xA;I recommend you that video also&#xA;&lt;a href=&quot;https://www.khanacademy.org/math/probability/regression/regression-correlation/v/regression-line-example&quot; rel=&quot;nofollow&quot;&gt;https://www.khanacademy.org/math/probability/regression/regression-correlation/v/regression-line-example&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2015-07-20T15:59:25.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6513" PostTypeId="1" CreationDate="2015-07-20T16:00:53.090" Score="2" ViewCount="314" Body="&lt;p&gt;I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable level(=0.05) I'm considering ways of controlling the false positives rate while keeping the sample size as low as I can(it's for a low traffic website)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aware of the Bonferroni, Benjiamimi-Hochberg and related methods designed to control the proportion of false positives in the multiple comparisons situation. However the problem of computing the sample size still remains, the simplest approach seems the one of computing the required sample size starting from alpha = 0.05/n as dictated by the Bonferroni correction and then perhaps use Benjiamimi-Hochberg for the actual testing. Is there a better way of computing the sample size when Benjiamimi-Hochberg is used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further as the A/B tests are designed to test possible improvements I was thinking that if the sequential tests are based on independent samples then then probability of two consecutive false positives would be alpha^2 = 0.025. My understanding is that the sequential tests would be independent if the respective samples do not refer to the same users. These could for example be users that joined the website after the previous test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the above thinking makes some sense, I could use a &quot;weak&quot; Bonferroni (i.e. alpha' = alpha/(n*0.2) )correction in order to run tests with a reasonable power and lower sample size and once a positive is found(the null hypothesis is rejected), I could repeat the test and if it's positive again then the result is positive with p &amp;lt; 0.025 which means that I would accept the result. This approach would allow for some control over the sample size but... is it sound?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any opinion would be greatly appreciated. Thanks in advance. AMAR&lt;/p&gt;&#xA;" OwnerUserId="10793" LastEditorUserId="11097" LastEditDate="2016-01-06T11:47:03.097" LastActivityDate="2017-10-12T23:30:07.010" Title="How to control false positives in sequential A/B testing while keeping a low sample size?" Tags="|data-mining|ab-test|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6514" PostTypeId="2" ParentId="6507" CreationDate="2015-07-20T16:10:10.593" Score="2" Body="&lt;p&gt;Try this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;as.POSIXct(Sys.Date(), format= &quot;%Y-%m-%d&quot;, tz=&quot;UCT&quot;) - as.POSIXct(&quot;11/25/2011&quot;, format = &quot;%m/%d/%Y&quot;,tz=&quot;UCT&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More vectorized operation in your situation&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;as.POSIXct(Sys.Date(), format= &quot;%Y-%m-%d&quot;, tz=&quot;UCT&quot;) - as.POSIXct(Last_tran, format = &quot;%m/%d/%Y&quot;,tz=&quot;UCT&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2015-07-20T16:10:10.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6515" PostTypeId="1" AcceptedAnswerId="6517" CreationDate="2015-07-20T16:15:54.270" Score="2" ViewCount="3784" Body="&lt;p&gt;I have a table, more or less in the following format&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col1 col2 col3 ... col100&#xA;val1 val2 val3 ... val100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where val* are doubles. Is there a way to extract for each row in which column is the highest value within row in Hive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for table like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col1 col2 col3&#xA;2     4   5&#xA;8     1   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col3&#xA;col1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5224" LastEditorUserId="2452" LastEditDate="2015-08-19T22:38:15.220" LastActivityDate="2015-11-18T00:17:28.080" Title="How to extract a column that has the highest value within row in Hive?" Tags="|sql|hive|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6517" PostTypeId="2" ParentId="6515" CreationDate="2015-07-20T17:19:25.467" Score="1" Body="&lt;p&gt;I can't test in hive, but a possible SQL query is as follows (greatest returns the maximum value from the list):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select &#xA;  case &#xA;   when col1 = greatest(col1,col2,col3) then 'col1' &#xA;   when col2 = greatest(col1,col2,col3) then 'col2'  &#xA;   when col3 = greatest(col1,col2,col3) then 'col3'    &#xA;  end as c1 &#xA;from test;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Additional note:&#xA;you should check, how ties are to be handled, in my solution I simple take the first column.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-07-20T17:19:25.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6518" PostTypeId="2" ParentId="6511" CreationDate="2015-07-20T17:22:41.453" Score="5" Body="&lt;p&gt;The way that you have phrased this question makes it tough for people to answer without first offering you some background on linear regression (LR).  Its great that you are interested in learning some ML and LR is a great place to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear regression is really just finding a line (or plain or hyperplain) that maps a relationship between two or more variables or features.  The important requirement is that the target variable be a continuous numerical value.  So its less about finding a problem for which linear regression &quot;works&quot; and more about finding some data that interests you and playing with it using linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you download some open data in the realms that you have mentioned.  There is plenty of open data in every topic that you mentioned that contain continuous numerical values that you can predict using other features of the data.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quandl.com/data/WIKI&quot; rel=&quot;nofollow&quot;&gt;stock trading&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.giantbomb.com/api/&quot; rel=&quot;nofollow&quot;&gt;video games&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://sdo.gsfc.nasa.gov/data/dataaccess.php&quot; rel=&quot;nofollow&quot;&gt;astronomy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.opensourcesports.com/&quot; rel=&quot;nofollow&quot;&gt;sports betting&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.firebase.com/docs/open-data/airports.html&quot; rel=&quot;nofollow&quot;&gt;flight time prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also think about taking some sort of online MOOC as this will help you gain some footing in the subject.  &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/home/info&quot; rel=&quot;nofollow&quot;&gt;Andrew Ng's Coursera on Machine Learning&lt;/a&gt; is highly recommended as a starting point and include some linear regression during the first portion and &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt; is a great Python based library.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-20T19:08:46.603" LastActivityDate="2015-07-20T19:08:46.603" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6519" PostTypeId="1" AcceptedAnswerId="6522" CreationDate="2015-07-20T20:13:53.770" Score="2" ViewCount="1645" Body="&lt;p&gt;I have multiple datasets, with slightly differing features. What tools can I use to make this a homogeneous dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;featureA,featureB,featureC&#xA;1,7,3&#xA;4,8,4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dataset2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;featureA,featureC,featureD,featureE&#xA;3,4,5,6&#xA;9,8,4,6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Homogeneous Dataset&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;featureA,featureB,featureC,featureD,featureE&#xA;1,7,3,,&#xA;4,8,4,,&#xA;3,,4,5,6&#xA;9,,8,4,6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10799" LastActivityDate="2015-07-24T15:55:45.900" Title="Combining Datasets with Different Features" Tags="|machine-learning|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6521" PostTypeId="2" ParentId="6506" CreationDate="2015-07-20T20:41:25.427" Score="11" Body="&lt;p&gt;First of all, if GloVe gives you &lt;em&gt;normalized&lt;/em&gt; unit vectors, then the two calculations are equivalent. In general, I would use the cosine similarity since it removes the effect of document length. For example, a postcard and a full-length book may be about the same topic, but will likely be quite far apart in pure &quot;term frequency&quot; space using the Euclidean distance. They will be right on top of each other in cosine similarity.&lt;/p&gt;&#xA;" OwnerDisplayName="user9424" LastActivityDate="2015-07-20T20:41:25.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6522" PostTypeId="2" ParentId="6519" CreationDate="2015-07-20T20:48:56.047" Score="2" Body="&lt;p&gt;You can use &lt;a href=&quot;http://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; to do that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.inside-r.org/packages/cran/gtools/docs/smartbind&quot; rel=&quot;nofollow&quot;&gt;The smartbind function&lt;/a&gt; is the perfect way to combine datsets in the way you are asking for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(gtools)&#xA;&#xA;d1&amp;lt;-as.data.frame(rbind(c(1,7,3),c(4,8,4))))&#xA;names(d1)&amp;lt;-c(&quot;featureA&quot;,&quot;featureB&quot;,&quot;featureC&quot;)&#xA;&#xA;d2&amp;lt;-as.data.frame(rbind(c(3,4,5,6),c(9,8,4,6)))&#xA;names(d2)&amp;lt;-c(&quot;featureA&quot;,&quot;featureC&quot;,&quot;featureD&quot;,&quot;featureE&quot;)&#xA;&#xA;d3&amp;lt;-smartbind(d1,d2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7714" LastActivityDate="2015-07-20T20:48:56.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6523" PostTypeId="2" ParentId="784" CreationDate="2015-07-20T21:51:46.383" Score="1" Body="&lt;p&gt;In anomaly detection (one sort of STL application), it's easier to see outliers if you can normalize the original time series to a residual series (called &quot;remainder&quot; in R's STL package). That involves identifying components that you can subtract from the original series. In STL, the components are the seasonal series and the trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a regular, recurring pattern in the data (i.e., a seasonality), then you don't want the expected ups/downs of that pattern to drive the anomaly detector. Instead you want to be able to say &quot;are we higher or lower than expected, factoring out the regular ups/downs that happen over the course of the day (or week, or month, etc.)&quot;. Identifying and removing the seasonal series allows you to ask that question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The decomposition you generated does not look right: the seasonal pattern doesn't seem to have captured much structure, the trend overfits the data, and the residuals have a regular structure that the seasonal series ought to have captured. I don't know whether this is because the dataset itself doesn't lend itself to an STL decomposition (the series tail appears qualitatively different than the initial part) or if it's because the parameters are wrong. Figure 6.10 on &lt;a href=&quot;https://www.otexts.org/fpp/6/5&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; shows a more prototypical case.&lt;/p&gt;&#xA;" OwnerDisplayName="user10803" LastActivityDate="2015-07-20T21:51:46.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6524" PostTypeId="2" ParentId="6144" CreationDate="2015-07-21T03:22:58.723" Score="2" Body="&lt;p&gt;There are many pitfalls to not scaling your data and it is generally very advisable to scale it. It is so easy to do, it is reversible, and it useful in other operations like removing outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon further analysis of the specifics of Hidden Markov Models using Multivariate Gaussians the theoretical accuracy should not suffer as a result of drastic differences in the scales of your features.  But, an important operation involving multivariate Gaussian distribution is matrix inversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though the theoretical invertibility won't change, the practical numerical solution to your problem will likely suffer inaccuracy due to the difference in scales.  Iterative methods will have issues with convergence and direct solve methods will suffer from stiffness.  This is especially true when common complications like linear dependence are present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a set of 3 lectures on HMM with some specifics on multivariate Gaussians (&lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/courses/281A-fall04/lectures/lec-10-21.pdf&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;-&lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/courses/281A-fall04/lectures/lec-10-26.pdf&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;-&lt;a href=&quot;http://www.cs.berkeley.edu/~jordan/courses/281A-fall04/lectures/lec-10-28.pdf&quot; rel=&quot;nofollow&quot;&gt;3&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know we've already discussed this in the comments, but I wanted to close out this question, so have added it as an answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-21T03:40:23.157" LastActivityDate="2015-07-21T03:40:23.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6527" PostTypeId="2" ParentId="6510" CreationDate="2015-07-21T07:26:51.743" Score="5" Body="&lt;p&gt;One key difference is that cross validation ensures all samples will appear in the training and test sets, so 100% of your data gets used at some point for training and for testing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the size of your dataset the bootstrapping , sampling with replacement, occurring in the random forest will not guarantee the splits the trees see will contain all instances. If you have enough trees in your forest the OOB estimate should asymptotically converge towards the best OOB estimate value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy for both methods will to some degree be data dependent so it may be prudent to compare both methods on the particular data you have in front of you and see if CV and RF OOB estimates give similar values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they do not, then it would be worth exploring further estimates of the true error rate, perhaps by much higher vales of K in CV.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-07-21T07:26:51.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6528" PostTypeId="1" AcceptedAnswerId="6529" CreationDate="2015-07-21T09:39:32.693" Score="1" ViewCount="456" Body="&lt;p&gt;Is there any implementation of Newton-Raphson or EM Algorithm? Can I get the source code of it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried googling, but didn't come across any. So asking here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="10810" LastActivityDate="2015-07-21T11:50:48.820" Title="Newton-Raphson or EM Algorithm in Python" Tags="|python|algorithms|" AnswerCount="1" CommentCount="2" ClosedDate="2015-07-21T18:05:25.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="6529" PostTypeId="2" ParentId="6528" CreationDate="2015-07-21T09:59:49.177" Score="1" Body="&lt;p&gt;scikit learn has the EM algorithm &lt;a href=&quot;http://scikit-learn.org/stable/modules/mixture.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source code is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you are an R fan the &lt;code&gt;mclust&lt;/code&gt; package is available &lt;a href=&quot;http://www.stat.washington.edu/mclust/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-07-21T11:50:48.820" LastActivityDate="2015-07-21T11:50:48.820" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6530" PostTypeId="1" CreationDate="2015-07-21T12:24:18.423" Score="2" ViewCount="8485" Body="&lt;p&gt;I would like to write an algorithm to convert unstructured texts (with contest descriptions) to structured data with the following fields:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;contest start date (optional)&lt;/li&gt;&#xA;&lt;li&gt;contest end date&lt;/li&gt;&#xA;&lt;li&gt;main prize&lt;/li&gt;&#xA;&lt;li&gt;additional prizes (optional)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have hundreds of text examples, that could be used for model learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to approach this task? Just in case this is important - the preferable language is Python. But I never worked on such tasks before. &#xA;What is the best practice to do that?&lt;/p&gt;&#xA;" OwnerUserId="10811" LastEditorUserId="76338" LastEditDate="2019-06-20T11:10:00.013" LastActivityDate="2019-06-20T11:10:00.013" Title="How to convert unstructured texts to structured data?" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6531" PostTypeId="1" CreationDate="2015-07-21T12:44:33.683" Score="-1" ViewCount="462" Body="&lt;p&gt;Can you please recommend me for a big dataset for k means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be cool if it will integrate easily with python, but any thing will be good.&lt;/p&gt;&#xA;" OwnerUserId="10813" LastActivityDate="2015-07-21T13:52:56.563" Title="K MEANS Big dataset" Tags="|python|dataset|k-means|" AnswerCount="1" CommentCount="1" ClosedDate="2015-07-21T18:03:09.130" ContentLicense="CC BY-SA 3.0" />
  <row Id="6532" PostTypeId="2" ParentId="6531" CreationDate="2015-07-21T13:52:56.563" Score="1" Body="&lt;p&gt;For the very first start I'd recoment synthetical data. Simple draw a K sets of random distributed numbers with different means and required dimensionality.&#xA;The big advantage is that you know the result clustering and you can easily verify the result. Also scaling the data is not problem.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-07-21T13:52:56.563" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6533" PostTypeId="1" CreationDate="2015-07-21T15:52:30.010" Score="2" ViewCount="737" Body="&lt;p&gt;I'm wondering about general approaches to storing complex tables and structures. For example, imagine I have a table like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               A1 A2 A3 A4 A5 B1 B2 B3 B4 B5 C1 ....Z5&#xA; individuals   &#xA; 1             .  .  .  .  . .  .  .  .  .&#xA; 2             .  .  .  .  . .  .  .  .  .&#xA; 3             .  .  .  .  . .  .  .  .  .&#xA; 4             .  .  .  .  . .  .  .  .  .&#xA; 5             .  .  .  .  . .  .  .  .  .&#xA; ...           .  .  .  .  . .  .  .  .  .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I'm just showing the row and column names (variables), but the table would be filled, or could be even much larger, with A1 to A999, B1 to B999 colums.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can want to perform some calculation on all $Ax$ columns, or on all $x3$ ones. Since it can be difficult to deal with so many columns, it may be better to write this as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; i  subcase  A B C ... Z&#xA; 1  1      .  .  .  .  &#xA; 1  2      .  .  .  .  &#xA; 1  3      .  .  .  .  &#xA; 1  4      .  .  .  .  &#xA; 1  5      .  .  .  .  &#xA; 1  6      .  .  .  .  &#xA; 1  7      .  .  .  .  &#xA; 1  8      .  .  .  .  &#xA; 1  9      .  .  .  .  &#xA; 2  1      .  .  .  .  &#xA; 2  2      .  .  .  .  &#xA; ...       .  .  .  .  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I remember seeing that kind of structure several times, sometimes referred to as &lt;em&gt;long format&lt;/em&gt;. Later, one can perform arbitrary operations on all rows meeting some condition on &quot;case&quot;. This has some advantages, but could be confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could average or &lt;em&gt;crate&lt;/em&gt; the differences of all the $\text{subcase}==1$ rows, only from $\text{individual}==1$, or when $\text{individual}==subcase$,  for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what would be the best way to store the resulting numbers--columnwise or rowise? If I wanted to store the average of all $\text{subcase}==1$ rows, I would need to repeat the number on every row meeting that condition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be to keep the &lt;em&gt;wide&lt;/em&gt; format structure, but having 2 rows for the headers—one for the letters and a second one for the numbers. I think it could be a natural way, but I've never seen it before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there other structures people can recommend for this problem, or for more complex situations, such as trees?&lt;/p&gt;&#xA;" OwnerUserId="10815" LastEditorUserId="13413" LastEditDate="2015-10-22T23:23:29.327" LastActivityDate="2015-10-22T23:23:29.327" Title="How to store complex tables and structures?" Tags="|data|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6534" PostTypeId="2" ParentId="6530" CreationDate="2015-07-21T17:45:54.623" Score="5" Body="&lt;p&gt;If I get you right, you would like to extract the mentioned entities from the unstructured text. This sounds as Named Entity Recognition (NER) problem for me. In your case I would define the following entities:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;START_DATE&lt;/li&gt;&#xA;&lt;li&gt;END_DATE&lt;/li&gt;&#xA;&lt;li&gt;MAIN_PRIZE&lt;/li&gt;&#xA;&lt;li&gt;EXTRA_PRIZE&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your text now can be tagged or annotated with those entities. You would like to build a machine which will automatically tag the encountered entities in the unstructured text (free text).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest to take a look at Conditional Random Fields (CRF), a classic approach to NER. There are also Recurrent Neural Networks (RNN) which are state of the art for sequence modeling but I am not an expert with those methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's begin the process of model building. You need to decide what will be a single training example. This can be a sentence and this can be also a paragraph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you did not provide any example, let me invent two:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The contest started on &amp;lt;START_DATE&amp;gt;july 1st&amp;lt;/START_DATE&amp;gt; and ended on &amp;lt;END_DATE&amp;gt;july 25&amp;lt;/END_DATE&amp;gt;.&#xA;&#xA;The winner got away with &amp;lt;MAIN_PRIZE&amp;gt;1 million dollars&amp;lt;/MAIN_PRIZE&amp;gt; and &amp;lt;EXTRA_PRIZE&amp;gt;bunch of flowers&amp;lt;/EXTRA_PRIZE&amp;gt;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice the way how the text is structured. Every word now is either label (surrounded by tags) or not labeled at all (outside of any tag). This is one way how to structure the text. This would be also the desired output on some unstructured text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The machine does not understand the text and annotation. You need to come up with features that you can extract from each word. It can be the word itself, the previous word, the next word, left bigram, right bigram, suffix of length 1,2,3, prefixes of length 1,2,3 and so on. I suggest you to look at Stanford NER features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you got the features, you need to create a training set. The training set is your annotated text (see my example) converted to label + feature information. It may look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;O   feature1 feature2 ...&#xA;O   feature1 feature2 ...&#xA;B-START_DATE   feature1 feature2 ...&#xA;I-START_DATE   feature1 feature3 ...&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This basically says that the first two words are not labeled, but the last two are part of an entity START_DATE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each line is essentially a word followed by features for it. The label &quot;O&quot; is the way to say that the word is not labeled, the labels B- and I- for DATE mean the first and second words of a named entity (e.g. &quot;july 1st&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sentences can be separated by empty lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've got the training set file ready, you can do the model training. With a model in hand, you can apply it on any unstructured data (extracted features) to get the desired entities extracted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a tool I suggest you CRFSuite (also available with Python bindings). I can detail more each step if you would like.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="7848" LastEditDate="2015-07-21T18:04:04.427" LastActivityDate="2015-07-21T18:04:04.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6535" PostTypeId="2" ParentId="6530" CreationDate="2015-07-21T19:03:19.130" Score="1" Body="&lt;p&gt;Look for NLTK, a Python package for analysing natural language data. With it, you can add some structure to text as written by humans. &lt;/p&gt;&#xA;" OwnerUserId="10171" LastActivityDate="2015-07-21T19:03:19.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6536" PostTypeId="2" ParentId="6533" CreationDate="2015-07-21T19:20:19.363" Score="0" Body="&lt;p&gt;It all depends on what you're trying to accomplish. For example, if you want to save space and you have sparse data, you want your table to grow &quot;long&quot; as you describe and let the absence of a row signify that you don't have data (rather than structuring your sparse field as a column, which would typically require space to store your empty data points). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need a data structure that makes it easiest to &quot;process&quot; the data, it depends on what your process is. Sometimes, a complex, nested data structure is best (e.g., JSON), other times, a flat table where some fields have repeating values is best. &lt;/p&gt;&#xA;" OwnerUserId="5010" LastActivityDate="2015-07-21T19:20:19.363" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6537" PostTypeId="2" ParentId="5987" CreationDate="2015-07-21T21:45:39.853" Score="6" Body="&lt;p&gt;I am first deriving the error for a convolutional layer below for simplicity for a one dimensional array (input) which can easily be transferred to a multidimensional then:&lt;/p&gt;&#xA;&lt;p&gt;We assume here that the &lt;span class=&quot;math-container&quot;&gt;$y^{l-1}$&lt;/span&gt; of length &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; are the inputs of the &lt;span class=&quot;math-container&quot;&gt;$l-1$&lt;/span&gt;-th conv. layer, &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; is the kernel-size of weights &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; denoting each weight by &lt;span class=&quot;math-container&quot;&gt;$w_i$&lt;/span&gt; and the output is &lt;span class=&quot;math-container&quot;&gt;$x^l$&lt;/span&gt;.&lt;br /&gt;&#xA;Hence we can write (note the summation from zero): &lt;span class=&quot;math-container&quot;&gt;$$x_i^l = \sum\limits_{a=0}^{m-1} w_a y_{a+i}^{l-1}$$&lt;/span&gt;&#xA;where &lt;span class=&quot;math-container&quot;&gt;$y_i^l = f(x_i^l)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; the activation function (e.g. sigmoidal).&#xA;With this at hand we can now consider some error function &lt;span class=&quot;math-container&quot;&gt;$E$&lt;/span&gt; and the error function at the convolutional layer (the one of your previous layer) given by &lt;span class=&quot;math-container&quot;&gt;$\partial E / \partial y_i^l $&lt;/span&gt;. We now want to find out the dependency of the error in one the weights in the previous layer(s):&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{equation} \frac{\partial E}{\partial w_a} = \sum\limits_{a=0}^{N-m} \frac{\partial E}{\partial x_i^l} \frac{\partial x_i^l}{\partial w_a} = \sum\limits_{a=0}^{N-m}\frac{\partial E}{\partial w_a} y_{i+a}^{l-1} \end{equation}&lt;/span&gt;&lt;br /&gt;&#xA;where we have the sum over all expression in which &lt;span class=&quot;math-container&quot;&gt;$w_a$&lt;/span&gt; occurs, which are &lt;span class=&quot;math-container&quot;&gt;$N-m$&lt;/span&gt;. Note also that we know the last term arises from the fact that &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial x_i^l}{\partial w_a}= y_{i+a}^{l-1}$&lt;/span&gt; which you can see from the first equation.&lt;br /&gt;&#xA;To compute the gradient we need to know the first term, which can be calculated by:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \frac{\partial E}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial y_i^l}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial}{\partial x_i^l} f(x_i^{l})$$&lt;/span&gt;&#xA;where again the first term is the error in the previous layer and &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; the nonlinear activation function.&lt;/p&gt;&#xA;&lt;p&gt;Having all necessary entities we are now able to calculate the error and propagate it back efficiently to the precious layer:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \delta^{l-1}_a = \frac{\partial E}{\partial y_i^{l-1} } = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x_{i-a}^l} \frac{\partial x_{i-a}^l}{\partial y_i^{l-1}} = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x^l_{i-a}} w_a^{flipped}$$&lt;/span&gt;&#xA;Note that the last step can be understood easy when writing down the &lt;span class=&quot;math-container&quot;&gt;$x_i^l$&lt;/span&gt;-s w.r.t. the &lt;span class=&quot;math-container&quot;&gt;$y_i^{l-1}$&lt;/span&gt;-s. The &lt;span class=&quot;math-container&quot;&gt;$flipped$&lt;/span&gt; refers to a transposed weight maxtrix (&lt;span class=&quot;math-container&quot;&gt;$T$&lt;/span&gt;).&lt;/p&gt;&#xA;&lt;p&gt;Therefore you can just calculate the error in the next layer by (now in vector notation):&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\delta^{l} = (w^{l})^{T} \delta^{l+1} f'(x^{l})$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;which becomes for a convolutional and subsampling layer:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\delta^{l} = upsample((w^{l})^{T} \delta^{l+1}) f'(x^{l})$$&lt;/span&gt;&#xA;where the &lt;span class=&quot;math-container&quot;&gt;$upsample$&lt;/span&gt; operation propagates the error through the max pooling layer.&lt;/p&gt;&#xA;&lt;p&gt;Please feel free to add or correct me!&lt;/p&gt;&#xA;&lt;p&gt;For references see:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Convolutional Neural Network by Stanford&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Convolutional Neural Network by Andrew Gibiansky&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;and for a C++ implementation &lt;a href=&quot;https://github.com/nyanp/tiny-cnn#supported-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (without requirement to install)&lt;/p&gt;&#xA;" OwnerUserId="10451" LastEditorUserId="98307" LastEditDate="2020-08-16T21:51:06.280" LastActivityDate="2020-08-16T21:51:06.280" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6538" PostTypeId="1" CreationDate="2015-07-22T00:28:50.967" Score="2" ViewCount="153" Body="&lt;p&gt;This question might be a huge topic, but let's talk about the simple case, say AR (auto-regressive) model. Usually people have various methods of estimating the AR model, like Burgs algorithm, Yule walker algorithm, or even OLS (ordinary least squares). Could some one tell me what is the motivation of using so many different skills to estimate the same model. Is there any literature of book discussing these topic? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot, &lt;/p&gt;&#xA;" OwnerUserId="10774" LastEditorUserId="1359" LastEditDate="2015-07-22T21:19:44.690" LastActivityDate="2015-07-22T21:19:44.690" Title="Different estimation methods of Time series model" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6543" PostTypeId="2" ParentId="661" CreationDate="2015-07-22T06:22:54.263" Score="1" Body="&lt;p&gt;The simplest way to setup a user directory is to login to Hue as an admin and create a user account. This will create a directory in HDFS located at /user/yourusername with the correct permissions to work in.&lt;/p&gt;&#xA;" OwnerUserId="406" LastActivityDate="2015-07-22T06:22:54.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6544" PostTypeId="1" AcceptedAnswerId="6545" CreationDate="2015-07-22T09:21:21.573" Score="2" ViewCount="2923" Body="&lt;p&gt;Normalized Euclidean Distance and Normalized Cross - Correlation can both be used as a metric of distance between vectors.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between these metrics?  It seems to me that they are the same, although I have not seen this explicitly stated in any textbook or literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you.  &lt;/p&gt;&#xA;" OwnerUserId="10512" LastActivityDate="2015-07-22T17:23:35.017" Title="Normalized Euclidean Distance versus cross correlation?" Tags="|time-series|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6545" PostTypeId="2" ParentId="6544" CreationDate="2015-07-22T11:27:06.997" Score="2" Body="&lt;p&gt;These two metrics are &lt;strong&gt;not&lt;/strong&gt; the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normalized Euclidean distance is the distance between two normalized vectors that have been normalized to length one.  If the vectors are identical then the distance is &lt;code&gt;0&lt;/code&gt;, if the vectors point in opposite directions the distance is &lt;code&gt;2&lt;/code&gt;, and if the vectors are orthogonal (perpendicular) the distance is &lt;code&gt;sqrt(2)&lt;/code&gt;.  It is a positive definite scalar value between &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QSIBn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QSIBn.png&quot; alt=&quot;NED equation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normalized cross-correlation is the dot product between the two normalized vectors.  If the vectors are identical, then the correlation is &lt;code&gt;1&lt;/code&gt;, if the vectors point in opposite directions the correlation is &lt;code&gt;-1&lt;/code&gt;, and if the vectors are orthogonal (perpendicular) the correlation is &lt;code&gt;0&lt;/code&gt;.  It is a scalar value between &lt;code&gt;-1&lt;/code&gt; and &lt;code&gt;1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XHjSu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XHjSu.png&quot; alt=&quot;NCC equation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This all comes with the understanding that in time-series analysis the cross-correlation is a measure of similarity of two series as a function of the lag of one relative to the other.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-22T17:23:35.017" LastActivityDate="2015-07-22T17:23:35.017" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6546" PostTypeId="1" AcceptedAnswerId="8545" CreationDate="2015-07-22T14:16:22.823" Score="15" ViewCount="106889" Body="&lt;p&gt;I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it into a DataFrame so I can use SparkSQL to select groups of players via their stats that meet certain criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I have the subset of players I am interested in looking at further, I would like to find the mean of a column; eg Batting Average or RBIs. From there I would like to break all the players into percentile groups based on their average performance compared to all players; the top 10%, bottom 10%, 40-50%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to use the DataFrame.describe() function to return a summary of a desired column (mean, stddev, count, min, and max) all as strings though. Is there a better way to get just the mean and stddev as Doubles, and what is the best way of breaking the players into groups of 10-percentiles?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far my thoughts are to find the values that bookend the percentile ranges and writing a function that groups players via comparators, but that feels like it is bordering on reinventing the wheel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following imports currently:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; import org.apache.spark.rdd.RDD &#xA; import org.apache.spark.sql.SQLContext &#xA; import org.apache.spark.{SparkConf, SparkContext} &#xA; import org.joda.time.format.DateTimeFormat  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10832" LastEditorUserId="10620" LastEditDate="2015-07-26T21:49:24.597" LastActivityDate="2023-10-07T14:22:00.120" Title="How to calculate the mean of a dataframe column and find the top 10%" Tags="|apache-spark|scala|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6547" PostTypeId="1" CreationDate="2015-07-22T14:26:58.660" Score="73" ViewCount="83020" Body="&lt;p&gt;&lt;strong&gt;Problem Background:&lt;/strong&gt;&#xA;I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, organized into hundreds/thousands of rows of various parameters. Each parameter is numeric (float) and there is a non-trivial/non-error value for each time point. My task is to monitor said log files for anomaly detection (spikes, falls, unusual patterns with some parameters being out of sync, strange 1st/2nd/etc. derivative behavior, etc.).&lt;/p&gt;&#xA;&lt;p&gt;On a similar assignment, I have tried Splunk with Prelert, but I am exploring open-source options at the moment.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Constraints:&lt;/strong&gt;&#xA;I am limiting myself to Python because I know it well, and would like to delay the switch to R and the associated learning curve. Unless there seems to be overwhelming support for R (or other languages/software), I would like to stick to Python for this task.&lt;/p&gt;&#xA;&lt;p&gt;Also, I am working in a Windows environment for the moment. I would like to continue to sandbox in Windows on small-sized log files but can move to Linux environment if needed.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Resources:&lt;/strong&gt;&#xA;I have checked out the following with dead-ends as results:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Some info &lt;a href=&quot;https://datascience.stackexchange.com/questions/5193/python-or-r-for-implementing-machine-learning-algorithms-for-fraud-detection&quot;&gt;here&lt;/a&gt; is helpful, but unfortunately, I am struggling to find the right package because:&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Twitter's &amp;quot;AnomalyDetection&amp;quot; is in R, and I want to stick to Python. Furthermore, the Python port &lt;a href=&quot;https://github.com/nicolasmiller/pyculiarity&quot; rel=&quot;noreferrer&quot;&gt;pyculiarity&lt;/a&gt; seems to cause issues in implementing in Windows environment for me.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Skyline, my next attempt, seems to have been pretty much discontinued (from &lt;a href=&quot;https://github.com/etsy/skyline/issues/119&quot; rel=&quot;noreferrer&quot;&gt;github issues&lt;/a&gt;). I haven't dived deep into this, given how little support there seems to be online.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;scikit-learn I am still exploring, but this seems to be much more manual. The down-in-the-weeds approach is OK by me, but my background in learning tools is weak, so would like something like a black box for the technical aspects like algorithms, similar to Splunk+Prelert.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;Problem Definition and Questions:&lt;/strong&gt;&#xA;I am looking for open-source software that can help me with automating the process of anomaly detection from time-series log files in Python via packages or libraries.&lt;/p&gt;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;Do such things exist to assist with my immediate task, or are they imaginary in my mind?&lt;/li&gt;&#xA;&lt;li&gt;Can anyone assist with concrete steps to help me to my goal, including background fundamentals or concepts?&lt;/li&gt;&#xA;&lt;li&gt;Is this the best StackExchange community to ask in, or is Stats, Math, or even Security or Stackoverflow the better options?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT [2015-07-23]&lt;/strong&gt;&#xA;Note that the latest update to &lt;a href=&quot;https://github.com/nicolasmiller/pyculiarity&quot; rel=&quot;noreferrer&quot;&gt;pyculiarity&lt;/a&gt; seems to be &lt;a href=&quot;https://github.com/nicolasmiller/pyculiarity/issues/1&quot; rel=&quot;noreferrer&quot;&gt;fixed&lt;/a&gt; for the Windows environment! I have yet to confirm, but should be another useful tool for the community.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT [2016-01-19]&lt;/strong&gt;&#xA;A minor update. I had not time to work on this and research, but I am taking a step back to understand the fundamentals of this problem before continuing to research in specific details. For example, two concrete steps that I am taking are:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Starting with the &lt;a href=&quot;https://en.wikipedia.org/wiki/Anomaly_detection&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia articles for anomaly detection&lt;/a&gt;, understanding fully, and then either moving up or down in concept hierarchy of other linked Wikipedia articles, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;, and then &lt;a href=&quot;https://en.wikipedia.org/wiki/Machine_learning&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Exploring techniques in the great surveys done by Chandola et al 2009 &lt;a href=&quot;http://www-users.cs.umn.edu/%7Ebanerjee/papers/09/anomaly.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;Anomaly Detection: A Survey&lt;/em&gt;&lt;/a&gt; and Hodge et al 2004 &lt;a href=&quot;http://eprints.whiterose.ac.uk/767/1/hodgevj4.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;A Survey of Outlier Detection Methodologies&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Once the concepts are better understood (I hope to play around with toy examples as I go to develop the practical side as well), I hope to understand which open source Python tools are better suited for my problems.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT [2020-02-04]&lt;/strong&gt;&#xA;It has been a few years since I worked on this problem, and am no longer working on this project, so I will not be following or researching this area until further notice. Thank you very much to all for their input. I hope this discussion helps others that need guidance on anomaly detection work.&lt;/p&gt;&#xA;&lt;p&gt;FWIW, if I had to do the same project now with the same resources (few thousand USD in expenses), I would pursue the deep learning/neural network approach. The ability of the method to automatically learn structure and hierarchy via hidden layers would've been very appealing since we had lots of data and (now) could spend the money on cloud compute. I would still use Python though ;).&lt;/p&gt;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="1406" LastEditorUserId="98307" LastEditDate="2020-10-25T10:07:45.663" LastActivityDate="2020-10-25T10:07:45.663" Title="Open source Anomaly Detection in Python" Tags="|machine-learning|python|data-mining|anomaly-detection|library|" AnswerCount="7" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="6548" PostTypeId="2" ParentId="6316" CreationDate="2015-07-22T18:14:36.907" Score="1" Body="&lt;p&gt;I am assuming that you have only 1 attribute (numeric).&#xA;What you can do is &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;modify the .arff file and set 0s to 1 and 1s to 0.&lt;/li&gt;&#xA;&lt;li&gt;Then you can again run j48 classifier and visualise the decision tree.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It would give you inverted result.&#xA;Also, try watching this video.&lt;a href=&quot;https://www.youtube.com/watch?v=l7R9NHqvI0Y&quot; rel=&quot;nofollow&quot;&gt;WEKA on MOOC&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10834" LastActivityDate="2015-07-22T18:14:36.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6549" PostTypeId="2" ParentId="6547" CreationDate="2015-07-22T18:55:05.250" Score="37" Body="&lt;p&gt;Anomaly Detection or Event Detection can be done in different ways:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Basic Way&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Derivative! If the deviation of your signal from its past &amp;amp; future is high you most probably have an event. This can be extracted by finding large zero crossings in derivative of the signal.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Statistical Way&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Mean of anything is its usual, basic behavior. if something deviates from mean it means that it's an event. Please note that mean in time-series is not that trivial and is not a constant but changing according to changes in time-series so you need to see the &lt;em&gt;&quot;moving average&quot;&lt;/em&gt; instead of average. It looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GCaNT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GCaNT.png&quot; alt=&quot;Events are peaks larger than 1 standard deviation from moving average&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Moving Average code can be found &lt;a href=&quot;https://stackoverflow.com/questions/11352047/finding-moving-average-from-data-points-in-python&quot;&gt;here&lt;/a&gt;. In signal processing terminology you are applying a &lt;em&gt;&quot;Low-Pass&quot;&lt;/em&gt; filter by applying the moving average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can follow the code bellow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MOV = movingaverage(TimeSEries,5).tolist()&#xA;STD = np.std(MOV)&#xA;events= []&#xA;ind = []&#xA;for ii in range(len(TimeSEries)):&#xA;    if TimeSEries[ii] &amp;gt; MOV[ii]+STD:&#xA;        events.append(TimeSEries[ii])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Probabilistic Way&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;They are more sophisticated specially for people new to Machine Learning. Kalman Filter is a great idea to &lt;a href=&quot;http://www.hamilton.ie/florian/infocom2008.pdf&quot; rel=&quot;noreferrer&quot;&gt;find the anomalies&lt;/a&gt;. Simpler probabilistic approaches using &lt;em&gt;&quot;Maximum-Likelihood Estimation&quot;&lt;/em&gt; also work well but my suggestion is to stay with moving average idea. It works in practice very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I could help :)&#xA;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-07-22T19:01:35.100" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6550" PostTypeId="1" AcceptedAnswerId="6991" CreationDate="2015-07-22T18:58:53.040" Score="2" ViewCount="312" Body="&lt;p&gt;What would be a good non cryptographic Hash function to use for converting string features to a numerical representation for feeding into machine learning algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain the scenario my feature set has both categorical data (e.g.: &lt;code&gt;Country&lt;/code&gt;) and non categorical data (e.g.: &lt;code&gt;IP Address&lt;/code&gt;, &lt;code&gt;Email address&lt;/code&gt;). I have used MurMur3 Hash function so far, is there some better algorithm?&lt;/p&gt;&#xA;" OwnerUserId="10836" LastActivityDate="2015-09-05T13:24:59.433" Title="What is a good non cryptographic Hash for string feature translation?" Tags="|machine-learning|data-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6552" PostTypeId="2" ParentId="6547" CreationDate="2015-07-22T21:42:19.010" Score="17" Body="&lt;p&gt;h2o has an &lt;a href=&quot;http://learn.h2o.ai/content/hands-on_training/anomaly_detection.html&quot; rel=&quot;noreferrer&quot;&gt;anomaly detection module&lt;/a&gt; and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit your bill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see an working example over &lt;a href=&quot;https://github.com/h2oai/h2o-3/blob/master/h2o-py/tests/testdir_algos/deeplearning/pyunit_anomaly_largeDeepLearning.py&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sys&#xA;sys.path.insert(1,&quot;../../../&quot;)&#xA;import h2o&#xA;&#xA;def anomaly(ip, port):&#xA;    h2o.init(ip, port)&#xA;&#xA;    print &quot;Deep Learning Anomaly Detection MNIST&quot;&#xA;&#xA;    train = h2o.import_frame(h2o.locate(&quot;bigdata/laptop/mnist/train.csv.gz&quot;))&#xA;    test = h2o.import_frame(h2o.locate(&quot;bigdata/laptop/mnist/test.csv.gz&quot;))&#xA;&#xA;    predictors = range(0,784)&#xA;    resp = 784&#xA;&#xA;    # unsupervised -&amp;gt; drop the response column (digit: 0-9)&#xA;    train = train[predictors]&#xA;    test = test[predictors]&#xA;&#xA;    # 1) LEARN WHAT'S NORMAL&#xA;    # train unsupervised Deep Learning autoencoder model on train_hex&#xA;    ae_model = h2o.deeplearning(x=train[predictors], training_frame=train, activation=&quot;Tanh&quot;, autoencoder=True,&#xA;                                hidden=[50], l1=1e-5, ignore_const_cols=False, epochs=1)&#xA;&#xA;    # 2) DETECT OUTLIERS&#xA;    # anomaly app computes the per-row reconstruction error for the test data set&#xA;    # (passing it through the autoencoder model and computing mean square error (MSE) for each row)&#xA;    test_rec_error = ae_model.anomaly(test)&#xA;&#xA;    # 3) VISUALIZE OUTLIERS&#xA;    # Let's look at the test set points with low/median/high reconstruction errors.&#xA;    # We will now visualize the original test set points and their reconstructions obtained&#xA;    # by propagating them through the narrow neural net.&#xA;&#xA;    # Convert the test data into its autoencoded representation (pass through narrow neural net)&#xA;    test_recon = ae_model.predict(test)&#xA;&#xA;    # In python, the visualization could be done with tools like numpy/matplotlib or numpy/PIL&#xA;&#xA;if __name__ == '__main__':&#xA;    h2o.run_test(sys.argv, anomaly)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10327" LastEditorUserId="10327" LastEditDate="2015-07-23T16:09:10.093" LastActivityDate="2015-07-23T16:09:10.093" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6553" PostTypeId="1" CreationDate="2015-07-22T22:34:43.403" Score="10" ViewCount="5057" Body="&lt;p&gt;I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as &quot;democracy&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A brief plan was&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take the books mentioning the term &quot;democracy&quot; as plain text&lt;/li&gt;&#xA;&lt;li&gt;In each book, replace &lt;code&gt;democracy&lt;/code&gt; with &lt;code&gt;democracy_%AuthorName%&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Train a &lt;code&gt;word2vec&lt;/code&gt; model on these books&lt;/li&gt;&#xA;&lt;li&gt;Calculate the distance between &lt;code&gt;democracy_AuthorA&lt;/code&gt;, &lt;code&gt;democracy_AuthorB&lt;/code&gt;, and other relabeled mentions of &quot;democracy&quot;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So each author's &quot;democracy&quot; gets its own vector, which is used for comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it seems that &lt;code&gt;word2vec&lt;/code&gt; requires much more than several books (each relabeled word occurs only in a subset of books) to train reliable vectors. The &lt;a href=&quot;https://code.google.com/p/word2vec/&quot; rel=&quot;noreferrer&quot;&gt;official page&lt;/a&gt; recommends datasets including billions of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just wanted to ask how large should be the subset of one author's books to make such inference with &lt;code&gt;word2vec&lt;/code&gt; or alternative tools, if available?&lt;/p&gt;&#xA;" OwnerUserId="5279" LastEditorUserId="85045" LastEditDate="2021-01-01T12:54:54.323" LastActivityDate="2021-01-01T12:54:57.057" Title="How much training data does Word2Vec need?" Tags="|text-mining|word-embeddings|word2vec|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="6554" PostTypeId="2" ParentId="5371" CreationDate="2015-07-22T23:27:40.203" Score="1" Body="&lt;p&gt;R's performance depends incredibly on how you write it.  For example, you mostly should never use for loops in R - they're horribly slow because they execute a function call with every iteration.  (One should vectorize and use the apply family of functions instead.  Weird, I know..)  Vectorization is king in R if you want fast code.  Assuming you vectorize both your R and Python code (and other factors), you should probably get the same order of magnitude in speed.  For data larger than memory (you can specify the limit), R starts to become a bad choice.  I don't know much about python's internals, so I can't speak on that.&lt;/p&gt;&#xA;" OwnerUserId="10839" LastActivityDate="2015-07-22T23:27:40.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6555" PostTypeId="1" CreationDate="2015-07-23T03:45:36.867" Score="12" ViewCount="5873" Body="&lt;p&gt;I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in applied machine learning and work quit a bit in data science. I use the tools for computing, rarely would I need to set them up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created 3 VMs (1 master, 2 slaves) and installed Spark successfully. Everything appears to be working as it should. My problem lies in creating a Jupyter server that can be connected to from a browser not running on a machine on the cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've installed &lt;a href=&quot;https://github.com/jupyter/notebook&quot;&gt;Jupyter notebook&lt;/a&gt; successfully... and it runs. I've added a new IPython profile connecting to a &lt;a href=&quot;http://ipython.org/ipython-doc/1/interactive/public_server.html&quot;&gt;remote server&lt;/a&gt; with Spark. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;now the problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The command&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$ ipython --profile=pyspark&#xA;&lt;/code&gt; &#xA;runs fine and it connects to the spark cluster. However, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$ ipython notebook --profile=pyspark&#xA;[&amp;lt;stuff is here&amp;gt;] Unrecognized alias: &quot;profile=pyspark&quot;, it will probably have no effect.&#xA;&lt;/code&gt; &#xA;defaults to the &lt;code&gt;default&lt;/code&gt; profile not the &lt;code&gt;pyspark&lt;/code&gt; profile. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My notebook config for &lt;code&gt;pyspark&lt;/code&gt; has: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;c = get_config()&#xA;c.NotebookApp.ip = '*'&#xA;c.NotebookApp.open_browser = False&#xA;c.NotebookApp.port = 8880&#xA;c.NotebookApp.server_extensions.append('ipyparallel.nbextension')&#xA;c.NotebookApp.password = u'some password is here'&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10846" LastEditorUserId="21" LastEditDate="2016-05-10T13:58:18.167" LastActivityDate="2020-08-03T09:31:05.810" Title="Issue with IPython/Jupyter on Spark (Unrecognized alias)" Tags="|python|apache-spark|pyspark|ipython|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6556" PostTypeId="2" ParentId="977" CreationDate="2015-07-23T05:25:56.953" Score="1" Body="&lt;p&gt;&lt;strong&gt;Stability-Plasticity Dilemma, Learning Rates, and Forgetting Algorithms:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, let me say that this is a really great question and is the type of thought provoking stuff that really improves one's understanding of ML algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this &quot;problem&quot; have a name that it can be referred to?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is generally referred to as &quot;stability&quot;.  What's funny is that stability is actually a useful concept in regular clustering i.e. not online. The &quot;stability&quot; of the algorithm is often chosen as a selection criterion for whether the right number of clusters have been selected.  More specifically, the online clustering stability issue that you have described is referred to as the &lt;code&gt;stability-plasticity dilemma&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Are there &quot;standard&quot; solutions to this and ...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;First, the big picture answer is that many online clustering algorithm are surprisingly stable when they have been well trained with a large cohort of initial data.  However, its still a problem if you want to really nail down the cluster identities of points while allowing the algorithm to react to new data.  The trickiness of you point is briefly addressed in &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/026201243X&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Introduction to Machine Learning&lt;/em&gt;&lt;/a&gt; By Ethem Alpaydin.  On &lt;a href=&quot;https://books.google.com/books?id=NP5bBAAAQBAJ&amp;amp;pg=PA319&amp;amp;lpg=PA319&amp;amp;dq=online+mini+batch+k-means+stability&amp;amp;source=bl&amp;amp;ots=HCI8IK47SX&amp;amp;sig=nJIyST9HJOnsXZr6GGNy1yBQ6uc&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0CEUQ6AEwBWoVChMIjuqesLvwxgIVCVyICh0_yALu#v=onepage&amp;amp;q=online%20mini%20batch%20k-means%20stability&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;page 319&lt;/a&gt; he derives the online k-means algorithm through the application of stochastic gradient descent, but mentions that the &lt;code&gt;stability-plasticity dilemma&lt;/code&gt; arises when choosing a value for the learning rate.  A small learning rate results in stability, but the system looses adaptability where as a larger learning rate gains adaptability, but looses cluster stability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe the best path forward is to choose an implementation of online clustering which allows you to control the stochastic gradient descent algorithm and then choose the learning rate so that you maximize stability and adaptability as best as you can using a sound cross-validation procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method that I've seen employed is some sort of forgetting algorithm e.g. forgetting older points as the data stream matures.  This allows for a fairly stable system on fast time scales and allows for evolution on slower time scales.  &lt;code&gt;Adaptive Resonance Theory&lt;/code&gt; was created to try to solve the &lt;code&gt;stability-plasticity dilemma&lt;/code&gt;.  You might find &lt;a href=&quot;http://techlab.bu.edu/files/resources/articles_tt/Soft-clustering%20and%20improved%20stability%20for%20adaptive%20resonance%20theory%20neural%20networks.pdf&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt; interesting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not well-versed enough in R to suggest an algorithm, but I suggest you look for a &lt;code&gt;mini-batch k-means&lt;/code&gt; algorithm that allows you to control the learning rate in its stochastic gradient descent algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-23T05:34:40.883" LastActivityDate="2015-07-23T05:34:40.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6557" PostTypeId="2" ParentId="6510" CreationDate="2015-07-23T09:36:01.793" Score="8" Body="&lt;p&gt;By default random forest picks up 2/3rd data for training and rest for testing for regression and almost 70% data for training and rest for testing during classification.By principle since it randomizes the variable selection during each tree split it's not prone to overfit unlike other models.However if you want to use CV using nfolds in sklearn you can still use the concept of hold out set such as oob_score(out of bag)=True which shows model performance with or without using CV.&#xA;So in a nutshell using oob_score=True with or without nfolds can itself tell whether using CV is good for your data.Generally if your target is following a certain distribution and you don't have much observation data with you then using CV will not give much improvement.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-23T09:36:01.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6558" PostTypeId="1" CreationDate="2015-07-23T17:15:10.287" Score="3" ViewCount="135" Body="&lt;p&gt;I am looking for a method that can realize if there is any semantic relation between two terms using ontologies. For example given two terms &lt;em&gt;&quot;kitchen&quot;&lt;/em&gt; and &lt;em&gt;&quot;Chef&quot;&lt;/em&gt; it can return s.th like &lt;em&gt;chef works in a kitchen&lt;/em&gt; and a correlation value (for example 0.7). What about proper nouns or noun phrases? The relatedness of two phrases such as &lt;code&gt;&quot;the biggest defense against the US Navy's Laser Weapon System&quot;&lt;/code&gt; and &lt;code&gt;&quot;China's crippling pollution problem&quot;&lt;/code&gt;.&#xA;I would be appreciated if you could give me some references.&lt;/p&gt;&#xA;" OwnerUserId="9218" LastEditorUserId="9218" LastEditDate="2015-08-15T09:22:44.227" LastActivityDate="2015-08-15T09:22:44.227" Title="semantic relation or semantic relatedness between terms or phrases" Tags="|machine-learning|nlp|similarity|correlation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6559" PostTypeId="2" ParentId="5257" CreationDate="2015-07-23T17:29:03.497" Score="1" Body="&lt;p&gt;My recommendation would be to explore some statistical approach to represent the reviews/rating pair as significance. For example, to translate the [# reviews, rating] into some test statistic type model, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;Student's t&lt;/a&gt; for starters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering your example numbers, some approaches can be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; 5/(1/sqrt(10))&#xA;15.811388300841896&#xA;&amp;gt;&amp;gt;&amp;gt; 3/(1/sqrt(10))&#xA;9.4868329805051381&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; 4/(1/sqrt(60))&#xA;30.98386676965934&#xA;&amp;gt;&amp;gt;&amp;gt; 5/(1/sqrt(10))&#xA;15.811388300841896&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) Or, diving deeper into stats,:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; (5-3)/(sqrt(1/10+1/10))&#xA;4.4721359549995796&#xA;&amp;gt;&amp;gt;&amp;gt; (4-5)/(sqrt(1/60+1/10))&#xA;-2.9277002188455996&#xA;&#xA;(you will need to do some work on the alpha level to get at the significance in ex 2 above)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see how these work out for you; if too basic/too many inaccurate assumptions/some other filtering that you need more resolution in your model/etc., you can explore ways to better represent:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the sample standard deviation (assumed to be 1 in my examples above), or&lt;/li&gt;&#xA;&lt;li&gt;the distribution (assumed Gaussian in examples above), or &lt;/li&gt;&#xA;&lt;li&gt;the appropriate statistical test (plow through Wikipedia), or &lt;/li&gt;&#xA;&lt;li&gt;etc&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Point is you can continuously refine your model, depending how much resolution you need. Only you can make that call. Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="1406" LastActivityDate="2015-07-23T17:29:03.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6561" PostTypeId="1" CreationDate="2015-07-23T20:52:46.680" Score="4" ViewCount="1132" Body="&lt;p&gt;I was looking for a cheat sheet of UNIX commands, which are specifically usable for data science. I mean an introduction to very basic commands (starting from &lt;code&gt;cd&lt;/code&gt;, &lt;code&gt;ls&lt;/code&gt;, &lt;code&gt;pwd&lt;/code&gt;, to some still simple but usable for data - e.g. &lt;code&gt;wc&lt;/code&gt;, a few simple things with pipes, &lt;code&gt;ssh&lt;/code&gt;, maybe &lt;code&gt;s3cmd&lt;/code&gt;, etc), with some minimalistic examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find one; or at least, nothing close to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://byte.kde.org/~zrusin/git/git-cheat-sheet-medium.png&quot; rel=&quot;nofollow&quot;&gt;Git Cheat Sheet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/&quot; rel=&quot;nofollow&quot;&gt;Regular Expressions Cheat Sheet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.rstudio.com/resources/cheatsheets/&quot; rel=&quot;nofollow&quot;&gt;series of R Cheat Sheets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am pretty much aware of &lt;a href=&quot;http://datascienceatthecommandline.com/&quot; rel=&quot;nofollow&quot;&gt;datascienceatthecommandline.com&lt;/a&gt;, but my goal would be to have a 1-2 A4 pages printout, with really simple stuff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is one (but I missed it), I would be grateful for posting it as an answer. If there isn't, then I am asking to post pieces of such list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If Moderator permits, we may create one list with Community Wiki.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: The closest thing I've found is here: &lt;a href=&quot;http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/&quot; rel=&quot;nofollow&quot;&gt;Useful Unix commands for data science&lt;/a&gt; (very nice, but missing the very introductory part).&lt;/p&gt;&#xA;" OwnerUserId="289" LastActivityDate="2015-07-24T02:21:36.687" Title="Cheat Sheet of UNIX commands for Data Science" Tags="|tools|beginner|processing|" AnswerCount="2" CommentCount="0" ClosedDate="2020-04-07T02:32:26.523" ContentLicense="CC BY-SA 3.0" />
  <row Id="6562" PostTypeId="2" ParentId="6561" CreationDate="2015-07-23T21:26:54.077" Score="3" Body="&lt;p&gt;I would recommend this one - &lt;a href=&quot;http://www.cheat-sheets.org/saved-copy/ubunturef.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cheat-sheets.org/saved-copy/ubunturef.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its comprehensive, but not data science specific. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of great cheat sheets here that may be what you are looking for:&lt;a href=&quot;https://dzone.com/refcardz&quot; rel=&quot;nofollow&quot;&gt;https://dzone.com/refcardz&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are wanting to create your own, this is a very nice tool: &lt;a href=&quot;http://www.cheatography.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.cheatography.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8986" LastActivityDate="2015-07-23T21:26:54.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6563" PostTypeId="2" ParentId="6510" CreationDate="2015-07-23T21:28:10.853" Score="4" Body="&lt;p&gt;I did some tests on a data set of 50k rows, using &lt;code&gt;sklearn.RandomForestRegressor&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get significantly different scores - I'm using a normalized gini for the metric - depending on whether I use &lt;code&gt;rf.oob_prediction_&lt;/code&gt; (0.2927) or a KFold CV (0.3258 for 7 folds and 0.3236 for 3 folds).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that, it appears your point about &quot;especially if comparing RFs with other algorithms on the same dataset&quot; is a strong consideration towards using manual CV rather than relying on the OOB prediction.&lt;/p&gt;&#xA;" OwnerUserId="10816" LastActivityDate="2015-07-23T21:28:10.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6565" PostTypeId="2" ParentId="6561" CreationDate="2015-07-24T02:21:36.687" Score="3" Body="&lt;p&gt;This is a post that starts from data science and walks you through options for completing common tasks:  &lt;a href=&quot;http://www.drbunsen.org/explorations-in-unix/&quot; rel=&quot;nofollow&quot;&gt;http://www.drbunsen.org/explorations-in-unix/&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="10861" LastActivityDate="2015-07-24T02:21:36.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6566" PostTypeId="2" ParentId="6547" CreationDate="2015-07-24T05:02:23.250" Score="6" Body="&lt;p&gt;I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html&quot; rel=&quot;noreferrer&quot;&gt;one-class SVM&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;noreferrer&quot;&gt;Novelty detection&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in case that your feature is an entire logfile, you need to first summarize it to some feature of same dimension, and then apply Novealty detection. &lt;/p&gt;&#xA;" OwnerUserId="10865" LastActivityDate="2015-07-24T05:02:23.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6567" PostTypeId="2" ParentId="6533" CreationDate="2015-07-24T05:53:55.760" Score="-1" Body="&lt;p&gt;To be frank, I really donot understand why this is an issue for you. If you want to a generic solution, I will say save it to a csv file. A file with 30,000 rows and 1,000 cols is NOT that big. Even a shell pipe with simple one-liners can do tasks you mentioned. For example, the following one-liner compute the mean of cells located at Ax rows and x3 cols of data.csv (I generate this data matrix of size 30,000x1,000 by filling random integers )&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat data.csv | perl -F',' -lane 'BEGIN{ $avg=0; $num=0; $row=0; } $row++; next if( $row%3!=0);foreach $i (0..$#F){ if($i%26 == 0){ $avg+=$F[$i]; $num++;} }END{ print &quot;avg=&quot; . ($avg/$num); print &quot;num=$num&quot;; }' &#xA;avg=0.1998&#xA;num=390000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It takes about 3 seconds on my laptop. You may put whatever interested conditions to filter data, collect filtered ones and compute stats. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since for whatever complicate condition you AT MOST scan each element once, the complexity is thus at most linear. Even for a big data file, this will not be a big issue. By the way, adding extra auxiliary is simply to 'pre-compute' something in advance. &lt;/p&gt;&#xA;" OwnerUserId="10865" LastActivityDate="2015-07-24T05:53:55.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6569" PostTypeId="2" ParentId="2293" CreationDate="2015-07-24T13:15:31.040" Score="0" Body="&lt;p&gt;You can refer to my response related to h2o R or Python anomaly detection method in &lt;a href=&quot;https://datascience.stackexchange.com/questions/6547/open-source-anomaly-detection-in-python&quot;&gt;stackexchange&lt;/a&gt;,since that is scalable too.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-07-24T13:15:31.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6570" PostTypeId="1" AcceptedAnswerId="6572" CreationDate="2015-07-24T13:21:32.583" Score="2" ViewCount="437" Body="&lt;p&gt;I have a Healthcare dataset. I have been told to look at non-parametric approach to solve certain questions related to the dataset. I am little bit confused about non-parametric approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do they mean density plot based approach (such as looking at the histogram)? &#xA;I know this is a vague question to ask here. However, I don't have access to anybody else whom I can ask and hence I am asking for some input from others in this forum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any response/thought would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and regards. &lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2015-07-24T14:56:44.693" Title="Non-parametric approach to healthcare dataset?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6572" PostTypeId="2" ParentId="6570" CreationDate="2015-07-24T14:56:44.693" Score="2" Body="&lt;p&gt;They are not specifically referring to a plot based approach.  They are referring to a class of methods that must be employed when the data is not normal enough or not well-powered enough to use regular statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parametric and nonparametric are two broad classifications of statistical procedures with loose definitions separating them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Parametric tests usually assume that the data are approximately normally distributed.&lt;/li&gt;&#xA;&lt;li&gt;Nonparametric tests do not rely on a normally distributed data assumption.&lt;/li&gt;&#xA;&lt;li&gt;Using parametric statistics on non-normal data could lead to incorrect results.&lt;/li&gt;&#xA;&lt;li&gt;If you are not sure that your data is normal enough or that your sample size is big enough (n &amp;lt; 30), use nonparametric procedures rather than parametric procedures.&lt;/li&gt;&#xA;&lt;li&gt;Nonparametric procedures generally have less power for the same sample&#xA;size than the corresponding parametric procedure if the data truly are normal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Take a look at some examples of parametric and analogous nonparametric tests from &lt;a href=&quot;http://www.mayo.edu/mayo-edu-docs/center-for-translational-science-activities-documents/berd-5-6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tanya Hoskin's Demystifying Summary&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q5JE7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q5JE7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some summary references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://changingminds.org/explanations/research/analysis/parametric_non-parametric.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Another general table with some different information&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statoek.wiso.uni-goettingen.de/veranstaltungen/graduateseminar/Nonpar2006.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nonparametric Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ksu.edu.sa/sites/py/ar/mpy/departments/math/learnResources/ResourceCenter/Documents/All%20of%20Nonparametric%20Statistics.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;All of Nonparametric Statistics, by Larry Wasserman&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r-tutor.com/elementary-statistics/non-parametric-methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;R tutorial&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://statsmodels-np.blogspot.ca/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nonparametric Econometrics with Python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-07-24T14:56:44.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6573" PostTypeId="2" ParentId="919" CreationDate="2015-07-24T15:48:54.190" Score="0" Body="&lt;p&gt;To be honest binary classification is the easiest type compared to multi-class classification as at times by error you can classify a wrong class to a right one.So if you have a dataset with multiclass you will need a good distribution among them,so the expectation is more sample will give better insight,i.e CV should be less.However in case of binary classification if your class distribution is balanced enough you can easily go fo CV=10 for 25k observations,however if the class distribution is skewed you better go with less CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in a nutshell in case of binary distribution CV value really depends on your class distribution and not much on number of observations.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastActivityDate="2015-07-24T15:48:54.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6574" PostTypeId="2" ParentId="6519" CreationDate="2015-07-24T15:55:45.900" Score="2" Body="&lt;p&gt;You can also use Python to do this. If you're familiar using &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Pandas&lt;/a&gt; data frames:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;d1 = pd.DataFrame({'A':[1,4], 'B':[7,8], 'C':[3,4]})&#xA;d2 = pd.DataFrame({'A':[3,4], 'C':[4,8], 'D':[5,4], 'E':[6,6]})&#xA;&#xA;d1.append(d2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will output what you're looking for, except the empty cells would have NaNs (not a number) in them.&lt;/p&gt;&#xA;" OwnerUserId="10844" LastActivityDate="2015-07-24T15:55:45.900" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6576" PostTypeId="1" CreationDate="2015-07-24T20:10:07.617" Score="13" ViewCount="1495" Body="&lt;p&gt;First of all this term sounds so obscure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways..I am a software programmer. One of the languages I can code is Python. Speaking of Data I can use SQL and can do Data Scraping. What I figured out so far after reading soo many articles that Data Science is all about good at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- Stats&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Algebra&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- Data Analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4- Visualisation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5- Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I know so far:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- Python Programming&#xA;2- Data scrapping in Python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you experts guide me or suggest a roadmap to brush up both theory and practical? I have given around 8 months of time frame to myself.&lt;/p&gt;&#xA;" OwnerUserId="10879" LastEditorUserId="3151" LastEditDate="2015-12-23T11:03:07.713" LastActivityDate="2022-08-02T11:05:59.237" Title="I am a programmer, how do I get into field of Data Science?" Tags="|beginner|career|" AnswerCount="8" CommentCount="5" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="6577" PostTypeId="1" AcceptedAnswerId="6580" CreationDate="2015-07-24T20:39:06.317" Score="1" ViewCount="104" Body="&lt;p&gt;I have a rate-step plan for consumption with which I'm trying to total up costs. I have a rate plant that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;First 20 Kwh is Free; &#xA;Second 5 Kwh is $.1&#xA;    Next 100 Kwh is $.053 &#xA;Next 875 Kwh is $.042 &#xA;    Over 1000Kwh is $.039&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone provide a worked example for me, e.g., with $\text{consumtpion}=625$? If possible, with R code for calculating the total cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my own approach, I was thinking of using a loop, but if it's a lot of accounts the loop will take too long.  Can anyone think of a vector-based approach for calculating the result?&lt;/p&gt;&#xA;" OwnerUserId="10881" LastEditorUserId="13413" LastEditDate="2015-10-23T07:43:20.510" LastActivityDate="2015-11-22T19:10:23.770" Title="how to perform Step calcuations in R" Tags="|r|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6578" PostTypeId="1" CreationDate="2015-07-24T21:06:16.493" Score="1" ViewCount="130" Body="&lt;p&gt;Can anyone suggest a machine learning algorithm that would be useful for identifying the organization a document belongs to? I'm working with a relatively large number of organizations (on the order of 1000s). Naturally, the input data is unstructured, but most documents look almost, but not quite, the same. Also, there are some organizations that follow more then a one template, again, in that &quot;almost but not quite&quot; manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Documents are in PDF format, and are reports of some kind and can contain several rectangular regions of text and tables. Most of them are not more than five pages long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New organizations can eventually appear - how does interfere with algorithm?&lt;/p&gt;&#xA;" OwnerUserId="10882" LastEditorUserId="13413" LastEditDate="2015-10-21T04:23:42.697" LastActivityDate="2015-10-25T10:37:43.900" Title="Identify document's owner by content" Tags="|classification|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6580" PostTypeId="2" ParentId="6577" CreationDate="2015-07-24T23:00:01.503" Score="3" Body="&lt;p&gt;Try a simple equation like this:&#xA;\begin{eqnarray}&#xA;&amp;amp;&amp;amp;\max(x-20,0)*0.1 + \max(0,x-25)*(-.047) + \\&#xA;&amp;amp;&amp;amp;\max(0, x-125)*(-.011) + \max(0,x-1000)*(-.003)&#xA;\end{eqnarray}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should take the usage $(x)$ and walk it through steps, initially charging 0.1 for each Kwh over 20: $\max(x-20,0)*0.1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, if $x$ is above 20, it takes away 0.047 (0.1-0.053) for every Kwh between 20 and 25: $\max(0, x-25)*(-0.047)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, if $x$ is above 125, it takes away 0.011 (0.053-0.042) for every Kwh between 25 and 125: $\max(0, x-125)*(-0.011)$&lt;/p&gt;&#xA;" OwnerUserId="10884" LastEditorUserId="10884" LastEditDate="2015-10-23T17:04:01.667" LastActivityDate="2015-10-23T17:04:01.667" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6581" PostTypeId="2" ParentId="253" CreationDate="2015-07-25T04:27:41.210" Score="1" Body="&lt;p&gt;I do think Leaning Hadoop framework (hard way) is not a requirement of being a Data Scientist.&#xA;General knowledge on all big data platforms is essential.&#xA;I will suggest to know concept on it and only part need from Hadoop is the MapReduce&#xA;&lt;a href=&quot;http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;A Data Scientist does not build cluster, administer ... is just make &amp;quot;magic&amp;quot; with data and does not care where is coming from.&#xA;The term &amp;quot;Hadoop&amp;quot; has come to refer not just to the base modules above, but also to the &amp;quot;ecosystem&amp;quot;, or collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Spark, and others.&lt;/p&gt;&#xA;&lt;p&gt;Most important is the Programing language, math and statistics for working with data (you'll need to find a way to connect with data and move forward).&#xA;I wish I had somebody to point me to the concept and do not spend weeks on learning framework and build from scratch nodes and clusters, because that part is Administrator role and not Data Engineer or Data Scientist.&#xA;Also one thing: all are changing and evolving but math, programing, statistics are still the requirements.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;accessing data from hdfs is essential, for example PROC Hadoop, Hive,  SparkContext or any other driver or pipe (treat hadoop as a point of accesing data or storage :)&lt;/p&gt;&#xA;&lt;p&gt;already are in place tools or frameworks what take care of resource allocation and management, performance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-11-13T21:56:52.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6582" PostTypeId="2" ParentId="6576" CreationDate="2015-07-25T04:49:31.850" Score="7" Body="&lt;p&gt;I do like Berkeley course on Data Science, will give a good foundation and taste for Data Science, After moved to udacity and coursera and many more resources. So if you have Programming skills than will need math and stat and a lot of visualization. Also will be great to get used to IPython because is essential to see every step(visualize)how it perform instead writing a whole script and test after (anaconda is easy to install and work with). Course is listed bellow: &#xA;bcourses.berkeley.edu/courses/1267848/wiki&#xA;also the stat i find good free course from SAS: Statistics 1: Introduction to ANOVA, Regression, and Logistic Regression support.sas.com/edu/schedules.html?ctry=us&amp;amp;id=1979&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting with ML will recommend: www.kaggle.com/c/titanic/details/getting-started-with-python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on left side is also for Excel using Pivot tables  and R. DataCamp has released the tutorial on how to use R. Once you complete this steps than more competitions in gaining experience are on kaggle (recently released one for San Francisco Crime Classification) and ultimately amazing video tutorials from www.dataschool.io&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope it helps ...&lt;/p&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="8752" LastEditDate="2015-07-26T19:39:37.227" LastActivityDate="2015-07-26T19:39:37.227" CommentCount="6" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="6583" PostTypeId="2" ParentId="6578" CreationDate="2015-07-25T07:36:25.153" Score="0" Body="&lt;p&gt;Try Decision Trees,Support Vector Machines or Naive-Bayes method with TF-IDF weighting for creating document vectors and check the Precision/Recall/F Measure scores. But this will not deal with unknown/new organizations, they will get classified to any other one. One way would be remodel/train again when there are so many unknown/new organization and the Precision/Recall/F Measure score too low.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-07-25T07:43:37.283" LastActivityDate="2015-07-25T07:43:37.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6584" PostTypeId="2" ParentId="6209" CreationDate="2015-07-25T13:27:49.453" Score="-1" Body="&lt;p&gt;simple... use advanced Text Editor&#xA;like...&#xA;Sublime text ,Notepad  ,Vim or Atom..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;paste all Bunch of data on any Editior and just remove '\n and replaced it by &quot;&quot; ..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;use FIND AND REPLACE function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notepad   ,sublime text,vim and atom&lt;/p&gt;&#xA;&#xA;&lt;p&gt;use Ctrl   H&lt;/p&gt;&#xA;&#xA;&lt;p&gt;find box type &quot;\n&quot; and in replace box &quot;&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;done :) ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tnx&lt;/p&gt;&#xA;" OwnerUserId="10891" LastActivityDate="2015-07-25T13:27:49.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6585" PostTypeId="1" CreationDate="2015-07-25T18:50:36.500" Score="1" ViewCount="41" Body="&lt;p&gt;I am working on the development of an ontology for my dissertation project. I have read plenty of resources and tutorials on how to develop ontologies, why are they useful and how to use them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to find examples of ontologies that include user interface such as textboxes, buttons etc. to study them for the development of my ontology. I checked databases such as ontoligua for similar projects but I didn't find anything to relate to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be grateful anyone can help me with that. &lt;/p&gt;&#xA;" OwnerUserId="10894" LastActivityDate="2015-07-25T18:50:36.500" Title="Ontologies with user interface" Tags="|knowledge-base|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6586" PostTypeId="2" ParentId="6576" CreationDate="2015-07-25T21:48:43.100" Score="19" Body="&lt;p&gt;Focus less on gaining skills and more on gaining experience. Try to actually solve some problems and post your work on github. You'll learn more in the process and be able to demonstrate knowledge and experience to employers, which is much more valuable than having a supposedly deep understanding of a topic or theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science is a pretty loaded field these days so I'm not sure what kind of work you specifically want to do, but assuming that machine learning is a component of it then kaggle.com is a good place to start. In terms of goals, if you're able to work with the data in pandas/numpy/scipy, build models in sci-kit learn and make some pretty graphs in seaborn, ggplot or even matplotlib then you won't have a problem getting a job from a skills perspective -- especially if you have code samples and examples to demonstrate your abilities. If you get stuck then stackexchange will either have the answer or you can post a questions and you'll have an answer shortly. Once you're doing the work for a living then you'll learn even more, likely from a senior team member who mentors you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-07-25T21:48:43.100" CommentCount="0" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="6587" PostTypeId="1" CreationDate="2015-07-26T10:15:25.393" Score="6" ViewCount="480" Body="&lt;p&gt;The  result of my computational simulation is a (time-dependent) system of large number (~100k) of moving points in a confined space. Each point has its own Cartesian coordinates as well as a weight (w) in the form of $(x_i,y_i,z_i;w_i)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a software/app/package to create a snapshot of the 3D spatial density map of these points. (&lt;a href=&quot;http://www.nature.com/ncomms/2013/130906/ncomms3414/fig_tab/ncomms3414_F3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;something like this&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a2TEG.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a2TEG.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like you see in this figure, the points are not going to be displayed individually, but only a transparent cloud will be drawn whose local intensity is proportional with the local number of points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final goal is to create a movie of the changing 3D spatial density with time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have tried R, Matlab, Origin, and ImageJ. But no success!&lt;/p&gt;&#xA;" OwnerUserId="10899" LastEditorUserId="31513" LastEditDate="2017-06-30T17:26:22.020" LastActivityDate="2024-02-23T19:02:30.287" Title="Visualizing large number of points as a 3D density map" Tags="|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6590" PostTypeId="1" AcceptedAnswerId="12016" CreationDate="2015-07-26T12:17:09.947" Score="9" ViewCount="1042" Body="&lt;p&gt;My question is three-fold&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of &quot;Kernelized&quot; support vector machines&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and the main motive behind introducing kernels to a SVM is to increase the dimensionality of the problem, in such a case reducing the dimensions by parameter reduction seems counter-intuitive&lt;/li&gt;&#xA;&lt;li&gt;If the answer to the 1st question is &quot;NO&quot;, then, On what conditions would the answer change that one should keep in mind ?&lt;/li&gt;&#xA;&lt;li&gt;Are there any good methods that have been tried to bring about feature reduction for SVMs in scikit-learn library of python - I have tried the SelectFpr method and am looking for people with experiences with different methods.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9061" LastEditorUserId="9061" LastEditDate="2015-07-26T14:10:04.970" LastActivityDate="2016-06-16T15:21:02.050" Title="Feature selection for Support Vector Machines" Tags="|svm|feature-selection|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6591" PostTypeId="1" CreationDate="2015-07-26T17:08:34.013" Score="3" ViewCount="93" Body="&lt;p&gt;I'm having some trouble describing in one line what I want, which is probably why I haven't had much luck with Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a game like 2048 where the possible actions each step are fixed (and more than two). I want to train a neural network that chooses a move, so I have 4 neurons in output layer and I make the move with the highest prediction. The output vector is normalized (softmax layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the training data I have is just the state, the move that was made, and whether that had good or bad results. If the chosen move is bad, I don't know which of the other ones was better (if any).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I train this? My current thought is like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Good move? -&gt; chosen action gets positive error (so prediction goes up)&lt;/li&gt;&#xA;&lt;li&gt;Bad move? -&gt; chosen step gets negative error (so prediction goes down)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I haven't found literature supporting this guess. There are alternatives:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Maybe I should also update the options that weren't chosen (in the opposite direction)?&lt;/li&gt;&#xA;&lt;li&gt;Is it a good idea to set error directly instead of using goal predictions?&lt;/li&gt;&#xA;&lt;li&gt;The error for correct and incorrect could be different, maybe to preserve normalization?&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(I'm doing 2048 and using neural networks, but I think this is not limited to this game or this method.)&lt;/p&gt;&#xA;" OwnerUserId="10907" LastEditorUserId="10907" LastEditDate="2015-07-27T19:26:27.800" LastActivityDate="2024-03-25T03:08:40.957" Title="Train a classifier for a game with feedback on chosen move instead of true labels" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="6593" PostTypeId="2" ParentId="6084" CreationDate="2015-07-27T02:12:12.203" Score="17" Body="&lt;p&gt;Wow, this was bit challenging but I was able to make one of these plots in python.&#xA;The two main components are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/24659005/radar-chart-with-multiple-scales-on-multiple-axes&quot;&gt;plotting multiple radial axes&lt;/a&gt; on a polar plot&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/12075001/set-radial-axis-on-matplotlib-polar-plots&quot;&gt;remapping radial axes&lt;/a&gt; for variables with reversed scales&#xA;&lt;a href=&quot;https://i.stack.imgur.com/XQpKQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XQpKQ.png&quot; alt=&quot;complex radar plot&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/kylerbrown/29ce940165b22b8f25f4&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns # improves plot aesthetics&#xA;&#xA;&#xA;def _invert(x, limits):&#xA;    &quot;&quot;&quot;inverts a value x on a scale from&#xA;    limits[0] to limits[1]&quot;&quot;&quot;&#xA;    return limits[1] - (x - limits[0])&#xA;&#xA;def _scale_data(data, ranges):&#xA;    &quot;&quot;&quot;scales data[1:] to ranges[0],&#xA;    inverts if the scale is reversed&quot;&quot;&quot;&#xA;    for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;        assert (y1 &amp;lt;= d &amp;lt;= y2) or (y2 &amp;lt;= d &amp;lt;= y1)&#xA;    x1, x2 = ranges[0]&#xA;    d = data[0]&#xA;    if x1 &amp;gt; x2:&#xA;        d = _invert(d, (x1, x2))&#xA;        x1, x2 = x2, x1&#xA;    sdata = [d]&#xA;    for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;        if y1 &amp;gt; y2:&#xA;            d = _invert(d, (y1, y2))&#xA;            y1, y2 = y2, y1&#xA;        sdata.append((d-y1) / (y2-y1) &#xA;                     * (x2 - x1) + x1)&#xA;    return sdata&#xA;&#xA;class ComplexRadar():&#xA;    def __init__(self, fig, variables, ranges,&#xA;                 n_ordinate_levels=6):&#xA;        angles = np.arange(0, 360, 360./len(variables))&#xA;&#xA;        axes = [fig.add_axes([0.1,0.1,0.9,0.9],polar=True,&#xA;                label = &quot;axes{}&quot;.format(i)) &#xA;                for i in range(len(variables))]&#xA;        l, text = axes[0].set_thetagrids(angles, &#xA;                                         labels=variables)&#xA;        [txt.set_rotation(angle-90) for txt, angle &#xA;             in zip(text, angles)]&#xA;        for ax in axes[1:]:&#xA;            ax.patch.set_visible(False)&#xA;            ax.grid(&quot;off&quot;)&#xA;            ax.xaxis.set_visible(False)&#xA;        for i, ax in enumerate(axes):&#xA;            grid = np.linspace(*ranges[i], &#xA;                               num=n_ordinate_levels)&#xA;            gridlabel = [&quot;{}&quot;.format(round(x,2)) &#xA;                         for x in grid]&#xA;            if ranges[i][0] &amp;gt; ranges[i][1]:&#xA;                grid = grid[::-1] # hack to invert grid&#xA;                          # gridlabels aren't reversed&#xA;            gridlabel[0] = &quot;&quot; # clean up origin&#xA;            ax.set_rgrids(grid, labels=gridlabel,&#xA;                         angle=angles[i])&#xA;            #ax.spines[&quot;polar&quot;].set_visible(False)&#xA;            ax.set_ylim(*ranges[i])&#xA;        # variables for plotting&#xA;        self.angle = np.deg2rad(np.r_[angles, angles[0]])&#xA;        self.ranges = ranges&#xA;        self.ax = axes[0]&#xA;    def plot(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;    def fill(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;&#xA;# example data&#xA;variables = (&quot;Normal Scale&quot;, &quot;Inverted Scale&quot;, &quot;Inverted 2&quot;, &#xA;            &quot;Normal Scale 2&quot;, &quot;Normal 3&quot;, &quot;Normal 4 %&quot;, &quot;Inverted 3 %&quot;)&#xA;data = (1.76, 1.1, 1.2, &#xA;        4.4, 3.4, 86.8, 20)&#xA;ranges = [(0.1, 2.3), (1.5, 0.3), (1.3, 0.5),&#xA;         (1.7, 4.5), (1.5, 3.7), (70, 87), (100, 10)]            &#xA;# plotting&#xA;fig1 = plt.figure(figsize=(6, 6))&#xA;radar = ComplexRadar(fig1, variables, ranges)&#xA;radar.plot(data)&#xA;radar.fill(data, alpha=0.2)&#xA;plt.show()    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7738" LastEditorUserId="21" LastEditDate="2016-02-14T19:28:31.743" LastActivityDate="2016-02-14T19:28:31.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6595" PostTypeId="1" AcceptedAnswerId="6609" CreationDate="2015-07-27T05:55:57.130" Score="0" ViewCount="1177" Body="&lt;p&gt;I have a dataset with 261 predictors scraped from a larger set of survey questions. 224 have values which are in a range of scale (some 1-10, some 1-4, some simply binary, all using 0 where no value is given), and the rest are unordered categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to perform classification using these predictors and identify the top n predictors. Am thinking of the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;convert the 224 ordered predictors into numeric, centered, and scaled.&lt;/li&gt;&#xA;&lt;li&gt;Run separate modeling (I use &lt;a href=&quot;https://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;caret&lt;/a&gt; from R): one for using the numeric predictors, another using the  remaining 37 categorical predictors (both cross-validated within each modeling exercise).&lt;/li&gt;&#xA;&lt;li&gt;Choose the respective best-fitting models modelN and modelC for the numeric and categorical predictors.&lt;/li&gt;&#xA;&lt;li&gt;Choose top n (say 10) predictors from model N and model C.&lt;/li&gt;&#xA;&lt;li&gt;Combine them in an ensemble model that can handle both numeric and categorical data (say, random forest).&lt;/li&gt;&#xA;&lt;li&gt;Choose top n predictors in the ensemble model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am going through this a roundabout way rather than directly fitting all predictors into an ensemble model to try and reduce the complexity of the problem first (and because in R, I'm having a problem with too many levels from the predictors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this be a valid approach to identifying the n most salient predictors? &#xA;Any possible issues to mitigate?&lt;/p&gt;&#xA;" OwnerUserId="1133" LastEditorUserId="26596" LastEditDate="2016-11-30T23:07:45.650" LastActivityDate="2016-11-30T23:07:45.650" Title="Identifying top predictors from a mix of categorical and ordinal data" Tags="|r|classification|feature-selection|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6597" PostTypeId="2" ParentId="6578" CreationDate="2015-07-27T11:46:01.080" Score="0" Body="&lt;p&gt;My team and I recently experienced a similar problem. We used a Random Forest to be able to predict between 10 authors with about an 82% accuracy, as the number of users went up the accuracy went down. We were then asked to try and identify a new author which we implemented by creating a dynamic threshold for votes from the individual trees in forest. With the new user identification and 10 authors we were at about 65% accurate. We were working with twitter data also, so our document sets were fairly small. I have seen documentation where other teams have simply used the count of stop words in documents and a random forest that was 80% accurate, they weren't trying to identify a new user however. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your situation, breaking up your problem into two different problems might yield better results. First identify if the new document belongs to any of your known users and if it does then try and decide which one.&lt;/p&gt;&#xA;" OwnerUserId="10914" LastActivityDate="2015-07-27T11:46:01.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6598" PostTypeId="2" ParentId="6078" CreationDate="2015-07-27T12:15:37.550" Score="0" Body="&lt;p&gt;Possible, yes, but why do you want to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topic modelling is a method of &lt;em&gt;unsupervised learning&lt;/em&gt; with all it weaknesses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the type of data you sketch you can used methods of &lt;em&gt;supervised learning&lt;/em&gt; that will be much more stable, less prone to overfitting, and more reliable in the long run.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-07-27T12:15:37.550" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6599" PostTypeId="2" ParentId="5257" CreationDate="2015-07-27T12:24:39.650" Score="1" Body="&lt;p&gt;It is an old question, but I want to suggest a different approach than the other answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Using a sentiment analysis on reviews&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that the data we have are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;5-stars rating (zero to five)&lt;/li&gt;&#xA;&lt;li&gt;N number of reviews for each entry&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We use a sentiment analysis model to calculate the &quot;mood&quot; of each review. The result would be &quot;negative&quot;, &quot;neutral&quot; and &quot;positive&quot;, based on the content of the review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read more about sentiment analysis and find some APIs in this &lt;a href=&quot;http://www.quora.com/Is-there-an-API-for-analyzing-the-emotion-in-text&quot; rel=&quot;nofollow&quot;&gt;Quora's question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you will use a multiplier for each mood.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0: negative mood&lt;/li&gt;&#xA;&lt;li&gt;0.5: neutral mood&lt;/li&gt;&#xA;&lt;li&gt;1: positive mood&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a result, you can add the review score like &lt;code&gt;multiplier*5&lt;/code&gt; for each review in your dataset. For example in pseudocode,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var current_5_star;&#xA;for each review in this_entry:&#xA;    multiplier = calc_sentiment(review)&#xA;    current_5_star += multiplier * 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to improve further the model, you can use the total number of reviews on your dataset to normalize the multiplier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up, you do not only use the number of reviews per se, but also the content of them to evaluate the exact impact of it in the rating score.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-07-27T12:24:39.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6600" PostTypeId="2" ParentId="6595" CreationDate="2015-07-27T12:30:12.220" Score="0" Body="&lt;p&gt;Just some random thoughts&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a mathematical model to base on? For example, you want to predict how pressure is changing against temperature. You won't drop any predictor that is related to 'temperature', no matter how remote it is. If so, that should govern your choice of predictors and you should start with that as that would lend more credibility to your final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not and you just want an algorithm to pick the best predictors, have you thought of running a regression model with L1 norm on? This will drive out insignificant factors and you can start with that set as a basis.&lt;/p&gt;&#xA;" OwnerUserId="10915" LastActivityDate="2015-07-27T12:30:12.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6601" PostTypeId="1" CreationDate="2015-07-27T17:27:24.527" Score="5" ViewCount="892" Body="&lt;p&gt;I am looking at time series security attack data where a given IP can either be labeled as (1) attack or (0) no attack. &#xA;In total we will have thousands of IPs and roughly an equal number of attacks and non attacks. The data is rather noisy and every time series sequence can have a different length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for advise on state of the art approaches to time series classification. I am past the stage of simple things like moving averages and I am looking for ways to improve my current methods or new things to try. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have currently implemented a few different techniques:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;K-nearest neighbor with DTW. I am successfully using &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/UCRsuite.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.ucr.edu/~eamonn/UCRsuite.html&lt;/a&gt; which provides state of the art performance. &lt;/li&gt;&#xA;&lt;li&gt;Logical shapelets (&lt;a href=&quot;http://www.cs.ucr.edu/~mueen/LogicalShapelet/&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.ucr.edu/~mueen/LogicalShapelet/&lt;/a&gt;). This seems promising but have not been able to get any existing code base to work.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest different technique to try? I have seen papers about discords and motifs but still need to investigate if they are relevant for my problem. &lt;/p&gt;&#xA;" OwnerUserId="802" LastEditorUserId="14994" LastEditDate="2015-12-30T08:28:25.667" LastActivityDate="2017-02-23T17:21:40.743" Title="Time series classification" Tags="|classification|time-series|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6602" PostTypeId="1" CreationDate="2015-07-27T21:15:03.970" Score="2" ViewCount="3511" Body="&lt;p&gt;I'm planning to upgrade my computer and also want to buy some new DDR3 RAM modules. Making some research I realized that there are 2 main characteristics of RAMs (beside capacity): frequency and latency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use my computer for data analysis as well (like Kaggle competitions). Usually I work in R and sometimes in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that there is a trade-off between latency and frequency (that is also reflected in the price sometimes). My question is which one is more important for data analysis purposes? How can it effect real world performance? How much difference should I expect between choosing a model with overall better and lower parameter values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;: I don't ask which RAM is better in general. I need answer from users who know how memory management happens in R (or Python). How are the different ML algorithms managed in memory and which RAM characteristics are preferred to support it. I would also appreciate personal experiences on this field. &lt;/p&gt;&#xA;" OwnerUserId="5211" LastEditorUserId="5211" LastEditDate="2015-07-28T20:10:05.260" LastActivityDate="2015-07-28T20:45:45.833" Title="What kind of RAM to choose for data analysis?" Tags="|r|python|" AnswerCount="1" CommentCount="3" ClosedDate="2015-07-28T20:45:54.253" ContentLicense="CC BY-SA 3.0" />
  <row Id="6603" PostTypeId="2" ParentId="5233" CreationDate="2015-07-27T21:32:58.897" Score="1" Body="&lt;p&gt;e1071 is a highly popular and therefore safe solution for this purpose. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if data starts to get bigger I would go with &lt;a href=&quot;https://cran.r-project.org/web/packages/classyfire/classyfire.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;classyfire&lt;/a&gt;. It's a quite fresh package built on e1071. It is highly optimized and have some other nice feature too. In my experience it produced similar results to e1071 but with a much faster speed as it was able to utilize all 4 core of my processor.&lt;/p&gt;&#xA;" OwnerUserId="5211" LastEditorUserId="24968" LastEditDate="2019-03-20T17:21:16.947" LastActivityDate="2019-03-20T17:21:16.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6604" PostTypeId="1" AcceptedAnswerId="6606" CreationDate="2015-07-28T08:53:18.033" Score="0" ViewCount="152" Body="&lt;p&gt;From &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#bayesian-ridge-regression&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/linear_model.html#bayesian-ridge-regression&lt;/a&gt;, they gave the bayesian ridge distribution as this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(w|\lambda) = \mathcal{N}(w|0,\lambda^{-1}{I_{p}})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there is a variable $I_p$ but it's unexplained what does the $I_p$ refer to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the variable $\mathcal{N}$ is unexplained but I'm not sure whether I've guessed correctly but is that the Gaussian prior as described in the Bayesian regression section above the Bayesian Ridge?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="21" LastEditDate="2015-07-28T11:07:07.237" LastActivityDate="2015-07-28T11:13:17.270" Title="What does the Ip mean in the Bayesian Ridge Regression formula?" Tags="|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6605" PostTypeId="2" ParentId="6604" CreationDate="2015-07-28T09:09:21.467" Score="0" Body="&lt;p&gt;&lt;code&gt;N&lt;/code&gt; is  function  and represents the Normal, or Gaussian, distribution. &lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-07-28T09:09:21.467" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6606" PostTypeId="2" ParentId="6604" CreationDate="2015-07-28T11:13:17.270" Score="3" Body="&lt;p&gt;$\mathcal{N}$ does indeed denote a (multivariate) normal / Gaussian distribution. $I_p$ is just an identity matrix of dimension $p$. So this a matrix with $\lambda^{-1}$ along the diagonal. Read this as the covariance matrix, so this is a spherical Gaussian (0 covariance between different dimensions) where each variable has variance $\lambda^{-1}$.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-07-28T11:13:17.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6607" PostTypeId="2" ParentId="6602" CreationDate="2015-07-28T12:39:20.250" Score="3" Body="&lt;p&gt;It's not easy to compare two RAMs with different frequency and latency, since both of them affect your performance with not the same way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The short answer is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have two RAMs with same capacity and frequency, choose the one with lower latency.&lt;/li&gt;&#xA;&lt;li&gt;If you have two RAMs with same capacity and latency, choose the one with the higher frequency.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://lifehacker.com/when-ram-speed-matters-and-how-it-affects-your-games-1436679680&quot; rel=&quot;nofollow&quot;&gt;lifehacker.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Essentially, you have two things to worry about when it comes to RAM &quot;speed&quot;: frequency, which deals with how much data can be transferred to the stick at one time, and latency, which is how quickly it responds to requests. In the current market, as you get to higher frequencies, latency tends to increase, so in many cases, they tend to balance each other out. Buying RAM with a higher speed doesn't matter a ton.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The long answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't worry about RAM. Yes, if you have a crappy RAM, you will also have a problem with your performance. But, the most important thing in Data Analysis is the performance of your code. An example from my personal experience:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a text dataset, about 30GB. I needed to create a corpus for a simple TF-IDF. The brute-force code, spent about 15 hours, whereas a map-reduce algorithm spent about 1.5 minute for the same dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work on a MacBook Pro with 16GB RAM. I didn't have any problem with performance. However, if you have huge datasets and even code improvements and RAM upgrades cannot help, you may reconsider to move to Hadoop or similar. &lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-07-28T12:39:20.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6608" PostTypeId="1" CreationDate="2015-07-28T12:40:39.007" Score="0" ViewCount="57" Body="&lt;p&gt;I have a job that results in a directory of part- files. I'd like to read it as if it were one file. Specifically, I'd like to read it that way over a web interface.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do either one of these things? Is there a hadoop component which makes this easy?&lt;/p&gt;&#xA;" OwnerUserId="10608" LastActivityDate="2015-07-30T04:21:27.090" Title="Accessing directory of small files as one file" Tags="|dataset|apache-hadoop|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6609" PostTypeId="2" ParentId="6595" CreationDate="2015-07-28T12:44:44.997" Score="1" Body="&lt;p&gt;Ricky,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Loose thoughts: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Depending on the algorithm you intend to use, centering might not be a good idea (e.g. if you go for SVM, centering will destroy sparsity)&lt;/li&gt;&#xA;&lt;li&gt;I would suggest not to handle ordered / unordered separately, as &lt;strong&gt;you are likely to miss interactions that way&lt;/strong&gt;. If the categorical ones don't have too many possible values, randomForest in R can handle factors.&lt;/li&gt;&#xA;&lt;li&gt;if that is an issue (as you seem to hint), I think you have two possibilities: binary indicators or response rates&lt;/li&gt;&#xA;&lt;li&gt;if it's feasible in terms of computational cost, i would convert all factors to binaries (use sparse matrices if necessary) and then try a greedy feature selection. caret, if memory serves, has rfe or somesuch.&lt;/li&gt;&#xA;&lt;li&gt;if that's too much trouble, try calculating response rates / average values per factor level (I don't see any info whether your problem is classification or regression): you split your set into folds, and then for each fold fit a mixed effects model (e.g. via lme4) on the remainder, using the factor of interest as the main variable. It's a bit of a pain to setup all the cv correctly, but it's the only way to avoid leaking information.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps,&#xA;K&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-07-28T12:44:44.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6610" PostTypeId="1" AcceptedAnswerId="6615" CreationDate="2015-07-28T14:55:13.340" Score="3" ViewCount="371" Body="&lt;p&gt;I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My belief is that a safer approach would be to use NoSQL storage such as &quot;Azure Table Storage&quot; in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't done a significant amount of research in this field, so would appreciate your input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?&lt;/li&gt;&#xA;&lt;li&gt;Would it be easy to initially start off with a simpler approach such as &quot;Azure Table Storage&quot; then onto Hadoop at a later point?&lt;/li&gt;&#xA;&lt;li&gt;If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?&lt;/li&gt;&#xA;&lt;li&gt;Is Hadoop necessary to invest from the get go as we will quickly incur major costs?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up. &lt;/p&gt;&#xA;" OwnerUserId="10940" LastActivityDate="2015-07-31T06:44:49.997" Title="Is our data &quot;Big Data&quot; (Startup)" Tags="|machine-learning|data-mining|bigdata|statistics|apache-hadoop|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6611" PostTypeId="1" CreationDate="2015-07-28T16:15:43.783" Score="7" ViewCount="4174" Body="&lt;p&gt;We're build an item-item recommender based on the text descriptions of the items. Our initial approach was to calculate the TF-IDF vectors for each item. We used a hashing tf with 5000 possible hashes for the words. Then approximate all-pairs using a sampling technique (DIMSUM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have an mxn matrix where m is the number of words, n is the number of items. The naive approach of calculating all column cosine similarities won't work here since we have n=10^7 m=10^4. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our first attempt was using DIMSUM &lt;a href=&quot;http://stanford.edu/~rezab/papers/dimsum.pdf&quot;&gt;http://stanford.edu/~rezab/papers/dimsum.pdf&lt;/a&gt; which is an all-pairs sampling technique. The problem with DIMSUM is it works for data where m &gt;&gt; n. Our matrix is short and wide n&gt;m.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Question:&#xA;What is a good approach for estimating all-pairs similarity of items based on words. Where number of items is 10^7, number of words is 10^4. We only want items pairs above a certain threshold of similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We don't have to do it this way, our task is to recommend a small set of items given one item, we have ways of doing this using collaborative filtering, but we want to also handle new items that we have no user data for so we're trying to find a way to use tf-idf vectors for that case.&lt;/p&gt;&#xA;" OwnerUserId="10943" LastActivityDate="2015-10-14T20:12:31.803" Title="Item-Item similarity based on text" Tags="|machine-learning|recommender-system|apache-spark|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6612" PostTypeId="1" CreationDate="2015-07-28T17:54:23.927" Score="10" ViewCount="12603" Body="&lt;p&gt;I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some dimensionality reduction method to scale the word embeddings with 300 dimensions down to 50 dimensions?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2023-05-07T18:20:39.717" Title="Reducing the dimensionality of word embeddings" Tags="|nlp|dimensionality-reduction|word-embeddings|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6613" PostTypeId="5" CreationDate="2015-07-28T17:54:44.533" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-07-28T17:54:44.533" LastActivityDate="2015-07-28T17:54:44.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6614" PostTypeId="4" CreationDate="2015-07-28T17:54:44.533" Score="0" Body="Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-25T12:54:30.460" LastActivityDate="2015-11-25T12:54:30.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6615" PostTypeId="2" ParentId="6610" CreationDate="2015-07-28T20:23:27.530" Score="5" Body="&lt;p&gt;Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. The Hadoop ecosystem has HBase for NoSQL; I suspect Azure has some kind of stream processing story.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bigger difference in your two alternatives is consuming a cloud provider's ecosystem vs Hadoop. The upside to Azure, or AWS, or GCE, is that these services optimize for integrating with each other, with billing, machine management, etc. The downside is being locked in to the cloud provider; you can't run Azure stuff anywhere but Azure. Hadoop takes more work to integrate since it's really a confederation of sometimes loosely-related projects. You're investing in both a distribution, and a place to run that distribution. But, you get a lot less lock-in, and probably more easy access to talent, and a broader choice of tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Azure road is also a &quot;big data&quot; solution in that it has a lot of the scalability properties you want for big data, and the complexity as well. It does not strike me as an easier route. Do you need to invest in distributed/cloud anything at this scale? given your IoT-themed use case, I believe you will need to soon, if not now, so yes. You're not talking about gigabytes, but many terabytes in just the first year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd give a fresh team 6-12 months to fully productionize something based on either of these platforms. That can certainly be staged as a POC, followed by more elaborate engineering.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-07-28T20:23:27.530" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6616" PostTypeId="1" AcceptedAnswerId="6617" CreationDate="2015-07-28T22:14:31.177" Score="2" ViewCount="63" Body="&lt;p&gt;GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on &lt;a href=&quot;http://stanford.edu/~rezab/sparkclass/slides/ankur_graphx.pdf&quot; rel=&quot;nofollow&quot;&gt;these slides&lt;/a&gt; (see slide 23). However, I am curious what characteristics of these algorithms make them parallelizable. &lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2015-07-29T11:27:48.940" Title="What makes a graph algorithm a good candidate for concurrency?" Tags="|algorithms|graphs|apache-spark|parallel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6617" PostTypeId="2" ParentId="6616" CreationDate="2015-07-29T00:51:18.110" Score="4" Body="&lt;p&gt;Two words: associative and commutative&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, the operations that the algorithm does need to be independent of how you order or group your data...this minimizes the need for cross-talk in the algorithm and leads to more efficiency.&lt;/p&gt;&#xA;" OwnerDisplayName="user9424" LastEditorDisplayName="user9424" LastEditDate="2015-07-29T11:27:48.940" LastActivityDate="2015-07-29T11:27:48.940" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6619" PostTypeId="2" ParentId="5233" CreationDate="2015-07-29T10:37:15.300" Score="0" Body="&lt;p&gt;Clarification: are we talking about multiclass&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Multiclass_classification&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or multilabel?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-label_classification&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Multi-label_classification&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-07-29T10:37:15.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6623" PostTypeId="1" CreationDate="2015-07-29T17:16:15.640" Score="1" ViewCount="36" Body="&lt;p&gt;Could someone point me to a standard data sharing agreement? I have an interesting data set that I want to share with my local university for  their Data Mining class. The data set includes time-series and email statistics. All domains and IPs are anonymized before release. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Main points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use at your own risk&lt;/li&gt;&#xA;&lt;li&gt;Protect my company and client's privacy&lt;/li&gt;&#xA;&lt;li&gt;Cite my company&lt;/li&gt;&#xA;&lt;li&gt;Do not re-share without permission&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Am I missing anything important here?&lt;/p&gt;&#xA;" OwnerUserId="10964" LastActivityDate="2015-07-29T17:16:15.640" Title="Sharing industry data with academic class contracts?" Tags="|data-mining|dataset|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6624" PostTypeId="1" CreationDate="2015-07-29T17:40:21.780" Score="2" ViewCount="107" Body="&lt;p&gt;We are working on a project to Predict the settings of the IoT devices(fan/light/AC) the user is using, based on his: location, outside temperature, humidity, time of the day etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training data is not available and the model should start building/adapting itself every time the user uses the IoT device, to predict his preferred settings in any new environment he goes. This model will be unique for each user. Possibly we can start off with a demo model but it should adapt itself as per each user's preferences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which (machine?)learning algorithm can be useful here? Any links to the same and some tips on implementation would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="10967" LastActivityDate="2015-07-30T17:30:32.120" Title="Adaptive learning of user's IoT setting preferences" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6626" PostTypeId="1" CreationDate="2015-07-29T20:50:40.583" Score="1" ViewCount="162" Body="&lt;p&gt;I've look at the questions on here regarding the different python libraries around for deep learning and neural nets. They include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Keras&lt;/li&gt;&#xA;&lt;li&gt;Caffe&lt;/li&gt;&#xA;&lt;li&gt;Lasagne&lt;/li&gt;&#xA;&lt;li&gt;PyLearn2&lt;/li&gt;&#xA;&lt;li&gt;Deepy&lt;/li&gt;&#xA;&lt;li&gt;Theano&lt;/li&gt;&#xA;&lt;li&gt;Torch&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My understanding is that Keras and Lasagne require the user to have varying degrees of interaction with Theano. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for the one that would be the easiest to start testing an algorithm I have already developed an apply deep learning on it in a packaged way. I am not looking to extend, modify or tweak anything. I certainly won't be doing any research in the field with it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only advanced feature I would need would be GPU support. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, what would be the best &quot;dummy&quot; library to test recursive convolutional neural nets? &lt;/p&gt;&#xA;" OwnerUserId="10949" LastActivityDate="2015-07-29T20:50:40.583" Title="&quot;Recursive ConvNets for Dummies&quot; Library" Tags="|neural-network|deep-learning|library|" AnswerCount="0" CommentCount="5" ClosedDate="2021-02-19T14:31:40.533" ContentLicense="CC BY-SA 3.0" />
  <row Id="6627" PostTypeId="2" ParentId="6624" CreationDate="2015-07-30T01:36:23.517" Score="1" Body="&lt;p&gt;This sounds like a fun problem but it very open ended!  I will provide some links from the &lt;a href=&quot;http://scikit-learn.org/stable/user_guide.html&quot; rel=&quot;nofollow&quot;&gt;Scikit-Learn User Guide&lt;/a&gt;.  There are many more reading options, but the SKL Userguide has lots of examples and usually links to academic publications for more in-depth reading.  Another great resource is: &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction to Statistical Learning&lt;/a&gt;, and if you are good at math: &lt;a href=&quot;http://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model Selection&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first big question is whether the settings you are trying to predict are ordinal values or continuous values?  Continuous value will allow you to use regression methods, where as ordinal values will give you a choice between regression methods and classification schemes.  I would suggest employing a couple of different models to start and then selecting the best one moving forward.  Some possible candidates are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html&quot; rel=&quot;nofollow&quot;&gt;Linear regression&lt;/a&gt; (with nonlinear features as appropriate) because linear models are usually insightful, &lt;/li&gt;&#xA;&lt;li&gt;Support Vector Classification/Regression (&lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow&quot;&gt;SVC&lt;/a&gt;/&lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#regression&quot; rel=&quot;nofollow&quot;&gt;SVR&lt;/a&gt;) as these are often very accurate classifiers. and either &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/naive_bayes.html&quot; rel=&quot;nofollow&quot;&gt;Naive Bayes&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html#forests-of-randomized-trees&quot; rel=&quot;nofollow&quot;&gt;Random Forests&lt;/a&gt; as these often give good results where other models fail.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Initial Training Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to prime your solution with some deterministic data or data from another dataset that you have adapted to fit your model.  I suggest priming the system with common sense values and then employing some sort of &lt;a href=&quot;http://arxiv.org/ftp/arxiv/papers/0709/0709.3965.pdf&quot; rel=&quot;nofollow&quot;&gt;algorithm to forget those values&lt;/a&gt; as the system becomes well-trained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Moving Forward&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have to decide whether each user's model will use other user's data or not.  This is sometimes solved via &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;nofollow&quot;&gt;clustering methods&lt;/a&gt; like &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#k-means&quot; rel=&quot;nofollow&quot;&gt;k-means&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#dbscan&quot; rel=&quot;nofollow&quot;&gt;DBSCAN&lt;/a&gt;), where by users are clustered into groups each cluster has a different model associated with it.  I would suggest retraining (fitting) the model on regular time frames initially, and then investigating the possibility of moving to an &lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning&quot; rel=&quot;nofollow&quot;&gt;online learning&lt;/a&gt; system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I absolutely suggest using a well known library for your problem. Scikit-Learn is great for the reasons mentioned above and for prototyping, but doesn't scale particularly well. &lt;a href=&quot;http://h2o.ai/&quot; rel=&quot;nofollow&quot;&gt;H20&lt;/a&gt; and &lt;a href=&quot;http://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout&lt;/a&gt; are high quality scalable libraries that I would recommend for a big data production system.  &lt;a href=&quot;http://www.manning.com/owen/&quot; rel=&quot;nofollow&quot;&gt;Mahout in Action&lt;/a&gt; is a great book to learn from, also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-30T17:30:32.120" LastActivityDate="2015-07-30T17:30:32.120" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6628" PostTypeId="2" ParentId="6608" CreationDate="2015-07-30T04:21:27.090" Score="1" Body="&lt;pre&gt;&lt;code&gt;hadoop fs -getmerge &amp;lt;hdfs-output-directory&amp;gt; &amp;lt;local-file&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This command can be used to concatenate the HDFS files into a single local file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref : &lt;a href=&quot;http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#getmerge&quot; rel=&quot;nofollow&quot;&gt;http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#getmerge&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-07-30T04:21:27.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6629" PostTypeId="1" CreationDate="2015-07-30T07:16:07.637" Score="5" ViewCount="416" Body="&lt;p&gt;I would like to know what are the best practices for anonymizing datasets? &#xA;Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be using some encryption functions? Hashing maybe? &lt;/p&gt;&#xA;" OwnerUserId="10979" LastActivityDate="2015-08-01T17:47:09.340" Title="Anonymizing Datasets" Tags="|data-mining|dataset|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6631" PostTypeId="1" AcceptedAnswerId="6638" CreationDate="2015-07-30T14:07:31.163" Score="4" ViewCount="47" Body="&lt;p&gt;Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a 2D data structure where 'orientation' doesn't matter. The examples I ran into:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The state of a &lt;a href=&quot;https://gabrielecirulli.github.io/2048/&quot; rel=&quot;nofollow&quot;&gt;2048&lt;/a&gt; game. In terms of symmetry groups this would be D&lt;sub&gt;4&lt;/sub&gt; / D&lt;sub&gt;8&lt;/sub&gt;, except that an operation doesn't yield an identical state, it just yields another state that has the same solution.&lt;/li&gt;&#xA;&lt;li&gt;Images of &lt;a href=&quot;https://www.kaggle.com/c/datasciencebowl&quot; rel=&quot;nofollow&quot;&gt;plankton&lt;/a&gt; or &lt;a href=&quot;https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge&quot; rel=&quot;nofollow&quot;&gt;galaxies&lt;/a&gt; (without background). Somewhat similar to above except that any rotation (not just 90&lt;sup&gt;o&lt;/sup&gt;) yields an equally valid image (and one might take scale into account, but let's forget about that).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In both cases I've wanted to transform all these equivalent states/images to remove all but one of the equivalent images. To illustrate with two that worked:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Image_moment&quot; rel=&quot;nofollow&quot;&gt;image moments&lt;/a&gt; M&lt;sub&gt;10&lt;/sub&gt; and M&lt;sub&gt;01&lt;/sub&gt; to transform horizontally and vertically mirrored equivalent data. E.g. apply horizontal mirroring iff it makes M&lt;sub&gt;10&lt;/sub&gt; bigger. This would transform a 2048 state and it's horizontal mirror image to the same state.&lt;/li&gt;&#xA;&lt;li&gt;I can use the eigenvector of the covariance matrix which has the largest eigenvalue as the orientation. Then I can rotate the image to align this eigenvector with some predetermined axis (e.g. horizontally).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That still leaves a lot of operations though (diagonal mirroring, rotations around the center, inversion). And these operations do not commute (D&lt;sub&gt;8&lt;/sub&gt; is non-Abelian). Is there any comprehensive approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I want to do this is to help machine learning methods by removing variance that isn't actually meaningful. Hopefully that makes sure they don't have to learn these equivalences, so possibly need less train data (and time).&lt;/p&gt;&#xA;" OwnerUserId="10907" LastActivityDate="2015-07-31T17:45:31.420" Title="Alignment of square nonorientable images/data" Tags="|data-cleaning|processing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6634" PostTypeId="1" CreationDate="2015-07-30T14:47:48.963" Score="4" ViewCount="229" Body="&lt;p&gt;I'd like to model the evolution of the sales of a store.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the data I have :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6FsZ8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6FsZ8.png&quot; alt=&quot;data table&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Customers are aggregated into monthly cohort depending on the date of the first purchase&#xA;&lt;em&gt;eg&lt;/em&gt; : customers who did their 1st purchase in Jan 2015 are in the cohort 1, customers who did their 1st purchase in Fev 2015 are in the cohort 2.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Sales : aggregated for each &lt;em&gt;Month&lt;/em&gt; * &lt;em&gt;Cohort&lt;/em&gt; (cohort 4 is not visible because there is no line for only 1 record)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xfAsk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xfAsk.png&quot; alt=&quot;sales over the time&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if we put all the cohorts at the same origin:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/kq0jQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kq0jQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; How to predict the spending of the next 2 months, ie :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the spending of the cohort 1 in its aging 5 and 6&lt;/li&gt;&#xA;&lt;li&gt;the spending of the cohort 2 in its aging 4 and 5&lt;/li&gt;&#xA;&lt;li&gt;the spending of the cohort 3 in its aging 3 and 4&lt;/li&gt;&#xA;&lt;li&gt;the spending of the cohort 4 in its aging 2 and 3&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But also for the cohort who don't exist yet :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the spending of the cohort 5 in its aging 1 and 2&lt;/li&gt;&#xA;&lt;li&gt;the spending of the cohort 6 in its aging 1 &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I have 2 methodologies in mind, but don't know if they're good:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Polynomial regression by using 2 parameters:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;aging: &lt;em&gt;because there is a clear evolution over the time, up then down.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;cohort number: &lt;em&gt;because more recent cohort seems to spend less.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Time series : I didn't perform time series recently I'm a bit rust so I would prefer to use something else but I think it could fit with this problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Indeed it seems obvious that y&lt;em&gt;t-1&lt;/em&gt; t in y&lt;em&gt;t-2&lt;/em&gt;... are good predictors of y&lt;em&gt;t&lt;/em&gt; but how to predict spending of future cohort with no history ?&lt;/p&gt;&#xA;" OwnerUserId="10983" LastEditorUserId="9420" LastEditDate="2015-07-31T02:37:53.037" LastActivityDate="2016-03-01T08:01:09.950" Title="Sales Predictions Over Time" Tags="|time-series|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6635" PostTypeId="1" CreationDate="2015-07-30T17:48:12.383" Score="2" ViewCount="3231" Body="&lt;p&gt;I am looking for a program that would allow me to fine-tune pre-trained word embeddings on my data set. Ideally, open source and working on Linux or Windows.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="21" LastEditDate="2015-07-31T08:32:13.697" LastActivityDate="2016-01-21T23:50:16.067" Title="Program to fine-tune pre-trained word embeddings on my data set" Tags="|nlp|word-embeddings|software-recommendation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6636" PostTypeId="5" CreationDate="2015-07-30T17:48:36.430" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-07-30T17:48:36.430" LastActivityDate="2015-07-30T17:48:36.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6637" PostTypeId="4" CreationDate="2015-07-30T17:48:36.430" Score="0" Body="Questions asking for specific applications that perform a given task or work with a given file format." OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-07-31T08:31:41.273" LastActivityDate="2015-07-31T08:31:41.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6638" PostTypeId="2" ParentId="6631" CreationDate="2015-07-30T21:14:40.950" Score="1" Body="&lt;p&gt;&lt;strong&gt;Fun with Group Theory!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are only 8 unique rotation-inversion operations for a square matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The four rotation operators are &lt;code&gt;(0,90,180,270)&lt;/code&gt;.  Further rotation or rotation in the reverse direction is the same as these four.  Two successive rotations just yields one of the rotation operators, so we will only consider these four rotations applied one time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The five inversion operators are &lt;code&gt;(0,/,\,|,-)&lt;/code&gt;.  Two successive inversions just yields a rotation, so we will only allow for a single inversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can thus derive all operators by combining these two vectors, which yields &lt;code&gt;4*5=20&lt;/code&gt; possible states.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0,90,180,270,0/,90/,180/,270/,0\,90\,180\,270\,0|,90|,180|,270|,0-,90-,180-,270-)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But there is still symmetry to be exploited in the inversion operators.  You can probably intuit that the 4x4 matrix only has 8 final states: inverted or not and rotated by (0,90,180,270).  It turns out that you can arrive at any possible state involving an inversion using any of the other inversion operators followed by one of the rotations.  So we only need to retain a single inversion operator!  So the final set of 8 orthogonal operations are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0,90,180,270,0|,90|,180|,270|)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If there is any symmetry in the matrix's members then some of the resulting states may be degenerate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of mapping possible states into a ground state, it makes sense to apply a set of successive deterministic rules to determine the ground state orientation.  I suggest finding the largest corner square and locating it in the lower right corner. If there are multiple candidates with equally large values in the corner square then use the next closest square as a tie breaker.  There are 16 squares, so you can eventually break all ties or declare degeneracy. There is one remaining &lt;code&gt;\&lt;/code&gt; operation that you can decide to apply in order to locate the larger of the two squares adjacent to the lower right corner at the bottom.  Again, you can use squares adjacent to these as tie breakers. &lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-07-31T17:45:31.420" LastActivityDate="2015-07-31T17:45:31.420" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6639" PostTypeId="1" AcceptedAnswerId="6644" CreationDate="2015-07-31T00:25:36.347" Score="26" ViewCount="11400" Body="&lt;p&gt;I'm working through a question from the &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap1.html&quot; rel=&quot;noreferrer&quot;&gt;online book&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I can understand that if the additional output layer is of 5 output neurons, I could probably set bias at 0.5 and weight of 0.5 each for the previous layer. But the question now ask for a new layer of four output neurons - which is more than enough to represent 10 possible outputs at &lt;span class=&quot;math-container&quot;&gt;$2^{4}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Can someone walk me through the steps involved in understanding and solving this problem?&lt;/p&gt;&#xA;&lt;p&gt;The exercise question:&lt;/p&gt;&#xA;&lt;p&gt;There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OqQ6N.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OqQ6N.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10990" LastEditorUserId="98307" LastEditDate="2020-08-05T11:06:35.280" LastActivityDate="2020-11-07T00:19:45.413" Title="Extra output layer in a neural network (Decimal to binary)" Tags="|neural-network|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6640" PostTypeId="2" ParentId="6635" CreationDate="2015-07-31T04:39:50.677" Score="4" Body="&lt;p&gt;While I am not aware of software &lt;em&gt;specifically&lt;/em&gt; for &lt;strong&gt;tuning&lt;/strong&gt; trained word embeddings, perhaps the following &lt;em&gt;open source software&lt;/em&gt; might be helpful, if you can figure out what parts can be modified for the fine-tuning part (just an idea off the top of my head - I'm not too familiar with the details):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://nlp.stanford.edu/projects/glove&quot; rel=&quot;nofollow&quot;&gt;GloVe: Global Vectors for Word Representation&lt;/a&gt; (part of &lt;em&gt;Stanford NLP Group&lt;/em&gt; software);&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ml.nec-labs.com/senna&quot; rel=&quot;nofollow&quot;&gt;SENNA&lt;/a&gt; (its use for word embeddings is mentioned in &lt;a href=&quot;http://lebret.ch/words&quot; rel=&quot;nofollow&quot;&gt;this blog post&lt;/a&gt;);&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/vaswani/WORD_OBSERVATION_RBM_NCE&quot; rel=&quot;nofollow&quot;&gt;Code on GitHub&lt;/a&gt; for Deep Learning-based word embeddings training;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://nlg.isi.edu/software/nplm&quot; rel=&quot;nofollow&quot;&gt;Neural Probabilistic Language Model Toolkit&lt;/a&gt;, mentioned above (also Deep Learning-based).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-07-31T04:39:50.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6641" PostTypeId="2" ParentId="6610" CreationDate="2015-07-31T06:44:49.997" Score="0" Body="&lt;p&gt;First, &lt;strong&gt;understand and solve the problem&lt;/strong&gt;. On manageable data. Gather experience on how to organize the data, and &lt;em&gt;where&lt;/em&gt; the difficulties are. Try to identify points where parallelism is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, &lt;strong&gt;parrallelize&lt;/strong&gt; and scale up as necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't do it backwards, popular mistake. Solving the wrong problem with the wrong tools will fail big, with big data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-07-31T06:44:49.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6642" PostTypeId="1" AcceptedAnswerId="9257" CreationDate="2015-07-31T07:08:07.880" Score="3" ViewCount="1263" Body="&lt;p&gt;I have a predictive model which I trained on a training set. I have written it in R. Now I want to deploy it as a web service so anyone can just input the data into it and get the output from the predictive model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to use Azure ML for deploying. I wanted to know whether I can drag and drop my already created/custom trained model to Azure ML studio instead of re-training it there ? I know we can train and save models in AML Studio but I am not sure about adding already created models and using them in AML solution. Help regarding this will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="8016" LastEditorUserId="843" LastEditDate="2015-12-07T18:34:20.597" LastActivityDate="2016-05-27T16:21:48.643" Title="Custom trained model in Azure ML" Tags="|machine-learning|r|predictive-modeling|azure-ml|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6643" PostTypeId="1" AcceptedAnswerId="6650" CreationDate="2015-07-31T08:21:49.083" Score="5" ViewCount="6912" Body="&lt;p&gt;I am new in deep learning. I am running a MacBook Pro yosemite (upgraded from Snowleopard). I don't have a CUDA-enabled card GPU, and running the code on the CPU is extremely slow. I heard that I can buy some instances on AWS, but it seems that they don't support macOS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, to continue with the deep learning, do I need to purchase a graphic card? Or is there other solution? I don't want to spend too much on this... &lt;/p&gt;&#xA;" OwnerUserId="10994" LastEditorUserId="8432" LastEditDate="2017-05-29T04:59:32.770" LastActivityDate="2017-05-29T04:59:32.770" Title="Do I need to buy a NVIDIA graphic card to run deep learning algorithm?" Tags="|deep-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6644" PostTypeId="2" ParentId="6639" CreationDate="2015-07-31T09:26:34.927" Score="18" Body="&lt;p&gt;The question is asking you to make the following mapping between old representation and new representation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Represent    Old                     New&#xA;0            1 0 0 0 0 0 0 0 0 0     0 0 0 0 &#xA;1            0 1 0 0 0 0 0 0 0 0     0 0 0 1 &#xA;2            0 0 1 0 0 0 0 0 0 0     0 0 1 0 &#xA;&#xA;3            0 0 0 1 0 0 0 0 0 0     0 0 1 1 &#xA;4            0 0 0 0 1 0 0 0 0 0     0 1 0 0 &#xA;5            0 0 0 0 0 1 0 0 0 0     0 1 0 1 &#xA;&#xA;6            0 0 0 0 0 0 1 0 0 0     0 1 1 0 &#xA;7            0 0 0 0 0 0 0 1 0 0     0 1 1 1 &#xA;8            0 0 0 0 0 0 0 0 1 0     1 0 0 0 &#xA;&#xA;9            0 0 0 0 0 0 0 0 0 1     1 0 0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because the old output layer has a simple form, this is quite easy to achieve. Each output neuron should have a positive weight between itself and output neurons which should be on to represent it, and a negative weight between itself and output neurons that should be off. The values should combine to be large enough to cleanly switch on or off, so I would use largish weights, such as +10 and -10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have sigmoid activations here, the bias is not that relevant. You just want to simply saturate each neuron towards on or off. The question has allowed you to assume very clear signals in the old output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So taking example of representing a 3 and using zero-indexing for the neurons in the order I am showing them (these options are not set in the question), I might have weights going from activation of old output $i=3$, $A_3^{Old}$ to logit of new outputs $Z_j^{New}$, where $Z_j^{New} = \Sigma_{i=0}^{i=9} W_{ij} * A_i^{Old}$ as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_{3,0} = -10$$&#xA;$$W_{3,1} = -10$$&#xA;$$W_{3,2} = +10$$&#xA;$$W_{3,3} = +10$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should clearly produce close to &lt;code&gt;0 0 1 1&lt;/code&gt; output when only the old output layer's neuron representing a &quot;3&quot; is active. In the question, you can assume 0.99 activation of one neuron and &amp;lt;0.01 for competing ones in the old layer. So, if you use the same magnitude of weights throughout, then relatively small values coming from +-0.1 (0.01 * 10) from the other old layer activation values will not seriously affect the +-9.9 value, and the outputs in the new layer will be saturated at very close to either 0 or 1.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-07-31T10:06:02.600" LastActivityDate="2015-07-31T10:06:02.600" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="6645" PostTypeId="1" CreationDate="2015-07-31T10:54:37.760" Score="2" ViewCount="3969" Body="&lt;p&gt;I am building a classification model using &lt;code&gt;randomForest&lt;/code&gt;. When trying to predict I get the below error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Type of predictors in new data do not match that of the training data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I made sure that the testing and training data has same levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also included &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;levels(test_var) &amp;lt;- levels(train_var)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to make sure that the levels are matching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But still I end up getting this error, is there anything else that I should look for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;**EDITED ON 3rd Aug 2015**&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here is the structure of the training dataset&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xnCH6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xnCH6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Structure of the test dataset&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gJQcP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gJQcP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Sapply training data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OhRLR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OhRLR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Sapply testing data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/evf0i.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/evf0i.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In order to make sure that the levels are matching between the training and test datasets, I wrote this loop to see if there are any differences exist between the datasets&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:28) &#xA;&#xA; {&#xA;   if(is.factor(testing[,i]))&#xA;    {&#xA;&#xA;    print(names((testing[i])))&#xA;    difference_in_test = setdiff(levels(testing[,i]), levels(training_data[,i]))&#xA;      print(difference_in_test)&#xA;&#xA;    }&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Results of the above For loop which shows that there are no differences exist between the levels.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1] &quot;VAR1&quot;&#xA;character(0)&#xA;[1] &quot;VAR2&quot;&#xA;character(0)&#xA;[1] &quot;VAR3&quot;&#xA;character(0)&#xA;[1] &quot;VAR4&quot;&#xA;character(0)&#xA;[1] &quot;VAR5&quot;&#xA;character(0)&#xA;[1] &quot;VAR7&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I still continue to get the error as mentioned below:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7Dyv0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7Dyv0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Edit on 4th Aug&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I started getting the above error only after including a filter in my dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;training_data &amp;lt;- subset(training_data, gender !=&quot;F&quot;)&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="9793" LastEditDate="2015-08-04T04:31:09.220" LastActivityDate="2015-08-04T04:31:09.220" Title="Error::Type of predictors in new data do not match that of the training data" Tags="|machine-learning|data-mining|r|classification|random-forest|" AnswerCount="0" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="6646" PostTypeId="2" ParentId="6642" CreationDate="2015-07-31T11:59:35.597" Score="0" Body="&lt;p&gt;I don't think you can currently upload a trained model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your options would be to either re-train the model in AzureML or expose them as a web-service using an Azure Virtual Machine running something like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/Rook/index.html&quot; rel=&quot;nofollow&quot;&gt;Rook&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shiny.rstudio.com/&quot; rel=&quot;nofollow&quot;&gt;Shiny&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.revolutionanalytics.com/whitepaper/using-deployr-to-solve-the-integration-problem&quot; rel=&quot;nofollow&quot;&gt;DeployR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10998" LastActivityDate="2015-07-31T11:59:35.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6647" PostTypeId="2" ParentId="6643" CreationDate="2015-07-31T14:21:03.577" Score="-1" Body="&lt;p&gt;AWS does &quot;support&quot; Mac OS. You can use any SSH client from your Mac to access the GPU instance. &lt;/p&gt;&#xA;" OwnerUserId="10949" LastActivityDate="2015-07-31T14:21:03.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6648" PostTypeId="1" AcceptedAnswerId="6651" CreationDate="2015-07-27T12:59:05.890" Score="5" ViewCount="2766" Body="&lt;p&gt;I have some features and I am using Weka to classify my instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Number of adj   number of adverb    number of punctuation&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;in my feature set. However, I would like to know the contribution of each feature in the feature set. So what metrics or parameters are helpful to get the contribution of features?&lt;/p&gt;&#xA;" OwnerUserId="11072" OwnerDisplayName="Hani" LastEditorUserId="29169" LastEditDate="2019-06-07T21:52:01.317" LastActivityDate="2019-06-07T21:52:01.317" Title="Contributions of each feature in classification?" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6649" PostTypeId="1" CreationDate="2015-07-31T17:20:10.873" Score="5" ViewCount="5118" Body="&lt;p&gt;I am trying to see if my data is multimodal (in fact, I am more interested in bimodality of the data). I performed dip test and it does evidence against unmodal data. However, I want to see, in particular, if it is bimodal. I believe silver man's test can be used. However, I couldn't find the implementation of it in either r or in python. (The one in R is old and not working with the current version of R). &#xA;Also, assuming that I have a bimodal data and that I am able to get the two components (using mixtools in R), how do I figure out how to find the point of intersection of the two components. For example, here is the histogram (overlaid with its density estimation) of the entire data. &lt;a href=&quot;https://i.stack.imgur.com/FO43X.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FO43X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the two components:&lt;a href=&quot;https://i.stack.imgur.com/gFoDm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gFoDm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to get the the x value where the curves intersected. I could have uploaded the data, but the length of the vector is rather long. Any general thought and idea is welcome including the R and/or Python packages are welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2016-03-29T23:07:32.927" Title="Testing bimodality of data" Tags="|machine-learning|r|python|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6650" PostTypeId="2" ParentId="6643" CreationDate="2015-07-31T18:58:25.117" Score="4" Body="&lt;p&gt;I would recommend familiarizing yourself with &lt;a href=&quot;https://www.youtube.com/watch?v=Py0VInjRSBE&quot; rel=&quot;nofollow&quot;&gt;AWS spot instances&lt;/a&gt;. It's the most practical solution I can think of for your problem, and &lt;a href=&quot;https://www.youtube.com/watch?v=NdR03RpCpac&quot; rel=&quot;nofollow&quot;&gt;it works your computer too&lt;/a&gt;. So, no you don't have to &lt;em&gt;buy&lt;/em&gt; an Nvidia card, but as of today you will want to use one since almost all the solutions rely on them.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-07-31T18:58:25.117" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6651" PostTypeId="2" ParentId="6648" CreationDate="2015-07-31T19:49:03.447" Score="3" Body="&lt;p&gt;This is called feature ranking, which is closely related to &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature selection&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;feature ranking = determining the importance of any individual feature&lt;/li&gt;&#xA;&lt;li&gt;feature selection = selecting a subset of relevant features for use in model construction. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So if you are able to ranked features, you can use it to select features, and if you can select a subset of useful features, you've done at least a partial ranking by removing the useless ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page&lt;/a&gt; and this &lt;a href=&quot;https://www.quora.com/How-do-I-perform-feature-selection?share=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora post&lt;/a&gt; should give some ideas. The distinction filter methods vs. wrapper based methods vs. embedded methods is the most common one.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;One straightforward approximate way is to use &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature importance with forests of trees&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/r7Io5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/r7Io5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other common ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;recursive feature elimination&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;stepwise regression (or &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#lars-lasso&quot; rel=&quot;nofollow noreferrer&quot;&gt;LARS Lasso&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you use scikit-learn, check out &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;module-sklearn.feature_selection&lt;/a&gt;. I'd guess Weka has some similar functions.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-08-01T17:41:58.990" LastActivityDate="2015-08-01T17:41:58.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6653" PostTypeId="2" ParentId="6290" CreationDate="2015-07-31T21:03:23.593" Score="3" Body="&lt;p&gt;Normalise using your training data statistics. Save the values used (e.g. mean and sd per feature), treating them as part of your model. Once you have used these values to transform input, they become fixed translate/scale factors in the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the same values to normalise test data or new inputs as required. You do not need to calculate new normalisation constants for new data. In fact doing so will most likely reduce the effectiveness of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same principle applies to interpreting output values if you need to scale those into range that your model produces. Scale according to your training data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-07-31T21:03:23.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6654" PostTypeId="2" ParentId="6558" CreationDate="2015-08-01T10:56:11.233" Score="1" Body="&lt;p&gt;Other-than Ontologies do check &lt;a href=&quot;https://code.google.com/p/word2vec/&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt;, this tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words and can be used to find the closest words for a user-specified word&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-08-01T10:56:11.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6655" PostTypeId="2" ParentId="6629" CreationDate="2015-08-01T17:47:09.340" Score="4" Body="&lt;p&gt;You should make the difference between the following concepts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Encryption&quot; rel=&quot;nofollow&quot;&gt;encryption&lt;/a&gt; = encoding information in such a way that only authorized parties can read it.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Data_anonymization&quot; rel=&quot;nofollow&quot;&gt;data anonymization&lt;/a&gt; = removing personally identifiable information from data sets, so that the people whom the data describe remain anonymous.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/De-identification&quot; rel=&quot;nofollow&quot;&gt;de-identification&lt;/a&gt; = preventing a person’s identity from being connected with information, while preserving identifying information which could only be re-linked by a trusted party in certain situations (unlike data anonymization which aims to be irreversible).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your choice depends on your use case.&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-08-01T17:47:09.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6659" PostTypeId="1" CreationDate="2015-08-02T09:22:41.890" Score="2" ViewCount="64" Body="&lt;p&gt;I'm intrigued by the open data provided by StackExchange, and have been running some really interesting queries on the &lt;em&gt;data.stackexchange.com&lt;/em&gt; page (using the Stackoverflow dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I would like to dive deep on this claim taken from the &lt;a href=&quot;https://stackoverflow.com/research/developer-survey-2015&quot;&gt;Stackoverflow Developer Survey 2015 summary page&lt;/a&gt;: &lt;a href=&quot;https://i.stack.imgur.com/oqrtI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oqrtI.png&quot; alt=&quot;Taken from the recap of the Stackoverflow developer survey page&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words: what is the &quot;value&quot; of Stackoverflow? Not in terms of its own economic success or funding amount, rather how it affects its users, for example accelerating a pay increase for active users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My ideal solution would be to correlate the activity of its users to some economic independent variable such as salary. Although the data on activity exists on Stackoverflow servers, this Salary data is not available (the downloadable StackOverflow Developer Survey dataset is anonymized).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other proxies/datasets could be used to conduct such a study? I would use github commits but it is more oriented towards productivity, and there already is &lt;a href=&quot;https://bvasiles.github.io/papers/socialcom13.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a great paper on this topic&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11028" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-08-02T09:22:41.890" Title="What proxies could be used to assess economic value of Stackoverflow for its users?" Tags="|data|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6660" PostTypeId="1" CreationDate="2015-08-02T11:48:27.367" Score="1" ViewCount="1957" Body="&lt;p&gt;I am just starting to learn to use R and am not sure how to find the best packages yet. I am looking for a package that will allow me to calculate user-based nearest neighbours as an input for a recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original data is approximately as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User    Item    Rating&#xA;Anna    Apple   2&#xA;Bob Apple   4&#xA;Carlos  Apple   1&#xA;Ethan   Apple   1&#xA;Gene    Apple   2&#xA;Anna    Banana  4&#xA;Dana    Banana  3&#xA;Ethan   Banana  5&#xA;Frank   Banana  5&#xA;Bob Carrot  1&#xA;Ethan   Carrot  2&#xA;Gene    Carrot  4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have used the package reshape2 and dcast method to generate a different datastructure where users are rows, items are columns and the rating is in the value section (similar to an Excel pivot).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User    Apple   Banana  Carrot&#xA;Anna    2   4   n/a&#xA;Bob 4   n/a 1&#xA;Carlos  1   n/a n/a&#xA;Dana    n/a 3   n/a&#xA;Ethan   1   5   2&#xA;Frank   n/a 5   n/a&#xA;Gene    2   n/a 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any R package that will allow me to compute the similarity between users based on the user vectors? Preferably I would like to use the Jaccard coefficient as my similarity metric. I tried to google for R packages with user-based nearest neighbour implementations but came up empty. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help of alternative suggestions will be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="11029" LastActivityDate="2018-11-07T13:00:18.783" Title="User-based nearest neighbour implementation in R?" Tags="|r|recommender-system|similarity|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6663" PostTypeId="2" ParentId="6649" CreationDate="2015-08-02T18:39:25.623" Score="2" Body="&lt;p&gt;There has been an answer on this question, please check it : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/51062/test-for-bimodal-distribution&quot;&gt;Test for bimodal distribution&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7714" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-08-02T18:39:25.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6664" PostTypeId="2" ParentId="6295" CreationDate="2015-08-03T06:27:50.220" Score="3" Body="&lt;p&gt;If you want to get but the &lt;code&gt;sum&lt;/code&gt; and &lt;code&gt;length&lt;/code&gt; of &lt;code&gt;id&lt;/code&gt; with &lt;code&gt;data.table&lt;/code&gt;, you can do it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt[,.(total=sum(id), n=length(id)), by=cid]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could even write this in shorter format with the &lt;code&gt;.N&lt;/code&gt; operator of &lt;code&gt;data.table&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt[,.(total=sum(id), .N), by=cid]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9253" LastEditorUserId="9253" LastEditDate="2015-08-03T07:48:23.380" LastActivityDate="2015-08-03T07:48:23.380" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6665" PostTypeId="1" CreationDate="2015-08-03T11:45:11.440" Score="5" ViewCount="396" Body="&lt;p&gt;I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics.  I combined all of these documents into a LDA and got a topic like the one below.  I'm listing the top 30 terms, in descending order.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TOPIC 83: ['produced', 'company', 'john', 'weight', 'motion', 'information', 'painting', 'original', 'training', 'people', 'health', 'life', 'jesus', 'feature', 'body', 'lucretia', 'computer', 'graphics', 'time', 'madonna', 'story', 'florence', 'type', 'animated', 'paul', 'diabetes', 'animation', 'exercise', 'peter', 'film']&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of these words do not co-occur with each other in any documents.  For instance, 'lucretia' does not co-occur with 'exercise'.  Yet how are these terms put together in the same topic, in the top 30 words? (30 out of 20K words or so)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps it has to do with my particular implementation?  I'm using the &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;noreferrer&quot;&gt;gensim&lt;/a&gt; library for Python.  Or is it a flaw of LDA generally?&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2015-08-03T13:31:35.363" Title="Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?" Tags="|python|topic-model|lda|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6666" PostTypeId="2" ParentId="6665" CreationDate="2015-08-03T12:19:06.493" Score="2" Body="&lt;p&gt;It is a feature (I won't call it flaw) of LDA generally. Usually, you get some very clearly defined topics and some very muddy topics (best termed &quot;Miscellaneous&quot; or so).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also advice not to overinterprete the topics from topic modelling. Just changing a little (e.g. the random seed, taking a few texts out, applying some &quot;small&quot; modifications to the corpus) changes a significant amount of the topics.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-08-03T12:19:06.493" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6667" PostTypeId="2" ParentId="6665" CreationDate="2015-08-03T13:31:35.363" Score="4" Body="&lt;p&gt;It could be partially due to the number of topics you selected but the fact that two words rank high for a given topic doesn't necessarily mean that the two words will frequently occur in the same document. Consider a topic that generally corresponds to &quot;medicine&quot;. You could have a number of medical documents associated with heart disease and a number of medical documents associated with injuries. They may all be linked by common terms such as &quot;diagnose&quot;, &quot;treatment&quot;, &quot;patient&quot;, etc. But you wouldn't necessarily expect that words like &quot;heart&quot; and &quot;aorta&quot; would appear in documents containing words such as &quot;sprain&quot; and &quot;fracture&quot;, even though they are all associated with the general topic of medicine. If one were to choose a larger number of topics, those words might be separated into more specific topics (e.g., &quot;heart disease&quot; and &quot;sports injuries&quot;).&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2015-08-03T13:31:35.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6670" PostTypeId="2" ParentId="6643" CreationDate="2015-08-03T22:08:42.203" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html&quot; rel=&quot;nofollow&quot;&gt;AWS GPU instances&lt;/a&gt; are an option, if you want to do CUDA development. If you don't want to leverage the cloud, you can look into the &lt;a href=&quot;https://developer.nvidia.com/jetson-tk1&quot; rel=&quot;nofollow&quot;&gt;Nvidia Jetson TK1&lt;/a&gt; development kits.  They are about 200 dollars (in July 2015), have 192 CUDA cores, a quad-core ARM processor, and 2GB of RAM.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conversely, the same amount of money could buy you a 640 CUDA core GeForce GTX 750 Ti, or maybe a 1024 CUDA core GTX 960.  A GT 720 with 1GB RAM and 192 cores could be had for 45 dollars (in July 2015).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't &lt;em&gt;have&lt;/em&gt; to use NVidia GPUs with deep learning.  GPUs will increase the speed dramatically, though.   There is very little support for non-NVidia GPUs with common deep learning toolkits.  &lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2015-08-03T22:08:42.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6672" PostTypeId="2" ParentId="6634" CreationDate="2015-08-04T01:14:53.830" Score="1" Body="&lt;p&gt;It sounds like you are making this more complicated than it is with trying to predict spend at the cohort level. It is best to use cohorts for understanding &quot;why&quot; something occurred and time series analysis for understanding &quot;what&quot; will occur. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Forecast package in R seems to be solution to your problem. Just forecast Monthly spend and be done with it! &lt;/p&gt;&#xA;" OwnerUserId="11054" LastActivityDate="2015-08-04T01:14:53.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6673" PostTypeId="2" ParentId="6576" CreationDate="2015-08-04T01:29:21.623" Score="5" Body="&lt;p&gt;Disagree with David, a true data scientist is an applied statistician who codes and knows how to use machine learning algorithms for the right reasons. Statistics is the base of all data science. It is the &quot;cake&quot; per se. Everything else is just icing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is what kind of data scientist do you want to be? Do you want to be a master of the subject (knowledge of how, why, when and when not to apply an algorithm or technique) or a Kaggle Script Kiddie using Scipy and thinking that he is a Data Scientist? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Stats&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Everything else&lt;/p&gt;&#xA;" OwnerUserId="11054" LastActivityDate="2015-08-04T01:29:21.623" CommentCount="11" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="6674" PostTypeId="1" CreationDate="2015-08-04T03:18:22.520" Score="3" ViewCount="5491" Body="&lt;p&gt;I am a green-hand in calculating the time complexity. Given a calculation as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\mathbf{x}=\mathbf{A^T}(\mathbf{AA^T}+\lambda\mathbf{I}_n)^{-1}\mathbf{b}&#xA;\end{equation}&#xA;where $\mathbf{A}\in R^{n\times p}$, $\mathbf{I}_n\in R^{n\times n}$, $\mathbf{b}\in R^{n}$ and $\lambda\in R$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could some one kindly give me the complexity of the above formula in details? Thanks a lot. &lt;/p&gt;&#xA;" OwnerUserId="9893" LastActivityDate="2020-01-29T21:58:45.860" Title="How to calculate a specific time complexity of inverse calculation of matrix?" Tags="|data-mining|algorithms|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6675" PostTypeId="1" CreationDate="2015-08-04T06:57:33.607" Score="0" ViewCount="760" Body="&lt;p&gt;I fount simple neural scenario sayinh : network having 3 layers, 3 inputs and 2 outputs. It should be trained to recognize a simple pattern - if inputs are correspondingly 6.0,7.0 and 8.0 then outputs should be 3.0 and 4.0, otherwise outputs should be 23.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/84WUh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/84WUh.png&quot; alt=&quot;3 layered neural network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I understand is, hidden layer should Check input pattern in sequence with simple if else condition. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if in1 == 6 and in2 == 7 and in3 ==8:&#xA;    out1, out2 = 3, 4&#xA;else&#xA;    out = 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do I understand the problem correctly or something I am missing?&lt;/p&gt;&#xA;" OwnerUserId="8840" LastActivityDate="2015-08-05T06:39:48.220" Title="identifying input sequence using neural network" Tags="|machine-learning|classification|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6676" PostTypeId="1" AcceptedAnswerId="9794" CreationDate="2015-08-04T08:11:30.990" Score="28" ViewCount="46092" Body="&lt;p&gt;A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is take a scikit-learn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;SGDClassifier&lt;/a&gt; and have it score the same as a Logistic Regression &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. However, I must be missing some machine learning enhancements, since my scores are not equivalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my current code. What am I missing on the SGDClassifier which would have it produce the same results as a Logistic Regression?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.linear_model import SGDClassifier&#xA;import numpy as np&#xA;import pandas as pd&#xA;from sklearn.cross_validation import KFold&#xA;from sklearn.metrics import accuracy_score&#xA;&#xA;# Note that the iris dataset is available in sklearn by default.&#xA;# This data is also conveniently preprocessed.&#xA;iris = datasets.load_iris()&#xA;X = iris[&quot;data&quot;]&#xA;Y = iris[&quot;target&quot;]&#xA;&#xA;numFolds = 10&#xA;kf = KFold(len(X), numFolds, shuffle=True)&#xA;&#xA;# These are &quot;Class objects&quot;. For each Class, find the AUC through&#xA;# 10 fold cross validation.&#xA;Models = [LogisticRegression, SGDClassifier]&#xA;params = [{}, {&quot;loss&quot;: &quot;log&quot;, &quot;penalty&quot;: &quot;l2&quot;}]&#xA;for param, Model in zip(params, Models):&#xA;    total = 0&#xA;    for train_indices, test_indices in kf:&#xA;&#xA;        train_X = X[train_indices, :]; train_Y = Y[train_indices]&#xA;        test_X = X[test_indices, :]; test_Y = Y[test_indices]&#xA;&#xA;        reg = Model(**param)&#xA;        reg.fit(train_X, train_Y)&#xA;        predictions = reg.predict(test_X)&#xA;        total += accuracy_score(test_Y, predictions)&#xA;    accuracy = total / numFolds&#xA;    print &quot;Accuracy score of {0}: {1}&quot;.format(Model.__name__, accuracy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Accuracy score of LogisticRegression: 0.946666666667&#xA;Accuracy score of SGDClassifier: 0.76&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8774" LastActivityDate="2018-02-16T07:01:59.707" Title="Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression" Tags="|python|logistic-regression|scikit-learn|gradient-descent|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6679" PostTypeId="1" AcceptedAnswerId="6704" CreationDate="2015-08-04T10:48:20.727" Score="1" ViewCount="338" Body="&lt;p&gt;I’m working on comparing bacteria metabolic models. Each model has a set of metabolites (around 2000) and their concentration for 200 time points. I’m in the process of comparing the models to cluster them based on their similarity. One method I followed is I did a pair wise comparison for each of the metabolite pairs in two models using Euclidean distance. Below is how my data look like. This is a &lt;a href=&quot;http://1drv.ms/1JD8PHX&quot; rel=&quot;nofollow noreferrer&quot;&gt;sample data file&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yvoPj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yvoPj.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I computed pair wise Euclidean distance for Met1 from Model A and Met1 from Model B. Likewise computed the distances for all the common metabolites between the 2 models (Met4 in Model A and Met4 in Model B) and summed up the distances to get a distance (dissimilarity) between the two models. Similarly I computed the dissimilarity matrix for all the models and I used hierarchical clustering to cluster them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned above, now I want to compute the dissimilarity of the models using Discrete Wavelet Transformation as my distance measure. I would like to know how to use Discrete Wavelet Transformation to compute a dissimilarity distance between 2 time series and hence for my models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Previously I used DWT as a distance measure with DBSCAN for clustering metabolites in one model according to their behavior. It worked fine.&lt;/p&gt;&#xA;" OwnerUserId="11063" LastActivityDate="2015-08-05T14:35:57.897" Title="R: Comparing dissimilarity between metabolic models with discrete wavelet transformation" Tags="|r|clustering|time-series|bioinformatics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6680" PostTypeId="1" AcceptedAnswerId="6685" CreationDate="2015-08-04T12:36:20.967" Score="1" ViewCount="361" Body="&lt;p&gt;Is it possible to use a sequence of numbers as one feature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, using libsvm data format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;label&amp;gt; &amp;lt;index1&amp;gt;:&amp;lt;value1&amp;gt; &amp;lt;index2&amp;gt;:&amp;lt;value2&amp;gt;&#xA;&#xA;+1 1:123.02 2:1.23 3:5.45,2.22,6.76&#xA;+1 1:120.12 2:2.23 3:4.98,2.55,4.45&#xA;-1 1:199.99 2:2.13 3:4.98,2.22,6.98&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any special machine learning algorithm for this kind of data?&lt;/p&gt;&#xA;" OwnerUserId="11064" LastEditorUserId="28347" LastEditDate="2017-03-05T22:39:52.887" LastActivityDate="2017-03-06T00:01:14.520" Title="Sequence of numbers as single feature" Tags="|machine-learning|dataset|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6681" PostTypeId="1" CreationDate="2015-08-04T13:44:22.333" Score="6" ViewCount="760" Body="&lt;p&gt;I'm trying to define a churn prediction model for an online service (betting/gambling). A lot of papers talk about churn analysis/prediction for telco companies where defining a churn user is straightforward: a churn user is a user who cancels his or her contract. I'm wondering how to define a churn user within the context of an industry where users don't cancel any contract/account?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can envision a general approach: if a user who plays every day for a month stops playing for 2 days, this is could be flagged, but a user who only plays once a week and then has no activity for 2 days would not. What's the best approach to encoding this logic algorithmically? Is survival analysis and good approach?&lt;/p&gt;&#xA;" OwnerUserId="11066" LastEditorUserId="13727" LastEditDate="2016-01-27T10:17:07.533" LastActivityDate="2019-09-30T07:28:37.727" Title="Data driven approach to define a churn user" Tags="|machine-learning|predictive-modeling|survival-analysis|churn|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6683" PostTypeId="1" CreationDate="2015-08-04T17:44:35.277" Score="13" ViewCount="17344" Body="&lt;p&gt;I have &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;noreferrer&quot;&gt;plotted the feature importances in random forests with scikit-learn&lt;/a&gt;. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to spot whether a feature is useless or even worse decrease of the random forests performance, based on the plot information? The plot is based on the attribute &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;feature_importances_&lt;/code&gt;&lt;/a&gt; and I use the classifier &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;sklearn.ensemble.RandomForestClassifier&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that there exist &lt;a href=&quot;https://datascience.stackexchange.com/a/6651/843&quot;&gt;other techniques for feature selection&lt;/a&gt;, but in this question I want to focus on how to use feature &lt;code&gt;feature_importances_&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Examples of such feature importance plots:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ySztK.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ySztK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xeXHY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xeXHY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-08-04T17:55:20.787" Title="Feature selection using feature importances in random forests with scikit-learn" Tags="|feature-selection|random-forest|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6684" PostTypeId="2" ParentId="6683" CreationDate="2015-08-04T17:55:20.787" Score="14" Body="&lt;p&gt;You can simply use the &lt;code&gt;feature_importances_&lt;/code&gt; attribute to select the features with the highest importance score. So for example you could use the following function to select the K best features according to importance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def selectKImportance(model, X, k=5):&#xA;     return X[:,model.feature_importances_.argsort()[::-1][:k]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or if you're using a pipeline the following class&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class ImportanceSelect(BaseEstimator, TransformerMixin):&#xA;    def __init__(self, model, n=1):&#xA;         self.model = model&#xA;         self.n = n&#xA;    def fit(self, *args, **kwargs):&#xA;         self.model.fit(*args, **kwargs)&#xA;         return self&#xA;    def transform(self, X):&#xA;         return X[:,self.model.feature_importances_.argsort()[::-1][:self.n]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.datasets import load_iris&#xA;&amp;gt;&amp;gt;&amp;gt; from sklearn.ensemble import RandomForestClassifier&#xA;&amp;gt;&amp;gt;&amp;gt; iris = load_iris()&#xA;&amp;gt;&amp;gt;&amp;gt; X = iris.data&#xA;&amp;gt;&amp;gt;&amp;gt; y = iris.target&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; model = RandomForestClassifier()&#xA;&amp;gt;&amp;gt;&amp;gt; model.fit(X,y)&#xA;RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',&#xA;            max_depth=None, max_features='auto', max_leaf_nodes=None,&#xA;            min_samples_leaf=1, min_samples_split=2,&#xA;            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,&#xA;            oob_score=False, random_state=None, verbose=0,&#xA;            warm_start=False)&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; newX = selectKImportance(model,X,2)&#xA;&amp;gt;&amp;gt;&amp;gt; newX.shape&#xA;(150, 2)&#xA;&amp;gt;&amp;gt;&amp;gt; X.shape&#xA;(150, 4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And clearly if you wanted to selected based on some other criteria than &quot;top k features&quot; then you can just adjust the functions accordingly.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-08-04T17:55:20.787" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6685" PostTypeId="2" ParentId="6680" CreationDate="2015-08-04T17:59:06.823" Score="3" Body="&lt;p&gt;2 solutions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You aggregate each sequence of numbers into a single number, which use as a feature. There exist plenty of aggregation functions, such as some derived from descriptive statistics root-mean-square, kurtosis, skewness, max, min, duration, standard deviation, crest factor, mean, or more specific aggregation such as fourier transforms or wavelet transforms.&lt;/li&gt;&#xA;&lt;li&gt;You use some model that accepts sequences as input. Sequences may be of variable length. Example of such model: &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;recurrent neural networks&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_Bayesian_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Bayesian networks&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-03-06T00:01:14.520" LastActivityDate="2017-03-06T00:01:14.520" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6687" PostTypeId="1" CreationDate="2015-08-04T19:25:34.437" Score="2" ViewCount="75" Body="&lt;p&gt;I have a grey valued image which is calculated as the mean of a series of images. The value of each pixel is therefore associated to a standard error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pixel values and the relative standard error (RSE) vary smoothly in pixel space, so that the pixels of a downscaled image resulting as the mean of the underlying downscaled images would exhibit a lower RSE. Meaning that a high resolution image is associated to high RSEs and a low resolution image to lower RSEs. However, the RSE vary in space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Effectively this behaves like the trade-off of an uncertainty relation: Resolution in space vs resolution in the mean value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For representing this I am looking for a way to divide the image into regions with constant RSEs, which would be larger in parts of the image with high RSEs and smaller in parts with low RSEs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aim would be to obtain irregularly sized superpixels with regular RSEs as opposed to my current image which contains pixels on a regular grid but with irregular RSEs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be equivalent to downscaling the image but not on a regular grid. One approach I tried is agglomerative clustering of the pixels computing new distances in every step, but so far I couldn't find the right distance and objective functions to obtain a more or less regular map of superpixels (generally one big cluster would eat up its neighbours).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of any existing approach for this or if not which would be a good clustering method to apply? Would quadtrees help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance!&lt;/p&gt;&#xA;" OwnerUserId="10463" LastEditorUserId="10463" LastEditDate="2015-08-05T09:29:04.283" LastActivityDate="2015-08-05T09:29:04.283" Title="Spatial clustering of data points on a grid to obtain variable resolution map with constant statistical confidence" Tags="|clustering|hierarchical-data-format|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6688" PostTypeId="2" ParentId="6675" CreationDate="2015-08-04T20:22:00.210" Score="2" Body="&lt;p&gt;A layer in a neural network does not process logic of the kind you present. It cannot &quot;check the input pattern with a simple if else pattern&quot;. You could make a model that processed your example inputs to example outputs as you show, but it would not be a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In machine-learning, neural networks have a formal, numeric structure. There are a few variations, but essentially in your diagram:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Each circle is a &quot;neuron&quot; which may have a numeric output value, also called its &lt;em&gt;activation&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each arrow is a connection between neurons that shows where values are passed between neurons. Each connection has a strength or &lt;em&gt;weight&lt;/em&gt; associated with it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When you have many arrows going into a neuron, you can calculate its new &lt;em&gt;activation&lt;/em&gt; starting with the sum of &lt;em&gt;weight&lt;/em&gt;s times the activation values of neurons that they are connected from. Call this sum $z$. Then the activation of the neuron you are looking at is $a = f(z)$ where f is an &quot;activation function&quot; (also called &quot;transfer function&quot;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above very roughly covers how many neural networks are constructed. As you can see though, there is no room in that for having your if/else statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, neural networks like the one you show (technically a &lt;em&gt;feed forward&lt;/em&gt; network also called a &lt;em&gt;multi-layer perceptron&lt;/em&gt; or MLP) are most often &lt;em&gt;trained by example&lt;/em&gt;. This involves repeatedly giving the network inputs where you know the output you want. To start with the network will give completely incorrect outputs, and you then adjust the weights to try and make the result closer. This is repeated very many times until the network becomes the best model it can for your purpose. A very common algorithm used to train a neural network in this way is &lt;a href=&quot;https://en.wikipedia.org/wiki/Backpropagation&quot; rel=&quot;nofollow&quot;&gt;backpropagation and gradient descent&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your problem as described, a neural network is &lt;em&gt;not&lt;/em&gt; a good model, and would not be used in the real world. That is because you already know the full logic of a simpler model that covers all possible inputs. At best with a neural network you will get an approximation to that already-perfect model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if this is for the purposes of understanding neural networks, your approach would be roughly this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Generate some training data with example inputs and outputs. Most example outputs will be the 23,23 default values, but when inputs are the special values 6,7,8 then the training outputs should be 3 and 4.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Build an initial network. Normally you would use an appropriate library - it is important you learn how the library works and understand any limitations. E.g. many activation functions have a maximum output of 1, so cannot learn the values 3,4, or 23.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train the network to fit the example data. There are very many options there, depending on the library you are using. Usually it is just a matter of selecting parameters for the learning that help it progress smoothly. You may have to run training for a large number of repeats over the whole dataset (called epochs)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Test that your network works as you expect. Even if you have trained it well, the response of the network is never as precise as your problem statement. If you gave it inputs of 5.8,7.2,8.1 (i.e. close to 6,7,8) then it would likely output something like 7.2,8.5 and not 23,23 .  . . it will interpolate between values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-08-05T06:39:48.220" LastActivityDate="2015-08-05T06:39:48.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6689" PostTypeId="2" ParentId="6681" CreationDate="2015-08-04T21:09:48.260" Score="2" Body="&lt;p&gt;You should first define what your churn event is which you have started. Is it global or individual? Is it has not gambled for 3 months or has changed his pattern? Global is better to model. You can use survival models for this. &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2015-08-04T21:09:48.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6690" PostTypeId="2" ParentId="6660" CreationDate="2015-08-04T21:39:31.060" Score="1" Body="&lt;p&gt;Maybe the best answer for Your question is &lt;strong&gt;recommenderlab&lt;/strong&gt; package for R available from CRAN. Here's the link to &lt;a href=&quot;https://cran.r-project.org/web/packages/recommenderlab/vignettes/recommenderlab.pdf&quot; rel=&quot;nofollow&quot;&gt;vignette&lt;/a&gt;. It has CF algorithms implemented and Jaccard coef. for 0/1 user feedback case.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only similarity(distance) computation in base stats R package there is &lt;em&gt;dist&lt;/em&gt; function. But it's limited to few distance measures. I know that when I search for distance measures in R package called &lt;em&gt;vegan&lt;/em&gt; claimed to have many methods implemented. But I have never used it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO most basic recommender systems algorithms are really simple. It worth spending few minutes and implement it to get a better understanding and intuition how it works. &#xA;Nevertheless, below my quick and dirty user-based kNN implementation for Your case. The code can be rewritten in better way (Adv.R is really good resource) - but this is only example for learning purpose and I do not take responsibilities for any damages :-)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(reshape2)&#xA;&#xA;ratings &amp;lt;- read.csv(con &amp;lt;- textConnection('&#xA;User,Item,Rating&#xA;Anna,Apple,2&#xA;Bob,Apple,4&#xA;Carlos,Apple,1&#xA;Ethan,Apple,1&#xA;Gene,Apple,2&#xA;Anna,Banana,4&#xA;Dana,Banana,3&#xA;Ethan,Banana,5&#xA;Frank,Banana,5&#xA;Bob,Carrot,1&#xA;Ethan,Carrot,2&#xA;Gene,Carrot,4&#xA;'), header = T)&#xA;&#xA;R &amp;lt;- ratings %&amp;gt;% dcast(Item ~ User)&#xA;ProductNames &amp;lt;- R[,1]&#xA;R &amp;lt;- R[,-1] # only ratings&#xA;&#xA;# tanimoto similarity (~Jaccard coefficiet)&#xA;tanimoto &amp;lt;- function(v1,v2) length(intersect(v1,v2))/length(union(v1,v2))&#xA;&#xA;# return top-k similar users&#xA;getKNN &amp;lt;- function(R, i, k, sim = tanimoto) {&#xA;  similarity &amp;lt;- array(0,length(R))&#xA;  for(j in 1:length(R)) {&#xA;    if(i!=j) similarity[j] = sim(which(!is.na(R[,i])),which(!is.na(R[,j])))&#xA;  }&#xA;  idx &amp;lt;- order(similarity, decreasing = T)[1:k]&#xA;  data.frame(idx = idx, similarity = similarity[idx])&#xA;}&#xA;&#xA;kNNRecommender &amp;lt;- function(R, k) {&#xA;  reco &amp;lt;- data.frame()&#xA;  for(u in 1:length(R)) {&#xA;    userItems &amp;lt;- which(!is.na(R[,u])) #items rated by user&#xA;    nn &amp;lt;- getKNN(R,u, k) # get user neighbours&#xA;&#xA;    # rating prediction - wiegted by similarity&#xA;    r &amp;lt;- array(0, c(dim(R)[1], k))&#xA;    for(i in 1:k) {&#xA;      r[,i] = R[,nn[i,]$idx] * nn[i,]$similarity&#xA;    }&#xA;    r[is.na(r)] &amp;lt;- 0&#xA;    userReco &amp;lt;- rowSums(r) / sum(nn$similarity)&#xA;    userRecoIdx &amp;lt;- order(userReco, decreasing = T)&#xA;&#xA;    # remove items already rated by user&#xA;    userRecoIdx &amp;lt;- userRecoIdx[-userItems]&#xA;&#xA;    # add to recommendations result&#xA;    reco &amp;lt;- rbind(reco, data.frame(&#xA;      User=u,&#xA;      Item=(if(length(userRecoIdx)==0) NA else userRecoIdx),&#xA;      Prediction=(if(length(userRecoIdx)==0) NA else userReco[userRecoIdx])&#xA;    ))&#xA;  }&#xA;  reco&#xA;}&#xA;&#xA;kNNRecommender(R, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9314" LastActivityDate="2015-08-04T21:39:31.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6691" PostTypeId="1" CreationDate="2015-08-04T22:15:52.833" Score="6" ViewCount="9102" Body="&lt;p&gt;I barely know about Data Analysis tools and techniques, so bare with me if I'm asking something too trivial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a Sentiment Analysis tool to process comments in Spanish. I do know some options for Sentiment analysis but those all work for English. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Is there a model/tool that already works with Spanish?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm language agnostic so it does not matter if it's a Java, Python or even Go code.&lt;/p&gt;&#xA;" OwnerUserId="11082" LastEditorUserId="31513" LastEditDate="2017-05-10T04:00:25.043" LastActivityDate="2021-08-13T20:16:07.070" Title="Sentiment Analysis model for Spanish" Tags="|machine-learning|nlp|social-network-analysis|sentiment-analysis|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6693" PostTypeId="1" CreationDate="2015-08-05T05:18:55.957" Score="5" ViewCount="301" Body="&lt;p&gt;Are there any limitations on the number of items to use in a transaction for applying apriori algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with just 20 records, but the number of items extend upto 900. I am getting memory issue when I apply the apriori algorithm on the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="3240" LastEditDate="2016-01-02T22:44:38.273" LastActivityDate="2016-02-02T10:02:30.633" Title="Limitations on the number of items to use in apriori algorithm?" Tags="|machine-learning|data-mining|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6694" PostTypeId="1" AcceptedAnswerId="10408" CreationDate="2015-08-05T05:58:44.543" Score="11" ViewCount="3643" Body="&lt;p&gt;I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can you suggest me some text books which would help me build a clear conception of Reinforcement Learning? &lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:36.497" LastActivityDate="2021-02-13T08:11:34.423" Title="Books on Reinforcement Learning" Tags="|machine-learning|books|reinforcement-learning|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6696" PostTypeId="2" ParentId="6694" CreationDate="2015-08-05T06:48:49.087" Score="7" Body="&lt;p&gt;There is a free online course on Reinforcement Learning by Udacity. Check : &lt;a href=&quot;https://www.udacity.com/course/machine-learning-reinforcement-learning--ud820&quot;&gt;Machine Learning: Reinforcement Learning&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-08-05T06:48:49.087" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6698" PostTypeId="2" ParentId="6086" CreationDate="2015-08-05T07:01:10.340" Score="3" Body="&lt;p&gt;As correctly pointed out above MinHash and SimHash both belong to Locality Sensitive Hashing. &#xA;Reference: &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main difference between the two is the way collision is handled,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;SimHash, uses cosine similarity&lt;/li&gt;&#xA;&lt;li&gt;MinHash, uses Jaccard Index.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Answers to your questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No. They uses different collision handling techniques to validate similarity. Also there is a variant on single Hash Function for Min Hash but it works differently. For more details, check the following reference out, &lt;a href=&quot;https://en.wikipedia.org/wiki/MinHash&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/MinHash&lt;/a&gt; (Variant with a single hash function) &lt;/li&gt;&#xA;&lt;li&gt;Yes, &lt;a href=&quot;https://github.com/chrisjmccormick/MinHash/blob/master/runMinHashExample.py&quot; rel=&quot;nofollow&quot;&gt;https://github.com/chrisjmccormick/MinHash/blob/master/runMinHashExample.py&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;I think the complexity can be reduced to $O(n \log n)$ with modified form of binary search while clustering.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5179" LastEditorUserId="816" LastEditDate="2015-08-05T09:58:28.050" LastActivityDate="2015-08-05T09:58:28.050" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6699" PostTypeId="2" ParentId="6691" CreationDate="2015-08-05T07:17:04.600" Score="1" Body="&lt;p&gt;Check the following links for Spanish sentiment analysis related links:&lt;/p&gt;&#xA;&lt;p&gt;Working Model :&lt;a href=&quot;http://dtminredis.housing.salle.url.edu:8080/EmoLib/es/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dtminredis.housing.salle.url.edu:8080/EmoLib/es/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Data: &lt;a href=&quot;http://www.daedalus.es/TASS2015/tass2015.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.daedalus.es/TASS2015/tass2015.php&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;API: &lt;a href=&quot;https://www.mashape.com/molinodeideas/sentiment-analysis-spanish&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.mashape.com/molinodeideas/sentiment-analysis-spanish&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="29169" LastEditDate="2021-08-13T20:15:28.557" LastActivityDate="2021-08-13T20:15:28.557" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6700" PostTypeId="1" AcceptedAnswerId="6707" CreationDate="2015-08-05T10:27:51.370" Score="3" ViewCount="180" Body="&lt;p&gt;Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update its value depending on its present value and the present value of the taking action from that state. Now&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How long do I keep updating value of each cell? Shall I keep updating unless the change in the previous and the present value function is the least? I am not able to understand how to implement the stopping mechanism in the grid-world scenario(discount not considered)&lt;br&gt;&lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Is the value function the values of all the grids in the grid world?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8820" LastEditDate="2016-01-18T14:37:01.720" LastActivityDate="2016-01-18T14:37:01.720" Title="When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld" Tags="|machine-learning|markov-process|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6701" PostTypeId="1" AcceptedAnswerId="6703" CreationDate="2015-08-05T10:53:17.293" Score="0" ViewCount="55" Body="&lt;p&gt;I am currently learning R and I have to solve an Issue were I have to extract values from a data set which are from a specific month and from this values I should calculate the mean Temp. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did it like that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data[data$X..Month.. == 6,]   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;mean(data$X..Temp.., na.rm=TRUE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It gave me the mean value but without taking my first statement into consideration. &#xA;What do I need to do that both statements are taken into consideration?&lt;/p&gt;&#xA;" OwnerUserId="11094" LastActivityDate="2015-08-05T11:28:25.060" Title="How to keep a subsetted value for calculating mean" Tags="|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6702" PostTypeId="2" ParentId="6701" CreationDate="2015-08-05T11:24:34.843" Score="1" Body="&lt;p&gt;You can answer this using tapply, which will give you the mean for all your subsets, e.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dummy data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = data.frame(mon=rep(month.abb, times=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;calculate mean for all months&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tapply(df$temp, df$mon, mean, na.rm=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8021" LastActivityDate="2015-08-05T11:24:34.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6703" PostTypeId="2" ParentId="6701" CreationDate="2015-08-05T11:28:25.060" Score="0" Body="&lt;p&gt;To just get the mean for month 6:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean(df$temp[df$mon==&quot;Jun&quot;], na.rm=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You were nearly there but didn't assign your subset to a value, if you had:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = data[data$X..Month.. == 6,]&#xA;mean(x$X..Temp.., na.rm=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that should work.&lt;/p&gt;&#xA;" OwnerUserId="8021" LastActivityDate="2015-08-05T11:28:25.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6704" PostTypeId="2" ParentId="6679" CreationDate="2015-08-05T14:35:57.897" Score="0" Body="&lt;p&gt;Take a look at the &lt;code&gt;TSclust&lt;/code&gt; package. Here how you would apply it to your sample data. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(TSclust)&#xA;&#xA;#read in the data&#xA;model_a &amp;lt;- read.csv(&quot;~/Desktop/Model A.csv&quot;, header = TRUE, stringsAsFactors = FALSE)&#xA;model_b &amp;lt;- read.csv(&quot;~/Desktop/Model B.csv&quot;, header = TRUE, stringsAsFactors = FALSE)&#xA;&#xA;#data must be in rows rather than columns&#xA;model_a &amp;lt;- as.data.frame(t(model_a))&#xA;&#xA;model_b &amp;lt;- as.data.frame(t(model_b))&#xA;&#xA;#calculate dissimlarities between metabolites in models 1 and 2&#xA;met1_DWT.diss &amp;lt;- as.numeric(diss.DWT(rbind(model_a['Met1', ], model_b['Met1', ])))&#xA;met1_DWT.diss&#xA;[1] 90.80332&#xA;&#xA;met2_DWT.diss &amp;lt;- as.numeric(diss.DWT(rbind(model_a['Met2', ], model_b['Met2', ])))&#xA;met2_DWT.diss&#xA;[1] 1.499241&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11021" LastActivityDate="2015-08-05T14:35:57.897" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6705" PostTypeId="2" ParentId="6694" CreationDate="2015-08-05T16:22:23.003" Score="4" Body="&lt;p&gt;I really enjoyed &lt;a href=&quot;https://books.google.cz/books?id=CAFR6IBF4xYC&quot; rel=&quot;nofollow&quot;&gt;Reinforcement Leraning: An introduction&lt;/a&gt; by Richard Sutton. It provides a very nice unifying view on RL, although it does not mention the newest approaches (it's from 1998).&lt;/p&gt;&#xA;" OwnerUserId="11077" LastActivityDate="2015-08-05T16:22:23.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6707" PostTypeId="2" ParentId="6700" CreationDate="2015-08-06T01:25:40.610" Score="3" Body="&lt;p&gt;1- You should set a threshold (a hyper-param) that will allow you to quit the loop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Let V the values for all state s and V' the new values after value iteration.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if $\sum_s|V(s) - V’(s)| \le threshold$, quit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - V is a function for every cell in the grid yes because you need to update every cell. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="11113" LastEditorUserId="836" LastEditDate="2015-08-06T21:19:08.257" LastActivityDate="2015-08-06T21:19:08.257" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="6709" PostTypeId="1" AcceptedAnswerId="6711" CreationDate="2015-08-06T06:03:20.613" Score="0" ViewCount="737" Body="&lt;p&gt;What I understood for value iteration while coding is that we need to have a policy fixed. According to that policy the value function of each state will be calculated. Right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in policy iteration the policy will change from time to time. Am I right?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8820" LastEditDate="2016-01-18T14:37:14.050" LastActivityDate="2017-03-31T17:14:15.470" Title="Confusion in Policy Iteration and Value iteration in Reinforcement learning in Dynamic Programming" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6710" PostTypeId="2" ParentId="6674" CreationDate="2015-08-06T09:09:00.657" Score="3" Body="&lt;p&gt;If you need complexity of this calculation in big O notation - it is: $$O(n^3)$$&#xA;Why? Because matrix inverse needs $$O(n^3)$$ operations, and it is biggest complexity here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiplication matrix by its transpose is $$O(n^2p)$$ (Because for computing every value in the resulting matrix of size NxN you have to compute p multiplications).&#xA;Matrix transpose is $$O(np)$$ But you can ignore any complexities lesser than $$O(n^3)$$ (like matrix sum, transpose, or multiplying by constant) because their order of growth much lesser than order of growth of inverse operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra&quot; rel=&quot;nofollow&quot;&gt;Computational_complexity_of_mathematical_operations&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10989" LastEditorUserId="10989" LastEditDate="2015-08-06T10:16:49.560" LastActivityDate="2015-08-06T10:16:49.560" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6711" PostTypeId="2" ParentId="6709" CreationDate="2015-08-06T09:25:34.667" Score="0" Body="&lt;p&gt;In policy iteration, you define a starting policy and iterate towards the best one, by estimating the state value associated with the policy, and making changes to action choices. So the policy is explicitly stored and tracked on each major step. After each iteration of the policy, you re-calculate the value function for that policy to within a certain precision. That means you also work with value functions that measure actual policies. If you halted the iteration just after the value estimate, you would have a non-optimal policy and the value function for that policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In value iteration, you implicitly solve for the state values under an ideal policy. There is no need to define an actual policy during the iterations, you can derive it at the end from the values that you calculate. You could if you wish, after any iteration, use the state values to determine what &quot;current&quot; policy is predicted. The values will likely not approximate the value function for that predicted policy, although towards the end they will probably be close.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-31T17:14:15.470" LastActivityDate="2017-03-31T17:14:15.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6714" PostTypeId="2" ParentId="6674" CreationDate="2015-08-06T16:32:40.243" Score="1" Body="&lt;p&gt;@Olologin is correct for very basic direct solves with no optimization, but there are optimizations that exist, so I wanted to add a few points...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A great many mathematicians spend their entire careers on speeding up and optimizing this problem.  So, the solution depends a lot on the composition of the matrix (&lt;code&gt;sparsity&lt;/code&gt;), whether the problem is &lt;code&gt;convex&lt;/code&gt;,  whether it is being solved &lt;code&gt;iteratively&lt;/code&gt; or &lt;code&gt;directly&lt;/code&gt;, and what algorithm is being used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For lots of data science problems $O(n^2)$ and better solves are readily achieved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some common libraries include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Basic Linear Algebra Subroutines (&lt;a href=&quot;http://www.netlib.org/blas/&quot; rel=&quot;nofollow&quot;&gt;BLAS&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.netlib.org/linpack/&quot; rel=&quot;nofollow&quot;&gt;LINPACK&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.netlib.org/lapack/&quot; rel=&quot;nofollow&quot;&gt;LAPACK&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are some surveys of the subject:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.nthu.edu.tw/~cchen/CS6331/ch2.pdf&quot; rel=&quot;nofollow&quot;&gt;General survey&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://web.stanford.edu/class/cme324/saad.pdf&quot; rel=&quot;nofollow&quot;&gt;Survey of Iterative Methods&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/LU_decomposition&quot; rel=&quot;nofollow&quot;&gt;LU decomposition&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Conjugate_gradient_method&quot; rel=&quot;nofollow&quot;&gt;Conjugate Gradient&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/ebooks/html/csa/node37.html&quot; rel=&quot;nofollow&quot;&gt;Direct Methods for Solving Linear Systems&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-07T15:15:50.507" LastActivityDate="2015-08-07T15:15:50.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6715" PostTypeId="1" AcceptedAnswerId="6718" CreationDate="2015-08-06T20:58:57.380" Score="40" ViewCount="65412" Body="&lt;p&gt;Is it necessary to standardize your data before cluster?  In the example from &lt;code&gt;scikit learn&lt;/code&gt; about DBSCAN, &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#example-cluster-plot-dbscan-py&quot;&gt;here&lt;/a&gt; they do this in the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = StandardScaler().fit_transform(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I do not understand why it is necessary.  After all, clustering does not assume any particular distribution of data - it is an unsupervised learning method so its objective is to explore the data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would it be necessary to transform the data?  &lt;/p&gt;&#xA;" OwnerUserId="10512" LastActivityDate="2021-04-17T05:28:30.623" Title="Is it necessary to standardize your data before clustering?" Tags="|python|clustering|anomaly-detection|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6716" PostTypeId="2" ParentId="6715" CreationDate="2015-08-06T21:38:01.087" Score="9" Body="&lt;p&gt;Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- &lt;code&gt;euclidean&lt;/code&gt;, otherwise known as the L2 norm. If one of your features has a range of values much larger than the others, clustering will be completely dominated by that one feature. To illustrate this look at the simple example below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import numpy as np&#xA;&amp;gt;&amp;gt;&amp;gt; from sklearn.preprocessing import StandardScaler&#xA;&amp;gt;&amp;gt;&amp;gt; from sklearn.metrics.pairwise import euclidean_distances&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; X = np.array([[1,2,100],[4,3,50],[1,1,75]])&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; print X&#xA;[[  1   2 100]&#xA; [  4   3  50]&#xA; [  1   1  75]]&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; print euclidean_distances(X)&#xA;[[  0.          50.0999002   25.01999201]&#xA; [ 50.0999002    0.          25.25866188]&#xA; [ 25.01999201  25.25866188   0.        ]]&#xA;&amp;gt;&amp;gt;&amp;gt; print euclidean_distances(StandardScaler().fit_transform(X))&#xA;[[ 0.          3.46410162  1.73205081]&#xA; [ 3.46410162  0.          3.46410162]&#xA; [ 1.73205081  3.46410162  0.        ]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this you should see that the euclidean distances between the non-standardized versions are dominated by the third column because its range of values is much larger than the other two. However when the data is standardized this no longer becomes an issue and weights each feature as being equal when calculating the distance between each data point.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-08-06T21:38:01.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6717" PostTypeId="2" ParentId="6715" CreationDate="2015-08-07T00:20:03.757" Score="1" Body="&lt;p&gt;It isn't strictly necessary to standardise, whether it is required or not may depend on the distance metric you choose. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance if you opt for the Mahalanobis distance then separation will be based upon the number of standard deviations separating points and not the absolute distance between them, as such it is a scale invariant metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As with many things in machine learning there is no hard and fast answer and the only way to know for sure is to apply a variety of techniques see which gives the most appropriate results for your data.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-08-07T00:20:03.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6718" PostTypeId="2" ParentId="6715" CreationDate="2015-08-07T00:22:14.480" Score="34" Body="&lt;p&gt;Normalization is not always required, but it rarely hurts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/21226/12359&quot;&gt;K-means&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;K-means clustering is &quot;isotropic&quot; in all directions of space and&#xA;  therefore tends to produce more or less round (rather than elongated)&#xA;  clusters. In this situation leaving variances unequal is equivalent to&#xA;  putting more weight on variables with smaller variance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Example in Matlab:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = [randn(100,2)+ones(100,2);...&#xA;     randn(100,2)-ones(100,2)];&#xA;&#xA;% Introduce denormalization&#xA;% X(:, 2) = X(:, 2) * 1000 + 500;&#xA;&#xA;opts = statset('Display','final');&#xA;&#xA;[idx,ctrs] = kmeans(X,2,...&#xA;                    'Distance','city',...&#xA;                    'Replicates',5,...&#xA;                    'Options',opts);&#xA;&#xA;plot(X(idx==1,1),X(idx==1,2),'r.','MarkerSize',12)&#xA;hold on&#xA;plot(X(idx==2,1),X(idx==2,2),'b.','MarkerSize',12)&#xA;plot(ctrs(:,1),ctrs(:,2),'kx',...&#xA;     'MarkerSize',12,'LineWidth',2)&#xA;plot(ctrs(:,1),ctrs(:,2),'ko',...&#xA;     'MarkerSize',12,'LineWidth',2)&#xA;legend('Cluster 1','Cluster 2','Centroids',...&#xA;       'Location','NW')&#xA;title('K-means with normalization')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/N2unM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/e5G2M.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(FYI: &lt;a href=&quot;https://www.quora.com/Machine-Learning/How-can-I-detect-if-my-dataset-is-clustered-or-unclustered-i-e-forming-one-single-cluster/answer/Franck-Dernoncourt&quot; rel=&quot;noreferrer&quot;&gt;How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.medwelljournals.com/fulltext/?doi=ijscomp.2009.168.172&quot; rel=&quot;noreferrer&quot;&gt;Distributed clustering&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The comparative analysis shows that the distributed clustering results&#xA;  depend on the type of normalization procedure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/4674770/395857&quot;&gt;Artificial neural network (inputs)&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If the input variables are combined linearly, as in an MLP, then it is&#xA;  rarely strictly necessary to standardize the inputs, at least in&#xA;  theory. The reason is that any rescaling of an input vector can be&#xA;  effectively undone by changing the corresponding weights and biases,&#xA;  leaving you with the exact same outputs as you had before. However,&#xA;  there are a variety of practical reasons why standardizing the inputs&#xA;  can make training faster and reduce the chances of getting stuck in&#xA;  local optima. Also, weight decay and Bayesian estimation can be done&#xA;  more conveniently with standardized inputs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html&quot; rel=&quot;noreferrer&quot;&gt;Artificial neural network (inputs/outputs)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should you do any of these things to your data? The answer is, it&#xA;  depends.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Standardizing either input or target variables tends to make the training&#xA;  process better behaved by improving the numerical condition (see &#xA;  &lt;a href=&quot;ftp://ftp.sas.com/pub/neural/illcond/illcond.html&quot; rel=&quot;noreferrer&quot;&gt;ftp://ftp.sas.com/pub/neural/illcond/illcond.html&lt;/a&gt;) of the optimization&#xA;  problem and ensuring that various default values involved in&#xA;  initialization and termination are appropriate. Standardizing targets&#xA;  can also affect the objective function. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Standardization of cases should be approached with caution because it&#xA;  discards information. If that information is irrelevant, then&#xA;  standardizing cases can be quite helpful. If that information is&#xA;  important, then standardizing cases can be disastrous.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Interestingly, changing the measurement units may even lead one to see a very different clustering structure: &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471735787&quot; rel=&quot;noreferrer&quot;&gt;Kaufman, Leonard, and Peter J. Rousseeuw.. &quot;Finding groups in data: An introduction to cluster analysis.&quot; (2005).&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In some applications, changing the measurement units may even lead one&#xA;  to see a very different clustering structure. For example, the age (in&#xA;  years) and height (in centimeters) of four imaginary people are given&#xA;  in Table 3 and plotted in Figure 3. It appears that {A, B ) and { C,&#xA;  0) are two well-separated clusters. On the other hand, when height is&#xA;  expressed in feet one obtains Table 4 and Figure 4, where the obvious&#xA;  clusters are now {A, C} and { B, D}. This partition is completely&#xA;  different from the first because each subject has received another&#xA;  companion. (Figure 4 would have been flattened even more if age had&#xA;  been measured in days.)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;To avoid this dependence on the choice of measurement units, one has&#xA;  the option of  standardizing the data. This converts the original&#xA;  measurements to unitless variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ppu5Y.png&quot; alt=&quot;enter image description here&quot;&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/zW6rI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0471735787&quot; rel=&quot;noreferrer&quot;&gt;Kaufman et al.&lt;/a&gt; continues with some interesting considerations (page 11):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;From a philosophical point of view, standardization does not really&#xA;  solve the problem. Indeed, the choice of measurement units gives rise&#xA;  to relative weights of the variables. Expressing a variable in smaller&#xA;  units will lead to a larger range for that variable, which will then&#xA;  have a large effect on the resulting structure. On the other hand, by&#xA;  standardizing one attempts to give all variables an equal weight, in&#xA;  the hope of achieving objectivity. As such, it may be used by a&#xA;  practitioner who possesses no prior knowledge. However, it may well be&#xA;  that some variables are intrinsically more important than others in a&#xA;  particular application, and then the assignment of weights should be&#xA;  based on subject-matter knowledge (see, e.g., Abrahamowicz, 1985). On&#xA;  the other hand, there have been attempts to devise clustering&#xA;  techniques that are independent of the scale of the variables&#xA;  (Friedman and Rubin, 1967). The proposal of Hardy and Rasson (1982) is&#xA;  to search for a partition that minimizes the total volume of the&#xA;  convex hulls of the clusters. In principle such a method is invariant&#xA;  with respect to linear transformations of the data, but unfortunately&#xA;  no algorithm exists for its implementation (except for an&#xA;  approximation that is restricted to two dimensions). Therefore, the&#xA;  dilemma of standardization appears unavoidable at present and the&#xA;  programs described in this book leave the choice up to the user.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-08-07T00:22:14.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6719" PostTypeId="1" AcceptedAnswerId="6720" CreationDate="2015-08-07T04:31:06.123" Score="4" ViewCount="138" Body="&lt;p&gt;Regarding Value Iteration of Dynamic Programming(reinforcement learning) in grid world, the value updation of each state is given by: &lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Y3Sar.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y3Sar.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;Now Suppose i am in say box (3,2). I can go to (4,2)(up) (3,3)(right) and (1,3)(left) and none of these are my final state so i get a reward of -0.1 for going in each of the states. The present value of all states are 0. The probability of going north is 0.8, and going left/right is 0.1 each. So since going left/right gives me more reward(as reward*probability will be negative) i go left or right. Is this the mechanism. Am I correct? But In the formula there is a summation term given. So I basically cannot understand this formula. Can anyone explain me with an example?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:57.447" LastActivityDate="2016-03-05T20:28:23.367" Title="Value Updation Dynamic Programming Reinforcement learning" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6720" PostTypeId="2" ParentId="6719" CreationDate="2015-08-07T05:15:00.393" Score="4" Body="&lt;p&gt;The probabilities you describe refer only to the go-north action. It means that if you want to go north, you have 80% chance of actually going north and 20% of going left or right, making the problem more difficult (non-deterministic). This rule applies to every direction. Also, the formula does not tell which action to chose, just how to update the values. In order to select an action, assuming a greedy-policy, you'd select the one with the highest expected value ($V(s')$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula says to sum the values for all possible outcomes from the best action. So, supposing go-north is indeed the best action, you have: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + 0) = -.1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But let us suppose that you still don't know which is the best action and want to select one greedily. Then you must compute the sum for each possible action (north, south, east, west). Your example has all values set to 0 and the same reward and so is not very interesting. Let's say you have a +1 reward to east (-0.1 for the remaining directions) and that south already has V(s) = 0.5 (0 for the remaining states). Then you compute the value for each action (let $\gamma = 1$, since it is a user-adjusted parameter):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;North: $.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (1 + 0) = -.08 - .01 + .1 = .01$&lt;/li&gt;&#xA;&lt;li&gt;South: $.8 * (-.1 + .5) + .1 * (-.1 + 0) + .1 * (1 + 0) = 0.32 - .01 + .1 = .41$&lt;/li&gt;&#xA;&lt;li&gt;East: $.8 * (1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + .5) = .8 - .01 + .04 = .83$&lt;/li&gt;&#xA;&lt;li&gt;West: $.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + .5) = -.08 - .01 + .04 = -.05$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So you would update your policy to go &lt;strong&gt;East&lt;/strong&gt; from the current state, and update the current state value to &lt;strong&gt;0.83&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9814" LastEditorUserId="9814" LastEditDate="2016-03-05T20:28:23.367" LastActivityDate="2016-03-05T20:28:23.367" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6721" PostTypeId="1" AcceptedAnswerId="6725" CreationDate="2015-08-07T10:43:50.747" Score="7" ViewCount="13683" Body="&lt;p&gt;I want to use some Decision Tree learning, such as the Random Forest classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to have consistent results?&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2015-08-08T04:33:31.700" Title="How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning" Tags="|data-mining|random-forest|data|decision-trees|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6722" PostTypeId="2" ParentId="6715" CreationDate="2015-08-07T11:02:11.387" Score="1" Body="&lt;p&gt;I found in some cases useful to define a &quot;business evaluation&quot; function, defining the &quot;importance&quot; of the dimensions used for clustering. E.g. for greengrocer clustering the customers, if apples are twice as expensive as oranges, the number of apples will be doubled.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-08-07T11:02:11.387" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6724" PostTypeId="2" ParentId="6576" CreationDate="2015-08-07T21:37:39.207" Score="4" Body="&lt;p&gt;If you want to be a practical person with true knowledge, start with math(calculus, probability + stat, linear algebra). On every step try to implement everything with programing, python is nice for this. When u get good ground, play with real data and solve for problems&lt;/p&gt;&#xA;&lt;p&gt;Courses.&#xA;Linear algebra - edx Laff or coding the matrix&#xA;Stat - edx stat 2x Barkley&#xA;Calculus - read...its simple&lt;/p&gt;&#xA;" OwnerUserId="11141" LastEditorUserId="11141" LastEditDate="2022-08-02T11:05:59.237" LastActivityDate="2022-08-02T11:05:59.237" CommentCount="0" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 4.0" />
  <row Id="6725" PostTypeId="2" ParentId="6721" CreationDate="2015-08-07T22:13:30.820" Score="13" Body="&lt;p&gt;One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is easiest or whatever is best for visualization, as long as it doesn't change the relative order of values within each data dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Categorical inputs, which have no sensible order, are a special case. If your random forest implementation doesn't have a built-in way to deal with categorical input, you should probably use a 1-hot encoding:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If a categorical value has $n$ categories, you encode the value using $n$ dimensions, one corresponding to each category. &lt;/li&gt;&#xA;&lt;li&gt;For each data point, if it is in category $k$, the corresponding $k$th dimension is set to 1, while the rest are set to 0.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This 1-hot encoding allows decision trees to perform category equality tests in one split since inequality splits on non-ordinal data doesn't make much sense.&lt;/p&gt;&#xA;" OwnerUserId="11142" LastEditorUserId="11142" LastEditDate="2015-08-08T04:33:31.700" LastActivityDate="2015-08-08T04:33:31.700" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6726" PostTypeId="1" CreationDate="2015-08-08T08:16:15.160" Score="1" ViewCount="2709" Body="&lt;p&gt;I am teaching myself some data science and have started a Kaggle project. &#xA;I have fitted a random forest classification model (using sci-kit learn) with a few millions rows of data. There are two possible outcomes for each row (0 or 1). When I run it against the test data, I get 0 for every row. This is a practical impossibility, but I am at a loss as to how to diagnose my model and how to move forward. &#xA;Is this an example of extreme overfitting ? Is it more likely a problem with my training data (or test data) that i am not seeing ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I simply increase the # of estimators? Is it possible that I have misformatted the input files in a subtle way that doesn't cause an error ? I am at a loss as to how to move forward.&lt;/p&gt;&#xA;" OwnerUserId="11146" LastEditorUserId="11146" LastEditDate="2015-08-08T08:26:21.600" LastActivityDate="2016-01-15T21:51:27.033" Title="Random forest model gives same result for all test data, Next step?" Tags="|python|random-forest|scikit-learn|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6727" PostTypeId="1" AcceptedAnswerId="6747" CreationDate="2015-08-08T11:14:35.630" Score="3" ViewCount="6004" Body="&lt;p&gt;I have a CSV file with around 1 Million rows. &#xA;Let say its have details like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name      |   Age   | Salary &#xA;name 1      52       10000&#xA;name 2      55       10043 &#xA;name 3      50       100054&#xA;name 2      55       10023&#xA;name 1      52       100322...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and soon .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but i need to merge the redundant details .&#xA;and need a output like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name      |   Age   | Salary &#xA;name 1      52       110322*&#xA;name 2      55       20066 *&#xA;name 3      50       100054 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you might notice that the repeating Name 1 and Name 2 details are merged and the Salary values are added .So i'm looking for a way to apply this change to my original data set. so i need a python script to fix my problem . &lt;/p&gt;&#xA;" OwnerUserId="9035" LastActivityDate="2015-08-14T17:21:38.663" Title="Merging repeating data cells in csv" Tags="|python|csv|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6729" PostTypeId="2" ParentId="6727" CreationDate="2015-08-08T13:22:01.227" Score="0" Body="&lt;p&gt;My solution may not be the best, but I would opt for this one since it is the simplest.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create an empty dictionary&lt;/li&gt;&#xA;&lt;li&gt;Iterate on the CSV. If the name is already in the dictionary, sum up the salaries. If not, then create a new key with the salary.&lt;/li&gt;&#xA;&lt;li&gt;After that, iterate again on the dictionary to write a new CSV with the new values.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are not familiar with Python code, just ask it and I will write it for you :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit with a code sample:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import csv&#xA;&#xA;## open CSV file and rea it&#xA;myfile  = open('test.csv', &quot;rb&quot;)&#xA;reader = csv.reader(myfile)&#xA;&#xA;## create an empty dictionary&#xA;mydictionary = {}&#xA;&#xA;rownum = 0&#xA;&#xA;for row in reader:&#xA;    ## check if it is the header&#xA;    if rownum == 0:&#xA;        pass&#xA;    else:&#xA;        ## split the line of CSV in elements..Use the name for the key in dictionary and the other two in a list&#xA;        line = row.split(&quot;,&quot;)&#xA;        key = line[0]&#xA;        age = line[1]&#xA;        salary = line[2]&#xA;&#xA;        if key in mydictionary:&#xA;            mydictionary[key][1] += salary&#xA;        else:&#xA;            mydictionary[key] = [age,salary]&#xA;&#xA;    rownum += 1&#xA;&#xA;ifile.close()&#xA;&#xA;## create a new list of lists with the data from the dictionary&#xA;newcsvfile = [&quot;name&quot;,&quot;age&quot;,&quot;salary&quot;]&#xA;&#xA;for i in mydictionary:&#xA;    newcsvfile.append(i,mydictionary[i][0],mydictionary[i][1])&#xA;&#xA;## write the new list of lists in a new CSV file&#xA;with open(&quot;output.csv&quot;, &quot;wb&quot;) as f:&#xA;    writer = csv.writer(f)&#xA;    writer.writerows(newcsvfile)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2015-08-08T14:24:51.130" LastActivityDate="2015-08-08T14:24:51.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6732" PostTypeId="2" ParentId="6726" CreationDate="2015-08-09T03:06:58.420" Score="2" Body="&lt;p&gt;Some of the &lt;strong&gt;possibilities&lt;/strong&gt; include the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The training data has class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Solution:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Train the model using CV = 5 or 10;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do a log transformation to make the target distribution more normal in nature;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check if you can add a weight variable that can fix the distribution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;2) Testing data is a small sub sample. Since it's Kaggle, I believe the testing data will be standard in nature, so the problem might be in the process that is generating the submission script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a beginner, you should use any benchmark script available in the forum to start with, which shouldn't have such issues.&lt;/p&gt;&#xA;" OwnerUserId="10327" LastEditorUserId="2452" LastEditDate="2015-08-10T04:28:05.877" LastActivityDate="2015-08-10T04:28:05.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6735" PostTypeId="1" CreationDate="2015-08-09T12:05:42.797" Score="3" ViewCount="1802" Body="&lt;p&gt;I have 5 variables - $v_{1}$, $v_{2}$, $v_{3}$, $v_{4}$, and $v_{5}$. All are categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I conducted a $\chi^{2}$ (chi-square) test to understand the relationship between these variables. I could see that $v_{3}$ and $v_{4}$ have p-values of 0.  I understand that these two variables are dependent on each other. Should I remove one of the variable from further analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if the same $v_{3}$ and $v_{4}$ are independent of other variables, like $v_{1}$, $v_{2}$, and $v_{5}$?&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="13413" LastEditDate="2015-12-05T23:28:51.917" LastActivityDate="2015-12-05T23:28:51.917" Title="Can i use chi square test to remove a particular variable from the model?" Tags="|statistics|correlation|linear-regression|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6736" PostTypeId="1" AcceptedAnswerId="6755" CreationDate="2015-08-09T13:56:21.477" Score="1" ViewCount="12641" Body="&lt;p&gt;I recently found this use cases on Kaggle for Data Science and Data Analytics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/wiki/DataScienceUseCases&quot; rel=&quot;nofollow&quot;&gt;Data Science Use Cases&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am curious to find examples and case studies of real reports from other professionals in some of those use cases on the link. Including hypothesis, testing, reports, conclusions (and maybe also the datasets they have used)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any kind of those reports to share?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of the difficulty to share this kind of details and this is why it is not a Google-search-and-find answer. &lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2019-07-26T03:36:33.270" Title="Do you have any real example of Data Science reports?" Tags="|usecase|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6737" PostTypeId="1" CreationDate="2015-08-09T14:27:33.883" Score="3" ViewCount="1623" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Natural_language_generation&quot; rel=&quot;nofollow&quot;&gt;Natural Language Generation (NLG)&lt;/a&gt; is the natural language&#xA;  processing task of generating natural language from a machine&#xA;  representation system such as a knowledge base or a logical form. — &#xA;  Wikipedia&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is NLG about building meaningful sentences, reports, etc.? Can NLG build valid dictionary words as well? For example, without consulting/reading from an English (or any language) dictionary, can an algorithm generate such words?&lt;/p&gt;&#xA;" OwnerUserId="4695" LastEditorUserId="2452" LastEditDate="2015-08-10T04:35:07.993" LastActivityDate="2015-08-11T12:22:01.997" Title="Can natural language generation algorithms generate valid words too?" Tags="|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6739" PostTypeId="2" ParentId="6737" CreationDate="2015-08-09T18:49:55.170" Score="3" Body="&lt;p&gt;Outside of context of NLG (thus not a direct answer to your whole question, but an answer to your question's title): Generating words from a character-level model has been done using RNNs exposed to large corpora of text, such as Wikipedia content, and trained to predict text &lt;em&gt;character-by-character&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Used to generate content, the model is normally fed a few starting characters and asked to predict the next one. A choice is made from its most-likely predictions and fed back to it to continue the sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;Here is a blog showing some examples trained on some Shakespear and Wikipedia.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such a network can and does generate nonsense words, although they are often fitting and might read like e.g. a noun or verb as you could expect depending on context. The sentence structure and grammar can come out sort of right, but the semantic content is usually complete gibberish.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-08-09T20:18:14.600" LastActivityDate="2015-08-09T20:18:14.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6740" PostTypeId="1" AcceptedAnswerId="6743" CreationDate="2015-08-09T19:20:37.687" Score="1" ViewCount="30" Body="&lt;p&gt;If I want to fit a nonlinear regression model with some parameters like $\sigma^2$(where $\sigma$ is the standard deviation, which is positive), how can I guarantee that $\hat{\sigma}$ is positive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, if I use maximum likelihood(and the model only have square term of $\sigma$), how does the optimization method know $\sigma$ is positive? (well, it seems OK that the result estimation of $\sigma$ is negative, but it looks weird, and this is why I ask this question.)&lt;/p&gt;&#xA;" OwnerUserId="1324" LastActivityDate="2015-08-10T08:53:30.417" Title="fit model with sd square" Tags="|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6743" PostTypeId="2" ParentId="6740" CreationDate="2015-08-10T08:53:30.417" Score="2" Body="&lt;p&gt;I would suggest parametrizing with a logarithm of volatility so you don't have to care about positivity, run the estimation and then invert back to original scale. Alternatively, you can consider constrained optimization routine. Without knowing more about the problem (at least the language you're using), that's about it.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-08-10T08:53:30.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6744" PostTypeId="1" CreationDate="2015-08-10T12:48:17.207" Score="1" ViewCount="278" Body="&lt;p&gt;Is it valid to normalise a dataset, reduce dimensionality with PCA and then to normalise the reduced dimension data.    Assuming this is performed on training data, should the same PCA coefficients be used to reduce the dimension of the test data.  Should the same max and min normalisation values be used for the test and training data. I have included a simplified example of the code I am using which may describe I said better. Thanks in advance. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%% Prepare Training Data&#xA;&#xA;&#xA;% Normalise training data&#xA;mindata=min(TRAINDATA); maxdata=max(TRAINDATA);&#xA;TRAINDATA = ((TRAINDATA-repmat(mindata,[size(TRAINDATA,1),1]))./(repmat(maxdata,[size(TRAINDATA,1),1])-repmat(mindata,[size(TRAINDATA,1),1])) - 0.5 ) *2;&#xA;&#xA;% Perform PCA&#xA;mTRAINDATA = mean(mean(TRAINDATA));&#xA;TRAINDATA = TRAINDATA - mTRAINDATA;&#xA;[Cpca,~,~,~,~]=princomp(TRAINDATA,'econ');&#xA;EigenRange = 1:2;&#xA;Cpca = Cpca(:,EigenRange);&#xA;TRAINDATA = TRAINDATA*Cpca;&#xA;TRAINDATA = TRAINDATA + mTRAINDATA;&#xA;&#xA;% Normalise training data second time&#xA;mindata2=min(TRAINDATA); maxdata2=max(TRAINDATA);&#xA;TRAINDATA = ((TRAINDATA-repmat(mindata2,[size(TRAINDATA,1),1]))./(repmat(maxdata2,[size(TRAINDATA,1),1])-repmat(mindata2,[size(TRAINDATA,1),1])) - 0.5 ) *2;&#xA;&#xA;&#xA;&#xA;%% Prepare Test Data&#xA;&#xA;% Normalise using first normalisation values from training data&#xA;TESTDATA = ((TESTDATA-repmat(mindata,[size(TESTDATA,1),1]))./(repmat(maxdata,[size(TESTDATA,1),1])-repmat(mindata,[size(TESTDATA,1),1])) - 0.5 ) *2;&#xA;&#xA;% Perform PCA&#xA;mTESTDATA = mean(mean(TESTDATA));&#xA;TESTDATA = TESTDATA - mTESTDATA;&#xA;TESTDATA = TESTDATA*Cpca;&#xA;TESTDATA = TESTDATA + mTRAINDATA;&#xA;&#xA;% Normalise using second normalisation values from training data&#xA;TESTDATA = ((TESTDATA-repmat(mindata2,[size(TESTDATA,1),1]))./(repmat(maxdata2,[size(TESTDATA,1),1])-repmat(mindata2,[size(TESTDATA,1),1])) - 0.5 ) *2;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12172" LastEditorUserId="12172" LastEditDate="2019-06-18T14:02:05.660" LastActivityDate="2019-06-18T14:02:05.660" Title="Prepare data for SVM, Is it valid to normalise the data before and after PCA dimension reduction" Tags="|machine-learning|svm|dimensionality-reduction|libsvm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6745" PostTypeId="2" ParentId="6727" CreationDate="2015-08-10T15:24:05.837" Score="0" Body="&lt;p&gt;This isn't a python program so strictly it isn't a solution to your problem, technically it isn't even a program - just a command line, but it does illustrate that are many ways to skin a cat, some of them quite elegantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dictionaries are a lovely feature of the modern language python but much earlier languages such as AWK, or it's linux equivalent GAWK, have had them for just short of 40 years, though in those days they were called associative arrays. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here is a one line command using &lt;code&gt;cat&lt;/code&gt; and &lt;code&gt;gawk&lt;/code&gt;, with its ability to accept code as an argument, to accomplish your task, demonstrating that the design principle of linux to be a framework for collaborating utilities is still alive and well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; $&amp;gt; cat in.dat | gawk '{if(NR==1)hdr=$0; else{emp[$1&quot; &quot;$2&quot; &quot;$3]=emp[$1&quot; &quot;$2&quot; &quot;$3]+$4}} END{print hdr; for(i in emp) print i,emp[i];}'  &amp;gt; out.dat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've taken your file formats literally, if you do actually require true &lt;code&gt;CSVs&lt;/code&gt; then a small mod will accomplish that.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat in.csv | awk -F&quot;,&quot; '{if(NR==1)hdr=$0; else{emp[$1&quot;,&quot;$2]=emp[$1&quot;,&quot;$2]+$3}} END{OFS=&quot;,&quot;;print hdr; for(i in emp) print i,emp[i];}' &amp;gt; out.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Name,Age,Salary &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;name 2,55,20066&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;name 1,52,110322&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;name 3,50,100054&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Designed as a text manipulation language by some very clever computer scientists, including Brian Kernighan of &lt;code&gt;C&lt;/code&gt; fame, AWK achieves this compactness by automatically parsing any input it is given into fields ( \$1, \$2, ... ) and keeping track of the number of lines processed. This allows a lot of the I/O management you might need to do in another language to be skipped over as it already exists.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-08-10T15:57:46.180" LastActivityDate="2015-08-10T15:57:46.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6746" PostTypeId="2" ParentId="6691" CreationDate="2015-08-10T15:38:47.583" Score="1" Body="&lt;p&gt;TM7 of The Netherlands has an advanced NLP engine named CARP that could do this for you. Their examples are largely in Dutch and English language, but they have established ontologies for Spanish for a number of their functional modules, including sentiment analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find the public 'playground' at &lt;a href=&quot;http://www.tm7.nl&quot; rel=&quot;nofollow&quot;&gt;http://www.tm7.nl&lt;/a&gt; . Select 'Carp Language Technologie' and follow the link in the blue box to the Playground.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Information about available APIs using SOAP services is available as well under the link 'SOAP Webservices' in English&lt;/p&gt;&#xA;" OwnerUserId="11114" LastActivityDate="2015-08-10T15:38:47.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6747" PostTypeId="2" ParentId="6727" CreationDate="2015-08-10T18:25:21.680" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pandas&lt;/a&gt; is a python library that you will find very useful for these types of tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/29583312/pandas-sum-of-duplicate-attributes&quot;&gt;Here is a stack overflow post&lt;/a&gt; that tells you how to do what you want to accomplish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It boils down to three very pythonic lines with a &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;groupby and transformation&lt;/a&gt; followed by a &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;drop_duplicates&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas&#xA;df = pandas.read_csv('csvfile.csv', header = 0)&#xA;df['Total'] = df.groupby(['Name', 'Age'])['Salary'].transform('sum')&#xA;df.drop_duplicates(take_last=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-08-10T18:25:21.680" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6748" PostTypeId="1" CreationDate="2015-08-10T18:45:08.413" Score="2" ViewCount="120" Body="&lt;p&gt;I am trying to plot &lt;em&gt;time threshold maps&lt;/em&gt;, proposed by Fryzlewicz. The relevant paper is: &lt;a href=&quot;http://stats.lse.ac.uk/fryzlewicz/allth/allth.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time-Threshold Maps: using information from wavelet&#xA;reconstructions with all threshold values simultaneously&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VD6RZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VD6RZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt; Is the top plot in &lt;code&gt;Figure 1&lt;/code&gt; a heat map? How can I plot this in &lt;code&gt;R&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="12177" LastEditorUserId="31513" LastEditDate="2017-05-25T09:23:19.893" LastActivityDate="2017-05-25T09:23:19.893" Title="Time threshold maps plotting in R" Tags="|r|time-series|visualization|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6750" PostTypeId="1" CreationDate="2015-08-11T02:15:24.130" Score="4" ViewCount="4682" Body="&lt;p&gt;I have a task at hand, where I have to explain &lt;em&gt;&lt;code&gt;decision tree&lt;/code&gt;&lt;/em&gt; algorithm to a person who has not much understanding of &lt;code&gt;machine learning&lt;/code&gt;. I have been looking around, but find it difficult to explain the algorithm in layman's terms, so that a person will understand what is happening in the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to describe the algorithm, possibly using some very simple basic example, so that the whole process of the algorithm is broken down to simple steps for understanding?&lt;/p&gt;&#xA;" OwnerUserId="11074" LastEditorUserId="11074" LastEditDate="2015-08-11T03:36:47.097" LastActivityDate="2015-08-12T03:54:32.253" Title="How to explain decision tree algortihm in layman's terms?" Tags="|machine-learning|bigdata|algorithms|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6751" PostTypeId="1" CreationDate="2015-08-11T05:04:52.300" Score="8" ViewCount="6249" Body="&lt;p&gt;In simple terms, if my classification is based on word2vec as features, what I am supposed to do, if a new word comes, which does not have a word2vec?&lt;/p&gt;&#xA;&lt;p&gt;I am trying to used word2vec or word vectors for classification based on entity.&lt;/p&gt;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&lt;p&gt;I have to classify the following words in a sentence as:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;quot;Google gives information about Nigeria&amp;quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Here, I would like to classify Nigeria as location.&lt;/p&gt;&#xA;&lt;p&gt;Suppose I have good word2vec vectors for each of the words, based on some readings I came to know that, recurrent neural networks can be used for this. So, word2vec will capture most locations with a kind of similar word vectors.&lt;/p&gt;&#xA;&lt;p&gt;But my questions are:&lt;/p&gt;&#xA;&lt;p&gt;a) Suppose a new location is there. lets say, Russia . So, do I need to assign a new word vector for this location ?&lt;/p&gt;&#xA;&lt;p&gt;b) If my input for training does not have grammatical sense. For example,&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot; Google information Nigeria &amp;quot; . Everything else Nigeria is associated with a non-location label. Will this condition work for find new location in non-grammatical sentences.&lt;/p&gt;&#xA;" OwnerUserId="12183" LastEditorUserId="29169" LastEditDate="2021-02-18T20:52:44.350" LastActivityDate="2021-02-18T20:52:44.350" Title="How word2vec can handle unseen / new words to bypass this for new classifications?" Tags="|machine-learning|nlp|deep-learning|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6752" PostTypeId="2" ParentId="5667" CreationDate="2015-08-11T06:40:58.583" Score="9" Body="&lt;p&gt;First of all let me tell you that I'm not a Spark expert; I've been using it quite a lot in the last few months, and I believe I now understand it, but I may be wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, answering your questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a.) they are equivalent, but not in the way you're seeing it; Spark will not optimize the graph if you are wondering, but the &lt;code&gt;customMapper&lt;/code&gt; will still be executed twice in both cases; this is due to the fact that for spark, &lt;code&gt;rdd1&lt;/code&gt; and &lt;code&gt;rdd2&lt;/code&gt; are two completely different RDDs, and it will build the transformation graph bottom-up starting from the leafs; so Option 1 will translate to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rdd0 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==0).cache()&#xA;rdd1 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==1).cache()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you said, &lt;code&gt;customMapper&lt;/code&gt; is executed twice (moreover, also &lt;code&gt;rddIn&lt;/code&gt; will be read twice, which means that if it comes from a database, it may be even slower).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b.) there is a way, you just have to move &lt;code&gt;cache()&lt;/code&gt; in the right place:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mappedRdd = rddIn.map(lambda x: customMapper(x)).cache()&#xA;rdd0 = mappedRdd.filter(lambda x: x[0]==0)&#xA;rdd1 = mappedRdd.filter(lambda x: x[0]==1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By doing this, we are telling spark that it can store the partial results of &lt;code&gt;mappedRdd&lt;/code&gt;; it will then use these partial results both for &lt;code&gt;rdd1&lt;/code&gt; and &lt;code&gt;rdd2&lt;/code&gt;. From the spark point of view this is equivalent to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mappedRdd = rddIn.map(lambda x: customMapper(x)).saveAsObjectFile('..')&#xA;# forget about everything&#xA;rdd0 = sc.objectFile('..').filter(lambda x: x[0]==0)&#xA;rdd1 = sc.objectFile('..').filter(lambda x: x[0]==1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12184" LastActivityDate="2015-08-11T06:40:58.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6754" PostTypeId="2" ParentId="6750" CreationDate="2015-08-11T09:20:53.177" Score="1" Body="&lt;p&gt;A decision tree is a graph that uses a branching method to illustrate every possible outcome of a decision. Here each internal node represents a &quot;test&quot; on an attribute (e.g. whether a coin flip comes up heads or tails), each branch represents the outcome of the test and each leaf node represents a class label (decision taken after computing all attributes). The paths from root to leaf represents classification rules. This can be represented as graph. Now you can parse through the possible nodes and edges as per the condition rules (if condition1 and condition2 and condition3 then outcome.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wikipedia link itself has some great examples :&lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Decision_tree&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do check these links also : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.echen.me/2011/03/14/laymans-introduction-to-random-forests/&quot; rel=&quot;nofollow&quot;&gt;http://blog.echen.me/2011/03/14/laymans-introduction-to-random-forests/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://whatis.techtarget.com/definition/decision-tree&quot; rel=&quot;nofollow&quot;&gt;http://whatis.techtarget.com/definition/decision-tree&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-08-11T09:20:53.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6755" PostTypeId="2" ParentId="6736" CreationDate="2015-08-11T09:25:24.340" Score="2" Body="&lt;p&gt;I think you can better go through various university thesis reports and data science related journal papers to get more details on &quot;Including hypothesis, testing, reports, conclusions&quot; of the above mentioned Data science related problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check these links from Stanford university :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs229.stanford.edu/projects2014.html&quot; rel=&quot;nofollow&quot;&gt;http://cs229.stanford.edu/projects2014.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs229.stanford.edu/projects2013.html&quot; rel=&quot;nofollow&quot;&gt;http://cs229.stanford.edu/projects2013.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs229.stanford.edu/projects2012.html&quot; rel=&quot;nofollow&quot;&gt;http://cs229.stanford.edu/projects2012.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-08-11T09:25:24.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6756" PostTypeId="2" ParentId="5001" CreationDate="2015-08-11T10:19:02.817" Score="2" Body="&lt;p&gt;Maybe there's another way to go. The idea would be to generate the dataset you will be processing with your algorithms. You define the models of behaviour of events (those you're looking for and those into which they are hiden). Then generate the data, then analyse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach has the benefit to let you control exactly what is inside the processed data. And make sure your algorithm identifies exactly what it is supposed to identify, no more, no less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With GEDIS Studio we model events behaviours with activity profiles and the generator produces those events. We have implemented generators for telecom CDR, credit card usages, smart metering, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are freely available online from the evaluation account on &lt;a href=&quot;http://www.data-generator.com&quot; rel=&quot;nofollow&quot;&gt;http://www.data-generator.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check the detailed CDR use case at &lt;a href=&quot;http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html&quot; rel=&quot;nofollow&quot;&gt;http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards &lt;/p&gt;&#xA;" OwnerUserId="12189" LastEditorUserId="12189" LastEditDate="2015-08-11T10:34:27.833" LastActivityDate="2015-08-11T10:34:27.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6757" PostTypeId="2" ParentId="6088" CreationDate="2015-08-11T11:01:44.517" Score="1" Body="&lt;p&gt;Is it possible that adding generated data to your data set will decrease the fraud/non fraud ration and make your dataset more representative / usable ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At GenieLog, we are producing test data for designing and testing fraud detection tools. We our generator GEDIS Studio we can define regular profiles and fraudster profiles, instantiating each category to a customizable ratio (for ex. 2 % of customer will have fraudulent usage of generated events.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We did it successfully for telecom CDR (&lt;a href=&quot;http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html&quot; rel=&quot;nofollow&quot;&gt;http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html&lt;/a&gt;) and Credit Card usages. There's a freely available access to the online generator on &lt;a href=&quot;http://www.data-generator.com&quot; rel=&quot;nofollow&quot;&gt;http://www.data-generator.com&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that even if the tool is not matching your needs at least the approach can be valuable. Otherwise I would be interested to read any objection :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="12189" LastActivityDate="2015-08-11T11:01:44.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6758" PostTypeId="2" ParentId="6751" CreationDate="2015-08-11T12:14:32.297" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose a new location is there. lets say, Russia . So, do I need to&#xA;  assign a new word vector for this location ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Define an Unknown Word Vector that is going to represent every word that is not in your list.&lt;/p&gt;&#xA;" OwnerUserId="12192" LastActivityDate="2015-08-11T12:14:32.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6759" PostTypeId="2" ParentId="6737" CreationDate="2015-08-11T12:22:01.997" Score="2" Body="&lt;p&gt;You are looking for Recurrent Neural Networks for character-level language models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at this. &lt;a href=&quot;https://github.com/karpathy/char-rnn&quot; rel=&quot;nofollow&quot;&gt;https://github.com/karpathy/char-rnn&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12193" LastActivityDate="2015-08-11T12:22:01.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6760" PostTypeId="1" CreationDate="2015-08-11T09:26:26.340" Score="2" ViewCount="825" Body="&lt;p&gt;I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic models seem a nice fit since they reduce the noise significantly, but still at the end there are quite a lot of trash topic lists which do not mean anything.What would be the best way to remove these topics. Is there a working example of this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using LDA. Probably I will use Gensim. Its going to be unsupervised and its a simple twitter data. So the whole corpora will be 1500 tweets of 160 characters. &lt;/p&gt;&#xA;" OwnerUserId="12240" OwnerDisplayName="Imo" LastEditorUserId="12240" LastEditDate="2015-08-13T19:34:04.057" LastActivityDate="2015-11-27T14:55:43.017" Title="How to discard trash topics from topic models?" Tags="|nlp|topic-model|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6761" PostTypeId="2" ParentId="6576" CreationDate="2015-08-11T12:27:50.903" Score="2" Body="&lt;p&gt;David has a good point , I would suggest you focus on whatever it is that drives your interest more. It's the only way to succeed in every kind of effort. If you want to build something cool start with it. If you want to read a book thats good too. The starting point doesn't matter. A few days ahead you will have a better understanding on what you want and should do next.&lt;/p&gt;&#xA;" OwnerUserId="12193" LastActivityDate="2015-08-11T12:27:50.903" CommentCount="0" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="6762" PostTypeId="5" CreationDate="2015-08-11T12:59:07.740" Score="0" Body="&lt;p&gt;If observations are words collected into documents, it posits that each document is a mixture of a small number of &lt;a href=&quot;http://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topics&lt;/a&gt; and that each word's creation is attributable to one of the document's topics. LDA represents documents as mixtures of topics that spit out words with certain probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Popular software packages to perform LDA include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mallet.cs.umass.edu/&quot; rel=&quot;nofollow&quot;&gt;Mallet&lt;/a&gt; (Java) &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/software/tmt/tmt-0.4/&quot; rel=&quot;nofollow&quot;&gt;Stanford Topic Modeling Toolbox&lt;/a&gt; (software)  &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;Gensim – Topic Modelling for Humans&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It should not be confused with Linear Discriminant Analysis, a supervised learning procedure for classifying observations into a set of categories.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-08-12T09:32:58.600" LastActivityDate="2015-08-12T09:32:58.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6763" PostTypeId="4" CreationDate="2015-08-11T12:59:07.740" Score="0" Body="Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling." OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-08-12T09:33:01.330" LastActivityDate="2015-08-12T09:33:01.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6764" PostTypeId="2" ParentId="6750" CreationDate="2015-08-11T17:10:07.043" Score="1" Body="&lt;p&gt;I had to do something similar for Regression Tree. I came up with as very simple complete example; I wrote out tables of dependent and independent variables on the board. The trickiest part is to explain how the first split happens, (that it minimizes MSE of dependent variable within the sub-samples), being able to point to the table helps. And after that we do the same thing just for a sub problem.&lt;/p&gt;&#xA;" OwnerUserId="3070" LastActivityDate="2015-08-11T17:10:07.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6765" PostTypeId="1" AcceptedAnswerId="6767" CreationDate="2015-08-11T19:28:29.793" Score="3" ViewCount="107" Body="&lt;p&gt;I am a CS graduate but am very new to data science. I could use some expert advise/insight on a problem I am trying to solve. I've been through the titanic tutorial on gaggle.com which I think was helpful but my problem is a bit different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict diabetes risk based upon Age, Sex...and other factors given this data: &lt;a href=&quot;http://www.healthindicators.gov/Indicators/Diabetes-new-cases-per-1000_555/Profile/ClassicData&quot; rel=&quot;nofollow&quot;&gt;http://www.healthindicators.gov/Indicators/Diabetes-new-cases-per-1000_555/Profile/ClassicData&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data gives new cases people per 1,000 people for each dimension (Age, Sex...etc). What I would like to do is devise a way to predict, given a list of dimensions (Age, Sex...etc) a probability factor for a new diagnosis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far my strategy is to load this data into R and use some package to create a decision tree, similar to what I saw in the titanic example on kaggle.com, then feed in a dimension list. However, I am a bit overwhelmed. Any direction on what I should be studying, packages/methods/examples would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="12203" LastActivityDate="2015-08-11T23:13:22.700" Title="Advise on making predictions given collection of dimensions and corresponding probabilities" Tags="|machine-learning|data-mining|r|predictive-modeling|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6767" PostTypeId="2" ParentId="6765" CreationDate="2015-08-11T21:37:41.603" Score="4" Body="&lt;h2&gt;Aggregate Data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Since you're only given aggregate data, and not individual examples, machine learning techniques like decision trees won't really help you much. Those algorithms gain a lot of traction by looking at correlations within a single example. For instance, the increase in risk from being both obese and over 40 might be much higher than the sum of the individual risks of being obese or over 40 (i.e. the effect is greater than the sum of its parts). Aggregate data loses this information.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The Bayesian Approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On the bright side, though, using aggregate data like this is fairly straightforward, but requires some probability theory. If $D$ is whether the person has diabetes and $F_1,\ldots,F_n$ are the factors from that link you provided, and if I'm doing my math correctly, we can use the formula:&#xA;$$ \text{Prob}(D\ |\ F_1,\ldots,F_n) \propto \frac{\prod_{k=1}^n \text{Prob}(D\ |\ F_k)}{\text{Prob}(D)^{n-1}} $$&#xA;(The proof for this is an extension of the one found &lt;a href=&quot;https://stats.stackexchange.com/a/112361&quot;&gt;here&lt;/a&gt;). This assumes that the factors $F_1,\ldots,F_n$ are conditionally independent given $D$, though that's usually reasonable. To calculate the probabilities, compute the outputs for $D=\text{Diabetes}$ and $\neg D=\text{No diabetes}$ and divide them both by their sum so that they add to 1.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Suppose we had a married, 48-year-old male. Looking at the 2010-2012 data, 0.73% of all people get diabetes ($\text{Prob}(D) = 0.73\%$), 0.77% of married people get diabetes ($\text{Prob}(D\ |\ F_1)$$= 0.77\%$), 1.02% of people age 45-54 get diabetes ($\text{Prob}(D\ |\ F_2) = 1.02\%$), and 0.70% of males get diabetes ($\text{Prob}(D\ |\ F_3) = 0.70\%$). This gives us the unnormalized probabilities:&#xA;$$ \begin{align*} P(D\ |\ F_1,F_2,F_3) &amp;amp;= \frac{(0.77\%)(1.02\%)(0.70\%)}{(0.73\%)^2} &amp;amp;= 0.0103 \\&#xA;P(\neg D\ |\ F_1,F_2,F_3) &amp;amp;= \frac{(99.23\%)(98.98\%)(99.30\%)}{(99.27\%)^2} &amp;amp;= 0.9897 \end{align*}$$&#xA;After normalizing these to add to one (which they already do in this case), we get a 1.03% chance of this person getting diabetes, and a 98.97% chance for them not getting diabetes.&lt;/p&gt;&#xA;" OwnerUserId="11142" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-08-11T23:13:22.700" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6768" PostTypeId="1" CreationDate="2015-08-11T22:21:42.453" Score="9" ViewCount="1161" Body="&lt;p&gt;I'm looking for recommendations as to the best way forward for my current machine learning problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outline of the problem and what I've done is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have 900+ trials of EEG data, where each trial is 1 second long. The ground truth is known for each and classifies state 0 and state 1 (40-60% split)&lt;/li&gt;&#xA;&lt;li&gt;Each trial goes through preprocessing where I filter and extract power of certain frequency bands, and these make up a set of features (feature matrix: 913x32)&lt;/li&gt;&#xA;&lt;li&gt;Then I use sklearn to train the model. cross_validation is used where I use a test size of 0.2. Classifier is set to SVC with rbf kernel, C = 1, gamma = 1 (I've tried a number of different values)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find a shortened version of the code here: &lt;a href=&quot;http://pastebin.com/Xu13ciL4&quot; rel=&quot;noreferrer&quot;&gt;http://pastebin.com/Xu13ciL4&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My issues:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When I use the classifier to predict labels for my test set, every prediction is 0&lt;/li&gt;&#xA;&lt;li&gt;train accuracy is 1, while test set accuracy is around 0.56&lt;/li&gt;&#xA;&lt;li&gt;my learning curve plot looks like this:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lkUnN.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lkUnN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this seems like a classic case of overfitting here. However, overfitting here is unlikely to be caused by a disproportionate number of features to samples (32 features, 900 samples). I've tried a number of things to alleviate this problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've tried using dimensionality reduction (PCA) in case it is because I have too many features for the number of samples, but accuracy scores and learning curve plot looks the same as above. Unless I set the number of components to below 10, at which point train accuracy begins to drop, but is this not somewhat expected given you're beginning to lose information?&lt;/li&gt;&#xA;&lt;li&gt;I have tried normalizing and standardizing the data. Standardizing (SD = 1) does nothing to change train or accuracy scores. Normalizing (0-1) drops my training accuracy to 0.6.&lt;/li&gt;&#xA;&lt;li&gt;I've tried a variety of C and gamma settings for SVC, but they don't change either score&lt;/li&gt;&#xA;&lt;li&gt;Tried using other estimators like GaussianNB, even ensemble methods like adaboost. No change&lt;/li&gt;&#xA;&lt;li&gt;Tried explcitly setting a regularization method using linearSVC but didn't improve the situation&lt;/li&gt;&#xA;&lt;li&gt;I tried running the same features through a neural net using theano and my train accuracy is around 0.6, test is around 0.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm happy to keep thinking about the problem but at this point I'm looking for a nudge in the right direction. Where might my problem be and what could I do to solve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's entirely possible that my set of features just don't distinguish between the 2 categories, but I'd like to try some other options before jumping to this conclusion. Furthermore, if my features don't distinguish then that would explain the low test set scores, but how do you get a perfect training set score in that case? Is that possible?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="12206" LastEditDate="2015-08-12T05:07:24.453" LastActivityDate="2015-12-11T09:36:19.037" Title="sklearn - overfitting problem" Tags="|machine-learning|python|svm|scikit-learn|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6769" PostTypeId="2" ParentId="904" CreationDate="2015-08-11T22:34:51.927" Score="1" Body="&lt;p&gt;R also has an &lt;a href=&quot;http://www.htmlwidgets.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;htmlwidgets&lt;/a&gt; package that incorporated D3 javascript visualizations.&lt;/p&gt;&#xA;" OwnerUserId="12207" LastEditorUserId="29169" LastEditDate="2021-03-11T19:06:05.303" LastActivityDate="2021-03-11T19:06:05.303" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6770" PostTypeId="2" ParentId="6750" CreationDate="2015-08-12T03:54:32.253" Score="3" Body="&lt;p&gt;I learn well by examples, so perhaps this will be helpful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say you are a resort considering your next advertizing campaign. You will be sending out 1,000 vouchers that give two &lt;em&gt;free&lt;/em&gt; days at the resort, with the hope that the customer will decide to extend it to a full week (who want's a two day vacation?!). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a rather extensive database of customer data, including where they are from, income range, duration of stay(s) and money spent at each stay, number of children, marital status, and employment status (working, unemployed, retired). You would like to create a decision tree to help select the customers to send the vouchers to (specifically, those who will spend at least $200 per day and stay at least one week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create the tree, a computer will perform the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;From the above list of customer data, find the property that best separates the customers into two groups.&lt;/li&gt;&#xA;&lt;li&gt;Repeat step one for each of the above groups using the remaining properties.&lt;/li&gt;&#xA;&lt;li&gt;In the end, you will have a tree where at each point, you can make one of two decisions. Following a path leads to a decision. The split points will be chosen to maximize the probability of a correct classification.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerDisplayName="user9424" LastActivityDate="2015-08-12T03:54:32.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6771" PostTypeId="2" ParentId="5178" CreationDate="2015-08-12T09:56:15.270" Score="13" Body="&lt;p&gt;What I am ending up using is a sort of hybrid solution:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;backup of the raw data&lt;/li&gt;&#xA;&lt;li&gt;git of the workflow&lt;/li&gt;&#xA;&lt;li&gt;manual snapshots of workflow + processed data, that are of relevance, e.g.:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;standard preprocessing&lt;/li&gt;&#xA;&lt;li&gt;really time-consuming&lt;/li&gt;&#xA;&lt;li&gt;for publication&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I believe it is seldom sensible to have a full revision history of large amount of binary data, because the time required to review the changes will eventually be so overwhelming that it will not pay off in the long run.&#xA;Maybe a semi-automatic snapshot procedure (eventually to save some disk-space, by not replicating the unchanged data across different snapshots) would be of help.&lt;/p&gt;&#xA;" OwnerUserId="12215" LastEditorUserId="12215" LastEditDate="2015-08-24T11:12:55.660" LastActivityDate="2015-08-24T11:12:55.660" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6773" PostTypeId="1" CreationDate="2015-08-12T14:01:23.257" Score="6" ViewCount="82678" Body="&lt;p&gt;I have a large amount of Data where I have to count meassurments per one ID. What I already did was creating a Data Frame over all Files and I omited the NAs. This part works properly. I was wondering if the nrow-function is the right function to solve this but I figured out that this will not lead me to the target as it returns a single number as output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is if you have entries like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1155 2010-05-02  2.7200    1&#xA;1156 2010-05-05  2.6000    3&#xA;1157 2010-05-08  2.6700    1&#xA;1158 2010-05-11  3.5700    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That I get a list:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID          Number of observations&#xA;1           2&#xA;2           1&#xA;3           1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11094" LastEditorUserId="8479" LastEditDate="2015-08-13T20:09:48.080" LastActivityDate="2020-05-20T13:08:21.487" Title="How to count observations per ID in R?" Tags="|r|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6774" PostTypeId="1" CreationDate="2015-08-12T14:16:42.160" Score="3" ViewCount="2558" Body="&lt;p&gt;I am learning hmm and try to implement it in  Python hmmlearn package(&lt;a href=&quot;http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples&quot; rel=&quot;nofollow&quot;&gt;http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am not quite understand what the documentation says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Classes in this module include MultinomialHMM, GaussianHMM, and&#xA;  GMMHMM. They implement HMM with emission probabilities determined by&#xA;  multimomial distributions, Gaussian distributions and mixtures of&#xA;  Gaussian distributions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Does this means when we try to estimate the model given an observation sequence, the  emission probability can only be these three kind? what if I want to type in the emission probability by meself?&lt;/p&gt;&#xA;" OwnerUserId="12219" LastEditorUserId="11097" LastEditDate="2015-12-27T03:31:05.410" LastActivityDate="2016-11-21T10:02:38.157" Title="emission probability using hmmlearn package in python" Tags="|python|markov-process|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6775" PostTypeId="1" CreationDate="2015-08-12T15:11:36.953" Score="7" ViewCount="3911" Body="&lt;p&gt;Is there algorithmic difference between analyzing video and an image, say for example,if I want object recognition? Or do I just have to analyze every frame of the the video just as an image?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, detecting an object in a single image is easy compared to video becuase time dimension is added to video. In addition, on video, in every frame, the object is most probably moving, which makes the frames in motion... So how can u handle the time factor and the &quot;in motion&quot; part in the video. Those are the problem I imagine in video, it would be nice if u add ur own thoughts on it. Thanks&lt;/p&gt;&#xA;" OwnerUserId="11141" LastEditorUserId="11141" LastEditDate="2015-08-12T15:21:13.803" LastActivityDate="2015-08-17T18:32:02.847" Title="algorithmic difference between image analysis and video analysis" Tags="|machine-learning|object-recognition|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6776" PostTypeId="1" CreationDate="2015-08-12T16:17:14.967" Score="9" ViewCount="1261" Body="&lt;p&gt;Recently Google publicized interesting deep dream. Besides art generation such as &lt;a href=&quot;http://deepdreamgenerator.com/&quot;&gt;http://deepdreamgenerator.com/&lt;/a&gt;, do you see any potential applications of deep dream in computer vision or machine learning?&lt;/p&gt;&#xA;" OwnerDisplayName="rudky martin" LastActivityDate="2015-09-17T22:12:50.787" Title="Any idea about application of deep dream?" Tags="|machine-learning|deep-learning|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6777" PostTypeId="2" ParentId="6773" CreationDate="2015-08-12T19:58:32.517" Score="1" Body="&lt;p&gt;OK if I understood correctly you can do something like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df$observations &amp;lt;- rep(1, nrow(df))&#xA;df &amp;lt;- df[ ,-file_name_column]&#xA;new_data &amp;lt;- data.frame(aggregate(df, by= ID, FUN=sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Caution: this might not work exactly since I am not sure what you data frame looks like. &lt;/p&gt;&#xA;" OwnerUserId="9698" LastActivityDate="2015-08-12T19:58:32.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6778" PostTypeId="1" CreationDate="2015-08-12T23:14:17.480" Score="0" ViewCount="690" Body="&lt;p&gt;Is there any Python library that provides ready-to-use metrics to analyze the performance of a classifier for a multioutput-multiclass classification task? scikit-learn doesn't have this option yet (as stated in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; and in the corresponding &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/3453&quot; rel=&quot;nofollow&quot;&gt;feature request on GitHub&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-09-30T01:10:13.797" Title="Python library to compute some metrics for multioutput-multiclass classification task" Tags="|performance|multiclass-classification|software-recommendation|multilabel-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6779" PostTypeId="5" CreationDate="2015-08-12T23:15:20.783" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-08-12T23:15:20.783" LastActivityDate="2015-08-12T23:15:20.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6780" PostTypeId="4" CreationDate="2015-08-12T23:15:20.783" Score="0" Body="Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A text might be about any of religion, politics, finance or education at the same time or none of these." OwnerUserId="43605" LastEditorUserId="43605" LastEditDate="2018-07-02T16:36:17.237" LastActivityDate="2018-07-02T16:36:17.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6781" PostTypeId="2" ParentId="6773" CreationDate="2015-08-13T04:21:31.877" Score="1" Body="&lt;p&gt;aggregate() should work, as the previous answer suggests. Another option is with the &lt;a href=&quot;http://plyr.had.co.nz/&quot; rel=&quot;nofollow&quot;&gt;plyr package&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;count(yourDF,c('id'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using more columns in the vector with 'id' will subdivide the count. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe ddply() (also part of plyr) has a summarize argument which can also do this, similar to aggregate().&lt;/p&gt;&#xA;" OwnerUserId="12197" LastActivityDate="2015-08-13T04:21:31.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6782" PostTypeId="2" ParentId="6768" CreationDate="2015-08-13T07:31:42.373" Score="1" Body="&lt;p&gt;To see if SVM can capture any signal at all, try to balance your data: create training and test sets that consist of exactly 50% positive and 50% negative samples (i.e., by subsampling randomly from whichever one is bigger). Also standardize the data (subtract the mean and divide by standard deviation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For balancing, you might try changing the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;class_weight&lt;/a&gt; parameter in sklearn, but we found the manual method (subsampling) to work better.)&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2015-08-13T07:31:42.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6783" PostTypeId="1" CreationDate="2015-08-13T07:37:46.547" Score="2" ViewCount="129" Body="&lt;p&gt;I am attempting to train a classifier to predict different prices for an item in different suburbs. I have several features, two of which are a latitude and longitude for the centroid of the suburb.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attempting to train the model to classify the price of an item in a bin of $10 size. The geospatial element will definitely affect the price of the item, however the training data I have will have gaps in it (i.e. I don't have prices for all suburbs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to engineer a feature that will include this geospatial information and be able to fill in the gaps in the training/test data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have tried creating new features for the bearing and distance from the capital city which seemed to work okay, as well as binning the latitude and longitude which performs worse than the bearing/distance. I did consider using a geohash, however I think that this will be too complex a feature for a classifier to understand.&lt;/p&gt;&#xA;" OwnerUserId="10998" LastActivityDate="2015-08-19T03:37:17.200" Title="Multiclass Classification that includes a Geospatial Element" Tags="|feature-extraction|multiclass-classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6784" PostTypeId="2" ParentId="6555" CreationDate="2015-08-13T07:45:45.927" Score="12" Body="&lt;p&gt;IPython has now moved to &lt;strong&gt;&lt;a href=&quot;http://blog.jupyter.org/2015/08/12/first-release-of-jupyter/&quot;&gt;version 4.0&lt;/a&gt;&lt;/strong&gt;, which means that if you are using it, it will be reading its configuration from &lt;code&gt;~/.jupyter&lt;/code&gt;, not &lt;code&gt;~/.ipython&lt;/code&gt;. You have to create a new configuration file with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;jupyter notebook --generate-config&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then edit the resulting &lt;code&gt;~/.jupyter/jupyter_notebook_config.py&lt;/code&gt; file according to your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More installation instructions &lt;a href=&quot;https://jupyter.readthedocs.org/en/latest/config.html&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4815" LastActivityDate="2015-08-13T07:45:45.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6786" PostTypeId="1" CreationDate="2015-08-13T13:52:11.463" Score="4" ViewCount="2022" Body="&lt;p&gt;I've searched quite a bit and haven't landed on any useful results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem statement is:&#xA;Given a set of vectors, I wish to find its approximate k-nearest neighbors.&#xA;The caveat here is that each of my dimensions resemble a different entity and hence we cannot use the same weight for each dimension while computing the distance.&#xA;Thus, solutions like kd-tree don't work as is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any data-structure or any alternate algorithm that I can use to find such approximate weighted k-nearest neighbors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Multiplying the initial input data with their weights so as to get a uniform weight is not an option.&lt;/p&gt;&#xA;" OwnerUserId="226" LastEditorUserId="226" LastEditDate="2015-08-13T14:51:09.187" LastActivityDate="2021-05-03T13:04:15.483" Title="Weighted k nearest neighbor search" Tags="|machine-learning|data|search|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6787" PostTypeId="1" CreationDate="2015-08-13T13:59:52.603" Score="38" ViewCount="58426" Body="&lt;p&gt;Recently a friend of mine was asked whether decision tree algorithms are linear or nonlinear algorithms in an interview. I tried to look for answers to this question but couldn't find any satisfactory explanation. Can anyone answer and explain the solution to this question? Also, what are some other examples of nonlinear machine learning algorithms?&lt;/p&gt;&#xA;" OwnerUserId="11074" LastEditorUserId="28175" LastEditDate="2019-02-25T12:38:19.573" LastActivityDate="2019-09-10T22:27:06.997" Title="Are decision tree algorithms linear or nonlinear" Tags="|machine-learning|classification|decision-trees|algorithms|pac-learning|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="6788" PostTypeId="2" ParentId="6776" CreationDate="2015-08-13T15:41:05.987" Score="1" Body="&lt;p&gt;It is impossible to prove a negative, but other than using the same pattern detection system in general to detect shapes/images and replace them with other similar images, possibly for use in automatic image correction or similar, I don't think it has real potential outside of modifying pictures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I may have to delete this answer if it is proven wrong.&lt;/p&gt;&#xA;" OwnerUserId="12222" LastActivityDate="2015-08-13T15:41:05.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6789" PostTypeId="1" CreationDate="2015-08-14T07:19:18.463" Score="3" ViewCount="2400" Body="&lt;p&gt;I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# install&#xA;Pkg.add(&quot;MNIST&quot;);&#xA;using MNIST&#xA;&#xA;# training data&#xA;X,y = traindata(); &#xA;m = size(X, 2);&#xA;inputLayerSize = size(X,1); &#xA;hiddenLayerSize = 300;&#xA;outputLayerSize = 10;&#xA;&#xA;# representing each output as an array of size of the output layer&#xA;eyeY = eye(outputLayerSize);&#xA;intY = [convert(Int64,i)+1 for i in y];&#xA;Y = zeros(outputLayerSize, m);&#xA;for i = 1:m&#xA;    Y[:,i] = eyeY[:,intY[i],];&#xA;end&#xA;&#xA;# weights with bias&#xA;Theta1 = randn(inputLayerSize+1, hiddenLayerSize); &#xA;Theta2 = randn(hiddenLayerSize+1, outputLayerSize); &#xA;&#xA;function sigmoid(z)&#xA;    g = 1.0 ./ (1.0 + exp(-z));&#xA;    return g;&#xA;end&#xA;&#xA;function sigmoidGradient(z)&#xA;  return sigmoid(z).*(1-sigmoid(z));&#xA;end&#xA;&#xA;# learning rate&#xA;alpha = 0.01;&#xA;# number of iterations&#xA;epoch = 20;&#xA;# cost per epoch&#xA;J = zeros(epoch,1);&#xA;# backpropagation algorithm&#xA;for i = 1:epoch&#xA;    for j = 1:m # for each input&#xA;        # Feedforward&#xA;        # input layer&#xA;        # add one bias element&#xA;        x1 = [1, X[:,j]];&#xA;&#xA;        # hidden layer&#xA;        z2 = Theta1'*x1;&#xA;        x2 = sigmoid(z2);&#xA;        # add one bias element&#xA;        x2 = [1, x2];&#xA;&#xA;        # output layer&#xA;        z3 = Theta2'*x2;&#xA;        x3 = sigmoid(z3);&#xA;&#xA;        # Backpropagation process&#xA;        # delta for output layer&#xA;        delta3 = x3 - Y[:,j];&#xA;        delta2 = (Theta2[2:end,:]*delta3).*sigmoidGradient(z2) ;&#xA;&#xA;        # update weights&#xA;        Theta1 = Theta1 - alpha* x1*delta2';&#xA;        Theta2 = Theta2 - alpha* x2*delta3';&#xA;    end&#xA;end&#xA;&#xA;function predict(Theta1, Theta2, X)&#xA;    m = size(X, 2); &#xA;    p = zeros(m, 1);&#xA;    h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);&#xA;    h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);&#xA;    # 1 index is for 0, 2 for 1 ...so forth&#xA;    for i=1:m&#xA;        p[i,:] = indmax(h2[:,i])-1;&#xA;    end&#xA;    return p;&#xA;end&#xA;&#xA;function accuracy(truth, prediction)&#xA;    m = length(truth);&#xA;    sum =0;&#xA;    for i=1:m&#xA;        if truth[i,:] == pred[i,:]&#xA;            sum = sum +1;&#xA;        end&#xA;    end&#xA;  return (sum/m)*100;&#xA;end&#xA;&#xA;pred = predict(Theta1, Theta2, X);&#xA;println(&quot;train accuracy: &quot;, accuracy(y, pred));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12250" LastEditorUserId="4647" LastEditDate="2015-08-21T16:37:10.417" LastActivityDate="2017-07-27T20:50:57.193" Title="Neural network for MNIST: very low accuracy" Tags="|machine-learning|neural-network|julia|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="6790" PostTypeId="2" ParentId="6787" CreationDate="2015-08-14T12:13:05.467" Score="24" Body="&lt;p&gt;A decision tree is a non-linear mapping of &lt;code&gt;X&lt;/code&gt; to &lt;code&gt;y&lt;/code&gt;. This is easy to see if you take an arbitrary function and create a tree to its maximum depth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if x = 1, y = 1&#xA;if x = 2, y = 15&#xA;if x = 3, y = 3&#xA;if x = 4, y = 27&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course, this is a completely over-fit tree and won't generalize. But it demonstrates why a decision tree is a non-linear mapping.&lt;/p&gt;&#xA;" OwnerUserId="10816" LastActivityDate="2015-08-14T12:13:05.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6792" PostTypeId="1" AcceptedAnswerId="8019" CreationDate="2015-08-14T16:43:19.007" Score="2" ViewCount="157" Body="&lt;p&gt;With companies and governments hungry for all the data about people, I was wondering if it was possible to gain some privacy by drowning relevant information in a sea of random data. For example a browser extension which keeps searching for random words and expressions in the background. Maybe sending generated emails to generated addresses. Or producing made up location data on my phone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would these measures help at all or the data miner AIs would just see right through them?&lt;/p&gt;&#xA;" OwnerUserId="12258" LastActivityDate="2015-09-06T10:33:39.820" Title="Privacy through fake data?" Tags="|bigdata|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6794" PostTypeId="1" CreationDate="2015-08-14T19:36:11.590" Score="2" ViewCount="323" Body="&lt;p&gt;There's really no &quot;standard,&quot; I assume, but how do you deal with null values when plotting data? An example of what I'm talking about is listed below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HSsHA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HSsHA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I'm plotting data on number of toys by color. Nearly half of these toys in my database have &lt;code&gt;null&lt;/code&gt; values for color. Do you guys include the toys with &lt;code&gt;null&lt;/code&gt; values in your visualization? How do you address that issue? Saying &quot;nearly half of the toys didn't have colors listed&quot; doesn't really add much to the conversation.&lt;/p&gt;&#xA;" OwnerUserId="12265" LastEditorUserId="8878" LastEditDate="2016-01-05T09:11:19.727" LastActivityDate="2016-01-05T09:12:48.950" Title="What is the standard for dealing with null values in plotting?" Tags="|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6795" PostTypeId="2" ParentId="6794" CreationDate="2015-08-14T21:12:39.333" Score="1" Body="&lt;p&gt;Imo, It depends on two major factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the purpose of the plot&lt;/li&gt;&#xA;&lt;li&gt;Does the &lt;code&gt;null&lt;/code&gt; value indicate incompleteness (&lt;em&gt;one of the values, but unknown which one&lt;/em&gt;), or does it indicate a last final option, (&lt;em&gt;like: multiple colors&lt;/em&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Imagine if the purpose of the plot is to compare which toy colors are most popular to a particular person/group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;code&gt;null&lt;/code&gt; indicates multi-colored toys, it may be important to include the data but rename it to something more meaningful. (&lt;em&gt;Multi-colored&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;code&gt;null&lt;/code&gt; indicates that we just don't know the color data for those toys, it would probably be best to exclude that data from the plot. However, with such a large percentage, it would be good to note that &quot;x%&quot; were excluded from your plot because they had no color information associated with them. Perhaps purple is in such a majority because this person/group really likes purple, or maybe its because purple toys almost never made it into the &lt;code&gt;null&lt;/code&gt; category for whatever reason.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The same line of thinking should be applied to whatever the purpose of the plot is. If the &lt;code&gt;null&lt;/code&gt; values are relevant to the purpose, include them. If not, don't - and maybe take a note of how much data is excluded if it could influence the results substantially (such as this case)&lt;/p&gt;&#xA;" OwnerUserId="12222" LastEditorUserId="8878" LastEditDate="2016-01-05T09:12:48.950" LastActivityDate="2016-01-05T09:12:48.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6796" PostTypeId="2" ParentId="6792" CreationDate="2015-08-14T22:44:27.743" Score="1" Body="&lt;p&gt;This was the idea behind paranoid linux which began as fiction and became a real project that did not reach fruition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Paranoid Linux is an operating system that assumes that its operator is under assault from the government (it was intended for use by Chinese and Syrian dissidents), and it does everything it can to keep your communications and documents a secret. It even throws up a bunch of &quot;chaff&quot; communications that are supposed to disguise the fact that you're doing anything covert. So while you're receiving a political message one character at a time, ParanoidLinux is pretending to surf the Web and fill in questionnaires and flirt in chat-rooms. Meanwhile, one in every five hundred characters you receive is your real message, a needle buried in a huge haystack.&#xA;~Cory Doctorow (Little Brother, 2008)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When those words were written, ParanoidLinux was just a fiction. It is our goal to make this a reality. The project officially started on May 14th, and has been growing ever since. We welcome your ideas, contributions, designs, or code. You can find us on freenode's irc server in the #paranoidlinux channel. Hope to see you there!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the most modern equivalent is &lt;a href=&quot;https://piratelinux.org/&quot; rel=&quot;nofollow&quot;&gt;Pirate Linux&lt;/a&gt; along with the &lt;a href=&quot;https://www.torproject.org/&quot; rel=&quot;nofollow&quot;&gt;TOR Project&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, Cory Doctorow's book &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0765323117&quot; rel=&quot;nofollow&quot;&gt;Little Brother&lt;/a&gt;&quot; is kind of a fun read if you are a geek.  If you are reading this then you probably &lt;strong&gt;ARE&lt;/strong&gt; a geek :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reality is that it is very difficult to hide one's true intent when high quality clustering, classification and anomaly/novelty detection is used.  I work with guys who have done this type of behavior detection for some very high profile three letter agencies and it is very very easy to detect nefarious behavior if you have enough data.  So much so that undercover good guys sometimes clearly show up in the data sets.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-19T03:26:00.770" LastActivityDate="2015-08-19T03:26:00.770" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6797" PostTypeId="1" CreationDate="2015-08-14T23:12:44.480" Score="2" ViewCount="2304" Body="&lt;p&gt;I have an unbalanced &lt;a href=&quot;http://s000.tinyupload.com/?file_id=03859222027158375634&quot; rel=&quot;nofollow&quot;&gt;data set&lt;/a&gt; with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. &lt;strong&gt;How should I tune hyperparameters in this setting?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;My suggestion: separate data into train/validation set, use probabilistic output together with cost matrix to assign predicted classes, tune hyperparameters to maximize &lt;a href=&quot;https://en.wikipedia.org/wiki/Accuracy_and_precision&quot; rel=&quot;nofollow&quot;&gt;accuracy&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I increase performance?&lt;/strong&gt; Currently I use random forest with &lt;em&gt;nodesize=1&lt;/em&gt; and &lt;em&gt;mtry=5&lt;/em&gt; which gives about 97% accuracy.&lt;/p&gt;&#xA;" OwnerUserId="10694" LastActivityDate="2016-03-17T08:49:10.570" Title="Tune hyperparameters for cost-sensitive classification" Tags="|r|classification|class-imbalance|hyperparameter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6798" PostTypeId="1" CreationDate="2015-08-15T01:08:48.490" Score="4" ViewCount="6268" Body="&lt;p&gt;Is there any comprehensive list of past, current and future NLP challenges?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. for NLP conferences, &lt;a href=&quot;http://www.cs.rochester.edu/~tetreaul/conferences.html&quot; rel=&quot;nofollow&quot;&gt;Joel Tetreault's unofficially official conference calendar&lt;/a&gt; and &lt;a href=&quot;http://www.wikicfp.com/cfp/call?conference=natural%20language%20processing&quot; rel=&quot;nofollow&quot;&gt;WikiCFP&lt;/a&gt; are pretty good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://aclweb.org/aclwiki/index.php?title=Competitions_and_Challenges&quot; rel=&quot;nofollow&quot;&gt;&quot;Competitions and Challenges&quot; page on the ACL wiki&lt;/a&gt; quite incomplete.&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2018-08-18T03:11:08.190" Title="List of NLP challenges" Tags="|nlp|reference-request|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6799" PostTypeId="2" ParentId="5357" CreationDate="2015-08-15T10:14:33.270" Score="2" Body="&lt;p&gt;There are some C++ tools for statistics and data science like ROOT &lt;a href=&quot;https://root.cern.ch/drupal/&quot; rel=&quot;nofollow&quot;&gt;https://root.cern.ch/drupal/&lt;/a&gt; , BAT &lt;a href=&quot;https://www.mppmu.mpg.de/bat/&quot; rel=&quot;nofollow&quot;&gt;https://www.mppmu.mpg.de/bat/&lt;/a&gt; , boost  , or OpenCV &lt;/p&gt;&#xA;" OwnerUserId="12275" LastActivityDate="2015-08-15T10:14:33.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6800" PostTypeId="2" ParentId="5357" CreationDate="2015-08-15T18:52:20.147" Score="12" Body="&lt;p&gt;I agree that the current trend is to use Python/R and to bind it to some C/C++ extensions for computationally expensive tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you want to stay in C/C++, you might want to have a look at &#xA;&lt;a href=&quot;http://dlib.net/&quot; rel=&quot;noreferrer&quot;&gt;Dlib&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dlib is a general purpose cross-platform C++ library designed using contract programming and modern C++ techniques. It is open source software and licensed under the Boost Software License. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QmR2X.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QmR2X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-08-15T18:52:20.147" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6801" PostTypeId="2" ParentId="6611" CreationDate="2015-08-15T18:52:22.243" Score="2" Body="&lt;p&gt;There are methods that approximate Jaccard similarity using hash functions, one of them &lt;a href=&quot;https://en.wikipedia.org/wiki/MinHash&quot; rel=&quot;nofollow&quot;&gt;MinHash&lt;/a&gt;. They scale well with the number of dimensions and  evaluate similarity of two items. If you want to do nearest neighbor queries (finding similar items given one item) you should look at locality sensitive hashing (LSH), which essentially maps similar items to the same hash value. There is a nice implementation in scikit-learn called &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LSHForest.html#sklearn.neighbors.LSHForest&quot; rel=&quot;nofollow&quot;&gt;LSHForest&lt;/a&gt;. It approximates cosine distance and scales well to large number of items and dimensions.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2015-08-15T18:52:22.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6802" PostTypeId="2" ParentId="5303" CreationDate="2015-08-15T19:12:31.760" Score="0" Body="&lt;p&gt;Take the cluster centers and project them back to the original feature vector using the inverse transform of your PCA. That gives you an intuition of what the clusters represent. As a second step you can take all original feature vectors that were assigned to one cluster and calculate histograms on each feature dimension. This will give you a feeling of in-cluster variation and cluster boundaries along the original feature dimensions.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2015-08-15T19:12:31.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6803" PostTypeId="2" ParentId="6394" CreationDate="2015-08-15T20:46:37.993" Score="4" Body="&lt;p&gt;Especially when your main goal is learning, I would break it into several phases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Get familiar with pandas dataframes and visualization using matplotlib. Try loading subsets of your datasets using pandas and visualize them using matplotlib, for example plot the time-series, or word count histograms. This will come in handy lateron to be able to understand the predictions/clusters you will create using machine learning methods. Pandas also provides functions to clean, aggregate and resample the data, which will be useful to align multiple time-series (e.g. weather and time-series) to the same sampling rate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Familiarize with scikit-learn using the articles and examples in the excellent online documentation. Play with the feature extraction, dimensionality reduction, classification and regression methods relevant to your data, apply them on smaller subsets and visualize the results. Learn about cross-validation and find a good algorithm and parameters that work well on a subset of your data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;One you have a pipeline setup and you want to run it on the full dataset, Spark can come in handy for feature extraction and cross-validation, as those tasks can be often run very well independently and in parallel. Install the Spark libraries on your machine, and reproduce the Pyspark examples (e.g. calculating Pi) in local mode. No need to setup a Hadoop cluster for that, Spark can make full use of your machine in local mode, reading from local files on disk. Once you have that running, try to express the expensive steps of your code from step 2 as Spark RDD operations. As Spark can make use of all cores of your machine, you should already see a speedup - debugging becomes much harder though compared to working with pure Python scripts.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Play with advanced stuff like deep learning, using the word2vec feature extraction. Compare with traditional feature extraction (bag of words).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(optional) If you want to experience the speedup you can get from Spark in a distributed setting, get an account in AWS, put your data on S3 and fire your Spark scripts against a multi-node cluster instantiated using Amazon EMR. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Regarding data storage, from my experience CSV files or similar work best to load the data into both Pandas and Spark. There are Spark connectors for databases (e.g. Cassandra), but I am not sure about MySQL and Solr. For processed data and intermediate results (e.g. models) I would always use Python's serialization framework pickle, so you can directly get the object instances back into Python without the need to parse file formats and reinstantiate objects.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2015-08-15T20:46:37.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6804" PostTypeId="2" ParentId="4971" CreationDate="2015-08-16T04:35:34.507" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://topsy.com/&quot; rel=&quot;nofollow&quot;&gt;Topsy&lt;/a&gt; - The site provides much more rate limit, and has archived all the tweets from 2006.&#xA;Its API provides a free trial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A solution that is paid will be &lt;a href=&quot;https://blog.twitter.com/2015/full-archive-search-api&quot; rel=&quot;nofollow&quot;&gt;Gnip's Full history search API&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12286" LastActivityDate="2015-08-16T04:35:34.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6805" PostTypeId="1" CreationDate="2015-08-16T19:14:01.430" Score="1" ViewCount="587" Body="&lt;p&gt;I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knn proba 1, knn proba 2.... Random forests proba 1 etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fantastic!... But when I use the model against new data the accuracy is very poor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In part it feels like a case of overfitting but surely the test set should be poor as well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why might the test data so good but new data so poor... What have I done wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &#xA;Chris&lt;/p&gt;&#xA;" OwnerUserId="12294" LastActivityDate="2015-08-17T12:47:39.600" Title="Ensemble model overfitting?" Tags="|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6806" PostTypeId="2" ParentId="6805" CreationDate="2015-08-17T04:31:05.600" Score="0" Body="&lt;p&gt;try cross-validation on training data, cross-validation combines (averages) measures of fit (prediction error) to correct for the optimistic nature of training error and derive a more accurate estimate of model prediction performance (eg : k-fold), check accuracy (Precision/recall/f measure) and then use this classifier on test data.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-08-17T04:31:05.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6807" PostTypeId="2" ParentId="6775" CreationDate="2015-08-17T05:18:34.563" Score="8" Body="&lt;p&gt;This is a huge topic, so I will just give you a high-level overview and some pointers to more info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there are definitely ways to work with video that are different from working with individual still images.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the simplest level, it could be running an object detector such as HoG (or a sliding-window convnet) on each frame, and then some means of assigning nearby detections in adjacent frames to be the same object, and discarding detections which do not seem to have continuity in time.  Many of the algorithms in this field seem to look at a single frame as a building block to looking at the whole sequence, where perhaps the data from adjacent frames is combined, aggregated, and/or used to disambiguate the current frame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to first estimate object motion between frames (using optical flow, phase correlation, pyramid block matching or another method) and then treat areas of multiple frames that are collocated after accounting for motion as being the same object.  This is very powerful but limited by the accuracy of the motion estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In newer research, there is a back-and-forth between finding where things are (detection) and how things move (tracking), where each task can help the other one, for example (Kalal 2010) or (Andriluka 2008), to the point that the two parts of the algorithm are no longer separable.  Kalal's &lt;a href=&quot;http://kahlan.eps.surrey.ac.uk/featurespace/tld/&quot; rel=&quot;noreferrer&quot;&gt;TLD&lt;/a&gt; algorithm is a recently famous version of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also algorithms which work directly in the spatiotemporal (or sometimes just temporal) domain.  An example of purely temporal would be detecting vehicles by the periodicity of variation of their wheel spokes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the frequently studied model problems are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;People, vehicles or other objects detection and tracking throughout a video sequence.  For example, using the &lt;a href=&quot;http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/&quot; rel=&quot;noreferrer&quot;&gt;CalTech Pedestrians&lt;/a&gt; dataset or other standard benchmarks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hand gesture recognition, either for games, user interface, or sometimes sign language recognition.  Often using depth data, for example Kinect video.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Activity recognition, for example walking vs standing.  Due to the nature of this problem, it is much more common to see pure spatiotemporal algorithms here, for example (Sadanand and Corso 2012).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Simultaneous location and mapping (SLAM).  Often found in robotics, this problem is basically to build a 3D model of the environment from video of a single camera (or stereo or depth video) which is moving around; the environment is often assumed to be static.  This is often done by running some kind of feature detector such as SIFT or SURF, matching up features in successive frames, and then building point clouds from the relative 3D motion implied by features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are some more references:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shah, Mubarak, and Ramesh Jain, eds. Motion-based recognition. Vol. 9. Springer Science &amp;amp; Business Media, 2013.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turk, Matthew. &quot;Gesture recognition.&quot; &lt;a href=&quot;http://www.springer.com/us/book/9780387307718&quot; rel=&quot;noreferrer&quot;&gt;Computer Vision: A Reference Guide&lt;/a&gt; (2014): 346-349.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosenfeld, Azriel, David Doermann, and Daniel DeMenthon, eds. Video mining. Vol. 6. Springer Science &amp;amp; Business Media, 2013.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kalal, Zdenek, Krystian Mikolajczyk, and Jiri Matas. &quot;&lt;a href=&quot;http://vision.stanford.edu/teaching/cs231b_spring1415/papers/Kalal-PAMI.pdf&quot; rel=&quot;noreferrer&quot;&gt;Tracking-learning-detection.&lt;/a&gt;&quot; Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.7 (2012): 1409-1422.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M. Andriluka, S. Roth, B. Schiele. &lt;a href=&quot;https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/research/people-detection-pose-estimation-and-tracking/people-tracking-by-detection-and-people-detection-by-tracking/&quot; rel=&quot;noreferrer&quot;&gt;People-Tracking-by-Detection and People-Detection-by-Tracking&lt;/a&gt;. Computer Vision and Pattern Recognition (CVPR) 2008&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sadanand, Sreemanananth, and Jason J. Corso. &quot;Action bank: A high-level representation of activity in video.&quot; Computer Vision and Pattern Recognition (CVPR), 2012&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2015-08-17T18:32:02.847" LastActivityDate="2015-08-17T18:32:02.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6809" PostTypeId="1" AcceptedAnswerId="6811" CreationDate="2015-08-17T08:07:58.280" Score="5" ViewCount="14637" Body="&lt;p&gt;I have done some clustering and I would like to visualize the results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the function I have written to plot my clusters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sklearn&#xA;from sklearn.cluster import DBSCAN&#xA;from sklearn import metrics&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;from sklearn.cluster import DBSCAN&#xA;from sklearn import metrics&#xA;from sklearn.datasets.samples_generator import make_blobs&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;&#xA;def plot_cluster(cluster, sample_matrix):&#xA;    '''Input:  &quot;cluster&quot;, which is an object from DBSCAN, &#xA;       e.g. dbscan_object = DBSCAN(3.0,4)&#xA;&quot;sample_matrix&quot; which is a data matrix:  &#xA;X = [&#xA;    [0,5,1,2],&#xA;    [0,4,1,3],&#xA;    [0,5,1,3],&#xA;    [0,5,0,2],&#xA;    [5,5,5,5],&#xA;    ]&#xA;        Output: Plots the clusters nicely.    &#xA;    '''&#xA;    import matplotlib.pyplot as plt&#xA;    import numpy as np&#xA;&#xA;    f = lambda row: [float(x) for x in row]&#xA;&#xA;    sample_matrix = map(f,sample_matrix)&#xA;    print sample_matrix&#xA;    sample_matrix = StandardScaler().fit_transform(sample_matrix)&#xA;&#xA;    core_samples_mask = np.zeros_like(cluster.labels_, dtype=bool)&#xA;    core_samples_mask[cluster.core_sample_indices_] = True&#xA;    labels = cluster.labels_&#xA;&#xA;    # Black removed and is used for noise instead.&#xA;    unique_labels = set(labels)&#xA;    colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))&#xA;    for k, col in zip(unique_labels, colors):&#xA;        if k == -1:&#xA;            # Black used for noise.&#xA;            col = 'k'&#xA;&#xA;        class_member_mask = (labels == k)  # generator comprehension &#xA;        # X is your data matrix&#xA;        X = np.array(sample_matrix)&#xA;&#xA;        xy = X[class_member_mask &amp;amp; core_samples_mask]&#xA;&#xA;        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,&#xA;                 markeredgecolor='k', markersize=14)&#xA;&#xA;        xy = X[class_member_mask &amp;amp; ~core_samples_mask]&#xA;        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=col,&#xA;                 markeredgecolor='k', markersize=6)&#xA;&#xA;    plt.ylim([0,10]) &#xA;    plt.xlim([0,10])    &#xA;#        plt.title('Estimated number of clusters: %d' % n_clusters_)&#xA;    plt.savefig('cluster.png')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The function above is copied almost verbatim from the scikit-learn demo &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#example-cluster-plot-dbscan-py&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet, when I try it on the following:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dbscan_object = DBSCAN(3.0,4)&#xA;&#xA;X = [&#xA;    [0,5,1,2],&#xA;    [0,4,1,3],&#xA;    [0,5,1,3],&#xA;    [0,5,0,2],&#xA;    [5,5,5,5],&#xA;    ]&#xA;&#xA;result = dbscan_object.fit(X)&#xA;print result.labels_&#xA;&#xA;print 'plotting '&#xA;plot_cluster(result, X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...It produces a single point.  What is the best way to plot clusters in python?&lt;/p&gt;&#xA;" OwnerUserId="10512" LastEditorUserId="13727" LastEditDate="2016-03-02T15:51:37.527" LastActivityDate="2016-03-02T15:51:37.527" Title="How to plot/visualize clusters in scikit-learn (sklearn)?" Tags="|python|scikit-learn|clustering|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6810" PostTypeId="2" ParentId="6805" CreationDate="2015-08-17T12:47:39.600" Score="1" Body="&lt;p&gt;Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;random_forest.fit(train_data,target)&#xA;random_forest_probabilities = random_forest.predict_proba(train_data)&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for &amp;lt;fold&amp;gt; in &amp;lt;crossvalidation&amp;gt;:&#xA;     random_forest.fit(&amp;lt;train_data_fold&amp;gt;,&amp;lt;target_fold&amp;gt;)&#xA;     &amp;lt;fold predictions&amp;gt; = random_forest.fit(&amp;lt;test_data_fold&amp;gt;)&#xA;     ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-08-17T12:47:39.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6811" PostTypeId="2" ParentId="6809" CreationDate="2015-08-17T13:56:53.857" Score="5" Body="&lt;p&gt;When I run the code you posted, I get three points on my plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KpbTX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KpbTX.png&quot; alt=&quot;clusters&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;point&quot; at (0, 4) corresponds to &lt;code&gt;X[1]&lt;/code&gt; and the &quot;point&quot; at (0, 5) is actually three points, corresponding to &lt;code&gt;X[0]&lt;/code&gt;, &lt;code&gt;X[2]&lt;/code&gt;, and &lt;code&gt;X[3]&lt;/code&gt;. The point at (5, 5) is the last point in your &lt;code&gt;X&lt;/code&gt; array. The data at (0, 4) and (0, 5) belong to one cluster, and the point at (5, 5) is considered noise (plotted in black).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue here seems to be that you're trying to run the &lt;code&gt;DBSCAN&lt;/code&gt; algorithm on a dataset containing 5 points, with at least 4 points required per cluster (the second argument to the &lt;code&gt;DBSCAN&lt;/code&gt; constructor). In the &lt;code&gt;sklearn&lt;/code&gt; example, the clustering algorithm is run on a dataset containing 750 points with three distinct centers. Try creating a larger &lt;code&gt;X&lt;/code&gt; dataset and running this code again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to remove the &lt;code&gt;plt.ylim([0,10])&lt;/code&gt; and &lt;code&gt;plt.xlim([0,10])&lt;/code&gt; lines from the code; they're making it a bit difficult to see the points on the edge of the plot! If you omit the &lt;code&gt;ylim&lt;/code&gt; and &lt;code&gt;xlim&lt;/code&gt; then &lt;code&gt;matplotlib&lt;/code&gt; will automatically determine the plot limits.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastActivityDate="2015-08-17T13:56:53.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6812" PostTypeId="2" ParentId="410" CreationDate="2015-08-17T14:13:39.080" Score="12" Body="&lt;p&gt;Selecting a learning rate is an example of a &quot;meta-problem&quot; known as &lt;em&gt;hyperparameter optimization&lt;/em&gt;. The best learning rate depends on the problem at hand, as well as on the architecture of the model being optimized, and even on the state of the model in the current optimization process! There are even software packages devoted to hyperparameter optimization such as &lt;a href=&quot;https://github.com/JasperSnoek/spearmint&quot;&gt;spearmint&lt;/a&gt; and &lt;a href=&quot;https://github.com/hyperopt/hyperopt&quot;&gt;hyperopt&lt;/a&gt; (just a couple of examples, there are many others!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from full-scale hyperparameter optimization, I wanted to mention one technique that's quite common for selecting learning rates that hasn't been mentioned so far. &lt;a href=&quot;https://en.wikipedia.org/wiki/Simulated_annealing&quot;&gt;Simulated annealing&lt;/a&gt; is a technique for optimizing a model whereby one starts with a large learning rate and gradually reduces the learning rate as optimization progresses. Generally you optimize your model with a large learning rate (0.1 or so), and then progressively reduce this rate, often by an order of magnitude (so to 0.01, then 0.001, 0.0001, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be combined with &lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot;&gt;early stopping&lt;/a&gt; to optimize the model with one learning rate as long as progress is being made, then switch to a smaller learning rate once progress appears to slow. The larger learning rates appear to help the model locate regions of general, large-scale optima, while smaller rates help the model focus on one particular local optimum.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastActivityDate="2015-08-17T14:13:39.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6813" PostTypeId="2" ParentId="731" CreationDate="2015-08-17T15:09:24.390" Score="15" Body="&lt;p&gt;While ffriend's answer gives some excellent pointers for learning more about how neural networks can be (extremely) difficult to tune properly, I thought it might be helpful to list a couple specific techniques that are currently used in top-performing classification architectures in the neural network literature.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Rectified linear activations&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The first thing that might help in your case is to switch your model's activation function from the &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_function&quot;&gt;logistic sigmoid&lt;/a&gt; -- $f(z) = \left(1 + e^{-z}\right)^{-1}$ -- to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)&quot;&gt;rectified linear (aka relu)&lt;/a&gt; -- $f(z) = \max(0, z)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relu activation has two big advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;its output is a true zero (not just a small value close to zero) for $z \le 0$ and&lt;/li&gt;&#xA;&lt;li&gt;its derivative is constant, either 0 for $z \le 0$ or 1 for $z &amp;gt; 0$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A network of relu units basically acts like an ensemble of exponentially many linear networks, because units that receive input $z \le 0$ are essentially &quot;off&quot; (their output is 0), while units that receive input $z &amp;gt; 0$ collapse into a single linear model for that input. Also the constant derivatives are important because a deep network with relu activations  tends to avoid the &lt;a href=&quot;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&quot;&gt;vanishing gradient problem&lt;/a&gt; and can be trained without layerwise pretraining.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &quot;Deep Sparse Rectifier Neural Networks&quot; by Glorot, Bordes, &amp;amp; Bengio (&lt;a href=&quot;http://jmlr.csail.mit.edu/proceedings/papers/v15/glorot11a/glorot11a.pdf&quot;&gt;http://jmlr.csail.mit.edu/proceedings/papers/v15/glorot11a/glorot11a.pdf&lt;/a&gt;) for a good paper about these topics.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Dropout&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Many research groups in the past few years have been advocating for the use of &quot;dropout&quot; in classifier networks to avoid overtraining. (See for example &quot;Dropout: A simple way to prevent neural networks from overfitting&quot; by Srivastava, Hinton, Krizhevsky, Sutskever, &amp;amp; Salakhutdinov &lt;a href=&quot;http://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot;&gt;http://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&lt;/a&gt;) In dropout, during training, some constant proportion of the units in a given layer are randomly set to 0 for each input that the network processes. This forces the units that aren't set to 0 to &quot;make up&quot; for the &quot;missing&quot; units. Dropout seems to be an extremely effective regularizer for neural network models in classification tasks. See a blog article about this at &lt;a href=&quot;http://fastml.com/regularizing-neural-networks-with-dropout-and-with-dropconnect/&quot;&gt;http://fastml.com/regularizing-neural-networks-with-dropout-and-with-dropconnect/&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastActivityDate="2015-08-17T15:09:24.390" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6814" PostTypeId="1" CreationDate="2015-08-17T20:34:15.330" Score="12" ViewCount="12420" Body="&lt;p&gt;I am working with the MovieLens10M dataset, predicting user ratings.  If I want to fairly evaluate my algorithm, how should I split my training v. test data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default, I believe the data is split into train v. test sets where 'test' contains movies previously unseen in the training set.  If my model requires each movie to have been seen at least once in the training set, how should I split my data?  Should I take all but N of each user's ratings for all the data and evaluate my performance on the held out NxUser_num  ratings?&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2022-08-11T14:57:18.013" Title="How to split train/test in recommender systems" Tags="|machine-learning|dataset|recommender-system|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6816" PostTypeId="1" CreationDate="2015-08-18T09:25:02.240" Score="3" ViewCount="260" Body="&lt;p&gt;Imagine that I have a set of short term time series data (signals) with different shapes and parameters. I would like to extract/select a set of features (parameters/functions/indeces) to identify each sequence and distinguish between them. Such features are simple features e.g. max and min value, mean, variance, start and end value, etc. or more complex ones e.g. entropy, no. peaks, power spectrum, etc. &#xA;I am looking for more &quot;representative&quot; features to express the signals and distinguish them. What features do you know (in signal processing or data mining fields) that can represent a parameter in signals?&lt;br&gt;&#xA;For example, in the following image, there are two signals with the same parameters such as mean, min-max values. But what are the features that differ them (for instance no. peaks, but generally what else?)&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Wb5a8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Wb5a8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10490" LastActivityDate="2015-09-18T00:36:40.417" Title="What are the features to distinguish two short term signals (time series)" Tags="|data-mining|time-series|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6817" PostTypeId="1" AcceptedAnswerId="6822" CreationDate="2015-08-18T09:34:51.143" Score="3" ViewCount="373" Body="&lt;p&gt;I have the following curve as the result of a azure machine learning classification experiment.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XcyS6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XcyS6.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;This obviously shows a lot of false negatives, but I'm struggling to interpret this, as my data has a high number of negatives and a low number of positives (I'm doing a binary classification).&lt;/p&gt;&#xA;&lt;p&gt;I am interpreting this as saying my data is almost entirely made up of positive values, with the majority being identified as negative. I believe that the false negatives on the chart are actually true negatives. Am I reading the chart correctly?&lt;/p&gt;&#xA;" OwnerUserId="12314" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-08-19T11:24:48.323" Title="Clasification - ROC Curve with very high number of false negatives" Tags="|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6818" PostTypeId="2" ParentId="6773" CreationDate="2015-08-18T10:11:05.097" Score="3" Body="&lt;p&gt;Another way is simply with the &quot;table&quot; function. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ids&amp;lt;-c(1,3,1,2)&#xA;counts&amp;lt;-data.frame(table(ids))&#xA;counts&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12316" LastActivityDate="2015-08-18T10:11:05.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6819" PostTypeId="2" ParentId="6783" CreationDate="2015-08-18T18:39:45.197" Score="2" Body="&lt;p&gt;If this is a U.S. model you can try County (FIPS) codes, which are coarser-grained areas than ZIP codes (there can be many ZIP codes in a county).  If bearing and distance worked for you, this should also work.  Your classifier would need to handle categorical data.  In R, this is pretty easy to do if you already have a ZIP code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(noncensus)&#xA;data(zip_codes)&#xA;my_dataframe['fips']&amp;lt;-zip_codes$fips[match(my_dataframe$zip, zip_codes$zip)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This assumes you already have a dataframe called &lt;code&gt;my_dataframe&lt;/code&gt; with ZIP codes as a column named 'zip'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a bonus you can use the &lt;a href=&quot;https://cran.r-project.org/web/packages/choroplethr/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;choroplethr&lt;/a&gt; package to create a nice visualization of some scalar value, for example, your item price:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(choroplethr)&#xA;df_to_plot&amp;lt;-data.frame()&#xA;df_to_plot['region']&amp;lt;-my_dataframe$fips&#xA;df_to_plot['value']&amp;lt;-my_dataframe$price&#xA;&#xA;county_choropleth(df_to_plot, title='price by location')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rtpvH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rtpvH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2015-08-19T03:37:17.200" LastActivityDate="2015-08-19T03:37:17.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6820" PostTypeId="2" ParentId="6773" CreationDate="2015-08-18T19:21:20.387" Score="1" Body="&lt;p&gt;This is similar to Jeremy's but using &lt;code&gt;dplyr&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;mytable &amp;lt;-&#xA;&quot;a    date        b         id&#xA; 1155 2010-05-02  2.7200    1&#xA; 1156 2010-05-05  2.6000    3&#xA; 1157 2010-05-08  2.6700    1&#xA; 1158 2010-05-11  3.5700    2&quot;&#xA;&#xA;mytable &amp;lt;- read.delim(textConnection(mytable), header=TRUE,  sep=&quot;&quot;)&#xA;mytable %&amp;gt;% count(id)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10253" LastActivityDate="2015-08-18T19:21:20.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6821" PostTypeId="2" ParentId="6816" CreationDate="2015-08-19T00:05:38.060" Score="3" Body="&lt;p&gt;The examples that you provided look similar to one another in duration and amplitude, so normalization might not be a problem.  But consider whether you will always be comparing the same duration and whether you want to compare the relative change in the time series or the absolute change.  The relative change can be measured by normalizing the series so that the first value (starting point) is set to 0.  Depending on your problem, this might be worth considering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, think about some sort of smoothing algorithm to get rid of some of the jitter in your data.  Examples include an &lt;a href=&quot;https://en.wikipedia.org/wiki/Moving_average&quot; rel=&quot;nofollow noreferrer&quot;&gt;Exponentially Weighted Moving Average&lt;/a&gt; or a second order &lt;a href=&quot;http://connor-johnson.com/2014/11/23/time-series-forecasting-in-python-and-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Holt-Winters EWMA&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The suggestions in the comments are good ones... both Fourier Transform via FFT or moving to a wavelet formulation are interesting, but they are still a one-to-one in terms of the amount of data in (in temporal space) being equal to the amount of data out (in frequency space).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest first including the raw data or a down-sampling of the raw data with some additional features like area under the curve, max, min.  I would stay away from procedures which amplify the jitter of your data (like derivatives) in favor of those which decrease the jitter (like integrals). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice option, which implicitly smooths would be to take a running integration of your data and then plot the running integral rather than the values themselves.  This wouldn't require apriori smoothing and is still deterministic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following this, I suggest doing a whole bunch of feature extraction/feature engineering and then trying a &lt;a href=&quot;https://stats.stackexchange.com/questions/17251/what-is-the-lasso-in-regression-analysis&quot;&gt;LASSO regression&lt;/a&gt; to pick out the most useful features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some feature engineering ideas include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fourier Transform via FFT&lt;/li&gt;&#xA;&lt;li&gt;Possibly only keeping the n (~10) largest Fourier Modes from above rather than the full set.&lt;/li&gt;&#xA;&lt;li&gt;Integrals of your data.&lt;/li&gt;&#xA;&lt;li&gt;Min.&lt;/li&gt;&#xA;&lt;li&gt;Max.&lt;/li&gt;&#xA;&lt;li&gt;Number of zero crossings.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Suggestions to look at the number of changes in the second derivative are tough with time series data due to its stochastic nature.  The slopes (derivatives) are very wild and depend largely on how much smoothing has been applied and how the data is being sampled.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-08-19T00:05:38.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6822" PostTypeId="2" ParentId="6817" CreationDate="2015-08-19T02:56:06.593" Score="3" Body="&lt;p&gt;Notice how the &lt;code&gt;precision&lt;/code&gt; is very high and all of the other metrics are very low.  Now look at the class balance of your problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$TP+FN=Actual Positive=31,245$$&#xA;$$TN+FP=Actual Negative=508$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So your data is heavily skewed toward positives.  To have gotten a model that is producing this poorly, I think you may have provided the model with the &lt;code&gt;precision&lt;/code&gt; as the cross validation metric.  The &lt;code&gt;precision&lt;/code&gt; is a very bad cross validation metric in this case since it will result in poor &lt;code&gt;accuracy&lt;/code&gt; and poor &lt;code&gt;recall&lt;/code&gt;.  &lt;code&gt;accuracy&lt;/code&gt; is also not a good metric as your model could classify everything as positive and get an accuracy of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$AC=\frac{31,245}{31,763}=.984$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For cases like this where the classes are grossly weighted toward one value, I suggest using the &lt;code&gt;F1-score&lt;/code&gt; as your cross validation metric.  The &lt;code&gt;F1-score&lt;/code&gt; is the &lt;code&gt;harmonic mean&lt;/code&gt; of &lt;code&gt;precision&lt;/code&gt; and &lt;code&gt;recall&lt;/code&gt; and hence balances these two factors nicely.  Wikipedia actually has a &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall#Definition_.28classification_context.29&quot; rel=&quot;nofollow&quot;&gt;very nice explanation of classification metrics here&lt;/a&gt; and &lt;a href=&quot;http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper is top notch&lt;/a&gt; if you even need to understand multi-class metrics and confusion matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-19T11:24:48.323" LastActivityDate="2015-08-19T11:24:48.323" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6823" PostTypeId="2" ParentId="6773" CreationDate="2015-08-19T04:31:09.787" Score="5" Body="&lt;p&gt;Using the &lt;code&gt;data.table&lt;/code&gt; structure (see the &lt;a href=&quot;https://github.com/Rdatatable/data.table/wiki&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt;),&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(data.table)&#xA;D &amp;lt;- data.table(x = c(1155, 1156, 1157, 1158),&#xA;                date = as.Date(c(&quot;2010-05-02&quot;, &quot;2010-05-05&quot;, &quot;2010-05-08&quot;, &quot;2010-05-11&quot;)),&#xA;                y = c(2.7200, 2.6000, 2.6700, 3.5700),&#xA;                id = c(1, 3, 1, 2))&#xA;counts &amp;lt;- D[, .(rowCount = .N), by = id]&#xA;counts&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will return&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;counts&#xA;##    id rowCount&#xA;## 1:  1        2&#xA;## 2:  3        1&#xA;## 3:  2        1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12328" LastEditorUserId="12384" LastEditDate="2015-08-27T07:42:42.883" LastActivityDate="2015-08-27T07:42:42.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6826" PostTypeId="2" ParentId="6797" CreationDate="2015-08-19T09:11:55.007" Score="0" Body="&lt;p&gt;I solve this task as follows: cross-validate train data in order to maximize F1-score then use probabilistic output and cost matrix for predicting a class for a new item. &#xA;Any suggestions are welcome.&lt;/p&gt;&#xA;" OwnerUserId="10694" LastActivityDate="2015-08-19T09:11:55.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6827" PostTypeId="1" AcceptedAnswerId="8020" CreationDate="2015-08-19T11:09:40.287" Score="1" ViewCount="257" Body="&lt;p&gt;I am currently collecting mobile device data about user location and the time at which an app is being used for a cohort of users and apps.  I am trying to predict which apps are likely to be used at a given time by a given user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which model should i use in order to predict the apps that are likely to be used by a particular user? For example, should I collect data for 5 days a week then use the 3 days for training and then the other 2 days for testing? Given location, time of day, and day, what model should I use to predict which app is likely to be used?&lt;/p&gt;&#xA;" OwnerUserId="12333" LastEditorUserId="9420" LastEditDate="2015-08-19T18:39:07.883" LastActivityDate="2015-10-06T11:15:40.863" Title="Machine learning to predict apps (recomendation)" Tags="|machine-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6828" PostTypeId="1" CreationDate="2015-08-19T11:22:29.927" Score="0" ViewCount="113" Body="&lt;p&gt;I have one Java Client which Reads 4 to 5 GBs of data and uses DLL file in order to process it and give O/P back to the JAVA client. &#xA;Can anyone tell me if this can be implemented using Apache Spark or not ?&#xA;And if yes can you back you Answer by some examples or documentation Links ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have knowledge about &lt;strong&gt;JNI&lt;/strong&gt; and how to do this in windows environement. &#xA;What I'm actually looking for is How do you implement such thing in Apache Spark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As all methods and processing is done via DLL files and JAVA Client is just having represntational logic so any documentation is helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using windows development env. but gonna use Ubuntu 14.04 for Apache Spark implemetation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; after Hold&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is really intending to ask where to find such documentation or help with which we can migrate such application smoothly. &lt;/p&gt;&#xA;" OwnerUserId="10210" LastEditorUserId="10210" LastEditDate="2015-08-19T19:10:23.623" LastActivityDate="2015-08-19T19:10:23.623" Title="How to migrate application which has backend running using DLL methods and front end in JAVA to Apache spark?" Tags="|apache-spark|java|" AnswerCount="1" CommentCount="2" ClosedDate="2015-08-19T18:40:11.693" ContentLicense="CC BY-SA 3.0" />
  <row Id="6829" PostTypeId="2" ParentId="6828" CreationDate="2015-08-19T11:46:46.080" Score="1" Body="&lt;p&gt;You will need to use the Java Native Interface (JNI) for this. Refer to a Stack Overflow question for &lt;a href=&quot;https://stackoverflow.com/a/5963294/2074605&quot;&gt;how to call DLL methods via Java over JNI&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10189" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-08-19T11:46:46.080" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6830" PostTypeId="1" AcceptedAnswerId="6833" CreationDate="2015-08-19T13:06:42.750" Score="4" ViewCount="4540" Body="&lt;p&gt;I have a set of user objects that I want to group using a $k$-means function from their quiz answers. Each quiz question had predefined answers with letter values &lt;em&gt;&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;&lt;/em&gt;. If a user answers the &lt;em&gt;question #1&lt;/em&gt; with letter &lt;em&gt;&quot;b&quot;&lt;/em&gt;, I put this answer into vector $(0, 1, 0, 0)$.  The $k$-means function I have to use takes a two-dimensional array of numbers as an input vector (in this case array[user][question]), and I can't figure out how to use it, because, instead of a number value representing a user's answer to question, I have a vector input.&#xA;How can I convert my vector values to numbers so that I can use my $k$-means function?&lt;/p&gt;&#xA;" OwnerUserId="12335" LastEditorUserId="13413" LastEditDate="2015-11-24T01:52:00.027" LastActivityDate="2015-11-24T01:52:00.027" Title="How to convert vector values to fit k-means algorithm function?" Tags="|clustering|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6831" PostTypeId="1" AcceptedAnswerId="6839" CreationDate="2015-08-19T15:21:53.017" Score="0" ViewCount="69" Body="&lt;p&gt;I have collected Temp vs. Time data for a controlled environment. I performed three tests, with the thermocouple in the same location for all three tests. I now have three Temp. vs. Time curves for this location. The graphs are shown here: &lt;a href=&quot;https://i.stack.imgur.com/3vCzj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3vCzj.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking to generate an empirical model based on the Temp. vs. Time at this location that can predict the temperature at this location over time. How should I approach this? My initial approach was to identify the local minima and maxima, and average the three points together (for each maximum and minimum). And then correlate that average to a sequence of time, for example, 30C between 750 and 1250s, or something like that. I really don't think this is an efficient approach...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you all think would be an efficient approach to this?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2015-08-20T06:08:22.557" Title="Developing an Empirical Model for Non-Linear Data" Tags="|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6832" PostTypeId="2" ParentId="6830" CreationDate="2015-08-19T15:56:45.850" Score="0" Body="&lt;p&gt;A two dimensional array is a list of vectors, so &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{{userid1,1a,1b,1c,1d,2a,2b,2c,2d,...,na,nb,nc,nd}&#xA;{userid2,1a,1b,1c,1d,2a,2b,2c,2d,...,na,nb,nc,nd},&#xA;...,&#xA;{useridk,1a,1b,1c,1d,2a,2b,2c,2d,...,na,nb,nc,nd}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;would be a suitable input for a test with &lt;code&gt;n&lt;/code&gt; questions and &lt;code&gt;k&lt;/code&gt; contestants, where &lt;code&gt;1a&lt;/code&gt; represents the response &lt;code&gt;a&lt;/code&gt; for question one.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="13123" LastEditDate="2015-11-22T11:11:28.447" LastActivityDate="2015-11-22T11:11:28.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6833" PostTypeId="2" ParentId="6830" CreationDate="2015-08-19T16:14:21.407" Score="3" Body="&lt;p&gt;You are 95% there, you just have one hangup...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vectorization that you are doing is alternatively known as binarization or &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow&quot;&gt;one-hot encoding&lt;/a&gt;.  The only thing you need to do now is break apart all of those vectors and think of them as individual features.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So instead of thinking of the question one vector as $(0,0,1,0)$ and the question two vector as $(0,1,0,0)$, you can now think of them as individual features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-      q1,        q2&#xA;-      (a,b,c,d), (a,b,c,d)&#xA;user1  (0,0,1,0), (0,1,0,0)&#xA;user2  (1,0,0,0), (0,0,0,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Becomes this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-      q1a,q1b,q1c,q1d,q2a,q2b,q2c,q2d&#xA;user1  0   0   1   0   0   1   0   0&#xA;user2  1   0   0   0   0   0   0   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And you can think of each one of those binary features as an orthogonal  dimension in your data that lies in a 8-dimensional space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-08-19T16:14:21.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6834" PostTypeId="1" CreationDate="2015-08-19T16:25:57.530" Score="5" ViewCount="889" Body="&lt;p&gt;I am looking for an algorithm that clusters a directed graph into a set of clusters that form a directed acyclic graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, given:&#xA;nodes: {A, B, C, D}&#xA;with edges: {(A,B), (B,A), (A,C), (C,D), (D,C)}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A valid output would be: {{A,B},{C,D})&#xA;An invalid output would be: {{A,C},{B,D}} (since the node edges imply a cycle between the clusters)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any pointers or suggestions!&lt;/p&gt;&#xA;" OwnerUserId="12343" LastActivityDate="2016-01-02T02:48:38.810" Title="Cluster directed graph into DAG" Tags="|clustering|graphs|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6835" PostTypeId="1" CreationDate="2015-08-19T18:15:00.463" Score="6" ViewCount="3127" Body="&lt;p&gt;I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this instance, I'm interested in results on the MovieLens10M dataset.  It seems to be referenced fairly frequently in literature, often using RMSE, but I have had trouble determining what might be considered state-of-the-art.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did find this site, but it is only for the 100K dataset and is far from inclusive:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results&quot; rel=&quot;noreferrer&quot;&gt;http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-09-20T11:16:54.937" Title="Bechmark for Movielens" Tags="|machine-learning|dataset|recommender-system|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6837" PostTypeId="1" CreationDate="2015-08-19T20:58:44.523" Score="1" ViewCount="534" Body="&lt;p&gt;In web analytics, one of the main issues is to find the &lt;em&gt;optimal path of navigation&lt;/em&gt;. &lt;strong&gt;Question:&lt;/strong&gt; Is there any literature in &lt;em&gt;graph theory&lt;/em&gt; area, which is focused on studying optimal website navigation path?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My Comment to Tom was pretty large. So putting it here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in network theory there's a concept of main node (forgot the exact terminology). This node is most important because it has most connection to other pages. Home page is a very simple kind of main node. Also different set of visitors might have different main node. So, if we could capture what is the main node, then if we directly drop the visitor to this page or some other page, then the conversion might increase. Because the visitor who will come to this main page, will able to easily navigate to his required page. And then able to get what he is looking for. Another method people use for shortest path is MCMC methodology. But I'll prefer network method. Figuring out the entire method is pretty cumbersome and I don't have much time. So I was wondering if somebody wrote some journal paper on it, I can refer that. &lt;/p&gt;&#xA;" OwnerUserId="10810" LastEditorUserId="10810" LastEditDate="2015-08-23T19:39:01.667" LastActivityDate="2015-08-23T19:39:01.667" Title="Website navigation path using graph theory" Tags="|graphs|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6838" PostTypeId="1" AcceptedAnswerId="6855" CreationDate="2015-08-20T04:16:43.303" Score="41" ViewCount="104025" Body="&lt;p&gt;&lt;strong&gt;When would one use &lt;code&gt;Random Forest&lt;/code&gt; over &lt;code&gt;SVM&lt;/code&gt; and vice versa?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that &lt;code&gt;cross-validation&lt;/code&gt; and model comparison is an important aspect of choosing a model, but here I would like to learn more about rules of thumb and heuristics of the two methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please explain the subtleties, strengths, and weaknesses of the classifiers as well as problems, which are best suited to each of them?&lt;/p&gt;&#xA;" OwnerUserId="12350" LastEditorUserId="26686" LastEditDate="2017-04-26T16:24:32.367" LastActivityDate="2022-06-11T01:09:55.867" Title="When to use Random Forest over SVM and vice versa?" Tags="|machine-learning|classification|random-forest|svm|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6839" PostTypeId="2" ParentId="6831" CreationDate="2015-08-20T06:08:22.557" Score="1" Body="&lt;p&gt;A couple of thoughts...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First as a (former) physicist:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a scientist/physicist I would likely never be satisfied to analyze the data that you have presented because there are some obvious issues with it.  I was initially thinking that it was way to schizophrenic, but then I noticed that you referred to the time units as seconds, in which case this is a pretty slowly varying time series.  Are the units really seconds?  If they happen to be milliseconds or microseconds then you might want to think about a probe that is less noisy (i.e. a larger probe with more of a temperature sink) and check weather you are seeing the effects of ground loop feedback in your system or just EMF interference from your electrical system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, there are clear patterns in the data and these do not line up very well, so I would rerun the experiment with things timed much more accurately.  For instance, the event that occurs midway through the experiment that drops the temperature (looks like an icicle or dagger) occurs 5-8 minutes apart in comparing the dark blue experiment to the purple experiment.  It would be great if these could be synced better.  The point being that if you can't sync them better, then the noise of the time series renders most phenomenon apart from a flat line, almost useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now as a data scientist:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This data is really noisy or at least fluctuates significantly!  You should stay away from averaging minima and maxima as you suggest in your post.  extrema are very sensitive to noise where as you are trying to reduce the noise in your system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should perhaps think about applying some sort of smoothing method to reduce the jitter in your data.  Perhaps a second-order exponentially weighted moving average like &lt;a href=&quot;http://connor-johnson.com/2014/11/23/time-series-forecasting-in-python-and-r/&quot; rel=&quot;nofollow&quot;&gt;Holt-Winters&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you could probably just average the three signals to produce a mean signal.  Means are more suseptable to outliers than medians,so the median signal is also an option. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other options include separating the trend from the general behavior using some type of &lt;a href=&quot;https://www.stat.auckland.ac.nz/~wild/ChanceEnc/Ch14.pdf&quot; rel=&quot;nofollow&quot;&gt;de-seasoning&lt;/a&gt; or &lt;a href=&quot;http://www.research.att.com/techdocs/TD_100381.pdf&quot; rel=&quot;nofollow&quot;&gt;double ARIMA&lt;/a&gt; procedure.  But, it seems like these are too extreme given that the dagger/icicle event looks significant and you don't want to transform it away.  So I would only use these if there is some type of periodicity to your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-08-20T06:08:22.557" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6840" PostTypeId="1" CreationDate="2015-08-20T09:12:13.357" Score="1" ViewCount="209" Body="&lt;p&gt;I have a dataset of what apps users downloaded and I try to estimate these users' gender based on what apps they downloaded, using machine learning algorithm. However, what kind of features of the apps should I focus on? As far as I know, app category plays a big role. Do Google Play and Apple App Store estimate users' gender based on what apps they download?&lt;/p&gt;&#xA;" OwnerUserId="12355" LastActivityDate="2015-08-21T08:21:09.977" Title="How to estimate a user's gender based on what apps the user download?" Tags="|machine-learning|data-mining|bigdata|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6841" PostTypeId="1" AcceptedAnswerId="6881" CreationDate="2015-08-20T09:49:51.230" Score="1" ViewCount="669" Body="&lt;p&gt;I have a data frame of the following format:&#xA;&lt;br&gt;Symbol  Date    Time    Profit&#xA;&lt;br&gt;$BANKNIFTY	4/1/2010	9:55:00	-1.18%&#xA;&amp;lt;br&amp;gt;$BANKNIFTY   4/1/2010    12:30:00    -2.84%&#xA;&lt;br&gt;$BANKNIFTY	4/1/2010	12:45:00	7.17%&#xA;&amp;lt;br&amp;gt;$BANKNIFTY   5/1/2010    11:40:00    -7.11%&#xA;&lt;br&gt;ZEEL    26/6/2012   13:50:00    24.75%&#xA;&lt;br&gt;ZEEL    27/6/2012   15:15:00    -1.90%&#xA;&lt;br&gt;ZEEL    28/6/2012   9:45:00 37.58%&#xA;&lt;br&gt;ZEEL    28/6/2012   14:55:00    23.95%&#xA;&lt;br&gt;ZEEL    29/6/2012   14:20:00    -4.65%&#xA;&lt;br&gt;ZEEL    29/6/2012   14:30:00    -6.01%&#xA;&lt;br&gt;ZEEL    29/6/2012   14:55:00    -12.23%&#xA;&lt;br&gt;ZEEL    29/6/2012   15:15:00    35.13%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to achieve is convert that data frame into a data frame which has dates for row names, symbol names for columns and sum of percentage profit for each day. Like in the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;Date    BankNifty   ZEEL&#xA;&lt;br&gt;4/1/2010    3.15%   0&#xA;&lt;br&gt;5/1/2010    -7.11%  0&#xA;&lt;br&gt;26/6/2012   0   24.75%&#xA;&lt;br&gt;27/6/2012   0   -1.90%&#xA;&lt;br&gt;28/6/2012   0   61.53%&#xA;&lt;br&gt;29/6/2012   0   12.24%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achieve that in R? &lt;code&gt;dplyr&lt;/code&gt; mutation or some apply function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a beginner in R programming. &#xA;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data in R is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(Symbol = structure(c(1L, 1L, 1L, 1L, 2L, 2L, 2L, &#xA;2L, 2L, 2L, 2L, 2L), .Label = c(&quot;BANKNIFTY&quot;, &quot;ZEEL&quot;), class = &quot;factor&quot;), &#xA;    Date = structure(c(5L, 5L, 5L, 6L, 1L, 2L, 3L, 3L, 4L, 4L, &#xA;    4L, 4L), .Label = c(&quot;26/6/2012&quot;, &quot;27/6/2012&quot;, &quot;28/6/2012&quot;, &#xA;    &quot;29/6/2012&quot;, &quot;4/1/2010&quot;, &quot;5/1/2010&quot;), class = &quot;factor&quot;), &#xA;    Time = structure(c(10L, 2L, 3L, 1L, 4L, 8L, 9L, 7L, 5L, 6L, &#xA;    7L, 8L), .Label = c(&quot;11:40:00&quot;, &quot;12:30:00&quot;, &quot;12:45:00&quot;, &quot;13:50:00&quot;, &#xA;    &quot;14:20:00&quot;, &quot;14:30:00&quot;, &quot;14:55:00&quot;, &quot;15:15:00&quot;, &quot;9:45:00&quot;, &#xA;    &quot;9:55:00&quot;), class = &quot;factor&quot;), Profit = structure(c(1L, 4L, &#xA;    12L, 7L, 9L, 2L, 11L, 8L, 5L, 6L, 3L, 10L), .Label = c(&quot;-1.18%&quot;, &#xA;    &quot;-1.90%&quot;, &quot;-12.23%&quot;, &quot;-2.84%&quot;, &quot;-4.65%&quot;, &quot;-6.01%&quot;, &quot;-7.11%&quot;, &#xA;    &quot;23.95%&quot;, &quot;24.75%&quot;, &quot;35.13%&quot;, &quot;37.58%&quot;, &quot;7.17%&quot;), class = &quot;factor&quot;)), .Names = c(&quot;Symbol&quot;, &#xA;&quot;Date&quot;, &quot;Time&quot;, &quot;Profit&quot;), class = &quot;data.frame&quot;, row.names = c(NA, &#xA;-12L))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12351" LastEditorUserId="12384" LastEditDate="2015-08-25T04:49:19.930" LastActivityDate="2015-08-25T04:49:19.930" Title="Data frame mutation in R" Tags="|r|data-cleaning|data|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6842" PostTypeId="1" CreationDate="2015-08-20T09:53:43.280" Score="7" ViewCount="6242" Body="&lt;p&gt;I am training and predicting on the same data-set, but I want to perform 10-fold cross-validation and predict on the left out fold and thus predict on the whole data set. How can I do this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The libraries which I am using are: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cross_validation&#xA;import xgboost as xgb&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12356" LastEditorUserId="303" LastEditDate="2020-03-14T08:46:50.780" LastActivityDate="2020-08-03T08:57:37.123" Title="Is there a way of performing stratified cross validation using xgboost module in python?" Tags="|python|scikit-learn|xgboost|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6843" PostTypeId="2" ParentId="6840" CreationDate="2015-08-20T09:59:59.090" Score="2" Body="&lt;p&gt;If you have data on how many downloads have been done for each category, you can use that data to perform logistic regression, where your target variable would be 0/1 classification of male/female. This will form your benchmark result and then you can improve upon the results. Especially look for feature selection, because there might be categories of apps which have nothing to do with target. You should also try bi variate profiling of variables with target to see if there is good correlation with target. &#xA;Try using some advanced techniques once you have done some data exploration. &#xA;KNN, Random Forest and Neural network for classification might give good results. Also cart might work fantastically here. &lt;/p&gt;&#xA;" OwnerUserId="12356" LastActivityDate="2015-08-20T09:59:59.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6844" PostTypeId="2" ParentId="6133" CreationDate="2015-08-20T10:52:36.830" Score="5" Body="&lt;p&gt;I'm working on a similar classification problem using Quinlans &lt;code&gt;C5.0&lt;/code&gt; and in my experience there are couple of tweaks inside that might help you. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First, I believe using such big number (trials=500) for boosting is&#xA;affecting the overfitting, since the model will repeatedly try and&#xA;improve on the misclassified instances in every iteration. Try using&#xA;a smaller boosting size, or even turning it off - it might help with&#xA;the overfitting problem.&lt;/li&gt;&#xA;&lt;li&gt;There are a couple of other parameters you might find useful. The minimal leaf size (&lt;code&gt;minCases&lt;/code&gt; in &lt;code&gt;C5.0Control()&lt;/code&gt; is something to you want to keep as big as possible when avoiding overfitting issues. &lt;/li&gt;&#xA;&lt;li&gt;Another one is the confidence factor (&lt;code&gt;CF&lt;/code&gt;) which lets you control&#xA;the severity of pruning of the tree, meaning lower factor levels will likely prune away the leaves which overspecify the classification. &lt;/li&gt;&#xA;&lt;li&gt;Finally, you might try using the &lt;code&gt;fuzzyThreshold&lt;/code&gt; control option which lets you 'soften' sharp thresholds with an inbetween grey area. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are all very well documented in the &lt;a href=&quot;https://cran.r-project.org/web/packages/C50/C50.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;C5.0&lt;/code&gt;&lt;/a&gt; vignette as well as on the &lt;a href=&quot;https://www.rulequest.com/see5-unix.html&quot; rel=&quot;nofollow&quot;&gt;informal tutorial page&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps you a bit. &lt;/p&gt;&#xA;" OwnerUserId="12341" LastEditorUserId="9420" LastEditDate="2015-08-21T07:33:02.843" LastActivityDate="2015-08-21T07:33:02.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6846" PostTypeId="2" ParentId="6838" CreationDate="2015-08-20T12:15:13.117" Score="9" Body="&lt;p&gt;SVM models perform better on sparse data than does trees in general.  For example in document classification you may have thousands, even tens of thousands of features and in any given document vector only a small fraction of these features may have a value greater than zero.  There are probably other differences between them, but this is what I found for my problems.  &lt;/p&gt;&#xA;" OwnerUserId="12360" LastActivityDate="2015-08-20T12:15:13.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6847" PostTypeId="1" AcceptedAnswerId="6849" CreationDate="2015-08-20T14:56:40.847" Score="2" ViewCount="75" Body="&lt;p&gt;I need an advice. I can resume my problem like that :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some travels in a database, for example :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Person1 travelled from CityA to CityB on Date1&#xA;Person1 travelled from CityB to CityC on Date2&#xA;Person2 travelled from CityB to CityD on Date3&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can consider that these cities are in the complete graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, according to all the travels in the database, I would like to know where a PersonX is likely to go. I can know when he come from (or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if I should use machine learning, data-mining or graph theory.&lt;/p&gt;&#xA;" OwnerUserId="12362" LastActivityDate="2015-08-20T17:52:15.133" Title="Estimating destination according to previous data" Tags="|machine-learning|data-mining|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6848" PostTypeId="1" CreationDate="2015-08-20T17:42:23.987" Score="4" ViewCount="1203" Body="&lt;p&gt;&lt;strong&gt;(I've posted this question on CV, but I feel it would also be great to hear from experts in DS community.)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a PhD student starting to think about dissertation topics, I am particularly interested in high-dimensional statistical learning. I wish to find some research review/survey/papers (or webpages, blogs, whatever...) about the state-of-the-art research in this research area, but there seems limited resources I can obtain. My first question is then,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Could you describe some current interesting research topics in&#xA;high-dimensional statistics? If you can list any relevant resources&#xA;(papers, webpages, etc.), that would be really helpful.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In addition, I've noticed that high-dimensional statistical learning is closed related with machine learning research. For example, the idea of penalized regularization in high-dimensional statistics was used in machine learning domain, like support vector machine, boosting tree, (sparse) additive models, etc. My question is,&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;what are good research papers about the interplay of&#xA;high-dimensional statistics and machine learning?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Last, since high-dimensional statistics was really motivated by genetic research (like gene-expression analysis, or genome-wide association studies), most of applications in high-dimensional research are devoted in that area.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there any successful applications of high-dimensional statistics&#xA;in areas other than genetics, particularly say, image/text mining,&#xA;recommendation, etc, areas where machine learning techniques have&#xA;long been used?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;A new question to machine learning researchers/practitioners&lt;/strong&gt;: I might be wrong, but as I understand, most machine learning algorithms are designed for low-dimensional problems (or at least the number of features is smaller that number of observations). Are there any successful applications of machine learning techniques for modeling high-dimensional data? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any resources/comments are highly appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="3465" LastActivityDate="2015-08-20T21:30:19.697" Title="Research in high-dimensional statistics vs. machine learning?" Tags="|machine-learning|statistics|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6849" PostTypeId="2" ParentId="6847" CreationDate="2015-08-20T17:52:15.133" Score="2" Body="&lt;p&gt;This is a spatio-temporal clustering problem that is likely best solved with a Markov model. You could reasonable group this into machine learning or data mining. Develop your model using machine learning and then (the data mining part) leverage those pattern recognition techniques (that have been developed in machine learning). I think there are at least one or two threads on this over at Cross-Validated that go into more detail.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a couple of papers to look at if you are just getting started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://link.springer.com/article/10.1007/s00779-003-0240-0&quot; rel=&quot;nofollow&quot;&gt;Using GPS to learn significant locations and predict movement across multiple users&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://lbsn2012.cmuchimps.org/papers/Paper16_Mathew.pdf&quot; rel=&quot;nofollow&quot;&gt;Predicting Future Locations with Hidden Markov Models&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11021" LastActivityDate="2015-08-20T17:52:15.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6850" PostTypeId="2" ParentId="6842" CreationDate="2015-08-20T19:43:11.243" Score="2" Body="&lt;p&gt;xgboost comes with an own cv method, see an &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/demo/guide-python/cross_validation.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;example here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3132" LastEditorUserId="98307" LastEditDate="2020-08-03T08:57:37.123" LastActivityDate="2020-08-03T08:57:37.123" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6851" PostTypeId="1" AcceptedAnswerId="6853" CreationDate="2015-08-20T20:46:07.740" Score="6" ViewCount="473" Body="&lt;p&gt;I'm a total amateur as far as data science goes, and I'm trying to figure out a way to do some string comparison on a large dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've a Google BigQuery table storing merchant transactions, but the store names are all over the board.  For example, there can be 'Wal-Mart Super Center' and 'Wal-Mart SC #1234', or 'McDonalds F2222' and 'McDonalds #321'.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need to do is group ALL 'Wal-mart' and 'McDonalds' and whatever else.  My first approach was doing a recursive reg-ex check, but that took forever and eventually timed-out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the best approach for doing that with a table of 20 million+ rows?  I'm open to trying out any technology that would fit this job. &lt;/p&gt;&#xA;" OwnerUserId="12364" LastEditorUserId="843" LastEditDate="2015-10-18T12:38:41.357" LastActivityDate="2015-10-18T12:38:41.357" Title="What's an efficient way to compare and group millions of store names?" Tags="|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6852" PostTypeId="2" ParentId="6848" CreationDate="2015-08-20T21:30:19.697" Score="2" Body="&lt;p&gt;This makes me think right off the bat of &lt;a href=&quot;https://en.wikipedia.org/wiki/Gaussian_process&quot; rel=&quot;nofollow&quot;&gt;Gaussian processes&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The GP was initially developed by geostatisticians in the 1970s as a model for the distribution of resources over a geographic area; this process is known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Kriging&quot; rel=&quot;nofollow&quot;&gt;kriging&lt;/a&gt; (or also Gaussian process regression). But more recently GPs have been used pretty widely in the movement/control/robotics area to model movements and trajectories of robotic limbs etc. over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistically, GPs are treated as priors over vectors of infinite dimension, such that the distribution of any subset of the values of a vector is Gaussian distributed. There's an &lt;a href=&quot;http://www.gaussianprocess.org/gpml/chapters/&quot; rel=&quot;nofollow&quot;&gt;online book&lt;/a&gt; if you want to read more!&lt;/p&gt;&#xA;" OwnerUserId="4647" LastActivityDate="2015-08-20T21:30:19.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6853" PostTypeId="2" ParentId="6851" CreationDate="2015-08-20T22:19:26.357" Score="7" Body="&lt;p&gt;This is an &lt;a href=&quot;http://www.umiacs.umd.edu/~getoor/Tutorials/ER_VLDB2012.pdf&quot;&gt;entity resolution&lt;/a&gt; aka &lt;a href=&quot;https://en.wikipedia.org/wiki/Record_linkage&quot;&gt;record linkage&lt;/a&gt; aka &lt;a href=&quot;http://www.springer.com/us/book/9783642311635&quot;&gt;data matching&lt;/a&gt; problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would solve this by removing all of the non-alphabetical characters including numbers, casting into all uppercase and then employing a hierarchical match.  First match up the exact cases and then move to a Levenshtein scoring between the fields.  Make some sort of a decision about how large you will allow the Levenshtein or normalized Levenshtein score to get before you declare something a non-match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assign every row an id and when you have a match, reassign the lower of the IDs to both members of the match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot;&gt;Levenshtein distance&lt;/a&gt; algorithm is simple but brilliant (&lt;a href=&quot;http://hetland.org/coding/python/levenshtein.py&quot;&gt;taken from here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def levenshtein(a,b):&#xA;    &quot;Calculates the Levenshtein distance between a and b.&quot;&#xA;    n, m = len(a), len(b)&#xA;    if n &amp;gt; m:&#xA;        # Make sure n &amp;lt;= m, to use O(min(n,m)) space&#xA;        a,b = b,a&#xA;        n,m = m,n&#xA;&#xA;    current = range(n+1)&#xA;    for i in range(1,m+1):&#xA;        previous, current = current, [i]+[0]*n&#xA;        for j in range(1,n+1):&#xA;            add, delete = previous[j]+1, current[j-1]+1&#xA;            change = previous[j-1]&#xA;            if a[j-1] != b[i-1]:&#xA;                change = change + 1&#xA;            current[j] = min(add, delete, change)&#xA;&#xA;    return current[n]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3642311636&quot;&gt;Data Matching&lt;/a&gt; book is a good resource and is free for seven days on Amazon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nominally, this is an $n^2$ algorithm without exploiting some sorting efficiencies, so I would expect to have to use multiple cores on $2\times10^7$ rows.  But this should run just fine on an 8 core &lt;a href=&quot;https://aws.amazon.com/&quot;&gt;AWS instance&lt;/a&gt;.  It will eventually finish on a single core, but might take several hours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-20T22:26:07.967" LastActivityDate="2015-08-20T22:26:07.967" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6854" PostTypeId="1" AcceptedAnswerId="6857" CreationDate="2015-08-20T22:21:12.060" Score="4" ViewCount="1375" Body="&lt;p&gt;As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, but it's made me wonder whether there are best practices out there for this sort of task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll describe what I've got, and maybe that will help provide some recommendations. So, I have an experiment that requires subjects to use their vision and move their body to complete a task. Each task is one trial, and each subject performs multiple trials to complete the experiment. During a trial I record the movement and the eye tracker (~200 channels) at regularly sampled time points (~100Hz). I store these in a CSV file (one file per trial), with one row per time point, and one column per variable (e.g., left-fingertip-x, left-fingertip-y, left-fingertip-z, etc. for the mocap, and left-eye-x, left-eye-y for the eyes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Associated with each trial is some metadata such as the experimental condition of the trial (e.g., how fast a target in the trial is moving, say). I store these values in the CSV filename itself, using a &quot;key=value&quot; sort of syntax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this works well enough for my purposes, it's really ad-hoc! I'd like to get a sense of whether other people have solved problems like this, and, if so, how?&lt;/p&gt;&#xA;" OwnerUserId="4647" LastEditorUserId="4647" LastEditDate="2015-08-21T14:26:03.203" LastActivityDate="2017-04-16T19:30:56.737" Title="Recommendations for storing time series data" Tags="|time-series|data|experiments|csv|metadata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6855" PostTypeId="2" ParentId="6838" CreationDate="2015-08-21T04:36:06.367" Score="47" Body="&lt;p&gt;I would say, the choice depends very much on what data you have and what is your purpose. A few &quot;rules of thumb&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest is intrinsically suited for multiclass problems, while SVM is intrinsically two-class. For multiclass problem you will need to reduce it into multiple binary classification problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest works well with a mixture of numerical and categorical features. When features are on the various scales, it is also fine. Roughly speaking, with Random Forest you can use data as they are. SVM maximizes the &quot;margin&quot; and thus relies on the concept of &quot;distance&quot; between different points. It is up to you to decide if &quot;distance&quot; is meaningful. As a consequence, one-hot encoding for categorical features is a must-do. Further, min-max or other scaling is highly recommended at preprocessing step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have data with $n$ points and $m$ features, an intermediate step in SVM is constructing an $n\times n$ matrix (think about memory requirements for storage) by calculating $n^2$ dot products (computational complexity).&#xA;Therefore, as a rule of thumb, SVM is hardly scalable beyond 10^5 points.&#xA;Large number of features (homogeneous features with meaningful distance, pixel of image would be a perfect example) is generally not a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a classification problem Random Forest gives you probability of belonging to class. SVM gives you distance to the boundary, you still need to convert it to probability somehow if you need probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those problems, where SVM applies, it generally performs better than Random Forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVM gives you &quot;support vectors&quot;, that is points in each class closest to the boundary between classes. They may be of interest by themselves for interpretation. &lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-08-21T04:36:06.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6856" PostTypeId="2" ParentId="6840" CreationDate="2015-08-21T08:21:09.977" Score="2" Body="&lt;p&gt;Regarding the features, except for the app category, I would try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;app size (in MBs) (i.e. avg of size of the downloaded apps, sum of different sizes, etc.) This would be important, since games tend to be huge, whereas productivity apps tend to have a smaller footprint. &lt;/li&gt;&#xA;&lt;li&gt;number of app downloads (again, avg and/or sum of the download counts for all the apps the user has downloaded).&lt;/li&gt;&#xA;&lt;li&gt;count of downloaded apps&lt;/li&gt;&#xA;&lt;li&gt;number of different categories (that gives an estimation of the variance of the categories the subject user inspects)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12374" LastActivityDate="2015-08-21T08:21:09.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6857" PostTypeId="2" ParentId="6854" CreationDate="2015-08-21T09:47:09.547" Score="8" Body="&lt;p&gt;There are two solutions that are worth looking at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.influxdb.com&quot;&gt;InfluxDB&lt;/a&gt; is an open source database platform specifically designed for time series data.&lt;/strong&gt; The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called &lt;a href=&quot;https://influxdb.com/chronograf/index.html&quot;&gt;Chronograf&lt;/a&gt;. I have not used this - but if its no good, you can also check out Grafana (which is very widely used and stable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The alternative strategy you may want to pursue is an elasticsearch index.&lt;/strong&gt; Elasticsearch is great at running aggregations and other mathematical functions on data. Many use it to store server log data and then query said data using Kibana.&lt;/p&gt;&#xA;" OwnerUserId="12234" LastActivityDate="2015-08-21T09:47:09.547" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6858" PostTypeId="2" ParentId="6837" CreationDate="2015-08-21T09:53:20.350" Score="1" Body="&lt;p&gt;I've done a lot of web analytics work and while we did do path analysis, we never focused so much on whether the user took the absolute shortest path to the page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike building a road system, a website should be built to provide different paths to the same content depending on the intent of the user. For example, some users know exactly what they want and will do a search. Other users are exploratory and want to browse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, sometimes you specifically want the user to step through a handful of pages to sell them on a conclusion. This is called a conversion funnel. If the user landed directly on the conversion point without seeing the info leading up to it, they'd bounce off the site and we'd lose the lead.&lt;/p&gt;&#xA;" OwnerUserId="12234" LastActivityDate="2015-08-21T09:53:20.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6859" PostTypeId="1" CreationDate="2015-08-21T10:27:19.047" Score="1" ViewCount="58" Body="&lt;p&gt;I have X clusters of size (n1,n2...,nX) and I want to fuse small clusters with less than T members with the nearest adjacent cluster i.e. the nearest neighboring cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is when I've got two small neighbor clusters, A-&gt;B and B-&gt;A which create an infinite loop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I try to avoid that by removing one of those which have this property [(A,B),(B,A)].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is there a better way to accomplish this task in sequential way ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to do the trick in distributed way ( Spark ) ?&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="9257" LastEditDate="2015-08-25T09:20:12.873" LastActivityDate="2015-08-25T09:20:12.873" Title="Best way to fuse clusters smaller than a certain size in a distributed way?" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6860" PostTypeId="2" ParentId="6835" CreationDate="2015-08-21T10:27:51.137" Score="5" Body="&lt;p&gt;As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve You should benchmark Your solution with suitable method. The best source for this can be publication, where as You noticed, many times results for ML dataset are given. What's more about benchmarking recommender systems, sometimes the best prediction results are not so important as other aspect e.g. novelty or diversity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most common methods try this benchmark results for librec project:&#xA;&lt;a href=&quot;http://www.librec.net/example.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.librec.net/example.html&lt;/a&gt;&#xA;There are no results for ML-10M, only ML-1M for rating and ML-100K for ranking. But You can always run chosen algorithm by yourself. There are other projects of this kind: &lt;a href=&quot;http://www.mymedialite.net/&quot; rel=&quot;noreferrer&quot;&gt;http://www.mymedialite.net/&lt;/a&gt;, &lt;a href=&quot;http://lenskit.org/&quot; rel=&quot;noreferrer&quot;&gt;http://lenskit.org/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saying about evaluation of recommender systems this project should be mention:&#xA;&lt;a href=&quot;https://github.com/recommenders/rival&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/recommenders/rival&lt;/a&gt;&#xA;It's dedicated for data preparation (splitting) and evaluation for recommender systems. It was presented at last RecSys'2014 [1]. And also look at this [2]. There are some benchmark results too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[1] A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” pp. 371–372, 2014.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[2] A. Said and A. Bellogín, “Comparative Recommender System Evaluation : Benchmarking Recommendation Frameworks,” pp. 129–136.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="9314" LastActivityDate="2015-08-21T10:27:51.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6861" PostTypeId="2" ParentId="6851" CreationDate="2015-08-21T10:47:02.427" Score="2" Body="&lt;p&gt;I'd be really tempted to be lazy and apply some old technology for a quick and dirty solution, with no programming, using the linux &lt;code&gt;sort&lt;/code&gt; command. &#xA;This will give you a lexicographically sorted list. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the store names are not the first field, if  just reorder them or tell &lt;code&gt;sort&lt;/code&gt; to use a different field via the &lt;code&gt;-k&lt;/code&gt; switch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Save the data to a plain CSV text file and then sort them:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$sort myStores.csv &amp;gt; sortedByStore.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can give sort a hand by allocating it plenty of memory, 16GB in this case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$sort -S16G myStores.csv &amp;gt; sortedByStore.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could go further and produce a list of unique store names and counts of instances for them to help you get a handle on what the data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$sort -S16G myStores.csv  | cut -f1 -d, | uniq -c &amp;gt; storeIdsAndCounts.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or to avoid resorting and  have only the unique IDs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$cat sortedByStore.csv   | cut -f1 -d, | uniq  &amp;gt; storeIds.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-08-21T11:17:20.473" LastActivityDate="2015-08-21T11:17:20.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6862" PostTypeId="2" ParentId="6859" CreationDate="2015-08-21T11:11:13.983" Score="1" Body="&lt;p&gt;My suggestion is just a greedy brute force approach. I am sure that more efficient solutions may exist, but it may be a first step towards your final solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Store in a table, for each cluster which size is lesser than T, the distance to its closest neighbouring cluster and the ID of that closest cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, sort the table entries in ascending distance order. Merge the cluster in the first entry with its closest neighbouring one. If the closest cluster also had an entry in the table, and its size got larger than T, remove its corresponding entry. Finally, update all the entries of the table, and sort them in ascending distance order again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat the steps above until there are not more entries in the table. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-08-21T11:11:13.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6863" PostTypeId="1" CreationDate="2015-08-21T11:18:27.883" Score="0" ViewCount="962" Body="&lt;p&gt;I have a data set with 50-60 independent variables and 1 categorical (1-0) dependent variable. Out of all the independent variables say, A is one variable which has majority of the effect on the target variable. For ex&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A        target variable&#xA;a              1 &#xA;a              1   &#xA;a              1    &#xA;a              1 &#xA;b              0  &#xA;b              1 &#xA;b              1 &#xA;b              1 &#xA;c              0 &#xA;c              0&#xA;c              0&#xA;c              1  &#xA;c              1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Clearly whatever analysis I do here, whatever probabilities I calculate for With respect to other variables, I would not be able to get the actual picture with respect to other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I do not want to build any model at this point, I just want to see the individual effects of other variables on the dependent variable after removing the effect of variable 'A' i.e. normalizing dependent variable with respect to 'A'. How do I do? If the dependent variable was a continuous variable, I would have simply divided it by the respective averages and removed the effect. How do I do this in this case where the dependent variable is 1-0?&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2015-08-24T01:53:27.273" Title="normalize dependent variable with respect to one independent variable" Tags="|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6865" PostTypeId="1" AcceptedAnswerId="6867" CreationDate="2015-08-21T13:49:10.677" Score="0" ViewCount="1036" Body="&lt;p&gt;According to this &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/dae/logit.htm&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;, it says SAS models for 0,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To model 1s rather than 0s, we use the descending option. We do this because by default, proc logistic models 0s rather than 1s  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What happens in case of R's glm function? Does it model for 1 or 0? Is there a way to change it? Does it matter?&lt;/p&gt;&#xA;" OwnerUserId="10050" LastActivityDate="2015-08-21T16:26:36.130" Title="Logistic Regression in R models for 1 or 0?" Tags="|r|logistic-regression|sas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6866" PostTypeId="1" AcceptedAnswerId="10150" CreationDate="2015-08-21T14:26:10.297" Score="3" ViewCount="2350" Body="&lt;p&gt;I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period that's meaningful within the domain. The problem is that this period is highly variable in length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've reviewed &lt;a href=&quot;https://datascience.stackexchange.com/questions/2673/training-neural-networks-with-unknown-length-of-input?rq=1&quot;&gt;this question&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/595/how-to-use-neural-networks-with-large-and-variable-number-of-inputs?rq=1&quot;&gt;this question&lt;/a&gt; and neither solve the problem, they are more about how to deal with missing values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seeing as denoising autoencoders are based off of matrix multiplication, this presents a serious problem. What is the standard approach in such a situation? Should I define an arbitrary (large) window size, and expand windows that are too small (and vice versa)? Or is there a better approach for variable length inputs? &lt;/p&gt;&#xA;" OwnerUserId="12381" LastEditorUserId="12381" LastEditDate="2015-08-21T14:37:46.083" LastActivityDate="2016-02-09T09:06:10.080" Title="Denoising Autoenoders with Variable Length Input" Tags="|neural-network|preprocessing|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6867" PostTypeId="2" ParentId="6865" CreationDate="2015-08-21T16:26:36.130" Score="1" Body="&lt;p&gt;I always learned to model for 1. See below for the impact of switching the encoding,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- glm(ans ~ x, data = simulated_data, family = binomial)&#xA;summary(model)&#xA;&#xA;# &#xA;# Call:&#xA;# glm(formula = ans ~ x, family = binomial, data = simulated_data)&#xA;# &#xA;# Deviance Residuals: &#xA;#       1        2        3        4        5  &#xA;#  1.6388  -0.6249  -1.2146  -0.8083   0.7389  &#xA;# &#xA;# Coefficients:&#xA;#             Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;# (Intercept)  -2.4753     2.5006   -0.99    0.322&#xA;# x             0.5957     0.6543    0.91    0.363&#xA;# &#xA;# (Dispersion parameter for binomial family taken to be 1)&#xA;# &#xA;#     Null deviance: 6.7301  on 4  degrees of freedom&#xA;# Residual deviance: 5.7505  on 3  degrees of freedom&#xA;# AIC: 9.7505&#xA;# &#xA;# Number of Fisher Scoring iterations: 4&#xA;# &#xA;&#xA;simulated_data$ans &amp;lt;- !simulated_data$ans&#xA;model_opp &amp;lt;- glm(ans ~ x, data = simulated_data, family = binomial)&#xA;summary(model_opp)&#xA;&#xA;# &#xA;# Call:&#xA;# glm(formula = ans ~ x, family = binomial, data = simulated_data)&#xA;# &#xA;# Deviance Residuals: &#xA;#       1        2        3        4        5  &#xA;# -1.6388   0.6249   1.2146   0.8083  -0.7389  &#xA;# &#xA;# Coefficients:&#xA;#             Estimate Std. Error z value Pr(&amp;gt;|z|)&#xA;# (Intercept)   2.4753     2.5006    0.99    0.322&#xA;# x            -0.5957     0.6543   -0.91    0.363&#xA;# &#xA;# (Dispersion parameter for binomial family taken to be 1)&#xA;# &#xA;#     Null deviance: 6.7301  on 4  degrees of freedom&#xA;# Residual deviance: 5.7505  on 3  degrees of freedom&#xA;# AIC: 9.7505&#xA;# &#xA;# Number of Fisher Scoring iterations: 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-21T16:26:36.130" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6868" PostTypeId="1" CreationDate="2015-08-21T18:10:54.847" Score="8" ViewCount="211" Body="&lt;p&gt;As a newcomer to the field, I find many of the &lt;a href=&quot;http://papers.nips.cc/book/advances-in-neural-information-processing-systems-27-2014&quot;&gt;NIPS 2014 papers&lt;/a&gt; fascinating, but it is difficult for me to evaluate which ones represent real progress over current approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which papers do you think are most significant and are likely to have a major impact?&lt;/p&gt;&#xA;" OwnerUserId="12386" LastActivityDate="2015-08-22T14:10:12.447" Title="Which of the NIPS 2014 papers are most significant, and why?" Tags="|machine-learning|research|state-of-the-art|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6869" PostTypeId="2" ParentId="6868" CreationDate="2015-08-22T14:10:12.447" Score="4" Body="&lt;p&gt;The &lt;a href=&quot;https://nips.cc/Conferences/2014/Awards&quot; rel=&quot;nofollow&quot;&gt;awards&lt;/a&gt; went to &quot;&lt;a href=&quot;http://papers.nips.cc/paper/5329-asymmetric-lsh-alsh-for-sublinear-time-maximum-inner-product-search-mips&quot; rel=&quot;nofollow&quot;&gt;Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)&lt;/a&gt;&quot; by Anshumali Shrivastava and Ping Li and &quot;&lt;a href=&quot;http://papers.nips.cc/paper/5449-a-sampling&quot; rel=&quot;nofollow&quot;&gt;A* Sampling&lt;/a&gt;&quot; by Chris J. Maddison, Daniel Tarlow, and Tom Minka.&lt;br&gt;&#xA;Further Quora discussion can be found &lt;a href=&quot;http://www.quora.com/What-were-the-scientific-highlights-of-the-NIPS-2014&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10319" LastActivityDate="2015-08-22T14:10:12.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6870" PostTypeId="1" CreationDate="2015-08-22T18:06:21.510" Score="0" ViewCount="1131" Body="&lt;p&gt;I am trying to get classification tree in R. I am using MNIST (handwritten digit dataset) to train my decision tree. The dataset has 28*28 pixel box. Pixels are organized in row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black). I have to calculate entropy to get best attributes among 784 attributes (pixel0 to pixel783). Can anyone suggest, how to calculate entropy or information gain for these dense attributes?&lt;/p&gt;&#xA;" OwnerUserId="12395" LastActivityDate="2015-08-26T13:31:42.913" Title="Entropy calculation for MNIST dataset to form classification decision tree" Tags="|r|decision-trees|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6874" PostTypeId="1" CreationDate="2015-08-23T15:49:14.167" Score="1" ViewCount="105" Body="&lt;p&gt;I'm trying to merge two datasets on Mergers &amp;amp; Acquisitions. They both consist of c.10'000 observations with c.50-100 variables each. One contains information about the actual M&amp;amp;A deal whereas the other one contains info on how a deal was financed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that there is no clear and unique identifier. For example, I could use the date that the deal was announced but that wouldn't be unique because on some days 10 deals were announced. Using company names is difficult since they mostly aren't identical in both datasets. For example if in one dataset I find &quot;Ebay&quot;, in the other the same company could be called &quot;eBay&quot;, &quot;Ebay Inc&quot;, or &quot;Ebay, Inc.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been working with the Fuzzy Lookup add-on for Excel, as well as concacenating various identifiers that are not unique but in their combination become useful (e.g. Date &amp;amp; Country &amp;amp; SIC Industry Classification Code, etc.). However I haven't been able to generate as many matches as I would have hoped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be grateful for any ideas or pointers towards resources that would help me merge the datasets more efficiently.&lt;/p&gt;&#xA;" OwnerUserId="12403" LastActivityDate="2015-08-24T00:16:19.567" Title="Merging two datasets with unclear identifier in Excel" Tags="|databases|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6875" PostTypeId="2" ParentId="6874" CreationDate="2015-08-23T16:59:32.163" Score="1" Body="&lt;p&gt;I typically proceed the following way,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;convert to the same case with &lt;code&gt;LOWER()&lt;/code&gt; or &lt;code&gt;UPPER()&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;remove all non-alphanumeric characters with several &lt;code&gt;SUBSTITUTE()&lt;/code&gt;, e.g. &lt;code&gt;SUBSTITUTE(A1, &quot;.&quot;, &quot;&quot;)&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;trim white spaces &lt;code&gt;TRIM()&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;extract the first word with &lt;code&gt;LEFT(A1, FIND(&quot; &quot;, A1) - 1)&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;trim white spaces again;&lt;/li&gt;&#xA;&lt;li&gt;perform a join on the key that you created.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Often this gives you a decent match. &lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-23T16:59:32.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6876" PostTypeId="2" ParentId="6184" CreationDate="2015-08-23T17:23:07.187" Score="1" Body="&lt;p&gt;It is a bit hard to solve your problem without data but I tried to give it a go&#xA;with a how I think the data would be encoded. I used R with &lt;a href=&quot;https://cran.r-project.org/web/packages/data.table/&quot; rel=&quot;nofollow&quot;&gt;data.tables&lt;/a&gt;. You can read data.tables with &lt;code&gt;fread()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Step 1&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; require(data.table)&#xA;&#xA; # Assume sample_data has the following format:&#xA; #   class: the class&#xA; #   division: the division&#xA; #   attendance: the attandance for a match&#xA; #&#xA; # I assume the table is in long format e.g. multiple rows exist per class with&#xA; # per class one or different divisions.&#xA;&#xA; # Make the list of classes with at least 5 divisions.&#xA; classes_of_interest &amp;lt;- &#xA;   sample_data[, &#xA;               .(num_divisions = length(unique(divisions))),&#xA;               by = class][num_divisions &amp;gt; 4, class]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Step 2&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; # Only consider the classes that were in at least 5 divisions.&#xA; attandance_by_division &amp;lt;- &#xA;   sample_data[class %in% classes_of_interest, &#xA;               .(attendance = sum(num_people)),&#xA;               by = list(division, class)]&#xA; setkey(attandance_by_division, &quot;class&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Step 3&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; # Merge the data set with a datas set that contains the required number&#xA; # of attendants per class.&#xA; # The  format is as follows:&#xA; #   class: the class&#xA; #   mininum_attendance: the minimum attendance&#xA; attendance_data &amp;lt;- &#xA;   merge(attendance_requirements, &#xA;         attandance_by_division, by = &quot;class&quot;)&#xA;&#xA; # Here I exploit the fact that the true/false condition will be converted&#xA; # to a 1 and 0. So I can sum and divide by the length of index subset created&#xA; # by aggregating on 'class'.&#xA; pct_of_division &amp;lt;- &#xA;   sample_data[, &#xA;               .(pct_with_min_attendance = (sum(attendance &amp;gt; minimum_attendance)&#xA;                                            / length(.I))),&#xA;                 by = class]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-23T17:23:07.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6877" PostTypeId="2" ParentId="6500" CreationDate="2015-08-23T17:30:57.120" Score="2" Body="&lt;p&gt;Your first attempt was more accurate. Since I do not have your data I will just use the iris data set,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; iris_ordered_on_Sepal_Length &amp;lt;- iris[order(iris$Sepal.Length),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or if you use &lt;a href=&quot;https://cran.r-project.org/web/packages/data.table/&quot; rel=&quot;nofollow&quot;&gt;data.table&lt;/a&gt;,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;iris_dt &amp;lt;- data.table(iris)&#xA;iris_dt[order(Sepal.Length)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;data.table will make renaming easier too,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;setnames(input, c(&quot;colname_11, &quot;colname_17&quot;, &quot;colname_23&quot;), validoutcomes[1 : 3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-23T17:30:57.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6878" PostTypeId="2" ParentId="6874" CreationDate="2015-08-23T18:59:54.223" Score="0" Body="&lt;p&gt;Use google search, which has some very sophisticated similarity algorithms,  to lookup the company names and see if the point to the same web site addresses using python, mathematica or import.io or a tool of your choice.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-08-24T00:16:19.567" LastActivityDate="2015-08-24T00:16:19.567" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6879" PostTypeId="1" CreationDate="2015-08-23T20:01:24.360" Score="1" ViewCount="77" Body="&lt;p&gt;I have a problem of process control which looks like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The outputs of the process are a number of outcome variables 0-100&#xA;where 0 is worst and 100 is best. There is no specific rule (already&#xA;known, or related to the nature of the problem) for how to combine&#xA;them into a single cost function.  There is some intuition about what results are good, however.&lt;/li&gt;&#xA;&lt;li&gt;All the variables are equally important, and scaled the same way. &#xA;The cost function should be symmetric with respect to permutation of&#xA;the values.&lt;/li&gt;&#xA;&lt;li&gt;If all the variables are 100 except one is 0, that is a bad&#xA;outcome.&lt;/li&gt;&#xA;&lt;li&gt;If all the variables are 90, that is a very good outcome.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The last two statements imply that the right way to combine variables into a cost function is something similar to min(v1, v2, ...).  But that has a problem, in that the direction of optimization would always be determined by the lowest variable alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it is desirable to trade off a slight worsening in one variable for a large improvement in another, even if the variable that is getting worse is already pretty bad, for example: 30 and 50 is less preferred than 28 and 75 (or something like that).  This implies some relative elasticity of variables which in this case is close to 100:1 for variables near the opposite ends of the range, and 1:1 for variables which are equal valued.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can a good cost function be constructed for this kind of problem in a general-purpose way?&lt;/strong&gt;  (with some adjustable parameters for elasticity, etc)&lt;/p&gt;&#xA;" OwnerUserId="26" LastActivityDate="2015-08-23T20:01:24.360" Title="Universal cost function for process control?" Tags="|optimization|loss-function|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6881" PostTypeId="2" ParentId="6841" CreationDate="2015-08-24T00:20:05.320" Score="1" Body="&lt;p&gt;The fastest way would be,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(data.table)&#xA;data &amp;lt;- data.table(data)&#xA;# Remove the percentage from your file and convert the field to numeric.&#xA;data[, Profit := as.numeric(gsub(&quot;%&quot;, &quot;&quot;, Profit))]&#xA;&#xA;data&#xA;##           Symbol      Date     Time Profit&#xA;##  1: BANKNIFTY  4/1/2010  9:55:00  -1.18&#xA;##  2: BANKNIFTY  4/1/2010 12:30:00  -2.84&#xA;##  3: BANKNIFTY  4/1/2010 12:45:00   7.17&#xA;##  4: BANKNIFTY  5/1/2010 11:40:00  -7.11&#xA;##  5:      ZEEL 26/6/2012 13:50:00  24.75&#xA;##  6:      ZEEL 27/6/2012 15:15:00  -1.90&#xA;##  7:      ZEEL 28/6/2012  9:45:00  37.58&#xA;##  8:      ZEEL 28/6/2012 14:55:00  23.95&#xA;##  9:      ZEEL 29/6/2012 14:20:00  -4.65&#xA;## 10:      ZEEL 29/6/2012 14:30:00  -6.01&#xA;## 11:      ZEEL 29/6/2012 14:55:00 -12.23&#xA;## 12:      ZEEL 29/6/2012 15:15:00  35.13&#xA;&#xA;# Melt the data so that we can easily dcast afterwards.&#xA;molten_data &amp;lt;- melt(data[, list(Symbol, Date, Profit)]&#xA;&#xA;# Create a summary by date and Symbol.&#xA;dcast(molten_data, id = c(&quot;Symbol&quot;, &quot;Date&quot;)), Date ~ variable + Symbol, fun = sum)&#xA;&#xA; ##         Date Profit_BANKNIFTY Profit_ZEEL&#xA; ## 1: 26/6/2012             0.00       24.75&#xA; ## 2: 27/6/2012             0.00       -1.90&#xA; ## 3: 28/6/2012             0.00       61.53&#xA; ## 4: 29/6/2012             0.00       12.24&#xA; ## 5:  4/1/2010             3.15        0.00&#xA; ## 6:  5/1/2010            -7.11        0.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-24T00:20:05.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6882" PostTypeId="2" ParentId="6776" CreationDate="2015-08-24T00:57:48.230" Score="4" Body="&lt;p&gt;There's already at least one application out, if you interpret 'application' broadly enough: &lt;a href=&quot;http://arxiv.org/abs/1506.04924&quot; rel=&quot;nofollow&quot;&gt;Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation by Hong, Noh and Han&lt;/a&gt;. They use it for &lt;a href=&quot;https://en.wikipedia.org/wiki/Image_segmentation&quot; rel=&quot;nofollow&quot;&gt;image segmentation&lt;/a&gt;. Standard image recognition networks can only give you a bounding box for each object recognized on an image. If you want to know which pixels constitute that object, you have to do image segmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, after finding a dog on an image, Hong et al's architecture back-propagates the dog-ness through the neural network down to the pixel level, to find the pixels that were the most responsible for the dog appearing. (They then use this heatmap as input for a supervised segmentation network, there's no deep dreaming in that part.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is already kind of an existence proof that the Deep Dream idea can be useful outside image manipulation. But I wouldn't downplay image manipulation itself either. I mention two things that are not immediate applications of Deep Dreaming, and we don't have them currently, but I can kinda see a plausible road from the original Deep Dream algorithm towards these:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Beautifying pictures and human faces and bodies. (Automating what a Photoshop retouch artist does.)&lt;/li&gt;&#xA;&lt;li&gt;CSI-style image upscaling with fake but believable interpolated detail.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12409" LastActivityDate="2015-08-24T00:57:48.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6883" PostTypeId="2" ParentId="6863" CreationDate="2015-08-24T01:53:27.273" Score="1" Body="&lt;p&gt;Though continuum regression methods appear to be significantly different from binary classification methods, the crux of the difference is usually tied up in a &lt;a href=&quot;https://en.wikipedia.org/wiki/Sigmoid_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;sigmoid function&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S(t)=\frac{1}{1+e^-t}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sigmoid literally means sigma like or &quot;S&quot; like, as becomes obvious in looking at its graphical representation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2Zz9l.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2Zz9l.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sigmoid provides a tractable means to map from continuum analytics into discrete space while retaining differentiability and integrability needed for some theoretical derivations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The important part&lt;/strong&gt; is that in classification methods, one has to choose a threshold value of the sigma function for mapping to a 0 or 1.  This is overwhelmingly chosen as 0.5, but there are cases of unbalanced classes where other values might work better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, you should be able to proceed exactly as you have already suggested: &quot;I would have simply divided it by the respective averages and removed the effect.&quot;  Do this, but then you have to turn the results back into 0's and 1's by choosing where your cutoff threshold lies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-08-24T01:53:27.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6884" PostTypeId="1" AcceptedAnswerId="6896" CreationDate="2015-08-24T07:38:03.147" Score="3" ViewCount="2646" Body="&lt;p&gt;I am trying to understand the abstract details that explain how h2o is faster than R and SAS for data science computations.&lt;/p&gt;&#xA;" OwnerUserId="12411" LastEditorUserId="8501" LastEditDate="2016-12-01T15:46:37.783" LastActivityDate="2016-12-01T15:46:37.783" Title="How is H2O faster than R or SAS?" Tags="|machine-learning|r|bigdata|performance|sas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6886" PostTypeId="2" ParentId="904" CreationDate="2015-08-24T16:36:51.427" Score="1" Body="&lt;p&gt;You could try &lt;a href=&quot;http://rstudio.github.io/shinydashboard/&quot; rel=&quot;nofollow&quot;&gt;ShinyDashboard&lt;/a&gt; and leave out any interactive aspects. In my opinion it is still nice to allow users to zoom in on plots or search for certain values in a column. Shiny will allow you to do this.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-24T16:36:51.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6887" PostTypeId="1" AcceptedAnswerId="6889" CreationDate="2015-08-24T16:51:45.647" Score="0" ViewCount="82" Body="&lt;p&gt;I am trying to filter a data frame where I need to search for a regular expression within the data frame. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dim(df)&#xA;[1] 10299 561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wanted to know if there is a way, I could find the name(s) of the column(s) where a particular regex may be available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, how do I find the name of the column where 'rsi' is present. I am referring to iris. &lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2015-08-24T18:51:56.090" Title="Find the column(s) name where the value of the variable matches a regex" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6889" PostTypeId="2" ParentId="6887" CreationDate="2015-08-24T18:51:56.090" Score="2" Body="&lt;p&gt;Would this work for you,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; names(iris)[apply(iris, 2, function(x) any(grepl(&quot;rsi&quot;, x)))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-24T18:51:56.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6890" PostTypeId="2" ParentId="6859" CreationDate="2015-08-25T03:15:11.023" Score="1" Body="&lt;p&gt;I'm not really seeing the issue with an infinite loop.  It seems like you are looping through your clusters and finding all the clusters that need to be merged, then in a separate loop doing the merging and are getting tripped up when two clusters are flagged in such a way that they both need to be merged into one another.  Instead, you can think of this as more of a queue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can loop through one time and merge the small clusters with their neighbors and then destroy them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have a &lt;code&gt;cluster&lt;/code&gt; class with attributes: &lt;code&gt;membercount&lt;/code&gt;, &lt;code&gt;members&lt;/code&gt;, and nearestNeighbor, and methods &lt;code&gt;addMembers()&lt;/code&gt; and &lt;code&gt;destroy()&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you could do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (cluster in clusterList):&#xA;    if cluster.membercount&amp;lt;T:&#xA;        cluster.nearestNeighbor.addMembers(cluster.members)&#xA;        cluster.destroy()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will only cycle through clusterList one time and will instantly merge and destroy the clusters that are smaller than T.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you have experience with object oriented programming, this would be pretty straightforward to implement.  Even if you do this within an array, you can just &lt;code&gt;pop&lt;/code&gt; clusters after you merge them with other clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-08-25T03:15:11.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6891" PostTypeId="2" ParentId="3745" CreationDate="2015-08-25T04:51:10.447" Score="0" Body="&lt;p&gt;A good method to reduce overfitting in Decision Trees is Pruning, i.e. not to generate the tree downto the last attribute (pre-prune), or generate the &quot;full&quot; tree and then post-prune. &#xA;Not sure if it pertains to Bayesian theory per se, but the leaf nodes of the &quot;pruned&quot; tree have probabilities about the various classifications that are constrained by the pruned branches.&#xA;There is an implementation in Weka (J48 classifier) and the accompanying book Data Mining - Practical Machine Learning Tools and Techniques:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0120884070&quot; rel=&quot;nofollow&quot;&gt;http://www.amazon.com/Data-Mining-Practical-Techniques-Management/dp/0120884070&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to read in more detail on what I just wrote above, it is explained in detail in Chapter 6.1&lt;/p&gt;&#xA;" OwnerUserId="12418" LastActivityDate="2015-08-25T04:51:10.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6894" PostTypeId="1" AcceptedAnswerId="6895" CreationDate="2015-08-25T12:37:54.423" Score="0" ViewCount="133" Body="&lt;p&gt;I'm performing a logistic regression in R.&lt;br&gt;&#xA;I wanted to know if the function &lt;em&gt;predict.glm&lt;/em&gt; outputs $p$ (probability of event occurring) or log odds i.e. $log(p/1-p)$?&lt;/p&gt;&#xA;" OwnerUserId="10050" LastActivityDate="2015-08-25T13:21:20.567" Title="For logistic regression, Predict.glm() outputs $p$ or $ln(p/1-p)$?" Tags="|r|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6895" PostTypeId="2" ParentId="6894" CreationDate="2015-08-25T13:21:20.567" Score="2" Body="&lt;p&gt;It returns the log odds. You can see that with this basic example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Create a perfect correlated data.&#xA;data &amp;lt;- data.frame(x = c(1,0,1,0), y = c(T, F, T, F))&#xA;&#xA;# Estimate the model.&#xA;model &amp;lt;- glm(y ~ x, data)&#xA;&#xA;# Show predictions.&#xA;predict(model)&#xA;##        1         2         3         4 &#xA;## 23.56607 -23.56607  23.56607 -23.56607&#xA;&#xA;# Compute the inverse of the log odd to come to your prediction. &#xA;boot::inv.logit(predict(model))&#xA;##           1            2            3            4 &#xA;##1.000000e+00 5.826215e-11 1.000000e+00 5.826215e-11 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If logit would return probabilities you could not take the inverse to come to your predicted values.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-08-25T13:21:20.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6896" PostTypeId="2" ParentId="6884" CreationDate="2015-08-25T13:32:01.330" Score="13" Body="&lt;p&gt;I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It is much harder to achieve the same efficiency in R alone. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why H2O is much faster is that they have a very good indexing of their data and their algorithms are written such that they exploit parallelism to the fullest. See &lt;a href=&quot;http://h2o.ai/blog/2014/03/h2o-architecture/&quot; rel=&quot;noreferrer&quot;&gt;http://h2o.ai/blog/2014/03/h2o-architecture/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R with the default matrix dynamic libraries can only use one CPU core. Revolution R community edition ships with the Intel Math Kernel Library. This allows for some matrix computations in parallel but definitely not as efficient as H2O. For SAS it is a bit harder to say anything considering it's closed source but based on my CPU utilization I would assume that they have a similar approach as Revolution R. Their matrix algebra exploits parallelism but they algorithms are not as efficient as H2O. Their data storage is also not as efficient as H2O.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, H2O with R comes at a very different price tag than SAS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this clarifies a bit.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="26596" LastEditDate="2016-12-01T14:43:11.920" LastActivityDate="2016-12-01T14:43:11.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6898" PostTypeId="1" AcceptedAnswerId="6900" CreationDate="2015-08-25T15:39:24.467" Score="1" ViewCount="212" Body="&lt;p&gt;Having n vectors of percentile ranks for a list of common users between group #1 and groups #2:n&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. vec1 = {0.25, 0.1, 0.8, 0.75, 0.5, 0.6}&#xA;     vec2 = {0.35, 0.2, 0.6, 0.45, 0.2, 0.9}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The percentile ranks represent activity frequency within the group for instance, opening times.&#xA;The goal is to find similarity between group #1 and groups #2..n in terms of these common users's rank.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The direction taken so far is to use dot product in order to take the magnitude into account (due to rank). The problem is that the scalar answer can take any value and so a threshold cannot be drawn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to draw a threshold with respect to the dot product of vec #1 (group 1) with itself? or is there another way to set a threshold (maybe dynamic) ?&lt;/p&gt;&#xA;" OwnerUserId="12433" LastActivityDate="2015-08-26T19:58:08.487" Title="users' percentile similarity measure" Tags="|similarity|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6899" PostTypeId="1" AcceptedAnswerId="24529" CreationDate="2015-08-25T16:04:08.067" Score="13" ViewCount="731" Body="&lt;p&gt;Suppose I have classifiers C_1 ... C_n that are disjoint in the sense that no two will return true on the same input (e.g. the nodes in a decision tree).  I want to build a new classifier that is the union of some subset of these (e.g. I want to decide on which leaves of a decision tree to give a positive classification).  Of course, in doing so there will be a trade off between sensitivity and positive predictive value.  So I would like to see a ROC curve.  In principle I could do this by enumerating all subsets of the classifiers and computing the resulting sensitivity and PPV.  However, this is prohibitively expensive if n is more than 30 or so.  On the other hand, there are almost certainly some combinations that are not Pareto optimal, so there might be some branch and bound strategy, or something, that avoids most of the computation in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like advice about whether this approach is likely to be fruitful and whether there is any work or if you have any ideas about efficiently computing the ROC curve in the situation above.&lt;/p&gt;&#xA;" OwnerUserId="12438" LastActivityDate="2017-11-09T16:10:35.763" Title="Efficient algorithm to compute the ROC curve for a classifier consisting of an ensemble of disjoint classifiers" Tags="|algorithms|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="6900" PostTypeId="2" ParentId="6898" CreationDate="2015-08-25T17:17:27.533" Score="2" Body="&lt;p&gt;As I understand it, you don't want to use the &lt;code&gt;dot product&lt;/code&gt;, because it is unbounded.  Yet you also don't want to use the &lt;a href=&quot;http://blog.christianperone.com/?p=2497&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;cosine similarity metric&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oFN5J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oFN5J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, which is just the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;dot product normalized by the length&lt;/a&gt; of the two vectors being dotted because it does not take magnitude into account e.g. $(1,1)\cdot(1,1)=1=(1,1)\cdot(2,2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you could therefor use the &lt;a href=&quot;https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;sorensen dice coefficient&lt;/a&gt; or possibly 1-soresen:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/75xmf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/75xmf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This varies from 0 to 1 and accounts for magnitudes of the constituent vectors e.g.:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s_{\nu}[(1,1),(1,1)]=1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, where as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$s_{\nu}[(1,1),(2,2)]=\frac{4}{5}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-08-25T18:21:07.960" LastActivityDate="2015-08-25T18:21:07.960" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6903" PostTypeId="2" ParentId="6898" CreationDate="2015-08-26T10:34:55.373" Score="0" Body="&lt;p&gt;As you are using percentile values, your vector components are effectively normalised on the interval $[0,1]$. As a result the element wise subtraction of components, normalised by the number of components in the vector, will give you a bounded similarity value between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for two vectors $x$ and $y$ of length $n$, similarity can be computed thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S(x,y)=1-\frac{1}{n}\sum_{i=1}^n{\lvert x_i-y_i\lvert}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can refine this approach by taking account of the variance for each component within the vector and standardising this around 0.5.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-08-26T19:58:08.487" LastActivityDate="2015-08-26T19:58:08.487" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6904" PostTypeId="1" AcceptedAnswerId="6906" CreationDate="2015-08-26T12:22:06.380" Score="2" ViewCount="201" Body="&lt;p&gt;We are working with a complex application i.e. a physical measurement in a lab, that has approximately 230 different input parameters, many of which are ranges or multiple-value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The application produces a single output, which is then verified in an external (physical) process. At the end of the process the individual tests are marked as &quot;success&quot; or &quot;fail&quot;. That is, despite the many input parameters, the output is assessed in a boolean manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When tests fail, the parameters are 'loosened' slightly and re-tested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have about 20,000 entries in our database, with both &quot;success&quot; and &quot;fail&quot;, and we are considering a machine learning application to help in two areas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Initial selection of optimum parameters&lt;br&gt;&#xA;2) Suggestions for how to tune the parameters after a &quot;fail&quot;    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of the input parameters are strongly related to each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I studied computer science in the mid-90s, when the focus was mostly expert systems and neural networks. We also have access to some free CPU hours of Microsoft Azure Machine Learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of machine learning would fit these use-cases?&lt;/p&gt;&#xA;" OwnerUserId="12454" LastEditorUserId="9420" LastEditDate="2015-08-27T14:09:17.757" LastActivityDate="2015-08-27T14:09:17.757" Title="Which type of machine learning to use" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="6906" PostTypeId="2" ParentId="6904" CreationDate="2015-08-26T14:53:36.273" Score="4" Body="&lt;p&gt;With using R, You could look at trees / randomforests. Since you have correlated variables, you could look into Projection pursuit classification trees (R package pptree). And there soon will be a ppforest package. But this is still under  development.  You could also combine randomforest with the package forestFloor to see the curvature of the randomforest and work from there.&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-08-26T14:53:36.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6907" PostTypeId="1" CreationDate="2015-08-26T15:35:29.083" Score="3" ViewCount="1420" Body="&lt;p&gt;I want to identify outliers from a very small group of numbers. How shall I do that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for the group of numbers: -0.4, 0.4, 52.1, actually 52.1 is an outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using interquartile range to identify the outliers, but it won't identify 52.1 as the outlier. &lt;/p&gt;&#xA;" OwnerUserId="12457" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:46.163" LastActivityDate="2017-05-04T19:57:46.163" Title="How to identify outliers from a small list of numbers?" Tags="|machine-learning|statistics|outlier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6908" PostTypeId="1" CreationDate="2015-08-26T17:37:49.187" Score="2" ViewCount="652" Body="&lt;p&gt;I am still new to data mining but I really want (and need) to learn it so badly. I know that before I can actually process my data in softwares like WEKA, I need to do some filtering like cleaning the data, integrating, transforming, etc to actually get your data cleaned from any kind of duplicate, missing value, noise, etc. But I only know all of these theoretically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I have now is I have a very big set of data that I need to filter first before going on to the processing part. But I don't know where to start. Fyi, my dataset is very big that the usual spreadsheet programs like Ms Excel, Libre Office, WPS,etc can't open it. I have to use Linux terminal and commands to count the number of rows, columns etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I do in the preprocessing start? How do I 'clean' my data? I have been thinking to use Linux commands to do all of these, but I am also wondering how real data scientists clean their data. Do they do these manually or they already have some sort of software to help them? Because seriously I don't know where to start or to do. Every reference I found in the internet only explain things theoretically. ANd I need something more practical to help me understand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I do with my dataset? Help please?&lt;/p&gt;&#xA;" OwnerUserId="12462" LastActivityDate="2017-09-27T13:09:10.870" Title="Preprocessing in Data mining?" Tags="|data-mining|dataset|data-cleaning|preprocessing|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6909" PostTypeId="2" ParentId="6908" CreationDate="2015-08-26T19:00:27.290" Score="3" Body="&lt;p&gt;The preprocessing will depend on what your data is like: textual? numerical? If in whatever analysis you want to do the data is not allowed to have duplicates, you'll have to filter those out. If not, they'll get to stay, etc. &#xA;If you already know exactly what you want to do theoretically and just need to find some software, python pandas is very good. Load the data, clean it, group it, perform elementary analysis.&lt;/p&gt;&#xA;" OwnerUserId="12464" LastActivityDate="2015-08-26T19:00:27.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6910" PostTypeId="1" AcceptedAnswerId="6916" CreationDate="2015-08-26T20:44:06.540" Score="5" ViewCount="1467" Body="&lt;p&gt;Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework?  Or are there restrictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess there may be some algorithms that is not possible to parallelize?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2016-11-14T17:51:33.400" Title="Can all statistical algorithms be parallelized using a Map Reduce framework" Tags="|machine-learning|apache-hadoop|map-reduce|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6911" PostTypeId="1" AcceptedAnswerId="6915" CreationDate="2015-08-26T20:49:40.957" Score="2" ViewCount="221" Body="&lt;p&gt;I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured data) as opposed to OLAP databases (Netezza)&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2015-08-27T14:08:36.973" Title="Can Hadoop be beneficial when data is in database tables and not in a file system" Tags="|apache-hadoop|databases|map-reduce|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6913" PostTypeId="2" ParentId="6908" CreationDate="2015-08-27T09:06:06.957" Score="3" Body="&lt;p&gt;I think different people probably have varying approaches to this depending upon their background. Often the linux toolset gets over looked, things like &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;sed&lt;/code&gt;, &lt;code&gt;grep&lt;/code&gt;, &lt;code&gt;cut&lt;/code&gt;, &lt;code&gt;paste&lt;/code&gt;, &lt;code&gt;sort&lt;/code&gt;, &lt;code&gt;uniq&lt;/code&gt; and so on, can be combined in many sophisticated ways and are very powerful and scalable, but they aren't for everyone. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect many people use a programming language that they already have familiarity with or that they  use for their data science work, &lt;code&gt;R&lt;/code&gt;, &lt;code&gt;Python&lt;/code&gt;, &lt;code&gt;Matlab&lt;/code&gt;, &lt;code&gt;Mathematica&lt;/code&gt;, &lt;code&gt;C++&lt;/code&gt;, &lt;code&gt;Julia&lt;/code&gt;, the list goes on. Others may use commercial products like SAS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Certainly Python, and it's relevant libraries, and R have large users bases and are popular choices. Ultimately you may have to look at a few options and decide which one suits you and your skill set best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you've already mentioned, &lt;code&gt;WEKA&lt;/code&gt; provides an accessible user interface to get started quickly and is certainly a good  place to start for early experimentation and to produce results quickly. It has a range of filters which can be used to preprocess data which may be of use to you. It exposes much more of its functionality via its &lt;code&gt;API&lt;/code&gt; which is available via &lt;code&gt;Java&lt;/code&gt; and this gives you the ability to be much more creative as you build your own solutions to your problems, but you will need to learn to program in &lt;code&gt;Java&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another, more eclectic, approach is that you draw from many toolboxes like &lt;code&gt;R&lt;/code&gt;, &lt;code&gt;Python&lt;/code&gt;, &lt;code&gt;WEKA&lt;/code&gt;, &lt;code&gt;C++&lt;/code&gt;, &lt;code&gt;Apache Mahout&lt;/code&gt; etc and fuse them using your own framework written in your favourite programming language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exactly which data cleansing operations you need to apply, missing value replacement, addition of noise, resampling, etc.,  will depend more heavily on the specific nature of your data and it's application. &lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-08-27T09:06:06.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6914" PostTypeId="2" ParentId="6911" CreationDate="2015-08-27T10:12:30.490" Score="1" Body="&lt;p&gt;I think you are referring to the HDFS part of Hadoop. If I'm correct, using Hadoop IMHO should not be considered as an alternative, but as a necessity. Hadoop answers the question : how can I take advantage advantage of the massive amount of data I have since I can't use it now as it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, HDFS is a distributed file system (that's why Hadoop DFS), and if you are not experiencing the limits of your database tables for now (or planning to in the future), then there is no reason to consider switching. In addition I think that depending on the data, you might not be able to do whatever you want with a bank database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should add that in the big data ecosystem there are alternative to databases (NoSQL system for example), but my answer is entirely conditioned on the fact that you are not mentioning any specific task/goal so my answer is basically :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;_ I have a computer which is sufficient for my work today and in the future, do I need to change it ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;_ If you believe it is sufficient for your future use, then no.&lt;/p&gt;&#xA;" OwnerUserId="8040" LastActivityDate="2015-08-27T10:12:30.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6915" PostTypeId="2" ParentId="6911" CreationDate="2015-08-27T13:03:55.960" Score="4" Body="&lt;p&gt;'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (Seems like everyone has their own version of it these days)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be honest though, if you're asking if it could be helpful, you probably don't need Hadoop (sorry in advance of that comes off harshly, it's not intended to). Lots an lots of places &lt;em&gt;think&lt;/em&gt; they need Hadoop, but very few actually &lt;em&gt;need&lt;/em&gt; Hadoop. There are of business that are down on the tech because they transitioned to it when the need didn't really exist. If you truly do need Hadoop or another distributed system it'd be almost impossible to determine which setup would be beneficial to your org without an intimate understanding of your data, and your specific business means.  &lt;/p&gt;&#xA;" OwnerUserId="11021" LastActivityDate="2015-08-27T13:03:55.960" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6916" PostTypeId="2" ParentId="6910" CreationDate="2015-08-27T13:54:30.117" Score="5" Body="&lt;p&gt;Indeed there are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Gradient Boosting is by construction sequential, so parallelization is not really possible&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Generalized Linear Models need all data at the same time, although technically you can parallelize some of the inner linear algebra nuts and bolts&lt;/li&gt;&#xA;&lt;li&gt;Support Vector Machines&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-08-27T13:54:30.117" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6917" PostTypeId="2" ParentId="6911" CreationDate="2015-08-27T14:08:36.973" Score="1" Body="&lt;p&gt;It depends on what your specific situation is - I think you need to add more detail for you to get a good answer on this. e.g. what problems are you experiencing with data storage currently? What are you trying to change or achieve in the future?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data (you don't mention what exactly the data is, but I assume this is bank transactions, customer info etc?) will be in a traditional relational database management system (RDBMS), probably SQL based, for a very good reason - traditional RDBMSs are extremely good at transactional processing, i.e. being able to add new records, update and delete existing records, extremely quickly and without 'messing up' the data - e.g. if two updates on the same customer records occur almost simultaneously, the database system will order the updates correctly, 'lock' the relevant records to prevent conflicts, etc. so that you are never left with inconsistent data. That would be EXTREMELY bad for a bank - imagine if someone's bank account accidentally got overwritten with the wrong balance! This is generally referred to as OLTP (online transaction processing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason I imagine most banks will continue to use these systems for many many years into the future, which are extremely mature, well tested, well understood, and there are tons of people to hire with the necessary skills. Also, to change the underlying storage system for a banking system would be extremely risky, hence why most banks run very old software - the risk of something going wrong by updating software is just too big.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are talking about just wanting to produce analytical reports on this data, and keep the original source data as-is, then yes Hadoop may well be a useful system to investigate. The general idea will be to export your raw data from your banking system into HDFS for example, and use Hadoop to produce analytical reports. the benefit of this kind of system is that it may be much faster (and safer) to create a query such as 'what is the average bank balance of all my customers', without locking-up your source database system while the query runs (again, that would be really, really bad for a bank!). Hadoop and similar systems excel at this kind of task, generally called OLAP (offline analytical processing), but are no good at all for OLTP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sounds like you need to really think about what you're trying to do here and do lots of research into the differences in OLAP and OLTP as a starter, and go from there. Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="9587" LastActivityDate="2015-08-27T14:08:36.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6918" PostTypeId="2" ParentId="6910" CreationDate="2015-08-27T14:30:25.843" Score="4" Body="&lt;p&gt;I would suggest that you can implement pretty much any kind of data processing you want in Map Reduce given enough time to code it, but the degree of parallelisation you will get will vary depending on what your data is and what you're doing to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine a simple scenario where parallelisation would be reduced dramatically. For example, a simple JOIN task with 4 mapper and 4 reducer nodes will be highly parallelised if the join keys (1, 2, 3 and 4) are evenly distributed across your mapper nodes, and there are the same number of each (4) - 4 reducers would then get an equal share of work for each join key:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mapper    1 2 3 4   |   1 2 3 4   |   1 2 3 4   |   1 2 3 4&#xA;             / \         / \            / \           / \&#xA;            v   v       v   v          v   v         v   v&#xA;&#xA;Reducer   1 1 1 1   |   2 2 2 2   |  3 3 3 3    |   4 4 4 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However if you have a situation where most of your join keys are the same (e.g. nearly all 1s), the reducer dealing with 1s will get swamped:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Mapper    1 1 1 1   |   1 1 1 1   |   1 1 1 1   |   1 1 1 2&#xA;             /          /            /             /     /&#xA;            v          v            v             v     v&#xA;&#xA;Reducer   1 1 1 1 1 1 1 1 1 1 1 1 1 1 1  |  2  |  nothing!  |  nothing!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because 1 reducer gets totally swamped while others do nothing, you will lose most of your parallelisation while Hadoop stalls and waits for the reducer to finish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caveat: There may be more sophisticated Map Reduce implementations out there which deal with this kind of problem, I only know the basics of Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I know this is nothing to do with statistical learning algorithms (which i know almost nothing about) but I imagine the principle is still true - some algorithms or data just won't divide into highly parallel sub-tasks.&lt;/p&gt;&#xA;" OwnerUserId="9587" LastActivityDate="2015-08-27T14:30:25.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6920" PostTypeId="2" ParentId="6907" CreationDate="2015-08-27T19:29:43.663" Score="5" Body="&lt;p&gt;An &quot;outlier&quot; is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is &lt;em&gt;unexpected&lt;/em&gt;? An observation that is highly &lt;em&gt;improbable&lt;/em&gt;? But then how do we know what's probable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you're able or willing to make some assumptions about the distribution that generated these numbers, you can't really declare things outliers. For example, quartiles don't help, since it will only help you find the x% largest or smallest values. But every data set has these -- indeed every data set has a min and max, and being the min or max doesn't mean being an outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, I sense there's an assumption lurking here, that the numbers are probably normally distributed about some mean with roughly some standard deviation. If you know what the mean and/or stdev is supposed to be, you can use it directly to decide how unlikely an observation is and discard it as an outlier if it exceeds a threshold you choose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take the mean and stdev of this &lt;em&gt;sample&lt;/em&gt; as a surrogate for that, and for a large enough sample, the sample mean and stdev could be close enough to the real population mean and stdev to work. Here, with such small sets, the outliers influence stats like the mean so much that they throw off attempts to evaluate them in terms of the stats they influence. It's kind of circular.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-08-27T19:29:43.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6921" PostTypeId="1" AcceptedAnswerId="6923" CreationDate="2015-08-27T19:36:53.297" Score="9" ViewCount="6315" Body="&lt;p&gt;I am looking at how to implement dropout on deep neural networks and found something counter intuitive. In the forward phase dropout mask activations with a random tensor of 1s and 0s to force net to learn the average of the weights. This help the net to generalize better. But during the update phase of the gradient descent the activations are not masked. This to me seems counter intuitive. If I mask connections activations with dropout, why I should not mask the gradient descent phase?&lt;/p&gt;&#xA;" OwnerUserId="10938" LastEditorUserId="29169" LastEditDate="2021-02-18T20:51:03.967" LastActivityDate="2021-02-18T20:51:03.967" Title="Understanding dropout and gradient descent" Tags="|neural-network|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="6922" PostTypeId="1" AcceptedAnswerId="6932" CreationDate="2015-08-27T20:04:03.597" Score="2" ViewCount="271" Body="&lt;p&gt;I am dealing with a data where I have only two documents and there are some words which are present in both. Now Term Frequencies (tf) of these words are very high for respective single document than the other. For e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Word1 is present in Documents D1 and D2, and&#xA;tf(Word1,D1) = 1000&#xA;tf(Word1,D2) = 3&#xA;But since Word1 is present in both the documents &#xA;IDF(Word1) = 0&#xA;TF-IDF(Word1,d) = 0 for all d belonging to {D1,D2}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So inspite having a very powerful presence in a single document, TF-IDF score will always be 0. One solution I could think is to take word1 as absent if &lt;code&gt;tf(Word1) &amp;lt; threshold&lt;/code&gt;. However I still don't feel this is good enough as the Word present in only one single document is given IDF score of only &lt;code&gt;0.5&lt;/code&gt;. I feel like TF-IDF is not a good measurement when number of documents are very low. Any suggestions here?&lt;/p&gt;&#xA;" OwnerUserId="8338" LastEditorUserId="85045" LastEditDate="2021-02-12T12:39:01.683" LastActivityDate="2021-02-12T14:26:38.457" Title="TF-IDF not a strong measure in this senario?" Tags="|text-mining|tfidf|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6923" PostTypeId="2" ParentId="6921" CreationDate="2015-08-27T20:43:32.347" Score="6" Body="&lt;p&gt;In dropout as described in &lt;a href=&quot;http://www.cs.toronto.edu/%7Ehinton/absps/JMLRdropout.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, weights are &lt;em&gt;not&lt;/em&gt; masked. Instead, the neuron activations are masked, &lt;em&gt;per example as it is presented for training&lt;/em&gt; (i.e. the mask is randomised for each run forward and gradient backprop, not ever repeated).&lt;/p&gt;&#xA;&lt;p&gt;The activations are masked during forward pass, and gradient calculations use the same mask during back-propagation of that example. This can be implemented as a modifier within a layer description, or as a separate dropout layer.&lt;/p&gt;&#xA;&lt;p&gt;During weight update phase, typically applied on a mini-batch (where each example would have had different mask applied) there is no further use of dropout masks. The gradient values used for update have already been affected by masks applied during back propagation.&lt;/p&gt;&#xA;&lt;p&gt;I found a useful reference for learning how dropout works, for maybe implementing yourself, is the &lt;a href=&quot;https://github.com/rasmusbergpalm/DeepLearnToolbox&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learn Toolbox&lt;/a&gt; for Matlab/Octave.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="98307" LastEditDate="2020-08-05T19:48:32.903" LastActivityDate="2020-08-05T19:48:32.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6924" PostTypeId="1" CreationDate="2015-08-27T22:06:09.470" Score="5" ViewCount="502" Body="&lt;p&gt;&lt;strong&gt;Tl;DR:&lt;/strong&gt; You can predict something, but how do you explain the prediction?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I have built a website that tries to answer this question with means of embedding / visually clustering data according to the path instances take in a decision tree here: &lt;a href=&quot;http://explicable.ai/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://explicable.ai/&lt;/a&gt;. Please do send me an email if that's something of interest to you!&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;Your usual classification/regression setup&lt;/h3&gt;&#xA;&lt;p&gt;Lets say the data is a classic regression/classification problem: several numerical columns, several nominal columns, and an event which we are trying to predict:&lt;/p&gt;&#xA;&lt;pre&gt;&#xA;user1, age:18, wealth:20000, likes:tomatoes, isInBigCity:yes, hasClicked:yes&#xA;user2, age:25, wealth:24000, likes:carrots , isInBigCity:no , hasClicked:no&#xA;...&#xA;&lt;/pre&gt;&#xA;&lt;p&gt;With the help of Random Forests, SVM, Logistic Regression, Deep Neural Network, or some other method we export a model that can output a probability of the event &lt;code&gt;hasClicked:yes&lt;/code&gt; for a new user faced with the choice of clicking.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;Extracting the inner topology surfaced by a model&lt;/h3&gt;&#xA;&lt;p&gt;Now, those algorithm do some dimensionality reduction, reducing those inputs to a single probability. My question is: how would you extract what those models are doing and show the dimensionality reduction steps to a human? How would you illustrate the inner topology of the dataset with regards to the predicted class?&lt;/p&gt;&#xA;&lt;p&gt;I am looking for either:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Visualizations of what a model produced by Random Forests, SVM, Logistic Regression, Deep Neural Network is doing.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clusterers being extracted from regression/classification models (Surely a single decision tree can be viewed as a hierarchical clusterer)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A model-specific way to project the input data in a space where the Euclidian distance of T-SNE makes sense.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A way to learn a T-SNE-compatible distance out of a regression/classification model.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clustering methods that optimise the separation of one variable while not using it to cluster.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clusterers built out of regression/classification models&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;The goal is to extract some sort market segmentation based on the behaviour of users. And give a high level visualization of it. Something that would expose clearly the &lt;strong&gt;reasons&lt;/strong&gt; why some users transform better than others.&lt;/p&gt;&#xA;&lt;p&gt;EDIT: Let's reduce the question to one estimator: Random Forests; what would be your answer?&lt;/p&gt;&#xA;" OwnerUserId="12485" LastEditorUserId="-1" LastEditDate="2023-05-24T14:51:27.013" LastActivityDate="2024-02-11T15:00:55.087" Title="Illustrating the dimensionality reduction done by a classification or regression model" Tags="|machine-learning|classification|svm|random-forest|dimensionality-reduction|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="6925" PostTypeId="1" CreationDate="2015-08-28T01:07:41.310" Score="0" ViewCount="81" Body="&lt;p&gt;How did the phrase &quot;big data&quot; come about? What were some of the new challenges that arose? How does big data differ from a small amount of data and how do we distinguish the two? &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="12443" LastEditDate="2015-08-31T20:30:18.857" LastActivityDate="2015-08-31T20:30:18.857" Title="What are some challenges with big data?" Tags="|bigdata|" AnswerCount="1" CommentCount="2" ClosedDate="2015-08-28T08:30:10.323" ContentLicense="CC BY-SA 3.0" />
  <row Id="6926" PostTypeId="1" CreationDate="2015-08-28T02:15:14.793" Score="2" ViewCount="1453" Body="&lt;p&gt;I want to cluster the documents I get for Google scholar search using the Bag of words model. I thought of using Java as the language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume for the keyword &lt;strong&gt;&lt;em&gt;k&lt;/em&gt;&lt;/strong&gt;, Google scholar gives me 50 results. If I have a predefined set of words &lt;strong&gt;&lt;em&gt;w1, w2, w3... wn,&lt;/em&gt;&lt;/strong&gt; how can I rank the the documents which have the predefined set of words most? How can I apply Bag of words model for this? Do I need a clustering algorithm like k-means? And do I need to perform NLP techniques as well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that the word &lt;strong&gt;&lt;em&gt;w1&lt;/em&gt;&lt;/strong&gt; has several synonyms. How can I consider those synonyms as well for document ranking? Will I have to create a corpus containing all the abbreviations, synonym etc for that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any good tutorials available for this? Will choosing Java over Python will be an disadvantage since most of the resources (Ex - Scikit) were in Python? &lt;/p&gt;&#xA;" OwnerUserId="12487" LastActivityDate="2015-08-30T11:24:44.310" Title="How to rank documents using Bag of words approach" Tags="|clustering|algorithms|java|information-retrieval|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6927" PostTypeId="2" ParentId="6925" CreationDate="2015-08-28T03:17:25.713" Score="1" Body="&lt;p&gt;There has/is been a lot of hype around the word &quot;Big Data&quot;. So, as someone who has been setting up an analytics infrastructure at a fast growing company, I would like to share my experiences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would like to check &lt;a href=&quot;https://stats.stackexchange.com/questions/18276/what-data-and-statistics-skills-are-currently-in-high-demand-and-where-are-they/164861#164861&quot;&gt;this&lt;/a&gt; out, which I have written for an almost similar question on CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the tools like H2O, R, Julia, Spark, etc; are being evolving for dealing and playing with Big Data, the size of the data is becoming a less of a problem these days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As there is a lots of data available, and a wonderful set of tools at disposal, the art of analytics and data science is becoming more of a pleasure for data scientists rather than pain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, the real challenge is the availability of people who know what they are doing, who know how to use these tools for mining gold from the data.&#xA;Just anyone who has run a Linear Regression on a data set can't call himself a data scientist. One has to explain what the data is, what can be pulled out of it, and why is the algorithm applied on it, and why not something else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do they differ from a small amount of data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, there isn't any huge differences. The only differences which I have found are &lt;a href=&quot;http://people.seas.harvard.edu/~minilek/cs229r/fall15/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;algorithms&lt;/a&gt; which help find the needle in the hay stack, like MinHash, HyperLogLog, etc; which had to evolve for dealing with the size.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-08-28T03:17:25.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6928" PostTypeId="2" ParentId="6926" CreationDate="2015-08-28T08:56:19.443" Score="2" Body="&lt;p&gt;If you're just looking to rank documents according to how many appearances your words w1,..,wn contain, then there's no need for clustering or machine learning in general: Clustering your 50 results will give you a partition of these results into clusters containing results that are similar to one another and different from the results in other clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just need to rank by word occ, just count the frequencies of your words in each document (including synonyms, which you can get e.g. from Wordnet automatically if you prefer) and sum them up. &lt;/p&gt;&#xA;" OwnerUserId="12492" LastActivityDate="2015-08-28T08:56:19.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6929" PostTypeId="1" AcceptedAnswerId="6933" CreationDate="2015-08-28T12:56:52.233" Score="4" ViewCount="171" Body="&lt;p&gt;Hello I am a layman trying to analyze game data from League of Legends, specifically looking at predicting the win rate for a given champion given an item build.&lt;/p&gt;&#xA;&lt;h3&gt;Outline&lt;/h3&gt;&#xA;&lt;p&gt;A player can own up to 6 items at the end of a game. They could have purchased these items in different orders or adjusted their inventory position during the course of the game.&lt;/p&gt;&#xA;&lt;p&gt;In this fashion the dataset may contain the following rows with:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;   champion id   |                 items ids               | win(1)/loss(0)&#xA;----------------------------------------------------------------------------&#xA;       45        |   [3089, 3135, 3151, 3157, 3165, 3285]  |       1&#xA;       45        |   [3151, 3285, 3135, 3089, 3157, 3165]  |       1&#xA;       45        |   [3165, 3285, 3089, 3135, 3157, 3151]  |       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;While the items are in a different order the build is the same, my initial thought would be to simply multiply the item ids as this would give me an integer value representing that combination of 6 items.&lt;/p&gt;&#xA;&lt;p&gt;While there are hundreds of items, in reality a champion draws off a small subset (~20) of those to form the core (3 items) of their build. A game may also finish before players have had time to purchase 6 items:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;                items ids               &#xA;------------------------------------------&#xA;   [3089, XXXX, 3151, 3285, 3165, 0000]&#xA;   [XXXX, 3285, XXXX, 3165, 3151, 0000]&#xA;   [3165, 3285, 3089, XXXX, 0000, 0000]&#xA;&#xA;XXXX item from outside core subset&#xA;0000 empty inventory slot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As item 3089 compliments champion 45 core builds that have item 3089 have a higher win rate than core builds which are missing item 3089.&lt;/p&gt;&#xA;&lt;p&gt;The size of the data set available for each champion varies between 10000 and 100000. The mean is probably around 35000.&lt;/p&gt;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this a suitable problem for supervised classification?&lt;/li&gt;&#xA;&lt;li&gt;How should I approach finding groups of core items and their win rates?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12497" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-08-28T18:07:22.253" Title="Classification problem where one attribute is a vector" Tags="|machine-learning|classification|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6930" PostTypeId="2" ParentId="6751" CreationDate="2015-08-28T14:17:43.353" Score="3" Body="&lt;p&gt;One way to do this is to use context information to represent each word along with the w2v vector. You can choose to represent this information in any way you like: add another 600 dimensions (100D w2v vectors for 3 left and 3 right context words), just another 100D as the sum of context vectors or any other fixed length representation for your context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you're training, you can use a version of 'word dropout' that will utilize this information. 20% of the time, set your w2v vectors to zero, forcing your classifier to use the context dimensions to represent the word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you encounter a new word, the hope is, the classifier learned to use the context information as well as it learned to use the w2v information.&lt;/p&gt;&#xA;" OwnerUserId="12501" LastActivityDate="2015-08-28T14:17:43.353" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6931" PostTypeId="1" AcceptedAnswerId="6941" CreationDate="2015-08-28T14:53:34.843" Score="0" ViewCount="1066" Body="&lt;p&gt;I tried SOMpy, though it is very crude now and works only with oldest versions of matplotlib. &#xA;Is there any fancy lib that can build SOM based on array and visualize it in Python? &lt;/p&gt;&#xA;" OwnerUserId="12503" LastActivityDate="2020-08-03T08:57:42.937" Title="Any usable libs to build and visualise SOM in python?" Tags="|python|visualization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6932" PostTypeId="2" ParentId="6922" CreationDate="2015-08-28T15:06:02.470" Score="3" Body="&lt;p&gt;Weighting scheme 2 in table &lt;strong&gt;Recommended TF-IDF weighting schemes&lt;/strong&gt; in &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf–idf | Wikipedia&lt;/a&gt; should solve your problem.&lt;/p&gt;&#xA;" OwnerUserId="12504" LastEditorUserId="29575" LastEditDate="2021-02-12T14:26:38.457" LastActivityDate="2021-02-12T14:26:38.457" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6933" PostTypeId="2" ParentId="6929" CreationDate="2015-08-28T17:38:30.597" Score="4" Body="&lt;p&gt;1) If you want to build a model with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: Items bought&#xA;Output: Win/Loss&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then you will probably want to learn a non-linear combination of the inputs to represent a build.  For example &lt;code&gt;item_X&lt;/code&gt; may have very different purpose when paired with &lt;code&gt;item_Y&lt;/code&gt; than with &lt;code&gt;item_Z&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the input format, you may consider creating a binary vector from the item list.  For example if there were only ten items, a game in which the champion purchased &lt;code&gt;items 1,4,5,9&lt;/code&gt; (in any order) would look like row 1; a game where he also purchased &lt;code&gt;item 2&lt;/code&gt; and &lt;code&gt;7&lt;/code&gt; would look like row 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;item_ID   | 0  1  2  3  4  5  6  7  8  9 &#xA;________________________________________    &#xA;champion_1| 0  1  0  0  1  1  0  0  0  1&#xA;champion_1| 0  1  1  0  1  1  0  1  0  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are a variety of models that might suit this task.  You might use &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;nofollow&quot;&gt;decision trees&lt;/a&gt; for interpretability.  A simple &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow&quot;&gt;neural net&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVM&lt;/a&gt; would likely also do a good job.  These should all be found in most basic ML packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The win rates of various items are directly computable.  Simply count the number of times a champion used the items in question and won and divide by the total number of times a champion used that item combination.  You can do this for any given group size (1 to 6)&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-08-28T18:07:22.253" LastActivityDate="2015-08-28T18:07:22.253" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6937" PostTypeId="1" CreationDate="2015-08-29T00:46:37.010" Score="6" ViewCount="10046" Body="&lt;p&gt;I've taken it upon myself to begin a career change. I have a decent background in mathematics, but lack in programming or data science specific skills (such as data munging). I have been looking through data science curricula and already feel overwhelmed by the amount of subjects a proper data science curriculum contains. Just to name a few: natural language processing, machine learning, R, Python, SQL, NoSQL, probability, statistics, numerical methods, algorithms, and the list goes on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't afford to go back to school, but if my goal is to get into data science, &lt;em&gt;what subjects should I focus my attention on&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fully understand &quot;data science&quot; is a very big field, but certainly there must be several subjects that are important to all, if not the vast majority, of these sub-fields. I suppose this is essentially what my question is asking.  I have not yet obtained the amount of exposure to determine what I want to specialize in, so at this point, my concern is obtaining a foundation that wouldn't restrict me from entering any particular sub-field, or in other words, studying the subjects that benefit all sub-fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Math isn't as much a concern for me compared to the other subjects.&lt;/p&gt;&#xA;" OwnerUserId="12516" OwnerDisplayName="tofu_bacon" LastEditorUserId="8820" LastEditDate="2016-02-08T05:46:24.050" LastActivityDate="2016-02-08T05:46:24.050" Title="What courses / subjects are most important to the field of Data Science?" Tags="|self-study|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6938" PostTypeId="2" ParentId="6274" CreationDate="2015-08-29T08:55:58.290" Score="1" Body="&lt;p&gt;&lt;strong&gt;Lets break this down...&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have 3 numerical attributes.  Great... &lt;a href=&quot;http://www.dataminingblog.com/standardization-vs-normalization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;standardize them&lt;/a&gt; by subtracting their means and dividing by their standard deviation.  You always need to standardize when you cluster in multiple dimensions or your clustering won't make much sense i.e. the distance vector only makes sense if it is agnostic to how it is oriented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have 2 categorical attributes.  One, however, is time, which doesn't seem very categorical.  Figure out how to turn time into a numerical value.  I suggest using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Unix_time&quot; rel=&quot;nofollow noreferrer&quot;&gt;unix time stamp&lt;/a&gt;.  Don't just blindly call it a categorical feature.  Your csv reader didn't know how to treat the colon the slash, so called it a factor, but you can convert it &lt;a href=&quot;https://stackoverflow.com/questions/23348992/convert-factor-to-date-time-in-r&quot;&gt;pretty easily and quickly&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would suggest first cluster the 3 numerical factors on their own.  Then I would do some &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_extraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature extraction&lt;/a&gt; from the time data.  You should be able to extract day of week, week of month, day of month, etc.  These could all be useful in seeing some sort of signal in your data.  I would also suggest plotting your numerical data as a function of these extracted features and this will give you insight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, think about the value that customer ID might or might not add to the problem.  You might be able to remove the letters and it could tell you when they first shopped at the store, or it could just contain useless information that will cloud your clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, think about why you are clustering your data and what insights you want to recover.  This will guide you in how to proceed.  Do you want to separate predominantly weekend shoppers from predominantly weekday shoppers?  Do you want to broadly classify shoppers as one of ten different types and try to ascertain the behavior of each of those types?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-08-29T08:55:58.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6939" PostTypeId="1" AcceptedAnswerId="8224" CreationDate="2015-08-29T10:27:02.597" Score="4" ViewCount="12505" Body="&lt;p&gt;Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the number of positive data sample to get the negative data sample. Will such a data set be useful?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="97" LastEditDate="2015-09-28T21:17:48.647" LastActivityDate="2021-02-12T12:15:39.243" Title="Ratio of positive to negative sample in data set for best classification" Tags="|classification|dataset|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6940" PostTypeId="2" ParentId="6939" CreationDate="2015-08-29T12:08:45.923" Score="1" Body="&lt;p&gt;It is recommended to have around equal number of instances in each class. If not, then you should normalize the data. 4 or 5 times more positive class size could produce highly biased classifier. However, if your negative class is not well defined then there are ways to train your classes with only one class as mentioned in the paper, &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2248178/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning from positive examples when the negative class is undetermined- microRNA gene identification&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8338" LastEditorUserId="85045" LastEditDate="2021-02-12T09:48:30.507" LastActivityDate="2021-02-12T09:48:30.507" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6941" PostTypeId="2" ParentId="6931" CreationDate="2015-08-29T12:29:50.363" Score="1" Body="&lt;p&gt;You could have a try on &lt;a href=&quot;http://www.pymvpa.org/examples/som.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this package&lt;/a&gt;.&lt;br /&gt;&#xA;There is a working example on this page.&lt;/p&gt;&#xA;&lt;p&gt;If what you are interested is the Manifold learning,&#xA;you could also apply &lt;a href=&quot;http://scikit-learn.org/stable/modules/manifold.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;many packages&lt;/a&gt; from sklearn.&lt;/p&gt;&#xA;" OwnerUserId="12237" LastEditorUserId="98307" LastEditDate="2020-08-03T08:57:42.937" LastActivityDate="2020-08-03T08:57:42.937" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6942" PostTypeId="2" ParentId="6899" CreationDate="2015-08-29T13:05:40.547" Score="0" Body="&lt;p&gt;I might suggest that you use a greedy methods. Give a classifier to start, you will include the classifier that make the ensemble get the best performance improvement. If no improvement could be get through include more classifiers, then stop. You will start with every classifiers. The complexity will be at most N*N. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have one more question, What do you mean by &quot;Pareto optimal&quot;, especially in your context? I found from wiki this explanation, &lt;a href=&quot;https://en.wikipedia.org/wiki/Pareto_efficiency&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Pareto_efficiency&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;through reallocation, improvements can be made to at least one participant's well-being without reducing any other participant's well-being.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The improvement for the Pareto efficiency is for each participant, which might correspond to each classifier. How do you define the improvement over one classifier?&lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2015-08-29T13:05:40.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6943" PostTypeId="2" ParentId="5178" CreationDate="2015-08-29T21:44:24.070" Score="6" Body="&lt;p&gt;We don't version control the actual data files. We wouldn't want to even if we stored it as CSV instead of in a binary form. As &lt;a href=&quot;https://datascience.stackexchange.com/a/6771/12328&quot;&gt;Riccardo M.&lt;/a&gt; said, we're not going to spend our time reviewing row-by-row changes on a 10M row data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, along with the processing code, I version control the metadata:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Modification date&lt;/li&gt;&#xA;&lt;li&gt;File size&lt;/li&gt;&#xA;&lt;li&gt;Row count&lt;/li&gt;&#xA;&lt;li&gt;Column names&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This gives me enough information to know &lt;strong&gt;if&lt;/strong&gt; a data file has changed and an idea of &lt;strong&gt;what&lt;/strong&gt; has changed (e.g., rows added/deleted, new/renamed columns), without stressing the VCS.&lt;/p&gt;&#xA;" OwnerUserId="12328" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-08-29T21:44:24.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6944" PostTypeId="1" CreationDate="2015-08-29T21:59:07.543" Score="1" ViewCount="1242" Body="&lt;p&gt;I need to decompose a series to remove seasonality. The series has 2 columns date and volume.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what my time series object looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;salestsDec &amp;lt;- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I ran the &lt;code&gt;decompose()&lt;/code&gt; function on a 'ts' object.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;salests = sales[, c(1,6)]&#xA;View(salests)&#xA;salestsDec &amp;lt;- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))&#xA;salestsDec &amp;lt;- decompose(salestsDec, type=c(&quot;additive&quot;))&#xA;plot(salestsDec)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Upon, running the &lt;code&gt;decompose()&lt;/code&gt; function, I get a list of 6 components, observed, trend, seasonal, random for both date and volume. I should only be seeing, observed, trend, seasonal and random component for Volume in my plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've attached an image of what the plot looks like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, when I try to remove seasonal component from the series, I am getting an error. It appears that it's the same underlying issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error in salests - salestsDec$seasonal : non-numeric argument to binary operator In addition: Warning message: Incompatible methods (&quot;Ops.data.frame&quot;, &quot;Ops.ts&quot;) for &quot;-&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pxzjf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pxzjf.png&quot; alt=&quot; Time series components plot after running decomposed() &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12533" LastEditorUserId="10253" LastEditDate="2015-08-30T18:24:37.093" LastActivityDate="2015-08-31T10:39:00.730" Title="remove seasonality from weekly time series data" Tags="|r|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6945" PostTypeId="2" ParentId="6937" CreationDate="2015-08-30T02:27:31.543" Score="3" Body="&lt;p&gt;&quot;Data Science&quot; is not very well defined term and people mean different things by it. For somebody it might mean working with huge data sets on distributed systems, to somebody it is basically data analysis, to somebody it is writing predictive models using sparse matrices (that's me). Therefore, you should approach what interests you. You don't have to know everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, Data Science is combination of two fields: Computer Science and Statistics. Fundamental understanding of both is crucial. &lt;/p&gt;&#xA;" OwnerUserId="3070" LastActivityDate="2015-08-30T02:27:31.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6946" PostTypeId="2" ParentId="6937" CreationDate="2015-08-30T03:20:29.350" Score="5" Body="&lt;p&gt;Probably the most useful (and fun) answer is provided by the excellent &lt;a href=&quot;http://nirvacana.com/thoughts/becoming-a-data-scientist/&quot; rel=&quot;noreferrer&quot;&gt;Swami Chandrasekaran&lt;/a&gt; ... in the form of a subway map:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bBuTr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bBuTr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would add just one comment to that:  school (or learning for the sake of learning) doesn't provide the same kind of experience as solving a real problem does.  Therefore, to learn, find real problems you can tackle.  To stay motivated, make them problems you really care about.  Kaggle competitions are a great place to start.  Even reproducing a winning solution (perhaps with some variations, or on a new dataset) will be a huge learning experience.&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2015-08-30T03:31:45.827" LastActivityDate="2015-08-30T03:31:45.827" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6947" PostTypeId="1" CreationDate="2015-08-30T05:14:24.837" Score="1" ViewCount="278" Body="&lt;p&gt;I have an extensive series of projects that require importing data from many different types of different sources (websites, web apis, sensors, legacy text files, etc). Is there a good framework (preferably open source) that has been built bottom up to tackle this type of problem? Preferably one leveraging an appropriate language (python, most likely) and already having an extensive plugin database.&lt;/p&gt;&#xA;" OwnerUserId="12536" LastActivityDate="2015-09-30T21:12:32.543" Title="Looking for language and framework for data munging/wrangling" Tags="|data-mining|data-cleaning|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6948" PostTypeId="2" ParentId="6926" CreationDate="2015-08-30T11:24:44.310" Score="0" Body="&lt;p&gt;If you are just looking to rank documents, @Sharon answer is what you need (+1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you have ranked them, indeed the next logical step would be to cluster the documents, to find out which of them are similar. In particular, if your list of words is heterogeneous, the rank by itself might not give you very clear information. This is quite a good &lt;code&gt;python notebook&lt;/code&gt;, everything is kept simple but at the same time it explains all the steps you should follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nbviewer.ipython.org/github/brandomr/document_cluster/blob/master/cluster_analysis_web.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Document clustering in Python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just wanted to detect the topics of your document (i.e. without your list of words), this is a good &lt;a href=&quot;https://stats.stackexchange.com/questions/155880/text-mining-how-to-cluster-texts-e-g-news-articles-with-artificial-intellige&quot;&gt;SO answer&lt;/a&gt;, which includes links to resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your question about Java v. Python, I cannot tell you for sure that Java would be a disadvantage. Even though all the links I provide are in python, I'm certain there's quite a few &lt;code&gt;nl&lt;/code&gt; resources in Java. However I don't know them well and I cannot compare.&lt;/p&gt;&#xA;" OwnerUserId="9584" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-08-30T11:24:44.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6949" PostTypeId="1" CreationDate="2015-08-31T05:45:00.513" Score="7" ViewCount="10546" Body="&lt;p&gt;I have an idea of how ADABOOST will be used for classification but I want to get the idea of how to re-weight and thus use ADABOOST in case of regression problems.&lt;/p&gt;&#xA;" OwnerUserId="12411" LastActivityDate="2020-08-18T04:19:34.023" Title="How will ADA Boost be used for solving regression problems?" Tags="|machine-learning|data-mining|bigdata|regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6950" PostTypeId="1" AcceptedAnswerId="6951" CreationDate="2015-08-31T10:13:43.080" Score="0" ViewCount="1248" Body="&lt;p&gt;I'm performing a logistic regression on my training data. I used the &lt;em&gt;glm&lt;/em&gt; function to get the model &lt;em&gt;m&lt;/em&gt;. Now using the below codes from this &lt;a href=&quot;https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;, I calculated AUC&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$test\$score&amp;lt;-predict(m,type = 'response',test)$&lt;br&gt;&#xA;  $pred &amp;lt;- prediction(test\$score,test\$good_bad)$&lt;br&gt;&#xA;  $perf &amp;lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)$  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where &lt;em&gt;score&lt;/em&gt; is the dependent variable (&lt;em&gt;0&lt;/em&gt; or &lt;em&gt;1&lt;/em&gt;).&lt;br&gt;&#xA;To score the &lt;em&gt;tpr&lt;/em&gt; (True positive rate) and &lt;em&gt;fpr&lt;/em&gt; (False positive rate), you have to classify the predicted probabilities into 1 or 0.&lt;br&gt;&#xA;What is the cutoff used for that? how can we change it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could not find anything useful in this &lt;a href=&quot;http://www.hpl.hp.com/techreports/2003/HPL-2003-4.pdf&quot; rel=&quot;nofollow&quot;&gt;main documentation&lt;/a&gt; as well.&lt;/p&gt;&#xA;" OwnerUserId="10050" LastActivityDate="2016-10-08T04:22:42.267" Title="How does Performance function classify predictions as positive or negative? Package:ROCR" Tags="|r|classification|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6951" PostTypeId="2" ParentId="6950" CreationDate="2015-08-31T10:33:28.057" Score="0" Body="&lt;p&gt;I cant access an R console at the moment to check, but I'm quite certain the cutoff is 0.5: if your glm model does prediction, it first produces real values and then applies the link function on top. To the best of my knowledge, you can't change it inside the glm function, so your best bet is probably to check ROC, find what the optimal threshold is and use that as cutoff. &lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-08-31T10:33:28.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6952" PostTypeId="2" ParentId="6944" CreationDate="2015-08-31T10:39:00.730" Score="1" Body="&lt;p&gt;I can't replicate (without having the data), but from what I see, you are trying to apply the &lt;em&gt;decompose&lt;/em&gt; function to the entire object - it's probably expecting a single vector, but receives a data-frame-like object and attempts to apply the decomposition to both columns. either use a different time series package (like zoo) which does not store timestamps as a column, or apply decompose explicitly to salestsDec[,2].&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-08-31T10:39:00.730" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6953" PostTypeId="2" ParentId="6949" CreationDate="2015-08-31T10:42:41.170" Score="0" Body="&lt;p&gt;AdaBoost is a meta algorithm, so the underlying principle is the same: i would suggest going through &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html#adaboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for references.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastEditorUserId="98307" LastEditDate="2020-08-18T04:19:34.023" LastActivityDate="2020-08-18T04:19:34.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6954" PostTypeId="1" CreationDate="2015-08-31T13:02:02.490" Score="1" ViewCount="6317" Body="&lt;p&gt;I am trying to classify set of text files whether it belong to category A or category B using KNN algorithm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the error that i get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; knn.pred &amp;lt;- knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand) :      dims of&#xA;  'test' and 'train' differ&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; dim(tdm.stack.nl)  &#xA;    [1]  184 1599  &#xA;    &amp;gt; dim(tdm.stack.nl_test)  &#xA;    [1]   1 992  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As obvious the number of words used by training dataset is different from train. How can I avoid this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to mention that this error i get when i divide the training and test dataset manually to (70:30) ratio. However i dont get this error if i divide the training and test dataset using samlpling method.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; # create hold-out   &#xA;&amp;gt; set.seed(500)  &#xA;&amp;gt; train.idx &amp;lt;- sample(nrow(tdm.stack), ceiling(nrow(tdm.stack) * 0.7))   &#xA;&amp;gt; test.idx &amp;lt;- (1:nrow(tdm.stack))[-train.idx]   &#xA;&amp;gt; &#xA;&amp;gt; # create model - knn clustering&#xA;&amp;gt; tdm.cand &amp;lt;- tdm.stack[, &quot;targetcandidate&quot;]&#xA;&amp;gt; tdm.stack.nl &amp;lt;- tdm.stack[, !colnames(tdm.stack) %in% &quot;targetcandidate&quot;]&#xA;&amp;gt; &#xA;&amp;gt; # set up model&#xA;&amp;gt; knn.pred &amp;lt;- knn(tdm.stack.nl[train.idx,], tdm.stack.nl[test.idx,],    tdm.cand[train.idx])  &#xA;&amp;gt; dim(tdm.stack.nl[train.idx,])  &#xA;[1]  129 1599  &#xA;&amp;gt; dim(tdm.stack.nl[test.idx,])  &#xA;[1]   55 1599   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However this method will not work for me as I want to classify a document in a real time using the model that was built before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help. Kindly let me know if you need any other info. &#xA;Arun&lt;/p&gt;&#xA;" OwnerUserId="12554" LastActivityDate="2019-11-23T14:32:47.967" Title="R - Error in KNN - Test and training differ" Tags="|machine-learning|data-mining|r|classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6955" PostTypeId="2" ParentId="6949" CreationDate="2015-08-31T15:08:04.943" Score="0" Body="&lt;p&gt;As a general principle Adaboost builds and ensemble by sequentially adding members which have been trained on those instances of data which are proving most difficult to correctly classify/predict. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each new classifier/predictor is given a training set where the difficult examples are increasingly represented, this is achieved either through weighting or resampling.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There should be very little difference in the approach for classification or regression.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-08-31T15:08:04.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6956" PostTypeId="2" ParentId="6954" CreationDate="2015-08-31T15:18:41.987" Score="1" Body="&lt;p&gt;You are avoiding the error with sampling purely by chance (the words are common). Simplest way would be to merge training and test, construct a tdm on a joined set, separate into training and test again and then purge some columns (the constant ones in the train set, as they correspond to words occurring in test only =&gt; useless for training).&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-08-31T15:18:41.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6958" PostTypeId="2" ParentId="6949" CreationDate="2015-08-31T18:26:55.673" Score="5" Body="&lt;p&gt;Here are link to some famous boost of regressor.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.314&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adaboost.R2: Improving Regressors using Boosting Techniques&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.4560&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adaboost.RT: Experiments with AdaBoost.RT, an Improved Boosting&#xA;Scheme for Regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Scikit-Learn have many implementations:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AdaBoostRegressor&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_adaboost_regression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision Tree Regression with AdaBoost&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12237" LastEditorUserId="98307" LastEditDate="2020-08-18T02:31:29.867" LastActivityDate="2020-08-18T02:31:29.867" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="6959" PostTypeId="1" CreationDate="2015-08-31T18:31:59.680" Score="0" ViewCount="940" Body="&lt;p&gt;I am using rpart to classify the documents whether it belongs to class a or class b. I want to prove or provide reasons  on what basis the document was classified correctly or wrongly. How can i do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I displayed the summary of the model. But it was not so convincing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help&lt;/p&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="843" LastEditDate="2017-03-08T15:21:44.533" LastActivityDate="2017-03-08T15:21:44.533" Title="r - rpart in text mining for classification of documents" Tags="|machine-learning|r|classification|rattle|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6960" PostTypeId="1" AcceptedAnswerId="7000" CreationDate="2015-08-31T18:39:36.183" Score="2" ViewCount="5644" Body="&lt;p&gt;I am trying to predict tags for stackoverflow questions and I am not able to decide which Machine Learning algorithm will be a correct approach for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input:&lt;/strong&gt; As a dataset I have mined stackoverflow questions, I have tokenized the data set and removed stopwords and punctuation from this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Things i have tried:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;TF-IDF&lt;/li&gt;&#xA;&lt;li&gt;Trained Naive Bayes on the dataset and then gave user defined input to predict tags, but its not working correctly&lt;/li&gt;&#xA;&lt;li&gt;Linear SVM&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which ML algorithm I should use Supervised or Unsupervised?&#xA;If possible please, suggest a correct ML approach from the scratch.&#xA;PS: I have the list of all tags present on StackOverflow so, will this help in anyway?&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="12557" LastActivityDate="2015-09-03T22:30:58.170" Title="StackOverflow Tags Predictor...Suggest an Machine Learning Approach please?" Tags="|machine-learning|classification|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="6961" PostTypeId="2" ParentId="6947" CreationDate="2015-08-31T19:09:54.973" Score="0" Body="&lt;p&gt;All of the scientific computing development going on around Python is a good option. &lt;a href=&quot;https://store.continuum.io/cshop/anaconda/&quot; rel=&quot;nofollow&quot;&gt;Continuum's Anaconda&lt;/a&gt; is very easy to get up and running. Also well supported in distributed environments.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Numpy for fast matrix operations&lt;/li&gt;&#xA;&lt;li&gt;Pandas for dataframes, munging, analytics, visualization&lt;/li&gt;&#xA;&lt;li&gt;Statsmodels and Scikit for models, metrics, ML algs and so on&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12388" LastActivityDate="2015-08-31T19:09:54.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6964" PostTypeId="1" CreationDate="2015-08-31T20:58:15.070" Score="12" ViewCount="32476" Body="&lt;p&gt;I saw some impressive result from LSTM models producing Shakespeare like texts. I was wondering if an LSTM package exists for R. I googled for it but only found packages for Python and Julia. (maybe there are some performance issue which explains why these programs are more preferred over R) Do You know about an LSTM (or at least an RNN) package for R? If exists are there any tutorials for using them?&lt;/p&gt;&#xA;" OwnerUserId="5211" LastEditorUserId="11097" LastEditDate="2016-03-08T06:53:05.527" LastActivityDate="2017-10-19T11:53:49.823" Title="LSTM or other RNN package for R" Tags="|r|neural-network|rnn|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6967" PostTypeId="2" ParentId="6959" CreationDate="2015-09-01T04:24:16.963" Score="0" Body="&lt;p&gt;In machine learning, we usually call &quot;correctly&quot; and &quot;wrongly&quot; as sensitivity, specificity, prediction accuracy etc. Please &lt;a href=&quot;https://en.wikipedia.org/wiki/Sensitivity_and_specificity&quot; rel=&quot;nofollow&quot;&gt;read&lt;/a&gt; this Wikipedia page for more details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use hold a validation subset by cross-validation to validate your model.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2015-09-01T04:24:16.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6969" PostTypeId="2" ParentId="6960" CreationDate="2015-09-01T09:08:25.243" Score="0" Body="&lt;p&gt;One interesting algorithm that I've once tested is called TopMine: &lt;a href=&quot;http://web.engr.illinois.edu/~elkishk2/&quot; rel=&quot;nofollow&quot;&gt;http://web.engr.illinois.edu/~elkishk2/&lt;/a&gt; (under Code and Datasets). It is able to extract bi-grams that could be used as key words and it can also assign them into topics.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2015-09-01T09:08:25.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6970" PostTypeId="1" AcceptedAnswerId="6979" CreationDate="2015-09-01T09:57:10.733" Score="5" ViewCount="11059" Body="&lt;p&gt;I am new to machine learning and I am confused with the terminology. Thus far, I used to view a hypothesis class as different instance of hypothesis function... Example: If we are talking about linear classification then different lines characterized by different weights would together form the hypothesis class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my understanding correct or can a hypothesis class represent anything which could approximate the target function?  For instance, can a linear or quadratic function that approximates the target function together form a single hypothesis class or both are from different hypothesis classes?&lt;/p&gt;&#xA;" OwnerUserId="12569" LastEditorUserId="9420" LastEditDate="2015-09-01T18:32:16.947" LastActivityDate="2015-09-01T21:54:01.573" Title="Newbie: What is the difference between hypothesis class and models?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6971" PostTypeId="1" CreationDate="2015-09-01T13:12:35.833" Score="1" ViewCount="246" Body="&lt;p&gt;Given: a data frame of 2.5 million records and 25 columns, which contains 1 grouping column with a variable number of groups (e.g. 2.5M records / 10 groups today; 2.5M records / 50 groups tomorrow).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I build a decision tree for each group?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was going to do this using Python but thought there might be an easier way in R (tapply?). Here was my initial thought process in Python:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;instantiate an empty dictionary to hold the prediction results&lt;/li&gt;&#xA;&lt;li&gt;identify the unique ids in the grouping column and iterate over the rows of data in a &lt;code&gt;for&lt;/code&gt; loop&lt;/li&gt;&#xA;&lt;li&gt;in each loop, build the decision tree, predict and store the results in the dictionary as a &lt;code&gt;{grouping_var:prediction}&lt;/code&gt; pair&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thought on how to do this faster or more efficiently in R?&lt;/p&gt;&#xA;" OwnerUserId="12572" LastActivityDate="2015-09-01T13:12:35.833" Title="How to build a unique decision tree for each subset of data based on a grouping variable?" Tags="|r|python|decision-trees|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6972" PostTypeId="1" AcceptedAnswerId="6981" CreationDate="2015-09-01T13:56:11.747" Score="3" ViewCount="1345" Body="&lt;p&gt;I wasn't sure if I had to ask it here or in Stackoverflow, but since I am also seeking research papers/algorithms and not only code, I decided to do it here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I have a text, I can manually write a regex to find all the possible outputs from what I want to extract from the file. What I want to do, is to find an algorithm or a research, which can let you highlight (set the input) different positions of the same (repeated) data you want to extract in the text file, train the algorithm and then identify all the others under the same contentions of those you set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say that I have a text with several titles which are following with &lt;code&gt;\n\n\n&lt;/code&gt; and starting with &lt;code&gt;\n\n&lt;/code&gt;. It is easy with regex, but I want to do it dynamically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An idea is to build an algorithm which will take examples and create regex automatically. But I am not aware of any research like this and maybe there are also other techniques that you can achieve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-11-01T01:26:14.960" Title="Identify given patterns in unstructured data like text files" Tags="|text-mining|algorithms|research|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6974" PostTypeId="2" ParentId="6960" CreationDate="2015-09-01T14:38:45.337" Score="0" Body="&lt;p&gt;If you have the tags of each of the questions that you mined, then supervised methods make sense.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might use the tf-idf representation of a given question, feed it to an SVM or neural net, and use that to predict 0/1 for each tag in your target set.  If there are too many possible classes (tags), it may be tricky to balance your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simpler approach might be to use the tf-idf vectors to compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow&quot;&gt;K-Nearest-Neighbors&lt;/a&gt; of a question; then you can use the tags of the most similar documents (by whatever &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_distance&quot; rel=&quot;nofollow&quot;&gt;distance metric&lt;/a&gt; does best) to predict the likelihood that question has each tag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have the tags for the questions you mined, you should consider unsupervised methods.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt;, for example, could identify &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topics&lt;/a&gt; within the questions and important words within those topics &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-09-01T14:38:45.337" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="6975" PostTypeId="2" ParentId="5547" CreationDate="2015-09-01T17:14:03.977" Score="1" Body="&lt;p&gt;Just out of curiousity: why use 3-channel images? I work in CV as well and from what I've seen, it's standard to grayscale the images. Especially with MNIST, which is not colored to begin with, it doesn't seem that using color has any benefit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also believe that there's an intuitive reason to use grayscale - often times color is a confounding variable. If you were to look at a red &quot;1&quot; and and blue &quot;1&quot;, you'd say &quot;Hey! Those are both ones! They're just different colors&quot;. However, computers are much stupider than you or I. Even with intelligent algorithms, a computer may still end up in the situation where it's looking at a blue &quot;1&quot; and say to itself, &quot;Oh crap, what is that?!? Why is it blue? I've never seen anything like this before!&quot; Keep in mind how large of a difference there is from the color (255, 0, 0) and (0, 255, 255) and how slight the differences in images can be (especially outside the comfortable playground of MNIST). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any rate, most of the NoLearn examples I've seen use data shapes like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;X = X.reshape(-1, 1, size, size)&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you grayscale your images you can coerce it into this shape. Unfortunately, I'm not sure of how you'd cram your colored data into noLearn and get the results you're looking for. &lt;/p&gt;&#xA;" OwnerUserId="12575" LastEditorUserId="12575" LastEditDate="2015-09-30T16:49:14.207" LastActivityDate="2015-09-30T16:49:14.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6978" PostTypeId="2" ParentId="6937" CreationDate="2015-09-01T18:53:47.930" Score="1" Body="&lt;p&gt;I am always inclined to the &lt;strong&gt;&quot;do while you learn&quot;&lt;/strong&gt; principle, when learning new skills. I am writing this completely based on my experience, which helped me solve Kaggle problems and do what I am doing now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this is roughly how you do it, and the courses which you need to take:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Introduction to Machine Learning&lt;/strong&gt; - And the best course for studying this is Andrew Ng's coursera course. You don't need to complete the HW's etc. It is self-paced and archived. So, just watch the courses, and do the assignments for a hands-on knowledge. I even tried doing the same assignments in both Octave and Python. Completing this course would also help you solve Kaggle problems.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Exploratory Analysis&lt;/strong&gt; - Doing preliminary cleaning and exploratory analysis is very important in every data science problem. So, learning those methods would help in learning visualizations, exploratory techniques etc. This &lt;a href=&quot;https://www.coursera.org/course/introstats&quot; rel=&quot;nofollow&quot;&gt;Coursera course&lt;/a&gt; is a nice one.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Databases&lt;/strong&gt; - Data is not always available in neat tsv's or csv's like in online courses. One needs to learn to handle data stored in databases. They might be SQL based, or NoSQL, they might be row-based or columnar. So, this is a &lt;a href=&quot;https://www.coursera.org/course/db&quot; rel=&quot;nofollow&quot;&gt;nice course&lt;/a&gt; for learning about databases getting a know of how to deal with them.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Algorithms&lt;/strong&gt; - Knowledge of basic data structures and algorithms would be enough to solve most Kaggle and real-world data problems (along with all the mentioned above, though). So, get yourself a nice Intro. to algos book.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, all those mentioned above are essential for understanding and tackling a data science problem. All the others are just improvements above these concepts, including NLP, Deep Learning, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, have these foundations nice and good, so that you can build upon these and get familiar and test the waters of much deeper concepts like neural nets, NLP, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, it is always useful to get a knack of Big Data and distributed algorithms and concepts. This &lt;a href=&quot;https://www.coursera.org/course/mmds&quot; rel=&quot;nofollow&quot;&gt;coursera course&lt;/a&gt; is a perfect start for learning the concepts of parallel computing and handling huge data.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-09-24T15:52:37.963" LastActivityDate="2015-09-24T15:52:37.963" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6979" PostTypeId="2" ParentId="6970" CreationDate="2015-09-01T21:54:01.573" Score="3" Body="&lt;p&gt;Your hypothesis class consists of all possible hypotheses that you are searching over, regardless of their form. For convenience's sake, the hypothesis class is usually constrained to be only one type of function or model at a time, since learning methods typically only work on one type at a time. This doesn't have to be the case, though: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Hypothesis classes don't have to consist of only one type of function. If you're searching over all linear, quadratic, and exponential functions, then those are what your combined hypothesis class contains.&lt;/li&gt;&#xA;&lt;li&gt;Hypothesis classes also don't have to consist of only simple functions. If you manage to search over all piecewise-$\tanh^2$ functions, then those functions are what your hypothesis class includes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The big tradeoff is that the larger your hypothesis class, the better the best hypothesis models the underlying true function, but the harder it is to find that best hypothesis. This is related to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff&quot; rel=&quot;nofollow&quot;&gt;bias–variance tradeoff&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11142" LastActivityDate="2015-09-01T21:54:01.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6980" PostTypeId="2" ParentId="6964" CreationDate="2015-09-02T00:07:15.973" Score="4" Body="&lt;p&gt;You might have to extend another package to implement LSTM and RNN in R. Here are some packages to get you started:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/deepnet/&quot; rel=&quot;nofollow&quot;&gt;deepnet&lt;/a&gt; Implements a variety of deep learning architectures&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/darch/index.htmlh&quot; rel=&quot;nofollow&quot;&gt;darch&lt;/a&gt; A deep architecture&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/h2o/index.html&quot; rel=&quot;nofollow&quot;&gt;H2O&lt;/a&gt; An open-source company with deep learning packages&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2015-09-02T00:07:15.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6981" PostTypeId="2" ParentId="6972" CreationDate="2015-09-02T00:22:32.530" Score="0" Body="&lt;p&gt;That is exactly what the &lt;a href=&quot;http://www.trifacta.com/&quot; rel=&quot;nofollow&quot;&gt;Trifecta&lt;/a&gt; product does (in addition to other features). It uses the Wrangle language which is a DSL (domain specific language) designed for data manipulation. There is a much earlier research project called &lt;a href=&quot;http://vis.stanford.edu/papers/wrangler&quot; rel=&quot;nofollow&quot;&gt;Wrangler&lt;/a&gt; from the same people. The Wrangler papers might give you ideas.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2015-09-02T00:22:32.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6982" PostTypeId="5" CreationDate="2015-09-02T09:01:54.150" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-09-02T09:01:54.150" LastActivityDate="2015-09-02T09:01:54.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6983" PostTypeId="4" CreationDate="2015-09-02T09:01:54.150" Score="0" Body="ELK (also called ELK stack) is combination of elasticsearch, logstash and kibana." OwnerUserId="82" LastEditorUserId="82" LastEditDate="2015-09-02T09:45:13.103" LastActivityDate="2015-09-02T09:45:13.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6984" PostTypeId="1" CreationDate="2015-09-02T11:40:08.637" Score="4" ViewCount="1653" Body="&lt;p&gt;I have to build a recommender system &amp;amp; it will be evaluated using map@10 criteria.&#xA;I have rolled up the data/rows at user-item level &amp;amp; is using Gradient Boosting in scikit learn to build the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Challenge&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is a big class imbalance where only 6%( out of 40 K use-item pairs) have actually purchased the item in the entire base. Given this challenge i want to know the accuracy metric I should use to tune the parameters of GBM model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about using recall, precision, f1 score etc. to tune the hyperparameters of GBM. But which one will maximize map@10 given the class imbalance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="1151" LastEditorUserId="1151" LastEditDate="2015-09-02T13:10:17.377" LastActivityDate="2015-09-02T13:10:17.377" Title="Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?" Tags="|machine-learning|classification|recommender-system|class-imbalance|loss-function|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="6985" PostTypeId="2" ParentId="6984" CreationDate="2015-09-02T13:02:04.277" Score="3" Body="&lt;p&gt;If you want to stay in Python, I would suggest going for xgboost instead of GradientBoostingClassifier - among other advantages, it supports map / ndcg as metrics. If you are considering R, gbm supports both map and ndcg.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-02T13:02:04.277" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="6986" PostTypeId="1" CreationDate="2015-09-02T13:34:53.967" Score="10" ViewCount="285" Body="&lt;p&gt;I found the following article on &lt;a href=&quot;http://statweb.stanford.edu/~jbien/jasa2011minimax.pdf&quot; rel=&quot;noreferrer&quot;&gt;&quot;Hierarchical Clustering With Prototypes&#xA;via Minimax Linkage&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is stated in Property 6 that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Minimax linkage cannot be written using &lt;a href=&quot;https://en.wikipedia.org/wiki/Ward%27s_method&quot; rel=&quot;noreferrer&quot;&gt;Lance–Williams&lt;/a&gt; updates.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A succinct proof using a counter-example is given:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Proof. &#xA;  Figure 9 shows a simple one-dimensional example that could not&#xA;  arise if minimax linkage followed Lance– Williams updates. The upper&#xA;  and lower panels show two configurations of points for which the&#xA;  right side of (4) is identical but the left side differs; in&#xA;  particular, &lt;span class=&quot;math-container&quot;&gt;$d(G_1 \cup G_2,H) = 9$&lt;/span&gt; for the upper panel, whereas&#xA;  &lt;span class=&quot;math-container&quot;&gt;$d(G_1 \cup G_2,H) = 8$&lt;/span&gt; for the lower panel.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I do not understand their proof. For both cases (upper and lower panels), &lt;span class=&quot;math-container&quot;&gt;$d(G_1,H) = 16$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$d(G_2,H) = 7$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$d(G_1,G_2) = 5$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot see any reason that &lt;span class=&quot;math-container&quot;&gt;$\alpha(G_2)$&lt;/span&gt; in the first case must equal &lt;span class=&quot;math-container&quot;&gt;$\alpha(G_2)$&lt;/span&gt; in the second case. For instance, &lt;span class=&quot;math-container&quot;&gt;$G_2$&lt;/span&gt; has not the same cardinal.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastEditorUserId="21519" LastEditDate="2019-03-08T04:44:02.020" LastActivityDate="2020-07-10T16:04:52.407" Title="Is Minimax Linkage a Lance-Williams hierarchical clustering?" Tags="|machine-learning|clustering|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="6987" PostTypeId="1" CreationDate="2015-09-02T14:49:33.520" Score="25" ViewCount="49060" Body="&lt;p&gt;I've recently started learning to work with &lt;code&gt;sklearn&lt;/code&gt; and have just come across this peculiar result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the &lt;code&gt;digits&lt;/code&gt; dataset available in &lt;code&gt;sklearn&lt;/code&gt; to try different models and estimation methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I tested a Support Vector Machine model on the data, I found out there are two different classes in &lt;code&gt;sklearn&lt;/code&gt; for SVM classification: &lt;code&gt;SVC&lt;/code&gt; and &lt;code&gt;LinearSVC&lt;/code&gt;, where the former uses &lt;em&gt;one-against-one&lt;/em&gt; approach and the other uses &lt;em&gt;one-against-rest&lt;/em&gt; approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't know what effect that could have on the results, so I tried both. I did a Monte Carlo-style estimation where I ran both models 500 times, each time splitting the sample randomly into 60% training and 40% test and calculating the error of the prediction on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regular SVC estimator produced the following histogram of errors:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/NkKQM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NkKQM.png&quot; alt=&quot;SVC Error Rate&quot;&gt;&lt;/a&gt;&#xA;While the linear SVC estimator produced the following histogram:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ajHfx.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ajHfx.png&quot; alt=&quot;Linear SVC Error Rate&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could account for such a stark difference? Why does the linear model have such higher accuracy most of the time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, relatedly, what could be causing the stark polarization in the results? Either an accuracy close to 1 or an accuracy close to 0, nothing in between.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For comparison, a decision tree classification produced a much more normally distributed error rate with an accuracy of around .85.&lt;/p&gt;&#xA;" OwnerUserId="12580" LastEditorUserId="12580" LastEditDate="2015-09-02T14:56:21.440" LastActivityDate="2015-12-18T07:54:21.370" Title="Can you explain the difference between SVC and LinearSVC in scikit-learn?" Tags="|svm|scikit-learn|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6988" PostTypeId="1" AcceptedAnswerId="8001" CreationDate="2015-09-02T15:21:23.047" Score="5" ViewCount="1870" Body="&lt;p&gt;I would like to understand regularization/shrinkage in the light of MLE/Gradient Descent. I know both concepts but I do not know/understand whether both are used to determine coefficients of a linear model. If so, what are the steps followed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further elaborate, regularization is used to reduce variance which is accomplished through penalizing coefficients of a linear model. The tuning parameter, lambda, is determined through cross-validation. Once, lambda is determined the coefficients are automatically determined, right? Hence, why do we need to minimize (RSS + regularization term) to find coefficients? Are the steps the following: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find lambda through cross-validation &lt;/li&gt;&#xA;&lt;li&gt;Minimize (RSS + regularization) through MLE or GD&lt;/li&gt;&#xA;&lt;li&gt;Find coefficients&lt;/li&gt;&#xA;&lt;li&gt;Penalize coefficients to decrease variance&lt;/li&gt;&#xA;&lt;li&gt;We are left with a small subset of coefficients&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12350" LastActivityDate="2015-09-04T04:48:33.520" Title="Connection between Regularization and Gradient Descent" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="6989" PostTypeId="1" AcceptedAnswerId="8006" CreationDate="2015-09-02T18:20:20.493" Score="3" ViewCount="1058" Body="&lt;p&gt;For the first time, I am playing around with a &lt;a href=&quot;http://docs.opencv.org/doc/user_guide/ug_traincascade.html&quot; rel=&quot;nofollow&quot;&gt;Cascade Classifier&lt;/a&gt; with the OpenCV package (also new to the latter). I realized that it would probably be faster to write my own GUI/script to generate the needed positive and negative images from the set of images I have than to open each file in Photoshop or Paint, but I also suspect this has been done many times before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I am looking for a GUI that lets users page through files in a directory and then use mouse clicks to draw rectangles on a particular image and have the coordinates of the rectangle recorded for later purposes. Any suggestions? If not, I'll  be sure to post a link when/if I finish this. It seems like something of general enough utility I am surprised I can't find it in the OpenCV package itself.  &lt;/p&gt;&#xA;" OwnerUserId="12586" LastEditorUserId="12586" LastEditDate="2015-09-04T15:26:46.787" LastActivityDate="2018-12-11T15:38:25.610" Title="Python script/GUI to generate positive/negative images for CascadeClassifier?" Tags="|classification|image-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6991" PostTypeId="2" ParentId="6550" CreationDate="2015-09-03T07:22:39.270" Score="1" Body="&lt;p&gt;See also: &lt;a href=&quot;https://datascience.stackexchange.com/questions/869/neural-network-parse-string-data&quot;&gt;Neural Network parse string data?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not see a problem with using MurMur3 per se.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the categorical labels, you can use one-hot encoding / one-of-k encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the strings, it's an application-specific question.  Presumably if you use exactly those strings as features, it will be very sparse.  The effect of this will depend on the algorithm that you are using, and how the training data compare to the data you see in practice.  You are running the risk that you will effectively either only create a traditional IP/email whitelist/blacklist OR throw out the feature altogether.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You must decide what you want (eg should a certain email address &lt;em&gt;always&lt;/em&gt; get a certain output label?) and have some intuition about the application so as to generate more features from &lt;em&gt;IP address&lt;/em&gt; and &lt;em&gt;email address&lt;/em&gt;.  For example, from &lt;em&gt;email address&lt;/em&gt; you can extract the &lt;em&gt;local part&lt;/em&gt; (eg &quot;john1972&quot;) and &lt;em&gt;domain&lt;/em&gt;, and from each of those you can extract:&lt;br&gt;&#xA;- &lt;em&gt;length&lt;/em&gt;&lt;br&gt;&#xA;- &lt;em&gt;character tri-grams&lt;/em&gt;&lt;br&gt;&#xA;- &lt;em&gt;count/proportion of numbers to alphachars&lt;/em&gt;&lt;br&gt;&#xA;- &lt;em&gt;number of hyphens&lt;/em&gt;&lt;br&gt;&#xA;- &lt;em&gt;dictionary validity&lt;/em&gt;&lt;br&gt;&#xA;...&lt;br&gt;&#xA;(From &lt;em&gt;domain&lt;/em&gt; you can also extract TLD and possibly subdomains.)&#xA;You can try to tokenise .  You can even hit external services to get information like &lt;em&gt;number of Google hits&lt;/em&gt;, &lt;em&gt;detected language&lt;/em&gt;, &lt;em&gt;spam score&lt;/em&gt; etc.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-03T07:22:39.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6992" PostTypeId="1" CreationDate="2015-09-03T09:57:16.537" Score="3" ViewCount="1113" Body="&lt;p&gt;I'm using the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;scikit-learn's Random Forest&lt;/a&gt; to perform some classification task but run out of memory because of the amount of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any mini-batch implementation of the Random Forest algorithm (or similar decision trees-based method)?&lt;/p&gt;&#xA;" OwnerUserId="9050" LastEditorUserId="9050" LastEditDate="2015-09-03T10:09:06.620" LastActivityDate="2015-11-04T02:52:47.043" Title="Any mini-batch implementation of Random Forest?" Tags="|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6993" PostTypeId="2" ParentId="6550" CreationDate="2015-09-03T11:41:03.390" Score="0" Body="&lt;p&gt;There's potentially some useful ideas for you in the answers to &lt;a href=&quot;https://datascience.stackexchange.com/questions/412/how-can-i-transform-names-in-a-confidential-data-set-to-make-it-anonymous-but-p/446#446&quot;&gt;this question about transforming names in a confidential data set but preserve some of the characteristics of the names&lt;/a&gt; I read a while ago and found very interesting. It's not exactly the right topic but there is some discussion on encrypting strings while preserving edit distance, which may not answer your question but perhaps will give you some ideas to think about if you're still stuck...&lt;/p&gt;&#xA;" OwnerUserId="9587" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-03T11:41:03.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6994" PostTypeId="2" ParentId="6987" CreationDate="2015-09-03T13:39:51.937" Score="5" Body="&lt;p&gt;If you used the default kernel in SVC(), the &lt;a href=&quot;http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;amp;doc=exercises/ex8/ex8.html&quot;&gt;Radial Basis Function&lt;/a&gt; (rbf) kernel, then you probably learned a more nonlinear decision boundary.  In the case of the &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/datasets/plot_digits_last_image.html&quot;&gt;digits dataset&lt;/a&gt;, this will vastly outperform a linear decision boundary &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/bottou-94.pdf&quot;&gt;on this task&lt;/a&gt; (see 3.1 'Baseline Linear Classifier')&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-09-03T13:39:51.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6995" PostTypeId="1" CreationDate="2015-09-03T13:45:43.010" Score="2" ViewCount="3898" Body="&lt;p&gt;&lt;strong&gt;Problem/Main objective/TLDR:&lt;/strong&gt; Train a classifier, then feed it a random review and get the correspondent predicted review rating (number of stars from 1 to 5) - only 60% accuracy! :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a big dataset with around 48000 tech product reviews (from many different writers and from different products - here this is not so important (?)) and corresponding ratings (1 to 5 stars) I randomly selected some reviews within each class:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 star: 173 reviews (could not pick 1000 because there were 173)&#xA;2 stars: 1000 reviews&#xA;3 stars: 1000 reviews&#xA;4 stars: 1000 reviews&#xA;5 stars: 1000 reviews&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Total: 4173 reviews - this data is organized in one file (all_reviews_labeled.txt) in tuple format, one review and rating for line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(‘review text’, ‘x star’)&#xA;(‘review text’, ‘x star’)&#xA;(‘review text’, ‘x star’)&#xA;(‘review text’, ‘x star’)&#xA;…&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My 1st “dummie” aproach was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Tokenize review text&#xA;POS tagging&#xA;&#xA;Get most frequent bigrams that folowing some POS tags rules for most frequent trigrams (I have seen this rules - using this POS patterns in “Automatic Star-rating Generation from Text Reviews” - pag.7 - paper from Chong-U Lim, Pablo Ortiz and Sang-Woo Jun):&#xA;&#xA;for (w1,t1), (w2,t2), (w3,t3) in nltk.trigrams(text):&#xA;    if (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'NN' or t2 == 'NNS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJR') and (t3 != 'NN' or t3 != 'NNS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'JJ' or t1 == 'JJS' or t1 == 'JJR') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'NN' or t1 == 'NNS') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS') and (t3 != 'NN' or t3 != 'NNS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'RB' or t1 == 'RBR' or t1 == 'RBS') and (t2 == 'VB' or t2 == 'VBD' or t2 == 'VBN' or t2 == 'VBG'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'DT') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    elif (t1 == 'VBZ') and (t2 == 'JJ' or t2 == 'JJS' or t2 == 'JJRS'):&#xA;        bi = unicode(w1 + ' ' + w2).encode('utf-8')&#xA;        bigrams.append(bi)&#xA;    else:&#xA;        continue&#xA;&#xA;Extract features (here is where I have more doubts - should I only look for this two features?):&#xA;&#xA;features={}&#xA;for bigram,freq in word_features:&#xA;    features['contains(%s)' % unicode(bigram).encode('utf-8')] = True&#xA;    features[&quot;count({})&quot;.format(unicode(bigram).encode('utf-8'))] = freq&#xA;return features&#xA;&#xA;featuresets = [(review_features(review), rating) for (review, rating) in tuples_labeled_reviews]&#xA;&#xA;Splits the training data into training size and testing size (90% training - 10% testing):&#xA;&#xA;numtrain = int(len(tuples_labeled_reviews) * 90 / 100)&#xA;train_set, test_set = featuresets[:numtrain], featuresets[numtrain:]&#xA;&#xA;Train SVMc:&#xA;&#xA;classifier = nltk.classify.SklearnClassifier(LinearSVC())&#xA;classifier.train(train_set)&#xA;&#xA;Evaluate the classifier:&#xA;&#xA;errors = 0&#xA;correct = 0&#xA;for review, rating in test_set:&#xA;    tagged_rating = classifier.classify(review)&#xA;    if tagged_rating == rating:&#xA;        correct += 1&#xA;        print(&quot;Correct&quot;)&#xA;        print &quot;Guess: &quot;, tagged_rating&#xA;        print &quot;Correct: &quot;, rating&#xA;    else:&#xA;        errors += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far I get only 60% accuracy… What can I do to improve my prediction results? Is something before, some text/reviews preprocessing (like removing stopwords/punctuation?) that is missing? Could you suggest me some other approaches? I am still a bit confused if is really a classification problem or a regression one... :/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please simple explanations, or give me a link to “machine learning for dummies”, or be my mentor, I promise to learn fast! My background in machine learning/language processing/data mining is very light, I have played a couple of times with weka (Java), but now I need to stick with Python (nltk + scikit-learn)!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Now I am also extracting unigrams as features, unigrams POS-tagged as 'JJ', 'NN','VB' and 'RB'. It improved a little the accuracy to 65%.&lt;/li&gt;&#xA;&lt;li&gt;I applied also do stemming and lemmatization in text before POS tagging. It improved the accuracy to +70%.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;I have feed the classifier all my reviews, the 48000, split into 90% training and 10% testing and the accuracy was 91%.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now I have 32000 new reviews (also labeled) and feed them all for testing and the mean accuracy was 62 % ... my confusion matrix is something like this image below (i divided by equal errors of +1/-1 star point, +2/-2, +3/-3 - because it is just an illustration):&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/efSpy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/efSpy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;What is happening? Why accuracy drops so much at 3 and 5 stars?&lt;/p&gt;&#xA;" OwnerUserId="12597" LastEditorUserId="12597" LastEditDate="2015-09-10T11:09:00.083" LastActivityDate="2015-09-10T11:09:00.083" Title="NLTK: Tuning LinearSVC classifier accuracy? - Looking for better approaches/advices" Tags="|machine-learning|data-mining|python|nlp|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="6997" PostTypeId="2" ParentId="6995" CreationDate="2015-09-03T14:25:53.153" Score="3" Body="&lt;p&gt;You might try looking into &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;nofollow&quot;&gt;sentiment analysis&lt;/a&gt;.  There was a &lt;a href=&quot;https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews&quot; rel=&quot;nofollow&quot;&gt;kaggle competition&lt;/a&gt; on it, and you might find insight there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treating this as either a regression or a classification problem is fair.  Also, it's important to judge your performance against the &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis#Evaluation&quot; rel=&quot;nofollow&quot;&gt;proper baselines&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your feature space might not be rich enough for the classes to be linearly separable.  You might do better using an &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;SVM with a non-linear kernel&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also appears you haven't &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_scaling&quot; rel=&quot;nofollow&quot;&gt;scaled&lt;/a&gt; the counts of the bigrams, which is generally helpful for SVMs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thought for an approach would be to apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; to the set of documents (reviews) and use the topics as your feature space (you'll have a topic vector per document).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some places to get python LDA implementations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pypi.python.org/pypi/gensim&quot; rel=&quot;nofollow&quot;&gt;gensim&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.princeton.edu/~blei/topicmodeling.html&quot; rel=&quot;nofollow&quot;&gt;Blei&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-09-03T17:43:09.420" LastActivityDate="2015-09-03T17:43:09.420" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="6999" PostTypeId="2" ParentId="2368" CreationDate="2015-09-03T17:43:29.930" Score="9" Body="&lt;p&gt;One more thing to consider, beyond everything that Ben Haley said, is to &lt;strong&gt;convert to user local time&lt;/strong&gt;. For example, if you are trying to predict something that occurs around 8pm for all users, if you look at UTC time, it will be harder to predict from.&lt;/p&gt;&#xA;" OwnerUserId="3592" LastActivityDate="2015-09-03T17:43:29.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="7000" PostTypeId="2" ParentId="6960" CreationDate="2015-09-03T22:30:58.170" Score="2" Body="&lt;p&gt;This exact problem was a kaggle competition sponsored by Facebook. The particular forum thread of interest for you is the one where many of the top competitors explained their methodology, this should provide you with more information than you were probably looking for: &lt;a href=&quot;https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/6650/share-your-approach&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/6650/share-your-approach&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, it appears that most people treated the problem as a supervised one. Their primary feature was a tf-idf, or unweighted BOW, representations of the text and they ensembled 1000s of single-tag models. Owen, the winner of the competition, noted that the title text was a more powerful feature than the content of the body of the post.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-09-03T22:30:58.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="7001" PostTypeId="2" ParentId="6693" CreationDate="2015-09-04T00:27:40.430" Score="1" Body="&lt;p&gt;A first count on items can prune some of them. If you have only 20 records you can simply count the occurrence of items in dataset and prune those items which did not happen enough because their supersets also do not happen frequently and them make item-sets based on the rest of items.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-09-04T00:27:40.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="7002" PostTypeId="2" ParentId="6834" CreationDate="2015-09-04T00:31:32.127" Score="1" Body="&lt;p&gt;What you are exactly looking for is a modification to the &lt;a href=&quot;http://kghost.de/cig_proc/full/paper_31.pdf&quot; rel=&quot;nofollow&quot;&gt;DEDICOM&lt;/a&gt; Algorithm (page 4). the DEDICOM itself gives you measure for relation between different components of a directed graph. You just need to be a bit creative to use it for converting a graph into DAG. Read the paper and if further help needed just drop me a comment.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-09-04T00:31:32.127" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8001" PostTypeId="2" ParentId="6988" CreationDate="2015-09-04T04:48:33.520" Score="4" Body="&lt;p&gt;The fitting procedure is the one that actually finds the coefficients of the model. The regularization term is used to indirectly find the coefficients by penalizing big coefficients &lt;em&gt;during the fitting procedure&lt;/em&gt;. A simple (albeit somewhat biased/naive) example might help illustrate this difference between regularization and gradient descent:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X, y &amp;lt;- read input data&#xA;for different values of lambda L&#xA;    for each fold of cross-validation using X,y,L&#xA;        theta &amp;lt;- minimize (RSS + regularization using L) via MLE/GD&#xA;        score &amp;lt;- calculate performance of model using theta on the validation set &#xA;    if average score across folds for L is better than the current best average score &#xA;        L_best &amp;lt;- L&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the fitting procedure (MLE or GD in our case) finds the best coefficients given the specific value of lambda.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, I would look at this answer &lt;a href=&quot;https://stats.stackexchange.com/questions/137481/how-bad-is-hyperparameter-tuning-outside-cross-validation&quot;&gt;here&lt;/a&gt; about tuning the regularization parameter, because it tends a little bit murky in terms of bias. &lt;/p&gt;&#xA;" OwnerUserId="11136" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-09-04T04:48:33.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8003" PostTypeId="2" ParentId="6681" CreationDate="2015-09-04T04:57:27.220" Score="3" Body="&lt;p&gt;First I want to say Dirk is basically correct that survival analysis doesn't have to model death it's used essentially the same way for looking at user groups in cohort analysis.  However, it is unreliable for specific individual behavior activities, for say fraud management. Revenue analysis and trend forecasting are great uses of survival models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That having been said the situation is not hopeless for individual behavior, simply not so straight-forward.  Having run into a similar situation in pre-paid card accounts for a mobile company ... you can model an individual's behavior in the way your trying to, by modeling it as a process control issue, and then following good practice. By randomly sampled data(important) of an individual's behavior (usage) and comparing it to similar behaviors of other user's (normalizing for environmental factors) and looking at the control points, more than 3 sequential out of control points in a period or more than 8 overall is base set.  This can then give you a set &quot;suspicious&quot; potentially &quot;churn or fraud&quot; activities which can then be analyzed and labeled to train better models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that's helpful :)&lt;/p&gt;&#xA;" OwnerUserId="12612" LastActivityDate="2015-09-04T04:57:27.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8004" PostTypeId="1" CreationDate="2015-09-04T10:20:30.410" Score="6" ViewCount="100" Body="&lt;p&gt;I am trying to start a meta-analysis for which I want to extract some 16S-based information from public databases. Moreover, I want to relate this information with any metadata found in the associated studies (everything from environmental variables to sequencing details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this, I realized some databases are available, like NCBI-Nucletotide, NCBI-SRA and EMBL-EBI-ENA, but I am not sure about which one to use or whether I can use them all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I filter only whole 16S sequences? Or even the most used 16S region?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more thing would be how can I extract all the corresponding metadata?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone here tried this or anything similar before?&lt;/p&gt;&#xA;" OwnerUserId="12614" LastEditorUserId="843" LastEditDate="2015-09-10T07:09:09.673" LastActivityDate="2015-09-10T07:09:09.673" Title="Meta-analysis of public 16S data" Tags="|metadata|bioinformatics|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8005" PostTypeId="1" CreationDate="2015-09-04T14:57:20.020" Score="2" ViewCount="562" Body="&lt;p&gt;I want to achieve during my internship a credit risk scoring system based on &quot;credit german dataset&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I develop a model with R using logistic regression and i validate it. Now i must use this model to calculate a new applicant score and deduce whether good or bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know how to proceed?&lt;/p&gt;&#xA;" OwnerUserId="12619" LastActivityDate="2015-09-07T09:21:14.393" Title="Develop a “Credit Risk Scoring System” with R" Tags="|data-mining|r|predictive-modeling|" AnswerCount="1" CommentCount="1" ClosedDate="2015-09-07T11:40:56.900" ContentLicense="CC BY-SA 3.0" />
  <row Id="8006" PostTypeId="2" ParentId="6989" CreationDate="2015-09-04T15:46:28.790" Score="1" Body="&lt;p&gt;So I wrote the script. It gave me an excuse to learn Tkinter. It's pasted below. Note this is a one-off, not a model of good programming practice! If anyone uses this and has bugs or suggestions, let me know. Here's the &lt;a href=&quot;https://github.com/sunnysideprodcorp/CascadeImagesorter/&quot; rel=&quot;nofollow&quot;&gt;git link&lt;/a&gt; and model code is pasted below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import Tkinter&#xA;import Image, ImageTk&#xA;from Tkinter import Tk, BOTH&#xA;from ttk import Frame, Button, Style&#xA;import cv2&#xA;import os&#xA;import time&#xA;import itertools&#xA;&#xA;IMAGE_DIRECTORY =            # Directory of existing files&#xA;POSITIVE_DIRECTORY =         # Where to store 'positive' cropped images&#xA;NEGATIVE_DIRECTORY =         # Where to store 'negative' cropped images (generated automatically based on 'positive' image cropping&#xA;IMAGE_RESIZE_FACTOR =        # How much to scale images for display purposes. Images are not scaled when saved.&#xA;&#xA;# Everything stuffed into one class, not exactly model programming but it works for now&#xA;class Example(Frame):&#xA;&#xA;    def __init__(self, parent, list_of_files, write_file):&#xA;        Frame.__init__(self, parent)           &#xA;        self.parent = parent&#xA;        self.list_of_files = list_of_files&#xA;        self.write_file = write_file&#xA;        self.image = None&#xA;        self.canvas = None&#xA;        self.corners = []&#xA;        self.index = -1&#xA;        self.loadImage()&#xA;        self.initUI()&#xA;        self.resetCanvas()&#xA;&#xA;&#xA;    def loadImage(self):&#xA;        self.index += 1&#xA;        img = cv2.imread(self.list_of_files[self.index])            &#xA;        print(self.list_of_files[self.index])&#xA;        while not img.shape[0]:&#xA;            self.index += 1&#xA;            img = cv2.imread(self.list_of_files[self.index])            &#xA;        self.cv_img = img&#xA;        img_small = cv2.resize(img, (0,0), fx = IMAGE_RESIZE_FACTOR, fy = IMAGE_RESIZE_FACTOR)&#xA;        b, g, r = cv2.split(img_small)&#xA;        img_small = cv2.merge((r,g,b))&#xA;        im = Image.fromarray(img_small)&#xA;        self.image = ImageTk.PhotoImage(image=im)       &#xA;&#xA;    def resetCanvas(self):        &#xA;        self.canvas.create_image(0, 0, image=self.image, anchor=&quot;nw&quot;)&#xA;        self.canvas.configure(height = self.image.height(), width = self.image.width())&#xA;        self.canvas.place(x = 0, y = 0, height = self.image.height(), width = self.image.width())&#xA;&#xA;    def initUI(self):&#xA;        self.style = Style()&#xA;        self.style.theme_use(&quot;default&quot;)&#xA;        self.pack(fill=BOTH, expand=1)&#xA;&#xA;        print &quot;width and height of image should be &quot;, self.image.width(), self.image.height()&#xA;        self.canvas = Tkinter.Canvas(self, width = self.image.width(), height = self.image.height())       &#xA;        self.canvas.bind(&quot;&amp;lt;Button-1&amp;gt;&quot;, self.OnMouseDown)&#xA;        self.canvas.pack()&#xA;&#xA;        nextButton = Button(self, text=&quot;Next&quot;, command=self.nextButton)&#xA;        nextButton.place(x=0, y=0)&#xA;&#xA;        restartButton = Button(self, text=&quot;Restart&quot;, command=self.restart)&#xA;        restartButton.place(x=0, y=22)       &#xA;&#xA;    def nextButton(self):&#xA;        new_img = self.cv_img[self.corners[0][1]/IMAGE_RESIZE_FACTOR:self.corners[1][1]/IMAGE_RESIZE_FACTOR, self.corners[0][0]/IMAGE_RESIZE_FACTOR:self.corners[1][0]/IMAGE_RESIZE_FACTOR]&#xA;        files = self.list_of_files[self.index].split(&quot;/&quot;)&#xA;        try:&#xA;            os.stat(POSITIVE_DIRECTORY+files[-2])&#xA;        except:&#xA;            os.mkdir(POSITIVE_DIRECTORY+files[-2])&#xA;        print(&quot;saving to &quot;, &quot;{}{}/{}&quot;.format(POSITIVE_DIRECTORY, files[-2], files[-1]))&#xA;        cv2.imwrite(&quot;{}{}/{}&quot;.format(POSITIVE_DIRECTORY, files[-2], files[-1]), new_img)&#xA;        self.saveNegatives(files)&#xA;        self.restart()&#xA;        self.loadImage()&#xA;        self.resetCanvas()&#xA;&#xA;    def saveNegatives(self, files):&#xA;        low_x = min(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR&#xA;        high_x = max(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR&#xA;        low_y = min(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR&#xA;        high_y = max(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR&#xA;&#xA;        try:&#xA;            os.stat(NEGATIVE_DIRECTORY+files[-2])&#xA;        except:&#xA;            os.mkdir(NEGATIVE_DIRECTORY+files[-2])&#xA;&#xA;        new_img = self.cv_img[ :low_y, :]&#xA;        cv2.imwrite(&quot;{}{}/{}{}&quot;.format(NEGATIVE_DIRECTORY, files[-2], &quot;LY&quot;, files[-1]), new_img)&#xA;        new_img = self.cv_img[ high_y: , :]&#xA;        cv2.imwrite(&quot;{}{}/{}{}&quot;.format(NEGATIVE_DIRECTORY, files[-2], &quot;HY&quot;, files[-1]), new_img)&#xA;&#xA;        new_img = self.cv_img[ :, :low_x ]&#xA;        cv2.imwrite(&quot;{}{}/{}{}&quot;.format(NEGATIVE_DIRECTORY, files[-2], &quot;LX&quot;, files[-1]), new_img)&#xA;        new_img = self.cv_img[:,  high_x: ]&#xA;        cv2.imwrite(&quot;{}{}/{}{}&quot;.format(NEGATIVE_DIRECTORY, files[-2], &quot;HX&quot;, files[-1]), new_img)&#xA;&#xA;&#xA;    def restart(self):&#xA;        self.corners = []&#xA;        self.index -=1&#xA;        self.canvas.delete(&quot;all&quot;)&#xA;        self.loadImage()&#xA;        self.resetCanvas()&#xA;&#xA;&#xA;    def OnMouseDown(self, event):&#xA;        print(event.x, event.y)&#xA;        self.corners.append([event.x, event.y])&#xA;        if len(self.corners) == 2:&#xA;            self.canvas.create_rectangle(self.corners[0][0], self.corners[0][1], self.corners[1][0], self.corners[1][1], outline ='cyan', width = 2)&#xA;&#xA;&#xA;def main():&#xA;&#xA;    root = Tk()&#xA;    root.geometry(&quot;250x150+300+300&quot;)&#xA;&#xA;    list_of_files = []&#xA;    file_names = []&#xA;    walker = iter(os.walk(IMAGE_DIRECTORY))&#xA;    next(walker)&#xA;    for dir, _, _ in walker:&#xA;        files = [dir + &quot;/&quot; +  file for file in os.listdir(dir)]            &#xA;        list_of_files.extend(files)&#xA;        file_names.extend(os.listdir(dir))&#xA;&#xA;    list_of_processed_files = []&#xA;    processed_file_names = []&#xA;    walker = iter(os.walk(POSITIVE_DIRECTORY))&#xA;    next(walker)&#xA;    for dir, _, _ in walker:&#xA;        files = [dir + &quot;/&quot; +  file for file in os.listdir(dir)]            &#xA;        list_of_processed_files.extend(files)&#xA;        processed_file_names.extend(os.listdir(dir))&#xA;&#xA;    good_names = set(file_names) - set(processed_file_names)&#xA;    list_of_files = [f for i, f in enumerate(list_of_files) if file_names[i] in good_names] &#xA;&#xA;    app = Example(root, list_of_files, IMAGE_DIRECTORY+&quot;positives&quot;)&#xA;    root.mainloop()  &#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    main()  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12586" LastEditorUserId="12586" LastEditDate="2015-09-04T22:00:38.370" LastActivityDate="2015-09-04T22:00:38.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8009" PostTypeId="1" AcceptedAnswerId="8163" CreationDate="2015-09-04T21:24:25.313" Score="5" ViewCount="2942" Body="&lt;p&gt;I am researching to implement RMSProp in a neural network project I am writing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a &lt;a href=&quot;https://www.coursera.org/course/neuralnets&quot; rel=&quot;noreferrer&quot;&gt;Coursera class presented by Geoffrey Hinton&lt;/a&gt; (lecture 6 I think). I don't think the approach has ever been formally published, despite many gradient-descent optimisation libraries having an option called &quot;RMSProp&quot;. In addition, my searches are showing up a few variations of the original idea, and it is not clear why they differ, or whether there is a clear reason to use one version over another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general idea behind RMSProp is to scale learning rates by a moving average of current gradient magnitude. On each update step, the existing squared gradients are averaged into a running average (which is &quot;decayed&quot; by a factor) and when the network weight params are updated, the updates are divided by the square roots of these averaged squared gradients. This seems to work by stochastically &quot;feeling out&quot; the second order derivatives of the cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naively, I would implement this as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Params:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\gamma$ geometric rate for averaging in [0,1]&lt;/li&gt;&#xA;&lt;li&gt;$\iota$ numerical stability/smoothing term to prevent divide-by-zero, usually small e.g 1e-6&lt;/li&gt;&#xA;&lt;li&gt;$\epsilon$ learning rate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Terms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$W$ network weights&lt;/li&gt;&#xA;&lt;li&gt;$\Delta$ gradients of weights i.e. $\frac{\partial E}{\partial W}$ for a specific mini-batch&lt;/li&gt;&#xA;&lt;li&gt;$R$ RMSProp matrix of running average squared weights&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Initialise:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$R \leftarrow 1$ (i.e. all matrix cells set to 1)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For each mini-batch:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$R \leftarrow (1-\gamma)R + \gamma \Delta^2$ (element-wise square, not matrix multiply)&lt;/li&gt;&#xA;&lt;li&gt;$W = W - \epsilon \frac{\Delta}{\sqrt{R + \iota}}$ (all element-wise)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have implemented and used a version similar to this before, but that time around I did something different. Instead of updating $R$ with a single $\Delta^2$ from the mini-batch (i.e. gradients summed across the mini-batch, then squared), I summed up each individual example gradient squared from the mini-batch. Reading up on this again, I'm guessing that's wrong. But it worked reasonably well, better than simple momentum. Probably not a good idea though, because of all those extra element-wise squares and sums needed, it will be less efficient if not required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I am discovering further variations that seem to work. They  call themselves RMSProp, and none seems to come with much rationale beyond &quot;this works&quot;. For example, the Python &lt;code&gt;climin&lt;/code&gt; library &lt;a href=&quot;http://climin.readthedocs.org/en/latest/rmsprop.html&quot; rel=&quot;noreferrer&quot;&gt;seems to implement what I suggest above&lt;/a&gt;, but then suggests a further combination with momentum with the teaser &quot;In some cases, adding a momentum term β is beneficial&quot;, with a partial explanation about adaptable step rates - I guess I'd need to get more involved in that library before fully understanding what they are. In another example the &lt;a href=&quot;http://downhill.readthedocs.org/en/stable/generated/downhill.adaptive.RMSProp.html&quot; rel=&quot;noreferrer&quot;&gt;downhill library's RMSProp implementation&lt;/a&gt; combines two moving averages - one is the same as above, but then another, the average of gradients without squaring is also tracked (it is squared and taken away from the average of squared weights).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd really like to understand more about these alternative RMSProp versions. Where have they come from, where is the theory or intuition that suggests the alternative formulations, and why do these libraries use them? Is there any evidence of better performance?&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2019-01-03T11:58:50.247" Title="Implementing RMSProp, but finding differences between reference versions" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8010" PostTypeId="2" ParentId="6681" CreationDate="2015-09-04T22:03:28.517" Score="5" Body="&lt;p&gt;Another approach would be to model &quot;churn&quot; (aka &quot;diminished use of the service, including non-use&quot;) as a &lt;strong&gt;&lt;em&gt;process&lt;/em&gt;&lt;/strong&gt; and not an event.  Years ago in retention marketing this was called a &quot;defection funnel&quot;, to mirror the &quot;sales funnel&quot; on the customer acquisition side (suspect -&gt; prospect -&gt; trial customer -&gt; repeat customer -&gt; loyal customer).  So a defection funnel might look like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;loyal/frequent customer -&gt; disaffected customer -&gt; infrequent customer -&gt; non-customer&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is proper to label this a &quot;funnel&quot; because a subset of each stage move on to the next stage.  Not all &quot;disaffected customers&quot; become &quot;infrequent customers&quot;, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two keys to defining a defection funnel.  First, define the behavioral characteristics for the funnel as a whole, and for each stage (category) within the funnel.  In your case, it might be a variation of the familiar RFM (&quot;recency&quot; &quot;frequency&quot; and &quot;monetary&quot;) score.  Second, you need to identify &lt;em&gt;other&lt;/em&gt; behavioral characteristics or patterns that show a &lt;em&gt;propensity&lt;/em&gt; to move from one stage to another.  These might include: complaints, being victim of trolling, experiencing service interruptions, experiencing betting losses, how long they have been using the service, or maybe it is just demographics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these definitions in place, you are in a position to build a predictive model and also track over time the population of customers in each category.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;By the way, this approach is not dependent on dividing the funnel into &quot;stages&quot;, nor does it matter fundamentally how many stages you have.  However, defining discrete stages has many practical benefits, including communicating your model and results to other people who are not quantitative or statistically minded.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-09-04T22:03:28.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8012" PostTypeId="2" ParentId="6992" CreationDate="2015-09-05T01:54:04.087" Score="2" Body="&lt;p&gt;Check out the CART algorithm. This is essentially a bootstrapping method with subsampling that I am sure you can generalize for batch processing.&lt;/p&gt;&#xA;" OwnerUserId="10148" LastActivityDate="2015-09-05T01:54:04.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8014" PostTypeId="2" ParentId="6550" CreationDate="2015-09-05T13:24:59.433" Score="0" Body="&lt;p&gt;I recommend doing a stacked model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing categorical prediction, you could use a Naive Bayes model to form a prediction on the string data using a bag of words model. That is now a continuous variable to be fed into the ANN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to process text is to term frequency/inverse document freq stats and use a cosine distance from reference docs. Let's say you have a document that speaks about topic A and one about topic B, you could convert unclassified docs to a numerical distance from A and B. (Typically you build a &quot;reference&quot; A vector from many A docs...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Tf&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Tf&lt;/a&gt;–idf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="8650" LastActivityDate="2015-09-05T13:24:59.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8015" PostTypeId="1" CreationDate="2015-09-05T13:49:06.513" Score="6" ViewCount="181" Body="&lt;p&gt;Working with the Naive Bayes spam filtering article on Wikipedia &#xA;(&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/K9cbG.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/oJGqu.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a binary multinomial equation or a Bernoulli form or some other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes? &#xA;&lt;a href=&quot;http://www.aueb.gr/users/ion/docs/ceas2006_paper.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.aueb.gr/users/ion/docs/ceas2006_paper.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper states that binary Multinomial NB performs best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is not multinomial, what changes are needed to make it so?&lt;/p&gt;&#xA;" OwnerUserId="8650" LastActivityDate="2018-02-04T13:07:03.410" Title="Naive about which Naive Bayes in article" Tags="|machine-learning|algorithms|predictive-modeling|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8016" PostTypeId="1" AcceptedAnswerId="8017" CreationDate="2015-09-05T16:42:25.473" Score="9" ViewCount="599" Body="&lt;p&gt;What does the term &lt;strong&gt;&quot;Intelligence&quot;&lt;/strong&gt; originally stand for in &lt;strong&gt;&quot;Business Intelligence&quot;&lt;/strong&gt; ? Does it mean as used in &lt;em&gt;&quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_intelligence&quot;&gt;Artificial Intelligence&lt;/a&gt;&quot;&lt;/em&gt; or as used in &lt;em&gt;&quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Intelligence_agency&quot;&gt;Intelligence Agency&lt;/a&gt;&quot;&lt;/em&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, does &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Business_intelligence&quot;&gt;Business Intelligence&lt;/a&gt;&quot; mean: &quot;Acting smart &amp;amp; intelligently in business&quot; or &quot;Gathering data and information about the business&quot; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question was the topic of a debate among some fellows in our data-science team, so I thought to ask about it from other experts. One might say that both meanings are applicable, but I'm asking for the original intended meaning of the word as proposed in the 1980s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An acceptable answer should definitely cite original references, and personal opinions are not what I'm seeking.&lt;/p&gt;&#xA;" OwnerUserId="10181" LastEditorUserId="12527" LastEditDate="2015-11-05T22:59:25.490" LastActivityDate="2023-09-02T13:53:48.573" Title="Original Meaning of &quot;Intelligence&quot; in &quot;Business Intelligence&quot;" Tags="|bigdata|definitions|terminology|history|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8017" PostTypeId="2" ParentId="8016" CreationDate="2015-09-05T21:11:42.217" Score="12" Body="&lt;p&gt;Howard Dresner, in 1989, is believed to have coined the term &amp;quot;business intelligence&amp;quot;, to describe &amp;quot;concepts and methods to improve business decision making by using fact-based support systems.&amp;quot;. When he was at Gartner Group. This is a common mantra, spread over the Web. I have not been able to trace the exact source for this origin yet. Many insist on he was not at Gartner group in 1989, which is confirmed in the &lt;a href=&quot;https://www.infoworld.com/article/2661157/the-maturing-of-bi.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;following interview&lt;/a&gt;. In his 2008 book, Performance Management Revolution: Improving Results Through Visibility and Actionable Insight, the termed is defined as:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;BI is knowledge gained through the access and analysis of business&#xA;information.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;He says, at the beginning, that&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In 1989, for example, I started-some might say incited-the BI&#xA;revolution with the premise that all users have a fundamental right to&#xA;access information without the help of IT.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;No apparent claim of the invention of the term on his side. Indeed, one can find older roots in H. P. Luhn, &lt;a href=&quot;http://dx.doi.org/10.1147/rd.24.0314&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Business Intelligence System&lt;/a&gt;, IBM Journal of Research and Development, 1958, Vol. 2, Issue 4, p. 314--319.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Abstract: An automatic system is being developed to disseminate information to the various sections of any industrial, scientific or government organization. This intelligence system will utilize data-processing machines for auto-abstracting and auto-encoding of documents and for creating interest profiles for each of the &amp;quot;action points&amp;quot; in an organization. Both incoming and internally generated documents are automatically abstracted, characterized by a word pattern, and sent automatically to appropriate action points. This paper shows the flexibility of such a system in identifying known information, in finding who needs to know it and in disseminating it efficiently either in abstract form or as a complete document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The author claims that:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The techniques proposed here to make these things possible are:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Auto-abstracting of documents;&lt;/li&gt;&#xA;&lt;li&gt;Auto-encoding of documents;&lt;/li&gt;&#xA;&lt;li&gt;Automatic creation and updating of action-point profiles.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;All of these techniques are based on statistical procedures which can&#xA;be performed on present-day data processing machines. Together with&#xA;proper communication facilities and input-output equipment a&#xA;comprehensive system may be assembled to accommodate all information&#xA;problems of an organization. We call this a Business Intelligence&#xA;System.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;He also gives the explanation of the terms &amp;quot;&lt;strong&gt;business&lt;/strong&gt;&amp;quot; and &amp;quot;&lt;strong&gt;intelligence&lt;/strong&gt;&amp;quot;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In this paper, business is a collection of activities carried on for&#xA;whatever purpose, be it science, technology, commerce, industry, law,&#xA;government, defense, et cetera. The communication facility serving the&#xA;conduct of a business (in the broad sense) may be referred to as an&#xA;intelligence system. The notion of intelligence is also defined here,&#xA;in a more general sense, as &amp;quot;the ability to apprehend the&#xA;interrelationships of presented facts in such a way as to guide action&#xA;towards a desired goal.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So the idea of &amp;quot;linking the facts&amp;quot; is already present in H. P. Luhn paper. To many sources, Howard Dresner has re-invented &amp;quot;Business Intelligence&amp;quot; to re-brand decision support system (DSS) and executive information system (EIS) when at DEC, and the term became famous throught the influence of the Gartner group.&lt;/p&gt;&#xA;&lt;p&gt;Apparently, the term has already been used way before, as in the book &lt;a href=&quot;http://www.worldcat.org/title/wholesale-business-intelligence-and-southern-and-western-merchants-pocket-directory-to-the-principal-mercantile-houses-in-the-city-of-philadelphia-for-the-year-1839/oclc/35144726&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wholesale Business Intelligence and Southern and Western Merchants' Pocket Directory to the Principal Mercantile Houses in the City of Philadelphia, for the Year 1839&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;As I could not fetch this source, I will stick to the Luhn/Dresner acception. It relates to the &lt;a href=&quot;http://www.etymonline.com/index.php?term=intelligence&quot; rel=&quot;nofollow noreferrer&quot;&gt;etymology of intelligence&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;late 14c., &amp;quot;faculty of understanding,&amp;quot; from Old French intelligence&#xA;(12c.), from Latin intelligentia, intellegentia &amp;quot;understanding, power&#xA;of discerning; art, skill, taste,&amp;quot; from intelligentem (nominative&#xA;intelligens) &amp;quot;discerning,&amp;quot; present participle of intelligere &amp;quot;to&#xA;understand, comprehend,&amp;quot; from inter- &amp;quot;between&amp;quot; (see inter-) + legere&#xA;&amp;quot;choose, pick out&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In Business Intelligence for Dummies (Scheps, 2008), the definition chapter plays on Military Intelligence:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Business Intelligence Defined: No CIA Experience Required So what the&#xA;heck is business intelligence, anyway? In essence, BI is any activity,&#xA;tool, or process used to obtain the best information to support the&#xA;process of making decisions.&lt;/p&gt;&#xA;&lt;p&gt;For our purposes, BI revolves around putting computing power (highly&#xA;specialized software in concert with other more common technology&#xA;assets) to work, to help make the best choices for your organization.&lt;/p&gt;&#xA;&lt;p&gt;Business intelligence is essentially timely, accurate, high-value, and&#xA;actionable business insights, and the work processes and technologies&#xA;used to obtain them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I would thus bend toward &amp;quot;Gathering data and information about the business&amp;quot;, maybe more &amp;quot;to better conduct business&amp;quot;. Additional historical comments can be found in &lt;a href=&quot;http://monashbi.blogspot.fr/2006/11/father-of-bi-is-he-having-laugh.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Father of BI? Is he having a laugh?!&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="138806" LastEditDate="2023-09-02T13:53:48.573" LastActivityDate="2023-09-02T13:53:48.573" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="8018" PostTypeId="1" CreationDate="2015-09-06T02:04:05.053" Score="8" ViewCount="761" Body="&lt;p&gt;I have data for each vehicle's lateral position over time and lane number as shown in these 3 plots in the image and sample data below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RfJJQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RfJJQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a&#xA;   Frame.ID   xcoord Lane&#xA;1       452 27.39400    3&#xA;2       453 27.38331    3&#xA;3       454 27.42999    3&#xA;4       455 27.46512    3&#xA;5       456 27.49066    3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The lateral position varies over time because a human driver does not have perfect control over vehicle's position. The lane change maneuver starts when the lateral position changes drastically and ends when the variation becomes 'normal' again. This can not be identified from the data directly. I have to manually look at each vehicle's plot to determine the start and end points of lane change maneuver in order to estimate the duration of lane change. But I have thousands of vehicles in the data set. Could you please direct me to any relevant image analysis/ machine learning algorithm that could be trained to identify these points? I work in R. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4933" LastEditorUserId="4933" LastEditDate="2015-09-06T03:53:18.940" LastActivityDate="2015-10-26T19:41:37.260" Title="Are there any machine learning techniques to identify points on plots/ images?" Tags="|machine-learning|r|" AnswerCount="3" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="8019" PostTypeId="2" ParentId="6792" CreationDate="2015-09-06T10:33:39.820" Score="0" Body="&lt;p&gt;There's an inherent problem that anybody who chooses to use a system designed primarily for extreme privacy is interesting to the surveillance org.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the surveillance org will likely learn about any such system or class of systems if it becomes significant, and develop means of identifying its traffic (and possibly filtering out the fake traffic), and then give special attention to the users sending that traffic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also consider the opposite approach: instead of auto-generating traffic for existing users with false negatives, auto-generating new users with traffic full of false positives.  However it suffers from the same &lt;a href=&quot;https://en.wikipedia.org/wiki/Cat_and_mouse&quot; rel=&quot;nofollow&quot;&gt;cat-and-mouse dynamic&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is for the tools used by most users (Android, iOS, telcos, Gmail, Facebook, Google Search, OEMs) to feature security capabilities client-side &lt;em&gt;by default&lt;/em&gt; AND to &lt;em&gt;not&lt;/em&gt; provide backdoors to surveillance orgs.  That would also make it more difficult to provide useful features like recommendations and well-targeted adverts.   For various reasons, in large part because most consumers do not demand it, they do not prioritise this, and that may not ever change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further reading: &lt;a href=&quot;https://blog.kaspersky.com/chrome_ext_encrypt_data_leaving_browser/5063/&quot; rel=&quot;nofollow&quot;&gt;https://blog.kaspersky.com/chrome_ext_encrypt_data_leaving_browser/5063/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2015-09-06T10:33:39.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8020" PostTypeId="2" ParentId="6827" CreationDate="2015-09-06T10:49:47.890" Score="1" Body="&lt;p&gt;Much depends on the data available to you.  Perhaps you can be more specific about the scale and scope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modelling time is the straightforward bit.  To understand how to conceptualise time as useful features, see &lt;a href=&quot;https://datascience.stackexchange.com/a/2370/12363&quot;&gt;this excellent answer&lt;/a&gt; on &lt;a href=&quot;https://datascience.stackexchange.com/questions/2368/machine-learning-features-engineering-from-date-time-data&quot;&gt;Machine learning - features engineering from date/time data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modelling the user is more complicated.  You will likely not have enough data on each user, but you can build some user models.  (Too few, then the system will make similar predictions for all users, without nuance.  Too many and there will be sparsity, overfitting, and generally the same problems as having no profile models at all, ie one model per actual user.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done supervised or unsupervised, finding representative clusters.&lt;br&gt;&#xA;(Search for &lt;em&gt;user profile categorisation&lt;/em&gt;, &lt;em&gt;user models&lt;/em&gt;, &lt;em&gt;user model clustering&lt;/em&gt;)&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-06T10:49:47.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8021" PostTypeId="2" ParentId="6555" CreationDate="2015-09-06T14:01:21.123" Score="4" Body="&lt;p&gt;Assume your configure file is ~/.ipython/profile_pyspark/ipython_notebook_config.py, you can still use this configure file by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ipython notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;jupyter-notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12650" LastActivityDate="2015-09-06T14:01:21.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8023" PostTypeId="2" ParentId="5302" CreationDate="2015-09-07T08:02:34.440" Score="0" Body="&lt;p&gt;In R I would strongly suggest the reshape2 package, in particular the cast / melt combo of functions.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-07T08:02:34.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8024" PostTypeId="1" AcceptedAnswerId="8348" CreationDate="2015-09-07T08:29:25.507" Score="8" ViewCount="1701" Body="&lt;p&gt;Can't reinforcement learning be used without the help of other learning algorithms like SVM and MLP back propagation? I consulted two papers:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.duke.edu/~parr/icml03.pdf&quot; rel=&quot;noreferrer&quot;&gt;Paper 1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5967372&amp;amp;tag=1&quot; rel=&quot;noreferrer&quot;&gt;Paper 2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;both have used other machine learning methods in the inner loop.&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="85045" LastEditDate="2021-02-18T08:55:51.597" LastActivityDate="2021-02-18T08:55:51.597" Title="Does reinforcement learning require the help of other learning algorithms?" Tags="|machine-learning|reinforcement-learning|algorithms|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8025" PostTypeId="2" ParentId="8005" CreationDate="2015-09-07T09:21:14.393" Score="3" Body="&lt;p&gt;First of all, you need to decide how you evaluate &quot;good&quot; vs &quot;bad&quot;, which means picking a metric. From the top of my head you have three choices:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;accuracy: simplest, but potentially misleading (if you have few defaults and predict all 0s, you still get high accuracy)&lt;/li&gt;&#xA;&lt;li&gt;logloss: penalize assigning a high def probability to good quality credit and vice versa&lt;/li&gt;&#xA;&lt;li&gt;ROC: sort the credit from best to worst.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Once you've made your pick, make sure your model can generate predictions in the right format (i.e. output probabilities and not just 0/1) and the go through the training / validation / test paradigm. This should get you started.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-07T09:21:14.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8026" PostTypeId="1" CreationDate="2015-09-07T14:42:14.130" Score="1" ViewCount="229" Body="&lt;p&gt;I have many data in the form of one dimensional histogram, to give an example consider the data at &lt;a href=&quot;http://pastebin.com/embed_js.php?i=1mNRuEHZ&quot; rel=&quot;nofollow noreferrer&quot;&gt; http://pastebin.com/embed_js.php?i=1mNRuEHZ&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fhps8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fhps8.jpg&quot; alt=&quot;histogram test&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect that these data are obtained from a pdf composed of a sum of Gaussian distributions but with their actual number being unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want then to fit these data with a model given by a sum of Gaussians&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;f_N(x) = \sum_{k=1}^N c_k \exp \left[ - \frac{(x-a_k)^2}{2 b_k^2} \right]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $a_k$, $b_k$, $c_k$ &lt;strong&gt;and&lt;/strong&gt; $N$ are in principle parameters to be fitted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem can be viewed as a problem of model selection between different $f_N$ for all the possible values of $N$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is then to make an usual fit by least squares at fixed $N$ and then compare the results for different $N$ via some statistical measure of the quality of the fit, as e.g. Akaike information criterion (AIC).&#xA;Is this an acceptable test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the data of the previous example (that I generated as a test from 3 gaussians) I will obtain&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; N        AIC&#xA; -------------&#xA; 1      +568.1&#xA; 2      +557.4&#xA; 3      -446.6&#xA; 4      -443.5&#xA; 5      -442.7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you see, the AIC decreases quickly till $N=3$, where it starts to increase again very slowly (due to overfitting). So it is clear that $N=3$ is the best choice here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mention that in some cases I have more data histograms extracted from the same pdf, so I could do some cross validation. Although this always a good test, the availability of my data changes from sample to sample, so I'd prefer to have a criterion for the single histogram (if the quality of the test is not too much different).  &lt;/p&gt;&#xA;" OwnerUserId="12665" LastEditorUserId="12665" LastEditDate="2015-09-11T16:34:14.003" LastActivityDate="2015-09-11T16:34:14.003" Title="Fittting histograms with combination of an unknown number of Gaussians" Tags="|statistics|beginner|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="8027" PostTypeId="1" CreationDate="2015-09-07T15:16:28.267" Score="1" ViewCount="464" Body="&lt;p&gt;I want to make prediction for quantity of stock that will be sufficient over a period of time i.e from one delivery to another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming, i want a store to have minimum amount of item between deliveries e.g 30 items, delivery interval can be 7 , 14, 30, days. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also i have history data of past delivery interval, amount of item delivered and the amount of item available before next delivery.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What will be the best way to relate these variable: delivery interval and on hand quantity to quantity delivered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also can i predict quantity to be delivered in other to get a given amount of on hand stock before next delivery.&lt;/p&gt;&#xA;" OwnerUserId="5027" LastActivityDate="2015-09-07T15:16:28.267" Title="Quantity Prediction Algorithm" Tags="|predictive-modeling|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8028" PostTypeId="1" CreationDate="2015-09-07T15:41:26.427" Score="1" ViewCount="221" Body="&lt;p&gt;I am studying data science education curriculum design and guidelines. All resources I have up to now is about data science in statistics education and library and information science education.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any work on data science in computer science/software engineering education (adding data science courses to a traditional software engineering curriculum) ,management and other disciplines? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of including the curriculum for statistics is this &lt;a href=&quot;https://www.researchgate.net/publication/266856309_Data_Science_in_Statistics_Curricula_Preparing_Students_to_Think_with_Data&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="11097" LastEditDate="2015-09-10T10:27:18.247" LastActivityDate="2015-09-29T17:35:34.363" Title="Data Science education curriculum design and guidelines in Computer Science and other Disciplines" Tags="|data|education|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8029" PostTypeId="1" AcceptedAnswerId="8059" CreationDate="2015-09-07T16:14:16.143" Score="3" ViewCount="491" Body="&lt;p&gt;Consider the application:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;We have a set of users and items.&lt;/li&gt;&#xA;&lt;li&gt;Users can perform different action types (think browsing, clicking, upvoting etc.) on different items.&lt;/li&gt;&#xA;&lt;li&gt;Users and items accumulate a &quot;profile&quot; for each action type.&lt;/li&gt;&#xA;&lt;li&gt;For users such profile is a list of items on which they had performed a given action type&lt;/li&gt;&#xA;&lt;li&gt;For items such profile is a list of users who performed a given action type on them&lt;/li&gt;&#xA;&lt;li&gt;We assume that accumulated profiles define future actions.&lt;/li&gt;&#xA;&lt;li&gt;We want to predict the action a user will take using supervised learning (classification with probability estimation)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Consider the following problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;These profiles can be very sparse (millions of items and 100 million users) and it is not feasible to use them directly as features&lt;/li&gt;&#xA;&lt;li&gt;We would like to compute &quot;compressed&quot; profiles (eigenprofiles?:)) with dimensionality &amp;lt; 300 that can then be efficiently stored and fed to different classification algorithms&lt;/li&gt;&#xA;&lt;li&gt;Before you say &quot;Use TruncatedSVD/Collapsed Gibbs Sampling/Random Projections on historical data&quot; bare with me for a second.&lt;/li&gt;&#xA;&lt;li&gt;Enters concept drift.&lt;/li&gt;&#xA;&lt;li&gt;New items and users are being introduced all the time to the system.&lt;/li&gt;&#xA;&lt;li&gt;Old users churn.&lt;/li&gt;&#xA;&lt;li&gt;Old items churn.&lt;/li&gt;&#xA;&lt;li&gt;At some point there are items with most of the users never seen in historical data and users with only fresh items.&lt;/li&gt;&#xA;&lt;li&gt;Before you say &quot;retrain periodically&quot;, remember that we have a classifier in the pipeline that was taught on the &quot;historic&quot; decomposition and the new decomposition could assign entirely different &quot;meaning&quot; to cells of output vectors (abs(decompose_v1(sample)[0] - decompose_v2(sample)[0]) &gt;&gt; epsilon) rendering this classifier unusable.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some requirements:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The prediction service has to be available 24/7.&lt;/li&gt;&#xA;&lt;li&gt;The prediction cannot take more than 15ms and should use a maximum of 4 cpu cores (preferably only one)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some ideas I had so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;We could retrain the classifier on the new decomposition but this would mean that we have to re-run the decomposition on the whole training dataset (with snapshot of profiles at the time of the event we want to predict) and the whole database (all current profiles) plus store it. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To make this work we would have to have a second database for storing the decomposed profiles that would be hot-swapped once the new retrained model is ready and all profiles have been decomposed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This approach is quite inefficient in both computational resources and storage resources (this is expensive storage because the retrieval has to be super-fast)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;ul&gt;&#xA;&lt;li&gt;We could retrain the classifier as in solution 1. But do the decomposition ad_hoc.&lt;/li&gt;&#xA;&lt;li&gt;This puts a lot of constraints on the speed of the decomposition (has to have sub-millisecond computation times for a single sample).&lt;/li&gt;&#xA;&lt;li&gt;This does a lot of redundant computation (especially for item profiles) unless we add an extra caching layer.&lt;/li&gt;&#xA;&lt;li&gt;This avoids redundant storage and redundant computation of churned users/items at the cost of extra prediction latency and extra caching layer complexity. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&amp;lt;---- Please help me here&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;We could use one of online learning algorithms such as VFT or Mondrian Forests for the classifier - so no more retraining + nice handling of concept drift.&lt;/li&gt;&#xA;&lt;li&gt;We would need an online algorithm for decomposition that satisfies strict requirements:&#xA;a) at least a part of output vectors should be stable between increments (batches).&#xA;b) it can introduce new features to account for new variance in the data but should do so at a controllable rate&#xA;c) should not break if it encounters new users/items&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Questions/points of action:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Please evaluate my proposed solutions and propose alternatives&lt;/li&gt;&#xA;&lt;li&gt;Please provide algorithms suitable for online learning and online decomposition (if they exist) as described in alternative 3. Preferably with efficient python/scala/java implementations with a sufficient layer of abstraction to use them in a web service (python scripts that take in a text file as dataset would be much less valuable than scikit modules)&lt;/li&gt;&#xA;&lt;li&gt;Please provide links to relevant literature that dealt with similar problems/describes algorithms that could be suitable&lt;/li&gt;&#xA;&lt;li&gt;Please share experiences/caveats/tips that you learned while dealing with similar problems  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some background reading that you may find useful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quora.com/What-machine-learning-algorithms-are-used-for-internet-advertising&quot; rel=&quot;nofollow&quot;&gt;quora question on ad click prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/41159.pdf&quot; rel=&quot;nofollow&quot;&gt;google &quot;view from the trenches&quot;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://people.csail.mit.edu/romer/papers/TISTRespPredAds.pdf&quot; rel=&quot;nofollow&quot;&gt;criteo paper on click prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.quinonero.net/Publications/predicting-clicks-facebook.pdf&quot; rel=&quot;nofollow&quot;&gt;facebook on predicting ad clicks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: Our application is not strictly ad conversion prediction and some problems such as rarity do not apply. The event we would like to predict has 8 classes and occurs c.a. 0.3%-3% of times a user browses an item. &lt;/p&gt;&#xA;" OwnerUserId="12663" LastActivityDate="2015-09-10T15:07:14.920" Title="Online/incremental unsupervised dimensionality reduction for use with classification for event prediction" Tags="|classification|performance|dimensionality-reduction|online-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8030" PostTypeId="2" ParentId="8024" CreationDate="2015-09-07T16:18:51.340" Score="3" Body="&lt;p&gt;&quot;Reinforcement Learning&quot; means the result of the learning algorithm is a policy; a function that takes a set of inputs and returns a decision. &quot;Supervised Learning&quot;, in contrast, learns a function that returns a prediction. They are different types of task.&lt;br&gt;&#xA;Multi-Layer Perceptron and Support Vector Machine are architectures i.e. forms for the learnt function in either case.&lt;br&gt;&#xA;There is no reason not to try different architectures on different tasks.&lt;/p&gt;&#xA;" OwnerUserId="10568" LastActivityDate="2015-09-07T16:18:51.340" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8031" PostTypeId="2" ParentId="289" CreationDate="2015-09-07T17:53:45.567" Score="2" Body="&lt;p&gt;You have the option of joining a PhD program in business school and information school as well. There are quantitative professors and data scientists in business schools and information schools as well (About US, I am sure there are a lot of schools). This way you are qualified or even over-qualified in terms of quantitative and technical skills and you can spend your time on reinforcing other skills.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-09-07T17:53:45.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8032" PostTypeId="1" CreationDate="2015-09-08T03:14:09.230" Score="25" ViewCount="79354" Body="&lt;p&gt;The below predict function is giving -ve values as well so it cannot be probabilities. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param &amp;lt;- list(max.depth = 5, eta = 0.01,  objective=&quot;binary:logistic&quot;,subsample=0.9)&#xA;bst &amp;lt;- xgboost(param, data = x_mat, label = y_mat,nround = 3000)&#xA;&#xA;pred_s &amp;lt;- predict(bst, x_mat_s2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I google &amp;amp; tried &lt;code&gt;pred_s &amp;lt;- predict(bst, x_mat_s2,type=&quot;response&quot;)&lt;/code&gt;&#xA;but it didn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to predict probabilities instead?&lt;/p&gt;&#xA;" OwnerUserId="1151" LastEditorUserId="123594" LastEditDate="2021-08-28T04:41:47.753" LastActivityDate="2022-06-06T09:06:55.177" Title="How to predict probabilities in xgboost using R?" Tags="|machine-learning|r|predictive-modeling|decision-trees|xgboost|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="8036" PostTypeId="2" ParentId="6642" CreationDate="2015-09-08T10:35:08.143" Score="0" Body="&lt;p&gt;Look at using Power Query.   For example: &lt;a href=&quot;http://www.poweredsolutions.co/2014/11/06/getting-api-results-from-azure-machine-learning-into-power-query/&quot; rel=&quot;nofollow&quot;&gt;http://www.poweredsolutions.co/2014/11/06/getting-api-results-from-azure-machine-learning-into-power-query/&lt;/a&gt;   As per the bottom of the url, power query can be a source into AML.&lt;/p&gt;&#xA;" OwnerUserId="12536" LastActivityDate="2015-09-08T10:35:08.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8038" PostTypeId="1" AcceptedAnswerId="8060" CreationDate="2015-09-08T12:11:47.187" Score="1" ViewCount="1240" Body="&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import linear_model&#xA;&#xA;X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])&#xA;Y = np.array(['C++', 'C#', 'java','python'])&#xA;clf = linear_model.SGDClassifier()&#xA;clf.fit(X, Y)&#xA;print (clf.predict([[1.7, 0.7]]))&#xA;#python&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict the values from arrays Y by giving a test case and training it on a training data which is &lt;strong&gt;X&lt;/strong&gt;, Now my problem is that, I want to change the training set &lt;strong&gt;X&lt;/strong&gt; to &lt;strong&gt;TF-IDF Feature Vectors&lt;/strong&gt;, so how can that be possible?&#xA;Vaguely, I want to do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import linear_model&#xA;&#xA;X = np.array_str([['abcd', 'efgh'], ['qwert', 'yuiop'], ['xyz','abc'],  ['opi', 'iop']])&#xA;Y = np.array(['C++', 'C#', 'java','python'])&#xA;clf = linear_model.SGDClassifier()&#xA;clf.fit(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12557" LastEditorUserId="2750" LastEditDate="2015-09-08T16:06:06.113" LastActivityDate="2020-08-03T09:46:14.697" Title="Passing TFIDF Feature Vector to a SGDClassifier from sklearn" Tags="|machine-learning|classification|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8039" PostTypeId="2" ParentId="8038" CreationDate="2015-09-08T13:49:19.217" Score="0" Body="&lt;p&gt;You can find a nice tutorial how to achieve that on &lt;a href=&quot;http://blog.christianperone.com/2011/10/machine-learning-text-feature-extraction-tf-idf-part-ii/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;This is acctually a part II. In part I the author discusses how and what is &amp;quot;Term frequency&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.christianperone.com/2011/09/machine-learning-text-feature-extraction-tf-idf-part-i/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link to part I&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="98307" LastEditDate="2020-08-03T09:46:14.697" LastActivityDate="2020-08-03T09:46:14.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8043" PostTypeId="1" AcceptedAnswerId="8061" CreationDate="2015-09-08T17:51:50.850" Score="5" ViewCount="2260" Body="&lt;p&gt;I am working on a problem that involves finding optimal parameter values for a black-box system. This system consists of 6 inputs and produces a single output. I can compare the returned values to observed values to determine whether the system is well calibrated based on the parameter values I specified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problem that I am having is that I do not know how these parameters are used in the black-box system to produce the output. The parameters may be correlated so changing the value of one parameter may affect the way the others behave in the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I do not know the function I am not sure how to optimize this problem efficiently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; what are some methods for optimization when the function is unknown?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edits:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variable types are each a vector of real numbers but we can make some assumptions if it is is helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost to run the black-box system is only time. Let's say that it takes 10 seconds to run the system but I happen to have five black boxes -- so if I can run my algorithm in parallel then I can cut down the run time. If the algorithm runs sequentially then I don't gain anything by having the extra boxes (expect perhaps choosing different starting positions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this problem I am interested in learning about how to solve this type of problem -- not trying to brute force my way to a solution. That is to say I am &lt;em&gt;less&lt;/em&gt; interested in obtaining an answer and &lt;em&gt;more&lt;/em&gt; finding out &lt;em&gt;how&lt;/em&gt; I would obtain the the answer. I think it is easy to say that given sufficient resources (time, processing power) that simply generating many many random numbers and choosing the combination that gives the best result is the simplest (and probably worst) approach.&lt;/p&gt;&#xA;" OwnerUserId="5103" LastEditorUserId="5103" LastEditDate="2015-09-08T18:33:55.383" LastActivityDate="2016-05-19T07:52:27.720" Title="Optimizing parameters for a closed (black-box) system" Tags="|optimization|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8045" PostTypeId="1" CreationDate="2015-09-09T12:10:46.467" Score="1" ViewCount="238" Body="&lt;p&gt;My apologies for cross-posting to stackoverflow and cross validated. Not really sure which one is the most relevant place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please shed some light on me with this task.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Description&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Assuming the training data looks like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;0 | X773579,Y2640,Y2072,Z4,Z15&#xA;1 | X374616379,X773579,X344420902,Y1940,Y1705,Z4,Z15,Z26&#xA;...&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One would like to predict 0/1 from those comma separated values, without further explanation about what exactly the X, Y, and Z with numbers mean. Some rare cases even have un-numbered X or Y (possibly just noises though). The length of comma separated values varies, but each row contains no repeated values. Also, the length of numbers seem correlated to X, Y, and Z.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data size is 1 million rows.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Feature&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In LibSVM format&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Label | Value_ID:1...&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;0 1:1 2:1 3:1 4:1 5:1&#xA;1 1:1 4:1 5:1 14:1 36:1 37:1 38:1 39:1&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've tried treating those values categorically and applying several different models of Spark MLlib, namely linear regression, logistic regression, decision tree, random forests, gradient-boosted trees, and SVM, along with LibSVM and Conditional Random Fields. Their 5-fold cross-validations got only 75%~79% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really want to know if there's any formal way to discover the signal from the data, build up models, extract and even select features accordingly. Admittedly I haven't done ANOVA yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Background (self-disclosure)&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is a private interview question and the employer didn't share any details (except &quot;no offer&quot; response :p). Although I specifically asked twice that I would like to learn from it, no reply so far.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Book studying/studied&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pattern classification&lt;/li&gt;&#xA;&lt;li&gt;Pattern recognition&lt;/li&gt;&#xA;&lt;li&gt;Foundations of Statistical Natural Language Processing&lt;/li&gt;&#xA;&lt;li&gt;Speech and Language Processing&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;What I may try next&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Based on a discussion with @NeilSlater :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have done some quick frequent pattern analysis and found certain n-grams. The problem was that they seemed associated with both labels 0 and 1. I will run some feature templates of bi-gram and tri-gram, and then get back to you. One thing that is also intriguing to me is, in NLP, sometimes labels also got dependencies, namely some 1 may be followed by certain patterns of 0 and vice versa.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Updates&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In terms of linear/logistic regressions and conditional random fields:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;n-gram is slightly harmful: tried adjacent bigram/trigram &amp;amp; pair/triple combinations, all got worse about 1%&lt;/li&gt;&#xA;&lt;li&gt;occurrences of X/Y/Z &amp;amp; mean/max/min/sum/avg of X/Y/Z can only reach 69%.&lt;/li&gt;&#xA;&lt;li&gt;X or Y or Z solely as categorical feature got 70%~75%. Y is the most useful category.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Other models listed in the original description all got far worse results.&lt;/p&gt;&#xA;" OwnerUserId="12699" LastEditorUserId="12699" LastEditDate="2015-09-13T15:19:42.773" LastActivityDate="2015-09-13T15:19:42.773" Title="Binary classification with unexplained data" Tags="|machine-learning|self-study|" AnswerCount="0" CommentCount="14" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8046" PostTypeId="1" CreationDate="2015-09-09T13:11:46.470" Score="2" ViewCount="126" Body="&lt;p&gt;I have a model at work that I am building and am running into some odd outputs from the random forest as it pertains to the probability of response. In my case, the class distributions are very unbalanced, around 45000 records in my training data, and only 300 are responders. So I took a smaller sample of the non-responders, 1000, so that my training data was 1300 total. This worked well, but it appears like the outputted predicted probability needs to be adjusted in some way given that I altered the distribution of the classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I create a gains table, the predicted response rate of the first decile comes out at 50%, but it should be around 2%. I dont have this problem with running an xgboost model with all training variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that in traditional logistic regression, you can weight the classes if you take a smaller sample so that it can accordingly adjust the predicted probability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way using random forest to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="12235" LastActivityDate="2015-09-09T13:11:46.470" Title="Random Forest Class Weighting for Logistic Probabilities" Tags="|r|logistic-regression|random-forest|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8048" PostTypeId="1" AcceptedAnswerId="8056" CreationDate="2015-09-09T14:46:20.600" Score="3" ViewCount="3419" Body="&lt;p&gt;I am preparing a lecture on data mining algorithms in R and I want to demonstrate the famous &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;Apriori algorithm&lt;/a&gt; in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question&lt;/strong&gt;&lt;br&gt;&#xA;Could anybody point me to a simple implementation of this algorithm in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am not looking for a package, e.g. arules, but for comprehensible source code of an implementation from scratch. It also does not help to look at the source code of arules: It just calls a C implementation of the algorithm!)&lt;/p&gt;&#xA;" OwnerUserId="690" LastActivityDate="2017-03-23T19:26:11.960" Title="Simple implementation of Apriori algorithm in R" Tags="|r|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8050" PostTypeId="2" ParentId="8028" CreationDate="2015-09-09T17:33:26.853" Score="2" Body="&lt;p&gt;Yes, data science work do exist in domains of management and computer science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think data science is a huge domain in itself to include in any of the curricula. So, I would explain how data science is used in the domains:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Management:&lt;/strong&gt; &#xA;Data Science used in a lot of ways. I use data science and analytics in every phase of the customer acquisition and retention funnel; which includes designing models for using the customer event tracking data and identify patterns and predict churn. Here is an &lt;a href=&quot;https://moz.com/devblog/deep-learning-for-customer-churn-prediction/&quot; rel=&quot;nofollow&quot;&gt;example article&lt;/a&gt; from the Moz blog, which explains about the use of deep for churn prediction and customer retention. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Computer Science:&lt;/strong&gt; &#xA;Most of the data science has roots in computer science basics. However, including the concepts of distributed computing would help the students understand and appreciate the concept of Big Data, when they later enter the industry.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, I think data science is a huge domain in itself, and has it's share of unique applications in a wide range of domains. However, bulk of data science concepts and techniques have taken roots in math, stats and computer science basics and concepts. So, I think the knowledge of those domains would be helpful and suffice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, these are some links which can help in curriculum re-design for accommodating data science concepts in the management and CS curricula:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;MBA speciality.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scu.edu/business/graduates/current-students/data-science.cfm&quot; rel=&quot;nofollow&quot;&gt;Curriculum of the Data Science and Business Analytics program&lt;/a&gt; at the Santa Clara University.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dsi.virginia.edu/curriculum&quot; rel=&quot;nofollow&quot;&gt;Curriculum&lt;/a&gt; of the Masters of Data Science in Virginia University.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The third bullet is something very close to how data science can be included or blended with the Computer Science degree.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-09-10T10:36:20.380" LastActivityDate="2015-09-10T10:36:20.380" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8051" PostTypeId="2" ParentId="8048" CreationDate="2015-09-09T17:58:20.117" Score="2" Body="&lt;p&gt;I found an implementation here: &lt;a href=&quot;https://books.google.de/books?id=ifpzBgAAQBAJ&amp;amp;lpg=PA50&amp;amp;ots=rkVf2uZvYo&amp;amp;pg=PA47#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Learning Data Mining with R by Bater Makhabel&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The source code can be downloaded here: &lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/learning-data-mining-r&quot; rel=&quot;nofollow&quot;&gt;https://www.packtpub.com/big-data-and-business-intelligence/learning-data-mining-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="690" LastActivityDate="2015-09-09T17:58:20.117" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8053" PostTypeId="2" ParentId="6478" CreationDate="2015-09-09T23:13:43.420" Score="1" Body="&lt;p&gt;as @image_doctor mentioned, this is clearly a non-trivial problem, but if you want a bit of help extracting numerical data from existing graphs, &lt;a href=&quot;http://arohatgi.info/WebPlotDigitizer/&quot; rel=&quot;nofollow&quot;&gt;WebPlotDigitizer&lt;/a&gt; might be of assistance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps investigating their methods via &lt;a href=&quot;https://github.com/ankitrohatgi/WebPlotDigitizer?&quot; rel=&quot;nofollow&quot;&gt;their source&lt;/a&gt; may be fruitful for you.&lt;/p&gt;&#xA;" OwnerUserId="11098" LastActivityDate="2015-09-09T23:13:43.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8054" PostTypeId="1" AcceptedAnswerId="16086" CreationDate="2015-09-09T23:32:26.267" Score="2" ViewCount="140" Body="&lt;p&gt;I have a profile hidden Markov model that I use to identify all instances of a user-defined pattern of symbols in a long sequence of symbols. I use the Viterbi algorithm to find the most probable path that generates that long sequence of symbols, and it all works very well to identify the user's pattern. But I'm now interested in extending this, and identifying the k-most probable paths through my model that would generate some long sequence. Is there an existing algorithm to do this? I've come across something called the &quot;1-best&quot; or &quot;k-best&quot; algorithm, but I can't find much of anything describing it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also considered what I'm sure would be a heuristic, by finding the most probable path and then returning the neighbourhood around it. This would be found by iterating over each move between states in the most probable path, setting that particular step to 0, and resolving using Viterbi. This would obviously increase the run-time by a factor of (length of the path ~= length of pattern), which I suspect will be reasonable for my purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody come across something like this, or can anybody see anything obviously wrong in my neighbourhood idea? I'd appreciate any feedback. As a note, this has been cross-posted to the Theoretical Computer Science forum.&lt;/p&gt;&#xA;" OwnerUserId="12710" LastActivityDate="2017-01-04T23:03:28.410" Title="How can I rank paths through an HMM?" Tags="|algorithms|markov-process|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8055" PostTypeId="2" ParentId="6937" CreationDate="2015-09-10T00:41:26.400" Score="2" Body="&lt;p&gt;I, like you, took it upon myself to change career paths into the growing field of data science. As a bit of background, I was working in a neuroscience research lab studying the effects of allelic variation on protein transport in rodent models of alzheimer's disease and post-traumatic stress disorder. I was a competent R, bash, and Matlab hacker and knew a bit of C and Java. I had a change of heart when I started applying for neuroscience PhD programs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over the next year, i took a few grad-level stats courses and got some of my programming skill back in order. Last year, I started an MS program for CS with the intent of 1) getting an job as a data scientist in industry 2) learning how to think algorithmically / mathematically about data 3) really sharpen my fundamental CS skills 4) have some fun in the process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I won't list specific courses that helped, but the major winning topics for me were:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Algorithms &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;both fundamental algos&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;complexity analysis, sorting and searching, graph algorithms, dynamic programming, randomized algos, etc. &lt;a href=&quot;https://www.coursera.org/course/algo&quot; rel=&quot;nofollow&quot;&gt;Tim Roughgarden&lt;/a&gt; has a great course for this.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;data mining/massive dataset/stream algorithms&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;locality sensitive hashing, sketch algorithms, kd/ball trees, reservoir sampling, sliding window methods, etc. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Databases/Data streams:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;noSQL, SQL, hadoop, etc. Working with them forces you to start appreciating that most of the data you will work with absolutely doesn't fit in memory and requires unique methods to not only extract information from it, but just to work with it and examine it. Learn how to use one by building a web scraper or data harvester to populate a database (or however else you want...)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Machine learning&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Learning the fundamental methods in the field, e.g., tree-based methods, optimization, neural networks, svms, regression, markov chains, graphical methods,  ensemble methods, overfitting, regularization, clustering, k-means, knn, etc. As mentioned, &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;Andrew Ng's coursera class&lt;/a&gt; is probably the best open-source solution for this one.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;EDIT&lt;/em&gt;: The Coursera course is good iff you supplement it with stuff from the real version (&lt;a href=&quot;https://www.youtube.com/watch?v=UzxYlbK2c7E&amp;amp;list=PLA89DCFA6ADACE599&quot; rel=&quot;nofollow&quot;&gt;lectures&lt;/a&gt;, &lt;a href=&quot;http://cs229.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;notes&lt;/a&gt;). I think the coursera class is a bit light and a nice overview of topics in ML, but doesn't go nearly deep enough into the mechanics of those topics. Tom Mitchell's &lt;a href=&quot;http://www.cs.cmu.edu/~tom/mlbook.html&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; is also a good resource. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;data visualization&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;nothing terribly fancy is totally necessary, but knowing how to visualize multidimensional data is quite useful. Learn a good plotting package and perhaps experiment with other tech, like D3.js or mapping visualizations. If you get &lt;a href=&quot;http://www.nytimes.com/interactive/2014/12/29/us/year-in-interactive-storytelling.html#data-visualization&quot; rel=&quot;nofollow&quot;&gt;really good at this&lt;/a&gt;, you'll probably have a great job forever.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Real-world&quot; experience&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I learned about as much in three months at a big web company doing data science than i did in the previous 1.5~ years  of grad school and self-study. This can be approximated by doing Kaggle competitions or the like, but honestly, data in the wild is &lt;em&gt;considerably&lt;/em&gt; harder to work with than most of what is on Kaggle (note that the microsoft malware detection project or some of the computer-vision projects are much more useful for learning to work with messy data that is also rather &quot;big&quot;). &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Notice that i didn't mention anything like &quot;learn R it's the best&quot; or &quot;learn python it's way betterz than R&quot;. I use Python, C, MySQL, MongoDB, and R for most of my work and current research (though I really prefer the python ecosystem these days). I'm sure that this will change in the future. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This falls a bit outside of your question, but perhaps the most critical thing about being an industry data scientist is the ability to work in a mostly unsupervised manner and communicate results/methodology clearly to a team of non-experts. Having a background in scientific research helps with this, as the questions you are trying to answer are difficult, often unstructured, and are often at the precipice of the knowledge ledge for your domain. My friends, acquaintances, and past coworkers working as data scientists in industry were nearly all ex-academic-path folks with MS or PhDs and at least a few publications under their belts. I absolutely do not believe this is a strict requirement and if i were in a hiring position, I'd never elect to filter out someone just because they didn't have an advanced degree, but the industry job postings do seem to be trending towards requiring an MS/PhD or equivalent experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bear in mind, all of this comes from just some dude from nowhere who hasn't been directly in the field all that long but whose transition seems to be going well. &lt;/p&gt;&#xA;" OwnerUserId="11098" LastEditorUserId="11098" LastEditDate="2015-09-10T06:48:48.087" LastActivityDate="2015-09-10T06:48:48.087" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8056" PostTypeId="2" ParentId="8048" CreationDate="2015-09-10T03:19:21.917" Score="2" Body="&lt;p&gt;Have you checked the following reference out?&#xA;Link: &lt;a href=&quot;http://www.borgelt.net/docs/apriori.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.borgelt.net/docs/apriori.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above link has the explanation along with the code. &lt;/p&gt;&#xA;" OwnerUserId="5179" LastEditorUserId="5179" LastEditDate="2015-09-14T16:14:13.707" LastActivityDate="2015-09-14T16:14:13.707" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="8057" PostTypeId="2" ParentId="6910" CreationDate="2015-09-10T04:39:05.057" Score="2" Body="&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Map reduce &lt;em&gt;by definition&lt;/em&gt; works at &lt;em&gt;one record at a time&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, &lt;strong&gt;you cannot compute a distance/similarity matrix&lt;/strong&gt; without abusing the programming model. Methods such as SVM &lt;em&gt;do&lt;/em&gt; require pairwise similarities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such abuses are really common though (which is why map reduce is dead). Most easily, you can map every object to the key 0, and do all the work in the &quot;reducer&quot;. More clever approaches divide the data into k partitions, so they can run at least some work in parallel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But all of these approaches are ugly hacks that hack around a deliberate design decision made for map reduce: in map reduce, every record &lt;em&gt;must&lt;/em&gt; be processed independently, to allow the platform to automatically optimize parallelism and recovery. In above hack, the map phase is abused to split the data into the manually chosen parallelism, and then the reducers are simple parallel jobs. It's no longer mapreduce, but the abstract model of parallel programming on partitioned data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After all the early hype, mapreduce is now dead. Tools such as Apache Mahout &lt;strong&gt;no longer accept mapreduce&lt;/strong&gt; but expect you to move on to more flexible pardigms that don't need such abusive hacks.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-09-10T04:39:05.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8058" PostTypeId="2" ParentId="6964" CreationDate="2015-09-10T08:07:49.920" Score="4" Body="&lt;p&gt;I found this page, but the R package seems not to be open source :&#xA;&lt;a href=&quot;http://www.cs.colostate.edu/~idfah/main/projects/rnn&quot; rel=&quot;nofollow&quot;&gt;link.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12712" LastActivityDate="2015-09-10T08:07:49.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8059" PostTypeId="2" ParentId="8029" CreationDate="2015-09-10T15:07:14.920" Score="2" Body="&lt;p&gt;My take:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I agree with the issues raised in 1., so not much to add here - retraining and storage is indeed inefficient&lt;/li&gt;&#xA;&lt;li&gt;Vowpal Wabbit &lt;a href=&quot;http://hunch.net/~vw/&quot; rel=&quot;nofollow&quot;&gt;http://hunch.net/~vw/&lt;/a&gt; would be my first choice&lt;/li&gt;&#xA;&lt;li&gt;stability of output between increments is really more of a data than algorithm feature - if you have plenty of variation on input, you won't have that much stability on output (at least not by default)&lt;/li&gt;&#xA;&lt;li&gt;hashing can take care of the variation - you can control by a combination of three parameters: the size of the hashing table and the l1/l2 regularization&lt;/li&gt;&#xA;&lt;li&gt;same for the new features / users (I &lt;em&gt;think&lt;/em&gt; - most of the applications i used it had a ercord representing a user clicking or not, so new users / ads were sort of treated &quot;the same&quot;)&lt;/li&gt;&#xA;&lt;li&gt;normally I use VW from the command line, but an example approach (not too elegant) for controlling from Python is given here: &lt;a href=&quot;http://fastml.com/how-to-run-external-programs-from-python-and-capture-their-output/&quot; rel=&quot;nofollow&quot;&gt;http://fastml.com/how-to-run-external-programs-from-python-and-capture-their-output/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;if you prefer sth purely Python, then a version (without decomposition) of an online learner in the Criteo spirit can be found here: &lt;a href=&quot;https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10537/beat-the-benchmark-with-less-than-400mb-of-memory&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/tradeshift-text-classification/forums/t/10537/beat-the-benchmark-with-less-than-400mb-of-memory&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;I am not sure how to handle the concept drift - haven't paid that much attention to it so far beyond rolling statistics: for the relevant variables of interest, keep track of mean / count over recent N periods. It is a crude approach, but does seem to get the job done it terms of capturing lack of &quot;stationarity&quot; &lt;/li&gt;&#xA;&lt;li&gt;helpful trick 1: single pass over data before first run to create per feature dictionary and flag certain values as rare (lump them into a single value)&lt;/li&gt;&#xA;&lt;li&gt;helpful trick 2: ensembling predictions from more than one model (varying interaction order, learning rate)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-10T15:07:14.920" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8060" PostTypeId="2" ParentId="8038" CreationDate="2015-09-10T15:38:23.237" Score="0" Body="&lt;p&gt;It's useful to do this with a &lt;code&gt;Pipeline&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import linear_model, pipeline, feature_extraction&#xA;&#xA;X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])&#xA;Y = np.array(['C++', 'C#', 'java','python'])&#xA;clf = pipeline.make_pipeline(&#xA;        feature_extraction.text.TfidfTransformer(use_idf=True),&#xA;        linear_model.SGDClassifier())&#xA;clf.fit(X, Y)&#xA;print(clf.predict([[1.7, 0.7]]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12717" LastActivityDate="2015-09-10T15:38:23.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8061" PostTypeId="2" ParentId="8043" CreationDate="2015-09-10T17:06:43.220" Score="3" Body="&lt;p&gt;There are lots of function optimising routines that could be applied, based on the description so far. Random search, grid search, hill-climbing, gradient descent, genetic algorithms, simulated annealing, particle swarm optimisation are all possible contenders that I have heard of, and I am probably missing a few.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trouble is, starting with next to zero knowledge of the black box, it is almost impossible to guess a good candidate from these search options. All of them have strengths and weaknesses. To start with, you seem to have no indication of &lt;em&gt;scale&lt;/em&gt; - should you be trying input parameters in any particular ranges? So you might want to try very crude searches through a range of magnitudes (positive and negative values) to find the area worth searching. Such a grid search is expensive - if you have $k$ dimensions and want to search $n$ different magnitudes, then you need to call your black box $n^k$ times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done in parallel though, and given you are confident that the function is roughly unimodal, you can start with a relatively low number of n (maybe check -10, -1, 0, +1, +10 for 15625 calls to your function taking roughly 8 hours 40 mins using 5 boxes). You may need to repeat with other params once you know whether you have found a bounding box for the mode or need to try yet more values, so this process could take a while longer - potentially days if the optimal value for param 6 is more like 20,000. You could also refine more closely, once you have a potential mode you might want to define another grid of values to search based around. This basic grid search might be my first point of attack on a black box system where I had no clue about parameter meaning, but some confidence that the black box output had a rough unimodal form. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the speed of response you should be storing all input and output values in a database for faster lookup and better model building later. No point repeating a call taking 10 seconds when a cache could look it up in 1 millisecond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have some range of values you think that a mode might be in, then it is time to pick a suitable optimiser.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the information so far, I would be tempted to run either more grid search (with a separate linear scaling between values of each param) and/or a random search, constrained roughly to the boxes defined by the set of $2^6$ corner points &lt;em&gt;around&lt;/em&gt; the best result found in initial order-of-magnitude search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At that point you could also consider graphing the data, to see if there is any intuition about which other algorithms could perform well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the possibility of parallel calls, then gradient descent might be a reasonable guess, because you can get approximate gradients by adding a small offset to each param and dividing difference that causes in the output by it. In addition, gradient descent (or simple hill climbing) has some chance of optimising with less calls to evaluate the function than approaches that rely on many iterations (simulated annealing) or lots of work in parallel (particle swarm or genetic algorithms). Gradient descent optimisers as used in neural networks, with additions like Nesterov momentum or RMSProp, can cope with changes in function output &quot;feature scale&quot; such as different sizes and heights of peaks, ridges, saddle points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, gradient descent and hill climbing algorithms are not robust against all function shapes. A graph or several of what your explorations are seeing may help you to decide on a different approach. So keep all the data and graph it in case you can get clues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, don't rule out random brute-force search, and being able to just accept &quot;best so far&quot; under time constraints. With low knowledge of the internals of the black box, it is a reasonable strategy.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-09-10T17:06:43.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8062" PostTypeId="1" CreationDate="2015-09-10T19:45:25.617" Score="2" ViewCount="169" Body="&lt;p&gt;I've been researching the history and use of k-nearest neighbor classification and regression, and various tweaks including k-d trees and LAESA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it is useful because it is simple and flexible, but can be computationally expensive and requires a lot of data storage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here's what I don't know:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there any class of problems for which nearest neighbor classification is the best or one of the best algorithms to use?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By 'class of problems' I mean either a class based on data structure (for instance, maybe KNN is great for low-dimensional data with a mix of nominal and numerical data), or a class of real-life problems (maybe KNN is useful in predicting diseases for insurance holders). &lt;/p&gt;&#xA;" OwnerUserId="12723" LastActivityDate="2015-10-15T00:13:18.130" Title="When, if ever, is nearest neighbor classification the best choice?" Tags="|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8063" PostTypeId="2" ParentId="8026" CreationDate="2015-09-11T01:29:52.387" Score="1" Body="&lt;p&gt;You're almost done with your results.&#xA;Try to substitute the AIM for the Likelihood of the sum of Gaussian with your data, and as soon as your Likelihood value stabilizes and doesn't increase with the number of Gaussians you've found the right value for your N.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can even try to look for the EM algorithm for the estimation of the Mixture of gaussians that fit your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value of L = f(N) increases as a log function and the so called &quot;knee&quot; is where your optimal value of N lies on!&lt;/p&gt;&#xA;" OwnerUserId="12730" LastActivityDate="2015-09-11T01:29:52.387" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8064" PostTypeId="1" CreationDate="2015-09-11T03:00:48.897" Score="10" ViewCount="11669" Body="&lt;p&gt;I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I couldn't find kappa as an evaluation_metric in scikit learn here &#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics&quot; rel=&quot;noreferrer&quot;&gt;sklearn.metrics&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there any workaround for including kappa in gridsearchcv?&lt;/li&gt;&#xA;&lt;li&gt;Is there any other better metric i can use instead of kappa in scikit learn?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1151" LastActivityDate="2015-11-26T07:01:38.963" Title="How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn?" Tags="|machine-learning|classification|python|predictive-modeling|scikit-learn|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8065" PostTypeId="1" AcceptedAnswerId="8066" CreationDate="2015-09-11T04:49:48.160" Score="0" ViewCount="165" Body="&lt;p&gt;Why is solr used in Big Data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is it's purpose?&lt;/p&gt;&#xA;" OwnerUserId="12732" LastEditorUserId="11097" LastEditDate="2015-09-12T04:04:26.417" LastActivityDate="2015-09-12T04:04:26.417" Title="What are the applications of Solr in Big Data?" Tags="|bigdata|" AnswerCount="1" CommentCount="1" ClosedDate="2015-09-11T14:46:10.837" ContentLicense="CC BY-SA 3.0" />
  <row Id="8066" PostTypeId="2" ParentId="8065" CreationDate="2015-09-11T05:37:09.750" Score="0" Body="&lt;p&gt;Solr is a highly scalable, fault tolerant search server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you can store files in the form of JSON, XML, CSV or binary over HTTP.&#xA;And you can query it via &lt;strong&gt;GET&lt;/strong&gt; and receive JSON, XML, CSV and binary results. It also has a distributed architecture for the purpose, which is the primary and most important concept in Big Data and it's handling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as it is highly scalable and fault tolerant for high traffic and huge documents, it is a choice for a reliable search engine for huge documents, or in short &lt;strong&gt;Big Data&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-09-11T05:37:09.750" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8069" PostTypeId="1" CreationDate="2015-09-11T18:02:44.743" Score="3" ViewCount="397" Body="&lt;p&gt;I have a general question on &lt;a href=&quot;https://en.wikipedia.org/wiki/Reservoir_sampling&quot; rel=&quot;nofollow&quot;&gt;reservoir sampling&lt;/a&gt;. When I use this method to sample a very large dataset for training machine learning classification algorithms, I am curious as to how to make my pipeline robust to fluctuations in  class distribution across samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose I am working on a binary classification problem. I want to sample a relatively small subset of data on which to evaluate my algorithms, so I used reservoir sampling. However, it seems to me (thought I could be wrong) that by chance I may draw a sample wherein the class distribution is significantly different from that seen in the larger population of data. If this is a correct inference, can anybody tell me how to remedy the situation? If reservoir sampling is not the answer, what other procedure are out there that I can leverage in this case (apart from using distributed environment)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am learning, so some explanation, hint, and direction would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="13413" LastEditDate="2015-10-21T08:05:17.897" LastActivityDate="2015-11-13T18:49:00.407" Title="Question on reservoir sampling" Tags="|machine-learning|data-mining|bigdata|sampling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8070" PostTypeId="1" CreationDate="2015-09-11T18:05:06.020" Score="4" ViewCount="2209" Body="&lt;p&gt;I'm visualizing results from lab tests. Most of these results are numerical and therefore easily communicated using line graphs, where the x-axis is time and the y-axis is determined by the range of possible values, like &lt;code&gt;[0, 100]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;EDIT: To be clear, I'm not measuring frequency. I'm displaying an individual's lab test results for one specific test (like cholesterol) over time. Since most people don't get more than 1 or 2 lab tests a year, most people's graphs will have less than a dozen (x,y) values.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm less sure how to visualize this information when the y-axis values are categorical:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Sometimes the possible values are themselves ranges. A test value will be something like &lt;code&gt;2-6&lt;/code&gt;, meaning that the value falls between 2 and 6 &amp;mdash; and that's all we need to know. Would it make sense to graph these values with something like &lt;a href=&quot;http://www.excel-easy.com/examples/images/error-bars/error-bars.png&quot; rel=&quot;nofollow&quot;&gt;error bars&lt;/a&gt;? Or would it make more sense to graph the average of the range (in this case, 4) as a regular line chart?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The other types involve categories which are non-numerical. Sometimes these are ordinal. In the ordinal case the categories might be something like &lt;code&gt;[none, few, moderate, many]&lt;/code&gt;. Does it make sense to just assign an index to each &lt;code&gt;[0, 1, 2, 3]&lt;/code&gt; and use a standard line graph? If not, what would make more sense?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Lastly, non-numerical, nominal types. These are categories with no clear rank hierarchy, like colors: &lt;code&gt;[yellow, orange, red]&lt;/code&gt;. In this case, I think it might be misleading to use a standard line graph, because it would imply that certain colors are &quot;higher&quot; or &quot;better&quot; than others. What would be appropriate?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12747" LastEditorUserId="12747" LastEditDate="2015-09-14T13:49:30.217" LastActivityDate="2016-01-01T06:34:08.317" Title="How to visualize categorical data over time?" Tags="|visualization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8074" PostTypeId="2" ParentId="8062" CreationDate="2015-09-12T02:53:08.243" Score="1" Body="&lt;p&gt;One situation where NN would be ideal is if the data are sample points of a piecewise constant function. In this case, the true function is composed of a tesselation of its domain, with a constant value for all points within each division.&lt;/p&gt;&#xA;" OwnerDisplayName="user9424" LastActivityDate="2015-09-12T02:53:08.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8077" PostTypeId="2" ParentId="8070" CreationDate="2015-09-12T10:01:17.140" Score="1" Body="&lt;p&gt;I'll give it a go, but a lot depends on how you want to communicate your results.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;try to keep the buckets, and add counts to them, so you end up with something like:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;date bucket count&#xA;2015-09-11 2-6 6&#xA;2015-09-12 2-6 9&#xA;2015-09-13 2-6 2&#xA;2015-09-11 10-16 9&#xA;2015-09-12 10-16 15&#xA;2015-09-13 10-16 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This you can communicate either via a line chart, but I would go with side by side bar charts per period, maybe aggregated to week or month depending your timeline. If you have many ranges, splitting them in different graphs. &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Don't index the ordinal values. The reader of the graph should not look somewhere else to figure out what the numbers mean. Just use counts with the ordinals and use line charts or bar charts, summary tables or a cross(table)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Count the occurences and use side-by-side or stacked bar charts and color the bars according to the colors you have. But human perception will always give a higher bar / line / area more weight than a smaller / lower one. Or just represent it as a table. That way you don't give any opinion except that some colors might occur more frequently than other. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-09-12T10:01:17.140" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8079" PostTypeId="1" CreationDate="2015-09-12T14:13:22.560" Score="1" ViewCount="35" Body="&lt;p&gt;I have some issue, during package installation in R version 3.0.2 (2013-09-25) ubuntu machine:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;randomForest&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Installing package into ‘/usr/local/lib/R/site-library’&lt;br/&gt;&#xA;       (as ‘lib’ is unspecified)&lt;br/&gt;&#xA;       Warning in install.packages :&lt;br/&gt;&#xA;       'lib = &quot;/usr/local/lib/R/site-library&quot;' is not writable&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How to resolve it?&lt;/p&gt;&#xA;" OwnerUserId="12732" LastEditorUserId="710" LastEditDate="2015-09-12T18:48:36.543" LastActivityDate="2015-09-12T18:48:36.543" Title="Rstudio - package installation problems" Tags="|r|" AnswerCount="0" CommentCount="1" ClosedDate="2015-09-12T16:10:24.873" ContentLicense="CC BY-SA 3.0" />
  <row Id="8082" PostTypeId="1" CreationDate="2015-09-13T13:53:21.273" Score="1" ViewCount="31" Body="&lt;p&gt;I am trying to implement a simple agent that creates sounds by building up signal building blocks (e.g. sound generators, filters etc.) that can generally be connected in the form of a directed acyclic graph, and each module has quasi-continuous parameter inputs (defined intervals of floating point numbers, e.g. for a frequency, an amplitude etc.). This agent should be able to operate in real-time along with me as a human agent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is to use something like &lt;strong&gt;reinforced learning&lt;/strong&gt; as it allows to operate in an exploratory mode where the state space is very large if not infinite, something that I understand is a disadvantage in a learning-by-example (supervised) mode.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you consider &lt;strong&gt;RL applicable&lt;/strong&gt; in this scenario?&lt;/li&gt;&#xA;&lt;li&gt;How would I go about &lt;strong&gt;defining the state space&lt;/strong&gt;? Is every module (generator or filter) a cell in that space? How about their connections and parameters? Should I introduce a partial ordering of the graph, or radically simplify the state? Perhaps the state should be a feature vector of the actually produced sound instead of the graph structure?&lt;/li&gt;&#xA;&lt;li&gt;I am thinking that I have to provide the &lt;strong&gt;rewards manually&lt;/strong&gt; depending on whether the current result is musically meaningful or bad. Does that make sense? Is there any literature for this &quot;human evaluated&quot; RL? I know there are human fitness functions in genetic programming, so I am thinking this would be an analogous situation.&lt;/li&gt;&#xA;&lt;li&gt;I am worried about the convergence rate. In automatic learning, I can easily perform thousands of steps, but if I have to assign manual reward and I have to listen to the actual sound production, the agent can only act at a fraction of the speed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12770" LastEditorUserId="12770" LastEditDate="2015-09-13T14:18:18.387" LastActivityDate="2015-09-13T14:18:18.387" Title="Designing an (RL) agent for a graph-based music improvisation system" Tags="|machine-learning|algorithms|graphs|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8083" PostTypeId="1" AcceptedAnswerId="8085" CreationDate="2015-09-13T20:28:07.537" Score="1" ViewCount="1555" Body="&lt;p&gt;Source: &lt;a href=&quot;https://d396qusza40orc.cloudfront.net/exdata%2Fdata%2FNEI_data.zip&quot; rel=&quot;nofollow&quot;&gt;https://d396qusza40orc.cloudfront.net/exdata%2Fdata%2FNEI_data.zip&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my data prep.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;NEI &amp;lt;- readRDS(&quot;summarySCC_PM25.rds&quot;)&#xA;Baltimore &amp;lt;- NEI[NEI$fips==&quot;24510&quot;, ]&#xA;Baltimore$type &amp;lt;- as.factor(Baltimore$type)&#xA;Total_Emmisssions &amp;lt;- aggregate(Baltimore$Emissions, &#xA;                               by=list(Baltimore$year, Baltimore$type),&#xA;                               FUN=sum)&#xA;names(Total_Emmisssions) &amp;lt;- c(&quot;Year&quot;,&quot;Type&quot;,&quot;Emissions&quot;)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Plot code  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;g &amp;lt;- ggplot(Total_Emmisssions,aes(x=Year, y=Emissions, colour=Type))&#xA;g1 &amp;lt;- g+geom_point()&#xA;g2 &amp;lt;- g1+facet_grid(. ~ Type)&#xA;g3 &amp;lt;- g2+geom_smooth(method=&quot;lm&quot;, se=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in this plot, the year is always 2002-2008. How do I change the scale / ticks to show from 1999-2008 at an interval of 3 years. Using &lt;code&gt;scale_x_discrete&lt;/code&gt; does not work and it messes the graph. Please help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;str(Total_Emmisssions)&#xA;'data.frame':   16 obs. of  3 variables:&#xA; $ Year     : int  1999 2002 2005 2008 1999 2002 2005 2008 1999 2002 ...&#xA; $ Type     : Factor w/ 4 levels &quot;NON-ROAD&quot;,&quot;NONPOINT&quot;,..: 1 1 1 1 2 2 2 2 3 3 ...&#xA; $ Emissions: num  522.9 240.8 248.9 55.8 2107.6 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="2961" LastEditDate="2015-09-13T22:10:10.143" LastActivityDate="2015-09-13T22:28:37.963" Title="Manage x-axis using ggplot()" Tags="|r|ggplot2|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8084" PostTypeId="1" AcceptedAnswerId="8088" CreationDate="2015-09-13T21:50:39.237" Score="2" ViewCount="2529" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/8739526/2623899&quot;&gt;This answer&lt;/a&gt; to &lt;a href=&quot;https://stackoverflow.com/q/8739227/2623899&quot;&gt;this question&lt;/a&gt; works only for situations in which the desired solution to the coupled functions is not restricted to a certain range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if, for example, we wanted a solution such that &lt;code&gt;0 &amp;lt; x &amp;lt; 10&lt;/code&gt; and &lt;code&gt;0 &amp;lt; y &amp;lt; 10&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are functions within &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/optimize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;scipy.optimize&lt;/code&gt;&lt;/a&gt; that find roots to a function within a given interval (e.g., &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.brentq.html#scipy.optimize.brentq&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;brentq&lt;/code&gt;&lt;/a&gt;), but these work only for functions of &lt;strong&gt;one variable&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does &lt;code&gt;scipy&lt;/code&gt; fall short of providing a root solver that works for multi-variable functions within specific ranges?  How might such a solver be implemented?&lt;/p&gt;&#xA;" OwnerUserId="12777" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-09-14T08:05:32.043" Title="Solve a pair of coupled nonlinear equations within certain limits" Tags="|python|optimization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8085" PostTypeId="2" ParentId="8083" CreationDate="2015-09-13T22:28:37.963" Score="3" Body="&lt;p&gt;Change &lt;code&gt;Year&lt;/code&gt; to a factor and add &lt;code&gt;group=1&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g &amp;lt;- ggplot(Total_Emmisssions,aes(x=factor(Year), y=Emissions, colour=Type, group=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can leave the rest the same (you'll also prbly want to change the &lt;code&gt;xlab&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1Xbqs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1Xbqs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="710" LastActivityDate="2015-09-13T22:28:37.963" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8087" PostTypeId="1" CreationDate="2015-09-14T08:01:06.213" Score="7" ViewCount="983" Body="&lt;p&gt;&lt;strong&gt;The goal:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to machine learning and experimenting with neural networks.&#xA;I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nVEjq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nVEjq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have done:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0KQbzm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0KQbzm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/63Qtpm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/63Qtpm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I am trying to do:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of &lt;em&gt;images&lt;/em&gt;, for two reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;&lt;/li&gt;&#xA;&lt;li&gt;By doing on the images, I can also compress the targets, since they are made of the same images.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What does not work:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fvg7dm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fvg7dm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/ybYxbm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ybYxbm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;24000 pixels =&gt; 50 sigmoids =&gt; 4800 sigmoids (= 4800 pixels)&lt;/li&gt;&#xA;&lt;li&gt;180 &quot;pixels&quot; =&gt; 50 sigmoids =&gt; 36 sigmoids (= 36 &quot;pixels&quot;)&lt;/li&gt;&#xA;&lt;li&gt;180 &quot;pixels&quot; =&gt; 50 sigmoids =&gt; 4800 sigmoids (= 4800 pixels)&lt;/li&gt;&#xA;&lt;li&gt;180 &quot;pixels&quot; =&gt; 50 sigmoids =&gt; 4800 sigmoids =&gt; 36 sigmoids (= 36 &quot;pixels&quot;)&lt;/li&gt;&#xA;&lt;li&gt;180 &quot;pixels&quot; =&gt; 50 sigmoids =&gt; 4800 sigmoids =&gt; 36 linear (= 36 &quot;pixels&quot;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?&lt;/p&gt;&#xA;" OwnerUserId="12776" LastEditorUserId="12776" LastEditDate="2015-09-16T15:15:34.633" LastActivityDate="2015-12-10T17:02:56.773" Title="Why does applying PCA on targets causes underfitting?" Tags="|machine-learning|neural-network|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8088" PostTypeId="2" ParentId="8084" CreationDate="2015-09-14T08:05:32.043" Score="1" Body="&lt;p&gt;As a workaround, you could minimize another function that includes both the objective and the constraints, then check if sol.fun is (numerically) equal to zero.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.optimize import minimize&#xA;import numpy as np&#xA;&#xA;f = lambda x: np.sin(x).sum() #the function to find roots of&#xA;L = np.array([-1,-1]) #lower bound for each coordinate&#xA;U = np.array([1, 1])  #upper bound for each coordinate&#xA;g = lambda x: f(x) **2 +  max(0, (L - x).max(), (x - U).max())&#xA;sol = minimize(g, [0.5,0.5])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, scipy.optimize seems to have some optimisers that support rectangular bounds, i.e. &lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution&quot; rel=&quot;nofollow&quot;&gt;differential_evolution&lt;/a&gt; (since version 0.15.0).&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2015-09-14T08:05:32.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8089" PostTypeId="2" ParentId="492" CreationDate="2015-09-14T09:28:54.003" Score="7" Body="&lt;p&gt;Two recent papers use a Deep learning architecture called CharWNN to address this problem. CharWNN was first used to &lt;a href=&quot;http://jmlr.org/proceedings/papers/v32/santos14.pdf&quot;&gt;get state of the art results&lt;/a&gt; (without handcrafted features) on Part of Speech (POS) tagging on an English corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://arxiv.org/abs/1505.05008&quot;&gt;second paper&lt;/a&gt; by the same author uses the same (or similar) architecture for predicting whether a word belongs to 10 Named Entity classes, with apparent state of the art results.&lt;/p&gt;&#xA;" OwnerUserId="12786" LastActivityDate="2015-09-14T09:28:54.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8090" PostTypeId="1" CreationDate="2015-09-13T17:24:06.520" Score="5" ViewCount="4084" Body="&lt;p&gt;As datasets and the number of parameters get larger, it becomes increasingly difficult to run validation locally because of limited disk size and computing power.  As a result, people might use an old laptop or a server in the cloud.  In particular, I'm interested in learning more about the second option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the setup, it sounds like I will need a storage system and a computing system.  Is this S3 and EC2?  Redshift and Amazon ML?  Combinations from some other vendor?  At the moment, I'm asking as an amateur participant for Kaggle, but I'd also be curious to know what the pros use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the machine itself, is there any way to interact with a GUI rather than a command line to set this up? Would I have to install Anaconda or some other Python distribution before getting started?  Is there a specific file structure to use? What are the main pitfalls to watch out for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, I am looking for the practical advice to set everything up.  If you're able to just offer links to documentation, that would be immensely helpful as well.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="14885" OwnerDisplayName="derekchen14" LastActivityDate="2019-10-06T17:38:29.960" Title="How do I setup a server in the cloud for machine learning?" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8091" PostTypeId="2" ParentId="8090" CreationDate="2015-09-13T19:51:12.860" Score="2" Body="&lt;p&gt;If you are looking for an external storage, then I would suggest you &lt;a href=&quot;http://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html&quot; rel=&quot;nofollow&quot;&gt;Redshift&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Redshift is a central warehouse and a columnar data store. It allows complex and huge data aggregations and joins; so it is a nice bet for serious Kaggle participants. (I personally use Redshift as data science architecture for kaggle.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Combinations from some other vendor?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No, a combination of Redshift and S3 is enough. The data is stored in S3, and then loaded into Redshift using the COPY command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any way to interact with a GUI rather than a command line to set this up?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there is a CLI interface for Amazon AWS. However, you can use the &lt;a href=&quot;http://boto.readthedocs.org/&quot; rel=&quot;nofollow&quot;&gt;boto&lt;/a&gt; library on Python for handling this stuff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Would I have to install Anaconda or some other Python distribution before getting started?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not a compulsion.  Anaconda is a wrapper of numerical and scientific libraries. You can install them on your own or intall Anaconda.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;https://spark.apache.org/docs/0.9.0/python-programming-guide.html&quot; rel=&quot;nofollow&quot;&gt;PySpark&lt;/a&gt;, which allows you to do handle BigData in Python.&lt;/p&gt;&#xA;" OwnerUserId="11097" OwnerDisplayName="Dawny33" LastActivityDate="2015-09-13T19:51:12.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8094" PostTypeId="1" CreationDate="2015-09-14T22:05:19.697" Score="0" ViewCount="6139" Body="&lt;p&gt;I have &lt;code&gt;correlation&lt;/code&gt; values for &lt;code&gt;profit&lt;/code&gt; based on three different attributes - &lt;code&gt;attribute1&lt;/code&gt;,&lt;code&gt;attribute2&lt;/code&gt;,&lt;code&gt;attribute3&lt;/code&gt;. Now I want to make &lt;code&gt;predictions&lt;/code&gt; for the &lt;code&gt;profit&lt;/code&gt; based on these 3 different &lt;code&gt;correlation&lt;/code&gt; values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted suggestions for some efficient algorithms that can help me to make predictions  from these &lt;code&gt;correlation&lt;/code&gt; values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I am new to this so I would appreciate some suggestions if there is some lack of understanding in me.   &lt;/p&gt;&#xA;" OwnerUserId="11074" LastActivityDate="2015-09-23T09:51:17.850" Title="How to make predictions based on correlations?" Tags="|algorithms|predictive-modeling|regression|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8095" PostTypeId="2" ParentId="8087" CreationDate="2015-09-14T23:16:13.443" Score="0" Body="&lt;p&gt;I've read your Post a few Times now. But I'm not absolutely sure if I understood your Experiments fully. But I have a guess what might be happening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the error is hidden in the part where you achieve this astonishing 0.06 percent. You're not only generalizing to new data but you predict the future in a way. This should not work, but I think it does because your net is horribly overfitted. So to say your network is still very dumb. It just learned if you show it pictures a, b it will answer c. The second Experiment contains a twist, if I understood you correctly. I assume you still present complete images to the network, but they are built out of the coefficients of their eigenimages. Now it has to spit out coefficients to build the answer from eigenimages. But this would require some understanding of the process, which the network didn't aquire.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I think there are a number of suggestions to improve your experiment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning of complete inputimages is just an extremely high dimensional problem which requires huge ammount of training data. Consider calculating features from the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if your learning problem is a sound one. Please consider to separate your training data from your prediction data and take more conventional data like handwritten digits.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2015-09-14T23:16:13.443" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8096" PostTypeId="2" ParentId="8094" CreationDate="2015-09-14T23:28:47.893" Score="1" Body="&lt;p&gt;The correlation coefficient will tell you, how much predictive power the individual attributes posess with respect to profit in a linear sense.&#xA;Easier: The correlation coefficient tells you how line-like your data is.&#xA;( There might still be a very predictive arc like structure in your data which the correlation coefficient can't capture )   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can either calculate a regression line for profit with the attribute having the highest correlation coefficient, or you can calculate a hyperplane for all three attributes to get the desired prediction. Regression lines and hyperplanes are usually calculated by least-squares fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But beware to overestimate predictive power, because it is extrapolation what you do. New data might behave different than old data which makes extrapolation potentially less exact the further you are away from known data.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2015-09-14T23:28:47.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8097" PostTypeId="2" ParentId="8062" CreationDate="2015-09-14T23:37:06.307" Score="0" Body="&lt;p&gt;If I remember it correctly, classical theory says it is ideal, when the data are gaussians with an identity-covariance matrix. Then it behaves as the Bayesian( Gaussian ) Classifier which is optimal with respect to the 0 - 1 Loss-function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which still doesn't say anything about the behaviour with a real dataset without infinite data like suggested before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The practical answer could be, to try it as a first guess. When it doesn't work apply PCA to your data ( to decorrelate your features ) and try it again. If it still doesn't work, try other stuff. &lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2015-09-14T23:37:06.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8098" PostTypeId="2" ParentId="8018" CreationDate="2015-09-15T00:01:23.723" Score="1" Body="&lt;p&gt;Looks like you could just look for a few seconds of higher then noise derivative.&#xA;Just calculate the absolute value of the finite difference from each timestep to the last ( or one of the former ) and wait for a series of high values. That's when a lane change occurs. &lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2015-09-15T00:01:23.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8099" PostTypeId="1" AcceptedAnswerId="8100" CreationDate="2015-09-15T03:15:28.163" Score="6" ViewCount="1208" Body="&lt;p&gt;I have a big data set of &lt;em&gt;fake&lt;/em&gt; transactions for a company. Each row contains the username, credit card number, time, device used, and amount of money in the transaction. I need to classify each transaction as either malicious or not malicious and I am lost for ideas on where to start. Doing it by hand would be silly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking possibly checking for how often a credit card is used, if it is consistently used at a certain time, or if it is used from lots of different devices (iOS AND Android, as an example) would be possible starting places. I'm still fairly new to all this and ML. Would there be some ML algorithm optimal for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, side question: what would be a good place to host the 600 or so GB of data for cheaps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="12801" LastActivityDate="2022-05-02T17:04:52.833" Title="Classifying transactions as malicious" Tags="|classification|bigdata|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8100" PostTypeId="2" ParentId="8099" CreationDate="2015-09-15T04:14:45.927" Score="10" Body="&lt;p&gt;This problem is popularly called the &amp;quot;&lt;strong&gt;&lt;a href=&quot;https://www.azleg.gov/ars/13/02105.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Credit Card Fraud Detection&lt;/a&gt;&lt;/strong&gt;&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;There are several classification algorithms, which aim to tackle this problem.&lt;/p&gt;&#xA;&lt;p&gt;With the knowledge of the dataset you possess, the Decision Trees algorithm can be employed for detecting malicious transactions from the non-malicious ones. This &lt;a href=&quot;http://www.iaeng.org/publication/IMECS2011/IMECS2011_pp442-447.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; is a nice resource to learn and develop the intuition about fraud detection and the usage of basic classification algorithms like the Decision Trees and the SVMs for solving the problem.&lt;/p&gt;&#xA;&lt;p&gt;There are &lt;a href=&quot;https://scholar.google.co.in/scholar?hl=en&amp;amp;q=Fraud%20Detection&amp;amp;btnG=&quot; rel=&quot;nofollow noreferrer&quot;&gt;several other papers&lt;/a&gt; which solve this problems employing algorithms like &lt;a href=&quot;https://ieeexplore.ieee.org/document/4280163&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks&lt;/a&gt;, Logistic Regression, Genetic Algorithms, etc. However, the paper which uses the decision trees algorithm is a nice place to start learning.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;what would be a good place to host the 600 or so GB of data for cheaps?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://what%20would%20be%20a%20good%20place%20to%20host%20the%20600%20or%20so%20GB%20of%20data%20for%20cheaps?&quot; rel=&quot;nofollow noreferrer&quot;&gt;Aws S3&lt;/a&gt; would be a nice, cheap way to do that. It also integrates nicely with Redshift, in case you want to do complex analytics on the data.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="25180" LastEditDate="2022-05-02T17:04:52.833" LastActivityDate="2022-05-02T17:04:52.833" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8101" PostTypeId="2" ParentId="8018" CreationDate="2015-09-15T09:27:35.667" Score="1" Body="&lt;p&gt;Try the &lt;a href=&quot;https://cran.r-project.org/web/packages/changepoint/changepoint.pdf&quot; rel=&quot;nofollow&quot;&gt;changepoint package&lt;/a&gt;. I used it in a similar case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Changepoint analysis is the statistical name for methods that detect changes between two &quot;regimes&quot;. A car staying in a lane is a line with gradient 0 at the midpoint of a lane. You can easily fit a statistical model to cars driving in lanes. A car changing lane is driving along a line with a gradient that isn't 0. The model has changed. Changepoint analysis, and the changepoint package, is exactly what you need to determine the point when a model changes from &lt;code&gt;y=a' (straight and level) to&lt;/code&gt;y=a+bx` (going up or down).&lt;/p&gt;&#xA;" OwnerUserId="12807" LastEditorUserId="471" LastEditDate="2015-10-26T19:41:37.260" LastActivityDate="2015-10-26T19:41:37.260" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8102" PostTypeId="2" ParentId="5024" CreationDate="2015-09-15T15:34:15.223" Score="3" Body="&lt;p&gt;Adding to Stephan's answer (I am unable to comment), there is actually an Optimization task view on CRAN that is even more relevant:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cran.r-project.org/web/views/Optimization.html&quot; rel=&quot;nofollow&quot;&gt;CRAN Task View: Optimization and Mathematical Programming&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12812" LastActivityDate="2015-09-15T15:34:15.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8105" PostTypeId="1" AcceptedAnswerId="8109" CreationDate="2015-09-16T13:54:07.770" Score="0" ViewCount="292" Body="&lt;p&gt;I am setting up a work computer and have free reign here. My typical go-to software packages are all freely available, such as Rstudio and Anaconda. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought about investing in commercial BI software such as Tableau or Spotfire, but nobody else in the group (or myself) is a proficient user. Are there other obvious choices I am missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the outset, much of what this group is working on will be exploratory. They are needing some justification to invest further into &quot;data science&quot;.&lt;/p&gt;&#xA;" OwnerUserId="12827" LastActivityDate="2015-09-17T07:29:40.237" Title="What commercial software should a data scientist purchase?" Tags="|bigdata|software-recommendation|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="8107" PostTypeId="2" ParentId="8105" CreationDate="2015-09-16T22:47:26.753" Score="1" Body="&lt;p&gt;I would echo @David. The question is what do you expect the tool to serve. May be SAS or SPSS is the way to go or even MATLAB. Each tool brings something to the table. Depending on what you need, one or more may fit the bill. Also look at your org's tech road map. If Open Source is a big push, your list may be different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tableau is a good tool (and easy to learn) for Business Users or people who are less technical. Even though it is costly ($2000 for a single user annual license), it has a good reader (free) and once a report is generated, the users can leverage the reader and update without dependence on the license holder. The benefit will lie as long as the reports are stable and standard in nature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;R has great graphic capabilities and hence users who do get involved in analytic may find it better suited. Similarly python can be a great tool especially if you need a web based solution for your users. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So think about your needs and strategy.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2015-09-16T22:47:26.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8108" PostTypeId="2" ParentId="8105" CreationDate="2015-09-17T00:58:38.983" Score="2" Body="&lt;p&gt;If you are looking for a technical package I would definitely go with &lt;a href=&quot;http://www.wolfram.com/mathematica/&quot; rel=&quot;nofollow&quot;&gt;Mathematica&lt;/a&gt;. It has functions for a very wide &lt;a href=&quot;http://reference.wolfram.com/language/&quot; rel=&quot;nofollow&quot;&gt;range of disciplines&lt;/a&gt; that come standard with it (no extra packages to buy). This allows you to expand into techniques as you grow without having to change platforms or buy extra packages.  It also has a great interactive format in &lt;a href=&quot;http://www.wolfram.com/cdf/&quot; rel=&quot;nofollow&quot;&gt;CDF&lt;/a&gt; with both a free and professional reader. The documentation is interactive as well which really helps a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you have to be open to a little bit of structure, typing &amp;amp; chaining functions together, and things like this.  Once you get started it comes very quickly.&lt;/p&gt;&#xA;" OwnerUserId="10814" LastActivityDate="2015-09-17T00:58:38.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8109" PostTypeId="2" ParentId="8105" CreationDate="2015-09-17T07:29:40.237" Score="1" Body="&lt;p&gt;If you are a data scientist, then there is very little use for pre-packaged, generally inflexible commercial tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part of the reason OSS is so prevalent and useful in data science is that you will often need to combine and/or modify a procedure to fit the needs at hand -- and then &lt;em&gt;deploy&lt;/em&gt; it without a bunch of lawyers and sales reps getting involved at every step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since data scientists are expected to be proficient programmers, you should be comfortable digging into the source code and adding functionality or making it more user friendly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've come close to recommending the purchase of non-free(as in GPL) a couple times only to find that some industrious person has set up a project in Git that provides most if not all of the functionality if commercial software. In the cases where it doesn't, it at least addresses the core issue and I can modify and extend from it. It's much easier to modify a prototype than start from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottom line: be wary of commercial software for data science unless you've done your due diligence in the OSS space and can honestly say that you could not find any projects that could be modified to suit your needs. &#xA;Commercial software is not only less flexible but you're effectively in a business partnership with these folks and that means your fates are somewhat intertwined (at least for the projects that depend in this software).&lt;/p&gt;&#xA;" OwnerDisplayName="user9424" LastActivityDate="2015-09-17T07:29:40.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8111" PostTypeId="2" ParentId="6776" CreationDate="2015-09-17T11:21:43.797" Score="0" Body="&lt;p&gt;Naritivly context aware, visual profanity filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other worlds, rendering physically realistic and thematically/stylistically appropriate clothes on people who are insufficiently dressed, to make the image more family safe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thats the idea, however at the moment it's both unreliable and when it does work inaccurate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, more tweaking of the parameters of the dream than I have access to, or possibly just using more iterations and a lower &quot;octave&quot; value than I can specify should make the results much more reliable.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before:&#xA;&lt;a href=&quot;http://s22.postimg.org/5sjpqbzoh/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s22.postimg.org/5sjpqbzoh/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After:&#xA;&lt;a href=&quot;http://s22.postimg.org/wew6fb3vl/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s22.postimg.org/wew6fb3vl/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://s13.postimg.org/c4urz139j/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s13.postimg.org/c4urz139j/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12844" LastEditorUserId="12844" LastEditDate="2015-09-17T16:56:59.210" LastActivityDate="2015-09-17T16:56:59.210" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8113" PostTypeId="1" AcceptedAnswerId="8121" CreationDate="2015-09-17T14:13:54.293" Score="0" ViewCount="5743" Body="&lt;p&gt;I am trying to learn &lt;code&gt;scikit-learn&lt;/code&gt; &lt;code&gt;neuralnetwork&lt;/code&gt; and am coming up against the same problem in regression where no matter the dataset I getting a horizontal straight line for my fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is an example using the Linear regression example from &lt;code&gt;scikit-learn&lt;/code&gt; and then using the &lt;code&gt;SKNN&lt;/code&gt; regressor , simple example code from the docs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;# -*- coding: utf-8 -*-&#xA;&#xA;# Code source: Jaques Grobler&#xA;# http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html&#xA;# License: BSD 3 clause&#xA;&#xA;&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn import datasets, linear_model&#xA;&#xA;# Load the diabetes dataset&#xA;diabetes = datasets.load_diabetes()&#xA;&#xA;&#xA;# Use only one feature&#xA;diabetes_X = diabetes.data[:, np.newaxis]&#xA;diabetes_X_temp = diabetes_X[:, :, 2]&#xA;&#xA;# Split the data into training/testing sets&#xA;diabetes_X_train = diabetes_X_temp[:-20]&#xA;diabetes_X_test = diabetes_X_temp[-20:]&#xA;&#xA;# Split the targets into training/testing sets&#xA;diabetes_y_train = diabetes.target[:-20]&#xA;diabetes_y_test = diabetes.target[-20:]&#xA;&#xA;# Create linear regression object&#xA;regr = linear_model.LinearRegression()&#xA;&#xA;# Train the model using the training sets&#xA;regr.fit(diabetes_X_train, diabetes_y_train)&#xA;&#xA;print &quot;Results of Linear Regression....&quot;&#xA;print &quot;================================\n&quot;&#xA;# The coefficients&#xA;print('Coefficients: ', regr.coef_)&#xA;# The mean square error&#xA;print(&quot;Residual sum of squares: %.2f&quot;&#xA;      % np.mean((regr.predict(diabetes_X_test) - diabetes_y_test) ** 2))&#xA;# Explained variance score: 1 is perfect prediction&#xA;print('Variance score: %.2f' % regr.score(diabetes_X_test, diabetes_y_test))&#xA;&#xA;# Plot outputs&#xA;plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')&#xA;plt.plot(diabetes_X_test, regr.predict(diabetes_X_test), color='blue',&#xA;         linewidth=3)&#xA;&#xA;plt.xticks(())&#xA;plt.yticks(())&#xA;&#xA;plt.show()&#xA;&#xA;# Now using the sknn regressor&#xA;# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html&#xA;#&#xA;&#xA;from sknn.mlp import Regressor, Layer&#xA;&#xA;nn = Regressor(&#xA;    layers=[&#xA;        Layer(&quot;Rectifier&quot;, units=200),&#xA;        Layer(&quot;Linear&quot;)],&#xA;    learning_rate=0.02,&#xA;    n_iter=10)&#xA;&#xA;nn.fit(diabetes_X_train, diabetes_y_train)&#xA;print &quot;Results of SKNN Regression....&quot;&#xA;print &quot;==============================\n&quot;&#xA;&#xA;&#xA;# The coefficients&#xA;print('Coefficients: ', regr.coef_)&#xA;# The mean square error&#xA;print(&quot;Residual sum of squares: %.2f&quot;&#xA;      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))&#xA;# Explained variance score: 1 is perfect prediction&#xA;print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))&#xA;&#xA;# Plot outputs&#xA;plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')&#xA;plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',&#xA;         linewidth=3)&#xA;&#xA;plt.xticks(())&#xA;plt.yticks(())&#xA;&#xA;plt.show()&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results of Linear Regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('Coefficients: ', array([ 938.23786125]))&#xA;Residual sum of squares: 2548.07&#xA;Variance score: 0.47&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;￼&#xA;Results of SKNN Regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('Coefficients: ', array([ 938.23786125]))&#xA;Residual sum of squares: 5737.52&#xA;Variance score: -0.19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Changing the number of iterations to 1000 results in a score of -0.15&lt;/p&gt;&#xA;" OwnerUserId="12294" LastEditorUserId="104237" LastEditDate="2020-09-14T13:08:57.510" LastActivityDate="2020-09-14T13:08:57.510" Title="SKNN regression problem" Tags="|python|neural-network|scikit-learn|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8114" PostTypeId="1" AcceptedAnswerId="8115" CreationDate="2015-09-17T18:11:38.857" Score="0" ViewCount="221" Body="&lt;p&gt;I'm attempting to classify text documents using a few different dimensions. I'm trying to create arbitrary topics to classify such as size and relevance, which are linear or gradual in nature. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;size: tiny, small, medium, large, huge. &#xA;relevance: bad, ok, good, excellent, awesome&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training the classifier by hand. For example, this document represents a 'small' thing, this other document is discussing a 'large' thing. When I try multi-label or multi-class SVM for this it does not work well and it also logically doesn't make sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which model should I use that would help me predict this linear type of data? I use scikit-learn presently with a tfidf vector of the words.&lt;/p&gt;&#xA;" OwnerUserId="9373" LastActivityDate="2015-09-17T18:41:18.710" Title="Classifying text documents using linear/incremental topics" Tags="|classification|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8115" PostTypeId="2" ParentId="8114" CreationDate="2015-09-17T18:41:18.710" Score="1" Body="&lt;p&gt;If you want these output dimensions to be continuous, simply convert your size and relevance metrics to real-valued targets.  Then you can perform &lt;a href=&quot;https://en.wikipedia.org/wiki/Regression_analysis&quot; rel=&quot;nofollow&quot;&gt;regression&lt;/a&gt; instead of classification, using any of a variety of models.  You could even attempt to train a multi target neural net to predict all of these outputs at once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you might consider first using  a &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic model&lt;/a&gt; such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; as your feature space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the values, it sounds like the &quot;relevance&quot; might be a variable best captured by techniques from &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;nofollow&quot;&gt;sentiment analysis&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-09-17T18:41:18.710" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8117" PostTypeId="1" AcceptedAnswerId="9927" CreationDate="2015-09-17T21:02:33.367" Score="8" ViewCount="1581" Body="&lt;p&gt;If word2vec encounters the same word multiple times in the same window, what occurs?  Obviously it is meaningless to decrease the distance between the vectors for the input word and the target word.  But will the repetition strengthen the relationship between the repeated word and the context words?&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="843" LastEditDate="2015-09-21T08:12:57.207" LastActivityDate="2016-10-20T12:59:46.543" Title="How does word2vec handle the input word being in the context?" Tags="|machine-learning|nlp|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8118" PostTypeId="2" ParentId="6776" CreationDate="2015-09-17T21:40:51.263" Score="1" Body="&lt;p&gt;Greyscale to Color&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://s15.postimg.org/3xq8jx03f/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s15.postimg.org/3xq8jx03f/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://s15.postimg.org/i5fx8kcsb/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s15.postimg.org/i5fx8kcsb/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://s15.postimg.org/c5s64wrzv/image.jpg&quot; rel=&quot;nofollow&quot;&gt;http://s15.postimg.org/c5s64wrzv/image.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tree wood seems unnaturally red but still, it's not bad. This has worked but less impressively with other greyscale images I've tried.&lt;/p&gt;&#xA;" OwnerUserId="12844" LastEditorUserId="12844" LastEditDate="2015-09-17T21:48:54.657" LastActivityDate="2015-09-17T21:48:54.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8119" PostTypeId="2" ParentId="6776" CreationDate="2015-09-17T22:12:50.787" Score="2" Body="&lt;p&gt;heres another application that is very new &amp;amp; just demonstrated within last few weeks. computers are filtering images to look like paintings in the distinctive style of different artists eg Van Gogh, Picasso, etc... and it seems possible since the technology can encompass different artistic styles it might be used for &lt;a href=&quot;https://en.wikipedia.org/wiki/Art_forgery&quot; rel=&quot;nofollow&quot;&gt;forgery detection&lt;/a&gt; in the art world at some point. (many very advanced analysis techniques are used in this area historically.) note that filtering methods are very popular on Instagram so it seems likely these will be commercially available at some point.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://qz.com/495614/computers-can-now-paint-like-van-gogh-and-picasso/&quot; rel=&quot;nofollow&quot;&gt;Computers can now paint like Van Gogh and Picasso&lt;/a&gt; / Quartz&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.dailymail.co.uk/sciencetech/article-3091562/The-science-Instagram-success-Filtered-photos-21-likely-viewed-warmer-tones-comments.html&quot; rel=&quot;nofollow&quot;&gt;The science of Instagram success: Filtered photos are 21% more likely to be viewed - and warmer tones get more comments&lt;/a&gt; / Dailymail&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="292" LastActivityDate="2015-09-17T22:12:50.787" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8120" PostTypeId="1" AcceptedAnswerId="9338" CreationDate="2015-09-17T23:06:50.597" Score="4" ViewCount="5324" Body="&lt;p&gt;I am using the xgboost library. My system runs a cronjob each night, where it pulls the data from the database and trains the model. However, I would like to remove the re-training of the model again and again, and just fine-tune it with any new data that came in the database. In sklearn's implemantation (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&lt;/a&gt;) one could use warm_start option, what about xgboost ? &lt;/p&gt;&#xA;" OwnerUserId="12858" LastEditorUserId="303" LastEditDate="2020-03-14T08:49:25.577" LastActivityDate="2021-05-13T06:17:43.140" Title="Can I fine tune the xgboost model instead of re-training it?" Tags="|machine-learning|scikit-learn|xgboost|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8121" PostTypeId="2" ParentId="8113" CreationDate="2015-09-18T04:06:50.197" Score="1" Body="&lt;p&gt;My best guess here is that your learning rate is &lt;em&gt;way&lt;/em&gt; too high for the problem. You also probably have far more neurons in your hidden network than you need, seeing as you're using just one feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall that learning rate is controlling the &quot;step size&quot; in gradient descent and that for your dataset, it is likely far too high. I made some minor changes to your code and got better results than linear regression. Notice the use of 2 hidden neurons, a 0.001 learning rate, and 20 iterations.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Now using the sknn regressor&#xA;# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html&#xA;&#xA;&#xA;from sknn.mlp import Regressor, Layer&#xA;&#xA;nn = Regressor(&#xA;    layers=[&#xA;        Layer(&quot;Rectifier&quot;, units=2),&#xA;        Layer(&quot;Linear&quot;)],&#xA;    learning_rate=0.001,&#xA;    n_iter=20)&#xA;&#xA;nn.fit(diabetes_X_train, diabetes_y_train)&#xA;print(&quot;Results of SKNN Regression....&quot;)&#xA;&#xA;&#xA;# The coefficients&#xA;print('Coefficients: ', regr.coef_)&#xA;# The mean square error&#xA;print(&quot;Residual sum of squares: %.2f&quot;&#xA;      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))&#xA;# Explained variance score: 1 is perfect prediction&#xA;print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))&#xA;&#xA;# Plot outputs&#xA;plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')&#xA;plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',&#xA;         linewidth=3)&#xA;&#xA;plt.xticks(())&#xA;plt.yticks(())&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;SKNN regression: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Results of SKNN Regression....&#xA;Coefficients:  [ 938.23786125]&#xA;Residual sum of squares: 6123.67&#xA;Variance score: 0.50&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11098" LastActivityDate="2015-09-18T04:06:50.197" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8122" PostTypeId="1" CreationDate="2015-09-18T05:32:26.250" Score="1" ViewCount="9642" Body="&lt;p&gt;This data when I plot - &lt;code&gt;60&lt;/code&gt; &lt;code&gt;85&lt;/code&gt; &lt;code&gt;72&lt;/code&gt; &lt;code&gt;59&lt;/code&gt; &lt;code&gt;37&lt;/code&gt; &lt;code&gt;75&lt;/code&gt; &lt;code&gt;93&lt;/code&gt; &lt;code&gt;7&lt;/code&gt; &lt;code&gt;98&lt;/code&gt; &lt;code&gt;63&lt;/code&gt; &lt;code&gt;41&lt;/code&gt; &lt;code&gt;90&lt;/code&gt; &lt;code&gt;5&lt;/code&gt; &lt;code&gt;17&lt;/code&gt; &lt;code&gt;97&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stem(data)&#xA;&#xA; 0 | 577&#xA; 2 | 7&#xA; 4 | 19&#xA; 6 | 0325&#xA; 8 | 50378&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now If I reconstruct data it is &lt;code&gt;5&lt;/code&gt;, &lt;code&gt;7&lt;/code&gt;, &lt;code&gt;7&lt;/code&gt;, &lt;code&gt;27&lt;/code&gt;, &lt;code&gt;60&lt;/code&gt;, &lt;code&gt;63&lt;/code&gt;, &lt;code&gt;62&lt;/code&gt;, &lt;code&gt;65&lt;/code&gt;, &lt;code&gt;85&lt;/code&gt;, &lt;code&gt;80&lt;/code&gt;, &lt;code&gt;83&lt;/code&gt;, &lt;code&gt;87&lt;/code&gt;, &lt;code&gt;88&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does not produce the input in reverse.&lt;/p&gt;&#xA;" OwnerUserId="12756" LastEditorUserId="710" LastEditDate="2015-09-20T10:38:35.267" LastActivityDate="2015-09-20T10:38:35.267" Title="I have few basic questions on R Stem and Leaf Plot" Tags="|r|" AnswerCount="2" CommentCount="0" ClosedDate="2015-09-18T20:28:03.713" ContentLicense="CC BY-SA 3.0" />
  <row Id="8123" PostTypeId="2" ParentId="8122" CreationDate="2015-09-18T08:32:19.853" Score="-1" Body="&lt;p&gt;try this&#xA;S=c(60, 85, 72, 59, 37, 75, 93, 7, 98, 63, 41, 90 ,5 ,17 ,97)&#xA;;stem(S,scale = 2)&#xA;A value of scale = 2 will cause the plot to be roughly twice as long as the default.&lt;/p&gt;&#xA;" OwnerUserId="12779" LastEditorUserId="12779" LastEditDate="2015-09-18T08:37:46.600" LastActivityDate="2015-09-18T08:37:46.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8124" PostTypeId="1" AcceptedAnswerId="8127" CreationDate="2015-09-18T09:50:09.740" Score="-2" ViewCount="211" Body="&lt;p&gt;I have string representations of text written by users in the form of parts of speech tags like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$NNDN,OVDANPN,PNVRV,^^V,^^!$^OV&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ^,G,#,!,N,R,$&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc. They are separated into two classes (0 or 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to cluster these such that I will be able to predict (or try to) what class the user in from their tags using the damerau levenshtein distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that even a few hundred strings is a huge calculation for any basic clustering that I am aware of (but I am very new to this). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using the counts of each tag to form a vector but applying SVM, knn classifier and Naive Bayes yielded poor results, even when using a KS test to get the best features. My gut feeling is that this seems like a problem that could be solved in the same way that scientists would compare and cluster genes.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should I be looking at different machine learning methods?  &lt;/li&gt;&#xA;&lt;li&gt;Is there another way of representing the strings that would be more appropriate?&lt;/li&gt;&#xA;&lt;li&gt;Is there another way of looking at the problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm using the scikit-learn library for Python.&lt;/p&gt;&#xA;" OwnerUserId="9025" LastActivityDate="2015-09-18T10:09:53.847" Title="Clustering large number of strings based on tags" Tags="|machine-learning|classification|python|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8125" PostTypeId="2" ParentId="8122" CreationDate="2015-09-18T10:08:01.810" Score="2" Body="&lt;p&gt;With the default scale, you see that numbers left of the bar go up by two - hence anything after the 4 is forty-something or fifty-something:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; stem(d,scale=1)&#xA;&#xA;  The decimal point is 1 digit(s) to the right of the |&#xA;&#xA;  0 | 577&#xA;  2 | 7&#xA;  4 | 19&#xA;  6 | 0325&#xA;  8 | 50378&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;scale=2&lt;/code&gt;, you'll see numbers left of the bar go up by one, so now you can get exact reconstruction of your input, since you input integers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; stem(d,scale=2)&#xA;&#xA;  The decimal point is 1 digit(s) to the right of the |&#xA;&#xA;  0 | 57&#xA;  1 | 7&#xA;  2 | &#xA;  3 | 7&#xA;  4 | 1&#xA;  5 | 9&#xA;  6 | 03&#xA;  7 | 25&#xA;  8 | 5&#xA;  9 | 0378&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Going further you can even split it by first and second five within each decade:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; stem(d,scale=4)&#xA;&#xA;  The decimal point is 1 digit(s) to the right of the |&#xA;&#xA;  0 | 57&#xA;  1 | &#xA;  1 | 7&#xA;  2 | &#xA;  2 | &#xA;  3 | &#xA;  3 | 7&#xA;  4 | 1&#xA;  4 | &#xA;  5 | &#xA;  5 | 9&#xA;  6 | 03&#xA;  6 | &#xA;  7 | 2&#xA;  7 | 5&#xA;  8 | &#xA;  8 | 5&#xA;  9 | 03&#xA;  9 | 78&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Stem plots &lt;em&gt;do not&lt;/em&gt; always guarantee you can reproduce the data by reversing the process, and that's not what they're for.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2015-09-18T10:08:01.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8127" PostTypeId="2" ParentId="8124" CreationDate="2015-09-18T10:09:53.847" Score="0" Body="&lt;p&gt;Using the Levenshtein distance does not make a lot of sense in this context, as it is made for comparing distances between words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A commonly used representation for texts is the bag-of-words representation, where a text is converted to a vector where every element in the vector represents the count of the corresponding word. In your case you could represent a text as a bag-of-tags.&#xA;The vector representation makes calculating distances a lot easier. However, I believe this is not necessary as you can classify the bags of words with Naive Bayes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have tried bag-of-words you can try more complicated representations like LDA, word2vec, and the like.&lt;/p&gt;&#xA;" OwnerUserId="5059" LastActivityDate="2015-09-18T10:09:53.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8128" PostTypeId="1" CreationDate="2015-09-18T16:16:42.860" Score="1" ViewCount="162" Body="&lt;p&gt;I have four different categories which the number of variables in each category is 7, 3, 8, and 9 (in total 27). The value of each variable changes from 1 to 5 but some of the variables were not applicable for some observations. In total, there are 20 observations which their data is available for 27 variables in the mentioned context (from 1 to 5 plus not applicable). How can I cluster these four categories for 20 observations to get three clusters for each category?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;edit&lt;/em&gt;: There are 4 categories (A, B, C, and D). Each category has a different vector including 7, 3, 8, and 9 values (from 1 to 5 plus NA). Now, it's 18 observations for each category of A, B, C, and D. I need to make three groups for each category. How can I find the centroid of each cluster for each category? It means I need three centroid in each category, totally 12(=4 category* 3 groups). Any feedback would be appreciated. Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/MAdCW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="12867" LastEditDate="2015-09-19T04:07:37.910" LastActivityDate="2015-09-21T14:22:17.430" Title="Clustering the data if some values in each vector is not applicable" Tags="|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8129" PostTypeId="1" CreationDate="2015-09-18T17:44:42.573" Score="3" ViewCount="1363" Body="&lt;p&gt;Is it possible take a training set of one million 120 x 120 pixel tiles, feed these tiles into a machine learning algorithm. And then make it &lt;em&gt;synthesize&lt;/em&gt; images that look like the original training set with a certain, adjustable level of entropy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What ML techniques should I consider?&lt;/p&gt;&#xA;" OwnerUserId="12859" LastEditorUserId="12859" LastEditDate="2015-09-18T21:38:35.040" LastActivityDate="2017-07-20T00:39:50.633" Title="Generate new images based on a dataset of images" Tags="|machine-learning|statistics|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8131" PostTypeId="1" AcceptedAnswerId="8152" CreationDate="2015-09-19T04:02:11.133" Score="1" ViewCount="276" Body="&lt;p&gt;I have collected data for a PhD thesis, and need help understanding how to build a road map to do analytical and statistical analysis. The PhD is not itself in statistics or machine learning, but I would like to understand what are the steps and type of analysis that I have to follow for analysing data for an advanced degree? In general, how should I approach such a problem?&lt;/p&gt;&#xA;&lt;p&gt;In the data I have collected, there are 623 observations including one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal) that are defined based on the researcher experience and literature review.&lt;/p&gt;&#xA;&lt;p&gt;I considered planning to do several regression analysis to predict the dependent variable and study the effective factors (if they are positive, negative, and their magnitude) on it. I've tried multiple linear regression including different transformation on independent variables. On the other hand, I'm not sure if I should study each independent variables through the time and  forecast their values in the time horizon?&lt;/p&gt;&#xA;&lt;p&gt;Here are the steps in my mind so far:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Plotting the scatter plots of different independent variables vs dependent variable to define outliers and check if the model is linear also with respect to coefficients&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Removing the potential outliers&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Splitting the data into two data sets to build the model and validate it after that.&lt;/p&gt;&#xA;&lt;p&gt;If the model is linear then:&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Performing the multiple linear regression&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Performing the multiple linear regression including different transformations to enhance the model&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Validating the model&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Doing the quantile regression&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Doing supervised learning machine etc.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;If the model is not linear, I may instead need to use non-linear statistical techniques.&lt;/p&gt;&#xA;&lt;p&gt;Any feedback would be highly appreciated. My goal is to build a clear and robust road map for this part of the work.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-09-22T07:42:59.590" Title="How to start analysing and modelling data for an academic project, when not a statistician or data scientist" Tags="|dataset|predictive-modeling|data-cleaning|linear-regression|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="8134" PostTypeId="2" ParentId="8128" CreationDate="2015-09-19T12:45:02.360" Score="1" Body="&lt;p&gt;First, are your categories independent of each other? If this is really four parallel problems, solve one at a time and it'll simplify things a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, the heart of clustering is the distance function between any two observations. Because your data has missing values, you need to explicitly decide how to handle them. A common approach is to ignore the problem by throwing away any rows that have any null values, which won't work at all for your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple approach would be to ignore any NA columns, but keep the rest of the columns--for category C, the difference between rows 1 and 2 would be 4 (columns 11, 14, 15, and 16 are all off by one). There's an issue with this approach that I'll get to later, but let's implement it for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each category, we need three rows of centroid positions (we might as well start off with randomly selected rows that have no missing values in them). We then calculate distance to each centroid using a bunch of summed ifs: if(matrix cell != NA, matrix cell - centroid cell, 0) across all the columns. (If you're doing this computation in a spreadsheet, I'd use intermediate cells to calculate the pieces instead of dealing with one huge formula.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then optimize the centroids with a standard iterative procedure: each row gets a group membership corresponding to the closest centroid, then each centroid becomes the mean of its group members (and it looks like your mean function automatically excludes NAs), then we repeat until satisfied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why did I specify that we had to seed with a row without NAs? If you allow NA measurements in the centroids, this will fail because a row of all NAs is as central as it gets! If our update step never lets us move to NAs, the NA column will be transient, but what happens when we take the mean of a missing column? (In your sample data, consider what happens if rows 4-12 are a group in the B category.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we want to allow NAs in the centroids, we need to modify our distance function to something like the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If both cells are numbers, the distance is the difference&lt;/li&gt;&#xA;&lt;li&gt;If both cells are NA, the distance is 2&lt;/li&gt;&#xA;&lt;li&gt;If only one cell is NA, the distance is 3.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those numbers are chosen such that for any full column of numbers, a numerical value in the centroid position will be closer, but if the column has sufficiently many NA rows, a NA value in the centroid will be closer. You might want to adjust the values so that the NA threshold looks like what you want it to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to find the centroid of a set of rows using our distance function, we should use an optimization procedure, but we can cheat: if the number of NAs in the group is above the threshold, we set it to NA, and otherwise we set it to the mean.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastEditorUserId="12876" LastEditDate="2015-09-21T14:22:17.430" LastActivityDate="2015-09-21T14:22:17.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8135" PostTypeId="1" AcceptedAnswerId="8136" CreationDate="2015-09-19T15:40:54.523" Score="1" ViewCount="538" Body="&lt;p&gt;I am trying to organize a cheat sheet of sorts for data science, and I am working with the basic distinction between description, inference, and prediction. As examples of the first I see unsupervised methods described, and for the last I see supervised methods. So my question is simply, do these two sets of categories align? Is unsupervised to supervised as description is to prediction?&lt;/p&gt;&#xA;" OwnerUserId="12881" LastActivityDate="2017-12-18T01:12:38.893" Title="Is supervised machine learning by definition predictive?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8136" PostTypeId="2" ParentId="8135" CreationDate="2015-09-19T17:44:53.190" Score="3" Body="&lt;p&gt;A description as any statistic drawn from your sample data, say the sample mean, quantiles, etc.. Inference is a conclusion drawn from your sample data about the population, e.g., rejecting or accepting some hypothesis or stating that a model is suitable or not for describing your data. Prediction is simply a guess about future observations, which hopefully uses your data and some function/model of the data in a way to formulate that guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both unsupervised and supervised learning methods aim to learn a function of the data that predicts another variable (typically called y) so both are drawing an inference (i.e., a model is well suited to describe your data, see the first sentence &lt;a href=&quot;https://en.wikipedia.org/wiki/Supervised_learning&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;). However, these two methods differ in what data is available. In supervised learning you are able to use an observed sample of y for training your model and in unsupervised learning, y is unobserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps!&lt;/p&gt;&#xA;" OwnerUserId="12884" LastEditorUserId="12884" LastEditDate="2015-09-19T17:52:13.387" LastActivityDate="2015-09-19T17:52:13.387" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8138" PostTypeId="1" AcceptedAnswerId="8148" CreationDate="2015-09-20T12:40:10.170" Score="7" ViewCount="373" Body="&lt;p&gt;I have a data set with text fragments having a fixed structure that can contain parameters.&#xA;Examples are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Temperature today is 20 centigrades&#xA; Temperature today is 28 centigrades&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Her eyes are blue and hair black.&#xA; Her eyes are green and hair brown.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first example show a template with one numerical parameter. The second one is a template with two factor parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of templates and the number of parameter is not know. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is to identify the templates and assign each text fragment to the corresponding template.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious first idea is to use clustering. The distance measure is defined as a number of non matching words. I.e. the records in example one have distance 1, in example two distance is 2. The distance between the record in example one and two is 7.&#xA;This approach works fine, providing the number of clusters is know, which is not the case, so it is not useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine a programmatic approach scanning the distance matrix searching for records with lot of neighbors in distance 1 (or 2,3,..), but I'm curious if I can apply some unsupervised machine learning algorithm to solve the problem. R is preferred, but not required.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="843" LastEditDate="2015-09-21T06:57:59.920" LastActivityDate="2015-09-27T09:51:41.237" Title="Identifying templates with parameters in text fragments" Tags="|machine-learning|r|nlp|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8139" PostTypeId="2" ParentId="8069" CreationDate="2015-09-20T21:42:48.270" Score="1" Body="&lt;p&gt;Consider the class to be the variable that you are sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your &quot;reservoir sample&quot; should still be as good as uniformly drawn from your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there may be fluctuations, in particular if you have small samples. If you sample a single observation, the class distribution in that sample will be 100% of one class, there is no way around that. ;-)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-09-20T21:42:48.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8140" PostTypeId="1" CreationDate="2015-09-21T01:31:19.573" Score="0" ViewCount="1111" Body="&lt;p&gt;I have a very unique problem that I would like to solve using machine learning. I have a set of 90 or so unique options. Each of these options has a unique set of features (5 to be specific) that vary over time and that can be represented mathematically as either scalars or booleans. I want to train a model to tell me what the best option at that moment is given the complete set of inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first, I figured a simple feedforward neural network would do the trick but I don't have a ton of experience with ML models so I ran into the following issue. I know my problem isn't really a classification problem since I am trying to pick the best option. I figured it could be easily modelled as a regression problem (as some mathematical combination of the inputs) which is fine but then I don't know what the expected outcome would be. For example, if the expected output is the weighted average of the inputs, I don't know what the final value should be. All I know is that option B should be suggested over option A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started reading about random forests and they seem like they might help my situation but I wanted to get other opinions first.&lt;/p&gt;&#xA;" OwnerUserId="12898" LastActivityDate="2015-11-02T13:38:24.937" Title="Machine Learning algorithm to predict an outcome where output is not known" Tags="|machine-learning|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8141" PostTypeId="2" ParentId="8140" CreationDate="2015-09-21T03:51:01.000" Score="1" Body="&lt;p&gt;What I was looking for is called ordinal regression or ranking learning (&lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_regression&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Ordinal_regression&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I will elaborate more on why this solved my issue&#xA;What I was having a problem with was finding a statistical model to appropriately predict the ordinality of multiple independent options. What I cared about was the order of the options and less so about what the value of the options was.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordered Logit/Probit describe such models. The provided wikipedia link explains them in far more detail than I can.&lt;/p&gt;&#xA;" OwnerUserId="12898" LastEditorUserId="12898" LastEditDate="2015-09-21T18:18:45.150" LastActivityDate="2015-09-21T18:18:45.150" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8142" PostTypeId="1" AcceptedAnswerId="8144" CreationDate="2015-09-21T11:15:45.813" Score="1" ViewCount="1065" Body="&lt;p&gt;I am working on a text classification work. The purpose of this work is to classify whether a particular document belong to class A or Class B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used KNN algorithm and i am able to get some decent results. However I want to know two things.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why a particular document has been classified as Class A or Class B? What keyword or information that made a document to be classified as such?&lt;/li&gt;&#xA;&lt;li&gt;How to perform mis-classification analysis?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Kindly help.&lt;/p&gt;&#xA;" OwnerUserId="9793" LastActivityDate="2015-09-21T15:21:13.133" Title="Error Analysis for misclassification of text documents" Tags="|data-mining|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8143" PostTypeId="2" ParentId="8138" CreationDate="2015-09-21T14:32:37.967" Score="3" Body="&lt;p&gt;You might consider using &lt;a href=&quot;https://code.google.com/p/word2vec/&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt; to identify &lt;a href=&quot;https://radimrehurek.com/gensim/models/phrases.html&quot; rel=&quot;nofollow&quot;&gt;phrases&lt;/a&gt; in the corpus.  The presence of a phrase (instead of single tokens) is likely to indicate a 'template.'  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here, the tokens most similar to your template phrase are likely to be the values for your parameters.   &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-09-21T14:32:37.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8144" PostTypeId="2" ParentId="8142" CreationDate="2015-09-21T15:21:13.133" Score="1" Body="&lt;p&gt;It seems to me that both of your questions could be answered by storing the retrieved neighbours on your test set and giving them a thorough analysis. Assuming you are using a unigram + tf-idf text representation and a cosine similarity distance metric for your K-NN retrieval, it would be trivial once you have a classified document to display the K neighbours and analyze their common unigrams and their respective tf-idf weights in order to see what influenced the classification. Moreover, doing it on your misclassified documents could help you understand which features caused the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be interested to know if there is a more systematized approach to those issues.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-21T15:21:13.133" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8145" PostTypeId="2" ParentId="8135" CreationDate="2015-09-21T15:33:29.793" Score="2" Body="&lt;p&gt;Peter Flach has a nice small matrix that describes what you are looking for in his book, &lt;a href=&quot;https://www.cs.bris.ac.uk/~flach/mlbook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning, the art and science of algorithms that make sense of data&lt;/a&gt;. I'll reproduce it below::&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;                      |     Predictive Model          |   Descriptive Model&#xA;------------------------------------------------------------------------------&#xA;Supervised learning   | Classification, Regression    |  Subgroup discovery&#xA;------------------------------------------------------------------------------&#xA;Unsupervised learning | Predictive clustering         |  Descriptive clustering,&#xA;                      |                               |  Association rule mining&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="12909" LastEditorUserId="29575" LastEditDate="2017-12-18T01:12:38.893" LastActivityDate="2017-12-18T01:12:38.893" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8146" PostTypeId="1" CreationDate="2015-09-21T18:06:44.903" Score="3" ViewCount="969" Body="&lt;ol&gt;&#xA;&lt;li&gt;I am working on a project for displaying products to customer by context, based on a search query. For example, I don't want customers to have to enter a specific product name, instead searching based on functionality (e.g., &quot;walls do not heat much&quot; would return product names such as &quot;Whirlpool NEO IC355 ROY 3S 340 L Double Door Refrigerator&quot;)  &lt;/li&gt;&#xA;&lt;li&gt;I have a training set comprised of the functionality associated products. I am planning to use Logistic Regression to train a model on these data. How do I process this data in Python or extract features to feed into logistic regression? I have heard of &quot;Bag of words model&quot;, but not sure how to use this, or is it even applicable here?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I know there are plenty of NLTK libraries available. But, I want to implement it from the scratch or using minimum external libraries possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help or are there any resources to refer to?&lt;/p&gt;&#xA;" OwnerUserId="12911" LastEditorUserId="13413" LastEditDate="2015-10-22T23:24:18.750" LastActivityDate="2015-10-22T23:24:18.750" Title="How to train a Text Based data for a Machine Learning problem?" Tags="|machine-learning|classification|nlp|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8147" PostTypeId="1" CreationDate="2015-09-21T18:51:35.353" Score="1" ViewCount="167" Body="&lt;p&gt;I have been trying to find the sum of a series defined by &lt;code&gt;Σ(1+r(i))^1:n&lt;/code&gt; in R where &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;n&lt;/code&gt; is in &lt;code&gt;(1:N)&lt;/code&gt;.  Both &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;n&lt;/code&gt; are of the same length. The solution needs to be two functions, where &lt;code&gt;function 1&lt;/code&gt; calculates the expression &lt;code&gt;(1+r)^n&lt;/code&gt; and the second &lt;code&gt;function 2&lt;/code&gt; recursively calls &lt;code&gt;function 1&lt;/code&gt; to calculate the summation. I am struggling to create a way to call &lt;code&gt;function 1&lt;/code&gt; within &lt;code&gt;function 2&lt;/code&gt; and having it calculate the values recursively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Rates   Tenor   Printrest  &#xA;1.20%   1   1.012  &#xA;1.35%   2   1.02718225  &#xA;1.90%   3   1.058089859  &#xA;2.01%   4   1.082856706  &#xA;1.80%   5   1.093298847  &#xA;1.70%   6   1.106434521  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Answer  6.379862183&lt;/p&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="11097" LastEditDate="2015-09-22T10:13:26.217" LastActivityDate="2015-09-24T01:23:20.170" Title="Sum of a Series" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8148" PostTypeId="2" ParentId="8138" CreationDate="2015-09-21T21:02:47.130" Score="3" Body="&lt;p&gt;The basic rationale behind the following suggestion is to associate &quot;eigenvectors&quot; and &quot;templates&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular one could use LSA on the whole corpus based on a a bag-of-words. The resulting eigenvectors would serve as surrogate templates; these eigenvectors should not be directly affected by the number of words in each template. Subsequently the scores could be used to cluster the documents together following a standard procedure (eg. $k$-means in conjunction with AIC).  As an alternative to LSA one could use NNMF. Let me point out that the LSA (or NNMF) would probably need to be done to the transformed TF-IDF rather than the raw word-counts matrix.&lt;/p&gt;&#xA;" OwnerUserId="12905" LastActivityDate="2015-09-21T21:02:47.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8149" PostTypeId="2" ParentId="8146" CreationDate="2015-09-21T21:50:09.343" Score="0" Body="&lt;p&gt;For your particular problem I'm not sure that using a supervised Logistic Regression approach is ideal, but I suppose that is a different and larger topic. To answer your question, yes you can use a &quot;bag of words&quot; representation of your text. Python's &lt;code&gt;sci-kit learn&lt;/code&gt; library offers this functionality via both &lt;code&gt;CountVectorizer&lt;/code&gt; and &lt;code&gt;TfidfVectorizer&lt;/code&gt;. This will result in a sparse matrix representation of n-grams and occurrences of those n-grams in your corpus. From here you basically have 2 options, (1) training your supervised model directly on the sparse matrix or (2) reducing the dimension of your sparse matrix so it be represented as a dense matrix. Luckily sklearn offers functionality for both these, their &lt;code&gt;LogisticRegression&lt;/code&gt; class supports sparse matrices and their &lt;code&gt;TruncatedSVD&lt;/code&gt; implementation of PCA/LSA supports sparse matrices as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should give what you need from a technical perspective to build a model, but I think the real question is what feature engineering you may be able to do in addition to simply training on the bag of words representation of the text.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2015-09-21T21:50:09.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8150" PostTypeId="1" AcceptedAnswerId="8151" CreationDate="2015-09-22T01:35:33.447" Score="4" ViewCount="246" Body="&lt;p&gt;I am asking this question because the &lt;a href=&quot;https://datascience.stackexchange.com/questions/8018/are-there-any-machine-learning-techniques-to-identify-points-on-plots-images&quot;&gt;previous&lt;/a&gt; one wasn't very helpful and I asked about a different solution for the same problem.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The Problem&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I have lateral positions, &lt;code&gt;xcoord&lt;/code&gt;, of vehicles over time which were recorded as the distances from the right edge of the road. This can be seen for one vehicle in the following plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s8BNW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s8BNW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each point on the plot represents the position of the front center of the vehicle. When the vehicle changes the lane (lane numbers not shown) there is a drastic change in the position as seen after the 'Start of Lane Change' on the plot.&lt;br&gt;&#xA;The data behind this plot are like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Vehicle.ID Frame.ID   xcoord Lane&#xA;1          2       13 16.46700    2&#xA;2          2       14 16.44669    2&#xA;3          2       15 16.42600    2&#xA;4          2       16 16.40540    2&#xA;5          2       17 16.38486    2&#xA;6          2       18 16.36433    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to identify the start and end data points of a lane change by clustering the data as shown in the plot. The data points in the plot circled in red are more similar to each other because the variation between them is smaller compared to the data points in the middle which see large variation in position (&lt;code&gt;xcoord&lt;/code&gt;).&lt;br&gt;&#xA;My questions are: Is it possible to apply any clustering technique to segment these data so that I could identify the start and end point of a lane change? If yes, which technique would be most suitable?&lt;br&gt;&#xA;I use R. I have tried Hierarchical clustering before but don't know how to apply it in this context. Please help. &lt;/p&gt;&#xA;" OwnerUserId="4933" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-22T06:55:02.183" Title="How to create clusters of position data?" Tags="|r|clustering|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8151" PostTypeId="2" ParentId="8150" CreationDate="2015-09-22T06:55:02.183" Score="3" Body="&lt;p&gt;I doubt any of the clustering algorithms will work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, you should look into:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;segmentation (yes, this is something different), specifically time series segmentation&lt;/li&gt;&#xA;&lt;li&gt;change detection (as you said, there is a rather constant distribution first, then a change, then a rather constant distribution again&lt;/li&gt;&#xA;&lt;li&gt;segment-wise regression may also work: try to find the best fit that is constant, linearly changing, and constant again. It's essentially four parameters to optimize in this restricted model: average before and after + beginning and end of transition.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-09-22T06:55:02.183" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8152" PostTypeId="2" ParentId="8131" CreationDate="2015-09-22T07:42:59.590" Score="1" Body="&lt;p&gt;Typically, quantitative analysis is planned and performed, based on research study's goals. Focusing on research goals and corresponding research questions, researcher would propose a model (or several models) and a set of hypotheses, associated with the model(s). Model(s) and its/their elements' types usually dictate (suggest) quantitative approaches that would make sense in a particular situation. For example, if your model includes latent variables, you would have to use appropriate methods to perform data analysis (i.e., structural equation modeling). Otherwise, you can apply a variety of other methods, such as time series analysis or, as you mentioned, multiple regression and machine learning. For more details on research workflow with latent variables, also see section #3 in &lt;a href=&quot;https://datascience.stackexchange.com/a/1006/2452&quot;&gt;my relevant answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last note: whatever methods you use, pay enough attention to the following two very important aspects - performing full-scale exploratory data analysis (EDA) (see &lt;a href=&quot;https://datascience.stackexchange.com/a/5095/2452&quot;&gt;my relevant answer&lt;/a&gt;) and trying to design and perform your analysis in the reproducible research fashion (see &lt;a href=&quot;https://datascience.stackexchange.com/a/759/2452&quot;&gt;my relevant answer&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-22T07:42:59.590" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8154" PostTypeId="2" ParentId="8146" CreationDate="2015-09-22T10:39:40.333" Score="1" Body="&lt;p&gt;Your problem looks more like a ranking problem than a classification problem to me. Have you tried a more naive method, like a 1-NN &quot;classifier&quot; with unigram text representation, Tf-Idf term weighting and a cosine similarity metric? It's far from the state of the art but it tends to give rather good results in retrieval and recommendation tasks.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-22T10:39:40.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8156" PostTypeId="1" AcceptedAnswerId="8160" CreationDate="2015-09-22T19:53:12.657" Score="1" ViewCount="804" Body="&lt;p&gt;Can somebody please give me some reference on implementing SVM using PySpark.&lt;/p&gt;&#xA;" OwnerUserId="10810" LastEditorUserId="11097" LastEditDate="2015-09-23T11:30:03.187" LastActivityDate="2015-09-23T11:30:03.187" Title="Reference of SVM Using Spark" Tags="|apache-spark|" AnswerCount="1" CommentCount="2" ClosedDate="2015-09-23T11:29:10.833" ContentLicense="CC BY-SA 3.0" />
  <row Id="8157" PostTypeId="1" CreationDate="2015-09-22T20:24:47.313" Score="1" ViewCount="11350" Body="&lt;p&gt;Here is my original dataset from a voting activity. Each participant voted for one option (A,B,C,D,E,F) listed below. The numerical value is the total number of participants who voted for each option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/olSQL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/olSQL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to bootstrap the voting for 1000 times &lt;strong&gt;(sample with replacement)&lt;/strong&gt; and make a comparison between the pre-event and post-event voting for each category using independent sample t-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't know how to start with generating the 1000 bootstrapping samples...I have the &quot;Boot&quot; packages installed on R. Could anyone give me a hint what kind of steps I should take in order to generate 1000 bootstrap samples based on the original data I provide above?&lt;/p&gt;&#xA;" OwnerUserId="12931" LastEditorUserId="12931" LastEditDate="2015-09-25T22:46:57.417" LastActivityDate="2015-09-26T08:27:05.450" Title="How to generate bootstrapping samples in R?" Tags="|r|statistics|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8159" PostTypeId="2" ParentId="6308" CreationDate="2015-09-23T00:01:05.647" Score="7" Body="&lt;p&gt;The Dataframe Python API exposes the RDD of a Dataframe by calling the following :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.rdd # you can save it, perform transformations of course, etc. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;df.rdd returns the content as an pyspark.RDD of Row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then map on that RDD of Row transforming every Row into a &lt;code&gt;numpy&lt;/code&gt; vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 1: &lt;code&gt;df&lt;/code&gt;is the variable define our Dataframe. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 2: this function is available since Spark 1.3&lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="5177" LastEditDate="2015-09-23T11:36:55.943" LastActivityDate="2015-09-23T11:36:55.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8160" PostTypeId="2" ParentId="8156" CreationDate="2015-09-23T03:47:13.190" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-linear-methods.html#linear-support-vector-machines-svms&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; of Apache Spark's MLib library has a neat and clear reference to the implementation of linear SVM's in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It supports two models: &lt;code&gt;SVMWithSGD&lt;/code&gt; and &lt;code&gt;SVMModel&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-09-23T03:47:13.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8161" PostTypeId="1" AcceptedAnswerId="8170" CreationDate="2015-09-23T06:47:48.473" Score="2" ViewCount="151" Body="&lt;p&gt;I am working on building a scoring algorithm for student data,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;say the attributes are :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name, location, age, class, school_name, skill1, skill2, skill3 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;based on these data I need to create a student score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to assign weight-ages for age, class, school_name skills and come up with a score for student.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;say I have 2 scoring models like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score_1 = x1*location_weight + x2*age_weight + x2*class_weight + x3*school_name_weight + x4*skill1_weight + x5*skill2_weight + x6*skill3_weight&#xA;&#xA;score_2 = y1*location_weight + y2*age_weight + y2*class_weight + y3*school_name_weight + y4*skill1_weight + y5*skill2_weight + y6*skill3_weight  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now how can I compare these models and evaluate them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is I don't have a test or validation set to prove or compare how accurate each of these model is, so in this case what is the best approach to compare and validate different models? also what is the best ways to build a validation set from scratch?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2015-09-24T06:55:24.830" LastActivityDate="2015-09-24T06:55:24.830" Title="Creating validation data for model comparison" Tags="|dataset|categorical-data|scoring|normalization|weighted-data|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8162" PostTypeId="1" CreationDate="2015-09-23T07:37:43.030" Score="1" ViewCount="752" Body="&lt;p&gt;I'm new to R and cannot make plotting work as desired. The problem is that R seems to draw the same four lines over and over again, redundantly. The details of the case I'm having are as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(dataset)&#xA;'data.frame':   57641 obs. of  3 variables:&#xA; $ duration : num  3 8 7 2 4 8 2 2 8 8 ...&#xA; $ graduated: logi  FALSE TRUE TRUE FALSE FALSE TRUE ...&#xA; $ group    : num  651 651 671 671 651 651 651 651 651 651 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I fit a Cox proportional hazards regression model to it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;survObj &amp;lt;- Surv(time = dataset$duration / 2, event = dataset$graduated)&#xA;model &amp;lt;- coxph(survObj ~ group, data=dataset)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, following &lt;a href=&quot;https://rpubs.com/daspringate/survival&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; example, I create a frame that would hopefully group the survival functions by the group number:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;frame &amp;lt;- data.frame(group = dataset$group)&#xA;&#xA;&amp;gt; str(frame)&#xA;'data.frame':   57641 obs. of  1 variable:&#xA; $ group: num  651 651 671 671 651 651 651 651 651 651 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There's four groups in the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; unique(dataset$group)&#xA;[1] 651 671 652 681&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this new frame, I create a fitted survival model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fitObjGrouped &amp;lt;- survfit(model, newdata = frame)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, I plot the thing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;color_set &amp;lt;- rainbow(4)&#xA;plot(fitObjGrouped, col=color_set)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result has the correct lines, but drawn many times over each other:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/r5tiz.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/r5tiz.jpg&quot; alt=&quot;Survival model plotted - overlapping lines&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, there's two red lines and two blue lines drawn last. They're the correct ones, one for each category, but closer observation reveals that there's a green or other color lines underneath each of them. When converting this to PDF the file size is 273 times larger than what it should be!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is: why is R drawing the lines so many many times and how could I achieve correct model fitting and plotting at the same time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody please help me to better understand the R commands I'm using? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="10764" LastEditorUserId="10764" LastEditDate="2015-09-23T07:44:17.433" LastActivityDate="2018-03-21T16:29:05.997" Title="R plot(surv(), newdata=...) draws same lines many times - why?" Tags="|r|survival-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8163" PostTypeId="2" ParentId="8009" CreationDate="2015-09-23T08:05:21.247" Score="3" Body="&lt;p&gt;RMSProp is indeed an unpublished method, and in the lecture Geoffrey Hinton gives just the general idea behind RMSProp - to divide the gradient by a moving average of the gradient magnitude. The lecture has disappeared from YouTube but you can find the slides in the end of this PDF:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When this principle is applied to Stochastic Gradient Descent, the update rule you showed is obtained. Since Hinton did not propose an exact algorithm, this principle has been applied to different optimization methods. I agree it's confusing all these methods call themselves RMSProp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;climin&lt;/code&gt; implements RMSProp with Nesterov momentum. Momentum methods try to avoid the oscillation that often happens with SGD by slowly changing the current direction of updates. The algorithm given in &lt;code&gt;climin&lt;/code&gt; documentation introduces the &lt;span class=&quot;math-container&quot;&gt;$\beta$&lt;/span&gt; parameter that controls how much of the previous update direction is retained. Nesterov momentum is implemented by first taking a step towards the previous update direction &lt;span class=&quot;math-container&quot;&gt;$v_t$&lt;/span&gt;, calculating gradient at that position, using the gradient to obtain the new update direction &lt;span class=&quot;math-container&quot;&gt;$v_{t+1}$&lt;/span&gt;, and finally updating the parameters. The &lt;code&gt;climin&lt;/code&gt; implementation also includes the smoothing term &lt;span class=&quot;math-container&quot;&gt;$\iota$&lt;/span&gt; inside the square root for stability (1e-8), even though it's not mentioned in the documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The implementation in the Downhill library is based on the algorithm described in the &lt;a href=&quot;https://arxiv.org/pdf/1308.0850.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper by A. Graves&lt;/a&gt;. In the article (equations 38–40) the square of the average gradient is subtracted from the average square gradient. Apparently the idea is to approximate the variance of the gradient instead of its magnitude (recall that the variance of a random variable is equal to the mean of the square minus the square of the mean).&lt;/p&gt;&#xA;" OwnerUserId="12932" LastEditorUserId="12932" LastEditDate="2019-01-03T11:58:50.247" LastActivityDate="2019-01-03T11:58:50.247" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="8164" PostTypeId="1" AcceptedAnswerId="8174" CreationDate="2015-09-23T08:16:45.387" Score="0" ViewCount="1897" Body="&lt;p&gt;I am getting &lt;code&gt;attr(, &quot;nn.index&quot;)&lt;/code&gt; as part of my KNN output in R. What is meant by that and how is this value getting calculated?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;knn.pred &amp;lt;- knn(tdm.stack.nl_train, tdm.stack.nl_Test, tdm.cand_train)&#xA;print(knn.pred)&#xA;&#xA;&amp;gt; knn.pred&#xA;&#xA;&#xA;[1] Silent      Silent      Silent      Silent      Silent      Silent      Silent     &#xA; [8] Silent      Silent      Silent    &#xA;&#xA;&#xA;attr(,&quot;nn.index&quot;)&#xA;&#xA;&#xA;      [,1]&#xA;[1,]  292  &#xA;[2,]  292  &#xA;[3,]  343  &#xA;[4,]  444  &#xA;[5,]  250  &#xA;[6,]  445  &#xA;[7,]  270  &#xA;[8,]  228  &#xA;[9,]  302  &#xA;[10,]  355  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9793" LastEditorUserId="11097" LastEditDate="2015-09-24T08:52:01.403" LastActivityDate="2015-09-24T08:52:01.403" Title="what is nn.index mean in KNN output" Tags="|r|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8165" PostTypeId="2" ParentId="8161" CreationDate="2015-09-23T09:26:46.420" Score="-1" Body="&lt;p&gt;I would start by fixing a seed (so that results are reproducible), then selecting a subsample (say, 10pct of data) and saving that as validation set. If you're using R, check the caret package &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/caret/index.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and if Python is your weapon of choice, familiarize yourself with the sklearn documentation in 3.1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/cross_validation.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both of those have multiple functions that might come in handy. If your computational resources allow it, I would suggest a cross-validation procedure (evaluating a model on multiple training-validation splits), both of the links I gave contain references to multiple functions that can help you achieve that. &lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-23T09:26:46.420" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8168" PostTypeId="1" AcceptedAnswerId="8171" CreationDate="2015-09-23T09:45:30.677" Score="0" ViewCount="267" Body="&lt;p&gt;I am using Matlab Neural Network toolbox for a classification problem. Now  considering a single set of data, if the inbuilt neural network is trained and classified with same data multiple number of time, different accuracy and different confusion matrix is obtained. Now which result should I take? Should I take all the vales obtained in all the training instances and average them fix on one particular result?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2015-09-23T14:02:10.540" Title="Multiple confusion matrix for multiple training instances. Which one to take?" Tags="|neural-network|matlab|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8169" PostTypeId="2" ParentId="8094" CreationDate="2015-09-23T09:51:17.850" Score="0" Body="&lt;p&gt;I am sorry to disappoint you but &lt;a href=&quot;https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Correlation does not imply causation&lt;/strong&gt;&lt;/a&gt; !.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to make sure that the data is statically significant (enough data, the difference is not negligible, etc), I would suggest Z-Test, or chi-square, depending on the nature of your data.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2015-09-23T09:51:17.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8170" PostTypeId="2" ParentId="8161" CreationDate="2015-09-23T11:34:31.400" Score="4" Body="&lt;p&gt;Predicting and scoring are two different tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And according to your answers and comments you are not solving prediction problem. You just want to set to each student a number in range [1,100] according to some rule. This is &lt;strong&gt;ranking&lt;/strong&gt; (or scoring, whatever).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the terms #prediction_model, #accuracy, #validation, #training_set are &lt;strong&gt;out of this scope&lt;/strong&gt;. You don't need to validate anything. You are not making predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want is to map ranks to students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But a problem is that you have mostely categorical data (school name, location etc) that cannot be 'ranked'. Some of them are useless at all: how does the student name refer to his school progress? :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you change it somehow to numerical (e.g. 'Skill_1_level', 'Skill_2_level', 'remoteness_of_location', 'school rank' etc) than you can do some ranking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normalize data: each of your factors&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cdJkC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cdJkC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Multiple by 100, as you want [0,100] range instead of [0,1]&lt;/li&gt;&#xA;&lt;li&gt;Set up weights based on your experience according to factor's importance. So that the sum of weights is 1.&lt;/li&gt;&#xA;&lt;li&gt;And finally build a rank (score):&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rank = 0.1 * skill_1_level + 0.2 * skill_2_level + 0.05 * remoteness_of_location + 0.5 * school_rank + ...&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-09-23T11:46:54.070" LastActivityDate="2015-09-23T11:46:54.070" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8171" PostTypeId="2" ParentId="8168" CreationDate="2015-09-23T14:02:10.540" Score="1" Body="&lt;p&gt;I can't check at the moment (no Matlab at hand), but I suppose the differences come from the different random seeds used to initialize the neural networks (at least this is the only part which i can think of that has a random component). I would suggest predicting class probabilities, averaging those and then viewing the resulting confusion matrix of the &quot;averaged&quot; prediction. This way you - to a degree - mitigating the effect of randomness resulting from different initializations of the weights.&lt;/p&gt;&#xA;" OwnerUserId="10936" LastActivityDate="2015-09-23T14:02:10.540" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8172" PostTypeId="1" AcceptedAnswerId="12213" CreationDate="2015-09-23T15:02:20.537" Score="4" ViewCount="2977" Body="&lt;p&gt;I built a predictive model using logistic regression for direct marking creatives. I built a “children’s” model which predicts buying behavior for those that purchase boys, girls and baby apparel.  But one may have a high propensity right now to buy baby clothes but in 12 months, when ones  kid is a toddler, their propensity drops for baby and increases for Girls/Boys.  I’ve been trying to brainstorm ways to take into account that scenario.  My question is, can anyone suggest options for adjustments to the model to based on that additional criteria?   &lt;/p&gt;&#xA;" OwnerUserId="12947" LastActivityDate="2017-05-11T22:03:52.360" Title="Propensity Modeling for Retail Marketing: Model Adjustments Based on Consumer Life Changes." Tags="|predictive-modeling|logistic-regression|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8173" PostTypeId="1" CreationDate="2015-09-23T16:11:07.463" Score="0" ViewCount="721" Body="&lt;p&gt;I have a batch of links with the following naming convention:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://api.cilabs.net/v1/conferences/ws15/info/attendees?page=1&quot; rel=&quot;nofollow&quot;&gt;https://api.cilabs.net/v1/conferences/ws15/info/attendees?page=1&lt;/a&gt;; and the page number incrementing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to extract the data from those web pages into a CSV file.&lt;/p&gt;&#xA;" OwnerUserId="12949" LastEditorUserId="11097" LastEditDate="2015-09-25T07:39:50.103" LastActivityDate="2015-09-25T07:39:50.103" Title="Batch convert json links to csv" Tags="|dataset|csv|" AnswerCount="1" CommentCount="6" ClosedDate="2015-09-24T08:52:38.973" ContentLicense="CC BY-SA 3.0" />
  <row Id="8174" PostTypeId="2" ParentId="8164" CreationDate="2015-09-23T16:48:30.630" Score="1" Body="&lt;p&gt;I guess you are using the &lt;code&gt;fnn&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;attr&lt;/code&gt; is a list of attributes which can be used for both &lt;code&gt;nn.index&lt;/code&gt; and &lt;code&gt;nn.dist&lt;/code&gt;. In this case, you are using &lt;code&gt;index&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, index returns an n x k matrix for the nearest neighbor indice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the definition of the nearest neighbor index is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The nearest neighbor index is expressed as the ratio of the observed&#xA;  distance divided by the expected distance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://edndoc.esri.com/arcobjects/9.2/net/shared/geoprocessing/spatial_statistics_tools/average_nearest_neighbor_spatial_statistics_.htm&quot; rel=&quot;nofollow&quot;&gt;Definition reference&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-09-23T16:48:30.630" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8175" PostTypeId="2" ParentId="8173" CreationDate="2015-09-23T17:50:37.233" Score="1" Body="&lt;p&gt;I guess what you are looking for is a crawler, which can crawl through the web pages and scrape the data for you into a csv file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really don't like the idea of giving away the full code. So, I would tell you how it's done. I would use Python for the example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define the url of the web page. Leave the last part (&lt;code&gt;page=&lt;/code&gt;), as it can go into a loop, so that as it needs to be incremented after every page gets scrapped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, identify the &lt;a href=&quot;https://en.wikipedia.org/wiki/XPath&quot; rel=&quot;nofollow&quot;&gt;xpaths&lt;/a&gt; of the data which you want to scrape. Store each of them in an array, and keep writing(and appending into a csv)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;a href=&quot;https://github.com/Dawny33/Cricket-Analytics/blob/master/Crawler.py&quot; rel=&quot;nofollow&quot;&gt;code of one such scrapper&lt;/a&gt; which I have written for a project of mine. Refer to it in case of a road-block. I have used the &lt;a href=&quot;http://lxml.de/&quot; rel=&quot;nofollow&quot;&gt;lxml&lt;/a&gt; library of Python. &#xA;You can use other fancy libraries like &lt;a href=&quot;http://scrapy.org/&quot; rel=&quot;nofollow&quot;&gt;Scrapy&lt;/a&gt;, &lt;a href=&quot;http://www.crummy.com/software/BeautifulSoup/&quot; rel=&quot;nofollow&quot;&gt;BeautifulSoup&lt;/a&gt;. But, they are just fancy wrappers over lxml, so it's faster.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-09-23T17:50:37.233" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8177" PostTypeId="2" ParentId="8157" CreationDate="2015-09-24T01:04:56.403" Score="3" Body="&lt;h2&gt;Creating samples&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You can create samples using &lt;code&gt;sample&lt;/code&gt; and &lt;code&gt;replicate&lt;/code&gt; functions.&#xA;I create &lt;code&gt;smp&lt;/code&gt; with the first given row.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; smp &amp;lt;- c(rep('A', 8), rep('B', 4), rep('C', 2), rep('D', 11), rep('E', 21), rep('F', 0))&#xA;&amp;gt; table(smp)&#xA;smp&#xA; A  B  C  D  E &#xA; 8  4  2 11 21 &#xA;&amp;gt; &#xA;&amp;gt; samples &amp;lt;- replicate(1000, sample(smp)) # creating 1000 samples&#xA;&amp;gt; dim(samples) # 1000 samples of 46 votes&#xA;[1]   46 1000&#xA;&amp;gt; &#xA;&amp;gt; samples[,1:3]&#xA;      [,1] [,2] [,3]&#xA; [1,] &quot;E&quot;  &quot;A&quot;  &quot;E&quot; &#xA; [2,] &quot;D&quot;  &quot;E&quot;  &quot;E&quot; &#xA; [3,] &quot;E&quot;  &quot;D&quot;  &quot;C&quot; &#xA; [4,] &quot;D&quot;  &quot;E&quot;  &quot;E&quot; &#xA; [5,] &quot;A&quot;  &quot;D&quot;  &quot;D&quot; &#xA; [6,] &quot;B&quot;  &quot;E&quot;  &quot;E&quot; &#xA; [7,] &quot;E&quot;  &quot;A&quot;  &quot;D&quot; &#xA; [8,] &quot;B&quot;  &quot;C&quot;  &quot;D&quot; &#xA; [9,] &quot;B&quot;  &quot;A&quot;  &quot;E&quot; &#xA;[10,] &quot;E&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[11,] &quot;A&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[12,] &quot;D&quot;  &quot;E&quot;  &quot;D&quot; &#xA;[13,] &quot;D&quot;  &quot;E&quot;  &quot;B&quot; &#xA;[14,] &quot;D&quot;  &quot;B&quot;  &quot;E&quot; &#xA;[15,] &quot;E&quot;  &quot;E&quot;  &quot;D&quot; &#xA;[16,] &quot;E&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[17,] &quot;E&quot;  &quot;B&quot;  &quot;B&quot; &#xA;[18,] &quot;A&quot;  &quot;D&quot;  &quot;E&quot; &#xA;[19,] &quot;E&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[20,] &quot;E&quot;  &quot;E&quot;  &quot;C&quot; &#xA;[21,] &quot;E&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[22,] &quot;E&quot;  &quot;D&quot;  &quot;A&quot; &#xA;[23,] &quot;D&quot;  &quot;D&quot;  &quot;B&quot; &#xA;[24,] &quot;E&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[25,] &quot;D&quot;  &quot;D&quot;  &quot;A&quot; &#xA;[26,] &quot;E&quot;  &quot;E&quot;  &quot;D&quot; &#xA;[27,] &quot;C&quot;  &quot;A&quot;  &quot;E&quot; &#xA;[28,] &quot;D&quot;  &quot;E&quot;  &quot;D&quot; &#xA;[29,] &quot;D&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[30,] &quot;D&quot;  &quot;A&quot;  &quot;E&quot; &#xA;[31,] &quot;A&quot;  &quot;D&quot;  &quot;E&quot; &#xA;[32,] &quot;E&quot;  &quot;E&quot;  &quot;A&quot; &#xA;[33,] &quot;E&quot;  &quot;E&quot;  &quot;D&quot; &#xA;[34,] &quot;E&quot;  &quot;E&quot;  &quot;A&quot; &#xA;[35,] &quot;A&quot;  &quot;E&quot;  &quot;E&quot; &#xA;[36,] &quot;D&quot;  &quot;C&quot;  &quot;A&quot; &#xA;[37,] &quot;A&quot;  &quot;D&quot;  &quot;A&quot; &#xA;[38,] &quot;E&quot;  &quot;D&quot;  &quot;D&quot; &#xA;[39,] &quot;E&quot;  &quot;D&quot;  &quot;E&quot; &#xA;[40,] &quot;A&quot;  &quot;E&quot;  &quot;A&quot; &#xA;[41,] &quot;B&quot;  &quot;A&quot;  &quot;A&quot; &#xA;[42,] &quot;C&quot;  &quot;A&quot;  &quot;D&quot; &#xA;[43,] &quot;A&quot;  &quot;A&quot;  &quot;D&quot; &#xA;[44,] &quot;E&quot;  &quot;B&quot;  &quot;E&quot; &#xA;[45,] &quot;E&quot;  &quot;D&quot;  &quot;B&quot; &#xA;[46,] &quot;E&quot;  &quot;B&quot;  &quot;E&quot; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If all you need are samples, &lt;code&gt;sample&lt;/code&gt; solves your problem.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Bootstraping&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;To bootstrap you need to compute a statistic. For example, I compute the weighted mean for a table of votes.&#xA;I'm using &lt;code&gt;boot::boot&lt;/code&gt; to bootsprap.&#xA;You must pass the original sample and a handler which receives the original sample &lt;code&gt;s&lt;/code&gt; and a vector with the indexes shuffled (&lt;code&gt;idx&lt;/code&gt;).&#xA;This function returns a &lt;code&gt;boot&lt;/code&gt; object that shows the bootstrap statistics.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; boot.smp &amp;lt;- boot::boot(smp, function(s, idx) {&#xA;+ tt &amp;lt;- table(s[idx])&#xA;+ weighted.mean(tt, as.numeric(factor(names(tt), labels=1:6, levels=LETTERS[1:6])))&#xA;+ }, 1000)&#xA;&amp;gt; boot.smp&#xA;&#xA;CASE RESAMPLING BOOTSTRAP FOR CENSORED DATA&#xA;&#xA;&#xA;Call:&#xA;boot::boot(data = smp, statistic = function(d, w) {&#xA;    tt &amp;lt;- table(d[w])&#xA;    weighted.mean(tt, as.numeric(factor(names(tt), labels = 1:6, &#xA;        levels = LETTERS[1:6])))&#xA;}, R = 1000)&#xA;&#xA;&#xA;Bootstrap Statistics :&#xA;    original    bias    std. error&#xA;t1*     11.4 0.4126077    1.274827&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;boot&lt;/code&gt; object has other methods to compute confidence interval and return much more information.&lt;/p&gt;&#xA;" OwnerUserId="12960" LastEditorUserId="12960" LastEditDate="2015-09-26T08:27:05.450" LastActivityDate="2015-09-26T08:27:05.450" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8178" PostTypeId="2" ParentId="8147" CreationDate="2015-09-24T01:23:20.170" Score="1" Body="&lt;p&gt;I think &lt;code&gt;within&lt;/code&gt; and &lt;code&gt;cumsum&lt;/code&gt; functions solve your problem.&#xA;The last row in &lt;code&gt;comp.acc&lt;/code&gt; column has what you want.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data.txt &amp;lt;- 'Rates   Tenor   Printrest  &#xA;+ 1.20   1   1.012  &#xA;+ 1.35   2   1.02718225  &#xA;+ 1.90   3   1.058089859  &#xA;+ 2.01   4   1.082856706  &#xA;+ 1.80   5   1.093298847  &#xA;+ 1.70   6   1.106434521'&#xA;&amp;gt; &#xA;&amp;gt; df &amp;lt;- read.table(text=data.txt, header = TRUE)&#xA;&amp;gt; &#xA;&amp;gt; within(df, {&#xA;+   comp &amp;lt;- (1 + Rates/100)^(Tenor)&#xA;+   comp.acc &amp;lt;- cumsum(comp)&#xA;+ })&#xA;  Rates Tenor Printrest comp.acc     comp&#xA;1  1.20     1  1.012000 1.012000 1.012000&#xA;2  1.35     2  1.027182 2.039182 1.027182&#xA;3  1.90     3  1.058090 3.097272 1.058090&#xA;4  2.01     4  1.082857 4.180129 1.082857&#xA;5  1.80     5  1.093299 5.273428 1.093299&#xA;6  1.70     6  1.106435 6.379862 1.106435&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12960" LastActivityDate="2015-09-24T01:23:20.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8181" PostTypeId="2" ParentId="8117" CreationDate="2015-09-24T04:17:47.300" Score="1" Body="&lt;p&gt;I think your last question is worth discussing, but forgive my careless on skipping the details of the model and just leaving a quick answer here :P&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeating a sentence in your corpus would definitely change the learning result, and strength the relationship of the words in this sentence, because one of the models behind word2vec is &lt;code&gt;skip-gram&lt;/code&gt;, which assume the center word can be used to predict its surroundings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I have to ask another question coming follows: what is our purpose of using word2vec? &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;To find similar words in semantic and synthetic, which is used to search and information retrieval.&lt;/li&gt;&#xA;&lt;li&gt;A skip-gram model is useful for modeling those like click-sequence data, which could be used in recommendation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1048" LastActivityDate="2015-09-24T04:17:47.300" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8183" PostTypeId="2" ParentId="6814" CreationDate="2015-09-24T06:17:23.487" Score="5" Body="&lt;p&gt;&lt;a href=&quot;https://www.cs.cmu.edu/%7Eschneide/tut5/node42.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Leave-one-out&lt;/code&gt;&lt;/a&gt; cross validation is probably the most straight-forward way to address this.  If you happen to be using a model that requires a lot of time to train, then leave n% out might be more appropriate.&lt;/p&gt;&#xA;&lt;p&gt;The method involves eliminating one known rating and trying to predict it.  If you want to remove n percent of the ratings, just choose them randomly rather than choosing a specific number of every user's ratings.  And keep n pretty small - on the order of 10% or less.&lt;/p&gt;&#xA;&lt;p&gt;Here is a good treatment of cross validation methods for recommender systems, &lt;a href=&quot;https://chrome.deib.polimi.it/images/8/8e/Cremonesi_2008_CROSSMEDIA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;An Evaluation Methodology for Collaborative Recommender Systems&lt;/a&gt;.  Generally:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Holdout is a method that splits a dataset into two parts: a training&#xA;set and a test set. These sets could have different proportions. In&#xA;the setting of recommender systems the partitioning is performed by&#xA;randomly selecting some ratings from all (or some of) the users. The&#xA;selected ratings constitute the test set, while the remaining ones are&#xA;the training set. This method is also called leave-k-out. In [17],&#xA;Sarwar et al. split the dataset into 80% training and 20% test data.&#xA;In [18] several ratios among training and test (from 0.2 to 0.95 with&#xA;an increment of&#xA;0.05) are chosen and for each one the experiment is repeated ten times with different training and test sets and finally the results are&#xA;averaged. In [13] the test set is made by 10% of users: 5 ratings for&#xA;each user in the test set are withheld.&lt;/p&gt;&#xA;&lt;p&gt;Leave-one-out is a method&#xA;obtained by setting k = 1 in the leave-k-out method. Given an active&#xA;user, we withhold in turn one rated item. The learning algorithm is&#xA;trained on the remaining data. The withheld element is used to&#xA;evaluate the correctness of the prediction and the results of all&#xA;evaluations are averaged in order to compute the final quality&#xA;estimate. This method has some disadvantages, such as the overfitting&#xA;and the high computational complexity. This technique is suitable to&#xA;evaluate the recommending quality of the model for users who are&#xA;already registered as members of the system. Karypis et al. [10]&#xA;adopted a trivial version of the leave-one-out creating the test set&#xA;by randomly selecting one of the non-zero entries for each user and&#xA;the remaining entries for training. In [7], Breese et al. split the&#xA;URM in training and test set and then, in the test set, withhold a&#xA;single randomly selected rating for each user.&lt;/p&gt;&#xA;&lt;p&gt;A simple variant of the&#xA;holdout method is the m-fold cross-validation. It consists in dividing&#xA;the dataset into m independent folds (so that folds do not overlap).&#xA;In turn, each fold is used exactly once as test set and the remaining&#xA;folds are used for training the model. According to [20] and [11], the&#xA;suggested number of folds is 10. This technique is suitable to&#xA;evaluate the recommending capability of the model when new users&#xA;(i.e., users do not already belong to the model) join the system. By&#xA;choosing a reasonable number of folds we can compute mean, variance&#xA;and confidence interval.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="103967" LastEditDate="2020-08-28T06:54:14.897" LastActivityDate="2020-08-28T06:54:14.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8186" PostTypeId="1" CreationDate="2015-09-24T14:58:08.870" Score="0" ViewCount="575" Body="&lt;p&gt;I have searched at all possible areas and haven't been able to find the solution to my problem. I am hoping that someone here would be able to help me out. If you feel that this question is already answered, I would appreciate the link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;code&gt;R3.2.2 64 bit&lt;/code&gt; on a &lt;code&gt;windows 7&lt;/code&gt; machine. I have a &lt;code&gt;fixed width file&lt;/code&gt; of size &lt;code&gt;1.9 GB&lt;/code&gt; and has approx &lt;code&gt;18 million&lt;/code&gt; records in it. I need to read the file based on a filter condition &lt;code&gt;RecordID = &quot;Cast&quot;&lt;/code&gt;. All my attempts to read the &lt;code&gt;file.dat&lt;/code&gt; has failed due to memory constraints. Any help on how to use package &lt;code&gt;ff&lt;/code&gt; for this purpose?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open to other methods also.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2019-09-25T02:37:12.540" Title="Big Data and a fwf files reading with filters" Tags="|r|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8187" PostTypeId="2" ParentId="8186" CreationDate="2015-09-24T15:19:10.283" Score="2" Body="&lt;p&gt;Try the &lt;code&gt;LaF&lt;/code&gt; package. See this:&#xA;&lt;a href=&quot;https://stackoverflow.com/a/24716798/1427069&quot;&gt;https://stackoverflow.com/a/24716798/1427069&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you create a &lt;code&gt;laf&lt;/code&gt; object, you can create a data frame or data table using your row filter condition.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(LaF)&#xA;laf &amp;lt;- laf_open_fwf(&quot;foo.dat&quot;,&#xA;                    column_widths = colWidths, &#xA;                    column_types=rep(&quot;character&quot;, length(colWidths)),&#xA;                    column_names = colNames)&#xA;castData &amp;lt;- laf[RecordID == &quot;Cast&quot;, ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12328" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-09-24T15:19:10.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8188" PostTypeId="1" CreationDate="2015-09-24T16:10:21.187" Score="3" ViewCount="1430" Body="&lt;p&gt;In Sentiment Analysis, I've seen the sentiment of an entire string of words (tweets, paragraphs, sentences, etc.) computed by summing the sentiment of all of the terms in the string.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there is a monotonic relationship between summing and averaging.  So, if sentences are only ever compared to other sentences, then of course it wouldn't make a difference.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, why is it that the average is not used instead of the sum?  Especially if there are cases where strings could contain only one word, or many words, it seems to me that an average sentiment would be the way to go.  Is that not true?  &lt;/p&gt;&#xA;" OwnerUserId="10512" LastEditorUserId="31513" LastEditDate="2017-05-09T16:03:02.963" LastActivityDate="2017-05-09T16:03:02.963" Title="Why use sum and not average for sentiment analysis?" Tags="|machine-learning|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8189" PostTypeId="2" ParentId="8188" CreationDate="2015-09-24T16:44:13.290" Score="6" Body="&lt;p&gt;Consider the following two sentences:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;My awesome girlfriend bought me a delicious popsicle at the store.&#xA;0--1-------0----------0------0--0-1---------0--------0--0---0-----:2:11&#xA;&#xA;My awesome girlfriend, Joyce, drove to the grocery store to buy me a delicious Dole popsicle.&#xA;0---1------0-----------0------0-----0--0---0-------0-----0--0---0--0-1---------0----0-------:2:16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See how you can sum the values and get &lt;code&gt;2&lt;/code&gt; for both, but average the sentiments and get &lt;code&gt;2/11&lt;/code&gt; vs. &lt;code&gt;2/16&lt;/code&gt;.  There are &lt;a href=&quot;http://www.matthewjockers.net/2015/04/28/cumulative-sentiments/&quot; rel=&quot;noreferrer&quot;&gt;cases where&lt;/a&gt; people have seen better results measuring the total number of good sentiment words rather than letting zero value words cloud the picture.  In this case, I would argue that the sentences have the same sentiment and averaging artificially pulls down the sentiment of the second sentence.  In certain methods like &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;TFIDF&lt;/code&gt;&lt;/a&gt;, the word vectors are normalized against are relevant corpus, so this may change things.  In other methods, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;stop words&lt;/code&gt;&lt;/a&gt; may have already been removed, so this has less effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In twitter examples, users are limited to 140 characters, so one might consider this to be an inherent denominator which makes averaging really averages of averages.  Its very dependent on the specific application and methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without a more specific reference, it is difficult to be more specific in my answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-09-24T16:44:13.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8190" PostTypeId="1" CreationDate="2015-09-24T17:17:03.003" Score="5" ViewCount="1655" Body="&lt;p&gt;&lt;a href=&quot;http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf&quot; rel=&quot;noreferrer&quot;&gt;Factorization machines&lt;/a&gt; are a great way to model interactions between different features, in a way that is more efficient than full-blown quadratic interactions. There exist efficient implementations like libFM and Vowpal Wabbit's &lt;code&gt;--lrq&lt;/code&gt; option. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is about higher order interactions: the FM paper mentions that it is &quot;easy&quot; to extend it to higher orders (Tensor FM?), see. Eq. (5) in &lt;a href=&quot;http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf&quot; rel=&quot;noreferrer&quot;&gt;the paper&lt;/a&gt;. Are there any good implementations of 3-way factorization machines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[moved from cross-validated]&lt;/p&gt;&#xA;" OwnerUserId="3592" LastEditorUserId="3592" LastEditDate="2015-09-24T17:48:43.007" LastActivityDate="2016-12-08T12:39:31.717" Title="3-way factorization machines" Tags="|machine-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8191" PostTypeId="2" ParentId="6842" CreationDate="2015-09-24T17:59:47.897" Score="4" Body="&lt;p&gt;What you are doing is a typical example of k-fold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;XGBoost&lt;/code&gt; is just used for boosting the performance and signifies &quot;distributed gradient boosting&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, run the cross-validation step:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kfld = sklearn.cross_validation.KFold(labels.size, n_folds=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, use the train and test indices in &lt;code&gt;kfld&lt;/code&gt; for constructing the XGBoost matrix and re-scaling weights by looping over them(the indices).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very neat implementation has been given as a Kaggle example &lt;a href=&quot;https://kaggle2.blob.core.windows.net/forum-message-attachments/44825/1290/test_xgboost.py?sv=2012-02-12&amp;amp;se=2015-12-14T21%3A28%3A18Z&amp;amp;sr=b&amp;amp;sp=r&amp;amp;sig=yQJmvA%2FpUqve9EDdkF1bWAaYjEtNFmcUfjQUHaTTD7U%3D&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, cross validation is not done with the &lt;code&gt;xgboost&lt;/code&gt; package, it is done with the &lt;code&gt;cross_validation&lt;/code&gt; module of &lt;code&gt;sklearn&lt;/code&gt;, and then the gradient boosting is done on the indices of the k-fold validation variable results.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="10462" LastEditDate="2015-12-12T04:05:59.983" LastActivityDate="2015-12-12T04:05:59.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8193" PostTypeId="1" AcceptedAnswerId="8198" CreationDate="2015-09-24T22:16:03.753" Score="2" ViewCount="2102" Body="&lt;p&gt;If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) and what can be done to solve this issue or gain more stability across runs?&lt;br&gt;&#xA;Thanks for the help.&lt;/p&gt;&#xA;" OwnerUserId="9296" LastActivityDate="2015-09-26T02:36:51.623" Title="replicability / reproducibility in topic modeling (LDA)" Tags="|topic-model|lda|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8194" PostTypeId="1" AcceptedAnswerId="8197" CreationDate="2015-09-24T23:46:59.960" Score="2" ViewCount="147" Body="&lt;p&gt;I am a first year PhD student in statistics. During this year I have analyzed the scopes of interest of my scientific advisor and found them unpromising. He is majored in &lt;em&gt;mixtures with varying concentrations models&lt;/em&gt; for which I have not found any references to authoritative sources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to change my PhD theme, however, there are no other scientific advisors in my university which majored in statistics. Therefore, I have 2 questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is it possible to write at least 5 articles together with a PhD thesis without scientific adviser? If yes, what is a proper way to do this? Here I mean how to choose a theme, where ask for a help and so on.&lt;/li&gt;&#xA;&lt;li&gt;Is it possible to find a remote adviser to consult with? If yes, how and where can I find him?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also I have no much time for the search. I am interested in statistics, especially in machine learning. I would like my PhD thesis to be of practical value, not pure research one which is popular in my department. Also I have commercial experience in programming (C/C++, R, Python) if that can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="10694" LastActivityDate="2015-09-25T06:11:52.020" Title="PhD program in statistics" Tags="|machine-learning|statistics|education|" AnswerCount="1" CommentCount="0" ClosedDate="2015-09-25T13:58:05.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="8195" PostTypeId="1" CreationDate="2015-09-25T00:47:46.873" Score="8" ViewCount="1760" Body="&lt;p&gt;A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does one check whether two distribution are statistically different?&#xA;Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference?&#xA;Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?&lt;/p&gt;&#xA;" OwnerUserId="12983" LastEditorUserId="13413" LastEditDate="2015-12-17T15:08:59.803" LastActivityDate="2016-02-17T08:49:52.093" Title="Difference between training and test data distributions" Tags="|machine-learning|classification|dataset|image-classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8197" PostTypeId="2" ParentId="8194" CreationDate="2015-09-25T06:11:52.020" Score="1" Body="&lt;p&gt;Certain ingredients are needed to give you the best chance of a successful PhD. One of the important ones is that you and your supervisor have mutual interests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second important ingredient, in my opinion, is that you immerse yourself in that environment. It's important to develop a network of colleagues. It helps to spread ideas, start collaborations, get help when needed, and to explore unthought of opportunities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what you have said, I think you will be missing out on these two important ingredients if you continue in the same place or if you work remotely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is also important is what you to do &lt;em&gt;after&lt;/em&gt; the PhD. PhD is required for academic position. But I think you will be in a weak position to get to the next step (fellowships, faculty positions, etc.) if you do what you proposed. In certain industrial positions it can be looked on favourably, not necessarily for the topic you pursued, but because it says something about your personally. Basically that you can get things gone, rise to a challenge, work independently, work as a team, communicate difficult topics and can bring creativity to solving a problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice would be to find a machine learning research group and apply for PhDs. If this is not possible why not consider following the topic of your supervisor and keep machine learning as a hobby? You will become and expert in statistic and so you will find manny concepts will translate between the various statistical disciplines. But only do this if you get along with her/him, and you can see yourself studying this topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you could try a compromise? Are there applications for &quot;mixing statistics&quot; in machine learning? Can you find one? Is there an unexplored opportunity to do something new?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a side note I find it ridiculous that PhD supervisor ask the student for topics. This always leads to problems because the student doesn't really have a clue about the research field. There is room for flexibility but often this hides supervisor laziness.&lt;/p&gt;&#xA;" OwnerUserId="12987" LastActivityDate="2015-09-25T06:11:52.020" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8198" PostTypeId="2" ParentId="8193" CreationDate="2015-09-25T06:17:41.417" Score="3" Body="&lt;p&gt;LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inference for many Bayesian models is done by Markov Chain Monte Carlo. Indeed the &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; on LDA suggests that Gibbs sampling is a popular inference technique for LDA. MCMC draws random samples to provide an approximation of the posterior distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variational inference methods should typically be deterministic, but I'm not too familiar with the &lt;a href=&quot;https://www.cl.cam.ac.uk/teaching/1011/L101/ml4lp-lect8.pdf&quot; rel=&quot;nofollow&quot;&gt;VB inference&lt;/a&gt; for this particular model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also one can typically replicate runs of random algorithms by setting of random number generation seeds (if your purpose is scientific).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In either case if the results from a Bayesian model show huge variability in the the parameters of interest, it may be telling you that the model is not a good fit for the dataset, or the dataset is not big enough for the model you are fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I don't know which inference method (Gibbs, VB etc.) the backend of your software is using, so it's not possible to determine what type (if any) of randomisation is going on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For scientific purposes, you'll probably want to read up some more on Bayesian inference. Standard software (e.g. the LDA in scikits.learn) will give you a summary of the outputs of the inference (e.g. most coders just want the best assignment of docs to topics). There's more information hanging around behind the scenes which you might be able to get access to, and could be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. (roughly) for scientific applications of Gibbs sampling methods we'd typically run multiple chains, drop the first N samples generated by each, and check if the resulting samples look like they came from the same distribution. If you're concerned about dependence on seeds etc. and your backend is the Gibbs sampler you will want to check out MCMC convergence diagnostics for this model.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastEditorUserId="5303" LastEditDate="2015-09-25T10:40:51.033" LastActivityDate="2015-09-25T10:40:51.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8199" PostTypeId="1" AcceptedAnswerId="8214" CreationDate="2015-09-25T06:40:34.843" Score="3" ViewCount="214" Body="&lt;p&gt;I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing analytics, MS in learning analytics and MS in urban informatics. Does anyone know any other domain-specific data science field of study?&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-12-09T18:56:32.327" LastActivityDate="2015-12-16T17:38:10.840" Title="Domain-specific data science programs" Tags="|data|education|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8200" PostTypeId="2" ParentId="8193" CreationDate="2015-09-25T08:23:08.340" Score="1" Body="&lt;p&gt;When you are using mallet, you can fix a random seed using the command line flag &lt;code&gt;--random-seed&lt;/code&gt;. This makes your results reproducible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does not and cannot remove the fact, that different random seeds produce different topic models.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-09-25T08:23:08.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8202" PostTypeId="1" CreationDate="2015-09-25T11:00:38.053" Score="1" ViewCount="81" Body="&lt;p&gt;I have to compose a feature which summarizes the blocks area of different sections of cities. (A block is defined as the space contained by streets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could compute the arithmetic average of areas, but the data is often very skewed in the sections. I could compute the arithmetic average of log(areas), which transform the data to a more normal distribution, but the problem is that the data is skewed differently section by section.&#xA;How would you create this feature?&lt;/p&gt;&#xA;" OwnerUserId="989" LastEditorUserId="11097" LastEditDate="2015-09-25T11:29:31.290" LastActivityDate="2015-09-30T15:14:08.353" Title="Averaging on skewed distribution" Tags="|feature-extraction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8203" PostTypeId="1" AcceptedAnswerId="8204" CreationDate="2015-09-25T15:28:09.677" Score="-1" ViewCount="146" Body="&lt;p&gt;I just want to know which books, courses,videos, links,etc do you recommend me to start in machine learning, neural networks, languajes most commonly used. I want to start from zero, just in the begging of all beacuse I have not experience in this kind of algorithms but it's something that call my attention. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="12996" LastEditorUserId="843" LastEditDate="2015-10-16T17:54:23.007" LastActivityDate="2015-10-16T17:54:23.007" Title="Beggining in machine learning" Tags="|machine-learning|beginner|" AnswerCount="1" CommentCount="2" ClosedDate="2015-09-25T17:03:19.223" ContentLicense="CC BY-SA 3.0" />
  <row Id="8204" PostTypeId="2" ParentId="8203" CreationDate="2015-09-25T16:10:13.913" Score="0" Body="&lt;p&gt;Coursera is currently offering a course on Machine learning with collaboration from MIT. Many says its strongly recommended.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/learn/machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I found the below course from Edx more interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/learn/machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also provides hands on the Microsofts New exclusive Machine learning platform On Azure. &lt;/p&gt;&#xA;" OwnerUserId="12952" LastActivityDate="2015-09-25T16:10:13.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8205" PostTypeId="1" CreationDate="2015-09-25T17:42:53.690" Score="1" ViewCount="48" Body="&lt;p&gt;I am developing an application for research purposes that uses randomly generated tweets to detect emerging conceptual topics. I need to collect a large amount of them (let's say, beyond 5 million) and due to the fact that I want the application to detect all kinds of topics, I need to generate a &lt;strong&gt;set of the most general or abstract search terms&lt;/strong&gt; I can find in order to query the stream. The problem is that I can't figure out how or where should I search for these search terms. Let's put it that way: if you had a dataset which consisted of the entire knowledge of the human population and you want to generate a representative subset of it, with which search terms would you query this dataset?&#xA;Is it going to be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Technology,&#xA;People,&#xA;Politics,&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a set of the most abstract concepts which I can study?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: Of course I know that Twitter hasn't got the entire knowledge of humans, but you get the picture ;)&lt;/p&gt;&#xA;" OwnerUserId="12999" LastActivityDate="2015-09-25T17:42:53.690" Title="Generate a set of abstract search terms" Tags="|data-mining|dataset|data|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8207" PostTypeId="1" CreationDate="2015-09-25T23:04:36.723" Score="5" ViewCount="184" Body="&lt;p&gt;I would like to know whether it's possible to build a predictive model where I could define a set of rows with their attributes, and a class belonging to that set of rows, instead of having the typical model one observation - one class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm trying to do is to create a model which should be able to predict the future level of a rider in professional cycling - more likely to be used to forecast young riders' level . The input data are the results of each rider (race,category of race, type of parcours, position, and so on). The class is represented with the value (from 1 to 10) each &quot;known&quot; rider has been assigned for each skill (mountain, flat, time trial, sprint, etc.), which means that in principle, I would make one predictive model for each skill (what means for each class). For example, for mountain class I would have the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Rider   Category    Race Type   Parcours    Distance    Position    Class(Mountain)&#xA;-----------------------------------------------------------------------------&#xA;Froome  GrandTour   Stage       Mountain    185         1           10&#xA;Froome  WorldTour   Stage       MidMountain 210         25          10&#xA;Nibali  Contin.Tour Stage       Mountain    170         15          9&#xA;Nibali  WorldTour   Classic     MidMountain 245         3           9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I see here two main challenges, apart from the fact of dealing with both numerical and categorical attributes.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A rider gathers a lot of results, and each of them will result in the same class value. For example, a victory in a Grand Tour category stage from Chris Froome would be rated with a 10 in mountain, but he would also be rated with a 10 in mountain when he ended up in a bad position in a lower level stage - just as in the example. Ok, it's supposed that with thousands of rows the model (regression model probably, rather than a classification algorithm, but not sure of that either) should be able to detect the &quot;common&quot; pattern (a good performance in mountain from a good climber like Froome), so it could be overcome. I guess it'd be a matter of testing and finding the best-fitted models.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I find the predictive part the most problematic one, because what I need isn't to predict the class value for each observation (which is represented by each result), but the class value for each rider. That's why I said at the beginning of the thread that I would need to somehow treat all the results of a rider as a unique observation (something like a set or group of observations), and assign the class value to that set. A workaround could be to take all the observations of the rider, predict the class for each of them, and then calculate the average, but it doesn't look very smart.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, any idea of how to build this kind of models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="13000" LastEditorUserId="97" LastEditDate="2015-09-26T10:32:17.323" LastActivityDate="2016-02-23T13:31:16.253" Title="Predictive models with class value belonging to a set of observations" Tags="|classification|predictive-modeling|regression|feature-selection|feature-construction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8208" PostTypeId="1" CreationDate="2015-09-25T23:13:55.427" Score="2" ViewCount="3530" Body="&lt;p&gt;Using a random forest is it possible to determine which features were the driving features to classify a specific sample as class A?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I can ask which features are more important to perform classification of ANY sample, but can I ask this for a specific sample?  E.g. Why was sample 1 classified as A?  Which of its features were much more like class A than class B?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it even make sense to ask this question of a random forest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus points on how to do it with sklearn in python :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question has been answered in a crosspost here:  &lt;a href=&quot;https://stats.stackexchange.com/questions/174229/feature-importance-for-random-forest-classification-of-a-sample&quot;&gt;https://stats.stackexchange.com/questions/174229/feature-importance-for-random-forest-classification-of-a-sample&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python implementation here:  &lt;a href=&quot;http://blog.datadive.net/random-forest-interpretation-with-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.datadive.net/random-forest-interpretation-with-scikit-learn/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13002" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-05T08:46:35.237" Title="Feature importance for random forest classification of a sample" Tags="|random-forest|scikit-learn|" AnswerCount="1" CommentCount="5" ClosedDate="2017-01-06T17:58:38.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="8210" PostTypeId="2" ParentId="8193" CreationDate="2015-09-26T02:36:51.623" Score="0" Body="&lt;p&gt;The answers about Gibbs Sampling are misleading, in my opinion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for this is that Gibbs Sampling leads to &lt;em&gt;some&lt;/em&gt; error due to resampling. All Bayesian methods that use MCMC techniques face this issue, and it's really not that much of a problem, as under standard conditions, you can put bounds on your MCMC error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is not one of those standard conditions, as Latent Dirichlet Allocation leads to a multi-modal posterior! As such, standard sampling methods will typically find a single mode and sample around that single mode. The sampling techniques may well miss other modes, which may in fact have a higher posterior probability than the one found. Often, in these types of problems, several starts from different initial parameters are run and the solutions are compared. There's more advanced samplers that attempt to deal with this problem as well, such as a sampler that occasionally attempts to sample far from the current mode in the hopes of hitting another mode. Odds are that if you find a mode, it's probably a pretty good solution, even if not optimal, but that is certainly not insured!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regards to making results replicable, if you want get the same solution, you have to start from the same place, with the same random seed. &lt;/p&gt;&#xA;" OwnerUserId="13005" LastActivityDate="2015-09-26T02:36:51.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8211" PostTypeId="2" ParentId="2454" CreationDate="2015-09-26T09:52:25.717" Score="0" Body="&lt;p&gt;You can try this.&#xA;&lt;a href=&quot;https://github.com/databricks/spark-redshift&quot; rel=&quot;nofollow&quot;&gt;https://github.com/databricks/spark-redshift&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise use Rahul's answer, unload data to s3 then load it into SPARK. EMR Spark is on top of YARN and Spark context's textFile method supports &quot;s3//&quot; natively.&lt;/p&gt;&#xA;" OwnerUserId="13009" LastActivityDate="2015-09-26T09:52:25.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8212" PostTypeId="2" ParentId="8207" CreationDate="2015-09-26T10:30:28.030" Score="1" Body="&lt;p&gt;Looks like you have two problems: time-dependant predictive modeling and feature engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) Time-dependant data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Key words:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;create a model which should be able to predict &lt;strong&gt;the future level&lt;/strong&gt; of a rider in professional cycling&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;future&lt;/em&gt; Level of the Rider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That means there is &lt;em&gt;current&lt;/em&gt; Level. And Level &lt;em&gt;in the past&lt;/em&gt;. And the history of Level change in time for each Rider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem you are trying to solve is time dependant. Your source data can look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eNQKD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eNQKD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) Build a variable to predict&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second. You should build the target feature. This Rider Level (Class).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understood:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the overall rider 'level' can be a function of his 'classes' in each particular race types&lt;/li&gt;&#xA;&lt;li&gt;and... the race 'observation' class is a kind of function of &lt;code&gt;{Category, Race Type, Parcours, Distance, Position}&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So the only thing you realy need to predict is race result.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TAwV6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TAwV6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-09-26T10:44:53.643" LastActivityDate="2015-09-26T10:44:53.643" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8213" PostTypeId="1" CreationDate="2015-09-26T11:58:37.963" Score="7" ViewCount="3373" Body="&lt;p&gt;I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of features are about 100+ and due to the sequence labelling task contiguous sequence of training samples share same features values. The feature values are mostly text based and most are categorical with long text based values  with large cardinality i.e many features may have huge number of possible values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The available RAM IS 32gb with 8 core CPU. On a small scale I tried Multinomial NB and linear SGD with sparse matrices which are extremely sparse. Used the scikit learns Dictvectorizer to vectorize the feature dictionary. Also will pandas dataframes help to optimize the overall configuration?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the scale of data involved I have only used about 20% of the data. And the features itself are 56GB of data from the 500MB input data. For the restricted training data, The precision too is very low does not reach above 12% with a very low  recall too(1-2%). Have gone through vowpal wabbit and  found the SEARN task but it seems that it is now no longer available in the latest version .&lt;/p&gt;&#xA;" OwnerUserId="13011" LastEditorUserId="13011" LastEditDate="2015-09-29T06:02:01.130" LastActivityDate="2018-09-25T09:26:36.703" Title="Classifier and Technique to use for large number of categories" Tags="|machine-learning|bigdata|dataset|scikit-learn|categorical-data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8214" PostTypeId="2" ParentId="8199" CreationDate="2015-09-27T05:24:24.880" Score="2" Body="&lt;p&gt;Unsurprisingly most programs in data science focus mostly on business or marketing applications. I too would like to see more programs focused on Biotechnology, Gaming or Energy for example but I think that in the future those kind of programs will become more and more available.&#xA;However there are a few I know of that offer a somewhat different approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding health informatics there are many offers where you can find anything between Public health to Computational Biology and Quantitative Genetics. Take a look at the full list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mastersindatascience.org/specialties/health-informatics/&quot; rel=&quot;nofollow&quot;&gt;http://www.mastersindatascience.org/specialties/health-informatics/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On that list there are a few certification programs but there also are full fledged programs that lead to a M.Sc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are more interested in Marketing Analytics you can find the same list but for marketing related programs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mastersindatascience.org/specialties/marketing-analytics/&quot; rel=&quot;nofollow&quot;&gt;http://www.mastersindatascience.org/specialties/marketing-analytics/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;other options like Information Systems are also present, just browse that site and you might find something that interests you.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-09-27T05:24:24.880" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8216" PostTypeId="2" ParentId="8138" CreationDate="2015-09-27T09:43:24.940" Score="1" Body="&lt;p&gt;The script below uses LSA with transformed IDF to cut off the parameters from templates. The idea is, that all terms with IDF higher that some threshold are considered as parameters and their frequency is reset to zero.  The threshold can be approximated with the average template occurrence in the corpus.&#xA;Eliminating the parameters, the distance of the records with the same template is zero.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(tm)&#xA; library(lsa)&#xA; df &amp;lt;- data.frame(TEMPLATE = c(rep(&quot;A&quot;,3),rep(&quot;B&quot;,3),rep(&quot;C&quot;,3)),&#xA; TEXT = c(&#xA; paste(&quot;Temperature today is&quot;,c(28,24,20),&quot;centigrades&quot;),&#xA; paste(&quot;Temperature today is&quot;,c(82,75,68),&quot;Fahrenheit&quot;),&#xA; paste(&quot;Her eyes are &quot;,c(&quot;blue&quot;,&quot;black&quot;,&quot;green&quot;), &quot;and hair&quot;,c(&quot;grey&quot;,&quot;brown&quot;,&quot;white&quot;))) , stringsAsFactors=FALSE)&#xA;&amp;gt; df     &#xA;   TEMPLATE                                TEXT&#xA; 1        A Temperature today is 28 centigrades&#xA; 2        A Temperature today is 24 centigrades&#xA; 3        A Temperature today is 20 centigrades&#xA; 4        B  Temperature today is 82 Fahrenheit&#xA; 5        B  Temperature today is 75 Fahrenheit&#xA; 6        B  Temperature today is 68 Fahrenheit&#xA; 7        C    Her eyes are  blue and hair grey&#xA; 8        C  Her eyes are  black and hair brown&#xA; 9        C  Her eyes are  green and hair white&#xA;&#xA; corpus &amp;lt;- Corpus(VectorSource(df$TEXT))&#xA; td &amp;lt;- as.matrix(TermDocumentMatrix(corpus,control=list(wordLengths = c(1, Inf)) ))&#xA;&#xA; &amp;gt; td             Docs&#xA; Terms         1 2 3 4 5 6 7 8 9&#xA;   20          0 0 1 0 0 0 0 0 0&#xA;   24          0 1 0 0 0 0 0 0 0&#xA;   28          1 0 0 0 0 0 0 0 0&#xA;   68          0 0 0 0 0 1 0 0 0&#xA;   75          0 0 0 0 1 0 0 0 0&#xA;   82          0 0 0 1 0 0 0 0 0&#xA;   and         0 0 0 0 0 0 1 1 1&#xA;   are         0 0 0 0 0 0 1 1 1&#xA;   black       0 0 0 0 0 0 0 1 0&#xA;   blue        0 0 0 0 0 0 1 0 0&#xA;   brown       0 0 0 0 0 0 0 1 0&#xA;   centigrades 1 1 1 0 0 0 0 0 0&#xA;   eyes        0 0 0 0 0 0 1 1 1&#xA;   fahrenheit  0 0 0 1 1 1 0 0 0&#xA;   green       0 0 0 0 0 0 0 0 1&#xA;   grey        0 0 0 0 0 0 1 0 0&#xA;   hair        0 0 0 0 0 0 1 1 1&#xA;   her         0 0 0 0 0 0 1 1 1&#xA;   is          1 1 1 1 1 1 0 0 0&#xA;   temperature 1 1 1 1 1 1 0 0 0&#xA;   today       1 1 1 1 1 1 0 0 0&#xA;   white       0 0 0 0 0 0 0 0 1&#xA;&#xA; ## supress terms with idf higher than template frequency&#xA; ## those terms are considered as parameters&#xA; template_freq &amp;lt;- 3&#xA; tdw &amp;lt;- lw_bintf(td) * ifelse(gw_idf(td)&amp;gt; template_freq,0, gw_idf(td))&#xA; dist &amp;lt;- dist(t(as.matrix(tdw)))&#xA;&#xA; &amp;gt; dist&#xA;          1        2        3        4        5        6        7        8&#xA; 2 0.000000                                                               &#xA; 3 0.000000 0.000000                                                      &#xA; 4 3.655689 3.655689 3.655689                                             &#xA; 5 3.655689 3.655689 3.655689 0.000000                                    &#xA; 6 3.655689 3.655689 3.655689 0.000000 0.000000                           &#xA; 7 6.901341 6.901341 6.901341 6.901341 6.901341 6.901341                  &#xA; 8 6.901341 6.901341 6.901341 6.901341 6.901341 6.901341 0.000000         &#xA; 9 6.901341 6.901341 6.901341 6.901341 6.901341 6.901341 0.000000 0.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The distance matrix clearly shows, that the records 1,2,3 are from the same template (distance = 0, with the synthetic data; in a real case some small threshold should be used). Same is valid for records 4,5,6  and 7,8,9.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2015-09-27T09:51:41.237" LastActivityDate="2015-09-27T09:51:41.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8219" PostTypeId="2" ParentId="8090" CreationDate="2015-09-27T16:18:25.053" Score="2" Body="&lt;p&gt;There are alternatives out there. Using cloud capabilities for computing is helpful for many different types of problems and I've been using the cloud for a couple years now. There are moments where you need more computational capabilities, for instance if you needed more ram to process data or if you needed GPUs for neural nets computation. The only drawback is the price of some of these services.&#xA;The quickest ways to use is just sign up and go through the tutorials on &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Amazon Machine Learning&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.aws.amazon.com/machine-learning/latest/dg/tutorial.html&quot; rel=&quot;nofollow&quot;&gt;https://docs.aws.amazon.com/machine-learning/latest/dg/tutorial.html&lt;/a&gt;&#xA;or &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Azure Machine Learning&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://azure.microsoft.com/en-us/services/machine-learning/&quot; rel=&quot;nofollow&quot;&gt;https://azure.microsoft.com/en-us/services/machine-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;RStudio on an Amazon instance&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Another alternative is to set up RStudio on an Amazon instance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.louisaslett.com/RStudio_AMI/&quot; rel=&quot;nofollow&quot;&gt;http://www.louisaslett.com/RStudio_AMI/&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://www.louisaslett.com/RStudio_AMI/video_guide.html&quot; rel=&quot;nofollow&quot;&gt;http://www.louisaslett.com/RStudio_AMI/video_guide.html&lt;/a&gt;&lt;br&gt;&#xA;This tutorial will teach you how to connect to an Amazon instance running R and showing the GUI on your browser.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;iPython HTML Notebook on Amazon's AWS&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The same can be done with python using an IPython notebook to interact with the remote instance.&#xA;&lt;a href=&quot;https://gist.github.com/iamatypeofwalrus/5183133&quot; rel=&quot;nofollow&quot;&gt;https://gist.github.com/iamatypeofwalrus/5183133&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Spark on the cloud&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If you know Apache Spark you can also use &#xA;&lt;a href=&quot;https://databricks.com/&quot; rel=&quot;nofollow&quot;&gt;https://databricks.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Advanced topic: Deep Learning on Amazon EC2 GPU with Python and nolearn&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Setup a GPU Amazon instance to run deep learning algorithms&#xA;&lt;a href=&quot;http://www.pyimagesearch.com/2014/10/13/deep-learning-amazon-ec2-gpu-python-nolearn/&quot; rel=&quot;nofollow&quot;&gt;http://www.pyimagesearch.com/2014/10/13/deep-learning-amazon-ec2-gpu-python-nolearn/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-09-27T16:18:25.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8223" PostTypeId="1" CreationDate="2015-09-28T20:04:57.783" Score="1" ViewCount="25" Body="&lt;p&gt;I'm working on a object detection problem, using convolutional neural networks which have a lot in common with how the human visual cortex works. I've previously seen remarks on tasks in computer vision like &quot;if a human can do it in x milliseconds, it can easily be solved by a computer&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of setting up an experiment where I flash an image for a set amount of time and the subject records whether (s)he detected the object or not. I would then have multiple subjects, a large set of images, and a varying flash length. Thus I could kind of quantify how hard my computer vision task is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether you know of any research, papers, books or other resources on this subject? Any resources specifically linked to neural networks and deep learning would be especially helpful.&lt;/p&gt;&#xA;" OwnerUserId="13054" LastActivityDate="2015-09-28T20:04:57.783" Title="Experiments on time spent by the visual cortex" Tags="|experiments|computer-vision|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8224" PostTypeId="2" ParentId="6939" CreationDate="2015-09-28T21:15:42.943" Score="1" Body="&lt;p&gt;I guess you are not limited to these 100 samples. Generate more, and let each 5th be negative. Then reduce number of positives by random removing 4/5 of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And check this out &lt;a href=&quot;https://datascience.stackexchange.com/questions/1107/quick-guide-into-training-highly-imbalanced-data-sets&quot;&gt;Training imbalanced data set&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is small quantity, you'd better have 50:50 negative vs positive.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-09-28T21:15:42.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8225" PostTypeId="1" AcceptedAnswerId="8228" CreationDate="2015-09-28T21:17:21.150" Score="4" ViewCount="1812" Body="&lt;p&gt;I have a system that manages equipments. When these equipments are faulty, they will be serviced. Imagine my dataset looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID&#xA;Type&#xA;# of times serviced&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|ID| Type       | #serviced |&#xA;|1 | iphone     | 1         |&#xA;|2 | iphone     | 0         |&#xA;|3 | android    | 1         |&#xA;|4 | android    | 0         |&#xA;|5 | blackberry | 0         |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I want to do is I want to predict &quot;of all the equipments that have not been serviced, which ones are likely to be serviced&quot; ? (ie) identify &quot;at risk&quot; equipments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is my training data will be #serviced &gt; 0. Any #serviced=0 will not be frozen and dont seem to be valid candidates to include in training. (ie) When it fails, it will be serviced hence the count will go up.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this a supervised or unsupervised problem ? (supervised because I have serviced and not-serviced labels, unsupervised because I want to cluster not-serviced with serviced and there by identify at-risk equipments)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What data should I include in training ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example is obviously simplified. In reality I have more features that describe the equipment.&lt;/p&gt;&#xA;" OwnerUserId="13056" LastEditorUserId="97" LastEditDate="2015-09-28T21:53:40.140" LastActivityDate="2015-09-30T02:06:37.910" Title="Equipment failure prediction" Tags="|machine-learning|classification|predictive-modeling|supervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8226" PostTypeId="2" ParentId="8225" CreationDate="2015-09-28T21:43:27.583" Score="0" Body="&lt;p&gt;This is supervised learning problem. &lt;code&gt;Type&lt;/code&gt; is a predictor. &lt;code&gt;#serviced&lt;/code&gt; classifier is target variable. Model is trained on samples set you already have. Best guess is that any model will not have substantual predictive ability. &lt;code&gt;Type&lt;/code&gt; is not enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try including more factors (predictors) into the model. Like &lt;code&gt;years_being_in_usage&lt;/code&gt;, &lt;code&gt;equipment_model&lt;/code&gt;, &lt;code&gt;have_been_in_service_before&lt;/code&gt; and so on. The more you get, the better model you can train.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="97" LastEditDate="2015-09-29T21:48:29.687" LastActivityDate="2015-09-29T21:48:29.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8227" PostTypeId="1" AcceptedAnswerId="8234" CreationDate="2015-09-28T22:39:45.037" Score="0" ViewCount="72" Body="&lt;p&gt;I'm looking for an optimal way to search a large body of text for a combination of words that resemble any CITY, STATE combination I have in a separate CITY-STATE database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My only idea would be to do a separate search against the body of text for each CITY, STATE in the database, but that would require a lot of time considering the amount of CITY, STATE combinations the database has in it. The desired result from this query would be to pull a single CITY, STATE for each body of text I am analyzing to tell the geographical side of the story for this data subset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone know of an optimal way/process to do such a query?&lt;/p&gt;&#xA;" OwnerUserId="13057" LastActivityDate="2015-09-29T13:23:16.520" Title="Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database)" Tags="|databases|search|parsing|data-indexing-techniques|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8228" PostTypeId="2" ParentId="8225" CreationDate="2015-09-28T23:40:09.620" Score="0" Body="&lt;p&gt;You should include data when the phone was serviced to create a survival model. These models are commonly used in reliability engineering as well as treatment efficacy. For reliability engineering it is very common to fit your data to a Weibull distribution. Even aircraft manufacturers consider the model to be reliable after calibrating with three to five data points. I can highly recommend the R package 'flexsurv' package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You cannot use typical linear or logistic regressions since some phones in your population will leave your observation period without ever being serviced. Survival models allow for this sort of missing information (this is called censoring). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically you would have the following data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|ID| Type       | serviced  | # months_since_purchase&#xA;|1 | iphone     | 1         | 12&#xA;|2 | iphone     | 0         | 15&#xA;|3 | android    | 1         | 2&#xA;|4 | android    | 0         | 10&#xA;|5 | blackberry | 0         | 5.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With that data you could create the following model in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(survival)&#xA;model &amp;lt;- survfit(Surv(months_since_purchase, serviced) ~ strata(Type) +&#xA; cluster(ID), data = phone_repairs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;survfit.formula&lt;/code&gt; &lt;code&gt;Surv(months_since_purchase, serviced) ~ strata(Type) + cluster(ID)&lt;/code&gt;&#xA;indicates that &lt;code&gt;months_since_purchase&lt;/code&gt; is the time at which an observation was&#xA;made, &lt;code&gt;serviced&lt;/code&gt; is 1 if the phone was serviced and 0 otherwise, &lt;code&gt;strata(Type)&lt;/code&gt;&#xA;will make sure that you create a different survival model for each phone,&#xA;&lt;code&gt;cluster(ID)&lt;/code&gt; will make sure that events relating to the same ID are considered&#xA;as a cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could extend this model with Joint Models such as &lt;code&gt;JM&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2015-09-30T02:06:37.910" LastActivityDate="2015-09-30T02:06:37.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8230" PostTypeId="1" CreationDate="2015-09-29T08:00:56.840" Score="2" ViewCount="415" Body="&lt;p&gt;I have a data set with keywords describing paragraphs in a car manual and the actual paragraphs. I want to rank those paragraphs by that keyword using PageRank algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I rank these paragraphs in relevance to the keywords?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The keywords are given an ID number while the paragraphs in the manual are texts. Let's say there is a keyword with ID number 89 and I have about 15 paragraphs that the keyword describes. How would the paragraphs be ranked in relevance to the keyword? Can I even use PageRank for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand for ranking websites, backlinks are used to sort the websites into ranks but for paragraphs of texts, how would I link and rank them? &lt;/p&gt;&#xA;" OwnerUserId="13067" LastEditorUserId="13067" LastEditDate="2015-09-30T00:00:23.577" LastActivityDate="2016-05-28T23:14:26.520" Title="How could PageRank be used to rank paragraphs in relevance to keywords?" Tags="|data-mining|text-mining|ranking|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8231" PostTypeId="1" AcceptedAnswerId="8232" CreationDate="2015-09-29T08:58:04.893" Score="1" ViewCount="66" Body="&lt;p&gt;Suppose I am using Neural Network for a 2 class classification. After training the network with the training set, I want to predict the class label of a dataset with no class label. Now with retraining, the same dataset gives different result. For example in one training session, a sample gave the output of it belonging to class 1 while in the other session it gave the output of it belonging to class 2. Then which value should be taken as the correct one?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2015-09-29T17:01:59.077" Title="Which value of output should be taken in multiple sessions of training Neural Network" Tags="|classification|neural-network|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8232" PostTypeId="2" ParentId="8231" CreationDate="2015-09-29T09:48:38.597" Score="2" Body="&lt;p&gt;This is normal behaviour of most classifiers. You are not guaranteed 100% accuracy in machine learning, and a direct consequence is that classifiers make mistakes. Different classifiers, even if trained on the same data, can make different mistakes. Neural networks with different starting weights will often converge to slightly different results each time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, perhaps in your problem the classification is an artificial construct over some spectrum (e.g. &quot;car&quot; vs &quot;van&quot; or &quot;safe&quot; vs &quot;dangerous&quot;) in which case the mistake in one case is entirely reasonable and expected?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should use the value from the classifier that you trust the most. To establish which one that is, use cross-validation on a hold-out set (where you know the true labels), and use the classifier with the best accuracy, or other metric, such as logloss or area under ROC. Which metric you should prefer depends on the nature of your problem, and the consequences of making a mistake.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could look at averaging the class probabilities to determine the best prediction - perhaps one classifier is really confident in the class assignment, and the other is not, so an average will go with the first classifier. Some kind of model aggregation will often boost accuracy, and is common in e.g. Kaggle competitions when you want the highest possible score and don't mind the extra effort and cost. However, if you want to use aggregation to solve your problem, again you should test your assumptions using validation and a suitable metric so you know whether or not it is really an improvement. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-09-29T17:01:59.077" LastActivityDate="2015-09-29T17:01:59.077" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8233" PostTypeId="2" ParentId="8213" CreationDate="2015-09-29T10:06:26.883" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not familiar with the task you are solving, but have you considered a lazy distance-based learner like k-nn and its variations? It could handle an arbitrary number of categories, learn incrementally and you could keep your model small-ish by using one of the case base maintenance algorithms out there. As for learning new categories, you could use an outlier detection algorithm to see when a new test example doesn't fit in its category (i.e. when it significantly reduces its intra-class consistency).&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-29T10:06:26.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8234" PostTypeId="2" ParentId="8227" CreationDate="2015-09-29T13:23:16.520" Score="0" Body="&lt;p&gt;The only thing I can see would be to separate both city and state lists and treat the problem as an automaton: parse your text, run through the n-grams, whenever you detect a CITY token (meaning a n-gram present in your list of cities or close to it in a similarity sense, as there might be misspellings) then look for a STATE token in its neighbourhood (similarly by looking into a list of states, using an edit distance metric to allow for misspellings). If you find one, then you can tag your text with that geographical location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, allowing for misspellings will bring some false positives but you could easily bypass that by doing a quick lookup through your corpus to see that &quot;SALAMI, OREGANO&quot; is different from &quot;SALEM, OREGON&quot; (because the frequency of the latter will hopefully be higher than the former)&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-29T13:23:16.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8235" PostTypeId="2" ParentId="8028" CreationDate="2015-09-29T17:35:34.363" Score="2" Body="&lt;p&gt;It seems computer science programs have not changed their curriculum. They just added some elective courses for data science or probably adding a concentration to let students reinforce their skills on data science and big data.  &lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-09-29T17:35:34.363" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8236" PostTypeId="1" CreationDate="2015-09-29T19:33:31.053" Score="4" ViewCount="145" Body="&lt;p&gt;I am working on a use case, and I'm unsure of the best way to proceed: in order to analyze the behavior of users of a web-based music application, we retain all songs each has played since 2009. We store this information in flat files, each containing the songs played in a day. Each file contains 50M of lines, and we have  19M users. Our entire song catalogue consists of 35M tracks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The format of these files is as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;id-user | country | id-artist | id-track&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: I would like to represent each user by the songs he or she has played; this profile would be used by the production site. Does anyone have a suggestion for what would be the best way to process the whole chain?&lt;/p&gt;&#xA;" OwnerUserId="8088" LastEditorUserId="13727" LastEditDate="2016-02-02T10:42:09.230" LastActivityDate="2016-02-02T10:42:09.230" Title="Approach to creating a user profile in music web application" Tags="|feature-construction|descriptive-statistics|aggregation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8237" PostTypeId="2" ParentId="8236" CreationDate="2015-09-29T21:32:45.653" Score="3" Body="&lt;p&gt;The first question is: what to do you want to see in user profile?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Top-10 tracks, top-10 artist by user?&lt;/li&gt;&#xA;&lt;li&gt;How many tracks/artists a user listens to in a day on average (may be, in last month)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;May be you want to get some general information related to the whole user base:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which artist/track is the most popular among users from different countries (top-N of them)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The second: you store and want to make aggregates for millions of records. It's not a text file deal. Make a database. Create a table &lt;code&gt;id-user | country | id-artist | id-track&lt;/code&gt;. Create another tables with some aggregates from #1, update it regularly and diplay on the front end.&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2015-09-29T21:32:45.653" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8238" PostTypeId="2" ParentId="8202" CreationDate="2015-09-30T00:57:13.990" Score="2" Body="&lt;p&gt;You could use the median. That protects against the &quot;Bill Gates walks into a bar, making everyone there a millionaire, on average&quot; effect. It's used with &lt;a href=&quot;http://www.ssa.gov/OACT/COLA/central.html&quot; rel=&quot;nofollow&quot;&gt;wage distributions&lt;/a&gt; for exactly the reason that they tend to be very skewed.&lt;/p&gt;&#xA;" OwnerUserId="12851" LastActivityDate="2015-09-30T00:57:13.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8239" PostTypeId="2" ParentId="8213" CreationDate="2015-09-30T01:55:58.737" Score="2" Body="&lt;p&gt;This sounds like a pretty gnarly problem, so a lot of finesse will be needed to solve it.  @Jérémie Clos has some good points, but I wanted to add some more general thoughts...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the size of your problem, you might want to think about a scalable framework like &lt;a href=&quot;http://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout&lt;/a&gt; or &lt;a href=&quot;http://h2o.ai/&quot; rel=&quot;nofollow&quot;&gt;H2O&lt;/a&gt; rather than scikit-learn, which is an awesome shared memory library, but does have &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/neighbors/plot_approximate_nearest_neighbors_scalability.html&quot; rel=&quot;nofollow&quot;&gt;scalability limitations&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a categorical feature with cardinality of 5000 and 80 million training cases, a uniform distribution will only give you 16,000 training cases for each positive result.  However, &lt;strong&gt;the distribution probably isn't uniform&lt;/strong&gt;.  I suggest that you eliminate the least common categories until you have thrown out 5% of the data and assess the cardinality of the remaining 95%.  This may give you a better clue about how to proceed i.e. what percent of the cardinality remains? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow&quot;&gt;multiclass classification algorithms&lt;/a&gt; are just lots of binary classifiers that are combined to produce a composite result, so will essentially do the work for you of splitting up the problem, binarizing the data (aka one-hot encoding), and training the classifiers, so you may not need to explicitly perform this splitting step.  Two exceptions to this, i.e. classifiers that can handle the multi-class problem intrinsically are &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow&quot;&gt;decision trees&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;random forests&lt;/a&gt; (&lt;a href=&quot;https://mahout.apache.org/users/classification/partial-implementation.html&quot; rel=&quot;nofollow&quot;&gt;Mahout version&lt;/a&gt;). I would try both of these and hold them as benchmarks moving forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of handling the addition of categories as data is added to the system, you will have to retrain sometimes.  This is sometimes referred to as the &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3732997/&quot; rel=&quot;nofollow&quot;&gt;stability-plasticity dilemma&lt;/a&gt;.  The best way to handle this is probably only to make predictions on categories that comprise some threshold of the data and throw out the rest as outlier categories that are not powered well enough to predict on e.g. similar to the sanity check I suggest above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-09-30T01:55:58.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8240" PostTypeId="1" AcceptedAnswerId="8659" CreationDate="2015-09-30T07:13:18.937" Score="1" ViewCount="180" Body="&lt;p&gt;I have many different strings of text. These strings of text are labels for particular things. But these labels are sloppy, sometimes one label is used for many different things. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Brown foxes edition 1999 series 1-6 EDI&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Light [old] seasons 1,2,3,4 other gibberish&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to answer the question: &quot;If the label contains a series, does that serie contain the value N?&quot; For the examples above 6 would be included in the first one, but not the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I thought of using regexes but that quickly grew out of hand. Digits appear everywhere, people can get very creative with separators and the location of the series in the label is not fixed.&#xA;There are many different ways the labels denote series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can do, however, &lt;em&gt;generate&lt;/em&gt; labels with series. I'll just grab a bunch of separators, a start and an end digit and iterate. This gives me a nice labeled training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naive Bayes comes to mind for this problem but I'm not sure what good features would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me try to clarify. &#xA;Given a label and given a number determine if that number is contained within that label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Given I'm looking for season 2 (the number). Does &quot;Pioneer One 2011 seasons 1-3&quot; contain season 2?&lt;/p&gt;&#xA;" OwnerUserId="12321" LastEditorUserId="12321" LastEditDate="2015-09-30T11:02:28.990" LastActivityDate="2015-10-30T12:35:16.940" Title="Detecting if a sentence contains a numeric series" Tags="|nlp|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8241" PostTypeId="2" ParentId="8240" CreationDate="2015-09-30T08:31:50.823" Score="0" Body="&lt;p&gt;I think your question is badly formulated, but if I understand it you are trying to separate numbers that are &quot;good&quot; numbers (here the number of the tv show season) from &quot;bad&quot; numbers (for example if the text contained the year &quot;1996&quot; and you were looking for &quot;6&quot;, or a 6 in any other context).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naive Bayes might do the trick, maybe using the neighbourhood of your number as a feature set with a predetermined window? I.e. take the X tokens on both sides of your number as they appear in the text and use them as your feature set. You could even enrich it with their part-of-speech and other contextual evidence.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-30T08:31:50.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8242" PostTypeId="1" CreationDate="2015-09-30T09:34:45.673" Score="0" ViewCount="111" Body="&lt;p&gt;I have some question concern similarity measure and i need your help (Iam new in statistics)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose that we have a matrix M where M(i, j) is the similarity measure between user i and user j .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question :&#xA;1- what similarity measure choose and why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- What is the maximum size of the matrix? What are its properties? How would you choose the store?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your help&#xA;bests&lt;/p&gt;&#xA;" OwnerUserId="8088" LastActivityDate="2015-09-30T10:28:10.390" Title="similarity measure" Tags="|statistics|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8243" PostTypeId="2" ParentId="8242" CreationDate="2015-09-30T10:28:10.390" Score="2" Body="&lt;p&gt;The matrix will be mxm (assuming you have m items) but you don't need to fill it entirely, since a similarity measure should ideally be symmetric (sim(a,b) = sim(b,a)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to choosing one, it depends entirely on what you store and what you plan to do with it. Assuming you are storing vectors of objects, some popular simple ones are &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow&quot;&gt;Jaccard similarity&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow&quot;&gt;Cosine similarity&lt;/a&gt;. &lt;a href=&quot;http://mines.humanoriented.com/classes/2010/fall/csci568/portfolio_exports/mvoget/similarity/similarity.html&quot; rel=&quot;nofollow&quot;&gt;This page&lt;/a&gt; has a bit more information on them (along with some other similarity measures). I would encourage you to search for books and research papers in the area that you are exploring and see what common similarity metrics are used by the community.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-09-30T10:28:10.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8244" PostTypeId="1" AcceptedAnswerId="8252" CreationDate="2015-09-30T10:43:15.157" Score="28" ViewCount="10751" Body="&lt;p&gt;What are some of the advantages of columnar data-stores which make them more suitable for data science and analytics?&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-04-12T19:15:16.777" LastActivityDate="2017-04-12T19:15:16.777" Title="What makes columnar databases suitable for data science?" Tags="|databases|tools|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8245" PostTypeId="1" AcceptedAnswerId="8351" CreationDate="2015-09-30T12:30:30.290" Score="1" ViewCount="126" Body="&lt;p&gt;I have devices on which I have time series data of one continuous variable. I have to evaluate the relation between the profile of that variable on those devices and &quot;events&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those events are given in terms of occurrences on a time period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first intention is to make clusters of similar behavior of that variable and compare those clusters with the low/middle/high events rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about doing a K-means with the min, max, quartilles, mean, normal q-q p value, Kurstosis, etc. as dimensions, but I don't think it's a good idea because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Those dimensions are not independant&lt;/li&gt;&#xA;&lt;li&gt;It's &quot;losing&quot; data and so potentially losing classification potential&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Do you have some suggestions to group similar devices together?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, do you have other ideas to establish that relationship?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Context:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;python3 with the scipy stack&lt;/li&gt;&#xA;&lt;li&gt;~ 3000 devices and hundreds of thousands of data per day; 5 months to consider&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="29575" LastEditDate="2017-12-18T16:15:42.357" LastActivityDate="2017-12-18T16:15:42.357" Title="Devices behavior in one continuous variable vs events rate" Tags="|machine-learning|clustering|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8246" PostTypeId="1" AcceptedAnswerId="8260" CreationDate="2015-09-30T13:04:37.247" Score="5" ViewCount="1411" Body="&lt;p&gt;What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across a &lt;a href=&quot;http://matlab.izmiran.ru/help/toolbox/nnet/backpr59.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;resource&lt;/a&gt;, but was unable to understand the difference between the two methods. It has mentioned in the procedure that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the next search direction is determined so that it is conjugate to previous search directions.&lt;br&gt;&#xA;  &lt;br&gt; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this mean? Also, what is line search mentioned in the web page?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please explain it with the help of a diagram?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="29169" LastEditDate="2019-06-08T03:09:04.587" LastActivityDate="2019-06-08T03:09:04.587" Title="What is conjugate gradient descent?" Tags="|neural-network|gradient-descent|supervised-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="8247" PostTypeId="1" CreationDate="2015-09-30T13:53:34.730" Score="4" ViewCount="1396" Body="&lt;p&gt;I am looking  to find some resources about what I wan't to do :I wan't to make some GUI of my machine learning models and  finally deploy them as a web  app.I find R Shiny to be somehow  ok , but it limits me only to use R,and I have also Python things.Looking at the azure ml studio  it brings  somekind of workflow  like in Disco and ProM from process mining, but I am not  sure  how much pain will be to feed everything  like ensembles/grid search etc to those  flow components,also the full paid  version is  very expensive.&#xA;Trying to do a web application  from scratch could also not be ok  cause it will be a lot  of pain  when I will modify my  models.&#xA;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2020-11-03T18:19:52.607" Title="Deploying machine learning modules" Tags="|python|scikit-learn|data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8248" PostTypeId="2" ParentId="8202" CreationDate="2015-09-30T15:14:08.353" Score="0" Body="&lt;p&gt;Alternatively you could consider the distribution of the blocks areas per section and use the most popular value (the top peak of the distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is possible to meanifully cluster the distribution curves, you may even define the feature as a factor - the nearest centroid of the cluster of distribution curves. &lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-09-30T15:14:08.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8250" PostTypeId="2" ParentId="8244" CreationDate="2015-09-30T16:00:00.923" Score="3" Body="&lt;p&gt;I haven't used a columnar database, but I've used an open source columnar file format called Parquet, and I think the benefits are probably the same -- faster processing of data when you only need to query a small subset of a large number of columns. I had a query running on about 50 terabytes of Avro files (a row oriented file format) with 673 columns that took about an hour and a half on a 140 node Hadoop cluster. With Parquet, the same query took about 22 minutes because I only needed 5 columns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had a small number of columns or were using a large proportion of your columns, I don't think a columnar database would make much of a difference vs a row oriented one because you would still have to basically scan all of your data. I believe columnar databases store columns separately whereas row oriented databases store rows separately. Your query will be faster any time you're able to read less data from disk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Column-oriented_DBMS&quot; rel=&quot;nofollow&quot;&gt;This link&lt;/a&gt; explains more of the details.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="21" LastEditDate="2015-10-18T12:38:37.760" LastActivityDate="2015-10-18T12:38:37.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8251" PostTypeId="2" ParentId="8247" CreationDate="2015-09-30T16:47:50.043" Score="4" Body="&lt;p&gt;Correct me if I'm wrong here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've already written the code that test/trains the model and you're looking for a way to put that model on the web somewhere so users can interact with it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a couple of way's you can go about that, but it depends on what you need out of the final product. Is this something clients will be using? Friends? Just you? &lt;a href=&quot;http://flask.pocoo.org/&quot; rel=&quot;nofollow&quot;&gt;Flask&lt;/a&gt; allows you to prototype quick web apps and still maintain a good amount of control over what the finished product looks like. &lt;a href=&quot;https://github.com/adamhajari/spyre&quot; rel=&quot;nofollow&quot;&gt;Spyre&lt;/a&gt; is an amazing way to make a web app at near light speed, although it may not be suited well to your use case, depending on what you're doing. It's a little harder to make unique-looking things with Spyre, but I always suggest it to people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At any rate, the standard thing to do here is to &lt;a href=&quot;https://docs.python.org/2/library/pickle.html&quot; rel=&quot;nofollow&quot;&gt;Pickle&lt;/a&gt; your model after it's trained and then load it to memory in a web app. After that you can call the &lt;code&gt;model.predict(X)&lt;/code&gt; on an &lt;code&gt;X&lt;/code&gt; that is sent to your program by a user. &lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2015-09-30T16:47:50.043" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8252" PostTypeId="2" ParentId="8244" CreationDate="2015-09-30T17:41:35.450" Score="30" Body="&lt;p&gt;A column-oriented database (=columnar data-store) stores the data of a table column by column on the disk, while a row-oriented database stores the data of a table row by row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two main advantages of using a column-oriented database in comparison&#xA;with a row-oriented database. The first advantage relates to the amount of data one’s&#xA;need to read in case we perform an operation on just a few features. Consider a simple&#xA;query:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT correlation(feature2, feature5)&#xA;FROM records&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A traditional executor would read the entire table (i.e. all the features):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4EZHo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4EZHo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, using our column-based approach we just have to read the columns&#xA;which are interested in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nGTKQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nGTKQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second advantage, which is also very important for large databases, is that column-based storage allows better compression, since the data in&#xA;one specific column is indeed homogeneous than across all the columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main drawback of a column-oriented approach is that manipulating (lookup, update or delete) an entire given row is inefficient. However the situation should&#xA;occur rarely in databases for analytics (“warehousing”),&#xA;which means most operations are read-only, rarely read many attributes in the same&#xA;table and writes are only appends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some RDMS offer a column-oriented storage engine option. For example, PostgreSQL&#xA;has natively no option to store tables in a column-based fashion, but Greenplum has&#xA;created a closed-source one (DBMS2, 2009). Interestingly, Greenplum is also behind&#xA;the open-source library for scalable in-database analytics, MADlib (Hellerstein et al.,&#xA;2012), which is no coincidence. More recently, CitusDB, a startup working on high speed, analytic database, released their own open-source columnar store extension for&#xA;PostgreSQL, CSTORE (Miller, 2014). Google’s system for large scale machine learning&#xA;Sibyl also uses column-oriented data format (Chandra et al., 2010). This trend&#xA;reflects the growing interest around column-oriented storage for large-scale analytics.&#xA;Stonebraker et al. (2005) further discuss the advantages of column-oriented DBMS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two concrete use cases: &lt;a href=&quot;http://qr.ae/RoQxgp&quot; rel=&quot;noreferrer&quot;&gt;How are most datasets for large-scale machine learning stored?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(most of the answer comes from Appendix C of: &lt;a href=&quot;http://francky.me/doc/2015beatdbwaveletsgp.pdf&quot; rel=&quot;noreferrer&quot;&gt;BeatDB: An end-to-end approach to unveil saliencies from massive signal data sets. Franck Dernoncourt, S.M, thesis, MIT Dept of EECS&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-09-30T17:47:31.427" LastActivityDate="2015-09-30T17:47:31.427" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8253" PostTypeId="1" AcceptedAnswerId="8254" CreationDate="2015-09-30T17:41:39.737" Score="10" ViewCount="8563" Body="&lt;p&gt;Suppose we have the following dataframe with multiple values for a certain column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    categories&#xA;0 - [&quot;A&quot;, &quot;B&quot;]&#xA;1 - [&quot;B&quot;, &quot;C&quot;, &quot;D&quot;]&#xA;2 - [&quot;B&quot;, &quot;D&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can we get a table like this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   &quot;A&quot;  &quot;B&quot;  &quot;C&quot;  &quot;D&quot;&#xA;0 - 1    1    0    0&#xA;1 - 0    1    1    1&#xA;2 - 0    1    0    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: I don't necessarily need a new dataframe, I'm wondering how to transform such DataFrames to a format more suitable for machine learning.&lt;/p&gt;&#xA;" OwnerUserId="13027" LastActivityDate="2020-08-01T15:44:28.253" Title="How to binary encode multi-valued categorical variable from Pandas dataframe?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8254" PostTypeId="2" ParentId="8253" CreationDate="2015-09-30T18:31:26.023" Score="7" Body="&lt;p&gt;If &lt;code&gt;[0, 1, 2]&lt;/code&gt; are numerical labels and is not the index, then &lt;code&gt;pandas.DataFrame.pivot_table&lt;/code&gt; works:&lt;/p&gt;&#xA;&lt;pre&gt;&#xA;In []:&#xA;data = pd.DataFrame.from_records(&#xA;    [[0, 'A'], [0, 'B'], [1, 'B'], [1, 'C'], [1, 'D'], [2, 'B'], [2, 'D']],&#xA;    columns=['number_label', 'category'])&#xA;data.pivot_table(index=['number_label'], columns=['category'], aggfunc=[len], fill_value=0)&#xA;&lt;/pre&gt;&#xA;&lt;pre&gt;&#xA;Out[]:&#xA;              len&#xA;category      A      B      C      D&#xA;number_label                       &#xA;0             1      1      0      0&#xA;1             0      1      1      1&#xA;2             0      1      0      1&#xA;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://pbpython.com/pandas-pivot-table-explained.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This blog&lt;/a&gt; post was helpful.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;If &lt;code&gt;[0, 1, 2]&lt;/code&gt; is the index, then &lt;code&gt;collections.Counter&lt;/code&gt; is useful:&lt;/p&gt;&#xA;&lt;pre&gt;&#xA;In []:&#xA;data2 = pd.DataFrame.from_dict(&#xA;    {'categories': {0: ['A', 'B'], 1: ['B', 'C', 'D'], 2:['B', 'D']}})&#xA;data3 = data2['categories'].apply(collections.Counter)&#xA;pd.DataFrame.from_records(data3).fillna(value=0)&#xA;&lt;/pre&gt;&#xA;&lt;pre&gt;&#xA;Out[]:&#xA;       A      B      C      D&#xA;0      1      1      0      0&#xA;1      0      1      1      1&#xA;2      0      1      0      1&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="13103" LastEditorUserId="98307" LastEditDate="2020-08-01T15:44:28.253" LastActivityDate="2020-08-01T15:44:28.253" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="8255" PostTypeId="1" AcceptedAnswerId="8259" CreationDate="2015-09-30T19:21:44.187" Score="2" ViewCount="194" Body="&lt;p&gt;I was posting on stats.stackexchange but perhaps I should be posting here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Context. Subscription business that charges users a monthly fee for access to the service. Management would like to predict &quot;churn&quot; - subscriptions who are likely to cancel. Management would like to create an email sequence in attempt to prevent high risk accounts from churning, perhaps with a discount code of some sort. So I need to identify those accounts at risk of leaving us.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with say 50k records. Each line item is an account number along with some variables. One of the variables is &quot;Churned&quot; with a value of &quot;yes&quot; (they cancelled) or &quot;No&quot; (they are active).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset I have is all data since the beginning of time for the business. About 20k records are active paying customers and about 30k are those who used to be paying customers but who have since cancelled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task is to build a model to predict which of the 20k active customers are currently likely to churn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is where I have tied my brain in a knott. I need to run the model (Predict) on the 20k records of active customers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I split my data between training, test and predict?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does predict data have to be exclusive of train and test data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I split the entire dataset of 50k into 0.8 train and 0.2 test, build a model and then predict on the 20k active accounts? That would imply I'm training and testing on data that I'm also going to predict on. Seems &quot;wrong&quot;. Is it?&lt;/p&gt;&#xA;" OwnerUserId="13106" LastEditorUserId="13727" LastEditDate="2016-01-27T10:17:14.433" LastActivityDate="2016-01-27T10:17:14.433" Title="Dividing data between test, learn and predict" Tags="|machine-learning|predictive-modeling|churn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8256" PostTypeId="2" ParentId="6947" CreationDate="2015-09-30T21:12:32.543" Score="1" Body="&lt;p&gt;If you are interested in a very high-level (enterprise architecture) framework, I suggest you to take a look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/MIKE2.0_methodology&quot; rel=&quot;nofollow&quot;&gt;MIKE2.0 Methodology&lt;/a&gt;. Being an &lt;em&gt;information management&lt;/em&gt; framework, MIKE2.0 has, certainly, much wider coverage than the domain of your interest, but it is a solid, interesting and open (licensed under the Creative Commons Attribution License) framework. A better fit for your focus is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Extract,_transform,_load&quot; rel=&quot;nofollow&quot;&gt;Extract, transform, load (ETL)&lt;/a&gt; framework, which is extremely popular in contexts of Business Intelligence and Data Warehousing. On a more practical note, you might want to check &lt;a href=&quot;https://www.quora.com/Is-there-an-open-source-customer-consumer-and-not-enterprise-customer-master-data-management-solution-preferably-cloud-based-available/answer/Aleksandr-Blekh?srid=OPcS&amp;amp;share=1&quot; rel=&quot;nofollow&quot;&gt;my answer&lt;/a&gt; on Quora on open source master data management (MDM) solutions. Pay attention to the Talend solutions (disclaimer: I am not affiliated with this or any company), which cover a wide spectrum of MDM, ETL and &lt;a href=&quot;http://www.talend.com/products/data-integration&quot; rel=&quot;nofollow&quot;&gt;data integration&lt;/a&gt; domains as open source and commercial offerings.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-09-30T21:12:32.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8257" PostTypeId="2" ParentId="8255" CreationDate="2015-09-30T21:15:16.140" Score="2" Body="&lt;p&gt;The docs from scikit-learn can help with best practices for splitting your data for cross-validation (training, testing, validating): &lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/cross_validation.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, all of your accounts are labeled as &quot;Churned&quot; with values either &quot;Yes&quot; or &quot;No&quot;. If you make a model that predicts Churned=Yes/No, then you're answering the question &quot;Is a given account currently active?&quot;, which I don't think is the question you want to answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand, I think you want to ask an actionable question about the Churned=No accounts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;What is the probability that a Churned=No account will become a Churned=Yes within the next 30 days?&quot;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Requires access to time-series data.&lt;/li&gt;&#xA;&lt;li&gt;Example approach:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use the 30k Churned=Yes accounts to create a supervised model (since you know when and how these accounts went from Churned=No to Churned=Yes) and apply it to the 20k Churned=No accounts.&lt;/li&gt;&#xA;&lt;li&gt;The predicted quantity is the probability of a Churned=No account becoming a Churned=Yes account within the next 30 days.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&quot;How 'close' is a Churned=No account to being a Churned=Yes account?&quot;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does not require access to time-series data.&lt;/li&gt;&#xA;&lt;li&gt;Example approach:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Again use the 30k Churned=Yes accounts to create a supervised model (may require multiple models for different clusters of similar accounts) and apply it to the 20k Churned=No accounts.&lt;/li&gt;&#xA;&lt;li&gt;The predicted quantity is a metric (i.e. a risk score) that measures the proximity in parameter space of a Churned=No account to the Churned=Yes accounts.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13103" LastEditorUserId="13103" LastEditDate="2015-09-30T22:37:04.277" LastActivityDate="2015-09-30T22:37:04.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8258" PostTypeId="2" ParentId="8230" CreationDate="2015-10-01T02:43:47.057" Score="1" Body="&lt;p&gt;If I understand, you have a set of paragraphs, and each paragraph is tagged with keywords. This defines a mapping from paragraph to keyword and vice-versa (an index and a reverse index). You can create a graph of paragraphs and keywords, but all of the edges are undirected (or they all have the same direction). Paragraphs only connect with keywords, and keywords only connect with paragraphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To establish relationships between paragraphs, you could define a transitive equivalence relation such as &quot;all paragraphs that connect to the same keyword are now connected to each other&quot;. This would allow you to rank the paragraphs that contain a given keyword by their number of connections to other paragraphs, but it may not be an appropriate measure of relevance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For PageRank to be most helpful, I think you need directional relationships (e.g. if &quot;see page 123&quot; was mentioned in a paragraph). To better represent the complexity of your data, creating additional relationships between keywords (e.g. semantic meaning, lexical relationships) and creating additional relationships between paragraphs (e.g. document classification) may help. &lt;a href=&quot;https://books.google.com/books?id=KGIbfiiP1i4C&quot; rel=&quot;nofollow&quot;&gt;Natural Language Processing with Python&lt;/a&gt; discusses these topics.&lt;/p&gt;&#xA;" OwnerUserId="13103" LastActivityDate="2015-10-01T02:43:47.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8259" PostTypeId="2" ParentId="8255" CreationDate="2015-10-01T03:33:52.937" Score="3" Body="&lt;p&gt;&lt;strong&gt;Supervised Learning:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do you have a saved time history of the data?&lt;/strong&gt;  For a supervised learning set you need some churned=&quot;No&quot; cases and some churned=&quot;Yes&quot; cases, but it sounds like you only have churned=&quot;Yes&quot; and the unknown cases e.g. current customers who may or may not churn.  With some time history you can go back in time and definitively label the current customers as churn=&quot;No&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then it is very easy to split up the data.  And no, you probably don't want to predict on any data that you trained on since you can only train on it if you already know the solution so it will be a waste of time and throw off any metrics you might use to assess accuracy (precision/recall/F1) in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Unsupervised Learning:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you don't have saved time history of the data&lt;/strong&gt; then this is an unsupervised learning set for which you have churned=&quot;yes&quot; and churned=&quot;maybe&quot;.  You could then employ anomaly or outlier detection on this set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;novelty detection:&lt;/strong&gt;&#xA;    The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;outlier detection:&lt;/strong&gt;&#xA;    The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do either one but novelty is more powerful.  This is kind of a flip around as the novelty here is Churned=&quot;No&quot; since all of your data is the confirmed Churn=&quot;Yes&quot; cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-10-01T03:33:52.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8260" PostTypeId="2" ParentId="8246" CreationDate="2015-10-01T04:17:22.643" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What does this sentence mean? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It means that the next vector should be perpendicular to all the previous ones with respect to a matrix. It's like how the &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_basis&quot; rel=&quot;noreferrer&quot;&gt;natural basis&lt;/a&gt; vectors are &lt;a href=&quot;https://en.wikipedia.org/wiki/Orthogonality&quot; rel=&quot;noreferrer&quot;&gt;perpendicular&lt;/a&gt; to each other, with the added twist of a matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathrm {x^T A y} = 0$ instead of $\mathrm{x^T y} = 0$&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And what is line search mentioned in the webpage?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Line_search&quot; rel=&quot;noreferrer&quot;&gt;Line search&lt;/a&gt; is an optimization method that involves guessing how far along a given direction (i.e., along a line) one should move to best reach the local minimum.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-10-01T04:17:22.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8261" PostTypeId="2" ParentId="6322" CreationDate="2015-10-01T10:57:42.560" Score="0" Body="&lt;p&gt;With (d=256, n=10) as well as  with (d=16000, n=1000) you are under the &lt;a href=&quot;https://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;curse of dimensionality&lt;/a&gt;. The essence of the curse (quoted from Wikipedia)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;nearly all of the high-dimensional space is &quot;far away&quot; from the centre. To put it another way, the high-dimensional unit hypercube can be said to consist almost entirely of the &quot;corners&quot; of the hypercube, with almost no &quot;middle&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For an intuitive explanation I also found &lt;a href=&quot;http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO, for your problem with n=10 you can use d&amp;lt;=2, for n=1000 d=3 (maybe 4, at most 5). Why d=3 for n=1000? Roughly speaking  this would correspond to 10 points along each dimension (10^3=1000), which is reasonable to fill the 3D space. For d=5 it is like 4 points in each dimension, which is not so good but not a disaster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMHO, you should try to reformulate your problem and significantly reduce dimensionality (maybe try to use SVD or PCA). This may automatically solve your problem of noisy data.   &lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-10-01T10:57:42.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8262" PostTypeId="2" ParentId="6959" CreationDate="2015-10-01T11:02:02.950" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/12046/organizing-a-classification-tree-in-rpart-into-a-set-of-rules&quot;&gt;This question&lt;/a&gt; on CrossValidated sort of answers your question. Rpart classifies via a decision tree, which means that if you could print the tree in one form or another it would be easy for you to take a document and see why it ended up being classified the way it was. The tree in itself is just a hierarchical set of rules applied to new documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The user was advised to use the &lt;a href=&quot;https://cran.r-project.org/web/packages/rattle/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;rattle&lt;/a&gt; library which also uses decision trees but is able to print the set of rules induced by the tree, which should help you in your task.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-10-01T11:02:02.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8263" PostTypeId="2" ParentId="6959" CreationDate="2015-10-01T11:08:47.600" Score="1" Body="&lt;p&gt;rpart is a decision tree model and as such is very much interpretable.&#xA;You should visualize your decision tree. See the examples &lt;a href=&quot;http://www.milbo.org/rpart-plot/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, one of them is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wPG1R.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wPG1R.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for each given document you can follow the tree path and see &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;what were the features on the way responsible for the classification &lt;/li&gt;&#xA;&lt;li&gt;what leaf your document arrived at and what is the percentage of correctly classified documents at these leaf.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2015-10-01T11:08:47.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8264" PostTypeId="1" CreationDate="2015-10-01T15:06:43.927" Score="4" ViewCount="209" Body="&lt;p&gt;we are trying to build a predicting model using machine learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a use case where the input data have a very high dimension. Each sample point has 20000 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have a decent training sample set with around 1 million training samples, if necessary, we can acquire more, say, 2 or 3 million.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are not very speed sensitive, it's not like a recommendation system that needs to respond within a second. The application allows us to spend up to a couple of minutes for one prediction. Nevertheless, we hope the algorithm can be run in parallel mode in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;given above description, what kind of algorithm would you suggest? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our biggest concern is over fitting, with so many features, it seems that we are doomed to over-fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We were trying to do something alone the line of nearest neighbor, but with these many features, calculating distance sounds like a mission impossible. Maybe we should do PCA to do dimension reduction first?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comment is welcomed!&lt;/p&gt;&#xA;" OwnerUserId="13120" LastEditorUserId="13120" LastEditDate="2015-10-01T15:16:13.827" LastActivityDate="2015-11-01T11:04:00.170" Title="when speed is not a major concern, what is the best ML algorithm for high dimensional data?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8265" PostTypeId="1" CreationDate="2015-10-01T15:19:58.877" Score="1" ViewCount="497" Body="&lt;p&gt;I have some question concern similarity measure&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that we have a matrix &lt;strong&gt;&lt;em&gt;M&lt;/em&gt;&lt;/strong&gt; where &lt;strong&gt;&lt;em&gt;M(i,j)&lt;/em&gt;&lt;/strong&gt; is the similarity measure between user &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt; and user &lt;strong&gt;&lt;em&gt;j&lt;/em&gt;&lt;/strong&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each user is characterised by : id-user | country | id-artist | id-track &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this I choose to use Jaccard similarity metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jaccard is determined to compute similarity between users based on the tracks that they listened.&#xA;My question is : is it possible to take account both id-track &lt;strong&gt;&lt;em&gt;and&lt;/em&gt;&lt;/strong&gt; id-artist to measure the similarity between users?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="8088" LastEditorUserId="8878" LastEditDate="2016-02-02T13:53:23.603" LastActivityDate="2016-02-02T13:53:23.603" Title="similarity measure with two features" Tags="|similarity|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8266" PostTypeId="1" AcceptedAnswerId="8273" CreationDate="2015-10-01T18:40:47.937" Score="10" ViewCount="6614" Body="&lt;p&gt;There is a package named segmented in R. Is there a similar package in python?&lt;/p&gt;&#xA;" OwnerUserId="12613" LastActivityDate="2021-12-06T11:22:08.087" Title="Is there a library that would perform segmented linear regression in python?" Tags="|python|regression|linear-regression|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8267" PostTypeId="2" ParentId="8265" CreationDate="2015-10-01T18:53:46.107" Score="4" Body="&lt;p&gt;Yes, multiple different ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id gives us no additional information beyond the track id, this will give the same result, whereas it will give different results if a particular track id could be associated with multiple artists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, we could compute the Jaccard similarity on tracks, and then the Jaccard similarity on artists, and then add the two (probably with some constant coefficient scaling the two). This way two users who listen to the same artists, but none of the same songs by those artists, will be rated as more similar than users who listen to different songs by different artists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What coefficient makes sense? Well, you could start with 0.5 for each (i.e. just .5*similarity_artist+.5*similarity_track), see if that's reasonable, and adjust if it's not.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-01T18:53:46.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8268" PostTypeId="1" CreationDate="2015-10-01T18:57:42.820" Score="3" ViewCount="66" Body="&lt;p&gt;What are the benefits of establishing data science programs as an independent discipline rather than creating data scientists in different fields? (for students, universities and the society as a whole)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am authoring a book on data science and I have mentioned some university programs as models for establishing data science programs in bachelor's, master's and doctorate level. I am thinking of some reasons to compare this model with enriching well-established programs and adding data science skills.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-10-03T15:43:25.610" LastActivityDate="2015-10-03T15:43:25.610" Title="Establishing data science programs as an independent discipline" Tags="|data|education|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8270" PostTypeId="2" ParentId="8230" CreationDate="2015-10-01T19:38:43.373" Score="1" Body="&lt;p&gt;Short version: I would not expect PageRank to be the right algorithm to use for this problem, because the thing you're interested in (relevance) is a feature of the content, not the context (i.e. the graph structure of the links). PageRank works on the web because the context is so closely related to the content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long version: PageRank is a way for weight to flow along directional links. Eventually, a stable distribution is reached where weight flows in and out in equal measure. (Generally, people talk about PageRank measuring &quot;importance&quot; or &quot;popularity&quot; because both of those have natural internal definitions--one item is popular because other popular items like it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can create something like PageRank with a graph with multiple classes of objects--no 'paragraph' links to other paragraphs, but it links to 'keyword's, and no keyword links to other keywords, but it links to other paragraphs. This gives you a minor constraint on the graph structure, which should probably be reflected in the randomized restart probability used by PageRank. Initiate the algorithm with half the weight on paragraphs and half the weight on keywords, and it'll iterate until it converges. (This is very similar to Samuel Harrold's solution, except the weighting of the links between paragraphs depends on the number of keywords they have in common.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what problem will that actually solve? Which paragraphs and keywords will be the most popular? The useful work of PageRank is &lt;em&gt;mostly&lt;/em&gt; being done by asymmetric links (knowing that Alice links to Bob but Bob does not link to Alice is evidence that Bob is more popular than Alice), and we only have symmetric links (unless we use different mechanisms to determine what keywords link to paragraphs than what paragraphs link to keywords). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The remaining work is being done by link scarcity--since a page's popularity is shared among all its links, a page that links to fewer pages transmits more popularity to them than a page that links to more pages. But this means that paragraphs that only match a few keywords make those keywords more popular, and keywords that only link to a few paragraphs make those paragraphs more popular. It's not obvious to me that this will have the effect that we want--a paragraph that receives weight from both &quot;spark plug&quot; and &quot;wheel&quot; is going to show up higher in the rankings than a paragraph that receives weight from just &quot;spark plug,&quot; which is probably the opposite of what the user will want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you might want to show paragraphs to the user ranked by their weight percentage from a given keyword--if paragraph A gets 100% of its weight from spark plug, paragraph B gets 30% of its weight from spark plug, and C gets 0% of its weight from spark plug, then maybe A is the best result to show for spark plug, followed by B. But is this going to be any different in practice from just ranking pages by what fraction of their total keywords are any particular keyword? It doesn't seem like it'll be by much.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-01T19:38:43.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8271" PostTypeId="1" CreationDate="2015-10-01T20:22:36.540" Score="1" ViewCount="1018" Body="&lt;p&gt;I know that a decision tree recursively splits along each attribute, greedily minimizing the wrong classifications/deviance at each split.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, what is the order in which the attributes are split?  In other words, for a regression tree in N dimensions, what determines which attribute gets split first?    &lt;/p&gt;&#xA;" OwnerUserId="10512" LastActivityDate="2015-10-02T10:23:38.967" Title="How to choose the order in which to split a decision tree?" Tags="|regression|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8272" PostTypeId="2" ParentId="8244" CreationDate="2015-10-01T21:07:26.540" Score="11" Body="&lt;p&gt;It depends on &lt;em&gt;what&lt;/em&gt; you do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column stores have two key benefits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;whole columns can be skipped&lt;/li&gt;&#xA;&lt;li&gt;run-length compression works better on columns (for certain data types; in particular with few distinct values)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However they also have drawbacks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;many algorithms will need all columns, and only record at a time (e.g. k-means) or may even need to compute a pairwise distance matrix&lt;/li&gt;&#xA;&lt;li&gt;compression techniques only work well on sparse data types and factors, but not well on double-valued continuous data&lt;/li&gt;&#xA;&lt;li&gt;appends on column stores are expensive, so it is not ideal for streaming / changing data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Columnar storage is really popular for OLAP aka &quot;stupid analytics&quot; (Michael Stonebraker) and of course for &lt;em&gt;preprocessing&lt;/em&gt; where you may indeed be interested in discarding whole columns (but you would need to have structured data first - you don't store JSONs in columnar format). Because the columnar layout is really nice for e.g. counting how many apples you have sold last week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For much of the scientific / data science use cases, &lt;strong&gt;array databases&lt;/strong&gt; appear to be the way to go (plus, of course, unstructured input data). E.g. SciDB and RasDaMan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many cases (e.g. deep learning), matrixes and arrays are the data types you need, not columns. MapReduce etc. can still be useful in preprocessing, of course. Maybe even column data (but array database usually support a column-like compression, too).&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2015-10-17T13:22:14.090" LastActivityDate="2015-10-17T13:22:14.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8273" PostTypeId="2" ParentId="8266" CreationDate="2015-10-01T23:38:12.950" Score="8" Body="&lt;p&gt;No, currently there isn't a package in Python that does segmented linear regression as thoroughly as those in R (e.g. &lt;a href=&quot;http://www.r-bloggers.com/r-for-ecologists-putting-together-a-piecewise-regression/&quot; rel=&quot;noreferrer&quot;&gt;R packages listed in this blog post&lt;/a&gt;). Alternatively, you can use a Bayesian Markov Chain Monte Carlo algorithm in Python to create your segmented model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segmented linear regression, as implemented by all the R packages in the above link, doesn't permit extra parameter constraints (i.e. priors), and because these packages take a frequentist approach, the resulting model doesn't give you probability distributions for the model parameters (i.e. breakpoints, slopes, etc). Defining a segmented model in &lt;a href=&quot;http://statsmodels.sourceforge.net/&quot; rel=&quot;noreferrer&quot;&gt;statsmodels&lt;/a&gt;, which is frequentist, is even more restrictive because the model requires a fixed x-coordinate breakpoint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can design a segmented model in Python using the Bayesian Markov Chain Monte Carlo algorithm &lt;a href=&quot;http://dan.iel.fm/emcee/current/&quot; rel=&quot;noreferrer&quot;&gt;emcee&lt;/a&gt;. Jake Vanderplas wrote a useful &lt;a href=&quot;http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/&quot; rel=&quot;noreferrer&quot;&gt;blog post&lt;/a&gt; and &lt;a href=&quot;http://arxiv.org/pdf/1411.5018.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; for how to implement emcee with comparisons to PyMC and PyStan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Segmented model with data:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QoGKH.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QoGKH.png&quot; alt=&quot;Segmented regression&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Probability distributions of fit parameters:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iJ5br.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iJ5br.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/stharrold/ARCHIVED_2015_Harrold_SDSSJ1600/blob/master/code/utils.py#L1241-L1546&quot; rel=&quot;noreferrer&quot;&gt;Link to code for segmented model&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/stharrold/33120aaf182b66983ee8&quot; rel=&quot;noreferrer&quot;&gt;Link to (large) ipython notebook&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13103" LastEditorUserId="13103" LastEditDate="2015-10-04T17:32:29.987" LastActivityDate="2015-10-04T17:32:29.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8274" PostTypeId="2" ParentId="8264" CreationDate="2015-10-02T00:00:04.043" Score="2" Body="&lt;p&gt;You didn't say whether you were building a regression or classification model, but here goes anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As ever, it depends ... though several neural network approaches, such as deep learning or RBF networks, have shown promise with high dimensional data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to have KNN approaches which use representative points, either as cluster centres or class boundaries, to reduce the computation burden.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a test I tried computing the Euclidean distance between a single vector and  20,000 , 20,000 feature vectors. This took around 7 seconds on a single core of a desktop machine using Mathematica. If you have the RAM and several cores KNN should be feasible in your time scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature engineering might gain you speed benefits but you would want to tune that in concert with prediction/classification accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several methods for achieving regularisation if you suspect you are overfitting,  you might want to explore Ridge Regression/Tikhonov regularisation or early stopping if you are following the neural network path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastEditorUserId="7980" LastEditDate="2015-10-02T08:54:42.810" LastActivityDate="2015-10-02T08:54:42.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8275" PostTypeId="5" CreationDate="2015-10-02T06:35:51.567" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-10-02T06:35:51.567" LastActivityDate="2015-10-02T06:35:51.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8276" PostTypeId="4" CreationDate="2015-10-02T06:35:51.567" Score="0" Body="education tag is used for data science programs including university programs and professional certificates." OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-10-02T14:47:00.673" LastActivityDate="2015-10-02T14:47:00.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8277" PostTypeId="1" AcceptedAnswerId="8292" CreationDate="2015-10-02T08:13:03.780" Score="3" ViewCount="3943" Body="&lt;p&gt;I am trying to figure out how the amount of money that a customer would want to withdraw on an ATM tell us if the transaction is fraudulent or not.There are other attributes, of course, but now I would want to hear your views on the amount of money that the customer wants to withdraw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data may be of this form:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume that a customer, for ten consecutive transactions, withdrew the following amounts: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;100.33, 384 , 458, 77.90, 456, 213.55, 500 , 500, 300, 304.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can we use this data to tell if the next transaction done on this account is fraudulent of not?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there specific algorithms that can be used for this classification?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I was thinking:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking to calculate the average amount of money, say for the last ten transactions, and check how far is the next transaction amount from the average. Too much deviation would signal an anomaly. But this does not sound much, does it?&lt;/p&gt;&#xA;" OwnerUserId="13132" LastActivityDate="2015-10-03T20:17:46.670" Title="Credit card fraud detection - anomaly detection based on amount of money to be withdrawn?" Tags="|machine-learning|algorithms|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8278" PostTypeId="1" CreationDate="2015-10-02T09:49:55.943" Score="11" ViewCount="4839" Body="&lt;p&gt;Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which contains a few hundred images.&lt;/p&gt;&#xA;" OwnerUserId="12983" LastActivityDate="2017-07-17T11:30:21.867" Title="Covariate shift detection" Tags="|machine-learning|classification|dataset|image-classification|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8279" PostTypeId="2" ParentId="8271" CreationDate="2015-10-02T10:23:38.967" Score="1" Body="&lt;p&gt;One option is to use a simple approach, like just choosing random attributes, or taking one at a time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to use a metric to decide, at each step, the attribute that best splits the data. Different tree algorithms apply different metrics, like for instance Gini impurity (CART algorithm), or information gain (ID3 and C4.5 algorithms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read a brief description of these methods in the &quot;Decision tree learning&quot; entry on the Wikipedia. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-10-02T10:23:38.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8280" PostTypeId="2" ParentId="8277" CreationDate="2015-10-02T12:20:32.303" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I was thinking to calculate the average amount of money, say for the&#xA;  last ten transactions, and check how far is the next transaction&#xA;  amount from the average.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This sounds like a good start. I'd look into Local Outlier Probabilities. For a given data point you could calculate the distance from &lt;code&gt;n&lt;/code&gt; nearest neighbors and figure out if the data point under consideration is an outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Local_outlier_factor&quot; rel=&quot;nofollow&quot;&gt;basic overview can be found here&lt;/a&gt;&#xA;I'd also consider the source, destination, volume and frequency of transactions as features. &lt;/p&gt;&#xA;" OwnerUserId="426" LastActivityDate="2015-10-02T12:20:32.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8282" PostTypeId="2" ParentId="8162" CreationDate="2015-10-02T17:50:26.450" Score="0" Body="&lt;p&gt;Try to make group a factor and you might want to make it a strata,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- coxph(survObj ~ strata(factor(group)), data=dataset)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Not sure whether this would help but it will definitely have impact on how Surv treats your information. Without the factor Surv will consider group a numeric variable where an increase in that number will lead to a higher number of cases. Converting it to a factor and strata will estimate different survival curves for each group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure whether this will help with your plot but it will certainly impact the outcome of your fits.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2015-10-02T17:50:26.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8283" PostTypeId="2" ParentId="8277" CreationDate="2015-10-02T20:40:07.630" Score="1" Body="&lt;p&gt;What's the underlying model of how much someone requests from an ATM? It doesn't seem like it's a simple distribution like a Gaussian, where comparing new amounts to the mean is sensible. Consider a person who always pulls out either \$40 or \$400. Ideally we want to build a distribution of what normal transactions from a user look like, and notice if new datapoints don't look like they're sampled from that distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;idclark's suggestion, to look at the nearest &lt;em&gt;n&lt;/em&gt; datapoints from that user and compute the distance from just them, is a good and fast implementation of that sort of test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One other possibility is to try to find similar users, and then aggregate data across users. If I only have 10 withdrawals from each user, I'm not going to be able to reject any new withdrawals with confidence, but if I have seven clusters of users, with a thousand withdrawals per cluster, I can notice when a user who was in a particular cluster deviates from the overall cluster distribution. (This also helps you make use of knowledge about which previous transactions were fraudulent.)&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-02T20:40:07.630" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8284" PostTypeId="2" ParentId="2323" CreationDate="2015-10-02T21:20:40.230" Score="3" Body="&lt;p&gt;Suppose (prior to one-hot-encoding) you have predictors/fields from a set $Z$ (say movie genre, user gender, and user race).  Suppose further, each predictor $z \in Z$ can take on one of $k_z$ values.  After one-hot encoding, you will have a new set of binary features $X$ of size $K := \sum_{z \in Z}k_z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a model with all interactions, you must estimate a matrix of interaction coefficients $Q$, which has $K\times (K+1) / 2$ unique terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The factorization machine puts structure on the matrix $Q$, and assumes that $Q \equiv W^{T} W$, where $W$ is of dimension $l \times K$, with $1\le l \le K$ some number specified by the user.  We estimate $W$ instead of $Q$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The field-aware factorization machine puts structure on $Q$ as well.  It partitions $Q$ into blocks based on $z$ (the original features).  If $q_{z_i, z_j}$ denotes the $z_i,z_j$ block of $Q$, we assume that $q_{z_i,z_j}$ comes from the $z_i,z_j$ block of $W_j^{T} W_i$, where $W_i$ is of dimension $l \times K$.  As with the FM, we estimate the $W_i$ instead of $Q$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The FM factorization of $Q$ has $K \times l $ parameters.  The &quot;feild-aware&quot; FM has $K\times l\times |Z|$ parameters.  A model with all interactions has $K \times (K+1)/2$ parameters.  &lt;/p&gt;&#xA;" OwnerUserId="13152" LastEditorUserId="13152" LastEditDate="2015-10-02T21:50:57.177" LastActivityDate="2015-10-02T21:50:57.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8285" PostTypeId="2" ParentId="8278" CreationDate="2015-10-03T00:44:15.000" Score="0" Body="&lt;p&gt;You don't give many clues about what properties of the images you might be considering, but it seems that what you might want to measure is the difference in the distributions of the training and tests sets. A useful place to start would be with the Kullback–Leibler divergence, which is a measure of the difference of two distributions.&lt;/p&gt;&#xA;" OwnerUserId="7980" LastActivityDate="2015-10-03T00:44:15.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8286" PostTypeId="1" CreationDate="2015-10-03T04:09:56.397" Score="33" ViewCount="10258" Body="&lt;p&gt;Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar to MS Excel, but that has R as the underlying language instead of VB.&lt;/p&gt;&#xA;" OwnerUserId="13154" LastEditorUserId="2452" LastEditDate="2015-10-03T06:18:50.037" LastActivityDate="2023-12-10T14:50:00.920" Title="Are there any tools for feature engineering?" Tags="|feature-selection|feature-extraction|feature-construction|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8287" PostTypeId="2" ParentId="8286" CreationDate="2015-10-03T08:05:06.717" Score="23" Body="&lt;p&gt;Very interesting question (+1). While I am not aware of any software tools that currently offer comprehensive functionality for &lt;em&gt;feature engineering&lt;/em&gt;, there is definitely a wide range of options in that regard. Currently, as far as I know, feature engineering is still largely a &lt;strong&gt;laborious&lt;/strong&gt; and &lt;strong&gt;manual&lt;/strong&gt; process (i.e., see &lt;a href=&quot;http://appliedpredictivemodeling.com/blog/2015/7/28/feature-engineering-versus-feature-extraction&quot;&gt;this blog post&lt;/a&gt;). Speaking about the feature engineering subject domain, &lt;a href=&quot;http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it&quot;&gt;this excellent article&lt;/a&gt; by Jason Brownlee provides a rather comprehensive overview of the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ben Lorica, Chief Data Scientist and Director of Content Strategy for Data at O'Reilly Media Inc., has written a &lt;a href=&quot;http://radar.oreilly.com/2014/06/streamlining-feature-engineering.html&quot;&gt;very nice article&lt;/a&gt;, describing the state-of-art (as of June 2014) approaches, methods, tools and startups in the area of &lt;em&gt;automating&lt;/em&gt; (or, as he put it, &lt;em&gt;streamlining&lt;/em&gt;) feature engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a brief look at some &lt;em&gt;startups&lt;/em&gt; that Ben has referenced and a product by &lt;a href=&quot;http://www.skytree.net&quot;&gt;Skytree&lt;/a&gt; indeed looks quite impressive, especially in regard to the subject of this question. Having said that, some of their claims sound really suspicious to me (i.e., &lt;em&gt;&quot;Skytree speeds up machine learning methods by up to 150x compared to open source options&quot;&lt;/em&gt;). Continuing talking about commercial data science and machine learning offerings, I have to mention solutions by Microsoft, in particular their &lt;em&gt;Azure Machine Learning Studio&lt;/em&gt;. This Web-based product is quite powerful and elegant and offers some feature engineering functionality (FEF). For an example of some simple FEF, see &lt;a href=&quot;http://datasciencedojo.com/tutorial-videos/feature-engineering-r-script-with-azure-ml-studio&quot;&gt;this nice video&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Returning to the question, I think that the simplest approach one can apply for automating feature engineering is to use corresponding &lt;strong&gt;IDEs&lt;/strong&gt;. Since you (me, too) are interested in R language as a data science backend, I would suggest to check, in addition to RStudio, another similar open source IDE, called &lt;a href=&quot;https://rkward.kde.org&quot;&gt;RKWard&lt;/a&gt;. One of the advantages of RKWard vs RStudio is that it supports &lt;a href=&quot;http://api.kde.org/doc/rkwardplugins&quot;&gt;writing plugins&lt;/a&gt; for the IDE, thus, enabling data scientists to automate feature engineering and streamline their R-based data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, on the other side of the spectrum of feature engineering solutions we can find some &lt;strong&gt;research projects&lt;/strong&gt;. The two most notable seem to be Stanford University's &lt;a href=&quot;http://i.stanford.edu/hazy/victor/columbus&quot;&gt;Columbus project&lt;/a&gt;, described in detail in the &lt;a href=&quot;http://cs.stanford.edu/people/chrismre/papers/mod539-zhang.pdf&quot;&gt;corresponding research paper&lt;/a&gt;, and &lt;em&gt;Brainwash&lt;/em&gt;, described in &lt;a href=&quot;http://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper82.pdf&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastActivityDate="2015-10-03T08:05:06.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8288" PostTypeId="2" ParentId="8286" CreationDate="2015-10-03T08:29:29.853" Score="1" Body="&lt;p&gt;You should consider checking the &lt;a href=&quot;https://azure.microsoft.com/en-us/services/machine-learning/&quot; rel=&quot;nofollow&quot;&gt;Azure Machine Learning&lt;/a&gt; platform. It is online and you can use it with a free account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Azure ML provides you with a workflow by using modules in a graphic user interface. Many of them are related with Data Munging and you can easily clean your data. If there is something that you cannot do in the GUI, then you can just add a module which let you run custom R or Python script to manipulate your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice part of it, is that you can easily visualise your data at any time and check simple stats like the &lt;code&gt;dataframe.describe()&lt;/code&gt; of the R.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-03T08:29:29.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8289" PostTypeId="2" ParentId="8277" CreationDate="2015-10-03T08:44:44.893" Score="0" Body="&lt;p&gt;I isn't actually answering your question, but it is an idea of how you can improve it. In my opinion, I don't believe that you will be able to build a classification model with only those data. And if you do it, it will not have high enough accuracy. In your position, I would start looking for more data to use as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ATM's code of the withdrawal. People use most of the time similar ATM in their daily routine. If you know the lat and long of their previous ATM, you can check if one of them is far away and combining it with the other features, you will increase your accuracy.&lt;/li&gt;&#xA;&lt;li&gt;Seconds spent on the ATM for each withdrawal. People tend to follow specific patterns when they withdraw money. If all of their previous data are similar on the spending time and then you see lower or higher time on a data point, you will be able to increase the accuracy of the model.&lt;/li&gt;&#xA;&lt;li&gt;Labeled data. In models like this, it is far better if you use supervised algorithms instead of unsupervised. Thus, I would seek for labeled data for fraud usages. This will also let you to calculate the actual accuracy of your model.&lt;/li&gt;&#xA;&lt;li&gt;Time between the two withdrawals. As I said before, people tends to follow patterns. An &quot;anomaly&quot; on this with a sooner withdrawal than the expected will also raise your accuracy.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As far as the algorithms are concerned, I am not keen on choosing one, because it is popular. If you have done all the Data Munging and the feature selection, you have the 90% of the job and the algorithm that you will choose is 2-3 lines on the code (in case you are using a language like Python). What I usually do, is to check all the possible algorithms and evaluate their accuracy. Then I either use a combination of them or the one with the highest accuracy.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-03T08:44:44.893" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="8290" PostTypeId="2" ParentId="8268" CreationDate="2015-10-03T10:34:01.960" Score="2" Body="&lt;p&gt;Advantages of setting up a dedicated data science as an independent discipline, which I can think of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There would be more and more &lt;strong&gt;data scientists&lt;/strong&gt; who can cater to the growing demand for them, in companies and industries in almost every domain.&lt;/li&gt;&#xA;&lt;li&gt;The concept of statistics, algorithms (ML would be included in this), and Big Data don't need to exist as chapters in separate domains. Now, they can exist as part of the data science workflow in this program.&lt;/li&gt;&#xA;&lt;li&gt;Data Scientists would be more aware how data science would be applicable in the domain of business. The concepts of business knowledge for data scientists is often found wanting. Data scientists with a knowledge of &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Business_statistics&quot; rel=&quot;nofollow&quot;&gt;Business Statistics&lt;/a&gt;&lt;/strong&gt; is in huge demand, and a domain dedicated program would help serving the needs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-10-03T13:43:46.043" LastActivityDate="2015-10-03T13:43:46.043" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8291" PostTypeId="2" ParentId="8286" CreationDate="2015-10-03T11:04:10.170" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://aws.amazon.com/machine-learning/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Amazon Machine Learning&lt;/strong&gt;&lt;/a&gt; is a tool, which I use for feature engineering some times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Amazon AWS services have shown a lot of promise and standard, I would definitely count on Amazon ML, with it's prospects and promises for making the workflow of data scientists simpler. But as of now, it's still small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, as you asked for a tool for feature engineering, so this is one of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some &lt;a href=&quot;https://aws.amazon.com/machine-learning/faqs/&quot; rel=&quot;nofollow&quot;&gt;FAQ's&lt;/a&gt; about/for using Amazon ML.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-03T11:04:10.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8292" PostTypeId="2" ParentId="8277" CreationDate="2015-10-03T14:27:49.387" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I was thinking to calculate the average amount of money, say for the&#xA;  last ten transactions, and check how far is the next transaction&#xA;  amount from the average. Too much deviation would signal an anomaly.&#xA;  But this does not sound much, does it?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A typical outlier detection approach. This would work in most cases. But, as the problem statement deals with credit card fraud detection, the detection technique/algorithm/implementation should be more robust.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to have a look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Mahalanobis_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Mahalanobis Distance&lt;/strong&gt;&lt;/a&gt; metric for this type of outlier detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to the algorithms for fraud detection, I would point out to the standards used in the industry (as I have no experience in this, but felt these resources would be useful to you).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check &lt;a href=&quot;https://datascience.stackexchange.com/questions/8099/classifying-transactions-as-malicious/8100#8100&quot;&gt;my answer&lt;/a&gt; for this question. It contains the popular approaches and algorithms used in the domain of fraud detection. The &lt;a href=&quot;https://en.wikipedia.org/wiki/Genetic_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Genetic Algorithm&lt;/a&gt; is the most popular amongst them.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-03T14:27:49.387" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8294" PostTypeId="1" CreationDate="2015-10-03T18:37:16.597" Score="8" ViewCount="10510" Body="&lt;p&gt;How do we use one hot encoding if the number of values which a categorical variable can take is large ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary features) in the training dataset which will immensely increase the complexity and hence the training time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do we deal with such cases ?&lt;/p&gt;&#xA;" OwnerUserId="13155" LastActivityDate="2021-04-24T05:19:41.053" Title="One Hot encoding for large number of values" Tags="|machine-learning|data-mining|classification|dataset|categorical-data|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8295" PostTypeId="2" ParentId="8294" CreationDate="2015-10-03T19:28:29.770" Score="2" Body="&lt;p&gt;You could  try  reducing  the dimmension  of the 56 dummy resulting features, if you have some categories that represent a small proportion compared to the majority by labeling them   the same.&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2015-10-03T19:28:29.770" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8296" PostTypeId="2" ParentId="8277" CreationDate="2015-10-03T20:17:46.670" Score="0" Body="&lt;p&gt;Firstly you should probably be creating models of classes/segments of users (unsupervised clustering).  Otherwise it is difficult to predict what a given user will do.  (More on that further below.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nextly, I think &quot;deviation from recent transactions&quot; is also fundamentally flawed.  Most likely there are time &lt;em&gt;patterns&lt;/em&gt; (time of day, days of the week, working hours, holidays and so on). To understand how to conceptualise time as useful features, see &lt;a href=&quot;https://datascience.stackexchange.com/a/2370/12363&quot;&gt;this excellent answer&lt;/a&gt; on &lt;a href=&quot;https://datascience.stackexchange.com/questions/2368/machine-learning-features-engineering-from-date-time-data&quot;&gt;Machine learning - features engineering from date/time data&lt;/a&gt;  And similarly there are amount patterns, partly having to do with practical reasons (eg by withdrawing 38, one can receive 20, 10, 5 and 1 denominations, although this is not possible in some markets, like the USA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modelling the user is more complicated. You will likely not have enough data on each user, but you can build some user models. (Too few, then the system will make similar predictions for all users, without nuance - eg &gt; $400 always detected as fraud. Too many and there will be sparsity, overfitting, and generally the same problems as having no profile models at all, ie one model per actual user - eg fraud incorrectly detected at every time a given user goes to a new ATM.)  This is basically unsupervised clustering.  (Search for user profile categorisation, user models, user model clustering)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much depends on the data available to you. Perhaps you can be more specific about the scale and scope.  In any case, I wish you luck - banks/Visa do this very poorly right now.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-03T20:17:46.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8297" PostTypeId="1" AcceptedAnswerId="8326" CreationDate="2015-10-04T07:00:55.790" Score="1" ViewCount="128" Body="&lt;p&gt;How do we use a correlation score between two variables for analysing data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of 20 features and need to predict 21st feature. Now is it necessary that correlation between any two features should be close to 1 ? If I have 2 features with corr score close to -1, then does this mean that they are contradicting and thereby decreasing the accuracy ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do we use a correlation score in analysis ?&lt;/p&gt;&#xA;" OwnerUserId="13155" LastActivityDate="2015-10-06T13:01:59.363" Title="Use of Correlation score" Tags="|dataset|feature-selection|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8298" PostTypeId="1" AcceptedAnswerId="8314" CreationDate="2015-10-04T08:02:57.263" Score="2" ViewCount="1029" Body="&lt;p&gt;As an example, let's say I have a very simple data set. I am given a csv with three columns, user_id, book_id, rating. The rating can be any number 0-5, where 0 means the user has &lt;em&gt;NOT&lt;/em&gt; rated the book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I randomly pick three users, and I get these feature/rating vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Martin: $&amp;lt;3,3,5,1,2,3,2,2,5&amp;gt;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jacob: $&amp;lt;3,3,5,0,0,0,0,0,0&amp;gt;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Grant: $&amp;lt;1,1,1,2,2,2,2,2,2&amp;gt;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The similarity calculations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;+--------------+---------+---------+----------+&#xA;|              | M &amp;amp; J   | M &amp;amp; G   | J &amp;amp; G    |&#xA;+--------------+---------+---------+----------+&#xA;| Euclidean    |   6.85  |    5.91 |    6.92  |&#xA;+--------------+---------+---------+----------+&#xA;| Cosine       |    .69  |    .83  |      .32 |&#xA;+--------------+---------+---------+----------+&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my expectation of similarity is that Martin and Jacob would be the most similar. I would expect this because they have EXACTLY the same ratings for the books that both of them have rated. But we end up finding that Martin and Grant are the most similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand mathematically how we get to this conclusion, but I don't understand how I can rely on Cosine Angular distance or Euclidean distance as a means of calculating similarity, if this type of thing occurs. For what interpretation are Martin and Grant more similar than Martin and Jacob?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thought I had was to just calculate Euclidean distance, but ignore all books for which one user hasn't rated the book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then end up with this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;+--------------+---------+---------+----------+&#xA;|              | M &amp;amp; J   | M &amp;amp; G   | J &amp;amp; G    |&#xA;+--------------+---------+---------+----------+&#xA;| Euclidean    |   0     |    5.91 |    6.92  |&#xA;+--------------+---------+---------+----------+&#xA;| Cosine       |    .69  |    .83  |      .32 |&#xA;+--------------+---------+---------+----------+&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course now I have a Euclidean distance of 0, which fits what I would expect of the recommender system. I see many tutorials and lectures use Cosine Angular distance to ignore the unrated books, rather than use Euclidean and ignore them, so I believe this must not work in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to experiment a little, I adjusted Jacob's feature vector to be much more similar:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jacob: $&amp;lt;3,3,5,1,2,3,2,0,0&amp;gt;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I calculate Cosine Angular distance with Martin, I still only get .82! Still less similar than Martin and Grant, yet by inspection I would expect these two to be very similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could somebody help explain where my thinking is wrong, and possibly suggest another similarity measure?&lt;/p&gt;&#xA;" OwnerUserId="13168" LastEditorUserId="13168" LastEditDate="2015-10-04T08:45:58.567" LastActivityDate="2015-10-05T13:27:27.030" Title="Correctly interpreting Cosine Angular Distance Similarity &amp; Euclidean Distance Similarity" Tags="|machine-learning|recommender-system|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8299" PostTypeId="2" ParentId="5973" CreationDate="2015-10-04T10:47:49.983" Score="3" Body="&lt;p&gt;After the Data Munging, this is the most difficult task on a prediction model. However, in order to answer it, we need more details. What do you mean by the &quot;best model&quot;? Do you want accuracy and long training time? Do you need something really fast with lower accuracy? Something between the two of them? What are your features? Have you just taken them or have you created new features from them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, I suggest that you should spend some time and read this perfect tutorial from &lt;a href=&quot;https://azure.microsoft.com/en-us/documentation/articles/machine-learning-algorithm-choice/&quot; rel=&quot;nofollow&quot;&gt;Microsoft&lt;/a&gt; about Machine Learning. A part of the tutorial to understand what I mean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regression&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Ordinal regression: Data in rank ordered categories&lt;/li&gt;&#xA;&lt;li&gt;Poisson regression: Predicting event counts&lt;/li&gt;&#xA;&lt;li&gt;Fast forest quantile regression: Predict a distribution&lt;/li&gt;&#xA;&lt;li&gt;Linear regression: Fast training, linear model&lt;/li&gt;&#xA;&lt;li&gt;Bayesian linear regression: Linear model, small data sets&lt;/li&gt;&#xA;&lt;li&gt;Neural Network regression: Accuracy, long training time&lt;/li&gt;&#xA;&lt;li&gt;Decision forest regression: Accuracy, fast training&lt;/li&gt;&#xA;&lt;li&gt;Boosted decision tree regression: Accuracy, fast training&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When I have a similar question and I don't know which one to choose, I usually end up to 3-4 different algorithms based on the cheat sheet of Microsoft or the one from the &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;, try them all and choose one or two of them with the best results. &lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-04T10:47:49.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8300" PostTypeId="2" ParentId="6735" CreationDate="2015-10-04T11:33:32.093" Score="2" Body="&lt;p&gt;What you described is the Filter Method. One of the three methods of Feature Selection. The most common mistake that one could do is to make the feature selection as an independent step on your whole process and then decide which model he will use. The most appropriate way to do it is to include your Feature Selection process in your accuracy test and try different kind of feature selections. The best way for the feature selection is the domain knowledge. If you don't have it, then you start working with one of the three methods, like the Filter Method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need to do is to test the chisquare or mutual information on your features with your label column. This will give you which of the features have low effect on your prediction and you can remove it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find a briefly detailed article about Feature Selection &lt;a href=&quot;http://machinelearningmastery.com/an-introduction-to-feature-selection/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume3/guyon03a/guyon03a.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-04T11:33:32.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8301" PostTypeId="1" CreationDate="2015-10-04T14:57:05.073" Score="2" ViewCount="528" Body="&lt;p&gt;I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say my data consists of observations made up of visitors to a website. The data is in clickstream/weblog format and so is made up of users cookie data. Lets say I can identify unique users via their IP address (as a basic example). How should I go about preparing my data so that it can be segmented to find users with similar behaviours? One of my thoughts around this, is that because the data is event-driven, the same user can obviously appear multiple times in the data even though it may all be related to the same session of that user. How are these types of problems tackled so that you can perform segmentation based on user behaviour?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="13175" LastActivityDate="2016-02-06T15:22:56.443" Title="How should clickstream data be prepared before user segmentation can be performed?" Tags="|data-mining|clustering|preprocessing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8302" PostTypeId="1" CreationDate="2015-10-04T16:39:09.470" Score="1" ViewCount="167" Body="&lt;p&gt;I have a training data set, which is something like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hour   windspeed   pollution&#xA;0              5           8&#xA;1              6           9&#xA;2              3           1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using scipy's RandomForestRegressor to estimate &lt;em&gt;pollution&lt;/em&gt; given &lt;em&gt;windspeed, hour&lt;/em&gt;. It is doing &quot;ok&quot;, with $r^2 = 0.7$ and $rmse = 35$. However, I would like to include the previous hours &lt;em&gt;pollution values&lt;/em&gt; as additional predictors, as in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hour   windspeed   pollution   pollution@t-1  polluion@t-2 ... pollution@t-n&#xA;0              5           8              NA           NA                 &#xA;1              6           9               8           NA&#xA;2              3           1               9           8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Eventually, I will be applying this to future weather conditions, estimating pollution, where pollution can only be known at time zero:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hour   windspeed  pollution  p@t-1  p@t-2  p@t-3  p@t-4    ...&#xA;1              5          ?     34     28     25     ...   ...         &#xA;2              6          ?     NA     34     28      25   ... &#xA;3              3          ?     NA     NA     34      28   ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should I best go about including this in a Random Forest? Since RFs don't tolerate null values, how best to impute them? Since '0' is not appropriate (given that 0 is likely very far from the current value), I was thinking that perhaps NaNs could be filled with the nearest known value, as in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hour   windspeed  pollution  p@t-1  p@t-2  p@t-3  p@t-4    ...&#xA;1              5          ?     34     28     25     ...   ...         &#xA;2              6          ?     34     34     28      25   ... &#xA;3              3          ?     34     34     34      28   ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13179" LastActivityDate="2020-08-03T13:32:10.827" Title="Using previous hour's value in time series data for inclusion in random forest" Tags="|python|regression|random-forest|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8303" PostTypeId="1" CreationDate="2015-10-04T16:49:46.320" Score="1" ViewCount="191" Body="&lt;p&gt;I am trying to understand how images are coded for classification/recognition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I have images that contain animals (e.g. dogs, cats, birds) -- does each image have a label for that animal or does the label only apply to the portion of the image with the animal in it? If the entire images are coded does it take a classifier longer to learn?&lt;/p&gt;&#xA;" OwnerUserId="5103" LastActivityDate="2015-12-03T20:27:49.133" Title="In image classification/recognition how are images coded?" Tags="|machine-learning|neural-network|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8305" PostTypeId="1" CreationDate="2015-10-04T17:34:41.407" Score="2" ViewCount="156" Body="&lt;p&gt;I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fmla &amp;lt;- as.formula(paste(&quot;ex ~ &quot;, paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= &quot;+&quot;)))&#xA;&amp;gt; bosque &amp;lt;- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)&#xA;&amp;gt; bosque&#xA;&#xA;Call:&#xA; randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE) &#xA;               Type of random forest: classification&#xA;                     Number of trees: 1000&#xA;No. of variables tried at each split: 12&#xA;&#xA;        OOB estimate of  error rate: 15.81%&#xA;Confusion matrix:&#xA;     No Si class.error&#xA;No 3999  1     0.00025&#xA;Si  758 42     0.94750&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As we see, the out of bag error for the level &quot;SI&quot; is 0.94750. However If I use the test set to get a sense of the error, the result is very different:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res.arbol &amp;lt;- predict(bosque,test.fullarbol,type=&quot;class&quot;)&#xA;&amp;gt; summary(res.arbol)&#xA;  No   Si &#xA;7761   43 &#xA;&amp;gt; table(res.arbol,test.fullarbol$ex)&#xA;&#xA;res.arbol   No   Si&#xA;       No 6937  824&#xA;       Si    4   39&#xA;&amp;gt; prop.table(table(res.arbol,test.fullarbol$ex),1)&#xA;&#xA;res.arbol         No         Si&#xA;       No 0.89382811 0.10617189&#xA;       Si 0.09302326 0.90697674&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we see thae error rate for the test set in the class &quot;Si&quot; is very low ,  it's equal to 0.093 and It doesn´t make sense to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?&lt;/p&gt;&#xA;" OwnerUserId="1018" LastEditorUserId="1018" LastEditDate="2015-10-04T17:50:26.207" LastActivityDate="2015-10-05T13:43:58.903" Title="Case when Out of Bag Error and Test error differs a lot in Random Forest" Tags="|machine-learning|predictive-modeling|random-forest|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8307" PostTypeId="2" ParentId="8303" CreationDate="2015-10-04T19:43:27.580" Score="1" Body="&lt;p&gt;This will depend on the type of task you want to perform:&#xA;Object recognition is a wide task than can be approached in different ways such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[Multi-part OR][1]&lt;/li&gt;&#xA;&lt;li&gt;[Multi-scale OR][2]&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jmlr.org/papers/volume11/escalera10a/escalera10a.pdf&quot; rel=&quot;nofollow&quot;&gt;Multi-class OR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;[Multi-label OR][4]&lt;/li&gt;&#xA;&lt;li&gt;[Multi-modal OR]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Each approach (and its combinations) uses different representations for feature vectors extracted from the images. Most typical representations try to be invariant to certain conditions on the images depending on it purpose, such as scale, rotation and illumination changes.&#xA;Some successful representations are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bag of Visual Words [5]&lt;/li&gt;&#xA;&lt;li&gt;Histogram of visual words [8]&lt;/li&gt;&#xA;&lt;li&gt;Haar-like features [2]&lt;/li&gt;&#xA;&lt;li&gt;Statistical-based structures (Active shape models, active appearance models) [9][10]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With the rise of deep networks, surprising results are being obtained feeding the images just as the raw pixels that represent them [6]. However, you will likely need hundreds of thousands of images to proper train a network that can abstract features to further recognize objects. You can actually take a look to the highly successful VGG Convolutional Network implementation at [Find actual links here][2]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The previous approaches will typically required that you have each example labeled with target value (cat, dog, etc). In case of deep learning you are being able to learn first representation of images with unsupervised data (not labeled) and the being able to train a classifier on top of the network in a supervised way (with labeled data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find most of the references below at: [Google Scholar][3]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.&#xA;Spatial weighting for bag-of-visual-words and its application in&#xA;content-based image retrieval. Advances in Knowledge Discovery&#xA;and Data Mining, 2009&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]: [Viola and Jones(2004)] Paul Viola and Michael J. Jones. Robust realtime&#xA;face detection. Int. J. Comput. Vision, 57(2):137–154, May 2004.&#xA;ISSN 0920-5691.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4]: [Jia Deng and Fei-Fei(2014)] Jonathan Krause Michael Bernstein&#xA;Alex Berg Jia Deng, Olga Russakovsky and Li Fei-Fei. Scalable&#xA;multi-label annotation. ACM Conference on Human Factors in&#xA;Computing Systems, 2014.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[5]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.&#xA;Spatial weighting for bag-of-visual-words and its application in&#xA;content-based image retrieval. Advances in Knowledge Discovery&#xA;and Data Mining, 2009.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[7]: [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex&#xA;Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet&#xA;classification with deep convolutional neural networks. In&#xA;P. Bartlett, F.c.n. Pereira, C.j.c. Burges, L. Bottou, and K.q.&#xA;Weinberger, editors, Advances in Neural Information Processing&#xA;Systems 25, pages 1106–1114. 2012.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[8]: [Lowe(2004)] David G. Lowe. Distinctive image features from scaleinvariant&#xA;keypoints. Int. J. Comput. Vision, 60(2):91–110, November&#xA;2004.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[9]: [van Ginneken(2002)] A.F. Staal J.J. ter Haar Romeny B.M. van Ginneken,&#xA;B.Frangi. Active shape model segmentation with optimal&#xA;features. Medical Imaging, IEEE Transactions on. Volume 21, 2002.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[10]: [Matthews and Baker(2004)] Iain Matthews and Simon Baker. Active&#xA;appearance models revisited. International Journal of Computer&#xA;Vision. Volume 60., 2004.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2015-10-04T19:43:27.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8309" PostTypeId="1" CreationDate="2015-10-04T23:06:52.520" Score="3" ViewCount="2598" Body="&lt;p&gt;I have a very large csv file which Apple Numbers won't open. I can open it with TextEdit, but each row is so long it forms multiple lines in the document and makes the document difficult to understand. Is there a tool for opening a csv and exploring different rows and columns?&lt;/p&gt;&#xA;" OwnerUserId="13182" LastEditorUserId="31513" LastEditDate="2017-05-25T10:35:24.827" LastActivityDate="2017-05-25T10:35:24.827" Title="Quick way to visually explore data?" Tags="|visualization|csv|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8310" PostTypeId="2" ParentId="8305" CreationDate="2015-10-05T07:54:19.927" Score="1" Body="&lt;p&gt;What type of sampling method did you use to split the data? I'm guessing, by looking at the ratio of classes in your OOB confusion matrix and your test set confusion matrix that it wasn't stratified, is this correct? For instance, the class ratio in the OOB data is 4000:800 whereas in the test set it is 7761:43. As a starting point, you may want to split your data using stratified sampling so that the classes are fairly represented (in a statistical sense) in both your training data and test data. The ratio of classes in the test set should be identical or almost the same as the ratio used in the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this would be a good starting point to figuring out the problem.&lt;/p&gt;&#xA;" OwnerUserId="13176" LastEditorUserId="13176" LastEditDate="2015-10-05T13:43:58.903" LastActivityDate="2015-10-05T13:43:58.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8311" PostTypeId="2" ParentId="8309" CreationDate="2015-10-05T07:56:16.367" Score="2" Body="&lt;p&gt;Unfortunately, the highest number of rows in both Numbers and Microsoft excel is 1,048,576. So, if you have a file bigger than this, you are not able to open it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Connect your file in an SQL database and work with it from there.&lt;/li&gt;&#xA;&lt;li&gt;Use another tool like &lt;a href=&quot;http://delimitware.com/&quot; rel=&quot;nofollow&quot;&gt;delimitware&lt;/a&gt; which let you open files up to 2 billions rows.&lt;/li&gt;&#xA;&lt;li&gt;Use Python Pandas with &lt;code&gt;pandas.read_csv('filename')&lt;/code&gt; and then &lt;code&gt;df.head(5)&lt;/code&gt; to check the first 5 rows.&lt;/li&gt;&#xA;&lt;li&gt;Split the file in smaller files with less than 1m rows per each chunk and use Numbers or Microsoft Excel to read it.&lt;/li&gt;&#xA;&lt;li&gt;Upload it to Azure ML and work on a new experiment with it. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-05T07:56:16.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8312" PostTypeId="2" ParentId="8309" CreationDate="2015-10-05T09:26:13.163" Score="1" Body="&lt;p&gt;Since you are on OS X you can use the terminal to explore your file without storing everything in memory. The &lt;code&gt;head -5 filename.csv&lt;/code&gt; command for example will display the first 5 lines of your file. You could even do &lt;code&gt;head -10 filename.csv &amp;gt; newfile.csv&lt;/code&gt; to store the first 10 lines in a new file and open it with Apple Numbers to examine what's inside. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For anything more than that I'm afraid you would need to write a script to run through each line of the file one by one and compute the relevant descriptive statistics.&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-10-05T09:26:13.163" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8313" PostTypeId="1" AcceptedAnswerId="8325" CreationDate="2015-10-05T10:35:09.597" Score="2" ViewCount="716" Body="&lt;p&gt;I'm working with &lt;a href=&quot;https://www.cs.cmu.edu/~./enron/&quot; rel=&quot;nofollow&quot;&gt;Enron email data set&lt;/a&gt;, and try to get some psychological intentions about email marketing. I know that we can derive &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;nofollow&quot;&gt;Sentimental analysis like emotions and mode&lt;/a&gt; and opinion mining like positive or negative through text data. what else can I do regarding human psychological intentions?&lt;/p&gt;&#xA;" OwnerUserId="9035" LastEditorUserId="609" LastEditDate="2015-10-05T17:05:01.810" LastActivityDate="2015-10-06T12:45:01.473" Title="How can I infer psychological intentions from email corpus using text mining?" Tags="|data-mining|nlp|text-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8314" PostTypeId="2" ParentId="8298" CreationDate="2015-10-05T13:27:27.030" Score="0" Body="&lt;p&gt;If you look at the definitions of the two distances, cosine distance is the normalized dot product of the two vectors and euclidian is the square root of the sum of the squared elements of the difference vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cosine distance between M and J is smaller than between M and G because the normalization factor of M's vector still includes the numbers for which J didn't have any ratings. Even if you make J's vector more similar, like you did, the remaining numbers of M (2 and 5) get you the number you get.&#xA;The number for M and G is this high because they both have non-zeroes for all the books. Even though they seem quite different, the normalization factors in the cosine are more &quot;neutralized&quot; by the non-zeroes for corresponding entries in the dot product. Maths don't lie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The books J didn't rate will be ignored if you make their numbers zero in the computation of the normalization factor for M. Maybe the fault in your thinking is that the books J didn't rate should be 0 while they shouldn't be any number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, for recommendation systems, I would like to refer to matrix factorization.&lt;/p&gt;&#xA;" OwnerUserId="13027" LastActivityDate="2015-10-05T13:27:27.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8315" PostTypeId="2" ParentId="8313" CreationDate="2015-10-05T13:46:24.837" Score="2" Body="&lt;p&gt;&lt;strong&gt;&lt;em&gt;DISCLAIMER :&lt;/strong&gt; I'm not sure that this will be the answer you are looking for but I'll give a try.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Noam Chomsky&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Language is a process of free creation; its laws and principles are&#xA;  fixed, but the manner in which the principles of generation are used&#xA;  is free and infinitely varied. Even the interpretation and use of&#xA;  words involves a process of free creation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;On one hand, data or text mining the combined large lexical data resources of linguistics and psychology suggests that the linguistic variable of 'word frequency' should be rehabilitated in psychology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you have to remember that, &lt;strong&gt;&lt;em&gt;correlation does not imply causation&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This said, there is not much way to apply cross text mining with psychological intuition. In other words, use cases are not too many.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can always try quantitative text analysis. It is a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of, usually, a casual personal text for the purpose of psychological profiling etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And by psychological profiling, I'm talking about offender profiling per example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get more information upon that on the following websites :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://criminvestresearch.com/&quot; rel=&quot;nofollow&quot;&gt;Criminal Investigative Research &amp;amp; Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.psychlotron.org.uk/&quot; rel=&quot;nofollow&quot;&gt;http://www.psychlotron.org.uk/&lt;/a&gt; &lt;a href=&quot;http://www.psychlotron.org.uk/newResources/criminological/A2_AQB_crim_whatIsProfiling.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt;  about criminal profiling.&lt;/li&gt;&#xA;&lt;li&gt;Here is a very intersting &lt;a href=&quot;http://www.newyorker.com/magazine/2007/11/12/dangerous-minds&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; about criminal profiling for beginners.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also try to take a look at this &lt;a href=&quot;http://begriffs.com/posts/2015-02-25-text-mining-in-r.html&quot; rel=&quot;nofollow&quot;&gt;presentation&lt;/a&gt; by Harold Baize who talks about the intersection of text mining in the content analysis of psychology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this can help!&lt;/p&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2015-10-05T13:46:24.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8316" PostTypeId="2" ParentId="8313" CreationDate="2015-10-05T14:05:41.850" Score="0" Body="&lt;p&gt;Actually, there isn't any limit of what you can do, if you have the right data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can remember a startup that was working on identify anxiety and suicidal thoughts on Tweets and Facebook status. First, they created a system where a user could raise a flag in case of a related Tweet/status about anxiety. They used those data to create a labeled dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have labeled data and you know the psychological intentions from them, then you can just train a model to classify all the other dataset to one of these labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not aware if there is any public dataset with already labeled text, but you can either search for one or create your own.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-10-05T14:05:41.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8317" PostTypeId="2" ParentId="8309" CreationDate="2015-10-05T17:26:14.550" Score="2" Body="&lt;p&gt;You may install &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; - you are limited only with your RAM&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ## read csv file&#xA; df &amp;lt;- read.csv(&quot;l.csv&quot;)&#xA;&#xA; ## column names&#xA; &amp;gt; colnames(df) &#xA; [1] &quot;a&quot; &quot;b&quot;&#xA;&#xA; &amp;gt; head(df)&#xA;     a  b&#xA;   1 1  1&#xA;   2 2  4&#xA; ...&#xA;&#xA; # quick overview&#xA; &amp;gt; summary(df)&#xA;        a                 b            &#xA;  Min.   :    1.0   Min.   :        1  &#xA;  1st Qu.: 2500.8   1st Qu.:  6253751  &#xA;  Median : 5000.5   Median : 25005000  &#xA;  Mean   : 5000.5   Mean   : 33338334  &#xA;  3rd Qu.: 7500.2   3rd Qu.: 56253750  &#xA;  Max.   :10000.0   Max.   :100000000  &#xA;&#xA; &amp;gt; plot(df2,type='l')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WGUcp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WGUcp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check docu for other graphic and statistics features.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2015-10-06T09:01:10.390" LastActivityDate="2015-10-06T09:01:10.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8319" PostTypeId="2" ParentId="8309" CreationDate="2015-10-05T20:15:43.173" Score="5" Body="&lt;p&gt;When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application, which is not constrained by arbitrary limits like 2&lt;sup&gt;20&lt;/sup&gt; rows by 2&lt;sup&gt;14&lt;/sup&gt; columns. (Of course, you are still constrained by the physical memory and the way that a particular language addresses it: R and Python tend to be more resource-intensive than C).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me give you an example using Python. You can install Python for your system from the &lt;a href=&quot;https://www.python.org/downloads/&quot; rel=&quot;nofollow noreferrer&quot;&gt;official site&lt;/a&gt;. If you install one of the latest versions, &lt;a href=&quot;http://python-packaging-user-guide.readthedocs.org/en/latest/installing/&quot; rel=&quot;nofollow noreferrer&quot;&gt;you will already&lt;/a&gt; get &lt;code&gt;pip&lt;/code&gt; as an installer of further Python packages. Install Pandas by starting a terminal (on OS X, there is a 'Terminal' application, on Windows this used to be a MS-DOS window ...), and typing into that terminal:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install pandas&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then start Python in an interactive session by typing &lt;code&gt;python&lt;/code&gt; in a terminal. In the Python shell, you can read the data as mentioned in &lt;strong&gt;@tasos&lt;/strong&gt; 's answer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt;&amp;gt; df = pd.read_csv('name_of_your_file.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here you can use the &lt;code&gt;describe&lt;/code&gt; method of the &lt;code&gt;df&lt;/code&gt; DataFrame to get some statistics about the read data. These numbers are for a randomly generated data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df.describe()&#xA;          0         1         2&#xA;count  3.000000  3.000000  3.000000&#xA;mean  -0.150869  0.444380 -0.117066&#xA;std    0.751421  0.697880  0.565328&#xA;min   -1.017424 -0.356764 -0.506030&#xA;25%   -0.386514  0.206466 -0.441313&#xA;50%    0.244396  0.769696 -0.376595&#xA;75%    0.282409  0.844952  0.077416&#xA;max    0.320422  0.920208  0.531427&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can generate plots for different columns, one against the other, using the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/visualization.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pandas plotting functions&lt;/a&gt;. Check the many examples in the linked documentation. Here is a small example from there:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;import bumpy as np&#xA;&#xA;df = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000),&#xA;                   'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])&#xA;plt.figure()&#xA;df.plot(kind='hist', alpha=0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Usually, you will need to use another call more in order to save it to a file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.save('output.png')  ## or *.svg, *.pdf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the file you should find something similar to this figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/OR9HL.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="1367" LastEditorUserId="15620" LastEditDate="2017-04-20T09:20:36.070" LastActivityDate="2017-04-20T09:20:36.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8320" PostTypeId="2" ParentId="5209" CreationDate="2015-10-05T20:39:30.453" Score="1" Body="&lt;p&gt;Given that you run a test against labeled data (the &lt;code&gt;-testFile&lt;/code&gt; option), then Stanford NER will report entity level F1 scores, and a micro-average entity-level F1 at the end of the run.&lt;/p&gt;&#xA;" OwnerUserId="13201" LastActivityDate="2015-10-05T20:39:30.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8321" PostTypeId="2" ParentId="6735" CreationDate="2015-10-06T07:54:23.917" Score="0" Body="&lt;p&gt;I agree with the answer by Tasos. Another approach to your work that you may wish to take, is using loglinear analysis on your categorical variables. This is a useful ploy whenever you are dealing with three or more categorical variables. Put simply, it's basically regression for categorical variables and is very simple to employ.&lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-06T07:54:23.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8322" PostTypeId="1" AcceptedAnswerId="8324" CreationDate="2015-10-06T10:51:52.883" Score="8" ViewCount="3386" Body="&lt;p&gt;What are all the options available for filling in missing data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do we deal with missing values if they are are lot of them?&lt;/p&gt;&#xA;" OwnerUserId="13155" LastEditorUserId="29169" LastEditDate="2019-06-08T03:14:47.970" LastActivityDate="2019-06-08T03:14:47.970" Title="Filling missing data with other than mean values" Tags="|data-mining|missing-data|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="8323" PostTypeId="2" ParentId="8322" CreationDate="2015-10-06T12:17:53.297" Score="2" Body="&lt;p&gt;There is a difference between data with missing values and sparse data. Missing values are generally there because of invalid input, loss or error during data collection or are created when cleaning or processing data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these values are very less in number, the corresponding instances can be ignored or if are around 5-10% of the data, can be filled using various methods (carry forward last observation, fill with mean/median, interpolate the data etc). If you are working in Python, go through Pandas documentation for &lt;em&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/missing_data.html&quot; rel=&quot;nofollow&quot;&gt;Working with Missing Values&lt;/a&gt;&lt;/em&gt;, to learn in detail about these options (even if you aren't working in Python, this is a good read).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if your data set has a large number of missing values (say ~ &gt;30% ), then the data is &lt;strong&gt;sparse&lt;/strong&gt;. Such data sets create various bias in your modelling, and there are special ways to deal with them, though I don't about them much.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastEditorUserId="11097" LastEditDate="2015-10-08T09:13:57.673" LastActivityDate="2015-10-08T09:13:57.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8324" PostTypeId="2" ParentId="8322" CreationDate="2015-10-06T12:31:04.540" Score="5" Body="&lt;p&gt;There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the &lt;em&gt;nature&lt;/em&gt; of your missing data. When you understand in detail why data is missing, the probability of coming up with a good solution will be much higher. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to start your investigation of missing data by finding out whether you have &lt;em&gt;informative&lt;/em&gt; or &lt;em&gt;non-informative&lt;/em&gt; missings. The first category is produced by random data loss; in this case, the observations with missing values are no different from the ones with complete data. As for &lt;em&gt;informative&lt;/em&gt; missing data, this one tells you something about your observation. A simple example is a customer record with a missing contract cancellation date meaning that this customer's contract has not been cancelled so far. You usually don't want to fill in informative missings with a mean or a median, but you may want to generate a separate feature from them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also find out that there are several kinds of missing data, being produced by different mechanisms. In this case, you might want to produce default values in different ways.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2015-10-06T12:31:04.540" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8325" PostTypeId="2" ParentId="8313" CreationDate="2015-10-06T12:45:01.473" Score="1" Body="&lt;p&gt;If you have a broad meaning of &quot;intentions&quot; in mind, you might be interested in research showing that a person's personality (in the sense of the &quot;big five&quot; psychological personality theory) can be inferred with remarkable accuracy from facebook likes. Original research showing this can be found &lt;a href=&quot;http://www.pnas.org/content/110/15/5802.full.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.pnas.org/content/112/4/1036.full.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Obviously, facebook likes are not the same as the textual information you work with, but it may be possible to infer certain likes and dislikes from the text.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2015-10-06T12:45:01.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8326" PostTypeId="2" ParentId="8297" CreationDate="2015-10-06T13:01:59.363" Score="0" Body="&lt;p&gt;Correlation should be as less as possible between different features, because correlated features mean that those features are giving out same kind of information/trend for the predictor to learn. Thus only one of them is actually useful for prediction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keeping more number of uninformative features (correlated features) would result in degraded accuracy if your sample size is similar to you feature set size. Feature selection using Recursive Feature elimination or PCA etc. can help you reduce your feature set to optimal size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We calculate correlation score in predictive analysis between features and Target variable. When using linear regression to model a data set, we first see if the plot between different features and target variable values follow an upward (+ve correlation) or downward trend (-ve correlation) and not scattered randomly. If such a relationship exists then regression modelling on the data would work well.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2015-10-06T13:01:59.363" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8329" PostTypeId="2" ParentId="8322" CreationDate="2015-10-06T15:24:38.363" Score="4" Body="&lt;p&gt;When it comes to missing data, there are many different methods of filling these values. However, the imputation method you choose, depends largely on the amount of missing data and the type of variable. For example, you won't impute the mean value for missing categorical data, you would choose the mode instead. No matter which method you choose, there will be some bias associated with it. One method which does a good job at reducing the bias associated with imputing missing values, is multiple imputation. It can be quite a long-winded approach but it is the most sound approach I've seen so far to imputing large amounts of missing values. I believe there may be an R library for multiple imputation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, another alternative may be that if variable x has 50% missing data for example, there may be a good explanation as to why this is. Rather than trying to impute it or lose the information associated with the variable, it can sometimes be useful to create a new variable, called variable_x_flag_missing. This would be a binary indicator variable where an observation is coded as 1 if variable x contains a missing value and coded as a 0 if it does not. &lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-06T15:24:38.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8331" PostTypeId="1" AcceptedAnswerId="8335" CreationDate="2015-10-06T18:45:53.400" Score="3" ViewCount="156" Body="&lt;p&gt;Suppose we collect data for 100,000 tosses of a fair coin and record &quot;Heads&quot; or &quot;Tails&quot; as the value for the attribute &lt;em&gt;outcome&lt;/em&gt; and also record the time, temprature and other irrelevant attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that the outcome of each toss is random so there should be no way of predicting future &lt;em&gt;unlabeled&lt;/em&gt; data instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do learning algorithms (support vector machines, for example) behave when we apply them on random data such as this?&lt;/p&gt;&#xA;" OwnerUserId="11044" LastActivityDate="2015-10-06T21:02:32.477" Title="Behaviour of Learning Algorithms on Random Data" Tags="|machine-learning|data-mining|classification|predictive-modeling|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8332" PostTypeId="1" CreationDate="2015-10-06T20:03:15.340" Score="1" ViewCount="801" Body="&lt;p&gt;So I'm a newcomer to the world of neural networks, and I've been getting a little familiar with the field and have started playing with my own networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using &lt;a href=&quot;https://github.com/Lasagne/Lasagne&quot; rel=&quot;nofollow&quot;&gt;Lasagne&lt;/a&gt;, and I'm finding that the training of my NN is taking an infeasible amount of time.  Maybe the problem I'm looking at is simply biting more than my computer can chew; I just don't know, because I'm not familiar enough with how these things go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to train a 1D convolutional NN, which has two 1D convolution layers, and two max-pooling layers, and two fully connected layers. All in all, the NN has around 95,000 parameters (weights + biases).  I'm running this on an Amazon EC2 GPU instance.  What I'm finding is that during training, it's taking 0.3 sec/sample.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By contrast, when I run the Lasagne MNIST CNN example, it's taking about 0.0003 sec/sample, 1000 times faster!  Now to be fair, my input samples are 18000x3 dimensional, and the MNIST samples are 28x28 dimensional; but the difference there is a factor of 68.  And the MNIST CNN actually has more parameters than mine, with 160,000 or so parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this indicate anything wrong with what I'm doing?  &lt;/p&gt;&#xA;" OwnerUserId="13216" LastEditorUserId="11097" LastEditDate="2015-10-08T09:58:30.900" LastActivityDate="2015-10-15T01:10:05.927" Title="Is my Lasagne/Theano neural network running too slow?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8334" PostTypeId="1" AcceptedAnswerId="8366" CreationDate="2015-10-06T21:01:20.730" Score="2" ViewCount="1742" Body="&lt;p&gt;How do we find maximum depth of Random Forest if we know the number of features ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is needed for regularizing random forest classifier.&lt;/p&gt;&#xA;" OwnerUserId="13155" LastActivityDate="2015-10-09T13:01:23.863" Title="finding maximum depth of random forest given the number of features" Tags="|machine-learning|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8335" PostTypeId="2" ParentId="8331" CreationDate="2015-10-06T21:02:32.477" Score="4" Body="&lt;p&gt;They will of course still learn some best decision boundary. We know it will be meaningless, but there will still be better and best coefficients for the algorithm to learn when fitting to this particular instance of data from this random process. It may produce better than 50% accuracy on the data set, but of course this is purely due to overfitting whatever the data happens to be. It will not predict future outcomes with more than 50% accuracy.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-10-06T21:02:32.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8337" PostTypeId="2" ParentId="8322" CreationDate="2015-10-06T21:43:50.660" Score="1" Body="&lt;p&gt;If the values are missing at random &lt;em&gt;and&lt;/em&gt; you are sure that your data matrix is of low rank, you can use nuclear norm basis pursuit method (also known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_completion&quot; rel=&quot;nofollow&quot;&gt;matrix completion&lt;/a&gt;). The method (among others) is implemented in &lt;a href=&quot;http://cvxr.com/tfocs/download/&quot; rel=&quot;nofollow&quot;&gt;TFOCS&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many real-world applications, the data matrix has rarely full rank, so the assumption of low-rank matrix can be acceptable. On the other hand, the values might not be missing truly at random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to use Singular Spectrum Analysis (&lt;a href=&quot;http://www.jds-online.com/files/JDS-396.pdf&quot; rel=&quot;nofollow&quot;&gt;SSA&lt;/a&gt;), also known as the Caterpillar algorithm. It can be used for time-series data with missing values. This algorithm is not very well-known but in literature it is sometimes called &quot;PCA for time-series data&quot;.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="11097" LastEditDate="2015-10-08T08:13:26.287" LastActivityDate="2015-10-08T08:13:26.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8338" PostTypeId="1" CreationDate="2015-10-06T21:55:44.410" Score="4" ViewCount="996" Body="&lt;p&gt;I'm trying to write a framework to compare a set of labels such as (for a sample of 5 yes/no answers to a question)  &lt;code&gt;[0, 1, 1, 1, 0]&lt;/code&gt; to a series of features to determine correlation.  For numerical non-sparse features, like &quot;number of words&quot; or &quot;average word length&quot;, I know I can use a variance-covariance matrix and get a sense for whether or not &quot;number of words&quot; or &quot;average word length&quot; is an informative feature for a model to answer the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to be able to do the same thing for term frequency (let's say using CountVectorizer in scikit-learn), but the resultant covariance matrix will be rather large and will only indicate whether or not that particular &lt;em&gt;term&lt;/em&gt; is an informative feature.  How do I get some kind of &quot;collapsed&quot; or &quot;aggregate&quot; measure of correlation?  Is this even possible?&lt;/p&gt;&#xA;" OwnerUserId="13226" LastActivityDate="2016-03-09T00:16:33.483" Title="Measure of correlation for term frequency" Tags="|statistics|feature-selection|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8339" PostTypeId="1" AcceptedAnswerId="8346" CreationDate="2015-10-06T23:08:51.327" Score="6" ViewCount="2517" Body="&lt;p&gt;I used following classifiers along with their accuracies:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Random forest - 85 %&lt;/li&gt;&#xA;&lt;li&gt;SVM - 78 %&lt;/li&gt;&#xA;&lt;li&gt;Adaboost - 82%&lt;/li&gt;&#xA;&lt;li&gt;Logistic regression - 80%&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When I used voting from above classifiers for final classification, I got lesser accuracy than the case when I used Random forest alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is this possible? All classifiers are giving more or less same accuracies when used individually, then how does Random Forest outperform their combined result ?&lt;/p&gt;&#xA;" OwnerUserId="13155" LastEditorUserId="5177" LastEditDate="2015-10-29T08:10:54.977" LastActivityDate="2015-10-29T09:11:50.207" Title="Voting combined results from different classifiers gave bad accuracy" Tags="|machine-learning|classification|logistic-regression|random-forest|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8340" PostTypeId="1" CreationDate="2015-10-06T23:42:44.063" Score="3" ViewCount="302" Body="&lt;p&gt;What are the best sites to display your data science skills &amp;amp; projects? Obviously there's kaggle, but any others that recruiters look at?&lt;/p&gt;&#xA;" OwnerUserId="13227" LastActivityDate="2015-10-07T12:25:28.967" Title="What are the best sites to display your data science skills &amp; projects?" Tags="|machine-learning|data-mining|beginner|" AnswerCount="1" CommentCount="1" ClosedDate="2015-10-08T09:58:12.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="8342" PostTypeId="2" ParentId="8340" CreationDate="2015-10-07T04:38:27.750" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;LinkedIn&lt;/li&gt;&#xA;&lt;li&gt;Github&lt;/li&gt;&#xA;&lt;li&gt;Personal Website (of course)&lt;/li&gt;&#xA;&lt;li&gt;Coderbits (not popular, but you can display your profile link on the resume/LinkedIn)&lt;/li&gt;&#xA;&lt;li&gt;Kaggle&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-07T04:38:27.750" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8346" PostTypeId="2" ParentId="8339" CreationDate="2015-10-07T12:20:00.673" Score="5" Body="&lt;p&gt;The approach you are considering is similar to a multi-class SVM or a &lt;strong&gt;one-vs-the-rest&lt;/strong&gt; approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is how I describe the problem. The support vector machine, per example, is fundamentally a two-class classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, however, we often have to tackle problems involving K &gt; 2 classes. Various methods have therefore been proposed for combining multiple two-class SVMs in order to build a multi-class classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One commonly used approach (Vapnik, 1998) is to construct K separate SVMs, in which the kth model y_k(x) is trained using the data from class C_k as the positive examples and the data from the remaining K − 1 classes as the negative examples. This is known as the one-versus-the-rest approach where : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y(x) = max_k y_k(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Unfortunately, this heuristic approach suffers from the problem that the different classifiers were trained on different tasks, and there is no guarantee that the real-valued quantities y_k(x) for different classifiers will have appropriate scales&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem with the one-versus-the-rest approach is that the training sets are imbalanced. For instance, if we have ten classes each with equal numbers of training data points, then the individual classifiers are trained on data sets comprising 90% negative examples and only 10% positive examples, and the symmetry of the original problem is lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefor, you got your &lt;em&gt;bad accuracy&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; &lt;em&gt;Accuracy&lt;/em&gt;, in most cases, is not a good measure for evaluating a classifier model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Vapnik, V. - Statistical Learning Theory. Wiley-Interscience, New York.&lt;/li&gt;&#xA;&lt;li&gt;Christopher M. Bishop - Pattern Recognition and Machine Learning.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2015-10-07T12:20:00.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8347" PostTypeId="2" ParentId="8309" CreationDate="2015-10-07T13:18:16.423" Score="1" Body="&lt;p&gt;There is a neat online tool called &lt;a href=&quot;http://raw.densitydesign.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;RAWGraphs&lt;/a&gt; which will chart your data in your browser.  You drag and drop your file and it will try and make sense of it. It's made for web designers, so in the end you export an HTML representation of your visualization, but it's a fine tool in its own right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Kr3Rk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Kr3Rk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="31513" LastEditDate="2017-05-25T10:35:19.810" LastActivityDate="2017-05-25T10:35:19.810" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8348" PostTypeId="2" ParentId="8024" CreationDate="2015-10-07T18:36:23.627" Score="10" Body="&lt;p&gt;You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative &lt;a href=&quot;https://en.wikipedia.org/wiki/Q-learning&quot; rel=&quot;noreferrer&quot;&gt;Q-learning&lt;/a&gt; can do very well - as well as a variety of similar techniques, such as Temporal Difference, SARSA. All these can be used without neural networks, provided your problem is not too big (typically under a few million state/action pairs). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest form of Q-learning just stores and updates a table of &lt;code&gt;&amp;lt;state, action&amp;gt; =&amp;gt; &amp;lt;estimated reward&amp;gt;&lt;/code&gt; pairs. There is no deeper statistical model inside that. Q-learning relies on estimates of reward from this table in order to take an action and then updates it with a more refined estimate after each action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q-learning and related techniques such as Temporal Difference are sometimes called &lt;em&gt;model free&lt;/em&gt;. However, this does not refer to the absence of a statistical model such as a neural net. Instead, it means that you do not need to have a model of the system you are learning to optimise available, such as knowing all the probabilities of results and consequences of actions in a game. In model free RL, all learning can be done simply by experiencing the system as an agent (if you do have a model then it may still be used for simulation or planning). When considering whether or not you need a neural network, then the term &lt;em&gt;tabular&lt;/em&gt; is used for systems that work with explicit value estimates for every possible state or state/action pair. And the term &lt;em&gt;function approximation&lt;/em&gt; is used to describe how a neural network is used in the context of RL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large, complex problems, which may even have infinite possible states, it is not feasible to use tabular methods, and you need good generalised value estimates based on some function of the state. In those cases, you can use a neural network to create a function approximator, that can estimate the rewards from similar states to those already seen. The neural network replaces the function of the simple table in tabular Q-Learning. However, the neural network (or other supervised ML algorithm) does not perform the learning process by itself, you still need an &quot;outer&quot; RL method that explores states and actions in order to provide data for the NN to learn.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-31T16:59:59.243" LastActivityDate="2017-03-31T16:59:59.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8349" PostTypeId="1" CreationDate="2015-10-07T19:14:44.173" Score="10" ViewCount="958" Body="&lt;p&gt;My dataset is comprised of vector sequences. Each vector has 50 real-valued dimensions. The number of vectors in a sequence range from 3-5 to 10-15. In other words, the length of a sequence is not fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some fair amount of the sequences (not vectors!) are annotated with a class label. My task is to learn a classifier that given a sequence of vectors, the class label for the whole sequence is computed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot tell the exact nature of the data but the nature of sequences is not temporal. Nevertheless, a vector $x_i$ cannot be interchanged with a vector $x_j$ without changing the label ($i \neq j$). In other words, the order of vectors is important. The vectors themselves are comparable, for example it makes sense to compute a dot product and use this similarity value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: what are the tools/algorithms that can help to classify such data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&#xA;The data has such a property that one or very few vectors influence strongly the class label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;POSSIBLE SOLUTION:&#xA;After some research it looks like the Recurrent Neural Networks (RNN) fit the bill pretty naturally. The overarching idea is to pick a context size $k$, concatenate word vectors, do max pooling and feed that through classical NN. At each possible context window position in a sentence, a feature vector is built. The final feature vector is built using max pooling for example. The backpropagation is done to adjust the network's parameters. I already got some positive results (GPU is a must).&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="7848" LastEditDate="2015-10-10T23:08:31.730" LastActivityDate="2015-10-12T21:22:43.730" Title="Classification of vector sequences" Tags="|classification|sequence|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8350" PostTypeId="2" ParentId="5442" CreationDate="2015-10-07T23:16:48.330" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;Voter registration files are public data and are available at the state or county level (since the passage of the Help America Vote Act states are required to maintain lists of eligible voters).&lt;/li&gt;&#xA;&lt;li&gt;A popular account of the Obama data analytics is in the book, &quot;The Victory Lab&quot; by Sash Issenberg &lt;a href=&quot;http://www.thevictorylab.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.thevictorylab.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Academic paper, &quot;Political Campaigns and Big Data&quot; &lt;a href=&quot;https://www3.nd.edu/~dnickers/files/papers/nickerson_rogers.2014.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www3.nd.edu/~dnickers/files/papers/nickerson_rogers.2014.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13254" LastActivityDate="2015-10-07T23:16:48.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8351" PostTypeId="2" ParentId="8245" CreationDate="2015-10-08T07:04:04.497" Score="1" Body="&lt;p&gt;Done with K-means clustering with descriptive statistics as features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I've tried the idea described in the question, even if I was thinking it won't work. Let the experience talk...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially had a list of devices data. Each element of the list were 2 columns, R rows matrix, and R was different for each device. So, per device:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;    [mesureValue, timestamp],&#xA;    ..., &#xA;    [mesureValue, timestamp],&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since I'm only interested in the measureValue distribution, I've transformed the inital data to a 8 columns, N rows matrix, where N = number of devices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns are, computed on the correponding device's measure value:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Arithmetic mean&lt;/li&gt;&#xA;&lt;li&gt;Median&lt;/li&gt;&#xA;&lt;li&gt;First quartille&lt;/li&gt;&#xA;&lt;li&gt;Third quartille&lt;/li&gt;&#xA;&lt;li&gt;Minimum&lt;/li&gt;&#xA;&lt;li&gt;Maximum&lt;/li&gt;&#xA;&lt;li&gt;Range&lt;/li&gt;&#xA;&lt;li&gt;Standard deviation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With this matrix, I've applied K-means clustering using scikit learn (python).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made the link between the matrix line and the physical device by using pandas Data Frames (python) who's line index are in fact the serial number of the device.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried with 5 clusters, and it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just in case of, if I need improvements in the future, I'm planning to add other statistics in the columns, especially for deviation vs. normality. So, for example Kurstosis and normal q-q plot p value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="3024" LastEditDate="2015-10-08T16:35:37.947" LastActivityDate="2015-10-08T16:35:37.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8352" PostTypeId="2" ParentId="8349" CreationDate="2015-10-08T08:21:38.153" Score="3" Body="&lt;p&gt;As you can't disclose much detail, I'm forced to be a bit generic in my answer. I hope it will be helpful nevertheless. First of all, I would only consider reducing the sequences before classification (be it by using the dot product or something else) if you can make sure that you don't lose information you need for classification afterwards. So this approach is only feasible if you have some insight into the nature of the classification. To give a simple example: if the class label is just the number of vectors in your sequence, you won't be very successfull in predicting the class label from the dot product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I would take the full sequence as an input for classification, and impose a maximum on the sequence length you want to consider. You might do this by first finding the maximum sequence length m in your training set and then turning each sequence of 50-dimensional vectors into one vector of dimension 50*m, possibly with some missing values at the end if your sequence doesn't have maximum length. You will probably want to get rid of these missing values and you may want to simply replace them by zeros. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's two roads you can go from here: &#xA;1.) You directly apply classification methods known to be suitable for high dimensions. Try something simple that doesn't need much tuning like naive Bayes. This way you can see whether this approach is feasible without losing too much time if it isn't.&#xA;2.) You try first to reduce the dimension and understand the nature of the classification better. You may want to use something like principal components analysis or analyse correlation/association between each vector component and the class label. If you're successful, you know how to properly reduce the dimension of your input before applying classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to follow any of these ideas, please keep in mind that the concrete details of your data and the classification may render any of the ideas proposed above infeasible. So please be careful to check against any details you know but can't post here before trying to make sure you're not wasting your time.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2015-10-08T08:21:38.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8353" PostTypeId="1" CreationDate="2015-10-08T13:11:56.483" Score="1" ViewCount="1071" Body="&lt;p&gt;Cross validation on a dataset with 22680 observations. Want the training set to contain 21420 entries. How many folds can you use for your cross validation?&#xA;How do you calculate the folds?&lt;/p&gt;&#xA;" OwnerUserId="13083" LastActivityDate="2015-10-08T13:27:36.613" Title="How many folds on cross validation" Tags="|statistics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8354" PostTypeId="2" ParentId="8353" CreationDate="2015-10-08T13:27:36.613" Score="5" Body="&lt;p&gt;The number of folds is nothing you can really calculate. It is more a parameter you choose by good judgement. Typically used values are between 5 and 10, but you can even go up to the so-called leave-one-out cross validation, in which each fold consists of all but one observation. The arguments that guide your decision are typically the following:&#xA;- computational cost: the more folds you use, the more computations you need. &#xA;- variance: the more folds you use, the higher the variance of the cross validation result&#xA;- bias: the more folds you use, the smaller the bias of the cross validation result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, depending on the exact context in which you want to apply cross validation, you will probably want to choose between 5 and 10 folds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details, you might want to consult &lt;a href=&quot;http://ai.stanford.edu/~ronnyk/accEst.pdf&quot; rel=&quot;noreferrer&quot;&gt;Kohavi's 1995 paper on cross validation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2015-10-08T13:27:36.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8357" PostTypeId="1" AcceptedAnswerId="8368" CreationDate="2015-10-09T00:02:34.463" Score="10" ViewCount="1287" Body="&lt;p&gt;I understand how a Hidden Markov Model is used in genomic sequences, such as finding a gene. But I don't understand how to come up with a particular Markov model. I mean, how many states should the model have? How many possible transitions? Should the model have a loop?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would they know that their model is optimal? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do they imagine, say 10 different models, benchmark those 10 models and publish the best one?&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="14713" LastEditDate="2019-09-28T07:36:19.890" LastActivityDate="2019-09-28T07:36:19.890" Title="How do scientists come up with the correct Hidden Markov Model parameters and topology to use?" Tags="|machine-learning|model-selection|hyperparameter|markov|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8358" PostTypeId="1" CreationDate="2015-10-09T01:54:16.800" Score="12" ViewCount="8075" Body="&lt;p&gt;When saved to disk using cPickle: &lt;a href=&quot;https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file&quot;&gt;https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file&lt;/a&gt;, my random forest is 6.57 GB. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with open('rforest.cpickle', 'wb') as f:&#xA;    cPickle.dump(rforest, f)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to use the forest itself to make predictions via a python API hosted on Heroku -- of course, that file size is unacceptable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the file size so large? There are 500 trees in the ensemble -- all I want to save are the completed trees themselves, since they will be used as prediction. Is it the actual nodes and edges that compose each of the 500 trees that requires nearly 7 GB of space on disk?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used scikitlearn's randomforestregressor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def buildForest(self, X_train, y_train):&#xA;    rf = RandomForestRegressor(n_estimators=500, verbose=1)&#xA;    rf.fit_transform(X_train, y_train)&#xA;    return rf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, if there is a better way to make my model accessible via API, that would also be good to know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update: I reduced it to 100 trees without losing much predictive power, and so now the saved size is 1.3 GB -- much more manageable, but still not great.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="13179" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2024-02-24T02:25:33.220" Title="Prohibitive size of random forest when saved to disk" Tags="|python|random-forest|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8359" PostTypeId="1" CreationDate="2015-10-09T03:44:07.940" Score="2" ViewCount="122" Body="&lt;p&gt;I have a dataset of this form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;chrX posX labelX&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;chrX&lt;/code&gt; refers to the chromosome number, &lt;code&gt;posX&lt;/code&gt; refers to location, and &lt;code&gt;labelX&lt;/code&gt; is a categorical variable with 3 labels.&#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chr1 3223 1&#xA;chr1 3200 2&#xA;chr1 3100  1&#xA;chr1 1000 2&#xA;chr2 1000 1&#xA;chr1 3210 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to be cluster these data with certain constraints (thus, this is no longer what 'clustering' typically is), such that, for each chromosome, the sites which are within, say $\pm 30$ positions are together, the constraint being they should have unique labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, an expected output would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;chr1 : 3223(1) 3200(2) 3210(3)&lt;/code&gt; Notice how sites like &lt;code&gt;chr1 3100 1&lt;/code&gt; are not part of the 'cluster'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach would be to simply calculate pairwise differences between all such sites, and grow the cluster taking care of unique labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there more elegant approaches to solve such problems?&lt;/p&gt;&#xA;" OwnerUserId="13274" LastEditorUserId="13413" LastEditDate="2015-10-21T08:05:14.653" LastActivityDate="2015-12-20T10:42:14.120" Title="Clustering with constraints" Tags="|machine-learning|clustering|categorical-data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8360" PostTypeId="2" ParentId="8358" CreationDate="2015-10-09T05:51:46.140" Score="8" Body="&lt;p&gt;The size of each tree depends very much on its depth. Thus, change the maximal depth (&lt;code&gt;max_depth&lt;/code&gt;). Try to set it to finite number (as opposed to the default &amp;quot;None&amp;quot;) and then try to reduce this number. In addition (or as alternative) try to increase &lt;code&gt;min_samples_split&lt;/code&gt; or &lt;code&gt;min_samples_leaf&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You can also analyze you features and keep only important ones. The simplest way would be to have a look at the &lt;code&gt;clf.feature_importances_&lt;/code&gt; of your forest. (In general, finding important features is an art and science on itself.) Exclude non-relevant features and rebuild the forest.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="118071" LastEditDate="2024-02-24T02:25:33.220" LastActivityDate="2024-02-24T02:25:33.220" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8361" PostTypeId="1" AcceptedAnswerId="8362" CreationDate="2015-10-09T06:11:18.740" Score="3" ViewCount="1809" Body="&lt;p&gt;What is the difference between support vector machine and Gaussian mixture model classifiers?&lt;/p&gt;&#xA;" OwnerUserId="13236" LastEditorUserId="11097" LastEditDate="2015-10-09T10:54:53.450" LastActivityDate="2015-10-09T10:54:53.450" Title="What is the difference between SVM and GMM classifier" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2015-10-19T07:57:13.127" ContentLicense="CC BY-SA 3.0" />
  <row Id="8362" PostTypeId="2" ParentId="8361" CreationDate="2015-10-09T06:31:39.117" Score="1" Body="&lt;p&gt;A Gaussian mixture model is a special case of a mixture distribution, which is a simple way of combining probability distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the SVM does not make any assumptions. It is just a function, which depends on the distance of a data point from another point/plane. Som there is absolutely no probabilistic assumptions.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-09T06:31:39.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8364" PostTypeId="2" ParentId="8359" CreationDate="2015-10-09T09:17:26.153" Score="1" Body="&lt;p&gt;Each clustering algorithm in my knwledge is based on distance, which is calculated as &lt;em&gt;pairwise differences&lt;/em&gt;. So I'm not sure if it is easy to find a general &lt;em&gt;more elegant approach&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In you case you can calculate the distance and than override it based on your &lt;em&gt;constraint&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here an simple example reduced to one numerical dimension with distance threshold 30.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; nlst &amp;lt;- c(1,3,5, 36,39,42, 1001,1003,1005, 1036,1039,1042)&#xA;&#xA; d &amp;lt;- dist(nlst, method=&quot;euclidean&quot;)&#xA; dr &amp;lt;- as.dist(ifelse(as.matrix(d) &amp;gt; 30, NA, 0))&#xA;&#xA;&#xA;     1  2  3  4  5  6  7  8  9 10 11&#xA; 2   0                              &#xA; 3   0  0                           &#xA; 4  NA NA NA                        &#xA; 5  NA NA NA  0                     &#xA; 6  NA NA NA  0  0                  &#xA; 7  NA NA NA NA NA NA               &#xA; 8  NA NA NA NA NA NA  0            &#xA; 9  NA NA NA NA NA NA  0  0         &#xA; 10 NA NA NA NA NA NA NA NA NA      &#xA; 11 NA NA NA NA NA NA NA NA NA  0   &#xA; 12 NA NA NA NA NA NA NA NA NA  0  0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A scan through the points collecting those with distance zero will give the result clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should be always possible if your constraints are local (i.e. depending only on the two points for which you calculate the distance).&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-10-09T09:17:26.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8366" PostTypeId="2" ParentId="8334" CreationDate="2015-10-09T13:01:23.863" Score="1" Body="&lt;p&gt;The maximum depth of a forest is a parameter which you set yourself. If you're asking how do you find the optimal depth of a tree given a set of features then this is through cross-validation. For example, create 5 rf's with 5 different tree depths and see which one performs the best on the validation set. &lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-09T13:01:23.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8367" PostTypeId="2" ParentId="8301" CreationDate="2015-10-09T13:14:55.737" Score="1" Body="&lt;p&gt;This sounds like a time-related predictive task...i.e. forecasting. The data you have available is event-driven, e.g. data is recorded when a user clicks on a link/refreshes a page/clicks on a Google ad etc. This means that you need to treat your data as if it were a time-series for each user. This would be easier if you had access to ESP (Event Stream Processing) but let's say that you do not. Therefore, you need to identify each user, aggregate their info/features so that you have features identifying the number of unique pages they visited, the number of sessions they had etc. etc. Then you can segment these users and you may or may not derive useful segments from them. You could even go a step further and do session analysis whereby you identify unique sessions and cluster/segment user sessions looking for nice visual patterns or anomalies. &lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-09T13:14:55.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8368" PostTypeId="2" ParentId="8357" CreationDate="2015-10-09T14:18:23.560" Score="6" Body="&lt;p&gt;I'm familiar with three main approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes, and so have 44 states for the hidden phoneme layer in a voice recognition model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Estimation. The number of states can often be estimated beforehand, perhaps by simple clustering on the observed features of the HMM. If the HMM transition matrix is triangular (which is often the case in failure prediction), the number of states determines the shape of the distribution of total time from the start state to the end state.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Optimization. Like you suggest, either many models are created and fit and the best model selected. One could also adapt the methodology that learns the HMM to allow the model to add or discard states as needed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-09T14:18:23.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8369" PostTypeId="1" AcceptedAnswerId="8372" CreationDate="2015-10-09T16:18:05.393" Score="1" ViewCount="68" Body="&lt;p&gt;In the data, there are 355 observations including one continuous dependent variable (Y: ranges from 15-55) and 12 independent variables (continuous, categorical, and ordinal). The X1 (2 levels) and X6 (3 levels) are considered as categorical variables. Here are some questions that I have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I assume that all the coefficients (except X1 and X6 which are categorical) are linear with respect to Y?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I consider X5 as continuous variable; however, it is ordinal and ranges from (1-7)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I get the X7 (year) as continuous variable; however, it’s ordinal and rages from 2002-2006 (In fact, year of data per se does not improve the response; it is the other factors occurring in the same time period which result in improvements and we don’t know those factors), does this approach seem logical?. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In general if I use different transformations on independent variables such as log, squared, square root, and inverse, do I need to standardize the data also?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is the scatter plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GYRpa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GYRpa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback and insights would be highly appreciated. Thank you&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="12867" LastEditDate="2015-10-09T20:16:04.263" LastActivityDate="2015-10-11T00:17:27.447" Title="Can I consider this pattern of data as a linear and use parametric multiple linear regression?" Tags="|data-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8370" PostTypeId="1" CreationDate="2015-10-09T16:31:44.320" Score="10" ViewCount="1309" Body="&lt;p&gt;Have corpus of over million documents     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given document want to find similar documents using cosine as in &lt;a href=&quot;https://en.wikipedia.org/wiki/Vector_space_model&quot; rel=&quot;nofollow&quot;&gt;vector space model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d_1 \cdot d_2 / ( ||d_1|| ||d_2|| )$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All tf have been normalized using augmented frequency, to prevent a bias towards longer documents as in this &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$tf(t,d)=0.5+0.5\frac{f(t,d)}{\mathrm{max}\{f(t,d): t\in d\}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have pre-calculated all $||d||$&lt;br&gt;&#xA;Have the values for the denominator pre-calculated&lt;br&gt;&#xA;So for a given $d1$ need to score over 1 million $d2$&lt;br&gt;&#xA;Have a threshold of 0.6 cosine for similarity  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can observe that for a given $||d_1||$ there is a fairly narrow range of $||d_2||$ for cosine $\ge$ 0.6&lt;br&gt;&#xA;For example in one search for similar for a cosine of $\ge$ 0.6 and a $||d_1||$ of 7.7631 then $||d_2||$ range from 7.0867 to 8.8339&lt;br&gt;&#xA;Where outside the threshold of cosine 0.6 $||d_2||$ range from to 0.7223 to 89.3395&lt;br&gt;&#xA;This was with standard tf document normalization&lt;br&gt;&#xA;It is looking at a LOT of $||d_2||$ that don't have a chance of being a cosine 0.6 match     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally the question:&lt;br&gt;&#xA;For a give $||d_1||$ and cosine of &gt;= 0.6 how can determine the range of $||d_2||$ that have a chance?&lt;br&gt;&#xA;Which $||d_2||$ can I safely eliminate?     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also know the number of terms in $d_1$ and $d_2$ if there is term count range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Via experimentation&lt;br&gt;&#xA;$||d2|| &amp;gt; .8 ||d1||$ and $||d2|| &amp;lt; ||d1|| / .8 $&#xA;seems to be safe but hopefully there is range that is proven to be safe  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Created some test cases with a very some unique terms, some not so unique, and some common.  Sure enough you can take the most unique term and increase that frequency in the compare.  The numerator will (dot product) go up and so will ||compare|| and will get a cosine very close to 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind of related and NOT the question.&lt;br&gt;&#xA;I am also using the tf-idf to group documents into groups. &#xA;The customer base I am selling into are used to near near dup groups. &#xA;There I am taking a related approach in I look as the smallest term count and evaluate it against term count up to 3x.  So a term count of 10 looks at 10  thru 30 (4-9 already had their shot at 10).  Here I can afford to miss one have it picked up in another.  I am 10% done and the biggest ratio is 1.8.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please identify the flaws in this analysis&lt;br&gt;&#xA;&lt;strong&gt;As pointed out by AN6U5 there is a flaw in this analysis&lt;/strong&gt;&lt;br&gt;&#xA;It is no longer a cosine if the document is normalized on weighted&lt;br&gt;&#xA;&lt;strong&gt;And as pointed out by Mathew also cannot conclude  d1⋅d2≤d1⋅d1&lt;/strong&gt;&lt;br&gt;&#xA;I am still hoping for something to give me a hard bound but people that seems to know this stuff are telling me no&lt;br&gt;&#xA;I don't want to change the question so just ignore this&lt;br&gt;&#xA;I will do some analysis and maybe post a separate question on document normalization&lt;br&gt;&#xA;For the purpose of this question assume the document is normalized on raw tf&lt;br&gt;&#xA;Sorry but I am just not good with what ever markup is used to make the   equations&lt;br&gt;&#xA;So in my notation&lt;br&gt;&#xA;||d1|| = sqrt(sum(w1 x w1))&lt;br&gt;&#xA;d1 dot d2 = sum(w1 X w2)&lt;br&gt;&#xA;Assume d1 is the shorter document&lt;br&gt;&#xA;The very best d1 dot d2 that can be achieved is d1 dot d1&lt;br&gt;&#xA;If d1 is marry 100 paul 20&lt;br&gt;&#xA;And d2 is marry 100 paul 20 peter 1&lt;br&gt;&#xA;Normalized&lt;br&gt;&#xA;d1 is marry 1 paul 1/5&lt;br&gt;&#xA;d2 is marry 1 paul 1/5  peter 1/100&lt;br&gt;&#xA;Clearly marry and paul have the same idf in both documents&lt;br&gt;&#xA;The best possible d1 dot d2 is d1 dot d1&lt;br&gt;&#xA;The maximum possible match to d1 is d1&lt;br&gt;&#xA;cos = d1 dot d1 / ||d1|| ||d2||&lt;br&gt;&#xA;square both sides&lt;br&gt;&#xA;cos X cos = (d1 dot d1) X (d1 dot d1) / ( (d1 dot d1) X (d2 dot d2) ) &#xA;cos X cos = (d1 dot d1) / (d2 dot d2)&lt;br&gt;&#xA;take the square root of both side&lt;br&gt;&#xA;cos = ||d1|| / ||d2||&lt;br&gt;&#xA;is ||d2|| not bounded by the cos?&lt;br&gt;&#xA;If I just use ||d2|| &gt;= cos ||d1|| and ||d2|| &amp;lt;= ||d1|| / cos I get the computational speed I need  &lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-10-15T19:45:25.843" LastActivityDate="2015-10-18T02:19:57.837" Title="Vector space model cosine tf-idf for finding similar documents" Tags="|text-mining|similarity|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8372" PostTypeId="2" ParentId="8369" CreationDate="2015-10-09T18:38:53.803" Score="2" Body="&lt;p&gt;I don't think &quot;can&quot; is the right question to ask; it's not going to give you a syntax error. The right question is &quot;what could go wrong?&quot;. Any modeling technique will have assumptions that may be broken, and knowing how those assumptions impact the results will help you know what to look for (and how much to care when those assumptions are broken).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The best test of whether or not linearity is appropriate is whether the residuals are white or structured. For example, it looks like X9 might have a nonlinear relationship with Y. But that might be an artifact of the interaction between X9 and other variables, especially categorical variables. Fit your full model, then plot the residuals against X9 and see what it looks like.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Treating it as continuous won't cause serious problems, but you might want to think about what this implies. Is the relationship between 1 and 2 in the same direction and half the strength as the relationship between 2 and 4? If not, you might want to transform this to a scale where you &lt;em&gt;do&lt;/em&gt; think the differences are linear.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Same as 2, except it's even more reasonable to see time as linear.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Standardization is not necessary for most linear regression techniques, as they contain their own standardization. The primary exception is techniques that use &lt;a href=&quot;https://en.wikipedia.org/wiki/Regularization_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;regularization&lt;/a&gt;, where the scale of the parameters is relevant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It's also worth pointing out that multivariate linear relationships, while they can capture general trends well, are very poor at capturing logical trends. For example, looking at X3 and X4, it could very well be that there are rules like Y&gt;X3 and Y&gt;X4 in place, which is hinted at but not captured by linear regression.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-09T18:38:53.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8373" PostTypeId="1" CreationDate="2015-10-10T06:14:50.927" Score="4" ViewCount="589" Body="&lt;p&gt;I am working on a project for prediction, using Fuzzy Cognitive Maps. I am new to data mining and want to learn about Fuzzy Cognitive Maps and its implementation. I also want to know what tools are required for this. I am beginner in this domain, so I need a little help to get started. I would be very grateful for help with some good resources and advice.&lt;/p&gt;&#xA;" OwnerUserId="13292" LastEditorUserId="2452" LastEditDate="2015-10-10T09:36:32.943" LastActivityDate="2023-10-09T04:05:32.807" Title="Fuzzy Cognitive Maps" Tags="|machine-learning|data-mining|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8374" PostTypeId="2" ParentId="6691" CreationDate="2015-10-10T10:35:22.093" Score="3" Body="&lt;p&gt;The &lt;a href=&quot;https://indico.io/docs#sentiment&quot; rel=&quot;nofollow&quot;&gt;Indico.io API&lt;/a&gt; supports Spanish (and&#xA;Chinese (Mandarin), Japanese, Italian, French, Russian, Arabic, German, English).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import indicoio&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.config.api_key = &amp;lt;YOUR_API_KEY&amp;gt;&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.sentiment(&quot;¡Jamás voy a usar esta maldita aplicación!  No funciona para nada.&quot;)&#xA;0.02919392219306888&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.sentiment(&quot;¡Es patrón!  La mejor que he visto.  Punto.&quot;)&#xA;0.8860221705630639&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this isn't your area, then that's probably the easiest sort of solution to integrate.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="12363" LastEditDate="2015-10-10T10:43:33.600" LastActivityDate="2015-10-10T10:43:33.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8376" PostTypeId="2" ParentId="8369" CreationDate="2015-10-10T13:46:02.047" Score="0" Body="&lt;p&gt;Thanks Matthew.&#xA;There is a confusion to me, you mean we’ll build the model through multiple linear regression method and then the residuals will be checked VS the significant factors to see if they are structureless. We shouldn’t first check whether or not the variables (coefficients) are linear with respect to y.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Since the X9 and X5 were not statistically significant, shall I check the scatter plot of residuals vs those two?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are your insights about the scatter plot of residuals vs time, do you think it still needs any transformation? If yes, can you explain the reason?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here are the scatterplot of residuals VS significant variables and different Residuals plots vs Y to check if the normality assumptions were met. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PzyHr.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PzyHr.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2pnmb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2pnmb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know your feedback. On the other hand, can you explain a bit more what your last paragraph meant (note that both X3 and X4 are statistically significant)?&#xA;Thank you again.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastActivityDate="2015-10-10T13:46:02.047" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8377" PostTypeId="2" ParentId="5893" CreationDate="2015-10-10T18:12:44.310" Score="0" Body="&lt;p&gt;Stopwords may be part of the solution at some point, but not the key.  In any case for any major languages good lists of stop words exist, it should not be domain specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also don't think that using TD-IDF alone is really correct.  There could be very rare (potentially garbage) words in poor quality strings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of trying to guess which exact features are useful: I would start by creating a data set by randomly selecting some of the data and labeling them by hand (as &lt;code&gt;good&lt;/code&gt; or &lt;code&gt;bad&lt;/code&gt;, or on a scale from 0.0 to 1.0).  Then code something up that pulls out many features (length, number of words (tokens), spam score, whether it contains URLs or botish chars, detected language, whether it has a question mark, whether it has proper capitalisation).  Also don't forget to include non-linguistic features that you may have, like country of the geoIP of the user that made the query, whether the user was logged in, how old the user's account is.  So at this point you will have a massive table/CSV, and a smaller one with one extra column for the label you've added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then train some machine learning package with those labeled examples to build a model that is accurate enough for you.  Then let that model run on the rest of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wish not to code too much, you could even just get those features into CSV form, and give them to the &lt;a href=&quot;https://cloud.google.com/prediction/docs/smart_autofill_add_on&quot; rel=&quot;nofollow&quot;&gt;Google Prediction API's spreadsheet interface&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="12363" LastEditDate="2015-10-12T10:37:54.667" LastActivityDate="2015-10-12T10:37:54.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8379" PostTypeId="2" ParentId="8369" CreationDate="2015-10-11T00:17:27.447" Score="0" Body="&lt;p&gt;In fact, I used additive method to calculate the X3 value (X3 is the summation of 21 binary variables) and multiplicative method to calculate X4 (X4 is the multiplication of 3 continues variables). I was wondering if this information will change the way of my calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the residuals plot versus X7, X5, and X9 plus probability plot of residuals for the X7 (time or year). It can be discovered that the residuals of X7 in each time horizon is normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IacoF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IacoF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/T600s.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/T600s.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm still confused about this point, you mean we’ll build the model through multiple linear regression and then the residuals will be checked VS the significant or none-significant factors to see if they are structureless and follow the normal distribution. We shouldn’t first check whether or not the variables (coefficients) are linear with respect to y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your feedback.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastActivityDate="2015-10-11T00:17:27.447" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8380" PostTypeId="1" CreationDate="2015-10-11T01:05:04.113" Score="6" ViewCount="2171" Body="&lt;p&gt;I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, other then the potential of expanding to more nodes in the future? What if you have no plans of ever having more then one node? Or is there an expectation that if you are going to use Hadoop you are also going to be setting up a cluster of many nodes?&lt;/p&gt;&#xA;" OwnerUserId="13307" LastActivityDate="2022-08-18T10:07:25.193" Title="Is there a benefit to using hadoop with only one node?" Tags="|apache-hadoop|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8381" PostTypeId="1" CreationDate="2015-10-11T03:41:00.940" Score="2" ViewCount="358" Body="&lt;p&gt;I have a basic understanding about Machine Learning in general. My question is how it is done in the practical applications of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I take the following definition of ML&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A computer program is said to learn from experience E with respect to&#xA;  some class of tasks T and performance measure P, if its performance at&#xA;  tasks in T, as measured by P, improves with experience E.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It talks about Experience E. What I understand from the above is the more data you give to the algorithm E increases, and that in turn increases P. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's consider a scenario where you build a decision tree model from 10,000 data rows available. Now I have the model, so can I say that my model has learned and just stop there!(use that model for prediction from that point on-wards forever)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the definition, I need to feed more data so the experience increases, and in turn I get a performance improvement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is Machine Learning a continuous process so that You cannot build the model and just stop there. Do we need to feed more data to the algorithm time by time and improve the model so that the model actually LEARNS? &lt;/p&gt;&#xA;" OwnerUserId="13309" LastActivityDate="2015-10-13T07:50:43.187" Title="How to do Machine Learning the right way?" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8382" PostTypeId="2" ParentId="8380" CreationDate="2015-10-11T08:04:58.277" Score="5" Body="&lt;p&gt;If you are testing or building something that will eventually be run on a distributed cluster, yes it make sense. If you're playing around to learn, yes. Otherwise no I don't see much value.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-10-11T08:04:58.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8384" PostTypeId="1" AcceptedAnswerId="8385" CreationDate="2015-10-11T08:25:59.450" Score="2" ViewCount="4486" Body="&lt;p&gt;What are some examples of the Three V's of Big Data? The three V's stand for: volume, velocity, variety.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt;&lt;br&gt;&#xA;Three V's of Big Data, provided by Norwegian University of Science and Technology.&lt;br&gt;&#xA;&lt;a href=&quot;https://www.ntnu.edu/ime/bigdata/what-is&quot; rel=&quot;nofollow&quot;&gt;https://www.ntnu.edu/ime/bigdata/what-is&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="9304" LastActivityDate="2016-09-13T19:21:06.823" Title="Examples of the Three V's of Big Data?" Tags="|bigdata|" AnswerCount="3" CommentCount="1" ClosedDate="2016-09-13T21:20:44.773" ContentLicense="CC BY-SA 3.0" />
  <row Id="8385" PostTypeId="2" ParentId="8384" CreationDate="2015-10-11T09:04:09.813" Score="1" Body="&lt;p&gt;&lt;strong&gt;Volume:&lt;/strong&gt;&lt;br&gt;&#xA;Simply stated, big data is to big to work on one computer. This is a relative definition, as what can't work on today's computer will easily work on computers in the future.&lt;br&gt;&#xA; - One Google search uses the computing power of the entire Apollo space mission.&lt;br&gt;&#xA; - Excel used to hold up to 65k rows in a single spreadsheet. Now it holds over a million.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Velocity:&lt;/strong&gt;&lt;br&gt;&#xA;Data is coming in extremely fast. Traditional scientific research methods of a few hundred cases could take weeks, months or even years to analyze and publish.&lt;br&gt;&#xA; - Iris flower data set&lt;br&gt;&#xA; - Statistical Programming Language R&lt;br&gt;&#xA; - Twitter Firehose (6,000 tweets per second)      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Variety:&lt;/strong&gt;&lt;br&gt;&#xA;Big Data that is contained in one specific data type or does not fit well within the format of a relational database. This data often comes in the form of unstructured text.&lt;br&gt;&#xA; - Estimated 80% of all enterprise data is unstructured&lt;br&gt;&#xA; - Open Data(Government)&lt;br&gt;&#xA; - noSql Databases&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Iris Data Set:&lt;/strong&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Iris_flower_data_set&lt;/a&gt;&lt;br&gt;&#xA;&lt;strong&gt;Open Data:&lt;/strong&gt; &lt;a href=&quot;https://www.data.gov/open-gov/&quot; rel=&quot;nofollow&quot;&gt;https://www.data.gov/open-gov/&lt;/a&gt;      &lt;/p&gt;&#xA;" OwnerUserId="9304" LastActivityDate="2015-10-11T09:04:09.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8387" PostTypeId="2" ParentId="8373" CreationDate="2015-10-11T13:13:20.747" Score="1" Body="&lt;p&gt;You could use &lt;a href=&quot;https://www.researchgate.net/profile/Elpiniki_Papageorgiou/publication/220199753_A_new_methodology_for_Decisions_in_Medical_Informatics_using_fuzzy_cognitive_maps_based_on_fuzzy_rule-extraction_techniques/links/53f4c17f0cf2888a74911729.pdf&quot; rel=&quot;nofollow&quot;&gt;competitive fuzzy cognitive maps&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13315" LastEditorUserId="11097" LastEditDate="2015-10-13T15:20:46.277" LastActivityDate="2015-10-13T15:20:46.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8388" PostTypeId="1" CreationDate="2015-10-11T17:54:04.907" Score="0" ViewCount="74" Body="&lt;p&gt;I am trying to cluster related areas of knowledge based in publications.For example a researcher has 3 keywords in a paper, in another paper he has 5 keywords, but 3 keywords are the same in the both papers. Then these 3 keywords are similar and It could be an area of knowledge. A sample of my file is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'Domain Ontology,Semantic Web'&#xA;'Linked Data,Domain Ontology,Use Case'&#xA;'Domain Ontology,Linked Data,Semantic Annotation'&#xA;'GIS,Open GIS,Integrated Geo Systems'&#xA;'Open GIS,GIS'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My file contains 48963 rows and 19000 keywords. I have tried grouping words (like in the sample), using StringToWordVector (STWV), but I don't have good results. So I tried different K for my cluster since 3 until 13000. When K is greater my log likelihood decrease. In many models my words are grouped perfectly, but the percentages are not really good for some clusters. For example for K=3000, the 2999 have between 0,1% and 1% of the data, but the 3000 have the 21%. I am using WEKA. Anyone know some work related, some advice or any help is helpful...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers.&lt;/p&gt;&#xA;" OwnerUserId="13323" LastActivityDate="2015-10-12T08:36:29.197" Title="Clustering related areas with k-means" Tags="|data-mining|clustering|text-mining|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="8389" PostTypeId="1" AcceptedAnswerId="8403" CreationDate="2015-10-11T20:58:19.907" Score="3" ViewCount="1075" Body="&lt;p&gt;My hypothesis &lt;code&gt;h&lt;/code&gt; depends on multiple categorical variables &lt;code&gt;(a,b,c)&lt;/code&gt; each with their corresponding set of possible values &lt;code&gt;(A,B,C)&lt;/code&gt;. Now each of my data point exist in this space where I have no control over the values (observational data). &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For e.g. Hypothesis to predict user shopping probability say depends&#xA;  on (Age, Country, Gender, Devicetype etc.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How could I sample the above data set so that it would give me a good representation. The techniques I have learned from the books very well apply to one dimension but that is a rare case in practice. If I sample across one dimension my other dimensions will be heavily skewed towards some values. Is there any standard algorithm to give good sampling? &lt;/p&gt;&#xA;" OwnerUserId="8338" LastEditorUserId="8338" LastEditDate="2015-10-12T14:34:32.033" LastActivityDate="2015-10-12T18:49:13.713" Title="Sampling for multi categorical variable" Tags="|sampling|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8390" PostTypeId="1" AcceptedAnswerId="8394" CreationDate="2015-10-12T03:31:10.867" Score="9" ViewCount="6097" Body="&lt;p&gt;I am trying to understand sentiment analysis and how to apply it using any language (R, Python etc). I would like to know if there is a good place on internet for tutorial that I can follow. I googled, but I wasn't very much satisfied because they were not tutorials but more of theory. I want theory and practical examples.&lt;/p&gt;&#xA;" OwnerUserId="3550" LastEditorUserId="31513" LastEditDate="2017-05-10T03:57:59.487" LastActivityDate="2017-05-10T03:57:59.487" Title="Sentiment Analysis Tutorial" Tags="|machine-learning|reference-request|sentiment-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8391" PostTypeId="2" ParentId="8390" CreationDate="2015-10-12T04:48:34.710" Score="4" Body="&lt;p&gt;The &lt;a href=&quot;http://www.nltk.org/book/&quot; rel=&quot;nofollow&quot;&gt;NLTK book&lt;/a&gt; is by far the best tutorial on basic NLP I have seen(in Python).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Coursera &lt;a href=&quot;https://www.coursera.org/course/nlp&quot; rel=&quot;nofollow&quot;&gt;course&lt;/a&gt; on NLP is also fairly good. It takes off from the basics and takes the student to a novice level.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-12T04:48:34.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8392" PostTypeId="2" ParentId="8384" CreationDate="2015-10-12T04:54:03.893" Score="1" Body="&lt;p&gt;Actually, it is four V's which define Big Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &lt;a href=&quot;https://stats.stackexchange.com/questions/173060/what-exactly-is-big-data/173065#173065&quot;&gt;answer of mine&lt;/a&gt; on CrossValidated to a &lt;a href=&quot;https://stats.stackexchange.com/questions/173060/what-exactly-is-big-data&quot;&gt;similar question&lt;/a&gt; explains the four V's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @MikeStratton has neatly explained the three V's, I would explain the fourth V, which is &lt;strong&gt;Veracity&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Veracity is the uncertainty in data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Examples can be poor data quality, inadequate data from surveys, etc.&#xA;This makes the fitted models highly biased and error-prone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to have a look at &lt;a href=&quot;https://datascience.stackexchange.com/questions/19/how-big-is-big-data&quot;&gt;this discussion&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-12T04:54:03.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8393" PostTypeId="2" ParentId="8390" CreationDate="2015-10-12T05:27:06.643" Score="2" Body="&lt;p&gt;As &lt;em&gt;Dawny&lt;/em&gt; said i would suggest &lt;a href=&quot;http://www.nltk.org/book/&quot; rel=&quot;nofollow&quot;&gt;NLTK Book&lt;/a&gt; too. And in R language &lt;a href=&quot;https://cran.r-project.org/web/packages/tm/index.html&quot; rel=&quot;nofollow&quot;&gt;'tm'&lt;/a&gt; library is pretty useful. So try &lt;a href=&quot;https://rstudio-pubs-static.s3.amazonaws.com/31867_8236987cf0a8444e962ccd2aec46d9c3.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for more related text mining stuff with R. &lt;a href=&quot;http://www.r-bloggers.com/text-mining/&quot; rel=&quot;nofollow&quot;&gt;r-blog&lt;/a&gt; also worth of consider.&#xA;good luck. &lt;/p&gt;&#xA;" OwnerUserId="9035" LastActivityDate="2015-10-12T05:27:06.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8394" PostTypeId="2" ParentId="8390" CreationDate="2015-10-12T05:49:31.807" Score="3" Body="&lt;p&gt;The &lt;a href=&quot;https://www.coursera.org/course/nlp&quot; rel=&quot;nofollow&quot;&gt;Stanford NLP course on Coursera&lt;/a&gt; covers Sentiment Analysis in &lt;a href=&quot;https://class.coursera.org/nlp/lecture/preview&quot; rel=&quot;nofollow&quot;&gt;week 3&lt;/a&gt;:&lt;br&gt;&#xA;- &lt;a href=&quot;https://class.coursera.org/nlp/lecture/31&quot; rel=&quot;nofollow&quot;&gt;What is Sentiment Analysis?&lt;/a&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;https://class.coursera.org/nlp/lecture/145&quot; rel=&quot;nofollow&quot;&gt;Sentiment Analysis: A baseline algorithm&lt;/a&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;https://class.coursera.org/nlp/lecture/35&quot; rel=&quot;nofollow&quot;&gt;Sentiment Lexicons&lt;/a&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;https://class.coursera.org/nlp/lecture/144&quot; rel=&quot;nofollow&quot;&gt;Learning Sentiment Lexicons&lt;/a&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;https://class.coursera.org/nlp/lecture/33&quot; rel=&quot;nofollow&quot;&gt;Other Sentiment Tasks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For coding tutorials see:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://streamhacker.com/2010/05/10/text-classification-sentiment-analysis-naive-bayes-classifier/&quot; rel=&quot;nofollow&quot;&gt;Stream Hacker's NLP tutorials&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://fjavieralba.com/basic-sentiment-analysis-with-python.html&quot; rel=&quot;nofollow&quot;&gt;Basic Sentiment Analysis with Python&lt;/a&gt;  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://andybromberg.com/sentiment-analysis-python/&quot; rel=&quot;nofollow&quot;&gt;Andy Bromberg's Sentiment Analysis tutorials&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.laurentluce.com/posts/twitter-sentiment-analysis-using-python-and-nltk/&quot; rel=&quot;nofollow&quot;&gt;Laurent Luce's Sentiment Analysis tutorials&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are really basic, so their performance will not be great in all cases.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="11097" LastEditDate="2015-10-13T15:20:26.840" LastActivityDate="2015-10-13T15:20:26.840" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8395" PostTypeId="1" AcceptedAnswerId="8396" CreationDate="2015-10-12T06:36:53.497" Score="3" ViewCount="493" Body="&lt;p&gt;I have historical data from an e-shop transactions. I want to write a prediction model and check if a specific user will buy with or without a discount, so I can do some targeting offers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If a user will buy the regular price, will not have an offer.&lt;/li&gt;&#xA;&lt;li&gt;If a user will not buy the regular price, check if he/she will buy with an offer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With this way, I will avoid to make an offer to someone who would buy with the regular price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am still in the brainstorming and trying to find a way for implementing the 1-2. Should I create two separate models to predict the 1) and then the 2) with the second model? Or should I join both in one prediction model?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-11-20T13:16:08.807" Title="Predict which user will buy with an offer - discount" Tags="|predictive-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8396" PostTypeId="2" ParentId="8395" CreationDate="2015-10-12T07:00:25.807" Score="0" Body="&lt;p&gt;You can use Decision trees for a single model prediction for both the set of users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good start would be to first &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow&quot;&gt;read up&lt;/a&gt; on Decision Trees and their applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can include the offer as a decision(as a boolean in this case). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;buying with offer&lt;/code&gt; and &lt;code&gt;buying without offer&lt;/code&gt; can be the decision criterion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can, in fact go ahead and put in the offer values also. For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;offer&amp;gt;10%&lt;/code&gt; and &lt;code&gt;offer &amp;lt;10%&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-12T07:00:25.807" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8397" PostTypeId="2" ParentId="8388" CreationDate="2015-10-12T08:36:29.197" Score="1" Body="&lt;p&gt;I don't think k-means is the correct approach to this problem. You should look at doing topic modelling using latent semantic indexing.&lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-12T08:36:29.197" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8398" PostTypeId="2" ParentId="5893" CreationDate="2015-10-12T09:23:40.853" Score="2" Body="&lt;p&gt;It depends on your application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are doing topic modelling, try the default stopwords first. When there are some words occurring prominently in many topics (note my rather vague formulation) they are good candidates for additional stopwords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., in a corpus with texts containing figures and tabular material, the words &quot;fig&quot;, &quot;figure&quot;, &quot;tab&quot;, or &quot;table&quot; are good additional stopwords. In the result, your topics become more well-defined.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-10-12T09:23:40.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8399" PostTypeId="5" CreationDate="2015-10-12T09:36:13.850" Score="0" Body="&lt;p&gt;A topic model is a type of statistical model for discovering the abstract &quot;topics&quot; that occur in a collection of documents. Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: &quot;dog&quot; and &quot;bone&quot; will appear more often in documents about dogs, &quot;cat&quot; and &quot;meow&quot; will appear in documents about cats (source: &lt;a href=&quot;http://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Generative models&lt;/strong&gt; (i.e. the statistical models used for topic modelling)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Latent Dirichlet Allocation (LDA)&lt;/li&gt;&#xA;&lt;li&gt;Hierarchical Dirichlet process (HDP)&lt;/li&gt;&#xA;&lt;li&gt;Non-Negative Matrix Factorisation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Software / Libraries&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mallet.cs.umass.edu/&quot; rel=&quot;nofollow&quot;&gt;Mallet&lt;/a&gt; (Java) &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/software/tmt/tmt-0.4/&quot; rel=&quot;nofollow&quot;&gt;Stanford Topic Modeling Toolbox&lt;/a&gt; (software)  &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;Gensim – Topic Modelling for Humans&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/lda/index.html&quot; rel=&quot;nofollow&quot;&gt;lda&lt;/a&gt; (R)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-10-12T14:39:38.330" LastActivityDate="2015-10-12T14:39:38.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8400" PostTypeId="4" CreationDate="2015-10-12T09:36:13.850" Score="0" Body="A topic model describes text from a large corpus as a probability distribution over topics which are probability distributions over words. There are quantified contributions from all topics to a specific text." OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-10-12T13:37:50.393" LastActivityDate="2015-10-12T13:37:50.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8401" PostTypeId="1" CreationDate="2015-10-12T13:30:09.357" Score="4" ViewCount="433" Body="&lt;p&gt;If this is a duplicate, I apologize. I'm not really sure what to even search for to try and find a duplicate/answer!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are working on a system for providing musical feedback to change the 'mood' of a subject. There are a number of parameters of the music that we can manipulate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the same time, we measure the physiology of the subject. I'm using a dynamic Bayesian network to reasonably accurately determine the level of frustration of the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a way to, based on the level of frustration, tell the music generation mechanism &quot;this set of parameters is effective&quot;, or not. The music parameters would then adjust automatically and iteratively, as we continue to make judgements of the subject's level of frustration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate any pointers I can get! Please let me know if I can provide any clarification!&lt;/p&gt;&#xA;" OwnerUserId="13341" LastActivityDate="2018-02-01T05:04:25.973" Title="&quot;Relearning&quot; parameters" Tags="|machine-learning|parameter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8402" PostTypeId="1" AcceptedAnswerId="8425" CreationDate="2015-10-12T15:23:01.260" Score="4" ViewCount="582" Body="&lt;p&gt;Looking at the first paper on RDDs/Apache Spark, I found a statement saying that &quot;RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-based operations&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are scan-based operations in the context of RDDs and which of the &lt;a href=&quot;http://spark.apache.org/docs/latest/programming-guide.html#transformations&quot; rel=&quot;nofollow&quot;&gt;Transformations in Spark&lt;/a&gt; are scan-based operations&lt;/p&gt;&#xA;" OwnerUserId="339" LastEditorUserId="339" LastEditDate="2015-10-14T08:56:40.373" LastActivityDate="2015-10-15T15:37:53.093" Title="Scan-based operations Apache Spark" Tags="|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8403" PostTypeId="2" ParentId="8389" CreationDate="2015-10-12T15:48:35.563" Score="1" Body="&lt;p&gt;Let me give you some pointers (assuming that I'm right on this, which might not necessarily be true, so proceed with caution :-). First, I'd figure out the applicable terminology. It seems to me that your case can be categorized as &lt;em&gt;multivariate sampling&lt;/em&gt; from a &lt;em&gt;categorical distribution&lt;/em&gt; (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Categorical_distribution#Sampling&quot; rel=&quot;nofollow&quot;&gt;this section&lt;/a&gt; on categorical distribution sampling). Perhaps, the simplest approach to it is to use &lt;strong&gt;R&lt;/strong&gt; ecosystem's rich functionality. In particular, standard &lt;code&gt;stats&lt;/code&gt; package contains &lt;code&gt;rmultinom&lt;/code&gt; function (&lt;a href=&quot;http://stat.ethz.ch/R-manual/R-devel/library/stats/html/Multinom.html&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more complex types of sampling, there are other packages that might be worth exploring, for example &lt;code&gt;sampling&lt;/code&gt; (&lt;a href=&quot;https://cran.r-project.org/web/packages/sampling&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;), &lt;code&gt;miscF&lt;/code&gt; (&lt;a href=&quot;https://cran.r-project.org/web/packages/miscF&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;), offering &lt;code&gt;rMultinom&lt;/code&gt; function (&lt;a href=&quot;http://www.inside-r.org/packages/cran/miscF/docs/rMultinom&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;). If your complex sampling is focused on survey data, consider reading &lt;a href=&quot;http://faculty.washington.edu/tlumley/survey-jsm-nup.pdf&quot; rel=&quot;nofollow&quot;&gt;this interesting paper&lt;/a&gt; &quot;Complex Sampling and R&quot; by Thomas Lumley.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use languages other than R, check &lt;code&gt;multinomial&lt;/code&gt; function from Python's &lt;code&gt;numpy&lt;/code&gt; package and, for Stata, &lt;a href=&quot;https://www.kai-arzheimer.com/sampling-from-a-multinomial-distribution-in-stata&quot; rel=&quot;nofollow&quot;&gt;this blog post&lt;/a&gt;. Finally, if you are interested in Bayesian statistics, the following two documents seems to be relevant: &lt;a href=&quot;https://theclevermachine.wordpress.com/tag/multivariate-sampling-methods&quot; rel=&quot;nofollow&quot;&gt;this blog post&lt;/a&gt; and &lt;a href=&quot;http://www.stat.ufl.edu/~aa/cda/bayes.pdf&quot; rel=&quot;nofollow&quot;&gt;this survey paper&lt;/a&gt;. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="2452" LastEditDate="2015-10-12T17:04:00.643" LastActivityDate="2015-10-12T17:04:00.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8405" PostTypeId="2" ParentId="8381" CreationDate="2015-10-12T16:24:49.857" Score="0" Body="&lt;p&gt;The &quot;experience&quot; here is the 10000 rows of training data. One expects that with less data, one produces an inferior model (e.g. if there was 1 row of data, or 10 rows).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, it is not uncommon for models to be specified based on some training set and then used for prediction.&lt;/p&gt;&#xA;" OwnerUserId="9597" LastActivityDate="2015-10-12T16:24:49.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8406" PostTypeId="2" ParentId="8401" CreationDate="2015-10-12T16:33:04.363" Score="0" Body="&lt;p&gt;One basic approach would be a hill-climbing algorithm to search for local optima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be more or less effective depending on the complexity of the space (e.g. convexity: in theory will this find the local optima, but also more practical concerns around the size/dimensionality of the space and easy of specifying neighboring points and the accuracy/precision of the frustration measurement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also answers a slightly different question: the acceptance criteria &quot;this set of parameters is effective&quot; seems linked to some threshold of &quot;frustration&quot; that you would define as opposed to simply a process to &quot;minimize the frustration function with respect to these parameters&quot;, although one can view the latter as the way to make the iterative adjustments to search for the former.&lt;/p&gt;&#xA;" OwnerUserId="9597" LastActivityDate="2015-10-12T16:33:04.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8407" PostTypeId="2" ParentId="8389" CreationDate="2015-10-12T18:49:13.713" Score="1" Body="&lt;p&gt;To clarify, you have at least one observation in every possible category combination, but you only want to perform analysis on a subset of the total data, and are trying to decide how to choose which points to keep and which points to throw away?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the right approach here will depend strongly on what your hypothesis &lt;code&gt;h&lt;/code&gt; is, what sort of statistical tests you want to run, and what your loss function is. If you're trying to answer a question which can be answered by the number of datapoints in each combination, for example, or by the mean and stdev of some continuous variable for each combination, reducing the size of your data by sampling will only hurt your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying to learn a classifier, for example, a classic question is whether to train on a set with equal numbers of all possible classes or with the underlying class distribution found in the wild. The first will train a &quot;superior&quot; classifier, especially if its prior on class membership is later reset to the actual distribution in the wild, by most reasonable loss functions. But is your loss function one of the ones where this is better? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to look into &lt;a href=&quot;https://en.wikipedia.org/wiki/Design_of_experiments&quot; rel=&quot;nofollow&quot;&gt;design of experiments&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Combinatorial_design&quot; rel=&quot;nofollow&quot;&gt;combinatorial design&lt;/a&gt; in particular, which is trying to solve a symmetrical problem--starting with no data but being able to choose the various values, what set of points should we test to get as much information as possible about the underlying functions?&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-12T18:49:13.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8408" PostTypeId="2" ParentId="8381" CreationDate="2015-10-12T19:04:25.647" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Do we need to feed more data to the algorithm time by time and improve the model so that the model actually LEARNS?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;nofollow&quot;&gt;online learning&lt;/a&gt;. It is recommended for any application where the underlying relationships might change, and where feedback is quickly available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that isn't always the case. It may be the case that once a model is learned, we don't expect it to change, and so this sort of 'continuing education' is unnecessary, and that feedback could be costly or dangerous. (Imagine playing Rock Paper Scissors with a bot whose updating algorithm you know; you should be able to reliably win by feeding it moves designed to give you an opportunity to know its move and outfox it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or it may be the case that feedback is available periodically, where my experience has been that people often retrain the model from scratch, rather than trying to do a warm start on the previous model.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-12T19:04:25.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8410" PostTypeId="1" CreationDate="2015-10-12T20:02:20.073" Score="-2" ViewCount="93" Body="&lt;p&gt;I want to type in &quot;car&quot; and get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One day, I came to a car and bought it.&#xA;Teacher told us that cars are pricey.&#xA;etc. etc. etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically just sentences where the word is used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Opensource and german preferred.&lt;/p&gt;&#xA;" OwnerUserId="13350" LastActivityDate="2015-10-20T23:30:54.157" Title="A service to generate examples of context of a word?" Tags="|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8411" PostTypeId="2" ParentId="8410" CreationDate="2015-10-12T20:30:50.213" Score="0" Body="&lt;p&gt;Wiktionary has ever-improving example sentences and open source tools for programmatically using such data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note also that Google Translate does this, but it is not offered in the official API.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2015-10-12T20:30:50.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8412" PostTypeId="2" ParentId="8349" CreationDate="2015-10-12T21:22:43.730" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The data has such a property that one or very few vectors&#xA;  influence strongly the class label.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The best (and easiest) approach would probably be to simply train a classifer on each vector and then average the predictions across the vectors for a given sequence. The important vectors would be strongly influential in their predictions, whereas the predictions for the unimportant vectors would be close to 0.5 (or similar for a non-binary classification problem).&lt;/p&gt;&#xA;" OwnerUserId="3044" LastActivityDate="2015-10-12T21:22:43.730" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8413" PostTypeId="2" ParentId="8380" CreationDate="2015-10-12T23:04:44.623" Score="2" Body="&lt;p&gt;Hadoop, HDFS etc. come at a substantial cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your application and code quality, the &lt;em&gt;break even&lt;/em&gt; is usually somewhere between 10 and 100 nodes: a beginners Hadoop job may easily run 100x slower than an average non-Hadoop job, so you need this many nodes (and data!) to make up for the overhead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gets much worse when the non-Java APIs are used. The native layer of Hadoop is Java, and running e.g. a Python or R map-reduce job will usually require multiple serializations and deserializations of data into text and back. For simple map jobs such as the famous word-count example, this overhead is more than the actual task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a rule of thumb: &lt;strong&gt;if your data fits into main memory, don't put it on hadoop&lt;/strong&gt; but process it locally. It may be much more effective to use multithreading or GPUs then. On the other hand, if your data is several TB, and you only need a tiny bit of that data, reading from multiple harddisks in parallel &lt;em&gt;does&lt;/em&gt; offer advantages. That is why a lot of the use cases follow the ETL pattern. Scan a large amount of raw data, extract the required parts, transform them into the desired format, and load them into a different system for further processing.&#xA;After that, the data usually is small enough to be processed with &quot;conventional&quot; tools. It's disappointing, but PCs have just a &lt;em&gt;lot&lt;/em&gt; of RAM these days... and CPU+GPU power is plenty, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;strong&gt;benchmark&lt;/strong&gt;. You'll be surprised how expensive the network and architecture overheads are.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2015-10-12T23:10:29.413" LastActivityDate="2015-10-12T23:10:29.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8414" PostTypeId="2" ParentId="8370" CreationDate="2015-10-12T23:55:22.950" Score="4" Body="&lt;p&gt;&lt;strong&gt;Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths.&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key point is that the cosine similarity metric normalizes based on length, so that only the unit vectors are considered.  I know this isn't necessarily the answer that you wanted, but the math clearly shows that the cosine similarity metrics is agnostic to vector length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Lets look at the math in more detail:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are applying a cosine similarity metric and requiring that that metric be larger than 0.6:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$similarity=\cos{(\theta)}=\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||}\geq0.6$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the scalar lengths on the bottom can be distributed into the cross products above (distributive property):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||} = \frac{\mathbf{A}}{||A||}\cdot\frac{\mathbf{B}}{||B||}=\hat{\mathbf{A}}\cdot\hat{\mathbf{B}}$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now $\hat{\mathbf{A}}$ and $\hat{\mathbf{B}}$ are vectors that point in the same direction as $\mathbf{A}$ and $\mathbf{B}$ but they have been normalized to length one.  So the definition of the cosine similarity metric is the take the original vectors, normalize them to length one, and then measure the dot product of the unit vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefor: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$similarity=\cos{(\theta)}=\frac{\mathbf{d1}\cdot\mathbf{d2}}{||d1|| ||d2||}=\hat{\mathbf{d1}}\cdot\hat{\mathbf{d2}}\geq0.6$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;depends only on the orientation of the vectors and not on their magnitude (i.e. length).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reconciling this with what you are doing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite what the results of the linear algebra show, you might still be seeing a statistically significant result.  Practically speaking you may be finding that the statistics show that length restrictions are valid for your data.  For instance, you might be finding that tweets never share a cosine similarity $\geq0.6$ when compared with Tolstoy's &quot;War and Peace&quot;.  If your statistics look good for using $||d2|| &amp;gt; .8 ||d1||$ and $||d2|| &amp;lt; ||d1|| / .8$ then I suggest you go with it as these type of canopy restrictions are very useful in saving computing time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can perhaps reconcile what you have been doing with distance metrics by also considering the Euclidean distance.  Where as the cosine similarity only returns a value between -1 and 1 based on the angle between the two vectors, the Euclidean distances will return values that depend on the lengths of the two vectors.  In some sense, you are combining aspects of the Euclidean distance with cosine similarity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes fairly good sense to require that the relative lengths be within 25% of one another in the sense that this combines an aspect of Euclidean distance to create group-by canopies, which cuts computation time, then length agnostic cosine similarity can be used as the final determinant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that 1/.8 =1.25, so d2&gt;=.8d1 is a tighter restriction than d2&amp;lt;=d1/.8.  I suggest using d2&gt;=.75d1 and d2&amp;lt;=1.25d1 as this is symmetric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-10-13T00:50:31.787" LastActivityDate="2015-10-13T00:50:31.787" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8415" PostTypeId="2" ParentId="8370" CreationDate="2015-10-12T23:55:44.340" Score="3" Body="&lt;p&gt;First, let's try to get some intuition why this would work. $||d_i||$ seems to serve as a word rarity measure, which seems plausible as something to filter on. If documents use dissimilar numbers of rare words, it'll be difficult for them to line up on the cosine similarity measure. But it seems unlikely to me that this cutoff will &lt;em&gt;only&lt;/em&gt; depend on $||d_i||$, rather than also on the structure in the tf or idf weights that go into $||d_i||$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To work through some algebra, let me introduce a few more terms (and rename some to shorter ones):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $d_1$ be a vector of tf weights $[t_1, t_2, ...]$ element-wise multiplied by a vector of idf weights $[w_1, w_2, ...]$ to get the final weights $[d_1, d_2, ...]$. We know that $0.5\le t_i\le 1$ and $0\le w_i\le 6$ (because of the corpus size and assuming we're using base 10, it doesn't matter if we're not). Let $D_1=||d_1||$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing $d_1$, we want to construct a delta vector $x$, such that $d_1+x$ has the minimal (or maximal) $X$ subject to the constraints that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X=\sqrt{\sum_i w_i^2 (t_i+x_i)^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0.6D_1X\le \sum_i w_i^2t_i(t_i+x_i)$ (1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0.5\le t_i+x_i \le 1$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because we didn't use the raw tf weight for $x$, $x_i=0\ \forall i$ is in the solution space. I'm also ignoring the more complicated constraint that at least one $d_i+x_i=1$, because we can't express that linearly. We'll leave it as is and hope that the optimizer ends up setting one of them to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it seems like the set of possible $x$ should be convex, but even if so we're already in the realm of &lt;a href=&quot;https://en.wikipedia.org/wiki/Quadratically_constrained_quadratic_program&quot; rel=&quot;nofollow&quot;&gt;quadratically constrained programming&lt;/a&gt;. Note that we can solve for minimal $X^2$ instead of minimal $X$, because $X&amp;gt;0$, but we probably can't use this methodology to maximize $X$ (i.e. minimize $-X$). But thankfully this'll be easily solvable if $P$ is &lt;a href=&quot;https://en.wikipedia.org/wiki/Positive-definite_matrix&quot; rel=&quot;nofollow&quot;&gt;positive semidefinite&lt;/a&gt;. So what is $P$? We need to rewrite (1) in the correct form, which starts by squaring both sides:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0\ge 0.36D_1^2\sum_i w_i^2 (t_i+x_i)^2-\sum_{i,j}w_i^4t_it_j(t_i+x_i)(t_j+x_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can rewrite this as $0\ge x^TPx+q^Tx+r$ where $P_{i,j}=0.36D_1^2-w_i^2t_it_j$ if $i=j$ and $-w_i^2t_it_j$ otherwise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's non-obvious to me that $P$ has to be positive semidefinite, but that'll be easy to check for any individual $d_1$. If so, pop this into a QP solver and you'll get a lower bound on $X$. If not, we're in trouble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we also get a practical upper bound? I'm not sure. Obviously there's some finite upper bound, since we can calculate the maximum possible $X$ from the idf vector $w$ easily. But the fact that the minimum tf weight is 0.5 instead of 0 is throwing off my intuitions about how to create an adversarial $x$ with maximum $X$, and so the best approach that I'm coming up with is gradient descent, which may or may not find the actual global maximum but will probably be close.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastEditorUserId="12876" LastEditDate="2015-10-13T00:13:42.093" LastActivityDate="2015-10-13T00:13:42.093" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8417" PostTypeId="2" ParentId="8381" CreationDate="2015-10-13T07:50:43.187" Score="2" Body="&lt;p&gt;There is not really a &quot;right way&quot; to use machine learning outside of running the algorithms, any more than there is a &quot;right way&quot; to use a sorting algorithm, or any other kind of complex automation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some guidelines/thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You can start using your model to make predictions, once those predictions are useful to you. This may be once they pass some threshold in performance that would make the predictions the best way of achieving some other goal. That may just mean &quot;better than the last ML model&quot;, but could also mean &quot;cheaper and faster than having some people doing it, even if it is less accurate&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Depending on how your training data is collected, and how the model works, it may be possible to &lt;em&gt;both&lt;/em&gt; use your model for predictions, and to continue training it on new data as it arrives.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;However, there is no &lt;em&gt;requirement&lt;/em&gt; for continuous learning. Learning systems that continuously adapt to new data are useful in some situations (example might be recommendation systems), but might be inappropriate in others (example might be automatic driver for a vehicle, where you don't want to risk reduction in performance whilst in use).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One thing that &lt;em&gt;is&lt;/em&gt; considered very standard practice is to demonstrate how the machine learning system is performing after a period of learning by using a test metric. The simplest variation is to keep some of your precious data to one side, deliberately not training on it, then measure the performance. Often two such sets are kept aside - a &lt;em&gt;cross validation&lt;/em&gt; set is used to help decide automatically between variations of your learning algorithm, and a &lt;em&gt;test&lt;/em&gt; set is used to assess the end result of that selection.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-10-13T07:50:43.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8418" PostTypeId="2" ParentId="8410" CreationDate="2015-10-13T09:33:02.640" Score="1" Body="&lt;p&gt;The Leipzig Wortschatz project offers such services on their Web interface, try &lt;a href=&quot;http://corpora.informatik.uni-leipzig.de/&quot; rel=&quot;nofollow&quot;&gt;http://corpora.informatik.uni-leipzig.de/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also download their corpora.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-10-13T09:33:02.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8419" PostTypeId="2" ParentId="1095" CreationDate="2015-10-13T13:55:36.163" Score="6" Body="&lt;p&gt;I took an example of Data with two people A and B with wealth of unit 1 and unit 3 respectively. &#xA;Gini Impurity as per Wikipedia = 1 - [ (1/4)^2 + (3/4)^2 ] = 3/8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gini coefficient as per Wikipedia would be  ratio of area between red and blue line to the total area under blue line in the following graph&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HDPUo.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HDPUo.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Area under red line is 1/2 + 1 + 3/2 = 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total area under blue line = 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So Gini coefficient = 3/4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly the two numbers are different. I will check more cases to see if they are proportional or there is an exact relationship and edit the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I checked for other combinations as well, the ratio is not constant. Below is a list of few combinations I tried.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/GjXjN.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GjXjN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9941" LastEditorUserId="9941" LastEditDate="2016-07-15T09:46:25.023" LastActivityDate="2016-07-15T09:46:25.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8421" PostTypeId="1" CreationDate="2015-10-13T16:17:28.133" Score="7" ViewCount="8289" Body="&lt;p&gt;I am building a recommender for web pages. For each web page in our data set, we wish to generate a list of web pages that other users have also visited. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our data only shows that a user has either visited a page, or they have not. Users do not provide any ratings of our web pages. This is a good task for item based recommendation. However, most of the algorithms (such as the one in Mahout) requires rating data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first solution I came up with was to use a graph database and write a query which does the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each page we want recommendations for, we search for all the users who have viewed that page. Then, for each of those users, we look up all other pages they have viewed. We then count the number of users which have viewed each page in this data set, and use those with the highest count as our recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this works pretty well, our data set has grown substantially and scaling the graph database is difficult. The queries become slower as the number of page views in our data set increases. We would like to consider a different implementation before we commit to moving to a distributed graph database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a more traditional item-based recommender (like Mahout's), is there a good way to 'fake' the ranking data, or is there a popular open source implementation which does not requires the ranking data?&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2016-05-31T16:18:48.583" Title="Item Based Collaborative Filtering with No Ratings" Tags="|recommender-system|open-source|apache-mahout|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8422" PostTypeId="1" CreationDate="2015-10-13T18:15:18.267" Score="1" ViewCount="47" Body="&lt;p&gt;What's the recommended approach for identifying new predictors when building gradient boosting or support vector machine predictive models? I know that visualizations are often used to come up with new predictors and/or transformations for linear models because linear models aren't able to capture non-linear behaviors or interactions in the same way that gradient boosting can. Are there visualization techniques that provide insight that you wouldn't get by just adding the candidate predictor to the model and seeing if it improves out-of-sample error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just using gradient boosting as an example, but I'm actually interested in &#xA;visualization techniques that could be applied to other powerful models like &#xA;support vector machines too. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2015-10-13T18:15:18.267" Title="Visualization techniques best suited for building non-linear models" Tags="|predictive-modeling|visualization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8424" PostTypeId="2" ParentId="8421" CreationDate="2015-10-13T21:20:25.580" Score="5" Body="&lt;p&gt;There might be different ways to do that, like considering implicit ratings like views or clicks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But basically, you can consider a rating of 1.0 for each user-item pair you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, your prediction will be between 0 and 1 which you can consider similar to a click prediction probability. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 being the probability on not having a click or the item by a user &lt;/li&gt;&#xA;&lt;li&gt;1 being the probability that a click would be made. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also set a threshold where you can drop a recommendation at a certain limit of your prediction level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. Let's say an user alpha has the following recommendations with a threshold of 0.7&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;i1 , 0.87&lt;/li&gt;&#xA;&lt;li&gt;i2, 0,75&lt;/li&gt;&#xA;&lt;li&gt;i3, 0,6&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can drop the i3 recommendation since you don't consider it good enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, the parameter of your recommendation engine must be determined of course with the help of your evaluation metrics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering software solutions, I use Apache Spark MLlib with Scala as a base for my recommendation engine algorithms where you can compute item cosine similarity easily per example where you are using in-house implementation or an approximation with the DIMSUM algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="5177" LastEditDate="2015-10-13T22:06:15.037" LastActivityDate="2015-10-13T22:06:15.037" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8425" PostTypeId="2" ParentId="8402" CreationDate="2015-10-13T22:22:02.087" Score="3" Body="&lt;p&gt;Scan based operations are basically all the operations that require evaluating the predicate on an RDD. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other terms, each time your create an RDD or a DataFrame in which you need to compute a &lt;em&gt;predicate&lt;/em&gt; like performing a filter, map on a case class, per example, or even explain method will be considered as a scan based operation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more clear, let's review the definition of a predicate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A predicate or a functional predicate is a logical symbol that may be applied to an object term to produce another object term. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Functional predicates are also sometimes called &lt;em&gt;mappings&lt;/em&gt;, but that term can have other meanings as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;// scan based transformation&#xA;rdd.filter(!_.contains(&quot;#&quot;)) // here the predicate is !_.contains(&quot;#&quot;)&#xA;&#xA;// another scan based transformation&#xA;rdd.filter(myfunc) // myfunc is a boolean function&#xA;&#xA;// a third also trivial scan based transformation followed by a non scan based one.&#xA;rdd.map(myfunc2) &#xA;   .reduce(myfunc3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to understand how spark internals work, I suggest that you watch the &lt;a href=&quot;https://spark-summit.org/2014/talk/a-deeper-understanding-of-spark-internals&quot; rel=&quot;nofollow&quot;&gt;presentation&lt;/a&gt; made by Databricks about the topics&lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="5177" LastEditDate="2015-10-15T15:37:53.093" LastActivityDate="2015-10-15T15:37:53.093" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="8426" PostTypeId="1" AcceptedAnswerId="8429" CreationDate="2015-10-13T22:23:39.020" Score="8" ViewCount="10517" Body="&lt;p&gt;I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two vectors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [2.7654870801855078, 0.35995355443076027, 0.016221679989074141, -0.012664358453398751, 0.0036888812311235068]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b = [-6.2588482809118942, -0.88952297609194686, 0.017336984676103874, -0.0054928004763216964, 0.011122959185936367]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running the following code will produce an output of ~1.999:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.spatial import distance&#xA;print(distance.cosine(a,b))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there something wrong with my input values? Anyone know why I am getting a result of &gt;1?&lt;/p&gt;&#xA;" OwnerUserId="13385" LastEditorUserId="11097" LastEditDate="2015-10-15T01:10:16.887" LastActivityDate="2015-10-15T01:10:16.887" Title="Cosine Distance &gt; 1 in scipy" Tags="|python|distance|cosine-distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8427" PostTypeId="1" CreationDate="2015-10-14T03:21:48.330" Score="4" ViewCount="715" Body="&lt;p&gt;I'm doing a keyword extraction using TF-IDF on a large number of documents. Currently, I'm splitting each sentence based on n-gram. More particularly, I'm using trigrams. However, this is not the best way to split each sentence into ints constituting keywords. For example a noun phrase like 'triple heart bypass' may not always get detected as one term. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other alternative to chunk each sentence into its constituting elements look to be part of speech tagging and [chunking][1] in [Open NLP][2]. In this approach phrase like 'triple heart bypass' always gets extracted as a whole but the downside is in TF-IDF the frequency of extracted terms (phrases) dramatically drops. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any suggestion on either or these two approaches or have any other ideas to improve the quality of the keywords?&lt;/p&gt;&#xA;" OwnerUserId="13390" LastEditorUserId="21" LastEditDate="2015-10-20T10:12:48.627" LastActivityDate="2016-11-14T05:54:29.920" Title="What is the best way to split a sentence for a keyword extraction task?" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8428" PostTypeId="1" AcceptedAnswerId="9475" CreationDate="2015-10-14T05:56:06.070" Score="1" ViewCount="1190" Body="&lt;p&gt;I am working on a 4-class classification task using an Artificial Neural Network. My aim is to visualize how well those 4 classes can be separated, moreover how consistent the data coming from each class is and how far they are &quot;away&quot; from one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach is to use 2 neurons in the last hidden layer and plot the output of the neurons in a scatterplot. However my current output usually looks like this where the different colors stand for the actual classes the data is coming from:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xbb9H.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xbb9H.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously this is not very useful. First of all I was hoping that the points would not all be all along the same line and second of all I am not sure if this can even be interpreted in the way that I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the neuralnet package in R to train the network using linear output. I am guessing that if this approach can work the choice of activation function would be crucial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to Neural Networks and I am not sure whether or not something like this can be done or not and I am hoping to get some comments.&lt;/p&gt;&#xA;" OwnerUserId="13392" LastActivityDate="2015-12-22T01:17:41.407" Title="Use the output of 2 hidden neurons in the last hidden layer of a NN to visualize the result of a 4-class classification task" Tags="|r|classification|neural-network|visualization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8429" PostTypeId="2" ParentId="8426" CreationDate="2015-10-14T07:46:51.447" Score="10" Body="&lt;p&gt;The cosine distance formula is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UmTFw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UmTFw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the formula used by the &lt;code&gt;cosine&lt;/code&gt; function of the &lt;code&gt;spatial&lt;/code&gt; class of scipy is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KLk4n.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KLk4n.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the actual cosine similarity metric is: -0.9998.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it signifies complete dissimilarity.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-10-14T07:46:51.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8432" PostTypeId="2" ParentId="8332" CreationDate="2015-10-14T13:32:46.583" Score="1" Body="&lt;h2&gt;Found the answer&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I found out why it was going slowly, after doing some profiling and reading &lt;a href=&quot;https://groups.google.com/forum/#!topic/theano-users/xX_p8xm3pqg&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out that if you do a convolution with a stride != 1, then it doesn't use the GPU for that operation (or perhaps the gradient of that operation), and makes things run much slower.  I was only using a non-1 stride to save execution time anyway, but turns out I'm better off getting rid of the stride.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that in your convoution layer, you have to completely get rid of &lt;code&gt;stride=X&lt;/code&gt;.  Just having &lt;code&gt;stride=1&lt;/code&gt; is not sufficient.&lt;/p&gt;&#xA;" OwnerUserId="13216" LastEditorUserId="11097" LastEditDate="2015-10-15T01:10:05.927" LastActivityDate="2015-10-15T01:10:05.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8433" PostTypeId="2" ParentId="5226" CreationDate="2015-10-14T19:18:50.540" Score="12" Body="&lt;p&gt;You need to encode your strings as numeric features that sci-kit can use for the ML algorithms. This functionality is handled in the preprocessing module (e.g.,  see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html#sklearn.preprocessing.LabelEncoder&quot; rel=&quot;noreferrer&quot;&gt;sklearn.preprocessing.LabelEncoder&lt;/a&gt; for an example).&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2015-10-14T19:28:20.260" LastActivityDate="2015-10-14T19:28:20.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8435" PostTypeId="1" AcceptedAnswerId="8463" CreationDate="2015-10-14T20:55:21.260" Score="1" ViewCount="1451" Body="&lt;p&gt;I am really struggling to replicate the output of the dist() function in R code without using 1 or 2 for loops. (If you're wondering why I'm doing this, it's so that I can play around with the distance calculation, and also to improve my R skills - so please only solutions that involve R!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overview: matrix is passed to dist(), which calculates the Euclidean distance row-wise and outputs a full distance matrix of the distance between each row (e.g. distance between rows 1 and 50 will be in distancematrix[1, 50] and distancematrix[50, 1]). The fast code looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distancematrix &amp;lt;- as.matrix(dist(myMatrix, method=&quot;euclidean&quot;, diag = T, upper = T))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have successfully produced the same output in R using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:nrow(myMatrix)) {&#xA;  for (j in 1:nrow(myMatrix)) {&#xA;    distancematrix[i, j] &amp;lt;- sum(abs(myMatrix[i,] - myMatrix[j,])) &#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, using two nested for loops is much slower than using dist(). I have read a lot about using apply() to optimise slower for loops, but I haven't been able to get my head around it so far. I believe that at least one of the for loops is definitely avoidable by just outputting a vector, and dealing with it at the end. However, I cannot for the life of me work out how to remove both for loops. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="13416" LastEditorUserId="29575" LastEditDate="2018-08-20T00:33:42.273" LastActivityDate="2018-08-20T00:33:42.273" Title="How can I vectorize this code in R? Maybe with the apply() function?" Tags="|machine-learning|r|programming|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8437" PostTypeId="2" ParentId="8357" CreationDate="2015-10-14T22:49:14.240" Score="1" Body="&lt;p&gt;A further approach is to sample from models with a countably infinite number of states. The answer to 'how many' would then emerge by averaging over the samplers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mlg.eng.cam.ac.uk/zoubin/papers/ihmm.pdf&quot; rel=&quot;nofollow&quot;&gt;http://mlg.eng.cam.ac.uk/zoubin/papers/ihmm.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-10-14T22:49:14.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8440" PostTypeId="1" AcceptedAnswerId="8452" CreationDate="2015-10-15T02:06:37.107" Score="3" ViewCount="3236" Body="&lt;p&gt;I'm working with decision trees in python's scikit learn. Unlike many use cases for this, I'm not so much interested in the accuracy of the classifier at this point so much as I am extracting the specific path a data point takes through the tree when I call &lt;code&gt;.predict()&lt;/code&gt; on it. Has anyone done this before? I'd like to build a data frame containing ($X_{i}$, path$_{i}$) pairs for use in a down-stream analysis.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-15T22:28:32.387" Title="Extract the &quot;path&quot; of a data point through a decision tree in sklearn" Tags="|machine-learning|python|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8441" PostTypeId="2" ParentId="1025" CreationDate="2015-10-15T03:26:35.007" Score="4" Body="&lt;p&gt;I recommend anybody interested in the topic take a look at the &lt;a href=&quot;http://arxiv.org/abs/1509.01549&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; which combines TDL and deep-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Roughly, you'll need to make the engine play games against each other. Record the mini-max evaluation for each position. At the end of the game, you'll get a reward, which is {0,1,-1} for chess. Then you'll need to adjust your parameters with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UPXEl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UPXEl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This equation tells us we should adjust the weights by temporal differences, weighted by how far you should do it. If you have a perfect evaluation, your temporal difference would always be zero, thus you wouldn't need to make any adjustment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you'll need use the new parameters to play a new game. Repeats until as many games as you can afford or when you think it's converged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few remarks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The paper I quote applies a discount factor. It's done for the backproportion algorithm for neural network. You don't need it.&lt;/li&gt;&#xA;&lt;li&gt;You'll need to experiment with the optimal learning rate (alpha in the equation). Too large will make your learning unstable, too little will take longer to converge. I've seen people using 0.70. The paper I quote used 1.0.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-03-29T04:52:04.283" LastActivityDate="2016-03-29T04:52:04.283" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8444" PostTypeId="1" AcceptedAnswerId="8446" CreationDate="2015-10-15T13:29:08.780" Score="1" ViewCount="73" Body="&lt;p&gt;I have 90k points in $\mathbb{R}^{32}$ (i.e., a 90k by 32 real matrix) which I want to visualize. I know I can cluster my points (k-means &amp;amp;c), but I want to select a few &quot;interesting&quot; 2-planes in $\mathbb{R}^{32}$, project the points there and scatter-plot them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I select the interesting 2-planes to project to?&lt;/p&gt;&#xA;" OwnerUserId="9867" LastEditorUserId="9867" LastEditDate="2015-10-15T15:31:35.297" LastActivityDate="2015-10-15T15:31:35.297" Title="Optimal projection for data visualization" Tags="|r|visualization|dimensionality-reduction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8446" PostTypeId="2" ParentId="8444" CreationDate="2015-10-15T14:48:40.253" Score="4" Body="&lt;p&gt;There are a few interesting plots and transformations you could start with, each dependent upon the purpose of your analysis. Below are some first steps I might take.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If simply visually looking for clusters:&lt;br&gt;&#xA;If clusters are what you are after, then I would recommend applying a Principal Component Analysis to the dataset and then plotting the dataset with the first 2 principal components as the axes. However, the major downside to PCA is that you will have to go &quot;unpack&quot; the principal components to find the original variables. In other words, you'll be may be able to identify cool clusters, but it will be a little harder to tie the findings back to your 32 variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If looking for quick relationships between variables to build on:&lt;br&gt;&#xA;With &quot;only&quot; 32 variables you could do a pairwise plot. However, a smarter way may be to first identify the relationships mathematically (e.g. correlation) and then plotting those variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If looking for quick relationships in each variable to build on:&#xA;Or, look at 32 histograms to start off with. Look out for clear bi-modal (or more) to begin piecing together an understanding of how your variables may contribute to an unsupervised model. If you end up look at 32 unimodal histograms then you can conclude early that no matter how you cluster, you will simply end up with a blob.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Actually, in usual analytics workflows I would go 3 &gt; 2 &gt; 1. But if I was fishing for clusters and just wanted to see if clusters will appear or not, PCA would be a good shortcut.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, feel free to sample your dataset. 90k points on your screen will likely do more harm than good.&lt;/p&gt;&#xA;" OwnerUserId="525" LastEditorUserId="9867" LastEditDate="2015-10-15T14:54:12.767" LastActivityDate="2015-10-15T14:54:12.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8449" PostTypeId="2" ParentId="6027" CreationDate="2015-10-15T15:30:11.107" Score="1" Body="&lt;p&gt;I would view my points as elements of $\mathbb{R}^{n+1}$ and use &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt; to project my points to $\mathbb{R}^3$ and then scale them to $S^2$.&lt;/p&gt;&#xA;" OwnerUserId="9867" LastActivityDate="2015-10-15T15:30:11.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8450" PostTypeId="1" CreationDate="2015-10-15T20:05:36.923" Score="1" ViewCount="395" Body="&lt;p&gt;I have over a million text documents that I would like to cluster. I used tf-idf modeling and term vector cosine for identifying similar documents in the corpus, which appeared to work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some documents are annotated with issues labels (e.g., a,b,c,d – twenty in all).  Each issue has as least 100 documents.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to compute the average document for each cluster. What would be the best approach to normalization in this context? Right now, I am normalizing using &lt;span class=&quot;math-container&quot;&gt;$0.5 + \big{(}0.5 * \frac{term}{\max{term}}\big{)}$&lt;/span&gt;. Would it be better to compute the average of the normalized vectors instead? Alternatively, I could compute the sum of all raw term frequencies and then normalize that. What is the best approach to create a normalized term vector for each issue?&lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="1330" LastEditDate="2020-05-25T19:05:04.610" LastActivityDate="2020-05-25T19:05:04.610" Title="How to normalized term vector for document clustering?" Tags="|clustering|cosine-distance|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="8451" PostTypeId="1" CreationDate="2015-10-15T20:07:41.827" Score="2" ViewCount="334" Body="&lt;p&gt;In performing ALS and getting an item matrix of latent features, what would be the best method for inferring the possible &quot;meaning&quot; of each latent factor in the item space? And as a corollary, are the any methods for successfully visualizing this data to gain some intuition or understanding?  &lt;/p&gt;&#xA;" OwnerUserId="13441" LastActivityDate="2016-01-14T23:30:29.043" Title="Visualizing Latent Features" Tags="|machine-learning|recommender-system|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8452" PostTypeId="2" ParentId="8440" CreationDate="2015-10-15T22:28:32.387" Score="2" Body="&lt;p&gt;Looks like this is easier to do in R, using the &lt;code&gt;rpart&lt;/code&gt; library in combination with the &lt;code&gt;partykit&lt;/code&gt; library. I'd ideally like to find a way to do this in python, but here's the code, for anyone who is interested (taken from &lt;a href=&quot;https://stackoverflow.com/questions/29618490/get-decision-tree-rule-path-pattern-for-every-row-of-predicted-dataset-for-rpart&quot;&gt;here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pathpred &amp;lt;- function(object, ...){&#xA;    ## coerce to &quot;party&quot; object if necessary&#xA;    if(!inherits(object, &quot;party&quot;)) object &amp;lt;- as.party(object)&#xA;&#xA;    ## get standard predictions (response/prob) and collect in data frame&#xA;    rval &amp;lt;- data.frame(response = predict(object, type = &quot;response&quot;, ...))&#xA;    rval$prob &amp;lt;- predict(object, type = &quot;prob&quot;, ...)&#xA;&#xA;    ## get rules for each node&#xA;    rls &amp;lt;- partykit:::.list.rules.party(object)&#xA;&#xA;    ## get predicted node and select corresponding rule&#xA;    rval$rule &amp;lt;- rls[as.character(predict(object, type = &quot;node&quot;, ...))]&#xA;&#xA;    return(rval)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Illustration using the iris data and rpart():&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;rpart&quot;)&#xA;library(&quot;partykit&quot;)&#xA;rp &amp;lt;- rpart(Species ~ ., data = iris)&#xA;rp_pred &amp;lt;- pathpred(rp)&#xA;rp_pred[c(1, 51, 101), ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Yielding,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       response prob.setosa prob.versicolor prob.virginica&#xA; 1       setosa  1.00000000      0.00000000     0.00000000&#xA; 51  versicolor  0.00000000      0.90740741     0.09259259&#xA; 101  virginica  0.00000000      0.02173913     0.97826087&#xA;                                           rule&#xA; 1                          Petal.Length &amp;lt; 2.45&#xA; 51   Petal.Length &amp;gt;= 2.45 &amp;amp; Petal.Width &amp;lt; 1.75&#xA; 101 Petal.Length &amp;gt;= 2.45 &amp;amp; Petal.Width &amp;gt;= 1.75&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which looks to be something I could at least use to derive shared parent node information.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-10-15T22:28:32.387" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8455" PostTypeId="1" CreationDate="2015-10-15T23:21:30.180" Score="1" ViewCount="260" Body="&lt;p&gt;I have a dataset where I need to explore using unsupervised technics (clustering and association rules). What are the best strategies to discretize the numeric attributes? Also, does this (attribute discretization) makes sense for clustering (using K-Means)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I'm using weka, I know that I can just use the Discretize filter, with findNumBins option or useEqualFrequency. But are there other strategies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, does it makes sense to use discretization on the attributes, and then look for clusters (using K-Means) or not? (I suspect that it doesn't make that much sense, but I just want to confirm)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="13447" LastEditorUserId="11097" LastEditDate="2015-11-15T18:02:15.587" LastActivityDate="2015-11-15T18:02:15.587" Title="Attribute discretization" Tags="|clustering|k-means|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8457" PostTypeId="1" CreationDate="2015-10-16T04:07:42.020" Score="27" ViewCount="39395" Body="&lt;p&gt;I am looking for a Python library that can perform &lt;a href=&quot;https://en.wikipedia.org/wiki/Segmented_regression#Segmented_linear_regression.2C_two_segments&quot; rel=&quot;noreferrer&quot;&gt;segmented regression (a.k.a. piecewise regression)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://onlinecourses.science.psu.edu/stat501/node/310&quot; rel=&quot;noreferrer&quot;&gt;Example&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZNoPv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZNoPv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2022-09-05T13:42:57.013" Title="Python library for segmented regression (a.k.a. piecewise regression)" Tags="|python|linear-regression|library|software-recommendation|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8459" PostTypeId="2" ParentId="8455" CreationDate="2015-10-16T06:27:14.760" Score="2" Body="&lt;p&gt;Most clustering algorithms work best with &lt;em&gt;continuous&lt;/em&gt; values (in particular k-means). Many cannot use categorial attributes and won't work very well with one-hot encodings either.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-10-16T06:27:14.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8460" PostTypeId="1" CreationDate="2015-10-16T06:45:35.140" Score="20" ViewCount="16490" Body="&lt;p&gt;What &lt;em&gt;stable&lt;/em&gt; Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, is there a more direct approach to performing a time-series analysis on a data-set using HMM?&lt;/p&gt;&#xA;" OwnerUserId="13450" LastActivityDate="2020-08-03T09:23:35.423" Title="Python library to implement Hidden Markov Models" Tags="|python|time-series|markov-process|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8463" PostTypeId="2" ParentId="8435" CreationDate="2015-10-16T13:16:05.703" Score="2" Body="&lt;p&gt;First of all it should be noted that the code you posted does not actually replicate the output of the &lt;code&gt;dist&lt;/code&gt; function, because the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distancematrix[i, j] &amp;lt;- sum(abs(myMatrix[i,] - myMatrix[j,]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;does not calculate the Euclidean distance; it should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distancematrix[i, j] &amp;lt;- sqrt(sum((myMatrix[i,] - myMatrix[j,]) ^ 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are two solutions that rely on &lt;code&gt;apply&lt;/code&gt;. They are simplified, and in particular do not take advantage of the symmetry of the distance matrix (which, if considered, would lead to a 2-fold speedup). First, generate some test data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Number of data points&#xA;N &amp;lt;- 2000&#xA;# Dimensionality&#xA;d &amp;lt;- 10&#xA;# Generate data&#xA;myMatrix = matrix(rnorm(N * d), nrow = N)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For convenience, define:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Wrapper for the distance function&#xA;d_fun &amp;lt;- function(x_1, x_2) sqrt(sum((x_1 - x_2) ^ 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first approach is a combination of &lt;code&gt;apply&lt;/code&gt; and &lt;code&gt;sapply&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time(&#xA;    D_1 &amp;lt;- &#xA;        apply(myMatrix, 1, function(x_i) &#xA;            sapply(1:nrow(myMatrix), function(j) d_fun(x_i, myMatrix[j, ]))&#xA;    )&#xA;)&#xA;&#xA;   user  system elapsed &#xA; 14.041   0.100  14.001 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while the second uses only &lt;code&gt;apply&lt;/code&gt; (but going over the indices, which are paired using &lt;code&gt;expand.grid&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time(&#xA;    D_2 &amp;lt;- &#xA;        matrix(apply(expand.grid(i = 1:nrow(myMatrix), j = 1:nrow(myMatrix)), 1, function(I) &#xA;            d_fun(myMatrix[I[[&quot;i&quot;]], ], myMatrix[I[[&quot;j&quot;]], ])&#xA;        )&#xA;    )&#xA;)&#xA;&#xA;   user  system elapsed &#xA; 39.313   0.498  39.561 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, as expected both are much slower than &lt;code&gt;dist&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;system.time(&#xA;    distancematrix &amp;lt;- as.matrix(&#xA;        dist(myMatrix, method = &quot;euclidean&quot;, diag = T, upper = T)&#xA;    )&#xA;)&#xA;&#xA;   user  system elapsed &#xA;  0.337   0.054   0.388&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12290" LastActivityDate="2015-10-16T13:16:05.703" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8464" PostTypeId="1" AcceptedAnswerId="9882" CreationDate="2015-10-16T13:35:02.133" Score="6" ViewCount="710" Body="&lt;p&gt;I am building a NLP application. My dataset has 0.6M datapoints each of 0.15M dimensions. My feature vector is highly sparse - mostly 0s, atmost 20 1s in any feature vector. I am using a 32 GB machine on AWS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Owing to size, Cant load entire dataset in one go in the memory. &lt;strong&gt;I am looking for a python based neural net library that supports&lt;/strong&gt; (if not all then atleast some) of the following [Owing to the dataset size and dimension, i defined below mentioned abilities]: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Ability to train the net on small batches &lt;/li&gt;&#xA;&lt;li&gt;library being able to exploit underlying multicore architecture (if available)&lt;/li&gt;&#xA;&lt;li&gt;A way to directly feed the feature vectors in compressed representation (like scipy.sparse) [in my case &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have had a good look at the following libraries - &lt;strong&gt;PyBrain, Pylearn2, NeuroLab, FANN&lt;/strong&gt;. To the best of my understanding, none of them has any of the desired abilities (Pybrain facilitates batch leaning. Even this takes close to 55 hrs).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions ? (cant reduce dimentionality)  Or in case &lt;em&gt;there is a shortcoming in my understanding and the any of the above libraries have (some) above capabilities&lt;/em&gt;, can you pls point out me to some sample code implementing/exploiting the same ?&lt;/p&gt;&#xA;" OwnerUserId="13457" LastActivityDate="2016-01-21T18:16:27.650" Title="Python + Neural Nets + Large dimension Large Dataset" Tags="|machine-learning|python|neural-network|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8467" PostTypeId="2" ParentId="8087" CreationDate="2015-10-16T17:10:38.963" Score="4" Body="&lt;p&gt;First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; Why do &lt;code&gt;perceptrons&lt;/code&gt; perform so much better than &lt;code&gt;generalized linear models&lt;/code&gt; for some problems? &lt;strong&gt;A:&lt;/strong&gt; Because they are inherently nonlinear models, with a great deal of flexibility.  The drawback is that the additional knobs require more data to correctly tune.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Big picture:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Less data can cause &lt;code&gt;high-bias&lt;/code&gt;.  &lt;code&gt;High-bias&lt;/code&gt; can be overcome by more data.  You have reduced your data from a 4800 feature dataset to a 38 feature dataset, so should expect to see increased bias.  &lt;code&gt;Neural networks&lt;/code&gt; require more data than models without hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Linearity vs. Nonlinearity&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your &lt;code&gt;artificial neural network&lt;/code&gt; (&lt;code&gt;perceptron&lt;/code&gt;) is an inherently nonlinear model, yet you are deciding to remove features from your dataset with a linear model (&lt;code&gt;PCA&lt;/code&gt;).  The presence of a single hidden layer explicitly creates second order terms in your data, and there are two additional nonlinear transformations (input==&gt;hidden and hidden==&gt;prediction), which add sigmoidal nonlinearity at each step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that the training data and the target data are both multiplied by the same matrix to achieve the reduced dimensionality really just means that your &lt;code&gt;perceptron&lt;/code&gt; needs to learn the matrix if it wants to reconstruct nonlinear aspects of the original data.  This requires more data or for you to reduce less of the dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An Experiment:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest trying an experiment where you perform second order &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;polynomial feature extraction&lt;/code&gt;&lt;/a&gt; on the full dataset and then performing &lt;code&gt;PCA&lt;/code&gt; on that enhanced dataset.  See how many features you end up with while retaining 99% of the variance of the enhanced dataset.  If it is larger than the dimensionality of your initial dataset then stick with the un-enhanced and un-reduced dataset.  If it is betweeen the dimensionality of the original data and 38 then try training the &lt;code&gt;perceptron&lt;/code&gt; with that data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A better idea:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than using the (linear) variance to determine the feature reduction of your PCA projection, try training and cross validating your model with different amounts of PCA dimensionality reduction.  You will likely find that there is a sweet spot for a given set of images.  For instance, an SVM on the MNIST digit data performs best when the 784 pixel features are reduced down to 50 linearly independent features using PCA.  Though this is not obvious from analyzing the variance of the principal components. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other options:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are nonlinear dimensionality reduction techniques, such as &lt;a href=&quot;http://wearables.cc.gatech.edu/paper_of_week/isomap.pdf&quot; rel=&quot;nofollow&quot;&gt;isomap&lt;/a&gt;.  You could investigate the use of nonlinear feature reduction since you are clearly loosing information with the linear PCA that you have applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also look into &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_extraction#Image_processing&quot; rel=&quot;nofollow&quot;&gt;image specific &lt;code&gt;feature extraction&lt;/code&gt; techniques&lt;/a&gt; to add some nonlinearity before reducing the dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-12-10T17:02:56.773" LastActivityDate="2015-12-10T17:02:56.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8468" PostTypeId="2" ParentId="8460" CreationDate="2015-10-16T17:33:33.103" Score="2" Body="&lt;p&gt;For an alternative approach, perhaps even to help foster understanding, you will probably find some utility in doing some analysis via R. Simple time series based tutorials abound for [wannabe] quants that should provide a bootstrap. &lt;a href=&quot;http://gekkoquant.com/2014/05/18/hidden-markov-models-model-description-part-1-of-4/&quot; rel=&quot;nofollow&quot;&gt;Part 1&lt;/a&gt;, &lt;a href=&quot;http://gekkoquant.com/2014/05/26/hidden-markov-models-forward-viterbi-algorithm-part-2-of-4/&quot; rel=&quot;nofollow&quot;&gt;Part 2&lt;/a&gt;, &lt;a href=&quot;http://gekkoquant.com/2014/09/07/hidden-markov-models-examples-in-r-part-3-of-4/&quot; rel=&quot;nofollow&quot;&gt;Part 3&lt;/a&gt;, &lt;a href=&quot;http://gekkoquant.com/2015/02/01/hidden-markov-models-trend-following-sharpe-ratio-3-1-part-4-of-4/&quot; rel=&quot;nofollow&quot;&gt;Part 4&lt;/a&gt;. These provide sources for data generation/intake as well as manipulation, allowing you to bypass much of the work to be able to see the actual HMM methods at work. There are direct analogues to the Python implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, for a more theoretical introduction, perhaps &lt;a href=&quot;http://www.robots.ox.ac.uk:5000/~vgg/rg/papers/hmm.pdf&quot; rel=&quot;nofollow&quot;&gt;Rabiner&lt;/a&gt; might provide some insights&lt;/p&gt;&#xA;" OwnerUserId="10663" LastActivityDate="2015-10-16T17:33:33.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8471" PostTypeId="2" ParentId="8451" CreationDate="2015-10-16T21:47:34.550" Score="2" Body="&lt;p&gt;Depending on the item domain You can expect different results. I would try to cluster data using cosinus similarity over item latent features. If number of items is large try DIMSUM (it's already implemented in Spark).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the visualization You can try one of the available implementation of t-SNE. Quick links: &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt; and &lt;a href=&quot;https://www.youtube.com/watch?v=RJVL80Gg3lA&quot; rel=&quot;nofollow&quot;&gt;YT - Google Talk about t-SNE Visualization&lt;/a&gt;. But here I cannot give You any more advice, because I have never tried to visualize it. I looking forward You will share Yours results.  &lt;/p&gt;&#xA;" OwnerUserId="9314" LastActivityDate="2015-10-16T21:47:34.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8472" PostTypeId="2" ParentId="8457" CreationDate="2015-10-16T22:29:51.053" Score="3" Body="&lt;p&gt;I've been looking for the same thing, and unfortunately it seems like there isn't one at this time. Some suggestions for how to proceed can be found in this &lt;a href=&quot;https://datascience.stackexchange.com/questions/8266/is-there-a-library-that-would-perform-segmented-linear-regression-in-python&quot;&gt;previous question&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could look into some R libraries eg segmented, SiZer, strucchange, and if something there works for you try embedding the R code in python with &lt;a href=&quot;http://rpy.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;rpy2&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Editing to add a link to &lt;a href=&quot;https://github.com/jcrudy/py-earth&quot; rel=&quot;nofollow noreferrer&quot;&gt;py-earth&lt;/a&gt;, &quot;A Python implementation of Jerome Friedman's Multivariate Adaptive Regression Splines&quot;. &lt;/p&gt;&#xA;" OwnerUserId="13469" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-16T23:10:23.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8475" PostTypeId="1" CreationDate="2015-10-17T13:05:41.787" Score="1" ViewCount="162" Body="&lt;p&gt;I'm an experienced developer but I'm only starting to discover data science.&#xA;I have a data set consisting of 62 parameters for each row and each row in that data set belongs to one of 7 groups (numbered from 0 to 6).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x0  x1  x2  x3  x4  x5  x6  x7  x8  x9  x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x20 x21 x22 x23 x24 x25 x26 x27 x28 x29 x30 x31 x32 x33 x34 x35 x36 x37 x38 x39 x40 x41 x42 x43 x44 x45 x46 x47 x48 x49 x50 x51 x52 x53 x54 x55 x56 x57 x58 x59 x60 x61 y&#xA;b4d8a653ea  16a14a2d17  06330986ed  ca63304de0  a62168d626  1746600cb0  1   1   -0.6887062641683063 7e5c97705a  e5df3eff9b  91bb549494  e33c63cf35  3694.0  6e40247e69  617a4ad3f9  718c61545b  c26d08129a  634e3cf3ac  dd9c9e0da2  17c99905b6  513a3e3f36  9aba4d7f51  40.57961189718329   -0.11269265451935975    -0.17219069579806134    1.1666666666666663  1.6745384722167482  0.6308894281294708  37.0    1.294921875 55.0    0.16666666666666666 10.0    0.0 0.0 1.0 9.0 0.0 1.0 23.0    Бер.67  0.12    1.935   02.Лют  0.625   0.25    0.125   0.0 0.813   0.07400000000000001 0.634   0.5479999999999999  0.2353332208066929  0.2649521447821752  0.0 0.3333333333333333  0.3333333333333333  0.3333333333333333  0.0 0.0 9.0 2&#xA;467f9617a3  16a14a2d17  06330986ed  ca63304de0  b7584c2d52  1746600cb0  1   1   0.8708708626728477  5624b8f759  fa0b797a92  669ea3d319  f178803074  18156.0 01ede04b4b  617a4ad3f9  718c61545b  d342e2765f  bb20e1ca06  8a6c8cef83  1b02793146  992153ed65  9aba4d7f51  28.76550293196428   2.6122849082704658  2.1590908057403015  4.0 1.7107137612171608  1.7135384162978815  0.16666666666666666 0.027669270833333325    109.0   0.0 31.0    0.0 0.0 1.0 244.0   1.0 1.0 68.0    17.25   0.57    3.452   4.0 0.409   0.619   0.579   0.248   0.34600000000000003 0.541   0.522   0.0 1.782346041542782   1.3224094711633876  0.011647254575707157    0.39767054908485855 0.2396006655574044  0.2495840266222961  0.06821963394342763 0.033277870216306155    601.0   4&#xA;190436e528  16a14a2d17  06330986ed  ca63304de0  b7584c2d52  1746600cb0  1   1   0.4376549941058605  5624b8f759  152af2cb2f  91bb549494  e33c63cf35  1178.0  cc69cbe29a  617a4ad3f9  e8a040423a  c82c3dbd33  ee3501282b  199ce7c484  5f17dedd5c  5c5025bd0a  9aba4d7f51  24.94393348850157   -0.8146595838365664 -0.7083080633874904 01.Тра  -0.5124221809900756 -0.7339666422629345 0.3333333333333333  14.837727864583336  11.0    0.0 24.0    0.0 0.0 1.0 29.0    0.0 3.0 11.0    Кві.42  0.15    0.161   0.2 1.0 1.0 1.0 1.0 1.0 0.52    0.5329999999999999  0.835   -0.5865396521883026 0.6724356815192951  0.0 0.6060606060606061  0.12121212121212124 0.21212121212121213 0.060606060606060615    0.0 33.0    3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know the relationship between parameters or what influence they have on the group number (y). I need to create and train a model that will predict a group based on 62 variables, with a high success rate (&gt; 80%). Where should I start?&lt;/p&gt;&#xA;" OwnerUserId="13480" LastEditorUserId="13413" LastEditDate="2015-10-17T18:29:47.603" LastActivityDate="2015-10-18T00:28:41.027" Title="Neural network, Support Vector Machine or something else to classify into 7 groups" Tags="|machine-learning|classification|neural-network|beginner|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8476" PostTypeId="2" ParentId="8475" CreationDate="2015-10-17T13:43:30.050" Score="1" Body="&lt;p&gt;Welcome to the site, @intellion! I can recommend a couple of things for you. First, if you think you'll continue your interest in data science and will be working on problems like this in the future, it is worthwhile to start familiarizing yourself with the field. There are a variety of introductory textbooks and courses you can choose from. If you have a strong mathematical background, I'd recommend Tibshirani's &quot;&lt;a href=&quot;https://books.google.com/books/about/The_Elements_of_Statistical_Learning.html?id=yPfZBwAAQBAJ&amp;amp;printsec=frontcover&amp;amp;source=kp_read_button&amp;amp;hl=en#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;&quot;, which is a solid classic in the field. If you're approaching this from a more application-oriented angle, I'd recommend something written specifically for the language you'd like to work in. If you plan on using R, there's &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1782162143&quot; rel=&quot;nofollow&quot;&gt;Machine Learning with R&lt;/a&gt;&quot;; if you plan on using python, there's &quot;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1118961749&quot; rel=&quot;nofollow&quot;&gt;Machine Learning in Python&lt;/a&gt;.&quot; There are of course a variety of free online resources as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To your question, you could certainly solve your problem with Support Vector Machines (SVMs) or Neural Networks, but I don't think that's the first place to start. To me, this looks like a classic multiple regression application situation (for an overview for doing this in R, see &lt;a href=&quot;http://www.statmethods.net/stats/regression.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; or &lt;a href=&quot;https://www.springer.com/us/book/9780387790534&quot; rel=&quot;nofollow&quot;&gt;this textbook&lt;/a&gt;). If you really want to try something in machine learning, perhaps decision trees are a good place to start, as they'll allow you to easily assess the influence of variables on the final model fit (e.g., using &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity&quot; rel=&quot;nofollow&quot;&gt;Gini Impurity&lt;/a&gt;). I've found the &lt;a href=&quot;https://cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;rpart&lt;/code&gt; library in R&lt;/a&gt; to be quite nice to work with, but it can be a little slow for larger data. You could also use &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow&quot;&gt;sklearn in python&lt;/a&gt;, if that's your preferred language.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-17T13:43:30.050" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8477" PostTypeId="2" ParentId="8475" CreationDate="2015-10-17T17:04:31.537" Score="1" Body="&lt;p&gt;It looks like your data includes strings and values. I suggest that you start with something simple as Logistic Regression. The strings can be hashed and the real values can be fed in as is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Split your dataset randomly in say 70% training and the remaining 30% for testing. Fit a model on the training data and evaluate the performance on the test data. This will give you a rough estimate of your classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot; rel=&quot;nofollow&quot;&gt;Vowpal Wabbit&lt;/a&gt; software package has everything you need to do the classification (and more). The format is simple, the learning is lightning fast - everything you need to experiment.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="836" LastEditDate="2015-10-18T00:28:41.027" LastActivityDate="2015-10-18T00:28:41.027" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="8478" PostTypeId="1" AcceptedAnswerId="8483" CreationDate="2015-10-17T18:37:01.587" Score="1" ViewCount="1282" Body="&lt;p&gt;R noob here..  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following data frame  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;data  &#xA; Value Multiplier&#xA;1  15    H&#xA;2   0    h&#xA;3   2    +&#xA;4   2    ?&#xA;5   2    k&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the multiplier is of class &lt;code&gt;factor&lt;/code&gt;. The values of &lt;code&gt;K&lt;/code&gt; &amp;amp; &lt;code&gt;k&lt;/code&gt; is &lt;code&gt;3&lt;/code&gt;, &lt;code&gt;+&lt;/code&gt; is &lt;code&gt;5&lt;/code&gt; and &lt;code&gt;?&lt;/code&gt; is &lt;code&gt;2&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data$Multiplier &amp;lt;-  revalue(data$Multiplier, c(&quot;+&quot;=&quot;5&quot;))&#xA;&amp;gt; data$Multiplier &amp;lt;-  revalue(data$Multiplier, c(&quot;?&quot;=&quot;2&quot;))&#xA;&amp;gt; data$Multiplier &amp;lt;-  revalue(data$Multiplier, c(&quot;K&quot;=&quot;3&quot;))&#xA;&amp;gt; data$Multiplier &amp;lt;-  revalue(data$Multiplier, c(&quot;k&quot;=&quot;3&quot;))  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a better way of doing it?&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2015-10-17T21:53:25.160" Title="R: Revalue multiple special characters in a data.frame" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8480" PostTypeId="1" AcceptedAnswerId="8487" CreationDate="2015-10-17T20:41:37.167" Score="0" ViewCount="213" Body="&lt;p&gt;I was trying to classify an outcome on some data using adaboost (the ada package in R) and I was playing around with the training data set of descriptors when I realized that removing a column in the descriptor matrix increased the accuracy of the output on the training data. Specifically, the number of false negatives dropped/true positives increased. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aside from removing a single column in the descriptors, I left everything else the same, including number of iterations.&lt;/p&gt;&#xA;" OwnerUserId="13485" LastActivityDate="2015-10-26T14:22:36.297" Title="Why would removing a variable in adaboost decrease error rate?" Tags="|r|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8481" PostTypeId="1" AcceptedAnswerId="8482" CreationDate="2015-10-17T21:10:40.957" Score="1" ViewCount="322" Body="&lt;p&gt;Question says most of it. I created a matrix of descriptors, set the vectors of responses, and input a set number of iterations. Each time I run the function with the same exact inputs, I get the same confusion table, but I get a different training error graph and different variable importance plots. &lt;/p&gt;&#xA;" OwnerUserId="13485" LastActivityDate="2015-10-17T21:24:01.290" Title="Why does ada (adaboost) in R return different training error graphs and variable importance plots when running the same function multiple times?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8482" PostTypeId="2" ParentId="8481" CreationDate="2015-10-17T21:24:01.290" Score="2" Body="&lt;p&gt;Boosting, together with bagging, falls into the realm of so cold ensemble models: you randomly draw a sample from the data, fit a model, adjust your predictions, sample once again. Unless your samples are fixed, every time you run the algo you'll get slightly different results.&lt;/p&gt;&#xA;" OwnerUserId="13406" LastActivityDate="2015-10-17T21:24:01.290" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8483" PostTypeId="2" ParentId="8478" CreationDate="2015-10-17T21:53:25.160" Score="2" Body="&lt;p&gt;That seems pretty straight forward to me. I'm pretty new too but in general I'm not sure if you can get better than one command. Though you could have combined all that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; newValueVector &amp;lt;- c(&quot;+&quot;=&quot;5&quot;, &quot;?&quot;=&quot;2&quot;, &quot;K&quot;=&quot;3&quot;, &quot;k&quot;=&quot;3&quot;)&#xA;&amp;gt; data$Multiplier &amp;lt;-  revalue(data$Multiplier, newValueVector)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13485" LastActivityDate="2015-10-17T21:53:25.160" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8484" PostTypeId="2" ParentId="8370" CreationDate="2015-10-18T02:19:57.837" Score="0" Body="&lt;p&gt;I post an answer but clearly I will award the bonus to someone else&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there is a maximum numerator if the document tf is normalized  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;d1⋅d2/(||d1||||d2||)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume d1 has same or less terms &#xA;(or just take the d with less terms)&lt;br&gt;&#xA;The maximum possible normalized tf is 1&lt;br&gt;&#xA;So the maximum possible numerator sum(tf1,i * idf,i * 1 * idf,i) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;||d2|| =  sum(tf1,i * idf,i * 1 * idf,i)  / ||d1|| / .6  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for a minimum I am working on that but clearly there is a minimum.&lt;br&gt;&#xA;If you are going to match you are going to have ||d||&lt;/p&gt;&#xA;" OwnerUserId="13285" LastActivityDate="2015-10-18T02:19:57.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8485" PostTypeId="2" ParentId="6227" CreationDate="2015-10-18T09:24:52.407" Score="1" Body="&lt;p&gt;There are several models for holding hierarchical data in a relational database. There's a good summary &lt;a href=&quot;http://www.slideshare.net/mobile/billkarwin/models-for-hierarchical-data&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. In summary they are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adjacency list: each row contains a foreign key to its parent. Fast for structure changes. Requires recursive processing to find the base ancestor.&lt;/li&gt;&#xA;&lt;li&gt;Path enumeration: each node stores its ancestry. An update requires all children to be updated. For a known node reaching the root requires no further work.&lt;/li&gt;&#xA;&lt;li&gt;Nested sets: additional columns hold the ancestry. The cost of structure changes depends on how big the moving subtree is. The root node is fast to find.&lt;/li&gt;&#xA;&lt;li&gt;Closure table: store all ancestor-descendant pairs. Structure changes can be expensive. Root node expensive to identify.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From the given information a path enumeration sounds like your best option.&lt;/p&gt;&#xA;" OwnerUserId="13498" LastActivityDate="2015-10-18T09:24:52.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8487" PostTypeId="2" ParentId="8480" CreationDate="2015-10-18T14:05:16.100" Score="3" Body="&lt;p&gt;Imagine that one of the column is just random data -- then it's not informative at all, so no classifier will be improved by including it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;code&gt;ada&lt;/code&gt;'s stochastic boosting implementations will always have some chance of including that variable in the classifier it generates. As a result, removing it has the potential to improve the classifiers generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In your case, you might check whether that variable is part of the final model generated.)&lt;/p&gt;&#xA;" OwnerUserId="9597" LastActivityDate="2015-10-18T14:05:16.100" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8488" PostTypeId="2" ParentId="749" CreationDate="2015-10-18T16:42:33.960" Score="9" Body="&lt;p&gt;Suppose you have &lt;code&gt;(MxN)&lt;/code&gt; sparse matrix, where &lt;code&gt;M&lt;/code&gt; -- stands for number of users who gave recommendations, and &lt;code&gt;N&lt;/code&gt; is the number of items recommended.   The $x_{ij}$ element of the matrix is the recommendation given, with some elements missing, i.e. to be predicted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then your matrix can be &quot;factorized&quot;, via introducing &lt;code&gt;K&lt;/code&gt; &quot;latent factors&quot;, so that instead of one matrix you have two: &lt;code&gt;(MxK)&lt;/code&gt; --for users, and &lt;code&gt;(KxN)&lt;/code&gt; -- for items, matrix multiplication of which produces the original matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, to your question: what are latent features in matrix factorization? They are unknown features (&lt;code&gt;K&lt;/code&gt;) in user tastes and recommended items, so that when these two matrices multiplied, they produce matrix of known recommendations. Particular weights (of user preferences towards a particular feature and amount of a feature in a particular item) are defined via so called Alternating Least Squares algo, more about which you can read &lt;a href=&quot;http://bugra.github.io/work/notes/2014-04-19/alternating-least-squares-method-for-collaborative-filtering/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13406" LastEditorUserId="13406" LastEditDate="2015-10-18T23:25:49.180" LastActivityDate="2015-10-18T23:25:49.180" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8492" PostTypeId="1" AcceptedAnswerId="8493" CreationDate="2015-10-19T02:35:49.983" Score="3" ViewCount="212" Body="&lt;p&gt;Given a two class multi dimensional classification problem, what reason would you give to choose Artificial Neural Network for carrying out the classification instead of Support Vector Machine or other classification methods?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2015-10-19T09:16:13.503" Title="Reason behind choosing Neural Network for classification" Tags="|classification|dataset|neural-network|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8493" PostTypeId="2" ParentId="8492" CreationDate="2015-10-19T03:52:45.837" Score="0" Body="&lt;p&gt;SVM is Parametric. Parametric models are something with fixed finite number of parameters independent of dataset size. Anything which is not parametric model is non-parametric model. ANN is non parametric. Also ANN has 'deep architectures&quot; which can represent &quot;intelligent&quot; behaviour/functions etc more efficiently than &quot;shallow architectures&quot; like SVMs. ANN may have any number of outputs, while support vector machines have only one. &lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-10-19T03:52:45.837" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8495" PostTypeId="5" CreationDate="2015-10-19T04:07:31.743" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-10-19T04:07:31.743" LastActivityDate="2015-10-19T04:07:31.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8496" PostTypeId="4" CreationDate="2015-10-19T04:07:31.743" Score="0" Body="Word2vec is a group of algorithms as well as an open-source tool that create word embeddings." OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-10-19T07:57:42.360" LastActivityDate="2015-10-19T07:57:42.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8497" PostTypeId="1" CreationDate="2015-10-19T05:08:06.053" Score="2" ViewCount="121" Body="&lt;p&gt;I am working in mining time series data for modeling user behavior. I collected the data from 37 smartphone users during 42 days(6 weeks). I want to use data during 3 weeks as model and the remaining as test (test performance of my model). The visualization of our data is following below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/afLS6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/afLS6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In here, I want to ask what is the good &lt;code&gt;window size&lt;/code&gt; for modeling user behavior, is one hour, one day, one week etc? or may be is there any good methods for determining window size?&lt;/p&gt;&#xA;" OwnerUserId="13512" LastActivityDate="2015-10-19T05:08:06.053" Title="How to determine window size for modeling user behavior" Tags="|data-mining|time-series|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8498" PostTypeId="1" CreationDate="2015-10-19T06:20:27.753" Score="1" ViewCount="807" Body="&lt;p&gt;Basically, I have data set contain inputs and outputs for unknown system, I want to use the LMS (Least Mean Square), to train a single perceptron (neuron) NN, and find the weights $ w_1, w_2, ... $, for the given output.&lt;/p&gt;&#xA;" OwnerUserId="13516" LastActivityDate="2015-10-19T06:20:27.753" Title="Use the LMS algorithm to train a single perceptron neural network by finding the weights for a given data" Tags="|machine-learning|neural-network|algorithms|optimization|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8499" PostTypeId="2" ParentId="8492" CreationDate="2015-10-19T09:16:13.503" Score="3" Body="&lt;p&gt;I wouldn't give any reason to make an choice &lt;em&gt;a priori&lt;/em&gt;, based on such a broad description.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead I would build models for NN and SVM (or any other that you were considering), train them on the data then compare using cross-validation and a suitable metric for your problem - such as accuracy, &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F1 score&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;AUROC&lt;/a&gt; or log loss. You may also want to note characteristics other than performance metric if you care about costs such as time to train, amount of computing resource required etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experience may tell you likely best choices from the problem domain. For instance, if you are classifying photographic or scanned images, then currently a convolutional neural network would be a very strong candidate, as these have repeatedly shown themselves as best performers with real-world image data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-10-19T09:16:13.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8500" PostTypeId="1" AcceptedAnswerId="8506" CreationDate="2015-10-19T10:14:50.240" Score="0" ViewCount="124" Body="&lt;p&gt;Hi guys I am working with a regular network which has the shape of a square grid and contains &lt;code&gt;100x100=10000&lt;/code&gt; nodes. The edges (links) between these nodes simply follow the shape of a chess table: each node which is not placed in the corner or along the boundary has 4 connections only, all of them involving its nearest neighbors. Accordingly, the nodes on the boundary have 3 connections only, while the nodes in the four corners have 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this brings me to my problem. If you plot such data you will figure out you have 4 nodes with links=2, 98*4 nodes with links=3, and 10000-(98*4)-4 with links=4. This translates into three tuples to plot: &lt;code&gt;(x,y)=(2,4),(3,392),(4,9604)&lt;/code&gt;. The resulting histogram is heavily skewed toward the right hand side:&#xA; &lt;a href=&quot;https://i.stack.imgur.com/nYDjE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nYDjE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;strong&gt;what kind of distribution do you think would fit this dataset?&lt;/strong&gt; I was thinking of a Poisson distribution (the x-axis values are discrete and not continuous) skewed to the right. &lt;strong&gt;I will appreciate any kind of help/guidance.&lt;/strong&gt; Thank you!&lt;/p&gt;&#xA;" OwnerUserId="13481" LastActivityDate="2015-10-19T13:01:29.480" Title="On fitting a Poisson distribution to make sense of data" Tags="|dataset|neural-network|sampling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8501" PostTypeId="1" CreationDate="2015-10-19T11:10:08.813" Score="1" ViewCount="135" Body="&lt;p&gt;For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500  of them are same posts.  How should I decide the category label now?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AnnotatorId  Post  Labels&#xA;-------------------------&#xA;Annotator1: post1: A,B,C&#xA;Annotator2: post1: A,D,E&#xA;Annotator3: post1: B,D,E&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Will the label for post1 be &lt;code&gt;A,B,D&lt;/code&gt; through majority voting? Or, is there some better, commonly used approaches?&lt;/p&gt;&#xA;" OwnerUserId="6514" LastActivityDate="2015-10-19T15:22:32.500" Title="Choosing class labels from annotated data" Tags="|machine-learning|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8502" PostTypeId="1" AcceptedAnswerId="8510" CreationDate="2015-10-19T11:13:06.367" Score="2" ViewCount="1591" Body="&lt;p&gt;It's possible to use Machine Learning techniques to cluster songs into musical-scale groups? I mean: &quot;this song was written in C&quot;... or &quot;this song was written in Am&quot; etc. I made a fast search about the subject and I found no software that can do this. If you know some software, or research (academic papers), related to that subject, could you link it here for me? I'm very interested in that subject but I'm not sure from where I can begin. I have a little experience with Random Forests and Neural Networks, maybe I can accomplish the classification task with one of those algorithms, but, again, I'm not sure which kinda of features I should pass to the algorithm. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="13521" LastEditorUserId="924" LastEditDate="2015-10-19T22:16:34.950" LastActivityDate="2015-10-20T10:56:55.360" Title="Using Machine Learning to Predict Musical Scales" Tags="|machine-learning|classification|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8504" PostTypeId="1" CreationDate="2015-10-19T11:31:55.477" Score="2" ViewCount="779" Body="&lt;p&gt;Let us say, I have 4 classification models on a training data set of various &lt;code&gt;examples&lt;/code&gt;. Now, I want to choose which 1 out 4 models (or what combination of the 4 models) should I choose to get the final probability for a particular &lt;code&gt;test example&lt;/code&gt;. So, basically I want to create a &lt;code&gt;meta model&lt;/code&gt; to get ultimate probabilities for all the test examples. I know I can combine all 4 models using regression but I am particularly interested in using a decision tree to combine all 4 models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the &lt;code&gt;meta decision tree model&lt;/code&gt; will have 4 features/variables (4 model probabilities) as per my current thoughts. Is this a right approach? Should I include more features in it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please explain how do I combine 4 models using a &lt;code&gt;meta decision tree model&lt;/code&gt; to choose the best &lt;code&gt;model out of 4&lt;/code&gt; for a particular &lt;code&gt;test example&lt;/code&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2015-10-20T12:23:46.497" Title="Pick a model from multiple models using a decision tree" Tags="|decision-trees|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8506" PostTypeId="2" ParentId="8500" CreationDate="2015-10-19T12:19:46.257" Score="1" Body="&lt;p&gt;Unlike other graphs, the degree distribution is a function of N. Specifically, the number of nodes with k=2 is constant (4), the  number of nodes with k=3 grows linearly with the length of one side $4l-4$, while the number of nodes with k=4 grows exponentially with the length of a side, $l^2-4l-4$. In the large $l$ (or $N$) limit, the degree distribution is 4. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get the exact distribution by normalizing this piecewise defined function (divide by $l^2=N$). &lt;/p&gt;&#xA;" OwnerUserId="7738" LastEditorUserId="7738" LastEditDate="2015-10-19T13:01:29.480" LastActivityDate="2015-10-19T13:01:29.480" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="8508" PostTypeId="2" ParentId="8504" CreationDate="2015-10-19T14:05:49.553" Score="2" Body="&lt;p&gt;Well, you can use this approach for stacking (AFAR this is how the process you are trying to perform is called):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Prepare a dataset in the format:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your original features which you used to fit the 4 estimators (remove the target variable from it)&lt;/li&gt;&#xA;&lt;li&gt;Add new target variable in it - the categorical output which shows which model should be picked up for that specific input. To obtain it - train your 4 estimators on your input and than label each input with the number of the estimator which has the closest output to the target&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;2) Train your stacker - on that dataset from #1 - for example the decision tree. Or any other classifier which will be able to accurately predict the number of the estimator which should be used for that specific input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) When using in combat mode - first run the stacker trained in #2 on new input and obtain an id of an estimator which best suites this example. Than run that estimator against new input record and get your response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that makes sense.&#xA;I tried it and it worked nicely for one of my problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you could try to employ some kind a 'blending' of these 4 predictors. Here is an approach you could take (my usage of terminology here becomes a bit frivolous since I am an engineer and not a scientist, so if someone provides a more elaborate answer on blending - please do).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, recall how ANNs for classification work - and would work for you example, if you will use ANN as a stacker. You have input layer which reads the input values, some hidden layers and an output layer with a set of neurons, each neuron is associated with a specific class. For our case it would be 4 neurons. After you feed input each of 4 output neurons will output some real value, for example:&#xA;Neuron 1 - 5.4&#xA;Neuron 2 - 0.3&#xA;Neuron 3 - 15.3&#xA;Neuron 4 - 3.3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That output means that ANN thinks that the input belongs to 'class 3' and that you should use 3rd predictor for that example to get best results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But! Let's think about probabilities. What is the probability of input to belong to ANY of 4 classes? It is 1.0 (since you have only 4 classes). And by default ANN doesn't have the knowledge that this the case. So for us it would be good if the ANN will output &lt;em&gt;probability of data belong to X class&lt;/em&gt; instead of just some real value. For example we would like to obtain such output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neuron 1 - 0.6&#xA;Neuron 2 - 0.2&#xA;Neuron 3 - 0.15&#xA;Neuron 4 - 0.05&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which will read as 'I am 60% sure that you should use estimator 1 for that input, 20% sure  - that estimator 2' etc. Such objective is called 'cross-entropy'. Modern libraries for building ANNs support it and you could tell network to train to output these probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By now I believe you already got what I suggest to do. Prepare the dataset as I described at the beginning of the answer, than train an ANN with cross-entropy objective to output the probabilities. When you run it in combat mode:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) For new input run it through ANN and obtain probabilities, like these:&#xA;Neuron 1 - 0.6&#xA;Neuron 2 - 0.2&#xA;Neuron 3 - 0.15&#xA;Neuron 4 - 0.05&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Run each predictor against input and obtain their predictions&#xA;3) Multiply each prediction by the associated probability and sum the results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;predictor1_result*0.6 + predictor2_result*0.2 + predictor3_result*0.15 + predictor4_result*0.05&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And get you final result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do that? First of all, we can't be sure that each time we realy pick up the best estimator, since our stacker (ANN in our case) could make mistakes. By using such approach we give the stacker a chance to compensate for possible mistakes - which minimizes the average prediction error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, consider that you get such output from ANN:&#xA;Neuron 1 - 0.45&#xA;Neuron 2 - 0.45&#xA;Neuron 3 - 0.05&#xA;Neuron 4 - 0.05&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such case the best option for us is to almost average the predictions from predictor 1 and 2 - since ANN is not really sure which of predictors to use.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastEditorUserId="7969" LastEditDate="2015-10-20T12:23:46.497" LastActivityDate="2015-10-20T12:23:46.497" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="8509" PostTypeId="2" ParentId="8501" CreationDate="2015-10-19T15:22:32.500" Score="5" Body="&lt;p&gt;That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall &lt;a href=&quot;https://en.wikipedia.org/wiki/Inter-rater_reliability&quot;&gt;inter-rater agreement&lt;/a&gt;. For example, &lt;a href=&quot;https://en.wikipedia.org/wiki/Cohen%27s_kappa&quot;&gt;Cohen's kappa&lt;/a&gt; is a commonly-used metric:&#xA;\begin{eqnarray}&#xA;\kappa &amp;amp;=&amp;amp; \frac{p_{o}-p_{e}}{1-p_{e}}\\&#xA;       &amp;amp;=&amp;amp; 1 - \frac{1-p_{o}}{1-p_{e}},&#xA;\end{eqnarray}&#xA;where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-19T15:22:32.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8510" PostTypeId="2" ParentId="8502" CreationDate="2015-10-19T16:35:09.303" Score="4" Body="&lt;p&gt;From a very high level -- You can convert the song to a spectrogram, there are a large number of implementations to do this. From there you can analyze the sound waves. In the case of the key, for instance, the note A is equal to 440 hz. Look into FFT as well.  Hope this helps get you started. I know spotify trains neural networks on spectrograms of songs to find similar songs based on &quot;sound&quot;.&lt;/p&gt;&#xA;" OwnerUserId="13529" LastActivityDate="2015-10-19T16:35:09.303" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8511" PostTypeId="1" CreationDate="2015-10-19T16:38:00.863" Score="5" ViewCount="261" Body="&lt;p&gt;The NBA has a system called Sports VU that tracks x-y coordinates of every player and the ball every 1/10th of a second for every game of the 2013-2014 and 2014-2015 seasons. With some fancy web scraping I now have access to this data and -- because I'm such an avid fan of the NBA -- I would like to identify each team's most common plays. Assume I don't have any knowledge about each team's plays beforehand (so I don't think supervised learning would work here). What would be the best unsupervised learning techniques to use? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I could trace each player's path over the course of the play, I imagine the problem would be similar to what you would see with image recognition/classification. Anyways, should I use PCA, some kind of neural  network? I understand this is a very broad question -- I don't need to know how to code it (I'm a proficient coder and machine learning practitioner); I'm just looking for high level unsupervised machine learning details. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2015-10-20T20:55:19.137" Title="Unsupervised learning to identify most common basketball plays from spatial data" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8512" PostTypeId="2" ParentId="8502" CreationDate="2015-10-19T17:02:02.380" Score="0" Body="&lt;p&gt;I kept searching and I recently found &lt;a href=&quot;http://jim.afim-asso.org/jim12/pdf/jim2012_08_p_osmalskyj.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://www.academia.edu/351159/Musical_Instruments_in_Random_Forest&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; articles that seems to be very promissory. I think that if I can identify a good number of notes from a music, I will be able to identify the music scale itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you guys have more material to suggest on this subject I'll be very grateful. &lt;/p&gt;&#xA;" OwnerUserId="13521" LastEditorUserId="13521" LastEditDate="2015-10-19T19:02:40.163" LastActivityDate="2015-10-19T19:02:40.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8513" PostTypeId="1" CreationDate="2015-10-19T18:05:57.420" Score="1" ViewCount="1078" Body="&lt;p&gt;I have a work order system. I found out that the work order completion times are exponentially distributed. Every ticket has some features. I understand that regression is &quot;a line that best fits the points&quot;. In my case, I want the line to be exponential curve. Is it possible ? Or is CDF for exponential distribution better option ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My use case is I want to predict the probability of a work order completing by deadline. I was thinking of querying sets of data through &quot;group by&quot; &amp;amp; then compute and store the lambda parameters. When a new ticket comes in, I will pull the corresponding lambda &amp;amp; use CDF to determine %age. I am wondering if an ML algorithm can simplify this process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have .net stack. I use Accord.net. I'm open to other .net libs.&lt;/p&gt;&#xA;" OwnerUserId="13056" LastEditorUserId="13413" LastEditDate="2015-10-19T18:55:18.397" LastActivityDate="2015-10-20T03:47:05.373" Title="regression to model exponential distribution" Tags="|regression|svm|.net|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8515" PostTypeId="2" ParentId="130" CreationDate="2015-10-19T22:42:14.050" Score="2" Body="&lt;p&gt;For a proper review and definition you may take a look at &lt;a href=&quot;http://link.springer.com/chapter/10.1007/11558958_113#page-1&quot; rel=&quot;nofollow&quot;&gt;Dimension Reduction vs. Variable Selection&lt;/a&gt; also in the book &lt;a href=&quot;https://books.google.ca/books/about/Feature_Extraction.html?id=x5hdbK8bIG0C&amp;amp;hl=en&quot; rel=&quot;nofollow&quot;&gt;Feature Extraction Foundations and Applications&lt;/a&gt;&#xA;feature extraction is decomposed in to two steps: feature construction and feature selection.&lt;/p&gt;&#xA;" OwnerUserId="13443" LastActivityDate="2015-10-19T22:42:14.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8517" PostTypeId="2" ParentId="8513" CreationDate="2015-10-20T03:47:05.373" Score="3" Body="&lt;p&gt;There was an MIT group that got a lot of attention a few years back for predicting twitter trends and doing so with really good &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow noreferrer&quot;&gt;ROC-AUC Scores&lt;/a&gt;.  The problem you are trying to solve is kind of reminiscent of the twitter trending problem.  The crux of the solution is that they avoided the hassle of parametric modeling through a grouping of similar time series followed by a classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a pretty good writeup of the method for the general public in &lt;a href=&quot;https://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stan Nikolov's Blog post&lt;/a&gt;.  Notice that there is a &lt;a href=&quot;https://www.youtube.com/watch?v=34Q1xP_5QzE&quot; rel=&quot;nofollow noreferrer&quot;&gt;video of a presentation by his thesis advisor also&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea of the method is to find the most common shapes in the time evolution hold those as the dominant classes.  These would be various exponential shapes in your case (from Nikolov's &lt;a href=&quot;https://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt;):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Z9obf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Z9obf.png&quot; alt=&quot;Grouping of time series into classes&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can take your training data and choose the best shape for each of your cases.  You can then use this data to train a classification algorithm.  Finally you can use this to predict the outcome of your new data (from Nikolov's &lt;a href=&quot;https://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt;):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/NdJh3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NdJh3.png&quot; alt=&quot;time series prediction through classification&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://dspace.mit.edu/handle/1721.1/85399&quot; rel=&quot;nofollow noreferrer&quot;&gt;full dissertation is also available online for download&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's actually an &lt;a href=&quot;http://blabpredicts.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;entire company&lt;/a&gt; that is using this methodology and trying to sell it to companies.  They have some pretty cool promotional &lt;a href=&quot;https://www.youtube.com/results?search_query=blab+predicts&quot; rel=&quot;nofollow noreferrer&quot;&gt;videos on youtube&lt;/a&gt; also, which won't teach you the methodology, but show how powerful it can be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-10-20T03:47:05.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8519" PostTypeId="1" AcceptedAnswerId="8529" CreationDate="2015-10-20T09:24:56.487" Score="1" ViewCount="3839" Body="&lt;p&gt;I have a string containing many words [not sentences], I want to know how I can extract all the words that correspond to a location in that string for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text&amp;lt;-c(&quot;China&quot;,&quot;Japan&quot;,&quot;perspective&quot;,&quot;United Kingdom&quot;,&quot;formatting&quot;,&quot;clear&quot;,&quot;India&quot;,&quot;Sudan&quot;,&quot;United States of America&quot;,&quot;Bagel&quot;,&quot;Mongolian&quot;,...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output should be:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; China, Japan, United Kingdom, Mongolian&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;something of the type. Basically I am looking at extracting locative information from random text.&#xA;This is a very general problem I am looking for guidance on how to model my solution, is there any dataset or something I can use to compare or extract information from. I don't want to carry out word by word comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked up OpenNLP but I am not sure how to use it's location-models for carrying out Named Entity Recognition in R. In the above example there are only countries but I would like to identify other places, such as provinces, states, counties, cities, etc. as well. &#xA;I am new to machine learning and R-programming, any guidance is greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="13508" LastEditorUserId="13508" LastEditDate="2015-10-20T17:32:28.920" LastActivityDate="2015-10-20T17:39:03.500" Title="Extracting list of locations from text using R" Tags="|machine-learning|r|information-retrieval|named-entity-recognition|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8520" PostTypeId="2" ParentId="8502" CreationDate="2015-10-20T10:56:55.360" Score="2" Body="&lt;p&gt;You can take a look at Music Information Retrieval Evaluation eXchange (MIREX). An annual competition with number of different tasks from MIR. The most relevant to your question being Audio Key Detection. Explanation papers for the used approaches can be found &lt;a href=&quot;http://nema.lis.illinois.edu/nema_out/mirex2013/results/akd/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of library for extracting knowledge from audio signal I'd recommend &lt;a href=&quot;http://essentia.upf.edu/documentation/&quot; rel=&quot;nofollow&quot;&gt;Essentia&lt;/a&gt;. There are numerous features you can compute over time window in a track including tonal descriptors (key and scale).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've built discriminative representation of a track for your task you can use any supervised classification model fed with labeled data. There are several musical datasets on the web labeled with track information. Take a look at GTZAN, each track has genre, tempo and key as meta. &lt;/p&gt;&#xA;" OwnerUserId="13563" LastActivityDate="2015-10-20T10:56:55.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8521" PostTypeId="1" AcceptedAnswerId="8527" CreationDate="2015-10-20T11:09:47.047" Score="1" ViewCount="2620" Body="&lt;p&gt;Just out of curiousity, is it generally a good idea to reduce the dimension of training set before using it to train SVM classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a collection of documents, each of them is represented by a vector with tf-idf weight calculated by scikit-learn's tfidf_transformer. The number of terms (feature?) is close to 60k, and with my training set that consist of about 2.5mil of documents, it makes the training process to go on forever.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides taking forever to train, the classification also wasn't accurate most probably due to the wrong model. Just to get an idea of what I am dealing with, I tried finding a way to visualize the data somehow. And I decomposed the document matrix into a (m, 2) matrix using SVD with scikit-learn (wanted to try other methods, but they all crashed halfway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is what the visualization looks like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/os28L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/os28L.png&quot; alt=&quot;Result of SVD&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is it generally a good practice to reduce the dimension, and then only proceed with SVM? Also in this case what can I do to improve the accuracy of the classifier? I am trying to use sklearn.svm.SVC and &lt;code&gt;kernel='poly'&lt;/code&gt;, and &lt;code&gt;degree=3&lt;/code&gt; and it is taking a very long time to complete.&lt;/p&gt;&#xA;" OwnerUserId="13562" LastActivityDate="2016-05-16T00:17:26.587" Title="Reduce dimension, then apply SVM" Tags="|python|svm|scikit-learn|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8522" PostTypeId="2" ParentId="8519" CreationDate="2015-10-20T12:04:44.297" Score="0" Body="&lt;p&gt;I use the function &lt;code&gt;grepl()&lt;/code&gt; for this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text&amp;lt;-c(&quot;China&quot;,&quot;Japan&quot;,&quot;perspective&quot;,&quot;United Kingdom&quot;,&quot;formatting&quot;,&quot;clear&quot;,&#xA;&quot;India&quot;,&quot;Sudan&quot;,&quot;United States of America&quot;,&quot;Bagel&quot;,&quot;Mongolian&quot;)&#xA;&#xA;#Make a logical vector&#xA;arg&amp;lt;-grepl(&quot;China&quot;,text)|grepl(&quot;Japan&quot;,text)|grepl(&quot;UnitedKingdom&quot;,text)|&#xA;grepl(&quot;Mongolian&quot;,text)&#xA;&#xA;#Pick out from text&#xA;text[arg]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13567" LastEditorUserId="11097" LastEditDate="2015-10-20T16:30:28.853" LastActivityDate="2015-10-20T16:30:28.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8523" PostTypeId="2" ParentId="8521" CreationDate="2015-10-20T13:04:23.000" Score="1" Body="&lt;p&gt;While performing PCA on your tfidf vectors or &lt;a href=&quot;http://www.nltk.org/book/ch03.html&quot; rel=&quot;nofollow&quot;&gt;stemming&lt;/a&gt; or eliminating infrequent words might reduce the dimensionality significantly, you might want to try &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic modeling&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While your initial problem of creating topics and assigning topics to documents will remain in high-dimenional space, the supervised portion will be in topic space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of fast implementations of &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; and similar topic models out there.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-20T13:04:23.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8524" PostTypeId="2" ParentId="8511" CreationDate="2015-10-20T13:59:07.417" Score="0" Body="&lt;p&gt;I agree that image classification is the right place to look for inspiration. But instead of viewing the 'image' as a grid of court positions, with 'color' being players, I would first see if you get anything useful out of seeing the 'image' as a grid of players, with 'color' being the x and y position (and probably also velocity) of the players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that you need to do some sort of 'clumping' of data: instead of trying to look at a full game, you want to look at segments that are closer to the length of a play. You can do this with a moving window (the equivalent of an &lt;a href=&quot;http://www.cs.toronto.edu/~kyros/courses/320/Lectures.2013s/lecture.2013s.03.pdf&quot; rel=&quot;nofollow&quot;&gt;image patch&lt;/a&gt;) or by joining together sufficiently similar player-seconds (the equivalent of &lt;a href=&quot;http://ivrl.epfl.ch/research/superpixels&quot; rel=&quot;nofollow&quot;&gt;superpixels&lt;/a&gt;) to create a graph that describes the game. My guess is that the moving window approach is the best place to start, as it may inform what sort of features are relevant for joining together pixels to make superpixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's likely the case that something like plays will fall out of doing k-means on moving windows of the game, but you'd probably benefit a lot from ensuring that the sort of structure that seems important (say, the distance to the closest enemy player) is already available as a feature to include in hypotheses.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-20T13:59:07.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8525" PostTypeId="2" ParentId="8519" CreationDate="2015-10-20T14:26:45.567" Score="0" Body="&lt;p&gt;I am not entirely clear on the evaluation criteria that are using but, given your example, you can use %in% to return matching elements in a vector. The %in% statement returns the same Boolean output that Daniel's grep example does. If you want the actually position of the matching elements you can use %in% and which.    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text&amp;lt;-c(&quot;China&quot;,&quot;Japan&quot;,&quot;perspective&quot;,&quot;United Kingdom&quot;,&quot;formatting&quot;,&quot;clear&quot;, &quot;India&quot;,&quot;Sudan&quot;,&quot;United States of America&quot;,&quot;Bagel&quot;,&quot;Mongolian&quot;)&#xA;eval &amp;lt;- c(&quot;China&quot;, &quot;Japan&quot;, &quot;United Kingdom&quot;, &quot;Mongolian&quot;)&#xA;&#xA;# Return matching elements&#xA;text[text %in% eval] &#xA;&#xA;# Return the index (position) of matching elements&#xA;which(text %in% eval) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8463" LastActivityDate="2015-10-20T14:26:45.567" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8526" PostTypeId="2" ParentId="8460" CreationDate="2015-10-20T15:14:12.613" Score="10" Body="&lt;p&gt;For another alternative approach, you can take a look at the PyMC library.&#xA;There is a good &lt;a href=&quot;https://gist.github.com/fonnesbeck/342989&quot; rel=&quot;nofollow noreferrer&quot;&gt;gist&lt;/a&gt; created by Fonnesbeck which walks you through the HMM creation.&lt;/p&gt;&#xA;&lt;p&gt;And if you become really eager about the PyMC, there is an awesome open-source &lt;a href=&quot;https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers&quot; rel=&quot;nofollow noreferrer&quot;&gt;book&lt;/a&gt; about Bayesian Modeling. It does not explicitly describe Hidden Markov Processes, but it gives a very good tutorial on the library itself with plenty of examples.&lt;/p&gt;&#xA;" OwnerUserId="4737" LastEditorUserId="98307" LastEditDate="2020-08-03T09:23:35.423" LastActivityDate="2020-08-03T09:23:35.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8527" PostTypeId="2" ParentId="8521" CreationDate="2015-10-20T16:26:08.090" Score="3" Body="&lt;p&gt;I'd recommend spending more time thinking about feature selection and representation for your SVM than worrying about the number of dimensions in your model. Generally speaking, SVM tends to be very robust to uninformative features (e.g., see &lt;a href=&quot;https://eldorado.tu-dortmund.de/bitstream/2003/2595/1/report23_ps.pdf&quot; rel=&quot;nofollow&quot;&gt;Joachims, 1997&lt;/a&gt;, or &lt;a href=&quot;https://eldorado.tu-dortmund.de/bitstream/2003/2596/1/report24.pdf&quot; rel=&quot;nofollow&quot;&gt;Joachims, 1999&lt;/a&gt; for a nice overview). In my experience, SVM doesn't often benefit as much from spending time on feature selection as do other algorithms, such as Naïve Bayes. The best gains I've seen with SVM tend to come from trying to encode your own expert knowledge about the classification domain in a way that is computationally accessible. Say for example that you're classifying publications on whether they contain information on protein-protein interactions. Something that is lost in the bag of words and tfidf vectorization approaches is the concept of proximity—two protein-related words occurring close to each other in a document are more likely to be found in documents dealing with protein-protein interaction. This can sometimes be achieved using $n$-gram modeling, but there are better alternatives that you'll only be able to use if you think about the characteristics of the types of documents you're trying to identify. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still want to try doing feature selection, &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/feature-selectionchi2-feature-selection-1.html&quot; rel=&quot;nofollow&quot;&gt;I'd recommend $\chi^{2}$&lt;/a&gt; (chi-squared) feature selection. To do this, you rank your features with respect to the objective&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\chi^{2}(\textbf{D},t,c) = \sum_{e_{t}\in{0,1}}\sum_{e_{c}\in{0,1}}\frac{(N_{e_{t}e_{c}}-E_{e_{t}e_{c}})^{2}}{E_{e_{t}}e_{c}},&#xA;\end{equation}&#xA;where $N$ is the observed frequency of a term in $\textbf{D}$, $E$ is its expected frequency, and $t$ and $c$ denote term and class, respectively. You can &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html&quot; rel=&quot;nofollow&quot;&gt;easily compute this in sklearn&lt;/a&gt;, unless you want the educational experience of coding it yourself $\ddot\smile$&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-20T16:26:08.090" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8528" PostTypeId="1" CreationDate="2015-10-20T17:11:20.453" Score="3" ViewCount="2056" Body="&lt;h2&gt;Question:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;What is the Time Decay formula that web analytics packages use to distribute credit across the multiple sessions associated with a user before a conversion?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Context:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;All web analytics packages like Google Analytics rely on the concept of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;USERS&lt;/strong&gt; &lt;em&gt;visitors, ie unique cookied browsers.&lt;/em&gt; Users have many...&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;SESSIONS&lt;/strong&gt; &lt;em&gt;visits, ie sets of pageviews within close time proximity.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;CONVERSIONS&lt;/strong&gt;  &lt;em&gt;a success event such as a sign-up which can occur during a session.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These analytics packages allow you to distribute the credit of a single conversion across all the preceding sessions because each played a part in leading the user to conversion. One such model is Time Decay, which gives a diminishing share of credit to sessions further from the conversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time Decay Visual: (1st session) ▁ ▂ ▃  ▅ ▆ (conversion)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://support.google.com/analytics/answer/1662518?hl=en&quot; rel=&quot;nofollow&quot;&gt;more info on attribution models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Sample Data:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am assuming the data required would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;n&lt;/code&gt; - the total number of sessions a user has before a conversion&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;t&lt;/code&gt; - the total duration of time from the user's first session to conversion&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;tbc&lt;/code&gt; - the individual time before conversion for each session &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So for three different sample users, data may look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;UserID  n   t   Session#    Timestamp   TBC Conversion in session?&#xA;User1   4   12  1           2015-10-01  12  No&#xA;User1   4   12  2           2015-10-06  7   No&#xA;User1   4   12  3           2015-10-11  2   No&#xA;User1   4   12  4           2015-10-13  0   YES&#xA;User2   3   4   1           2015-10-09  4   No&#xA;User2   3   4   2           2015-10-09  4   No&#xA;User2   3   4   3           2015-10-13  0   YES&#xA;User3   1   0   1           2015-10-14  0   YES&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="13574" LastEditorUserId="13574" LastEditDate="2015-10-20T21:56:40.380" LastActivityDate="2016-08-05T03:40:53.860" Title="Time Decay Formula for Conversion Attribution" Tags="|consumerweb|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8529" PostTypeId="2" ParentId="8519" CreationDate="2015-10-20T17:28:51.060" Score="0" Body="&lt;p&gt;This might be better for &lt;a href=&quot;http://opendata.stackexchange.com&quot;&gt;opendata&lt;/a&gt;, but nonetheless, you have a few options. One would be to go to &lt;a href=&quot;http://www.geohive.com/cntry/&quot; rel=&quot;nofollow&quot;&gt;geohive&lt;/a&gt; which has other pages, including &lt;a href=&quot;http://www.geohive.com/earth/population_now.aspx&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;. There is also the UN categorization, available on &lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_sovereign_states&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt; which uses membership within the United Nations system divides the 206 listed states into three categories: 193 member states,[1] two observer states, and 11 other states. The sovereignty dispute column indicates states whose sovereignty is undisputed (190 states) and states whose sovereignty is disputed (16 states).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can &lt;code&gt;read.table&lt;/code&gt; or &lt;code&gt;rvest&lt;/code&gt; those sources and grab them at runtime.&lt;/p&gt;&#xA;" OwnerUserId="10663" LastActivityDate="2015-10-20T17:28:51.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8530" PostTypeId="2" ParentId="8519" CreationDate="2015-10-20T17:39:03.500" Score="0" Body="&lt;p&gt;This is an interesting problem! You're right in thinking the easiest approach to this is to use a look-up table for identifying locations. Of course, this approach is going to be only as good as your data set and is still going to be somewhat prone to misclassification. One resource I've found is the &lt;a href=&quot;https://www.maxmind.com/en/free-world-cities-database&quot; rel=&quot;nofollow&quot;&gt;Free World Cities Database&lt;/a&gt;. One caveat—this isn't still actively maintained, and the countries are listed as country code, which may require further resolution on your part. Another possibility is the &lt;a href=&quot;http://www.geonames.org/export/&quot; rel=&quot;nofollow&quot;&gt;geonames data set&lt;/a&gt;. There, it looks like you'd want their &lt;code&gt;allCountries.zip&lt;/code&gt; dataset.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-20T17:39:03.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8531" PostTypeId="2" ParentId="8511" CreationDate="2015-10-20T20:55:19.137" Score="1" Body="&lt;p&gt;This is a really interesting problem! Like most really interesting problems, you're unlikely to find an out-of-box solution for this, but I think the field of graph/subgraph similarity has some promise here. I'll go into more detail, but, at a high level, I think you can view players' paths during a play as a collection of five traversals through x,y space, with a vertex being any x,y point you have available (presumably there's some level of time granularity here), and an edge describing a player's movement from one point in x,y space to the next over time. It should be possible to cluster your data using a similarity metric (e.g., see &lt;a href=&quot;https://www.cs.cmu.edu/~jingx/docs/DBreport.pdf&quot; rel=&quot;nofollow&quot;&gt;Koutra et al., 2011&lt;/a&gt; for a nice overview). Then, using your own domain expertise, you should be able to identify whether the clusters you've derived have some real-world meaning in basketball. &lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-20T20:55:19.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8532" PostTypeId="2" ParentId="8427" CreationDate="2015-10-20T21:34:08.533" Score="1" Body="&lt;p&gt;I think the tagging approach has some merit here. The frequency drop you're observing as a consequence of using this is to be expected, I think. After all, keywords are the words that help differentiate a document from others in a corpus. If you have access to an ontology related to the subject matter of your corpus, you could try to map your rare keyword tags to the ontology, and use parent-level information for each to get a slightly more general set of keyword tags! If this is an approach that interests you, Stanford's open source &lt;a href=&quot;http://protege.stanford.edu/&quot; rel=&quot;nofollow&quot;&gt;protégé&lt;/a&gt; system is a good framework for working on this.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-20T21:34:08.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8534" PostTypeId="2" ParentId="6209" CreationDate="2015-10-20T23:27:03.737" Score="0" Body="&lt;p&gt;Complete code in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;contents = open('path/to/file', 'r').read()&#xA;matches = re.findall(r'^(.*?)\n(.*?)$', contents)&#xA;joined = ['\t'.join([i[0] i[1]]) for i in matches]&#xA;print '\n'.join(joined)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rubular.com/r/Uo0ZOKhk7T&quot; rel=&quot;nofollow&quot;&gt;I tested this out on rubular&lt;/a&gt;, and it works on your example. It should generalize to others as well. To walk you through the regular expression, &lt;code&gt;^&lt;/code&gt; and &lt;code&gt;$&lt;/code&gt; match the beginning and end of lines, respectively, while (.*?) uses non-greedy wildcard matching to capture, first, everything up to the newline character, then everything up to the next end of line.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-20T23:27:03.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8536" PostTypeId="2" ParentId="3825" CreationDate="2015-10-21T01:08:06.420" Score="1" Body="&lt;p&gt;This seems to me as the plain old recommendation problem. The Accounts are the USERS and the Symptoms are the ITEMS. Each time an Account shows a particular Symptom your system will increment a count value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Creating the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ACCOUNT, SYMPTOM, COUNT&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can use/implement any sort of recommender system (Mahout is only an option, have you seen &lt;a href=&quot;http://mymedialite.net/&quot; rel=&quot;nofollow&quot;&gt;MyMediaLite&lt;/a&gt;) or you can even implement yours. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's reuse your ideas:&#xA;* You'd like to use a user-based similarity &#xA;* If an Account has shown 2 or more times the same symptom it seems to be important&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can filter out the Account, Symptom pairs with less than 2 counts, and with the rest you create the following datasets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;User, Item dataset:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ACCOUNT, SYMPTOM&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Table with a unique column containing all Users:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ACCOUNT&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Table with a unique column containing all Items:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;SYMPTOM&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can use directly the &lt;a href=&quot;http://mymedialite.net/documentation/doxygen/class_my_media_lite_1_1_item_recommendation_1_1_user_k_n_n.html&quot; rel=&quot;nofollow&quot;&gt;User-KNN algorithm&lt;/a&gt; from MyMediaLite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the recommender model already trained you can pass any ACCOUNT as input and it will give you a ranked list of the most probable SYMPTOMS that might appear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obs.: Initially ignore the time, then you could use it to partition your data in past/future and evaluate the recommendation in a more realistic way. ;-)&lt;/p&gt;&#xA;" OwnerUserId="13583" LastActivityDate="2015-10-21T01:08:06.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8537" PostTypeId="2" ParentId="6064" CreationDate="2015-10-21T01:29:59.400" Score="1" Body="&lt;p&gt;One starting approach is doing the cross-correlation as you've said. The correlation will fail to capture the time shifted behavior. But you should calculate it for every pair of Milk/Drug you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/acf.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;ccf()&lt;/code&gt;&lt;/a&gt; function in R does it for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're not successful i suggest you to take a look at the multiple time-series plotted side by side (like &lt;a href=&quot;https://stackoverflow.com/questions/19921842/plotting-multiple-time-series-on-the-same-plot-using-ggplot&quot;&gt;here&lt;/a&gt;) and check if really there exists this time-shifted pattern in your customer behavior.&lt;/p&gt;&#xA;" OwnerUserId="13583" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-10-21T01:29:59.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8538" PostTypeId="2" ParentId="2473" CreationDate="2015-10-21T01:46:30.197" Score="0" Body="&lt;p&gt;All answers above are quite good paths to follow. But if you want to select between multiple algorithms (all in R =D) and understand why selecting them, try this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Dimensionality_Reduction/Feature_Selection&quot; rel=&quot;nofollow&quot;&gt;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Dimensionality_Reduction/Feature_Selection&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last tip: The Feature Ranking approach in that page is exaclty what you want. ;-)&lt;/p&gt;&#xA;" OwnerUserId="13583" LastActivityDate="2015-10-21T01:46:30.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8539" PostTypeId="1" CreationDate="2015-10-21T11:18:43.607" Score="4" ViewCount="1434" Body="&lt;p&gt;I am trying to solve a decision making problem. In it, information evolves and increases with time for each event observed, and the history of the event may be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is as follows: in an electric network at time $t_0$ there is signal that automatically triggers the possible need of a technical intervention in a network to correct the problem. The system at $t_0$ has a large set of general indicators describing the state of the network based on which I'd like to determine whether to proceed with an intervention at $t_{0}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's possible to carry specific measurements at the point where the problem was detected (this takes some time $\Delta t$). At this stage we can again ask whether the intervention is necessary or not? We can iterate this procedure a few times. Then, my question is, which is the machine learning method best suited for this classification problem? Can it be formulated as a Survival problem?&lt;/p&gt;&#xA;" OwnerUserId="13593" LastEditorUserId="13593" LastEditDate="2015-10-21T14:55:10.750" LastActivityDate="2016-01-22T20:51:30.577" Title="Time dependent Classification problem" Tags="|machine-learning|classification|survival-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8540" PostTypeId="1" AcceptedAnswerId="8566" CreationDate="2015-10-21T11:57:51.553" Score="2" ViewCount="520" Body="&lt;p&gt;Let's say I need to classify addresses with scikit-learn, so if I want my classifier to be able to classify addresses by the street name, and post/zip code, should I do a OneVsRest classifier, or separate them into two different classifiers (for the same training set)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried both, and it seems like having multiple classifiers might be a better choice, as it feels faster to train multiple smaller classifiers. Is this how it is supposed to be done?&lt;/p&gt;&#xA;" OwnerUserId="13562" LastEditorUserId="13562" LastEditDate="2015-10-21T12:03:43.703" LastActivityDate="2018-01-31T15:27:01.020" Title="One multilabel classifier or one for each type of label?" Tags="|classification|python|svm|scikit-learn|multilabel-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8541" PostTypeId="2" ParentId="8395" CreationDate="2015-10-21T12:26:12.900" Score="0" Body="&lt;p&gt;Your requirements suggest you might want to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Uplift_modelling&quot; rel=&quot;nofollow&quot;&gt;uplift Modelling&lt;/a&gt;. This has a neat R package references in the wiki article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stochasticsolutions.com/pdf/sig-based-up-trees.pdf&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; looks like a practical paper.&lt;/p&gt;&#xA;" OwnerUserId="13598" LastActivityDate="2015-10-21T12:26:12.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8542" PostTypeId="1" AcceptedAnswerId="8543" CreationDate="2015-10-21T12:30:53.157" Score="4" ViewCount="7739" Body="&lt;p&gt;I am trying to classify a large-ish number of small strings (millions) into about 10 disjunct categories. Examples of classes and strings for each class include:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;email: &quot;foo@bar.com&quot;&#xA;phone: &quot;55&quot;, &quot;22334455&quot;&#xA;personName: &quot;John&quot;, &quot;Q.&quot;, &quot;Public&quot;&#xA;organizationName: &quot;Reuters&quot;, &quot;IBM&quot;&#xA;date: &quot;Dec.&quot;, &quot;22.10.2010&quot;&#xA;nameAndEmail: &quot;Larsonlarson@gmail.com&quot; (a last name has been concatenated with an email address.)&#xA;phoneAndEmail: &quot;55larson@gmail.com&quot; (part of a phone number has been concatenated with an emal address)&#xA;separator: &quot;,&quot;, &quot;and&quot;, &quot;or&quot;&#xA;other: stuff that does not fit in other categories.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Carrying out classification via a set of heuristics seems tedious, so I have been experimenting with an SVM. I'm not able to get more than about 80% accuracy without doing a lot of work manually encoding features like &quot;has an ampersand&quot;, &quot;is uppercase&quot;, &quot;is mostly numeric&quot;, &quot;has a colon in the middle&quot;, etc., which sort of defeats the purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering whether part of the challenge with using an SVM is that the &quot;salient features&quot; of a given string, like the '@' character, are not in a fixed position in the string, so it will not get the same feature index for each string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have suggestions for a more appropriate approach to this task, or can anyone recommend further reading?&lt;/p&gt;&#xA;" OwnerUserId="13596" LastEditorUserId="13413" LastEditDate="2015-10-21T17:32:24.590" LastActivityDate="2015-10-21T17:37:36.267" Title="Appropriate algorithm for string (not document) classification?" Tags="|machine-learning|classification|text-mining|multiclass-classification|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8543" PostTypeId="2" ParentId="8542" CreationDate="2015-10-21T13:29:57.420" Score="5" Body="&lt;p&gt;You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings.  With &lt;code&gt;N = 4&lt;/code&gt; or greater, you would capture things like &quot;.com&quot; in emails, &quot;##.#&quot; in dates, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might also help to encode all single digits as one reserved number-only-character.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy way to to this might be to create all the n-gram substrings for each string in your dataset, then simply treat that list of words as a document.  Then you could use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer&quot; rel=&quot;noreferrer&quot;&gt;term frequency&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;tf-idf&lt;/a&gt; vectors in your supervised step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example to create the substring uni-, bi-, and tri-grams for &quot;whatever@gmail.com&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = &quot;whatever@gmail.com&quot; &#xA;b = set([a[x:x+y] for y in range(0,4) for x in range(0,len(a))]) &#xA;&#xA;set(['', 'co', 've', 'ai', 'eve', 'r@', 'at', '.co', 'gm', 'ev', 'tev', 'er', '@gm', 'ver', '@g', 'r@g', 'ail', 'il.', 'gma', '.', 'te', 'hat', '@', 'wha', 'om', 'wh', 'er@', 'mai', 'ma', 'ha', 'l.c', 'a', 'c', 'ate', 'e', 'g', 'i', 'h', '.c', 'm', 'l', 'o', 'l.', 'r', 't', 'w', 'v', 'com', 'il'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-10-21T13:47:43.257" LastActivityDate="2015-10-21T13:47:43.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8544" PostTypeId="2" ParentId="8542" CreationDate="2015-10-21T14:13:34.260" Score="1" Body="&lt;p&gt;One can copy the 'bag of words' model for documents, though it becomes a 'bag of characters' model. (jamesmf extends this to n-grams, which will be more useful.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you want a feature to be salient to the algorithm, you need to give the algorithm access to that feature. You don't need to tell it what classes that feature is relevant for--just whether or not the feature is present. For example, the length of the string, number of digits in the string, and number of capital letters in the string are probably very useful for differentiating several of the classes you described, but will not be part of the feature vector unless you put them there. But all you need to do is identify them, assign them numbers, and then let the SVM take care of deciding how relevant they are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, having a binary variable for whether or not the last four characters are &lt;code&gt;.com&lt;/code&gt; (or whether &lt;code&gt;.com&lt;/code&gt; appears in the string if it's not always at the end) in the feature vector will convey the same amount of helpful information without also giving the algorithm a bunch of worthless information--there may be other 4-grams that are important, but it seems unlikely to me that it's worth the cost of considering all 4-grams and 3-grams in order to get the known features of &lt;code&gt;.com&lt;/code&gt;, &lt;code&gt;.net&lt;/code&gt;, &lt;code&gt;.org&lt;/code&gt;, and so on.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-10-21T14:13:34.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8545" PostTypeId="2" ParentId="6546" CreationDate="2015-10-21T15:00:24.857" Score="26" Body="&lt;p&gt;This is the import you need, and how to get the mean for a column named &quot;RBIs&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.sql.functions._&#xA;df.select(avg($&quot;RBIs&quot;)).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the standard deviation, see&#xA;&lt;a href=&quot;https://stackoverflow.com/questions/31789939/calculate-the-standard-deviation-of-grouped-data-in-a-spark-dataframe&quot;&gt;scala - Calculate the standard deviation of grouped data in a Spark DataFrame - Stack Overflow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For grouping by percentiles, I suggest defining a new column via a user-defined function (UDF), and using groupBy on that column.  See&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/sql-programming-guide.html#udf-registration-moved-to-sqlcontextudf-java--scala&quot; rel=&quot;noreferrer&quot;&gt;Spark SQL and DataFrames - Spark 1.5.1 Documentation - udf registration&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9146" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-10-21T15:18:22.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8546" PostTypeId="1" CreationDate="2015-10-21T15:48:01.713" Score="5" ViewCount="2975" Body="&lt;p&gt;I'm building a decision tree in R using the &lt;code&gt;rpart&lt;/code&gt; function, available in the library of the same name, but am experiencing some serious performance issues when partitioning data using more than one feature (there is roughly 5000 rows in my data set, and the features in question have around 30 unique values each). Running rpart with the one feature completes in a couple hours (running single thread on a CentOS server with about 250GB ram), but, after 6 days, a model with a second feature added has yet to complete!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created an approach to extracting the path through the tree new data takes (see my question/solution &lt;a href=&quot;https://datascience.stackexchange.com/questions/8440/extract-the-path-of-a-data-point-through-a-decision-tree-in-sklearn&quot;&gt;here&lt;/a&gt;), which is my main motivation here, but the method requires an object that can be type-cast as a &lt;code&gt;party&lt;/code&gt; object. A colleague of mine told me that he's seen performance boosts using the &lt;code&gt;ctree&lt;/code&gt; function, but the output of this is an inscrutable S4 object that I can't figure out how to convert. My questions are two:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does anyone know a way to speed up &lt;code&gt;rpart&lt;/code&gt; beyond what I've already done here:&lt;code&gt;clf &amp;lt;- rpart(Y ~ X$var1 + X$var2, rpart.control(maxcompete=1, maxsurrogate=0))&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Does anyone know how to convert a ctree object to a party object? Alternatively, does anyone have concurring or contradicting observations on ctree performance?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2023-01-05T17:25:23.797" Title="Slow convergence with rpart" Tags="|r|performance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8547" PostTypeId="2" ParentId="8542" CreationDate="2015-10-21T16:15:33.810" Score="0" Body="&lt;p&gt;For a first pass why not use regex?&lt;br&gt;&#xA;There are examples of regex for email&lt;br&gt;&#xA;You could use regex for email with a digit before @&lt;br&gt;&#xA;(and you cannot conclude 55 is a phone number)&lt;br&gt;&#xA;From there you would just need to decide if it has a name&lt;br&gt;&#xA;And you could use regex to extract the component before the @&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regex for date  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regex for phone number  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use regex to extract date and email from documents&lt;br&gt;&#xA;It is fast and as accurate as the regex&lt;br&gt;&#xA;Every regex is accurate it just may not pick up some edge cases&lt;br&gt;&#xA;You expand the regex or pick those up via the algorithm  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you also want to also classify invalid email at least you have valid email to compare to  &lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-10-21T17:37:36.267" LastActivityDate="2015-10-21T17:37:36.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8549" PostTypeId="1" CreationDate="2015-10-21T18:17:22.190" Score="12" ViewCount="47655" Body="&lt;p&gt;I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: &lt;code&gt;IPYTHON_OPTS=&quot;notebook&quot; pyspark&lt;/code&gt;. This opens a webpage listing all my IPython notebooks. I can select one of them, opening it in a second webpage. SparkContext (sc) is available already, and my first command in the notebook is &lt;code&gt;help(sc)&lt;/code&gt;, which runs fine. The problem I'm having is that I am getting a Java heap space error that I don't know how to address. How do I view my current Java heap setting, and how do I increase it within the context of my setup. The error message I'm getting follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.&#xA;: org.apache.spark.SparkException: Job aborted due to stage failure: Task 19 in stage 247.0 failed 1 times, most recent failure: Lost task 19.0 in stage 247.0 (TID 953, localhost): java.lang.OutOfMemoryError: Java heap space&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13348" LastEditorUserId="21" LastEditDate="2016-05-10T13:59:50.790" LastActivityDate="2019-07-31T10:40:16.913" Title="How do I set/get heap size for Spark (via Python notebook)" Tags="|apache-spark|pyspark|ipython|anaconda|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8550" PostTypeId="2" ParentId="3783" CreationDate="2015-10-21T18:49:41.357" Score="2" Body="&lt;p&gt;tl;dr: This is not a great candidate for machine learning solutions.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It is not exactly true that &quot;the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output&quot;.  Instead, strength of an encryption scheme is how resistant it is to &lt;a href=&quot;https://en.wikipedia.org/wiki/Cryptanalysis&quot; rel=&quot;nofollow&quot;&gt;cryptanalysis&lt;/a&gt;. Yes, input $\rightarrow$ output maps that show patterns are relatively easy to break. But the input $\rightarrow$ output map might be very random (high entropy) from many perspectives but still might be defeated through feasible cryptanalysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You offer another definition: &quot;informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable.&quot;  This is usefully different from your first definition because it focuses on the information that might be gleaned about the encryption scheme from a stream of input $\rightarrow$ output pairs.  This is different and easier than the usual cryptanalysis problem where all you have is a data set of (encrypted) outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Framed this way, I think you are heading toward a &lt;a href=&quot;https://en.wikipedia.org/wiki/Model_selection&quot; rel=&quot;nofollow&quot;&gt;model selection problem.&lt;/a&gt;  While this may be relatively straight forward with weak encryption schemes, it becomes very hard, very quickly with strong schemes.  In fact, I believe you walk right in to the &lt;a href=&quot;https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization&quot; rel=&quot;nofollow&quot;&gt;No Free Lunch theorem&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;In formal terms, there is no free lunch when the probability distribution on problem instances is such that all problem solvers have identically distributed results. In the case of search, a problem instance is an objective function, and a result is a sequence of values obtained in evaluation of candidate solutions in the domain of the function. For typical interpretations of results, search is an optimization process. There is no free lunch in search if and only if the distribution on objective functions is invariant under permutation of the space of candidate solutions.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Translated: if you find yourself needing to search over all possible cryptanalysis methods, there exist input $\rightarrow$ output streams where no method is probabilistically better than any other. This will defeat any model selection method based on optimization of some objective function.&lt;/p&gt;&#xA;" OwnerUserId="609" LastActivityDate="2015-10-21T18:49:41.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8551" PostTypeId="1" AcceptedAnswerId="8556" CreationDate="2015-10-21T19:52:39.080" Score="0" ViewCount="157" Body="&lt;p&gt;I am trying to implement section 3.4 of paper &lt;a href=&quot;http://arxiv.org/abs/1505.04803&quot; rel=&quot;nofollow&quot;&gt;Predicting Important Objects for Egocentric Video Summarization&lt;/a&gt; where they have created a distance matrix of frame histograms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, let say &lt;code&gt;Ω&lt;/code&gt; is mean of distances between all frames,&lt;code&gt;DV&lt;/code&gt;is distance matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't understand what is meant by this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We next perform complete-link agglomerative clustering&#xA;  with DV , grouping frames until the smallest maximum interframe&#xA;  distance is larger than two standard deviations beyond&#xA;  Ω&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can this be achieved by setting the &lt;code&gt;cutoff&lt;/code&gt; value to &lt;code&gt;2Ω&lt;/code&gt; in Matlab's &lt;a href=&quot;http://www.mathworks.com/help/stats/clusterdata.html&quot; rel=&quot;nofollow&quot;&gt;clusterdata function&lt;/a&gt; ? &lt;/p&gt;&#xA;" OwnerUserId="13606" LastEditorUserId="13413" LastEditDate="2015-10-21T19:58:46.113" LastActivityDate="2015-10-22T06:55:40.367" Title="Agglomerative Clustering Stopping Criteria" Tags="|clustering|matlab|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8552" PostTypeId="1" CreationDate="2015-10-21T20:40:31.937" Score="2" ViewCount="400" Body="&lt;p&gt;I have 1,000 photoshoot-quality images of pianos on white background with very little noise (people in the background, etc). How would I go about finding the piano image that looks most like the others? Alternatively, how would I find five pianos that look most like the &quot;average&quot; piano. I'm particularly looking for python libraries that may be helpful (I'm currently looking at skimage).&lt;/p&gt;&#xA;" OwnerUserId="13529" LastEditorUserId="13413" LastEditDate="2015-10-21T21:02:48.430" LastActivityDate="2015-10-22T10:32:16.303" Title="Find most representative image" Tags="|machine-learning|python|image-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8554" PostTypeId="2" ParentId="8552" CreationDate="2015-10-22T04:50:25.360" Score="3" Body="&lt;p&gt;Find a low-dimensional emdedding of the images using a nonlinear dimensionality reduction method like autoencoding, then select the image closest to the centroid in the embedding space.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-10-22T04:50:25.360" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8556" PostTypeId="2" ParentId="8551" CreationDate="2015-10-22T06:55:40.367" Score="1" Body="&lt;p&gt;Clearly, 2 standard deviations beyond Omega is not the same as twice the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently, their process is this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;compute the distance matrix&lt;/li&gt;&#xA;&lt;li&gt;compute the mean&lt;/li&gt;&#xA;&lt;li&gt;compute the standard deviation&lt;/li&gt;&#xA;&lt;li&gt;compute hierarchical clustering with maximum linkage&lt;/li&gt;&#xA;&lt;li&gt;cut the tree at mu+2*sigma&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Because complete linkage is in O(n^3), this approach will not scale to longer videos or higher frame rates.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-10-22T06:55:40.367" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8557" PostTypeId="1" CreationDate="2015-10-22T10:11:39.940" Score="3" ViewCount="84" Body="&lt;p&gt;I would like to ask a question about recommender systems. We are showing some movies to users and they have to decide if they like them or not. These movies have only a few attributes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Title&#xA;Director&#xA;Category&#xA;Duration&#xA;Nationality&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We show the attributes as a block. The users can decide if they like (the block) or not. We aggregate these decisions to recommend them more movies in the future. My question is: Are there any methods that can tell us whether a user is uninterested in a given attribute? For example, if a user is never considering duration when saying yes or no.&lt;/p&gt;&#xA;" OwnerUserId="13618" LastEditorUserId="13413" LastEditDate="2015-10-22T16:44:51.410" LastActivityDate="2015-10-23T14:43:41.973" Title="Discovering non-interesting attributes" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8558" PostTypeId="2" ParentId="8552" CreationDate="2015-10-22T10:32:16.303" Score="2" Body="&lt;p&gt;You could use the OpenCV library, specifically, the &lt;a href=&quot;http://docs.opencv.org/doc/user_guide/ug_traincascade.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Haar-Cascade technique&lt;/strong&gt;&lt;/a&gt; . The OpenCV is a computer vision library that have a python interface. You could take a look in &lt;a href=&quot;http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;this tutorial&lt;/strong&gt;&lt;/a&gt; to start. &lt;/p&gt;&#xA;" OwnerUserId="13521" LastActivityDate="2015-10-22T10:32:16.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8560" PostTypeId="1" CreationDate="2015-10-22T12:54:38.083" Score="0" ViewCount="3277" Body="&lt;p&gt;I have some time-series data, which I need to use to predict a binary label for a given time-stamp. I was initially doing it using a Logistic Regression Model but I later figured that a time-series based problem could be better solved using Hidden Markov Models. I might also save on time since this prediction is to be made for ~200,000 time-stamps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been able to obtain the next &lt;em&gt;n&lt;/em&gt; hidden states using the &lt;a href=&quot;http://github.com/hmmlearn/hmmlearn&quot; rel=&quot;nofollow&quot;&gt;hmmlearn&lt;/a&gt; implementation, but what I really need are 0/1 labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I (if possible) use HMM as a binary classifier?&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2015-10-22T13:00:12.593" LastActivityDate="2016-10-14T20:04:02.820" Title="Can HMM be used as a binary classifier?" Tags="|classification|time-series|markov-process|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8561" PostTypeId="1" CreationDate="2015-10-22T14:48:58.663" Score="3" ViewCount="396" Body="&lt;p&gt;I'm training a 2-layer, 1024 node, dropout of 0.5 RNN over natural text. Specifically, I'm using &lt;a href=&quot;https://github.com/karpathy/char-rnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;karpathy's char-rnn&lt;/a&gt; which I found to work quite well for most of my use cases. Sometimes however, late in the training, my errors drastically increase, and I can't figure out why this is happening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is specifically asking why am I observing the drastic increase in error, and how this is handled in practice. What can be done to prevent this besides taking the last-known good model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uq6YB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uq6YB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12963" LastEditorUserId="11097" LastEditDate="2016-03-08T06:53:20.090" LastActivityDate="2023-11-29T14:05:05.610" Title="What causes the error in a RNN to increase late in training?" Tags="|machine-learning|rnn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8563" PostTypeId="2" ParentId="8557" CreationDate="2015-10-22T18:21:41.997" Score="2" Body="&lt;p&gt;As a simple measure, you could simply take the &lt;a href=&quot;https://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;point biserial correlation&lt;/a&gt; between duration and rating for a given user.  If they aren't correlated (positively or negatively), this variable is likely not important to that user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to determine the impact of a variable is by training another model excluding that variable and determining how that affects the quality of the predictions.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-10-23T14:43:41.973" LastActivityDate="2015-10-23T14:43:41.973" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8564" PostTypeId="2" ParentId="8146" CreationDate="2015-10-22T18:30:05.927" Score="0" Body="&lt;p&gt;You might train a &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic model&lt;/a&gt; (like &lt;a href=&quot;https://www.quora.com/What-is-a-good-explanation-of-Latent-Dirichlet-Allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt;) to approach this problem.  This would allow you to represent your queries (which won't necessarily contain the exact name of the products desired) in terms of topic vectors.  You can also represent your products in terms of their topic vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then rank the products in terms of &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_%28mathematics%29&quot; rel=&quot;nofollow&quot;&gt;similarity&lt;/a&gt; (in topic vector space) to the query.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-22T18:30:05.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8565" PostTypeId="2" ParentId="6533" CreationDate="2015-10-22T21:42:58.660" Score="1" Body="&lt;p&gt;In &lt;a href=&quot;https://en.wikipedia.org/wiki/Third_normal_form&quot; rel=&quot;nofollow&quot;&gt;third normal form&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;IndvidualID&lt;br&gt;&#xA;CatID&lt;br&gt;&#xA;CaseID&lt;br&gt;&#xA;Value   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Primary Key:  IndvidualID, CatID, CaseID&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need to constrain CatID and/or CaseID then have a FK   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From that structure you can easily loop and create long format or multidimensional array.&lt;br&gt;&#xA;It is also space efficient as a null is simply no row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sql an avg &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select CatID, avg(value)  &#xA;  from table &#xA; where CaseID = x &#xA; group by CatID  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you can just loop on those same constraints&lt;br&gt;&#xA;Or a tool like LINQ     &lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-10-22T22:10:05.993" LastActivityDate="2015-10-22T22:10:05.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8566" PostTypeId="2" ParentId="8540" CreationDate="2015-10-22T21:49:56.613" Score="4" Body="&lt;p&gt;Both ways are valid and both are commonly used. Sometimes, a classifier that claims to be multilabel may just be separating the labels into multiple OneVsRest classifiers under-the-hood and conveniently joining the results together at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are cases where the methods are fundamentally different. For instance, in training a neural net with multiple targets (labels), you can setup the structure of the network such that there is shared structure. The shared nodes will end up learning features that are useful for all the targets, which could be very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you're classes (labels) are &quot;cat-pet&quot;, &quot;cat-big&quot;, and &quot;dog&quot;, you may want an algorithm that first learns to distinguish between any cat and any dog, and then in a later step learns to separate cats that are pets from cats that are big (like a lion!). This is called hierarchy, and if your classifier can exploit hierarchy you may gain better accuracy. If your classes are completely independent however, it may not make any difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you start with the method that is easiest (i.e. OneVsRest), and see if the performance is suitable to your needs, then move to more complicated methods (multilabel, hierarchical methods, etc) only once you need better performance.&lt;/p&gt;&#xA;" OwnerUserId="10683" LastEditorUserId="10683" LastEditDate="2015-10-22T23:26:36.043" LastActivityDate="2015-10-22T23:26:36.043" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8568" PostTypeId="1" CreationDate="2015-10-22T22:32:24.597" Score="3" ViewCount="313" Body="&lt;p&gt;I'm training a support vector machine classifier (SVM) on 100.000 observations. I'd like to try different parameter combinations (including kernel types) using cross-validation. However, it is computationally expensive to optimize over these different parameter values. Therefore, I'd like to test them on a subset of 5.000 observations on a validation set and then based on those results choose the parameters for the final model that is trained on 100.000 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is then whether there is something intrinsic in these parameters that will make this approach problematic. More specifically, is there a parameter, where training on the full data set will require a different value for optimal performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the following kernels with their parameters in parentheses:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rbf (gamma, C)&lt;/li&gt;&#xA;&lt;li&gt;poly (degree, C)&lt;/li&gt;&#xA;&lt;li&gt;sigmoid (gamma, C)&lt;/li&gt;&#xA;&lt;li&gt;linear (gamma).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="13413" LastEditDate="2015-10-22T23:23:37.663" LastActivityDate="2016-01-21T01:16:21.307" Title="Does importance of SVM parameters vary for subsample of data?" Tags="|machine-learning|classification|svm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8569" PostTypeId="2" ParentId="8568" CreationDate="2015-10-22T22:48:13.960" Score="2" Body="&lt;p&gt;In theory, if you have a large enough random sample of your data set, it should be representative of the characteristics in the larger population of data that will affect the relationship between parameter values and performance. It seems to me that a $5\%$ sample of your data might be too small for what you're trying to do. When I'm performing cross-validation on a test/dev data set, I usually set aside somewhere in the neighborhood of $10\%$ for the hold-out test/evaluation data, and use $90\%$ for my development work, but, as you pointed out, this may be too big given the computational constraints you're up against. Depending on the type of application you're working on, you could look at feature distribution in your sampled data and compare it to that in the larger population. If the two sets are comparable, then you might be ok to do your dev. work on the smaller data set. I say it depends on the type of application you're developing, because many statisticians would consider this cheating, in that it's generally not good model evaluation practice to look at hold-out data at all. Here's the procedure I'd recommend:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Separate out $10\%$ of your data for hold-out evaluation.&lt;/li&gt;&#xA;&lt;li&gt;Divide the remaining $90\%$ into $5\%-10\%$ parameter optimization, and the remainder for final parameter eval.&lt;/li&gt;&#xA;&lt;li&gt;Compare the feature distributions of the parameter optimization and final parameter eval data sets, if they're not comparable, redraw the samples (within the context of the $90\%$ development sample)&lt;/li&gt;&#xA;&lt;li&gt;When you have a good subsample, run your parameter optimization experiments on the small parameter optimization data set.&lt;/li&gt;&#xA;&lt;li&gt;With your final parameter settings, evaluate with cross-validation on the the $90\%$ combined parameter optimization and final parameter eval data sets.&lt;/li&gt;&#xA;&lt;li&gt;Perform final model analysis by training on the $90\%$ data set and classifying on the $10\%$ hold-out evaluation set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It might also be worth looking into optimizing your analytics pipeline. For example, is file I/O, feature generation and feature extraction part of the the workflow you're using, or have you done all of that off-line and are just concerned with the SVM evaluation part?&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-10-22T22:48:13.960" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8570" PostTypeId="2" ParentId="5142" CreationDate="2015-10-23T06:43:34.580" Score="0" Body="&lt;p&gt;Personally, I don't use vagrant with a local provisioning. I have installed a Spark cluster locally without HDFS which allows me to experiment and develop easily without the overwhelm of a virtual machine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HDFS is not a requirement for local clusters, and it's also a kind of a system administration nightmare if you just need it for local testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark works very fine with local file systems which you'll also have to port when deploy on your Cloud of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, You can use vagrant with AWS provisioning to create a cluster for heavier testing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: AWS works with S3 and not HDFS. AWS's HDFS version is ephemeral and if you put down the cluster, you'll loose all your computations. For persistence, you'll need to write back to S3. &lt;/p&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2015-10-23T06:43:34.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8571" PostTypeId="2" ParentId="6639" CreationDate="2015-10-23T07:41:01.997" Score="4" Body="&lt;p&gt;The code below from SaturnAPI answers this question. See and run the code at &lt;a href=&quot;https://saturnapi.com/artitw/neural-network-decimal-digits-to-binary-bitwise-conversion&quot; rel=&quot;noreferrer&quot;&gt;https://saturnapi.com/artitw/neural-network-decimal-digits-to-binary-bitwise-conversion&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% Welcome to Saturn's MATLAB-Octave API.&#xA;% Delete the sample code below these comments and write your own!&#xA;&#xA;% Exercise from http://neuralnetworksanddeeplearning.com/chap1.html&#xA;% There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.&#xA;&#xA;% Inputs from 3rd layer&#xA;xj = eye(10,10)&#xA;&#xA;% Weights matrix&#xA;wj = [0 0 0 0 0 0 0 0 1 1 ;&#xA;      0 0 0 0 1 1 1 1 0 0 ;&#xA;      0 0 1 1 0 0 1 1 0 0 ;&#xA;      0 1 0 1 0 1 0 1 0 1 ]';&#xA;&#xA;% Results&#xA;wj*xj&#xA;&#xA;&#xA;% Confirm results&#xA;integers = 0:9;&#xA;dec2bin(integers)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13625" LastEditorUserId="13625" LastEditDate="2015-10-23T07:48:34.083" LastActivityDate="2015-10-23T07:48:34.083" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8573" PostTypeId="5" CreationDate="2015-10-23T15:32:52.940" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-10-23T15:32:52.940" LastActivityDate="2015-10-23T15:32:52.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8574" PostTypeId="4" CreationDate="2015-10-23T15:32:52.940" Score="0" Body="Support Vector Machines (SVM) are a popular supervised machine learning algorithm that can be used for classification or regression." OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2015-10-24T00:21:10.973" LastActivityDate="2015-10-24T00:21:10.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8575" PostTypeId="2" ParentId="6577" CreationDate="2015-10-23T17:08:58.483" Score="1" Body="&#xA;&#xA;&lt;p&gt;Here's a naïve implementation of your pseudocode in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;calculate_cost &amp;lt;- function(kwh){&#xA;    if(kwh &amp;lt; 20){&#xA;        return(0)&#xA;    }else{&#xA;         kwh &amp;lt;- kwh - 20&#xA;    if(kwh &amp;lt;= 5){&#xA;        return(kwh * 0.1)&#xA;    }else{&#xA;        final_cost &amp;lt;- kwh * 0.1&#xA;        kwh &amp;lt;- kwh - 5&#xA;        if(kwh &amp;lt;= 100){&#xA;            return(final_cost + (kwh * 0.053))&#xA;        }else{&#xA;            final_cost &amp;lt;- final_cost + (kwh * 0.053)&#xA;            kwh &amp;lt;- kwh - 100&#xA;            if(kwh &amp;lt;= 875){&#xA;                return(final_cost + (kwh * 0.042))&#xA;            }else{&#xA;                final_cost &amp;lt;- final_cost + (kwh * 0.042)&#xA;                kwh &amp;lt;- kwh - 875&#xA;                return(final_cost + (kwh * 0.039))&#xA;                }&#xA;            }&#xA;        }&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; calculate_cost(kwh=625)&#xA;[1] 113.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="10683" LastEditDate="2015-10-23T17:45:25.407" LastActivityDate="2015-10-23T17:45:25.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8576" PostTypeId="2" ParentId="8549" CreationDate="2015-10-23T17:10:12.590" Score="7" Body="&lt;p&gt;I solved it by creating a &lt;strong&gt;&lt;code&gt;spark-defaults.conf&lt;/code&gt;&lt;/strong&gt; file in &lt;code&gt;apache-spark/1.5.1/libexec/conf/&lt;/code&gt; and adding the following line to it:&#xA;&lt;strong&gt;&lt;code&gt;spark.driver.memory 14g&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That solved my issue. But then I ran into another issue of &lt;code&gt;exceeding max result size of 1024MB&lt;/code&gt;. The solution was to add another line in the file above:&#xA;&lt;strong&gt;&lt;code&gt;spark.driver.maxResultSize 2g&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="13348" LastEditorUserId="59101" LastEditDate="2019-01-09T16:05:18.357" LastActivityDate="2019-01-09T16:05:18.357" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="8577" PostTypeId="1" CreationDate="2015-10-23T17:27:00.107" Score="1" ViewCount="627" Body="&lt;p&gt;I'm trying to perform classification on a large dataset with mixed numerical and categorical features. The dataset is all semi-structured text, so everything is a String. Does anyone know of a library I could use to automatically identify numeric features (e.g. &quot;width (in.)&quot;, which corresponds to the width in inches)? This is important because numeric and categorical features are encoded/handled differently in my analytics pipeline. I think this is basically a named entity recognition problem wherein the entity is a numeric variable or feature that exists in the data set as a String/text.&lt;/p&gt;&#xA;" OwnerUserId="13348" LastEditorUserId="13348" LastEditDate="2015-10-23T18:25:36.033" LastActivityDate="2015-10-24T08:21:31.120" Title="Recognizing numerical entities" Tags="|feature-extraction|named-entity-recognition|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="8578" PostTypeId="2" ParentId="8528" CreationDate="2015-10-23T17:30:01.997" Score="2" Body="&lt;h2&gt;Here's one solution:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;using the following data:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;USER-LEVEL:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;fss&lt;/code&gt; &lt;em&gt;first session start (timestamp) - time of first pageview of&#xA;first session for that user&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;css&lt;/code&gt; &lt;em&gt;converting session start (timestamp) - time of the first pageview of session where user converted&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;ns&lt;/code&gt; &lt;em&gt;number of sessions (int) - total # of sessions&#xA;before conversion for user&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SESSION-LEVEL:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;ss&lt;/code&gt; &lt;em&gt;session start (timestamp) - timestamp of first pageview in session&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;FORMULA:&lt;/strong&gt; &lt;em&gt;calculated for each session&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;FOR FIRST SESSION: Attribution Credit is just linear: &lt;em&gt;(or really whatever you want to make it)&lt;/em&gt;&lt;br&gt;&lt;br&gt; &lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=sc%5C%20%3D%5C%20%5Cfrac%7B1%7D%7Bns%7D&quot; alt=&quot;linear credit&quot;&gt;&lt;br&gt;&lt;br&gt; &lt;/li&gt;&#xA;&lt;li&gt;ALL OTHER SESSIONS: &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First calculate the SUM of all session's &quot;time before conversion&quot;:&lt;br&gt;&lt;br&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=sum(tbc)=%5Cfrac%7B((css-fss)%5C%20%5Ctimes%5C%20ns)-sum(ss)%7D%7Bcss-fss%7D&quot; alt=&quot;sum(tbc)&quot;&gt;&lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Then Calculate the credit for each session:&lt;br&gt;&lt;br&gt;&lt;img src=&quot;https://chart.googleapis.com/chart?cht=tx&amp;amp;chl=sc%5C%20%3D%5C%20%5Cfrac%7B1%5C%20-%5C%20%5Cfrac%7Bcss%5C%20-%5C%20ss%7D%7Bcss%5C%20-%5C%20fss%7D%7D%7Bns%5C%20-%5C%20%5Cfrac%7Bsum(tbc)%7D%7Bcss%5C%20-%5C%20fss%7D%7D%5Ctimes%5C%20%5Cfrac%7Bns-1%7D%7Bns%7D&quot; alt=&quot;sc formula&quot;&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;it isn't perfect because it distributes credit based on the relative position of a session in the User's lifetime.  Ideally you have a preset halflife (like 7 days) so that comparison is more consistent.&lt;/p&gt;&#xA;" OwnerUserId="13574" LastActivityDate="2015-10-23T17:30:01.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8580" PostTypeId="2" ParentId="8577" CreationDate="2015-10-24T08:21:31.120" Score="1" Body="&lt;p&gt;There are many good libraries for identifying number-like &lt;em&gt;values&lt;/em&gt;, but identifying corresponding &lt;em&gt;fieldnames&lt;/em&gt; is trickier and likely very problem-specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;purely data driven approach&lt;/strong&gt; might look for co-occurrences with numbers, for example:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if [*number-like* capacity] or [capacity: *number-like*] occurs in &amp;gt; x% of &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the instances where &lt;em&gt;&quot;capacity&quot;&lt;/em&gt; occurs,&#xA;    then &lt;em&gt;&quot;capacity&quot;&lt;/em&gt; can be guessed to be a label of number-like things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even the nature of these relationships (eg, whether it should be before, after, capitalised) could be learnt from existing known labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done with &lt;a href=&quot;https://cloud.google.com/bigquery/bigquery-web-ui&quot; rel=&quot;nofollow&quot;&gt;Google BigQuery&lt;/a&gt; using the public &lt;a href=&quot;https://bigquery.cloud.google.com/table/publicdata:samples.trigrams&quot; rel=&quot;nofollow&quot;&gt;tri-gram sample data&lt;/a&gt;.  But I assume that's out of scope and impractical here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;practical approach&lt;/strong&gt;, might aim to classify these strings as &lt;code&gt;numerical_units&lt;/code&gt;, &lt;code&gt;categorical_units&lt;/code&gt; or &lt;code&gt;unsure&lt;/code&gt; or even more classes, and then do human review of the latter.  (There are some very tricky cases, for example, &lt;em&gt;&quot;capacity&quot;&lt;/em&gt; is numerical, but &lt;em&gt;&quot;capacity type&quot;&lt;/em&gt; would not be.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a starting list you can use:&lt;br&gt;&#xA;&lt;a href=&quot;https://en.wiktionary.org/wiki/Category:en:Units_of_measure&quot; rel=&quot;nofollow&quot;&gt;wiktionary:Category:en:Units_of_measure&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://en.wiktionary.org/wiki/Category:Symbols_for_SI_units&quot; rel=&quot;nofollow&quot;&gt;wiktionary:Category:Symbols_for_SI_units&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://en.wiktionary.org/wiki/Category:en:Mathematics&quot; rel=&quot;nofollow&quot;&gt;wiktionary:Category:en:Mathematics&lt;/a&gt;&lt;br&gt;&#xA;Note that they are in the singular.  For your domain considering finding other such lists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a string is a full match with one of those labels, you can consider it a &lt;code&gt;numerical_unit&lt;/code&gt;.&#xA;Your next concern is tolerant, fuzzy matching.  You could treat '.' as a wildcard (so &lt;em&gt;&quot;in*&quot;&lt;/em&gt; matches &lt;em&gt;&quot;inch&quot;&lt;/em&gt;), or find the actual abbreviations of units like &lt;em&gt;&quot;inch&quot;&lt;/em&gt;.  These you can label as &lt;code&gt;unsure&lt;/code&gt; and then review.  Likewise if the word is simply contained in the string, eg &lt;em&gt;&quot;arch length&quot;&lt;/em&gt; contains &lt;em&gt;&quot;length&quot;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think once you have done this you can make some refinements and add some special cases.  Without seeing even a sample of your data, it is difficult to say more.  If most of your strings are numerical units, then it may be easier to identify categorical units instead.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2015-10-24T08:21:31.120" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8581" PostTypeId="2" ParentId="8560" CreationDate="2015-10-24T09:30:55.723" Score="2" Body="&lt;p&gt;I have no experience in hmmlearn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you want to receive the labels for the underlying states of the process you have to do 'decoding' by using the 'Viterbi' algorithm. It calculates the most probable state sequence for your data. So if you only have two hidden states it calculates a 0/1 label for each of your time-stamps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead you're trying to discern between two different processes and recognize them, you have to train two different HMMs for the two processes and calculate the foward probabilty up to time t. Then you can choose the process with the higher probability. &lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2015-10-24T09:30:55.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8582" PostTypeId="2" ParentId="8539" CreationDate="2015-10-24T09:54:27.783" Score="1" Body="&lt;p&gt;From the description of the problem, you can just choose whatever classification algorithm you like to classify the need of intervention at  $t_{0}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general rule for choosing a classifier for this is to start simple--e.g., by using nearest neighbor and iterate to more powerful classifiers until you get enough accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If additional measurements at $t_{0}$ can be made a classifier can be used to determine the necessity of intervention without this additional data, then based on the probability output of this classifier and a threshold the additional measurements can be triggered. Based on this measurements another probability can be output which hopefully uses more informative features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you care about $\Delta t$, you should think about wether there is a real time dependency. Does the need of intervention depend upon the &lt;strong&gt;series&lt;/strong&gt; of classifications? Is the order of determining interventions as necessary important for you? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, you do not have to care about ordering for this application.&#xA;Rather, I would suggest pooling classifications for one instance according to the most frequent label or something like that. &lt;/p&gt;&#xA;" OwnerUserId="12798" LastEditorUserId="13413" LastEditDate="2015-10-24T18:49:18.850" LastActivityDate="2015-10-24T18:49:18.850" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8583" PostTypeId="1" AcceptedAnswerId="8584" CreationDate="2015-10-24T13:24:09.340" Score="3" ViewCount="848" Body="&lt;p&gt;I've built a neural network for a speech recognition task using the &lt;a href=&quot;https://catalog.ldc.upenn.edu/LDC93S1&quot; rel=&quot;nofollow noreferrer&quot;&gt;timit dataset&lt;/a&gt;. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 PLP values, 13 about first order derivative and 13 about second order derivative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to improve my dataset. The only thing I've tried thus far is normalizing the dataset using a standard scaler (standardizing features with mean 0 and variance 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Since my dataset has high dimensionality, is there a way to visualize? For now, I've just plotted the dataset values using a heat map.&lt;/li&gt;&#xA;&lt;li&gt;Are there any methods for separating my sample even more, making it easier to differentiate between the classes?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My heat map is below, representing 20 samples. In this heatmap there are 5 different phonemes, related to vowels, in particular, &lt;em&gt;uh, oy, aw, ix,&lt;/em&gt; and &lt;em&gt;ey.&lt;/em&gt;&#xA;As you can see, each phoneme is not really distinguishable from the others. Does anyone know how could I improve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A1Yby.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A1Yby.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13648" LastEditorUserId="13413" LastEditDate="2015-10-24T13:42:39.870" LastActivityDate="2015-10-25T17:36:12.733" Title="How to visualize data of a multidimensional dataset (TIMIT)" Tags="|machine-learning|neural-network|feature-selection|visualization|preprocessing|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8584" PostTypeId="2" ParentId="8583" CreationDate="2015-10-24T15:47:05.650" Score="1" Body="&lt;p&gt;Like I said in the comment, you'll need to perform dimension reduction, otherwise you'll not be able to visualize the $\mathbb{R}^n$ vector space and this is why :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visualization of high-dimensional data sets is one of the traditional applications of dimensionality reduction methods such as PCA (Principal components analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In high-dimensional data, such as experimental data where each dimension corresponds to a different measured variable, dependencies between different dimensions often restrict the data points to a manifold whose dimensionality is much lower than the dimensionality of the data space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many methods are designed for manifold learning, that is, to find and unfold the lower-dimensional manifold. There has been a research boom in manifold learning since 2000, and there now exist many methods that are known to unfold at least certain kinds of manifolds successfully.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the most used methods for dimension reduction is called PCA or Principal component analysis. PCA is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. You can read more on this topics &lt;a href=&quot;http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So once you reduce your high dimensional space into a ${\mathbb{R}^3}$ or ${\mathbb{R}^2}$ space you will able to project it using your adequate visualization method.&#xA;References :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume11/venna10a/venna10a.pdf&quot; rel=&quot;nofollow&quot;&gt;Information Retrieval Perspective to Nonlinear Dimensionality&#xA;Reduction for Data Visualization&lt;/a&gt; - Jarkko Venna&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; &lt;em&gt;To avoid confusion for some concerning PCA and Dimension Reduction&lt;/em&gt;, I add the following details :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA will allow you compute the principal components of your vector model, so the information are not lost but &quot;synthesized&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately there is no other imaginable way to display 39 dimensions on a 2/3 dimension screen. If you wish to analyze correlations between your 39 features, maybe you should consider another visualization technique. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend a scatter plot matrix in this case.&lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="5177" LastEditDate="2015-10-25T17:36:12.733" LastActivityDate="2015-10-25T17:36:12.733" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8586" PostTypeId="1" AcceptedAnswerId="8588" CreationDate="2015-10-24T20:19:43.473" Score="3" ViewCount="205" Body="&lt;p&gt;Experienced in signal/image analysis, and new to data science, I recently was challenged with a relatively simple dataset: 100 to 200 items, about 10-20 numerical variables (in the [0-1] or percentage range), with only one variable used at present time for ranking, and 5 to 10 categorical variables, each with few options. A categorical variable takes about 2 to 4 different values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like first to get insight on potential structures in such data. I have browsed Agresti's &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470082895,subjectCd-STZ0.html&quot; rel=&quot;nofollow&quot;&gt;Analysis of Ordinal Categorical Data&lt;/a&gt;, some have advised me to invest on TDA (&lt;a href=&quot;https://en.wikipedia.org/wiki/Topological_data_analysis&quot; rel=&quot;nofollow&quot;&gt;Topological data Analysis&lt;/a&gt;). Yet I do not know where to start from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have guidelines, best practices on such REAL data to gradually address the aforementioned issues, from visualization to genuine processing/inference?&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2015-10-25T00:10:49.983" Title="Steps in exploratory methods for mild-sized data with mixed categorical and numerical values?" Tags="|data-mining|categorical-data|linear-regression|numerical|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8587" PostTypeId="1" CreationDate="2015-10-24T20:32:11.577" Score="1" ViewCount="369" Body="&lt;p&gt;&lt;em&gt;Historical background&lt;/em&gt;:  &lt;a href=&quot;https://en.wikipedia.org/wiki/Hugo_Steinhaus&quot; rel=&quot;nofollow&quot;&gt;Hugo Steinhaus&lt;/a&gt; can be considered as an early father of data science. He authored the paper &lt;a href=&quot;http://www.laurent-duval.eu/Documents/Steinhaus_H_1956_j-bull-acad-polon-sci_division_cmp-k-means.pdf&quot; rel=&quot;nofollow&quot;&gt;Sur la division des corps matériels en parties&lt;/a&gt;, Bulletin de l’Académie Polonaise des Sciences, 1956. The title could read &quot;On the division of material bodies into parts&quot; in English. As the original paper is hard to obtain, the above link is a retranscription from  &lt;a href=&quot;http://tex.stackexchange.com&quot;&gt;latex&lt;/a&gt; that I made in the summer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is considered as one of the first known origins of the $K$-means algorithms (see &lt;a href=&quot;http://dx.doi.org/10.1016/j.patrec.2009.09.011&quot; rel=&quot;nofollow&quot;&gt;Data Clustering: 50 Years Beyond K-Means&lt;/a&gt;, Anil K. Jain, Pattern recognition letters, 2010 or &lt;a href=&quot;http://www.jehps.net/Decembre2008/Bock.pdf&quot; rel=&quot;nofollow&quot;&gt;Origins and extensions of the k-means algorithm in cluster analysis&lt;/a&gt;, Hans-Hermann Bock, Electronic Journ@l for History of Probability and Statistics, 2008).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;https://en.wikipedia.org/wiki/Hugo_Steinhaus&quot; rel=&quot;nofollow&quot;&gt;his wikipedia page&lt;/a&gt;, H. Steinhaus  has collaborated with engineers, geologists, economists, physicians, biologists (and even lawyers), and lacking of trustworthy information during World War II, he invented a statistical tool to estimate 'German' losses, using necrologic news from German soldiers on the front. He notably used the mention that the soldier killed was the first, second or third child from a family.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for references (possibly second-hand notes, translated papers) to this applied and &quot;missing value&quot; statistical tool related works? Namely:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Regarding his war lacunary inferences, where is this algorithm described (as I would like to be able to re-implement it)?&lt;/li&gt;&#xA;&lt;li&gt;What are specific examples of his applied works with biologists for instance, were they published, and where?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="12527" LastEditDate="2016-08-27T07:30:56.430" LastActivityDate="2016-08-27T07:30:56.430" Title="What are references explaining Hugo Steinhaus early &quot;data science&quot; work?" Tags="|clustering|k-means|missing-data|history|" AnswerCount="0" CommentCount="4" ClosedDate="2020-05-26T00:29:54.087" ContentLicense="CC BY-SA 3.0" />
  <row Id="8588" PostTypeId="2" ParentId="8586" CreationDate="2015-10-25T00:10:49.983" Score="4" Body="&lt;p&gt;You can get a reasonably good approximation of steps for &lt;em&gt;exploratory data analysis (EDA)&lt;/em&gt; by reviewing the &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/eda.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;EDA section&lt;/a&gt; of the &lt;em&gt;NIST Engineering Statistics Handbook&lt;/em&gt;. Additionally, you might find helpful parts of &lt;a href=&quot;https://datascience.stackexchange.com/a/1006/2452&quot;&gt;my related answer&lt;/a&gt; here on &lt;em&gt;Data Science SE&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Methods, related to EDA, are too diverse that it is not feasible to discuss them in a single answer. I will just mention several approaches. If you are interested in applying &lt;em&gt;classification&lt;/em&gt; to your data set, you might find information, mentioned in &lt;a href=&quot;https://stats.stackexchange.com/a/135842/31372&quot;&gt;my other answer&lt;/a&gt; helpful. In order to &lt;em&gt;detect structures&lt;/em&gt; in a data set, you can try to apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;principal component analysis (PCA)&lt;/a&gt;. If, on the other hand, you are interested in exploring &lt;em&gt;latent structures&lt;/em&gt; in data, consider using &lt;a href=&quot;https://en.wikipedia.org/wiki/Exploratory_factor_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;exploratory factor analysis (EFA)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2452" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-25T00:10:49.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8590" PostTypeId="1" AcceptedAnswerId="8617" CreationDate="2015-10-25T05:42:48.993" Score="1" ViewCount="166" Body="&lt;p&gt;If anyone has used the Neural Network toolbox in matlab, what does the two values in the performance section means? Like considering this image:&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/c1CoK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/c1CoK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;br&gt;&#xA;The performance section has the value 0.484 and 1.32e-07. what are these two values. Among them which is the mean squared error value. If one is the mean squared error value, then what is the other value?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2015-10-27T04:58:18.030" LastActivityDate="2015-10-27T13:53:26.117" Title="Query related to Matlab Neural Network toolbox" Tags="|machine-learning|neural-network|algorithms|tools|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8591" PostTypeId="2" ParentId="6578" CreationDate="2015-10-25T06:31:33.783" Score="0" Body="&lt;p&gt;Document authorship attribution is a  valuable technique in many areas such as plagiarism detection, and lawsuit prosecution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have few research papers on source code authorship attribution. In these papers, we have discussed simple models such as Naive Bayes classifier and few advanced techniques such as Deep Learning. Though, our papers are specifically on source code authorship attribution, you can apply algorithms we discussed in our papers for general document classification problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Papers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]. &lt;a href=&quot;http://www.ijmlc.org/papers/50-A243.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ijmlc.org/papers/50-A243.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]. &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0167865512003571&quot; rel=&quot;nofollow&quot;&gt;http://www.sciencedirect.com/science/article/pii/S0167865512003571&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3]. &lt;a href=&quot;http://link.springer.com/chapter/10.1007/978-3-642-42042-9_46&quot; rel=&quot;nofollow&quot;&gt;http://link.springer.com/chapter/10.1007/978-3-642-42042-9_46&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3040" LastActivityDate="2015-10-25T06:31:33.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8592" PostTypeId="2" ParentId="6578" CreationDate="2015-10-25T10:37:43.900" Score="1" Body="&lt;p&gt;Additional to the content of the document, the form could contain even more valuable information. I can't comment on the formatting of pdf files, but it should be possible to exploit the pdf metadata. &lt;em&gt;Author, Title, subject&lt;/em&gt; or &lt;em&gt;keywords&lt;/em&gt; could provide some clue pointing to a common origin in the same organization.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-10-25T10:37:43.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8593" PostTypeId="1" CreationDate="2015-10-25T12:21:40.167" Score="0" ViewCount="568" Body="&lt;p&gt;I have a sorted sequence of integers, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1,2,480,1000,1100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;representing volumes in some categories. The task is to separate the valid data (high volumes) from the outliers (low volumes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example I expect following obvious split&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1,2   |   480,1000,1100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is of course trivial, if there is a definition of outliers, such as &lt;em&gt;less than .1% of the total volume&lt;/em&gt;, but I'd prefer to have an &#xA;algorithm for calculating this threshold as a result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know anything about the distribution of the values, except for the outliers have low values and the regular categories have high volumes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, I'd say the approach should pass all possible N-1 splits of the sequence and evaluate some criterion, but I fail to&#xA;find a meaningful one (e.g. I distinctly failed comparing the WSS of the sequence with the sum of the WSS of both splits).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate hints to the algorithm or to transformation of this problem in other well known unsupervised problem.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="13413" LastEditDate="2015-10-26T11:49:06.693" LastActivityDate="2015-10-26T11:49:06.693" Title="Identifying outliers in an unknown distribution" Tags="|machine-learning|outlier|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8597" PostTypeId="1" CreationDate="2015-10-25T20:32:16.883" Score="2" ViewCount="18" Body="&lt;p&gt;I am interesting in knowing an approximate amount of &lt;a href=&quot;https://en.wikipedia.org/wiki/Speeded_up_robust_features&quot; rel=&quot;nofollow&quot;&gt;SURF features&lt;/a&gt; used to do e.g. image-recognition on 224x224x3 images. Is there a standard amount of features used? Or perhaps a well-acclaimed article using SURF features that I could look at? I am not going to work with the features, but I am just trying to understand some aspects of their usage.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="1330" LastEditDate="2022-04-18T18:13:06.087" LastActivityDate="2022-04-18T18:13:06.087" Title="Number of SURF features normally used?" Tags="|image-classification|feature-engineering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8598" PostTypeId="1" CreationDate="2015-10-25T22:05:51.123" Score="5" ViewCount="8135" Body="&lt;p&gt;As I understand it one can use cross validation to help find the optimal pruning of a classification or regression tree, for example, implemented here in &lt;a href=&quot;http://au.mathworks.com/help/stats/classificationtree.cvloss.html#bunpajd-1&quot; rel=&quot;noreferrer&quot;&gt;Matlab&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand the following about this process. Since each cross validation fold has random data, it may be that the trees fit on each data set may be different and have differing depths. How can one find the best pruning level over all such trees then?&lt;/p&gt;&#xA;" OwnerUserId="13623" LastActivityDate="2017-07-26T16:48:30.007" Title="How is cross validation used to prune a decision tree" Tags="|decision-trees|cross-validation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8599" PostTypeId="1" AcceptedAnswerId="8602" CreationDate="2015-10-25T23:37:21.817" Score="0" ViewCount="150" Body="&lt;p&gt;Using the automated upgrade, when I try to upgrade I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“YARN Timeline state preserving restart should be enabled&#xA;Reason: YARN should have state preserving restart enabled for the Timeline server. The yarn-site.xml property yarn.timeline-service.recovery.enabled should be set to true”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, using Ambari, whenever I go to change the setting in yarn-site.xml it gets me to create a new configuration group and changes it there. I have moved all of the servers into that config group but the check still fails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I modify the default yarn-site.xml?&#xA;Alternatively, does anyone know the actual script (or how to find it) that is kicked off when the automated upgrade is invoked?&lt;/p&gt;&#xA;" OwnerUserId="13671" LastActivityDate="2015-10-26T06:48:17.057" Title="Yarn timeline recovery not enabled error upgrading via ambari" Tags="|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8602" PostTypeId="2" ParentId="8599" CreationDate="2015-10-26T06:48:17.057" Score="0" Body="&lt;p&gt;Run the following from the ambari server's shell:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;/var/lib/ambari-server/resources/scripts/configs.sh set mace gamma yarn-site &quot;yarn.timeline-service.recovery.enabled&quot; &quot;true&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where mace is the ambari server's host name and gamma is the cluster name&lt;/p&gt;&#xA;" OwnerUserId="13671" LastActivityDate="2015-10-26T06:48:17.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8603" PostTypeId="2" ParentId="8593" CreationDate="2015-10-26T07:12:40.513" Score="1" Body="&lt;p&gt;If the high volume and low volume data both appear in non-negligible probability, you may view the problem as a clustering problem with two clusters. Gaussian mixture model or K-means algorithm may help you in this case. &lt;/p&gt;&#xA;" OwnerUserId="13496" LastActivityDate="2015-10-26T07:12:40.513" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8604" PostTypeId="1" AcceptedAnswerId="8619" CreationDate="2015-10-26T08:50:41.190" Score="0" ViewCount="85" Body="&lt;p&gt;I have some archive data that show the variation of a certain quantity X measured over 1-year intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm now facing the problem of understanding how X correlates with another variable Y, that I can measure much more frequently, even daily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, is it really necessary to collect data for Y so often? This seems such a waste of resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I would do is to collect data for Y only for 10% of the days of each year and I would select those days randomly. This way I'm sure that the observations are independent because the sample size is not too large and days have been selected randomly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think of that? Is that the approach you would follow as well?&lt;/p&gt;&#xA;" OwnerUserId="13680" LastActivityDate="2015-10-27T15:15:57.083" Title="how often should one sample a dataset?" Tags="|statistics|data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8605" PostTypeId="2" ParentId="8598" CreationDate="2015-10-26T09:01:05.700" Score="0" Body="&lt;p&gt;Develop a training set, validation set and test set. Training set contains 40% of observations, validation and test set contain 30% each of the observations (assuming you have a fairly large dataset in the first place). &#xA;Develop 5 decision trees, each with differing parameters that you would like to test. Run these decision trees on the training set and then validation set and see which decision tree has the lowest ASE (Average Squared Error) on the validation set. You can use a different validation criterion if you so choose but I prefer the ASE. If you feel that the best performing tree can still perform better given the ASE on the validation set, then go back and tweak the parameters of this tree and re-run on training and validation set. Once you are happy, use your final tree and predict on your test set.   &lt;/p&gt;&#xA;" OwnerUserId="13176" LastActivityDate="2015-10-26T09:01:05.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8606" PostTypeId="1" AcceptedAnswerId="8865" CreationDate="2015-10-26T11:50:41.147" Score="3" ViewCount="3835" Body="&lt;p&gt;Definition of Click Through Rates:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;CTR is the number of clicks that your ad receives divided by the&#xA;  number of times your ad is shown expressed as a percentage (clicks ÷&#xA;  impressions = CTR).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-09T14:30:16.700" Title="What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?" Tags="|r|python|optimization|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8607" PostTypeId="2" ParentId="8606" CreationDate="2015-10-26T12:09:50.823" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;nofollow&quot;&gt;Logistic regression&lt;/a&gt; is very common and efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will provide you with a probability of click for a given impression. You will likely need additional information for optimization, like keywords or A/B testing.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-10-26T12:09:50.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8608" PostTypeId="2" ParentId="8480" CreationDate="2015-10-26T14:22:36.297" Score="0" Body="&lt;p&gt;In AdaBoost I guess at each step the model is &quot;greedily&quot; updated and given that typically you regularize heavily and there are bounds on model complexity this can have unexpected results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the deterministic AdaBoost I think adding a feature f1 which is random can't degrade performance if the tree depth has no bounds (so the tree can become as complex as it needs to be-remember we are talking of minimizing the training error so overfitting is ok).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In actual implementations however the tree depth has an upper bound which might lead to premature pruning of the tree and hence you might end up getting a higher training error.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2015-10-26T14:22:36.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8609" PostTypeId="2" ParentId="664" CreationDate="2015-10-26T14:57:27.840" Score="1" Body="&lt;p&gt;The only way to be sure is by benchmarking, but for glm Fisher scoring should be faster than coordinate descent.  Fisher scoring is a special case of Newton Raphson, which has a faster &lt;a href=&quot;https://en.wikipedia.org/wiki/Rate_of_convergence&quot; rel=&quot;nofollow&quot;&gt;rate of convergence&lt;/a&gt; than coordinate descent (Newton-Raphson is quadratically convergent, while coordinate descent is linearly convergent.)  So while the computation of second-derivative information means each step takes more time, it can require many fewer steps than coordinate descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the lasso, the special form of the penalty term makes it a very special case (and in fact absolute value isn't differentiable anyway, though sometimes you can finesse this).  For this special problem, coordinate descent proves to be particularly fast.  There are many other optimization problems where in practice Newton-Raphson is faster.&lt;/p&gt;&#xA;" OwnerUserId="13687" LastActivityDate="2015-10-26T14:57:27.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8613" PostTypeId="2" ParentId="8018" CreationDate="2015-10-26T16:35:02.110" Score="2" Body="&lt;p&gt;A first derivative, on the surface, would do it.  However, the data you show have a great deal of noise in them, so we need a way to evaluate the first derivative in a somewhat noiseless way, or at least within a frequency domain that eliminates the jitter and preserves the major derivative change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wavelet analysis could achieve this for you, especially if you use the first derivative of a Gaussian as your mother wavelet.  R has some decent wavelet packages (see &lt;a href=&quot;https://cran.r-project.org/web/packages/wavelets/wavelets.pdf&quot; rel=&quot;nofollow&quot;&gt;r-project.org&lt;/a&gt; for starters).  If you do the wavelet transform at short scales, this will identify the locations of the bits of jitter in the steering.  If you do it at larger scales (i.e. lower frequency), you can likely find just the lane shifts and not the little jitters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you train the transform up with a reasonable data set, you should be able to identify a scale or range of scales that corresponds to lane changes.  But note that if you don't figure that out, this is something like O(n^2), so try to narrow the scale range down a bit to save compute time.&lt;/p&gt;&#xA;" OwnerUserId="13484" LastActivityDate="2015-10-26T16:35:02.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8614" PostTypeId="2" ParentId="5860" CreationDate="2015-10-27T02:48:34.590" Score="2" Body="&lt;p&gt;You could start by learning R or Python and Machine Learning in one of those languages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other free options for books are &lt;em&gt;The LION Way: Machine Learning plus Intelligent Optimization&lt;/em&gt; and &lt;em&gt;Learning Deep Architectures for AI&lt;/em&gt;, which are both freely available online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also the free and open &lt;a href=&quot;https://www.edx.org/course/artificial-intelligence-uc-berkeleyx-cs188-1x&quot; rel=&quot;nofollow&quot;&gt;Artificial Intelligence course by Berkeley&lt;/a&gt; at edX.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have listed all the free books I could find on &lt;a href=&quot;http://www.learndatasci.com/free-books/&quot; rel=&quot;nofollow&quot;&gt;machine learning and artificial intelligence books&lt;/a&gt; at LearnDataSci, if you're interested.&lt;/p&gt;&#xA;" OwnerUserId="12521" LastActivityDate="2015-10-27T02:48:34.590" CommentCount="1" CommunityOwnedDate="2015-10-27T02:48:34.590" ContentLicense="CC BY-SA 3.0" />
  <row Id="8616" PostTypeId="1" CreationDate="2015-10-27T09:55:31.540" Score="12" ViewCount="415" Body="&lt;p&gt;I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence : &lt;a href=&quot;https://i.stack.imgur.com/NjMvs.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NjMvs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was expecting to see a plateau sooner on the ngram setup since it needless context. However, one thing I wasn't expecting was that the prediction rate drops.&#xA;In my understanding since we already have a context of 3 words, the plateau should converge asymptotically to its highest value.&#xA;But both the recurrent network and the Ngram models are experiencing this drop.&#xA;I have no idea why it would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note RNNLM is the name of the framework used to build the recurrent neural net, it uses 500 neurons and 100M direct connections, RNN25 is the same setup but with a training base divided by for)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the sentence size distribution :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/MxSY1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MxSY1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="10232" LastEditorUserId="83275" LastEditDate="2019-11-18T08:53:48.340" LastActivityDate="2021-03-29T22:01:00.090" Title="ngram and RNN prediction rate wrt word index" Tags="|neural-network|nlp|statistics|rnn|ngrams|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="8617" PostTypeId="2" ParentId="8590" CreationDate="2015-10-27T13:53:26.117" Score="1" Body="&lt;p&gt;I don't see it explicitly discussed in the documentation, but it appears that the numbers on the right side of the screen are the stopping criteria.  That run appears to have stopped because the error on the validation set increased six times (explained &lt;a href=&quot;http://www.mathworks.com/help/nnet/ref/plotperform.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;).  I'm guessing it would also stop after 1000 epochs, when 0 MSE was achieved, the gradient was below 1e-6, or after 4 resets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the number to the left of the bar should be the MSE on the epoch with the best validation error*.  I'm guessing the value of the blue bar is the lowest that MSE got during the iterations that were actually increasing the validation error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the docs:&#xA;&quot;In the default setup, the training stops after six consecutive increases in validation error, and the best performance is taken from the epoch with the lowest validation error.&quot;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-27T13:53:26.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8619" PostTypeId="2" ParentId="8604" CreationDate="2015-10-27T15:15:57.083" Score="1" Body="&lt;p&gt;In order to show a correlation between X and Y they should be aggregated over the same time period. Otherwise, there there may be seasonality or other features in Y that may have been averaged out in X. Is it possible re-index X to get monthly aggregates? I would not sub-sample Y. Find a common time period for both X and Y to make your comparisons.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-10-27T15:15:57.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8622" PostTypeId="1" AcceptedAnswerId="122986" CreationDate="2015-10-27T17:16:08.807" Score="10" ViewCount="3357" Body="&lt;p&gt;Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or tutorial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases I just create a function, add some noise, and plot it, but there are many times when I wish I could just click my mouse on a graph to generate points. For instance, when I want to generate a fairly complex relationship between x and y, it's a hassle to think of the right formulation to generate the points programmatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does there exist a tool that will allow me to generate data points using my mouse, with an option to export to CSV (or other simple format)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I am currently learning how to use mutual information and entropy as a metric of dependence between variables. I'd like to see what happens when I have data that is clearly dependent but does not have a linear relationship, so I drew this image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PtwUR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PtwUR.png&quot; alt=&quot;Non-linear dependence&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I just need a way to export the coordinates of the points to CSV. I realize this is a simple program and I could write my own, but surely someone has already done this and created a tool to do so? It could be a website, an .exe, Python source, or any other application.&lt;/p&gt;&#xA;" OwnerUserId="10683" LastActivityDate="2023-07-28T07:16:39.323" Title="Tool to Generate 2D Data via Mouse Clicking" Tags="|data|tools|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8623" PostTypeId="1" AcceptedAnswerId="8634" CreationDate="2015-10-27T20:02:55.957" Score="1" ViewCount="130" Body="&lt;p&gt;I am working on a litigation support application using the &lt;a href=&quot;https://www.cs.cmu.edu/~./enron/&quot; rel=&quot;nofollow&quot;&gt;Enron corpus&lt;/a&gt;, which contains about 600,000 unique text documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In litigation, one is often concerned with whether a document is responsive or non-responsive. One produces responsive documents to the opposing side, unless they are privileged (e.g., attorney client communication).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, I have sample sets of over 200 responsive and non-responsive documents.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge here is that the topic of the responsive documents is about one thing, whereas the non-responsive documents could be about any number of topics,  ranging from spam, to soccer practice, to business documents, etc.  The non-responsive is what I would call &lt;em&gt;diluted&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what those non-responsive classes are up front, and there is no purpose or value to breaking them out up front. If a document is non-responsive, then it needs to be quickly (and cheaply) dismissed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the samples are random when the classification is applied to the corpus my customers expect the split between corpus responsive and non-responsive to be close to the split of the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best approach to classify responsive versus non-responsive in this situation?    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below I briefly describe what I have tried. If there is a better approach, please share.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Using tf-idf, create an average vector for each document class (responsive and non-responsive).&lt;/li&gt;&#xA;&lt;li&gt;Take the first 1000 terms (sorted by weight), such that each vector is the same length. &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Normalized the vectors.&lt;br&gt;&#xA;Note: these two vectors are only a 0.03 cosine similarity to each other.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For each document in the responsive sample set, calculate the cosine similarity to the single average vector.&lt;br&gt;&#xA;Note: the average is 0.06.  It is a small number, as documents in the sample set have around 40 terms, as compared to an average vector of 1000. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the non-responsive perform the same analysis (compare each non-responsive sample document to the non-responsive average vector).&lt;br&gt;&#xA;In the non-responsive that same comparison is 0.03. Basically, the non-responsive average vector is diluted.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Based on this approach, I cannot really conclude if document has a higher cosine similarity to responsive compared to non-responsive then it is responsive.  By that measure 1/3 of the documents in the non-responsive sample set would be classified as responsive.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The non-responsive needs to be handicapped.  In this approach, how do I handicap it?  Is there another approach that would accomplish the same thing? &lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-10-27T22:04:07.287" LastActivityDate="2015-11-03T21:33:06.700" Title="Classification when one class is other" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8624" PostTypeId="2" ParentId="6774" CreationDate="2015-10-28T00:42:02.707" Score="1" Body="&lt;p&gt;According to that &lt;a href=&quot;http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples&quot; rel=&quot;nofollow&quot;&gt;same documentation you provided&lt;/a&gt;, the way to use custom emission probabilities is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;strong&gt;1.1.3. Implementing HMMs with custom emission probabilities&lt;/strong&gt; &lt;br&gt;&#xA;If you want to implement other emission probability (e.g. Poisson), you have to implement a new HMM class by inheriting the _BaseHMM and overriding the methods &lt;strong&gt;init&lt;/strong&gt;, _compute_log_likelihood, _set and _get for additional parameters, _initialize_sufficient_statistics, _accumulate_sufficient_statistics and _do_mstep.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: The documentation pages are for hmm.learn version 0.1.* which is no longer the most recent one. That is, it seems that the documentation page available at any given time might not correspond to the most recent version of hmmlearn. &lt;a href=&quot;https://github.com/hmmlearn/hmmlearn/issues/54&quot; rel=&quot;nofollow&quot;&gt;For example&lt;/a&gt;, sample code for hmm  0.1.* does not run with hmm learn 0.20.&lt;/p&gt;&#xA;" OwnerUserId="13720" LastActivityDate="2015-10-28T00:42:02.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8625" PostTypeId="1" AcceptedAnswerId="8632" CreationDate="2015-10-28T02:21:19.663" Score="19" ViewCount="33630" Body="&lt;p&gt;I'm looking for a Python package that implements multivariate linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Terminological note: &lt;em&gt;multivariate&lt;/em&gt; regression deals with the case where there are more than one dependent variables while &lt;em&gt;multiple&lt;/em&gt; regression deals with the case where there is one dependent variable but more than one independent variables.)&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2015-10-30T03:30:28.643" LastActivityDate="2017-08-28T23:07:58.010" Title="Multivariate linear regression in Python" Tags="|python|regression|library|software-recommendation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8626" PostTypeId="2" ParentId="8622" CreationDate="2015-10-28T07:24:42.917" Score="9" Body="&lt;p&gt;I recently discovered this site: &lt;a href=&quot;https://guoguibing.github.io/librec/datagen.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://guoguibing.github.io/librec/datagen.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Outputs list of points, and color ID (class) for each point.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Screenshot:&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/2lEiH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2lEiH.png&quot; alt=&quot;Datagen screenshot&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13726" LastEditorUserId="10683" LastEditDate="2021-10-01T05:04:24.113" LastActivityDate="2021-10-01T05:04:24.113" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="8628" PostTypeId="2" ParentId="810" CreationDate="2015-10-28T07:47:26.290" Score="5" Body="&lt;p&gt;Separate the operational and the training scenarios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The operational scenario is the one in which your classifier will be measure on.&#xA;This is where you should perform well.&#xA;Use should have a dataset that is representative of this scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training scenario is whatever you are doing in order to build a classifier that will perform well on the operational scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many time the datasets in both scenarios are of the same nature so there is no need to distinct them.&#xA;For example, you have some online store so you use for training past usage in order to perform well on the future usage. &#xA; However, while training you can use a different dataset than a one that represents the operational scenario.&#xA;Actually, if you sleep, dream of a classifier, validate it on your operational scenario (this step should be done after waking up), you are just as good as after going the usual machine learning path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distinction between operational and training scenarios becomes important when the dataset is imbalanced. Most algorithms won't perform well on such a dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, don't hesitate to use two datasets -&#xA;You can use a balanced dataset for the training.&#xA;Once you are done, validate your classifier of the operational dataset.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2016-03-17T12:20:47.600" LastActivityDate="2016-03-17T12:20:47.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8632" PostTypeId="2" ParentId="8625" CreationDate="2015-10-28T13:58:41.143" Score="14" Body="&lt;p&gt;You can still use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression&quot; rel=&quot;noreferrer&quot;&gt;sklearn.linear_model.LinearRegression&lt;/a&gt;.  Simply make the output &lt;code&gt;y&lt;/code&gt; a matrix with as many columns as you have dependent variables.  If you want something &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#polynomial-regression-extending-linear-models-with-basis-functions&quot; rel=&quot;noreferrer&quot;&gt;non-linear&lt;/a&gt;, you can try different basis functions, use polynomial features, or use a different method for regression (like a NN).&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-28T13:58:41.143" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8633" PostTypeId="1" AcceptedAnswerId="8636" CreationDate="2015-10-28T14:17:49.470" Score="0" ViewCount="56" Body="&lt;p&gt;The initial task can be described like this:&#xA;I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I need to know is answer to the question &quot;Has my system ever seen this element before?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most close data structure I was able to find so far is a bloom filter, but it has false positives which better to avoid in my task as it results in data loss&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example providing I account to store at least 2^32 items, with positive error rate of just 1%(which means 1% of all urls won't be visited) I would need at least&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n = 4,294,967,296, p = 0.01 (1 in 100) → m = 41,167,512,262 (4.79GB), k = 7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;4.79GB of memory...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task itself is a high scale web crawler, so I need to keep track of already visited urls(or sha1 hashes of this urls)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is welcome&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="13735" LastActivityDate="2015-10-28T15:40:05.240" Title="Memory efficient structure for membership checking without false positive" Tags="|bigdata|scalability|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8634" PostTypeId="2" ParentId="8623" CreationDate="2015-10-28T14:23:47.867" Score="1" Body="&lt;p&gt;As a first step you should try &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt; on your tf-idf vectors.  It is simple to implement and would provide a good baseline for comparison.  You can find an implementation in whatever language you're using.  You could also try some kind of (&lt;a href=&quot;https://www.cs.princeton.edu/~blei/papers/BleiMcAuliffe2007.pdf&quot; rel=&quot;nofollow&quot;&gt;perhaps supervised&lt;/a&gt;) &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic modeling&lt;/a&gt; to create a better feature space, but that would be more involved.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-28T14:23:47.867" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="8635" PostTypeId="2" ParentId="8622" CreationDate="2015-10-28T14:28:10.890" Score="3" Body="&lt;p&gt;In R:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First set up a blank plot with whatever x and y scale limits you need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(NA, xlim=c(11,20),ylim=c(10,99))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then click click click with mouse-button 1 and end with mouse-button 2 (probably):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pts = data.frame(locator(type=&quot;p&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then save as a CSV file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;write.csv(pts,&quot;pts.csv&quot;,row.names=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;producing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;x&quot;,&quot;y&quot;&#xA;20.9461142167608,54.0921852908633&#xA;11.6463003491398,24.5409354249845&#xA;14.4239385175408,44.1769632963908&#xA;14.7755382856928,29.5957544809901&#xA;14.7931182741004,62.8409105801038&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2015-10-28T14:28:10.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8636" PostTypeId="2" ParentId="8633" CreationDate="2015-10-28T15:21:19.960" Score="2" Body="&lt;p&gt;For web crawler scale why not use a distributed database like Apache Cassandra? Lookups on indexes are efficient and no false positives.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-10-28T15:21:19.960" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8638" PostTypeId="1" AcceptedAnswerId="8650" CreationDate="2015-10-28T20:10:27.143" Score="0" ViewCount="89" Body="&lt;p&gt;I am looking for the best method to go from a sequence of events such as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time event&#xA;1 a&#xA;2 b&#xA;3 a&#xA;4 b&#xA;5 c&#xA;6 d&#xA;7 c&#xA;8 d&#xA;9 e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where each letter corresponds to a certain event that occurs at a time. I want to reduce the number of events by aggregating frequently occurring events into a new event. A possible solution data set would look like,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 a'&#xA;2 a'&#xA;3 a'&#xA;4 a' &#xA;5 b'&#xA;6 b'&#xA;7 b'&#xA;8 b'&#xA;9 e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the clusters are created because they occur in a sequence following each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was looking at the text mining algorithms in R with tm or RNA sequenceing&#xA;with edgeR. But I have no experience in this so I was hoping that someone can&#xA;shed me some light on a common approach for this type of problems.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2015-10-29T12:57:13.050" LastActivityDate="2015-10-29T16:41:20.990" Title="Unsupervised sequence identification" Tags="|feature-extraction|sequence|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8640" PostTypeId="2" ParentId="8339" CreationDate="2015-10-29T09:11:50.207" Score="3" Body="&lt;p&gt;Simply combining by voting some classifiers can naturally give bag results. Consider a toy example like having a set of data with $100$ instances. Suppose you have $3$ classifiers. Let's say on first $70$ instances all classifiers match perfectly. Than on next $10$ first classifier is good, the others are bad, on the next $10$, the 2nd is good, others bad, and on the last $10$ the 3rd is good, others are bad. All three models goes on with $0.80$ accuracy. Voting them would lead to $0.70$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also some other potential difficulties like unscaled scores for base learners (it often happen with SVM, if it is not scaled by a logistic or whatever). Another reason is the model family provided by different learners. Bagging (or bootstrap aggregating which is used in RF for example) work simply because they assume that the model comes from the same family (it is the same tree model, or statistical speaking it is the majority class over a region), and only the samples are drawn randomly. Thus bootstrapping which requires independent and identically distributed sub-samples does not work in your case, since your models (which can be considered functions over samples) are not identically distributed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are however solutions to this problem, one of them is stacked generalization (or simply stacking). The simplest way is stacking with a logistic regressor. The idea is to train some base classifiers (which you already have), and take their results as input variables for a logistic regression. The top logistic regression would be able to find the 'proper balance' between the predictions and usually you get some increased accuracy as a benefit among others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further references see: &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning#Stacking&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page on ensemble learning&lt;/a&gt;. Search on Google for 'stacked generalization'. For papers start with the seminal paper of &lt;a href=&quot;http://machine-learning.martinsewell.com/ensembles/stacking/Wolpert1992.pdf&quot; rel=&quot;nofollow&quot;&gt;David Wolpert - Stacked Generalization - 1992&lt;/a&gt; which started all the discussion on this topic.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-10-29T09:11:50.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8641" PostTypeId="1" CreationDate="2015-10-29T09:48:39.013" Score="3" ViewCount="89" Body="&lt;p&gt;I just graduated in Computer Science, with a very theoretical background but without any kind of Data Science or Artificial Intelligence experience, and I working on my own to discover those two fields. More precisely, I try to work on a toy subspace clustering k-means-like algorithm, and I think I successfully learned basic optimization techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I have some ad-hoc subspace clustering algorithm, and... I'm stuck. How do I validate it? During my studies, I studied a lot of formal proving or model checking techniques, but I feel they are totally irrelevant here. I read several AI papers, and it seems that there is a strong tradition of experimental validation in Clustering, with a lot of validity measures. My problem is that they mean nothing to me. I don't understand what they prove, or even why they are relevant. I would be very interested in a general course about the experimental validation methodology (if it actually is a thing!), if possible with theoretical justification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, I don't really know what are the good properties people are looking for in clustering algorithms. What general theorems should I aim for? I understand this may be a question too related to what I am working on - a generic answer for k-means-like algorithms would be enough.&lt;/p&gt;&#xA;" OwnerUserId="13754" LastEditorUserId="31513" LastEditDate="2017-07-11T11:16:48.757" LastActivityDate="2017-07-11T11:16:48.757" Title="How do I learn experimental methodology? When is it relevant?" Tags="|clustering|experiments|methodology|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8642" PostTypeId="1" CreationDate="2015-10-29T10:30:39.640" Score="6" ViewCount="2993" Body="&lt;p&gt;I've recently read about concept learning in a machine learning class. They defined a concept as (translated from German):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Concept:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Describes a subset of objects or events defined on a bigger set.&lt;/li&gt;&#xA;  &lt;li&gt;Boolean function defined on bigger set&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;Given: Examples which are labeled as members or not-members&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Searched: Automatically infer the definition of the underlying concept&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Definition of concept learning: Infering a boolean-valued function from training data of its input and its output.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would call this a 2-class classification problem. Is there a difference?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-03T08:47:37.627" Title="What is the difference between concept learning and classification?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8644" PostTypeId="2" ParentId="8641" CreationDate="2015-10-29T12:15:41.227" Score="2" Body="&lt;p&gt;On the metrics. A common technique is to make simulated data with known labels. Then see if your algorithm can reproduce the clustering; if it can't there's a problem, if it can there may still be problems but you're off the starting blocks. Typical measures would include ROC and AUC (essentially how many times does the algo correctly place object i and j in the same class, balanced against false positives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another tactic is to use well known real datasets to benchmark against. E.g. the classic iris dataset. See how your algo performs in terms of speed and accuracy versus other algos.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to what people are looking for this could mean a multitude of things. You might do well to consider where your algo is suited, find an application in that area, and find out whether it performs for that purpose.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-10-29T12:15:41.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8645" PostTypeId="1" AcceptedAnswerId="8662" CreationDate="2015-10-29T12:33:07.310" Score="10" ViewCount="13408" Body="&lt;p&gt;I'm using seaborn library to generate bar plots in python. I'm wondering what statistics are used to compute the error bars, but can't find any reference to this in the &lt;a href=&quot;https://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.barplot.html&quot; rel=&quot;noreferrer&quot;&gt;seaborn's barplot documentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the bar values are computed based on mean in my case (the default option), and I assume the error bars are computed based on a Normal distribution 95% confidence interval, but I'd like to be sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aT584.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aT584.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="31513" LastEditDate="2017-05-29T12:05:57.030" LastActivityDate="2017-05-29T12:05:57.030" Title="Python Seaborn: how are error bars computed in barplots?" Tags="|python|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8646" PostTypeId="1" AcceptedAnswerId="8652" CreationDate="2015-10-29T12:42:35.213" Score="5" ViewCount="4183" Body="&lt;p&gt;I'm trying to predict rare events, meaning less than 1% of positive cases. I basically try to predict if a subject will have 0, 1, 2 ... , 6, &gt; 6 failures (there are cases in all those categories).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried several algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;decision trees&lt;/li&gt;&#xA;&lt;li&gt;random forest&lt;/li&gt;&#xA;&lt;li&gt;adaboost&lt;/li&gt;&#xA;&lt;li&gt;grouping using k-means clustering and finding associations with failures (which group has most failure)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In any case, learning either goes to no failure or has too much variance (leading poor reasults on C.V. set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know any machine learning algorithms which are better suited for rare events?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it surprising that I get those bad results using those algorithms, which means that my features list is not good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2015-10-29T19:21:27.530" Title="Predictive analysis of rare events" Tags="|machine-learning|algorithms|predictive-modeling|random-forest|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8649" PostTypeId="1" AcceptedAnswerId="8651" CreationDate="2015-10-29T16:18:45.837" Score="0" ViewCount="130" Body="&lt;p&gt;I am trying to solve the following problem, but am having some difficulty. Can anyone give me some guidance?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are lots of tourists in Grandeville. The streets in Grandeville run&#xA;  east to west and go from&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;..., S. 2nd St., S. 1st St., Broadway St., N. 1st St., N. 2nd St., ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;The avenues run north to south and go from&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;..., E. 2nd Ave., E. 1st Ave., Broadway Ave., W. 1st Ave., W. 2nd Ave., ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;These streets form a square block grid. For each of the questions&#xA;  below, the tourist starts at the intersection of Broadway St. and&#xA;  Broadway Avenue and moves one block in each of the four cardinal&#xA;  directions with equal probability.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q1)&lt;/strong&gt; What is the probability that the tourist is at least 3 city blocks (as&#xA;the crow flies) from Broadway and Broadway after 10 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q2)&lt;/strong&gt; What is the probability that the tourist is at least 10 city blocks&#xA;(as the crow flies) from Broadway and Broadway after 60 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q3)&lt;/strong&gt; What is the probability that the tourist is ever at least 5 city&#xA;blocks (as the crow flies) from Broadway and Broadway within 10 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q4)&lt;/strong&gt; What is the probability that the tourist is ever at least 10 city&#xA;blocks (as the crow flies) from Broadway and Broadway within 60 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q5)&lt;/strong&gt; What is the probability that the tourist is ever east of East 1st&#xA;Avenue but ends up west of West 1st Avenue in 10 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q6)&lt;/strong&gt; What is the probability that the tourist is ever east of East 1st&#xA;Avenue but ends up west of West 1st Avenue in 30 moves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q7)&lt;/strong&gt; What is the average number of moves until the first time the tourist&#xA;is at least 10 city blocks (as the crow flies) from Broadway and&#xA;Broadway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q8)&lt;/strong&gt; What is the average number of moves until the first time the tourist&#xA;is at least 60 city blocks (as the crow flies) from Broadway and&#xA;Broadway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running this matlab code. But i am not sure how to find probabilities even to the first question?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function random_walk_2d_simulation ( step_num, walk_num )&#xA;&#xA;%*****************************************************************************80&#xA;%&#xA;%% RANDOM_WALK_2D_SIMULATION simulates a random walk in 2D.&#xA;%&#xA;%  Discussion:&#xA;%&#xA;%    The expectation should be that, the average distance squared D^2 &#xA;%    is equal to the time, or number of steps N.&#xA;%&#xA;%    Or, equivalently&#xA;%&#xA;%      average ( D ) = sqrt ( N )&#xA;%&#xA;%    The program makes a plot of both the average and the maximum values&#xA;%    of D^2 versus time.  The maximum value grows much more quickly,&#xA;%    and that curve is much more jagged.&#xA;%&#xA;%  Licensing:&#xA;%&#xA;%    This code is distributed under the GNU LGPL license.&#xA;%&#xA;%  Modified:&#xA;%&#xA;%    03 November 2009&#xA;%&#xA;%  Author:&#xA;%&#xA;%    John Burkardt&#xA;%&#xA;%  Parameters:&#xA;%&#xA;%    Input, integer STEP_NUM, the number of steps to take in one test.&#xA;%&#xA;%    Input, integer WALK_NUM, the number of walks to take.&#xA;%&#xA;&#xA;%&#xA;%  Set up arrays for plotting.&#xA;%&#xA;  time = 0 : step_num;&#xA;  d2_ave = zeros(step_num+1,1);&#xA;  d2_max = zeros(step_num+1,1);&#xA;%&#xA;%  Take the walk WALK_NUM times.&#xA;%&#xA;  for walk = 1 : walk_num&#xA;&#xA;    x = zeros(step_num+1,1);&#xA;    y = zeros(step_num+1,1);&#xA;&#xA;   for step = 2 : step_num + 1&#xA;%&#xA;%  We are currently at ( X(STEP-1), Y(STEP-1) ).&#xA;%  Consider the four possible points to step to.&#xA;%&#xA;      destination = [ x(step-1) + 1.0, y(step-1); ...&#xA;                      x(step-1) - 1.0, y(step-1); ...&#xA;                      x(step-1),       y(step-1) + 1.0; ...&#xA;                      x(step-1),       y(step-1) - 1.0 ];&#xA;%&#xA;%  Choose destination 1, 2, 3 or 4.&#xA;%&#xA;      k = ceil ( 4.0 * rand );&#xA;%&#xA;%  Move there.&#xA;%&#xA;      x(step) = destination(k,1);&#xA;      y(step) = destination(k,2);&#xA;%&#xA;%  Update the sum of every particle's distance at step J.&#xA;%&#xA;      d2 = x(step)^2 + y(step)^2;&#xA;      d2_ave(step) = d2_ave(step) + d2;&#xA;      d2_max(step) = max ( d2_max(step), d2 );&#xA;&#xA;    end&#xA;&#xA;  end&#xA;%&#xA;%  Average the squared distance at each step over all walks.&#xA;%&#xA;  d2_ave(:,1) = d2_ave(:,1) / walk_num;&#xA;%&#xA;%  Make a plot.&#xA;%&#xA;  clf&#xA;  plot ( time, d2_ave, time, d2_max, 'LineWidth', 2 );&#xA;&#xA;  xlabel ( 'Time' )&#xA;  ylabel ( 'Distance squared' )&#xA;  title_string = sprintf ( '2D Random Walk Ave and Max - %d walks, %d steps', walk_num, step_num );&#xA;  title ( title_string );&#xA;&#xA;  return&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the plot of running the following command&#xA;random_walk_2d_simulation(60,10000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nLARF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nLARF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13767" LastEditorUserId="13767" LastEditDate="2015-10-30T14:12:45.850" LastActivityDate="2015-10-30T14:12:45.850" Title="crow probability problem" Tags="|probability|" AnswerCount="1" CommentCount="1" ClosedDate="2015-10-30T14:51:02.993" ContentLicense="CC BY-SA 3.0" />
  <row Id="8650" PostTypeId="2" ParentId="8638" CreationDate="2015-10-29T16:41:20.990" Score="1" Body="&lt;p&gt;You may use &lt;code&gt;ngram&lt;/code&gt;s to get the frequent sequences of you events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here a little example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(tau)&#xA; seq &amp;lt;- &quot;ababcdcde&quot;&#xA; textcnt(seq, method=&quot;ngram&quot;, n=3L, decreasing=TRUE)&#xA;&#xA;   _   a  ab   b   c  cd   d  _a _ab aba abc  ba bab  bc bcd cdc cde  dc dcd  de de_   e  e_ &#xA;   2   2   2   2   2   2   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Select the &lt;em&gt;longest&lt;/em&gt; and &lt;em&gt;most frequent&lt;/em&gt; ngram, here &lt;strong&gt;ab&lt;/strong&gt; and &lt;strong&gt;cd&lt;/strong&gt;.&#xA;You may also trade off the length and frequency to get maximal compression. The &lt;strong&gt;n&lt;/strong&gt; parameter limits the length of the ngram.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-10-29T16:41:20.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8651" PostTypeId="2" ParentId="8649" CreationDate="2015-10-29T19:01:21.140" Score="0" Body="&lt;p&gt;This is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_walk&quot; rel=&quot;nofollow&quot;&gt;random walk&lt;/a&gt; problem.  You could write a fairly simple random walk simulation which ends as soon as the tourist is 60 city blocks away from the starting location.  At any given time you only need to know the tourist's x,y coordinates.  Then each time-step, you pick a direction and increment either x or y.  You then need to keep track of distance from the origin (0,0).  Presumably this should be Euclidean distance, since it's all as the crow flies.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;decide direction to move&lt;/li&gt;&#xA;&lt;li&gt;update x,y and distance&lt;/li&gt;&#xA;&lt;li&gt;check if any of the distance requirements mentioned are met and store the timestep at which it occured&lt;/li&gt;&#xA;&lt;li&gt;if distance &gt; 60 blocks, terminate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-10-29T19:01:21.140" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="8652" PostTypeId="2" ParentId="8646" CreationDate="2015-10-29T19:21:27.530" Score="5" Body="&lt;p&gt;When you have an unbalanced data set, the algorithm is going to weight its success on each data point equally, meaning the majority class comes out as much more important than the minority class. The typical solution is to sample down the majority class until it's the same size as the minority class, and an alternate (similar) solution is to adjust the cost function so that the minority class is weighted appropriately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See these similar questions for more:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/810/should-i-go-for-a-balanced-dataset-or-a-representative-dataset/&quot;&gt;Should I go for a 'balanced' dataset or a 'representative' dataset?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/1107/quick-guide-into-training-highly-imbalanced-data-sets/&quot;&gt;Quick Guide into training highly imbalanced data sets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/454/what-are-the-implications-for-training-a-tree-ensemble-with-highly-biased-datase&quot;&gt;What are the implications for training a tree ensemble with highly biased datasets?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/736/skewed-multi-class-data/&quot;&gt;Skewed multi-class data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/6939/ratio-of-positive-to-negative-sample-in-data-set-for-best-classification/&quot;&gt;Ratio of positive to negative sample in data set for best classification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12876" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-10-29T19:21:27.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8654" PostTypeId="1" CreationDate="2015-10-29T22:02:46.530" Score="6" ViewCount="1348" Body="&lt;p&gt;My question is about creating &lt;a href=&quot;https://en.wikipedia.org/wiki/Classifier_chains&quot; rel=&quot;noreferrer&quot;&gt;classifier chains&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I am working with is a multiclass problem where I have to assign one of 7 possible classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started by training a single random forest based classifier for this purpose and the results were not that great. The expectation is about 95%+ for Precision, whereas I was at 79.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I started working with training 7 different classifiers for each class. When I take class 1, for the purpose of training, I convert the class labels of other classes to ‘Other’.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By doing this process, the performance of the individual classifiers registered ~ 95%+.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I thought, may be lets chain the classifiers (in a waterfall manner) in a way that the most accurate model gets first attempt at classification, the next most accurate gets left over rows to make prediction on, the third most accurate one goes after that so on and so forth. The training process was still the same(as for 7 individual classifiers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concretely, let us say classifier 1 predicts on 100 rows and finds 10 to be of Class 1, the other 90 will be ‘Other’. Classifier 2 will then do predictions on the remaining 90 which will be marked as ‘Other’. Classifier 3 will then go on the left over and so on and so forth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem is, the results are nowhere near the when they go individually. It is down to high 70s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am  trying to figure out what might be the reason for the steep decline in performance and wondering if this is expected or I need to modify something to achieve somewhat comparative performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate any input/suggestions/comments.&lt;/p&gt;&#xA;" OwnerUserId="13770" LastEditorUserId="11097" LastEditDate="2015-11-03T08:47:40.013" LastActivityDate="2016-07-30T15:25:04.523" Title="Classifier Chains" Tags="|machine-learning|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8655" PostTypeId="1" CreationDate="2015-10-30T01:03:04.533" Score="5" ViewCount="1433" Body="&lt;p&gt;I'm looking to use a tree kernel (e.g. Zelenko et al., 2003, Moschitti 2006, etc.) for text classification in Python. However, I'm still a little unclear on the implementation. Is there an existing implementation (in any language) that I could use as a reference, or is there a 3rd-party tree kernel library I could use?&lt;/p&gt;&#xA;" OwnerUserId="13772" LastEditorUserId="843" LastEditDate="2015-10-31T18:26:53.300" LastActivityDate="2020-08-03T10:26:29.840" Title="Implementation of Tree Kernels in Python" Tags="|machine-learning|python|statistics|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8656" PostTypeId="1" CreationDate="2015-10-30T02:49:08.590" Score="3" ViewCount="115" Body="&lt;p&gt;I've got sets of time series data collected from Weibo. It contains the number of posts under certain topics over a year. It can be found that from time to time there're bursts of discussion on certain topic. What's interesting is some burst die down quickly while others sustain on the agenda for quite a long time and die down gradually. How to predict and model the decays given a time series data. Also, I've got data of some exogenous data including time series data of links, pictures, and retweeting and original data in the community. Can I predict the duration of decay from these variables?&lt;/p&gt;&#xA;" OwnerUserId="13709" LastEditorUserId="13709" LastEditDate="2015-10-30T03:11:07.783" LastActivityDate="2015-10-30T17:40:08.800" Title="How to predict the duration of burst given several series" Tags="|bigdata|predictive-modeling|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8657" PostTypeId="1" AcceptedAnswerId="8663" CreationDate="2015-10-30T04:21:27.737" Score="7" ViewCount="19110" Body="&lt;p&gt;I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression.  As a first pass I'm just trying to do a binary classification on part of the iris data set.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One problem I think I have encountered is that the negative log-loss (computed with loss and stored in loss_vec) doesn't change much from one iteration to the next.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another challenge I am facing is trying to figure out how to plot the decision boundary once I have learned the logistic regression coefficients. Using the coefficients to plot the 0.5 decision boundary is way off.  This makes me think I have made a mistake somewhere&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/&quot; rel=&quot;noreferrer&quot;&gt;http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn import linear_model, datasets&#xA;&#xA;&#xA;iris = datasets.load_iris()&#xA;X = iris.data[:, :2]  # we only take the first two features.&#xA;Y = iris.target&#xA;&#xA;&#xA;X = X[:100,:]&#xA;Y = Y[:100]&#xA;&#xA;def phi(t):&#xA;    # logistic function, returns 1 / (1 + exp(-t))&#xA;    idx = t &amp;gt; 0&#xA;    out = np.empty(t.size, dtype=np.float)&#xA;    out[idx] = 1. / (1 + np.exp(-t[idx]))&#xA;    exp_t = np.exp(t[~idx])&#xA;    out[~idx] = exp_t / (1. + exp_t)&#xA;    return out&#xA;&#xA;&#xA;def loss(x0, X, y, alpha):&#xA;    # logistic loss function, returns Sum{-log(phi(t))}&#xA;    #x0 is the weight vector w are the paramaters, c is the bias term&#xA;    w, c = x0[:X.shape[1]], x0[-1]&#xA;    z = X.dot(w) + c&#xA;    yz = y * z&#xA;    idx = yz &amp;gt; 0&#xA;    out = np.zeros_like(yz)&#xA;    out[idx] = np.log(1 + np.exp(-yz[idx]))&#xA;    out[~idx] = (-yz[~idx] + np.log(1 + np.exp(yz[~idx])))&#xA;    out = out.sum() / X.shape[0] + .5 * alpha * w.dot(w)&#xA;    return out&#xA;&#xA;&#xA;def gradient(x0, X, y, alpha):&#xA;    # gradient of the logistic loss&#xA;    w, c = x0[:X.shape[1]], x0[-1]&#xA;    z = X.dot(w) + c&#xA;    z = phi(y * z)&#xA;    z0 = (z - 1) * y&#xA;    grad_w = X.T.dot(z0) / X.shape[0] + alpha * w&#xA;    grad_c = z0.sum() / X.shape[0]&#xA;    return np.concatenate((grad_w, [grad_c]))&#xA;&#xA;&#xA;def bgd(X, y, alpha, max_iter):&#xA;    step_sizes = np.array([100,10,1,.1,.01,.001,.0001,.00001])&#xA;    iter_no = 0&#xA;    x0 = np.random.random(X.shape[1] + 1) #initialize weight vector&#xA;&#xA;    #placeholder for coefficients to test against the loss function&#xA;    next_thetas = np.zeros((step_sizes.shape[0],X.shape[1]+1) )   &#xA;    J = loss(x0,X,y,alpha)&#xA;    running_loss = []&#xA;    while iter_no &amp;lt; max_iter:&#xA;        grad = gradient(x0,X,y,alpha)&#xA;        next_thetas = -(np.outer(step_sizes,grad)-x0)&#xA;        loss_vec = []&#xA;        for i in range(np.shape(next_thetas)[0]):&#xA;            loss_vec.append(loss(next_thetas[i],X,y,alpha))&#xA;        ind = np.argmin(loss_vec)&#xA;        x0 = next_thetas[ind]&#xA;        if iter_no % 500 == 0:&#xA;            running_loss.append(loss(x0,X,y,alpha))&#xA;        iter_no += 1  &#xA;    return next_thetas&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13775" LastEditorUserId="13775" LastEditDate="2015-10-30T17:28:51.990" LastActivityDate="2018-11-11T23:18:44.517" Title="Trying to understand Logistic Regression Implementation" Tags="|python|logistic-regression|gradient-descent|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8659" PostTypeId="2" ParentId="8240" CreationDate="2015-10-30T12:35:16.940" Score="1" Body="&lt;p&gt;So there are many ways to denote a series.  How are you going to parse the series down to determine the values if you don't know the format?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Determining if the label has a series does not get you to the specific numbers in the series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2,3,5,7  parses out to 4 numbers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is 6 in 1996?  I assume that is one number and 1996 != 6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;55,56,57&quot; is series with a 6 but but not the number 6 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does 7-9 parse out to 2 numbers or 3 number&lt;br&gt;&#xA;Is 6 in 7-9?&lt;br&gt;&#xA;If 6 is in 7-9 identifying that as a series does not answer that question.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many ways can there be represent a series that regex got out of hand?  For each format of series you also need to parse the values.  You need to know the format of the series to parse out the numbers.  I would have a set of regex mapped to set of parsers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe use machine learning to identity new series formats but you are still going to need to parse out the series. &lt;/p&gt;&#xA;" OwnerUserId="13285" LastActivityDate="2015-10-30T12:35:16.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8660" PostTypeId="1" CreationDate="2015-10-30T15:59:38.850" Score="10" ViewCount="825" Body="&lt;p&gt;I've just read Geoff Hinton's paper on transforming autoencoders &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hinton, Krizhevsky and Wang: &lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/transauto6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Transforming Auto-encoders&lt;/a&gt;. In Artificial Neural Networks and Machine Learning, 2011.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and would quite like to play around with something like this. But having read it I couldn't get enough detail from the paper on how I might actually implement it. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does anyone know how the mapping between input pixels to capsules should work? &lt;/li&gt;&#xA;&lt;li&gt;What exactly should be happening in the recognition units?&lt;/li&gt;&#xA;&lt;li&gt;How it should be trained? Is it just standard back prop between every connection?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Even better would be a link to some source code for this or something similar.&lt;/p&gt;&#xA;" OwnerUserId="13630" LastEditorUserId="26562" LastEditDate="2018-01-23T20:05:55.657" LastActivityDate="2018-01-23T23:01:34.960" Title="Transforming AutoEncoders" Tags="|neural-network|deep-learning|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8661" PostTypeId="2" ParentId="8656" CreationDate="2015-10-30T17:40:08.800" Score="1" Body="&lt;p&gt;I guess, a nice place to start is seminal work of John Kleinberg:  &lt;a href=&quot;https://www.cs.cornell.edu/home/kleinber/bhs.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.cs.cornell.edu/home/kleinber/bhs.pdf&lt;/a&gt;&#xA;Yes, this paper is outdated, but anyway it's still very influential and simple. Also, digging articles, citing this paper, could be very helpful. By the way, these guys solve very similar task ( and they even use Weibo logs as well!): &lt;a href=&quot;http://www.csce.uark.edu/~xintaowu/publ/pakdd15.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.csce.uark.edu/~xintaowu/publ/pakdd15.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13787" LastActivityDate="2015-10-30T17:40:08.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8662" PostTypeId="2" ParentId="8645" CreationDate="2015-10-30T17:41:10.997" Score="11" Body="&lt;p&gt;Looking at the source (seaborn/seaborn/categorical.py, line 2166), we find&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def barplot(x=None, y=None, hue=None, data=None, order=None, hue_order=None,&#xA;        estimator=np.mean, ci=95, n_boot=1000, units=None,&#xA;        orient=None, color=None, palette=None, saturation=.75,&#xA;        errcolor=&quot;.26&quot;, ax=None, **kwargs):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so the default value is, indeed, .95, as you guessed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: How CI is calculated: &lt;code&gt;barplot&lt;/code&gt; calls &lt;code&gt;utils.ci()&lt;/code&gt; which has&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;seaborn/seaborn/utils.py&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def ci(a, which=95, axis=None):&#xA;    &quot;&quot;&quot;Return a percentile range from an array of values.&quot;&quot;&quot;&#xA;    p = 50 - which / 2, 50 + which / 2&#xA;    return percentiles(a, p, axis)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this call to &lt;code&gt;percentiles()&lt;/code&gt; is calling:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def percentiles(a, pcts, axis=None):&#xA;    &quot;&quot;&quot;Like scoreatpercentile but can take and return array of percentiles.&#xA;    Parameters&#xA;    ----------&#xA;    a : array&#xA;        data&#xA;    pcts : sequence of percentile values&#xA;        percentile or percentiles to find score at&#xA;    axis : int or None&#xA;        if not None, computes scores over this axis&#xA;    Returns&#xA;    -------&#xA;    scores: array&#xA;        array of scores at requested percentiles&#xA;        first dimension is length of object passed to ``pcts``&#xA;    &quot;&quot;&quot;&#xA;    scores = []&#xA;    try:&#xA;        n = len(pcts)&#xA;    except TypeError:&#xA;        pcts = [pcts]&#xA;        n = 0&#xA;    for i, p in enumerate(pcts):&#xA;        if axis is None:&#xA;            score = stats.scoreatpercentile(a.ravel(), p)&#xA;        else:&#xA;            score = np.apply_along_axis(stats.scoreatpercentile, axis, a, p)&#xA;        scores.append(score)&#xA;    scores = np.asarray(scores)&#xA;    if not n:&#xA;        scores = scores.squeeze()&#xA;    return scores&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;axis=None&lt;/code&gt; so &lt;code&gt;score = stats.scoreatpercentile(a.ravel(), p)&lt;/code&gt; which is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scipy.stats.scoreatpercentile(a, per, limit=(), interpolation_method='fraction', axis=None)[source]&#xA;Calculate the score at a given percentile of the input sequence.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For example, the score at per=50 is the median. If the desired quantile lies between two data points, we interpolate between them, according to the value of interpolation. If the parameter limit is provided, it should be a tuple (lower, upper) of two values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Parameters: &#xA;a : array_like&#xA;A 1-D array of values from which to extract score.&#xA;per : array_like&#xA;Percentile(s) at which to extract score. Values should be in range [0,100].&#xA;limit : tuple, optional&#xA;Tuple of two scalars, the lower and upper limits within which to compute the percentile. Values of a outside this (closed) interval will be ignored.&#xA;interpolation_method : {‘fraction’, ‘lower’, ‘higher’}, optional&#xA;This optional parameter specifies the interpolation method to use, when the desired quantile lies between two data points i and j&#xA;fraction: i + (j - i) * fraction where fraction is the fractional part of the index surrounded by i and j.&#xA;lower: i.&#xA;higher: j.&#xA;axis : int, optional&#xA;Axis along which the percentiles are computed. Default is None. If None, compute over the whole array a.&#xA;Returns:    &#xA;score : float or ndarray&#xA;Score at percentile(s).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and looking in the source for &lt;strong&gt;scipy.stats.stats.py&lt;/strong&gt; we see the signature&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def scoreatpercentile(a, per, limit=(), interpolation_method='fraction',&#xA;                      axis=None):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so since seaboard calls it with no param for &lt;code&gt;interpolation&lt;/code&gt; it is using &lt;code&gt;fraction&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note, there is a warning of future obsolescence in &lt;code&gt;stats.scoreatpercentile()&lt;/code&gt;, namely &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This function will become obsolete in the future. For Numpy 1.9 and higher, numpy.percentile provides all the functionality that scoreatpercentile provides. And it’s significantly faster. Therefore it’s recommended to use numpy.percentile for users that have numpy &gt;= 1.9.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="10663" LastEditorUserId="10663" LastEditDate="2015-10-31T17:50:35.460" LastActivityDate="2015-10-31T17:50:35.460" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8663" PostTypeId="2" ParentId="8657" CreationDate="2015-10-30T18:07:44.627" Score="11" Body="&lt;p&gt;There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Primary takeaways&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: &lt;em&gt;Try to start from the math behind the model.&lt;/em&gt; The logistic regression is a relatively simple one. Find the two equations you need and stick to them, replicate them letter by letter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B: &lt;em&gt;Vectorize.&lt;/em&gt; It will save you a lot of unnecessary steps and computations, if you step back for a bit and think of the best vectorized implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C: &lt;em&gt;Write more comments in the code.&lt;/em&gt; It will help those trying to help you. It will also help you understand each part better and maybe uncover errors yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's go over the code step by step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. The sigmoid function&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a reason for such a complicate implementation in &lt;code&gt;phi(t)&lt;/code&gt;? Assuming that &lt;code&gt;t&lt;/code&gt; is a vector (a numpy array), then all you really need is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def phi(t):&#xA;   1. / (1. + np.exp(-t))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As &lt;code&gt;np.exp()&lt;/code&gt; operates element-wise over arrays. Ideally, I'd implement it as a function that can also return its derivative (not necessary here, but might be handy if you try to implement a basic neural net with sigmoid activations):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def phi(t, dt = False):&#xA;   if dt:&#xA;      phi(t) * (1. - phi(t))&#xA;   else:&#xA;      1. / (1. + np.exp(-t))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Cost function&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, the logistic cost function is defined as a log cost in the following way (vectorized): $ \frac{1}{m} (-(y^T \log{(\phi(X\theta))})-(1-y^T)(\log{(1 - \phi(X\theta))}) + \frac{\lambda}{2m} \theta^{1T}\theta $  where $\phi(z)$ is the logistic (sigmoid) function, $\theta$ is the full parameter vector (including bias weight), $\theta^1$ is parameter vector with $\theta_1=0$ (by convention, bias is not regularized) and $\lambda$ is the regularization parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I really don't understand is the part where you multiply &lt;code&gt;y * z&lt;/code&gt;. Assuming &lt;code&gt;y&lt;/code&gt; is your label vector $y$, why are you multiplying it with your &lt;code&gt;z&lt;/code&gt; before applying the sigmoid function? And why do you need to split the cost function into zeros and ones and calculate losses for either sample separately? &#xA;I think the problem in your code really lies in this part: you must be erroneously multiplying $y$ with $X\theta$ before applying $\phi(.)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, this bit here: &lt;code&gt;X.dot(w) + c&lt;/code&gt;. So &lt;code&gt;c&lt;/code&gt; is your bias parameter, right? Why are you adding it to every element of $X\theta$? It shouldn't be added - it should be the first element of the vector $X\theta$. Yes, you don't regularize it, but you need to use in the &quot;prediction&quot; part of the loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your code, I also see the cost function as being overly complicated. Here's what I would try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def loss(X,y,w,lam):&#xA;   #calculate &quot;prediction&quot;&#xA;   Z = np.dot(X,w)&#xA;   #calculate cost without regularization&#xA;   #shape of X is (m,n), shape of w is (n,1)&#xA;   J = (1./len(X)) * (-np.dot(y.T, np.log(phi(Z))) * np.dot((1-y.T),np.log(1 - phi(Z))))&#xA;   #add regularization&#xA;   #temporary weight vector&#xA;   w1 = copy.copy(w) #import copy to create a true copy&#xA;   w1[0] = 0&#xA;   J += (lam/(2.*len(X))) * np.dot(w1.T, w)&#xA;   return J&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Gradient&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, let's first go over the formula for the gradient of the logistic loss, again, vectorized: $\frac{1}{m} ((\phi(X\theta) - y)^TX)^T + \frac{\lambda}{m}\theta^1$.&#xA;This will return a vector of derivatives (i.e. gradient) for all parameters, regularized properly (without the bias term regularized).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here again, you've multiplied by $y$ way too soon: &lt;code&gt;phi(y * z)&lt;/code&gt;. In fact, you shouldn't have multiplied by $y$ in gradient at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I would do for the gradient:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gradient(X, y, w, lam):&#xA;   #calculate the prediction&#xA;   Z = np.dot(X,w)&#xA;   #temporary weight vector&#xA;   w1 = copy.copy(w) #import copy to create a true copy&#xA;   w1[0] = 0&#xA;   #calc gradient&#xA;   grad = (1./len(X)) * (np.dot((phi(Z) - y).T,X).T) + (lam/len(X)) * w1&#xA;   return grad&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The actual gradient descent implementation seems ok to me, but because there are errors in the gradient and cost function implementations, it fails to deliver :/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this will help you get on track.&lt;/p&gt;&#xA;" OwnerUserId="12580" LastActivityDate="2015-10-30T18:07:44.627" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8664" PostTypeId="1" CreationDate="2015-10-30T23:10:31.113" Score="1" ViewCount="120" Body="&lt;p&gt;I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern.&#xA;I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?&lt;/p&gt;&#xA;" OwnerUserId="7848" LastActivityDate="2015-11-03T15:03:29.300" Title="Is this a correct way improving a statistical model?" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8665" PostTypeId="1" CreationDate="2015-10-31T00:28:41.197" Score="1" ViewCount="255" Body="&lt;p&gt;Many Universities now offer Masters Degrees in data science; I'm curious as to people's opinions on whether a traditional master of statistics is a better to fit for working in the field of in data science.&lt;/p&gt;&#xA;" OwnerUserId="12611" LastEditorUserId="13413" LastEditDate="2015-10-31T18:26:46.040" LastActivityDate="2015-10-31T18:26:46.040" Title="Master of Statistics or Master of Data Science" Tags="|statistics|education|" AnswerCount="2" CommentCount="2" ClosedDate="2015-10-31T18:27:07.097" ContentLicense="CC BY-SA 3.0" />
  <row Id="8666" PostTypeId="2" ParentId="8665" CreationDate="2015-10-31T01:57:54.070" Score="0" Body="&lt;p&gt;As everyone here knows, data science as a field is broad, and similarly the experience required for a particular job is broad. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on where you want to end up, either degree could suit you better. If you would like more of a traditional statistician's job which has been rebranded to data scientist in order to gain more attention, then perhaps the MSc Stats would be more applicable. However, if your field requires more complex computer programming or database management, you may want to look into the masters in data science or masters in computer science, depending on your background and interest. Going with a &quot;hot&quot; degree like data science means that you will likely learn a lot of common analyses in superficial detail in order to be employable quickly, but with a more traditionally academic MSc Stats or MSc Com sci, you will learn the theoretical basis with the potential loss of applicability. My personal bias is towards to academic side (MSc/PhD Stats or applied math), but I think this is a decision you should think hard about. Regardless of your decision, in general data science / analysis / programming / whatever this field is requires quite a lot of self directed learning, so whatever program you choose &lt;strong&gt;must&lt;/strong&gt; provide you with the skills and tools to be able to teach yourself whatever you need to know.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would personally take a hard look at what specific kinds of jobs appeal to you, and which programs may prepare you better. Talk to the admissions officers and look up the career paths of past students. Where did they end up? Is that where you want to be? Keep in mind the kind of life you want to live and the kind of work you want to do. I personally find anything financially or profit driven absolutely soul-sucking, but enjoying doing almost the same work in the medical field. Different skill sets, different programming languages, and different vocabulary are required, and the more you know before you enter the field the better you can prepare yourself for the the journey to come.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, have fun, and don't worry too much about making mistakes. No one knows how things will work out in the long run. You can prepare yourself with as much knowledge as possible, but you'll never know which was the truly right choice, just which one was your choice. &lt;/p&gt;&#xA;" OwnerUserId="9131" LastActivityDate="2015-10-31T01:57:54.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8667" PostTypeId="1" AcceptedAnswerId="8679" CreationDate="2015-10-31T05:21:03.620" Score="8" ViewCount="1109" Body="&lt;p&gt;Suppose I have a data set : &lt;code&gt;Amount of money (100, 50, 150, 200, 35, 60 ,50, 20, 500)&lt;/code&gt;. I have &lt;a href=&quot;https://www.google.co.zw/webhp?sourceid=chrome-instant&amp;amp;ion=1&amp;amp;espv=2&amp;amp;ie=UTF-8#q=Outlier+detection+Algorithms&quot;&gt;Googled&lt;/a&gt; the web looking for techniques that can be used to find a possible outlier in this data set but I ended up confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is&lt;/strong&gt;: Which algorithms, techniques or methods can be used to detect possible outlier in this data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS&lt;/strong&gt;:Consider that the data does not follow a normal distribution. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="13132" LastActivityDate="2015-11-02T07:22:33.490" Title="Which algorithms or methods can be used to detect an outlier from this data set?" Tags="|algorithms|outlier|" AnswerCount="3" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8668" PostTypeId="2" ParentId="8665" CreationDate="2015-10-31T09:28:22.000" Score="0" Body="&lt;p&gt;I'd probably do stats; but that's because it's the choice I made ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your undergrad was in pure math, maybe doing data science would be a good bet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise doing MSc stats is a good way to get better at math. Getting better at math is way more difficult to do outside of academia than getting better at programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It partly depends what kind of data scientist you're aiming to be and what kind of work you want to do. If you're leaning towards doing visualisations rather than modelling the data science programmes might be more relvevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say beware of a focus on learning particular computer tools as this is a rapidly changing area and everything could be different in five years.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-10-31T09:28:22.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8671" PostTypeId="1" CreationDate="2015-10-31T13:35:09.233" Score="1" ViewCount="92" Body="&lt;p&gt;I am using $\chi^{2}$ kernel for non-linear SVM (using libSVM) for classifying MNIST digits. I am getting very bad performance (worse than random guessing). The $\chi^{2}$ kernel code (in MATLAB) is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function D = chi2_kernel(X,Y)&#xA;%Computes the chi2 kernel for two Matrices X and Y&#xA;&#xA; D = zeros(size(X,1),size(Y,1));&#xA;&#xA;for i = 1:size(Y,1)&#xA;&#xA;    d = bsxfun(@minus,X, Y(i,:));&#xA;    s = bsxfun(@plus,X,Y(i,:));&#xA;    D(:,i) = sum(d.^2 ./ (s/2),2);&#xA;end&#xA;&#xA;D = exp(-0.0001.*(1-D));&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this I computed the kernel matrix as follows :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% Gram matrix for chi2 kernel&#xA; dist = chi2_kernel(X_train2,X_train2);&#xA;dist1 = chi2_kernel(X_test,X_train2);&#xA;&#xA;% Gram matrix for training set to training set&#xA;&#xA;&#xA;KTrT = [(1:numTrain/50)', dist];&#xA;&#xA;% Gram matrix for training to test set &#xA;&#xA;KTT = [(1:numTest)', dist1];&#xA;&#xA;&#xA;model = svmtrain(Y_train2, KTrT,'-t 4');&#xA;&#xA;predClass = svmpredict(Y_test, KTT,model);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone have suggestions for how to better optimize this code, or an explanation for why I'm observing such poor performance?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="13413" LastEditDate="2015-10-31T13:52:31.003" LastActivityDate="2015-10-31T13:52:31.003" Title="$\chi^{2}$ kernel SVM performance issue" Tags="|svm|matlab|libsvm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8673" PostTypeId="5" CreationDate="2015-10-31T16:59:23.013" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-10-31T16:59:23.013" LastActivityDate="2015-10-31T16:59:23.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8674" PostTypeId="4" CreationDate="2015-10-31T16:59:23.013" Score="0" Body="Autoencoders are a type of neural network that learns a useful encoding for data in an unsupervised manner." OwnerUserId="843" LastEditorUserId="38887" LastEditDate="2019-12-05T19:24:59.120" LastActivityDate="2019-12-05T19:24:59.120" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8675" PostTypeId="2" ParentId="8667" CreationDate="2015-10-31T18:51:06.720" Score="3" Body="&lt;p&gt;A simple approach would be using the same thing as box plots does: away than 1.5 (median-q1) or 1.5 (q3-median) = outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find it useful in lots of cases even it not perfect and maybe too simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has the advantage to not suppose normality.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2015-10-31T18:51:06.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8677" PostTypeId="2" ParentId="8264" CreationDate="2015-11-01T11:04:00.170" Score="4" Body="&lt;h2&gt;There is no one-size-fits-all&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Also, data dimensionality is not the same everywhere. Text data, which is inherently sparse, has a very different &lt;strong&gt;intrinsic dimensionality&lt;/strong&gt; than e.g. random Gaussians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For text data, linear SVMs are known to work very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RBF kernels do not work well with high-dimensional data, because they are distance-based at the core, and choosing the sigma parameter becomes next to impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can &quot;fold&quot; dimensions, you also get very different behavior. Im image recognition, you usually have thousands of pixels. However, you &lt;em&gt;never look at all of them at once&lt;/em&gt;. Instead, you use convolutional kernels that move over the data space, and they may have only say 32x32 pixels. That is still 1024 dimensions, but not millions anymore.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-11-01T11:04:00.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8679" PostTypeId="2" ParentId="8667" CreationDate="2015-11-01T14:50:27.320" Score="5" Body="&lt;p&gt;You can use BoxPlot for outlier analysis. I would show you how to do that in Python:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider your data as an array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [100, 50, 150, 200, 35, 60 ,50, 20, 500]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, use seaborn to plot the boxplot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sn&#xA;sn.boxplot(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, you would get a plot which looks somewhat like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cQv7b.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cQv7b.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems like 500 is the only outlier to me.  But, it all depends on the analysis and the tolerance level of the analyst or the statistician and also the problem statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have a look at &lt;a href=&quot;https://stats.stackexchange.com/a/178106/84191&quot;&gt;one of my answers&lt;/a&gt; on the CrossValidated SE for more tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there are several nice questions on outliers and the algorithms and techniques for detecting them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My personal favourite is the &lt;a href=&quot;https://stackoverflow.com/questions/9935271/one-dimensional-mahalanobis-distance-in-python&quot;&gt;Mahalanobis distance technique&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-11-02T07:22:33.490" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8680" PostTypeId="2" ParentId="8667" CreationDate="2015-11-01T20:12:32.123" Score="5" Body="&lt;p&gt;One way of thinking of outlier detection is that you're creating a predictive model, then you're checking to see if a point falls within the range of predictions.  From an information-theoretic point of view, you can see how much each observation increases the entropy of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are treating this data as just a collection of numbers, and you don't have some proposed model for how they're generated, you might as well just look at the average.  If you're certain the numbers aren't normally distributed, you can't make statements as to how far 'off' a given number is from the average, but you can just look at it in absolute terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying this, you can take the average of all the numbers, then exclude each number and take the average of the others.  Whichever average is most different from the global average is the biggest outlier.  Here's some python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def avg(a):&#xA;    return sum(a)/len(a)&#xA;&#xA;l = [100, 50, 150, 200, 35, 60 ,50, 20, 500]&#xA;m = avg(l)&#xA;for idx in range(len(l)):&#xA;    print(&quot;outlier score of {0}: {1}&quot;.format(l[idx], abs(m - avg([elem for i, elem in enumerate(l) if i!=idx]))))&#xA;&amp;gt;&amp;gt;&#xA;outlier score of 100: 4&#xA;outlier score of 50: 10&#xA;outlier score of 150: 3&#xA;outlier score of 200: 9&#xA;outlier score of 35: 12&#xA;outlier score of 60: 9&#xA;outlier score of 50: 10&#xA;outlier score of 20: 14&#xA;outlier score of 500: 46 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12626" LastActivityDate="2015-11-01T20:12:32.123" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8681" PostTypeId="1" CreationDate="2015-11-02T04:12:53.367" Score="11" ViewCount="28036" Body="&lt;p&gt;My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level. &lt;a href=&quot;https://i.stack.imgur.com/3jIFZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3jIFZ.png&quot; alt=&quot;Snapshot of data&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Data is of shape (n=7219, p=105).&lt;/p&gt;&#xA;&lt;p&gt;Couple things:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12533" LastEditorUserId="12533" LastEditDate="2020-07-12T05:50:07.547" LastActivityDate="2020-07-12T05:50:07.547" Title="Clustering for mixed numeric and nominal discrete data" Tags="|clustering|k-means|scikit-learn|categorical-data|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="8682" PostTypeId="2" ParentId="8642" CreationDate="2015-11-02T05:15:53.350" Score="3" Body="&lt;p&gt;The objective is rather different.&#xA;For classification problems, we want to know: GIVEN X, WHAT WILL BE Y?&#xA;For concept-learning problems, we are asking: WHAT KIND OF X WILL GIVE US Y?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may seem similar, but for classification problems, we may use a model that is unable to generalize and answer &quot;WHAT KIND OF X WILL GIVE US Y&quot; until you plug X into the model (e.g. K-nearest neighbor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, concept-learning problems, the main issue is to generalize and answer what can be X specifically.&lt;/p&gt;&#xA;" OwnerUserId="13818" LastActivityDate="2015-11-02T05:15:53.350" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8683" PostTypeId="2" ParentId="5479" CreationDate="2015-11-02T05:25:24.690" Score="0" Body="&lt;p&gt;I think the dataset is generated using dynamic network generator made available at &lt;a href=&quot;http://mlg.ucd.ie/snam/&quot; rel=&quot;nofollow&quot;&gt;this address&lt;/a&gt; which is dynamic version of static LFR. It's an excellent dynamic networked data generator.&lt;/p&gt;&#xA;" OwnerUserId="13476" LastEditorUserId="21" LastEditDate="2015-11-02T20:46:53.037" LastActivityDate="2015-11-02T20:46:53.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8684" PostTypeId="2" ParentId="8641" CreationDate="2015-11-02T05:38:00.973" Score="1" Body="&lt;p&gt;Data Scientist differs from Data Engineer as it is &quot;science vs engineering&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scientific methods (such as statistics, validation, experimental design, etc) is what separates data scientists from data engineers. While the program skills are still essential, it is the &quot;tool&quot; for solving problem instead of &quot;problem solving&quot; itself - we use programming tools to gather, process and analyze data but it is the scientific methods that solidify the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you to learn more about statistics. I was a computer science graduate as well and I personally learn statistics from Udacity (and other sources afterwards). I feel it's a nice place to start.&lt;/p&gt;&#xA;" OwnerUserId="13818" LastActivityDate="2015-11-02T05:38:00.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8685" PostTypeId="2" ParentId="8070" CreationDate="2015-11-02T05:58:23.197" Score="1" Body="&lt;p&gt;X and Y, Size, Texture, Shape, Orientation, Color are some common encodings for data visualization. Some of them have implied orders (X and Y, Size). In your case, I feel that you may use an unordered encoding such as color.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYR. The following website have a rather comprehensive explanation.&#xA;&lt;a href=&quot;https://www.safaribooksonline.com/library/view/designing-data-visualizations/9781449314774/ch04.html&quot; rel=&quot;nofollow&quot;&gt;https://www.safaribooksonline.com/library/view/designing-data-visualizations/9781449314774/ch04.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13818" LastActivityDate="2015-11-02T05:58:23.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8686" PostTypeId="1" CreationDate="2015-11-02T06:36:09.557" Score="2" ViewCount="1331" Body="&lt;p&gt;Which all are the equivalent or advanced libraries in Python for building recommendation systems like &lt;a href=&quot;https://mahout.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Mahout&lt;/a&gt; for  Collaborative Filtering and Content Based Filtering ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also is there a way to integrate Mahout with Python?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2020-08-03T10:27:19.217" Title="Building Recommendation engine with Python" Tags="|python|apache-mahout|software-recommendation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8688" PostTypeId="1" CreationDate="2015-11-02T10:29:59.463" Score="4" ViewCount="1902" Body="&lt;p&gt;I am trying to visualize a dataset that registers the career of people in an organisation and their backgrounds before starting in that particular organisation. I 'd like to show how individuals are positioned in the space based on their background areas and show the evolution through time in the beginning of each decade. I have seen the answers &lt;a href=&quot;https://datascience.stackexchange.com/questions/8070/how-to-visualize-categorical-data-over-time&quot;&gt;here&lt;/a&gt; but my data are much more categorical and also, ideally, &lt;em&gt;I 'd like to display all three background categories (see below) at the same time&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample of my data follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Person  BkgArea1    BkgArea2    BkgArea3    TotalSize   YearFrom    YearTo&#xA;P1  3   2   0   5   1959    1965&#xA;P2  0   2   1   3   1959    1967&#xA;P3  3   0   2   5   1991    2008&#xA;P4  3   0   0   3   1959    1980&#xA;P5  3   0   0   3   1981    1998&#xA;P6  0   2   1   3   2006    2014&#xA;P7  2   2   2   6   2008    2014&#xA;P8  0   0   3   3   1991    1996&#xA;P9  3   3   1   7   1966    1972&#xA;P10 3   0   2   5   1975    1991&#xA;P11 3   2   0   5   1998    2013&#xA;P12 3   1   0   4   2004    2013&#xA;P13 0   2   0   2   1998    2010&#xA;P14 0   3   0   3   2003    2008&#xA;P15 1   2   3   6   1998    2008&#xA;P16 3   0   2   5   1973    1977&#xA;P17 1   3   0   4   1998    2012&#xA;P18 3   0   2   5   1996    2008&#xA;P19 0   3   1   4   1998    2011&#xA;P20 3   2   0   5   1998    2006&#xA;P21 3   0   0   3   1986    1989&#xA;P22 0   3   2   5   1996    2014&#xA;P23 3   2   3   8   1959    1976&#xA;P24 3   2   0   5   1998    2001&#xA;P25 0   0   3   3   1998    2011&#xA;P26 3   3   2   8   1965    1992&#xA;P27 2   3   2   7   2010    2014&#xA;P28 3   2   2   7   1986    1998&#xA;P29 3   3   0   6   2013    2014&#xA;P30 0   3   2   5   1976    1977&#xA;P31 3   3   0   6   1986    1987&#xA;P32 0   2   2   4   1980    1990&#xA;P33 3   0   2   5   1975    1986&#xA;P34 0   3   3   6   1977    1991&#xA;P35 3   2   2   7   1963    1974&#xA;P36 3   3   0   6   1998    2001&#xA;P37 0   2   3   5   1998    2004&#xA;P38 0   3   2   5   1974    1980&#xA;P39 3   2   3   8   1989    1998&#xA;P40 0   3   3   6   1991    1998&#xA;P41 0   1   2   3   1998    2001&#xA;P42 0   2   0   2   2003    2012&#xA;P43 3   3   3   9   1973    1986&#xA;P44 3   2   1   6   1978    1986&#xA;P45 0   2   0   2   2002    2012&#xA;P46 0   3   3   6   1982    1988&#xA;P47 0   3   0   3   1992    1998&#xA;P48 3   2   2   7   1998    2004&#xA;P49 2   2   0   4   2012    2014&#xA;P50 3   2   2   7   2008    2014&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The values in columns &lt;code&gt;BkgArea1-3&lt;/code&gt; are essentially categorical values, that indicate the strength of an individual in a specific category. It is thus&#xA;&lt;code&gt;0: no background, 1: weak background, 2: average background, 3: strong background&lt;/code&gt;. The &lt;code&gt;TotalSize&lt;/code&gt; column is the sum of the scores of the individuals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The initial idea was to assign basic colors to BkgAreas 1-3, namely Red, Green, Blue, and then colour the individuals according to the mixture of their backgrounds, weighing each color with the strength of each area, i.e. &lt;code&gt;2/3 * Red&lt;/code&gt; if an individual has score 2 in BkgArea1, etc. I then tried to display the area as 3d scatter plot but it looks very complicated and hard to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried to apply &lt;em&gt;Multiple Correspondence Analysis&lt;/em&gt; but I haven't got very satisfying results. Do you have any ideas as of how to represent this dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After people having posted various useful answers but not 100% covering the representation needs, I summarize what I would like to achieve:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Group/cluster the individuals according to their backgrounds and relative to the three background areas. if someone has scores in more than one of them, they should be displayed somewhere in the middle.&lt;/li&gt;&#xA;&lt;li&gt;Show, visually if possible, the number of individuals and the &lt;code&gt;TotalSize&lt;/code&gt; of the individuals. This could be a cloud &lt;/li&gt;&#xA;&lt;li&gt;Show the development of the organization. I think this might be easy to do with stacked column charts or histograms&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/visualization.html#visualization-radviz&quot; rel=&quot;nofollow noreferrer&quot;&gt;RadViz&lt;/a&gt; can be a method to solve 1 and 2, or at least I am inclined toward that representation. The problem in my data is that individuals under RadViz will be placed at very specific points so we lose somewhat the track of how many individuals have specific background characteristics&lt;/p&gt;&#xA;" OwnerUserId="13778" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-11-08T15:22:56.140" Title="How to visualise multidimensional categorical data with additional time dimension" Tags="|r|python|visualization|categorical-data|tableau|" AnswerCount="4" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8689" PostTypeId="2" ParentId="8686" CreationDate="2015-11-02T13:15:27.973" Score="1" Body="&lt;p&gt;I don't know of a well documented python option for collaborative filtering, but one option is to use &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;Spark's MLib&lt;/a&gt;. There is a wrapper that allows you to run it in python (&lt;a href=&quot;https://spark.apache.org/docs/0.9.1/python-programming-guide.html&quot; rel=&quot;nofollow&quot;&gt;pyspark&lt;/a&gt;) and it is &lt;a href=&quot;http://spark.apache.org/docs/latest/spark-standalone.html#spark-standalone-mode&quot; rel=&quot;nofollow&quot;&gt;straightforward&lt;/a&gt; to get it running on your local machine. &lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-02T14:28:29.870" LastActivityDate="2015-11-02T14:28:29.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8693" PostTypeId="2" ParentId="8688" CreationDate="2015-11-02T14:03:03.140" Score="1" Body="&lt;p&gt;There might be a couple of different things you could do.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(data.table)&#xA;d &amp;lt;- data.table(read.csv('/tmp/x.csv', sep='\t', quote='', header=T, na.strings=''))&#xA;# bucket start date to decade&#xA;d[, start.decade:=10*floor(YearFrom/10)]&#xA;# create quantiles to represent the inner quartile range&#xA;xy.plot &amp;lt;- d[, .(bg1.lower=quantile(BkgArea1, .25),&#xA;       bg1.upper=quantile(BkgArea1, .75),&#xA;       bg2.lower=quantile(BkgArea2, .25), &#xA;       bg2.upper=quantile(BkgArea2, .75), &#xA;       bg3.lower=quantile(BkgArea3, .25), &#xA;       bg3.upper=quantile(BkgArea3, .75)), by=start.decade]&#xA;plot(xy.plot$start.decade, xy.plot$bg1.lower, col='red', type='l')&#xA;lines(xy.plot$start.decade, xy.plot$bg1.upper, col='red')&#xA;lines(xy.plot$start.decade, xy.plot$bg2.lower, col='green')&#xA;lines(xy.plot$start.decade, xy.plot$bg2.upper, col='green')&#xA;lines(xy.plot$start.decade, xy.plot$bg3.lower, col='blue')&#xA;lines(xy.plot$start.decade, xy.plot$bg3.upper, col='blue')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RKB2Q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RKB2Q.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is a little messy. You could just show one pair at a time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mosaic plots are nice for categorical data. The relative area is the density of users in the cross-sectional pair. Changing the numbers to text would make it more clear.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mosaicplot(BkgArea1 ~ BkgArea3 + BkgArea2, d[start.decade==2010])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0UU2p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0UU2p.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could have one per decade or visualize the total column against the decade, depending on what you want.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-02T14:03:03.140" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8694" PostTypeId="1" AcceptedAnswerId="12474" CreationDate="2015-11-02T15:42:45.377" Score="6" ViewCount="1676" Body="&lt;p&gt;I frequently use Random Forest, Regularized Random Forest, Guided Random Forest, and similar tree models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The size of the data that I'm dealing with has grown beyond what I can work around using HPC and parallelism. It's typically large due to row length (observations) not columns (features). The data is also often not normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to make a choice between: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Running a small number of trees (i.e. 50 or less) with either complete data or a relatively large and comparative sample&lt;/li&gt;&#xA;&lt;li&gt;Running several times the number of trees, but with a correspondingly scaled down sample size&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are work-arounds and for any 1 case -- for instance, I can do some ad hoc tests to see which I think will work better, but what I'm wondering &lt;strong&gt;&lt;em&gt;is if there is a good theoretical (or robust empirical) reasoning&lt;/em&gt;&lt;/strong&gt; to either guide the choice of approach over the other or to describe the tradeoff being made?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, I'm hoping that someone more comfortable with the math, statistics, and theory underlying this (type of) algorithm can offer some generalizable insight.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2015-11-03T14:33:36.903" LastActivityDate="2016-06-28T18:05:32.680" Title="Random Forests with Big Data - number of trees v. number of observations" Tags="|bigdata|random-forest|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8695" PostTypeId="2" ParentId="8686" CreationDate="2015-11-02T22:05:06.603" Score="1" Body="&lt;p&gt;Have you considered using &lt;a href=&quot;http://dato.com/products/create/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GraphLab Create&lt;/a&gt;? It's a Python tool for doing many different types of Machine Learning.&lt;/p&gt;&#xA;&lt;p&gt;General info about their Recommender System can be found &lt;a href=&quot;https://dato.com/solutions/use-cases/recommendation-engine.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;A hands-on coding example can be found &lt;a href=&quot;https://dato.com/learn/gallery/notebooks/basic_recommender_functionalities.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Using this product you should be able to create a Recommender System in as few as five lines of code.&lt;/p&gt;&#xA;&lt;p&gt;One downside, if you're using it for a commercial purpose, you'll need to get a license for the product. Although they do have a free trial period.&lt;/p&gt;&#xA;" OwnerUserId="13853" LastEditorUserId="98307" LastEditDate="2020-08-03T10:27:19.217" LastActivityDate="2020-08-03T10:27:19.217" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8696" PostTypeId="2" ParentId="8654" CreationDate="2015-11-03T02:21:53.237" Score="1" Body="&lt;p&gt;Try running &lt;em&gt;all&lt;/em&gt; of the models for each of the predictions. You will need get the probability of the prediction. Take as the classification the individual model which gives the maximum probability. Using your proposed &quot;waterfall&quot; does not allow the other models to have a say in the prediction of each new input. Also, are you saying that when you switched form one model with 7 classes to 7 models with 2 classes each the accuracy went from ~70 to ~95? That is interesting. Maybe you should look to see which features are the strongest predictors in each of your binary models (see variable importance).&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-03T02:21:53.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8697" PostTypeId="1" CreationDate="2015-11-03T06:26:49.093" Score="1" ViewCount="36" Body="&lt;p&gt;In machine learning, one can use Euclidean distance to measure a cluster&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mu\in R^k,&#xA;$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;over data points &#xA;$$&#xA;\{x_i\}^N_{i=1}\in R^{k\times N},&#xA;$$ &#xA;with the measure &#xA;$$&#xA;\text{error}_i = ||\mu-x_i||^2_2&#xA;$$&#xA;The total error can be calculated as&#xA;$$&#xA;\text{error} = \sum^N_{i=1}error_i&#xA;$$ &#xA;To estimate the parameters, one formulates the optimization problem&#xA;$$&#xA;\min_{\mu_{new}} \big{(}\text{error}\big{)}&#xA;$$&#xA;This has two underlying assumptions: first, that samples are independent, and, second, that the dimensions are independent for each sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain how I should analyze when either or both assumptions are not satisfied?&lt;/p&gt;&#xA;" OwnerUserId="9893" LastEditorUserId="1330" LastEditDate="2020-05-25T19:03:54.253" LastActivityDate="2020-05-25T19:03:54.253" Title="How to cluster when samples are independent and/or dimensions are not independent for each sample?" Tags="|machine-learning|data-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8698" PostTypeId="1" AcceptedAnswerId="15620" CreationDate="2015-11-03T13:09:48.553" Score="1" ViewCount="5766" Body="&lt;p&gt;I am occuring a strange &lt;code&gt;hive&lt;/code&gt;-client &lt;code&gt;beeline&lt;/code&gt; behavior. &#xA;In the outputed file with query results there is also a queary at the beggining and at the end. Is there any option to disable such behavior? I can't see such option in the &lt;code&gt;beeline -help&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-bash-4.2$ beeline -help&#xA;Usage: java org.apache.hive.cli.beeline.BeeLine &#xA;   -u &amp;lt;database url&amp;gt;               the JDBC URL to connect to&#xA;   -n &amp;lt;username&amp;gt;                   the username to connect as&#xA;   -p &amp;lt;password&amp;gt;                   the password to connect as&#xA;   -d &amp;lt;driver class&amp;gt;               the driver class to use&#xA;   -i &amp;lt;init file&amp;gt;                  script file for initialization&#xA;   -e &amp;lt;query&amp;gt;                      query that should be executed&#xA;   -f &amp;lt;exec file&amp;gt;                  script file that should be executed&#xA;   --hiveconf property=value       Use value for given property&#xA;   --hivevar name=value            hive variable name and value&#xA;                                   This is Hive specific settings in which variables&#xA;                                   can be set at session level and referenced in Hive&#xA;                                   commands or queries.&#xA;   --color=[true/false]            control whether color is used for display&#xA;   --showHeader=[true/false]       show column names in query results&#xA;   --headerInterval=ROWS;          the interval between which heades are displayed&#xA;   --fastConnect=[true/false]      skip building table/column list for tab-completion&#xA;   --autoCommit=[true/false]       enable/disable automatic transaction commit&#xA;   --verbose=[true/false]          show verbose error messages and debug info&#xA;   --showWarnings=[true/false]     display connection warnings&#xA;   --showNestedErrs=[true/false]   display nested errors&#xA;   --numberFormat=[pattern]        format numbers using DecimalFormat pattern&#xA;   --force=[true/false]            continue running script even after errors&#xA;   --maxWidth=MAXWIDTH             the maximum width of the terminal&#xA;   --maxColumnWidth=MAXCOLWIDTH    the maximum width to use when displaying columns&#xA;   --silent=[true/false]           be more silent&#xA;   --autosave=[true/false]         automatically save preferences&#xA;   --outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv]  format mode for result display&#xA;                                   Note that csv, and tsv are deprecated - use csv2, tsv2 instead&#xA;  --truncateTable=[true/false]    truncate table column when it exceeds length&#xA;   --delimiterForDSV=DELIMITER     specify the delimiter for delimiter-separated values output format (default: |)&#xA;   --isolation=LEVEL               set the transaction isolation level&#xA;   --nullemptystring=[true/false]  set to true to get historic behavior of printing null as empty string&#xA;   --help                          display this message&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5224" LastActivityDate="2024-03-26T09:29:47.480" Title="how to disable query from beeline results" Tags="|apache-hadoop|hive|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8699" PostTypeId="1" CreationDate="2015-11-03T14:46:10.123" Score="2" ViewCount="319" Body="&lt;p&gt;I am using decision tree in Weka and I have some continuous data, so when I use Weka it automatically find the threshold for me but for some reason I want to implement Decision Tree by myself so I need to know what approach to use to find the threshold to discretize my continuous data?  &lt;/p&gt;&#xA;" OwnerUserId="11072" LastEditorUserId="1330" LastEditDate="2020-05-25T19:00:31.383" LastActivityDate="2020-05-25T19:00:31.383" Title="Method of finding threshold in Decision tree for continuous data" Tags="|machine-learning|weka|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8700" PostTypeId="2" ParentId="8664" CreationDate="2015-11-03T15:03:29.300" Score="1" Body="&lt;p&gt;No offense, but let me guess, you work for a large company where there is no person with a full understanding of the process overseeing this? I say this because it sounds as if the testers want you to manually and intentionally over-fit your models. There is no such thing as a perfectly accurate model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, I would say that there is bias in your model building. New model input should be a random selection not by finding and labeling similar instances to ones that were misclassified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your production model will degrade over time (the distribution in the population has changed). You should have a sense of a threshold, an error rate at which is no longer exceptable and you will need to retrain. To retrain, you could add a random selection of new labeled data to your existing data, then train, validate, select model, deploy, and monitor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may &quot;fix&quot; an error report with a new retraining, but it is an unrealistic expectation that this will also cover all previous error reports. Your model needs to generalize. See here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Test_set&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Test_set&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the image at the bottom. In essence you are being asked to create the orange model were you want to create the green model, which generalizes better to unseen data. &lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-03T15:03:29.300" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8701" PostTypeId="1" CreationDate="2015-11-03T16:24:33.707" Score="1" ViewCount="36" Body="&lt;p&gt;I've modified the &lt;code&gt;Blocks&lt;/code&gt; &lt;a href=&quot;http://blocks.readthedocs.org/en/latest/tutorial.html&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt;, to train an MLP neural net with a dataset provided for an assignment in my ML course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to evaluate the accuracy of the network with varying parameterization, but I am not sure how to obtain the accuracy in first place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inspecting the &lt;code&gt;main_loop&lt;/code&gt; object useing &lt;code&gt;dir()&lt;/code&gt; &amp;amp; &lt;code&gt;vars()&lt;/code&gt;, I'm not coming across anything other than &lt;code&gt;the test_cost_with_regularization&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to record means of different &lt;code&gt;Theano&lt;/code&gt; variables via the monitor classes, so perhaps the answer lies there within?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Final Output&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------&#xA;TRAINING HAS BEEN FINISHED:&#xA;---------------------------------------------------------&#xA;Training status:&#xA;     batch_interrupt_received: False&#xA;     epoch_interrupt_received: False&#xA;     epoch_started: False&#xA;     epochs_done: 1&#xA;     iterations_done: 16&#xA;     received_first_batch: True&#xA;     resumed_from: None&#xA;     training_started: True&#xA;Log records from the iteration 16:&#xA;     test_cost_with_regularization: 2.3032071347&#xA;     training_finish_requested: True&#xA;     training_finished: True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11122" LastActivityDate="2015-11-03T16:24:33.707" Title="Making sense of blocks python package output" Tags="|python|neural-network|accuracy|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8702" PostTypeId="2" ParentId="8623" CreationDate="2015-11-03T21:33:06.700" Score="0" Body="&lt;p&gt;This is more what I did then an answer to the stated question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I also want is a rank.  Clearly a similarity of 0.9 versus 0.1 is a higher rank than 0.5 versus 0.4.   My problem is 0.43 non-responsive is versus 0.45 responsive is actually non-responsive (other).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did for the raw score was the harmonic mean of the difference and log of the ratio.  I could get .35 versus a .20 and 0.1 versus a 0.01.  Using the log of the ratio seem to give absolute differences more even ground with relative differences.  Then the harmonic mean kept the bigger number from dominating as much.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I went down the ordered by the exact ratio of the sample set and used that as the offset so a score of 0 was don't know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This rank seems to be effective0 as the top 1/2 by rank had zero errors when looking at sample set versus the classification of the sample set.  And the stuff right in the middle had the most errors.   &lt;/p&gt;&#xA;" OwnerUserId="13285" LastActivityDate="2015-11-03T21:33:06.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8703" PostTypeId="1" AcceptedAnswerId="8752" CreationDate="2015-11-04T03:50:42.330" Score="0" ViewCount="1649" Body="&lt;p&gt;The neural network I am trying to evolve uses the tanh as an activation function in each neuron and has a topology of 1-5-1, so I need at least 5 weights. The solution of the GA is a real-number vector of length 5, which represents the weights of the network and each weight should take values between -5 to 5. I wrote an R function to use as a fitness function which returns the mean squared error (MSE) of the output data in comparison to the desired output. I want it to learn the cubic function. The input data I am using is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input&amp;lt;-seq(-1,1,0.02)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the output data is its cubic function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;des_out&amp;lt;-input^3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The evaluation function is the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;evalnn&amp;lt;-function(x){&#xA;&#xA;mse&amp;lt;-0&#xA;for(i in 1:length(input)){&#xA;&#xA;     nn.out &amp;lt;- tanh(x[1]*input[i]) + tanh(x[2]*input[i]) + tanh(x[3]*input[i]) + tanh(x[4]*input[i]) + tanh(x[5]*input[i]) &#xA;     mse &amp;lt;- (des.out[i] - nn.out)^2 + mse&#xA;&#xA;}&#xA;return(-(mse)/length(input))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have set the return value to negative, because I want the smallest value to be thought as the best fit.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gann&amp;lt;-ga(type=&quot;real-valued&quot;, fitness=evalnn,min=c(-5,-5,-5,-5,-5),max=c(5,5,5,5,5),popSize=100,maxiter=150,pmutation=0.01,pcrossover=0.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I always get back from the GA are weights that make up a linear function although I have been experimenting with the ga's parameters quite a lot, i.e. I have tried all crossover methods and mutations. &lt;a href=&quot;https://cran.r-project.org/web/packages/GA/GA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/GA/GA.pdf&lt;/a&gt;.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/iJVXi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iJVXi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;The cubic function plotted together with my output function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used a linear function as training data before the cubic with this implentation and worked. It has trouble with the non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anybody could figure out why do I get this, is there something I have missed. &#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="13875" LastEditorUserId="13875" LastEditDate="2015-11-04T17:24:16.893" LastActivityDate="2017-12-12T05:24:40.550" Title="Using the GA R package to optimize the weights of a MLP neural network" Tags="|r|neural-network|genetic-algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8704" PostTypeId="1" AcceptedAnswerId="8706" CreationDate="2015-11-04T04:25:31.113" Score="1" ViewCount="267" Body="&lt;p&gt;I'm new to Data Science. I'm trying to understand cosine similarity and it seems like the equation is about finding the distance between two vectors. From what I've Googled, a vector needs to have magnitude and direction. But in CS, it seems like it's a 1-dimensional array. Is vector in CS the same as vector in Physics? If so, what is the direction of a vector. And if a vector is like this &lt;code&gt;[1, 0, 1, 0]&lt;/code&gt; what is the magnitude of this vector? &lt;/p&gt;&#xA;" OwnerUserId="10549" LastActivityDate="2015-11-04T09:27:04.000" Title="Is Vector in Cosine Similarity the same as vector in Physics?" Tags="|cosine-distance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8705" PostTypeId="1" CreationDate="2015-11-04T05:18:41.420" Score="0" ViewCount="1891" Body="&lt;p&gt;What are the advantages and disadvantages of using &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;Collaborative filtering&lt;/a&gt; based recommendation using machine learning approach and graph based approach ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have user purchase data (user_name, user_location, user_company_name, product_name, product_price, product_ingredients) and would like to recommend product for user based on what other user from the same location, company are buying, based on product price, ingredients etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to decide on which of them is suitable for a given use case?&#xA;I would like to evaluate Neo4j (Graph database) and Mahout (Machine learning) for Collaborative filtering.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-11-04T14:46:15.060" Title="collaborative filtering using graph and machine learning" Tags="|machine-learning|graphs|apache-mahout|neo4j|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8706" PostTypeId="2" ParentId="8704" CreationDate="2015-11-04T05:25:52.270" Score="2" Body="&lt;p&gt;As you ask specifically for the Cosine Similarity technique, it has magnitude and direction, and similar to a vector which is used in Physics, as Cosine Similarity deals with vectors in an inner product space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RxFDJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RxFDJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the magnitude of vectors is exactly the same as the formula in Physics (summating over the squares of the vector elements.)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-04T05:25:52.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8707" PostTypeId="1" CreationDate="2015-11-04T07:34:21.203" Score="10" ViewCount="1591" Body="&lt;p&gt;I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is trained with user generated text paragraphs (like comments/questions/answers)&lt;/li&gt;&#xA;&lt;li&gt;Each item in the training set will be tagged with . So for e.g. (&quot;category 1&quot;, , &quot;text paragraph&quot;)&lt;/li&gt;&#xA;&lt;li&gt;There will be hundreds of categories&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What would be the best approach to build such a system? I have been looking at a few different options and the following is a list of possible solutions. Is Word2Vec/NN the best solution at the moment? &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Recursive Neural Tensor Network fed with averaged Word2Vec data &lt;/li&gt;&#xA;&lt;li&gt;RNTN and The Paragraph Vector (&lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot;&gt;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&lt;/a&gt;)?&lt;/li&gt;&#xA;&lt;li&gt;TF-IDF used in a Deep Belief Network&lt;/li&gt;&#xA;&lt;li&gt;TF-IDF and Logistic Regression&lt;/li&gt;&#xA;&lt;li&gt;Bag of words and Naive Bayes classification&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13881" LastActivityDate="2015-12-04T16:48:36.490" Title="Text-Classification-Problem: Is Word2Vec/NN the best approach?" Tags="|neural-network|text-mining|multiclass-classification|word-embeddings|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8708" PostTypeId="2" ParentId="19" CreationDate="2015-11-04T08:11:37.630" Score="4" Body="&lt;p&gt;I think that big data starts at the point where the size prevents you from doing what you want to.&#xA;In most scenarios, there is a limit on the running time that is considered feasible.&#xA;In some cases it is an hour, in some cases it might be few weeks.&#xA;As long as the data is not big enough that only O(n) algorithms can run in the feasible time frame, you didn't reach big data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like this definition since it is agnostic to volume, technology level and specific algorithms. It is not agnostic to resources so a grad student will reach the point of big data way before Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to be able to quantify how big is the data, I like to consider the time needed to backup it. Since the technology advances, volumes that were considered big some years ago are now moderate. Backup time improves, as the technology improves, just as the running time of the learning algorithms.&#xA;I feel it is more sensible to talk about a dataset it takes X hours to backup and not of a dataset of Y bytes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to note that even if you reached the big data point and you can not run algorithms of complexity more than O(n) in the straight forward way, there is plenty you can do in order to still benefit from such algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, Feature selection can reduce the number of features that many algorithms running time depends on.&#xA;In many long tail distribution focusing in the few items in the head might be of benefit.&#xA;You can use a sample and run on it the slower algorithms.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-11-04T08:11:37.630" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8709" PostTypeId="1" CreationDate="2015-11-04T08:18:22.727" Score="1" ViewCount="955" Body="&lt;p&gt;I'm interested to find out how to implement NMF for facial recognition. I understand that the NMF works by taking V, which is a matrix of face images (n resolution x m persons), and factorize V = WH, where we get r basis vectors. I presume that this step is done using the training set. But I don't know how to proceed from here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With PCA, after building the eigenfaces from the training set, we project the test image onto the face-space, then classify the image by comparing the weight coefficients between the test image and training images. But what about NMF? Do I use the basis vectors? What do I do with them?&lt;/p&gt;&#xA;" OwnerUserId="13884" LastActivityDate="2019-05-06T07:19:29.797" Title="Using non-negative matrix factorization (NMF) for facial recognition" Tags="|object-recognition|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8710" PostTypeId="2" ParentId="8704" CreationDate="2015-11-04T09:27:04.000" Score="1" Body="&lt;p&gt;Yes, they are the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The array &lt;code&gt;[1, 0, 1, 0]&lt;/code&gt; represents a vector in $4$ dimensional euclidean space ($\mathbb{R}^4$) with tails at &lt;code&gt;[0, 0 , 0 ,0]&lt;/code&gt; and head at  &lt;code&gt;[1, 0, 1, 0]&lt;/code&gt;. So your vector is &quot;pointing&quot; in the direction of your given set of coordinates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might be easier to visualize in $2$ dimensions: for example, the array &lt;code&gt;[1,0]&lt;/code&gt; corresponds to the unit vector along the $x$-axis centered at the origin.  &lt;/p&gt;&#xA;" OwnerUserId="12806" LastActivityDate="2015-11-04T09:27:04.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8711" PostTypeId="2" ParentId="5379" CreationDate="2015-11-04T11:15:29.960" Score="0" Body="&lt;p&gt;In this situation, I use RapidMiner (&lt;a href=&quot;http://rapidminer.com/&quot; rel=&quot;nofollow&quot;&gt;RapidMiner&lt;/a&gt;). There a multiple solutions such as Hadoop and Radoop or RapidMiner cloud service etc.&lt;/p&gt;&#xA;" OwnerUserId="13676" LastActivityDate="2015-11-04T11:15:29.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8712" PostTypeId="2" ParentId="5192" CreationDate="2015-11-04T11:17:44.937" Score="2" Body="&lt;p&gt;For my research on medical data, I am very pleased with the features RapidMiner offers. &lt;a href=&quot;http://rapidminer.com&quot; rel=&quot;nofollow&quot;&gt;Rapidminer&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="13676" LastActivityDate="2015-11-04T11:17:44.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8714" PostTypeId="2" ParentId="8705" CreationDate="2015-11-04T12:48:06.270" Score="4" Body="&lt;p&gt;I can only speak about Graphs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using graphs, you can easily find products bought/rated by users that bought or liked an item, or users that have similar &quot;taste&quot; to another user. From my experience the traversal process is fast enough.&lt;/li&gt;&#xA;&lt;li&gt;Closely matched products are easy to find, depending on how you model your graph: e.g. &lt;code&gt;(userA)-[]-&amp;gt;(basketA)-[with]-&amp;gt;(productA)&amp;lt;-(basketB)-[]-(userB)&lt;/code&gt;. Finding similar products by basket is computationally cheap here&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disadvantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Depending on how you rank your results, you can easily run into the trap of constantly recommending the same products. Users that bought A, bought B. So you suggest B. People buy/like B. Next time you run the query, B will keep coming on top. &lt;/li&gt;&#xA;&lt;li&gt;Your results will rarely uncover new products. You'd have to find smart approaches around this, e.g. find other products, within this price range, with this category&lt;/li&gt;&#xA;&lt;li&gt;If you have very large data sets, you'll have to use a distributed graph database that performs well (this is not as easy as it sounds, unless you're willing to pay large sums of money)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My little experience with ML for collaborative filtering, is that when your data grows large (50GB+), building a model takes a considerable amount of time (hours, days), and you're not likely to get good recommendations on new products. Having to update your model becomes a huge problem too. From my experience, I lean towards graphs for small use cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that for both cases, you never recommend newly added products that have not been bought or liked by someone. I say this because the goal of recommendations is to help users uncover new products. There is a blog post I wrote, where I talk about modelling graphs, and I discuss recommendations in some examples &lt;a href=&quot;https://thinkingthread.com/modelling-graphs-with-neo4j/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, so you can skip to that section to get an idea on how to model this kind of problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I advise you to read up on Amazon's paper on their approach to collaborative filtering, which is pretty simple, in theory, and yields good results for them. I think you'd always want to implement a few different approaches combined, to tackle different parts of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck.&lt;/p&gt;&#xA;" OwnerUserId="13889" LastEditorUserId="13889" LastEditDate="2015-11-04T12:54:56.360" LastActivityDate="2015-11-04T12:54:56.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8717" PostTypeId="1" AcceptedAnswerId="8719" CreationDate="2015-11-04T14:38:45.187" Score="0" ViewCount="968" Body="&lt;p&gt;I am using SPARK 1.5.1, and I have DataFrame that looks like follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labelsCol, featureCol&#xA;(Label1, Label2, Label 32), FeatureVector&#xA;(Label1, Label10, Label16, Label30, Label48), FeatureVector&#xA;...&#xA;(Label1, label 95), FeatureVector&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first column is the list of labels for that sample, and in total I have 100 label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to build a binary classifier for each label, so I want to transform the labels list column into a binary vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The binary vector will have a length of 100 and the value will be 0 or 1 depends on the existence of the label for sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any strait forward solution for this?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastActivityDate="2015-11-04T14:50:14.170" Title="SPARK 1.5.1: Convert multi-labeled data into binary vector" Tags="|classification|apache-spark|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8718" PostTypeId="2" ParentId="8705" CreationDate="2015-11-04T14:46:15.060" Score="3" Body="&lt;p&gt;One advantage of many ML-based recommendation techniques is they allow you to work in a lower-dimensional space.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Non-negative_matrix_factorization&quot; rel=&quot;nofollow&quot;&gt;Matrix factorization&lt;/a&gt; techniques for example, allow you to view a user or an item in terms of a learned latent-variable space.  This allows for easier computations after the model has been trained (which is often very expensive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the best results on the MovieLens recommendation dataset have been &lt;a href=&quot;http://users.cecs.anu.edu.au/~akmenon/papers/autorec/paper.pdf&quot; rel=&quot;nofollow&quot;&gt;achieved by autoencoders&lt;/a&gt;.  These also have the benefit of reducing the dimensionality of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore learned representations of the data likely represent (perhaps non-linear, in the case of auto-encoders) combinations of features.  Navigating the edges of a graph is likely to focus on one feature at a time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your idea for using neo4j came from &lt;a href=&quot;https://linkurio.us/using-neo4j-to-build-a-recommendation-engine-based-on-collaborative-filtering/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, one thing to remember is that the data you're talking about is not just ratings/likes data (common in collaborative filtering), but also &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system#Content-based_filtering&quot; rel=&quot;nofollow&quot;&gt;content-based&lt;/a&gt; data.  You might want to read about hybrid recommender systems to leverage both content-based and collaborative filtering based recommendation.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-04T14:46:15.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8719" PostTypeId="2" ParentId="8717" CreationDate="2015-11-04T14:50:14.170" Score="4" Body="&lt;p&gt;Spark only recently implemented &lt;a href=&quot;https://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/feature/CountVectorizer.html&quot; rel=&quot;nofollow&quot;&gt;CountVectorizer&lt;/a&gt;, which will take the labels (as strings) and encode them as your 100-dimensional vector (assuming all 100 labels show up somewhere in your dataset).  Once you have those vectors, it should be a simple step to threshold them to make them 0/1 instead of a frequency.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-04T14:50:14.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8720" PostTypeId="2" ParentId="8709" CreationDate="2015-11-04T15:57:38.840" Score="2" Body="&lt;p&gt;One major issue with this approach is that if you represent your image as a vector of pixel intensities, you have zero translation-invariance.  Two pictures that are exactly the same but shifted over 10 pixels will likely look like dissimilar vectors.  That is why people try to use other methods that handle translation, like &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;nofollow&quot;&gt;convolutional neural networks&lt;/a&gt; or &lt;a href=&quot;http://courses.cs.washington.edu/courses/cse576/11sp/notes/SIFT_white2011.pdf&quot; rel=&quot;nofollow&quot;&gt;SIFT features&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-04T15:57:38.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8721" PostTypeId="2" ParentId="8707" CreationDate="2015-11-04T16:45:26.883" Score="5" Body="&lt;p&gt;1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. &lt;a href=&quot;http://arxiv.org/pdf/1405.4053v2.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/pdf/1405.4053v2.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gensim(python) has a nice Doc2vec model.&lt;/p&gt;&#xA;" OwnerUserId="13896" LastActivityDate="2015-11-04T16:45:26.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8722" PostTypeId="1" CreationDate="2015-11-04T17:10:29.887" Score="2" ViewCount="323" Body="&lt;p&gt;I have a DataFrame representing an annotated dataset with 300 labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The DataFrame looks like follow (the first row is just to explain the columns):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;binary Vector column for labeling, feature column&#xA;[0, 1, 1, 0, 0 , 0.... ], featureVec&#xA;[0, 0, 1, 0, 1 , 0.... ], featureVec&#xA;......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The labelling column, represent the presence or the absence of each label as an annotation of a feature vector (which we can see in the second column).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I am wrong: &lt;strong&gt;I can't train one multi-class classifier because the labels are not exclusives, so I would like to train one binary classifier per label.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have 300 labels, then I need to train and optimise 300 classifier (say logistic regression classifier for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way and best practices to train my classifiers with N-cross validation for parameter optimisation? any example code or reference is highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the classifiers are optimised, what is the best way and best practices to save the best models and use them to classify the new data?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastActivityDate="2017-04-18T08:42:39.047" Title="Spark 1.5.1: Train many binary classifiers, save them, then use them on new data" Tags="|classification|multiclass-classification|multilabel-classification|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8723" PostTypeId="2" ParentId="8688" CreationDate="2015-11-04T17:59:47.003" Score="2" Body="&lt;p&gt;Slightly variation of the @init-random's first plot is use of &lt;em&gt;Boxplot&lt;/em&gt;. Here example for the &lt;strong&gt;TotalSize&lt;/strong&gt; using same setup for &lt;code&gt;d&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; xy &amp;lt;- boxplot(TotalSize ~ start.decade    ,   &#xA; data =  d,        &#xA; ylab=&quot;sum of skill levels&quot;, xlab=&quot;start.decade&quot;, main= &quot;Skill per Decade - TotalSize&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sAHNl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sAHNl.png&quot; alt=&quot;Boxplot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this actually ignores the factor and interprets it as numbers.&#xA;To draw a factor frequency in time, you may use &lt;strong&gt;levelplot&lt;/strong&gt; interpreting the third dimension as color.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(lattice)&#xA; library(RColorBrewer)&#xA; library(reshape)&#xA;&#xA; df &amp;lt;- data.frame(d)&#xA; df &amp;lt;- as.data.frame.matrix(table(df[,c(&quot;TotalSize&quot;,&quot;start.decade&quot;)]))&#xA; df$TotalSize &amp;lt;- rownames(df)&#xA; library(reshape)&#xA; dfm &amp;lt;- melt(df, id = c(&quot;TotalSize&quot;))&#xA; colnames(dfm) &amp;lt;- c(&quot;TotalSize&quot;, &quot;decade&quot;, &quot;count&quot;)&#xA; dfm$decade &amp;lt;- as.factor(dfm$decade)&#xA; dfm$TotalSize &amp;lt;- as.factor(dfm$TotalSize)&#xA;&#xA; xy &amp;lt;- levelplot(count ~ decade + TotalSize , data=dfm, cuts=8, &#xA; col.regions=brewer.pal(9,&quot;YlOrRd&quot;)  ,   &#xA; ylab=&quot;Total Skill&quot;, xlab=&quot;Decade&quot;, main= &quot;Total Skill per Decade&quot;)&#xA; print (xy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TfGOg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TfGOg.png&quot; alt=&quot;level plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2015-11-05T11:01:20.723" LastActivityDate="2015-11-05T11:01:20.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8725" PostTypeId="1" CreationDate="2015-11-05T06:10:40.180" Score="2" ViewCount="39" Body="&lt;p&gt;How can I understand &quot;Markov chain Monte Carlo allows sampling from a large class of distributions and which scales well with the dimensionality of the sample space&quot;&lt;/p&gt;&#xA;" OwnerUserId="13905" LastActivityDate="2015-11-05T06:10:40.180" Title="Why Markov Chain Monte Carlo allows sampling from a large class of distributions and scales well with the dimensionality of the sample space?" Tags="|sampling|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8726" PostTypeId="2" ParentId="8265" CreationDate="2015-11-05T11:32:54.160" Score="1" Body="&lt;p&gt;Yes, you can do it (indeed in many ways).&#xA;I like to reduce the problem into a classification problem and derive the proper way to combine them.&#xA;For details see:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/61351/how-to-combine-multiple-similarity-measures/166419#166419&quot;&gt;https://stats.stackexchange.com/questions/61351/how-to-combine-multiple-similarity-measures/166419#166419&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-05T11:32:54.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8729" PostTypeId="1" AcceptedAnswerId="8730" CreationDate="2015-11-05T15:02:35.457" Score="2" ViewCount="358" Body="&lt;p&gt;I was wondering if it makes sense to apply normal standardization on a feature like timestamp ?&#xA;The data that I process are network packets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="13915" LastActivityDate="2015-11-05T15:53:49.797" Title="Does it makes sense to apply feature scaling on timestamp" Tags="|feature-scaling|preprocessing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8730" PostTypeId="2" ParentId="8729" CreationDate="2015-11-05T15:53:49.797" Score="2" Body="&lt;p&gt;For time series analysis.  &lt;strong&gt;Yes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, turning data into a computable object for using in the ML computation?  &lt;strong&gt;No&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the data as a feature.  Then, &lt;strong&gt;Yes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would give you a general time series example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the number of days in months. The irregularity causes friction while analyzing the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oMTB6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oMTB6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, such type of transformations would be helpful which analyzing time series', which can help reduce friction in the models sensibly.&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/165166/number-of-days-in-a-monthly-forecast/165169#165169&quot;&gt;Link of the explanation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-05T15:53:49.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8731" PostTypeId="1" CreationDate="2015-11-05T17:06:27.647" Score="9" ViewCount="973" Body="&lt;p&gt;For our final course project in Data Science, we proposed the following-&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Give the &lt;a href=&quot;http://jmcauley.ucsd.edu/data/amazon/&quot;&gt;Amazon Reviews Dataset&lt;/a&gt;, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that determines a strategic position for placing ads on Amazon. For example, there are millions of products on Amazon. And the dataset gives you an idea of what products are related, what products were brought together, viewed together etc. ( We can construct a graph with this info of also viewed and also bought ) It also gives you the reviews associated with each product over 14 years. Using all these info, we will rate/rank products on Amazon. Now, you are a vendor on Amazon who wants to improve traffic to their product page. Our algorithm helps you identify strategic positions in the graph where you can place your ad so that you can derive max traffic.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, our Professor's question is, how will you validate your algorithm without real users? We said-&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We can model a fixed set of users. Some users follow &lt;code&gt;also_bought&lt;/code&gt; and &lt;code&gt;also_viewed&lt;/code&gt; links to the third hop more often than first or fifth hop. There users' behavior is normally distributed. Some other users hardly navigate beyond the first hop. This set of users' behavior is exponentially distributed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Our Professor's said - Whatever distribution the users follow, users are navigating using links for similar products. Your ranking algorithm also considers similarity b/w 2 products to rank products. So using this validation algorithm is kinda &lt;code&gt;cheating&lt;/code&gt;. Come with some other user behavior, something more realistic and orthogonal to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to model the users' behavior? I am happy to provide more details about the algo.&lt;/p&gt;&#xA;" OwnerUserId="13919" LastActivityDate="2016-06-30T14:48:37.857" Title="How to model user's buying behavior on Amazon?" Tags="|machine-learning|data-mining|dataset|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8732" PostTypeId="1" AcceptedAnswerId="8851" CreationDate="2015-11-05T19:21:27.620" Score="4" ViewCount="527" Body="&lt;p&gt;I'm fairly new to machine learning so please pardon me if I mess up terms. I'm working on a wearables project that involves hand gesture detection, looking for any pointers on potential algorithms that would be applicable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically there will be sensors that pick up tendon movement at the wrist to provide input, so for simplicity I'm assuming that would mean 5 features (one for each finger), each with values ranging from say 0-100. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also 5 classes that the gesture could be classified as. The user would do each type of gesture several times to provide training data, and then for each gesture class I'd need an algorithm to determine the best model for it based on that training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After initial training, whenever the user does a gesture it would input the sensor readings into each model to see which one is the closest match.&#xA;Additionally, since this is a wearable anything that optimizes for low memory and low processing power would be best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone has any input on either the type of models or the algorithm with which to generate them that I should look into I'd appreciate it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Here is some &lt;a href=&quot;https://gist.github.com/lfowlie/779ef9025c9d650173f0&quot; rel=&quot;nofollow&quot;&gt;sample data&lt;/a&gt; from 4 sensors during 3 different gestures. These are basically just pressure readings from sensors lying along tendons on the wrist.&lt;/p&gt;&#xA;" OwnerUserId="13917" LastEditorUserId="13917" LastEditDate="2015-11-19T03:04:12.403" LastActivityDate="2016-01-24T19:36:51.307" Title="Algorithm for gesture classification in a wearable" Tags="|machine-learning|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8733" PostTypeId="2" ParentId="19" CreationDate="2015-11-06T02:18:46.443" Score="6" Body="&lt;p&gt;I'll share what Big Data is like in genomics, in particular de-novo assembly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we sequence your genome (eg: detect novel genes), we take billions of next-generation short reads. Look at the image below, where we try to assemble some reads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TaZ5S.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TaZ5S.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks simple? But what if you have billion of those reads? What if those reads contain sequence errors? What if your RAM doesn't have enough memory to keep the reads? What about repetitive DNA regions, such as the very common &lt;a href=&quot;https://en.wikipedia.org/wiki/Alu_element&quot; rel=&quot;noreferrer&quot;&gt;Alu Element&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;De-novo assembly is done by constructing a &lt;a href=&quot;https://en.wikipedia.org/wiki/De_Bruijn_graph&quot; rel=&quot;noreferrer&quot;&gt;De-Bruijn graph&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/juiy3.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/juiy3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The graph is a clever-mined data-structure to represent overlapping reads. It's not perfect but it's better than generating all possible overlaps and store them in an array.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assembly process could take days to complete, because there are quite a number of paths that an assembler would need to traverse and collapse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In genomics, you have a big data when:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can't brute force all combinations&lt;/li&gt;&#xA;&lt;li&gt;Your computer doesn't have enough physical memory to store the data&lt;/li&gt;&#xA;&lt;li&gt;You need to reduce the dimensions (eg: collapsing redundant graph paths)&lt;/li&gt;&#xA;&lt;li&gt;You get pissed off because you'd have to wait days to do anything&lt;/li&gt;&#xA;&lt;li&gt;You need a special data structure to represent the data&lt;/li&gt;&#xA;&lt;li&gt;You need to filter your data-set for errors (eg: sequencing errors) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/De_Bruijn_graph&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/De_Bruijn_graph&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2015-11-06T14:33:19.680" LastActivityDate="2015-11-06T14:33:19.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8734" PostTypeId="1" CreationDate="2015-11-06T04:12:26.920" Score="1" ViewCount="1685" Body="&lt;p&gt;I am trying to build a model for classifying MNIST dataset using SVM. With raw features I am getting accuracy of around 94% (using linear kernel). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I tried it with PCA, with different number of components (35,50,250,500) I am getting accuracy around 11%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can possibly the reason for this?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2015-11-06T13:17:22.930" Title="PCA on MNIST dataset results in very poor performance" Tags="|svm|performance|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8735" PostTypeId="2" ParentId="8734" CreationDate="2015-11-06T05:29:43.860" Score="0" Body="&lt;p&gt;Firstly, please read up on what exactly is a PCA algorithm and when can it be used and what purpose does it solve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just throwing it on a problem and expect it to improve your results would work most of the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA reduces the features in the dataset into dimensions, and it is for the user to select the number of dimensions he/she want to use in the model as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;a href=&quot;https://stats.stackexchange.com/questions/44060/choosing-number-of-principal-components-to-retain&quot;&gt;choosing the right number of features&lt;/a&gt; to use or eliminate from the model would dictate your accuracy.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-06T05:29:43.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8736" PostTypeId="2" ParentId="424" CreationDate="2015-11-06T07:54:07.367" Score="4" Body="&lt;p&gt;This is one of the nice problem where the scope might vary from an homework assignment to a Google size project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, you can start with co-occurrence of the words (e.g., conditional probability). &#xA;You will discover quickly that you get the list of &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow&quot; title=&quot;stop words&quot;&gt;stop words&lt;/a&gt; as related the most of the words simply because they are very popular.&#xA;Using the lift of conditional probability will take care of the stop words but will make the relation prone to error in small number (most of your cases).&#xA;You might try &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow&quot; title=&quot;Jaccard&quot;&gt;Jacard&lt;/a&gt; but since it is symmetric there will be many relations it won't find. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you might consider relations that appear only in short distance from the base word. You can (and should) consider relations base on general corpus's (e.g., Wikipedia) and user specific (e.g., his emails).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very shortly you will have plenty of relatedness measures, when all the measures are good and have some advantage over the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to combine such measures, I like to reduce the problem into a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should build a data set of paris of words and label them as &quot;is related&quot;.&#xA;In order to build a large labeled dataset you can:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use sources of known related words (e.g., good old Wikipedia categories) for positives&lt;/li&gt;&#xA;&lt;li&gt;Most of the word not known as related are not related.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then use all the measures you have as features of the pairs.&#xA;Now you are in the domain of supervised classification problem.&#xA;Build a classifier on the data set, evaluated according to your needs and get a similarity measure that fits your needs.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-11-06T07:54:07.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8737" PostTypeId="2" ParentId="678" CreationDate="2015-11-06T08:15:59.400" Score="1" Body="&lt;p&gt;It is useful to have in you bag of tools the family of &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow&quot;&gt;locality sensitive hashing algorithms&lt;/a&gt;.&#xA;This family is not semantic at all. &#xA;Actually is consider the text as a sequence of bits.&#xA;I find it useful in dirty data sets when the same text appears many times with slight differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;a href=&quot;http://ssdeep.sourceforge.net/usage.html&quot; rel=&quot;nofollow&quot; title=&quot;ssdeep&quot;&gt;ssdeep&lt;/a&gt; (which is based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Nilsimsa_Hash&quot; rel=&quot;nofollow&quot;&gt;Nilsimsa hash&lt;/a&gt;) for identifying such documents.&#xA;Ssdeep was originally planned for the domain of spam. Spammers often do small changes in the message (add a space) in order to prevent detection by exact signature (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/MD5&quot; rel=&quot;nofollow&quot;&gt;md5&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since many version of almost the same document in the same data set will cause havoc to statistical methods that will be applied on it, doing such a cleanup can be very beneficial.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-11-06T08:15:59.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8738" PostTypeId="1" AcceptedAnswerId="8743" CreationDate="2015-11-06T09:28:56.003" Score="0" ViewCount="80" Body="&lt;p&gt;Suppose I apply tri-gram indexing for my document collection, and is implementing a vector-space model to help retrieving the document. In &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/k-gram-indexes-for-wildcard-queries-1.html&quot; rel=&quot;nofollow&quot;&gt;the text&lt;/a&gt; it is mentioned implementing a trigram will introduce a new step in filtering the result. However, what are the problems that I need to be aware of if I implement tfidf/vector-space model? The reason I am exploring this option is to try handling basic spelling error handling, does it really work in practice?&lt;/p&gt;&#xA;" OwnerUserId="13562" LastActivityDate="2015-11-06T16:33:56.420" Title="Typing error handling n-gram character index and vector space model" Tags="|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8739" PostTypeId="2" ParentId="8734" CreationDate="2015-11-06T13:17:22.930" Score="1" Body="&lt;p&gt;There are a few thing that you may be doing, but worth mentioning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did you &quot;whiten&quot; your data---mean center and normalize by the standard deviation? This step may be implementation-dependent.&#xA;&lt;a href=&quot;http://deeplearning.stanford.edu/wiki/index.php/PCA&quot; rel=&quot;nofollow&quot;&gt;http://deeplearning.stanford.edu/wiki/index.php/PCA&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did you check to see how much variablility was explained by the number of chosen parameters? For images +80% is common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure you understand the implementation of the PCA you are using. It always confuses me by the terminology different implementations uses, e.g. score, rotation, etc. I suggest following a tutorial and matching their output with your implementation to get a deeper understanding.&#xA;&lt;a href=&quot;http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html&quot; rel=&quot;nofollow&quot;&gt;http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model is now built on transformed data. To predict make sure you are using these same transformations on the new input data.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-06T13:17:22.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8741" PostTypeId="2" ParentId="8688" CreationDate="2015-11-06T15:11:04.593" Score="3" Body="&lt;p&gt;This is something you can get in Tableau in 10 minutes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HE5go.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HE5go.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need is year binning, skills weights and length(year_end - year_start) for Gannt chart:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sTAAt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sTAAt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="97" LastActivityDate="2015-11-06T15:11:04.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8743" PostTypeId="2" ParentId="8738" CreationDate="2015-11-06T16:33:56.420" Score="2" Body="&lt;p&gt;Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help.  You need some form of &lt;a href=&quot;https://en.wikipedia.org/wiki/Approximate_string_matching&quot; rel=&quot;nofollow&quot;&gt;fuzzy matching&lt;/a&gt; for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example the string, &quot;I like dosg too&quot; would fool a unigram model because &quot;dosg&quot; is likely &quot;dogs&quot; misspelled, and it will encode it as &lt;code&gt;&quot;dosg&quot; : 1&lt;/code&gt;.  But you have the same problem in a trigram model.  It will encode &lt;code&gt;&quot;I like dosg&quot; : 1&lt;/code&gt;, &lt;code&gt;&quot;like dosg too&quot; : 1&lt;/code&gt;.  Which is not really better, as it will still not match any trigrams with the word &quot;dogs&quot; in it.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-06T16:33:56.420" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="8744" PostTypeId="1" CreationDate="2015-11-06T17:41:28.683" Score="2" ViewCount="143" Body="&lt;p&gt;Since NB is a generative classifier, we assume that the data points are all generated from a distribution, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But since we can compute the MLE of &lt;strong&gt;p(x_i|y)&lt;/strong&gt; by counting (MLE of p(x_i|y) = # of (x_i,y) / # of y), do we really need a specific distribution, if at all, to model the likelihood(p(x_i|y))?&lt;/p&gt;&#xA;" OwnerUserId="13943" LastEditorUserId="8820" LastEditDate="2016-01-27T06:39:14.640" LastActivityDate="2016-01-27T10:45:48.673" Title="Does Naive Bayes Classifier require assuming a specific distribution for training?" Tags="|machine-learning|naive-bayes-classifier|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8745" PostTypeId="1" AcceptedAnswerId="8930" CreationDate="2015-11-06T22:25:43.853" Score="1" ViewCount="3175" Body="&lt;p&gt;This is more of a hypothetical than something I'm actively trying to solve. It just struck me that a machine learning algorithm that specifically looked at two pieces of data and had to label one as greater than the other or something of that nature might be inherently different than classifying each separately and comparing the strength of the classifications.&lt;/p&gt;&#xA;" OwnerUserId="9768" LastActivityDate="2015-11-20T09:05:33.487" Title="Are there any machine learning algorithms that focus on comparing items, rather than classification or regression?" Tags="|machine-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8747" PostTypeId="2" ParentId="8688" CreationDate="2015-11-07T14:01:29.810" Score="1" Body="&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Picking up from where we left off in the original post a quick interactive view to of the background areas can be made. I show this in a gif file here as you would need either the free Wolfram CDF Player or Mathematica installed to see the interactivity (there is also the Wolfram Cloud that does not need an install but I don't want you all using up my cloud credits while you play with this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ELX69.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ELX69.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The controls look better in Mathematica but the gif export uses more plain representations of them. In any case your report could lead with this interactive view of the backgrounds and use the static table with all the employees as an appendix. You could write your report in Mathematica and sent it out as a CDF (they would have to install the reader). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mathematica code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;Manipulate&lt;/code&gt; call may look a big but it only has two parts. The first &lt;code&gt;Grid&lt;/code&gt; is for the table and it uses the sorting variables defined in the second &lt;code&gt;Grid&lt;/code&gt; call. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(* Pregroup the data by decade *)&#xA;backgroundByDecade = Function[{decade},&#xA;      decade -&amp;gt; &#xA;       background[&#xA;        Select[IntervalMemberQ[Interval[{#[&quot;YearFrom&quot;], #[&quot;YearTo&quot;]}],&#xA;            decade] &amp;amp;]]&#xA;      ] /@ decades // Association // Dataset;&#xA;&#xA;(* Create interactive information grid *)&#xA;Manipulate[&#xA; Grid[Transpose[Normal[&#xA;    backgroundByDecade[All,&#xA;      Function[{bkgDecade},&#xA;         Column[&#xA;          Normal@bkgDecade[&#xA;             SortBy[{firstDesc #[firstCol], secondDesc #[secondCol], &#xA;                thirdDesc #[thirdCol]} &amp;amp;]][All, #[&quot;BkgChart&quot;] &amp;amp;], &#xA;          Spacings -&amp;gt; 0]]][KeyValueMap[{#1, #2} &amp;amp;]]&#xA;    ]],&#xA;  Alignment -&amp;gt; {Center, Top},&#xA;  Dividers -&amp;gt; {Gray, {None, {2 -&amp;gt; Gray}}},&#xA;  BaseStyle -&amp;gt; {FontFamily -&amp;gt; &quot;Calibri&quot;}],&#xA; Style[&quot;Sory By&quot;, 14, Bold],&#xA; Grid[{&#xA;   {&quot;First&quot;, &#xA;    Control[{{firstCol, &quot;BkgArea1&quot;, &quot;&quot;}, {&quot;BkgArea1&quot;, &quot;BkgArea2&quot;, &#xA;       &quot;BkgArea3&quot;}}], &#xA;    Control[{{firstDesc, -1, &quot;Desc&quot;,}, {1, -1}, Checkbox}]},&#xA;   {&quot;Second&quot;, &#xA;    Control[{{secondCol, &quot;BkgArea2&quot;, &quot;&quot;}, {&quot;BkgArea1&quot;, &quot;BkgArea2&quot;, &#xA;       &quot;BkgArea3&quot;}}], &#xA;    Control[{{secondDesc, -1, &quot;Desc&quot;}, {1, -1}, Checkbox}]},&#xA;   {&quot;Third&quot;, &#xA;    Control[{{thirdCol, &quot;BkgArea3&quot;, &quot;&quot;}, {&quot;BkgArea1&quot;, &quot;BkgArea2&quot;, &#xA;       &quot;BkgArea3&quot;}}], &#xA;    Control[{{thirdDesc, -1, &quot;Desc&quot;}, {1, -1}, Checkbox}]}&#xA;   }, Alignment -&amp;gt; Left],&#xA; ControlPlacement -&amp;gt; Left, Paneled -&amp;gt; False&#xA; ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Original Post&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand you would like to see a person's background area at the start of each decade that they are present. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below a colour for each background area is used and each is lightened to show the strength/weakness of the background.  The background number is also given but these can be removed and placed in a tooltip of the boxes if you want to. Each box set contains the 3 dimensions of background area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These sets are arranged in time columns for the start of each decade. The background areas align within the columns so you can scan down a decade background area to see the distribution of its strength across persons.  A summary background area box set could be added for each decade column as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TUtcO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TUtcO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;Mathematica code&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = Import[FileNameJoin[{NotebookDirectory[], &quot;BackgroundArea.txt&quot;}], &quot;Table&quot;];&#xA;&#xA;(* Background area chart function *)&#xA;bkgAreaBarChart[bkgArea1_, bkgArea2_, bkgArea3_] :=&#xA; BarChart[{{1, 1, 1}}, ChartLayout -&amp;gt; &quot;Stacked&quot;,&#xA;  ChartStyle -&amp;gt; {EdgeForm[Gray], &#xA;    Inner[#1[#2] &amp;amp;, &#xA;     Function[{value}, &#xA;       Nest[Lighter, #, 3 - value]] &amp;amp; /@ {Purple,Orange, Blue}, {bkgArea1, bkgArea2, bkgArea3}, List]}, &#xA;  Axes -&amp;gt; False,&#xA;  PlotRangePadding -&amp;gt; None,&#xA;  BarOrigin -&amp;gt; Left,&#xA;  ChartLabels -&amp;gt; Placed[{bkgArea1, bkgArea2, bkgArea3}, Center],&#xA;  BaseStyle -&amp;gt; {Background -&amp;gt; None},&#xA;  ImagePadding -&amp;gt; None,&#xA;  ImageSize -&amp;gt; {Automatic, 20},&#xA;  AspectRatio -&amp;gt; 1/3]&#xA;&#xA;(* Create Dataset from data and add chart column *)&#xA;background = Dataset[&#xA;  AssociationThread[&#xA;     Join[First@data, {&quot;BkgChart&quot;}],&#xA;     Join[#, {bkgAreaBarChart[Sequence @@ #[[{2, 3, 4}]]]}]&#xA;     ] &amp;amp; /@ Rest@data]&#xA;&#xA;(* Calculate decades from dataset *)&#xA;decades = &#xA; Range[Sequence @@ Normal[background[MinMax, {&quot;YearFrom&quot;, &quot;YearTo&quot;}][Round[#, 10] &amp;amp;]], 10]&#xA;&#xA;(* Create information grid *)&#xA;Grid[&#xA; Join[&#xA;  {Join[{&quot;&quot;}, decades]},&#xA;  Transpose[&#xA;   Join[{background[All, &quot;Person&quot;] // Normal},&#xA;    Function[{decade},&#xA;      background[All, &#xA;        If[IntervalMemberQ[Interval[{#[&quot;YearFrom&quot;], #[&quot;YearTo&quot;]}], decade],&#xA;          #[&quot;BkgChart&quot;], &#xA;          Graphics[ImageSize -&amp;gt; {Automatic, 20},&#xA;           AspectRatio -&amp;gt; 1/3]] &amp;amp;] // Normal] /@ decades&#xA;    ]],&#xA;  {Join[{&quot;&quot;}, decades]}&#xA;  ],&#xA; Alignment -&amp;gt; {{Left, {Center}}, Center},&#xA; Spacings -&amp;gt; {0, 0},&#xA; Dividers -&amp;gt; {{None, {Gray}}, {None, {2 -&amp;gt; Gray, -2 -&amp;gt; Gray}}},&#xA; Background -&amp;gt; {None, {None, {LightGray, None}, None}},&#xA; BaseStyle -&amp;gt; {FontFamily -&amp;gt; &quot;Calibri&quot;}]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This was interesting to consider.&lt;/p&gt;&#xA;" OwnerUserId="10814" LastEditorUserId="10814" LastEditDate="2015-11-08T15:22:56.140" LastActivityDate="2015-11-08T15:22:56.140" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8749" PostTypeId="2" ParentId="8745" CreationDate="2015-11-07T16:04:53.320" Score="0" Body="&lt;p&gt;Not sure if I read the question correctly, but I have implemented multi-variate sub-sample optimisation routines using the Kolmogrov-Smirnov test? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage is that if two distributions exactly track each other (so would be significantly correlated) but are at quite different scales (e.g. one is half the value of the other), they would still rank as very different distributions at similar scales. Only when both distribution and scale match is the ranking high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spatially per-pixel comparison algorithms are also quite common in raster analysis?&lt;/p&gt;&#xA;" OwnerUserId="13955" LastActivityDate="2015-11-07T16:04:53.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8750" PostTypeId="2" ParentId="5302" CreationDate="2015-11-07T16:46:45.183" Score="0" Body="&lt;p&gt;Been a while since I used SAS but I think you could use a data step where you make an ID var on which you can aggregate, then vars for each &quot;cat1-4.&quot; You could then use proc transpose or do a proc SQL sum() with a &quot;groupby&quot; statement on the ID variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So first step is to get to:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;'Topic_ID' | 'Cat1' | 'Cat2' | 'Cat3' | 'Cat4'&#xA;     1     |   1.5  |    0   |    0   |    0&#xA;     1     |    0   |    3   |    0   |    0&#xA;     1     |    0   |    0   |    1   |    0&#xA;     1     |    0   |    0   |    0   |    4&#xA;     2     |    3   |    0   |    0   |    0&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If Topic_number isn't explicit in your dataset, you can always calculate it for each observation by floor(obs_#/4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then by performing a Proc SQL sum() using &lt;a href=&quot;http://www.w3schools.com/sql/sql_groupby.asp&quot; rel=&quot;nofollow&quot;&gt;group by&lt;/a&gt;, you can reduce the data to look like this&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;'Topic_ID' | 'Cat1' | 'Cat2' | 'Cat3' | 'Cat4'&#xA;     1     |   1.5  |    3   |    1   |    4&#xA;     2     |    3   |   ...  |  ...   |  ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This isn't necessarily the most efficient method, but it's simple to implement with SAS.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-08T04:28:52.527" LastActivityDate="2015-11-08T04:28:52.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8752" PostTypeId="2" ParentId="8703" CreationDate="2015-11-08T01:26:47.457" Score="-1" Body="&lt;p&gt;I finally got the results I was looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently a single hidden layer neural network cannot learn the cubic function. Instead of just evolving the weights, I included the topology of the neural network into the chromosome. My neural network can have maximum 5 hidden layers with maximum 5 neurons in each layer and the chromosome contains information for the structure of the network and the selection of the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I ran the GA with population of 200 and for 100 iterations I got a full topology of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1(input) - 4-2-3-2-2 - 1(output), a mean squared error of 0.0051 and the following plot&lt;a href=&quot;https://i.stack.imgur.com/87WsZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/87WsZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be glad if anyone has any comments or questions.&lt;/p&gt;&#xA;" OwnerUserId="13875" LastActivityDate="2015-11-08T01:26:47.457" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8753" PostTypeId="1" AcceptedAnswerId="9278" CreationDate="2015-11-08T04:17:56.450" Score="30" ViewCount="19865" Body="&lt;p&gt;This is more like a general NLP question. &#xA;What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or should each article be a document in said corpus?&#xA;This is just an example using python and gensim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Corpus split by sentence:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SentenceCorpus = [[&quot;first&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;first&quot;, &quot;article.&quot;],&#xA;                  [&quot;second&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;first&quot;, &quot;article.&quot;],&#xA;                  [&quot;first&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;second&quot;, &quot;article.&quot;],&#xA;                  [&quot;second&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;second&quot;, &quot;article.&quot;]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Corpus split by article:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ArticleCorpus = [[&quot;first&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;first&quot;, &quot;article.&quot;,&#xA;                  &quot;second&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;first&quot;, &quot;article.&quot;],&#xA;                 [&quot;first&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;second&quot;, &quot;article.&quot;,&#xA;                  &quot;second&quot;, &quot;sentence&quot;, &quot;of&quot;, &quot;the&quot;, &quot;second&quot;, &quot;article.&quot;]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Training Word2Vec in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from gensim.models import Word2Vec&#xA;&#xA;wikiWord2Vec = Word2Vec(ArticleCorpus)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="29575" LastEditDate="2018-03-24T22:30:43.570" LastActivityDate="2020-08-06T13:01:29.880" Title="What is a better input for Word2Vec?" Tags="|nlp|word-embeddings|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8755" PostTypeId="1" AcceptedAnswerId="8776" CreationDate="2015-11-08T07:01:32.580" Score="5" ViewCount="1455" Body="&lt;p&gt;Say I had the following training set for a Naive Bayes algorithm. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Outlook    Person    Play Golf?&#xA;   -------    ------    ----------&#xA;   Sunny      Joe       Yes&#xA;   Sunny      Mary      Yes&#xA;   Raining    Joe       Yes&#xA;   Raining    Mary      No&#xA;   Raining    Harry     Yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If try to predict whether Harry will play golf on a sunny day (which I have no data for). &#xA;Would it be correct to exclude the person attribute and use the remaining outlook attribute to calculate the probability of this happening? Or could that potentially cause problems with a larger data set that I'm unaware of?&lt;/p&gt;&#xA;" OwnerUserId="13959" LastEditorUserId="8820" LastEditDate="2016-01-27T00:03:41.853" LastActivityDate="2016-01-27T00:03:41.853" Title="Predicting New Data with Naive Bayes" Tags="|machine-learning|classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8756" PostTypeId="2" ParentId="8338" CreationDate="2015-11-08T14:05:24.280" Score="1" Body="&lt;p&gt;If you did a simple linear regression for each of the 5 outputs, your r-squared would be a good measure of the &quot;aggregate correlation.&quot; If you wanted to compare to the other results, it would be helpful to do the same for each of your single features as well.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-08T14:05:24.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8757" PostTypeId="1" CreationDate="2015-11-08T20:26:21.163" Score="3" ViewCount="109" Body="&lt;p&gt;Suppose I have a data set with points $x_i$ and a dissimilarity measure $d_{ij}$ between each pair, as well as a weight $w_{ij}$ that qualifies the quality of this dissimilarity. I have two problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The first one is how to introduce weights when performing spectral clustering. Like in the Shi-Malik, Ng-Jordan-Weiss versions, where only the distances are used.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The second one is that my dissimilarity measure does not exists sometimes. In these cases, it has zero weight. How to introduce these cases in the algorithm? I suppose the zero weight solves the problem if I know how to introduce them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;An idea I had is to perform the similarity transformation as:&#xA;$$&#xA;s_{ij} = w_{ij}  e^{-\frac{d^2_{ij}}{\sigma^2}}&#xA;$$&#xA;but I feel it does not make sense since the weigth should reflect the importance of the measure, not the measure itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is not Euclidean, so I can use only this dissimilarity measure. Any reference is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="13978" LastEditorUserId="13978" LastEditDate="2015-11-08T21:03:10.713" LastActivityDate="2015-11-11T15:16:22.027" Title="Introducing weights into spectral clustering" Tags="|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8762" PostTypeId="1" AcceptedAnswerId="8767" CreationDate="2015-11-09T09:40:19.857" Score="6" ViewCount="7204" Body="&lt;p&gt;I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the similarities of my data points). What tactic can I use for this?&lt;/p&gt;&#xA;" OwnerUserId="13915" LastEditorUserId="13296" LastEditDate="2015-11-09T18:36:23.563" LastActivityDate="2018-05-01T18:20:20.260" Title="How to define a distance measure between two IP addresses?" Tags="|feature-extraction|preprocessing|featurization|" AnswerCount="7" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="8763" PostTypeId="2" ParentId="8762" CreationDate="2015-11-09T09:54:22.597" Score="2" Body="&lt;p&gt;IP (v4) addresses are a 32-bit integer, which trivially gives you a metric. However, it may not be a particularly &lt;em&gt;useful&lt;/em&gt; metric - 10.255.255.255 and 11.0.0.0 are almost certainly significantly more different than 192.168.1.1 and 192.168.1.2.&lt;/p&gt;&#xA;" OwnerUserId="13296" LastActivityDate="2015-11-09T09:54:22.597" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8764" PostTypeId="2" ParentId="8762" CreationDate="2015-11-09T10:09:25.230" Score="4" Body="&lt;p&gt;That's a very interesting question. Similarity here should be computed component-wise, but the thing is from a &quot;business logic&quot; perspective, the similarity of the last 3 numbers doesn't matter if the other 3 sets of numbers are not the same. Keeping that in mind, I would probably do something like the following (there is probably a more elegant way of doing it, and I don't have much time to think about it so forgive me if it doesn't answer your question and for the poor formatting).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming IPv4 of the form aaa.bbb.ccc.ddd, I would so something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If aaa_1 == aaa_2:&#xA;  If bbb_1 == bbb_2:&#xA;    If ccc_1 == ccc_2:&#xA;        If ddd_1 == ddd_2:&#xA;            Dist = 1;&#xA;        Else:&#xA;            Dist = (3 + distance(ddd_1,ddd_2))/4;&#xA;        End if;&#xA;    Else:&#xA;        Dist = (2 + distance(ccc_1,ccc_2))/4;&#xA;    End if;&#xA;  Else:&#xA;    Dist = (1 + distance(bbb_1,bbb_2))/4;&#xA;  End if;&#xA; Else:&#xA;  Dist = distance(aaa_1,aaa_2);&#xA;  Return 1/Dist;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-11-09T10:09:25.230" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8765" PostTypeId="2" ParentId="8338" CreationDate="2015-11-09T11:06:56.353" Score="1" Body="&lt;p&gt;From what you describe, it seems that something like &lt;a href=&quot;https://en.wikipedia.org/wiki/Pointwise_mutual_information&quot; rel=&quot;nofollow&quot;&gt;Pointwise Mutual Information&lt;/a&gt; could satisfy your requirements. It's often used in text mining and opinion analysis to analyse the degree of correlation between specific terms and classes (e.g. a term in a product review and the &quot;positive reviews&quot; class).&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2015-11-09T11:06:56.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8766" PostTypeId="2" ParentId="8762" CreationDate="2015-11-09T12:19:02.747" Score="1" Body="&lt;p&gt;&lt;strong&gt;2nd Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The below can be improved as it does not consider the hierarchical structure of an IP address.  To account for this the elements of the IP vectors can be non-linearly scaled before computing the distance vector and its norm. This gives more weight to the elements higher in the hierarchy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mathematica code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once we have the 4D vectors from the 1st update each element is scaled based on its position $[x^{2}_{1},x^{\frac{3}{2}}_{2},x^{1}_{3},x^{\frac{1}{2}}_{4}]$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Subtract @@ (MapIndexed[&#xA;       Function[{value, index}, &#xA;        value^((5 - First@index)/2)], #] &amp;amp; /@ {ip1, ip2}) // Norm // N&#xA;(* 2209.17 *)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is information lost in collapsing from 4D down to 1D but this can't be helped if you are looking for a 1D distance metric.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1st Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An IP address is made up of 4 numbers. Takes these as vectors in 4D and calculate the distance between them ( &lt;a href=&quot;https://en.wikipedia.org/wiki/Distance&quot; rel=&quot;nofollow&quot;&gt;Distance in Euclidean space&lt;/a&gt; ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mathematica code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(* Make some IP addresses *)&#xA;{ip1, ip2} = &#xA; StringRiffle[#, &quot;.&quot;] &amp;amp; /@ &#xA;  Map[ToString, RandomInteger[{1, 255}, {2, 4}], {2}]&#xA;(* {&quot;50.229.29.146&quot;, &quot;27.167.216.58&quot;} *)&#xA;&#xA;(* Extract 4D vector *)&#xA;{ip1, ip2} = Map[FromDigits, StringSplit[#, &quot;.&quot;] &amp;amp; /@ {ip1, ip2}, {2}]&#xA;(* {{50, 229, 29, 146}, {27, 167, 216, 58}} *)&#xA;&#xA;(* Calculate distance *)&#xA;Norm[ip1 - ip2] // N&#xA;(* 216.993 *)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Consider an IP address as a 4D vector. Subtract and calculate the norm.&lt;/p&gt;&#xA;" OwnerUserId="10814" LastEditorUserId="10814" LastEditDate="2015-11-09T14:25:49.203" LastActivityDate="2015-11-09T14:25:49.203" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="8767" PostTypeId="2" ParentId="8762" CreationDate="2015-11-09T21:04:51.970" Score="5" Body="&lt;p&gt;If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative to euclidean distance would be to calculate a hamming distance bit-wise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edmund's first update would be better than his second. The reason is simple to state: his 2nd update tries to define a distance measure by considering a &lt;strong&gt;non-linear&lt;/strong&gt; function of the coordinates of a 4D vector. That however will most likely destroy the key properties that it needs to satisfy in order to be a metric, namely &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Injectivity: $d(IP_1,IP_2)=0 \iff IP_1=IP_2$, &lt;/li&gt;&#xA;&lt;li&gt;Symmetry: $d(IP_1,IP_2)=d(IP_2,IP_1)$, and &lt;/li&gt;&#xA;&lt;li&gt;Triangular inequality: $d(IP_1,IP_2)\leq d(IP_1,IP_3)+d(IP_3,IP_2)\,\forall IP_3$. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The latter is key for later interpreting small distances as close points in IP space. One would need a linear (in the coordinates) distance function. However, simple euclidean distance is not enough as you saw. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Physics (well, differential geometry actually) could lead to a nice solution to this problem: define a metric tensor $g$. In plain english, give weights to each &lt;strong&gt;pair of coordinates&lt;/strong&gt;, take each pair difference, square it and multiply it by its weight, and then add those products. Take the square root of that sum and define it as your distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the sake of simplicity, one could start trying with a diagonal metric tensor.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Example: Say you take $g=\begin{pmatrix}1000 &amp;amp;0 &amp;amp;0 &amp;amp;0 \\0 &amp;amp;100&amp;amp;0&amp;amp;0\\0&amp;amp;0&amp;amp;10&amp;amp;0\\0&amp;amp;0&amp;amp;0&amp;amp;1\end{pmatrix}$ $IP_1=(x_1,x_2,x_3,x_4)$ and&#xA;  $IP_2=(y_1,y_2,y_3,y_4)$. Then the square of the distance is given by&#xA;  $$d(IP_1,IP_2)^2=1000*(x_1-y_1)^2+100*(x_2-y_2)^2+\\ \,+10*(x_3-y_3)^2+1*(x_4-y_4)^2$$&#xA;  For $IP_1=192.168.1.1,\,IP_2=192.168.1.2$ the distance is clearly 1.&#xA;  However, for $192.168.1.1$ and $191.168.1.1$ the distance is&#xA;  $\sqrt{1000}\approx 32$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Eventually you could play around with different weights and set a kind of normalization where you could fix the value of the maximal distance $d(0.0.0.0,FF.FF.FF.FF)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, this set up allows for more complex descriptions of your data where the relevant distance would contain &quot;cross-products&quot; of coordinates like say $g_{13}*(x_1-y_1)*(x_3-y_3)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: While this would be a better &quot;weighting&quot; method than using those other I addressed, I realize now it is actualy meaningless: As Anony-Mousse and Phillip mention, IP are indeed 32 dimensional. This means in particular that giving the same weight to all bits in say the 2nd group is in general not sound: One bit could be part of the netmask while the other not. See Anony-Mousse answer for additional objections. &lt;/p&gt;&#xA;" OwnerUserId="12603" LastEditorUserId="12603" LastEditDate="2015-11-10T01:23:31.757" LastActivityDate="2015-11-10T01:23:31.757" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8768" PostTypeId="1" AcceptedAnswerId="8769" CreationDate="2015-11-09T22:55:49.043" Score="4" ViewCount="3226" Body="&lt;p&gt;It appears to me that &quot;feature extraction&quot; and &quot;feature learning&quot; are equivalent  concepts, however there are 2 separate wikipedia articles dedicated to them that are notably different. In particular, only in the Feature Learning article Neural Networks/Deep Learning are mentioned. However, it seems like they would be equally appropriate for either because autoencoders extract features from the raw (typically image) data and those now extracted features feed into the next layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what's the real difference between these terms?&lt;/p&gt;&#xA;" OwnerUserId="14009" LastEditorUserId="14009" LastEditDate="2015-11-10T00:42:44.357" LastActivityDate="2015-11-10T00:42:44.357" Title="Is there any difference between feature extraction and feature learning?" Tags="|machine-learning|feature-selection|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8769" PostTypeId="2" ParentId="8768" CreationDate="2015-11-09T23:11:48.677" Score="4" Body="&lt;p&gt;Yes I think so. Just by looking at &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_learning&quot; rel=&quot;nofollow&quot;&gt;Feature Learning&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_extraction&quot; rel=&quot;nofollow&quot;&gt;Feature extraction&lt;/a&gt; you can see it's a different problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature extraction&lt;/strong&gt; is just transforming your raw data into a sequence of feature vectors (e.g. a dataframe) that you can work on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;feature learning&lt;/strong&gt;, you don't know what feature you can extract from your data. In fact, you will probably apply machine learning techniques just to  discover what are good features to extract from your dataset. Then you can extract them them apply machine learning to the extracted features. Deep learning techniques are one example of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;strong&gt;word2vec&lt;/strong&gt; toolkit, for instance, you extract vectors from documents which can't be easily interpreted by a human, you can't look at it and tell what &lt;em&gt;features&lt;/em&gt; have been extracted at all. It's just a mass of vectors which, for some reason, give good empirical results.&lt;/p&gt;&#xA;" OwnerUserId="413" LastActivityDate="2015-11-09T23:11:48.677" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8770" PostTypeId="2" ParentId="5357" CreationDate="2015-11-09T23:14:37.530" Score="1" Body="&lt;p&gt;Not sure whether it's been mentioned yet, but there's also &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit&quot; rel=&quot;nofollow&quot;&gt;vowpal wabbit&lt;/a&gt; but it might be specific to certain kinds of problem only.&lt;/p&gt;&#xA;" OwnerUserId="413" LastActivityDate="2015-11-09T23:14:37.530" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8771" PostTypeId="2" ParentId="8762" CreationDate="2015-11-09T23:48:53.397" Score="3" Body="&lt;p&gt;20 years ago, I would have suggested to use the &lt;strong&gt;length of the shared prefix&lt;/strong&gt; as similarity measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you take two IPs. In their 32 bit representation, not the &quot;pretty printed&quot; x.y.z.w form; the real &quot;int&quot; reoresentation your network stack uses. Then XOR them, count the leading zeros, and you get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distance = 32 - leadingZeros(ip1 XOR ip2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, we have exhausted the IPv4 namespace long ago. The last 10 years, the few remaining netblock have been more or less &quot;randomly&quot; (at least from a similarity perspective) been distributed. &lt;strong&gt;IP ranges have been relocated&lt;/strong&gt; and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lomg time ago, people would have told you routing happens on trees, based on their prefix. So If you wanted to read an IP 10.2.3.4 it would go to 10.0.0.0 then 10.2.0.0 then 10.2.3.0. But that was just the theory. If you &lt;em&gt;manually&lt;/em&gt; configured your router, that is what you would do. 10.2 isthe second building, 10.2.3. is the third floor router.&#xA;History.&#xA;Within networks, IPs are assigned by DHCP, often first-come-first-served. Within intranet, you have mostly switches not routers. And on the global level, the &lt;strong&gt;BGP&lt;/strong&gt; is responsible for taking care of the &lt;strong&gt;big mess&lt;/strong&gt; of todays routing tables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words:&#xA;use some database like GeoIP to map the IPs to (approximate) coordinates. Best you can do.&#xA;IP based similarity is mostly useful on a /24 prefix, but a binary yes/no similarity won't make you happy I guess.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-11-09T23:48:53.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8773" PostTypeId="2" ParentId="694" CreationDate="2015-11-10T01:00:51.743" Score="19" Body="&lt;p&gt;&lt;a href=&quot;http://www.tensorflow.org/&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt; (by Google, &lt;a href=&quot;https://googleblog.blogspot.com/2015/11/tensorflow-smarter-machine-learning-for.html&quot; rel=&quot;noreferrer&quot;&gt;released&lt;/a&gt; on 2015-11-09) looks promising.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;open source (Apache 2.0 License) (&lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;noreferrer&quot;&gt;GitHub&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Python (backend in C++)&lt;/li&gt;&#xA;&lt;li&gt;CPU/GPU&lt;/li&gt;&#xA;&lt;li&gt;Auto-Differentiation&lt;/li&gt;&#xA;&lt;li&gt;Portable (even works on mobile devices)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Suhdk.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Suhdk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYI: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/zer0n/deepframeworks/blob/master/README.md&quot; rel=&quot;noreferrer&quot;&gt;Evaluation of Deep Learning Toolkits&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;noreferrer&quot;&gt;Easy benchmarking of all public open-source implementations of convnets.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Soheil Bahrampour, Naveen Ramakrishnan, Lukas Schott, Mohak Shah. &lt;a href=&quot;http://arxiv.org/abs/1511.06435&quot; rel=&quot;noreferrer&quot;&gt;Comparative Study of Deep Learning Software Frameworks&lt;/a&gt;.  arXiv:1511.06435.&lt;/li&gt;&#xA;&lt;li&gt;Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu. &lt;a href=&quot;http://arxiv.org/abs/1608.07249&quot; rel=&quot;noreferrer&quot;&gt;Benchmarking State-of-the-Art Deep Learning Software Tools&lt;/a&gt;. arXiv:1608.07249&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://skeptics.stackexchange.com/q/31794/7654&quot;&gt;Was a new deep learning toolkit released once every 22 days in 2015?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:46:46.723" LastActivityDate="2016-08-27T15:28:11.023" CommentCount="5" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="8775" PostTypeId="1" AcceptedAnswerId="8777" CreationDate="2015-11-10T08:06:57.003" Score="3" ViewCount="923" Body="&lt;p&gt;We have the feeling that behavior of a device in terms of continuous variables (fans speeds, temperatures, voltages, ...) has influence on rare events happening (components failures).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now have to build a predictive model for that, to proof the influence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those continuous features are given as time series, and events are punctual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've made a model based on descriptive statistics of those variable (see &lt;a href=&quot;https://datascience.stackexchange.com/q/8245/3024&quot;&gt;this question&lt;/a&gt;) with decision tree, random forest, adaboost, and clustering but it doesn't works. I will still improve by balancing classes, but I'm convinced it is not the best approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that there are nicer algorithms for such predictions (this is quite common problem), but I don't find anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I'm working with Python and cython&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-06-28T09:48:34.990" Title="Events prediction with time series of continuous variables as features" Tags="|predictive-modeling|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8776" PostTypeId="2" ParentId="8755" CreationDate="2015-11-10T11:46:16.910" Score="2" Body="&lt;p&gt;Among Naive Bayes assumptions the main one is that features are conditionally independent. For our problem we would have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Play|Outlook,Person) \propto P(Play)P(Outlook|Play)P(Name|Play)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address question &lt;em&gt;is Harry going to play on a sunny day?&lt;/em&gt;, you have to compute the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Yes|Sunny,Harry) = P(Yes)P(Sunny|Yes)P(Harry|Yes)$$&#xA;$$P(No|Sunny,Harry) = P(No)P(Sunny|No)P(Harry|No)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and choose the probability with bigger value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is what theory says. To address your question I will rephrase the main assumption of Naive Bayes. The assumptions that features are independent given the output means basically that the information given by joint distribution can be obtained by product of marginals. In plain English: assume you can find if Harry plays on sunny days if you only know how much Harry plays in general and how much anybody plays on sunny days. As you can see, &lt;em&gt;you simply you would not use the fact that Harry plays on sunny days&lt;/em&gt; &lt;strong&gt;even if you would have had that record in your data&lt;/strong&gt;. Simply because Naive Bayes assumes there is no useful information in the interaction between the features, and this is the precise meaning of conditional independence, which Naive Bayes relies upon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said if you would want to use the interaction of features than you would have either to use a different model, or simply add a new &lt;em&gt;combined&lt;/em&gt; feature like a concatenation of factors of names and outlook. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a conclusion when you do not include names in your input features you will have a &lt;em&gt;general wisdom&lt;/em&gt; classifier like &lt;em&gt;everybody plays no matter outlook&lt;/em&gt;, since most of the instances have play=yes. If you include the name in your input variables you allow to alter that &lt;em&gt;general wisdom&lt;/em&gt; with something specific to player. So your classifier &lt;em&gt;wisdom&lt;/em&gt; would look like &lt;em&gt;players prefer in general to play, no matter outlook, but Marry like less to play less on Rainy&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is however a potential problem with Naive Bayes on your data set. This problem is related with the potential big number of levels for variable Name. In order to approximate the probability there is a general thing that happens: more data, better estimates. This probably would happen with variable Outlook, since there are two levels and adding more data would probably not increase number of levels. So the estimates for Outlook would be probably better with more data. However for name you will not have the same situation. Adding more instances would be possible perhaps only by adding more names. Which means that on average the number of instances for each name would be relatively stable. And if you would have a single instance, like it is the case for Harry, you do not have enough data to estimate $P(Harry|No)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it happens this problem can be alleviated using smoothing. Perhaps Laplace smoothing (or a more general for like Lindstone) is very helpful. The reason is that estimates based on maximum likelihood have big problems with cases like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it answers at least partially your question.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-11-10T11:46:16.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8777" PostTypeId="2" ParentId="8775" CreationDate="2015-11-10T12:59:30.270" Score="2" Body="&lt;p&gt;First of all, you will not be able to prove anything with a model, you will have false positives/negatives. With a good model you may be able show what variables may be an indicator of component failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In problems like this feature generation can have the most important influence on the accuracy of the model. The time stamps can be used for aggregation. For example, you may aggregate metrics per device per hour. The metrics/features that you might create for input into the model might be average/max temperature or fan speed, rate of change in temperature or fan speed, number of seconds device was above some threshold temperature or fan speed, boolean indicator of voltage spike, etc. There could be any number of features you may create. You can then find which features are not strong predictors and remove these columns to reduce noise, if need be.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-10T12:59:30.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8778" PostTypeId="2" ParentId="5996" CreationDate="2015-11-10T13:09:21.747" Score="1" Body="&lt;p&gt;For the numbers you mention, I think all alternatives should work (read: you'll be able to finish your analysis in reasonable time).&#xA;I recommend on a design that can lead to significantly faster results.&lt;/p&gt;&#xA;&lt;p&gt;As answered before, in general postgresql is faster than mongo, some times more than 4 times faster.&lt;br /&gt;&#xA;See for example &lt;a href=&quot;http://www.enterprisedb.com/postgres-plus-edb-blog/marc-linster/postgres-outperforms-mongodb-and-ushers-new-developer-reality&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You said that you are interested in improving performance in joins.&#xA;I assume that you are interested in calculating similarities among the entities (e.g., post, author) so you'll mainly join the table with it self (e.g., by post or author) and aggregate.&lt;/p&gt;&#xA;&lt;p&gt;Add to that the fact that after the initial loading your database will be read only, what make the problem very suitable to index usage. You won't pay for index update since you won't have any and I guess you have the extra storage for the index.&lt;/p&gt;&#xA;&lt;p&gt;I would have use postgres and store the data in two tables:&lt;/p&gt;&#xA;&lt;p&gt;create table posts(&#xA;post_id integer,&#xA;url varchar(255),&#xA;author_id integer&#xA;)&#xA;;&lt;/p&gt;&#xA;&lt;p&gt;-- Load data and then create the indices.&#xA;-- That will lead to a faster load and better indices&#xA;alter table posts add constraint posts_pk primary key(post_id);&#xA;create  index post_author on posts(author_id);&lt;/p&gt;&#xA;&lt;p&gt;create table comments(&#xA;comment_id integer,&#xA;post_id integer,&#xA;author_id integer,&#xA;comment varchar(255)&#xA;)&#xA;;&#xA;alter table comments add constraint comments_pk primary key(comment_id);&#xA;create  index comment_author on comments(author_id);&#xA;create  index comment_post on comments(post_id);&lt;/p&gt;&#xA;&lt;p&gt;Then you can compute author similarity based on comments in queries like&#xA;select&#xA;m. author_id as m_author_id,&#xA;a. author_id as a_author_id,&#xA;count(distinct m.post_id) as posts&#xA;from&#xA;comments as m&#xA;join&#xA;comments as a&#xA;using&#xA;(post_id)&#xA;group by&#xA;m.author_id&#xA;, a. author_id&lt;/p&gt;&#xA;&lt;p&gt;In case you are interested in tokenzing the words in the comment for nlp, add another table for that but remember that it will increase the volume of your data significantly.Usually it is better not to represent the entire tokenization in the database.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="98307" LastEditDate="2020-08-16T18:02:30.027" LastActivityDate="2020-08-16T18:02:30.027" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8779" PostTypeId="2" ParentId="8762" CreationDate="2015-11-10T13:11:38.847" Score="1" Body="&lt;p&gt;First you need to distinguish private and public addresses (check wikipedia IP_address#Private_addresses).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Private IPs:&lt;/strong&gt; the best you can do is compute if 2 addresses COULD be on the same network or not, then you need clues from other features to KNOW if it is the case or no.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Public IPs:&lt;/strong&gt;&#xA;For geographic distance, you may want to check web services/API that try to map IP and geographical locations (one google search turned this one for instance &lt;a href=&quot;https://www.db-ip.com/&quot; rel=&quot;nofollow&quot;&gt;enter link description here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point which could be interesting is the &quot;organisational distance&quot;, from the IP address you can try to identify the owner of the address (the ISP), check ARIN for instance &lt;a href=&quot;http://whois.arin.net/rest/net/NET-8-8-8-0-1/pft?s=8.8.8.8&quot; rel=&quot;nofollow&quot;&gt;http://whois.arin.net/rest/net/NET-8-8-8-0-1/pft?s=8.8.8.8&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From there you can try to figure out if two addresses belong to the same organisation or not. You will have to find some way to tell if the organisation is an ISP with private customers or a company with their own network. Please be also aware that some organisations have started to resell blocks from their IPV4 addresses to other companies for money with the effect that addresses that used to be in the same organisation/location can now be thousands of miles away in different companies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it would be wise to consider these informations as probabilities only.&lt;/p&gt;&#xA;" OwnerUserId="14003" LastActivityDate="2015-11-10T13:11:38.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8784" PostTypeId="1" AcceptedAnswerId="8798" CreationDate="2015-11-10T17:00:58.527" Score="1" ViewCount="87" Body="&lt;p&gt;I have a list of accounts as data set and I need to group the accounts that refer to the same user using many features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking to use machine learning( but I'm new in this domain), because I know the group of each account for the training data set.&lt;br&gt;&#xA;ex of training data:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;account-id   Feature1    Feature2    class(Group)&#xA;1            T1          P4          Gr1&#xA;2            T2          P4          Gr1&#xA;3            T3          P2          Gr2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is in the testing of data and when a new account arrive for a new group not learned before in the training set.&#xA;&lt;br&gt;&#xA;ex of testing data:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;account-id   Feature1   Feature2&#xA;4             T5         P5&#xA;5             T6         P5&#xA;6             T3         P2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The groups of the testing data should be as following:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;account-id   Feature1   Feature2   class(Group)&#xA;4             T5         P5         Gr3&#xA;5             T6         P5         Gr3&#xA;6             T3         P2         Gr2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The accounts 4 and 5 are in a new group (Gr3) which is not learned before in the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how could I group the new data under a new class that is not defined before in the learning phase ? and which algorithm can I use to solve this issue ?&lt;/p&gt;&#xA;" OwnerUserId="14023" LastEditorUserId="11097" LastEditDate="2015-11-11T08:32:16.633" LastActivityDate="2015-11-11T12:20:43.383" Title="how to make new class from the test data" Tags="|machine-learning|classification|multiclass-classification|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8785" PostTypeId="1" AcceptedAnswerId="8787" CreationDate="2015-11-10T18:16:17.677" Score="6" ViewCount="13958" Body="&lt;p&gt;I am working on a breast cancer dataset (&lt;a href=&quot;http://kdd.org/kdd-cup/view/kdd-cup-2008&quot; rel=&quot;noreferrer&quot;&gt;http://kdd.org/kdd-cup/view/kdd-cup-2008&lt;/a&gt;). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A section of the report that I have to write is &quot;benchmark models&quot; and I have no idea what that means. I googled the term and it doesn't seem to be something well defined in data mining. Any idea what that means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="14029" LastActivityDate="2015-11-10T18:34:19.713" Title="What is a benchmark model?" Tags="|data-mining|classification|decision-trees|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8786" PostTypeId="2" ParentId="694" CreationDate="2015-11-10T18:28:17.970" Score="6" Body="&lt;p&gt;I wrote up this post detailing some of my personal favorites:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stackabuse.com/the-best-machine-learning-libraries-in-python/&quot; rel=&quot;noreferrer&quot;&gt;The Best Machine Learning Libraries in Python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since over 30 different libraries are mentioned, I won't post them all here, but these are among the most popular:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://github.com/scikit-learn/scikit-learn&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/Theano/Theano&quot; rel=&quot;noreferrer&quot;&gt;Theano&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://github.com/lisa-lab/pylearn2&quot; rel=&quot;noreferrer&quot;&gt;Pylearn2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/perone/Pyevolve&quot; rel=&quot;noreferrer&quot;&gt;Pyevolve&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://github.com/numenta/nupic&quot; rel=&quot;noreferrer&quot;&gt;NuPIC&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://github.com/clips/pattern&quot; rel=&quot;noreferrer&quot;&gt;Pattern&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/BVLC/caffe&quot; rel=&quot;noreferrer&quot;&gt;Caffe&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strike&gt;(Sorry, can't link to the Github repos since my rep is still &amp;lt; 10...)&lt;/strike&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Added links to Github repos.&lt;/p&gt;&#xA;" OwnerUserId="14030" LastEditorUserId="14030" LastEditDate="2015-11-11T15:03:48.870" LastActivityDate="2015-11-11T15:03:48.870" CommentCount="3" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="8787" PostTypeId="2" ParentId="8785" CreationDate="2015-11-10T18:34:19.713" Score="10" Body="&lt;p&gt;Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which you can benchmark, you might compare to the best naive solution (guessing the mean, guessing the majority class, etc) or a very simple model (a simple regression, K Nearest Neighbors).  If the field is well studied, you should probably benchmark against the current published state of the art (and possibly against human performance when relevant).&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-10T18:34:19.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8788" PostTypeId="2" ParentId="1177" CreationDate="2015-11-10T19:19:26.100" Score="1" Body="&lt;p&gt;Your problem is separable. You can 1) transform your vectors into a weighted combination of the &quot;raw&quot; vector dimensions and &lt;strong&gt;then&lt;/strong&gt; 2) compute similarity or distance with things like correlation or Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem 1: Weights are typically linear, just a matrix multiplied by your feature vector to rotate and scale it however you like. A matrix multiplication can even reduce your vectors dimensions (all the way down to a scalar score, if you like). The state of the art for dimension reduction and optimal feature vector transformation is SVD/PCA/LSI/LDA/NMF. FYI, the &quot;eigenvectors&quot; that some of these approaches produce are simply the columns or rows of your transformation matrix, the eigenvalues are the relative importance/weight of each of your new vectors. Some eigenvalues will be zero if you're doing dimension reduction and there's redundancy in your features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem 2: Some common distance metrics based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Norm_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;vector norms&lt;/a&gt; are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;L_1 (Manhattan)&lt;/li&gt;&#xA;&lt;li&gt;L_2 (Euclidian)&lt;/li&gt;&#xA;&lt;li&gt;L_inf (Supremum)&lt;/li&gt;&#xA;&lt;li&gt;Cosine (vector product)&lt;/li&gt;&#xA;&lt;li&gt;Fractional (p-norm)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The state-of-the-art distance metric for high-dimensional problems is &lt;a href=&quot;https://en.wikipedia.org/wiki/Norm_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;fractional distance or p-norm&lt;/a&gt; which is like Euclidean distance/norm, but the exponent is set to a value between 0 and 1 rather than 2.&lt;/p&gt;&#xA;" OwnerUserId="14031" LastEditorUserId="14031" LastEditDate="2015-11-11T16:50:46.357" LastActivityDate="2015-11-11T16:50:46.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8789" PostTypeId="1" AcceptedAnswerId="8791" CreationDate="2015-11-10T20:02:19.077" Score="3" ViewCount="503" Body="&lt;p&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to model compositional data problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compositional data is when each example or sample is a vector that sums to 1 (or 100%). In my case, I am interested in the composition of minerals in a rock and I have sensors that tell me the sum of the minerals but not the components that make up the sum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, lets say I have two minerals, $m_1$ and $m_2$, that are made up of 3 elements (like copper and other elements from the periodic table) which form a vector of length 3:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 = [0.1, 0.3, 0.6]&#xA;m2 = [0.6, 0.2, 0.2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If a rock has 25% of $m_1$ and 75% of $m_2$, the sensor reading produces the sum of the two minerals (shown in bottom-left subplot below): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align}&#xA;&amp;amp;0.25*m_1 + 0.75*m_2 \\&#xA;=&amp;amp;0.25*[0.1, 0.3, 0.6] + 0.75*[0.6, 0.2, 0.2] \\&#xA;=&amp;amp;[0.475, 0.225, 0.3]&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LHbDs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LHbDs.png&quot; alt=&quot;Mineral composition example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know how to model and solve the problem of unmixing a composition into its underlying components, where the sum of the elements is normalized to 100% (e.g. $0.25m_1 + 0.75m_2$ has the same composition as $0.50m_1 + 1.50m_2$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, my example is simplistic; in reality a composition can have more than just 2 minerals (up to 3000) and each mineral is made up of 118 elements, not just 3 (all the elements of periodic table - though many elements will be zero). The elemental composition of a mineral is assumed to be known (definition of $m_1$ and $m_2$ in the example). Also, the sensor reading is noisy - each element of the observed composition is assumed to have Gaussian noise.&lt;/p&gt;&#xA;" OwnerUserId="10683" LastEditorUserId="10683" LastEditDate="2017-11-14T20:13:58.530" LastActivityDate="2017-11-15T08:51:10.637" Title="How to model compositional data?" Tags="|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8790" PostTypeId="2" ParentId="8789" CreationDate="2015-11-10T20:44:50.153" Score="2" Body="&lt;p&gt;If I understand correctly, in your problem the mineral-element vector $m1$ and $m2$ is &lt;em&gt;known&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is the case, I would suggest LASSO regression. Suppose you have $n$ minerals $m_1$, $m_2$, ..., $m_n\in\mathbb{R}^p$ arranged into a matrix $M=[m_1;m_2,...,m_n]\in\mathbb{R}^{n\times p}$, and an unknown mixing vector $x\in\mathbb{R}^n$ (in your case, $x=[0.25; 0.75]$). Your sensor measures $y=M^\top x\in\mathbb{R}^n$ and you want to recover $x$ from $y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The LASSO regression aims to solve $\hat{x}=\arg\min_x ||y-M^\top x||_2 + \lambda||x||_1$. The $\ell_1$ penalty promotes sparsity in $x$ and could be beneficial when you have a lot of candidate minerals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think you need any special handling due to the compositional nature of the data, other than normalizing the final $\hat{x}$ if needed.&lt;/p&gt;&#xA;" OwnerUserId="14006" LastActivityDate="2015-11-10T20:44:50.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8791" PostTypeId="2" ParentId="8789" CreationDate="2015-11-10T20:45:53.720" Score="2" Body="&lt;p&gt;First normalize the result vector. E.g. &lt;code&gt;[.95, .45, .6]&lt;/code&gt; by dividing by 2 (sum of the members); giving &lt;code&gt;[.475, .225, .3]&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $x$ be the share of the first mineral, than $(x-1)$ is the share of the second mineral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solve the three linear equation which must give the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$.1 * x + .6 * (1-x) = .475$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$.3 * x + .2 * (1-x) = .225$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$.6 * x + .2 * (1-x) = .3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result is $x = 1/4$ as expected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above proposed solution works of course only if the number of minerals is less or equal&#xA;the number of elements.&#xA;In the update of the question is clearly stated, that this is not the case (3000 minerals and 118 elements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets simulate this oposite case on a small example with 3 minarals and 2 elements.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 &amp;lt;- c(0.2, 0.8)  &#xA;m2 &amp;lt;- c(0.4, 0.6)  &#xA;m3 &amp;lt;- c(0.9, 0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and with the mix of minerals&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; x &amp;lt;- c(.25, .65, .1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which produce a measurement of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; t(matrix(c(m1,m2,m3),3,2, byrow= T))  %*% x&#xA;&#xA; [,1]&#xA; [1,]  0.4&#xA; [2,]  0.6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives following linear equations &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_1 + m_2 + m_3 = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$.2 m_1 + .4 m_2 + .9 m_3 = .4$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$.8 m_1 + .6 m_2 + .1 m_3 = .6$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution of the equations is not unique &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_3 \in &amp;lt;0, 1 / 3.5&amp;gt;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_1 = 2 - 2 * m_2 - 4.5 * m_3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_2 = 1 - 3.5 * m_3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some alternative solutions are provided below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  0 1 0&#xA;  0.125 0.825 0.050&#xA;  0.375 0.475 0.150&#xA;  0.5 0.3 0.2&#xA;  0.625 0.125 0.250&#xA;  0.71428571 0.00000000 0.28571429&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is of course oversimplified example, but shows that you should carefully select the optimization goal,&#xA;as there could be more &quot;equaly good&quot; solutions.&#xA;For example with promoting sparsity you will find the solution &lt;code&gt;[0 1 0]&lt;/code&gt; which is far from our used mix.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10683" LastEditDate="2017-11-15T08:51:10.637" LastActivityDate="2017-11-15T08:51:10.637" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8792" PostTypeId="1" CreationDate="2015-11-11T01:32:56.953" Score="4" ViewCount="87" Body="&lt;p&gt;Is there a fundamental difference between building a set on N logistic regressions in 1 vs all fashion, as compared to training a single mutlinomial logistic regression? Put another way, are there any optimization techniques that treat the 1 to N classes logistic regression problem in a way that's markedly different from N independent regressions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively is seems like the answer ought to be yes, since there should be a lot of information sharing between various problems if two classes are similar. But since I'm not entirely well versed on how  common 1 to N solvers actually work, I can't tell if I'm right or these problems are treated in ways that fundamentally the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that we can see that there could be a difference between the two models, but I'm not entirely sure. Googling the matter revealed a few arcane discussing about the subject, but I was unable to find an authoritative discussion of the matter. &lt;/p&gt;&#xA;" OwnerUserId="7999" LastActivityDate="2016-02-09T03:44:58.210" Title="Difference between multinomia logistic regression and N 1 vs all binary" Tags="|logistic-regression|multiclass-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8793" PostTypeId="2" ParentId="8792" CreationDate="2015-11-11T01:44:01.247" Score="1" Body="&lt;p&gt;Quote from &lt;em&gt;Alan, Agresti. &quot;Categorical data analysis.&quot; A John Wiley and Sons, Inc. Publication, Hoboken, New Jersey, USA (2002)&lt;/em&gt;, page 273:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The separate-fitting estimates differ from the ML estimates for simultaneous fitting of the J-1 logits. They are less efficient, tending to have larger standard errors. However, Begg and Gray 1984 showed that the efficiency loss is minor when the response category having highest prevalence is the baseline.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-11-11T01:44:01.247" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8794" PostTypeId="1" AcceptedAnswerId="8795" CreationDate="2015-11-11T06:34:17.577" Score="1" ViewCount="344" Body="&lt;p&gt;I'm dumb but still trying to understand the code provided from this &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap6.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;e-book&lt;/a&gt; on deep learning, but it doesn't explain where the &lt;code&gt;n_in=40*4*4&lt;/code&gt; comes from. &lt;code&gt;40&lt;/code&gt; is from the 40 previous feature maps, but what about the &lt;code&gt;4*4&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AfDJj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AfDJj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; net = Network([&#xA;        ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28), &#xA;                      filter_shape=(20, 1, 5, 5), &#xA;                      poolsize=(2, 2), &#xA;                      activation_fn=ReLU),&#xA;        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 12), &#xA;                      filter_shape=(40, 20, 5, 5), &#xA;                      poolsize=(2, 2), &#xA;                      activation_fn=ReLU),&#xA;        FullyConnectedLayer(&#xA;            n_in=40*4*4, n_out=1000, activation_fn=ReLU, p_dropout=0.5),&#xA;        FullyConnectedLayer(&#xA;            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),&#xA;        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)], &#xA;        mini_batch_size)&#xA;&amp;gt;&amp;gt;&amp;gt; net.SGD(expanded_training_data, 40, mini_batch_size, 0.03, &#xA;            validation_data, test_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For instance, what if I do a similar analysis in 1D as shown below, which should that &lt;code&gt;n_in&lt;/code&gt; term be?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; net = Network([&#xA;        ConvPoolLayer(image_shape=(mini_batch_size, 1, 81, 1), &#xA;                      filter_shape=(20, 1, 5, 1), &#xA;                      poolsize=(2, 1), &#xA;                      activation_fn=ReLU),&#xA;        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 1), &#xA;                      filter_shape=(40, 20, 5, 1), &#xA;                      poolsize=(2, 1), &#xA;                      activation_fn=ReLU),&#xA;        FullyConnectedLayer(&#xA;            n_in=40*???, n_out=1000, activation_fn=ReLU, p_dropout=0.5),&#xA;        FullyConnectedLayer(&#xA;            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),&#xA;        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)], &#xA;        mini_batch_size)&#xA;&amp;gt;&amp;gt;&amp;gt; net.SGD(expanded_training_data, 40, mini_batch_size, 0.03, &#xA;            validation_data, test_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="13625" LastEditorUserId="13625" LastEditDate="2015-11-11T06:44:00.210" LastActivityDate="2021-02-18T20:50:22.667" Title="Understanding convolutional pooling sizes (deep learning)" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8795" PostTypeId="2" ParentId="8794" CreationDate="2015-11-11T07:10:12.170" Score="0" Body="&lt;p&gt;In the given example from the e-book, the number &lt;span class=&quot;math-container&quot;&gt;$4$&lt;/span&gt; comes from &lt;span class=&quot;math-container&quot;&gt;$(12-5+1) \over 2$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$12$&lt;/span&gt; is the input image size &lt;span class=&quot;math-container&quot;&gt;$(12*12)$&lt;/span&gt; of the second constitutional layer; 5 is the filter size (5*5) used in that layer; and &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt; is the poolsize.&lt;br /&gt;&#xA;This is similar to how you get the number &lt;span class=&quot;math-container&quot;&gt;$12$&lt;/span&gt; from the first constitutional layer: &lt;span class=&quot;math-container&quot;&gt;$12= {(28-5+1) \over 2}$&lt;/span&gt;. It's well explained in your linked chapter.&lt;/p&gt;&#xA;&lt;p&gt;Regarding your &amp;quot;For instance&amp;quot; code, your 6th line is not correct:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 1),&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The number 12 should be replaced by &lt;span class=&quot;math-container&quot;&gt;$(81-5+1)\over2$&lt;/span&gt; which unfortunately is not an integer. You may want to change the filter_shape in the first layer to (6,1) to make it work. In that case, your 6th line should be:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;ConvPoolLayer(image_shape=(mini_batch_size, 20, 38, 1),&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;and your 11th line should be:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;n_in=40&lt;em&gt;17&lt;/em&gt;1, n_out=1000, activation_fn=ReLU, p_dropout=0.5),&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14006" LastEditorUserId="29169" LastEditDate="2021-02-18T20:50:22.667" LastActivityDate="2021-02-18T20:50:22.667" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8796" PostTypeId="2" ParentId="806" CreationDate="2015-11-11T07:32:15.217" Score="4" Body="&lt;p&gt;I'd like to refer to how you should choose a performance measure.&#xA;Before that I'll refer to the specific question of accuracy and AUC.&lt;/p&gt;&#xA;&lt;p&gt;As answered before, on imbalanced dataset using the majority run as a classifier will lead to high accuracy what will make it a misleading measure.&#xA;AUC aggregate over confidence threshold, for good and bad.&#xA;For good, you get a weight result for all confidence level.&#xA;The bad is that you are usually care only about the confidence level you will actually use and the rest are irrelevant.&lt;/p&gt;&#xA;&lt;p&gt;However, I want to remark about choosing a proper performance measure for a model.&#xA;You should compare a model by its goal. The goal of a model is not a question of machine learning or statistic, it is a question of the business domain and its needs.&lt;/p&gt;&#xA;&lt;p&gt;If you are digging for gold (a scenario in which you have huge benefit from a true positive, not too high cost of a false positive) then recall is a good measure.&lt;/p&gt;&#xA;&lt;p&gt;If you are trying to decide whether to perform a complex medical procedure on people (high cost of false positive, hopefully a low cost of false negative), precision is the measure you should use.&lt;/p&gt;&#xA;&lt;p&gt;There are plenty of measures you can use.&#xA;You can also combine them in various ways.&lt;/p&gt;&#xA;&lt;p&gt;However, there is no universal &amp;quot;best&amp;quot; measure.&#xA;There is the best measure for your needs, the one that you should maximize to maximize your benefit.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="-1" LastEditDate="2022-02-06T17:46:54.090" LastActivityDate="2022-02-06T17:46:54.090" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8798" PostTypeId="2" ParentId="8784" CreationDate="2015-11-11T12:20:43.383" Score="0" Body="&lt;p&gt;I think you need to read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;nofollow&quot;&gt;Online learning&lt;/a&gt;, it refers to learning when new data is being constantly added. In these cases you need an algorithm that can update itself as new data arrives (i.e. it doesn't need to recalculate itself from scratch). In other words, &lt;em&gt;incrementally&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are incremental versions for support vector machines (SVMs) and for neural networks. Also, bayesian networks can be made to work incrementally.&lt;/p&gt;&#xA;" OwnerUserId="413" LastActivityDate="2015-11-11T12:20:43.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8799" PostTypeId="1" AcceptedAnswerId="8898" CreationDate="2015-11-11T14:27:51.393" Score="4" ViewCount="2655" Body="&lt;p&gt;I have some data that I would like to cluster with k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the features is the hour of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the hour '23' is considered far from the hour '0'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I map the data so that the boundary will be cyclic?&lt;/p&gt;&#xA;" OwnerUserId="14044" LastActivityDate="2015-11-19T07:39:28.247" Title="Boundary conditions for clustering" Tags="|k-means|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8800" PostTypeId="2" ParentId="8799" CreationDate="2015-11-11T15:06:18.373" Score="0" Body="&lt;p&gt;Modular arithmetic. Generally you would do &lt;code&gt;end - start mod 24&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;julia&amp;gt; mod(-23,24)&#xA;1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I seem to recall some programming languages treat mod of negative numbers in a different way, so check your implementation first.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2015-11-11T15:06:18.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8801" PostTypeId="2" ParentId="8757" CreationDate="2015-11-11T15:16:22.027" Score="1" Body="&lt;p&gt;What happens in Spectral Clustering is simply finding some blocks in the data according to the property for which your matrix is defined. If it's similarity you get the similar data points. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;The first one is how to introduce weights when performing spectral clustering. Like in the Shi-Malik, Ng-Jordan-Weiss versions, where only the distances are used.&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Do not get confused. All what you need is to introduce your similarity measure which can be weighted. Just insert all the weights into a classical spectral clustering version (Shi-Malik or Ng or whatever) and there you go!&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;The second one is that my dissimilarity measure does not exists sometimes. In these cases, it has zero weight. How to introduce these cases in the algorithm? I suppose the zero weight solves the problem if I know how to introduce them.&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think this should not be a big issue. Just put zero and try your algorithm with a mock data and see what happens. If didn't work, try to fill missing values by some statistical indicators e.g. mean of the data and see the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please write a comment about how it worked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-11-11T15:16:22.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8802" PostTypeId="1" CreationDate="2015-11-11T15:37:59.103" Score="6" ViewCount="96" Body="&lt;p&gt;Let's say I have 2 datasets, each from a set of experiments.  Dataset A measures a set of properties X for set S, while dataset B measures properties Y for set T.  X and Y are highly correlated, and S and T have (not perfect) overlap.  To give an example, I might have &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A   |   ID   |   pKa    |  log P &#xA;    |  cid12 |   3.51   |   1.2&#xA;    |  cid51 |   2.32   |   .9&#xA;&#xA;B   |   ID   | Pol Srf A|  cLogP &#xA;    |  cid12 |   48.5   |   1.5&#xA;    |  cid88 |   61.2   |   .9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to use the shared information here (since all of these are physical properties) in a multitask neural net setup.  For examples in both S and T, I have a target vector as long as X union Y.  But for members exclusive to S or T, I am missing values.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the examples, I have &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; C   |   ID   |   pKa    |  log P  | Pol Srf A|  cLogP &#xA;     |  cid12 |   3.51   |   1.2   |   48.5   |   1.5&#xA;     |  cid51 |   2.32   |   .9    |    na    |   na&#xA;     |  cid88 |    na    |   na    |   61.2   |   .9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I train, do I simply zero fill these outputs, then not backpropagate error coming from these nodes?  So for  cid12, error from all nodes backpropagates, but for cid51, only error from pKa and logP?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the case, is that a standard procedure implemented in most NN libraries (Theano, for example) and what is that called?&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2020-02-10T19:02:00.263" Title="Error on multitask neural nets where all outputs not observed for every example" Tags="|machine-learning|multitask-learning|backpropagation|theano|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8804" PostTypeId="1" CreationDate="2015-11-11T18:31:14.463" Score="1" ViewCount="1895" Body="&lt;p&gt;As far as I understand, canopy clustering preprocessing procedure is not implemented in scikit-learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a python library for this?&lt;/p&gt;&#xA;" OwnerUserId="14048" LastActivityDate="2015-11-11T18:31:14.463" Title="Canopy clustering implementation in Python" Tags="|python|clustering|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8805" PostTypeId="2" ParentId="1253" CreationDate="2015-11-11T19:37:13.993" Score="0" Body="&lt;p&gt;Deep Learning has been around for a long time. CNN,RNN,Boltzmann Machines sure look like new techniques but they were developed long time back. Check the &lt;a href=&quot;https://en.wikipedia.org/wiki/Deep_learning#History&quot; rel=&quot;nofollow noreferrer&quot;&gt;history of deep learning&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The resurgence of Deep Learning is due to the fact that the computational powers increased exponentially from that time. With a laptop equipped with GPUs you can train a complex Deep Learning model is a very small time as compared to its previous days. Deep learning models are also very efficient empirically. Current state of the art in Images,Speech and many fields is Deep Learning models.&lt;/p&gt;&#xA;&lt;p&gt;I believe due to these factors we can see lot of NLP/ML community has shifted its focus on Deep Learning.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="98307" LastEditDate="2020-08-05T08:48:18.683" LastActivityDate="2020-08-05T08:48:18.683" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8806" PostTypeId="1" AcceptedAnswerId="8807" CreationDate="2015-11-11T19:52:07.653" Score="1" ViewCount="2016" Body="&lt;p&gt;I have performed a linear regression analysis to two series of data, each of which has &lt;code&gt;50&lt;/code&gt; values. I did the analysis in SPSS and as a result got a table which says that my &lt;code&gt;adjusted R squared&lt;/code&gt; is &lt;code&gt;0.145&lt;/code&gt; and its significance is &lt;code&gt;0.004&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Being &lt;code&gt;0.004 &amp;lt; 0.05&lt;/code&gt;, I assume my &lt;code&gt;adjusted R squared&lt;/code&gt; is significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Does it mean my &lt;code&gt;adjusted R squared&lt;/code&gt; is &lt;em&gt;credible&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What does happen if you get a significance which is &lt;code&gt;&amp;gt; 0.05&lt;/code&gt;? Does it imply the &lt;code&gt;adjusted R squared&lt;/code&gt; can be trusted with credibility but also that the two datasets are not or poorly correlated?&lt;/p&gt;&#xA;" OwnerUserId="13481" LastActivityDate="2023-03-06T09:27:25.537" Title="On interpreting the statistical significance of R squared" Tags="|statistics|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8807" PostTypeId="2" ParentId="8806" CreationDate="2015-11-11T20:09:31.253" Score="2" Body="&lt;p&gt;The p-value is the strength of evidence against the null hypotheses. In this case the null is that the coefficient is equal to zero. So your p-value says that this is very weak evidence against the null so you model is likely to be describing the underlying system of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R-squared describes the percent of variation that is explained by the model. Your value is very low; 14.5%. Of all the &quot;activity&quot; in the data your model is only explaining 14.5% of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you have a situation were model is most likely explaining variation in data but not explaining very much of it.  I would suggest altering the model and refitting. &lt;/p&gt;&#xA;" OwnerUserId="10814" LastActivityDate="2015-11-11T20:09:31.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8808" PostTypeId="2" ParentId="8806" CreationDate="2015-11-11T22:08:33.383" Score="0" Body="&lt;p&gt;The accepted answer is correct in terms of explaining the interpretation of R^2 as being the amount of variation in the dependent variable caused by the independent variable. It ranges from 0.0 (0%) to 1.00 (100%) of correlated variation (for a linear regression), so if it 100%, all of the changes in Y (dependent or predictor variable) can be attributed to changes in X (independent variable). Regression coefficients represent the mean change in the response variable for one unit of change in the predictor variable while holding other predictors in the model constant. This statistical control that regression provides is important because it isolates the role of one variable from all of the others in the model.&lt;/p&gt;&#xA;&lt;p&gt;The p-value interpretation is the likelihood one can reject the null hypothesis. The p-value for each term tests the null hypothesis that the coefficient is equal to zero (no effect). A low p-value (say &amp;lt; 0.05) indicates that you can reject the null hypothesis. In other words, a predictor that has a low p-value is likely to be a meaningful addition to your model because changes in the predictor's value are related to changes in the response variable.&lt;/p&gt;&#xA;&lt;p&gt;Conversely, a larger (insignificant) p-value suggests that changes in the predictor are not associated with changes in the response.&lt;/p&gt;&#xA;&lt;p&gt;You will never get a p = 0 in a real scenario, because there is no way to eliminate the case that the null is correct, because there will be some error in the observations due to measurement alone, leaving out other factors. You need to establish how certain you need to be to feel comfortable. There is nothing magical in the p = 0.05 number, it is one that was established as a standard and now is taken as doctrinal by many not understanding it. If you can feel comfortable in your situation with an 80% certainty that the null can be rejected, then there is nothing wrong with that level.&lt;/p&gt;&#xA;&lt;p&gt;The real reason that I wanted to add an answer is that the other answer doesn't deal with your using &lt;strong&gt;adjusted R^2&lt;/strong&gt;, sometimes called &lt;strong&gt;R bar squared&lt;/strong&gt;. &lt;strong&gt;Adjusted R^2&lt;/strong&gt; is &lt;strong&gt;not&lt;/strong&gt; R^2 and should not be confused for it. R^2 &amp;gt;= adjusted R^2 and if you are only dealing with correlation between two variables, as hinted at in your question, you should use R^2. R^2 measures the correlation between two variables. Adjusted R^2 has additional factors that attempt to take account of the phenomenon of the R2 automatically and spuriously increasing when extra explanatory variables are added to the model.&lt;/p&gt;&#xA;&lt;p&gt;Adjusted R2 is particularly useful in the feature selection stage of model building.&lt;/p&gt;&#xA;" OwnerUserId="10663" LastEditorUserId="91724" LastEditDate="2023-03-06T09:27:25.537" LastActivityDate="2023-03-06T09:27:25.537" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="8809" PostTypeId="1" CreationDate="2015-11-12T00:11:39.770" Score="2" ViewCount="136" Body="&lt;p&gt;In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;               &lt;img src=&quot;https://i.stack.imgur.com/XNGnC.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.&lt;/p&gt;&#xA;" OwnerUserId="14053" LastEditorUserId="14053" LastEditDate="2015-11-12T16:08:35.237" LastActivityDate="2015-11-13T09:11:31.793" Title="What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8810" PostTypeId="2" ParentId="8709" CreationDate="2015-11-12T02:24:41.690" Score="1" Body="&lt;p&gt;NMF is not a classifier -- it is a transformation (or, more specifically, a factorization).  We have $V \sim W H$, where (depending on your perspective) each column of $H$ corresponds to some &quot;proto-image&quot;, and each row of $W$ corresponds to the amount that the corresponding row of $V$ is made up of each image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach that I might suggest (and this is not the only approach) is to take your lower-dimensional data $W$ and use some other classification method on top of that.  From this perspective there is no need for the basis vectors in $H$; you operate only with $W$.  So this is like seeing NMF as a preprocessing dimensionality reduction step before some out-of-the-box classifier.  An advantage here is that the rank of your decomposition is often much lower than your original dimensionality, which can (among other things) reduce the necessary training time.&lt;/p&gt;&#xA;" OwnerUserId="14050" LastActivityDate="2015-11-12T02:24:41.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8811" PostTypeId="1" CreationDate="2015-11-12T07:16:06.783" Score="1" ViewCount="301" Body="&lt;p&gt;I am working to create a SVM binary classifier for classification of Tweets based on news class &quot;Crime&quot; and &quot;Non Crime&quot;. I have downloaded a dataset of 6400 rows from various sources and training my model on it. While I am able to achieve over 95% accuracy, what concerns me is about its performance on new datasets. The system will be deployed on live streaming tweets, so how can I put in a feedback system such as my model is continuously updates itself? As in, say for new crime type news source which were not covered in training model, how to incorporate them further?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am asking a broad methodology question, not some specific program related question as I would like to figure out the implementation myself :)&lt;/p&gt;&#xA;" OwnerUserId="14057" LastActivityDate="2016-01-11T17:11:03.493" Title="Improving SVM binary classification model on new dataset" Tags="|data-mining|classification|text-mining|svm|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8812" PostTypeId="2" ParentId="8032" CreationDate="2015-11-12T07:18:48.460" Score="17" Body="&lt;p&gt;Know I'm a bit late, but to get probabilities from &lt;code&gt;xgboost&lt;/code&gt; you should specify &lt;code&gt;multi:softmax&lt;/code&gt; objective like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgboost(param, data = x_mat, label = y_mat,nround = 3000, objective='multi:softprob')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the &lt;code&gt;?xgb.train&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;multi:softprob same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The result contains predicted probabilities of each data point belonging to each class.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14059" LastActivityDate="2015-11-12T07:18:48.460" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8813" PostTypeId="1" AcceptedAnswerId="9114" CreationDate="2015-11-12T09:44:13.153" Score="3" ViewCount="404" Body="&lt;p&gt;I performed a regression analysis with two datasets, each of which has size &lt;code&gt;50&lt;/code&gt;. One dataset is called &lt;code&gt;Spatial %&lt;/code&gt; and the other &lt;code&gt;Min values&lt;/code&gt;, and I wanted to check whether the two are correlated. I did the analysis in &lt;code&gt;SPSS&lt;/code&gt; and the resulting scatterplot is as follows:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GO49G.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GO49G.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am not that much experienced but it seems to me that a line is not the perfect fit for this scatterplot. Would a power line fit better? Or what else do you suggest?&lt;/p&gt;&#xA;" OwnerUserId="13481" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-11-29T08:50:05.273" Title="Non-linear regression line fit" Tags="|regression|correlation|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8814" PostTypeId="2" ParentId="8813" CreationDate="2015-11-12T13:10:01.687" Score="1" Body="&lt;p&gt;You're right, a basic linear regression is unlikely to fit this data.  You need some form of &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Nonlinear_regression&quot; rel=&quot;nofollow&quot;&gt;non-linear regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basic forms of this (including exponential regression, mentioned by @Dawny33)  can be found in most spreadsheet software, including Excel. Packages like scikit learn and others will allow for more flexibility.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="11097" LastEditDate="2015-11-19T17:42:13.157" LastActivityDate="2015-11-19T17:42:13.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8815" PostTypeId="2" ParentId="8811" CreationDate="2015-11-12T13:17:55.257" Score="3" Body="&lt;p&gt;Online learning for SVMs is supported only for &lt;a href=&quot;https://stackoverflow.com/questions/23056460/does-the-svm-in-sklearn-support-incremental-online-learning&quot;&gt;certain SVM options&lt;/a&gt; (a linear SVM with SGD can easily update for example).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are implementations out there to address the issue of online learning for all SVMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/26041/can-svm-do-stream-learning-one-example-at-a-time&quot;&gt;https://stats.stackexchange.com/questions/26041/can-svm-do-stream-learning-one-example-at-a-time&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model isn't terribly large it also probably isn't too bad to train a new one on all data periodically.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-11-12T15:52:56.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8816" PostTypeId="2" ParentId="8813" CreationDate="2015-11-12T14:09:13.323" Score="1" Body="&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt; Linear Regression is not a nice fit for this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-linear regression, as @jamesmf suggested, can be a nice option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, this looks like a nice fit for &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat501/node/372&quot; rel=&quot;nofollow noreferrer&quot;&gt;exponential regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The graph of exponential regression looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5QjCl.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5QjCl.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, adjusting the parameters to fit your data should go good.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://onlinestatbook.com/2/transformations/box-cox.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Box-Cox transformation&lt;/a&gt; can also be used for fitting the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have taken a sample data set, and fitted a Box-Plot transformation, with relevant parameters for transforming it to look somewhat like the plot of your data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FpgnL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FpgnL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the noise, as it is a quick and dirty implementation.  But, yeah a &lt;a href=&quot;http://onlinestatbook.com/2/transformations/box-cox.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Box Cox transformation&lt;/a&gt; should also be a nice way to fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R code for the above plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lambda = +9.6 &#xA;plot(BoxCox(elec, lambda))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/fpp/fpp.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;elec&lt;/a&gt; is a sample data set.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-12T15:06:32.857" LastActivityDate="2015-11-12T15:06:32.857" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8817" PostTypeId="2" ParentId="8809" CreationDate="2015-11-12T14:38:57.153" Score="4" Body="&lt;p&gt;The equation that you show &lt;em&gt;calculates&lt;/em&gt; (the negative of) a gradient to the objective function. The value $\delta_j$ is the rate of change of the objective function for an isolated change of output of a neuron &lt;em&gt;before&lt;/em&gt; applying the activation function for the neuron. The factor of $f′(net_i)$ is required because it is &lt;em&gt;correct&lt;/em&gt; - it is due to &lt;a href=&quot;https://en.wikipedia.org/wiki/Chain_rule&quot; rel=&quot;nofollow&quot;&gt;the chain rule&lt;/a&gt; applied twice - once due to the linear relationship linking weights between layers, and again to get the gradient pre-activation function in the next layer down. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An interpretation of $\delta_i$ is &quot;the (negative) gradient of objective function with respect to a pre-activation value of a neuron the network&quot;. If, somehow, the pre-activation value could be changed - ignoring &lt;em&gt;how&lt;/em&gt; it could be changed - then $\delta_i$ tells you the linear scale of the impact on the objective function, at least for very small changes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This value is not normally used directly in weight updates, but is fed deeper backwards into the networks layers if there are yet lower layers to update, because it can be used for weight updates in those layers using the same rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The property of low gradients associated with strong activations is because you previously chose &lt;code&gt;tanh&lt;/code&gt; as an activation function, and is not because of the way you calculate the gradient. Once you chose &lt;code&gt;tanh&lt;/code&gt;, you were not really free to choose a different backpropagation gradient calculation. Or rather, there is not much to gain - and potentially a lot to lose - by doing so.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the purpose of this for weight updating? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The value shown is not used directly, but used to derive $\frac{\partial J}{\partial W_{ij}}$ gradients for weights in deeper layers (where $J$ is value of your objective function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of your equations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial J}{\partial W_{ij}} = -\delta_j y_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very simple, so it has been folded in to your delta update rule. However, doing that is hiding something important. Out of all the intermediate values you calculate during back propagation, $\frac{\partial J}{\partial W_{ij}}$ is the key one. It is the gradient of the objective function &lt;em&gt;with respect to parameters that you can change&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a set of gradients for  $\frac{\partial J}{\partial W_{ij}}$, this can be used in various ways.  Using this gradient &lt;em&gt;directly&lt;/em&gt; for weight updates using the delta rule (as in your equation $\Delta W_{ij} = \eta \delta_j y_i$) is a basic approach. The purpose of this rule is to reduce the value of the objective function by taking a small step in the direction that you have just calculated &lt;em&gt;will&lt;/em&gt; reduce the value of objective function. The step size needs to be small (multiplied by a factor $\eta$) because you have only measured the gradient at one point, and don't know for what step size the relationship still holds - complicated by the fact that you are making a similar step in multiple dimensions at once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The delta update rule is a relatively weak update method compared to e.g. Nesterov momentum, Adagrad, Rmsprop optimisers which will use the calculated gradient as input, but adjust the update weights based on history of previous gradients. These optimisers can deal with low but consistent gradient values and make large updates.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How would the networks behave differently &#xA;  if one would remove or change this factor?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not advisable as effectively you would be optimising against a &lt;em&gt;different&lt;/em&gt; network architecture than the feed-forward network you had constructed. The most likely way they would behave is to either diverge or settle on a larger value of your objective function than you would otherwise see. Making small changes (such as a minimum absolute gradient) can work, but usually such a tweak is best done consistently, by changing the activation function as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a few edge cases, you might consider &quot;lying&quot; in the gradient calculation. A trivial case could be considered the ReLU activation function ($y = 0 (if x \lt 0), y = x (otherwise)$) where the gradient is technically undefined at $x=0$, but can safely be treated as either 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alteration to updates that is more usually done (by e.g. momentum variants, Adagrad, RMSProp et al) is to use the gradient as-is, but then multiply update amounts by various factors based on history or estimates of higher-order derivatives, because you can guess that the gradient will not switch sign if you update a weight by a smaller or larger amount. These approaches replace the delta update rule with update mechanisms that are more robust when gradients become small.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-11-13T09:11:31.793" LastActivityDate="2015-11-13T09:11:31.793" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="8818" PostTypeId="1" CreationDate="2015-11-12T15:41:30.003" Score="3" ViewCount="98" Body="&lt;p&gt;I want to increase the coverage of wordnet by increasing the synset count.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way in which I can proceed and increase sysnets. I am trying to work on Hindi wordnet with Princeton wordnet  as the  standard&lt;/p&gt;&#xA;" OwnerUserId="12378" LastEditorUserId="11097" LastEditDate="2015-11-13T08:55:49.560" LastActivityDate="2016-03-12T10:54:13.663" Title="how to increase the coverage of wordnet using wikipedia" Tags="|knowledge-base|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8819" PostTypeId="2" ParentId="8818" CreationDate="2015-11-12T15:56:47.663" Score="1" Body="&lt;p&gt;You should look at Open Multilingual Wordnet (&lt;a href=&quot;http://compling.hss.ntu.edu.sg/omw/&quot; rel=&quot;nofollow&quot;&gt;http://compling.hss.ntu.edu.sg/omw/&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They used wiktionary (not wikipedia, because wikipedia lemmas are nouns only) to get translations and synsets. EDIT: In the extended version, there is also a Hindi wordnet with 3,593 synsets, see &lt;a href=&quot;http://compling.hss.ntu.edu.sg/omw/summx.html&quot; rel=&quot;nofollow&quot;&gt;http://compling.hss.ntu.edu.sg/omw/summx.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a non-open Hindi wordnet at &lt;a href=&quot;https://catalog.ldc.upenn.edu/LDC2008L02&quot; rel=&quot;nofollow&quot;&gt;https://catalog.ldc.upenn.edu/LDC2008L02&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-11-12T16:59:50.057" LastActivityDate="2015-11-12T16:59:50.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8820" PostTypeId="1" CreationDate="2015-11-12T16:09:57.543" Score="13" ViewCount="23531" Body="&lt;p&gt;I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried various algorithms (Naive Bayes, Random Forest, AODE, C4.5) and all of them have unacceptable false negative ratios. Random Forest has the highest overall prediction accuracy (99.5%) and the lowest false negative ratio, but still misses 79% of positive classes (i.e. fails to detect 79% of malignant tumors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how I can improve this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="14029" LastActivityDate="2019-05-06T21:45:59.437" Title="Unbalanced classes -- How to minimize false negatives?" Tags="|classification|random-forest|decision-trees|class-imbalance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8821" PostTypeId="2" ParentId="8820" CreationDate="2015-11-12T16:21:22.213" Score="10" Body="&lt;p&gt;&lt;a href=&quot;http://www.cs.cmu.edu/~qyj/IR-Lab/ImbalancedSummary.html&quot; rel=&quot;noreferrer&quot;&gt;Class imbalance&lt;/a&gt; is a very common problem.  You can either oversample the positive class (or undersample the negative) or add class weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing to remember in this case is that accuracy is not a very useful metric here.  You might consider &lt;a href=&quot;http://www.r-bloggers.com/illustrated-guide-to-roc-and-auc/&quot; rel=&quot;noreferrer&quot;&gt;AUC&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;noreferrer&quot;&gt;F1 score&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Changing your decision threshold may seem appealing, but will obviously lead to (in this case likely drastically) increased false positives (though perhaps FPs aren't as bad as FNs in the case of medical diagnosis, if tests aren't prohibitively expensive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more in-depth analysis of options in the case of class imbalance is provided &lt;a href=&quot;http://www.cs.ox.ac.uk/people/vasile.palade/papers/Class-Imbalance-SVM.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-13T00:37:16.063" LastActivityDate="2015-11-13T00:37:16.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8824" PostTypeId="2" ParentId="8820" CreationDate="2015-11-12T19:01:19.127" Score="3" Body="&lt;p&gt;Even though the answer in reality is always &lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;, you can make your class attributes not labels but floating point numbers, ie 1.0 and 0.0 (or 100 and 0).  That is, you can frame it is a &lt;em&gt;regression&lt;/em&gt; problem, not classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the predicted output will likewise be numbers on that spectrum, ie &lt;em&gt;probabilities&lt;/em&gt; not labels.  Your current implementation is essentially equivalent to a regression model with a threshold of 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With such an output, you or your client can define a threshold that is acceptable (eg 0.3).  Of course there will be more false positives then, but for some applications, like detecting cancer, that is optimal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oversampling (the positives) or undersampling (the negatives) are also ways to fix this, but must be done thoughtfully, can sacrifice accuracy, and still sacrifices the control to move the threshold after creating the data and training.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="12363" LastEditDate="2019-05-06T20:08:27.493" LastActivityDate="2019-05-06T20:08:27.493" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="8825" PostTypeId="1" CreationDate="2015-11-13T04:36:59.007" Score="4" ViewCount="231" Body="&lt;p&gt;I have a data set of user rating for movie as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_name, product_name, user_rating&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I am using this data to recommend new movie to user (collaborative filtering). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have another data set which show relationships between user like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_1, CO_WORKER, user_2&#xA;user_1, FATHER_OF, user_3&#xA;user_3, FRIEND_OF, user_4 etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now what are the ways to include 'relationship data' along with 'ratings data' to produce better recommendations ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example : user_1 might like the movies which user_2 likes since they are co_workers&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2020-08-18T16:27:13.150" Title="Make use of relationships on recommendation systems" Tags="|machine-learning|data-mining|python|recommender-system|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8826" PostTypeId="2" ParentId="8825" CreationDate="2015-11-13T05:26:22.873" Score="1" Body="&lt;p&gt;There are a number of ways to make use of this information, including various graph-based methods of recommendation.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://snap.stanford.edu/class/cs224w-2013/projects2013/cs224w-038-final.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Collaborative Filtering using Weighted BiPartite Graph&#xA;Projection&lt;br /&gt;&#xA;A Recommendation System for Yelp&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;It's also common to use multiple approaches to build a hybrid recommendation system.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="98307" LastEditDate="2020-08-18T16:27:13.150" LastActivityDate="2020-08-18T16:27:13.150" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8827" PostTypeId="2" ParentId="5417" CreationDate="2015-11-13T07:17:47.380" Score="0" Body="&lt;p&gt;It depends. If you need a fast way to mine streams of data and use adaptative training of data sets, the best tool is SAMOA, because it could be easily integrated with Storm or S4 stream processing engines. If you need only to process batch data in a fast and distributed manner, the Spark MLLib would be the best solution among them.&lt;/p&gt;&#xA;" OwnerUserId="14078" LastActivityDate="2015-11-13T07:17:47.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8829" PostTypeId="1" AcceptedAnswerId="8830" CreationDate="2015-11-13T09:29:35.127" Score="1" ViewCount="298" Body="&lt;p&gt;I created a DocumentTermMatrix  for text mining using RTextTools. The rows for this DocumentTermMatrix  correspond to dataframe rows and matix columns correspond to words. My question is : How can I get the words (labels vector) for examining the DocumentTermMatrix  ? In other words, How can I get the vector of these 904 words?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(RTextTools,quietly=TRUE)&#xA;data(USCongress)&#xA;&#xA;doc_matrix &amp;lt;- create_matrix(USCongress$text, language=&quot;english&quot;,   removeNumbers=TRUE, stemWords=TRUE, removeSparseTerms=.998)&#xA;&#xA;dim(USCongress)&#xA;[1] 4449    6&#xA;&#xA;dim(doc_matrix)&#xA;[1] 4449  904&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-11-13T10:00:41.030" Title="Examining a DocumentTermMatrix in RTextTools" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8830" PostTypeId="2" ParentId="8829" CreationDate="2015-11-13T10:00:41.030" Score="2" Body="&lt;p&gt;a documenttermmatrix is a simple_triplet_matrix. You can turn this into a simple matrix with the as.matrix command and then use all matrix functions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# turn into simple matrix&#xA;mat &amp;lt;- as.matrix(doc_matrix)&#xA;&#xA;# vector of the words &#xA;word_vector &amp;lt;- colnames(mat)&#xA;&#xA;# Dataframe containing words and their frequency&#xA;df_words &amp;lt;- data.frame(words = colnames(mat), frequency = colSums(mat), row.names = NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-11-13T10:00:41.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8833" PostTypeId="2" ParentId="8401" CreationDate="2015-11-13T13:40:09.927" Score="1" Body="&lt;p&gt;This can be thought of as an &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Active_learning_(machine_learning)&quot; rel=&quot;nofollow&quot;&gt;active learning&lt;/a&gt; problem. You want feedback from the learning process to influence what kind of input examples you next train on.  Active learning is still a difficult problem in most settings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @cohoz points out, hill climbing is an intuitive option in this case, but only if you always want to increase or decrease your measure of frustration. If instead the goal is to explore frustration as a function of as much of the music parameter feature space as possible, you will want to read about active learning and uncertainty.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-13T14:59:39.310" LastActivityDate="2015-11-13T14:59:39.310" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8834" PostTypeId="1" CreationDate="2015-11-13T17:10:42.153" Score="2" ViewCount="76" Body="&lt;p&gt;I am reading about Boltzmann machines and according the formulas the joint probability of the states of all units is $$ P(X = x) = \frac{1}{Z} e^{-\frac{1}{2T} \sum_i\sum_j {x_i x_j w_{ij}}} $$&#xA;$$ Z = \sum_x e^{-\frac{1}{2T}\sum_i\sum_j x_i x_j w_{ij}} $$&#xA;In thse formulas it is assumed that $ w_{ii} = 0$. However it is infeasible to calculate the quantity $Z$ directly. Therefore people use Gibbs sampling to sample different states. I don't understand how exactly the sampling is used to estimate probabilities.&lt;/p&gt;&#xA;" OwnerUserId="14091" LastActivityDate="2015-11-13T17:10:42.153" Title="Estimating probability using boltzmann machine" Tags="|machine-learning|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8835" PostTypeId="2" ParentId="8732" CreationDate="2015-11-13T17:57:42.067" Score="0" Body="&lt;p&gt;The types of methods that would work depend on the nature of the data being captured in relation to the output classification.  If the gestures are linearly separable in the 5-dimensional input space, a simple logistic regression is cheap and fast.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively a weak but very cheap method would be to find the mean input vector for each of the gestures, and simply decide which gesture vector the input is closest to at any point.  The downside there you might need another class for &quot;not gesturing.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible to gather a lot of test data from many users to pre-train a model, and long as readings don't vary too drastically between users,you wouldn't need to train on a by-user basis (on the wearable itself).  In most cases prediction is efficient, so you wouldn't be too limited by computational power. &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-13T17:57:42.067" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8836" PostTypeId="2" ParentId="8069" CreationDate="2015-11-13T18:49:00.407" Score="0" Body="&lt;p&gt;If you evaluate the effectiveness of your classifier using a &lt;a href=&quot;https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf&quot; rel=&quot;nofollow&quot;&gt;ROC curve&lt;/a&gt;, you don't have to worry so much about class skew:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ROC curves have an attractive property: they are insensitive to&#xA;  changes in class distribution. If the proportion of positive to&#xA;  negative instances changes in a test set, the ROC curves will not&#xA;  change.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are using R, the &lt;a href=&quot;https://rocr.bioinf.mpi-sb.mpg.de/&quot; rel=&quot;nofollow&quot;&gt;ROCR package&lt;/a&gt; is excellent for constructing ROC curves (and many other metrics, too). &lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2015-11-13T18:49:00.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8837" PostTypeId="1" CreationDate="2015-11-13T19:16:49.497" Score="1" ViewCount="514" Body="&lt;p&gt;I have a large number of different XML documents with associated style sheets (SEC financial filings). Are there any natural language processing (or other) tools that I can use to try to automatically extract section headings?&lt;/p&gt;&#xA;" OwnerUserId="6403" LastActivityDate="2016-01-15T13:47:00.920" Title="How can I figure out section headings in a document?" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8838" PostTypeId="1" AcceptedAnswerId="8839" CreationDate="2015-11-13T21:41:18.970" Score="1" ViewCount="3872" Body="&lt;p&gt;I saw this piece of code in my project:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%let num = test;&#xA;%let x=%sysfunc(trim(&amp;amp;num));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why could not  I write: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%let x= %trim(&amp;amp;num);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why did I need to use sysfunc?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under what circumstances can I call a function inside a macro without using sysfunc?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2016-04-01T11:56:31.933" Title="Why do we need to use sysfunc when we call a SAS function inside a SAS macro" Tags="|sas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8839" PostTypeId="2" ParentId="8838" CreationDate="2015-11-13T23:03:59.497" Score="2" Body="&lt;p&gt;Without the sysfunc(), the expression will not be evaluated. You will not be assigning the value of the expression trim(&amp;amp;num) to the macrovariable, but rather the whole expression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to store the result of an expression, you need to execute that function with sysfunc()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#z3514sysfunc.htm&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#z3514sysfunc.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-13T23:03:59.497" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8840" PostTypeId="1" CreationDate="2015-11-14T05:51:38.297" Score="2" ViewCount="2008" Body="&lt;p&gt;I am working on a certain kaggle competition and users there say that they are using &gt;5000 features and training a XGBoost or Random Forest on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mentioned post is here: &lt;a href=&quot;https://www.kaggle.com/c/walmart-recruiting-trip-type-classification/forums/t/17258/feature-counts&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/walmart-recruiting-trip-type-classification/forums/t/17258/feature-counts&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried doing so myself after exploding the feature space and creating interaction features, but it takes forever to run on my 16GB macbook. &#xA;For example: (96000,1000) data took around 5 hrs to finish to train an RF with n_estimators=1000,max_depth=40, cv=3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tuning such a model would take weeks and it only uses 1000 predictors....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now here are some of my doubts:&#xA;1. Is my python scikit environment properly loaded. I use anaconda distribution. Are these running times normal?&lt;/p&gt;&#xA;" OwnerUserId="3211" LastActivityDate="2016-12-11T08:56:01.853" Title="How many features do you generally use for your ML Model?" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ClosedDate="2016-12-11T20:46:00.623" ContentLicense="CC BY-SA 3.0" />
  <row Id="8841" PostTypeId="1" CreationDate="2015-11-14T05:58:56.993" Score="4" ViewCount="3243" Body="&lt;p&gt;I am trying to solve a classification problem on a highly imbalanced data set. I am using SMOTE to over sample the minority samples and down sample the majority ones. After creating a balanced data set, I applied the random forest model. But, the prediction error for the minority class is extremely high even after using a balanced data set. What could possibly be going wrong?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(DMwR)&#xA;new.data &amp;lt;- SMOTE(Clicked ~ ., train, perc.over = 600, perc.under =  80)&#xA;table(new.data$Clicked)&#xA;&#xA;rand.forest &amp;lt;- randomForest(Clicked ~., data=new.data, mtry = 7,&#xA;                            importance = TRUE, proximity=TRUE, ntree = 1000&#xA;                            )&#xA;&#xA;&#xA;#confusion matrix&#xA;table(yhat.rf, test$Clicked)&#xA;&#xA;yhat.rf   0   1&#xA;      0 889  47&#xA;      1  57   7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11110" LastEditorUserId="25849" LastEditDate="2020-10-03T08:15:58.730" LastActivityDate="2020-10-03T08:15:58.730" Title="oversampling plus down sampling using smote not working on random forests" Tags="|machine-learning|classification|r|class-imbalance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8842" PostTypeId="2" ParentId="8841" CreationDate="2015-11-14T06:22:14.190" Score="2" Body="&lt;p&gt;Balancing your dataset does not guarantee an even prediction split.  Imagine the case where your features cannot separate between positive and negative examples at all. In this case, even if you balance the dataset, you will learn a decision boundary that essentially randomly guesses on each example. You would therefore expect that your prediction error would mirror the distribution of the majority/minority classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this scenario you might but have strongly predictive features or you may have insufficient data.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-14T06:22:14.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8846" PostTypeId="1" CreationDate="2015-11-14T19:49:28.633" Score="1" ViewCount="72" Body="&lt;p&gt;Fuzzy logic was utilized to derive a performance indicator of some manufacturing facilities in an uncertain environment. There are some historical data obtained from some manufacturing facilities’ rigorous assessments. How come can I use this data to verify and validate the fuzzy rules? Any insights and feedback would be appreciated.   &lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="11097" LastEditDate="2015-11-18T12:27:28.307" LastActivityDate="2015-11-18T12:27:28.307" Title="Does anyone have any thought how come the fuzzy rules can be verified and validated?" Tags="|data-mining|reference-request|fuzzy-logic|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8847" PostTypeId="1" AcceptedAnswerId="8849" CreationDate="2015-11-15T00:45:03.647" Score="21" ViewCount="44721" Body="&lt;p&gt;In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am completely lost when it comes to feature extraction techniques in python. My advisor convinced me to use images which haven't been covered in class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone direct me towards meaningful documentation or reading or suggest methods to consider?&lt;/p&gt;&#xA;" OwnerUserId="14112" LastEditorUserId="11097" LastEditDate="2016-01-29T06:44:44.977" LastActivityDate="2020-08-03T10:25:09.250" Title="Feature extraction of images in Python" Tags="|python|feature-extraction|image-recognition|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8848" PostTypeId="2" ParentId="8847" CreationDate="2015-11-15T06:53:23.637" Score="8" Body="&lt;p&gt;This great tutorial covers the basics of convolutional neuraltworks, which are currently achieving state of the art performance in most vision tasks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot;&gt;http://deeplearning.net/tutorial/lenet.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of options for CNNs in python, including Theano and the libraries built on top of it (I found keras to be easy to use).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefer to avoid deep learning, you might look into OpenCV, which can learn many other types of features, line Haar cascades and SIFT features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_feature2d/py_table_of_contents_feature2d/py_table_of_contents_feature2d.html&quot;&gt;http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_feature2d/py_table_of_contents_feature2d/py_table_of_contents_feature2d.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-15T06:53:23.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8849" PostTypeId="2" ParentId="8847" CreationDate="2015-11-15T07:34:27.787" Score="12" Body="&lt;p&gt;In images, some frequently used techniques for feature extraction are &lt;strong&gt;binarizing&lt;/strong&gt; and &lt;strong&gt;blurring&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binarizing:&lt;/strong&gt; converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how you do it in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import Image&#xA;&#xA;%matplotlib inline  &#xA;&#xA;#Import an image&#xA;image = Image.open(&quot;xyz.jpg&quot;)&#xA;&#xA;image&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example Image: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mkf97.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mkf97.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, convert into gray-scale:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;im = image.convert('L')&#xA;&#xA;im&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will return you this image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AGxy6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AGxy6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the matrix can be seen by running this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array(im)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The array would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[213, 213, 213, ..., 176, 176, 176],&#xA;       [213, 213, 213, ..., 176, 176, 176],&#xA;       [213, 213, 213, ..., 175, 175, 175],&#xA;       ..., &#xA;       [173, 173, 173, ..., 204, 204, 204],&#xA;       [173, 173, 173, ..., 205, 205, 204],&#xA;       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, use a histogram plot and/or a contour plot to have a look at the image features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pylab import *&#xA;&#xA;# create a new figure&#xA;figure()&#xA;gray()&#xA;# show contours with origin upper left corner&#xA;contour(im, origin='image')&#xA;axis('equal')&#xA;axis('off')&#xA;&#xA;&#xA;figure()&#xA;&#xA;&#xA;hist(im_array.flatten(), 128)&#xA;&#xA;show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would return you a plot, which looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/56A0K.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/56A0K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/tWpNy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tWpNy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Blurring:&lt;/strong&gt; Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is how you do it in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import *&#xA;&#xA;&#xA;figure()&#xA;p = image.convert(&quot;L&quot;).filter(ImageFilter.GaussianBlur(radius = 2))&#xA;p.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the blurred image is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0Dx8q.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0Dx8q.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-15T07:34:27.787" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8850" PostTypeId="1" CreationDate="2015-11-15T15:06:41.813" Score="7" ViewCount="1208" Body="&lt;p&gt;I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a way to build the model on any computer and send it up to a server for training and have it return the graphs/accuracies/weights etc. I know there are paid solutions for this but I'm looking for a distributed solution I can run myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a server set up at home which is about to get a CPU and GPU upgrade. I'd like to be able to set it up so that when I'm working on the LAN, or when I'm working remotely on my laptop, I can send code to the server and have it train the model and return to me the results (or save the results if the sender machine is switched off)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any existing solutions to accomplish something like this? I'm not tied to any specific library, but would prefer to stick with Python if possible&lt;/p&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2019-10-06T17:35:19.840" Title="Python distributed machine learning" Tags="|machine-learning|python|neural-network|distributed|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8851" PostTypeId="2" ParentId="8732" CreationDate="2015-11-15T20:27:56.330" Score="3" Body="&lt;p&gt;It might be really useful to provide a bit more details regarding the actual data, produced by your sensor. Can you provide a simple example for any simple gesture? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, let's try to address the problem in the following way: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sensor can be represented as a function &lt;code&gt;f(t)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;range of the function will be &lt;code&gt;{1,2,3,..100}&lt;/code&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So we can draw something like the following (can we?): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/okVvh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/okVvh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can extract various high-level features out of that data, but I would suggest to go with inflection points as the most straightforward option:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qBHIA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qBHIA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those points, basically, define a pattern. I'll use a figure from the &lt;a href=&quot;https://stackoverflow.com/a/19071980/1435741&quot;&gt;different answer&lt;/a&gt; to illustrate the idea. For example, the naive pattern below is used to identify a circle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qzN7M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qzN7M.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case situation is a bit simpler, because sensor output is the only data you can work with. Naturally, you can combine multiple sensors in order to define a bit more complex interaction scenarios:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MKJn1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MKJn1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can take into account &lt;code&gt;f'(t)&lt;/code&gt; in order to distinguish between slow and fast movements, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the data. It does look like these three gestures looks pretty much the same from the data perspective (see visualization below). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uhdcR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uhdcR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It really looks like several features are required to identify the gesture.&lt;/p&gt;&#xA;" OwnerUserId="12469" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-24T19:36:51.307" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8852" PostTypeId="2" ParentId="8850" CreationDate="2015-11-15T21:02:55.333" Score="1" Body="&lt;p&gt;This shouldn't be terribly complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Big picture&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you have Linux on your server: SSH to your server from your work laptop, train the network, receive results (the trained network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Details&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Train the network&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to stick with Python, there are two basic options that I am familiar with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is &lt;em&gt;PyBrain&lt;/em&gt;, a library built specifically for training neural nets. The syntax is fairly straightforward. The dataset structure is a bit unusual (especially if you are used to just using numpy arrays in &lt;em&gt;scikit-learn&lt;/em&gt;), but otherwise, it works rather well. However, it doesn't support GPUs AFAIK, and I don't know how optimized/fast it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second is Google's &lt;em&gt;Tensor Flow&lt;/em&gt;. A bit of a heavy-weight if you just want to run vanilla neural nets, but the syntax is also pretty easy to get around (although very different from &lt;em&gt;PyBrain&lt;/em&gt;). It's also probably faster than anything else you could find in Python. It also supports GPU training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Get the results back&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This depends on the library you choose (or it could be definitely up to you write your own implementation). But both libraries mentioned above will let you return the trained weights, which you can save as csv files and download upon training. Or you could pickle them and get back that way. Also, Tensor Flow has this Tensor Board functionality which should let you visualize the training and the network structure, but I haven't tried it yet, so can't help on that front.&lt;/p&gt;&#xA;" OwnerUserId="12580" LastActivityDate="2015-11-15T21:02:55.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8853" PostTypeId="1" CreationDate="2015-11-16T01:53:50.673" Score="5" ViewCount="79" Body="&lt;p&gt;Given a matrix &lt;code&gt;A&lt;/code&gt;, I want to complete Multidimensional Scaling by hand, instead of using any given R functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, I have calculated the centered matrix &lt;code&gt;B&lt;/code&gt; with the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; n&amp;lt;-nrow(A)&#xA; id&amp;lt;-diag(n)&#xA; e&amp;lt;-diag(id)&#xA; H &amp;lt;- id - (1/n)*e %*% etranspose&#xA; B &amp;lt;- (-1/2)* H %*% A %*% H&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: how can I use my B matrix to complete multidimensional scaling on my A matrix, without the &lt;code&gt;cmdscale&lt;/code&gt; function or anything along those lines?&lt;/p&gt;&#xA;" OwnerUserId="14137" LastEditorUserId="64760" LastEditDate="2018-12-28T00:50:33.870" LastActivityDate="2020-01-22T04:04:15.743" Title="Completing MDS manually in R" Tags="|machine-learning|statistics|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8855" PostTypeId="1" CreationDate="2015-11-16T08:15:55.440" Score="3" ViewCount="128" Body="&lt;p&gt;As far as I understood it, the pooling layer doesn't learn anything. It has several parameters, most important its &lt;code&gt;pool_size&lt;/code&gt; and &lt;code&gt;stride&lt;/code&gt; (see &lt;a href=&quot;http://lasagne.readthedocs.org/en/latest/modules/layers/pool.html#lasagne.layers.MaxPool2DLayer&quot; rel=&quot;nofollow&quot;&gt;Lasagne documentation&lt;/a&gt; for more), but none of those is learned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to learn these two (or one of those) parameters? Are there papers about it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I would guess that it is not possible to add this to the objective function in a meaningful way... but I'd like to see if people thought about it.)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2022-06-17T23:17:05.163" LastActivityDate="2022-06-17T23:17:05.163" Title="Can the size of a pooling layer be learned?" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8856" PostTypeId="2" ParentId="8855" CreationDate="2015-11-16T10:17:28.597" Score="4" Body="&lt;p&gt;No it is not possible to learn those meta parameters from a data set using a learning algorithm. There is no way to calculate a gradient of any objective function with respect to pool_size and stride params. Even if there were, they are typically discrete values and relatively small integers, so cannot be updated in the same way as e.g. the weights are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is search through different values looking for optimal ones, using cross-validation to pick the best. The size and stride meta parameters can also have a large effect on memory use and speed of learning, so it may be the case you are looking to find a good compromise. With stride in lower layers for instance, it may be more effective (in terms of training time required for similar level of accuracy) to have a higher stride value but increase the overall training set size by adding random x/y translations to the training set images.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2015-11-16T10:17:28.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8857" PostTypeId="1" AcceptedAnswerId="8868" CreationDate="2015-11-16T10:57:53.013" Score="6" ViewCount="222" Body="&lt;p&gt;Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into the training set is often questionable (I don't even know how to start separating good from bad ones). I have spent much more time trying to pre-process the files in order to increase the validity of the data, than actually building the prototype.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have only dumped a part of the data that is claimed to be better. There are still data from other sources to complement the missing part in the first set of training data. Given the amount of data, it is practically impossible for me to spend too much time on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is, how do people deal with really horribly, inconsistent data in real life (assuming I am working with addresses, where people often write the wrong postcode due to bad city planning or lack of clear instruction, misspell the name of neighbourhood, plus data got &quot;autocorrected&quot; by excel such that &quot;1-2&quot; becomes &quot;1-Feb&quot;, etc. etc.)? Specifically, how do I ensure the quality of the train classifier given the training set (and cross validation set) has questionable quality?&lt;/p&gt;&#xA;" OwnerUserId="13562" LastEditorUserId="11097" LastEditDate="2015-11-18T02:20:43.477" LastActivityDate="2015-11-18T07:45:28.037" Title="Dealing with training set of questionable quality" Tags="|data-cleaning|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8858" PostTypeId="2" ParentId="8837" CreationDate="2015-11-16T11:30:44.943" Score="1" Body="&lt;p&gt;It seems you are lucky and the section headings are XML-tagged. So you can use XSLT transformations or Python lxml.etree to get rid of them or to extract them.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2015-11-16T12:40:57.647" LastActivityDate="2015-11-16T12:40:57.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8859" PostTypeId="2" ParentId="8857" CreationDate="2015-11-16T12:44:05.323" Score="5" Body="&lt;p&gt;You can use techniques of semi-supervised learning where you have a small clean training set and some dirty data. You than extend your data base by judging how good the other data are and incorporate the &quot;best&quot; data points into your training set.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-11-16T12:44:05.323" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8860" PostTypeId="1" AcceptedAnswerId="8866" CreationDate="2015-11-16T14:41:08.553" Score="17" ViewCount="14812" Body="&lt;p&gt;Bagging is the generation of multiple predictors that works as ensamble as a single predictor. Dropout is a technique that teach to a neural networks to average all possible subnetworks. Looking at the most important Kaggle's competitions seem that this two techniques are used together very often. I can't see any theoretical difference besides the actual implementation. Who can explain me why we should use both of them in any real application? and why performance improve when we use both of them?&lt;/p&gt;&#xA;" OwnerUserId="10938" LastActivityDate="2018-10-18T20:34:24.577" Title="Bagging vs Dropout in Deep Neural Networks" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8861" PostTypeId="1" CreationDate="2015-11-16T15:14:37.750" Score="4" ViewCount="97" Body="&lt;p&gt;Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based.  Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list.  If a ranked list contains n pages, for instance, these metrics give a sum of n terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my own literature review, it seems no metrics incorporate words.  Although some theoretically incorporate &quot;information nuggets&quot;, none incorporate &quot;information nuggets=words&quot; in practice.  I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics.  Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted.  Thoughts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: One obvious objection is that TREC-based assessment of &quot;word relevance&quot; is a vague and even expensive endeavor (asking assessors to mark for words for relevance).  But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2016-01-15T17:58:00.187" Title="Why are there currently no content-based evaluation metrics for information retrieval?" Tags="|information-retrieval|model-evaluations|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8862" PostTypeId="1" CreationDate="2015-11-16T15:40:51.043" Score="4" ViewCount="177" Body="&lt;p&gt;I have a dataset that gives information of a population. For instance, I know the fraction of people that are males (&lt;code&gt;M&lt;/code&gt;) and that are within a certain age range (&lt;code&gt;A&lt;/code&gt;), &lt;code&gt;P(M &amp;amp; A)&lt;/code&gt;, and then I know the fraction of males that live in a certain area (&lt;code&gt;L&lt;/code&gt;), &lt;code&gt;P(M &amp;amp; L)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm interested in computing is &lt;code&gt;P(M &amp;amp; A &amp;amp; L)&lt;/code&gt;, which is the fraction of people that are males, are within a certain age range and live in a certain area. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Baye's formula I can say that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(M &amp;amp; A &amp;amp; L) = P(M &amp;amp; A | L) P(L)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But my dataset only gives &lt;code&gt;P(L)&lt;/code&gt; and not &lt;code&gt;P(M &amp;amp; A | L)&lt;/code&gt;. However, if I assumed that &lt;code&gt;M &amp;amp; A&lt;/code&gt; and &lt;code&gt;L&lt;/code&gt; are independent I have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(M &amp;amp; A | L) = P(M &amp;amp; A) P(L)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How large is the error on &lt;code&gt;P(M &amp;amp; A | L)&lt;/code&gt; if I make this assumption. Do you know of any other method I could use to estimate &lt;code&gt;P(M &amp;amp; A | L)&lt;/code&gt; without assuming independence?&lt;/p&gt;&#xA;" OwnerUserId="13680" LastActivityDate="2016-01-16T23:21:35.497" Title="how to find probability of one or more events to happen from an incomplete data set" Tags="|statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8863" PostTypeId="2" ParentId="8861" CreationDate="2015-11-16T16:50:46.557" Score="3" Body="&lt;p&gt;Using a word-based metric would explicitly favor word-level retrieval methods.  The theory is that (just as you suggest with dwell time), the URL-level metric measures more directly the desired result.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More concretely, consider a search of &quot;alcohol from potatoes.&quot; Assume we have two pages: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) A page that is simply a grocery list (containing &quot;alcohol&quot; and &quot;potatoes&quot;) &#xA;2) A page that describes vodka or moonshine which only uses the words &quot;ethanol&quot;&#xA;   and &quot;potatoes&quot; but not &quot;alcohol.&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A word-level metric would prefer the first result, since it contains both alcohol and potatoes..  A URL-level metric would likely label the second result as more relevant, which would allow a model to form associations beyond simple word-matching (i.e. it could learn that &quot;alcohol&quot; and &quot;vodka&quot; are associated).&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-16T16:50:46.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8865" PostTypeId="2" ParentId="8606" CreationDate="2015-11-16T17:45:01.217" Score="2" Body="&lt;p&gt;There are a group of algorithms (or techniques) called the &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-armed_bandit&quot; rel=&quot;nofollow&quot;&gt;Bandit algorithms&lt;/a&gt;, which deal especially with the problem statement, which is the optimization of Click-through rates of advertisements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is framed in a setting of multiple bandits with vending machines. There are various strategies which can be implemented:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Epsilon-greedy strategy&lt;/li&gt;&#xA;&lt;li&gt;Epsilon-first strategy&lt;/li&gt;&#xA;&lt;li&gt;Epsilon-decreasing strategy&lt;/li&gt;&#xA;&lt;li&gt;Contextual Epsilon strategy&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs_ab.html&quot; rel=&quot;nofollow&quot;&gt;Reference on why Bandit algorithms are better than A/B testing frameworks.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-16T17:45:01.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8866" PostTypeId="2" ParentId="8860" CreationDate="2015-11-16T18:48:32.353" Score="23" Body="&lt;p&gt;Bagging and dropout do not achieve quite the same thing, though both are types of model averaging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bootstrap_aggregating&quot;&gt;Bagging&lt;/a&gt; is an operation across your entire dataset which trains models on a subset of the training data.  Thus some training examples are not shown to a given model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot;&gt;Dropout&lt;/a&gt;, by contrast, is applied to features within each training example.  It is true that the result is functionally equivalent to training exponentially many networks (with shared weights!) and then equally weighting their outputs.  But dropout works on the feature space, causing certain features to be unavailable to the network, not full examples.  Because each neuron cannot completely rely on one input, representations in these networks tend to be more distributed and the network is less likely to overfit.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-16T18:48:32.353" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8867" PostTypeId="2" ParentId="8087" CreationDate="2015-11-16T22:12:09.917" Score="2" Body="&lt;p&gt;It is possible that most of the variance in the dataset exists between input images (or between input and output images). In that case, the most informative principal components serve to separate input examples or to separate the input from the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If only the less informative PCs describe the variance b/w outputs,  it will be harder to distinguish between  outputs than it was in the original feature space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said,  PCA in images is rarely all that helpful. In this example I can see it being appealing, but it would probably much more meaningful and memory-friendly to learn a different lower dimensional feature representation. You could try a simple autoencoder, SIFT/SURF features, or haar-like features.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-17T13:21:04.217" LastActivityDate="2015-11-17T13:21:04.217" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8868" PostTypeId="2" ParentId="8857" CreationDate="2015-11-17T03:01:21.017" Score="6" Body="&lt;p&gt;Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online.  Those are super polished and refined.  But, the real world data is not so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The step where one does the cleaning and scrubbing is called the data pre-processing step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, some nice data cleaning techniques, in addition to @jknappen's excellent answer are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Elimination of zero variance columns/predictors:&lt;/strong&gt; These columns are not important, and they cause the model and the fit to crash and leak errors. So, eliminating them would make complete sense.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Correlated Predictors:&lt;/strong&gt; Reducing the level of correlation between the predictors would be a very nice step in the pre-processing process.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Scaling:&lt;/strong&gt; You must be knowing why scaling is important during pre-process. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Predictor Transformations&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://www.kaggle.com/c/MerckActivity/forums/t/2900/what-do-you-use-to-preprocess-the-data/26554&quot; rel=&quot;nofollow&quot;&gt;nice reference from Kaggle forums&lt;/a&gt; where the pre-processing and cleaning of data sets is discussed.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-18T07:45:28.037" LastActivityDate="2015-11-18T07:45:28.037" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8869" PostTypeId="1" CreationDate="2015-11-17T03:06:51.357" Score="8" ViewCount="1515" Body="&lt;p&gt;I have a database of about 200 documents who's ngrams I have extracted. I want to find the document in my database that is most similar to a query document. In otherwords, I want to find the document in the database that shares the most number of ngrams with the query document. Right now, I can go through each one and compare it one by one, but this will take O(N) time and is expensive if N is very large. I was wondering if there are any efficient data structures or methods in doing efficient similarity search. Thanks&lt;/p&gt;&#xA;" OwnerUserId="14154" LastEditorUserId="13727" LastEditDate="2015-11-17T10:20:03.763" LastActivityDate="2019-05-26T18:32:18.270" Title="Best way to search for a similar document given the ngram" Tags="|nlp|similarity|search|information-retrieval|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="8870" PostTypeId="2" ParentId="8869" CreationDate="2015-11-17T07:21:41.843" Score="1" Body="&lt;p&gt;The data structure typically uses is &lt;a href=&quot;https://en.wikipedia.org/wiki/Inverted_index&quot; rel=&quot;nofollow&quot;&gt;inverted index&lt;/a&gt; (e.g., in databases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that matching all ngram is a good heuristic but you might want to improve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking into account the probability of each term and &lt;a href=&quot;https://en.wikipedia.org/wiki/Stemming&quot; rel=&quot;nofollow&quot;&gt;stemming&lt;/a&gt; are directions you might benefit from.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-11-17T07:21:41.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8871" PostTypeId="1" CreationDate="2015-11-17T07:38:41.220" Score="5" ViewCount="507" Body="&lt;p&gt;What are the features &amp;amp; models that can be used to compute the probability of a certain customer accepting an offer/product from a bank? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some research, I came to know of what is called '&lt;a href=&quot;https://en.wikipedia.org/wiki/Propensity_score_matching&quot; rel=&quot;nofollow&quot;&gt;Propensity Scoring&lt;/a&gt;' and it's definition is very relevant to what I am trying to find. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I failed to find any technical papers going into any appreciable detail. I tried Google Scholar but reached a dead end because the term 'Propensity modeling' is actually a marketing term not exactly academic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also know from my search that a broader term is 'Campaign Response Models' but this turns out to be too broad and I am looking for something very specific. Any references/links that contain some technical details about the modeling process, the features or the choice of models will be very helpful.&lt;/p&gt;&#xA;" OwnerUserId="9540" LastEditorUserId="11097" LastEditDate="2015-11-17T09:59:44.243" LastActivityDate="2016-06-15T14:02:52.080" Title="Features &amp; Models to compute the probability of certain customer accepting an offer/product from a bank?" Tags="|predictive-modeling|reference-request|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8872" PostTypeId="2" ParentId="8869" CreationDate="2015-11-17T09:21:08.463" Score="1" Body="&lt;p&gt;table  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ngram&lt;br&gt;&#xA;docID  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PK (primary key) ngram, docID  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;depending the database may change a bit but this is for TSQL&lt;br&gt;&#xA;x is the document you are matching &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select top(1) with ties *    &#xA;from &#xA;(  select tm.docID, count(*) as count &#xA;     from table td&#xA;     join table tm&#xA;       on tm.docID &amp;lt;&amp;gt; td.docID &#xA;      and tm.ngram = td.ngram &#xA;      and td.docID = x&#xA;    group by tm.docID &#xA;) tt &#xA;order by count desc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The join is on an index (PK) so this is very fast. I do this on a million documents in just a few seconds (with more advanced conditions).     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is going to favor larger documents but that is what you asked for.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question seems to be changing  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;declare table @query (varchar ngram);&#xA;insert into @query values ('ng1'), ('ng2'), ('ng3');&#xA;select top(10) with ties *    &#xA;from &#xA;(  select tm.docID, count(*) as count &#xA;     from table td&#xA;     join @query&#xA;       on tm.ngram = @query.ngram&#xA;    group by tm.docID &#xA;) tt &#xA;order by count desc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-11-17T17:14:27.130" LastActivityDate="2015-11-17T17:14:27.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8873" PostTypeId="1" CreationDate="2015-11-17T16:10:20.000" Score="10" ViewCount="273" Body="&lt;p&gt;Its very common in recommender that we have user product data which have label as an e.g. &quot;click&quot;. In order to learn the model, I need click and no-click data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplest approach to generate is to take user-products pairs which are not found in click data. However, that may be misleading. Example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;user1, product1 (click)&#xA; user2, product2 (click)&#xA; user2, product3 (click)&#xA; user3, product2 (click)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can take user1 with all products except product1 and label them as &quot;no_click&quot; and so on. But this may not be true. Maybe user1 would have clicked product2 if he was shown product2. But just because he was shown other sets of products - he had no opportunity to decide to click/no-click product2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how to address unary data problem?&lt;/p&gt;&#xA;" OwnerUserId="14167" LastActivityDate="2020-08-16T21:51:03.223" Title="User-product positive (click data) available. How to generate negative (no-click data)?" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8874" PostTypeId="2" ParentId="6015" CreationDate="2015-11-17T16:59:45.853" Score="3" Body="&lt;p&gt;Reproducing the accepted answer from CrossValidated here, as it is the most complete and the best answer to this question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to understand this, remember the &quot;ingredients&quot; of random forest classifier (there are some modifications, but this is the general pipeline):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;At each step of building individual tree we find the &lt;strong&gt;best split&lt;/strong&gt; of data&lt;/li&gt;&#xA;&lt;li&gt;While building a tree we use not the whole dataset, but &lt;strong&gt;bootstrap sample&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;We aggregate the individual tree outputs by &lt;strong&gt;averaging&lt;/strong&gt; (actually 2 and 3 means together more general &lt;a href=&quot;http://en.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;nofollow noreferrer&quot;&gt;bagging procedure&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Assume first point. It is not always possible to find the best split. For example in the following dataset each split will give exactly one misclassified object.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/aG3Ps.png&quot; alt=&quot;Example of the dataset with no best split&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I think that exactly this point can be confusing: indeed, the behaviour of the individual split is somehow similar to the behaviour of Naive Bayes classifier: if the variables are dependent - there is no better split for Decision Trees and Naive Bayes classifier also fails (just to remind: independent variables is the main assumption that we make in Naive Bayes classifier; all other assumptions come from the probabilistic model that we choose).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here comes the great advantage of decision trees: we take &lt;strong&gt;any&lt;/strong&gt; split and &lt;strong&gt;continue&lt;/strong&gt; splitting further. And for the following splits we will find a perfect separation (in red).&#xA;&lt;img src=&quot;https://i.stack.imgur.com/ptLS4.png&quot; alt=&quot;Example of the decision boundary&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as we have no probabilistic model, but just binary split, we don't need to make any assumption at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That was about Decision Tree, but it also applies for Random Forest. The difference is that for Random Forest we use Bootstrap Aggregation. It has no model underneath, and the only assumption that it relies is that &lt;strong&gt;sampling is representative&lt;/strong&gt;. But this is usually a common assumption. For example, if one class consist of two components and in our dataset one component is represented by 100 samples, and another component is represented by 1 sample - probably most individual decision trees will see only the first component and Random Forest will misclassify the second one.&#xA;&lt;img src=&quot;https://i.stack.imgur.com/Ndk6t.png&quot; alt=&quot;Example of weakly represented second component&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-17T16:59:45.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8875" PostTypeId="1" AcceptedAnswerId="8876" CreationDate="2015-11-17T18:56:02.027" Score="3" ViewCount="594" Body="&lt;p&gt;I have a database containing sets of words. So for example, I have a database that has:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{happy, birthday, to, you}&#xA;{how, are, you}&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Given a query set, lets say {how, was your, birthday}, I want to find the top K sets in the database that is most similar to my query. Similarity metric used can be something like Jaccard's Index. Right now, I go through the database one by one and calculate the Jaccard Index and keep track of the top K scores found so far. I was wondering if there are any data structures or methods that would allow me to more efficiently find the top K scores. Right now it's a linear search. Thanks &lt;/p&gt;&#xA;" OwnerUserId="14154" LastActivityDate="2015-11-17T20:59:01.573" Title="Finding the top K most similar sets" Tags="|similarity|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8876" PostTypeId="2" ParentId="8875" CreationDate="2015-11-17T20:59:01.573" Score="2" Body="&lt;p&gt;Do you have any information about your data set? Is it sparse, will most similarities be zero? Is the total dictionary small? You could consider a inverted index. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word  query_id&#xA;W1    [1, 3, 6]&#xA;W2    [2, 5]&#xA;W3    [1, 3, 4]&#xA;W4    [2, 3, 4]&#xA;W5    [2, 3, 6]&#xA;&#xA;query_id  query&#xA;1         W1 W3&#xA;2         W2 W4 W4&#xA;3         W1 W3 W4 W5&#xA;4         W3 W4&#xA;5         W2&#xA;6         W1 W5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here W_i is a word, e.g. birthday and query_id is the id of the query in the database. e.g. {how, are, you} might have id 22. Now you get a query {W1 W3 W5}. Aggregate counts on the inverted index. W1 was seen in queries 1, 3, and 6. W3 in 1, 3, and 4, etc.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;query_id  count&#xA;1         2&#xA;2         1&#xA;3         3&#xA;4         1&#xA;6         2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The count will the number of words in common with the incoming query, this is the numerator of the jaccard similarity. So, to find the top k you can start with the queries with the highest count. query_id 3 has the highest count and its similarity is 3/4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a massive database there are techniques like locality sensitive hashing which will basically reduce the search space into a small bucket. The incoming query gets hashed and lands in a bucket. You can then do a linear search with all the queries in this bucket to find the nearest k.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-11-17T20:59:01.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8877" PostTypeId="2" ParentId="8862" CreationDate="2015-11-17T22:41:32.403" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow&quot;&gt;Bayes Theorem&lt;/a&gt; applies to conditional probabilities. &#xA;$&#xA;     P(A|B) = \frac{P(A).P(B|A)}{P(B)} &#xA;    $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question you have posed is one of multiple events occurring together. If the events can be considered to be independent of each other, then,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(A &amp;amp; B &amp;amp; C) = P(A).P(B).P(C)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In your question above, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(M &amp;amp; A &amp;amp; L) = P(M &amp;amp; A).P(L) = P(M).P(A).P(L)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Based on the data provided by you, you should have these probabilities or can derive them by appropriate summations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that .1 probability of males are between 1-10 years of age. Further, probability of males in location L is 0.2. So the probability of males in 1-10 years of age in Location L is 0.1 X 0.2 = 0.02 . This is assuming that all locations have the same probability distribution of age ranges. &lt;/p&gt;&#xA;" OwnerUserId="14175" LastActivityDate="2015-11-17T22:41:32.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8881" PostTypeId="1" CreationDate="2015-11-18T05:47:48.480" Score="3" ViewCount="744" Body="&lt;p&gt;Is it possible to get  recommendation on similar product using Mahout ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data set of movies with following attributes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Movie_name, Actor_1, Actor_2, Actress_1, Actress_2, Director, Theme, Language&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now given a Movie_name the system should  recommend top 3 similar movies based on the attributes .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this be done using Mahout. If yes how ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-12-21T21:46:56.083" Title="Content based recommendation on Mahout" Tags="|python|recommender-system|apache-mahout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8883" PostTypeId="1" AcceptedAnswerId="9933" CreationDate="2015-11-18T09:12:27.920" Score="14" ViewCount="6576" Body="&lt;p&gt;Let's say we are predicting the sales of a shop and my training data has two sets of features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One about the store sales with the dates (the field &quot;Store&quot; is not unique)&lt;/li&gt;&#xA;&lt;li&gt;One about the store types (the field &quot;Store&quot; is unique here)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So the matrix would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-------+-----------+------------+---------+-----------+------+-------+--------------+&#xA;| Store | DayOfWeek |    Date    |  Sales  | Customers | Open | Promo | StateHoliday |&#xA;+-------+-----------+------------+---------+-----------+------+-------+--------------+&#xA;|   1   |     5     | 2015-07-31 |  5263.0 |   555.0   |  1   |   1   |      0       |&#xA;|   2   |     5     | 2015-07-31 |  6064.0 |   625.0   |  1   |   1   |      0       |&#xA;|   3   |     5     | 2015-07-31 |  8314.0 |   821.0   |  1   |   1   |      0       |&#xA;|   4   |     5     | 2015-07-31 | 13995.0 |   1498.0  |  1   |   1   |      0       |&#xA;|   5   |     5     | 2015-07-31 |  4822.0 |   559.0   |  1   |   1   |      0       |&#xA;|   6   |     5     | 2015-07-31 |  5651.0 |   589.0   |  1   |   1   |      0       |&#xA;|   7   |     5     | 2015-07-31 | 15344.0 |   1414.0  |  1   |   1   |      0       |&#xA;|   8   |     5     | 2015-07-31 |  8492.0 |   833.0   |  1   |   1   |      0       |&#xA;|   9   |     5     | 2015-07-31 |  8565.0 |   687.0   |  1   |   1   |      0       |&#xA;|   10  |     5     | 2015-07-31 |  7185.0 |   681.0   |  1   |   1   |      0       |&#xA;+-------+-----------+------------+---------+-----------+------+-------+--------------+&#xA;[986159 rows x 4 columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-------+-----------+------------+---------------------+&#xA;| Store | StoreType | Assortment | CompetitionDistance |&#xA;+-------+-----------+------------+---------------------+&#xA;|   1   |     c     |     a      |         1270        |&#xA;|   2   |     a     |     a      |         570         |&#xA;|   3   |     a     |     a      |        14130        |&#xA;|   4   |     c     |     c      |         620         |&#xA;|   5   |     a     |     a      |        29910        |&#xA;|   6   |     a     |     a      |         310         |&#xA;|   7   |     a     |     c      |        24000        |&#xA;|   8   |     a     |     a      |         7520        |&#xA;|   9   |     a     |     c      |         2030        |&#xA;|   10  |     a     |     a      |         3160        |&#xA;+-------+-----------+------------+---------------------+&#xA;[1115 rows x 4 columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second matrix describes the store type, the assortment groups of item each of them sell and the distance from the nearest competitor store.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in my test data, I only have information in the first matrix without the &lt;code&gt;Customers&lt;/code&gt; and &lt;code&gt;Sales&lt;/code&gt; fields. The aim is to predict the sales field given the &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Store&lt;/li&gt;&#xA;&lt;li&gt;DayofWeek&lt;/li&gt;&#xA;&lt;li&gt;Date&lt;/li&gt;&#xA;&lt;li&gt;Open (whether the store is open)&lt;/li&gt;&#xA;&lt;li&gt;Promo (whether the store is having a promotion)&lt;/li&gt;&#xA;&lt;li&gt;StateHoliday (whether it is a state holiday)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I can easily train a classifier based on the bulleted fields above to predict &lt;code&gt;Sales&lt;/code&gt; but &lt;strong&gt;how can I make use of the second matrix in my training data that I would not get in test data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it logical to assume that the second matrix about the Store types is static and I can easily join it to the test data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens if there are holes in my test data feature set, let's say for some rows in the test data, I don't have the &quot;Promo&quot; values.&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2018-09-05T20:25:08.817" Title="What to do when testing data has less features than training data?" Tags="|machine-learning|regression|feature-selection|feature-construction|missing-data|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8884" PostTypeId="5" CreationDate="2015-11-18T11:18:18.503" Score="0" Body="&lt;p&gt;Neural networks traditionally refer to a network or circuit of biological neurons. The modern usage of the term often refers to &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow&quot;&gt;artificial neural networks&lt;/a&gt; (ANN), which are composed of artificial neurons or nodes - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network designer having had a model of a real system. The goal is good, or human-like, predictive ability.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-18T12:28:10.790" LastActivityDate="2015-11-18T12:28:10.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8885" PostTypeId="4" CreationDate="2015-11-18T11:18:18.503" Score="0" Body="Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network designer having had a model of a real system." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-18T12:28:28.800" LastActivityDate="2015-11-18T12:28:28.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8886" PostTypeId="5" CreationDate="2015-11-18T11:20:47.033" Score="0" Body="&lt;p&gt;&quot;Regression&quot; is a general term for a wide variety of techniques to analyze the relationship between one (or more) dependent variables and independent variables. Typically the dependent variables are modeled with probability distributions whose parameters are assumed to vary (deterministically) with the independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinary least squares (OLS) regression affords a simple example in which the expectation of one dependent variable is assumed to depend linearly on the independent variables. The unknown coefficients in the assumed linear function are estimated by choosing values for them that minimize the sum of squared differences between the values of the dependent variable and the corresponding fitted values.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-02-21T10:53:50.843" LastActivityDate="2016-02-21T10:53:50.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8887" PostTypeId="4" CreationDate="2015-11-18T11:20:47.033" Score="0" Body="Techniques for analyzing the relationship between one (or more) &quot;dependent&quot; variables and &quot;independent&quot; variables.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-02-21T10:53:44.110" LastActivityDate="2016-02-21T10:53:44.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8888" PostTypeId="5" CreationDate="2015-11-18T11:23:07.643" Score="0" Body="&lt;p&gt;A &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_set&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt;, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_set&quot; rel=&quot;nofollow noreferrer&quot;&gt;data set&lt;/a&gt;, is a collection of data - the data points of which are typically related in some way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most commonly a dataset corresponds to the contents of a single database table, or a single statistical data matrix, where every column of the table represents a particular variable, and each row corresponds to a given member of the dataset in question. The dataset lists values for each of the variables, such as height and weight of an object, for each member of the dataset. Each value is known as a datum or data point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term dataset may also be used more loosely, to refer to the data in a collection of closely related tables.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="29575" LastEditDate="2018-01-01T18:57:11.923" LastActivityDate="2018-01-01T18:57:11.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8889" PostTypeId="4" CreationDate="2015-11-18T11:23:07.643" Score="0" Body="A dataset is a collection of data, often in tabular or matrix form.&#xD;&#xA;&#xD;&#xA;This tag is NOT intended for data requests (&quot;where can I find a dataset about ...&quot;) --&gt; see OpenData" OwnerUserId="11097" LastEditorUserId="23305" LastEditDate="2019-07-02T20:02:53.647" LastActivityDate="2019-07-02T20:02:53.647" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8891" PostTypeId="2" ParentId="8799" CreationDate="2015-11-18T12:14:08.953" Score="1" Body="&lt;h2&gt;K-means uses the &lt;em&gt;mean&lt;/em&gt;.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;K-means is designed for least-squares. It only works reliably with (variants of) squared Euclidean distance (= sum of squared deviations).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Counterexample:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Assume you have the two hours 0 and 23.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they get assigned to the same cluster, k-means will compute the &lt;strong&gt;mean&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mean of the two values is 11.5. It is &lt;em&gt;not&lt;/em&gt; 23.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Abusing k-means with a cyclic &quot;distance&quot; may &lt;strong&gt;no longer converge&lt;/strong&gt;, and will return nonsense results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are more cases where the concept of a &lt;em&gt;cluster center&lt;/em&gt; is not viable on cyclic data. For example, &lt;strong&gt;given an event on every full hour, what is the center&lt;/strong&gt;? The arithmetic mean is 12 - but if you take cyclic space into account &lt;strong&gt;every hour is an equally good choice&lt;/strong&gt; in cyclic space. Therefore, the concept of a &quot;center&quot; in cyclic space is fragile.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Alternate clustering algorithms&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You can try e.g. PAM or DBSCAN instead, with an appropriate similarity measure.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Projection techniques&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As pointed out by other answers, you &lt;em&gt;can&lt;/em&gt; project the time to the unit circle via sin/cos(time/24*2pi). By computing the angle of the centroids, you can map this back to a point in time. But once you want additional attributes it gets hard to meaningfully normalize the data (to combine attributes), and you can get undefined time (e.g. if there are two points in a cluster, one at 6 and one at 18).&#xA;I didn't discuss this because I wanted to point out that modifying the distance function is &lt;em&gt;not&lt;/em&gt; a good idea for k-means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2015-11-19T07:39:28.247" LastActivityDate="2015-11-19T07:39:28.247" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8892" PostTypeId="2" ParentId="8606" CreationDate="2015-11-18T12:47:07.477" Score="1" Body="&lt;p&gt;Factorization machines - &lt;a href=&quot;http://www.libfm.org/&quot; rel=&quot;nofollow&quot;&gt;libfm&lt;/a&gt; open source software has been used widely among top companies in recommender applications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check it out. It contains various algorithms to play with (e.g. stochastic gradient descent).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess your data has like &quot;click/view&quot; label for each pair of user-product or say cookie-advertisement saying &lt;em&gt;&quot;whether user clicked or just viewed product&quot;&lt;/em&gt;. You can treat this as binary classification with two classes click or just view.&lt;/p&gt;&#xA;" OwnerUserId="14167" LastEditorUserId="11097" LastEditDate="2015-11-18T15:04:53.363" LastActivityDate="2015-11-18T15:04:53.363" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8893" PostTypeId="2" ParentId="8860" CreationDate="2015-11-18T12:48:39.317" Score="2" Body="&lt;p&gt;I found a comparison of the two kind of nets in &lt;a href=&quot;http://jmlr.csail.mit.edu/proceedings/papers/v28/goodfellow13.pdf&quot; rel=&quot;nofollow&quot;&gt;Max Out Networks&lt;/a&gt; which says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dropout training is similar to bagging (Breiman, 1994), where many different models are trained on different subsets of the data. Dropout training differs from bagging in that each model is trained for only one step and all of the models share parameters. For this training procedure (dropout) to behave as if it is training an ensemble rather than a single model, each update must have a large effect, so that it makes the sub-model induced by that µ fit the current input v well.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope it will be useful.&lt;/p&gt;&#xA;" OwnerUserId="10938" LastEditorUserId="13123" LastEditDate="2015-11-22T11:10:59.393" LastActivityDate="2015-11-22T11:10:59.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8894" PostTypeId="2" ParentId="8873" CreationDate="2015-11-18T14:48:24.850" Score="6" Body="&lt;p&gt;So, there are two problems.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Recording impressions (shows)&lt;/li&gt;&#xA;&lt;li&gt;How to deal with non-impressions&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;For (1) you &lt;em&gt;should&lt;/em&gt; be recording this information. If it is not currently being recorded you should start recording this information. Given that you do not have this information you want to provide recommendations. Fortunately, with just click data you can still create a utility matrix, see &lt;a href=&quot;http://i.stanford.edu/%7Eullman/mmds/ch9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;9.1.1&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;You could then use user- or item-based collaborative filtering as described in the paper. This is basically an exercise in populating the utility matrix and trying to find &amp;quot;scores&amp;quot; for unclicked items. Your recommendation would be an unclicked item with the highest score.&lt;/p&gt;&#xA;&lt;p&gt;For (2) you will still make recommendations on unclicked items. So, that alone is not an issue. You will want to optimize your impressions however. You also cannot have full knowledge where a user can see all possible options. You need to record impressions and understand a number of things.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;show rate of an item&lt;/li&gt;&#xA;&lt;li&gt;click rate of an item&lt;/li&gt;&#xA;&lt;li&gt;how to incorporate new items&lt;/li&gt;&#xA;&lt;li&gt;how to optimize which items to show&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;This is a huge topic and basically this is the problem domain of online advertising. However, a recommendation engine tries to find items of interest in the long tail, which is a bit different from ad optimization. This is a feedback loop to evaluate your recommendation. A/B tests are common. You will want to test click rates and recommendation errors between your current system and new system.&lt;/p&gt;&#xA;&lt;p&gt;Also see &lt;a href=&quot;http://cs.brynmawr.edu/Courses/cs380/fall2006/Herlocker2004.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.cs.umd.edu/%7Esamir/498/Amazon-Recommendations.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorUserId="98307" LastEditDate="2020-08-16T21:51:03.223" LastActivityDate="2020-08-16T21:51:03.223" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8895" PostTypeId="1" AcceptedAnswerId="8897" CreationDate="2015-11-18T20:14:30.133" Score="17" ViewCount="5542" Body="&lt;p&gt;I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is that, as you may know, the dataset is very unbalanced : Around 1300 negative examples (non click) for 1 positive example (click).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load the data&lt;/li&gt;&#xA;&lt;li&gt;Split the dataset into 3 datasets : &#xA;    A = Training (60%)&#xA;    B = Validating (20%)&#xA;    C = Testing (20%)&lt;/li&gt;&#xA;&lt;li&gt;For each dataset (A, B, C), do an under-sampling on each negative class in order to have a ratio of 5 (5 negative example for 1 positive example). This give me 3 new datasets which are more balanced:&#xA;    A’&#xA;    B’&#xA;    C’&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Then I train my model with the dataset A’ and logistic regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Which dataset do I have to use for validation ? B or B’ ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Which dataset do I have to use for testing ? C or C’&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Which metrics are the most relevant to evaluate my model? F1Score seems to be a well used metric. But here due to the unbalanced class (if I use the datasets B and C), the precision is low (under 0.20) and the F1Score is very influenced by low recall/precision. Would that be more accurate to use aucPR or aucROC ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I want to plot the learning curve, which metrics should I use ? (knowing that the %error isn’t relevant if I use the B’ dataset for validating)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for your time !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards.&lt;/p&gt;&#xA;" OwnerUserId="14200" LastEditorUserId="14200" LastEditDate="2015-11-18T20:47:17.757" LastActivityDate="2015-11-19T07:05:58.493" Title="With unbalanced class, do I have to use under sampling on my validation/testing datasets?" Tags="|machine-learning|dataset|sampling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8896" PostTypeId="2" ParentId="8895" CreationDate="2015-11-18T22:38:55.063" Score="5" Body="&lt;p&gt;For &lt;code&gt;1)&lt;/code&gt; and &lt;code&gt;2)&lt;/code&gt;, you want to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) choose a model that performs well on data distributed as you &#xA;   expect the real data will be &#xA;2) evaluate the model on data distributed the same way&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So for those datasets, you shouldn't need to balance the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also try using class weights instead of under/oversampling, as this takes care of this decision for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;3)&lt;/code&gt; you likely want to optimize using whatever metric you will be scored on (if it's a competition).  But if that isn't a consideration, all those models are fine choices.  F1 may be influenced by the low precision, but you want that to be captured.  It's precisely when naive models (like guessing the majority class) can score well by some metrics that scores like F1 are relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for &lt;code&gt;4)&lt;/code&gt; there is nothing wrong with showing whichever metric you end up optimizing on.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-18T22:38:55.063" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8897" PostTypeId="2" ParentId="8895" CreationDate="2015-11-19T02:05:24.543" Score="12" Body="&lt;p&gt;Great question...  Here are some specific answers to your numbered questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;1)&lt;/code&gt; You should cross validate on B not B`.  Otherwise, you won't know how well your class balancing is working.  It couldn't hurt to cross validate on both B &lt;strong&gt;and&lt;/strong&gt; B` and will be useful based on the answer to 4 below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;2)&lt;/code&gt; You should test on both C and C` based on 4 below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;3)&lt;/code&gt; I would stick with F1 and it could be useful to use ROC-AUC and this provides a good sanity check.  Both tend to be useful with unbalanced classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;4)&lt;/code&gt; This gets really tricky.  The problem with this is that the best method requires that you reinterpret what the learning curves should look like or use both the re-sampled and original data sets.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classic interpretation of learning curves is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Overfit&lt;/strong&gt; - Lines don't quite come together; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Underfit&lt;/strong&gt; - Lines come together but at too low an F1 score;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Just Right&lt;/strong&gt; - Lines come together with a reasonable F1 score.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, if you are training on A` and testing on C, the lines will never completely come together.  If you are training on A` and testing on C` the results won't be meaningful in the context of the original problem.  So what do you do?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is to train on A` and test on B`, but also test on B.  Get the F1 score for B` where you want it to be, then check the F1 score for B.  Then do your testing and generate learning curves for C.  The curves won't ever come together, but you will have a sense of the acceptable bias... its the difference between F1(B) and F1(B`).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the new interpretation of your learning curves is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Overfit&lt;/strong&gt; - Lines don't come together and are farther apart than F1(B`)-F1(B);&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Underfit&lt;/strong&gt; - Lines don't come together but the difference is less than F1(B`)-F1(B) and the F1(C) score is under F1(B);&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Just right&lt;/strong&gt; - Lines don't come together but the difference is less than F1(B`)-F1(B) with an F1(C) score similar to F1(B).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General&lt;/strong&gt;: I strenuously suggest that for unbalanced classes you first try adjusting your class weights in your learning algorithm instead of over/under-sampling as it avoids all of the rigor moral that we've outlined above.  Its very easy in libraries like scikit-learn and pretty easy to hand code in anything that uses a sigmoid function or a majority vote.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-11-19T02:05:24.543" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8898" PostTypeId="2" ParentId="8799" CreationDate="2015-11-19T02:21:26.980" Score="7" Body="&lt;p&gt;&lt;strong&gt;Since you accepted another answer, which says this can't be done, I am editing this to include an example of it being done.  Hope this helps!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Original Answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The &lt;code&gt;x&lt;/code&gt; position swings back and forth out of sink with the &lt;code&gt;y&lt;/code&gt; position. For a 24-hour clock you can accomplish this with &lt;code&gt;x=sin(2pi*hour/24)&lt;/code&gt;,&lt;code&gt;y=cos(2pi*hour/24)&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need both variables or the proper movement through time is lost.  This is due to the fact that the derivative of either sin or cos changes in time where as the &lt;code&gt;(x,y)&lt;/code&gt; position varies smoothly as it travels around the unit circle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method works really well for clustering and for keeping the distance between 15 minutes after midnight and 5 minutes before midnight &quot;close&quot; in Euclidean space.  All of the modulo suggestions don't accomplish this and the cyclic representation that they do accomplish is pretty clumsy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, consider whether it is worthwhile to add a third feature to trace linear time, which can be constructed by hours (or minutes or seconds) from the start of the first record or a Unix time stamp or something similar.  These three features then provide proxies for both the cyclic and linear progression of time e.g. you can pull out cyclic phenomenon like sleep cycles in people's movement and also linear growth like population vs. time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example of if being accomplished:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Enable inline plotting&#xA;%matplotlib inline&#xA;&#xA;#Import everything I need...&#xA;&#xA;import numpy as np&#xA;import matplotlib as mp&#xA;&#xA;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;&#xA;# Grab some random times from here: https://www.random.org/clock-times/&#xA;# put them into a csv.&#xA;from pandas import DataFrame, read_csv&#xA;df = read_csv('/Users/angus/Machine_Learning/ipython_notebooks/times.csv',delimiter=':')&#xA;df['hourfloat']=df.hour+df.minute/60.0&#xA;df['x']=np.sin(2.*np.pi*df.hourfloat/24.)&#xA;df['y']=np.cos(2.*np.pi*df.hourfloat/24.)&#xA;&#xA;df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1UTYj.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1UTYj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def kmeansshow(k,X):&#xA;&#xA;    from sklearn import cluster&#xA;    from matplotlib import pyplot&#xA;    import numpy as np&#xA;&#xA;    kmeans = cluster.KMeans(n_clusters=k)&#xA;    kmeans.fit(X)&#xA;&#xA;    labels = kmeans.labels_&#xA;    centroids = kmeans.cluster_centers_&#xA;    #print centroids&#xA;&#xA;    for i in range(k):&#xA;        # select only data observations with cluster label == i&#xA;        ds = X[np.where(labels==i)]&#xA;        # plot the data observations&#xA;        pyplot.plot(ds[:,0],ds[:,1],'o')&#xA;        # plot the centroids&#xA;        lines = pyplot.plot(centroids[i,0],centroids[i,1],'kx')&#xA;        # make the centroid x's bigger&#xA;        pyplot.setp(lines,ms=15.0)&#xA;        pyplot.setp(lines,mew=2.0)&#xA;    pyplot.show()&#xA;    return centroids&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now lets try it out:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kmeansshow(6,df[['x', 'y']].values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MkVNg.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MkVNg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can just barely see that there are some after midnight times included with the before midnight green cluster.  Now lets reduce the number of clusters and show that before and after midnight can be connected in a single cluster in more detail:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kmeansshow(3,df[['x', 'y']].values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bWg2B.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bWg2B.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See how the blue cluster contains times that are from before and after midnight that are clustered together in the same cluster...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;QED!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-11-19T05:39:30.620" LastActivityDate="2015-11-19T05:39:30.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8899" PostTypeId="5" CreationDate="2015-11-19T02:45:13.793" Score="0" Body="&lt;p&gt;Text Mining is a process of deriving high-quality information from unstructured (textual) information. Possible applications for text-mining are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Comments of Survey responses&lt;/li&gt;&#xA;&lt;li&gt;Customer messages, emails, complaints etc.&lt;/li&gt;&#xA;&lt;li&gt;Investigating competitors by crawling their web sites&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;More about text mining in below links.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Text_mining&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wiki Links&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statsoft.com/textbook/text-mining/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text Mining Concept&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2018-01-01T18:57:18.520" LastActivityDate="2018-01-01T18:57:18.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8900" PostTypeId="4" CreationDate="2015-11-19T02:45:13.793" Score="0" Body="Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-19T09:27:22.323" LastActivityDate="2015-11-19T09:27:22.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8901" PostTypeId="5" CreationDate="2015-11-19T02:46:12.347" Score="0" Body="&lt;p&gt;Predictive modelling is a set of statistical techniques used for predicting outcomes. Each such techniques is called a predictive model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Major areas of application of predictive modelling are health care, customer relationship management, and marketing.&lt;/p&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="13526" LastEditDate="2017-06-18T08:05:59.500" LastActivityDate="2017-06-18T08:05:59.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8902" PostTypeId="4" CreationDate="2015-11-19T02:46:12.347" Score="0" Body="Statistical techniques used for predicting outcomes." OwnerUserId="31513" LastEditorUserId="13526" LastEditDate="2017-06-18T08:05:56.647" LastActivityDate="2017-06-18T08:05:56.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8903" PostTypeId="5" CreationDate="2015-11-19T02:47:23.013" Score="0" Body="&lt;p&gt;&lt;strong&gt;Overview&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Time series are data observed over time (either in continuous time or at discrete time periods).&lt;/p&gt;&#xA;&lt;p&gt;Time series analysis includes:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Trend identification (e.g. autocorrelation)&lt;/li&gt;&#xA;&lt;li&gt;Temporal pattern recognition&lt;/li&gt;&#xA;&lt;li&gt;Spectral analysis / wavelet analysis&lt;/li&gt;&#xA;&lt;li&gt;Forecasting (e.g. regression)&lt;/li&gt;&#xA;&lt;li&gt;Anomaly detection&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The salient characteristic of methods of time series analysis (as opposed to more general methods to analyze relationships among data) is accounting for the possibility of &lt;em&gt;serial correlation&lt;/em&gt; (also known as temporal correlation) among the data.  Positive serial correlation means successive observations in time tend to be close to one another, whereas negative serial correlation means successive observations tend to oscillate between extremes.  Time series analysis also differs from analyses of more general stochastic processes by focusing on the &lt;em&gt;inherent direction&lt;/em&gt; of time, creating a potential asymmetry between past and future.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="41637" LastEditDate="2021-03-02T17:36:27.070" LastActivityDate="2021-03-02T17:36:27.070" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8904" PostTypeId="4" CreationDate="2015-11-19T02:47:23.013" Score="0" Body="Time series are data observed over time (either in continuous time or at discrete time periods).&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-19T09:27:25.550" LastActivityDate="2015-11-19T09:27:25.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8905" PostTypeId="5" CreationDate="2015-11-19T02:47:56.697" Score="0" Body="&lt;p&gt;&quot;Regression&quot; is a general term for a wide variety of techniques to analyze the relationship between one (or more) dependent variables and independent variables. Typically the dependent variables are modeled with probability distributions whose parameters are assumed to vary (deterministically) with the independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinary least squares (OLS) regression affords a simple example in which the expectation of one dependent variable is assumed to depend linearly on the independent variables. The unknown coefficients in the assumed linear function are estimated by choosing values for them that minimize the sum of squared differences between the values of the dependent variable and the corresponding fitted values.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-19T09:28:13.583" LastActivityDate="2015-11-19T09:28:13.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8906" PostTypeId="4" CreationDate="2015-11-19T02:47:56.697" Score="0" Body="Techniques for analyzing the relationship between one (or more) &quot;dependent&quot; variables and &quot;independent&quot; variables.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-19T09:28:42.207" LastActivityDate="2015-11-19T09:28:42.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8908" PostTypeId="2" ParentId="694" CreationDate="2015-11-19T04:16:41.117" Score="7" Body="&lt;p&gt;&lt;a href=&quot;https://mxnet.readthedocs.org/en/latest/&quot; rel=&quot;noreferrer&quot;&gt;MXNet&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;written in C++ but has an API in Python (and a few other programming languages such as R, Julia, and Go)&lt;/li&gt;&#xA;&lt;li&gt;Scales up to multi GPUs and distributed setting with auto parallelism.&lt;/li&gt;&#xA;&lt;li&gt;Automatic differentiation&lt;/li&gt;&#xA;&lt;li&gt;Decent performances:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lImAG.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lImAG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-11-19T04:16:41.117" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="8909" PostTypeId="1" AcceptedAnswerId="8923" CreationDate="2015-11-19T06:04:53.053" Score="13" ViewCount="12129" Body="&lt;p&gt;I have been using this library for basic neural network construction and analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it does not have support for building multi-layered neural networks, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I would like to know of any nice libraries for doing advanced neural networks and Deep Learning in Julia.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2020-08-05T13:58:11.387" Title="Best Julia library for neural networks" Tags="|machine-learning|neural-network|julia|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8911" PostTypeId="2" ParentId="8895" CreationDate="2015-11-19T07:05:58.493" Score="2" Body="&lt;p&gt;You should test your classifier on a dataset that represents the why it will be used. The best is usually unmodified distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During the learning, modify the dataset in anyway that helps you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For details, see &lt;a href=&quot;https://datascience.stackexchange.com/questions/810/should-i-go-for-a-balanced-dataset-or-a-representative-dataset/8628#8628&quot;&gt;Should I go for a &amp;#39;balanced&amp;#39; dataset or a &amp;#39;representative&amp;#39; dataset?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-11-19T07:05:58.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8915" PostTypeId="2" ParentId="8909" CreationDate="2015-11-19T10:37:11.273" Score="9" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;MXNet Julia Package - flexible and efficient deep learning in Julia&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/MXNet.jl&quot;&gt;https://github.com/dmlc/MXNet.jl&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pros&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fast&lt;/li&gt;&#xA;&lt;li&gt;Scales up to multi GPUs and distributed setting with auto parallelism.&lt;/li&gt;&#xA;&lt;li&gt;Lightweight, memory efficient and portable to smart devices.&lt;/li&gt;&#xA;&lt;li&gt;Automatic Differentiation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cons&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Doesn't have yet low level operations for algorithm implementation. But they are working on this issue (&lt;a href=&quot;https://github.com/dmlc/mxnet/issues/586&quot;&gt;https://github.com/dmlc/mxnet/issues/586&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9634" LastEditorUserId="9634" LastEditDate="2015-11-19T12:56:19.593" LastActivityDate="2015-11-19T12:56:19.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8916" PostTypeId="1" CreationDate="2015-11-19T10:48:24.853" Score="2" ViewCount="132" Body="&lt;p&gt;I have been trying the find the best support and confidence values for associative rules mining. I came across the following approach from an &lt;a href=&quot;https://www.quora.com/How-do-I-pick-appropriate-support-confidence-value-when-doing-basket-analysis-with-Apriori-algorithm/answer/Boxun-Zhang&quot; rel=&quot;nofollow&quot;&gt;answer on Quora&lt;/a&gt; - &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Picking the &quot;appropriate&quot; values for support and confidence can be&#xA;  difficult, as it is very much an unsupervised process.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;However, if you transform the output of Apriori algorithm (association&#xA;  rules) into features for a supervised machine learning algorithm, you&#xA;  can examine the effect of having different support and confidences&#xA;  values (while having other features fixed) on the performance of that&#xA;  supervised model (ROC, RMSE, and etc.). Then, the best values for&#xA;  support and confidence are the values that maximize, or minimize, the&#xA;  performance metric.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Being a rookie in this field, I am not sure on how to do this. Could you please help me on this?&lt;/p&gt;&#xA;" OwnerUserId="3327" LastActivityDate="2015-11-19T10:48:24.853" Title="Using classification to find the best support and confidence measure in associative rule mining" Tags="|data-mining|classification|unsupervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8917" PostTypeId="2" ParentId="5789" CreationDate="2015-11-19T11:07:35.660" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://lxml.de/&quot; rel=&quot;nofollow&quot;&gt;lxml&lt;/a&gt; is a nice web scrapping library in Python. Beautiful Soup is a wrapper over lxml.  So, lxml is faster than both scrapy and beautiful soup and has a much easier learning curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/Dawny33/Cricket-Analytics/blob/master/Crawler.py&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is an example of a scraper which I built with it for a personal project, which can iterate over web pages.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-19T11:07:35.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8918" PostTypeId="2" ParentId="6787" CreationDate="2015-11-19T11:34:07.250" Score="2" Body="&lt;p&gt;Decision trees are non linear. Unlike Linear regression there is no equation  to express relationship between independent and dependent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear regression - &#xA;Price of fruit = b0 + b1*Freshness + b2*Size&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision tree - &#xA;Nodes:&#xA;Ripe - Yes or no | &#xA;Fresh -  Yes or No | &#xA;Size - &amp;lt;5, &gt;5 but &amp;lt;10 and &gt;10 |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second case there is no linear relationship between independent and dependent variables. &lt;/p&gt;&#xA;" OwnerDisplayName="user14204" LastActivityDate="2015-11-19T11:34:07.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8919" PostTypeId="2" ParentId="6787" CreationDate="2015-11-19T11:43:50.750" Score="16" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Recently a friend of mine was asked whether decision tree algorithm a&#xA;  linear or nonlinear algorithm in an interview&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Decision trees is a non-linear classifier like the neural networks, etc. It is generally used for classifying non-linearly separable data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even when you consider the regression example, decision tree is non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a linear regression line would look somewhat like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GXj8T.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GXj8T.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The red dots are the data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a decision tree regression plot would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JvuRJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JvuRJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, clearly &lt;strong&gt;decision trees are non-linear&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-05-31T04:47:04.803" LastActivityDate="2017-05-31T04:47:04.803" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8920" PostTypeId="1" CreationDate="2015-11-19T12:04:51.340" Score="2" ViewCount="294" Body="&lt;p&gt;I have just gotten my feet wet with Hidden Markov Models. Now I want to apply them to tell whether a transaction from an ATM is suspicious or not. I have great confusion in defining my Hidden States.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My attempt:&lt;/strong&gt; A transaction is either suspicious or non-suspicious. Since I cannot tell, &lt;code&gt;(&quot;this is hidden to me&quot;)&lt;/code&gt;, whether a transaction is suspicious or not before analyzing it, therefore my &lt;code&gt;Hidden States(Suspicious, Non-Suspicious)&lt;/code&gt;. &lt;strong&gt;Does this hold&lt;/strong&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My great confusion:&lt;/strong&gt; When I read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;Hidden Markov Models on Weather example&lt;/a&gt;, I understood it this way: &quot;weather can be &lt;code&gt;rainy&lt;/code&gt; or &lt;code&gt;sunny&lt;/code&gt; - I mean its the 'same' weather that is switching between the two states&quot;. In my &lt;code&gt;transaction&lt;/code&gt; case - one transaction cannot switch between being suspicious and non-suspicious. &lt;em&gt;So, does my hidden states hold&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt; Does my hidden states hold? Can I apply the HMM that way? Please advice, thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S&lt;/strong&gt;: I had my Observations as location, time and amount of money but as of now I want clarification on Hidden states.&lt;/p&gt;&#xA;" OwnerUserId="13132" LastActivityDate="2015-11-20T00:37:55.537" Title="Can I apply Hidden Markov Models this way?" Tags="|machine-learning|markov-process|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8921" PostTypeId="2" ParentId="6070" CreationDate="2015-11-19T12:32:59.207" Score="4" Body="&lt;p&gt;This is similar to the fundamental information theory problem that Shannon explored.  In that domain, it is framed this way: given two rvs, X and Y, what information does X convey about Y?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example would be that I create sequence of numbers/bit/letters from a known PDF (X) and you receive a distorted version of those values for which you also know the PDF (Y).  The mutual information is the number of bits that Y communicates about X which can be thought of as a type of correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of time varying PDFs (ie stochastic processes), information theory would just treat the ensemble of rvs as one joint rv and then calculate the mutual information of the joint PDFs.  If the PDFs are iid, then significant simplifications are possible.  Joint Gaussian PDFs also make things easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another information theory concept that might be useful is the entropy rate of a process.  ER is a quantification of the amount of information contained in a process.  Depending on your problem, you might be able to compute the ER for each process and then use those values as realizations of a feature that can be grouped with a clustering algorithm.&lt;/p&gt;&#xA;" OwnerUserId="10384" LastActivityDate="2015-11-19T12:32:59.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8922" PostTypeId="1" AcceptedAnswerId="8924" CreationDate="2015-11-19T12:59:40.403" Score="28" ViewCount="174404" Body="&lt;p&gt;I have a dataset like the one below. I would like to remove all characters after the character ©. How can I do that in R?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;data_clean_phrase &amp;lt;- c(&amp;quot;Copyright © The Society of Geomagnetism and Earth&amp;quot;, &#xA;&amp;quot;© 2013 Chinese National Committee &amp;quot;)&#xA;&#xA;data_clean_df &amp;lt;- as.data.frame(data_clean_phrase)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="29169" LastEditDate="2021-03-11T19:07:56.217" LastActivityDate="2021-03-11T19:07:56.217" Title="Removing strings after a certain character in a given text" Tags="|r|data-cleaning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="8923" PostTypeId="2" ParentId="8909" CreationDate="2015-11-19T13:17:24.097" Score="7" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/pluskid/Mocha.jl&quot;&gt;Mocha.jl&lt;/a&gt; - Mocha is a Deep Learning framework for Julia, inspired by the C++ framework Caffe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Project with good &lt;a href=&quot;http://mochajl.readthedocs.org/en/latest/&quot;&gt;documentation&lt;/a&gt; and examples.&#xA;Can be run on CPU and GPU backend.&lt;/p&gt;&#xA;" OwnerUserId="9314" LastActivityDate="2015-11-19T13:17:24.097" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8924" PostTypeId="2" ParentId="8922" CreationDate="2015-11-19T13:52:28.787" Score="31" Body="&lt;p&gt;For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rs&amp;lt;-c(&quot;copyright @ The Society of mo&quot;,&quot;I want you to meet me @ the coffeshop&quot;)&#xA; s&amp;lt;-gsub(&quot;@.*&quot;,&quot;&quot;,rs)&#xA; s&#xA; [1] &quot;copyright &quot;             &quot;I want you to meet me &quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or, if you want to keep the @ character:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; s&amp;lt;-gsub(&quot;(@).*&quot;,&quot;\\1&quot;,rs)&#xA; s&#xA; [1] &quot;copyright @&quot;             &quot;I want you to meet me @&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: If what you want is to remove everything from the last @ on you just have to follow this previous example with the appropriate regex. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rs&amp;lt;-c(&quot;copyright @ The Society of mo located @ my house&quot;,&quot;I want you to meet me @ the coffeshop&quot;)&#xA;s&amp;lt;-gsub(&quot;(.*)@.*&quot;,&quot;\\1&quot;,rs)&#xA;s&#xA;[1] &quot;copyright @ The Society of mo located &quot; &quot;I want you to meet me &quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Given the matching we are looking for, both sub and gsub will give you the same answer.&lt;/p&gt;&#xA;" OwnerUserId="12603" LastEditorUserId="12603" LastEditDate="2015-11-19T17:25:33.983" LastActivityDate="2015-11-19T17:25:33.983" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8926" PostTypeId="1" CreationDate="2015-11-19T15:19:35.033" Score="4" ViewCount="208" Body="&lt;p&gt;I have a corpus of text documents, some of which are labelled by analysts with label L. I am using this data to train an SVM for predicting if a new document should have label L.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far it's straight-forward, but there is an issue: The analysts have not evaluated all documents in the training set, so there are in fact three classes of documents:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Documents labeled L&lt;/li&gt;&#xA;&lt;li&gt;Documents the analysts have looked at, and chosen not label L (so you could say they're labelled not-L)&lt;/li&gt;&#xA;&lt;li&gt;Documents the analysts have not looked at.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, at training time, I have no way to separate documents in 2. and 3, or not-L and unlabelled documents. I believe this is a problem, because a not-L label gives information to the SVM, but an unlabelled document is more &quot;neutral&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I estimate the impact of this issue on predicting if a new document should have label L?&lt;/p&gt;&#xA;" OwnerUserId="13596" LastActivityDate="2015-11-25T15:00:16.763" Title="Impact of unlabelled documents for label prediction via SVM" Tags="|machine-learning|statistics|svm|multilabel-classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8927" PostTypeId="2" ParentId="8460" CreationDate="2015-11-19T16:34:49.927" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;http://ghmm.org/&quot; rel=&quot;nofollow&quot;&gt;ghmm&lt;/a&gt; library might be the one which you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it is said in their website:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is used for implementing efficient data structures and algorithms&#xA;  for basic and extended HMMs with discrete and continuous emissions. It&#xA;  comes with Python wrappers which provide a much nicer interface and&#xA;  added functionality.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It also has a nice documentation and a step-by-step tutorial for getting your feet wet.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-19T16:34:49.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8928" PostTypeId="2" ParentId="8920" CreationDate="2015-11-20T00:37:55.537" Score="1" Body="&lt;p&gt;This is a reasonable application for a hidden markov model only if you have reason to believe that the transactions have the markov property - that the next state depends only on the present state, and not on any of the previous ones. This often makes sense for models in time series (like in the weather example, in which the fact that it's already raining makes it more likely that in the next time step it will still be raining). Of course, the model can be useful even if it's not a perfect fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way you've formulated the model is saying that the hidden state (suspicious/non-suspicious) of an ATM may be predictive of the emitted, or observed states (location, time, and amount of money). At least for the first two, this feels a bit off, particularly if there are multiple ATMs involved, as is suggested by the presence of a location variable. I'll try to explain what I think the way you've formulated the model suggests in terms of observed variables and how it might be nonsensical, and suggest some alternatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, this seems to suggest that suspicious/non-suspicious is a shared state of all ATMs, and that that state somehow influences where an ATM is being used (each hidden state has a particular distribution from the location is drawn). It may make more sense to have a separate model for each ATM location. Maybe this is already what you were intending. If that is the case, location wouldn't really be an observed variable. Instead of thinking of the transactions switching between states, which doesn't seem to make much sense, as you say, perhaps you want to be thinking of the ATMs switching states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, it's not clear how time would be a random variable that would be emitted based on a state of suspiciousness. Instead, time would probably make more sense as the way of ordering the observations you feed into the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want to combine all transactions into a single model, maybe an SVM would be more straightforward as a binary classifier.&lt;/p&gt;&#xA;" OwnerUserId="14218" LastActivityDate="2015-11-20T00:37:55.537" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8929" PostTypeId="2" ParentId="6331" CreationDate="2015-11-20T03:16:58.080" Score="0" Body="&lt;p&gt;Echoing @James Allen's suggestion, the total amount of data at any point in time is important to know. It will tell you how much disk you need. Also important is the rate at which new data will be received and old data deleted as this will be limited by the IOPS of your hardware&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your use case any popular DBMS on commodity hardware should suffice. Something like PostgreSQL can handle terabytes of data at hundreds of transactions per second on a run-of-the-mill four core server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at the &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R programming language&lt;/a&gt; for analysis. SQL is great for reads and writes but a histogram or linear regression is very tricky in comparison.&lt;/p&gt;&#xA;" OwnerUserId="13498" LastActivityDate="2015-11-20T03:16:58.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8930" PostTypeId="2" ParentId="8745" CreationDate="2015-11-20T09:05:33.487" Score="6" Body="&lt;p&gt;There are ranking algorithms based on machine learning that are aimed to build ranking models. Training data for these models is given in the form of partial ordering between each pair of elements in a sample. A brief description, together with a list of useful references, is given in the corresponding &lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank&quot;&gt;Wikipedia page&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2015-11-20T09:05:33.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8931" PostTypeId="5" CreationDate="2015-11-20T11:11:34.763" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-20T11:11:34.763" LastActivityDate="2015-11-20T11:11:34.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8932" PostTypeId="4" CreationDate="2015-11-20T11:11:34.763" Score="0" Body="An algorithm is a set of one or more computations that will produce a calculated result. All statistics methods are algorithms. Algorithms can be simple, such as calculating a percentage, or can be very complex and require a computer for fast and accurate results.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:06:37.747" LastActivityDate="2015-11-20T12:06:37.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8933" PostTypeId="5" CreationDate="2015-11-20T11:13:45.567" Score="0" Body="&lt;p&gt;The &lt;em&gt;logistic function&lt;/em&gt; is $$ f(x) = \frac{1}{1+e^{-x}}, $$ which maps real numbers to $(0,1)$. One common use of the logistic function is &lt;a href=&quot;http://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;, which is a standard method of quantifying the effect of a set of predictors $\{X_1, ..., X_p\}$ on a binary outcome, $Y$. The model can be written as &#xA;$$ P(Y=1|X) = f(\beta_0 + \beta_1X_1 + ... + \beta_p X_p)$$ The logistic regression model has the nice property that the exponentiated regression coefficients can be interpreted as &lt;a href=&quot;http://en.wikipedia.org/wiki/Odds_ratio&quot; rel=&quot;nofollow&quot;&gt;odds ratios&lt;/a&gt; associated with a one unit increase in the predictor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second use of the logistic function (but unrelated to logistic regression) is the logistic distribution, which has $f(x)$ as its quantile function. &lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:06:48.970" LastActivityDate="2015-11-20T12:06:48.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8934" PostTypeId="4" CreationDate="2015-11-20T11:13:45.567" Score="0" Body="Refers generally to statistical procedures that utilize the logistic function, most commonly various forms of logistic regression&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:06:09.530" LastActivityDate="2015-11-20T12:06:09.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8935" PostTypeId="5" CreationDate="2015-11-20T11:14:26.570" Score="0" Body="&lt;p&gt;&lt;strong&gt;Feature selection&lt;/strong&gt;, also called &lt;strong&gt;attribute selection&lt;/strong&gt; or &lt;strong&gt;feature reduction&lt;/strong&gt;, refers to techniques for identifying a subset of features of a data set that are relevant to a given problem. By removing irrelevant and redundant features, successful feature selection can avoid the &lt;a href=&quot;http://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;curse of dimensionality&lt;/a&gt; and improve the performance, speed, and interpretability of subsequent models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature selection includes &lt;strong&gt;manual methods&lt;/strong&gt; (such those those based on domain knowledge) and automatic methods.  &lt;strong&gt;Automatic methods&lt;/strong&gt; are often categorized into &lt;strong&gt;filter&lt;/strong&gt;, &lt;strong&gt;wrapper&lt;/strong&gt;, and &lt;strong&gt;embedded&lt;/strong&gt; approaches.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Filter approaches&lt;/strong&gt; perform feature selection as a separate preprocessing step before the learning algorithm.  Filter approaches thus look only at the intrinsic properties of the data. Filter methods include Wilcoxon rank sum tests and &lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_selection#Correlation_feature_selection&quot; rel=&quot;nofollow&quot;&gt;Correlation based tests&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Wrapper approaches&lt;/strong&gt; uses performance of a learning algorithm to select features. A search algorithm is “wrapped” around the learning algorithm to ensure the space of feature subsets is adequately searched.  As such, wrapper methods can be seen as conducting the model hypothesis search within the feature subset search. Examples of wrapper approaches are &lt;a href=&quot;http://en.wikipedia.org/wiki/Simulated_annealing&quot; rel=&quot;nofollow&quot;&gt;simulated annealing&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Beam_search&quot; rel=&quot;nofollow&quot;&gt;beam search&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Embedded approaches&lt;/strong&gt; incorporate variable selection as a part of the training process, with feature relevance obtained analytically from the objective of the learning model. Embedded methods can be seen as a search in the combined space of feature subsets and hypotheses.  Examples of embedded approaches are &lt;a href=&quot;http://en.wikipedia.org/wiki/Boosting_%28machine_learning%29&quot; rel=&quot;nofollow&quot;&gt;boosting&lt;/a&gt; and recursive ridge regression.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:07:38.743" LastActivityDate="2015-11-20T12:07:38.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8936" PostTypeId="4" CreationDate="2015-11-20T11:14:26.570" Score="0" Body="Methods and principles of selecting a subset of attributes for use in further modelling&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:05:58.170" LastActivityDate="2015-11-20T12:05:58.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8937" PostTypeId="5" CreationDate="2015-11-20T11:15:01.420" Score="0" Body="&lt;p&gt;&lt;strong&gt;Overview&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Random forests are an ensemble learning method for classification or regression that operate by constructing a multitude of decision trees at training time and aggregating their outputs: in regression by averaging, in classification either by averaging probabilities or by majority vote. The randomness in the decision trees follows from (a) using a new bootstrapped version of the original sample for each tree and (b) using a random subsample of the explanatory variables at each node of each tree.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.berkeley.edu/%7Ebreiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest page&lt;/a&gt; maintained by Leo Breiman and Adele Cutler, the creators of the algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Wikipedia pages on &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_tree_%28disambiguation%29&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Trees&lt;/a&gt;, &lt;a href=&quot;http://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt; and &lt;a href=&quot;http://%20http://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensemble Learning&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="55122" LastEditDate="2022-06-14T14:19:14.890" LastActivityDate="2022-06-14T14:19:14.890" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8938" PostTypeId="4" CreationDate="2015-11-20T11:15:01.420" Score="0" Body="Random forest is a machine learning ensemble method based on choosing random subsets of observations and variables for each of many decision trees.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="55122" LastEditDate="2022-06-14T14:19:14.890" LastActivityDate="2022-06-14T14:19:14.890" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8939" PostTypeId="5" CreationDate="2015-11-20T11:15:36.970" Score="0" Body="&lt;p&gt;Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software. Exploratory data analysis techniques are often used to identify problems.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:07:57.570" LastActivityDate="2015-11-20T12:07:57.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8940" PostTypeId="4" CreationDate="2015-11-20T11:15:36.970" Score="0" Body="Data cleaning is a preliminary step to statistical analysis in which the data-set is edited to correct errors and to put it into a form suitable for processing by statistical software.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-20T12:06:41.447" LastActivityDate="2015-11-20T12:06:41.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8941" PostTypeId="1" AcceptedAnswerId="8945" CreationDate="2015-11-20T11:21:03.500" Score="23" ViewCount="3664" Body="&lt;p&gt;This is a similar question like the &lt;a href=&quot;https://stats.stackexchange.com/questions/1904/statistics-conferences&quot;&gt;Statistics Conferences question at CrossValidated&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the most significant annual Data Science conferences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rules:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Include a link to the conference&lt;/li&gt;&#xA;&lt;li&gt;Please include links for the talks (be it youtube, the conference site or some other video streaming site)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-09-29T00:51:13.743" Title="Data Science conferences?" Tags="|community|" AnswerCount="14" CommentCount="1" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8942" PostTypeId="2" ParentId="8871" CreationDate="2015-11-20T13:31:29.410" Score="1" Body="&lt;p&gt;I don't think anyone can help you with features without more information about your particular domain, but you could do worse than logistic regression for modeling:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Discrete_choice&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Discrete_choice&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Logistic_regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14229" LastActivityDate="2015-11-20T13:31:29.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8943" PostTypeId="1" CreationDate="2015-11-20T14:15:37.753" Score="2" ViewCount="72" Body="&lt;p&gt;I have some time series data (time, value) such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;281,97.0&#xA;284,98.0&#xA;287,98.0&#xA;290,98.0&#xA;293,98.0&#xA;297,99.0&#xA;309,100.0&#xA;320,101.0&#xA;333,101.0&#xA;346,103.0&#xA;359,104.0&#xA;372,105.0&#xA;385,107.0&#xA;396,108.0&#xA;405,108.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict when the value will be 160.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is somewhat noisy. The values go mostly up, but could go down sharply every now and then, or stay the same for a short time. When these distortions occur, the prediction is far off for a period of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithms or heuristics would work the best to deliver a robust prediction that doesn't vary to much?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using a linear fit over a time window of 60 data points. The results are not stable enough.&lt;/p&gt;&#xA;" OwnerUserId="14231" LastActivityDate="2015-11-20T14:15:37.753" Title="What's the most robust way to predict the value of some noisy time series data?" Tags="|predictive-modeling|time-series|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="8944" PostTypeId="2" ParentId="8871" CreationDate="2015-11-20T14:30:49.877" Score="1" Body="&lt;p&gt;How much help this community can be depends on where in the process you are.  If you have a dataset and are looking on ways to &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_engineering&quot; rel=&quot;nofollow&quot;&gt;determine the most predictive features&lt;/a&gt;, then you could include some information as to what kind of data you've collected/have access to.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead your question is specifically &quot;what data should I collect to predict purchases from banks, and what models should I use,&quot; you are unlikely to get a very specific answer.  &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-20T14:30:49.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8945" PostTypeId="2" ParentId="8941" CreationDate="2015-11-20T19:30:24.200" Score="10" Body="&lt;p&gt;PyData - talks about Python Data tools&lt;br&gt;&#xA;Link: &lt;a href=&quot;http://pydata.org/events/&quot; rel=&quot;noreferrer&quot;&gt;http://pydata.org/events/&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is one PyData conference on the east coast and one on the west coast each year.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;NIPS - Neural Information Processing Systems (NIPS)&lt;br&gt;&#xA;Link: &lt;a href=&quot;https://nips.cc/&quot; rel=&quot;noreferrer&quot;&gt;https://nips.cc/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one of the hardest / most prestigious academic Machine Learning conferences to get an abstract / poster accepted.    &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The 5th International Workshop on Parallel and Distributed Computing for Large Scale Machine Learning and Big Data Analytics (IEEE IPDPS 2016)&lt;br&gt;&#xA;Link: &lt;a href=&quot;http://parlearning.ecs.fullerton.edu/&quot; rel=&quot;noreferrer&quot;&gt;http://parlearning.ecs.fullerton.edu/&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This one is also an academic conference with paper submission.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Note:&#xA;I am not sure if you want academic or nor academic conferences (have conference proceedings / papers associated with the conference).&#xA;Some conferences are not about new data science methodologies  but the tools and libraries (e.g. PyData) that implement existing methodologies.&#xA;Also, data science is very broad and includes Stat, Machine Learning and data warehousing / mining etc.&lt;/p&gt;&#xA;" OwnerUserId="6396" LastEditorUserId="6396" LastEditDate="2015-11-23T17:52:22.077" LastActivityDate="2015-11-23T17:52:22.077" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8946" PostTypeId="1" AcceptedAnswerId="10311" CreationDate="2015-11-20T20:52:08.420" Score="3" ViewCount="666" Body="&lt;p&gt;In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?&lt;/p&gt;&#xA;" OwnerUserId="13578" LastActivityDate="2016-02-19T03:26:46.010" Title="Feature Scaling and Mean Normalization" Tags="|data-cleaning|gradient-descent|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8947" PostTypeId="2" ParentId="8941" CreationDate="2015-11-20T21:11:17.623" Score="1" Body="&lt;p&gt;Annual Wolfram Technology Conference.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.wolfram.com/events/technology-conference/2015/&quot; rel=&quot;nofollow&quot;&gt;Wolfram Technology Conference 2015&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Technical training, analysis topics on a wide range of industries, software development, curated data, on so on. &lt;/p&gt;&#xA;" OwnerUserId="10814" LastActivityDate="2015-11-20T21:11:17.623" CommentCount="2" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8948" PostTypeId="2" ParentId="8946" CreationDate="2015-11-20T21:41:21.573" Score="2" Body="&lt;p&gt;Mean normalization is a form of feature scaling, so these are not really two different approaches. Feature scaling is just a more general term. What kind of feature scaling, e.g. mean normalization,  you need to use ultimately depends on the data.&lt;/p&gt;&#xA;" OwnerUserId="14177" LastActivityDate="2015-11-20T21:41:21.573" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8949" PostTypeId="2" ParentId="8941" CreationDate="2015-11-20T22:39:07.593" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.kdd.org/&quot; rel=&quot;nofollow&quot;&gt;ACM KDD&lt;/a&gt; is the top conference for both industry and academia. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last time it took place in Sydney. Have a look to &lt;a href=&quot;http://www.kdd.org/kdd2015/&quot; rel=&quot;nofollow&quot;&gt;the program and participants&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="418" LastActivityDate="2015-11-20T22:39:07.593" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8950" PostTypeId="2" ParentId="8941" CreationDate="2015-11-21T00:34:08.703" Score="2" Body="&lt;p&gt;My favorites ones are Wrangle, Spark Summit and ampcamp.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;&lt;p&gt;Wrangle is a new, single-day, single-track industry event about the principles, practice, and application of Data Science, across multiple data-rich industries. It includes talks from data scientists from companies like Salesforce, Pinterest, Facebook, and Uber about the hardest problems they've faced, and the solutions they found for them.&#xA;  If you're a practicing Data Scientist, Wrangle is for you!&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;spark-summit. DATA SCIENCE AND ENGINEERING AT SCALE &lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;AMP Camps are Big Data training events organized by the UC Berkeley AMPLab about big data analytics, machine learning, and popular open-source software projects produced by the AMPLab. All AMP Camp curricula, and whenever possible videos of instructional talks presented at AMP Camps, are published here and accessible for free.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.wrangleconf.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.wrangleconf.com/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://spark-summit.org/&quot; rel=&quot;nofollow&quot;&gt;https://spark-summit.org/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/&quot; rel=&quot;nofollow&quot;&gt;http://ampcamp.berkeley.edu/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2015-11-21T00:34:08.703" CommentCount="1" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8951" PostTypeId="5" CreationDate="2015-11-21T06:21:43.267" Score="0" Body="&lt;p&gt;&lt;strong&gt;Overview&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data visualization refers to techniques for presenting results in graphical form, such as histograms, scatterplots, or boxplots. Data visualization is a special challenge for data with high dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your question is only about how to get particular software to produce a specific effect, then it is likely not on topic here.  Programming questions (for example, in Python, or in &lt;code&gt;R&lt;/code&gt; with &lt;code&gt;ggplot&lt;/code&gt;, etc.) for which you can supply a reproducible example are usually welcomed on &lt;a href=&quot;http://stackoverflow.com/&quot;&gt;StackOverflow&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following question contains references to data visualization resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/22434/27581&quot;&gt;Modern successor to Exploratory Data Analysis by Tukey?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="3117" LastEditDate="2017-08-27T14:55:36.443" LastActivityDate="2017-08-27T14:55:36.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8952" PostTypeId="4" CreationDate="2015-11-21T06:21:43.267" Score="0" Body="Constructing meaningful and useful graphical representations of data. (If your question is only about how to get particular software to produce a specific effect, then it is likely not on topic here.)&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-21T12:16:50.487" LastActivityDate="2015-11-21T12:16:50.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8953" PostTypeId="5" CreationDate="2015-11-21T06:22:47.047" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-21T06:22:47.047" LastActivityDate="2015-11-21T06:22:47.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8954" PostTypeId="4" CreationDate="2015-11-21T06:22:47.047" Score="0" Body="In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-21T12:16:29.350" LastActivityDate="2015-11-21T12:16:29.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8955" PostTypeId="5" CreationDate="2015-11-21T06:23:57.420" Score="0" Body="&lt;h2&gt;What is Deep Learning?&lt;/h2&gt;&#xA;&lt;p&gt;Deep Learning is an area of &lt;a href=&quot;/questions/tagged/machine-learning&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;machine-learning&amp;#39;&quot; rel=&quot;tag&quot;&gt;machine-learning&lt;/a&gt; which attempts to build &lt;a href=&quot;/questions/tagged/neural-networks&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;neural-networks&amp;#39;&quot; rel=&quot;tag&quot;&gt;neural-networks&lt;/a&gt; to learn complex functions by using special architectures composed of many layers (hence the term &amp;quot;deep&amp;quot;).&lt;/p&gt;&#xA;&lt;p&gt;Deep architectures allow more complex tasks to be learned because, in addition to these neural networks having more layers to perform transformations, the larger number of layers and more complex architectures of the neural network allow a hierarchical organization of functionality to emerge.&lt;/p&gt;&#xA;&lt;p&gt;Deep Learning was introduced into machine learning research with the intention of moving machine learning closer to artificial intelligence. A significant impact of deep learning lies in feature learning, mitigating much of the effort going into manual feature engineering in non-deep learning neural networks.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;New to Deep Learning?&lt;/h2&gt;&#xA;&lt;p&gt;There are a variety of resources including books, tutorials/workshops, etc. for those looking to learn more about Deep Learning.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A popular introductory tutorial is:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;SciPy 2020 Conference Tutorial:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=CW8hTe21LPg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning from Scratch with PyTorch&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;Some popular introductory books:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1617294438&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Deep Learning with Python&lt;/em&gt;&lt;/a&gt;, by François Chollet&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Deep Learning with PyTorch&lt;/em&gt;&lt;/a&gt;, by Eli Stevens, Luca Antiga, and Thomas Viehman&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.deeplearningbook.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Deep Learning&lt;/em&gt;&lt;/a&gt;, by Ian Goodfellow&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h1&gt;Resources&lt;/h1&gt;&#xA;&lt;h2&gt;Papers&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1404.7828&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning in Neural Networks: An Overview&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2&gt;Books&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks and Deep Learning&lt;/a&gt; By &lt;a href=&quot;http://michaelnielsen.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Michael Nielsen&lt;/a&gt; - this is a free book with associated Python source code on &lt;a href=&quot;https://github.com/mnielsen/neural-networks-and-deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1519514212&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning Made Easy with R: A Gentle Introduction For Data Science&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1601988141&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning: Methods and Applications&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.springer.com/la/book/9783319056029&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autonomous Robotics and Deep Learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2&gt;Videos&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=bxe2T-V8XRs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks Demystified&lt;/a&gt; - accompanied by a set of &lt;a href=&quot;https://github.com/stephencwelch/Neural-Networks-Demystified&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jupyter Notebooks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://coursera.org/specializations/deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning by Andrew Ng&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2&gt;Stack Exchange Sites&lt;/h2&gt;&#xA;&lt;p&gt;Other StackExchange sites with Deep Learning tag:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com&quot;&gt;Cross Validated&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com&quot;&gt;Stack Overflow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29169" LastEditDate="2021-10-17T06:43:43.290" LastActivityDate="2021-10-17T06:43:43.290" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8956" PostTypeId="4" CreationDate="2015-11-21T06:23:57.420" Score="0" Body="a new area of Machine Learning research concerned with the technologies used for learning hierarchical representations of data, mainly done with deep neural networks (i.e. networks with two or more hidden layers), but also with some sort of Probabilistic Graphical Models." OwnerUserId="11097" LastEditorUserId="21519" LastEditDate="2019-03-06T10:27:54.070" LastActivityDate="2019-03-06T10:27:54.070" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8957" PostTypeId="5" CreationDate="2015-11-21T06:25:13.837" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-21T06:25:13.837" LastActivityDate="2015-11-21T06:25:13.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8958" PostTypeId="4" CreationDate="2015-11-21T06:25:13.837" Score="0" Body="k-means is a family of cluster analysis methods in which you specify the number of clusters you expect. This is as opposed to hierarchical cluster analysis methods.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-21T12:16:05.543" LastActivityDate="2015-11-21T12:16:05.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8959" PostTypeId="5" CreationDate="2015-11-21T06:26:15.180" Score="0" Body="&lt;p&gt;From &lt;a href=&quot;http://en.wikipedia.org/wiki/Database&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A database is an organized collection of data. The data is typically&#xA;  organized to model relevant aspects of reality (for example, the&#xA;  availability of rooms in hotels), in a way that supports processes&#xA;  requiring this information (for example, finding a hotel with&#xA;  vacancies).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A large proportion of websites and applications rely on databases. They are a crucial component of telecommunications systems, banking systems, video games, and just about any other software system or electronic device that maintains some amount of persistent information. In addition to persistence, database systems provide a number of other properties that make them exceptionally useful and convenient: reliability, efficiency, scalability, concurrency control, data abstraction, and high-level query languages. Databases are so ubiquitous and important that computer science graduates frequently cite their database class as the one most useful to them in their industry or graduate-school careers.&lt;sup&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_management_system&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;database&lt;/em&gt; should not be confused with &lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_management_system&quot; rel=&quot;nofollow&quot;&gt;Database Management System&lt;/a&gt;&lt;/em&gt; (DBMS). A DBMS is the system software used to create and manage databases and provide users and applications with access to the database(s). A database is to a DBMS as a document is to a word processor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some useful references:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dba.stackexchange.com/questions/18700/what-is-a-database/18702#18702&quot;&gt;What is database?&lt;/a&gt; from Database Administrators SE.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_design&quot; rel=&quot;nofollow&quot;&gt;design&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.tldp.org/HOWTO/Sybase-ASA-HOWTO/manage.html&quot; rel=&quot;nofollow&quot;&gt;managing a database&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_tuning&quot; rel=&quot;nofollow&quot;&gt;tuning of a database&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_security&quot; rel=&quot;nofollow&quot;&gt;database security&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_connection&quot; rel=&quot;nofollow&quot;&gt;Database connections&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Database_normalization&quot; rel=&quot;nofollow&quot;&gt;Database normalization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-21T12:16:01.663" LastActivityDate="2015-11-21T12:16:01.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8960" PostTypeId="4" CreationDate="2015-11-21T06:26:15.180" Score="0" Body="A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-21T12:16:44.813" LastActivityDate="2015-11-21T12:16:44.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8961" PostTypeId="1" CreationDate="2015-11-21T07:17:14.963" Score="2" ViewCount="109" Body="&lt;p&gt;I have a correlation matrix for dependent var Vs independent variable as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Year   1       Sales&#xA;Sales -0.5453  1.0000&#xA;PriceIndex    -0.6089 &#xA;Income        -0.5033 &#xA;Interest      -0.3842 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we can see that all independent var have negative correlation with the dependent variable. However as one of the requirement that i have for my project is that based on these values can i predict correlation between my independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly suggest if we can predict the same.&#xA;Thanks, Shivi&lt;/p&gt;&#xA;" OwnerUserId="9052" LastActivityDate="2023-12-14T09:06:35.803" Title="Can we predict correlation between independent variables based on dependent variables" Tags="|statistics|correlation|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8962" PostTypeId="2" ParentId="8941" CreationDate="2015-11-21T07:50:45.983" Score="3" Body="&lt;p&gt;Annual UseR! conference:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;link: &lt;a href=&quot;http://user2015.math.aau.dk/&quot; rel=&quot;nofollow&quot;&gt;UseR!2015&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;link: &lt;a href=&quot;http://user2016.org/&quot; rel=&quot;nofollow&quot;&gt;UseR!2016&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Data lead has an annual conference. Last year in the US, this year in Paris:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.datalead2015.com/index.php&quot; rel=&quot;nofollow&quot;&gt;http://www.datalead2015.com/index.php&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;H2O has a conference &lt;a href=&quot;http://h2oworld.h2o.ai/&quot; rel=&quot;nofollow&quot;&gt;H2O world&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Data Science is a very broad subject:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Effective applications of the R language:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.earl-conference.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.earl-conference.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Strata + Hadoop World:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://conferences.oreilly.com/strata&quot; rel=&quot;nofollow&quot;&gt;http://conferences.oreilly.com/strata&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Joint Statistical Meetings:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.amstat.org/meetings/jsm.cfm&quot; rel=&quot;nofollow&quot;&gt;https://www.amstat.org/meetings/jsm.cfm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And of course your local &lt;a href=&quot;http://www.meetup.com/&quot; rel=&quot;nofollow&quot;&gt;meetups&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-11-21T07:50:45.983" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8963" PostTypeId="1" CreationDate="2015-11-21T07:58:22.937" Score="7" ViewCount="734" Body="&lt;p&gt;In a few weeks I'm starting a new job that will be involved in machine learning and data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a masters degree in probability / mathematics but I have no knowledge of machine learning and data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any online resources like courses or books that could help me prepare for my new job? Basically I'm trying to get a head start on my new job. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now the only things I really know related to this fields are regression methods like linear regression and logistic regression, and I have a decent grasp of mathematical concepts so I'd prefer something that is not totally dumbed-down&lt;/p&gt;&#xA;" OwnerUserId="14240" LastEditorUserId="11097" LastEditDate="2015-12-16T18:00:29.947" LastActivityDate="2016-01-07T12:09:29.697" Title="data science / machine learning resources?" Tags="|machine-learning|reference-request|" AnswerCount="1" CommentCount="0" ClosedDate="2015-11-21T12:17:29.093" ContentLicense="CC BY-SA 3.0" />
  <row Id="8964" PostTypeId="2" ParentId="8941" CreationDate="2015-11-21T08:17:54.233" Score="4" Body="&lt;p&gt;Data science is still a domain in fusion, borrowing from neighboring fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recent interesting contender is DSAA, &quot;IEEE International Conference on Data Science and Advanced Analytics&quot;. The first edition (&lt;a href=&quot;http://datamining.it.uts.edu.au/conferences/dsaa14/&quot; rel=&quot;nofollow&quot;&gt;DSAA 2014&lt;/a&gt;) held in Shanghai. The second &lt;a href=&quot;http://dsaa2015.lip6.fr/&quot; rel=&quot;nofollow&quot;&gt;DSAA 2015&lt;/a&gt; was in Paris, the &lt;a href=&quot;http://www.idawise.com/News/Dsaa2016.html&quot; rel=&quot;nofollow&quot;&gt;DSAA 2016&lt;/a&gt; edition is announced in Montreal, Canada, on October 17-19, 2016. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nuit-blanche.blogspot.fr/2015/11/pcmi-summer-school-mathematics-of-data.html&quot; rel=&quot;nofollow&quot;&gt;Nuit Blanche&lt;/a&gt; recently anounced the PCMI Summer school &quot;&lt;a href=&quot;https://pcmi.ias.edu/program-index/2016&quot; rel=&quot;nofollow&quot;&gt;The mathematics of data&lt;/a&gt;&quot;, June-July 2016, Midway, Utah, USA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other related conferences and workshops are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;COLT: &lt;a href=&quot;http://www.learningtheory.org/&quot; rel=&quot;nofollow&quot;&gt;Annual Conference on Learning Theory&lt;/a&gt; (&lt;a href=&quot;http://www.learningtheory.org/colt2016/&quot; rel=&quot;nofollow&quot;&gt;COLT 2016&lt;/a&gt;, 23-26 June, New-York, USA) &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;videos: &lt;a href=&quot;http://videolectures.net/colt2013_princeton/&quot; rel=&quot;nofollow&quot;&gt;2013 Princeton&lt;/a&gt;, &lt;a href=&quot;http://videolectures.net/colt2014_barcelona/&quot; rel=&quot;nofollow&quot;&gt;2014 Barcelona&lt;/a&gt;, &lt;a href=&quot;http://videolectures.net/colt2015_paris/&quot; rel=&quot;nofollow&quot;&gt;2015 Paris&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;MMDS: &lt;a href=&quot;http://mmds-data.org/&quot; rel=&quot;nofollow&quot;&gt;Workshop on Algorithms for Modern Massive Data Sets&lt;/a&gt; (&lt;a href=&quot;http://mmds-data.org/&quot; rel=&quot;nofollow&quot;&gt;MMDS 2016&lt;/a&gt; edition Berkeley CA 21-24/06/2016)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;videos: &lt;a href=&quot;https://www.youtube.com/user/MMDSworkshop&quot; rel=&quot;nofollow&quot;&gt;MMDSworkshop YouTube channel&lt;/a&gt;, and follow tabs for MMDS 2012 videos, or &lt;a href=&quot;https://www.youtube.com/playlist?list=PLl_tHArJvfnBS-OPQa2jFYhb1ZrCAnyyL&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;ICML: &lt;a href=&quot;http://www.machinelearning.org/icml.html&quot; rel=&quot;nofollow&quot;&gt;International Conference on Machine Learning&lt;/a&gt; (&lt;a href=&quot;http://icml.cc/2016/&quot; rel=&quot;nofollow&quot;&gt;ICML 2016&lt;/a&gt;, June, New York, USA)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;videos: &lt;a href=&quot;http://videolectures.net/icml2015_lille/&quot; rel=&quot;nofollow&quot;&gt;2015 Lille&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;NIPS: &lt;a href=&quot;https://nips.cc/&quot; rel=&quot;nofollow&quot;&gt;Annual Conference on Neural Information Processing Systems&lt;/a&gt; with NIPS 2016 in Barcelona, 5-10/12/2016&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;videos: NIPS 2015 &lt;a href=&quot;https://nips.cc/Conferences/2015/Schedule?type=Tutorial&quot; rel=&quot;nofollow&quot;&gt;tutorials&lt;/a&gt;, &lt;a href=&quot;https://nips.cc/Conferences/2015/Schedule?type=Invited%20Talk&quot; rel=&quot;nofollow&quot;&gt;invited speakers&lt;/a&gt;, &lt;a href=&quot;https://nips.cc/Conferences/2015/Schedule?type=Spotlight&quot; rel=&quot;nofollow&quot;&gt;spotlights&lt;/a&gt;, &lt;a href=&quot;https://www.youtube.com/user/NeuralInformationPro&quot; rel=&quot;nofollow&quot;&gt;NIPS 2014&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="12527" LastEditDate="2016-05-03T13:02:21.973" LastActivityDate="2016-05-03T13:02:21.973" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8965" PostTypeId="2" ParentId="8963" CreationDate="2015-11-21T09:19:31.403" Score="12" Body="&lt;p&gt;Start with the &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;Coursera's Machine Learning&lt;/a&gt; course. It does a really good job in introducing the student to the domain of Machine Learning and helps you lay a solid foundation in the concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case, you feel that the math is a bit dumbed down in that course, you can take &lt;a href=&quot;https://www.youtube.com/playlist?list=PLA89DCFA6ADACE599&quot; rel=&quot;nofollow&quot;&gt;this course&lt;/a&gt;, taught by the same professor and is more math-intensive than the former.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you would have a clear intuition about the basic concepts of Machine Learning. Take &lt;a href=&quot;https://www.youtube.com/playlist?list=PLD63A284B7615313A&quot; rel=&quot;nofollow&quot;&gt;this course&lt;/a&gt;, which can be said as a follow-up or a supplement to the course of Andrew Ng.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://homepages.inf.ed.ac.uk/rbf/IAPR/researchers/MLPAGES/mltut.htm&quot; rel=&quot;nofollow&quot;&gt;This resource from IAPR&lt;/a&gt; has in-depth notes on a lot of ML concepts like cross-validation, regularization, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also have a look at these &lt;a href=&quot;https://www.quora.com/profile/Eren-Golge/Machine-Learning/Some-of-useful-machine-learning-resources-from-beginner-to-intermediate&quot; rel=&quot;nofollow&quot;&gt;amazing list of resources&lt;/a&gt; compiled into a blog on Quora.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for diving into advanced concepts of neural networks and deep learning, you can make use of &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow&quot;&gt;this free book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the free e-book: &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning&lt;/a&gt; is a wonderful book for beginners in ML or Statistical Learning.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="15197" LastEditDate="2016-01-07T12:09:29.697" LastActivityDate="2016-01-07T12:09:29.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8967" PostTypeId="1" AcceptedAnswerId="26803" CreationDate="2015-11-21T23:02:02.990" Score="4" ViewCount="2177" Body="&lt;p&gt;My main question is it looks like &lt;a href=&quot;http://zeppelin-project.org/&quot; rel=&quot;nofollow&quot;&gt;Zeppelin&lt;/a&gt; limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default plotting tool of Zeppelin also plot the first 1000 results.&#xA;Is there a configuration or a way to make the plotting tool plot all the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If no, is there any equivalent for Matlibplot to Scala on Zeppelin?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="11097" LastEditDate="2015-11-22T11:11:32.640" LastActivityDate="2022-03-05T04:19:35.897" Title="Plotting libraries for Scala on Zeppelin" Tags="|visualization|scala|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="8968" PostTypeId="5" CreationDate="2015-11-22T05:40:44.037" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;correlation&lt;/a&gt; between two random variables $X$ and $Y$ is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ {\rm cor}(X,Y) = \frac{ E(XY) - E(X)E(Y) }{ \sqrt{ {\rm var}(X) {\rm var}(Y) } }$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and is bounded between $-1$ (perfect negative linear relationship) and $1$ (perfect positive linear relationship). The numerator of ${\rm cor}(X,Y)$ is known as the &lt;a href=&quot;http://en.wikipedia.org/wiki/Covariance&quot; rel=&quot;nofollow&quot;&gt;covariance&lt;/a&gt; between $X$ and $Y$. &lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:10:32.100" LastActivityDate="2015-11-22T11:10:32.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8969" PostTypeId="4" CreationDate="2015-11-22T05:40:44.037" Score="0" Body="A measure of the degree of linear association among a pair of variables." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:10:12.697" LastActivityDate="2015-11-22T11:10:12.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8970" PostTypeId="5" CreationDate="2015-11-22T05:41:19.620" Score="0" Body="&lt;p&gt;&lt;em&gt;From &lt;a href=&quot;http://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/&lt;/a&gt;:&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To run programs faster, Spark offers a general execution model that can optimize arbitrary operator graphs, and supports in-memory computing, which lets it query data faster than disk-based engines like &lt;a href=&quot;/questions/tagged/hadoop&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;hadoop&amp;#39;&quot; rel=&quot;tag&quot;&gt;hadoop&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark is not tied to the two-stage &lt;a href=&quot;/questions/tagged/mapreduce&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;mapreduce&amp;#39;&quot; rel=&quot;tag&quot;&gt;mapreduce&lt;/a&gt; paradigm, and promises performance up to 100 times faster than Hadoop MapReduce&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark provides primitives for in-memory cluster computing that allows user programs to load data into a cluster's memory and query it repeatedly, making it well suited to machine learning algorithms.&#xA;To make programming faster, Spark provides clean, concise APIs in &lt;a href=&quot;/questions/tagged/scala&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;scala&amp;#39;&quot; rel=&quot;tag&quot;&gt;scala&lt;/a&gt;, &lt;a href=&quot;/questions/tagged/java&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;java&amp;#39;&quot; rel=&quot;tag&quot;&gt;java&lt;/a&gt; and &lt;a href=&quot;/questions/tagged/python&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;python&amp;#39;&quot; rel=&quot;tag&quot;&gt;python&lt;/a&gt;. You can also use Spark interactively from the &lt;a href=&quot;/questions/tagged/scala&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;scala&amp;#39;&quot; rel=&quot;tag&quot;&gt;scala&lt;/a&gt; and &lt;a href=&quot;/questions/tagged/python&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;python&amp;#39;&quot; rel=&quot;tag&quot;&gt;python&lt;/a&gt; shells to rapidly query big datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark runs on &lt;a href=&quot;/questions/tagged/yarn&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;yarn&amp;#39;&quot; rel=&quot;tag&quot;&gt;yarn&lt;/a&gt;, &lt;a href=&quot;/questions/tagged/mesos&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;mesos&amp;#39;&quot; rel=&quot;tag&quot;&gt;mesos&lt;/a&gt;, standalone, or in the cloud. It can access diverse data sources including &lt;a href=&quot;/questions/tagged/hdfs&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;hdfs&amp;#39;&quot; rel=&quot;tag&quot;&gt;hdfs&lt;/a&gt;, &lt;a href=&quot;/questions/tagged/cassandra&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;cassandra&amp;#39;&quot; rel=&quot;tag&quot;&gt;cassandra&lt;/a&gt;, &lt;a href=&quot;/questions/tagged/hbase&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;hbase&amp;#39;&quot; rel=&quot;tag&quot;&gt;hbase&lt;/a&gt;, and &lt;a href=&quot;/questions/tagged/amazon-s3&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;amazon-s3&amp;#39;&quot; rel=&quot;tag&quot;&gt;amazon-s3&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recommended reference sources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://spark.apache.org/documentation.html&quot; rel=&quot;nofollow&quot;&gt;Spark Documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920028512.do&quot; rel=&quot;nofollow&quot;&gt;Learning Spark&lt;/a&gt; - Lightning-Fast Big Data Analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/5/&quot; rel=&quot;nofollow&quot;&gt;AMP Camp 5&lt;/a&gt; (Berkeley, CA, November 20-21, 2014) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/4/&quot; rel=&quot;nofollow&quot;&gt;AMP Camp 4&lt;/a&gt; (Strata Santa Clara, Feb 2014) — focus on BlinkDB, MLlib, GraphX, Tachyon&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/3/&quot; rel=&quot;nofollow&quot;&gt;AMP Camp 3&lt;/a&gt; (Berkeley, CA, Aug 2013)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/amp-camp-two-strata-2013/&quot; rel=&quot;nofollow&quot;&gt;AMP Camp 2&lt;/a&gt; (Strata Santa Clara, Feb 2013)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/agenda-2012/&quot; rel=&quot;nofollow&quot;&gt;AMP Camp 1&lt;/a&gt; (Berkeley, CA, Aug 2012)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:47.883" LastActivityDate="2015-12-03T22:02:47.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8971" PostTypeId="4" CreationDate="2015-11-22T05:41:19.620" Score="0" Body="Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:31.603" LastActivityDate="2015-12-03T22:02:31.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8972" PostTypeId="5" CreationDate="2015-11-22T05:41:54.860" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-22T05:41:54.860" LastActivityDate="2015-11-22T05:41:54.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8973" PostTypeId="4" CreationDate="2015-11-22T05:41:54.860" Score="0" Body="Social network data consists of a collection of &quot;nodes&quot; (which can be any sort of entity - e.g. people, corporations) and &quot;links&quot; (which can be any sort of relationship - e.g. friend, sharing a board member). " OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:10:15.960" LastActivityDate="2015-11-22T11:10:15.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8974" PostTypeId="5" CreationDate="2015-11-22T05:42:25.853" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-22T05:42:25.853" LastActivityDate="2015-11-22T05:42:25.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8975" PostTypeId="4" CreationDate="2015-11-22T05:42:25.853" Score="0" Body="Dimensionality reduction refers to techniques for reducing many variables into a smaller number while keeping as much information as possible. One prominent method is [tag pca]" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:11:25.570" LastActivityDate="2015-11-22T11:11:25.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8976" PostTypeId="5" CreationDate="2015-11-22T05:42:59.207" Score="0" Body="&lt;p&gt;For analysis, categorical values are considered as abstract entities without any mathematical structure such as an order or a topology, regardless of how they are coded and stored.  For more, see &lt;a href=&quot;http://en.wikipedia.org/wiki/Categorical_variable&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:11:07.397" LastActivityDate="2015-11-22T11:11:07.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8977" PostTypeId="4" CreationDate="2015-11-22T05:42:59.207" Score="0" Body="Categorical data can take on a limited (usually fixed) number of possible values called categories. Categorical values &quot;label&quot;, they do not &quot;measure&quot;. Nominal and dichotomous/binary scale types are categorical. Some people consider ordinal scale categorical too." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-22T11:11:19.650" LastActivityDate="2015-11-22T11:11:19.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8978" PostTypeId="1" AcceptedAnswerId="8979" CreationDate="2015-11-22T16:18:47.337" Score="6" ViewCount="1420" Body="&lt;p&gt;Is XGBoost complete by itself for prod-strength machine learning?  If not, with which other tools or libs is it typically combined, and how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I recently read a description of a stack that included ca 5 pieces, including XGBoost and Keras.)&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="11097" LastEditDate="2015-11-22T16:44:20.307" LastActivityDate="2015-11-22T17:00:41.070" Title="How to place XGBoost in a full stack for ML?" Tags="|machine-learning|python|tools|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8979" PostTypeId="2" ParentId="8978" CreationDate="2015-11-22T17:00:41.070" Score="4" Body="&lt;p&gt;Yes, it is a full-strength Machine Learning paradigm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBoost is basically Extreme Gradient Boosting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It only takes in numeric matrix data. So, you might want to convert your data such that it is compatible with XGBoost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://xgboost.readthedocs.org/en/latest/parameter.html&quot; rel=&quot;nofollow&quot;&gt;wide range of parameters&lt;/a&gt; of the xgboost paradigm is what makes it so diverse. Boosting can be done on trees and linear models, and then more parameters can be defined depending on the model you have selected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, yes it is a complete paradigm in itself. But, when you want more than the limitations of xgboost like linear and tree models, then you can use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;concept of ensembling&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of ensembles, the tools/libraries which can be used depends on the data scientist who is conducting the experiment. It can be Keras or Theano or TensorFlow, or anything which he/she is comfortable with. (opinion-based)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-22T17:00:41.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="8980" PostTypeId="2" ParentId="8941" CreationDate="2015-11-22T17:42:25.770" Score="4" Body="&lt;p&gt;Just went to this one last week (Open Data Science Conference): &lt;a href=&quot;http://odsc.com/&quot; rel=&quot;nofollow&quot;&gt;http://odsc.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was really enjoyable. Heavy focus on open source technologies. Speakers from high profile (within the community) projects. And they have posted a lot of the videos from it too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/playlist?list=PLB2SCq-tZtVl2qWH4CjizIb-IrkT_RA1f&quot; rel=&quot;nofollow&quot;&gt;Playlist&lt;/a&gt; of the talk video.&lt;/p&gt;&#xA;" OwnerUserId="14258" LastEditorUserId="11097" LastEditDate="2016-02-09T14:32:02.853" LastActivityDate="2016-02-09T14:32:02.853" CommentCount="3" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8982" PostTypeId="1" CreationDate="2015-11-23T05:47:01.023" Score="1" ViewCount="184" Body="&lt;p&gt;I have a data containing 200K ads (sale and rent property in the United Arab Emirates, or UAE). Here is a sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;190459 obs. of  29 variables:&#xA; $ id                  : chr  &quot;tnydu.biz/DafdVO&quot; &quot;tnydu.biz/DafloP&quot; &quot;tnydu.biz/Dafmvy&quot; &quot;tnydu.biz/Dafuy5&quot; ...&#xA; $ AdType              : chr  &quot;sale&quot; &quot;sale&quot; &quot;sale&quot; &quot;sale&quot; ...&#xA; $ AgencyFees          : num  NA NA NA NA NA NA NA NA NA NA ...&#xA; $ Amenities           : chr  &quot;CentralA/C&amp;amp;Heating,SharedPool,SharedGym,Security,BuiltinWardrobes,ViewofWater,ViewofLandmark&quot; &quot;Study,CentralA/C&amp;amp;Heating,Balcony,Security,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchenAppliances,Vi&quot;| __truncated__ &quot;CentralA/C&amp;amp;Heating,SharedSpa,Security,ConciergeService,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchen&quot;| __truncated__ &quot;CentralA/C&amp;amp;Heating,Balcony,Security,CoveredParking,BuiltinWardrobes,BuiltinKitchenAppliances,ViewofLandmark&quot; ...&#xA; $ AnnualCommunityFee  : chr  &quot;9000&quot; NA NA NA ...&#xA; $ AreaDescription     : chr  &quot;\r\n            \r\n                \r\n                    1.9 km from Najmat Reem Marina\r\n                \r\n            \&quot;| __truncated__ &quot;\r\n            \r\n                \r\n                    0.3 km from The Dubai Mall\r\n                \r\n            \r\n &quot;| __truncated__ &quot;\r\n            \r\n                \r\n            \r\n        &quot; &quot;\r\n            \r\n                \r\n                    0.7 km from The Dubai Mall\r\n                \r\n            \r\n &quot;| __truncated__ ...&#xA; $ Bathrooms           : num  1 NA NA NA 5 3 NA NA 3 3 ...&#xA; $ Bedrooms            : num  0 3 0 1 3 2 2 2 3 3 ...&#xA; $ Building            : chr  &quot;HydraAvenueTowers&quot; &quot;BurjVista1&quot; &quot;TheAddressDubaiMall&quot; &quot;TheAddressDowntown&quot; ...&#xA; $ City                : chr  &quot;AbuDhabi&quot; &quot;Dubai&quot; &quot;Dubai&quot; &quot;Dubai&quot; ...&#xA; $ PublishDate         : POSIXct, format: &quot;2015-10-30&quot; &quot;2015-11-11&quot; &quot;2015-11-13&quot; &quot;2015-11-09&quot; ...&#xA; $ DealerCode          : chr  &quot;599942&quot; &quot;604296&quot; &quot;604296&quot; &quot;604296&quot; ...&#xA; $ DealerName          : chr  &quot;STARWOOD PROPERTIES BROKER&quot; &quot;BLUE PALACE REAL ESTATE BROKERS&quot; &quot;BLUE PALACE REAL ESTATE BROKERS&quot; &quot;BLUE PALACE REAL ESTATE BROKERS&quot; ...&#xA; $ Developer           : chr  &quot;HydraProperties&quot; &quot;EMAAR&quot; &quot;EMAAR&quot; NA ...&#xA; $ Furnished           : chr  NA NA NA NA ...&#xA; $ ListedBy            : chr  &quot;Agent&quot; &quot;Agent&quot; &quot;Agent&quot; &quot;Agent&quot; ...&#xA; $ Location            : chr  &quot;City of Lights, Tamouh Marina Square&quot; &quot;Downtown Dubai, Dubai&quot; &quot;Downtown, Cairo&quot; &quot;Downtown Dubai, Dubai&quot; ...&#xA; $ LocationGPSLatitude : num  24.5 25.2 30 25.2 25.1 ...&#xA; $ LocationGPSLongitude: num  54.4 55.3 31.3 55.3 55.1 ...&#xA; $ Price               : num  900000 3822888 2150000 3200000 5500000 ...&#xA; $ PriceSqFt           : num  1129 2185 3909 3422 2431 ...&#xA; $ PropertyReference   : chr  NA &quot;BP9801&quot; &quot;DT-K20&quot; &quot; BP8958&quot; ...&#xA; $ PropertyType        : chr  &quot;apartment&quot; &quot;apartment&quot; &quot;apartment&quot; &quot;apartment&quot; ...&#xA; $ ReadyBy             : POSIXct, format: &quot;2015-04-30&quot; NA NA NA ...&#xA; $ RentIsPaid          : chr  NA NA NA NA ...&#xA; $ ShortLink           : chr  &quot;tnydu.biz/DafdVO&quot; &quot;tnydu.biz/DafloP&quot; &quot;tnydu.biz/Dafmvy&quot; &quot;tnydu.biz/Dafuy5&quot; ...&#xA; $ Size                : num  797 1749 550 935 2262 ...&#xA; $ TotalClosingFee     : chr  NA NA NA NA ...&#xA; $ VirtualView         : chr  NA NA NA NA ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm looking for help figuring out what kind of questions I can answer using these data.&#xA;For example,&#xA;1. What the mean price per year for a 1-bedroom apartment to rent in different areas of Dubai?&#xA;2. How are supplies for apartments and villas distributed on the city map?&#xA;3. Does the number of bathrooms influence on sale/rent price?&#xA;4. Where is the best place for investing to get more profit from apartment cost and rent revenue? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already done some of this (plotted the map with mean price for studio in Dubai in thousands AED/year, 1USD = 3.66AED):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HDm3t.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HDm3t.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your experience, what other questions can I ask? What I have to be aware in this data set? I can share my data set if anyone want to examine it (it is a 12mb zip).&lt;/p&gt;&#xA;" OwnerUserId="14267" LastEditorUserId="13413" LastEditDate="2015-11-24T01:52:12.407" LastActivityDate="2016-03-23T20:55:47.693" Title="What kind of research I can do with my data set?" Tags="|visualization|data|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8983" PostTypeId="1" CreationDate="2015-11-23T09:40:01.580" Score="2" ViewCount="553" Body="&lt;p&gt;I'm participating in a Kaggle contest &lt;a href=&quot;https://www.kaggle.com/c/whats-cooking&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;What's cooking&quot;&lt;/a&gt;. The goal is to know wich kind of cuisine we have, depending on some ingredients. So it's a multiclass classification problem. I have an existing model, and I have been trying to improve it for 2 weeks without result. I am using scikit learn, and my existing model is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipelineLR = Pipeline([&#xA;('vect', CountVectorizer(max_df=0.75)),&#xA;('lr', LogisticRegression(C=0.4,solver='lbfgs',multi_class='ovr',fit_intercept=True,warm_start=True)),&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's really basic. But the problem of this dataset is that we have unbalanced data. Also we have a a huge sparse matrix because of the CountVectorizer. I tried a lot of different model, with and without TF-IDF and this model is the best I can get. I have also confusion matrix, classification report and learning curve for it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can notice is that is have high-variance. I think I can do something by preprocessing the data but I am really stuck. I don't know if my model is good enough to be improved, or if I have to find a totally new approach to it. Also what I can get from the classification report is that the model is working well for classes where I have a lot of data, but it's making a lot of mistakes for classes with less data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am very new to data science and I am trying to learn by myself. But I think I really need to have another point of view with this problem, because I can't manage to improve my score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8LrDq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8LrDq.png&quot; alt=&quot;Classification report&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/JBUEP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JBUEP.png&quot; alt=&quot;Learning curves&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14268" LastEditorUserId="31513" LastEditDate="2017-05-11T14:26:29.900" LastActivityDate="2017-05-11T15:20:05.090" Title="What's cooking Kaggle - Improve model" Tags="|multiclass-classification|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8984" PostTypeId="2" ParentId="8941" CreationDate="2015-11-23T10:42:18.230" Score="1" Body="&lt;p&gt;Lots of nice conferences already mentioned by the existing answers. Here are some which I think deserve a place in the top conferences list:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mlconf.com/&quot; rel=&quot;nofollow&quot;&gt;MLConf&lt;/a&gt;:  As the name puts: &lt;em&gt;THE MACHINE LEARNING CONFERENCE&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Purpose of the conference:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MLconf gathers communities to discuss the recent research and&#xA;  application of Algorithms, Tools, and Platforms to solve the hard&#xA;  problems that exist within organizing and analyzing massive and noisy&#xA;  data sets.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/channel/UCjeM1xxYb_37bZfyparLS3Q&quot; rel=&quot;nofollow&quot;&gt;Videos link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.baylearn.org/&quot; rel=&quot;nofollow&quot;&gt;Baylearn&lt;/a&gt;: The Bay Area Machine Learning symposium.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Purpose:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The BayLearn Symposium aims at gathering scientists in machine&#xA;  learning from the San Francisco Bay Area. While it promotes community&#xA;  building between local researchers from academic and industrial&#xA;  institutions, it also welcomes visitors. This one-day event combines&#xA;  invited talks and posters to foster exchange of ideas.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/channel/UCnf609gsCjoJd_GvQ31SFVA&quot; rel=&quot;nofollow&quot;&gt;Link to videos&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://fifthelephant.in&quot; rel=&quot;nofollow&quot;&gt;Fifth Elephant&lt;/a&gt;: &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Fifth Elephant is a community-powered conference on the big data&#xA;  ecosystem, covering the technology and its applications in a range of&#xA;  industries.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://hasgeek.tv/fifthelephant/&quot; rel=&quot;nofollow&quot;&gt;Link to videos&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-23T10:42:18.230" CommentCount="2" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8985" PostTypeId="5" CreationDate="2015-11-23T10:57:43.013" Score="0" Body="&lt;p&gt;&lt;strong&gt;Overview&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Markov process is any stochastic process $Y_{t}$ such that the future is conditionally independent of the past, given the present; the distribution of the process only depends on where the process is, not where it has been: &#xA;$$ P(Y_{t+1}=y_{t+1} |Y_t = y_{t}, Y_{t-1} = y_{t-1}, ..., Y_{1} = y_{1}) =  P(Y_{t+1}=y_{t+1} |Y_t = y_{t}) $$&#xA;This property is known as the Markov property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following threads on &lt;a href=&quot;https://math.stackexchange.com/&quot;&gt;math.se&lt;/a&gt; provide references to resources on Markov processes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://math.stackexchange.com/q/15431/84650&quot;&gt;Good introductory book for Markov processes&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://math.stackexchange.com/q/27514/84650&quot;&gt;Nice references on Markov chains/processes?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="3117" LastEditDate="2017-08-27T14:55:33.893" LastActivityDate="2017-08-27T14:55:33.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8986" PostTypeId="4" CreationDate="2015-11-23T10:57:43.013" Score="0" Body="A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-22T07:48:52.403" LastActivityDate="2016-01-22T07:48:52.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8987" PostTypeId="5" CreationDate="2015-11-23T10:58:13.177" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-23T10:58:13.177" LastActivityDate="2015-11-23T10:58:13.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8988" PostTypeId="4" CreationDate="2015-11-23T10:58:13.177" Score="0" Body="Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the data during model fitting and quantifying how accurate the withheld data are predicted and repeating this process to get a measure of prediction accuracy." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-25T12:54:25.243" LastActivityDate="2015-11-25T12:54:25.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8989" PostTypeId="5" CreationDate="2015-11-23T10:59:40.870" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-23T10:59:40.870" LastActivityDate="2015-11-23T10:59:40.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8990" PostTypeId="4" CreationDate="2015-11-23T10:59:40.870" Score="0" Body="Structured Query Language (SQL) is a language for querying databases. Questions should include code examples, table structure, sample data, and a tag for the DBMS implementation (e.g. MySQL, PostgreSQL, Oracle, MS SQL Server, IBM DB2, etc.) being used." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-04T15:37:58.157" LastActivityDate="2019-04-04T15:37:58.157" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="8991" PostTypeId="5" CreationDate="2015-11-23T11:01:16.990" Score="0" Body="&lt;p&gt;CSV is a file format describing a plain text file with information separated by commas (&lt;code&gt;,&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While technically plain text files (&lt;code&gt;.txt&lt;/code&gt;), these files are given the &lt;code&gt;.csv&lt;/code&gt; extension to indicate that the data is delimited by commas and can be afforded special parsing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MIME type for CSV files is &lt;code&gt;text/csv&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Information is often stored as CSV format to make it easy to transfer tables of data between applications. Each row of a table is represented as a list of plain text (human-readable) values with commas as delimiters between each discrete piece of data. Values may be enclosed in quotes, especially if they contain spaces (which might otherwise be parsed as delimiters), or if the data itself contains commas or line breaks. The first row of data often contains the headings of the table columns, which describe the meaning of the data in each column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tabular format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-------+-------------+----------+----------------------+&#xA;| Time  | Temperature | Humidity | Description          |&#xA;+-------|-------------|----------|----------------------+&#xA;| 08:00 |     70      |    35    | Sunny and Clear      |&#xA;| 11:45 |     94      |    90    | Hazy, Hot, and Humid |&#xA;| 14:30 |     18      |          | Freezing             |&#xA;+-------+-------------+----------+----------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;CSV format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Time,Temperature,Humidity,Description&#xA;08:00,70,35,&quot;Sunny and Clear&quot;&#xA;11:45,94,90,&quot;Hazy, Hot, and Humid&quot;&#xA;14:30,18,,Freezing&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example, the first row of CSV data serves as the &quot;header&quot; which describes the corresponding data below it. Each successive line of the CSV file would then neatly fit into the same field as the first line. There is no inherent way to describe within a CSV file whether the first row is a header row or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that empty fields (fields with no available data, such as the third field in the last line) are place-held with commas so that the fields that follow may be correctly placed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions tagged &lt;a href=&quot;/questions/tagged/csv&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;csv&amp;#39;&quot; rel=&quot;tag&quot;&gt;csv&lt;/a&gt; are expected to relate to programming in some way, for example, parsing/importing CSV files or creating them programmatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Related links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stackoverflow.com/q/1382604/190829&quot;&gt;What to set as mimetype for csv files to open in spreadsheet applications&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://tools.ietf.org/html/rfc4180&quot; rel=&quot;nofollow&quot;&gt;RFC 4180&lt;/a&gt; (describes official CSV format)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://stackoverflow.com/tags/csv/info&quot;&gt;Stack Overflow SE&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-25T12:54:19.507" LastActivityDate="2015-11-25T12:54:19.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8992" PostTypeId="4" CreationDate="2015-11-23T11:01:16.990" Score="0" Body="Comma-Separated Values are a list of plain text values delimited by commas, or a file containing one or more lists in that format." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-25T12:54:50.027" LastActivityDate="2015-11-25T12:54:50.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8993" PostTypeId="1" CreationDate="2015-11-23T16:49:32.403" Score="2" ViewCount="56" Body="&lt;p&gt;How to select a bunch of optimized data from a larger data set? I have a data set containing the products to be sold to the customers. And the products were sold before, it has some information, like the review, ratings, how many were sold per product from the customers. Now, I want to select a bunch of optimized data from a larger data set based on the number of units sold, ratings, review etc all the information. The purpose is to increase the number of units to be sold in the future. How to do this? What kind of model, method, algorithms should be used here? If it is related to machine learning, what kind of machine learning tools should I use here? Thank you so much. &lt;/p&gt;&#xA;" OwnerUserId="4619" LastActivityDate="2015-11-23T16:49:32.403" Title="How to select a bunch of optimized data from a larger data set?" Tags="|optimization|data|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="8994" PostTypeId="2" ParentId="8941" CreationDate="2015-11-23T20:39:48.233" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://passbaconference.com/2016/Home.aspx&quot; rel=&quot;nofollow&quot;&gt;PASS Analytics&lt;/a&gt; - The Professional Association for SQL Server has an analytics focused conference. While some of the conference tracks may not be for highly qualified data scientists, many are specifically relevant as Microsoft continues to support Hadoop and integrates R into SQL Server among many other things.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Whether you’re just starting out in the data analytics field, are a&#xA;  data pro who sees the value in expanding your skillset, or are a&#xA;  seasoned business analytics pro, the PASS Business Analytics&#xA;  Conference is the place to be.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="13578" LastActivityDate="2015-11-23T20:39:48.233" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="8997" PostTypeId="2" ParentId="8982" CreationDate="2015-11-24T00:56:24.300" Score="0" Body="&lt;p&gt;Feature Selection is a pretty good starting point. U can use statistical or information theoretic approaches (e.g. variance, entropy, ...) to choose &lt;em&gt;k&lt;/em&gt; best features which influence the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering seems attractive here as well. Cluster houses based on different properties to disclose hidden pattern e.g. the relation between geographic location and price as a simple one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also build a model from which one can predict the price of a house given its features.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-11-24T00:56:24.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8998" PostTypeId="2" ParentId="8982" CreationDate="2015-11-24T09:04:15.257" Score="0" Body="&lt;p&gt;Geographically weighted regression would be one way to see what factors correlate with price and how that model changes over space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Used with caution it can also help predict expected prices in the intervening locations i.e. if the data points are suitably distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GWR is available in e.g. R and ArcGIS&lt;/p&gt;&#xA;" OwnerUserId="13955" LastActivityDate="2015-11-24T09:04:15.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="8999" PostTypeId="1" AcceptedAnswerId="9010" CreationDate="2015-11-24T09:13:14.750" Score="5" ViewCount="2817" Body="&lt;p&gt;I recently came across a paper about doing semantic segmentation using deconvolutional network: &lt;strong&gt;&lt;a href=&quot;http://arxiv.org/abs/1505.04366&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Deconvolution Network for Semantic Segmentation&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The basic structure of the network is like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/lm5wv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lm5wv.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The goal is to generate a probability map in the end. I'm having trouble figuring out how to realize the deconvolution layer. In the paper, it says:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The output of an unpooling layer is an enlarged, yet sparse activation map. The deconvolution layers densify the sparse activations obtained by unpooling through convolution-like operations with multiple learned filters. However, contrary to convolutional layers, which connect multiple input activations within a filter window to a single activation, deconvolutional layers associate a single input activation with multiple outputs.&lt;/p&gt;&#xA;&lt;p&gt;The output of the deconvolutional layer is an enlarged and dense activation map. We crop the boundary of the enlarged activation map to keep the size of the output map identical to the one from the preceding unpooling layer.&lt;/p&gt;&#xA;&lt;p&gt;The learned filters in deconvolutional layers correspond to bases to reconstruct shape of an input object. Therefore, similar to the convolution network, a hierarchical structure of deconvolutional layers are used to capture different level of shape details. The filters in lower layers tend to capture overall shape of an object while the class-specific fine details are encoded in the filters in higher layers. In this way, the network directly takes class-specific shape information into account for semantic segmentation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Can anyone explain how the deconvolution works? I'm guessing it's not a simple interpolation.&lt;/p&gt;&#xA;" OwnerUserId="14293" LastEditorUserId="98307" LastEditDate="2020-08-13T11:24:52.417" LastActivityDate="2020-08-13T11:24:52.417" Title="Deconvolutional Network in Semantic Segmentation" Tags="|neural-network|deep-learning|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9000" PostTypeId="1" AcceptedAnswerId="9004" CreationDate="2015-11-24T10:36:18.830" Score="16" ViewCount="3302" Body="&lt;p&gt;What are some podcasts which are related to data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a similar question to the &lt;a href=&quot;https://stats.stackexchange.com/questions/927/statistical-podcasts&quot;&gt;reference request question on CrossValidated&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Details/rules:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The podcasts (the theme and the episodes) should be related to data science. (For example: A podcast which is about some other domain, with an episode which speaks about data science in that domain, is not a good reference/answer.)&lt;/li&gt;&#xA;&lt;li&gt;Personal opinions/reviews (if any) would be very helpful too.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2023-01-24T04:52:11.627" Title="Data Science Podcasts?" Tags="|reference-request|" AnswerCount="7" CommentCount="1" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9001" PostTypeId="2" ParentId="9000" CreationDate="2015-11-24T10:45:09.197" Score="6" Body="&lt;p&gt;In &lt;a href=&quot;http://dataskeptic.com/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Data Skeptic&lt;/strong&gt;&lt;/a&gt; they talk about different aspects of data science:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data Skeptic is a podcast that alternates between short mini episodes with the host explaining concepts from data science to his non-data scientist wife, and longer interviews featuring practitioners and experts on interesting topics related to data, all through the eye of scientific skepticism.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a good intro to Data Science which also explains the basic concepts in a not too technical way.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A bit more technical is &lt;a href=&quot;http://www.partiallyderivative.com/&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;Partially Derivative&lt;/strong&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Partially Derivative is a podcast about the data of everything. Each week we look at a new way that data has changed how we understand and experience the world. From ancient warfare to modern love and everything in between.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A related Quora question: &lt;a href=&quot;https://www.quora.com/What-are-the-best-data-science-podcasts&quot; rel=&quot;nofollow&quot;&gt;What are the best data science podcasts?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13123" LastEditorUserId="13123" LastEditDate="2015-11-24T10:57:51.290" LastActivityDate="2015-11-24T10:57:51.290" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9002" PostTypeId="1" CreationDate="2015-11-24T11:01:43.763" Score="1" ViewCount="79" Body="&lt;p&gt;I am working on developing a predictive model using Random Forest. There are a lot of users that log in to the site but only a fraction of them actually monetize on that day. I am trying to predict the probability of monetization of a user during a login. The predictor variables have been selected. Now the issue is that one user can log in on multiple days and have different values of predictor variables and the label (Monetized/ Not Monetized) for each log in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try to train the data on the last 30 days' data, some users would occur multiple times and others would only occur one time. This may lead to a higher weight-age being given to the users who logged in on more days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I ensure that equal weightage is assigned to all the users even though the users might have different numbers of logins and hence different counts of data points.&lt;/p&gt;&#xA;" OwnerUserId="14294" LastActivityDate="2015-11-24T11:01:43.763" Title="Consolidating values for predictive modelling through Random Forest" Tags="|machine-learning|r|predictive-modeling|random-forest|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9003" PostTypeId="1" AcceptedAnswerId="9005" CreationDate="2015-11-24T11:50:46.290" Score="8" ViewCount="12619" Body="&lt;p&gt;I'm wondering if sometimes, to validate a model, it's not better to use aucPR instead of aucROC? Do these cases only depend on the &quot;domain  &amp;amp; business understanding&quot; ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially, I'm thinking about the &quot;unbalanced class problem&quot; where, it seems more logical to use the aucPR because recall and precision are well-used metrics for this problem.&lt;/p&gt;&#xA;" OwnerUserId="14200" LastEditorUserId="11097" LastEditDate="2015-11-24T11:53:44.003" LastActivityDate="2015-11-24T13:50:29.083" Title="When do I have to use aucPR instead of auROC? (and vice versa)" Tags="|machine-learning|data-mining|cross-validation|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9004" PostTypeId="2" ParentId="9000" CreationDate="2015-11-24T12:27:30.943" Score="13" Body="&lt;p&gt;I strongly suggest &lt;a href=&quot;http://www.thetalkingmachines.com&quot;&gt;Talking Machines&lt;/a&gt;. It's a very well put together podcast from a professor at Harvard. They cater to both machine learning experts and enthusiasts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their interviews are often done from NIPS, and the guests are usually top tier practitioners.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-24T12:27:30.943" CommentCount="1" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9005" PostTypeId="2" ParentId="9003" CreationDate="2015-11-24T13:50:29.083" Score="18" Body="&lt;p&gt;Yes, you are correct that the dominant difference between the area under the curve of a receiver operator characteristic curve (&lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;noreferrer&quot;&gt;ROC-AUC&lt;/a&gt;) and the area under the curve of a Precision-Recall curve (&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html&quot; rel=&quot;noreferrer&quot;&gt;PR-AUC&lt;/a&gt;) lies in its tractability for unbalanced classes.  They are very similar and have been shown to contain essentially the same information, however PR curves are slightly more finicky, but a well drawn curve gives a more complete picture.  The issue with PR-AUC is that its difficult to interpolate between points in the PR curve and thus numerical integration to achieve an area under the curve becomes more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf&quot; rel=&quot;noreferrer&quot;&gt;Check out this discussion of the differences and similarities.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting Davis' 2006 abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Receiver Operator Characteristic (ROC)&#xA;  curves are commonly used to present results&#xA;  for binary decision problems in machine&#xA;  learning. However, when dealing&#xA;  with highly skewed datasets, Precision-Recall&#xA;  (PR) curves give a more informative picture&#xA;  of an algorithm’s performance. We show that&#xA;  a deep connection exists between ROC space&#xA;  and PR space, such that a curve dominates&#xA;  in ROC space if and only if it dominates&#xA;  in PR space. A corollary is the notion of&#xA;  an achievable PR curve, which has properties&#xA;  much like the convex hull in ROC space;&#xA;  we show an efficient algorithm for computing&#xA;  this curve. Finally, we also note differences&#xA;  in the two types of curves are significant for&#xA;  algorithm design. For example, in PR space&#xA;  it is incorrect to linearly interpolate between&#xA;  points. Furthermore, algorithms that optimize&#xA;  the area under the ROC curve are not&#xA;  guaranteed to optimize the area under the&#xA;  PR curve.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/forums/f/15/kaggle-forum/t/7517/precision-recall-auc-vs-roc-auc-for-class-imbalance-problems/41179&quot; rel=&quot;noreferrer&quot;&gt;This was also discussed on Kaggle recently.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/7207/roc-vs-precision-and-recall-curves&quot;&gt;There is also some useful discussion on Cross Validated.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-24T13:50:29.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9006" PostTypeId="2" ParentId="9000" CreationDate="2015-11-24T14:00:52.920" Score="4" Body="&lt;p&gt;Some which I regularly hear to, are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://fivethirtyeight.com/tag/whats-the-point/&quot; rel=&quot;nofollow&quot;&gt;What's the point&lt;/a&gt; by FiveThirtyEight&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is a very nice podcast, where not only the concepts, but also the applications of data science to a wide range of domains, are discussed.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/podcasts/linear-digressions&quot; rel=&quot;nofollow&quot;&gt;Linear digressions&lt;/a&gt; by Udacity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It focuses more on ML and concepts of data science. Hosted by two really nice data scientists Katie and Ben.  All their episodes are fun to watch.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Partially Derivative  (Already mentioned by agold)&lt;/li&gt;&#xA;&lt;li&gt;Data Skeptic  (Already mentioned by agold)&lt;/li&gt;&#xA;&lt;li&gt;Talking Machine  (Already mentioned by jamesmf)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://freakonomics.com/radio/freakonomics-radio-podcast-archive/&quot; rel=&quot;nofollow&quot;&gt;Freakonomics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is a fun podcast which focuses on data and it's importance in various domains and how to make sense of it.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-24T14:00:52.920" CommentCount="1" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9007" PostTypeId="2" ParentId="8926" CreationDate="2015-11-24T16:33:41.683" Score="1" Body="&lt;p&gt;It's a slight variation on your problem as described, but if your goal is to build a good model for predicting documents labeled $L$, I would initially formulate this as a recommender system problem, until you've reached a desirable point in the learning curve of your system. I did exactly this in &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3872296/&quot; rel=&quot;nofollow&quot;&gt;a publication a few years back&lt;/a&gt;. In my approach, all labeled documents are used to train the SVM, while all those unlabeled are used for classifying. I use the signed margin distance of each classified document for ranking, and take the top-$n$ as the next $n$ documents the reviewers should assign a label $\in(L, L_{not})$. A side effect of this process is that you get to iteratively evaluate the performance of your classifier as you add new labeled documents to the model, which should get you what you need. If this approach sounds useful to you, I recommend looking over my paper, as I describe some performance metrics useful for this, in addition to outlining the specific procedure to use that will ensure you're not biasing your model. &lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-11-24T16:33:41.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9008" PostTypeId="1" CreationDate="2015-11-24T19:09:10.280" Score="0" ViewCount="50" Body="&lt;p&gt;What is the technical term for a report or dataset that compares data of an interval of time with that of the same interval in a previous week, month, or year?  &lt;/p&gt;&#xA;" OwnerUserId="14317" LastActivityDate="2015-11-24T21:07:46.043" Title="term for data that compares a day to same day of a week/month/year ago" Tags="|terminology|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9009" PostTypeId="2" ParentId="9008" CreationDate="2015-11-24T21:07:46.043" Score="1" Body="&lt;p&gt;A repeated &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-sectional_study&quot; rel=&quot;nofollow&quot;&gt;cross-sectional study&lt;/a&gt;. Cross-sectional refers to data collected at one specific point in time or over a short period. Repeated since you compare at least two time frames. If there are several time frames, one could talk about &lt;a href=&quot;http://as.nyu.edu/docs/IO/2576/beck.pdf&quot; rel=&quot;nofollow&quot;&gt;time-series cross-sectional (TSCS)&lt;/a&gt; data. Repeated cross-sectional studies may become pseudolongitudinal, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Longitudinal_study&quot; rel=&quot;nofollow&quot;&gt;longitudinal&lt;/a&gt; if carried over a longer lapse of time.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2015-11-24T21:07:46.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9010" PostTypeId="2" ParentId="8999" CreationDate="2015-11-24T22:24:44.447" Score="4" Body="&lt;p&gt;There are two main functions they undo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html#maxpooling&quot; rel=&quot;nofollow&quot;&gt;pooling layers&lt;/a&gt; in the convolutional neural network downsample the image by (usually) taking the maximum value within the receptive field.  Each &lt;code&gt;rxr&lt;/code&gt; image region is downsampled to a single value.  What this implementation does is store which unit had the maximum activation in each of these pooling steps.  Then, at each &quot;unpooling&quot; layer in the deconvolutional network, they upsample back to a &lt;code&gt;rxr&lt;/code&gt; image region, only propagating the activation to the location that produced the original max-pooled value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, &quot;the output of an unpooling layer is an enlarged, yet sparse activation map.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convolutional layers learn a filter for each image region that maps from a region of size &lt;code&gt;r x r&lt;/code&gt; to a single value, where &lt;code&gt;r&lt;/code&gt; is the receptive field size. The point of the deconvolutional layer is to learn the opposite filter.  This filter is a set of weights that projects an &lt;code&gt;rxr&lt;/code&gt; input into a space of size &lt;code&gt;sxs&lt;/code&gt;, where s is the size of the next convolutional layer.  These filters are learned in the same way that as regular convolutional layers are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the mirror image of a deep CNN, the low-level features of the network are really high-level, class-specific features.  Each layer in the network then localizes them, enhancing class-specific features while minimizing noise.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-24T22:24:44.447" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9011" PostTypeId="2" ParentId="9000" CreationDate="2015-11-24T23:47:14.847" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://soundcloud.com/nssd-podcast&quot; rel=&quot;nofollow&quot;&gt;Not So Standard Deviations&lt;/a&gt; by Hilary Parker and Roger Peng! Fun little podcast where they talk about a variety of things. They only have 4 episodes out currently, but they seem to be releasing one every two weeks, so there's that.&lt;/p&gt;&#xA;" OwnerUserId="13543" LastActivityDate="2015-11-24T23:47:14.847" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9012" PostTypeId="1" AcceptedAnswerId="9015" CreationDate="2015-11-25T00:03:55.700" Score="0" ViewCount="99" Body="&lt;p&gt;I have a CSV file that, by all appearances, is totally normal except that each line ends with &lt;code&gt;,^M&lt;/code&gt;.  Not sure if that has anything to do with my issue or not, but I try reading in the file in R with the usual command &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df &amp;lt;- read.csv('file.csv')&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then when I try to inspect it by typing &lt;code&gt;df&lt;/code&gt; it prints out this result that I don't know how to interpret:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function (x, df1, df2, ncp, log = FALSE) &#xA;{&#xA;    if (missing(ncp)) &#xA;        .External(C_df, x, df1, df2, log)&#xA;    else .External(C_dnf, x, df1, df2, ncp, log)&#xA;}&#xA;&amp;lt;bytecode: 0x000000000b083720&amp;gt;&#xA;&amp;lt;environment: namespace:stats&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Anybody know what's going on here?  I've read in CSV files before and had no problem.&lt;/p&gt;&#xA;" OwnerUserId="14327" LastActivityDate="2015-11-25T12:55:34.337" Title="Reading a CSV using R" Tags="|r|csv|" AnswerCount="2" CommentCount="3" ClosedDate="2017-12-18T05:14:31.533" ContentLicense="CC BY-SA 3.0" />
  <row Id="9013" PostTypeId="2" ParentId="9012" CreationDate="2015-11-25T01:30:54.147" Score="0" Body="&lt;p&gt;The ^M looks like the file came from a computer with a different OS from yours - maybe Unix and you're on Windows? Look on Google for a Unix2DOS utility, or maybe you can open it in Excel and then re-export to fix it.&lt;/p&gt;&#xA;" OwnerUserId="13320" LastEditorUserId="13413" LastEditDate="2015-11-25T12:55:34.337" LastActivityDate="2015-11-25T12:55:34.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9015" PostTypeId="2" ParentId="9012" CreationDate="2015-11-25T04:58:15.010" Score="0" Body="&lt;p&gt;Sorry, never mind, I just realized that I thought I was running the command to set my working directory to the correct file and it turns out I wasn't so when I was importing the file I was importing the wrong one.  All fixed.&lt;/p&gt;&#xA;" OwnerUserId="14327" LastActivityDate="2015-11-25T04:58:15.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9016" PostTypeId="1" CreationDate="2015-11-25T06:01:48.613" Score="1" ViewCount="78" Body="&lt;p&gt;I have data &amp;amp; a R script that creates a report from the data. I can't expose the data to internet. Also, I cant expose my script to internet / user. But I would like to eliminate myself from the work, and allow couple of users (yeah, only three users that use this script to generate reports, but they do it weekly) to run the script and generate report for them. &#xA;I would like the community to suggest me how I can achieve this. Any information helps. &lt;/p&gt;&#xA;" OwnerUserId="7923" LastEditorUserId="3151" LastEditDate="2015-12-01T20:27:21.113" LastActivityDate="2015-12-01T20:27:21.113" Title="How do I allow user to run a R script without making a website or web application?" Tags="|r|data-product|" AnswerCount="1" CommentCount="2" ClosedDate="2015-11-27T13:52:42.403" ContentLicense="CC BY-SA 3.0" />
  <row Id="9019" PostTypeId="2" ParentId="9000" CreationDate="2015-11-25T08:56:04.333" Score="2" Body="&lt;p&gt;Here are two other podcasts not mentioned previously:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ocdqblog.com/podcast/&quot; rel=&quot;nofollow&quot;&gt;OCDQ&lt;/a&gt;: for some interesting discussions on data quality and Big Data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.learningmachines101.com/&quot; rel=&quot;nofollow&quot;&gt;Learning Machine 101&lt;/a&gt;: for begginer level discussion on machine learning.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-11-25T08:56:04.333" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9020" PostTypeId="1" AcceptedAnswerId="9230" CreationDate="2015-11-25T11:11:25.923" Score="13" ViewCount="4106" Body="&lt;p&gt;I have a vector X with n features previously standardized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new features after the computing ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because knowing that my original features are already between 0 and 1, the new polynomial features will also be between 0 and 1. But the higher the degree is, the higher the &quot;ratio&quot; between the original one and the polynomial one will be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also,  would that be better to do the square computing on the values of the non standardized features, and then standardize these new features ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, when I say &quot;standardization&quot;, I m thinking about substrat each value by the mean and divide it by the std.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards.&lt;/p&gt;&#xA;" OwnerUserId="14200" LastActivityDate="2015-12-06T00:53:21.600" Title="Do I have to standardize my new polynomial features?" Tags="|machine-learning|dataset|data-cleaning|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9022" PostTypeId="1" AcceptedAnswerId="9031" CreationDate="2015-11-25T12:13:29.793" Score="7" ViewCount="20347" Body="&lt;p&gt;There are two &lt;code&gt;pd.DataFrame&lt;/code&gt;. First is like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;print df1&#xA;&#xA;        id        date    month  is_buy&#xA;     0  17  2015-01-16  2015-01       1&#xA;     1  17  2015-01-26  2015-01       1&#xA;     2  17  2015-01-27  2015-01       1&#xA;     3  17  2015-02-11  2015-02       1&#xA;     4  17  2015-03-14  2015-03       1&#xA;     5  18  2015-01-28  2015-01       1&#xA;     6  18  2015-02-12  2015-02       1&#xA;     7  18  2015-02-25  2015-02       1&#xA;     8  18  2015-03-04  2015-03       1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In second data frame there are some aggregated data by month from the first one:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df2 = df1[df1['is_buy'] == 1].groupby(['id', 'month']).agg({'is_buy': np.sum})&#xA;     &#xA;print df2&#xA;&#xA;        id    month       buys&#xA;     0  17  2015-01          3&#xA;     1  17  2015-02          1&#xA;     2  17  2015-03          1&#xA;     3  18  2015-01          1&#xA;     4  18  2015-02          2&#xA;     5  18  2015-03          1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I'm trying to get new &lt;code&gt;df2&lt;/code&gt; column named &lt;code&gt;'last_week_buys'&lt;/code&gt; with aggregated buys by last 7 days from first day of each &lt;code&gt;df1['month']&lt;/code&gt;. In other words, I want to get this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;        id    month       buys    last_week_buys&#xA;     0  17  2015-01          3               NaN&#xA;     1  17  2015-02          1                 2&#xA;     2  17  2015-03          1                 0&#xA;     3  18  2015-01          1               NaN&#xA;     4  18  2015-02          2                 1&#xA;     5  18  2015-03          1                 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Are there any ideas to get this column?&lt;/p&gt;&#xA;" OwnerUserId="14339" LastEditorUserId="98307" LastEditDate="2020-07-31T14:54:51.557" LastActivityDate="2020-08-03T13:41:42.427" Title="pandas count values for last 7 days from each date" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9023" PostTypeId="1" CreationDate="2015-11-25T14:28:46.053" Score="1" ViewCount="117" Body="&lt;p&gt;If you are asked to do text categorization using clustering. Which algorithm would you use and why?&lt;/p&gt;&#xA;" OwnerUserId="14341" LastEditorDisplayName="user10169" LastEditDate="2015-11-25T19:27:19.667" LastActivityDate="2015-11-26T03:03:59.650" Title="Which algorithm would you use for text categorisation using clustering?" Tags="|data-mining|clustering|" AnswerCount="3" CommentCount="1" ClosedDate="2015-11-27T13:52:50.097" ContentLicense="CC BY-SA 3.0" />
  <row Id="9024" PostTypeId="2" ParentId="8926" CreationDate="2015-11-25T15:00:16.763" Score="1" Body="&lt;p&gt;I do not believe classic SVM is the right tool in the situation simply because you have no negatives. I'd try to use one of the following: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Semi-supervised approach: if you data can be clustered well enough, then you can determine which of the clusters corresponds to the positive examples using your data and use this as your classifier&lt;/li&gt;&#xA;&lt;li&gt;Use 1-class SVM&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14009" LastActivityDate="2015-11-25T15:00:16.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9025" PostTypeId="1" AcceptedAnswerId="9107" CreationDate="2015-11-25T15:00:50.553" Score="1" ViewCount="346" Body="&lt;p&gt;I've been writing a java library that I want to use to build Bayesian Belief Networks. I have classes that I use to build a Directed Graph&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;public class Node{&#xA;    private String label;&#xA;    private List&amp;lt;Node&amp;gt; adjacencyList = new ArrayList&amp;lt;Node&amp;gt;();&#xA;    private Frequency&amp;lt;String&amp;gt; distribution = new Frequency&amp;lt;String&amp;gt;();&#xA;&#xA;    public String getLabel() {&#xA;        return label;&#xA;    }&#xA;&#xA;    public void setLabel(String label) {&#xA;        this.label = label;&#xA;    }&#xA;&#xA;    public List&amp;lt;Node&amp;gt; getAdjacencyList(){&#xA;        return adjacencyList;&#xA;    }&#xA;&#xA;    public void addNeighbour(Node neighbour){&#xA;        adjacencyList.add(neighbour);&#xA;    }&#xA;&#xA;    public void setDistribution(List&amp;lt;String&amp;gt; data){&#xA;&#xA;        for(String s:data){&#xA;            distribution.addValue(s);&#xA;        }&#xA;    }&#xA;&#xA;    public double getDistributionValue(String value){&#xA;        return distribution.getPct(value);&#xA;    }&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Graph&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;public class DirectedGraph {&#xA;&#xA;Map&amp;lt;String,Node&amp;gt; graph = new HashMap&amp;lt;String,Node&amp;gt;();&#xA;&#xA;public void addVertex(String label){&#xA;    Node vertex = new Node();&#xA;    vertex.setLabel(label);&#xA;    graph.put(label, vertex);&#xA;}&#xA;&#xA;public void addEdge(String here, String there){&#xA;    Node nHere = graph.get(here);&#xA;    Node nThere = graph.get(there);&#xA;    nThere.addNeighbour(nHere);&#xA;    graph.put(there,nThere);&#xA;}&#xA;&#xA;public List&amp;lt;Node&amp;gt; getNeighbors(String vertex){&#xA;    return graph.get(vertex).getAdjacencyList();&#xA;}&#xA;&#xA;public int degree(String vertex){&#xA;    return graph.get(vertex).getAdjacencyList().size();&#xA;}&#xA;&#xA;public boolean hasVertex(String vertex){&#xA;    return graph.containsKey(vertex);&#xA;}&#xA;&#xA;public boolean hasEdge(String here, String there){&#xA;    Set&amp;lt;Node&amp;gt; nThere = new HashSet&amp;lt;Node&amp;gt;(graph.get(there).getAdjacencyList());&#xA;    boolean thereConHere = nThere.contains(here);&#xA;    return (thereConHere);&#xA;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have a class that I use to keep track of the probability distribution of a data set&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;public class Frequency&amp;lt;T extends Comparable&amp;lt;T&amp;gt;&amp;gt; {&#xA;&#xA;private Multiset event = HashMultiset.create();&#xA;private Multimap event2 = LinkedListMultimap.create();&#xA;&#xA;public void addValue(T data){&#xA;    if(event2.containsKey(data) == false){&#xA;        event2.put(data,data);&#xA;    }&#xA;&#xA;    event.add(data);&#xA;}&#xA;&#xA;public void clear(){&#xA;&#xA;    this.event = null;&#xA;    this.event2 = null;&#xA;&#xA;    this.event = HashMultiset.create();&#xA;    this.event2 = LinkedListMultimap.create();&#xA;}&#xA;&#xA;public double getPct(T data){&#xA;    int numberOfIndElements = event.count(data);&#xA;    int totalNumOfElements = event.size();&#xA;    return (double) numberOfIndElements/totalNumOfElements;&#xA;}&#xA;&#xA;public int getNum(T data){&#xA;    int numberOfIndElements = event.count(data);&#xA;    return numberOfIndElements;&#xA;}&#xA;&#xA;public int getSumFreq(){&#xA;    return event.size();&#xA;}&#xA;&#xA;public int getUniqueCount(){&#xA;    return event.entrySet().size();&#xA;}&#xA;&#xA;public String[] getKeys(){&#xA;    Set&amp;lt;String&amp;gt; test = event2.keySet();&#xA;    Object[] keys = test.toArray();&#xA;    String[] keysAsStrings = new String[keys.length];&#xA;&#xA;    for(int i=0;i&amp;lt;keys.length;i++){&#xA;        keysAsStrings[i] = (String) keys[i];&#xA;    }&#xA;&#xA;    return keysAsStrings;&#xA;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;as well as another function that I can use to calculate conditional probabilities&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;public double conditionalProbability(List&amp;lt;String&amp;gt; interestedSet,&#xA;                                     List&amp;lt;String&amp;gt; reducingSet,&#xA;                                     String interestedClass,&#xA;                                     String reducingClass){&#xA;    List&amp;lt;Integer&amp;gt; conditionalData = new LinkedList&amp;lt;Integer&amp;gt;();&#xA;    double returnProb = 0;&#xA;    iFrequency.clear();&#xA;    rFrequency.clear();&#xA;&#xA;    this.setInterestedFrequency(interestedSet);&#xA;    this.setReducingFrequency(reducingSet);&#xA;&#xA;&#xA;    for(int i = 0;i&amp;lt;reducingSet.size();i++){&#xA;        if(reducingSet.get(i).equalsIgnoreCase(reducingClass)){&#xA;            if(interestedSet.get(i).equalsIgnoreCase(interestedClass)){&#xA;                conditionalData.add(i);&#xA;            }&#xA;        }&#xA;    }&#xA;&#xA;    int numerator = conditionalData.size();&#xA;    int denominator = this.rFrequency.getNum(reducingClass);&#xA;&#xA;    if(denominator !=0){&#xA;        returnProb = (double)numerator/denominator;&#xA;    }&#xA;&#xA;    iFrequency.clear();&#xA;    rFrequency.clear();&#xA;    return returnProb;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;However, I'm still not sure how to hook everything up in order to perform classification.&lt;/p&gt;&#xA;&lt;p&gt;I was reading over a paper entitled &lt;a href=&quot;http://arxiv.org/pdf/1301.6684.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Comparing Bayesian Network Classifiers &lt;/a&gt; to try and get an understanding.&lt;/p&gt;&#xA;&lt;p&gt;Let's say that I am trying to predict a person's sex based on the attributes height, weight and shoe size. My understanding is that I would have Sex as my parent/classification node and height, weight and shoe size would by my child nodes.&lt;/p&gt;&#xA;&lt;p&gt;This is what I'm confused about. The various classification nodes only keep track of the probability distribution of their respective attributes, but I'd need the conditional probabilities in order to perform classification.&lt;/p&gt;&#xA;&lt;p&gt;I have an older version of Naive Bayes that I wrote&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;public void naiveBayes(Data data,List&amp;lt;String&amp;gt; targetClass, BayesOption bayesOption,boolean headers){&#xA;    //intialize variables&#xA;    int numOfClasses = data.getNumOfKeys();//.getHeaders().size();&#xA;    String[] keyNames = data.getKeys();//  data.getHeaders().toArray();&#xA;    double conditionalProb = 1.0;&#xA;    double prob = 1.0;&#xA;    String[] rClass;&#xA;    String priorName;&#xA;&#xA;&#xA;    iFrequency.clear();&#xA;    rFrequency.clear();&#xA;&#xA;    if(bayesOption.compareTo(BayesOption.TRAIN) == 0){&#xA;        this.setInterestedFrequency(targetClass);&#xA;        this.targetClassKeys = Util.convertToStringArray(iFrequency.getKeys());&#xA;&#xA;        for(int i=0;i&amp;lt;this.targetClassKeys.length;i++){&#xA;            priors.put(this.targetClassKeys[i],iFrequency.getPct(this.targetClassKeys[i]));&#xA;        }&#xA;&#xA;    }&#xA;&#xA;&#xA;    //for each classification in the target class&#xA;    for(int i=0;i&amp;lt;this.targetClassKeys.length;i++){&#xA;&#xA;        //get all of the different classes for that variable&#xA;        for(int j=0;j&amp;lt;numOfClasses;j++){&#xA;&#xA;            String reducingKey = Util.convertToString(keyNames[j]);&#xA;            List&amp;lt;String&amp;gt; reducingClass = data.dataColumn(reducingKey,DataOption.GET,true);// new ArrayList(data.getData().get(reducingKey));&#xA;            this.setReducingFrequency(reducingClass);&#xA;            Object[] reducingClassKeys = rFrequency.getKeys();&#xA;            rClass = Util.convertToStringArray(reducingClassKeys);&#xA;&#xA;&#xA;            for(int k=0;k&amp;lt;reducingClassKeys.length;k++){&#xA;&#xA;                if(bayesOption.compareTo(BayesOption.TRAIN) == 0){&#xA;                    conditionalProb = conditionalProbability(targetClass, reducingClass, this.targetClassKeys[i], rClass[k]);&#xA;                    priorName = this.targetClassKeys[i]+&amp;quot;|&amp;quot;+rClass[k];&#xA;                    priors.put(priorName,conditionalProb);&#xA;                }&#xA;&#xA;                if(bayesOption.compareTo(BayesOption.PREDICT) == 0){&#xA;                    priorName = this.targetClassKeys[i]+&amp;quot;|&amp;quot;+rClass[k];&#xA;                    prob = prob * priors.get(priorName);&#xA;&#xA;                }&#xA;            }&#xA;            rFrequency.clear();&#xA;&#xA;        }&#xA;&#xA;        if(BayesOption.PREDICT.compareTo(bayesOption) == 0){&#xA;            prob = prob * priors.get(this.targetClassKeys[i]);&#xA;            Pair&amp;lt;String,Double&amp;gt; pred = new Pair&amp;lt;String, Double&amp;gt;(this.targetClassKeys[i],prob);&#xA;            this.predictions.add(pred);&#xA;        }&#xA;&#xA;    }&#xA;&#xA;    this.iFrequency.clear();&#xA;    this.rFrequency.clear();&#xA;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So I generally understand how the math works, but I'm not quite sure how I'm supposed to get things to work with this specific architecture.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;How do I calculate the conditional probabilities?&lt;/p&gt;&#xA;&lt;p&gt;Can somebody explain this discrepancy to me please?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14340" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-11-28T19:43:43.387" Title="How do I perform Naive Bayes Classification with a Bayesian Belief Network?" Tags="|machine-learning|data-mining|classification|statistics|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9026" PostTypeId="2" ParentId="9023" CreationDate="2015-11-25T15:04:04.387" Score="0" Body="&lt;p&gt;Text categorization using clustering can be done in a lot of ways. Some of which are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using the &lt;a href=&quot;http://www.codeproject.com/Articles/439890/Text-Documents-Clustering-using-K-Means-Algorithm&quot; rel=&quot;nofollow&quot;&gt;K-means algorithm&lt;/a&gt;  (&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/text/document_clustering.html&quot; rel=&quot;nofollow&quot;&gt;Another link&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=985318&quot; rel=&quot;nofollow&quot;&gt;Feature Projections&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In fact, most(or almost all) the &lt;a href=&quot;https://en.wikipedia.org/wiki/Category:Data_clustering_algorithms&quot; rel=&quot;nofollow&quot;&gt;clustering algorithms&lt;/a&gt; can be applied for classifying text through clustering.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-26T02:21:01.703" LastActivityDate="2015-11-26T02:21:01.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9027" PostTypeId="1" AcceptedAnswerId="9032" CreationDate="2015-11-25T15:14:36.440" Score="0" ViewCount="146" Body="&lt;p&gt;In a typical supervised learning setting with a few positive and a few negative examples, it is clear that unlabeled data carries some information that can benefit learning and that is not captured in the labeled data. For example one can estimate mean values, bounds and some other geometrical characteristics of the data-set with much higher precision if you do not discard the (massive) unlabeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the most common ML algorithms from Neural Networks to SVM do not take advantage of this information (at least in their standard, most common form). My question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any theoretical framework where unlabeled data is treated in the supervised setting?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I can think of semi-supervised ways to approach this (first cluster and then label the clusters). Are there any other?&lt;/p&gt;&#xA;" OwnerUserId="14009" LastActivityDate="2015-11-26T18:47:10.297" Title="Theoretical treatment of unlabeled samples" Tags="|machine-learning|data-mining|supervised-learning|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9028" PostTypeId="2" ParentId="9027" CreationDate="2015-11-25T15:36:27.167" Score="0" Body="&lt;p&gt;You already dropped the key word &quot;semi-supervised&quot; in your question. Indeed, semi-supervised learning is the answer to your question. Search for this term with your favourite search engine or library catalog to find out about algorithms for semi-supervised learning.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-11-25T15:36:27.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9029" PostTypeId="2" ParentId="9023" CreationDate="2015-11-25T16:22:36.677" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf vectors&lt;/a&gt; are an easy start, but it can be hard to cluster very high dimensional data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might try &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic modeling&lt;/a&gt; (LDA, LSI for example) to reduce the dimensionality of your features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A newer approach is &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://cs.stanford.edu/~quocle/paragraph_vector.pdf&amp;amp;ved=0ahUKEwiAz52Ija3JAhXLKCYKHRdeAcIQFgghMAA&amp;amp;usg=AFQjCNE-2SsR07iQm78dQqVV-6_Y3ERurA&amp;amp;sig2=O64Artz1ekwZ2KqjFCMabQ&quot; rel=&quot;nofollow&quot;&gt;paragraph vectors&lt;/a&gt;, which learn a distributed representation of arbitrary-length text. &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is an implementation in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning a reasonable, lower dimensional representation of the text can help with the issues that arise trying to cluster high dimensional data.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-26T03:03:59.650" LastActivityDate="2015-11-26T03:03:59.650" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9030" PostTypeId="2" ParentId="9023" CreationDate="2015-11-25T16:37:48.423" Score="1" Body="&lt;p&gt;It will depend on the purpose and the text.  Many options this is what I have used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow&quot;&gt;k-means clustering&lt;/a&gt; with &lt;a href=&quot;http://tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;TF-IDF&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did was limit the cluster and any document vector to the top 1000 terms sorted by weight.  This not only results in faster processing but you get some multi modal clusters and will have thousands and thousands of terms.  What happens is those really long vectors dilute the similarity when compared to a even a long single document.  I think you also get faster convergence.  If you want to keep documents out of multi modal clusters then don't truncate the cluster vector.&lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-11-25T18:14:52.830" LastActivityDate="2015-11-25T18:14:52.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9031" PostTypeId="2" ParentId="9022" CreationDate="2015-11-25T20:39:36.057" Score="6" Body="&lt;p&gt;The main obstacle is figuring out whether a date is within the last 7 days of the month. I'd recommend something hacky like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from datetime import datetime, date, timedelta&#xA;def last7(datestr):&#xA;    orig = datetime.strptime(datestr,'%Y-%m-%d')&#xA;    plus7 = orig+timedelta(7)&#xA;    return plus7.month != orig.month&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once you have that, it's relatively simple to adapt your previous code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df3 = df1[df1['is_buy'] == 1 &amp;amp;&amp;amp; last7(df1['date'])].groupby(['id', 'month']).agg({'is_buy': np.sum})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we just join together &lt;code&gt;df2&lt;/code&gt; and &lt;code&gt;df3&lt;/code&gt; and we're done.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2015-11-25T20:39:36.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9032" PostTypeId="2" ParentId="9027" CreationDate="2015-11-25T21:16:44.900" Score="0" Body="&lt;p&gt;In a neural network model, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow&quot;&gt;autoencoders&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea of an autoencoder is to learn a hidden layer of features by creating a network that simply copies the input vector at the output. So the training features and training &quot;labels&quot; are initially identical, no supervised labels are required. This can work using a classic &lt;em&gt;triangular&lt;/em&gt; network architecture with progressively smaller layers that capture a compressed and hopefully useful set of derived features. The network's hidden layers learn representations based on the larger unsupervised data set. These layers can then be used to initialise a regular supervised learning network to be trained using the actual labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar idea is pre-training layers using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;Restricted Boltzmann Machine&lt;/a&gt;, which can be used in a very similar way, although based on different principles. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-11-26T18:47:10.297" LastActivityDate="2015-11-26T18:47:10.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9034" PostTypeId="2" ParentId="8064" CreationDate="2015-11-25T22:03:57.350" Score="13" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#cohen-s-kappa&quot; rel=&quot;noreferrer&quot;&gt;Cohen's kappa&lt;/a&gt; was introduced in scikit-learn 0.17.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can wrap it in &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#scoring&quot; rel=&quot;noreferrer&quot;&gt;make_scorer&lt;/a&gt; for use in GridSearchCV.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import cohen_kappa_score, make_scorer&#xA;from sklearn.grid_search import GridSearchCV&#xA;from sklearn.svm import LinearSVC&#xA;&#xA;kappa_scorer = make_scorer(cohen_kappa_score)&#xA;grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14320" LastActivityDate="2015-11-25T22:03:57.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9036" PostTypeId="1" CreationDate="2015-11-26T00:10:57.433" Score="3" ViewCount="422" Body="&lt;p&gt;Most of us want to build a recommendation engine as accurate as possible, however, an experienced chief data scientist believes a &lt;em&gt;practical&lt;/em&gt; machine learning algorithm should randomly shuffle the results, therefore non-optimal results. This is known as &quot;Results Dithering&quot;. Slide 15 at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cikm2013.org/slides/ted.pptx&quot; rel=&quot;nofollow&quot;&gt;http://cikm2013.org/slides/ted.pptx&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I understand &lt;strong&gt;how&lt;/strong&gt; to do it (adding a small fixed Gaussian noise), but &lt;strong&gt;why&lt;/strong&gt; would we do it to make the performance worse.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2015-11-26T11:40:51.800" LastActivityDate="2017-07-30T21:10:31.250" Title="Result Dithering? Why randomly shuffle results?" Tags="|machine-learning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9037" PostTypeId="1" CreationDate="2015-11-26T03:45:19.333" Score="1" ViewCount="2160" Body="&lt;p&gt;When apply clustering algorithm with the multi-class data set and class numbers are not equal to the result cluster numbers(For example , When we use K-Means algorithm by setting K = 3 apply with &quot;yeast&quot; data set that has 14 classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the proper way to evaluate the result in this situation ? &lt;/p&gt;&#xA;" OwnerUserId="14357" LastActivityDate="2015-12-12T13:29:13.163" Title="How to evaluate the clustering result when cluster numbers are not equal to data set class" Tags="|clustering|multiclass-classification|model-evaluations|unsupervised-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9038" PostTypeId="1" AcceptedAnswerId="9157" CreationDate="2015-11-26T04:28:17.827" Score="30" ViewCount="3490" Body="&lt;p&gt;There are many techniques for visualizing high dimension datasets, such as T-SNE, isomap, PCA, supervised PCA, etc. And we go through the motions of projecting the data down to a 2D or 3D space, so we have a &quot;pretty pictures&quot;. Some of these embedding (manifold learning) methods are described &lt;a href=&quot;http://scikit-learn.org/stable/modules/manifold.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H3FBv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H3FBv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But is this &quot;pretty picture&quot; actually meaningful? What possible insights can someone grab by trying to visualize this embedded space?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask because the projection down to this embedded space is usually meaningless. For example, if you project your data down to principal components generated by PCA, those principal components (eiganvectors) don't correspond to features in the dataset; they're their own feature space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, t-SNE projects your data down to a space, where items are near each other if they minimize some KL divergence. This isn't the original feature space anymore. (Correct me if I'm wrong, but I don't even think there is a large effort by the ML community to use t-SNE to aid classification; that's a different problem than data visualization though.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just very largely confused why people make such a big deal about some of these visualizations. &lt;/p&gt;&#xA;" OwnerUserId="8774" LastActivityDate="2020-08-20T18:49:05.380" Title="Purpose of visualizing high dimensional data?" Tags="|machine-learning|dimensionality-reduction|visualization|" AnswerCount="8" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9039" PostTypeId="2" ParentId="9037" CreationDate="2015-11-26T06:28:37.397" Score="1" Body="&lt;p&gt;The first step would be to train a couple of different algorithms/classifiers to get a rough idea of how good your results can be without changing your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, once you do your kmeans, you can use that information to possibly improve your data and then run it through your classifiers again to see if your results improve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I say &quot;use that information to possibly improve your data&quot;, here's what I mean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say your data has 5 features(columns)...see following totally made up data...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2,5,6,3,2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4,5,2,3,7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5,8,2,3,5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is this data that you ran through your kmeans...using 3 clusters, each row of your data will 'belong' to a cluster...so add it to your data and your data becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2,5,6,3,2,1,0,0  belongs to cluster 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4,5,2,3,7,0,1,0  belongs to cluster 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5,8,2,3,5,0,0,1  belongs to cluster 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could just add one column and make it 1,2,3 but some algorithms prefer the 0/1 method (one-hot encoding, dummy variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See, now you have a bit more information in your data than you started with. Having the before and after comparison models will let you know if it is actually useful information!  &lt;/p&gt;&#xA;" OwnerUserId="3457" LastActivityDate="2015-11-26T06:28:37.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9040" PostTypeId="2" ParentId="8064" CreationDate="2015-11-26T07:01:38.963" Score="2" Body="&lt;p&gt;In addition to the link in the existing answer, there is also a Scikit-Learn laboratory, where methods and algorithms are being experimented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you are okay with working with bleeding edge code, &lt;a href=&quot;http://skll.readthedocs.org/en/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;this library&lt;/a&gt; would be a nice reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Cohen's Kappa is &lt;a href=&quot;http://skll.readthedocs.org/en/latest/api/metrics.html#skll.metrics.kappa&quot; rel=&quot;nofollow&quot;&gt;also one of the metrics in the library&lt;/a&gt;, which takes in &lt;code&gt;true labels&lt;/code&gt;, &lt;code&gt;predicted labels&lt;/code&gt;, &lt;code&gt;weights&lt;/code&gt; and &lt;code&gt;allowing one off?&lt;/code&gt; as the input parameters. Obviously, the metric would range from &lt;code&gt;[-1, 1]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also have a look at the &lt;a href=&quot;http://skll.readthedocs.org/en/latest/_modules/skll/metrics.html&quot; rel=&quot;nofollow&quot;&gt;implementation code&lt;/a&gt;, just in case you would like to contribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Cohen's Kappa is also &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py#L260-L308&quot; rel=&quot;nofollow&quot;&gt;implemented in Scikit-Learn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, &lt;a href=&quot;http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/&quot; rel=&quot;nofollow&quot;&gt;there&lt;/a&gt; &lt;a href=&quot;http://blog.dato.com/how-to-evaluate-machine-learning-models-part-2a-classification-metrics&quot; rel=&quot;nofollow&quot;&gt;are&lt;/a&gt; &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html&quot; rel=&quot;nofollow&quot;&gt;alternatives&lt;/a&gt; to the Cohen Kappa metric.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-26T07:01:38.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9041" PostTypeId="2" ParentId="9037" CreationDate="2015-11-26T07:07:15.133" Score="1" Body="&lt;p&gt;Even if the number of clusters were equal, kt rarely holds that there is a one on one match of clusters to classes. For example, one class may be split into two clusters, and two classes may be indiscernible and thus in the same cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact &lt;strong&gt;clustering is most interesting if it &lt;em&gt;differs&lt;/em&gt; from your classes&lt;/strong&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compare clusterings (and classes) a number of metrics have been developed. Most are based on &lt;strong&gt;pair counting&lt;/strong&gt;, and the Rand index is one of the most popular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of objects, you look at &lt;strong&gt;pairs of objects&lt;/strong&gt;: are they in the same cluster and same class? Then the pair is &quot;true positive&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Wikipedia for more details.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-11-26T07:07:15.133" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9042" PostTypeId="5" CreationDate="2015-11-26T09:53:45.897" Score="0" Body="&lt;p&gt;A &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;nofollow&quot;&gt;confusion matrix&lt;/a&gt; is a special contingency table used to evaluate the predictive accuracy of a classifier.  Predicted classes are listed in rows (or columns) &amp;amp; actual classes in columns (or rows), with counts of the cases in each combination listed in each cell.  All cases represented along the main diagonal are accurately classified, while the off-diagonal elements are misclassified.  Inspection of the confusion matrix can identify which classes tend to be 'confused' for each other.  The confusion matrix also allows the calculation of model performance metrics such as sensitivity and specificity, precision and recall, positive and negative predictive value, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example confusion matrix from a model of Fisher's iris data with good accuracy.  The model occasionally confuses &lt;code&gt;versicolor&lt;/code&gt; and &lt;code&gt;virginica&lt;/code&gt;, but never misclassified either as &lt;code&gt;setosa&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                      Actual:&#xA;             setosa versicolor virginica &#xA;Predicted:&#xA;  setosa         50          0         0  &#xA;  versicolor      0         47         3  &#xA;  virginica       0          4        46 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:50:32.723" LastActivityDate="2015-11-27T13:50:32.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9043" PostTypeId="4" CreationDate="2015-11-26T09:53:45.897" Score="0" Body="A confusion matrix is a special contingency table used to evaluate the predictive accuracy of a classifier. Predicted classes are listed in rows and actual classes in columns, with counts of respective cases in each cell." OwnerUserId="11097" LastEditorUserId="21" LastEditDate="2015-11-27T13:51:25.560" LastActivityDate="2015-11-27T13:51:25.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9044" PostTypeId="5" CreationDate="2015-11-26T09:54:27.533" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T09:54:27.533" LastActivityDate="2015-11-26T09:54:27.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9045" PostTypeId="4" CreationDate="2015-11-26T09:54:27.533" Score="0" Body="Survival analysis is concerned with modelling the time before subjects change state, typically time until death or failure. One key feature of such data is that they can be censored, that is, some subjects will not have changed state before the study ends. " OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:50:55.330" LastActivityDate="2015-11-27T13:50:55.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9046" PostTypeId="5" CreationDate="2015-11-26T09:55:44.800" Score="0" Body="&lt;p&gt;Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal).&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:51:00.557" LastActivityDate="2015-11-27T13:51:00.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9047" PostTypeId="4" CreationDate="2015-11-26T09:55:44.800" Score="0" Body="Supervised learning is a type of machine learning algorithm that learns a mapping function y = f(x) between input variables (x) and output variables (y). The two most common supervised learning tasks are classification and regression.&#xD;&#xA;&#xD;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-08-01T02:32:33.930" LastActivityDate="2017-08-01T02:32:33.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9048" PostTypeId="5" CreationDate="2015-11-26T09:56:54.963" Score="0" Body="&lt;p&gt;Ranking means ordering data from highest to lowest or &lt;em&gt;vice versa&lt;/em&gt;.  Statistical &quot;ranks&quot; are normally given the values 1, 2, 3, ..., up to the sample size, assigned (respectively) to values or attributes which have the highest (or lowest) rank, the second highest (/lowest) rank, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A standard notation is to bracket subscripts when denoting ranks: that is, for a batch of numbers $(x_1, x_2, \ldots, x_n)$ (written in any order), the sorted values are written $(x_{[1]}, x_{[2]}, \ldots, x_{[n]})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the original values being ranked can sometimes be tied (have a &lt;em&gt;weak order&lt;/em&gt;), tied ranks can occur where more than one rank is to be assigned to indistinguishable values. This issue is resolved in various ways, but commonly in statistics the average rank allocated to the tied (indistinguishable) observations is assigned to each of them.  Allocating &lt;em&gt;average ranks&lt;/em&gt; is sometimes called &lt;em&gt;fractional ranking&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://en.wikipedia.org/wiki/Ranking#Ranking_in_statistics&quot; rel=&quot;nofollow&quot;&gt;Wikipedia: Ranking in statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:52:24.987" LastActivityDate="2015-11-27T13:52:24.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9049" PostTypeId="4" CreationDate="2015-11-26T09:56:54.963" Score="0" Body="Ranking is ordering data from highest to lowest or &#xD;&#xA;*vice versa.* For questions about *constructing* scores to use in ranking, please use the &quot;valuation&quot; tag, too." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:51:05.820" LastActivityDate="2015-11-27T13:51:05.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9050" PostTypeId="5" CreationDate="2015-11-26T09:58:43.803" Score="0" Body="&lt;p&gt;SAS is a proprietary cross-platform general-purpose statistical software package (&lt;a href=&quot;http://www.sas.com/index.html&quot; rel=&quot;nofollow&quot;&gt;Official Website&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Various Resources for SAS include&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://support.sas.com/documentation/&quot; rel=&quot;nofollow&quot;&gt;online documentation&lt;/a&gt; includes the SAS User's Guide which contains quite detailed information on SAS procedures, including syntax, theoretical details, and examples.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://support.sas.com/events/sasglobalforum/previous/online.html&quot; rel=&quot;nofollow&quot;&gt;proceedings of SAS Global Forum&lt;/a&gt;. The proceedings were formerly referred to by the acronym of SUGI, which stood for SAS User Group International, but in 2007 the group was renamed to SAS Global Forum. The proceedings include short papers related to programming aspects of SAS as well as statistical analysis.&lt;/li&gt;&#xA;&lt;li&gt;The UCLA Academic Technology Services provides &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/default.htm&quot; rel=&quot;nofollow&quot;&gt;various tutorials&lt;/a&gt; on conducting statistical analysis and programming in SAS&lt;/li&gt;&#xA;&lt;li&gt;Chris Hemedinger on his blog, &lt;a href=&quot;http://blogs.sas.com/content/sasdummy/&quot; rel=&quot;nofollow&quot;&gt;The SAS Dummy&lt;/a&gt;, has &lt;a href=&quot;http://blogs.sas.com/content/sasdummy/2011/09/07/google-reader-bundles-for-sas-related-blogs/&quot; rel=&quot;nofollow&quot;&gt;posted a listing of various SAS blogs&lt;/a&gt; he follows (including his own) in two Google reader bundles. Even if you don't use Google reader it is possible to follow the trail to the various webpages.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.lexjansen.com/&quot; rel=&quot;nofollow&quot;&gt;Lex Jansen&lt;/a&gt; has indexed all of SUGI/SGF as well as many SAS regional.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other Forums in which you can ask SAS specific programming questions (in descending order of activity)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;http://listserv.uga.edu/archives/sas-l.html&quot; rel=&quot;nofollow&quot;&gt;SAS-L&lt;/a&gt; listserve&lt;/li&gt;&#xA;&lt;li&gt;Stackoverflow has (as of 12/14/2011) over 400 &lt;a href=&quot;http://stackoverflow.com/questions/tagged/sas&quot;&gt;questions tagged with sas&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.runsubmit.com/&quot; rel=&quot;nofollow&quot;&gt;RunSubmit&lt;/a&gt; (a Stack-alike site specifically for SAS)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:50:28.973" LastActivityDate="2015-11-27T13:50:28.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9051" PostTypeId="4" CreationDate="2015-11-26T09:58:43.803" Score="0" Body="SAS is a proprietary cross-platform general-purpose statistical software package." OwnerUserId="11097" LastEditorUserId="25180" LastEditDate="2019-12-06T19:10:51.663" LastActivityDate="2019-12-06T19:10:51.663" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9052" PostTypeId="5" CreationDate="2015-11-26T09:59:25.590" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T09:59:25.590" LastActivityDate="2015-11-26T09:59:25.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9053" PostTypeId="4" CreationDate="2015-11-26T09:59:25.590" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T09:59:25.590" LastActivityDate="2015-11-26T09:59:25.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9054" PostTypeId="5" CreationDate="2015-11-26T10:00:06.537" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T10:00:06.537" LastActivityDate="2015-11-26T10:00:06.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9055" PostTypeId="4" CreationDate="2015-11-26T10:00:06.537" Score="0" Body="&quot;References&quot; is our generic tag for questions seeking information about books, papers, presentations, videos of lectures, on-line tutorials, etc., regarding any subject matter that is on-topic for Data Science." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:50:24.050" LastActivityDate="2015-11-27T13:50:24.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9056" PostTypeId="5" CreationDate="2015-11-26T10:00:43.780" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T10:00:43.780" LastActivityDate="2015-11-26T10:00:43.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9057" PostTypeId="4" CreationDate="2015-11-26T10:00:43.780" Score="0" Body="Recommendations for software or libraries." OwnerUserId="843" LastEditorUserId="843" LastEditDate="2016-01-08T19:26:33.237" LastActivityDate="2016-01-08T19:26:33.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9058" PostTypeId="5" CreationDate="2015-11-26T10:01:14.917" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-26T10:01:14.917" LastActivityDate="2015-11-26T10:01:14.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9059" PostTypeId="4" CreationDate="2015-11-26T10:01:14.917" Score="0" Body="Indicates questions asking about the use and meaning of specific technical words/concepts in statistics." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-27T13:52:18.080" LastActivityDate="2015-11-27T13:52:18.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9060" PostTypeId="1" AcceptedAnswerId="9062" CreationDate="2015-11-25T10:30:37.023" Score="1" ViewCount="170" Body="&lt;p&gt;I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have found many tools with which I can analyse the data where each tool has its own advantages and disadvantages. Big data being really new to me, I find it very difficult to choose a platform to start with. There should be other people here who should have dealt with similar situations.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What criteria should I mainly take into account before selecting a&#xA;tool for analysing big data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some of the Criteria that I have taken into account : Visualization, Interaction, Security, Data Access and integration, Speed of response, Integrated Data Mining, Pattern Matching, Ease of use etc. As you can see the list that I have made for the criteria comes from the extensive reading of different articles on the topic. But I can't narrow down the list nor find the individual contribution of these criteria in the various tools available for analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me also list some of the tools that I found after googling : Knime, Statistica 2, Rapidminer, Orange, WEKA, KEEL, R and RATTLE. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On what basis could I choose a tool from a list of tools that perform similar tasks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE based on Comment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aim : To develop a software that analyses the data coming from the wind mills and generate reports. The software should be able to predict when a wind mill can fail based on the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The project is still in the phase of gathering User Requirements. Maybe i am so early to come into conclusions about what tool should be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone else suggested that I should be finalise the requirements and then think about a tool that can help me get things done. So is it possible that I find what and how things should be analysed before finding a tool? And is it also possible that I find an algorithm for predictive analysis without knowing what would be the results of the tool after analysis. &lt;/p&gt;&#xA;" OwnerUserId="14332" OwnerDisplayName="Vini Vasundharan" LastEditorUserId="11097" LastEditDate="2015-11-26T10:41:44.153" LastActivityDate="2015-11-26T10:41:44.153" Title="Dealing with big data" Tags="|bigdata|software-recommendation|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9061" PostTypeId="2" ParentId="9060" CreationDate="2015-11-25T14:02:38.900" Score="3" Body="&lt;p&gt;Syncfusion provides a &lt;a href=&quot;http://www.syncfusion.com/products/big-data&quot; rel=&quot;nofollow&quot;&gt;Big data platform&lt;/a&gt; which is an easy to use Hadoop distribution for Windows. It can help you get started quickly. Syncfusion also provides a &lt;a href=&quot;http://www.syncfusion.com/products/data-science/predictive-analytics&quot; rel=&quot;nofollow&quot;&gt;PMML processing library&lt;/a&gt; using which you can execute predictive analytic models. There is a &lt;a href=&quot;http://www.syncfusion.com/products/dashboard&quot; rel=&quot;nofollow&quot;&gt;Dashboard platform&lt;/a&gt; also available that can help visualize the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the above are available for free through the &lt;a href=&quot;http://www.syncfusion.com/products/communitylicense&quot; rel=&quot;nofollow&quot;&gt;community license&lt;/a&gt; if you qualify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I work for Syncfusion.&lt;/p&gt;&#xA;" OwnerUserId="98096" OwnerDisplayName="Davis Jebaraj" LastActivityDate="2015-11-25T14:02:38.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9062" PostTypeId="2" ParentId="9060" CreationDate="2015-11-26T10:40:40.813" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What criteria should I mainly take into account before selecting a&#xA;  tool for analysing big data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are a lot of criterion which are to be taken into account when the tool selection is concerned. The can be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Structure of the data. (The data model Ex: Hierarchical, tabular, etc)&lt;/li&gt;&#xA;&lt;li&gt;Type of data and what is the problem statement.   (time series, or classification, etc)&lt;/li&gt;&#xA;&lt;li&gt;Speed&lt;/li&gt;&#xA;&lt;li&gt;Security&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Aim : To develop a software that analyses the data coming from the&#xA;  wind mills and generate reports. The software should be able to&#xA;  predict when a wind mill can fail based on the analysis.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Almost all the existing analytics tools like Python, Julia, R, etc can do this.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And is it also possible that I find an algorithm for predictive&#xA;  analysis without knowing what would be the results of the tool after&#xA;  analysis.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes.&lt;/strong&gt;  The predictive algorithm or technique can be inferred by looking at the data and the contents of the data. &lt;strong&gt;It is not dependant on the tool.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some points which I would like to include which I believe would be useful to you:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Select the database depending on your data, and it's type.  According to your data, a &lt;a href=&quot;https://en.wikipedia.org/wiki/NoSQL&quot; rel=&quot;nofollow&quot;&gt;NoSQL database&lt;/a&gt; would be more relevant and suitable.&lt;/li&gt;&#xA;&lt;li&gt;Select the algorithms and techniques only after you have a clear knowledge about the problem statement and takeaways and also after clearly looking at the data for an &lt;a href=&quot;https://en.wikipedia.org/wiki/Exploratory_data_analysis&quot; rel=&quot;nofollow&quot;&gt;exploratory analysis&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If you want more flexibility, then use a tool/programming language like Python, R and Julia. Else, you can use a tool like Knime, Orange (it has a &lt;a href=&quot;https://pypi.python.org/pypi/Orange/2.7&quot; rel=&quot;nofollow&quot;&gt;Python library&lt;/a&gt; too.), RapidMiner, etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-26T10:40:40.813" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9063" PostTypeId="1" CreationDate="2015-11-26T13:07:02.697" Score="4" ViewCount="3914" Body="&lt;p&gt;I have thousands of CV / resumes with me. We want to build a parser which can extract company names from resume. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far we have tried&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Maintained a list of common words present in companies (Eg. Org, Ltd, Limited, Technologies etc.) and use them to identify probable companies. But this list is limited and many times many companies don't get extracted.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using HTML of CV we have tried to give more score to probable companies which have a certain feature (like Bold, Italics) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Since CV is not only text and we always have some structural information along with it. There should be better ways to extract information. Maybe training some model which could predict companies mentioned in the resume. We are open to any better approaches/suggestions we could incorporate into our system for better accuracy. The precision so far is really bad (less than 45%). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have already done the segmentation of work experience in CV. So we are able to extract the segment containing work experience with very high precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also have a comprehensive list of companies (millions). Although it contains duplicates and needs significant amount of cleaning. But yes we have a lot of data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other approaches we are trying - &#xA;We try to predict the important phrases inside text using N-Grams and then mark them as probable companies. Then we do a search on companies corpus with us to find any match. How useful is this technique ? Any better approaches ?&lt;/p&gt;&#xA;" OwnerUserId="14368" LastEditorUserId="14368" LastEditDate="2015-11-27T07:30:00.110" LastActivityDate="2018-08-19T19:59:32.867" Title="How to extract important phrases (which may contain company name) from resume?" Tags="|machine-learning|data-mining|nlp|parsing|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9064" PostTypeId="2" ParentId="9063" CreationDate="2015-11-26T13:32:00.800" Score="0" Body="&lt;p&gt;Have you tried the XML package? In a similar question here, in SE, the most upvoted answer suggested using some packages for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here: &lt;a href=&quot;https://stackoverflow.com/questions/3195522/is-there-a-simple-way-in-r-to-extract-only-the-text-elements-of-an-html-page&quot;&gt;https://stackoverflow.com/questions/3195522/is-there-a-simple-way-in-r-to-extract-only-the-text-elements-of-an-html-page&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you find further instructions: &lt;a href=&quot;https://stackoverflow.com/questions/1844829/how-can-i-read-and-parse-the-contents-of-a-webpage-in-r&quot;&gt;https://stackoverflow.com/questions/1844829/how-can-i-read-and-parse-the-contents-of-a-webpage-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14370" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-04T00:28:30.063" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9065" PostTypeId="2" ParentId="9063" CreationDate="2015-11-26T15:34:36.003" Score="3" Body="&lt;p&gt;Sounds like you want &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow&quot;&gt;named entity recognition&lt;/a&gt;.  There are a variety of approaches to NER, and plenty of implementations, like the &lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow&quot;&gt;Stanford NER package&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you find named entities, determining what the named entity refers to is called &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Entity_linking&quot; rel=&quot;nofollow&quot;&gt;concept normalization&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-11-26T15:34:36.003" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9066" PostTypeId="2" ParentId="6760" CreationDate="2015-11-26T16:21:28.057" Score="2" Body="&lt;p&gt;I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics.  If no words are strongly associated with a given topic, it may not be a useful topic.  If you find many of these, you can re-run the experiment with fewer topics.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead there are words strongly associated with these topics, you just don't see the cohesive meaning behind the words associated with the topic, then these are not exactly &quot;trash&quot; topics.  They are still capturing the latent structure of the data, it just isn't something easily deemed a &quot;topic&quot; in the traditional sense of the word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing to remember is that the hyperparameters also control the shape of your resulting distributions.  &lt;a href=&quot;https://stats.stackexchange.com/a/37444/95325&quot;&gt;This answer&lt;/a&gt; does a great job explaining alpha and beta.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-26T16:21:28.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9067" PostTypeId="2" ParentId="9037" CreationDate="2015-11-26T16:40:11.530" Score="2" Body="&lt;p&gt;&lt;strong&gt;If no pair of your &lt;em&gt;classes&lt;/em&gt; are overlapping and they do classify each and every element of the data set, then they constitute a &lt;a href=&quot;https://en.wikipedia.org/wiki/Partition_of_a_set&quot; rel=&quot;nofollow&quot;&gt;partition&lt;/a&gt; of that set. So does your clustering&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In such a case&lt;/strong&gt;, I would suggest to use a distance metric to compare two partitions, instead of any bona fide index, even if it's popular -unless that index happens to be a metric as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One such distances for comparing partitions is the $VI$ index, rediscovered by Maria Meila$\phantom{,}^\dagger$ a few years back, but its origin dating back to the 80's in a physics paper by Zurek$\phantom{,}^{\ddagger}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to calculate distance between two partitions P and Q&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Determine $P\wedge Q, $the &quot;intersection&quot; (I always mix up the words join and meet here) of P and Q. This is simply the partition formed by taking the intersection of each cluster of P against each cluster of Q.&lt;/li&gt;&#xA;&lt;li&gt;For all three partitions $P$, $Q$, $P\wedge Q$ calculate their entropy $H$ defined as follows: $$H(P)=-\frac{1}{N}\sum_{i=1}^Np_i\log p_i\;,\;p_i\equiv\frac{n_i}{N}$$ where $N$ is the total number of elements you clustered/classified and $n_i$ is the number of elements within cluster/class $i$ of a given partition $P$. &lt;/li&gt;&#xA;&lt;li&gt;The distance $d(P,Q)$ between partitions P and Q is then given by $$d(P,Q)=2H(P\wedge Q)-H(P)-H(Q)$$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The reason for using a real metric is simple: a metric satisfy the triangular inequality and our intuitions of &quot;close&quot; and &quot;similar&quot; heavily relies on that. Any ad-hoc measure of dis-similarity (distance) that doesn't satisfy the triangular inequality will eventually give you results like this: A is &quot;$d$ away&quot; from B (d being concrete number), with $d$ smaller than a given bound, say $D$. Then you find A close to C by an amount $d'&amp;lt;D$. If this distance were not a metric, it could then happen that the distance $d''$ between B and C is say $2D$. For a sound metric distance, it should be &#xA;$d''\leq d+d'&amp;lt;2D$. In other words A would be &quot;similar&quot; to B and C, but the last two would be &quot;different&quot;/&quot;not similar&quot; from each other. This is counterintuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside is that the VI distance may be computationally more expensive than calculating say a Rand index. The upside is that it's mathematically sound and &lt;em&gt;makes sense&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use R to calculate the VI distance. You can also use Partanalyzer a program I wrote some time ago (&lt;a href=&quot;https://github.com/MASantos/Partanalyzer&quot; rel=&quot;nofollow&quot;&gt;https://github.com/MASantos/Partanalyzer&lt;/a&gt;). It needs a rewrite to make it more efficient, but it's open source, so you can modify it at will.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\dagger$: Meila, M. Journal of Multivariate Analysis, 98, 873 (2007). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\ddagger$: W.H. Zurek, Nature, vol.341, p.119 (1989).&lt;/p&gt;&#xA;" OwnerUserId="12603" LastEditorUserId="12603" LastEditDate="2015-11-26T22:44:04.030" LastActivityDate="2015-11-26T22:44:04.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9068" PostTypeId="2" ParentId="869" CreationDate="2015-11-26T17:31:49.537" Score="6" Body="&lt;p&gt;Both the answers from @Emre and @Madison May make good points about the issue at hand.  The problem is one of representing your string as a feature vector for input to the NN.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the problem depends on the size of the string you want to process.  Long strings containing may tokens (usually words) are often called documents in this setting.  There are separate methods for dealing with individual tokens/words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of ways to represent documents.  Many of them make the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow&quot;&gt;bag-of-words&lt;/a&gt; assumption.  The simplest types represent the document as a vector of the counts of words, or term frequency (tf).  In order to eliminate the effects of document length, usually people prefer to normalize by the number of documents a term shows up in, document frequency (&lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is topic modeling, which learns a latent lower-dimensional representation of the data.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_semantic_indexing&quot; rel=&quot;nofollow&quot;&gt;LSI/LSA&lt;/a&gt; are typical choices, but it's important to remember this is unsupervised.  The representation learned will not necessarily be ideal for whatever supervised learning you're doing with your NN.  If you want to do topic modeling, you might also try &lt;a href=&quot;https://www.cs.princeton.edu/~blei/papers/BleiMcAuliffe2007.pdf&quot; rel=&quot;nofollow&quot;&gt;supervised topic models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For individual words, you can use &lt;a href=&quot;https://code.google.com/p/word2vec/&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt;, which leverages NNs to embed words into an arbitrary-sized space.  Similarity between two word vectors in this learned space tends to correspond to semantic similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more recently pioneered approach is that of &lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow&quot;&gt;paragraph vectors&lt;/a&gt;, which first learns a word2vec-like word model, then builds on that representation to learn a distributed representation of sets of words (documents of any size).  This has shown state-of-the-art results in many applications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using NNs in NLP, people often use different architectures, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow&quot;&gt;Recurrent Neural Nets&lt;/a&gt; (like &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network#Long_short_term_memory_network&quot; rel=&quot;nofollow&quot;&gt;Long Short Term Memory&lt;/a&gt; networks).  In &lt;a href=&quot;http://www.aclweb.org/anthology/C14-1008&quot; rel=&quot;nofollow&quot;&gt;some cases&lt;/a&gt; people have even used &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/&quot; rel=&quot;nofollow&quot;&gt;Convolutional Neural Networks&lt;/a&gt; on text.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-28T04:54:54.333" LastActivityDate="2015-11-28T04:54:54.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9069" PostTypeId="1" CreationDate="2015-11-27T02:54:52.780" Score="3" ViewCount="500" Body="&lt;p&gt;I am working on &lt;code&gt;Spark MLlib&lt;/code&gt; and have a project where I have to make predictions for numeric data based on non-numeric features. I am a bit confused about which &lt;code&gt;regression algorithm&lt;/code&gt; to use from &lt;code&gt;Spark MLlib&lt;/code&gt; library primarily due to being new at this. The algorithms present in Spark MLlib library are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-linear models (SVMs, logistic regression, linear regression)&#xA;-naive Bayes&#xA;-decision trees&#xA;-ensembles of trees (Random Forests and Gradient-Boosted Trees)&#xA;-isotonic regression &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone provide me some guidance as to which algorithm will be suitable for predictions for numeric data based on non-numeric features?&lt;/p&gt;&#xA;" OwnerUserId="14383" LastActivityDate="2015-11-27T05:54:47.637" Title="Which Spark MLlib regression algorithm is suitable for numeric predictions based on non-numeric features?" Tags="|machine-learning|regression|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9070" PostTypeId="2" ParentId="9069" CreationDate="2015-11-27T03:48:58.973" Score="1" Body="&lt;p&gt;I would suggest &lt;code&gt;-ensembles of trees (Random Forests and Gradient-Boosted Trees)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/20416/how-should-decision-tree-splits-be-implemented-when-predicting-continuous-variab&quot;&gt;Here&lt;/a&gt; is a nice reference for handling such data with decision trees.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-27T03:48:58.973" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9071" PostTypeId="2" ParentId="9069" CreationDate="2015-11-27T04:15:23.570" Score="0" Body="&lt;p&gt;A linear regression would work, but the real issue here is &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-feature-extraction.html&quot; rel=&quot;nofollow&quot;&gt;feature extraction&lt;/a&gt;. You have to encode your &lt;a href=&quot;https://en.wikipedia.org/wiki/Categorical_variable#Categorical_variables_and_regression&quot; rel=&quot;nofollow&quot;&gt;categorical features&lt;/a&gt; somehow, likely by vectorizing them. You can one-hot encode your features, treat them as text and countVectorize them, etc.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-11-27T05:54:47.637" LastActivityDate="2015-11-27T05:54:47.637" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9073" PostTypeId="1" CreationDate="2015-11-27T08:35:19.797" Score="4" ViewCount="1892" Body="&lt;p&gt;If I understand it correctly, then the labels of &lt;a href=&quot;http://image-net.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageNet&lt;/a&gt; are based on WordNet:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ImageNet is an image dataset organized according to the WordNet hierarchy. Each meaningful concept in WordNet, possibly described by multiple words or word phrases, is called a &quot;synonym set&quot; or &quot;synset&quot;. There are more than 100,000 synsets in WordNet, majority of them are nouns (80,000+). In ImageNet, we aim to provide on average 1000 images to illustrate each synset. Images of each concept are quality-controlled and human-annotated. In its completion, we hope ImageNet will offer tens of millions of cleanly sorted images for most of the concepts in the WordNet hierarchy.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;http://image-net.org/about-overview&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://image-net.org/about-overview&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The text above states already that WordNet defines a hierarchy. You can see that by using the Python &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;nltk&lt;/code&gt; package&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import wordnet&#xA;nltk.download()&#xA;def get_hyponyms(synset):&#xA;    hyponyms = set()&#xA;    for hyponym in synset.hyponyms():&#xA;        hyponyms |= set(get_hyponyms(hyponym))&#xA;    return hyponyms | set(synset.hyponyms())&#xA;dog = wordnet.synset('dog.n.01')&#xA;print(get_hyponyms(dog))&#xA;harrier = wordnet.synset('harrier.n.02')&#xA;print(get_hyponyms(harrier))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(thanks to &lt;a href=&quot;https://stackoverflow.com/a/33662890/562769&quot;&gt;Stephan Ds answer&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;gives&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;set([Synset('harrier.n.02'), Synset('water_spaniel.n.01'), Synset('standard_poodle.n.01'), Synset('dandie_dinmont.n.01'), Synset('courser.n.03'), Synset('wirehair.n.01'), Synset('toy_manchester.n.01'), Synset('puppy.n.01'), Synset('briard.n.01'), Synset('beagle.n.01'), Synset('siberian_husky.n.01'), Synset('manchester_terrier.n.01'), Synset('bloodhound.n.01'), Synset('gordon_setter.n.01'), Synset('leonberg.n.01'), Synset('king_charles_spaniel.n.01'), Synset('yorkshire_terrier.n.01'), Synset('sealyham_terrier.n.01'), Synset('american_water_spaniel.n.01'), Synset('skye_terrier.n.01'), Synset('clumber.n.01'), Synset('pembroke.n.01'), Synset('wire-haired_fox_terrier.n.01'), Synset('shih-tzu.n.01'), Synset('newfoundland.n.01'), Synset('retriever.n.01'), Synset('cocker_spaniel.n.01'), Synset('springer_spaniel.n.01'), Synset('american_foxhound.n.01'), Synset('large_poodle.n.01'), Synset('lapdog.n.01'), Synset('bull_mastiff.n.01'), Synset('affenpinscher.n.01'), Synset('irish_water_spaniel.n.01'), Synset('terrier.n.01'), Synset('keeshond.n.01'), Synset('vizsla.n.01'), Synset('dalmatian.n.02'), Synset('bird_dog.n.01'), Synset('irish_terrier.n.01'), Synset('miniature_poodle.n.01'), Synset('dachshund.n.01'), Synset('australian_terrier.n.01'), Synset('blenheim_spaniel.n.01'), Synset('weimaraner.n.01'), Synset('soft-coated_wheaten_terrier.n.01'), Synset('doberman.n.01'), Synset('kelpie.n.02'), Synset('water_dog.n.02'), Synset('feist.n.01'), Synset('attack_dog.n.01'), Synset('french_bulldog.n.01'), Synset('papillon.n.01'), Synset('bedlington_terrier.n.01'), Synset('foxhound.n.01'), Synset('labrador_retriever.n.01'), Synset('great_dane.n.01'), Synset('kerry_blue_terrier.n.01'), Synset('miniature_schnauzer.n.01'), Synset('pariah_dog.n.01'), Synset('border_terrier.n.01'), Synset('staghound.n.01'), Synset('norwegian_elkhound.n.01'), Synset('redbone.n.01'), Synset('pooch.n.01'), Synset('old_english_sheepdog.n.01'), Synset('police_dog.n.01'), Synset('welsh_terrier.n.01'), Synset('spitz.n.01'), Synset('boxer.n.04'), Synset('tibetan_terrier.n.01'), Synset('shetland_sheepdog.n.01'), Synset('boarhound.n.01'), Synset('border_collie.n.01'), Synset('wolfhound.n.01'), Synset('lhasa.n.02'), Synset('scotch_terrier.n.01'), Synset('coondog.n.01'), Synset('giant_schnauzer.n.01'), Synset('japanese_spaniel.n.01'), Synset('german_short-haired_pointer.n.01'), Synset('entlebucher.n.01'), Synset('griffon.n.03'), Synset('griffon.n.02'), Synset('welsh_springer_spaniel.n.01'), Synset('clydesdale_terrier.n.01'), Synset('hound.n.01'), Synset('brittany_spaniel.n.01'), Synset('corgi.n.01'), Synset('pekinese.n.01'), Synset('mastiff.n.01'), Synset('flat-coated_retriever.n.01'), Synset('sennenhunde.n.01'), Synset('schipperke.n.01'), Synset('english_toy_spaniel.n.01'), Synset('ibizan_hound.n.01'), Synset('airedale.n.01'), Synset('cardigan.n.02'), Synset('miniature_pinscher.n.01'), Synset('bluetick.n.01'), Synset('west_highland_white_terrier.n.01'), Synset('seizure-alert_dog.n.01'), Synset('pomeranian.n.01'), Synset('english_foxhound.n.01'), Synset('bernese_mountain_dog.n.01'), Synset('norfolk_terrier.n.01'), Synset('greater_swiss_mountain_dog.n.01'), Synset('collie.n.01'), Synset('chow.n.03'), Synset('pug.n.01'), Synset('scottish_deerhound.n.01'), Synset('groenendael.n.01'), Synset('golden_retriever.n.01'), Synset('schnauzer.n.01'), Synset('irish_setter.n.01'), Synset('german_shepherd.n.01'), Synset('walker_hound.n.01'), Synset('english_setter.n.01'), Synset('english_springer.n.01'), Synset('sporting_dog.n.01'), Synset('afghan_hound.n.01'), Synset('cairn.n.02'), Synset('rhodesian_ridgeback.n.01'), Synset('chesapeake_bay_retriever.n.01'), Synset('irish_wolfhound.n.01'), Synset('fox_terrier.n.01'), Synset('sled_dog.n.01'), Synset('toy_dog.n.01'), Synset('staffordshire_bullterrier.n.01'), Synset('bullterrier.n.01'), Synset('seeing_eye_dog.n.01'), Synset('samoyed.n.03'), Synset('bouvier_des_flandres.n.01'), Synset('otterhound.n.01'), Synset('kuvasz.n.01'), Synset('cur.n.01'), Synset('guide_dog.n.01'), Synset('malinois.n.01'), Synset('malamute.n.01'), Synset('poodle.n.01'), Synset('curly-coated_retriever.n.01'), Synset('toy_spaniel.n.01'), Synset('basset.n.01'), Synset('toy_terrier.n.01'), Synset('tibetan_mastiff.n.01'), Synset('basenji.n.01'), Synset('field_spaniel.n.01'), Synset('mexican_hairless.n.01'), Synset('setter.n.02'), Synset('great_pyrenees.n.01'), Synset('american_staffordshire_terrier.n.01'), Synset('rottweiler.n.01'), Synset('standard_schnauzer.n.01'), Synset('black-and-tan_coonhound.n.01'), Synset('borzoi.n.01'), Synset('bulldog.n.01'), Synset('sausage_dog.n.01'), Synset('sussex_spaniel.n.01'), Synset('spaniel.n.01'), Synset('working_dog.n.01'), Synset('belgian_sheepdog.n.01'), Synset('watchdog.n.02'), Synset('silky_terrier.n.01'), Synset('eskimo_dog.n.01'), Synset('brabancon_griffon.n.01'), Synset('whippet.n.01'), Synset('plott_hound.n.01'), Synset('liver-spotted_dalmatian.n.01'), Synset('coonhound.n.01'), Synset('saluki.n.01'), Synset('toy_poodle.n.01'), Synset('hearing_dog.n.01'), Synset('chihuahua.n.03'), Synset('lakeland_terrier.n.01'), Synset('shepherd_dog.n.01'), Synset('rat_terrier.n.01'), Synset('italian_greyhound.n.01'), Synset('komondor.n.01'), Synset('saint_bernard.n.01'), Synset('norwich_terrier.n.01'), Synset('pointer.n.04'), Synset('appenzeller.n.01'), Synset('maltese_dog.n.01'), Synset('hunting_dog.n.01'), Synset('smooth-haired_fox_terrier.n.01'), Synset('housedog.n.01'), Synset('boston_bull.n.01'), Synset('pinscher.n.01'), Synset('greyhound.n.01')])&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;set([])&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are all images of ImageNet in the leaves, e.g. &lt;code&gt;'harrier.n.02'&lt;/code&gt; or are some images only labeled as &lt;code&gt;'dog.n.01'&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Sub-question: ImageNet was labeled by ordinary people (not specialists in the topic, e.g. for dogs not biologists with specialization in dogs or something similar) via Amazon Mechanical Turk, if I remember it correctly. How did the people there know all these different kinds of dogs? I wouldn't know what a borzoi is... so how did they check if the label was correct and specific enough?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the 21841 synsets (&lt;a href=&quot;http://image-net.org/about-stats&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;) leaves or also inner nodes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are all images in ImageNet in the leaves? How many leaves are there?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-03-23T07:51:10.453" Title="Are all images in ImageNet in the leaves?" Tags="|dataset|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9074" PostTypeId="1" AcceptedAnswerId="26495" CreationDate="2015-11-27T09:05:46.147" Score="6" ViewCount="11170" Body="&lt;p&gt;Until recently, I thought that &quot;labeling&quot; and &quot;classification&quot; are synonyms. But when I started another question about &lt;a href=&quot;https://stackoverflow.com/q/33947823/562769&quot;&gt;terminology in computer vision&lt;/a&gt; I thought about it: Is there a difference between &quot;labeling&quot; and &quot;classification&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought that the &quot;class&quot; is the concept you want to detect and &quot;label&quot; is what you assign to data. So the &quot;class&quot; is a concept which leads to the data and &quot;label&quot; is only the name. Hence &quot;labeling&quot; would be the same as &quot;classification&quot; as both want to make a statement about the underlying class which lead to the data.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Articles&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A quick search via Google Scholar revealed that some articles use both terms in the title:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Markus Eich, Malgorzata Dabrowska, and Frank Kirchner: &quot;Semantic Labeling: Classification of 3D Entities Based on Spatial Feature Descriptors&quot;&lt;/li&gt;&#xA;&lt;li&gt;Chunlin Li, Dmitry B. Goldgof, and Lawrence 0. Hall: &quot;Knowledge-based classification and tissue labeling of MR images of human brain&quot;&lt;/li&gt;&#xA;&lt;li&gt;Ray Blanchard: &quot;The classification and labeling of nonhomosexual gender dysphorias&quot; - another research area but probably it is the same difference between the two words?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So I guess there is a difference between &quot;labeling&quot; and &quot;classification&quot;. What is the difference?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Google N-Gram&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zpn9y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zpn9y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;classification seems to be a much boarder term.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-01-10T18:32:19.417" Title="Is there a difference between &quot;classification&quot; and &quot;labeling&quot;?" Tags="|classification|terminology|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9075" PostTypeId="2" ParentId="9000" CreationDate="2015-11-27T09:11:51.690" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.r-podcast.org/&quot; rel=&quot;nofollow&quot;&gt;The R-Podcast&lt;/a&gt; is a podcast about data analysis in R. Other cases are &lt;a href=&quot;https://itunes.apple.com/us/podcast/oreilly-data-show/id944929220?intcmp=il-data-na-na-na_20150826_radar_data_show_podcast_post_top_subscribe_cta&quot; rel=&quot;nofollow&quot;&gt;O'Reilly Data Show&lt;/a&gt; and &lt;a href=&quot;http://radar.oreilly.com/tag/podcast&quot; rel=&quot;nofollow&quot;&gt;O'Reilly Radar Podcasts&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-11-27T09:18:57.340" LastActivityDate="2015-11-27T09:18:57.340" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:49.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="9076" PostTypeId="1" AcceptedAnswerId="9077" CreationDate="2015-11-27T09:26:20.967" Score="1" ViewCount="129" Body="&lt;p&gt;I have the dataset test_stopword and I want to remove some words from the dataset based on a vector. How can I do that in R?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;texts &amp;lt;- c(&quot;This is the first document.&quot;, &#xA;       &quot;Is this a text?&quot;, &#xA;       &quot;This is the second file.&quot;, &#xA;       &quot;This is the third text.&quot;, &#xA;       &quot;File is not this.&quot;) &#xA;&#xA;test_stopword &amp;lt;- as.data.frame(texts)&#xA;ordinal_stopwords  &amp;lt;- c(&quot;first&quot;,&quot;primary&quot;,&quot;second&quot;,&quot;secondary&quot;,&quot;third&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-11-27T11:18:03.633" Title="removing words based on a predefined vector" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9077" PostTypeId="2" ParentId="9076" CreationDate="2015-11-27T09:48:24.633" Score="2" Body="&lt;pre&gt;&lt;code&gt;texts &amp;lt;- c(&quot;This is the first document.&quot;, &#xA;       &quot;Is this a text?&quot;, &#xA;       &quot;This is the second file.&quot;, &#xA;       &quot;This is the third text.&quot;, &#xA;       &quot;File is not this.&quot;) &#xA;&#xA;test_stopword &amp;lt;- as.data.frame(texts)&#xA;ordinal_stopwords  &amp;lt;- c(&quot;first&quot;,&quot;primary&quot;,&quot;second&quot;,&quot;secondary&quot;,&quot;third&quot;)&#xA;&#xA;(newdata &amp;lt;- as.data.frame(sapply(texts, function(x)   gsub(paste(ordinal_stopwords, collapse = '|'), '', x))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is getting skewed when added in a code block (&lt;a href=&quot;https://meta.stackexchange.com/q/270069/302377&quot;&gt;maybe a bug in SE&lt;/a&gt;). But, you would get the desired output.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-03-20T10:32:30.217" LastActivityDate="2015-11-27T11:18:03.633" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9078" PostTypeId="2" ParentId="9016" CreationDate="2015-11-27T09:50:28.027" Score="0" Body="&lt;p&gt;If you work on Linux, you may create a script that loads data and runs R inside and give the users the access to execute the main script. I do not know the details but this may be a possible solution.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-11-27T09:50:28.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9079" PostTypeId="2" ParentId="8941" CreationDate="2015-11-27T10:39:27.050" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://conferences.oreilly.com/strata&quot; rel=&quot;nofollow&quot;&gt;Strata + Hadoop World&lt;/a&gt; by O'Reilly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dataedge.ischool.berkeley.edu/&quot; rel=&quot;nofollow&quot;&gt;DataEDGE&lt;/a&gt; by Berkeley School of Information. &lt;a href=&quot;https://dataedge.ischool.berkeley.edu/2015/video&quot; rel=&quot;nofollow&quot;&gt;Link&lt;/a&gt; to the videos.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="11097" LastEditDate="2015-11-27T10:41:05.850" LastActivityDate="2015-11-27T10:41:05.850" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="9081" PostTypeId="2" ParentId="6760" CreationDate="2015-11-27T14:55:43.017" Score="0" Body="&lt;p&gt;I recommend calculating 1-gram, 2-grams and ... frequencies, sorting based on frequencies and observing what pops up. Sometimes you see patterns that need extra data cleansing. &lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-11-27T14:55:43.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9082" PostTypeId="1" AcceptedAnswerId="9084" CreationDate="2015-11-27T16:11:32.240" Score="1" ViewCount="38" Body="&lt;p&gt;I would like to ask if the matrix that is returned from the R cor() function applied on a dataset, is considered a confusion matrix. Thank you&lt;/p&gt;&#xA;" OwnerUserId="13875" LastActivityDate="2015-11-27T16:21:19.570" Title="result of correlation function in r" Tags="|r|correlation|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9083" PostTypeId="1" AcceptedAnswerId="9092" CreationDate="2015-11-27T16:18:14.920" Score="0" ViewCount="362" Body="&lt;p&gt;I know that Spark enhances performance relative to mapreduce by doing in-memory computations. But, since the caching is explicitly decided by the programmer, one can also proceed without doing that. I believe even in such cases, Spark is 10x faster than map reduce. What features in the framework make this possible?&lt;/p&gt;&#xA;" OwnerUserId="14391" LastEditorUserId="13413" LastEditDate="2015-11-28T06:20:59.060" LastActivityDate="2015-11-28T09:07:14.723" Title="Spark vs Map reduce" Tags="|apache-spark|map-reduce|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9084" PostTypeId="2" ParentId="9082" CreationDate="2015-11-27T16:21:19.570" Score="1" Body="&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;, it is not a confusion matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please have a look at the &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html&quot; rel=&quot;nofollow&quot;&gt;docs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.inside-r.org/node/86995&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is how the confusion matrix is created in R.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-11-27T16:21:19.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9085" PostTypeId="1" CreationDate="2015-11-27T17:44:20.980" Score="7" ViewCount="238" Body="&lt;p&gt;I  have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current way I am doing that is by having a &quot;reasonable window&quot; of what I consider to be a legitimate vote.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For items with less than 10 votes;&#xA;I currently do nothing and take the mean as it's rating. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Window = 4.5 - Log(TotalVotes, 10);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So a reasonable vote range is then &lt;code&gt;(Mean - Window) thru (Mean + Window)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the reasonable vote range is found, the &quot;Rating&quot; is just the mean of all the reasonable votes (those who fall in the reasonable range).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means an item with a real mean of 4.2 with 100 votes would have a window of &lt;code&gt;4.5-Log(100,10) = 2.5&lt;/code&gt;, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has worked good in general but the issue is when an item's &lt;code&gt;Mean - Window&lt;/code&gt; is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.&lt;/p&gt;&#xA;" OwnerUserId="14392" LastActivityDate="2015-12-16T14:20:18.467" Title="Reducing the effect of down voters with rating system" Tags="|statistics|normalization|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9086" PostTypeId="1" CreationDate="2015-11-27T18:11:03.323" Score="10" ViewCount="17347" Body="&lt;p&gt;I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and Sparks MLlib(or PredictionIO).An example of the desired result would be to be able to predict based on the exception logs gathered to be able to predict which user is more likely to cause the next exception and at which feature(and bunch of other stuff to keep track and improve optimization of the application).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have successfully been able to ingest data from ElasticSearch into spark and create DataFrames and map the data needed. What I would like to know is how do I approach the Machine Learning aspect of my implementation. I've been through articles and papers that talk about Data preprocessing, training the data models and creating labels and then generating predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions I have are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;How do I approach transforming the exiting log data into numerical vectors which can be used to datasets to be trained.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What algorithms do I use to train my dataset(with the limited knowledge i've gathered the past couple days, i was thinking bout implementing linear regression, please suggest which implementation would be best)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just looking for suggestions on how to approach this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank You.&lt;/p&gt;&#xA;" OwnerUserId="14393" LastActivityDate="2016-12-02T17:40:46.663" Title="Server log analysis using machine learning" Tags="|machine-learning|predictive-modeling|apache-spark|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9087" PostTypeId="1" AcceptedAnswerId="9112" CreationDate="2015-11-27T19:04:28.730" Score="3" ViewCount="6100" Body="&lt;p&gt;I would like to ask if the Pearson correlation between fields (but not the class field) of a dataset affects somehow the performance of Naive Bayes when applying it to the dataset in order to predict the class field.&lt;/p&gt;&#xA;" OwnerUserId="13875" LastEditorUserId="38887" LastEditDate="2020-05-08T10:41:50.037" LastActivityDate="2020-05-08T10:41:50.037" Title="Correlation and Naive Bayes" Tags="|predictive-modeling|correlation|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9088" PostTypeId="2" ParentId="9083" CreationDate="2015-11-27T21:45:39.147" Score="1" Body="&lt;p&gt;When you script out steps for spark on an RDD, it does not begin executing the operations until the data needs to be accessed.  Instead of actually manipulating the data, spark builds a graph of how to go from the data, to the desired result set, and will not store intermediate data.  For example, if you enter the following into a shell&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_rdd = sc.parallelize(some_data)&#xA;mapped_data = my_rdd.map(...)&#xA;filtered_data = mapped_data.filter(...)&#xA;filtered_data.take(3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you'll notice that when you enter lines 2 and 3, no execution occurs.  This is because spark isn't creating mapped_data or filtered_data, but rather, is building a graph of how to get from my_rdd to filtered data.  The only reason that you would really write it in this way, is if you need to access mapped_data at a later state, in which case you would want to use a cache.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-11-27T21:45:39.147" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9089" PostTypeId="1" AcceptedAnswerId="9090" CreationDate="2015-11-28T01:40:58.530" Score="1" ViewCount="84" Body="&lt;p&gt;This may be a silly question, apologies if it is. And further apologies if I've said something wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Network graphs, or visualisations look somewhat like &lt;a href=&quot;https://flowingdata.com/wp-content/uploads/2013/06/revere-network-reduced1-620x574.png&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;, with each node representing a thing connected to another thing. So for my example, I'd like to model population changes for refugees from their origin country to their destination country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been tasked to create a graph which shows a similar change over a period of years, but I am really not sure how years could be modelled? e.g. how would a graph like the example above, show a change in years?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: preferably using the NodeXL plugin&lt;/p&gt;&#xA;" OwnerUserId="14397" LastEditorUserId="11097" LastEditDate="2015-11-28T02:48:33.530" LastActivityDate="2015-11-28T04:51:34.707" Title="Modelling population changes with years on a network graph" Tags="|visualization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9090" PostTypeId="2" ParentId="9089" CreationDate="2015-11-28T02:48:00.047" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Sankey_diagram&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sankey diagram&lt;/a&gt; would be nicely suited for your problem statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The flow width can be the number of refugees migrated, and the end nodes can be the destinations (from and to) of the refugees, and the flow width would be the magnitude of refugee migration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to model the graph on a geographical map, it can look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VA78L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VA78L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure whether it can be done in NodeXL or not.  But, a google search has returned me this &lt;a href=&quot;http://djjr-courses.wikidot.com/ppol225:sankey-diagram&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T04:51:34.707" LastActivityDate="2015-11-28T04:51:34.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9091" PostTypeId="2" ParentId="9089" CreationDate="2015-11-28T04:21:48.200" Score="0" Body="&lt;p&gt;It seems that our first issue is to model the graph before visualizing it. I would recommend &lt;a href=&quot;http://neo4j.com/&quot; rel=&quot;nofollow&quot;&gt;Neo4j&lt;/a&gt; and its &lt;a href=&quot;http://neo4j.com/developer/cypher-query-language/&quot; rel=&quot;nofollow&quot;&gt;Cypher&lt;/a&gt; query language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the data modeling, states can be the nodes with a population property (or even population for each year), relationship could be the directed migration with the number of refugees and a year property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now with such a graph at hand you can easily retrieve the total number of refugees per year by using the year property from the nodes or the edges. &lt;/p&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-11-28T04:21:48.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9092" PostTypeId="2" ParentId="9083" CreationDate="2015-11-28T09:07:14.723" Score="4" Body="&lt;p&gt;No, this is not in general true. For a map-only job, or map and reduce, MapReduce is a bit faster. It's more optimized for this pattern and a lot easier to scale / tune. However, the problem is that few problems are just one or two operations. Once you have a chain of them to execute, considering the entire DAG and execution plan is a win even if memory is not used for persistence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Developer productivity also related to efficiency. If I don't have to spend my time writing lower-level code then I can spend more time designing a more sophisticated distributed application that could be faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see this in Crunch, for example, which gives you a high-level language similar to Spark Java on MapReduce, and I'd say you can write more efficient Crunch jobs in a fixed time than M/R.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2015-11-28T09:07:14.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9093" PostTypeId="5" CreationDate="2015-11-28T14:59:17.057" Score="0" Body="&lt;p&gt;Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the items are unlabelled, there's nothing that points toward the &quot;correct&quot; labels, as there is with supervised learning. Unsupervised learning uses methods like clustering and principal components analysis to discover structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference:&lt;br&gt;&#xA;&lt;a href=&quot;http://en.wikipedia.org/wiki/Unsupervised_learning&quot; rel=&quot;nofollow&quot;&gt;Wikipedia - Unsupervised learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:28.187" LastActivityDate="2015-11-28T16:55:28.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9094" PostTypeId="4" CreationDate="2015-11-28T14:59:17.057" Score="0" Body="Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:19.880" LastActivityDate="2015-11-28T16:55:19.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9095" PostTypeId="5" CreationDate="2015-11-28T14:59:51.690" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-28T14:59:51.690" LastActivityDate="2015-11-28T14:59:51.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9096" PostTypeId="4" CreationDate="2015-11-28T14:59:51.690" Score="0" Body="Gradient Boosting Machine&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:36.527" LastActivityDate="2015-11-28T16:55:36.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9097" PostTypeId="5" CreationDate="2015-11-28T15:01:23.757" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-28T15:01:23.757" LastActivityDate="2015-11-28T15:01:23.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9098" PostTypeId="4" CreationDate="2015-11-28T15:01:23.757" Score="0" Body="The discipline at the crossroads of computing and the biological sciences, which involves organizing, maintaining, and analyzing data from such fields as molecular biology, genetics, and genomics" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:06.613" LastActivityDate="2015-11-28T16:55:06.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9099" PostTypeId="5" CreationDate="2015-11-28T15:02:56.000" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-28T15:02:56.000" LastActivityDate="2015-11-28T15:02:56.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9100" PostTypeId="4" CreationDate="2015-11-28T15:02:56.000" Score="0" Body="Excel is a commercial spreadsheet program created by Microsoft." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:16.590" LastActivityDate="2015-11-28T16:55:16.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9101" PostTypeId="5" CreationDate="2015-11-28T15:03:42.977" Score="0" Body="&lt;p&gt;The expectation maximization (EM) algorithm is an optimization algorithm. It is an instance of a broader class of optimization algorithms known as Majorization-Minimization (MM). It is a cornerstone of statistics since it particularly suitable for &quot;skipping&quot; local maxima which often arise in likelihood maximization problems, especially in the presence of missing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, the &lt;a href=&quot;http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm&quot; rel=&quot;nofollow&quot;&gt;EM algorithm&lt;/a&gt; is an iterative method for finding maximum likelihood estimates. The typical form of the EM algorithm works as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Expectation Step&lt;/em&gt;: Compute the expected value of the log-likelihood function based on the current estimate.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Maximization Step&lt;/em&gt;: Update the estimate by maximizing the result of the expectation step.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:54:40.127" LastActivityDate="2015-11-28T16:54:40.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9102" PostTypeId="4" CreationDate="2015-11-28T15:03:42.977" Score="0" Body="An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data. " OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:54:53.670" LastActivityDate="2015-11-28T16:54:53.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9103" PostTypeId="5" CreationDate="2015-11-28T15:04:14.043" Score="0" Body="&lt;p&gt;Descriptive statistics summarize features of a sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Common descriptive statistics include mean and standard deviations, particular quantiles like the median and quartiles, the maximum and minimum, range and interquartile range, five number summaries and so on, but with multiple variables, may include correlations and crosstabs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Descriptive statistics may include visual displays such as boxplots, histograms and scatterplots.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:32.907" LastActivityDate="2015-11-28T16:55:32.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9104" PostTypeId="4" CreationDate="2015-11-28T15:04:14.043" Score="0" Body="Descriptive statistics summarize features of a sample, such as mean and standard deviations, median and quartiles, the maximum and minimum. With multiple variables, may include correlations and crosstabs. Can include visual displays - boxplots, histograms, scatterplots and so on.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:55:14.300" LastActivityDate="2015-11-28T16:55:14.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9105" PostTypeId="5" CreationDate="2015-11-28T15:12:29.987" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-11-28T15:12:29.987" LastActivityDate="2015-11-28T15:12:29.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9106" PostTypeId="4" CreationDate="2015-11-28T15:12:29.987" Score="0" Body="Given multivariate data split into several subsamples (classes) the analysis finds linear combinations of variables, called discriminant functions, which discriminate between classes and are uncorrelated. The functions are applied then to assign old or new observations to the classes. Discriminant analysis is both dimensionality reduction and classification technique." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-11-28T16:54:51.050" LastActivityDate="2015-11-28T16:54:51.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9107" PostTypeId="2" ParentId="9025" CreationDate="2015-11-28T16:04:50.477" Score="2" Body="&lt;p&gt;After reading some more papers, I realize that I misunderstood how the graphs work. The graphs are supposed to contain the conditional probabilities based on their parent(s). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This clears up the doubts that I had before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information, see &lt;a href=&quot;http://www.csse.monash.edu.au/bai/book/BAI_Chapter2.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; book chapter.&lt;/p&gt;&#xA;" OwnerUserId="14340" LastActivityDate="2015-11-28T16:04:50.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9108" PostTypeId="2" ParentId="9074" CreationDate="2015-11-28T19:19:51.470" Score="1" Body="&lt;p&gt;The way I view it: 'Classification' (in the context of machine learning) is &lt;strong&gt;a type of problem&lt;/strong&gt; in which you assign a 'label' to an object. Formally, 'Classification' is a type of problem whereas labeling is a function from an object to a set of labels (maybe infinite).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much the same way Regression is a type of problem where you, again, assign a label to an object only this time the label is some real number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both in Classification and in Regression you are attempting to find the 'best' labeling function with respect to some metric/loss function.&lt;/p&gt;&#xA;" OwnerUserId="14403" LastActivityDate="2015-11-28T19:19:51.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9109" PostTypeId="1" CreationDate="2015-11-28T19:36:54.963" Score="3" ViewCount="651" Body="&lt;p&gt;I am making some stochastic training ensemble classes in Python, and I want to get hyperparameters values. Grid search will take too long for moderate data sets, because in my stochastic training I train different learners on different dropouts, as in cross-validation, then I average or weight-average on best error for dropout. Having this set, I want to get best parameter values at every iteration, so I thought to use a genetic algorithm approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found one example on Github, but failed to install, and I am looking for more references of genetic algorithms for doing this. Can anyone help me?&lt;/p&gt;&#xA;" OwnerUserId="10094" LastEditorUserId="11097" LastEditDate="2015-11-29T05:48:42.870" LastActivityDate="2019-04-25T10:13:24.880" Title="Genetic Algorithm to find best parameter values of an estimaor" Tags="|optimization|hyperparameter|genetic-algorithms|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9110" PostTypeId="2" ParentId="9025" CreationDate="2015-11-28T19:43:43.387" Score="0" Body="&lt;p&gt;I think , if you are  not   in a particular  interest  to use  Java(which is not  a deditcated  language for machine learning) you could  find a lot  of  references in R and Python:&#xA;&lt;a href=&quot;https://pymc-devs.github.io/pymc/&quot; rel=&quot;nofollow&quot;&gt;https://pymc-devs.github.io/pymc/&lt;/a&gt;&#xA;&lt;a href=&quot;http://www.bayespy.org/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.bayespy.org/index.html&lt;/a&gt;&#xA;&lt;a href=&quot;http://www.bnlearn.com/examples/&quot; rel=&quot;nofollow&quot;&gt;http://www.bnlearn.com/examples/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2015-11-28T19:43:43.387" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9111" PostTypeId="2" ParentId="9074" CreationDate="2015-11-28T20:58:34.853" Score="0" Body="&lt;p&gt;Short answer: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No, there is no difference between labelling and classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Class - a set or category of things having some property or attribute in common and differentiated from others by kind, type or quality. See 'category'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Label - word or phrase indicating that what follows belongs in a particular category or class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To classify something is to label it, they are the necessarily the same thing. The term labelling probably evolved because &quot;label&quot; allows you to avoid saying &quot;class&quot; which has other connotations in Computer Science. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Label is much simpler, and in all cases, classification is just the act of putting labels on objects (or learning to correctly do so).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The discrepancy you're seeing in the use of labelling/classification comes from the simple fact that a title like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Semantic Classification: Classification of 3D Entities Based on Spatial Feature Descriptors&quot;&#xA;or&#xA;&quot;Knowledge-based classification and tissue classification of MR images of human brain&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sound really awkward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like most academic paper titles, these are just overly complex descriptions of what is in the paper that explain exactly what is going on without sounding redundant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TL;DR - Don't get hung up on terminology!&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2015-11-28T20:58:34.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9112" PostTypeId="2" ParentId="9087" CreationDate="2015-11-28T22:25:05.030" Score="4" Body="&lt;p&gt;As you probably know, naive here implies that the &quot;fields&quot; are independent. So your question boils down to does correlation imply dependence. Yes, it does. See here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/113417/does-non-zero-correlation-imply-dependence&quot;&gt;https://stats.stackexchange.com/questions/113417/does-non-zero-correlation-imply-dependence&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if your features show correlation then this will have an adverse effect on the naive assumption. Despite this fact, Naive Bayes has been shown to be robust against this assumption. If your model still suffers from this, however, you could consider transforming the space to be independent with methods such as PCA.  &lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-11-28T22:25:05.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9114" PostTypeId="2" ParentId="8813" CreationDate="2015-11-29T08:09:29.833" Score="2" Body="&lt;p&gt;I tried to estimate a few of your data values from the scatter plot you provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I performed a power model regression and came up with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = (5.777 \cdot 10^{-16}) \cdot 1.404^{x}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ni7RV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ni7RV.png&quot; alt=&quot;Graph of model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimated values I used are below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(70, 0.01), (75, 0.012), (80, 0.015), (90, 0.025), (95, 0.075), (98, 0.15), (99, 0.20), (99.5, 0.25), (99.9, 0.32)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, your actual model will differ because you have the actual data set. I just eyeballed a few points so I could test the power fit.&lt;/p&gt;&#xA;" OwnerUserId="14381" LastActivityDate="2015-11-29T08:09:29.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9115" PostTypeId="2" ParentId="8813" CreationDate="2015-11-29T08:50:05.273" Score="1" Body="&lt;p&gt;I suspect that your $x$ values, as they are percentages, lie in $[0,100[$. Regarding $y$ values, they seem positive. But many are very close to $0$. So I would first decide whether $y$s below a threshold should be put aside first as outliers, as they will have a huge influence on the first basic fits. You can reintroduce them afterward with robust fitting procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important question is: are $y$s bounded or not? The slope seems very steep, so you have to guess if the derivative is infinite at $x=100$, to help you chose models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe a first idea is to perform a change of variable on the $x$ axis, with $x' = \frac{1}{{(100-x)}^\alpha}$ and try some $\alpha $ values, to see if clearer patterns appear.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2015-11-29T08:50:05.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9116" PostTypeId="2" ParentId="8941" CreationDate="2015-11-29T09:51:01.640" Score="1" Body="&lt;p&gt;Startup.ml conference has had a couple good conferences. The talks are done by experts in the field and I found more technical and practical than other conferences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://conf.startup.ml&quot; rel=&quot;nofollow&quot;&gt;http://conf.startup.ml&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="588" LastEditorUserId="588" LastEditDate="2015-11-29T21:41:11.117" LastActivityDate="2015-11-29T21:41:11.117" CommentCount="0" CommunityOwnedDate="2016-04-14T10:22:59.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="9117" PostTypeId="1" CreationDate="2015-11-29T10:18:27.317" Score="0" ViewCount="119" Body="&lt;p&gt;I have an extremely unbalanced data set: around 200 positive samples and 70,000 negative samples. To overcome this problem I have tried to over-sample the minority class as suggested in previous questions here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since training the classifier (in my case I am using SVM) takes a long time on my computer I first trained a classifier on a small subset of my data picked randomly, the classifier gave solid results when tested on the test set, yet when I trained a classifier using the full training data the classifier produced very poor results. Does anyone have insight into why this is happening and what should i do? I have also tried using weights but it did not produce a different result.&lt;/p&gt;&#xA;" OwnerUserId="14411" LastActivityDate="2016-01-30T20:43:18.070" Title="Subset of training set produces good results while full training set produces poor results" Tags="|svm|class-imbalance|cross-validation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9118" PostTypeId="2" ParentId="9085" CreationDate="2015-11-29T11:18:31.997" Score="5" Body="&lt;p&gt;You should look into other estimators of location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want is a &lt;em&gt;robust&lt;/em&gt; estimator, with a &lt;em&gt;high break-down point&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The extreme approach would be the median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you may get more numerically interesting results with a &lt;strong&gt;trimmed mean&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-11-29T11:18:31.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9119" PostTypeId="1" AcceptedAnswerId="9124" CreationDate="2015-11-29T12:51:16.560" Score="3" ViewCount="91" Body="&lt;p&gt;I have a graph with vertices that represent some entities and the edges are weighted as the correlation between two such entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to break this graph into several subgraphs with high inner-correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that I have a few 'hubs', with high correlation to a lot of different entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I detect such hubs in order to remove them?&lt;/p&gt;&#xA;" OwnerUserId="14044" LastActivityDate="2015-11-30T05:28:55.880" Title="Hub removal from graphs" Tags="|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9120" PostTypeId="2" ParentId="9038" CreationDate="2015-11-29T15:59:03.413" Score="3" Body="&lt;p&gt;Excellent question. In chapter 4 of &quot;Illuminating the Path, The Research and Development Agenda for Visual Analytics&quot; by James J. Thomas and Kristin A. Cook is a discussion on data representations and data transformations. In my research I have approached this question in the context of PCA and factor analysis. My brief answer is that the visualizations are useful if one has the data transformation to move from the visualization space to the original data space. This would additionally be conducted within a visual analytics framework.&lt;/p&gt;&#xA;" OwnerUserId="14343" LastEditorUserId="14343" LastEditDate="2015-12-03T22:52:39.377" LastActivityDate="2015-12-03T22:52:39.377" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9124" PostTypeId="2" ParentId="9119" CreationDate="2015-11-30T05:28:55.880" Score="2" Body="&lt;p&gt;Depends on how you define &quot;hubs&quot;. In Network Science, a hub is simply a node with high degree i.e. those node who contribute to the power-law nature of the degree distribution the most. But you can also find other definitions for instance according to the information flow in the network where hubs are defined as those node that are critical in the process information flow (also called central nodes). &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My Suggestions&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Degree Distribution:&lt;/strong&gt; The simplest approach would be to choose high degree nodes as hubs. To ensure your results a bit more I recommend to calculate the summation of all correlations correspond to each node and have a look at these numbers as well. In this case you are looking for nodes which have high degrees and contain a larger value of correlation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Centrality Measure:&lt;/strong&gt; or &lt;em&gt;Betweenness&lt;/em&gt; introduced by Linton Freeman which again somehow measures the influence of a node in the process of information flow over the network. Calculating the centrality for a vertex $v$ in a graph has basically 3 steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For each pair of vertices $(s,t)$, compute the shortest paths between them.&lt;/li&gt;&#xA;&lt;li&gt;For each pair of vertices $(s,t)$, determine the fraction of shortest paths that pass through the vertex in question (here, vertex $v$).&lt;/li&gt;&#xA;&lt;li&gt;Sum this fraction over all pairs of vertices $(s,t)$.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For more information read &lt;a href=&quot;https://en.wikipedia.org/wiki/Centrality&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; carefully and in case you need any help (specially for implementation) just drop me a line in the comments :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-11-30T05:28:55.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9125" PostTypeId="2" ParentId="2587" CreationDate="2015-11-30T05:51:36.977" Score="1" Body="&lt;p&gt;Unfortunately, there isn't any.  However, the &lt;code&gt;AbstractDifferenceRecommenderEvaluator&lt;/code&gt; can be extended for the purpose. The class would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public abstract class AbstractKFoldRecommenderEvaluator extends AbstractDifferenceRecommenderEvaluator {&#xA;&#xA;private final Random random;&#xA;public double noEstimateCounterAverage = 0.0;&#xA;public double totalEstimateCount = 0.0;&#xA;public double totalEstimateCountAverage = 0.0;&#xA;&#xA;private static final Logger log = LoggerFactory&#xA;        .getLogger(AbstractKFoldRecommenderEvaluator.class);&#xA;&#xA;public AbstractKFoldRecommenderEvaluator() {&#xA;    super();&#xA;    random = RandomUtils.getRandom();&#xA;}&#xA;&#xA;public double getNoEstimateCounterAverage(){&#xA;    return noEstimateCounterAverage;&#xA;}&#xA;&#xA;public double getTotalEstimateCount(){&#xA;    return totalEstimateCount;&#xA;}&#xA;&#xA;public double getTotalEstimateCountAverage(){&#xA;    return totalEstimateCountAverage;&#xA;}&#xA;&#xA;/**&#xA; * We use the same evaluate function from the RecommenderEvaluator interface&#xA; * the trainingPercentage is used as the number of folds, so it can have&#xA; * values bigger than 0 to the number of folds.&#xA; */&#xA;@Override&#xA;public double evaluate(RecommenderBuilder recommenderBuilder,&#xA;        DataModelBuilder dataModelBuilder, DataModel dataModel,&#xA;        double trainingPercentage, double evaluationPercentage)&#xA;        throws TasteException {&#xA;    Preconditions.checkNotNull(recommenderBuilder);&#xA;    Preconditions.checkNotNull(dataModel);&#xA;    Preconditions.checkArgument(trainingPercentage &amp;gt;= 0.0,&#xA;            &quot;Invalid trainingPercentage: &quot; + trainingPercentage);&#xA;&#xA;    Preconditions.checkArgument(evaluationPercentage &amp;gt;= 0.0&#xA;            &amp;amp;&amp;amp; evaluationPercentage &amp;lt;= 1.0,&#xA;            &quot;Invalid evaluationPercentage: &quot; + evaluationPercentage);&#xA;&#xA;    log.info(&quot;Beginning evaluation using {} of {}&quot;, trainingPercentage,&#xA;            dataModel);&#xA;&#xA;    int numUsers = dataModel.getNumUsers();&#xA;&#xA;    // Get the number of folds&#xA;    int noFolds = (int) trainingPercentage;&#xA;&#xA;    // Initialize buckets for the number of folds&#xA;    List&amp;lt;FastByIDMap&amp;lt;PreferenceArray&amp;gt;&amp;gt; folds = new ArrayList&amp;lt;FastByIDMap&amp;lt;PreferenceArray&amp;gt;&amp;gt;();&#xA;    for (int i = 0; i &amp;lt; noFolds; i++) {&#xA;        folds.add(new FastByIDMap&amp;lt;PreferenceArray&amp;gt;(&#xA;                1 + (int) (i / noFolds * numUsers)));&#xA;    }&#xA;&#xA;    // Split the dataModel into K folds per user&#xA;    LongPrimitiveIterator it = dataModel.getUserIDs();&#xA;    while (it.hasNext()) {&#xA;        long userID = it.nextLong();&#xA;        if (random.nextDouble() &amp;lt; evaluationPercentage) {&#xA;            splitOneUsersPrefs2(noFolds, folds, userID, dataModel);&#xA;        }&#xA;    }&#xA;&#xA;    double result = Double.NaN;&#xA;    List&amp;lt;Double&amp;gt; intermediateResults = new ArrayList&amp;lt;&amp;gt;();&#xA;    List&amp;lt;Integer&amp;gt; unableToRecoomend = new ArrayList&amp;lt;&amp;gt;();&#xA;    List&amp;lt;Integer&amp;gt; averageEstimateCounterIntermediate = new ArrayList&amp;lt;&amp;gt;();&#xA;&#xA;    noEstimateCounterAverage = 0.0;&#xA;    totalEstimateCount = 0.0;&#xA;    totalEstimateCountAverage = 0.0;&#xA;    int totalEstimateCounter = 0;&#xA;&#xA;    // Rotate the folds. Each time only one is used for testing and the rest&#xA;    // k-1 folds are used for training&#xA;    for (int k = 0; k &amp;lt; noFolds; k++) {&#xA;        FastByIDMap&amp;lt;PreferenceArray&amp;gt; trainingPrefs = new FastByIDMap&amp;lt;PreferenceArray&amp;gt;(&#xA;                1 + (int) (evaluationPercentage * numUsers));&#xA;        FastByIDMap&amp;lt;PreferenceArray&amp;gt; testPrefs = new FastByIDMap&amp;lt;PreferenceArray&amp;gt;(&#xA;                1 + (int) (evaluationPercentage * numUsers));&#xA;&#xA;        for (int i = 0; i &amp;lt; folds.size(); i++) {&#xA;&#xA;            // The testing fold&#xA;            testPrefs = folds.get(k);&#xA;&#xA;            // Build the training set from the remaining folds&#xA;            if (i != k) {&#xA;                for (Map.Entry&amp;lt;Long, PreferenceArray&amp;gt; entry : folds.get(i)&#xA;                        .entrySet()) {&#xA;                    if (!trainingPrefs.containsKey(entry.getKey())) {&#xA;                        trainingPrefs.put(entry.getKey(), entry.getValue());&#xA;                    } else {&#xA;                        List&amp;lt;Preference&amp;gt; userPreferences = new ArrayList&amp;lt;Preference&amp;gt;();&#xA;                        PreferenceArray existingPrefs = trainingPrefs&#xA;                                .get(entry.getKey());&#xA;                        for (int j = 0; j &amp;lt; existingPrefs.length(); j++) {&#xA;                            userPreferences.add(existingPrefs.get(j));&#xA;                        }&#xA;&#xA;                        PreferenceArray newPrefs = entry.getValue();&#xA;                        for (int j = 0; j &amp;lt; newPrefs.length(); j++) {&#xA;                            userPreferences.add(newPrefs.get(j));&#xA;                        }&#xA;                        trainingPrefs.remove(entry.getKey());&#xA;                        trainingPrefs.put(entry.getKey(),&#xA;                                new GenericUserPreferenceArray(&#xA;                                        userPreferences));&#xA;&#xA;                    }&#xA;                }&#xA;            }&#xA;        }&#xA;&#xA;        DataModel trainingModel = dataModelBuilder == null ? new GenericDataModel(&#xA;                trainingPrefs) : dataModelBuilder&#xA;                .buildDataModel(trainingPrefs);&#xA;&#xA;        Recommender recommender = recommenderBuilder&#xA;                .buildRecommender(trainingModel);&#xA;&#xA;        Double[] retVal = getEvaluation(testPrefs, recommender);&#xA;        double intermediate = retVal[0];&#xA;        int noEstimateCounter = ((Double)retVal[1]).intValue();&#xA;        totalEstimateCounter += ((Double)retVal[2]).intValue();&#xA;        averageEstimateCounterIntermediate.add(((Double)retVal[2]).intValue());&#xA;&#xA;        log.info(&quot;Evaluation result from fold {} : {}&quot;, k, intermediate);&#xA;        log.info(&quot;Average Unable to recommend  for fold {} in: {} cases out of {}&quot;, k, noEstimateCounter, ((Double)retVal[2]).intValue());&#xA;        intermediateResults.add(intermediate);&#xA;        unableToRecoomend.add(noEstimateCounter);&#xA;&#xA;    }&#xA;&#xA;    double sum = 0;&#xA;    double noEstimateSum = 0;&#xA;    double totalEstimateSum = 0;&#xA;    // Sum the results in each fold&#xA;    for (int i = 0; i &amp;lt; intermediateResults.size(); i++) {&#xA;        if (!Double.isNaN(intermediateResults.get(i))) {&#xA;            sum += intermediateResults.get(i);&#xA;            noEstimateSum+=unableToRecoomend.get(i);&#xA;            totalEstimateSum+=averageEstimateCounterIntermediate.get(i);&#xA;        }&#xA;    }&#xA;&#xA;    if (sum &amp;gt; 0) {&#xA;        // Get an average for the folds&#xA;        result = sum / intermediateResults.size();&#xA;    }&#xA;&#xA;    double noEstimateCount = 0;&#xA;    if(noEstimateSum&amp;gt;0){&#xA;        noEstimateCount = noEstimateSum / unableToRecoomend.size();&#xA;    }&#xA;&#xA;    double avgEstimateCount = 0;&#xA;    if(totalEstimateSum&amp;gt;0){&#xA;        avgEstimateCount = totalEstimateSum / averageEstimateCounterIntermediate.size();&#xA;    }&#xA;&#xA;    log.info(&quot;Average Evaluation result: {} &quot;, result);&#xA;    log.info(&quot;Average Unable to recommend in: {} cases out of avg. {} cases or total {} &quot;, noEstimateCount, avgEstimateCount, totalEstimateCounter);&#xA;&#xA;    noEstimateCounterAverage = noEstimateCount;&#xA;    totalEstimateCount = totalEstimateCounter;&#xA;    totalEstimateCountAverage = avgEstimateCount;&#xA;    return result;&#xA;}&#xA;&#xA;/**&#xA; * Split the preference values for one user into K folds, randomly&#xA; * Generate random number until is not the same as the previously generated on&#xA; * in order to make sure that at least two buckets are populated.&#xA; * &#xA; * @param k&#xA; * @param folds&#xA; * @param userID&#xA; * @param dataModel&#xA; * @throws TasteException&#xA; */&#xA;private void splitOneUsersPrefs(int k,&#xA;        List&amp;lt;FastByIDMap&amp;lt;PreferenceArray&amp;gt;&amp;gt; folds, long userID,&#xA;        DataModel dataModel) throws TasteException {&#xA;&#xA;    List&amp;lt;List&amp;lt;Preference&amp;gt;&amp;gt; oneUserPrefs = Lists&#xA;            .newArrayListWithCapacity(k + 1);&#xA;    for (int i = 0; i &amp;lt; k; i++) {&#xA;        oneUserPrefs.add(null);&#xA;    }&#xA;&#xA;    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);&#xA;&#xA;    int size = prefs.length();&#xA;    int previousBucket = -1;&#xA;    Double rand = -2.0;&#xA;    for (int i = 0; i &amp;lt; size; i++) {&#xA;        Preference newPref = new GenericPreference(userID,&#xA;                prefs.getItemID(i), prefs.getValue(i));&#xA;        do {&#xA;            rand = random.nextDouble() * k * 10;&#xA;&#xA;            rand = (double) Math.floor(rand / 10);&#xA;            // System.out.println(&quot;inside Rand &quot;+rand);&#xA;&#xA;        } while (rand.intValue() == previousBucket);&#xA;        // System.out.println(&quot;outside rand &quot;+rand);&#xA;        if (oneUserPrefs.get(rand.intValue()) == null) {&#xA;            oneUserPrefs.set(rand.intValue(), new ArrayList&amp;lt;Preference&amp;gt;());&#xA;        }&#xA;        oneUserPrefs.get(rand.intValue()).add(newPref);&#xA;&#xA;        previousBucket = rand.intValue();&#xA;&#xA;    }&#xA;&#xA;    for (int i = 0; i &amp;lt; k; i++) {&#xA;        if (oneUserPrefs.get(i) != null) {&#xA;            folds.get(i).put(userID,&#xA;                    new GenericUserPreferenceArray(oneUserPrefs.get(i)));&#xA;        }&#xA;    }&#xA;&#xA;}&#xA;&#xA;/**&#xA; * Split the preference values for one user into K folds, by shuffling.&#xA; * First Shuffle the Preference array for the user. Then distribute the item-preference pairs&#xA; * starting from the first buckets to the k-th bucket, and then start from the beggining.&#xA; * &#xA; * @param k&#xA; * @param folds&#xA; * @param userID&#xA; * @param dataModel&#xA; * @throws TasteException&#xA; */&#xA;private void splitOneUsersPrefs2(int k, List&amp;lt;FastByIDMap&amp;lt;PreferenceArray&amp;gt;&amp;gt; folds, long userID, DataModel dataModel) throws TasteException {&#xA;&#xA;    List&amp;lt;List&amp;lt;Preference&amp;gt;&amp;gt; oneUserPrefs = Lists.newArrayListWithCapacity(k + 1);&#xA;    for (int i = 0; i &amp;lt; k; i++) {&#xA;        oneUserPrefs.add(null);&#xA;    }&#xA;&#xA;    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);&#xA;    int size = prefs.length();&#xA;&#xA;&#xA;    List&amp;lt;Preference&amp;gt; userPrefs = new ArrayList&amp;lt;&amp;gt;();&#xA;    Iterator&amp;lt;Preference&amp;gt; it = prefs.iterator();&#xA;    while (it.hasNext()) {&#xA;        userPrefs.add(it.next());&#xA;    }&#xA;&#xA;    // Shuffle the items&#xA;    Collections.shuffle(userPrefs);&#xA;&#xA;    int currentBucket = 0;&#xA;    for (int i = 0; i &amp;lt; size; i++) {&#xA;        if (currentBucket == k) {&#xA;            currentBucket = 0;&#xA;        }&#xA;&#xA;        Preference newPref = new GenericPreference(userID, userPrefs.get(i).getItemID(), userPrefs.get(i).getValue());&#xA;&#xA;        if (oneUserPrefs.get(currentBucket) == null) {&#xA;            oneUserPrefs.set(currentBucket, new ArrayList&amp;lt;Preference&amp;gt;());&#xA;        }&#xA;        oneUserPrefs.get(currentBucket).add(newPref);&#xA;        currentBucket++;&#xA;    }&#xA;&#xA;    for (int i = 0; i &amp;lt; k; i++) {&#xA;        if (oneUserPrefs.get(i) != null) {&#xA;            folds.get(i).put(userID, new GenericUserPreferenceArray(oneUserPrefs.get(i)));&#xA;        }&#xA;    }&#xA;&#xA;}&#xA;&#xA;private Double[] getEvaluation(FastByIDMap&amp;lt;PreferenceArray&amp;gt; testPrefs, Recommender recommender) throws TasteException {&#xA;    reset();&#xA;    Collection&amp;lt;Callable&amp;lt;Void&amp;gt;&amp;gt; estimateCallables = Lists.newArrayList();&#xA;    AtomicInteger noEstimateCounter = new AtomicInteger();&#xA;    AtomicInteger totalEstimateCounter = new AtomicInteger();&#xA;    for (Map.Entry&amp;lt;Long, PreferenceArray&amp;gt; entry : testPrefs.entrySet()) {&#xA;        estimateCallables.add(new PreferenceEstimateCallable(recommender, entry.getKey(), entry.getValue(), noEstimateCounter, totalEstimateCounter));&#xA;    }&#xA;    log.info(&quot;Beginning evaluation of {} users&quot;, estimateCallables.size());&#xA;    RunningAverageAndStdDev timing = new FullRunningAverageAndStdDev();&#xA;    execute(estimateCallables, noEstimateCounter, timing);&#xA;&#xA;&#xA;    Double[] retVal = new Double[3];&#xA;    retVal[0] = computeFinalEvaluation();&#xA;    retVal[1] = (double) noEstimateCounter.get();&#xA;    retVal[2] = (double) totalEstimateCounter.get();&#xA;    //retVal.put(computeFinalEvaluation(), noEstimateCounter.get());&#xA;    //return computeFinalEvaluation();&#xA;    return retVal;&#xA;}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The implementation can be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public class RMSRecommenderEvaluatorModified extends AbstractKFoldRecommenderEvaluator {&#xA;&#xA;private RunningAverage average;&#xA;&#xA;&#xA;@Override&#xA;protected void reset() {&#xA;    average = new FullRunningAverage();&#xA;}&#xA;&#xA;@Override&#xA;protected void processOneEstimate(float estimatedPreference, Preference realPref) {&#xA;    double diff = realPref.getValue() - estimatedPreference;&#xA;    average.addDatum(diff * diff);&#xA;}&#xA;&#xA;@Override&#xA;protected double computeFinalEvaluation() {&#xA;    return Math.sqrt(average.getAverage());&#xA;}&#xA;&#xA;@Override&#xA;public String toString() {&#xA;    return &quot;RMSRecommenderEvaluator&quot;;&#xA;}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/30207754/4993513&quot;&gt;Source&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-11-30T05:51:36.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9126" PostTypeId="2" ParentId="9038" CreationDate="2015-11-30T06:05:30.840" Score="11" Body="&lt;p&gt;First of all your explanation about the methods are right. The point is that Embedding algorithms are not to only visualize but basically reducing the dimentionality to cope with two main problems in Statistical Data Analysis, namely &lt;strong&gt;Curse of Dimentionaliy&lt;/strong&gt; and &lt;strong&gt;Low-Sample Size Problem&lt;/strong&gt; so that they are not supposed to depict physically understood features and they are not only &lt;em&gt;meaningful&lt;/em&gt; but also necessary for data analysis!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually the visualization is almost the last usage of embedding methods. Projecting high-dimensional data into a lower-dimension space helps to preserve the actual pair-wise distances (mainly Euclidean one) which get distorted in the high dimensions or capturing the most information embedded in the variance of different features.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2016-01-06T18:00:56.617" LastActivityDate="2016-01-06T18:00:56.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9128" PostTypeId="2" ParentId="9038" CreationDate="2015-11-30T11:30:14.440" Score="4" Body="&lt;p&gt;Based on the statements and the discussions, I think there is an important point to distinct. A transformation to a lower dimensional space may &lt;em&gt;reduce&lt;/em&gt; the information, which is something different from making the information &lt;em&gt;meaningless&lt;/em&gt;. Let me use a following analogy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observing (2D) pictures of our world (3D) is a usual practice. A visualization method provides only different “glasses” to see a high dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good thing to “trust” a visualization method is to understand the internals. My favourite example is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow&quot;&gt;MDS&lt;/a&gt;  . It is easy possible to implement this method at your own using some optimization tool (e.g. R &lt;em&gt;optim&lt;/em&gt;). So you can &lt;em&gt;see&lt;/em&gt; how the method words, you may &lt;em&gt;measure the error&lt;/em&gt; of the result etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end you get a picture preserving the similarity of the original data with some degree of precision. Not more, but not less.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-11-30T11:30:14.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9129" PostTypeId="1" CreationDate="2015-11-30T11:31:55.753" Score="1" ViewCount="236" Body="&lt;p&gt;Based on three datasets, I have produced the scatterplot below in Python:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/C6oMN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/C6oMN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to fit a line on each dataset, but when I check the metrics this is what I get:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Set 1 (red):     R squared=0.002, p-value=0.651 &lt;/li&gt;&#xA;&lt;li&gt;Set 2 (purple):  R squared=0.008, p-value=0.378&lt;/li&gt;&#xA;&lt;li&gt;Set 3 (blue):    R squared=0.001, p-value=0.714&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt; are such data sets impossible to fit? Is there any kind of data transformation I could apply, based on the scatterplot shape?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to predict from regression.&lt;/p&gt;&#xA;" OwnerUserId="13481" LastEditorUserId="13481" LastEditDate="2015-11-30T14:56:53.597" LastActivityDate="2020-07-15T18:14:30.943" Title="Regression: Scatterplot with low R squared and high p-values" Tags="|dataset|regression|" AnswerCount="0" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="9130" PostTypeId="2" ParentId="9038" CreationDate="2015-11-30T14:24:17.620" Score="11" Body="&lt;p&gt;Richard Hamming is attributed with the sentence: &quot;The purpose of computing is insight, not numbers.&quot;  In this 1973 &lt;a href=&quot;http://ww.w.lithoguru.com/scientist/statistics/Anscombe_Graphs%20in%20Statistical%20Analysis_1973.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;academic paper&lt;/a&gt; (see discussion in &lt;a href=&quot;https://stats.stackexchange.com/questions/141614/what-is-the-famous-data-set-that-looks-totally-different-but-has-similar-summary/180805&quot;&gt;What is the famous data set that looks totally different but has similar summary stats?&lt;/a&gt;), Francis Anscombe argues that &quot;graphs are essential to good statistical analysis.&quot; Anscombe's quartet is a long time favorite: same stats and regression, low dimension, yet very different behavior, regarding noise, outliers, dependancy. The projection of data in 11 dimensions onto two dimensions shown below is quite misleading: one has correlation and  dispersion, the second (bottom down) has  exact match, except one outlier. The third has clear relationship, but not linear. The fourth shows the variables are potentially not related, except for a threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Anscombe&amp;#39;s_quartet_3.svg/500px-Anscombe&amp;#39;s_quartet_3.svg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Anscombe&amp;#39;s_quartet_3.svg/500px-Anscombe&amp;#39;s_quartet_3.svg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the book &lt;a href=&quot;http://eu.wiley.com/WileyCDA/WileyTitle/productCd-EHEP002310.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multivariate Analysis for the Biobehavioral and Social Sciences&lt;/a&gt; by Bruce L. Brown &lt;em&gt;et al.&lt;/em&gt;, we can find:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In his 1990 work &quot;Drawing Things Together,&quot; Latour claims that the&#xA;  mindset of hard scientists is one of intense “obsession” with graphism&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whether limited to 3D space, up to &lt;a href=&quot;https://www.youtube.com/watch?v=OdnhKE95AqM&quot; rel=&quot;nofollow noreferrer&quot;&gt;six dimension plots&lt;/a&gt; (space, color, shape and time), or &lt;a href=&quot;http://www.tenthdimension.com/medialinks.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;even imagining the tenth dimension&lt;/a&gt;, humans have limited sights. Relationships between observable phenomena: not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, the curse of dimensions is assorted with even low dimension paradoxes, to give a few: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/149936/why-is-the-curse-of-dimensionality-also-called-the-empty-space-phenomenon&quot;&gt;Why is the curse of dimensionality also called the empty space phenomenon?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.slideshare.net/NikhilSharma6/curse-of-dimensionality&quot; rel=&quot;nofollow noreferrer&quot;&gt;Curse of dimensionality&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/29627/euclidean-distance-is-usually-not-good-for-sparse-data&quot;&gt;Why is Euclidean distance not a good metric in high dimensions?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://sbjoshi.wordpress.com/2013/03/27/paradox-about-high-dimensional-spheres/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Paradox about high-dimensional spheres!&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bit-player.org/2011/the-n-ball-game&quot; rel=&quot;nofollow noreferrer&quot;&gt;The n-ball game&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Even if all norms are equivalent in finite dimensions, relationships between variables might be misleading. This is one reason for preserving distances from one space to another. Such concepts are at of the heart of lower dimension embeddings for signals (such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Compressed_sensing&quot; rel=&quot;nofollow noreferrer&quot;&gt;compressive sensing&lt;/a&gt; and the &lt;a href=&quot;https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma&quot; rel=&quot;nofollow noreferrer&quot;&gt;Johnson-Lindenstauss lemma&lt;/a&gt; concerning low-distortion embeddings of points from high-dimensional into low-dimensional Euclidean space) or features (&lt;a href=&quot;http://www.di.ens.fr/data/scattering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scattering transforms&lt;/a&gt; for classifications).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So visualization is another help in getting insights in the data, and it goes hand in hand with calculations, including dimension reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last example: put touching   $n$-spheres in an $n$-cube (the bubble inside the box, taken from &lt;a href=&quot;https://www.quora.com/Do-good-mathematicians-visualize-everything-even-algebra&quot; rel=&quot;nofollow noreferrer&quot;&gt;Do good mathematicians visualize everything (even algebra)?&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t57Ls.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t57Ls.png&quot; alt=&quot;Pizza box paradox&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In two dimensions, the  center blue  ball is small. In 3D too. But very quickly, the center ball grows and its radius exceeds that of the cube. This insight is vital n clustering, for instance.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-12-05T12:58:55.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9131" PostTypeId="1" CreationDate="2015-11-30T14:32:56.113" Score="4" ViewCount="221" Body="&lt;p&gt;I have 623 observations with one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal), defined based on researcher experience and literature review.&#xA;I considered planning to do several regression analysis to estimate the dependent variable and study the predictive factors (if they are positive, negative, and their magnitude) on it. Data are provided for 10 years. Since the latest observations are more important, I’m interested in using weighted observations. How can I approach this problem and validate my approach?&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="13413" LastEditDate="2015-11-30T22:11:07.677" LastActivityDate="2019-12-18T07:01:40.277" Title="How can I weigh observations differently that were provided for a time horizon?" Tags="|data-mining|dataset|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9133" PostTypeId="2" ParentId="5438" CreationDate="2015-11-30T15:59:29.730" Score="0" Body="&lt;p&gt;&quot;I now declare you man and wife&quot;, an example of utterance by means of which the speaker performs a particular act, is an instance of a &lt;strong&gt;performative&lt;/strong&gt; act, which changes the thing being modeled and thus changes the concept. Reflexive, iterative, recursive, retroactive, &lt;a href=&quot;http://dictionary.reference.com/browse/self-generating&quot; rel=&quot;nofollow&quot;&gt;self-generating&lt;/a&gt;, are related terms.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2015-11-30T15:59:29.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9134" PostTypeId="1" AcceptedAnswerId="23339" CreationDate="2015-11-30T16:08:12.507" Score="6" ViewCount="6805" Body="&lt;p&gt;I'm trying to fully understand the gradient boosting (GB) method. I've read some wiki pages and papers about it, but it would really help me to see a full simple example carried out step-by-step. Can anyone provide one for me, or give me a link to such an example? Straightforward source code without tricky optimizations will also meet my needs.&lt;/p&gt;&#xA;" OwnerUserId="14434" LastEditorUserId="14434" LastEditDate="2015-11-30T19:24:45.387" LastActivityDate="2018-09-29T22:17:22.910" Title="Gradient boosting algorithm example" Tags="|machine-learning|classification|algorithms|xgboost|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9135" PostTypeId="2" ParentId="9134" CreationDate="2015-11-30T17:02:03.633" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is the repository where the xgboost package resides. That is from where the xgboost library for major languages like Julia, Java, R, etc are forked from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/tree/master/python-package&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is the Python implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Walking through the source code (as instructed in the &quot;How to contribute&quot;) docs would help you understand the intuition behind it.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-01T11:53:31.563" LastActivityDate="2015-12-01T11:53:31.563" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9136" PostTypeId="1" AcceptedAnswerId="9149" CreationDate="2015-11-30T17:45:34.930" Score="1" ViewCount="436" Body="&lt;p&gt;I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...&lt;/p&gt;&#xA;&lt;p&gt;Author provided: The goal of my ...&lt;/p&gt;&#xA;&lt;p&gt;The goal of my ... END OF TRANSCRIPT&lt;/p&gt;&#xA;&lt;p&gt;The goal of my ... END, SPONSORED BY COMPANY XYZ&lt;/p&gt;&#xA;&lt;p&gt;The goal of my ... SPONSORED: COMPANY XYZ, All rights reserved, date: 10/21&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?&lt;/p&gt;&#xA;" OwnerUserId="12963" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-12-01T17:24:11.440" Title="Detecting boilerplate in text samples" Tags="|text-mining|data-cleaning|data-wrangling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9137" PostTypeId="2" ParentId="9131" CreationDate="2015-12-01T02:41:58.287" Score="0" Body="&lt;p&gt;Converting the date into an integer timestamp will put a higher weight on the more recent observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you write :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Since the latest observations are more important, I’m interested in using weighted observations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Do you mean that you know already that the date will be a predictive factor in your analysis, or that you want to artificially make this variable a predictive factor for the regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is the former, then the integer conversion above could do the trick.&#xA;If it is the latter, you need to combine the date with your target variable (by  multiplication for example).&lt;/p&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-12-01T02:41:58.287" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9138" PostTypeId="1" CreationDate="2015-12-01T04:53:16.017" Score="2" ViewCount="193" Body="&lt;p&gt;This is the corpus&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abcdef&#xA;abcd&#xA;xabcd&#xA;b&#xA;bcef&#xA;ef&#xA;aef&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;what I want to extract is &lt;code&gt;abcd&lt;/code&gt;, &lt;code&gt;ef&lt;/code&gt;, then I try to count all n-grams:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abcdef 1&#xA;abcd 3&#xA;abc 3&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is how to distinguish &lt;code&gt;abcd&lt;/code&gt; with &lt;code&gt;abcdef&lt;/code&gt; and &lt;code&gt;abc&lt;/code&gt;, since &lt;code&gt;abcdef&lt;/code&gt; just appear once, and &lt;code&gt;abc&lt;/code&gt; is the sub-string of 'abcd'. Of course, I can filter with this two conditions, but just want to know any existing algorithm target on solving this problem.&lt;/p&gt;&#xA;" OwnerUserId="14445" LastEditorUserId="14445" LastEditDate="2015-12-01T05:57:43.327" LastActivityDate="2016-02-02T04:34:30.600" Title="how to extract high frequency phrase" Tags="|data-mining|dataset|statistics|text-mining|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9140" PostTypeId="1" AcceptedAnswerId="9143" CreationDate="2015-12-01T06:13:35.210" Score="1" ViewCount="147" Body="&lt;p&gt;I have a dataset of houses like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;HouseID  Latitude Longitude PriceIndex&#xA;  1          1.4     103.120    1.21&#xA;  2          1.42    103.112    2.01 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to find houses which are similar to each other both on the basis of their position as well as their price index.[Also would need to Rank in order of similarities, given one house] I tried using hclust package in R  and was able to extract 9 classes. However the groups don't seem to have any interpretable similarities (for example the points are spread all across the city etc). I haven't done clustering based projects before so any help in the right direction will be helpful. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Removing the price index column from the clustering data-set actually clusters spatially. But adding the price shows only price-based clustering&lt;/p&gt;&#xA;" OwnerUserId="14427" LastEditorUserId="14427" LastEditDate="2015-12-01T08:06:44.253" LastActivityDate="2015-12-01T09:26:11.457" Title="How to cluster Houses on the basis of similarity of features+location?" Tags="|r|clustering|similarity|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9141" PostTypeId="2" ParentId="9086" CreationDate="2015-12-01T07:33:56.700" Score="12" Body="&lt;p&gt;I don't think you necessarily need to convert the individual log entries into vectors for use in an algorithm.  I would guess that what you are interested in is a sequence of log entries, which represent a series of events, ordered in time, which together make up a series of 'cases'.  Here the relationship between a series of collected log entries is important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the case then you might want to consider using &lt;a href=&quot;http://www.processmining.org/start&quot; rel=&quot;noreferrer&quot;&gt;Process Mining&lt;/a&gt; techniques.  This allows you to build models of your process (the use of your application) and determine patterns of process steps, along with errors and rework steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a good introduction course on Coursera, &lt;a href=&quot;https://www.coursera.org/course/procmin&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.  There are even some developed, commercial packages like 'disco' to help you with the analysis and visualisation&lt;/p&gt;&#xA;" OwnerUserId="14450" LastActivityDate="2015-12-01T07:33:56.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9143" PostTypeId="2" ParentId="9140" CreationDate="2015-12-01T09:26:11.457" Score="1" Body="&lt;p&gt;Check the ranges of your dimensions and consider scaling if you see a large difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would interpret your described  behaviour due to much larger range if the index that the two other dimensions.&#xA;See also the &lt;a href=&quot;https://datascience.stackexchange.com/questions/6715/is-it-necessary-to-standardize-your-data-before-clustering/6722&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-01T09:26:11.457" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9147" PostTypeId="2" ParentId="9138" CreationDate="2015-12-01T14:13:41.883" Score="1" Body="&lt;p&gt;This uses python and NLTK&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import nltk&#xA;&#xA;text = &quot;&quot;&quot;abcdef&#xA;abcd&#xA;xabcd&#xA;b&#xA;bcef&#xA;ef&#xA;aef&#xA;&quot;&quot;&quot;&#xA;&#xA;def tokens(lines):&#xA;    for line in lines:&#xA;        for n in range(1, len(line)):&#xA;            ts = nltk.ngrams(line, n)&#xA;            for t in ts: yield str.join('', t) &#xA;        yield line&#xA;&#xA;ts = tokens(text.splitlines())&#xA;freq = nltk.FreqDist(ts)&#xA;print(freq.most_common())&#xA;# [('b', 5), ('c', 4), ('ef', 4), ('bc', 4), ('e', 4)...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are other solutions, of course, but I think this meets your criteria. You may need to optimize if your lines are huge as this will create every possible n-gram.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-01T14:13:41.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9148" PostTypeId="2" ParentId="9074" CreationDate="2015-12-01T16:14:33.330" Score="2" Body="&lt;p&gt;&lt;strong&gt;Totally disagree with @Derek Janni.&lt;/strong&gt; Be careful about notation however you should not get lost in terminology. Those papers you mentioned used the term &quot;labeling&quot; literally but in Machine Learning/Data Mining community, &lt;strong&gt;labeling is the process of preparing data for supervised learning (classification)! It has nothing to do with the ML task!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those papers used the term to show that after supervised learning they can recognize different labels of different objects so they used the term labeling but you probably can not find in any literature that they use these two terms as synonyms.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-12-01T16:14:33.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9149" PostTypeId="2" ParentId="9136" CreationDate="2015-12-01T17:24:11.440" Score="1" Body="&lt;p&gt;This might get you started. Phrase length is determined by the range() function. Basically this tokenizes and creates n-grams. Then it counts each token. Tokens with a high mean over all documents (occurs often across documents) is printed out in the last line.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;import numpy as np&#xA;import nltk&#xA;&#xA;text = &quot;&quot;&quot;DESCRIPTION PROVIDED BY AUTHOR: The goal of my a...&#xA;Author provided: The goal of my b...&#xA;The goal of my c... END OF TRANSCRIPT&#xA;The goal of my d... END SPONSORED BY COMPANY XYZ&#xA;The goal of my e... SPONSORED: COMPANY XYZ All rights reserved date: 10/21&#xA;&quot;&quot;&quot;&#xA;&#xA;def todocuments(lines):&#xA;    for line in lines:&#xA;        words = line.lower().split(' ')&#xA;        doc = &quot;&quot;&#xA;        for n in range(3, 6):&#xA;            ts = nltk.ngrams(words, n)&#xA;            for t in ts: doc = doc + &quot; &quot; + str.join('_', t) &#xA;        yield doc&#xA;&#xA;cv = CountVectorizer(min_df=.5)&#xA;&#xA;fit = cv.fit_transform(todocuments(text.splitlines()))&#xA;vocab_idx = {b: a for a, b in cv.vocabulary_.items()}&#xA;&#xA;means = fit.mean(axis=0)&#xA;arr = np.squeeze(np.asarray(means))&#xA;[vocab_idx[idx] for idx in np.where(arr &amp;gt; .95)[0]]&#xA;# ['goal_of_my', 'the_goal_of', 'the_goal_of_my']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-01T17:24:11.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9150" PostTypeId="2" ParentId="9109" CreationDate="2015-12-01T20:08:28.413" Score="2" Body="&lt;p&gt;Having used genetic algorithms extensively in both professional projects and for my own entertainment, I think it is best to just code your own set of functions/classes for the specific project.  As you have seen, there are quite a few well-intended open-source GA packages out there. However, many of the fine details and tweaks necessary for a good GA are problem-specific.  It is also more convenient to work directly with your existing data structures, rather than adapt to someone else's worldview.  In other words, cost-benefit does not seem to work out when applying someone else's GA package.  You need a function for creating a random instance, a function for checking it is valid, a function for scoring it, and a mating function.  That's pretty much it!     &lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2015-12-01T20:16:36.043" LastActivityDate="2015-12-01T20:16:36.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9151" PostTypeId="2" ParentId="9117" CreationDate="2015-12-01T20:08:57.823" Score="1" Body="&lt;p&gt;For the 200 positive samples, are there many different &quot;types&quot;? In other words, will future positive samples look nothing like the current positive samples? If this is the case, you may try anomaly detection in machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a lecture by Andrew Ng in Coursera, if you have small number of positive samples (e.g., 20), large number of negative samples (e.g., 10000), and it's hard for any algorithm to learn from the positive samples, you should use anomaly detection, otherwise, you can use supervised learning.  &lt;/p&gt;&#xA;" OwnerUserId="14463" LastActivityDate="2015-12-01T20:08:57.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9152" PostTypeId="2" ParentId="9038" CreationDate="2015-12-01T21:07:45.980" Score="4" Body="&lt;p&gt;Sometimes, it is meaningful to visualize high dimensional data since it may tell us physics.&lt;/p&gt;&#xA;&lt;p&gt;There is at least one example in astrophysics where you project your data down to principal components generated by PCA and those principal components correspond to much physical insight about the galaxies. For detail, see the &lt;a href=&quot;http://iopscience.iop.org/article/10.1086/425626/pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Here is the basic idea. The authors apply PCA to many spectra (e.g., 10,000) from a telescope. Each spectrum has ~1000 attributes. Since this data set has large dimensions, it's difficult to visualize it. However, the first 4 components from PCA reveal much physics about the spectra (see sections 4.1-4.4 in the paper above).&lt;/p&gt;&#xA;" OwnerUserId="14463" LastEditorUserId="98307" LastEditDate="2020-08-20T18:49:05.380" LastActivityDate="2020-08-20T18:49:05.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9156" PostTypeId="1" CreationDate="2015-12-01T21:26:44.943" Score="2" ViewCount="160" Body="&lt;p&gt;I’m trying to create a model to forecast demand/expenditure for an individual based on historical data of many individuals over time, but I’m having trouble finding examples of this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a simple example of what I’m trying to do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let’s say there’s an ice-cream store that children in the surrounding area visit every week. We’ve got the history of all purchases at the store and details of who made the purchase, so we can track the purchase history of each individual. We also have details about each individual, e.g. age, gender, school, no. of siblings etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can assume that a child will buy from the store once a week until they ‘leave’ for good and stop purchasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is create purchase forecasts for each child over a few weeks. The bit I’m struggling with is where a new child, say Timmy, has only been purchasing for a few weeks and so doesn't have enough of a history to forecast based solely on his own data (but we do have his details). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My assumption is that demand can be better predicted by taking a child's details into account, so I'd like to find a group in the historical data that best represents Timmy, and leverage their data in some way when creating Timmy's forecast. Are there any standard methods/practices for this type of forecasting? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice or guidance would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="14472" LastEditorUserId="924" LastEditDate="2015-12-02T07:26:35.797" LastActivityDate="2016-01-31T08:47:10.373" Title="Forecasting an individual based on a representative group" Tags="|machine-learning|predictive-modeling|forecast|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9157" PostTypeId="2" ParentId="9038" CreationDate="2015-12-01T22:05:41.837" Score="14" Body="&lt;p&gt;I take Natural Language Processing as an example because that's the field that I have more experience in so I encourage others to share their insights in other fields like in Computer Vision, Biostatistics, time series, etc. I'm sure in those fields there are similar examples.&lt;/p&gt;&#xA;&lt;p&gt;I agree that sometimes model visualizations can be meaningless but I think the main purpose of visualizations of this kind are to help us check if the model actually relates to human intuition or some other (non-computational) model. Additionally, Exploratory Data Analysis can be performed on the data.&lt;/p&gt;&#xA;&lt;p&gt;Let's assume we have a word embedding model built from Wikipedia's corpus using &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gensim&lt;/a&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = gensim.models.Word2Vec(sentences, min_count=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;We would then have a 100 dimension vector for each word represented in that corpus that's present at least twice. So if we wanted to visualize these words we would have to reduce them to 2 or 3 dimensions using the t-sne algorithm. Here is where very interesting characteristics arise.&lt;/p&gt;&#xA;&lt;p&gt;Take the example:&lt;/p&gt;&#xA;&lt;p&gt;vector(&amp;quot;king&amp;quot;) + vector(&amp;quot;man&amp;quot;) - vector(&amp;quot;woman&amp;quot;) = vector(&amp;quot;queen&amp;quot;)&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/OQpTJ.gif&quot; alt=&quot;http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a-thousand-vectors/&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here each direction encode certain semantic features. The same can be done in 3d&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZcNDo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZcNDo.png&quot; alt=&quot;https://www.tensorflow.org/versions/master/images/linear-relationships.png&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://www.tensorflow.org/versions/master/images/linear-relationships.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow.org&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;&lt;p&gt;See how in this example past tense is located in a certain position respective to its participle. The same for gender. Same with countries and capitals.&lt;/p&gt;&#xA;&lt;p&gt;In the word embedding world, older and more naive models, didn't have this property.&lt;/p&gt;&#xA;&lt;p&gt;See this Stanford lecture for more details.&#xA;&lt;a href=&quot;https://www.youtube.com/watch?v=T8tQZChniMk&quot; rel=&quot;nofollow noreferrer&quot;&gt;Simple Word Vector representations: word2vec, GloVe&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;They only were limited to clustering similar words together without regard for semantics (gender or verb tense weren't encoded as directions). Unsurprisingly models which have a semantic encoding as directions in lower dimensions are more accurate. And more importantly, they can be used to explore each data point in a more appropriate way.&lt;/p&gt;&#xA;&lt;p&gt;In this particular case, I don't think t-SNE is used to aid classification per se, it's more like a sanity check for your model and sometimes to find insight in the particular corpus you are using. As for the problem of the vectors not being in original feature space anymore. Richard Socher explains in the lecture (link above) that low dimensional vectors share statistical distributions with its own larger representation as well as other statistical properties which make plausible visually analyse in lower dimensions embedding vectors.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Additional resources &amp;amp; Image Sources:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://multithreaded.stitchfix.com/blog/2015/03/11/word-is-worth-a-thousand-vectors/&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Word is Worth a Thousand Vectors&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec/index.html#motivation_why_learn_word_embeddings%3F&quot; rel=&quot;nofollow noreferrer&quot;&gt;Motivation Why Learn Word Embeddings&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="98307" LastEditDate="2020-08-20T18:48:21.690" LastActivityDate="2020-08-20T18:48:21.690" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9158" PostTypeId="2" ParentId="9156" CreationDate="2015-12-01T23:35:37.437" Score="1" Body="&lt;p&gt;I think you are doing a supervised classification problem.  You have labelled data — previous customers, which you could label as different groups (e.g. high spend / low spend) along with predictors about the customer — their details (e.g. age).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could think of each of these customer details representing dimensions on a plot, wth your task being to find out where different groups of customers are clustered along these dimensions.  Once you know that you can predict what group a new customer will likely belong to, based on their details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many methods for solving this sort of problem, including but not limited to support &lt;em&gt;vector machines&lt;/em&gt;, &lt;em&gt;gradient descent&lt;/em&gt; and &lt;em&gt;nearest neighbour&lt;/em&gt;.  Which one you choose will depend on the amount and sort of data you have.  A really good guide if you are starting out is &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html&quot; rel=&quot;nofollow&quot;&gt;this flowchart&lt;/a&gt; in the SciKit learn documentation, which isn't exhaustive, but gives a useful overview.&lt;/p&gt;&#xA;" OwnerUserId="14450" LastActivityDate="2015-12-01T23:35:37.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9159" PostTypeId="1" CreationDate="2015-12-02T01:06:28.243" Score="16" ViewCount="78516" Body="&lt;p&gt;I am working on a project and I am having difficulty in deciding which algorithm to choose for &lt;code&gt;regression&lt;/code&gt;. I want to know under what conditions should one choose a &lt;code&gt;linear regression&lt;/code&gt; or &lt;code&gt;Decision Tree regression&lt;/code&gt; or &lt;code&gt;Random Forest regression&lt;/code&gt;? Are there any specific characteristics of the data that would make the decision to go towards a specific algorithm amongst the tree mentioned above? What are those characteristics that I should look in my dataset to make the decision? And are there some reasons that would make one choose a &lt;code&gt;decision tree&lt;/code&gt; or &lt;code&gt;random forest&lt;/code&gt; algorithm even if the same correctness can be achieved by &lt;code&gt;linear regression&lt;/code&gt;? &lt;/p&gt;&#xA;" OwnerUserId="14384" LastActivityDate="2019-02-23T04:41:04.703" Title="When to choose linear regression or Decision Tree or Random Forest regression?" Tags="|machine-learning|algorithms|random-forest|linear-regression|decision-trees|" AnswerCount="2" CommentCount="1" ClosedDate="2015-12-02T14:44:54.173" ContentLicense="CC BY-SA 3.0" />
  <row Id="9161" PostTypeId="2" ParentId="8278" CreationDate="2015-12-02T02:28:11.103" Score="5" Body="&lt;p&gt;There are methods like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;noreferrer&quot;&gt;Kullback-Leibler divergence model&lt;/a&gt;, the &lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/eda/section3/eda35d.htm&quot; rel=&quot;noreferrer&quot;&gt;Wald-Wolfowitz test&lt;/a&gt; for detecting non-randomness and covariance shift. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple test for quick analysis of covariance test would be to build a machine learning model, where the model is repeatedly tested with inputting training data and the production data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case, the model can make out the difference between the training and production datasets then it can be a sign of covariance shift.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-02T02:28:11.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9162" PostTypeId="2" ParentId="9159" CreationDate="2015-12-02T06:41:14.460" Score="4" Body="&lt;p&gt;As far as I know, there is not a rule to say which algorithm works for which dataset. Just make sure your dataset and variables of interest fulfill the pre-assumptions of running each algorithm and give it a try. For example, linear regression has some pre-assumptions such as normality of resuduals, homoscedasticity (the variability in the response variable is the same at all levels of the explanatory variable) and so on. Just check these for your variables and give the algorithm a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a point and click software to see the results without getting involved in the code and parameter setting. If you are an R user, rattle package will be a very useful tool at this stage. You do your job in point and click mode and you have access to the code behind it.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-12-02T06:41:14.460" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9163" PostTypeId="2" ParentId="9159" CreationDate="2015-12-02T06:50:55.213" Score="17" Body="&lt;p&gt;Let me explain it using some examples for clear intuition:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When do you use linear regression vs Decision Trees?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this case, you can use the decision trees, which do a better job at capturing the non-linearity in the data by dividing the space into smaller sub-spaces depending on the questions asked.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When do you use Random Forest vs Decision Trees?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I guess the &lt;a href=&quot;https://www.quora.com/Random-Forests/How-do-random-forests-work-in-laymans-terms/answer/Edwin-Chen-1&quot; rel=&quot;noreferrer&quot;&gt;Quora answer&lt;/a&gt; here would do a better job than me, at explaining the difference between them and their applications. Let me quote that for you:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose you're very indecisive, so whenever you want to watch a movie,&#xA;  you ask your friend Willow if she thinks you'll like it. In order to&#xA;  answer, Willow first needs to figure out what movies you like, so you&#xA;  give her a bunch of movies and tell her whether you liked each one or&#xA;  not (i.e., you give her a labeled training set). Then, when you ask&#xA;  her if she thinks you'll like movie X or not, she plays a 20&#xA;  questions-like game with IMDB, asking questions like &quot;Is X a romantic&#xA;  movie?&quot;, &quot;Does Johnny Depp star in X?&quot;, and so on. She asks more&#xA;  informative questions first (i.e., she maximizes the information gain&#xA;  of each question), and gives you a yes/no answer at the end.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Thus, Willow is a decision tree for your movie preferences.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But Willow is only human, so she doesn't always generalize your&#xA;  preferences very well (i.e., she overfits). In order to get more&#xA;  accurate recommendations, you'd like to ask a bunch of your friends,&#xA;  and watch movie X if most of them say they think you'll like it. That&#xA;  is, instead of asking only Willow, you want to ask Woody, Apple, and&#xA;  Cartman as well, and they vote on whether you'll like a movie (i.e.,&#xA;  you build an ensemble classifier, aka a forest in this case).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Now you don't want each of your friends to do the same thing and give&#xA;  you the same answer, so you first give each of them slightly different&#xA;  data. After all, you're not absolutely sure of your preferences&#xA;  yourself -- you told Willow you loved Titanic, but maybe you were just&#xA;  happy that day because it was your birthday, so maybe some of your&#xA;  friends shouldn't use the fact that you liked Titanic in making their&#xA;  recommendations. Or maybe you told her you loved Cinderella, but&#xA;  actually you &lt;em&gt;really really&lt;/em&gt; loved it, so some of your friends should&#xA;  give Cinderella more weight. So instead of giving your friends the&#xA;  same data you gave Willow, you give them slightly perturbed versions.&#xA;  You don't change your love/hate decisions, you just say you love/hate&#xA;  some movies a little more or less (you give each of your friends a&#xA;  bootstrapped version of your original training data). For example,&#xA;  whereas you told Willow that you liked Black Swan and Harry Potter and&#xA;  disliked Avatar, you tell Woody that you liked Black Swan so much you&#xA;  watched it twice, you disliked Avatar, and don't mention Harry Potter&#xA;  at all.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;By using this ensemble, you hope that while each of your friends gives&#xA;  somewhat idiosyncratic recommendations (Willow thinks you like vampire&#xA;  movies more than you do, Woody thinks you like Pixar movies, and&#xA;  Cartman thinks you just hate everything), the errors get canceled out&#xA;  in the majority. Thus, your friends now form a bagged (bootstrap&#xA;  aggregated) forest of your movie preferences.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;There's still one problem with your data, however. While you loved&#xA;  both Titanic and Inception, it wasn't because you like movies that&#xA;  star Leonardio DiCaprio. Maybe you liked both movies for other&#xA;  reasons. Thus, you don't want your friends to all base their&#xA;  recommendations on whether Leo is in a movie or not. So when each&#xA;  friend asks IMDB a question, only a random subset of the possible&#xA;  questions is allowed (i.e., when you're building a decision tree, at&#xA;  each node you use some randomness in selecting the attribute to split&#xA;  on, say by randomly selecting an attribute or by selecting an&#xA;  attribute from a random subset). This means your friends aren't&#xA;  allowed to ask whether Leonardo DiCaprio is in the movie whenever they&#xA;  want. So whereas previously you injected randomness at the data level,&#xA;  by perturbing your movie preferences slightly, now you're injecting&#xA;  randomness at the model level, by making your friends ask different&#xA;  questions at different times.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;And so your friends now form a random forest.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2019-02-23T04:41:04.703" LastActivityDate="2019-02-23T04:41:04.703" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9164" PostTypeId="2" ParentId="9138" CreationDate="2015-12-02T07:12:16.303" Score="1" Body="&lt;p&gt;I'm not sure what is the goal of the procedure you are asking about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be interested in &lt;a href=&quot;https://en.wikipedia.org/wiki/Huffman_coding&quot; rel=&quot;nofollow&quot;&gt;Huffman coding&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Huffman codes are used for text compression.&#xA;They identify high frequency sequences and assign to them short codes, getting a a reduction in the overall text size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tree constructed for the code might help you with what you look for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an extra bonus, it is a classical algorithm so you will be able to find many implementations for it.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-12-02T07:12:16.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9165" PostTypeId="1" AcceptedAnswerId="9166" CreationDate="2015-12-02T08:50:48.877" Score="2" ViewCount="50" Body="&lt;p&gt;I have three datasets, let's call them &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y1&lt;/code&gt; and &lt;code&gt;Y2&lt;/code&gt;. A scatterplot is produced out of them, with &lt;code&gt;Y1&lt;/code&gt; and &lt;code&gt;Y2&lt;/code&gt; sharing them same &lt;code&gt;X&lt;/code&gt; dataset (or support).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt; if the two regression lines are different in both slope and intercept, is there a way to evaluate if the &lt;code&gt;X&lt;/code&gt; dataset has more influence on &lt;code&gt;Y1&lt;/code&gt; or &lt;code&gt;Y2&lt;/code&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the image below, this is to say - &lt;strong&gt;which Y dataset is more influenced by the X dataset?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Blue slope (Y1): -112&lt;/li&gt;&#xA;&lt;li&gt;Red slope (Y2): -90&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is visible that an increase in X produces a decrease in both Y1 and Y2. My question could be interpreted as follows: &lt;strong&gt;which Y dataset decreases the most, given the increase in X? Is the slope everything I need?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/AaHEa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AaHEa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13481" LastEditorUserId="13481" LastEditDate="2015-12-02T08:59:26.327" LastActivityDate="2015-12-02T09:01:48.113" Title="Regression: how to interpret different linear relations?" Tags="|predictive-modeling|regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9166" PostTypeId="2" ParentId="9165" CreationDate="2015-12-02T09:01:48.113" Score="2" Body="&lt;p&gt;What you are looking for is the Analysis of Covariance (&lt;a href=&quot;https://en.wikipedia.org/wiki/Analysis_of_covariance&quot; rel=&quot;nofollow&quot;&gt;ANCOVA&lt;/a&gt;) analysis, which is used to compare two or more regression lines by testing the effect of a categorical factor on a dependent variable (y-var) while controlling for the effect of a continuous co-variable (x-var).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://r-eco-evo.blogspot.in/2011/08/comparing-two-regression-slopes-by.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is an example for carrying out the ANCOVA analysis using R.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-02T09:01:48.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9167" PostTypeId="1" CreationDate="2015-12-02T11:38:06.480" Score="4" ViewCount="22359" Body="&lt;p&gt;I am working on &lt;code&gt;Decision Tree&lt;/code&gt; algorithm and at the end I calculate &lt;code&gt;RMSE&lt;/code&gt; value based on actual labels and predicted values (for regression). Now what I am having difficulty in is in understanding the significance of the &lt;code&gt;RMSE&lt;/code&gt; value that I get. I know that the lower the &lt;code&gt;RMSE&lt;/code&gt; better is the performance of the model but what &lt;code&gt;RMSE&lt;/code&gt; value is considered reasonable low or high? Suppose my &lt;code&gt;RMSE&lt;/code&gt; value comes as &lt;code&gt;20&lt;/code&gt; so does that mean the performance of my model is &lt;code&gt;80%&lt;/code&gt; (means 80% of predictions are correct)?&lt;/p&gt;&#xA;" OwnerUserId="14384" LastActivityDate="2018-12-13T09:41:13.043" Title="What does RMSE points about performance of a model in machine learning?" Tags="|machine-learning|performance|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9168" PostTypeId="2" ParentId="9167" CreationDate="2015-12-02T12:19:27.893" Score="3" Body="&lt;p&gt;The root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model and the values actually observed. The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you've badly over fit the data.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2015-12-02T12:19:27.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9169" PostTypeId="1" AcceptedAnswerId="9174" CreationDate="2015-12-02T13:56:10.107" Score="3" ViewCount="9158" Body="&lt;p&gt;When building a model in Machine Learning, it's more than common to have several &quot;parameters&quot; (I'm thinking of real parameter like the step of gradient descent, or things like features) to tune. We validate these parameters on a validating set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: what is the best way of tuning these multiple parameters? For example, let say we have 3 parameters A, B and C that take 3 values each:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A = [ A1, A2, A3 ]&lt;/li&gt;&#xA;&lt;li&gt;B = [ B1, B2, B3 ]&lt;/li&gt;&#xA;&lt;li&gt;C = [ C1, C2, C3 ]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Two methods come to my mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vary all the parameters at the same time and test different combinations randomly, such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Test1 = [A1,B1,C1]&lt;/li&gt;&#xA;&lt;li&gt;Test2 = [A2,B2,C2]&lt;/li&gt;&#xA;&lt;li&gt;Test3 = [A3,B3,C3]&lt;/li&gt;&#xA;&lt;li&gt;Test4 = [A1,B2,C3]&lt;/li&gt;&#xA;&lt;li&gt;Test5= [A3,B1,C2]&lt;/li&gt;&#xA;&lt;li&gt;etc..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Method 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fix all the parameters except one:&#xA; - TestA1 = [A1,B1,C1]&#xA; - TestA2 = [A2,B1,C1]&#xA; - TestA3 = [A3,B1,C1]&#xA;In that way, we can find the best value for parameter A, then we fix this value and use it to find the best value for B, and finally the best for C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems more logical to me to use the method 2 which seems more organized. But may be we will miss a combination which can be found only in method 1 that doesn’t appear in method 2, such as [A1,B2,C3] for example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which method is the best? Is there another method more accurate for tuning multiple parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards.&lt;/p&gt;&#xA;" OwnerUserId="14200" LastEditorUserId="13023" LastEditDate="2015-12-06T14:01:24.930" LastActivityDate="2015-12-09T02:27:46.880" Title="What are the best ways to tune multiple parameters?" Tags="|data-mining|dataset|data-cleaning|parameter|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9172" PostTypeId="1" AcceptedAnswerId="9176" CreationDate="2015-12-02T16:16:08.317" Score="1" ViewCount="211" Body="&lt;p&gt;Imagine you have $k$ classes. Every class $i$ has points which are follow a probability distribution, such that their distance to 0 is $i$ in mean, but this distance follows a normal distribution. The direction is uniformly distributed. So all classes are in shells around the origin 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can $k$-means get these shells when you choose the &quot;right&quot; distance metric?&#xA;(Obviously it can't find it if you take the euclidean metric, but I wonder if there is any metric at all or if this problem is inherently unsolvable by $k$-means, even if you know the number of clusters $k$)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-02T22:16:52.933" Title="Can k-means clustering get shells as clusters?" Tags="|clustering|k-means|distance|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9173" PostTypeId="2" ParentId="9167" CreationDate="2015-12-02T16:48:12.337" Score="3" Body="&lt;p&gt;There are multiple factors to consider, but the first thing to realize is that in regression, you don't want to think about whether an example is &quot;correct&quot; or &quot;incorrect&quot; but rather how close it was to the true target value.  Therefore you can ignore your original intuition about &quot;80% of predictions are 'correct'.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second remember that &lt;a href=&quot;https://www.kaggle.com/wiki/RootMeanSquaredError&quot; rel=&quot;nofollow&quot;&gt;RMSE&lt;/a&gt; is in the same space as your target values.  So it is relative to the variance in your target values.  The benchmark of random guessing should get you an &lt;code&gt;RMSE = standard_deviation&lt;/code&gt;.  So lower than this, your model is demonstrating some ability to learn; above that number, you haven't even learned to guess the mean correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There isn't a cutoff for &quot;my model is doing well&quot; in RMSE space, just like with other metrics.  Everything is relative to a naive solution/benchmark or the state-of-the-art.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-02T16:48:12.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9174" PostTypeId="2" ParentId="9169" CreationDate="2015-12-02T16:59:16.860" Score="7" Body="&lt;p&gt;Generally people perform a &lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html&quot;&gt;grid search&lt;/a&gt;, which in its simplest &quot;exhaustive&quot; form is similar to Method 1.  However there are also more 'intelligent' ways to choose what to explore, which optimize in parameter space in a fashion similar to how each individual model is optimized. It can be tricky to do greedy optimization in this space, as it is often strongly non-convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperparameter_optimization&quot;&gt;This page&lt;/a&gt; describes the basics of optimizing model parameters.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-02T16:59:16.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9175" PostTypeId="1" AcceptedAnswerId="9384" CreationDate="2015-12-02T21:53:17.183" Score="55" ViewCount="35440" Body="&lt;p&gt;This question boils down to &quot;how do convolution layers &lt;em&gt;exactly&lt;/em&gt; work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have an $n \times m$ greyscale image. So the image has one channel.&#xA;In the first layer, I apply a $3\times 3$ convolution with $k_1$ filters and padding. Then I have another convolution layer with $5 \times 5$ convolutions and $k_2$ filters. How many feature maps do I have?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Type 1 convolution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the second layer gets applied. Every single filter gets applied separately to &lt;strong&gt;each of the feature maps&lt;/strong&gt;. This results in $k_2$ feature maps for every of the $k_1$ feature maps. So there are $k_1 \times k_2$ feature maps after the second layer. Every single pixel of each of the new feature maps got created by taking $5 \cdot 5 = 25$ &quot;pixels&quot; of the padded feature map from before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Type 2.1 convolution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Like before: The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike before: Then the second layer gets applied. Every single filter gets applied to the same region, but &lt;strong&gt;all feature maps&lt;/strong&gt; from before. This results in $k_2$ feature maps in total after the second layer got executed. Every single pixel of each of the new feature maps got created by taking $k_2 \cdot 5 \cdot 5 = 25 \cdot k_2$ &quot;pixels&quot; of the padded feature maps from before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Type 2.2 convolution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Like above, but instead of having $5 \cdot 5 = 25$ parameters per filter which have to be learned and get simply copied for the other input feature maps, you have $k_1 \cdot 3 \cdot 3 + k_2 \cdot k_1 \cdot 5 \cdot 5$ paramters which have to be learned.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is type 1 or type 2 typically used?&lt;/li&gt;&#xA;&lt;li&gt;Which type is used in &lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf&quot; rel=&quot;noreferrer&quot;&gt;Alexnet&lt;/a&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Which type is used in &lt;a href=&quot;http://arxiv.org/abs/1409.4842&quot; rel=&quot;noreferrer&quot;&gt;GoogLeNet&lt;/a&gt;?&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you say type 1: Why do $1 \times 1$ convolutions make any sense? Don't they only multiply the data with a constant?&lt;/li&gt;&#xA;&lt;li&gt;If you say type 2: Please explain the quadratic cost (&quot;For example, in a deep vision network, if two convolutional layers are chained,&#xA;any uniform increase in the number of their filters results in a quadratic increase of computation&quot;)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For all answers, please give some evidence (papers, textbooks, documentation of frameworks) that your answer is correct.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Bonus question 1&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Is the pooling applied always only per feature map or is it also done over multiple feature maps?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Bonus question 2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'm relatively sure that type 1 is correct and I got something wrong with the GoogLe paper. But there a 3D convolutions, too. Lets say you have 1337 feature maps of size $42 \times 314$ and you apply a $3 \times 4 \times 5$ filter. How do you slide the filter over the feature maps? (Left to right, top to bottom, first feature map to last feature map?) Does it matter as long as you do it consistantly?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My research&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've read the two papers from above, but I'm still not sure what is used.&lt;/li&gt;&#xA;&lt;li&gt;I've read the &lt;a href=&quot;http://lasagne.readthedocs.org/en/latest/modules/layers/conv.html&quot; rel=&quot;noreferrer&quot;&gt;lasagne documentation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;I've read the &lt;a href=&quot;http://deeplearning.net/software/theano/library/tensor/nnet/conv.html&quot; rel=&quot;noreferrer&quot;&gt;theano documentation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;I've read the answers on &lt;a href=&quot;https://stats.stackexchange.com/q/85767/25741&quot;&gt;Understanding convolutional neural networks&lt;/a&gt; (without following all links)&lt;/li&gt;&#xA;&lt;li&gt;I've read &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;noreferrer&quot;&gt;Convolutional Neural Networks (LeNet)&lt;/a&gt;. Especially figure 1 makes me relatively sure that Type 2.1 is the right one. This would also fit to the &quot;quadratic cost&quot; comment in GoogLe Net and to some practical experience I had with Caffee.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2022-06-16T19:07:46.600" LastActivityDate="2022-06-16T19:07:46.600" Title="How do subsequent convolution layers work?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9176" PostTypeId="2" ParentId="9172" CreationDate="2015-12-02T22:16:52.933" Score="2" Body="&lt;p&gt;You cannot just use arbitrary distance functions with k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;because the algorithm is &lt;strong&gt;not based on metric properties but on variance&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/81481/7828&quot;&gt;https://stats.stackexchange.com/q/81481/7828&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fact is that k-means minimizes the &lt;strong&gt;sum of squares&lt;/strong&gt;. This does not even give you the &quot;smallest distances&quot; but only the smallest &lt;em&gt;squared&lt;/em&gt; distances. This is &lt;em&gt;not&lt;/em&gt; the same (see: difference between median and mean) - if you want to minimize Euclidean distances, use k-medians or if you want other distances PAM (k-medoids).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can generalize k-means to use a few more distances known as &quot;Bergman divergences&quot; and you can do some variant of the kernel trick. But that is not very powerful, because you don't have labels for optimizing the kernel parameters! Still, that could be what this exercise question is up to... If your &quot;shells&quot; are indeed centered at 0, then you can transform your data (read: kernel trick done wrong) to angle+distance from origin, and k-means may be able to cluster the projected data (dependent on the not well defined scaling of the axes).&#xA;Or the textbook did &lt;em&gt;not&lt;/em&gt; realize that a kernel k-means has been proposed long ago. Then the argument is probably this: &lt;strong&gt;the mean of each shell is 0, and thus the shells cannot be distinguished&lt;/strong&gt;. This clearly holds for &lt;em&gt;unmodified&lt;/em&gt; k-means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-12-02T22:16:52.933" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9177" PostTypeId="1" CreationDate="2015-12-02T22:16:52.963" Score="2" ViewCount="464" Body="&lt;p&gt;I am currently reading slides about the $k$-means algorithm. In the analysis, the professor writes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Minimize Schwarz Criterion: $W(C) + \lambda m k \log R$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$W(C)$ is Within-class scatter. I guess $\lambda$ is a weighting factor which has to be chosen by the developer and $k$ is the number of clusters. But what is $m$ and what is $R$?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-01-03T19:12:39.270" Title="How is the Schwarz Criterion defined?" Tags="|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9178" PostTypeId="1" CreationDate="2015-12-03T00:31:32.943" Score="2" ViewCount="1207" Body="&lt;p&gt;Here is the problem&#xA;The initial four cluster partition {c1, c2, c3, c4} for the text collection is provided by this link &lt;a href=&quot;http://userpages.umbc.edu/~kogan/teaching/m710/15fall/project/list.txt&quot; rel=&quot;nofollow&quot;&gt;text vs cluster&lt;/a&gt;. Assuming that&#xA;the ground-truth partition is given by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cacm texts belong to cluster1&#xA;cisi texts belong to cluster2&#xA;cran texts belong to cluster3&#xA;med texts belong to cluster4 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;build the confusion matrix (CM) for the partition provided. That is build a 4X4 matrix whose row i shows distribution of ci elements&#xA;among cluster1, cluster2, cluster3, and cluster4. For example the first row is computed as follows:&#xA;CM11=|c1∩cluster1|,  CM12=|c1∩cluster2|,  CM13=|c1∩cluster3|, and CM14=|c1∩cluster4| &#xA;I understand the problem just not sure how to code it up.&#xA;I was thinking along the lines of this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//open text file&#xA;fileID = fopen('list.txt');&#xA;C = textscan(fileID,'%s %s');&#xA;fclose(fileID);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What this does is it orders the names as $C\{1\}\{j\}$is the file name and $C\{2\}\{j\}$ is the cluster name.&#xA;By the way I am using Matlab&lt;/p&gt;&#xA;" OwnerUserId="14495" LastEditorUserId="-1" LastEditDate="2015-12-03T16:10:06.183" LastActivityDate="2016-02-01T17:33:41.027" Title="Clustering and confusion matrix" Tags="|data-mining|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9179" PostTypeId="1" CreationDate="2015-12-03T04:10:28.387" Score="4" ViewCount="14061" Body="&lt;p&gt;I'm reading a &lt;a href=&quot;https://www.robots.ox.ac.uk/~vgg/publications/2010/Lempitsky10b/lempitsky10b.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; on counting cells, humans, etc. Is there an off-the-shelf library for Python/Theano or TensorFlow I could use?&lt;/p&gt;&#xA;" OwnerUserId="13625" LastEditorUserId="11097" LastEditDate="2015-12-03T06:00:33.070" LastActivityDate="2020-08-03T13:41:35.077" Title="Library to count number of objects in image?" Tags="|python|theano|tensorflow|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9180" PostTypeId="2" ParentId="9179" CreationDate="2015-12-03T05:49:55.020" Score="0" Body="&lt;p&gt;Not sure I understand the question...  TensorFlow is OpenSourced, &lt;a href=&quot;https://www.tensorflow.org/tutorials/mnist/beginners/index.html&quot; rel=&quot;nofollow&quot;&gt;here is a tutorial&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To import the TensorFlow library, all you need to do is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14497" LastActivityDate="2015-12-03T05:49:55.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9181" PostTypeId="2" ParentId="9179" CreationDate="2015-12-03T06:03:56.253" Score="1" Body="&lt;p&gt;Yes, there are libraries for TensorFlow and Theano.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/Theano/Theano&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano Library&lt;/a&gt;:  An &lt;a href=&quot;http://www.deeplearning.net/software/theano/&quot; rel=&quot;nofollow noreferrer&quot;&gt;extensive tutorial&lt;/a&gt; for the Theano library is also available online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow library&lt;/a&gt;: An in-depth &lt;a href=&quot;http://www.tensorflow.org/tutorials/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;official tutorial&lt;/a&gt; of TensorFlow.  (Highly recommend, if you're going with TensorFlow)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, I would also like to suggest you to have a look at this &lt;a href=&quot;https://datascience.stackexchange.com/q/694/11097&quot;&gt;question&lt;/a&gt; regarding open source Python deep learning libraries.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-03T06:03:56.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9182" PostTypeId="5" CreationDate="2015-12-03T06:24:50.837" Score="0" Body="&lt;h2&gt;Interface&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;TensorFlow provides a Python API for building and executing data flow graphs. There is also a C++ API for executing graphs, and adding new operation kernels. Other languages can invoke TensorFlow using its low-level C API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow also includes TensorBoard, which is a web-based visualization tool for understanding performance and network features/parameters.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Speed&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;TensorFlow achieves its performance by parallelizing execution across multiple CPUs and GPUs. The kernel of each operation is implemented in C++, using efficient libraries such as Eigen and cuDNN for the best performance.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:59.413" LastActivityDate="2015-12-03T22:02:59.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9183" PostTypeId="4" CreationDate="2015-12-03T06:24:50.837" Score="0" Body="TensorFlow is an open source library for machine learning and machine intelligence.  TensorFlow uses data flow graphs with tensors flowing along edges. For details, see https://www.tensorflow.org.  TensorFlow is released under an Apache 2.0 License." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:02:56.790" LastActivityDate="2015-12-03T22:02:56.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9184" PostTypeId="5" CreationDate="2015-12-03T06:28:53.410" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;pandas&lt;/code&gt;&lt;/a&gt; is a &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;python&lt;/code&gt;&lt;/a&gt; library for PAN-el DA-ta manipulation and analysis, i.e. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance.  &lt;code&gt;pandas&lt;/code&gt; is implemented primarily using &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;numpy&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;http://cython.org/&quot; rel=&quot;nofollow&quot;&gt;Cython&lt;/a&gt;; it is intended to be able to integrate very easily with other &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;numpy&lt;/code&gt;&lt;/a&gt;-based scientific libraries, such as &lt;a href=&quot;http://statsmodels.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;statsmodels&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Main Features:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data structures: for 1, 2, and 3 dimensional labeled data sets (respectively &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/dsintro.html#series&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;Series&lt;/code&gt;&lt;/a&gt;, &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;DataFrames&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/dsintro.html#panel&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;Panels&lt;/code&gt;&lt;/a&gt;). Some of their main features include:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automatically aligning data and interpolation&lt;/li&gt;&#xA;&lt;li&gt;Handling missing observations in calculations&lt;/li&gt;&#xA;&lt;li&gt;Convenient slicing and reshaping (&quot;reindexing&quot;) functions&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/categorical.html&quot; rel=&quot;nofollow&quot;&gt;Categorical data&lt;/a&gt; types &lt;/li&gt;&#xA;&lt;li&gt;Provide 'group by' aggregation or transformation functionality&lt;/li&gt;&#xA;&lt;li&gt;Tools for merging / joining together data sets&lt;/li&gt;&#xA;&lt;li&gt;Simple &lt;a href=&quot;http://matplotlib.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;matplotlib&lt;/code&gt;&lt;/a&gt; integration for plotting&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Date tools: objects for expressing date offsets or generating date ranges; some functionality similar to scikits.timeseries.  Dates can be aligned to a specific timezone and converted / compared at-will&lt;/li&gt;&#xA;&lt;li&gt;Statistical models: convenient ordinary least squares and panel OLS implementations for in-sample or rolling time series / cross-sectional regressions. These will hopefully be the starting point for implementing other models&lt;/li&gt;&#xA;&lt;li&gt;Intelligent Cython offloading; complex computations are performed rapidly due to these optimizations.&lt;/li&gt;&#xA;&lt;li&gt;Static and moving statistical tools: mean, standard deviation, correlation, covariance&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/&quot; rel=&quot;nofollow&quot;&gt;Rich User Documentation&lt;/a&gt;, using &lt;a href=&quot;http://sphinx.pocoo.org/&quot; rel=&quot;nofollow&quot;&gt;Sphinx&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Resources:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/&quot; rel=&quot;nofollow&quot;&gt;Docs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/pydata/pandas&quot; rel=&quot;nofollow&quot;&gt;Github&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Homepage&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Books:&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920023784.do&quot; rel=&quot;nofollow&quot;&gt;Python for Data Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.packtpub.com/learning-ipython-for-interactive-computing-and-data-visualization/book&quot; rel=&quot;nofollow&quot;&gt;Learning IPython for Interactive Computing and Data Visualization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/mastering-pandas-finance&quot; rel=&quot;nofollow&quot;&gt;Mastering pandas for Finance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:03:02.157" LastActivityDate="2015-12-03T22:03:02.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9185" PostTypeId="4" CreationDate="2015-12-03T06:28:53.410" Score="0" Body="pandas is a python library for Panel Data manipulation and analysis, e.g. multidimensional time series and cross-sectional data sets commonly found in statistics, experimental science results, econometrics, or finance." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-03T22:03:47.727" LastActivityDate="2015-12-03T22:03:47.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9186" PostTypeId="2" ParentId="9179" CreationDate="2015-12-03T08:34:51.567" Score="3" Body="&lt;p&gt;I think the OP was asking for a specific library &lt;em&gt;for&lt;/em&gt; Theano or TensorFlow that provides off-the-shelf functionality for object identification.&lt;/p&gt;&#xA;&lt;p&gt;I would look first into &lt;code&gt;scikit image&lt;/code&gt;, an image processing library for Python in the Scipy/Numpy stack. It has all sorts of edge detectors, blob detectors, shape identifications, color separations... Look at their &lt;a href=&quot;http://scikit-image.org/docs/dev/auto_examples/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gallery&lt;/a&gt; of examples.&lt;/p&gt;&#xA;&lt;p&gt;Btw, counting humans in a crowd may not be so trivial to automate...&lt;/p&gt;&#xA;" OwnerUserId="12580" LastEditorUserId="98307" LastEditDate="2020-08-03T12:57:04.993" LastActivityDate="2020-08-03T12:57:04.993" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9188" PostTypeId="1" AcceptedAnswerId="9190" CreationDate="2015-12-03T10:33:27.963" Score="1" ViewCount="1550" Body="&lt;p&gt;I wasn't sure where to ask this question so forgive me if the question seems out of place (and please guide me where to ask it !)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an archive of 9GAG images and I want to Cluster them based on their content and their similarity... 9GAG images are mostly memes so it's natural that you'd find many of them pretty similar to each other...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find any application which does this out of the box (if there is, could you please refer me to it ?) and what I found was a vast number of Papers about Image Clustering but no real application based on them...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a Ruby, Python or Java Program which could simply get the directory of the Images and Cluster them into groups (folders) based on their Similarity to each other ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much...&lt;/p&gt;&#xA;" OwnerUserId="14504" LastActivityDate="2015-12-04T11:01:26.237" Title="Cluster Similar Images into Folders" Tags="|clustering|image-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9189" PostTypeId="2" ParentId="8294" CreationDate="2015-12-03T11:38:53.413" Score="10" Body="&lt;p&gt;If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), after your one hot encoding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But know that &quot;56 features&quot; isn't really large and it's highly common in the industry to have thousands, millions or even billions of features. &lt;/p&gt;&#xA;" OwnerUserId="14200" LastEditorUserId="11097" LastEditDate="2015-12-03T12:32:26.467" LastActivityDate="2015-12-03T12:32:26.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9190" PostTypeId="2" ParentId="9188" CreationDate="2015-12-03T11:39:54.640" Score="1" Body="&lt;p&gt;I would impressed if there is already a dedicated 9gag clustering :P&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you can read this blog post about &lt;a href=&quot;http://www.janeriksolem.net/2009/04/hierarchical-clustering-in-python.html&quot; rel=&quot;nofollow&quot;&gt;Hierarchical clustering in Python&lt;/a&gt; for images, which is close to what you want. The problem is that the author uses the average color of the image as a feature and it could proved crude and not inefficient. You may find something more interesting to use. But in the end, you need to experiment a lot with your own dataset.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-12-03T11:39:54.640" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9191" PostTypeId="2" ParentId="9134" CreationDate="2015-12-03T12:14:27.500" Score="2" Body="&lt;p&gt;As it states, the following presentation is a 'gentle' introduction to Gradient Boosting, I found it quite helpful when figuring out Gradient Boosting; there is a fully explained example included. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14372" LastActivityDate="2015-12-03T12:14:27.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9193" PostTypeId="2" ParentId="9178" CreationDate="2015-12-03T14:41:58.103" Score="1" Body="&lt;p&gt;Assume you clustering results are store in R. You have the partition id &lt;em&gt;part_id&lt;/em&gt; and the real &lt;em&gt;clust_id&lt;/em&gt;, here in a small example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- read.table(text = &quot;obj,  part_id, clust_id &#xA;X, 1, 1&#xA;Y, 2, 2&#xA;Z, 3, 3 &#xA;U, 1, 3&#xA;V, 2, 3&#xA;W, 2, 3&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Than the &lt;strong&gt;table&lt;/strong&gt; will do the work for you&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; table(df[,c(&quot;part_id&quot;,&quot;clust_id&quot;)])&#xA;&#xA;       clust_id&#xA;part_id 1 2 3&#xA;      1 1 0 1&#xA;      2 0 1 2&#xA;      3 0 0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For completeness you may also use SQL (as proposed in a comment), a sample query is shown below on a table identical to the data-frame above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will see imediately, that there is a limitation because you must define the exact column list in the query - so the query must be adjusted based on the cluster / partition list.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with clust_agg as (&#xA;select part_id, clust_id, count(*) cnt from clust&#xA;group by part_id, clust_id&#xA;)&#xA;select * from clust_agg&#xA;pivot(sum(cnt) clust  for (clust_id) in &#xA;     (1 as &quot;1&quot;,&#xA;      2 as &quot;2&quot;,&#xA;      3 as &quot;3&quot;))&#xA;order by 1;&#xA;, header = TRUE, sep=&quot;,&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   PART_ID    1_CLUST    2_CLUST    3_CLUST&#xA;---------- ---------- ---------- ----------&#xA;         1          1                     1 &#xA;         2                     1          2 &#xA;         3                                1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2015-12-03T14:47:14.060" LastActivityDate="2015-12-03T14:47:14.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9195" PostTypeId="1" CreationDate="2015-12-03T21:30:57.907" Score="10" ViewCount="386" Body="&lt;p&gt;Are there any papers published which show differences of the regularization methods for neural networks, preferably on different domains (or at least different datasets)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am asking because I currently have the feeling that most people seem to use only dropout for regularization in computer vision. I would like to check if there would be a reason (not) to use different ways of regularization.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2022-06-16T19:07:29.757" LastActivityDate="2022-06-16T19:07:29.757" Title="Are there studies which examine dropout vs other regularizations?" Tags="|neural-network|convolutional-neural-network|computer-vision|regularization|dropout|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9196" PostTypeId="2" ParentId="9188" CreationDate="2015-12-04T00:28:11.300" Score="1" Body="&lt;p&gt;Extract color histograms from each image. Then cluster them with ELKI, which has a number of relevant similarity measures for images, such as histogram intersection distance.&#xA;Maybe start with hierarchical clustering first, then also try OPTICS.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-12-04T00:28:11.300" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9197" PostTypeId="1" AcceptedAnswerId="9198" CreationDate="2015-12-04T06:40:57.113" Score="3" ViewCount="1986" Body="&lt;p&gt;I have a set of predictor variables and another target variable .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am really confused on what method to use to forecast the target variable .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For e.g my data set have customer profit(which is my target variable) and a set of predictor variable(balances of different account) for one year for each customer .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to predict profit of next 5 years .I am confused in the part that I dont have the data(predictor variables) for future .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my possible choices of modelling .Please assist .&lt;/p&gt;&#xA;" OwnerUserId="14353" LastActivityDate="2016-12-23T15:24:02.080" Title="Predicting future value with regression Model" Tags="|r|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9198" PostTypeId="2" ParentId="9197" CreationDate="2015-12-04T07:59:25.257" Score="3" Body="&lt;p&gt;You should distinct between a &lt;em&gt;time series&lt;/em&gt; prediction, where from a known history of some attribute the future is predicted and &lt;em&gt;model prediction&lt;/em&gt; where based on the predictor variables the target variable is calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you could combine both approaches, i.e. use time series prediction on the customer balances and apply the regression model to calculate the profit on the result.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-12-04T07:59:25.257" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9199" PostTypeId="2" ParentId="9197" CreationDate="2015-12-04T10:57:55.683" Score="0" Body="&lt;p&gt;One way to generate a set of predictor variables is by adding noise.&#xA;In your case, this might work well unless there is a lot of variation due to uncontrolled conditions like financial crisis.&#xA;You have to be careful about adding noise. It should thoroughly test your model for robustness. One way would be to add fractions of the variance of each attribute. &lt;/p&gt;&#xA;" OwnerUserId="12785" LastActivityDate="2015-12-04T10:57:55.683" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9200" PostTypeId="2" ParentId="9188" CreationDate="2015-12-04T11:01:26.237" Score="0" Body="&lt;p&gt;I guess classification is a better term here.&#xA;One idea would be to calculate histogram of an image.&#xA;Make 32 bins from the histogram.&#xA;So if an image has Red, Blue, Green channels, each having 256 values, you will get a vector of 32*3=96 dimensions.&#xA;You can run SVM or Naive Bayes on it.&lt;/p&gt;&#xA;" OwnerUserId="12785" LastActivityDate="2015-12-04T11:01:26.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9201" PostTypeId="2" ParentId="8294" CreationDate="2015-12-04T11:05:08.427" Score="-1" Body="&lt;p&gt;When there are large number of categorical variables, it is advisable to do one versus rest.&lt;/p&gt;&#xA;" OwnerUserId="12785" LastActivityDate="2015-12-04T11:05:08.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9202" PostTypeId="1" AcceptedAnswerId="9211" CreationDate="2015-12-04T11:36:40.260" Score="5" ViewCount="14603" Body="&lt;p&gt;I am trying to fully understand difference between categorical and ordinal data when doing regression analysis. For now, what is clear:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Categorical feature and data example:&lt;/strong&gt;&lt;br&gt; &#xA;Color: red, white, black &lt;br&gt; &#xA;Why categorical: &lt;code&gt;red &amp;lt; white &amp;lt; black&lt;/code&gt; is logically &lt;strong&gt;incorrect&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ordinal feature and data example:&lt;/strong&gt;&lt;br&gt; &#xA;Condition: old, renovated, new &lt;br&gt; &#xA;Why ordinal: &lt;code&gt;old &amp;lt; renovated &amp;lt; new&lt;/code&gt; is logically &lt;strong&gt;correct&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Categorical-to-numeric and ordinal-to-numeric encoding methods: &lt;br&gt;&lt;/strong&gt;&#xA;One-Hot encoding for categorical data &lt;br&gt;&#xA;Arbitrary numbers for ordinal data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Categorical data to numeric:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = {'color': ['blue', 'green', 'green', 'red']}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Numeric format after One-Hot encoding:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   color_blue  color_green  color_red&#xA;0           1            0          0&#xA;1           0            1          0&#xA;2           0            1          0&#xA;3           0            0          1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ordinal data to numeric:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = {'con': ['old', 'new', 'new', 'renovated']}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Numeric format after using mapping: Old &amp;lt; renovated &amp;lt; new → 0, 1, 2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    0&#xA;1    2&#xA;2    2&#xA;3    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my data I have 'color' feature. As color changes from white to black price increases. From above mentioned rules I probably have to use one-hot encoding for categorical 'color' data. But why I cannot use ordinal representation. Below I provided my observations from where my question arised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me start with introducing formula for linear regression:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mbiK6.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mbiK6.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;br&gt;Let have a look at data representations for color:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Veb54.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Veb54.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Let's predict price for 1-st and 2-nd item using formula for both data representations:&lt;br&gt;&#xA;&lt;strong&gt;One-hot encoding:&lt;/strong&gt; &#xA;In this case different thetas for different colors will exist. I assume that thetas already derived from regression (20, 50 and 100). Prediction will be: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Price (1 item) = 0 + 20*1 + 50*0 + 100*0 = 20$  (thetas are assumed for example)&#xA;Price (2 item) = 0 + 20*0 + 50*1 + 100*0 = 50$  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ordinal encoding for color:&lt;/strong&gt;&#xA;In this case all colors will have 1 common theta but  my assigned multipliers (10, 20, 30) differ:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Price (1 item) = 0 + 20*10 = 200$  (theta assumed for example)&#xA;Price (2 item) = 0 + 20*20 = 400$  (theta assumed for example)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my model White &amp;lt; Red &amp;lt; Black in prices. Seem to be that correlation works correctly and it is logical predictions in both cases. For ordinal and categorical representations. So I can use any encoding for my regression regardless of the data type (categorical or ordinal)? This division in data representations is just a matter of conventions and software-oriented representations rather than a matter of regression logic?&lt;/p&gt;&#xA;" OwnerUserId="14525" LastActivityDate="2015-12-04T15:44:19.133" Title="Categorical and ordinal feature data representation in regression analysis?" Tags="|machine-learning|regression|categorical-data|linear-regression|" AnswerCount="1" CommentCount="2" ClosedDate="2016-08-29T09:33:37.913" ContentLicense="CC BY-SA 3.0" />
  <row Id="9204" PostTypeId="5" CreationDate="2015-12-04T12:17:25.853" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-12-04T12:17:25.853" LastActivityDate="2015-12-04T12:17:25.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9205" PostTypeId="4" CreationDate="2015-12-04T12:17:25.853" Score="0" Body="Computer Vision is a subfield of computer science which deals with analyzing and understanding images. This includes detection of objects like faces in images or segmenting images." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2015-12-05T16:58:25.357" LastActivityDate="2015-12-05T16:58:25.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9207" PostTypeId="1" CreationDate="2015-12-04T14:11:04.620" Score="2" ViewCount="433" Body="&lt;p&gt;I'd like to find frequent patterns in data that has been created by an accelerometer of a Smart Watch.&#xA;The algorithm should return the parts of the data that occur after a pattern.&#xA;In the best case, the algorithm finds the movement automatically that shows how we open doors, grab our phone etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, until now I only was able to find algorithms that find frequent &lt;strong&gt;item sets&lt;/strong&gt; in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know a solution on how to find those patterns in &lt;strong&gt;continuous&lt;/strong&gt; data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much! :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/Doz0S.png&quot; alt=&quot;patterns&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="14531" LastActivityDate="2015-12-04T14:11:04.620" Title="Finding Patterns in continuous data" Tags="|machine-learning|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9208" PostTypeId="1" AcceptedAnswerId="9239" CreationDate="2015-12-04T14:24:09.310" Score="8" ViewCount="886" Body="&lt;p&gt;I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow?  As of Dec 2015 has the community started to lean one direction or another?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://deeplearning4j.org/compare-dl4j-torch7-pylearn.html&quot;&gt;link&lt;/a&gt; has given me concern about getting tied to TenserFlow.&lt;/p&gt;&#xA;" OwnerUserId="14535" LastEditorUserId="11097" LastEditDate="2015-12-04T15:42:32.287" LastActivityDate="2015-12-07T01:03:55.753" Title="Pylearn2 vs TensorFlow" Tags="|machine-learning|python|neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9210" PostTypeId="2" ParentId="9208" CreationDate="2015-12-04T15:42:09.467" Score="3" Body="&lt;p&gt;As far as I know, the existing (almost all) libraries in Python can handle very complex models of Neural Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow is however not polished as of now. It still has a long way to grow before getting accepted as a mainstream library for ML. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, going ahead with the existing libraries like PyLearn/Keras/Torch, etc makes sense as of now (also as they have wide and dedicated communities already), as you need to concentrate on research rather than worrying on bugs and technical problems of a library.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-04T15:42:09.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9211" PostTypeId="2" ParentId="9202" CreationDate="2015-12-04T15:44:19.133" Score="4" Body="&lt;p&gt;The distinction between ordinal and categorical does matter. If in truth the difference between white and red was drastically different from red and black, your (10,20,30) ordinal model would not have performed well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One hot encoding can learn the relationship between the ordinal values more finely, but throws out the information that the variables are related. Similarly, with insufficient data it is more likely to overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinal variables lessen those  problems but at the cost of forcing you to define the interval. There are a number of methods for defining the values of your ordinal variables, like rologit.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-04T15:44:19.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9212" PostTypeId="1" AcceptedAnswerId="9213" CreationDate="2015-12-04T15:45:40.557" Score="4" ViewCount="2097" Body="&lt;p&gt;I've recently read about maxout in slides of a lecture and in the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is maxout the same as max pooling?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:53:28.843" LastActivityDate="2021-02-16T15:53:28.843" Title="Is Maxout the same as max pooling?" Tags="|machine-learning|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9213" PostTypeId="2" ParentId="9212" CreationDate="2015-12-04T15:45:40.557" Score="5" Body="&lt;p&gt;They are almost identical:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The second key reason that maxout performs well is&#xA;  that it improves the bagging style training phase of&#xA;  dropout. Note that the arguments in section 7 motivating&#xA;  the use of maxout also apply equally to rectified&#xA;  linear units (Salinas &amp;amp; Abbott, 1996; Hahnloser, 1998;&#xA;  Glorot et al., 2011). &lt;strong&gt;The only difference between maxout&#xA;  and max pooling over a set of rectified linear units&#xA;  is that maxout does not include a 0 in the max&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;http://arxiv.org/pdf/1302.4389v4.pdf&quot; rel=&quot;noreferrer&quot;&gt;Maxout Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-04T15:45:40.557" CommentCount="0" CommunityOwnedDate="2015-12-04T15:45:40.557" ContentLicense="CC BY-SA 3.0" />
  <row Id="9215" PostTypeId="2" ParentId="9212" CreationDate="2015-12-04T17:35:20.373" Score="2" Body="&lt;p&gt;That paper of Goodfellow is a little bit cryptic. For what I understand, the max out networks let the net to find the best activation function for the specific problem. This is done through a winner take all algorithm in  intermediate layers. Accidentally this kind of networks compress the information in term of paths. This means that the information is stored in the net in term weights and paths that the net create. In my opinion this is not the same as the max pooling, even though we use a winner take all in both cases, we perform a max pool in a layer to reduce the dimensionality of the problem. &lt;/p&gt;&#xA;" OwnerUserId="10938" LastActivityDate="2015-12-04T17:35:20.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9216" PostTypeId="1" CreationDate="2015-12-04T17:50:56.987" Score="2" ViewCount="121" Body="&lt;p&gt;In Tableau, I commonly want to carry out a task like distinguishing 2 groups by various measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example below shows the difference in the count of 2 treatment and control for variable levels of a factor (measure):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6xuNV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6xuNV.jpg&quot; alt=&quot;Tableau&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This view is misleading though, because the overall group sizes of treatment and control are not equal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to do is to represent this as a proportion for each levels of the factor (measure) instead of using just a count. For instance, perhaps 25% of the treatment group is in Business Development whereas only 10% of the control group is in Business Development. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this possible to do in Tableau, or does it strictly require pre-processing?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2017-02-01T11:39:03.860" Title="Change raw count to a proportion based on a filter in Tableau" Tags="|tableau|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9217" PostTypeId="2" ParentId="9038" CreationDate="2015-12-04T18:11:57.240" Score="4" Body="&lt;p&gt;Taking a slightly different approach than the other great answers here, the &quot;pretty picture&quot; is worth a thousand words. Ultimately, you will need to convey your findings to someone who is not as statistically literate, or who simply does not have the time, interest, or whatever, to grasp the full situation. That doesn't mean we cannot help the person to understand, at least a general concept or a piece of the reality. This is what books like Freakonomics do - there's little to no math, no data sets, and yet the findings are still presented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the arts, look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Adolphe_Yvon#/media/File:Adolphe_Yvon_%281817-1893%29_-_Marshall_Ney_at_retreat_in_Russia.jpg&quot; rel=&quot;nofollow&quot;&gt;Marshal Ney at Retreat in Russia&lt;/a&gt;. This massive oversimplification of the Napoleonic wars nevertheless conveys great meaning and allows people with even the most ignorant knowledge of the war to understand the brutality, the climate, the landscape, the death, and decorum that permeated the invasion of Russia.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately the charts are simply communication, and for better or worse, human communication is often times focused on conflation, simplification, and brevity.&lt;/p&gt;&#xA;" OwnerUserId="13578" LastActivityDate="2015-12-04T18:11:57.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9218" PostTypeId="2" ParentId="9177" CreationDate="2015-12-04T18:13:24.560" Score="1" Body="&lt;p&gt;See also &lt;a href=&quot;https://stackoverflow.com/questions/1793532/how-do-i-determine-k-when-using-k-means-clustering&quot;&gt;this StackExchange post&lt;/a&gt;. The Schwarz criterion, or BIC, is very general, so the question is, &quot;how does this formula apply the BIC to $k$-means?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page on Bayesian Information Criterion&lt;/a&gt; (BIC) uses this definition of BIC:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$BIC = -2 \ln \hat{L} + \kappa \ln {n}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$n$ is the #observations&lt;/li&gt;&#xA;&lt;li&gt;$\kappa$ is the #free parameters (I use Greek $\kappa$ to distinguish from your $k$)&lt;/li&gt;&#xA;&lt;li&gt;$\hat{L}$ is the maximized likelihood after fitting parameters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Pelleg and Moore (2000) use $R$ for the #observations.  Assuming your professor does to, that means:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\kappa = \lambda m k$, the # free parameters.  &lt;/li&gt;&#xA;&lt;li&gt;$W(C) = -2 \ln \hat{L}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, what are $\lambda$ and $m$?  I can't tell which is which, but it has to come out something like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$k$ is the number of clusters [as you say]&lt;/li&gt;&#xA;&lt;li&gt;$\lambda$ is the number of dimensions (columns) in the data&lt;/li&gt;&#xA;&lt;li&gt;$m$ is the number of parameters to estimate for each dimension (2 for the usual Gaussian: mean and variance).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It makes sense that $W(C)$, the &quot;within-class scatter&quot; would be measured by log likelihood.  Note, $W(C)$ has to sum the log likelihood for all clusters.&lt;/p&gt;&#xA;" OwnerUserId="12417" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-12-04T18:13:24.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9221" PostTypeId="2" ParentId="9169" CreationDate="2015-12-05T05:53:41.113" Score="3" Body="&lt;p&gt;I just want to add more information about these  more &quot;intelligent&quot; ways to pick hyperparameters. One in particular that's becoming more and more popular. &lt;a href=&quot;http://arxiv.org/pdf/1206.2944.pdf&quot; rel=&quot;nofollow&quot;&gt;BAYESIAN  OPTIMIZATION  OF  MACHINE  LEARNING ALGORITHMS&lt;/a&gt; by Jasper Snoek, Hugo Larochelle and Ryan P. Adam. Which has proven effective in algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithm on Github&#xA;&lt;a href=&quot;https://github.com/JasperSnoek/spearmint&quot; rel=&quot;nofollow&quot;&gt;https://github.com/JasperSnoek/spearmint&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method is particularly useful when the algorithm is complicated (leaning towards a black box model) and/or the data is massive and it's computationally expensive to train several times a model.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-12-05T05:53:41.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9222" PostTypeId="1" CreationDate="2015-12-05T10:24:32.533" Score="3" ViewCount="7338" Body="&lt;p&gt;I am trying to understand how the shape of the image changes after deconvolution ?&#xA;I am trying to understand the example code of &lt;a href=&quot;https://github.com/NervanaSystems/neon/blob/master/examples/conv_autoencoder.py&quot; rel=&quot;nofollow&quot;&gt;convolutional autoencoder from neon&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layers = [Conv((4, 4, 8), init=init_uni, activation=Rectlin()),&#xA;          Pooling(2),&#xA;          Conv((4, 4, 32), init=init_uni, activation=Rectlin()),&#xA;          Pooling(2),&#xA;          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),&#xA;          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),&#xA;          Deconv(fshape=(4, 4, 1), init=init_uni, strides=2, padding=0)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The input_shapes and output_shapes of each layer are as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Convolution Layer 'ConvolutionLayer': 1 x (28x28) inputs, 8 x (25x25) outputs, padding 0, stride 1&#xA;Pooling Layer 'PoolingLayer': 8 x (25x25) inputs, 8 x (12x12) outputs&#xA;Convolution Layer 'ConvolutionLayer': 8 x (12x12) inputs, 32 x (9x9) outputs, padding 0, stride 1&#xA;Pooling Layer 'PoolingLayer': 32 x (9x9) inputs, 32 x (4x4) outputs&#xA;Deconvolution Layer 'DeconvolutionLayer': 32 x (4x4) inputs, 8 x (7x7) outputs&#xA;Deconvolution Layer 'DeconvolutionLayer': 8 x (7x7) inputs, 8 x (13x13) outputs&#xA;Deconvolution Layer 'DeconvolutionLayer': 8 x (13x13) inputs, 1 x (28x28) outputs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand how the shapes change after convolution ('valid'). (Thanks to &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow&quot;&gt;http://cs231n.github.io/convolutional-networks/&lt;/a&gt;)&#xA;How does the stride affect the size of matrix when deconvolution (full convolution) is used ? &lt;/p&gt;&#xA;" OwnerUserId="10396" LastEditorUserId="29169" LastEditDate="2021-02-16T15:53:18.987" LastActivityDate="2021-02-16T15:53:18.987" Title="How does strided deconvolution works?" Tags="|deep-learning|convolutional-neural-network|autoencoder|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9224" PostTypeId="1" AcceptedAnswerId="9226" CreationDate="2015-12-05T15:04:43.810" Score="4" ViewCount="857" Body="&lt;p&gt;I'm currently building an app that strives to predict how the users uses different apps and give the user a suggestion based on which apps it think the user will currently use (a ranked list based on the user's current conditions). I've been collecting some data over the past week now, and I'm not really sure which approach to take. I've been thinking about using multiple seasonalities (correct me if I'm using the wrong terminology) such as time of day, day of week, week of month, month and quarter. I also want to use location, and other sensor data (such as the user state &quot;walking&quot; or &quot;sitting&quot; later on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've summarised the usage over the last week on an hourly level for some apps. &#xA;The bars represents each time the app was opened during that time, and the green line is a weighted moving average which has a weight of 0.5 on its closest neighbours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you think my model is a good one for this problem&lt;/li&gt;&#xA;&lt;li&gt;How do I account for ageing data?&lt;/li&gt;&#xA;&lt;li&gt;How do add up the different seasonalities/states/location? Multiply them?&lt;/li&gt;&#xA;&lt;li&gt;Does it make sense smoothening the curve as I do?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here some data for the last week on an hourly basis:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/N4fjl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N4fjl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14556" LastActivityDate="2023-12-04T05:44:21.233" Title="Predicting app usage on mobile phone" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9226" PostTypeId="2" ParentId="9224" CreationDate="2015-12-05T19:00:34.283" Score="2" Body="&lt;p&gt;So you have collected data that shows which app is being used at any time, binned into hours in the day. And you have several apps. You mention other dimensions, like user state when using the app (walking, not-walking), (active, not-active - it appears to me that you are not collecting much usage 2-6. is that because the usage is from a self-directed ping from an app while the user is truly away?), location (is this going to be all possible values, or are you going to use something like the fact that this location has been seen often before?). Another interesting relationship could be pairing apps, i.e., mining for a relationship between App A being used after using App B or before App B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, then you will definitely have many different dimensions upon which to measure the usage characteristics for any particular usage measurement, and so you are definitely going to have a multiple dimensional problem. You might try to visualize this as an N-space problem, with an axis of measurement for each of your characteristics. Each of your previous measurements represents vectors and you are producing a new vector with your next measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this, you want to predict future behavior based on measuring the input characteristics from your usage space. You could go for something that classifies as nearest neighbor, and you probably want to do this for your first stab at the problem. You might end up wanting to make the predictive model more sophisticated by adding probabilities to the classifier and acting on that. This means getting estimates of class membership probability rather than just simple classifications. But I would build the whole thing incrementally. Start simple and add complexity as you require it. The increased complexity will also have effects on performance, so why not baseline with something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the aging of data, are you wanting to reduce the predictive power of characteristics that are too long in the tooth? If so, be explicit with yourself about what that means, quantitively. Do I trust the usage data from last month less than yesterday's data? Perhaps so, but then why? is my usage different because I am different or because last month was special compared to yesterday, or vice-versa? Again, you might benefit from ignoring this at first, but then trying to &lt;strong&gt;search&lt;/strong&gt; for &quot;seasonal&quot; or periodicity characteristics from the data. Once you determine if/how it changes, you can weight that contribution compared to your immediate usage in different ways. Perhaps you want to amplify the contribution of a similar period (same time of day &amp;amp;&amp;amp; same location &amp;amp;&amp;amp; same previous app usage). Perhaps you want to provide an exponential dampening on historical data because the usage is always adapting and changing, and recent usage seems to be a &lt;strong&gt;much&lt;/strong&gt; better predictor than 3xcurrent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For all of this, the proper data science perspective is to let the data lead you. &lt;/p&gt;&#xA;" OwnerUserId="10663" LastActivityDate="2015-12-05T19:00:34.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9227" PostTypeId="2" ParentId="5224" CreationDate="2015-12-05T20:18:42.887" Score="34" Body="&lt;p&gt;The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from:&#xA;&lt;a href=&quot;https://benanne.github.io/2015/03/17/plankton.html&quot;&gt;Classifying plankton with deep neural networks&lt;/a&gt;. I suggest you read it, specially the part about &lt;strong&gt;Pre-processing and data augmentation&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;- Resize Images&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for different sizes, resolutions or distances you can do the following. You can simply rescale the largest side of each image to a fixed length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to use openCV or scipy.&#xA;and this will resize the image to have 100 cols (width) and 50 rows (height):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;resized_image = cv2.resize(image, (100, 50)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Yet another option is to use scipy module, by using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;small = scipy.misc.imresize(image, 0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;- Data Augmentation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Augmentation always improves performance though the amount depends on the dataset. If you want to augmented the data to artificially increase the size of the dataset you can do the following if the case applies (it wouldn't apply if for example were images of houses or people where if you rotate them 180degrees they would lose all information but not if you flip them like a mirror does):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rotation: random with angle between 0° and 360° (uniform)&lt;/li&gt;&#xA;&lt;li&gt;translation: random with shift between -10 and 10 pixels (uniform)&lt;/li&gt;&#xA;&lt;li&gt;rescaling: random with scale factor between 1/1.6 and 1.6&#xA;(log-uniform)&lt;/li&gt;&#xA;&lt;li&gt;flipping: yes or no (bernoulli)&lt;/li&gt;&#xA;&lt;li&gt;shearing: random with angle between -20° and 20° (uniform)&lt;/li&gt;&#xA;&lt;li&gt;stretching: random with stretch factor between 1/1.3 and 1.3&#xA;(log-uniform)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can see the results on the Data Science bowl images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pre-processed images&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0S0Y0.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0S0Y0.png&quot; alt=&quot;Pre-processed images&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;augmented versions of the same images&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KJXZK.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KJXZK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-Other techniques&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These will deal with other image properties like lighting and are already related to the main algorithm more like a simple pre-processing step. Check the full list on: &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/unsupervised/PCAWhitening/&quot;&gt;UFLDL Tutorial&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2015-12-17T19:27:26.240" LastActivityDate="2015-12-17T19:27:26.240" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9228" PostTypeId="1" AcceptedAnswerId="9229" CreationDate="2015-12-05T22:24:29.063" Score="17" ViewCount="46112" Body="&lt;p&gt;In which cases is it better to use a Decision tree and other cases a KNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why use one of them in certain cases? And the other in different cases? (By looking at its functionality, not at the algorithm)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone have some explanations or references about this?&lt;/p&gt;&#xA;" OwnerUserId="14566" LastActivityDate="2021-03-10T02:55:21.323" Title="Decision tree vs. KNN" Tags="|machine-learning|data-mining|decision-trees|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9229" PostTypeId="2" ParentId="9228" CreationDate="2015-12-06T00:09:57.890" Score="10" Body="&lt;p&gt;They serve different purposes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;KNN is unsupervised, Decision Tree (DT) supervised.&#xA;(&lt;strong&gt;KNN is supervised learning while K-means is unsupervised, I think this answer causes some confusion.&lt;/strong&gt;)&#xA;KNN is used for clustering, DT for classification. (&lt;strong&gt;Both are used for classification.&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KNN determines neighborhoods, so there must be a distance metric. This implies that all the features must be numeric. Distance metrics may be affected by varying scales between attributes and also high-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DT, on the other hand, predicts a class for a given input vector. The attributes may be numeric or nominal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you want to find similar examples you could use KNN. If you want to classify examples you could use DT.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorUserId="81999" LastEditDate="2020-03-11T14:40:25.473" LastActivityDate="2020-03-11T14:40:25.473" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="9230" PostTypeId="2" ParentId="9020" CreationDate="2015-12-06T00:53:21.600" Score="10" Body="&lt;p&gt;It's better to do the square computing on the values of the non standardized features, and then standardize these new features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point of standardizing features is to put features in a similar scale, e.g. 0-1. If you didn't standardize these new features after the computing, although your new features are between 0-1, your new features will be one or more magnitude smaller than your old features. &lt;/p&gt;&#xA;" OwnerUserId="14463" LastActivityDate="2015-12-06T00:53:21.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9231" PostTypeId="1" CreationDate="2015-12-06T09:17:40.570" Score="0" ViewCount="54" Body="&lt;p&gt;If I am given 60 features along with test label and I was to find values of other features what is the best way to do it ?&lt;/p&gt;&#xA;" OwnerUserId="14573" LastActivityDate="2015-12-06T11:57:37.600" Title="Missing features for classifier" Tags="|machine-learning|parameter|" AnswerCount="1" CommentCount="1" ClosedDate="2015-12-06T14:00:50.037" ContentLicense="CC BY-SA 3.0" />
  <row Id="9232" PostTypeId="1" CreationDate="2015-12-06T10:56:34.090" Score="2" ViewCount="1053" Body="&lt;p&gt;I have two models: one a simple linear regression and the other a multi linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The strongest relationship is between the two variables in the simple linear regression. The multi linear regression includes two extra variables that can be connected with the dependent, however, statistically they are insignificant (P &gt; 0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I interpret whether the best model to use is the simple linear regression or the multi-linear regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some values from the regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple Linear : F(1, 77) = 21.07 , Prob &gt; F = 0.0000 , Rsquared = 0.2148, Adj Rsquared = 0.2046&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multi Linear: F(3, 75) = 7.29 , Prob &gt; F = 0.0002, Rsquared = 0.2258, Adj Rsquared = 0.1948&lt;/p&gt;&#xA;" OwnerUserId="14533" LastEditorUserId="11097" LastEditDate="2015-12-06T11:08:32.977" LastActivityDate="2016-12-30T11:18:07.327" Title="How to test for Overall Significance?" Tags="|statistics|linear-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9233" PostTypeId="1" AcceptedAnswerId="9237" CreationDate="2015-12-06T11:41:30.750" Score="23" ViewCount="8394" Body="&lt;p&gt;I am currently preparing for an exam on neural networks. In several protocols from former exams I read that the activation functions of neurons (in multilayer perceptrons) have to be monotonic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that activation functions should be differentiable, have a derivative which is not 0 on most points, and be non-linear. I do not understand why being monotonic is important/helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the following activation functions and that they are monotonic:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ReLU&lt;/li&gt;&#xA;&lt;li&gt;Sigmoid&lt;/li&gt;&#xA;&lt;li&gt;Tanh&lt;/li&gt;&#xA;&lt;li&gt;Softmax: I'm not sure if the definition of monotonicity is applicable for functions &lt;span class=&quot;math-container&quot;&gt;$f: \mathbb{R}^n \rightarrow \mathbb{R}^m$&lt;/span&gt; with &lt;span class=&quot;math-container&quot;&gt;$n, m &amp;gt; 1$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;Softplus&lt;/li&gt;&#xA;&lt;li&gt;(Identity)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I still can't see any reason why for example &lt;span class=&quot;math-container&quot;&gt;$\varphi(x) = x^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do activation functions have to be monotonic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Related side question: is there any reason why the logarithm/exponential function is not used as an activation function?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2019-06-08T03:34:00.167" LastActivityDate="2019-08-04T12:24:25.033" Title="Why do activation functions have to be monotonic?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="9234" PostTypeId="2" ParentId="9231" CreationDate="2015-12-06T11:57:37.600" Score="1" Body="&lt;p&gt;A good answer to this question has to rely on the specific dataset / domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions I would ask myself are (in this order):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can I solve my classification task without those features? → Just remove those features&lt;/li&gt;&#xA;&lt;li&gt;Do I know of a relationship between features I know and the (partially) missing feature? → Find hard rules to fill those missing ones&lt;/li&gt;&#xA;&lt;li&gt;Otherwise: Train a classifier / regression model to fill those missing features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-06T11:57:37.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9235" PostTypeId="2" ParentId="5349" CreationDate="2015-12-06T12:17:40.473" Score="1" Body="&lt;h2&gt;SOMs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Self-organizing maps (SOMs) are one specific type of neural networks. In contrast to multilayer perceptrons (MLPs; they are used much more often) the SOMs neurons have a position on a regular grid.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Training of SOMs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;SOMs are usually trained in a stochastic way (source: [1]). This means one training example is used at a time. After every single training example the network learns something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, batch learning means the network is presented the complete batch of all training-examples (or a big part of usually 128 examples; then you call it mini-batch). The parameters are only updated after all of the examples were presented to the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast to MLPs, you don't apply gradient descent (as far as I know; I've never used them, but only read about SOMs). I wouldn't even know to which function you would apply the gradient descent algorithm, because there is no error function in SOMs. (At least no standard error function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For MLPs, I know that the advantage of batch training compared to stochastic training is that the weights are not jumping as much. So you rather go in the right direction. However, the advantage of stochastic training is that you make updates much more often. You want to get the best compromise, so you make mini-batch training.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Online Learning&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&quot;Online&quot; has quite a lot of different meanings:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&quot;online&quot; as in &quot;in the internet&quot;.&lt;/li&gt;&#xA;&lt;li&gt;&quot;online&quot; as in &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Online_algorithm&quot; rel=&quot;nofollow&quot;&gt;online algorithms&lt;/a&gt;&quot; which get input piece-by-piece and have to make a decision before knowing all of the input (e.g. when you have severe time constraints or when you can't store all the data like in CERN where you have to throw most of the data away right after you get it)&lt;/li&gt;&#xA;&lt;li&gt;&quot;online&quot; in handwriting recognition in contrast to the offline OCR methods (see &lt;a href=&quot;http://arxiv.org/abs/1511.09030&quot; rel=&quot;nofollow&quot;&gt;On-line Recognition of Handwritten Mathematical Symbols&lt;/a&gt; if you're interested). Online methods use the way how symbols were written, offline methods only use the image.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;https://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Online machine learning is used when data becomes available in a sequential order to determine a mapping from data set corresponding labels. The difference between online learning and batch learning (or &quot;offline&quot; learning) techniques, is that in online learning the mapping is updated after the arrival of every new data point in a scale fashion, whereas batch techniques are used when one has access to the entire training data set at once.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Sources&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;[1] Andries P. Engelbrecht: Computational Intelligence: An Introduction. Page 62.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-06T12:17:40.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9236" PostTypeId="1" AcceptedAnswerId="9277" CreationDate="2015-12-06T17:35:35.983" Score="3" ViewCount="3937" Body="&lt;p&gt;In the simple example where I have &lt;strong&gt;n&lt;/strong&gt; input neurons, I can consider this to be a point in a n-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the output layer is just one neuron with value &lt;strong&gt;0&lt;/strong&gt; or &lt;strong&gt;1&lt;/strong&gt;, if I get convergence, the neural net should define a hyperplane dividing the two classes of points - those mapped to &lt;strong&gt;0&lt;/strong&gt; and those mapped to &lt;strong&gt;1&lt;/strong&gt;. How do I calculate this hyperplane using the &lt;strong&gt;(n,1)&lt;/strong&gt; matrix of weights I calculated?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="10663" LastEditDate="2015-12-06T22:04:12.220" LastActivityDate="2015-12-09T04:54:45.217" Title="How to draw a hyperplane using the weights calculated" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9237" PostTypeId="2" ParentId="9233" CreationDate="2015-12-06T18:23:15.390" Score="18" Body="&lt;p&gt;The monotonicity criterion helps the neural network to converge easier into an more accurate classifier. &#xA;See this &lt;a href=&quot;https://cs.stackexchange.com/questions/45281/should-activation-function-be-monotonic-in-neural-networks&quot;&gt;stackexchange answer&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Activation_function&quot; rel=&quot;noreferrer&quot;&gt;wikipedia article&lt;/a&gt; for further details and reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the monotonicity criterion is not mandatory for an activation function - It is also possible to train neural nets with non-monotonic activation functions. It just gets harder to optimize the neural network. &#xA;See &lt;a href=&quot;https://www.quora.com/Can-non-monotonic-activation-function-neural-networks-be-trained-using-the-same-backpropagation-algorithm-as-monotonic-activation-function-neural-networks&quot; rel=&quot;noreferrer&quot;&gt;Yoshua Bengio's answer&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14578" LastEditorUserId="-1" LastEditDate="2017-04-13T12:48:38.763" LastActivityDate="2015-12-06T18:23:15.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9238" PostTypeId="2" ParentId="9236" CreationDate="2015-12-06T20:48:55.103" Score="3" Body="&lt;p&gt;I think a better to way to think about a neural net as you have it written is that it is a function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{nn} : \mathbb{R}^n \rightarrow \{0, 1\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is generally a non-linear decision boundary. You did not indicate your network structure and activation function. If your data is linearly separable then a single layer perceptron will define weights for you in the input layer, plus a bias input node for the intercept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you specifically want a separating hyperplane you could consider support vector machines. See here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/8021219/431895&quot;&gt;https://stackoverflow.com/a/8021219/431895&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-12-06T20:48:55.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9239" PostTypeId="2" ParentId="9208" CreationDate="2015-12-07T01:03:55.753" Score="6" Body="&lt;p&gt;You might want to take into consideration that &lt;a href=&quot;https://github.com/lisa-lab/pylearn2/pull/1565&quot;&gt;Pylearn2 has no more developer&lt;/a&gt;, and now &lt;a href=&quot;https://github.com/lisa-lab/pylearn2/commit/48380daba195db8cf2a53ec86139a1ae7af89d7b#diff-88b99bb28683bd5b7e3a204826ead112&quot;&gt;points to other Theano-based libraries&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are other machine learning frameworks built on top of Theano that could interest you, such as: Blocks, Keras and Lasagne.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As Dawny33 says, TensorFlow is just getting started, but it is interesting to note that the number of questions on TensorFlow (244) on Stack Overflow already surpasses Torch (166) and will probably catch up with Theano (672) in  2016.&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-12-07T01:03:55.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9240" PostTypeId="2" ParentId="694" CreationDate="2015-12-07T01:10:34.357" Score="5" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/NervanaSystems/neon&quot; rel=&quot;noreferrer&quot;&gt;neon&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python based&lt;/li&gt;&#xA;&lt;li&gt;Open source (Apache 2.0 License)&lt;/li&gt;&#xA;&lt;li&gt;Developed by &lt;a href=&quot;http://www.nervanasys.com/&quot; rel=&quot;noreferrer&quot;&gt;Nervana Systems&lt;/a&gt; (if you don't know them: &lt;a href=&quot;http://venturebeat.com/2015/06/04/deep-learning-startup-nervana-raises-20-5m/&quot; rel=&quot;noreferrer&quot;&gt;Deep learning startup Nervana raises $20.5M&lt;/a&gt;;  &lt;a href=&quot;https://gigaom.com/2014/08/21/nervana-systems-raises-3-3m-to-build-hardware-designed-for-deep-learning/&quot; rel=&quot;noreferrer&quot;&gt;Nervana Systems raises 3.3M USD to build hardware designed for deep learning&lt;/a&gt; )&lt;/li&gt;&#xA;&lt;li&gt;Supports CPU and GPU &lt;/li&gt;&#xA;&lt;li&gt;Basic automatic differentiation support&lt;/li&gt;&#xA;&lt;li&gt;Framework for visualization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-12-07T01:10:34.357" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="9241" PostTypeId="1" CreationDate="2015-12-07T01:34:51.480" Score="7" ViewCount="1349" Body="&lt;p&gt;I've been going through the &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;noreferrer&quot;&gt;Standford/Coursera Machine Learning course&lt;/a&gt;; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).&lt;/p&gt;&#xA;&lt;p&gt;The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.&lt;/p&gt;&#xA;&lt;p&gt;Here's a brief run through of what I do;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Set all Theta's (weights) to a small random number&#xA;&#xA;for each training example&#xA;&#xA;set activation 0 on layer 0 as 1 (bias)&#xA;set layer 1 activations = inputs&#xA;&#xA;forward propagate;&#xA; Z(j+1) = Theta(j) x activation(j)  [matrix operations]&#xA; activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid]&#xA;&#xA;Set Hx = final layer activations&#xA;&#xA;Set bias of each layer (activation 0,0) = 1&#xA;&#xA;[back propagate]&#xA;calculate delta;&#xA;delta(last layer) = activation(last layer) - Y  [Y is the expected answer from training set]&#xA;&#xA;delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j))&#xA;&#xA;[where ones is a matrix of 1's in every cell; and .* is the element wise multiplication]&#xA;[Don't calculate delta(0) since there ins't one for input layer]&#xA;&#xA;&#xA;DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j))&#xA;&#xA;Next [End for]&#xA;&#xA;Calculate D;&#xA;&#xA;D(j) = 1/#Training * DeltaCap(j) (for j = 0)&#xA;&#xA;D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0)&#xA;&#xA;&#xA;[calculate cost function]&#xA;&#xA;J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2&#xA;&#xA;Recalculate Theta&#xA;&#xA;Theta = Theta - alpha * D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.&lt;/p&gt;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&lt;p&gt;Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xIcEi.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xIcEi.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.&lt;/p&gt;&#xA;&lt;p&gt;Edit #2;&lt;/p&gt;&#xA;&lt;p&gt;I've ran a gradient checking algorithm with the following sort of results;&lt;/p&gt;&#xA;&lt;p&gt;Numerical calculation: -0.0074962585205895493&#xA;Value from propagation: 0.62021047431540277&lt;/p&gt;&#xA;&lt;p&gt;Numerical calculation: 0.0032635827218463476&#xA;Value from propagation: -0.39564819922432665&lt;/p&gt;&#xA;&lt;p&gt;etc. Clearly there's something wrong here; I'm going to work through it.&lt;/p&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-02-06T04:58:36.187" Title="Troubleshooting Neural Network Implementation" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9242" PostTypeId="2" ParentId="8244" CreationDate="2015-12-07T06:16:39.233" Score="0" Body="&lt;p&gt;Note: This is my question, and I'm really thankful for the wonderful answers here, which helped me grasp the concept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I would explain the concept the way which I have understood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, the data in the databases are stored in the memory in the following formats:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this datum:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           X1         X2&#xA;1   0.7091409 -1.4061361&#xA;2  -1.1334614 -0.1973846&#xA;3   2.3343391 -0.4385071&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a relational row-based store, it is stored like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1, 0.7091409, -1.4061361, 2, -1.1334614, -0.1973846, 3, 2.3343391, -0.4385071 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in the form of rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the columnar store, it would be stored like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1, 2, 3, 0.7091409 ,-1.1334614, 2.3343391, -1.4061361, -0.1973846, -0.4385071&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in the form of columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So, what does this mean?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that insertion(and updating) and deletions are fast in the row-based column store as it is just removal of the last few values or the first few values.  However, it is not the case in columnar stores as the value in each block store needs to be removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when there is the need for columnar aggregates and operations, the columnar stores have an edge over their row-based counterparts, as they are stored column-wise, and as a result, accessing individual columns is very easy.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-07T06:16:39.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9243" PostTypeId="2" ParentId="683" CreationDate="2015-12-07T07:08:39.933" Score="4" Body="&lt;p&gt;Your problem belongs to the framework of &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow&quot;&gt;PU learning&lt;/a&gt; (only positives, a lot of unlabelled).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also close to the more common frameworks of  &lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;nofollow&quot;&gt;Semi supervised learning&lt;/a&gt; (few positives and negatives, a lot of unlabeled).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many survey papers that you can look up on the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A classical method in the field, that was also tested on spam as in your case, is &lt;a href=&quot;https://en.wikipedia.org/wiki/Co-training&quot; rel=&quot;nofollow&quot;&gt;co-training&lt;/a&gt;&#xA;In co training you build two independent learners (e.g, one based on the mail content and one based on the sending scheme) and you use the results of one of the to train the other and vice versa.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-12-07T07:08:39.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9245" PostTypeId="1" CreationDate="2015-12-07T11:08:27.253" Score="1" ViewCount="30" Body="&lt;p&gt;I want to use several models to find a relation btw. life satisfaction and several independent variables like children, unemployment, marriage and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dependent var has an ordinal scale [1:10] and the independent vars have nominal and ordinal scales but no metric ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would set up an ordered model like an ordered probit, but I do not know what else I could use. Could you give me some hints?&lt;/p&gt;&#xA;" OwnerUserId="14596" LastEditorUserId="11097" LastEditDate="2015-12-07T11:14:34.293" LastActivityDate="2015-12-07T11:14:34.293" Title="which model for ordinal scaled dependent variable" Tags="|data-mining|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9247" PostTypeId="1" CreationDate="2015-12-07T12:10:37.997" Score="1" ViewCount="65" Body="&lt;p&gt;I am working with an algorithm studying the response of a system to an external factor (Phenomenon A). The system can either respond perfectly, or can be completely disrupted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am plotting the system's response in the following graph, which shows kind of an &lt;code&gt;oscillating trend&lt;/code&gt;. High &lt;code&gt;y&lt;/code&gt; values imply poor responses (upper limit: &lt;code&gt;100&lt;/code&gt;, &lt;code&gt;total collapse&lt;/code&gt;), whereas low &lt;code&gt;y&lt;/code&gt; values mean very good responses (lower limit: &lt;code&gt;0&lt;/code&gt;, &lt;code&gt;no influence&lt;/code&gt; whatsoever).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running this experiment based on an arbitrary number of simulations - in this case, 1000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question:&lt;/strong&gt; is there a metric which can help me identify the most appropriate number of simulations? I was thinking of something which could associate an optimal number of runs with a reduction in oscillation (&lt;code&gt;convergence&lt;/code&gt;). What metric/formula could be useful here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nwxgI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nwxgI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13481" LastActivityDate="2015-12-07T12:10:37.997" Title="Convergence line for oscillating trend" Tags="|predictive-modeling|simulation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9248" PostTypeId="1" CreationDate="2015-12-07T15:40:19.207" Score="4" ViewCount="10077" Body="&lt;p&gt;Say I have a dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hotel  HasPool AvgPrice&#xA; 1        1      $123&#xA; 2        0      $234 &#xA; 3        1      $200     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Currently I have broken down the dataset into 2 (one containing all continuous, other all categorical). The continuous one, I am calculating euclidean distance b/w all observation whereas for the categorical one I calculate Cosine distance . Is there a way to combine both scores effectively. Or, is there a distance function which works for both data types ?&#xA;My final output should look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hotel  1   2   3&#xA;  1    1  0.3  0.7 &#xA;  2    0.3  1  0.5 &#xA;  3    0.7  0.5 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated! Thanks&lt;/p&gt;&#xA;" OwnerUserId="14427" LastEditorUserId="14427" LastEditDate="2015-12-07T15:57:14.730" LastActivityDate="2020-03-16T08:42:04.633" Title="How to find similarity/distance matrix with mixed Continuous and Categorical data?" Tags="|similarity|cosine-distance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9249" PostTypeId="1" AcceptedAnswerId="23399" CreationDate="2015-12-07T16:42:04.107" Score="24" ViewCount="10461" Body="&lt;p&gt;&lt;a href=&quot;http://keras.io/&quot;&gt;Keras&lt;/a&gt; supports both &lt;strong&gt;TensorFlow&lt;/strong&gt; and &lt;strong&gt;Theano&lt;/strong&gt; as backend: what are the pros/cons of choosing one versus the other, besides the fact that currently not all operations are implemented with the TensorFlow backend?&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="10663" LastEditDate="2015-12-07T19:32:09.237" LastActivityDate="2020-08-05T09:21:31.550" Title="Choosing between TensorFlow or Theano as backend for Keras" Tags="|neural-network|deep-learning|theano|tensorflow|keras|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9250" PostTypeId="5" CreationDate="2015-12-07T16:43:44.823" Score="0" Body="&lt;h2&gt;What is Keras?&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; is a popular, open-source deep learning API for Python built on top of &lt;a href=&quot;https://www.tensorflow.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow&lt;/a&gt; and is useful for fast implementation. Topics include efficient low-level tensor operations, computation of arbitrary gradients, scalable computations, export of graphs, etc.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;New to Keras?&lt;/h2&gt;&#xA;&lt;p&gt;There are various resources including books, tutorials/workshops, etc. for those looking to learn how to use Keras.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Introductions on the Keras website:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/getting_started/intro_to_keras_for_engineers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Keras for Engineers&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/getting_started/intro_to_keras_for_researchers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Keras for Researchers&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A popular introductory book is:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1617294438&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning with Python&lt;/a&gt;&lt;/em&gt;, by François Chollet.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;&lt;a href=&quot;/questions/tagged/keras&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;keras&amp;#39;&quot; rel=&quot;tag&quot;&gt;keras&lt;/a&gt; Tag usage&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;When posting questions about Keras, please take the following into consideration:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When tagging questions with the &lt;a href=&quot;/questions/tagged/keras&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;keras&amp;#39;&quot; rel=&quot;tag&quot;&gt;keras&lt;/a&gt; tag, users should make sure to post sufficient information regarding model construction (layers, input shape, activation functions, etc.) and include all other relevant tags that pertain to the specific topic at hand.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explicit programming related questions are more suitable for Stack Overflow and should not be posted on Stack Exchange Data Science.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Questions should include sufficient details and clarity to be able to provide support for the problem at hand. This includes linking to underlying data used, providing code used for the model's construction, highlighting relevant outputs, etc.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;External Resources&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt;: Documentation Page&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h2&gt;Important Links&lt;/h2&gt;&#xA;&lt;p&gt;Issue Tracker: &lt;a href=&quot;https://github.com/keras-team/keras/issues&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/issues&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2021-04-01T21:24:29.687" LastActivityDate="2021-04-01T21:24:29.687" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9251" PostTypeId="4" CreationDate="2015-12-07T16:43:44.823" Score="0" Body="Keras is a popular, open-source deep learning API for Python built on top of TensorFlow and is useful for fast implementation. Topics include efficient low-level tensor operations, computation of arbitrary gradients, scalable computations, export of graphs, etc." OwnerUserId="843" LastEditorUserId="29169" LastEditDate="2021-04-01T21:24:14.097" LastActivityDate="2021-04-01T21:24:14.097" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9252" PostTypeId="5" CreationDate="2015-12-07T18:36:03.537" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2015-12-07T18:36:03.537" LastActivityDate="2015-12-07T18:36:03.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9253" PostTypeId="4" CreationDate="2015-12-07T18:36:03.537" Score="0" Body="Azure Machine Learning offers a streamlined experience for all data scientist skill levels, from setting up with only a web browser, to using drag and drop gestures and simple data flow graphs to set up experiments. Can create amusing visualizations on the data." OwnerUserId="843" LastEditorUserId="40853" LastEditDate="2017-12-07T06:12:34.223" LastActivityDate="2017-12-07T06:12:34.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9255" PostTypeId="1" CreationDate="2015-12-07T21:39:27.877" Score="12" ViewCount="91651" Body="&lt;p&gt;I have a pandas data frame (X11) like this: In actual I have 99 columns up to dx99&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    dx1      dx2    dx3    dx4&#xA;0   25041   40391   5856    0&#xA;1   25041   40391   25081   5856&#xA;2   25041   40391   42822   0&#xA;3   25061   40391   0       0&#xA;4   25041   40391   0       5856&#xA;5   40391   25002   5856    3569&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to create additional column(s) for cell values like 25041,40391,5856 etc. So there will be a column 25041 with value as 1 or 0 if 25041 occurs in that particular row in any dxs columns. I am using this code and it works when number of rows are less.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mat = X11.as_matrix(columns=None)&#xA;values, counts = np.unique(mat.astype(str), return_counts=True)&#xA;&#xA;for x in values:&#xA;    X11[x] = X11.isin([x]).any(1).astype(int)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting result like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dx1     dx2     dx3    dx4  0   25002   25041   25061   25081   3569    40391   42822   5856&#xA;25041   40391   5856    0   0   0       1       0       0       0          1        0       1&#xA;25041   40391   25081  5856 0   0       1       0       1       0            1      0       1&#xA;25041   40391   42822   0   0   0       1       0       0       0           1       1       0&#xA;25061   40391   0       0   0   0       0       1       0       0          1        0       0&#xA;25041   40391   0    5856   0   0       1       0       0       0          1        0       1&#xA;40391   25002 5856   3569   0   1       0       0       0       1          1        0       1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When number of rows are many thousands or in millions, it hangs and takes forever and I am not getting any result. Please see that cell values are not unique to column, instead repeating in multi columns. For ex, 40391 is occurring in dx1 as well as in dx2 and so on for 0 and 5856 etc. Any idea how to improve the logic mentioned above?&lt;/p&gt;&#xA;" OwnerUserId="14604" LastEditorUserId="14604" LastEditDate="2015-12-21T17:50:55.737" LastActivityDate="2015-12-21T17:50:55.737" Title="Creating new columns by iterating over rows in pandas dataframe" Tags="|python|data-cleaning|pandas|anaconda|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9256" PostTypeId="2" ParentId="6744" CreationDate="2015-12-07T23:29:01.027" Score="1" Body="&lt;p&gt;Depending on the language of your choice, you may or may not have to normalise the data yourself - for example the &lt;code&gt;e1071&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt; does this automatically for you. As this is built off the 'libsvm' library, it might be that this is also the case there - the library documentation is your best source. As for the normalisation values, you should definitely use the same [min,max] values from the &lt;strong&gt;training&lt;/strong&gt; set for the test set as well. Also, I would reccomend reducing the dimensionality of your data first, than normalising before running a svm. hth &lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2015-12-07T23:29:01.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9257" PostTypeId="2" ParentId="6642" CreationDate="2015-12-08T02:36:55.850" Score="1" Body="&lt;p&gt;No you cannot. I had a discussion with someone in their development/support team on the MSDN forums and currently they don't support 'drag and drop' type of functionality. However you CAN serialize the model output and then de-serialize them in Azure. &lt;a href=&quot;https://i.stack.imgur.com/QIILN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QIILN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note that the answer in the image is a bit outdated&lt;/strong&gt; and there is the 'Create R Script' module to replace the serialization-deserialization steps &lt;em&gt;within&lt;/em&gt; Azure. However I believe you can still serialize outside Asure (in your Desktop version of R) and deserialize them in Azure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link to the conversation in Image:&#xA;&lt;a href=&quot;https://social.msdn.microsoft.com/Forums/azure/en-US/5944c342-79ac-4ada-8006-8edf40f36ee1/r-script-as-a-trained-model?forum=MachineLearning&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://social.msdn.microsoft.com/Forums/azure/en-US/5944c342-79ac-4ada-8006-8edf40f36ee1/r-script-as-a-trained-model?forum=MachineLearning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14427" LastActivityDate="2015-12-08T02:36:55.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9258" PostTypeId="2" ParentId="9255" CreationDate="2015-12-08T03:16:43.023" Score="5" Body="&lt;p&gt;It looks like you want to create dummy variable from a pandas dataframe column. Fortunately, pandas has a special method for it: &lt;code&gt;get_dummies()&lt;/code&gt;. &#xA;Here is a code snippet that you can adapt for your need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;data = pd.read_clipboard(sep=',')&#xA;&#xA;#get the names of the first 3 columns&#xA;colN = data.columns.values[:3]&#xA;&#xA;#make a copy of the dataframe&#xA;data_transformed = data&#xA;&#xA;#the get_dummies method is doing the job for you&#xA;for column_name in colN:&#xA;    dummies = pd.get_dummies(data_transformed[column_name], prefix='value', prefix_sep='_')&#xA;    col_names_dummies = dummies.columns.values&#xA;&#xA;    #then you can append new columns to the dataframe&#xA;    for i,value in enumerate(col_names_dummies):&#xA;        data_transformed[value] = dummies.iloc[:,i]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the output of &lt;code&gt;data_transformed&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         dx1    dx2    dx3   dx4    dx5    dx6    dx7  value_25041  value_25061  0  25041  40391   5856     0  V4511  V5867  30000            1            0   &#xA;    1  25041  40391  25081  5856   5363   3572      0            1            0   &#xA;    2  25041  40391  42822     0   5856      0      0            1            0   &#xA;    3  25061  40391      0     0      0      0      0            0            1   &#xA;    4  25041  40391      0  5856  25081  V4511  25051            1            0   &#xA;&#xA;      value_40391  value_0  value_5856  value_25081  value_42822  &#xA;    0            1        0           1            0            0  &#xA;    1            1        0           0            1            0  &#xA;    2            1        0           0            0            1  &#xA;    3            1        1           0            0            0  &#xA;    4            1        1           0            0            0  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-12-08T03:16:43.023" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9259" PostTypeId="2" ParentId="9241" CreationDate="2015-12-08T04:02:56.427" Score="1" Body="&lt;p&gt;Are your inputs &lt;a href=&quot;https://stackoverflow.com/questions/4674623/why-do-we-have-to-normalize-the-input-for-an-artificial-neural-network&quot;&gt;scaled&lt;/a&gt;? Not doing so can cause the weights to immediately blow up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's common practice to preprocess your input to be between -1 and 1 or at least in that range. Otherwise you are running the risk of your gradients exploding or vanishing (covered &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap5.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2015-12-08T04:02:56.427" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9260" PostTypeId="2" ParentId="9249" CreationDate="2015-12-08T06:07:07.413" Score="9" Body="&lt;p&gt;If I were given the option, &lt;strong&gt;I would go with Theano&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;State of the art RNN implementations and API's&lt;/strong&gt;, which TensorFlow unfortunately doesn't possess, and &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/7&quot; rel=&quot;noreferrer&quot;&gt;has a long way to go&lt;/a&gt;. And in a domain where RNN's are the trending rage, Theano has a big edge there.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Very wide range of implementations&lt;/strong&gt;. TensorFlow has a long way to catch up. A lot of recent ML models have been done with the help of Theano, so it is something like a standard when it comes to neural networks.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimized and &lt;a href=&quot;http://deeplearning.net/software/theano/tutorial/loop.html&quot; rel=&quot;noreferrer&quot;&gt;improved looping&lt;/a&gt;:&lt;/strong&gt; Theano's scan is a wonderful way for looping in neural networks, which makes use of the awesome map-reduce framework. But, I'm pretty sure TensorFlow would improve on this as its creator Jeff Dean is the daddy of Map Reduce.  However, as of now; it's Theano&lt;/li&gt;&#xA;&lt;li&gt;Huge edge when it comes to video analytics.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, TensorFlow supports both the cpp and Python interfaces which might be an advantage with the cpp community. But, when it comes to ML and data science products, Python has been the standard, so it wouldn't be a huge edge IMO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, model deployment and ease of use in production is where TensorFlow has the real advantage. As it uses Eigen for improved and easy deployment, it would be a darling for engineers. If it gets compatible with Windows, then you'll see a huge migration. But, I've got used to the Python overhead, I can wait till it gets more polished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Theano for now. I can happily wait for TensorFlow to catch up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are deploying simple to average complexity neural networks, go with Tensorflow.  If deep learning, then Theano.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="843" LastEditDate="2015-12-08T14:32:41.897" LastActivityDate="2015-12-08T14:32:41.897" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9261" PostTypeId="2" ParentId="9255" CreationDate="2015-12-08T06:12:59.660" Score="7" Body="&lt;p&gt;There's a much more pythonic solution in pandas...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This takes less than a second on 10 Million rows on my laptop:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for x in X11.E.unique():&#xA;    X11[x]=(X11.E==x).astype(int)&#xA;X11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here are the details laid out:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple small dataframe - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD'))&#xA;X11['E'] = [25223, 112233,25223,14333,14333,112233]&#xA;X11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DBgSz.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DBgSz.png&quot; alt=&quot;simple small dataframe&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binarization method - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for x in X11.E.unique():&#xA;    X11[x]=(X11.E==x).astype(int)&#xA;X11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NANqV.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NANqV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataframe with 10 Million rows - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.set_option(&quot;display.max_rows&quot;,20)&#xA;X12 = pd.DataFrame(np.random.randn(10000000,4), columns=list('ABCD'))&#xA;foo = [25223, 112233,25223,14333,14333,112233]&#xA;bar=[]&#xA;import random&#xA;for x in range(10000000):&#xA;    bar.append(random.choice(foo))&#xA;X12['E'] = bar&#xA;X12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hJAAV.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hJAAV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Timed binarization (aka one-hot encoding) on 10 million row dataframe - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import time&#xA;start = time.clock()&#xA;&#xA;for x in X12.E.unique():&#xA;    X12[x]=(X12.E==x).astype(int)&#xA;elapsed = (time.clock() - start)&#xA;&#xA;print &quot;This is the time that this took in seconds: &quot;,elapsed&#xA;&#xA;X12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vFkob.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vFkob.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2015-12-19T17:51:54.010" LastActivityDate="2015-12-19T17:51:54.010" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9262" PostTypeId="1" AcceptedAnswerId="9271" CreationDate="2015-12-08T10:37:44.050" Score="46" ViewCount="112485" Body="&lt;p&gt;I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I am using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; to try and do this. The problem that I'm running into is that the value returned is the same for any 2 lists of numbers (its 1.3862943611198906). I have a feeling that I'm making some sort of theoretical mistake here but can't spot it.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;values1 = [1.346112,1.337432,1.246655]&#xA;values2 = [1.033836,1.082015,1.117323]&#xA;metrics.mutual_info_score(values1,values2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;That is an example of what I'm running - just that I'm getting the same output for any 2 input. Any advice/help would be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="8206" LastEditorUserId="98307" LastEditDate="2020-08-03T13:41:37.703" LastActivityDate="2021-05-21T20:18:52.677" Title="Calculating KL Divergence in Python" Tags="|python|clustering|scikit-learn|" AnswerCount="6" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="9264" PostTypeId="2" ParentId="9262" CreationDate="2015-12-08T10:59:33.703" Score="15" Body="&lt;p&gt;I'm not sure with the scikit-learn implementation, but here is a quick implementation of the KL divergence in Python:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;def KL(a, b):&#xA;    a = np.asarray(a, dtype=np.float)&#xA;    b = np.asarray(b, dtype=np.float)&#xA;&#xA;    return np.sum(np.where(a != 0, a * np.log(a / b), 0))&#xA;&#xA;&#xA;values1 = [1.346112,1.337432,1.246655]&#xA;values2 = [1.033836,1.082015,1.117323]&#xA;&#xA;print KL(values1, values2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output:  &lt;code&gt;0.775279624079&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;There might be &lt;a href=&quot;https://datascience.stackexchange.com/a/8429/11097&quot;&gt;conflict of implementation&lt;/a&gt; in some libraries, so make sure you read their docs before using.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="29169" LastEditDate="2021-02-16T20:09:37.360" LastActivityDate="2021-02-16T20:09:37.360" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="9265" PostTypeId="2" ParentId="9262" CreationDate="2015-12-08T13:19:10.360" Score="25" Body="&lt;p&gt;Scipy's &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html&quot; rel=&quot;noreferrer&quot;&gt;entropy function&lt;/a&gt; will calculate KL divergence if feed two vectors p and q, each representing a probability distribution. If the two vectors aren't pdfs, it will normalize then first.&lt;/p&gt;&#xA;&lt;p&gt;Mutual information is &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;noreferrer&quot;&gt;related to, but not the same&lt;/a&gt; as KL Divergence.&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;This weighted mutual information is a form of weighted KL-Divergence, which is known to take negative values for some inputs, and there are examples where the weighted mutual information also takes negative values&amp;quot;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="118071" LastEditDate="2021-05-21T20:18:52.677" LastActivityDate="2021-05-21T20:18:52.677" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9266" PostTypeId="1" AcceptedAnswerId="9267" CreationDate="2015-12-08T14:12:05.660" Score="3" ViewCount="737" Body="&lt;p&gt;I am trying to understand some simple neural net case using theano.&#xA;The deeplearning.net site gives the following simple code for implementing a logistic regression application to a simple case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import theano&#xA;import theano.tensor as T&#xA;rng = numpy.random&#xA;&#xA;N = 400&#xA;feats = 784&#xA;D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=2))&#xA;training_steps = 10000&#xA;&#xA;# Declare Theano symbolic variables&#xA;x = T.matrix(&quot;x&quot;)&#xA;y = T.vector(&quot;y&quot;)&#xA;w = theano.shared(rng.randn(feats), name=&quot;w&quot;)&#xA;b = theano.shared(0., name=&quot;b&quot;)&#xA;print(&quot;Initial model:&quot;)&#xA;print(w.get_value())&#xA;print(b.get_value())&#xA;&#xA;# Construct Theano expression graph&#xA;p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b))   # Probability that target = 1&#xA;prediction = p_1 &amp;gt; 0.5                    # The prediction thresholded&#xA;xent = -y * T.log(p_1) - (1-y) * T.log(1-p_1) # Cross-entropy loss function&#xA;cost = xent.mean() + 0.01 * (w ** 2).sum()# The cost to minimize&#xA;gw, gb = T.grad(cost, [w, b])             # Compute the gradient of the cost&#xA;                                          # (we shall return to this in a&#xA;                                          # following section of this tutorial)&#xA;&#xA;# Compile&#xA;train = theano.function(&#xA;          inputs=[x,y],&#xA;          outputs=[prediction, xent],&#xA;          updates=((w, w - 0.1 * gw), (b, b - 0.1 * gb)))&#xA;predict = theano.function(inputs=[x], outputs=prediction)&#xA;&#xA;# Train&#xA;for i in range(training_steps):&#xA;    pred, err = train(D[0], D[1])&#xA;&#xA;print(&quot;Final model:&quot;)&#xA;print(w.get_value())&#xA;print(b.get_value())&#xA;print(&quot;target values for D:&quot;)&#xA;print(D[1])&#xA;print(&quot;prediction on D:&quot;)&#xA;print(predict(D[0]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand most of it, p_1 is the logistic regression function, the prediction is whether the value will be in the 0 class or 1 class, xent is the loss function, i.e. how far from correct is our prediction. I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss? &#xA;What is the cost function representing here?&#xA;Also, why is the bias initially set to 0 and not a random number like the weights?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2015-12-08T14:40:38.523" Title="Theano logistic regression example" Tags="|python|logistic-regression|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9267" PostTypeId="2" ParentId="9266" CreationDate="2015-12-08T14:40:38.523" Score="6" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss? What is the cost function representing here? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The cost is the error (xent.mean()) + some regularization (0.01 * (w ** 2).sum())&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why is the bias initially set to 0 and not a random number like the weights?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details &lt;a href=&quot;http://cs231n.github.io/neural-networks-2/#init&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-12-08T14:40:38.523" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9268" PostTypeId="1" CreationDate="2015-12-08T16:37:35.857" Score="7" ViewCount="4786" Body="&lt;p&gt;Can you explain what is finite precision? Why is finite precision a problem in machine learning?&lt;/p&gt;&#xA;" OwnerUserId="1151" LastEditorDisplayName="user10640" LastEditDate="2019-02-13T03:14:16.283" LastActivityDate="2019-02-13T06:26:11.607" Title="Why is finite precision a problem in machine learning?" Tags="|machine-learning|terminology|definitions|finite-precision|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9269" PostTypeId="1" CreationDate="2015-12-08T17:49:04.497" Score="3" ViewCount="597" Body="&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4781121&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; describes a technique for making recommendations when the feedback is implicit, that is, $r_{ui}$ is only a guess. The recommendation problems boils down to the following optimisation problem:&#xA;$$&#xA;\min_{x^*, y^*}\ \ \sum_{u,i}c_{ui}(p_{ui}-x_i^Ty_i)^2+\lambda \left(\sum_u\|x_u\|^2+\|y_u\|^2 \right) &#xA;$$&#xA;where $c_{ui} = 1+\alpha r_{ui}$.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following concerns:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The algorithm above approximates the binary indicator function $p_{ui}$, weighted by the term $c_{ui}$. However, most people use RMSE between $r_{ui}$ and $x_i^Ty_i$ for assessing performance. I am not sure why that would make sense. &lt;/li&gt;&#xA;&lt;li&gt;Why using RMSE at all? With implicit feedback, it would make more sense to me to use rank-based metrics. I would imagine RMSE is more suitable in machine learning competition where the objective metric is usually RMSE.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14455" LastEditorUserId="14455" LastEditDate="2015-12-10T13:49:10.643" LastActivityDate="2018-03-27T11:21:32.357" Title="Performance metric in recommender systems with implicit feedback" Tags="|bigdata|recommender-system|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9271" PostTypeId="2" ParentId="9262" CreationDate="2015-12-08T22:29:45.167" Score="31" Body="&lt;p&gt;First of all, &lt;code&gt;sklearn.metrics.mutual_info_score&lt;/code&gt; implements &lt;em&gt;mutual information&lt;/em&gt; for evaluating clustering results, not &lt;em&gt;pure&lt;/em&gt; Kullback-Leibler divergence!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is equal to the Kullback-Leibler divergence of the joint distribution with the product distribution of the marginals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;KL divergence (and any other such measure) expects the input data to &lt;strong&gt;have a sum of 1&lt;/strong&gt;. Otherwise, they are &lt;em&gt;not&lt;/em&gt; proper &lt;strong&gt;probability distributions&lt;/strong&gt;. If your data does not have a sum of 1, most likely it is usually not proper to use KL divergence! (In some cases, it may be admissible to have a sum of less than 1, e.g. in the case of missing data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that it is common to use base 2 logarithms. This only yields a constant scaling factor in difference, but base 2 logarithms are easier to interpret and have a more intuitive scale (0 to 1 instead of 0 to log2=0.69314..., measuring the information in bits instead of nats).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sklearn.metrics.mutual_info_score([0,1],[1,0])&#xA;0.69314718055994529&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as we can clearly see, the MI result of sklearn is scaled using natural logarithms instead of log2. This is an unfortunate choice, as explained above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kullback-Leibler divergence is fragile, unfortunately. On above example it is not well-defined: &lt;code&gt;KL([0,1],[1,0])&lt;/code&gt; causes a division by zero, and tends to infinity. It is also &lt;em&gt;asymmetric&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2015-12-09T17:29:04.477" LastActivityDate="2015-12-09T17:29:04.477" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9272" PostTypeId="2" ParentId="9169" CreationDate="2015-12-09T02:27:46.880" Score="1" Body="&lt;p&gt;In the vein of bayesian optimization, I prefer Hyperopt, available on github at &lt;a href=&quot;https://github.com/hyperopt/hyperopt&quot; rel=&quot;nofollow&quot;&gt;https://github.com/hyperopt/hyperopt&lt;/a&gt; or through pip, homepage of author at &lt;a href=&quot;https://github.com/hyperopt/hyperopt&quot; rel=&quot;nofollow&quot;&gt;https://github.com/hyperopt/hyperopt&lt;/a&gt;. The tree-parzen-estimator algorithm behind it is described in the paper at &lt;a href=&quot;http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf&quot; rel=&quot;nofollow&quot;&gt;http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf&lt;/a&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can define an arbitrary nested search space, and then tell it to find the optimum of a black-box function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hyperopt also supports a built-in sklearn search space out of the box, but I have not used that, I've typically defined my own. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also a few other bayesian optimization schemes around. I can't claim to know which one of them will come out on top as the &quot;best&quot; in the long run. &lt;/p&gt;&#xA;" OwnerUserId="7786" LastActivityDate="2015-12-09T02:27:46.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9273" PostTypeId="2" ParentId="9268" CreationDate="2015-12-09T02:41:37.340" Score="4" Body="&lt;p&gt;Finite precision is decimal representation of a number which has been rounded or truncated. There many cases where this may be necessary or appropriate. For example 1/3 and the transcendental numbers $e$ and $\pi$ all have infinite decimal representations. In the programming language C, a double value is 8 bit and precise to approximately 16 digits. See here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.learncpp.com/cpp-tutorial/25-floating-point-numbers/&quot; rel=&quot;noreferrer&quot;&gt;http://www.learncpp.com/cpp-tutorial/25-floating-point-numbers/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To concretely represent one of these numbers on a (finite) computer there must be some sort of compromise. We could write 1/3 to 9 digits as .333333333 which is less than 1/3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These compromises are compounded with arithmetic operations. Unstable algorithms are prone to arithmetic errors. This is why SVD is often used to compute PCA (instability of the covariance matrix). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sandia.gov/~smartin/presentations/SMartin_Stability.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.sandia.gov/~smartin/presentations/SMartin_Stability.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Numerical_stability&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Numerical_stability&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the naive bayes classifier you will often see multiplication transformed into a sum of logarithms, which is less prone to rounding errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-09T02:41:37.340" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9274" PostTypeId="2" ParentId="8753" CreationDate="2015-12-09T03:02:02.420" Score="1" Body="&lt;p&gt;For the former, gensim has the Word2Vec class. For the latter, &lt;a href=&quot;http://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2Vec&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorUserId="98307" LastEditDate="2020-08-06T13:01:29.880" LastActivityDate="2020-08-06T13:01:29.880" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9275" PostTypeId="1" CreationDate="2015-12-09T04:10:02.643" Score="4" ViewCount="6118" Body="&lt;p&gt;I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.”&#xA;(&lt;a href=&quot;http://blog.minitab.com/blog/adventures-in-statistics/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit&quot; rel=&quot;nofollow&quot;&gt;Source&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since kNN is not a linear model (it is nonparametric), is it appropriate to use $R^2$ to evaluate it? If not, what error metric would be appropriate? I'm using the kNN classifier provided by scikit-learn in python.&lt;/p&gt;&#xA;" OwnerUserId="838" LastEditorUserId="11097" LastEditDate="2015-12-15T10:52:50.037" LastActivityDate="2015-12-20T00:21:27.720" Title="Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?" Tags="|classification|python|scikit-learn|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9276" PostTypeId="2" ParentId="9248" CreationDate="2015-12-09T04:51:55.550" Score="0" Body="&lt;p&gt;To combine the cosine distance and euclidian distance, I would first normalize the euclidian distance on the same scale as the cosine distance [o-1], and then combine them by average. But I am sure you already try that approach. Thus I am wondering what is your definition of &quot;combine both scores effectively&quot; ? Do you need to weight differently the two metrics? How did you came to the final matrix you are showing? Is it intuition? If so, you can investigate this further to better define the weights you want to put on each feature and it associated metrics.&lt;/p&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-12-09T04:51:55.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9277" PostTypeId="2" ParentId="9236" CreationDate="2015-12-09T04:54:45.217" Score="3" Body="&lt;p&gt;If you have only an input layer, one set of weights, and an output layer, you can solve this directly with $$ X \cdot w = threshold $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if you add in hidden layers, you no longer necessarily have a &lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperplane&quot; rel=&quot;nofollow&quot;&gt;hyperplane&lt;/a&gt;, as in order to be a hyperplane it must be able to be expressed as the &quot;solution of a single algebraic equation of degree 1.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you can't solve directly, you can still get a sense for the response surface by evaluating your network's output over a wide range of inputs. &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-09T04:54:45.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9278" PostTypeId="2" ParentId="8753" CreationDate="2015-12-09T06:19:59.753" Score="18" Body="&lt;p&gt;The answer to this question is that &lt;em&gt;it depends&lt;/em&gt;. The primary approach is to pass in the tokenized sentences (so &lt;code&gt;SentenceCorpus&lt;/code&gt; in your example), but depending on what your goal is and what the corpus is you're looking at, you might want to instead use the entire article to learn the embeddings. This is something you might not know ahead of time -- so you'll have to think about how you want to evaluate the quality of the embeddings, and do some experiments to see which 'kind' of embeddings are more useful for your task(s).&lt;/p&gt;&#xA;" OwnerUserId="11136" LastActivityDate="2015-12-09T06:19:59.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9279" PostTypeId="2" ParentId="8606" CreationDate="2015-12-09T10:08:11.757" Score="2" Body="&lt;p&gt;I've tried the following algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf&quot; rel=&quot;nofollow&quot;&gt;Factorization Machines&lt;/a&gt; by Steffen Rendle&lt;/strong&gt; - &#xA;Really good algorithm for sparse feature sets, in construct to polynomial-regression.&#xA;Measures the interaction between features. Model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing interactions between features in this new space. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Field-Aware Factorization Machines&lt;/strong&gt; - an improvement of the FM model.&#xA;Recently, have been used to win two Kaggle's click-through rate prediction competitions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;FTRL - &quot;Follow The (Proximally) Regularized Leader&quot; algorithm&lt;/strong&gt; - regularized online logistic regression. Equivalent to Online (Stochastic) Gradient Descent when no regularization is used. Very easy to implement. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://research.microsoft.com/apps/pubs/?id=122779&quot; rel=&quot;nofollow&quot;&gt;AdPredictor&lt;/a&gt; algorithm by Microsoft&lt;/strong&gt; &#xA;The algorithm is based on a probit regression model that maps discrete or real-valued input features to probabilities. It maintains Gaussian beliefs over weights of the model and performs Gaussian online updates derived from approximate message passing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1093" LastEditorUserId="1093" LastEditDate="2015-12-09T13:53:28.953" LastActivityDate="2015-12-09T13:53:28.953" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9280" PostTypeId="2" ParentId="9197" CreationDate="2015-12-09T10:26:11.503" Score="0" Body="&lt;p&gt;For simple extrapolation, use the &lt;code&gt;predict()&lt;/code&gt; function; call it with  the &lt;code&gt;newdata=&lt;/code&gt; Argument. This is a dataframe with the attribute values you for the next n years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, check the &lt;code&gt;forecast()&lt;/code&gt; function from the &lt;code&gt;forecast&lt;/code&gt; package. It is recommended by the R cheatsheet &lt;a href=&quot;https://s3.amazonaws.com/quandl-static-content/Documents/Quandl+-+R+Cheat+Sheet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="16853" LastEditDate="2016-12-23T15:24:02.080" LastActivityDate="2016-12-23T15:24:02.080" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9281" PostTypeId="1" CreationDate="2015-12-09T11:29:23.027" Score="1" ViewCount="90" Body="&lt;p&gt;From the tutorial slides: &lt;a href=&quot;http://mt-class.org/jhu/slides/lecture-tuning.pdf&quot; rel=&quot;nofollow&quot;&gt;http://mt-class.org/jhu/slides/lecture-tuning.pdf&lt;/a&gt;, (slide 37) the powell search algorithm goes as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: sentences with n-best list of translations, initial parameter values&#xA;1: repeat&#xA;2:   for all parameter do&#xA;3:     set of threshold points T = {}&#xA;4:     for all sentence do&#xA;5:       for all translation do&#xA;6:         compute line l: parameter value → score&#xA;7:       end for&#xA;8:       find line l with steepest descent&#xA;9:       while find line l2 that intersects with l first do&#xA;10:        add parameter value at intersection to set of threshold points T&#xA;11:        l = l2&#xA;12:      end while&#xA;13:    end for&#xA;14:    sort set of threshold points T by parameter value&#xA;15:    compute score for value before first threshold point&#xA;16:    for all threshold point t ∈ T do&#xA;17:      compute score for value after threshold point t&#xA;18:      if highest do record max score and threshold point t&#xA;19:    end for&#xA;20:    if max score is higher than current do update parameter value&#xA;21:    end for&#xA;22: until no changes to parameter values applied&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And on slide 31, there's a formula to &lt;code&gt;compute the line l&lt;/code&gt; which is used for line 6 of the pseudo code above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p(ei|f) = λai + bi&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it's unclear what should the values be for &lt;code&gt;ai&lt;/code&gt; and &lt;code&gt;bi&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a sentence from the nbest list as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 ||| including options , чтобы buy 20 больше planes , соотношении volume - в том 26 миллиардов долларов .  ||| LexicalReordering0= -3.10978 0 0 -6.72232 0 0 Distortion0= 0 LM0= -111.207 WordPenalty0= -18 PhrasePenalty0= 16 TranslationModel0= -13.1426 -8.45991 -12.0762 -11.3076 ||| -746.351&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When computing the score for this line for the first parameter of &lt;code&gt;LexicalReordering0&lt;/code&gt;, is the &lt;code&gt;p(ei|f)&lt;/code&gt; this?:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p(ei|f) = λ*-3.10978 + bi&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But what is the value of &lt;code&gt;bi&lt;/code&gt; in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is &lt;code&gt;p(ei|f) = -746.351&lt;/code&gt; for this sentence ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the &lt;code&gt;→ score&lt;/code&gt; on line 6 of the pseudo code mean -746.351 from the nbest list or does it mean the MT evaluation metric score for this translation with respect to the reference translation (e.g. BLEU)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What should line 6 of the pseudo code do?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="122" LastEditDate="2015-12-09T14:34:45.460" LastActivityDate="2015-12-09T14:34:45.460" Title="Minimum Error Rate Training using Powell Search for Machine Translation" Tags="|machine-learning|algorithms|machine-translation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9282" PostTypeId="2" ParentId="8606" CreationDate="2015-12-09T14:30:16.700" Score="2" Body="&lt;p&gt;You can see nice algorithms in kaggle competitions about CTR:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/c/avito-context-ad-clicks&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/avito-context-ad-clicks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/c/avazu-ctr-prediction&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/avazu-ctr-prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just go to forum of each competition and search for winning solutions ;)&lt;/p&gt;&#xA;" OwnerUserId="14636" LastActivityDate="2015-12-09T14:30:16.700" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9283" PostTypeId="2" ParentId="6088" CreationDate="2015-12-09T14:43:35.640" Score="0" Body="&lt;p&gt;I think that it depends on your data set. There are many ways to handle unbalanced data sets, just search, for example &lt;a href=&quot;https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set&quot; rel=&quot;nofollow&quot;&gt;https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set&lt;/a&gt; . I think that the simplest way is to use the same distribution of classes in the train and test sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have really small amount of minority class you can try &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow&quot;&gt;one-class classification&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastEditorUserId="11097" LastEditDate="2015-12-09T16:03:33.357" LastActivityDate="2015-12-09T16:03:33.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9284" PostTypeId="2" ParentId="9275" CreationDate="2015-12-09T14:49:27.510" Score="0" Body="&lt;p&gt;$R^2$ is for data with a numerical range. Classification by k Nearest Neighbours assigns class labels that are just labels (even if you choose them to be numbers, they aren't like real numbers). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You use kNN in a supervised setting, typical quality assessment consists in splitting up your data in training and test sets (n-fold cross validation) and determining precision, recall, and F-score.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2015-12-09T14:49:27.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9285" PostTypeId="1" CreationDate="2015-12-09T17:10:27.163" Score="3" ViewCount="43" Body="&lt;p&gt;I'm trying to find the term for a type of calculations or values that cannot be simpply added or multiplied when zooming in or out from a temporal scale. I know it's not very clear, if it were I would just google the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example: say you're comparing 2014 vs 2015 sales of ice cream and temperature at day's level. If you want to do an analysis by week, you can just add the sales for seven days, and compare weeks between them, the information still makes sense: there's a fact (amount of money deposited in the bank that week) that is grounded in reality that relate to the new figure. You could also calculate an average by day from the total by week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, adding temperatures for a total would not make any sense in reality, you need to calculate an average for that kind of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is there a term for this type of values that can only make sense as average and never be added or multiplied (except as an intermediate step for calculating an average)?&lt;/p&gt;&#xA;" OwnerUserId="9503" LastEditorUserId="9503" LastEditDate="2015-12-09T22:10:32.397" LastActivityDate="2015-12-10T20:02:49.097" Title="Term for calculated values that lose pertinence when changing scale" Tags="|terminology|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9287" PostTypeId="2" ParentId="8199" CreationDate="2015-12-09T18:56:08.837" Score="2" Body="&lt;p&gt;I found a PhD in Energy Informatics at &lt;a href=&quot;http://www.energystatusdata.kit.edu/index.php&quot; rel=&quot;nofollow&quot;&gt;Karlsruhe Institute of Technology (KIT)&lt;/a&gt; and PhD in Learning Analytics at &lt;a href=&quot;http://utscic.edu.au/research/phd/&quot; rel=&quot;nofollow&quot;&gt;University of Technology Sydney&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-12-16T17:38:10.840" LastActivityDate="2015-12-16T17:38:10.840" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9288" PostTypeId="1" CreationDate="2015-12-09T19:26:25.173" Score="2" ViewCount="272" Body="&lt;p&gt;I'm searching a book as a refresher in machine learning (I have taken a lecture in machine learning sometime ago). I will be applying machine learning in a project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have searched a lot of books and found the following three the one which best fits me:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Applied Predictive Modeling by Kuhn and Johnson&lt;/li&gt;&#xA;&lt;li&gt;Machine learning an Algorithmic Perspective (second edition) by Marsland&lt;/li&gt;&#xA;&lt;li&gt;An Introduction to Statistical Learning by Hastie&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Applied Predictive Modeling and An Introduction to Statistical Learning uses R as examples and Machine learning an Algorithmic Perspective uses Python. The used language does not matter that much because I'm not really interested in language examples (I'm using Matlab) and I will probably skip such examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of Applied Predictive Modeling seems to be that it covers the whole machine learning procedure (feature selection etc.) and seems to be very well written. The advantage of Machine learning an Algorithmic Perspective (second edition) is that it covers more topics (ensemble learning, graphical models, gaussian processes) and has perhaps a bit more math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which one of these three books would you recommend (and why)?&lt;/p&gt;&#xA;" OwnerUserId="14641" LastEditorUserId="11097" LastEditDate="2015-12-10T03:33:00.013" LastActivityDate="2015-12-10T03:33:00.013" Title="Which Machine Learning book to choose (APM, MLAP or ISL)?" Tags="|machine-learning|reference-request|books|" AnswerCount="1" CommentCount="0" ClosedDate="2015-12-10T12:39:42.000" ContentLicense="CC BY-SA 3.0" />
  <row Id="9289" PostTypeId="2" ParentId="9285" CreationDate="2015-12-09T19:31:14.810" Score="0" Body="&lt;p&gt;I don't get the assertion that a sum of temperature loses pertinence&lt;br&gt;&#xA;$$&#xA;\text{avg}(T)  = \frac{\text{sum}(T)}{n}&#xA;$$&#xA;$$&#xA;\frac{\text{avg}(\text{Revenue})}{\text{avg}(\text{Temp})} = \frac{\frac{\text{sum}(\text{Revenue})}{ n}}{\frac{\text{sum}(\text{Temp})}{n}} = \frac{\text{sum}(\text{Revenue})}{\text{sum}(\text{Temp}) }&#xA;$$&#xA;Now multiply does lack meaning if the value is not 0 based&lt;br&gt;&#xA;200 degrees °F (Fahrenheit) is not twice as hot 100 °F&lt;br&gt;&#xA;As °F is not zero based&lt;br&gt;&#xA;You need to translate to absolute - °R (Rankine)&lt;br&gt;&#xA;(200 + 459.67) / (100 + 459.67) = 1.17867672&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure what term you want to call it but I think you are describing linear zero based properties   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like $\log$ is not linear $\frac{\log(a)}{\log(b)} \ne \frac{a}{b}$ (even if $a$ and $b$ are zero based)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spacial data does not add, subtract, or average in a simple linear manner.&lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="2981" LastEditDate="2015-12-10T07:41:20.520" LastActivityDate="2015-12-10T07:41:20.520" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9290" PostTypeId="1" CreationDate="2015-12-09T19:34:03.510" Score="3" ViewCount="2192" Body="&lt;p&gt;I have a Pandas dataframe X(20346, 4116). All independent columns have binary variable as 0 or 1. Whereas dependent column has continuous variable. When I execute below code using the scikit-learn library, it takes for ever and I am not getting any result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X13, y, train_size=0.8)&#xA;&#xA;rf = RandomForestRegressor(n_estimators=100,  max_features='auto', oob_score=True,&#xA;                            n_jobs=1, verbose=0)&#xA;rf.fit(X_train, y_train) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using this for feature selection. In case of less number of rows I can get result. But when number of rows go up then this problem starts happening. Do you have any idea how to make it work?&lt;/p&gt;&#xA;" OwnerUserId="14604" LastEditorUserId="9420" LastEditDate="2016-01-09T06:39:26.507" LastActivityDate="2016-01-10T08:05:09.040" Title="Scikit Learn's RandomForestRegressor is not giving results on large data set" Tags="|python|random-forest|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9292" PostTypeId="1" CreationDate="2015-12-09T22:12:27.573" Score="4" ViewCount="1449" Body="&lt;p&gt;Does anyone know of a predictive model that can combine the linear regression model and time series model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some data about some products. The data has two parts, some attributes about the products themselves, and some time series data of month-by-month sales amounts for each product. We want to predict future month-by-month sales amount for each product. with the products attributes data, we can build a regression model, or an svm model, to predict sales amounts. With the time series sales amount, we can build a time series model to predict sales amount, but is there any predictive model that can combine the regression, svm, and time series models to do the prediction?&lt;/p&gt;&#xA;" OwnerUserId="14642" LastEditorUserId="13413" LastEditDate="2015-12-10T22:22:08.847" LastActivityDate="2018-01-10T14:23:16.647" Title="Combining Linear Regression and Time Series" Tags="|time-series|regression|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9293" PostTypeId="2" ParentId="8762" CreationDate="2015-12-09T22:17:04.550" Score="2" Body="&lt;p&gt;Like someone else mentioned, treating IPs as int automatically gives higher bits higher weights.&#xA;I've used variance of IPs which is log scaled.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;math.log(np.std([IP(ip).int() for ip in ips]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14643" LastActivityDate="2015-12-09T22:17:04.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9294" PostTypeId="1" AcceptedAnswerId="9313" CreationDate="2015-12-09T22:24:43.523" Score="1" ViewCount="1078" Body="&lt;p&gt;In order to build a predict model with two categories (buy or not buy),I want to use RandomForest and predict with type='prob', so I can have a prob of someone buy or not buy. So, with this outcome I can clusterize and make groups, like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;group A: costumer who has [100 to 80]% of buy.&#xA;group B: costumer who has [81 to 60]% of buy.&#xA;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't know the appropriate evaluation metric to measure the accuracy of this model. I guess that I can't use a confusion matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe I can use a ROC curve, and or measure KS between the buy group with the not buy group. But I'm not sure about this metrics. &lt;/p&gt;&#xA;" OwnerUserId="14644" LastActivityDate="2015-12-10T17:02:37.683" Title="What is the appropriate evaluation metric for RandomForest with probability in R?" Tags="|r|predictive-modeling|random-forest|probability|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9295" PostTypeId="2" ParentId="9294" CreationDate="2015-12-09T22:48:12.113" Score="0" Body="&lt;p&gt;Group the entires by the probability score you assigned them (e.g. 100-95%, 95-90%,....) and then measure the likelihood of their purchases from ground truth.  You can then perform a Chi-squared calculation between your predicted and measured outcomes.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2015-12-09T22:48:12.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9296" PostTypeId="1" CreationDate="2015-12-10T00:53:03.687" Score="1" ViewCount="604" Body="&lt;p&gt;I have a problem to name data processing step.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step name?&lt;/li&gt;&#xA;&lt;li&gt;I have an attribute that contain 200 tweets of a twitter user. I make a new attribute that describe URL ratio in 200 tweets (number of URL in 200 tweet / 200). is it aggregation or feature creation or else?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14647" LastEditorUserId="11097" LastEditDate="2015-12-10T03:32:28.750" LastActivityDate="2015-12-10T03:32:28.750" Title="Data preprocessing : Aggregation, feature creation, or else?" Tags="|data-mining|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9297" PostTypeId="2" ParentId="9296" CreationDate="2015-12-10T01:08:47.270" Score="1" Body="&lt;p&gt;You are looking for names to attribute to the two items listed? For (1) I would just call it a transformation as it is a straight mapping with no change in distribution of the feature. When working with data, make sure you make copies of your data transformation and do not alter the original data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For (2), since it is a single number per group, where group here is the full data set I would call it an aggregation. Likewise if you did a similar calculation per user. If however, you computed a new value from existing features for each record, this would be feature generation or creation. &lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-10T01:08:47.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9298" PostTypeId="2" ParentId="9290" CreationDate="2015-12-10T01:20:42.543" Score="0" Body="&lt;p&gt;Have you tried increasing  n_jobs or setting oob_score to False? I am not sure what the logic is in sklearn to default max_features to auto (=n_features). Using sqrt is common in random forest and should gain efficiency.&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-10T01:20:42.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9299" PostTypeId="2" ParentId="9288" CreationDate="2015-12-10T03:26:52.917" Score="0" Body="&lt;p&gt;&lt;strong&gt;An Introduction to Statistical Learning by Hastie&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is like a bible for all those who want to get started at statistical analysis and Machine Learning.  It gives an excellent intuition about the theory behind all those models and algorithms, enough to pursue research and/or dive into another advanced ML book like the &lt;code&gt;Applied Predictive Modeling by Kuhn and Johnson&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-10T03:26:52.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9300" PostTypeId="2" ParentId="9296" CreationDate="2015-12-10T03:30:23.020" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I have attribute that contain string or null. i want to change the&#xA;  record of attribute to 0 if null and 1 if not null. What preprocessing&#xA;  step name?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This step is &lt;a href=&quot;http://www.ats.ucla.edu/stat/mult_pkg/faq/general/dummy.htm&quot; rel=&quot;nofollow&quot;&gt;dummy coding&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;i have attribute that contain 200 tweet of twitter user. i make new&#xA;  attribute that describe URL ratio in 200 tweet (number of URL in 200&#xA;  tweet / 200). is it aggregation or feature creation or else?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Feature aggregation&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-10T03:30:23.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9301" PostTypeId="1" AcceptedAnswerId="9323" CreationDate="2015-12-10T05:25:41.537" Score="1" ViewCount="795" Body="&lt;p&gt;I am doing linear regression with multiple variables. In my data I have &lt;strong&gt;n = 143&lt;/strong&gt; features and &lt;strong&gt;m = 13000&lt;/strong&gt; training examples. Some of my features are continuous (ordinal) variables (area, year, number of rooms). But I also have categorical variables (district, color, type). For now I visualized some of my feautures against predicted price. For example here is the plot of &lt;code&gt;area&lt;/code&gt; against predicted &lt;code&gt;price&lt;/code&gt;:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/n3Mc7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/n3Mc7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since &lt;code&gt;area&lt;/code&gt; is continuous ordinal variable I had no troubles visualizing the data. But now I wanted to somehow visualize dependency of my categorical variables (such as district) on predicted price.&#xA;For categorical variables I used one-hot (dummy) encoding. &#xA;&lt;br&gt;For example that kind of data:&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/hoFqt.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hoFqt.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;turned to this format:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/jQBuN.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jQBuN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were using ordinal encoding for districts this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DistrictA - 1&#xA;DistrictB - 2&#xA;DistrictC - 3&#xA;DistrictD - 4&#xA;DistrictE - 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would plot this values against predicted price pretty easy by putting 1-5 to X axis and price to Y axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I used dummy coding and now I do not know how can I show (visualize) dependency between price and categorical variable 'District' represented as series of zeros and ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I make a plot showing a regression line of districts against predicted price in case of using dummy coding?&lt;/p&gt;&#xA;" OwnerUserId="14525" LastActivityDate="2015-12-10T22:42:41.480" Title="How to visualize (make plot) of regression output against categorical input variable?" Tags="|machine-learning|visualization|categorical-data|linear-regression|" AnswerCount="1" CommentCount="2" ClosedDate="2016-08-29T09:34:07.820" ContentLicense="CC BY-SA 3.0" />
  <row Id="9302" PostTypeId="1" CreationDate="2015-12-10T06:22:48.927" Score="151" ViewCount="165980" Body="&lt;p&gt;In the &lt;a href=&quot;https://www.tensorflow.org/versions/master/tutorials/mnist/beginners/index.html&quot; rel=&quot;noreferrer&quot;&gt;MNIST For ML Beginners&lt;/a&gt; they define cross-entropy as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question 1&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an &quot;obvious&quot; &lt;code&gt;1&lt;/code&gt; labeled as &lt;code&gt;3&lt;/code&gt;. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question 2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I've learned that cross-entropy is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="50743" LastEditDate="2018-04-19T19:17:28.040" LastActivityDate="2021-09-11T16:22:44.930" Title="The cross-entropy error function in neural networks" Tags="|machine-learning|tensorflow|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9303" PostTypeId="1" CreationDate="2015-12-10T08:53:58.557" Score="0" ViewCount="161" Body="&lt;p&gt;I am a recent graduate.Currently I work in a late stage startup (product company) as a Software Engineer where my responsibilities are to interact with customers, understand their environment and write code so that they can use our product. I work on C#, web services and relational databases. I also work rarely on Big Data space to migrate some small components of our product to big data platform. I've got an opportunity to work in a large firm as a big data engineer to work on apache projects like yarn and hadoop and solve customer issues. And also to contribute to open source community. In curreny company, there is no one to review my work and mentor me. &#xA;Will it be a wise decision to leave a company where I've an opportunity to see product from end to end and go to a company where I'll not the same visibility of product ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my next opportunity a good one if I want to make my career in big data space ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please share your inputs to help me take my decision ?&lt;/p&gt;&#xA;" OwnerUserId="14656" LastEditorUserId="11097" LastEditDate="2015-12-10T10:33:30.590" LastActivityDate="2015-12-10T11:41:52.540" Title="Seeking Career guidance" Tags="|bigdata|" AnswerCount="3" CommentCount="1" ClosedDate="2015-12-10T12:39:48.020" ContentLicense="CC BY-SA 3.0" />
  <row Id="9304" PostTypeId="2" ParentId="9303" CreationDate="2015-12-10T09:25:34.180" Score="3" Body="&lt;p&gt;I think your question is off-topic, but I'll still try to answer because it might help you. Getting a job in data science isn't something everybody can do, it's a competitive field. Well done, for landing yourself in data science. However, data science is a big area, so you'll need to think what you want to do and plan ahead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you want to get yourself into machine learning? Do you want to get yourself into statistics? Right now, you're more like a software engineer in the field, still not quite being a data scientist. Do you want to become a real data scientist? Or are you happy being a software engineer in the field? If you're planning to become a data scientist, you should train yourself hard in statistics, machine learning etc. Being a Hadoop user is simply not enough to step yourself fully into the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I give you my own experience. I was a software engineer myself. It wasn't easy to get a job in data science. I had to do a postgraduate degree in mathematics. Trained myself as much as possible. In the end, I landed myself a job as a bioinformatician.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Recommendation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't leave just because nobody mentors you. Learn as much as possible and prepare yourself. Once you're comfortable, move on.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2015-12-10T09:25:34.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9305" PostTypeId="2" ParentId="9303" CreationDate="2015-12-10T10:09:58.930" Score="0" Body="&lt;p&gt;I am voting to close it as off-topic.  (StudentT has already given a wonderful answer :) )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, it is up to you to take the decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to have a passion for data, and willing to get your feet dirty at it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not the end, if your colleagues are not mentoring you. No one mentored me when I was preparing myself. I have written &lt;a href=&quot;https://www.quora.com/How-can-I-find-mentors-to-guide-me-to-become-a-data-scientist-at-Facebook-Google-or-LinkedIn/answer/Jalem-Raj-Rohit&quot; rel=&quot;nofollow&quot;&gt;an answer on Quora&lt;/a&gt; about how I disturbed a lot of people till they mentored me. In fact, they were very happy to.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Will it be a wise decision to leave a company where I've an&#xA;  opportunity to see product from end to end and go to a company where&#xA;  I'll not the same visibility of product ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The domain of data science is very difficult. Let me cite an example from my team.  Me, being the lead, have an idea of the entire product end-to-end, however the ones in my team working on customer success analytics don't have the complete idea on the entire product(in fact, they don't need to), and vice versa.  So, it is completely dependant on the role.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is my next opportunity a good one if I want to make my career in big&#xA;  data space ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Big data space?  &lt;strong&gt;Yes&lt;/strong&gt;  In data science? Maybe, but you have a &lt;a href=&quot;http://nirvacana.com/thoughts/becoming-a-data-scientist/&quot; rel=&quot;nofollow&quot;&gt;lot of ground to &#xA;cover&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2015-12-10T11:41:52.540" LastActivityDate="2015-12-10T11:41:52.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9306" PostTypeId="2" ParentId="9303" CreationDate="2015-12-10T10:40:41.863" Score="1" Body="&lt;p&gt;Reading this short free eBook might give you some insights on what to do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920029014.do&quot; rel=&quot;nofollow&quot;&gt;Analyzing the Analyzers&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-12-10T10:40:41.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9307" PostTypeId="1" CreationDate="2015-12-10T12:50:10.270" Score="4" ViewCount="71746" Body="&lt;p&gt;I am getting the error in the subject line for the following piece of code in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r&amp;lt;- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)&#xA;for (u in 1:nrow(r))&#xA;  for (x in 1:ncol(r))&#xA;    r_hat&amp;lt;- function(u,x) {&#xA;&#xA;&#xA;      if (r_hat &amp;gt;r[u,x]) {&#xA;        print(r[u,x]-r_hat)&#xA;      }&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I get rid of the error?&lt;/p&gt;&#xA;" OwnerUserId="14427" LastEditorUserId="1352" LastEditDate="2016-04-07T15:24:51.160" LastActivityDate="2023-08-17T03:34:57.057" Title="R: comparison (6) is possible only for atomic and list types" Tags="|r|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9308" PostTypeId="1" AcceptedAnswerId="9310" CreationDate="2015-12-10T13:41:33.530" Score="4" ViewCount="119" Body="&lt;p&gt;Are there any web services that can be used to analyse data in social networks with respect to a specific research question (e.g. mentioning of certain products in social media discussions)?&lt;/p&gt;&#xA;" OwnerUserId="14660" LastEditorUserId="13023" LastEditDate="2015-12-11T12:03:10.920" LastActivityDate="2015-12-11T12:03:10.920" Title="Web services to mine the social web?" Tags="|data-mining|social-network-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9309" PostTypeId="2" ParentId="9302" CreationDate="2015-12-10T14:08:05.680" Score="7" Body="&lt;p&gt;Those issues are handled by the tutorial's use of softmax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 1) you're correct that softmax guarantees a non-zero output because it exponentiates it's input. For activations that do not give this guarantee (like relu), it's simple to add a very small positive term to every output to avoid that problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for 2), they aren't the same obviously, but I the softmax formulation they gave takes care of the the issue. If you didn't use softmax, this would cause you to learn huge bias terms that guess 1 for every class for any input. But since they normalize the softmax across all classes, the only way to maximize the output of the correct class is for it to be large relative to the incorrect classes.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-10T14:08:05.680" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9310" PostTypeId="2" ParentId="9308" CreationDate="2015-12-10T15:48:15.770" Score="3" Body="&lt;p&gt;Twitter's API is one of the best sources of social network data. You can extract off twitter pretty much everything you can imagine, you just need an account and a developer ID. The documentation is rather big so I will let you navigate it.&#xA;&lt;a href=&quot;https://dev.twitter.com/overview/documentation&quot; rel=&quot;nofollow&quot;&gt;https://dev.twitter.com/overview/documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As usual there are wrappers that make your life easier.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/bear/python-twitter&quot; rel=&quot;nofollow&quot;&gt;python-twitter&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/twitteR/index.html&quot; rel=&quot;nofollow&quot;&gt;twitteR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are also companies who offer detailed twitter analytics and historic datasets for a fee.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://gnip.com/historical/&quot; rel=&quot;nofollow&quot;&gt;Gnip&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://datasift.com/products/stream-for-social-data/&quot; rel=&quot;nofollow&quot;&gt;Datasift&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Check them out!&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-12-10T15:48:15.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9311" PostTypeId="2" ParentId="9302" CreationDate="2015-12-10T16:10:21.970" Score="32" Body="&lt;p&gt;The first logloss formula you are using is for multiclass log loss, where the $i$ subscript enumerates the different classes in an example. The formula &lt;em&gt;assumes&lt;/em&gt; that a single $y_i'$ in each example is 1, and the rest are all 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That means the formula only captures error on the target class. It discards any notion of errors that you might consider &quot;false positive&quot; and does not care how predicted probabilities are distributed other than predicted probability of the true class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another assumption is that $\sum_i y_i = 1$ for the predictions of each example. A softmax layer does this automatically - if you use something different you will need to scale the outputs to meet that constraint.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question 1&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Isn't it a problem that the $y_i$ (in $log(y_i)$) could be 0?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes that can be a problem, but it is usually not a practical one. A randomly-initialised softmax layer is extremely unlikely to output an exact &lt;code&gt;0&lt;/code&gt; in any class. But it is possible, so worth allowing for it. First, don't evaluate $log(y_i)$ for any $y_i'=0$, because the negative classes always contribute 0 to the error. Second, in practical code you can limit the value to something like &lt;code&gt;log( max( y_predict, 1e-15 ) )&lt;/code&gt; for numerical stability - in many cases it is not required, but this is sensible defensive programming.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Question 2&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I've learned that cross-entropy is defined as $H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This formulation is often used for a network with one output predicting two classes (usually positive class membership for 1 and negative for 0 output). In that case $i$ may only have one value - you can lose the sum over $i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you modify such a network to have two opposing outputs and use softmax plus the first logloss definition, then you can see that in fact it &lt;em&gt;is the same error measurement&lt;/em&gt; but folding the error metric for two classes into a single output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is more than one class to predict membership of, and the classes are &lt;em&gt;not exclusive&lt;/em&gt; i.e. an example could be any or all of the classes at the same time, then you will need to use this second formulation. For digit recognition that is not the case (a written digit should only have one &quot;true&quot; class)&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2015-12-17T09:40:51.580" LastActivityDate="2015-12-17T09:40:51.580" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="9313" PostTypeId="2" ParentId="9294" CreationDate="2015-12-10T17:02:37.683" Score="0" Body="&lt;p&gt;You should select some threshold, let's say 0.5 and treat customers with probability below threshold as not buy and above as buy. Based on this you can compute accuracy of your model. You can also check the ROC curve of the model. You can check various metric for binary classifier &lt;a href=&quot;https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14636" LastActivityDate="2015-12-10T17:02:37.683" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9314" PostTypeId="2" ParentId="9308" CreationDate="2015-12-10T17:45:32.630" Score="5" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/tweepy/tweepy&quot;&gt;Tweepy&lt;/a&gt; is one of the best libraries for analyzing and hacking around with the Twitter API.  (Being a contributor for tweepy, I can vouch for it's stability and quality)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a Python wrapper for the Facebook graph API, you can use the &lt;a href=&quot;https://github.com/debrouwere/facebook-insights&quot;&gt;Facebook-Insights library&lt;/a&gt;, which is well-maintained and neat documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are services out there which can mine you information, but they are limited to the complexity of the query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: &quot;How many people tweeted about banana on Monday?&quot; can be answered from existing tools&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;How many people happy people tweeted about banana on Monday&quot; can also be done, but would require more efforts and the software or tool should be able to detect emotion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you are into research, I would advise you to go with the API's and good ol' programming!&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-10T17:45:32.630" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9315" PostTypeId="2" ParentId="9285" CreationDate="2015-12-10T20:02:49.097" Score="1" Body="&lt;p&gt;Let us think of sales ($S$) as a property of a system.  You could imagine Week 1 to be a system and Week 2 to be another system.  Call the Week 1 system $W_1$ and the Week 2 system $W_2$.  Then,&#xA;$$&#xA;  S(W_1) + S(W_2) = S(W_1 + W_2)&#xA;$$&#xA;A property that has this additive character is called &lt;strong&gt;extensive&lt;/strong&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider the temperature ($T$) as the property with values $T(W_1)$ and $T(W_2)$ for weeks 1 and 2.  Assume that the two $T$ values are identical.  Merge the Week 1 and Week 2 systems together to get another system $W$ for the two weeks together.  Then&#xA;$$&#xA;  T(W) = T(W_1+W_2)&#xA;$$ &#xA;Properties that follow this rule are called &lt;strong&gt;intensive&lt;/strong&gt; and cannot be added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These definitions were invented for physical properties and systems and one has to be careful in applying the definitions to general situations.  I'm not sure how much thought along these lines has gone into the social sciences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more, see &lt;a href=&quot;https://en.wikipedia.org/wiki/Intensive_and_extensive_properties&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Intensive_and_extensive_properties&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2981" LastActivityDate="2015-12-10T20:02:49.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9316" PostTypeId="1" CreationDate="2015-12-10T20:22:54.097" Score="3" ViewCount="946" Body="&lt;p&gt;We have information about what the user likes in our app and we want to recommend content to similar users even those who may not have explicitly like a particular content but are similar to those who have explicitly liked that content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was planning to use the implicit feedback &lt;a href=&quot;http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4781121&quot; rel=&quot;nofollow&quot;&gt;variation&lt;/a&gt; of Alternating Least Squares using Apache Spark to implement a recommender system which will find me the additional users that I can target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the tuning parameters is a boolean flag &lt;code&gt;nonnegative&lt;/code&gt; for non-negative matrix factorization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be counting the number of likes per user and tag so I won't have negative values. Can I say &lt;code&gt;nonnegative=true&lt;/code&gt; or does it mean something completely different. My understanding is that the nonnegative constraint is for the values inside the two matrices which the algorithm will decompose my original matrix to,  but I don't have the knowledge to know if those values will be non-negative for my scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref:&#xA;&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;Spark ALS Algorithm&lt;/a&gt;. Not exactly since there is another API but good enough for this question.&lt;/p&gt;&#xA;" OwnerUserId="14667" LastActivityDate="2015-12-11T10:13:12.463" Title="How to determine Nonnegativity in Matrix Factorization?" Tags="|recommender-system|apache-spark|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9317" PostTypeId="1" CreationDate="2015-12-10T21:07:40.313" Score="2" ViewCount="16597" Body="&lt;p&gt;I am working in R. I have two vectors; A and B of lengths 5913 and 3733 respectively (with entries are repeated). I want to extract those values (with repetitions) that are present in both A and B. &#xA;I have done (A %in% B) (lets call it C) and (B %in% A) (lets call it D). &#xA;The length of C is 3906 and that of D is 3607 (so 2007 elements in A are not in B and 126 elements in B are not in A). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But How do I find the common values ? I don't think I can use intersect() method as it expects vectors with no duplicate values. I have many many repetitions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Due to the length of these vectors, I couldn't mention them here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2016-06-09T17:00:58.990" Title="How to get common values between two multi sets?" Tags="|r|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9318" PostTypeId="2" ParentId="9307" CreationDate="2015-12-10T21:45:17.327" Score="2" Body="&lt;p&gt;The code works and leads to no error. The result is a definition of one variable &lt;code&gt;r&lt;/code&gt;and one function &lt;code&gt;r_hat&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You get the same with this snippet (I removed the loop as not relevant, what you do is creating the same function repeatedly in a loop)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; r&amp;lt;- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)&#xA;&amp;gt; &#xA;&amp;gt; r_hat&amp;lt;- function(u,x) {&#xA;+       if (r_hat &amp;gt;r[u,x]) {&#xA;+         print(r[u,x]-r_hat)&#xA;+       }&#xA;+ } &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is in the function, after calling it you get the mentioned error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; r_hat(0,0)   &#xA;Error in r_hat &amp;gt; r[u, x] : &#xA;  comparison (6) is possible only for atomic and list types&#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You reference &lt;code&gt;r_hat&lt;/code&gt; within the function, are you trying to define a recursive function? Than the recursive call should have the parameters and the function some return value...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is you tries to compare the &lt;em&gt;function definition&lt;/em&gt; with an element of a matrix, which leads to this error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not enough for you and you explains what the code should deliver, I'll try to complete my answer.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-12-10T21:45:17.327" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9319" PostTypeId="2" ParentId="130" CreationDate="2015-12-10T21:49:06.973" Score="5" Body="&lt;p&gt;Feature selection is about choosing some of features based on some statistical score but feature extraction is using techniques to extract some second layer information from the data e.g. interesting frequencies of a signal using Fourier transform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dimensionality reduction is all about transforming data into a low-dimensional space in which data preserves its euclidean structure but does not suffer from curse of dimensionality.&#xA;For instance assume you extract some word features $[x_1,...,x_n]$ from a data set where each document can be modeled as a point in n-dimensional space and n is too large (a toy example). In this case many algorithms do not work according to the distance distortion of high-dimensional space. Now you need to reduce dimensionality by either selecting most informative features or transforming them into a low-dimensional manifold using dimensionality reduction methods e.g. PCA, LLE, etc.&lt;/p&gt;&#xA;" OwnerUserId="14672" LastActivityDate="2015-12-10T21:49:06.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9320" PostTypeId="2" ParentId="8825" CreationDate="2015-12-10T22:02:30.910" Score="1" Body="&lt;p&gt;Go for network analysis! make a network in which the links are relations and try to make network analysis to get your answer. Tones of tutorials on the web. You can have a look at GraphLab Create as a start.&lt;/p&gt;&#xA;" OwnerUserId="14672" LastActivityDate="2015-12-10T22:02:30.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9321" PostTypeId="1" AcceptedAnswerId="9333" CreationDate="2015-12-10T22:03:16.547" Score="0" ViewCount="458" Body="&lt;p&gt;I am trying to understand logistic regression for a multi-class example and I have the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import theano&#xA;import theano.tensor as T&#xA;rng = numpy.random&#xA;&#xA;num_classes = 3&#xA;#N = number of examples&#xA;N = 100&#xA;#feats = number of input neurons&#xA;feats = 784&#xA;#training rate&#xA;tr_rate = 0.1 &#xA;&#xA;D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=num_classes))&#xA;training_steps = 1000&#xA;&#xA;# Declare Theano symbolic variables&#xA;x = T.matrix(&quot;x&quot;)&#xA;y = T.ivector(&quot;y&quot;)&#xA;&#xA;w = theano.shared(rng.randn(feats), name=&quot;w&quot;)&#xA;b = theano.shared(0.01, name=&quot;b&quot;)&#xA;&#xA;# Construct Theano expression graph&#xA;sigma = T.nnet.softmax(T.dot(x,w) + b) &#xA;prediction = T.argmax(sigma, axis=1)          # The class with highest probability&#xA;&#xA;# Cross-entropy loss function&#xA;xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])&#xA;&#xA;cost = xent.mean() + 0.01 * (w ** 2).sum()      # Regularisation &#xA;gw, gb = T.grad(cost, [w, b])                   # Compute the gradient of the cost &#xA;&#xA;# Compile&#xA;train = theano.function(&#xA;          inputs=[x,y],&#xA;          outputs=[xent],&#xA;          updates=((w, w - tr_rate * gw), (b, b - tr_rate * gb)),&#xA;          allow_input_downcast=True)&#xA;predict = theano.function(inputs=[x], outputs=prediction)&#xA;&#xA;# Train&#xA;for i in range(training_steps):&#xA;    train(D[0], D[1])      &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However at this point the code gives me the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-246-c4753ce8ccc7&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 # Train&#xA;      2 for i in range(training_steps):&#xA;----&amp;gt; 3     train(D[0], D[1])&#xA;&#xA;/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)&#xA;    869                     node=self.fn.nodes[self.fn.position_of_error],&#xA;    870                     thunk=thunk,&#xA;--&amp;gt; 871                     storage_map=getattr(self.fn, 'storage_map', None))&#xA;    872             else:&#xA;    873                 # old-style linkers raise their own exceptions&#xA;&#xA;/Users/vzocca/gitProjects/Theano/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)&#xA;    312         # extra long error message in that case.&#xA;    313         pass&#xA;--&amp;gt; 314     reraise(exc_type, exc_value, exc_trace)&#xA;    315 &#xA;    316 &#xA;&#xA;/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)&#xA;    857         t0_fn = time.time()&#xA;    858         try:&#xA;--&amp;gt; 859             outputs = self.fn()&#xA;    860         except Exception:&#xA;    861             if hasattr(self.fn, 'position_of_error'):&#xA;&#xA;ValueError: number of rows in x (1) does not match length of y (100)&#xA;Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Elemwise{Add}[(0, 0)].0, Alloc.0, y)&#xA;Toposort index: 12&#xA;Inputs types: [TensorType(float64, row), TensorType(float64, vector), TensorType(int32, vector)]&#xA;Inputs shapes: [(1, 100), (100,), (100,)]&#xA;Inputs strides: [(800, 8), (8,), (4,)]&#xA;Inputs values: ['not shown', 'not shown', 'not shown']&#xA;Outputs clients: [[Sum{acc_dtype=float64}(CrossentropySoftmaxArgmax1HotWithBias.0)], [], []]&#xA;&#xA;Backtrace when the node is created:&#xA;  File &quot;&amp;lt;ipython-input-244-834f04b2fb36&amp;gt;&quot;, line 16, in &amp;lt;module&amp;gt;&#xA;    xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help understand what is wrong?&#xA;Thank you! &lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="11097" LastEditDate="2015-12-11T03:03:34.257" LastActivityDate="2015-12-11T07:41:24.760" Title="Multi-class logistic regression" Tags="|logistic-regression|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9322" PostTypeId="1" CreationDate="2015-12-10T22:13:30.213" Score="2" ViewCount="169" Body="&lt;p&gt;I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the logistic regression code available online, I am trying to understand this line of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is basically saying perform this average:&#xA;$\begin{align}&#xA;J(\theta) = - \left[ \sum_{i=1}^{b} \sum_{k=1}^{K}  1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right]&#xA;\end{align}$ where $b$ is the batch size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is it related to theano code, or it is just a python notation ?&#xA;I am interested exactly on this piece of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;T.log(self.p_y_given_x)[T.arange(y.shape[0]), y]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you could please give me some explanation.&lt;/p&gt;&#xA;" OwnerUserId="14673" LastActivityDate="2015-12-11T09:05:17.410" Title="Theano logistic regression understanding code" Tags="|python|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9323" PostTypeId="2" ParentId="9301" CreationDate="2015-12-10T22:42:41.480" Score="1" Body="&lt;p&gt;One possible first step is to convert the data back to the original coding.&#xA;This is called in SQL &lt;em&gt;unpivot&lt;/em&gt;, in R &lt;em&gt;melt&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here an R example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; my.df &amp;lt;- read.table(&#xA;+ text = &quot;DistrictA     DistrictB    DistrictC    DistrictD     DistrictE     Price&#xA;+         1             0            0            0             0             10000&#xA;+         0             1            0            0             0             20000&#xA;+         0             0            1            0             0             30000&#xA;+         0             0            0            1             0             40000&#xA;+         0             0            0            0             1             50000&quot;&#xA;+      , header = TRUE)&#xA;&amp;gt; my.df&#xA;  DistrictA DistrictB DistrictC DistrictD DistrictE Price&#xA;1         1         0         0         0         0 10000&#xA;2         0         1         0         0         0 20000&#xA;3         0         0         1         0         0 30000&#xA;4         0         0         0         1         0 40000&#xA;5         0         0         0         0         1 50000&#xA;&#xA;&amp;gt; library(reshape)&#xA;&amp;gt; subset(melt(my.df, id=&quot;Price&quot;, variable = &quot;District&quot;),value == 1)[,c(1,2)]&#xA;   Price  District&#xA;1  10000 DistrictA&#xA;7  20000 DistrictB&#xA;13 30000 DistrictC&#xA;19 40000 DistrictD&#xA;25 50000 DistrictE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After that you plot the Price dependent on a factor variable. You may additionally consider to order the factor based on the predicted price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I provide no details, as you don't tagged your tool, but I would recommend additional to a scatter plot to consider a box plot and/or density plot - always combined with the prediction value from the model for each factor level.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-12-10T22:42:41.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9324" PostTypeId="2" ParentId="8825" CreationDate="2015-12-11T00:25:06.727" Score="0" Body="&lt;p&gt;GraphLab is a good choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;NetworkX&lt;/a&gt;. You could make this really easy on yourself by eliminating the father of/friend of/etc. distinctions and just treat every relationship as fundamentally the same. Then its as simple as building a graph that tracks the connections between your users. After that you could limit collaborative filtering to people in this person's close circle of friends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also make distinct Friend/Family/Coworker implementations, but unless you have a LOT of data (and a lot of Family members) collaborative filtering could be a little disappointing.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2015-12-11T00:25:06.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9325" PostTypeId="1" CreationDate="2015-12-11T02:54:26.433" Score="9" ViewCount="16241" Body="&lt;p&gt;I'm looking for a Python library that can compute the &lt;a href=&quot;https://stats.stackexchange.com/q/21551/12359&quot;&gt;confusion matrix for multi-label classification&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;FYI:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;scikit-learn &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/3452&quot; rel=&quot;noreferrer&quot;&gt;doesn't&lt;/a&gt; support multi-label for confusion matrix)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/168945/12359&quot;&gt;What is the difference between Multiclass and Multilabel Problem&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-08-14T14:09:53.570" LastActivityDate="2020-08-03T12:56:58.390" Title="Python library that can compute the confusion matrix for multi-label classification" Tags="|python|software-recommendation|multilabel-classification|" AnswerCount="7" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9326" PostTypeId="2" ParentId="9322" CreationDate="2015-12-11T03:04:26.597" Score="3" Body="&lt;p&gt;I agree the code may be hard to read at the first glance, but I find the &lt;a href=&quot;http://deeplearning.net/tutorial/code/logistic_sgd.py&quot; rel=&quot;nofollow&quot;&gt;comment&lt;/a&gt; pretty clear:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;y.shape[0]&lt;/code&gt; is (symbolically) the number of rows in &lt;code&gt;y&lt;/code&gt;, i.e., number of examples (call it &lt;code&gt;n&lt;/code&gt;) in the minibatch &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;T.arange(y.shape[0])&lt;/code&gt; is a symbolic vector which will contain &lt;code&gt;[0,1,2,... n-1]&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;T.log(self.p_y_given_x)&lt;/code&gt; is a matrix of Log-Probabilities (call it &lt;code&gt;LP&lt;/code&gt;) with one row per example and one column per class &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;LP[T.arange(y.shape[0]),y]&lt;/code&gt; is a vector v containing &lt;code&gt;[LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;T.mean(LP[T.arange(y.shape[0]),y])&lt;/code&gt; is the mean (across minibatch examples) of the elements in v, i.e., the mean log-likelihood across the minibatch. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Everything is Python notation (but sometimes using Theano functions).&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-12-11T03:04:26.597" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9327" PostTypeId="2" ParentId="9317" CreationDate="2015-12-11T03:39:17.913" Score="1" Body="&lt;p&gt;Since you have multiple repetitions in A and B, and these two vectors are of different lengths (5913 and 3733), it is expected that vector C and D are of different size. However, C and D might contain the same unique elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take the example proposed by Biswajit Banerjee:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A = c(1,1,2,2,3,3,4,4,5,5,6,6,9,11,15,17,19,19)&#xA;B = c(1,2,2,2,3,3,3,4,5,6,7,8,8,8)&#xA;&#xA;C= A[(A %in% B)]&#xA;D = B[(B %in% A)]&#xA;&#xA;all(unique(C) == unique(D))&#xA;TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could verify that your vectors C and D of length 2007 and 126 contains the same unique values. Does it solve your problem ?&lt;/p&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2015-12-11T03:39:17.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9328" PostTypeId="1" CreationDate="2015-12-11T06:02:34.457" Score="2" ViewCount="121" Body="&lt;p&gt;Suppose we have an English to French translation task in a company, and there are 100s of workers who are proficient in doing this task, but each worker has its own unique attributes which enables them to do certain translations in a better way than the others (e.g., a translator who is a doctor would do better translations of medical documents). Consider 1000s of documents for translation received per day, and each document takes one day to translate. So, there is a two-fold problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Allocating the tasks to available translators using a queuing model or any other efficient allocation mechanism.&lt;/li&gt;&#xA;&lt;li&gt;Learning the worker model based on the past performance of each (e.g., correctness scores, static parameters like skill set, qualifications, experience, etc.) using a machine learning system.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there a setup/system/model which solves both of the above problems in one cohesive system? From my reading, this problem calls for the application of the assignment problem, user modelling, queuing, and machine learning optimization. I am searching for existing models or frameworks which integrate all of this, can anyone give me some direction for this task of optimum allocation?&lt;/p&gt;&#xA;" OwnerUserId="13011" LastEditorUserId="13011" LastEditDate="2016-02-10T13:58:25.100" LastActivityDate="2016-02-10T13:58:25.100" Title="Machine learning worker performance features for optimum allocation of tasks to workers" Tags="|machine-learning|predictive-modeling|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9329" PostTypeId="1" AcceptedAnswerId="9331" CreationDate="2015-12-11T06:13:25.480" Score="1" ViewCount="110" Body="&lt;p&gt;I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on size of data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be specific to the title, I have to predict missing values in a given temperature data set and i am unaware of anything that i can use. Could someone guide me through?&lt;/p&gt;&#xA;" OwnerUserId="14681" LastEditorUserId="11097" LastEditDate="2015-12-11T06:42:19.597" LastActivityDate="2015-12-15T10:26:52.910" Title="Using predictive modelling for temperature data set" Tags="|data-mining|missing-data|data-imputation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9331" PostTypeId="2" ParentId="9329" CreationDate="2015-12-11T06:41:49.733" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I am not able to understand how and what modelling techniques do we&#xA;  use?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Every data science workflow has the follwing steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pre-processing (data cleaning and wrangling)&lt;/li&gt;&#xA;&lt;li&gt;Exploratory analytics&lt;/li&gt;&#xA;&lt;li&gt;Model selection&lt;/li&gt;&#xA;&lt;li&gt;Prediction and testing.  (And re-iteration)&lt;/li&gt;&#xA;&lt;li&gt;(Optional) Reporting the workflow&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does it depend on the data type?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, the entire workflow is dependent on the type and features of the data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does it depend on size of data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Size of data makes a difference in the tools and sometimes(very rarely) the algorithms used.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have to predict missing values in a given temperature data set and i&#xA;  am unaware of anything that i can use&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;a href=&quot;https://datascience.stackexchange.com/questions/tagged/missing-data?sort=newest&amp;amp;pageSize=50&quot;&gt;lot of&lt;/a&gt; &lt;a href=&quot;https://stats.stackexchange.com/questions/tagged/missing-data&quot;&gt;material&lt;/a&gt; and algorithms on how to impute missing data, which you can refer to and use them accordingly depending on the type of data and the problem statement.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-11T06:41:49.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9333" PostTypeId="2" ParentId="9321" CreationDate="2015-12-11T07:41:24.760" Score="1" Body="&lt;p&gt;I found my own answer. I had defined &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = theano.shared(rng.randn(feats), name=&quot;w&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and that was wrong.&#xA;The correct definition is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = theano.shared(rng.randn(feats, num_classes), name=&quot;w&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;since the weights link 'feats'-number of input neuron to 'num_classes'-number of output neurons.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2015-12-11T07:41:24.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9335" PostTypeId="1" CreationDate="2015-12-11T09:07:07.943" Score="2" ViewCount="49" Body="&lt;p&gt;I have a data set DS.&#xA;I want to substitute the statements contained with others following a pattern-replacement schema like the following one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0YZe5.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0YZe5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used the followng function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from_distincts_to_origins &amp;lt;- function(patternST, replacementST,originalFile){&#xA;&#xA;  #---------------------------------------&#xA;  patternST&amp;lt;-paste0('\\b',patternST,'\\b')&#xA;  #---------------------------------------&#xA;  # we have to point back to the original data set&#xA;  file_origin_DF &amp;lt;- as.data.frame(originalFile)&#xA;&#xA;  file_output &amp;lt;- file_origin_DF&#xA;  for(i in 1:length(patternST))   { &#xA;    where &amp;lt;- file_output[[1]]#vector&#xA;    file_output &amp;lt;- apply(file_output,  &#xA;                         2,&#xA;                         function(x) sub(patternST[i], &#xA;                                         replacementST[i], &#xA;                                         where, &#xA;                                         #ignored.case=FALSE, &#xA;                                         perl=FALSE, &#xA;                                         fixed=FALSE, useBytes=FALSE)&#xA;    )&#xA;&#xA;    file_output &amp;lt;- as.data.frame(file_output, stringsAsFactors =F) # in order to work&#xA;  }       &#xA;&#xA;  return(file_output)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What will happen is that R will start substituting all the &quot;I like football&quot; with &quot;football&quot;. When it will pass to the second row it will look for &quot;I like football, W Lazio&quot; but it will only find &quot;&lt;strong&gt;football, W Lazio&lt;/strong&gt;&quot;, so those rows won't be substituted with &quot;Lazio&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible solution is to order the Pattern such to have the longest strings at the top, but it would only &lt;em&gt;reduce&lt;/em&gt; the substitution missings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there another method in R to do that?&lt;/p&gt;&#xA;" OwnerUserId="10024" LastActivityDate="2016-01-08T10:45:16.107" Title="R: substituting only whole statements (not parts of them)" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9336" PostTypeId="2" ParentId="9317" CreationDate="2015-12-11T10:04:33.210" Score="1" Body="&lt;p&gt;Does &lt;code&gt;C &amp;lt;- intersect(unique(A), unique(B))&lt;/code&gt; work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can do &lt;code&gt;A[A %in% C]&lt;/code&gt; to get them from  &lt;code&gt;A&lt;/code&gt; with duplicates.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2015-12-11T10:04:33.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9337" PostTypeId="2" ParentId="9316" CreationDate="2015-12-11T10:13:12.463" Score="3" Body="&lt;p&gt;Yes, the parameter &lt;code&gt;nonnegative&lt;/code&gt; is for constraining values for both matrices: user features and item features to not have values below zeros while optimizing using ALS algorithm. It drive which solver to use: CholeskySolver or NNLS(conjugated gradient). Both are implemented in Spark mllib. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The flag is of course connected with the input. If You have ratings below zeros and You directly pump all of Your data to model, the prediction of negatives values while You have &lt;code&gt;nonnegative&lt;/code&gt; constrain check will be somehow hard :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Your case, what's important is that You have implicit feedback. So, You should set appropriate flag or use &lt;code&gt;trainImplicit&lt;/code&gt; method. The easiest way to find out what works for Your case best with the &lt;code&gt;nonnegative&lt;/code&gt; flag is to threat it as another hyper-parameter which needs cross-validation check. And for me, the measurement of the model while validating is a more crucial aspect. If You are trying to do top-N recommendations, go for ranking measures like Recall@N, MRR@N or AUC.&lt;/p&gt;&#xA;" OwnerUserId="9314" LastActivityDate="2015-12-11T10:13:12.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9338" PostTypeId="2" ParentId="8120" CreationDate="2015-12-11T13:07:06.583" Score="2" Body="&lt;p&gt;I see that in the current version of python wrapper of xgboost you can specify file name or existing xgboost model (class Booster) in train function.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastActivityDate="2015-12-11T13:07:06.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9339" PostTypeId="2" ParentId="155" CreationDate="2015-12-11T16:13:05.043" Score="8" Body="&lt;p&gt;This subreddit lists a lot of known Datasets&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.reddit.com/r/datasets&quot; rel=&quot;noreferrer&quot;&gt;Reddit Datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of dataset requests on that subreddit, several of which have been answered.&lt;/p&gt;&#xA;" OwnerUserId="14695" LastActivityDate="2015-12-11T16:13:05.043" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="9340" PostTypeId="1" CreationDate="2015-12-11T16:59:33.750" Score="2" ViewCount="511" Body="&lt;p&gt;I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no  photographs or icons), and different documents may have a different shape (height and width). I'd like my classification approach to use all pages in each document, rather than just the first page. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, the input to a CNN (like theano) needs a standardized shape. My first thought was to create a single image array that has all the pages concatenated. But I would then have to resize/zero pad all concatenated pages to match the length of the document with the largest page count, and use that as my lowest height and width for the set of concatenated pages. If I don't use this strategy I risk losing resolution on the words in the image, but that is a giant input vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like splitting each document's pages into separate samples would be a better approach to standardize input preprocessing, but I'm at a loss for how to classify the whole document if I'm just training on loose pages of each document. Can anyone advise on me on this?&lt;/p&gt;&#xA;" OwnerUserId="14694" LastEditorUserId="11097" LastEditDate="2015-12-12T04:33:43.187" LastActivityDate="2015-12-15T14:14:30.910" Title="How can I preprocess multi-page image inputs in a theano/lasagne network?" Tags="|neural-network|image-classification|preprocessing|theano|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9341" PostTypeId="2" ParentId="9328" CreationDate="2015-12-11T18:09:06.853" Score="1" Body="&lt;p&gt;Your problem is going a lot of directions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have known worker attributes, like doctor, engineer, or accountant, you could have mirror document classification, and use machine learning to classify.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are going to measure performance, you will need to get a sample from each user. For example, take the top $\frac{1}{4}$ performance and have a target vector for the user.  Map a document to a user's strength.  If that user is over-queued, pick the next highest $\frac{1}{4}$ and so on.  You could use cosine similarity to give you a score. This can get tricky, as you may want to normalize on productivity.  You may get a document that a high productivity user is better at, but a low productivity user is OK at. By way of analogy, an american football player may be the best receiver and half back, but you might have a good backup receiver and no good backup halfback, so you let your backup play receiver.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Static parameters would be different, and I am not sure you should use them if you have a performance measure. My guess would be to queue before you have performance for a worker.  But even at 4 documents, you can select a top $\frac{1}{4}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also create clusters and measure a user's performance versus the cluster. This is the easiest approach to queuing, as you just take the next document out of the cluster that is a user's best cluster.  You only need a few cross-cluster workers to level queues. The downside is, if you have 20 clusters, it would take a while to get user performance versus cluster, so you would need to use static attributes until you get a valid sampling.&lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13413" LastEditDate="2015-12-11T18:48:31.750" LastActivityDate="2015-12-11T18:48:31.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9342" PostTypeId="1" CreationDate="2015-12-11T19:59:49.920" Score="3" ViewCount="272" Body="&lt;p&gt;I am trying to write a piece of code for R to fetch data from an api with the following command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- read.csv(&quot;https://...../api/dataservice?sql = select * from db where name IN (?)&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The parenthesis are where I need to input 218 different name tags. How can I do this programmatically? It is possible to write down few of them, but over 200 is not possible. Is there a way this? Can I use a character vector containing the names?&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="13413" LastEditDate="2015-12-11T20:04:42.387" LastActivityDate="2015-12-11T22:56:10.587" Title="How do write the following piece of code in R?" Tags="|r|sql|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9343" PostTypeId="2" ParentId="9342" CreationDate="2015-12-11T20:02:25.103" Score="1" Body="&lt;p&gt;If you already have the names in R, you can iterate over the vector of names, and use the &lt;code&gt;paste&lt;/code&gt; function. &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/paste.html&quot; rel=&quot;nofollow&quot;&gt;Here's the help page for it&lt;/a&gt;. &lt;code&gt;sprintf&lt;/code&gt; would be &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/sprintf.html&quot; rel=&quot;nofollow&quot;&gt;another option&lt;/a&gt;, if you prefer that approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of this would be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names &amp;lt;- c(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;)&#xA;querystring_a &amp;lt;- &quot;https://...../api/dataservice?sql = select * from db where name IN &quot;&#xA;querystring_b &amp;lt;- '\\'&#xA;querystring_c &amp;lt;- &quot;\'&quot;&#xA;for(n in names){&#xA;    cat(paste(querystring_a, querystring_b, querystring_c, n, querystring_c, sep=&quot;&quot;), sep=&quot;\n&quot;)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Return:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;https://...../api/dataservice?sql = select * from db where name IN \'a'&#xA;https://...../api/dataservice?sql = select * from db where name IN \'b'&#xA;https://...../api/dataservice?sql = select * from db where name IN \'c'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2015-12-11T22:56:10.587" LastActivityDate="2015-12-11T22:56:10.587" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9344" PostTypeId="1" CreationDate="2015-12-11T21:29:32.397" Score="1" ViewCount="2521" Body="&lt;p&gt;I am trying to build a long string of the form.....&lt;code&gt;\'name1\',\'name2\',\'name3\',..etc&lt;/code&gt;. How do I do it ? I am trying to do it using R ? By hand is impossible as there are currently over 200 such names and in future I may have more. So I need to automate the process. However, every time I tried to use R's paste0 function I am getting a different result. Here is what I did:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t &amp;lt;- c()&#xA;for( nms in name){ t &amp;lt;- paste0(t, paste0(&quot;\\'&quot;,nms,&quot;\\'&quot;, sep = &quot;,&quot;) ) }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the result turns out to include double backslashes instead of one, like this &lt;code&gt;&quot;\\'name1\\',\\'name2\\',\\'name3\\'&lt;/code&gt; etc. I couldn't get a single backslash as I want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2017-12-01T19:10:32.653" Title="How to build the following the string in R?" Tags="|r|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9345" PostTypeId="1" CreationDate="2015-12-11T23:01:37.850" Score="3" ViewCount="716" Body="&lt;p&gt;So, I'm having problems learning for a data analysis lecture. The topic is the construction of the &quot;central interval&quot; as our lecturer calls it. The concept shall path the way to understand confidence &amp;amp; credible intervals. But I don't even get his concept and therefore wanted to ask for help. &#xA;I will give an excerpt of the passage I don't get (it is in the context of a bionmial distribution where r is the possible outcome of a bernoulli experiment):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;To this end, we define the &lt;strong&gt;Central Probability Interval&lt;/strong&gt; and Smallest&#xA;Probability Interval containing a probability at least 1-$\alpha$.&#xA;This  interval  is  constructed  by  defining  a  smallest  member  of  the  set  of possible outcomes, $r_{1}$, and a largest member, $r_{2}$, subject to the criterion that&#xA;$$P(r&amp;lt;r_{1}) &amp;lt;= \alpha/2$$and $$P(r&amp;gt;r_{2}) &amp;lt;= \alpha/2$$ while maximizing these probablities.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First of all, $\alpha$ seems to be an arbitrary factor because it isn't mentioned anytime before. After choosing such an $\alpha$, we look for two numbers $r_{1,2}$. These numbers are limits which, when r surpasses these number, will yield a probability smaller than $\alpha$/2. But what about the maximizing of the probability? What probability?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;Stating this precisely, we write:&#xA;$$&#xA;r_{1} = 0 \quad\text{if}\quad P(r=0|N,p) &amp;gt; \alpha/2\quad\text{ else}\quad r_{1} =  \sup_{r\in0,...,N}[\sum_{i=0}^{r}P(i|N,p) &amp;lt;= \alpha/2]+1&#xA;$$&#xA;and&#xA;$$&#xA;r_{2} = N\quad\text{if}\quad P(r=N|N,p) &amp;gt; \alpha/2\quad\text{ else}\quad r_{2} =  \sup_{r\in0,...,N}[\sum_{i=r}^{N}P(i|N,p) &amp;lt;= \alpha/2]-1&#xA;$$&#xA;The set of our results in our Central Interval is&#xA;$$&#xA;O^{C}_{1-\alpha}=\{ r_{1}, r_{1}+1,...r_{2}\}.&#xA;$$&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now, I cannot make anything of the equations simply because I don't know how to read and process them properly. Maybe someone has an idea or can give me a good example?&#xA;Your help is much appreciated.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;I found a hint online and I think, now I get it.&#xA;We construct an interval which contains a prbability 1-$\alpha$. Therefore, the probability of all r values outside the interval [$r_{1}$, $r_{2}$] has to be $\alpha$. If you split this symmetric you get the criterion stated above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now as for the equations. They add up all the probabilities which contribute to an outcome smaller than $\alpha$/2. Taking the $\sup_{r \in 0,...,N}$ now gives the r value for which this sum is maximized. Then you have to add the +1 in order to take the next r which then is $r_{1}$.&lt;/p&gt;&#xA;" OwnerUserId="14698" LastEditorUserId="14698" LastEditDate="2015-12-12T11:40:58.333" LastActivityDate="2015-12-12T11:40:58.333" Title="Central Probability Interval" Tags="|statistics|data|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9346" PostTypeId="1" AcceptedAnswerId="9515" CreationDate="2015-12-11T23:43:50.733" Score="11" ViewCount="1398" Body="&lt;p&gt;I learn best through experimentation and example. I'm learning about neural networks and have (what I think) is a pretty good understanding of classification and regression and also supervised and unsupervised learning, but I've stumbled upon something I can't quiet figure out;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted to train an AI to play a complicated game; I'm thinking something like a RTS (eg. Age of Empires, Empire Earth etc.). In these types of games there is typically a number of entities controlled by the player (units, buildings) each with different capabilities. It seem like the problem of that the AI does would be classification (eg. choose that unit, and that action), however since the number of units is a variable how does one handle a classification problem in this way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing I can think of is multiple networks that do different stages (one for overall strategy, one for controlling this type of unit, one for that type of building etc.); but this seems like I'm making the problem to complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any good example of machine learning/neural networks learning complex games (not specifically RTS, but more complicated the &lt;a href=&quot;https://www.google.co.uk/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjoy7Cf_tTJAhWKtxQKHfYjDCwQyCkIJDAA&amp;amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dqv6UVOQ0F44&amp;amp;usg=AFQjCNFHQQt6wIfZEm4jzRCo8B0CWEF65w&amp;amp;sig2=ur7NzjxwjqTtVUmOX4Hcow&quot;&gt;Mario&lt;/a&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="14994" LastEditDate="2015-12-27T12:02:45.683" LastActivityDate="2021-03-10T02:21:25.713" Title="How do AI's learn to act when the problem space is too big" Tags="|machine-learning|neural-network|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9347" PostTypeId="1" CreationDate="2015-12-12T13:15:44.547" Score="2" ViewCount="204" Body="&lt;p&gt;I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Professional service&lt;/strong&gt;: Statistician involvement to build regression model or any other kind of predictive model that fits specifically to client data and can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Issue:&lt;/em&gt;&lt;/strong&gt; So on the long run there will be issues around scalability as one analyst cannot build model simultaneously for hundreds of clients who want to come on board and use this service. Hiring 1 analyst for 100 clients is not an option for me as i do not run this operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Product&lt;/strong&gt;: Plug and play with least human intervention. i.e. Some kind of self learning/artificial algorithm that works for any and every client. Additionally, it adjusts to new trends that appear over time which might not have been captured in the existing model. &lt;strong&gt;Personally, i will be biased towards this kind of solution.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Issue&lt;/em&gt;&lt;/strong&gt;: This is the ideal solution, however i am not sure how to build this and how to approach in this direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give context, a client in Europe might have same data structure in their sql database in terms of name, variable type in comparison to sql database of a client who is based in Singapore. However, Singapore data might have different trend, seasonality and pattern in comparison to data in Europe. Is there any way that a single Machine Learning or Artificial Intelligence or Deep Learning or Neural Net algorithm can be built and used for all regions and clients without having to update that model manually for any new client?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My background:&lt;/strong&gt; i have previously worked in a professional service landscape and have statistics/predictive modelling background.&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="13100" LastEditDate="2015-12-13T15:28:21.457" LastActivityDate="2015-12-13T15:28:21.457" Title="How to create Self learning data product" Tags="|machine-learning|recommender-system|deep-learning|scalability|data-product|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="9348" PostTypeId="2" ParentId="9037" CreationDate="2015-12-12T13:29:13.163" Score="1" Body="&lt;p&gt;In cases where the number of discovered clusters differs from the number of real classes, I like to use the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.6189&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Maximum-match measure&lt;/em&gt;&lt;/a&gt;. Basically it tries to find an optimal matching between your results and the true classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a 3-class database and you found 4 clusters. Then you can get the following sample results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\left[\matrix{2 &amp;amp; 7 &amp;amp; 3 \\ 6 &amp;amp; 2 &amp;amp; 3 \\ 3 &amp;amp; 3 &amp;amp; 1 \\ 2 &amp;amp; 1 &amp;amp; 8}\right]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which can be read like this: cluster 1 has 2 elements from class 1, 7 from class 2 and 3 from class 3, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximum-match tries to find the 3x3 matrix with maximum trace (diagonal sum). In this case, it'd be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\left[\matrix{ 6 &amp;amp; 2 &amp;amp; 3 \\ 2 &amp;amp; 7 &amp;amp; 3 \\ 2 &amp;amp; 1 &amp;amp; 8}\right]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final measure would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$mm = \frac{6+7+8}{41} = 0.51$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where 41 is the sum of all the elements of the 4x3 matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This measure penalizes clustering methods that find extra clusters.&lt;/p&gt;&#xA;" OwnerUserId="14665" LastActivityDate="2015-12-12T13:29:13.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9349" PostTypeId="1" CreationDate="2015-12-12T13:35:11.297" Score="9" ViewCount="11664" Body="&lt;p&gt;In spectral clustering, it's standard practice to solve the eigenvector problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L v = \lambda v$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: why bother taking the graph Laplacian? Couldn't I just solve the eigenvector problem for the graph (affinity matrix) itself, &lt;a href=&quot;https://www.youtube.com/watch?v=P-LEH-AFovE&quot; rel=&quot;nofollow noreferrer&quot;&gt;like the guy did in this video&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I made &lt;a href=&quot;https://stats.stackexchange.com/questions/183227/what-is-the-difference-between-affinity-matrix-eigenvectors-and-graph-laplacian&quot;&gt;this same question&lt;/a&gt; in CrossValidated but I think this is a more appropriate channel. Forgive me if I'm wrong.&lt;/p&gt;&#xA;" OwnerUserId="14665" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-12-13T03:02:27.553" Title="What is the difference between affinity matrix eigenvectors and graph Laplacian eigenvectors in the context of spectral clustering?" Tags="|machine-learning|clustering|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9352" PostTypeId="1" CreationDate="2015-12-12T16:14:27.670" Score="0" ViewCount="1323" Body="&lt;p&gt;I am stemming my text data in R. I am using a solution proposed by Yanchang Zhao for the latest version of tm package but found this very slow. I am looking for an alternative way.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myCorpusCopy &amp;lt;- myCorpus&#xA;&#xA;stemCompletion2 &amp;lt;- function(x, dictionary) {&#xA;x &amp;lt;- unlist(strsplit(as.character(x), &quot; &quot;))&#xA;x &amp;lt;- x[x != &quot;&quot;]&#xA;x &amp;lt;- stemCompletion(x, dictionary=dictionary)&#xA;x &amp;lt;- paste(x, sep=&quot;&quot;, collapse=&quot; &quot;) &#xA;PlainTextDocument(stripWhitespace(x)) }&#xA;&#xA;&#xA; myCorpus &amp;lt;- lapply(myCorpus, stemCompletion2, dictionary=myCorpusCopy)&#xA;&#xA; myCorpus &amp;lt;- Corpus(VectorSource(myCorpus))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-12-12T17:34:03.967" LastActivityDate="2015-12-12T17:34:03.967" Title="Stemming text in R" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9353" PostTypeId="2" ParentId="9352" CreationDate="2015-12-12T16:45:36.380" Score="1" Body="&lt;p&gt;Apply &lt;strong&gt;stemming&lt;/strong&gt; during preprocessing to reduce words to their basic form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Porter stemmer is okay for English, and rather fast.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-12-12T16:45:36.380" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9355" PostTypeId="1" AcceptedAnswerId="9357" CreationDate="2015-12-12T17:16:45.253" Score="2" ViewCount="1198" Body="&lt;p&gt;I am trying to understand the code for the logistic regression on the official documentation, but I am struggling to understand the logic behind this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# early-stopping parameters&#xA;patience = 5000  # look as this many examples regardless&#xA;patience_increase = 2     # wait this much longer when a new best is&#xA;                              # found&#xA;improvement_threshold = 0.995  # a relative improvement of this much is&#xA;                               # considered significant&#xA;validation_frequency = min(n_train_batches, patience/2)&#xA;                              # go through this many&#xA;                              # minibatches before checking the network&#xA;                              # on the validation set; in this case we&#xA;                              # check every epoch&#xA;&#xA;best_params = None&#xA;best_validation_loss = numpy.inf&#xA;test_score = 0.&#xA;start_time = time.clock()&#xA;&#xA;done_looping = False&#xA;epoch = 0&#xA;while (epoch &amp;lt; n_epochs) and (not done_looping):&#xA;    # Report &quot;1&quot; for first epoch, &quot;n_epochs&quot; for last epoch&#xA;    epoch = epoch + 1&#xA;    for minibatch_index in xrange(n_train_batches):&#xA;&#xA;        d_loss_wrt_params = ... # compute gradient&#xA;        params -= learning_rate * d_loss_wrt_params # gradient descent&#xA;&#xA;        # iteration number. We want it to start at 0.&#xA;        iter = (epoch - 1) * n_train_batches + minibatch_index&#xA;        # note that if we do `iter % validation_frequency` it will be&#xA;        # true for iter = 0 which we do not want. We want it true for&#xA;        # iter = validation_frequency - 1.&#xA;        if (iter + 1) % validation_frequency == 0:&#xA;&#xA;            this_validation_loss = ... # compute zero-one loss on validation set&#xA;&#xA;            if this_validation_loss &amp;lt; best_validation_loss:&#xA;&#xA;                # improve patience if loss improvement is good enough&#xA;                if this_validation_loss &amp;lt; best_validation_loss * improvement_threshold:&#xA;&#xA;                    patience = max(patience, iter * patience_increase)&#xA;                best_params = copy.deepcopy(params)&#xA;                best_validation_loss = this_validation_loss&#xA;&#xA;        if patience &amp;lt;= iter:&#xA;            done_looping = True&#xA;            break&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could any one, explain to me, what do the variables: patience, patience_increase, improvement_threshold, validation_frequency, iter,  represent ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this condition do ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if (iter + 1) % validation_frequency == 0:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14674" LastActivityDate="2015-12-12T23:10:13.957" Title="Early-Stopping for logistic regression. Theano" Tags="|logistic-regression|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9357" PostTypeId="2" ParentId="9355" CreationDate="2015-12-12T23:10:13.957" Score="1" Body="&lt;p&gt;&lt;code&gt;Patience&lt;/code&gt; is the number of training batches to do before you stop. &lt;code&gt;iter&lt;/code&gt; is the number of training batches you've seen. Each iteration, you decide whether or not your validation is lower than your previous best. The improvement is only stored if the score is lower than &lt;code&gt;improvement_threshold * validation_score&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears that &lt;code&gt;patience_increase&lt;/code&gt; is a multiplier.  Every time you have a new lowest score, you up the total number or training batches to &lt;code&gt;iter*patience_increase&lt;/code&gt;, but not below the current value of &lt;code&gt;patience&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;validation_frequency&lt;/code&gt; is just the number of batches between times you check the validation score.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-12T23:10:13.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9358" PostTypeId="2" ParentId="9349" CreationDate="2015-12-13T03:02:27.553" Score="5" Body="&lt;p&gt;The concept is the same but you are getting confused by the type of data. Spectral Clustering as &lt;a href=&quot;http://ai.stanford.edu/~ang/papers/nips01-spectral.pdf&quot; rel=&quot;noreferrer&quot;&gt;Ng et al.&lt;/a&gt; explain is about clustering standard data while the Laplacian matrix is a graph derived matrix used in algebraic graph theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the point is that whenever you encode the similarity of your objects into a matrix, this matrix could be used for spectral clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have standard data i.e. a sample-feature matrix, you can find the proximity or affinity or whatever you want to call it as a matrix and apply spectral clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a graph, this affinity would be anything like adjacency matrix, distance matrix or Laplacialn matrix and solving the eigenfunction for such a matrix gives you the corresponding result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point about using Laplacian rather than adjacency is to keep the so-called affinity matrix positive semi-definite (and normalized Laplacian matrix is a better choice as it gives you normalized eigenvalues between 0 and 2 and reveals the structure of graph much better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the long story short is that as long as you have a matrix containing the affinity of your data you can use spectral clustering in general. The difference is in details (i.g. the property of normalized Laplacian I just mentioned)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2015-12-13T03:02:27.553" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9359" PostTypeId="1" CreationDate="2015-12-13T05:04:15.860" Score="0" ViewCount="104" Body="&lt;p&gt;I'm mainly interested in creative things, but also interested in the science behind viewership and marketing a piece of media. Think about having a stable career in data science, where do you think I should begin? Are media analytics as big as others like business intelligence? Ive checked online courses, like google analytics academy and others like these:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edx.org/xseries/data-science-analytics-context&quot; rel=&quot;nofollow&quot;&gt;https://www.edx.org/xseries/data-science-analytics-context&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/specializations/jhudatascience&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specializations/jhudatascience&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right path to take? And is it easy to find jobs in such a field even if the person is a beginner in programming?&lt;/p&gt;&#xA;" OwnerUserId="14714" LastEditorUserId="3151" LastEditDate="2015-12-13T18:17:41.770" LastActivityDate="2015-12-13T18:17:41.770" Title="Should i pick this as a career?" Tags="|bigdata|predictive-modeling|beginner|" AnswerCount="1" CommentCount="2" ClosedDate="2015-12-13T11:34:00.343" ContentLicense="CC BY-SA 3.0" />
  <row Id="9360" PostTypeId="2" ParentId="9359" CreationDate="2015-12-13T05:52:33.113" Score="1" Body="&lt;p&gt;Data Science is all about creativity and do/see things what makes sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would describe Data Scientist career in couple words: Take this path only and only if you are passionate about science, working with data and are curious. Do not take this way if you are thinking about money or job security ... this field is not so simple and you stay weeks and months experimenting and try to find something in that datasets ... &#xA;One example: as data scientist you are mostly starting to look at data and try to find something inside and solve problems, some dta does not provide any info and after weeks of &quot;magic&quot; is put on the side and moved to other project, after a while you returning and look into data to find information on other business problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at Data Science as pure science and only after that about money, also most of the data scientist's primary goal is not remuneration ... media is promoting salaries but not the real work what we doing ... how much we love data ...&lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2015-12-13T05:52:33.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9361" PostTypeId="2" ParentId="9228" CreationDate="2015-12-13T06:43:55.510" Score="4" Body="&lt;p&gt;From Sebastian Raschka's &lt;em&gt;&lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning&quot; rel=&quot;nofollow&quot;&gt;Python Machine Learning&lt;/a&gt;&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The main advantage of such a memory-based approach [the KNN] is that&#xA;  the classifier immediately adapts as we collect new training data.&#xA;  However, the downside is that the computational complexity for&#xA;  classifying new samples grows linearly with the number of samples in&#xA;  the training dataset in the worst-case scenario—unless the dataset has&#xA;  very few dimensions (features) and the algorithm has been implemented&#xA;  using efficient data structures such as KD-trees. J. H. Friedman, J.&#xA;  L. Bentley, and R. A. Finkel. An algorithm for finding best matches in&#xA;  logarithmic expected time. ACM Transactions on Mathematical Software&#xA;  (TOMS), 3(3):209–226, 1977. Furthermore, we can't discard training&#xA;  samples since no training step is involved. Thus, storage space can&#xA;  become a challenge if we are working with large datasets.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The decision tree, however, can rapidly classify new examples. You're just running a series of boolean comparisons.&lt;/p&gt;&#xA;" OwnerUserId="14715" LastEditorUserId="11097" LastEditDate="2015-12-13T07:10:38.633" LastActivityDate="2015-12-13T07:10:38.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9362" PostTypeId="1" CreationDate="2015-12-13T09:26:04.977" Score="4" ViewCount="5393" Body="&lt;p&gt;Still I have no clear concept when I should choose linear OR Logistic regression. &#xA;&lt;br&gt; In addition, when I can say either linear or logistic regression performing better? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I want to know When should I choose Linear Regression and When to choose Logistic regression? &lt;/p&gt;&#xA;" OwnerUserId="14716" LastEditorUserId="3151" LastEditDate="2015-12-14T18:23:27.327" LastActivityDate="2016-02-15T08:49:11.300" Title="When to use linear or logistic regression?" Tags="|logistic-regression|linear-regression|model-selection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9363" PostTypeId="2" ParentId="9362" CreationDate="2015-12-13T10:39:27.733" Score="5" Body="&lt;p&gt;&lt;strong&gt;Linear Regression&lt;/strong&gt; is used for predicting continuous variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Logistic Regression&lt;/strong&gt; is used for predicting variables which has only limited values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me quote a nice example which can help you make the difference between the both:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For instance, if X contains the area in square feet of houses, and Y&#xA;  contains the corresponding sale price of those houses, you could use&#xA;  linear regression to predict selling price as a function of house&#xA;  size. While the possible selling price may not actually be any, there&#xA;  are so many possible values that a linear regression model would be&#xA;  chosen.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;If, instead, you wanted to predict, based on size, whether a house&#xA;  would sell for more than 200K, you would use logistic regression. The&#xA;  possible outputs are either Yes, the house will sell for more than&#xA;  $200K, or No, the house will not.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-13T10:39:27.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9364" PostTypeId="1" AcceptedAnswerId="9368" CreationDate="2015-12-13T14:19:54.510" Score="33" ViewCount="44617" Body="&lt;p&gt;XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I applied the parameters for a recent Kaggle problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param &amp;lt;- list(  objective           = &quot;reg:linear&quot;, &#xA;                booster = &quot;gbtree&quot;,&#xA;                eta                 = 0.02, # 0.06, #0.01,&#xA;                max_depth           = 10, #changed from default of 8&#xA;                subsample           = 0.5, # 0.7&#xA;                colsample_bytree    = 0.7, # 0.7&#xA;                num_parallel_tree   = 5&#xA;                # alpha = 0.0001, &#xA;                # lambda = 1&#xA;)&#xA;&#xA;&#xA;clf &amp;lt;- xgb.train(   params              = param, &#xA;                    data                = dtrain, &#xA;                    nrounds             = 3000, #300, #280, #125, #250, # changed from 300&#xA;                    verbose             = 0,&#xA;                    early.stop.round    = 100,&#xA;                    watchlist           = watchlist,&#xA;                    maximize            = FALSE,&#xA;                    feval=RMPSE&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All I do to experiment is randomly select (with intuition) another set of parameters for improving on the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anyway I automate the selection of optimized(best) set of parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Answers can be in any language. I'm just looking for the technique)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2019-05-11T18:59:04.247" Title="Hypertuning XGBoost parameters" Tags="|r|python|xgboost|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9365" PostTypeId="2" ParentId="9364" CreationDate="2015-12-13T17:06:10.410" Score="5" Body="&lt;p&gt;You could use the caret package to do hyperparameter space search, either through a &lt;a href=&quot;http://topepo.github.io/caret/training.html#custom&quot; rel=&quot;noreferrer&quot;&gt;grid search&lt;/a&gt; , or through &lt;a href=&quot;http://topepo.github.io/caret/random.html&quot; rel=&quot;noreferrer&quot;&gt;random search&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="14598" LastActivityDate="2015-12-13T17:06:10.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9366" PostTypeId="2" ParentId="9085" CreationDate="2015-12-13T23:32:10.927" Score="0" Body="&lt;p&gt;Record all votes  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ratio number of 1 votes when on the front page compared to not   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only apply a fraction of number 1 votes while on the first page&lt;br&gt;&#xA;Basically remove the page 1 bias based on the page 1 bias as a whole  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 vote applied = 1 votes item when on first page * (1 votes second page total / 1 votes first page total) &lt;/p&gt;&#xA;" OwnerUserId="13285" LastEditorUserId="13285" LastEditDate="2015-12-15T17:13:17.883" LastActivityDate="2015-12-15T17:13:17.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9368" PostTypeId="2" ParentId="9364" CreationDate="2015-12-14T02:02:02.803" Score="42" Body="&lt;p&gt;Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Caret&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost.&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees&quot;&gt;How to tune hyperparameters of xgboost trees?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;strong&gt;Custom Grid Search&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I often begin with a few assumptions based on &lt;a href=&quot;https://www.kaggle.com/owenzhang1&quot;&gt;Owen Zhang&lt;/a&gt;'s slides on &lt;a href=&quot;http://www.slideshare.net/OwenZhang2/tips-for-data-science-competitions&quot;&gt;tips for data science&lt;/a&gt; P. 14&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9GgQK.jpg&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9GgQK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can see that you'll mostly need to tune row sampling, column sampling and maybe maximum tree depth. This is how I do a custom row sampling and column sampling search for a problem I am working on at the moment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;searchGridSubCol &amp;lt;- expand.grid(subsample = c(0.5, 0.75, 1), &#xA;                                colsample_bytree = c(0.6, 0.8, 1))&#xA;ntrees &amp;lt;- 100&#xA;&#xA;#Build a xgb.DMatrix object&#xA;DMMatrixTrain &amp;lt;- xgb.DMatrix(data = yourMatrix, label = yourTarget)&#xA;&#xA;rmseErrorsHyperparameters &amp;lt;- apply(searchGridSubCol, 1, function(parameterList){&#xA;&#xA;    #Extract Parameters to test&#xA;    currentSubsampleRate &amp;lt;- parameterList[[&quot;subsample&quot;]]&#xA;    currentColsampleRate &amp;lt;- parameterList[[&quot;colsample_bytree&quot;]]&#xA;&#xA;    xgboostModelCV &amp;lt;- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE, &#xA;                           metrics = &quot;rmse&quot;, verbose = TRUE, &quot;eval_metric&quot; = &quot;rmse&quot;,&#xA;                           &quot;objective&quot; = &quot;reg:linear&quot;, &quot;max.depth&quot; = 15, &quot;eta&quot; = 2/ntrees,                               &#xA;                           &quot;subsample&quot; = currentSubsampleRate, &quot;colsample_bytree&quot; = currentColsampleRate)&#xA;&#xA;    xvalidationScores &amp;lt;- as.data.frame(xgboostModelCV)&#xA;    #Save rmse of the last iteration&#xA;    rmse &amp;lt;- tail(xvalidationScores$test.rmse.mean, 1)&#xA;&#xA;    return(c(rmse, currentSubsampleRate, currentColsampleRate))&#xA;&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And combined with some ggplot2 magic using the results of that apply function you can plot a graphical representation of the search.&lt;a href=&quot;https://i.stack.imgur.com/jaAuy.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jaAuy.png&quot; alt=&quot;My xgboost hyperparameter search&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot lighter colors represent lower error and each block represents a unique combination of column sampling and row sampling. So if you want to perform an additional search of say eta (or tree depth) you will end up with one of these plots for each eta parameters tested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see you have a different evaluation metric (RMPSE), just plug that in the cross validation function and you'll get the desired result. Besides that I wouldn't worry too much about fine tuning the other parameters because doing so won't improve performance too much, at least not so much compared to spending more time engineering features or cleaning the data.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;strong&gt;Others&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Random search and Bayesian parameter selection are also possible but I haven't made/found an implementation of them yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a good primer on bayesian Optimization of hyperparameters by Max Kuhn creator of caret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html&quot;&gt;http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-06-07T20:00:13.573" LastActivityDate="2016-06-07T20:00:13.573" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9369" PostTypeId="1" CreationDate="2015-12-14T04:36:30.757" Score="2" ViewCount="135" Body="&lt;p&gt;I am using the Gensim &lt;a href=&quot;http://radimrehurek.com/gensim/models/lsimodel.html&quot; rel=&quot;nofollow&quot;&gt;LsiModel&lt;/a&gt;. I have a set of documents and fixed set of topics. Some of the documents are already categorized, others are not. The goal is to categorize the uncategorized documents with the most relevant category. I am using a similarity search as described here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://radimrehurek.com/gensim/tut3.html&quot; rel=&quot;nofollow&quot;&gt;http://radimrehurek.com/gensim/tut3.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am comparing each uncategorized document to the categorized corpus to find the most relevant category. I am seeing very good performance on documents which have an appropriate category. However, it is to be expected that some documents will not have an relevant category, e.g. it is in Spanish, it is spam, or it just does not fit into an existing category. With this model every document is categorized and the best fit is the category with the highest similarity score. My question is, &lt;strong&gt;how can I determine when there is not relevant category&lt;/strong&gt;? My assumption is that the similarity measures for the documents should all be low, but this is not always true. This also seems to be an arbitrary measure. Are there better ways to say a particular document does not fit well into existing categories?&lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastEditorDisplayName="user13684" LastEditDate="2015-12-14T13:00:46.073" LastActivityDate="2015-12-14T13:00:46.073" Title="Latent Semantic Indexing False Positive Detection" Tags="|gensim|lsi|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9370" PostTypeId="1" AcceptedAnswerId="9387" CreationDate="2015-12-14T10:33:28.333" Score="3" ViewCount="2792" Body="&lt;p&gt;I'm new to datascience. I was wondering how one should treat an extremely dominant feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one of the features is &quot;on&quot;/&quot;off&quot;, and when it's &quot;off&quot;, none of the other features matter and the output will just always be 0. So should I drop all rows where it's &quot;off&quot; in my train/test data sets? I feel like I would get a better fit that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I delete those rows, I'm concerned about how I would handle those rows in the test set. For example, I'd have to write code to loop through the data and put a 0 in the prediction column for those rows, as well as make sure everything else lines up. (This is all Kaggle related, so the training set is several columns of features and a y_column, whereas the test set doesn't have the y_column and we're supposed to predict it.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Python and Scikit Learn's random forest, if that matters.&lt;/p&gt;&#xA;" OwnerUserId="14729" LastEditorUserId="13413" LastEditDate="2015-12-15T09:28:43.800" LastActivityDate="2015-12-15T13:59:18.510" Title="Extremely dominant feature?" Tags="|random-forest|feature-selection|scikit-learn|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9371" PostTypeId="2" ParentId="9370" CreationDate="2015-12-14T12:51:44.017" Score="2" Body="&lt;p&gt;If you are using RandomForest, then I am sure it will pick up this rule given that $\text{off} \rightarrow 0$ provides complete node purity. However, your intuition is correct, it is not uncommon to preprocess data. You can think of this rule as a single-level decision tree or &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_stump&quot; rel=&quot;nofollow&quot;&gt;decision stump&lt;/a&gt;, (aka 1-rule algorithm). Basically, you would remove these records from test and training to reduce noise. During classification you would also preprocess, if the input vector matches the rule then classify as 0 otherwise classify input vector with your model. &lt;/p&gt;&#xA;" OwnerDisplayName="user13684" LastActivityDate="2015-12-14T12:51:44.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9372" PostTypeId="2" ParentId="9340" CreationDate="2015-12-14T14:09:46.347" Score="1" Body="&lt;p&gt;First point is that convolutional neural networks would be incredibly expensive to train on images that large. The minimum size you could allow would be &lt;code&gt;num_pixels_per_letter x num_letters_per_row_or_col&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you want to achieve two tasks: OCR then document classification. Given that you have predominantly text in these documents, you would almost certainly be better served by using a more a traditional method for OCR. If you want or need to use NNs for the OCR piece, you will almost certainly need to use a sliding window. For inspiration on how you might do that, you could look to &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow&quot;&gt;Recurrent Neural Networks&lt;/a&gt; that &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;process images in sequences&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have the text there are again much simpler methods to deal with text than NNs, but they can achieve great results. To deal with different document lengths, you can use an RNN architecture like &lt;a href=&quot;http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf&quot; rel=&quot;nofollow&quot;&gt;LSTM&lt;/a&gt;. Or you can use paragraph vectors, which do most of the heavy lifting for you and give you an N-dimensional representation of your text.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-12-15T14:14:30.910" LastActivityDate="2015-12-15T14:14:30.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9373" PostTypeId="1" CreationDate="2015-12-14T14:44:54.697" Score="3" ViewCount="207" Body="&lt;p&gt;Since PyLearn2 is build upon Theano, is it possible to do anything I can do in Theano in Pylearn2? For example, if I have some snippets of Theano code, can I run them as-is in Pylearn2, or would this not work? If it wouldn't, why not?&lt;/p&gt;&#xA;" OwnerUserId="14734" LastEditorUserId="12473" LastEditDate="2015-12-14T16:56:42.643" LastActivityDate="2016-09-10T09:45:06.977" Title="Can PyLearn do everything that Theano can?" Tags="|machine-learning|deep-learning|theano|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9374" PostTypeId="1" CreationDate="2015-12-14T15:09:13.993" Score="2" ViewCount="149" Body="&lt;p&gt;I am working on a problem where I have access to a database with news articles, their publication date and the number of views they got 24hrs they got published.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The objective is to be able to predict the number of views for any new article.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to add a feature that, for a given article, gives an indication of the number of views / publications of all articles with a similar theme that happened a short time before the publication of this article. This would capture the &quot;buzz&quot; phenomenon. (If there has been a lot of articles on elections, then it is quite likely that a new article on elections will get lots of views.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is my strategy :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We classify all articles among X themes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We segment the timeline into Y timesteps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each timestep:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each theme:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;we count the number of views of articles of this theme published in this timestep&lt;/li&gt;&#xA;&lt;li&gt;ditto for the number of publications&lt;/li&gt;&#xA;&lt;li&gt;we add these two values as features for all articles published in the next timestep&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, I was wondering if this violates the i.i.d. assumption? (Because, if I randomly separate the data into train/test, this injects information from the target values of the test set as a feature of the training set.) Is this &quot;allowed&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something? Do you see another strategy?&lt;/p&gt;&#xA;" OwnerUserId="14735" LastActivityDate="2015-12-14T15:09:13.993" Title="&quot;Trending&quot; feature to predict number of views" Tags="|machine-learning|nlp|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9375" PostTypeId="1" CreationDate="2015-12-14T16:58:41.163" Score="5" ViewCount="291" Body="&lt;p&gt;I'm trying to build a model to solve a regression task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplified, the data look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;7.16, A 1, B 4, C 15, D 3&#xA;8.21, C 3, D 2, G  7, M 4, Y 9&#xA;1.85, D 3, N 1, L  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first column is a label, consequent columns are a class and respective number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the first row demonstrates that there is one of &lt;strong&gt;A&lt;/strong&gt;, four of &lt;strong&gt;B&lt;/strong&gt;, fifteen of &lt;strong&gt;C&lt;/strong&gt;, and three of &lt;strong&gt;D&lt;/strong&gt; - such a configuration gives us 7.16. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Primary Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I represent my data so that I can fit a Random Forest Regression model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Thoughts:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Figure out the whole list of classes (there are 26 letters, but I have way more classes) and put zeros for all of the unknown classes into each record. &lt;code&gt;Vector.sparse()&lt;/code&gt; might help here as it can reduce the amount of required memory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Figure out the entire list of classes, correspond a number starting from 0 and convert each record into a vector (i.e. the first one would look): 0 1 1 4 2 15 3 3. Then each column is treated as a separate feature, but in this case I have to specify that some of my features are categorical and that's not possible with Spark since different records are of different length (In Spark I have to provide columns of features, which are categorical with number of classes. In the example that would be a map(0 -&gt; 26, 2 -&gt; 26, 4 -&gt; 26) - like every second column is a class with 26 categories).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It's possible to build features like A:1, B:4, C:15, D:3 and treat all of them as categorical. Or move decision burden (categorical vs continuous) to Spark itself.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14740" LastEditorUserId="26596" LastEditDate="2016-12-05T19:54:53.060" LastActivityDate="2017-01-05T22:00:31.897" Title="Random Forest Regression. How to represent really long list of categories for processing" Tags="|dataset|regression|random-forest|apache-spark|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9376" PostTypeId="2" ParentId="9375" CreationDate="2015-12-14T17:17:18.263" Score="0" Body="&lt;p&gt;I think that your first and last approach are the same - as a result you have sparse feature vector. In your last approach you just don't list features with value 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is hard to say if you should use categorical or continuous values - it depends on each feature.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastActivityDate="2015-12-14T17:17:18.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9377" PostTypeId="2" ParentId="9373" CreationDate="2015-12-14T17:19:44.423" Score="0" Body="&lt;p&gt;PyLearn2 is built on top of Theano. If you have Theano snippets you can run them with PyLearn2 snippets.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastActivityDate="2015-12-14T17:19:44.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9378" PostTypeId="1" AcceptedAnswerId="9404" CreationDate="2015-12-14T17:21:21.047" Score="5" ViewCount="2103" Body="&lt;p&gt;[I've cross-posted it to cross.validated because I'm not sure where it fits best]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the training set)? I have thought of three different possibilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch starts. We sample and feedforward one minibatch only, get the error and backprop it, i.e. update the weights. Epoch over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch starts. We sample and feedforward a minibatch, get the error and backprop it, i.e. update the weights. We repeat this until we have sampled the full data set. Epoch over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch starts. We sample and feedforward a minibatch, get the error and store it. We repeat this until we have sampled the full data set. We somehow average the errors and backprop them by updating the weights. Epoch over.&lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2015-12-15T19:31:07.267" Title="How to update weights in a neural network using gradient descent with mini-batches?" Tags="|machine-learning|neural-network|gradient-descent|backpropagation|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9379" PostTypeId="2" ParentId="9378" CreationDate="2015-12-14T17:45:18.323" Score="2" Body="&lt;p&gt;When you train with mini-batches then you have the second option, network is updated after each mini-batch, and epoch ends after presenting all samples.&lt;/p&gt;&#xA;&lt;p&gt;Please see these &lt;a href=&quot;https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks&quot;&gt;responses&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14636" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2015-12-15T14:41:00.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9380" PostTypeId="2" ParentId="9175" CreationDate="2015-12-14T17:50:53.123" Score="2" Body="&lt;p&gt;Check this &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;lecture&lt;/a&gt; and this &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;visualization&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Usually it is used type 2.1 convolution. In the input you have &lt;span class=&quot;math-container&quot;&gt;$NxMx1$&lt;/span&gt; image, then after first convolution you will obtain &lt;span class=&quot;math-container&quot;&gt;$N_1xM_1xk_1$&lt;/span&gt;, so your image after first convolution will have &lt;span class=&quot;math-container&quot;&gt;$k_1$&lt;/span&gt; channels. The new dimension &lt;span class=&quot;math-container&quot;&gt;$N_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$M_1$&lt;/span&gt; will depend on your stride &lt;span class=&quot;math-container&quot;&gt;$S$&lt;/span&gt; and padding &lt;span class=&quot;math-container&quot;&gt;$P: N_1 = (N - 3 + 2P)/S + 1$&lt;/span&gt;, you compute &lt;span class=&quot;math-container&quot;&gt;$M_1$&lt;/span&gt; in analogy. For the first conv layer you will have &lt;span class=&quot;math-container&quot;&gt;$3x3xk_1 + k_1$&lt;/span&gt; weights. There is added &lt;span class=&quot;math-container&quot;&gt;$k_1$&lt;/span&gt; for biases in nonlinear function.&lt;/p&gt;&#xA;&lt;p&gt;In the second layer you have as an input image with size &lt;span class=&quot;math-container&quot;&gt;$N_1xM_1xk_1$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$k_1$&lt;/span&gt; is new number of channels. And after second convolution you obtain &lt;span class=&quot;math-container&quot;&gt;$N_2xM_2xk_2$&lt;/span&gt; image (array). You have &lt;span class=&quot;math-container&quot;&gt;$5x5xk_2xk_1+k_2$&lt;/span&gt; parameters in the second layer.&lt;/p&gt;&#xA;&lt;p&gt;For &lt;span class=&quot;math-container&quot;&gt;$1x1$&lt;/span&gt; convolution with &lt;span class=&quot;math-container&quot;&gt;$k_3$&lt;/span&gt; filters and input &lt;span class=&quot;math-container&quot;&gt;$NxMxC$&lt;/span&gt; (&lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; is number of input channels) you will obtain new image (array) &lt;span class=&quot;math-container&quot;&gt;$NxMxk_3$&lt;/span&gt;, so &lt;span class=&quot;math-container&quot;&gt;$1x1$&lt;/span&gt; make sense. They were introduced in this &lt;a href=&quot;http://arxiv.org/pdf/1312.4400v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Bonus 1: pooling is applied per feature map.&lt;/p&gt;&#xA;&lt;p&gt;For details please see &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;slides&lt;/a&gt; for CNN course on Stanford - you have there nice visualisation how convolution is summed from several input channels.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastEditorUserId="29169" LastEditDate="2021-02-07T21:04:20.057" LastActivityDate="2021-02-07T21:04:20.057" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="9383" PostTypeId="1" CreationDate="2015-12-14T21:58:59.800" Score="4" ViewCount="239" Body="&lt;p&gt;I'm using a random forest model. One of my independent variables almost certainly has a parabolic effect on the dependent variable. In a linear regression, I would include the variable as both linear and squared in order to capture this effect. Should I do the same in a random forest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By extension, same question for variables that have logistic effects (age, for example)?&lt;/p&gt;&#xA;" OwnerUserId="14749" LastActivityDate="2020-03-24T21:01:00.077" Title="Do I need to include a squared and linear variable in a random forest to achieve a parabolic effect?" Tags="|machine-learning|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9384" PostTypeId="2" ParentId="9175" CreationDate="2015-12-14T22:21:15.120" Score="10" Body="&lt;p&gt;I am not sure about the alternatives described above, but the commonly used methodology is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before the application of the non-linearity, each filter output depends linearly on all of the feature maps before within the patch, so you end up with $k_2$ filters after the second layers. The overall number of parameters is $3 \dot{} 3\dot{}k_1 + k_1\dot{} 5 \dot{} 5 \dot{} k_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus 1: Pooling is done per feature map, separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus 2: The order of &quot;sliding&quot; does not matter. In fact, each output is computed based on the previous layer, so the output filter responses do not depend on each other. They can be computed in parallel.&lt;/p&gt;&#xA;" OwnerUserId="14750" LastActivityDate="2015-12-14T22:21:15.120" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9385" PostTypeId="2" ParentId="22" CreationDate="2015-12-15T01:23:14.193" Score="25" Body="&lt;p&gt;This question seems really about representation, and not so much about clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called &quot;color&quot; that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm will think that red (1) is actually closer to blue (2) than it is to yellow (3). We need to use a representation that lets the computer understand that these things are all actually equally different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple way is to use what's called a &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;noreferrer&quot;&gt;one-hot&lt;/a&gt; representation, and it's exactly what you thought you should do. Rather than having one variable like &quot;color&quot; that can take on three values, we separate it into three variables. These would be &quot;color-red,&quot; &quot;color-blue,&quot; and &quot;color-yellow,&quot; which all can only take on the value 1 or 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This increases the dimensionality of the space, but now you could use any clustering algorithm you like. It does sometimes make sense to zscore or whiten the data after doing this process, but the your idea is definitely reasonable.&lt;/p&gt;&#xA;" OwnerUserId="9483" LastActivityDate="2015-12-15T01:23:14.193" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9386" PostTypeId="2" ParentId="8681" CreationDate="2015-12-15T01:36:15.547" Score="2" Body="&lt;p&gt;I appended my answer to &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;this&lt;/a&gt; question below - you guys essentially asked the same thing.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This question seems really about representation, and not so much about clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called &quot;color&quot; that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm will think that red (1) is actually closer to blue (2) than it is to yellow (3). We need to use a representation that lets the computer understand that these things are all actually equally different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple way is to use what's called a one-hot representation, and it's exactly what you thought you should do. Rather than having one variable like &quot;color&quot; that can take on three values, we separate it into three variables. These would be &quot;color-red,&quot; &quot;color-blue,&quot; and &quot;color-yellow,&quot; which all can only take on the value 1 or 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This increases the dimensionality of the space, but now you could use any clustering algorithm you like. It does sometimes make sense to zscore or whiten the data after doing this process, but the your idea is definitely reasonable.&lt;/p&gt;&#xA;" OwnerUserId="9483" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-15T01:36:15.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9387" PostTypeId="2" ParentId="9370" CreationDate="2015-12-15T01:43:06.890" Score="3" Body="&lt;p&gt;Actually, it shouldn't really matter what classification algorithm you use. The whole point of machine learning is that the algorithm learns how to combine the available features to achieve the desired result. If one feature has the ability to 'turn the others off,' the algorithm will learn that (It'll also learn lots of things that you probably aren't aware of). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in short, no, modifying the data this way probably won't affect classification performance. Not needing to incorporate these kinds of things into the training set is part of what makes machine learning so cool!&lt;/p&gt;&#xA;" OwnerUserId="9483" LastActivityDate="2015-12-15T01:43:06.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9389" PostTypeId="1" AcceptedAnswerId="9396" CreationDate="2015-12-15T03:10:26.323" Score="-2" ViewCount="470" Body="&lt;p&gt;I have downloaded some data to learn about machine learning and distributed computing. I used WinZip to uncompress a .bz2. Now I have text (opened in Notepad) which looks like this (this file is long, this is just a sample):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    {&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450635&quot;,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299an&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:1,&quot;author&quot;:&quot;bostich&quot;,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;test&quot;,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299an&quot;,&quot;retrieved_on&quot;:1427426409}&#xA;    {&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;much smoother.\r\n\r\nIm just glad reddit is back, #reddit in mIRC was entertaining but I had no idea how addicted I had become. Thanks for making the detox somewhat short.&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;author&quot;:&quot;igiveyoumylife&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299ao&quot;,&quot;retrieved_on&quot;:1427426409,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450639&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299ao&quot;}&#xA;{&quot;author&quot;:&quot;Arve&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:0,&quot;body&quot;:&quot;Can we please deprecate the word \&quot;Ajax\&quot; now? \r\n\r\n(But yeah, this _is_ much nicer)&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299ap&quot;,&quot;created_utc&quot;:&quot;1192450643&quot;,&quot;parent_id&quot;:&quot;t1_c02999p&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:0,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299ap&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true}&#xA;{&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;[deleted]&quot;,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;[deleted]&quot;,&quot;name&quot;:&quot;t1_c0299aq&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;created_utc&quot;:&quot;1192450646&quot;,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299aq&quot;,&quot;distinguished&quot;:null}&#xA;{&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299ar&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author&quot;:&quot;gigaquack&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:3,&quot;body&quot;:&quot;Oh, I see. Fancy schmancy \&quot;submitting....\&quot;&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299ar&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;created_utc&quot;:&quot;1192450646&quot;,&quot;parent_id&quot;:&quot;t1_c0299ah&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:3}&#xA;{&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;body&quot;:&quot;testing ...&quot;,&quot;score_hidden&quot;:false,&quot;author&quot;:&quot;Percept&quot;,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299as&quot;,&quot;author_flair_css_class&quot;:null,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:1,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450656&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299as&quot;,&quot;distinguished&quot;:null}&#xA;{&quot;ups&quot;:3,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c0299ar&quot;,&quot;created_utc&quot;:&quot;1192450658&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299at&quot;,&quot;distinguished&quot;:null,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;I like it. One more time...&quot;,&quot;score&quot;:3,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;gigaquack&quot;,&quot;name&quot;:&quot;t1_c0299at&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false}&#xA;{&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c02999j&quot;,&quot;created_utc&quot;:&quot;1192450662&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;id&quot;:&quot;c0299au&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot; try refreshing yor cache, that worked for me \n//edit: trying to edit&quot;,&quot;score&quot;:2,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;mcm69&quot;,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299au&quot;,&quot;author_flair_css_class&quot;:null,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;}&#xA;{&quot;controversiality&quot;:0,&quot;ups&quot;:3,&quot;parent_id&quot;:&quot;t1_c0299at&quot;,&quot;created_utc&quot;:&quot;1192450670&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299av&quot;,&quot;body&quot;:&quot;K. I lied. Just one more...&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;author&quot;:&quot;gigaquack&quot;,&quot;score&quot;:3,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299av&quot;,&quot;retrieved_on&quot;:1427426409,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;}&#xA;{&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299aw&quot;,&quot;author&quot;:&quot;deki&quot;,&quot;score&quot;:2,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;body&quot;:&quot;I also wonder what the differences are...&quot;,&quot;score_hidden&quot;:false,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299aw&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450681&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:2}&#xA;{&quot;created_utc&quot;:&quot;1192450683&quot;,&quot;parent_id&quot;:&quot;t1_c0299av&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:3,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299ax&quot;,&quot;distinguished&quot;:null,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;author&quot;:&quot;gigaquack&quot;,&quot;score&quot;:3,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;So addictive...&quot;,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;name&quot;:&quot;t1_c0299ax&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null}&#xA;{&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299ay&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:1,&quot;created_utc&quot;:&quot;1192450687&quot;,&quot;parent_id&quot;:&quot;t3_5yba1&quot;,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299ay&quot;,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;body&quot;:&quot;I can't post a story to proggit - I got \&quot;you are trying to submit too fast\&quot; on my first submission.&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;author&quot;:&quot;llimllib&quot;,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba1&quot;}&#xA;{&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;Alright I'm done.&quot;,&quot;author&quot;:&quot;gigaquack&quot;,&quot;score&quot;:3,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;name&quot;:&quot;t1_c0299az&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;controversiality&quot;:0,&quot;ups&quot;:3,&quot;parent_id&quot;:&quot;t1_c0299ax&quot;,&quot;created_utc&quot;:&quot;1192450691&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299az&quot;,&quot;distinguished&quot;:null}&#xA;{&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299b0&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;created_utc&quot;:&quot;1192450696&quot;,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;ups&quot;:12,&quot;controversiality&quot;:0,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b0&quot;,&quot;score&quot;:12,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;igiveyoumylife&quot;,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;Is anyone else's \&quot;recommended\&quot; page completely empty? Mine is, and it is usually jam-packed.&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0}&#xA;{&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299b1&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c02999j&quot;,&quot;created_utc&quot;:&quot;1192450700&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299b1&quot;,&quot;retrieved_on&quot;:1427426409,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;body&quot;:&quot;[deleted]&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:1,&quot;author&quot;:&quot;[deleted]&quot;}&#xA;{&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450705&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:2,&quot;id&quot;:&quot;c0299b2&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;author&quot;:&quot;[deleted]&quot;,&quot;score&quot;:2,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;Ok, I guess we need to submit comments to test? &quot;,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b2&quot;,&quot;author_flair_css_class&quot;:null}&#xA;{&quot;name&quot;:&quot;t1_c0299b3&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;body&quot;:&quot;I can't submit any stories -- even the first time, i get, \&quot;you are trying to submit too fast\&quot; .. anyone else seeing this?\n\nEdit: This appears to be fixed.&quot;,&quot;score_hidden&quot;:false,&quot;author&quot;:&quot;raldi&quot;,&quot;score&quot;:5,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;id&quot;:&quot;c0299b3&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null,&quot;controversiality&quot;:0,&quot;ups&quot;:5,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450709&quot;}&#xA;{&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299b4&quot;,&quot;distinguished&quot;:null,&quot;controversiality&quot;:0,&quot;ups&quot;:1,&quot;created_utc&quot;:&quot;1192450737&quot;,&quot;parent_id&quot;:&quot;t1_c0299a5&quot;,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b4&quot;,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;body&quot;:&quot;Working fine with normal Adblock.&quot;,&quot;score_hidden&quot;:false,&quot;author&quot;:&quot;deki&quot;,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;}&#xA;{&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299b5&quot;,&quot;distinguished&quot;:null,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450739&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:12,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b5&quot;,&quot;author_flair_css_class&quot;:null,&quot;author&quot;:&quot;jezmck&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:12,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;can't see beta.reddit.com from here...&quot;}&#xA;{&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299b6&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c02999j&quot;,&quot;created_utc&quot;:&quot;1192450748&quot;,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b6&quot;,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;body&quot;:&quot;I had a problem as well, with my comment being lost. Then I did a \&quot;super-refresh\&quot; (shift-Reload) and it seemed to work.&quot;,&quot;score_hidden&quot;:false,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;score&quot;:2,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;sickofthisshit&quot;}&#xA;{&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299b7&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450748&quot;,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299b7&quot;,&quot;retrieved_on&quot;:1427426409,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;hfaber&quot;,&quot;body&quot;:&quot;hm?&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0}&#xA;{&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299b8&quot;,&quot;author_flair_css_class&quot;:null,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;paternoster&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;I think so...&quot;,&quot;id&quot;:&quot;c0299b8&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;parent_id&quot;:&quot;t1_c0299b2&quot;,&quot;created_utc&quot;:&quot;1192450770&quot;,&quot;ups&quot;:1,&quot;controversiality&quot;:0}&#xA;{&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299b9&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;parent_id&quot;:&quot;t1_c0299ac&quot;,&quot;created_utc&quot;:&quot;1192450771&quot;,&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299b9&quot;,&quot;retrieved_on&quot;:1427426409,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;author&quot;:&quot;igiveyoumylife&quot;,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;well... I read Ron Paul's website.&quot;,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null}&#xA;{&quot;parent_id&quot;:&quot;t1_c0299ap&quot;,&quot;created_utc&quot;:&quot;1192450772&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:2,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299ba&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;author&quot;:&quot;zoomzoom83&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;No&quot;,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299ba&quot;,&quot;retrieved_on&quot;:1427426409}&#xA;{&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;body&quot;:&quot;Cry me a river.  Most patents and\r\ncopyrights these days are theft  of\r\npublic domain under the color of\r\nlaw, to an absurd degree.\r\n\r\nMickey Mouse copyright extended to\r\n100 years as a favor to everyone's\r\nfavorite political donor, Will Eisner.\r\nAmazon's one click patent.\r\n\r\nIt's probably been 20 years \r\nsince the Constitutional authority\r\nto grant patents and copyrights\r\nactually served, on the whole, to\r\n\&quot;advance the arts and commerce\&quot;,\r\nrather than to stifle them.\r\n&quot;,&quot;score_hidden&quot;:false,&quot;link_id&quot;:&quot;t3_2zxa6&quot;,&quot;score&quot;:1,&quot;author&quot;:&quot;donh&quot;,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299bb&quot;,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t3_2zxa6&quot;,&quot;created_utc&quot;:&quot;1192450781&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;id&quot;:&quot;c0299bb&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null}&#xA;{&quot;parent_id&quot;:&quot;t1_c0299b2&quot;,&quot;created_utc&quot;:&quot;1192450789&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:1,&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299bc&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;author&quot;:&quot;tashbarg&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:1,&quot;body&quot;:&quot;jup&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299bc&quot;,&quot;retrieved_on&quot;:1427426409}&#xA;{&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299bd&quot;,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;body&quot;:&quot;As long as we can still say \&quot;Web 2.0\&quot;...&quot;,&quot;score_hidden&quot;:false,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;author&quot;:&quot;newton_dave&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299bd&quot;,&quot;distinguished&quot;:null,&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c0299ap&quot;,&quot;created_utc&quot;:&quot;1192450790&quot;}&#xA;{&quot;ups&quot;:3,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t1_c0299b7&quot;,&quot;created_utc&quot;:&quot;1192450797&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299be&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;what's new except the asynchronous submit (which alone isn't really worth being dubbed 'new comment system')?&quot;,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:3,&quot;author&quot;:&quot;hfaber&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299be&quot;,&quot;retrieved_on&quot;:1427426409,&quot;edited&quot;:false,&quot;subreddit&quot;:&quot;reddit.com&quot;}&#xA;{&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299bf&quot;,&quot;score&quot;:6,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;paternoster&quot;,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;Yep&quot;,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299bf&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;parent_id&quot;:&quot;t1_c0299b0&quot;,&quot;created_utc&quot;:&quot;1192450809&quot;,&quot;ups&quot;:6,&quot;controversiality&quot;:0}&#xA;{&quot;name&quot;:&quot;t1_c0299bg&quot;,&quot;retrieved_on&quot;:1427426409,&quot;author_flair_css_class&quot;:null,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;score_hidden&quot;:false,&quot;body&quot;:&quot;[deleted]&quot;,&quot;score&quot;:1,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;author&quot;:&quot;[deleted]&quot;,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;id&quot;:&quot;c0299bg&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;distinguished&quot;:null,&quot;ups&quot;:1,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450827&quot;}&#xA;{&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299bh&quot;,&quot;distinguished&quot;:null,&quot;archived&quot;:true,&quot;downs&quot;:0,&quot;created_utc&quot;:&quot;1192450831&quot;,&quot;parent_id&quot;:&quot;t1_c0299az&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:2,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299bh&quot;,&quot;author_flair_css_class&quot;:null,&quot;author&quot;:&quot;newton_dave&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;body&quot;:&quot;You can stop anytime you want.\n\nReally.&quot;,&quot;score_hidden&quot;:false}&#xA;{&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299bi&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;created_utc&quot;:&quot;1192450849&quot;,&quot;parent_id&quot;:&quot;t1_c02999p&quot;,&quot;controversiality&quot;:0,&quot;ups&quot;:3,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;author_flair_css_class&quot;:null,&quot;retrieved_on&quot;:1427426409,&quot;name&quot;:&quot;t1_c0299bi&quot;,&quot;author&quot;:&quot;lyon&quot;,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:3,&quot;body&quot;:&quot;hmm only worked after refresh...&quot;,&quot;score_hidden&quot;:false,&quot;gilded&quot;:0,&quot;author_flair_text&quot;:null}&#xA;{&quot;body&quot;:&quot;okay, dokay, lets see how the comment work now ...  PING !!!&quot;,&quot;score_hidden&quot;:false,&quot;author_flair_text&quot;:null,&quot;gilded&quot;:0,&quot;link_id&quot;:&quot;t3_5yba3&quot;,&quot;score&quot;:2,&quot;author&quot;:&quot;joe24pack&quot;,&quot;author_flair_css_class&quot;:null,&quot;name&quot;:&quot;t1_c0299bj&quot;,&quot;retrieved_on&quot;:1427426409,&quot;subreddit&quot;:&quot;reddit.com&quot;,&quot;edited&quot;:false,&quot;ups&quot;:2,&quot;controversiality&quot;:0,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;created_utc&quot;:&quot;1192450869&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;distinguished&quot;:null,&quot;id&quot;:&quot;c0299bj&quot;,&quot;subreddit_id&quot;:&quot;t5_6&quot;}&#xA;{&quot;distinguished&quot;:null,&quot;subreddit_id&quot;:&quot;t5_6&quot;,&quot;id&quot;:&quot;c0299bk&quot;,&quot;downs&quot;:0,&quot;archived&quot;:true,&quot;created_utc&quot;:&quot;1192450870&quot;,&quot;parent_id&quot;:&quot;t3_5yba3&quot;,&quot;controversiality&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14757" LastEditorUserId="11097" LastEditDate="2015-12-15T03:40:25.877" LastActivityDate="2015-12-15T09:11:31.150" Title="How should I analyze this data from reddit (sample text included)" Tags="|data-cleaning|" AnswerCount="1" CommentCount="1" ClosedDate="2015-12-15T09:27:20.790" ContentLicense="CC BY-SA 3.0" />
  <row Id="9390" PostTypeId="1" AcceptedAnswerId="9403" CreationDate="2015-12-15T03:30:28.397" Score="1" ViewCount="99" Body="&lt;p&gt;I have constructed a lookup table using locality-sensitive hashing for comparing nearly similar documents/records. If two records (columns) have the same hash value in a row, they are considered to be similar. For example, the structure is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   R1 R2 R3 R4 R5&#xA;b1 a1 a2 a3 a2 a5 .....&#xA;b2 a2 a4 a1 a4 a4 ..... &#xA;b3 a3 a5 a3 a7 a4....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since each band holds similar records, similar record sets will be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S1 = {R2,R4}&#xA;S2 = {R2, R4, R5}&#xA;S3 = {R1,R3}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the merged similarity structures will be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S1' = {R2,R4,R5}&#xA;S2' = {R1,R3}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to span the matrix and map all similarity structures so I have buckets of similar records. The dimensionality of the hash table is large, hence a naive approach is not likely to work. What types of algorithms should I consider to achieve this efficiently?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1:&#xA;Question updated to clarify more on the objective &lt;/p&gt;&#xA;" OwnerUserId="14758" LastEditorUserId="13413" LastEditDate="2015-12-15T18:42:54.880" LastActivityDate="2015-12-15T19:21:12.943" Title="Algorithm to construct similarity structure from hash lookup table" Tags="|algorithms|graphs|named-entity-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9391" PostTypeId="2" ParentId="9370" CreationDate="2015-12-15T03:42:50.740" Score="2" Body="&lt;p&gt;You seem to be using the Random Forest model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't see how that feature would influence the model. It actually doesn't make a difference, as random forest divides the sample space iteratively, and your sample space would be divided as &lt;code&gt;switch = 0&lt;/code&gt; and &lt;code&gt;switch = 1&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the presence of those sample points do not affect the model.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-15T03:42:50.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9392" PostTypeId="1" AcceptedAnswerId="9394" CreationDate="2015-12-15T05:54:18.307" Score="3" ViewCount="312" Body="&lt;p&gt;I have data in the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;table 1&lt;br&gt;&#xA;id, feature1, predict &lt;br&gt;&#xA;1, xyz,yes&lt;br&gt;&#xA;2, abc, yes&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;table2&lt;br&gt;&#xA;id, feature2&lt;br&gt;&#xA;1, class1&lt;br&gt;&#xA;1, class2&lt;br&gt;&#xA;1, class3&lt;br&gt;&#xA;2, class2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could perform a one many join and train on the resultant set- which is one way to go about it. But If I rather wanted to maintain the length of the resultant set equal length of table 1, what is the technique?&lt;/p&gt;&#xA;" OwnerUserId="14763" LastActivityDate="2015-12-15T08:23:17.867" Title="Handling a feature with multiple categorical values for the same instance value" Tags="|machine-learning|dataset|data-cleaning|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9393" PostTypeId="2" ParentId="9248" CreationDate="2015-12-15T07:26:34.523" Score="3" Body="&lt;p&gt;Similarity measures are subjective and so are they ways to combine them.&#xA;You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I like to reduce similarity problems into classification problems. Given the dataset of items you have, create a new dataset of item pairs. The concept should be whether the two items in a pair are similar. Each similarity measure you have is a feature of the pair. Note that now you are in the good old classification framework. You can evaluate the similarity measures by computing the mutual information/accuracy/your chosen metric given the concept. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I would build the dataset this way. First choose pairs of items on which both your measures agree on. Choose pairs that are close by both the Euclidian distance and the cosine distance or pairs that are far by both measures. Note that since the similarity measures agree on these pair, combining them usually  leads to agreement too and not that important. However, you should use such pairs so they will stabilize your weighting function. Other than that, you can easily generate these pair, which is always an advantage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you should choose pair on which your similarity function disagree. Choose some that are considered close by the first and far by the second and vice versa. Manually label the pairs in order to decide whether they are close or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you built the dataset, you can present your subjective definition of similarity as a set of pairs and distances. Now run any classifier (in your case a linear regression might fit what you look for) and use the result as the combined similarity measure. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-12-15T07:26:34.523" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9394" PostTypeId="2" ParentId="9392" CreationDate="2015-12-15T08:23:17.867" Score="2" Body="&lt;p&gt;One possible approach is to perform an encoding, where each level of the feature2 corresponds to a new feature (column).&#xA;This way you may describe the 1:N relation between the feature 1 and 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here a small example in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; table1  &amp;lt;- data.frame(id = c(1,2),  feature1 = c(&quot;xyz&quot;,&quot;abc&quot;), predict = c(T,T))&#xA;&amp;gt; table2  &amp;lt;- data.frame(id = c(1,1,1,2), feature2 = c(&quot;class1&quot;, &quot;class2&quot;, &quot;class3&quot;, &quot;class2&quot;))&#xA;&amp;gt; &#xA;&amp;gt; ## encoding&#xA;&amp;gt; table(table2)&#xA;   feature2&#xA;id  class1 class2 class3&#xA;  1      1      1      1&#xA;  2      0      1      0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The new object contains the (now unique) &lt;code&gt;id&lt;/code&gt; and setting of the feature2.&#xA;You need only to merge (join) the result to the &lt;code&gt;table1&lt;/code&gt; (basically same task a DB join - which variance: inner, outer or full depends on your requirements).&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2015-12-15T08:23:17.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9395" PostTypeId="2" ParentId="9085" CreationDate="2015-12-15T08:24:10.153" Score="1" Body="&lt;p&gt;I like @Anony-Mousse's answer. Using robust estimators is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to add a different direction to cope with the problem.&#xA;It seems that there are some &quot;malicious&quot; users casting these down votes so you might want to identify them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a dataset of the users and use &quot;casted &lt;strong&gt;unjustified&lt;/strong&gt; down vote on leading item&quot; as the label. You can use &quot;casted down vote on leading item&quot; as the default value and then manually modify them and make the rule more delicate like &quot;casted more than twice  down vote on leading item after the  item reached the top charts&quot;&#xA;I guess that features like number of low votes, number of low votes to leading items, etc will be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you are in a supervised learning framework. Once you identify malicious users, ignore their votes and avoid the manipulations.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2015-12-15T08:24:10.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9396" PostTypeId="2" ParentId="9389" CreationDate="2015-12-15T09:11:31.150" Score="3" Body="&lt;p&gt;Data Science is not an algorithm to run on your data. It is the process that helps you answer a specific question. The key to be a data scientist is to ask the right questions. So, first, since you want to be familiar with machine learning, examine your data and try to understand what they can answer for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can I cluster the reddit questions in categories?&lt;/li&gt;&#xA;&lt;li&gt;Can you predict the score of a question based on the number of comments?&lt;/li&gt;&#xA;&lt;li&gt;Can you predict the sub-reddit that a user will ask a question?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="201" LastActivityDate="2015-12-15T09:11:31.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9397" PostTypeId="1" AcceptedAnswerId="9423" CreationDate="2015-12-15T10:12:37.867" Score="3" ViewCount="2266" Body="&lt;p&gt;For solving a prediction problem I'm willing to use the &lt;strong&gt;Factorization Machines&lt;/strong&gt;, a model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing interactions between features in this new space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was told that performing the hashing trick to convert categorical features to 1-of-k binary features (using sklearn’s &lt;strong&gt;DictVectorizer&lt;/strong&gt;, which returns sparse matrix) can destroy feature interaction and I should try regular &lt;strong&gt;one-hot encoding&lt;/strong&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain why? &lt;/p&gt;&#xA;" OwnerUserId="1093" LastEditorUserId="11097" LastEditDate="2015-12-15T10:14:54.837" LastActivityDate="2015-12-17T09:30:24.847" Title="Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding" Tags="|python|categorical-data|scikit-learn|feature-engineering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9398" PostTypeId="1" CreationDate="2015-12-15T13:25:39.840" Score="2" ViewCount="110" Body="&lt;p&gt;I try to predict temperatures values as function of time and different parameters. The temperature curve look like a &quot;ramp&quot; with some &quot;gauss peaks&quot; on regular intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I try to build a regression model for the following equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat{T}_{a, \mu, \sigma, s, c, m, p}\left(t\right) = a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to establish the values of $a$, $\mu$, $\sigma$, $s$, $c$, $m$, $p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first intention was to use the gradient descent, but I'm pretty sure that the derivative of the RSS based cost function won't be convex, so I might get stuck at local mimina.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also wondering if there is no better alternative to solve that problem. It concerns especially methods not involving minimizing derivatives. I think (but not sure) that neuronal networks can help me with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in short, how would you solve the following?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;	\left[\begin{array}{l}&#xA;			a \\&#xA;			\mu \\&#xA;			\sigma \\&#xA;			s \\&#xA;			c \\&#xA;			m \\&#xA;			p&#xA;	\end{array}\right] \leftarrow \textrm{argmin}_{a, \mu, \sigma, s, c, m, p}\frac{1}{2N} \sum_{i=1}^N{\left(a \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{\left(\sin\left(\frac{t_i-s}{c}\right) - \mu\right)^2}{2 \sigma^2}} + m t + p - T_i\right)^2}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: I'm using python 3 with scipy stack&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="3024" LastEditDate="2015-12-15T14:01:10.060" LastActivityDate="2023-12-14T10:04:55.610" Title="Regression problem - too complex for gradient descent" Tags="|predictive-modeling|regression|optimization|gradient-descent|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="9399" PostTypeId="2" ParentId="9362" CreationDate="2015-12-15T13:57:06.523" Score="1" Body="&lt;p&gt;My two cents...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not 100% accurate, but can give you a rough idea...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear regression makes a linearization of a problem where $y = f(x)$, with $x$ and $y$ are continuous variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now imagine that you want to predict a kind of boolean behavior (yes/no) based on a $x$ value. For example, based on your salary, are you happy or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can say happy = 1 and not happy = 0. You can make a scatter plot with all pairs (salary, happy) (happy in the vertical axis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try to make a line to separate the happy people from the unhappy ones, but you'll see quickly that's not working well (what is a value in the middle, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better idea would be to draw a kind of s curve which will pass as best through the points you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's what the logistics regression makes. It basically makes linear the s curve by transforming the $y$ values: this is the Logit function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then say that we predict &quot;True&quot; if the predicted logit is higher than a threshold. This threshold corresponds often to 0.5, which is the inflexion point of the curve (some tools are only allowing to use 0.5). That threshold is in fact the probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we have more than yes/no possibilities, one solution is to make the logistics regression for all the possibilities (e.g. if you have A, B, C, that will be A/not A, B/not B, C/not C) and take the possibility that give you highest probability. This is called the &quot;one-vs-all&quot; approach.&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="3024" LastEditDate="2016-02-15T08:49:11.300" LastActivityDate="2016-02-15T08:49:11.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9400" PostTypeId="2" ParentId="9370" CreationDate="2015-12-15T13:59:18.510" Score="4" Body="&lt;p&gt;Rather than discarding the dominant feature (which will discard information), try reducing the number of features randomly selected when making each partition. In scikit's syntax this is &lt;code&gt;max_features&lt;/code&gt; (&lt;code&gt;mtry&lt;/code&gt; in R's randomForest). By default this is set to compare the square root of all features (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&lt;/a&gt;), instead try much smaller (perhaps even 2 aka decision stump). You'll probably also need many more trees vs. higher settings. This will allow you to extract information from more subtle features without losing all that great information provided by the dominant feature.&lt;/p&gt;&#xA;" OwnerUserId="14572" LastActivityDate="2015-12-15T13:59:18.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9402" PostTypeId="2" ParentId="9085" CreationDate="2015-12-15T17:24:34.147" Score="1" Body="&lt;p&gt;To robustify your estimator, you might model your ratings as a Gaussian mixture model (GMM) that is a mixture of two Gaussian rvs: 1) true ratings, 2) junk rating that are equal to one.  Scikit-learn already has a canned GMM classifier: &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_classifier.html#example-mixture-plot-gmm-classifier-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_classifier.html#example-mixture-plot-gmm-classifier-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Digging in a little more, a simple approach would be to let scikit-learn partition your ratings into two gaussians.  If one of the partitions ends up with a mean near one, then we can throw out those ratings.  Or, more elegantly, we can take the mean of the other, non-near-one Gaussian, as the true rating mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a bit of code for a ipython notebook that does this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.mixture import GMM&#xA;import numpy as np&#xA;%matplotlib inline&#xA;import matplotlib.pyplot as plt&#xA;import collections&#xA;&#xA;def make_ratings(mean,std,rating_cnt):&#xA;    rating_sample = np.random.randn(rating_cnt)*std + mean&#xA;    return np.clip(rating_sample,1,5).astype(int)&#xA;&#xA;def make_collection(true_mean,true_std,true_cnt,junk_count):&#xA;    true_ratings = make_ratings(true_mean,true_std,true_cnt)&#xA;    junk_ratings = make_ratings(1,0,junk_count)&#xA;    return np.hstack([true_ratings,junk_ratings])[:,np.newaxis]&#xA;&#xA;def robust_mean(X, th = 2.5, agg_th=2.5, default_agg=np.mean):&#xA;    classifier = GMM(n_components=2)&#xA;    classifier.fit(X)&#xA;    if np.min(classifier.means_) &amp;gt; th or default_agg(X)&amp;lt;agg_th:&#xA;        return default_agg(X)&#xA;    else:&#xA;        return np.max(classifier.means_)&#xA;&#xA;r_mean = 4.2&#xA;X = make_collection(r_mean,2,40,10)&#xA;plt.hist(X,5)&#xA;classifier = GMM(n_components=2)&#xA;classifier.fit(X)&#xA;plt.show()&#xA;print &quot;vars =&quot;,classifier.covars_.flatten()&#xA;print &quot;means = &quot;,classifier.means_.flatten()&#xA;print &quot;mean = &quot;,np.mean(X)&#xA;print &quot;median = &quot;,np.median(X)&#xA;print &quot;robust mean = &quot;, robust_mean(X)&#xA;print &quot;true mean = &quot;, r_mean&#xA;print &quot;prob(rating=1|class) = &quot;,classifier.predict_proba(1).flatten()&#xA;print &quot;prob(rating=true_mean|class) = &quot;,classifier.predict_proba(r_mean).flatten()&#xA;print &quot;prediction: &quot;, classifier.predict(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output for one run looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vars = [ 0.22386589  0.56931527]&#xA;means =  [ 1.32310978  4.00603523]&#xA;mean =  2.9&#xA;median =  3.0&#xA;robust mean =  4.00603523034&#xA;true mean =  4.2&#xA;prob(rating=1|class) =  [  9.99596493e-01   4.03507425e-04]&#xA;prob(rating=true_mean|class) =  [  1.08366762e-08   9.99999989e-01]&#xA;prediction:  [1 0 1 0 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 0 1 0 1 1 0&#xA; 1 1 1 0 0 0 0 0 0 0 0 0 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can simulate how will this works with a few monte carlo trials:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;true_means = np.arange(1.5,4.5,.2)&#xA;true_ratings = 40&#xA;junk_ratings = 10&#xA;true_std = 1&#xA;m_out = []&#xA;m_in = []&#xA;m_reg = []&#xA;runs = 40&#xA;for m in true_means:&#xA;    Xs = [make_collection(m,true_std,true_ratings,junk_ratings) for x in range(runs)]&#xA;    m_in.append([[m]*runs])&#xA;    m_out.append([[robust_mean(X, th = 2.5, agg_th=2,default_agg=np.mean) for X in Xs]])&#xA;    m_reg.append([[np.mean(X) for X in Xs]])&#xA;&#xA;m_in = np.array(m_in).T[:,0,:]&#xA;m_out = np.array(m_out).T[:,0,:]&#xA;m_reg = np.array(m_reg).T[:,0,:]&#xA;&#xA;plt.plot(m_in,m_out,'b.',alpha=.25)&#xA;plt.plot(m_in,m_reg,'r.',alpha=.25)&#xA;plt.plot(np.arange(0,5,.1),np.arange(0,5,.1),'k.')&#xA;plt.xlim([0,5])&#xA;plt.ylim([0,5])&#xA;plt.xlabel('true mean')&#xA;plt.ylabel('predicted mean')&#xA;plt.title(&quot;true_ratings=&quot; + str(true_ratings)&#xA;          + &quot;; junk_ratings=&quot; + str(junk_ratings)&#xA;         + &quot;; std=&quot;+str(true_std))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is pasted below.  The red is the mean rating and the blue is the proposed rating.  You can tweak the parameters to get slightly different behaviors.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/BjmBj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BjmBj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10384" LastActivityDate="2015-12-15T17:24:34.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9403" PostTypeId="2" ParentId="9390" CreationDate="2015-12-15T19:21:12.943" Score="1" Body="&lt;p&gt;The algorithm you are asking about is very straightforward. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you do, you are looking for &lt;a href=&quot;https://en.wikipedia.org/wiki/Connected_component_%28graph_theory%29&quot; rel=&quot;nofollow&quot;&gt;connected components&lt;/a&gt; in a certain graph where edges are determined by matching hash values. You can implement this with a modification of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Disjoint-set_data_structure&quot; rel=&quot;nofollow&quot;&gt;disjoint-set data structure&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your particular variation would be that apart from keeping track of vertices in each component, you have to keep track of m (number of rows) sets of hash values that have been found for each component.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2015-12-15T19:21:12.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9404" PostTypeId="2" ParentId="9378" CreationDate="2015-12-15T19:31:07.267" Score="7" Body="&lt;p&gt;Let us say that the output of one neural network given it's parameters is $$f(x;w)$$&#xA;Let us define the loss function as the squared L2 loss (in this case).&#xA;$$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2$$&#xA;In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of the subset being equal to your batch-size, and use the gradient normalized under this batch. We do this until we have exhausted every data-point in the dataset. Then the epoch is over. The gradient in this case:&#xA;$$\frac{\partial L(X,y;w)}{\partial w} = \frac{1}{n}\sum_{i=0}^{n}[f(X_i;w)-y_i]\frac{\partial f(X_i;w)}{\partial w}$$&#xA;Using batch gradient descent normalizes your gradient, so the updates are not as sporadic as if you have used stochastic gradient descent. &lt;/p&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2015-12-15T19:31:07.267" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9405" PostTypeId="2" ParentId="8850" CreationDate="2015-12-16T09:37:25.713" Score="1" Body="&lt;p&gt;Sounds like you want something like &lt;a href=&quot;http://spark.apache.org/docs/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;Apache Spark with its Python API&lt;/a&gt;. It is designed to run locally, or on a single server, OR in a distributed way, and its &quot;distributedness&quot; is  hidden away from you as much as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has a big community - I took this &lt;a href=&quot;https://www.edx.org/course/introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x&quot; rel=&quot;nofollow&quot;&gt;MOOC&lt;/a&gt; in summer 2015 together with 80000 participants, and it was a great introduction to the topic. It is still open for enrollments but unmaintained by the instructors.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2015-12-16T10:31:20.727" LastActivityDate="2015-12-16T10:31:20.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9406" PostTypeId="1" CreationDate="2015-12-16T11:39:09.137" Score="11" ViewCount="2629" Body="&lt;p&gt;Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $f$ wherever I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are regression trees capable of finding a smooth model of the function (hence a tiny change in the input should only give a tiny change in the output)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I've read in &lt;a href=&quot;http://www.stat.cmu.edu/~cshalizi/350-2006/lecture-10.pdf&quot;&gt;Lecture 10: Regression Trees&lt;/a&gt; it seems to me that regression trees basically put the function values into bins:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For classic regression trees, the model in each cell is just a constant estimate of&#xA;  Y .&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As they write &quot;classic&quot; I guess there is a variant where the cells do something more interesting?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-02T12:34:05.413" Title="Can regression trees predict continuously?" Tags="|predictive-modeling|regression|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9407" PostTypeId="1" CreationDate="2015-12-16T12:18:51.933" Score="0" ViewCount="85" Body="&lt;p&gt;I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also random, I should expect the result to follow a standard distribution. I have ten classes and 10 initial examples, so I am expecting my accuracy to be close to 10, most of the time, however I am often (too often) getting results that are quite far from 10 (like 0 or 49). I should expect to get those results sometimes, but not so often. Is there something wrong in my code?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import theano&#xA;import theano.tensor as T&#xA;import numpy as np&#xA;rng = np.random&#xA;#rng = np.random.RandomState(0)&#xA;&#xA;ninputs = 100&#xA;nfeatures = 784&#xA;noutputs = 10&#xA;nhiddens = 50&#xA;&#xA;#training rate&#xA;tr_rate = 0.1 &#xA;training_steps = 1&#xA;&#xA;D = (rng.randn(ninputs, nfeatures), rng.randint(size=ninputs, low=0, high=noutputs))&#xA;&#xA;# Feed-forward&#xA;x = T.dmatrix('x')&#xA;t = T.ivector('t')&#xA;&#xA;w = theano.shared(rng.randn(nfeatures, noutputs), name=&quot;w&quot;)&#xA;b = theano.shared(np.full(noutputs, 0.01), name=&quot;b&quot;)&#xA;sigma = T.nnet.softmax(T.dot(x,w) + b) &#xA;prediction = T.argmax(sigma, axis=1)   &#xA;&#xA;&#xA;parameters = [w, b]&#xA;&#xA;xent = -T.mean(T.log(sigma)[T.arange(t.shape[0]), t])          # Cost function&#xA;&#xA;cost = xent.mean() + 0.01 * (w ** 2).sum()                     # Regularisation &#xA;&#xA;gparameters = [T.grad(cost, param) for param in parameters]    # Compute the gradient of the cost &#xA;&#xA;pars = zip(parameters, gparameters)&#xA;&#xA;train = theano.function(&#xA;          inputs=[x,t],&#xA;          outputs=[xent],&#xA;          updates = [(param, param - tr_rate * gparam) for param, gparam in zip(parameters, gparameters)],&#xA;          allow_input_downcast = True )&#xA;&#xA;&#xA;predict = theano.function(inputs=[x], outputs=prediction)&#xA;&#xA;# Train&#xA;for i in range(training_steps):&#xA;    train(D[0], D[1])&#xA;&#xA;&#xA;result = predict(D[0]) - D[1]&#xA;&#xA;N = ninputs&#xA;&#xA;error = 0&#xA;for index in result:&#xA;    if result[index] != 0:&#xA;        error += 1&#xA;&#xA;correct_guesses = N - error&#xA;accuracy = (N - error)*100/N&#xA;&#xA;print&#xA;print &quot;correct predictions = %f over %i examples&quot; % (correct_guesses, N)&#xA;print &quot;accuracy = %i%%&quot; % accuracy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="11097" LastEditDate="2015-12-16T14:38:09.227" LastActivityDate="2015-12-17T07:07:04.163" Title="Logistic regression program does not give statistically correct result" Tags="|python|logistic-regression|theano|" AnswerCount="2" CommentCount="2" ClosedDate="2015-12-20T08:18:00.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="9408" PostTypeId="2" ParentId="9302" CreationDate="2015-12-16T13:29:30.290" Score="127" Body="&lt;p&gt;One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data &lt;span class=&quot;math-container&quot;&gt;$y_i'$&lt;/span&gt;, under a model &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Namely, suppose that you have some fixed model (a.k.a. &quot;hypothesis&quot;), which predicts for &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; classes &lt;span class=&quot;math-container&quot;&gt;$\{1,2,\dots, n\}$&lt;/span&gt; their hypothetical occurrence probabilities &lt;span class=&quot;math-container&quot;&gt;$y_1, y_2,\dots, y_n$&lt;/span&gt;. Suppose that you now observe (in reality) &lt;span class=&quot;math-container&quot;&gt;$k_1$&lt;/span&gt; instances of class &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$k_2$&lt;/span&gt; instances of class &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$k_n$&lt;/span&gt; instances of class &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt;, etc. According to your model the likelihood of this happening is:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}.&#xA;$$&lt;/span&gt;&#xA;Taking the logarithm and changing the sign:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;-\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i&#xA;$$&lt;/span&gt;&#xA;If you now divide the right-hand sum by the number of observations &lt;span class=&quot;math-container&quot;&gt;$N = k_1+k_2+\dots+k_n$&lt;/span&gt;, and denote the empirical probabilities as &lt;span class=&quot;math-container&quot;&gt;$y_i'=k_i/N$&lt;/span&gt;, you'll get the cross-entropy:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;-\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y)&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the log-likelihood of a dataset given a model can be interpreted as a measure of &quot;encoding length&quot; - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This follows from the observation that an independent event with probability &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt; requires at least &lt;span class=&quot;math-container&quot;&gt;$-\log_2 y_i$&lt;/span&gt; bits to encode it (assuming efficient coding), and consequently the expression &#xA;&lt;span class=&quot;math-container&quot;&gt;$$-\sum_i y_i'\log_2 y_i,$$&lt;/span&gt;&#xA;is literally the expected length of the encoding, where the encoding lengths for the events are computed using the &quot;hypothesized&quot; distribution, while the expectation is taken over the actual one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, instead of saying &quot;measure of expected encoding length&quot; I really like to use the informal term &quot;measure of surprise&quot;. If you need a lot of bits to encode an expected event from a distribution, the distribution is &quot;really surprising&quot; for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With those intuitions in mind, the answers to your questions can be seen as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Question 1&lt;/strong&gt;. Yes. It is a problem &lt;em&gt;whenever the corresponding &lt;span class=&quot;math-container&quot;&gt;$y_i'$&lt;/span&gt; is nonzero at the same time&lt;/em&gt;. It corresponds to the situation where your model believes that some class has zero probability of occurrence, and yet the class pops up in reality. As a result, the &quot;surprise&quot; of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That is why you get infinity as your cross-entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality, people tend to use sigmoid or &quot;softmax&quot; functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use some other hypothesis model, it is up to you to &lt;em&gt;regularize&lt;/em&gt; (aka &quot;smooth&quot;) it so that it would not hypothesize zeros where it should not.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Question 2&lt;/strong&gt;. In this formula, one usually assumes &lt;span class=&quot;math-container&quot;&gt;$y_i'$&lt;/span&gt; to be either &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;, while &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt; is the model's probability hypothesis for the corresponding input. If you look closely, you will see that it is simply a &lt;span class=&quot;math-container&quot;&gt;$-\log P[data|model]$&lt;/span&gt; for binary data, an equivalent of the second equation in this answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as &lt;em&gt;cross-entropy&lt;/em&gt; is that it is, in fact, a &lt;em&gt;sum&lt;/em&gt; over binary cross-entropies for individual points in the dataset:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\sum_i H(y_i', y_i),&#xA;$$&lt;/span&gt;&#xA;where &lt;span class=&quot;math-container&quot;&gt;$y_i'$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt; have to be interpreted as the corresponding binary distributions &lt;span class=&quot;math-container&quot;&gt;$(y_i', 1-y_i')$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$(y_i, 1-y_i)$&lt;/span&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14519" LastEditorUserId="14519" LastEditDate="2018-11-25T11:08:09.020" LastActivityDate="2018-11-25T11:08:09.020" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="9409" PostTypeId="2" ParentId="9085" CreationDate="2015-12-16T14:20:18.467" Score="0" Body="&lt;p&gt;While it is technically probably easiest to implement one of the above solutions, I think you should also consider disincentivising the voters to downvote. For example, if the downvotes come from a minority of users who are clearly abusing the system, their repeated downvotes should count (negatively) toward their reputation - like this site. &lt;/p&gt;&#xA;" OwnerUserId="14598" LastActivityDate="2015-12-16T14:20:18.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9410" PostTypeId="2" ParentId="9407" CreationDate="2015-12-16T14:41:34.937" Score="1" Body="&lt;p&gt;If you initialized the weights randomly and predicted immediately, you should probably expect &lt;code&gt;1/num_classes&lt;/code&gt; accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you are training first. The data may be random, but it is still possible to overfit it. With a sample that small and wide, you could imagine some degree of separability of the classes by chance. In that scenario, your network can learn weights that are better than random (far from perfect in only one iteration).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect would likely be lessened by using much more data or holding out/using new random data to test accuracy.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-16T14:41:34.937" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9411" PostTypeId="1" AcceptedAnswerId="10004" CreationDate="2015-12-16T14:55:27.443" Score="7" ViewCount="1456" Body="&lt;p&gt;Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 training examples for each category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All looks good and the classifier is working with acceptable precision on unseen documents. But what I am noticing that often when classifying a new document, the classifier is often assigning a very high probability to one of the categories (the prediction probability is almost equal to 1), while the other categories receive very low probabilities (close to zero).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the possible reasons for this phenomenon?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I can think of one possible reason, which is all the words in some documents did not appear with some categories in the training dataset? but I am not totally convinced of that especially that for most of the results, there is always one category who has very high probability and all the others have very low probability.&lt;/li&gt;&#xA;&lt;li&gt;Is there any other explanations or reasons?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to add that in SPARK ML there is something called &quot;raw prediction&quot; and when I look at it, I can see negative numbers but they have more or less comparable magnitude, so even the category with the high probability has comparable raw prediction score, but I am finding difficulties in interpreting this scores.&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="8820" LastEditDate="2016-01-27T06:39:38.793" LastActivityDate="2016-01-27T21:18:58.680" Title="SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction" Tags="|classification|apache-spark|multiclass-classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9412" PostTypeId="2" ParentId="9406" CreationDate="2015-12-16T16:06:42.623" Score="0" Body="&lt;p&gt;In classic regression trees you have a one value in the leaf, but in the leaf you can have a linear regression model, check &lt;a href=&quot;https://stats.stackexchange.com/questions/78563/regression-tree-algorithm-with-linear-regression-models-in-each-leaf&quot;&gt;this&lt;/a&gt; ticket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use ensemble of trees (Random Forest or Gradient Boosting Machines) to have continuous output value.&lt;/p&gt;&#xA;" OwnerUserId="14636" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-12-16T16:06:42.623" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9413" PostTypeId="1" AcceptedAnswerId="9414" CreationDate="2015-12-16T16:13:42.943" Score="2" ViewCount="119" Body="&lt;p&gt;With a team of researchers we were given the assignment to make a scale for the &lt;em&gt;move kindness&lt;/em&gt; (how inviting a room or place is for exercise--e.g., a gym). In order to get objective results we were asked to make a measurement device. This device receives input through various sensors and maybe some true/false questions, and then analyze it with training information in mind. After the analysis it returns a number from, say: 1 to 10, indicating the move kindness. So a gym gets, for example an eight and a classroom a three. The problem with this is, that move kindness is very subjective, so we have conducted some surveys. For example one of the criteria is the temperature of the room/place. While the survey was being conducted we measured the temperature:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;From a scale from 1 to 10, what is your opinion about the temperature?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And then we measured the temperature. We have put all these information into some  spreadsheets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rating (Move Kindness): 8&lt;br&gt;&#xA;Temperature: 18 degrees Celsius&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of this survey we asked them to give the move kindness a rating.&#xA;So we have this, for example: &lt;br&gt;&#xA;Temperature: 8, 18 &lt;br&gt;&#xA;Light: 7, 300 &lt;br&gt;&#xA;Humidity: 8, 50 &lt;br&gt;&#xA;.... &lt;br&gt;&#xA;Rating (Move Kindness): 8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, what's the best way to analyse these data for a reliable measurement device using python?&#xA;We were thinking of using neural networks, because they can be trained, but logistic regression or some other machine learning algorithm is also an option. Can anyone give me some direction on this?&lt;/p&gt;&#xA;" OwnerUserId="14800" LastEditorUserId="13413" LastEditDate="2015-12-17T15:09:32.183" LastActivityDate="2021-04-21T02:47:11.873" Title="How to analyse move kindness in python (Logistic regression, neural network, etc.)?" Tags="|machine-learning|python|neural-network|logistic-regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9414" PostTypeId="2" ParentId="9413" CreationDate="2015-12-16T17:48:22.550" Score="1" Body="&lt;p&gt;Okay, so from what I understand, you have a regression problem taking into account a variety of physical features. The reason I say that this is a regression problem, verses a classification problem is because the scale you are trying to predict is an ordinal scale.&lt;/p&gt;&#xA;&lt;p&gt;There are a couple approaches to this. If your features are discriminative and linear enough, a simple least squares linear regression might work. If you believe the problem you have is to complicated for linear regressions, a simple vanilla neural network with one single output. I would recommend using the scikit-learn library in python for all models that are not neural networks. &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a link to the generalized linear regression page.&lt;br /&gt;&#xA;That link has code samples and mathematical explanations. If you decide to use neural networks, and you don't have a great amount of samples or a need to use the GPU, the &lt;a href=&quot;http://pybrain.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pyBrain&lt;/a&gt; library is great.&lt;/p&gt;&#xA;&lt;p&gt;I wouldn't recommend using a logistic regression (since you mentioned it in your question), simply because a logistic regression is a classification problem, and I believe you would be better off approaching this from a regression standpoint.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastEditorUserId="25180" LastEditDate="2021-04-21T02:47:11.873" LastActivityDate="2021-04-21T02:47:11.873" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9415" PostTypeId="1" CreationDate="2015-12-16T21:20:08.390" Score="6" ViewCount="595" Body="&lt;p&gt;I have a dataset with large (4500 variables) and 3000 observations (n &amp;lt; p case). I am using LASSO (from the glmnet package in R) to reduce dimensionality of the problem. I have couple of questions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) After I ran LASSO the first time, the number of variables with non-zero coefficients goes down to ~80 (which seems reasonable). But the correlation matrix shows that some of these variables are very highly correlated themselves (FORMS distinct groups if you plot it using the &lt;code&gt;corplot&lt;/code&gt; package. So I ran LASSO exclusively on these variables whose coefficients were non-zero during the 1st run. This time, the algorithm picked up even less number of variables.I was expecting that they would remain more or less similar. I want to know if running LASSO multiple times is advisable or it introduces bias of any form in anyway that interferes with interpretation/prediction ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) My second question is about the magnitude of the coefficients. Some of the coefficients, in absolute value, turns out to be very close to 0 (~1E-14) while some are larger. I mean the spread is very large. Does it indicate that the variables with this small coefficients are making the model worse in any way. Is it advisable to get rid off those variables and rerun LASSO ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts/comments are most welcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2016-07-24T21:46:46.360" Title="Is it advisable to rerun LASSO multiple (2) times?" Tags="|machine-learning|data-mining|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9416" PostTypeId="1" CreationDate="2015-12-16T21:21:35.683" Score="1" ViewCount="249" Body="&lt;p&gt;I'm using varImpPlot() to evaluate the importance of variables from my rf model, and I can't decide wich variable I have to delete from my model. I'm not sure about the diference between &quot;MeanDecreaseAccuracy&quot; and &quot;MeanDecreaseGini&quot;, I guess the last one can measure the purity of the tree nodes, but as an example of my model: I have a variable that has a MeanDecreaseGini of about 100 and at the same time a MeanDecreaseAccuracy of about zero.! Because of this I can't decide if this variable is good or bad.&lt;/p&gt;&#xA;" OwnerUserId="14644" LastActivityDate="2016-09-12T16:32:49.787" Title="Importance of variables in RandomForest in R" Tags="|r|random-forest|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9417" PostTypeId="1" AcceptedAnswerId="9419" CreationDate="2015-12-16T21:49:31.960" Score="0" ViewCount="151" Body="&lt;p&gt;I want to know, why python? Instead of C#, Java, C\C++ and so on?&lt;/p&gt;&#xA;" OwnerUserId="14808" LastActivityDate="2015-12-17T01:55:19.170" Title="Why in deep learning or another AI technique python is used rather than other languages?" Tags="|machine-learning|python|java|" AnswerCount="1" CommentCount="0" ClosedDate="2015-12-17T15:10:03.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="9418" PostTypeId="1" CreationDate="2015-12-17T01:42:57.980" Score="3" ViewCount="109" Body="&lt;p&gt;I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types of preprocessing are common ?&lt;/p&gt;&#xA;" OwnerUserId="543" LastActivityDate="2015-12-17T01:42:57.980" Title="Is it common to preprocess image data before sending it through a deep net?" Tags="|image-classification|preprocessing|object-recognition|" AnswerCount="0" CommentCount="0" ClosedDate="2015-12-17T22:29:50.580" ContentLicense="CC BY-SA 3.0" />
  <row Id="9419" PostTypeId="2" ParentId="9417" CreationDate="2015-12-17T01:55:19.170" Score="2" Body="&lt;p&gt;Short answers&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Tooling.&lt;/strong&gt; Python has fantastic math, statistics, and linear algebra libraries.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Less Code, Same Result.&lt;/strong&gt; Python provides quick and simple ways of achieving programming solutions compared to C#, Java, C++, etc. This means you'll write less code and achieve the same result.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14812" LastActivityDate="2015-12-17T01:55:19.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9420" PostTypeId="1" AcceptedAnswerId="10279" CreationDate="2015-12-17T02:44:52.977" Score="12" ViewCount="3745" Body="&lt;p&gt;My colleague and I are trying to wrap our heads around the difference between logistic regression and an SVM. Clearly they are optimizing different objective functions. Is an SVM as simple as saying it's a discriminative classifier that simply optimizes the hinge loss? Or is it more complex than that? How do the support vectors come into play? What about the slack variables? Why can't you have deep SVM's the way you can't you have a deep neural network with sigmoid activation functions?&lt;/p&gt;&#xA;" OwnerUserId="1301" LastActivityDate="2016-02-17T11:12:45.037" Title="What's the relationship between an SVM and hinge loss?" Tags="|svm|logistic-regression|hinge-loss|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9421" PostTypeId="2" ParentId="9407" CreationDate="2015-12-17T07:07:04.163" Score="0" Body="&lt;p&gt;Found the issue, a simple bug in my code. I wrote:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for index in result:&#xA;    if result[index] != 0:&#xA;        error += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while it should have been something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for index in range(N):&#xA;    if result[index] != 0:&#xA;        error += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2015-12-17T07:07:04.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9422" PostTypeId="1" CreationDate="2015-12-17T07:58:13.057" Score="1" ViewCount="262" Body="&lt;p&gt;I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.&#xA;The operations on the data are simple (JOIN and STORE), but the data is not getting loaded completely, and the program is terminating with a java outofmemory exception.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried increasing the Pig Heap size to 8192, but that hasn't worked, however my code works fine if I use only 25% of the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the last line of the complete log file:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;20124550 [Service Thread] DEBUG &lt;br&gt;&#xA;  org.apache.pig.impl.util.SpillableMemoryManager  - memory handler call - &lt;br&gt;Collection threshold init = 374341632(365568K) used =&#xA;  698749896(682372K) committed = 698875904(682496K) max =&#xA;  698875904(682496K)&lt;br&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;java.lang.OutOfMemoryError: Java heap space&#xA;  -XX:OnOutOfMemoryError=&quot;kill -9 %p&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's my code for loading the dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;log  = LOAD s3://some-path/{20151101..20151130}/{00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23}/' USING PigStorage ('\n') AS (line:chararray);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've specified each date individually, so that is not the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I've tried using 35 machines in the cluster, but that hasn't helped. I could potentially add more, but that is not the type of solution I am looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone can suggest better approach to coding this problem, or some other solution?&lt;/p&gt;&#xA;" OwnerUserId="13508" LastEditorUserId="13413" LastEditDate="2015-12-18T22:14:58.593" LastActivityDate="2016-12-28T10:45:15.867" Title="Pig is not able to read the complete data" Tags="|apache-hadoop|map-reduce|apache-pig|" AnswerCount="0" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9423" PostTypeId="2" ParentId="9397" CreationDate="2015-12-17T09:30:24.847" Score="5" Body="&lt;p&gt;I decided to expand a bit on my comment and make a full answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the reason why somebody may say that performing the hashing trick can destroy interactions is because it may map different features to the same bucket. But usually it's not a big deal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;DictVectorizer&lt;/a&gt; doesn't perform the hashing trick:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When feature values are strings, this transformer will do a binary one-hot (aka one-of-K) coding: one boolean-valued feature is constructed for each of the possible string values that the feature can take on. For instance, a feature “f” that can take on the values “ham” and “spam” will become two features in the output, one signifying “f=ham”, the other “f=spam”.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To do it, you need to use a different vectorizer: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;HashingVectorizer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2015-12-17T09:30:24.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9424" PostTypeId="1" CreationDate="2015-12-17T10:52:10.013" Score="7" ViewCount="8426" Body="&lt;p&gt;I am using &lt;code&gt;LogisticRegressionWithLBFGS&lt;/code&gt; to train a multi-class classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to get the probability of all classes (not only the top candidate class) when I test the model on new unseen samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I am not necessarily obliged to use the LBFGS classifier but I would like to use the logistic regression in my problem. So if there is a solution by using another LR classifier type, I would go for it.&lt;/p&gt;&#xA;" OwnerUserId="13891" LastActivityDate="2016-09-09T11:53:49.333" Title="SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?" Tags="|classification|apache-spark|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9425" PostTypeId="1" CreationDate="2015-12-17T16:32:03.567" Score="4" ViewCount="1954" Body="&lt;p&gt;I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of experience I am confused on how to clean them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, 6 out of 11 columns have more than 277000 NA values. In that case I know that I cannot remove them because they are a big portion of the dataset so I have to impute them. Usually I would do mean substitution but I've read that this approach might create bias in the data and I wouldn't like to have that. I have tried the packages Amelia and mice in R, mice couldn't run properly and after a while it was giving me an error, Amelia was very fast and completed 5 imputations but it introduced many negative values in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone with similar experience on that?&lt;/p&gt;&#xA;" OwnerUserId="10894" LastActivityDate="2015-12-18T07:05:01.303" Title="Handling many missing values" Tags="|r|missing-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9426" PostTypeId="2" ParentId="9425" CreationDate="2015-12-17T18:48:27.133" Score="0" Body="&lt;p&gt;This is not a technical question. You should decide which one is more important for you: the columns or the rows that contain these 277000 NA values. Probably the columns are more important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend deleting the rows at first, building  a model (starting from one variable and adding more variables gradually), calculating the model measures (such as adjusted R-Squared) . At this point you have some baseline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then restore these deleted rows and see how your model measures change. Maybe these 6 columns are not predictors of your response variable and you are worried for nothing. There is no magic to say which method works for your data. Give it a try. This is exactly what is done in statistics and machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After understanding your data better, you can decide to fill your missing values or not. Keep in mind that data analysis is an iterative process.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2015-12-17T18:48:27.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9428" PostTypeId="1" CreationDate="2015-12-18T01:31:06.703" Score="3" ViewCount="116" Body="&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: Can Stata present or output the actual betas/coefficients in a datasheet when it performs the conditional logit model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that Stata has a &quot;predict&quot; capability, but I need to store the betas/coefficients and use them later, after new factor values have been calculated.&lt;/p&gt;&#xA;" OwnerUserId="5053" LastEditorUserId="11097" LastEditDate="2015-12-18T10:26:53.477" LastActivityDate="2016-04-08T21:11:05.903" Title="Conditional logit in Stata" Tags="|predictive-modeling|logistic-regression|stata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9429" PostTypeId="2" ParentId="9425" CreationDate="2015-12-18T07:05:01.303" Score="1" Body="&lt;p&gt;There are a number of different ways to impute data when you have missing values. Far to many people just delete those records, which often contain useful training data.  Further, some people advocate for simple methods like using the mean of the feature when it is missing.  I don't like this method as it seems too simplistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.amstat.org/sections/srms/webinarfiles/ModernMethodWebinarMay2012.pdf&quot; rel=&quot;nofollow&quot;&gt;Here is a good presentation&lt;/a&gt; detailing a couple of methods and particularly drawing attention to imputation through maximum likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my favorite method is collaborative filtering, which is &lt;a href=&quot;http://papersdb.cs.ualberta.ca/~papersdb/uploaded_files/916/paper_SAC08_DM101.pdf&quot; rel=&quot;nofollow&quot;&gt;detailed in this paper&lt;/a&gt;. The nice thing about collaborative filtering is that it is easy to write from scratch and is also very popular, so is usually contained in most machine learning libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also think about taking a look at the imputation techniques that might be available in the specific machine learning library that you may be using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a word of caution... imputing data adds some linear dependence, so you should expect that you might have to increase your regularization slightly in order to avoid high variance scenarios. I would experiment with (cross validate) removing features, removing records, then &lt;a href=&quot;http://papersdb.cs.ualberta.ca/~papersdb/uploaded_files/967/paper_08_WI.pdf&quot; rel=&quot;nofollow&quot;&gt;imputing data using collaborative filtering&lt;/a&gt; and see what produces the best results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-12-18T07:05:01.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9430" PostTypeId="2" ParentId="6987" CreationDate="2015-12-18T07:54:21.370" Score="28" Body="&lt;p&gt;A regular SVM with default values uses a radial basis function as the SVM kernel.  This is basically a Gaussian kernel aka bell-curve.  Meaning that the no man's land between different classes is created with a Gaussian function.  The linear-SVM uses a linear kernel for the basis function, so you can think of this as a ^ shaped function.  It is much less tunable and is basically just a linear interpolation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People are kind of hammering this question because you didn't provide a lot of information, but looking deeply into what you posted... this hits on some fundamental aspects of really understanding the details of bias and variance and the difference between linear and nonlinear basis functions in SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-overfitting-machine-learning-overview/&quot; rel=&quot;noreferrer&quot;&gt;this image&lt;/a&gt; describing the four regions of high and low bias and high and low variance.  Obviously the best place to be is  low variance and low bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jyHqx.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jyHqx.png&quot; alt=&quot;darts bias-variance image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First lets assess variance -&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now take a look at your plots:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/fIVsR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fIVsR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nonlinear basis function has higher variance.  See how it is noisier than the linear kernel!  The linear kernel has lower variance.  See how it is less noisy!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now lets assess bias -&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which kernel is more accurate?  We can add the errors that you provided.  The nonlinear kernel has a total error of ~550+325=~875.  The linear kernel has an error of ~690+~50=~740.  So the linear kernel &lt;strong&gt;seems&lt;/strong&gt; to do better overall, but they are pretty close overall.  This is were things get tricky!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Putting it all together&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See how the linear kernel did a poor job on 1's and a really great job on 0's.  This is pretty unbalanced.  Where as the nonlinear kernel is more balanced.  It kind of seems like the sweet spot might be to create a balanced model that doesn't have such high variance.  How do we control for high variance?  Bingo - regularization.  We can add regularization to the nonlinear model and we will probably see much better results.  This is the C parameter in scikit learn SVMs, which you will want to increase from the default.  We could also play with the gamma parameter.  Gamma controlls the width of the Gaussian.  Maybe try increasing that one slightly to get less noisy results e.g. a larger no-man's land between classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-12-18T07:54:21.370" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9431" PostTypeId="2" ParentId="5706" CreationDate="2015-12-18T10:50:59.480" Score="147" Body="&lt;p&gt;Let's review how the ReLU (Rectified Linear Unit) looks like :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/8CGlM.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8CGlM.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input to the rectifier for some input $x_n$ is &#xA;$$z_n=\sum_{i=0}^k w_i a^n_i$$ for weights $w_i$, and activations from the previous layer $a^n_i$ for that particular input $x_n$. The rectifier neuron function is $ReLU = max(0,z_n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming a very simple error measure &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$error = ReLU - y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the rectifier has only 2 possible gradient values for the deltas of backpropagation algorithm:&#xA;$$\frac{\partial error}{\partial z_n} = \delta_n = \left\{&#xA;\begin{array}{c l}     &#xA;    1 &amp;amp; z_n \geq 0\\&#xA;    0 &amp;amp; z_n &amp;lt; 0&#xA;\end{array}\right.$$&#xA;(if we use a proper error measure, then the 1 will become something else, but the 0 will stay the same)&#xA;and so for a certain weight $w_j$ :&#xA;$$\nabla error = \frac{\partial error}{\partial w_j}=\frac{\partial error}{\partial z_n} \times \frac{\partial z_n}{\partial w_j} = \delta_n \times a_j^n = \left\{&#xA;\begin{array}{c 1}&#xA;    a_j^n &amp;amp; z_n \geq 0\\&#xA;    0 &amp;amp; z_n &amp;lt; 0&#xA;\end{array}\right.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One question that comes to mind is how actually ReLU works &quot;at all&quot; with the gradient $=$ 0 on the left side. What if, for the input $x_n$, the current weights put the ReLU on the left flat side while it optimally should be on the right side for this particular input ? The gradient is 0 and so the weight will not be updated, not even a tiny bit, so where is &quot;learning&quot; in this case? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The essence of the answer lies in the fact that Stochastic Gradient Descent will not only consider a single input $x_n$, but many of them, and the hope is that not all inputs will put the ReLU on the flat side, so the gradient will be non-zero for &lt;strong&gt;some&lt;/strong&gt; inputs (it may be +ve or -ve though). If at least one input $x_*$ has our ReLU on the steep side, then the ReLU is still &lt;strong&gt;alive&lt;/strong&gt;  because there's still learning going on and weights getting updated for this neuron. If all inputs put the ReLU on the flat side, there's no hope that the weights change at all and the neuron is &lt;strong&gt;dead&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A ReLU may be alive then die due to the gradient step for some &lt;strong&gt;input batch&lt;/strong&gt; driving the weights to smaller values, making $z_n &amp;lt; 0$ for all inputs. A large learning rate amplifies this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @Neil Slater mentioned, a fix is to modify the flat side to have a small gradient, so that it becomes $ReLU=max(0.1x,x)$ as below, which is called LeakyReLU.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LZk6i.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LZk6i.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="860" LastEditorUserId="-1" LastEditDate="2016-01-08T19:26:10.760" LastActivityDate="2016-01-08T19:26:10.760" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9432" PostTypeId="1" AcceptedAnswerId="9609" CreationDate="2015-12-18T10:54:15.307" Score="4" ViewCount="497" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I was planning to use a graph theoretic and page-rank approach to finding the most influential person in an organization.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Influential person is someone who drives a lot of activity in the&#xA;  organization. When he assigns a task, most people do it. When he sends&#xA;  a mail, most people reply to him. When he assigns some training, most&#xA;  people do it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But there is one drawback to using graph theory here. For example, say Person A is very influential, and Person B has just joined an organization and works under A. B puts in a leave request and sends it A for approval. A approves the leave request. This seems contradictory, since A is very influential but A responded to B. According to graph theory, wouldn't this make Person B also very influential? This approach would give very bad results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can i overcome this limitation? Can anyone suggest an alternative approach?&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I am using number of interactions to calculate an &lt;code&gt;affinity scores&lt;/code&gt; between two users. Based on this, I suggest what content should be shown to a user (like FaceBook does). User A responds to User B most of the time.  so (A-&gt;B) has a high affinity score. So, in user B's newsfeed I intend to show content coming from User A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;is there a better way than just counting number of interactions?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If two users A &amp;amp; C have same affinity scores with B, if A is more influential than B, A's content will be shown first to user B.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerDisplayName="user14204" LastEditorUserId="8878" LastEditDate="2016-01-04T13:21:15.567" LastActivityDate="2016-01-04T13:21:15.567" Title="Alternatives to Graph theory?" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9433" PostTypeId="1" CreationDate="2015-12-18T16:19:00.960" Score="3" ViewCount="1718" Body="&lt;p&gt;I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I would do it in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Vector to sample from&#xA;x &amp;lt;- c(&quot;User1&quot;,&quot;User2&quot;,&quot;User3&quot;,&quot;User4&quot;,&quot;User5&quot;)&#xA;&#xA;# Occurenciens from which to obtain sampling probabilities&#xA;y &amp;lt;- c(2,4,4,3,2)&#xA;&#xA;# Calculate sampling probabilities&#xA;p &amp;lt;- y / sum(y)&#xA;&#xA;# Draw sample with replacement of size 10&#xA;s &amp;lt;- sample(x, 10, replace = TRUE, prom = p)&#xA;&#xA;# Which yields (for example):&#xA;[1] &quot;User5&quot; &quot;User1&quot; &quot;User1&quot; &quot;User5&quot; &quot;User2&quot; &quot;User4&quot; &quot;User4&quot; &quot;User2&quot; &quot;User1&quot; &quot;User3&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I do the same in Scala / Spark?&lt;/p&gt;&#xA;" OwnerUserId="14852" LastActivityDate="2017-08-05T19:03:05.413" Title="Sampling with replacement, specify the probabilities" Tags="|r|apache-spark|scala|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9435" PostTypeId="1" AcceptedAnswerId="9436" CreationDate="2015-12-18T19:31:32.010" Score="16" ViewCount="4795" Body="&lt;p&gt;I am a self-taught web developer and am interested in teaching myself data science, but I'm unsure of how to begin. In particular, I'm wondering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What fields are there within data science? (e.g., Artificial Intelligence, machine learning, data analysis, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Are there online classes people can recommend?&lt;/li&gt;&#xA;&lt;li&gt;Are there projects available out there that I can practice on (e.g., open datasets).&lt;/li&gt;&#xA;&lt;li&gt;Are there certifications I can apply for or complete?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8960" LastEditorUserId="3151" LastEditDate="2015-12-19T09:25:44.767" LastActivityDate="2015-12-19T11:34:53.270" Title="How to self-learn data science?" Tags="|beginner|self-study|" AnswerCount="3" CommentCount="0" ClosedDate="2015-12-19T09:25:37.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="9436" PostTypeId="2" ParentId="9435" CreationDate="2015-12-18T20:50:13.463" Score="16" Body="&lt;p&gt;Welcome to the site, Martin! That's a pretty broad question, so you're probably going to get a variety of answers. Here's my take.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;Data Science&lt;/em&gt; is an interdisciplinary field generally thought to combine classical statistics, machine learning, and computer science (again, this depends on who you ask, but other might include business intelligence here, and possible information visualization or knowledge discovery as well; for example, &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_science&quot;&gt;the wikipedia article on data science&lt;/a&gt;). A good data scientist is also skilled at picking up on the domain-specific characteristics of the domain in which they working, as well. For example, a data scientist working on analytics for hospital records is much more effective if they have a background in Biomedical Informatics.&lt;/li&gt;&#xA;&lt;li&gt;There are many options here, depending on the type of analytics you're interested in. &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot;&gt;Andrew Ng's coursera course is the first resource mentioned by most&lt;/a&gt;, and rightly so. If you're interested in machine learning, that's a great starting place. If you want an in-depth exploration of the mathematics involved, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot;&gt;Tibshirani's The Elements of Statistical Learning&lt;/a&gt; is excellent, but fairly advanced text. There are many online courses available on coursera in addition to Ng's, but you should select them with a mind for the type of analytics you want to focus on, and/or the domain in which you plan on working.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/&quot;&gt;Kaggle&lt;/a&gt;. Start with kaggle, if you want to dive in on some real-world analytics problems. Depending on your level of expertise, it might be good to start of simpler, though. &lt;a href=&quot;https://projecteuler.net/&quot;&gt;Project Euler&lt;/a&gt; is a great resource for one-off practice problems that I still use as warm-up work.&lt;/li&gt;&#xA;&lt;li&gt;Again, this probably depends on the domain you wish to work in. However, I know Coursera offers a data science certificate, if you complete a series of data science-related courses. This is probably a good place to start.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Good luck! If you have any other specific questions, feel free to ask me in the comments, and I'll do my best to help!&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-12-18T20:50:13.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9438" PostTypeId="2" ParentId="9435" CreationDate="2015-12-19T04:30:36.620" Score="1" Body="&lt;p&gt;I recommend starting from Coursera specializations in data science.  The data science specialization by Johns Hopkins is the oldest running specialization. I do not recommend books and kaggle. They only confuse you in the beginning. Keep in mind that coding is the easiest part of data science and you have to learn a lot.&#xA;For getting an idea about the field, &lt;a href=&quot;http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram&quot; rel=&quot;nofollow&quot;&gt;this Venn Diagram&lt;/a&gt; is a good start.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2015-12-19T11:34:53.270" LastActivityDate="2015-12-19T11:34:53.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9439" PostTypeId="2" ParentId="9435" CreationDate="2015-12-19T05:55:50.047" Score="5" Body="&lt;p&gt;I am a self-taught data scientist, and I'd try my best to explain you how to go about it.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What fields are there within data science? (e.g., Artificial&#xA;  Intelligence, machine learning, data analysis, etc.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Data Science is a very wide domain. It is about the science of data. So, any field which uses data to take decisions come under this domain. Some of the fields include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;AI&lt;/li&gt;&#xA;&lt;li&gt;Pattern Recognition and Analytics&lt;/li&gt;&#xA;&lt;li&gt;Bio-statistics&lt;/li&gt;&#xA;&lt;li&gt;Statistical Learning&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning&lt;/li&gt;&#xA;&lt;li&gt;Data Aesthetics(or data visialization)&lt;/li&gt;&#xA;&lt;li&gt;Data Journalism&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there online classes people can recommend?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have answered a &lt;a href=&quot;https://datascience.stackexchange.com/a/8965/11097&quot;&gt;similar question&lt;/a&gt;. So I'd quote it here:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Start with the &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera's Machine Learning&lt;/a&gt; course. It does a&#xA;  really good job in introducing the student to the domain of Machine&#xA;  Learning and helps you lay a solid foundation in the concepts.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In case, you feel that the math is a bit dumbed down in that course,&#xA;  you can take &lt;a href=&quot;https://www.youtube.com/playlist?list=PLA89DCFA6ADACE599&quot; rel=&quot;nofollow noreferrer&quot;&gt;this course&lt;/a&gt;, taught by the same professor and is&#xA;  math-intensive than the former.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Now, you would have a clear intuition about the basic concepts of&#xA;  Machine Learning. Now, take &lt;a href=&quot;https://www.youtube.com/playlist?list=PLD63A284B7615313A&quot; rel=&quot;nofollow noreferrer&quot;&gt;this course&lt;/a&gt;, which can be said as a&#xA;  follow-up or a supplementary for the course of Andrew Ng.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;http://homepages.inf.ed.ac.uk/rbf/IAPR/researchers/MLPAGES/mltut.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;This resource from IAPR&lt;/a&gt; has in-depth notes on a lot of ML&#xA;  concepts like cross-validation, regularization, etc.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;You can also have a look at these &lt;a href=&quot;https://www.quora.com/profile/Eren-Golge/Machine-Learning/Some-of-useful-machine-learning-resources-from-beginner-to-intermediate&quot; rel=&quot;nofollow noreferrer&quot;&gt;amazing list of resources&lt;/a&gt;&#xA;  compiled into a blog on Quora.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Now, for diving into advanced concepts of neural networks and deep&#xA;  learning, you can make use of &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this free book&lt;/a&gt;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Finally, the free e-book: &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elements of Statistical Learning&lt;/a&gt; is a&#xA;  wonderful book for beginners in ML or Statistical Learning.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I addition to that, do check out this &lt;a href=&quot;https://www.quora.com/topic/Data-Science/faq&quot; rel=&quot;nofollow noreferrer&quot;&gt;repository of data science references by Quora&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there projects available out there that I can practice on (e.g.,&#xA;  open datasets).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have started doing projects with open datasets of India. However, I would recommend you to check out this amazing &lt;a href=&quot;https://www.quora.com/Data-Science/What-are-some-good-toy-problems-in-data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;discussion here&lt;/a&gt;, and after doing those projects, you can start with Kaggle.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there certifications I can apply for or complete?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In my opinion, there aren't any &lt;em&gt;data science&lt;/em&gt; certifications. Yeah, there are a lot of Big Data certifications out there, but I didn't see them being really useful for a budding data scientist, so I recommend you not to chase them atleast till you are confident enough with your ML and data skills.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-19T05:55:50.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9440" PostTypeId="1" CreationDate="2015-12-19T07:52:15.327" Score="9" ViewCount="8784" Body="&lt;p&gt;For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?&lt;/p&gt;&#xA;" OwnerUserId="10915" LastActivityDate="2016-12-06T16:43:49.917" Title="How to customise cost function in Scikit learn's model？" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9441" PostTypeId="1" AcceptedAnswerId="9442" CreationDate="2015-12-19T16:57:31.583" Score="1" ViewCount="5720" Body="&lt;p&gt;How can I vectorize this code in R?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data.frame(A = rep(5, 5), B = rep(0, 5))&#xA;data$abstract &amp;lt;- c(&quot;no abstract available&quot;, &quot;A&quot;, &quot;A&quot;, &quot;B&quot;, &quot;no abstract available&quot;)&#xA;&#xA;for (row in (1:nrow(data))){&#xA;  data [row,&quot;abstract&quot;] &amp;lt;- gsub(&quot;no abstract available&quot;,&quot; &quot;,data[row,&quot;abstract&quot;])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="13023" LastEditDate="2015-12-20T03:40:23.893" LastActivityDate="2015-12-20T03:40:23.893" Title="Code Vectorization of gsub in R" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9442" PostTypeId="2" ParentId="9441" CreationDate="2015-12-19T18:42:13.853" Score="3" Body="&lt;p&gt;You have plenty of alternatives for this problem&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Using sapply&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data$abstract &amp;lt;- sapply(data$abstract, &#xA;                        function(x){gsub(pattern = &quot;no abstract available&quot;,&#xA;                        replacement = &quot; &quot;, x)})&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Using mapply&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data$abstract &amp;lt;- mapply(gsub, pattern = &quot;no abstract available&quot;,&#xA;                        replacement = &quot; &quot;, data$abstract)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Using the stringr package&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(stringr)&#xA;data$abstract &amp;lt;- str_replace(data$abstract, &quot;no abstract available&quot;, &#xA;                             &quot; &quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, check this &lt;a href=&quot;https://stackoverflow.com/questions/19424709/r-gsub-pattern-vector-and-replacement-vector&quot;&gt;this question on StackOverflow&lt;/a&gt; for more information, like solutions with match and the qdap package.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-12-19T18:42:13.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9443" PostTypeId="1" AcceptedAnswerId="9447" CreationDate="2015-12-19T19:30:35.527" Score="172" ViewCount="123303" Body="&lt;p&gt;I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the difference between &lt;code&gt;OHE&lt;/code&gt;, &lt;code&gt;LabelEncoder&lt;/code&gt; and &lt;code&gt;DictVectorizor&lt;/code&gt; in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?&lt;/p&gt;&#xA;" OwnerUserId="10462" LastEditorUserId="134" LastEditDate="2018-12-28T00:50:39.957" LastActivityDate="2022-02-20T22:17:56.370" Title="When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?" Tags="|scikit-learn|categorical-data|feature-engineering|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="9444" PostTypeId="1" CreationDate="2015-12-19T19:35:24.460" Score="1" ViewCount="206" Body="&lt;p&gt;&lt;strong&gt;Introduction:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to build a neural network similar to the following one &lt;a href=&quot;http://www.nature.com/articles/srep11476&quot; rel=&quot;nofollow&quot;&gt;http://www.nature.com/articles/srep11476&lt;/a&gt; to predict two different features at the same time:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Secondary structure: discrete label with three classes (Helix, Sheet and Coil)&lt;/li&gt;&#xA;&lt;li&gt;Psi/Phi angles: continuous value (-pi to pi in rad or -180/180 in degrees)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To do so, I use an output layer with softmax activation for the secondary structure and a tanh layer for the angles to predict sin(phi), cos(phi), sin(psi) and cos(psi), which are subsequently converted back to angles phi/psi. Note that this is done to avoid problems with the periodicity of angles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost for the individual features is a cross entropy for the secondary structure and a mean square error for the angles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict both features at the same time by having two different output layers and then train the network with a cost that takes into account both features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: the mean square error cost should be done at the sin/cos level or at the angle level (rad or deg)? sin/cos is constrained from -1/1, the rad from -pi/pi and the deg from -180 to 180&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 2: how can I merge the cross entropy cost for the secondary structure with the mean squared error for the angles in order to make a single cost? I cannot use a simple sum since they have different ranges.&lt;/p&gt;&#xA;" OwnerUserId="14875" LastActivityDate="2015-12-19T19:35:24.460" Title="Neural network with multiple kinds of output" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9446" PostTypeId="2" ParentId="9275" CreationDate="2015-12-20T00:21:27.720" Score="2" Body="&lt;p&gt;$R^{2}$ is a cursory measure of goodness of fit of a linear model to the data and it is used in regression analysis. It appears to be a popular choice when deciding between linear and non-linear regression models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems you intend to use kNN for classification, which has different evaluation metrics than regression. Scikit-learn provides 'accuracy', 'true-positive', 'false-positive', etc (TP,FP,TN,FN), 'precision', 'recall', 'F1 score', etc. for evaluating performance of a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;bonus: kNN are simple classifiers that are used when data shows local structure but not global structure, i.e. data with same label are found in multiple localized clusters in the data space. The key parameter to tune is 'k', which should be based on the size of these localized clusters as you would stand to achieve considerably different results with different values of 'k'. You can do a grid-search (also in scikit-learn) to tune your parameters (including 'k' of course).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;evaluation metric: accuracy or ROC or mean Average Precision or F1-score: There are several metrics to choose from and the choice is based on the task you are performing and what your priorities are. (you can look up these metrics for more details) but you can begin with accuracy for starters.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2015-12-20T00:21:27.720" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9447" PostTypeId="2" ParentId="9443" CreationDate="2015-12-20T00:29:13.517" Score="208" Body="&lt;p&gt;There are some cases where &lt;code&gt;LabelEncoder&lt;/code&gt; or &lt;code&gt;DictVectorizor&lt;/code&gt; are useful, but these are quite limited in my opinion due to ordinality.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;LabelEncoder&lt;/code&gt; can turn &lt;code&gt;[dog,cat,dog,mouse,cat]&lt;/code&gt; into &lt;code&gt;[1,2,1,3,2]&lt;/code&gt;, but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and &lt;code&gt;LabelEncoder&lt;/code&gt; can be used to store values using less disk space.&lt;/p&gt;&#xA;&lt;p&gt;One-Hot-Encoding has the advantage that the result is binary rather than ordinal and that everything sits in an orthogonal vector space.  The disadvantage is that for high cardinality, the feature space can really blow up quickly and you start fighting with the curse of dimensionality. In these cases, I typically employ one-hot-encoding followed by PCA for dimensionality reduction.  I find that the judicious combination of one-hot plus PCA can seldom be beat by other encoding schemes.  PCA finds the linear overlap, so will naturally tend to group similar features into the same feature.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="29169" LastEditDate="2021-06-19T22:53:50.397" LastActivityDate="2021-06-19T22:53:50.397" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="9448" PostTypeId="2" ParentId="22" CreationDate="2015-12-20T01:09:49.760" Score="28" Body="&lt;p&gt;(In addition to the excellent answer by Tim Goodman)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of k-modes is definitely the way to go for stability of the clustering algorithm used.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The clustering algorithm is free to choose any distance metric / similarity score. Euclidean is the most popular. But any other metric can be used that scales according to the data distribution in each dimension /attribute, for example the Mahalanobis metric.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/fjv5L.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fjv5L.jpg&quot; alt=&quot;Illustrating the distance of data points from the center based on the distance metric used.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;With regards to mixed (numerical and categorical) clustering a good paper that might help is: &lt;a href=&quot;http://users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/p1127.pdf&quot; rel=&quot;noreferrer&quot;&gt;INCONCO: Interpretable Clustering of Numerical and Categorical Objects&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Beyond k-means: Since plain vanilla k-means has already been ruled out as an appropriate approach to this problem, I'll venture beyond to the idea of thinking of clustering as a model fitting problem. Different measures, like information-theoretic metric: Kullback-Liebler divergence work well when trying to converge a parametric model towards the data distribution.&#xA;(Of course parametric clustering techniques like GMM are slower than Kmeans, so there are drawbacks to consider)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fuzzy k-modes clustering also sounds appealing since fuzzy logic techniques were developed to deal with something like categorical data. See &lt;a href=&quot;http://biosoft.kaist.ac.kr/~dhlee/pubs/papers/2004FuzzyClustgeringCategorical.pdf&quot; rel=&quot;noreferrer&quot;&gt;Fuzzy clustering of categorical data using fuzzy centroids&lt;/a&gt; for more information.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also check out: &lt;a href=&quot;http://theory.stanford.edu/~sudipto/mypapers/categorical.pdf&quot; rel=&quot;noreferrer&quot;&gt;ROCK: A Robust Clustering Algorithm for Categorical Attributes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2515" LastEditorUserId="-1" LastEditDate="2019-11-25T19:53:21.473" LastActivityDate="2019-11-25T19:53:21.473" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9449" PostTypeId="2" ParentId="934" CreationDate="2015-12-20T06:02:38.193" Score="0" Body="&lt;p&gt;In weka, you could assign the weights to your learning instances. Assign the weights in inverse proportion to the class frequency weight and you should be good to go.&#xA;Another way would be to play around with sampling.&lt;/p&gt;&#xA;" OwnerUserId="14763" LastActivityDate="2015-12-20T06:02:38.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9451" PostTypeId="1" CreationDate="2015-12-20T09:54:24.217" Score="1" ViewCount="2810" Body="&lt;p&gt;I have an assignment, i have to collect movie reviews and classify them with poz neg notr tags, and turn them info arff files. I did all of this stuff with regex etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for experiment i have to choose ten classify algorithm. I did some research but couldn't figure out which algortihm should i use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please help?&#xA;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="14882" LastActivityDate="2015-12-21T21:07:12.467" Title="Sentiment Analysis algorithm in Weka" Tags="|classification|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9453" PostTypeId="1" CreationDate="2015-12-20T17:09:48.473" Score="6" ViewCount="3760" Body="&lt;p&gt;When trying to train a SVM on some Kaggle data, I have encountered a situation where the linear kernel fails to give any results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't make sense to me because the RBF kernel works just fine, and my understanding is that a linear kernel is a strictly simpler version that doesn't map the data into higher dimensions.  In fact, &lt;a href=&quot;https://stats.stackexchange.com/questions/73032/linear-kernel-and-non-linear-kernel-for-support-vector-machine&quot;&gt;all&lt;/a&gt; &lt;a href=&quot;https://stackoverflow.com/questions/20566869/where-is-it-best-to-use-svm-with-linear-kernel&quot;&gt;my&lt;/a&gt; &lt;a href=&quot;https://www.quora.com/How-does-one-decide-on-which-kernel-to-choose-for-an-SVM-RBF-vs-linear-vs-poly-kernel&quot; rel=&quot;nofollow noreferrer&quot;&gt;research&lt;/a&gt; suggests that not only is a linear kernel possible, but that in most cases, it should be faster to converge (with the trade-off that the results might not be as accurate).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this has not proved to be the case.  While the RBF kernel was able to produce a result during cross-validation after ~6 minutes, the linear kernel just sat there with no output after ~6 hours.  After this point, I just force quit the training and moved onto testing other SVM parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A breakdown of some facts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Roughly 60,000 examples, around 120 features (Sometimes cutting down to 30 features helps, and linear kernel will produce a result after many hours)&lt;/li&gt;&#xA;&lt;li&gt;I'm using SKLearn GridSearchCV to perform my training&lt;/li&gt;&#xA;&lt;li&gt;I have tested this used &lt;code&gt;SVC(kernel='linear')&lt;/code&gt; as well as &lt;code&gt;LinearSVC()&lt;/code&gt;, both with the same outcome&lt;/li&gt;&#xA;&lt;li&gt;This issue has come up with multiple data sets across different competitions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My current hypothesis is that the training stops possibly because the data is not linearly separable.  I might be missing something else obvious though.  Any guidance is appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="14885" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-05-17T10:29:06.630" Title="Linear kernel in SVM performing much worse than RBF or Poly" Tags="|svm|scikit-learn|cross-validation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9454" PostTypeId="1" CreationDate="2015-12-21T05:06:49.210" Score="4" ViewCount="909" Body="&lt;p&gt;I'm constrained to use a perceptron based method. I have a user-item matrix filled with rating data on scale of 1 to 5 like this, with around 50% of the matrix with no data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r&amp;lt;- matrix(c(2,4, NA,5,NA,3, NA,5,NA,1,NA,3,NA,5,NA,4,4,NA,NA,NA,1,1,2,NA,1,1,1,1,NA,NA,NA,NA,2,3,4,2,NA,NA,NA,NA,3,4,5,1,NA,NA,2,3,NA), nrow=7).&#xA;#one row represents one user, hile one column represents one item&#xA;r&#xA;    [,1] [,2] [,3] [,4] [,5] [,6] [,7]&#xA;[1,]    2    5   NA    1   NA    2    5&#xA;[2,]    4   NA    4    2   NA   NA    1&#xA;[3,]   NA    1    4   NA   NA   NA   NA&#xA;[4,]    5   NA   NA    1   NA   NA   NA&#xA;[5,]   NA    3   NA    1    2   NA    2&#xA;[6,]    3   NA   NA    1    3    3    3&#xA;[7,]   NA    5    1    1    4    4   NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm recoding the NA's to 0's, which makes the perceptron give out only 2 classes (one containing all the 1's and 0's , and the other containing all the 2,3,4 and 5. Which is understandable I guess.) How do I deal with this? I tried mean imputation but the results are not good (20% accuracy)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I can't give exact code due to this being a proprietary code of my company, but any perceptron based method should work for this example)&lt;/p&gt;&#xA;" OwnerUserId="14427" LastEditorUserId="29575" LastEditDate="2017-12-18T01:11:56.517" LastActivityDate="2018-03-21T18:51:28.020" Title="How to deal with a sparse matrix when using a perceptron based recommender system?" Tags="|machine-learning|recommender-system|perceptron|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9455" PostTypeId="2" ParentId="9453" CreationDate="2015-12-21T05:40:40.410" Score="0" Body="&lt;p&gt;One quick note: gridsearch gets &lt;em&gt;very&lt;/em&gt; slow as your hyperparameter space grows. Randomized search often leads to just-as-good hyperparameters, and is much faster:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14715" LastActivityDate="2015-12-21T05:40:40.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9456" PostTypeId="1" CreationDate="2015-12-21T06:50:07.703" Score="1" ViewCount="248" Body="&lt;p&gt;I did a lot of Googling but could not find a paper that presents an algorithm which will produce dense feature vectors for short text input. I would be happy to find feature extraction algorithm which would perform at least as good as sparse word unigram and bigram feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am exploring the idea of using LDA (Latent Dirichlet Allocation) but there are problems with processing short text (2-7 words per document).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task at hand is short text classification. The number of classes for my data ranges from 10 to 20 classes. The classes are fairly well represented and the word unigram and bigram features work well. I would like to compute dense feature vectors for other experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers to papers, preferrably simple to implement, would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastActivityDate="2017-02-14T15:49:46.903" Title="Dense text representations" Tags="|classification|text|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9457" PostTypeId="1" CreationDate="2015-12-21T07:39:44.953" Score="5" ViewCount="285" Body="&lt;p&gt;I am trying to build and train a machine learning data science algorithm that correctly predicts what president won in what county. I have the following information for training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to the field and this is my first time building an artificial neural network and I don't know what to do to start (so forgive me if my question is extremely broad). What I have done so far is read the wikipedia page for artificial neural networks. What I want to do next is implement supervised learning using the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any help getting started. Please let me know if you have used any good tutorials or libraries to help build something similar. I was thinking of using the library Lasagne.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few specific questions I have as I tried to use Lasagne - What is the error I am to calculate when building an algorithm using training data? How many layers do I need and what does each layer signify?&lt;/p&gt;&#xA;" OwnerUserId="14898" LastEditorUserId="14898" LastEditDate="2015-12-21T07:45:09.397" LastActivityDate="2015-12-21T17:13:49.767" Title="How to build a supervised artificial neural network?" Tags="|machine-learning|neural-network|algorithms|supervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9458" PostTypeId="1" AcceptedAnswerId="9479" CreationDate="2015-12-21T10:26:19.633" Score="3" ViewCount="1776" Body="&lt;p&gt;I have been told that nlp possibly holds the key for allowing researchers to infer the affective state of a person when writing. For instance, by using nlp analysis on online note taking you could infer whether a student is stressed. I work in educational cognitive science, so if this were the case it would be a valuable resource for me, however I am struggling to find evidence that this is indeed the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have begun preliminary research and am trying to learn a bit about the technical aspects of nlp - I'm taking an &lt;a href=&quot;https://www.coursera.org/course/nlp&quot; rel=&quot;nofollow noreferrer&quot;&gt;online course by Jurafsky and Manning&lt;/a&gt;, &lt;a href=&quot;https://www.coursera.org/course/nlangp&quot; rel=&quot;nofollow noreferrer&quot;&gt;another by Michael Collins&lt;/a&gt; - and I have been reading about what what can be inferred using nlp, specifically around opinion mining and sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is two part:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Firstly, most resources I have come across say something along the&#xA;lines of nlp can be used for opinion mining and sentiment analysis&#xA;and we will talk about the implications for opinion mining'. Can&#xA;someone point me in the direction of more sentiment analysis oriented&#xA;resources?&lt;/li&gt;&#xA;&lt;li&gt;Secondly, as I understand it, sentiment analysis is deriving the&#xA;&lt;em&gt;stated&lt;/em&gt; sentiment within text, e.g. I was happy with, I loved, was tasty, enjoyed, hated, frustrated, etc. Can nlp, or something else,&#xA;be used to derive &lt;em&gt;unstated&lt;/em&gt; affect? And is this the same thing as&#xA;sentiment analysis?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;[Apologies if this is asked in the wrong exchange. I've been trying to find the best fit, but there were a few candidates and I wasn't sure which would be most appropriate.]&lt;/p&gt;&#xA;" OwnerUserId="14900" LastEditorUserId="31513" LastEditDate="2017-05-10T08:05:47.560" LastActivityDate="2017-05-10T08:05:47.560" Title="nlp - opinion mining vs sentiment analysis" Tags="|nlp|sentiment-analysis|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9459" PostTypeId="1" AcceptedAnswerId="9517" CreationDate="2015-12-21T10:38:45.997" Score="8" ViewCount="7259" Body="&lt;p&gt;I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV.&#xA;I've used the excellent tutorial at &lt;a href=&quot;http://www.pyimagesearch.com/2015/11/09/pedestrian-detection-opencv/&quot; rel=&quot;noreferrer&quot;&gt;pymagesearch&lt;/a&gt;, which explains what the algorithm does and furnishes hints on how to set the parameters of the &lt;em&gt;detectMultiScale&lt;/em&gt; method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# initialize the HOG descriptor&#xA;hog = cv2.HOGDescriptor()&#xA;&#xA;# Set the support vector machine to be pre-trained for people detection&#xA;hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())&#xA;&#xA;# Detect people in the image&#xA;(rects, weights) = hog.detectMultiScale(image,&#xA;                                        winStride=(4, 4),&#xA;                                        padding=(8, 8),&#xA;                                        scale=1.05)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The parameters are chosen according to a fine tuning on both accuracy and performance, following the explanations in the tutorial itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that this method, which seems like the currently best method to recognise humans in a picture according to literature (the original &lt;a href=&quot;http://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; is dated 2005) seems to perform pretty poorly on my images. &#xA;I have images containing clothes, both with a model and without it and I'm trying this approach to recognise those with the model. On a subset of 300 images which I manually scanned to tag them for containing the model or not, the method fails 30% of the times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are some images as examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here it detected a missing human:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aaR8D.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aaR8D.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here it did not get the full human: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/uExXC.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uExXC.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here it did not recognise it at all:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qPjs5.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qPjs5.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the detector works for upright humans. Should they also be full-figure? &#xA;My images encompass half-figure, figures with no head or no feet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before this, I have tried a Haar-feature based cascade classifier to recognise the face in an image and the accuracy on the same set of image has been 90%, so I was trying to improve on this. Also, I'm interested in understanding why things are not working here.&lt;/p&gt;&#xA;" OwnerUserId="982" LastActivityDate="2016-09-06T10:05:06.727" Title="Recognition human in images through HOG descriptor and SVM classifier performs poorly" Tags="|python|computer-vision|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9460" PostTypeId="1" AcceptedAnswerId="9463" CreationDate="2015-12-21T11:18:31.080" Score="15" ViewCount="43774" Body="&lt;p&gt;I have a pandas DataFrame which has the following columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_0&#xA;n_1&#xA;p_0&#xA;p_1&#xA;e_0&#xA;e_1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to transform it to have columns and sub-columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0&#xA;    n&#xA;    p&#xA;    e&#xA;1&#xA;    n&#xA;    p&#xA;    e&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've searched in the documentation, and I'm completely lost on how to implement this. Does anyone have any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="29575" LastEditDate="2018-02-24T23:13:41.547" LastActivityDate="2020-07-31T14:54:46.530" Title="Pandas: how can I create multi-level columns" Tags="|pandas|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9461" PostTypeId="2" ParentId="9458" CreationDate="2015-12-21T12:25:11.593" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Sentiment analysis&lt;/em&gt;, &lt;em&gt;sentiment detection&lt;/em&gt; and &lt;em&gt;opinion mining&lt;/em&gt;&lt;/a&gt; all cover a set of problems, and can generally be considered to be one and the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;sentiment analysis&lt;/em&gt; seems to be more popular in the press and in industry.  In practice, as of 2015, it is mostly about giving a score, to text, between 0.0 and 1.0, for negative to positive sentiment.  (Strictly speaking that is only a subproblem and one of many possible formulations.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But whenever any of the terms is used, you should define or ask for definition of the exact problem.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2015-12-21T12:25:11.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9462" PostTypeId="2" ParentId="9457" CreationDate="2015-12-21T12:43:00.493" Score="1" Body="&lt;p&gt;Note that I am cross-posting this answer because the question is cross-posted, and one of the questions will likely be deleted. If this is a breach of etiquette, please let me know. Cross-post is &lt;a href=&quot;https://stats.stackexchange.com/questions/187723/how-to-build-a-supervised-artificial-neural-network&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question has two parts, at least implicitly.  The second part is, generally speaking, how to train and predict from neural networks (in Python).  The first part is (implicitly) how do we build a model that predicts (correctly) which person won a presidency based on demographics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to the second question is quite easy to state, but harder to do.  It boils down to the rather unhelpful statement:  &lt;strong&gt;learn about neural networks, and use that knowledge&lt;/strong&gt;.  Google for tutorials on neural networks. I would make recommendations, but I don't know your mathematical, statistical, and computer science background, and thus cannot really make suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question, however, I can answer, although likely not in the way you'd like.  Just using the data you've listed, it's highly unlikely you'll get any sort of accuracy out of your neural networks, or any other model you fit.  Why?  For several reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is that unless you can always classify the president into one of two (or more) distinct groups, as you can in the US (Democrat, Republican), your data are changing all the time.  You can't classify anything if it doesn't have a class, and that's difficult in some countries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, even if there are distinct classes, demographic data is likely not a set of good predictors.  Demographic data changes depending on how good the local governance is, as well as the overall governance.  You might, might be able to code that in somehow.  But demographic data also changes due to a lot of other things. When it comes down to voting, people choose to vote based on a bunch of things, and their demographic status is only one factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third reason is more a caution than anything.  Neural networks and all kinds of machine learning/statistical modelling methods can give good predictions with the right kind of data.  Something like predicting a president is relatively difficult, especially in smaller countries.  (Nate Silver did it well in the US, but there was a lot more data there.)  And even if you've got great data and a great model, there's invariably something you miss.  George Box said it right:  &quot;All models are wrong, but some are useful.&quot;  When you mention predicting &lt;strong&gt;correctly&lt;/strong&gt;, all sorts of alarm bells go off in my head.  The problem with trying for &quot;correct&quot; prediction is that there are no guarantees your prediction will be correct, often even in simple cases.  No model is perfect.&lt;/p&gt;&#xA;" OwnerUserId="14903" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2015-12-21T12:43:00.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9463" PostTypeId="2" ParentId="9460" CreationDate="2015-12-21T12:44:47.847" Score="2" Body="&lt;p&gt;Finally, I found a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find the example script below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python3&#xA;import pickle&#xA;import pandas as pd&#xA;import itertools&#xA;import numpy as np&#xA;&#xA;data = pd.DataFrame(np.random.randn(10, 5), columns=('0_n', '1_n', '0_p', '1_p', 'x'))&#xA;&#xA;indices = set()&#xA;groups = set()&#xA;others = set()&#xA;for c in data.columns:&#xA;    if '_' in c:&#xA;        (i, g) = c.split('_')&#xA;        c2 = pd.MultiIndex.from_tuples((i, g),)&#xA;        indices.add(int(i))&#xA;        groups.add(g)&#xA;    else:&#xA;        others.add(c)&#xA;columns = list(itertools.product(groups, indices))&#xA;columns = pd.MultiIndex.from_tuples(columns)&#xA;ret = pd.DataFrame(columns=columns)&#xA;for c in columns:&#xA;    ret[c] = data['%d_%s' % (int(c[1]), c[0])]&#xA;for c in others:&#xA;    ret[c] = data['%s' % c]&#xA;ret.rename(columns={'total': 'total_indices'}, inplace=True)&#xA;&#xA;print(&quot;Before:&quot;)&#xA;print(data)&#xA;print(&quot;&quot;)&#xA;print(&quot;After:&quot;)&#xA;print(ret)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sorry for this...&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2015-12-21T12:44:47.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9464" PostTypeId="2" ParentId="9458" CreationDate="2015-12-21T14:36:31.693" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can someone point me in the direction of more sentiment analysis&#xA;  oriented resources?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are &lt;a href=&quot;https://www.google.co.in/search?q=top%20research%20papers%20on%20sentiment%20analysis&amp;amp;oq=top%20research%20papers%20on%20sentiment%20analysis&amp;amp;aqs=chrome..69i57.12126j0j4&amp;amp;sourceid=chrome&amp;amp;es_sm=91&amp;amp;ie=UTF-8#q=sentiment%20analysis%20site%3Aarxiv.org&quot; rel=&quot;nofollow&quot;&gt;tonnes of exceptionally good papers&lt;/a&gt; on sentiment analysis for your reference. Especially the &lt;a href=&quot;https://www.google.co.in/search?q=top%20research%20papers%20on%20sentiment%20analysis&amp;amp;oq=top%20research%20papers%20on%20sentiment%20analysis&amp;amp;aqs=chrome..69i57.12126j0j4&amp;amp;sourceid=chrome&amp;amp;es_sm=91&amp;amp;ie=UTF-8#q=sentiment%20analysis%20site%3Aarxiv.org&quot; rel=&quot;nofollow&quot;&gt;ones&lt;/a&gt; on social network sites like Twitter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I think the book by Manning and his course are a nice start as references to NLP and opinion mining.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can nlp, or something else, be used to derive unstated affect?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, it can be. This would be my approach:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Consider I already have a hand-classified training data) Using the bag of words approach, the most used words and phrases for a mood (stressful, jovial, etc) can be captured, and the subsequent training data can be ranked accordingly as (60% stressful, 40% jovial).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you can also define thresholds for getting a single mood. Like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if mood[score] &amp;gt; .5:&#xA;    current_mood = mood&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-21T14:36:31.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9465" PostTypeId="2" ParentId="9453" CreationDate="2015-12-21T14:41:34.360" Score="3" Body="&lt;p&gt;This depends very much on your dataset: There is data, which can &lt;em&gt;not&lt;/em&gt; be separated using a linear kernel. It is impossible, and you won't get any reasonable result. Take for example the simple XOR function. The plot of the XOR looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YXSMs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YXSMs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter how hard you try, you will never be able to separate this using a linear SVM, or any linear method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some short MATLAB code which illustrates the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = [0 0; 0 1; 1 0; 1 1];    % training points&#xA;y = [0; 1; 1; 0];            % target: XOR&#xA;linearSvm = fitcsvm(x,y);    % train with linear kernel&#xA;linearSvm.predict(x)&#xA;&#xA;&amp;gt;&amp;gt; ans = &#xA;        0&#xA;        0&#xA;        0&#xA;        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The SVM has no chance of finding a solution. Now if we use a kernel, we create a mapping to a higher-dimensional space. The kernel trick allows us to implicitly do this mapping, by calculating a kernel function, which returns the result of a scalar product in this high-dimensional space, without needing to map the input values to this space. The RBF kernel is special, as it corresponds to an infinite-dimensional space. &#xA;Now if we specify a RBF kernel and run the same example again, then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gaussSvm = fitcsvm(x,y,'KernelFunction','rbf');  % RBF kernel&#xA;gaussSvm.predict(x)&#xA;&#xA;&amp;gt;&amp;gt; ans = &#xA;        0&#xA;        1&#xA;        1&#xA;        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the SVM easily finds the correct result. The same result (in this XOR case) is also found when using a polynomial kernel. There are many more examples, where data simply can't be separated linearly, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/viKHY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/viKHY.png&quot; alt=&quot;circles&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if your data is almost linearly separable, then you can (and probably will) get good results with a linear kernel. Using soft-margin SVMs, which allow for a few misclassifications, often leads to good results, as they can ignore outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the linear kernel gives no result, you will have to accept that your data is not (well) linearly separable. You'll either have to switch to a kernel, or allow for more misclassifications in the soft-margin SVM.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2015-12-21T14:41:34.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9466" PostTypeId="2" ParentId="2464" CreationDate="2015-12-21T15:31:56.623" Score="5" Body="&lt;p&gt;Assuming that LDA produced a list of topics and put a score against each topic for each document, you could represent the document and it's scores as a vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document | Prime | Minister | Controversial | TopicN | ...&#xA;   1       0.041    0.042      0.041          ...&#xA;   2       0.052    0.011      0.042          ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get the scores for each document, you can run the document. as a bag of words, through a trained LDA model. From the gensim documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; lda = LdaModel(corpus, num_topics=100)  # train model&#xA;&amp;gt;&amp;gt;&amp;gt; print(lda[doc_bow]) # get topic probability distribution for a document&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, you could run the k-means on this matrix and it should group documents that are similar together. &#xA;K-means by default is a hard clustering algorithm implying that it classifies each document into one cluster. You could use soft clustering mechanisms that will give you a probability score that a document fits within a cluster - this is called &lt;a href=&quot;https://mahout.apache.org/users/clustering/fuzzy-k-means.html&quot; rel=&quot;noreferrer&quot;&gt;fuzzy k-means&lt;/a&gt;. &#xA; &lt;a href=&quot;https://gist.github.com/mblondel/1451300&quot; rel=&quot;noreferrer&quot;&gt;https://gist.github.com/mblondel/1451300&lt;/a&gt; is a Python gist showing how you can do it with scikit learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps: I cant post more than 2 links&lt;/p&gt;&#xA;" OwnerUserId="14175" LastActivityDate="2015-12-21T15:31:56.623" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9467" PostTypeId="1" AcceptedAnswerId="9477" CreationDate="2015-12-21T16:21:25.913" Score="3" ViewCount="233" Body="&lt;p&gt;What would be the equivalent of geom_density2d in lattice? In essence I'm trying to create this graph with lattice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5hoQj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5hoQj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think contourplot or levelplot is what i want and when trying it, it gives me a blank plot?&lt;/p&gt;&#xA;" OwnerUserId="11116" LastEditorUserId="11097" LastEditDate="2015-12-21T16:22:56.343" LastActivityDate="2015-12-21T23:06:24.260" Title="R: lattice equivalent of density2d in ggplot?" Tags="|r|ggplot2|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9468" PostTypeId="2" ParentId="9457" CreationDate="2015-12-21T17:13:49.767" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Please let me know if you have used any good tutorials or libraries to&#xA;  help build something similar.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would highly recommend the online book of Michael Nielsen: &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks and Deep Learning&lt;/a&gt;. Coming to the libraries suggestion, I found &lt;a href=&quot;http://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; extremely useful and easy when I was learning neural networks. However, Lasagna is also a nice choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can learn about more libraries from &lt;a href=&quot;https://datascience.stackexchange.com/q/694/11097&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the error I am to calculate when building an algorithm using&#xA;  training data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The most commonly used error function is the squared error function, which is also used in the BackPropogation algorithm, which is used for optimizing the error function.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How many layers do I need and what does each layer signify?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Layers add sophistication, and help in finding subtle patterns in the data. However, you have to make sure that the network don't &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;overfit&lt;/a&gt;. So, as long as the number of layers don't overfit the model, you are good to go. For ensuring that, you need to know what is &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross-validation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2015-12-21T17:13:49.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9469" PostTypeId="1" CreationDate="2015-12-21T17:26:36.720" Score="2" ViewCount="306" Body="&lt;p&gt;In a typical neural network you have a certain number of neurons in each layer, all neurons have incoming connections from each neuron in the previous layer (and visa-versa). This I have no problem figuring out how to propagate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However when you have a network that doesn't have the neurons in layers (I'm looking into the NEAT algorithm); where neurons are just connected based on genetic algorithm (random creation - then testing of fitness) there isn't a simple way of figuring how to forward propagate, for example;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 input neuron - connected to neurons 2 and 3&#xA;Neuron 2 is also connected to neuron 3&#xA;Neuron 3 is connected to the output.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now in this case, if I just go through numerically then everything works out just fine. However with this subtle change;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 input neuron - connected to neurons 2 and 3&#xA;Neuron **3** is also connected to neuron **2**&#xA;Neuron **2** is connected to the output.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The going through; 1, 2, then 3 would mean neuron 2 doesn't include the input from neuron 3. I can see that there's a brute force way through this, but is there a better, more efficient way?&lt;/p&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="14586" LastEditDate="2015-12-21T17:32:54.330" LastActivityDate="2016-03-10T09:32:36.593" Title="Forward propagation in complicated neural networks" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9471" PostTypeId="1" CreationDate="2015-12-21T19:40:44.957" Score="1" ViewCount="37" Body="&lt;h1&gt;Setting&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I have data about a board game which has been played multiple times with different players per game (though players reoccur in all kinds of different constellations). During the game one can score points in one of seven categories. The general question is: What is the best strategy? On what category should players concentrate?&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Details about the data and performed transformations&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The original data has the variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;game - the number of the game&lt;/li&gt;&#xA;&lt;li&gt;name - the name of the player&lt;/li&gt;&#xA;&lt;li&gt;military points&lt;/li&gt;&#xA;&lt;li&gt;money points&lt;/li&gt;&#xA;&lt;li&gt;world wonder points&lt;/li&gt;&#xA;&lt;li&gt;... points&lt;/li&gt;&#xA;&lt;li&gt;sum of points&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;which I transferred into&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;game&lt;/li&gt;&#xA;&lt;li&gt;name&lt;/li&gt;&#xA;&lt;li&gt;sum of points (a little repetitive, since its the same for a game and name)&lt;/li&gt;&#xA;&lt;li&gt;place in game (a little repetitive, since its the same for a game and name)&lt;/li&gt;&#xA;&lt;li&gt;type of points (one of the seven categories)&lt;/li&gt;&#xA;&lt;li&gt;number of points&lt;/li&gt;&#xA;&lt;li&gt;percentage this type of points contributed to the total amount of points of this player for this game&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;My Question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Asking the right question is been said to be the most important thing in Data Science and in most situations I managed to find a good concrete (programmable) question, but this one seems tough to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: &lt;strong&gt;How do I ask a specific question to answer the general question?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course I have already asked/answered simple questions regarding the winning players of each game. For example: &quot;What is the type of points that contributes most to the total points for winning players (whose success proved them right)?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'd like to make analyses over the entire field of players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An even more advanced question would consider that player reoccur across games.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I have no idea what tag to use here.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="14909" LastEditDate="2015-12-23T16:39:22.560" LastActivityDate="2015-12-23T16:39:22.560" Title="Asking a specific question from a general one" Tags="|simulation|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9472" PostTypeId="1" CreationDate="2015-12-21T19:53:49.627" Score="2" ViewCount="839" Body="&lt;p&gt;I have a data set in which users have input free text (think &quot;APPLE&quot;, &quot;Apple&quot;, &quot;apple&quot;, &quot;appl&quot;) and I'm trying to group the input by response type (e.g. organize all of the preceding examples by &quot;Apple&quot;). What's the best way to do this in Python? &lt;/p&gt;&#xA;" OwnerUserId="14910" LastEditorUserId="14910" LastEditDate="2015-12-22T00:02:49.283" LastActivityDate="2017-08-09T20:33:02.733" Title="Which text clustering algorithms should I use to group free text fields in Python?" Tags="|machine-learning|python|nlp|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9473" PostTypeId="1" CreationDate="2015-12-21T20:14:42.187" Score="4" ViewCount="2921" Body="&lt;p&gt;I've seen that there are four neural net packages in R:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;neural&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;neuralnet&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;nnet&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;RSNNS&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;H2O&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What are the advantages/disadvantages of those compared with each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I found out &lt;code&gt;NeuralNetTools&lt;/code&gt; only provides additional tools.&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="14909" LastEditDate="2015-12-23T10:27:22.433" LastActivityDate="2016-10-15T14:30:12.527" Title="What are the advantages/disadvantages of neural net packages in R?" Tags="|r|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9474" PostTypeId="2" ParentId="9451" CreationDate="2015-12-21T21:07:12.467" Score="1" Body="&lt;p&gt;There is a good tutorial here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.unal.edu.co/diracad/einternacional/Weka.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.unal.edu.co/diracad/einternacional/Weka.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically you want to use a StringToWordVector filter to convert your string into word vectors (!) and then train a Bayes classifier on that data.&lt;/p&gt;&#xA;" OwnerUserId="14912" LastActivityDate="2015-12-21T21:07:12.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9475" PostTypeId="2" ParentId="8428" CreationDate="2015-12-21T21:42:06.333" Score="2" Body="&lt;p&gt;I'm somewhat new to the topic as well, but I think what you are looking for is an autoencoder.  I have only used it in the h2o deeplearning package, but it seems to work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea behind an auto-encoder is to begin with your inputs, encode down to less nodes (two is nice for visual representation), then decode back to your original outputs (ass accurately as possible).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Zwsmz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Zwsmz.png&quot; alt=&quot;Autoencoder&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the auto-encoder can decode back to an accurate representation of your inputs, then there was sufficient important information withheld in those two nodes.  You can then plot the features that those two nodes held in a two-dimensional plot that would be analogous to a bounded PCA plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also - the auto-encoder is not constrained to linearity like the PCA is.  Below is a quick snippet of some code, hope it helps.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(h2o)&#xA;localH2O = h2o.init(ip = &quot;localhost&quot;, port = 54321, startH2O = TRUE, min_mem_size = &quot;3g&quot;, max_mem_size = &quot;4g&quot;, nthreads = -1)&#xA;dat_h2o &amp;lt;- as.h2o(train.df)&#xA;&#xA;unsupervised &amp;lt;- &#xA;  h2o.deeplearning(x = 24:356,  # column numbers to use&#xA;                   training_frame = dat_h2o, # data in H2O format&#xA;                   autoencoder = TRUE, ## unsupervised autoencoding&#xA;                   activation = &quot;Tanh&quot;, # or 'Tanh' 'Rectifier' 'WithDropout' node activation function, Tanh seems to work best for autoencoding&#xA;                   hidden = c(5,2,5), # three layers of nodes, with 5/2/5 nodes, respectively&#xA;                   epochs = 1) # max. no. of epochs &#xA;&#xA;## layer 1 corresponds to hidden[1], so it will reduce to 5 variables&#xA;## below is roughly similar to using predict(pca.object, newdata)&#xA;training_data &amp;lt;- h2o.deepfeatures(unsupervised, dat_h2o, layer = 1)&#xA;testing_data &amp;lt;- h2o.deepfeatures(unsupervised, test_h2o, layer = 1)&#xA;&#xA;## explore the second layer with 2 nodes, similar to exploring PC1 vs PC2&#xA;train_supervised_features2 = h2o.deepfeatures(unsupervised, dat_h2o, layer=2)&#xA;&#xA;plotdata2 = as.data.frame(train_supervised_features2)&#xA;plotdata2$label = as.character(as.vector(dat_h2o[,364]))&#xA;&#xA;## L2 corresponds to layer 2, so use L2&#xA;qplot(DF.L2.C1, DF.L2.C2, data = plotdata2, color = label, main = 'Neural network: 5-2-5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2015-12-21T21:42:06.333" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9476" PostTypeId="2" ParentId="8881" CreationDate="2015-12-21T21:46:56.083" Score="2" Body="&lt;p&gt;Generally, this is done using &lt;code&gt;spark-rowsimilarity&lt;/code&gt; algorithm - it is a class of content based recommendation. However, the actual process of doing this is quite simple. Here are the steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For each movie, convert your categorical variables into columns. For lets say that actor_1 has Brad Pitt, Daniel Craig, and Vin Diesel for different movies. This will become three columns with a 1 denoting which movies have each actor. Your movie matrix will look something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Movie Name, Has_Brad_Pitt, Has_Daniel_Craig, Has_Vin_Diesel, ...&#xA;MI-6      ,     1        ,       0         ,     0         , ...&#xA;Fast&amp;amp;Furios,    0        ,       0         ,     1         , ...&#xA;Casino Royale,  0        ,       1         ,     0         , ...&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now, to find similarity score of movies, you can just compute the cross product of the two vectors. Higher the value, more they are similar. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This can be done by the &lt;a href=&quot;https://github.com/apache/mahout/blob/master/spark/src/main/scala/org/apache/mahout/drivers/RowSimilarityDriver.scala&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;spark-rowsimilarity&lt;/code&gt;&lt;/a&gt; algorithm in one shot. You may have to do some work in encoding categorical variables.&lt;/p&gt;&#xA;" OwnerUserId="14175" LastActivityDate="2015-12-21T21:46:56.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9477" PostTypeId="2" ParentId="9467" CreationDate="2015-12-21T23:06:24.260" Score="2" Body="&lt;p&gt;Use a custom panel function in which the density is estimated (e.g. using &lt;code&gt;MASS::kde2d()&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;lattice&quot;)&#xA;library(&quot;MASS&quot;)&#xA;&#xA;set.seed(1);&#xA;dat &amp;lt;- data.frame(x=rnorm(100), y=rt(100, 5))&#xA;&#xA;xyplot(y~x, data=dat,&#xA;       panel=function(x, y, ...) {&#xA;         dens &amp;lt;- kde2d(x=dat$x, y=dat$y, n=50)&#xA;         tmp &amp;lt;- data.frame(x=dens$x,&#xA;                       y=rep(dens$y, each=length(dens$x)),&#xA;                       z=as.vector(dens$z))&#xA;         panel.levelplot(tmp$x, tmp$y, tmp$z, contour=TRUE,&#xA;                         subscripts=1:nrow(tmp), region=FALSE, ...)&#xA;         panel.xyplot(x, y, ...)&#xA;       })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/d2rSi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/d2rSi.png&quot; alt=&quot;lattice plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2961" LastActivityDate="2015-12-21T23:06:24.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9478" PostTypeId="2" ParentId="8428" CreationDate="2015-12-22T01:17:41.407" Score="0" Body="&lt;p&gt;First of all, I would think that plot is useful. The classes do seen fairly separable in that space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A first pass option for visualization should be PCA, though as pointed out in other answers, it will consider only linear combinations of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to visualize the separability of the classes in the context of how well your classifier is doing, the plot you made is a good option. It shows that while there is some confusion (for green in particular), you could fairly easily group each class on that plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead you want to visualize the natural separability of the classes, PCA or a low-dimensional autoencoder is the way to go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is that if what you care about is the former, it's a good thing that your representation is optimized to separate the classes. If you care about the latter, the unsupervised nature of autoencoders is more appropriate.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-22T01:17:41.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9479" PostTypeId="2" ParentId="9458" CreationDate="2015-12-22T01:42:09.480" Score="2" Body="&lt;p&gt;I think the key is that most &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;Recurrent Neural Networks&lt;/a&gt; problems are formulated in terms of either a regression (with low values indicating negative sentiment, and high values positive) or a binary classification (is this text positive?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you seem to be interested in is a much more nuanced definition of sentiment. This doesn't present any inherent problem, as the same algorithms might well work to predict more complex sentiments. The issue is simply labeled data. Because this kind of classification is difficult even for humans, it isn't easy to reliably gather data on, say, how stressed a writer is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if you're interested in assembling a dataset of that nature, you'd be able to apply the same methods (&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;Recurrent Neural Networks&lt;/a&gt; are a popular option) to do the classification. Many researchers in the field use &lt;a href=&quot;https://www.mturk.com/mturk/welcome&quot; rel=&quot;nofollow&quot;&gt;Amazon Mechanical Turk&lt;/a&gt; or something similar to gather labeled data at a reasonable cost.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2015-12-22T01:49:09.613" LastActivityDate="2015-12-22T01:49:09.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9480" PostTypeId="1" CreationDate="2015-12-22T04:11:21.253" Score="1" ViewCount="43" Body="&lt;p&gt;I have access to a few thousand files binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. What techniques can be used to figure out the format of the files, given this information?&lt;/p&gt;&#xA;" OwnerUserId="13145" LastEditorUserId="13145" LastEditDate="2015-12-22T07:49:36.220" LastActivityDate="2018-03-30T14:41:58.127" Title="Given the files and their parsed content how can I reverse engineer the file format?" Tags="|data-formats|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9481" PostTypeId="1" CreationDate="2015-12-22T09:36:33.997" Score="5" ViewCount="371" Body="&lt;p&gt;I'm trying to analyze &lt;em&gt;human intentions in clicking google ad word keywords&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;https://code.google.com/p/open-advertising-dataset/downloads/list&quot; rel=&quot;nofollow&quot;&gt;this dataset&lt;/a&gt; I have the usual adword details, for example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CTR = Clicks / Impressions&#xA;CPC = Cost / Clicks&#xA;CPA = Cost / Converted Clicks&#xA;ROI = Total Conversion Value / Cost&#xA;CVR = Converted Clicks / Clicks&#xA;Weighted Avg Pos = WAP / Impressions&#xA;and Ad position placement.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, I can imagine a relationship between Keywords and CTR. I can do some psychological analysis, like opinion mining and emotion detection, like a vice. Still, I need to show different types of analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what kind of advanced statistical analysis methods can be applied to this data set? My data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Keyword |Average Position|    Average CPC|    Clicks| CTR|    Cost|   Impressions&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;android app developers    ,1  ,0.5,   21.79,  8.7%,   10.99,  250  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9035" LastEditorUserId="13413" LastEditDate="2015-12-23T11:03:25.890" LastActivityDate="2016-05-30T14:31:37.593" Title="What kind of statistical analyses can I do with my data?" Tags="|data-mining|statistics|csv|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9483" PostTypeId="1" AcceptedAnswerId="9485" CreationDate="2015-12-22T11:34:37.877" Score="12" ViewCount="14286" Body="&lt;p&gt;I am a newbie to XGBoost so pardon my ignorance. Here is the python code : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import xgboost as xgb&#xA;&#xA;df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})&#xA;X_train = df.drop('y',axis=1)&#xA;Y_train = df['y']&#xA;T_train_xgb = xgb.DMatrix(X_train, Y_train)&#xA;&#xA;params = {&quot;objective&quot;: &quot;reg:linear&quot;}&#xA;gbm = xgb.train(dtrain=T_train_xgb,params=params)&#xA;Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))&#xA;print Y_pred&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 24.126194  24.126194]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see the input data is simply a straight line. So the output I expect is &lt;code&gt;[40,50]&lt;/code&gt;.  What am I doing wrong here?&lt;/p&gt;&#xA;" OwnerUserId="14925" LastEditorUserId="816" LastEditDate="2016-11-20T12:04:55.270" LastActivityDate="2016-11-29T10:51:41.853" Title="XGBoost Linear Regression output incorrect" Tags="|python|linear-regression|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9484" PostTypeId="1" CreationDate="2015-12-22T12:56:20.123" Score="1" ViewCount="96" Body="&lt;p&gt;Here is my code to implement the learning of my neural network using the backpropagation learning. The algorithms is stable but I don't learn correctly the output. Do you see anything wrong in my learning process?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//### Parameter ###&#xA;#define Nb_entry 2&#xA;#define coeff_app 0.01 &#xA;#define par_momentum 0.8 &#xA;#define par_nb_test 0.2&#xA;#define para_tolerence 0.05&#xA;#define para_stop_learning 1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are my learning functions&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;void fonction_neuron(double * input, Neuron * neuron_info)&#xA;{&#xA;    int i=0; // loop variable&#xA;    double net=0;&#xA;&#xA;    // Computation of the net value type: net = w[0]*bias + sum(w[i]*imp[i])&#xA;    net=(*neuron_info).weight[0]; //bias = 1&#xA;    for(i=1;i&amp;lt;(*neuron_info).nb_input;i++) net+=(*neuron_info).weight[i]*input[i];&#xA;&#xA;    (*neuron_info).output=net;&#xA;    //print_stat_neuron(neuron_info);&#xA;}&#xA;&#xA;void fonction_network(Neuron N_network[][10], int nb_layer, int *nb_neuron_per_layer,double *input)&#xA;{&#xA;    int i,j; //loop variable&#xA;    double previous_layer_output[10]={0};&#xA;&#xA;    //Propagation of the signal into the neural network&#xA;    for(i=0;i&amp;lt;nb_layer;i++)&#xA;    {&#xA;        if(i!=0) for(j=0;j&amp;lt;nb_neuron_per_layer[i-1];j++) &#xA;        {&#xA;            previous_layer_output[j]=N_network[i-1][j].output;//save previous layer output&#xA;            //printf(&quot;previous_layer_output[%d]=%f\n&quot;,j,previous_layer_output[j]); &#xA;        } &#xA;&#xA;        for(j=0;j&amp;lt;nb_neuron_per_layer[i];j++)&#xA;        {&#xA;            //printf(&quot; i=%d j=%d \n&quot;, i,j);&#xA;            N_network[i][j].old_output=N_network[i][j].output; //save previous value&#xA;            if(i==0)fonction_neuron(input, &amp;amp;(N_network[i][j])); //first layer               &#xA;            else fonction_neuron(previous_layer_output, &amp;amp;(N_network[i][j])); //other layer using the previous layer output&#xA;        }&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14926" LastActivityDate="2015-12-23T11:02:59.960" Title="Neural network with multiple layer: learning function" Tags="|neural-network|unsupervised-learning|c|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9485" PostTypeId="2" ParentId="9483" CreationDate="2015-12-22T12:58:33.743" Score="23" Body="&lt;p&gt;It seems that XGBoost uses &lt;strong&gt;regression trees&lt;/strong&gt; as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not extrapolate the patterns in the training data, so any input above 3 or below 1 will not be predicted correctly in your case. Your model is trained to predict outputs for inputs in the interval &lt;code&gt;[1,3]&lt;/code&gt;, an input higher than 3 will be given the same output as 3, and an input less than 1 will be given the same output as 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, regression trees do not really see your data as a &lt;em&gt;straight line&lt;/em&gt; as they are non-parametric models, which means they can theoretically fit any shape that is more complicated than a straight line. Roughly, a regression tree works by assigning your new input data to some of the training data points it have seen during training, and produce the output based on that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is in contrast to parametric regressors (like &lt;strong&gt;linear regression&lt;/strong&gt;) which actually look for the best parameters of a hyperplane (straight line in your case) to fit your data. Linear regression &lt;strong&gt;does&lt;/strong&gt; see your data as a straight line with a slope and an intercept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can change the base learner of your XGBoost model to a GLM (generalized linear model) by adding &lt;code&gt;&quot;booster&quot;:&quot;gblinear&quot;&lt;/code&gt; to your model &lt;code&gt;params&lt;/code&gt; :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import xgboost as xgb&#xA;&#xA;df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})&#xA;X_train = df.drop('y',axis=1)&#xA;Y_train = df['y']&#xA;T_train_xgb = xgb.DMatrix(X_train, Y_train)&#xA;&#xA;params = {&quot;objective&quot;: &quot;reg:linear&quot;, &quot;booster&quot;:&quot;gblinear&quot;}&#xA;gbm = xgb.train(dtrain=T_train_xgb,params=params)&#xA;Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))&#xA;print Y_pred&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In general, to debug why your XGBoost model is behaving in a particular way, see the model parameters :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gbm.get_dump()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If your base learner is linear model, the get_dump output is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['bias:\n4.49469\nweight:\n7.85942\n']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In your code above, since you tree base learners, the output will be :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=2.85\n\t\t4:leaf=5.85\n\t2:leaf=8.85\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=1.995\n\t\t4:leaf=4.095\n\t2:leaf=6.195\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=1.3965\n\t\t4:leaf=2.8665\n\t2:leaf=4.3365\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.97755\n\t\t4:leaf=2.00655\n\t2:leaf=3.03555\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.684285\n\t\t4:leaf=1.40458\n\t2:leaf=2.12489\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.478999\n\t\t4:leaf=0.983209\n\t2:leaf=1.48742\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.3353\n\t\t4:leaf=0.688247\n\t2:leaf=1.04119\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.23471\n\t\t4:leaf=0.481773\n\t2:leaf=0.728836\n',&#xA; '0:[x&amp;lt;3] yes=1,no=2,missing=1\n\t1:[x&amp;lt;2] yes=3,no=4,missing=3\n\t\t3:leaf=0.164297\n\t\t4:leaf=0.337241\n\t2:leaf=0.510185\n',&#xA; '0:[x&amp;lt;2] yes=1,no=2,missing=1\n\t1:leaf=0.115008\n\t2:[x&amp;lt;3] yes=3,no=4,missing=3\n\t\t3:leaf=0.236069\n\t\t4:leaf=0.357129\n']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Tip : I actually prefer to use xgb.XGBRegressor or xgb.XGBClassifier classes, since they follow the &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;noreferrer&quot;&gt;sci-kit learn&lt;/a&gt; API. And because sci-kit learn has so many machine learning algorithm implementations, using XGB as an additional library does not disturb my workflow only when I use the sci-kit interface of XGBoost.&lt;/p&gt;&#xA;" OwnerUserId="860" LastEditorUserId="26596" LastEditDate="2016-11-29T10:51:41.853" LastActivityDate="2016-11-29T10:51:41.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9488" PostTypeId="1" AcceptedAnswerId="9489" CreationDate="2015-12-22T17:19:37.870" Score="35" ViewCount="45591" Body="&lt;p&gt;Is there a way to add more importance to points which are more recent when analyzing data with xgboost? &lt;/p&gt;&#xA;" OwnerUserId="14895" LastActivityDate="2017-07-18T12:21:52.893" Title="xgboost: give more importance to recent samples" Tags="|xgboost|weighted-data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9489" PostTypeId="2" ParentId="9488" CreationDate="2015-12-22T17:26:15.973" Score="15" Body="&lt;p&gt;You could try building multiple xgboost models, with some of them being limited to more recent data, then weighting those results together.  Another idea would be to make a customized evaluation metric that penalizes recent points more heavily which would give them more importance.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2015-12-22T17:26:15.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9490" PostTypeId="2" ParentId="9484" CreationDate="2015-12-22T18:18:07.993" Score="1" Body="&lt;p&gt;Here are some questions to help with the implementation of the algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- You did the sum, but where/what is the activation function (sigmoid, hyperbolic tangent, Gaussian, etc.)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Is multilayer? How did you forward? Did you use the sum along with activation of each neuron past for the other layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- Have you stored the derivatives of activations to the backpropagation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4- Will you use the traditional backpropagation algorithm or genetic algorithm to adjust the network weights?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5- If you choose the sigmoid and backpropagation, &lt;a href=&quot;http://image.slidesharecdn.com/2-150507064358-lva1-app6891/95/25-backpropagation-14-638.jpg?cb=1430981260&quot; rel=&quot;nofollow&quot;&gt;here is an example algorithm&lt;/a&gt; (this algorithm the derivative of the sigmoid is already calculated $O_{j}$ $(1-O_{j})$.&lt;/p&gt;&#xA;" OwnerUserId="14934" LastEditorUserId="13413" LastEditDate="2015-12-23T11:02:59.960" LastActivityDate="2015-12-23T11:02:59.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9491" PostTypeId="1" CreationDate="2015-12-22T18:26:05.800" Score="2" ViewCount="63" Body="&lt;p&gt;Has anyone defined a spline function (i.e. defineFuction) in PMML?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are quite a few parameters that need to be defined, with a fairly lengthy math. For example, for a predictor with a simple 3-knot restrictive cubic spline, I have seven parameters (including two coefficients, three knots, one reference point and one predictor name). The math includes a bit of exponential, max statements, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For that matter, is there a library of functions other than those described on the PMML website documentation? It seems like it would be helpful to have a library of descriptions for even relatively simple functions like polynomials.&lt;/p&gt;&#xA;" OwnerUserId="14894" LastActivityDate="2015-12-22T18:26:05.800" Title="Has anyone defined a spline function (i.e. defineFuction) in PMML?" Tags="|predictive-modeling|algorithms|regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9492" PostTypeId="2" ParentId="6576" CreationDate="2015-12-22T23:22:26.827" Score="1" Body="&lt;p&gt;Data Science is so broad, there's many different paths to get into it. It is usually split into 4 or 5 different types for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/P3ePHl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/P3ePHl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could see from the other posts in this topic people coming from an Applied Statistics background (applying the right algorithm), Programming background (participating in Kaggle), and others applying it to a business background&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Savvy companies could refer to a programming skewed person as a &quot;Data Engineer&quot;. Big companies also use each type for their data science team, so demonstrating good T-shaped skills would be a good thing.&lt;/p&gt;&#xA;" OwnerUserId="14940" LastEditorUserId="14940" LastEditDate="2015-12-23T17:30:34.410" LastActivityDate="2015-12-23T17:30:34.410" CommentCount="0" CommunityOwnedDate="2018-07-31T02:01:34.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="9493" PostTypeId="2" ParentId="9488" CreationDate="2015-12-23T00:38:44.573" Score="43" Body="&lt;p&gt;Just add weights based on your time labels to your xgb.DMatrix.&#xA;The following example is written in R but the same principle applies to xgboost on Python or Julia.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data.frame(feature = rep(5, 5),&#xA;                   year = seq(2011, 2015), &#xA;                   target = c(1, 0, 1, 0, 0))&#xA;weightsData &amp;lt;- 1 + (data$year - max(data$year)) * 5 * 0.01&#xA;&#xA;#Now create the xgboost matrix with your data and weights&#xA;xgbMatrix &amp;lt;- xgb.DMatrix(as.matrix(data$feature), &#xA;                         label = data$target, &#xA;                         weight = weightsData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2015-12-23T00:38:44.573" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9495" PostTypeId="1" AcceptedAnswerId="9873" CreationDate="2015-12-23T06:48:19.680" Score="1" ViewCount="1495" Body="&lt;p&gt;I've heard that a multilayer perceptron can approximate any function arbitrarily exact, given enough neurons. I wanted to try it, so I wrote the following code:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;&quot;&quot;&quot;Example for learning a regression.&quot;&quot;&quot;&#xA;&#xA;&#xA;import tensorflow as tf&#xA;import numpy&#xA;&#xA;&#xA;def plot(xs, ys_truth, ys_pred):&#xA;    &quot;&quot;&quot;&#xA;    Plot the true values and the predicted values.&#xA;&#xA;    Parameters&#xA;    ----------&#xA;    xs : list&#xA;        Numeric values&#xA;    ys_truth : list&#xA;        Numeric values, same length as `xs`&#xA;    ys_pred : list&#xA;        Numeric values, same length as `xs`&#xA;    &quot;&quot;&quot;&#xA;    import matplotlib.pyplot as plt&#xA;    truth_plot, = plt.plot(xs, ys_truth, '-o', color='#00ff00')&#xA;    pred_plot, = plt.plot(xs, ys_pred, '-o', color='#ff0000')&#xA;    plt.legend([truth_plot, pred_plot],&#xA;               ['Truth', 'Prediction'],&#xA;               loc='upper center')&#xA;    plt.savefig('plot.png')&#xA;&#xA;&#xA;# Parameters&#xA;learning_rate = 0.1&#xA;momentum = 0.6&#xA;training_epochs = 1000&#xA;display_step = 100&#xA;&#xA;# Generate training data&#xA;train_X = []&#xA;train_Y = []&#xA;&#xA;# First simple test: a linear function&#xA;f = lambda x: x+4&#xA;&#xA;# Second, more complicated test: x^2&#xA;# f = lambda x: x**2&#xA;&#xA;for x in range(-20, 20):&#xA;    train_X.append(float(x))&#xA;    train_Y.append(f(x))&#xA;train_X = numpy.asarray(train_X)&#xA;train_Y = numpy.asarray(train_Y)&#xA;n_samples = train_X.shape[0]&#xA;&#xA;# Graph input&#xA;X = tf.placeholder(tf.float32)&#xA;reshaped_X = tf.reshape(X, [-1, 1])&#xA;Y = tf.placeholder(&quot;float&quot;)&#xA;&#xA;# Create Model&#xA;W1 = tf.Variable(tf.truncated_normal([1, 100], stddev=0.1), name=&quot;weight&quot;)&#xA;b1 = tf.Variable(tf.constant(0.1, shape=[1, 100]), name=&quot;bias&quot;)&#xA;mul = tf.matmul(reshaped_X, W1)&#xA;h1 = tf.nn.sigmoid(mul) + b1&#xA;W2 = tf.Variable(tf.truncated_normal([100, 100], stddev=0.1), name=&quot;weight&quot;)&#xA;b2 = tf.Variable(tf.constant(0.1, shape=[100]), name=&quot;bias&quot;)&#xA;h2 = tf.nn.sigmoid(tf.matmul(h1, W2)) + b2&#xA;W3 = tf.Variable(tf.truncated_normal([100, 1], stddev=0.1), name=&quot;weight&quot;)&#xA;b3 = tf.Variable(tf.constant(0.1, shape=[1]), name=&quot;bias&quot;)&#xA;# identity as activation to get arbitrary output&#xA;activation = tf.matmul(h2, W3) + b3&#xA;&#xA;# Minimize the squared errors&#xA;l2_loss = tf.reduce_sum(tf.pow(activation-Y, 2))/(2*n_samples)&#xA;optimizer = tf.train.MomentumOptimizer(learning_rate, momentum).minimize(l2_loss)&#xA;&#xA;# Initializing the variables&#xA;init = tf.initialize_all_variables()&#xA;&#xA;# Launch the graph&#xA;with tf.Session() as sess:&#xA;    sess.run(init)&#xA;&#xA;    # Fit all training data&#xA;    for epoch in range(training_epochs):&#xA;        for (x, y) in zip(train_X, train_Y):&#xA;            sess.run(optimizer, feed_dict={X: x, Y: y})&#xA;&#xA;        # Display logs per epoch step&#xA;        if epoch % display_step == 0:&#xA;            cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})&#xA;            print(&quot;cost=%s\nW1=%s&quot; % (cost, sess.run(W1)))&#xA;&#xA;    print(&quot;Optimization Finished!&quot;)&#xA;    print(&quot;cost=%s W1=%s&quot; %&#xA;          (sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y}),&#xA;           sess.run(W1)))  # &quot;b2=&quot;, sess.run(b2)&#xA;&#xA;    # Get output and plot it&#xA;    ys_pred = []&#xA;    ys_truth = []&#xA;&#xA;    test_X = []&#xA;    for x in range(-40, 40):&#xA;        test_X.append(float(x))&#xA;&#xA;    for x in test_X:&#xA;        ret = sess.run(activation, feed_dict={X: x})&#xA;        ys_pred.append(list(ret)[0][0])&#xA;        ys_truth.append(f(x))&#xA;    plot(train_X.tolist(), ys_truth, ys_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This kind of works for linear functions (at least for the training data, not so much for the testing data outside of the range):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cIRSR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cIRSR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it doesn't work at all for non-linear function $x^2$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/q6JbH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q6JbH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does this neural network not work for such simple function approximation? What do I have to change to make the same network topology work for both functions?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2015-12-23T09:56:36.720" LastActivityDate="2016-01-20T14:40:27.380" Title="How should a neural network for unbound function approximation be structured?" Tags="|machine-learning|neural-network|tensorflow|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9498" PostTypeId="1" CreationDate="2015-12-23T23:11:25.907" Score="1" ViewCount="45" Body="&lt;p&gt;With the amount of geographic diversity that is popping up over time with Chipotle and the outbreak of e-coli, if you were investigating the source of all these issues, and where it could occur again in the future, where would you begin? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I initially thought it would be very simple to narrow it down, the most recent CDC articles indicate they have suspicions but nothing for sure yet to indicate how to track it down. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am relatively new to machine learning algorithms, but I would think a feature classification that would take into account the ingredients involved in positive cases, the sources of the ingredients, and possibly even the machinery used or methods of cooking to help classify the inputs that relate to a positive or negative output would be a good start. &lt;/p&gt;&#xA;" OwnerUserId="14972" LastActivityDate="2015-12-23T23:11:25.907" Title="You are in charge of investing Chipotle's E-Coli source/s, what methods do you use?" Tags="|machine-learning|feature-selection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9499" PostTypeId="1" AcceptedAnswerId="9503" CreationDate="2015-12-24T00:13:09.110" Score="1" ViewCount="1263" Body="&lt;p&gt;I am confused about the difference between a binary and multiclass neural network classification. If I am writing an algorithm that has 2 output classes (Obama or Romney), but not yes or no (so not like Obama or not Obama), then is it a binary neural network or a multi class (2 class) neural network classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I do know:&#xA;A binary neural network classification outputs 1 unit. If a multi class classification neural network has k classes, then it has k outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I think:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is a binary neural network classification because I am really only trying to output one thing, whether a county votes for Romney or Obama. I am confused because I thought binary was more like Romney or not Romney classification and I am not sure if that is the same as Romney or Obama. Just wanted to double check and clarify my understanding.&lt;/p&gt;&#xA;" OwnerUserId="14898" LastActivityDate="2017-02-24T14:46:28.553" Title="Binary Neural Network Classification or Multiclass Neural Network Classification?" Tags="|neural-network|multiclass-classification|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9501" PostTypeId="1" CreationDate="2015-12-24T00:35:23.427" Score="1" ViewCount="117" Body="&lt;p&gt;Say I want to do a &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;many-to-one&lt;/a&gt; classification task such as sentiment analysis with a sequence of ~100 elements using a &lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;nofollow&quot;&gt;LSTM&lt;/a&gt;. Might it help to train the network using a many-to-many setup instead to allow the gradients to flow more easily to the initial elements in the long sequence? Each $y_t$ would just be a replica of the true value $y$.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastActivityDate="2015-12-24T00:35:23.427" Title="Include label for each time step to allow gradient flow in LSTM" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9502" PostTypeId="1" AcceptedAnswerId="9504" CreationDate="2015-12-24T00:54:52.500" Score="9" ViewCount="757" Body="&lt;p&gt;I am trying to build and train a multilayer perceptron neural network that correctly predicts what president won in what county for the first time. I have the following information for training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's 14 columns of training data and the 15th column is what the output should be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use Keras to build a multilayer perceptron neural network, but I need some help understanding a few properties and the pros of cons of choosing different options for these properties.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ACTIVATION FUNCTION&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I know my first step is to come up with an activation function. I always studied neural networks used sigmoid activation functions. Is a sigmoid activation function the best? How do you know which one to use? Keras additionally gives the options of using a softmax, softplus, relu, tanh, linear, or hard_sigmoid activation function. I'm okay with using whatever, but I just want to be able to understand why and the pros and cons.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;PROBABILITY INITIALIZAIONS&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I know initializations define the probability distribution used to set the initial random weights of Keras layers. The options Keras gives are uniform&#xA;lecun_uniform, normal, identity, orthogonal, zero, glorot_normal, glorot_uniform, he_normal, and he_uniform. How does my selection here impact my end result or model? Shouldn't it not matter because we are &quot;training&quot; whatever random model we start with and come up with a more optimal weighting of the layers anyways?&lt;/p&gt;&#xA;" OwnerUserId="14898" LastActivityDate="2016-01-11T04:05:25.160" Title="Properties for building a Multilayer Perceptron Neural Network using Keras?" Tags="|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9503" PostTypeId="2" ParentId="9499" CreationDate="2015-12-24T09:50:39.853" Score="4" Body="&lt;p&gt;I think you are making things more confusing then they are. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case you have two possible outputs: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Obama = 1. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Not-Obama (who in this case can only be Romney) = 0. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multi-Class&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case you have &lt;em&gt;k&lt;/em&gt; possible outputs, for example when &lt;em&gt;k = 4&lt;/em&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;k = 0&lt;/em&gt;: Obama&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;k = 1&lt;/em&gt;: Romney&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;k = 2&lt;/em&gt;: Clinton&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;k = 3&lt;/em&gt;: Bush&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are approaches to tackle multi-class classification as binary classification which are called &lt;em&gt;One-vs-rest classification&lt;/em&gt; and &lt;em&gt;One-vs-one classification&lt;/em&gt;, other classifiers, such as Random Forests, are able to deal with a multiclass setting in a natural way. For a brief report, see &lt;a href=&quot;https://www.cs.utah.edu/~piyush/teaching/aly05multiclass.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="14372" LastEditDate="2017-02-24T14:46:28.553" LastActivityDate="2017-02-24T14:46:28.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9504" PostTypeId="2" ParentId="9502" CreationDate="2015-12-24T15:18:53.523" Score="6" Body="&lt;p&gt;1) Activation is an architecture choice, which boils down to a hyperparameter choice. You can make a theoretical argument for using any function, but the best way to determine this is to try several and evaluate on a validation set. It's also important to remember you can mix and match activations of various layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) In theory yes, many random initializations would be the same if your data was extremely well behaved and your network ideal. But in practice initializations seek to ensure the gradient starts off reasonable and the signal can be backpropagated correctly. Likely in this case any of those initializations would perform similarly, but the best approach is to try them out, switching if you get undesirable results.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastEditorUserId="12473" LastEditDate="2016-01-11T04:05:25.160" LastActivityDate="2016-01-11T04:05:25.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9505" PostTypeId="1" AcceptedAnswerId="12141" CreationDate="2015-12-24T17:21:26.850" Score="6" ViewCount="1568" Body="&lt;p&gt;I am working on Azure ML Studio and try to create a regression model to predict a numerical value. I will try to describe my features and what I have done until now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data with about 3 million rows :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;8 integer features from 1 to 25&lt;/li&gt;&#xA;&lt;li&gt;2 boolean features with 0 and 1&lt;/li&gt;&#xA;&lt;li&gt;3 integer features from 1 to 10&lt;/li&gt;&#xA;&lt;li&gt;2 integer feature from 0 to 500.000 (and 1.000.000 respectively) with about 4.500 unique values&lt;/li&gt;&#xA;&lt;li&gt;1 integer feature from 20 to 50&lt;/li&gt;&#xA;&lt;li&gt;1 integer feature from 1 to 15&lt;/li&gt;&#xA;&lt;li&gt;1 integer feature from 0 to 100&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Label:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Integer from 10.000 to 100.000.000 with about 5.000 unique values&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I have done:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Split the dataset to 80% (train) and 20% (test). Then I split the training dataset again to 60% (actual train) and 40% (validation).&lt;/li&gt;&#xA;&lt;li&gt;Normalize the features with many unique values (4th bullet in the above list)&lt;/li&gt;&#xA;&lt;li&gt;Train a model of Boosted Decision Tree Regression.&lt;/li&gt;&#xA;&lt;li&gt;Use the Sweep Parameters module to find the best combination&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I tried also Neural Networks, Bayesian Linear Regression, but BDTR gave the best score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to exclude columns and start with only a few (based on what I think it will affect the model) and then add more columns one by one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the least MSE I could achieved was 1.500.000 (plus I had many negative scored values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I was thinking what other techniques I could use to improve the model. &lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2021-08-24T16:30:17.467" Title="Improve a regression model and feature selection" Tags="|predictive-modeling|regression|feature-selection|feature-engineering|azure-ml|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9506" PostTypeId="2" ParentId="172" CreationDate="2015-12-24T20:13:18.980" Score="5" Body="&lt;p&gt;There is the useful &lt;a href=&quot;https://github.com/blaze/dask&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;dask&lt;/code&gt;&lt;/a&gt; library for parallel numpy/pandas jobs&lt;/p&gt;&#xA;" OwnerUserId="14987" LastEditorUserId="98307" LastEditDate="2020-08-02T12:40:19.397" LastActivityDate="2020-08-02T12:40:19.397" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9507" PostTypeId="1" AcceptedAnswerId="9551" CreationDate="2015-12-24T21:09:29.900" Score="8" ViewCount="7478" Body="&lt;p&gt;Is there a machine learning algorithm that maps a single input to an output list of variable length? If so, are there any implementations of the algorithm for public use? If not, what do you recommend as a workaround?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, the input is a single scalar and the output is a list of scalars with variable length. For example, suppose I wanted to output a list of ones given the length of the list as input. Then &amp;lt;input, output&amp;gt; could be &amp;lt;1, &amp;#91;1&amp;#93;&gt;, &amp;lt;2, [1, 1]&gt;, etc. A small tweak would be providing the square root of the length in which case &amp;lt;2, [1, 1, 1, 1]&gt; would be an answer. Note: the input need not be tied directly to the output.&#xA;&lt;hr&gt;&#xA;For a more complex example, suppose I want to learn the &lt;a href=&quot;https://en.wikipedia.org/wiki/Look-and-say_sequence&quot; rel=&quot;noreferrer&quot; title=&quot;Look-and-say sequence&quot;&gt;look-and-say sequence&lt;/a&gt;. Valid &amp;lt;input, output&amp;gt; pairs would be: &amp;lt;1, &amp;#91;1&amp;#93;&gt;, &amp;lt;2, [1, 1]&gt;, &amp;lt;3, [2, 1]&gt;, &amp;lt;4, [1, 2, 1, 1]&gt;, &amp;lt;5, [1, 1, 1, 2, 2, 1]&gt;, etc. My problem is also similar in that I can generate more examples; I am not restricted to a finite set of examples.&lt;/p&gt;&#xA;" OwnerUserId="14989" LastEditorUserId="14989" LastEditDate="2015-12-27T18:11:23.927" LastActivityDate="2018-09-11T13:46:07.517" Title="Machine Learning: Single input to variable number of outputs" Tags="|machine-learning|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9508" PostTypeId="2" ParentId="8871" CreationDate="2015-12-24T21:11:01.720" Score="2" Body="&lt;p&gt;Narrowing down your question is a binary classification task. Neural networks and svm are equally sufficient for the job. However, only decision trees use &quot;scores&quot; and actually show you the reasoning behind the classification while the other two are a black box. What you are after is called uplifting modelling. Also   it is worth checking feature selection and sensitivity due to the nature of the task.&lt;/p&gt;&#xA;" OwnerUserId="14981" LastActivityDate="2015-12-24T21:11:01.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9509" PostTypeId="1" CreationDate="2015-12-25T07:40:52.577" Score="1" ViewCount="34" Body="&lt;p&gt;We have an Oracle database in which the main table &quot;A1&quot; is partitioned by the hour of insertion of the row. A row once inserted, may be updated depending upon data in 6 other tables (B,C,D,E,F,G) in the same database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A partition in table A1 typically gets updated for a few hours after it starts receiving data. After the updates to the partition are finished, the partition gets archived, i.e, it is moved from production to an archive database. The time of creation of a partition and the time of its archival are logged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my question is given the past history of creation and archival times of partitions, how can I predict when a partition created now in A1 will be archived? What kind of model (regression, etc.) is the best to answer this question? How can I set up a model that tracks the dependence of the archival time of a partition on the attributes in other tables (B,C,D,E,F,G)?&lt;/p&gt;&#xA;" OwnerUserId="13145" LastEditorUserId="14994" LastEditDate="2015-12-26T17:47:49.570" LastActivityDate="2017-04-24T13:15:10.417" Title="Predicting when a partition in Oracle database will be archived" Tags="|predictive-modeling|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9510" PostTypeId="1" AcceptedAnswerId="9511" CreationDate="2015-12-25T08:08:22.430" Score="9" ViewCount="3797" Body="&lt;p&gt;Long short-term memory is a recurrent neural network architecture introduced in the paper &lt;a href=&quot;http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf&quot; rel=&quot;noreferrer&quot;&gt;Long short-term memory&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please tell me where the name comes from?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&quot;Memory&quot;, as the network can store information because of the recurrence - but where does the &quot;Long short-term&quot; come from?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-25T09:10:38.390" Title="Where does the name 'LSTM' come from?" Tags="|machine-learning|neural-network|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9511" PostTypeId="2" ParentId="9510" CreationDate="2015-12-25T08:59:31.813" Score="9" Body="&lt;p&gt;In &lt;a href=&quot;http://goo.gl/JSaqb4&quot; rel=&quot;noreferrer&quot;&gt;Sepp Hochreiter's original paper on the LSTM &lt;/a&gt;  where he introduces the algorithm and method to the scientific community, he explains that the long term memory refers to the learned weights and the short term memory refers to the gated cell state values that change with each step through time &lt;em&gt;t&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt; quote from paper &lt;em&gt;&quot;Recurrent networks can in principle use their feedback connections to store representations of recent input events in the form of activations (&quot;short-term memory&quot;, as opposed to &quot;long-term memory embodied by slowly changing weights)&lt;/em&gt;&quot;&lt;/p&gt;&#xA;" OwnerUserId="4853" LastEditorUserId="4853" LastEditDate="2015-12-25T09:10:38.390" LastActivityDate="2015-12-25T09:10:38.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9512" PostTypeId="1" AcceptedAnswerId="9513" CreationDate="2015-12-25T09:04:20.853" Score="2" ViewCount="318" Body="&lt;p&gt;When reading about SVMs (e.g. on the German Wikipedia) there is a sentence like &quot;an svm is a large-margin classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there other large margin classifiers than SVMs?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-25T09:56:21.907" Title="Are there other large margin classifiers than SVMs?" Tags="|machine-learning|classification|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9513" PostTypeId="2" ParentId="9512" CreationDate="2015-12-25T09:56:21.907" Score="3" Body="&lt;p&gt;Yes, one famous example are boosting techniques like &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow&quot;&gt;Adaboost&lt;/a&gt;. It uses small classifiers to create a big one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can find more info about &lt;a href=&quot;https://en.wikipedia.org/wiki/Margin_classifier&quot; rel=&quot;nofollow&quot;&gt;margin classifiers&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T09:56:21.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9514" PostTypeId="2" ParentId="9505" CreationDate="2015-12-25T11:37:47.143" Score="5" Body="&lt;p&gt;I think the next option you have to take is to add more features. You have a huge amount of training examples, which is good, but the number of features is very low. Adding more features is one of the most used methods to improve performance in machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, it would be good to try to understand how your features affect your model. Imagine you have a linear model like &lt;code&gt;y = theta1*feat1 + theta2*feat2 + theta3*feat3&lt;/code&gt;. If theta3 is near to 0, then feat3 is not affecting the model.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T11:37:47.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9515" PostTypeId="2" ParentId="9346" CreationDate="2015-12-25T11:54:24.150" Score="4" Body="&lt;p&gt;That is a good question and many scientists around the world are asking the same. Well, first a game like Age of Empires is not considered to have a really big solution space, there are not so many things you can do. It's the same in games like Mario Bros. The problem of learning in easy games like Atari games was solve by the guys of DeepMind (here the &lt;a href=&quot;https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;), that was acquired by Google. They used an implementation of Reinforcement Learning with Deep Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going back to your question. A really big problem is how to imitate the amount of decisions a human being takes every day. Wake up, have breakfast, take a shower, leave your house... All these action need a really high level of intelligence and many actions to develop. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many people working on this problem, I'm one of them. I don't know the solution but I can tell you in which way I'm looking. I follow the theories of Marvin Minsky, he is one of the fathers of AI. This book, the Emotion Machine, tells a very good view of the problem. He suggested that the way to create a machine that imitates the human behavior is not by constructing an unified compact theory of artificial intelligence. On the contrary, he argues that our brain contains resources that compete&#xA;between each other to satisfy different goals at the same moment. They called this &lt;strong&gt;Ways to Think&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T11:54:24.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9516" PostTypeId="2" ParentId="9481" CreationDate="2015-12-25T12:12:17.340" Score="0" Body="&lt;p&gt;You have to wonder what is what you want to improve in your page. In my opinion, the most important analysis in every page is the conversion, this is people that enter the page versus people that buys something (or interact). Everything in the page should be optimized to increase this value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important thing would be cost versus conversion. To see how the cost affects your conversion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your content can be share it is important also number of visits versus number of shares.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T12:12:17.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9517" PostTypeId="2" ParentId="9459" CreationDate="2015-12-25T12:16:45.900" Score="5" Body="&lt;p&gt;You are using the training set that opencv is giving you which it doesn't correspond to the kind of images you are using. The data you are using comes from &lt;code&gt;getDefaultPeopleDetector&lt;/code&gt; and the kind of images that the default detector uses are pictures of many people, not a female model from a fashion ecommerce. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to distinguish between models and garments you can try to train your own classifier with HOG or other features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another path you can take is to detect if there is a face or not. You could use haar cascades for that.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastEditorUserId="14994" LastEditDate="2016-09-06T10:05:06.727" LastActivityDate="2016-09-06T10:05:06.727" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9519" PostTypeId="1" CreationDate="2015-12-25T15:10:51.277" Score="0" ViewCount="1465" Body="&lt;p&gt;I have cluster consisting of two points and I am working with non-Euclidean distance. I wonder if it is appropriate to talk about center of cluster in this case and how we can interpret it? We can start with simpler case when distance is weighted Euclidean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Here is clarification. Assume we are using Manhattan distance to find centroid of our 2 point cluster. As far as I understand centroid is not unique in this case if we use PAM algorithm. We need to work with whole set of centroids  for one cluster. Hence, clustering might produce random results on each iteration. It looks to me that problem is not well posed. &lt;/p&gt;&#xA;" OwnerUserId="14999" LastEditorUserId="14999" LastEditDate="2016-01-29T15:53:25.987" LastActivityDate="2016-01-29T15:53:25.987" Title="Definition - center of the cluster with non-Euclidean distance" Tags="|clustering|distance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9520" PostTypeId="2" ParentId="8195" CreationDate="2015-12-25T16:21:29.437" Score="1" Body="&lt;p&gt;A good way to measure the difference between two probabilistic distributions is &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;Kullbak-Liebler&lt;/a&gt;. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T16:21:29.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9521" PostTypeId="2" ParentId="8883" CreationDate="2015-12-25T16:40:10.677" Score="2" Body="&lt;p&gt;I think there might be a problem in the way you are stating the problem. You say that you test data doesn't have two fields, but that can not be correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to take all your data and split it into 2 groups, the training set and the test set. In a proportion of 80%-20% or 70%-30%. Then you train your algorithm with the data in the training set, and test the accuracy of the model with the data in the test set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy you get is the probability that your model is correct. Or said in another way, the next time you use your model to predict a sale, the accuracy is the probability that your prediction is real&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-25T16:40:10.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9523" PostTypeId="2" ParentId="9519" CreationDate="2015-12-25T23:35:34.560" Score="4" Body="&lt;p&gt;The easiest solution for a non-Euclidean cluster center is the &lt;em&gt;medoid&lt;/em&gt;, as in the algorithm PAM. This works with arbitrary metrics, but unfortunately for a 2-element cluster the result by definition is random (by metric properties, each point is an equally good medoid).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you step back and rethink what the &lt;em&gt;center&lt;/em&gt; is. It is the point which has the &lt;em&gt;least squared error&lt;/em&gt;. So &quot;all&quot; you need to do is solve some math for your distance function, how to optimize this least-squares estimation. People have been solving such problems since Gauss. For example with the deviation abs(x-mu) the least-squares objective is minimized by the arithmetic mean, an the one dimensional result carries over to the n-dimensional case and thus Euclidean distance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, this &lt;em&gt;needs&lt;/em&gt; to be done for every distance function again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you want to &lt;em&gt;interpolate&lt;/em&gt; a center for an other distance function, solve that optimization problem (efficiently, but try to find an existing solution first, of course).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2015-12-25T23:35:34.560" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9524" PostTypeId="2" ParentId="9519" CreationDate="2015-12-26T00:34:24.270" Score="1" Body="&lt;p&gt;I guess the centre of a cluster of two points is a point that is the same distance from each of them. Is there such a point in your system? If not, maybe you should go to the point in your space with the smallest difference in distance from each of your other points, in other words, C such that |d(A, C) - d(A, B)| is minimised, where A and B are the points in your cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What distance metric are you using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you planning to generalise this to clusters with more points?&lt;/p&gt;&#xA;" OwnerUserId="15002" LastActivityDate="2015-12-26T00:34:24.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9525" PostTypeId="1" CreationDate="2015-12-26T03:47:48.800" Score="14" ViewCount="17687" Body="&lt;p&gt;I was working on word2vec gensim model and found it really interesting.&#xA;I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this possible?&#xA;Can word2vec be tweaked for this?&#xA;Or the training corpus needs to have all the words of which i want to find similarity.&lt;/p&gt;&#xA;" OwnerUserId="12378" LastActivityDate="2021-02-18T20:48:07.957" Title="How word2vec can be used to identify unseen words and relate them to already trained data" Tags="|nlp|deep-learning|word-embeddings|unsupervised-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9526" PostTypeId="2" ParentId="6078" CreationDate="2015-12-26T04:17:24.980" Score="1" Body="&lt;p&gt;Certainly. A part of topic modeling output is typically some vector for each document describing it as a distribution over the topics in your model. You can use these as continuous-valued predictors for a classifier trained with the labels you have. I've done this before, but, depending on the domain characteristics of your problem, it may not be the best approach, but it is certainly a testable one!&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2015-12-26T04:17:24.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9527" PostTypeId="1" CreationDate="2015-12-26T19:31:41.540" Score="1" ViewCount="539" Body="&lt;p&gt;The code below doesn't give any errors but it doesn't iterate after the first observation, so it's useless. I tried using sapply without any improvement and I don't know if it's a smart approach to use it in conjunctions with the other functions...&#xA;Any idea how to improve my code is much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:nrow(train)) {&#xA;  if (is.na(train[i,5])) {&#xA;    if (grepl(&quot;Master&quot;, train$Name, fixed=TRUE) == TRUE) {&#xA;          train$Age[i] = round(mean(train$Age[train$Name == &quot;Master&quot;], na.rm = TRUE), digits = 2)&#xA;        } else if (grepl(&quot;Miss&quot;,train$Name, fixed=TRUE) == TRUE) {&#xA;      train$Age[i] = round(mean(train$Age[train$Name == &quot;Miss&quot;], na.rm = TRUE), digits = 2)&#xA;        } else if (grepl(&quot;Mrs&quot;,train$Name, fixed=TRUE) == TRUE) {&#xA;      train$Age[i] = round(mean(train$Age[train$Name == &quot;Mrs&quot;], na.rm = TRUE), digits = 2)&#xA;        } else if (grepl(&quot;Mr&quot;,train$Name, fixed=TRUE) == TRUE) {&#xA;      train$Age[i] = round(mean(train$Age[train$Name == &quot;Mr&quot;], na.rm = TRUE), digits = 2)&#xA;        } else if (grepl(&quot;Dr&quot;,train$Name, fixed=TRUE) == TRUE) {&#xA;      train$Age[i] = round(mean(train$Age[train$Name == &quot;Dr&quot;], na.rm = TRUE), digits = 2)&#xA;    } &#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15010" LastActivityDate="2015-12-26T19:31:41.540" Title="How to replace the missing values in Age column from Titanic/kaggle project" Tags="|r|feature-engineering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9528" PostTypeId="1" AcceptedAnswerId="9533" CreationDate="2015-12-26T19:42:51.460" Score="8" ViewCount="258" Body="&lt;p&gt;I've understood that SVMs are binary, linear classifiers (without the kernel trick). They have training data $(x_i, y_i)$ where $x_i$ is a vector and $y_i \in \{-1, 1\}$ is the class. As they are binary, linear classifiers the task is to find a hyperplane which separates the data points with the label $-1$ from the data points with the label $+1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume for now, that the data points are linearly separable and we don't need slack variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I've read that the training problem is now the following optimization problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;${\min_{w, b} \frac{1}{2} \|w\|^2}$&lt;/li&gt;&#xA;&lt;li&gt;s.t. $y_i ( \langle w, x_i \rangle + b) \geq 1$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think I got that minizmizing $\|w\|^2$ means maximizing the margin (however, I don't understand why it is the square here. Would anything change if one would try to minimize $\|w\|$?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also understood that $y_i ( \langle w, x_i \rangle + b) \geq 0$ means that the model has to be correct on the training data. However, there is a $1$ and not a $0$. Why?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-07-06T08:11:48.710" Title="Where exactly does $\geq 1$ come from in SVMs optimization problem constraint?" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9529" PostTypeId="1" AcceptedAnswerId="9535" CreationDate="2015-12-26T20:10:00.473" Score="1" ViewCount="4090" Body="&lt;p&gt;I am going to do regression analysis with multiple variables. In my data I have &lt;strong&gt;n = 23&lt;/strong&gt; features and &lt;strong&gt;m = 13000&lt;/strong&gt; training examples. Here is the plot of my training data (area of houses against price):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9lggy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9lggy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 13000 training examples on the plot. As you can see it is relatively noisy data. My question is which regression algorithm is more appropriate and reasonable to use in my case. I mean is it more logical to use simple linear regression or some nonlinear regression algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more clear I provide some examples.&lt;br&gt;&#xA;Here is some unrelated example of linear regression fit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DyTKO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DyTKO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And some unrelated example of nonlinear regression fit:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/uB7pm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uB7pm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now I provide some hypothetic regression lines for my data:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/q8FzC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q8FzC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;AFAIK primitive linear regression for my data will generate very high error cost because it is very noisy and scattered data. On the other hand, there is no apparent nonlinear pattern (for example sinusoidal). What regression algorithm will be more reasonable to use in my case (house prices data) in order to get more or less appropriate houses' price prediction and why this algorithm (linear or nonlinear) is more reasonable? &lt;/p&gt;&#xA;" OwnerUserId="14525" LastActivityDate="2015-12-27T16:31:59.260" Title="How to select regression algorithm for noisy (scattered) data?" Tags="|machine-learning|regression|linear-regression|" AnswerCount="1" CommentCount="3" ClosedDate="2016-08-29T09:31:54.047" ContentLicense="CC BY-SA 3.0" />
  <row Id="9530" PostTypeId="2" ParentId="9509" CreationDate="2015-12-26T21:18:35.373" Score="1" Body="&lt;p&gt;I'd suggest first visualize your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;time to archive a partition&lt;/em&gt; could be defined as a difference between the partition start (first possible insert) and archiving time (last possible update).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot the total distribution of this data for some time interval - check if it follows some distribution. Is the average, median or sd meaningful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make a boxplot for each hour to see inter quartile interval and the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot it as a time series - does it have a trend or daily/weekly pattern?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your simple set up (with one predictor and one response) this will give you impression about the accuracy that you can expect from a model. The choose of the model depends on the result of the visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally would start with the simplest models:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;all partition archived after $N$ hours;&lt;/li&gt;&#xA;&lt;li&gt;independent model for each hour predicting say $.9$ quantile of some&#xA;history interval;&lt;/li&gt;&#xA;&lt;li&gt;more advanced 2.: consider the weekday and/or trend and/or&#xA;ignoring outliers.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Always check the accuracy and stop if expectations are met.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="31513" LastEditDate="2017-04-24T13:15:10.417" LastActivityDate="2017-04-24T13:15:10.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9531" PostTypeId="2" ParentId="9525" CreationDate="2015-12-27T00:34:21.437" Score="3" Body="&lt;p&gt;The training corpus needs to have all the words of which you want to find similarity.&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2015-12-27T00:34:21.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9532" PostTypeId="1" AcceptedAnswerId="15108" CreationDate="2015-12-27T02:19:02.473" Score="5" ViewCount="1078" Body="&lt;p&gt;The cost vs iteration graph while using the ReLU activation function has number of spikes &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Zz7U1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Zz7U1.png&quot; alt=&quot;ReLU activation function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy on MNIST dataset (both train and test) is around 95%, also on using sigmoid as an activation function I get a smooth downward sloping curve so I think the implementation is correct. Are these spikes expected for ReLU? How would you explain this property?&lt;/p&gt;&#xA;" OwnerUserId="12250" LastActivityDate="2016-11-14T16:06:06.620" Title="spikes in the cost vs iteration graph using ReLU activation function" Tags="|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9533" PostTypeId="2" ParentId="9528" CreationDate="2015-12-27T10:16:56.583" Score="11" Body="&lt;p&gt;&lt;strong&gt;First problem:&lt;/strong&gt; Minimizing $\|w\|$ or $\|w\|^2$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is correct that one wants to maximize the margin. This is actually done by maximizing $\frac{2}{\|w\|}$. This would be the &quot;correct&quot; way of doing it, but it is rather inconvenient. Let's first drop the $2$, as it is just a constant. Now if $\frac{1}{\|w\|}$ is maximal, $\|w\|$ will have to be as small as possible. We can thus find the identical solution by &lt;em&gt;minimizing&lt;/em&gt; $\|w\|$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\|w\|$ can be calculated by $\sqrt{w^T w}$. As the square root is a monotonic function, any point $x$ which maximizes $\sqrt{f(x)}$ will also maximize $f(x)$. To find this point $x$ we thus don't have to calculate the square root and can minimize $w^T w = \|w\|^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, as we often have to calculate derivatives, we multiply the whole expression by a factor $\frac{1}{2}$. This is done very often, because if we derive $\frac{d}{dx} x^2 = 2 x$ and thus $\frac{d}{dx} \frac{1}{2} x^2 = x$. &#xA;This is how we end up with the problem: minimize $\frac{1}{2} \|w\|^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;tl;dr&lt;/em&gt;: yes, minimizing $\|w\|$ instead of $\frac{1}{2} \|w\|^2$ would work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second problem:&lt;/strong&gt; $\geq 0$ or $\geq 1$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As already stated in the question, $y_i \left( \langle w,x_i \rangle + b \right) \geq 0$ means that the point has to be on the correct side of the hyperplane. However this isn't enough: we want the point to be at least as far away as the margin (then the point is a support vector), or even further away.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember the definition of the hyperplane,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{H} = \{ x \mid \langle w,x \rangle + b = 0\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This description however is not unique: if we scale $w$ and $b$ by a constant $c$, then we get an equivalent description of this hyperplane. To make sure our optimization algorithm doesn't just scale $w$ and $b$ by constant factors to get a higher margin, we define that the distance of a support vector from the hyperplane is always $1$, i.e. the margin is $\frac{1}{\|w\|}$. A support vector is thus characterized by $y_i \left( \langle w,x_i \rangle + b \right) = 1 $. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As already mentioned earlier, we want all points to be either a support vector, or even further away from the hyperplane. In training, we thus add the constraint $y_i \left( \langle w,x_i \rangle + b \right) \geq 1$, which ensures exactly that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;tl;dr&lt;/em&gt;: Training points don't only need to be correct, they have to be on the margin or further away.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="14906" LastEditDate="2016-07-06T08:11:48.710" LastActivityDate="2016-07-06T08:11:48.710" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9534" PostTypeId="2" ParentId="422" CreationDate="2015-12-27T13:01:57.797" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://networkrepository.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Repository&lt;/a&gt; has tons of social networks, web graphs, bio and brain networks, etc. Best of all, they also have interactive visual analytic tools to compare/explore the various social networks.&lt;/p&gt;&#xA;" OwnerUserId="15022" LastEditorUserId="29169" LastEditDate="2021-02-09T04:27:20.537" LastActivityDate="2021-02-09T04:27:20.537" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9535" PostTypeId="2" ParentId="9529" CreationDate="2015-12-27T16:31:59.260" Score="2" Body="&lt;p&gt;The model I would use is the one that minimizes the accumulated quadratic error. Both models you are using, linear and quadratic, looks good. You can compute which one has the lowest error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use an advanced method you can use &lt;a href=&quot;https://en.m.wikipedia.org/wiki/RANSAC&quot; rel=&quot;nofollow&quot;&gt;RANSAC&lt;/a&gt;. It is an iterative method for regression that assumes that there are outliers and remove them from the optimization. So your model should be more accurate that just using the first approach I told you.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-27T16:31:59.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9536" PostTypeId="1" AcceptedAnswerId="9543" CreationDate="2015-12-27T16:52:01.120" Score="5" ViewCount="612" Body="&lt;p&gt;An 1024*1024 pixel image has around one million pixels. If I would like to connect each pixel to an R,G,B input neuron, then more than 3 million neurons are needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be really hard, to train a neural network, which has millions of inputs. How is it possible, to reduce the number of neurons?&lt;/p&gt;&#xA;" OwnerUserId="15024" LastActivityDate="2015-12-31T10:24:18.860" Title="How is it possible to process an image with a few neurons?" Tags="|machine-learning|neural-network|deep-learning|image-classification|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9537" PostTypeId="2" ParentId="9525" CreationDate="2015-12-27T17:38:30.030" Score="10" Body="&lt;p&gt;Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithms tend to ignore words that are outside their vocabulary. However there are ways to reframe your problem such that there are essentially no Out-Of-Vocabulary words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that words are simply &quot;tokens&quot; in word2vec. They could be ngrams or they could be letters. One way to define your vocabulary is to say that every word that occurs at least X times is in your vocabulary. Then the most common &quot;syllables&quot; (ngrams of letters) are added to your vocabulary. Then you add individual letters to your vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this way you can define any word as either &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A word in your vocabulary&lt;/li&gt;&#xA;&lt;li&gt;A set of syllables in your vocabulary&lt;/li&gt;&#xA;&lt;li&gt;A combined set of letters and syllables in your vocabulary&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2015-12-27T17:38:30.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9538" PostTypeId="1" CreationDate="2015-12-27T23:41:51.893" Score="1" ViewCount="27" Body="&lt;p&gt;I have a county level dataset with a dichotomous variable which represents whether or not any charter schools exist in a county.  I am performing a nested logistic regression. That variable is the response and the explanatory variables are social factors such as education level, language spoken at home, household type, etc.  When I include all counties in the U.S. (Including ten states which have no charter authorization laws and thus every county in those states is represented as a zero in the response variable) my results show significant associations between the response and the explanatory variables.  When I exclude those ten states the significance disappears.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas as to why this might be happening?  &lt;/p&gt;&#xA;" OwnerUserId="15032" LastActivityDate="2015-12-28T07:17:28.600" Title="Significance shifts when cases are excluded" Tags="|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9539" PostTypeId="1" CreationDate="2015-12-28T07:15:56.377" Score="0" ViewCount="762" Body="&lt;p&gt;Any binary one-hot encoding is aware of only values seen in training, so features not encountered during fitting will be silently ignored. For real time, where you have millions of records in a second, and features have very high cardinality, you need to keep your hasher/mapper updated with the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we do an incremental update to the hasher (rather calculating the entire fit() every time we incounter a new feature-value pair)? What is the suggested approach here the tackle this?&lt;/p&gt;&#xA;" OwnerUserId="1093" LastActivityDate="2015-12-28T21:30:18.467" Title="Using sklearn DictVectorizer in real-time systems" Tags="|python|categorical-data|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9540" PostTypeId="2" ParentId="9538" CreationDate="2015-12-28T07:17:28.600" Score="1" Body="&lt;p&gt;Sounds like you came across &lt;a href=&quot;https://en.wikipedia.org/wiki/Simpson%27s_paradox&quot; rel=&quot;nofollow&quot;&gt;Simpson's Paradox&lt;/a&gt;. Roughly speaking you have a result in one group of data but not when you look at different groups or subsets. This often happens when the variables depend on hidden dimensions in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Simpson%27s_paradox&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt; explains it a bit better and in more detail.&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2015-12-28T07:17:28.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9541" PostTypeId="1" CreationDate="2015-12-28T10:15:19.117" Score="4" ViewCount="723" Body="&lt;p&gt;Regard to K-Means++ algorithm, which is an algorithm for choosing the initial values (or &quot;seeds&quot;) for the k-means clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/K-means%2B%2B&quot; rel=&quot;nofollow&quot;&gt;K-Means++ algorithm in Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exact algorithm is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose one center uniformly at random from among the data points.&lt;/li&gt;&#xA;&lt;li&gt;For each data point x, compute D(x), the distance between x and the nearest center that has already been chosen.&lt;/li&gt;&#xA;&lt;li&gt;Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)2.&lt;/li&gt;&#xA;&lt;li&gt;Repeat Steps 2 and 3 until k centers have been chosen.&lt;/li&gt;&#xA;&lt;li&gt;Now that the initial centers have been chosen, proceed using standard k-means clustering.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I dont understand step 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)^2.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is probability proportional  ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I do not misunderstand  . . .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next centroid x I choose must the distance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;D(x) = D(x)^2 / summation of all distances from all data points square &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that right ?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I still wonder about implementation. I try this in java but it does not work , the chance is very low and it make the selection distort. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public static double euclidean(Data a, Data b) {&#xA;&#xA;    double accumValue = 0;&#xA;    double res;&#xA;&#xA;    for (int i = 0; i &amp;lt; 72; i++) {&#xA;&#xA;        res = a.features[i] - b.features[i];&#xA;&#xA;        res = Math.pow(res, 2);&#xA;&#xA;        accumValue += res;&#xA;&#xA;    }&#xA;&#xA;    double finalRes = Math.sqrt(accumValue);&#xA;&#xA;    return finalRes;&#xA;&#xA;}&#xA;&#xA;public static double accumeratedSqrDistanceCal(ArrayList&amp;lt;Data&amp;gt; dataList, ArrayList&amp;lt;Data&amp;gt; centroids) {&#xA;&#xA;    double[] squareDistanceCollection = new double[dataList.size()];&#xA;&#xA;    for (int i = 0; i &amp;lt; dataList.size(); i++) {&#xA;&#xA;        double minDistance = minDistanceFromClosetCentroidsCal(dataList.get(i), centroids);&#xA;&#xA;        squareDistanceCollection[i] = Math.pow(minDistance, 2);&#xA;&#xA;    }&#xA;&#xA;    double accumerateDistance = 0;&#xA;&#xA;    for (int i = 0; i &amp;lt; dataList.size(); i++) {&#xA;&#xA;        accumerateDistance += squareDistanceCollection[i];&#xA;&#xA;    }&#xA;&#xA;    return accumerateDistance;&#xA;&#xA;}&#xA;&#xA;public static double minDistanceFromClosetCentroidsCal(Data d, ArrayList&amp;lt;Data&amp;gt; centroids) {&#xA;&#xA;    double minDistance = 100000;&#xA;&#xA;    for (int i = 0; i &amp;lt; centroids.size(); i++) {&#xA;&#xA;        double distance = euclidean(d, centroids.get(i));&#xA;&#xA;        if (distance &amp;lt; minDistance) {&#xA;&#xA;            minDistance = distance;&#xA;        }&#xA;&#xA;    }&#xA;&#xA;    return minDistance;&#xA;&#xA;}&#xA;public static void main(String[] args) {&#xA;&#xA;for (int i = 0; i &amp;lt; CENTROIDS_SIZE; i++) {&#xA;&#xA;            for (int j = 0; j &amp;lt; dataList.size(); j++) {&#xA;&#xA;                double accumerateDistance = accumeratedSqrDistanceCal(dataList, centroids);&#xA;&#xA;                double rand = Math.random();&#xA;&#xA;                double distance = minDistanceFromClosetCentroidsCal(dataList.get(j), centroids);&#xA;&#xA;                double distanceSquare = Math.pow(distance, 2);&#xA;&#xA;                double chance = distanceSquare / accumerateDistance;&#xA;&#xA;                if (chance &amp;gt; rand) {&#xA;&#xA;                    centroids.add(dataList.get(j));&#xA;                    break;&#xA;&#xA;                }&#xA;&#xA;            }&#xA;        }&#xA;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="21" LastEditDate="2015-12-30T11:16:08.493" LastActivityDate="2016-01-29T11:18:08.007" Title="Details of the k-means++ algorithm that is used to seed k-means" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9542" PostTypeId="2" ParentId="9541" CreationDate="2015-12-28T10:28:41.260" Score="1" Body="&lt;p&gt;You want to spread out the initial clusters, the article argues that this gives (on average) faster convergence and lower error. This means that points with a higher distance to the closest center are likely to be better candidates for initial clusters. So the probability distribution for a new initial clustering point is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbb{P}(C = c) = \frac{D(c)^2}{\sum_{\forall x\in X}D(x)^2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $X$ is the collection of all candidate points and $C$ the chosen point distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means the higher the distance, the higher the chance it will be picked as initial point.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2015-12-28T10:28:41.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9543" PostTypeId="2" ParentId="9536" CreationDate="2015-12-28T11:46:45.243" Score="6" Body="&lt;p&gt;There are several ways to make this big number trainable:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot;&gt;CNNs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Auto-Encoders (see &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/science.pdf&quot;&gt;Reducing the Dimensionality of Data with Neural Networks&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Dimensionality reduction of the input&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scale the image down&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot;&gt;PCA&lt;/a&gt; / &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot;&gt;LDA&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Troll-Answer&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If you really meant &quot;only a few neurons&quot; then you might want to have a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Spiking_neural_network&quot;&gt;Spiking neural networks&lt;/a&gt;. Those are incredibly computationally intensive, need a lot of hand-crafting and still get worse performance than normal neural networks for most tasks ... but you only need very little of them.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2015-12-28T12:00:49.753" LastActivityDate="2015-12-28T12:00:49.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9544" PostTypeId="1" CreationDate="2015-12-28T13:32:36.103" Score="-1" ViewCount="4162" Body="&lt;pre&gt;&lt;code&gt;set.seed(42)&#xA;iris_matrix &amp;lt;- as.matrix(iris[,1:4])&#xA;image(iris_matrix)&#xA;g&amp;lt;-cor(iris_matrix)&#xA;g&#xA;library(igraph)&#xA;independent.vertex.sets(g, min=NULL, max=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error in independent.vertex.sets(g, min = NULL, max = NULL) : &#xA;       Not a graph object&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to find independent sets but problems in converting the correlation  matrix into a graph object.&lt;/p&gt;&#xA;" OwnerUserId="13953" LastActivityDate="2015-12-31T12:14:42.187" Title="How to convert a 2D matrix into a graph object in R" Tags="|r|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9547" PostTypeId="1" AcceptedAnswerId="9550" CreationDate="2015-12-28T19:38:08.663" Score="1" ViewCount="194" Body="&lt;p&gt;I am wondering why my stacked features do not help me to improve against my loss metric. Here's what I'm doing: I am adding new features which are simple the predictions originated from train, predict of other   models to the original train/test features. Every time I have tried this method, it has failed. I am curious  what the issue could be the with this. Can anyone give me some advice?&lt;/p&gt;&#xA;" OwnerUserId="14946" LastEditorUserId="13413" LastEditDate="2015-12-28T21:43:22.860" LastActivityDate="2015-12-29T09:52:09.657" Title="Stacked features not helping" Tags="|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9548" PostTypeId="2" ParentId="9539" CreationDate="2015-12-28T21:30:18.467" Score="1" Body="&lt;p&gt;If you're doing anything at the &lt;em&gt;millions of records per second&lt;/em&gt; level, you're going to need something more high-performance than Python. Spark/PySpark is ideal for this since it interfaces with your existing Python knowledge and supports HDFS/distributed technologies that are made to take on the millions per second sized tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for sklearn: you might be out of luck here. The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html&quot; rel=&quot;nofollow&quot;&gt;fit_transform() or fit()&lt;/a&gt; methods necessarily have to consider every record that you pass to them to keep the hash up to date, so if you have millions of data points coming in you're going to have to do millions of computations to keep moving, whether you do them one at a time or in a batch. The source code for the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/c957249/sklearn/feature_extraction/dict_vectorizer.py#L269&quot; rel=&quot;nofollow&quot;&gt;transform()&lt;/a&gt; method should convince you that doing either is equivalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One caveat is that if you have predictable features, e.g. maybe you're receiving counts of foo's bar's and baz's, like in the first link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you don't need to update the hash every time, all you have to do is transform &lt;code&gt;D&lt;/code&gt; into the vectorized format and add it to your collection. As long as you don't need to fit new features this can be done relatively fast. However, if you're doing something less structured like NLP this won't give you hardly any useful information.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2015-12-28T21:30:18.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9550" PostTypeId="2" ParentId="9547" CreationDate="2015-12-29T09:52:09.657" Score="3" Body="&lt;p&gt;As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features to another model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First such kind of construction used logistic regression as a final ensemble and and class probabilities from each base learner as input features. Now, what I have described is a technical layout, the intuition behind is the following: considering that there are no models which are good over all joint probability space of features, one can combine their results in order to get the best from each one. In other words we can state the we explore the richness of models (seen as function spaces) to get a combined thing. This strategy does not work always but often it works. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you do something wrong. I think is better to use original features only for base learners. Be careful to use scores or probabilities if possible, instead of final classifications from base learners, it gives more space for improvements. Often is better to stack learners from different families, not the same model with different parameters (better to use a gradient boost and random forest than two gradient boosts). All of those advices are not rules which cannot be broken, and even if you take them all there is no guarantee that there will be improvements.  &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-12-29T09:52:09.657" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9551" PostTypeId="2" ParentId="9507" CreationDate="2015-12-29T22:02:40.703" Score="3" Body="&lt;p&gt;I would try to set a &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow&quot;&gt;multilabel classification algorithm&lt;/a&gt; and make the output standard by adding zeros. So if your data is like this: &amp;lt;1, &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow&quot;&gt;1&lt;/a&gt;&gt;, &amp;lt;2, [1, 1]&gt;, &amp;lt;3, [2, 1]&gt;, &amp;lt;4, [1, 2, 1, 1]&gt;, &amp;lt;5, [1, 1, 1, 2, 2, 1]&gt;. The maximum number of output is 6. So you could transform your data into something like: &#xA;&amp;lt;1, [1,0,0,0,0,0]&gt;, &amp;lt;2, [1, 1,0,0,0,0]&gt;, &amp;lt;3, [2, 1,0,0,0,0]&gt;, &amp;lt;4, [1, 2, 1, 1,0,0]&gt;, &amp;lt;5, [1, 1, 1, 2, 2, 1]&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option that occurs to me is to add the limit dynamically. Let say you have your training and test set. You can search for the biggest length and create an algorithm that adds the zeros to both datasets. Then let's say a new data you want to predict has a bigger length, then you'll need to recompute all training and test with for this new prediction. You can even check how extending the limit affects your model.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastEditorUserId="14994" LastEditDate="2015-12-29T22:18:49.837" LastActivityDate="2015-12-29T22:18:49.837" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9552" PostTypeId="2" ParentId="6601" CreationDate="2015-12-29T23:06:19.167" Score="1" Body="&lt;p&gt;I would suggest &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow&quot;&gt;Recurrent Neural Nets&lt;/a&gt;. They are good for time series, however they need a huge dataset to get good performance. Here you can find an &lt;a href=&quot;https://github.com/Element-Research/rnn&quot; rel=&quot;nofollow&quot;&gt;implementation in torch&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2015-12-29T23:06:19.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9553" PostTypeId="1" AcceptedAnswerId="9584" CreationDate="2015-12-30T02:46:00.233" Score="3" ViewCount="138" Body="&lt;p&gt;Suppose my dataset contains some very small documents (about 20 words each). And each of them may have words in at least two languages (combination of malay and english, for instance). Also there are some numbers inside each of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just out of curiosity, while usually customizable, why are some tokenizers choose to ignore tokens that are just numbers by default, or anything that doesn't meet certain length? For example, the &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow&quot;&gt;CountVectorizer&lt;/a&gt; in scikit-learn ignores words that do not have more than 1 alphanumeric. And the &lt;a href=&quot;http://radimrehurek.com/gensim/utils.html#gensim.utils.tokenize&quot; rel=&quot;nofollow&quot;&gt;tokenizer&lt;/a&gt; utility in gensim ignores words with digits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used CountVectorizer in the end and made it to accept words containing digits and words with length 1 as well. This is because I needed exact match, as slight differences in those words of length 1 may point to a different document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently trying topic modeling (&lt;a href=&quot;http://radimrehurek.com/gensim/models/lsimodel.html&quot; rel=&quot;nofollow&quot;&gt;gensim's LSI&lt;/a&gt;) to perform topic analysis, but the main intention of doing that is to be able to reduce the dimension so I can feed it to spotify's &lt;a href=&quot;https://github.com/spotify/annoy&quot; rel=&quot;nofollow&quot;&gt;annoy library&lt;/a&gt; for quick approximate searching (from 58k features, to 500 topics). Also I expect it to reduce the time and memory taken to compute classifier models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is really, if I tokenize words of length 1, would it makes sense to perform topic modeling? Even if I do a brute force search by just comparing cosine similarity (no fancy ANN), would it affect the precision or accuracy (i.e. would it be able to recognize the slight change in words of length 1 in query and retrieve the right document)?&lt;/p&gt;&#xA;" OwnerUserId="13562" LastEditorUserId="13562" LastEditDate="2015-12-30T02:52:31.690" LastActivityDate="2016-01-02T10:15:40.907" Title="Tokenizing words of length 1, what would happen if I do topic modeling?" Tags="|information-retrieval|lsi|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9554" PostTypeId="1" CreationDate="2015-12-30T09:37:40.547" Score="18" ViewCount="22120" Body="&lt;p&gt;Contribution into open source projects is typically a good way to get some practice for newbies, and try a new area for experienced data scientists and analysts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which projects do you contribute? Please provide some intro + link on Github.&lt;/p&gt;&#xA;" OwnerUserId="97" LastEditorUserId="108" LastEditDate="2015-12-30T20:01:36.423" LastActivityDate="2021-03-11T19:15:10.317" Title="Open source data science projects to contribute" Tags="|beginner|open-source|" AnswerCount="5" CommentCount="1" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 3.0" />
  <row Id="9556" PostTypeId="2" ParentId="9554" CreationDate="2015-12-30T14:38:51.157" Score="6" Body="&lt;p&gt;There are plenty of them available. I do not know if I am allowed to do this (please let me know if it is wrong), but I develop one and it has already over 2 years on git hub (it actually started one years before github). &#xA;The project is called rapaio, is on git hub &lt;a href=&quot;https://github.com/padreati/rapaio&quot;&gt;here&lt;/a&gt; and recently I started to write a manual for it (some of my friends asked me about that). The manual can be found &lt;a href=&quot;https://aureliantutuianu.gitbooks.io/rapaio-manual/content/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It fits your needs if you are willing to develop in Java 8, if you like to do yourself any tool and if you like to experiment. There are only two principles which I enforce. The first one is &lt;em&gt;write something only when you need it&lt;/em&gt;. That is because I strongly believe that only when you need a tool you also know what you really want from it in terms of output, performance, information. The second principle is &lt;em&gt;you depend only on jdk, if you need something you will write it&lt;/em&gt;. I can agree that I am old fashioned, but you can tailor any feature for your purpose in this way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am not allowed to do that as an aswer, again, please let me know. Although, since it's an open source initiative, a &lt;em&gt;give something back to the people with no profit&lt;/em&gt; type of project I see not reason why I could not do it. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2015-12-30T14:38:51.157" CommentCount="3" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 3.0" />
  <row Id="9557" PostTypeId="2" ParentId="9554" CreationDate="2015-12-30T15:28:37.507" Score="16" Body="&lt;p&gt;The &lt;a href=&quot;https://github.com/JuliaLang/julia&quot;&gt;Julia project&lt;/a&gt; is one which I actively contribute to, including the advanced computing and XGBoost libraries. So, I can definitely vouch for it's maintenence and the quality of the community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some really good open source data science projects where even the beginners can contribute are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md&quot;&gt;Sklearn&lt;/a&gt;: Always developing at a rapid pace, the sklearn community is always open to new developers and contributors.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://h2o.ai/&quot;&gt;H2O&lt;/a&gt;: H2O is another fast growing data science projects, working on scalable machine learning and Deep Learning solutions. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/datasciencemasters/go&quot;&gt;Go&lt;/a&gt;: Open source data science road map and resources. Not really a technical project, but is very helpful for absolute beginners and aspiring analysts.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/lisa-lab/pylearn2&quot;&gt;Pylearn2&lt;/a&gt;: Another fast growing Machine Learning and Deep Learning project. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot;&gt;Vowpal Wabbit&lt;/a&gt;: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://www.quora.com/What-are-some-interesting-open-source-data-science-projects-to-contribute-on-GitHub&quot;&gt;Quora discussion&lt;/a&gt; on such projects and some more which are not mentioned in this answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a another &lt;a href=&quot;http://www.kdnuggets.com/2015/06/top-20-python-machine-learning-open-source-projects.html&quot;&gt;nice discussion&lt;/a&gt; about open source Data Science and ML projects in Python.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2015-12-30T15:28:37.507" CommentCount="2" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 3.0" />
  <row Id="9558" PostTypeId="1" AcceptedAnswerId="9566" CreationDate="2015-12-30T16:11:20.963" Score="2" ViewCount="797" Body="&lt;p&gt;I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a sequence. But, in an application to text messaging or any text-based conversation, where the language which is most likely used will be more informal or colloquial. Can still a neural networks perform well than n-gram LM? Considering the data to be fed are the text messages (colloquial phrases). If so, please enlighten me, thanks.&lt;/p&gt;&#xA;" OwnerUserId="15079" LastActivityDate="2015-12-31T12:37:01.817" Title="Neural Networks for Predictive typing" Tags="|machine-learning|neural-network|nlp|language-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9560" PostTypeId="1" AcceptedAnswerId="9561" CreationDate="2015-12-30T19:32:09.643" Score="4" ViewCount="177" Body="&lt;p&gt;I'm curious as to whether research been done into random forests that combine unsupervised with supervised learning in a way allowing a single algorithm to find patterns in, and work with, multiple different data sets. I have googled every possible way to find research on this, and have come up empty. Can anyone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="15083" LastEditorUserId="13413" LastEditDate="2015-12-30T23:41:19.363" LastActivityDate="2015-12-31T00:17:17.063" Title="Research in random forest algorithms able to switch data sets" Tags="|random-forest|supervised-learning|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9561" PostTypeId="2" ParentId="9560" CreationDate="2015-12-31T00:17:17.063" Score="4" Body="&lt;p&gt;&lt;strong&gt;Semi-Supervised Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The combination of unsupervised learning and supervised learning is referred to as &lt;strong&gt;&lt;code&gt;semi-supervised learning&lt;/code&gt;&lt;/strong&gt;, which is the concept that I believe you are searching for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.slideshare.net/dav009/label-propagation-semisupervised-learning-with-applications-to-nlp&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;Label propagation&lt;/code&gt;&lt;/a&gt; is often cited when outlining the heuristics of &lt;code&gt;semi-supervised learning&lt;/code&gt;.  The essence is to employ clustering, but to use a tiny set of known cases in order to derive (or propogate) the labels of the clusters.  Hence one is able to use a small set of labeled cases to classify a much larger set of unsupervised data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here are some references:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia has an entry on the &lt;code&gt;semi-supervised learning&lt;/code&gt;&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/label_propagation.html&quot; rel=&quot;noreferrer&quot;&gt;The scikit learn User Guide is often a useful starting point and has a label propogation routine&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;There are, in fact, papers treating &lt;a href=&quot;http://www.icg.tugraz.at/publications/pdf/srf.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;semi-supervised&lt;/code&gt; &lt;code&gt;random forest&lt;/code&gt; models&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/155552/decisionForests_MSR_TR_2011_114.pdf&quot; rel=&quot;noreferrer&quot;&gt;Another one here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2015-12-31T00:17:17.063" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9562" PostTypeId="1" AcceptedAnswerId="9563" CreationDate="2015-12-31T03:53:38.003" Score="3" ViewCount="691" Body="&lt;p&gt;I'm new in this field and I started working with data by using R. Because of that, I find R much easier to approach a data project. However, apparently an employer wants you to know an object-oriented programing language (a language like Python). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it would be smart to think I can use Python just when I need to deal with a complex programming process, like replacing the na's in Titanic/kaggle project with the average based on name and to use R for anything else? So use them both interchangeably?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beside the fact that Python is more programming oriented I don't see why somebody would use it over R...&lt;/p&gt;&#xA;" OwnerUserId="15010" LastEditorUserId="29169" LastEditDate="2019-06-09T15:20:13.980" LastActivityDate="2019-06-09T15:20:13.980" Title="Using R and Python together" Tags="|r|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9563" PostTypeId="2" ParentId="9562" CreationDate="2015-12-31T05:08:46.897" Score="5" Body="&lt;p&gt;Several clarifications: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;you can program with object-oriented (OOP) concepts in R, even though OOP in R has slightly different syntax from other languages. Methods do not bind to objects. In R, different method versions will be involved based on the input argument classes (and types). (Ref: &lt;a href=&quot;http://adv-r.had.co.nz/OO-essentials.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Advanced R&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;you can also replace nans with mean / any stat. / value in R using a mask if you store the data in a dataframe (See &lt;a href=&quot;https://stackoverflow.com/questions/25835643/replacing-missing-values-in-r-with-column-mean&quot;&gt;SO post&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is no problem using them interchangeably. I use R from Python using the package RPy2. I assume it is equally easy to do it the other way round. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At the end of the day, any language is only as good as how much the users know about it. Use one that you are more familiar with and try to learn it properly using the vast online resources online.&lt;/p&gt;&#xA;" OwnerUserId="6396" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2015-12-31T09:30:24.533" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9564" PostTypeId="2" ParentId="9536" CreationDate="2015-12-31T10:24:18.860" Score="2" Body="&lt;p&gt;Convolutional neural nets share weights across the whole input image and this drastically reduces the number of weights. For example you could have a 3x3 grid where each point has its own weight, this scans over each of the 3 channels in the 1024x1024 image. Only 9 weights are used for each channel making 27 weights. If you have, say 10 of these grids then that's only 270 weights! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore by sharing weights you create some translation invariance over the input space which is a desirable property in things like object recognition. &lt;/p&gt;&#xA;" OwnerUserId="14047" LastActivityDate="2015-12-31T10:24:18.860" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9565" PostTypeId="2" ParentId="9544" CreationDate="2015-12-31T12:14:42.187" Score="1" Body="&lt;p&gt;You have to &lt;strong&gt;convert your matrix into a graph&lt;/strong&gt;.&#xA;For example, started from your &quot;g&quot; :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;G &amp;lt;- as.undirected(graph.adjacency(g, weighted = T))&#xA;plot(G, edge.color=c(&quot;red&quot;,&quot;green&quot;)[sign(E(G)$weight)/2 + 1.5], &#xA;     edge.width = 3 *abs(E(G)$weight))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15099" LastActivityDate="2015-12-31T12:14:42.187" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9566" PostTypeId="2" ParentId="9558" CreationDate="2015-12-31T12:37:01.817" Score="4" Body="&lt;p&gt;A neural network is in principle a good choice when you have A LOT of similar data and classification tasks. Predicting the next character (or word... which is just multiple characters) is such a szenario. I don't think it really matters which kind of language you have, as long as you have enough training data of the same kind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;The Unreasonable Effectiveness of Recurrent Neural Networks&lt;/a&gt; for a nice article where a recurrent neural network (RNN) was used as a character predictor to write complete texts. They also have code on &lt;a href=&quot;https://github.com/karpathy/char-rnn&quot; rel=&quot;nofollow&quot;&gt;github.com/karpathy/char-rnn&lt;/a&gt; ready to train / go. You can feed it with a start string and ask for the next characters / words.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2015-12-31T12:37:01.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9567" PostTypeId="2" ParentId="9074" CreationDate="2015-12-31T17:19:42.423" Score="0" Body="&lt;p&gt;Labels come up in conjunction with classification when the object does not belong to a single class but to a broader set, hence the term &quot;multilabel learning&quot; or &quot;multilabel classification&quot;. Since they refer to discrete classes, they can be used synonymously, but I would recommend using the traditional terminology (&lt;em&gt;classification&lt;/em&gt; when a single class is to be assigned) to avoid confusion.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2015-12-31T17:19:42.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9568" PostTypeId="1" CreationDate="2015-12-31T19:18:24.280" Score="6" ViewCount="385" Body="&lt;p&gt;I am planning to set up a JSON storage system. It will store tens of millions of JSON records, all in the same format. I'd like to be able to query the data using Apache Drill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like there is Drill support for MongoDB and Postgres. However, I'm unsure of the pros and cons of each, and how I'd structure the schema if I'd choose Postgres.&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2016-07-21T16:29:46.587" Title="Database options for JSON storage, queried with Apache Drill" Tags="|bigdata|mongodb|json|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9572" PostTypeId="1" AcceptedAnswerId="9575" CreationDate="2016-01-01T00:30:14.717" Score="10" ViewCount="615" Body="&lt;p&gt;I often use &lt;a href=&quot;http://docs.python-guide.org/en/latest/writing/tests/&quot;&gt;Nose, Tox or Unittest&lt;/a&gt; when testing my python code, specially when it has to be integrated with other modules or other pieces of code. However, now that I've found myself using R more than python for ML modelling and development. I realized that I don't really test my R code (And more importantly I really don't know how to do it well). So my question is, what are good packages that allow you to test R code in a similar manner as Nose, Tox or Unittest do in Python. Additional references such as tutorials will be greatly appreciated as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus points for packages in R similar to &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://hypothesis.readthedocs.org/en/latest/&quot;&gt;Hypothesis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://feature-forge.readthedocs.org/en/latest/&quot;&gt;Feature Forge&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Related Talk:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=GEqM9uJi64Q&quot;&gt;Trey Causey: Testing for Data Scientists&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-01-02T22:06:50.740" LastActivityDate="2016-01-07T21:47:52.183" Title="Software Testing for Data Science in R" Tags="|r|software-development|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9573" PostTypeId="1" AcceptedAnswerId="9574" CreationDate="2016-01-01T08:11:03.380" Score="5" ViewCount="8283" Body="&lt;p&gt;In &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; wiki page, I came across with the following phrase.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When data is not labeled, a supervised learning is not possible, and&#xA;  an unsupervised learning is required&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I cannot figure out why supervised learning is not possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate any help to resolve this ambiguity.&lt;/p&gt;&#xA;" OwnerUserId="15115" LastEditorUserId="15091" LastEditDate="2016-12-30T07:52:27.277" LastActivityDate="2018-07-18T03:11:48.957" Title="supervised learning and labels" Tags="|machine-learning|supervised-learning|unsupervised-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9574" PostTypeId="2" ParentId="9573" CreationDate="2016-01-01T08:54:20.330" Score="5" Body="&lt;p&gt;The main difference between supervised and unsupervised learning is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In supervised learning you have a set of labelled data, meaning that you have the values of the inputs and the outputs. What you try to achieve with machine learning is to find the &lt;strong&gt;true&lt;/strong&gt; relationship between them, what we usually call &lt;strong&gt;the model&lt;/strong&gt; in math. There are many different algorithms in machine learning that allow you to obtain a model of the data. The objective that you seek, and how you can use machine learning, is to predict the output given a new input, once you know the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In unsupervised learning you don't have the data labelled. You can say that you have the inputs but not the outputs. And the objective is to find some kind of pattern in your data. You can find groups or &lt;strong&gt;clusters&lt;/strong&gt; that you think that belong to the same group or output. Here you also have to obtain a model. And again, the objective you seek is to be able to predict the output given a new input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, going back to your question, if you don't have labels you can not use supervised learning, you have to use unsupervised learning.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-01-01T08:54:20.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9575" PostTypeId="2" ParentId="9572" CreationDate="2016-01-01T09:09:22.127" Score="7" Body="&lt;p&gt;Packages for unit testing and assertive testing that are actively maintained:&#xA;Packages for unit testing&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;testthat: more information on how to use you can find &lt;a href=&quot;http://r-pkgs.had.co.nz/tests.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; or on &lt;a href=&quot;https://github.com/hadley/testthat&quot; rel=&quot;noreferrer&quot;&gt;github&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Runit: &lt;a href=&quot;https://cran.r-project.org/web/packages/RUnit/index.html&quot; rel=&quot;noreferrer&quot;&gt;Cran page&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Packages for assertions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;assertthat: info on &lt;a href=&quot;https://github.com/hadley/assertthat&quot; rel=&quot;noreferrer&quot;&gt;github&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;assertive: Assertive has a lot of subpackages available in case you do not need all of them. check on cran&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;assertr: info on &lt;a href=&quot;https://github.com/tonyfischetti/assertr&quot; rel=&quot;noreferrer&quot;&gt;github&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ensurer: info on &lt;a href=&quot;https://github.com/smbache/ensurer&quot; rel=&quot;noreferrer&quot;&gt;github&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;tester: info on &lt;a href=&quot;https://github.com/gastonstat/tester&quot; rel=&quot;noreferrer&quot;&gt;github&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It is a matter of preference what you want to use for assertions. &#xA;Read &lt;a href=&quot;http://bioconductor.org/developers/unitTesting-guidelines/&quot; rel=&quot;noreferrer&quot;&gt;this bioconductor&lt;/a&gt; page for more info on the difference between RUnit and testthat.&lt;/p&gt;&#xA;" OwnerUserId="10517" LastEditorUserId="10517" LastEditDate="2016-01-03T08:13:26.770" LastActivityDate="2016-01-03T08:13:26.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9576" PostTypeId="2" ParentId="694" CreationDate="2016-01-01T09:19:31.513" Score="5" Body="&lt;p&gt;Just to add some more resources. Recently there was a paper studying the differences between several packages of neural networks and deep neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can find the &lt;a href=&quot;https://github.com/zer0n/deepframeworks/blob/master/README.md&quot; rel=&quot;noreferrer&quot;&gt;information&lt;/a&gt;. It looks that &lt;a href=&quot;https://github.com/torch/torch7&quot; rel=&quot;noreferrer&quot;&gt;Torch&lt;/a&gt; and &lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt; are the winners. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: not all of them are in python. However, I posted it to open the discussion.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-01-01T09:19:31.513" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="9577" PostTypeId="1" CreationDate="2016-01-01T09:45:59.920" Score="1" ViewCount="2683" Body="&lt;p&gt;I am researching on overlapping clustering (Clusters are non-disjoint).I found that &lt;a href=&quot;http://bigdata.ices.utexas.edu/publication/non-exhaustive-overlapping-k-means/&quot; rel=&quot;nofollow&quot;&gt;Neo-K-Means&lt;/a&gt; is probably the state-of-the-art now.But, when I tried  implementing the algorithm with the multi-label data set (music-emotion/scene).I hadn't got the high result as declared in the paper (My results are around 0.4 F-measure , the paper declare 0.55 for music and 0.626 for scene ). In spite of , I had initialized the experiments with the best situation for K-Means (centroid seeds are means of each class and the total cluster assignments are equal to reality) . I wonder what wrong with my implementation method , orDo I have to do something extra for getting higher result ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. I have found the &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2783398&quot; rel=&quot;nofollow&quot;&gt;further research of Neo-Kmeans&lt;/a&gt; , but I think I should clear this point before go further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;while (count &amp;lt; TIMES) {&#xA;&#xA;            DC = new ArrayList&amp;lt;DistanceCollection&amp;gt;();&#xA;&#xA;            for (int i = 0; i &amp;lt; K; i++) {&#xA;&#xA;                cluster[i] = new Cluster();&#xA;&#xA;            }&#xA;&#xA;            for (int i = 0; i &amp;lt; dataList.size(); i++) {&#xA;&#xA;                for (int j = 0; j &amp;lt; K; j++) {&#xA;&#xA;                    DistanceCollection dist = new DistanceCollection();&#xA;                    dist.dataNum = dataList.get(i).dataNum;&#xA;                    dist.clusterNum = j;&#xA;                    dist.distanceFromCluster = euclidean(centroids[j], dataList.get(i));&#xA;                    DC.add(dist);&#xA;&#xA;                }&#xA;&#xA;            }&#xA;&#xA;            // sort the distances for argmin(i,j) checking &#xA;            Collections.sort(DC, new DistanceCollectionComparator());&#xA;&#xA;&#xA;&#xA;            int totalAssignment = 2585;&#xA;            int assignedCluster = -1;&#xA;            int assignmentCount = 0;&#xA;            int[] dataSelectionCheck = new int[2407];&#xA;            int dataMatrix[][] = new int[6][2407];&#xA;&#xA;            for (int i = 0;  assignmentCount &amp;lt; dataList.size(); i++) {&#xA;&#xA;                int clusterNum = DC.get(i).clusterNum;&#xA;                int dataNum = DC.get(i).dataNum;&#xA;&#xA;                if (dataMatrix[clusterNum][dataNum] == 0 &amp;amp;&amp;amp; dataSelectionCheck[dataNum] == 0) {&#xA;&#xA;                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));&#xA;                    dataMatrix[clusterNum][dataNum] = 1;&#xA;                    dataSelectionCheck[dataNum] = 1;&#xA;&#xA;                    assignmentCount++;&#xA;&#xA;                }&#xA;&#xA;            }&#xA;&#xA;            for (int i = 0; assignmentCount &amp;lt; totalAssignment; i++) {&#xA;&#xA;                int clusterNum = DC.get(i).clusterNum;&#xA;                int dataNum = DC.get(i).dataNum;&#xA;&#xA;                if (dataMatrix[clusterNum][dataNum] == 0) {&#xA;&#xA;                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));&#xA;                    dataMatrix[clusterNum][dataNum] = 1; &#xA;&#xA;                    assignmentCount++;&#xA;&#xA;                }&#xA;&#xA;            }&#xA;&#xA;&#xA;&#xA;            for (int i = 0; i &amp;lt; K; i++) {&#xA;&#xA;                if (cluster[i].dataMembers.size() &amp;gt; 0) {&#xA;&#xA;                    for (int j = 0; j &amp;lt; centroids[i].features.length ; j++) {&#xA;&#xA;                        double accumFeaturesValue = 0;&#xA;&#xA;                        for (int k = 0; k &amp;lt; cluster[i].dataMembers.size(); k++) {&#xA;&#xA;                            accumFeaturesValue += cluster[i].dataMembers.get(k).features[j];&#xA;&#xA;                        }&#xA;&#xA;                        centroids[i].features[j] = accumFeaturesValue / cluster[i].dataMembers.size();&#xA;&#xA;                    }&#xA;                }&#xA;&#xA;            }&#xA;&#xA;&#xA;            count++ ;&#xA;&#xA;        }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="924" LastEditDate="2016-01-01T22:40:24.650" LastActivityDate="2016-01-01T22:40:24.650" Title="The implementation of overlapping clustering (NEO-K-Means)" Tags="|machine-learning|data-mining|clustering|k-means|research|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9578" PostTypeId="2" ParentId="9577" CreationDate="2016-01-01T11:39:24.200" Score="2" Body="&lt;p&gt;I have never heard of NEO-k-means. So I wouldn't call it &quot;state of the art&quot;: which tools does include it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plenty of methods are published every week and have zero impact because they are redundant, badly written, only work on a single data set, hard to parameterize, or outright irreproducible (As I have not read NEO-k-Means, I'm not claiming any of this applies to this particular algorithm!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, it is not the first to do non-overlapping clustering.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fuzzy c-means (FCM): a soft-asssignment k-means variant&lt;/li&gt;&#xA;&lt;li&gt;EM (Gaussian Mixture Modeling) is even more powerful, and has a sound theorertical backing&lt;/li&gt;&#xA;&lt;li&gt;Subspace and correlation clusterings are usually overlapping, too.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd start with GMM and FCM, as these have been successfully used by many people.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-01-01T11:39:24.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9582" PostTypeId="1" CreationDate="2016-01-02T04:45:46.130" Score="3" ViewCount="131" Body="&lt;p&gt;I have downloaded comments from a website which asked people whether they supported or opposed the implementation of a certain political policy related to immigration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to get any resources or ideas on how to extract aggregate support/opposition to this policy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I need a method that correctly identifies that all of the following comments are &quot;anti-immigration&quot; in some senses such as: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We need to give Americans first priority in the job market. &lt;/li&gt;&#xA;&lt;li&gt;We should not let Americans suffer more unemployment on account of immigrants. &lt;/li&gt;&#xA;&lt;li&gt;Immigrants should not be allowed to take American jobs. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Similarly the method should be able to identify &quot;pro-immigration&quot; comments, such as:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Providing this service to immigrants will be good for the economy. &lt;/li&gt;&#xA;&lt;li&gt;The American economy will suffer if immigrants aren't allowed to continue to work here. &lt;/li&gt;&#xA;&lt;li&gt;I do not think that passing this law will be detrimental to American jobs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15128" LastEditorUserId="5177" LastEditDate="2016-07-02T15:36:12.350" LastActivityDate="2016-09-01T16:55:22.240" Title="Sentiment Analysis of comments to understand support on a topic" Tags="|text-mining|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9583" PostTypeId="1" CreationDate="2016-01-02T09:01:43.163" Score="-4" ViewCount="2512" Body="&lt;p&gt;Can somebody answer that? It would be good if the answer comes with evidences or some research paper. I'm not asking for opinions&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-01-02T18:52:21.370" Title="What are the top 10 problems yet to solve in machine learning?" Tags="|machine-learning|ranking|" AnswerCount="1" CommentCount="7" ClosedDate="2016-01-02T22:44:25.493" ContentLicense="CC BY-SA 3.0" />
  <row Id="9584" PostTypeId="2" ParentId="9553" CreationDate="2016-01-02T10:15:40.907" Score="2" Body="&lt;p&gt;The libraries usually exclude 1-length tokens and tokens with no alpha-numeric characters because typically they are noise and do not have any descriptive power. That is, these tokens are usually not helpful, say, in distinguishing between relevant vs not relevant documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if in your domain you feel like 1-length tokens can be helpful, feel free to use them as well. For example, if all the document that contain &lt;code&gt;1&lt;/code&gt; belong to the same topic, it may be a good idea to preserve this token. &lt;code&gt;1&lt;/code&gt; has descriptive power in this case: it can help distinguish between one particular topic and the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, your next question is about LSI. For LSI there is no difference if a column in the document-term matrix corresponds to a 1-char token or to a 5-char token. So you can use LSI in your analysis. &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-01-02T10:15:40.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9585" PostTypeId="1" CreationDate="2016-01-02T17:02:04.953" Score="1" ViewCount="119" Body="&lt;p&gt;In the biology class we've learned, that the neurons are connected. And if two or more neurons interact with each other often, then the connection gets stronger and stronger, and new connections may form. But if a connection is unused for a while, then it gets weaker and weaker, and it may lost forever. That is how the human brain/memory learns and forgets something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a bit confused now. Because I've started to learn about artificial neurons, and the method of learning is very different. We analyze, how efficient the network was, and before restarting it, we update the weights a bit, based on the relative difference between the returned and the expected value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: How is a real-world brain capable of learning? And does it have a default configuration?&lt;/p&gt;&#xA;" OwnerUserId="15024" LastActivityDate="2016-01-02T21:49:31.620" Title="How do a real brain actually learn?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="2" ClosedDate="2016-01-02T22:43:53.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="9586" PostTypeId="2" ParentId="9583" CreationDate="2016-01-02T17:09:31.627" Score="1" Body="&lt;p&gt;I found a similar question to yours from 2013 on Quora. To my knowledge, the problems given in the post are still mostly unsolved. Here you can find the &lt;a href=&quot;http://www.quora.com/What-were-the-Top-10-problems-in-Machine-Learning-for-2013&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think #9 is perhaps the greatest obstacle.  How can we create a machine that hears, sees, and learns like a human?  Just like a human child starts at zero and learns from his/her senses, perhaps that is the best way to create a truly intelligent machine.         &lt;/p&gt;&#xA;" OwnerUserId="15083" LastEditorUserId="14994" LastEditDate="2016-01-02T18:52:21.370" LastActivityDate="2016-01-02T18:52:21.370" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9587" PostTypeId="2" ParentId="9585" CreationDate="2016-01-02T21:40:25.217" Score="1" Body="&lt;p&gt;Artificial Neural Networks are algorithms loosely based in how a brain functions so they shouldn't be treated as the equivalent of brain learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The science that studies all the details of how neurons and brains learn is &lt;a href=&quot;http://www.scholarpedia.org/article/Category:Computational_neuroscience&quot; rel=&quot;nofollow&quot;&gt;Computational Neuroscience&lt;/a&gt; and the question you ask is pretty much an open question but it has many great hypothesis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method you described about how artificial neural networks update their &quot;weights&quot; is called backpropagation and some authors included O'Reilly, Randall; Munakata, Yuko (2000) in Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain argue that backpropagation is actually the algorithm that drives learning in the brain. In later publications they propose architectures that include the Hippocampus region to re-create short term memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However in recent years, with the advent of mainstream deep learning new ideas have arrived along with old ideas with new approaches. The ones that look to replicate the brain and its function or at least approximate it as much as we currently can are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Spiking_neural_network&quot; rel=&quot;nofollow&quot;&gt;Spiking neural networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_temporal_memory&quot; rel=&quot;nofollow&quot;&gt;Hierarchical temporal memory / Numenta&lt;/a&gt; (They also have resources and solutions for data science problems like anomaly detection and classification)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scholarpedia.org/article/Category:Computational_Intelligence&quot; rel=&quot;nofollow&quot;&gt;Other theories in Computational Intelligence&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Spiking neural networks are computationally expensive and hard to tune, their performance isn't close to what deep learning can do, however natural brains use some form of Spiking neural networks which tells you about the potential of these kind of models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another interesting mention is &lt;a href=&quot;http://arxiv.org/abs/1410.5401&quot; rel=&quot;nofollow&quot;&gt;Neural Turing Machines&lt;/a&gt;. It's an interesting approach on how certain brain inspired processes can be used to compute diverse problems.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-01-02T21:49:31.620" LastActivityDate="2016-01-02T21:49:31.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9588" PostTypeId="1" CreationDate="2016-01-03T02:10:10.643" Score="8" ViewCount="97969" Body="&lt;p&gt;testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'&lt;/p&gt;&#xA;" OwnerUserId="15144" LastEditorUserId="21" LastEditDate="2016-05-10T13:56:58.010" LastActivityDate="2017-11-27T16:26:01.727" Title="How to select particular column in Spark(pyspark)?" Tags="|apache-spark|pyspark|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9590" PostTypeId="1" CreationDate="2016-01-03T11:29:14.937" Score="3" ViewCount="2084" Body="&lt;p&gt;I have no background AT ALL in data science/stats/mathematics. However, I've always been interested in what data shows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a huge dataset right now - daily attendance figures for a factory of ~300 for the past 10 years. I'm interested in finding out answers to questions like is there a pattern of leaves correlated with public holidays? For example, around which holiday (+/- 2 days) are workers most likely to take a leave? This is an expected pattern. Or, was there a significant increase (+10%) in on-time reporting after bonuses were issued. Maybe there are hidden patterns which an algorithm can find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a tool I can plug this data in which can help me find these patterns? Google tells me there's a tool &lt;a href=&quot;http://www.i-programmer.info/news/84-database/3501-mine-finding-patterns-in-big-data.html&quot; rel=&quot;nofollow&quot;&gt;http://www.i-programmer.info/news/84-database/3501-mine-finding-patterns-in-big-data.html&lt;/a&gt; but I'm not sure if this is the right direction for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate any advice!&lt;/p&gt;&#xA;" OwnerUserId="15140" LastActivityDate="2016-09-24T11:50:54.717" Title="What tool to find expected and hidden patterns in data?" Tags="|data-mining|" AnswerCount="1" CommentCount="5" ClosedDate="2017-03-04T10:37:06.863" ContentLicense="CC BY-SA 3.0" />
  <row Id="9594" PostTypeId="1" AcceptedAnswerId="9610" CreationDate="2016-01-03T18:31:14.720" Score="0" ViewCount="57" Body="&lt;p&gt;I've successfully used Spike Slab in the past, but with this data it seems like something is going wrong. My code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;require(spikeslab)&#xA;&amp;gt;set.seed(2)&#xA;&amp;gt;model1_ss&amp;lt;-with(data_use,spikeslab(gdeaths_per_100_thousand~gdealers_per_100_thousand+Physically.Unhealthy.Days+Mentally.Unhealthy.Days, na.rm=TRUE))&#xA;&amp;gt;summary(model1_ss)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the first 10 lines of data_use is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Physically.Unhealthy.Days  Mentally.Unhealthy.Days gdeaths_per_100_thousand    gdealers_per_100_thousand&#xA; 5.2        4.1     19.70210418  30.11828855&#xA; 4.9        4.1     21.6723146   32.83851989&#xA; 5.1        4.3     23.52217417  39.4076506&#xA; 5.1        3.6     NA           38.03625359&#xA; 3.4        4.1     17.78983533  27.28191219&#xA; 3.4        4.2     19.4576324   32.2906761&#xA; 3.3        3.8     13.9248167   40.75855495&#xA; 3.3        3.8     12.05513916  36.2194827&#xA; 5.9        3.9     NA           31.52340694&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And it returns this, which is totally unlike what I expect from from Spike Slab:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Length Class      Mode     &#xA;summary           4  data.frame list     &#xA;verbose          10  -none-     list     &#xA;terms             3  terms      call     &#xA;sigma.hat         1  -none-     numeric  &#xA;y              2132  -none-     numeric  &#xA;xnew          12792  -none-     numeric  &#xA;x             12792  -none-     numeric  &#xA;y.center          1  -none-     numeric  &#xA;x.center          6  -none-     numeric  &#xA;x.scale           6  -none-     numeric  &#xA;names             6  -none-     character&#xA;bma               6  -none-     numeric  &#xA;bma.scale         6  -none-     numeric  &#xA;gnet              6  -none-     numeric  &#xA;gnet.scale        6  -none-     numeric  &#xA;gnet.path         2  -none-     list     &#xA;gnet.obj         16  lars       list     &#xA;gnet.obj.vars     6  -none-     numeric  &#xA;gnet.parms        6  -none-     numeric  &#xA;phat              1  -none-     numeric  &#xA;complexity      500  -none-     numeric  &#xA;ridge           500  -none-     list     &#xA;model           500  -none-     list &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to interpret this that I don't know about? Or is there something wrong with how I'm running it? Thanks so much for any help, I have no idea what to try.&lt;/p&gt;&#xA;" OwnerUserId="15154" LastActivityDate="2016-01-04T14:38:12.230" Title="Spike Slab in r, bad output" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9595" PostTypeId="2" ParentId="9590" CreationDate="2016-01-03T21:24:47.673" Score="1" Body="&lt;p&gt;You are unlikely to find a tool that will truly do this for you. Various bits of software may produce some nice graphs. However we're not at a stage where tools can generate interesting questions to ask, ask them, and then explain the results to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is commercial, your best bet is to hire someone with an appropriate background and set them some questions. Tell them what your biggest goals and costs are, explain the data to them, and see what happens.  So the answer is a contractor.&lt;/p&gt;&#xA;" OwnerUserId="5303" LastActivityDate="2016-01-03T21:24:47.673" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9596" PostTypeId="1" AcceptedAnswerId="9606" CreationDate="2016-01-03T23:32:48.993" Score="3" ViewCount="1246" Body="&lt;p&gt;So I have a large collection of blog posts containing &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;content&lt;/code&gt;, &lt;code&gt;category&lt;/code&gt;, &lt;code&gt;tags&lt;/code&gt; and &lt;code&gt;geo-location&lt;/code&gt; fields and I'm looking to achieve three things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign a category (or multiple categories) to all the posts and any new ones. I have a strict vocabulary of categories.&lt;/li&gt;&#xA;&lt;li&gt;Add new tags to the posts that might be relevant to the post.&lt;/li&gt;&#xA;&lt;li&gt;Mark the post if it contains information about a place. For example: &lt;code&gt;Lorem ipsum dolor sit amet San Francisco, consectetur adipiscing elit&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I've been looking into different machine learning algorithms, most recently decision trees, but I don't feel that is the best algorithm to work out the problems above (or that I haven't understood them enough).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of these posts already contain &lt;code&gt;categories&lt;/code&gt;, &lt;code&gt;tags&lt;/code&gt; and &lt;code&gt;geo-location&lt;/code&gt; data. Some do not contain any information and some have only a few details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best machine learning algorithm to look into to solve each of the three areas?&lt;/p&gt;&#xA;" OwnerUserId="15157" LastEditorUserId="8820" LastEditDate="2016-01-04T12:20:23.690" LastActivityDate="2019-01-18T09:40:53.570" Title="Machine learning algorithm to classify blog posts" Tags="|machine-learning|data-mining|classification|algorithms|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9597" PostTypeId="2" ParentId="9596" CreationDate="2016-01-03T23:54:03.493" Score="1" Body="&lt;p&gt;For text data, &lt;strong&gt;linear SVMs&lt;/strong&gt; are still state of the art.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For named entity recognition, look up some NER toolkits.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-01-03T23:54:03.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9598" PostTypeId="1" AcceptedAnswerId="9599" CreationDate="2016-01-04T00:17:58.200" Score="10" ViewCount="4572" Body="&lt;p&gt;I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield, since harvesting of crop happens at end of growing season of crop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am familiar with performing machine learning using scikit-learn. However, not sure how to represent this problem. The tricky part here is that temperature and precipitation are daily but yield is just 1 value per year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I approach this?&lt;/p&gt;&#xA;" OwnerUserId="12985" LastActivityDate="2020-07-31T14:32:22.763" Title="Building a machine learning model to predict crop yields based on environmental data" Tags="|python|scikit-learn|pandas|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9599" PostTypeId="2" ParentId="9598" CreationDate="2016-01-04T00:24:37.313" Score="3" Body="&lt;p&gt;For starters, you can predict the yield for the upcoming year based on the daily data for the previous year. You can estimate the model parameters by considering each year's worth of data as one &quot;point&quot;, then validate the model using cross-validation. You can extend this model by considering more than the past year, but look back too far and you'll have trouble validating your model and overfit.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-04T00:24:37.313" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9600" PostTypeId="2" ParentId="9596" CreationDate="2016-01-04T01:29:57.740" Score="1" Body="&lt;p&gt;You might want to look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes classifier&lt;/a&gt; or have a look at this page on &lt;a href=&quot;https://stackoverflow.com/questions/13788229/very-simple-text-classification-by-machine-learning&quot;&gt;Very simple text classification by machine learning&lt;/a&gt;. You could also look at &lt;a href=&quot;http://nlp.stanford.edu:8080/ner/process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Named Entity Tagger&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15159" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-04T01:29:57.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9602" PostTypeId="1" CreationDate="2016-01-04T04:49:51.893" Score="4" ViewCount="60" Body="&lt;p&gt;Consider this scenario. There is a website that ranks every type of sandwich. For each type of sandwich, users fill out a survey about the following: A) how tasty it is (on a scale from 1-5); B) how filling it is (on a scale from 1 to 5); C*) if it is spicy (-1 or 1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The website displays the average value of the answers to Question A, the average value of the answers to Question B, and the sum of the answers to Question C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question A and Question B are mandatory. Question C, however, is optional. I have the number of user-submitted ratings for each sandwich, and I have the sum of the answers to C. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to compare the spiciness of different sandwiches. Since different sandwiches have different rankings, I cannot simply compare the sum of the answers to Question C. I also cannot weight it as &quot;Sum of Answers to Question C/Number of Ratings&quot; since Question C is optional and some people won't answer it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best way to go about comparing the spiciness of different sandwiches under these constraints?&lt;/p&gt;&#xA;" OwnerUserId="15163" LastActivityDate="2016-01-04T14:39:34.630" Title="How to deal with analyzing optional survey data" Tags="|statistics|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9603" PostTypeId="2" ParentId="6224" CreationDate="2016-01-04T07:03:09.920" Score="1" Body="&lt;p&gt;Check &lt;a href=&quot;http://web.maths.unsw.edu.au/~adelle/Garvan/Assays/GoodnessOfFit.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A value closer to 0 indicates that the model has a smaller random error component, and that the fit will be more useful for prediction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="15167" LastActivityDate="2016-01-04T07:03:09.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9604" PostTypeId="1" AcceptedAnswerId="9623" CreationDate="2016-01-04T10:38:13.970" Score="5" ViewCount="7607" Body="&lt;p&gt;I have seen many examples online regarding the MNIST dataset, but it's all in black and white. In that case, a 2D array can be constructed where the values at each array element represent the intensity of the corresponding pixel. However, what if I want to do colored images? What's the best way to represent the RGB data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a very brief discussion of it &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap6.html#exercise_683491&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, which I quote below. However, I still don't get how the RGB data should be organized. Additionally, is there some OpenCV library/command we should use to preprocess the colored images?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the feature detectors in the second convolutional-pooling layer have&#xA;  access to all the features from the previous layer, but only within&#xA;  their particular local receptive field*&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;*This issue would have arisen in the first layer if the input images were in color. In that case we'd have 3 input features for each pixel,&#xA;  corresponding to red, green and blue channels in the input image. So&#xA;  we'd allow the feature detectors to have access to all color&#xA;  information, but only within a given local receptive field.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="13625" LastActivityDate="2016-01-04T22:49:17.717" Title="How to prepare colored images for neural networks?" Tags="|neural-network|deep-learning|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9605" PostTypeId="2" ParentId="9554" CreationDate="2016-01-04T10:38:29.700" Score="4" Body="&lt;p&gt;Check &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; project on github. It contains a comprehensive list of open source projects grouped by language, with some short descriptions. I think you can find there some of them which meet you needs.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="29169" LastEditDate="2021-03-11T19:15:10.317" LastActivityDate="2021-03-11T19:15:10.317" CommentCount="0" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 4.0" />
  <row Id="9606" PostTypeId="2" ParentId="9596" CreationDate="2016-01-04T12:05:47.320" Score="2" Body="&lt;h1&gt;Question 1: Category prediction&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;To predict the category of a new blog post, you could do the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Build a MLP (multilayer Perceptron, a very simple neural network). Each category gets an output node, each tag is a binary input node. However, this will only work if the number of tags is very little. As soon as you add new tags, you will have to retrain the network.&lt;/li&gt;&#xA;&lt;li&gt;Build a MLP with &quot;important words&quot; as features.&lt;/li&gt;&#xA;&lt;li&gt;If you have internal links, you might want to have a look at &quot;On Node Classification in Dynamic Content-based Networks&quot;. In case you're German, you might also like &lt;a href=&quot;http://arxiv.org/abs/1512.04469&quot; rel=&quot;nofollow&quot;&gt;Über die Klassifizierung von Knoten in dynamischen Netzwerken mit Inhalt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;You could take all words you currently have, see those as a vector space. Fix that vocabulary (and probably remove some meaningless words like &quot;with&quot;, &quot;a&quot;, &quot;an&quot; - this is commonly called a &quot;stopword&quot;). For each text, you count the different words you have in your vocabulary. A new blog post is point in this space. Use $k$ nearest neighbor for classification.&lt;/li&gt;&#xA;&lt;li&gt;Use combinations of different predictiors by letting each predictor give a vote for a classification.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Yiming Yang, Jan O. Pedersen: &lt;a href=&quot;http://www.surdeanu.info/mihai/teaching/ista555-spring15/readings/yang97comparative.pdf&quot; rel=&quot;nofollow&quot;&gt;A Comparative Study on Feature Selection in Text Categorization&lt;/a&gt;, 1997.&lt;/li&gt;&#xA;&lt;li&gt;Scikit-learn: &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow&quot;&gt;Working With Text Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Question 2: Tagging texts&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This can be treated the same way like question 1.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Question 3: Finding locations&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Download a database of countries / cities (e.g. &lt;a href=&quot;https://www.maxmind.com/de/free-world-cities-database&quot; rel=&quot;nofollow&quot;&gt;maxmind&lt;/a&gt;) and just search for a match.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-04T12:11:13.647" LastActivityDate="2016-01-04T12:11:13.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9607" PostTypeId="1" AcceptedAnswerId="9608" CreationDate="2016-01-04T12:51:14.620" Score="0" ViewCount="102" Body="&lt;p&gt;I'm new to computer vision, and I'm looking for a good place to start from, what's better between open cv in python or open cv in c++&lt;/p&gt;&#xA;" OwnerUserId="15171" LastActivityDate="2016-01-04T14:30:59.730" Title="Open cv and computer vision" Tags="|computer-vision|" AnswerCount="1" CommentCount="0" ClosedDate="2016-01-04T13:30:05.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="9608" PostTypeId="2" ParentId="9607" CreationDate="2016-01-04T13:05:35.400" Score="1" Body="&lt;p&gt;Depends on your programming skills. Here is the summary:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OpenCV is a great tool originally developed in C++ and after a while a Python interface was added. In industry (and also many academic research group) C++ is the popular language for Computer Vision as the nature of data needs efficiency in computation. So starting with python is an easy way to learn Computer Vision and OpenCV capabilities but when you are applying for a job in industry or academia, they most probably need you to get your hands dirty with C++.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2016-01-04T14:30:59.730" LastActivityDate="2016-01-04T14:30:59.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9609" PostTypeId="2" ParentId="9432" CreationDate="2016-01-04T13:20:15.520" Score="1" Body="&lt;h2&gt;According to graph theory, wouldn't this make Person B also very influential?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;No. You need to think about how to set up your graph first and then go further. In this case your graph is not simple but both &lt;strong&gt;weighted&lt;/strong&gt; and &lt;strong&gt;directed&lt;/strong&gt; thus handling a request of B by A neither reduces A's influence nor improves B's. Because the weights of directed edges are determining the influence.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Is there a better way than just counting number of interactions?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1511.01523&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Community Detection&lt;/em&gt;&lt;/a&gt; or what computer scientists call &lt;a href=&quot;http://dollar.biz.uiowa.edu/~street/graphClustering.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Graph Clustering&lt;/em&gt;&lt;/a&gt;. If you do not have to limit yourself to hierarchies, you can extract communities who have the most interaction and share suggestions inside the community. &lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-04T13:20:15.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9610" PostTypeId="2" ParentId="9594" CreationDate="2016-01-04T14:38:12.230" Score="0" Body="&lt;p&gt;There seems to be no &lt;code&gt;summary&lt;/code&gt; method for objects of class &lt;code&gt;spikeslab&lt;/code&gt;, so the default summary method is used which prints the internal list elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just print the object to get a nice summary - maybe that's what you want? Is this the output you expected?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; model1_ss&#xA;------------------------------------------------------------------- &#xA;Variable selection method     : AIC &#xA;Big p small n                 : FALSE &#xA;Screen variables              : FALSE &#xA;Fast processing               : TRUE &#xA;Sample size                   : 7 &#xA;No. predictors                : 3 &#xA;No. burn-in values            : 500 &#xA;No. sampled values            : 500 &#xA;Estimated mse                 : 7.5765 &#xA;Model size                    : 2 &#xA;&#xA;&#xA;---&amp;gt; Top variables:&#xA;                            bma  gnet bma.scale gnet.scale&#xA;Mentally.Unhealthy.Days   2.290 2.331    13.003     13.236&#xA;Physically.Unhealthy.Days 0.926 0.951     1.084      1.113&#xA;------------------------------------------------------------------- &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-01-04T14:38:12.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9611" PostTypeId="2" ParentId="9602" CreationDate="2016-01-04T14:39:34.630" Score="2" Body="&lt;p&gt;Your best bet is to change the way you ask question C and the way you measure it. Giving a yes / no response for C will not let you gauge the actual spiciness so you need to adjust it to a scale and allow individuals to rank it. While all of your questions are fairly subjective, C is the most so as tolerance to spice plays a big role in if one thinks it spicy or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest using a 1-5 scale, as you have for the others, and give comparative suggestions for each level of the scale e.g. 1 is eating bread while 5 is equivalent to eating a habnero pepper.&lt;/p&gt;&#xA;" OwnerUserId="15173" LastActivityDate="2016-01-04T14:39:34.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9612" PostTypeId="1" AcceptedAnswerId="9618" CreationDate="2016-01-04T15:17:44.630" Score="3" ViewCount="453" Body="&lt;p&gt;I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield (since harvesting of crop happens at end of growing season of crop).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per, &lt;a href=&quot;https://datascience.stackexchange.com/questions/9598/building-a-machine-learning-model-to-predict-crop-yields-based-on-environmental&quot;&gt;Building a machine learning model to predict crop yields based on environmental data&lt;/a&gt;, I am using &lt;code&gt;sklearn.cross_validation.LabelKFold&lt;/code&gt; to assign each year the same label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is that since I have a single target value per year, do I need to interpolate to fill in target values for all the other days of the year? Should I just use the same target value for each day of the year?&lt;/p&gt;&#xA;" OwnerUserId="12985" LastEditorUserId="29169" LastEditDate="2019-06-07T16:52:18.083" LastActivityDate="2019-06-07T16:52:18.083" Title="Assigning values to missing target vector values in scikit-learn" Tags="|python|scikit-learn|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9613" PostTypeId="2" ParentId="9598" CreationDate="2016-01-04T15:55:41.277" Score="1" Body="&lt;p&gt;You can use a &lt;a href=&quot;https://math.stackexchange.com/questions/93195/bayesian-network-for-dummies&quot;&gt;Bayesian Belief Network&lt;/a&gt; for prediction. Here is a link for basic explanation: &lt;a href=&quot;https://math.stackexchange.com/questions/93195/bayesian-network-for-dummies&quot;&gt;Bayesian Network&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15174" LastEditorUserId="29575" LastEditDate="2020-07-31T14:32:22.763" LastActivityDate="2020-07-31T14:32:22.763" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9614" PostTypeId="1" AcceptedAnswerId="9622" CreationDate="2016-01-04T16:43:21.400" Score="1" ViewCount="67" Body="&lt;p&gt;I have data on my customers (age, location, gender) and number of interactions with customer by channel (#calls, #sms, #letters) and data on whether they have bought a product (yes/no).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to predict what combination of interaction (e.g. 3 calls, 2 letters) is the best approach when contacting different types of customers. What would be an appropriate statistical/ machine learning technique to use to calculate this?&lt;/p&gt;&#xA;" OwnerUserId="15176" LastEditorUserId="8005" LastEditDate="2016-01-05T08:21:16.523" LastActivityDate="2016-01-05T08:50:56.897" Title="How can I use data on customer interactions to drive communication strategy?" Tags="|statistics|marketing|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9616" PostTypeId="1" CreationDate="2016-01-04T18:35:57.540" Score="13" ViewCount="35244" Body="&lt;p&gt;I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it would be a map of the US with each state having a color from a color gradient that corresponds to a quantitative value. I don't know what this plot type is called, so I'm having trouble figuring out how to do this in MATLAB or another program, but does anyone know an easy way to do this? I have a vector of data (of length 50) and just need to make a US map plot.&lt;/p&gt;&#xA;" OwnerUserId="15180" LastEditorUserId="29169" LastEditDate="2021-03-13T23:29:05.313" LastActivityDate="2021-03-13T23:29:05.313" Title="How to create US state choropleth map" Tags="|data|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9617" PostTypeId="1" CreationDate="2016-01-04T19:14:51.783" Score="5" ViewCount="1762" Body="&lt;p&gt;In &lt;a href=&quot;https://www.youtube.com/watch?v=ZYUnyyVgtyA&quot; rel=&quot;nofollow&quot;&gt;Neural networks [3.8] : Conditional random fields - Markov network&lt;/a&gt; by Hugo Larochelle it seems to me that a Markov Random Field is a special case of a CRF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the Wikipedia article &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=Markov_random_field&amp;amp;oldid=695876526#Conditional_random_fields&quot; rel=&quot;nofollow&quot;&gt;Markov random field&lt;/a&gt; it says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One notable variant of a Markov random field is a conditional random field, in which each random variable may also be conditioned upon a set of global observations o.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This would mean that CRFs are a special case of MRFs.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Definitions&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Markov Random Field&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Again, according to Wikipedia&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given an undirected graph $G=(V,E)$, a set of random variables $X = (X_v)_{v\in V}$ indexed by $V$  form a Markov random field with respect to $G$  if they satisfy the local Markov properties:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Pairwise Markov property&lt;/strong&gt;: Any two non-adjacent variables are conditionally independent given all other variables:&#xA;  $X_u \perp\!\!\!\perp X_v \mid X_{V \setminus \{u,v\}} \quad \text{if } \{u,v\} \notin E$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Local Markov property&lt;/strong&gt;: A variable is conditionally independent of all other variables given its neighbors:&#xA;  $X_v \perp\!\!\!\perp X_{V\setminus \operatorname{cl}(v)} \mid X_{\operatorname{ne}(v)}$&#xA;  where ${\textstyle \operatorname{ne}(v)}$ is the set of neighbors of $v$, and $\operatorname{cl}(v) = v \cup \operatorname{ne}(v)$ is the closed neighbourhood of $v$.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Global Markov property&lt;/strong&gt;: Any two subsets of variables are conditionally independent given a separating subset:&#xA;  $X_A \perp\!\!\!\perp X_B \mid X_S$&#xA;  where every path from a node in $A$ to a node in $B$ passes through $S$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please note if you know a citable source which gives a good definition.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Conditional Random Fields&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=Conditional_random_field&amp;amp;oldid=695232246#Description&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Lafferty, McCallum and Pereira[1] define a CRF on observations $\boldsymbol{X}$ and random variables $\boldsymbol{Y}$ as follows:&lt;/p&gt;&#xA;  &#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Let $G = (V , E)$ be a graph such that&lt;/p&gt;&#xA;    &#xA;    &lt;p&gt;$\boldsymbol{Y} = (\boldsymbol{Y}_v)_{v\in V}$, so that $\boldsymbol{Y}$ is indexed by the vertices of $G$. Then $(\boldsymbol{X}, \boldsymbol{Y})$ is a conditional random field when the random variables $\boldsymbol{Y}_v$, conditioned on $\boldsymbol{X}$, obey the Markov property with respect to the graph: $$p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \neq v) = p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \sim v)$$ where $\mathit{w} \sim v$ means that $w$ and $v$ are neighbors in G.&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;  &#xA;  &lt;p&gt;What this means is that a CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets $\boldsymbol{X}$ and $\boldsymbol{Y}$, the observed and output variables, respectively; the conditional distribution $p(\boldsymbol{Y}|\boldsymbol{X})$ is then modeled.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;What is the relationship between Markov Random Fields and Conditional Random Fields?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-18T12:22:30.613" LastActivityDate="2016-01-18T12:22:30.613" Title="What is the relationship between Markov Random Fields and Conditional Random Fields?" Tags="|machine-learning|markov|pgm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9618" PostTypeId="2" ParentId="9612" CreationDate="2016-01-04T20:12:13.473" Score="2" Body="&lt;p&gt;The model likely won't have much predictive power if the input is a single day. No weather patterns longer than one day can be captured that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead you should aggregate the days together. You can come up with different features that describe your larger, aggregated unit of time (months, year). For example mean precipitation is a very simple one. Binning the data and using counts within those bins would also work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More advanced options would roll the time all the way up to a full year and learn a feature set at that level.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2016-01-04T20:12:13.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9619" PostTypeId="1" CreationDate="2016-01-04T20:22:46.043" Score="6" ViewCount="881" Body="&lt;p&gt;I am working on this project where I have to predict the acceptance probability of a literary article, let's say a research paper by a publisher, which in this case would be a Journal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use a machine learning algorithm for this project which would learn from the pool of articles/research papers published by the Publisher/Journal till date. Later I want to use the current article/research paper at hand to predict the probability of acceptance of this article/research paper on the insights that the algorithm has learnt. Here are my queries:-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What algorithm are there to tackle such task?&lt;/li&gt;&#xA;&lt;li&gt;How should I make these algorithms learn the word vector, grammars, topics etc?&lt;/li&gt;&#xA;&lt;li&gt;How do I convert the article into a feature set from which the algorithms can learn?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15182" LastActivityDate="2016-01-06T23:17:43.363" Title="How can I predict the acceptance of an article by publisher?" Tags="|machine-learning|data-mining|nlp|predictive-modeling|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9620" PostTypeId="2" ParentId="9616" CreationDate="2016-01-04T20:48:57.710" Score="10" Body="&lt;p&gt;Plotly in Python &lt;a href=&quot;https://plot.ly/python/choropleth-maps/&quot; rel=&quot;nofollow noreferrer&quot;&gt;provides a nice solution&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-python prettyprint-override&quot;&gt;&lt;code&gt;!pip install chart_studio&#xA;from chart_studio import plotly as py&#xA;import pandas as pd&#xA;&#xA;df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_us_ag_exports.csv')&#xA;&#xA;for col in df.columns:&#xA;    df[col] = df[col].astype(str)&#xA;&#xA;scl = [[0.0, 'rgb(242,240,247)'],[0.2, 'rgb(218,218,235)'],[0.4, 'rgb(188,189,220)'],\&#xA;            [0.6, 'rgb(158,154,200)'],[0.8, 'rgb(117,107,177)'],[1.0, 'rgb(84,39,143)']]&#xA;&#xA;df['text'] = df['state'] + '&amp;lt;br&amp;gt;' +\&#xA;    'Beef '+df['beef']+' Dairy '+df['dairy']+'&amp;lt;br&amp;gt;'+\&#xA;    'Fruits '+df['total fruits']+' Veggies ' + df['total veggies']+'&amp;lt;br&amp;gt;'+\&#xA;    'Wheat '+df['wheat']+' Corn '+df['corn']&#xA;&#xA;data = [ dict(&#xA;        type='choropleth',&#xA;        colorscale = scl,&#xA;        autocolorscale = False,&#xA;        locations = df['code'],&#xA;        z = df['total exports'].astype(float),&#xA;        locationmode = 'USA-states',&#xA;        text = df['text'],&#xA;        marker = dict(&#xA;            line = dict (&#xA;                color = 'rgb(255,255,255)',&#xA;                width = 2&#xA;            )&#xA;        ),&#xA;        colorbar = dict(&#xA;            title = &amp;quot;Millions USD&amp;quot;&#xA;        )&#xA;    ) ]&#xA;&#xA;layout = dict(&#xA;        title = '2011 US Agriculture Exports by State&amp;lt;br&amp;gt;(Hover for breakdown)',&#xA;        geo = dict(&#xA;            scope='usa',&#xA;            projection=dict( type='albers usa' ),&#xA;            showlakes = True,&#xA;            lakecolor = 'rgb(255, 255, 255)',&#xA;        ),&#xA;    )&#xA;    &#xA;fig = dict( data=data, layout=layout )&#xA;&#xA;url = py.plot( fig, filename='d3-cloropleth-map' )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uhhaw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uhhaw.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="111929" LastEditDate="2021-03-12T16:01:32.867" LastActivityDate="2021-03-12T16:01:32.867" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="9622" PostTypeId="2" ParentId="9614" CreationDate="2016-01-04T22:47:13.593" Score="0" Body="&lt;p&gt;I would do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, cluster your customers into several groups, based on age, location, gender etc. Second, for each group, use different combination of interaction as features to predict 'yes/no', and select which combination of interaction performs best.&lt;/p&gt;&#xA;" OwnerUserId="14463" LastActivityDate="2016-01-04T22:47:13.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9623" PostTypeId="2" ParentId="9604" CreationDate="2016-01-04T22:49:17.717" Score="6" Body="&lt;p&gt;Your R,G, and B pixel values can be broken into 3 separate channels (and in most cases this is done for you). These channels are treated no differently than feature maps in higher levels of the network. Convolution extends naturally to more than 2 dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine the greyscale, single-channel example. Say you have N feature maps to learn in the first layer. Then the output of this layer (and therefore the input to the second layer) will be comprised of N channels, each of which is the result of convolving a feature map with each window in your image. Having 3 channels in your first layer is no different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tutorial does a nice job on convolution in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot;&gt;http://deeplearning.net/tutorial/lenet.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2016-01-04T22:49:17.717" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9626" PostTypeId="1" CreationDate="2016-01-05T04:02:38.443" Score="4" ViewCount="2342" Body="&lt;p&gt;I'm working on a use case where the user will be provided a text box to enter the details of the transaction application.&lt;br&gt;&#xA;For Example user might enter the below text and I have to parse the data and accordingly create the transaction. (I took basic example as a startup)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ex 1 : Transfer $100 from Account 1 into Account 2&lt;br&gt;&#xA;  Ex 2: Move Money of total 400 dollars into account owned by Mr.xxxx from Acct#&lt;br&gt;&#xA;  Ex 3: Deposit an amount of five thousand dollars into acct#&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For above examples I have to parse and convert into a tabular format like &#xA;Trx_Type (Deposit or Transfer), Amt, From_Acct#, To_Acct#, Acct_Holder_Name,etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to look options by doing Named Entity (NER) or Classifier modelling using NLTK in python. Can someone please share ideas from where to start and guide me with a draft way to proceed.&lt;br&gt;&#xA;Looking forward for a response. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="15187" LastEditorUserId="14738" LastEditDate="2016-01-06T12:17:52.043" LastActivityDate="2016-01-08T02:21:50.947" Title="Information Extraction from Free-form text to create Transactions" Tags="|python|nlp|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9627" PostTypeId="1" CreationDate="2016-01-05T04:59:01.667" Score="2" ViewCount="1411" Body="&lt;p&gt;I've created a graph from an adjacency matrix using the igraph package but I want edges to appear in the graph only when the value in the adjacency matrix is beyond a particular threshold. Using the present method I am getting all the edges which have non zero weights.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(42)&#xA;iris_matrix &amp;lt;- as.matrix(iris[,1:4])&#xA;image(iris_matrix)&#xA;g&amp;lt;-cor(iris_matrix)&#xA;library(igraph)&#xA;R&amp;lt;-graph.adjacency(g, mode=c(&quot;undirected&quot;), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)&#xA;plot(R)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13953" LastActivityDate="2016-01-05T09:41:56.137" Title="How to make edges appear in a graph when adjacency matrix value is beyond a particular threshold?" Tags="|r|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9628" PostTypeId="2" ParentId="9598" CreationDate="2016-01-05T08:50:35.083" Score="1" Body="&lt;p&gt;You have 10 data points with each data point having 365 (temperature for each day) + 365 (precipitation for each day) dimensions. Ideally, I would first reduce dimensions via machine learning methods, e.g. PCA. Then use machine learning methods to build a prediction model. However, due to the small dataset, I don't think machine learning techniques are appropriate to your problem.&lt;/p&gt;&#xA;" OwnerUserId="14463" LastActivityDate="2016-01-05T08:50:35.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9629" PostTypeId="2" ParentId="9614" CreationDate="2016-01-05T08:50:56.897" Score="0" Body="&lt;p&gt;This is how you can approach this problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Customer Segmentation:&lt;/strong&gt; Segment each customer as a separate data point, and if possible have another project for clustering the customers depending on various factors (Example: Depending on geographical regions, age, etc). This would help in simplifying the problem statement and narrowing it too.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Now choose the best combination:&lt;/strong&gt; Now after segmentation, choose the best(average) combination of interaction mechanisms needed to be implemented for customer conversion. The mean would answer you the question: &lt;code&gt;What is the average number of interactions by different media taken by a customer before he/she is converted&lt;/code&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bonus&lt;/strong&gt;: In addition to calculating the mean, calculating the median also helps, as it helps in fixing skew due to overly compulsive customer reps who interact a bit more frequently than some, and difficult customers.  &lt;a href=&quot;https://datascience.stackexchange.com/a/8865/11097&quot;&gt;Useful reference&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-01-05T08:50:56.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9630" PostTypeId="2" ParentId="9627" CreationDate="2016-01-05T09:06:51.717" Score="1" Body="&lt;p&gt;Line 7 is what you need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(42)&#xA;iris_matrix &amp;lt;- as.matrix(iris[,1:4])&#xA;image(iris_matrix)&#xA;g&amp;lt;-cor(iris_matrix)&#xA;library(igraph)&#xA;R&amp;lt;-graph.adjacency(g, mode=c(&quot;undirected&quot;), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)&#xA;R[R &amp;gt; thresh] &amp;lt;- 0 &#xA;plot(R)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2016-01-05T09:41:56.137" LastActivityDate="2016-01-05T09:41:56.137" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9632" PostTypeId="1" AcceptedAnswerId="9640" CreationDate="2016-01-05T10:57:35.867" Score="7" ViewCount="15751" Body="&lt;p&gt;I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at  DEAP documentation, but the examples there were pretty hard for me to follow. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def  function(x,y):&#xA;     return x*y+3*x-x**2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I  am looking  for some references on how  I can make a genetic algorithm in which I can feed some initial random values for both x and y (not coming from the same dimensions). Can someone with experience creating and using genetic algorithms give me some guidance on this?&lt;/p&gt;&#xA;" OwnerUserId="14946" LastEditorUserId="13413" LastEditDate="2016-01-05T16:58:45.380" LastActivityDate="2016-01-05T19:47:52.763" Title="Simple example of genetic alg minimization" Tags="|python|optimization|genetic-algorithms|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9633" PostTypeId="1" CreationDate="2016-01-05T12:28:49.013" Score="1" ViewCount="547" Body="&lt;p&gt;I would like to use RF in order to get a score probability of a binary outcome, in order to know groups with higher probability of buy something. Like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit&amp;lt;-randomForest(formula = buy ~.  , data = pract,importance = TRUE ,ntree =1000) &#xA;&#xA;&#xA;pract$PREDICTION &amp;lt;- as.vector(predict(fit,type='prob',newdata=pract)[,2])&#xA;&#xA;test$PREDICTION &amp;lt;- as.vector(predict(fit,type='prob',newdata=test)[,2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First Question, When I perfom auc of the pract$PREDICTION I get 1! It is because overfiting? Should I use out of bag instead of pract, like this?:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pract$PREDICTION &amp;lt;- as.vector(predict(fit,type='prob')[,2]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second Question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With test$PREDICTION I want to make groups ordered by the score prob, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test$score[test$PREDICTION &amp;gt;= 0 &amp;amp; test$PREDICTION &amp;lt;=0.3] &amp;lt;-  1&#xA;    test$score[test$PREDICTION &amp;gt; 0.3 &amp;amp; test$PREDICTION &amp;lt;=0.6] &amp;lt;- 2&#xA;...&#xA;test$score[test$PREDICTION &amp;gt; 0.9 &amp;amp; test$PREDICTION &amp;lt;=1.0] &amp;lt;- 10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I measure how good my model is? by doing the same segmentation group in the pract$PREDICTION and then comparing how right and how wrong both tests are in each group ? This make any sense?&lt;/p&gt;&#xA;" OwnerUserId="14644" LastActivityDate="2016-01-05T12:28:49.013" Title="Score probability groups with Random Forest in R" Tags="|r|predictive-modeling|random-forest|probability|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9635" PostTypeId="2" ParentId="9554" CreationDate="2016-01-05T13:26:27.517" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://elki.dbs.ifi.lmu.de/&quot; rel=&quot;nofollow&quot;&gt;ELKI&lt;/a&gt; (also on &lt;a href=&quot;https://github.com/elki-project/elki&quot; rel=&quot;nofollow&quot;&gt;GitHub&lt;/a&gt;) is data mining and data science open-source project. It is unique with respect to its modular architecture: you can combine algorithms, distance functions, and indexes for acceleration with very few limitations (of course, algorithms that do not use distances cannot be combined with distances). It is not the easiest code because of efficiency. For data mining, you need to be careful about memory - using &lt;code&gt;ArrayList&amp;lt;Integer&amp;gt;&lt;/code&gt; is a no-go if you want scalability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the modular architecture, it is easy to contribute just small modules, like a single distance function or algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We keep a list of &lt;a href=&quot;http://elki.dbs.ifi.lmu.de/wiki/ProjectIdeas&quot; rel=&quot;nofollow&quot;&gt;data mining project ideas&lt;/a&gt;, roughly grouped by difficulty. Most projects are the implementation of some variant of an algorithm. ELKI aims at allowing comparative studies of algorithms, so we try to allow any combination, and cover also variants of algorithms. For example with k-means, we not only have Lloyds algorithm, but 10 variants of the general k-means theme. Over 220 articles have been (at least partially) reimplemented in ELKI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By implementing everything in the same tool, we get much more comparable results. If you use R for benchmarking, you are usually comparing apples and oranges. k-means in R itself is actually an old Fortran program, and very fast. k-means in R but in the &quot;flexclust&quot; package is 100x slower, because it is written in real R code. So don't trust a benchmark in R... also, R modules tend to be incompatible, so you often can't use distance A from modules A with algorithm B from module B. in ELKI we try to share as much code as possible across implementations to reduce such artifacts (it will, of course, never be possible to have a 100% fair benchmark - there is always room for optimization), but also to allow combining modules easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could start with something small such as the Hartigan&amp;amp;Wong k-means variant, and then continue into spherical k-means (which is meant for sparse data, where different performance optimizations may become necessary) and continue into adding better support for categorical data; or adding indexing functionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also &lt;strong&gt;love to see a better UI for ELKI&lt;/strong&gt;, but that is a major effort.&lt;/p&gt;&#xA;" OwnerUserId="2920" LastActivityDate="2016-01-05T13:26:27.517" CommentCount="0" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 3.0" />
  <row Id="9636" PostTypeId="1" AcceptedAnswerId="9638" CreationDate="2016-01-05T13:27:19.253" Score="2" ViewCount="76" Body="&lt;p&gt;I've no experience in data science so this will be one of &lt;em&gt;those&lt;/em&gt; questions...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data from &gt;100k purchases made via a webshop regarding a catalogue of around &gt;100 items. The history of purchases flattened out looks like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item1 Item2 ... ItemN Sex State&#xA;    5     0         0   M    NY&#xA;   25    15         0   F    IL&#xA;    0     1         1   ?    NY&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By playing around with the data, I can deduce simple facts like &quot;90% of all purchases include at least 3 Item1&quot;, &quot;If there are at least 4 of Item2, it is likely that Item3 is 0&quot; or &quot;60% of all customers from NY are male, but only 40% of those from IL are&quot;. Given the amount of combinations and data there is, the most obvious question: How can I approach wringing out more information from a data set like the above? I'm mostly interested in how one item does or does not entails inclusion of another... &lt;/p&gt;&#xA;" OwnerUserId="15196" LastActivityDate="2016-01-05T14:42:39.350" Title="Mine webshop history for clusters" Tags="|data-mining|classification|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9637" PostTypeId="1" CreationDate="2016-01-05T13:28:18.940" Score="8" ViewCount="4063" Body="&lt;p&gt;I need to use reinforcement learning to teach a neural net a policy for a board game. I chose Q-learining as the specific alghoritm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like a neural net to have the following structure:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;layer - &lt;code&gt;rows * cols + 1&lt;/code&gt; neurons - input - values of consecutive fields on the board (&lt;code&gt;0&lt;/code&gt; for empty, &lt;code&gt;1&lt;/code&gt; or &lt;code&gt;2&lt;/code&gt; representing a player), action (natural number) in that state&lt;/li&gt;&#xA;&lt;li&gt;layer - (??) neurons - hidden&lt;/li&gt;&#xA;&lt;li&gt;layer - 1 neuron - output - value of action in given state (float)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My first idea was to begin with creating a map of states, actions and values, and then try to teach neural network. If the teaching process would not succeed I could increase the number of neurons and start over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;a href=&quot;https://stackoverflow.com/questions/34597386/how-to-improve-performance-of-a-script-operating-on-large-amount-of-data&quot;&gt;I quickly ran into performance problems&lt;/a&gt;. Firstly, I needed to switch from simple in-memory Python &lt;code&gt;dict&lt;/code&gt; to a database (not enough RAM). Now the database seems to be a bottleneck (simply speaking there are so many possible states that the retrieval of actions' values is taking a noticeable amount of time). Calculations would take weeks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I guess it would be possible to teach neural network on the fly, without the layer of a map in the middle. But how would I choose a right number of neurons on the hidden layer? How could I figure out that I'm loosing periously saved (learned) data?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="15197" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-02-15T15:54:55.473" Title="How to teach neural network a policy for a board game using reinforcement learning?" Tags="|machine-learning|neural-network|reinforcement-learning|q-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9638" PostTypeId="2" ParentId="9636" CreationDate="2016-01-05T14:42:39.350" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;Frequent Item-Set Mining&lt;/a&gt; is what you are looking for. You can see the tree structure of your frequent itemsets and the association rules afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your data I'd suggest to look at the whole for a while to get a sense on what you have in hand. Playing with concepts like Probability Distributions, &lt;a href=&quot;https://en.wikipedia.org/wiki/Entropy_%28information_theory%29&quot; rel=&quot;nofollow&quot;&gt;Entropy&lt;/a&gt;, etc would be really helpful in case you can reduce the size of your features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt; also gives you the opportunity of projecting your data into a low-dimenstional space and you can see also plots showing first several PCs in 2-D or 3-D and get an impression about your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before all above I strongly suggest to see if you have &lt;a href=&quot;https://www.utexas.edu/cola/prc/_files/cs/Missing-Data.pdf&quot; rel=&quot;nofollow&quot;&gt;Missing Values&lt;/a&gt; and if yes try to cope with them.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-05T14:42:39.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9639" PostTypeId="2" ParentId="9617" CreationDate="2016-01-05T17:17:03.907" Score="3" Body="&lt;p&gt;Conditinal Random Fields (CRFs) are a special case of Markov Random Fields (MRFs).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1.5.4 Conditional Random Field&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A Conditional Random Field (CRF) is a form of MRF that defines a posterior for&#xA;  variables x given data z, as with the hidden MRF above. Unlike the hidden MRF,&#xA;  however, the factorization into the data distribution P (x|z) and the prior P&#xA;  (x) is not made explicit [288]. This allows complex dependencies of x on z to&#xA;  be written directly in the posterior distribution, without the factorization&#xA;  being made explicit. (Given P (x|z), such factorizations always exist,&#xA;  however—infinitely many of them, in fact—so there is no suggestion that the CRF&#xA;  is more general than the hidden MRF, only that it may be more convenient to&#xA;  deal with.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: Blake, Kohli and Rother: Markov random fields for vision and image processing. 2011.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A conditional random field or CRF (Lafferty et al. 2001), sometimes a&#xA;  discriminative random field (Kumar and Hebert 2003), is just a version of an&#xA;  MRF where all the clique potentials are conditioned on input features:&#xA;  [...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The advantage of a CRF over an MRF is analogous to the advantage of a discriminative&#xA;  classifier over a generative classifier (see Section 8.6), namely, we don’t need to “waste resources”&#xA;  modeling things that we always observe. [...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The disadvantage of CRFs over MRFs is that they require labeled training data, and they are slower to train[...]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: Kevin P. Murphy: &lt;a href=&quot;https://www.cs.ubc.ca/~murphyk/MLbook/pml-print3-ch19.pdf&quot; rel=&quot;nofollow&quot;&gt;Machine Learning: A Probabilistic Perspective&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-05T17:44:35.570" LastActivityDate="2016-01-05T17:44:35.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9640" PostTypeId="2" ParentId="9632" CreationDate="2016-01-05T17:40:49.800" Score="6" Body="&lt;p&gt;&lt;a href=&quot;http://lethain.com/genetic-algorithms-cool-name-damn-simple/&quot; rel=&quot;noreferrer&quot;&gt;Here is a trivial example&lt;/a&gt;, which captures the essence of genetic algorithms more meaningfully than the polynomial you provided.  The polynomial you provided is solvable via &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;stochastic gradient descent&lt;/code&gt;&lt;/a&gt;, which is a simpler minimimization technique.  For this reason, I am instead suggesting this excellent article and example by Will Larson.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://lethain.com/genetic-algorithms-cool-name-damn-simple/&quot; rel=&quot;noreferrer&quot;&gt;Quoted from the original article&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Defining a Problem to Optimize&lt;/strong&gt; Now we're going to put together a simple example of using a genetic algorithm in Python. We're going to&#xA;  optimize a very simple problem: trying to create a list of N numbers&#xA;  that equal X when summed together.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;If we set N = 5 and X = 200, then these would all be appropriate&#xA;  solutions. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lst = [40,40,40,40,40]&#xA;lst = [50,50,50,25,25]&#xA;lst = [200,0,0,0,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Take a look at the &lt;a href=&quot;http://lethain.com/genetic-algorithms-cool-name-damn-simple/&quot; rel=&quot;noreferrer&quot;&gt;entire article&lt;/a&gt;, but here is the &lt;strong&gt;complete code&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Example usage&#xA;from genetic import *&#xA;target = 371&#xA;p_count = 100&#xA;i_length = 6&#xA;i_min = 0&#xA;i_max = 100&#xA;p = population(p_count, i_length, i_min, i_max)&#xA;fitness_history = [grade(p, target),]&#xA;for i in xrange(100):&#xA;    p = evolve(p, target)&#xA;    fitness_history.append(grade(p, target))&#xA;&#xA;for datum in fitness_history:&#xA;   print datum&#xA;&quot;&quot;&quot;&#xA;from random import randint, random&#xA;from operator import add&#xA;&#xA;def individual(length, min, max):&#xA;    'Create a member of the population.'&#xA;    return [ randint(min,max) for x in xrange(length) ]&#xA;&#xA;def population(count, length, min, max):&#xA;    &quot;&quot;&quot;&#xA;    Create a number of individuals (i.e. a population).&#xA;&#xA;    count: the number of individuals in the population&#xA;    length: the number of values per individual&#xA;    min: the minimum possible value in an individual's list of values&#xA;    max: the maximum possible value in an individual's list of values&#xA;&#xA;    &quot;&quot;&quot;&#xA;    return [ individual(length, min, max) for x in xrange(count) ]&#xA;&#xA;def fitness(individual, target):&#xA;    &quot;&quot;&quot;&#xA;    Determine the fitness of an individual. Higher is better.&#xA;&#xA;    individual: the individual to evaluate&#xA;    target: the target number individuals are aiming for&#xA;    &quot;&quot;&quot;&#xA;    sum = reduce(add, individual, 0)&#xA;    return abs(target-sum)&#xA;&#xA;def grade(pop, target):&#xA;    'Find average fitness for a population.'&#xA;    summed = reduce(add, (fitness(x, target) for x in pop))&#xA;    return summed / (len(pop) * 1.0)&#xA;&#xA;def evolve(pop, target, retain=0.2, random_select=0.05, mutate=0.01):&#xA;    graded = [ (fitness(x, target), x) for x in pop]&#xA;    graded = [ x[1] for x in sorted(graded)]&#xA;    retain_length = int(len(graded)*retain)&#xA;    parents = graded[:retain_length]&#xA;    # randomly add other individuals to&#xA;    # promote genetic diversity&#xA;    for individual in graded[retain_length:]:&#xA;        if random_select &amp;gt; random():&#xA;            parents.append(individual)&#xA;    # mutate some individuals&#xA;    for individual in parents:&#xA;        if mutate &amp;gt; random():&#xA;            pos_to_mutate = randint(0, len(individual)-1)&#xA;            # this mutation is not ideal, because it&#xA;            # restricts the range of possible values,&#xA;            # but the function is unaware of the min/max&#xA;            # values used to create the individuals,&#xA;            individual[pos_to_mutate] = randint(&#xA;                min(individual), max(individual))&#xA;    # crossover parents to create children&#xA;    parents_length = len(parents)&#xA;    desired_length = len(pop) - parents_length&#xA;    children = []&#xA;    while len(children) &amp;lt; desired_length:&#xA;        male = randint(0, parents_length-1)&#xA;        female = randint(0, parents_length-1)&#xA;        if male != female:&#xA;            male = parents[male]&#xA;            female = parents[female]&#xA;            half = len(male) / 2&#xA;            child = male[:half] + female[half:]&#xA;            children.append(child)&#xA;    parents.extend(children)&#xA;    return parents&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think it could be quite pedagogically useful to also solve your original problem using this algorithm and then also construct a solution using &lt;code&gt;stochastic grid search&lt;/code&gt; or &lt;code&gt;stochastic gradient descent&lt;/code&gt; and you will gain a deep understanding of the juxtaposition of those three algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-01-05T19:47:52.763" LastActivityDate="2016-01-05T19:47:52.763" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9642" PostTypeId="2" ParentId="987" CreationDate="2016-01-06T04:20:10.207" Score="17" Body="&lt;p&gt;If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Perform dimensionality reduction (such as LSA via &lt;code&gt;TruncatedSVD&lt;/code&gt;) on your sparse data to make it dense and combine the features into a single dense matrix to train your model(s). &lt;/li&gt;&#xA;&lt;li&gt;Add your few dense features to your sparse matrix using something like scipy's &lt;code&gt;hstack&lt;/code&gt; into a single sparse matrix to train your model(s).&lt;/li&gt;&#xA;&lt;li&gt;Create a model using only your sparse text data and then combine its predictions (probabilities if it's classification) as a dense feature with your other dense features to create a model (ie: ensembling via stacking). If you go this route remember to only use CV predictions as features to train your model otherwise you'll likely overfit quite badly (you can make a quite class to do this all within a single &lt;code&gt;Pipeline&lt;/code&gt; if desired). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All three approaches are valid and have their own pros and cons. Personally, I find (1) to typically be the worst because it is, relatively speaking, extremely slow. I also find (3) to usually be the best, being both sufficiently fast and resulting in very good predictions. You can obviously do a combination of them as well if you're willing to do some more extensive ensembling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the algorithms you use, they can essentially all fit within that framework. Logistic regression performs surprisingly well most of the time, but others may do better depending on the problem at hand and how well you tune them. I'm partial to GBMs myself, but the bottom line is that you can try as many algorithms as you would like and even doing simple weighted ensembles of their predictions will almost always lead to a better overall solution.&lt;/p&gt;&#xA;" OwnerUserId="947" LastActivityDate="2016-01-06T04:20:10.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9644" PostTypeId="1" CreationDate="2016-01-06T08:39:14.273" Score="1" ViewCount="46" Body="&lt;p&gt;In the problem of association rules mining, the transaction database with $N$ transactions and $M$ items can be represented by a binary matrix $D$ with dimension $N \times M$. My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is $D$ low-rank or approximately low-rank? Equivalently, are there only few large singular values dominate the overall singular values of $D$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Intuitively, only few items on average are contained in each transaction because a customer generally buy only few items. However, is that true?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It will be much appreciated if someone familiar this field could answer my questions or even only provide any clues.  &lt;/p&gt;&#xA;" OwnerUserId="15211" LastActivityDate="2016-01-06T08:39:14.273" Title="Is the binary matrix of transaction database low-rank in the problem of association rules mining?" Tags="|data-mining|bigdata|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9648" PostTypeId="2" ParentId="9616" CreationDate="2016-01-06T12:16:03.760" Score="2" Body="&lt;p&gt;R, with &lt;a href=&quot;https://cran.r-project.org/web/packages/googleVis/vignettes/googleVis_examples.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;googlevis package&lt;/a&gt;, also provides great solution (interactive html map):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(datasets)&#xA;states &amp;lt;- data.frame(state.name, state.x77)&#xA;GeoStates &amp;lt;- gvisGeoChart(states, &quot;state.name&quot;, &quot;Illiteracy&quot;,&#xA;                          options=list(region=&quot;US&quot;, &#xA;                                       displayMode=&quot;regions&quot;, &#xA;                                       resolution=&quot;provinces&quot;,&#xA;                                       width=600, height=400))&#xA;plot(GeoStates)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WsaPX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WsaPX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15215" LastActivityDate="2016-01-06T12:16:03.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9649" PostTypeId="2" ParentId="4977" CreationDate="2016-01-06T12:52:19.833" Score="6" Body="&lt;p&gt;You might also want to take a look at &lt;a href=&quot;https://flink.apache.org/&quot;&gt;Apache Flink&lt;/a&gt;. It is a new contender to Apache Spark as it can do both, realtime and batch analysis. It also claims to be faster than Apache Spark, but I think mostly in depends on your usecase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good thing is: Apache Flink project has some Storm compatibility layer in its development version. In this way you can take your existing Storm project and simply run it inside Flink. For me it was at least 2 times faster than Storm in a very simple topology. So give it a try! :)&lt;/p&gt;&#xA;" OwnerUserId="15217" LastActivityDate="2016-01-06T12:52:19.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9650" PostTypeId="2" ParentId="9626" CreationDate="2016-01-06T13:04:21.167" Score="3" Body="&lt;p&gt;My answer is based on couple of assumptions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;user input is more or less standard, so there won't be &quot;Ex 20000&quot;&lt;/li&gt;&#xA;&lt;li&gt;you have at least majority of forms of input covered&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In every representative example of transaction description you would need to mark words of interest, be it name of holder and account number. You can start small with 10-20 examples for start, and then, when you have all required fields marked, you can train a sequence labelling model, or, say it, custom Named Entity Recognition model, which will parse new text and extract required data for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to train actual model is answered in that question: &lt;a href=&quot;https://datascience.stackexchange.com/questions/4980/help-regarding-ner-in-nltk&quot;&gt;Help regarding NER in NLTK&lt;/a&gt; , and how to mark up the data - is more of a question you should answer, since only you do know, what should be marked as account number, account holder etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would be digging into NER training, I would advise you not to rely only on current word features, but add some regexp-alike features and dictionaries as features, since task seems very limited in terms of context variation.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-01-06T13:04:21.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9651" PostTypeId="1" AcceptedAnswerId="9657" CreationDate="2016-01-06T14:21:05.260" Score="1" ViewCount="1402" Body="&lt;p&gt;Suppose I have such a JSON file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;  {&#xA;    &quot;id&quot;: &quot;0&quot;,&#xA;    &quot;name&quot;: &quot;name0&quot;, &#xA;    &quot;first_sent&quot;: &quot;date0&quot;,  &#xA;    &quot;analytics&quot;: [&#xA;        {&#xA;            &quot;a&quot;: 1,&#xA;            ...&#xA;        }, &#xA;        {&#xA;            &quot;a&quot;: 2, &#xA;            ...&#xA;        }&#xA;    ]&#xA;  }&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I want to parse it with Pandas. So I load it with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_json('file.son')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's all good until I try to access and count the number of dictionaries in the &quot;analytics&quot; field for each item, for which task I haven't found any better way than &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(df.shape[0]):&#xA;    num = len(df[i:i+1]['analytics'][i])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this looks totally non-elegant and it's missing the point of using Pandas in the first place. I need to be able to access the fields within &quot;analytics&quot; for each item.&#xA;The question is how to use Pandas to access fields within a field (which maps to a Series object), without reverting to non-Pandas approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A head of the DataFrame looks like this (only fields 'id' and 'analytics' reported):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    [{u'a': 0.0, u'b...&#xA;1    [{u'a': 0.01, u'b...&#xA;2    [{u'a': 0.4, u'b...&#xA;3    [{u'a': 0.2, u'b...&#xA;Name: analytics, dtype: object&#xA;0      '0'&#xA;1      '1'&#xA;2      '2'&#xA;3      '3'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first number is obviously the index, the string is the 'id', and it is clear that 'analytics' appears as a Series.&lt;/p&gt;&#xA;" OwnerUserId="982" LastEditorUserId="982" LastEditDate="2016-01-06T16:11:28.687" LastActivityDate="2020-08-03T15:11:50.693" Title="Pandas: access fields within field in a DataFrame" Tags="|python|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9652" PostTypeId="1" AcceptedAnswerId="9654" CreationDate="2016-01-06T15:07:08.563" Score="2" ViewCount="280" Body="&lt;p&gt;I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data analysis (plots - preferably interactive) and export custom reports in html or pdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering how people do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Store the data. For the moment I have plain text files (each file has rows of a fixed number of columns - most of the data are categorical). Would it make sense to save those in some database (SQL) or hdf5? Any hints on which is preferrable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Which plotting library would you propose for the graphs? I have seen about bookeh and matplotlib supports interactive widgets but I don't know what people normally use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Could I export the analysis results in an IPython notebook and then in html programmatically?&lt;/p&gt;&#xA;" OwnerUserId="14487" LastEditorUserId="14487" LastEditDate="2016-01-06T16:05:26.653" LastActivityDate="2016-01-06T16:30:57.220" Title="Writing custom data analysis program" Tags="|python|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9653" PostTypeId="2" ParentId="9616" CreationDate="2016-01-06T15:48:31.523" Score="5" Body="&lt;p&gt;There are &lt;a href=&quot;https://www.google.com/search?q=choropleth+map+r&quot; rel=&quot;noreferrer&quot;&gt;many ways&lt;/a&gt; to plot &lt;a href=&quot;https://en.wikipedia.org/wiki/Choropleth_map&quot; rel=&quot;noreferrer&quot;&gt;choropleth maps&lt;/a&gt; in R. Here's another one that uses the popular grammar of graphics library ggplot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(dplyr)&#xA;choro &amp;lt;- left_join(&#xA;  map_data(&quot;state&quot;), &#xA;  USArrests %&amp;gt;% &#xA;    add_rownames(&quot;region&quot;) %&amp;gt;% &#xA;    mutate(region=tolower(region))&#xA;)&#xA;ggplot(choro, aes(long, lat)) +&#xA;  geom_polygon(aes(group = group, fill = Assault)) + &#xA;  coord_quickmap()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zSKVa.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zSKVa.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-01-06T15:48:31.523" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9654" PostTypeId="2" ParentId="9652" CreationDate="2016-01-06T16:30:57.220" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Store the data. For the moment I have plain text files (each file&#xA;  has rows of a fixed number of columns - most of the data are&#xA;  categorical). Would it make sense to save those in some database (SQL)&#xA;  or hdf5? Any hints on which is preferable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes,&lt;/strong&gt; it would make sense to store in a local database, rather than using large csv/text files. As you say that the data is derived from a NoSQL source, I assume unstructured data. So, using a SQL/relational store is out of question. As you say you are using Python, I would suggest you use &lt;a href=&quot;http://tinydb.readthedocs.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;TinyDB&lt;/a&gt;, which is both light-weight and easy to handle.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2) Which plotting library would you propose for the graphs? I have&#xA;  seen about bookeh and matplotlib supports interactive widgets but I&#xA;  don't know what people normally use.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Matplotlib would be good enough. Actually, this question is more opinion-based than anything else. There are a lot of visualization libraries you can use, like Bokeh, Seaborn, etc.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;3) Could I export the analysis results in an IPython notebook and then&#xA;  in html programmatically?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, you can do the analytics directly in an Ipython notebook(Jupyter), which also supports Markdown and HTML cells.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, you can also use &lt;a href=&quot;https://github.com/ipython/ipywidgets&quot; rel=&quot;nofollow&quot;&gt;widgets and interactive visualization with Jupyter Ipy notebooks&lt;/a&gt; and Matplotlib.  &lt;a href=&quot;https://github.com/ipython/ipywidgets/blob/master/examples/notebooks/Index.ipynb&quot; rel=&quot;nofollow&quot;&gt;Tutorials for the same&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-06T16:30:57.220" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9655" PostTypeId="1" CreationDate="2016-01-06T20:18:26.413" Score="3" ViewCount="359" Body="&lt;p&gt;I read about Recursive Neural Networks that they can convert Documents to distributed word representation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of new article recommendation, I am thinking to use this model to convert all news articles to vectors and then recommend to a  particular user, news articles similar to ones he browsed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In vector space this will boil down to finding 'similar' vectors to a given vector(user's news read). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How likely is that this model will work well in practice? Any comments and/or suggestions?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="13518" LastEditDate="2016-01-07T05:17:14.370" LastActivityDate="2016-01-07T05:17:14.370" Title="Deep Learning for Recommender System" Tags="|recommender-system|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9656" PostTypeId="2" ParentId="9619" CreationDate="2016-01-06T20:57:55.773" Score="1" Body="&lt;p&gt;First of all I think that what you're trying to do is very difficult. A research paper success depends not only of the words but the math, the time when it was published, the journal, etc. You have many features you should take into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would try deep learning. For input, I would add all the features above and perhaps more. As the output I would use the number of citations or maybe something more sophisticated like a custom function of popularity. It's not the same to be published in Nature than in other journals.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-01-06T20:57:55.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9657" PostTypeId="2" ParentId="9651" CreationDate="2016-01-06T21:19:02.200" Score="2" Body="&lt;p&gt;Multi-indexing &lt;em&gt;might&lt;/em&gt; be helpful. See &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/advanced.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But the below was the immediate solution that came to mind. I think it's a little more elegant than what you came up with (fewer obscure numbers, more interpretable natural language):&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.read_json('test_file.json')&#xA;df = df.append(df) # just to give us an extra row to loop through below&#xA;df.reset_index(inplace=True) # not really important except to distinguish your rows&#xA;for _ , row in df.iterrows():&#xA;    currNumbDict = len(row['analytics'])&#xA;    print(currNumbDict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14715" LastEditorUserId="98307" LastEditDate="2020-08-03T15:11:50.693" LastActivityDate="2020-08-03T15:11:50.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9658" PostTypeId="2" ParentId="6513" CreationDate="2016-01-06T21:30:26.947" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;=run a maximum of n tests or &lt;strong&gt;stop earlier if a good improvement is found&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Don't do that. Don't stop before n tests have been run if you like what you've seen so far....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.evanmiller.org/how-not-to-run-an-ab-test.html&quot; rel=&quot;nofollow&quot;&gt;http://www.evanmiller.org/how-not-to-run-an-ab-test.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, however, figuring out how big an N you need given an expected effect size and alpha level is the domain of power analysis. Read more here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.uta.edu/faculty/sawasthi/Statistics/stpowan.html&quot; rel=&quot;nofollow&quot;&gt;http://www.uta.edu/faculty/sawasthi/Statistics/stpowan.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14715" LastActivityDate="2016-01-06T21:30:26.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9659" PostTypeId="2" ParentId="9655" CreationDate="2016-01-06T22:45:56.380" Score="2" Body="&lt;p&gt;There are many ways to calculate similarity between articles. I have not seen anybody doing a vector conversion and running comparisons. However, there is a text mining strategy called &quot;Term-Frequency / Inverse-Document-Frequency&quot; which is a clever way to find unique words and phrases in documents. You can run this on multiple documents, and compare the extracted keywords to match them for recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out my ebook for more details: &lt;a href=&quot;https://lizrush.gitbooks.io/algorithms-for-webdevs-ebook/content/chapters/tf-idf.html&quot; rel=&quot;nofollow&quot;&gt;https://lizrush.gitbooks.io/algorithms-for-webdevs-ebook/content/chapters/tf-idf.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to leverage this technique on web documents (like a blog), there is a free service to do so: &lt;a href=&quot;https://algorithmia.com/recommends&quot; rel=&quot;nofollow&quot;&gt;https://algorithmia.com/recommends&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3466" LastActivityDate="2016-01-06T22:45:56.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9660" PostTypeId="2" ParentId="9619" CreationDate="2016-01-06T23:17:43.363" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://appliedpredictivemodeling.com/&quot; rel=&quot;nofollow&quot;&gt;Applied Predictive Modeling book&lt;/a&gt; has a &lt;a href=&quot;https://www.kaggle.com/c/unimelb&quot; rel=&quot;nofollow&quot;&gt;case study of acceptance rate of grant proposals, which was a Kaggle competition&lt;/a&gt;. You can get many good ideas from there. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;from the dataset, what kinds of features you would look at&lt;/li&gt;&#xA;&lt;li&gt;from the book, step-by-step explanations of how to build models&lt;/li&gt;&#xA;&lt;li&gt;from the Kaggle competition, approaches by the contestants&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think it is very hard to predict the quality of the paper from the text of the paper, but you might have much more luck by including features such as academic rank of the author(s), their past publishing history and success, acceptance rate of the journals the paper was submitted to, and things such as even the length of the paper. Such proxy features for paper quality actually might be very successful... &lt;/p&gt;&#xA;" OwnerUserId="14598" LastActivityDate="2016-01-06T23:17:43.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9661" PostTypeId="2" ParentId="9655" CreationDate="2016-01-07T01:31:41.430" Score="1" Body="&lt;p&gt;You've just described the matrix factorization model, which works well. In fact, it even works without explicit features; i.e., any properties of the items. But you can introduce prior information by augmenting the feature vectors, e.g., with the document embedding, as you describe. I do not see any reason why it should not work, esp. since I've seen it done.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-07T01:31:41.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9662" PostTypeId="1" AcceptedAnswerId="10174" CreationDate="2016-01-07T02:33:51.253" Score="4" ViewCount="469" Body="&lt;p&gt;In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we dropped the &quot;sql&quot; requirement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my best option Hive and as many servers I can't get my hands on? Would spark be of any benefit since this size of dataset wouldn't come close to fitting into memory? I have other related questions /ideas but don't want to bog this down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Background&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is originally stored in a binary format that gets converted into an ASCII. The ASCII file is getting turned into CSV (actually tab separated but whatever). 1 binary file = 1 ASCII file = 1 CSV file. The data format is pretty simple, 200 to 500 header columns, each row is a sample of an attribute in 1-second intervals.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example (first row is header):&#xA;t,attribute1,attribute2...&amp;lt;attributeX&amp;gt;&#xA;1,val1,val2...&amp;lt;valx&amp;gt;&#xA;2,val1,val2...&amp;lt;valx&amp;gt;&#xA;3,val1,val2...&amp;lt;valx&amp;gt;&#xA;...&#xA;X,val1,val2...&amp;lt;valx&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example queries that could be ran. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Take t value (t is non-unique, as each file starts at 1 and ends at some random time) and compare different attribute values between t value of 100-1000 for all other attributes that were sampled at 100-1000. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)There is UTC value as an attribute value (one of the columns), and same idea as #1 we might want to compare all attributes between X dates and compare those attributes to Y data that is from a different UTC range. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3)We might want to figure out a single particular attribute and trend that from the earliest time we saw that attribute for a different particular attribute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are my best examples as provided to me but other types of queries could exist. If you do have a question of why don't we cut out the middle man of the conversions of the binary data to CSV data, we could theoretically. However, the program that converts the files is quite old and is very involved. &lt;/p&gt;&#xA;" OwnerUserId="14829" LastEditorUserId="14829" LastEditDate="2016-01-09T23:15:21.227" LastActivityDate="2016-02-10T12:06:23.797" Title="Tools to perform SQL analytics on 350TB of csv data" Tags="|bigdata|dataset|data-wrangling|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9663" PostTypeId="1" CreationDate="2016-01-07T04:28:45.337" Score="8" ViewCount="10761" Body="&lt;p&gt;I have categorized 800,000 documents into 500 categories using the Mahout topic modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name for the group using any existing algorithm.&#xA;For the time being, I have used the following algorithm to arrive at the name for the topic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each topic&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Take all the documents belonging to the topic (using the document-topic distribution output)&lt;/li&gt;&#xA;&lt;li&gt;Run python nltk to get the noun phrases&lt;/li&gt;&#xA;&lt;li&gt;Create the TF file from the output&lt;/li&gt;&#xA;&lt;li&gt;name for the topic is the phrase (limited towards max 5 words)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please suggest a approach to arrive at more relevant name for the topics.&lt;/p&gt;&#xA;" OwnerUserId="15192" LastEditorDisplayName="user10169" LastEditDate="2016-01-07T11:25:15.600" LastActivityDate="2020-08-06T02:36:24.913" Title="How to give name to topics created using LDA?" Tags="|machine-learning|data-mining|nlp|text-mining|topic-model|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9664" PostTypeId="1" CreationDate="2016-01-07T06:01:43.687" Score="2" ViewCount="2258" Body="&lt;p&gt;In all articles I've read say &quot;K-means is not converge with non-convex function&quot;.I just know that convex function is the function that able to find the global minimum. However , I do not understand how this relate to K-Means algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to ask&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Which situation cause the non-convex problem ? (If possible please visualize it and show the equation)&lt;/li&gt;&#xA;&lt;li&gt;In some articles say that the bad initialization( the poor seeds) cause the not-convex problem, How are they related?&lt;/li&gt;&#xA;&lt;li&gt;I'm not quite sure the referred function is subject to which variables.Are that clusters and their centroid ? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Ok , It's my misunderstanding. K-Means is not converged with non-convex shape data set .Nothing relate with convex function,sorry.&lt;/p&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="21" LastEditDate="2017-12-18T05:13:48.790" LastActivityDate="2017-12-18T05:13:48.790" Title="The problem of K-Means with non-convex function" Tags="|machine-learning|data-mining|clustering|k-means|" AnswerCount="0" CommentCount="3" ClosedDate="2016-02-07T08:52:12.807" ContentLicense="CC BY-SA 3.0" />
  <row Id="9667" PostTypeId="1" CreationDate="2016-01-07T08:32:42.970" Score="13" ViewCount="720" Body="&lt;p&gt;I need help on what should be my next step in an algorithm I am designing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to NDAs, I can't disclose much, but I'll try to be generic and understandable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, after several steps in the algorithms, I have this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each customer that I have, and events that they do during a month, during first steps I have clustered the events into several categories (each customer will have the events separated into categories that go from 1 to x being x between 1 to 25, generally the first categories have more density of events than the others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each category and customer I have created a time series aggregating the events of the month per hour (getting patterns of when these events are being done). Also I am using a couple of normalizing variables based on the number of days over a month (30 days) that the guy performs at least one event, and the number of days with at least one event over the total of days with at least one event (aggregating all clusters). The first one gives me a ratio of how active the customer is during the month, and the second one weights the category against the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final table looks like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|*Identifier*|  *firstCat* | *feature1* | *feature2*  |   {      *(TIME SERIES)*   }&#xA;&#xA;CustomerID  |  ClusterID |  DaysOver30 | DaysOverTotal | Events9AM Events10AM ... &#xA;&#xA; xx | 1 | 0,69 |  0,72 |  0,2   0,13   ...&#xA;&#xA; xx | 2 | 0,11 |  0,28 |  0,1   0,45   ...&#xA;&#xA; xy | 1 | 0,23 |  0,88 |  0,00  0,60   ...&#xA;&#xA; xy | 2 | 0,11 |  0,08 |  1,00  0,00   ...&#xA;&#xA; xy | 3 | 0,10 |  0,04 |  0,40  0,60   ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The time series variables are the percentage over the total of events per day on each specific category (this means that per each row adding up all variables should be 1). The reason of doing it like that is because for example a time series with events &lt;code&gt;0 0 0 1 0&lt;/code&gt; and &lt;code&gt;1 1 1 2 1&lt;/code&gt; are completely different, and standardizing to normal would give similar results. And due to high skew between different categories, I check the values on the time series independently with the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need to do now is to identify these categories (remember, they can be from 1 to x being x any number from 1 to 25) into 3 tags: tag A, tag B and None of Them. Looking at these variables I can manually identify which tag they belong to, and the idea is to identify manually as much as I can and use any classifier algorithm to learn from that and identify all of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea was to use multiple logistic regressions on the table, but all the variables of the time series are correlated (since they are a linear combination of each other), so I thought I better use a clustering algorithm only over the time series using euclidean distance to categorize the different patterns and use the result and the other two normalizing variables in the logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other concern that I have is that this approach takes each row independently from the others, and in theory, for each customer there should be only 0 or 1 tag A, 0 or 1 tag B and the remaining of them should be None (another tip is that normally Tag A and B are between first categories, because is highly dependent on the normalizing features (if days over total is High, there is a high possibility that the row is either A or B, depending on the Time Series Pattern). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: This is no longer a concern, I will just perform two different logistic regressions, one for Tag A or Other and another for Tag B or another, with the result probabilities I can select only the best of each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is enormous and the final algorithm needs to be applied using SQL (on Teradata), but for getting the coefficients of the logistic regression, or the centers of the clustering I get a sample and use R.&lt;/p&gt;&#xA;" OwnerUserId="15230" LastEditorUserId="29575" LastEditDate="2017-12-18T14:29:31.870" LastActivityDate="2017-12-18T14:29:31.870" Title="Classify Customers based on 2 features AND a Time series of events" Tags="|classification|clustering|time-series|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9668" PostTypeId="1" CreationDate="2016-01-07T08:51:03.393" Score="4" ViewCount="3711" Body="&lt;p&gt;I have a classification problem with 60 data points in a 2-dimensional feature space. The data originally is divided into 2 classes. Earlier I was using Statistics Toolbox of Matlab so it was giving me fairly good results.It was giving 1 false negative and no false positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SVMstruct = svmtrain(point(1:60,:),T(1:60),'Kernel_Function','polynomial','polyorder',11,'Showplot',true);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using a polynomial kernel and with polynomial order of 11. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XSWKG.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XSWKG.jpg&quot; alt=&quot;Boundries Generated after training data using svmTrain from Matlab&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I use same kernel configuration in scikit-learn SVC it does not gives the same result rather it gives very undesirable result with classifying all of them to single class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ftFJj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ftFJj.png&quot; alt=&quot;Boundries Generated after training data using SVC from Scikit-learn in python&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using it as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svc = svm.SVC(kernel='poly', degree=11, C=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have used with many values of C too. No major difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why there is so much difference in results ?&#xA;How can I get same result as I got using Matlab ?&#xA;For me it is compulsory to do with python-scikit.&lt;/p&gt;&#xA;" OwnerUserId="14789" LastEditorUserId="11097" LastEditDate="2016-01-07T09:45:49.693" LastActivityDate="2016-01-13T06:02:40.970" Title="Same SVM configuration, same input data gives different output using Matlab and scikit-learn implementation of SVM, in a classification problem" Tags="|classification|svm|scikit-learn|matlab|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9669" PostTypeId="2" ParentId="9663" CreationDate="2016-01-07T10:03:23.543" Score="2" Body="&lt;p&gt;If you don't want to dig into much NLP in that task, I suggest you to generate a set of most frequent NGrams (of lengths 2-5) from your documents and find the most distinct ngrams for each category using TF*IDF metric as sense importance of a particular ngram (normalizing measure by word count) and selecting those Ngrams that are used in a particular category and are not (or rarely) used in others.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-01-07T10:03:23.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9670" PostTypeId="1" AcceptedAnswerId="9671" CreationDate="2016-01-07T10:04:14.607" Score="3" ViewCount="455" Body="&lt;p&gt;I need to pull the names of companies out of resumes. Thousands of them. I was thinking of using NLTK to create a list of possible companies, and then cross-referencing the list of strings with something like SEC.gov.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already been able to successfully pull the candidate's name, and contact info off of the resumes with some RegEx, but this one has me quite stumped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm thinking is that I could use NLTK to create a list of strings of proper nouns from the resume's, and then search SEC.gov, or some other database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a link to the SEC page I would be searching: SEC company search page&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Read Resume1&#xA;Get all potential company names as list of strings potentialCompanies&#xA;    IF searching for string1 in SEC gets result, THEN add to candidateCompanies&#xA;        ELSE remove from potentialCompanies, go to next string&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My Questions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To people that have used NLTK, would there be a better way of getting the potential companies from the text besides using proper nouns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would there be a better place to search for companies than the SEC site?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have never done any web scraping before, and don't really know where to start if it is needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I had posted this on Stack Overflow but they told me that it might be better suited for here...)&lt;/p&gt;&#xA;" OwnerUserId="15233" LastEditorUserId="11097" LastEditDate="2016-01-07T10:06:46.510" LastActivityDate="2016-01-07T10:14:52.510" Title="Python: validating the existence of NLTK data with database search" Tags="|python|nlp|nltk|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9671" PostTypeId="2" ParentId="9670" CreationDate="2016-01-07T10:14:52.510" Score="2" Body="&lt;p&gt;NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow&quot;&gt;NLTK book&lt;/a&gt; (look for section &quot;5 Named Entity Recognition&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if your input text has organizations in a very specific context that wasn't seen by NLTK NER model, performance might be quite low. In that case you should be looking into training your own NER model, what would extract company names. For that you would require to manually markup a small amount of your dataset.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-01-07T10:14:52.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9672" PostTypeId="1" AcceptedAnswerId="10185" CreationDate="2016-01-07T10:26:48.333" Score="5" ViewCount="1840" Body="&lt;p&gt;I've just read &lt;a href=&quot;http://papers.nips.cc/paper/207-the-cascade-correlation-learning-architecture.pdf&quot; rel=&quot;nofollow&quot;&gt;The Cascade-Correlation Learning Architecture&lt;/a&gt; by Scott E. Fahlman and Christian Lebiere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I've got the overall concept (or at least the &quot;cascade&quot; part - a &lt;a href=&quot;https://www.youtube.com/watch?v=1E3XZr-bzZ4&quot; rel=&quot;nofollow&quot;&gt;4min YouTube video&lt;/a&gt; how I think it works):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Start with a minimal network with input and output units only&lt;/li&gt;&#xA;&lt;li&gt;Learn those weights with standard algorithms (e.g. gradient descent - they seem to use another training objective which I don't quite understand, so it is gradient ascent in the paper)&lt;/li&gt;&#xA;&lt;li&gt;When the network doesn't improve, add a single new hidden unit. This unit gets input from all input nodes and all hidden nodes which were added before. Its output goes to all output nodes only.&lt;/li&gt;&#xA;&lt;li&gt;Repeat step 3&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, I don't understand the details of step 3: The input weights to hidden units are frozen (indicated by boxes in the paper). When exactly do they get frozen? Are they just initialized by random and never learned at all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also don't understand this paragraph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To create a new hidden unit, we begin with a candidate unit that receives&#xA;  trainable input connections from all of the network's external inputs and from&#xA;  all pre-existing hidden units. The output of this candidate unit is not yet&#xA;  connected to the active network. We run a number of passes over the examples of&#xA;  the training set, adjusting the candidate unit's input weights after each pass.&#xA;  The goal of this adjustment is to maximize $S$, the sum over all output units $o$&#xA;  of the magnitude of the correlation (or, more precisely, the covariance)&#xA;  between $V$, the candidate unit's value, and Eo, the residual output error&#xA;  observed at unit o. We define S as&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$$S = \sum_{o} | \sum_p (V_p - \bar V) (E_{p,o} - \bar{E_o}) |$$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where $o$ is the network output at which the error is measured and p is the&#xA;  training pattern. The quantities $\bar V$ and $\bar{E_o}$ are the values of&#xA;  $V$ and $E_o$ averaged over all patterns.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is an &quot;residual output error&quot;? Is $V_p$ simply the activation of the unit given the pattern $p$? What does the term $S$ mean and why do we want to maximize it?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-07T19:44:29.623" LastActivityDate="2018-09-15T00:13:20.923" Title="How exactly does adding a new unit work in Cascade Correlation?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9673" PostTypeId="5" CreationDate="2016-01-07T11:23:52.943" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-01-07T11:23:52.943" LastActivityDate="2016-01-07T11:23:52.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9674" PostTypeId="4" CreationDate="2016-01-07T11:23:52.943" Score="0" Body="Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward." OwnerUserId="15197" LastEditorUserId="15197" LastEditDate="2016-01-08T19:26:58.623" LastActivityDate="2016-01-08T19:26:58.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9675" PostTypeId="5" CreationDate="2016-01-07T11:24:42.380" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-01-07T11:24:42.380" LastActivityDate="2016-01-07T11:24:42.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9676" PostTypeId="4" CreationDate="2016-01-07T11:24:42.380" Score="0" Body="A model-free reinforcement learning technique." OwnerUserId="15197" LastEditorUserId="15197" LastEditDate="2016-01-08T19:26:45.357" LastActivityDate="2016-01-08T19:26:45.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9677" PostTypeId="1" CreationDate="2016-01-07T11:42:34.537" Score="1" ViewCount="632" Body="&lt;p&gt;I'm interested in various machine learning methods and wonder how can I aim for a career in the field.&#xA;What are most basic skills to have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean the matters strictly connected to the machine learning - not programming or libraries.&lt;/p&gt;&#xA;" OwnerUserId="15197" LastEditorUserId="15197" LastEditDate="2016-01-07T14:42:28.833" LastActivityDate="2016-01-07T14:42:28.833" Title="How can one can aim for a career in machine learning?" Tags="|machine-learning|beginner|career|" AnswerCount="2" CommentCount="5" ClosedDate="2016-01-09T09:03:16.377" ContentLicense="CC BY-SA 3.0" />
  <row Id="9678" PostTypeId="2" ParentId="9677" CreationDate="2016-01-07T12:04:25.113" Score="4" Body="&lt;p&gt;Machine Learning is a really big field. Depending on what exactly you want to do, there might be huge differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, the following skills are helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Programming, especially Python / C++&lt;/li&gt;&#xA;&lt;li&gt;Frameworks like &lt;a href=&quot;http://tensorflow.org&quot; rel=&quot;nofollow&quot;&gt;TensorFlow&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;sklearn&lt;/a&gt;, &lt;a href=&quot;http://torch.ch/&quot; rel=&quot;nofollow&quot;&gt;Torch&lt;/a&gt;, ...&lt;/li&gt;&#xA;&lt;li&gt;Algorithms like neural networks (especially gradient descent), SVM, decision trees, clustering algorithms, Q-Learning, ...&lt;/li&gt;&#xA;&lt;li&gt;Mathematics (e.g. probability theory / statistics, linear algebra, analysis for calculating gradients&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Knowing how to program the GPU directly (CUDA) is probably not always necessary, but certainly a big bonus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you're interested what you learn at KIT (Karlsruhe, Germany) in an introductory machine learning course, see &lt;a href=&quot;http://martin-thoma.com/machine-learning-1-course/&quot; rel=&quot;nofollow&quot;&gt;my blog post for exam preparation&lt;/a&gt; (I'm sorry, it is mostly in German).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Online Courses&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There are lots of online material for machine learning. For example an &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;Coursera ML course&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-07T12:21:17.540" LastActivityDate="2016-01-07T12:21:17.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9679" PostTypeId="2" ParentId="9677" CreationDate="2016-01-07T14:38:02.570" Score="2" Body="&lt;p&gt;I have already answered similar questions &lt;a href=&quot;https://datascience.stackexchange.com/a/9439/11097&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/8965/11097&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, that would help you get an idea of how you need to design your own learning path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why I said that this question is opinion-based is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data Science is a huge domain in itself, and attaining complete&#xA;  knowledge of everything in it, is close to impossible.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;So, if I am interested in pursuing the concept of Neural Networks&#xA;  alone, and subsequently Deep Learning; then my learning path would be&#xA;  completely different to someone who wants to learn a different concept like time series.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;And so does that of someone who wants to learn some other concept. The&#xA;  concepts might be intertwined, but something which I absolutely&#xA;  require in my learning path might not be necessary to someone else's.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, those above resources are a rough guide to get started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to that, here is a list of &lt;a href=&quot;https://datascience.stackexchange.com/q/8941/11097&quot;&gt;conferences&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/q/9000/11097&quot;&gt;podcasts&lt;/a&gt; of Data Science, which help you with your learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And also keep an eye on the &lt;a href=&quot;/questions/tagged/beginner&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;beginner&amp;#39;&quot; rel=&quot;tag&quot;&gt;beginner&lt;/a&gt; tag here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional reading:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/174140/84191&quot;&gt;Applying machine learning in a real world example&lt;/a&gt;&#xA;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;A really good neural networks and deep learning book&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-01-07T14:38:02.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9680" PostTypeId="1" AcceptedAnswerId="9706" CreationDate="2016-01-07T15:11:21.470" Score="3" ViewCount="605" Body="&lt;p&gt;I am looking for a method to approximate how similar a test set (i.e., test set  features) to a train set. For example, something like, for each row in test: is there a similar enough data point in train? I've been thinking about using a mixture model approach, but I haven't been able to find a good reference on this. Can anyone suggest a good approach, or provide good references for how to use mixture models for this application?&lt;/p&gt;&#xA;" OwnerUserId="14946" LastEditorUserId="13413" LastEditDate="2016-01-07T18:05:58.310" LastActivityDate="2016-01-08T22:18:16.683" Title="Approximating density of test set in train" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9681" PostTypeId="1" AcceptedAnswerId="9682" CreationDate="2016-01-07T15:31:37.570" Score="7" ViewCount="13812" Body="&lt;p&gt;RStudio automatically recognizes headers in an R script that are set via comments:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LnGac.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LnGac.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would like to exploit that feature, but I don't quite understand what the rules are for RStudio to recognize them as headers. Can someone explain?&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="14909" LastEditDate="2018-02-06T09:39:56.553" LastActivityDate="2020-08-07T09:18:53.697" Title="Rules by which RStudio sets Headings" Tags="|r|rstudio|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9682" PostTypeId="2" ParentId="9681" CreationDate="2016-01-07T16:35:46.120" Score="10" Body="&lt;p&gt;Check out &lt;a href=&quot;https://support.rstudio.com/hc/en-us/articles/200484568-Code-Folding-and-Sections&quot;&gt;Code Folding and Sections&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Code sections allow you to break a larger source file into a set of&#xA;  discrete regions for easy navigation between them&lt;/strong&gt;. Code sections are&#xA;  automatically foldable—for example, the following source file has&#xA;  three sections (one expanded and the other two folded):&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;To insert a new code section you can use the Code -&gt; Insert Section&#xA;  command. Alternatively, any comment line which includes at least four&#xA;  trailing dashes (-), equal signs (=), or pound signs (#) automatically&#xA;  creates a code section. For example, all of the following lines create&#xA;  code sections:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;# Section One ---------------------------------&lt;br&gt;&#xA;   # Section Two =================================&lt;br&gt;&#xA;   ### Section Three ############################# &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Note that as illustrated above the line can start with any number of&#xA;  pound signs (#) so long as &lt;strong&gt;it ends with four or more -, =, or #&#xA;  characters&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(highlights by myself)&lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-01-07T16:35:46.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9683" PostTypeId="1" AcceptedAnswerId="9694" CreationDate="2016-01-07T16:49:38.773" Score="2" ViewCount="380" Body="&lt;p&gt;Suppose A's possible values are ON or OFF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I represent it as: if A ON then feature f=1 else f=0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, suppose I represent it with 2 features, where: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-if A is ON then f1=1 and f2=0 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-if A is OFF then f1=0 and f2=1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How this kind of representation affects neural networks?&lt;/p&gt;&#xA;" OwnerUserId="15244" LastEditorUserId="11097" LastEditDate="2016-01-07T17:22:27.467" LastActivityDate="2016-01-08T05:57:34.127" Title="How data representation affects neural networks?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9684" PostTypeId="2" ParentId="9663" CreationDate="2016-01-07T17:32:32.017" Score="7" Body="&lt;p&gt;I can suggest several papers on this topic:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automatic Labelling of Topic Models&lt;/li&gt;&#xA;&lt;li&gt;Automatic Labeling Hierarchical Topics&lt;/li&gt;&#xA;&lt;li&gt;Representing Topics Labels for Exploring Digital Libraries&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find more by looking at their citations.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-07T17:32:32.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9685" PostTypeId="2" ParentId="9668" CreationDate="2016-01-07T17:51:11.043" Score="3" Body="&lt;p&gt;You have to be sure that the algorithm is the same and the kernel functions are really the same. &#xA;If you look at this python documentation page for &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#svm-kernels&quot; rel=&quot;nofollow&quot;&gt;kernels in scikit learn&lt;/a&gt; you will see there a description of poly kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that you have a gamma and a degree. Gamma is by default 'auto' which is evaluated at 1/n_samples. For the same kernel you have 'coef0' (a great name for a parameter), which is used in poly as the free term. I do not know how matlab put this values as defaults, but the usual formula for poly kernel in literature I found it to be $poly(x_1, x_2) = (&amp;lt;x_1,x_2&amp;gt; + 1)^{d}$. So no gamma and the free term is $1$. I think matlab uses that. (Anyway I found the 'improvements' in scikit learn to have a not so good smell).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in this &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;SVC documentation page&lt;/a&gt; they state that there is a parameter called shrinking. I really do not know it's effect, but its auto, which means is enabled. Might be an issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Later edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this documentation page for &lt;a href=&quot;http://uk.mathworks.com/discovery/support-vector-machine.html&quot; rel=&quot;nofollow&quot;&gt;svm in matlab&lt;/a&gt; which describes the kernel in the way I stated (no degree, free term with $1$). Also it states that 'SMO' is used by default, make it sure you use 'SMO' in python also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand you have to understand that these kind of algorithms are solved by optimization methods which are usually iterative and to save some memory, or cycles their implementations can be different in small details, which will almost produce different results. I agree however that the results should be similar.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2016-01-07T17:57:44.077" LastActivityDate="2016-01-07T17:57:44.077" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9686" PostTypeId="1" AcceptedAnswerId="9687" CreationDate="2016-01-07T19:11:01.153" Score="3" ViewCount="5328" Body="&lt;p&gt;So I'm a recent transfer 2nd year student from Computer Science major to Mathematics major. Though I do have a bit of an issue here. I can choose between the applied mathematics, pure mathematics and statistics concentrations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along with this major, I'm doing a minor in Data Science with courses focused on Economics and Statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the future, I'm interested in doing a master's degree in Data Science and a career in data analysis for businesses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I know any degree can be used to get into graduate school, I still want to which would be most beneficial for education in the future as well as opportunities for internships, research, and jobs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="15253" LastActivityDate="2016-01-17T19:41:13.670" Title="Mathematics major for data science" Tags="|data|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9687" PostTypeId="2" ParentId="9686" CreationDate="2016-01-07T19:36:58.410" Score="7" Body="&lt;p&gt;If you know that you want to become a data scientist, you can pretty much rule out pure mathematics.  Note, I'm not saying that pure mathematicians cannot become data scientists, but it's not the most natural transition.  Between the other two branches, stats is probably the most natural path.  Both will having you think about applying math to answer real world problems, but stats is very specifically geared towards focusing on larger scale data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;@rocinante mentions the need for software skills, and suggests CS over econ as a minor.  I would say this really depends; if you are in a bigger data science team, you'll probably work alongside dedicated programers, who will be able to implement your analytics more efficiently than you would be expected to as an analyst.  If you know that you want to apply D.S. techniques to either finance or business, domain specific knowledge is helpful.  Additionally, if you know that it is your desire to end up in data science, you'll be able to look for ways to use programing along the way, and keep those skills sharp.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastEditorUserId="8041" LastEditDate="2016-01-17T19:41:13.670" LastActivityDate="2016-01-17T19:41:13.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9688" PostTypeId="2" ParentId="9686" CreationDate="2016-01-07T20:21:59.707" Score="2" Body="&lt;p&gt;I agree with Gartner about the importance of statics. However, if your data science minor courses focus on economics, I would re-evaluate going through with it. Instead, I would switch to a computer science minor, or use up your electives to take programming courses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a person who did applied math and stats, I felt that my programming knowledge was really deficient when it came to handling real life huge, messy data sets. All my program required was two Java courses, so I ended up having to spend an inordinate amount of time learning more about scripting, useful programming languages, etc.&lt;/p&gt;&#xA;" OwnerUserId="6529" LastActivityDate="2016-01-07T20:21:59.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9689" PostTypeId="2" ParentId="9686" CreationDate="2016-01-07T21:39:33.300" Score="1" Body="&lt;p&gt;I recently answered in &lt;a href=&quot;https://math.stackexchange.com/questions/1592421/why-does-topology-rarely-come-up-outside-of-topology/1592463#1592463&quot;&gt;Why does topology rarely come up outside of topology?&lt;/a&gt; that pure mathematics (topology) can be useful in data science. I believe that mixed skills are required for data science: programming (computer science, algorithmics), practicing (statistics, signal processing), theorizing (logic, mathematics). You can get a &lt;a href=&quot;https://bringdata.wordpress.com/2013/11/14/data-science-metromap/&quot; rel=&quot;nofollow noreferrer&quot;&gt;metromap of data science skills&lt;/a&gt;, which are numerous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://bringdata.files.wordpress.com/2013/11/roadtodatascientist1-1024x831.png?w=512&amp;amp;h=415&quot; rel=&quot;nofollow noreferrer&quot;&gt;metro map of data scientist skills&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that it can be better to learn first what is more difficult, and what you cannot easily learn by yourself or with online lecture. A little bit of math seems interesting, it can give you (if properly chosen) fuel for the future, by learning abstractions you rarely learn elsewhere.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2016-01-07T21:39:33.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9690" PostTypeId="2" ParentId="9686" CreationDate="2016-01-07T21:46:15.240" Score="3" Body="&lt;p&gt;If you are considering having data science career you should start developing background on statistical theory, software engineering principles, database, data structure, algorithms development and statistical computing.&lt;/p&gt;&#xA;" OwnerUserId="14416" LastActivityDate="2016-01-07T21:46:15.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9691" PostTypeId="2" ParentId="9572" CreationDate="2016-01-07T21:47:52.183" Score="1" Body="&lt;p&gt;For a testing package similar to Hypothesis and based on Haskell's quickcheck, there's the R package from Revolution Analytics called &lt;a href=&quot;https://github.com/RevolutionAnalytics/quickcheck&quot; rel=&quot;nofollow&quot;&gt;quickcheck&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2666" LastActivityDate="2016-01-07T21:47:52.183" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9692" PostTypeId="1" CreationDate="2016-01-07T22:39:43.080" Score="-2" ViewCount="1604" Body="&lt;p&gt;I am thinking of creating a data science company (team) that aims to make money by winning Kaggle competitions. It will be actually a onsite data science learning service but I want to create the company's brand identity like a sport team instead of as a &quot;school.&quot; Have you ever heard of such an idea?&lt;/p&gt;&#xA;" OwnerUserId="15258" LastActivityDate="2016-01-08T00:44:24.087" Title="create a company to make money by winning Kaggle competition" Tags="|data|kaggle|" AnswerCount="1" CommentCount="6" ClosedDate="2016-01-09T09:04:13.350" ContentLicense="CC BY-SA 3.0" />
  <row Id="9693" PostTypeId="2" ParentId="9692" CreationDate="2016-01-08T00:44:24.087" Score="1" Body="&lt;p&gt;To me, this is a crazy idea because there is absolutely no guarantee your &quot;team&quot; can win anything. You'll need to place yourself like in the top-three, but remember you don't have access to the private leaderboard until the end of the competition. Furthermore, the prize money is minimal. The prize money is designed to offer to an individual, not enough for you to make a living.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-01-08T00:44:24.087" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9694" PostTypeId="2" ParentId="9683" CreationDate="2016-01-08T01:23:37.253" Score="4" Body="&lt;p&gt;&lt;strong&gt;It will have very little effect&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer most will give is that it will have no effect, but adding one more feature will decrease the ratio of records to features so will slightly increase the bias and will hence make your model slightly less accurate.  Unless, of course, you have overfit your model , in which case it will make your model slightly more accurate (a good data scientist would never do this because they understand the importance of cross-validation :-).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you normalize your data and then attempt some sort of dimensionality reduction, your algorithm will immediately eliminate the feature that you added since it is perfectly negatively (linearly) correlated with the first feature.  &lt;strong&gt;In this case it will have no effect&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Please also consider the following:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always see big red flags when someone asks a very fundamental data science question with the words &lt;code&gt;neural network&lt;/code&gt; included.  Neural networks are very powerful and receive a great deal of attention in the media and on Kaggle, but they take more data to train, are difficult to configure, and require much more computing power.  If you are just starting out, I suggest getting a foundation in linear regression, logistic regression, clustering, SVMs, decision trees, random forests, and naive Bayes before delving into artificial neural networks.  Just some food for thought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-01-08T05:57:34.127" LastActivityDate="2016-01-08T05:57:34.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9696" PostTypeId="1" CreationDate="2016-01-08T04:26:22.997" Score="3" ViewCount="502" Body="&lt;p&gt;If I get some posts on reddict.com, how can I predict whether this post will (trending/hot/popular) in the future or not? I would like to use the hidden markov model to predict it, but I don`t know how to define the hidden states and observation sequence...can anyone give me any suggestion? Thanks~&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/io9IX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/io9IX.png&quot; alt=&quot;I only have these time series data, such asu comments..&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15261" LastActivityDate="2016-10-13T16:01:29.120" Title="How can I predict the post popularity of reddit.com with hidden markov model(HMM)?" Tags="|markov|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9698" PostTypeId="2" ParentId="9335" CreationDate="2016-01-08T10:45:16.107" Score="1" Body="&lt;p&gt;I believe it will help if you add the exact regex anchors to the pattern you are trying to match. In your case adding &lt;code&gt;&quot;^I like football$&quot;&lt;/code&gt; as the pattern you are trying to match will look for strings starting and ending exactly like that. That way you have:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;grepl(&quot;^I like football$&quot;, &quot;I like football, W Lazio&quot;)&#xA;[1] FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will eliminate at least the problem you are facing now.&lt;br&gt;&#xA; I am not sure if I understand the rest of the question though...as I get it, you have some &lt;em&gt;other&lt;/em&gt; character vector called &lt;code&gt;text&lt;/code&gt; for example, and inside you are trying to look for elements of the vector &lt;code&gt;pattern&lt;/code&gt; and, if matched, replace them with the corresponding vector element from &lt;code&gt;replacement&lt;/code&gt; right?&lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2016-01-08T10:45:16.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9699" PostTypeId="1" CreationDate="2016-01-08T10:58:31.153" Score="3" ViewCount="136" Body="&lt;p&gt;I have a very skewed, 10-dimensional data set. I need approximate nearest neighbours for my use case and I was looking into Locality senstive hashing. However after scaling and randomly generating hyperplanes through the origin and coding the data points like that I got very skewed buckets because of the nature of the data. After thinking about it for a bit I came up with an idea to take random points from the data and use these as cluster centers for the hash. Every point will be mapped to the ID of the closest randomly picked center. My question is if the expected size of the bucket of a specific point is the same for all the other data points. I think that is the case but others say it shouldn't. My rationale is that more dense areas have more randomly decided cluster points while outliers will not be picked very often. I can't find anything about this in literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I did run some tests and they seem to support my hypothesis somewhat but variance is relatively high because there is a high dependence between sizes of clusters (data points wise)&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-03-03T21:06:01.783" Title="Expectation number of points in initial clustering for LSH" Tags="|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9700" PostTypeId="2" ParentId="9699" CreationDate="2016-01-08T11:12:37.197" Score="0" Body="&lt;p&gt;It's not a proof, but I just found a paper that claims this empirically, and also had the exact same idea for the same problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://web.kaist.ac.kr/~kyomin/2012BigLearn.pdf&quot; rel=&quot;nofollow&quot;&gt;http://web.kaist.ac.kr/~kyomin/2012BigLearn.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone does happen to have a proof for this, I'm still very interested though.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-01-08T11:12:37.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9701" PostTypeId="1" AcceptedAnswerId="9704" CreationDate="2016-01-08T11:51:00.473" Score="0" ViewCount="102" Body="&lt;p&gt;I am looking for a good python api for timeseries models such as ARIMA. Please list some well tested apis and few more advance models possible for financial time-series analysis.&lt;/p&gt;&#xA;" OwnerUserId="837" LastActivityDate="2016-01-08T16:42:36.143" Title="Please list some well tested api's for arima model" Tags="|python|predictive-modeling|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9702" PostTypeId="1" CreationDate="2016-01-08T11:57:50.097" Score="9" ViewCount="9497" Body="&lt;p&gt;I trained a prediction model with Scikit Learn in Python (Random Forest Regressor) and I want to extract somehow the weights of each feature to create an excel tool for manual prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing that I found is the &lt;code&gt;model.feature_importances_&lt;/code&gt; but it doesn't help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to achieve it?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def performRandomForest(X_train, y_train, X_test, y_test):&#xA;&#xA;    '''Perform Random Forest Regression'''&#xA;&#xA;    from sklearn.ensemble  import  RandomForestRegressor&#xA;&#xA;    model  =  RandomForestRegressor()&#xA;    model.fit( X_train , y_train )&#xA;&#xA;    #make predictions&#xA;    expected  = y_test&#xA;    predicted  = model.predict( X_test )&#xA;&#xA;    #summarize the fit of the model&#xA;    mse  = np.mean(( predicted - expected )** 2)&#xA;    accuracy = ( model.score ( X_train , y_train ))&#xA;&#xA;    return model, mse, accuracy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At the moment, I use the &lt;code&gt;model.predict([features])&lt;/code&gt; to do it, but I need it in an excel file.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2019-05-13T12:00:34.000" Title="Export weights (formula) from Random Forest Regressor in Scikit-Learn" Tags="|python|predictive-modeling|regression|random-forest|scikit-learn|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9703" PostTypeId="2" ParentId="9699" CreationDate="2016-01-08T12:18:48.240" Score="0" Body="&lt;p&gt;A counterexample in dimension 1: take three points: 1, 3 and 4 on the x axis. When you sample two points at random, the expected sizes of the bucket each point falls in are 4/3, 2 and 5/3 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But maybe your hypothesis is &quot;approximately true&quot; for data from some good enough distribution (for example, uniformly random points on a sphere).&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2016-01-08T12:18:48.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9704" PostTypeId="2" ParentId="9701" CreationDate="2016-01-08T16:42:36.143" Score="1" Body="&lt;p&gt;&lt;strong&gt;Statsmodels:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Statsmodels&lt;/a&gt; is your best bet for a python library that includes ARIMA.  I have used it fairly extensively and am quite happy with it. But, its certainly not as well tested as R based ARIMA models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something &quot;well-tested&quot; then your best bet is likely to use &lt;a href=&quot;http://rpy.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;Rpy2&lt;/a&gt; to call an &lt;a href=&quot;https://cran.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; based &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/arima.html&quot; rel=&quot;nofollow&quot;&gt;ARIMA library&lt;/a&gt; from python. Rpy2 can be a bit tricky based on version reconciliation between python, R and Rpy2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a &lt;a href=&quot;https://sites.google.com/site/aslugsguidetopython/data-analysis/pandas/calling-r-from-python&quot; rel=&quot;nofollow&quot;&gt;turorial on calling R from python using Rpy2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-01-08T16:42:36.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9706" PostTypeId="2" ParentId="9680" CreationDate="2016-01-08T19:54:12.367" Score="4" Body="&lt;p&gt;The approach that comes to mind, is to calculate the kullback-leibler divergence between the kernel density estimations of your train dataset and of your test dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The kernel density estimation of each of your datasets will give you an approximation to the pdf's of your datasets. The kullback-leibler divergence will give you a number that will represent the divergence in bits from one distribution to another (if you use base 2 for your logarithm). Below are some references I think you would fine useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Kernel_density_estimation&lt;/a&gt;&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&lt;/a&gt;&#xA;&lt;a href=&quot;https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/&quot; rel=&quot;nofollow&quot;&gt;https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like me to show the math behind this method. Feel free to ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: Added math as asked by author of question&lt;/strong&gt;    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $\hat x_1, \hat x_2,\hat x_3...,\hat x_n$ be your training dataset while $x_1,x_2,x_3...,x_n$ is your testing dataset, where both $\hat x_i$ and $x_i$ belong to $\mathbb{R}^d$.&#xA;$$\hat f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-\hat x_i;H)$$&#xA;$$f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-x_i;H)$$&#xA;$\hat f$and $f$ represent the kernel density estimation for training set and testing set respectively. The parameter $H$ represents the bandwidth parameter and is a symmetric positive definite $d \times d$ matrix. $K(u;H)$ can be rewritten as $$|H|^{-\frac{1}{2}} K(H^{\frac{1}{2}} u)$$ where K can be any kernel function. I would recommend for simplicity purposes the standard multivariate normal kernel. Okay so now that we have the kernel density estimations of both our training and our testing dataset, we can use the kullback-leibler divergence in order to estimate the difference between the two.&lt;br&gt;&#xA;Optimally we would like to calculate the kullback-leibler divergence with respect to every point in our space. Mathematically speaking.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_X f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})dx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is computationally unpractical to compute. We can approximate this integral by sampling a set of points from the $f(x;H)$ and then computing the discrete sum.&#xA;$$\sum_{x \in X} f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Quick Note:&lt;/strong&gt; To sample from a kernel density estimator, uniformly randomly select a point from the respective dataset. Then sample from the kernel of choice centered around the point chosen. &lt;/p&gt;&#xA;" OwnerUserId="14779" LastEditorUserId="14779" LastEditDate="2016-01-08T22:18:16.683" LastActivityDate="2016-01-08T22:18:16.683" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9708" PostTypeId="1" CreationDate="2016-01-09T01:16:15.027" Score="2" ViewCount="173" Body="&lt;p&gt;I've built a multivariate regression model for predicting the price of an item given a couple of other factors about that item - age, condition, etc. So, given values for age and condition I can return a single value for price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what I want to do is return a price &lt;em&gt;range&lt;/em&gt;, representing, say, 95% confidence that the true value will fall within this range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've googled a bunch, but I can't find a clear description of how to achieve this. The closest I've come is what's described in this paper: &lt;a href=&quot;http://pareonline.net/getvn.asp?v=7&amp;amp;n=2&quot; rel=&quot;nofollow&quot;&gt;http://pareonline.net/getvn.asp?v=7&amp;amp;n=2&lt;/a&gt;, in the section &quot;Calculating a Predicted Score, and Confidence Intervals Around That Score&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that the confidence of a regression model &quot;bows&quot; in around the mean for each value, I feel like this is more complex than just taking +/-X% of the predicted value. The range should be greater for items that are at the extremes of age and condition, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Grateful for any help.&lt;/p&gt;&#xA;" OwnerUserId="15281" LastActivityDate="2016-01-09T01:16:15.027" Title="Confidence range for individual prediction from multiple regression" Tags="|machine-learning|regression|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9712" PostTypeId="2" ParentId="9290" CreationDate="2016-01-09T06:59:39.420" Score="2" Body="&lt;p&gt;&lt;strong&gt;Some of the settings are likely too aggressive for your machine:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As others have pointed out, &lt;code&gt;n_jobs&lt;/code&gt; might help by taking advantage of openMP threading that is likely available on a multicore processor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But lets also look at &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow&quot;&gt;some other parameters from the documentation&lt;/a&gt;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RandomForestRegressor(n_estimators=10, criterion='mse', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=1, random_state=None, verbose=0, warm_start=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why don't you set &lt;code&gt;n_estimators&lt;/code&gt; to 10 and then slowly ramp it up.&lt;/li&gt;&#xA;&lt;li&gt;max_depth should probably be set both so you trees aren't too computationally intensive and so you don't overfit your model (are you cross validating?)&lt;/li&gt;&#xA;&lt;li&gt;You have set &lt;code&gt;max_features&lt;/code&gt; to &lt;code&gt;auto&lt;/code&gt; which specifies that it use &lt;code&gt;n_features&lt;/code&gt;, but you have not set &lt;code&gt;n_features&lt;/code&gt;.  You could consider turning this down to some percentage of your full feature set.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general you need to cross validate your model and tinker with the parameters in order to find something that runs on your box in finite time using the available memory and performs adequately.  You could also consider tuning and cross validating a simpler decision tree and then using that insight to tweak the parameters of the random forest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample the data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Big datasets often require lots of computing power.  I sometimes drastically reduce the size of my training data while I am zeroing in on the best model and then time the scaling as I increase the dataset.  This will give you a better idea of how much time you should expect for the full problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other models:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, maybe you have a problem for which 'bootstrap aggregation' is a bad fit. Maybe another type of aggregation would work better like 'boosting' in which case a 'random jungle' or 'ada-boost' algo might work better.  Or perhaps you might want to move to a computationally less intense method that doesn't require aggregation like a 'support vector regressor'.  &lt;code&gt;Scikit-learn&lt;/code&gt; has a great benefit in that it is very simple to swap out different models and try different pipelines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cloud Computing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what box or laptop you are running this on, but it might just be easier to spin up an AWS (Google/MS/IBM) instance and run this on a well-powered box in the cloud.  The AWS spot market is crazy affordable over the weekends or in the middle of the night, so you can work on some very powerful 8 core machines for $.50 per hour, so a 24 hour job is only $12.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-01-09T06:59:39.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9713" PostTypeId="1" CreationDate="2016-01-09T14:43:38.303" Score="1" ViewCount="783" Body="&lt;p&gt;I'm using a Random Forest algorithm in order to construct a classification model, and I HAVE to check the accuracy of my rf model in the training sample, but as you can see in this answers : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/112052/90446&quot;&gt;https://stats.stackexchange.com/a/112052/90446&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/66546/90446&quot;&gt;https://stats.stackexchange.com/a/66546/90446&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can't evaluate the accuracy considering the training samples like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(model, data=train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not confortable with the idea of use OOB to get accuracy of the training sample, because the OOB was not used to build the model, how could this be right? I don't know what should I do to predict the fit in the training sample and get the accuracy of the training sample, is it possible or make any sense? When a check the AUC of the prediction of my training sample I get something near of 0.98, but the AUC of the test sample is about 0.7. Is this  due to the limitations of prediction at the training sample or due to Overfitting? &lt;/p&gt;&#xA;" OwnerUserId="14644" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-10-26T11:15:34.603" Title="Prediction in the training sample with randomforest in r" Tags="|machine-learning|r|predictive-modeling|random-forest|accuracy|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9714" PostTypeId="2" ParentId="9668" CreationDate="2016-01-09T15:17:31.107" Score="3" Body="&lt;p&gt;&lt;strong&gt;SVC Parameters:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What prompted you to use a polynomial kernel?  There are cases where this might make sense, but it certainly wouldn't be my first choice.  A radial basis function is likely a better fit, which is why it is the default in SKLearn. Step back a minute and take in the big picture perspective. Does your human mind want to create a decision surface using an 11th order polynomial given the points that you showed? It looks like there are a couple islands of white points in your second image that could easily be wrapped with two &lt;code&gt;Gaussian functions&lt;/code&gt; aka &lt;code&gt;radial basis functions&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does the SVC with only default parameters do? I've seen plenty of cases where a regularization parameter as high as 10 or 100 is needed, but you haven't spoken to the cross validation procedure you used to arrive at $C=10$, so I wonder how you got there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest running SVC with default parameters and seeing what this returns.  Then maybe try tuning gamma (this is effectively the buffer zone between the decision surface and points around it) and C (this is the regularization term that controls overfitting aka high variance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-01-09T15:17:31.107" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9715" PostTypeId="1" AcceptedAnswerId="9717" CreationDate="2016-01-09T16:07:08.427" Score="5" ViewCount="289" Body="&lt;p&gt;I have been doing machine learning for a while, but bits and pieces come together even after some time of practicing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In neural networks, you adjust the weights by doing one pass (forward pass), and then computing the partial derivatives for the weights (backward pass) after each training example - and subtracting those partial derivatives from the initial weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in turn, the calculation of the new weights is mathematically complex (you need to compute the partial derivative of the weights, for which you compute the error at every layer of the neural net - but the input layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that not by definition an online algorithm, where cost and new weights are calculated after each training example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="9197" LastEditorUserId="9197" LastEditDate="2016-01-09T16:42:05.333" LastActivityDate="2016-01-10T19:06:31.767" Title="is neural networks an online algorithm by nature?" Tags="|machine-learning|neural-network|online-learning|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9716" PostTypeId="1" CreationDate="2016-01-09T16:47:45.160" Score="1" ViewCount="477" Body="&lt;p&gt;One of the applications of Ternary content addressable memory(TCAM) is artificial neural network. How does TCAM do prediction or recognition? How TCAM is related to artificial neural network?&lt;/p&gt;&#xA;" OwnerUserId="15293" LastActivityDate="2017-07-13T15:48:19.887" Title="Artificial neural network on ternary content addressable memory" Tags="|machine-learning|data-mining|bigdata|predictive-modeling|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9717" PostTypeId="2" ParentId="9715" CreationDate="2016-01-09T16:48:25.530" Score="1" Body="&lt;p&gt;You can train after each example, or after each epoch. This is the difference between stochastic gradient descent, and batch gradient descent. See pg 84 of Sebastian Raschka's Python Machine Learning book for more.&lt;/p&gt;&#xA;" OwnerUserId="14715" LastActivityDate="2016-01-09T16:48:25.530" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9718" PostTypeId="2" ParentId="9662" CreationDate="2016-01-10T01:31:18.250" Score="2" Body="&lt;p&gt;There are many tools that are able to support such queries (as you mentioned Hive or Spark), and it is really up to your requirements in terms of number of queries, number of people who are going to query the data, what kind of BI or reporting tools you might want to use with it, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More than that, your requirements are probably going to change in the future. If you provide easier ways to query the data, the more people around you are going to use it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you choose to load that data into Amazon Redshift (&lt;a href=&quot;https://aws.amazon.com/redshift/&quot; rel=&quot;nofollow&quot;&gt;https://aws.amazon.com/redshift/&lt;/a&gt;), it will give you a full SQL flexibility, very fast performance and the ability to connect to several BI, visualization and reporting tools. When you COPY the data into Redshift it is being compressed (column based encoding) 5-8 times, and you can put all your data in a cluster of 6 nodes of DS2.8XL of Redshift, for a couple of weeks for your POC. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept of compression can be applied in different solutions as well. For example, instead of using CSV format, you can convert it to Parquet (&lt;a href=&quot;https://parquet.apache.org/&quot; rel=&quot;nofollow&quot;&gt;https://parquet.apache.org/&lt;/a&gt;) format, and save on your storage and increase the performance as you need to move around and scan less bytes of data. Once you have your data in Parquet, you can query it for example with Presto, like Netflix are doing - &lt;a href=&quot;http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html&quot; rel=&quot;nofollow&quot;&gt;http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding Spark, it can help even if you don't fit all the data into memory. Spark can also help you with the usage of Zeppelin (&lt;a href=&quot;https://zeppelin.incubator.apache.org/&quot; rel=&quot;nofollow&quot;&gt;https://zeppelin.incubator.apache.org/&lt;/a&gt;) as an interactive notebook, with nice visualization features, as well as write code with Scala, Python or even R with SparkR. &lt;/p&gt;&#xA;" OwnerUserId="15300" LastActivityDate="2016-01-10T01:31:18.250" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9719" PostTypeId="1" CreationDate="2016-01-10T07:48:55.200" Score="1" ViewCount="1618" Body="&lt;p&gt;I'm looking  to  find some  tutorials  involving embedding a Machine Learning Model into a  Web Application eventualy with some storage and database behind( with things like slack  etc)  .&lt;/p&gt;&#xA;" OwnerUserId="14946" LastEditorUserId="14946" LastEditDate="2016-01-10T08:57:01.620" LastActivityDate="2016-01-10T09:17:00.887" Title="Embedding a Machine Learning Model into a Web Application" Tags="|machine-learning|consumerweb|" AnswerCount="1" CommentCount="1" ClosedDate="2016-01-18T17:55:36.290" ContentLicense="CC BY-SA 3.0" />
  <row Id="9721" PostTypeId="2" ParentId="9719" CreationDate="2016-01-10T09:17:00.887" Score="1" Body="&lt;p&gt;I am not sure you would get any such tutorial regarding the same. It is more like asking how to build a software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider you have to build an ML model for doing Sentiment Analysis before a post comes onto your feed, this is how you need to build the system:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The ML model would constitute the backend of the software, as it is just another algorithm we are using for building the software.&lt;/li&gt;&#xA;&lt;li&gt;The plots or the feed you want to display to the end user would constitute the front end, which would include the templates and the views. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above can be achieved through an MVC framework. The data can be sent back and forth the server and the client through a protocol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/How-can-I-implement-machine-learning-algorithms-in-a-web-application&quot; rel=&quot;nofollow&quot;&gt;Useful link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case, you want to integrate it with an external system, then create an API for the above system which you have built and let the other systems use your software/app by calling your api.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, create a simple SlackBot which can interact with your API with some certain commands, and display the relevant results in the group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/kengz/jarvis&quot; rel=&quot;nofollow&quot;&gt;Example of such a system&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-10T09:17:00.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9722" PostTypeId="1" CreationDate="2016-01-10T10:49:20.447" Score="11" ViewCount="2888" Body="&lt;p&gt;I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword (&quot;kw1&quot;) and wants to find semantically relevant terms for &quot;kw1&quot; using word2vec. For example if the keyword is &quot;apple&quot; I would expect to see related terms such as &quot;ipad&quot; &quot;os&quot; &quot;mac&quot;... based on the input file. So this set of related terms for &quot;kw1&quot; would be different for each input file as word2vec would be trained on individual files (eg., 5 input files, run word2vec 5 times on each file). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to find sets of related terms for each input file given the common keyword (&quot;kw1&quot;), which would be used for some other purposes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions/doubts are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does it make sense to use word2vec for a task like this? is it technically right to use considering the small size of an input file?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have downloaded the code from code.google.com: &lt;a href=&quot;https://code.google.com/p/word2vec/&quot;&gt;https://code.google.com/p/word2vec/&lt;/a&gt; and have just given it a dry run as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; time ./word2vec -train $file -output vectors.bin -cbow 1 -size 200 -window 10 -negative 25 -hs 1 -sample 1e-3 -threads 12 -binary 1 -iter 50&#xA;&#xA;./distance vectors.bin &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;From my results I saw I'm getting many noisy terms (stopwords) when I'm using the 'distance' tool to get related terms to &quot;kw1&quot;. So I did remove stopwords and other noisy terms such as user mentions. But I haven't seen anywhere that word2vec requires cleaned input data...?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How do you choose right parameters? I see the results (from running the distance tool) varies greatly when i change parameters such as '-window', '-iter'. Which technique should I use to find the correct values for the parameters. (manual trial and error is not possible for me as I'll be scaling up the dataset). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15303" LastActivityDate="2016-01-11T10:26:43.330" Title="applying word2vec on small text files" Tags="|machine-learning|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9723" PostTypeId="2" ParentId="9715" CreationDate="2016-01-10T12:36:27.090" Score="4" Body="&lt;p&gt;There are three training modes for neural networks&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;stochastic gradient descent&lt;/strong&gt;: Adjust the weights after every single training example&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;batch training&lt;/strong&gt;: Adjust the weights after going through all data (an epoch)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;mini-batch training&lt;/strong&gt;: Adjust the weights after going through a &lt;em&gt;mini-batch&lt;/em&gt;. This is usually 128 training examples.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Most of the time, mini-batch training seems to be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answer is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No, the neural network learning algorithm is not online algorithm.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-10T19:06:31.767" LastActivityDate="2016-01-10T19:06:31.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9724" PostTypeId="1" CreationDate="2016-01-10T14:11:47.697" Score="2" ViewCount="497" Body="&lt;p&gt;Could you please help me to understand it because I'm not sure if I got it correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a dataset, of persons, with 100 features, various characteristics like height, weight, age, etc. I want to classify if are normal or abnormal. By abnormal I mean if a 20 years old man is 170cm and 150kg to identify it as abnormal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I use Self Organising Map to reduce the dimensionality (these 100 features) and then K-means to classify them into normal and abnormal? Is that a correct approach? Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many features can I use with K-NN? All the examples I've found so far use just two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?&lt;/p&gt;&#xA;" OwnerUserId="15308" LastActivityDate="2016-01-13T16:41:50.893" Title="Deciding about dimensionality reduction, classification and clustering?" Tags="|machine-learning|classification|clustering|dimensionality-reduction|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9725" PostTypeId="2" ParentId="6555" CreationDate="2016-01-10T14:41:39.533" Score="1" Body="&lt;p&gt;The issue is that pyspark is not on os sys path by default. After several failed attempt to add it manually to my config files/paths/etc, I came across &lt;a href=&quot;https://github.com/minrk/findspark&quot; rel=&quot;nofollow noreferrer&quot;&gt;this GitHub&lt;/a&gt; repository called findspark.&lt;/p&gt;&#xA;&lt;p&gt;I cloned this repository using&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;git clone https://github.com/minrk/findspark.git&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Then I ran &lt;code&gt;pip install findspark&lt;/code&gt; from the findspark root.&lt;/p&gt;&#xA;&lt;p&gt;Started a Jupyter notebook, created a new Python3 notebook and added&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import findspark  &#xA;import os  &#xA;findspark.init()  &#xA;import pyspark  &#xA;sc = pyspark.SparkContext() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Before &lt;code&gt;findspark.init()&lt;/code&gt;, &lt;code&gt;import pyspark&lt;/code&gt; came back with an error.&lt;/p&gt;&#xA;&lt;p&gt;To test I just typed &lt;code&gt;sc&lt;/code&gt; and got back:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;pyspark.context.SparkContext at 0x4526d30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;All working for me now.&lt;/p&gt;&#xA;" OwnerUserId="15306" LastEditorUserId="98307" LastEditDate="2020-08-03T09:31:05.810" LastActivityDate="2020-08-03T09:31:05.810" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9726" PostTypeId="1" AcceptedAnswerId="9743" CreationDate="2016-01-10T17:10:59.837" Score="0" ViewCount="55" Body="&lt;p&gt;I am currious  how good  such  a procedure could be :&#xA;I  get some predictions  of some 10  learners  trained   on train set  and predicted on train set also .&#xA;Then I am column binding   those predictions  to the original train.&#xA;Could this  be  a valid procedure  on improving  learning process?&lt;/p&gt;&#xA;" OwnerUserId="14946" LastActivityDate="2016-01-11T11:07:23.760" Title="Predicted features combined with original ones" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9727" PostTypeId="1" CreationDate="2016-01-10T17:30:52.217" Score="1" ViewCount="2603" Body="&lt;p&gt;I've got a program that analyses some data based on a bunch of values then spits out a true/false value if it was correct or not. I'm using the Weka software to graph this all up and I just want to show the ROC curve for my data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use the ZeroR classifier does it change the data in any way or does it just display the data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to just show the results and get the ROC without changing/inferring any data - can I use ZeroR for this?&lt;/p&gt;&#xA;" OwnerUserId="15311" LastEditorUserId="15311" LastEditDate="2016-01-11T12:41:58.593" LastActivityDate="2018-09-10T18:12:08.817" Title="ZeroR Classifier" Tags="|classification|predictive-modeling|weka|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9729" PostTypeId="1" CreationDate="2016-01-10T19:44:59.887" Score="8" ViewCount="5394" Body="&lt;p&gt;I'm trying to use &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt; and &lt;a href=&quot;https://github.com/jterrace/pyssim&quot; rel=&quot;noreferrer&quot;&gt;pyssim&lt;/a&gt; for clustering a set of images - less than 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The end goal is to place the images into several buckets (clusters) according to the calculated similarity measures - CW-SSIM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task seems to be trivial, but I can't figure out the best way to handle &quot;similarity based&quot; clustering in scikit-learn. K-Means clustering looks like a good choice, but it doesn't accept any &quot;comparison functions&quot; or custom distance functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how to handle the comparison based (similarity based) clustering in scikit-learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about &quot;comparison matrices&quot; with 1 (similar) or 0 (not similar) per cell according to the calculated CW-SSIM similarity values. This matrix will be used for fitting into K-Means clustering. But then we will face the scalability issue, because such matrix will have dimensions equal to the amount of images ... which might grow to 1+ million in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is an easier option in R than in Python, then I'm ready to review as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE from Jan 18, 2016&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created some code on GitHub about this topic: &lt;a href=&quot;https://github.com/llvll/imgcluster&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/llvll/imgcluster&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This project also includes IP[y] Notebook with step-by-step instructions and extra comments: &lt;a href=&quot;https://github.com/llvll/imgcluster/blob/master/ip%5By%5D/imgcluster.ipynb&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/llvll/imgcluster/blob/master/ip%5By%5D/imgcluster.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15315" LastEditorUserId="15315" LastEditDate="2016-01-17T23:40:47.650" LastActivityDate="2016-01-17T23:40:47.650" Title="Image clustering by similarity measurement (CW-SSIM)" Tags="|machine-learning|r|python|scikit-learn|k-means|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9730" PostTypeId="2" ParentId="9729" CreationDate="2016-01-10T21:29:01.780" Score="4" Body="&lt;p&gt;I would use a regular clustering algorithm and replace the objective function, which is usually the MSE, with a differentiable loss function of your choice. Another way is to learn an embedding that optimizes your similarity metric using a neural network and just cluster that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would rather do similarity-based clustering, here are some papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A Similarity-Based Robust Clustering Method&lt;/li&gt;&#xA;&lt;li&gt;A Discriminative Framework for Clustering via Similarity Functions&lt;/li&gt;&#xA;&lt;li&gt;Similarity-Based Clustering by Left-Stochastic Matrix Factorization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;sklearn implements two similarity clustering methods: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.html#sklearn.cluster.AffinityPropagation&quot; rel=&quot;nofollow&quot;&gt;Affinity propagation&lt;/a&gt;, and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html#sklearn.cluster.SpectralClustering&quot; rel=&quot;nofollow&quot;&gt;spectral clustering&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-01-11T05:52:30.573" LastActivityDate="2016-01-11T05:52:30.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9731" PostTypeId="1" AcceptedAnswerId="10295" CreationDate="2016-01-11T01:07:17.193" Score="1" ViewCount="408" Body="&lt;p&gt;First a theoretical question and then a practical one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural nets backpropagation is the computation of the weights derivatives or the computation of the new weights (that is, the original weights minus  the weight derivatives times the learning rate - simplified)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may well be a semantics issue but important nevertheless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if anyone is familiar with Torch, nn class&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gradInput = module:backward(input, gradOutput)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is gradinput the weight set for the next forward pass or is it the derivatives of the weights of the previous forward pass?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="9197" LastActivityDate="2017-03-30T21:07:03.073" Title="Torch and conceptual question about neural nets backpropagation" Tags="|neural-network|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9734" PostTypeId="1" AcceptedAnswerId="9764" CreationDate="2016-01-11T05:56:19.680" Score="4" ViewCount="128" Body="&lt;p&gt;I recently came across this term &lt;code&gt;recurrent heavy subgraph&lt;/code&gt; in a talk. I don't seem to understand what it means and Google doesn't seem to show any good results. Can someone explain what this means in detail.&lt;/p&gt;&#xA;" OwnerUserId="15324" LastEditorUserId="2932" LastEditDate="2016-01-12T16:40:48.607" LastActivityDate="2016-01-12T16:40:48.607" Title="What is a Recurrent Heavy Subgraph?" Tags="|data-mining|graphs|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9735" PostTypeId="1" AcceptedAnswerId="9737" CreationDate="2016-01-11T07:07:23.557" Score="9" ViewCount="6183" Body="&lt;p&gt;As reading &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensemble methods&lt;/a&gt; on scikit-learn docs, it says that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;bagging methods work best with strong and complex models (e.g., fully&#xA;  developed decision trees), in contrast with boosting methods which&#xA;  usually work best with weak models (e.g., shallow decision trees).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But search on google it always return information about &lt;code&gt;Decision Tree&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I'd like to know the detail of the two trees mentioned in the doc, what's the &lt;code&gt;fully developed&lt;/code&gt; and &lt;code&gt;shallow&lt;/code&gt; meanings.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;About why &lt;code&gt;bagging&lt;/code&gt; work best with &lt;code&gt;fully developed&lt;/code&gt; and why &lt;code&gt;boosting&lt;/code&gt; work best with &lt;code&gt;shallow&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I think &lt;code&gt;complex models (e.g., fully developed decision trees)&lt;/code&gt; means such a data set has a complex format be called as &lt;code&gt;fully developed decision trees&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I read above quote over 20 times and rapaio's answer, I think my poor English lead me to the wrong road(misunderstand). I also mistake &lt;code&gt;shallow&lt;/code&gt; as &lt;code&gt;shadow&lt;/code&gt; , which make me confusing a long time....Now I understand the meanings of &lt;code&gt;fully developed&lt;/code&gt; and &lt;code&gt;shallow&lt;/code&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the quote is saying &lt;code&gt;bagging&lt;/code&gt; work best with a models(already trained) which algorithm is complex. And &lt;code&gt;boosting&lt;/code&gt; only need simple model.&#xA;Both &lt;code&gt;bagging&lt;/code&gt; and &lt;code&gt;boosting&lt;/code&gt; need  many &lt;code&gt;estimators&lt;/code&gt;, as &lt;code&gt;n_estimators=100&lt;/code&gt; in scikit-learn examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;code&gt;n_estimators=100&lt;/code&gt; :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;bagging&lt;/code&gt; need 100 &lt;code&gt;fully developed decision trees&lt;/code&gt; estimators(models)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;boosting&lt;/code&gt; need 100 &lt;code&gt;shallow decision trees&lt;/code&gt; estimators(models)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does my thoughts is right?&#xA;Hope my update can help non-native speakers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;e.g.&lt;/code&gt; means for example, so there are other models can use for bagging and boosting. How about changing the model to &lt;code&gt;svm&lt;/code&gt; or something else? Or both of they need a tree base model?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="15325" LastEditDate="2017-03-09T01:08:47.527" LastActivityDate="2017-04-13T08:49:48.457" Title="what is the difference between &quot;fully developed decision trees&quot; and &quot;shallow decision trees&quot;?" Tags="|scikit-learn|decision-trees|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9736" PostTypeId="1" AcceptedAnswerId="9751" CreationDate="2016-01-11T07:16:58.747" Score="25" ViewCount="19298" Body="&lt;p&gt;What are the hallmarks or properties that indicate that a certain learning problem can be tackled using support vector machines?&lt;/p&gt;&#xA;&lt;p&gt;In other words, what is it that, when you see a learning problem, makes you go &amp;quot;oh I should definitely use SVMs for this&amp;quot; rather than neural networks or decision trees or anything else?&lt;/p&gt;&#xA;" OwnerUserId="11044" LastEditorUserId="29169" LastEditDate="2021-11-17T18:46:49.397" LastActivityDate="2021-11-17T18:46:49.397" Title="What kinds of learning problems are suitable for Support Vector Machines?" Tags="|machine-learning|svm|supervised-learning|unsupervised-learning|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="9737" PostTypeId="2" ParentId="9735" CreationDate="2016-01-11T09:15:44.103" Score="16" Body="&lt;p&gt;[&lt;strong&gt;Later edit - Rephrase everything&lt;/strong&gt;]&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Types of trees&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;A &lt;em&gt;shallow tree&lt;/em&gt; is a small tree (most of the cases it has a small depth). A &lt;em&gt;full grown tree&lt;/em&gt; is a big tree (most of the cases it has a large depth). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a training set of data which looks like a non-linear structure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7Y2k1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7Y2k1.png&quot; alt=&quot;Synthetic sample with 2 regression trees. Red line is for tree with depth 3, blue line is for tree with depth 10&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Bias variance decomposition as a way to see the learning error&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Considering bias variance decomposition we know that the learning error has 3 components:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Err = \text{Bias}^2 + \text{Var} + \epsilon$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bias&lt;/strong&gt; is the error produced by the fit model when it is not capable to represent the true function; it is in general associated with &lt;em&gt;underfitting&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Var&lt;/strong&gt; is the error produced by the fit model due to sampled data, it describes how unstable a model is if the training data changes; it is in general associated with &lt;em&gt;overfitting&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;$\epsilon$&lt;/strong&gt; is the irreducible error which envelops the true function; this can't be learned&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://onlinecourses.science.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/lesson04/model_complexity.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://onlinecourses.science.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/lesson04/model_complexity.png&quot; alt=&quot;Model compexity&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering our shallow tree we can say that the model has low variance since changing the sample does not change too much the model. It needs too many changed data points to be considered unstable. At the same time we can say that has a high bias, since it really can't represent the sine function which is the true model. We can say also that it has a low complexity. It can be described by 3 constants and 3 regions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consequently, the full grown tree has low bias. It is very complex since it can be described only using many regions and many constants on those regions. This is why it has low bias. The complexity of the model impacts also variance which is high. In some regions at least, a single point sampled differently can change the shape of the fitted tree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule of thumb when a model has low bias it has also high variance and when it has low variance it has high bias. This is not true always, but it happens very often. And intuitively is a correct idea. The reason is that when you are getting close to the points in the sample you learn the patterns but also learn the errors from sample, when you are far away from sample you are instead very stable since you do not incorporate errors from sample. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;How can we build ensembles using those kind of trees?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bagging&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The statistical model behind bagging is based on bootstrapping, which is a statistical procedure to evaluate the error of a statistic. Assuming you have a sample and you want to evaluate the error of a statistical estimation, bootstrapping procedure allows you to approximate the distribution of the estimator. But trees are only a simple function of sample &quot;split the space into regions and predict with the average, a statistic&quot;. Thus if one builds multiple trees from bootstrap samples and averages the trees can be considered i.i.d. and the same principle works to reduce variance. Because of that bagging allows one to reduce the variance without affecting too much the bias. Why it needs full depth trees? Well it is a simple reason, perhaps not so obvious at first sight. It need classifiers with high variance in order to reduce it. This procedure does not affect the bias. If the underlying models have low variance high bias, the bagging will reduce slightly the already small variance and produce nothing for bias. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Boosting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does boosting? Many compare boosting with model averaging, but the comparison is flawed. The idea of boosting is that the ensemble is an iterative procedure. It is true that the final rule for classification looks like a weighted average of some weak models, but the point is that those models were built iteratively. Has nothing to do with bagging and how it works. Any $k$ tree is build using information learned from all previous $k-1$ trees. So we have initially a weak classifier which we fit to data, where all the points have the same importance. This importance can be changed by weights like in adaboost or by residuals like in gradient boosting, it really does not matter. The next weak classifier will not treat in the same way all the points, but those previously classified correctly has smaller importance than those classifier incorrectly. The consequence is that the model enriches it's complexity, it's ability to reproduce more complex surfaces. This is translated in the fact that it reduces the bias, since it can go closer to data. A similar intuition is behind: if the classifier has already a low bias, what will happen when I boost it? Probably a much unbearable overfit, that's all.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Which one is better?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There is no clear winner. It depends too much on data set and on other parameters. For example bagging can't hurt. It is possible to be useless ut usually does not hurt performance. Boosting can lead to overfit. That is because you can go eventually too close to data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The is a lot of literature which says that when the irreducible error is high, the bagging is much better and boosting does not progress too much. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Can we decrease both variance and bias?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Pure bootstrap and bagging approaches serves a single purpose. Either reduce variance either reduce bias. However modern implementations changed various things in how those approaches works. Sampling can be used in boosting and it seems to work towards reducing also the variance. There are bagging procedures which takes some ideas prom boosting, for example iterative bagging (or adaptive bagging) published by Breiman. So, the answer is yes, is possible.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Can we use other learners other than trees?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Of course. Often you will see boosting to use this approach. I read some papers on bagging svm or another learners also. I tried myself to bag some svms but without much success. The trees are the preferred way, however, for a very simple reason, they are simple to build, simple to adapt and simple to control their effect. My personal opinion is that it is not everything said regarding ensembles of trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: a last note on the number of weak classifiers: this depends entirely on the complexity of the data set mixed with the complexity of the learner. There is no recipe. Often 20 of them are enough to get most of the information and additional ones are only for tiny tuning. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="-1" LastEditDate="2017-04-13T08:49:48.457" LastActivityDate="2017-04-13T08:49:48.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9738" PostTypeId="1" AcceptedAnswerId="9740" CreationDate="2016-01-11T09:24:36.577" Score="2" ViewCount="805" Body="&lt;p&gt;I am trying to learn TensorFlow, and I could understand how it uses the &lt;em&gt;batch&lt;/em&gt; in &lt;a href=&quot;https://www.tensorflow.org/versions/master/tutorials/mnist/pros/index.html#start-tensorflow-interactivesession&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))&#xA;train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)&#xA;correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, &quot;float&quot;))&#xA;sess.run(tf.initialize_all_variables())&#xA;for i in range(20000):&#xA;  batch = mnist.train.next_batch(50)&#xA;  if i%100 == 0:&#xA;    train_accuracy = accuracy.eval(feed_dict={&#xA;        x:batch[0], y_: batch[1], keep_prob: 1.0})&#xA;    print(&quot;step %d, training accuracy %g&quot;%(i, train_accuracy))&#xA;  train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})&#xA;&#xA;print(&quot;test accuracy %g&quot;%accuracy.eval(feed_dict={&#xA;    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is, why it get a batch of 50 training data, but only use the first one for training. Maybe I did not understand the code correctly.&lt;/p&gt;&#xA;" OwnerUserId="3167" LastActivityDate="2016-01-11T10:29:30.257" Title="Question about train example code for TensorFlow" Tags="|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9739" PostTypeId="2" ParentId="9722" CreationDate="2016-01-11T10:26:43.330" Score="9" Body="&lt;p&gt;Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a concise word similarity.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-01-11T10:26:43.330" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9740" PostTypeId="2" ParentId="9738" CreationDate="2016-01-11T10:29:30.257" Score="0" Body="&lt;p&gt;If I understood you correctly, you are asking about this line of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here you only specify which part of batch is used for features and which for your predicted class.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2016-01-11T10:29:30.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9741" PostTypeId="1" CreationDate="2016-01-11T10:38:54.740" Score="1" ViewCount="417" Body="&lt;p&gt;The objective function of kernel K-means is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$      \sum\limits_{c=1}^k \sum\nolimits_{a_i \in c}   w_i \Vert \phi(a_i)- m_c \Vert^2 \ $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ m_c = \frac{\sum\nolimits_{a_i \in c }w_i\phi(a_i)}{\sum\nolimits_{a_i \in c  }w_i}  $$&#xA;I need to know how to determine wi &lt;/p&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="14357" LastEditDate="2016-01-11T11:19:16.470" LastActivityDate="2016-01-11T12:11:36.337" Title="How to set weight in Weighted Kernel K-Means?" Tags="|machine-learning|data-mining|k-means|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9742" PostTypeId="2" ParentId="9727" CreationDate="2016-01-11T11:00:13.157" Score="1" Body="&lt;p&gt;ZeroR classifier uses only the target (dependent variable) to build a majority classifier. As a consequence, it does not fit your purpose. You can built it from code as described here: &lt;a href=&quot;https://weka.wikispaces.com/Generating+ROC+curve&quot; rel=&quot;nofollow&quot;&gt;Generating ROC curve example&lt;/a&gt;. Basically you do not need a classifier, you need only the results and an instance of &lt;a href=&quot;http://weka.sourceforge.net/doc.dev/weka/classifiers/evaluation/ThresholdCurve.html&quot; rel=&quot;nofollow&quot;&gt;ThresholdCurve&lt;/a&gt; class (this is how it is named in weka library).&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-01-11T11:00:13.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9743" PostTypeId="2" ParentId="9726" CreationDate="2016-01-11T11:07:23.760" Score="3" Body="&lt;p&gt;This procedure exists and is called &lt;em&gt;stacked generalization&lt;/em&gt; or simply &lt;em&gt;stacking&lt;/em&gt;. See stacking section from &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;wikipedia page&lt;/a&gt;. Strating from there you can red more by following references from the page. The first paper on subject was published by Wolpert in 1992. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Later edit]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not combine the results with original features, but keep only the predictions combined only with the target. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-01-11T11:07:23.760" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9744" PostTypeId="2" ParentId="9741" CreationDate="2016-01-11T12:11:36.337" Score="1" Body="&lt;p&gt;These weights should be introduced by a user. With a weight you tell the K-means algorithm, that one feature is more important than the other.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[0] These might represent a measure of importance, a frequency count, or some other information. The intent is that a point with a weight of 5.0 is twice as &quot;important&quot; as a point with a weight of 2.5, for instance. This gives rise to the &quot;weighted&quot; K-Means problem. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;[0] &lt;a href=&quot;http://people.sc.fsu.edu/~jburkardt/m_src/kmeans/kmeans.html&quot; rel=&quot;nofollow&quot;&gt;http://people.sc.fsu.edu/~jburkardt/m_src/kmeans/kmeans.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2016-01-11T12:11:36.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9745" PostTypeId="1" CreationDate="2016-01-11T12:19:27.103" Score="1" ViewCount="1348" Body="&lt;p&gt;Can Anybody Explain me the role of ZooKeeper and ZooKeeper Quorum in a BigData structure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd appreciate if you explain me with a simple practical example instead of abstract theoretical definitions.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Please&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As you would explain for a non-technical person&lt;/p&gt;&#xA;" OwnerUserId="14672" LastEditorUserId="14672" LastEditDate="2016-01-11T15:02:25.627" LastActivityDate="2016-01-11T19:18:07.320" Title="A Simple Explanation of ZooKeeper in Hadoop" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9746" PostTypeId="1" CreationDate="2016-01-11T12:36:55.430" Score="3" ViewCount="1914" Body="&lt;p&gt;Can anybody explain Apache Kafka for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is it used for? What does messaging mean? Messaging between what?!! At which stage of a BigData analysis is it used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what are prerequisites for learning it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: &#xA;Please explain as you would explain for a non-technical person&lt;/p&gt;&#xA;" OwnerUserId="14672" LastEditorUserId="11097" LastEditDate="2016-01-11T15:36:55.507" LastActivityDate="2018-10-25T14:36:46.717" Title="Simple Explanation of Apache Kafka" Tags="|bigdata|apache-kafka|" AnswerCount="2" CommentCount="1" ClosedDate="2018-10-25T21:54:47.583" ContentLicense="CC BY-SA 3.0" />
  <row Id="9747" PostTypeId="1" CreationDate="2016-01-11T12:46:06.030" Score="4" ViewCount="705" Body="&lt;p&gt;Can anybody explain Apache Flume for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is it used for? At which stage of a BigData analysis is it used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what are prerequisites for learning it?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Please&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As you would explain for a non-technical person&lt;/p&gt;&#xA;" OwnerUserId="14672" LastEditorUserId="14672" LastEditDate="2016-01-11T15:01:44.097" LastActivityDate="2018-03-29T13:06:06.583" Title="Simple Explanation of Apache Flume" Tags="|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9748" PostTypeId="2" ParentId="9696" CreationDate="2016-01-11T13:55:47.207" Score="1" Body="&lt;p&gt;I don't think it makes a lot of sense to use HMM's for this problem. What I would suggest is some kind of text-based classifier. &#xA;If you want to use a cool technique, you could use a neural network to learn based on the text of successful posts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, If you want to use an easy technique you could make a predictor for the popularity such as a regression model (try to predict upvotes).&lt;/p&gt;&#xA;" OwnerUserId="15332" LastActivityDate="2016-01-11T13:55:47.207" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9749" PostTypeId="2" ParentId="9746" CreationDate="2016-01-11T14:39:49.243" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can anybody explain Apache Kafka for me in a plain language?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As the official documentation of Kafka says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Apache Kafka is publish-subscribe messaging rethought as a distributed&#xA;  commit log. It follows the publish-subscribe messaging style, with speed and durability built in.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, let me give a dumbed down explanation of the same:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A publish-subscribe(commonly called pub-sub) architecture basically contains a publisher(who sends the messages, for example: A twitter stream) and a subscriber(who receives the messages Ex: the analyst/the user's activity stream)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A clearer explanation would be:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In software architecture, publish–subscribe is a messaging pattern&#xA;  where senders of messages, called publishers, do not program the&#xA;  messages to be sent directly to specific receivers, called&#xA;  subscribers, but instead characterize published messages into classes&#xA;  without knowledge of which subscribers, if any, there may be.&#xA;  Similarly, subscribers express interest in one or more classes and&#xA;  only receive messages that are of interest, without knowledge of which&#xA;  publishers, if any, there are.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And keep in mind that Kafka is a &lt;strong&gt;distributed&lt;/strong&gt; pub-sub messaging system, designed to scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gcBD0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gcBD0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;At which stage of a BigData analysis is it used?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is basically used in the &lt;strong&gt;Extraction&lt;/strong&gt; step of the ETL pipeline. It can contain high volumes and speed of data flow (we call it high throughput in technical terms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can store messages also, but it is a persistent store; which means that the data is not stored forever. The stored data has an expiry date.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And what are prerequisites for learning it?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a tough question, as everyone who can program can get started with it, but if you really want to implement this in your architecture, then these are the pre-requisites which I can think of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Know what is a persistent data store and where to use them&lt;/li&gt;&#xA;&lt;li&gt;Know what a pub-sub messaging system is&lt;/li&gt;&#xA;&lt;li&gt;Know what an extraction step is, in ETL. And also know how to deal with data with huge volume coming at you at high velocity. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further reading:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kafka has an &lt;a href=&quot;http://kafka.apache.org/documentation.html&quot; rel=&quot;noreferrer&quot;&gt;excellent documentation&lt;/a&gt;, which can help to get started with it.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-01-11T14:50:10.947" LastActivityDate="2016-01-11T14:50:10.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9750" PostTypeId="2" ParentId="9747" CreationDate="2016-01-11T18:35:57.990" Score="5" Body="&lt;p&gt;&lt;strong&gt;What is it used for?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Data ingestion into a distributed datastore (e.g. HDFS). See image (I did not make the image and am only including the image for visual aid). There are other tools that will help you with the ingestion of data as well (Storm and Sqoop are mentioned).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;At which stage of a BigData analysis is it used?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;It is used for data ingestion into your distributed datastore (e.g. HDFS). So for example a webserver is running logging information into /var/logs/webserver.log. Apache Flume can look at that file, grab what it needs out of it and send it to HDFS. Once the data gets put into your datastore you can then utilize other tools to analyze the imported data (e.g. Hive, Pig, MR, etc.)&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;And what are prerequisites for learning it?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Understanding of how to write scripts, edit configuration settings, and your way around Linux would be the absolute minimum to get started. This set of instructions are old but a starting point would be to look at the hortwonworks tutorial on flume. &lt;a href=&quot;http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/&quot; rel=&quot;noreferrer&quot;&gt;http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;If you would like me to elaborate more I would be glad to but I wanted to try and meet your requirement of a simple, short explanation.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/raU0I.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/raU0I.jpg&quot; alt=&quot;HDP Data Lake Solution Architecture&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14829" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-01-11T18:35:57.990" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9751" PostTypeId="2" ParentId="9736" CreationDate="2016-01-11T19:04:01.747" Score="12" Body="&lt;p&gt;SVM can be used for classification (distinguishing between several groups or classes) and regression (obtaining a mathematical model to predict something). They can be applied to both linear and non linear problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Until 2006 they were the best general purpose algorithm for machine learning.  I was trying to find a paper that compared many implementations of the most known algorithms: svm, neural nets, trees, etc. I couldn't find it sorry (you will have to believe me, bad thing). In the paper the algorithm that got the best performance was svm, with the library libsvm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 2006 Hinton came up with deep learning and neural nets. He improved the current state of the art by at least 30%, which is a huge advancement. However deep learning only get good performance for huge training sets. If you have a small training set I would suggest to use svm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore you can find here a useful infographic about &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;noreferrer&quot;&gt;when to use different machine learning algorithms&lt;/a&gt; by scikit-learn. However, to the best of my knowledge there is no agreement among the scientific community about if a problem has X,Y and Z features then it's better to use svm. I would suggest to try different methods. Also, please don't forget that svm or neural nets is just a method to compute a model. It is very important as well the features you use.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastEditorUserId="14994" LastEditDate="2016-01-12T09:30:54.953" LastActivityDate="2016-01-12T09:30:54.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9752" PostTypeId="2" ParentId="9745" CreationDate="2016-01-11T19:05:27.367" Score="4" Body="&lt;p&gt;I noticed you are asking several questions related to the Hadoop ecosystem. It is hard to judge how much you know while explaining the things you are asking in a very simplistic manner without knowing that. Might be worthwhile giving some background of what you are trying to understand. But I digress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you are building an application that requires coordination (think race conditions) you more than likely won't need Zookeeper directly. Many tools within the Hadoop ecosystem do use it for this exact purpose (e.g. HBase).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is probably the best and most simplistic answer I can give. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In a nutshell, ZooKeeper gives you the tools to help build distributed applications. For example, suppose you have a distributed web server application running on 10 nodes. Say, you want to get total real-time hit count. One way to do this is to write an application which connects to the 10 nodes, gets count from each and present the sum. Alternatively, you can have each web server application write their hit counts to ZooKeeper on regular intervals and then query ZooKeeper to get the count.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://10kloc.wordpress.com/2013/12/21/quick-intro-to-apache-zookeeper/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://10kloc.wordpress.com/2013/12/21/quick-intro-to-apache-zookeeper/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ZooKeeper Quorum is &quot;A replicated group of servers in the same application is called a quorum, and in replicated mode, all servers in the quorum have copies of the same configuration file.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more detailed answer of what Zookeeper is and how it is beneficial can be found on SO (&lt;a href=&quot;https://stackoverflow.com/questions/3662995/explaining-apache-zookeeper&quot;&gt;https://stackoverflow.com/questions/3662995/explaining-apache-zookeeper&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="14829" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-11T19:18:07.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9753" PostTypeId="1" CreationDate="2016-01-11T22:50:16.850" Score="7" ViewCount="1663" Body="&lt;p&gt;I'm referring to the gamma in the Value function:&#xA;&lt;img src=&quot;https://image.slidesharecdn.com/ml-sep-09-091009141615-phpapp01/95/regretbased-reward-elicitation-for-markov-decision-processes-42-728.jpg?cb=1255098159&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="13165" LastEditorUserId="-1" LastEditDate="2017-03-10T09:42:41.897" LastActivityDate="2021-08-02T08:58:38.303" Title="How do I choose a discount factor in Markov Decision Problems?" Tags="|machine-learning|markov-process|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9755" PostTypeId="2" ParentId="9746" CreationDate="2016-01-12T05:59:35.087" Score="-1" Body="&lt;p&gt;It might be easier for you to take a look at the events streaming services that are offered in Amazon Web Services (AWS), mainly Kinesis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest of them in Kinesis Firehose (&lt;a href=&quot;http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html&quot; rel=&quot;nofollow&quot;&gt;http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html&lt;/a&gt;), which gives you an endpoint where you can write your events to, and they will appear in Amazon Redshift, which is a data warehouse service that can scale to huge sizes (&lt;a href=&quot;https://aws.amazon.com/redshift/&quot; rel=&quot;nofollow&quot;&gt;https://aws.amazon.com/redshift/&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;AWS also announced that they will add Kinesis Analytics, which will allow you to write SQL code on top of the events stream of Kinesis. You can calculate different statistics such as moving agerages and other relevant statistic on your incoming events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These events that are processed in near real time, can be clicks on your web site, purchases in your e-commerce site, security events that should be monitored for potential intrusion, etc. &lt;/p&gt;&#xA;" OwnerUserId="15300" LastActivityDate="2016-01-12T05:59:35.087" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9756" PostTypeId="1" CreationDate="2016-01-12T06:38:51.017" Score="2" ViewCount="99" Body="&lt;p&gt;Does Apche Kafka support sockets-based communication?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: After running both Kafka producer and consumer.if I send a message through putty(TCP/IP client), will the consumer receive it or not?&lt;/p&gt;&#xA;" OwnerUserId="15348" LastEditorUserId="11097" LastEditDate="2016-01-12T06:54:45.977" LastActivityDate="2016-01-12T06:54:45.977" Title="Does Apche Kafka support sockets-based communication?" Tags="|apache-kafka|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9757" PostTypeId="2" ParentId="9736" CreationDate="2016-01-12T07:44:43.370" Score="5" Body="&lt;p&gt;Let's assume that we are in a classification setting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;svm&lt;/code&gt; feature engineering is cornerstone:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the sets have to be linearly separable. Otherwise the data needs to be transformed (eg using Kernels). This is not done by the algo itself and might blow out the number of features.&lt;/li&gt;&#xA;&lt;li&gt;I would say that &lt;code&gt;svm&lt;/code&gt; performance suffers as we increase the number of dimensions faster than other methodologies (tree ensemble). This is due to the constrained optimization problem that backs &lt;code&gt;svm&lt;/code&gt;s. Sometimes feature reduction is feasible, sometimes not and this is when we can't really pave the way for an effective use of &lt;code&gt;svm&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;svm&lt;/code&gt; will likely struggle with a dataset where the number of features is much larger than the number of observations. This, again, can be understood by looking at the constrained optimizatiom problem.&lt;/li&gt;&#xA;&lt;li&gt;categorical variables are not handled out of the box by the &lt;code&gt;svm&lt;/code&gt; algorithm. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1244" LastActivityDate="2016-01-12T07:44:43.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9758" PostTypeId="1" AcceptedAnswerId="9760" CreationDate="2016-01-12T09:57:38.883" Score="11" ViewCount="10162" Body="&lt;p&gt;In the below graph,&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;x-axis =&amp;gt; &lt;strong&gt;Data set Size&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;y-axis =&amp;gt; &lt;strong&gt;Cross validation Score&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4VAFr.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4VAFr.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Red line&lt;/strong&gt; is for Training Data&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Green line&lt;/strong&gt; is for Testing Data&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;In a tutorial that I'm referring to, the author says that the point where the &lt;strong&gt;red line&lt;/strong&gt; and the &lt;strong&gt;green line&lt;/strong&gt; overlap means,&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Collecting more data is unlikely to increase the generalization&#xA;performance and &lt;strong&gt;we're in a region that we are likely to underfit the&#xA;data. Therefore it makes sense to try out with a model with more capacity&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I cannot quite understand meaning of the &lt;strong&gt;bold&lt;/strong&gt; phrase and how it happens.&lt;/p&gt;&#xA;&lt;p&gt;Appreciate any help.&lt;/p&gt;&#xA;" OwnerUserId="15115" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2020-02-19T13:45:45.400" Title="Overfitting/Underfitting with Data set size" Tags="|machine-learning|cross-validation|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9760" PostTypeId="2" ParentId="9758" CreationDate="2016-01-12T12:09:19.850" Score="7" Body="&lt;p&gt;So, the underfitting means that you still have capacity for improving your learning while overfitting means that you have used a capacity more than needed for learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Green area is where testing error is rising i.e. you should continue providing capacity (either data points or model complexity) to gain better results. More green line goes, more flat it becomes i.e. you are reaching the point where the provided capacity (which is data) is enough and better to try providing the other type of capacity which is model complexity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it does not improve your test score or even reduce it that means that the combination of Data-Complexity was somehow optimal and you can stop training.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-12T12:09:19.850" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9761" PostTypeId="1" AcceptedAnswerId="9763" CreationDate="2016-01-12T13:01:58.427" Score="1" ViewCount="80" Body="&lt;p&gt;I want to cluster a 5 feature data-set. Firstly to explore the data I did a correlation matrix to see if some features where highly correlated so I could reduce them. Then I saw a feature that have close to zero correlation against all the other features. This got me wondering if I should exclude this parameter since it acts as a kind of &quot;noise&quot; relatively to all the other features. What's your opinion? &lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-01-12T18:38:23.660" Title="Feauture selection for clustering regarding zero-correlated feature" Tags="|clustering|feature-selection|correlation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9762" PostTypeId="1" AcceptedAnswerId="9766" CreationDate="2016-01-12T13:54:34.180" Score="5" ViewCount="3823" Body="&lt;p&gt;I have the following variables along with sales data going back a few years:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;date # simple date, can be split in year, month etc &lt;/li&gt;&#xA;&lt;li&gt;shipping_time  (0-6 weeks) # 0 weeks means in stock, more weeks means the product is out of stock but a shipment is on the way to the warehouse. Longer shipping times have a siginificant impact on sales. &lt;/li&gt;&#xA;&lt;li&gt;sales # amount of products sold&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I need to predict the sales (which vary seasonally) while taking into account the shipping time. What would be a simple regression model that would produce reasonable results? I tried linear regression with only date and sales, but this does not account for seasonality, so the prediction is rather weak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: As a measure of accuracy, I will withold a random sample of data from the input and compare against the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extra points if it can be easily done in python/scipy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data can look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;--------------------------------------------------&#xA;| date           | delivery_time| sales          |&#xA;--------------------------------------------------&#xA;| 2015-01-01     | 0            |10              |&#xA;--------------------------------------------------&#xA;| 2015-01-01     | 7            |2               |&#xA;--------------------------------------------------&#xA;| 2015-01-02     | 7            |3               |&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15356" LastEditorUserId="15356" LastEditDate="2016-01-12T15:21:36.423" LastActivityDate="2017-07-25T20:52:35.583" Title="Best regression model to use for sales prediction" Tags="|predictive-modeling|regression|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9763" PostTypeId="2" ParentId="9761" CreationDate="2016-01-12T14:41:46.947" Score="1" Body="&lt;p&gt;Lack of correlation with other features is not a reason to omit a feature. On the contrary, it is usually a reason to keep the feature because it may provide unique information. Typically, highly correlated features provide redundant information and feature reduction techniques (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;Principal Components Analysis&lt;/a&gt;) are used to remove the redundancy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it is &lt;em&gt;possible&lt;/em&gt; that the uncorrelated feature is noise, you should not make that assumption. It could be that the uncorrelated feature is the &lt;em&gt;only&lt;/em&gt; one containing information and the other 4 features are all correlated noise.&lt;/p&gt;&#xA;" OwnerUserId="964" LastEditorUserId="964" LastEditDate="2016-01-12T18:38:23.660" LastActivityDate="2016-01-12T18:38:23.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9764" PostTypeId="2" ParentId="9734" CreationDate="2016-01-12T16:20:16.453" Score="3" Body="&lt;p&gt;The term may best be expressed as a &lt;strong&gt;Recurrent&lt;/strong&gt;, &lt;strong&gt;Heavy&lt;/strong&gt; &lt;strong&gt;Subgraph&lt;/strong&gt;. That is, a subgraph which is both &lt;strong&gt;Recurrent&lt;/strong&gt; &lt;em&gt;and&lt;/em&gt; &lt;strong&gt;Heavy&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Heaviness&lt;/strong&gt; of a subgraph refers to heavily connected vertices- that is, nodes which are connected many times (&quot;many&quot; being relative to the network in question). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Recurrent&lt;/strong&gt; refers to the propensity of a subgraph to occur more than once. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, a &lt;strong&gt;Recurrent Heavy Subgraph&lt;/strong&gt;  is a densely connected set of vertices which occurs several times in the overall network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These subgraphs are often used to determine properties of a network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: In a network of emails interactions within a company organized into 4-person teams with one member acting as the lead, each team's email activity (if they email between themselves sufficiently to be considered &quot;heavy&quot;) could be described as a &lt;strong&gt;Heavy&lt;/strong&gt; &lt;strong&gt;Subgraph&lt;/strong&gt;. The fact that these subgraphs occur many times in the network make them &lt;strong&gt;Recurrent Heavy Subgraphs&lt;/strong&gt;. If one was searching for structure in the network, noticing that these recurrent, heavy subgraphs exist would go a long way toward determining the organization of the network as a whole.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2016-01-12T16:36:45.500" LastActivityDate="2016-01-12T16:36:45.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9765" PostTypeId="2" ParentId="9758" CreationDate="2016-01-12T17:53:48.103" Score="6" Body="&lt;p&gt;While Kasra Manshaei gives a good general answer (+1), I would like to give an easy to understand example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of a very simple problem: Fitting a function $f:[0, 1] \rightarrow \mathbb{R}$. To do so, you take a model out of the polynomial class. For the sake of argument, let's say you take a polynomial of degree 0. This models &lt;strong&gt;capacity&lt;/strong&gt; is very limited as it can only fit constants. It will basically guess the mean value (depends on the error function, of course, but keep it simple). So relatively quick you will have a pretty good estimate of what the best parameters for this kind of model are. Your test- and training error will be almost identical, no matter how many examples you add. The problem is not that you don't have enough data, the problem is that your model is not powerful enough: You &lt;strong&gt;underfit&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So lets go the other way around: Say you have 1000 data points. Knowing a bit of math, you choose a polynomial of degree 999. Now you can fit the training data perfectly. However, your data might just fit the data too perfectly. For example, see (from &lt;a href=&quot;http://martin-thoma.com/spline-interpolation/&quot; rel=&quot;noreferrer&quot;&gt;my blog&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fYSlo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fYSlo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, you have other models which also fit the data perfectly. Obviously, the blue model seems kind of unnatural between the datapoints. The model itself might not be able to capture the kind of distribution well, so restricting the model to something simpler might actually help it. This can be an example of &lt;strong&gt;overfitting&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-12T18:01:21.473" LastActivityDate="2016-01-12T18:01:21.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9766" PostTypeId="2" ParentId="9762" CreationDate="2016-01-12T20:49:00.640" Score="5" Body="&lt;p&gt;This is a pretty classic &lt;strong&gt;ARIMA&lt;/strong&gt; dataset. &lt;strong&gt;ARIMA&lt;/strong&gt; is implemented in the StatsModels package for Python, the documentation for which is available &lt;a href=&quot;http://statsmodels.sourceforge.net/stable/index.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &lt;strong&gt;ARIMA&lt;/strong&gt; model with seasonal adjustment may be the simplest reasonably successful forecast for a complex time series such as sales forecasting. It may (probably will) be that you need to combine the method with an additional model layer to detect additional fluctuation beyond the auto-regressive function of your sales trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, simple linear regression models tend to fare quite poorly on time series data.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2016-01-12T21:15:43.100" LastActivityDate="2016-01-12T21:15:43.100" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9767" PostTypeId="1" CreationDate="2016-01-12T21:19:16.740" Score="1" ViewCount="113" Body="&lt;p&gt;Hey fellow data lovers!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data modelling problem for a clustering analysis that I can't wrap my head around. Perhaps I'm thinking to advanced and I should simplify my analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 data sets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Persons with their properties:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;PersonId&lt;/li&gt;&#xA;&lt;li&gt;Gender&lt;/li&gt;&#xA;&lt;li&gt;Age&lt;/li&gt;&#xA;&lt;li&gt;Country&lt;/li&gt;&#xA;&lt;li&gt;Etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Conversations and their properties:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ConversationId&lt;/li&gt;&#xA;&lt;li&gt;Sentences of the conversations&lt;/li&gt;&#xA;&lt;li&gt;Keyword of a sentence + it's frequency&lt;/li&gt;&#xA;&lt;li&gt;Polarity of the sentence (pos, neg or neutral)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to cluster terms that have the same polarity for a group of people. Eg. male persons between 20 and 35 speak positively about economy and neg about privacy&#xA;Eg. male persons from BE, NL and DE speak neg about climat changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem is that I need to cluster multiple terms for an undefined group of people. For one term (eg. privacy) it would be 'easy' to determine the properties of the different segments as this is a classification problem. As I would like to cluster multiple terms together, I'm strugling to model my data as this means I have multiple records per person (they can talk about multiple terms). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assumption: you can assume I have one record per person per term.&lt;/p&gt;&#xA;" OwnerUserId="15366" LastActivityDate="2017-04-07T05:40:33.170" Title="Datamodel for cluster analysis terms &amp; segmentation" Tags="|classification|clustering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9768" PostTypeId="2" ParentId="9767" CreationDate="2016-01-12T21:30:25.100" Score="0" Body="&lt;p&gt;If you're just looking for a data model, this one should handle your stated requirements:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Person&#xA;    -------------------&#xA;    PersonId&#xA;    Age&#xA;    Gender&#xA;    Country&#xA;    Etc&#xA;    --------------------&#xA;&#xA;    Person_Conversation&#xA;    --------------------&#xA;    PersonId&#xA;    ConversationId&#xA;    --------------------&#xA;&#xA;    Conversation&#xA;    --------------------&#xA;    ConversationId&#xA;    ....&#xA;    --------------------&#xA;&#xA;    ---- if a sentence can be part of more than one conversation&#xA;    ---- use:&#xA;&#xA;    Conversation_Sentence &#xA;    --------------------&#xA;    ConversationId&#xA;    SentenceId&#xA;    --------------------&#xA;&#xA;    Sentence&#xA;    --------------------&#xA;    SentenceId&#xA;    Keyword&#xA;    Polarity&#xA;    --------------------&#xA;&#xA;    ----Otherwise use:&#xA;&#xA;    Sentence&#xA;    --------------------&#xA;    SentenceId&#xA;    ConversationId&#xA;    Keyword&#xA;    Polarity&#xA;    --------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2016-01-12T21:46:33.133" LastActivityDate="2016-01-12T21:46:33.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9769" PostTypeId="1" CreationDate="2016-01-12T23:39:55.800" Score="10" ViewCount="2524" Body="&lt;p&gt;Suppose you have an input layer with n neurons and the first hidden layer has $m$ neurons, with typically $m &amp;lt; n$. Then you compute the actication $a_j$ of the $j$-th neuron in the hidden layer by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_j =  f\left(\sum\limits_{i=1..n} w_{i,j} x_i+b_j\right)$, where $f$ is an activation function like $\tanh$ or $\text{sigmoid}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train the network, you compute the reconstruction of the input, denoted $z$, and minimize the error between $z$ and $x$. Now, the $i$-th element in $z$ is typically computed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;z_i = f\left ( \sum\limits_{j=1..m}  w_{j,i}' a_j+b'_i \right)&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering why are the reconstructed $z$ are usually computed with the same activation function instead of using the inverse function, and why separate $w'$ and $b'$ are useful instead of using tied weights and biases? It seems much more intuitive to me to compute the reconstructed with the inverse activation function $f^{-1}$, e.g., $\text{arctanh}$, as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;z_i' = \sum\limits_{j=1..m} \frac{f^{-1}(a_j)-b_j}{w_{j,i}^T}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that here tied weights are used, i.e., $w' = w^T$, and the biases $b_j$ of the hidden layer are used, instead of introducing an additional set of biases for the input layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a very related question: To visualize features, instead of computing the reconstruction, one would usually create an identity matrix with the dimension of the hidden layer. Then, one would use each column of the matrix as input to a reactivation function, which induces an output in the input neurons. For the reactivation function, would it be better to use the same activation function (resp. the $z_i$) or the inverse function (resp. the $z'_i$)?&lt;/p&gt;&#xA;" OwnerUserId="15372" LastEditorUserId="15372" LastEditDate="2016-01-15T03:50:44.043" LastActivityDate="2016-01-15T15:01:58.393" Title="Why is Reconstruction in Autoencoders Using the Same Activation Function as Forward Activation, and not the Inverse?" Tags="|machine-learning|visualization|deep-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9770" PostTypeId="2" ParentId="6113" CreationDate="2016-01-13T00:10:30.310" Score="1" Body="&lt;p&gt;K-means clustering is known to give local minima, depending on your initial initialisation of the cluster centres.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, k-means segmentation can, I think, be solved globally, since we do not permute anything in finding the solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can see from your comments that you did manage eventually to reach a segmentation. Would you be able to give some feedback, please? Is your solution the best solution? Or did you settle for a good enough solution?&lt;/p&gt;&#xA;" OwnerUserId="15373" LastActivityDate="2016-01-13T00:10:30.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9771" PostTypeId="2" ParentId="9729" CreationDate="2016-01-13T00:37:33.207" Score="4" Body="&lt;p&gt;It seems like you do not have fixed numbers of centroid(clusters) so centroid based clustering for example k-means can not be used in your case. However, you can use density based clustering for example DBSCAN. &lt;/p&gt;&#xA;" OwnerUserId="14416" LastActivityDate="2016-01-13T00:37:33.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9772" PostTypeId="2" ParentId="2527" CreationDate="2016-01-13T01:19:43.893" Score="0" Body="&lt;p&gt;One way to classify the text is by calculating  the Term Frequency and Inverse Document Frequency. You can refer this paper: &lt;a href=&quot;http://www.oracle.com/technetwork/testcontent/feature-preparation-130942.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.oracle.com/technetwork/testcontent/feature-preparation-130942.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14416" LastActivityDate="2016-01-13T01:19:43.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9773" PostTypeId="2" ParentId="9762" CreationDate="2016-01-13T03:28:17.657" Score="2" Body="&lt;p&gt;Did you try time series modelling?  If not, then you should.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I tried linear regression with only date and sales, but this does not&#xA;  account for seasonality&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Moving-average_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;moving average model&lt;/a&gt; is something which would fit nicely to your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as you say that your model is exhibiting seasonality, you need to adjust the moving averages so that it takes the seasonality into account. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the best model for your dataset would be the SARIMA model. It is just the Auto-Regressive Integrated Moving Average (ARIMA) model but with seasonal adjustments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/165169/84191&quot;&gt;Here&lt;/a&gt; is one of the questions which I have answered which further helps you understand minor seasonality and trend adjustments, along with the R code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;a href=&quot;https://www.otexts.org/fpp/8&quot; rel=&quot;nofollow noreferrer&quot;&gt;Further reading&lt;/a&gt;]&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-01-13T03:28:17.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9774" PostTypeId="2" ParentId="9668" CreationDate="2016-01-13T06:02:40.970" Score="0" Body="&lt;p&gt;I am clueless about the difference coming both the implementation yet, but found how to find a decision boundary for the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At First step I scaled the data to a range [0,100] which earlier was [0,1] for the input. &#xA;(Still do not know why scaling creates a difference)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then used a RBF kernal with low gamma and high C value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;svc = svm.SVC(kernel='rbf', gamma=.004, C=1000).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives me the following result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZLppV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZLppV.png&quot; alt=&quot;Separation boundary created using above mentioned kernal&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14789" LastActivityDate="2016-01-13T06:02:40.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9776" PostTypeId="2" ParentId="9769" CreationDate="2016-01-13T13:00:10.607" Score="5" Body="&lt;p&gt;I don't think that your assumption $w' = w^T$ holds. Or rather is not necessary, and if it is done, it is not in order to somehow automatically reverse the calculation to create the hidden layer features. It is not possible to reverse the compression in general, going from n to smaller m, directly in this way. If that was the goal, then you would want a form of matrix inversion, not simple transpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead we just want $w_{ij}$ for the compressed higher-level feature representation, and will discard $w'_{ij}$ after auto-encoder is finished. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;can&lt;/em&gt; set $w' = w^T$ and tie the weights. This can help with regularisation - helping the autoencoder generalise. But it is not necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the autoencoder to function it doesn't actually matter what activation function you use after the layer that you are pre-training, provided the last layer of the autoencoder can express the range of possible inputs. However, you may get varying quality of results depending on what you use, as normal for a neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is quite reasonable to use the same activation function that you are building the pre-trained layer for, as it is the simplest choice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using an inverse function is possible too, but not advisable for sigmoid or tanh, because e.g. arctanh is not defined &amp;lt; -1 or &gt; 1, so likely would not be numerically stable.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-01-15T15:01:58.393" LastActivityDate="2016-01-15T15:01:58.393" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9777" PostTypeId="1" CreationDate="2016-01-13T13:46:57.730" Score="3" ViewCount="10212" Body="&lt;p&gt;I have one categorical variable of string type in my dataset. I need to convert it to numerical value for further processing. I know standard way to represent categorical data is to use one-hot encoding. But that will convert each entry of the variable to a vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;LabelEncoder of sklearn converts each entry to a scalar value. I realise this is a very naive and possibly stupid question but which representation is more commonly used and is there a reason for the bias?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2018-02-08T15:58:02.240" Title="One-Hot Vector representation vs Label Encoding for Categorical Variables" Tags="|categorical-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9778" PostTypeId="2" ParentId="9777" CreationDate="2016-01-13T14:09:03.943" Score="7" Body="&lt;p&gt;The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode &quot;red&quot; as 1, &quot;blue&quot; as 2 and &quot;green&quot; as 3, but there is no reason why red is more similar to blue than to green).&lt;/p&gt;&#xA;" OwnerUserId="12909" LastActivityDate="2016-01-13T14:09:03.943" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9779" PostTypeId="2" ParentId="9724" CreationDate="2016-01-13T16:05:14.540" Score="2" Body="&lt;h2&gt;Should I use Self Organising Map to reduce the dimensionality?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Not the first choice. For dimensionality reduction you have more handy and basic methods like &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Non-negative_matrix_factorization&quot; rel=&quot;nofollow&quot;&gt;NMF&lt;/a&gt; (if you don't have negative values) or &lt;a href=&quot;https://www.researchgate.net/profile/Christian_Bauckhage/publication/262898400_Kernel_Archetypal_Analysis_for_Clustering_Web_Search_Frequency_Time_Series/links/5416e0770cf2bb7347db7a99.pdf&quot; rel=&quot;nofollow&quot;&gt;Archetypal Analysis&lt;/a&gt; which are recommended to be used first.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;and then K-means to classify them into normal and abnormal?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Red alarm! Kmeans is not for classification but clustering. Be careful about the fundamental understanding of what you are about to do otherwise you get confused. If you have labels K-NN for instance is a supervised (thus proper) method.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Impossible. As you already know you have the problem called &lt;a href=&quot;https://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;Curse of Dimensionality&lt;/a&gt; where the euclidean distances are highly distorted. You need to reduce the dimenstionality before K-NN. The only class of methods I know which may not need dimentionality reduction are &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_method&quot; rel=&quot;nofollow&quot;&gt;Kernel Methods&lt;/a&gt; e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;SVM&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;How many features can I use with K-NN?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Once I examined the Curse of Dimentionality with K-Means in MATLAB and I saw after 7 dimensions everything was distorted. It of course depends on the nature of the data (distribution,etc) but this is what I got from my experiment.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This question is so fuzzy and about the goal of Machine Learning. In Machine Learning you get a set of answers (called labels) and try to learn them. You are not supposed to say if they are right or not. So if you are said a specific guy is abnormal you should ask the one who provided labels (in the terminology we call him &lt;em&gt;expert&lt;/em&gt;) about the reason. If a guy with a certain weight and hight is called abnormal you only know this weight and hight indicate abnormal people. Why? Well, who knows?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I did not get the last question correctly please drop me a line in comments. Would be glad to help.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My Suggestion&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I recommend you to apply PCA to your data and take the first 5 PCs and feed them to K-NN or &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt;. You can plot some of these PCs against each other and see how they discriminate your classes. It gives you a good impression of what is happening. You can also plot the variance or entropy of all features beforehand and drop those features whose variance or entropy is lower than a threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-13T16:05:14.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9780" PostTypeId="1" CreationDate="2016-01-13T16:20:48.900" Score="2" ViewCount="718" Body="&lt;p&gt;I am trying to identify 6 clusters in a graph of authors. Authors are connected with an edge when they have co-authored on a paper. I have already created a clustering with K-means (6-Means) and with Agglomerative clustering. Now, I want to combine these two clustering approaches in order to get more accurate clusters.&#xA;I am able to find corresponding clusters, but I have no clue on how to decide to which cluster an author belongs when the algorithms do not agree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly I could use some form of a linear combination, but I do not know how to apply this properly. Other methods might be possible, but I am not sure on that.&lt;/p&gt;&#xA;" OwnerUserId="15393" LastActivityDate="2021-12-27T16:00:50.090" Title="Combining K-means clustering with Agglomerative clustering" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9781" PostTypeId="2" ParentId="6676" CreationDate="2016-01-13T16:28:38.463" Score="3" Body="&lt;p&gt;You should also do a grid search for the &quot;alpha&quot; hyperparameter for the SGDClassifier. It is explicitly mentioned in the sklearn documentation and from my experience has a big impact on accuracy. &#xA;Second hyperparameter you should look at is &quot;n_iter&quot; - however I saw a smaller effect with my data.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-01-13T16:28:38.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9782" PostTypeId="2" ParentId="9724" CreationDate="2016-01-13T16:41:50.893" Score="0" Body="&lt;p&gt;Without labels, I guess you are looking for Unsupervised anomaly detection technique. You may want to do some research on this topic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be very careful that you are not &quot;classifying&quot; since your data is not labeled. You are actually looking for outliers of the cluster of normal people. Your data is not labeled so any supervised technique is not applicable including K-NN. Clustering methods like K-means can be used, and you want to look for the outliers and label them as &quot;abnormal&quot;. There are other techniques available like one class support vector machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With labels, it become classification problem. Any classification methods can be applied. Feature selection can be done to remove irrelevant or correlated features.&lt;/p&gt;&#xA;" OwnerUserId="9489" LastActivityDate="2016-01-13T16:41:50.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9783" PostTypeId="1" AcceptedAnswerId="9787" CreationDate="2016-01-14T04:12:54.237" Score="4" ViewCount="1089" Body="&lt;p&gt;I'm new to machine learning and have spent the last couple months having a blast using Sci-Kit Learn to try to understand the basics of building feature sets and predictive models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm trying to use ML on a data set not to predict future values but to understand the importance and direction (positive or negative) of each feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My features (X) are boolean and integer values that describe a product. My target (y) is the sales of the product. I have ~15,000 observations with 16 features a piece.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With my limited ML knowledge to this point, I'm confident that I can predict (with some level of accuracy) a new y based on a new set of features X. However I'm struggling to coherently identify, report on and present &lt;em&gt;the importance and direction of each feature that makes up X&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus far, I've taken a two-step approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use a linear regression to observe coefficients&lt;/li&gt;&#xA;&lt;li&gt;Use a random forest to observe feature importance&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I try to get the directional impact of each feature:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import linear_model&#xA;linreg = linear_model.LinearRegression()&#xA;linreg.fit(X, y)&#xA;coef = linreg.coef_&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second, I try to get the importance of each feature:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import ensemble&#xA;forest = ensemble.RandomForestRegressor()&#xA;forest.fit(X, y)&#xA;importance = forest.feature_importances_&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I multiply the two derived values together for each feature and end up with some value that maybe perhaps could be the information I'm looking for!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd love to know if I'm on the right track with any of this. Is this a common use case for ML? Are there tools, ideas, packages I should focus on to help guide me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="15398" LastActivityDate="2022-02-01T06:13:20.777" Title="Using machine learning specifically for feature analysis, not predictions" Tags="|machine-learning|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9785" PostTypeId="1" AcceptedAnswerId="10417" CreationDate="2016-01-14T07:13:45.810" Score="28" ViewCount="24108" Body="&lt;p&gt;Given a sentence:&#xA;&amp;quot;When I open the &lt;strong&gt;??&lt;/strong&gt; door it starts heating automatically&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;I would like to get the list of possible words in ?? with a probability.&lt;/p&gt;&#xA;&lt;p&gt;The basic concept used in word2vec model is to &amp;quot;predict&amp;quot; a word given surrounding context.&lt;/p&gt;&#xA;&lt;p&gt;Once the model is build, what is the right context vectors operation to perform my prediction task on new sentences?&lt;/p&gt;&#xA;&lt;p&gt;Is it simply a linear sum?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model.most_similar(positive=['When','I','open','the','door','it','starts' ,&#xA;                   'heating','automatically'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15402" LastEditorUserId="29169" LastEditDate="2021-02-09T20:17:39.023" LastActivityDate="2021-02-09T20:17:39.023" Title="Predicting a word using Word2vec model" Tags="|nlp|predictive-modeling|word-embeddings|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9787" PostTypeId="2" ParentId="9783" CreationDate="2016-01-14T09:05:42.040" Score="5" Body="&lt;p&gt;You don't need the linear regression to understand the effect of features in your random forest, you're better off looking at the partial dependence plots directly, this what you get when you hold all the variables fixed, and you vary one at a time. You can plot these using &lt;code&gt;sklearn.ensemble.partial_depence.plot_partial_dependence&lt;/code&gt;. Take a look at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html#partial-dependence&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt; for an example of how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another type of model that can be useful for exploratory data analysis is a &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;, you can produce a graphical representation of this using &lt;code&gt;export_graphviz&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10975" LastActivityDate="2016-01-14T09:05:42.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9791" PostTypeId="1" AcceptedAnswerId="9792" CreationDate="2016-01-14T13:48:53.553" Score="1" ViewCount="216" Body="&lt;p&gt;Given person's name, e.g. 'Adjutor Ferguson'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to define is it a male or female?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution came to my mind:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     I have found Person NLP training dataset here mbejda.github.io.&#xA;&#xA;      And via a machine learning software like Apache Mahout, train it and provide real data.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I am not sure about the accuracy of the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May be another approach exist? (e.g. scikit-learn.org)&lt;/p&gt;&#xA;" OwnerUserId="15410" LastActivityDate="2016-01-14T14:34:22.810" Title="How to define person's gender from the fullname?" Tags="|classification|nlp|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9792" PostTypeId="2" ParentId="9791" CreationDate="2016-01-14T14:34:22.810" Score="1" Body="&lt;p&gt;That dataset looks like a good starting point. Keep in mind that when you make your own dataset from those datasets you'll want to keep the male to female ratio balanced if you want it to predict both well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should not matter what machine learning software you use (Apache Mahout, scikit-learn, weka, etc.). Pick one that fits your language of choice since speed will probably not be too much of a concern with the smallish dataset size. As for features, you'd generally use ngrams as your baseline for NLP classification tasks. If you use ngrams here you won't end up with anything very interesting because the model won't generalize to any unseen names. I'd suggest as a feature baseline that you try character ngrams, and maybe something like syllable ngrams for something slightly more advanced (for syllable tokenization see &lt;a href=&quot;https://stackoverflow.com/questions/405161/detecting-syllables-in-a-word&quot;&gt;https://stackoverflow.com/questions/405161/detecting-syllables-in-a-word&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="13354" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-14T14:34:22.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9793" PostTypeId="1" AcceptedAnswerId="9796" CreationDate="2016-01-14T16:37:46.247" Score="1" ViewCount="1216" Body="&lt;p&gt;I am new to scikit-learn. I went through the examples given in the docs and I downloaded the script for recognizing images of hand-written digits. When I made the script to run on my laptop, I got the following errors:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;   File &amp;quot;C:\Python34\plot_digits_classification.py&amp;quot;, line 22, in &amp;lt;module&amp;gt;&#xA;    from sklearn import datasets, svm, metrics&#xA;    File &amp;quot;C:\Python34\lib\site-packages\sklearn\__init__.py&amp;quot;, line 57, in&amp;lt;module&amp;gt;&#xA;    from .base import clone&#xA;   File &amp;quot;C:\Python34\lib\site-packages\sklearn\base.py&amp;quot;, line 11, in &amp;lt;module&amp;gt;&#xA;  from .utils.fixes import signature&#xA;  File &amp;quot;C:\Python34\lib\site-packages\sklearn\utils\__init__.py&amp;quot;, line 11, in &amp;lt;module&amp;gt;&#xA;   from .validation import (as_float_array,&#xA;   File &amp;quot;C:\Python34\lib\site-packages\sklearn\utils\validation.py&amp;quot;, line 16, in &amp;lt;module&amp;gt;&#xA;   from ..utils.fixes import signature&#xA;  File &amp;quot;C:\Python34\lib\site-packages\sklearn\utils\fixes.py&amp;quot;, line 324, in &amp;lt;module&amp;gt;&#xA;   from scipy.sparse.linalg import lsqr as sparse_lsqr&#xA;   File &amp;quot;C:\Python34\lib\site-packages\scipy\sparse\linalg\__init__.py&amp;quot;, line 109, in &amp;lt;module&amp;gt;&#xA;   from .isolve import *&#xA;  File &amp;quot;C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\__init__.py&amp;quot;, line 6, in &amp;lt;module&amp;gt;&#xA;  from .iterative import *&#xA;  File &amp;quot;C:\Python34\lib\site-packages\scipy\sparse\linalg\isolve\iterative.py&amp;quot;, line 7, in &amp;lt;module&amp;gt;&#xA;  from . import _iterative&#xA;  ImportError: DLL load failed: The specified module could not be found.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Please help me. Also, I want to know if I want to load data for prediction, how should I do that? For example, if I want to test a hand written digit that is stored somewhere on my disk, how to prepare that data for loading and passing into this model for prediction?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-30T16:04:09.527" LastActivityDate="2020-12-30T16:04:09.527" Title="Running examples from scikit-learn tutorials" Tags="|machine-learning|python|scikit-learn|prediction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9794" PostTypeId="2" ParentId="6676" CreationDate="2016-01-14T16:39:58.943" Score="23" Body="&lt;p&gt;The comments about iteration number are spot on. The default &lt;code&gt;SGDClassifier&lt;/code&gt; &lt;code&gt;n_iter&lt;/code&gt; is &lt;code&gt;5&lt;/code&gt; meaning you do &lt;code&gt;5 * num_rows&lt;/code&gt; steps in weight space. The &lt;a href=&quot;http://scikit-learn.org/stable/modules/sgd.html#tips-on-practical-use&quot; rel=&quot;noreferrer&quot;&gt;sklearn rule of thumb&lt;/a&gt; is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with &lt;code&gt;SGDClassifier&lt;/code&gt; because it's hitting iteration limit before tolerance so you are &quot;early stopping&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modifying your code quick and dirty I get:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Added n_iter here&#xA;params = [{}, {&quot;loss&quot;: &quot;log&quot;, &quot;penalty&quot;: &quot;l2&quot;, 'n_iter':1000}]&#xA;&#xA;for param, Model in zip(params, Models):&#xA;    total = 0&#xA;    for train_indices, test_indices in kf:&#xA;        train_X = X[train_indices, :]; train_Y = Y[train_indices]&#xA;        test_X = X[test_indices, :]; test_Y = Y[test_indices]&#xA;        reg = Model(**param)&#xA;        reg.fit(train_X, train_Y)&#xA;        predictions = reg.predict(test_X)&#xA;        total += accuracy_score(test_Y, predictions)&#xA;&#xA;    accuracy = total / numFolds&#xA;    print &quot;Accuracy score of {0}: {1}&quot;.format(Model.__name__, accuracy)&#xA;&#xA;Accuracy score of LogisticRegression: 0.96&#xA;Accuracy score of SGDClassifier: 0.96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="92" LastEditorUserId="92" LastEditDate="2017-11-30T18:17:59.610" LastActivityDate="2017-11-30T18:17:59.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9795" PostTypeId="2" ParentId="9793" CreationDate="2016-01-14T18:10:37.080" Score="1" Body="&lt;p&gt;Seems like you've got to install the &lt;a href=&quot;http://scikit-learn.org/stable/install.html&quot; rel=&quot;nofollow&quot;&gt;module,&lt;/a&gt; for example with the &lt;code&gt;pip&lt;/code&gt; package manager. Perhaps you could install anaconda, which is highly recommended&lt;/p&gt;&#xA;" OwnerUserId="1244" LastEditorUserId="1244" LastEditDate="2016-01-14T18:12:34.773" LastActivityDate="2016-01-14T18:12:34.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9796" PostTypeId="2" ParentId="9793" CreationDate="2016-01-14T18:51:20.203" Score="0" Body="&lt;p&gt;I installed older version of numpy,that's the problem. If you have installed scikit-learn using windows binaries, then you must first install numpy+mkl from the windows binaries site. It is a prerequisite for scikit-learn.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-01-14T18:51:20.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9797" PostTypeId="1" CreationDate="2016-01-14T20:18:07.897" Score="4" ViewCount="1219" Body="&lt;p&gt;I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning.&#xA;I'm mostly interested in methods of sharing Q-table between processes (or joining/syncing them together if each process have it's own).&#xA;I'd also appreciate a brief description of method used in linked/mentioned sources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that I use neural network (PyBrain) as approximation.&lt;/p&gt;&#xA;" OwnerUserId="15197" LastEditorUserId="11097" LastEditDate="2016-01-15T10:47:16.137" LastActivityDate="2016-02-23T15:10:46.217" Title="Parallel Q-learning" Tags="|neural-network|parallel|reinforcement-learning|q-learning|pybrain|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9799" PostTypeId="1" CreationDate="2016-01-15T01:41:59.553" Score="2" ViewCount="40" Body="&lt;p&gt;I have knowledge that is organized as in the example below, where items or nodes can belong to multiple hierarchies and can have arbitrary numbers of children and parents:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sports&#xA;    golf&#xA;    baseball&#xA;    tennis&#xA;equipment&#xA;    golf&#xA;        clubs&#xA;            club1&#xA;    baseball&#xA;        bats&#xA;            bat1&#xA;    tennis&#xA;        racket&#xA;            racket1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to be able to run queries like &quot;Show me all sports that use clubs or rackets, and their associated equipment.&quot;  The queries do not need to be posed in natural language, but the results must be displayed in a hierarchy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sports&#xA;    golf&#xA;        club1&#xA;    tennis&#xA;        racket1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm going crazy trying to figure out what I need.  It doesn't seem to quite be a relational database, an ontology, or a taxonomy.  Or maybe I don't quite understand how to use one or more of those to do what I need.  What standard approach is most similar to what I'm trying to do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;As always with stackexchange questions, I'm utterly terrified of asking a question that doesn't fit the exact requirements of a particular forum.  I tried to make it as specific and answerable as possible.  I BEG YOU PLEASE, do not delete or lock this question without at least explaining why it is unacceptable or offering a suggestion for making it acceptable.  Thank you!&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="15420" LastEditorUserId="15420" LastEditDate="2016-01-15T03:13:35.397" LastActivityDate="2016-06-13T11:15:07.270" Title="What knoweldge framework supports this type of query?" Tags="|knowledge-base|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9800" PostTypeId="2" ParentId="9799" CreationDate="2016-01-15T08:11:59.377" Score="1" Body="&lt;p&gt;Well, to answer my own question, it turns out the issues I was struggling with are almost exactly the issues that motivated the development of and that are addressed by &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_database&quot; rel=&quot;nofollow&quot;&gt;graph databases&lt;/a&gt;.  I would have been out of luck if this was 1999, but it turns out 2016 is a good time to be alive for databases that go beyond the traditional relational model!  I guess all this NoSQL stuff that the kids are always talking about is more than just hype.  :)&lt;/p&gt;&#xA;" OwnerUserId="15420" LastActivityDate="2016-01-15T08:11:59.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9802" PostTypeId="1" AcceptedAnswerId="14550" CreationDate="2016-01-15T11:00:41.637" Score="13" ViewCount="8802" Body="&lt;p&gt;What is the difference in R in xgboost between binary:logistic and reg:logistic? Is it only in evaluation metric? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, how does RMSE on binary classification compare to error rate? Is the relationship between the metrics more or less monotonic, output from tuning on one metric should not differ significantly between those two approaches?&lt;/p&gt;&#xA;" OwnerUserId="15430" LastActivityDate="2016-10-15T22:54:42.727" Title="What is the difference in xgboost binary:logistic and reg:logistic" Tags="|r|logistic-regression|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9804" PostTypeId="2" ParentId="9406" CreationDate="2016-01-15T17:19:25.270" Score="2" Body="&lt;p&gt;Regression trees, particularly gradient boosting (essentially many trees), tend to do very well on continuous predictions, often outperforming models that are truly continuous like linear regression when. This is especially true when there are variable interactions and when you have a sufficiently large dataset (over 10,000 records) so that overfitting is less likely. If your primary objective is simply predictive power, then whether the model is 100% continuous or pseudo continuous should be irrelevant. If making your regression trees more continuous enhances out of sample predictive power, then you can simply increase tree depth or add more trees.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-01-15T17:19:25.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9806" PostTypeId="2" ParentId="6726" CreationDate="2016-01-15T21:51:27.033" Score="0" Body="&lt;p&gt;1) Like 0xF suggested: &#xA;Please check the distribution of the label you are predicting i.e. number of 0's and 1's.&#xA;If there's a class imbalance problem and you have assigned equal weight to all instances, this problem is most likely to occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Assuming, you took care of class imbalance, also, use the function predict_proba() along with predict().&#xA;Now, take the ration of prob(1) by prob(0). It doesn't matter in case of divide by zero error as you can replace those numbers by some very large number. Now, adjust the margins of classification according to your convenience to minimize false positives and false negatives.&lt;/p&gt;&#xA;" OwnerUserId="14763" LastActivityDate="2016-01-15T21:51:27.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9810" PostTypeId="1" CreationDate="2016-01-16T16:50:36.893" Score="2" ViewCount="220" Body="&lt;p&gt;I have a dataset from some health institues. The data contains information about malaria cases on a 52 week range. The dataset has 52 columns, one for each week and about 16 rows one for each hospital reporting the number of cases in the particular week diagnosed in that hospital. Example of the dataset with 9 weeks entries :-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SAD Lakhwar 0   0   0   0   0   1   4   3   1&#xA;Rural Health Center 2   0   0   6   0   2   0   2   2&#xA;Herbertpur Christian Hospital   1   0   1   0   2   0   1   0   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have used Hierarchical and K-Means clustering for both determining clusters of hospitals as well as clusters of weeks but my real goal is to cluster in such a manner that outbreaks can be detected using data from consecutive weeks and at the same time the cluster of hospitals in which the outbreak is noticed is also found out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The techniques I have used till now find me clusters where in some cases weeks are away from each other, for example the 7th and 37th week fall into the same cluster as can be seen in the output given below but I want to achieve continuity in weeks because outbreaks span over a few weeks I understand the reason of the results I am getting but I want continuity, if anyone could help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result of trying to cluster the weeks into 4 clusters using k means&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Week No x&#xA;1   2&#xA;2   4&#xA;3   4&#xA;4   2&#xA;5   4&#xA;6   2&#xA;7   1&#xA;8   2&#xA;9   2&#xA;10  2&#xA;11  2&#xA;12  4&#xA;13  1&#xA;14  4&#xA;15  4&#xA;16  4&#xA;17  1&#xA;18  4&#xA;19  1&#xA;20  4&#xA;21  1&#xA;22  1&#xA;23  1&#xA;24  1&#xA;25  1&#xA;26  1&#xA;27  1&#xA;28  1&#xA;29  1&#xA;30  1&#xA;31  1&#xA;32  3&#xA;33  1&#xA;34  1&#xA;35  1&#xA;36  1&#xA;37  1&#xA;38  3&#xA;39  3&#xA;40  3&#xA;41  3&#xA;42  3&#xA;43  3&#xA;44  3&#xA;45  3&#xA;46  3&#xA;47  3&#xA;48  3&#xA;49  3&#xA;50  1&#xA;51  4&#xA;52  4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dput of the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(V1 = structure(c(13L, 15L, 6L, 10L, 3L, 17L, 12L, &#xA;1L, 2L, 11L, 4L, 14L, 8L, 9L, 7L, 5L), .Label = c(&quot;CHC Sahaspur&quot;, &#xA;&quot;Comb. Hosp. Premnagar&quot;, &quot;Doon Hospital&quot;, &quot;FRI Hospital&quot;, &quot;Herbertpur                     Christian Hospital&quot;, &#xA;&quot;HIHT Jollygrant&quot;, &quot;Kalindi Hospital&quot;, &quot;MAX Hospital&quot;, &quot;PHC Herbatpur &quot;, &#xA;&quot;PHC Kalsi&quot;, &quot;PHC Rajawala&quot;, &quot;Rural Health Center&quot;, &quot;SAD Lakhwar&quot;, &#xA;&quot;Shubharti Hospital&quot;, &quot;SPS Rishikesh&quot;, &quot;Total&quot;, &quot;Vaish Nursing Home&quot;&#xA;), class = &quot;factor&quot;), V2 = c(0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, &#xA;0, 0, 0, 0, 0), V3 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA;0, 0, 0), V4 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA;0), V5 = c(0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0), V6 = c(0, &#xA;0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), V7 = c(1, 0, 0, &#xA;0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0), V8 = c(4, 0, 0, 0, 0, &#xA;0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0), V9 = c(3, 0, 0, 0, 0, 0, 2, &#xA;0, 0, 0, 0, 0, 1, 0, 0, 0), V10 = c(1, 0, 0, 0, 0, 1, 2, 0, 0, &#xA;0, 0, 0, 0, 0, 0, 0), V11 = c(0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, &#xA;0, 0, 0, 0, 0), V12 = c(0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, &#xA;0, 0, 0), V13 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA;0), V14 = c(0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V15 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V16 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V17 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V18 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1), &#xA;V19 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V20 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V21 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V22 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2), &#xA;V23 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0), &#xA;V24 = c(2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V25 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0), &#xA;V26 = c(0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1), &#xA;V27 = c(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V28 = c(0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V29 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0), &#xA;V30 = c(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V31 = c(0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V32 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V33 = c(0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), &#xA;V34 = c(5, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0), &#xA;V35 = c(1, 0, 22, 0, 1, 1, 0, 0, 0, 0, 0, 38, 0, 2, 0, 0), &#xA;V36 = c(0, 2, 4, 2, 1, 0, 0, 0, 0, 0, 0, 23, 0, 2, 0, 1), &#xA;V37 = c(0, 0, 10, 0, 2, 0, 0, 0, 0, 0, 0, 10, 0, 2, 0, 7), &#xA;V38 = c(1, 2, 2, 1, 2, 0, 0, 0, 0, 0, 0, 16, 2, 2, 0, 7), &#xA;V39 = c(0, 1, 9, 0, 28, 2, 0, 0, 0, 0, 8, 12, 0, 1, 0, 2), &#xA;V40 = c(1, 0, 2, 0, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V41 = c(0, 0, 3, 0, 10, 0, 0, 0, 1, 0, 0, 18, 0, 0, 0, 1), &#xA;V42 = c(0, 0, 1, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0), &#xA;V43 = c(0, 1, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V44 = c(1, 0, 1, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V45 = c(0, 0, 9, 0, 6, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1), &#xA;V46 = c(0, 0, 4, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V47 = c(0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V48 = c(0, 0, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0), &#xA;V49 = c(0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V50 = c(0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V51 = c(0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V52 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), &#xA;V53 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)), .Names =                 c(&quot;V1&quot;, &#xA;&quot;V2&quot;, &quot;V3&quot;, &quot;V4&quot;, &quot;V5&quot;, &quot;V6&quot;, &quot;V7&quot;, &quot;V8&quot;, &quot;V9&quot;, &quot;V10&quot;, &quot;V11&quot;, &#xA;&quot;V12&quot;, &quot;V13&quot;, &quot;V14&quot;, &quot;V15&quot;, &quot;V16&quot;, &quot;V17&quot;, &quot;V18&quot;, &quot;V19&quot;, &quot;V20&quot;, &#xA;&quot;V21&quot;, &quot;V22&quot;, &quot;V23&quot;, &quot;V24&quot;, &quot;V25&quot;, &quot;V26&quot;, &quot;V27&quot;, &quot;V28&quot;, &quot;V29&quot;, &#xA;&quot;V30&quot;, &quot;V31&quot;, &quot;V32&quot;, &quot;V33&quot;, &quot;V34&quot;, &quot;V35&quot;, &quot;V36&quot;, &quot;V37&quot;, &quot;V38&quot;, &#xA;&quot;V39&quot;, &quot;V40&quot;, &quot;V41&quot;, &quot;V42&quot;, &quot;V43&quot;, &quot;V44&quot;, &quot;V45&quot;, &quot;V46&quot;, &quot;V47&quot;, &#xA;&quot;V48&quot;, &quot;V49&quot;, &quot;V50&quot;, &quot;V51&quot;, &quot;V52&quot;, &quot;V53&quot;), row.names = c(NA, &#xA;16L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13953" LastActivityDate="2016-01-18T20:19:09.400" Title="Clustering weekly data to detect outbreaks by having some sort of continuity among the adjacent weeks" Tags="|r|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9812" PostTypeId="1" CreationDate="2016-01-16T18:54:01.930" Score="2" ViewCount="198" Body="&lt;p&gt;I'm new to data mining, so this might sound like a very simple task to some.&lt;/p&gt;&#xA;&lt;p&gt;I work in reliability engineering in aviation and have a set of data that is generated on a daily basis regarding system failures and failure rectification. This data is categorized using numerical tags of maintenance manual tasks (reference data) by chapter, section, and paragraph. However, since the data is entered manually by people, sometimes, the wrong chapter/section tags are entered and would require being manually checked to ensure the data's validity.&lt;/p&gt;&#xA;&lt;p&gt;The failure/resolution data is available in table format (&lt;code&gt;CSV&lt;/code&gt;, &lt;code&gt;Excel&lt;/code&gt;...) and I also have the maintenance manual data keywords including their chapters/sections in table format.&lt;/p&gt;&#xA;&lt;p&gt;My question is, is it possible using RapidMiner to take these tables, cross-check some keywords in the text (failure/rectification) and compare them with the reference data, and output it with the proper reference tags (of the chapters, sections...), taking into account spelling errors, acronyms, and abbreviations. Or, is there a program/application that is more specialized than RapidMiner to do these functions?&lt;/p&gt;&#xA;&lt;p&gt;Example: Failure on system X was rectified and logged on the database. System X is under chapter 4 section 33; however, when entering the data, the person put it under chapter 3 section 44. I have a document that has the reference for system X being under Chapter 4 Section 33. Is it possible for RapidMiner to check the text in the failure text and the rectification text and cross-check it with a predefined list where system X is under Chapter 4 Section 33 and give me the output with the right Chapter/Section, taking into account spelling mistakes and the fact that some people write abbreviations and acronyms differently (example, &lt;code&gt;I.B.M&lt;/code&gt;/&lt;code&gt;IBM&lt;/code&gt;/&lt;code&gt;I.B.M&lt;/code&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="15456" LastEditorUserId="29169" LastEditDate="2021-07-14T16:24:02.517" LastActivityDate="2021-07-14T16:24:02.517" Title="Natural Language text categorization using RapidMiner" Tags="|classification|data-mining|data-cleaning|rapidminer|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9813" PostTypeId="1" AcceptedAnswerId="9814" CreationDate="2016-01-17T04:36:02.373" Score="1" ViewCount="774" Body="&lt;p&gt;I am new to machine learning. I want to develop a face recognition system using scikit-learn. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html#example-applications-face-recognition-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is the example given in the tutorials of scikit-learn.&lt;/p&gt;&#xA;&lt;p&gt;I am not getting how the input is being provided to the program. How should I load a particular image and make my program run to predict the label for that?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-30T17:06:42.760" LastActivityDate="2020-12-30T17:06:42.760" Title="Face Recognition using Eigenfaces and SVM" Tags="|machine-learning|python|scikit-learn|svm|object-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9814" PostTypeId="2" ParentId="9813" CreationDate="2016-01-17T06:02:00.537" Score="1" Body="&lt;p&gt;Take a look at the code that you linked to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Download the data, if not already on disk and load it as numpy arrays&#xA;&#xA;lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_lfw_people.html#sklearn.datasets.fetch_lfw_people&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;fetch_lfw-poeple&lt;/code&gt;&lt;/a&gt; is a routine that loads the data and is detailed &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_lfw_people.html#sklearn.datasets.fetch_lfw_people&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-01-17T06:02:00.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9815" PostTypeId="1" CreationDate="2016-01-17T08:51:12.870" Score="0" ViewCount="90" Body="&lt;p&gt;In Fuzzy c-means, we have to put how many centers(centroids) in the code. I am wondering how many centroids are suitbale ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to define a suitbale number of centroids for fuzzy c-means? &lt;/p&gt;&#xA;" OwnerUserId="12882" LastActivityDate="2017-11-24T22:15:59.247" Title="How to define a suitbale number of centroids for fuzzy c-means?" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9816" PostTypeId="2" ParentId="9815" CreationDate="2016-01-17T10:45:18.583" Score="1" Body="&lt;p&gt;You can try the exact same heuristics you would use for k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But usually, you would do this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;try some parameters by experience&lt;/li&gt;&#xA;&lt;li&gt;run the algorithm&lt;/li&gt;&#xA;&lt;li&gt;carefully inspect the result&lt;/li&gt;&#xA;&lt;li&gt;go back to 1 and try other parameters until you have found something interesting or are tired of trying&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There is no &quot;correct&quot; solution in clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's about discovering some &lt;em&gt;new&lt;/em&gt; pattern and you cannot out the &quot;new&quot; into an optimization equation. The point is to try, and try again. It is not a drop-in replacement for classification.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-01-17T10:45:18.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9817" PostTypeId="2" ParentId="9810" CreationDate="2016-01-17T11:12:16.170" Score="0" Body="&lt;p&gt;Do you really want clustering here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I'd look at &lt;em&gt;segmentation&lt;/em&gt; and &lt;em&gt;change detection&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you could compute the absolute (or relative) change from one row to another, and segment at the largest change points.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-01-17T11:12:16.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9818" PostTypeId="1" AcceptedAnswerId="9820" CreationDate="2016-01-17T13:04:57.100" Score="60" ViewCount="17439" Body="&lt;p&gt;Neural networks get top results in Computer Vision tasks (see &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot;&gt;MNIST&lt;/a&gt;, &lt;a href=&quot;http://www.image-net.org/challenges/LSVRC/&quot;&gt;ILSVRC&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2014/04/18/winning-the-galaxy-challenge-with-convnets/&quot;&gt;Kaggle Galaxy Challenge&lt;/a&gt;).  They seem to outperform every other approach in Computer Vision. But there are also other tasks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/&quot;&gt;Kaggle Molecular Activity Challenge&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Regression: &lt;a href=&quot;http://blog.kaggle.com/2016/01/04/how-much-did-it-rain-ii-winners-interview-1st-place-pupa-aka-aaron-sim/&quot;&gt;Kaggle Rain prediction&lt;/a&gt;, also the &lt;a href=&quot;http://blog.kaggle.com/2015/12/17/how-much-did-it-rain-ii-2nd-place-luis-andre-dutra-e-silva/&quot;&gt;2nd place&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.kaggle.com/2015/09/29/grasp-and-lift-eeg-detection-winners-interview-2nd-place-daheimao/&quot;&gt;Grasp and Lift 2nd&lt;/a&gt; also &lt;a href=&quot;http://blog.kaggle.com/2015/10/05/grasp-and-lift-eeg-detection-winners-interview-3rd-place-team-hedj/&quot;&gt;third place&lt;/a&gt; - Identify hand motions from EEG recordings&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm not too sure about ASR (automatic speech recognition) and machine translation, but I think I've also heard that (recurrent) neural networks (start to) outperform other approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently learning about Bayesian Networks and I wonder in which cases those models are usually applied. So my question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any challenge / (Kaggle) competition, where the state of the art are Bayesian Networks or at least very similar models?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Side note: I've also seen &lt;a href=&quot;http://blog.kaggle.com/2015/12/21/rossmann-store-sales-winners-interview-1st-place-gert/&quot;&gt;decision trees&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/11/09/profiling-top-kagglers-gilberto-titericz-new-1-in-the-world/&quot;&gt;2&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/10/30/dato-winners-interview-2nd-place-mortehu/&quot;&gt;3&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/10/21/recruit-coupon-purchase-winners-interview-2nd-place-halla-yang/&quot;&gt;4&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/10/20/caterpillar-winners-interview-3rd-place-team-shift-workers/&quot;&gt;5&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/09/28/liberty-mutual-property-inspection-winners-interview-qingchen-wang/&quot;&gt;6&lt;/a&gt;, &lt;a href=&quot;http://blog.kaggle.com/2015/09/22/caterpillar-winners-interview-1st-place-gilberto-josef-leustagos-mario/&quot;&gt;7&lt;/a&gt; win in several recent Kaggle challenges)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-18T12:16:29.323" LastActivityDate="2022-12-07T19:52:40.220" Title="Is there any domain where Bayesian Networks outperform neural networks?" Tags="|machine-learning|pgm|" AnswerCount="9" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9819" PostTypeId="1" CreationDate="2016-01-17T13:14:44.827" Score="17" ViewCount="25856" Body="&lt;p&gt;There's an &lt;code&gt;iter&lt;/code&gt; parameter in the &lt;code&gt;gensim&lt;/code&gt; Word2Vec implementation&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;class gensim.models.word2vec.Word2Vec(sentences=None, size=100,&#xA;alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=0,&#xA;seed=1, workers=1, min_alpha=0.0001, sg=1, hs=1, negative=0,&#xA;cbow_mean=0, hashfxn=&amp;lt;built-in function hash&amp;gt;, **iter=1**, null_word=0,&#xA;trim_rule=None, sorted_vocab=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;that specifies the number of epochs, i.e.:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;iter = number of iterations (epochs) over the corpus.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Does anyone know whether that helps in improving the model over the corpus?&lt;/p&gt;&#xA;&lt;p&gt;Is there any reason why the &lt;code&gt;iter&lt;/code&gt; is set to 1 by default? Is there not much effect in increasing the no. of epochs?&lt;/p&gt;&#xA;&lt;p&gt;Is there any scientific/empirical evaluation of how to set the no. of epochs?&lt;/p&gt;&#xA;&lt;p&gt;Unlike classification/regression task, the grid search method wouldn't really work since the vectors are generated in an unsupervised manner and the objective function is simply by either hierarchical softmax or negative sampling.&lt;/p&gt;&#xA;&lt;p&gt;Is there an early stopping mechanism to cut short the no. of epochs once vectors converges? And can the hierarchical softmax or negative sampling objective converge?&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="29169" LastEditDate="2021-03-11T20:18:49.367" LastActivityDate="2022-12-02T21:32:31.593" Title="Number of epochs in Gensim Word2Vec implementation" Tags="|gensim|word2vec|convergence|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9820" PostTypeId="2" ParentId="9818" CreationDate="2016-01-17T16:42:08.180" Score="38" Body="&lt;p&gt;One of the areas where Bayesian approaches are often used, is where one needs interpretability of the prediction system. You don't want to give doctors a Neural net and say that it's 95% accurate. You rather want to explain the assumptions your method makes, as well as the decision process the method uses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar area is when you have a strong prior domain knowledge and want to use it in the system. &lt;/p&gt;&#xA;" OwnerUserId="15469" LastActivityDate="2016-01-17T16:42:08.180" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9821" PostTypeId="2" ParentId="9818" CreationDate="2016-01-17T16:44:48.993" Score="2" Body="&lt;p&gt;Bayesian networks are preferred for genome interpretation. See, for example, &lt;a href=&quot;https://jscholarship.library.jhu.edu/handle/1774.2/37140&quot; rel=&quot;nofollow&quot;&gt;this dissertation&lt;/a&gt; discussing computational methods for genome interpretation.&lt;/p&gt;&#xA;" OwnerUserId="15470" LastActivityDate="2016-01-17T16:44:48.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9822" PostTypeId="2" ParentId="8869" CreationDate="2016-01-17T17:59:09.063" Score="4" Body="&lt;p&gt;You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. This is equivalent to the database join on an index, but may have less overhead.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-01-17T17:59:09.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9823" PostTypeId="1" CreationDate="2016-01-17T18:26:54.320" Score="21" ViewCount="21677" Body="&lt;p&gt;From Keras RNN Tutorial: &lt;em&gt;&quot;RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this is more a general question about tuning the hyperparameters of a LSTM-RNN on Keras. I would like to know about an approach to finding the best parameters for your RNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I began with the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py&quot;&gt;IMDB example on Keras' Github&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the main model looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,&#xA;                                                      test_split=0.2)&#xA;&#xA;max_features = 20000&#xA;maxlen = 100  # cut texts after this number of words (among top max_features most common words)&#xA;batch_size = 32&#xA;&#xA;model = Sequential()&#xA;model.add(Embedding(max_features, 128, input_length=maxlen))&#xA;model.add(LSTM(128))  &#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(1))&#xA;model.add(Activation('sigmoid'))&#xA;&#xA;# try using different optimizers and different optimizer configs&#xA;model.compile(loss='binary_crossentropy',&#xA;          optimizer='adam',&#xA;          class_mode=&quot;binary&quot;)&#xA;&#xA;print(&quot;Train...&quot;)&#xA;model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=3,&#xA;      validation_data=(X_test, y_test), show_accuracy=True)&#xA;score, acc = model.evaluate(X_test, y_test,&#xA;                        batch_size=batch_size,&#xA;                        show_accuracy=True)&#xA;&#xA;print('Test accuracy:', acc)&#xA;Test accuracy:81.54321846&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;81.5 is a fair score and more importantly it means that the model, even though not fully optimized, it works. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is Time Series and the task is binary prediction, the same as the example. And now my problem looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Training Data&#xA;train = genfromtxt(os.getcwd() + &quot;/Data/trainMatrix.csv&quot;, delimiter=',', skip_header=1)&#xA;validation = genfromtxt(os.getcwd() + &quot;/Data/validationMatrix.csv&quot;, delimiter=',', skip_header=1)&#xA;&#xA;#Targets&#xA;miniTrainTargets = [int(x) for x in genfromtxt(os.getcwd() + &quot;/Data/trainTarget.csv&quot;, delimiter=',', skip_header=1)]&#xA;validationTargets = [int(x) for x in genfromtxt(os.getcwd() + &quot;/Data/validationTarget.csv&quot;, delimiter=',', skip_header=1)]&#xA;&#xA;#LSTM&#xA;model = Sequential()&#xA;model.add(Embedding(train.shape[0], 64, input_length=train.shape[1]))&#xA;model.add(LSTM(64)) &#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(1))&#xA;model.add(Activation('sigmoid'))&#xA;&#xA;# try using different optimizers and different optimizer configs&#xA;model.compile(loss='binary_crossentropy',&#xA;          optimizer='adam',&#xA;          class_mode=&quot;binary&quot;)&#xA;&#xA;model.fit(train, miniTrainTargets, batch_size=batch_size, nb_epoch=5,&#xA;      validation_data=(validation, validationTargets), show_accuracy=True)&#xA;valid_preds = model.predict_proba(validation, verbose=0)&#xA;roc = metrics.roc_auc_score(validationTargets, valid_preds)&#xA;print(&quot;ROC:&quot;, roc)&#xA;ROC:0.5006526&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The model is basically the same as the IMDB one. Though the result means it's not learning anything. However, when I use a vanilla MLP-NN I don't have the same problem, the model learns and the score increases. I tried increasing the number of epochs and increasing-decreasing the number of LTSM units but the score won't increase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would like to know a standard approach to tuning the network because in theory the algorithm should perform better than a multilayer perceptron network specially for this time series data.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="31513" LastEditDate="2017-04-26T16:24:27.147" LastActivityDate="2020-08-03T14:25:33.807" Title="Hyperparameter search for LSTM-RNN using Keras (Python)" Tags="|python|neural-network|deep-learning|keras|hyperparameter|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9824" PostTypeId="2" ParentId="9818" CreationDate="2016-01-17T19:53:08.783" Score="22" Body="&lt;p&gt;Bayesian networks and neural networks are not exclusive of each other. In fact, Bayesian networks are just another term for &quot;directed graphical model&quot;. They can be very useful in designing objective functions neural networks. Yann Lecun has pointed this out here: &lt;a href=&quot;https://plus.google.com/+YannLeCunPhD/posts/gWE7Jca3Zoq&quot;&gt;https://plus.google.com/+YannLeCunPhD/posts/gWE7Jca3Zoq&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variational auto encoder and derivatives are directed graphical models of the form $$p(x) = \int_z p(x|z)p(z) dz.$$ A neural networks is used to implemented $p(x|z)$ and an approximation to its inverse: $q(z|x) \approx p(z|x)$.&lt;/p&gt;&#xA;" OwnerUserId="1193" LastEditorUserId="1193" LastEditDate="2016-01-18T15:56:26.757" LastActivityDate="2016-01-18T15:56:26.757" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9825" PostTypeId="2" ParentId="9818" CreationDate="2016-01-18T03:20:15.597" Score="17" Body="&lt;p&gt;Excellent answers already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One domain which I can think of, and is working extensively in, is the &lt;strong&gt;customer analytics&lt;/strong&gt; domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to understand and predict the moves and motives of the customers in order to inform and warn both the customer support, the marketing and also the growth teams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here, neural networks do a really good job in churn prediction, etc. But, I found and prefer the Bayesian networks style, and here are the reasons for preferring it:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Customers always have a pattern. They always have a &lt;em&gt;reason&lt;/em&gt; to act. And that reason would be something which my team has done for them, or they have learnt themselves. So, everything has a prior here, and in fact that reason is very important as it fuels most of the decision taken by the customer.&lt;/li&gt;&#xA;&lt;li&gt;Every move by the customer and the growth teams in the marketing/sales funnel is cause-effect. So, prior knowledge is vital when it comes to converting a prospective lead into a customer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, the concept of &lt;em&gt;prior&lt;/em&gt; is very important when it comes to customer analytics, which makes the concept of Bayesian networks very important to this domain.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Suggested Learning: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/slides/bayesian.pdf&quot;&gt;Bayesian Methods for Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://fedcsis.org/proceedings/2012/pliks/190.pdf&quot;&gt;Bayesian networks in business analytics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-18T03:20:15.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9826" PostTypeId="1" AcceptedAnswerId="10786" CreationDate="2016-01-18T05:37:40.897" Score="7" ViewCount="5475" Body="&lt;p&gt;I have set of categories and I want to compare a document vector with word vector of categories to find best matching category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to compare a word vector with document vector? If yes, is there any literature which gives proof of concept for this?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="8501" LastEditDate="2018-12-15T05:12:19.713" LastActivityDate="2018-12-15T05:12:19.713" Title="Can we compare a word2vec vector with a doc2vec vector?" Tags="|deep-learning|information-retrieval|word-embeddings|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9827" PostTypeId="1" CreationDate="2016-01-18T08:18:54.693" Score="2" ViewCount="190" Body="&lt;p&gt;&lt;strong&gt;DISCLAIMER:&lt;/strong&gt; &lt;em&gt;I understand this could be considered a subjective type of question, but I'm hoping it is specific enough to not be flagged. Please give it a chance.&lt;/em&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing a GridSearch in scikit-learn on a corpus of 50 books (all kinda average commercially published texts...each ~200 pages of text).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I began playing with GridSearch, I got to thinking. Surely there must be some sort of 'best practices' for building a pipeline?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, I can include and remove parameters, but I'm really just sorta stabbing around in the dark for the best score.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly, it could make sense to create a smaller, exploratory pipeline using the larger, more effective parameters first. After running this basic pipeline, more parameters that have a smaller effect (in general) could be run next.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say I'm GridSearching a TfidfTransformer.  I am assuming that flipping the &lt;code&gt;use_idf&lt;/code&gt; switch on or off is going to make a real impact on the scoring, whereas perhaps specifying &lt;code&gt;alpha=0.000010&lt;/code&gt; versus &lt;code&gt;alpha=0.000015&lt;/code&gt; won't.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, when using a pipeline, is there an alternative to &quot;throw in the kitchen sink&quot; that results in a more cognizant, informative (and possibly faster) experience?&lt;/p&gt;&#xA;" OwnerUserId="9670" LastActivityDate="2016-02-20T12:18:12.850" Title="Is there an appropriate sequence of parameters to be considered when building a pipeline in Scikit learn?" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9828" PostTypeId="2" ParentId="8195" CreationDate="2016-01-18T08:45:04.647" Score="0" Body="&lt;p&gt;If you are working with large dataset. Training and test set disribution may not be too different. In theory &quot;law of large numbers&quot; ensures that the distribution remains same. For smaller set of data probably this is a good point to take care of distribution. As said by Hoap Humanoid &quot;Kullbak-Liebler&quot; can be used to find the difference of distributions of two sets.&lt;/p&gt;&#xA;" OwnerUserId="837" LastActivityDate="2016-01-18T08:45:04.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9829" PostTypeId="5" CreationDate="2016-01-18T12:18:28.227" Score="0" Body="&lt;p&gt;Examples for Probabilistic Graphical Models (PGMs) are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Markov Random Fields (MRFs, undirected)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Conditional Random Fields (CRFs)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Bayesian networks (BNs, directed)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-18T14:36:29.827" LastActivityDate="2016-01-18T14:36:29.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9830" PostTypeId="4" CreationDate="2016-01-18T12:18:28.227" Score="0" Body="A probabilistic graphical model (PGM) is a probabilisic model for which conditional dependencies are expressed with a graph G = (X, E) where X are random variables." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-18T14:37:09.920" LastActivityDate="2016-01-18T14:37:09.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9831" PostTypeId="2" ParentId="9810" CreationDate="2016-01-18T12:53:01.577" Score="1" Body="&lt;p&gt;Clustering allows you to find clusters of similar hospitals. In your case, hospitals in the same cluster will have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;around the same number of patients&lt;/li&gt;&#xA;&lt;li&gt;around the same time&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But depending on the number of clusters you specify and your data, you might not get results that feel satisfactory. Moreover, clustering gives you &lt;em&gt;an&lt;/em&gt; answer, but it might not be an answer to the question you want answered.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exemple 1:&lt;/strong&gt; Hospital 1 at week 1 might be grouped with Hospital 2 at week 1, while having respectively 0 and 5 patients. This might make sense in the model, but might not in &lt;em&gt;real life&lt;/em&gt;, where 0 or 5 patients could be very different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To adress this problem, you might want to &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_scaling#Standardization&quot; rel=&quot;nofollow&quot;&gt;standardize&lt;/a&gt; your patient count. It will help the model to make sense of &lt;em&gt;how different&lt;/em&gt; 0 and 5 patients are.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exemple 2:&lt;/strong&gt; Hospital 1 at week 1 might be grouped with hospital 3 at week 12, if they are the only one having 23 patients while every Hospital never gets more than 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the other problem you are facing with clustering; it helps you find similarities, but similarities might not be helpful. K-means can not fully use the fact that the data at week 2 depends from the data at week 1. It only tries to find how &lt;em&gt;close&lt;/em&gt; two points are in the 2-dimensional space formed by the week number and patient count that week.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So you want&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To detect outbreaks&lt;/li&gt;&#xA;&lt;li&gt;To find the hospital in which the outbreak is happening&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Anony-Mousse proposed a very simple way of finding &quot;outbreaks&quot;; Compute the difference in patients from week to week. Here would be a way to elaborate on this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Set week 1 at 0 and week $i$ at (number of patients at week $i$ $-$ the number of patients at week $i-1$)&lt;/li&gt;&#xA;&lt;li&gt;Compute the sum of the change over all hospitals with past week, and use a threshold to determine outbreaks. If there are 10 new patients or more that week, there is an &quot;outbreak&quot;&lt;/li&gt;&#xA;&lt;li&gt;Use the initial number of patient per hospital at the week the outbreak is detected to determine which hospital are touched. Use 2-means for (High/Low), 3-means for (High/Medium/Low), and so on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is just a simple example, but should give you an hint on to where to search next.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Additional &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is an &lt;em&gt;outbreak&lt;/em&gt;? If you can define more clearly what you are searching for, you will have better results in the end. Clustering tries to find &lt;em&gt;structure&lt;/em&gt; in the data, and outbreaks &lt;em&gt;might&lt;/em&gt; be part of that structure, but you should focus on methods specifically for it.&lt;/li&gt;&#xA;&lt;li&gt;If you want to apply data science/machine learning techniques to a problem, try to formulate the problem in that language. It is a framework to reason about data, but a good portion of the job is to translate your problem in ways it can be handled.&lt;/li&gt;&#xA;&lt;li&gt;Learn about methods specifically designed to handle this type of problems. A good place to start would be &lt;a href=&quot;https://en.wikipedia.org/wiki/Change_detection&quot; rel=&quot;nofollow&quot;&gt;Change detection&lt;/a&gt;, as pointed out by Anony-Mousse.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15501" LastActivityDate="2016-01-18T12:53:01.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9832" PostTypeId="1" AcceptedAnswerId="10358" CreationDate="2016-01-18T13:51:25.520" Score="68" ViewCount="81283" Body="&lt;p&gt;It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got something wrong.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Definitions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Q- and V-learning are in the context of &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_decision_process#Definition&quot; rel=&quot;noreferrer&quot;&gt;Markov Decision Processes&lt;/a&gt;. A &lt;strong&gt;MDP&lt;/strong&gt; is a 5-tuple $(S, A, P, R, \gamma)$ with&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$S$ is a set of states (typically finite)&lt;/li&gt;&#xA;&lt;li&gt;$A$ is a set of actions (typically finite)&lt;/li&gt;&#xA;&lt;li&gt;$P(s, s', a) = P(s_{t+1} = s' | s_t = s, a_t = a)$ is the probability to get from state $s$ to state $s'$ with action $a$.&lt;/li&gt;&#xA;&lt;li&gt;$R(s, s', a) \in \mathbb{R}$ is the immediate reward after going from state $s$ to state $s'$ with action $a$. (It seems to me that usually only $s'$ matters).&lt;/li&gt;&#xA;&lt;li&gt;$\gamma \in [0, 1]$ is called discount factor and determines if one focuses on immediate rewards ($\gamma = 0$), the total reward ($\gamma = 1$) or some trade-off.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;policy $\pi$&lt;/strong&gt;, according to &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;noreferrer&quot;&gt;Reinforcement Learning: An Introduction&lt;/a&gt; by Sutton and Barto is a function $\pi: S \rightarrow A$ (this could be probabilistic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://www.cs.upc.edu/~mmartin/Ag4-4x.pdf&quot; rel=&quot;noreferrer&quot;&gt;Mario Martins slides&lt;/a&gt;, the &lt;strong&gt;$V$ function&lt;/strong&gt; is&#xA;$$V^\pi(s) = E_\pi \{R_t | s_t = s\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s\}$$&#xA;and the &lt;strong&gt;Q function&lt;/strong&gt; is&#xA;$$Q^\pi(s, a) = E_\pi \{R_t | s_t = s, a_t = a\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s, a_t=a\}$$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The $V$ function states what the expected overall value (not reward!) of a state $s$ under the policy $\pi$ is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $Q$ function states what the value of a state $s$ and an action $a$  under the policy $\pi$ is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means,&#xA;$$Q^\pi(s, \pi(s)) = V^\pi(s)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right? So why do we have the value function at all? (I guess I mixed up something)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="836" LastEditDate="2017-12-07T17:01:09.630" LastActivityDate="2023-07-24T04:22:25.943" Title="What is the Q function and what is the V function in reinforcement learning?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9834" PostTypeId="2" ParentId="9818" CreationDate="2016-01-18T14:15:40.530" Score="21" Body="&lt;p&gt;&lt;strong&gt;Sometimes you care as much about changing the outcome as predicting the outcome.&lt;/strong&gt;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A neural network given enough training data will tend to predict the outcome better, but once you can predict the outcome, you then may wish to predict the effect of making changes in the input features on the outcome.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example from real life, knowing that someone is likely to have a heart attack is useful, but being able to tell the person that if they stopped doing XX, the risk would reduce by 30% is of much greater benefit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise for customer retention, knowing why customers stop shopping with you, is worth as much as predicting the customers that are likely to stop shopping with you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also a simpler Bayesian Network that predicts less well but &lt;strong&gt;leads to more action being taken&lt;/strong&gt; may often be better than a more “correct” Bayesian Network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The biggest advantage of Bayesian networks over neural networks is that they can be used for causal inference . This branch is of fundamental importance to statistics and machine learning and Judea Pearl has won the &lt;a href=&quot;http://amturing.acm.org/award_winners/pearl_2658896.cfm&quot; rel=&quot;noreferrer&quot;&gt;Turing award&lt;/a&gt; for this research.&lt;/p&gt;&#xA;" OwnerUserId="1272" LastEditorUserId="21" LastEditDate="2016-05-30T23:50:10.567" LastActivityDate="2016-05-30T23:50:10.567" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9838" PostTypeId="2" ParentId="9818" CreationDate="2016-01-18T17:09:03.993" Score="7" Body="&lt;p&gt;Bayesian networks might outperform Neural Networks in small data setting. If the prior information is properly managed via the network structure, priors and other hyperparameters, it might have an edge over Neural Networks. Neural Networks, especially the ones with more layers, are very well known to be data hungry. Almost by definition lots of data is necessary to properly train them.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastActivityDate="2016-01-18T17:09:03.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9839" PostTypeId="1" CreationDate="2016-01-18T19:48:29.680" Score="5" ViewCount="279" Body="&lt;p&gt;I am exploring using Bayesian Networks to identify the best parameters within a system design, to improve its performance. I'm trying to find any case studies where this has been used successfully or unsuccessfully.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example consider an automatic assisted anti-collision system on a car.  It could use some combination of ultrasonic, radar &amp;amp; IR seekers to sense and measure the closing speed with the car in front.  These all have different costs / reliabilities and perform differently in different weather conditions.  I am interested in understanding how effective Bayesian Networks can be in evaluating the trade space and identifying the best solutions to these types of problems.&lt;/p&gt;&#xA;" OwnerUserId="15519" LastEditorUserId="14994" LastEditDate="2016-01-21T06:46:43.030" LastActivityDate="2017-11-24T00:40:52.003" Title="Any case studies using Bayesian Networks for system design trades?" Tags="|data-mining|predictive-modeling|bayesian-networks|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9840" PostTypeId="2" ParentId="9810" CreationDate="2016-01-18T20:19:09.400" Score="0" Body="&lt;p&gt;If you are trying to find both the clusters of weeks and the clusters of hospitals where outbreaks happen then you might be more successful using &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) a simple moving average filter on the weeks,  playing with the number of weeks you are averaging - trying 3-5 weeks as the averaging window.&lt;br&gt;&#xA;2) now look for hospitals that have high moving averages in the same weeks&lt;/p&gt;&#xA;" OwnerUserId="15519" LastActivityDate="2016-01-18T20:19:09.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9841" PostTypeId="2" ParentId="9823" CreationDate="2016-01-18T23:07:52.433" Score="5" Body="&lt;p&gt;An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, &lt;code&gt;[[4], [20]] -&amp;gt; [[0.25, 0.1], [0.6, -0.2]]&lt;/code&gt;. This representation conversion is learned automatically with the embedding layer in Keras (see the &lt;a href=&quot;http://keras.io/layers/embeddings/&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it seems that your data does not need any such embedding layer to perform a conversion. Having an unnecessary embedding layer is likely why you cannot get your LSTM to work properly. If that is the case then you should simply remove the embedding layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first layer in your network should then have the &lt;code&gt;input_shape&lt;/code&gt; argument added with information on the dimensions of your data (see &lt;a href=&quot;http://keras.io/examples/&quot; rel=&quot;noreferrer&quot;&gt;examples&lt;/a&gt;). Note that you can add this argument to any layer - it will not be present in the documentation for any specific layer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;By the way, hyperparameters are often tuned using random search or Bayesian optimization. I would use RMSProp and focus on tuning batch size (sizes like 32, 64, 128, 256 and 512), gradient clipping (on the interval 0.1-10) and dropout (on the interval of 0.1-0.6). The specifics of course depend on your data and model architecture.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="3044" LastEditDate="2016-01-18T23:12:55.587" LastActivityDate="2016-01-18T23:12:55.587" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9842" PostTypeId="2" ParentId="6365" CreationDate="2016-01-19T00:01:42.273" Score="1" Body="&lt;p&gt;If Python 3 is an option, &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow&quot;&gt;Orange 3&lt;/a&gt; features FP-growth in its &lt;a href=&quot;https://pypi.python.org/pypi/Orange3-Associate&quot; rel=&quot;nofollow&quot;&gt;Orange3-Associate add-on&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip3 install Orange3-Associate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's well documented: &lt;a href=&quot;http://orange3-associate.readthedocs.org/&quot; rel=&quot;nofollow&quot;&gt;http://orange3-associate.readthedocs.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apriori due to its slowness and resource requirements, as you'd noticed, really isn't an algorithm to consider anymore.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="15527" LastEditDate="2016-02-29T14:53:32.657" LastActivityDate="2016-02-29T14:53:32.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9843" PostTypeId="2" ParentId="9554" CreationDate="2016-01-19T00:19:09.127" Score="4" Body="&lt;p&gt;If one likes cross-platform visual programming tools, &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt; is an option. Having recently moved to Python 3, they haven't yet got all the widgets ported.&#xA;It's bringing the PyData stack (NumPy, SciPy, SciKit Learn, ...) to Python 3, PyQt, PyQtGraph, and it's GPL'd &lt;a href=&quot;https://github.com/biolab/orange3&quot; rel=&quot;nofollow noreferrer&quot;&gt;on GitHub&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YnQh6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YnQh6.png&quot; alt=&quot;Orange screenshow&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="15527" LastEditDate="2016-01-19T03:01:46.333" LastActivityDate="2016-01-19T03:01:46.333" CommentCount="0" CommunityOwnedDate="2016-06-08T13:17:42.357" ContentLicense="CC BY-SA 3.0" />
  <row Id="9844" PostTypeId="2" ParentId="5458" CreationDate="2016-01-19T03:05:25.887" Score="1" Body="&lt;p&gt;I don't know about RapidMiner, but for a beginner, &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt; seems an awesome open-source tool for the job. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly if you want to do unsupervised learning, i.e. clustering. You'd probably want to preprocess your textual data in some &lt;em&gt;bag of words&lt;/em&gt; model and save it as CSV, I guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also hear Orange has a &lt;a href=&quot;https://pypi.python.org/pypi/Orange3-Text/&quot; rel=&quot;nofollow noreferrer&quot;&gt;text mining add-on&lt;/a&gt;, but I haven't tried it yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YnQh6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YnQh6.png&quot; alt=&quot;Orange screenshow&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-01-19T03:05:25.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9845" PostTypeId="2" ParentId="9818" CreationDate="2016-01-19T06:11:24.240" Score="6" Body="&lt;p&gt;I've posted &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/41dij6/is_there_any_domain_where_bayesian_networks/&quot; rel=&quot;noreferrer&quot;&gt;this link on Reddit&lt;/a&gt; and got a lot of feedback. Some have posted their answers here, others didn't. This answer should sum the reddit post up. (I made it community wiki, so that I don't get points for it)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/abs/1312.6114v10&quot; rel=&quot;noreferrer&quot;&gt;Auto-Encoding Variational Bayes&lt;/a&gt; is a combination of a Bayes Network and a neural network. The paper &lt;a href=&quot;http://arxiv.org/abs/1506.02158v5&quot; rel=&quot;noreferrer&quot;&gt;Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference&lt;/a&gt; seems to go in the same direction.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot; rel=&quot;noreferrer&quot;&gt;Dropout: A Simple Way to Prevent Neural Networks from&#xA;Overfitting&lt;/a&gt; is an example where Bayesian neural networks outperform their dropout approach (see section 6.4 &quot;Comparison with Bayesian Neural Networks&quot;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://web.mit.edu/cocosci/Papers/Science-2015-Lake-1332-8.pdf&quot; rel=&quot;noreferrer&quot;&gt;Human-level concept learning through probabilistic program induction&lt;/a&gt; is a paper &quot;on a Bayesian net network that does one-shot classification that way outperformed neural networks&quot; (according to trashacount12345 - I didn't check that by now).&lt;/li&gt;&#xA;&lt;li&gt;Yann LeCun wrote a &lt;a href=&quot;https://plus.google.com/+YannLeCunPhD/posts/gWE7Jca3Zoq&quot; rel=&quot;noreferrer&quot;&gt;Google+ post&lt;/a&gt; in which he argues that neural networks and probabilisitc graphical models are not orthogonal concepts.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-19T06:18:44.937" LastActivityDate="2016-01-19T06:18:44.937" CommentCount="0" CommunityOwnedDate="2016-01-19T06:11:24.240" ContentLicense="CC BY-SA 3.0" />
  <row Id="9846" PostTypeId="1" CreationDate="2016-01-19T06:26:00.080" Score="4" ViewCount="170" Body="&lt;p&gt;For the sake of my own exploration, I am working on a sales prediction project. I am using text extracted from a set of books to build a predictive model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With scikit learn, I have created a Tfidf, and together with the numeric sales numbers, I created a SGDRegressor. But I'd like to practice with other models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking over the options available to me, and I'm wondering, what other algorithms might be useful in this kind of regression scenario? And/or, what other algorithms in scikit learn will take a tfidf as the dataset?   &lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="11097" LastEditDate="2016-01-19T06:33:41.093" LastActivityDate="2016-01-21T13:30:26.147" Title="What regressors are recommended with text modeling?" Tags="|nlp|text-mining|regression|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9848" PostTypeId="1" CreationDate="2016-01-19T09:03:57.597" Score="9" ViewCount="14662" Body="&lt;p&gt;Suppose I have mixed data and (python) code which is capable of doing &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;noreferrer&quot;&gt;PCA&lt;/a&gt; (principal component analysis) on continuous predictors and &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiple_correspondence_analysis&quot; rel=&quot;noreferrer&quot;&gt;MCA&lt;/a&gt; (multiple correspondence analysis) on nominal predictors. Is it possible to combine results from PCA and MCA into one?&lt;/p&gt;&#xA;" OwnerUserId="15148" LastEditorUserId="12527" LastEditDate="2016-01-19T12:10:39.727" LastActivityDate="2018-08-20T15:47:10.087" Title="How to combine PCA and MCA on mixed data?" Tags="|python|categorical-data|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9849" PostTypeId="1" CreationDate="2016-01-19T09:11:00.450" Score="1" ViewCount="712" Body="&lt;p&gt;I would like to know how should I provide the inputs to a BiLSTM if I am going to classify speech files (.wav) files. What is the proper way to label the data? Do I have to label each individual frame with the target label (which I think is meaningless because content of a single frame doesn't imply that the end label of the given file is what it is. Eg: First frame of utterance_0.wav doesn't directly say output label is 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore what would be the correct way of labelling these data? And how should I maintain the .mfcc values calculated per each .wav files. (I have around 4500 files and trying to generate coefficients when we try to do the training would add extra overhead) Please correct me if I am wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I am new to this ML approaches and training networks and RNN, I would like to have some guidance on above questions.&lt;/p&gt;&#xA;" OwnerUserId="15536" LastActivityDate="2016-01-19T09:11:00.450" Title="How to implement a BiLSTM to classify speech data" Tags="|machine-learning|classification|deep-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9850" PostTypeId="1" AcceptedAnswerId="9870" CreationDate="2016-01-19T11:48:29.337" Score="60" ViewCount="53506" Body="&lt;p&gt;I am using &lt;a href=&quot;https://en.wikipedia.org/wiki/TensorFlow&quot;&gt;TensorFlow&lt;/a&gt; for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the &quot;correct&quot; cost function for specific problems because overall I could be considered a beginner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before coming to TensorFlow I coded some fully-connected MLPs and some recurrent networks on my own with &lt;a href=&quot;https://en.wikipedia.org/wiki/Python_(programming_language)&quot;&gt;Python&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/NumPy&quot;&gt;NumPy&lt;/a&gt; but mostly I had problems where a simple squared error and a simple gradient descient was sufficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, since TensorFlow offers quite a lot of cost functions itself as well as building custom cost functions, I would like to know if there is some kind of tutorial maybe specifically for cost functions on neural networks? (I've already done like half of the official TensorFlow tutorials but they're not really explaining &lt;strong&gt;why&lt;/strong&gt; specific cost functions or learners are used for specific problems - at least not for beginners)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To give some examples:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_output, y_train))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess it applies the softmax function on both inputs so that the sum of one vector equals 1. But what exactly is cross entropy with logits? I thought it sums up the values and calculates the cross entropy...so some metric measurement?! Wouldn't this be very much the same if I normalize the output, sum it up and take the squared error?&#xA;Additionally, why is this used e.g. for MNIST (or even much harder problems)? When I want to classify like 10 or maybe even 1000 classes, doesn't summing up the values completely destroy any information about &lt;em&gt;which&lt;/em&gt; class actually was the output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;cost = tf.nn.l2_loss(vector)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is this for? I thought l2 loss is pretty much the squared error but TensorFlow's API tells that it's input is just one tensor. Doesn't get the idea at all?!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides I saw this for &lt;strong&gt;cross entropy&lt;/strong&gt; pretty often:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = -tf.reduce_sum(y_train * tf.log(y_output))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...but why is this used? Isn't the loss in cross entropy mathematically this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-1/n * sum(y_train * log(y_output) + (1 - y_train) * log(1 - y_output))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where is the &lt;code&gt;(1 - y_train) * log(1 - y_output)&lt;/code&gt; part in most TensorFlow examples? Isn't it missing?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answers:&lt;/strong&gt; I know this question is quite open, but I do not expect to get like 10 pages with every single problem/cost function listed in detail. I just need a short summary about when to use which cost function (in general or in TensorFlow, doesn't matter much to me) and some explanation about this topic. And/or some source(s) for beginners ;)&lt;/p&gt;&#xA;" OwnerUserId="13809" LastActivityDate="2019-11-13T04:45:11.677" Title="Neural networks: which cost function to use?" Tags="|machine-learning|python|neural-network|statistics|tensorflow|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9851" PostTypeId="2" ParentId="9848" CreationDate="2016-01-19T13:55:58.777" Score="8" Body="&lt;p&gt;You may want to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Factor_analysis_of_mixed_data&quot; rel=&quot;noreferrer&quot;&gt;Factor analysis of mixed data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It allows you to do dimension reduction on a complete data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A R implementation could be found in the FactoMineR package. But this function struggle when you have a high number of data/columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not aware of the existence of the equivalent in python.&lt;/p&gt;&#xA;" OwnerUserId="15543" LastActivityDate="2016-01-19T13:55:58.777" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9852" PostTypeId="1" CreationDate="2016-01-19T14:00:09.217" Score="5" ViewCount="345" Body="&lt;p&gt;After thinking about a question &lt;a href=&quot;https://datascience.stackexchange.com/questions/9818/is-there-any-domain-where-bayesian-networks-outperform-neural-networks&quot;&gt;comparing Bayesian Network to Neural Network&lt;/a&gt;, I am now wondering if they may not be one and the same thing!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, my maths (unused for over 20 years fails me!).   But....&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If infinite random test data was used&lt;/li&gt;&#xA;&lt;li&gt;The output of the Bayesian Network was found for each data item&lt;/li&gt;&#xA;&lt;li&gt;And the Neural Network was then trained on it&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Would the Neural Network not tent towards being the same as the Bayesian Network?&lt;/p&gt;&#xA;" OwnerUserId="1272" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2018-07-19T10:12:39.317" Title="For every Bayesian Network, is there a Neural Network that gives the same output?" Tags="|machine-learning|bayesian-networks|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9853" PostTypeId="1" CreationDate="2016-01-19T14:21:40.870" Score="2" ViewCount="34" Body="&lt;p&gt;In &lt;a href=&quot;http://ucrel.lancs.ac.uk/acl/H/H93/H93-1039.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, Brown et al. proposed the usage of dictionary entries in addition to the machine translation probabilities that we're used to in the &lt;a href=&quot;http://www.aclweb.org/anthology/J93-2003&quot; rel=&quot;nofollow noreferrer&quot;&gt;Math of Statistical Machine Translation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He describes:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We formalize an intuitive account of lexicographic behavior as&#xA;  follows. We imagine that a lexicographer, when constructing an entry&#xA;  for the English word or phrase e, first chooses a random size s, and&#xA;  then selects at random a sample of s instances of the use of e, each&#xA;  with its French translation. We imagine, further, that he includes in&#xA;  his entry for e a list consisting of all of tile translations that&#xA;  occur at least once in his random sample. The probability that he&#xA;  will, in this way, obtain tile list fi, ..., f,,, is&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/drgci.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/drgci.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That much is understandable, and the &lt;code&gt;Pr(f|e)&lt;/code&gt; in the formula is what we get from the normal machine translation probabilities. and then reweighted by the binomial recursion of finding the probability of the lexicographer choosing the sample sentence given the english word/phrase, i.e. &lt;code&gt;Pr(s|e)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then he went on to simply the formula and estimated a Poisson distribution instead of a binomial one and that will simplify the computation to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JYJ0U.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JYJ0U.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything still looks okay but there's an additional variable &lt;code&gt;exp^lambda(e)&lt;/code&gt; and that he explains is the mean of the Possion distribution of selecting the sample examples given the English source word. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So the question is how did he estimate that &lt;code&gt;exp^lambda(e)&lt;/code&gt; from the corpus?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2016-01-19T14:21:40.870" Title="How to estimate the lambda e to reweighing translation probabilities of entries in a dictionary" Tags="|algorithms|machine-translation|parameter-estimation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9854" PostTypeId="1" AcceptedAnswerId="9861" CreationDate="2016-01-19T15:14:38.403" Score="0" ViewCount="1936" Body="&lt;p&gt;In a binary classification, how can I use sklearn.naive_bayes python module to predict the class of inputs with 5 categorical variables (not binary)?&lt;/p&gt;&#xA;" OwnerUserId="3433" LastActivityDate="2016-01-19T21:48:19.997" Title="sklearn.naive_bayes VS categorical variables" Tags="|machine-learning|data-mining|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9855" PostTypeId="2" ParentId="9852" CreationDate="2016-01-19T17:17:04.363" Score="4" Body="&lt;p&gt;We don't have to do anything crazy to prove this, we can simply leverage the UAT. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;nofollow&quot;&gt;universal approximation theorem&lt;/a&gt;, yes, you could concieve of a neural network that precisely matched your Bayesian network.&lt;/p&gt;&#xA;" OwnerUserId="15090" LastEditorUserId="8878" LastEditDate="2016-01-19T20:11:45.987" LastActivityDate="2016-01-19T20:11:45.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9856" PostTypeId="1" CreationDate="2016-01-19T17:26:39.433" Score="2" ViewCount="236" Body="&lt;p&gt;I have the following code, where x in the input data, w is the weight matrix, bv and bh are the biases for the visible and hidden units.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import theano.tensor as T&#xA;&#xA;x_states = x &amp;gt; numpy.random.rand(training_examples, feats)&#xA;&#xA;hid =  T.nnet.sigmoid(T.dot(x_states, w) + bh)    # Activation function of hidden layer&#xA;hid_states = hid &amp;gt; numpy.random.rand(training_examples, nhidden)&#xA;&#xA;# Construct Theano expression graph&#xA;vis = T.nnet.sigmoid(T.dot(hid_states, w.T) + by) &#xA;vis_states = vis &amp;gt; numpy.random.rand(training_examples, feats)&#xA;&#xA;hid2 = T.nnet.sigmoid(T.dot(vis_states, w) + bh)&#xA;hid2_states = hid2 &amp;gt; numpy.random.rand(training_examples, nhidden)&#xA;&#xA;xent = T.sum((x - vis)**2)&#xA;parameters = [w, bh, by]  # this line defines all parameters for each layer&#xA;cost = xent.mean()&#xA;&#xA;pos_associations = T.dot(x.T, hid)&#xA;neg_associations = T.dot(vis.T, hid2)  &#xA;&#xA;wparameters = [w]&#xA;byparameters = [by]&#xA;bhparameters = [bh]&#xA;&#xA;update = []&#xA;for wparam, byparam, bhparam in zip(wparameters, byparameters, bhparameters):&#xA;    update.append((wparam, wparam + tr_rate * (pos_associations - neg_associations)))&#xA;    update.append((byparam, byparam + tr_rate*(T.sum(x.T[:]) - T.sum(vis.T[:]))))      &#xA;    update.append((bhparam, bhparam + tr_rate*(T.sum(hid.T[:]) - T.sum(hid2.T[:]))))&#xA;&#xA;&#xA;train = theano.function(&#xA;          inputs=[x],&#xA;          outputs=[cost],&#xA;          updates=update)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the correct implementation for the update? In my example this is clearly not working. I tested this on the classic MNIST dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the result I get for the cost. As you can see it does not decreases. What is wrong?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch: 0&#xA;cost =  [ 969572.73014003]&#xA;Epoch: 1&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 2&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 3&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 4&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 5&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 6&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 7&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 8&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 9&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 10&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 11&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 12&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 13&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 14&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 15&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 16&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 17&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 18&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 19&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 20&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 21&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 22&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 23&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 24&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 25&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 26&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 27&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 28&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 29&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 30&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 31&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 32&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 33&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 34&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 35&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 36&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 37&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 38&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 39&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 40&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 41&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 42&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 43&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 44&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 45&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 46&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 47&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 48&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 49&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 50&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 51&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 52&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 53&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 54&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 55&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 56&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 57&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 58&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 59&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 60&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 61&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 62&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 63&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 64&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 65&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 66&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 67&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 68&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 69&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 70&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 71&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 72&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 73&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 74&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 75&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 76&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 77&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 78&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 79&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 80&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 81&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 82&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 83&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 84&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 85&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 86&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 87&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 88&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 89&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 90&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 91&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 92&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 93&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 94&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 95&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 96&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 97&#xA;cost =  [ 258872.77507019]&#xA;Epoch: 98&#xA;cost =  [ 2003326.79850769]&#xA;Epoch: 99&#xA;cost =  [ 258872.77507019]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="8820" LastEditDate="2016-02-08T05:01:42.710" LastActivityDate="2016-02-08T05:01:42.710" Title="Contrastive divergence in RBM" Tags="|neural-network|deep-learning|theano|rbm|" AnswerCount="0" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="9857" PostTypeId="1" CreationDate="2016-01-19T19:02:45.530" Score="1" ViewCount="315" Body="&lt;p&gt;I'm trying to build a model using GBM in r in order to get probability of two classes ( 'yes','no'). My data are unbalanced, and because of this I trained my model using a balanced data(undersampling the 'no' class). I trained my model With the balanced sample and then I evaluate the performance on my test sample ( 30% of the balanaced sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is any problem to predict the probability with my fit in the unbalanced sample? When I performe this, the probability distribution are equal of the probability distribution of my balanced sample, and because of this maybe my score is meaningless&lt;/p&gt;&#xA;" OwnerUserId="14644" LastEditorUserId="14644" LastEditDate="2016-01-19T19:11:10.133" LastActivityDate="2016-01-19T19:11:10.133" Title="Unbalanced data fit in gbm" Tags="|machine-learning|r|gbm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9858" PostTypeId="1" CreationDate="2016-01-19T20:59:28.040" Score="10" ViewCount="3864" Body="&lt;p&gt;I have been trying to understand the different k-means clustering algorithms mainly that are implemented in the &lt;code&gt;stats&lt;/code&gt; package of the &lt;code&gt;R&lt;/code&gt; language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the Lloyd's algorithm and MacQueen's online algorithm. The way I understand them is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Lloyd's Algorithm:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially ‘k’ random observations are chosen that will serve as the centroids of the ‘k’ clusters. Then the following steps occur in iteration till the centroids converge.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The Euclidean distance between each observation and the chosen centroids is calculated. &lt;/li&gt;&#xA;&lt;li&gt;The observations that are closest to each centroids are tagged within ‘k’ buckets. &lt;/li&gt;&#xA;&lt;li&gt;The mean of all the observations in each bucket serves as new centroids. &lt;/li&gt;&#xA;&lt;li&gt;The new centroids replace the old centroids and the iteration goes back to step 1 if the old and new centroids have not converged.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The conditions to converge are the following: the old and the new centroids are exactly identical, the difference between the centroids is small (of the order of 10^-3) or the maximum number of iterations (10 or 100) are reached.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MacQueen's Algorithm:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an online version where the first 'k' instances are chosen as centroids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then each instance is placed in buckets depending on which centroid is closest to that instance. The respective centroid is recalculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat this step till each instance is placed in the appropriate bucket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This algorithm only has one iteration and the loop goes on for 'x' instances&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hartigan-Wong Algorithm:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign all the points/instances to random buckets and calculate the respective centroid.&lt;/li&gt;&#xA;&lt;li&gt;Starting from the first instance find the nearest centroid and assing that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change&lt;/li&gt;&#xA;&lt;li&gt;Loop through all the points and get new centroids.&lt;/li&gt;&#xA;&lt;li&gt;Do a second iteration of points 2 and 3 which performs sort of a clean-up operation and reassigns stray points to correct buckets.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So this algorithm performs 2 iterations before we see the convergence result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I am unsure if what I think in point 4 in the Hartigan-Wong algorithm is the correct method of the algorithm. My question is, if the following method for Hartigan-Wong is the correct method to implement k-means? Are there only two iterations for this method? if not, what is the condition for convergence (when to stop)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible implementation explanation what I understand is.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign all the points/instances to random buckets and calculate the respective centroid.&lt;/li&gt;&#xA;&lt;li&gt;Starting from the first instance find the nearest centroid and assign that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change.&lt;/li&gt;&#xA;&lt;li&gt;Once there is a change in the bucket for any point, go back to the first instance and repeat the steps again.&lt;/li&gt;&#xA;&lt;li&gt;The iteration ends when all the instances are iterated and none of the points change buckets.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This way there are a lot of iterations that start from the beginning of the dataset again and again every time when an instance changes buckets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any explanations would be helpful and please let me know if I my understanding for any of these methods is wrong.&lt;/p&gt;&#xA;" OwnerUserId="15552" LastEditorUserId="15552" LastEditDate="2016-01-19T22:55:43.443" LastActivityDate="2018-03-21T21:25:15.160" Title="Convergence in Hartigan-Wong k-means method and other algorithms" Tags="|r|clustering|k-means|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9860" PostTypeId="1" AcceptedAnswerId="9878" CreationDate="2016-01-19T21:26:28.150" Score="0" ViewCount="8136" Body="&lt;p&gt;I am trying to read in a .csv file containing some data. I only need to read in specific chunks of rows from the file, such as line 15- line 20, line 45-line 50, and so on. However, the file contains text and copy write information like, such as &lt;code&gt;©1990-2016 AAR,All rights reserved&lt;/code&gt; in several places. Such lines seem to be producing the error &lt;code&gt;ValueError: No columns to parse from file&lt;/code&gt;, because when I just copy lines without such information using &lt;code&gt;pd.read_csv()&lt;/code&gt;, it works fine. My goal is to automate the process of downloading these files from the web and reading them into pandas to grab chunks of rows and then do some processing with it, so I can't just manually specify the windows of text lacking such characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I tried:&lt;code&gt;pd.read_csv(&quot;filename.csv&quot;,encoding=utf-8, skiprows = 14)&lt;/code&gt; and &lt;code&gt;pd.read_csv(&quot;filename.csv&quot;,encoding=utf-16, skiprows = 15)&lt;/code&gt;, after looking at similar answers in stack exchange, but this didn't work. Can anyone give me some guidance on this?&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="13413" LastEditDate="2016-01-21T03:05:23.423" LastActivityDate="2016-01-21T07:42:08.003" Title="How can I read in a .csv file with special characters in it in pandas?" Tags="|python|pandas|scraping|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9861" PostTypeId="2" ParentId="9854" CreationDate="2016-01-19T21:48:19.997" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow&quot;&gt;Hot encode&lt;/a&gt; the categorical variables and use Bernoulli naive Bayes. Hot encoding is usually the trick one uses in representing categorical variables.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-19T21:48:19.997" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9862" PostTypeId="1" AcceptedAnswerId="9868" CreationDate="2016-01-19T22:19:43.050" Score="3" ViewCount="3525" Body="&lt;p&gt;I was trying to build a 0-1 classifier using xgboost R package. My question is how predictions are made? For example in random forests, trees &quot;vote&quot; against each option and the final prediction is based on majority. As regard xgboost, the regression case is simple since prediction on whole model is equal to sum of predcitions for weak learners (boosted trees), but what about classification? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does xgboost classifier works the same as in the random forest (I don't think so, since it can return predictive probabilities, not class membership).&lt;/p&gt;&#xA;" OwnerUserId="13384" LastEditorUserId="11097" LastEditDate="2016-01-20T12:30:32.187" LastActivityDate="2016-01-20T12:46:55.283" Title="Classification using xgboost - predictions" Tags="|classification|predictive-modeling|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9863" PostTypeId="1" CreationDate="2016-01-20T01:18:02.193" Score="3" ViewCount="1368" Body="&lt;p&gt;I'm starting to experiment with neural networks in Google's tensorflow framework, and in particular am working on a model that has an LSTM layer. I am using the &lt;a href=&quot;https://www.tensorflow.org/versions/master/tutorials/recurrent/index.html&quot; rel=&quot;nofollow&quot;&gt;RNN tutorial&lt;/a&gt; to get me started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train my model, I have sequences of data that are several thousand data points long. In the tutorial, the author is processing small sequences of data (short sentences of paragraphs) and concatenates those sequences together, separated by &lt;code&gt;&amp;lt;end of stream&amp;gt;&lt;/code&gt;, divides the entire data set equally into &lt;code&gt;batch_size&lt;/code&gt; number of chunks and processes those chunks in parallel. I would like to process each of my sequences of data separately from start to finish, but would like to be able to process several of them in parallel, as the &lt;code&gt;rnn_cell&lt;/code&gt; objects are set up to do (they process &lt;code&gt;batch_size&lt;/code&gt; sequences at onces).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be straight forward to do if the sequences were of the same length, but they are not. An &lt;code&gt;rnn_cell.LSTMCell&lt;/code&gt; expects the data to come in with a fixed dimensionaltiy, &lt;code&gt;batch_size x input_size&lt;/code&gt;, so there's a problem once some of the sequences reach their end and my &lt;code&gt;batch_size&lt;/code&gt; wants to decrease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I work around this? I don't think I can 'left' or 'right' justify my data so that they line up without affecting the training.&lt;/p&gt;&#xA;" OwnerUserId="15422" LastEditorUserId="11097" LastEditDate="2016-03-08T06:52:22.333" LastActivityDate="2017-11-08T03:08:57.087" Title="Tensorflow RNN: Batching data of different length" Tags="|tensorflow|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9864" PostTypeId="2" ParentId="9852" CreationDate="2016-01-20T06:13:41.527" Score="2" Body="&lt;p&gt;I would say &lt;strong&gt;Yes.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Jacob neatly put, the Universal Approximation algo. can be leveraged for proving the fact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, as neural networks are highly flexible, owing to their portable structure definitions and a huge set of activation functions one can choose from, I don't see why a Bayesian network cannot be mimicked by a neural network. But, from my experience of both research and coding deep belief networks, I'd advise you &lt;strong&gt;not to try this at home.&lt;/strong&gt;  Cause, you would often end up with painfully complex networks. :D&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, I am skeptical about the vice-versa process. (owing to my limited knowledge of Bayesian networks)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-20T06:13:41.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9865" PostTypeId="1" AcceptedAnswerId="9867" CreationDate="2016-01-20T06:33:54.923" Score="13" ViewCount="29588" Body="&lt;p&gt;As discussed with Sean in &lt;a href=&quot;https://datascience.meta.stackexchange.com/q/2200/11097&quot;&gt;this Meta post&lt;/a&gt;, I thought it would be nice to have a question which can help people who were confused like me, to know about the differences between text mining and NLP!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what are the differences between &lt;a href=&quot;/questions/tagged/nlp&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;nlp&amp;#39;&quot; rel=&quot;tag&quot;&gt;nlp&lt;/a&gt; and &lt;a href=&quot;/questions/tagged/text-mining&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;text-mining&amp;#39;&quot; rel=&quot;tag&quot;&gt;text-mining&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I have included my understanding as an answer. If possible, please explain your answer with a brief example!&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-03-16T16:42:03.127" LastActivityDate="2020-03-10T10:11:24.800" Title="What is the difference between NLP and text mining?" Tags="|nlp|text-mining|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9866" PostTypeId="2" ParentId="9865" CreationDate="2016-01-20T06:39:32.650" Score="9" Body="&lt;p&gt;I have had this doubt since a long time. So, &lt;a href=&quot;http://discuss.analyticsvidhya.com/t/difference-between-nlp-and-text-mining/2977/2&quot; rel=&quot;noreferrer&quot;&gt;this post&lt;/a&gt; here helped me figure the differences between the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this is the difference between text mining and NLP:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Text Mining deals with the text itself, while NLP deals with the&#xA;  underlying/latent metadata.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Answering questions like - frequency counts of words, length of the&#xA;  sentence, presence/absence of certain words etc. is text mining.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;NLP on the other hand allows you to answer questions like;&#xA;  - What is the sentiment?&#xA;  - What are the keywords? (using POS tagging &amp;amp; parsers)&#xA;  - What category of content it falls under?&#xA;  - Which are the entities in the sentence? And more&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="8878" LastEditDate="2016-01-20T09:23:36.963" LastActivityDate="2016-01-20T09:23:36.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9867" PostTypeId="2" ParentId="9865" CreationDate="2016-01-20T06:56:14.430" Score="12" Body="&lt;p&gt;I agree with Sean's answer.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Natural_language_processing&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLP&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Text_mining&quot; rel=&quot;nofollow noreferrer&quot;&gt;text mining&lt;/a&gt; are usually used for different goals.&#xA;Also, there is indeed an overlap and both definitions are vogue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than the difference in goal, there is a difference in methods.&#xA;Text mining techniques are usually shallow and do not consider the text structure. Usually, text mining will use bag-of-words, n-grams and possibly stemming over that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In NLP methods usually involve the text structure. You can find there sentence splitting, part-of-speech tagging and parse tree construction. Also, NLP methods provide several techniques to capture context and meaning from text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical text mining method will consider the following sentences to indicate happiness while typical NLP methods detect that they are not&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I am not happy&lt;/li&gt;&#xA;&lt;li&gt;I will be happy when it will rain&lt;/li&gt;&#xA;&lt;li&gt;If it will rain, I'll be happy.&lt;/li&gt;&#xA;&lt;li&gt;She asked whether I am happy&lt;/li&gt;&#xA;&lt;li&gt;Are you happy?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="4879" LastEditDate="2020-03-10T10:11:24.800" LastActivityDate="2020-03-10T10:11:24.800" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="9868" PostTypeId="2" ParentId="9862" CreationDate="2016-01-20T09:53:43.583" Score="1" Body="&lt;p&gt;The gradient boost algorithm create a set of decision tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction process used &lt;a href=&quot;https://gist.github.com/shanebutler/5456942&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; use these steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;for each tree, create a temporary &quot;predicted variable&quot;, applying the tree to the new data set.&lt;/li&gt;&#xA;&lt;li&gt;use a formula to aggregate all these tree. Depending on the model:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;bernoulli: 1/(1 + exp(-(intercept + SUM(temporary pred))))&lt;/li&gt;&#xA;&lt;li&gt;poisson, gamma: exp(intercept + SUM(temporary pred))&lt;/li&gt;&#xA;&lt;li&gt;adaboost: 1 /(1 + exp(-2*(intercept + SUM(temporary pred))))&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The temporary &quot;predicted variable&quot; is a probability, having no sense by its own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more tree you have, the more smooth is your prediction.( as for each tree, only a finite set of value is spread across your observations)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R process is probably optimised, but it is enough to understand the concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the h2o implementation of the gradient boost, the output is a flag 0/1.&#xA;I think the &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F1 score&lt;/a&gt; is used by default to convert probability into flag. I'll do some search/test to confirm that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that same implementation, one of the default output for a binary outcome is a confusion matrix, which is a great way to assess your model ( and open a whole new bunch of interrogations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intercept is &quot;the initial predicted value to which trees make adjustments&quot;. Basically,just an initial adjustment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition: &lt;a href=&quot;http://h2o-release.s3.amazonaws.com/h2o/rel-tibshirani/8/docs-website/h2o-docs/booklets/GBM_Vignette.pdf&quot; rel=&quot;nofollow&quot;&gt;h2o.gbm documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15543" LastEditorUserId="15543" LastEditDate="2016-01-20T12:46:55.283" LastActivityDate="2016-01-20T12:46:55.283" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9869" PostTypeId="1" CreationDate="2016-01-20T12:53:31.883" Score="1" ViewCount="531" Body="&lt;p&gt;I am currently reading &lt;em&gt;Boosting the Performance of RBF Networks with Dynamic Decay Adjustment&lt;/em&gt; by Michael R. Berthold and Jay Diamond (&lt;a href=&quot;http://kops.uni-konstanz.de/handle/123456789/5427&quot; rel=&quot;nofollow&quot;&gt;online&lt;/a&gt;) to understand how Dynamic Decay Adjustment (DDA; a constructive trainining algorithm for RBF networks). Doing so, I stumbled over the word &lt;strong&gt;prototype&lt;/strong&gt; a couple of times:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Unfortunately PRCE networks do not adjust the standard deviation of their prototypes individually, using only one global value for this parameter.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;This paper introduces the Dynamic Decay Adjustment (DDA) algorithm which utilizes the constructive nature of the PRCE algorithm together with independent adaptation of each prototype's decay factor.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;PNNs are not suitable for large databases because they commit one new prototype for each training pattern they encounter, eeffectively becoming a referential memory scheme.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've tried to find it in the only resource they referenced in this context (D.L. Reilly, L.N. Cooper, C. Elbaum: &quot;A Neural Model for Category Learning&quot;), but sadly I don't have access to that one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found an explanation on &lt;a href=&quot;https://chrisjmccormick.wordpress.com/2013/08/15/radial-basis-function-network-rbfn-tutorial/&quot; rel=&quot;nofollow&quot;&gt;https://chrisjmccormick.wordpress.com/2013/08/15/radial-basis-function-network-rbfn-tutorial/&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An RBFN performs classification by measuring the input’s similarity to examples from the training set. Each RBFN neuron stores a “prototype”, which is just one of the examples from the training set. When we want to classify a new input, each neuron computes the Euclidean distance between the input and its prototype. Roughly speaking, if the input more closely resembles the class A prototypes than the class B prototypes, it is classified as class A.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So a prototype is just the parameters (center and radius, assuming Gaussians are used) of an RBFs neuron?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rephrasing the first quoted sentence, does it mean that the RBF networks usually only learn the center and the radius is fixed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is if I understood it correct. Please add a reference (which is not a random blog article) which makes it more clear.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-06-21T10:46:06.970" Title="What are prototypes in RBF networks?" Tags="|machine-learning|neural-network|reference-request|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9870" PostTypeId="2" ParentId="9850" CreationDate="2016-01-20T13:41:33.503" Score="40" Body="&lt;p&gt;This answer is on the &lt;em&gt;general&lt;/em&gt; side of cost functions, not related to TensorFlow, and will mostly address the &quot;some explanation about this topic&quot; part of your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most examples/tutorial I followed, the cost function used was somewhat arbitrary. The point was more to introduce the reader to a specific method, not to the cost function specifically. It should not stop you to follow the tutorial to be familiar with the tools, but my answer should help you on how to choose the cost function for your own problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want answers regarding Cross-Entropy, Logit, L2 norms, or anything specific, I advise you to post multiple, more specific questions. This will increase the probability that someone with specific knowledge will see your question.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Choosing the right cost function for achieving the desired result is a critical point of machine learning problems. The basic approach, if you do not know exactly what you want out of your method, is to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;noreferrer&quot;&gt;Mean Square Error (Wikipedia)&lt;/a&gt; for regression problems and Percentage of error for classification problems. However, if you want &lt;em&gt;good&lt;/em&gt; results out of your method, you need to &lt;em&gt;define good&lt;/em&gt;, and thus define the adequate cost function. This comes from both domain knowledge (what is your data, what are you trying to achieve), and knowledge of the tools at your disposal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not believe I can guide you through the cost functions already implemented in TensorFlow, as I have very little knowledge of the tool, but I can give you an example on how to write and assess different cost functions.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To illustrate the various differences between cost functions, let us use the example of the binary classification problem, where we want, for each sample &lt;span class=&quot;math-container&quot;&gt;$x_n$&lt;/span&gt;, the class &lt;span class=&quot;math-container&quot;&gt;$f(x_n) \in \{0,1\}$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting with &lt;strong&gt;computational properties&lt;/strong&gt;; how two functions measuring the &quot;same thing&quot; could lead to different results. Take the following, simple cost function; the percentage of error. If you have &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; samples, &lt;span class=&quot;math-container&quot;&gt;$f(y_n)$&lt;/span&gt; is the predicted class and &lt;span class=&quot;math-container&quot;&gt;$y_n$&lt;/span&gt; the true class, you want to minimize&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{1}{N} \sum_n \left\{&#xA;\begin{array}{ll}&#xA;1 &amp;amp; \text{ if } f(x_n) \not= y_n\\&#xA;0 &amp;amp; \text{ otherwise}\\&#xA;\end{array} \right. = \sum_n y_n[1-f(x_n)] + [1-y_n]f(x_n)$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This cost function has the benefit of being easily interpretable. However, it is not smooth; if you have only two samples, the function &quot;jumps&quot; from 0, to 0.5, to 1. This will lead to inconsistencies if you try to use gradient descent on this function. One way to avoid it is to change the cost function to use probabilities of assignment; &lt;span class=&quot;math-container&quot;&gt;$p(y_n = 1 | x_n)$&lt;/span&gt;. The function becomes&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{1}{N} \sum_n y_n p(y_n = 0 | x_n) + (1 - y_n) p(y_n = 1 | x_n)$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This function is smoother, and will work better with a gradient descent approach. You will get a 'finer' model. However, it has other problem; if you have a sample that is ambiguous, let say that you do not have enough information to say anything better than &lt;span class=&quot;math-container&quot;&gt;$p(y_n = 1 | x_n) = 0.5$&lt;/span&gt;. Then, using gradient descent on this cost function will lead to a model which increases this probability as much as possible, and thus, maybe, overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem of this function is that if &lt;span class=&quot;math-container&quot;&gt;$p(y_n = 1 | x_n) = 1$&lt;/span&gt; while &lt;span class=&quot;math-container&quot;&gt;$y_n = 0$&lt;/span&gt;, you are certain to be right, but you are wrong. In order to avoid this issue, you can take the log of the probability, &lt;span class=&quot;math-container&quot;&gt;$\log p(y_n | x_n)$&lt;/span&gt;. As &lt;span class=&quot;math-container&quot;&gt;$\log(0) = \infty$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\log(1) = 0$&lt;/span&gt;, the following function does not have the problem described in the previous paragraph:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{1}{N} \sum_n y_n \log p(y_n = 0 | x_n) + (1 - y_n) \log p(y_n = 1 | x_n)$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This should illustrate that in order to optimize the &lt;em&gt;same thing&lt;/em&gt;, the percentage of error, different definitions might yield different results if they are easier to make sense of, computationally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;It is possible for cost functions &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$B$&lt;/span&gt; to measure the &lt;em&gt;same concept&lt;/em&gt;, but &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; might lead your method to better results than &lt;span class=&quot;math-container&quot;&gt;$B$&lt;/span&gt;.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now let see how different costs function can measure different concepts. In the context of information retrieval, as in google search (if we ignore ranking), we want the returned results to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;have high &lt;em&gt;precision&lt;/em&gt;, not return irrelevant information&lt;/li&gt;&#xA;&lt;li&gt;have high &lt;em&gt;recall&lt;/em&gt;, return as much relevant results as possible&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;noreferrer&quot;&gt;Precision and Recall (Wikipedia)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that if your algorithm returns &lt;em&gt;everything&lt;/em&gt;, it will return every relevant result possible, and thus have high recall, but have very poor precision. On the other hand, if it returns only &lt;em&gt;one&lt;/em&gt; element, the one that it is the most certain is relevant, it will have high precision but low recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to judge such algorithms, the common cost function is the &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;noreferrer&quot;&gt;&lt;span class=&quot;math-container&quot;&gt;$F$&lt;/span&gt;-score (Wikipedia)&lt;/a&gt;. The common case is the &lt;span class=&quot;math-container&quot;&gt;$F_1$&lt;/span&gt;-score, which gives equal weight to precision and recall, but the general case it the &lt;span class=&quot;math-container&quot;&gt;$F_\beta$&lt;/span&gt;-score, and you can tweak &lt;span class=&quot;math-container&quot;&gt;$\beta$&lt;/span&gt; to get&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Higher recall, if you use &lt;span class=&quot;math-container&quot;&gt;$\beta &amp;gt; 1$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;Higher precision, if you use &lt;span class=&quot;math-container&quot;&gt;$\beta &amp;lt; 1$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In such scenario, &lt;strong&gt;choosing the cost function is choosing what trade-off your algorithm should do&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example that is often brought up is the case of medical diagnosis, you can choose a cost function that punishes more false negatives or false positives depending on what is preferable:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;More healthy people being classified as sick (But then, we might treat healthy people, which is costly and might hurt them if they are actually not sick)&lt;/li&gt;&#xA;&lt;li&gt;More sick people being classified as healthy (But then, they might die without treatment)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In conclusion, defining the cost function is defining the goal of your algorithm. The algorithm defines how to get there.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Side note: Some cost functions have nice algorithm ways to get to their goals. For example, a nice way to the minimum of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hinge_loss&quot; rel=&quot;noreferrer&quot;&gt;Hinge loss (Wikipedia)&lt;/a&gt; exists, by solving the dual problem in &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;noreferrer&quot;&gt;SVM (Wikipedia)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15501" LastEditorUserId="21519" LastEditDate="2019-03-06T08:23:26.230" LastActivityDate="2019-03-06T08:23:26.230" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9873" PostTypeId="2" ParentId="9495" CreationDate="2016-01-20T14:40:27.380" Score="2" Body="&lt;p&gt;This does not answere your question directly, but might include some helpful pointers:&lt;/p&gt;&#xA;&lt;p&gt;In the recent &lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Submitted on 10 Dec 2015&quot;&gt;Deep Residual Learning for Image Recognition&lt;/a&gt; paper it is written:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;If one hypothesizes that multiple nonlinear layers can asymptotically&#xA;approximate complicated functions[...]&lt;/p&gt;&#xA;&lt;p&gt;This hypothesis, however, is still an open question. See [&lt;a href=&quot;http://arxiv.org/abs/1402.1869&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;G. Montufar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014.&quot;&gt;28&lt;/a&gt;].&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14390" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-01-20T14:40:27.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9874" PostTypeId="2" ParentId="9839" CreationDate="2016-01-20T18:55:57.010" Score="-1" Body="&lt;p&gt;There is a paper that was published at Systems&amp;amp;Cybernetics journal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See how they use Bayesian Networks with sensor fusion:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1605392&amp;amp;tag=1&quot; rel=&quot;nofollow&quot;&gt;Active and dynamic information fusion for multi-sensor systems with dynamic bayesian networks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastEditorUserId="25640" LastEditDate="2016-10-29T11:33:18.740" LastActivityDate="2016-10-29T11:33:18.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9875" PostTypeId="1" CreationDate="2016-01-20T19:03:33.203" Score="3" ViewCount="2759" Body="&lt;p&gt;Cloudera recently added the &lt;a href=&quot;https://github.com/cloudera/spark-timeseries&quot; rel=&quot;nofollow&quot;&gt;spark-time series library to github&lt;/a&gt;. According to the &lt;a href=&quot;http://cloudera.github.io/spark-timeseries/0.2.0/scaladocs/index.html#com.cloudera.sparkts.models.ARIMA$&quot; rel=&quot;nofollow&quot;&gt;user docs&lt;/a&gt;, it definitely can fit autoregressive integrated moving average (&lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMA&lt;/a&gt;) models, but I see no mention of ARIMAX, which takes into account explanatory variables. Does anyone with more experience with this library know whether ARIMAX is possible in the library's current implementation? Mathematically, I understand the difference between ARIMA and ARIMAX, and I know extending it myself wouldn't be terribly difficult, but I'm right now looking for an off-the-shelf solution in spark. Can anyone recommend an alternative spark implementation?&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2016-01-20T19:18:31.220" LastActivityDate="2016-01-20T21:15:52.153" Title="ARIMAX with spark-timeseries" Tags="|time-series|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9876" PostTypeId="2" ParentId="9875" CreationDate="2016-01-20T21:15:52.153" Score="1" Body="&lt;p&gt;Blast. It looks as though ARIMAX is &lt;a href=&quot;https://github.com/cloudera/spark-timeseries/issues/85&quot; rel=&quot;nofollow&quot;&gt;currently a work in progress&lt;/a&gt; for the cloudera/spark-timeseries community. Although &lt;a href=&quot;https://github.com/cloudera/spark-timeseries/pull/51&quot; rel=&quot;nofollow&quot;&gt;some work has been done&lt;/a&gt; on the model development side, it doesn't appear to have been merged in. &lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-01-20T21:15:52.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9877" PostTypeId="1" AcceptedAnswerId="9903" CreationDate="2016-01-21T00:40:12.583" Score="2" ViewCount="3817" Body="&lt;p&gt;For what it's worth my particular case is a symmetrical matrix, but this question should be answered more generally.&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2016-01-22T19:17:40.530" Title="What is the best file format to store an uncompressed 2D matrix?" Tags="|data-formats|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9878" PostTypeId="2" ParentId="9860" CreationDate="2016-01-21T07:42:08.003" Score="1" Body="&lt;p&gt;There is &lt;code&gt;df.drop&lt;/code&gt; command that can be used as follows to remove certain rows (in this case, 15 &amp;amp; 16):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.drop(df.index[[15,16]])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the rows you don't need are regular (e.g. you never need row 15) then this is a quick and dirty solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only want to drop arbitrary rows containing some value, this should do the trick:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df = df.drop([df.column_name == ©1990-2016 AAR])&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-01-21T07:42:08.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9879" PostTypeId="2" ParentId="9877" CreationDate="2016-01-21T09:39:22.573" Score="2" Body="&lt;p&gt;I would go with &lt;em&gt;.csv&lt;/em&gt; as it's universally accepted and can be read in different programming languages easily. Moreover you can simply open it with an office software. If you are using your matrix only in Python I also recommend &lt;a href=&quot;https://wiki.python.org/moin/UsingPickle&quot; rel=&quot;nofollow&quot;&gt;Pickle library&lt;/a&gt; which writes your matrix in a &lt;em&gt;.p&lt;/em&gt; format and can be easily read in Python with a simple load function.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-21T09:39:22.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9880" PostTypeId="1" AcceptedAnswerId="10306" CreationDate="2016-01-21T11:02:06.503" Score="3" ViewCount="151" Body="&lt;p&gt;I am trying to analyze soccer's data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W_OVER_2_5  PREDICTED MATCH_DATE LEAGUE HOME AWAY MATCH_HOME MATCH_DRAW MATCH_AWAY  MATCH_U2_50 MATCH_O2_50&#xA;0            0        1105135200    5   260  289    2.05        3.00        4.50        1.65       2.30&#xA;0            1        1105308000    16  715  700    2.50        3.30        3.05        1.80       2.14&#xA;1            1        1105308000    11  445  479    1.36        5.25       12.00        2.15       1.78&#xA;0            1        1105308000    11  453  474    3.00        3.35        2.62        1.75       2.20....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I selected 'the best estimator' - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,&#xA;          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,&#xA;          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,&#xA;          verbose=1, warm_start=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with best coefs - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. -2.40477246e-10  &#xA;2. -5.57611571e-02  &#xA;3. -1.32010761e-04   &#xA;4.  1.51666398e-03&#xA;5.  7.54521399e-02   &#xA;6.  6.38889247e-02  &#xA;7. -2.25746953e-01  &#xA;8. -3.79313902e-01&#xA;9.  3.70514297e-02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I have a question - how should I understand coefs in terms of real strategy? As example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If `MATCH_HOME` is min among all [`MATCH_HOME`, `MATCH_DRAW`, `MATCH_WAY`] AND `MATCH_O2_50' = 1 &#xA;THEN PREDICTED := 1&#xA;ELSE PREDICTED := 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PS. I would very appreciated for any science papers about that thematic :)&lt;/p&gt;&#xA;" OwnerUserId="14684" LastEditorUserId="11097" LastEditDate="2016-01-21T11:11:58.667" LastActivityDate="2016-02-18T14:25:11.430" Title="How should I convert Logistic Regression's coefs into action strategy?" Tags="|machine-learning|classification|python|predictive-modeling|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9881" PostTypeId="2" ParentId="1186" CreationDate="2016-01-21T11:29:32.980" Score="4" Body="&lt;p&gt;I don't see a huge problem here. So, I would try to answer all of your questions from the production-level point of view: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do I confirm that classification will not take much time?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Take a subset of the corpus data you have (you can randomly do it, no need of sampling), and test your algorithm on it, and they approximate/generalize it to the overall dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(SVM is comparatively faster. Nevertheless, do the above process just to be sure.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And do test it in the development environment before pushing to production.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is using Python's scikit library for websites/web applications&#xA;  suitable for this?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, it is. It is &lt;a href=&quot;http://scikit-learn.org/stable/testimonials/testimonials.html&quot; rel=&quot;nofollow&quot;&gt;already being used&lt;/a&gt; by a nice chunk of companies out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third question about Amazon and Flipkart cannot be answered by someone outside their teams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, I would advise you to use the mapreduce techniques for training your models. And as already advised, pickle your models so that you don't need to train them with every request.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-21T11:29:32.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9882" PostTypeId="2" ParentId="8464" CreationDate="2016-01-21T11:59:55.633" Score="2" Body="&lt;p&gt;I experimented with PyBrain and PyLearn2&#xA;I rather find Keras [&lt;a href=&quot;http://keras.io/&quot; rel=&quot;nofollow&quot;&gt;http://keras.io/&lt;/a&gt; ] as an excellent choice. It way ahead of PyBrain for sure. PyBrain development has been stopped&lt;/p&gt;&#xA;" OwnerUserId="13457" LastEditorUserId="13457" LastEditDate="2016-01-21T18:16:27.650" LastActivityDate="2016-01-21T18:16:27.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9883" PostTypeId="1" CreationDate="2016-01-21T12:14:59.203" Score="2" ViewCount="222" Body="&lt;p&gt;I found some realisation of pretty interesting ML algorithm. &lt;a href=&quot;https://github.com/EderSantana/DeepEEG&quot; rel=&quot;nofollow&quot;&gt;https://github.com/EderSantana/DeepEEG&lt;/a&gt; We have dataset X, matrix of spacial filter W, matrix of time filter V, and matrices U and B - parameters of classyfier. &#xA;$Xfilt_i=ln(W*(X_i*v)^2))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Z=(Xfilt)^T * U+B$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y\_pred = argmax(softmax(Z))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, quite simple logistic regression classification. But most inreresting thing was the way authors tune W,V,U and B  matrices. W was initialized by CSP algorith. V was initialized by ones, U and B by zeros. And then cross-entropy cost function optimised (mini-batch gradient descent) respect to this parameters. And accuracy, achived by this way pretty high. My quiestion is, why we need non-linearity here (squaring and log)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Xfilt_i=ln(W*(X_i*v)^2))$&#xA;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS In experiments optimisation algorithm did not converge without this nonlineraity.  &lt;/p&gt;&#xA;" OwnerUserId="15595" LastEditorUserId="15595" LastEditDate="2016-01-25T11:12:53.823" LastActivityDate="2016-01-25T11:12:53.823" Title="Non-linear data preprocessing before mini-batch gradient descent" Tags="|machine-learning|classification|gradient-descent|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9884" PostTypeId="2" ParentId="9846" CreationDate="2016-01-21T13:30:26.147" Score="1" Body="&lt;p&gt;In sklearn Anything that will take sparse data can take output from TFIDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sklearn basically all data any model can take is either dense (normal array) or sparse (that only stores the location of values != 0). You can convert from sparse to dense, but chances are you'll run out of memory if you try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite a big fan of using linear algorithms on text data (sgdregressor for example have a ton of different options you can play with). But other algorithms like randomforest that Sergey mentioned, and naive bayes models can also work with that kind of data. Basically what you are looking for is anything that can take sparse input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(One thing I've done in the past when working with text+other data is taking output from an sgd's analysis of the data and feeding that + the other data to another algorithm like randomforest. It's a simple and quite powerful method)&lt;/p&gt;&#xA;" OwnerUserId="15599" LastActivityDate="2016-01-21T13:30:26.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9885" PostTypeId="1" AcceptedAnswerId="13019" CreationDate="2016-01-21T14:47:08.167" Score="11" ViewCount="2479" Body="&lt;p&gt;There is a similar question here in CrossValidated, and I have read the answers. My question is a bit different. I don't want to merely visualize my data, and indeed what I want to visualize is not easy to visualize with either package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two sets of points ($x,y$ coordinates) on my plot. I want to add edges and make it similar to a graph. If I have $n$ points, then based on the nature of my problem, I need to add $(n/2)^2$ edges to the plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I want to add some interactive features to my plot. For instance, by clicking on one point (vertex), all other vertices which are not connected to this clicked vertex must disappear to have a more lucid and focused plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to find a network visualization package suited to my project, but it seems all of them use the data about connection of edges or nodes and provide a network based on some specific layout algorithm. My data is different. I have the coordinates of nodes and I want to establish some edges between some of them. In my data the nodes have their own fixed locations and must not be moved around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I narrowed down the packages to &lt;code&gt;ggvis&lt;/code&gt; and &lt;code&gt;ggplot2&lt;/code&gt;. &lt;code&gt;ggvis&lt;/code&gt; is new and fresh in addition to having intentional design to generate interactive plots. But I found out that &lt;code&gt;ggplot2+Shiny&lt;/code&gt; can also produce interactive plots. (Is that correct?) Besides, I figured that &lt;code&gt;ggplot2&lt;/code&gt;'s capabilities are more varied and matured than &lt;code&gt;ggvis&lt;/code&gt;: for instance &lt;code&gt;ggplot2&lt;/code&gt; has a zooming feature while &lt;code&gt;ggvis&lt;/code&gt; does not have it. More important, as &lt;code&gt;ggvis&lt;/code&gt; is growing and changing, I don't want to invest a lot of time on a work that becomes technically obsolete or buggy in a year or so. I thought this is the case if I code based on &lt;code&gt;ggvis&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now may you please evaluate my choice (&lt;code&gt;ggplot2+shiny&lt;/code&gt;) and advise me about possibility to do what I want to do with it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I did not know any of the mentioned packages, I have started learning &lt;code&gt;ggplot2&lt;/code&gt; from Wickham's book and I love it! But I am afraid of spending a few weeks and finding out that this amazing package cannot do what I want to do even with the aid of &lt;code&gt;Shiny&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: I want to put the final code on github so probable users can download and run it. Therefore, I do not need to make a web application, there is no need to any website or putting the interactive map online. I just need to have an interactive plot which can be run from inside of Rstudio. &lt;/p&gt;&#xA;" OwnerUserId="15604" LastEditorUserId="15604" LastEditDate="2016-01-22T11:32:03.317" LastActivityDate="2018-12-03T12:42:56.173" Title="ggvis vs. ggplot2+Shiny; which one to choose for interactive visualization?" Tags="|r|visualization|ggplot2|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9886" PostTypeId="1" AcceptedAnswerId="9902" CreationDate="2016-01-21T16:08:59.477" Score="6" ViewCount="1441" Body="&lt;p&gt;I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like &quot;stronger leadership&quot;, &quot;better customer service&quot;, &quot;programs&quot;, and etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, can I use this as a training data set in order to develop a model that I can use in the future as we get more survey responses? We would like to be able to tag, label, or classify each survey response into (up to) 3 of the 70 categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this even possible? Or do I have to use a NB with simple words? Can you please guide me to tutorials, examples, etc.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using R in this exercise.&lt;/p&gt;&#xA;" OwnerUserId="15609" LastEditorUserId="15609" LastEditDate="2016-01-21T16:36:58.603" LastActivityDate="2016-01-22T08:36:18.620" Title="Classifying survey response text SVM" Tags="|machine-learning|r|text-mining|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9887" PostTypeId="2" ParentId="155" CreationDate="2016-01-21T17:05:30.203" Score="4" Body="&lt;p&gt;Yahoo just released a &lt;a href=&quot;http://webscope.sandbox.yahoo.com/catalog.php?datatype=l&quot; rel=&quot;nofollow&quot;&gt;huge dataset&lt;/a&gt; for research community. Enjoy it!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-21T17:05:30.203" CommentCount="0" CommunityOwnedDate="2016-03-16T10:12:10.880" ContentLicense="CC BY-SA 3.0" />
  <row Id="9888" PostTypeId="2" ParentId="9886" CreationDate="2016-01-21T17:06:52.900" Score="3" Body="&lt;p&gt;Can you clarify what you're trying to predict with these responses?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial reaction is that with open-ended surveys you'll have a tough time implementing classification algorithms. Open-ended-ness means you don't have a finite feature space and thus you can't do the usual transformation of responses into a feature matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there may be other ways to make this work. If you have 5 questions, for instance, you may be able to use sentiment analysis or other methods to come up with metadata about the open ended response that can help you design a classification scheme.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-01-21T17:06:52.900" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="9889" PostTypeId="1" AcceptedAnswerId="9897" CreationDate="2016-01-21T17:59:08.770" Score="3" ViewCount="109" Body="&lt;p&gt;I'm new to ML. I'm taking over a Classification project which involves analyzing data for customers which returned a product and I need to determine the return reason (~10 categories). &#xA;This data was captured at the counter, and could include words like: LGTM (Looks good to me) NFF (No fault found), etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a training set of 1000 records and when using Google Prediction API I get a &quot;classificationAccuracy&quot; value of &quot;0.82&quot; and 10 labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&#xA;1. Any recommended API to analyze this type of data?.&#xA;2. What is a good  &quot;classificationAccuracy&quot; value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2017-03-08T22:31:39.140" LastActivityDate="2017-03-08T22:31:39.140" Title="Analyzing customer response" Tags="|machine-learning|data-mining|classification|google-prediction-api|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9890" PostTypeId="1" CreationDate="2016-01-21T19:36:21.350" Score="0" ViewCount="1221" Body="&lt;p&gt;I am trying to read in an excel file into a pandas data frame using &lt;code&gt;pandas.read_excel()&lt;/code&gt; function. The first column is &lt;code&gt;'Date'&lt;/code&gt;. The first entry into that column has asterisk mark which I want to remove. Here are the first 2 rows of the data frame &lt;code&gt;DF&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;0   1/4/1996*   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN &#xA;1   1996-01-11 00:00:00 13881   11735   2532    4188    32064   21989   3819    2586    0   NaN NaN`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In order to remove that &lt;code&gt;*&lt;/code&gt;, I am doing this:&lt;code&gt;DF['Date'][0] = '1/4/1996'&lt;/code&gt;. However, once I do that, I am getting a warning message&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:3: SettingWithCopyWarning: &#xA;A value is trying to be set on a copy of a slice from a DataFrame&#xA;&#xA;See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy&#xA;  app.launch_new_instance()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;and it changes the cell value, but, it also attaches another &lt;code&gt;'Date'&lt;/code&gt; column at the end of the &lt;code&gt;df&lt;/code&gt; like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;0   1/4/1996   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN 1/4/1996&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I don't understand why this is happening.&lt;br /&gt;&#xA;I would greatly appreciate it if someone can explain this or point out the correct way of changing the cell value.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="98307" LastEditDate="2020-07-31T14:25:46.103" LastActivityDate="2020-07-31T14:25:46.103" Title="Problem encountered while updating a particular cell of a pandas data frame in Python" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9892" PostTypeId="1" AcceptedAnswerId="9893" CreationDate="2016-01-21T20:15:04.757" Score="14" ViewCount="8895" Body="&lt;p&gt;I know someone who is working on a project that involves ingesting files of data without regard to the columns or data types. The task is to take a file with any number of columns and various data types and output summary statistics on the numerical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, he is unsure of how to go about dynamically assigning data types for certain number-based data. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    CITY&#xA;    Albuquerque&#xA;    Boston&#xA;    Chicago&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is obviously not numerical data and will be stored as text. However,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    ZIP&#xA;    80221&#xA;    60653&#xA;    25525&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;are not clearly marked as categorical. His software would assign the ZIP code as numerical and output summary statistics for it, which does not make sense for that sort of data.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A couple ideas we had were:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If a column is all integers, label it as categorical. This clearly wouldn't work, but it was an idea.&lt;/li&gt;&#xA;&lt;li&gt;If a column has fewer than &lt;em&gt;n&lt;/em&gt; unique values and is numeric, label it categorical. This might be closer, but there could still be issues with numerical data falling through.&lt;/li&gt;&#xA;&lt;li&gt;Maintain a list of common numeric data that should actually be categorical and compare the column headers to this list for matches. For example, anything with &quot;ZIP&quot; in it would be categorical.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My gut tells me that there is no way to accurately assign numeric data as categorical or numerical, but was hoping for a suggestion. Any insight you have is greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="15617" LastActivityDate="2016-01-22T12:37:10.353" Title="How can I dynamically distinguish between categorical data and numerical data?" Tags="|categorical-data|numerical|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="9893" PostTypeId="2" ParentId="9892" CreationDate="2016-01-21T22:25:51.053" Score="10" Body="&lt;p&gt;I'm not aware of a foolproof way to do this. Here's one idea off the top of my head:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Treat values as categorical by default.&lt;/li&gt;&#xA;&lt;li&gt;Check for various attributes of the data that would imply it is actually continuous. Weight these attributes based on how likely they are to correlate with continuous data. Here are some possible examples:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Values are integers: +.7&lt;/li&gt;&#xA;&lt;li&gt;Values are floats: +.8&lt;/li&gt;&#xA;&lt;li&gt;Values are normally distributed: +.3&lt;/li&gt;&#xA;&lt;li&gt;Values contain a relatively small number of unique values: +.3&lt;/li&gt;&#xA;&lt;li&gt;Values aren't all the same number of characters: +.1&lt;/li&gt;&#xA;&lt;li&gt;Values don't contain leading zeros: +.1&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Treat any columns that sum to greater than 1 as being numerical. Adjust the factors and weights based on testing against different data sets to suit your needs. You could even build and train a separate machine learning algorithm just to do this.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2016-01-21T22:25:51.053" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9894" PostTypeId="1" CreationDate="2016-01-21T22:51:03.573" Score="2" ViewCount="2928" Body="&lt;p&gt;I am working on &lt;code&gt;Random Forest algorithm&lt;/code&gt; in &lt;code&gt;PySpark&lt;/code&gt; &lt;code&gt;MLlib&lt;/code&gt; and have a doubt regarding the number of trees parameter that we pass to the model. The standard format of &lt;code&gt;Random Forest modeling&lt;/code&gt; in &lt;code&gt;PySpark&lt;/code&gt; &lt;code&gt;MLlib&lt;/code&gt; is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = RandomForest.trainRegressor(trainingData, categoricalFeaturesInfo={},&#xA;                                    numTrees=3, featureSubsetStrategy=&quot;auto&quot;,&#xA;                                    impurity='variance', maxDepth=4, maxBins=32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The doubt that I have is how to decide the optimum value of trees to pass to &lt;code&gt;numTrees&lt;/code&gt; parameter? I assume the more the number of trees better should be the performance but would it keep on improving with increase in number of trees? Is there a point after which it will start to have negative impact in performance? If yes then how can I find the optimum number of trees for my dataset?&lt;/p&gt;&#xA;" OwnerUserId="14384" LastEditorUserId="21" LastEditDate="2016-05-10T13:57:19.243" LastActivityDate="2016-05-10T13:57:19.243" Title="How to decide the number of trees parameter for Random Forest algorithm in PySpark MLlib?" Tags="|machine-learning|regression|random-forest|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9895" PostTypeId="2" ParentId="6635" CreationDate="2016-01-21T23:50:16.067" Score="2" Body="&lt;p&gt;If you use SGNS or CBOW, &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;https://radimrehurek.com/gensim/&lt;/a&gt; will let you continue continue training a pre-trained model on a different corpus if that is what you mean by fine tuning.&lt;/p&gt;&#xA;" OwnerUserId="15626" LastActivityDate="2016-01-21T23:50:16.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9896" PostTypeId="2" ParentId="9823" CreationDate="2016-01-21T23:58:17.550" Score="5" Body="&lt;p&gt;I would recommend Bayesian Optimization for hyper parameter search and had good results with &lt;a href=&quot;https://github.com/HIPS/Spearmint&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spearmint&lt;/a&gt;. You might have to use an older version for commercial use.&lt;/p&gt;&#xA;" OwnerUserId="15626" LastEditorUserId="98307" LastEditDate="2020-08-03T14:25:28.380" LastActivityDate="2020-08-03T14:25:28.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9897" PostTypeId="2" ParentId="9889" CreationDate="2016-01-22T01:00:00.070" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the above tutorial to get acquaintance with text classification. Afterwards it should be easier to formulate nontrivial questions to move even further.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-01-22T01:00:00.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9898" PostTypeId="1" AcceptedAnswerId="9901" CreationDate="2016-01-22T04:47:28.773" Score="2" ViewCount="224" Body="&lt;p&gt;When I run Kmeans on my dataset, I notice that some centroids become stale in the they are no longer the closest centroid to any point after some iteration. Right now I am skipping these stale centroids in my next iteration because I think those centroids no longer represent any useful set of the data, however I wanted to know if there are other reasonable ways to deal with these centroids.&lt;/p&gt;&#xA;" OwnerUserId="15631" LastActivityDate="2016-01-22T08:12:36.267" Title="What to do with stale centroids in K-means" Tags="|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9899" PostTypeId="1" AcceptedAnswerId="10259" CreationDate="2016-01-22T05:10:26.040" Score="7" ViewCount="279" Body="&lt;p&gt;Here my goal is…&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Find Product 5 (New Product) is really influencing other product sales (product 1 to 4) or not?&lt;/li&gt;&#xA;&lt;li&gt;If it is influencing other product sales, how much?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;New to R and tried several related posts but didn’t find exact answer to my question. I love R and learning every day something new which helping us in taking data driven decisions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My sample Dataset is like below (Week and Product 1 to Product 5 Sales per each week)&#xA;Here my new product is Product 5 and launched on Week 5.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Week    Product-1   Product-2   Product-3   Product-4   Product-5&#xA;1   2   4   5   5   &#xA;2   4   4   6   4   &#xA;3   4   4   6   5   &#xA;4   4   4   6   6   4&#xA;5   4   6   5   3   5&#xA;6   2   7   6   4   3&#xA;7   3   8   7   5   6&#xA;8   2   9   9   3   6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Here my questions are&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best process or model to show the influence of product 5 (statistically)?&lt;/li&gt;&#xA;&lt;li&gt;Do I need to run co-integration tests before I run correlation?&#xA;Example some of these products are never be correlated with Product 5&#xA;(example: growth in cockle growth vs. growth in electricity demand)&lt;/li&gt;&#xA;&lt;li&gt;How I know correlation vs. causation in this mix?&lt;/li&gt;&#xA;&lt;li&gt;Since my new&#xA;product launched on week 5, where I can start my correlations? Is it&#xA;from week 5 or from earlier weeks?&lt;/li&gt;&#xA;&lt;li&gt;Do I need to test for stationarity first? and bring the data to stationary?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9663" LastEditorUserId="15527" LastEditDate="2016-01-24T21:26:45.070" LastActivityDate="2016-02-16T22:15:52.360" Title="Time series data: How I measure influence of new product sales on existing product sales (statistically)?" Tags="|machine-learning|r|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9900" PostTypeId="2" ParentId="9890" CreationDate="2016-01-22T06:15:55.073" Score="4" Body="&lt;p&gt;Use a combination of string processing and type casting to process the whole column in one swoop:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;excelDF[datecol] = pandas.to_datetime(excelDF[datecol].str.replace('*',''))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I tried using &lt;code&gt;str.rstrip('*')&lt;/code&gt; but I could not get it to work.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-22T06:15:55.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9901" PostTypeId="2" ParentId="9898" CreationDate="2016-01-22T08:12:36.267" Score="2" Body="&lt;p&gt;&lt;em&gt;k-means&lt;/em&gt; finds only a local optima. Thus a wrong number of cluster or simply some random state of equilibrium in the attracting forces could lead to empty clusters. Technically k-means does not provide a procedure for that, but you can enrich the algorithm with no problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two approaches which I found that are useful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;remove the stale cluster, choose a random instance from your data set and create a new cluster with centroid equal with the chosen random point&lt;/li&gt;&#xA;&lt;li&gt;remove the stale cluster, choose the farthest distant point from any other centroids, create a new cluster with centroid in that point&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both procedures can lead to indefinite running time, but if the number of this kind of adjustments is finite (and usually it is) that it will converge with no problem. To guard yourself from infinite running time you can set an upper bound for the number of &lt;em&gt;adjustments&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure itself is not practical if you have a huge data set a a large number of clusters. The running time can became prohibitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another procedure to decrease the chances for that to happen is to use a better initialization procedure, like &lt;em&gt;k-means++&lt;/em&gt;. In fact the second suggestion is an idea from &lt;em&gt;k-means++&lt;/em&gt;. There are no guarantees, however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally a note regarding implementation. If you can't change to code of the algorithm to make those improvements on the fly, your only option which comes to my mind is to start a new clustering procedure where you initialize the centroid positions for non-stale clusters, and follow procedures for stale clusters.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-01-22T08:12:36.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9902" PostTypeId="2" ParentId="9886" CreationDate="2016-01-22T08:36:18.620" Score="3" Body="&lt;p&gt;Assigning ~3 of 70 categories means you would be performing &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-label_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-label classification&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, it doesn't make much difference if you use Naive Bayes or SVM; they are both families of algorithms that translate provided independent variables (your feature space) into hopefully correct dependent variables (target classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to construct a good feature space. The state of the art approaches in text mining are (or were) first tokenizing words, stripping punctuation and &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow noreferrer&quot;&gt;stop words&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/16069406/text-mining-with-the-tm-package-word-stemming&quot;&gt;stemming&lt;/a&gt; or &lt;a href=&quot;https://stackoverflow.com/questions/28214148/how-to-perform-lemmatization-in-r&quot;&gt;lemmatizing&lt;/a&gt; them, creating a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag-of-words model&lt;/a&gt; of those words' &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;relative frequencies&lt;/a&gt; and perhaps the frequencies of those words' &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow noreferrer&quot;&gt;bigrams or trigrams&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then run your classification learners on that. Assume the resulting feature space table might get really wide (lots of words and combinations of words), so you might want to consider some form of &lt;a href=&quot;https://stats.stackexchange.com/questions/3048/how-to-do-dimensionality-reduction-in-r&quot;&gt;dimensionality reduction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you will have to repeat the same filtering process with exact same parameters for each new survey you want to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's another good batch of answers on &lt;a href=&quot;https://stats.stackexchange.com/questions/12907/off-the-shelf-tool-for-multi-label-classification&quot;&gt;multi-label text classification&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-22T08:36:18.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9903" PostTypeId="2" ParentId="9877" CreationDate="2016-01-22T08:48:23.160" Score="2" Body="&lt;p&gt;The most compatible format is surely CSV/TSV. It's text and you can usually Gzip it on the fly with the software package you are using. There is no widely standardized format for storing matrix array data. Matlab has its *.mat files, NumPy has *.npz, Stata and SAS have their own, ... Best just use a clear-text file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the matrix is symmetric, if it is very large or if there will be a lot of them, you could spare 50% in space requirement by storing only the lower (or upper) triangular part of it. If you chose to do so, there is, again, no widely accepted format. Just store the shape first and then the flattened, 1D data.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-01-22T08:48:23.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9904" PostTypeId="2" ParentId="9892" CreationDate="2016-01-22T11:07:40.893" Score="4" Body="&lt;p&gt;If you have, for example, number of children of a family (which could range, for example, between 0 and 5), is it a categorical or numerical variable? Actually it depends on your problem and how you intend to solve it. In this sense, you can do the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Compute the number of unique values of that column&lt;/li&gt;&#xA;&lt;li&gt;Divide this number by the total number of rows&lt;/li&gt;&#xA;&lt;li&gt;If this ratio is bellow some threshold (for example, 20%), you consider it categorical.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In case of discrete values, one additional test could be: use a regression model to estimate some of the parameters and check if the estimated values are contained in the original set of values. If this is not true, you are probably dealing with categorical data (as it is the case of ZIP).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It worked relatively well for me in the past...&lt;/p&gt;&#xA;" OwnerUserId="15215" LastActivityDate="2016-01-22T11:07:40.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9905" PostTypeId="2" ParentId="9890" CreationDate="2016-01-22T11:38:22.840" Score="2" Body="&lt;p&gt;The issue is in how Pandas indices / selectors work. When you only do &lt;code&gt;Dataframe['Column']&lt;/code&gt;, you are selecting a copy/view of the dataframe. If you do &lt;code&gt;Dataframe['NewColumn']&lt;/code&gt;, so the &lt;code&gt;'NewColumn'&lt;/code&gt; column doesn't exist yet, it creates this new column. But if it does exist, it will raise this warning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you should do is use &lt;code&gt;.iloc&lt;/code&gt;, &lt;code&gt;.ix&lt;/code&gt; or something similar to select data from the frame and be able to alter it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or follow @Emre advice.&lt;/p&gt;&#xA;" OwnerUserId="12580" LastActivityDate="2016-01-22T11:38:22.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9906" PostTypeId="2" ParentId="9892" CreationDate="2016-01-22T12:13:07.807" Score="2" Body="&lt;p&gt;Since this question has been cross-posted, the initial comments by @nickcox on Cross Validated are highly relevant and true. My views are slightly different. For instance, I would rephrase the question, decomposing it into two parts: first, there is the issue of &lt;em&gt;how&lt;/em&gt; one would go about classifying a stream of unknown information by data type and, second, what kind misclassification rate can be expected. Both questions are immediately answerable, if someone wants to take the time to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first issue amounts to developing a rules-based expert system along the lines of what has been preliminarily proposed in the comments and answers on these threads using samples of real-world data that are readily available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, given that this real-world data contains known data types, benchmarking the classification error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A third step would involve iteratively updating the rules to more accurately reflect the information that is being passed through -- to the extent that this is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my mind, while tedious, such a project wouldn't involve a prohibitive amount of work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;---- Additional Comment ----&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good discussion of feature extraction as it relates to the data generating process popped up on CV here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/191935/what-does-the-process-that-generates-the-data-mean-and-how-does-feature-selec&quot;&gt;https://stats.stackexchange.com/questions/191935/what-does-the-process-that-generates-the-data-mean-and-how-does-feature-selec&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature extraction, selection and the DGP is an excellent point of departure for developing an expert system.&lt;/p&gt;&#xA;" OwnerUserId="9974" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-01-22T12:37:10.353" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9907" PostTypeId="1" CreationDate="2016-01-22T15:14:05.493" Score="5" ViewCount="1382" Body="&lt;p&gt;I have a labelled time-series dataset; for example, a sequential input length nn from kk sensors corresponds to class '1' (see below for n=50,k=2n=50,k=2). How can I correctly classify it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use RNN with infinite input length, the input layer will have 2 units (2 channels), but what about the output? Label '1' applies to the full sequence, not just a single input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, there are two related questions: 1) how do I design an output unit? It has to reflect the input of the full sequence, 2) how do I compute an error? It also needs to account for the error of the whole sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use ANN with a sliding window, the problem remains virtually the same - different classes can have comparable values in the window, so the output must correspond to the full sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, how do I match a time series to just 1 number?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, there is an $n \times k$ array labelled '1' (this can be for example readings off $n$ sensors, which match 1 class, like a gesture: 'snap'), and another $m \times j$ array labelled '0', where $n$ is the number of features/input neurons and $k \neq j$ is the length/size of the input data. If I use an RNN to feedforward the first array I do it $k$ times, obviously. But how do I label it? Do I use a unity vector length $k$, get an error each time and them average out to get the sample error and backprop? I don't think so, but not too sure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/i2Sh2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/i2Sh2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="1426" LastEditDate="2016-01-27T10:51:55.107" LastActivityDate="2016-01-27T10:51:55.107" Title="Correct output labelling for RNNs/ANNs with sequential data and single label" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9908" PostTypeId="2" ParentId="9894" CreationDate="2016-01-22T15:42:46.927" Score="2" Body="&lt;p&gt;Maybe you will start to have negative impact of performance by increasing the number of trees. But definitely, at some point, increasing the number of trees will not add much accuracy in your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual way is to perform a k-fold cross-validation for different number of trees (and any other combination of model's parameters) and choose the one with the best performance. If you have any question about cross-validation, check &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;. Instead of using simple accuracy, try to use other metrics that take into account the complexity of the model, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow&quot;&gt;AIC&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obs.: the final accuracy of the algorithm should be calculated using a separate dataset, not used in the cross-validation (usually called test set).&lt;/p&gt;&#xA;" OwnerUserId="15215" LastEditorUserId="15215" LastEditDate="2016-01-22T18:23:58.153" LastActivityDate="2016-01-22T18:23:58.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9909" PostTypeId="1" CreationDate="2016-01-22T17:56:30.493" Score="5" ViewCount="737" Body="&lt;p&gt;In this example I have a RBM with a visible layer and a hidden layer. The original data is &quot;data&quot;, the values of the hidden neurons is &quot;hid&quot;, the values of the visible neurons calculated from &quot;hid&quot; is &quot;vis&quot;, and then the value for the hidden neurons calculated from &quot;vis&quot;is &quot;hid2&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I calculate the positive_association and the negative_association by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pos_associations = T.dot(data.T, hid)&#xA;neg_associations = T.dot(vis.T, hid2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where T.dot represents the usual matrix calculation. I then update the weights using contrastive divergence (CD)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = w + (training_rate)*(pos_associations - neg_associations)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;like described in this &lt;a href=&quot;http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do get a good reconstruction (vis looks like data) and I can see features representing the weights. However, it takes a while and many epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead of using CD I simply calculate the quadratic differnce between data and vis&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost = T.sum((data - vis)**2).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I update the weights using the gradient of cost, I get a faster approximation of the data. So my question is, what is the advantage of using CD?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="8820" LastEditDate="2016-02-08T05:01:59.320" LastActivityDate="2017-09-17T19:55:38.970" Title="What are the advantages of contrastive divergence vs the gradient of the quadratic difference between the original data and the reconstructed data?" Tags="|neural-network|deep-learning|rbm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9911" PostTypeId="2" ParentId="9877" CreationDate="2016-01-22T19:17:40.530" Score="1" Body="&lt;p&gt;As &lt;a href=&quot;https://fr.wikipedia.org/wiki/Carriage_Return_Line_Feed&quot; rel=&quot;nofollow&quot;&gt;carriage return/line feed (CR/LF)&lt;/a&gt; might cause  issues depending on the system, I suggest to store the matrix &quot;inline&quot; or in &lt;a href=&quot;https://en.wikipedia.org/wiki/Raster_graphics&quot; rel=&quot;nofollow&quot;&gt;raster-style&lt;/a&gt; with a short header to specify your convention, a versioning number (you might change your mind later, and increase the version), etc. and &lt;strong&gt;at least&lt;/strong&gt; the matrix size, and the rest in some text format (CSV,TSV). This is similar to what is done in the &lt;a href=&quot;http://netpbm.sourceforge.net/doc/pgm.html&quot; rel=&quot;nofollow&quot;&gt;portable gray map&lt;/a&gt; or bitmap image format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used that to store text &lt;a href=&quot;http://dx.doi.org/10.1109/TSP.2009.2023947&quot; rel=&quot;nofollow&quot;&gt;filter bank coefficients&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A minimum example  could be:&#xA;&lt;code&gt;2,3,0.1,1.2,2.3,3.4,4.5,5.6&lt;/code&gt; for the $2\times 3$ matrix:&#xA;\begin{array}&#xA;0.1&amp;amp;1.2&amp;amp;2.3\\3.4&amp;amp;4.5&amp;amp;5.6&#xA;\end{array}&#xA;but you can use for instance &lt;code&gt;#2,#3,0.1,1.2,2.3,3.4,4.5,5.6&lt;/code&gt;&#xA;so that aliens (think about the &lt;a href=&quot;https://en.wikipedia.org/wiki/Pioneer_plaque&quot; rel=&quot;nofollow&quot;&gt;golden Pioneer plaque&lt;/a&gt;) understand that the  first two integers are &quot;different&quot; and  provide hints on how the following numbers should be read. With a square matrix (typical of symmetrical matrices), this is even more interesting, as you only need one header number &lt;code&gt;#n&lt;/code&gt; (the side) and the acute readers will see that the remaining numbers are in $n^2$ quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also have a look at other &lt;a href=&quot;http://www.netlib.org/lapack/lug/node121.html&quot; rel=&quot;nofollow&quot;&gt;Matrix Storage Schemes&lt;/a&gt;, and if your matrix is sparse, &lt;a href=&quot;http://netlib.org/linalg/html_templates/node91.html&quot; rel=&quot;nofollow&quot;&gt;Compressed Row Storage (CRS)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2016-01-22T19:17:40.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9912" PostTypeId="2" ParentId="9909" CreationDate="2016-01-22T23:15:09.463" Score="5" Body="&lt;p&gt;It is natural that RBM requires some time to converge, but if you see that the reconstruction is decent, then your algorithm works well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand the idea of optimizing the model by your objective function:&#xA;&lt;code&gt;cost = T.sum((data - vis)**2).mean()&lt;/code&gt;.&#xA;I think you just cannot train RBM (or any other stochastic model) that way because vis is a nonlinear function of the data. If you omit sampling of the hidden vector you might use this formula and end up having model similar to the feed forward network that learns an identity transformation. It fact it can be used as a feature extractor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also notice that your second network no longer will be a proper generative model, because its learning rule doesn't follow the gradient of RBM's cost function. Hence it won't properly model either an energy function or a joint probability P(v, h) of your visible and hidden vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also be interested in reading &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Although it is convenient, the reconstruction error is actually a very poor measure of the progress of learning.  It is not the function that CD learning is approximately optimizing, [...] and it systematically confounds two different quantities that are changing during the learning.  The&#xA;  frst  is  the  difference  between  the  empirical  distribution  of  the  training  data  and  the  equilibrium&#xA;  distribution of the RBM. The second is the mixing rate of the alternating Gibbs Markov chain.  If&#xA;  the mixing rate is very low, the reconstruction error will be very small even when the distributions of&#xA;  the data and the model are very different.  As the weights increase the mixing rate falls, so decreases&#xA;  in reconstruction error do not necessarily mean that the model is improving and, conversely, small&#xA;  increases do not necessarily mean the model is getting worse.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So what is the cost function that RBM tries to optimize? &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow&quot;&gt;good description can be found here&lt;/a&gt;&#xA;$$-\frac{\partial log(p(x))}{\partial \theta }=\frac{\partial F(x)}{\partial \theta}-\sum_{\tilde{x}}p(\tilde{x})\frac{\partial F(\tilde{x})}{\partial \theta}$$&#xA;On the left hand side we have minus log probability of the data which we must minimize in order to increase the probability of the data. On the right hand side we have two terms. The function $F(x)$ coresponds to the free energy of the network given the data $x$ and current weights $\theta$ so minimization of this function increases the probability of our data. On the other hand we must take into account the normalizing factor $Z$- its role is to give correct probability distribution of the input vectors and this sum in the equation corresponds to the normalizing factor. In other words we must maximize the free energy (decrease probability) over all set of inputs proportionally to its current probability $p(\tilde{x})$. If we omit to maximize this sum our model won't converge to the optimal solution. Contrastive Divergence is only crude approximation of this objective funtion but it works very well in practice. I recommend that you watch &lt;a href=&quot;https://www.youtube.com/watch?v=AyzOUbkUf3M&quot; rel=&quot;nofollow&quot;&gt;this film by Geoffrey Hinton&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15656" LastEditorUserId="15656" LastEditDate="2016-01-23T17:45:46.987" LastActivityDate="2016-01-23T17:45:46.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9913" PostTypeId="1" CreationDate="2016-01-23T05:44:51.323" Score="3" ViewCount="10796" Body="&lt;p&gt;Here is my idea and my early work.    &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;&lt;em&gt;My target&lt;/em&gt;&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fetch 1-hour resolution air pollution data from China's goverment continuously.      &lt;/li&gt;&#xA;&lt;li&gt;The website's data which collected from the monitor sites over the country update per hour .      &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;&lt;em&gt;My Code&lt;/em&gt;&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Now, I can grab the useful information for a single hour. Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Input the website links for different pollution(co,no2,pm10, etc)&lt;/em&gt;     &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;html_co = urllib.urlopen(&quot;http://www.pm25.in/api/querys/co.json?city=beijing&amp;amp;token=5j1znBVAsnSf5xQyNQyq&quot;).read().decode('utf-8')&#xA;html_no2 = urllib.urlopen(&quot;http://www.pm25.in/api/querys/no2.json?city=beijing&amp;amp;token=5j1znBVAsnSf5xQyNQyq&quot;).read().decode('utf-8')&#xA;html_pm10 = urllib.urlopen(&quot;http://www.pm25.in/api/querys/pm10.json?city=beijing&amp;amp;token=5j1znBVAsnSf5xQyNQyq&quot;).read().decode('utf-8')&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Get the content of the html doc.&lt;/em&gt;      &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;soup_co = BeautifulSoup(html_co)&#xA;soup_no2 = BeautifulSoup(html_no2)&#xA;soup_pm10 = BeautifulSoup(html_pm10)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Extract the useful information from the whole content.&lt;/em&gt;        &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l = soup_co.p.get_text() &#xA;co= json.loads(l) &#xA;l = soup_no2.p.get_text() &#xA;no2= json.loads(l) &#xA;l = soup_pm10.p.get_text() &#xA;pm10= json.loads(l)       &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Tight the raw data into neat Pandas.Dataframe.&lt;/em&gt;          &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = {&quot;time&quot;:[],&quot;station&quot;:[],&quot;code&quot;:[],&quot;co&quot;:[],&quot;no2&quot;:[],&quot;pm10&quot;:[]}&#xA;for i in range(0,len(pm10)-1,1):&#xA;    ## 'station' is the monitor station's name in Chinese&#xA;    data[&quot;station&quot;].append(co[i][&quot;position_name&quot;])&#xA;    data[&quot;time&quot;].append(co[i][&quot;time_point&quot;])&#xA;    data[&quot;co&quot;].append(co[i][&quot;co&quot;])&#xA;    ## 'code' is the monitor station's index&#xA;    data[&quot;code&quot;].append(co[i][&quot;station_code&quot;])&#xA;    data[&quot;no2&quot;].append(no2[i][&quot;no2&quot;])&#xA;    data[&quot;pm10&quot;].append(pm10[i][&quot;pm10&quot;])&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;&lt;em&gt;My result&lt;/em&gt;&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Some pre-explanation&lt;/em&gt;      &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ignore the chinese character in the table. &lt;/li&gt;&#xA;&lt;li&gt;I only grab one city(Beijing here)'s data, and index from 0-11 notify that Beijing has 12 monitor site.         &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The columns 'co'/'NO2'/'PM10' represent the concentration of these air pollutants.      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ohyod.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ohyod.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;&lt;em&gt;My problem&lt;/em&gt;&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Now, I can grab the web data manually according my code above. But, I want to &lt;strong&gt;achieve the working flow below hourly  automatically&lt;/strong&gt;.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hour i      &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Execute the code     &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(1) Grab the data for Hour i's air pollutants data from website;     &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(2) Save the data into .csv based on the true date(like 20160101.csv)   &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After one hour.     &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Execute the code     &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(1) Grab the data for Hour i+1's air pollutants data from website;     &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(2) Save the data into .csv based on the true date.&lt;br&gt;&#xA;  if it's the same day liken to hour i --&gt; same .csv(like 2016-01-01.csv)&lt;br&gt;&#xA;  if the present day has past --&gt; creat a new .csv(like 2016-01-02.csv)      &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I havn't done these kind of stuff before. Can somebody offer me some advice?&lt;br&gt;&#xA;So, I can get an useful data scraping tool run in the background and I don't have to worry about it.      &lt;/p&gt;&#xA;" OwnerUserId="15663" LastEditorUserId="15620" LastEditDate="2017-04-20T09:20:45.257" LastActivityDate="2018-10-24T16:03:10.980" Title="Periodically executing a scraping script with Python" Tags="|python|dataset|scraping|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9914" PostTypeId="2" ParentId="9913" CreationDate="2016-01-23T06:08:11.307" Score="6" Body="&lt;p&gt;If you are on a Unix-based OS, use a scheduler &lt;a href=&quot;https://www.maketecheasier.com/cron-alternatives-linux/&quot; rel=&quot;nofollow&quot;&gt;like cron&lt;/a&gt; to run your script periodically. cron is the most common one, and you can find many tutorials for it. I guess this is the part you are unfamiliar with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it executes your script, make it check the time like so: &lt;code&gt;fname=datetime.datetime.now().strftime('%Y-%m-%d-%H.csv'&lt;/code&gt;). Open the appropriate file in append mode (&lt;code&gt;with open(fname, 'a') as csv_file&lt;/code&gt;) ; if it does not exist it will be created. That's all there's to it!&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-01-24T03:50:07.863" LastActivityDate="2016-01-24T03:50:07.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9915" PostTypeId="1" CreationDate="2016-01-23T06:20:17.123" Score="1" ViewCount="250" Body="&lt;p&gt;Say I have an fMRI experiment where I have a task that is suppose to measure response time, and the corresponding &quot;activation&quot; in the brain. I then have participants complete this task. Then I measure some variables through tests or assessments, and questionaries such as IQ, working memory, age, and years of education. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fMRI experiments, as I am aware typically voxels (the name for the 3d fmri representation of a pixel) that remain &quot;active&quot; after statistical tests, while the participant completes a task, say something about how that brain region where those voxels are modulates that cognitive task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what if I want to use the variables that I measured, IQ, age, working memory, and years of education in a multiple linear regression test, with the brain activation to this task as the response variable, or dependent variable. What does this tell me? &lt;/p&gt;&#xA;" OwnerUserId="10584" LastEditorUserId="10584" LastEditDate="2016-01-23T19:46:14.610" LastActivityDate="2017-01-19T14:22:23.590" Title="Multiple linear regression, fMRI" Tags="|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9916" PostTypeId="1" CreationDate="2016-01-23T08:04:13.130" Score="2" ViewCount="1761" Body="&lt;p&gt;I'm dealing with a regression prediction challenge where the evaluation metric is (pearson) correlation.&#xA;However, I have the impression that this metric is kind of arbitrary.&#xA;While I can keep the RMSE stable the correlation can have great variety.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please explain this metric and how to optimise for it?&lt;/p&gt;&#xA;" OwnerUserId="15664" LastEditorUserId="15664" LastEditDate="2016-01-23T08:11:23.093" LastActivityDate="2016-01-25T17:39:33.803" Title="Correlation as an evaluation metric for regression" Tags="|machine-learning|predictive-modeling|correlation|model-evaluations|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9917" PostTypeId="1" CreationDate="2016-01-23T08:11:03.673" Score="6" ViewCount="2037" Body="&lt;p&gt;I built a prediction model and predicted on new data. I now want to specify a value of my confidence in this predicted value, e.g. ranging from 0 to 1.&#xA;Three methods come to my mind&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Built 100 models on bootstrapped data, predict 100 times on each new observation, then calculate confidence intervals. Smaller intervals mean higher confidence. High computational effort.&lt;/li&gt;&#xA;&lt;li&gt;use oob predictions of a random forest from every tree&lt;/li&gt;&#xA;&lt;li&gt;Bayesian methods can give confidence intervals through the posterior&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Are there other/better ones?&lt;/p&gt;&#xA;" OwnerUserId="15664" LastActivityDate="2016-11-12T22:33:07.800" Title="How to measure confidence in prediction?" Tags="|machine-learning|predictive-modeling|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="9918" PostTypeId="1" AcceptedAnswerId="9922" CreationDate="2016-01-23T13:04:17.457" Score="5" ViewCount="3182" Body="&lt;p&gt;I am computing co-occurence matrix for a fixed windows size in python using scipy's lil_matrix for storing the counts and computing the counts by sliding the context window over each word and then counting in the window. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the code is taking too much time for relatively small corpus size also (100 MB Wikipedia dump). The code is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gen_coocur(window_size=5):&#xA;        '''&#xA;        Generates coocurrence matrix&#xA;        '''&#xA;        # vocab is precomputed. &#xA;        coocur_matrix = lil_matrix((len(vocab)+1, len(vocab)+1), dtype=np.float64)&#xA;        for page in self.wiki_extract.get_page():&#xA;            # word_tokenize is tokenizer from nltk &#xA;            doc_tokens = word_tokenize(page.decode('utf-8'))&#xA;            N = len(doc_tokens)&#xA;            for token in self.vocab:&#xA;                for i in xrange(0,window_size):&#xA;                    if (token in doc_tokens[0:i] or token in doc_tokens[i:(i+window_size+1)]) and token != doc_tokens[i]:&#xA;                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1&#xA;                for i in xrange(window_size, (N-window_size)):&#xA;                    if token in doc_tokens[(i-window_size):(i+window_size+1)] and token != doc_tokens[i]:&#xA;                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1&#xA;                for i in xrange(N-window_size, N):&#xA;                    if (token in doc_tokens[i:N] or token in doc_tokens[i-window_size:N]) and token != doc_tokens[i]:&#xA;                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1            &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;vocab is a dictionary which maps words -&gt; wordId. How can I optimize this code to run faster?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="13518" LastEditDate="2016-01-23T13:22:19.480" LastActivityDate="2016-08-20T22:33:35.367" Title="Optimizing co-occurrence matrix computation" Tags="|python|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9919" PostTypeId="2" ParentId="9907" CreationDate="2016-01-23T16:11:23.523" Score="4" Body="&lt;p&gt;Recurrent Neural Networks can learn very complex mappings between sequences (unlike Feed Forward Nets which are capable of transforming only fixed-sized vectors). You really only need a RNN which takes the sequence of the form $(a_t, b_t)$ as the input (assuming $t \in \{1...T\}$ you have seq. $a$ and $b$ both with $T$ samples and you have to classify these signals as one of $k$ classes). Then you can run the following procedure (pseudocode) for the sampling (given already trained model):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for t in (1..T)           //loop over an input sequence&#xA;    rnn.input[0] = a[t]   //assign t-th point of the blue seq.&#xA;    rnn.input[1] = b[t]   //assign t-th point of the green seq.&#xA;    rnn.forward()         //make forward pass&#xA;rnn.output.normalize()    //renormalize output probabilities&#xA;class_prob = rnn.output.max()&#xA;class      = rnn.output.argmax()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example we have the following sequence with length 4:&lt;br&gt;&#xA;signal A: {1, 3, 4, 6}&lt;br&gt;&#xA;signal B: {4, 3, 0, 1}&lt;br&gt;&#xA;Then in the 1st timestep we assign vector $(1, 4)$ to the input layer, then $(3, 3)$ and so on. At each timestep we propagate the input forward the network updating its hidden state which is responsible for remembering past datapoints. After every forward pass our neural network generates a prediction at the output layer which is temporarily uninmportant. This fluctuating prediction reflects the rnn's belief about the part of our dataset- it have seen only $t$ first input vectors. After we propagated entire signals we can do the classification.&lt;br&gt;&#xA;Your RNN's output layer should be $k$-dimensional and its $i$-th component is the pseudo-probability of the signal beeing in class $i$. For example you can do argmax (take the index with greatest prediction) or sample with a softmax distribution. For example if you have class $a$ and class $b$ and your output vector is $(5, 15)$ then your signal is classified as $b$. Or if you do softmax then $softmax(5, 15) = (0.000045, 0.999955)$ so you assign the class $b$ with nearly 100% probability.&lt;br&gt;&#xA;Training RNN is much more triky. I would suggest you taking some existing toolkit which have RNN such as LSTM already implemented. To train Recurrent NN we can apply the above procedure but at the end (after iterating over the sequence) we calculate Gradient of the loss function and do a backward pass. However this technique is called Backpropagation through time (it is really similar to ordinary FFN backprop, but we must &quot;unfold&quot; out RNN in time to adjust recurrent connections). Also notice that if you choose to have softmax output units your loss function should be a Cross-Entropy loss function.&lt;br&gt;&#xA;But what is the expected output vector to compute gradient loss from? If your dataset contains $N$ (green-blue) sequences each of them is in $k_n$ class then their expected output vector is simply one-hot vector $(0...,1,...0)$ where $1$ is at $k_n$-th component only. For example if you have a sequence classified as $b$, then its output vector target is $(0,1)$. @EDIT: Suppose that as in a previous example your sequence have class $b$ so you expect to see the vector $(0,1)$. You take now the prediction of the net (say $(0.34, 0.57)$) and calculate the error function as normal f.e. squared error or cross-entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information you can learn from &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;karpathy&lt;/a&gt; and &lt;a href=&quot;https://www.youtube.com/watch?v=vShMxxqtDDs&quot; rel=&quot;nofollow&quot;&gt;hinton&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15656" LastEditorUserId="15656" LastEditDate="2016-01-26T22:19:55.147" LastActivityDate="2016-01-26T22:19:55.147" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="9920" PostTypeId="5" CreationDate="2016-01-23T17:39:45.100" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://orange.biolab.si/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt; is a free and open-source, component-based, data mining and machine learning software suite, featuring a visual programming front-end for explorative data analysis and visualization. It includes a comprehensive set of components for data preprocessing, feature scoring and filtering, modeling, model evaluation, and various exploration techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is maintained by the &lt;a href=&quot;https://fri.uni-lj.si/en/laboratory/biolab&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bioinformatics Laboratory of the Faculty of Computer and Information Science&lt;/a&gt;, &lt;a href=&quot;https://www.uni-lj.si/eng/&quot; rel=&quot;nofollow noreferrer&quot;&gt;University of Ljubljana&lt;/a&gt;, &lt;a href=&quot;https://www.slovenia.info/?lng=2&quot; rel=&quot;nofollow noreferrer&quot;&gt;Slovenia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="29575" LastEditDate="2019-06-09T13:41:49.470" LastActivityDate="2019-06-09T13:41:49.470" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9921" PostTypeId="4" CreationDate="2016-01-23T17:39:45.100" Score="0" Body="Use for questions about Orange, the free, open-source, component-based, data mining and machine learning software suite." OwnerUserId="15527" LastEditorUserId="29575" LastEditDate="2019-06-09T13:41:49.470" LastActivityDate="2019-06-09T13:41:49.470" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9922" PostTypeId="2" ParentId="9918" CreationDate="2016-01-23T19:15:14.310" Score="2" Body="&lt;p&gt;From easiest to hardest:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Try running it in &lt;a href=&quot;http://pypy.org&quot; rel=&quot;nofollow&quot;&gt;pypy&lt;/a&gt; or &lt;a href=&quot;http://numba.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;numba&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/search?q=co-occurrence&amp;amp;type=Repositories&quot; rel=&quot;nofollow&quot;&gt;Find a faster implementation&lt;/a&gt;. Unfortunately I can not recommend one.&lt;/li&gt;&#xA;&lt;li&gt;Parallelize the loop over the documents. Not so hard since your vocabulary is precomputed. (Even if it weren't you could get away with it using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_hashing#Feature_vectorization_using_the_hashing_trick&quot; rel=&quot;nofollow&quot;&gt;hashing trick&lt;/a&gt;.) Combine this with the first bullet.&lt;/li&gt;&#xA;&lt;li&gt;Rewrite the &lt;a href=&quot;http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#putting-the-inner-loop-in-cython&quot; rel=&quot;nofollow&quot;&gt;inner loop&lt;/a&gt; in &lt;a href=&quot;http://docs.cython.org/src/userguide/numpy_tutorial.html&quot; rel=&quot;nofollow&quot;&gt;Cython&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Rewrite the whole thing in a faster language like C++ or Scala.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-23T19:15:14.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9923" PostTypeId="1" CreationDate="2016-01-23T23:07:16.237" Score="1" ViewCount="29" Body="&lt;p&gt;Is there a way to directly calculate an approximate quad weighted kappa measure from an OOB estimate, obtained from a gradient boosting model with subsampling without going through cross validation?&lt;/p&gt;&#xA;" OwnerUserId="15445" LastActivityDate="2016-01-23T23:07:16.237" Title="Convert Out-of-bag (OOB) estimate to quad weighted kappa score" Tags="|cross-validation|gbm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9924" PostTypeId="1" CreationDate="2016-01-24T01:12:43.523" Score="2" ViewCount="1361" Body="&lt;p&gt;I need to design and implement my 3rd year project which is a real-time alert system for parents to keep their kids safe with their smartphones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is time-based and unsupervised, which means I don't have score for any events, it contains events from : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;mobile browsing ( http requests )&lt;/li&gt;&#xA;&lt;li&gt;gps coordinates&lt;/li&gt;&#xA;&lt;li&gt;calls ( outgoing &amp;amp; incoming ) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Should I choose cluster based / density based approach ? &#xA;How do I find rare &amp;amp; suspicious events that parents should be notified for with the help of machine learning ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to present the results visually ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is a week of data enough for training ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any good open sources in Java to detect anomaly ?&lt;/p&gt;&#xA;" OwnerUserId="15686" LastActivityDate="2016-01-24T04:42:16.733" Title="anomaly detection alert system" Tags="|java|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9925" PostTypeId="2" ParentId="9913" CreationDate="2016-01-24T04:24:00.450" Score="6" Body="&lt;p&gt;As Emre already put it nicely, you can use cron for the job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I would suggest you to use any one of these:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Luigi&lt;/li&gt;&#xA;&lt;li&gt;Airflow&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are dependency management, scheduler libraries for Python, which are very easy to use and much better than vanilla cron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as I trigger hundreds of Airflow scripts everyday in production, I can as well vouch for it's ease of use and usefulness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this is a screenshot from my computer, of a very basic Airflow script running:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PZHP2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PZHP2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, here the tasks are dependent on each other. For example, the second task is dependant on the first one. So, if the first task fails, the second automatically fails, so that you don't need to fire-fight with the code/database everytime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you need the analytics task &lt;strong&gt;depends&lt;/strong&gt; on the scripting task, it would make sure both occur as they are intended to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, you can do this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Set the Airflow script to run every hour automatically for you&lt;/li&gt;&#xA;&lt;li&gt;Trigger emails whenever an error occurs (which is very helpful when the task runs when you aren't monitoring them)&lt;/li&gt;&#xA;&lt;li&gt;Neat visualization of the failed, running and completed tasks.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Helpful reading:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://softwarerecs.stackexchange.com/a/24885/18022&quot;&gt;My answer for DAG based script execution scheduler/planner with GUI; similar to airbnb's airflow or BODS or Informatica&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:04.587" LastActivityDate="2016-01-24T04:46:34.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9926" PostTypeId="2" ParentId="9924" CreationDate="2016-01-24T04:42:16.733" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I choose cluster based / density based approach?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes,&lt;/strong&gt; but depends on how good the algorithm is, and how effectively is the algorithm identifying important clusters. There is &lt;a href=&quot;http://www.cs.stir.ac.uk/courses/msc/projects/PastDissertations/2012-2013/Dissertations/Elangovan.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a lot of literature&lt;/a&gt; where people have taken the clustering approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to go with the clustering approach, I would go with the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;hierarchical clustering technique&lt;/a&gt;, cause it would help me re-check my clusters and build hierarchies in them so that the seriousness of the fraud can also be explored.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do I find rare &amp;amp; suspicious events that parents should be notified&#xA;  for with the help of machine learning ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would point out &lt;a href=&quot;https://datascience.stackexchange.com/a/8100/11097&quot;&gt;an answer&lt;/a&gt; which I have written for a very similar question, which also explores the aspect of fraud detection and anomaly analysis. And as I have said in the answer, the genetic algorithm still goes strong when it comes to fraud detection, owing to it's neat method of fitness calculation of the data points.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Any suggestions on how to present the results visually ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally such charts are presented through pie charts and network graphs. you can also use infographics if you have a nice idea about design and aesthetics.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is a week of data enough for training ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally, data science teams at Paypal, Palantir, etc churn years of data for sensible insights. As you say it's a class project, data of several months should do good. &lt;strong&gt;You wouldn't really get anything suspicious or sensible from a week of data.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any good open sources in Java to detect anomaly ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, there are some:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jgap.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Genetic algorithms library&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jubat.us/en/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Distributed Online Machine Learning Framework in java&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-01-24T04:42:16.733" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9927" PostTypeId="2" ParentId="8117" CreationDate="2016-01-24T06:36:04.600" Score="2" Body="&lt;p&gt;We can look at the source for guidance.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How does word2vec handle the input word being in the context?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is skipped; for both the &lt;a href=&quot;https://github.com/piskvorky/gensim/blob/f267abf94e84484047fb7569ebacba5626bc8391/gensim/models/word2vec.py#L129&quot; rel=&quot;nofollow&quot;&gt;skip-gram&lt;/a&gt; and &lt;a href=&quot;https://github.com/piskvorky/gensim/blob/f267abf94e84484047fb7569ebacba5626bc8391/gensim/models/word2vec.py#L154&quot; rel=&quot;nofollow&quot;&gt;CBOW&lt;/a&gt; models.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If word2vec encounters the same word multiple times in the same window, what occurs?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/piskvorky/gensim/blob/f267abf94e84484047fb7569ebacba5626bc8391/gensim/models/word2vec.py#L131&quot; rel=&quot;nofollow&quot;&gt;The relationship is strengthened&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-24T06:36:04.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9930" PostTypeId="1" AcceptedAnswerId="9932" CreationDate="2016-01-24T16:49:11.727" Score="8" ViewCount="8402" Body="&lt;p&gt;A recent paper by He et al. (&lt;a href=&quot;http://arxiv.org/pdf/1512.03385v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deep Residual Learning for Image Recognition&lt;/a&gt;, Microsoft Research, 2015) claims that they use up to 4096 layers (not neurons!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to understand the paper, but I stumble about the word &quot;residual&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could somebody please give me an explanation / definition what residual means in this case?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Examples&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We explicitly reformulate the layers as learning&#xA;  residual functions with reference to the layer inputs, instead&#xA;  of learning unreferenced functions.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Instead&#xA;  of hoping each few stacked layers directly fit a&#xA;  desired underlying mapping, we explicitly let these layers&#xA;  fit a residual mapping. Formally, denoting the desired&#xA;  underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear&#xA;  layers fit another mapping of $\mathcal{F}(x) := \mathcal{H}(x)−x$. The original&#xA;  mapping is recast into $\mathcal{F}(x)+x$. We hypothesize that it&#xA;  is easier to optimize the residual mapping than to optimize&#xA;  the original, unreferenced mapping&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-01-24T18:58:14.580" Title="What is a &quot;residual mapping&quot;?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9932" PostTypeId="2" ParentId="9930" CreationDate="2016-01-24T18:58:14.580" Score="3" Body="&lt;p&gt;It's $F(x)$; the difference between the mapping $H(x)$ and its input $x$. It's a &lt;a href=&quot;https://en.wikipedia.org/wiki/Residual_%28numerical_analysis%29&quot; rel=&quot;nofollow&quot;&gt;common term in mathematics&lt;/a&gt; (&lt;a href=&quot;https://de.wikipedia.org/wiki/Residuum_%28Numerische_Mathematik%29&quot; rel=&quot;nofollow&quot;&gt;DE&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-24T18:58:14.580" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9933" PostTypeId="2" ParentId="8883" CreationDate="2016-01-24T19:02:29.723" Score="6" Body="&lt;p&gt;Use the extra features for unsupervised learning. You might enjoy Vladimir Vapnik's take on this in the context of SVMs, which he calls privileged learning: &lt;a href=&quot;https://www.microsoft.com/en-us/research/video/invited-talk-learning-with-intelligent-teacher-similarity-control-and-knowledge-transfer/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="24968" LastEditDate="2018-09-05T20:25:08.817" LastActivityDate="2018-09-05T20:25:08.817" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9935" PostTypeId="2" ParentId="9916" CreationDate="2016-01-24T19:36:38.207" Score="1" Body="&lt;p&gt;The organizers might deem the &lt;em&gt;direction&lt;/em&gt; of predicted change more important than the &lt;em&gt;magnitude&lt;/em&gt;, i.e., it is more important that your prediction is high when the known value is high (and vice versa) than to get as close to the known value as possible. The measurements might be noisy anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One, fairly robust way of optimizing for it would be by &lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperparameter_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;grid-searching&lt;/a&gt; for the local optimum, like in &lt;a href=&quot;https://datascience.stackexchange.com/questions/8064/how-to-use-cohens-kappa-as-the-evaluation-metric-in-gridsearchcv-in-scikit-lear&quot;&gt;this QA&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you should also take note that algorithms tweak ther internal parameters when fitting according to some &lt;a href=&quot;https://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;loss function&lt;/a&gt;. Some algorithms &lt;a href=&quot;https://www.google.com/search?num=20&amp;amp;q=custom+(loss+OR+cost)+function&quot; rel=&quot;nofollow noreferrer&quot;&gt;accept custom cost functions and derivatives&lt;/a&gt;, yet &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;some implementations&lt;/a&gt; &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;don't&lt;/a&gt;. &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_gain_in_decision_trees&quot; rel=&quot;nofollow noreferrer&quot;&gt;Information-theoretic measureas&lt;/a&gt; are standard in classification while MSE in regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretically, you should be able to tell your Random Forest (&lt;a href=&quot;http://www.scipy-lectures.org/advanced/mathematical_optimization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;or another algorithm accordingly&lt;/a&gt;) what the optimal split is as a function of &lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;q=pearson+correlation+derivative&quot; rel=&quot;nofollow noreferrer&quot;&gt;pearson correlation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-01-25T17:39:33.803" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9938" PostTypeId="1" CreationDate="2016-01-25T10:33:46.983" Score="3" ViewCount="375" Body="&lt;p&gt;I'd like to define our app user churn,  Generally, people will define their app user churn by some simply way, like: if the user do not logon continually 7 days or 1 month, we will define them churn user. So is there a better way?&lt;/p&gt;&#xA;" OwnerUserId="15715" LastEditorUserId="13727" LastEditDate="2016-01-26T08:32:08.400" LastActivityDate="2016-01-26T08:32:08.400" Title="How to define user churn" Tags="|bigdata|churn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9939" PostTypeId="1" CreationDate="2016-01-25T11:03:17.520" Score="2" ViewCount="1330" Body="&lt;p&gt;I am trying to come up with a formula or machine learning algorithm using which I can approximately predict the weekly or monthly users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What to keep in mind is that I already have counts for the &lt;strong&gt;unique visitors per day&lt;/strong&gt; for the week/months that I would like to make a near accurate prediction. Here, simply summing the daily unique users would not work, as they can be unique on one day but not on two days as they can have a session lasting over 2 days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method is to serve as an alternative to running a Spark job on the whole week/month data in order to save time and resources - Is this possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at Time Series and Linear Regression, but need more clarification on the possible approaches and also on any work-arounds?&lt;/p&gt;&#xA;" OwnerUserId="15638" LastActivityDate="2016-04-07T18:22:01.810" Title="How to predict an approximate weekly/monthly number, when the Unique Daily Visitors for that week/month are already known" Tags="|machine-learning|predictive-modeling|regression|time-series|apache-spark|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9940" PostTypeId="2" ParentId="9938" CreationDate="2016-01-25T11:51:20.740" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Churn_rate&quot; rel=&quot;nofollow&quot;&gt;churn rate&lt;/a&gt; is users the ratio of active users that stop being active after a given period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The churn rate for application user is exactly the one used for other cases, like employees. &#xA;However, when dealing with employees it is clear whether the employee is active,&#xA;The problem with application users is the identification that they are no longer active.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is to identify a period of in activity after which, the vast majority of these user never return.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, take the users that were active during January of the previous year and were not active for 7 days since then. If 99% of them were not active for the rest of the year than 7 days are a good indication of being in active.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The suitable period is usually determined by the domain (e.g., some applications are used in periods longer than 7 days) and the required confidence level (99%, 90%).&#xA;Note that you don't want to be too safe because if you'll decide that a user is inactive only after 2 years, this definition will prevent you from identifying users that are practicality lost and take actions to reduce the churn rate.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-01-25T11:51:20.740" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9941" PostTypeId="1" CreationDate="2016-01-25T12:18:00.553" Score="4" ViewCount="2626" Body="&lt;p&gt;I have the following problem that I don't know how to solve:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the data for different market baskets with a corresponding class. So for example I know:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Student - {beer, milk, water}&lt;/li&gt;&#xA;&lt;li&gt;Professional -  {nuts, pizza, bananas}&lt;/li&gt;&#xA;&lt;li&gt;Student - {oranges, tomatoes, beer}&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there a method to create a classification model so that I can use the content of the market basket in order to determine the corresponding class (Student, Professional, ...)? Thank you!&lt;/p&gt;&#xA;" OwnerUserId="15720" LastEditorUserId="13727" LastEditDate="2016-01-28T10:45:32.283" LastActivityDate="2016-01-28T10:45:32.283" Title="Perform classification on market basket analysis" Tags="|machine-learning|classification|methods|market-basket-analysis|association-rules|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9942" PostTypeId="2" ParentId="9562" CreationDate="2016-01-25T12:23:46.063" Score="0" Body="&lt;p&gt;Using &lt;a href=&quot;http://orgmode.org/worg/org-contrib/babel/&quot; rel=&quot;nofollow&quot;&gt;org-babel&lt;/a&gt; could be a solution. In a single file you can have executable code blocks containing different languages. The caveat is that you have to use &lt;a href=&quot;https://www.gnu.org/software/emacs/&quot; rel=&quot;nofollow&quot;&gt;emacs&lt;/a&gt; and &lt;a href=&quot;http://orgmode.org/&quot; rel=&quot;nofollow&quot;&gt;org-mode&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have an example of Python and R together, but I did use Python and Bash for instance in the same document. Note that, I assume you won't be able to share results generated by Python inside R block and vice versa.&lt;/p&gt;&#xA;" OwnerUserId="3591" LastActivityDate="2016-01-25T12:23:46.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9943" PostTypeId="1" AcceptedAnswerId="10206" CreationDate="2016-01-25T13:57:24.367" Score="2" ViewCount="9513" Body="&lt;p&gt;My ultimate goal is to use Jupyter together with Python for data analysis using Spark. The current hurdle I face is loading the external &lt;code&gt;spark_csv&lt;/code&gt; library. I am using Mac OS and Anaconda as the Python distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark import SparkContext&#xA;sc = SparkContext('local', 'pyspark')&#xA;sqlContext = SQLContext(sc)&#xA;df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')&#xA;df.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;when invoked from Jupyter yields: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Py4JJavaError: An error occurred while calling o22.load.&#xA;: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are more details:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Setting Spark together with Jupyter&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I managed to set up Spark/PySpark in Jupyter/IPython (using Python 3.x). &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;System initial setting&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On my OS X I installed Python using Anaconda. The default version of Python I have currently installed is 3.4.4 (Anaconda 2.4.0). Note, that I also have installed also 2.x version of Python using &lt;code&gt;conda create -n python2 python=2.7&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Installing Spark&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is actually the simplest step; download the latest binaries into &lt;code&gt;~/Applications&lt;/code&gt; or some other directory of your choice. Next, untar the archive &lt;code&gt;tar -xzf spark-X.Y.Z-bin-hadoopX.Y.tgz&lt;/code&gt;.&#xA;For easy access to Spark create a symbolic link to the Spark:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ln -s ~/Applications/spark-X.Y.Z-bin-hadoopX.Y ~/Applications/spark&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, add the Spark symbolic link to the PATH:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;export SPARK_HOME=~/Applications/spark&#xA;export PATH=$SPARK_HOME/bin:$PATH&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can now run Spark/PySpark locally: simply invoke &lt;code&gt;spark-shell&lt;/code&gt; or &lt;code&gt;pyspark&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Setting Jupyter&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In order to use Spark from within a Jupyter notebook, prepand the following to &lt;code&gt;PYTHONPATH&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;export PYTHONPATH=$SPARKHOME/python/lib/py4j-0.8.2.1-src.zip:$SPARKHOME/python/:$PYTHONPATH&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Further details can be found &lt;a href=&quot;https://github.com/drorata/Jupyter-Spark-Python-setup&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3591" LastEditorUserId="21" LastEditDate="2016-05-10T13:57:56.713" LastActivityDate="2016-05-10T13:57:56.713" Title="Use spark_csv inside Jupyter and using Python" Tags="|python|apache-spark|pyspark|jupyter|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9944" PostTypeId="2" ParentId="9939" CreationDate="2016-01-25T13:58:56.627" Score="1" Body="&lt;p&gt;To calculate the (estimated) value of the monthly/weekly unique visitors based on the per day count you may use the &lt;a href=&quot;https://en.wikipedia.org/wiki/HyperLogLog&quot; rel=&quot;nofollow&quot;&gt;HyperLogLog&lt;/a&gt; algorihm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give you the historical values and you can use any time series approach to predict future counts&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-01-25T13:58:56.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9945" PostTypeId="1" CreationDate="2016-01-25T15:57:55.647" Score="16" ViewCount="10571" Body="&lt;p&gt;Are there any machine learning packages for R that can make use of the GPU to improve training speed (something like theano from the python world)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see that there is a package called gputools which allows execution of code on the gpu, but I'm looking for a more complete library for machine learning.&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="31513" LastEditDate="2017-05-28T02:19:44.437" LastActivityDate="2021-03-11T20:15:33.507" Title="R: machine learning on GPU" Tags="|machine-learning|r|gpu|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9947" PostTypeId="2" ParentId="9941" CreationDate="2016-01-25T16:30:28.977" Score="1" Body="&lt;p&gt;You are looking into s classification task where text in the left (like Student etc.) is the target class, while list of shoppings is an almost ready-to-go list of features - variables on which you will be learning a classification model.&#xA;While you should looking into classification algorithm family in general, really most basic algorithms you may use are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;Naive Bayes classifier&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow&quot;&gt;Support Vector Machines classifier&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Implementation of the first is so damn simple, that you may code it in mostly any language straight away to just blow the dust of coding skills, while more sophisticated algorithms usually have a number of implementations, which you should be looking based on your language of choice.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-01-25T16:30:28.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9948" PostTypeId="1" AcceptedAnswerId="9951" CreationDate="2016-01-25T17:33:24.343" Score="0" ViewCount="2721" Body="&lt;p&gt;I have a time series. Data points are available for each year from 1966 to 2000. Using R, I want to decompose this time series into trend, seasonal and random components. When I run the decompose command, I get the error &quot;time series has no or less than 2 periods&quot;. Since my data is annual I have specified a frequency of 1. What am I doing wrong? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the R code that I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat=c(37.2,37,37.4,37.5,37.7,37.7,37.4,37.2,37.3,37.2,36.9,36.7,36.7,36.5,&#xA;      36.3,35.9, 35.8,35.9,36,35.7,35.6, 35.2, 34.8, 35.3,35.6,35.6, 35.6,&#xA;      35.9,36,35.7, 35.7, 35.5, 35.6, 36.3, 36.5)&#xA;&#xA;whts &amp;lt;- ts(dat, frequency = 1, start=1966, end=2000)  &#xA;is.ts(whts)  &#xA;plot.ts(whts)  &#xA;whtimeseriescomponents &amp;lt;- decompose(whts)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15730" LastEditorUserId="11097" LastEditDate="2016-01-25T17:57:12.077" LastActivityDate="2019-07-04T15:01:56.623" Title="Decompose annual time series in R" Tags="|r|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9949" PostTypeId="1" AcceptedAnswerId="9952" CreationDate="2016-01-25T18:41:09.973" Score="2" ViewCount="977" Body="&lt;p&gt;I have a big .CSV database of 25k users with various attributes of the last user's activity and events during the past 6 weeks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an example of the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;username        (B)      (C)             (D)        (E)  &#xA;nicole          524      329             203        787&#xA;asteria         197      186             286        120 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to create a common behavior pattern based on the values of the attributes of each user and run an algorithm to find a common pattern that defines this group's behavior and to find out if there is any correlation in the dimensions values and which dimension define this list of users.&#xA;I am fully aware that correlation does not necessarily equal causation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the model for this problem ? &#xA;What kind of Algorithm is the best to deal with this situation?&#xA;What is the tools that you recommend to use of the project ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas would be great.&lt;/p&gt;&#xA;" OwnerUserId="15695" LastEditorUserId="11097" LastEditDate="2016-02-09T11:21:50.280" LastActivityDate="2016-02-09T11:21:50.280" Title="What model should I use to find a common pattern for a specific user group based on the other dimensions?" Tags="|machine-learning|r|data-mining|bigdata|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9950" PostTypeId="1" AcceptedAnswerId="9967" CreationDate="2016-01-25T18:41:24.083" Score="22" ViewCount="14604" Body="&lt;p&gt;In NLP, there is the concept of &lt;code&gt;Gazetteer&lt;/code&gt; which can be quite useful for creating annotations. As far as I understand: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A gazetteer consists of a set of lists containing names of entities&#xA;  such as cities, organisations, days of the week, etc. These lists are&#xA;  used to ﬁnd occurrences of these names in text, e.g. for the task of&#xA;  named entity recognition.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So it is essentially a lookup. Isn't this kind of a cheat? If we use a &lt;code&gt;Gazetteer&lt;/code&gt; for detecting named entities, then there is not much &lt;code&gt;Natural Language Processing&lt;/code&gt; going on. Ideally, I would want to detect named entities using &lt;code&gt;NLP&lt;/code&gt; techniques. Otherwise, how is it any better than a regex pattern matcher?&lt;/p&gt;&#xA;" OwnerUserId="15735" LastEditorUserId="29169" LastEditDate="2019-06-09T16:46:31.853" LastActivityDate="2019-06-09T16:46:31.853" Title="NLP - Is Gazetteer a cheat?" Tags="|nlp|named-entity-recognition|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="9951" PostTypeId="2" ParentId="9948" CreationDate="2016-01-25T18:43:14.507" Score="2" Body="&lt;p&gt;Seasonal decomposition doesn't make sense in this situation. You're sampling frequency needs to be greater than 1 for this to work!&#xA;I know this changes your model, but just for the sake of example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dat=c(37.2,37,37.4,37.5,37.7,37.7,37.4,37.2,37.3,37.2,36.9,36.7,36.7,36.5, 36.3,35.9, 35.8,35.9,36,35.7,35.6, 35.2, 34.8, 35.3,35.6,35.6, 35.6, 35.9,36,35.7, 35.7, 35.5, 35.6, 36.3, 36.5)&#xA;&amp;gt; whts &amp;lt;- ts(dat, frequency=2, start=1966, end=2000)&#xA;&amp;gt; decompose(whts)&#xA;$x&#xA;Time Series:&#xA;Start = c(1966, 1) &#xA;End = c(2000, 1) &#xA;Frequency = 2 &#xA; [1] 37.2 37.0 37.4 37.5 37.7 37.7 37.4 37.2 37.3 37.2 36.9 36.7 36.7 36.5 36.3 35.9 35.8 35.9 36.0 35.7 35.6&#xA;[22] 35.2 34.8 35.3 35.6 35.6 35.6 35.9 36.0 35.7 35.7 35.5 35.6 36.3 36.5 37.2 37.0 37.4 37.5 37.7 37.7 37.4&#xA;[43] 37.2 37.3 37.2 36.9 36.7 36.7 36.5 36.3 35.9 35.8 35.9 36.0 35.7 35.6 35.2 34.8 35.3 35.6 35.6 35.6 35.9&#xA;[64] 36.0 35.7 35.7 35.5 35.6 36.3&#xA;&#xA;$seasonal&#xA;Time Series:&#xA;Start = c(1966, 1) &#xA;End = c(2000, 1) &#xA;Frequency = 2 &#xA; [1] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA; [9] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[17] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[25] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[33] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[41] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[49] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[57] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266  0.007141266&#xA;[65] -0.007141266  0.007141266 -0.007141266  0.007141266 -0.007141266&#xA;&#xA;$trend&#xA;Time Series:&#xA;Start = c(1966, 1) &#xA;End = c(2000, 1) &#xA;Frequency = 2 &#xA; [1]     NA 37.150 37.325 37.525 37.650 37.625 37.425 37.275 37.250 37.150 36.925 36.750 36.650 36.500 36.250&#xA;[16] 35.975 35.850 35.900 35.900 35.750 35.525 35.200 35.025 35.250 35.525 35.600 35.675 35.850 35.900 35.775&#xA;[31] 35.650 35.575 35.750 36.175 36.625 36.975 37.150 37.325 37.525 37.650 37.625 37.425 37.275 37.250 37.150&#xA;[46] 36.925 36.750 36.650 36.500 36.250 35.975 35.850 35.900 35.900 35.750 35.525 35.200 35.025 35.250 35.525&#xA;[61] 35.600 35.675 35.850 35.900 35.775 35.650 35.575 35.750     NA&#xA;&#xA;$random&#xA;Time Series:&#xA;Start = c(1966, 1) &#xA;End = c(2000, 1) &#xA;Frequency = 2 &#xA; [1]           NA -0.157141266  0.082141266 -0.032141266  0.057141266  0.067858734 -0.017858734 -0.082141266&#xA; [9]  0.057141266  0.042858734 -0.017858734 -0.057141266  0.057141266 -0.007141266  0.057141266 -0.082141266&#xA;[17] -0.042858734 -0.007141266  0.107141266 -0.057141266  0.082141266 -0.007141266 -0.217858734  0.042858734&#xA;[25]  0.082141266 -0.007141266 -0.067858734  0.042858734  0.107141266 -0.082141266  0.057141266 -0.082141266&#xA;[33] -0.142858734  0.117858734 -0.117858734  0.217858734 -0.142858734  0.067858734 -0.017858734  0.042858734&#xA;[41]  0.082141266 -0.032141266 -0.067858734  0.042858734  0.057141266 -0.032141266 -0.042858734  0.042858734&#xA;[49]  0.007141266  0.042858734 -0.067858734 -0.057141266  0.007141266  0.092858734 -0.042858734  0.067858734&#xA;[57]  0.007141266 -0.232141266  0.057141266  0.067858734  0.007141266 -0.082141266  0.057141266  0.092858734&#xA;[65] -0.067858734  0.042858734 -0.067858734 -0.157141266           NA&#xA;&#xA;$figure&#xA;[1] -0.007141266  0.007141266&#xA;&#xA;$type&#xA;[1] &quot;additive&quot;&#xA;&#xA;attr(,&quot;class&quot;)&#xA;[1] &quot;decomposed.ts&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-01-25T18:43:14.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9952" PostTypeId="2" ParentId="9949" CreationDate="2016-01-25T19:37:47.280" Score="1" Body="&lt;p&gt;The most common approach is to create business rules handmade, based on the univariate and multivariate analysis of the variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, do some frequency count, see if you could isolate some subset of your data just looking at one or two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then when you have your labels, create a linear or so model with this new variable as output. For exemple, a &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;linear discriminant analysis&lt;/a&gt;. The analysis will supply you with new insights on your group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to rely on an algorithm, two solutions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you don't seems to have a lot of variables, a non-supervised segmentation could do the job. For exemple, a k-Nearest Neighbor or a decision tree are basic and good aproachs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a few more variables, I like is to do a &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;principal component analysis&lt;/a&gt; then a non-supervised classification to define your group on the result of the PCA.&#xA;Note that a PCA + handmade rules based on the analysis of your PCA result may be enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each time, in the end, a discriminant analysis and a profile of your groups to asses the quality of your results.&lt;/p&gt;&#xA;" OwnerUserId="15543" LastEditorUserId="15543" LastEditDate="2016-01-26T17:30:00.683" LastActivityDate="2016-01-26T17:30:00.683" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9953" PostTypeId="2" ParentId="9938" CreationDate="2016-01-25T20:11:41.210" Score="2" Body="&lt;p&gt;Two small amendments to the previous complete answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may classify the churn based on cause, this is typically &lt;em&gt;active&lt;/em&gt; and &lt;em&gt;passive&lt;/em&gt; churn. In your case this will be &lt;strong&gt;churn due to inactivity&lt;/strong&gt; and &lt;strong&gt;churn due to lock account by the administrator&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To handle the uncertainty of the inactivity churn, you may introduce a measure called &lt;strong&gt;dechurn&lt;/strong&gt; (probably not a stable term), measuring the reactivation of a churned user.   &lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-01-25T20:11:41.210" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9954" PostTypeId="1" CreationDate="2016-01-25T21:00:28.350" Score="2" ViewCount="174" Body="&lt;p&gt;Suppose you have a big number of polynomials such as &lt;code&gt;xyz+xy+z&lt;/code&gt; where &lt;code&gt;xyz&lt;/code&gt;, &lt;code&gt;xy&lt;/code&gt; and &lt;code&gt;z&lt;/code&gt; are monomials. You wanna visualize the polynomials together something such as atlas below and show how they are related graphically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Vs1KS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Vs1KS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example is from &lt;a href=&quot;https://github.com/networkx/networkx/tree/master/examples/drawing&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The edges could correspond to sum while the vertices could be monomials. It is also possible to do it so that each variable in monomial has their own vertex while two different kinds of edges are drawn: edges inside monomials and distinct edges to connect monomials of the same polynomial. Other visualizations are possible as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can you visualize a big number of polynomials graphically?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;P.s. Aspiring &lt;a href=&quot;http://chat.stackexchange.com/transcript/message/27092695#27092695&quot;&gt;discussion in Mathematica SE&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="15738" LastEditorUserId="31513" LastEditDate="2017-08-08T07:38:42.023" LastActivityDate="2017-08-08T07:38:42.023" Title="Visualizing a big number of polynomials as atlas of graphs?" Tags="|visualization|graphs|infographics|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9955" PostTypeId="1" CreationDate="2016-01-25T21:13:50.570" Score="1" ViewCount="62" Body="&lt;p&gt;I am new to machine learning and AI, and I am working on a problem in which I need to clean a table (database) of text/words. For example I should delete words like and, the, etc. , or replace words like COMP with company or substitute other acronyms with the suitable words. I wanted to find some resources on the techniques proposed for this purpose and to find what else I can do to better clean the table. Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="15739" LastActivityDate="2016-01-25T22:28:44.190" Title="Which machine learning/AI method is this problem related to?" Tags="|machine-learning|data-mining|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9957" PostTypeId="2" ParentId="9955" CreationDate="2016-01-25T22:28:44.190" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;For example I should delete words like and, the, etc. ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In natural language processing, this first task is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow&quot;&gt;stop word&lt;/a&gt; removal. You can identify them by looking at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;words' frequency over the documents&lt;/a&gt;; uninformative words appear very often.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-25T22:28:44.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9958" PostTypeId="1" AcceptedAnswerId="9960" CreationDate="2016-01-25T23:45:28.583" Score="2" ViewCount="134" Body="&lt;p&gt;&lt;a href=&quot;http://pastebin.com/K0eq8cyZ&quot; rel=&quot;nofollow&quot;&gt;http://pastebin.com/K0eq8cyZ&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I went through each season of &quot;It's Always Sunny in Philadelphia&quot; and determined the character groupings (D=Dennis, F=Frank, C=Charlie, M=Mac, B=Sweet Dee) for each episode. I also starred &quot;winners&quot; for some episodes. How best could I organize this data, in what type of database, and what data science tools would extract the most information out of it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of making an SQL table like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             (1)     (2)      (3)     (4)     (5)&#xA;Episode# | Dennis | Frank | Charlie | Mac | Sweet Dee &#xA;008      |    5   |  3,4  |  2,4    | 2,3 |    1&#xA;010      |    5   |  3,4,6|  2,4,6  |2,3,6|    1  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...where all the values are arrays of ints. 6 represents that the character won the episode and each number represents one of the 5 characters. Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2016-01-26T06:09:04.190" Title="What would be the best way to structure and mine this set of data?" Tags="|machine-learning|clustering|dataset|visualization|data-cleaning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9959" PostTypeId="1" AcceptedAnswerId="9970" CreationDate="2016-01-26T01:51:21.140" Score="1" ViewCount="944" Body="&lt;p&gt;When using an auto encoder to create non-linear dimensional reduced featires, is it more common to use the output of the network (the prediction of the input features) or to use the weights from the (or 1 of the if there are multiple) hidden layers? If the hidden layer is used, do you use the hidden layer activation as features or weights from the hidden layer to the output?&lt;/p&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2016-01-26T14:12:26.117" Title="Autoencoders for feature creation" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9960" PostTypeId="2" ParentId="9958" CreationDate="2016-01-26T02:02:48.413" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How best could I organize this data, in what type of database?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A simple relational database should do, but you could also use a &quot;fancy&quot; graph database if you want. One table for the users, and one for the &quot;interactions&quot;. Each interaction would have foreign key columns for the two participants, labeled winner and loser, and the number of the episode the interaction it occurred.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Also any ideas on the best way to visually represent this data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A graphical representation for &lt;a href=&quot;https://en.wikipedia.org/wiki/Social_network_analysis&quot; rel=&quot;nofollow&quot;&gt;social network analysis&lt;/a&gt; suggests itself. Here are &lt;a href=&quot;https://www.researchgate.net/publication/225765648_Social_network_analysis_in_a_movie_using_character-net&quot; rel=&quot;nofollow&quot;&gt;some&lt;/a&gt; &lt;a href=&quot;http://bi.snu.ac.kr/Publications/Conferences/International/ASONAM2015_CJNan.pdf&quot; rel=&quot;nofollow&quot;&gt;papers&lt;/a&gt; and a &lt;a href=&quot;https://www.reddit.com/r/sna/&quot; rel=&quot;nofollow&quot;&gt;subreddit&lt;/a&gt; for inspiration. In your case, there is a concept of competition with clear winners/losers, so you could make your graph directed. Have the characters be the nodes, and add &lt;em&gt;directed&lt;/em&gt; edges from the winning party to the losing party for each interaction. Collapse repeated interactions, etc. This approach would let you quickly identify overall winners and losers, as well as simply who interacts with whom. &lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-01-26T06:09:04.190" LastActivityDate="2016-01-26T06:09:04.190" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="9963" PostTypeId="1" CreationDate="2016-01-26T02:57:59.407" Score="2" ViewCount="69" Body="&lt;p&gt;I'm new to ML and trying to find some practical use to it I encountered with the chance of saving the connections and disconnections (the binary variable) of a bunch of users like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;User A connected at 10:02:33&lt;/p&gt;&#xA;&#xA;&lt;p&gt;User A discconected at 10:05:02&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering how could I determine whenever the user is going to connect again, analyzing his pasts connections, disconnections and time online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My concers are the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Would this be posible?&lt;/li&gt;&#xA;&lt;li&gt;What's the best method for this?&lt;/li&gt;&#xA;&lt;li&gt;How much samples per user would I need?&lt;/li&gt;&#xA;&lt;li&gt;What would be the best way to structure the data, and besides the user identifier, action(connect/disconnect) and time the action take place what other info would be useful?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;An just as an extra question: Would it be possible to add more data in real-time to improve the algorithm prediction? How?&lt;/p&gt;&#xA;" OwnerUserId="15748" LastActivityDate="2016-01-26T06:13:10.360" Title="Predicting or patron find of a binary variable over time" Tags="|machine-learning|dataset|predictive-modeling|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9965" PostTypeId="2" ParentId="9963" CreationDate="2016-01-26T06:01:48.653" Score="1" Body="&lt;p&gt;Since you say you are a beginner, I suggest using a simple binary classifier: logistic regression. It takes the input vector, and transforms it by taking its dot product with a weight parameter (estimating which is the goal) before passing it through the logistic function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat y(x) = \dfrac{1}{1+\exp(-\left&amp;lt; w, x \right&amp;gt;)}$ and $y=\begin{cases}1, \text{user present} \\ 0, \text{user absent} \end{cases}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For details read &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;the wikipedia article&lt;/a&gt;. For features, I would use the time of day encoded as $\left( \sin (2\pi t/T), \cos (2\pi t/T) \right)$, a hot-encoded categorical variable for the day of the week, a binary variable to indicate weekends, and so on. It's a crude model that leaves a lot of data on table, such as correlations between users, but as a beginner's exercise it is appropriate.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-01-26T06:13:10.360" LastActivityDate="2016-01-26T06:13:10.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9966" PostTypeId="1" CreationDate="2016-01-26T09:00:35.757" Score="3" ViewCount="591" Body="&lt;p&gt;I have a sample dataset containing around 77k of rows and I have clustered them by using k mean clustering and colored the clusters using color manager. Now i would like to view them, However I could not set the No. of rows to display as 77k in Scatter Plot and Is there any way to visualize the clusters in Knime? &lt;/p&gt;&#xA;" OwnerUserId="13034" LastActivityDate="2019-12-10T08:01:50.447" Title="Knime : Scatter Plot" Tags="|data-mining|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9967" PostTypeId="2" ParentId="9950" CreationDate="2016-01-26T09:31:52.940" Score="16" Body="&lt;p&gt;Gazetteer or any other option of intentionally fixed size feature seems a very popular approach in &lt;em&gt;academic&lt;/em&gt; papers, when you have a problem of finite size, for example NER in a fixed corpora, or POS tagging or anything else. I would not consider it cheating unless the only feature you will be using is Gazetteer matching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when you train any kind of NLP model, which does rely on dictionary while training, you may get real world performance way lower than your initial testing would report, unless you can include all objects of interest into the gazetteer (and why then you need that model?) because  your trained model will rely on the feature at some point and, in a case when other features will be too weak or not descriptive, new objects of interest would not be recognized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do use a Gazetteer in your models, you should make sure, that that feature has a counter feature to let model balance itself, so that simple dictionary match won't be the only feature of positive class (and more importantly, gazetteer should match not only positive examples, but also negative ones).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, assume you do have a full set of infinite variations of all person names, which makes general person NER irrelevant, but now you try to decide whether the object mentioned in text is capable of singing. You will rely on features of inclusion into your Person gazetteer, which will give you a lot of false positives; then, you will add a verb-centric feature of &quot;&lt;strong&gt;Is Subject of verb sing&lt;/strong&gt;&quot;, and that would probably give you false positives from all kind of objects like birds, your tummy when you're hungry and a drunk fellow who &lt;em&gt;thinks&lt;/em&gt; he can sing (but let's be honest, he can not) -- but that verb-centric feature will balance with your person gazetteer to assign positive class of 'Singer' to persons and not animals or any other objects. Though, it doesn't solve the case of drunk performer.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-01-26T09:31:52.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9968" PostTypeId="1" CreationDate="2016-01-26T11:25:57.627" Score="2" ViewCount="116" Body="&lt;p&gt;Data Scientist positions are not the most common entry level jobs. They require hard work, knowledge and most of all experience. The problem is, I am uncertain what entry level jobs can help you get experience in data science. I apologise for not asking a technical question, but I have been searching an answer and most importantly advice from experts and data science practitioners. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am studying to become a Data Scientist and I was wondering what are good entry level jobs that can help me build a good foundation for future success in the field?&lt;/p&gt;&#xA;" OwnerUserId="15756" LastEditorUserId="13023" LastEditDate="2016-01-30T20:07:17.467" LastActivityDate="2016-01-30T20:07:17.467" Title="Jobs for gaining experience in preperation for data science" Tags="|beginner|career|" AnswerCount="0" CommentCount="0" ClosedDate="2016-01-26T13:09:27.117" ContentLicense="CC BY-SA 3.0" />
  <row Id="9969" PostTypeId="1" CreationDate="2016-01-26T12:34:54.873" Score="5" ViewCount="417" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;In theory we would always like to predict qualitative responses using&#xA;  the Bayes classifier. But for real data, we do not know the&#xA;  conditional distribution of Y given X, and so computing the Bayes&#xA;  classifier is impossible. Therefore, the Bayes classifier serves as an&#xA;  unattainable gold standard against which to compare other methods.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In a real life situation why we are not able to calculate the &lt;strong&gt;Bayes Classifier&lt;/strong&gt; ? &lt;/li&gt;&#xA;&lt;li&gt;Why can't we calculate the conditional distribution of Y given X ?&lt;/li&gt;&#xA;&lt;li&gt;Can some one explanation with an example?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated. &lt;/p&gt;&#xA;" OwnerUserId="15115" LastEditorUserId="15115" LastEditDate="2016-01-27T02:18:16.167" LastActivityDate="2016-01-27T02:18:16.167" Title="Bayes Classification" Tags="|classification|logistic-regression|supervised-learning|bayesian-networks|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9970" PostTypeId="2" ParentId="9959" CreationDate="2016-01-26T14:12:26.117" Score="2" Body="&lt;p&gt;When you want to use Auto-Encoders (AEs) for dimensionality reduction, you usally add a bottleneck layer. This means, for example, you have 1234-dimensional data. You feed this into your AE, and - as it is an AE - you have an output of dimension 1234. However, you might have many layers in that network and one of them has significantly less dimensions. Lets say you have the topology &lt;code&gt;1234:1024:784:1024:1234&lt;/code&gt;. You train it like this, but you only use the weights from the &lt;code&gt;1234:1024:784&lt;/code&gt; part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you get new input, you just feed it into this network. You can see it as a kind of preprocessing. For the later stages, this is a black box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is manly useful when you have a lot of unlabeled data. It is called &lt;em&gt;Semi Supervised Learning&lt;/em&gt; (SSL).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-01-26T14:12:26.117" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9972" PostTypeId="1" CreationDate="2016-01-26T16:15:33.010" Score="1" ViewCount="266" Body="&lt;p&gt;I am trying to come up with some rules to detect named entities, specifically company or organization names in text. I think it makes sense to focus on verbs. There are a lot of &lt;code&gt;POS Taggers&lt;/code&gt; that can easily detect proper nouns. I personally like &lt;code&gt;StanfordPOSTagger&lt;/code&gt;. Now, once i have the proper noun, i know that it is a named entity. However, to be certain that it is the name of a company, i need to come up with rules and possibly &lt;code&gt;Gazetteers&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of focusing on verbs. Is there a set of common verbs that occur frequently around company names?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could create an annotated corpus and explicitly train a Machine Learning classifier to predict such verbs, but that is a LOT of work. It would be great if someone has already done some research on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, can some other &lt;code&gt;POS&lt;/code&gt; tags give clues? Not just verbs.&lt;/p&gt;&#xA;" OwnerUserId="15735" LastActivityDate="2016-01-26T16:15:33.010" Title="NLP : What are some common verbs surrounding organization names in text" Tags="|python|nlp|named-entity-recognition|nltk|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="9973" PostTypeId="2" ParentId="9945" CreationDate="2016-01-26T18:07:06.920" Score="16" Body="&lt;p&gt;As for a complete machine learning package on GPU's, no such package exists.  However, there are actually a handful of R packages that can use GPU's.  You can see these packages on the &lt;a href=&quot;https://cran.r-project.org/web/views/HighPerformanceComputing.html&quot;&gt;CRAN High Performance Computing page&lt;/a&gt;.  You should note that most of these packages do require you to have a NVIDIA card.  Of the packages available, there are three packages you most likely would utilize unless you have a special case.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/gputools/index.html&quot;&gt;gputools&lt;/a&gt; - if interested in distance computations (only NVIDIA).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/gmatrix/index.html&quot;&gt;gmatrix&lt;/a&gt; - general numeric computations (only NVIDIA).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/gpuR/index.html&quot;&gt;gpuR&lt;/a&gt; - general numeric computations (any GPU via OpenCL).*&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;* &lt;strong&gt;NOTE&lt;/strong&gt; - At the risk of self promotion I am the author of the gpuR package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can likely use the latter two packages to reproduce existing machine learning algorithms.  I am actually using my gpuR package to create a GPU accelerated neuralnet package but this is in progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in summary, if you are determined, the basic resources are available in R.  But if you need something in the immediate future, you will need to explore other resources/approaches as pointed out by @YCR.&lt;/p&gt;&#xA;" OwnerUserId="6385" LastActivityDate="2016-01-26T18:07:06.920" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9974" PostTypeId="2" ParentId="9969" CreationDate="2016-01-26T18:15:29.430" Score="4" Body="&lt;p&gt;You can't calculate the conditional distribution because that distribution is defined over the entire population and assuming a probabilistic model which you do not know. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In real life you have only a sample of the population. More often than not, in real life you do not have the structure of the model also. You do not know if it is a linear model, a polynomial or another type. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probabilistic approach is to assume a model, consider some assumptions for that model and fit the model's parameters using data. That is not &lt;em&gt;the Bayes classifier&lt;/em&gt;, it's only an approximation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trick is 'all models are wrong, some models are useful'. There are many approaches to learn this approximation. You can estimate the joint probability (generative classifiers), you can estimate directly the conditional probability (discriminative classifiers) or you can use a different approach (like SVM, for example) and twist it somehow to approximate the conditional probability. But all of them are approximations. The real one, &lt;em&gt;THE Bayes classifier&lt;/em&gt; will remain unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Later edit]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page&lt;/a&gt; for Bayes classifier it states that: &quot;The Bayes classifier is a useful benchmark in statistical classification.&quot;. My opinion is that this statement is confusing, if not dead wrong. Probably this is related with the wrong usage of expression &quot;Bayes classifier&quot; to designate the Naive Bayes classifier. If one search on google for &quot;Bayes classifier&quot;, he will see a lot of materials related with Naive Bayes. Personally I do not agree with the statement from wikipedia. Bayes classifier is that state of nature that we do not know but we aim to approximate as well as possible.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2016-01-26T18:31:41.070" LastActivityDate="2016-01-26T18:31:41.070" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9975" PostTypeId="2" ParentId="6107" CreationDate="2016-01-26T21:24:44.633" Score="65" Body="&lt;p&gt;I think one way to get a really basic level intuition behind convolution is that you are sliding K filters, which you can think of as K stencils, over the input image and produce K activations - each one representing a degree of match with a particular stencil. The inverse operation of that would be to take K activations and expand them into a preimage of the convolution operation. The intuitive explanation of the inverse operation is therefore, roughly, image reconstruction given the stencils (filters) and activations (the degree of the match for each stencil) and therefore at the basic intuitive level we want to blow up each activation by the stencil's mask and add them up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to approach understanding deconv would be to examine the deconvolution layer implementation in Caffe, see the following relevant bits of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DeconvolutionLayer&amp;lt;Dtype&amp;gt;::Forward_gpu&#xA;ConvolutionLayer&amp;lt;Dtype&amp;gt;::Backward_gpu&#xA;CuDNNConvolutionLayer&amp;lt;Dtype&amp;gt;::Backward_gpu&#xA;BaseConvolutionLayer&amp;lt;Dtype&amp;gt;::backward_cpu_gemm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see that it's implemented in Caffe exactly as backprop for a regular forward convolutional layer (to me it was more obvious after i compared the implementation of backprop in cuDNN conv layer vs ConvolutionLayer::Backward_gpu implemented using GEMM). So if you work through how backpropagation is done for regular convolution you will understand what happens on a mechanical computation level. The way this computation works matches the intuition described in the first paragraph of this blurb.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However, I don't know how the learning of convolutional layers works. (I understand how simple MLPs learn with gradient descent, if that helps).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To answer your other question inside your first question, there are two main differences between MLP backpropagation (fully connected layer) and convolutional nets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) the influence of weights is localized, so first figure out how to do backprop for, say a 3x3 filter convolved with a small 3x3 area of an input image, mapping to a single point in the result image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) the weights of convolutional filters are shared for spatial invariance. What this means in practice is that in the forward pass the same 3x3 filter with the same weights is dragged through the entire image with the same weights for forward computation to yield the output image (for that particular filter). What this means for backprop is that the backprop gradients for each point in the source image are summed over the entire range that we dragged that filter during the forward pass. Note that there are also different gradients of loss wrt x, w and bias since dLoss/dx needs to be backpropagated, and dLoss/dw is how we update the weights. w and bias are independent inputs in the computation DAG (there are no prior inputs), so there's no need to do backpropagation on those.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(my notation here assumes that convolution is y = x*w+b where '*' is the convolution operation)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15766" LastEditorUserId="45264" LastEditDate="2018-06-25T21:16:49.607" LastActivityDate="2018-06-25T21:16:49.607" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="9976" PostTypeId="1" AcceptedAnswerId="10035" CreationDate="2016-01-26T22:07:26.397" Score="2" ViewCount="193" Body="&lt;p&gt;I am looking for ways to perform clustering on an aggregated category while still using individual data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, assume we have a multitude users that we have acquired through different channels. I would like to cluster groups of channels together, so that it is easier to deal with similar channels in similar ways. However, if I aggregate the user data into averages for each channel, then because these averages are heavily driven by outliers (high-spenders), it doesn't give that much insight to the behavior of the majority of users from that channel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to use the individual user data but still gain insight into and cluster based on the channel they come from?&lt;/p&gt;&#xA;" OwnerUserId="15768" LastActivityDate="2016-01-30T23:19:11.103" Title="Perform clustering on individual data in categories" Tags="|r|clustering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9977" PostTypeId="2" ParentId="9976" CreationDate="2016-01-26T23:45:55.990" Score="0" Body="&lt;p&gt;The simplest thing that comes to mind is cluster the users for each channel then compare the channels by their cluster parameters. If you use a non-probabilistic model such as k-means, you could compare the cluster parameters directly. With a large number of clusters this would not be a good idea, for somewhat technical reasons. If you use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Mixture_model&quot; rel=&quot;nofollow&quot;&gt;probabilistic model&lt;/a&gt; you would look at something like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;KLD&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-26T23:45:55.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9978" PostTypeId="1" CreationDate="2016-01-27T01:05:15.363" Score="1" ViewCount="637" Body="&lt;p&gt;I've started working on a project about the trajectory data mining from videos - for example: snowboarding video from GoPro action camera.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the continuation of my previous experiment (MotionML), which was based on the accelerometer datasets and KNN-DTW classification: &lt;a href=&quot;https://github.com/llvll/motionml&quot; rel=&quot;nofollow&quot;&gt;https://github.com/llvll/motionml&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visual odometry and monocular SLAM topics have been researched with the following algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;PTAM&lt;/li&gt;&#xA;&lt;li&gt;DTAM&lt;/li&gt;&#xA;&lt;li&gt;LSD-SLAM&lt;/li&gt;&#xA;&lt;li&gt;ORB-SLAM&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;At this stage I've selected ORB-SLAM algorithm for the initial experiments: &lt;a href=&quot;https://github.com/raulmur/ORB_SLAM&quot; rel=&quot;nofollow&quot;&gt;https://github.com/raulmur/ORB_SLAM&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The implementation will be done in Python using Scikit-Learn and FastDTW for the model training purposes based on the trajectory data. The data mining itself, based on OpenCV and ORB-SLAM, will execute on Apache Spark for video file processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following results are expected:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Extracted trajectory data.&lt;/li&gt;&#xA;&lt;li&gt;Discovered and classified motion patterns - similar to MotionML project above.&lt;/li&gt;&#xA;&lt;li&gt;Video georeferencing or spatio-temporal referencing - GPS geotagging might be present for frame #1, trajectory data will extrapolate geotagging to the next frames. Clock synchronization should take place.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Two questions from my side:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;DTW for trajectory classification - I'm still doubting about DTW and reviewing HMM (Hidden Markov Models) or LSTM neural nets for the same purpose. Any practical advices are really appreciated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm also looking for any examples or advices about SLAM-based feature extraction for machine learning. Ideally for trajectory data mining, but other examples will work as well. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you would like to cooperate or to participate in this project, then please write me directly: oleg.v.puzanov@gmail.com&lt;/p&gt;&#xA;" OwnerUserId="15315" LastActivityDate="2016-01-27T01:05:15.363" Title="Trajectory data mining and pattern recognition using ORB-SLAM and KNN-DTW" Tags="|machine-learning|python|neural-network|time-series|computer-vision|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9979" PostTypeId="2" ParentId="9966" CreationDate="2016-01-27T01:38:22.920" Score="1" Body="&lt;p&gt;Can't help if you're fixed on KNIME, but if you can use &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange Data Mining&lt;/a&gt;, I've had success clustering (took a minute or so) and previewing a scatter plot of a dataset with 32k rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IFDOA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IFDOA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-01-27T01:38:22.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9980" PostTypeId="1" CreationDate="2016-01-27T01:40:51.273" Score="3" ViewCount="238" Body="&lt;p&gt;I'm trying to build a model in R that predicts when a customer will purchase a product again. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't know what kind of model can handle time and predict this kind of outcome. I'm wondering to use: buy till you die package, but this package output the frequency of future purchases, and not when this purchases will happening.&lt;/p&gt;&#xA;" OwnerUserId="14644" LastEditorUserId="1244" LastEditDate="2016-01-27T13:46:44.813" LastActivityDate="2016-01-27T13:47:03.163" Title="Opposite of churn?" Tags="|machine-learning|r|predictive-modeling|churn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9981" PostTypeId="1" CreationDate="2016-01-27T03:50:11.847" Score="2" ViewCount="208" Body="&lt;p&gt;I'm new to machine learning programming and am working on an application which will calculate weights of the attributes to calculate the ETAT (Estimated Turn Around Time) of some tasks in workflow. The set of attributes is not fixed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: ETAT will depend on-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Language   &lt;/li&gt;&#xA;&lt;li&gt;No of properties   &lt;/li&gt;&#xA;&lt;li&gt;Experience level of resources and many&#xA;more.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For each workflow entity there can by a set of these attributes, i.e. a single workflow may be like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Language - English &lt;/li&gt;&#xA;&lt;li&gt;No. of properties - 1200 &lt;/li&gt;&#xA;&lt;li&gt;Experience level of&#xA;resources - Expert &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Language - Russian&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;No. of properties - 21000&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Experience level of resources - Basic&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now for the above set of attributes, how can I design my machine learning algorithm? Any pointers will be highly useful.&lt;/p&gt;&#xA;" OwnerUserId="8412" LastEditorUserId="8412" LastEditDate="2016-01-27T05:15:45.143" LastActivityDate="2016-02-03T21:34:24.403" Title="Machine learning algorithm for a dynamic set of attributes" Tags="|machine-learning|python|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9982" PostTypeId="2" ParentId="9981" CreationDate="2016-01-27T06:19:25.230" Score="1" Body="&lt;p&gt;I would train a &lt;a href=&quot;https://en.wikipedia.org/wiki/Gamma_distribution&quot; rel=&quot;nofollow&quot;&gt;gamma-distributed&lt;/a&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Generalized_linear_model&quot; rel=&quot;nofollow&quot;&gt;GLM&lt;/a&gt;; one for each language. Now, I assume the set of parameters is fixed within each language. You can incorporate ordinal variables (basic -&gt; intermediate -&gt; expert) using dummy coding (i.e., treat them as categorical variables). If there is no intermediate level one bit will do (0 = basic, 1 = expert). And so on for the other properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find free software to implement GLMs in various language, including &lt;a href=&quot;http://statsmodels.sourceforge.net/devel/glm.html&quot; rel=&quot;nofollow&quot;&gt;python&lt;/a&gt; and &lt;a href=&quot;http://data.princeton.edu/R/glms.html&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-01-27T06:35:16.677" LastActivityDate="2016-01-27T06:35:16.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9983" PostTypeId="5" CreationDate="2016-01-27T07:56:42.260" Score="0" Body="&lt;p&gt;Naive Bayes classifiers make use of Bayes theorem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\overbrace{P(c | X)}^{\text{A posteriori}} = \frac{\overbrace{P(X | c)}^{\text{Likelihood}} \cdot \overbrace{P(c)}^{\text{A priori} } }{\underbrace{P(X)}_{\text{evidence}}}$$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-27T10:17:23.237" LastActivityDate="2016-01-27T10:17:23.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9984" PostTypeId="4" CreationDate="2016-01-27T07:56:42.260" Score="0" Body="Naive Bayes classifiers makes the naive assumption that the features are independent. They make use of Bayes theorem." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-27T10:17:28.877" LastActivityDate="2016-01-27T10:17:28.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9985" PostTypeId="5" CreationDate="2016-01-27T08:06:17.083" Score="0" Body="&lt;p&gt;The hinge loss function $\mathcal{l}$ is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{l}(y, t) = \max(0, 1 - t \cdot y)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $t$ is the target value and $y$ is the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It  has values in $[0,  \infty)$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hinge_loss&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-27T10:17:25.850" LastActivityDate="2016-01-27T10:17:25.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9986" PostTypeId="4" CreationDate="2016-01-27T08:06:17.083" Score="0" Body="The hinge loss function is defined as $l(y, t) = max(0, 1 - t \cdot y)$" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-04-23T02:59:55.693" LastActivityDate="2021-04-23T02:59:55.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="9987" PostTypeId="2" ParentId="9950" CreationDate="2016-01-27T08:33:33.873" Score="4" Body="&lt;p&gt;Using a list of entities has few disadvantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The list is closed&lt;/li&gt;&#xA;&lt;li&gt;The list is not context sensitive. You need context in order to differ between &quot;a white house&quot; and &quot;the white house&quot;.&lt;/li&gt;&#xA;&lt;li&gt;List building require a lot of labor&lt;/li&gt;&#xA;&lt;li&gt;List might also contain errors.&lt;/li&gt;&#xA;&lt;li&gt;It does feel like cheating (or at list no NLP insights are used).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can cope with these disadvantages by going along the direction @emre suggested and use the list in order to learn a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you can use tokens near the entity and learn rule like that &quot;I live at X&quot; is an indicator of a place and &quot;I talked with X&quot; is an indicator of a person.&#xA;You can play this game few rounds by increasing your list by the hits of the rules and use the new list to learn more rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please not that in this learning you will introduce noisy into the data so in most cases the learning should be so straight forward. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-01-27T08:33:33.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9988" PostTypeId="2" ParentId="8744" CreationDate="2016-01-27T10:45:48.673" Score="1" Body="&lt;p&gt;The most harrowing problem would be &quot;interpolation&quot;. That is, computing $p(x_i | y)$ for $x_i$ and $y$ you have not yet observed, but fall within the convex hull of your observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to bin events together, or come up with a wacky interpolation scheme. This &lt;em&gt;does&lt;/em&gt; work. However, If you assume a distribution, $p(x_i | y)$ becomes continuous, and you end up with a more statisically sound means of interpolation. &lt;/p&gt;&#xA;" OwnerUserId="15090" LastActivityDate="2016-01-27T10:45:48.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9989" PostTypeId="2" ParentId="9980" CreationDate="2016-01-27T13:47:03.163" Score="4" Body="&lt;p&gt;&lt;strong&gt;Short Answer:&lt;/strong&gt; You could just build a model to predict purchase (0/1) and call it a day. Create a feature that is count of past purchases to control for repeat customers. BUT if you want to specifically deal with the &lt;em&gt;time&lt;/em&gt; component (which is certainly more interesting)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TTE Models:&lt;/strong&gt; You should look at Time-to-Event models (&lt;a href=&quot;https://en.wikipedia.org/wiki/Accelerated_failure_time_model&quot; rel=&quot;nofollow&quot;&gt;accelerated time failure model&lt;/a&gt;, etc.) Normally, the event (0/1) is usually something phrased as a negative outcome (like death or loan default), but this is just because of the research fields these models stem from. There is no reason you can't re-pose your outcome to be a purchase event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Watch Out For:&lt;/strong&gt; Dealing with Censoring in the data. &lt;a href=&quot;https://en.wikipedia.org/wiki/Censoring_(statistics)&quot; rel=&quot;nofollow&quot;&gt;See Here&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Get creative&lt;/strong&gt;, I can't really comment on great detail here because you didn't give a feel for your exact dataset but you might try and work your dependent variable into something more concrete, for example, time between first view of customer and first purchase, or time between most resent 2 purchases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about the &lt;strong&gt;purchase time&lt;/strong&gt; distribution. Essentially, start to think of ways of parameterizing the purchase-time distribution given some conditionals (customer tenure, total purchases, etc.) As you expand on this vein of thought you'll come across methods that will help. For (a simple) example, using EM to estimate the parameters of various purchase time conditional distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="15784" LastActivityDate="2016-01-27T13:47:03.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9990" PostTypeId="1" CreationDate="2016-01-27T14:02:57.430" Score="6" ViewCount="332" Body="&lt;p&gt;I want to estimate the current maximum capacity (in kWh) having the current power consumption (in kWh) and the state of charge of the battery (in %) available in a time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not have a full battery charge circle recorded but only a snippet with the state of charge going from ~95% to ~35% in a 1 second data recording interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first, i cumulated the current power consumption and noticed that the progression between cumulated power consumption and state of charge was almost the same (see figure 1 and 2). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;progression of state of charge vs. cumulated power consumption&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Y5tTp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y5tTp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i tried to use a linear regression model to predict two values of cumulated power consumption at 0% and 100% state of charge. In my assumption the delta of these two leaves me with the current maximum capacity of the battery.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally i want to monitor the capacity during the data recording. Which means i only have data of a few percent of state of charge.&#xA;In figure 3 and 4 you can see my attempt of trying to create a linear regression model for every 2% state of charge. As you can see in the right plot, the estimated capacity has a very high variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;LEFT: regression lines (green) of every 2% state of charge; RIGHT: estimated capacity of every 2% state of charge&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qOmlY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qOmlY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since i am not an expert in neither the matter of regression nor batteries/physics, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a much simple or more exact way (or both) to estimate the current maximum capacity of the battery?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If yes, is there a good way to estimate an sufficiently exact capacity with an even smaller snippet in an ongoing process, lets say every 2% state of charge? (maybe using a characteristic curve of the discharge process)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is a sample of my data: &lt;a href=&quot;http://www.filedropper.com/sampledata&quot; rel=&quot;nofollow noreferrer&quot;&gt;sample file&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="15690" LastEditorUserId="15690" LastEditDate="2016-04-18T08:34:54.337" LastActivityDate="2016-05-10T04:52:36.767" Title="Estimating the battery capacity using current power consumption and battery percentage" Tags="|r|predictive-modeling|time-series|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="9991" PostTypeId="1" CreationDate="2016-01-27T14:12:48.323" Score="6" ViewCount="227" Body="&lt;p&gt;On several tutorials on object and pattern recognition in images (such as the typical &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot; rel=&quot;nofollow&quot;&gt;mnist digit recognition&lt;/a&gt; problem), when working with a sample of $m$ images of $n \times n$ pixels each, the first step is to build a matrix of $m \times (n * n)$, where each image is converted into a single row-vector with all of its pixels, one next to the other. Different techniques are then applied to this matrix, such as PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: when converting a two-dimensional image to a single-dimensional vector by placing each image row next to another, the fact of which pixels are neighbours (in the same column, different row) seems to be lost. For instance, in image:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a b c&#xA;d e f&#xA;g h i&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is converted to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a b c d e f g h i&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fact that &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;e&lt;/code&gt; were next to each other seems to be lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that conclusion correct? Does that fact even matter for analysis? Are there other ways to represent the image that do preserve that piece of information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies in advance if the question is too vague or basic, I'll be glad to reformulate if needed. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="15786" LastEditorUserId="11097" LastEditDate="2016-01-27T16:01:33.973" LastActivityDate="2016-01-27T16:01:33.973" Title="Is information lost when converting an image to a single vector for analysis?" Tags="|image-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9992" PostTypeId="2" ParentId="9991" CreationDate="2016-01-27T14:56:58.933" Score="8" Body="&lt;p&gt;The short answer is it depends on two things: what definition of information you use and if you really use that information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From information theory point of view, if your transformation is reversible than the information is there. This happens because you can apply the inverse transformation to recover the original. So nothing is lost. This is similar with what happens in various lose-less compression algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, information is there, but is encoded in another place, specifically in the transformation function. As a consequence, to learn also the position you have to create those new features to express that information. Not also that when you have a classical learner this information is not understood by learner. Take for example a linear model. There is no difference between $f_1 = \beta_0 +\beta_1 x_1 + \beta_2 x_2$ and $f_2 = \beta_2 x_2 + \beta_0 + \beta_1 x_1$. So learning with a simple linear model is impossible to use directly this locality information. This happens with most learners. So, even in the original space the locality is an information in your head, and not in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand if your neural net is tailored for images and expects to receive patches of images where the order of the values is important than yes, your neural net perhaps understands those relations and use them. Even so, perhaps that custom neural net has a custom way to specify the location information, so what you will have to do would be to encode this information in a format specific to that custom neural net. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand if we are talking about a standard back propagation neural net, the order is irrelevant. The model is simply not able to use the location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a conclusion, if you have a standard generic learning algorithm your only your only chance would be to encode this locality information somehow in the features. For example (this is a made-up example, I do not even know if that would word) one can add a new set of input variable, one for each square of 4 adjacent pixels and use the average of pixels intensity as value. You do not made that explicitly enough, but you created a new set of features which is created using the locality information. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-01-27T14:56:58.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9993" PostTypeId="2" ParentId="6391" CreationDate="2016-01-27T16:11:58.653" Score="8" Body="&lt;p&gt;As David states in the comments if you want to interpret a model you likely want to explore something besides neural nets.  That said it you want to intuitively understand the network plot it is best to think of it with respect to images (something neural networks are very good at).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The left-most nodes (i.e. input nodes) are your raw data variables.&lt;/li&gt;&#xA;&lt;li&gt;The arrows in black (and associated numbers) are the &lt;strong&gt;weights&lt;/strong&gt; which you can think of as &lt;strong&gt;how much that variable contributes to the next node.&lt;/strong&gt;  The blue lines are the bias weights.  You can find the purpose of these weights in the excellent answer &lt;a href=&quot;https://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The middle nodes (i.e. anything between the input and output nodes) are your hidden nodes.  This is where the image analogy helps.  &lt;strong&gt;Each of these nodes constitute a component that the network is learning to recognize.&lt;/strong&gt;  For example a nose, mouth, or eye.  This is not easily determined and is far more abstract when you are dealing with non-image data.&lt;/li&gt;&#xA;&lt;li&gt;The far-right (output node(s)) node is the final output of your neural network.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that this all is omitting the activation function that would be applied at each layer of the network as well.&lt;/p&gt;&#xA;" OwnerUserId="6385" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-01-27T16:11:58.653" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="9994" PostTypeId="2" ParentId="1143" CreationDate="2016-01-27T16:52:11.450" Score="2" Body="&lt;p&gt;Well, I would say knowing the intricacies of 1 or 2 algorithms in detail( like the inner workings of their parameters) is definitely better than knowing how to run a bunch of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been in the Analytics area for about 11 years and a Data Scientist for 2.5 years and I'm speaking from experience. On the other hand, you should definitely be aware of other things out there (more recent algorithms like deep learning, SVM, XGboost etc.) which might be more applicable to your problem at hand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think Dr. Andrew Ng's course goes into a quite a few details of some algorithms and it's a good start. As others have pointed out, &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;http://statweb.stanford.edu/~tibs/ElemStatLearn/&lt;/a&gt; is a good book and it has videos to go with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my personal opinion, the algorithms which you shouldn't miss out on are:&#xA;(Know these in detail):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Multiple linear regression&#xA;2) Logistic regression&#xA;3) Common techniques of dimensionality reduction like PCA&#xA;4) K-means clustering&#xA;5) Non-linear regression&#xA;6) Optimization methods: gradient based search methods, linear programming and discrete optimization&#xA;7) Concepts and algorithms in feature engineering&#xA;8) Simple time-series forecasting methods&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More Esoteric algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Random Forests&#xA;2) SVM&#xA;3) deep learning &#xA;4) Other methods of dimensionality reduction like LDA&#xA;5) Other kernel based methods&#xA;6) Genetic algorithms&#xA;7) XgBoost&#xA;8) Dynamic regression&#xA;9) GARCH/ARCH methods&#xA;10) Structural equation modeling&#xA;11) Box Jenkins methods in time-series forecasting&#xA;12) Information theory: information gain, mutual gain etc.&lt;/p&gt;&#xA;" OwnerUserId="15790" LastActivityDate="2016-01-27T16:52:11.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9996" PostTypeId="2" ParentId="5255" CreationDate="2016-01-27T17:11:39.913" Score="1" Body="&lt;p&gt;An ideal data scientist is 60-70% Statistician and 30-40% a computer scientist and so the old name of &quot;Data scientist' was somebody who was part statistician and part computer science guy.&lt;/p&gt;&#xA;" OwnerUserId="15790" LastActivityDate="2016-01-27T17:11:39.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="9997" PostTypeId="1" CreationDate="2016-01-27T18:53:31.557" Score="4" ViewCount="1412" Body="&lt;p&gt;I have a table with many financial information of thousands of companies around the world (revenue, income, number of employees, total assets etc) and an extra information indicating whether that company owns a specific high value machine (&quot;1&quot; if the company has and &quot;0&quot; if the company hasn't).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to identify which companies don't have this machine, but are potential customers. My first approach was to use a classification algorithm, such as SVM, random Forest etc and use those financial characteristics as predictors of the &quot;has machine&quot;/&quot;hasn't machine&quot; column. The &quot;potential customers&quot; would be, actually, the individuals the model classified as &quot;1&quot; (has machine), but they are actually &quot;0&quot; (hasn't machine). In other words, the false positives of this model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not comfortable with this approach (after all, a perfect model wouldn't indicate a single potential customer!), but I don't know any other way I could approach this problem. I was wondering if someone could give me some direction!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="15215" LastActivityDate="2016-01-28T18:02:08.440" Title="Classification of potential customers" Tags="|machine-learning|classification|marketing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="9999" PostTypeId="1" AcceptedAnswerId="10012" CreationDate="2016-01-27T19:56:50.257" Score="4" ViewCount="2781" Body="&lt;p&gt;I'm trying to train an algorithm to copy some of the top traders on various forex social trading sites. The problem is that the traders only trade around say 10 times per month so even if I only look at minute resolution numbers that's .02% of the time [ 10/(60*24*30)*100 ].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using random forest and it gives an error rate of around 2% which is unacceptable and from what I've read most machine learning algorithms have similar errors rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a better approach?&lt;/p&gt;&#xA;" OwnerUserId="15797" LastActivityDate="2016-01-28T21:54:28.953" Title="What's a good machine learning algorithm for low frequency trading?" Tags="|machine-learning|classification|random-forest|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10000" PostTypeId="1" CreationDate="2016-01-27T19:58:00.083" Score="15" ViewCount="7539" Body="&lt;p&gt;I have read that HMMs, Particle Filters and Kalman filters are special cases of dynamic Bayes networks. However, I only know HMMs and I don't see the difference to dynamic Bayes networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could somebody please explain?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be nice if your answer could be similar to the following, but for bayes Networks:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Hidden Markov Models&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A Hidden Markov Model (HMM) is a 5-tuple $\lambda = (S, O, A, B, \Pi)$:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$S \neq \emptyset$: A set of states (e.g. &quot;beginning of phoneme&quot;, &quot;middle of phoneme&quot;, &quot;end of phoneme&quot;)&lt;/li&gt;&#xA;&lt;li&gt;$O \neq \emptyset$: A set of possible observations (audio signals)&lt;/li&gt;&#xA;&lt;li&gt;$A \in \mathbb{R}^{|S| \times |S|}$: A stochastic matrix which gives probabilites $(a_{ij})$ to get from state $i$ to state $j$.&lt;/li&gt;&#xA;&lt;li&gt;$B \in \mathbb{R}^{|S| \times |O|}$: A stochastic matrix which gives probabilites $(b_{kl})$ to get in state $k$ the observation $l$.&lt;/li&gt;&#xA;&lt;li&gt;$\Pi \in \mathbb{R}^{|S|}$: Initial distribution to start in one of the states.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is usually displayed as a directed graph, where each node corresponds to one state $s \in S$ and the transition probabilities are denoted on the edges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hidden Markov Models are called &quot;hidden&quot;, because the current state is hidden. The algorithms have to guess it from the observations and the model itself. They are called &quot;Markov&quot;, because for the next state only the current state matters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For HMMs, you give a fixed topology (number of states, possible edges). Then there are 3 possible tasks&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Evaluation&lt;/strong&gt;: given a HMM $\lambda$, how likely is it to get observations $o_1, \dots, o_t$ (Forward algorithm)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Decoding&lt;/strong&gt;: given a HMM $\lambda$ and a observations $o_1, \dots, o_t$, what is the most likely sequence of states $s_1, \dots, s_t$ (Viterbi algorithm)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning&lt;/strong&gt;: learn $A, B, \Pi$: &lt;a href=&quot;https://en.wikipedia.org/wiki/Baum%E2%80%93Welch_algorithm&quot; rel=&quot;noreferrer&quot;&gt;Baum-Welch algorithm&lt;/a&gt;, which is a special case of Expectation maximization.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Bayes networks&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Bayes networks are directed acyclical graphs (DAGs) $G = (\mathcal{X}, \mathcal{E})$. The nodes represent random variables $X \in \mathcal{X}$. For every $X$, there is a probability distribution which is conditioned on the parents of $X$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X|\text{parents}(X))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There seem to be (please clarify) two tasks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Inference&lt;/strong&gt;: Given some variables, get the most likely values of the others variables. Exact inference is NP-hard. Approximately, you can use MCMC.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Learning&lt;/strong&gt;: How you learn those distributions depends on the exact problem (&lt;a href=&quot;http://www.eng.tau.ac.il/~bengal/BN.pdf&quot; rel=&quot;noreferrer&quot;&gt;source&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;known structure, fully observable:  maximum likelihood estimation (MLE)&lt;/li&gt;&#xA;&lt;li&gt;known structure, partially observable: Expectation Maximization (EM) or Markov Chain Monte Carlo (MCMC)&lt;/li&gt;&#xA;&lt;li&gt;unknown structure, fully observable: search through model space&lt;/li&gt;&#xA;&lt;li&gt;unknown structure, partially observable: EM + search through model space&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Dynamic Bayes networks&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I guess dynamic Bayes networks (DBNs) are also directed probabilistic graphical models. The variability seems to come from the network changing over time. However, it seems to me that this is equivalent to only copying the same network and connecting every node at time $t$ with every the corresponding node at time $t+1$. Is that the case?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-01-27T21:00:53.283" LastActivityDate="2017-12-27T20:43:21.200" Title="What is the difference between a (dynamic) Bayes network and a HMM?" Tags="|bayesian-networks|pgm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10001" PostTypeId="2" ParentId="5255" CreationDate="2016-01-27T20:02:29.903" Score="1" Body="&lt;p&gt;In several subfields, some were simply called &lt;a href=&quot;https://en.wikipedia.org/wiki/Analyst&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;analysts&lt;/strong&gt;&lt;/a&gt;. If you go back earlier in time, in a pre-science era, I tend to believe that people involved in &lt;a href=&quot;https://en.wikipedia.org/wiki/Divination&quot; rel=&quot;nofollow&quot;&gt;divination&lt;/a&gt; or astrology (several of them because they were paid for that, much more than for serious science) were precursors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a lighter note, I like the neologism &lt;em&gt;dedomenology&lt;/em&gt; (in French, &lt;a href=&quot;http://laurent-duval.blogspot.fr/search/label/dedomenology&quot; rel=&quot;nofollow&quot;&gt;dédoménologie&lt;/a&gt;), and I tend to consider that &lt;a href=&quot;http://laurent-duval.blogspot.fr/2015/08/steinhaus-k-means-clustering-french.html&quot; rel=&quot;nofollow&quot;&gt;Hugo Steinhaus (the inventor of $k$-means)&lt;/a&gt; was, due  to his interest in many applied fields, one of the first to spark the fire of data science.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2016-01-27T20:02:29.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10003" PostTypeId="2" ParentId="9997" CreationDate="2016-01-27T21:01:09.243" Score="6" Body="&lt;p&gt;What you described is the more-or-less standard approach, with a couple of caveats.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Leakers&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Remember that we live in a stochastic world, so a perfect model is immediately suspect of using &lt;em&gt;leakers&lt;/em&gt;, i.e., variables which have a causal connection with the target. In your case, e.g., it would be the presence of a service contract for the machine (which would indicated that the company actually does own one). This, building an interpretable model and examining the most powerful variable for such causal connections (and eliminating them) would be a good idea.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Better Approach&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;However, one could improve on this approach in a very powerful way by replacing the &lt;strong&gt;current&lt;/strong&gt; state of the company with its &lt;strong&gt;historical&lt;/strong&gt; state at the time of the machine purchase. This way one eliminates the leakers automatically and also gets a better model because the predictor values which are more relevant: after all, you are trying to predict which company is ready to &lt;strong&gt;buy&lt;/strong&gt; the machine &lt;strong&gt;now&lt;/strong&gt; rather that which company bought it in the past when its conditions were very different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, your approach works precisely because it is an approximation of this approach (approximating the company state at the time it bought the machine with its current state) and the absurdity/uselessness of the perfect model that you mention is the artifact of your approach being &lt;em&gt;an approximation&lt;/em&gt; of The Right Thing.&lt;/p&gt;&#xA;" OwnerUserId="9867" LastEditorUserId="9867" LastEditDate="2016-01-28T18:02:08.440" LastActivityDate="2016-01-28T18:02:08.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10004" PostTypeId="2" ParentId="9411" CreationDate="2016-01-27T21:10:51.063" Score="4" Body="&lt;p&gt;The reason NB is called &quot;Naive&quot; is that is makes the assumption that the predictive variables are all &lt;strong&gt;independent&lt;/strong&gt;.&#xA;This assumption usually skews the model scores (which, under the above naive assumption are unbiased probability estimates) towards 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, e.g., the presence of words &lt;code&gt;flower&lt;/code&gt; and &lt;code&gt;petal&lt;/code&gt; indicate &lt;code&gt;gardening&lt;/code&gt; category, but, because the presence of these words is not independent (if one is present, the other is likely to be present too), the model will over-value their appearance. Taking the extreme case, if words &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; appear only together, then &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(Category=X | A &amp;amp; B) = P(Category=X | B) = P(Category=X | A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and thus one should multiply by the odds ratio of &lt;code&gt;A&amp;amp;B&lt;/code&gt; once, not twice, as the Naive Bayes algorithm requires.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your remedy is to use &lt;em&gt;calibration&lt;/em&gt; (a separate model which maps model scores to probabilities).&lt;/p&gt;&#xA;" OwnerUserId="9867" LastEditorUserId="9867" LastEditDate="2016-01-27T21:18:58.680" LastActivityDate="2016-01-27T21:18:58.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10005" PostTypeId="1" CreationDate="2016-01-27T22:05:51.333" Score="6" ViewCount="465" Body="&lt;p&gt;I try to understand the difference between radial basis neuron network and support vector regression with Gaussian Kernel. I watched Andrew Ng presentation about kernels in SVN, I have read also an introduction to RBFNN in Bishop's book and Wikipedia's article about SVR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it correctly in both cases I try to find a function from features space to R by computation of gaussians of a distances between data from training set to some centers distributed in feature space (activation of rbfs). Next I try to find the best set of linear combination of coefficients, to approximate my data as a linear combination of activated RBFs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However in some papers these methods are mentioned as two different, competitive methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is what exactly is the difference? &lt;/p&gt;&#xA;" OwnerUserId="15801" LastEditorUserId="11097" LastEditDate="2016-01-28T02:37:58.710" LastActivityDate="2018-03-24T03:03:10.023" Title="Difference between rbfnn and svr with gaussian kernel" Tags="|machine-learning|neural-network|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10006" PostTypeId="1" CreationDate="2016-01-27T22:35:26.847" Score="2" ViewCount="180" Body="&lt;p&gt;I'm using Google API to categorize and predict problems with laptops (Hardware, Software, Network, Customer satisfaction, No reason). I inserted my training data and categories are very well evaluated and predicted. &#xA;Training data only uses 5 main categories as it comes from free text and being specific requires some manual work.&#xA;Now I want to go a level deeper and tag these categories: Example. Laptop was returned because Display was cracked:&#xA;Hardware category was defined already, but I have a dictionary where I can also look for the word &quot;Display&quot;. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hardware:&#xA;   Display:&#xA;      failure: True&#xA;   Info:&#xA;      Physical_damage: Display cracked&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unable to install Skype:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Software:&#xA;   Applications:&#xA;      Skype:&#xA;       failure: True&#xA;   Info:&#xA;      Description: Unable to upgrade    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to categorize first using main categories and after main category is chosen, look into my text for possible options from a bag of words (Example: &lt;code&gt;{&quot;display&quot;: [&quot;screen&quot;, &quot;display&quot;, &quot;monitor&quot;] }, {&quot;applications&quot;, &quot;skype&quot;,&quot;office&quot;,&quot;adobe&quot;,&quot;spotify&quot;}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I run these in 2 stages? Any recommended pattern or solution?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-02-02T19:18:51.270" LastActivityDate="2016-02-02T19:18:51.270" Title="Data categorization" Tags="|machine-learning|classification|nlp|multilabel-classification|nltk|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10007" PostTypeId="2" ParentId="5255" CreationDate="2016-01-28T03:15:22.817" Score="2" Body="&lt;p&gt;Some really nice answers already. However, I would break the entire process of breaking the work of a data scientist into who actually did those:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Getting the data from databases and other sources:&lt;/strong&gt; Generally, it used to be the &lt;strong&gt;DBA&lt;/strong&gt; who gets the data from the DB's and the people who collects data from other sources are called &lt;em&gt;data guys&lt;/em&gt;, they don't really have a specific name (atleast in India). And the scraping and crawling scripts are written by &lt;strong&gt;software engineers&lt;/strong&gt; who are hired especially for that purpose.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Analytics and prediction:&lt;/strong&gt; Done by people called the &lt;strong&gt;statisticians&lt;/strong&gt; or the &lt;strong&gt;mathematicians&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Visualizations and reporting:&lt;/strong&gt; Done by people called &lt;strong&gt;business analysts&lt;/strong&gt; or the MBA guys in the company.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Big Data and pipelining stuff:&lt;/strong&gt; Done by &lt;strong&gt;software engineers&lt;/strong&gt; hired especially for thar particular purpose.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-01-28T03:15:22.817" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10008" PostTypeId="2" ParentId="9941" CreationDate="2016-01-28T06:22:11.380" Score="5" Body="&lt;p&gt;In case that the number of items is quite small, turning the problem into a classification problem will be the most convenient solution. Use each item as a feature, the class as a concept. Now you can apply the methods that @OvisAmmon recommended on or other classification methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, usually in market basket analysis there are many items and each user buy few of them.  That leads to a very sparse matrix and most classification methods have problems dealing with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common method to analyze market basket data is using &#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;noreferrer&quot;&gt;Association rules&lt;/a&gt; which are implemented by first locating frequent item sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use association rules on order to learn on market basket in the following way:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Add item for every class&lt;/li&gt;&#xA;&lt;li&gt;Add the owner class to its basket (e.g, transform {milk, beer} basket of a student in to the basket {mlik, beer, student}&lt;/li&gt;&#xA;&lt;li&gt;Use a frequent item set algorithm to find them&lt;/li&gt;&#xA;&lt;li&gt;Remove all frequent item sets that don't have a class member in them. Such sets can't help deduce the class.&lt;/li&gt;&#xA;&lt;li&gt;From the frequent items set find rules that imply the classes (with good enough support and confidence&lt;/li&gt;&#xA;&lt;li&gt;Use the these rules that match a basket in order to predict class&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You should note some points:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some association rules with different implication might apply to the same basket. You might have the rules milk -&gt; X and beer -Y and the basket {milk, beer}. If the set {milk, beer} is frequent enough you might have a rule specifically to it. Otherwise you'll need a met rule in order to base by the rules. Such rules might be taking the rule with the highest confidence or the most likely class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Association rules are based only on the existence of item, not on the lack of them. If students never buy diapers, you will not find such association rule. You can add &quot;no diapers&quot; as an item to your data set but that will turn it from a sparse one to a full one and the storage might be unfeasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complexity of association rules is exponential. The is simply since the output it self is exponential , as can be noted when you have a &lt;a href=&quot;https://en.wikipedia.org/wiki/Clique_(graph_theory)&quot; rel=&quot;noreferrer&quot;&gt;Clique&lt;/a&gt; in the data.&#xA;Usually people don't try to find all association rules due to that problem, which is the practical thing to do. &#xA;However, it means that there might be association rules in your data that you won't find.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-01-28T06:22:11.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10009" PostTypeId="2" ParentId="8236" CreationDate="2016-01-28T09:52:40.290" Score="2" Body="&lt;p&gt;You can download a free as in beer software Qlikview that allows you to do interactive data discovery via graphical interface similar to Excel but also featuring a powerful scripting language for data load and transformation. Huge flat files is no problem at all. It is an in-memory technology so you'd need a computer with a lot of RAM. &#xA;The advantage though is that it can load billions of records in a star schema but still allows you to do discovery ad hoc (second or sub-second time) without writing and rewriting SQL. I am always using it to screen the data and run descriptive statistics + visual exploration on it. &#xA;From Data Science standpoint this is a very advanced column based data engine integrated with a powerful dictionary of descriptive stat functions and interactive graphics UI. You will be surprised what is possible with your data.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-01-28T09:52:40.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10010" PostTypeId="2" ParentId="3788" CreationDate="2016-01-28T12:41:51.127" Score="3" Body="&lt;p&gt;Speaking as a Graph/Complex Networks guy I'd recommend &lt;a href=&quot;http://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;Networkx&lt;/a&gt; package in Python. This is the main library I used for my master thesis and my research during last 2 years. As long as your graph is not gigantic (millions of nodes) you can handle it using Networkx.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what you need is not only the library but a philosophy to convert your data into a graph. What is your data exactly? If you have some addresses, what would be the node and what would define an edge? This is important actually. Graph is not constructed from your addresses unless you have a definition for vertices and edges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you convert your data into a graph Networkx provides you all fancy algorithms for everything (shortest path, community detection, statistical analysis, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you provide info in the comment here I'll answer your question in details.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-01-28T12:41:51.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10011" PostTypeId="1" AcceptedAnswerId="15713" CreationDate="2016-01-28T13:13:52.680" Score="4" ViewCount="62" Body="&lt;p&gt;I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy. &#xA;The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow&quot;&gt;precision&lt;/a&gt; metric).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My validation prediction is as following: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Randomly split the data to $k$ stratified folds&lt;/li&gt;&#xA;&lt;li&gt;Fit $k$ models &lt;/li&gt;&#xA;&lt;li&gt;Estimate each model according to $I$ (which results $k$-crossvalidated values of $I$)&lt;/li&gt;&#xA;&lt;li&gt;The final model prediction performance is calculated by the average of $k$-crossvalidated $I$ measures.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to perform some &lt;strong&gt;significance testing&lt;/strong&gt; - to be able to say the &lt;em&gt;confidence&lt;/em&gt; for the model prediction performance - to what extent I am “sure” in this $I$?&lt;/p&gt;&#xA;" OwnerUserId="1093" LastActivityDate="2016-12-13T08:22:22.083" Title="Assessing significance / confidence of a crossvalidated performance measure" Tags="|statistics|cross-validation|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10012" PostTypeId="2" ParentId="9999" CreationDate="2016-01-28T14:34:36.233" Score="2" Body="&lt;p&gt;Random forests, GBM or even the newer and fancier xgboost are not the best candidates for binary classification (predicting ups and down) of stocks predictions or forex trading or at least not as the main algorithm. The reason is that, for this particular problem, they require a huge amount of trees (and tree depth in case of GBM or xgboost) to obtain reasonable accuracy (Breiman suggested using at least 5000 trees and to &quot;not be stingy&quot; and in fact his main ML paper on RF he used 50,000 trees per run).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some quants use random forests as feature selectors while others use it to generate new features. It all depends on the characteristics of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest you read this &lt;a href=&quot;https://quant.stackexchange.com/questions/9313/machine-learning-vs-regression-and-or-why-still-use-the-latter/9317#9317&quot;&gt;question and answers on quant.stackexchange&lt;/a&gt; where people discuss what methods are the best and when to use them, among them ISOMAP, Laplacian eigenmaps, ANNs, swarm optimization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out the &lt;a href=&quot;https://quant.stackexchange.com/tags/machine-learning/hot&quot;&gt;machine-learning tag on the same site&lt;/a&gt;, there you might find information related to your particular dataset.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-01-28T21:54:28.953" LastActivityDate="2016-01-28T21:54:28.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10013" PostTypeId="1" CreationDate="2016-01-28T16:02:34.740" Score="2" ViewCount="102" Body="&lt;p&gt;R function plotmo &lt;a href=&quot;http://www.milbo.users.sonic.net/gallery/plotmo.html&quot; rel=&quot;nofollow&quot;&gt;http://www.milbo.users.sonic.net/gallery/plotmo.html&lt;/a&gt; is used for plotting dependence plot from e.g. decision tree model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases, resulting graph does not print full names of features on each plot. What I am saying is, e.g. the &quot;feature1&quot; is automatically renamed &quot;f1&quot; or something similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want full name. But plotmo just adjust automatically to short name (I believe for memory convenience, but not useful for perception).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe Plotmo has parameter forcing it to produce full name? I just see -main parameter. But what values to set as to produce full original feature names for each generated plot?&lt;/p&gt;&#xA;" OwnerUserId="14167" LastActivityDate="2016-01-28T16:02:34.740" Title="full name of each variable in Plotmo graph" Tags="|machine-learning|r|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10015" PostTypeId="1" AcceptedAnswerId="10498" CreationDate="2016-01-29T03:10:22.990" Score="6" ViewCount="12460" Body="&lt;p&gt;There are many resources online about how to implement MLP in tensorflow, and most of the samples do work :) But I am interested in a particular one, that I learned from &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot;&gt;https://www.coursera.org/learn/machine-learning&lt;/a&gt;. In which, it uses a &lt;em&gt;cost&lt;/em&gt; function defined as follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ -y_k^{(i)} \log((h_\theta(x^{(i)}))_k - (1 - y_k^{(i)}) \log(1 - (h_\theta(x^{(i)}))_k \right]&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_\theta$ is the &lt;em&gt;sigmoid&lt;/em&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there's my implementation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# one hidden layer MLP&#xA;&#xA;x = tf.placeholder(tf.float32, shape=[None, 784])&#xA;y = tf.placeholder(tf.float32, shape=[None, 10])&#xA;&#xA;W_h1 = tf.Variable(tf.random_normal([784, 512]))&#xA;h1 = tf.nn.sigmoid(tf.matmul(x, W_h1))&#xA;&#xA;W_out = tf.Variable(tf.random_normal([512, 10]))&#xA;y_ = tf.matmul(h1, W_out)&#xA;&#xA;# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)&#xA;cross_entropy = tf.reduce_sum(- y * tf.log(y_) - (1 - y) * tf.log(1 - y_), 1)&#xA;loss = tf.reduce_mean(cross_entropy)&#xA;train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)&#xA;&#xA;correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&#xA;# train&#xA;with tf.Session() as s:&#xA;    s.run(tf.initialize_all_variables())&#xA;&#xA;    for i in range(10000):&#xA;        batch_x, batch_y = mnist.train.next_batch(100)&#xA;        s.run(train_step, feed_dict={x: batch_x, y: batch_y})&#xA;&#xA;        if i % 100 == 0:&#xA;            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})&#xA;            print('step {0}, training accuracy {1}'.format(i, train_accuracy))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think the definition for the layers are correct, but the problem is in the &lt;em&gt;cross_entropy&lt;/em&gt;. If I use the first one, &lt;em&gt;the one got commented out&lt;/em&gt;, the model converges quickly; &lt;strong&gt;but if I use the 2nd one, which I think/hope is the translation of the previous equation, the model won't converge.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="3167" LastActivityDate="2016-03-03T03:11:39.900" Title="Implement MLP in tensorflow" Tags="|machine-learning|tensorflow|" AnswerCount="1" CommentCount="6" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10016" PostTypeId="5" CreationDate="2016-01-29T06:50:57.143" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-01-29T06:50:57.143" LastActivityDate="2016-01-29T06:50:57.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10017" PostTypeId="4" CreationDate="2016-01-29T06:50:57.143" Score="0" Body="A form of signal processing where the input is an image. Usually treating the digital image as a two-dimensional signal (or multidimensional). This processing may include image restoration and enhancement (in particular, pattern recognition and projection)." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-01-29T09:48:35.823" LastActivityDate="2016-01-29T09:48:35.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10018" PostTypeId="5" CreationDate="2016-01-29T06:51:48.630" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-01-29T06:51:48.630" LastActivityDate="2016-01-29T06:51:48.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10019" PostTypeId="4" CreationDate="2016-01-29T06:51:48.630" Score="0" Body="Variables (used for prediction or explication) used in regression or regression-like models (like clustering, discrimination). Use this tag for questions about constructing such variables or selecting the best among them." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-02-07T08:53:14.817" LastActivityDate="2016-02-07T08:53:14.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10020" PostTypeId="5" CreationDate="2016-01-29T06:52:59.090" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-01-29T06:52:59.090" LastActivityDate="2016-01-29T06:52:59.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10021" PostTypeId="4" CreationDate="2016-01-29T06:52:59.090" Score="0" Body="Java (not to be confused with JavaScript) is a general-purpose object-oriented programming language designed to be used in conjunction with the Java Virtual Machine (JVM). &quot;Java platform&quot; is the name for a computing system that has installed tools for developing and running Java programs. Use this tag for questions referring to Java programming language or Java platform tools." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-01-29T09:48:41.690" LastActivityDate="2016-01-29T09:48:41.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10022" PostTypeId="5" CreationDate="2016-01-29T06:54:47.960" Score="0" Body="&lt;p&gt;NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2021-03-18T20:36:55.853" LastActivityDate="2021-03-18T20:36:55.853" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10023" PostTypeId="4" CreationDate="2016-01-29T06:54:47.960" Score="0" Body="NLTK is a free, open-source natural language processing toolkit for python. It is used primarily for text processing applications and includes libraries specifically made for classification, tokenization, stemming, tagging, parsing, and semantic reasoning." OwnerUserId="11097" LastEditorUserId="29169" LastEditDate="2021-03-18T19:40:53.550" LastActivityDate="2021-03-18T19:40:53.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10024" PostTypeId="2" ParentId="5804" CreationDate="2016-01-29T14:22:55.907" Score="2" Body="&lt;p&gt;your grammar is correct!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grammar = &quot;&quot;&quot;MEDIA: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;           RELATION: {&amp;lt;V.*&amp;gt;}&#xA;                     {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;           ENTITY: {&amp;lt;NN.*&amp;gt;}&quot;&quot;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;by specifying &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RELATION: {&amp;lt;V.*&amp;gt;}&#xA;          {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you are indicating that there are two ways to generate the &lt;code&gt;RELATION&lt;/code&gt; chunk i.e. &lt;code&gt;{&amp;lt;V.*&amp;gt;}&lt;/code&gt; or &lt;code&gt;{&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grammar = &quot;&quot;&quot;MEDIA: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;               RELATION: {&amp;lt;V.*&amp;gt;}&#xA;                         {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN.*&amp;gt;+}&#xA;               ENTITY: {&amp;lt;NN.*&amp;gt;}&quot;&quot;&quot;&#xA;    chunkParser = nltk.RegexpParser(grammar)&#xA;    tagged = nltk.pos_tag(nltk.word_tokenize(&quot;adventure movies between 2000 and 2015 featuring performances by daniel craig&quot;))&#xA;&#xA;    tree = chunkParser.parse(tagged)&#xA;&#xA;    for subtree in tree.subtrees():&#xA;        if subtree.label() == &quot;RELATION&quot;: &#xA;            print(&quot;RELATION: &quot;+str(subtree.leaves()))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RELATION: [('featuring', 'VBG')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15735" LastActivityDate="2016-01-29T14:22:55.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10025" PostTypeId="1" AcceptedAnswerId="20112" CreationDate="2016-01-29T15:39:26.277" Score="12" ViewCount="7102" Body="&lt;p&gt;I was looking into the possibility to classify sound (for example sounds of animals) using spectrograms. The idea is to use a deep convolutional neural networks to recognize segments in the spectrogram and output one (or many) class labels.  This is not a new idea (see for example &lt;a href=&quot;http://danielnouri.org/notes/2014/01/10/using-deep-learning-to-listen-for-whales/&quot; rel=&quot;noreferrer&quot; title=&quot;using it to classify whale sounds&quot;&gt;whale sound classification&lt;/a&gt; or &lt;a href=&quot;http://papers.nips.cc/paper/3674-unsupervised-feature-learning-for-audio-classification-using-convolutional-deep-belief-networks.pdf&quot; rel=&quot;noreferrer&quot;&gt;music style recognition&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem that I'm facing is that I have sound files of different length and therefore spectrograms of different sizes. So far, every approach I have seen uses a fixed size sound sample but I can't do that because my sound file might be 10 seconds or 2 minutes long. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With, for example, a bird sound in the beginning and a frog sound at the end (output should be &quot;Bird, Frog&quot;). My current solution would be to add a temporal component to the neural network (creating more of a recurrent neural network) but I would like to keep it simple for now. Any ideas, links, tutorials, ...?&lt;/p&gt;&#xA;" OwnerUserId="15847" LastEditorUserId="836" LastEditDate="2017-07-02T11:46:50.503" LastActivityDate="2017-07-02T11:46:50.503" Title="Deep Learning with Spectrograms for sound recognition" Tags="|deep-learning|multilabel-classification|audio-recognition|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10026" PostTypeId="2" ParentId="6601" CreationDate="2016-01-29T17:22:56.890" Score="1" Body="&lt;p&gt;I would suggest &quot;&lt;a href=&quot;http://marcocuturi.net/Papers/cuturi11fast.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generalized Alignment Kernels&lt;/a&gt;&quot; by Marco Suturi for time-series classification. The idea is to use the well known DTW distance in the SVM. The problem which was solved in the paper is that the DTW distance does not result in a valid SVM kernel; there are some tricks employed by the authors to enable that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran this algorithm on some popular time-series datasets and noticed:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The algorithm is very fast&lt;/li&gt;&#xA;&lt;li&gt;The performances are very good on the respective datasets with respect to the published test sets (&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/time_series_data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCR Time Series&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finally, the source code is available.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastEditorUserId="7848" LastEditDate="2017-02-23T17:21:40.743" LastActivityDate="2017-02-23T17:21:40.743" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10027" PostTypeId="1" AcceptedAnswerId="10384" CreationDate="2016-01-29T17:56:53.437" Score="3" ViewCount="946" Body="&lt;p&gt;I need to build a time series model with explanatory variables, and &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMAX&lt;/a&gt; seems to be the one that comes up most frequently in practice, based on my survey of related work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_model&quot; rel=&quot;nofollow&quot;&gt;ARX&lt;/a&gt; solves a similar problem, but I'm having trouble wrapping my mind around the practical differences in what an ARX representation of my data would have compared with an ARIMAX approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know ARX lacks the moving average component, but I'm curious whether anyone can point me toward some best practice for choosing one approach over the other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there certain characteristics in my data I should look for to make an informed decision?&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="11097" LastEditDate="2016-01-29T18:13:51.863" LastActivityDate="2016-02-25T16:20:06.223" Title="ARIMAX v. ARX Time Series Modeling" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10028" PostTypeId="1" CreationDate="2016-01-29T19:39:59.317" Score="1" ViewCount="887" Body="&lt;p&gt;I have read a lot of documentation surrounding &lt;code&gt;NP&lt;/code&gt; chunking but what about Verb Phrases? Has there been a fixed set of rules for &lt;code&gt;VP&lt;/code&gt; chunking? &lt;/p&gt;&#xA;" OwnerUserId="15735" LastActivityDate="2017-11-24T23:19:03.433" Title="NLP : Rules for chunking Verb Phrases" Tags="|nlp|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10030" PostTypeId="1" CreationDate="2016-01-30T11:18:52.890" Score="0" ViewCount="3805" Body="&lt;p&gt;I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Medicine_Name&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Disease_Name&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Body_Part&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please give me some ideas to implement it, specifically how to prepare the training example and implement the training phase. &lt;/p&gt;&#xA;" OwnerUserId="15861" LastEditorUserId="13727" LastEditDate="2016-02-01T09:09:14.767" LastActivityDate="2016-02-01T09:09:14.767" Title="Algorithm for classification of words into given categories" Tags="|machine-learning|nlp|text-mining|named-entity-recognition|" AnswerCount="1" CommentCount="2" ClosedDate="2016-02-07T08:51:37.010" ContentLicense="CC BY-SA 3.0" />
  <row Id="10031" PostTypeId="1" CreationDate="2016-01-30T16:51:26.207" Score="3" ViewCount="96" Body="&lt;p&gt;I need to specify semantically close words in a corpus. For instance &lt;em&gt;newspaper/magazine&lt;/em&gt; in the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read [newspaper/magazine] yesterday.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example both words are close in meaning. I need to find these kind of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find those words?&lt;/p&gt;&#xA;" OwnerUserId="12378" LastEditorUserId="8878" LastEditDate="2016-02-01T16:44:11.773" LastActivityDate="2016-02-01T16:44:11.773" Title="how to provide alternate for a word in corpus for word2vec model" Tags="|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10032" PostTypeId="1" CreationDate="2016-01-30T18:41:55.730" Score="7" ViewCount="1146" Body="&lt;p&gt;Recently researchers at Google DeepMind published a &lt;a href=&quot;http://deepmind.com/alpha-go.html&quot;&gt;paper&lt;/a&gt;, where they described a Go playing system that beat the best current computer programs and the human European champion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a quick look at that paper, and it seems it is using many interesting ideas from previous papers. What did they do differently that allowed them to achieve this spectacular improvement? &lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2018-06-24T09:22:42.943" Title="What is the novelty in AlphaGo, Google Deepmind's Go playing system?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10033" PostTypeId="2" ParentId="10031" CreationDate="2016-01-30T20:35:13.647" Score="2" Body="&lt;p&gt;Add a regularization term for synonymity to the objective function (to be maximized):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log P(w_i|h) \rightarrow \log P(w_i|h) + \lambda \frac{1}{|S|} \sum_{(i,j) \in S} sim(w_i, w_j)$ where $S$ is the set of synonyms, and $sim$ is your word similarity function (e.g., cosine similarity).&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-30T20:35:13.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10034" PostTypeId="1" CreationDate="2016-01-30T20:37:07.527" Score="3" ViewCount="690" Body="&lt;p&gt;I have 130 records in one CSV dataset and I'm using j48 decision tree. I used the whole training set for testing, and the result was 79 correctly classified and 51 incorrectly classified records.&#xA;Now! I want a result between 95% - 100% and don't have ability to add or remove any record to those 130. But I'm allowed to play a little with my dataset, for example use 10 out of 130 in such a way that all of those 10 must be classified correctly.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The Quality of tree is not Important&lt;/li&gt;&#xA;&lt;li&gt;The Number of test samples is not so important (10-15 is good)&lt;/li&gt;&#xA;&lt;li&gt;The Only Important thing is Accuracy between 95-100 %  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;FIRST, I tried to move 10 correct samples to bottom of dataset and use &quot;split by percentage&quot; (92.30%) but it wasn't useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At SECOND, I tried to choose them by &quot;try and error&quot; and chose randomly, and I was just testing the last sample (130th), the I found some correct samples and keep it there, then tried another below the last successful sample and tested the last two lines, the result should have been 50% or 100%, but surprisingly was 0% (Total instants:2 correct:0 Incorrect:2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me please?&#xA;I already thank a lot dear readers...&lt;/p&gt;&#xA;" OwnerUserId="15871" LastActivityDate="2016-02-01T12:03:47.260" Title="Making fake result in data mining using weka j48 algorithm" Tags="|data-mining|decision-trees|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10035" PostTypeId="2" ParentId="9976" CreationDate="2016-01-30T23:19:11.103" Score="3" Body="&lt;p&gt;As always clustering is about what is the meaning of distance, since that encodes at least some part of your question. So you question is which channels are similar, where similarity is defined on users. Usually you do not assume some nesting on your instances, but what you have is basically a nesting of users in channels. So you have to incorporate this kind of nesting into the distance / similarity function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would start with some observation on similarity function. We denote similarity between instance $i$ and $j$ with $d(i,j)$. Usually this function obeys the following condition $d(i,j)\ge0$ for any $i$ and $j$. Note that we usually have equality only on identical data points. The main consequence of this observation is that we can have basically an identity at user level. What you want is an identity at channel level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way would be to define the similarity function like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d_c(i,j) = I_{c_i\ne c_j}d(i,j)$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $I_{c_i\ne c_j}$ is $1$ when channel of instance $i$ is different that channel of instance $j$. The main effect is that now all the points from the same channel are considered identical since the distance between them is zero. When the identity function is $1$ the distance is given by your real business distance which should be constructed by you and answer your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use such kind of function in a hierarchical clustering basically it will start to find first some clusters which are identically with your grouping on channels and later on it will join clusters which are similar. This kind of approach will work even with a kmeans algorithm and perhaps with most of the clustering approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A slight different approach would be to define your $I$ function to return $1-\lambda$ when cluster is different and $\lambda$ when channels are equal, and have $\lambda$ a positive value close to $0$. This will not guarantee that all the clients will go into the same cluster, but it gives you the benefit that you have a slider which you can use to fine tune the compromise between &lt;em&gt;all user of the same channel goes into the same cluster&lt;/em&gt; and &lt;em&gt;a distance measure which is more robust to outliers&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A totally different approach from an implementation point of view would be to define a more complex function directly on channel samples. This would be similar with how hierarchical clustering works since in order to joins to clusters it should have a distance function which would measure inter clusters similarity. See more on &lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_clustering#Linkage_criteria&quot; rel=&quot;nofollow&quot;&gt;linkage-criteria&lt;/a&gt;. For example average linkage clustering. Note that I said that the approach is different only algorithmic. I would bet that the results would be similar with the first approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A totally different approach would be to use a more robust criteria. It is known that sample average is not robust since one point can blow away the estimation. Median instead is much stable. You can use a median or a trimmed mean to have a more robust aggregation values. This would have the advantage that the clustering would be much faster since you would work with channels instead of clients and the running time for computing the clustering would be reduced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally, another approach which comes to my mind would be to go further with comparing channels, but this time using a distance which would be based on statistical tests. I will give you a scenario to clarify. Suppose that your users have an attribute named &lt;em&gt;Weight&lt;/em&gt;, which as expected would be a continuous variable. How could you define a distance between the &lt;em&gt;Weight&lt;/em&gt; of users of one channel and &lt;em&gt;Weight&lt;/em&gt; of users of other channel? If you can assume a distribution, like a Gaussian on that weight you can build a two sample t-test on the two samples which are the two clusters. If you can't assume a distribution you can employ a homogeneity / independent test like &lt;a href=&quot;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test&quot; rel=&quot;nofollow&quot;&gt;two sample KS test&lt;/a&gt;. KS test is valuable since does not assume any distribution and is sensitive to both changes is shape and location. If you have nominal attributes you can employ a &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson&amp;#39;s_chi-squared_test#Test_of_independence&quot; rel=&quot;nofollow&quot;&gt;chi-square independence test&lt;/a&gt;. Be careful with what you can use from that tests. In order to have equal contribution for each attribute used in distance function you have to you p-values. Also note that if the test is significant it will have a small p-value, since the null hypotheses for both tests is the independence assumption, which can be translated as same marginal for both samples. So, smaller p-values means bigger distance. You can use $1-\text{p_value}$ or even you can try $\frac{1}{\text{p_value}}$.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-01-30T23:19:11.103" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10036" PostTypeId="2" ParentId="10032" CreationDate="2016-01-30T23:56:07.940" Score="6" Body="&lt;p&gt;The existing programs before AlphaGo were based on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Convolutional neural networks (CNN), trained on a database of existing games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Monte Carlo tree search (MCTS)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;AlphaGo is based on a combination of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) reinforcement learning: train networks by letting versions of CNN's (see above) play against eachother.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AND&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) MCTS using moves generated by step A)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On top of that, the performance was improved even further by using distributed computing with large amounts of CPU's and GPU's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the novelty was the combination of the above techniques A) and B).&lt;/p&gt;&#xA;" OwnerUserId="9186" LastEditorUserId="9186" LastEditDate="2018-06-24T09:22:42.943" LastActivityDate="2018-06-24T09:22:42.943" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="10037" PostTypeId="1" AcceptedAnswerId="10041" CreationDate="2016-01-31T10:05:49.300" Score="2" ViewCount="90" Body="&lt;p&gt;I am trying to synthetize clients Data in order to do clustering. &lt;strong&gt;My problem is for 1 customer I have severals rows&lt;/strong&gt;. I would like to synthetize informations to get 1 row per customer. This clustering is about how customers use fidelity program. Here is a picture of my table : By Column (left to right) : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1) CustomerID 2)Date at which their use their points 3) Category number (Ex: 1 is gift card, 2 is a flight etc) 4) How many points they used 5) How many items they purchased with points&lt;/em&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pCvIh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pCvIh.png&quot; alt=&quot;Table&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how could I have 1 customer per row without loosing informations.&#xA;Maybe Pivot Table? But I don&quot;t know how it work exactly. I am new to statistic btw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cédric &lt;/p&gt;&#xA;" OwnerUserId="15879" LastActivityDate="2016-01-31T21:03:18.233" Title="Filter Data for clustering" Tags="|clustering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10038" PostTypeId="2" ParentId="10030" CreationDate="2016-01-31T14:47:21.820" Score="2" Body="&lt;p&gt;The task you are asking about is &lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow&quot;&gt;Named entity recognition&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way to identify a word category is by using patterns for the context of the word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you can use patterns like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;I took X&quot; for a medicine name&lt;/li&gt;&#xA;&lt;li&gt;&quot;I had pain in my X&quot; for a body part&lt;/li&gt;&#xA;&lt;li&gt;&quot;I suffered from X&quot; for a disease &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The easiest and fastest way is to discuss the pattern with a domain expert and do a sanity check on the data you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more systematic approach is to deduce the pattern from your data.&#xA;I guess you can get a list of samples of the items.&#xA;The technical details depend on the patterns you use but for this example let consider tokenization. &#xA;Take the context of k-words next to the items on the list. Compare it to the context of other words. Contexts that are related to your items are suitable as patterns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bare in mind that these pattern will have very different precision and recall and they will lead to errors. It is important that you will have a labeled dataset so you will be able to estimate the patterns performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will be fairly easy to get pretty good classification. It might be very hard to get a very good classification.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2016-01-31T14:57:30.387" LastActivityDate="2016-01-31T14:57:30.387" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10039" PostTypeId="1" CreationDate="2016-01-31T18:10:33.100" Score="5" ViewCount="134" Body="&lt;p&gt;I was looking into the definition of SVD and trying to understand which are the conditions needed to be met in order to be able to use it. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there any hypothesis concerning the distribution of the data that I want to apply SVD on ? &lt;/li&gt;&#xA;&lt;li&gt;Is the condition (number of data vectors &gt; the dimension of the vectors space) strict ? Or is there always a way to decompose a dataset using SVD ? In the definition of SVD, all websites that I could find start by the same condition without fearther explanitions :&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;given an $m\times n$ real matrix A with $n&amp;gt;m$, A can be written as ...  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="15885" LastEditorUserId="29575" LastEditDate="2017-12-18T01:13:32.660" LastActivityDate="2017-12-23T10:06:19.833" Title="Amount of data needed and hypothesis for SVD" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10040" PostTypeId="1" AcceptedAnswerId="10042" CreationDate="2016-01-31T19:09:31.433" Score="5" ViewCount="1798" Body="&lt;p&gt;I have a classification problem for which a feedforward, fully connected neural net works reasonably well (two classes, true positive and true negative rate close to 80%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to get these rates to 90%, and more features is one of the catalysts for improvements I can think of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do autoencoders to learn additional, interesting features work well for problems that do not involve images?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9197" LastActivityDate="2016-08-30T15:03:57.933" Title="do autoencoders work well for non images?" Tags="|classification|neural-network|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10041" PostTypeId="2" ParentId="10037" CreationDate="2016-01-31T21:03:18.233" Score="0" Body="&lt;p&gt;If you can afford to do the full join once, do it and learn which columns are useful through &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow&quot;&gt;feature selection&lt;/a&gt;. Then you can only SELECT these columns for subsequent iterations, when the database is updated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a survey: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.295.8115&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;Feature Selection for Clustering: A Review&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-31T21:03:18.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10042" PostTypeId="2" ParentId="10040" CreationDate="2016-01-31T21:06:06.353" Score="4" Body="&lt;p&gt;Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are lots of parameters, and not every configuration will work well. Even the optimization algorithm is a hyperparameter. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-01-31T21:06:06.353" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10046" PostTypeId="2" ParentId="10034" CreationDate="2016-02-01T12:03:47.260" Score="2" Body="&lt;p&gt;I don't understand the purpose when the quality of the tree is not important but you should be able to make a tree to have 100% accuracy on training data set easily. Just avoid any pruning and let the tree grow as maximum as possible. As far as I remember Weka set a pruning policy for J48 by default, you should disable that. Also check to see that the split nodes with minimum number of instances possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that I an considering better, there are cases when full accuracy is impossible. Think for example a simple toy data set with 10 instances. Suppose that all instances are the same for input variables, but for target variable 5 are positive and 5 are negative. Either way you take a decision for prediction, the accuracy on training data would be 0.5. In this case you simply do not have enough data to discriminate on. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-02-01T12:03:47.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10047" PostTypeId="1" AcceptedAnswerId="10051" CreationDate="2016-02-01T12:58:14.967" Score="6" ViewCount="1862" Body="&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/QGKbr/model-selection-and-train-validation-test-sets&quot; rel=&quot;noreferrer&quot;&gt;Model Selection and Train/Validation/Test Sets - Stanford University | Coursera:&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At &lt;strong&gt;10:59&lt;/strong&gt;~&lt;strong&gt;11:10&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One final note: I should say that in the machine learning as of this practice today, there &lt;strong&gt;aren't&lt;/strong&gt; many people that will do that early thing that I talked about, and said that, you know...​&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is my comprehension correct?&#xA;Because English subtitles on coursera sometimes are not correct.&#xA;As I konw, here what Chinese subtitle means is opposite to what English one does.&#xA;So I am not sure whether Andrew Ng said &quot;there &lt;strong&gt;aren't&lt;/strong&gt;&quot; or &quot;there &lt;strong&gt;are&lt;/strong&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your reading.​&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I would like to ask &lt;strong&gt;another one&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/yCAup/diagnosing-bias-vs-variance&quot; rel=&quot;noreferrer&quot;&gt;Diagnosing Bias vs. Variance - Stanford University | Coursera:&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At &lt;strong&gt;02:34&lt;/strong&gt;~&lt;strong&gt;02:36&lt;/strong&gt;, what Andrew Ng said is not quite clear as well as the English subtitle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My comprehension is as following&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If d equals 1,.... to be high training error.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's not that complete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would anyone like to identify that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you...&lt;/p&gt;&#xA;" OwnerUserId="15908" LastEditorUserId="15908" LastEditDate="2016-02-01T17:49:53.790" LastActivityDate="2016-02-01T17:49:53.790" Title="On coursera what exactly does Andrew Ng say in videos Lectures 60 &amp; 61 of machine learning?" Tags="|machine-learning|cross-validation|model-selection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10048" PostTypeId="1" AcceptedAnswerId="10052" CreationDate="2016-02-01T15:18:33.907" Score="35" ViewCount="70716" Body="&lt;p&gt;I am working on research, where need to classify one of three event WINNER=(&lt;code&gt;win&lt;/code&gt;, &lt;code&gt;draw&lt;/code&gt;, &lt;code&gt;lose&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;WINNER  LEAGUE  HOME    AWAY    MATCH_HOME  MATCH_DRAW  MATCH_AWAY  MATCH_U2_50 MATCH_O2_50&#xA;3         13    550      571          1.86        3.34        4.23       1.66     2.11&#xA;3         7     322     334           7.55         4.1         1.4       2.17     1.61&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My current model is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def build_model(input_dim, output_classes):&#xA;    model = Sequential()&#xA;    model.add(Dense(input_dim=input_dim, output_dim=12, activation=relu))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(output_dim=output_classes, activation='softmax'))&#xA;    model.compile(loss='categorical_crossentropy', optimizer='adadelta')&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I am not sure that is the correct one for multi-class classification&lt;/li&gt;&#xA;&lt;li&gt;What is the best setup for binary classification?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;EDIT: #2 - Like that?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(input_dim=input_dim, output_dim=12, activation='sigmoid'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(output_dim=output_classes, activation='softmax'))&#xA;model.compile(loss='binary_crossentropy', optimizer='adadelta')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14684" LastEditorUserId="31513" LastEditDate="2017-05-03T15:19:41.160" LastActivityDate="2020-01-06T08:02:10.323" Title="What is the best Keras model for multi-class classification?" Tags="|python|neural-network|classification|clustering|keras|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10049" PostTypeId="1" AcceptedAnswerId="10056" CreationDate="2016-02-01T16:05:41.110" Score="5" ViewCount="174" Body="&lt;p&gt;I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if anyone could refer me to a method for solving my problem, or tell me how this problem is referred to in the academic literature, so that I can look for resources online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is the problem :&lt;/strong&gt; &#xA;From some text (wikipedia articles, for example), I would like to extract the hierarchy of different concepts that can be found in it. By hierarchy I mean a tree wherein A is a descendant of B if A or one of A's parents (transitive) is defined by B. For instance, normal distribution would be a descendant of probability (since normal distribution is defined using probabilities) and probability would be a descendant (or child) of mathematics. Since it is transitive, normal distribution would also be a child of mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I thought about solving this is by looking at the number of times a word A is used alone (called A), the words A and B are used together (called A AND B, 'together' could be, for instance, in the same article or in the same paragraph, or in the same sentence), and the number of times the word B is used alone (called B). Let A be mathematics and B be probability. Then, if the ratios (A AND B)/A and (A AND B)/B are low, then it could imply that there is no direct link between A and B (but a link could exist through transitivity). Conversely, if A is bigger than B, A is a bigger concept than B. If A and B are almost the same then they are probably siblings (children of the same parent).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take 3 examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mathematics (A) and carrot (B). A AND B is really low compared to A and B, so there is no direct link between them (or only an indirect link by transitivity).&lt;/li&gt;&#xA;&lt;li&gt;Mathematics (A) and probabilities (B). A AND B is quite high compared to B, and A is much bigger than B, so B should be a child of A (probabilities is a child of mathematics).&lt;/li&gt;&#xA;&lt;li&gt;Topology (A) and Probabilities (B). A AND B is relativaly high (the texts that present the different areas of mathematics will likely speak about the 2), A and B are about the same order of magnitude, so A and B should be the children of a same parent. Indeed, Topology and Probabilities are the children of Mathematics.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This way of solving the problem is far from perfect, for instance 'the' (A) and 'probability' (B) would probably end up saying probability is a child of the (because A AND B is huge and A is much bigger than B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone knows some papers on this or has any ideas on how I might solve this problem, I would appreciate some direction. Also, does my solution seem viable? How could it be improved?&lt;/p&gt;&#xA;" OwnerUserId="10500" LastEditorUserId="13413" LastEditDate="2016-02-01T18:35:50.087" LastActivityDate="2016-02-01T21:43:53.117" Title="Inferring Relational Hierarchies of Words" Tags="|nlp|unsupervised-learning|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10050" PostTypeId="1" AcceptedAnswerId="10086" CreationDate="2016-02-01T16:17:26.047" Score="3" ViewCount="363" Body="&lt;p&gt;&lt;strong&gt;TLDR:&lt;/strong&gt; Please help me understand the graph representation of the network in the image below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hi, this is pretty stupid, but I'm just have trouble visualising what I'm actually doing with this neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read about neural networks and multilayer perceptrons for some time and I'm just getting started with actually using them. I started with a super simple example, just to get warmed up but now I've confused myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I artificially generated some data and used nntools in matlab to attempt to &quot;predict&quot; the results. I built a neural network with the following parameters:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;feed forward backprop network.&lt;/li&gt;&#xA;&lt;li&gt;Gradient Descent training algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Gradient Descent learniing algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Performance/loss function of mean squared error.&lt;/li&gt;&#xA;&lt;li&gt;two layers:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;first with three neurons and Tansig activation function.&lt;/li&gt;&#xA;&lt;li&gt;the second with one neuron and linear activation.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I end up with something looking like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EPajk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EPajk.png&quot; alt=&quot;My network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't know what this actually represents, I'm all sorts of confused right now. Could someone please explain/upload an image/draw some ascii to represent the neurons and edges in the above network? It would really help clear my head.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I think it's like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           T  L&#xA;           o&#xA;          / \&#xA;         /   \&#xA;   IN &amp;gt; o--o--o--o &amp;gt; OUT&#xA;         \   /&#xA;          \ /&#xA;           o&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With linear activations in columns L and Tanh activations in columns T. Is that right? Doesn't make sense to me.&lt;/p&gt;&#xA;" OwnerUserId="14617" LastEditorUserId="14617" LastEditDate="2016-02-01T19:50:51.213" LastActivityDate="2016-02-04T09:58:07.610" Title="Simple ANN visualisation" Tags="|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10051" PostTypeId="2" ParentId="10047" CreationDate="2016-02-01T16:19:04.107" Score="7" Body="&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;, he actually says the opposite:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One final note: I should say that in the machine learning as of this practice today, there &lt;strong&gt;are&lt;/strong&gt; many people that will do that early thing that I talked about, and said that, you know...​&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then he says (the &quot;early thing&quot; he talked about):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;selecting your model as a test set and then using the same test set to report the error&#xA;  ...&#xA;  unfortunately many people do that&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In this lesson he explains about separating the data set:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;training set&lt;/em&gt; to train the model;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;cross validation set&lt;/em&gt; to find the right parameters;&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;test set&lt;/em&gt; to find the final generalization error (of the function with the best parameter values found during using the &lt;em&gt;cross validation set&lt;/em&gt;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So Andrew Ng is complaining that many people us the same data set to find the right parameters, and then report the error of that data set as final &lt;em&gt;generalization error&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13123" LastActivityDate="2016-02-01T16:19:04.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10052" PostTypeId="2" ParentId="10048" CreationDate="2016-02-01T16:41:58.290" Score="39" Body="&lt;p&gt;Your choices of &lt;code&gt;activation='softmax'&lt;/code&gt; in the last layer and compile choice of &lt;code&gt;loss='categorical_crossentropy'&lt;/code&gt; are good for a model to predict multiple mutually-exclusive classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding more general choices, there is rarely a &quot;right&quot; way to construct the architecture. Instead that should be something you test with different meta-params (such as layer sizes, number of layers, amount of drop-out), and should be results-driven (including any limits you might have on resource use for training time/memory use etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a cross-validation set to help choose a suitable architecture. Once done, to get a more accurate measure of your model's general performance, you should use a separate test set. Data held out from your training set separate to the CV set should be used for this. A reasonable split might be 60/20/20 train/cv/test, depending on how much data you have, and how much you need to report an accurate final figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Question #2, you can either just have two outputs with a softmax final similar to now, or you can have &lt;em&gt;final&lt;/em&gt; layer with one output, &lt;code&gt;activation='sigmoid'&lt;/code&gt; and &lt;code&gt;loss='binary_crossentropy'&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Purely from a gut feel from what might work with this data, I would suggest trying with &lt;code&gt;'tanh'&lt;/code&gt; or &lt;code&gt;'sigmoid'&lt;/code&gt; activations in the hidden layer, instead of &lt;code&gt;'relu'&lt;/code&gt;, and I would also suggest increasing the number of hidden neurons (e.g. 100) and reducing the amount of dropout (e.g. 0.2). Caveat: Gut feeling on neural network architecture is not scientific. Try it, and test it.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-02-01T16:47:52.520" LastActivityDate="2016-02-01T16:47:52.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10053" PostTypeId="1" AcceptedAnswerId="10054" CreationDate="2016-02-01T17:36:16.720" Score="2" ViewCount="417" Body="&lt;p&gt;I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task I'm currently struggling is: user needs to enter data column by column, which would consist of a number n of ID's , a number m of attributes and a number k of classes, with the conditions that n, m &gt; 0 and k &gt;= 0. Data is heterogeneous, so indexes can be both numeric or text, and the same goes for attributes and classes. I'm supposing there will be no null in the data for simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently thinking on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Creating a table with more than enough columns (all with null values), so that I can work using only the non-null columns (which will be got from user input). However this would limit the size of the datasets people could input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Resorting to create an specialized data structure on a programming language, do all the work there and finally, create a table dynamically  to store the result data there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Using a database specialized for this kind of data (maybe a document-based DB).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Create a data structure on the RDBMS itself (I'm using PostgreSQL), let's say a variable size array, so that I can create the table directly from the user input, using only 3 variable arrays (one for indexes, one for attributes and one for classes). However, I keep in mind that attributes and indexes could be of different types, so the array would have to support heterogeneous data type and I don't know if this is possible on a RDBMS or SQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been looking for information on information but got no result until now. Any guidances to a package, language library, extension or paper, thesis, technical report with relevant information would be appreciated.&#xA;Also, personal experiences with doing something similar could be useful.&lt;/p&gt;&#xA;" OwnerUserId="13688" LastEditorUserId="21" LastEditDate="2017-05-19T16:11:17.853" LastActivityDate="2017-05-19T16:11:17.853" Title="Data structure design for supporting arbitrary number of columns in table or database" Tags="|dataset|data|databases|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10054" PostTypeId="2" ParentId="10053" CreationDate="2016-02-01T19:44:47.643" Score="1" Body="&lt;p&gt;I've done something like you're describing using &lt;a href=&quot;https://www.mongodb.com/&quot; rel=&quot;nofollow&quot;&gt;mongoDB&lt;/a&gt;--I think you'll best use your time using some sort of NoSQL approach, rather than creating a specialized one-off solution. If you're using Python, I've had excellent experiences using &lt;a href=&quot;https://api.mongodb.org/python/current/&quot; rel=&quot;nofollow&quot;&gt;PyMongo&lt;/a&gt; to handle reads and writes from within my code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would strongly caution you against adopting your approach #1. This could break very easily in the future, and there are databases designed to handle your exact problem!&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-02-01T19:44:47.643" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="10055" PostTypeId="1" CreationDate="2016-02-01T21:26:08.853" Score="0" ViewCount="228" Body="&lt;p&gt;I have a tif file that shows a map. On white background the border of ditricts are drawn with black, nothing else is on the picture. I want to color the districts according to certain values. Is it possible to do that in R? (since the picture doesn't &quot;know&quot; the name of districts in advance I suppose I have to do some manual work but it's ok for me)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: the picture is similar to this one: &lt;a href=&quot;http://www.geoindex.hu/wp-content/uploads/evk_106_20140101.png&quot; rel=&quot;nofollow&quot;&gt;http://www.geoindex.hu/wp-content/uploads/evk_106_20140101.png&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5211" LastEditorUserId="5211" LastEditDate="2016-02-02T21:40:38.867" LastActivityDate="2016-02-03T13:27:54.427" Title="How to color a tif file in R?" Tags="|r|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ClosedDate="2016-02-05T01:10:33.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="10056" PostTypeId="2" ParentId="10049" CreationDate="2016-02-01T21:43:53.117" Score="5" Body="&lt;p&gt;Look up taxonomy/ontology construction/induction. Relevant papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees&lt;/li&gt;&#xA;&lt;li&gt;Topic Models for Taxonomies&lt;/li&gt;&#xA;&lt;li&gt;OntoLearn Reloaded. A Graph-Based Algorithm for Taxonomy Induction&lt;/li&gt;&#xA;&lt;li&gt;Ontology Population and Enrichment: State of the Art&lt;/li&gt;&#xA;&lt;li&gt;Probabilistic Topic Models for Learning Terminological Ontologies&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-01T21:43:53.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10057" PostTypeId="1" CreationDate="2016-02-01T21:54:04.777" Score="1" ViewCount="163" Body="&lt;p&gt;Currently using networkX to build a probability based network for unsupervised learning, it is a basic reader understanding relational connectivity through probabilities, network slows significantly after about 40 pages (8000 words) any suggestions for a better network platform would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="15921" LastActivityDate="2017-12-07T21:11:46.693" Title="Best network structure for unsupervised learning" Tags="|nlp|unsupervised-learning|" AnswerCount="1" CommentCount="2" ClosedDate="2017-12-08T03:28:51.523" ContentLicense="CC BY-SA 3.0" />
  <row Id="10058" PostTypeId="1" AcceptedAnswerId="10068" CreationDate="2016-02-01T22:11:56.673" Score="2" ViewCount="495" Body="&lt;p&gt;I'm totally new to the topic of real-time bidding in which I know Machine Learning algorithms are used pretty often.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody explain me the system in a plain language i.e. a language for a non-technical person?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the bidding? Who bids on what? Where does Machine Learning get involve?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is &lt;a href=&quot;https://developers.google.com/ad-exchange/rtb/cookie-guide?hl=en&quot; rel=&quot;nofollow&quot;&gt;cookie matching&lt;/a&gt; mainly about?&lt;/p&gt;&#xA;" OwnerUserId="14672" LastActivityDate="2016-02-02T19:32:30.473" Title="Analysis of Real-Time Bidding" Tags="|data-stream-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10060" PostTypeId="1" AcceptedAnswerId="10067" CreationDate="2016-02-02T01:42:38.053" Score="6" ViewCount="3190" Body="&lt;p&gt;&lt;strong&gt;Our weapons&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. &lt;a href=&quot;https://askubuntu.com/questions/725444/help-me-win-this-war&quot;&gt;Help me win this war!&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The battlefield&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have articles, which belong to &lt;em&gt;c&lt;/em&gt; categories, where &lt;em&gt;c&lt;/em&gt; is fixed. I am vectorizing the contents of the articles to &lt;strong&gt;TF-IDF&lt;/strong&gt; features. Now I am running a naive k-means algorithm, which takes &lt;code&gt;c&lt;/code&gt; centroids to begin with and starts, iteratively, grouping articles (i.e. rows of the TF-IDF matrix, where you can see &lt;a href=&quot;https://stackoverflow.com/questions/35109424/how-to-make-tf-idf-matrix-dense&quot;&gt;here&lt;/a&gt; how I built it), until converenge occurs.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Special notes&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Initial centroids: Tried with random from within each category or&#xA;with the mean of all the articles from each category.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Distance function: Euclidean.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question(s)&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy is poor, as expected, can I do any better, by making another choice for the initial centroids, or/and pick another distance function?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;print &quot;Hello Data Science site!&quot; :)&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="15927" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-02-02T18:28:38.880" Title="Improve k-means accuracy" Tags="|python|text-mining|apache-hadoop|k-means|distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10061" PostTypeId="2" ParentId="9138" CreationDate="2016-02-02T04:34:30.600" Score="1" Body="&lt;p&gt;If you have a lot of data you could split your words into sentences of single letters and treat each sentence as a separate document. Then feed it into CountVectorizer from sklearn to create a sparse matrix containing vectors for ngrams of length from 2 to max length. Use binary=True to avoid double counting of a repeating pattern inside the word. Last step is to calculate which columns of the matrix have the highest number of 'ones' and look up which ngrams that were from the vectorizer dictionary.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-02T04:34:30.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10062" PostTypeId="1" CreationDate="2016-02-02T07:36:26.677" Score="1" ViewCount="279" Body="&lt;p&gt;Given a very large population, is there an accepted recall estimation method to use when one is validating a trained model against the population? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, a trained model was tested on a disjoint data set, so it has test precision and recall values, but the test data is an extremely small fraction the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the model is run against 'live' data, and when I look at randomly sampled results, I observe an overfit. I want to estimate how bad the situation is, so it makes sense to estimate precision and recall estimates in the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a way to do this for precision calculations, as one can randomly select from the classifier's positive outcomes at various times to calculate a sample mean and standard deviation for precision. However, it does not appear to be easy to estimate recall in the case of drastic class imbalance, and samples drawn from the population keep coming up with zero positive-class data points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone come across a situation like this? I would appreciate any pointers or reference materials on the subject.&lt;/p&gt;&#xA;" OwnerUserId="15930" LastEditorUserId="8878" LastEditDate="2016-02-03T17:20:00.557" LastActivityDate="2016-02-03T17:20:00.557" Title="Estimating precision &amp; recall" Tags="|text-mining|model-evaluations|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10063" PostTypeId="1" CreationDate="2016-02-02T08:36:55.023" Score="7" ViewCount="8689" Body="&lt;p&gt;For clustering, DBSCAN surpass k-means in terms of handling arbitrary  shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data sets that have special features (for example, the moon-shaped data sets). Can someone point me to some real world datasets where DBSCAN outperforms K-means. &lt;/p&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="924" LastEditDate="2016-02-03T09:49:48.607" LastActivityDate="2016-02-12T06:33:58.687" Title="For which real world data sets does DBSCAN surpass K-means.?" Tags="|machine-learning|dataset|k-means|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10064" PostTypeId="1" AcceptedAnswerId="11302" CreationDate="2016-02-02T09:05:18.940" Score="11" ViewCount="4963" Body="&lt;p&gt;Today, in a lecture it was claimed that the direction of edges in a Bayes network doesn't really matter. They don't have to represent causality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is obvious that you cannot switch any single edge in a Bayes network. For example, let $G = (V, E)$ with $V = \{v_1, v_2, v_3\}$ and $E=\{(v_1, v_2), (v_1, v_3), (v_2, v_3)\}$. If you would switch $(v_1, v_3)$ to $(v_3, v_1)$, then $G$ would no longer be acyclical and hence not a Bayes network. This seems to be mainly a practical problem how to estimate the probabilities then. This case seems to be much more difficult to answer, so I will skip it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This made me ask the following questions for which I hope to get answers here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is it possible for any directed acyclical graph (DAG) to reverse all edges and still have a DAG?&lt;/li&gt;&#xA;&lt;li&gt;Assume a DAG $G$ and data is given. Now we construct the inverse DAG $G_\text{inv}$. For both DAGs, we fit the data to the corresponding Bayes networks. Now we have a set of data for which we want to use the Bayes network to predict the missing attributes. Could there be different results for both DAGs? (Bonus if you come up with an example)&lt;/li&gt;&#xA;&lt;li&gt;Similar to 2, but simpler: Assume a DAG $G$ and data is given. You may create a new graph $G'$ by inverting any set of edges, as long as $G'$ remains acyclical. Are the Bayes networks equivalent when it comes to their predictions?&lt;/li&gt;&#xA;&lt;li&gt;Do we get something if we have edges which do represent causality?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-02T09:16:25.737" LastActivityDate="2016-04-20T09:30:21.023" Title="Is the direction of edges in a Bayes Network irrelevant?" Tags="|bayesian-networks|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10065" PostTypeId="1" CreationDate="2016-02-02T09:38:57.447" Score="2" ViewCount="43" Body="&lt;p&gt;I am having around 1000 DAG(Directed Acyclic Graph) of different files showing java.io.BufferedReader usage. Following is representation of one of the graphs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;digraph G {&#xA;  9 [ label=&quot;9 : ROOT:setup()#0&quot; ];&#xA;  10 [ label=&quot;10 : START IF&quot; ];&#xA;  12 [ label=&quot;12 : java.net.URL.openConnection()#1&quot; ];&#xA;  11 [ label=&quot;11 : END IF&quot; ];&#xA;  13 [ label=&quot;13 : java.net.URL.openConnection()#0&quot; ];&#xA;  14 [ label=&quot;14 : START IF&quot; ];&#xA;  16 [ label=&quot;16 : java.net.HttpURLConnection.setRequestProperty()#2&quot; ];&#xA;  15 [ label=&quot;15 : END IF&quot; ];&#xA;  17 [ label=&quot;17 : START IF&quot; ];&#xA;  19 [ label=&quot;19 : java.net.HttpURLConnection.addRequestProperty()#2&quot; ];&#xA;  18 [ label=&quot;18 : END IF&quot; ];&#xA;  21 [ label=&quot;21 : java.net.HttpURLConnection.setReadTimeout()#1&quot; ];&#xA;  22 [ label=&quot;22 : java.net.HttpURLConnection.setConnectTimeout()#1&quot; ];&#xA;  23 [ label=&quot;23 : java.net.HttpURLConnection.setUseCaches()#1&quot; ];&#xA;  24 [ label=&quot;24 : &amp;lt;static&amp;gt;java.net.HttpURLConnection.setFollowRedirects()#1&quot; ];&#xA;  25 [ label=&quot;25 : START IF&quot; ];&#xA;  27 [ label=&quot;27 : java.net.HttpURLConnection.setRequestMethod()#1&quot; ];&#xA;  28 [ label=&quot;28 : java.net.HttpURLConnection.setDoOutput()#1&quot; ];&#xA;  29 [ label=&quot;29 : java.net.HttpURLConnection.setDoInput()#1&quot; ];&#xA;  30 [ label=&quot;30 : java.net.HttpURLConnection.getOutputStream()#0&quot; ];&#xA;  31 [ label=&quot;31 : java.io.DataOutputStream.&amp;lt;init&amp;gt;()#1&quot; ];&#xA;  32 [ label=&quot;32 : java.io.DataOutputStream.writeBytes()#1&quot; ];&#xA;  33 [ label=&quot;33 : java.io.DataOutputStream.flush()#0&quot; ];&#xA;  26 [ label=&quot;26 : END IF&quot; ];&#xA;  34 [ label=&quot;34 : java.net.HttpURLConnection.getInputStream()#0&quot; ];&#xA;  35 [ label=&quot;35 : java.io.InputStreamReader.&amp;lt;init&amp;gt;()#1&quot; ];&#xA;  36 [ label=&quot;36 : java.io.BufferedReader.&amp;lt;init&amp;gt;()#1&quot; ];&#xA;  9 -&amp;gt; 10;&#xA;  10 -&amp;gt; 12;&#xA;  12 -&amp;gt; 11;&#xA;  10 -&amp;gt; 13;&#xA;  12 -&amp;gt; 13;&#xA;  13 -&amp;gt; 11;&#xA;  11 -&amp;gt; 14;&#xA;  14 -&amp;gt; 16;&#xA;  12 -&amp;gt; 16;&#xA;  13 -&amp;gt; 16;&#xA;  16 -&amp;gt; 15;&#xA;  15 -&amp;gt; 17;&#xA;  17 -&amp;gt; 19;&#xA;  12 -&amp;gt; 19;&#xA;  13 -&amp;gt; 19;&#xA;  16 -&amp;gt; 19;&#xA;  19 -&amp;gt; 18;&#xA;  18 -&amp;gt; 20;&#xA;  16 -&amp;gt; 20;&#xA;  19 -&amp;gt; 20;&#xA;  20 -&amp;gt; 21;&#xA;  12 -&amp;gt; 21;&#xA;  13 -&amp;gt; 21;&#xA;  19 -&amp;gt; 21;&#xA;  21 -&amp;gt; 22;&#xA;  12 -&amp;gt; 22;&#xA;  13 -&amp;gt; 22;&#xA;  16 -&amp;gt; 22;&#xA;  19 -&amp;gt; 22;&#xA;  22 -&amp;gt; 23;&#xA;  12 -&amp;gt; 23;&#xA;  13 -&amp;gt; 23;&#xA;  16 -&amp;gt; 23;&#xA;  19 -&amp;gt; 23;&#xA;  21 -&amp;gt; 23;&#xA;  23 -&amp;gt; 24;&#xA;  24 -&amp;gt; 25;&#xA;  25 -&amp;gt; 27;&#xA;  12 -&amp;gt; 27;&#xA;  13 -&amp;gt; 27;&#xA;  16 -&amp;gt; 27;&#xA;  19 -&amp;gt; 27;&#xA;  21 -&amp;gt; 27;&#xA;  22 -&amp;gt; 27;&#xA;  23 -&amp;gt; 27;&#xA;  27 -&amp;gt; 28;&#xA;  12 -&amp;gt; 28;&#xA;  13 -&amp;gt; 28;&#xA;  16 -&amp;gt; 28;&#xA;  19 -&amp;gt; 28;&#xA;  21 -&amp;gt; 28;&#xA;  22 -&amp;gt; 28;&#xA;  23 -&amp;gt; 28;&#xA;  28 -&amp;gt; 29;&#xA;  12 -&amp;gt; 29;&#xA;  13 -&amp;gt; 29;&#xA;  16 -&amp;gt; 29;&#xA;  19 -&amp;gt; 29;&#xA;  21 -&amp;gt; 29;&#xA;  22 -&amp;gt; 29;&#xA;  23 -&amp;gt; 29;&#xA;  27 -&amp;gt; 29;&#xA;  29 -&amp;gt; 30;&#xA;  12 -&amp;gt; 30;&#xA;  13 -&amp;gt; 30;&#xA;  16 -&amp;gt; 30;&#xA;  19 -&amp;gt; 30;&#xA;  21 -&amp;gt; 30;&#xA;  22 -&amp;gt; 30;&#xA;  23 -&amp;gt; 30;&#xA;  27 -&amp;gt; 30;&#xA;  28 -&amp;gt; 30;&#xA;  30 -&amp;gt; 31;&#xA;  31 -&amp;gt; 32;&#xA;  32 -&amp;gt; 33;&#xA;  31 -&amp;gt; 33;&#xA;  33 -&amp;gt; 26;&#xA;  26 -&amp;gt; 34;&#xA;  12 -&amp;gt; 34;&#xA;  13 -&amp;gt; 34;&#xA;  16 -&amp;gt; 34;&#xA;  19 -&amp;gt; 34;&#xA;  21 -&amp;gt; 34;&#xA;  22 -&amp;gt; 34;&#xA;  23 -&amp;gt; 34;&#xA;  27 -&amp;gt; 34;&#xA;  28 -&amp;gt; 34;&#xA;  29 -&amp;gt; 34;&#xA;  30 -&amp;gt; 34;&#xA;  34 -&amp;gt; 35;&#xA;  35 -&amp;gt; 36;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to cluster these graphs.I am trying to implement K-Medoids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to start with K-Medoid implementation?&#xA;How can i find distance between two graphs?&lt;/p&gt;&#xA;" OwnerUserId="15931" LastActivityDate="2022-05-02T11:04:50.347" Title="How to apply K-Medoids in many CFG?" Tags="|clustering|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10066" PostTypeId="2" ParentId="6693" CreationDate="2016-02-02T10:02:30.633" Score="1" Body="&lt;p&gt;The question asks about the &lt;code&gt;Limitations on the number of items to use in apriori algorithm?&lt;/code&gt; and I would answer in terms of space and time complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say that number of input transactions are N(=20) and the number of unique elements be R(approx 900). Assuming your threshold count is quite small (which means very few candidates are pruned), the time and space complexity for size i candidates would be &lt;code&gt;O(R^i)&lt;/code&gt;. So you see, if very few candidates are pruned, the space (and time) requirements become exponential. It might seem unintutive at first, given that you have only 20 transactions. But the bottleneck is the number of candidates which increases exponentially with number of items.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-02-02T10:02:30.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10067" PostTypeId="2" ParentId="10060" CreationDate="2016-02-02T17:25:00.407" Score="4" Body="&lt;p&gt;Great question, @gsamaras! The way you've set up this experiment makes a lot of sense to me, from a design point of view, but I think there are a couple aspects you can still examine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, it's possible that uninformative features are distracting your classifier, leading to poorer results. In text analytics, we often talk about &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow&quot;&gt;stop word&lt;/a&gt; filtering, which is just the process of removing such text (e.g., the, and, or, etc.). There are standard stop word lists you can easily find online (e.g., &lt;a href=&quot;http://www.ranks.nl/stopwords&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;), but they can sometimes be heavy-handed. The best approach is to build a table relating feature frequency to class, as this will get at domain-specific features that you won't likely find in such look-up tables. There is varying evidence as to the efficacy of stop word removal in the literature, but I think these findings mostly have to do with classifier-specific (for example, support vector machines tend to be less affected by uninformative features than does a naive bayes classifier. I suspect k-means falls into the latter category).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, you might consider a different feature modeling approach, rather than tf-idf. Nothing against tf-idf--it works fine for many problems--but I like to start with binary feature modeling, unless I have experimental evidence showing a more complex approach leads to better results. That said, it's possible that k-means could respond strangely to the switch from a floating-point feature space to a binary one. It's certainly an easily-testable hypothesis!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you might look at the expected class distribution in your data set. Are all classes equally likely? If not, you may get better results from either a sampling approach, or using a different distance metric. &lt;a href=&quot;https://www.erpublication.org/admin/vol_issue1/upload%20Image/IJETR021251.pdf&quot; rel=&quot;nofollow&quot;&gt;k-means is known to respond poorly in skewed class situations&lt;/a&gt;, so this is something to consider as well! There is probably research available in your specific domain describing how others have handled this situation.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-02-02T17:25:00.407" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10068" PostTypeId="2" ParentId="10058" CreationDate="2016-02-02T19:32:30.473" Score="4" Body="&lt;p&gt;A simplified example: suppose two companies: the webpage owner and a book store. And a customer named Jane, interested in reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The web page owner has some free space on its page, which can be sold for advertisement. The book store wants to place an advertisement on the same web page, in order to increase sales. Both companies meet eachother at an auction, where the web space is sold to the highest bidder in real-time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the book store is bidding on the right to show an advertisement to Jane, who is visiting the page of the webpage owner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The machine learning is done on the part of the book store, who receives information about the web page visitor. This can be all sort of information that the web page owner wants to release, and that could be of any use to the bookstore. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this information, it is decided weather to make a bidding for the advertisement space or not, and the amount up to which the book store will want to make a bidding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without cookie matching, the webpage visitor Jane will probably not be identifiable by the bookstore, so the store must decide on bidding, based on parameters like geographical location of the customer, browser version (just to name a few ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With cookie matching, each visitor/customer gets a unique identifier at the book store. Based on this identifier, the book store has more information of the visitor, like: what ads have been served before, and how long ago ? This visit to the web page can be linked to earlier visits and this information will likely ease the decision making process for doing the bidding on the advertisement space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;( There is more to it as there can be two more intermediary companies: one that holds the auction and one that delivers the ad )&lt;/p&gt;&#xA;" OwnerUserId="9186" LastActivityDate="2016-02-02T19:32:30.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10069" PostTypeId="2" ParentId="10064" CreationDate="2016-02-02T20:26:11.310" Score="1" Body="&lt;p&gt;This might be a bit unsatisfying, so feel free not to accept this answer, and apologies in advance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a Bayes net, nodes represent random variables, and edges represent conditional dependences. When you interpret the nodes a certain way, conditioning flows a certain way naturally. Arbitrarily reversing them doesn't really make sense in the context of modeling data. And a lot of time, the arrows do represent causality.&lt;/p&gt;&#xA;" OwnerUserId="15942" LastActivityDate="2016-02-02T20:26:11.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10070" PostTypeId="1" CreationDate="2016-02-02T21:49:39.920" Score="3" ViewCount="3452" Body="&lt;p&gt;I've collected around half a million of unmarked comments from a newspage. The newspage has anti-foreign background. Because of that, a relativley  high number of comments contain hate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have an idea how to start creating a dictionary out of the comments? The dictionary shall contain words, which are connected to hate. With the dictionary I want to perform a text analysis for hate, later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because there is no such dictionary for German language I tried some things. For instance, I checked how often words, which are connected to emotions, appear in the comments. But it turned up, that they seem to be contained as often as any other word. I also thought about a Bag-of-Words and some other things, but I don't know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm doing the analysis with Python 3 and R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kind regards and thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="15946" LastEditorUserId="15946" LastEditDate="2016-02-04T18:57:45.550" LastActivityDate="2017-10-20T12:42:26.087" Title="Sentiment Analysis: Creating dictionary from dataset" Tags="|r|python|dataset|nlp|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10072" PostTypeId="2" ParentId="6547" CreationDate="2016-02-03T06:14:48.873" Score="8" Body="&lt;p&gt;I am currently on same stage like you. I am finding best option for anomaly detection, doing some research.&lt;/p&gt;&#xA;&lt;p&gt;What I have found is I think best matches your need and is better compare to what you have seen. i.e., TwitterAnomalyDetection, SkyLine.&lt;/p&gt;&#xA;&lt;p&gt;I have found better is Numenta's NAB (Numenta Anomaly Benchmark).&#xA;It also have a very good community support and for you plus point is its open source &amp;amp; developed in Python. You can add your algorithm in it.&lt;/p&gt;&#xA;&lt;p&gt;In case of algorithm, I found LOF, or CBLOF are good option.&lt;/p&gt;&#xA;&lt;p&gt;So, &lt;a href=&quot;https://github.com/numenta/nab&quot; rel=&quot;nofollow noreferrer&quot;&gt;check it out&lt;/a&gt; once. It may help you out.&lt;/p&gt;&#xA;&lt;p&gt;If you found better option, please share.&lt;/p&gt;&#xA;" OwnerUserId="15955" LastEditorUserId="98307" LastEditDate="2020-08-03T09:30:58.113" LastActivityDate="2020-08-03T09:30:58.113" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10073" PostTypeId="2" ParentId="10070" CreationDate="2016-02-03T07:34:40.943" Score="4" Body="&lt;p&gt;When looking on the probability of word occurrence, you will get &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow noreferrer&quot;&gt;stop words&lt;/a&gt; and other popular words. You are interested in words that appear more in the comments (assumed hate related) than in normal use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Get a neutral  resource (e.eg., a German newspaper, the German wikipedia, maybe &lt;a href=&quot;https://archive.org/details/google_ngrams-german&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google ngrams for German&lt;/a&gt;). Compute the probability on the neutral source $P_{neutral}(word)$, the probability of the comments $P_{comments}(word)$ and look for the words of lift  $\frac{P_{comments}(word)}{P_{neutral}(word)} &amp;gt; 1$. These are the words that are more popular at the comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @chi wrote, many repositories can both give you a head start and help you tune the needed lift threshold (you might want words that appear much more often in the comments).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this phase you might need do do a finer analysis. For example, I guess that there will be politicians names that will appear more often in the comments. See &lt;a href=&quot;https://datascience.stackexchange.com/questions/10030/algorithm-for-classification-of-words-into-given-categories/10038#10038&quot;&gt;here&lt;/a&gt; for a possible approach.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-02-03T07:34:40.943" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10074" PostTypeId="2" ParentId="10070" CreationDate="2016-02-03T08:25:58.970" Score="2" Body="&lt;p&gt;You could restate your problem as a text classification &quot;hate vs neutral or compassion&quot;. The standard text classification methods then apply. Get yourself a neutral or &quot;compassion&quot; corpus and label their elements as such. Then run a classification learner pipeline. It's features dictionary for the &quot;hate&quot; category will be what you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that does not work out of the box or you don't have contrasting corpus, you could try to emulate the classifier and do the selection manually. Run the texts through a vectorizer with German stopwords, try both TfidfVectorizer and CountVectorizer. Then sort their resulting dictionary by the weight descending and just collect the words manually.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="15361" LastEditDate="2016-02-03T21:22:37.810" LastActivityDate="2016-02-03T21:22:37.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10076" PostTypeId="2" ParentId="10063" CreationDate="2016-02-03T09:55:59.357" Score="5" Body="&lt;p&gt;There is no good evaluation data &lt;em&gt;at all&lt;/em&gt; for clustering that would allow such conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There isn't even good real data where you could say variant 1 of k-means is better than variant 2 of k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also no good evaluation &lt;em&gt;measure&lt;/em&gt; that would handle the notion of &quot;noise&quot; well either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: &lt;strong&gt;don't go by some number&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is about solving a data problem. You have data, you try algorithms and parameters and &lt;em&gt;study&lt;/em&gt; (important) the result. Until you find something &lt;em&gt;interesting&lt;/em&gt;. This cannot be automated, or measured (well, you could ask 100 scientists on what worked for them - i remember having seen e.g. some astronomy papers that used DBSCAN with success - but of course they can't put a number 0 to 1 on this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to look e.g. at this &lt;a href=&quot;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/Density-Based_Clustering#Case_Study:_Identifying_Celestial_Entities&quot; rel=&quot;noreferrer&quot;&gt;Case Study with DBSCAN&lt;/a&gt; - I doubt k-means is of much help on this data set. In particular, it will assign the noise points to &lt;em&gt;some&lt;/em&gt; cluster, when they should not be assigned at all.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2016-02-03T18:15:17.730" LastActivityDate="2016-02-03T18:15:17.730" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10077" PostTypeId="1" AcceptedAnswerId="15497" CreationDate="2016-02-03T10:56:51.447" Score="24" ViewCount="23854" Body="&lt;p&gt;Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already implemented text summarization using standard word-frequency approaches and sentence-ranking, but I'd like to explore the possibility of using deep learning techniques for this task. I have also gone through some implementations given on &lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&quot; rel=&quot;noreferrer&quot;&gt;wildml.com&lt;/a&gt; using Convolutional Neural Networks (CNN) for sentiment analysis; I'd like to know how one could use libraries such as TensorFlow or Theano for text summarization and keyword extraction. Its been about a week since I started experimenting with Neural nets, and I am really excited to see how the performance of these libraries compares to my previous approaches to this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am particularly looking for some interesting papers and github projects related to text summarization using these frameworks. Can anyone provide me with some references?&lt;/p&gt;&#xA;" OwnerUserId="13508" LastEditorUserId="13413" LastEditDate="2016-02-03T17:00:31.027" LastActivityDate="2018-03-09T02:03:26.293" Title="Keyword/phrase extraction from Text using Deep Learning libraries" Tags="|neural-network|text-mining|deep-learning|beginner|tensorflow|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10078" PostTypeId="2" ParentId="10055" CreationDate="2016-02-03T13:27:54.427" Score="2" Body="&lt;p&gt;As I said, maybe there is already a public domain vector file that fits your needs, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(raster)&#xA;map &amp;lt;- getData(&quot;GADM&quot;, country=&quot;HUN&quot;, level=2) &#xA;plot(map, col=rainbow(length(map)), border=NA)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Otherwise, try a dedicated tool to polygonize your raster image and save it as a shapefile. E.g. using QGIS (German): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sA5xc.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sA5xc.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After saving the shapefile, you can load and process it in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(raster)&#xA;map &amp;lt;- shapefile(&quot;evk_106_20140101.shp&quot;)&#xA;plot(map, col=rainbow(length(map)), border=NA)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gujpr.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gujpr.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(Side note: I assume it's ok for any copyright owner oto modify and use the image for demonstrational purposes in the above way. If not, drop a comment and I'll delete it.)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-02-03T13:27:54.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10079" PostTypeId="2" ParentId="10070" CreationDate="2016-02-03T15:26:18.303" Score="2" Body="&lt;p&gt;It is not completely clear whether your dataset has any kind of mark up (like 'comment', 'neutral', 'positive'), and, from my point of view and experience, to get a quite precise dictionary of any kind, you should take human insights as source and stick with supervised learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your dataset does contain such information, you may use Dan Levin's approach which seems quite promising and probability-wise comprehensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively to it, you may use advanced vector-space representations of words (word2vec) in a following manner:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;train a word2vec model on large German text bank. That text shouldn't be of any specific character though you would benefit from presence of all your comment-words in that dataset.&lt;/li&gt;&#xA;&lt;li&gt;then, for particular words in your hate-comments, find &lt;strong&gt;similar&lt;/strong&gt; words  using word2vec model. In that way, given your word2vec text bank will be representative enough, you may get a dictionary even richer that the one people tend to use in comments.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Anyway, keep us posted on achieved results :)&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-02-03T15:26:18.303" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10080" PostTypeId="1" CreationDate="2016-02-03T20:25:08.253" Score="2" ViewCount="121" Body="&lt;p&gt;I am trying to learn Holt Winters exponential smoothing. In the algorithm there are three indices involved (level, trend, seasonality) while forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;What is the interpretation of these 3 indices?&#xA;How do I differentiate between different values of these indices? For example, what is the difference between level 100 and level 200 or what is the difference between trend 2 and trend 5?&#xA;How do we initiate these values?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help will be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2016-02-03T20:25:08.253" Title="Interpretation of level, trend and seasonal indices in holt winters exponential smoothing" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10081" PostTypeId="1" CreationDate="2016-02-03T20:36:41.050" Score="2" ViewCount="494" Body="&lt;p&gt;In follow up to my &lt;a href=&quot;https://datascience.stackexchange.com/questions/9886/classifying-survey-response-text-svm&quot;&gt;previous question&lt;/a&gt;, I was able to label survey responses based on 70 categories using RTextTools. Each survey gets one label while if manually coded some would have up to 8 labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good news is my training data set does have multi-labels. So my question is, would be possible to train on 8 labels all at once? And then on the test data, get probabilities associated with each label and sort based on probability to get the top 8 labels?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or can anyone suggest a better approach?&lt;/p&gt;&#xA;" OwnerUserId="15609" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-02-04T14:19:17.643" Title="RTextTools multi-label classification" Tags="|machine-learning|r|text-mining|multilabel-classification|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10082" PostTypeId="2" ParentId="10053" CreationDate="2016-02-03T20:42:24.000" Score="1" Body="&lt;p&gt;You should be using a combination of 2) and 3) in your scenario above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 2) use JSON as it allows you to construct an arbitrary structure on the fly with as many columns as you need without tying yourself to a fixed schema. JSON also has wide language adoption (Python, R, Java, Scala eg).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 3) I agree that MongoDB is the simplest as it is designed to be used for JSON.&lt;/p&gt;&#xA;" OwnerUserId="8762" LastActivityDate="2016-02-03T20:42:24.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10083" PostTypeId="1" CreationDate="2016-02-04T01:46:37.107" Score="2" ViewCount="80" Body="&lt;p&gt;I'm new to machine learning but am interested in decoding applications like the Higgs Boson challenge on Kaggle. I just wondered, what are the best algorithms or approaches for classifying such a small signal in large amounts of noise? I think supervised and unsupervised are equally fine to use for my project, but hearing about your thoughts in general would help a lot! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to start with the Higgs Boson challenge to explore machine learning methods for improving discovery significance of scientific data. The experiment and data are available at &lt;a href=&quot;https://www.kaggle.com/c/higgs-boson&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/higgs-boson&lt;/a&gt; . The code used by the winner was released here: &lt;a href=&quot;https://www.kaggle.com/c/higgs-boson/forums/t/10425/code-release&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/higgs-boson/forums/t/10425/code-release&lt;/a&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the basic theory of supervised and unsupervised learning, and have studied and used Linear Regression and SVM. I've read about neural networks also but on a basic approach level. I'm just learning about deep learning coming from the natural sciences, to improve data analysis, and found SVM very useful but otherwise don't know more advanced techniques. &lt;/p&gt;&#xA;" OwnerUserId="15980" LastEditorUserId="11097" LastEditDate="2016-02-05T05:16:01.630" LastActivityDate="2016-02-05T05:16:01.630" Title="Classification in noise" Tags="|machine-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10084" PostTypeId="1" CreationDate="2016-02-04T05:39:51.250" Score="2" ViewCount="73" Body="&lt;p&gt;I am trying to find the top 10 useful item recommendation. Items are divided into categories and then top10 in each category is calculated.&#xA;There are six features based on which a score is assigned to each item and then they are sorted in decreasing order to get the top 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure are these the best top items? How to validate the output? I need to see which feature is dominating the most in calculating the score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already calculated correlation, any other statistical measure?&lt;/p&gt;&#xA;" OwnerUserId="13046" LastEditorUserId="13046" LastEditDate="2016-02-05T07:08:59.497" LastActivityDate="2016-02-06T17:40:47.583" Title="Validate output" Tags="|regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10085" PostTypeId="1" AcceptedAnswerId="10087" CreationDate="2016-02-04T08:43:12.847" Score="10" ViewCount="1884" Body="&lt;p&gt;Which of the below set of steps options is the correct one when creating a predictive model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First eliminate the most obviously bad predictors, and preprocess the remaining if needed, then train various models with cross-validation, pick the few best ones, identify the top predictors each one has used, then retrain those models with those predictors only and evaluate accuracy again with cross-validation, then pick the best one and train it on the full training set using its key predictors and then use it to predict the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First eliminate the most obviously bad predictors, then preprocess the remaining if needed, then use a feature selection technique like recursive feature selection (eg. RFE with rf ) with cross-validation for example to identify the ideal number of key predictors and what these predictors are, then train different model types with cross-validation and see which one gives the best accuracy with those top predictors identified earlier. Then train the best one of those models again with those predictors on the full training set and then use it to predict the test set.&lt;/p&gt;&#xA;" OwnerUserId="15984" LastEditorUserId="20381" LastEditDate="2016-06-10T15:02:25.400" LastActivityDate="2016-06-10T15:02:25.400" Title="Machine Learning Steps" Tags="|machine-learning|predictive-modeling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10086" PostTypeId="2" ParentId="10050" CreationDate="2016-02-04T09:52:53.810" Score="4" Body="&lt;p&gt;I believe this is the representation you're after, please excuse the rough sketch but I think it explains the structure appropriately.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Single input going to three hidden units, each with a bias and tansig activation.&lt;/li&gt;&#xA;&lt;li&gt;The outputs of the hidden layer are summed (via linear activation) with a bias to produce the output.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CdMAf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CdMAf.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5144" LastEditorUserId="5144" LastEditDate="2016-02-04T09:58:07.610" LastActivityDate="2016-02-04T09:58:07.610" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10087" PostTypeId="2" ParentId="10085" CreationDate="2016-02-04T10:27:20.423" Score="17" Body="&lt;p&gt;I found both of your options slightly faulty. So, this is generally (very broadly) how a predictive modelling workflow looks like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data Cleaning&lt;/strong&gt;: Takes the most time, but every second spent here is worth it. The cleaner your data gets through this step, the lesser would your total time spent would be. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Splitting the data set&lt;/strong&gt;: The data set would be splitted into training and testing sets, which would be used for the modelling and prediction purposes respectively. In addition, an additional split as a cross-validation set would also need to be done.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Transformation and Reduction:&lt;/strong&gt; Involves processes like transformations, mean and median scaling, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature Selection:&lt;/strong&gt; This can be done in a lot of ways like threshold selection, subset selection, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Designing predictive model:&lt;/strong&gt; Design the predictive model on the training data depending on the features you have at hand.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Cross Validation:&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Final Prediction, Validation&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-03-13T14:09:37.667" LastActivityDate="2016-03-13T14:09:37.667" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="10090" PostTypeId="1" CreationDate="2016-02-05T07:31:09.190" Score="2" ViewCount="64" Body="&lt;p&gt;I have to find the best 10 items from the set of items for x number of given features. I don't know what the best recommendation will be. User data is not available to validate it. After reading a number of papers I came to know about voting algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to implement something like this-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find x number of features for static variables( that doesn't change over time), dynamic variables (that change very frequently).&lt;/li&gt;&#xA;&lt;li&gt;Assume some weights for the weights on the basis of requirement for dynamic variables.&lt;/li&gt;&#xA;&lt;li&gt;Try to come up with 3-4 regression models by combining the static and dynamic variables and find the best 10 items from each&lt;/li&gt;&#xA;&lt;li&gt;Take the common one out, and then calculate the error.&lt;/li&gt;&#xA;&lt;li&gt;Back-propagate the error to get the new weights.&lt;/li&gt;&#xA;&lt;li&gt;Keep on iterating till we get at-least common top 8 elements.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is the approach right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any guidance for what to take as error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach to error calculation was-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Mean of common items feature values (X).&lt;/li&gt;&#xA;&lt;li&gt;For each model MSE where mean is X and Xi is feature values for uncommon items of that model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am new to it. Can anyone please guide me through it.&lt;/p&gt;&#xA;" OwnerUserId="13046" LastActivityDate="2016-02-05T07:31:09.190" Title="Modified Voting Algorithm to find the best recommendation" Tags="|regression|backpropagation|error-handling|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10091" PostTypeId="1" AcceptedAnswerId="10092" CreationDate="2016-02-05T07:50:28.437" Score="6" ViewCount="9280" Body="&lt;p&gt;I have a set of images that are considered as good quality image and other set that are considered as bad quality image. I have to train a classification model so that any new image can be said good/bad.&#xA;SVM seems to be the best approach to do it. I know how to do it in MATLAB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But,can anyone suggest how to do it in python? What are the libraries?&#xA;For SVM scikit is there, what about feature extraction of image and PCA?&lt;/p&gt;&#xA;" OwnerUserId="13046" LastEditorUserId="11097" LastEditDate="2016-02-05T08:19:06.823" LastActivityDate="2018-03-09T03:05:24.763" Title="Image classification in python" Tags="|python|image-classification|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10092" PostTypeId="2" ParentId="10091" CreationDate="2016-02-05T08:17:26.803" Score="3" Body="&lt;p&gt;As this question highly overlaps with a similar question I have already answered, I would include that answer here (linked in the comments underneath the question):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In images, some frequently used techniques for feature extraction are &lt;strong&gt;binarizing&lt;/strong&gt; and &lt;strong&gt;blurring&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binarizing:&lt;/strong&gt; converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how you do it in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import Image&#xA;&#xA;%matplotlib inline  &#xA;&#xA;#Import an image&#xA;image = Image.open(&quot;xyz.jpg&quot;)&#xA;&#xA;image&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example Image: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mkf97.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mkf97.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, convert into gray-scale:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;im = image.convert('L')&#xA;&#xA;im&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will return you this image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AGxy6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AGxy6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the matrix can be seen by running this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array(im)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The array would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[213, 213, 213, ..., 176, 176, 176],&#xA;       [213, 213, 213, ..., 176, 176, 176],&#xA;       [213, 213, 213, ..., 175, 175, 175],&#xA;       ..., &#xA;       [173, 173, 173, ..., 204, 204, 204],&#xA;       [173, 173, 173, ..., 205, 205, 204],&#xA;       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, use a histogram plot and/or a contour plot to have a look at the image features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pylab import *&#xA;&#xA;# create a new figure&#xA;figure()&#xA;gray()&#xA;# show contours with origin upper left corner&#xA;contour(im, origin='image')&#xA;axis('equal')&#xA;axis('off')&#xA;&#xA;&#xA;figure()&#xA;&#xA;&#xA;hist(im_array.flatten(), 128)&#xA;&#xA;show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would return you a plot, which looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/56A0K.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/56A0K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/tWpNy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tWpNy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Blurring:&lt;/strong&gt; Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is how you do it in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import *&#xA;&#xA;&#xA;figure()&#xA;p = image.convert(&quot;L&quot;).filter(ImageFilter.GaussianBlur(radius = 2))&#xA;p.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the blurred image is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0Dx8q.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0Dx8q.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The entire analytics is done with the &lt;a href=&quot;http://www.pythonware.com/products/pil/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PIL package&lt;/a&gt;. I wouldn't claim that it's a one-stop shop for Image analytics, but for a starter to novice level, it is pretty much it.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-02-05T16:16:11.363" LastActivityDate="2016-02-05T16:16:11.363" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10093" PostTypeId="1" AcceptedAnswerId="13184" CreationDate="2016-02-05T13:52:08.660" Score="7" ViewCount="33152" Body="&lt;p&gt;I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence level of a given value of that population. In other words, given a a value of the population, I would like to know if it is within 95% (confidence level) of the whole population or 68% or 50% and so on. Usually, we can find the z value and confidence interval and given confidence level as explained here &lt;a href=&quot;http://www.wikihow.com/Calculate-Confidence-Interval&quot; rel=&quot;noreferrer&quot;&gt;How to find the confidence Interval&lt;/a&gt;. But I would like to find the confidence level given the z value (which in this case is a given value from the population).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I tackle this? If possible it should be in python or in R&lt;/p&gt;&#xA;" OwnerUserId="10240" LastActivityDate="2020-04-22T10:07:43.387" Title="How to find a confidence level given the z value" Tags="|r|python|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10095" PostTypeId="2" ParentId="10093" CreationDate="2016-02-05T15:23:40.550" Score="1" Body="&lt;p&gt;To convert between z-scores and confidence values with python, use the &lt;code&gt;cdf&lt;/code&gt; and &lt;code&gt;ppf&lt;/code&gt; functions in &lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.norm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy.stats.norm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a nice example of how to use them in the answer for &lt;a href=&quot;https://stackoverflow.com/questions/20864847/probability-to-z-score-and-vice-versa-in-python&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="964" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-02-05T19:03:20.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10096" PostTypeId="2" ParentId="9753" CreationDate="2016-02-05T15:35:00.860" Score="3" Body="&lt;p&gt;This is the typical value function of Reinforcement Learning. The discount factor evaluates the importance of the accumulated future events in your current value. The smaller the number, the less important are the future events in the current action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually this number is selected heuristically. I usually select 0.9. If I don't want any discount then I would select 1.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-02-05T15:35:00.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10099" PostTypeId="1" CreationDate="2016-02-05T21:26:05.873" Score="0" ViewCount="328" Body="&lt;p&gt;I have a normal distribution and I would like to know what is the confidence level of mu+0.1sigma (where mu is the mean and sigma the standard deviation). How do i get the confidence level?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using python or R&lt;/p&gt;&#xA;" OwnerUserId="10240" LastActivityDate="2016-02-06T12:32:22.060" Title="How to get the mu+0.1sigma from a normal distribution?" Tags="|r|python|statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10100" PostTypeId="1" AcceptedAnswerId="10101" CreationDate="2016-02-06T02:38:20.750" Score="6" ViewCount="2652" Body="&lt;p&gt;After setting up a &lt;a href=&quot;https://gsamaras.wordpress.com/code/hadoop-cluster-with-pc-and-virtualbox/&quot; rel=&quot;nofollow noreferrer&quot;&gt;2-noded Hadoop cluster&lt;/a&gt;, understanding &lt;a href=&quot;http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hadoop and Python&lt;/a&gt; and based on this naive &lt;a href=&quot;https://codereview.stackexchange.com/questions/80050/k-means-clustering-algorithm-in-python&quot;&gt;implementation&lt;/a&gt;, I ended up with this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def kmeans(data, k, c=None):&#xA;    if c is not None:&#xA;        centroids = c&#xA;    else:&#xA;        centroids = []&#xA;        centroids = randomize_centroids(data, centroids, k)&#xA;&#xA;    old_centroids = [[] for i in range(k)] &#xA;&#xA;    iterations = 0&#xA;    while not (has_converged(centroids, old_centroids, iterations)):&#xA;        iterations += 1&#xA;&#xA;        clusters = [[] for i in range(k)]&#xA;&#xA;        # assign data points to clusters&#xA;        clusters = euclidean_dist(data, centroids, clusters)&#xA;&#xA;        # recalculate centroids&#xA;        index = 0&#xA;        for cluster in clusters:&#xA;            old_centroids[index] = centroids[index]&#xA;            centroids[index] = np.mean(cluster, axis=0).tolist()&#xA;            index += 1&#xA;&#xA;&#xA;    print(&quot;The total number of data instances is: &quot; + str(len(data)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tested it for serial execution and it is OK. How to make it distributed in Hadoop? In other words, what should go to the reducer and what to the mapper?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Please&lt;/em&gt; note that &lt;em&gt;if&lt;/em&gt; possible, I would like to follow the tutorial's style, since it's something I have understood.&lt;/p&gt;&#xA;" OwnerUserId="15927" LastEditorUserId="-1" LastEditDate="2017-04-13T12:41:09.900" LastActivityDate="2020-08-11T08:46:37.487" Title="How to make k-means distributed?" Tags="|python|apache-hadoop|k-means|map-reduce|distributed|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10101" PostTypeId="2" ParentId="10100" CreationDate="2016-02-06T03:14:11.303" Score="3" Body="&lt;p&gt;Unless you are trying to do this as a learning exercise, just use Spark which has ML libraries made for distributed computing. See &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10384" LastEditorUserId="98307" LastEditDate="2020-08-11T08:46:37.487" LastActivityDate="2020-08-11T08:46:37.487" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10102" PostTypeId="2" ParentId="10099" CreationDate="2016-02-06T12:32:22.060" Score="1" Body="&lt;p&gt;From your question, I imagine you are interested in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\alpha = P[\mu - k \, \sigma \le x \le \mu + k \, \sigma]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where, in your case, $k = 0.1$. In other words, by &quot;confidence level&quot; you mean alpha.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, you can obtain that via function &lt;code&gt;pnorm&lt;/code&gt; (empirical cumulative distribution function for the standardized normal distribution):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;alpha &amp;lt;- 2 * (pnorm(k) - 0.5)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which, for &lt;code&gt;k = 0.1&lt;/code&gt; produces 8%.&lt;/p&gt;&#xA;" OwnerUserId="12290" LastActivityDate="2016-02-06T12:32:22.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10103" PostTypeId="1" AcceptedAnswerId="28772" CreationDate="2016-02-06T14:19:10.243" Score="28" ViewCount="21888" Body="&lt;p&gt;I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(&lt;code&gt;doc2vec&lt;/code&gt;). &#xA;I am using &lt;code&gt;TSNE&lt;/code&gt; implementation from &lt;code&gt;sklearn.manifold&lt;/code&gt; module for it and the major problem is time complexity. Even with &lt;code&gt;method = barnes_hut&lt;/code&gt;, the speed of computation is still low. Some time even it runs out of Memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running it on a 48 core processor with 130G RAM. Is there a method to run it parallely or make use of the plentiful resource to speed up the process. &lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="11097" LastEditDate="2016-02-06T14:23:23.533" LastActivityDate="2020-08-06T13:00:54.440" Title="Improve the speed of t-sne implementation in python for huge data" Tags="|python|bigdata|nlp|scikit-learn|dimensionality-reduction|" AnswerCount="5" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="10106" PostTypeId="1" CreationDate="2016-02-06T17:34:16.887" Score="3" ViewCount="668" Body="&lt;p&gt;I have a file that contains words. I have to calculate cosine similarity of each word with every other words. For that I used the following formula in gensim:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;model.most_similar([word])&lt;/li&gt;&#xA;&lt;li&gt;model.save('text.txt')&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to save the result of cosine-similarity of each words with respective to other words in such a way that I can used them further in classification and clustering. But don't know how to retrieve the results which are saved using gensim model ?&lt;/p&gt;&#xA;" OwnerUserId="16031" LastEditorUserId="8878" LastEditDate="2016-02-12T11:14:16.117" LastActivityDate="2016-11-08T17:53:44.523" Title="How to retrive the results saved in model of gensim?" Tags="|text-mining|word-embeddings|gensim|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10107" PostTypeId="2" ParentId="10084" CreationDate="2016-02-06T17:40:47.583" Score="3" Body="&lt;p&gt;There are a lot of ways to see how well a recommender works and I think it really depends on your final goal.  A list of &lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank&quot; rel=&quot;nofollow&quot;&gt;evaluation metrics&lt;/a&gt; are here that might be useful to looks through. Although a measure like Accuracy/Precision/RMSE might suit your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, lets say you have a year worth of historical sales.  You can build your system on the first 8 months of sales, then for each following month you can see what products would be recommended in that month and what products were actually purchased in that month.  So within that you can get an idea of how well it is performing (of the 5 products someone bought, we had all 5 in the  in the spots 1,2,6,8,9 of the top-10 list).  You would want the product purchases to be closer to the top, which is why ranking metrics could be useful for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course there is a lot more to consider, but I think this is a good place to start. Are you trying to increase the pure number of sales or the revenue? Do certain products sell more depending on seasons/holidays?  Are you using ratings or just associated sales (how do you recommend new products?).  Factors like these can change your recommender and how you would want to evaluate it.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-02-06T17:40:47.583" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10108" PostTypeId="1" AcceptedAnswerId="10117" CreationDate="2016-02-06T18:29:08.427" Score="8" ViewCount="6944" Body="&lt;p&gt;I presently receive files from a device in a semi-csv format. I have a written a simple recursive descent parser for getting information out of these files. Every time the device updates firmware, I have a new version of the parser for the changes the update brings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Down the road, we will be taking data from other devices, which means another parser and more updates to firmware. I'm wondering if I could define a basic structure of &quot;this is the data I need&quot; and use a neural network to get the parsed data without having to write a parser for each new file type that comes in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a pipe dream or is it a valid application of machine learning? I'm much more of a software engineer than I am a data scientist, but I'm starting to dip my toes into the machine learning realm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="16032" LastActivityDate="2018-01-30T06:56:42.813" Title="Is parsing files an application of machine learning?" Tags="|machine-learning|parsing|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10110" PostTypeId="2" ParentId="10091" CreationDate="2016-02-06T20:52:06.760" Score="0" Body="&lt;p&gt;Can you tell us which criteria do you use for considering a picture &quot;good quality image&quot; or &quot;bad quality image&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An idea of how to handle it could be for example choosing unfocused pictures as bad ones. One way to discriminate it could be compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Sobel_operator&quot; rel=&quot;nofollow&quot;&gt;Sobel operator&lt;/a&gt; over the image in order to obtain the edges. A blurry image will contain less edges than a good one, but also it will depend on the type of image (it's not the same a landscape with just the sea and the sand than a picture of a table full of stuff), so you will need to normalize your image, but no idea yet how to handle this normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion on libraries for python is &lt;a href=&quot;https://opencv-python-tutroals.readthedocs.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;OpenCv&lt;/a&gt;. It's documentation for Python language was awful when I tried to use some years ago, but surely it has improved a lot since then.&lt;/p&gt;&#xA;" OwnerUserId="16040" LastActivityDate="2016-02-06T20:52:06.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10112" PostTypeId="1" AcceptedAnswerId="10124" CreationDate="2016-02-06T22:17:04.323" Score="4" ViewCount="1974" Body="&lt;p&gt;I apologize for lack of terminology, I'm no computer scientist.&#xA;I have a problem of validating paths in a directed graph with complex nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full description is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have a decent set (about 1K) of directed graphs;&lt;/li&gt;&#xA;&lt;li&gt;Each node contains a complex data structure (it is a hierarchical data structure, not a picture or sound);&lt;/li&gt;&#xA;&lt;li&gt;I have some of paths in those graphs known as &quot;correct&quot; paths (based&#xA;mostly on data in nodes);&lt;/li&gt;&#xA;&lt;li&gt;And I have some of paths in those graphs known as &quot;incorrect&quot; paths&#xA;(with classification why it is incorrect).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd like to predict given a graph with those complex nodes and a path, is this path &quot;correct&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which machine learning algorithm will suit me best?&#xA;In general, what approach I should use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Each full graph is either have app paths processed&#xA;(correct/incorrect) or completely blank (no path is processed);&lt;/li&gt;&#xA;&lt;li&gt;Correctness depends on both position of node in a graph AND data in&#xA;the node;&lt;/li&gt;&#xA;&lt;li&gt;Humans would need heuristics to decide or guess which paths are&#xA;correct;&lt;/li&gt;&#xA;&lt;li&gt;Most of the paths are &quot;correct&quot;;&lt;/li&gt;&#xA;&lt;li&gt;I hope to convert human heuristics to some kind of &quot;correctness&quot;&#xA;recognition.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16042" LastEditorUserId="16042" LastEditDate="2016-02-07T20:12:55.557" LastActivityDate="2019-04-15T18:28:42.470" Title="Which machine learning approach/algorithm do I choose for path validation?" Tags="|machine-learning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10113" PostTypeId="1" CreationDate="2016-02-07T02:55:46.907" Score="2" ViewCount="759" Body="&lt;p&gt;I am new to ML and learnt a lot from your valuable posts. I need your advise with the following situation and guidance on if the steps make sense.&#xA;I have a binary classification problem, my dataset has a severe imbalance approximately 2% positive cases (4,000 cases) out of a total of 200,000 cases. I separated my dataset into a train and a test (80/20 stratified split). My train now has total of 160,000 cases (3,200 positive cases) and test has total of 40,000 (800 positive cases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, from Train I created a (50-50) SMOTE sample which has ~9,000 positive cases (original 3,200 and SMOTE create synthetic samples of ~ 5,800 positives) and ~ 9,000 negative cases (so a total of 18,000 cases in the &quot;new train&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;next, I have developed bagged/ boosted tree based classifiers on the &quot;new train&quot; with ROC training criteria in R caret (I get high ROC around 0.9 which is usual). I then applied the model/s on the test set which has the original class imbalance and got the predicted probabilities (the AUC is now 0.65). I identified the optimal threshold for classification based on the pROC package for creation of the final class predictions on test dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need your advise on if I am transitioning from the trained SMOTE model with a 50/50 imbalance to the test model with the original imbalance of 50/1 correctly. is there anything fundamentally wrong in what I am doing and any suggestions on how to improve the process will be very helpful. are any corrections needed to the SMOTE based probabilities on test or is thresholding the best I can do?&lt;/p&gt;&#xA;" OwnerUserId="16046" LastEditorUserId="1330" LastEditDate="2022-04-18T18:45:39.923" LastActivityDate="2022-04-18T18:45:39.923" Title="class imbalance - applied SMOTE - next steps" Tags="|r|class-imbalance|smote|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10114" PostTypeId="1" CreationDate="2016-02-07T04:25:04.640" Score="0" ViewCount="1004" Body="&lt;p&gt;Where I can get the complete guide (step by step )on building a recommender system for example using movielens datsets building content based, collaborative or may be hybrid system.&lt;/p&gt;&#xA;" OwnerUserId="16048" LastActivityDate="2016-02-07T11:51:18.740" Title="using movielens dataset build recommendation engine" Tags="|recommender-system|" AnswerCount="1" CommentCount="1" ClosedDate="2016-02-08T14:51:18.653" ContentLicense="CC BY-SA 3.0" />
  <row Id="10115" PostTypeId="1" CreationDate="2016-02-07T04:36:41.013" Score="2" ViewCount="422" Body="&lt;p&gt;i create a decision tree model using c4.5 algorithm. After create the model, i evaluate model using 10 fold cross validation and classify model using test data to get accuracy. And then I run prune tree with REP. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what data should i use to compare accuracy (after prune and before prune) to decide to remove the leaf? and from where i get the accuracy? using cross validation or classify test data?&lt;/p&gt;&#xA;" OwnerUserId="14647" LastActivityDate="2016-03-11T13:32:10.947" Title="Pruning tree using REP" Tags="|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10117" PostTypeId="2" ParentId="10108" CreationDate="2016-02-07T09:31:46.453" Score="6" Body="&lt;p&gt;I would answer the question at two levels. The first level is &quot;&lt;strong&gt;can it be done using machine learning&lt;/strong&gt;?&quot; I would say that machine learning is essentially about learning. So given that you prepare sufficient examples of sample documents and the output to expect from those documents, you can train a network to learn the structure of documents and extract the relevant information. The more general form of extracting information from documents is a well-researched problem and is more commonly known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_retrieval&quot;&gt;Information Retrieval&lt;/a&gt;. And it is not limited to just machine learning techniques, you can use Natural Language Processing tools as well. So, in its general form, it is actually being done in practice. Coming to the second level, &quot;&lt;strong&gt;should you be doing it using machine learning&lt;/strong&gt;?&quot;. I would agree to what @NeilSlater said. The better and more feasible approach would be to use good programming practices so that you can reuse parts of your parser as your dataset evolves.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-02-07T09:31:46.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10118" PostTypeId="2" ParentId="10114" CreationDate="2016-02-07T11:51:18.740" Score="2" Body="&lt;p&gt;There are plenty of ways to realize this. Originally everybodies approach to the Netflix Prize (quite similar to the movielense dataset) were nearest neighbor methods. Their intuition should be fairly clear. Matrix Factorization became the next big thing and was nicely described by Simon Funk in his journal: &lt;a href=&quot;http://sifter.org/~simon/journal/20061211.html&quot; rel=&quot;nofollow&quot;&gt;http://sifter.org/~simon/journal/20061211.html&lt;/a&gt; . If you are more interested in these two topics and don't mind a bit of bad english you may consider this thesis: &lt;a href=&quot;http://patrick-ott.de/dl/diploma_ott.pdf&quot; rel=&quot;nofollow&quot;&gt;http://patrick-ott.de/dl/diploma_ott.pdf&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crowd then went on to Restricted Boltzmann Machines for the task, e.g. (cs.utoronto.ca/~hinton/absps/netflixICML.pdf) . I think it is beyond this answer to explain these for this particular task -- maybe worth another question,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All three approaches though (Nearest Neighbor, Matrix Factorization and RBMs) never we are able to reach to goal set by Netflix, which let to the development of more or less clever methods to combine their predictions, which finally was enough to get to the point Netflix set out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are all somewhat old approaches to the problem, their might be newer ones. However, the most recent machine learning hype, i.e. Deep Learning has so far not given many possible solutions although dominating in other fieds.&lt;/p&gt;&#xA;" OwnerUserId="16041" LastActivityDate="2016-02-07T11:51:18.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10119" PostTypeId="1" CreationDate="2016-02-07T12:42:49.030" Score="7" ViewCount="513" Body="&lt;p&gt;I have a 2M instances dataset with millions of very very sparse dummy variables created using the &lt;code&gt;hashing trick&lt;/code&gt; = &lt;code&gt;hash(orig_feature_name + orig_feature_value)=1&lt;/code&gt;. Note that the data is sparse both on rows (every instance has only a limited &amp;lt;100 features=1) and on columns (most features are relevant only to very few instances &amp;lt; 1%) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I discovered that in such sparse scenarios the &lt;code&gt;follow-the-regularized-leader FTRL proximal gradient descent&lt;/code&gt; is very popular:&#xA;&lt;a href=&quot;http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, &lt;a href=&quot;https://github.com/mlandry22/kaggle/blob/master/modern_logistic_regression/fast_solution_v3.py&quot; rel=&quot;nofollow&quot;&gt;reference implementation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'm not sure why shouldn't I prefer a &lt;code&gt;batch&lt;/code&gt; gradient descent algorithm? FTRL for all its merits is still an &lt;code&gt;online-learning&lt;/code&gt; algorithm that sees one instance at a time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what are the advantages and disadvantages of using FTRL vs. a well known sparse least squares algorithm such as &lt;code&gt;LSQR&lt;/code&gt; (&lt;a href=&quot;https://web.stanford.edu/class/cme324/paige-saunders2.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, &lt;a href=&quot;https://github.com/scipy/scipy/blob/master/scipy/sparse/linalg/isolve/lsqr.py&quot; rel=&quot;nofollow&quot;&gt;reference implementation&lt;/a&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is that if possible to use all the data for each iteration, we should do it, but I'm not sure... &lt;/p&gt;&#xA;" OwnerUserId="16050" LastEditorUserId="83275" LastEditDate="2019-11-14T13:53:06.917" LastActivityDate="2019-11-14T13:53:06.917" Title="differences between LSQR and FTRL when working with very sparse data" Tags="|machine-learning|linear-regression|gradient-descent|online-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10120" PostTypeId="1" CreationDate="2016-02-07T15:34:18.730" Score="2" ViewCount="740" Body="&lt;p&gt;Has anyone attempted Principal component analysis on time series and acceleration data(ie. data from accelerometer and sensors) and tried compressing it as well as regenerating back the data with minimum error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to know if this is possible using PCA. Kindly suggest alternate methods available to compress and regenrate my data.&lt;/p&gt;&#xA;" OwnerUserId="16058" LastActivityDate="2016-02-08T04:53:55.723" Title="PCA on acceleration time series data" Tags="|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10121" PostTypeId="1" CreationDate="2016-02-07T16:25:39.260" Score="1" ViewCount="489" Body="&lt;p&gt;I use logistic regression. We know that it is a supervised method and needs calculated feature values both in training and test data. There are six features. Although the functions produce these features’ values are different and their maximum value can be 1, there are four features (both in training and test data) that have very low values. e.g. they range between 0 and 0.1 and are never 1, even more than 0.1!!!. Thus these features’ values are very close to each other. Other features are distributed normally (they range between 0 and 0.9). So the difference between these two kinds of features is high, I think this causes trouble in learning process for logistic regression. Am I right?! Does it need any transforming/normalizing these features? Any help would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="9218" LastActivityDate="2016-02-07T17:38:05.697" Title="features' range in logistic regression" Tags="|machine-learning|classification|logistic-regression|feature-selection|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10122" PostTypeId="2" ParentId="10121" CreationDate="2016-02-07T17:38:05.697" Score="3" Body="&lt;p&gt;You do not &lt;em&gt;have&lt;/em&gt; to normalise features used in logistic regression, but sometimes it can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should normalise features used in logistic regression, if you are using a gradient-based optimiser (e.g. SGD) to find the optimum weights. That is because the optimiser will perform better when partial derivatives of the cost function are of similar magnitude in each direction. When the derivatives vary too much, you will need a lower learning rate to compensate (making learning slower, and more likely to get stuck) or the optimiser will not converge - it may oscillate or start to diverge instead.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-02-07T17:38:05.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10124" PostTypeId="2" ParentId="10112" CreationDate="2016-02-07T18:47:21.247" Score="3" Body="&lt;p&gt;This is a good question but it's rather complicated. I can suggest two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Graphical_model&quot; rel=&quot;nofollow&quot;&gt;Graphical models&lt;/a&gt;; specifically &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow&quot;&gt;Bayesian networks&lt;/a&gt; since your graph is directed.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow&quot;&gt;Recurrent neural networks&lt;/a&gt;. Here's a talk on popular recent model: &lt;a href=&quot;http://research.microsoft.com/apps/video/?id=239083&quot; rel=&quot;nofollow&quot;&gt;Sequence to Sequence Learning with Neural Networks&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-02-07T20:31:02.837" LastActivityDate="2016-02-07T20:31:02.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10126" PostTypeId="2" ParentId="731" CreationDate="2016-02-07T20:48:02.950" Score="8" Body="&lt;p&gt;You might be interested in reading the following paper by researchers of Microsoft Research:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: &lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot;&gt;Deep Residual Learning for Image Recognition&lt;/a&gt; on arxiv, 2015.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They had similar problems as you had:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When deeper networks are able to start converging, a&#xA;  degradation problem has been exposed: with the network&#xA;  depth increasing, accuracy gets saturated (which might be&#xA;  unsurprising) and then degrades rapidly. &lt;strong&gt;Unexpectedly,&#xA;  such degradation is not caused by overfitting, and adding&#xA;  more layers to a suitably deep model leads to higher training&#xA;  error&lt;/strong&gt;, as reported in [11, 42] and thoroughly verified by&#xA;  our experiments.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To solve the problem, they have made use of a skip architecture. With that, they trained very deep networks (1202 layers) and achieved the best result in the ILSVRC 2015 challenge.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-02-07T20:48:02.950" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10127" PostTypeId="1" CreationDate="2016-02-07T20:55:57.947" Score="7" ViewCount="488" Body="&lt;p&gt;Restricted Boltzmann machines are stochastic neural networks. The neurons form a complete bipartite graph of visible units and hidden units. The &quot;restricted&quot; is exactly the bipartite property: There may not be a connection between any two visible units and there may not be a connection between two hidden units.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Restricted Boltzmann machines are trained with Contrastive Divergence (CD-k, see &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;noreferrer&quot;&gt;A Practical Guide to Training&#xA;Restricted Boltzmann Machines&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I wonder: How are non-restricted Boltzmann Machines trained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I google for &quot;Boltzmann Machine&quot;, I only finde RBMs.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2019-10-24T16:42:06.270" Title="How are non-restricted Boltzmann machines trained?" Tags="|machine-learning|rbm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10128" PostTypeId="5" CreationDate="2016-02-07T20:58:14.257" Score="0" Body="&lt;p&gt;A restricted Boltzmann maschine (RBM) is a stochastic neural network. RBMs can be stacked to deep networks.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scholarpedia.org/article/Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;Scholarpedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-08T05:02:12.687" LastActivityDate="2016-02-08T05:02:12.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10129" PostTypeId="4" CreationDate="2016-02-07T20:58:14.257" Score="0" Body="A restricted Boltzmann machine (RBM) is a stochastic neural network." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-08T05:01:31.233" LastActivityDate="2016-02-08T05:01:31.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10130" PostTypeId="1" CreationDate="2016-02-07T22:27:42.207" Score="2" ViewCount="334" Body="&lt;p&gt;I want to predict future user activities (e.g., account cancellation) but I don't know how I'm supposed to represent the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The raw data is a sequence of all activities by all users:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2015-01-01T04:04|user1|created account&#xA;2015-01-01T05:04|user2|created account&#xA;2015-01-01T06:04|user1|changed plan&#xA;2015-01-01T07:04|user2|changed plan&#xA;2015-01-01T08:04|user1|cancelled submission&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the simplest prediction I'd like to label a user as 0=Cancelled and 1=Not Cancelled and subsequently would train a classifier for this label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how would I prepare the data for a machine learning algorithm (e.g. SVM or logistic regression)?. I now have multiple rows per user which are ordered by the time they happened, but I'm thinking of having only one row per user. How would I represent the activities? Furthermore, the number of activities per user is not limited (1 to &lt;code&gt;n&lt;/code&gt; where &lt;code&gt;n&lt;/code&gt; would hardly ever exceed &lt;code&gt;10&lt;/code&gt;), and the the number of features per activity can be large (up to 100 features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd really appreciate it if anyone could help me out here :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thx&lt;/p&gt;&#xA;" OwnerUserId="16068" LastActivityDate="2016-02-07T22:27:42.207" Title="User activity representation for Prediction/ML" Tags="|machine-learning|preprocessing|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10131" PostTypeId="2" ParentId="9753" CreationDate="2016-02-07T22:28:31.130" Score="1" Body="&lt;p&gt;Selecting the discount factor &lt;span class=&quot;math-container&quot;&gt;$\gamma$&lt;/span&gt; depends on the problem. As explained by &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton &amp;amp; Barto&lt;/a&gt; the value is always between 0 and 1: &lt;span class=&quot;math-container&quot;&gt;$0&amp;lt;=\gamma&amp;lt;=1.0$&lt;/span&gt;. If &lt;span class=&quot;math-container&quot;&gt;$\gamma=0$&lt;/span&gt; the policy will be greedy, i.e. it will choose the best action only for the current state. And if &lt;span class=&quot;math-container&quot;&gt;$\gamma&amp;gt;0$&lt;/span&gt; then (possible) future rewards will be taken into account.&#xA;When &lt;span class=&quot;math-container&quot;&gt;$\gamma&amp;lt;1$&lt;/span&gt; then the infinite sum is finite as long as the reward sequence is bounded.&lt;/p&gt;&#xA;&lt;p&gt;As also commented in &lt;a href=&quot;https://stats.stackexchange.com/a/137595/90930&quot;&gt;this related answers&lt;/a&gt;, with a higher &lt;span class=&quot;math-container&quot;&gt;$\gamma$&lt;/span&gt; the policy is optimized for gains further in time, but will take more time to converge.&lt;/p&gt;&#xA;" OwnerUserId="13123" LastEditorUserId="13123" LastEditDate="2021-08-02T08:58:38.303" LastActivityDate="2021-08-02T08:58:38.303" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10133" PostTypeId="2" ParentId="10120" CreationDate="2016-02-08T04:53:55.723" Score="2" Body="&lt;p&gt;&lt;strong&gt;PCA doesn't really make sense for a time series if it is left as a time series:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are times when its useful to &lt;a href=&quot;http://faculty.washington.edu/ezivot/econ424/Working%20with%20Time%20Series%20Data%20in%20R.pdf&quot; rel=&quot;nofollow&quot;&gt;aggregate time series data into a rectangular representation&lt;/a&gt; in order to gain insight into interesting aspects of the data.  This is a feature engineering where the time series provides lots of the features.  For instance, you can create features for 'minute of hour', 'hour of day', 'morning', 'afternoon','evening', 'day of week', 'day of month', 'week of month', 'week of year', 'season', ...  You can even join this data with other data, like historical weather data or economic index, to see how daily temperature or economics affect your phenomenon. In these cases you could do all of this feature engineeting and then use PCA to find the most important features (radar plot) or simply to select the most important orthogonal features from the post PCA mashup (linear transformation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Time Series Smothing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a bunch of different methods to smooth and down-sample time series data. In this case, you are turning the quantized stochastic events and either turning them into regularly spaced data, or taking your already regularly spaced data and downsampling and smoothing it. Smoothing algorithms include Exponentially weighted moving average &lt;a href=&quot;https://en.wikipedia.org/wiki/Moving_average&quot; rel=&quot;nofollow&quot;&gt;(EWMA)&lt;/a&gt; or Holt-Winters smoothing.  Connor Johnson has a &lt;a href=&quot;http://connor-johnson.com/2014/02/01/smoothing-with-exponentially-weighted-moving-averages/&quot; rel=&quot;nofollow&quot;&gt;nice blog write-up&lt;/a&gt; on the these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMA&lt;/a&gt; is a more complete algorithm that includes smoothing, de-seasoning, and forecasting. It is very powerful, but &lt;a href=&quot;http://people.duke.edu/~rnau/411arim.htm&quot; rel=&quot;nofollow&quot;&gt;takes some time to master.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-02-08T04:53:55.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10134" PostTypeId="5" CreationDate="2016-02-08T05:45:45.807" Score="0" Body="&lt;p&gt;This is a special tag that is designed to be used only by the system for questions that have had all of their other tags removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All questions in the system must have at least one tag. Normally, it is not possible to submit new or edited questions that do not have tags; the system will reject the submission until a valid tag is applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few unusual situations that can cause this tag to be applied. When a tag has existed for six months and only been used on one question, and no tag wiki has been created for the tag, the system deletes the tag. When the community decides that a given tag can never be appropriate &amp;#8212; for example, because it is misspelled or is off-topic &amp;#8212; moderators may remove the tag from the site &lt;em&gt;en masse&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When one of these deletions removes the last tag from a question, the system adds the &lt;a href=&quot;/questions/tagged/untagged&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;untagged&amp;#39;&quot; rel=&quot;tag&quot;&gt;untagged&lt;/a&gt; tag to that question so that the &quot;all questions must have at least one tag&quot; rule is not broken.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can help clean up some of these questions! If they're good questions, edit them to remove the &lt;a href=&quot;/questions/tagged/untagged&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;untagged&amp;#39;&quot; rel=&quot;tag&quot;&gt;untagged&lt;/a&gt; tag and add more descriptive tags. If they're not so good, use the usual tools: downvoting, flagging and voting to close/delete.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-08T09:10:07.780" LastActivityDate="2016-02-08T09:10:07.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10135" PostTypeId="4" CreationDate="2016-02-08T05:45:45.807" Score="0" Body="This is a special tag that is designed to be used only by the system for questions that have had all of their other tags removed.  Do not add this tag to existing questions, it is reserved for use by the system to identify posts with no valid tags." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-08T09:10:00.367" LastActivityDate="2016-02-08T09:10:00.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10136" PostTypeId="5" CreationDate="2016-02-08T05:49:06.980" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-02-08T05:49:06.980" LastActivityDate="2016-02-08T05:49:06.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10137" PostTypeId="4" CreationDate="2016-02-08T05:49:06.980" Score="0" Body="Interpolation is a set of methods to construct intermediate points between known points." OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-08T07:34:02.953" LastActivityDate="2016-02-08T07:34:02.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10138" PostTypeId="1" AcceptedAnswerId="10141" CreationDate="2016-02-08T06:03:16.453" Score="6" ViewCount="690" Body="&lt;p&gt;I have time series data of one month plotted day-wise as&#xA;&lt;a href=&quot;https://i.stack.imgur.com/tazsC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tazsC.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Notice &lt;strong&gt;every day follows a different pattern&lt;/strong&gt;. Now, I want to show this &amp;quot;diversity in pattern&amp;quot; of each day in much compact form in a research paper. What are the different ways/options of representing this in compact form using R.&lt;/p&gt;&#xA;" OwnerUserId="13291" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-02-10T18:56:11.913" Title="Represent time-series data in much compact form" Tags="|r|visualization|ggplot2|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10139" PostTypeId="1" CreationDate="2016-02-08T06:38:59.600" Score="4" ViewCount="203" Body="&lt;p&gt;I understand the working of NIPALS algorithm but while doing the regression using PLS how exactly the relation between known and unknown is established using Principle Component Analysis.&#xA;The idea is clear up to the points both dependent and independent are expressed in form of their principle components. After I am not able to understand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I have read text and some papers about it but they are very difficult to understand. This Question may sound a trivial one, but please bear with me. Correct me if I am wrong.&lt;/p&gt;&#xA;" OwnerUserId="14789" LastEditorUserId="29169" LastEditDate="2020-12-18T03:53:19.090" LastActivityDate="2020-12-18T03:53:19.090" Title="How exactly dependent variable is expressed in terms of independent variables using Partial Least Square Regression Method?" Tags="|regression|linear-regression|supervised-learning|dimensionality-reduction|partial-least-squares|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10140" PostTypeId="1" AcceptedAnswerId="10145" CreationDate="2016-02-08T11:36:29.633" Score="2" ViewCount="614" Body="&lt;p&gt;I have recently used the heatmap widget in Orange 3. All the documentation says is &quot;Clustering (clusters data by similarity)&quot;. Is this using hierarchical or k-means or some other type of clustering? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On that note, is there a way to look at the code being run by all the widgets to see whats going on under the hood? It would be nice if after you finish the workflow you would get a file with the script run to perform the analysis.  &lt;/p&gt;&#xA;" OwnerUserId="16087" LastActivityDate="2016-02-09T01:32:29.220" Title="Orange 3 Heatmap clustering under the hood" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10141" PostTypeId="2" ParentId="10138" CreationDate="2016-02-08T12:07:51.693" Score="7" Body="&lt;p&gt;Simulate some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(purrr)&#xA;library(ggthemes)&#xA;&#xA;days &amp;lt;- seq(as.Date(&quot;2015-08-01&quot;), as.Date(&quot;2015-08-31&quot;), by=&quot;1 day&quot;)&#xA;hours &amp;lt;- sprintf(&quot;%02d&quot;, 0:23)&#xA;&#xA;map_df(days, function(x) {&#xA;  map_df(hours, function(y) {&#xA;    data.frame(day=x, hour=y, val=sample(2500, 1), stringsAsFactors=FALSE)&#xA;  })&#xA;}) -&amp;gt; df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Check it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(df, aes(x=hour, y=val, group=day)) +&#xA;  geom_line() +&#xA;  facet_wrap(~day) +&#xA;  theme_tufte(base_family=&quot;Helvetica&quot;) +&#xA;  labs(x=NULL, y=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/p6Oj2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/p6Oj2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you're only trying to convey the scope of the variation, perhaps use a boxplot of the values of hours across days?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(df, aes(x=hour, y=val)) +&#xA;  geom_boxplot(fill=&quot;#2b2b2b&quot;, alpha=0.25, width=0.75, size=0.25) +&#xA;  scale_x_discrete(expand=c(0,0)) +&#xA;  scale_y_continuous(expand=c(0,0)) +&#xA;  coord_flip() +&#xA;  theme_tufte(base_family=&quot;Helvetica&quot;) +&#xA;  theme(axis.ticks=element_blank()) +&#xA;  labs(x=NULL, y=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B1EZD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B1EZD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That can be tweaked to fit into most publication graphics slots and the boxplot shows just how varied each day's readings are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use &lt;code&gt;boxplot.stats&lt;/code&gt; to get the summary data and plot it on a line chart:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;library(tidyr)&#xA;&#xA;bps &amp;lt;- function(x) {&#xA;  cnf &amp;lt;- boxplot.stats(x)$conf&#xA;  data.frame(as.list(set_names(cnf, c(&quot;lwr&quot;, &quot;upr&quot;))), mean=mean(x))&#xA;}&#xA;&#xA;group_by(df, hour) %&amp;gt;% &#xA;  do(bps(.$val)) %&amp;gt;% &#xA;  ggplot(aes(x=hour, y=mean, ymin=lwr, ymax=upr, group=1)) +&#xA;  geom_ribbon(fill=&quot;#2b2b2b&quot;, alpha=0.25) +&#xA;  geom_line(size=0.25) +&#xA;  theme_tufte(base_family=&quot;Helvetica&quot;) +&#xA;  theme(axis.ticks=element_blank()) +&#xA;  labs(x=NULL, y=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RMQRa.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RMQRa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="710" LastActivityDate="2016-02-08T12:07:51.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10142" PostTypeId="1" CreationDate="2016-02-08T16:49:53.303" Score="2" ViewCount="308" Body="&lt;p&gt;My academical background is in physics and analysis (PDE's), but now I'am reading about data science. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm reading about &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcess.html#sklearn.gaussian_process.GaussianProcess&quot; rel=&quot;nofollow&quot;&gt;Gaussian Process implementation in Sci-Kit Learn&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could not find a simple explanation about those hyper-parameters theta0, thetaL and thetaU. Can someone explain to me what those parameters are, and how to choose then?&lt;/p&gt;&#xA;" OwnerUserId="16096" LastEditorUserId="11097" LastEditDate="2016-02-19T16:17:54.703" LastActivityDate="2020-08-03T15:11:55.513" Title="Hyperparameters in Gaussian Process" Tags="|python|statistics|regression|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10143" PostTypeId="1" CreationDate="2016-02-08T16:58:47.657" Score="12" ViewCount="6499" Body="&lt;p&gt;I'm looking for information on how should a Python Machine Learning project be organized. For Python usual projects there is &lt;a href=&quot;http://cookiecutter.readthedocs.org/en/latest/&quot;&gt;Cookiecutter&lt;/a&gt; and for R &lt;a href=&quot;http://projecttemplate.net/getting_started.html&quot;&gt;ProjectTemplate&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my current folder structure, but I'm mixing Jupyter Notebooks with actual Python code and it does not seems very clear.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;.&#xA;├── cache&#xA;├── data&#xA;├── my_module&#xA;├── logs&#xA;├── notebooks&#xA;├── scripts&#xA;├── snippets&#xA;└── tools&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I work in the scripts folder and currently adding all the functions in files under my_module, but that leads to errors loading data(relative/absolute paths) and other problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could not find proper &lt;em&gt;best practices&lt;/em&gt; or good examples on this topic besides some kaggle competition solutions and some Notebooks that have all the functions condensed at the start of such Notebook.&lt;/p&gt;&#xA;" OwnerUserId="16097" LastActivityDate="2020-08-03T15:11:52.947" Title="Python Machine Learning/Data Science Project Structure" Tags="|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10144" PostTypeId="1" AcceptedAnswerId="10146" CreationDate="2016-02-08T19:55:06.057" Score="4" ViewCount="7727" Body="&lt;p&gt;What are some reference sources for understanding Markov switching models?&lt;/p&gt;&#xA;" OwnerUserId="16103" LastEditorUserId="11097" LastEditDate="2016-02-09T05:24:00.793" LastActivityDate="2023-03-14T04:02:40.627" Title="Markov switching models" Tags="|time-series|markov-process|reference-request|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10145" PostTypeId="2" ParentId="10140" CreationDate="2016-02-09T01:32:29.220" Score="0" Body="&lt;p&gt;It appears the widget uses hierarchical clustering. I guess the metric is Euclidean distance by default and there doesn't seem to be a way to specify another one (except by using &lt;em&gt;Distances&lt;/em&gt; widget and connecting it into the &lt;em&gt;Distance Map&lt;/em&gt; widget).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think it is possible to export the widget's workflow as pure code, but you can look at what the widget does in &lt;a href=&quot;https://github.com/biolab/orange3/blob/master/Orange/widgets/visualize/owheatmap.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;the source code&lt;/a&gt; (seems pretty low-level, though). What you &lt;em&gt;can&lt;/em&gt; do, however, is select subsets of data (can be saved with &lt;em&gt;Save Data&lt;/em&gt; widget) for further analysis if that's of any help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YXUzC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YXUzC.png&quot; alt=&quot;Orange Heatmap widget screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-02-09T01:32:29.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10146" PostTypeId="2" ParentId="10144" CreationDate="2016-02-09T05:23:06.453" Score="7" Body="&lt;p&gt;Firstly, for understanding the Markov switching models, a nice knowledge of Markov models and the way they work. Most importantly, an idea of time series models and how they work, is very important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found &lt;a href=&quot;http://homepage.ntu.edu.tw/~ckuan/pdf/Lec-Markov_note_spring%202010.pdf&quot; rel=&quot;noreferrer&quot;&gt;this tutorial&lt;/a&gt; good enough for getting up to speed with the concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://econweb.ucsd.edu/~jhamilto/palgrav1.pdf&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; is another tutorial on a similar application of the switching model, which is the regime switching model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://statsmodels.sourceforge.net/&quot; rel=&quot;noreferrer&quot;&gt;statsmodels&lt;/a&gt; library has a nice support for building the Morkov switching models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ChadFulton/pymar&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; is one simple and quick Python tutorial which uses the statsmodels library.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-02-22T08:41:32.153" LastActivityDate="2017-02-22T08:41:32.153" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10147" PostTypeId="1" CreationDate="2016-02-09T08:10:30.340" Score="3" ViewCount="274" Body="&lt;p&gt;Multi-State Time Delay Neural Networks (MS-TDNNs) were introduced in&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Haffner, Patrick and Waibel, Alex: &lt;a href=&quot;http://isl.anthropomatik.kit.edu/downloads/0135_Kopie_.pdf&quot; rel=&quot;nofollow&quot;&gt;Multi-state time delay networks for continuous speech recognition&lt;/a&gt;. In Advances in neural information processing systems, 1992.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They are an extension of TDNNs. TDNNs are convolutional neural networks for automatic speech recognition (ASR), where the convolution happens over the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aim of MS-TDNNs seems to be to get rid of the hybrid approach in ASR, where you need dynamic programming / HMMs to chunk the audio stream and then neural networks to recognize the phonemes. Somehow MS-TDNNs seem to do the segmentation as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand how. Could somebody please explain it to me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Related side questions: Are MS-TDNNs recurrent networks? Where exactly does the name &quot;multi-state&quot; come from?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-09T08:15:40.120" LastActivityDate="2018-11-13T18:14:26.403" Title="How do MS-TDNNs work?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10149" PostTypeId="2" ParentId="9469" CreationDate="2016-02-09T08:53:31.053" Score="1" Body="&lt;p&gt;You can do &lt;a href=&quot;https://en.wikipedia.org/wiki/Topological_sorting&quot; rel=&quot;nofollow&quot;&gt;topological sorting&lt;/a&gt;. For example, here is some pythonic pseudocode&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def order(current_neuron, level):&#xA;    current_neuron.level = max(level, current_neuron.level)&#xA;    for child in sorted(current_neuron.children, key=children level, reverse=True):&#xA;        order(child, level + 1)&#xA;&#xA;for input_neuron in input_neurons:&#xA;    order(input_neuron, level=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Take this pseudo code with a grain of salt. This is just what came directly to my mind; it is not tested.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can calculate the activations in order of &lt;code&gt;node.level&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-02-09T08:53:31.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10150" PostTypeId="2" ParentId="6866" CreationDate="2016-02-09T09:06:10.080" Score="4" Body="&lt;p&gt;Recurrent Neural Networks can deal with variable length data. You might want to have a look at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Andrej Karpathy: &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;The Unreasonable Effectiveness of Recurrent Neural Networks&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Christopher Olah: &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow&quot;&gt;Understanding LSTM Networks&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Hochreiter, Schmidthuber: &lt;a href=&quot;http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=6795963&quot; rel=&quot;nofollow&quot;&gt;Long short-term memory&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another idea (which I have not tested so far and  just came to my mind) is using a histogram approach: You could probably make fixed-size windows, get the data from those windows and make it discrete (e.g. vector quantization, k-means). After that, you can make a histogram of how often those vectors appeared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;HMMs&lt;/a&gt; for recognition of variable length data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transformations (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Fourier_transform&quot; rel=&quot;nofollow&quot;&gt;Fourier transform&lt;/a&gt; from the time domain in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Frequency_domain&quot; rel=&quot;nofollow&quot;&gt;frequency domain&lt;/a&gt;) might also come in handy.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-02-09T09:06:10.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10151" PostTypeId="1" AcceptedAnswerId="10210" CreationDate="2016-02-09T09:32:54.870" Score="5" ViewCount="5848" Body="&lt;p&gt;I am currently reading:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Stephen Jose Hanson: &lt;a href=&quot;http://papers.nips.cc/paper/227-meiosis-networks.pdf&quot; rel=&quot;nofollow&quot;&gt;Meiosis Networks&lt;/a&gt;, 1990.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and I stumbled about this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is possible to precisely characterize the search problem in terms of the resources or degress of freedom in the learning model. If the task the learning system is to perform is classification then the system can be analyzed in terms of its ability to dichotomize stimulus points in feature space.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;Dichotomization Capability: Network Capacity&lt;/em&gt; Using a linear fan-in or hyperplane type neuron we can characterize the degrees of freedom inherent in a network of units with thresholded output. For example, with linear boundaries, consider 4 points, well distributed in a 2-dimensional feature space. There are exactly 14 linearly separable dichotomies that can be formed with the 4 target points. However, there are actually 16 ($2^4$) possible dichotomies of 4 points in 2 dimensions consequently, the number of possible dichotomies or arbitrary categories that are linearly implementable can be thought of as a capacity of the linear network in $k$ dimensions with $n$ examples. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is a &quot;dichonomy&quot; in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Side questions: what is a fan-in type neuron?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="12527" LastEditDate="2016-02-11T21:29:01.537" LastActivityDate="2016-02-11T21:29:01.537" Title="What is a Dichotomy?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10152" PostTypeId="2" ParentId="10151" CreationDate="2016-02-09T09:32:54.870" Score="3" Body="&lt;p&gt;It seems to be about classification with 2 classes. To understand where the 14 comes from, just try all cases of 4 points being in one of two classes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;&quot;&quot;&quot;Print all cases of 4 points being in one of two classes in 2D.&quot;&quot;&quot;&#xA;&#xA;&#xA;def print_pattern(pattern):&#xA;    &quot;&quot;&quot;Print four points in 2D.&quot;&quot;&quot;&#xA;    print(&quot;%s\t%s&quot; % (pattern[0], pattern[1]))&#xA;    print(&quot;%s\t%s&quot; % (pattern[2], pattern[3]))&#xA;&#xA;&#xA;get_bin = lambda x, n: format(x, 'b').zfill(n)&#xA;&#xA;# 16 possible cases how 4 points can belong to 2 classes&#xA;for i in range(16):&#xA;    bin_ = get_bin(i, 4)&#xA;    print_pattern(bin_)&#xA;    print(&quot;-&quot;*60)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    0&#xA;0    0&#xA;------------------------------------------------------------&#xA;0    0&#xA;0    1&#xA;------------------------------------------------------------&#xA;0    0&#xA;1    0&#xA;------------------------------------------------------------&#xA;0    0&#xA;1    1&#xA;------------------------------------------------------------&#xA;0    1&#xA;0    0&#xA;------------------------------------------------------------&#xA;0    1&#xA;0    1&#xA;------------------------------------------------------------&#xA;0    1        --- Not linearly seperable&#xA;1    0&#xA;------------------------------------------------------------&#xA;0    1&#xA;1    1&#xA;------------------------------------------------------------&#xA;1    0&#xA;0    0&#xA;------------------------------------------------------------&#xA;1    0        --- Not linearly seperable&#xA;0    1&#xA;------------------------------------------------------------&#xA;1    0&#xA;1    0&#xA;------------------------------------------------------------&#xA;1    0&#xA;1    1&#xA;------------------------------------------------------------&#xA;1    1&#xA;0    0&#xA;------------------------------------------------------------&#xA;1    1&#xA;0    1&#xA;------------------------------------------------------------&#xA;1    1&#xA;1    0&#xA;------------------------------------------------------------&#xA;1    1&#xA;1    1&#xA;------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm still not quite sure what a dichtonomy is, but this seems to be part of the answer.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-02-09T09:32:54.870" CommentCount="0" CommunityOwnedDate="2016-02-09T09:32:54.870" ContentLicense="CC BY-SA 3.0" />
  <row Id="10153" PostTypeId="1" CreationDate="2016-02-09T11:16:43.310" Score="1" ViewCount="26" Body="&lt;p&gt;I am working on a Multivariate Analysis on my dataset, I have a large dataset of 50k users let's call it (Loyal Users) with various numerical attributes +13 tracking the user's activity and events within a timeframe.&#xA;I want to come up with Set of rules to define this group of users so that i can come up with a common pattern.&#xA;This is an example of the data that i am working on&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;username        Like     Share           Select     Send  Verify  Register&#xA;&#xA;Alex            524      329             203        787    1       1&#xA;Kristina        197      186             286        120    0       1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is an example of rules that i want to make : &#xA;If Share &gt; 23 and Verify =1 and Like &gt;120  then Loyal User&lt;/p&gt;&#xA;&#xA;&lt;p&gt;would that be possible after doing a principal components analysis (PCA) and linear discriminant analysis (LDA) ? And how can i do that ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas would be great.&lt;/p&gt;&#xA;" OwnerUserId="15695" LastActivityDate="2016-02-09T11:16:43.310" Title="Generating a set of Rules that defines a Group of users ( Multivariate Analysis )" Tags="|machine-learning|r|data-mining|predictive-modeling|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10154" PostTypeId="1" CreationDate="2016-02-09T11:22:31.957" Score="3" ViewCount="1070" Body="&lt;p&gt;I have data of latitude, longitude and timestamp. I am trying to build a graph based on pincodes (in India) (equivalent to zipcode). Based on this graph and trajectory data that I have, I want to predict ETA given source, destination and time for dispatch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am very new to this field, So I dont know how to handle and make structures using this data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already gone through this question&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/5802/learning-time-of-arrival-eta-from-historical-location-data-of-vehicle&quot;&gt;Learning time of arrival (ETA) from historical location data of vehicle&lt;/a&gt; but it is not very clear to me. Any kind of help/suggestions is appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : Pincodes are equivalent to zipcodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Graph : Nodes of the graph will be pincode boundaries and if these pincodes share boundary then there will be edge between them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trajectory : It is the data collected from GPS when vehicle is travelling on the road. Data is in format (Latitude, Longitude, TimeStamp, vehicleId)&lt;/p&gt;&#xA;" OwnerUserId="15906" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2020-04-28T12:13:45.213" Title="How to predict Estimated Time for Arrival given only trajectory data and time?" Tags="|predictive-modeling|graphs|linear-regression|geospatial|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10157" PostTypeId="2" ParentId="10147" CreationDate="2016-02-09T14:22:41.033" Score="2" Body="&lt;p&gt;So I think you understood that a MS-TDNN has &lt;strong&gt;two parts&lt;/strong&gt;: a conventional &lt;strong&gt;TDNN&lt;/strong&gt; that calculates state probabilities for each frame and kind of a &lt;strong&gt;perceptron&lt;/strong&gt; on top of it linking the states of several frames to a word. The later part is doing the segmentation. &lt;strong&gt;Its connections aren't trained&lt;/strong&gt;, but just used for running BP and train the TDNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The segmentation is obtained by some other algorithm (I don't remember exactly, but something like dynamic time warping). In normal DNN-HMM hybrid systems the DNNs are trained separately to predict the correct state for each frame. The &lt;strong&gt;error function is calculated on frame level!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not optimal as we do not care so much about each state being correct, but the final text. &lt;strong&gt;WER is calculated on world level&lt;/strong&gt;. MS-TDNNs try to solve this by having this extra layer that allows passing a word level based error to BP. This is very similar to todays methods called sequence training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Neither TDNN nor MS-TDNN are recurrent networks.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If you are interested in RNNs and ASR systems that do not require HMMs I would recommend you to look at CTC objective that does indeed learn the alignment automatically.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="16124" LastActivityDate="2016-02-09T14:22:41.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10158" PostTypeId="1" CreationDate="2016-02-09T14:42:16.560" Score="1" ViewCount="483" Body="&lt;p&gt;I'm exploring follow-the-regularized-leader FTRL proximal gradient descent:&#xA;&lt;a href=&quot;http://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, &lt;a href=&quot;https://github.com/mlandry22/kaggle/blob/master/modern_logistic_regression/fast_solution_v3.py&quot; rel=&quot;nofollow&quot;&gt;reference implementation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everywhere FTRL is mentioned, the loss surface for the gradient decent is the &lt;code&gt;LogLoss&lt;/code&gt;, and the model for prediction is &lt;code&gt;Logistic regression&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use the same algorithm for a linear least squares model? I have a problem I want to model with a &lt;code&gt;linear model&lt;/code&gt; and define the loss by &lt;code&gt;least squares&lt;/code&gt;, and then do FTRL to find the optimal solution - do you see any problem with that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2016-11-18T03:04:06.497" Title="Can FTRL be applied on linear least squares? or is it just for logistic regression models?" Tags="|logistic-regression|linear-regression|gradient-descent|online-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10159" PostTypeId="1" CreationDate="2016-02-09T15:12:59.843" Score="2" ViewCount="66" Body="&lt;p&gt;This is perhaps not an ideal question by StackExchange rules, but I don't know where to find a more informed community, so I'll attempt to make it as specific as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to wrap my brain around 'data analysis', and I'm looking for attempts to analyze the field, to perhaps create a taxonomy of the analytic process, or in some other way to impose an order on 'analysis', where analysis might include both descriptive and predictive modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the field of data graphics, there are several well known examples: One could look at &lt;em&gt;Visualizing Data&lt;/em&gt; by Cleveland, in which he organizes graphical techniques by dimensionality, or more broadly, at Bertin's &lt;em&gt;Semiology of Graphics&lt;/em&gt;, or Wilkinson's &lt;em&gt;The Grammar of Graphics&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a related domain (a sub-domain of analysis, actually), Codd's relational algebra attempts to organize the filtering and summarization of tabular data. (I'm very much focused on tabular data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm asking for are pointers to a proposed &quot;grammar&quot; of analytics.   &lt;/p&gt;&#xA;" OwnerUserId="16126" LastEditorUserId="11097" LastEditDate="2016-02-18T06:24:25.587" LastActivityDate="2016-02-18T06:24:25.587" Title="An analysis of analysis?" Tags="|statistics|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10162" PostTypeId="1" AcceptedAnswerId="10167" CreationDate="2016-02-09T16:29:52.363" Score="11" ViewCount="20762" Body="&lt;p&gt;I would like to use the knn distance plot to be able to figure out which eps value should I choose for the DBSCAN algorithm.&#xA;Based on &lt;a href=&quot;http://www.sthda.com/english/wiki/dbscan-density-based-clustering-for-discovering-clusters-in-large-datasets-with-noise-unsupervised-machine-learning&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; page:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The idea is to calculate, the average of the distances of every point&#xA;  to its k nearest neighbors.  The value of k will be specified by the&#xA;  user and corresponds to MinPts. Next, these k-distances are plotted in&#xA;  an ascending order. The aim is to determine the “knee”, which&#xA;  corresponds to the optimal eps parameter.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using python with numpy/sklearn, I have the following points, with the following distance for 6-knn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])&#xA;nbrs = NearestNeighbors(n_neighbors=len(X)).fit(X)&#xA;distances, indices = nbrs.kneighbors(X)&#xA;&#xA;# Indices&#xA;&#xA;[[0 1 2 3 4 5]&#xA; [1 0 2 3 4 5]&#xA; [2 1 0 3 4 5]&#xA; [3 4 5 0 1 2]&#xA; [4 3 5 0 1 2]&#xA; [5 4 3 0 1 2]]&#xA;&#xA;# Distances&#xA;[[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]&#xA;[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]&#xA;[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]&#xA;[ 0.          1.          2.23606798  2.82842712  3.60555128  5.        ]&#xA;[ 0.          1.          1.41421356  3.60555128  4.47213595  5.83095189]&#xA;[ 0.          1.41421356  2.23606798  5.          5.83095189  7.21110255]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I computed the average distance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distances.mean()&#xA;2.9269575028354495&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is I don't understand how exactly could I represent the same plot as them with distances in y-axis and number of points according to the distances on the x-axis using python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank for your help.&lt;/p&gt;&#xA;" OwnerUserId="13915" LastEditorUserId="8878" LastEditDate="2016-03-02T15:50:11.347" LastActivityDate="2019-04-22T06:49:49.127" Title="Knn distance plot for determining eps of DBSCAN" Tags="|python|clustering|parameter-estimation|dbscan|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10163" PostTypeId="1" CreationDate="2016-02-09T16:54:06.513" Score="2" ViewCount="84" Body="&lt;p&gt;In the paper, '&lt;a href=&quot;https://arxiv.org/pdf/1312.6082&quot; rel=&quot;nofollow noreferrer&quot;&gt;Goodfellow, I., et al. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. ICLR, 2014&lt;/a&gt;', on page 10 there is a table which calculate $\log(P(S|H))$ given $\log(P(s_1,s_2,...,s_l))$ and $\log(P(l))$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the equation given in page 4 it should be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(P(S|H)) = \log(P(s_1,s_2,...,s_l)) + \log(P(l))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but if you see from the first two lines in the lower table it is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(P(S|H)) = 10\cdot\log(P(s_1,s_2,...,s_l)) + \log(P(l))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and on third line it is correctly specified as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(P(S|H)) = \log(P(s_1,s_2,...,s_l)) + \log(P(l))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did I miss anything??&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8vuym.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8vuym.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is the table of probability (idealized) predicted by the model:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V9hoY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V9hoY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16129" LastEditorUserId="29169" LastEditDate="2021-02-16T15:53:09.157" LastActivityDate="2021-02-16T15:53:09.157" Title="Discrepancy in probability calculations in paper 'Multi-digit Number Recognition...'" Tags="|deep-learning|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10165" PostTypeId="1" CreationDate="2016-02-09T19:10:00.017" Score="4" ViewCount="2306" Body="&lt;p&gt;I have some numeric data that has come 'binned', but the bins are not of equal sizes in terms of scale or quantile&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, an age variable that is [0-16), [16-21), [21-30), [30-45), [45-65), [65, ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I leave it as a categorical variable, a tree will treat each category separately and discount the ordered relationship between the factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I change it to an ordinal variable, e.g. [0, 1, 2, 3, 4, 5] and keep an array of labels for later reporting, the tree might split on, e.g. &amp;lt;2.5, which seems more natural to me, but then again the distance between 1 and 2 is not the same as the distance between 2 and 3 by any measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm leaning towards the second solution, but I would love some input!&lt;/p&gt;&#xA;" OwnerUserId="1264" LastActivityDate="2017-08-02T14:12:42.947" Title="What's the best way to use binned data in a tree-based model?" Tags="|random-forest|feature-extraction|categorical-data|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10166" PostTypeId="2" ParentId="694" CreationDate="2016-02-09T19:29:40.513" Score="1" Body="&lt;p&gt;If you are interested in performance comparisons, Soumith Chintala maintains a set of convnet benchmarks that cover several of the already mentioned python frameworks (&lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;nofollow&quot;&gt;TensorFlow&lt;/a&gt;, &lt;a href=&quot;https://github.com/pfnet/chainer&quot; rel=&quot;nofollow&quot;&gt;chainer&lt;/a&gt;, &lt;a href=&quot;https://github.com/NervanaSystems/neon&quot; rel=&quot;nofollow&quot;&gt;neon&lt;/a&gt;, &lt;a href=&quot;https://github.com/Theano/Theano&quot; rel=&quot;nofollow&quot;&gt;Theano&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow&quot;&gt;https://github.com/soumith/convnet-benchmarks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16134" LastActivityDate="2016-02-09T19:29:40.513" CommentCount="1" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="10167" PostTypeId="2" ParentId="10162" CreationDate="2016-02-09T19:34:15.977" Score="9" Body="&lt;p&gt;You &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;take the last column of that matrix&lt;/li&gt;&#xA;&lt;li&gt;sort descending&lt;/li&gt;&#xA;&lt;li&gt;plot index, distance&lt;/li&gt;&#xA;&lt;li&gt;hope to see a knee (if the distance does not work well. there might be none)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-02-09T19:34:15.977" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10168" PostTypeId="1" AcceptedAnswerId="11072" CreationDate="2016-02-09T19:43:01.207" Score="6" ViewCount="4123" Body="&lt;p&gt;I'm captivated by autoencoders and really like the idea of convolution.  It seems though that both Theano and TensorFlow only support conv2d to go from an array of 2D-RGB (n 3D arrays) to an array of higher-depth images.  That makes sense from the traditional tensor-product math, c_ijkl = sum{a_ijk*b_klm}, but means it's hard to 'de-convolve' an image.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases, if I have an image (in #batch, depth, height, width form), I can do a conv to get (#batch, num_filters, height/k, width/k).  I'd really like to do the opposite, like going from (#batch, some_items, height/k, width/k) to (#batch, depth, height, width).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow had the hidden deconv2d function for a while (in 0.6, I think, undocumented), but I'd like to know if there's a math trick I can use to get a bigger output in the last two dimensions after a convolution than the input.  I'd settle for a series of differentiable operations, like conv -&gt; resize, but I want to avoid just doing a dense matrix multiplication -&gt; resize like I've been doing so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: As of today (2016/02/17) TensorFlow 0.7 has the tf.depth_to_space method, which helps greatly in this endeavor.  (&lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/depth_to_space&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/api_docs/python/tf/depth_to_space&lt;/a&gt;)  I would still love a Theano based solution, too, to complete my understanding of the material.&lt;/p&gt;&#xA;" OwnerUserId="16135" LastEditorUserId="29169" LastEditDate="2021-02-16T15:50:54.480" LastActivityDate="2021-02-16T15:50:54.480" Title="Stuck on deconvolution in Theano and TensorFlow" Tags="|tensorflow|convolutional-neural-network|autoencoder|theano|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10169" PostTypeId="2" ParentId="10165" CreationDate="2016-02-09T21:35:25.743" Score="3" Body="&lt;p&gt;For a complete decision tree, either of your proposed models would be able to represent the same set of concepts since a decision tree can be decomposed into a disjunction of conjunctions. I don't think the fact that the bins have different sizes is really an issue because placing any significance on the exact location of splits occurring &quot;inside&quot; a bin is probably not a good idea anyway. From a performance perspective, what probably matters most is how your data are distributed (i.e., does your concept split the data between a single pair of adjacent bins or is it more complicated).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have the &quot;random-forest&quot; tag associated with your question, I'm inclined to agree with you that keeping them ordered is probably better because if you are generating many trees, the ordered representation will likely result in more terse/smaller trees.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-02-09T21:35:25.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10170" PostTypeId="2" ParentId="9222" CreationDate="2016-02-09T23:28:57.067" Score="1" Body="&lt;p&gt;The strided deconvolution is expanding the size of the layer (you can think of it as undo-ing the effect of the pooling without doing unpooling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out slide 18 &lt;a href=&quot;http://www.slideshare.net/nervanasys/anil-thomas-object-recognition?ref=http://neon.nervanasys.com/docs/latest/introductory_resources.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. That might help.&lt;/p&gt;&#xA;" OwnerUserId="16140" LastActivityDate="2016-02-09T23:28:57.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10174" PostTypeId="2" ParentId="9662" CreationDate="2016-02-10T12:06:23.797" Score="1" Body="&lt;p&gt;Talking about SQL world: from your requirements point of view parallel column based DB Vertica seems to fit well. It is not free of charge though. &#xA;Even more powerful solution if you have good budget is Teradata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd need to have at least 70 spindles to store your data on HDDs (5TB) on site +/- compression and redundancy. May be add more RAM or SDD if you can cache. That could fit in 3-10 boxes, depending on the processing power you require.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rest is a software question on a trade-off between data management, feature set, optimization, your own time spent and the price.&#xA;Teradata and Vertica will surely reduce your time spent writing queries and optimizing them, managing data. In case you'd need joins that advantage will be much bigger yet.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-10T12:06:23.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10175" PostTypeId="2" ParentId="10085" CreationDate="2016-02-10T12:40:19.980" Score="3" Body="&lt;p&gt;Where the feature selection finds a place in your pipeline depends on the problem. If you know your data well, you can select features based on this knowledge manually. If you don't - the experimentation with the models using cross validation may be best. Reducing number of features a priory with some additional technique like chi2 or PCA may actually reduce model accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience with text classification with SGD classifier for example leaving all hundred thousands words encoded as  binary features brought better results compared to reducing to a few thousands or hundreds. Training time is actually faster with all features as feature selection is rather slow with my toolset (sklearn) because it is not stochastic like SGD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multicollinearity is something to watch out for, but the feature interpretability might equally be important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then people report getting best result with ensembles of models. Each model capturing a particular part of information space better than the others. That would also preclude you from selecting the features before fitting all models you'd include into your ensemble.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-10T12:40:19.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10177" PostTypeId="1" CreationDate="2016-02-10T14:50:49.610" Score="2" ViewCount="502" Body="&lt;p&gt;We are trying to build a recommendation system for a supermarket with diverse item types (ranging from fast-moving grocery to low-moving electronic items). Some items are purchased more frequently in high volume and some items are purchased only once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have purchase history data of 4 months from 25K+ customers across 30K+ SKU's from 100+ departments. We ran ALS-WR in Spark to generate recommendations. To our surprise, we are receiving top 15 recommendations for each customer quite generic without much variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have tried several means to diversify the recommendations as below- &#xA;- computed &quot;rating&quot; = normalized # of purchase &#xA;- computed &quot;rating&quot; = log of # of purchase &#xA;- computed &quot;rating&quot; = 1 (if purchase # &gt; 1) &#xA;- We have used following combination of parameters - lambda = 0.01 to 300, alpha = 5 to 50, rank = 10, 20, 30 and # of iterations = 10, 20 &#xA;- preference considered is explicit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think that ALS can be used for such heterogeneous data? If yes, what modifications will make the recommendations diverse &amp;amp; personalized?&lt;/p&gt;&#xA;" OwnerUserId="16147" LastActivityDate="2024-02-14T08:02:25.423" Title="Spark ALS-WR giving the same recommended items for all users" Tags="|machine-learning|recommender-system|apache-spark|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10178" PostTypeId="1" AcceptedAnswerId="10180" CreationDate="2016-02-10T16:37:43.837" Score="0" ViewCount="200" Body="&lt;p&gt;I have a pandas data frame of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r1    r2    r3    r4    r5&#xA;&#xA;0    1    12    0    4&#xA;1    1    2    9    2&#xA;32   5    0    0    0&#xA;12   14   3    1    23&#xA;0    2    43    5    2&#xA;9    3    5    1    1&#xA;0    0    0    0    1&#xA;1    0    0    0    0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to check if any column: &lt;code&gt;r1, r2, r3, r4, r5&lt;/code&gt; significantly differs from any of the other. Should I do a t test or an anova? And how would I set it up for the computation? &lt;/p&gt;&#xA;" OwnerUserId="10584" LastActivityDate="2016-02-10T16:46:04.273" Title="t test or anova" Tags="|python|statistics|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10179" PostTypeId="1" AcceptedAnswerId="10308" CreationDate="2016-02-10T16:40:34.743" Score="26" ViewCount="27663" Body="&lt;p&gt;As far as I know, to train learning to rank models, you need to have three things in the dataset:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;label or relevance&lt;/li&gt;&#xA;&lt;li&gt;group or query id&lt;/li&gt;&#xA;&lt;li&gt;feature vector&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For example, the &lt;a href=&quot;http://research.microsoft.com/en-us/um/beijing/projects/letor/letor4dataset.aspx&quot; rel=&quot;noreferrer&quot;&gt;Microsoft Learning to Rank dataset&lt;/a&gt; uses this format (label, group id, and features).&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;1 qid:10 1:0.031310 2:0.666667 ...&#xA;0 qid:10 1:0.078682 2:0.166667 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am trying out XGBoost that utilizes GBMs to do pairwise ranking. They have an &lt;a href=&quot;https://github.com/dmlc/xgboost/tree/master/demo/rank&quot; rel=&quot;noreferrer&quot;&gt;example for a ranking task&lt;/a&gt; that uses the C++ program to learn on the Microsoft dataset like above.&lt;/p&gt;&#xA;&lt;p&gt;However, I am using their Python wrapper and cannot seem to find where I can input the group id (&lt;code&gt;qid&lt;/code&gt; above). I can train the model using just the features and relevance score, but I am missing something.&lt;/p&gt;&#xA;&lt;p&gt;Here is a sample script.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;gbm = XGBRegressor(objective=&amp;quot;rank:pairwise&amp;quot;)&#xA;&#xA;X =  np.random.normal(0, 1, 1000).reshape(100, 10)&#xA;y = np.random.randint(0, 5, 100)&#xA;&#xA;gbm.fit(X, y) ### --- no group id needed???&#xA;&#xA;print gbm.predict(X)&#xA;&#xA;# should be in reverse order of relevance score&#xA;print y[gbm.predict_proba(X)[:, 1].argsort()][::-1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="588" LastEditorUserId="121138" LastEditDate="2021-07-30T17:11:07.820" LastActivityDate="2021-07-30T17:11:07.820" Title="How fit pairwise ranking models in XGBoost?" Tags="|search|ranking|xgboost|gbm|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10180" PostTypeId="2" ParentId="10178" CreationDate="2016-02-10T16:46:04.273" Score="2" Body="&lt;p&gt;This is typical statistics problem.  When you have multiple 'classes' that you assume are normally distributed you first run an ANOVA.  Then, IFF (if-and-only-if) the ANOVA is significant, then run post-hoc pairwise t-tests with an appropriate correction (e.g. Bonferroni).&lt;/p&gt;&#xA;" OwnerUserId="6385" LastActivityDate="2016-02-10T16:46:04.273" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10181" PostTypeId="2" ParentId="10127" CreationDate="2016-02-10T17:19:47.570" Score="4" Body="&lt;p&gt;I will apologize in advance for linking to the Wikipedia article, but I'm not sure I can do a better job explaining it.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Boltzmann_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Boltzmann_machine&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Will extract selected passages to prevent link rot shortly.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the reasons we see so much talk about training RBMs and not much talk about the other is, by my recollection of a talk by Hinton, &quot;Intractable, except for very small problems or limited cases.&quot;  The activation of a hidden node depends not only on the state of visible nodes, but on its compatriots. This means our clever contrastive divergence and gradient decent methods don't work!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you didn't ask why it's difficult, you asked how to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short, short answer is &lt;a href=&quot;https://en.wikipedia.org/wiki/Simulated_annealing&quot; rel=&quot;nofollow noreferrer&quot;&gt;simulated annealing&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The long short answer, we select an initial state for the network, then perform updates until the network reaches a global thermal equilibrium, then try and adjust the weights while reducing the temperature until our activations approximate a global minimum.  The process repeats until we're happy with the outcome.&lt;/p&gt;&#xA;" OwnerUserId="16135" LastEditorUserId="39830" LastEditDate="2019-10-24T16:42:06.270" LastActivityDate="2019-10-24T16:42:06.270" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10183" PostTypeId="2" ParentId="10138" CreationDate="2016-02-10T18:56:11.913" Score="3" Body="&lt;p&gt;Two alternative way are a &lt;em&gt;density plot&lt;/em&gt; for each hour and &lt;em&gt;level plot&lt;/em&gt; of hour and factorized value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm showing ganarated data with two different unified distribution (for day and night hours)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Densityplot&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;xy &amp;lt;- densityplot(~ value,  groups = hour ,   &#xA;plot.points=FALSE, &#xA;data =  df,   &#xA;scales=list(x=list(rot=90, cex= .9 ),y=list(cex=.9)),par.strip.text=list(cex=.8),   &#xA;ylab=&quot;density&quot;, xlab=&quot;value&quot;, main=paste(  &quot;DensityPlot&quot;  ) )&#xA;print (xy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rJAPX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rJAPX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Levelplot&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(RColorBrewer)&#xA;xy &amp;lt;- levelplot(cnt ~    hour + value,   &#xA; col.regions=colorRampPalette(brewer.pal(9,&quot;YlOrRd&quot;))(16) ,&#xA;data =  df,   &#xA;scales=list(x=list(rot=90, cex= .9 ),y=list(cex=.9)),par.strip.text=list(cex=.8),   &#xA;ylab=&quot;value&quot;, xlab=&quot;hour&quot;, main=paste( &quot;LevelPlot&quot;  ) )&#xA;print (xy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V0sPJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V0sPJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-02-10T18:56:11.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10184" PostTypeId="1" CreationDate="2016-02-10T19:32:57.467" Score="1" ViewCount="164" Body="&lt;p&gt;The ICD code scheme has gone through many revisions: &lt;a href=&quot;https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any compatibility between the different versions? (e.g. can ICD-9 code be directly used where ICD-10 codes are expected?)&lt;/p&gt;&#xA;" OwnerUserId="16170" LastActivityDate="2016-03-13T05:58:55.897" Title="Are ICD codes forward compatible?" Tags="|data-formats|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10185" PostTypeId="2" ParentId="9672" CreationDate="2016-02-10T20:51:16.730" Score="4" Body="&lt;p&gt;I've been reading up on cascade correlation quite a bit recently and made a python implementation &lt;a href=&quot;https://github.com/DanielSlater/CascadeCorrelation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/DanielSlater/CascadeCorrelation&lt;/a&gt; (though it still needs a bit of cleaning up/extra work and has a bunch of me mucking around with using Particle Swarm Optimization for selecting candidates, definitely not production ready).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To try and explain step 3.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Start by creating a number of candidate hidden nodes with random weights. These have incoming connection from all existing hidden nodes and input nodes.&lt;/li&gt;&#xA;&lt;li&gt;We then use that equation $$S=\sum_o \left | \sum_p(V_p - \overline{V})(E_{p,o} - \overline{E_o}) \right |$$ to train the candidate nodes. &lt;/li&gt;&#xA;&lt;li&gt;The residual output error is the difference between the output of the network and the target value(think sum of squared error without the square).&lt;/li&gt;&#xA;&lt;li&gt;$S$ is the correlation between the activation of our candidate node and the the residual error.&lt;/li&gt;&#xA;&lt;li&gt;$V_p$ is the activation of the candidate node given input $p$.&lt;/li&gt;&#xA;&lt;li&gt;After a bit of backprop training against $S$ we choose our best candidate. This becomes a new hidden node.&lt;/li&gt;&#xA;&lt;li&gt;This is when it's weights are frozen. That is to say after random initialization then back-prop training, then selecting the best one.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13630" LastEditorUserId="29575" LastEditDate="2018-09-15T00:13:20.923" LastActivityDate="2018-09-15T00:13:20.923" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10186" PostTypeId="2" ParentId="10165" CreationDate="2016-02-10T21:11:01.363" Score="3" Body="&lt;p&gt;Instead of using a random forest classifier, you could instead use a random forrest regression analysis.  Use the mean value of the bin as the value (which will take into account the relative values as you suggest).  This has the added benefit that if you get a dataset that has actual ages, you don't need to change the training, and you'll be used to translating the scores (ages) provided by the analytic to your bins.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-02-10T21:11:01.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10188" PostTypeId="1" AcceptedAnswerId="10192" CreationDate="2016-02-10T21:52:30.730" Score="115" ViewCount="70605" Body="&lt;p&gt;I'm just getting started with some machine learning, and until now I have been dealing with linear regression over one variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have learnt that there is a hypothesis, which is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_\theta(x)=\theta_0+\theta_1x$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find out good values for the parameters $\theta_0$ and $\theta_1$ we want to minimize the difference between the calculated result and the actual result of our test data. So we subtract&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_\theta(x^{(i)})-y^{(i)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for all $i$ from $1$ to $m$. Hence we calculate the sum over this difference and then calculate the average by multiplying the sum by $\frac{1}{m}$. So far, so good. This would result in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{1}{m}\sum_{i=1}^mh_\theta(x^{(i)})-y^{(i)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is not what has been suggested. Instead the course suggests to take the square value of the difference, and to multiply by $\frac{1}{2m}$. So the formula is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{1}{2m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is that? Why do we use the square function here, and why do we multiply by $\frac{1}{2m}$ instead of $\frac{1}{m}$?&lt;/p&gt;&#xA;" OwnerUserId="16148" LastEditorUserId="44002" LastEditDate="2018-01-01T17:27:08.807" LastActivityDate="2019-02-25T11:03:05.913" Title="Why do cost functions use the square error?" Tags="|machine-learning|linear-regression|loss-function|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10189" PostTypeId="1" AcceptedAnswerId="10194" CreationDate="2016-02-10T22:53:49.620" Score="5" ViewCount="3449" Body="&lt;p&gt;I want to implement K-means algorithm in Spark. I am looking for a starting point and I found &lt;a href=&quot;http://www.cs.berkeley.edu/~rxin/ampcamp-ecnu/machine-learning-with-spark.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Berkeley's naive implementation&lt;/a&gt;. However, is that distributed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean I see no mapreduce operations. Or maybe, when submitted in Spark, the framework actually makes the needed tricks under the hood to distribute the algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;I also found that &lt;a href=&quot;http://www.infoworld.com/article/2982475/big-data/spark-15-shows-mapreduce-the-exit.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spark shows mapreduce the exit&lt;/a&gt; and I am using Spark 1.6.&lt;/sub&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT: This code produces a runtime error, check &lt;a href=&quot;https://stackoverflow.com/questions/35373478/mapping-of-elements-gone-bad&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15927" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-02-12T23:02:16.177" Title="Distributed k-means in Spark" Tags="|clustering|k-means|apache-spark|distributed|scala|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10190" PostTypeId="2" ParentId="10189" CreationDate="2016-02-10T23:04:51.403" Score="2" Body="&lt;p&gt;I don't know about that specific implementation, but we use the &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-clustering.html&quot; rel=&quot;nofollow&quot;&gt;mllib k-means&lt;/a&gt; here at my work, to some degree of success. It is distributed and runs on Spark!&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-02-10T23:04:51.403" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10191" PostTypeId="2" ParentId="10188" CreationDate="2016-02-10T23:28:08.403" Score="31" Body="&lt;p&gt;The 1/2 coefficient is merely for convenience; it makes the derivative, which is the function actually being optimized, look nicer. The 1/m is more fundamental; it suggests that we are interested in the &lt;em&gt;mean&lt;/em&gt; squared error. This allows you to make fair comparisons when changing the sample size, and prevents overflow. So called &quot;stochastic&quot; optimizers use a subset of the data set (m' &amp;lt; m). When you introduce a regularizer (an additive term to the objective function), using the 1/m factor allows you to use the same coefficient for the regularizer regardless of the sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the question of why the square and not simply the difference: don't you want underestimates to be penalized similarly to overestimates? Squaring eliminates the effect of the sign of the error. Taking the absolute value (L1 norm) does too, but its derivative is undefined at the origin, so it requires more sophistication to use. The L1 norm has its uses, so keep it in mind, and perhaps ask the teacher if (s)he's going to cover it.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-02-11T00:00:14.410" LastActivityDate="2016-02-11T00:00:14.410" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10192" PostTypeId="2" ParentId="10188" CreationDate="2016-02-11T01:18:54.240" Score="73" Body="&lt;p&gt;Your loss function would not work because it incentivizes setting &lt;span class=&quot;math-container&quot;&gt;$\theta_1$&lt;/span&gt; to any finite value and &lt;span class=&quot;math-container&quot;&gt;$\theta_0$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$-\infty$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's call &lt;span class=&quot;math-container&quot;&gt;$r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} -y$&lt;/span&gt; the &lt;em&gt;residual&lt;/em&gt; for &lt;span class=&quot;math-container&quot;&gt;$h$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your goal is to make &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; &lt;strong&gt;as close to zero&lt;/strong&gt; as possible, &lt;strong&gt;not just minimize it&lt;/strong&gt;. A high negative value is just as bad as a high positive value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; You can counter this by artificially limiting the parameter space &lt;span class=&quot;math-container&quot;&gt;$\mathbf{\Theta} $&lt;/span&gt;(e.g. you want &lt;span class=&quot;math-container&quot;&gt;$|\theta_0| &amp;lt; 10$&lt;/span&gt;). In this case, the optimal parameters would lie on certain points on the boundary of the parameter space. See &lt;a href=&quot;https://math.stackexchange.com/q/896388/12467&quot;&gt;https://math.stackexchange.com/q/896388/12467&lt;/a&gt;. This is not what you want.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Why do we use the square loss&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The squared error forces &lt;span class=&quot;math-container&quot;&gt;$h(x)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; to match. It's minimized at &lt;span class=&quot;math-container&quot;&gt;$u=v$&lt;/span&gt;, if possible, and is always &lt;span class=&quot;math-container&quot;&gt;$\ge 0$&lt;/span&gt;, because it's a square of the real number &lt;span class=&quot;math-container&quot;&gt;$u-v$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$|u-v|$&lt;/span&gt; would also work for the above purpose, as would &lt;span class=&quot;math-container&quot;&gt;$(u-v)^{2n}$&lt;/span&gt;, with &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; some positive integer. The first of these is actually used (it's called the &lt;span class=&quot;math-container&quot;&gt;$\ell_1$&lt;/span&gt; loss; you might also come across the &lt;span class=&quot;math-container&quot;&gt;$\ell_2$&lt;/span&gt; loss, which is another name for squared error). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, why is the squared loss better than these? This is a &lt;em&gt;deep&lt;/em&gt; question related to the link between &lt;em&gt;Frequentist&lt;/em&gt; and &lt;em&gt;Bayesian&lt;/em&gt; inference. In short, the squared error relates to &lt;strong&gt;Gaussian Noise&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data does not fit all points exactly, i.e. &lt;span class=&quot;math-container&quot;&gt;$h(x)-y$&lt;/span&gt; is not zero for some point no matter what &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt; you choose (as will always happen in practice), that might be because of &lt;strong&gt;noise&lt;/strong&gt;. In any complex system there will be many small &lt;strong&gt;independent&lt;/strong&gt; causes for the difference between your &lt;em&gt;model&lt;/em&gt; &lt;span class=&quot;math-container&quot;&gt;$h$&lt;/span&gt; and &lt;em&gt;reality&lt;/em&gt; &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt;: measurement error, environmental factors etc. By the &lt;a href=&quot;https://en.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Central Limit Theorem&lt;/strong&gt;&lt;/a&gt;(CLT), the total noise would be distributed &lt;em&gt;Normally&lt;/em&gt;, i.e. according to the &lt;strong&gt;Gaussian distribution&lt;/strong&gt;. We want to pick the best fit &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt; taking this noise distribution into account. Assume &lt;span class=&quot;math-container&quot;&gt;$R = h(X)-Y$&lt;/span&gt;, the part of &lt;span class=&quot;math-container&quot;&gt;$\mathbf{y}$&lt;/span&gt; that your model cannot explain, follows the Gaussian distribution &lt;span class=&quot;math-container&quot;&gt;$\mathcal{N}(\mu,\sigma)$&lt;/span&gt;. We're using capitals because we're talking about random variables now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Gaussian distribution has two parameters, mean &lt;span class=&quot;math-container&quot;&gt;$\mu = \mathbb{E}[R] = \frac{1}{m} \sum_i h_\theta(X^{(i)})-Y^{(i))}$&lt;/span&gt; and variance &lt;span class=&quot;math-container&quot;&gt;$\sigma^2 = E[R^2] = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$&lt;/span&gt;. See &lt;a href=&quot;https://math.stackexchange.com/questions/518281/how-to-derive-the-mean-and-variance-of-a-gaussian-random-variable&quot;&gt;here&lt;/a&gt; to understand these terms better.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider &lt;span class=&quot;math-container&quot;&gt;$\mu$&lt;/span&gt;, it is the &lt;em&gt;systematic error&lt;/em&gt; of our measurements. Use &lt;span class=&quot;math-container&quot;&gt;$h'(x) = h(x) - \mu$&lt;/span&gt; to correct for systematic error, so that &lt;span class=&quot;math-container&quot;&gt;$\mu' = \mathbb{E}[R']=0$&lt;/span&gt; (exercise for the reader). Nothing else to do here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; represents the &lt;em&gt;random error&lt;/em&gt;, also called &lt;em&gt;noise&lt;/em&gt;. Once we've taken care of the systematic noise component as in the previous point, the best predictor is obtained when &lt;span class=&quot;math-container&quot;&gt;$\sigma^2 = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$&lt;/span&gt; is minimized. Put another way, the best predictor is the one with the tightest distribution (smallest variance) around the predicted value, i.e. smallest variance. &lt;strong&gt;Minimizing the the least squared loss is the same thing as minimizing the variance!&lt;/strong&gt; That explains why the least squared loss works for a wide range of problems. The underlying noise is very often Gaussian, because of the CLT, and minimizing the squared error turns out to be the &lt;em&gt;right&lt;/em&gt; thing to do!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To simultaneously take both the mean and variance into account, we include a &lt;em&gt;bias&lt;/em&gt; term in our classifier (to handle systematic error &lt;span class=&quot;math-container&quot;&gt;$\mu$&lt;/span&gt;), then minimize the square loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Followup questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Least squares loss = Gaussian error. Does every other loss function also correspond to some noise distribution?&lt;/strong&gt; Yes. For example, the &lt;span class=&quot;math-container&quot;&gt;$\ell_1$&lt;/span&gt; loss (minimizing absolute value instead of squared error) corresponds to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Laplace_distribution&quot; rel=&quot;noreferrer&quot;&gt;Laplace distribution&lt;/a&gt; (Look at the formula for the PDF in the infobox -- it's just the Gaussian with &lt;span class=&quot;math-container&quot;&gt;$|x-\mu|$&lt;/span&gt; instead of &lt;span class=&quot;math-container&quot;&gt;$(x-\mu)^2$&lt;/span&gt;). A popular loss for probability distributions is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;noreferrer&quot;&gt;KL-divergence&lt;/a&gt;.&#xA;-The Gaussian distribution is very well motivated because of the &lt;strong&gt;Central Limit Theorem&lt;/strong&gt;, which we discussed earlier. When is the Laplace distribution the right noise model? There are some circumstances where it comes about naturally, but it's more commonly as a regularizer &lt;a href=&quot;https://math.stackexchange.com/q/1904767/12467&quot;&gt;to enforce &lt;strong&gt;sparsity&lt;/strong&gt;&lt;/a&gt;: the &lt;span class=&quot;math-container&quot;&gt;$\ell_1$&lt;/span&gt; loss is the &lt;em&gt;least convex&lt;/em&gt; among all convex losses. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/14904/jan-van-der-vegt&quot;&gt;Jan&lt;/a&gt; mentions in the comments, the minimizer of &lt;em&gt;squared&lt;/em&gt; deviations is the mean and the minimizer of the sum of &lt;strong&gt;absolute&lt;/strong&gt; deviations is the &lt;strong&gt;median&lt;/strong&gt;. Why would we want to find the median of the residuals instead of the mean? Unlike the mean, the median isn't thrown off by one very large outlier. So, the &lt;span class=&quot;math-container&quot;&gt;$\ell_1$&lt;/span&gt; loss is used for increased robustness. Sometimes a combination of the two is used.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Are there situations where we minimize both the Mean and Variance?&lt;/strong&gt; Yes. Look up &lt;a href=&quot;https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff&quot; rel=&quot;noreferrer&quot;&gt;Bias-Variance Trade-off&lt;/a&gt;. Here, we are looking at a set of classifiers &lt;span class=&quot;math-container&quot;&gt;$h_\theta \in H$&lt;/span&gt; and asking which among them is best. If we ask which &lt;em&gt;set&lt;/em&gt; of classifiers is the best for a problem, minimizing both the bias and variance becomes important. It turns out that there is always a trade-off between them and we use &lt;strong&gt;regularization&lt;/strong&gt; to achieve a compromise.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Regarding the &lt;span class=&quot;math-container&quot;&gt;$\frac{1}{2}$&lt;/span&gt; term&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The 1/2 does not matter and actually, neither does the &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; - they're both constants. The optimal value of &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt; would remain the same in both cases. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The expression for the gradient becomes prettier with the &lt;span class=&quot;math-container&quot;&gt;$\frac{1}{2}$&lt;/span&gt;, because the 2 from the square term cancels out.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When writing code or algorithms, we're usually concerned more with the gradient, so it helps to keep it concise. You can check progress just by checking the norm of the gradient. The loss function itself is sometimes omitted from code because it is used only for validation of the final answer.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; is useful if you solve this problem with gradient descent. Then your gradient becomes the average of &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; terms instead of a sum, so its' scale does not change when you add more data points.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've run into this problem before: I test code with a small number of points and it works fine, but when you test it with the entire dataset there is loss of precision and sometimes over/under-flows, i.e. your gradient becomes &lt;code&gt;nan&lt;/code&gt; or &lt;code&gt;inf&lt;/code&gt;. To avoid that, just normalize w.r.t. number of data points.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;These aesthetic decisions are used here to maintain consistency with future equations where you'll add &lt;strong&gt;regularization&lt;/strong&gt; terms. If you include the &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt;, the regularization parameter &lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt; will not depend on the dataset size &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; and it will be more interpretable across problems.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13505" LastEditorUserId="23546" LastEditDate="2019-02-25T11:03:05.913" LastActivityDate="2019-02-25T11:03:05.913" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="10193" PostTypeId="1" AcceptedAnswerId="10196" CreationDate="2016-02-11T01:47:47.487" Score="8" ViewCount="25519" Body="&lt;p&gt;I'm trying to work out if I'm correctly interpreting a decision tree found online.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The dependent variable of this decision tree is Credit Rating which&#xA;has two classes, Bad or Good. The root of this tree contains all 2464&#xA;observations in this dataset.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The most influential attribute to determine how to classify a good or bad credit rating is the Income Level attribute.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The majority of the people (454 out of 553) in our sample that had a less than low income also had a bad credit rating. If I was to launch a premium credit card without a limit I should ignore these people.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I were to use this decision tree for predictions to classify new observations, are the largest number of class in a leaf used as the prediction? E.g. Observation x has medium income, 7 credit cards and 34 years old. Would the predicted classification for credit rating = &quot;Good&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another new observation could be Observation Y, which has less than low income so their credit rating = &quot;Bad&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is this the correct way to interpret a decision tree or have I got this completely wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QI0QU.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QI0QU.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16179" LastEditorUserId="11097" LastEditDate="2016-02-11T02:22:52.397" LastActivityDate="2016-02-11T06:34:10.360" Title="How to interpret a decision tree correctly?" Tags="|predictive-modeling|decision-trees|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10194" PostTypeId="2" ParentId="10189" CreationDate="2016-02-11T01:55:10.150" Score="6" Body="&lt;p&gt;In that link you posted, you can look at the python full solution &lt;a href=&quot;http://www.cs.berkeley.edu/~rxin/ampcamp-ecnu/machine-learning-with-spark.html#solution_11&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; at the end and go through it to see what all is distributed. &#xA;In short, some parts are distributed, like reading data from the file, but the very important parts like the distance computation are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running down, we see:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;sc = SparkContext(&quot;local[6]&quot;, &quot;PythonKMeans&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This instantiates the context and creates a local cluster which the jobs will be submitted to&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;lines = sc.textFile(..)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is still setting up. No operations have taken place yet. You can verify this by putting timing statements in the code&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;data = lines.map(lambda x: (x.split(&quot;#&quot;)[0], parseVector(x.split(&quot;#&quot;)[1])))&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The lambda here will be applied to lines, so this operation will split the file in parallel. Note that the actual line also has a &lt;code&gt;cache()&lt;/code&gt; at the end (see &lt;a href=&quot;http://spark.apache.org/docs/latest/quick-start.html#caching&quot; rel=&quot;nofollow&quot;&gt;cache&lt;/a&gt;]). &lt;code&gt;data&lt;/code&gt; is just a reference to the spark object in memory. (I may be wrong here, but I think the operation still doesn't happen yet)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;count = data.count()&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This forces the parallel computation to start, and the count to be stored. At the end, the reference data is still valid, and we'll use it for further computations. I'll stop with detailed explanations here, but wherever &lt;code&gt;data&lt;/code&gt; is being used is a possible parallel computation. The python code itself is single threaded, and interfaces with the Spark cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An interesting line is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;tempDist = sum(np.sum((centroids[x] - y) ** 2) for (x, y) in newCentroids.iteritems())&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;centroids&lt;/code&gt; is an object in python memory, as is &lt;code&gt;newCentroids&lt;/code&gt;. So, at this point, all computations are being done in memory (and &lt;strong&gt;on the client&lt;/strong&gt;, typically clients are slim, i.e. have limited capabilities, or the client is an SSH shell, so the computers resources are shared. You should ideally &lt;em&gt;never&lt;/em&gt; do any computation here), so no parallelization is being used. You could optimize this method further by doing this computation in parallel. Ideally you want the python program to never directly handle individual points' $x$ and $y$ values.&lt;/p&gt;&#xA;" OwnerUserId="13505" LastEditorUserId="13505" LastEditDate="2016-02-11T02:18:15.983" LastActivityDate="2016-02-11T02:18:15.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10195" PostTypeId="2" ParentId="10193" CreationDate="2016-02-11T02:12:34.163" Score="2" Body="&lt;p&gt;Yes, your interpretation is correct. Each level in your tree is related to one of the variables (this is not always the case for decision trees, you can imagine them being more general). &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;X has medium income, so you go to Node 2, and more than 7 cards, so you go to Node 5. Now, you've reached a leaf node. you see that in your dataset, you had 54 people like X, who you determined has a Bad rating (a human presumably did this rating based on other factors. And you had 336 people like X who had a Good rating. So, based on only this information, you can say X probably has a Good rating. So, the decision tree has given you a quick, though approximate answer.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Side point: The 54 vs 336 here gives you a measure of confidence. For example, you could treat it like a probability. You could say $P(Good)=336/(54+336) \approx 0.86$, and also calculate a confidence interval in various ways.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Y has Low income, so you can immediately look at the tree and go to Node 1, and say he probably has a Bad rating, with $P(Bad)=454/(454+99) \approx 0.82$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Regarding the comment about &quot;most influential&quot; attribute, this really depends on the way the tree is constructed, and what definition of &quot;influential&quot; you use. So, you would have to ask the person/software/algorithm that made the tree. It is certainly an important attribute, as you can see from the table itself. &lt;/p&gt;&#xA;" OwnerUserId="13505" LastActivityDate="2016-02-11T02:12:34.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10196" PostTypeId="2" ParentId="10193" CreationDate="2016-02-11T02:22:10.373" Score="4" Body="&lt;p&gt;Let me evaluate each of your observations one by one, so that it would be more clear:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The dependent variable of this decision tree is Credit Rating which&#xA;  has two classes, Bad or Good. The root of this tree contains all 2464&#xA;  observations in this dataset.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If &lt;code&gt;Good, Bad&lt;/code&gt; is what you mean by credit rating, then &lt;strong&gt;Yes&lt;/strong&gt;. And you are right with the conclusion that all the 2464 observations are contained in the root of the tree.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The most influential attribute to determine how to classify a good or&#xA;  bad credit rating is the Income Level attribute.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Debatable&lt;/strong&gt; Depends on how you consider something to be &lt;em&gt;influential&lt;/em&gt;. Some might argue that the number of cards might be the most influential, and some might agree with your point. So, you are both right and wrong here.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The majority of the people (454 out of 553) in our sample that had a&#xA;  less than low income also had a bad credit rating. If I was to launch&#xA;  a premium credit card without a limit I should ignore these people.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, but it would also be better if you consider the probability of getting a bad credit from these people. But, even that would turn out to be NO for this class, which makes your observation correct again.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I were to use this decision tree for predictions to classify new&#xA;  observations, are the largest number of class in a leaf used as the&#xA;  prediction? E.g. Observation x has medium income, 7 credit cards and&#xA;  34 years old. Would the predicted classification for credit rating =&#xA;  &quot;Good&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Depends on the probability&lt;/strong&gt;. So, &lt;a href=&quot;https://social.msdn.microsoft.com/Forums/sqlserver/en-US/97c9ce39-024f-450f-8b21-a2d2961d8be7/decision-trees-how-is-prediction-probability-calculated?forum=sqldatamining&quot; rel=&quot;nofollow&quot;&gt;calculate the probability&lt;/a&gt; from the leaves and then make a decision depending on that. Or much simpler, use a library like the Sklearn's decision tree classifier to do that for you.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Another new observation could be Observation Y, which has less than&#xA;  low income so their credit rating = &quot;Bad&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Again, same as the explanation above.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is this the correct way to interpret a decision tree or have I got&#xA;  this completely wrong?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, this is a correct way of interpreting decision trees. You might be tempted to sway when it comes to selection of &lt;em&gt;influential&lt;/em&gt; variables, but that is dependant on a lot of factors, including the problem statement, construction of the tree, analyst's judgement, etc.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-02-11T06:34:10.360" LastActivityDate="2016-02-11T06:34:10.360" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10197" PostTypeId="1" AcceptedAnswerId="10285" CreationDate="2016-02-11T09:47:09.200" Score="3" ViewCount="359" Body="&lt;p&gt;I would like to develop a soccer field segmentation method. For this purpose, I prepared a training image data set and annotated field and non-field pixels. Following is a gr-chromacity plot of all training samples, colored with respect to their labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QObI3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QObI3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train a classifier for inferring the label of a new sample. The first approach comes to my mind is using Gaussian mixture models to model both distributions. Would you recommend another method for this purpose?&lt;/p&gt;&#xA;" OwnerUserId="16195" LastActivityDate="2016-02-17T17:37:15.280" Title="Soccer Field Segmentation" Tags="|machine-learning|classification|statistics|svm|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10198" PostTypeId="1" CreationDate="2016-02-11T10:02:59.413" Score="0" ViewCount="275" Body="&lt;p&gt;What are the source and target functions in igraph package in R?  so that source(s-&gt;t) = s and target(s-&gt;t) = t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on this directed graph example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(igraph, quietly=TRUE)&#xA;bsk &amp;lt;-read.table(&quot;http://www.dimiter.eu/Data_files/edgesdata3.txt&quot;, sep='\t', dec=',', header=T)&#xA;&#xA;bsk.network &amp;lt;- graph.data.frame(bsk, directed=T) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;edit: I am trying to write a function like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f_source &amp;lt;- function(str,network){&#xA; result &amp;lt;- head_of(network, E(network)[str]); &#xA;}&#xA;&#xA;f_source(AA%-&amp;gt;%DD,bsk.network)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is the string. It includes quotations while E(network)[str] does not work with quotations.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2016-02-11T18:21:28.053" LastActivityDate="2016-02-11T18:21:28.053" Title="source and target functions in igraph package in R" Tags="|r|graphs|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10199" PostTypeId="1" CreationDate="2016-02-11T11:08:06.757" Score="2" ViewCount="67" Body="&lt;p&gt;[If this is the wrong place to ask this question, please let me know where I can receive better answers:)]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm building an internal tool for an digital media organization. The tool aims to track the number of hours (or any other measure of effort) spent on each brand by different people-roles (Eg: Designers, Content Writers, etc). Using the data collected for the first few months, the organization aims to achieve a better estimate for future quotes given to new clients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought about the following roles and their metrics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Designers: Hours&lt;/li&gt;&#xA;&lt;li&gt;Content Writers: Hours&lt;/li&gt;&#xA;&lt;li&gt;SEO Guys: Hours&lt;/li&gt;&#xA;&lt;li&gt;Client-Facing people. (Who actually talk to all the clients): Effort on a scale of 1-10&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Whenever somebody starts work on a client (brand), they will start their timer and when done, they'll stop the timer. The tool will log the amount of time spent. Also, the current quotes to the different brands will be entered into the system initially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think about this as a regression problem with the hours put in by different roles as the features and the 'quote' as the output. But I don't think there will be enough data collected in the first 2 months to give a realistic estimate for quotes to new clients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can my cost function be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be my first project of this kind. Please give me some ideas. Even new approaches are welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="16197" LastActivityDate="2016-02-11T18:29:38.173" Title="Ideas on cost function" Tags="|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10200" PostTypeId="1" CreationDate="2016-02-11T13:36:17.973" Score="1" ViewCount="120" Body="&lt;p&gt;I am working on a project which processes big data (size ~3TB) every day. The first stage of our data pipeline copies the data from our MS-SQL Server to a host machine(linux server) using a tool called &lt;code&gt;freebcp&lt;/code&gt;. More info on freebcp is &lt;a href=&quot;http://linux.die.net/man/1/freebcp&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this tool from the linux server, we run a set of stored procedures on SQL Server and export the data and transfer it in bulk. Recently I am observing that if the data is huge (~200GB), the data transfer is getting stalled after sometime. I ran a couple of commands (&lt;code&gt;sp_who2&lt;/code&gt;, and &lt;code&gt;dbcc inputbuffer(spid)&lt;/code&gt;) to monitor the execution of stored procedure on SQL server. We observe the CPU time and Disk IO utilized by this procedure. If this is not changing for a few minutes, we assume the job is stalled, and manually kill these stored procedures to continue our data processing tasks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the probable reasons for this stalling of data copy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any better way to copy the data in bulk from SQL server to linux host? May be an alternative to freebcp. After this, we load this data into hadoop file system run our map reduce tasks.&lt;/p&gt;&#xA;" OwnerUserId="16203" LastActivityDate="2016-02-11T13:36:17.973" Title="freebcp getting stalled for huge data" Tags="|bigdata|apache-hadoop|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10201" PostTypeId="2" ParentId="10184" CreationDate="2016-02-11T14:03:27.103" Score="1" Body="&lt;p&gt;9 and 10 have different schema. For example, a displaced transverse fracture of the right femur would be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;821.01RT in 9&#xA;S72.321 in 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;10 is also more specific. In the example, 9 doesn't tell you that the fracture is transverse and displaced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since ICD10 is basically a better organized and more precise superset of the data represented by ICD9, someone has probably created a mapping. However, I haven't looked into that. &lt;/p&gt;&#xA;" OwnerUserId="13643" LastActivityDate="2016-02-11T14:03:27.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10202" PostTypeId="2" ParentId="10103" CreationDate="2016-02-11T15:00:59.437" Score="6" Body="&lt;p&gt;Since, there are no answers in SO, I have asked myself in github page and the issue has been closed by stating the following reply by GaelVaroquaux..&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you only want to parallelise vector operation, then you should use a&#xA;  build of numpy compiled with MKL (don't attempt to do it yourself, it's&#xA;  challenging).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;There could be approaches to high-level parallelism in the algorithm&#xA;  itself, which would probably lead to larger gains. However, after a quick&#xA;  look at the code, I didn't see any clear way of doing that.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;I am going to ahead and close this issue, as it is more of a blue-sky&#xA;  whish list. I completely agree, I would like TSNE to go faster, and it&#xA;  would be great is parallelism was easy. But in the current state of&#xA;  affairs, more work is required to be in a state where we can tackle such&#xA;  wish list.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-02-11T15:00:59.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10203" PostTypeId="2" ParentId="10199" CreationDate="2016-02-11T15:32:05.060" Score="1" Body="&lt;p&gt;In that great &lt;a href=&quot;http://www.joelonsoftware.com/items/2007/10/26.html&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt;, Joel Spolsky use Monte-Carlo simulation to give an estimation of the end of a project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thinking about a similar concept whould be to replace the notion of velocity by efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$efficiency = \frac{quote}{hours} $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more discrete the scales of the quotes, the better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should work even with a few weeks of data.&lt;/p&gt;&#xA;" OwnerUserId="15543" LastEditorUserId="15543" LastEditDate="2016-02-11T18:29:38.173" LastActivityDate="2016-02-11T18:29:38.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10204" PostTypeId="1" AcceptedAnswerId="10207" CreationDate="2016-02-11T16:35:19.590" Score="9" ViewCount="3877" Body="&lt;p&gt;When implementing mini-batch gradient descent for neural networks, is it important to take random elements in each mini-batch? Or is it enough to shuffle the elements at the beginning of the training once?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm also interested in sources which definitely say what they do.)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2020-08-05T19:48:35.610" Title="Should I take random elements for mini-batch gradient descent?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10205" PostTypeId="2" ParentId="10198" CreationDate="2016-02-11T17:22:39.867" Score="1" Body="&lt;p&gt;Given&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(igraph)&#xA;set.seed(1)&#xA;g &amp;lt;- sample_pa(5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you got&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ends(g, E(g))&#xA;#      [,1] [,2]&#xA;# [1,]    2    1&#xA;# [2,]    3    1&#xA;# [3,]    4    1&#xA;# [4,]    5    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head_of(g, E(g))&#xA;# + 4/5 vertices:&#xA;# [1] 2 3 4 5&#xA;&#xA;tail_of(g, E(g))&#xA;# + 4/5 vertices:&#xA;# [1] 1 1 1 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;source()&lt;/code&gt; is an important base function. &lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-02-11T17:22:39.867" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10206" PostTypeId="2" ParentId="9943" CreationDate="2016-02-11T17:40:00.273" Score="3" Body="&lt;p&gt;Assuming the rest of your configuration is correct all you have to do is to make &lt;code&gt;spark-csv&lt;/code&gt; jar available to your program. There are a few ways you can achieve this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;manually download required jars including  &lt;a href=&quot;http://central.maven.org/maven2/com/databricks/spark-csv_2.10/1.3.0/spark-csv_2.10-1.3.0.jar&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;spark-csv&lt;/code&gt;&lt;/a&gt; and csv parser (for example &lt;a href=&quot;http://mvnrepository.com/artifact/org.apache.commons/commons-csv/1.2&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;org.apache.commons.commons-csv&lt;/code&gt;&lt;/a&gt;) and put them somewhere on the &lt;code&gt;CLASSPATH&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;using &lt;code&gt;--packages&lt;/code&gt; option (use Scala version which has been used to build Spark. Pre-built versions use 2.10):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;using &lt;code&gt;PYSPARK_SUBMIT_ARGS&lt;/code&gt; environmental variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;export PACKAGES=&quot;com.databricks:spark-csv_2.11:1.3.0&quot;&#xA;export PYSPARK_SUBMIT_ARGS=&quot;--packages ${PACKAGES} pyspark-shell&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;adding Gradle string to &lt;code&gt;spark.jars.packages&lt;/code&gt; in &lt;code&gt;conf/spark-defaults.conf&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;spark.jars.packages    com.databricks:spark-csv_2.11:1.3.0&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12632" LastActivityDate="2016-02-11T17:40:00.273" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10207" PostTypeId="2" ParentId="10204" CreationDate="2016-02-11T17:42:35.960" Score="7" Body="&lt;p&gt;It should be enough to shuffle the elements at the beginning of the training and then to read them sequentially. This really achieves the same objective as taking random elements every time, which is to break any sort of predefined structure that may exist in your original dataset (e.g. all positives in the beginning, sequential images, etc).&lt;/p&gt;&#xA;&lt;p&gt;While it would work to fetch random elements every time, this operation is typically not optimal performance-wise. Datasets are usually large and are not saved in your memory with fast random access, but rather in your slow HDD. This means sequential reads are pretty much the only option you have for good performance.&lt;/p&gt;&#xA;&lt;p&gt;Caffe for example uses LevelDB, which does not support efficient random seeking. See &lt;a href=&quot;https://github.com/BVLC/caffe/issues/1087&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, which confirms that the dataset is trained with images always in the same order.&lt;/p&gt;&#xA;" OwnerUserId="15234" LastEditorUserId="98307" LastEditDate="2020-08-05T19:48:35.610" LastActivityDate="2020-08-05T19:48:35.610" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10209" PostTypeId="1" CreationDate="2016-02-11T19:21:28.653" Score="2" ViewCount="70" Body="&lt;p&gt;I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like &quot;Cant update application&quot; is tagged with #Apps #General #CantUpdate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My categories exist in 3 different levels.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Level1  Level2  Level3      Description&#xA;&#xA;Apps    General CantUpdate   &quot;Cannot update applications&quot;&#xA;Network Wifi    CantConnect  &quot;Cant connect to internet&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to perform multiple tag categorization.&#xA;I was thinking in have multiple lines for this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Tag     Description&#xA;&#xA;Apps   &quot;Cannot update applications&quot;&#xA;General &quot;Cannot update applications&quot;&#xA;CantUpdate &quot;Cannot update applications&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any better idea?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-02-12T21:52:32.393" LastActivityDate="2016-02-12T21:52:32.393" Title="Tag categorizer" Tags="|data|categorical-data|google|google-prediction-api|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10210" PostTypeId="2" ParentId="10151" CreationDate="2016-02-11T19:51:36.483" Score="6" Body="&lt;p&gt;In a machine learning context, a &lt;a href=&quot;https://en.wikipedia.org/wiki/Dichotomy&quot; rel=&quot;noreferrer&quot;&gt;dichotomy&lt;/a&gt; is simply a split of a set into two mutually exclusive subsets whose union is the original set. The point being made in your quoted text is that for four points, a linear boundary can not form all possible dichotomies (i.e., it does not &lt;a href=&quot;https://en.wikipedia.org/wiki/Shattered_set&quot; rel=&quot;noreferrer&quot;&gt;shatter&lt;/a&gt; the set). For example, if the four points are arranged on the corners of a square, a linear boundary can be used to create all possible dichotomies &lt;em&gt;except&lt;/em&gt; it cannot produce a boundary that splits the two points lying along one diagonal from the other two points (and vice versa), as you indicated in your own answer.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-02-11T19:51:36.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10211" PostTypeId="1" AcceptedAnswerId="10795" CreationDate="2016-02-11T21:52:50.030" Score="4" ViewCount="7499" Body="&lt;p&gt;Does anybody knows python library to retrieve sentiment from Russian text. The dictionary with sentiment parameterization will be ok to. The idea of library something like in GPOMS in &lt;a href=&quot;http://arxiv.org/abs/1010.3003&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9992" LastEditorUserId="31513" LastEditDate="2017-05-11T14:26:24.317" LastActivityDate="2019-01-10T18:52:31.710" Title="Sentiment retriving from text (Russian)" Tags="|python|nlp|sentiment-analysis|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10212" PostTypeId="1" CreationDate="2016-02-11T22:21:42.730" Score="2" ViewCount="71" Body="&lt;p&gt;I'm searching a lot about what people call Data Scientist. From my brief search, I've came with the conclusion that Data Scientists' are probably what you may call the future evolution of a Business Intelligence Professional, by focusing important topics such has machine learning, big data, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this search It's being difficult to define what may be the most important sectors, departments (inside companies) and main profiles to this subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone help me on understanding what is happening to the business world around this subject?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Most important sectors? (for example, is it the retail and bank sector? There are more?)&lt;/li&gt;&#xA;&lt;li&gt;Most important departments? (are the majority of Data Scientists working on IT departments? Risk departments? Marketing? I have no idea.)&lt;/li&gt;&#xA;&lt;li&gt;Most important topics about the profile of a Data Scientist (what are the imperative skills to have?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16217" LastActivityDate="2016-02-11T22:21:42.730" Title="Data Scientists | Sectors , Departments and Profiles" Tags="|machine-learning|data-mining|bigdata|data|" AnswerCount="0" CommentCount="0" ClosedDate="2016-02-12T15:11:55.073" ContentLicense="CC BY-SA 3.0" />
  <row Id="10213" PostTypeId="1" CreationDate="2016-02-11T22:58:56.057" Score="1" ViewCount="77" Body="&lt;p&gt;I have a dataset with 10 columns and 158 rows. I try to predict my test dataset which is 1 columns with 158 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made cross-validations, grid-search and use ElasticNet algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also before the evaluate the model I check the pearson correlation between 10 columns which I used for train with other 1 column which I try to predict. The correlation is not good but when I evaluate model the R^2 score is near 0.98 . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I be ensure that this score is confidental ? Because I didn't expect a R^2 like this. This is too high that I expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="16218" LastActivityDate="2016-02-13T17:32:20.820" Title="How can I ensure about my R^2 score?" Tags="|machine-learning|data-mining|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10214" PostTypeId="2" ParentId="10106" CreationDate="2016-02-12T01:24:49.527" Score="1" Body="&lt;p&gt;The question is not really clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the question is how can you save and retrieve the cosine-similarity of each word with every other word, then you can save them in an array and use &lt;a href=&quot;https://wiki.python.org/moin/UsingPickle&quot; rel=&quot;nofollow&quot;&gt;pickle&lt;/a&gt; for saving and loading them again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, If the question is how you can retrieve the &lt;code&gt;gensim model&lt;/code&gt;, then the function is &lt;code&gt;gensim.models.word2vec.Word2Vec.load()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-02-12T01:24:49.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10216" PostTypeId="1" AcceptedAnswerId="10230" CreationDate="2016-02-12T02:22:01.940" Score="22" ViewCount="17774" Body="&lt;p&gt;I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you need to have each sentence / paragraph / document with its own unique label (e.g. &quot;Sent_123&quot;)? This seems useful if you want to say &quot;what words or sentences are most similar to a single specific sentence labeled &quot;Sent_123&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you have the labels be repeated based on content? For example if each sentence / paragraph / document is about a certain product item (and there are multiple sentence / paragraph / document for a given product item) can you label the sentences based on the item and then compute the similarity between a word or a sentence and this label (which I guess would be like an average of all those sentences that had to do with the product item)?&lt;/p&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2016-02-12T22:27:54.073" Title="Doc2Vec - How to label the paragraphs (gensim)" Tags="|machine-learning|text-mining|word-embeddings|word2vec|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10217" PostTypeId="2" ParentId="6169" CreationDate="2016-02-12T03:19:49.117" Score="3" Body="&lt;p&gt;I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine: &#xA;SPARK_HOME: C:\spark-1.6.0-bin-hadoop2.6\&#xA;I use Enthought Canopy, so Python is already integrated in my system path. Next, launch Python or IPython and use the following code. If you get an error, check what you get for 'spark_home'. Otherwise, it should run just fine.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;&#xA;import sys&#xA;&#xA;spark_home = os.environ.get('SPARK_HOME', None)&#xA;&#xA;if not spark_home:&#xA;&#xA;raise ValueError('SPARK_HOME environment variable is not set')&#xA;&#xA;sys.path.insert(0, os.path.join(spark_home, 'python'))&#xA;&#xA;sys.path.insert(0, os.path.join(spark_home, 'C:/spark-1.6.0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip')) ## may need to adjust on your system depending on which Spark version you're using and where you installed it.&#xA;&#xA;execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ou0CE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ou0CE.png&quot; alt=&quot;pySpark on IPython&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2016-02-12T03:19:49.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10218" PostTypeId="2" ParentId="10063" CreationDate="2016-02-12T06:33:58.687" Score="8" Body="&lt;p&gt;I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you've ever gone hiking or mountain climbing on a day with fog or a low cloud cover, there are times where you get to the top of the peak and can only see the surrounding peaks poking up through the clouds.  I like to use this analogy when I think of DBSCAN.  The density filtering allows one to select a threshold where data is kept, and all of the remaining data is filtered out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at this Seattle crime incident data.  Suppose I want to cluster the data by location to form pseudo neighborhoods i.e. neighborhoods that are roughly defined the the geography of where criminal events tend to occur. This is an example of k-means working just fine in location analytics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Seattle crime data superimposing a primitive map with k-means clustering&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3DhWo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3DhWo.png&quot; alt=&quot;Seattle Crime data clustered with k-means&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know Seattle at all, you can see that the clusters tend to pick out neighborhoods and this works quite well in breaking up neighborhoods.  Now suppose I want to pick out the high crime areas of Seattle in order to identify the hot spots. No matter how I adjust k, the k-means clustering doesn't really provide any additional insight.  But the density filtering in DBSCAN does a wonderful job of identifying high crime areas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Seattle crime data superimposing a primitive map with DBSCAN clustering&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/J2WSO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J2WSO.png&quot; alt=&quot;Seattle Crime data clustered with DBSCAN&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this gives the jest of some strengths that can be exploited using the two algorithms.  There's nothing special about the crime data.  For what its worth, I have a &lt;a href=&quot;http://angusmacnab.com/Location_SKLearn_Tut_MSData_0.2.html&quot; rel=&quot;noreferrer&quot;&gt;tutorial that goes through the analytics of identifying a user's home and work locations from their cell phone GPS pings&lt;/a&gt;. This is another case where DBSCAN was very useful, but the DBSCAN specific parts a buried a ways down in the tutorial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-02-12T06:33:58.687" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10219" PostTypeId="1" CreationDate="2016-02-12T07:27:22.183" Score="2" ViewCount="1129" Body="&lt;p&gt;Now, I have enormous number of resumes of someone who search for next jobs.&#xA;There are infomation about how old they are, what schools they graduated,and &#xA;what jobs they experienced.etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to make a recommender system using those data.&#xA;In the system, if I get new resume, I want to recommend certain jobs for him.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some idea what models will I use, for example,&#xA;collaborative filtering, recurrent neural network, word2vec etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I couldn't find best method for this problem.&#xA;If someone Knows good idea, please tell me that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an example of one data, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[age,education,skills,past_job1,job1's occupational category, past_job2,job2's occupational category,&#xA;...,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All data is stored in string.&lt;/p&gt;&#xA;" OwnerUserId="16101" LastActivityDate="2017-08-06T00:35:20.780" Title="What model should I choose for Jobs recommender system?" Tags="|python|nlp|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10222" PostTypeId="2" ParentId="10213" CreationDate="2016-02-12T13:42:10.460" Score="2" Body="&lt;p&gt;$ R^2 $ shows what variation of your purpose variable is described by independent variables. So their synergetic effect could give you good better answer than their correlation. Better use $ R^2 adjusted $. Look at p-values of your variables and think about their real correlation. Are they important in real life. If they have any adequate relationship, so your regression is right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it will help.  &lt;/p&gt;&#xA;" OwnerUserId="16119" LastActivityDate="2016-02-12T13:42:10.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10223" PostTypeId="1" AcceptedAnswerId="10701" CreationDate="2016-02-12T15:25:52.077" Score="0" ViewCount="1176" Body="&lt;p&gt;I am using rpart package in order to create a segmentation of my data using decision tree. As final result I want to obtain a classification of my data. For exemple, if the rpart devide data into 3 classes, I want to divide my data onto this three subsets such that I know that row n°1 is in subset 1, row n°2 in subset 3, etc. I can't find how could I get this information from the results of rpart object?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tree.res &amp;lt;- rpart(x ~ ., data, method=&quot;class&quot;, parms=list(split = &quot;gini&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that I can retrieve the results of this classification as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(tree)    #plot the tree&#xA;text(tree)&#xA;printcp(tree) #Displays CP table for Fitted Rpart Object&#xA;predict(tree) #displays prediction results of x variable&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I get the information on the segmentation of my data set? As first attempt I used : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(tree, type = &quot;vector&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this a correct approach?&lt;/p&gt;&#xA;" OwnerUserId="16152" LastActivityDate="2016-03-14T15:44:26.867" Title="How to retrieve the clustering results of rpart" Tags="|machine-learning|r|clustering|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10224" PostTypeId="1" CreationDate="2016-02-12T17:00:51.213" Score="1" ViewCount="77" Body="&lt;p&gt;An example of some time series input I have in mind. Suppose I have multiple users working on a 20-minute search task, using a commercial search engine like Google, Yahoo, or Bing. It's the same task, but users may finish after 10 minutes if they feel they have completed the task to the best of their ability. The y-axis is a &quot;performance metric&quot; (like normalized discounted cumulative gain), and measures are taken either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;At each query&lt;/li&gt;&#xA;&lt;li&gt;At each minute&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both the # of queries and time are obviously not constant across users.  The goal here is for visualization/visual comparison, but if there's methods that are more suited to recommendation/prediction, I'm happy to hear those solutions, too!&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2016-02-12T17:00:51.213" Title="What are standard ways to normalize lengths for multiple time series?" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10225" PostTypeId="1" CreationDate="2016-02-12T17:52:48.120" Score="11" ViewCount="6527" Body="&lt;p&gt;I have a large SQL table that is essentially a log. The data is pretty complex and I'm trying to find some way to identify anomalies without me understanding all the data. I've found lots of tools for Anomaly Detection but most of them require a &quot;middle-man&quot; of sorts, ie Elastic Search, Splunk, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a tool that can run against a SQL table which builds a baseline and alerts of anomalies automagically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may sound lazy but I've spent dozens of hours writing individual reporting scripts as I learn what each event type means and which other fields go with each event and I don't feel any closer to being able to alert on real problems in a meaningful way. The table has 41 columns and just hit 500 million rows (3 years of data).&lt;/p&gt;&#xA;" OwnerUserId="16235" LastActivityDate="2016-06-23T18:20:35.073" Title="Tools for automatic anomaly detection on a SQL table?" Tags="|sql|anomaly-detection|outlier|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10227" PostTypeId="2" ParentId="10216" CreationDate="2016-02-12T21:12:01.790" Score="11" Body="&lt;p&gt;&lt;code&gt;doc2vec&lt;/code&gt; model gets its algorithm from &lt;code&gt;word2vec&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;code&gt;word2vec&lt;/code&gt; there is no need to label the words, because every word has their own semantic meaning in the vocabulary.&#xA;But in case of &lt;code&gt;doc2vec&lt;/code&gt;, there is a need to specify that how many number of words or sentences convey a semantic meaning, so that the algorithm could identify it as a single entity. For this reason, we are specifying &lt;code&gt;labels&lt;/code&gt; or &lt;code&gt;tags&lt;/code&gt; to sentence or paragraph depending on the level of semantic meaning conveyed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we specify a single label to multiple sentences in a paragraph, it means that all the sentences in the paragraph are required to convey the meaning. On the other hand, if we specify variable labels to all the sentences in a paragraph, it means that each conveys a semantic meaning and they may or may not have similarity among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In simple terms, a &lt;code&gt;label&lt;/code&gt; means semantic meaning of something.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-02-12T21:12:01.790" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10228" PostTypeId="1" CreationDate="2016-02-12T22:05:41.193" Score="112" ViewCount="138483" Body="&lt;p&gt;Can someone practically explain the rationale behind &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity&quot; rel=&quot;noreferrer&quot;&gt;Gini impurity&lt;/a&gt; vs &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Information_gain&quot; rel=&quot;noreferrer&quot;&gt;Information gain&lt;/a&gt; (based on Entropy)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Metrics&quot; rel=&quot;noreferrer&quot;&gt;metric&lt;/a&gt; is better to use in different scenarios while using decision trees?&lt;/p&gt;&#xA;" OwnerUserId="16236" LastEditorUserId="144646" LastEditDate="2023-07-15T17:39:20.247" LastActivityDate="2023-07-15T17:39:20.247" Title="When should I use Gini Impurity as opposed to Information Gain (Entropy)?" Tags="|machine-learning|decision-trees|information-theory|gini-index|entropy|" AnswerCount="9" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="10229" PostTypeId="1" CreationDate="2016-02-12T22:17:56.217" Score="1" ViewCount="42" Body="&lt;p&gt;I have a set of game play event data. One of our goals is to provide a picture of how much time the person has spent playing. This sounds simple but for the fact that events are often dropped and result in vast amounts of time between events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are in field trials right now and don't have a huge pool of data to use as reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have much experience in statics, though I'm a competent programmer. My question is: What might I do to account for the missing data? How do I ensure that these missing events don't skew my data?&lt;/p&gt;&#xA;" OwnerUserId="16238" LastActivityDate="2020-01-24T15:00:42.320" Title="dealing with dropped event data" Tags="|missing-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10230" PostTypeId="2" ParentId="10216" CreationDate="2016-02-12T22:27:54.073" Score="12" Body="&lt;p&gt;Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The TaggedDocument constructor takes a list of tags. (If you happen to limit yourself to to plain ints ascending from 0, the Doc2Vec model will use those as direct indexes into its backing array, and you'll save a lot of memory that would otherwise be devoted to a string -&gt; index lookup, which could be important for large datasets. But you can use string doctags or even a mixture of int and string doctags.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll have to experiment with what works best for your needs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some classification tasks, an approach that's sometimes worked better than I would have expected is skipping per-text IDs entirely, and just training the Doc2Vec model with known-class examples, with the desired classes as the doctags. You then get 'doc vectors' just for the class doctags – not every document – a potentially much smaller model. Later inferring vectors for new texts results in vectors meaningfully close to related class doc vectors.&lt;/p&gt;&#xA;" OwnerUserId="8278" LastActivityDate="2016-02-12T22:27:54.073" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10232" PostTypeId="1" CreationDate="2016-02-13T15:57:15.663" Score="1" ViewCount="2449" Body="&lt;p&gt;&lt;em&gt;Updated: see bottom of this question.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure if I am at the right channel here but I have good hopes someone here might be able to help me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to process and analyse data from a &lt;a href=&quot;http://www.rsscan.com/footscan/entry-level/#tab_products&quot; rel=&quot;nofollow&quot;&gt;Footscan system&lt;/a&gt; that is exported from the &lt;a href=&quot;http://www.rsscan.com/footscan/entry-level/#software-9&quot; rel=&quot;nofollow&quot;&gt;Footscan 9 Gait Essentials&lt;/a&gt; software to a .rsdb database. This database is a standard SQLite database with a different file extension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no problems accessing the data via Python but the most interesting data (the raw data from the sensor) is stored as a 'blob' (see &lt;a href=&quot;https://www.sqlite.org/datatype3.html&quot; rel=&quot;nofollow&quot;&gt;sqlite documentation&lt;/a&gt; for more details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exporting this data from SQLitePro to .csv shows that the data in the blob looks like this (although much longer):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;9c4e79cb cbd09de5 c380fc78 a74e819f ... ab2172c7 d9cf311a 357aebac&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is&lt;/strong&gt;: what would be my first steps to try to discover how this data is encoded? And does anyone maybe recognize this type of encoding?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extra hints: The data represents all forces of a force plate with 64x64 force sensors, sampled at 300Hz, so it's most probably a multidimensional array.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am making some progress. As commented below, the values are most probably hexadecimal values, but I'm not sure if the values are base 16: most values seem to be &gt;10^9...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore: I found a table in the database called 'Contact' which linkes to entries in the blob table which contain the most hexadecimal values (&gt;40000). My guess is that these are the entries which are recognized as actual steps. I extracted the data from one of these steps &lt;a href=&quot;https://www.dropbox.com/s/4ss303k624ihcos/rsdb_export_24?dl=0&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (Contact id 2; blob id 24). This blob contains 50792 hexadecimal values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the entry for this step in 'Contacts':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id: {35d9a73e-608d-4ba3-9823-de0540f11c93}&#xA;timestamp: 2&#xA;deleted: NULL&#xA;FrameWidth: 64&#xA;Frames: 24&#xA;FrameOffset: 12204&#xA;FrameCount: 214&#xA;FrameHeight: 64&#xA;FrameStoreWithContacts_Contacs: {5d10546e-5ef3-4469-b898-d7b9bd58458f}&#xA;OriginalId: {32ae0f73-8022-4151-92e7-510c0e86d09d}&#xA;orphan: false&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Full export of the blob table can be downloaded &lt;a href=&quot;https://www.dropbox.com/s/ihvpyw3fx2xhoqz/BlobTable.csv?dl=0&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&#xA;The full .rsdb can be found &lt;a href=&quot;https://www.dropbox.com/s/0pb449qbi2pzcnl/Aart_goossens_2016-02-10_20-11-27.rsdb?dl=0&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (you might have to rename to &lt;code&gt;.sqlite&lt;/code&gt;).&#xA;The tables 'Foot' and 'Region' in the full database might contain useful data too, but I'm not yet sure how...&lt;/p&gt;&#xA;" OwnerUserId="16249" LastEditorUserId="16249" LastEditDate="2016-02-16T20:42:47.213" LastActivityDate="2018-04-03T14:40:02.560" Title="Decode SQLite database blobs: how to start?" Tags="|databases|parsing|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="10233" PostTypeId="2" ParentId="10213" CreationDate="2016-02-13T17:32:20.820" Score="0" Body="&lt;p&gt;Try regression diagnostics. There are a couple of methods to help you make sense of the data and the model. Try &lt;a href=&quot;https://us.sagepub.com/en-us/nam/an-r-companion-to-applied-regression/book233899&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt;, go to resources tab and download Chapter 6 - Diagnosing Problems in Linear and Generalized Linear Models. The code is in R. However you can find the python equivalent of the code.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-02-13T17:32:20.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10234" PostTypeId="1" CreationDate="2016-02-13T18:35:09.057" Score="-1" ViewCount="321" Body="&lt;p&gt;I am looking for datasets that have the same columns and their content (rows) are different. When I run a decision tree (classification) I'll get different models (trees) for each of them. It can be data of customers of different bank or insurance companies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for clarification the datasets should have the following criteria:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Have same columns&#xA;2. Categorized target column (i.e. I can build decision tree)&#xA;3. Each dataset can have enough data (over 1000 tuples)&#xA;4. The decision trees that I create (2) from each dataset is different&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16251" LastEditorUserId="16251" LastEditDate="2016-02-18T17:51:06.513" LastActivityDate="2016-02-22T19:01:47.150" Title="Finding databases with same columns and different content/ model" Tags="|dataset|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="10236" PostTypeId="5" CreationDate="2016-02-14T02:46:41.803" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-02-14T02:46:41.803" LastActivityDate="2016-02-14T02:46:41.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10237" PostTypeId="4" CreationDate="2016-02-14T02:46:41.803" Score="0" Body="gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models." OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2016-02-14T13:51:13.687" LastActivityDate="2016-02-14T13:51:13.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10239" PostTypeId="1" CreationDate="2016-02-14T16:21:21.510" Score="2" ViewCount="166" Body="&lt;p&gt;&lt;em&gt;NOTE : I'm not sure if this is the right forum for this question. if not, please advice.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt; : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect the data at ~1second intervals for about 2 hours, which gives almost 7200 data sets. These are the parameters :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Timestamp (milliseconds)&lt;/li&gt;&#xA;&lt;li&gt;Latitude&lt;/li&gt;&#xA;&lt;li&gt;Longitude&lt;/li&gt;&#xA;&lt;li&gt;Speed&lt;/li&gt;&#xA;&lt;li&gt;Acceleration&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now I was looking at ways to simplify this data, as processing and rendering these many date points, especially on mobile devices is not a good idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT :&lt;/strong&gt; as answer to &lt;em&gt;Spacedman&lt;/em&gt;'s comment : I want to simplify the data because most of it is redundant. The data is collected every 1 second regardless of whether the values are constant or changing. Accuracy is not crucial as its only used for displaying visual graphs on a website, drawing a polyline on a map. etc. So I want to keep only the minimum necessary data points required to reproduce the graph/line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While searching, I came across the &lt;a href=&quot;https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ramer–Douglas–Peucker algorithm&lt;/a&gt; and also found a &lt;a href=&quot;http://mourner.github.io/simplify-js/&quot; rel=&quot;nofollow noreferrer&quot;&gt;library&lt;/a&gt; that implements this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The issue&lt;/strong&gt; : I am a bit confused as to how to simplify this data. I could :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Somehow simply the entire thing by considering each data as a 5D point&#xA;, &lt;strong&gt;or&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Generate three sets of arrays, namely : &lt;code&gt;[Lat, Long, time]&lt;/code&gt;, &lt;code&gt;[Speed, time]&lt;/code&gt; and &lt;code&gt;[acceleration, time]&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So my question is :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any advantages/disadvantages of using either of the two approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking that - as each metric will have different patterns of variations, will combining them reduce the efficiency of the whole simplification process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it best to keep the metrics separate, so that each will be simplified to its maximum efficiency?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a Java/Obj-C developer, normally active on &lt;a href=&quot;https://stackoverflow.com/users/3894781/shahim&quot;&gt;SO&lt;/a&gt;, and I am &lt;strong&gt;not&lt;/strong&gt; an expert on these things, So I would like to know what you guys think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="16266" LastEditorUserId="1330" LastEditDate="2022-04-18T18:38:06.573" LastActivityDate="2022-04-18T18:38:06.573" Title="Combining parameters for Douglas-Peucker Simplification" Tags="|dataset|data-cleaning|bigdata|javascript|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10244" PostTypeId="2" ParentId="10143" CreationDate="2016-02-15T02:05:23.760" Score="3" Body="&lt;p&gt;The University of Washington has released a project template for small scientific python projects (including data science projects) called &lt;a href=&quot;https://github.com/uwescience/shablona&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;shablona&lt;/code&gt;&lt;/a&gt;. Is that more or less what you're looking for?&lt;/p&gt;&#xA;" OwnerUserId="9351" LastEditorUserId="98307" LastEditDate="2020-08-03T15:11:52.947" LastActivityDate="2020-08-03T15:11:52.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10246" PostTypeId="2" ParentId="6084" CreationDate="2016-02-15T03:54:53.160" Score="8" Body="&lt;p&gt;Here is an R version:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The codes here seem outdated for ggplot2: 2.0.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try my package zmisc: &lt;code&gt;devtools:install_github(&quot;jerryzhujian9/ezmisc&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you install it, you will be able to run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = mtcars&#xA;df$model = rownames(mtcars)&#xA;&#xA;ez.radarmap(df, &quot;model&quot;, stats=&quot;mean&quot;, lwd=1, angle=0, fontsize=0.6, facet=T, facetfontsize=1, color=id, linetype=NULL)&#xA;ez.radarmap(df, &quot;model&quot;, stats=&quot;none&quot;, lwd=1, angle=0, fontsize=1.5, facet=F, facetfontsize=1, color=id, linetype=NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The main codes were adapted from &lt;a href=&quot;http://www.cmap.polytechnique.fr/~lepennec/R/Radar/RadarAndParallelPlots.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cmap.polytechnique.fr/~lepennec/R/Radar/RadarAndParallelPlots.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5PL78.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5PL78.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16267" LastEditorUserId="16267" LastEditDate="2016-04-18T18:44:14.403" LastActivityDate="2016-04-18T18:44:14.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10248" PostTypeId="1" AcceptedAnswerId="10251" CreationDate="2016-02-15T08:30:39.100" Score="0" ViewCount="89" Body="&lt;p&gt;As a total beginner I am trying to apply some &quot;predictions&quot; on top of a bunch of csv files which contains house transactions for the last 20 years divided per area. What I would like to predict is the trend of the transactions for lets say the next year for a specific area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What general steps would you follow, to analyse those data and then predict?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read different articles but what I am looking for it is a sort of &quot;best general practice&quot; for this sort of problem.&lt;/p&gt;&#xA;" OwnerUserId="16280" LastActivityDate="2016-02-15T09:42:51.110" Title="How to make data predictions" Tags="|data-mining|predictive-modeling|" AnswerCount="1" CommentCount="0" ClosedDate="2016-02-15T12:58:12.883" ContentLicense="CC BY-SA 3.0" />
  <row Id="10250" PostTypeId="1" AcceptedAnswerId="10263" CreationDate="2016-02-15T09:34:13.660" Score="13" ViewCount="11415" Body="&lt;p&gt;The title says it all: I have seen three terms for functions so far, that seem to be the same / similar:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;error function&lt;/li&gt;&#xA;&lt;li&gt;criterion function&lt;/li&gt;&#xA;&lt;li&gt;cost function&lt;/li&gt;&#xA;&lt;li&gt;objective function&lt;/li&gt;&#xA;&lt;li&gt;loss function&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I was working on classification problems&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $W$ are the weights, $E$ is the evaluation set, $t_x$ is the desired output (the class) of $x$ and $o(x)$ is the given output. This function&#xA;seems to be commonly called &quot;error function&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But while reading about this topic, I've also seen the terms &quot;criterion function&quot; and &quot;objective function&quot;. Do they all mean  the same for neural nets?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://class.coursera.org/neuralnets-2012-001/lecture/47&quot; rel=&quot;noreferrer&quot;&gt;Geoffrey Hinton&lt;/a&gt; called cross-entropy for softmax-neurons and $E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$ a &lt;em&gt;cost function&lt;/em&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-16T15:01:00.563" LastActivityDate="2020-06-02T09:29:53.200" Title="What is the difference between (objective / error / criterion / cost / loss) function in the context of neural networks?" Tags="|terminology|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10251" PostTypeId="2" ParentId="10248" CreationDate="2016-02-15T09:42:51.110" Score="2" Body="&lt;p&gt;Regression will work well if your data set is large, but only for predicting current house prices (say, for example, estimating the value of your house). That's what people generally mean when they talk about predicting house prices from current house sales data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question of how house prices will behave in the next year is much, but much more complicated, and would not depend simply on the data you currently have. You would need to involve other information and a much more complex model, which would need to involve things like current level of household debt, inflation rate, economic outlook, etc. Daunting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, speculative prices follow some stochastic process. They depend on the current value, but diverge more and more the farther you go in the future.&lt;/p&gt;&#xA;" OwnerUserId="16279" LastActivityDate="2016-02-15T09:42:51.110" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10253" PostTypeId="2" ParentId="10228" CreationDate="2016-02-15T12:17:55.757" Score="36" Body="&lt;p&gt;Generally, your performance will not change whether you use Gini impurity or Entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Laura Elena Raileanu and Kilian Stoffel compared both in &quot;&lt;a href=&quot;https://www.unine.ch/files/live/sites/imi/files/shared/documents/papers/Gini_index_fulltext.pdf&quot; rel=&quot;noreferrer&quot;&gt;Theoretical comparison between the gini index and information gain criteria&lt;/a&gt;&quot;. The most important remarks were:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It only matters in 2% of the cases whether you use gini impurity or entropy. &lt;/li&gt;&#xA;&lt;li&gt;Entropy might be a little slower to compute (because it makes use of the logarithm). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I was once told that both metrics exist because they emerged in different disciplines of science. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="39729" LastEditDate="2017-09-27T02:35:11.537" LastActivityDate="2017-09-27T02:35:11.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10256" PostTypeId="1" AcceptedAnswerId="10257" CreationDate="2016-02-15T15:52:00.317" Score="1" ViewCount="116" Body="&lt;p&gt;I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to measure fame for a large group of scientists. The quick method is to measure the amount of Google results when searching their name. I do not care about the results, only the number of them. That method has it's flaws I know, so I am not opposed to an alternative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My scientist data is composed of thousands of entries. Which is causing issues. I tried to programmatically search Google, but less then 1000 searches later they blocked the program. I also looked into their Search API, but that is limited to 100 searches a day unless I pay for more, but since I am a poor college student that isn't an option. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping someone here may be able to offer suggestions on build a dataset with some way of measuring fame. &lt;/p&gt;&#xA;" OwnerUserId="16291" LastActivityDate="2016-08-21T14:39:57.857" Title="Gathering the number of Google results from a large amount of searches." Tags="|dataset|search|google|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10257" PostTypeId="2" ParentId="10256" CreationDate="2016-02-15T16:52:04.893" Score="2" Body="&lt;p&gt;With any search engine you will be limited by number of requests and any way of outcoming those limits will be a gray zone of violation of end user agreement (and, eventually, you will get banned for some time, of course). You should be looking into Search APIs of known search engines, for example, Bing gives you 5000 searches per month for free which - for a proof of concept research - might be enough. Also, 5k/month will give you some 20-30k until summer, so your data set will become bigger while you will be polishing your idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Google's free tier search is limited to 100 requests per day. Which gives you completely legal 3k per month as well. Combined (given, you treat Google and Bing results as equal) you get 8k per month.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-02-15T16:52:04.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10258" PostTypeId="1" CreationDate="2016-02-15T17:41:46.373" Score="2" ViewCount="191" Body="&lt;p&gt;I have a dataset of organization names that is quite messy. I used all the popular NER tools on it without much success(I assume it's because they lack context). I resolved to using OpenRefine but I reached a dead end with it's filters not picking up a lot of similar strings. I would like to use the data that I cleaned so far with OpenRefine for a (preferably supervised) machine learning algorithm that can afterwards continue the cleaning.&lt;br&gt;&#xA;Are there any resources that could help with this? &lt;/p&gt;&#xA;" OwnerUserId="16294" LastEditorUserId="16284" LastEditDate="2016-04-12T20:49:58.253" LastActivityDate="2021-01-09T22:18:23.707" Title="Is there a tool/ library/ algorithm that can learn from the steps I perform in cleaning text and apply them to similar data?" Tags="|machine-learning|data-mining|data-cleaning|supervised-learning|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10259" PostTypeId="2" ParentId="9899" CreationDate="2016-02-16T03:53:46.017" Score="2" Body="&lt;p&gt;You could build an ARIMAX model. This would permit to include autoregressive (AR) terms as well as well as the sales in product 5 as an Exogenous Input (X). This would give you a potential model where the sales for a product $i$ at time $t$ is given by $s^i_t$ then,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s^1_t=\alpha_1 s^1_{t-1} + \alpha_2 s^1_{t-2} + \ldots + \beta_0 s^5_t + \beta_1 s^5_{t-1} + \ldots $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you may need to make the series stationary first, but see more on that below. You could estimate this model with the &lt;a href=&quot;https://cran.r-project.org/web/packages/seasonal/&quot; rel=&quot;nofollow noreferrer&quot;&gt;seasonal&lt;/a&gt; R package that relies on the &lt;a href=&quot;https://www.census.gov/srd/www/x13as/&quot; rel=&quot;nofollow noreferrer&quot;&gt;X-13ARIMA-SEATS software developed by the US Census Bureau&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend to ensure that your time series are all stationary, see for example &lt;a href=&quot;https://stats.stackexchange.com/questions/27332/how-to-know-if-a-time-series-is-stationary-or-non-stationary&quot;&gt;this post&lt;/a&gt; before you use X13. I would also run cointegration tests. See for more explanation this &lt;a href=&quot;http://www.econ.uiuc.edu/~econ508/R/e-ta8_R.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;excellent post&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you only have data on week 5 I would start modeling in week 5 but you could include autoregressive (AR) terms related to the sales of product 1 prior to week 5.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-02-16T04:06:23.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10260" PostTypeId="2" ParentId="8546" CreationDate="2016-02-16T04:04:00.967" Score="1" Body="&lt;p&gt;I have faced similar challenges. You could try to use the &lt;a href=&quot;https://learn.microsoft.com/en-us/machine-learning-server/r-reference/revoscaler/rxdtree&quot; rel=&quot;nofollow noreferrer&quot;&gt;RevolutionR rxDTree function&lt;/a&gt; which performs reasonably well on large data sets and can be converted to a rpart object.&lt;/p&gt;&#xA;&lt;p&gt;I found to achieve a reasonable performance boost by specifying the minimum bin size with &lt;code&gt;rpart.control(minbucket=2)&lt;/code&gt;. You could also tune the complexity parameter. Note that both may lead to poor performance.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="30186" LastEditDate="2023-01-05T17:25:23.797" LastActivityDate="2023-01-05T17:25:23.797" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10261" PostTypeId="1" CreationDate="2016-02-16T08:50:44.807" Score="2" ViewCount="130" Body="&lt;p&gt;I am reading the docs of MLlib, but I am new with it and I don't know which would be the best algorithm to predict prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen that with RandomRDDs I can use a generator for random numbers that follow a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, this is not good enought, since the prices I want to simulate doesn't follow a normal distribution. The demand changes depending on if its daytime or night, so the prices also change. The seasonality also affects the demand and influence the resulting prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of that, I would need a model that I can feed with historical data in order to generate new prices for simulations that have &quot;sense&quot; depending on the day of the year or the hour (day/night) I want to simulate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said, I'm looking for a solution with MLlib since I'm working with Spark. I'm blocked in finding the appropiate algorithm when reading the docs. I only would need a name or a few names in order to continue investigating (I, myself, can enrich later the answer in order to help other people that reach this post)&lt;/p&gt;&#xA;" OwnerUserId="16312" LastEditorUserId="16312" LastEditDate="2016-02-17T11:45:23.877" LastActivityDate="2016-02-17T11:45:23.877" Title="Which is the most appropiate algorithm to use with Mlib for predicting prices" Tags="|apache-spark|beginner|" AnswerCount="0" CommentCount="2" ClosedDate="2016-02-17T11:13:33.570" ContentLicense="CC BY-SA 3.0" />
  <row Id="10262" PostTypeId="1" CreationDate="2016-02-16T10:26:52.147" Score="3" ViewCount="7052" Body="&lt;p&gt;Mathematically, in case of a constant function (&lt;code&gt;f(x) = c&lt;/code&gt;) the correlation coefficient of a constant function with function input is not defined. Neverthless, from the function plot, we see that there is no correlation between the function input and output, so it must be returning 0.&lt;/p&gt;&#xA;" OwnerUserId="14688" LastEditorUserId="10024" LastEditDate="2016-02-17T09:39:17.343" LastActivityDate="2016-04-18T10:40:31.850" Title="Why is the correlation coefficient of a constant function with function input is not defined?" Tags="|statistics|correlation|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10263" PostTypeId="2" ParentId="10250" CreationDate="2016-02-16T11:13:06.967" Score="8" Body="&lt;p&gt;The &lt;strong&gt;error function&lt;/strong&gt; is the function representing the difference between the values computed by your model and the real values. In the optimization field often they speak about two phases: a training phase in which the model is set, and a test phase in which the model tests its behaviour against the real values of output. In the training phase the error is necessary to improve the model, while in the test phase the error is useful to check if the model works properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;objective function&lt;/strong&gt; is the function you want to maximize or minimize. When they call it &quot;&lt;strong&gt;cost function&lt;/strong&gt;&quot; (again, it's the objective function) it's because they want to only minimize it. I see the cost function and the objective function as the same thing seen from slightly different perspectives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;&lt;strong&gt;criterion&lt;/strong&gt;&quot; is usually the rule for stopping the algorithm you're using. Suppose you want that your model find the minimum of an objective function, in real experiences it is often hard to find the exact minimum and the algorithm could continuing to work for a very long time. In that case you could accept to stop it &quot;near&quot; to the optimum with a particular stopping criterion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I gave you a correct idea of these topics.&lt;/p&gt;&#xA;" OwnerUserId="10024" LastActivityDate="2016-02-16T11:13:06.967" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10265" PostTypeId="1" CreationDate="2016-02-16T13:04:42.263" Score="2" ViewCount="64" Body="&lt;p&gt;From a data stream i'm receiving a pair of measurements consisting of a current consumption and a current percentage every second. By accumulating the consumption over time it will represent eventually the maximum capacity when the percentage reaches from 100% to 0%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict the maximum capacity in (almost) real time using linear regression with a small sample size window of two percent. However, when i compare the models of these local regressions of every two percent with the model of the whole data regression, i get very different results due to perhaps local fluctuation. (see figure)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to bring the local regression models closer to the whole data model? (in a way that i can see the differences due to fluctuation but overall closer predictions to the whole data model)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e1c8w.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e1c8w.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15690" LastEditorUserId="15690" LastEditDate="2016-02-16T13:19:41.633" LastActivityDate="2016-02-16T13:19:41.633" Title="local regression with streaming data" Tags="|r|predictive-modeling|time-series|linear-regression|data-stream-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10266" PostTypeId="2" ParentId="10262" CreationDate="2016-02-16T13:49:00.593" Score="0" Body="&lt;p&gt;When you have a constant function &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f(x) = c, for each x \in \R^n&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the correlation between the input and output is 0. In other terms, you can change x as you want, but in output we'll always have 0. The correlation exists and it is zero.&lt;/p&gt;&#xA;" OwnerUserId="10024" LastActivityDate="2016-02-16T13:49:00.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10270" PostTypeId="2" ParentId="9899" CreationDate="2016-02-16T22:15:52.360" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;How I know correlation vs. causation in this mix?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finding the causal affect of one variable on another is a difficult one because there are probably a few hidden variables that are the driving factors behind product 5 and all others. For example the true causal effect could be that the weather improved causing sales to increase for product 5 and others, making them correlated but not having any causal relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of trying to remove bias in determining causality is through the use of the following &lt;a href=&quot;https://en.wikipedia.org/wiki/Instrumental_variable&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Instrumental_variable&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14047" LastActivityDate="2016-02-16T22:15:52.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10272" PostTypeId="1" CreationDate="2016-02-17T06:12:09.650" Score="7" ViewCount="408" Body="&lt;p&gt;I'm currently trying to understand regularization for logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I'm not quite sure whether I &lt;em&gt;really&lt;/em&gt; got it. Basically, the problem is that when we add an additional features to a model we might overfit the training set. This leads to an algorithm which perfectly matches the training set, but fails for future values (as it too exactly fits what has been there in the past).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this is why we introduce regularization. With this we decrease the impact of the features, so the model may produce an algorithm which does a (seemingly) worse job on the training set, but overall works better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct?&lt;/p&gt;&#xA;" OwnerUserId="16148" LastEditorUserId="11097" LastEditDate="2016-02-17T06:37:10.057" LastActivityDate="2016-02-17T15:22:38.343" Title="Understanding regularization" Tags="|logistic-regression|regularization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10273" PostTypeId="2" ParentId="10228" CreationDate="2016-02-17T06:13:06.997" Score="79" Body="&lt;p&gt;Gini impurity and Information Gain Entropy are pretty much the same. And people do use the values interchangeably. Below are the formulae of both:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\textit{Gini}: \mathit{Gini}(E) = 1 - \sum_{j=1}^{c}p_j^2$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\textit{Entropy}: H(E) = -\sum_{j=1}^{c}p_j\log p_j$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Given a choice, I would use the Gini impurity, as it doesn't require me to compute logarithmic functions, which are computationally intensive. The closed-form of its solution can also be found.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Which metric is better to use in different scenarios while using&#xA;decision trees?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The Gini impurity, for reasons, stated above.&lt;/p&gt;&#xA;&lt;p&gt;So, &lt;strong&gt;they are pretty much the same when it comes to CART analytics.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/rasbt/python-machine-learning-book/blob/master/faq/decision-tree-binary.md&quot; rel=&quot;noreferrer&quot;&gt;Helpful reference for computational comparison of the two methods&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="71781" LastEditDate="2021-09-23T21:17:16.370" LastActivityDate="2021-09-23T21:17:16.370" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="10274" PostTypeId="1" CreationDate="2016-02-17T06:52:49.147" Score="3" ViewCount="325" Body="&lt;p&gt;I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and give names to these clusters. But I want to map demographic features on these clusters to validate my clusters whether they make sense or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for example I have original data set of 100 customers with 90 females and 10 males.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose find 4 clusters and each is of size 25. In Cluster No: 1,2 and 3, I have all males and no females. In Cluster No: 4, I have 15 males and 10 females.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to map genders to these clusters to describe clusters as male or female dominating ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I making sense here ?&lt;/p&gt;&#xA;" OwnerUserId="12411" LastEditorUserId="8878" LastEditDate="2016-02-17T17:40:33.437" LastActivityDate="2024-03-23T08:07:00.773" Title="Finding dominating attributes with in the clusters generated" Tags="|machine-learning|data-mining|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10275" PostTypeId="2" ParentId="10234" CreationDate="2016-02-17T07:41:17.917" Score="1" Body="&lt;p&gt;If I understand your goal correctly, you want to build classifier on similar entities taken from different distributions and than compare the classifier/source distributions. Am I right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, if it is indeed so you might find the area of &lt;a href=&quot;https://en.wikipedia.org/wiki/Domain_adaptation&quot; rel=&quot;nofollow&quot;&gt;domain adaptation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case I would go in your direction of splitting a database (since you need the same entities and features). In order the splitting will make sense I recommend doing the splitting by a feature that should change the entities behavior (e.g., country, far periods in time). That dataset you should use must be quite large in order to have large enough partitioned datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specific such dataset that you might find useful are &lt;a href=&quot;http://www.census.gov/main/www/cen2000.html&quot; rel=&quot;nofollow&quot;&gt;US census 2000&lt;/a&gt; (plenty of options for concept and splitting features).&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-02-17T07:41:17.917" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="10277" PostTypeId="1" CreationDate="2016-02-17T09:49:45.097" Score="1" ViewCount="249" Body="&lt;p&gt;I have to predict a quantitative response variable Y from p quantitative and q qualitative variables. There's around 100,000 rows in the data frame. What predictive methods can I use for that purpose? &lt;/p&gt;&#xA;" OwnerUserId="16337" LastEditorUserId="11097" LastEditDate="2016-02-17T09:52:48.323" LastActivityDate="2016-02-17T16:10:14.620" Title="Predict quantitative response variable from quantitative and qualitative variables" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10278" PostTypeId="1" CreationDate="2016-02-17T09:53:10.923" Score="0" ViewCount="576" Body="&lt;p&gt;I was trying to implement Item based Recommender System with the boolean dataset, Dataset example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;      User-id | movie-id | Action | Comedy | Drama&#xA;        1              200    0         1        1&#xA;        2              210    1         1        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And I tried implementing it with item-based similarity algorithm as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;package prediction.contentrecommender;&#xA;&#xA;import java.io.File;&#xA;import java.io.IOException;&#xA;import java.util.List;&#xA;&#xA;import org.apache.mahout.cf.taste.common.TasteException;&#xA;import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;&#xA;import org.apache.mahout.cf.taste.impl.recommender.GenericItemBasedRecommender;&#xA;import org.apache.mahout.cf.taste.impl.similarity.TanimotoCoefficientSimilarity;&#xA;import org.apache.mahout.cf.taste.model.DataModel;&#xA;import org.apache.mahout.cf.taste.recommender.RecommendedItem;&#xA;import org.apache.mahout.cf.taste.similarity.ItemSimilarity;&#xA;import org.apache.mahout.cf.taste.impl.recommender.GenericBooleanPrefItemBasedRecommender;&#xA;&#xA;&#xA;public class ContentRecommender {&#xA;&#xA;    /**&#xA;     * @param args&#xA;     */&#xA;    public static void main(String[] args) {&#xA;        int userid=10;&#xA;           int noOfRecommendations=1;&#xA;           &#xA;        &#xA;        try {&#xA;            &#xA;            DataModel dm = new FileDataModel(new File(&amp;quot;data/dataset.csv&amp;quot;));&#xA;            ItemSimilarity sim = new TanimotoCoefficientSimilarity(dm);&#xA;            GenericItemBasedRecommender recommender =&#xA;                new GenericBooleanPrefItemBasedRecommender(dm,sim);&#xA;&#xA;         List&amp;lt;RecommendedItem&amp;gt; recommendations =&#xA;             recommender.recommend(userid, noOfRecommendations);&#xA;                for(RecommendedItem recommendation : recommendations)&#xA;                 {&#xA;                    System.out.println(recommendation.getItemID());&#xA;                }&#xA;                &#xA;            &#xA;        } &#xA;        catch (IOException e) &#xA;        {&#xA;            System.out.println(&amp;quot;there is an error&amp;quot;);&#xA;            e.printStackTrace();&#xA;        } &#xA;       catch (TasteException e) &#xA;        {&#xA;            System.out.println(&amp;quot;error&amp;quot;);&#xA;            e.printStackTrace();&#xA;        }&#xA;&#xA;        &#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;When I am running this code, I am not getting any output, although there is no error.I don't know what can be the problem. Can anyone figure out how to implement item-based similarity algorithm with Boolean data-set.&lt;/p&gt;&#xA;" OwnerUserId="16176" LastEditorUserId="153991" LastEditDate="2023-09-02T18:51:49.180" LastActivityDate="2023-09-02T18:51:49.180" Title="Recommendation for boolean dataset with apache mahout" Tags="|machine-learning|apache-mahout|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10279" PostTypeId="2" ParentId="9420" CreationDate="2016-02-17T11:12:45.037" Score="8" Body="&lt;p&gt;They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more if they are close to the boundary. The points near the boundary are therefore more important to the loss and therefore deciding how good the boundary is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVM uses a hinge loss, which conceptually puts the emphasis on the boundary points. Anything farther than the closest points contributes nothing to the loss because of the &quot;hinge&quot; (the max) in the function. Those closest points are the support vectors, simply. Therefore it actually reduces to picking a boundary that creates the largest margin -- distance to closest point. The theory is that the boundary case is all that really matters to generalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside is that hinge loss is not differentiable, but that just means it takes more math to discover how to optimize it via Lagrange multipliers. It doesn't really handle the case where data isn't linearly separable. Slack variables are a trick that lets this possibility be incorporated cleanly into the optimization problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use hinge loss with &quot;deep learning&quot;, e.g. &lt;a href=&quot;http://arxiv.org/pdf/1306.0239.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/pdf/1306.0239.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-02-17T11:12:45.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10280" PostTypeId="1" CreationDate="2016-02-17T11:32:54.833" Score="2" ViewCount="122" Body="&lt;p&gt;On a neural network, how can I find out what features most contribute for some label (the dependent variable y)?&lt;/p&gt;&#xA;" OwnerUserId="15244" LastEditorUserId="11097" LastEditDate="2016-02-17T11:35:11.733" LastActivityDate="2016-02-17T20:37:03.703" Title="What features most contribute for label y?" Tags="|machine-learning|neural-network|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10281" PostTypeId="2" ParentId="10272" CreationDate="2016-02-17T14:39:40.497" Score="5" Body="&lt;p&gt;That's correct. Without regularization your model would fit to an irrelevant noise present in your dataset. It means that training set will fit better but the overall predictive power will decrease. &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html#overfitting_and_regularization&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; is a good article about overfitting and regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can get some intuition by watching these images:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/KxBK9.png&quot; rel=&quot;noreferrer&quot;&gt;Without regularization&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/043U7.png&quot; rel=&quot;noreferrer&quot;&gt;With regularization&lt;/a&gt;&#xA;You can clearly see that the shape at the 1st image has problems with prediction at the upper left corner but it perfectly fits to the training points.&lt;/p&gt;&#xA;" OwnerUserId="15656" LastEditorUserId="15656" LastEditDate="2016-02-17T15:22:38.343" LastActivityDate="2016-02-17T15:22:38.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10282" PostTypeId="2" ParentId="10272" CreationDate="2016-02-17T15:15:28.297" Score="4" Body="&lt;p&gt;Yes, that is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, think of a polynomial $a_n x^n + a_{n-1} x^{n-1} + \dots + a_2 x^2 + a_1 x^1 + a_0 x^0$ which should fit 100 data points $(x_, y_i)$ where all $y_i$ were generated by one polynomial with some noise. Of course, you could always perfectly fit the model to the data making the MSE error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$MSE = \sum_{(x_i, y_i)} (y_i - \text{predict}_{\text{Model}}(x_i))^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;be 0. But considering that there is noise, you might actually prefer a &quot;simpler&quot; model. One way to think about simplicity is having weights $a_i$ which are smaller. This is done by adding a regularization term to the error. One common regularization is $L_1$ regularization ($+ \sum_{1}^{100} | a_i |$), another one is $L_2$ regulariazion ($+ \sum_{1}^{100} a_i^2$).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-02-17T15:15:28.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10283" PostTypeId="2" ParentId="10277" CreationDate="2016-02-17T16:10:14.620" Score="1" Body="&lt;p&gt;You should use one of the regression methods for predicting a quantitative variable. For the qualitative features, you can binarize them to pass to the learning algorithm. If you use Python, scikit-learn provide classes for doing that in the preprocessing module.&lt;/p&gt;&#xA;" OwnerUserId="13061" LastActivityDate="2016-02-17T16:10:14.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10284" PostTypeId="1" CreationDate="2016-02-17T16:50:36.510" Score="1" ViewCount="56" Body="&lt;p&gt;I want to estimate how many people are in a household by looking at account transactions. It would be also interesting to understand if in the household there are children.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought that a possible way to proceed could be to compare the living expenses with that of the average individual according to the national statistics. Moreover I could also perform text analytics to retrieve informations about what products people bought (in order to understand if they spent money to buy children products).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As model I could maybe use a random forest for classification (but please correct me if I'm wrong).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there better way to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="848" LastActivityDate="2016-02-17T20:56:27.763" Title="Classification problem: estimate the number of people in a household through account transactions" Tags="|machine-learning|classification|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10285" PostTypeId="2" ParentId="10197" CreationDate="2016-02-17T17:37:15.280" Score="1" Body="&lt;p&gt;I would not suggest GMM at this point as the distribution of points in the space is not well-shaped enough. Even if you want to use it it's better to look at your data in PC space (i.e. using PCA). My suggestion would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Think of your features. What are they? Are you going to use these gr-chromacity as features? If yes you should know that kernel methods work better on this as the features are highly nonlinear. The image show that you need a feature mapping anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) It seems you have already thought of kernel methods as you put SVM as a tag. you can use it for classification. Might work better than GMM here. Also think of probabilistic graphical models as they have been used intensively for image segmentation and your images are structured enough (a football field has its fixed position in the image anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) If you have raw labeled dataset, I'd recommend to think of smarter features for segmentation. in gr-chromacity you already loose some information about colors which is the most important thing for you here. I would recommend taking the position of pixels into account as well. Then a PCA on the new data may reveal some more linearly separated classes.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-02-17T17:37:15.280" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10286" PostTypeId="1" CreationDate="2016-02-17T18:59:27.833" Score="4" ViewCount="290" Body="&lt;p&gt;I am thinking of the following image where we have two weights and an error (so we can make a 3D visualization). In this case a &quot;long valley&quot; looks like $x^2$ in the plane of the gradient, but in a perpendicular plane the function can still be minimized:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5JyJB.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5JyJB.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to my understanding, all optimization algorithms which are based on the gradient should be &quot;trapped&quot; at the saddle point like SGD is. Why does having a momentum term help in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm sorry, I don't know who made those images. &lt;a href=&quot;https://i.stack.imgur.com/ptfob.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;More of them&lt;/a&gt; are available.)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-03-19T08:48:51.407" Title="When being in a perfect &quot;Long Valley&quot; situation, does momentum help?" Tags="|neural-network|optimization|gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10287" PostTypeId="2" ParentId="10280" CreationDate="2016-02-17T20:37:03.703" Score="1" Body="&lt;p&gt;I have good news and bad news.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bad news is that due to the nonlinear nature of neural networks, the same feature will contribute differently depending on the other features.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a simple nonlinear function: f(a,b,c) = y = ba^3 + bc + c.  One could argue that 'a' contributes most to this function, but when b is negative, a more positive a might make things worse.  When b is zero, c contributes most.  When b is negative, a very negative a will contribute most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good news is that you may be able to visualize some simple single-layer networks.  There's a fun trick for this used in deep learning.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you've got x for your examples, W for your weights, and y for your outcomes, set the feature you want to show in y to 1 and the rest to zero, then multiply by W-transpose.  You should get some output values for x which show you how connected everything is.  It's not a perfect solution, but if you get some huge values coming out after the multiplication, you can say those values are highly-connected for y.  (Usually.)&lt;/p&gt;&#xA;" OwnerUserId="16135" LastActivityDate="2016-02-17T20:37:03.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10288" PostTypeId="1" AcceptedAnswerId="10294" CreationDate="2016-02-17T20:43:08.740" Score="2" ViewCount="395" Body="&lt;p&gt;I have a data set that's a dictionary of tuples. Each key represents an ID number and each tuple is (yesvotes, totalvotes). Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;{17: (6, 10), 18: (1, 1), 21: (0, 2), 26: (1, 1), 27: (3, 4), 13: (2, 2)}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to find the max key of the set. I want to assign weights so, for instance, key 17 would be ranked higher than key 18 because even though the ratio is much smaller, it has ten times the total votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an optimal way to do this? My best guess is simply calculate new ratios by &lt;code&gt;(yesvotes/totalvotes)*(totalvotes+1)&lt;/code&gt; but that doesn't seem right... Is there some kind of standardized field of study concerning fair-voting?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastEditorUserId="15527" LastEditDate="2016-02-18T14:26:15.610" LastActivityDate="2016-02-18T14:26:15.610" Title="Sort by average votes/ratings" Tags="|python|data-cleaning|weighted-data|data-wrangling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10289" PostTypeId="2" ParentId="10286" CreationDate="2016-02-17T20:48:33.153" Score="0" Body="&lt;p&gt;The animation is by &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/2gopfa/visualizing_gradient_optimization_techniques/cklhott&quot; rel=&quot;nofollow&quot;&gt;Alec Radford&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Momentum helps by &quot;pushing through&quot; the local minimum or saddle point. Of course, it does not push in the optimal direction, which is why it exhibits oscillation, as you can see above, but it eventually recovers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further study, I suggest reading about &lt;a href=&quot;https://en.wikipedia.org/wiki/Preconditioner#Preconditioning_in_optimization&quot; rel=&quot;nofollow&quot;&gt;preconditioning&lt;/a&gt; and the &lt;a href=&quot;https://en.wikipedia.org/wiki/Condition_number&quot; rel=&quot;nofollow&quot;&gt;condition number&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-17T20:48:33.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10290" PostTypeId="2" ParentId="10284" CreationDate="2016-02-17T20:56:27.763" Score="1" Body="&lt;p&gt;I think regression makes more sense here than classification; you are trying to predict a number. In particular I would recommend &lt;a href=&quot;https://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow&quot;&gt;Poisson regression&lt;/a&gt; since you are modeling count data. Software is readily available for various languages.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-17T20:56:27.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10291" PostTypeId="2" ParentId="10262" CreationDate="2016-02-17T21:19:57.337" Score="4" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;correlation coefficient&lt;/a&gt; between two random variables is a rigorously defined mathematical parameter. It is undefined when either of the random variables have zero variance. Therefore &quot;NaN&quot; is a very appropriate value to return in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are welcome to define it to be zero when one of the random variables is constant, if you find this more practical. But this does not agree with my mathematical intuition using limits. For example, if $X$ is $Uniform[-1,1]$, then the correlation coefficient between $X$ and $Y = c + \epsilon X$ is 1 for all $\epsilon &amp;gt; 0$, -1 for all $\epsilon &amp;lt; 0$. You can modify this example to get functions that get closer and closer to a constant function as $\epsilon$ goes to zero, while the correlation between X and its image is any number from the interval [-1,1].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also like &lt;a href=&quot;https://stats.stackexchange.com/a/18427&quot;&gt;this explanation&lt;/a&gt; by David Epstein: the cosine of an angle between a non-zero and a zero vector is fundamentally undefined.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-02-18T09:03:57.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10293" PostTypeId="2" ParentId="1069" CreationDate="2016-02-17T22:16:03.997" Score="1" Body="&lt;p&gt;I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-02-17T22:16:03.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10294" PostTypeId="2" ParentId="10288" CreationDate="2016-02-17T22:25:38.420" Score="3" Body="&lt;p&gt;Yes, this is a well-studied problem: rank aggregation. &lt;a href=&quot;http://www.evanmiller.org/how-not-to-sort-by-average-rating.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a solution with code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the quantity you are trying to estimate, the &quot;score&quot; of the item, is subject to noise. The fewer votes you have the greater the noise. Therefore you want to consider the variance of your estimates when ranking them.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-17T22:25:38.420" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10295" PostTypeId="2" ParentId="9731" CreationDate="2016-02-17T23:51:18.347" Score="2" Body="&lt;p&gt;I have been using torch for a few months now but I will give it a go (apologies if incorrect).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes a weight $w$ is updated as follows;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ w_{new} = w_{old} - \gamma \partial E/ \partial w_{old}  $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $ \gamma $ is your learning rate and $E$ is the error calculated using something like &lt;code&gt;criterion:forward(output,target)&lt;/code&gt;. The criterion could be, for example, &lt;code&gt;nn.MSECriterion()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate $\partial E/\partial W$  you need $\partial E/\partial y$ &lt;code&gt;gradOutput = criterion:backward(output,target)&lt;/code&gt;(gradient respect to output) as well as the input &lt;code&gt;input&lt;/code&gt; to the net i.e. your $X$ (e.g. image data) to generate the recursive set of equations which multiply with &lt;code&gt;gradOutput&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model:backward(input, gradOutput)&lt;/code&gt; therefore serves to update the weights so that they are ready for the next &lt;code&gt;model:forward(input)&lt;/code&gt; as it generates a big derivative tensor $dE/dW_{old}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is then combined with a optimiser such as &lt;code&gt;optim.sgd&lt;/code&gt; using &lt;code&gt;optimMethod&lt;/code&gt; and the old weights $W_{old}$ to generate the new weights in the first equation. Of course you can just update the weights without an optimiser with &lt;code&gt;model:updateParameters(learningRate)&lt;/code&gt; but you miss useful stuff like momentum, weight decay etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Got a bit side tracked there but hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="14047" LastActivityDate="2016-02-17T23:51:18.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10296" PostTypeId="1" AcceptedAnswerId="10298" CreationDate="2016-02-18T00:08:09.197" Score="0" ViewCount="60" Body="&lt;p&gt;machine learning is being hyped since the deep neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me, that you have to program in order to do machine learning.&#xA;But is the process of training data and labeling data is the same of every problem. Why isn't there an Excel like application that enables thousands of non experts to do machine learning ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer : I am not a data scientist .&lt;/p&gt;&#xA;" OwnerUserId="16353" LastActivityDate="2016-02-18T02:35:39.410" Title="Why is there no end user Application, yet?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2016-02-18T14:25:33.200" ContentLicense="CC BY-SA 3.0" />
  <row Id="10297" PostTypeId="2" ParentId="10286" CreationDate="2016-02-18T01:22:14.330" Score="1" Body="&lt;p&gt;The answer is simple: in a perfect situation never. These optimizers are 1st order methods (unlike for example Hessian-Free optimization). The reason for they escaping from the stationary point above is that there is some small gradient (too small for ordinary SGD) in the &quot;correct&quot; direction. Momentum and Nesterov gradient can build up the velocity over time in that direction and Adagrad/Adadelta/RMS can immediately scale the gradient (increase it in the direction for which the past gradients were small).&lt;/p&gt;&#xA;" OwnerUserId="15656" LastEditorUserId="15656" LastEditDate="2016-02-18T08:22:04.017" LastActivityDate="2016-02-18T08:22:04.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10298" PostTypeId="2" ParentId="10296" CreationDate="2016-02-18T02:35:39.410" Score="1" Body="&lt;p&gt;Listing 2 examples:&#xA;&lt;a href=&quot;https://www.ibm.com/marketplace/cloud/watson-analytics/us/en-us&quot; rel=&quot;nofollow&quot;&gt;IBM Watson Analytics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.ibm.com/marketplace/cloud/watson-analytics/us/en-us&quot; rel=&quot;nofollow&quot;&gt;Amazon ML use case&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preparing the data for supervised learning is require skills. Not all data came labeled and in form to be used as in need to solving problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also many more platforms/Api are in market now but for sure you can't solve a problem only with 1 algorithm, is needed much more ... Hope it help. &lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2016-02-18T02:35:39.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10299" PostTypeId="1" AcceptedAnswerId="15438" CreationDate="2016-02-18T04:25:38.627" Score="2" ViewCount="7140" Body="&lt;p&gt;I am trying to find a resource to understand non-negative matrix factorization. Apart from Wikipedia, I couldn't find anything useful.&lt;/p&gt;&#xA;" OwnerUserId="14652" LastEditorUserId="26596" LastEditDate="2016-11-30T23:07:55.847" LastActivityDate="2016-11-30T23:07:55.847" Title="What is a good explanation of Non Negative Matrix Factorization?" Tags="|nlp|text-mining|dimensionality-reduction|feature-engineering|reference-request|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10301" PostTypeId="1" CreationDate="2016-02-18T09:59:43.063" Score="6" ViewCount="6953" Body="&lt;p&gt;I have a data from GPS in the form &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.('latitude', 'longitude','Timestamp').&#xA;2.('latitude', 'longitude','Timestamp').&#xA;3.('latitude', 'longitude','Timestamp').&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am changing this data into the below form &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'latitude_1', 'longitude_1', 'Timestamp_1', 'latitude_2', 'longitude_2', 'Timestamp_2, Timestamp_2-Timestamp_1.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this format I am training a &lt;code&gt;LinearRegressionWithSGD&lt;/code&gt; model of spark where label is &lt;code&gt;Timestamp_2-Timestamp_1&lt;/code&gt; and features are &lt;code&gt;latitude_1, longitude_1, latitude_2, longitude_2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I am giving Origin ( latitude and longitude ) and destination ( latitude and longitude ) the results are very bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly guide me whether this approach is the right approach ? and if not then how to build a prediction model from given data to predict &lt;code&gt;Estimated Time of Arrival&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15906" LastEditorUserId="16284" LastEditDate="2016-04-21T13:54:30.647" LastActivityDate="2017-10-24T20:40:57.423" Title="How to predict ETA using Regression?" Tags="|machine-learning|linear-regression|geospatial|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10302" PostTypeId="1" AcceptedAnswerId="10362" CreationDate="2016-02-18T10:11:23.997" Score="9" ViewCount="773" Body="&lt;p&gt;I am trying to understand reinforcement learning and markov decision processes (MDP) in the case where a neural net is being used as the function approximator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having difficulty with the relationship between the MDP where the environment is explored in a probabilistic manner, how this maps back to learning parameters and how the final solution/policies are found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct to assume that in the case of Q-learning, the neural-network essentially acts as a function approximator for q-value itself so many steps in the future? How does this map to updating parameters via backpropagation or other methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, once the network has learned how to predict the future reward, how does this fit in with the system in terms of actually making decisions? I am assuming that the final system would not probabilistically make state transitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="5144" LastEditorUserId="5144" LastEditDate="2016-02-18T10:24:21.377" LastActivityDate="2018-12-28T16:53:00.080" Title="Understanding Reinforcement Learning with Neural Net (Q-learning)" Tags="|machine-learning|neural-network|q-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10303" PostTypeId="2" ParentId="10288" CreationDate="2016-02-18T10:31:17.507" Score="2" Body="&lt;p&gt;Have a look at the example on &lt;a href=&quot;http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter4_TheGreatestTheoremNeverTold/Chapter4.ipynb#Example:-How-to-order-Reddit-comments&quot; rel=&quot;nofollow&quot;&gt;&quot;How to order Reddit comments&quot;&lt;/a&gt; using their up- &amp;amp; down-votes in Cam Davidson Pilon's &lt;a href=&quot;https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac{a}{a+b} - 1.65\sqrt{\frac{a b}{(a+b)^2(a+b+1)}}&#xA;$$&#xA;where&#xA;$$&#xA;a = 1 + u&#xA;$$&#xA;$$&#xA;b = 1 + d&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$u$ is the number of yes votes and $d$ is the number of no votes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorting your data using the score obtained from that formula results in the following table (highest score first):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id  total  yes  no     score&#xA;13      2    2   0  0.430479&#xA;27      4    3   1  0.372679&#xA;17     10    6   4  0.357720&#xA;18      1    1   0  0.277758&#xA;26      1    1   0  0.277758&#xA;21      2    0   2 -0.069521&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-02-18T10:31:17.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10304" PostTypeId="2" ParentId="10301" CreationDate="2016-02-18T12:26:20.643" Score="2" Body="&lt;p&gt;To predict timestamps from two predictor variables longitude and latitude, you want to train a multiple linear regression model of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Timestamp = \alpha + \beta_0 \cdot Longitude + \beta_1 \cdot Latitude.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a new latitude-longitude pair of you destination, you can then compute the ETA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark's &lt;code&gt;LinearRegressionWithSGD&lt;/code&gt; model should be able to perform multiple linear regression out of the box, using &lt;code&gt;Timestamp&lt;/code&gt; as label and &lt;code&gt;latitude&lt;/code&gt; and &lt;code&gt;longitude&lt;/code&gt; as features. There's no need to transform the data beforehand.&lt;/p&gt;&#xA;" OwnerUserId="16366" LastActivityDate="2016-02-18T12:26:20.643" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10305" PostTypeId="2" ParentId="5458" CreationDate="2016-02-18T13:42:13.210" Score="1" Body="&lt;p&gt;RapidMiner is actually pretty good at multivariate analysis. The difficult part is getting the data into an appropriate format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to perform clustering, the data should be converted to a format so that two jobs can be &lt;em&gt;compared&lt;/em&gt; based on skills. For example, each job can be represented as a 12-dimensional boolean vector $$[x_1,\ldots,x_{12}]$$ where each $x_i$ represents a skill which is $true$ if the job covers the skill and $false$ otherwise. A distance measure between jobs is then for example &lt;a href=&quot;https://en.wikipedia.org/wiki/Hamming_distance&quot; rel=&quot;nofollow&quot;&gt;Hamming-distance&lt;/a&gt;. RapidMiner offers a wide range of distance measures between different kinds of feature vectors (Hamming distance is called &lt;code&gt;NominalDistance&lt;/code&gt; in RapidMiner, but others might be better).&lt;/p&gt;&#xA;" OwnerUserId="16366" LastActivityDate="2016-02-18T13:42:13.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10306" PostTypeId="2" ParentId="9880" CreationDate="2016-02-18T14:25:11.430" Score="3" Body="&lt;p&gt;To understand the coefficients you just need to understand how the logistic regression model that you fit uses the coefficients to make predictions. No, it does not work like a decision tree. It's a linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really, predictions are based on the dot product of the coefficients and the values from some new instance to predict. This is just the sum of their products. The higher the dot product, the more positive the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can understand it as computing something like &lt;code&gt;-2.40477246e-10 * MATCH_HOME + -5.57611571e-02 * MATCH_AWAY + ...&lt;/code&gt; (I don't know what coefficients go with what feature in your model.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That generally means that inputs with bigger coefficients matter more, and inputs with positive coefficients correlate positively with a positive prediction. That's most of what you can interpret here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first of those conclusions is only really valid if inputs have been normalized to be on the same scale though. I'm not clear that you've done that here. You should also in general use L1 regularization if you intend to interpret the coefficients this way.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-02-18T14:25:11.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10307" PostTypeId="1" CreationDate="2016-02-18T14:38:46.413" Score="5" ViewCount="182" Body="&lt;p&gt;I'm trying out a &lt;a href=&quot;http://www.kaggle.com&quot; rel=&quot;nofollow&quot;&gt;Kaggle&lt;/a&gt; competition, which puts me in the unusual position of being able to get feedback on my models' &quot;true&quot; performance (you can submit several predictions per day and they give you your score on a performance metric -- in this case MSE -- each time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, I've been observing a situation I can't explain -- my cross-validation results for my first stage Random Forest model, constructed with &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow&quot;&gt;sklearn in Python&lt;/a&gt;, are consistently accurate within 1 percentage point of the true MSE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, my first  stage model will give a CV MSE of about .22 and the &quot;ground truth&quot; (actual) MSE will turn out to be about .23. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OTOH, my 2nd stage Random Forest, which is a different type of RF constructed in R, gives way &quot;better&quot; (smaller) training and cross validation MSE errors, like 0.05, but consistently turns out to have a much higher true MSE, like .23. In the 2nd stage Random Forest I have the predictions from the first stage and a few additional engineered features as regressors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a statistical explanation for this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example is it the result of some sort of recursion from stacking the models? If so, given that most winners of these Machine Learning contests are stacking many models, what's the appropriate way to do so that would avoid this statistical problem?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-02-19T15:30:35.797" LastActivityDate="2016-02-19T15:30:35.797" Title="2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd" Tags="|machine-learning|random-forest|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10308" PostTypeId="2" ParentId="10179" CreationDate="2016-02-18T15:21:31.547" Score="12" Body="&lt;p&gt;According to the &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/tutorials/input_format.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost documentation&lt;/a&gt;, XGboost expects:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the examples of a same group to be consecutive examples,&lt;/li&gt;&#xA;&lt;li&gt;a list with the size of each group (which you can set with &lt;code&gt;set_group&lt;/code&gt; method of &lt;code&gt;DMatrix&lt;/code&gt; in Python).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14089" LastEditorUserId="14089" LastEditDate="2020-11-03T14:36:04.833" LastActivityDate="2020-11-03T14:36:04.833" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="10309" PostTypeId="2" ParentId="9827" CreationDate="2016-02-18T15:39:32.197" Score="2" Body="&lt;p&gt;What you are asking for is a gradient based hyperparameter optimization. To enable such a grid search one would have to define a metric on the space of all hyper parameter combinations for the current pipeline. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think this makes sense in your example because of the binary nature of the &lt;code&gt;use_idf&lt;/code&gt; flag. Lets say your parameter combinations are &lt;code&gt;(use_idf, alpha)&lt;/code&gt;. Then you would have to do decide what the distance between &lt;code&gt;(True, 0.00015)&lt;/code&gt; and &lt;code&gt;(True, 0.00010)&lt;/code&gt; is. In your case it should be smaller than the distance between &lt;code&gt;(True, 0.00015)&lt;/code&gt;and &lt;code&gt;(False, 0.00015)&lt;/code&gt;. So&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;d((True, 0.00015), (True, 0.00010)) &amp;lt; d((True, 0.00015), (False, 0.00015))&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about the distance between &lt;code&gt;(True, 0.00015)&lt;/code&gt; and &lt;code&gt;(True, 0.05)&lt;/code&gt;? Will it be still smaller than &lt;code&gt;d((True, 0.00015), (False, 0.00015))&lt;/code&gt;? I think you get the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By choosing the metric you will subjectively influence the search for the best hyper parameters. Probably a random search would be more suitable as suggested in the following paper&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@article{bergstra2012random,&#xA;  title={Random search for hyper-parameter optimization},&#xA;  author={Bergstra, James and Bengio, Yoshua},&#xA;  journal={The Journal of Machine Learning Research},&#xA;  volume={13},&#xA;  number={1},&#xA;  pages={281--305},&#xA;  year={2012},&#xA;  publisher={JMLR. org}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16370" LastEditorUserId="11097" LastEditDate="2016-02-18T15:42:53.020" LastActivityDate="2016-02-18T15:42:53.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10310" PostTypeId="1" CreationDate="2016-02-18T22:06:24.440" Score="5" ViewCount="10603" Body="&lt;p&gt;I am trying to build a classifier that will classify data points of different classes, but which significantly overlap, but I'm not sure how to best approach the problem. I have read that tree models can perform well in this situation, and that Support Vector Machines might as well, but I'm unclear what other approaches might be good for tackling this problem. Can anyone give me some direction?&lt;/p&gt;&#xA;" OwnerUserId="16380" LastEditorUserId="13413" LastEditDate="2016-05-28T02:41:57.623" LastActivityDate="2016-12-24T05:29:50.223" Title="Classification with heavily overlapping data points of different labels" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10311" PostTypeId="2" ParentId="8946" CreationDate="2016-02-19T03:26:46.010" Score="1" Body="&lt;p&gt;I agree with the existing answer that &lt;em&gt;feature scaling&lt;/em&gt; is a superset into which techniques like mean normalization, residual normalization, etc falls under.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, assuming that by &lt;em&gt;feature scaling&lt;/em&gt;, you mean the techniques other than mean normalization, I would attempt to answer your questions:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Can I mix and match these two approaches? e.g. Feature Scale x1 and&#xA;  Mean Normalize x2?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In most cases &lt;strong&gt;No&lt;/strong&gt;. Generally, only one normalization technique is used and it pretty much suffices the need. In addition to that argument, it should also be noted that any normalization technique introduces duplication in the data records (not necessarily redundant duplication).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, pretty much a single normalization technique would suffice most of the times.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2) How do you determine which of these options to apply? It seems that&#xA;  either could accomplish the task of increasing your convergence&#xA;  rates... I suppose you just need to know your data set to understand&#xA;  which will reliably reduce your values while leaving as few outliers&#xA;  as possible?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt;, you are right. The selection of the technique depends on the data. And the feature scaling (and normalization) process comes under the process of data cleaning. So, it is done immediately after the selection of the relevant data for the analytics process.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-02-19T03:26:46.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10312" PostTypeId="2" ParentId="334" CreationDate="2016-02-19T05:48:30.247" Score="2" Body="&lt;p&gt;I think the effect of the certification from coursera is dependent on the individual as well as the classes. The requirement says min 3-5 hours a week, if you put more, and the material do open up for a lot more than the 3-5 hours, then these classes and certifications can be equivalent to strong knowledge base and experience in the field. Science comes to those who request it.&lt;/p&gt;&#xA;" OwnerUserId="16392" LastActivityDate="2016-02-19T05:48:30.247" CommentCount="0" CommunityOwnedDate="2016-02-19T14:15:54.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="10313" PostTypeId="2" ParentId="10307" CreationDate="2016-02-19T07:47:26.503" Score="6" Body="&lt;p&gt;To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the first RF and some other additional features. The error on CV training data is very low and on test data are very high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we will analyze the performance of the second RF without considering how it was built. Usually if you fit data very well on training set and very poor on test set it means you simply overfit data. This happens when your model learns too much from the data. The point is that in your training data you have two kinds of information. The first one is the real structure of the data, and the second one is the noise around that data, assuming that you have enough data and your model is able to handle the useful structure. The ideal case happens when you learn only the useful structure and when leave out the noise.When that happens you will have close errors in training and test data, which means that what your model learnt is at least almost only useful data. When you learn too much, which means you learn useful and noise together, your training error decreases because the results have the noise structure which is in training data, and on test data the errors grows since the noise from test data does not look like the noise from training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the first learner looks like it does not over fit. This of course does not always mean that you can't do better. It only means that what your learn is a structure which is also found in test data and thus has predictive power. So is understandable that you aim to improve the performance of the first predictor. One way would be to try to improve that predictor, and you can search for that by studying how that model learns. For example you can study if that model improves performance with more data by training repeatedly on samples of increased size and see if stabilize itself or not. There are many approaches. Another approach would be ensembles and a prominent one is stacking. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you did not understood the idea of stacking. The whole point of stacking can be expressed in the following statement: explore the model space by building various &lt;em&gt;different&lt;/em&gt; model families and in the end get the best from all of them. So stacking starts with the assumption that since we do not know which would be an ideal model for our data, we should try many imperfect ones. Some of them will behave well in some regions of space, some others in other regions of space, but none of them will be able to handle all the space at high performance. The key point is to have models from different families. Sometimes the same algorithm can produce very different models, and sometimes not. For example two SVMs with very different kernels can produce totally different surfaces. Sometimes even for the same kernel an SVM can produce very different results for very different complexity parameter. Often for RF this is not the case, but it is not impossible to get different surfaces with different parameters on RF. So the conclusion is that stacking aims to explore different ways of learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you have done is to apply stacking on a single base learner by adding new features. First of all is that in general it does not work. The predictions from the first learner has a string structure, it comes from averaging on regions. So, this predictions compared with additional variables from the second model are more powerful. This means that every wrong thing learned in the first RF becomes very powerful on the second RF, because it uses the same averaging over regions procedure for learning. So the errors from the first learner amplifies in the second learner. I do not know about the additional features, but in this conditions it looks like it is very easy to overfit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turning back to what's in a data set: this data set has a structured content which you have to learn and an unstructured content (noise) around that. Your purpose of learning is to learn only the structure. The first RF is able to distinguish the structure from noise simply because the noise is not altered. The second RF is not able anymore to make that distinction, since the predictions of the first RF are already smooth in regions, which means it has a powerful structure, at least from the point of view of the second RF, which learns in the same way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A last idea is that it looks like what you have done is somehow similar with a boosting procedure. The first learner gives you a structure which is used as a baseline by the second learner. But there is a big difference than in boosting. While in boosting you have a base line which you aim to improve, you improve that baseline by looking again at the original features, which have the noise unstructured. In your case at the second iteration you do not have again that chance and everything is structured. Thus errors amplifies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My 2 cents: try multiple learners, eventually using additional features in those learners, try different models for base learners, start with very simple (low complexity or small degrees of freedom for) the stacking learner, like a penalized logistic regression and more further to more complex ones in a controlled fashion. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-02-19T07:47:26.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10314" PostTypeId="1" CreationDate="2016-02-19T10:05:44.533" Score="1" ViewCount="290" Body="&lt;p&gt;I was trying to do class balancing on the image semantic segmentation problem for some classes in the images are in the minority. The weight for each class is calculated as mentioned in &lt;a href=&quot;http://arxiv.org/pdf/1511.00561v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;we weight each pixel by αc = median_freq / freq(c) where freq(c) is the number of pixels of class c divided by the total number of pixels in images where c is present, and median_freq is the median of these frequencies.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Then I weighted the cross entropy loss as follows, the size of the label is (img_col, img_row, num_class) for the labels are one-hot-labels:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def weighted_cce(coding_dist, true_dist, weights):&#xA;    # calculate weighted cross entropy loss&#xA;    # true_dist: ground truth, coding_dist: predicted&#xA;    coding_dist = T.clip(coding_dist, 10e-8, 1.0-10e-8)&#xA;    return -T.sum(weights * true_dist * T.log(coding_dist), axis=coding_dist.ndim-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;What's strange is that instead of producing a more balanced output, the result is even more biased than without class balancing, namely the network now can only recognize the most dominant classes in the images.&lt;/p&gt;&#xA;&lt;p&gt;Could anyone share some thoughts on this? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="14293" LastEditorUserId="98307" LastEditDate="2020-08-03T16:41:53.237" LastActivityDate="2020-08-03T16:41:53.237" Title="Regarding Class Balancing in Deep Neural Network" Tags="|python|deep-learning|image-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10316" PostTypeId="2" ParentId="9827" CreationDate="2016-02-19T11:54:02.797" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://auto-sklearn.readthedocs.org/en/master/&quot; rel=&quot;nofollow&quot;&gt;auto-sklearn&lt;/a&gt; team is working on finding a way to fully automate the pipeline. It has delivered good results for me so far. Better as what I have got manually with the GridsearchCV.&#xA;This can be an answer to your question if you 're happy to use such tool or read their source code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And at &lt;a href=&quot;http://www.ml4aad.org/algorithm-configuration/&quot; rel=&quot;nofollow&quot;&gt;ML4AAD&lt;/a&gt; you can find further links to the attempts to solve this problem.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="15361" LastEditDate="2016-02-20T12:18:12.850" LastActivityDate="2016-02-20T12:18:12.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10318" PostTypeId="1" AcceptedAnswerId="10319" CreationDate="2016-02-19T15:14:30.483" Score="2" ViewCount="1257" Body="&lt;p&gt;I'm rather new to PCA and was hoping to have some confusion cleared up. Lets say for example we have a feature matrix that's nx100 and I want to get it down to something a bit smaller, p-dimensions, without losing too much variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After applying PCA and receiving and new feature matrix nxp, I would use x_reduced to predict some target variable y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, after the transformation, the new reduced feature matrix has been rotated by the eigenvectors and is sitting on a new basis. Yet, our y has not changed relative to X_reduced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure about how y_original and x_reduced can be used for training since y has not changed with respect to x_reduced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to correct for this or am I not thinking about it correctly?&lt;/p&gt;&#xA;" OwnerUserId="16404" LastEditorUserId="16404" LastEditDate="2016-02-19T19:43:16.143" LastActivityDate="2016-02-19T19:43:16.143" Title="PCA and maintaining relationship with target variable" Tags="|statistics|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10319" PostTypeId="2" ParentId="10318" CreationDate="2016-02-19T15:29:30.033" Score="4" Body="&lt;p&gt;The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced.  While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format.  You lose a lot of interpretability as far as what the actual numbers mean which is why it may seem confusing, but it's just a transformed representation of the x_original that (hopefully) contains enough of the x_original variability to make it useful.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-02-19T15:29:30.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10320" PostTypeId="1" AcceptedAnswerId="10321" CreationDate="2016-02-19T16:08:12.400" Score="1" ViewCount="6168" Body="&lt;p&gt;I have a set of points where I performed a KMeans classification. How make a plot where the color of the point is based on the cluster they belong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: for clarification, having the set of points, I want to use the values of the array generated from &lt;code&gt;KMeans.predict()&lt;/code&gt; ( from sklearn) to choose the color of each point.&lt;/p&gt;&#xA;" OwnerUserId="16096" LastEditorUserId="11097" LastEditDate="2016-02-19T16:17:22.797" LastActivityDate="2016-02-19T16:37:04.990" Title="Colouring points based on cluster on matplotlib" Tags="|python|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10321" PostTypeId="2" ParentId="10320" CreationDate="2016-02-19T16:37:04.990" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn documentation&lt;/a&gt; shows you how:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colors = np.array([x for x in 'bgrcmykbgrcmykbgrcmykbgrcmyk'])&#xA;colors = np.hstack([colors] * 20)&#xA;...&#xA;if hasattr(algorithm, 'cluster_centers_'):&#xA;        centers = algorithm.cluster_centers_&#xA;        center_colors = colors[:len(centers)]&#xA;        plt.scatter(centers[:, 0], centers[:, 1], s=100, c=center_colors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/quQKq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/quQKq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-19T16:37:04.990" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10322" PostTypeId="1" CreationDate="2016-02-19T17:22:37.827" Score="6" ViewCount="39108" Body="&lt;p&gt;I'm new to Pandas and Bokeh; I'd to create a bar plot that shows two different variables next to each other for comparison. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, with the following Pandas data frame, I'd like to see how the amount of Recalled compares to the amount of Recovered for each year.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    year    Recalled    Recovered&#xA;0   1994    11.472  10.207&#xA;1   1995    11.810  10.326&#xA;2   1996    10.632  10.094&#xA;3   1997    13.857  12.944&#xA;4   1998    13.861  12.588&#xA;5   1999    13.375  11.951&#xA;6   2000    11.278  nan&#xA;7   2001    12.827  nan&#xA;8   2002    12.687  nan&#xA;9   2003    10.859  nan&#xA;10  2004    nan nan&#xA;11  2005    11.782  11.047&#xA;12  2006    12.089  10.194&#xA;13  2007    14.351  13.401&#xA;14  2008    14.921  13.886&#xA;15  2009    11.759  10.815&#xA;16  2010    12.987  11.482&#xA;17  2011    13.262  10.730&#xA;18  2012    9.980   9.520&#xA;19  2013    10.626  9.591&#xA;20  2014    12.199  10.270&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16407" LastActivityDate="2020-08-03T16:42:03.233" Title="How to plot multiple variables with Pandas and Bokeh" Tags="|python|pandas|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10323" PostTypeId="2" ParentId="10142" CreationDate="2016-02-19T18:21:54.060" Score="1" Body="&lt;p&gt;As you probably know, the Gaussian distribution is fully specified by its mean and covariance. theta0, thetaL, and thetaU are, respectively, initial values, lower bounds, and upper bounds on the covariance. You can set them based on your knowledge of the particulars of the problem. To learn more about Gaussian processes, I suggest visiting &lt;a href=&quot;http://www.gaussianprocess.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;this great Web site&lt;/a&gt;. It even offers a free ebook.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="98307" LastEditDate="2020-08-03T15:11:55.513" LastActivityDate="2020-08-03T15:11:55.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10324" PostTypeId="2" ParentId="10322" CreationDate="2016-02-19T18:44:15.090" Score="5" Body="&lt;p&gt;Just pandas, no bokeh (copy the data to the clipboard before running):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas, seaborn&#xA;DF = pandas.read_clipboard()&#xA;DF.plot.bar(x='year')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Dyw3p.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Dyw3p.png&quot; alt=&quot;pandas plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-19T18:44:15.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10325" PostTypeId="1" CreationDate="2016-02-19T20:44:06.953" Score="3" ViewCount="817" Body="&lt;p&gt;I wonder how Orange decides which attribute is declared as a string? I have few discrete attributes in Excel, which I can't use as Orange does not recognize them as discrete. Have tried export excel to csv format as well, no difference. Where in Orange I can enforce change from meta to discrete attribute?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance,&#xA;Marko &lt;/p&gt;&#xA;" OwnerUserId="16411" LastActivityDate="2018-02-18T17:57:04.833" Title="How Orange makes decision about declaration of a meta attribute" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10327" PostTypeId="2" ParentId="6378" CreationDate="2016-02-20T05:39:30.017" Score="2" Body="&lt;p&gt;The details are just a &lt;code&gt;help(summary.lm)&lt;/code&gt; in R. The meaning of the codes and the terms are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering R = Correlation Coefficient&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The &lt;strong&gt;residual standard error&lt;/strong&gt; is nothing more than the positive square root of the mean square error. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Regression degrees of freedom&lt;/strong&gt; = number of independent variables (factors) in $y$ = $a_1x_1 + a_2x_2 + a_3x3 + ...$   &lt;a href=&quot;https://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia link&lt;/a&gt; for a better in-depth explanation&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;R Square&lt;/strong&gt; = $(Multiple R)^2$ = $R^2$ =  $1 - (Residual SS / Total SS) = (Regress SS / Total SS)$&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Adjusted R Square&lt;/strong&gt; = $1 - (Total df / Residual df)(Residual SS / Total SS)$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://stats.stackexchange.com/a/17148/84191&quot;&gt;wonderful post&lt;/a&gt; on how to understand degrees of freedom.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-02-20T05:39:30.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10328" PostTypeId="1" CreationDate="2016-02-20T06:18:40.880" Score="3" ViewCount="167" Body="&lt;p&gt;In many image processing papers, I've seen that they used fuzzy logic for segmentation&#xA;I wonder how fuzzification impact the result that made Fuzzy-C-Means better than ordinary K-Means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. If possible could you provide me the sample data sets for case study&lt;/p&gt;&#xA;" OwnerUserId="14357" LastEditorUserId="924" LastEditDate="2016-02-20T09:52:47.093" LastActivityDate="2018-12-21T07:01:50.373" Title="Why does image segmentation benefit from fuzzy clustering?" Tags="|machine-learning|clustering|k-means|image-classification|fuzzy-logic|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10329" PostTypeId="1" AcceptedAnswerId="11420" CreationDate="2016-02-20T07:24:56.140" Score="9" ViewCount="10796" Body="&lt;p&gt;Would it be possible for a an amateur who is interested in getting  some &quot;hands-on&quot; experience in desining and training deep neural networks, to use an ordinary laptop for that purpose (no GPU), or is it hopeless to get good results in reasonable time without a powerful computer/cluster/GPU?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more specific, the laptop's CPU is an Intel Core i7 5500U fith generation, with 8GB RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, since I haven't specified what problems I would like to work on, I'll frame my questions in a different way: which deep architectures would you recommend that I try to implement with my hardware, such that the following goal is achieved: Acquiring intuition and knowledge about how and when to use techniques that were introduced in the past 10 years and were essential to the uprising of deep nets (such as understanding of initialisations, drop-out, rmsprop, just to name a few).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read about these techniques, but of course without trying them out myself I wouldn't know exactly how and when to implement these in an effective way. On the other hand, I'm afraid that if I try using a PC which isn't strong enough, then my own learning rate will be so slow that it would be meaningless to say that I've acquired any better understanding. And if I try using these techniques on shallow nets, maybe I wouldn't be building the right intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imagine the process of (my) learning as follows: I implement a neural net, let it practice for up to several hours, see what I've got, and repeat the process. If I do this once or twice a day, I would be happy if after, say, 6 months I will have gained practical knowledge which is comparable to what a professional in the field should know.&lt;/p&gt;&#xA;" OwnerUserId="16424" LastEditorUserId="836" LastEditDate="2016-02-20T19:17:24.003" LastActivityDate="2016-04-29T15:41:45.560" Title="Training Deep Nets on an Ordinary Laptop" Tags="|machine-learning|deep-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10330" PostTypeId="1" CreationDate="2016-02-20T10:11:17.397" Score="2" ViewCount="417" Body="&lt;p&gt;i am currently working on preprocessing unstructured data (emails,logs,bug reports and irc chats).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i wish to prove preprocessing improves the content quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;are there metrics available to prove this? that is to compare the raw data and the preprocessed data and show some quality improvement in numbers.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;content quality - i mean to say that the preprocessed corpus is more suitable for mining with topic models than the raw one.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of perplexity. i dont want to use this metric as it only evaluates the topic model performance and not the corpus.Also i wish to evaluate the corpus quality even before the topic model is applied.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can topic coherence be used as a metric to evaluate the quality of the corpus?&#xA;can more coherent topics be generated from preprocessed documents than the raw one?&lt;/p&gt;&#xA;" OwnerUserId="16427" LastEditorUserId="16427" LastEditDate="2016-02-20T18:02:31.100" LastActivityDate="2020-02-01T04:03:40.390" Title="What metrics must i use in my data(unstructured) preprocessing research?" Tags="|text-mining|data-cleaning|information-retrieval|topic-model|preprocessing|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10333" PostTypeId="1" AcceptedAnswerId="10342" CreationDate="2016-02-21T00:45:37.420" Score="1" ViewCount="2342" Body="&lt;p&gt;I've looked and surprisingly have not found too much discussion on the relative strengths of the &lt;code&gt;caret&lt;/code&gt; and &lt;code&gt;e1071&lt;/code&gt; package.  From my understanding, these packages perform many of the same ML algorithms.  With that in mind, I'm interested in what those practitioners who have experience with both think these packages relative strengths and weaknesses are.  What do you think about readability of code, ease of coding, methods, speed, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular I'm interested in multi-class SVM&lt;/p&gt;&#xA;" OwnerUserId="10845" LastEditorUserId="10845" LastEditDate="2016-02-21T03:05:11.860" LastActivityDate="2016-02-26T01:07:23.793" Title="ML packages in R: caret v e1071" Tags="|r|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10334" PostTypeId="5" CreationDate="2016-02-21T05:39:00.670" Score="0" Body="&lt;p&gt;Some good beginner resources for getting started in Data Science are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/topic/Data-Science&quot; rel=&quot;nofollow&quot;&gt;Data Science wiki of Quora&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some good beginner resources for getting started in Machine Learning are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;Professor Andrew Ng's course on Machine Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="21" LastEditDate="2016-02-21T10:54:33.530" LastActivityDate="2016-02-21T10:54:33.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10335" PostTypeId="4" CreationDate="2016-02-21T05:39:00.670" Score="0" Body="For questions which concern getting started in Data Science or any of its related subdomains." OwnerUserId="11097" LastEditorUserId="21" LastEditDate="2016-02-21T10:54:14.877" LastActivityDate="2016-02-21T10:54:14.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10336" PostTypeId="1" CreationDate="2016-02-21T05:41:22.113" Score="2" ViewCount="206" Body="&lt;p&gt;Page 359 of &lt;a href=&quot;http://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elements Of Statistical Learning 2nd edition&lt;/a&gt; says the below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PgHFz.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PgHFz.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Can someone explain the intuition &amp;amp; simplify it in layman terms?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Questions&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the reason/intuition &amp;amp; math behind fitting each successive tree in GBM on the negative gradient of the loss function?&lt;/li&gt;&#xA;&lt;li&gt;Is it done to make GBM more generalization on unseen test dataset? If so how does fitting on negative gradient achieve this generalization on test data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1151" LastEditorUserId="1151" LastEditDate="2016-02-21T05:51:34.477" LastActivityDate="2024-02-14T07:07:45.960" Title="Why is each successive tree in GBM fit on the negative gradient of the loss function?" Tags="|machine-learning|optimization|gradient-descent|gbm|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10338" PostTypeId="1" CreationDate="2016-02-21T12:58:57.120" Score="3" ViewCount="3081" Body="&lt;p&gt;I've been reading Bing Liu's book on Sentiment Analysis. He mentions all of these slightly different approaches seen in research since 2004, but doesn't talk much about efficacy at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That leaves me - someone who has not done any sentiment analysis before - wondering what approaches are seeing the best results currently. And it specifically needs to be an approach that can extract the sentiments of individual topics found in the text, not just if a document/sentence is positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it makes a huge difference: the object of analysis will be reddit comments, not tweets or reviews which seem to be the most common source of data.&lt;/p&gt;&#xA;" OwnerUserId="16443" LastEditorUserId="31513" LastEditDate="2017-05-10T03:58:18.760" LastActivityDate="2023-09-23T11:18:59.713" Title="What is the current state-of-the-art within aspect-based sentiment analysis?" Tags="|machine-learning|nlp|sentiment-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10339" PostTypeId="1" CreationDate="2016-02-21T14:28:49.490" Score="4" ViewCount="310" Body="&lt;p&gt;If you take the following sentence from an article on deep neural networks &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;to regularize the classifier layer by estimating the marginalized effect of &lt;strong&gt;label-dropout&lt;/strong&gt; during training.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does label-dropout mean&lt;/p&gt;&#xA;" OwnerUserId="15408" LastEditorUserId="843" LastEditDate="2016-04-08T23:22:57.887" LastActivityDate="2022-02-06T12:40:36.800" Title="In Neural Networks and deep neural networks what does label-dropout mean" Tags="|classification|neural-network|deep-learning|terminology|dropout|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10340" PostTypeId="1" CreationDate="2016-02-21T15:48:03.543" Score="3" ViewCount="2494" Body="&lt;p&gt;I am seraching for a  while  an example  on how to use  PyMc/PyMc3 to do  classification task, but have not  found  an concludent   example  regarding  on how to do   the  predicton on a  new data point.&#xA;I am currious if some   could   give  me some references.&lt;/p&gt;&#xA;" OwnerUserId="14946" LastActivityDate="2020-01-21T04:05:35.657" Title="Bayesian network for classification using PyMc or PyMc3" Tags="|bayesian-networks|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10341" PostTypeId="1" CreationDate="2016-02-21T16:49:54.067" Score="2" ViewCount="80" Body="&lt;p&gt;I'm trying to create an analytical model for performance of a greedy scheduler in a particular domain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created a preliminary model, but I think it's very confusing to others and was wondering if someone could provide feedback.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem domain follows. A greedy scheduler has the capacity to schedule c tasks and will select the c highest rewarding tasks. Tasks arrive dynamically online from an arrival pattern, A--that is for every time unit within a time horizon X tasks will arrive, where X is an arbitrary random variable. Task details are all drawn from an arbitrary predefined distribution. Task details include, reward, maximum waiting time before being serviced (i.e if a task is not serviced after m time units, the task will fail), and duration. I'll label the respective distributions as R for reward, M for maximum waiting time, and D for duration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this problem is very large, so I would like to focus on a much smaller component of this larger problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a greedy agent with capacity c and no tasks currently scheduled with a task mix of T, where the size of T is larger than c and T's task's rewards are representative of the R reward distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what should be the expected reward (or even better the distribution of rewards) for the c selected tasks from T. This is similar to a k-order statistics problem; however, I've been unable to format this problem into a k-order statistics problem due to my basic/limiting understanding of statistics. (I study computer science and only know basic statistical analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach follows; however, I feel it could be improved. In my approach, I try to calculate the lowest reward of the highest rewarding tasks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the scheduler can choose c tasks from T and c &amp;lt; |T|, the scheduler will choose the highest rewarding c tasks. Since T's task's rewards are representative of R, The expected reward of a single task will be the expected reward of the upper c / |T| portion of the R distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate the expected value of a reward distribution, one takes the &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;integral from x= -infinity to x=+infinity of P(R=x)*x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly, to calculate the expected value of a distribution of the highest c rewards, we take the &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;integral from x = Y to x = +infinity of P(R=x)*x&lt;/code&gt;&#xA; where y is the lowest of the highest rewarding task's rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate the upper c/ |T| portion of the R distribution, I let the lowest reward of the highest rewarding task be Y and thus c / |T| = 1 - CDF(Y), where CDF is the cumulative distribution function applied to the reward distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c / |T| = 1 - CDF(Y) states that:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The proportion of schedulable tasks to total tasks is equal to the proportion of the upper half highest rewarding tasks of the reward distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By solving for Y, we get that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = inverse_CDF(1 - c/|T|)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can then plug the highest rewarding task's reward into our expected value equation earlier and then the expected value of the highest c rewarding tasks are then the integral from x = Y to x = +infinity of P(R=x)*x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback on how I can improve clarity or improve my analytical equation would be appreciated or if there is a more standard/correct way to describe what I'm doing any suggestions would be appreciated. Also, I'm unsure if this community is the correct community to post this question. If it is not, please provide suggestions as to where I should post this question. I've looked at similar work, and they typically use omniscient scheduling and I'm not too understanding of their syntax/approach.&lt;/p&gt;&#xA;" OwnerUserId="16449" LastEditorUserId="16449" LastEditDate="2016-02-21T17:04:13.117" LastActivityDate="2016-02-21T17:04:13.117" Title="Analytical Model for Greedy Scheduler" Tags="|statistics|predictive-modeling|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10342" PostTypeId="2" ParentId="10333" CreationDate="2016-02-21T18:36:25.060" Score="5" Body="&lt;p&gt;I think that &lt;code&gt;caret&lt;/code&gt; and &lt;code&gt;e1071&lt;/code&gt; serve difference purposes. First lets discuss &lt;code&gt;caret&lt;/code&gt;, its closest competitor is the &lt;code&gt;mlr&lt;/code&gt; package. Both are meta packages that allow to optimize models across parameters. Take for example a problem where you are not sure whether you would want to use Lasso or Ridge to create a model. As explained &lt;a href=&quot;https://rstudio-pubs-static.s3.amazonaws.com/22067_48fad02fb1a944e9a8fb1d56c55119ef.html#ridge-regression-and-lasso&quot;&gt;here&lt;/a&gt; &lt;code&gt;caret&lt;/code&gt; allows to choose the optimal lambda based on different type of cross validations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;e1071&lt;/code&gt; package instead is a bag of functions developed by the TU Wien. It is probably one of the most popular packages for support vector machines which are used by &lt;code&gt;caret&lt;/code&gt;. But its goal is very different than &lt;code&gt;caret&lt;/code&gt;, it implements learning algorithms and other functions whereas &lt;code&gt;caret&lt;/code&gt; seeks to find the best parameter in learning models.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-02-21T18:36:25.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10343" PostTypeId="1" CreationDate="2016-02-21T20:45:40.750" Score="2" ViewCount="869" Body="&lt;p&gt;am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more &quot;outliers&quot; than other by class. The issue I have however is that there is a lot of missing data. I would like to replace these NAs (they are numerical) with something like -9999. Should I first scale my data and then assign -9999 to NAs or first assign then scale?&lt;/p&gt;&#xA;" OwnerUserId="16452" LastActivityDate="2016-02-22T15:44:49.743" Title="Feature scaling data with missing values" Tags="|feature-scaling|missing-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10344" PostTypeId="2" ParentId="10343" CreationDate="2016-02-22T01:42:39.323" Score="1" Body="&lt;p&gt;Well -9999 can matter more or less depending on the variability within your data (is -9999 an extreme outlier value, or a moderate value somewhat close to the mean/median&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depends a lot on your data.  A value like that can skew it a lot, and can affect modeling quite a bit depending on the technique you use.  Are the NA's biased for a class?  Consider decision trees that classify on optimal splits.  They would detect a bias (e.g. if 90% of your NA's a certain class) and split at -9999 value to manifest that bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also check out Pareto scaling, which emphasizes small to medium changes in your data.  Might be along the lines you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastEditorUserId="14913" LastEditDate="2016-02-22T01:53:59.747" LastActivityDate="2016-02-22T01:53:59.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10347" PostTypeId="1" CreationDate="2016-02-22T12:05:42.773" Score="2" ViewCount="226" Body="&lt;p&gt;I'm working on trying to predict physical traffic volume over a network of intersections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have is sorted per intersection and consists of the time (in sec after epoch) they passed and the direction someone came from. A mock sample here (I have months worth of data for 60 intersections):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Intersection 1:{&#xA;     (1456140051, 1),&#xA;     (1456140066, 2),&#xA;     (1456140070, 2),&#xA;     (1456140073, 3),&#xA;     (1456140078, 1),&#xA;     (1456140091, 1)}&#xA;   Intersection 2:{&#xA;     (1456140051, 1),&#xA;     (1456140058, 4),&#xA;     (1456140060, 3),&#xA;     (1456140063, 2),&#xA;     (1456140067, 2),&#xA;     (1456140071, 4)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Note: the data is not for the complete network, there are many smaller intersections for which no data is gathered)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've done some fourier analysis which gives me strong weekly and daily periodicity (obviously) and thus calculated daily/weekly averages/noise etc. for each intersection, and I've looked for some correlations between some neighbouring intersections. The next step for me is to use the data from all the intersections to estimate the traffic volume at one intersection, and then do the same for the traffic volume 10 minutes in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found various machine learning methods that should work well with time series with constant time intervals and differing values (e.g. 1 data point of varying value every minute), and I can aggregate my data so I get a value of traffic volume each minute or so (typical traffic is between 5 and 100 /minute /intersection). However, I feel I would lose a lot of information that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I'm looking for it a method that can work with the data I have (or some derivative of that without much information loss) as input. Can anyone recommend some articles or point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="16199" LastActivityDate="2016-02-22T12:05:42.773" Title="Machine learning on data with only time stamps" Tags="|machine-learning|time-series|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10348" PostTypeId="2" ParentId="10343" CreationDate="2016-02-22T15:44:49.743" Score="0" Body="&lt;p&gt;If you can't use -inf, NaN or just remove the data, you should try something a little better than using a fixed value. For example, -1000*min(data)&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-02-22T15:44:49.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10349" PostTypeId="1" AcceptedAnswerId="10383" CreationDate="2016-02-22T17:04:58.207" Score="4" ViewCount="1874" Body="&lt;p&gt;I have tried to build a model to forecast the count of a particular variable.The model that was used for the purpose was poisson .Unfortunately ,i don't have enough stat knowledge to analyze the model performance .If somebody can provide some insights as of how the model is performing,as well as some tweaks to improve the model performance will be greatly helpful. I am also willing to try out other models if it performs better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using python with the statmodels package to build the model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Attaching a graph which shows the fitted and the actual&#xA;  values(&lt;strong&gt;Green&lt;/strong&gt; shows the actual values and &lt;strong&gt;Blue&lt;/strong&gt; shows the fitted&#xA;  values)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YtRYb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YtRYb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Also,providing the summary() output of the model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                 Generalized Linear Model Regression Results                  &#xA;==============================================================================&#xA;Dep. Variable:         Work_Item_Type   No. Observations:                  581&#xA;Model:                            GLM   Df Residuals:                      574&#xA;Model Family:                 Poisson   Df Model:                            6&#xA;Link Function:                    log   Scale:                             1.0&#xA;Method:                          IRLS   Log-Likelihood:                -16752.&#xA;Date:                Mon, 22 Feb 2016   Deviance:                       31268.&#xA;Time:                        21:59:12   Pearson chi2:                 1.05e+05&#xA;No. Iterations:                     9                                         &#xA;===============================================================================&#xA;                  coef    std err          z      P&amp;gt;|z|      [95.0% Conf. Int.]&#xA;-------------------------------------------------------------------------------&#xA;Intercept       2.8492      0.051     55.426      0.000         2.748     2.950&#xA;Weekday        -0.2066      0.032     -6.446      0.000        -0.269    -0.144&#xA;day_of_week    -0.0926      0.007    -13.367      0.000        -0.106    -0.079&#xA;wom             0.1122      0.007     16.996      0.000         0.099     0.125&#xA;week           -0.0411      0.001    -53.597      0.000        -0.043    -0.040&#xA;TimeDelta       0.0001    5.1e-05      2.933      0.003      4.96e-05     0.000&#xA;month_of_yr     0.2192      0.004     60.981      0.000         0.212     0.226&#xA;===============================================================================&#xA;&#xA;&#xA;Also attaching a sample of the dataset used &#xA;&#xA;clear_date  Count_Work_Item_Type&#xA;7/7/2014    1&#xA;7/10/2014   1&#xA;7/11/2014   5&#xA;7/17/2014   2&#xA;7/22/2014   1&#xA;7/24/2014   1&#xA;7/29/2014   3&#xA;7/30/2014   4&#xA;8/13/2014   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since i had only the date and the variable to be forecast i created a bunch of other variables like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Weekday(binomial)?&#xA;Day of week&#xA;Week of Month&#xA;Week&#xA;Time Delta (Starts from 0 increment by one until end)&#xA;Month of Year&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also,i haven't done any kind of transformation on the variables.&#xA;Please do comment if you need additional information:&#xA;Thhanks&lt;/p&gt;&#xA;" OwnerUserId="13515" LastActivityDate="2020-08-03T18:11:20.417" Title="Analyze performance Poisson regression model on a time series(count forecasting)" Tags="|python|time-series|forecast|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10350" PostTypeId="2" ParentId="10301" CreationDate="2016-02-22T18:15:10.043" Score="4" Body="&lt;p&gt;I suggest to calculate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Haversine_formula&quot; rel=&quot;nofollow noreferrer&quot;&gt;Haversine&lt;/a&gt; distance between two points, and fit a linear regression to find the relation between the Haversine distance and the trip duration.&#xA;So your regression will be&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$duration_t = timestamp_t - timestamp_{t-1} = \alpha + \beta*d(point_t,point_{t-1})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Where &lt;span class=&quot;math-container&quot;&gt;$d$&lt;/span&gt; is the Haversine distance. &lt;span class=&quot;math-container&quot;&gt;$point_t$&lt;/span&gt; is a lat/long pair at time &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Note however that there's an assumption that the user drives at the same speed. If half of your data was gathered while walking and half while driving, then your relation between time and ETA is possibly not linear.&lt;/p&gt;&#xA;" OwnerUserId="8113" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-09-19T12:13:46.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10351" PostTypeId="2" ParentId="10234" CreationDate="2016-02-22T18:54:45.957" Score="1" Body="&lt;p&gt;I hope this &lt;a href=&quot;http://www.modelingonlineauctions.com/datasets&quot; rel=&quot;nofollow&quot;&gt;data set might&lt;/a&gt; suits you.This contains details of eBay Online Auctions (2012). very similar to your bank customer database you looking for. And all CSV format. Therefore you can import it to SQL also.Just Tryout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;File Format&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;All files are available in comma separated format (CSV).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This dataset contain details of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Field Description&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;auctionid&lt;/em&gt; - unique identifier of an auction&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;bid&lt;/em&gt; - the proxy bid placed by a bidder&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;bidtime&lt;/em&gt; - the time (in days) that the bid was placed, from the start of the auction&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;bidder&lt;/em&gt; - eBay username of the bidder&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;bidderrate&lt;/em&gt; - eBay feedback rating of the bidder&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;openbid - the opening bid set by the seller&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;em&gt;price&lt;/em&gt; - the closing price that the item sold for (equivalent to the second highest bid + an increment)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Or&lt;/strong&gt; &#xA;My personal favorite &lt;a href=&quot;https://code.google.com/archive/p/open-advertising-dataset/downloads&quot; rel=&quot;nofollow&quot;&gt;Google Adwords dataset&lt;/a&gt;. You need to really check it to know how much information it has...&lt;/p&gt;&#xA;" OwnerUserId="9035" LastEditorUserId="9035" LastEditDate="2016-02-22T19:01:47.150" LastActivityDate="2016-02-22T19:01:47.150" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10352" PostTypeId="2" ParentId="10301" CreationDate="2016-02-22T23:19:26.997" Score="0" Body="&lt;p&gt;The problem is with how you are training the problem.  If you use Timestamp1 and Timestamp2 as training parameters, they will carry 100% predictive power, and the algorithm will completely disregard any location parameters.  If you want to make predictions based on only an origin and destination, you need to train your model using only those parameters.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-02-22T23:19:26.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10353" PostTypeId="1" CreationDate="2016-02-23T02:17:59.543" Score="1" ViewCount="31" Body="&lt;p&gt;Is there a generalisation of $k$-handed bandit to $n$-handed player in a sense that the sample is taken not from just one distribution (one bandit from $k$ bandits) but from n distributons ($n$ bandits) where $n&amp;lt;k$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words instead of predicting best ad banner in terms of click though rate I'd need to predict a set of ad banners that will be shown simultaneously on the page (but only one clicked or not and it does not matter which). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or yet another way to formulate is to say that the generalized $n$-arm is a vector of bandits arms instead of one banidts arm. The pool of bandits is fixed.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-23T02:17:59.543" Title="generalisation of k-handed bandit to n-handed player" Tags="|machine-learning|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10354" PostTypeId="2" ParentId="10338" CreationDate="2016-02-23T02:45:38.143" Score="2" Body="&lt;p&gt;Basically, there are two types of sentiment analysis.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Symbolic approach - the sentiment of the sentence is classified based on the kind of words used in the sentence. Involves a lexical database such as wordnet and some rules based on the grammatical structure of the language as knowledge representation.&lt;/li&gt;&#xA;&lt;li&gt;Vector models - rather than detecting the sentiment in a sentence, the query sentence is classified into one of the category based on the training dataset and the features are represented as word vectors. Distributed and Distributional representations are types of it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Whether it is coarse-classification (positive - negative) or fine-classification (types of emotions in it), a training dataset is needed if you are going to approach it via vector models as it is much more easier to play with.&lt;/p&gt;&#xA;&lt;p&gt;The skip-gram model and continuous bag of words model developed in Google based on Recurrent neural network language model made a breakthrough in Natural language processing seems to be the state-of-art model in Distributional representation.&lt;/p&gt;&#xA;&lt;p&gt;Since the code has been open-sourced, the c-code is available &lt;a href=&quot;https://code.google.com/archive/p/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and the python library is maintained &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It doesn't matter if you are going to use reddit comments, tweets or reviews. What matters is how well the sentences are classified for the training data. Because garbage in is garbage out.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="98307" LastEditDate="2020-08-07T09:28:58.413" LastActivityDate="2020-08-07T09:28:58.413" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="10356" PostTypeId="2" ParentId="8024" CreationDate="2016-02-23T14:05:49.773" Score="1" Body="&lt;p&gt;Although not required, it is extremely common to use some approximation scheme once you start working with larger problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are thinking of value-based model-free RL, a typical problem is that the state-space if huge. Using some approximation scheme is necessary not only to store but also useful if you can generalise and take advantage some structure in your data. It is then any function approximation method (regression, neural networks, etc.) can be beneficial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For model-based approaches something different happens. You need to build up a model of the environment based on data. Using sampled data, you try to approximate your transition and reward functions so you can later use planning methods. Again, supervised learning is applicable.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2016-02-24T21:42:25.553" LastActivityDate="2016-02-24T21:42:25.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10357" PostTypeId="1" AcceptedAnswerId="10361" CreationDate="2016-02-23T14:28:29.563" Score="1" ViewCount="62" Body="&lt;p&gt;How do linear learning systems, such as the simple &quot;closest to the class average&quot; algorithm or SVMs, classify datapoints that fall on the hyperplane?&lt;/p&gt;&#xA;" OwnerUserId="11044" LastActivityDate="2016-02-24T06:54:18.073" Title="How do linear learning systems classify datapoints that fall on the hyperplane" Tags="|machine-learning|classification|algorithms|svm|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10358" PostTypeId="2" ParentId="9832" CreationDate="2016-02-23T14:51:06.107" Score="36" Body="&lt;p&gt;Q-values are a great way to the make actions explicit so  you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are not so nice and Q-values are not so convenient. Think of a huge number of actions or even continuous action-spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a sampling perspective, the dimensionality of &lt;span class=&quot;math-container&quot;&gt;$Q(s, a)$&lt;/span&gt; is higher than &lt;span class=&quot;math-container&quot;&gt;$V(s)$&lt;/span&gt; so it might get harder to get enough &lt;span class=&quot;math-container&quot;&gt;$(s, a)$&lt;/span&gt; samples in comparison with &lt;span class=&quot;math-container&quot;&gt;$(s)$&lt;/span&gt;. If you have access to the transition function sometimes &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; is good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also other uses where both are combined. For instance, the advantage function where &lt;span class=&quot;math-container&quot;&gt;$A(s, a) = Q(s, a) - V(s)$&lt;/span&gt;. If you are interested, you can find a recent example using advantage functions here:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1511.06581&quot; rel=&quot;noreferrer&quot;&gt;Dueling Network Architectures for Deep Reinforcement Learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;by Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot and Nando de Freitas.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2019-02-19T09:27:34.170" LastActivityDate="2019-02-19T09:27:34.170" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10359" PostTypeId="2" ParentId="9797" CreationDate="2016-02-23T15:10:46.217" Score="2" Body="&lt;p&gt;I think you will like the following two papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Available from: &lt;a href=&quot;http://arxiv.org/abs/1507.04296&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1507.04296&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nair A, Srinivasan P, Blackwell S, Alcicek C, Fearon R, De Maria A, et al. Massively Parallel Methods for Deep Reinforcement Learning. arXiv preprint arXiv:150704296&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Available from: &lt;a href=&quot;http://arxiv.org/abs/1602.01783&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1602.01783&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mnih V, Badia AP, Mirza M, Graves A, Lillicrap TP, Harley T, et al. Asynchronous Methods for Deep Reinforcement Learning. arXiv:160201783&lt;/p&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2016-02-23T15:10:46.217" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10360" PostTypeId="1" AcceptedAnswerId="10365" CreationDate="2016-02-23T17:00:22.933" Score="1" ViewCount="4835" Body="&lt;p&gt;I am trying to replicate the below sql query in R &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select a.*, b.key from Table1 a &#xA;LEFT OUTER JOIN Table2 b &#xA;on a.key = b.key where b.key is null&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have read through this post however I am still struggling to code my specific case. &lt;a href=&quot;https://stackoverflow.com/questions/1299871/how-to-join-merge-data-frames-inner-outer-left-right&quot;&gt;https://stackoverflow.com/questions/1299871/how-to-join-merge-data-frames-inner-outer-left-right&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried the below but the result doesnot allow me to filter for &lt;code&gt;b.key IS NULL&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LoansToInsert_stg1 &amp;lt;- merge(x = Prior_stg2, y = BankLoans_stg2, &#xA;                            by = &quot;Account_ID&quot;, all.x = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any insights?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Key1 &amp;lt;- c(&quot;A1&quot;,&quot;A2&quot;,&quot;A3&quot;,&quot;A4&quot;,&quot;A5&quot;)  &#xA;Key2 &amp;lt;- c(&quot;A1&quot;,&quot;A2&quot;,&quot;A3&quot;,&quot;B4&quot;,&quot;B5&quot;)  &#xA;BV1 &amp;lt;- c(100, 200, 300, 400, 500)  &#xA;BV2 &amp;lt;- c(150, 250, 350, 450, 550)  &#xA;df1 &amp;lt;- as.data.frame(cbind(Key1, BV1))  &#xA;df2 &amp;lt;- as.data.frame(cbind(Key2, BV2))  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Expected Output as a new df:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Key1 BV1 Key2 BV2&#xA;A1 100 A1 150&#xA;A2 200 A2 250&#xA;A3 300 A3 350&#xA;A4 400 NA NA&#xA;A5 500 NA NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-02-23T21:43:08.693" Title="Left Join with b.key being NULL in R" Tags="|r|sql|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10361" PostTypeId="2" ParentId="10357" CreationDate="2016-02-23T17:08:03.497" Score="6" Body="&lt;p&gt;Linear, binary classifiers can choose either class (but consistently) when the datapoint which is to classify is on the hyperplane. It just depends on how you programmed it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it doesn't really matter. This is very unlikely to happen. In fact, if we had arbitrary precision computing and normal distributed features, there would be a probability of 0 (exactly, not rounded) that this would happen. We have IEEE 754 floats, so the probability is not 0, but still so small that there are much more important factors to worry about.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-02-24T06:54:18.073" LastActivityDate="2016-02-24T06:54:18.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10362" PostTypeId="2" ParentId="10302" CreationDate="2016-02-23T18:24:58.230" Score="4" Body="&lt;p&gt;In Q-Learning, on every step you will use observations and rewards to update your Q-value function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xA;Q_{t+1}(s_t,a_t) = Q_t(s_t,a_t) +  \alpha [R_{t+1}+ \gamma  \underset{a'}{\max} Q_t(s_{t+1},a') - Q_t(s_t, a_t)]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are correct in saying that the neural network is just a function approximation for the q-value function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, the approximation part is just a standard supervised learning problem. Your network uses (s,a) as input and the output is the q-value. As q-values are adjusted, you need to train these new samples to the network. Still, you will find some issues as you as using correlated samples and SGD will suffer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking at the DQN paper, things are slightly different. In that case, what they are doing is putting samples in a vector (experience replay). To teach the network, they sample tuples from the vector, bootstrap using this information to obtain a new q-value that is taught to the network. When I say teaching, I mean adjusting the network parameters using stochastic gradient descent or your favourite optimisation approach. By not teaching the samples in the order that are being collected by the policy the decorrelate them and that helps in the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, in order to make a decision on state $ s $, you choose the action that provides the highest q-value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;a^*(s)= \underset{a}{argmax} \space Q(s,a)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your Q-value function has been learnt completely and the environment is stationary, it is fine to be greedy at this point. However, while learning, you are expected to explore. There are several approaches being $\varepsilon$-greedy one of the easiest and most common ways.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastActivityDate="2016-02-23T18:24:58.230" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10363" PostTypeId="1" AcceptedAnswerId="10364" CreationDate="2016-02-23T18:25:24.797" Score="2" ViewCount="9445" Body="&lt;p&gt;In one field I have entries like 'U$ 192,0'. Working on pandas, how I ignore non numerical data and get only the numerical part?&lt;/p&gt;&#xA;" OwnerUserId="16096" LastEditorUserId="16096" LastEditDate="2016-02-23T19:10:43.467" LastActivityDate="2019-06-09T07:24:57.953" Title="Ignoring symbols and select only numerical values with pandas" Tags="|python|data-cleaning|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10364" PostTypeId="2" ParentId="10363" CreationDate="2016-02-23T18:59:11.567" Score="3" Body="&lt;p&gt;Use &lt;code&gt;str.strip&lt;/code&gt; if the prefix is fixed or &lt;code&gt;str.replace&lt;/code&gt; if not:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = pandas.Series([&quot;U$ 192.0&quot;])&#xA;data.str.replace('^[^\d]*', '').astype(float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This removes all the non-numeric characters to the left of the number, and casts to float.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-23T18:59:11.567" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10365" PostTypeId="2" ParentId="10360" CreationDate="2016-02-23T19:40:37.380" Score="2" Body="&lt;p&gt;If I understand correctly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Table1 &amp;lt;- data.frame(key = seq(1,100),a.data = rnorm(100))&#xA;Table2 &amp;lt;- data.frame(key = c(seq(1,30),rep(NA,30)), b.data = seq(1,60))&#xA;&#xA;&#xA;##Assuming this is what you want&#xA;library(sqldf)&#xA;sql.ans &amp;lt;- sqldf(&quot;select a.*, b.key from Table1 a LEFT OUTER JOIN Table2 b on a.key = b.key where b.key is null&quot;)&#xA;&#xA;## dplyr version&#xA;library(dplyr)&#xA;&#xA;dplyr.ans &amp;lt;- Table1 %&amp;gt;% filter(!key %in% Table2$key)&#xA;&#xA;## Regular R version&#xA;R.ans &amp;lt;- Table1[which(!Table1$key %in% Table2$key),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT after dummy data and expected output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dplyr.ans2 &amp;lt;- left_join(df1,df2, by = c(&quot;Key1&quot; = &quot;Key2&quot;))&#xA; Key1 BV1  BV2&#xA; 1   A1 100  150&#xA; 2   A2 200  250&#xA; 3   A3 300  350&#xA; 4   A4 400 &amp;lt;NA&amp;gt;&#xA; 5   A5 500 &amp;lt;NA&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8113" LastEditorUserId="8113" LastEditDate="2016-02-23T21:42:36.893" LastActivityDate="2016-02-23T21:42:36.893" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10368" PostTypeId="1" AcceptedAnswerId="10782" CreationDate="2016-02-24T04:17:12.343" Score="6" ViewCount="3679" Body="&lt;p&gt;I am trying to implement demo of Image Captioning system from &lt;a href=&quot;http://keras.io/examples/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras documentation&lt;/a&gt;. From the documentation I could understand training part. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;max_caption_len = 16&#xA;vocab_size = 10000&#xA;&#xA;# first, let's define an image model that&#xA;# will encode pictures into 128-dimensional vectors.&#xA;# it should be initialized with pre-trained weights.&#xA;image_model = VGG-16 CNN definition&#xA;image_model.load_weights('weight_file.h5')&#xA;&#xA;# next, let's define a RNN model that encodes sequences of words&#xA;# into sequences of 128-dimensional word vectors.&#xA;language_model = Sequential()&#xA;language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))&#xA;language_model.add(GRU(output_dim=128, return_sequences=True))&#xA;language_model.add(TimeDistributedDense(128))&#xA;&#xA;# let's repeat the image vector to turn it into a sequence.&#xA;image_model.add(RepeatVector(max_caption_len))&#xA;&#xA;# the output of both models will be tensors of shape (samples, max_caption_len, 128).&#xA;# let's concatenate these 2 vector sequences.&#xA;model = Merge([image_model, language_model], mode='concat', concat_axis=-1)&#xA;# let's encode this vector sequence into a single vector&#xA;model.add(GRU(256, 256, return_sequences=False))&#xA;# which will be used to compute a probability&#xA;# distribution over what the next word in the caption should be!&#xA;model.add(Dense(vocab_size))&#xA;model.add(Activation('softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', optimizer='rmsprop')&#xA;&#xA;model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But now I am confused in how to generate caption for test image. Input here is [image, partial_caption] pair, now for test image how to input partial caption?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="29575" LastEditDate="2017-10-23T07:55:27.233" LastActivityDate="2017-10-23T07:55:27.233" Title="Image Captioning in Keras" Tags="|neural-network|deep-learning|image-classification|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10369" PostTypeId="1" CreationDate="2016-02-24T06:07:13.507" Score="5" ViewCount="502" Body="&lt;p&gt;Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the best ways to find the most influential columns/features? I know it's important to know what models to run first, but is there a general way to find columns that's more important than others? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've already removed the columns that are 80%+ empty, but I still have over 600 columns and I have no idea where to start. Please let me know your thoughts! I'm kind of new so any pointers will help.&lt;/p&gt;&#xA;" OwnerUserId="16511" LastEditorUserId="13727" LastEditDate="2016-02-24T12:36:45.890" LastActivityDate="2020-08-17T20:32:47.020" Title="General way to reduce features" Tags="|data-mining|feature-selection|data|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10370" PostTypeId="2" ParentId="10369" CreationDate="2016-02-24T06:49:50.923" Score="5" Body="&lt;p&gt;Another way of looking at 600 columns is having a vector with 600 dimensions. So, the answer to reduce the 600 dimensional vectors to fewer dimensions without losing lot of information will be dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal component analysis(PCA) is a commonly used one. T-distributed stochastic neighbour embedding (TSNE) has been the state of the art for reducing higher dimensional vectors. &lt;a href=&quot;https://indico.io/blog/visualizing-with-t-sne/&quot; rel=&quot;nofollow&quot; title=&quot;data visualisaiton using TSNE&quot;&gt;Data visualisation using TSNE&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the quick intuition about the vectors, the best thing is to reduce it to 2 or 3 dimensional vectors and plot them in a graph. Data visualisation always helps to get an overview of the data you are dealing with.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2016-02-24T16:07:58.640" LastActivityDate="2016-02-24T16:07:58.640" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10371" PostTypeId="2" ParentId="10369" CreationDate="2016-02-24T07:17:11.247" Score="5" Body="&lt;p&gt;Since you didn't mention labels of any kind, I presume you are not doing supervised learning. Note, if you were, there would be no need to throw out features. You could use something like lasso regression to build a model that sparsely selects the influential features while fitting a predictive model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unsupervised methods exist to reduce your feature set size but could potentially lose their original feature interpretability (if that's important to you), like PCA or auto encoders. But there are alternatives to these as well, so depends what you are looking for. &lt;/p&gt;&#xA;" OwnerUserId="283" LastEditorUserId="283" LastEditDate="2016-02-24T08:06:06.083" LastActivityDate="2016-02-24T08:06:06.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10373" PostTypeId="2" ParentId="6201" CreationDate="2016-02-24T10:59:20.057" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://danielweitzenfeld.github.io/passtheroc/blog/2014/10/28/bayes-premier-league/&quot; rel=&quot;nofollow&quot;&gt;http://danielweitzenfeld.github.io/passtheroc/blog/2014/10/28/bayes-premier-league/&lt;/a&gt; uses a bayesian approach on data taken from wikipedia (You can use any source in fact.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It uses the following paper: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statistica.it/gianluca/Research/BaioBlangiardo.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.statistica.it/gianluca/Research/BaioBlangiardo.pdf&lt;/a&gt;&#xA;Bayesian hierarchical model for the prediction of football/soccer results&lt;/p&gt;&#xA;" OwnerUserId="16520" LastEditorUserId="75" LastEditDate="2016-02-26T12:35:34.247" LastActivityDate="2016-02-26T12:35:34.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10374" PostTypeId="2" ParentId="10369" CreationDate="2016-02-24T12:19:09.397" Score="2" Body="&lt;p&gt;There are a lot of feature selection techniques, but depends if you have the &amp;quot;labels&amp;quot; of the samples. &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;One simple technique&lt;/a&gt; involves the variance which&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;removes all features whose variance doesn’t meet some threshold&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;By default, it removes all zero-variance features&amp;quot;&lt;/p&gt;&#xA;" OwnerUserId="11007" LastEditorUserId="98307" LastEditDate="2020-08-17T20:32:47.020" LastActivityDate="2020-08-17T20:32:47.020" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10375" PostTypeId="2" ParentId="2495" CreationDate="2016-02-24T15:33:03.407" Score="0" Body="&lt;p&gt;As far as I know there are no parallel implementations of SVM available directly in R. The benefit of running on RevolutionR is that it will parallelize the matrix computations through the use of the Intel MKL library. Since SVM relies on linear algebra you will achieve probably some performance gain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option that you could try and which may lead to better results is using Spark and &lt;a href=&quot;https://amplab-extras.github.io/SparkR-pkg/&quot; rel=&quot;nofollow&quot;&gt;SparkR&lt;/a&gt; which could expose the &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-linear-methods.html#linear-support-vector-machines-svms&quot; rel=&quot;nofollow&quot;&gt;SVM in MLib&lt;/a&gt; that is part of Spark. I have not tried this myself yet but if you are stuck this might help.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-02-24T15:33:03.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10377" PostTypeId="1" AcceptedAnswerId="11451" CreationDate="2016-02-24T17:54:04.143" Score="2" ViewCount="1113" Body="&lt;p&gt;I’ve a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables).&#xA;Here is the code that I used,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;set.seed(1)&#xA;train=sample(seq(678),452,replace=FALSE)&#xA;train&#xA;&#xA;regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method=&quot;exhaustive&quot;)&#xA;&#xA;val.errors=rep(NA,14)&#xA;&#xA;x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])     &#xA;for(i in 1:14){&#xA;  coefi=coef(regfit.exh,id=i)&#xA;  pred=x.test[,names(coefi)]%*%coefi&#xA;  val.errors[i]=mean((Mydata$HPV[-train]-pred)^2)&#xA;}&#xA;&#xA;plot(sqrt(val.errors),ylab=&quot;Root MSE&quot;,ylim=c(3,12), pch=11, type=&quot;b&quot;)&#xA;points(sqrt(regfit.exh$rss[-1]/452),col=&quot;blue&quot;,pch=11,type=&quot;b&quot;)&#xA;legend(&quot;topright&quot;,legend=c(&quot;Training&quot;,&quot;Validation&quot;),col=c(&quot;blue&quot;,&quot;black&quot;),pch=11)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R4OwJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R4OwJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How come the validation root MSE could always beat the training?. Any feedback would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="11097" LastEditDate="2016-02-25T02:33:05.247" LastActivityDate="2016-04-30T03:07:50.653" Title="Comparing training and validation data set Root MSE for a best subset regression?" Tags="|r|regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10378" PostTypeId="2" ParentId="9716" CreationDate="2016-02-24T17:58:35.893" Score="1" Body="&lt;p&gt;It is mostly a performance benefit thing. You can run a neural network on a computer (slow or not). A TCAM is the only real hardware suited to the problem of seeing the cause and supplying the effect, TCAM can perform hopfield net operations very natively in pure hardware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It puts the network in a hardware spreadsheet formfactor.  It's really simple, and can be bewilderingly so, you have a linear amount of patterns, (which I like to call bullets, because it's like a machine gun ready to fire off and enter in patterns really damn fast, faster than a minigun)    and requires a lot of miniaturization to have a large amount, say a billion bullets,  more so than just ordinary RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you ever wanted to see a &quot;real&quot; robot in action,  it's going to use hardware TCAM.&lt;/p&gt;&#xA;" OwnerUserId="16528" LastEditorDisplayName="user380" LastEditDate="2017-07-13T15:48:19.887" LastActivityDate="2017-07-13T15:48:19.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10379" PostTypeId="2" ParentId="8024" CreationDate="2016-02-24T19:04:11.023" Score="1" Body="&lt;p&gt;I don't think you need prior knowledge like SVM and MLP. In fact, reinforcement is another type of machine learning beside supervised learning, which including SVM, and unsupervised learning, which includes MLP. Reinforcement learning is actually very different from the latter two, as it is learning for interaction (agent-environment interaction). The trade-off between exploration and exploit is the key point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Markov decision process is the basic framework for reinforcement learning, which is very different from the other two types of learning. I highly recommend the textbook &quot;&lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning: An Introduction&lt;/a&gt;&quot; by Richard S. Sutton and Andrew G. Barto&quot;. That's the book I am reading now. The language is very easy to follow and the content is comprehensive.&lt;/p&gt;&#xA;" OwnerUserId="16530" LastEditorUserId="836" LastEditDate="2017-12-07T17:04:47.457" LastActivityDate="2017-12-07T17:04:47.457" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10382" PostTypeId="2" ParentId="10369" CreationDate="2016-02-25T00:23:39.297" Score="4" Body="&lt;p&gt;Purging columns with &gt;80% zeros might have been a mistake. Those non-zero values may be the most significant bits in your data. There's a whole field of study in text mining dedicated to discovering knowledge from &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag of words&lt;/a&gt; models which are tables like yours except can be hundreds of thousands of features wide and &gt;99% empty (i.e. sparse).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, an idiots approach to feature selection is to fit a linear curve (with linear or logistic regression; or another simple model) to the data, then extract the individual feature coefficients from the model. Those are your influence weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple tool to visualize feature importances is &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt;. I find it's Rank widget works for me most of the time.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/q4AF0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q4AF0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-02-25T00:23:39.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10383" PostTypeId="2" ParentId="10349" CreationDate="2016-02-25T00:39:31.590" Score="2" Body="&lt;p&gt;I'm not sure what you mean by &quot;performance&quot;, but if what you mean is fit the answer is clear. You need to be using the log-likelihood to differentiate between different models. Basically, when you are fitting the model you are trying to maximize the log-likelihood. Thus the log-likelihood is giving you some sense of how well the parameters of your model are doing at fitting the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you want to get log-likelihood to be as close to zero as possible. Now this is kind of terrible advice, because if you were clever enough to come up with a feature for every observation, you could get a perfect fit. That's bad because your model would be completely useless. There are functions that take your log-likelihood as an input and transform it to penalize you for adding more variables, etc. We won't worry about those right now. Just keep in mind not to mindlessly lower the log-likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a model that you can live with you should run some sort of cross-validation, and/or hold out set. Then you can use any number of metrics to validate the predictive performance of your model. I think that is the more important of the two issues. You could calculate the mean square error on your hold out set. $$MSE=\sum_{i=1}^n(\hat y_i - y_i)^2$$&#xA;This would give you a really basic metric to assess how well your model is predictive of the output.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-02-25T00:39:31.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10384" PostTypeId="2" ParentId="10027" CreationDate="2016-02-25T00:42:47.980" Score="2" Body="&lt;p&gt;Yes, there are characteristics. You can use a correlogram to inform you as to the error structure in your data. That will tell you whether or not your data needs to account for AR and/or MA terms. Also check for unit roots to tell you whether you need to difference the time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat510/node/60&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; has a good introduction to analyzing the correlogram that shows the pattern for AR(1) process. It is charecterized by a declining or oscillating but declining values. Depending on whether or not you have positive or negative autocorrelation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ACF for an MA(1) process is below. It is characterized by one significant value and then non-significant oscillating values thereafter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YJrrw.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YJrrw.jpg&quot; alt=&quot;MA(1) Process&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To tell if you need to do differencing you should use a test like the augmented dickey-fuller test to check for the unit roots (i.e. your data is integrated).&lt;/p&gt;&#xA;" OwnerUserId="16538" LastEditorUserId="16538" LastEditDate="2016-02-25T16:20:06.223" LastActivityDate="2016-02-25T16:20:06.223" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10385" PostTypeId="1" CreationDate="2016-02-25T05:42:07.063" Score="1" ViewCount="1091" Body="&lt;p&gt;I am working on a data mining project on NBA data. I want to make a recommendation system similar to the google one, where you search for players and you get recommendation for similar players. I intend to make such system based on their stats. My data looks like &lt;a href=&quot;https://github.com/joed7/Data-Mining-Project/blob/master/data/active.csv&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;14  Tony Allen  709     357     15386   2178    4580    0.476   102     363     0.281   2076    4217    0.492   0.487   1220    1701    0.717   804     1567    2371    942     995     299     964     1675    5678&#xA;&#xA;15  Al-Farouq Aminu     431     228     9988    1119    2592    0.432   188     607     0.31    931     1985    0.469   0.468   530     731     0.725   618     1724    2342    488     413     245     540     813     2956&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One approach I have in mind is using k-means clustering, but I am not sure how I will use the clusters to find similar players. Each cluster will have hundreds of players, I will need further filtering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have any suggestions/tips for handling this problem?&lt;/p&gt;&#xA;" OwnerUserId="16540" LastEditorUserId="10024" LastEditDate="2016-02-25T13:34:58.643" LastActivityDate="2016-02-25T13:34:58.643" Title="Can i find similar players using a clustering method like the k-mean algorithm?" Tags="|clustering|recommender-system|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10386" PostTypeId="1" CreationDate="2016-02-25T09:53:29.630" Score="4" ViewCount="328" Body="&lt;p&gt;I am Bachelors of Engineering in Software Engineering field and having 3 years professional work experience in Oracle ERP/Database related technologies. Now I'm planning to pursue MSc in Data Science. What I need to know that what sort of Mathematics skills should a Data Scientist have? As we had learned numerous subjects like Numerical Analysis, Statistics, Linear Algebra etc.&lt;/p&gt;&#xA;" OwnerUserId="16542" LastEditorUserId="11097" LastEditDate="2016-02-25T13:35:56.203" LastActivityDate="2016-02-25T13:35:56.203" Title="Role and need of Mathematics in Data Science" Tags="|career|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10387" PostTypeId="2" ParentId="10385" CreationDate="2016-02-25T10:08:44.567" Score="2" Body="&lt;p&gt;You can go as further or specific you want with this kind of data.&#xA;I would suggest first to analyze the data thoroughly before deciding which algorithms to apply. Evaluate the mean, quartiles, max and min of each parameter (such as in a box plot), verify if you have missing values and in that case decide a way of dealing with that (removing data, imputating - predicting missing value from data - or average, as common techniques). Also verify if is possible to create a new representative feature from the information you have, or even mix a set of features to reduce (a priori feature selection; just ok if it makes a lot of sense from your knowledge on the context - NBA). For example, a player position can be considerably representative, in the sense of giving context to data (e.g. it will come up that defensive players steal more balls than attacking players but score less).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Afterwards I would suggest some simple algorithms such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Hierarchical clustering&lt;/strong&gt;:  can help you to build a genealogy of players, so you see which ones cluster together directly and which others influence this cluster (so they are similar in certain more general aspects but not so much so to be clustered together).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;K-Means&lt;/strong&gt;: Well, its always good to try it; it won't be that expensive in computational terms. The thing is that determining K will be rather complicated, but it can give you hints from a Visualization point of view, while not actually helping directly the task of information retrieval.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;K-Nearest-Neighbors&lt;/strong&gt;: Also determining K is complex (but not that much) it can give a first hint for retrieving similar players based on their distance in the space. This is a form of case-based reasoning, but actually is quite practical for your purposes. The key thing here is determining the right distance measure. Start with Euclidean as a typical approach. Then you can try some other distance measures such as the Malahanobis Distance, which takes into account the covariance of features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can have a first glance and even a functional solution following these approaches. Once observing the results you can get deeper (if necessary).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember to normalize or standarize the data previously. You could also visualize the first principal components after applying PCA or LDA (this last if you have a target variable) so to observe how well the data distributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same methods can be applied after refining the data through feature selection and/or extraction techniques; which in some cases produces very interesting results.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="5143" LastEditDate="2016-02-25T10:49:55.887" LastActivityDate="2016-02-25T10:49:55.887" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10388" PostTypeId="2" ParentId="10385" CreationDate="2016-02-25T10:14:56.000" Score="3" Body="&lt;p&gt;For &lt;strong&gt;finding similar&lt;/strong&gt; objects, k-means does not make much sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;k-means is based on similar objects. If squared deviations don't work for your data then k-means will not work either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, instead of using k-means, just use a &lt;strong&gt;nearest neighbor search&lt;/strong&gt; without k-means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-02-25T10:14:56.000" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10389" PostTypeId="1" AcceptedAnswerId="10393" CreationDate="2016-02-25T10:30:42.690" Score="3" ViewCount="293" Body="&lt;p&gt;When we have linearly inseparable datasets and we are using machine learning algorithms such as SVMs, we use kernels to &lt;em&gt;implicitly&lt;/em&gt; map datapoints into a feature space that makes them linearly separable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But how do we know if a kernel has indeed, &lt;em&gt;implicitly&lt;/em&gt;, been successful in making the datapoints linearly separable in the new feature space? What is the guarantee?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="11044" LastActivityDate="2016-02-25T17:47:01.463" Title="How do we know Kernels are successful in making data linearly Separable?" Tags="|machine-learning|classification|svm|supervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10390" PostTypeId="2" ParentId="10386" CreationDate="2016-02-25T13:25:42.463" Score="3" Body="&lt;p&gt;Firstly, I think this question is both broad and opinion-based, however most people don't think so. I'll explain why:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science is a domain which various very much with the industry it is applied in, so very much application-based.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Someone like me in the growth and customer support segment, I use markov models and reinforcement learning, as customers moves can most likely be modeled as a (stochastic) graph. So, when I look out for hiring someone into my team, I look for graph theory skills, linear algebra (obviously) and statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for someone working in the segment where NLP is used, the math needed is different, and so does with every other application of data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some really very important courses which help someone in the path of a data scientist according to me are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Linear Algebra&lt;/li&gt;&#xA;&lt;li&gt;Statistics&lt;/li&gt;&#xA;&lt;li&gt;Probability theory&lt;/li&gt;&#xA;&lt;li&gt;Discrete Math&lt;/li&gt;&#xA;&lt;li&gt;Analytical methodology (Numerical Analysis would be a &lt;em&gt;close to precise&lt;/em&gt; name for that)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some would argue that &lt;em&gt;Information Theory&lt;/em&gt; is also very important, however I haven't really used it heavily in my day job, so I don't think it's absolutely necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/816/11097&quot;&gt;Helpful Post&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-02-25T13:25:42.463" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10391" PostTypeId="1" CreationDate="2016-02-25T14:12:30.200" Score="2" ViewCount="161" Body="&lt;p&gt;I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www2007.org/papers/paper570.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www2007.org/papers/paper570.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://chrisjmccormick.wordpress.com/2015/06/12/minhash-tutorial-with-python-code/&quot; rel=&quot;nofollow&quot;&gt;https://chrisjmccormick.wordpress.com/2015/06/12/minhash-tutorial-with-python-code/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The data that I am working with consists of interactions between users and items. There are 2.2M distinct users and 440M distinct items. Across all of the data, there are only 905M records, so it is very sparse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my approach, I am calculating H minimum hash values for each user by reordering the items (of which there are 440M). Users have a wide range of item interactions. The user with the most interactions as 2.5M interactions, the lowest is 1 interaction, the average is 403, and the median is only 26.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In google's doc about Google News, they recommend concatenating 2-4 keys (LSH) and doing this 10-20 times. I imagine this works well when a user has interacted with a smaller amount of items like news articles, but it is woefully low for what I am doing. When I test this number of keys for users that have 1,000+ interactions, many do not have any concatenated min has matches with another user. This is a problem because I can manually calculate cosine or jaccard similarity for some of these users and see an acceptable amount of similarity for my needs. I have found better results by not concatenating hash keys and using as many as 200.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most of my hash key groups, there are around 2M distinct hash keys for the 2.24M users. As such, there is a fairly low amount of collision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you guys have any tips for increasing the amount of clustering? I am considering using 1,000 hash keys and pairing users if they match on more than one. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="16548" LastActivityDate="2024-02-24T10:06:45.833" Title="Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions" Tags="|data-mining|bigdata|clustering|recommender-system|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10392" PostTypeId="1" CreationDate="2016-02-25T16:13:17.880" Score="3" ViewCount="800" Body="&lt;p&gt;I'm hoping you have some research or experience with determining the completeness of a data set. I'm trying to use a twitter dataset I scraped myself and want to have an indication on the completeness. &#xA;Obviously, I will miss some data but I am wondering if there is a formula or method to calculate a probability on the completeness.&lt;/p&gt;&#xA;" OwnerUserId="16552" LastActivityDate="2016-03-26T22:56:44.720" Title="Determining completeness of dataset" Tags="|dataset|statistics|research|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10393" PostTypeId="2" ParentId="10389" CreationDate="2016-02-25T17:47:01.463" Score="1" Body="&lt;p&gt;You cannot guarantee this. Some data &lt;em&gt;is&lt;/em&gt; not separable by any kernel because of duplicates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By trying too hard, you will &lt;strong&gt;cause overfitting&lt;/strong&gt;. Essentially, you force the implicit mapping to be so complex it contains a copy of your training data (which is exactly what happens if you choose a too small bandwidth with RBF).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a good generalization performance, you will have to tolerate some errors, and use e.g. soft-margin and such techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perfect separation is not something to aim for. Such a guarantee is just a guarantee of being able to overfit! Use cross-validation to reduce the risk of overfitting and find the right balance between being optimal on training data and actual performance.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-02-25T17:47:01.463" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10394" PostTypeId="1" AcceptedAnswerId="10439" CreationDate="2016-02-25T18:11:34.943" Score="-1" ViewCount="196" Body="&lt;p&gt;As I'm starting my career in analytics, I have to choose between SAS in-memory analytics and Openware and largely adopted R, which one should I choose now so that I will have good market value in short-term as well as long-term?&lt;/p&gt;&#xA;" OwnerUserId="6558" LastEditorUserId="11097" LastEditDate="2016-02-29T10:21:18.180" LastActivityDate="2016-02-29T10:21:18.180" Title="which technology should I invest into SAS In-memory analytics or R?" Tags="|r|sas|" AnswerCount="5" CommentCount="2" ClosedDate="2016-02-29T10:35:39.317" ContentLicense="CC BY-SA 3.0" />
  <row Id="10395" PostTypeId="2" ParentId="10394" CreationDate="2016-02-25T18:49:53.537" Score="1" Body="&lt;p&gt;This all depends. SAS is great and is backed by the SAS Institute, meaning if you're working for an organization that has invested in SAS, you can contact the support team for anything funky happening with the software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R is free and open source, and there are also organizations being created that are building on to and supporting R much like SAS Institute has done. The difference being that SAS has been around much longer and is much more structured than R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They both have their ups and downs, and it depends on what type of analytics you plan on doing in your career. So, to answer your questions, I would decide what you envision yourself doing. To my understanding R is good for Big Data applications and Machine Learning, SAS is great for statistical analysis (ARIMA, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SAS and R comparisons:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://support.sas.com/resources/papers/proceedings13/348-2013.pdf&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/resources/papers/proceedings13/348-2013.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.learnanalytics.in/blog/?p=9&quot; rel=&quot;nofollow&quot;&gt;http://www.learnanalytics.in/blog/?p=9&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16556" LastActivityDate="2016-02-25T18:49:53.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10396" PostTypeId="2" ParentId="10394" CreationDate="2016-02-25T19:30:55.660" Score="1" Body="&lt;p&gt;In my experience, there are a few things to consider here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What field you're going into&lt;/li&gt;&#xA;&lt;li&gt;What kinds of technology you believe you'll be working with&lt;/li&gt;&#xA;&lt;li&gt;What kinds of teams you believe you'll be working with&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Field&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a huge determiner. To my understanding, SAS is standard in finance, banking, some biostats, and other industries. R, on the other hand, is open source, free, and is receiving a lot of attention currently from Microsoft after their acquisition of Revolution Analytics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Technology&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you thinking you'll be at a large corporation or a small shop? Do you think you'll need to work with creating production ready algorithms or simply producing insights and analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason this matters is that open source technology can often times be a bit easier to convert to production ready use. The cost of software can also be a barrier at smaller companies and may dictate what you are capable of using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Collaboration with Other Teams&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you sit firmly on the business side, it's possible that everyone you work with may use SAS or may be comfortable with the outputs. In this case, you may not need to collaborate across any additional technology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you work with technology, however, it may be difficult to integrate SAS or proprietary solutions into other workflows. An example would be creating a real-time user scoring system. If you were able to program this into R or Python, you may be able to pass the code directly to a developer for implementation. This would be more difficult if a proprietary solution was involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Considerations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analytics space is evolving rapidly. On top of the above, Python is coming out as a very popular technology for use in machine learning and data mining and pairs well with Spark as well as some other large data technology. A specific example of a library here would be scikit-learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Closing Thoughts/My Experience&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm an R/Python user by experience, though I have some experience with SAS in school as well as in work. Generally, I've found the level of support with R and Python to be great - especially since they've started to rise in popularity. SAS has its uses and has definitely carved out piece of the industry for itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, however - getting a solid base in statistical theory, programming (scripting), and understanding the application and value that analysis can provide will go a long way. The syntax between these tools is, generally, not worlds apart. SAS can be a bit strange compared to R and Python, but all of these tools generally have syntax which is fairly readable and is not difficult to adapt to another tool.&lt;/p&gt;&#xA;" OwnerUserId="16558" LastActivityDate="2016-02-25T19:30:55.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10398" PostTypeId="2" ParentId="10392" CreationDate="2016-02-25T22:30:45.597" Score="2" Body="&lt;p&gt;If you are talking about exploring your data for patterns of missing data you can try using Self-Organizing Maps [&lt;a href=&quot;https://en.wikipedia.org/wiki/Self-organizing_map]&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Self-organizing_map]&lt;/a&gt;, which are a special flavor of neural networks. Here is a small &lt;a href=&quot;http://users.ics.aalto.fi/sami/thesis/node18.html#SECTION00074000000000000000&quot; rel=&quot;nofollow&quot;&gt;research paper&lt;/a&gt; explaining the concept a bit. Here is another [link][3] with historical info and links to some original papers about the topic, specifically by Kohonen &lt;em&gt;et al.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Wikipedia: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Self-organizing maps are different from other artificial neural&#xA;  networks as they apply competitive learning as opposed to&#xA;  error-correction learning (such as backpropagation with gradient&#xA;  descent), and in the sense that they use a neighborhood function to&#xA;  preserve the topological properties of the input space. This makes&#xA;  SOMs useful for visualizing low-dimensional views of high-dimensional&#xA;  data...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Essentially this method will allow you to look for patterns in your data. From there you can determine what level of completeness you are dealing with in your data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically for your example for Twitter data, I would imagine there are many fields in the JSON data with missing values. Maybe some users choose not to fill in their gender or age etc. Visualizing your data as well calculating summary statistics will help you to paint an overall picture of your data. And when you have high dimensional data using tools to visualize it in a lower dimensional space can always be handy. Hope this helps you!&lt;/p&gt;&#xA;" OwnerUserId="14654" LastActivityDate="2016-02-25T22:30:45.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10399" PostTypeId="1" AcceptedAnswerId="10438" CreationDate="2016-02-25T23:16:02.823" Score="4" ViewCount="3757" Body="&lt;p&gt;Question: A image has multiple labels. Given a set of image with labels, how to adapt the softmax layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Encode multiple labels to 0-1 variables. Use logistics regression as the output layer.&lt;/li&gt;&#xA;&lt;li&gt;Choose the top X labels in softmax output. But I do not know &#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;how to determine the threshold.&lt;/li&gt;&#xA;&lt;li&gt;how to modify the code in a pretrained net. Let's say &lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;      &quot;op&quot;: &quot;null&quot;,&#xA;      &quot;param&quot;: {},&#xA;      &quot;name&quot;: &quot;softmax_label&quot;,&#xA;      &quot;inputs&quot;: [],&#xA;      &quot;backward_source_id&quot;: -1&#xA;    },&#xA;    {&#xA;      &quot;op&quot;: &quot;Softmax&quot;,&#xA;      &quot;param&quot;: {&quot;grad_scale&quot;: &quot;1&quot;},&#xA;      &quot;name&quot;: &quot;softmax&quot;,&#xA;      &quot;inputs&quot;: [[510, 0], [511, 0]],&#xA;      &quot;backward_source_id&quot;: -1&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any recommendations on tutorials are appreciated. I have some theory background from the book ESL.&lt;/p&gt;&#xA;" OwnerUserId="16310" LastEditorUserId="21" LastEditDate="2017-05-19T16:10:50.067" LastActivityDate="2018-02-21T17:04:05.567" Title="How to adapt the softmax layer for multiple labels?" Tags="|neural-network|image-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10400" PostTypeId="2" ParentId="10399" CreationDate="2016-02-25T23:56:11.483" Score="5" Body="&lt;p&gt;I don't know what's going on in your code, but you seem to be close: to get multiple labels, simply replace the softmax output layer with a logistic layer (or something else that maps a real number to a probability), then optimize the cross entropy. That way you will have a probability associated with each label such that their sum across labels no longer need to add to unity.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-25T23:56:11.483" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10401" PostTypeId="2" ParentId="10333" CreationDate="2016-02-26T01:07:23.793" Score="1" Body="&lt;p&gt;It depends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All most all the professors (in my univ) would show students examples in &lt;code&gt;e1071&lt;/code&gt;. The somewhat weird name &lt;code&gt;e1071&lt;/code&gt; is from a course name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you have hand-on experience on industry or kaggle, you would find the preprocessing functions of &lt;code&gt;caret&lt;/code&gt; is quiet useful. Specify what you want by several arguments. You get normalization, fill NAs, cross validation etc. Extremely easy. No headache at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If time permits, play both.&lt;/p&gt;&#xA;" OwnerUserId="16310" LastActivityDate="2016-02-26T01:07:23.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10402" PostTypeId="2" ParentId="10330" CreationDate="2016-02-26T06:00:07.287" Score="1" Body="&lt;p&gt;The corpus quality can be determined using the zipf law ,vocabulary growth curve and the pareto distribution. An interesting thesis which explains this can be found at   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ilk.uvt.nl/downloads/pub/papers/hait/camp2008.pdf&quot; rel=&quot;nofollow&quot;&gt;http://ilk.uvt.nl/downloads/pub/papers/hait/camp2008.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16427" LastActivityDate="2016-02-26T06:00:07.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10404" PostTypeId="1" CreationDate="2016-02-26T06:04:11.280" Score="11" ViewCount="9021" Body="&lt;p&gt;The wiki page for bootstrapping says that you use it in the case where the underlying distribution is unknown. Why is bootstrapping, or sampling with replacement, better than just calculating the variance and other properties from the data directly?&lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2018-03-06T12:48:01.363" Title="Why use bootstrapping?" Tags="|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10405" PostTypeId="2" ParentId="10404" CreationDate="2016-02-26T07:34:23.683" Score="11" Body="&lt;p&gt;In general when you are evaluating statistics there are two main approaches: either you use some distributional assumptions (a generating model for your data) or not. Whenever you can you would probably go with a statistical model. But often the distributional assumptions can be wrong or they are not available to you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing inferences about the mean of the population, than you can rely on central limit theorem and build a statistic for that. The same for variance. However what about estimating something which does not rely on general assumptions? Bootstrap works for any kind of statistic, this is where it lies its power. It's simple, and does require only minimal assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there is another problem which appears in practice. Estimating mean rely on central limit theorem. It is true that your set of assumptions required only independent and identical distributed data. But it's something more. CLT works on large samples, it works in the limit. For small samples it can go wrong more often. And what about if your distribution is skewed. While it's true that CLT relies only on i.i.d. the needed sample size for estimating a mean on a skewed distribution is in general greater that for a symmetric distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found bootstrapping very useful in two main situations: when the sample is fairly small (but not tiny) and when the distribution is not clean (suppose it's a mixture of two distributions). And is very useful because you can build not only estimates of the population parameters, but also estimations for confidence intervals of p-values. And because it's easy to apply I also found sometimes useful even when you are assuming some model. Doing in parallel can help you to see if there is something really spooky in your sample or assumptions if inferences produces too different results, it makes you think or re-evaluate your setup.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-02-26T07:34:23.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10406" PostTypeId="1" CreationDate="2016-02-26T13:27:48.553" Score="5" ViewCount="4519" Body="&lt;p&gt;Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.&lt;/p&gt;&#xA;" OwnerUserId="16570" LastActivityDate="2016-02-29T21:09:03.920" Title="What are the methods to ensure that the population split for A/B test is random?" Tags="|data-mining|statistics|data|experiments|ab-test|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10407" PostTypeId="1" CreationDate="2016-02-26T13:54:54.900" Score="4" ViewCount="452" Body="&lt;p&gt;Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is expected to be low. How to choose a good decision boundary for such cases. &lt;/p&gt;&#xA;" OwnerUserId="16570" LastActivityDate="2016-02-27T00:30:13.717" Title="How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?" Tags="|data-mining|classification|class-imbalance|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10408" PostTypeId="2" ParentId="6694" CreationDate="2016-02-26T14:51:42.707" Score="11" Body="&lt;p&gt;Here you have some good references on Reinforcement Learning:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Classic&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;The draft for the second edition is available for free: &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning: An Introduction&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Russell/Norvig Chapter 21:&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Russell SJ, Norvig P, Davis E. Artificial intelligence: a modern approach. Upper Saddle River, NJ: Prentice Hall; 2010.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;More technical&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Szepesvári C. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 2010;4(1):1–103.&lt;/em&gt;  &lt;a href=&quot;http://www.ualberta.ca/%7Eszepesva/RLBook.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Algorithms of Reinforcement Learning | Csaba Szepesvári&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Bertsekas DP. Dynamic Programming and Optimal Control. 3rd edition. Belmont, Mass.: Athena Scientific; 2007. 1270 p.&lt;/em&gt;&#xA;Chapter 6, vol 2 is available for free: &lt;a href=&quot;http://web.mit.edu/dimitrib/www/dpchapter.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Programming and Optimal Control&#xA;3rd Edition, Volume II | Massachusetts Institute of Technology&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;For more recent developments&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Wiering M, van Otterlo M, editors. Reinforcement Learning. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012&lt;/em&gt; Available from: &lt;a href=&quot;http://link.springer.com/10.1007/978-3-642-27645-3&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning | SpringerLink&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Kochenderfer MJ, Amato C, Chowdhary G, How JP, Reynolds HJD, Thornton JR, et al. Decision Making Under Uncertainty: Theory and Application. 1 edition. Cambridge, Massachusetts: The MIT Press; 2015. 352 p.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Multi-agent reinforcement learning&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Buşoniu L, Babuška R, Schutter BD. Multi-agent Reinforcement Learning: An Overview. In: Srinivasan D, Jain LC, editors. Innovations in Multi-Agent Systems and Applications - 1 . Springer Berlin Heidelberg; 2010 p. 183–221.&lt;/em&gt; Available from: &lt;a href=&quot;http://link.springer.com/chapter/10.1007/978-3-642-14435-6_7&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-agent Reinforcement Learning: An Overview&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Schwartz HM. Multi-agent machine learning : a reinforcement approach. Hoboken, New Jersey: Wiley; 2014.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Videos / Courses&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I would also suggest David Silver course in YouTube: &lt;a href=&quot;https://www.youtube.com/watch?v=2pWv7GOvuf0&amp;amp;list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB&quot; rel=&quot;nofollow noreferrer&quot;&gt;RL Course by David Silver&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="85045" LastEditDate="2021-02-13T08:11:34.423" LastActivityDate="2021-02-13T08:11:34.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10409" PostTypeId="1" CreationDate="2016-02-26T15:26:10.943" Score="4" ViewCount="64" Body="&lt;p&gt;I have a dataset with physiological measures of subjects along time. I would like to create (or select) a mean prototype example in order to be able to identify in new examples how far are they from the mean prototype. A second issue will be to select a threshold to determine what is considered near or far.&#xA;Each example has 20 numeric features and I have around 300 examples per subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First ideas (disregarding outliers):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To iterate through all the examples of a subject and find the one who has the minimum mean distance to all the other examples. This will select a specific example from the dataset.&lt;/li&gt;&#xA;&lt;li&gt;To use an evolutionary algorithm to find a prototypical example which has the lowest mean distance to all the other examples. This will create a new example that can be used as prototype.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now I would like to determine when a new example is close, far or very far from the prototype (mean). A possible approach is to set two thresholds distance to determine to which class or case corresponds the new example (close, far or very far). How could I determine these thresholds? Possibly using number of standard deviations? What other approaches can be followed to perform all this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume the distance metric is already selected.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2016-02-27T10:19:05.907" Title="How to build a mean prototype from data" Tags="|clustering|distance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10410" PostTypeId="2" ParentId="10406" CreationDate="2016-02-26T15:58:37.930" Score="2" Body="&lt;p&gt;Just split them 50/50 (or another percentage, depending on what you want). Once you want to test a specific metric, make sure that if you can only generate that for a portion of the A group because of some specific property (ie you need information C for this metric) to compare it only to the people in group B that also have information C to avoid bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: I have 200,000 customers. I split them 50/50. I'm testing my new recommendation method against a random advertisement. However my recommender only works if they have searched in the last month. After getting back the results, I will have to check that against the control group filtered to have only people that have searched in the past month. People that have searched in the past month are much more likely to have a stronger kind of connection to you, so if you would check against the whole group you would overestimate the performance of your recommender.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could already split up the groups knowing whether they have information C or not, but you might want to test different things so splitting before hand can give you extra info, at the small cost that your groups might be slightly unequal, but for a big sample size this is not an issue at all.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-02-26T15:58:37.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10411" PostTypeId="2" ParentId="10406" CreationDate="2016-02-26T16:10:44.310" Score="2" Body="&lt;p&gt;In some cases it is useful to do stratified sampling. For example, imagine you have 4 different groups in your population each with differing n (e.g., A = 10k, B = 20k, C = 30k D = 40k). In creating your train/test you may want to ensure that each subset maintains the same proportions of subgroup. This can ensure better equivalence of train/test than pure random assignment (which could result in lopsided train/test).&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-02-26T16:10:44.310" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10412" PostTypeId="1" AcceptedAnswerId="10415" CreationDate="2016-02-26T18:28:57.417" Score="0" ViewCount="166" Body="&lt;p&gt;I'm new to machine learning so I apologize if this question is silly. I'm using Stanford NER's english 4class classifier with good results. However, since my dataset is mostly focused on organizations, I think the results could be improved if I could boost the probability for an entity to be an organization to the detriment of other classes.(Ex: I would prefer &quot;Carl Zeis&quot; to be identified as an organization rather than a person). Is my supposition correct? If so, can it be achieved in an easier way than retraining the model?&lt;/p&gt;&#xA;" OwnerUserId="16294" LastActivityDate="2016-02-26T21:45:25.570" Title="Stanford NER - Increase probability for a certain class" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10413" PostTypeId="2" ParentId="10407" CreationDate="2016-02-26T19:00:49.500" Score="2" Body="&lt;p&gt;You don't necessarily have to train any differently if you are happy with the result. If you aren't, undersample the majority class and/or adjust the class weights in your loss function. The thing you really have to be more careful about is the evaluation of the result because a naive metric like the accuracy would give you false hope since the imbalance allows you to achieve a high accuracy by simply always predicting the majority class or randomly predicting each class according to its probability of occurrence. It is this baseline against which you have to judge your success; like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Coefficient_of_determination&quot; rel=&quot;nofollow&quot;&gt;coefficient of determination&lt;/a&gt; in regression, where the baseline is the sample mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use a graphical approach (ROC or precision/recall curves), or report a size-weighted average of the class positive rate instead of the accuracy. Here are some other ideas: &lt;a href=&quot;http://ipl.ece.ubc.ca/bci_files/C8.pdf&quot; rel=&quot;nofollow&quot;&gt;Comparison of Evaluation Metrics in Classification Applications with Imbalanced Datasets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-02-26T21:34:52.667" LastActivityDate="2016-02-26T21:34:52.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10414" PostTypeId="2" ParentId="10409" CreationDate="2016-02-26T19:33:43.400" Score="0" Body="&lt;p&gt;For your prototype: for each time segment t1-t300, for each feature f1-f20, compute the mean feature value across subjects s1-sn. This is your prototype subject about which you can compute a confindence interval set to your target threshold (e.g, 95%) or set a threshold to +/- Nsd depending upon your definition of an outlier. &lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-02-26T19:33:43.400" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10415" PostTypeId="2" ParentId="10412" CreationDate="2016-02-26T21:45:25.570" Score="0" Body="&lt;p&gt;Your data set will influence the labeling results. If it is focused on organizations, the NER should favor them simply by virtue of the data it's fed. So you might need to do anything. But if you do observe undesirable behavior in the resulting NER, you can adjust the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's the same with any machine learning algorithm. A humorous demonstration was provided by Google's Deep Dream: &lt;a href=&quot;http://rhizome.org/editorial/2015/jul/10/deep-dream-doggy-monster/?ref=tags_dogs_post_readbtn&quot; rel=&quot;nofollow&quot;&gt;it kept seeing dogs every where&lt;/a&gt;. Why? Because the data set they used for training had an abundance of dogs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(And Carl Zeis should be labeled as a person. The company is Carl Zeiss.)&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-26T21:45:25.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10417" PostTypeId="2" ParentId="9785" CreationDate="2016-02-27T00:29:41.117" Score="13" Body="&lt;p&gt;Word2vec works in two models CBOW and skip-gram. Let's take CBOW model, as your question goes in the same way that predict the target word, given the surrounding words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fundamentally, the model develops input and output weight matrices, which depends upon the input context words and output target word with the help of a hidden layer. Thus back-propagation is used to update the weights when the error difference between predicted output vector and the current output matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically speaking, predicting the target word from given context words is used as an equation to obtain the optimal weight matrix for the given data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the second part, it seems a bit complex than just a linear sum.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Obtain all the word vectors of context words&lt;/li&gt;&#xA;&lt;li&gt;Average them to find out the hidden layer vector &lt;code&gt;h&lt;/code&gt; of size &lt;code&gt;Nx1&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Obtain the output matrix &lt;code&gt;syn1&lt;/code&gt;(&lt;code&gt;word2vec.c&lt;/code&gt; or &lt;code&gt;gensim&lt;/code&gt;) which is of size &lt;code&gt;VxN&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Multiply &lt;code&gt;syn1&lt;/code&gt; by &lt;code&gt;h&lt;/code&gt;, the resulting vector will be &lt;code&gt;z&lt;/code&gt; with size &lt;code&gt;Vx1&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Compute the probability vector &lt;code&gt;y = softmax(z)&lt;/code&gt; with size &lt;code&gt;Vx1&lt;/code&gt;, where the highest probability denotes the one-hot representation of the target word in vocabulary.&#xA;&lt;code&gt;V&lt;/code&gt; denotes size of vocabulary and &lt;code&gt;N&lt;/code&gt; denotes size of embedding vector.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Source : &lt;a href=&quot;http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Long short term memory models are currently doing a great work in predicting the next words. &lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;noreferrer&quot;&gt;seq2seq&lt;/a&gt; models are explained in &lt;a href=&quot;https://www.tensorflow.org/tutorials/seq2seq&quot; rel=&quot;noreferrer&quot;&gt;tensorflow tutorial&lt;/a&gt;. There is also a &lt;a href=&quot;https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/&quot; rel=&quot;noreferrer&quot;&gt;blog post&lt;/a&gt; about text generation.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2017-08-24T12:50:14.460" LastActivityDate="2017-08-24T12:50:14.460" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10418" PostTypeId="2" ParentId="10407" CreationDate="2016-02-27T00:30:13.717" Score="0" Body="&lt;p&gt;Check whether the classifier offers class_weight='balanced' or 'auto' option . I had better results as when down or up-sampled. Best classifier for me turned out to be SGD from sklearn. Auto-sklearn shows even better results.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-27T00:30:13.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10419" PostTypeId="2" ParentId="10394" CreationDate="2016-02-27T00:41:11.417" Score="3" Body="&lt;p&gt;Go for R. SAS is a monster and is not fun. Having said that your market value will be determined more by what you can and want to do and the general education level than just by this or that technology.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-02-27T00:41:11.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10420" PostTypeId="2" ParentId="10322" CreationDate="2016-02-27T03:16:34.320" Score="1" Body="&lt;p&gt;This is my answer just using matplotlib and numpy.     &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My code seem to be very lengthy comparing with the accepted answer.&#xA;  Hence, if someone can help me to improve it, that will be great!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Code here: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Reading the data&#xA;year = np.arange(1994,2015,1)&#xA;type = [&quot;Recalled&quot;, &quot;Recovered&quot;]&#xA;value_1 = [11.472, 11.81, 10.632, 13.857, 13.861, 13.375, 11.278, 12.827, 12.687, 10.859, np.nan, 11.782, 12.089,\&#xA;       14.351, 14.921, 11.759, 12.987, 13.262, 9.98, 10.626, 12.199]&#xA;value_2 = [10.207, 10.326, 10.094, 12.944, 12.588, 11.951, np.nan, np.nan, np.nan, np.nan, np.nan, 11.047, 10.194,\&#xA;       13.401, 13.886, 10.815, 11.482, 10.73, 9.52, 9.591, 10.27] &#xA;&#xA;&#xA;&#xA;## Changing data into ndarray format&#xA;dpoints = np.array([type[0], year[0], value_1[0]])&#xA;conditions =  np.unique(dpoints[:,0])&#xA;categories =  np.unique(dpoints[:,1]).tolist()&#xA;&#xA;for i in range(1,len(year),1):&#xA;    dpoints = np.vstack([dpoints,np.array([type[0],year[i],value_1[i]])])&#xA;for i in range(0,len(year),1):&#xA;    dpoints = np.vstack([dpoints,np.array([type[1],year[i],value_2[i]])])&#xA;&#xA;# Plot it!&#xA;fig = plt.figure(figsize=(16,5))&#xA;ax = plt.subplot()&#xA;#the space between each set of bars&#xA;space = 0.2&#xA;n = len(conditions)&#xA;width = (1 - space) / (len(conditions))&#xA;&#xA;# Create a set of bars at each position&#xA;for i,cond in enumerate(conditions):&#xA;    indeces = range(1, len(categories)+1)&#xA;    vals = dpoints[dpoints[:,0] == cond][:,2].astype(np.float)&#xA;    pos = [j - (1 - space) / 2. + i * width for j in indeces]&#xA;&#xA;    ax.bar(pos, vals, width=width, label =cond,lw = 0,color=  [&quot;blue&quot;,&quot;r&quot;][i],alpha = 0.6)&#xA;&#xA;&#xA;    # Set the x-axis tick labels to be equal to the categories&#xA;    ax.set_xticks(indeces)&#xA;    ax.set_xticklabels(categories)&#xA;    ax.set_xlim(0,22)&#xA;    plt.setp(plt.xticks()[1], rotation=0,fontsize = 12)&#xA;    ax.set_ylabel('Variables',fontsize =15)&#xA;    ax.set_xlabel(&quot;Year&quot;,fontsize =15)&#xA;&#xA;&#xA;    # Add a legend&#xA;    handles, labels = ax.get_legend_handles_labels()&#xA;    ax.legend(handles[::-1], labels[::-1], loc='upper left',frameon =False,fontsize =12)    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Figure here&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/GSQMK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GSQMK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15663" LastEditorUserId="15663" LastEditDate="2016-03-23T09:05:24.667" LastActivityDate="2016-03-23T09:05:24.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10422" PostTypeId="2" ParentId="10310" CreationDate="2016-02-27T06:52:09.120" Score="1" Body="&lt;p&gt;I would say that it strongly depends on the nature of your labels. Why they overlap? I'm use to work with fish and we are use to get several false-negative data. It's simple, there are not enocuh fish to occupy every suitable place. In that case we usually favour presence data. However, the input variables/features could be not good for an adequate discrimination and that's why they appear overlapped ... (We would need additional info.). You can favour one or more labels penalizing others however that's always at the expense of accuracy. In that case SVMs with case weights may work. Likewise others will if they allow the inclusion of case weights. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="16560" LastActivityDate="2016-02-27T06:52:09.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10424" PostTypeId="1" CreationDate="2016-02-27T10:01:28.513" Score="5" ViewCount="6228" Body="&lt;p&gt;I have few key words say - RAM, Speaker, Brand, Display etc. I have made a dictionary of all possible values these keys can have. Like - RAM can have 12, 12gb, 12GB; Display can have 12inch, 12&quot;, 12 inch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am given a title say - &quot;Apple iPhone 5s (Space Grey, 16GB)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to extract tokens from these titles and map to the dictionary values and assign appropriate keyword to each token.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like for this - Brand - Apple, Color - Grey, Storage - 16GB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should i tokenize these sentences? Doing it just by space will not be enough- like in this title - &quot;Samsung Guru Music 2 SM-B310E (White)&quot; 2 will be one token and can map to RAM,Display anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will any NLP library help with this? I am using python to code and new to NLP.&lt;/p&gt;&#xA;" OwnerUserId="13046" LastActivityDate="2020-08-03T17:06:39.277" Title="Tokenize sentence based on a dictionary" Tags="|python|nlp|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10425" PostTypeId="2" ParentId="10409" CreationDate="2016-02-27T10:19:05.907" Score="3" Body="&lt;p&gt;The object with the lowest mean distance (= the object with the lowest distance sum) is known as the &lt;strong&gt;medoid&lt;/strong&gt; and the basis of k-medoids algorithms such as PAM. Because you must not use k-means with arbitrary distances.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-02-27T10:19:05.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10426" PostTypeId="2" ParentId="10424" CreationDate="2016-02-27T11:29:31.560" Score="1" Body="&lt;p&gt;Look at the &lt;a href=&quot;http://www.nltk.org/_modules/nltk/tokenize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK&lt;/a&gt; library for Python, there are functions to facilitate tokenizing sentences.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="98307" LastEditDate="2020-08-03T16:41:58.487" LastActivityDate="2020-08-03T16:41:58.487" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10428" PostTypeId="2" ParentId="10424" CreationDate="2016-02-27T12:50:57.053" Score="0" Body="&lt;p&gt;This seems like it might a good fit to regular expressions but lets first talk about how to restrict the search space so you don't have to use regex. Before you even start though you need to make sure your requirements are possible - it looks like you want to match 16 to storage and 12 to screen size without requiring GB or inches at the end, which means you need to find some way to distinguish those numbers. Here's a few ideas of how to make these restrictions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Storage&lt;/strong&gt;: this is very likely to be powers of 2, (8, 16, 32, 64...) so you only really need to check if the string has a power of two in it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[storage in &quot;iPhone 5s 16 GB&quot; for storage in [&quot;8&quot;, &quot;16&quot;, &quot;32&quot;, &quot;64&quot;]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will return you &lt;code&gt;[False, True, False, False]&lt;/code&gt;, telling you that there's a 16 in there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Display&lt;/strong&gt;: again, these are likely to be quite restricted - as you say, 12, 13, etc. You could probably get away with something similar to my storage trick above and ignore trying to find inches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Brand, Model&lt;/strong&gt;: These should be really simple - again, make a list of the brands you're searching for, and then do the list comprehension above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this doesn't work, you can try regexes. These are more complex and usually more trouble than they're worth - they're hard to read, maintain and I've often got bitten by them. However:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regex &lt;code&gt;(?i)[0-9]+(\s|)GB&lt;/code&gt; in python will do a case insensitive (&lt;code&gt;(?i)&lt;/code&gt;) search for strings containing one of more numbers 0-9 (&lt;code&gt;[0-9]+&lt;/code&gt;), zero or more white space (&lt;code&gt;\s*&lt;/code&gt;) and then the letters GB (&lt;code&gt;GB&lt;/code&gt;). GB will be matched in upper or lower case because of the &lt;code&gt;(?i)&lt;/code&gt; option are the start. Calling&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;re.search('(?i)[0-9]+\s*GB', &quot;Apple iPhone 5s (Space Grey, 16GB)&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;will return a match object giving you the matched string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find regexs quite complicated so general use a &lt;a href=&quot;http://regexr.com&quot; rel=&quot;nofollow&quot;&gt;helpful website&lt;/a&gt; to at least try catch bugs before I code them.&lt;/p&gt;&#xA;" OwnerUserId="16601" LastActivityDate="2016-02-27T12:50:57.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10429" PostTypeId="1" CreationDate="2016-02-27T15:34:34.027" Score="5" ViewCount="7030" Body="&lt;p&gt;till now I was under the impression that machine learning algorithms (gbm, random forest, xgboost etc) can handle bad features (variable) present in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In one of my problems, there are around 150 features and with xgboost I am getting a logloss of around 1 if I use all features. But if I remove around 10 bad features (found using some technique) I am observing a logloss of .45. That is huge improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, can bad features really make such big differences?&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2016-12-06T20:51:47.023" Title="Can xgboost (or any other algorithm) give bad results with some bad features?" Tags="|machine-learning|random-forest|gbm|xgboost|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10430" PostTypeId="1" CreationDate="2016-02-27T15:59:49.043" Score="4" ViewCount="169" Body="&lt;p&gt;I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't know which keywords to search for to find existing literature(/implementations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at a time, I might have already computed function values at 18 points, with 15 still being computed, and I want to start evaluating the function another point. Without the running jobs, I could make a model and find which next point might provide the most information. But now, I need to somehow tell the model that there are also 15 points for which I don't have values, but near which I don't want to evaluate the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I'm looking for something with minimal assumptions and no gradient information. But I'll be happy if I just know the keyword(s) to search for. (I can come up with several hacks to that kind of work, but I wonder if there is any 'real' solution).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;For illustration, say every column is a function evaluation point and every row is a timestep in the below ascii-figure. An &lt;code&gt;X&lt;/code&gt; means the evaluation is still being computed and a &lt;code&gt;*&lt;/code&gt; in the bottom row means it's done. A new evaluation starts as soon as an old one terminates, with others still busy.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;XXXXXXXXXXXXXXXX&#xA;XXXXXXXXXXXXXXXX&#xA; XXXXXXXXXXX XXXXX&#xA; XXXXXXXXXXX XXXXX&#xA; XXXXXXXXXXX XXXXX&#xA; XXXXXXXXXXX X  XXX X&#xA; XXXX XXXXXX X  XXX XX&#xA; XXXX XXXXXX X  XXX  XX&#xA; XXXX XXXXXX X  XXX  XX&#xA; XXXX XXXXXX X  XXX  XX&#xA; XXXX XXXXXX X  XXX  XX&#xA; XXXX X XXXX X  XXX  XXX&#xA; XXXX X XXXX X  X X  XXXX&#xA; XXXX X XXXX X  X X  XXXX&#xA;  XXX X XXXX X  X X  XXXXX&#xA;  XXX X XXXX X  X X  XXXXX&#xA;  XXX X XXXX X  X X  XXXXX&#xA;  XXX X XXXX X  X X  XXXXX&#xA;  XXX X XX X X  X X  XXXXXX&#xA;  XXX   XX X X  X X  XXXXXXX&#xA;  XXX   XX X X  X X  XXXXXXX&#xA;  XXX   XX X X  X X  XXXXXXX&#xA;  XXX   XX   X  X    X XXXXXXXX&#xA;  XXX   XX   X  X    X XXXXXXXX&#xA;  XXX   XX   X  X    X XXXXXX XX&#xA;  XXX   XX   X  X    X XXXXXX XX&#xA;  XXX   XX   X       X XXXXXX XXX&#xA;  XX    XX   X       X XXXXXX XXXX&#xA;**  ****  *** ******* *      *&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Each time an evaluation completes, how to choose the next point to evaluate, given several complete and several incomplete evaluations?&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="10907" LastActivityDate="2019-12-12T01:01:30.210" Title="Parallel active optimization" Tags="|optimization|sampling|parallel|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10431" PostTypeId="2" ParentId="10430" CreationDate="2016-02-27T18:42:38.753" Score="0" Body="&lt;p&gt;I would use a parallelized Gaussian process regression model. The next points to query in this framework are given by the posterior extrema or those of &lt;a href=&quot;http://people.orie.cornell.edu/pfrazier/Presentations/2011.11.INFORMS.Tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;maximum expected improvement&lt;/a&gt;. &lt;a href=&quot;https://github.com/arikcj/pgpr&quot; rel=&quot;nofollow&quot;&gt;Here's an implementation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-02-27T18:42:38.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10432" PostTypeId="1" AcceptedAnswerId="10442" CreationDate="2016-02-27T19:14:08.533" Score="6" ViewCount="155" Body="&lt;p&gt;For nonlinear data, when we are using Support Vector Machines, we can use kernels such as Gaussian RBF, Polynomial, etc to achieve linearity in a different (potentially unknown to us) feature space and &lt;em&gt;the algorithm learns a maximal separating hyperplane in that feature space&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do we create heatmaps such as the one seen in the image below to show this max. separating hyperplane in our original space and how should it be interpreted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tZYnK.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tZYnK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11044" LastActivityDate="2016-02-29T08:39:00.030" Title="How are Hyperplane Heatmaps created and how should they be interpreted?" Tags="|machine-learning|python|svm|supervised-learning|discriminant-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10433" PostTypeId="1" AcceptedAnswerId="10434" CreationDate="2016-02-28T07:05:01.517" Score="0" ViewCount="100" Body="&lt;p&gt;I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Xij : number of times word i occurs in context of word j in a window of size w.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now is this context limited to sentence i.e do we context word of a main word in a sentence only or across the whole document?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the corpus :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;human interface computer&#xA;survey user computer system response time&#xA;eps user interface system&#xA;system human system eps&#xA;user response time&#xA;trees&#xA;graph trees&#xA;graph minors trees&#xA;graph minors survey&#xA;I like graph and stuff&#xA;I like trees and stuff&#xA;Sometimes I build a graph&#xA;Sometimes I build trees&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now for word graph and context window 10 do we look for only 4th last line or do we search for words across line also considering whole doc as one line?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-02-28T11:21:19.167" Title="Problem in constructing co-occurence matrix" Tags="|text-mining|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10434" PostTypeId="2" ParentId="10433" CreationDate="2016-02-28T11:21:19.167" Score="2" Body="&lt;p&gt;Whether to window only over sentences or over the whole corpus depends on the context of your problem and on the structure of your corpus. In this corpus the order of your sentences does not matter, if they were switched around it would still be the same corpus. This means that you should stick to only the lines and not see it as one line. If your corpus would be a book for example, spilling over sentences makes a lot more sense. If you are unsure, try both, it is just a hyperparameter of your model.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-02-28T11:21:19.167" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10435" PostTypeId="2" ParentId="10429" CreationDate="2016-02-28T11:25:05.730" Score="7" Body="&lt;p&gt;Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if you have 400 data points 150 features is way too much. How can you learn something when there is so little to learn from? Overfitting is a real issue for almost any algorithm, and noisy features make it more easy to learn things that are not actually there.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-02-28T11:25:05.730" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10437" PostTypeId="1" CreationDate="2016-02-28T17:25:28.147" Score="3" ViewCount="2356" Body="&lt;p&gt;I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of iterations set to 100 and say that Spark mllib can run very fast for big data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This the code I am having problems with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    def train(data: RDD[TrainFeature], scalerModel: StandardScalerModel): LinearRegressionModel = {&#xA;    val labeledData = data map { trainFeature =&amp;gt;&#xA;      LabeledPoint(trainFeature.relevance.value, toVector(trainFeature.feature, scalerModel))&#xA;    }&#xA;    labeledData.cache()&#xA;&#xA;    val algorithm = new LinearRegressionWithSGD()&#xA;    algorithm.optimizer&#xA;      .setNumIterations(10)&#xA;      .setRegParam(0.01)&#xA;      .setStepSize(0.1)&#xA;&#xA;    algorithm run labeledData&#xA;  }&#xA;&#xA;private def toVector(feature: Feature, scalerModel: StandardScalerModel): Vector = scalerModel transform toVector(feature)&#xA;&#xA;private def toVector(feature: Feature): Vector = Vectors dense feature.coordinates.toArray&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have scaled the data first and then run the algorithm to train the model. Even when I use 10 iterations it takes around 10 minutes to train model for 70,000 entries with feature vector of size 2. And the results I get are not good at all. I start getting decent results after numberOfIterations = 1000, but that would take ages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it normal for linear regression with SGD to be this slow for 70,000 vectors of size 2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My JVM min and initial memory is set to 4g.&#xA;I have tried setting the following as well (devastated try): &lt;code&gt;System.setProperty(&quot;spark.executor.memory&quot;, &quot;3g&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running this locally and since normal LinearRegression written in MatLab would finish the job very fast, I am wondering what am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: When I look at the spark UI in jobs section, I see that it is creating way too many jobs for gradient descent. Is there a way that I could tell Spark to create very little jobs - i.e. don't split data, run everything in a single thread? Maybe that can help me debug the problem further.&lt;/p&gt;&#xA;" OwnerUserId="16612" LastEditorUserId="16612" LastEditDate="2016-03-01T11:33:31.160" LastActivityDate="2023-12-22T12:06:35.767" Title="Why is Spark's LinearRegressionWithSGD very slow locally?" Tags="|predictive-modeling|regression|apache-spark|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10438" PostTypeId="2" ParentId="10399" CreationDate="2016-02-28T19:39:41.407" Score="0" Body="&lt;p&gt;I have read related tutorials over the past several days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two possible solutions&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;transfer the problem into several multi-classication problem.&lt;/li&gt;&#xA;&lt;li&gt;use algorithms that can handle multi-label problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On the software side:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;extract network parameters except the last layer.&lt;/li&gt;&#xA;&lt;li&gt;use a SVM(or other methods) to handle the last layer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;stanford computer vision online course&lt;/li&gt;&#xA;&lt;li&gt;caffe tutorial&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16310" LastActivityDate="2016-02-28T19:39:41.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10439" PostTypeId="2" ParentId="10394" CreationDate="2016-02-28T21:51:28.710" Score="1" Body="&lt;p&gt;You'll need both or even more for your &quot;career in analytics&quot;.  Start with SAS, SAS studio is intuitive and easy, once you are comfortable with the &quot;stats&quot; and looking for next level than is coming R in place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SAS also have free courses for intro in stat and their platform.&#xA;&lt;a href=&quot;http://support.sas.com/training/tutorial/index.html&quot; rel=&quot;nofollow&quot;&gt;SAS free courses&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For long term believe me that is not about the tools/platform. You can check the free book listed bellow and most important modeling and prediction techniques listed in the book. Theoretical and practical understanding of many important methods is essential and after that you will be able to compute in any language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow&quot;&gt;An Introduction to Statistical Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2016-02-28T21:51:28.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10440" PostTypeId="2" ParentId="10325" CreationDate="2016-02-29T00:53:07.890" Score="1" Body="&lt;p&gt;As far as I know, Orange supports advanced annotation with a &lt;a href=&quot;http://orange-visual-programming.readthedocs.org/loading-your-data/index.html&quot; rel=&quot;nofollow&quot;&gt;three-line header&lt;/a&gt; in both CSV and Excel files. So if you have a discrete attribute that gets marked as string, you should annote it as &lt;code&gt;discrete&lt;/code&gt; on the second header line, or prefix the attribute name on the first line with &lt;code&gt;D#&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When no advanced annotation is provided, it seems Orange's heuristic treats attributes containing general strings (as opposed to numbers) as discrete only if they contain fewer than approximately $\sqrt{|data|}$ unique values.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-02-29T00:53:07.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10441" PostTypeId="2" ParentId="10394" CreationDate="2016-02-29T04:25:23.520" Score="1" Body="&lt;p&gt;@ImperativelyAblative has many good points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding to that, many companies, especially banks, that rooted in SAS have started to experiment with R. This is because of the cost saving and maturing enterprise support, think of RStudio Server and Cloudera. SAS is great, but it comes with a premium price tag (some may argue it's even overpriced). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a management consultant working at a top tier firm with specialty in data science. We serve most of the global companies and work with their executives and operating teams. Moving to open source tools, such as R, has gained the blessing of many top executives and building tractions among core analytics teams (Marketing, Modelling, etc.). The point being - the market is moving towards open source tools&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other consideration, in my opinion, is the role you aspire to play in an organization. Here are some archetypes based on my experience in working with the analytics teams: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;General Business Analyst&lt;/strong&gt;: R, Interactive visualization tools (Tableau / QlikView), some SQL, and Excel / Powerpoint (of course) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Modeller&lt;/strong&gt;: SAS is must, R (my clients are all trying to pick this up), deep in SQL / HQL (querying on Hadoop stack), and strong domain knowledge (risk model, pricing, operation optimization, etc.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Application Developer&lt;/strong&gt; (people who put things into production): SAS is must, Python, and automation language &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, this is not an exhaustive list, but hope it provides some industry trend for your reference. There is always the superstar in any organization who knows all these stuff, could be you :) &lt;/p&gt;&#xA;" OwnerUserId="16614" LastActivityDate="2016-02-29T04:25:23.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10442" PostTypeId="2" ParentId="10432" CreationDate="2016-02-29T08:39:00.030" Score="6" Body="&lt;p&gt;I think I can answer that, since I implement such a thing in my own library, even if I really don't know how it's implemented in other libraries. Although I am confident that if there are other ways, they don't differ too much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It took my a few weeks to understand how such a graph can be drawn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's start with a general function $f:\mathbb{R} \times \mathbb{R} \to \mathbb{R}$. What you want is to draw points with a colour which signifies the value of the function. One way would be to simplify the whole problem and draw one point for each pixel. This would work, but will draw only shaded surfaces and it's impossible to draw lines with various formats (dotted lines with some colours and line widths. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real solution which I found makes two simplifications. The first one would the that instead of colouring with a gradient you can colour bands. Suppose that your function $f$ takes values in $[-1, 1]$. You can split your co-domain into many subintervals like: $[-1, -0.9]$, $[-0.9, -0.8]$ and so on. Now what you have to do would be to paint a polygon filled with appropriate color, for each interval. So your original problem is simplified to draw multiple instances of a simpler problem. Note that when your intervals are small enough it will look like a gradient and even a trained eye would not notice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second simplification would be to split the space which needs to be drawn into a grid of small rectangles. So instead of drawing a polygon on the whole surface, you need to fill each small rectangle with appropriate much simpler polygon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's not obvious, the problem is much much simplified. Take for example a one of the rectangle. This rectangle has four corners and you can take an additional point in the center of that rectangle (you might need that point in some certain situations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to fill with appropriate colour the proper region? &#xA;You need to evaluate function in all four corners and in the center. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some specific cases:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;function evaluated in all corners are smaller than the beginning of the interval =&gt; you need to do nothing &lt;/li&gt;&#xA;&lt;li&gt;functions evaluated in all corners are greater than the end of the interval =&gt; you need to do nothing&lt;/li&gt;&#xA;&lt;li&gt;functions evaluated in all corners are within interval =&gt; fill the whole rectangle with an appropriate color&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can stop here if you want, but your figures would looks non-smooth. You can go further:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;left-up, left-down, right-down in interval, right-up bigger =&gt; there are two points, one on up side and one on right side which contains the function evaluated at max value from interval =&gt; those two points together with top-right corner forms a triangle which can be filled&lt;/li&gt;&#xA;&lt;li&gt;many other cases which requires only common judgement to decide which polygons to be formed and should be filled.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Using this algorithm you can fill polygons or draw lines. In the specific case of SVM you need to know that the line which corresponds with $f(x,y)=0$ is the line which classifies points into positive and negative samples. Also, lines which evaluates the function at $-1$ or $1$ corresponds with the maximal margins of SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some time I found that this kind of approach is named iso lines, or iso curves. Perhaps are more similar algorithms like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My implementation is named mesh contour (I did not found a proper name at that time in the beginning) and you can find source &lt;a href=&quot;https://github.com/padreati/rapaio/blob/master/src/main/java/rapaio/graphics/plot/plotcomp/MeshContour.java&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oE7rK.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oE7rK.png&quot; alt=&quot;Mesh curves for grained colour step&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WQsvq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WQsvq.png&quot; alt=&quot;Mesh curves for fine colour steps&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lhlhf.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lhlhf.png&quot; alt=&quot;SVM on synthetic data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-02-29T08:39:00.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10445" PostTypeId="2" ParentId="10424" CreationDate="2016-02-29T12:28:00.573" Score="1" Body="&lt;p&gt;If a couple of words appears together many times in your corpus, e.g., &lt;code&gt;'new'&lt;/code&gt;, &lt;code&gt;'york'&lt;/code&gt;, you can use &lt;a href=&quot;https://radimrehurek.com/gensim/models/phrases.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. It works with more than 2 tokens. This way you have one token called &lt;code&gt;&amp;quot;new_york&amp;quot;&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also for your dict, you can use &lt;a href=&quot;https://radimrehurek.com/gensim/corpora/dictionary.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, it does exactly what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="11007" LastEditorUserId="98307" LastEditDate="2020-08-03T17:06:39.277" LastActivityDate="2020-08-03T17:06:39.277" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10448" PostTypeId="1" CreationDate="2016-02-29T14:26:42.980" Score="1" ViewCount="88" Body="&lt;p&gt;I am working on a project that should predict Alarm based on the input data. I am trying to use a supervised learning algorithm. But I do not have the exact target value based on the input for feeding the learning algorithm. I am trying to generate the target value manually based on the input data (Current Target = previous input[frequency] +constant). But the plot of the target is not a  smooth curve. A sample is given below.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/waKuf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/waKuf.png&quot; alt=&quot;Sample&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the graph given below the yellow line plots the current alarm (manually set), pink is the generated target alarm value and red is the input (frequency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the pink plot isn't a smooth curve, it cannot be used as a efficient target output for learning. I have also tried normalization of the graph which gives a really undesired output. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What method should be used to generate the target value (There are no fixed rule for the alarm value)?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;How can the curve be smoothed desirably?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Project background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working on Knime with a dataset of 6000 rows. &lt;/p&gt;&#xA;" OwnerUserId="13034" LastActivityDate="2016-02-29T14:26:42.980" Title="Trend Decline Graph Normalization" Tags="|machine-learning|bigdata|predictive-modeling|time-series|optimization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10449" PostTypeId="1" AcceptedAnswerId="10451" CreationDate="2016-02-29T16:48:33.507" Score="1" ViewCount="1096" Body="&lt;p&gt;Let's say if I have 2 TB of data, what is the best sample size to pick? I understand that there is a limit on how much RAM/processing power I have and hence I should make my sampling decision around that. But let's say if the processing power is not a concern for me right now. What would be a good way to approach my sample size?&lt;/p&gt;&#xA;" OwnerUserId="16511" LastActivityDate="2016-02-29T18:18:58.380" Title="What is a &quot;good&quot; sample size" Tags="|data-mining|sampling|" AnswerCount="1" CommentCount="1" ClosedDate="2016-02-29T20:16:26.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="10451" PostTypeId="2" ParentId="10449" CreationDate="2016-02-29T18:18:58.380" Score="5" Body="&lt;p&gt;This is a tough question to answer without more information. I'm going to assume that this is for model building, but without more detail it's hard to recommend something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are some things which should &lt;strong&gt;generally&lt;/strong&gt; be known:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Population size&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How large is the population? Does your 2TB of data comprise the total population, or is this a sample of a given timeframe? What frame of data are you looking at - is this 2 days worth of data that is only representative for a given subset of the population, or is this everything? You'll need to know this to know what conclusions you can draw from this dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Variance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the variance of the sample? If it's categorical data, how many unique values are there? Having a metric around this will help determine the number of samples you'll need. If this is a low variance set, you may only need a few hundred/thousand observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Stratification/grouping&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is your data grouped in a meaningful way? If so, you'll need to factor this into your sample. Depending on what you're doing, you'll want a meaningful representation of the population. If the data is not grouped, but has distinct groups within it that you care about, you may need to stratify or pre-process your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model and goals&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of this ends up coming down to what you're trying to do. If you're trying to classify or parse a set of unique entities, you may be better off streaming a large set of your data rather than trying to sample it. If you're trying to classify images or customers based on behavior, you may only need a small subset depending on how these groups differ.&lt;/p&gt;&#xA;" OwnerUserId="16558" LastActivityDate="2016-02-29T18:18:58.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10452" PostTypeId="2" ParentId="9637" CreationDate="2016-02-29T18:21:35.043" Score="4" Body="&lt;p&gt;You need to use some function approximation scheme. In addition, experience replay would be useful for two reasons: (1) you want to keep past memories (2) you need to decorrelate the way to teach your network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at Deepmind's DQN on ATARI games. What you are describing is basically what they have solved. The paper is in their website: &lt;a href=&quot;http://deepmind.com/dqn.html&quot; rel=&quot;nofollow&quot;&gt;http://deepmind.com/dqn.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature. 2015 Feb 25;518(7540):529–33.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With respect to the network architecture, it will definitely require some experimentation. For an alternative, you can also have a look at HyperNEAT. They evolve the network topology:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Hausknecht M, Khandelwal P, Miikkulainen R, Stone P. HyperNEAT-GGP: A HyperNEAT-based Atari general game player. In: Proceedings of the 14th annual conference on Genetic and evolutionary computation p. 217–24. Available from: &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2330195&quot; rel=&quot;nofollow&quot;&gt;http://dl.acm.org/citation.cfm?id=2330195&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more strategic games. Maybe you can have a look at &quot;Giraffe: Using Deep Reinforcement Learning to Play Chess&quot; &lt;a href=&quot;http://arxiv.org/abs/1509.01549&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1509.01549&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2016-02-29T18:28:17.900" LastActivityDate="2016-02-29T18:28:17.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10454" PostTypeId="2" ParentId="10406" CreationDate="2016-02-29T21:09:03.920" Score="2" Body="&lt;p&gt;You might want to provide some more information here. This is either very straightforward or incredibly difficult, depending on your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Brandon Loudermilk is dead on for recommending stratification, especially for something like purchase rate, prior product purchased, etc. This will get you a long ways, but there are a few places this is heavily dependent, and a lot of this boils down to experimental design:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are you hoping to test&lt;/li&gt;&#xA;&lt;li&gt;What defines success? A 10% improvement? A 0.001% improvement?&lt;/li&gt;&#xA;&lt;li&gt;How many observations do you expect in each group?&lt;/li&gt;&#xA;&lt;li&gt;Are you assigning users to groups on the fly, or are you designing this prior?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you're doing this on the fly, for instance, if your tool can split over a given variable/quartile of that variable, it should place users into A or B depending on the strata they belong to. That would cause every using coming in to have a 50/50 chance of landing in the right bucket - you should never end up with only high spenders in one bucket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said - if you can design this before the test, you should be able to ensure this split fairly easily by doing this prior to the test. However, you might want to look up blocking/experimental design if you have a complicated experimental setup. As a final note, remember that you cannot test for randomness - only test for signs against randomness.&lt;/p&gt;&#xA;" OwnerUserId="16558" LastActivityDate="2016-02-29T21:09:03.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10455" PostTypeId="1" CreationDate="2016-02-29T23:32:53.733" Score="2" ViewCount="77" Body="&lt;p&gt;Are there any automated or semi-automated tools for finding matching &quot;similar&quot; or data in two columnar data sets?  The data I'm working with was collected (and handled) by different organizations.  Some rows describe the same events and even carry the same unique identifier (about 70% of the smaller data set can be matched using the unique ids).  For the unmatched cases, there are slight differences such as typos in the unique id, date differences (+/- 1-2 days), and slight differences in value variables... 997.5 v. 1000, etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, by inspection, the value variables, date, and location are close, I call it a match.  Sometimes it is even more obvious such when a one or more digits in the unique ID digit have been transposed or the wrong one has been duplicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus far, I've been doing this by hand.  I've been using R for data cleaning and Excel for comparison and matching (I export comparable subsets, make a crosswalk of old-new unique IDs by hand, and re-import the crosswalk).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd really like is a tool that automates this process by presenting me with information such as: &quot;record 1 in data set A is really similar (for the variables that matter) to records 10, 20, and 25 in data set B.&quot;  Then I could quickly examine the differences, make a choice (say record 20), and then move on.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like it to feel very similar to the way Ancestry.com helps you match up birth, death, marriage, and census records.  But, anything that simplifies and automates this process would be welcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts? &lt;/p&gt;&#xA;" OwnerUserId="16640" LastEditorUserId="16640" LastEditDate="2016-03-01T01:54:54.900" LastActivityDate="2016-03-01T01:54:54.900" Title="What tools are available for semi-automated matching of dirty columnar data" Tags="|data-cleaning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10456" PostTypeId="1" CreationDate="2016-03-01T01:56:31.823" Score="-6" ViewCount="2512" Body="&lt;p&gt;Hello there community im looking for a way to merge 2 live video feeds onto one screen but still keeping 2 different perspectives from individual cameras in their own windows while in right side of the left camera and the left side of the right camera join both of their feed to create one single frame so that at the end your left with 3 video feeds, far left from left cam middle feed from both cameras and the right cam having its far right side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i know it can sound confusing so here is a example &#xA;&lt;a href=&quot;https://i.stack.imgur.com/truV6.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/truV6.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16646" LastActivityDate="2016-03-01T05:20:34.457" Title="merging two video camera into one screen?" Tags="|visualization|computer-vision|" AnswerCount="1" CommentCount="4" ClosedDate="2016-03-01T23:01:29.260" ContentLicense="CC BY-SA 3.0" />
  <row Id="10457" PostTypeId="1" AcceptedAnswerId="10548" CreationDate="2016-03-01T02:46:31.693" Score="6" ViewCount="172" Body="&lt;p&gt;I came across this &lt;a href=&quot;http://media.cs.tsinghua.edu.cn/~taopin/ML2005/intro2Kernel-basedLearn-TNN-2001.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper&lt;/a&gt; which contained this figure &lt;a href=&quot;https://i.stack.imgur.com/CycSG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CycSG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as polygon (or is it a point?).&#xA;I'm having trouble understanding this figure, and since it seems to provide a geometric interpretation of when an SVM performs poorly or not, I'm interested in it. Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="11044" LastActivityDate="2016-03-06T11:11:38.457" Title="Geometric Interpretation of Whether SVMs are performing well or not" Tags="|machine-learning|svm|visualization|performance|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10458" PostTypeId="2" ParentId="10456" CreationDate="2016-03-01T05:11:56.833" Score="3" Body="&lt;p&gt;This problem is called &lt;em&gt;real-time video stitching&lt;/em&gt;. It's pretty hard to do yourself if you're not familiar with computer vision, so &lt;a href=&quot;http://www.mytechlogy.com/IT-blogs/9702/5-real-time-video-stitching-tools&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; are some off-the-shelf tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: If you are looking for a paper: &lt;a href=&quot;http://research.microsoft.com/apps/pubs/default.aspx?id=112596&quot; rel=&quot;nofollow&quot;&gt;Stitching videos streamed by mobile phones in real-time&lt;/a&gt;, or &lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5326097&quot; rel=&quot;nofollow&quot;&gt;Real-time photogrammetric stitching of high resolution video on COTS hardware&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-01T05:20:34.457" LastActivityDate="2016-03-01T05:20:34.457" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10459" PostTypeId="1" AcceptedAnswerId="10461" CreationDate="2016-03-01T05:56:37.497" Score="38" ViewCount="193052" Body="&lt;p&gt;I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of area of activity (theater, cloth stores, food ...) and other data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to create a new data frame and insert a column with the income of all kinds of stores that belong to the same category, and the returning data frame has only the first column filled and the rest is full of NaN's.&#xA;The code that I tired: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;corr = pd.DataFrame()&#xA;for at in activity:&#xA;    stores.loc[stores['Activity']==at]['income']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to do so, so I can use &lt;code&gt;.corr()&lt;/code&gt; to gave the correlation matrix between the category of stores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, I would like to know how I can plot the matrix values (-1 to 1, since I want to use Pearson's correlation) with matplolib.&lt;/p&gt;&#xA;" OwnerUserId="16096" LastEditorUserId="11097" LastEditDate="2016-03-01T08:03:21.320" LastActivityDate="2018-08-30T09:25:03.187" Title="Calculation and Visualization of Correlation Matrix with Pandas" Tags="|python|statistics|visualization|pandas|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10460" PostTypeId="2" ParentId="6634" CreationDate="2016-03-01T08:01:09.950" Score="1" Body="&lt;p&gt;Firstly, cohorts are used mostly for descriptive and inferential analytics, rather than for predictive analytics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon a close look at your dataset, I see a clear seasonal periodicity and there might be trends which can be observed. So, my advise would be to go with the time series analysis (Start with the basic seasonal naive (&lt;code&gt;snaive&lt;/code&gt; in R) algorithm). The &lt;code&gt;snaive&lt;/code&gt; line would give you a nice idea whether to go forward with time series or whether to use a simpler estimation like the regression approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is seasonality and trend, then the &lt;code&gt;snaive&lt;/code&gt; fit would be more accurate than simple regression. Of course, the confidence interval would get wider and wider as you move from &lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-03-01T08:01:09.950" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10461" PostTypeId="2" ParentId="10459" CreationDate="2016-03-01T08:49:41.437" Score="28" Body="&lt;p&gt;&lt;strong&gt;I suggest some sort of play on the following:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the UCI Abalone data for this example...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;&#xA;# Read file into a Pandas dataframe&#xA;from pandas import DataFrame, read_csv&#xA;f = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data'&#xA;df = read_csv(f)&#xA;df=df[0:10]&#xA;df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/efITk.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/efITk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Correlation matrix plotting function:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;# Correlation matric plotting function &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def correlation_matrix(df):&#xA;    from matplotlib import pyplot as plt&#xA;    from matplotlib import cm as cm&#xA;&#xA;    fig = plt.figure()&#xA;    ax1 = fig.add_subplot(111)&#xA;    cmap = cm.get_cmap('jet', 30)&#xA;    cax = ax1.imshow(df.corr(), interpolation=&quot;nearest&quot;, cmap=cmap)&#xA;    ax1.grid(True)&#xA;    plt.title('Abalone Feature Correlation')&#xA;    labels=['Sex','Length','Diam','Height','Whole','Shucked','Viscera','Shell','Rings',]&#xA;    ax1.set_xticklabels(labels,fontsize=6)&#xA;    ax1.set_yticklabels(labels,fontsize=6)&#xA;    # Add colorbar, make sure to specify tick locations to match desired ticklabels&#xA;    fig.colorbar(cax, ticks=[.75,.8,.85,.90,.95,1])&#xA;    plt.show()&#xA;&#xA;correlation_matrix(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YZsF8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YZsF8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2017-01-24T00:35:09.207" LastActivityDate="2017-01-24T00:35:09.207" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10462" PostTypeId="1" CreationDate="2016-03-01T15:15:25.953" Score="1" ViewCount="2179" Body="&lt;p&gt;I'm currently working on implementing a bag of visual words in Python. I get the general gist of how it works but I can't seem to find any sources that explain it in more detail to a level where I can implement it. I'm guessing scikit learn and scikit image would come in but I can't seem to point myself in the right direction. Any help?&lt;/p&gt;&#xA;" OwnerUserId="16661" LastEditorUserId="11097" LastEditDate="2016-03-01T15:59:52.733" LastActivityDate="2020-08-03T18:11:17.553" Title="What is a visual bag of words and how is it implemented?" Tags="|python|classification|scikit-learn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10463" PostTypeId="1" CreationDate="2016-03-01T15:59:41.713" Score="1" ViewCount="124" Body="&lt;p&gt;Say you have a binomial distribution with $p$ very small ($\approx 0.001$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are asked to predict the conditional success rate $SR=S/T$ with $S$ successes out of $T$ trials given a set of conditions $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One would expect (correct me if I'm wrong, though I ran simulations and am quite confident) $SR$ to have a downward with increasing $T$ when $T$ is still small, and to approach $p$ when $T$ is large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution of trials and successes is not uniform across $X$, so the train set tends to give higher $SR$ to conditions with smaller $T$ values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you treat this data skew when constructing a model?&lt;/p&gt;&#xA;" OwnerUserId="14044" LastActivityDate="2016-04-01T15:16:46.927" Title="Skewed binomial data for small p" Tags="|predictive-modeling|feature-engineering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10464" PostTypeId="2" ParentId="10463" CreationDate="2016-03-01T16:15:42.543" Score="1" Body="&lt;p&gt;When fitting a GLM (at least in R), I know there is a optional weight vector that you can include.  This weight is not to give more importance to an observation, but to rather weight observations based on $T$ for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The R documentation says:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; For a binomial GLM prior weights are used to give the number of trials&#xA; when the response is the proportion of successes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I believe this would help adjust for your non-uniformity.  How to choose those weights can be tricky I suppose and can vary a lot with your data, but its worth looking into if it looks like it will help your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D7TPT.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D7TPT.jpg&quot; alt=&quot;Simulation of problem&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have enough data where you can play with dropping data, or maybe if you simulate how much the $SR$ is inflated at low $T$ you can try to adjust for it somehow.  It looks like after about 1000 trials the high success rate issue begins to mellow out, and after 3000 or so you start getting more reliable measures.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastEditorUserId="14913" LastEditDate="2016-03-02T15:11:04.297" LastActivityDate="2016-03-02T15:11:04.297" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10465" PostTypeId="1" CreationDate="2016-03-01T16:29:54.667" Score="0" ViewCount="56" Body="&lt;p&gt;So Now I was working on my project (&quot;Failure prediction models in FoF&quot;) and the Tool I will be using is RapidMinder but I need to have a dataset to run the tool on, I found a nice description of dataset that could be helpful on Nasa website &lt;a href=&quot;https://data.nasa.gov/external-dataset?datasetId=dc5d-zf5v&quot; rel=&quot;nofollow&quot;&gt;https://data.nasa.gov/external-dataset?datasetId=dc5d-zf5v&lt;/a&gt; , However I do not know how to access the data itself in form of CSV file.&lt;/p&gt;&#xA;" OwnerUserId="16664" LastActivityDate="2016-03-01T19:24:10.957" Title="How can I access dataset from Nasa website" Tags="|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10466" PostTypeId="1" CreationDate="2016-03-01T16:44:40.450" Score="6" ViewCount="2324" Body="&lt;p&gt;If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?&lt;/p&gt;&#xA;" OwnerUserId="16665" LastEditorUserId="201" LastEditDate="2019-04-07T12:44:01.273" LastActivityDate="2020-12-31T09:22:26.583" Title="How does Seaborn calculate error bars when using estimators other than the arithmetic mean?" Tags="|python|pandas|descriptive-statistics|seaborn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10467" PostTypeId="2" ParentId="10465" CreationDate="2016-03-01T18:29:57.163" Score="1" Body="&lt;p&gt;It &lt;a href=&quot;http://techport.nasa.gov/view/9418&quot; rel=&quot;nofollow&quot;&gt;seems that there is no downloadable data&lt;/a&gt; (I clicked &lt;em&gt;Show the complete Project library&lt;/em&gt;), but the project was completed in 2012, so you could ask the listed contact, Gary Jahns, explaining your motivation, to be sure: &lt;code&gt;gary.c.jahns@nasa.gov&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-01T19:24:10.957" LastActivityDate="2016-03-01T19:24:10.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10469" PostTypeId="1" AcceptedAnswerId="10472" CreationDate="2016-03-01T21:26:04.460" Score="4" ViewCount="2934" Body="&lt;p&gt;I want to plot large heatmaps (say a matrix $500 \times 500$). I can do it in Python/matplotlib.pyplot with pcolor, but it is not interactive (and I need an interactive heatmap). I have tried with D3.js but what I found is aiming at displaying small heatmaps: &lt;a href=&quot;http://bl.ocks.org/tjdecke/5558084&quot; rel=&quot;nofollow&quot;&gt;http://bl.ocks.org/tjdecke/5558084&lt;/a&gt;&#xA;Naively extending this example with a bigger matrix (e.g. $500 \times 500$) can crash the web-browser.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, can anyone point me toward a good way of displaying and interacting with large heatmaps with a web-based technology: I want to be able to interact on a web-page or a ipython notebook.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2016-03-02T01:48:44.923" Title="How to plot large web-based heatmaps?" Tags="|visualization|javascript|ipython|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10470" PostTypeId="2" ParentId="10469" CreationDate="2016-03-02T00:18:37.360" Score="1" Body="&lt;p&gt;If you can find a way to convert python code into JavaScript, Highcharts JS might support a large-enough heatmap:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.highcharts.com/demo/heatmap-canvas&quot; rel=&quot;nofollow&quot;&gt;http://www.highcharts.com/demo/heatmap-canvas&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-03-02T00:18:37.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10471" PostTypeId="1" AcceptedAnswerId="10474" CreationDate="2016-03-02T01:09:39.420" Score="22" ViewCount="9884" Body="&lt;p&gt;I want to predict some value $Y(x)$ and I am trying to get some prediction $\hat Y(x)$ that optimizes between being as low as possible, but still being larger than $Y(x)$. In other words:&#xA;$$\text{cost}\left\{ Y(x) \gtrsim \hat Y(x) \right\}  &amp;gt;&amp;gt; \text{cost}\left\{ \hat Y(x) \gtrsim Y(x) \right\}  $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a simple linear regression should do totally fine.&#xA;So I somewhat know how to implement this manually, but I guess I'm not the first one with this kind of problem. Are there any packages/libraries (preferably python) out there doing what I want to do?&#xA;What's the keyword I need to look for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if I knew a function $Y_0(x) &amp;gt; 0$ where $Y(x) &amp;gt; Y_0(x)$. What's the best way to implement these restrictions?&lt;/p&gt;&#xA;" OwnerUserId="16683" LastActivityDate="2022-01-11T11:27:06.777" Title="Linear regression with non-symmetric cost function?" Tags="|machine-learning|logistic-regression|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10472" PostTypeId="2" ParentId="10469" CreationDate="2016-03-02T01:48:44.923" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://plot.ly/api/&quot; rel=&quot;nofollow&quot;&gt;Plotly&lt;/a&gt; and &lt;a href=&quot;http://lightning-viz.org/&quot; rel=&quot;nofollow&quot;&gt;Lightning&lt;/a&gt; are [supposedly] able to visualize extremely large data sets.&lt;/p&gt;&#xA;" OwnerUserId="10277" LastActivityDate="2016-03-02T01:48:44.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10474" PostTypeId="2" ParentId="10471" CreationDate="2016-03-02T03:13:43.577" Score="16" Body="&lt;p&gt;If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal L: (x,\alpha) \to x^2 \left( \mathrm{sgn} x + \alpha \right)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $-1 &amp;lt; \alpha &amp;lt; 1$ is a parameter you can use to trade off the penalty of underestimation against overestimation. Positive values of $\alpha$ penalize overestimation, so you will want to set $\alpha$ negative. In python this looks like &lt;code&gt;def loss(x, a): return x**2 * (numpy.sign(x) + a)**2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zCSNb.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zCSNb.png&quot; alt=&quot;Loss functions for two values of a&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next let's generate some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;x = numpy.arange(-10, 10, 0.1)&#xA;y = -0.1*x**2 + x + numpy.sin(x) + 0.1*numpy.random.randn(len(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XNhIG.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XNhIG.png&quot; alt=&quot;Arbitrary function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, we will do our regression in &lt;code&gt;tensorflow&lt;/code&gt;, a machine learning library from Google that supports automated differentiation (making gradient-based optimization of such problems simpler). I will use &lt;a href=&quot;https://github.com/nlintz/TensorFlow-Tutorials/blob/master/1_linear_regression.py&quot; rel=&quot;noreferrer&quot;&gt;this example&lt;/a&gt; as a starting point.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;X = tf.placeholder(&quot;float&quot;) # create symbolic variables&#xA;Y = tf.placeholder(&quot;float&quot;) &#xA;&#xA;w = tf.Variable(0.0, name=&quot;coeff&quot;)&#xA;b = tf.Variable(0.0, name=&quot;offset&quot;)&#xA;y_model = tf.mul(X, w) + b&#xA;&#xA;cost = tf.pow(y_model-Y, 2) # use sqr error for cost function&#xA;def acost(a): return tf.pow(y_model-Y, 2) * tf.pow(tf.sign(y_model-Y) + a, 2)&#xA;&#xA;train_op = tf.train.AdamOptimizer().minimize(cost)&#xA;train_op2 = tf.train.AdamOptimizer().minimize(acost(-0.5))&#xA;&#xA;sess = tf.Session()&#xA;init = tf.initialize_all_variables()&#xA;sess.run(init)&#xA;&#xA;for i in range(100):&#xA;    for (xi, yi) in zip(x, y): &#xA;#         sess.run(train_op, feed_dict={X: xi, Y: yi})&#xA;        sess.run(train_op2, feed_dict={X: xi, Y: yi})&#xA;&#xA;print(sess.run(w), sess.run(b))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;cost&lt;/code&gt; is the regular squared error, while &lt;code&gt;acost&lt;/code&gt; is the aforementioned asymmetric loss function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use &lt;code&gt;cost&lt;/code&gt; you get&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1.00764 -3.32445&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5pW89.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5pW89.png&quot; alt=&quot;cost&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use &lt;code&gt;acost&lt;/code&gt; you get&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1.02604 -1.07742&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s6IDA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s6IDA.png&quot; alt=&quot;acost&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;acost&lt;/code&gt; clearly tries not to underestimate. I did not check for convergence, but you get the idea.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-02T03:19:19.020" LastActivityDate="2016-03-02T03:19:19.020" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10475" PostTypeId="1" CreationDate="2016-03-02T04:36:11.830" Score="3" ViewCount="1346" Body="&lt;p&gt;I am trying to build a fuzzy inference system in Python using skfuzzy library. I have 4 variables depending on which output class is decided.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def fuzzInferenceself():&#xA;&#xA;### Input ###&#xA;    hf_very_poor = fuzz.trimf(hotel_facility, [0, 0.15, 0.3])&#xA;    hf_poor = fuzz.trimf(hotel_facility, [2.5,0.3,0.45])&#xA;    hf_average = fuzz.trimf(hotel_facility, [0.4, 0.5, 0.75])&#xA;    hf_good = fuzz.trimf(hotel_facility, [0.7, 0.85, 0.9])&#xA;    hf_very_good = fuzz.trimf(hotel_facility, [0.875, 0.92, 1.0])&#xA;    vc_less = fuzz.trimf(visited_count, [0, 0.05, 0.1])&#xA;    vc_average = fuzz.trimf(visited_count, [0.05, 0.2, 0.35])&#xA;    vc_many = fuzz.trapmf(visited_count, [0.3,0.45,0.55,0.7])&#xA;    vc_a_lot = fuzz.trapmf(visited_count, [0.65,0.8,0.9,1.0])&#xA;    rm_very_poor = fuzz.trimf(hotel_facility, [0, 0.15, 0.3])&#xA;    rm_poor = fuzz.trimf(hotel_facility, [2.5,0.3,0.45])&#xA;    rm_average = fuzz.trimf(hotel_facility, [0.4, 0.5, 0.75])&#xA;    rm_good = fuzz.trimf(hotel_facility, [0.7, 0.8, 0.9])&#xA;    rm_very_good = fuzz.trimf(hotel_facility, [0.85, 0.9,1.0])&#xA;&#xA;## output ####&#xA;    class_very_poor = fuzz.gaussmf(class_score,1,0.5)&#xA;    class_poor = fuzz.gaussmf(class_score,1.75,0.65)&#xA;    class_average = fuzz.gaussmf(class_score,2.25,0.75)&#xA;    class_good = fuzz.gaussmf(class_score,3,0.25)&#xA;    class_very_good = fuzz.gaussmf(class_score, 3.5, 0.5)&#xA;&#xA;&#xA;def hotelFaclilityClassification(self,A):&#xA;    hf_vp= fuzz.interp_membership(hotel_facility, hf_very_poor, A)&#xA;    hf_p= fuzz.interp_membership(hotel_facility, hf_poor, A)&#xA;    hf_av= fuzz.interp_membership(hotel_facility, hf_average, A)&#xA;    hf_gd= fuzz.interp_membership(hotel_facility, hf_good, A)&#xA;    hf_vg= fuzz.interp_membership(hotel_facility, hf_very_good, A) &#xA;    return dict(hfVP = hf_vp, hfP = hf_p, hfAV = hf_av,hGD = hf_gd, hVG = hf_vg)&#xA;&#xA;def visitCountClassification(B):&#xA;    vc_l = fuzz.interp_membership(visited_count,vc_less)&#xA;    vc_av = fuzz.interp_membership(visited_count,vc_average)&#xA;    vc_mn = fuzz.interp_membership(visited_count,vc_many)&#xA;    vc_al = fuzz.interp_membership(visited_count,vc_a_lot)&#xA;    return dict(vcL = vc_l, vcAV=vc_av, vcMN = vc_mn, vcAL = vc_al )&#xA;&#xA;def roomFacilityClassification(C):&#xA;    rm_vp = fuzz.interp_membership(room_facility,rm_very_poor)&#xA;    rm_p =  fuzz.interp_membership(room_facility,rm_poor)&#xA;    rm_av = fuzz.interp_membership(room_facility,rm_average)&#xA;    rm_gd = fuzz.interp_membership(room_facility,rm_good)&#xA;    rm_vg = fuzz.interp_membership(room_facility,rm_very_good)&#xA;    return dict(rmVP = rm_vp, rmP = rm_p, rmAV = rm_av, rmGD = rm_gd, rmVG = rm_vg)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;A similar function for price is: &lt;code&gt;def priceClassification(D)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The rules are as follows:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;If Hotel facility score is Very good, visited count is a lot, room facility is very good, prices is less then class is very good.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I do not understand how to code the rules. All sources I have seen takes one input and one output variable. But this is not the case in my code.&lt;/p&gt;&#xA;&lt;p&gt;Can anyone give me a good resource or idea about how to code this rule?&lt;/p&gt;&#xA;" OwnerUserId="13046" LastEditorUserId="25750" LastEditDate="2022-10-06T19:52:18.647" LastActivityDate="2022-10-06T19:52:18.647" Title="Fuzzy Rules with more than two variable in python" Tags="|python|scikit-learn|fuzzy-logic|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10476" PostTypeId="1" AcceptedAnswerId="10477" CreationDate="2016-03-02T06:14:13.313" Score="7" ViewCount="4345" Body="&lt;p&gt;I want to make a sklearn pipeline using the custom Artificial Neural Network I already have. I want to make pipeline in which input goes to ANN and its output goes to the sklearn.svm.SVC model and final prediction is made.&#xA;So, how can I do this using sklearn pipeline?&lt;/p&gt;&#xA;" OwnerUserId="14310" LastEditorUserId="14310" LastEditDate="2016-03-04T06:13:38.087" LastActivityDate="2016-03-04T06:13:38.087" Title="how to make sklearn pipeline using custom model?" Tags="|neural-network|svm|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10477" PostTypeId="2" ParentId="10476" CreationDate="2016-03-02T07:59:43.067" Score="10" Body="&lt;p&gt;Implementing a custom transformer is simple. You have to implement the fit and transform methods like below. Since your ANN is already trained (right?) the fit method has to do nothing, just return self. And the transform method has to pass the incoming data to the ANN and return its output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.base import BaseEstimator, TransformerMixin&#xA;&#xA;class MyANNTransformer(BaseEstimator, TransformerMixin):&#xA;    def __init__(self, ann):&#xA;        self.ann = ann&#xA;&#xA;    def fit(self, X, y):&#xA;        return self&#xA;&#xA;    def transform(self, X)&#xA;        return self.ann.predict(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can include that in pipelines:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.pipeline import make_pipeline&#xA;from sklearn.svm import SVC&#xA;&#xA;pipe = make_pipeline(MyANNTransformer(ann),&#xA;                     SVC())&#xA;&#xA;pipe.fit(Xtrain, ytrain)&#xA;pipe.predict(Xtest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-03-02T07:59:43.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10478" PostTypeId="1" CreationDate="2016-03-02T08:54:10.503" Score="25" ViewCount="16411" Body="&lt;p&gt;I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of &lt;code&gt;gather()&lt;/code&gt;, &lt;code&gt;separate()&lt;/code&gt; and &lt;code&gt;spread()&lt;/code&gt;, which can be found in R &lt;code&gt;tidyr&lt;/code&gt; tool.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastEditorUserId="29169" LastEditDate="2021-02-09T20:17:00.320" LastActivityDate="2021-06-25T18:59:23.340" Title="Is there any data tidying tool for python/pandas similar to R tidyr tool?" Tags="|r|python|dataset|data-cleaning|pandas|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10479" PostTypeId="1" CreationDate="2016-03-02T09:27:41.087" Score="3" ViewCount="3062" Body="&lt;p&gt;How do Hyperbolic Tangent Kernels work? That is what is the intuition behind them? Can you provide proofs and examples for illustration?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hyperbolic Tangent Kernels are defined as:&#xA;$$ K(x, x^\prime) = tanh\bigg(\alpha (x\cdot x^\prime) + c\bigg)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for Gaussian RBF kernel, the intuition is that the support vectors affect the decision surface based on the locality of influence. What is the analog for Hyperbolic Tangent (Sigmoid Kernels)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some references on the Hyperbolic Tangent Kernels are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.csie.ntu.edu.tw/~cjlin/papers/tanh.pdf&quot; rel=&quot;nofollow&quot;&gt;Hsuan-Tien Lin and Chih-Jen Lin&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://perso.lcpc.fr/tarel.jean-philippe/publis/jpt-icme05.pdf&quot; rel=&quot;nofollow&quot;&gt;Sabri Boughorbel, Jean-Philippe Tarel, Nozha Boujemaa&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="11044" LastEditorUserId="11097" LastEditDate="2016-03-08T06:46:08.820" LastActivityDate="2016-03-09T04:24:53.393" Title="On the properties of Hyperbolic Tangent Kernel" Tags="|machine-learning|svm|supervised-learning|reference-request|" AnswerCount="1" CommentCount="2" ClosedDate="2016-03-10T11:04:20.293" ContentLicense="CC BY-SA 3.0" />
  <row Id="10480" PostTypeId="1" CreationDate="2016-03-02T09:34:14.203" Score="1" ViewCount="13" Body="&lt;p&gt;this is my first post on the Stack Exchange site, and I'm looking for some guidance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of 10x input numbers (positive), and a set of 10x output numbers (negative), as well as the real net change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a perfect world, adding those twenty numbers up would be equal to the 'real net change'. Unfortunately, the set in incomplete, and every week there is an error. If it's of any help, the error is always in the same direction (modelled change too big) - it appears that the set of output numbers is incomplete. Sometimes though, the error is non-existent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What approach would you recommend to model the real change?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used Weka for regression analysis, but it appears to me that weka is much more powerful when used with categorising problems rather than continuous data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate all comments and apologise if the problem stated is too general.&lt;/p&gt;&#xA;" OwnerUserId="16692" LastActivityDate="2016-03-02T09:34:14.203" Title="Beginner: how to model net change given an incomplete set of input/output figures" Tags="|regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10481" PostTypeId="1" CreationDate="2016-03-02T10:09:15.650" Score="2" ViewCount="201" Body="&lt;p&gt;This question is more about inference and decision making based on statistical data, so I hope I'm in the right place. If not, let me know if I should migrate this question elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set specifying the predicted shoe size of different people for various shoe models, and also the manually measured correct shoe size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance person #1 was predicted to have a shoe size of 8 for shoe model A, and the actual size she wears is 8.5. Each record includes the person, shoe model, predicted size and actual size. I already calculated the delta between these two values for each record, then summarized the average offset and standard deviation for all shoe models. Here's an example of the offset data (&lt;em&gt;n&lt;/em&gt; is the sample size from the average offset and standard deviation was calculated for each model):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-------+----------------+-----+----+&#xA;| model | Average Offset | SD  | n  |&#xA;+-------+----------------+-----+----+&#xA;|     1 | 0.4            | 1.0 | 16 |&#xA;|     2 | -0.8           | 0.8 |  5 |&#xA;|     3 | 0.8            | 0.7 | 10 |&#xA;|     4 | 0.5            | 0.9 | 12 |&#xA;|     5 | 0.7            | 0.8 |  6 |&#xA;|     6 | 0.1            | 0.9 | 28 |&#xA;|     7 | 0.5            | 0.8 | 16 |&#xA;|     8 | 0.1            | 0.7 | 18 |&#xA;|     9 | 0.3            | 0.5 |  6 |&#xA;|    10 | 2.7            | 0.3 |  5 |&#xA;|    11 | -0.2           | 0.6 | 33 |&#xA;|    12 | 1.0            | 0.5 |  6 |&#xA;|    13 | 0.0            | 0.0 |  5 |&#xA;|    14 | -0.1           | 0.6 | 13 |&#xA;|    15 | 0.0            | 0.4 |  4 |&#xA;|    16 | -0.9           | 0.5 |  7 |&#xA;|    17 | 0.2            | 0.8 |  9 |&#xA;|    18 | -0.2           | 0.8 | 20 |&#xA;|    19 | -1.1           | 0.7 |  9 |&#xA;|    20 | -0.1           | 0.6 | 14 |&#xA;|    21 | -1.1           | 0.8 | 55 |&#xA;|    22 | -1.2           | 0.8 | 12 |&#xA;|    23 | -0.3           | 0.5 | 12 |&#xA;|    24 | -0.1           | 0.4 | 10 |&#xA;|    25 | 0.6            | 0.9 | 29 |&#xA;+-------+----------------+-----+----+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And as a chart:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/w2irU.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w2irU.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're wondering why analyze this per shoe model at all, then the answer is that we know from shop experience that different shoe models have particular structures and materials that have of consistent effect on shoe size preference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to my actual question:&#xA;My goal is to plug average offsets back into the prediction calculation to adjust it based on previously recorded offsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is - when does it make sense to use an offset value, and when that value is unusable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to disregard average offsets where the SD is close to the size of the average, but mathematically, plugging in such an average to adjust all predictions for this model is likely to do more good than harm. Or does it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a good approach to decide which of these averages is useful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestion for additional insights or analysis techniques are welcome.&lt;/p&gt;&#xA;" OwnerUserId="16693" LastEditorUserId="16693" LastEditDate="2016-03-02T10:45:09.680" LastActivityDate="2016-03-02T10:45:09.680" Title="Adjusting predicted values based on average offsets from past predictions" Tags="|statistics|predictive-modeling|descriptive-statistics|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10483" PostTypeId="2" ParentId="5121" CreationDate="2016-03-02T12:00:50.990" Score="12" Body="&lt;p&gt;Jaccard similarity is used for two types of binary cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Symmetric, where 1 and 0 has equal importance (gender, marital status,etc)&lt;/li&gt;&#xA;&lt;li&gt;Asymmetric, where 1 and 0 have different levels of importance (testing positive for a disease)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity is usually used in the context of text mining for comparing documents or emails. If the cosine similarity between two document term vectors is higher, then both the documents have more number of words in common&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another difference is 1 - Jaccard Coefficient can be used as a dissimilarity or distance measure, whereas the cosine similarity has no such constructs. A similar thing is the Tonimoto distance, which is used in taxonomy.&lt;/p&gt;&#xA;" OwnerUserId="11027" LastActivityDate="2016-03-02T12:00:50.990" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10484" PostTypeId="1" CreationDate="2016-03-02T12:49:00.370" Score="1" ViewCount="3810" Body="&lt;p&gt;This follows a recent post where I got my answer concerning possible web-based techno for plotting big heatmaps ($600 \times 600$ dissimilarity matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, that I have achieved that, I'm struggling with setting the color scale. &#xA;I would like some fine continuous color scale. For now I have plugged a discretization into 17 colors... not enough. With Python, pretty much is done with pyplot.pcolor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I obtain such a 'continuous' color scale?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the difference between the two heatmaps parameterized by the same matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using matplotlib.pyplot in Python&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qNbgX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qNbgX.png&quot; alt=&quot;using matplotlib.pyplot in python&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using D3.js and the code below&#xA;&lt;a href=&quot;https://i.stack.imgur.com/1W7Lr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1W7Lr.png&quot; alt=&quot;using d3.js and the code below&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;script src=&quot;http://d3js.org/d3.v3.js&quot;&amp;gt;&amp;lt;/script&amp;gt;&#xA;&amp;lt;canvas id=&quot;heatmap&quot;&amp;gt;&amp;lt;/canvas&amp;gt;&#xA;&amp;lt;script&amp;gt;&#xA;var tsvName = 'wasserstein_dists.tsv';&#xA;var canvas = document.getElementById('heatmap');&#xA;var ctx = canvas.getContext(&quot;2d&quot;);&#xA;&#xA;var scale = 1;&#xA;var colors = [&quot;#6363FF&quot;, &quot;#6373FF&quot;, &quot;#63A3FF&quot;, &quot;#63E3FF&quot;, &quot;#63FFFB&quot;, &quot;#63FFCB&quot;,&#xA;           &quot;#63FF9B&quot;, &quot;#63FF6B&quot;, &quot;#7BFF63&quot;, &quot;#BBFF63&quot;, &quot;#DBFF63&quot;, &quot;#FBFF63&quot;, &#xA;           &quot;#FFD363&quot;, &quot;#FFB363&quot;, &quot;#FF8363&quot;, &quot;#FF7363&quot;, &quot;#FF6364&quot;];&#xA;var heatmapColor = d3.scale.linear()&#xA;    .domain(d3.range(0, 1, 1.0 / (colors.length - 1)))&#xA;    .range(colors);&#xA;&#xA;&#xA;// On charge le tsv :&#xA;d3.tsv(tsvName, function (d) {&#xA;  return {&#xA;    x: +d.x,&#xA;    y: +d.y,&#xA;    v: +d.v,&#xA;  };&#xA;}, function (err, data) {&#xA;  if (err != null) { return alert('No tsv file'); }&#xA;&#xA;&#xA;  var size = Math.sqrt(data.length);&#xA;  canvas.width = size * scale;&#xA;  canvas.height = size * scale;&#xA;&#xA;  var c = d3.scale.linear()&#xA;    .domain([d3.min(data, function (d) { return d.v; }), d3.max(data, function (d) { return d.v; })])&#xA;    .range([0,1]);&#xA;&#xA;  data.forEach(function (d) {&#xA;    ctx.fillStyle = heatmapColor(c(d.v));&#xA;    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);&#xA;  });&#xA;});&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;/p&gt;&#xA;" OwnerUserId="7966" LastEditorUserId="7966" LastEditDate="2016-03-02T12:59:35.060" LastActivityDate="2016-03-02T13:44:39.057" Title="heatmap color and d3.js" Tags="|visualization|javascript|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10486" PostTypeId="2" ParentId="10484" CreationDate="2016-03-02T13:44:39.057" Score="2" Body="&lt;p&gt;Actually you are almost there. Just use the min/max function directly in the  color scale instead of using the c var. Use the max/min function in the domain and the start/stop color in the range. The d3.scale is already a gradient function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;script src=&quot;http://d3js.org/d3.v3.js&quot;&amp;gt;&amp;lt;/script&amp;gt;&#xA;&amp;lt;canvas id=&quot;heatmap&quot;&amp;gt;&amp;lt;/canvas&amp;gt;&#xA;&amp;lt;script&amp;gt;&#xA;var tsvName = 'wasserstein_dists.tsv';&#xA;var canvas = document.getElementById('heatmap');&#xA;var ctx = canvas.getContext(&quot;2d&quot;);&#xA;&#xA;var scale = 1;&#xA;&#xA;// On charge le tsv :&#xA;d3.tsv(tsvName, function (d) {&#xA;  return {&#xA;    x: +d.x,&#xA;    y: +d.y,&#xA;    v: +d.v,&#xA;  };&#xA;}, function (err, data) {&#xA;  if (err != null) { return alert('No tsv file'); }&#xA;&#xA;  var size = Math.sqrt(data.length);&#xA;  var heatmapColor = d3.scale.linear()&#xA;    .domain([d3.min(data, function(d) { return d.v; }), d3.max(data, function(d) { return d.v; })])&#xA;    .range([&quot;#6363FF&quot;,  &quot;#FF6364&quot;]);&#xA;  canvas.width = size * scale;&#xA;  canvas.height = size * scale;&#xA;&#xA;  data.forEach(function (d) {&#xA;    ctx.fillStyle = heatmapColor(d.v);&#xA;    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);&#xA;  });&#xA;});&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of cause you have to first define data and then the color gradient with the min/max function.&lt;/p&gt;&#xA;" OwnerUserId="16465" LastActivityDate="2016-03-02T13:44:39.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10489" PostTypeId="1" AcceptedAnswerId="10492" CreationDate="2016-03-02T22:10:32.293" Score="1" ViewCount="175" Body="&lt;p&gt;I have a classification task for people with 3 categories. I want to apply machine learning for that. I have 10 sources of data, which have the same fields (say 4: age, job title, a number of organizations, a number of followers). Data is incomplete, some fields can be missing in some profiles. The training set is limited (say, 300 examples).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two strategies for feature engineering, and I don't know which one to use.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Expand features: take 40 features (Profile 1 age, Profile 1 job title, ..., Profile 10 age, Profile 10 job title).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compact features: take 4 features, and apply some heuristics to merge the values from different profiles. Say, take age and job title which occur most frequently, take a maximum number of organizations, take a sum of numbers of followers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What strategy is generally used to give best results and why?&lt;/p&gt;&#xA;" OwnerUserId="16680" LastEditorUserId="11097" LastEditDate="2016-03-03T00:37:08.883" LastActivityDate="2016-03-03T00:52:19.427" Title="Expand or compact features?" Tags="|machine-learning|classification|feature-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10490" PostTypeId="1" AcceptedAnswerId="10491" CreationDate="2016-03-02T22:16:43.580" Score="5" ViewCount="3138" Body="&lt;p&gt;I'm applying ML for classification task in Python using sklearn/pandas. I'm going to try various things to get the best results, and I wonder how do I effectively store and analyze all the parameters and results of the classification? Parameters include:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Number of training examples (which can be extended as I get more labeled data).&lt;/li&gt;&#xA;&lt;li&gt;Set of features.&lt;/li&gt;&#xA;&lt;li&gt;Classification algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Algorithm hyperparameters.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Results include:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Precision/recall for each of the classes.&lt;/li&gt;&#xA;&lt;li&gt;Overall precision/recall.&lt;/li&gt;&#xA;&lt;li&gt;Support for each class, etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Of course, I can manually copy the parameters and results to an Excel spreadsheet every time, but it's not an optimal solution. Are there any Python libraries (or modules of sklearn/pandas) which allow to easily store and display the parameters and results for later analysis? How do you solve this task?&lt;/p&gt;&#xA;" OwnerUserId="16680" LastActivityDate="2017-07-05T18:06:54.633" Title="How to store and analyze classification results with Python?" Tags="|machine-learning|python|classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10491" PostTypeId="2" ParentId="10490" CreationDate="2016-03-02T22:51:19.100" Score="5" Body="&lt;p&gt;&lt;a href=&quot;http://sacred.readthedocs.org/&quot; rel=&quot;noreferrer&quot;&gt;Sacred&lt;/a&gt; is a &lt;a href=&quot;https://github.com/IDSIA/sacred&quot; rel=&quot;noreferrer&quot;&gt;python library&lt;/a&gt; developed by the &lt;a href=&quot;http://www.idsia.ch/&quot; rel=&quot;noreferrer&quot;&gt;IDSIA lab&lt;/a&gt; that &quot;facilitates automated and reproducible experimental research&quot;. It is available through pip as &lt;code&gt;sacred&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a related discussion see &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/3npg0d&quot; rel=&quot;noreferrer&quot;&gt;reddit&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-02T22:51:19.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10492" PostTypeId="2" ParentId="10489" CreationDate="2016-03-03T00:05:55.693" Score="3" Body="&lt;p&gt;The way I see it is that your 10 sources of data, they all refer to the same set of people. Depending on the attributes, some can be expanded, some can be merged ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attributes such as age should be unique, so it doesn't make sense to expand it to Profile 1 age, profile 2 age ... One simple way is merge them is by using the average or use max. Expanding age only add redundant data to your feature matrix, and increase its dimensionality, in most cases, this doesn't help generalization performance of your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, number of followers can be expanded. Depending on the data source, a guy has 10 followers on Twitter but 1000 followers on Google+ might simply mean that he barely uses Twitter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, the way you pick your features or engineer new features should increase your model performance, so if expanding number of followers actually decrease Cross Validation or Test performance, compared to the one using sum of followers then you can simply use sum of followers. &lt;/p&gt;&#xA;" OwnerUserId="14313" LastEditorUserId="14313" LastEditDate="2016-03-03T00:52:19.427" LastActivityDate="2016-03-03T00:52:19.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10493" PostTypeId="1" AcceptedAnswerId="10495" CreationDate="2016-03-03T00:09:55.310" Score="1" ViewCount="63" Body="&lt;p&gt;I have a table formatted like the following table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Feature amount  ID  Location&#xA;Feat1      2    1   US&#xA;Feat2      0    1   US&#xA;Feat3      0    1   US&#xA;Feat4      1    1   US&#xA;Feat2      2    2   US&#xA;Feat4      0    2   US&#xA;Feat3      0    2   US&#xA;Feat6      1    2   US&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's say I have 200 different IDs.&#xA;I want to convert all different features into variables and amount into observations, so I combine rows with same ID into one row. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Feat1 Feat2 Feat3 Feat4 Feat5 Feat6 ID Location&#xA;  2     0     0     1    NA    NA   1    US&#xA; NA     2     0     0    NA    1    2    US&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a good way to do it either in Python (pandas) or R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="16677" LastActivityDate="2016-03-03T01:23:15.633" Title="convert some observations into variables" Tags="|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10494" PostTypeId="1" CreationDate="2016-03-03T00:21:26.643" Score="6" ViewCount="620" Body="&lt;p&gt;I have a series of seemingly random data dripping in one value at a time through time. Although it appears to be random, the data forms clusters when certain attributes are analysed which the charts show. I'm trying to avoid the fallacy of seeing patterns that are not there, but across several datasets the data does indeed appear to form clusters through time. I'm interested in trying to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;detect the clusters of either the blue squares and / or the white squares in the data I have&lt;/li&gt;&#xA;&lt;li&gt;ignoring data which are outliers&lt;/li&gt;&#xA;&lt;li&gt;predicting the future shape of the clusters as data is dripped in one column at a time. (There appears to be a certain momentum through time of the cluster shapes where the current cluster areas of the new data is in nearby region of the preceding clusters)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They appear to take on regular semi-predictable shapes which (to my eyes at least) follow a momentum from the data before. How could I detect this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is this cluster analysis, and&lt;/li&gt;&#xA;&lt;li&gt;what would be the best way to detect the white squares and / or the blue areas. I'm guessing the blue areas would be the easiest to cluster?&lt;/li&gt;&#xA;&lt;li&gt;the white squares appear fairly close to the previous y-values and may continue at a certain angle up or down in value. How is the best way to predict this?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MjTMN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MjTMN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/zMEjM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zMEjM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/0LVAh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0LVAh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Tro3y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Tro3y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2861" LastEditorUserId="2861" LastEditDate="2016-03-03T17:30:14.940" LastActivityDate="2016-10-21T13:38:37.663" Title="Is this cluster analysis / prediction?" Tags="|clustering|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="10495" PostTypeId="2" ParentId="10493" CreationDate="2016-03-03T01:23:15.633" Score="1" Body="&lt;p&gt;Assume your table can be put into a pandas DataFrame object &lt;b&gt;data&lt;/b&gt; with 4 columns as above. One way to achieve what you want is to do a GROUPBY using ID and Location. Then gradually assign values to each row of new table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    newdata = pd.DataFrame(columns=['ID', 'Location', 'Feat1', 'Feat2', 'Feat3', 'Feat4', 'Feat5', 'Feat6'])&#xA;    grouped = data.groupby(['ID', 'Location'])&#xA;&#xA;    for index, (group_name, d) in enumerate(grouped):&#xA;        newdata.loc[index, 'ID'] = group_name[0]&#xA;        newdata.loc[index, 'Location'] = group_name[1]&#xA;        for feature, amount in zip(d['Feature'], d['amount']):&#xA;            newdata.loc[index, feature] = amount&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-03T01:23:15.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10497" PostTypeId="1" CreationDate="2016-03-03T03:09:33.047" Score="1" ViewCount="704" Body="&lt;p&gt;I've been doing machine learning and neural networks for a couple months, and finished a 5 months online course of it last year, but I'm struggling in an specific scenario:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we've got this data: &lt;strong&gt;X1&lt;/strong&gt;, &lt;strong&gt;X2&lt;/strong&gt;, &lt;strong&gt;F3&lt;/strong&gt;, &lt;strong&gt;F4&lt;/strong&gt; and &lt;strong&gt;Y1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X1 and X2 are possible to change.&#xA;F3 and F4 change only with time and arbitrary.&#xA;Y1 only changes as a result of X1, X2, F3 and F4 changing.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've collected thousands of samples ready to train, but I don't know what type of algorithm should I use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This is how I want to use it:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to &quot;reach&quot; a desired target value of &lt;strong&gt;Y1&lt;/strong&gt;, setting the combination of values of &lt;strong&gt;X1&lt;/strong&gt; and &lt;strong&gt;X2&lt;/strong&gt;, respecting the existing and unchangeable values of &lt;strong&gt;F3&lt;/strong&gt; and &lt;strong&gt;F4&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[X1, X2] = run_algorithm(Y1, F3, F4);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And what if I also decide to include these &lt;strong&gt;X1&lt;/strong&gt; and &lt;strong&gt;X2&lt;/strong&gt; in the algorithm too, so it finds the most efficient way of doing it knowing the already existing values of &lt;strong&gt;X1&lt;/strong&gt; and &lt;strong&gt;X2&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[X1, X2] = run_algorithm(Y1, F3, F4, X1, X2);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16712" LastEditorUserId="16712" LastEditDate="2016-03-03T11:16:33.180" LastActivityDate="2016-03-03T11:16:33.180" Title="Machine Learning with fixed and variable inputs and multiple outputs" Tags="|machine-learning|algorithms|regression|random-forest|linear-regression|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10498" PostTypeId="2" ParentId="10015" CreationDate="2016-03-03T03:11:39.900" Score="3" Body="&lt;p&gt;You made three mistakes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You omitted the offset terms before the nonlinear transformations (variables b_1 and b_out). This increases the representative power of the neural network.&lt;/li&gt;&#xA;&lt;li&gt;You omitted the softmax transformation at the top layer. This makes the output a probability distributions, so you can calculate the cross-entropy, which is the usual cost function for classification.&lt;/li&gt;&#xA;&lt;li&gt;You used the binary form of the cross-entropy when you should have used the multi-class form.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When I run this I get accuracies over 90%:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;&#xA;mnist = input_data.read_data_sets('/tmp/MNIST_data', one_hot=True)&#xA;&#xA;x = tf.placeholder(tf.float32, shape=[None, 784])&#xA;y = tf.placeholder(tf.float32, shape=[None, 10])&#xA;&#xA;W_h1 = tf.Variable(tf.random_normal([784, 512]))&#xA;b_1 = tf.Variable(tf.random_normal([512]))&#xA;h1 = tf.nn.sigmoid(tf.matmul(x, W_h1) + b_1)&#xA;&#xA;W_out = tf.Variable(tf.random_normal([512, 10]))&#xA;b_out = tf.Variable(tf.random_normal([10]))&#xA;y_ = tf.nn.softmax(tf.matmul(h1, W_out) + b_out)&#xA;&#xA;# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)&#xA;cross_entropy = tf.reduce_sum(- y * tf.log(y_), 1)&#xA;loss = tf.reduce_mean(cross_entropy)&#xA;train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)&#xA;&#xA;correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&#xA;# train&#xA;with tf.Session() as s:&#xA;    s.run(tf.initialize_all_variables())&#xA;&#xA;    for i in range(10000):&#xA;        batch_x, batch_y = mnist.train.next_batch(100)&#xA;        s.run(train_step, feed_dict={x: batch_x, y: batch_y})&#xA;&#xA;        if i % 1000 == 0:&#xA;            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})&#xA;            print('step {0}, training accuracy {1}'.format(i, train_accuracy))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-03T03:11:39.900" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10499" PostTypeId="1" CreationDate="2016-03-03T09:36:36.193" Score="1" ViewCount="696" Body="&lt;p&gt;When trying to model as a recommendation problem the selection of an item that can be selected (and rated) by the same user many times, I can't find references of previous work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example User1 can select item A (and rate it) then item A again (and give it a different rate) and then again item A can be selected n times.. then the same for item B, but item A could be again selected at any time point, and that could happen for any of the A, B, C... Z items. Could this be modelled somehow as a recommender system?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When asked in SO I was told that can't be modelled even as a multi-modal recommneder, then I think my question should be extended to how could I model this for prediction of next rate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/35767636/collaborative-filtering-when-multiple-items-are-rated-multiple-times-by-same-use?noredirect=1#comment59207724_35767636&quot;&gt;SO question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16723" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-03-03T16:57:57.617" Title="Collaborative filtering when multiple items are rated multiple times by same user" Tags="|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10502" PostTypeId="1" AcceptedAnswerId="10524" CreationDate="2016-03-03T16:25:23.677" Score="2" ViewCount="1110" Body="&lt;p&gt;In &lt;code&gt;Word2Vec&lt;/code&gt;, If I train a set of sentences multiple times with change in order (as it increases the vector representations), will the frequency of a word get changed due to it.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I have the word &quot;deer&quot; in my corpus 4 times and If I set the &lt;code&gt;min_count&lt;/code&gt; to be 5, does training the model 3 times repeatedly count &quot;deer&quot; with frequency 12 and will be included in the model ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it knows it is the same corpus then how it is possible to differentiate, if I retrain the model with a new corpus.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-03-04T09:34:33.680" Title="Does traning the Word2Vec model multiple times affect `min_count` parameter?" Tags="|predictive-modeling|word-embeddings|word2vec|gensim|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10503" PostTypeId="2" ParentId="10499" CreationDate="2016-03-03T16:31:55.730" Score="1" Body="&lt;p&gt;I don't think there is any academic work on the subject, at least that I know of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple way of using that data would be to use the &lt;strong&gt;mean&lt;/strong&gt; of the ratings or other average like measures such as a &lt;strong&gt;moving average&lt;/strong&gt;, a &lt;strong&gt;time weighted average&lt;/strong&gt;, the &lt;strong&gt;median&lt;/strong&gt;, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this approach is probably not exactly what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to look at collaborative filtering approaches with temporal dynamics, there might be something interesting for you.&lt;/p&gt;&#xA;" OwnerUserId="11121" LastActivityDate="2016-03-03T16:31:55.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10504" PostTypeId="2" ParentId="10499" CreationDate="2016-03-03T16:57:57.617" Score="0" Body="&lt;p&gt;As to predicting the next rate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very simple (kind of brute force)&#xA;Would be a Dataming approach: Treat it as a timeseries or a lagged model. All or a certain number of past rates are used as inputs to explain the current rate. Input the data into your favorite classifier algorithm (should be something able to handle nominal target variables). Of cause you need a rather large number of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefere a graph approach you might want to take a look into random fields or markov models for sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is going to be an online recommendation problem, a graph database would offer the structure and capabilities to efficiently compute the most likely next outcome.&lt;/p&gt;&#xA;" OwnerUserId="16465" LastActivityDate="2016-03-03T16:57:57.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10505" PostTypeId="2" ParentId="6435" CreationDate="2016-03-03T17:38:03.550" Score="0" Body="&lt;p&gt;All in all, what you are trying to reach is Pareto efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to make it cooperative, you need to define a single &lt;a href=&quot;https://stats.stackexchange.com/questions/149231/why-and-when-does-one-have-to-learn-the-reward-function-from-samples-in-reinfo&quot;&gt;reward function&lt;/a&gt; that is shared by all the players (it could be a function that combines in some way individual reward functions). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somehow, you need to weight the rewards that you obtain from one product with respect to the others.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-11-29T10:53:43.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10506" PostTypeId="2" ParentId="10490" CreationDate="2016-03-03T18:21:17.377" Score="1" Body="&lt;p&gt;I ended up using &lt;a href=&quot;https://github.com/machinalis/featureforge/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Forge&lt;/a&gt;, which is a python library that &quot;provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, etc.), and particularly helpful if you use scikit-learn (although this can work if you have a different algorithm)&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose it because it seems to be a more specific tool for machine learning than Sacred, which appears to be a more generic tool. Feature Forge also provides a convenient framework for describing features in a scalable and reusable class-based way. It also has functions to store experimental results in a Mongo database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to @Emre for pointing to &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/3npg0d&quot; rel=&quot;nofollow noreferrer&quot;&gt;reddit discussion&lt;/a&gt;, where featureforge was mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPD&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote a library based on Feature Forge to support the full machine learning experimentation pipeline: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;classifier configuration management&lt;/li&gt;&#xA;&lt;li&gt;feature description&lt;/li&gt;&#xA;&lt;li&gt;training/testing classifiers&lt;/li&gt;&#xA;&lt;li&gt;storing/analyzing results. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Feel free to contribute: &lt;a href=&quot;https://github.com/goldan/machinery&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/goldan/machinery&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16680" LastEditorUserId="16680" LastEditDate="2017-07-05T18:06:54.633" LastActivityDate="2017-07-05T18:06:54.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10508" PostTypeId="1" CreationDate="2016-03-03T19:46:52.493" Score="5" ViewCount="676" Body="&lt;p&gt;I have a problem that I think machine learning can solve. I'm new to it however and don't know where to start so any pointers to appropriate solving methods/guides etc for the problem are appreciated. &#xA;I've searched a bit and couldn't find any questions similar to this, but most likely because I dont know exactly what to search for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say we have a bunch of fruits: &lt;code&gt;a,b,c,d,e,f&lt;/code&gt; which all have two factors associated with it, a price and a quality. Every day in a store the customers can choose between two bundles&lt;code&gt;(g1,g2)&lt;/code&gt;, each consisting of two fruits with varying price and quality&lt;code&gt;(p,q)&lt;/code&gt;. The 4 fruits every day must be unique, so g1 cannot have a fruit in g2.&#xA;The customers then buy one bundle each. They value the price and quality, but are probably also influenced by outside factors such as what fruit it is and what it is combined with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to create a training set from this?&lt;br&gt; I came up with two ways to represent this:&lt;br&gt;&#xA;The simplest way i could think of: let a fruit x be represented as &lt;code&gt;x:{x_p,x_q}&lt;/code&gt;, a group as &lt;code&gt;g1(a,b)&lt;/code&gt;, and a training element would then be defined as &lt;code&gt;day(g1,g2)&lt;/code&gt; which we will use to train our machine.&lt;br&gt;&#xA;The second way was to let the fruits contain a present flag to mark if they are in a group or not, &lt;code&gt;x:{x_p,x_q,present}&lt;/code&gt; and a group would then become &lt;code&gt;g1(a,b,c,d,e,f)&lt;/code&gt; and a day still &lt;code&gt;day(g1,g2)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to do it the first way without data for several fruits, or is it just implementation specific?&lt;br&gt;&#xA;How would one train for the mirroring case of &lt;code&gt;day(g1(a,b),g2(c,d))&lt;/code&gt; vs &lt;code&gt;day(g1(c,d),g2(a,b))&lt;/code&gt;, by just creating the mirrored training element or is something that the machine could figure out on its own?&lt;br&gt;&#xA;While this is a minimum example, what would be different if we were to cover more fruits(5/bundle) from a larger fruit pool(20)?&lt;br&gt;&#xA;Can we use it to predict which bundle is more likely to be picked even if that group/fruit combo hasn't been evaluated in the set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way I thought of was training on the groups, but I think that would result in a flat score for the groups and most likely miss relationships between the groups, such as g1 is preferred over g2 which is preferred over g3 which is preferred over g1? Is this true or would a machine be able to understand that too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was looking a bit at the javascript libraries brain and mind and thought that it might be possible with one of them, but I dont know what type of algorithm/solving method I'm looking for. Any suggestions for libraries to solve this in, preferrably python/php/java/javascript are appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if someone could add suitable tags to the question would be great!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: same as with the group mirroring, how do we distinguish the groups, do we need to sort the fruits alphabetically or can a machine handle it too? (apple, banana) should be identical to (banana,apple)&lt;/p&gt;&#xA;" OwnerUserId="16735" LastEditorUserId="16735" LastEditDate="2016-03-03T20:47:32.403" LastActivityDate="2017-04-29T17:52:11.117" Title="Is it possible to predict groups vs groups in machine learning?" Tags="|machine-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10509" PostTypeId="1" AcceptedAnswerId="10510" CreationDate="2016-03-03T20:29:22.797" Score="7" ViewCount="9860" Body="&lt;p&gt;I would like to train a binary classifier on feature vectors.&#xA;One of the features is categorical feature with string, it is the zip codes of a country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, there is thousands of zip codes, and in my case they are strings.&#xA;How can convert this feature into numerical?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not think that using one-hot-encoding is good as a solution for my case.&#xA;Am I right by saying that? If yes, what would be a suitable solution?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="13891" LastEditDate="2016-03-03T20:33:48.407" LastActivityDate="2019-09-18T15:39:51.347" Title="How to deal with categorical feature of very high cardinality?" Tags="|classification|categorical-data|feature-extraction|feature-construction|" AnswerCount="3" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="10510" PostTypeId="2" ParentId="10509" CreationDate="2016-03-03T20:36:09.343" Score="7" Body="&lt;p&gt;One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be &lt;em&gt;much&lt;/em&gt; wider (millions, billions even), but if you really want you could aggregate area codes into regions, such as states. Of course, you should not use strings, but bit vectors. Two other dimensionality reduction options are &lt;a href=&quot;https://github.com/esafak/mca&quot; rel=&quot;noreferrer&quot;&gt;MCA&lt;/a&gt; (PCA for categorical variables) and &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_projection&quot; rel=&quot;noreferrer&quot;&gt;random projection&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-03T20:36:09.343" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10512" PostTypeId="2" ParentId="10508" CreationDate="2016-03-03T21:39:07.360" Score="0" Body="&lt;p&gt;It's quite hard to understand what your actual goal is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understood you want to calculate optimal fruit bundles from a daily given set of fruits which may not have any fruit in common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to go about would be to just accumulate data how often each fruit is bought and take this as a measure of value for the customer.&#xA;Then you could assume the bundling does not matter. From here you can just split up the fruit baskets by ranking the fruits according to their value and assign every odd element to basket one and every even element to basket two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are really sure the composition of the basket matters because there are interactions between fruits you have to do a different strategy.&#xA;If you can accumulate vast amounts of data, you could just sell every possible combination of baskets and again take the sales as a value measure.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2016-03-03T21:39:07.360" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10514" PostTypeId="2" ParentId="10462" CreationDate="2016-03-03T23:19:01.583" Score="2" Body="&lt;p&gt;The necessary information can be found on &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model_in_computer_vision&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;when we use more technical features such as colour histograms&amp;quot;&#xA;Judging from this sentence I guess you need to understand the &amp;quot;Codebook&amp;quot; generation.&lt;/p&gt;&#xA;&lt;p&gt;First step is to extract features of patches in the image. For efficiency you only want to take patches which are interesting and calculate discriminative features on them. SIFT is one Method which performs both steps for you. It takes care of finding good spots and it calculates features on this spot.&lt;/p&gt;&#xA;&lt;p&gt;Now you can generate your codebook. A codebook will map every possible featurevector ( after all they're just numeric vectors ) to a certain output codeword. One possibility to do this, is to use k-means for codebook generation. After you built your codebook, a vector is mapped to a code by finding the minimal distance to all the entries ( since you used k-means you can use euclidean distance ).&lt;/p&gt;&#xA;&lt;p&gt;Now you have a complete realization of the bag of words model. You can now dive into using it for classification.&#xA;The required algorithms can be implemented using the libraries you mentioned.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastEditorUserId="98307" LastEditDate="2020-08-03T18:11:17.553" LastActivityDate="2020-08-03T18:11:17.553" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10515" PostTypeId="2" ParentId="1100" CreationDate="2016-03-03T23:34:21.943" Score="2" Body="&lt;p&gt;In my opinion you're looking into the wrong methods to solve your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have strictly no numeric data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics based machine learning has a very hard time with such problems.&#xA;Your problem sounds more like one of the problems which should be solved with rule based systems. My first instinct would be to try to understand the rules and put them into code leading to a classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are however methods for learning such rule based systems based on logic. They are only quite unfashionable for machine learning today. &lt;a href=&quot;https://en.wikipedia.org/wiki/Rule-based_system&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Rule-based_system&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2016-03-03T23:34:21.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10516" PostTypeId="2" ParentId="6440" CreationDate="2016-03-03T23:39:18.740" Score="1" Body="&lt;p&gt;Every Blob in caffe can be assigned a nonzero loss weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can have an arbitrary number of outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means you can just learn n different networks on the same data with different targets and assign every loss function it's own weight. Caffe takes care of adding up all the loss.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2016-03-03T23:39:18.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10517" PostTypeId="1" CreationDate="2016-03-03T23:41:57.497" Score="2" ViewCount="168" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Nurw0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Nurw0.png&quot; alt=&quot;These are 7 features in scatter plots across my output variable &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly you can see that there is no linear relationship. If there was one it would be a vertical line which is not that useful. I need some type of regression rather than classification as the output is continuous. The only options I can think of are neural networks and SVR (support vector regressors). Is linear regression a waste of time here? What would you use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use case is - I am trying to predict the rgb color of a polygon based on its area, length, index, complexity, #of lines, # of curves, and intensity (black and white). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this first step I am merely trying to predict the amount of red from 0 to 1. (scaled down from 255).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here are the labels - rgb values: &lt;a href=&quot;http://www.heypasteit.com/clip/2HZX&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.heypasteit.com/clip/2HZX&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is the data with the 7 features: &lt;a href=&quot;http://www.heypasteit.com/clip/2HZY&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.heypasteit.com/clip/2HZY&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16743" LastEditorUserId="16744" LastEditDate="2016-03-04T00:32:37.897" LastActivityDate="2016-03-05T03:56:20.117" Title="If my data looks like this, is linear regression just never gonna be useful?" Tags="|neural-network|regression|linear-regression|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10520" PostTypeId="2" ParentId="1100" CreationDate="2016-03-04T07:23:58.337" Score="2" Body="&lt;p&gt;In line with the above comment, I suggest you try a rule-based approach. For each server you have, query its services. If all of the services are down on a server, then you have a server problem. For each service, if no server reports that the service is running, then you have a problem with the service. In the case that it's both, you'll get notices for each and then be able to go in an inspect what's going on with any of the constituents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost, maintenance, and risk of bad results using an ANN model all exceed the simple, rule-based solution and your boss will probably pat you on the back for just doing what makes sense here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're really serious about keeping your servers and processes functional, I suggest you invest in a APM service that gives you reliable, real-time notifications on what's going on in your production environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case that you're just trying to learn how ANN work - try a different problem. Any well-known dataset for classification or anomaly detection will provide you with a lot more insight into how ANN work than a custom dataset, which can be horribly hard to coerce into an effective learning scheme.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastEditorUserId="12575" LastEditDate="2016-03-07T17:23:56.783" LastActivityDate="2016-03-07T17:23:56.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10521" PostTypeId="1" CreationDate="2016-03-04T08:00:39.950" Score="3" ViewCount="225" Body="&lt;p&gt;Given the machine learning libraries available for many different languages, it's possible to utilise algorithms where you don't need to have a detailed understanding of their application or workings or datascience in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried to find a list of algorithms that are suitable for sequence prediction from a window of previous input values, which I could then apply in such a library e.g. Accord for C# but have failed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each observation I have is an array of 2-dimensonal data in the following form: {MODE1/2/3, int 1-98}. That is, multiple inputs produce multiple outputs. I'd like to examine preceding sets of values from either &lt;code&gt;t-1&lt;/code&gt; or &lt;code&gt;t-x&lt;/code&gt; where x could be a variable amount of preceding value sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I'd like to ask what algorithms excel at prediction from a window of sequence data, and what are their strengths / weaknesses.&lt;/p&gt;&#xA;" OwnerUserId="2861" LastEditorUserId="2861" LastEditDate="2016-03-04T21:42:10.563" LastActivityDate="2016-06-10T15:01:31.150" Title="Output prediction from a sequence of data" Tags="|predictive-modeling|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10522" PostTypeId="1" AcceptedAnswerId="10537" CreationDate="2016-03-04T08:33:36.833" Score="8" ViewCount="11978" Body="&lt;p&gt;I have a dataset with users connection and I want to create a directed network graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nodes: ~20.000&#xA;Edges: ~33.000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used to work before on NodeXL and Gephi, but now I am on Mac and I don't have access on NodeXL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried Gephi and another tool called Cytoscape. Both of them were lagging and I could barely do some statistics. Visualizing the graph was a dream. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any software (or maybe Python library) that can handle such a big network Graph?&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="31513" LastEditDate="2017-04-24T17:38:18.853" LastActivityDate="2018-05-09T08:15:21.933" Title="Visualize graph network with more than 30k edges" Tags="|visualization|graphs|social-network-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10523" PostTypeId="1" CreationDate="2016-03-04T09:32:17.287" Score="42" ViewCount="51577" Body="&lt;p&gt;I have been using neural networks for a while now. However, one thing that I constantly struggle with is the selection of an optimizer for training the network (using backprop). What I usually do is just start with one (e.g. standard SGD) and then try other others pretty much randomly. I was wondering if there's a better (and less random) approach to finding a good optimizer, e.g. from this list:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SGD (with or without momentum)&lt;/li&gt;&#xA;&lt;li&gt;AdaDelta&lt;/li&gt;&#xA;&lt;li&gt;AdaGrad&lt;/li&gt;&#xA;&lt;li&gt;RMSProp&lt;/li&gt;&#xA;&lt;li&gt;Adam&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In particular, I am interested if there's some theoretical justification for picking one over another given the training data has some property, e.g. it being sparse. I would also imagine that some optimizers work better than others in specific domains, e.g. when training convolutional networks vs. feed-forward networks or classification vs. regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any of you have developed some strategy and/or intuition on how you pick optimizers, I'd be greatly interested in hearing it. Furthermore, if there's some work that provides theoretical justification for picking one over another, that would be even better.&lt;/p&gt;&#xA;" OwnerUserId="16753" LastEditorUserId="134" LastEditDate="2018-12-29T02:28:01.330" LastActivityDate="2021-04-16T12:21:48.847" Title="Guidelines for selecting an optimizer for training neural networks" Tags="|neural-network|optimization|backpropagation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10524" PostTypeId="2" ParentId="10502" CreationDate="2016-03-04T09:34:33.680" Score="3" Body="&lt;p&gt;The question has been answered in google groups by Gordon mohr.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Normally there's one read of the corpus to build the vocabulary (which includes initializing the model based on the learned vocabulary size), then any number of extra passes for training. It's only after the one vocabulary-learning scan that word counts are looked at (and compared to &lt;code&gt;min_count&lt;/code&gt; for trimming).&lt;/p&gt;&#xA;&lt;p&gt;If you supply a corpus (as a restartable iterator) as one of the arguments to the initial creation of the Word2Vec model, all these steps are done automatically: one read of the corpus (through the &lt;code&gt;build_vocab()&lt;/code&gt; method) to collect words/counts, then one or more passes (as controlled by the 'iter' parameter and done through the &lt;code&gt;train()&lt;/code&gt; method) for training. Still, only the count for the single pass over the supplied corpus matters for frequency decisions.&lt;/p&gt;&#xA;&lt;p&gt;If you don't supply a corpus at model-initialization, you can then call &lt;code&gt;build_vocab(…)&lt;/code&gt; and &lt;code&gt;train(…)&lt;/code&gt; yourself. It's only what's passed to &lt;code&gt;build_vocab()&lt;/code&gt; that matters for retained frequency counts (and the estimate of corpus size). You can then call &lt;code&gt;train(…)&lt;/code&gt; in other ways, or repeatedly – it just keeps using the vocabulary from the one earlier &lt;code&gt;build_vocab(…)&lt;/code&gt; call.&lt;/p&gt;&#xA;&lt;p&gt;(Note that &lt;code&gt;train(…)&lt;/code&gt; does try to reuse the single-pass corpus size, remembered from the vocab-scanning pass, to give accurate progress-estimates and schedule the decay of the training-rate &lt;code&gt;alpha&lt;/code&gt;. So if you give a different-sized corpus to &lt;code&gt;train(…)&lt;/code&gt;, you should also use its other optional parameters to give it a hint of the size.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-03-04T09:34:33.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10525" PostTypeId="1" CreationDate="2016-03-04T10:03:04.807" Score="2" ViewCount="2546" Body="&lt;p&gt;Assume that a sales rep has actions A, B, C, D.... X that he can perform to influence a physician to write his drug&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict what the next best action a sales rep should take to have the highest impact on a physician&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if a sales rep has already performed A, D, F then i want to predict the next best action for the sales rep to perform to convert the physician to a writer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What models/algorithms are suitable for this type of a problem?&lt;/p&gt;&#xA;" OwnerUserId="16754" LastActivityDate="2020-12-14T13:19:08.183" Title="What models are used to get the Next Best Action to convert a physician from a non writer to a write?" Tags="|bigdata|statistics|predictive-modeling|model-selection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10526" PostTypeId="2" ParentId="10525" CreationDate="2016-03-04T11:42:27.550" Score="1" Body="&lt;p&gt;Expanding on my comment, the simplest predictive model would be to just compute the conversion rates Pr(conversion | action) for each action {A,B,C,D, ... }.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A solution in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;library(tidyr)&#xA;&#xA;set.seed(100) # for reproducibility&#xA;action &amp;lt;- sample(letters[1:5], 100, replace = TRUE) # dummy up sales actions (independent variable)&#xA;converted &amp;lt;- as.logical(rbinom(100, 1, prob = .5))  # dummy up physician conversion (dependent variable)&#xA;&#xA;(df &amp;lt;- data.frame(action = action, converted = converted))&#xA;&#xA;# Compute conversion rates for each action&#xA;conv_df &amp;lt;- df %&amp;gt;% &#xA;  group_by(action, converted) %&amp;gt;% # unique combinations of action by converted&#xA;  summarize(freq = n()) %&amp;gt;% # compute freq of unique combinations&#xA;  spread(converted, freq) %&amp;gt;% # spread rows to cols&#xA;  mutate(rate = `TRUE`/sum(`TRUE`,`FALSE`)) # compute conversion rate&#xA;&#xA;conv_df[order(-conv_df$rate),] #reverse sort&#xA;&#xA;##Source: local data frame [5 x 4]&#xA;##Groups: action [5]&#xA;##&#xA;##  action FALSE  TRUE      rate&#xA;##  (fctr) (int) (int)     (dbl)&#xA;##1      d     8    13 0.6190476&#xA;##2      e     7    11 0.6111111&#xA;##3      a     5     6 0.5454545&#xA;##4      c    13    10 0.4347826&#xA;##5      b    17    10 0.3703704&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The solution above assumes the application is to be some sort of guide for sales reps... if you are wanting to model the distribution of actions in sales contexts, you would need to compute the prevalence of each action as well.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-03-04T11:50:19.057" LastActivityDate="2016-03-04T11:50:19.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10528" PostTypeId="1" CreationDate="2016-03-04T13:00:54.080" Score="2" ViewCount="67" Body="&lt;p&gt;I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values prevents me from using exponential and power law regression lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Zero values are not shown as the Y axis starts from 1, but they are present. My bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the scatterplot. To me it looks like this is some sort of logarithmic trend but I am not sure about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What model will best interpret this trend?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/J2WVX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J2WVX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13481" LastEditorUserId="1330" LastEditDate="2021-05-21T18:52:48.470" LastActivityDate="2021-05-21T18:57:53.570" Title="What model to fit pseudo-logarithmic trend?" Tags="|dataset|regression|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="10529" PostTypeId="1" CreationDate="2016-03-04T16:49:25.550" Score="6" ViewCount="1340" Body="&lt;p&gt;I am doing a project where I have to train a LSTM neural network to generate music. The first step is of course training it. I am using Keras on top of Theano for this task and music21 for feature extraction from MIDI files.&#xA;So far the data I extract from MIDI files has the following format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[[69, -7, 1, 0.75],&#xA; [62, -1, 0, 0.25] ... ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the 4 features that I take from the files. They are then placed in a 3D numpy array with the shape - (number of notes in sequence, 1 (time steps per sample), 4 (features) ).&#xA;I create another array with the expected output for each time step being the data in the next time step. I've built the following model: &lt;a href=&quot;http://pastebin.com/AHCJj1wa&quot; rel=&quot;noreferrer&quot;&gt;http://pastebin.com/AHCJj1wa&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that when it starts training the model I get some nonsensical output. First the epochs' output looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/100&#xA;0s - loss: 15.2986 - acc: 0.9738&#xA;Epoch 2/100&#xA;0s - loss: -3.3534e+02 - acc: 0.8265&#xA;Epoch 3/100&#xA;0s - loss: -3.8183e+02 - acc: 0.8265&#xA;Epoch 4/100&#xA;0s - loss: -3.8218e+02 - acc: 0.8265&#xA;Epoch 5/100&#xA;0s - loss: -3.8256e+02 - acc: 0.8265&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems to be taking 0s per epoch and the loss just becomes larger and larger or stops changing at all (differs between runs).&#xA;Also at the end of the training cycle when I call .predict it prints something like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 0.11846249 -0.13457553  0.08614471 -0.1007532 ]&#xA; [ 0.14596225 -0.17533173  0.10007301 -0.08799653]&#xA; [ 0.15234099 -0.19707087  0.10395571 -0.08419057]&#xA; [ 0.14596225 -0.17533173  0.10007301 -0.08799653]&#xA; [ 0.14340727 -0.1664145   0.0987886  -0.09037785]&#xA; [ 0.14596225 -0.17533173  0.10007301 -0.08799653]&#xA; [ 0.14340727 -0.1664145   0.0987886  -0.09037785]&#xA; [ 0.14802463 -0.17770432  0.10296948 -0.08740482]&#xA; ..... ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think that there is something fundamentally wrong with my model and/or representation of the data. Any ideas on what I can do or at least some pointers to where I can read about this?&lt;/p&gt;&#xA;" OwnerUserId="16764" LastActivityDate="2017-07-26T07:46:14.037" Title="LSTM neural network for music generation" Tags="|machine-learning|python|neural-network|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10531" PostTypeId="1" AcceptedAnswerId="10532" CreationDate="2016-03-04T18:31:17.653" Score="4" ViewCount="1405" Body="&lt;p&gt;My task is to classify free text originated from customer complaints about our product. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created a Taxonomy and have around 10 different categories. I've realized that these categories include keywords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Customer doesn't understand how to use the product&quot;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Keywords: understand, knowledge, know, aware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Record:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training, Customer doesn't understand how to use the product&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using &lt;a href=&quot;https://cloud.google.com/prediction/docs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Prediction API&lt;/a&gt;. When training the model, I would categorize previous text as: &quot;Customer doesn't understand how to use the product&quot; - Training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I add keywords to free text/training data to help the model perform better and provide a better confidence level?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data in training set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training, understand knowledge know aware&#xA;Training, Customer doesn't understand how to use the product&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Right now, I'm adding Keywords into same training data, but looking for a better suggestion.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="29169" LastEditDate="2019-06-09T15:47:14.117" LastActivityDate="2019-06-09T15:47:14.117" Title="Creating training data" Tags="|nlp|data-cleaning|nltk|google-prediction-api|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="10532" PostTypeId="2" ParentId="10531" CreationDate="2016-03-04T19:17:18.713" Score="4" Body="&lt;p&gt;Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is balanced, &lt;a href=&quot;https://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow noreferrer&quot;&gt;stratified sampling&lt;/a&gt;. Assuming sufficient labelled data, ensure that your final training set has a balanced number of text examples for each class/label. Depending on your situation, you may need to over/under sample or somehow deal with the problem of highly imbalanced classes (see &lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;8 tactics to combat imbalanced classes&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest NLP approach to use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag of words&lt;/a&gt; technique, simply indicating the presence/absence of a word in the sentence. Thus each sentence becomes represented as vector of length n, where n = the number of unique words in your data set. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_set &amp;lt;- c(&quot;the big dog&quot;, &quot;the small cat&quot;, &quot;the big and fat cow&quot;)&#xA;words &amp;lt;- strsplit(data_set, split = &quot; &quot;) #tokenize sentences&#xA;words&#xA;##[[1]]&#xA;##[1] &quot;the&quot; &quot;big&quot; &quot;dog&quot;&#xA;##&#xA;##[[2]]&#xA;##[1] &quot;the&quot;   &quot;small&quot; &quot;cat&quot;  &#xA;##&#xA;##[[3]]&#xA;##[1] &quot;the&quot; &quot;big&quot; &quot;and&quot; &quot;fat&quot; &quot;cow&quot;&#xA;&#xA;&#xA;vec &amp;lt;- unique(unlist(words)) #vector representation of sentences&#xA;##[1] &quot;the&quot;   &quot;big&quot;   &quot;dog&quot;   &quot;small&quot; &quot;cat&quot;   &quot;and&quot;  &#xA;##[7] &quot;fat&quot;   &quot;cow&quot; &#xA;&#xA;m &amp;lt;- matrix(nrow = length(data_set), ncol = length(vec))&#xA;&#xA;for (i in 1:length(words)) { #iterate the index of tokenized sentences&#xA;  vec_rep &amp;lt;- vec %in% words[[i]] #create binary word-feature vector&#xA;  m[i,] &amp;lt;- vec_rep #update matrix&#xA;}&#xA;&#xA;df &amp;lt;- data.frame(m, row.names = NULL)&#xA;names(df) &amp;lt;- vec&#xA;df&#xA;##   the   big   dog small   cat   and   fat   cow&#xA;##1 TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE&#xA;##2 TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE&#xA;##3 TRUE  TRUE FALSE FALSE FALSE  TRUE  TRUE  TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sometimes you can increase performance by adding &lt;a href=&quot;https://en.wikipedia.org/wiki/Bigram&quot; rel=&quot;nofollow noreferrer&quot;&gt;bi-gram&lt;/a&gt; and tri-gram features. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##given: &quot;the big dog&quot;&#xA;##unigrams &amp;lt;- {the, big, dog}&#xA;##bigrams &amp;lt;- {the big, big dog}&#xA;##trigrams &amp;lt;- {the big dog}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sometimes weighting words by their frequency improves performance or computing the &lt;a href=&quot;http://blog.christianperone.com/2011/09/machine-learning-text-feature-extraction-tf-idf-part-i/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to increase performance, in my experience, has been custom language feature engineering, especially if the data is from social media sources replete with spelling errors, acronyms, slang, and other word variants. Standard NLP approaches will typically remove &lt;a href=&quot;https://datascience.stackexchange.com/questions/5893/how-to-create-a-good-list-of-stopwords?rq=1&quot;&gt;stop words&lt;/a&gt; (e.g., the, a/an, this/that, etc.) from vector representations (because closed class, high frequency words often don't help discriminate among class/label boundaries). Because vector representations are typically highly dimensional (approximately num of unique words in the corpus/data set), dimensionality reductions techniques can increase performance. For example, one can compute chi-sq, info gain, etc. on a word feature's distribution across classes -- only keep those features/words above some threshold or below some pre-established p value). &lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-03-05T17:27:57.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10533" PostTypeId="2" ParentId="10517" CreationDate="2016-03-05T03:27:53.870" Score="2" Body="&lt;p&gt;I tried running a &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_ransac.html&quot; rel=&quot;nofollow&quot;&gt;RANSAC model&lt;/a&gt; on your data, but got worse results than a straight linear regressor. The ten-fold cross-validated mean absolute error over all three response variables (r,g,b) for the linear model was about 0.37. I also ran a random forest model for comparison, and got about the same score. This suggests that the linear model is not shabby, but it is up to you to decide if either is good enough.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sklearn.linear_model, sklearn.cross_validation, sklearn.ensemble, pandas&#xA;&#xA;labels = pandas.read_csv('labels.csv', header=None, names=['r', 'g', 'b'])&#xA;features = pandas.read_csv('features.csv', header=None, names=['area', 'length', 'index', 'complexity', 'lines', 'curves', 'intensity'])&#xA;&#xA;sklearn.cross_validation.cross_val_score(sklearn.linear_model.LinearRegression(), features, labels[:len(features)], 'mean_absolute_error', 10, -1).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also tried visualizing the data with the outliers clipped, but did not find it very enlightening, so I did not include it here.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-05T03:56:20.117" LastActivityDate="2016-03-05T03:56:20.117" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10534" PostTypeId="1" CreationDate="2016-03-05T08:09:29.460" Score="3" ViewCount="2225" Body="&lt;p&gt;How do we derive/extrapolate the feature map for inhomogeneous polynomial kernels for degree $d$, define as $$K(x, x^\prime) = \bigg( (x \cdot x^\prime) + c\bigg)^d$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know we don't need to know a valid feature map to use the inhomogeneous polynomial kernels, I'm simply curious as to how one would derive/extrapolate it.&lt;/p&gt;&#xA;" OwnerUserId="11044" LastEditorUserId="11044" LastEditDate="2016-03-08T06:59:32.270" LastActivityDate="2016-03-10T03:40:06.340" Title="Deriving Feature Map formula for Inhomogeneous Polynomial Kernel" Tags="|machine-learning|classification|neural-network|svm|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10535" PostTypeId="1" CreationDate="2016-03-05T08:36:11.283" Score="2" ViewCount="95" Body="&lt;p&gt;I have N time varying feature vectors obtained by recording different parameters over time.This results in N*N similarity matrix which contains one to one correlations value for each feature. We need to consider only the upper triangle matrix since it's symmetric, so the correlation value for one feature is low corresponding to all other feature. Is there any learning method that can identify that feature autonomously.&#xA;For Example the matrix looks something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             Feature 1   Feature2 Feature 3 Feature 4&#xA;   Feature1  1           0.91      0.81       0.44&#xA;   Feature2  0.91          1       0.98       0.31&#xA;   Feature3  0.81       0.98       1          0.32&#xA;   Feature4  0.44       0.31       0.32        1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we know Feature 4 is behaving differently . Any learning method that can learn this difference and identify the corresponding feature.Sorry, if this is a very trivial question, I'm new to data analysis.&lt;/p&gt;&#xA;" OwnerUserId="16749" LastEditorUserId="16749" LastEditDate="2016-03-07T03:34:14.810" LastActivityDate="2024-02-28T06:06:00.763" Title="Accept any suggestion to create training data from correlation matrix to find odd one out to identify difference in variation" Tags="|neural-network|clustering|random-forest|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10536" PostTypeId="1" CreationDate="2016-03-05T11:18:28.513" Score="5" ViewCount="230" Body="&lt;p&gt;I am a researcher in deep learning, and have given courses on the theory. Recently I was approached to give the same course adding sections using some very concrete examples of deep learning for companies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My audience will be industry managers never exposed to deep learning before. What are the most easily performed examples I could provide without having to develop millions of lines of code that can be of general interest to managers never exposed to deep learning before? These examples need to be quite concrete and easily applicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, they look for examples that can have the label &quot;big data&quot; slapped on them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know about using ML for detecting digits, or all the other applications on kaggle, but they do not seem to care for those.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2016-03-05T12:59:31.673" LastActivityDate="2016-03-07T21:37:51.123" Title="What are the most concrete and easiest to understand applications of deep learning in the industry?" Tags="|bigdata|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10537" PostTypeId="2" ParentId="10522" CreationDate="2016-03-05T14:18:09.057" Score="6" Body="&lt;p&gt;Yes There are!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;noreferrer&quot;&gt;Networkx&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I think 20k-30k node-edge would be OK on Networkx, IF YOU HAVE A GOOD MACHINE! Networkx is a great library in Python particularly for Graph Analysis so you have access to great analysis tools beside visualizing but visualizing 20k vertices needs much RAM and takes long.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://igraph.org/redirect.html&quot; rel=&quot;noreferrer&quot;&gt;Igraph&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Igraph is another great tool for Graph Analysis with APIs in Python, R and C. You can do a lot with Igraph library including nice plotting facilities.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://snap.stanford.edu/&quot; rel=&quot;noreferrer&quot;&gt;SNAP&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;SNAP was developed by Jure Leskovec and his colleagues at Stanford. A large scale graph processing library natively written in C with API in Python. This library can handle huge graphs on a single personal machine.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;PS.1&lt;/strong&gt; Both of these guys have interfaces with different graph softwares like Gephi, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;PS.2&lt;/strong&gt; For large graphs I strongly recommend to work in C++ if you can. Speaking as a Network/Graph guy, C++ solves many problems out there!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-03-05T14:18:09.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10538" PostTypeId="1" CreationDate="2016-03-05T17:43:06.277" Score="1" ViewCount="135" Body="&lt;p&gt;I have an &lt;a href=&quot;http://www.imdb.com/title/tt0472954/epdate&quot; rel=&quot;nofollow&quot;&gt;average rating of all votes&lt;/a&gt; as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) multiply each average rating by the total votes cast for that episode&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) take the array of all those values and normalize it so the lowest is 0 and the highest is 10&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) use that new value as the actual rating&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2016-03-05T17:43:06.277" Title="What's the best way to rank aggregate imdb rating data?" Tags="|dataset|data-cleaning|ranking|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10539" PostTypeId="2" ParentId="10536" CreationDate="2016-03-05T18:37:21.273" Score="3" Body="&lt;p&gt;This seems like a matter of opinion to me, but here are some applications of machine learning that you seen &lt;em&gt;very&lt;/em&gt; often in Silicon Valley:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Advertising&lt;/strong&gt;: what to show whom and when? The data comes fast and furious so the engineering part is quite sophisticated.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Recommendation&lt;/strong&gt;: similar to the advertising problem, except there are no advertisers, and the volume is slower, so you can afford to use more sophisticated machine algorithms.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Security&lt;/strong&gt;: did somebody outside penetrate your system? did somebody inside abuse your system?&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Text/audio/video processing&lt;/strong&gt;: information retrieval on media. Requires finding good representations of the items using machine learning; e.g., fetch all pictures similar to the input, where the similarity can be construed in various ways, such as similarity of texture, style, color, content, etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are not necessarily simple, but they're representative. If you want to give a demo, I suggest the last one. You could show them how to find similar images.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-05T18:37:21.273" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10540" PostTypeId="1" AcceptedAnswerId="10586" CreationDate="2016-03-05T19:07:07.393" Score="7" ViewCount="957" Body="&lt;p&gt;I'm using scikit-learn to do a genome-wide association study with a feature vector of about 100K SNPs. My goal is to tell the biologists which SNPs are &quot;interesting&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.RandomizedPCA.html&quot;&gt;RandomizedPCA&lt;/a&gt; really improved my models, but I'm having trouble interpreting the results. Can scikit-learn tell me which features are used in each component? &lt;/p&gt;&#xA;" OwnerUserId="15877" LastEditorUserId="381" LastEditDate="2016-03-05T19:23:22.780" LastActivityDate="2016-03-08T02:34:58.633" Title="Interpreting the results of randomized PCA in scikit-learn" Tags="|feature-selection|scikit-learn|pca|randomized-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10541" PostTypeId="1" CreationDate="2016-03-05T20:56:59.990" Score="2" ViewCount="1884" Body="&lt;p&gt;Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this project where I'm trying to design a system to predict which &quot;store&quot; a user is going to buy a given item from. However, the set of stores a user can potentially buy from is already known for each user (because this promotion only works at limited stores they signed up for). On average, the set of stores the average user can buy from is around 3 but the # of distinct stores across all users is about 10000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory, for a single user this seems like a simple classification problem. We have historical information such as the time/day/month the user bought item Y from store X along with other features such as location (postal code) where the user lives and features related to the type of item they are buying (cost, weight, brand).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the issue is that there's currently around 6000 users so going with this approach seems like I would need a separate model for each user but that doesn't seem like an efficient solution to me or at least not how I've generally seen ML algorithms used. Unfortunately, I don't see any other way I could take into account the fact that the set of stores a user can already buy from is known for each user already. I suppose I could have a categorical variable for each buy-able store as a feature but then that would be equivalent to adding 10000 features and I'm not sure if this would scale as the number of distinct stores increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be really helpful if anyone has any insight on how to apply machine learning techniques to this kind of problem in general as this is sort my first time working on a &quot;real&quot; problem. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="16788" LastEditorUserId="16788" LastEditDate="2016-03-05T22:27:34.947" LastActivityDate="2017-01-19T09:43:24.863" Title="Multiclass classification with large number of classes but for each user the set of target classes is known" Tags="|machine-learning|classification|dataset|multiclass-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10542" PostTypeId="1" CreationDate="2016-03-05T21:29:48.683" Score="4" ViewCount="148" Body="&lt;p&gt;I am part of a small team data analysts that has been trying to optimise the process we use to keep track of the assumptions and design choices that are made over the lifetime of a complex data science project (several months) - ultimately we want to (i) catch (conceptual) errors when they are made (ii) ensure that it is at all times clear what assumptions underlie the results being produced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our current process relies on code-generated and manually maintained excel spreadsheets - the process is (i) time consuming/results in too many files, (ii) introduces a new source of error (the copying of results into the spreadsheet) and (iii) is not dynamic, i.e. cannot easily be adapted as a project evolves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While there are a large number of questions/answers focused on version control, reproducible research and project organisation more broadly we are not looking for a new suite of tools but a light-weight process that addresses a very specific problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ideal process: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is easy to  maintain &amp;amp; as much as possible automated to reduce&#xA;errors&lt;/li&gt;&#xA;&lt;li&gt;Captures all design choices and assumptions made&lt;/li&gt;&#xA;&lt;li&gt;At a glance reflects the impact of any changes, i.e. reflects (changes&#xA;in) the 'key metrics' that resulted from the change&lt;/li&gt;&#xA;&lt;li&gt;Links all changes with the related code file versions&lt;/li&gt;&#xA;&lt;li&gt;Can easily be shared with other analysts &amp;amp; Principal Investigators&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16791" LastActivityDate="2017-03-24T01:54:50.313" Title="How to keep track of assumptions/design choices and avoid errors in complex projects" Tags="|r|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10544" PostTypeId="1" AcceptedAnswerId="11000" CreationDate="2016-03-06T04:51:12.607" Score="1" ViewCount="803" Body="&lt;p&gt;I want to use R for implementing a fuzzy inference system. There are 4 input variables and one output. Each rule is dependent on all input variables and based on there membership the output class is decided. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest a good library and source examples to begin with? The library that have good visualisation support also.&#xA;Should I go with 'frbs' R package?&lt;/p&gt;&#xA;" OwnerUserId="13046" LastActivityDate="2017-03-29T00:12:18.537" Title="Fuzzy Inference system in R" Tags="|machine-learning|r|fuzzy-logic|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10545" PostTypeId="1" CreationDate="2016-03-06T08:17:18.480" Score="4" ViewCount="386" Body="&lt;p&gt;I am trying out POS tagging using RNN but not able to figure out what's wrong in my implementation because of which the gradient check is failing. Please help. I am pasting the relevant part below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# weights&#xA;Wxh = randn(inputLayerSize, hiddenLayerSize)*0.01; # input to hidden&#xA;Whh = randn(hiddenLayerSize, hiddenLayerSize)*0.01; # hidden to hidden&#xA;Bh = zeros(1, hiddenLayerSize); # hidden bias&#xA;Why = randn(hiddenLayerSize, outputLayerSize)*0.01; # hidden to output&#xA;By = zeros(1, outputLayerSize); # output bias&#xA;&#xA;function forwardRNN(x, y, h, p, hPrev)&#xA;  global Wxh, Whh, Why, Bh, By;&#xA;  cost = 0;&#xA;  for time in 1:length(x)&#xA;    if time == 1&#xA;      h[time] = tanh(x[time]'*Wxh + hPrev*Whh .+ Bh);&#xA;    else&#xA;      h[time] = tanh(x[time]'*Wxh + h[time-1]*Whh .+ Bh);&#xA;    end&#xA;    score = h[time]*Why .+ By;&#xA;    p_softmax = exp(score) / sum(exp(score));&#xA;    p[time] = vec(p_softmax);&#xA;    cost += -sum(log(y[time]'*p[time]));&#xA;  end&#xA;  return cost;&#xA;end&#xA;&#xA;function backwardRNN(x, y, h, p, hPrev)&#xA;  global Wxh, Whh, Why, Bh, By;&#xA;  dWxh = zeros(size(Wxh));&#xA;  dWhh = zeros(size(Whh));&#xA;  dBh = zeros(size(Bh));&#xA;  dWhy = zeros(size(Why));&#xA;  dBy = zeros(size(By));&#xA;  dh = zeros(size(Bh));&#xA;  dhnext = zeros(size(h[1]));&#xA;  for time in length(x):-1:1&#xA;    dy = p[time] - y[time];&#xA;    dWhy = dWhy + (dy * h[time])';&#xA;    dBy = dBy + dy';&#xA;    dh = (Why*dy)' + dhnext;&#xA;    dhRaw = (1 - (h[time].*h[time])) .* dh;&#xA;    dWxh = dWxh + (x[time] * dhRaw);&#xA;    dBh = dBh + dhRaw;&#xA;    if time == 1&#xA;      dWhh = dWhh + (hPrev' * dhRaw);&#xA;    else&#xA;      dWhh = dWhh + (h[time-1]' * dhRaw);&#xA;    end&#xA;    dhnext = dhRaw*Whh;&#xA;  end&#xA;  return dWxh, dWhh, dBh, dWhy, dBy;&#xA;end&#xA;&#xA;# gradient checking&#xA;function gradCheck(inputs, targets, h, p, hPrev)&#xA;  paramNameList = [&quot;Wxh&quot;, &quot;Whh&quot;, &quot;Bh&quot;, &quot;Why&quot;, &quot;By&quot;];&#xA;  global Wxh, Whh, Why, Bh, By;&#xA;  paramList = [x for x=(Wxh, Whh, Bh, Why, By)];&#xA;  num_checks = 2;&#xA;  delta = 1e-5;&#xA;  cost = forwardRNN(inputs, targets, h, p, hPrev);&#xA;  dWxh, dWhh, dBh, dWhy, dBy = backwardRNN(inputs, targets, h, p, hPrev);&#xA;  dParamList = [x for x=(dWxh, dWhh, dBh, dWhy, dBy)];&#xA;  for (param,dparam,name) in zip(paramList, dParamList, paramNameList)&#xA;    s0 = size(dparam);&#xA;    s1 = size(param);&#xA;    if s0 != s1&#xA;      println(&quot;Error dims dont match: &quot;, s0,&quot; and &quot;,s1);&#xA;    end&#xA;    println(name)&#xA;    for i in 1:num_checks&#xA;      ri = rand(1:length(param));&#xA;      old_val = param[ri];&#xA;      param[ri] = old_val + delta;&#xA;      cg0 = forwardRNN(inputs, targets, h, p, hPrev);&#xA;      param[ri] = old_val - delta;&#xA;      cg1 = forwardRNN(inputs, targets, h, p, hPrev);&#xA;      param[ri] = old_val&#xA;      grad_analytic = dparam[ri];&#xA;      grad_numerical = (cg0 - cg1) / ( 2 * delta );&#xA;      rel_error = abs(grad_analytic - grad_numerical) / abs(grad_numerical + grad_analytic);&#xA;      println(grad_numerical,&quot;, &quot;, grad_analytic, &quot; =&amp;gt; &quot;,rel_error);&#xA;      if rel_error &amp;gt; 1e-5&#xA;        error(&quot;Gradient check failed.&quot;);&#xA;      end&#xA;      println(&quot;Gradient check passed.&quot;)&#xA;    end&#xA;  end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code is in Julia programming language and is inspired from Karpathy's &lt;a href=&quot;https://gist.github.com/karpathy/d4dee566867f8291f086&quot; rel=&quot;nofollow&quot;&gt;min-char-rnn.py&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12250" LastEditorUserId="11097" LastEditDate="2016-03-08T06:52:40.477" LastActivityDate="2020-08-07T07:24:09.640" Title="Gradient Check is failing for RNN" Tags="|machine-learning|nlp|deep-learning|julia|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10546" PostTypeId="2" ParentId="10541" CreationDate="2016-03-06T09:52:06.700" Score="2" Body="&lt;p&gt;so it sounds like you have historical transaction data. If you do, you probably want to train a model using store label as your dependent variable, and all the others independent variables. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(100)&#xA;items &amp;lt;- c(&quot;milk&quot;, &quot;cheese&quot;, &quot;steak&quot;, &quot;apple&quot;, &quot;eggs&quot;)    &#xA;df &amp;lt;- data.frame(trans_id = 1:100, &#xA;                       cust_id = sample(1:10, 100, replace = TRUE),&#xA;                       item = sample(items, 100, replace = TRUE)&#xA;                       )&#xA;&#xA;df$store &amp;lt;- as.factor(sapply(df$cust_id, function(x) {&#xA;    sample(letters[cust_id:(cust_id+2)],1)&#xA;  }))&#xA;&#xA;head(df)&#xA;##  trans_id cust_id   item store&#xA;##1        1       4 cheese     e&#xA;##2        2       3 cheese     f&#xA;##3        3       6   milk     f&#xA;##4        4       1 cheese     f&#xA;##5        5       5  steak     d&#xA;##6        6       5  apple     f&#xA;&#xA;#train General linear model&#xA;glm(formula = store ~., family = binomial(link = &quot;logit&quot;), &#xA;     data = df)&#xA;&#xA;##all:  glm(formula = store ~ ., family = binomial(link = &quot;logit&quot;), data = df)&#xA;##&#xA;##Coefficients:&#xA;##(Intercept)     trans_id      cust_id   itemcheese  &#xA;##   1.320095    -0.007399    -0.066363     0.024632  &#xA;##   itemeggs     itemmilk    itemsteak  &#xA;##   0.084018     0.047022     1.069399  &#xA;&#xA;library(&quot;e1071&quot;)&#xA;#train naive bayes&#xA;naiveBayes(formula = store ~., data = df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Depending on business constraints you may or may not want to include customer id as a predictor variable (include if you will only be predicting stores for a finite known pool of customers; don't include if your model needs to generalize across customers). Similarly, if this were a real data set, you probably wouldn't include a transaction id as a predictor variable, because it won't generalize to new cases (transaction ids are unique)&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-03-06T10:28:18.203" LastActivityDate="2016-03-06T10:28:18.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10547" PostTypeId="1" CreationDate="2016-03-06T10:33:09.410" Score="2" ViewCount="109" Body="&lt;p&gt;I'm training a neural network for predicting the location of a single object, so the prediction consists of 4 values: x, y, width and height. Without weight decay, the loss starts at around 0.3. If I add weight decay with a factor of 0.004, the loss jumps to around 22. Does this mean the network will give more importance to minimizing the weights over finding good coordinates? How important is the balance between the components of a loss function?&lt;/p&gt;&#xA;" OwnerUserId="15234" LastActivityDate="2016-06-04T15:52:12.630" Title="How important is the balance between the components of a loss value?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10548" PostTypeId="2" ParentId="10457" CreationDate="2016-03-06T11:11:38.457" Score="3" Body="&lt;p&gt;A most interesting paper! Following &lt;em&gt;Opper and Haussler&lt;/em&gt;&lt;sup&gt;$\dagger$&lt;/sup&gt;, the authors define a &lt;em&gt;version space&lt;/em&gt;; the set of unit vectors which separate the training samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal V \equiv \{ \mathtt w | y_i f(\mathtt x_i) &amp;gt;0 , i= 1, \ldots, n, \| \mathtt w \|_2 = 1 \}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that we are dealing with a classification problem where $y_i \in \{-1, +1\}$, so we want our classifier $f(x_i)$ to have the same sign as $y_i$. It might help to recall that $\mathtt w = \sum_i \alpha_i \phi(\mathtt x_i)$ and $f(\mathtt x) = \left&amp;lt; \mathtt w, \phi(\mathtt x) \right&amp;gt; = \sum_i \alpha_i k(\mathtt x_i, \mathtt x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally $y_i \left( \left&amp;lt; \mathtt w , \phi(\mathtt x_i) \right&amp;gt; + b \right) \ge 1$. What they've done is to set $b=0$ and eliminate the margin (RHS). The length constraint is to ensure uniqueness.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The version space is illustrated as a region on the sphere as shown in&#xA;  Figs. 5 and 6. If the version space is shaped as in Fig. 5, the SVM&#xA;  solution is near to the optimal point. However, if it has an elongated&#xA;  shape as in Fig. 6, the SVM solution is far from the optimal one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The reason there is an inscribed sphere is that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the SVM solution coincides with the Tchebycheff-center of the version&#xA;  space, which is the center of the largest sphere contained&#xA;  in $\mathcal V$. However, the theoretical optimal point in version&#xA;  space yielding a Bayes-optimal decision boundary is the&#xA;  Bayes point, which is known to be closely approximated by&#xA;  the center of mass.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In other words, they are saying that the SVM classifier is not always Bayes optimal. Please see the references for proof. The hypersphere comes about when you consider the margin $\gamma \equiv \min y_i f(\mathtt x_i)$. The support vectors are then the boundaries of the version space which are tangent to the hypersphere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A cited paper* follows this idea to develop an algorithm to actually estimate the center of mass &quot;by averaging over the trajectory of a billiard ball bouncing in version space.&quot; If you're interested in diagnosing SVM failures, perhaps it would help to read that paper too, since it claims to offer a better algorithm instead.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;$\dagger$ M. Opper and D. Haussler, “Generalization performance of Bayes optimal&#xA;classification algorithm for learning a perceptron,” &lt;em&gt;Phys. Rev. Lett.&lt;/em&gt;,&#xA;vol. 66, p. 2677, 1991.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* T. Graepel, R. Herbrich, and C. Campbell, “Bayes point machines: Estimating&#xA;the bayes point in kernel space,” in Proc. IJCAI Workshop Support Vector Machines, 1999, pp. 23–27&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-06T11:11:38.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10550" PostTypeId="1" CreationDate="2016-03-06T14:08:50.243" Score="4" ViewCount="231" Body="&lt;p&gt;I have a list of email subjects like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;XYZ&amp;gt; commented on &amp;lt;ABC&amp;gt;&#xA;Weekly review for &amp;lt;Company&amp;gt;&#xA;Your account is ready &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to find patterns in them so I can group them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a well known algorithm I can use? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Preferably with wide language implementations or easy re-implementation.&lt;/li&gt;&#xA;&lt;li&gt;The algorithm should be unsupervised.&lt;/li&gt;&#xA;&lt;li&gt;The number of different emails is not known.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Update:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I can break this down into two problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Group subjects by the similar words they use, resulting in the following. Each group should be very distinct from the rest (they should be almost perfectly exclusive) and the algorithm should give relatively small number of groups with good length of the common words.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[commented, on]&#xA;[weekly, review]&#xA;[your, account, is, ready]&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once grouped, it should be easy to find a state automaton that accepts only the group's subject and thus eliminates variable&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I can go back and check if there are any intersections and tweak the variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Having said that, is it better to use a completley different approach like neural nets maybe? I have zero experience with those, but if it makes more sense, I am open to learning.&lt;/p&gt;&#xA;" OwnerUserId="16808" LastEditorUserId="29169" LastEditDate="2019-06-07T17:25:34.467" LastActivityDate="2019-06-07T17:25:34.467" Title="How would you categorize email subjects to find similar emails?" Tags="|text-mining|algorithms|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10551" PostTypeId="2" ParentId="10547" CreationDate="2016-03-06T15:20:10.303" Score="3" Body="&lt;p&gt;The cost function controls the algorithm completely - the new regularization from weight decay is likely responsible for the jump in loss. If you only added a small amount of regularization and the loss ballooned, you were likely very overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is &lt;a href=&quot;https://www.cs.cmu.edu/~schneide/tut5/node42.html&quot; rel=&quot;nofollow&quot;&gt;cross validation.&lt;/a&gt; The only way to know the optimal amount if regularization is to see what does the best on held-out data. &lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2016-03-06T15:20:10.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10552" PostTypeId="1" CreationDate="2016-03-06T15:46:50.050" Score="5" ViewCount="230" Body="&lt;p&gt;I have an hypothesis but i don't know if it's true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the cluster is dense and we apply a supervised learning on this data, the model generated by this cluster will be more efficient for new data falling into this cluster than other.&#xA;Thus we have multiple cluster with associate models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How choose the right associate model to optimize our prediction ?&#xA;We can compare the value of the point to cluster's mean and apply the right model !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions ? &lt;/p&gt;&#xA;" OwnerUserId="9257" LastActivityDate="2016-08-04T22:25:37.160" Title="Predictive clustering" Tags="|clustering|supervised-learning|semi-supervised-learning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10553" PostTypeId="2" ParentId="10550" CreationDate="2016-03-06T15:54:29.363" Score="0" Body="&lt;p&gt;You can associate the right semantic to each letter, start with the definition of the semantic. Build your model from scratch, use rigor to moove on forward.&#xA;At this point you have the semantic of letter but which is the semantic of a word ? Can we reduce it to something more dense ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was jocking but not :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is, how can you categorise data ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to put rules/limits to define something.&#xA;If you want your work works for a mean interpretation, thus use the mean meaning of each letter/word/sentance/pararagraph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can make disjonctions, for example, good(0) bad(1) =&gt; lucky = 0 , unlucky = 1&#xA;But you can say that be lucky for a moment is be unlucky later thus lucky = 1&#xA;Thus you have to put hard limits and build on it, you have 2 options, look backward or moove forward !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the hope to be helpful !  &lt;/p&gt;&#xA;" OwnerUserId="9257" LastActivityDate="2016-03-06T15:54:29.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10554" PostTypeId="2" ParentId="989" CreationDate="2016-03-06T16:05:46.777" Score="8" Body="&lt;p&gt;With such a huge dataset I think you'd be better off using a neural network, deep learning, random forest (they are surprisingly good), etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned in earlier replies, the time taken is proportional to the third power of the number of training samples. Even the prediction time is polynomial in terms of number of test vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really must use SVM then I'd recommend using GPU speed up or reducing the training dataset size. Try with a sample (10,000 rows maybe) of the data first to see whether it's not an issue with the data format or distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned in other replies, linear kernels are faster.&lt;/p&gt;&#xA;" OwnerUserId="16812" LastEditorUserId="16812" LastEditDate="2018-01-11T22:10:38.190" LastActivityDate="2018-01-11T22:10:38.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10555" PostTypeId="1" AcceptedAnswerId="10563" CreationDate="2016-03-06T16:36:33.270" Score="3" ViewCount="1354" Body="&lt;p&gt;I am using scikit-learn's SVM for the &lt;a href=&quot;https://www.kaggle.com/c/digit-recognizer/data&quot; rel=&quot;nofollow&quot;&gt;MNIST digit classification dataset&lt;/a&gt;. In order to improve the performance I extended the dataset by adding rotated samples. I was aware that SVM takes O(N^3) time to train the data, where N is the number of training vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However even prediction seems to take increase polynomially, the number of test vectors is the same. Is there any explanation for this or some equation that relates prediction time to the number of training samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a 3rd degree polynomial as the kernel with C=100.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: I am doing a group project to compare the performance of various methods so I can't use any other method as my teammates would have used those. I referred to &lt;a href=&quot;http://www.cs.berkeley.edu/~malik/cs294/decoste-scholkopf.pdf&quot; rel=&quot;nofollow&quot;&gt;a paper by Decoste and Scholkoph&lt;/a&gt; which uses Virtual SVM. However I don't think I can run this on my current system if I can't run a simple extended training set.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="16812" LastActivityDate="2016-03-06T19:15:42.447" Title="SVM prediction time increase with number of test cases" Tags="|svm|scikit-learn|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10556" PostTypeId="2" ParentId="10552" CreationDate="2016-03-06T17:56:36.010" Score="0" Body="&lt;p&gt;If the cluster is nice and dense, your classifier on all of the data will work just fine. Relying on a clustering is risky.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, the classifier in a cluster may get easier, but overall I would expect the error to increase.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-03-06T17:56:36.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10557" PostTypeId="1" AcceptedAnswerId="10569" CreationDate="2016-03-06T17:57:54.723" Score="11" ViewCount="4683" Body="&lt;p&gt;When trying to do e.g. a classification, my approach currently is to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;try out various algorithm first and benchmark them&lt;/li&gt;&#xA;&lt;li&gt;perform feature selection on the best algorithm from 1 above&lt;/li&gt;&#xA;&lt;li&gt;tune the parameters using the selected features and algorithm&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, I often cannot convince myself that there may be a better algorithm then the selected one, if the other algorithms have been optimized with the best parameter / most suitable features. At the same time, doing a search across all algorithms * parameters * features are just too time-consuming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion on the right approach / sequence?&lt;/p&gt;&#xA;" OwnerUserId="1133" LastEditorUserId="21" LastEditDate="2016-04-14T10:23:59.853" LastActivityDate="2023-02-25T05:52:21.990" Title="Which one first: algorithm benchmarking, feature selection, parameter tuning?" Tags="|feature-selection|parameter-estimation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10558" PostTypeId="1" CreationDate="2016-03-06T18:33:58.750" Score="3" ViewCount="1967" Body="&lt;p&gt;What are the basic principles/tools necessary to make something like Alexa utterance parsing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, Alexa allows a designer to define phrases with &quot;placeholders&quot; that will be filled in. For example, the phrase &quot;what is the horoscope for gemini&quot; would match the underlying model (below), and return &lt;code&gt;Sign=gemini&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;what's the horoscope for {Sign}&#xA;what is the horoscope for {Sign}&#xA;give me the horoscope for {Sign}&#xA;tell me the horoscope for {Sign}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To clarify: I am interested in the theory behind how the language model and parsing (with regard to theory and algorithms) work, so I can build my own version.&lt;/p&gt;&#xA;" OwnerUserId="16816" LastEditorUserId="16816" LastEditDate="2016-03-06T22:30:39.167" LastActivityDate="2017-10-09T18:00:16.267" Title="How does Alexa utterance parsing work?" Tags="|nlp|language-model|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10560" PostTypeId="1" CreationDate="2016-03-06T10:59:19.857" Score="1" ViewCount="1205" Body="&lt;p&gt;I have a list of email subjects like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;XYZ&amp;gt; commented on &amp;lt;ABC&amp;gt;&#xA;Weekly review for &amp;lt;Company&amp;gt;&#xA;Your account is ready &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to find patterns in them so I can group them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a well known algorithm I can use? &lt;/p&gt;&#xA;" OwnerUserId="16808" OwnerDisplayName="hakunin" LastActivityDate="2016-03-06T19:04:44.247" Title="How would you categorise email subjects to find similar emails?" Tags="|algorithms|machine-learning|" AnswerCount="1" CommentCount="5" ClosedDate="2016-03-06T21:31:26.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="10561" PostTypeId="2" ParentId="10560" CreationDate="2016-03-06T18:53:08.607" Score="3" Body="&lt;p&gt;You'll probably have to experiment a bit with different approaches.  Let me outline two different kinds of approaches you could try.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Topic modelling&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You could try applying unsupervised topic modelling to your subject lines.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;LDA&lt;/a&gt; is probably the most widely used method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topic modelling tries to find a limited number of &quot;topics&quot;, and assigns each subject line to one or more &quot;topics&quot; based on the words in the subject line.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Clustering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You could try using clustering.  Broadly, you would find some way map each subject line to a feature vector, and then apply some unsupervised clustering method.  There are many options for each of those two steps.  To get feature vectors, you might try any of the existing &lt;a href=&quot;https://en.wikipedia.org/wiki/Word_embedding&quot; rel=&quot;nofollow&quot;&gt;word embeddings&lt;/a&gt;; e.g., you might try &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt;.  For clustering, there are many, many &lt;a href=&quot;https://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow&quot;&gt;clustering algorithms&lt;/a&gt;; e.g., you might try k-means.  I recommend you do a little reading on these topics and then experiment with them a bit.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Caveats&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A warning.  Don't set your expectations too high. Subject lines are typically very short, and that will make it hard for these techniques to find clusterings.  In other words, you are operating in a regime that's known to be hard for existing NLP and ML techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can find any other features that might assist with clustering similar emails (e.g., the identify of the sender? the mailing list it was sent on, if any?), including that information might improve the quality of the clusters.&lt;/p&gt;&#xA;" OwnerUserId="8560" OwnerDisplayName="D.W." LastActivityDate="2016-03-06T18:53:08.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10562" PostTypeId="1" CreationDate="2016-03-06T19:08:40.770" Score="1" ViewCount="265" Body="&lt;p&gt;I am trying to determine a good tool that will help me generate a probability of a sale for a list of of 300,000 products. I have a table of historical sales data (with about 300,000 records) that contains around 10 continuous variables along with a dependent variable that has a yes/no (i.e., binary outcome) value indicating whether product in the list has had a sale in the past 12 months. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The historical data essentially looks like this. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Product1,2,3 etc&#xA;Variable 1&#xA;Variable 2 &#xA;Variable 3&#xA;Variable 4&#xA;Variable 5&#xA;Variable 6&#xA;Variable 7&#xA;Variable 8&#xA;Variable 9&#xA;Variable 10&#xA;Sold in past 12 months (Yes or No)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last variable in the list is of course the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I want to do is to find a tool that is going to be the best or easiest to use, so that I can assign a probability to each product in the list, essentially giving me the chance to condense my list to the products that are the highest likelihood to generate a sale, so that I can list those products instead of the others that have lower probability of generating a sale. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, the tool could do a quick logistic regression, or some other probability calculation based on the available variables, and thereby give me a (RVU-like) number (perhaps a probability ranging from 0 to 1) for each product, allowing me to quickly select the top 50,000 products to list on a website, since they have the higher probability of generating a sale according to the available variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am of course assuming that the variables are somehow correlated to the outcome, but perhaps the tool will help me determine that. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does anyone have any suggestions of a good tool to accomplish this? I would presume that there is a simple way to set this up in Microsoft Excel, but if not, then a piece of software that does this would of course be great too. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am also open to suggestions as to which type of regression analysis (or other analysis) is the best to accomplish this. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks for any suggestions. &lt;/p&gt;&#xA;" OwnerUserId="16817" LastEditorUserId="381" LastEditDate="2016-03-06T19:49:38.037" LastActivityDate="2016-03-07T17:37:17.993" Title="Best tool for analyzing inventory that has binary sales outcome of yes or no" Tags="|regression|logistic-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10563" PostTypeId="2" ParentId="10555" CreationDate="2016-03-06T19:15:42.447" Score="3" Body="&lt;p&gt;The number of support vectors must be increasing. The prediction time is proportional to that; after all, the kernel classifier is $f(x) = \sum_i \alpha_i k(x, x_i)$, where the summation is over the support vectors. With sklearn you can find out how many you have by inspecting &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;n_support_&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-06T19:15:42.447" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10564" PostTypeId="1" CreationDate="2016-03-06T19:58:23.093" Score="1" ViewCount="82" Body="&lt;p&gt;I'm implementing a CNN with Numpy and Scipy to solidify my understanding, but I'm encountering a rather strange problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My layers are: input -&gt; conv -&gt; pool -&gt; FC&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the relevant functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def unpackTheta (theta, filterSize, numFilters, pooledDim, outputSize):&#xA;    a = filterSize ** 2 * numFilters&#xA;    b = a + numFilters&#xA;    c = b + pooledDim ** 2 * numFilters * outputSize&#xA;    d = c + outputSize&#xA;    return (np.reshape(theta[0 : a], (filterSize, filterSize, numFilters)),&#xA;            np.reshape(theta[a : b], (numFilters)),&#xA;            np.reshape(theta[b : c], (pooledDim ** 2 * numFilters, outputSize)),&#xA;            np.reshape(theta[c : d], (outputSize)))&#xA;&#xA;def getConvolution (features, Wc, Bc, numFilters, numImages, filteredSize):&#xA;    conv = np.zeros((filteredSize, filteredSize, numFilters, numImages))&#xA;    for imageNum in range(numImages):&#xA;        image = features[:, :, imageNum]&#xA;        for filterNum in range(numFilters):&#xA;            filterLayer = np.rot90(Wc[:, :, filterNum], 2)&#xA;            conv[:, :, filterNum, imageNum] = sigmoid(convolve2d(image, filterLayer, 'valid') + Bc[filterNum])&#xA;    return conv&#xA;&#xA;def getPooled (features, poolSize, pooledSize, filteredSize, numFilters, numImages):&#xA;    ret = np.empty((pooledSize, pooledSize, numFilters, numImages))&#xA;    poolFilter = np.ones((poolSize, poolSize)) / poolSize ** 2&#xA;    for i in range(numFilters): &#xA;        for j in range(numImages):&#xA;            pooledLayer = convolve2d(features[:, :, i, j], poolFilter, 'valid')&#xA;            for x in range(0, filteredSize, poolSize):&#xA;                for y in range(0, filteredSize, poolSize):&#xA;                    ret[x / poolSize, y / poolSize, i, j] = pooledLayer[x, y]&#xA;    return ret&#xA;&#xA;def getCost (theta, *args) :&#xA;    ## Initializing variables&#xA;    features, inputSize, filterSize, numFilters, poolSize, outputSize, reg = args&#xA;    filteredSize = inputSize - filterSize + 1&#xA;    pooledSize = filteredSize / poolSize&#xA;    (Wc, Bc, Wr, Br) = unpackTheta(theta, filterSize, numFilters, pooledSize, outputSize)&#xA;    numImages = np.size(features, 0)&#xA;&#xA;    actualValues = np.eye(outputSize)[features[:, 0]]&#xA;    features = np.reshape(features[:, 1:] / 255.0, (inputSize, inputSize, numImages))&#xA;&#xA;    ## Forward propagation&#xA;    # Convolution&#xA;    conv = getConvolution(features, Wc, Bc, numFilters, numImages, filteredSize)&#xA;&#xA;    # Mean pooling&#xA;    pooled = np.reshape(getPooled(conv, poolSize, pooledSize, filteredSize, numFilters, numImages), (-1, numImages))&#xA;&#xA;    # Logistic regression&#xA;    calcValues = sigmoid(pooled.transpose().dot(Wr) + Br)&#xA;&#xA;    ## Calculating cost&#xA;    cost = -np.sum(actualValues * np.log(calcValues) + (1 - actualValues) * np.log(1 - calcValues)) / numImages&#xA;    cost += reg * (np.sum(Wc ** 2) + np.sum(Wr ** 2)) / (2 * numImages)&#xA;&#xA;    ## Back propagation&#xA;&#xA;    #print Wr&#xA;    #print Br&#xA;    outputError = (calcValues - actualValues) &#xA;&#xA;    pooledError = Wr.dot(outputError.transpose())&#xA;    pooledError = np.reshape(pooledError, (pooledSize, pooledSize, numFilters, numImages))&#xA;&#xA;    convError = np.empty((filteredSize, filteredSize, numFilters, numImages))&#xA;    convErrorFilter = np.ones((poolSize, poolSize)) / (poolSize ** 2)&#xA;    for i in range(numFilters):&#xA;        for j in range(numImages):&#xA;            convError[:, :, i, j] = np.kron(pooledError[:, :, i, j], convErrorFilter)&#xA;&#xA;    convError *= conv * (1 - conv)&#xA;&#xA;    ## Gradient&#xA;&#xA;    WrGrad = pooled.dot(outputError)&#xA;    BrGrad = np.sum(outputError, 0)&#xA;&#xA;    WcGrad = np.zeros((filterSize, filterSize, numFilters))&#xA;    BcGrad = np.empty(numFilters)&#xA;&#xA;    for i in range(numFilters):&#xA;        BcGrad[i] = np.sum(convError[:, :, i, :])&#xA;        for j in range(numImages): &#xA;            filterLayer = np.rot90(convError[:, :, i, j], 2)&#xA;            WcGrad[:, :, i] += convolve2d(features[:, :, j], filterLayer, 'valid')&#xA;&#xA;    WrGrad /= numImages&#xA;    WcGrad /= numImages&#xA;    BrGrad /= numImages&#xA;    BcGrad /= numImages&#xA;&#xA;    WrGrad += reg * Wr / numImages&#xA;    WcGrad += reg * Wc / numImages&#xA;&#xA;    global costCache&#xA;    costCache = cost&#xA;    return (cost, np.concatenate((WcGrad.flatten(), BcGrad.flatten(), WrGrad.flatten(), BrGrad.flatten()), axis=0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I implemented a numerical gradient to verify if it was returning the right values, and it was. However, my softmax regression seems to be predicting the same value across the entire batch when I do SGD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I use scipy.minimize function, it gives me different values for softmax regression&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="16818" LastEditorUserId="16818" LastEditDate="2016-03-07T03:34:56.517" LastActivityDate="2016-03-07T03:34:56.517" Title="Error with Convolutional Neural Network (SGD?)" Tags="|machine-learning|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10565" PostTypeId="2" ParentId="10522" CreationDate="2016-03-06T23:15:56.633" Score="1" Body="&lt;h2&gt;&lt;a href=&quot;http://reference.wolfram.com/language/tutorial/GraphDrawingOverview.html&quot; rel=&quot;nofollow&quot;&gt;Mathematica's Graph Drawing&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Mathematica can draw both 2D (&lt;a href=&quot;http://reference.wolfram.com/language/ref/GraphPlot.html&quot; rel=&quot;nofollow&quot;&gt;GraphPlot&lt;/a&gt;) and 3D (&lt;a href=&quot;http://reference.wolfram.com/language/ref/GraphPlot3D.html&quot; rel=&quot;nofollow&quot;&gt;GraphPlot3D&lt;/a&gt;) graphs (static and interactive) and has a host of options available to fine tune the display and interactive content.  There is a very detailed &lt;a href=&quot;http://reference.wolfram.com/language/tutorial/GraphDrawingOverview.html&quot; rel=&quot;nofollow&quot;&gt;Graph Drawing&lt;/a&gt; language overview that has tutorials and examples through out. &lt;/p&gt;&#xA;" OwnerUserId="10814" LastActivityDate="2016-03-06T23:15:56.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10566" PostTypeId="2" ParentId="9943" CreationDate="2016-03-07T00:46:28.223" Score="2" Body="&lt;p&gt;Use the following procedure on your Mac--&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;vi to ~/.bash_profile (~/.zshrc if you're on that train)&lt;/li&gt;&#xA;&lt;li&gt;Paste the following entry (be sure to specify your desired version of spark-csv)--&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;export PYSPARK_SUBMIT_ARGS=&quot;--packages com.databricks:spark-csv_2.11:1.3.0 $PYSPARK_SUBMIT_ARGS&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From there, run 'ipython notebook' and test with something like this--&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pyspark as ps&#xA;from pyspark.sql import SQLContext&#xA;&#xA;sc = ps.SparkContext()&#xA;&#xA;input_csv = 'file:////PATH_TO_CSV_ON_LOCAL_FILESYSTEM'&#xA;&#xA;df=sqlContext.read.load(input_csv,format='com.databricks.spark.csv', header='true', inferSchema='true')&#xA;&#xA;df.dtypes  # Returns the csv's schema breakdown with types&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16827" LastActivityDate="2016-03-07T00:46:28.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10567" PostTypeId="2" ParentId="10562" CreationDate="2016-03-07T01:38:50.730" Score="1" Body="&lt;p&gt;Logistic regression would be an ideal candidate for assessing the probability of a sale, but it would be wise to take Emre's comments on 'how many' into account.  In this case even a basic aggregation of your top 50000 with a 'yes' dependent variable would get you there-- but why stop there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to isolate whether or not seasonal (or other cyclical) factors are influencing purchasing behavior (e.g. if you were attempting to optimize inventory levels for the summer purchasing habits).  In this case, linear regression with dummy variables accounting for the season or month may be a good way to delve deeper (this touches on the 'classification' point made by Emre).  The more historical data you have here, the better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to your point on 'fast'-- you mention Excel so I am assuming Python or R are being ruled out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;RapidMiner or Knime would fit your requirement for 'fast' in any of these modeling contexts.  Otherwise, if you're curious, scikit-learn (python) has about 3 dozen other general linear models freely available (R has a vast library as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attached is a link on how to do this kind of modeling within RapidMiner--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Logistic Regression Example (RapidMiner)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=0npgGPK0zR8&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=0npgGPK0zR8&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy modeling!&lt;/p&gt;&#xA;" OwnerUserId="16827" LastActivityDate="2016-03-07T01:38:50.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10568" PostTypeId="1" CreationDate="2016-03-07T03:37:53.590" Score="1" ViewCount="568" Body="&lt;p&gt;What is the best option or rather options to predict how much order a customer will place in the future, say next 3 months on a monthly basis. Also will a customer place an order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used ARIMA to predict the future order but the MAPE is quite high. Basically what i have done is predict the probability of a customer placing an order using survival analysis and then multiplied that with the predicted order quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that make sense or is there something fundamentally wrong with the approach?&lt;/p&gt;&#xA;" OwnerUserId="16829" LastActivityDate="2016-03-07T03:37:53.590" Title="Predicting purchase order?" Tags="|time-series|forecast|survival-analysis|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10569" PostTypeId="2" ParentId="10557" CreationDate="2016-03-07T06:16:29.040" Score="8" Body="&lt;p&gt;I assume you mean &lt;strong&gt;feature selection&lt;/strong&gt; as &lt;strong&gt;feature engineering&lt;/strong&gt;. The process I usually follow and I see some people do is &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Feature engineering&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try a few algorithms, usually highly performant ones such as RandomForest, Gradient Boosted Trees, Neutral Networks, or SVM on the features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1 Do simple parameter tuning such as grid search on a small range of parameters&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the result of step 2 is not satisfactory, go back to step 1 to generate more features, or remove redundant features and keep the best ones, people usually call this &lt;strong&gt;feature selection&lt;/strong&gt;. If running out of ideas for new features, try more algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the result is alright or close to what you want, then move to step 3 &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Extensive parameter tuning&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The reason for doing this is that classification is all about &lt;strong&gt;feature engineering&lt;/strong&gt;, and unless you know some incredible powerful classifier such as deep learning customized for a particular problem, such as Computer Vision. Generating good features is the key. Choosing a classifier is important but not crucial. All the classifiers mentioned above are quite comparable in terms of performance, and most of the time, best classifier turns out to be one of them.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parameter tuning can boost performance, in some cases, quite a lot. But without good features, tuning doesn't help much. Keep in mind, you always have time for parameter tuning. Also, there's no point of tuning parameter extensively then you discover a new feature and redo the whole thing. &lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-07T06:16:29.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10570" PostTypeId="2" ParentId="10557" CreationDate="2016-03-07T06:29:05.133" Score="1" Body="&lt;p&gt;I have already answered a similar question &lt;a href=&quot;https://datascience.stackexchange.com/a/10087/11097&quot;&gt;here&lt;/a&gt;. The process would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Transformation and Reduction:&lt;/strong&gt; Involves processes like transformations, mean and median scaling, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature Selection:&lt;/strong&gt; This can be done in a lot of ways like threshold selection, subset selection, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Designing predictive model:&lt;/strong&gt; Design the predictive model on the training data depending on the features you have at hand.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Cross Validation and parameter tuning:&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Final Prediction, Validation&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Always try and do feature engineering before model selection. Then, select the model according to the best features (or the features which tend to influence the problem/dependant variable better.)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-03-07T06:29:05.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10571" PostTypeId="2" ParentId="989" CreationDate="2016-03-07T09:24:09.103" Score="1" Body="&lt;p&gt;Leave it to run overnight or better for 24 hours. &#xA;What is your CPU utilization? If none of the cores is running at 100% then you have a problem. Probably with memory. Have you checked whether your dataset fits into 8GB at all?&#xA;Have you tried the SGDClassifier? It is one of the fastest there. Worth giving it a try first hoping it completes in an hour or so.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-03-07T09:24:09.103" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10572" PostTypeId="1" CreationDate="2016-03-07T10:05:09.100" Score="2" ViewCount="1027" Body="&lt;p&gt;I have a data set which has thousands of rows of {latitute, longitude, crime-type} tuples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;41.757366519   -87.642992854   THEFT&#xA;41.910469677   -87.585822373   ROBBERY&#xA;41.751270452   -87.690708662   BURGLARY&#xA;41.757366519   -87.642992854   THEFT&#xA;41.757366519   -87.642992854   THEFT&#xA;..             ..              ..&#xA;..             ..              ..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to cluster these based upon the crime types. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if in any  region, THEFT has a high frequency of occurrence, based on the data set, it should show up as a cluster. I have tried clustering using the lat-long data only, and that does not seem to have any meaning for this crime dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am fairly new to data mining, and gradually figuring my way out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I cluster the data using the latitude and longitude values based such that the clusters are related to each other through the crime-type? Is there any tool available that can use the lat-long data and cluster them on the crime-type basis? Otherwise, I can even write a script once I understand how this can be done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, has anyone had any previous experience in crime-data-mining? In what other ways can I find interesting patterns from a crime data-set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot! &lt;/p&gt;&#xA;" OwnerUserId="2481" LastActivityDate="2016-03-07T20:56:56.597" Title="Clustering Crime Data which has {latitute, longitude, crime-type} tuples" Tags="|data-mining|clustering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10574" PostTypeId="1" CreationDate="2016-03-07T12:58:22.040" Score="4" ViewCount="174" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'm currently preparing for a paper in which I will discuss the ability to predict the most likely application to be opened by the user at the given time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The application will collect information and learn to predict which application that is most likely to be run based on the learned patterns.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Proposed method&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'm collecting the following features for any given program run on the machine:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Time of day (To predict if the program is being run)&lt;/li&gt;&#xA;&lt;li&gt;Day of week (To learn when it is used)&lt;/li&gt;&#xA;&lt;li&gt;Most recently opened program (To see connections between programs)&lt;/li&gt;&#xA;&lt;li&gt;CPU load (Read below)&lt;/li&gt;&#xA;&lt;li&gt;GPU load (Read below)&lt;/li&gt;&#xA;&lt;li&gt;Memory usage (To connect programs to heavy usage)&lt;/li&gt;&#xA;&lt;li&gt;Screen use (To see if the program has connections with an active screen)&lt;/li&gt;&#xA;&lt;li&gt;Last mouse movement (To see if the program is actively used when it is&#xA;run)&lt;/li&gt;&#xA;&lt;li&gt;Last keyboard usage (Same as above)&lt;/li&gt;&#xA;&lt;li&gt;App running (Whether or not the application is running)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The mentioned features should allow for some seasonality to be present and should therefore yield some interesting results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the data has been collected, An ANN will be trained with the normalized inputs and each application as a separate output representing the possibility of the program being run. (0-1)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Is the proposed method adequate for such a task? I've read about using Bayesian classifiers but it doesn't seem like it's what I'm after. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also read about this being a &quot;time series&quot; task - something I couldn't quite wrap my head around but I assume I am exploiting through the proposed method. Is this a time series? Does it have to be treated as such?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should each application be present as a separate output or should the current application be fed in as an input, with the output presenting the possibility of the given app being run?&lt;/p&gt;&#xA;" OwnerUserId="16843" LastEditorDisplayName="user380" LastEditDate="2017-07-14T14:05:48.887" LastActivityDate="2017-07-14T14:05:48.887" Title="Predicting most likely application to be opened" Tags="|machine-learning|neural-network|predictive-modeling|time-series|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10575" PostTypeId="2" ParentId="10572" CreationDate="2016-03-07T14:04:56.227" Score="-1" Body="&lt;p&gt;Separate the training data according to every possible value of crime type and form chunks and after that apply K-mean clustering for longitude and latitude on each separate chunk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex, if possible crime type is &quot;murder&quot; and &quot;burglary&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then separate data in which &quot;murder&quot; is crime type and &quot;burglary&quot; is crime type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have two chunk of data. So apply K-Mean clustering on each chunk.&lt;/p&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2016-03-07T14:04:56.227" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10576" PostTypeId="2" ParentId="10574" CreationDate="2016-03-07T14:52:17.330" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;How do you plan to predict the exact app?  Are you thinking of a user level model (you can't predict that a user will open app X if she doesn't have it installed in the first place) Or maybe you want to predict apps at a segment level( instead of predicting individual apps)&lt;/li&gt;&#xA;&lt;li&gt;A neural network should work well if you have enough data since it has good &quot;capacity&quot;&lt;/li&gt;&#xA;&lt;li&gt;The time series part comes from the fact that the probability of opening app &lt;strong&gt;depends on what app was opened&lt;/strong&gt; before it. This is called &quot;structured prediction&quot; and is quite common in language models etc.&lt;/li&gt;&#xA;&lt;li&gt;The output will be a distribution over the apps and you can use a soft max layer for the final output&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-03-08T06:28:00.377" LastActivityDate="2016-03-08T06:28:00.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10577" PostTypeId="2" ParentId="10541" CreationDate="2016-03-07T15:09:06.030" Score="0" Body="&lt;p&gt;To add to what @Brandon has said: you can add another set of variables like customer ID* other predictors ( product of one hot customer IDs with other predictors). These additional predictors will implicitly capture the fact that different customers have different set of candidate stores. A cleaner approach is to build customer level models if you have enough data &#xA;Note: you'll have #customers*#predictors additional variables and will need to regularize using L1 penalty &lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-03-07T15:09:06.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10578" PostTypeId="2" ParentId="10562" CreationDate="2016-03-07T17:37:17.993" Score="0" Body="&lt;p&gt;If you can trancode the data into a regular CSV file, then &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;A visual programming tool for data science which includes several classification and regression algorithms.&quot;&gt;Orange&lt;/a&gt; can open it. Orange is a visual programming tool for data science. It includes several classication and regression algorithms which you can use to easily predict any unlabeled data you might have. There are also widgets for clustering and cluster analysis, which you may find useful (e.g. find if there is a group of products that better fits together than rest).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/YnQh6.png&quot; alt=&quot;Orange clustering example&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-03-07T17:37:17.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10579" PostTypeId="2" ParentId="10558" CreationDate="2016-03-07T17:43:49.207" Score="1" Body="&lt;p&gt;If you're interested more generally in speech understanding or speech-to-text, some approaches to natural language parsing and speech-to-text use recurrent neural networks or Hidden Markov processes for learning, as well as a number of signal-processing algorithms to extract more data from the input stream that just raw audio. Keep in mind people have spent their whole careers on this work, so it's not a good problem to just pick up and run with unless you're a MS/PhD candidate looking for a capstone/dissertation project. Here's the iconic paper from &lt;a href=&quot;http://arxiv.org/pdf/cmp-lg/9608018.pdf&quot; rel=&quot;nofollow&quot;&gt;Bell Labs&lt;/a&gt; that inspired a lot of the DFA/HMM solutions. I've yet to find a paper that does a good job of explaining how to actually implement the RNN style solutions, but &lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/RNN13.pdf&quot; rel=&quot;nofollow&quot;&gt;here's one&lt;/a&gt; in case you're interested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's likely that Alexa uses some combination of these methods, but I doubt you're going to get any good answer out of anyone here. After all, it's an important Amazon project and it's not like their engineers are going to come on Stack Overflow and start giving away trade secrets.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-03-07T17:43:49.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10580" PostTypeId="1" AcceptedAnswerId="10581" CreationDate="2016-03-07T19:39:18.703" Score="10" ViewCount="9786" Body="&lt;p&gt;I'm looking deeper into collaborative filtering. One really interesting paper is &quot;A Comparative Study of Collaborative Filtering Algorithms&quot; &lt;a href=&quot;http://arxiv.org/pdf/1205.3193.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/pdf/1205.3193.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to select which CF algorithm should be used the paper refers to the density of the dataset. What it doesn't do is explain how you actually calculate the density of your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in the context of that above paper can anyone help explain to me how I would calculate the density of a dataset? The paper refers regularly density in the 1-5% range.&lt;/p&gt;&#xA;" OwnerUserId="16854" LastActivityDate="2016-03-08T18:21:44.057" Title="How do you calculate how dense or sparse a dataset is?" Tags="|machine-learning|dataset|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10581" PostTypeId="2" ParentId="10580" CreationDate="2016-03-07T20:08:43.027" Score="7" Body="&lt;p&gt;It's actually defined on the first page:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... sparsity level (ratio of observed to total ratings) ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In other words, the fraction of the user/item rating matrix that is not empty. Remember that the problem is that most user-item pairs have no rating, and we wish to estimate them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let there be three users and four products. The number of &lt;em&gt;possible&lt;/em&gt; ratings is $3\times4 = 12$. If every user rates only one product each (regardless of which product), the density is 3/12 = 25%.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-08T18:21:44.057" LastActivityDate="2016-03-08T18:21:44.057" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10582" PostTypeId="2" ParentId="10572" CreationDate="2016-03-07T20:13:01.097" Score="1" Body="&lt;p&gt;There is no &lt;em&gt;meaningful&lt;/em&gt; way to combine &quot;type&quot; with a distance in meters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion is that you:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;split the data set by type.&lt;/li&gt;&#xA;&lt;li&gt;cluster each type, with DBSCAN, haversine distance, and the same minpts/eps values for each crime type&lt;/li&gt;&#xA;&lt;li&gt;compare the resulting clusters for similarity and differences&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-03-07T20:13:01.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10583" PostTypeId="2" ParentId="10572" CreationDate="2016-03-07T20:56:56.597" Score="0" Body="&lt;p&gt;I'm not pretty sure about what is the final objective of doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheles, I have some ideas. You could hash or dict the crime type (asign a number to each class) and then do clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or given a couple [lat, long] you could use an algorithm (like KNN) to predict which crime type is &quot;most probable&quot; to be.&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-03-07T20:56:56.597" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10584" PostTypeId="2" ParentId="10536" CreationDate="2016-03-07T21:37:51.123" Score="1" Body="&lt;p&gt;The most important application is even cross-domain usable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's known as CLD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You buy some expensive hardware, or rent some external expertise. And while you are waiting for the company administration to make a move and allow you to install and operate the hardware (which cannot not centrally administered, because the GPUs fall out of their usual active directory policy admin pattern) you have to make sure you have something fancy to show off. So you get a number of images from your corporate intranet, mostly with company logos. Lots of pictures. And then you feed all of them into a neural network using the inception pattern, and train this on your home gaming PC. Do not use the pretrained puppy-snail model! So after you have pretrained the network, you can now &quot;make Google dream your company logo&quot; in arbitrary images. so you generate a few of them to hang on your office walls while waiting for your systems to be approved. And when the C-level execs come by and want to see results while you still are not even started, you can give them some of this artwork to take home. They will all be impressed by your expertise of deep learning! And that is why I call this application the CLD, the c-level distractor.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-03-07T21:37:51.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10585" PostTypeId="1" AcceptedAnswerId="11099" CreationDate="2016-03-07T23:03:06.100" Score="7" ViewCount="1888" Body="&lt;p&gt;Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix.  I train some model (lets say a neural net for this example) and I find my AUC is really poor and I believe the problem to be the fact that I only have 1000 observations - so I don't have enough data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am now unable to collect anymore data. One thing I could do, is take that 30minute time series, and slice it up into 30 1-minute sections. Then use these 1 minute series as rows in my data. So I would end up with 30 observations per person, and 1000 people, giving me 30000 rows in my feature matrix. I've now increased the size of my training set by 30x&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to do this when doing statistical/inferential tests I would be violating the assumption of independence, and I'd be forced to model this with some multilevel model to correct&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the same as the IID assumption in machine learning, where each of your rows must be independent of each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For inferential tests, (one of) the reason why this assumption matters is because it affects the Type 1 error of your inference. However, in machine learning we're not doing any inferential tests, so what effect does violating IID actually have on results? In other words, &lt;em&gt;why&lt;/em&gt; do rows need to be independent of each other? Especially in a case like above where I can drastically increase the size of my training set by reusing different parts of 1 person's data&lt;/p&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2016-04-08T07:36:55.980" Title="IID violation in machine learning" Tags="|machine-learning|neural-network|statistics|data|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10586" PostTypeId="2" ParentId="10540" CreationDate="2016-03-08T02:34:58.633" Score="4" Body="&lt;p&gt;Yes, through the &lt;code&gt;components_&lt;/code&gt; property:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy, seaborn, pandas, sklearn.decomposition&#xA;data = numpy.random.randn(1000, 3) @ numpy.random.randn(3,3)&#xA;seaborn.pairplot(pandas.DataFrame(data, columns=['x', 'y', 'z']));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9bp8r.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9bp8r.png&quot; alt=&quot;Faceted scatter plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sklearn.decomposition.RandomizedPCA().fit(data).components_&#xA;&#xA;&amp;gt; array([[ 0.43929754,  0.81097276,  0.38644644],&#xA;       [-0.54977152,  0.58291122, -0.59830243],&#xA;       [ 0.71047094, -0.05037554, -0.70192119]])&#xA;&#xA;sklearn.decomposition.RandomizedPCA(2).fit(data).components_&#xA;&#xA;&amp;gt; array([[ 0.43929754,  0.81097276,  0.38644644],&#xA;       [-0.54977152,  0.58291122, -0.59830243]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We see that the truncated decomposition is simply the truncation of the full decomposition. Each row contains the coefficients of the corresponding principal component.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-08T02:34:58.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10587" PostTypeId="2" ParentId="10534" CreationDate="2016-03-08T04:43:20.280" Score="1" Body="&lt;p&gt;Let $x, y \in \mathbb R^N, \mathbf \alpha \equiv (\alpha_0, \dots, \alpha_N) \equiv (\alpha_0, \mathbf \alpha'_0), \; \mathbf z^\alpha \equiv z_0^{\alpha_0} \dots z_N^{\alpha_N}$ and ${d \choose \alpha}$ be the &lt;a href=&quot;https://en.wikipedia.org/wiki/Multinomial_theorem#Alternate_expression&quot; rel=&quot;nofollow&quot;&gt;multinomial coefficient&lt;/a&gt;. Then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left(\left&amp;lt;\mathbf x, \mathbf y \right&amp;gt; + c\right)^d = \sum_{|\mathbf \alpha| = d} {d \choose \alpha} c^{\alpha_0} (\mathbf x \mathbf y)^{\mathbf \alpha'_0}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you redistribute the coefficient ${d \choose \alpha} c^{\alpha_0}$ between $\Phi(\mathbf x)$ and $\Phi(\mathbf y)$, it follows that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Phi(\mathbf z) = \left(\sqrt{ {d \choose \alpha} c^{\alpha_0}} \mathbf z^{\alpha'_0} \right)_{\forall |\mathbf \alpha| = d}$$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://research.microsoft.com/pubs/67119/svmtutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;A Tutorial on Support Vector Machines for Pattern Recognition&lt;/a&gt; (cf. &lt;em&gt;6.1. The VC Dimension for Polynomial Kernels&lt;/em&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.rpi.edu/~stewart/lec23-post/kernels.pdf&quot; rel=&quot;nofollow&quot;&gt;Data Mining and Analysis: Fundamental Concepts and Algorithms&lt;/a&gt;, Ch.5, pp 149-151. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-10T03:40:06.340" LastActivityDate="2016-03-10T03:40:06.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10588" PostTypeId="2" ParentId="10479" CreationDate="2016-03-08T06:39:19.383" Score="1" Body="&lt;p&gt;Sigmoid kernels owe their popularity to neural networks, which traditionally used the sigmoid activation function. Sigmoid kernels de-emphasize extreme correlation. In a way they behave a bit like correlation coefficients, which also has a limited range, emphasizing similarity in orientation. $c$ shifts the operating point on the sigmoid, affecting the relative emphasis of the angle between the inputs. Perhaps &lt;a href=&quot;http://vip.uwaterloo.ca/files/publications/Carrington%20et%20al%20-%20A%20New%20Mercer%20Sigmoid%20Kernel.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this visualization&lt;/a&gt; (for $c=0$) might help mentally visualize this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dDEF4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dDEF4.png&quot; alt=&quot;A New Mercer Sigmoid Kernel for Clinical Data Classification&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first reference states that sigmoid kernels behave like RBFs for certain parameters. This makes them suited to nonlinear classification. You probably know that the sigmoid kernel is only conditionally PSD, and thus sometimes does not correspond to the kernel function of any implicit feature map per &lt;a href=&quot;https://en.wikipedia.org/wiki/Mercer%27s_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mercer's theorem&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Someone this interested in kernel methods should have a copy of &lt;a href=&quot;https://mitpress.mit.edu/books/learning-kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning with Kernels&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-08T06:51:26.537" LastActivityDate="2016-03-08T06:51:26.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10589" PostTypeId="1" CreationDate="2016-03-08T06:48:40.410" Score="7" ViewCount="25895" Body="&lt;p&gt;I'm using a basic RNN as in the figure below (say for translation).&#xA;The model has the following structure:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;\begin{aligned} s_t &amp;amp;= \tanh(Ux_t + Ws_{t-1}) \\ o_t &amp;amp;= \mathrm{softmax}(Vs_t) \end{aligned}&lt;/span&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assume that the vocabulary size is &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; and that of the hidden layer is &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If &lt;span class=&quot;math-container&quot;&gt;$x_{t}=\{0,1\}^{m}$&lt;/span&gt; and U is a &lt;span class=&quot;math-container&quot;&gt;$ n \times m$&lt;/span&gt;  matrix then W is a &lt;span class=&quot;math-container&quot;&gt;$ n \times n $&lt;/span&gt; matrix.&lt;/li&gt;&#xA;&lt;li&gt;If &lt;span class=&quot;math-container&quot;&gt;$o_{t}$&lt;/span&gt; is &lt;span class=&quot;math-container&quot;&gt;$\mathbb{R}^{k}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$s_{t}$&lt;/span&gt; is &lt;span class=&quot;math-container&quot;&gt;$\mathbb{R}^{n}$&lt;/span&gt;then V is a &lt;span class=&quot;math-container&quot;&gt;$ k \times n $&lt;/span&gt; matrix.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;What are the # parameters for this RNN model?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EhhBD.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EhhBD.jpg&quot; alt=&quot;a basic RNN&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="-1" LastEditDate="2023-12-10T14:57:17.110" LastActivityDate="2023-12-10T14:57:17.110" Title="Number of parameters in an RNN" Tags="|deep-learning|rnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10590" PostTypeId="5" CreationDate="2016-03-08T06:51:38.410" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-03-08T06:51:38.410" LastActivityDate="2016-03-08T06:51:38.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10591" PostTypeId="4" CreationDate="2016-03-08T06:51:38.410" Score="0" Body="A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.&#xD;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-03-08T10:34:36.463" LastActivityDate="2016-03-08T10:34:36.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10592" PostTypeId="2" ParentId="10589" CreationDate="2016-03-08T07:14:54.857" Score="10" Body="&lt;p&gt;The entities W , U and V are shared by all steps of the RNN and these are the only parameters in the model described in the figure. Hence number of parameters to be learnt while training = $dim(W)+ dim(V)+ dim(U)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on data in the question this = $ n^{2}+ kn + nm$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;n - dimension of hidden layer&lt;/li&gt;&#xA;&lt;li&gt;k - dimension of output layer&lt;/li&gt;&#xA;&lt;li&gt;m - dimension of input layer&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="31373" LastEditDate="2017-04-20T17:59:30.687" LastActivityDate="2017-04-20T17:59:30.687" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10595" PostTypeId="1" CreationDate="2016-03-08T12:11:25.040" Score="4" ViewCount="2669" Body="&lt;p&gt;In xgboost R package, there are two objectives given with booster &lt;code&gt;gbtree&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;reg:logistic&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;binary:logistic&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;See,page 22 (first 2 lines) &lt;a href=&quot;https://cran.r-project.org/web/packages/xgboost/xgboost.pdf&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/xgboost/xgboost.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering, what is the difference between these two methods?&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2016-03-09T03:59:10.647" Title="difference between logistic regression and binary logistic regression" Tags="|machine-learning|r|logistic-regression|xgboost|gbm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10596" PostTypeId="2" ParentId="9818" CreationDate="2016-03-08T14:19:07.903" Score="2" Body="&lt;p&gt;I did a small example for this once. From that, I think Bayesian Networks are preferred if you want to capture a distribution but your input training set doesn't cover the distribution well. In such cases, even a neural network that generalised well would not be able to reconstruct the distribution.&lt;/p&gt;&#xA;" OwnerUserId="16812" LastActivityDate="2016-03-08T14:19:07.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10597" PostTypeId="1" CreationDate="2016-03-08T16:51:45.650" Score="1" ViewCount="1733" Body="&lt;p&gt;Suppose I have a dataset which has the columns date, time, latitude, longitude, place name visited by a user. I am wondering whether there is a way to write any algorithm to predict the path, the user would take at a particular time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have the last two months date, time, latitude and longitude, can we predict the path the user would take at a particular date next month. I am not sure whether this is possible. But if any one can give me a idea of algorithm or any packages in R or python, it would be helpful. I can research on it.&lt;/p&gt;&#xA;" OwnerUserId="16887" LastActivityDate="2018-04-13T04:19:04.047" Title="Path prediction using the historic path taken by person" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10598" PostTypeId="2" ParentId="10597" CreationDate="2016-03-08T17:09:53.697" Score="1" Body="&lt;p&gt;I think we've covered this question before but I can't find my response... A lot of work has been done in this area. I don't know of any packages, but the basic idea is to discretize the search space. One way is to just to use the cellular grid, which makes sense if you have that information. Another is to first extract the &quot;locations&quot; from the location trace; i.e., places the users revisits and spends significant time in. Then you can apply well-known techniques such as HMMs to estimate the next discrete location from the recent past. Most of the papers I read on this subject are about a decade old, so they came out before the deep learning flurry. If they were revisiting the subject I'm sure I'd see papers using recurrent neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some papers for inspiration:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Learning and Inferring Transportation Routines&lt;/li&gt;&#xA;&lt;li&gt;A personal route prediction system based on trajectory data mining&lt;/li&gt;&#xA;&lt;li&gt;A Comparison of First- and Second-Order HMMs in the Task of Predicting the Next Locations of Mobile Individuals&lt;/li&gt;&#xA;&lt;li&gt;Using GPS to Learn Significant Locations and Predict Movement Across Multiple Users&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-08T17:09:53.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10600" PostTypeId="2" ParentId="10595" CreationDate="2016-03-08T18:16:32.237" Score="6" Body="&lt;p&gt;&lt;code&gt;binary:logistic&lt;/code&gt; is used for binary classification where the target variable takes &lt;strong&gt;binary output&lt;/strong&gt; [0, 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;reg:logistic&lt;/code&gt; is used for regression where the target variable is &lt;strong&gt;continuous&lt;/strong&gt; between [0, 1] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quote from xgboost doc:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;We use linear regression here, if we want use objective = reg:logistic logistic regression, the label needed to be pre-scaled into [0,1]&lt;/code&gt; &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/tree/master/demo/regression&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/dmlc/xgboost/tree/master/demo/regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14313" LastEditorUserId="14313" LastEditDate="2016-03-09T03:59:10.647" LastActivityDate="2016-03-09T03:59:10.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10602" PostTypeId="1" CreationDate="2016-03-08T21:27:18.810" Score="2" ViewCount="210" Body="&lt;p&gt;Where and when should I consider R^2 as a measure of goodness-of-fit for regressions?&lt;br&gt;&#xA;Usually I choose the least squared model as the best model.&lt;br&gt;&#xA;Is it possible that the least squared model does not have the highest R squared?&lt;br&gt;&#xA;Should I use both as the evaluation of regressions?&lt;br&gt;&#xA;Could you give me an intuitive example?&lt;/p&gt;&#xA;" OwnerUserId="15064" LastEditorUserId="381" LastEditDate="2016-03-08T23:27:47.543" LastActivityDate="2016-03-08T23:27:47.543" Title="Difference between coefficient of determination and least squared error?" Tags="|regression|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10603" PostTypeId="2" ParentId="10523" CreationDate="2016-03-08T22:02:08.687" Score="6" Body="&lt;p&gt;My personal approach is to pick the optimizer that is newest (i.e. newest-published-in-a-peer-reviewed-journal), because they usually report results on standard datasets, or beat state of the art, or both. When I use Caffe for example, I always use &lt;a href=&quot;http://arxiv.org/abs/1412.6980&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adam&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9667" LastEditorUserId="98307" LastEditDate="2020-08-05T08:47:41.707" LastActivityDate="2020-08-05T08:47:41.707" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10605" PostTypeId="2" ParentId="10602" CreationDate="2016-03-08T22:16:14.060" Score="3" Body="&lt;p&gt;R squared is one of many model diagnostics. One calculation of it is &lt;code&gt;(1-var(residuals)/var(response variable))&lt;/code&gt;. Since the response variable is constant, you want to minimize the average squared residuals (the variance), which is &lt;strong&gt;identical to least squares&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue with using only R-squared is that if you have a lot of variables in a model, you will likely get a high R-squared, but that doesn't mean the model is good.  In fact, if you have 100 rows of data, 100 independent variables and 1 response variable, there's a good chance you will get an R-squared of 1, but the model will be completely overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 common solutions to this problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Add a penalty term that reduces (or increases, depending on what is optimal) the &quot;score&quot; of a model when it has more variables.&lt;/li&gt;&#xA;&lt;li&gt;Use a form of validation to see if the model performs well on data it wasn't trained on.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Oftentimes both of these are used to varying degrees, but here are a few examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;AIC, or Aikaike Information Criterion, is &lt;code&gt;-2*log(P(Y|X)) + 2*p&lt;/code&gt;, where &lt;code&gt;P(Y|X)&lt;/code&gt; is the probability density of Y given X under the current model and &lt;code&gt;p&lt;/code&gt; is the number of predictor (&lt;code&gt;X&lt;/code&gt;) variables.  The goal is to minimize AIC.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In K-fold cross-validation, where K is the number of subsets you divide data into, you train the data on K-1 of the sets combined and then test it on the remaining set and report the error.  You repeat this process for each combination of K-1 training sets and 1 remaining set and combine the K results (usually by averaging error).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first method is usually a safe bet for linear regression models, but oftentimes, especially when you are using more complex algorithms, you will want to validate the algorithm to make sure it isn't overfitting the training data.  Also, using validation allows comparisons between different models (especially non-parametric ones) to be made much more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more ways of evaluating models, I would suggest reading into different regularization techniques and different validation techniques.&lt;/p&gt;&#xA;" OwnerUserId="9565" LastActivityDate="2016-03-08T22:16:14.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10607" PostTypeId="2" ParentId="10585" CreationDate="2016-03-08T23:11:30.407" Score="0" Body="&lt;p&gt;I don't feel secure enough to give a definitive answer but this described situation arises in phoneme classification when the data is split up in arbitray small parts. Here even though it is the same problem it does not cause any problems I know of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would just give it a try violating this assumption and just see if it works. This approach is often used in machine learning. For example Naive Bayes is sometimes used when the training data does not behave like a diagonal covariance gaussian etc.&lt;/p&gt;&#xA;" OwnerUserId="12798" LastActivityDate="2016-03-08T23:11:30.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10608" PostTypeId="2" ParentId="8338" CreationDate="2016-03-09T00:16:33.483" Score="1" Body="&lt;p&gt;if you want a correlation for term frequency, assuming you have a frequency vector representation for each row, then you can compute the similarity between any two rows by computing &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow&quot;&gt;cosine similarity&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how one could do it in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_set &amp;lt;- c(&quot;the big bad dog&quot;, &#xA;              &quot;the small cat and the orange cat&quot;, &#xA;              &quot;the big big dog&quot;)&#xA;words &amp;lt;- strsplit(data_set, split = &quot; &quot;) #tokenize sentences&#xA;vec &amp;lt;- unique(unlist(words)) #vector representation of sentences&#xA;&#xA;m &amp;lt;- matrix(nrow = length(data_set), ncol = length(vec))&#xA;&#xA;for (i in 1:length(words)) { #iterate the index of tokenized sentences&#xA;  tokens &amp;lt;- words[[i]]&#xA;  vec_rep &amp;lt;- as.integer(sapply(vec, function(w){sum( w == tokens)})) #create binary word-feature vector&#xA;  m[i,] &amp;lt;- vec_rep #update matrix&#xA;}&#xA;&#xA;df &amp;lt;- data.frame(m, row.names = NULL)&#xA;names(df) &amp;lt;- vec&#xA;df&#xA;&#xA;##  the big bad dog small cat and orange&#xA;##1   1   1   1   1     0   0   0      0&#xA;##2   2   0   0   0     1   2   1      1&#xA;##3   1   2   0   1     0   0   0      0&#xA;&#xA;cosineSimilarity &amp;lt;- function(df, row1, row2){&#xA;  x &amp;lt;- as.numeric(df[row1,])&#xA;  y &amp;lt;- as.numeric(df[row2,])&#xA;  (x %*% y) / (sqrt(x%*%x * y%*%y))&#xA;}&#xA;&#xA;cosineSimilarity(df,1,1) #1.00&#xA;cosineSimilarity(df,1,2) #0.30&#xA;cosineSimilarity(df,1,3) #0.82&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you have labelled training data, you can create a prototype vector representation for each label. Then during classification of unlabelled data you just compute cosine similarity between target text and each prototype, and then assign the label to prototype that maximizes similarity score.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-03-09T00:16:33.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10609" PostTypeId="1" CreationDate="2016-03-09T00:24:50.050" Score="2" ViewCount="1044" Body="&lt;p&gt;What are the common techniques for encoding discrete features with tree-path value (e.g. file/folder path) without expanding tree into multiple features corresponding to each node?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, I am looking for encoding which can be used with different algorithms (e.g. linear regression, not only clustering).&#xA;Something like &lt;a href=&quot;https://en.wikipedia.org/wiki/Huffman_coding&quot; rel=&quot;nofollow&quot;&gt;Huffman coding&lt;/a&gt; but usable with machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider this tree described with lisp-like notation:&#xA;&lt;code&gt;(a (b b1 b2) (c c1 c2 c3))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be encoded by creating feature for each node&#xA;but this will be very expensive for any sufficiently large tree.&#xA;&lt;code&gt;&#xA;a  b  b1 b2 c  c1 c2 c3 Path&#xA;1  1  0  0  0  0  0  0  /a/b&#xA;1  0  0  0  1  0  1  0  /a/c/c2&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of creating feature for each node,&#xA;there could probably be a feature for each depth level.&#xA;&lt;code&gt;&#xA;L1 L2 L3 Path&#xA;1  1  0  /a/b&#xA;1  2  6  /a/c/c2&#xA;&lt;/code&gt;&#xA;where&lt;br/&gt;&#xA;L2 can have value of 0,1,2 for none,b,c correspondingly;&lt;br/&gt;&#xA;L3 can have value of 0,1,2,5,6,7 for none,b1,b2,c1,c2,c3 correspondingly&#xA;(with gap between b2, and c1 so that there is numerical distance between nodes with different parents).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some obvious problems with it though,&#xA;e.g. c1 and c3 are further away from each other than c1 and c2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that encoding really depends on how you want to interpret the tree and relation between nodes. I was hoping there might be some article/paper which describes different approaches and trade-offs.&lt;/p&gt;&#xA;" OwnerUserId="16897" LastEditorUserId="16897" LastEditDate="2016-03-14T08:40:37.780" LastActivityDate="2016-03-17T02:20:28.220" Title="Encoding feature with tree-path data" Tags="|machine-learning|data-cleaning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10611" PostTypeId="2" ParentId="10609" CreationDate="2016-03-09T06:12:04.713" Score="2" Body="&lt;h1&gt;Clustering&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I think a tree is a perfectly appropriate data structure in this case. You don't need an embedding to do clustering; there are similarity-based approaches too, and defining a similarity function in your case is straightforward: I'd say it's a function of the common depth between the two items in consideration. You will probably want to define a root to &quot;normalize&quot; the scores. For example, given the root &quot;a/b/&quot; and the paths:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a/b/c/d/foo1&#xA;a/b/c1/foo2&#xA;a/b/c1/foo3&#xA;a/b/c/d/foo4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The shared depth between foo1 and foo2 and 0, discounting the root. For foo1 and foo4 it is 2. For foo2 and foo3 it is 1. You can define the similarity as a transformation of this through the function $f:x \to 1-\exp(-ax)$, where $a$ is a parameter you can use to tweak the clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For implementation, in python, you can try &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;nofollow&quot;&gt;sklearn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;General&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You can featurize a path string by creating a set from its parent folders, which can then be represented as a sparse bit string by &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_hashing#Hashing_strings&quot; rel=&quot;nofollow&quot;&gt;hashing the set elements&lt;/a&gt;. For example, &quot;a/b/c/d/foo1&quot; becomes (&quot;c&quot;, &quot;c/d&quot;), if we let &quot;a/b&quot; be the root, as before. (The notion of a root is not strictly necessary here except to ensure that the baseline similarity is zero.) The path similarity is simply then the set similarity or, after conversion to bit strings, the $L_p$ distance.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-17T02:20:28.220" LastActivityDate="2016-03-17T02:20:28.220" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10612" PostTypeId="1" AcceptedAnswerId="21662" CreationDate="2016-03-09T08:37:20.947" Score="16" ViewCount="10038" Body="&lt;p&gt;&lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;noreferrer&quot;&gt;https://radimrehurek.com/gensim/models/doc2vec.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we have trained doc2vec with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;aaaaaAAAAAaaaaaa&quot; - &quot;label 1&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“bbbbbbBBBBBbbbb&quot; - &quot;label 2&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can we infer “aaaaAAAAaaaaAA” is label 1 using Doc2vec?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know Doc2vec can train word vectors and label vectors.&#xA;Using this vectors, can we infer unseen sentences(combination of trained words) in which label?&lt;/p&gt;&#xA;" OwnerUserId="16865" LastActivityDate="2021-07-12T06:51:22.930" Title="Doc2vec(gensim) - How can I infer unseen sentences’ label?" Tags="|gensim|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10613" PostTypeId="1" AcceptedAnswerId="10617" CreationDate="2016-03-09T09:51:44.833" Score="1" ViewCount="432" Body="&lt;p&gt;I have generated a model for predicting the future input trend with a sample data using linear regression in Knime. I want to validate the model using a different data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the data set I used for creating the model came from the sensor of device A. The prediction is accurate for the dataset from device A. Lets say I saved the model as a PMML file  called A. Can I use the same PMML file A to predict values of Device B (Values of B are not comparable). If not, should I create a different model for all data set that I have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I combine all the models generated so that it predicts any given data set? Is it possible?&lt;/p&gt;&#xA;" OwnerUserId="14332" LastActivityDate="2016-03-09T13:57:55.067" Title="Using a model for a different dataset" Tags="|data-mining|bigdata|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10615" PostTypeId="1" CreationDate="2016-03-09T11:14:20.163" Score="59" ViewCount="78199" Body="&lt;p&gt;How many parameters does a single stacked LSTM have? The number of parameters imposes a &lt;strong&gt;lower bound on the number of training examples&lt;/strong&gt; required and also influences the training time. Hence knowing the number of parameters is useful for training models using LSTMs.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="11097" LastEditDate="2016-03-09T14:27:11.463" LastActivityDate="2020-07-28T05:45:56.057" Title="Number of parameters in an LSTM model" Tags="|deep-learning|rnn|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10616" PostTypeId="2" ParentId="10615" CreationDate="2016-03-09T11:15:28.157" Score="43" Body="&lt;p&gt;The LSTM has a set of 2 matrices: U and W for each of the (3) gates. The (.) in the diagram indicates multiplication of these matrices with the input &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and output &lt;span class=&quot;math-container&quot;&gt;$h$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;U has dimensions &lt;span class=&quot;math-container&quot;&gt;$n \times m$&lt;/span&gt; &lt;/li&gt;&#xA;&lt;li&gt;W has dimensions &lt;span class=&quot;math-container&quot;&gt;$n \times n$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;there is a different set of these matrices for each of the three gates(like &lt;span class=&quot;math-container&quot;&gt;$U_{forget}$&lt;/span&gt; for the &lt;em&gt;forget&lt;/em&gt; gate etc.)&lt;/li&gt;&#xA;&lt;li&gt;there is another set of these matrices for updating the cell state S&lt;/li&gt;&#xA;&lt;li&gt;on top of the mentioned matrices, you need to count the biases (not in the picture)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hence total # parameters = &lt;span class=&quot;math-container&quot;&gt;$4(nm+n^{2} + n)$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DT6Xc.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DT6Xc.png&quot; alt=&quot;LSTM abstract block&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="23492" LastEditDate="2018-10-20T12:11:21.740" LastActivityDate="2018-10-20T12:11:21.740" CommentCount="4" CommunityOwnedDate="2016-05-04T10:45:16.563" ContentLicense="CC BY-SA 4.0" />
  <row Id="10617" PostTypeId="2" ParentId="10613" CreationDate="2016-03-09T13:57:55.067" Score="1" Body="&lt;p&gt;I think you answered yourself by &quot;values of B are not comparable&quot;. Learning for predictions is based on a fundamental assumption which is the data for prediction has the same joint distribution as the data for learning. This is the link between those processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you want to handle that in a meaningful way you have to know somehow the source type. In your example the device type. One way would be to introduce the device type as a different column in your data set, so that the model can have the chance to differentiate between source type. Obviously you have to have training data for all the device types. Supposing you have 2 device types, A and B. Your training data should have some columns for the signal and also a factor column with type A and B. Also you have to have enough instances of data with type A and type B. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-03-09T13:57:55.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10618" PostTypeId="1" CreationDate="2016-03-09T13:58:57.540" Score="2" ViewCount="113" Body="&lt;p&gt;Here is my problem: I need to use hupossion in MCMCglmm package. Here is my prior:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; prior = list(R = list(V = diag(2), nu = 0.002),&#xA; +                G = list(G1 = list(V = diag(10), nu = 0.002),&#xA; +                           G2 = list(V = 1, nu = 0.002)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here is my training:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; myTrain &amp;lt;- MCMCglmm(pipenumyearnosameday ~ trait - 1 +&#xA;+                         at.level(trait,1):lastyearnumnosameday + &#xA;+                         at.level(trait,1):pipeInsideSizemm,&#xA;+                     random = ~idh(pipematerialgroup1):installyeargroup1,&#xA;+                     data = train,&#xA;+                     prior = prior,&#xA;+                     verbose = F,&#xA;+                     pr=T,&#xA;+                     family = &quot;hupoisson&quot;,&#xA;+                     rcov = ~idh(trait):units)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are 5 levels in the attribute pipematerialroup1. I know sth must be wrong with the prior. I got the following information:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in MCMCglmm(pipenumyearnosameday ~ trait - 1 + at.level(trait, 1):lastyearnumnosameday + : prior$G has the wrong number of structures&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;there is one more thing I need to say:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myPredict &amp;lt;- predict(myTrain,newdata = myTest,type = &quot;response&quot;,marginal = ~idh(trait):pipematerialgroup1,interval = &quot;confidence&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is my prediction part when I just take pipematerialgroup1 as random effect. If combine pipematerialgroup1 and installyeargroup1, what this prediction would be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone help me get the prior right? Thanks a lot.&lt;/p&gt;&#xA;" OwnerUserId="16911" LastActivityDate="2016-03-09T13:58:57.540" Title="How can I use idh and random when I use hupossion in mcmcglmm?" Tags="|r|glm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10619" PostTypeId="1" CreationDate="2016-03-09T14:23:20.817" Score="4" ViewCount="291" Body="&lt;p&gt;Nowadays there's a trend towards using architectures of &quot;deep&quot;  RNNs i.e. vertically stacked RNNs. &lt;a href=&quot;http://www.deeplearningbook.org/contents/rnn.html&quot; rel=&quot;nofollow&quot;&gt;RNN chapter from Bengio's book&lt;/a&gt;These networks seem to work well in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the intuition around using vertically stacked layers of RNNs (beyond the obvious fact that they increase the &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://bengio.abracadoudou.com/lectures/theory.pdf&amp;amp;ved=0ahUKEwjsqYzP57PLAhWMA44KHcY5DKcQFggfMAA&amp;amp;usg=AFQjCNEaA9h_j-EykLpmgblOE2QQoZ4r-A&amp;amp;sig2=Mi0f7Za7ujvJadNMlwz7hw&quot; rel=&quot;nofollow&quot;&gt;capacity&lt;/a&gt; by increasing the # parameters)?&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2024-02-08T17:00:55.107" Title="Importance/intuition behind stacking RNNs" Tags="|deep-learning|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10620" PostTypeId="2" ParentId="6406" CreationDate="2016-03-09T15:23:54.927" Score="3" Body="&lt;p&gt;Silhouette Score, it was mentioned in a previous answer but I don't have the reputation to comment. &#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; are useful links if you are using python to implement clustering. Check section 2.3.9.4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Silhouette Score takes overfitting into consideration I think. &#xA;For example if I have a dataset with 24 points to cluster, if I put them in 23 clusters the score is 0.0263. &#xA;If I put them in 6 cluster using K-Means then I get a score of 0.2705.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in your GA, you can use this quite nicely as a fitness function.&lt;/p&gt;&#xA;" OwnerUserId="16812" LastEditorUserId="16812" LastEditDate="2018-01-11T22:12:33.503" LastActivityDate="2018-01-11T22:12:33.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10621" PostTypeId="1" CreationDate="2016-03-09T15:27:17.250" Score="1" ViewCount="288" Body="&lt;p&gt;I was hoping to get some help on a task I have been given. I have a data set with the chemical breakdown of a drink. My job is to find some sort of analysis with the given data. The data has 10 different attributes and a numerical value associated with each attribute and roughly 300 instances of it. I have to use Weka and perform some sort of Clustering, Association, or Classification algorithm. I cannot think of a good analysis that can be done with this data!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first I thought of using the clustering algorithm but that does not tell me much besides clustering the the data into the number of clusters I assign and telling me the mean composition of each chemical per cluster. Not much of analysis there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping someone might be able to guide me in coming up with a good analysis prompt I can do with this data!&lt;/p&gt;&#xA;" OwnerUserId="16913" LastActivityDate="2016-03-09T15:27:17.250" Title="Numerical data and different algorithms" Tags="|classification|clustering|algorithms|weka|association-rules|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10622" PostTypeId="1" CreationDate="2016-03-09T17:10:16.333" Score="2" ViewCount="38" Body="&lt;p&gt;I have a problem with outputting the terms for a logistic regression model in R.  For a given list of independent values, say list l of terms {w,y,z} to determine dependent variable {x}, I want to find out what the biggest regressor is when we pair two terms together.  I want to be able to group multiple independent variables together and say &quot;when a record has this combination of values, then they have a very strong chance of predicting X&quot;.  I tried to just add the interactions when calling the glm function like glm(x~y + w + z + w:z + y:z + y:w, data = l).  But the results come out very hard to explain, because they of how they are measured between themselves and not just measured against the mean. Does anyone know a way to do this?  &lt;/p&gt;&#xA;" OwnerUserId="16676" LastActivityDate="2016-03-09T17:10:16.333" Title="Outputting risk groups for a logistic regression model" Tags="|r|predictive-modeling|logistic-regression|glm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10624" PostTypeId="1" CreationDate="2016-03-10T01:25:22.640" Score="1" ViewCount="17" Body="&lt;p&gt;I am trying to find out the processes on our linux servers that are responsible for load on the system.&#xA;We have an application server and an Oracle database server (both running Linux).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The application server runs processes which are mostly shell scripts, perl scripts and standard applications like Apache. Many of the processes open sessions in database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sampling the load average on both servers once a minute using 'uptime' command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also collecting the output of 'ps' command once a minute, to list the processes running.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the above information, what algorithms or machine learning techniques can I use to classify processes into 10 different buckets with 10 indicating the processes that cause heaviest load on the system and 1 the lightest load?&lt;/p&gt;&#xA;" OwnerUserId="13145" LastActivityDate="2016-03-10T01:25:22.640" Title="Associating processes with load on the system" Tags="|machine-learning|classification|association-rules|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10625" PostTypeId="1" CreationDate="2016-03-10T03:07:38.000" Score="2" ViewCount="2264" Body="&lt;p&gt;I want to ask a question on how to calculate WOE and IV from weighted good bad indicator in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found the &lt;strong&gt;woe&lt;/strong&gt; package in R, but it only calculate WOE and IV from non weighted good bad indicator and it seems the indicator must be on 0 and 1 only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone knows how to manipulate the good bad indicator in this &lt;strong&gt;woe&lt;/strong&gt; package or got information of WOE and IV calculation package with weighted indicator?&lt;/p&gt;&#xA;" OwnerUserId="16927" LastEditorUserId="11097" LastEditDate="2016-03-10T05:29:26.090" LastActivityDate="2016-03-10T05:29:26.090" Title="Calculate Information Value and Weight of Evidence in R" Tags="|r|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10626" PostTypeId="1" AcceptedAnswerId="10627" CreationDate="2016-03-10T03:42:07.697" Score="1" ViewCount="3066" Body="&lt;p&gt;Let's say I do a Decision Tree analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the performance characteristics are nothing great (e.g. ROC is nothing great).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything I can do with this &quot;not so great&quot; tree. Or do I typically need to trash it and try something else (either a new data set or a new analysis on the same data)?&lt;/p&gt;&#xA;" OwnerUserId="16928" LastActivityDate="2016-03-14T21:21:15.620" Title="What can I do with a Decision Tree with poor ROC" Tags="|decision-trees|performance|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10627" PostTypeId="2" ParentId="10626" CreationDate="2016-03-10T07:27:23.577" Score="4" Body="&lt;p&gt;Decision trees has one big quality and one big drawback. Their big quality is that they are what is known as glass-box models. What I mean by that is that they expose what they learn in a very clear and intuitive way. The name comes from the fact that you can see through a glass box. So, because of that, decision trees are very useful for analysis, they are a nice support to understand the relations between variables, which variables are important, in which way they are important. And even if they does not provide crystal clear information, they can bring you ideas about that. This might be very helpful especially if you have domain expert knowledge and you can put things together in a meaningful manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their main drawback is their high variability. This problem is mainly caused by their greedy approach. Each decision in the first level nodes shape differently the tree. You can even go further and see that a single additional data point is enough in many cases to get a totally different tree, especially if the sample is small or if the data is noisy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two types of approaches to solve this issue. The first type of approach tries to improve the single tree you built. This kind of approaches are known as pruning. A simple example would be reduced error pruning. This approach is simple and produce good results. You train your tree on a sample of data. Then you take another sample of data, you fit the new data on the tree, and then evaluate again the nodes of the tree from the perspective of the new data. If a non-leaf node get at least the same error if would not be split than if it would be split, then you can decide to cut the child nodes and transform that node into a leaf node. There are however much nicer pruning strategies, which are stronger, perhaps based on cross validation or some other criteria, mostly statistical based. Notice however that for reduced error pruning you need additional data, or to split your original sample in two, one for training, the other for pruning. If you go further to estimate the prediction error you need a third sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second approach would be either to build multiple times some trees and chose some based on cross validation, bootstrapping or whatever method you you, or use a tree ensembles like bagging or boosting algorithms. Note that for boosting and bagging you loose glass box property. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately you have to choose between understanding and performance, having as a decent compromise the pruning procedure. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2016-03-10T09:01:08.403" LastActivityDate="2016-03-10T09:01:08.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10628" PostTypeId="1" CreationDate="2016-03-10T09:21:41.117" Score="1" ViewCount="101" Body="&lt;p&gt;I have questions regarding this Normalized weights and Initial inputs video on &lt;a href=&quot;https://www.udacity.com/course/viewer#!/c-ud730/l-6370362152/m-7119160655&quot; rel=&quot;nofollow&quot;&gt;Udacity&lt;/a&gt; course Deep Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this video the lecturer talks about variables that go into Big-loss function should have zero mean and equal variances. I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function? And how  would zero mean and equal variance help in optimization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other question I have is, in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma? And where optimizer will move the point (initialized weight), up or down to find the local minima?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?&lt;/p&gt;&#xA;" OwnerUserId="16939" LastEditorUserId="8820" LastEditDate="2016-03-10T17:53:02.880" LastActivityDate="2016-04-09T18:38:37.303" Title="Questions regarding deep learning?" Tags="|machine-learning|deep-learning|reference-request|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10630" PostTypeId="1" CreationDate="2016-03-10T10:48:12.450" Score="1" ViewCount="802" Body="&lt;p&gt;I have been given 6 data sets out of which 5 are training sets. The 5 data sets correspond to 5 drivers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row of a data set consists of a time-stamp followed by a poly-line. The poly-line gives the GPS coordinates (latitude,longitude) of the driver taken every 15th second. So the length of this line gives the time duration of each trip. I have to classify the poly-lines corresponding to each time-stamp to one of these 5 drivers in the last (test) data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need assistance in choosing a classification algorithm. I have only implemented Naive Bayes (text classification) before but I don't think it will work here. I'm having trouble visualizing the approach to solve this problem.&lt;/p&gt;&#xA;" OwnerUserId="16943" LastEditorUserId="16284" LastEditDate="2016-04-22T17:04:11.213" LastActivityDate="2016-04-22T17:04:11.213" Title="Which classification algorithm to choose for classifying driving patterns (GPS coordinates) and mapping them to drivers?" Tags="|classification|geospatial|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10631" PostTypeId="1" AcceptedAnswerId="10673" CreationDate="2016-03-10T10:53:59.060" Score="2" ViewCount="360" Body="&lt;p&gt;I want to implement MAB(Multi arm bandit algorithm ) in python. There is a package called bandit in R &lt;a href=&quot;https://cran.r-project.org/web/packages/bandit/bandit.pdf&quot; rel=&quot;nofollow&quot;&gt;Check here&lt;/a&gt;, is there anything similar to this in python&lt;/p&gt;&#xA;" OwnerUserId="15905" LastActivityDate="2016-03-12T05:37:45.277" Title="Is there an equivalent of the Bandit package in r (Used in implementation of Multi arm bandit algorithm) in python?" Tags="|r|python|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10633" PostTypeId="1" CreationDate="2016-03-10T15:52:55.427" Score="1" ViewCount="2422" Body="&lt;p&gt;I am trying to write a model that will extract certain details from financial documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It must be able to extract the; contract start date, contract duration, contract value and all named entities. Preferably it would also be able to categorise the named entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have roughly 600 tagged documents that I can use for training and testing purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What model should I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had a look at different named entity recognition models, such as the Skip-Gram model. However these models do not utilise the fact that each one of these documents will contain all of this information. To illustrate this, the Skip-Gram model is often implemented in such a way that only the words in the same sentence contribute to its semantic vector.&lt;/p&gt;&#xA;" OwnerUserId="16951" LastActivityDate="2016-03-11T22:36:56.720" Title="What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem" Tags="|information-retrieval|named-entity-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10634" PostTypeId="1" CreationDate="2016-03-10T15:56:50.617" Score="1" ViewCount="738" Body="&lt;p&gt;I'm working on an implementation of a neural network so I can really grasp how these magic boxes work. However the neural network I've written code for doesn't work and I &lt;em&gt;think&lt;/em&gt; it's due to my implementation of backpropagation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[pseudo code]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'last layer only&#xA;for each node (i) in last layer&#xA;error(i) = outputError(i)&#xA;end for&#xA;&#xA;'all hidden/inner layers&#xA;for each layer (i) in layers (not including layer 0 or last)&#xA; for each neuron (j) in layer i&#xA;  for each neuron (k) in layer i+1&#xA;   error(i)(j) +=error(i+1)(k) * weight(i)(j,k)&#xA;  end for&#xA;&#xA;  errors(i)(j) = errors(i)(j) * activation(i)(j) * (1-activation(i)(j))&#xA;&#xA;  for each neuron in layer i+1&#xA;   deltaWeight(i)(j,k) = -error(i)(k)*activation(i)(k)*learningRate/NumExamples&#xA;   deltaBias(i)(k) = -error(i)(k) * bias(i)(k)*learningRate/NumExample&#xA;  end for&#xA; end for&#xA;end for&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I update all the weights with this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weight(i)(j,k) = weight(i)(j,k) + deltaWeight(i)(j,k)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the problem I have is that the output doesn't seem to get any better. It certainly changes but it doesn't seem to minimise the cost function at all, does anyone know why?&lt;/p&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="5041" LastEditDate="2016-04-10T15:45:13.360" LastActivityDate="2016-10-10T21:06:00.037" Title="Neural Network Backpropagation problems" Tags="|neural-network|backpropagation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10636" PostTypeId="1" AcceptedAnswerId="10890" CreationDate="2016-03-10T17:36:50.013" Score="-2" ViewCount="636" Body="&lt;p&gt;Fallowing this tutorial &lt;a href=&quot;https://templates.prediction.io/PredictionIO/template-scala-parallel-universal-recommendation&quot; rel=&quot;nofollow&quot;&gt;https://templates.prediction.io/PredictionIO/template-scala-parallel-universal-recommendation&lt;/a&gt;, when trying to train I get the fallowing error message &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ERROR] [TaskSetManager] Task 0 in stage 23.0 failed 1 times; aborting job&#xA;Exception in thread &quot;main&quot; org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 1 times, most recent failure: Lost task 0.0 in stage 23.0 (TID 17, localhost): java.lang.NegativeArraySizeException&#xA;    at org.apache.mahout.math.DenseVector.&amp;lt;init&amp;gt;(DenseVector.java:57)&#xA;    at org.apache.mahout.sparkbindings.SparkEngine$$anonfun$5.apply(SparkEngine.scala:78)&#xA;    	at org.apache.mahout.sparkbindings.SparkEngine$$anonfun$5.apply(SparkEngine.scala:77)&#xA;    	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:706)&#xA;    	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:706)&#xA;    	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)&#xA;    	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)&#xA;    	at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)&#xA;    	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)&#xA;    	at org.apache.spark.scheduler.Task.run(Task.scala:88)&#xA;    	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;    at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;Driver stacktrace:&#xA;    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)&#xA;    	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)&#xA;    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)&#xA;    	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)&#xA;    	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)&#xA;    	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)&#xA;    	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)&#xA;    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)&#xA;    	at scala.Option.foreach(Option.scala:236)&#xA;    	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)&#xA;    	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)&#xA;    	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)&#xA;    	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)&#xA;    	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)&#xA;    	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)&#xA;    	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)&#xA;    	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1942)&#xA;    	at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1003)&#xA;    	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)&#xA;    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)&#xA;    	at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)&#xA;    	at org.apache.spark.rdd.RDD.reduce(RDD.scala:985)&#xA;    	at org.apache.mahout.sparkbindings.SparkEngine$.numNonZeroElementsPerColumn(SparkEngine.scala:86)&#xA;    at org.apache.mahout.math.drm.CheckpointedOps.numNonZeroElementsPerColumn(CheckpointedOps.scala:37)&#xA;    at org.apache.mahout.math.cf.SimilarityAnalysis$.sampleDownAndBinarize(SimilarityAnalysis.scala:286)&#xA;    	at org.apache.mahout.math.cf.SimilarityAnalysis$.cooccurrences(SimilarityAnalysis.scala:66)&#xA;    at org.apache.mahout.math.cf.SimilarityAnalysis$.cooccurrencesIDSs(SimilarityAnalysis.scala:141)&#xA;    	at a.test.URAlgorithm.calcAll(URAlgorithm.scala:143)&#xA;    	at a.test.URAlgorithm.train(URAlgorithm.scala:117)&#xA;    	at a.test.URAlgorithm.train(URAlgorithm.scala:102)&#xA;    	at io.prediction.controller.P2LAlgorithm.trainBase(P2LAlgorithm.scala:46)&#xA;    	at io.prediction.controller.Engine$$anonfun$18.apply(Engine.scala:688)&#xA;    at io.prediction.controller.Engine$$anonfun$18.apply(Engine.scala:688)&#xA;    	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)&#xA;    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)&#xA;    	at scala.collection.immutable.List.foreach(List.scala:318)&#xA;    	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)&#xA;    	at scala.collection.AbstractTraversable.map(Traversable.scala:105)&#xA;    	at io.prediction.controller.Engine$.train(Engine.scala:688)&#xA;    	at io.prediction.controller.Engine.train(Engine.scala:174)&#xA;    	at io.prediction.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:65)&#xA;    	at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:247)&#xA;    	at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)&#xA;    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&#xA;    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)&#xA;    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#xA;    	at java.lang.reflect.Method.invoke(Method.java:497)&#xA;    	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)&#xA;    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)&#xA;    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)&#xA;    	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)&#xA;    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)&#xA;Caused by: java.lang.NegativeArraySizeException&#xA;    at org.apache.mahout.math.DenseVector.&amp;lt;init&amp;gt;(DenseVector.java:57)&#xA;    at org.apache.mahout.sparkbindings.SparkEngine$$anonfun$5.apply(SparkEngine.scala:78)&#xA;    	at org.apache.mahout.sparkbindings.SparkEngine$$anonfun$5.apply(SparkEngine.scala:77)&#xA;    	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:706)&#xA;    	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:706)&#xA;    	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)&#xA;    	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)&#xA;    	at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)&#xA;    	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)&#xA;    	at org.apache.spark.scheduler.Task.run(Task.scala:88)&#xA;    	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;    at java.lang.Thread.run(Thread.java:745)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I configure HBase properly ?&lt;/p&gt;&#xA;" OwnerUserId="16096" LastEditorUserId="11097" LastEditDate="2016-03-25T13:07:27.903" LastActivityDate="2017-04-20T21:45:41.107" Title="Problem while training UR on PredictionIO" Tags="|predictive-modeling|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10637" PostTypeId="1" CreationDate="2016-03-10T17:50:00.297" Score="1" ViewCount="2287" Body="&lt;p&gt;I designed a neural network in Torch for prediction of articles. I want to predict, which article will be read by user based on your reading history. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input to the network is tf-idf vector of article and another information about user.&#xA;Output from the network should be tf-idf vector of next article, which user read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tf-idf vector is normalized by SoftMax function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the best solution would be, if a output was a representation of classes but there are a lot of articles and by that there will be a lot of classes. So I represent articles with tf-idf vector on the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried train with DistKLDivCriterion ( Kullback–Leibler divergence ) but it didn't converge. How to design a network for prediction tf-idf vectors? Which criterion should I use? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tmodel = nn.Sequential()                                               &#xA;tmodel:add(nn.Sequencer(nn.FastLSTM(537, 300, rho )))&#xA;tmodel:add(nn.Sequencer(nn.Dropout(0,5)))&#xA;tmodel:add(nn.Sequencer(nn.Linear(300,527))) &#xA;tmodel:add(nn.Sequencer(nn.LogSoftMax(527)))&#xA;&#xA;criterion = nn.SequencerCriterion(nn.DistKLDivCriterion())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15137" LastEditorUserId="15137" LastEditDate="2016-03-10T17:56:42.957" LastActivityDate="2016-04-11T09:37:41.670" Title="How to predict tf-Idf vector with Neural Network?" Tags="|neural-network|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10638" PostTypeId="2" ParentId="10628" CreationDate="2016-03-10T18:04:23.910" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I can’t figure about which variables the lecturer is talking about. Are they weights and biases or are they soft-max function and labels in the Big-loss function?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the case of neural networks, the loss function depends on the weights and the biases (which are often not mentioned extra; they are also weights).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The loss function itself is parametrized heavily. Its parameters is the data (input and labels). Its variables (by which the derivatives are calculated) are weights.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And how would zero mean and equal variance help in optimization?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Have a look at the derivative of the sigmoid function. It is biggest at 0. This means the gradient there can be big. This helps by learning, because the basic learning rule is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$w \gets w_i + \Delta w_i\;\;\; \text{ with } \Delta w_i = - \eta \frac{\partial E}{\partial w_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can get bigger adjustments if you normalize it to have mean of 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The part about the variance ... hm. That's harder to explain. I'm not entirely sure about it. One thought is that you want the data to be in a very restricted, similar domain (independent of the application) so that you can treat results independently from your application. Also, it might help with mini-batches not varying too much.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;in the video the lecturer talks about weight initialization randomly using Gaussian distribution, I cannot understand how can we initialize weights using Gaussian distribution with zero mean and standard deviation sigma?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not sure what exactly the question is.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Practically: Just use a function. For example, &lt;a href=&quot;http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.random.normal.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;numpy.random.normal&lt;/code&gt;&lt;/a&gt; if you're using Python.&lt;/li&gt;&#xA;&lt;li&gt;Mathematically: This is called &lt;em&gt;sampling&lt;/em&gt;. You have a process which has a random distribution and you take examples from that.&lt;/li&gt;&#xA;&lt;li&gt;Technically: If you really want to implement this yourself, you will most likely end up using a random number generator which generates samples from a uniform distribution in [0, 1].&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Once you have those sample numbers, you simply assign those numbers to the weights.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And where optimizer will move the point (initialized weight), up or down to find the local minima?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The optimizer calculates the gradient of the error function. I recommend having a look at the early chapters of the Udacity course. I'm pretty sure this was covered there, too. Another resource would be &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html&quot; rel=&quot;nofollow&quot;&gt;neuralnetworksanddeeplearning.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So can you recommend me a book that I should study before taking this course, so that problems like these don't occur?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Tom Mitchells book &quot;Machine Learning&quot; covers similar topics as the course. &lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-03-10T18:09:27.423" LastActivityDate="2016-03-10T18:09:27.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10639" PostTypeId="2" ParentId="10630" CreationDate="2016-03-10T18:04:42.157" Score="1" Body="&lt;p&gt;I have done some previous work classifying vehicles (heavy or light) based on their driving behaviors. This required calculating speed and accelerations, which you can easily do by using numerical formulas such as the five-point stencil. You already know that points are separated by 0.25 seconds and the distances can be calculated using the haversine formula. More features can be derived such as the driving range, maximum and average speeds, number of left and right turns, hard breaks and sharp accelerations. Try plotting the polylines on Google Earth (or a similar mapping tool) to see if there is a distinctive geographical pattern (are they far apart from each other, or all in the same region?). If patterns are visible, a clustering algorithm may help.&lt;/p&gt;&#xA;" OwnerUserId="3439" LastActivityDate="2016-03-10T18:04:42.157" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10640" PostTypeId="1" CreationDate="2016-03-10T19:39:16.190" Score="22" ViewCount="12836" Body="&lt;p&gt;I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People are performing a lot of feature engineering on these features. I am wondering how exactly one is able perform feature engineering on features which are unknown? Can somebody please help me understand this and some tips on how can I perform feature engineering on unknown features?&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2021-09-08T15:40:43.193" Title="How to perform feature engineering on unknown features?" Tags="|machine-learning|feature-selection|feature-extraction|feature-engineering|kaggle|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10642" PostTypeId="1" CreationDate="2016-03-10T21:01:20.697" Score="13" ViewCount="5416" Body="&lt;p&gt;I am looking to find a pre-trained weights of an already trained models like &lt;strong&gt;Google News data&lt;/strong&gt; etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated.&#xA;Thanks in advance :)&lt;/p&gt;&#xA;" OwnerUserId="16964" LastEditorUserId="16964" LastEditDate="2019-08-18T07:12:01.907" LastActivityDate="2019-08-18T07:12:01.907" Title="Can we take of benefit of using transfer learning while training a word2vec models?" Tags="|machine-learning|bigdata|word2vec|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10643" PostTypeId="1" CreationDate="2016-03-10T21:48:42.187" Score="4" ViewCount="3516" Body="&lt;p&gt;I would like to know how I can measure the performance of an imputation technique. I have read a lot about this. Most literature on the web are applying a classifier after the data has been completed. So this classifier will be used in order to make predictions. However, I am not interested to use such classifier to make predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know how good an imputation method performs on a dataset. We can measure this for example my variance, mean, mean squared error and so on. I would like to know if there are similar techniques to measure the quality of the data. I will use the dataset for descriptive analysis(not for predictive analysis, e.g. training classifier)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I am thinking in the wrong context/corner. Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Regards&lt;/p&gt;&#xA;" OwnerUserId="16966" LastActivityDate="2016-11-16T23:46:35.400" Title="How to measure the performance of an imputation technique" Tags="|data-mining|descriptive-statistics|data-imputation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10645" PostTypeId="2" ParentId="10640" CreationDate="2016-03-10T23:52:45.690" Score="4" Body="&lt;p&gt;You can take different combinations of features such as sum of features: &lt;code&gt;feat_1 + feat_2 + feat_3&lt;/code&gt; ..., or product of those. Or you can transform features by log, or exponential, sigmoid ... or even discretize the numeric feature into a categorical one. It's an infinite space to explore. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whatever combination or transformation that increases your Cross-Validation or Test Set performance then you should use it.&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-10T23:52:45.690" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10646" PostTypeId="2" ParentId="10634" CreationDate="2016-03-11T00:06:47.993" Score="0" Body="&lt;p&gt;Have a look at Simon Haykin's &quot;Neural Networks and Learning Machines&quot;. There is a good treatment of back-propagation there for both sigmoid and radial-basis function networks.&lt;/p&gt;&#xA;" OwnerUserId="16969" LastActivityDate="2016-03-11T00:06:47.993" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10648" PostTypeId="2" ParentId="10637" CreationDate="2016-03-11T01:07:34.517" Score="0" Body="&lt;p&gt;You might want to check out algorithm from &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system&quot; rel=&quot;nofollow&quot;&gt;recommendation system&lt;/a&gt; perspective. It would be more appropriate to use something like &lt;a href=&quot;http://www.libfm.org/&quot; rel=&quot;nofollow&quot;&gt;matrix factorization&lt;/a&gt; (assuming you have more than one user in your dataset), rather than neutral network. TF-IDF is often a very fat matrix, with a lot more columns than rows, so your target variable is a very long vector. Your model eventually suffers from the &lt;a href=&quot;https://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow&quot;&gt;curse of dimensionality&lt;/a&gt; due to lack of training example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you have enough training examples to train the model, the output is the TF-IDF of some document. So you would still have a non-trivial problem of matching this TF-IDF matrix with some document in your corpus.&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-11T01:07:34.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10649" PostTypeId="1" AcceptedAnswerId="10651" CreationDate="2016-03-11T01:09:37.010" Score="2" ViewCount="2507" Body="&lt;p&gt;I have a dataset that has been trained on word2vec. Is it a good idea to cluster the output vectors?.&lt;/p&gt;&#xA;" OwnerUserId="16971" LastActivityDate="2016-03-11T01:21:06.473" Title="Can KMeans clustering be used on word2vec output?" Tags="|clustering|" AnswerCount="1" CommentCount="1" ClosedDate="2016-03-11T09:11:41.260" ContentLicense="CC BY-SA 3.0" />
  <row Id="10650" PostTypeId="2" ParentId="10642" CreationDate="2016-03-11T01:12:33.637" Score="1" Body="&lt;p&gt;Take a look at this paper&lt;a href=&quot;http://www.aclweb.org/anthology/W15-1830&quot; rel=&quot;nofollow&quot;&gt; [PDF]&lt;/a&gt;. The main focus is about NER task but the idea is the same - take pre-trained word2vec vectors and adapt them for a specific application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many common neural network based applications for NLP frequently start with pre-trained vectors. For example a very recent paper &lt;a href=&quot;http://arxiv.org/abs/1603.01354&quot; rel=&quot;nofollow&quot;&gt;[PDF]&lt;/a&gt; (NER and POS tagging tasks) does just this.&lt;/p&gt;&#xA;" OwnerUserId="7848" LastActivityDate="2016-03-11T01:12:33.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10651" PostTypeId="2" ParentId="10649" CreationDate="2016-03-11T01:21:06.473" Score="2" Body="&lt;p&gt;It's totally fine to cluster word2vec output to know semantically similar words. KMeans is an option, you might also want to checkout some approximate neighbor scheme such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow&quot;&gt;Locality Sensitive Hashing&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-11T01:21:06.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10652" PostTypeId="2" ParentId="10626" CreationDate="2016-03-11T05:47:35.290" Score="1" Body="&lt;p&gt;Depending on the software / language you are using, you should be able to implement some form of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Boosting - &lt;a href=&quot;https://en.wikipedia.org/wiki/Boosting_(machine_learning)&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; - (AdaBoost) - Reweights incorrect predictions and re-evaluates the tree&lt;/li&gt;&#xA;&lt;li&gt;Bagging - &lt;a href=&quot;https://en.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; - builds multiple decision trees by repeatedly resampling training data with replacement, and voting the trees for a consensus prediction.&lt;/li&gt;&#xA;&lt;li&gt;Pruning - &lt;a href=&quot;https://en.wikipedia.org/wiki/Pruning_(decision_trees)&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt; - Decision tree algorithms can create overly complex trees with too many branches and nodes that do not generalize well. This is known as overfitting. Pruning removes the extra nodes that do not provide any valuable information.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16166" LastEditorUserId="16166" LastEditDate="2016-03-11T14:06:51.313" LastActivityDate="2016-03-11T14:06:51.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10653" PostTypeId="2" ParentId="19" CreationDate="2016-03-11T06:31:47.807" Score="2" Body="&lt;p&gt;I tend to agree with what @Dan Levin has already said. Ultimately since we want to draw useful insights from the data rather than just storing it, it's the &lt;strong&gt;ability of learning algorithms/systems&lt;/strong&gt; which should determine what is called &quot;Big data&quot;. &#xA;As ML systems evolve what was Big data today will no longer be Big Data tomorrow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of defining Big data could be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Big data&lt;/strong&gt;: Data on which you can't build ML models in &lt;em&gt;reasonable time ( 1-2 hours)&lt;/em&gt; on a typical workstation ( with say 4GB RAM)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Non-Big data&lt;/strong&gt;: complement of the above&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming this definition, as long as the memory occupied by an individual row (all variables for a single data point) does not exceed machine RAM we should be be in the &lt;em&gt;Non-big data&lt;/em&gt; regime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki/v6.1_tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;Vowpal Wabbit&lt;/a&gt; (by far the fastest ML system as of today) can learn on any data set as long as an individual row ( data point) is &amp;lt; RAM ( say 4GB). The number of rows is &lt;strong&gt;not a limitation&lt;/strong&gt; because it uses SGD on multiple cores. Speaking from experience you can train a model with 10k features and 10MN rows on a laptop in a day.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-03-11T06:39:43.660" LastActivityDate="2016-03-11T06:39:43.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10654" PostTypeId="1" CreationDate="2016-03-11T07:35:28.530" Score="2" ViewCount="198" Body="&lt;p&gt;I'd like to convert a set ducoments to term weight matrix(features) by tf-idf.&#xA;Then calculate the similarity of two document by their features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;similarity is compute by &lt;code&gt;result = matrix*matrix.T &amp;gt; 0.9&lt;/code&gt; &lt;a href=&quot;https://stackoverflow.com/questions/17627219/whats-the-fastest-way-in-python-to-calculate-cosine-similarity-given-sparse-mat&quot;&gt;detail here&lt;/a&gt;, and group them by the result(loop result, if &lt;code&gt;result[1,0] &amp;gt;0.9&lt;/code&gt;, then index 1 is similar with index 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have a lot of resource to group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have some books with different names(more complex in fact)&#xA;I can roughly group these books name by similarity, like below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;step1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group 1:&#xA;    1.The Three Body Problem vol 1&#xA;    2.[Chinese]The Three Body Problem no 1&#xA;    3.The Three Body Problem 2&#xA;    4.The Three Body Problem vol 3[Japanese]&#xA;    5.Problem of Three Body vol 3&#xA;    6.(xx)The Three Body Problem 2&#xA;    7.The Three Body Problem 1[English]&#xA;group 2:&#xA;    1.Another book 1&#xA;    ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But &lt;code&gt;xxx 2&lt;/code&gt; and &lt;code&gt;xxx vol3&lt;/code&gt; is needless when I want to find &lt;code&gt;xxx vol 1&lt;/code&gt; , &#xA;So have to do &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;step2:&lt;/strong&gt;&#xA;   tokenize each book name again, use some patterns/rules to extract the book number to distinguish them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to add some term(such as Arabic numerals:0-1, English number:one- twenty) with high dissimilarity weight, to make &lt;strong&gt;step1&lt;/strong&gt; result &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group 1:&#xA;    1.The Three Body Problem vol 1&#xA;    2.[Chinese]The Three Body Problem no 1&#xA;    7.The Three Body Problem 1[English]&#xA;group 2:&#xA;    3.The Three Body Problem 2&#xA;    6.(xx)The Three Body Problem 2&#xA;group 3:&#xA;    4.The Three Body Problem vol 3[Japanese]&#xA;    5.Problem of Three Body vol 3&#xA;group 4:&#xA;    another book 1&#xA;    [xx]another book vol.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are some titles like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. There are 2 man vol.1&#xA;2. (xx)There man 2 boy 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need add a lot detection(number position or something else), that's why I want a way to add a extra weight to somewhere(to avoid step 2, the redundant tokennizing and custom extraction rule).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the similarity weight may work like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tfidf weight two title, plus each number weight, then calculate the similarity matrix.&lt;br&gt;&#xA;But now I am using tfidf matrix power to get similarity matrix,&lt;br&gt;&#xA;I don't know how to add a extra weight to  tfidf weight result, the weight meaning is different between tfidf weight and what mentioned extra weight.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know where and how to add the proper extra weight, how calculate it value?&lt;/p&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-11-13T03:37:37.120" Title="Add extra term weight when grouping strings by similarity?" Tags="|text-mining|similarity|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10656" PostTypeId="2" ParentId="10640" CreationDate="2016-03-11T09:40:57.230" Score="1" Body="&lt;p&gt;Could you please explain more with specific examples about your following statement?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I'm not sure how can you do Feature Engineering without good understanding of your Data Set and the given attributes!!&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tabular data is described in terms of observations or instances (rows) that are made up of variables or attributes (columns). An attribute could be a feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The idea of a feature, separate from an attribute, makes more sense in the context of a problem. A feature is an attribute that is useful or meaningful to your problem. It is an important part of an observation for learning about the structure of the problem that is being modeled.&#xA;... ... ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In computer vision, an image is an observation, but a feature could be a line in the image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In natural language processing, a document or a tweet could be an observation, and a phrase or word count could be a feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In speech recognition, an utterance could be an observation, but a feature might be a single word or phoneme. &quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please visit the following URL for more:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&quot; rel=&quot;nofollow&quot;&gt;Discover Feature Engineering, How to Engineer Features and How to Get Good at It&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16870" LastActivityDate="2016-03-11T09:40:57.230" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10657" PostTypeId="1" CreationDate="2016-03-11T10:11:24.820" Score="1" ViewCount="171" Body="&lt;p&gt;I am new to Data Science but really enthusiastic about it. Really appreciate communities like this one, so thanks in anticipation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I entered a competition to solve a data mining challenge for an e-commerce website. The challenge is as follows: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Build a model based on clickstream data, to detect the phase of the&#xA;  customer in the customer journey (or related intent of customer).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We are not given specific definitions of the customer phases, instead we are supposed to come up with them ourselves based on observable behaviours of users (and maybe academic literature). I believe there are three main shopping states: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Directed buyers - Goal oriented, specific product in mind.&lt;/li&gt;&#xA;&lt;li&gt;Deliberate searchers - Know the basic category, but not a specific product in mind.&lt;/li&gt;&#xA;&lt;li&gt;Hedonic browsers - Similar to window shoppers in real life. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After cleaning up the data, I have a table with session-level summary variables such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Category breadth: How many unique categories a user visited in a given session&lt;/li&gt;&#xA;&lt;li&gt;Product to category ratio: Time spent in category pages vs product pages&lt;/li&gt;&#xA;&lt;li&gt;Session source: (Direct, Search Engine, Social media, Referral)&lt;/li&gt;&#xA;&lt;li&gt;Added item to cart: Yes/No&lt;/li&gt;&#xA;&lt;li&gt;Ordered item: Yes/No&lt;/li&gt;&#xA;&lt;li&gt;etc&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have the raw analytics export data (you can see all variables in the IBM source bellow), so can basically make any other variables I want. But thought these are good enough to explain myself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method would you use in order to come up with Latent Shopping States for these users? Preferably I would like to obtain the probability that each individual user is in a specific shopping state. As well as some sort of weight for the factors that compose each state (e.g. Directed shoppers have a very high Category Breadth)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I thought some sort of clustering algorithm such as K-Means, but that does not give me probabilities. I am now leaning towards Latent Class Analysis, but its still a bit abstract to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I can provide you with any additional information let me know! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Ramachandran, Vandana; Viswanathan, Siva; and Lucas, Hank, &quot;STATE OF&#xA;SHOPPING AND THE VALUE OF INFORMATION: INSIGHTS FROM THE CLICKSTREAM&quot;&#xA;(2010). ICIS 2010 - &lt;a href=&quot;http://aisel.aisnet.org/icis2010_submissions/254/&quot; rel=&quot;nofollow&quot;&gt;http://aisel.aisnet.org/icis2010_submissions/254/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;IBM Digital Analytics Export info - &lt;a href=&quot;http://www.ibm.com/support/knowledgecenter/SSPG9M/Export/Export/digitaldatafeed.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ibm.com/support/knowledgecenter/SSPG9M/Export/Export/digitaldatafeed.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16977" LastActivityDate="2016-03-11T10:11:24.820" Title="Modelling latent online shopping states from e-commerce website clickstream data - What type of method to use?" Tags="|classification|clustering|k-means|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10658" PostTypeId="2" ParentId="10654" CreationDate="2016-03-11T10:26:06.867" Score="1" Body="&lt;p&gt;When you have a similarity measure fixed already, you can use correlation clustering to turn your similarity matrix into groups as you wanted. Here's a good resource about correlation clustering &lt;a href=&quot;http://www.francescobonchi.com/CCtuto_kdd14.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.francescobonchi.com/CCtuto_kdd14.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second step to attempt to make sure volume 1 and volume 2 are not the same, I would add a number of rules that try to find the difference. Look for numerals, if they don't match they have 0 similarity. Play around a bit with this. After you have your new similarity matrix you can use correlation clustering which should seperate volumes 1 and 2 into different groups.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-03-17T07:28:31.967" LastActivityDate="2016-03-17T07:28:31.967" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10659" PostTypeId="2" ParentId="10643" CreationDate="2016-03-11T10:48:15.793" Score="2" Body="&lt;p&gt;I think there is no answer to your question since there is no absolute universal &quot;good&quot;. Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does not alter your answer or at least measure some effects of the missing values. I will give some simple examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One widespread technique is to replace missing values with a marginal central estimator for each variable. This might work for a classifier like decision tree, if there are no missing patterns in your data. However using this imputation would alter your confidence intervals if you study linear relations between variables using regression. This is simply because it will alter sample variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another much sophisticated imputation method is to use EM algorithm to fit the maximum likelihood estimator of the variance-covariance matrix. This estimator is unbiased and using this variance covariance matrix you can recover the linear model in an unbiased way. You then can go to analyze linear relations. But this works only for linear and log linear models and requires a lot of data and also requires to have missing data at random, which is not always the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another one is multiple imputation. What you actually do is to draw data at random for missing values according with a supposed distribution. You do that multiple times, let's say at least 30 times. You analyze each data set with traditional tools and later you aggregate those results into a single set of results. This works well mostly when data is missing at random, but is tedious, sometimes inconsistent and duo to randomness it can produce different results. Ultimately, sometimes is very hard to find proper ways to aggregate the results, it depends on the analysis you use.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-03-11T10:48:15.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10661" PostTypeId="2" ParentId="10637" CreationDate="2016-03-11T13:01:09.090" Score="-1" Body="&lt;p&gt;You should not use neural networks for this kind of problem. Neural network works well for classification problems. Here, your problem is somewhat like recommender system problem. You should search some other algorithm for this problem.&lt;/p&gt;&#xA;" OwnerUserId="16985" LastActivityDate="2016-03-11T13:01:09.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10662" PostTypeId="2" ParentId="2537" CreationDate="2016-03-11T13:03:41.843" Score="0" Body="&lt;p&gt;I have never read (or tried) subsampling techniques on each node of the tree. I do not say that they might not exist, but it looks odd the statement &quot;like is a common practice with random forests&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that, subsampling or bootstrapping the sample for each tree from random forests gives stochastic gradient boosting &lt;a href=&quot;https://statweb.stanford.edu/~jhf/ftp/stobst.pdf&quot; rel=&quot;nofollow&quot;&gt;Friedman, 1999&lt;/a&gt;. They give better results with subsampling than without and more details are given also in &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;nofollow&quot;&gt;Elements of Statistical Learning, page 358, 10.12.2 Subsampling&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-03-11T13:03:41.843" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10663" PostTypeId="2" ParentId="10115" CreationDate="2016-03-11T13:32:10.947" Score="2" Body="&lt;p&gt;The reduce error pruning strategy works in the following way:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train a tree on a training data set&lt;/li&gt;&#xA;&lt;li&gt;Fit a pruning data set (which is different than the training data set) to the tree. What you will have is the tree as was learned at step one, but for each node you will have some instances from the pruning data set.&lt;/li&gt;&#xA;&lt;li&gt;Prune each node from the tree if the misclassification error computed for the instances from the pruned data set is not larger than the original misclassification error rate computed on the training data. For example, suppose that you do a binary classification and you build a tree from a training data set. Now you fit data from the pruning data set and some instances arrives in a given non-leaf node. Suppose that at that node arrives $2$ positive instances and $8$ negative instances. This gives an error rate of $0.2$. Now from that node the instances are split given to the criteria already learned into two children node, one will have $2$ positive instances and $2$ negative instances, and in the other child node goes the other $6$ negative instances. The cumulative error on children is also $0.2$. This means that going further with the split would make no sense, since the data varies a lot. That means you can actually cut the child nodes and make the original non-leaf node a leaf node. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The point of using fresh data (pruning data set) is to check if the split is useful or the noise from data covers the signal given by the split. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To evaluate the performance of both models, before and after pruning, you need another &lt;strong&gt;different&lt;/strong&gt; data set, named validation data. To see way you have to understand that pruning is actually also part of extended training. The only difference than usual learning is that in training with REM you learn how to extend using some data and you learn how to unlearn using some other data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the same data from training to do pruning you simply do not progress (you can stop early). To use for pruning the data set for testing will invalidate you model selection criteria, since it will almost always be the case that the pruning tree performs better (original tree learns from training data set, pruned tree learn from both). I dare to say that if you would like to asset the prediction error, you will have to use a different data set, or envelope everything in a cross validation. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-03-11T13:32:10.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10664" PostTypeId="2" ParentId="10640" CreationDate="2016-03-11T13:34:11.047" Score="20" Body="&lt;p&gt;You do not need domain knowledge (the knowledge of what your data &lt;em&gt;mean&lt;/em&gt;) in order to do feature engineering (finding more expressive ways of &lt;em&gt;framing&lt;/em&gt; your data).&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/a/10645/15501&quot;&gt;Tu N. explained&lt;/a&gt;, you can find &amp;quot;quick and dirty&amp;quot; combinations of features that could be helpful pretty easily. Given an output &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; and an individual feature &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;, you can take the following transforms, &lt;span class=&quot;math-container&quot;&gt;$x' \in \{e^x, \log(x), x^2, x^3, \tanh(x)\}$&lt;/span&gt;. A quick check of the usefulness of the transformation is if the correlation between &lt;span class=&quot;math-container&quot;&gt;$\{y,x'\}$&lt;/span&gt; is higher than the correlation between &lt;span class=&quot;math-container&quot;&gt;$\{y,x\}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Warning on correlation:&lt;/strong&gt; Correlation does not show everything and depending on the model you are using (highly non-linear such as NN or RF) and interaction with other variables, a change in correlation could mean nothing.&lt;/p&gt;&#xA;&lt;p&gt;However, if you are using a simple linear model like logistic regression, it is an OK indicator of performance. The best way to evaluate such a transformation, however, as noted by &lt;a href=&quot;https://datascience.stackexchange.com/a/10656/15501&quot;&gt;Fokhruz Zaman&lt;/a&gt;, would be to build a model with and without your transformed feature, and see how the validation error (on your Cross-Validation folds) evolves.&lt;/p&gt;&#xA;&lt;p&gt;It is rather easy to spot single-feature transformations this way. Those apply to a lot of data, where a more expressive relation between your input and output could be on a different scale. For example, the relationship between &lt;a href=&quot;https://en.wikipedia.org/wiki/Easterlin_paradox#Controversy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Income and &amp;quot;Happiness&amp;quot;&lt;/a&gt; appears to be logarithmic, but you would never record the log of a participant income directly.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Finding combinations of features is more difficult. For a start, if you want to test every addition of 2 features, and you have &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; features, you have an order of &lt;span class=&quot;math-container&quot;&gt;$D^2$&lt;/span&gt; transformations to test. In order to find such transformations, you can apply a nonlinear model (such as NN or RF) to the problem and try to see what it is that it is learning. If you can identify what an intermediate layer in a NN is doing, you can pre-compute its result and add it as a new feature. It will not need to compute it again, and it will probably try to learn something new.&lt;/p&gt;&#xA;&lt;p&gt;It can be hard to interpret the internal representation of a NN, or even interpret feature importance in a Random Forest. An easier, and probably more suited method for this purpose, the model would be &lt;a href=&quot;https://en.wikipedia.org/wiki/Boosting_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boosting&lt;/a&gt; with decision trees. There are a lot of libraries implementing Boosting, and if you are into Kaggle competition as your post seems to imply, &lt;a href=&quot;https://github.com/dmlc/xgboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost&lt;/a&gt; seems used by a lot of participants, so you might find some help/tutorials on what I'm going to describe.&lt;/p&gt;&#xA;&lt;p&gt;First, run your boosting algorithm using &lt;em&gt;only stumps&lt;/em&gt;, 1-level decision trees. Stumps are very weak, but Boosting makes it a reasonable model. This will act as your baseline. Depending on the library you are using, you should be able to display pretty easily which are the most used features, and you should plot them against the response (or do a histogram if the response is categorical) to identify some pattern. This might give you an intuition on what would be a good single feature transformation.&lt;/p&gt;&#xA;&lt;p&gt;Next, run the Boosting algorithm with 2-level decision trees. This model is a lot more complex than the previous one; if two variables taken together have more power than taken individually, this model should outperform your previous one (again, not in terms of training error, but on validation error!). Based on this, you should be able to extract which variables are often used together, and this should lead you to potential multi-feature transformations.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;On related material, I would advise the following videos as they are easy to follow&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=bL4b1sGnILU&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Agnosticism: Feature Engineering Without Domain Expertise; SciPy 2013 Presentation&lt;/a&gt;: An example of how &amp;quot;simple&amp;quot; feature engineering can be.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=IXZKgIsZRm0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Peter Prettenhofer - Gradient Boosted Regression Trees in scikit-learn&lt;/a&gt;: The speaker touches quickly on analyzing feature importance at the end.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15501" LastEditorUserId="122809" LastEditDate="2021-09-08T15:40:43.193" LastActivityDate="2021-09-08T15:40:43.193" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10665" PostTypeId="1" AcceptedAnswerId="10941" CreationDate="2016-03-11T15:03:59.843" Score="5" ViewCount="3052" Body="&lt;p&gt;I need to detect plateaus in time series data online. The data I am working with represents the magnitude of acceleration of a tri-axis accelerometer. I want to find a reference time window that I can use for calibration purposes. Because of that, the system must not move and hence only gravity should influence the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find such plateaus or is there even a more principled approach that I can take?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EnFMy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EnFMy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15690" LastActivityDate="2016-03-29T08:40:20.463" Title="online detection of plateaus in time series" Tags="|time-series|online-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10666" PostTypeId="1" CreationDate="2016-03-11T15:25:33.093" Score="1" ViewCount="841" Body="&lt;p&gt;I have a model with 13 independent variables (all of them are significant and 2 of them are categorical variables) and 678 observations. I was wondering if there is any way to check the coefficients stability. My approach is to start from the model with one variable and then find that coefficients. Then I’ll add another variable by subset selection method and find the new variables and coefficients. I’ll continue until that all variables are in the final model. Hence, I have 15 models (2 categorical variable have 3 levels each) which have different variables (1 to 15 variables including the categorical variables levels). I was wondering if I can show all the coefficients in one graph. For instance, for X1 which is in all models I’ll have 15 coefficients, for X2 which I’ll have 14 coefficients etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback about how can I have all the coefficients in one graph and check their variation for all models or any other approach that help me to check the Regression coefficient stability would be grateful. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastActivityDate="2016-03-11T15:25:33.093" Title="Regression coefficient stability measurement in a multiple linear regression particularly in r?" Tags="|regression|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10667" PostTypeId="1" CreationDate="2016-03-11T16:01:53.930" Score="1" ViewCount="57" Body="&lt;p&gt;I have a model with 13 independent variables (all of them are significant and 2 of them are categorical variables) and 678 observations. I ran the ridge regression and lasso on the data set using glmnet package. Here is the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZHKMg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZHKMg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ll41b.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ll41b.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering how I can relate the variables to their number and what L1 norm mean.&#xA;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="12867" LastActivityDate="2016-03-11T16:01:53.930" Title="How to relate the regularized regression coefficients with their number in the plot?" Tags="|regression|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10668" PostTypeId="1" AcceptedAnswerId="10698" CreationDate="2016-03-11T16:49:04.193" Score="0" ViewCount="106" Body="&lt;p&gt;I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for &lt;code&gt;col1&lt;/code&gt; and all 12 months for column &lt;code&gt;mon&lt;/code&gt; and past 20 years for &lt;code&gt;year&lt;/code&gt;. I then impute values and add new columns, however I need new factors to be added for further analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each &lt;code&gt;mon&lt;/code&gt; and &lt;code&gt;year&lt;/code&gt; combination, a new level should exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample df:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col1 &amp;lt;- c(rep(&quot;a&quot;,4),rep(&quot;b&quot;,4)) &#xA;col2 &amp;lt;- c(1:4)  &#xA;mon &amp;lt;- c(rep(c(&quot;Jan&quot;,&quot;Feb&quot;, &quot;Mar&quot;,&quot;Apr&quot;), 4))  &#xA;year &amp;lt;- c(rep(&quot;2016&quot;,8), rep(&quot;2015&quot;,8))  &#xA;df &amp;lt;- as.data.frame(cbind(col1,col2,mon,year))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;head(df,8)&lt;/strong&gt;  # edited to make it readable&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  col1 col2   mon year  &#xA;1 a    1      Jan 2016  &#xA;2 a    2      Feb 2016  &#xA;3 a    3      Mar 2016  &#xA;4 a    4      Apr 2016  &#xA;5 b    1      Jan 2016  &#xA;6 b    2      Feb 2016  &#xA;7 b    3      Mar 2016  &#xA;8 b    4      Apr 2016  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Expected Output&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   col1 col2   mon year  &#xA;1  a    1      Jan 2016  &#xA;2  a    2      Feb 2016  &#xA;3  a    3      Mar 2016  &#xA;4  a    4      Apr 2016  &#xA;5  b    1      Jan 2016  &#xA;6  b    2      Feb 2016  &#xA;7  b    3      Mar 2016  &#xA;8  b    4      Apr 2016&#xA;9  c    NA     Jan 2016  # New level c for each mon and year&#xA;10 c    NA     Feb 2016  # New level c for each mon and year&#xA;11 c    NA     Mar 2016  # New level c for each mon and year&#xA;12 c    NA     Apr 2016  # New level c for each mon and year&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I go about reaching the expected df?&lt;/p&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="471" LastEditDate="2016-04-12T13:14:30.480" LastActivityDate="2016-04-12T13:14:30.480" Title="Add new factor across multiple groups" Tags="|r|dataset|data-cleaning|data|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10670" PostTypeId="2" ParentId="10626" CreationDate="2016-03-11T21:11:42.930" Score="1" Body="&lt;p&gt;To be short: yes, trash it. Any ML model you use has one job - to predict. If it can't do that job then why on earth would you ever use it? The whole impetus for using ML algorithms is to predict better than guessing randomly. Ideally, much, much better than guessing randomly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Understanding &lt;em&gt;why&lt;/em&gt; a model makes a decision has some value, sure, but only insofar as that decision has value to the end user/solves a business problem.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-03-11T21:11:42.930" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10671" PostTypeId="2" ParentId="10633" CreationDate="2016-03-11T22:36:56.720" Score="4" Body="&lt;p&gt;Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming &quot;unfashionable&quot; as of late due to the prevalence of deep learning, I would suggest that you try using &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow&quot;&gt;Conditional Random Fields (CRF)&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;CRFs offer very competative performance in this space and are often used for named entity recognition, &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Part-of-speech_tagging&quot; rel=&quot;nofollow&quot;&gt;part of speech tagging&lt;/a&gt; and variants thereof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another great thing about CRFs is that they takes into account the position of the current 'field' in a document and also the preceeding and succeeding fields in order to calculate what the current class should be so you don't have to spend too much time engineering structural features into your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had a great deal of success with the library  &lt;a href=&quot;http://www.chokkan.org/software/crfsuite/manual.html&quot; rel=&quot;nofollow&quot;&gt;crfsuite&lt;/a&gt; in particular which is written in C but had a python wrapper. &lt;/p&gt;&#xA;" OwnerUserId="4853" LastActivityDate="2016-03-11T22:36:56.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10672" PostTypeId="1" CreationDate="2016-03-12T00:22:25.197" Score="1" ViewCount="43" Body="&lt;p&gt;I'm searching for the right literature on how to optimally divide a set into a fixed number of bins with pre-determined sizes where the contribution of each element to the objective depends on the other elements in the same bin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in a scheduling context, certain employees work better than others so that the total output of one worker includes both the individual worker's productivity &lt;strong&gt;and&lt;/strong&gt; his/her added (subtracted) productivity due to the other members on the team. Suppose I want to create a fixed $K$ number of teams of size $P$ that maximizes overall productivity, or the one that creates teams with the greatest difference in productivity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example from finance would be one where I want to group assets together to minimize variance. Since assets covary, grouping assets together depends both on the variance of each asset and its covariance with other assets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've considered several streams of literature that don't quite fit:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bin Packing - doesn't fit since I have a fixed number of bins&lt;/li&gt;&#xA;&lt;li&gt;Knapsack - again, fixed number of objects that I want to take, but with the added complexity that the value of an object is both its inherent value and the additional value it gains when taken with something else.&lt;/li&gt;&#xA;&lt;li&gt;$k$-means clustering - I think this is the closest, but the measures of distance does not depend on which other objects are in the same cluster, and also cluster sizes are not fixed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on literature that is related if it is already out there or perhaps thoughts on ways to tackle the problem? If this is the wrong forum, is there a better place to ask this question?&lt;/p&gt;&#xA;" OwnerUserId="16992" LastEditorUserId="924" LastEditDate="2016-03-12T09:07:56.127" LastActivityDate="2016-03-12T09:07:56.127" Title="Extending k-means clustering to include interactions between elements" Tags="|optimization|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10673" PostTypeId="2" ParentId="10631" CreationDate="2016-03-12T05:37:45.277" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Check this one from a &lt;a href=&quot;https://github.com/johnmyleswhite/BanditsBook&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; on MAB&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There's also a &lt;a href=&quot;http://pythonhosted.org/Flask-MAB/&quot; rel=&quot;nofollow&quot;&gt;flask&lt;/a&gt;implementation if you want a web service&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-03-12T05:37:45.277" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10674" PostTypeId="1" AcceptedAnswerId="10675" CreationDate="2016-03-12T15:21:23.430" Score="11" ViewCount="15083" Body="&lt;p&gt;Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem?&lt;/p&gt;&#xA;&lt;p&gt;say for instance I have a dataset with the following format where the target variable takes 1 or 0:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; dt.head()&#xA;   ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \&#xA;0   1     2     23                   0                        0   &#xA;1   3     2     34                   0                        0   &#xA;2   4     2     23                   0                        0   &#xA;3   8     2     37                   0                      195   &#xA;4  10     2     39                   0                        0   &#xA;&#xA;   imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  TARGET  &#xA;0                        0                        0       0  &#xA;1                        0                        0       0  &#xA;2                        0                        0       0  &#xA;3                      195                        0       0  &#xA;4                        0                        0       0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Computing the correlation matrix gives the following values&lt;/p&gt;&#xA;&lt;div class=&quot;s-table-container&quot;&gt;&#xA;&lt;table class=&quot;s-table&quot;&gt;&#xA;&lt;thead&gt;&#xA;&lt;tr&gt;&#xA;&lt;th&gt;ID&lt;/th&gt;&#xA;&lt;th&gt;var3&lt;/th&gt;&#xA;&lt;th&gt;var15&lt;/th&gt;&#xA;&lt;th&gt;imp_ent_var16_ult1&lt;/th&gt;&#xA;&lt;th&gt;imp_op_var39_comer_ult1&lt;/th&gt;&#xA;&lt;th&gt;imp_op_var39_comer_ult3&lt;/th&gt;&#xA;&lt;th&gt;imp_op_var40_comer_ult1&lt;/th&gt;&#xA;&lt;th&gt;TARGET&lt;/th&gt;&#xA;&lt;/tr&gt;&#xA;&lt;/thead&gt;&#xA;&lt;tbody&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;ID&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;-0.00102533166614&lt;/td&gt;&#xA;&lt;td&gt;-0.00213549813966&lt;/td&gt;&#xA;&lt;td&gt;-0.00311137548461&lt;/td&gt;&#xA;&lt;td&gt;-0.00143645708778&lt;/td&gt;&#xA;&lt;td&gt;-0.00413114484307&lt;/td&gt;&#xA;&lt;td&gt;-0.00727672024906&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;var3&lt;/td&gt;&#xA;&lt;td&gt;-0.00102533166614&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;-0.00445177129541&lt;/td&gt;&#xA;&lt;td&gt;0.0018681447614&lt;/td&gt;&#xA;&lt;td&gt;0.00598903116859&lt;/td&gt;&#xA;&lt;td&gt;0.00681691701467&lt;/td&gt;&#xA;&lt;td&gt;0.00151753041397&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;var15&lt;/td&gt;&#xA;&lt;td&gt;-0.00213549813966&lt;/td&gt;&#xA;&lt;td&gt;-0.00445177129541&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;0.0437222608106&lt;/td&gt;&#xA;&lt;td&gt;0.0947624170998&lt;/td&gt;&#xA;&lt;td&gt;0.101177078747&lt;/td&gt;&#xA;&lt;td&gt;0.0427540973727&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;imp_ent_var16_ult1&lt;/td&gt;&#xA;&lt;td&gt;-0.00311137548461&lt;/td&gt;&#xA;&lt;td&gt;0.0018681447614&lt;/td&gt;&#xA;&lt;td&gt;0.0437222608106&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;0.0412213212518&lt;/td&gt;&#xA;&lt;td&gt;0.0348787079026&lt;/td&gt;&#xA;&lt;td&gt;0.00989582043194&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;imp_op_var39_comer_ult1&lt;/td&gt;&#xA;&lt;td&gt;-0.00143645708778&lt;/td&gt;&#xA;&lt;td&gt;0.00598903116859&lt;/td&gt;&#xA;&lt;td&gt;0.0947624170998&lt;/td&gt;&#xA;&lt;td&gt;0.0412213212518&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;0.886476049204&lt;/td&gt;&#xA;&lt;td&gt;0.342709191344&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;imp_op_var39_comer_ult3&lt;/td&gt;&#xA;&lt;td&gt;-0.00413114484307&lt;/td&gt;&#xA;&lt;td&gt;0.00681691701467&lt;/td&gt;&#xA;&lt;td&gt;0.101177078747&lt;/td&gt;&#xA;&lt;td&gt;0.0348787079026&lt;/td&gt;&#xA;&lt;td&gt;0.886476049204&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;td&gt;0.316671244555&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;imp_op_var40_comer_ult1&lt;/td&gt;&#xA;&lt;td&gt;-0.00727672024906&lt;/td&gt;&#xA;&lt;td&gt;0.00151753041397&lt;/td&gt;&#xA;&lt;td&gt;0.0427540973727&lt;/td&gt;&#xA;&lt;td&gt;0.00989582043194&lt;/td&gt;&#xA;&lt;td&gt;0.342709191344&lt;/td&gt;&#xA;&lt;td&gt;0.316671244555&lt;/td&gt;&#xA;&lt;td&gt;1.0&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;tr&gt;&#xA;&lt;td&gt;TARGET&lt;/td&gt;&#xA;&lt;td&gt;0.0031484687227&lt;/td&gt;&#xA;&lt;td&gt;0.00447479817554&lt;/td&gt;&#xA;&lt;td&gt;0.101322098561&lt;/td&gt;&#xA;&lt;td&gt;-1.74602537678e-05&lt;/td&gt;&#xA;&lt;td&gt;0.0103531295754&lt;/td&gt;&#xA;&lt;td&gt;0.0035169224417&lt;/td&gt;&#xA;&lt;td&gt;0.00311938694896&lt;/td&gt;&#xA;&lt;/tr&gt;&#xA;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;/div&gt;&#xA;&lt;p&gt;Is it valid, to dismiss all features where the correlation with target is lower than a threshold (say for instance, 0.1)?&lt;/p&gt;&#xA;&lt;p&gt;What if there is a strong inter-attributes correlation as high as 1 where the correlated attributes are continuous variables, does this mean that these features hold redundant information for the learner? can I safely remove one of them without risking to lose information?&lt;/p&gt;&#xA;" OwnerUserId="9774" LastEditorUserId="103781" LastEditDate="2022-10-26T16:12:00.190" LastActivityDate="2022-10-26T16:12:00.190" Title="Dissmissing features based on correlation with target variable" Tags="|feature-selection|correlation|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10675" PostTypeId="2" ParentId="10674" CreationDate="2016-03-12T15:47:06.863" Score="17" Body="&lt;p&gt;You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Low correlation means there's no &lt;em&gt;linear&lt;/em&gt; relationship; it doesn't mean there's no information in the feature that predicts the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you're really looking for &lt;em&gt;mutual information&lt;/em&gt;, in this case between continuous and categorical variables. (I assume your other inputs are continuous?) This is a little involved; see &lt;a href=&quot;https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical&quot;&gt;https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're attempting to do feature selection then you could perform a logistic regression with L1 regularization and select features based on the absolute value of their coefficients.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-03-12T15:47:06.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10676" PostTypeId="1" AcceptedAnswerId="10677" CreationDate="2016-03-12T17:44:12.350" Score="2" ViewCount="1447" Body="&lt;p&gt;I have a set of variables that I want to use for a regression or a classification problem. &#xA;Having computed the correlation matrix of these variables, I discovered that some of them has an inter-variables Pearson correlation&#xA;values as high as 1. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this mean that these variables hold redundant information for the learner?&lt;/li&gt;&#xA;&lt;li&gt;Is it safe to remove one of them without risking information loss? if yes, how to chose the one to remove? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9774" LastActivityDate="2016-03-14T16:08:21.680" Title="Detecting redundancy with Pearson correlation in continuous features" Tags="|feature-selection|correlation|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10677" PostTypeId="2" ParentId="10676" CreationDate="2016-03-12T18:52:20.617" Score="4" Body="&lt;p&gt;If the correlation between two features $x_1$ and $x_2$ is 1 that means that you can write $x_1 = c\cdot x_2 + a$. The only knew knowledge there is are those two constants, the individual values can be retrieved knowing this. I highly doubt there is anything a machine learning algorithm can learn from this and it is a fact that for some having this kind of correlation between features can hurt your performance quite a bit, so I would test it a bit but I would say it's very likely you can remove one of the two, and which one is not going to matter.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-03-12T18:52:20.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10678" PostTypeId="1" CreationDate="2016-03-12T19:49:04.710" Score="2" ViewCount="213" Body="&lt;p&gt;I'm trying to resolve a paradox. Suppose that I have a bunch of data points $\{x_i,y_i\}$ and calculate a slope and intercept, $m$ and $b$ such that $y=mx+b$. Also, I can calculate the error in $m$, $\sigma_m$ and the error in $b$, $\sigma_b$. Suppose further that the true model is a line $y=ax+c +\epsilon$ where $\epsilon$ is an error term that is normally distributed and has a standard deviation of $\sigma$. If I were to calculate the error in my calculated value as a result of the errors in my coefficients I would get $error^2 = \sigma_m^2 x^2 +\sigma_b^2$. This error increases with increasing $x$, a very strange property given that the standard deviation of y at any given point x is constant, i.e. $\sigma$. What am I missing? What is the resolution of this paradox? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: The standard error of the slope $\sigma_m$ is given by $\frac{\sigma^2}{\sum_i (x_i -\bar{x})^2}$ so $\sigma_m$ goes down inversely with the standard deviation of the X values. I thought that perhaps this resolved the paradox. However, x,and $\bar{x}$ are not constrained by the standard deviations. All of the x values can be within two units of each other and yet x may be a thousand, million, billion or more. &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="12443" LastEditDate="2016-03-12T19:57:55.140" LastActivityDate="2016-04-12T06:44:08.780" Title="Error of linear regression as a result of the coefficients" Tags="|regression|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10679" PostTypeId="1" AcceptedAnswerId="10965" CreationDate="2016-03-12T20:43:03.863" Score="4" ViewCount="2415" Body="&lt;p&gt;I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the Google DeepMind &amp;amp; AlphaGo's stuff) to parse and extract information from text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, could it be a competitive approach to structured prediction such as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Named Entity Recognition (NER), i.e. the task of labelling New York by &quot;city&quot;, and New York Times by &quot;organization&quot;&#xA;Part-of-speech tagging (POS), i.e. classifying words as determinant, noun, etc.&#xA;information extraction, i.e. finding and labelling some target information in texts, for instance 12/03 is date given the context meaning 3 December and has the label &quot;expiry date&quot;&#xA;What would be a relevant modelling to do these tasks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather naively I would think of a pointer that read the text from start to end and annotate each 'letter' by a label. Maybe it would learn that neighbouring letters in a 'word' share the same label, etc. Would it be able to learn long-term dependencies with this approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested by any ideas or references related to this subject.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastEditorUserId="-1" LastEditDate="2016-08-21T10:55:00.863" LastActivityDate="2019-03-12T09:27:04.767" Title="Information extraction with reinforcement learning, feasible?" Tags="|text-mining|reinforcement-learning|parsing|named-entity-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10681" PostTypeId="2" ParentId="10184" CreationDate="2016-03-13T05:58:55.897" Score="1" Body="&lt;p&gt;ICD 10 is a vast generalization of ICD 9, in that each procedure and diagnostic type now has many more codes. In addition, there are new diagnostics and procedures in ICD 10 that are not in ICD 9. ICD 9 is forwards compatible with ICD 10, if you settle on the approximate ICD10 code that's most similar &lt;a href=&quot;http://icd10coded.com/convert/?s=aw&amp;amp;gclid=CLenkbv8vMsCFceBfgodXfwPVw&quot; rel=&quot;nofollow&quot;&gt;for example here&lt;/a&gt;. If you try  ICD 9: 81.54 (total knee replacement), then you'll see a list of approximations &lt;a href=&quot;http://icd10coded.com/convert/?code=81.54&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. As you can see, the specificity of ICD 10 is something you'll have to compromise on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ICD 10 will not in general be backwards compatible, except in cases similar to the above knee replacement, where you'd settle on something. One compromise here would be to use &lt;a href=&quot;https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccsfactsheet.jsp&quot; rel=&quot;nofollow&quot;&gt;Clinical Classification Software (CCS)&lt;/a&gt; grouping which collapses ICD 9 and 10 codes into multilevel groups (for example, &quot;Knee Arthroscopy&quot;) , so you can compare procedures and diagnostics by groups&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-03-13T05:58:55.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10682" PostTypeId="2" ParentId="10678" CreationDate="2016-03-13T06:05:25.770" Score="1" Body="&lt;p&gt;You have some inherent error in $m$, say that you predict $m=m_0+a$, where $m_0$ is the true value and where you can control the size of $a$ in terms of your error bounds on $m$. Then you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=(m_0+a)x+b=m_0x+ax+b.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $ax$ term is what will cause your error to grow as you increase/decrease $x$ toward $\pm\infty$. In other words, your predicted line will ultimately be infinitely far from the true line (as can be visualized by drawing two lines with slightly different slopes).&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-03-13T06:05:25.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10683" PostTypeId="1" CreationDate="2016-03-13T07:29:41.990" Score="1" ViewCount="214" Body="&lt;p&gt;Currently I'm writing some models to predict outcomes of team games, but I'm having a hard time featurizing the teams so that the order of teams doesn't matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, currently I duplicate all the data and reverse it to reduce the effect of ordering on the outcome, but that's not proving perfect. If I have &quot;Ravens vs Panthers = Panthers win&quot;, I also put in &quot;Panthers vs Ravens = Panthers wins&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I then go to my model and ask for the output of Panthers vs Ravens and Ravens vs Panthers, I get different (but similar) results. For example the probabilities of Panther's winning may be 75% in the first ane 72% in the second.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How else might I make it 100% ambiguous as to which team is team 1 and which is team 2?&lt;/p&gt;&#xA;" OwnerUserId="17006" LastActivityDate="2016-08-10T16:40:40.590" Title="Predicting outcome of games - how to make team 1/2 interchangable" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10684" PostTypeId="2" ParentId="10683" CreationDate="2016-03-13T09:32:44.693" Score="0" Body="&lt;p&gt;The simplest solution would be to average the predictions from &quot;Team A vs Team B&quot; and &quot;Team B vs Team A&quot; as inputs, and consider that part of your algorithm. You are already doing something similar when augmenting training data by considering both arrangements. This solution is also used by image classification models, which may output predictions averaged over more than one random patch from an input image, to get better translation invariance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inherently the problem you face is one of dimensionality - the input features are linked by some symmetry (e.g. Team A's defensive strength is equivalent to Team B's defensive strength), but this is not encoded into the learning model. Depending on your model class, you might be able to link weights or make adjustments to the features to better encode the symmetry. But this is a lot more work than creating a simple meta-model that predicts on both input arrangements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another simple alternative is to have a sorting algorithm apply between the features, such that which is Team A and which Team B as presented to the ML and prediction is decided by some function of the feature values (e.g. always make the team with highest value of a specific predictive feature the first team). This makes the training and prediction deterministic and avoids the issue of needing to treat the teams equivalently. In effect it breaks the symmetry, instead of &quot;Team A&quot; and &quot;Team B&quot; features you have &quot;Team with highest X&quot; and &quot;Team with lowest X&quot; features, and predict whether &quot;Team with highest X&quot; will win. A disadvantage of this approach is that which feature or rule you pick to sort by becomes a meta-param of your ML routine. It may help to pick a feature with a known strong influence on the result (sorting by team name is not likely to be effective). &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-03-13T09:49:14.040" LastActivityDate="2016-03-13T09:49:14.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10685" PostTypeId="2" ParentId="10674" CreationDate="2016-03-13T12:41:54.463" Score="3" Body="&lt;p&gt;Please note that Pearson correlation (and mutual information) considers the concept and the single feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are cases in which a single feature is useless but given more features it becomes important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy).   &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-03-13T12:41:54.463" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10687" PostTypeId="1" AcceptedAnswerId="10691" CreationDate="2016-03-13T17:46:59.960" Score="0" ViewCount="30" Body="&lt;p&gt;I am interested what type of combinatorics is using for following bookmakers system called &quot;S3&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We have  N={1..8} &lt;code&gt;events&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We build &lt;code&gt;express pairs&lt;/code&gt; C=8 like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[(1, 2, 4), (1, 3, 6), (1, 6, 8), (2, 3, 5),&#xA; (2, 5, 8), (3, 7, 8), (4, 5, 7), (4, 6, 7)]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Each &lt;code&gt;event&lt;/code&gt; repeats 3-times: 1 =&gt;  (&lt;strong&gt;1&lt;/strong&gt;, 2, 4), (&lt;strong&gt;1&lt;/strong&gt;, 3, 6), (&lt;strong&gt;1&lt;/strong&gt;, 6, 8)..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not &lt;code&gt;permutation&lt;/code&gt;, &lt;code&gt;combination&lt;/code&gt;... Please advice what is it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want as example generate in similar way &lt;code&gt;4 from 10&lt;/code&gt; if this possible. After I'd like to predict the optimal way for building the `pairs'&lt;/p&gt;&#xA;" OwnerUserId="14684" LastActivityDate="2016-03-14T01:19:41.613" Title="Combinatoric system which using in bookies &quot;S3&quot; system" Tags="|machine-learning|data-mining|classification|clustering|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10690" PostTypeId="1" CreationDate="2016-03-14T00:37:25.940" Score="4" ViewCount="704" Body="&lt;p&gt;I'm working on a project involved with identifying different types of sounds (such as screams, singing, and bangs) from each other.  We've got our data a reasonable number of different transformations (e.g.: spectrograms, chromagrams, MFCCs, etc.), but since most of our features are 2-dimensional matrices (some are actually 1-dimensional vectors), &lt;strong&gt;we'd like to reduce this information in some way, so that the machine learning we're hoping to do takes a &quot;feasible&quot; amount of time.&lt;/strong&gt;  &lt;em&gt;However, I don't quite know enough about math and statistics to make an educated decision on this.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our data consists of small sound files from ~1-10 seconds long.  There are recordings of screams, singing, bangs (and other man-made noises), and birds (and other natural noises).  We are hoping to be able to differentiate and identify each source type from the others.  See &lt;a href=&quot;https://github.com/BenSandeen/surveillance_sound_classifier/blob/master/Project.ipynb&quot; rel=&quot;nofollow&quot;&gt;https://github.com/BenSandeen/surveillance_sound_classifier/blob/master/Project.ipynb&lt;/a&gt; for the different plots we have made to guide our selection of features to use.  Focus mainly on the 3x3 plots, as that's where the comparisons are being made.  These plots are primarily &lt;strong&gt;time vs. frequency, with amplitude represented by color.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking that maybe we could &quot;collapse&quot; each matrix down to a vector by somehow choosing some representative frequency/amplitude-related feature at each time slice (we're using &lt;em&gt;Short Time Fourier Transforms&lt;/em&gt; to analyze the sounds) and then get a vector of some length, containing a bunch of scalars.  Although this could make accounting for different lengths of sounds difficult.  Would it be reasonable to just set the shorter sounds' vectors to be filled in with zeros if they have no useful data?  That would effectively make these sounds a projection onto some lower-dimensional space.  Then, maybe we could just use dot products to compare the vectors; if they're parallel, they'll have a large for product, but of they're nearly perpendicular, they'll have a dot product near zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, I was thinking that something like a trace of our matrices, or finding their characteristic polynomial, might be useful direction to pursue.  I've read a bit about &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow&quot;&gt;PCA&lt;/a&gt;, but I'm not quite understanding it enough to know if this may be what I'm looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone think of any other ways of handling and reducing this data?  For what it's worth, we're currently planning on using Sci-Kit Learn (sorry that I can't use more than 2 links yet) to perform our machine learning.&lt;/p&gt;&#xA;" OwnerUserId="17022" LastEditorUserId="17022" LastEditDate="2016-03-14T05:22:52.177" LastActivityDate="2016-05-04T22:03:07.247" Title="How to reduce dimensionality of audio data that comes in form of matrices and vectors?" Tags="|python|scikit-learn|dimensionality-reduction|feature-engineering|audio-recognition|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10691" PostTypeId="2" ParentId="10687" CreationDate="2016-03-14T01:19:41.613" Score="1" Body="&lt;p&gt;This can be interpreted as an unusual error correcting code with a non-binary signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, you want the tuples to be very different, i.e. there are no two tuples that agree in two positions. That is also a simple strategy for producing such codes (the more interesting question is what (x,y) combination is best.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of it this way: assuming you only know part of the tuple. Say the first is unknown: (?, 2, 4) then you want to be able still conclude the correct triple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Forward_error_correction&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Forward_error_correction&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-03-14T01:19:41.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10692" PostTypeId="1" CreationDate="2016-03-14T07:06:05.487" Score="3" ViewCount="1688" Body="&lt;p&gt;The Neural Network language Model (NNLM) by &lt;a href=&quot;http://www.cs.columbia.edu/~blei/seminar/2016_discrete_data/readings/BengioDucharmeVincentJanvin2003.pdf&quot; rel=&quot;nofollow&quot;&gt;Bengio et.al&lt;/a&gt; is a structure extensively used in machine translation, text summarization based on deep learning. What's the computational complexity of this model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing the complexity in terms of number of parameters helps in choosing the size of the training set and in determining what compute infrastructure is required.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2018-10-26T09:14:22.120" LastActivityDate="2018-10-26T09:14:22.120" Title="Number of parameters in Neural Network Language Model" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10693" PostTypeId="2" ParentId="10692" CreationDate="2016-03-14T08:32:13.420" Score="1" Body="&lt;p&gt;NNLM has following sets of parameters ). &#xA;Using $V$ to denote the number of words in the vocabulary:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The matrix which creates the embedding for each word in the context . This distributed embedding is in $\mathbb{R}^{m}$ space. This matrix is $C :{V \times m}$&lt;/li&gt;&#xA;&lt;li&gt;The matrix which transforms the concatenated list of word embeddings (active in the current context of size $n-1$) to the hidden layer (of size $h$). This matrix is $H:{(n-1)m\times h}$&lt;/li&gt;&#xA;&lt;li&gt;The matrix which maps the hidden layer to unmormalized probabilities for each word in the vocabulary. This matrix is $V :{V \times h}$&lt;/li&gt;&#xA;&lt;li&gt;The matrix connecting the context word embeddings to the output layer. This connection is optional and has dimensions $W:{V \times (n-1)m}$&#xA;Note: while training using SGD, for  a single example only $n-1$ words in the context are active out of words $V$ in the vocabulary. I have also omitted the parameter vectors for the bias terms $b:V \times 1$   when computing unnormalized outputs and when computing the hidden layer ($d: h \times 1$).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hence my current understanding is that  number of parameters in NNLM is:&#xA;$$dim(C)+ dim(H)+ dim(V) + dim(W)= (n-1)m \times V + h \times (n-1)m+ V \times h + V \times (n-1)m$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xqHzq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xqHzq.png&quot; alt=&quot;NNLM&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-03-19T03:43:14.667" LastActivityDate="2016-03-19T03:43:14.667" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10694" PostTypeId="1" CreationDate="2016-03-14T09:27:38.383" Score="1" ViewCount="316" Body="&lt;p&gt;I am a beginner in data science. I have a data set of drivers that has the following attributes available- &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Time stamp&lt;/li&gt;&#xA;&lt;li&gt;Speed&lt;/li&gt;&#xA;&lt;li&gt;Acceleration&lt;/li&gt;&#xA;&lt;li&gt;GPS co-ordinates&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I need to build a driver rating system to rate drivers on a scale of 1-100 on parameter of speeding. The data set has the data from many drivers and also their past data. &#xA;Which is the best algorithm that I can use in this situation?&lt;/p&gt;&#xA;" OwnerUserId="17034" LastActivityDate="2016-04-13T22:20:20.450" Title="Algorithms to generate a rating system based on history" Tags="|data-mining|predictive-modeling|recommender-system|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10695" PostTypeId="1" CreationDate="2016-03-14T09:47:28.813" Score="21" ViewCount="44285" Body="&lt;p&gt;I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset pretrained model). I have been struggling with it for a couple of weeks. Now, I just found out that in gesim there is a function that can help me initialize the weights of my model with pre-trained model weights.&lt;/p&gt;&#xA;&lt;p&gt;That is mentioned below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;reset_from(other_model)&#xA;&#xA;    Borrow shareable pre-built structures (like vocab) from the other_model. Useful if testing multiple models in parallel on the same corpus.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I do not know this function can do the same thing or not. Please help!&lt;/p&gt;&#xA;" OwnerUserId="16964" LastEditorUserId="29169" LastEditDate="2021-03-11T19:05:12.607" LastActivityDate="2021-03-11T19:05:37.727" Title="How to initialize a new word2vec model with pre-trained model weights?" Tags="|python|nlp|word-embeddings|word2vec|gensim|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="10696" PostTypeId="2" ParentId="10694" CreationDate="2016-03-14T10:35:42.447" Score="0" Body="&lt;p&gt;You seem to be trying to do supervised learning without having labeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a general understanding about Machine Learning? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually the ML problems are divided into 2 categories:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Supervised Learning&lt;/strong&gt;: Where you have data samples and a label for each sample, the goal is then to predict what are the labels of new samples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Unsupervised Learning&lt;/strong&gt;: You just have the data samples and no labels, you want to find some way to group the data or to find some patterns in it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You should check the &lt;a href=&quot;http://scikit-learn.org/stable/tutorial&quot; rel=&quot;nofollow&quot;&gt;scikit-learn tutorials&lt;/a&gt; there you can learn the basics about ML and probably learn what you need for your project. &lt;/p&gt;&#xA;" OwnerUserId="11121" LastActivityDate="2016-03-14T10:35:42.447" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10697" PostTypeId="2" ParentId="10694" CreationDate="2016-03-14T10:59:29.430" Score="0" Body="&lt;p&gt;You need examples if you want to regress ratings. I would get someone with domain knowledge (product owner for example) to take a look at a subset of your data, let them rate those data points and use this subset to train a model on your features and maybe some added artificial features. Then you can use this model to rate the other drivers.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-03-14T10:59:29.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10698" PostTypeId="2" ParentId="10668" CreationDate="2016-03-14T13:01:54.977" Score="1" Body="&lt;p&gt;Several possibilities. For example, to add &lt;code&gt;c&lt;/code&gt; for existing &lt;code&gt;mon&lt;/code&gt;-&lt;code&gt;year&lt;/code&gt; combinations in your data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rbind(df, transform(df[!duplicated(df[, 3:4]), ], col1=&quot;c&quot;, col2=NA))&#xA;#     col1 col2 mon year&#xA;# 1      a    1 Jan 2016&#xA;# 2      a    2 Feb 2016&#xA;# 3      a    3 Mar 2016&#xA;# 4      a    4 Apr 2016&#xA;# 5      b    1 Jan 2016&#xA;# 6      b    2 Feb 2016&#xA;# 7      b    3 Mar 2016&#xA;# 8      b    4 Apr 2016&#xA;# 9      a    1 Jan 2015&#xA;# 10     a    2 Feb 2015&#xA;# 11     a    3 Mar 2015&#xA;# 12     a    4 Apr 2015&#xA;# 13     b    1 Jan 2015&#xA;# 14     b    2 Feb 2015&#xA;# 15     b    3 Mar 2015&#xA;# 16     b    4 Apr 2015&#xA;# 17     c &amp;lt;NA&amp;gt; Jan 2016&#xA;# 21     c &amp;lt;NA&amp;gt; Feb 2016&#xA;# 31     c &amp;lt;NA&amp;gt; Mar 2016&#xA;# 41     c &amp;lt;NA&amp;gt; Apr 2016&#xA;# 91     c &amp;lt;NA&amp;gt; Jan 2015&#xA;# 101    c &amp;lt;NA&amp;gt; Feb 2015&#xA;# 111    c &amp;lt;NA&amp;gt; Mar 2015&#xA;# 121    c &amp;lt;NA&amp;gt; Apr 2015&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To add &lt;code&gt;c&lt;/code&gt; for all possible combinations of existing &lt;code&gt;mon&lt;/code&gt; values and existing &lt;code&gt;year&lt;/code&gt; values: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rbind(df, data.frame(col1=&quot;c&quot;, col2=NA, expand.grid(mon=levels(df$mon), year=levels(df$year))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To add &lt;code&gt;c&lt;/code&gt; for all possible combinations of all months names and existing &lt;code&gt;year&lt;/code&gt; values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rbind(df, data.frame(col1=&quot;c&quot;, col2=NA, expand.grid(mon=month.abb, year=levels(df$year))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on.&lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-03-14T13:01:54.977" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10699" PostTypeId="1" CreationDate="2016-03-14T14:33:11.313" Score="2" ViewCount="217" Body="&lt;p&gt;I have tried to train a neural network for a simple x^2 function    &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I developed training data in excel. First column (X) is =RANDBETWEEN(-5,5) i.e random integer between -5 and 5&lt;/li&gt;&#xA;&lt;li&gt;Second column simply squares first column&lt;/li&gt;&#xA;&lt;li&gt;And third column, my output 'y' column is 0 or 1. 0 if second column is less than 12.5 else 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I made 850 training examples and used the first column as 'X' and third column as 'y'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However &lt;strong&gt;I am only able to get a training accuracy of 63%!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where could I have gone wrong? I changed input_layer to 1 and tried hidden units between 5 and 35. Tried regularization lambda 0 to 2 but still only 63% accuracy! Where could I have gone wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My predict function is p = 1 if h2(i)&gt;0.5 else 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be much appreciated! :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I also noticed that my neural network's output is 0.3XXX for all training examples&lt;/strong&gt;...how is this possible?? &lt;/p&gt;&#xA;" OwnerUserId="17047" LastActivityDate="2016-03-17T16:31:16.180" Title="Applying neural network for simple x^2 function for demonstration purpose" Tags="|machine-learning|classification|neural-network|beginner|octave|" AnswerCount="2" CommentCount="7" FavoriteCount="0" ClosedDate="2016-03-18T13:20:58.333" ContentLicense="CC BY-SA 3.0" />
  <row Id="10700" PostTypeId="2" ParentId="10642" CreationDate="2016-03-14T15:04:59.870" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;Distributed representations (Glove)&lt;/em&gt;&lt;/a&gt; based on training on a large corpus are directly available from Stanford NLP group. You can use those word embeddings directly in your application (instead of using 1 hot encoded vectors and then training the network to get the embeddings). If your task is not too specialized starting with this set of embeddings will work well in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will &lt;strong&gt;save you from training an additional $m \times V$ number of parameters&lt;/strong&gt; where $V$ is the vocabulary size and  $m$ is the dimension of the embedding space you want to project into.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-03-19T03:39:37.863" LastActivityDate="2016-03-19T03:39:37.863" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10701" PostTypeId="2" ParentId="10223" CreationDate="2016-03-14T15:44:26.867" Score="0" Body="&lt;p&gt;I found an answer to my question using the package partykit, I post it to help others.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tree_party &amp;lt;- as.party(tree) #coercing the rpart.object into partykit&#xA;&#xA;tree_fit &amp;lt;- fitted(tree_party) #a two-column data.frame with the fitted node numbers and the observed responses on the training data.&#xA;&#xA;tree_pred &amp;lt;- predict(tree_party, newdata = data, type = &quot;node&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16152" LastActivityDate="2016-03-14T15:44:26.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10702" PostTypeId="2" ParentId="10676" CreationDate="2016-03-14T16:08:21.680" Score="2" Body="&lt;p&gt;Yes and Yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variance Inflation Factor is a common method for addressing your concerns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/177498&quot;&gt;vif answer&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Variance_inflation_factor&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Variance_inflation_factor&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation based feature selection is another approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection#Correlation_feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Feature_selection#Correlation_feature_selection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8650" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-03-14T16:08:21.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10703" PostTypeId="1" CreationDate="2016-03-14T16:16:25.640" Score="1" ViewCount="58" Body="&lt;p&gt;I am a beginner and I have developed code in Octave to train a neural network (as part of Andrew Ng's coursera course).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to generate some datasets in Excel for simple mathematical functions and try and train my neural network implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried y = x^2 and y = sin(x), &lt;strong&gt;but my training accuracy is only around 67%.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have varied the number of hidden layer units, varied the lambda, but still don't get good results. Where could I be wrong? &lt;/p&gt;&#xA;" OwnerUserId="17047" LastActivityDate="2016-03-14T17:23:50.040" Title="Please suggest some simple functions that I can use to test my neural network implementation" Tags="|machine-learning|neural-network|beginner|matlab|octave|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10704" PostTypeId="2" ParentId="9460" CreationDate="2016-03-14T16:32:42.607" Score="-2" Body="&lt;p&gt;There is a simpler solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  data.columns = data.columns.str.split('_', expand=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To arrange column names one can also do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; data.sort_index(axis=1, inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To change column levels:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; data = data.reorder_levels([1,0], axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17049" LastEditorUserId="17049" LastEditDate="2016-03-14T16:42:16.703" LastActivityDate="2016-03-14T16:42:16.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10705" PostTypeId="2" ParentId="8032" CreationDate="2016-03-14T17:07:51.323" Score="22" Body="&lt;p&gt;Just use &lt;code&gt;predict_proba&lt;/code&gt; instead of &lt;code&gt;predict&lt;/code&gt;. You can leave the objective as &lt;code&gt;binary:logistic&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2016-03-14T17:07:51.323" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10706" PostTypeId="1" AcceptedAnswerId="10845" CreationDate="2016-03-14T17:18:16.080" Score="3" ViewCount="2087" Body="&lt;p&gt;I am training a &lt;em&gt;normal&lt;/em&gt; and a &lt;em&gt;balanced&lt;/em&gt; &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow&quot;&gt;Linear SVM&lt;/a&gt; using imbalance data, and testing both using F1-score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it &quot;oversamples&quot; from the minority class and &quot;undersamples&quot; from the majority class.)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                   normal vs balanced&#xA;target 01 - scores: 0.272 vs 0.608&#xA;target 02 - scores: 0.391 vs 0.587&#xA;target 03 - scores: 0.433 vs 0.546&#xA;target 04 - scores: 0.659 vs 0.655&#xA;target 05 - scores: 0.000 vs 0.257&#xA;target 06 - scores: 0.431 vs 0.475&#xA;target 07 - scores: 0.000 vs 0.249&#xA;target 08 - scores: 0.000 vs 0.053&#xA;target 09 - scores: 0.576 vs 0.155&#xA;target 10 - scores: 0.000 vs 0.550&#xA;OneVsRest - scores: 0.565 vs 0.540&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code I have used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# -*- coding: utf-8 -*-&#xA;&#xA;from sklearn.svm import LinearSVC&#xA;from sklearn.multiclass import OneVsRestClassifier&#xA;from sklearn.cross_validation import StratifiedShuffleSplit&#xA;from sklearn.metrics import f1_score&#xA;import numpy as np&#xA;&#xA;d = np.loadtxt('yeast.csv', delimiter=',')&#xA;X = d[:, 0:-1]&#xA;y = d[:, -1]&#xA;&#xA;print '                     none vs balanced'&#xA;for target in np.unique(y):&#xA;    scores = [0] * 2&#xA;    k = 6&#xA;    for tr, ts in StratifiedShuffleSplit(y, k, 0.2):&#xA;        for i, w in enumerate([None, 'balanced']):&#xA;            _y = (y == target).astype(int)&#xA;            yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts])&#xA;            scores[i] += f1_score(_y[ts], yp) / k&#xA;    print 'target %02d - scores: %.3f vs %.3f' % (target, scores[0], scores[1])&#xA;&#xA;scores = [0] * 2&#xA;for tr, ts in StratifiedShuffleSplit(y, k, 0.2):&#xA;    for i, w in enumerate([None, 'balanced']):&#xA;        m = OneVsRestClassifier(LinearSVC(class_weight=w))&#xA;        yp = m.fit(X[tr], y[tr]).predict(X[ts])&#xA;        scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / k&#xA;print 'OneVsRest - scores: %.3f vs %.3f' % (scores[0], scores[1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the dataset, I have used here &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Yeast&quot; rel=&quot;nofollow&quot;&gt;Yeast&lt;/a&gt; (UCI), for which a sklearn-ready version can be found &lt;a href=&quot;https://www.dropbox.com/s/fgb0lgf665ney7z/yeast.csv?dl=1&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                     none vs balanced&#xA;target 01 - scores: 0.241 vs 0.612&#xA;target 02 - scores: 0.402 vs 0.604&#xA;target 03 - scores: 0.444 vs 0.552&#xA;target 04 - scores: 0.666 vs 0.650&#xA;target 05 - scores: 0.000 vs 0.286&#xA;target 06 - scores: 0.370 vs 0.500&#xA;target 07 - scores: 0.000 vs 0.280&#xA;target 08 - scores: 0.000 vs 0.082&#xA;target 09 - scores: 0.563 vs 0.147&#xA;target 10 - scores: 0.000 vs 0.511&#xA;OneVsRest - scores: 0.567 vs 0.543&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I think the fact that one model wins pretty much every &quot;battle&quot;, but not the &quot;war&quot; is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] &gt; score[k2] we can assume it prefers k1 &gt; k2. See &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-03-16T11:19:59.197" LastActivityDate="2016-03-23T13:51:55.687" Title="Balanced Linear SVM wins every class except One vs All" Tags="|classification|scikit-learn|multiclass-classification|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10707" PostTypeId="2" ParentId="10703" CreationDate="2016-03-14T17:23:50.040" Score="1" Body="&lt;p&gt;Without code you could have made to many mistakes to answer your question directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I suggest two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Gradient checking (e.g. &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;this explanation&lt;/a&gt;, but there are many more - pretty sure also in the course)&lt;/li&gt;&#xA;&lt;li&gt;XOR problem with a 2:2:1 network with only sigmoid activations. Plot the error surface. Compare the decision surface with that of &lt;a href=&quot;https://phiresky.github.io/neural-network-demo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this interactive demo&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It should look similar to this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pqGoh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pqGoh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, please keep in mind that the problem might not be your implementation, but rather the network architecture / hyperparameters such as the number of epochs you're training or the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Also, very important&lt;/strong&gt;: I doubt that you will get good results for $y = x^2$, except if $x$ is restricted to $[-1,1]$ or something similar simple. Please keep the domain of your output layer in mind.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-03-14T17:23:50.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10708" PostTypeId="2" ParentId="6169" CreationDate="2016-03-14T20:00:45.083" Score="1" Body="&lt;p&gt;Johnnyboycurtis answer works for me. If you are using python 3, use below code. His code doesnt work in python 3. I am editing only the last line of his code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;import sys&#xA;&#xA;&#xA;spark_home = os.environ.get('SPARK_HOME', None)&#xA;print(spark_home)&#xA;if not spark_home:&#xA;    raise ValueError('SPARK_HOME environment variable is not set')&#xA;sys.path.insert(0, os.path.join(spark_home, 'python'))&#xA;sys.path.insert(0, os.path.join(spark_home, 'C:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2.6/python/lib/py4j-0.9-src.zip')) ## may need to adjust on your system depending on which Spark version you're using and where you installed it.&#xA;&#xA;&#xA;filename=os.path.join(spark_home, 'python/pyspark/shell.py')&#xA;exec(compile(open(filename, &quot;rb&quot;).read(), filename, 'exec'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17054" LastActivityDate="2016-03-14T20:00:45.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10709" PostTypeId="2" ParentId="10626" CreationDate="2016-03-14T21:21:15.620" Score="2" Body="&lt;p&gt;Two things can be done. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One is apply ensemble methods like random forest, Adaboost, etc..&lt;/li&gt;&#xA;&lt;li&gt;Second one is try to optimise your decision tree and find the best&#xA;parameters using techniques like GridSearchCV.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17055" LastActivityDate="2016-03-14T21:21:15.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10710" PostTypeId="2" ParentId="10694" CreationDate="2016-03-14T21:53:14.007" Score="1" Body="&lt;p&gt;You need to define what speeding means and how to rate the drivers for amount of such. As you note that you have speed in your dataset this should be rather trivial. Not sure at which stage you want to involve machine learning? Seems like you don't need any.&#xA;Machine learning is not the same thing as learning about cars and their drivers.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-03-14T21:53:14.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10711" PostTypeId="1" CreationDate="2016-03-14T22:35:56.710" Score="0" ViewCount="137" Body="&lt;p&gt;I'm very new to text mining. I am trying to build a classifier which assigns zero, one or more topics to each document of the Reuters 21578 dataset. However, I'm not sure how to make it work using standard R packages. I've looked at a lot of simple online examples using packages such as RTextTools and RWeka, etc. which creates classifier to assign one topic to each document and I'm stuck as I can't directly follow the examples to achieve what I desire. Thanks in advance!  &lt;/p&gt;&#xA;" OwnerUserId="17057" LastActivityDate="2016-11-04T14:58:39.103" Title="Text classification: how to assign 1 or more classes to each document?" Tags="|machine-learning|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10712" PostTypeId="1" CreationDate="2016-03-14T22:39:16.153" Score="1" ViewCount="46" Body="&lt;p&gt;I have a problem that could be described as logistic regression with all dichotomous variables: 1 response variable (DV) &lt;strong&gt;Y&lt;/strong&gt; (I would call it later as a feature/violet star) and 5 explanatory variables (IVs) &lt;strong&gt;A, B, C, D&lt;/strong&gt; and &lt;strong&gt;E&lt;/strong&gt; (also binary).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What's most important, I'm trying to infer which IVs contribute the most to Y and preferably somehow assess quantitatively their severity&#xA;(something like ranking predictors, variable selection, etc.)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Of course, before I sent this post, I've made some reconnaissance, but I'm not sure if articles I've found reflect my problem properly,&#xA;because of it's strong cooperative nature.&#xA;It's very similar to coalitional games and Shapley value, but I'm not sure whether I can use it in this particular case and also I would like not to &quot;preach to the converted&quot; and would rather use something proven and well-tried.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Could you have a look on the situation below and recommend some methods, articles, books, etc. how to infer which explanatory and to what extent contribute the most to response Y (feature/violet star).&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, we have five different objects denoted as labelled balls: A, B, C, D, E creating some subsets and the single special feature of interest&#xA;marked as violet star. If the star is filled, the feature is present in a specific set, whereas hollow star denotes its absence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fxryw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fxryw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any subset of these five objects (there're $2^5 - 1$ possible types of such nonempty subsets) can appear many times, sometimes with the feature being present and sometimes&#xA;without it, as depicted below, where there're six instances of the full set and $\frac{2}{3}$ of them have the feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can assume (if it's needed) there's always at least one instance of the full set with the&#xA;feature present (for example the one marked with thicker red envelope).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For every of the $2^5 - 1$ possible nonempty subsets we have three values, e.g. for (A, C, E):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$F_{ACE}$ - number of all occurrences of subset (A, C, E) \textbf{with} the feature.&lt;/li&gt;&#xA;&lt;li&gt;$N_{ACE}$ - number of all occurrences of subset (A, C, E) \textbf{without} the feature.&lt;/li&gt;&#xA;&lt;li&gt;$T_{ACE} = F_{ACE} + N_{ACE}$ - simply total number of all appearances of subset (A, C, E).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The cardinalities of instances $T_X$ of every subset type $X$ can be different, but there's a very obvious and rather trivial rule that&#xA;      if $X \subset Y$ then $T_X \geq T_Y$. For example, $(A, C, E) \subset (A, B, C, D, E)$ thus $T_{ACE} \geq T_{ABCDE}$.&#xA;      That's because any instance of subset (A, B, C, D, E) is at the same time an instance of (A, C, E), but for narrower (A, C, E)&#xA;      we can have more instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In this description I use six instances for each subset type just bacause of limited space for pictures and those six&#xA;              instances are intended to reflect the overall proportion between instances having and lacking the feature.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among all these $2^5 - 1$ types, some of them are more likeli to have&#xA;the feature (for this particular subset/type, there're more instances with the feature than without it), whereas other types are more prone to lack it. Please look at two figures below and the following numbers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dNL94.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dNL94.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/cefwh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cefwh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;$$&#xA;\frac{F_{ACE}}{T_{ACE}} = \frac{5}{6}, ~~&#xA;\frac{F_{BD}}{T_{BD}} = \frac{1}{6}, ~~&#xA;\frac{F_{ABCDE}}{T_{ABCDE}} = \frac{4}{6} ~\text{(the situation of the very first picture)}&#xA;$$&#xA;A blatantly obvious property is that in my model the ratio $\frac{F}{T}$ is not monotonic (in neither direction) with respect to&#xA;inclusion relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally the most crucial property I would like to make my model had, is some kind of cooperative interaction between objects in subset,&#xA;which is the last thing I'm going to describe beneath.&#xA;A moment before, we've spotted that the subset (A, C, D) is exceptionally favorable for the feature appearance, so we would pay a special attention to its constituent singletons and their corresponding ratios:&#xA;$$&#xA;\frac{F_A}{T_A} = \frac{2}{6} ~~ \text{(2nd &amp;amp; 4th)}, ~~ \frac{F_C}{T_C} = \frac{2}{6} ~~ \text{(2nd &amp;amp; 3rd)},&#xA;~~ \frac{F_E}{T_E} = \frac{1}{6} ~~ \text{(2nd only)}&#xA;$$&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ZO4u0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZO4u0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4Ujpn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Ujpn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/lqLNJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lqLNJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even now, we can clearly see a sort of cooperative, since those three singletons (A), (C) and (E) all together cover only three subsets&#xA;{2nd, 3rd, 4th} = {2nd, 4th} $\cup$ {2nd, 3rd} $\cup$ {2nd},&#xA;whereas subset (A, C, E) triggers off the feature in most subsets except the 1st one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, merging A with E or C with E gives none additional coalitional effect, so E itself contributes poorly to overall effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s5Aep.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s5Aep.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Rdb2t.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rdb2t.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, merging A with C gives a significant interaction resulting with the feature apperance in more instances than it follows from simply&#xA;summing instances: $\{2nd, 3rd, 4th, 6th\}$ vs. $\{2nd, 3rd, 4th\}$. The occurrence in the 6th instance is this coalitional effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OB4Vc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OB4Vc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nonetheless, the E is necessary to trigger off the full potential and reveal the feature also in the 5th instance, as depicted in the second figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up, I would like to infer from the data which single object are favourable for the feature appearance. In our example we can&#xA;clearly see that A and C are very conducive (in cooperation with other), E is not so useful, but required to reach the full potential,&#xA;whereas B and D seem to have no cooperative effect on the feature.&lt;/p&gt;&#xA;" OwnerUserId="14876" LastActivityDate="2016-03-14T22:39:16.153" Title="Coalitional effect in logistic regression and assessing explanarory variable contribution" Tags="|predictive-modeling|regression|logistic-regression|feature-selection|feature-extraction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10713" PostTypeId="1" CreationDate="2016-03-14T23:38:12.397" Score="6" ViewCount="10379" Body="&lt;p&gt;Learning about random forest for the first time and I'm not clear if at each level, if we've already used a feature to make a decision, whether we can use the same feature again at further levels. &lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2024-01-16T12:25:40.300" Title="Does random forest re-use features at each node when generating a decision tree?" Tags="|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10714" PostTypeId="2" ParentId="10699" CreationDate="2016-03-15T01:08:58.617" Score="3" Body="&lt;p&gt;I re-implemented your set-up in python using keras. I used a hidden layer size of 25, and all my activations were sigmoid's. I got to an accuracy of 99.88%. Try running your algorithm for a greater amount of epochs. Use binary cross entropy as the loss function and try decreasing the learning rate of your gradient descent algorithm. This should help increase your accuracy. My only explanation for the poor performance would be that you are getting stuck at a local minimum, if that is the case different initiations of your weights should fix that problem.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastEditorUserId="14779" LastEditDate="2016-03-15T20:30:26.870" LastActivityDate="2016-03-15T20:30:26.870" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10715" PostTypeId="2" ParentId="10713" CreationDate="2016-03-15T01:44:24.583" Score="8" Body="&lt;p&gt;If the feature is discrete/categorial/binary, it won't use the already used feature to split at further levels in any given branch of the tree. But key thing to note here is that the same feature can be used in a different branch of the tree at a different level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/l6r4F.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/l6r4F.jpg&quot; alt=&quot;Decision Tree&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above diagram, we can see that same 'size' feature has been used at two levels 'level 1' and 'level 2', but in different branches of the tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if the variable is a continuous value, it uses threshold splits at each level and in this case, same feature can be used multiple times in any given branch of the decision tree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since random forests is an ensemble of decision trees, the same theory applies to Random Forests as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://www.cse.msu.edu/~cse802/DecisionTrees.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.cse.msu.edu/~cse802/DecisionTrees.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13957" LastActivityDate="2016-03-15T01:44:24.583" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10716" PostTypeId="2" ParentId="10642" CreationDate="2016-03-15T03:16:21.697" Score="11" Body="&lt;p&gt;Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pre-trained word and phrase vectors &lt;a href=&quot;https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing&quot;&gt;https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then load the vectors in binary format in your model using gensim as shown below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)  # C text format&#xA;&amp;gt;&amp;gt;&amp;gt; model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)  # C binary format&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a different pre-built model for English Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent&quot;&gt;https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://github.com/idio/wiki2vec/&quot;&gt;https://github.com/idio/wiki2vec/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a prebuilt model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Get python 2.7&#xA;Install gensim: pip install gensim&#xA;uncompress downloaded model: tar -xvf model.tar.gz&#xA;Load model in gensim:&#xA;from gensim.models import Word2Vec&#xA;model = Word2Vec.load(&quot;path/to/word2vec/en.model&quot;)&#xA;model.similarity('woman', 'man')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also use Stanford NLP Glove &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot;&gt;http://nlp.stanford.edu/projects/glove/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/stanfordnlp/GloVe&quot;&gt;https://github.com/stanfordnlp/GloVe&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is a great compilation of pre-trained word2vec models.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models&quot;&gt;https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some additional pre-trained models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.zuccon.net/ntlm.html&quot;&gt;http://www.zuccon.net/ntlm.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://code.google.com/archive/p/word2vec/&quot;&gt;https://code.google.com/archive/p/word2vec/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;More on gensim and code here:  &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot;&gt;https://radimrehurek.com/gensim/models/word2vec.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quora forum with a similar questions &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/Where-can-I-find-pre-trained-models-of-word2vec-and-sentence2vec-trained-on-Wikipedia-or-other-large-text-corpus&quot;&gt;https://www.quora.com/Where-can-I-find-pre-trained-models-of-word2vec-and-sentence2vec-trained-on-Wikipedia-or-other-large-text-corpus&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/Where-can-I-find-some-pre-trained-word-vectors-for-natural-language-processing-understanding&quot;&gt;https://www.quora.com/Where-can-I-find-some-pre-trained-word-vectors-for-natural-language-processing-understanding&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13957" LastActivityDate="2016-03-15T03:16:21.697" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10721" PostTypeId="1" CreationDate="2016-03-15T10:17:24.553" Score="1" ViewCount="546" Body="&lt;p&gt;I've been assigned a task to extract features/attributes from product description. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Levi Strauss slim fit jeans&#xA;Big shopping bag in pink and gold&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to be able to extract out attributes such as &quot;Jeans&quot; and &quot;slim fit&quot; or &quot;shopping bag&quot; and &quot;pink&quot; and &quot;gold&quot;.&#xA;The product description listings are not just for clothes, they can basically be anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure how to approach this problem. I tried implementing a Named Entity Recognizer solution and also a POS implementations, The NER implementation fails to recognize any token and most of the tokens show up as NNP(Proper Nouns) in he POS solution, which doesn't help me out a lot. I need a way to be able to distinguish between the brand name and the features of the Product(like if it is a t-shirt, the color or design(round neck, v-neck) etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did implement a KMean solution which did cluster like products together, but then again it is not the result I am looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just looking for someone to direct me in the correct direction.&lt;/p&gt;&#xA;" OwnerUserId="14393" LastActivityDate="2016-03-15T10:17:24.553" Title="Extract Product Attributes/Features" Tags="|machine-learning|data-mining|nlp|feature-extraction|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10722" PostTypeId="1" CreationDate="2016-03-15T16:35:42.080" Score="3" ViewCount="670" Body="&lt;p&gt;I am coming from the world of linear/logistic regression where residuals are assumed to be normal and the output includes some sort of standard error of residuals that one can use to make forecasts. Admittedly, I have very little experience in fitting and interpreting multinomial models..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, I am thinking about a specific problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have lots of data on voters that have already cast a vote. With this data we wish to train a model to forecast future elections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the world of normal regression, there seems to be standard errors in forecasting each vote that could potentially be summed to create a random variable for the sums of votes for each candidate. These random variables could then be used to try to forecast the probability of each candidate winning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, what is the standard way of assessing the &quot;standard error&quot; of classifier models using tools like Stochastic Gradient Descent Classifiers for example. Or any of the more modern tools for classification. Some sort of bootstrapping approach?&lt;/p&gt;&#xA;" OwnerUserId="16972" LastActivityDate="2017-02-09T06:09:49.193" Title="Standard Deviation of Classifier Models" Tags="|classification|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10723" PostTypeId="2" ParentId="10478" CreationDate="2016-03-15T17:54:10.540" Score="8" Body="&lt;p&gt;I'd start with the &lt;code&gt;melt()&lt;/code&gt; function in pandas.  I wrote an article about it:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17080" LastEditorUserId="29169" LastEditDate="2021-02-09T20:17:11.830" LastActivityDate="2021-02-09T20:17:11.830" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10724" PostTypeId="1" CreationDate="2016-03-15T17:57:57.203" Score="6" ViewCount="9570" Body="&lt;p&gt;The explanation from the official doc2vec paper &lt;a href=&quot;https://arxiv.org/abs/1405.4053&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Distributed Representations of Sentences and Documents&quot;&lt;/a&gt; for PV-DBOW is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;According to the paper the word vectors are not stored and PV-DBOW is said to work similar to skip gram in word2vec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Skip-gram is explained in &lt;a href=&quot;https://arxiv.org/pdf/1411.2738.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec Parameter Learning&lt;/a&gt;. In the skip gram model the word vectors are mapped to the hidden layer. The matrix that performs this mapping is updated during the training. In PV-DBOW the dimension of the hidden layer should be the dimension of one paragraph vector. When I want to multiply the word vector of a sampled example with the paragraph vector they should have the same size. The original representation of a word is of size (vocabulary size x 1). Which mapping is performed to get the right size (paragraph dimension x 1) in the hidden layer. And how is this mapping performed when the word vectors are not stored? I assume that word and paragraph representation should have the same size in the hidden layer because of equation 26 in &lt;a href=&quot;https://arxiv.org/pdf/1411.2738.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17081" LastEditorUserId="17081" LastEditDate="2020-01-21T18:29:36.053" LastActivityDate="2020-01-21T18:29:36.053" Title="How does PV-DBOW (doc2vec) work?" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10725" PostTypeId="2" ParentId="10724" CreationDate="2016-03-15T19:57:19.950" Score="2" Body="&lt;p&gt;I haven't read the paper you linked, but I have followed a &lt;a href=&quot;http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf&quot; rel=&quot;nofollow&quot;&gt;lecture notes&lt;/a&gt; by Richard Socher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, basically that mapping matrix is called weight matrices. There are two weight matrices &lt;code&gt;W1&lt;/code&gt; and &lt;code&gt;W2&lt;/code&gt; for input and output mapping. Thus for each word, vectors in both the matrices are updated via backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question, a word is represented by one-hot sparse vector which has the size of &lt;code&gt;Vx1&lt;/code&gt; (V is size of vocabulary), with a value of 1 in one of the position. and when this vector is multiplied with the weight matrix &lt;code&gt;W1&lt;/code&gt; with size &lt;code&gt;NxV&lt;/code&gt;, the corresponding embedding vector of size &lt;code&gt;Nx1&lt;/code&gt; (N is size of the required embedding vector) is used in the hidden layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document has no one-hot represented vectors. So, basically there is a document matrix &lt;code&gt;D&lt;/code&gt; of size &lt;code&gt;Nxd&lt;/code&gt; (d is number of documents) where each column represents a document. In other words, the matrices &lt;code&gt;W1&lt;/code&gt; and &lt;code&gt;W2&lt;/code&gt; need the one-hot representation only for mathematical steps, other than that they are representing each word in each column just as the document matrix &lt;code&gt;D&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-03-15T19:57:19.950" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="10726" PostTypeId="2" ParentId="10722" CreationDate="2016-03-15T21:56:08.110" Score="1" Body="&lt;p&gt;Classifiers are commonly scored on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy&quot; rel=&quot;nofollow&quot;&gt;cross entropy&lt;/a&gt; or some other &lt;a href=&quot;https://en.wikipedia.org/wiki/Divergence_(statistics)&quot; rel=&quot;nofollow&quot;&gt;divergence&lt;/a&gt;, since we're talking about distributions. A multi-class classifier will give you a &quot;score&quot; associated with each class (option, outcome). It is usually easy to &lt;a href=&quot;https://en.wikipedia.org/wiki/Calibration_(statistics)&quot; rel=&quot;nofollow&quot;&gt;transform this score into a probability&lt;/a&gt;, so that the options add up to one. Furthermore, it is possible to associate with each class a confidence distribution. I don't know if this is what you are asking about, but &lt;a href=&quot;http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html&quot; rel=&quot;nofollow&quot;&gt;here's a great blog post on it&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-15T21:56:08.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10728" PostTypeId="1" CreationDate="2016-03-16T01:03:29.313" Score="7" ViewCount="496" Body="&lt;p&gt;I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records are incomplete or contain outliers. The dependent variable is Defaulted / Good (0,1). We are trying to train a neural network to predict defaults based on the training data. I have experience with neural networks and used them for sample data with great results, however, I never had to normalize noisy real world data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts what I should keep in mind in respect to:&#xA; - How to normalize the categories. Can I assign an indexed number? Should I tried to stratify them? &#xA;- How to deal with missing data. Assign 0 ?&#xA;- How to deal with the fact that defaults are only about 5 % of the data set. What transfer function would be useful to predict these low probabilities.&#xA;- Basically any other real world data advice is very much appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="17085" LastActivityDate="2016-04-01T01:48:26.150" Title="Neural Networks: How to prepare real world data to detect low probability events?" Tags="|machine-learning|neural-network|data-cleaning|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10730" PostTypeId="1" CreationDate="2016-03-16T04:28:50.893" Score="13" ViewCount="6050" Body="&lt;p&gt;How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.&lt;/p&gt;&#xA;" OwnerUserId="14898" LastActivityDate="2016-03-16T12:54:20.037" Title="How many dimensions to reduce to when doing PCA?" Tags="|pca|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10731" PostTypeId="2" ParentId="10730" CreationDate="2016-03-16T05:32:30.787" Score="14" Body="&lt;p&gt;After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components removed and see if it makes an 'elbow' where it makes sense. A lot of this depends on your use case though.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-03-16T05:32:30.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10738" PostTypeId="2" ParentId="10730" CreationDate="2016-03-16T12:54:20.037" Score="3" Body="&lt;p&gt;I normally check for percentage of the information held by K value. Let's say out of 8 fields, 2 of them hold 90% of the information. Then there is no point in including the other 6 or 5 fields. If u know mnist data, out of 768 input, I only used 250, which bumped my accuracy from 83 to 96%. The fact is more dimensionality brings more problem. So cut them off. I usually only take K who only hold 90% of the info, and it works for me. &lt;/p&gt;&#xA;" OwnerUserId="11141" LastActivityDate="2016-03-16T12:54:20.037" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10739" PostTypeId="1" AcceptedAnswerId="10747" CreationDate="2016-03-16T13:28:39.270" Score="1" ViewCount="1948" Body="&lt;p&gt;I have several user names and their salaries. &#xA;Now I need to cluster user based on their salaries.&#xA;I am using KMeans clustering and following is my code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import KMeans&#xA;from sklearn.preprocessing import LabelEncoder&#xA;import pandas as pd&#xA;&#xA;le = LabelEncoder()&#xA;data = pd.read_csv('kmeans.data',header=None, names =['user', 'salary'])&#xA;&#xA;# Numerical conversion&#xA;data['user'] = le.fit_transform(data['user'])&#xA;km = KMeans(n_clusters=4, random_state= 10, n_init=10, max_iter=500)&#xA;km.fit(data)&#xA;&#xA;data['labels'] = le.inverse_transform(data['user'])&#xA;data['cluster'] = km.labels_&#xA;&#xA;print data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But my results are bad and there are lot of overlapping salaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything wrong in the code ? How to improve the results ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or whether clustering is not a right approach here ?&#xA;Then how can I cluster users only based on salary ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; km.fit(data['salary'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I figured out a way to solve my problem using &lt;a href=&quot;http://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html&quot; rel=&quot;nofollow&quot;&gt;numpy.reshape&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;km.fit(data['salary'].reshape(-1,1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2016-03-17T07:52:33.083" LastActivityDate="2016-03-17T07:52:33.083" Title="K Means giving poor results" Tags="|machine-learning|python|clustering|k-means|scikit-learn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10740" PostTypeId="2" ParentId="10739" CreationDate="2016-03-16T13:44:02.077" Score="0" Body="&lt;p&gt;I think the problem here is using the name as a dimension. You can, but you have to use a more robust distance metric between names (string). As far as I know, LabelEncoder just assign an int considering the element's order of ocurrence in a unique list. You could try a different hashing (string to int) or &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow&quot; title=&quot;Levenshtein_distance&quot;&gt;Levenshtein_distance&lt;/a&gt; as a distance metric&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-03-16T13:44:02.077" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10741" PostTypeId="1" CreationDate="2016-03-16T14:10:07.437" Score="3" ViewCount="1919" Body="&lt;p&gt;I have implemented my own mini neural network program&lt;a href=&quot;https://github.com/yashchandak/Neural-Nets/tree/master/CNN&quot; rel=&quot;nofollow&quot; title=&quot;sorry, not well documented yet&quot;&gt;1&lt;/a&gt;.&#xA;Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was trying to implement batch updates and batch normalisation&lt;a href=&quot;http://arxiv.org/abs/1502.03167&quot; rel=&quot;nofollow&quot;&gt;2&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) For simple batch updates, instead of updating parameters each time, for each image of the batch size of 'n' I should backpropagate and accumulate the deltas for all the parameters and finally update them once after the end of the batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)For batch normalisation (BN), I went through the paper and I am sort of the clear with the idea but I am confused regarding how to implement it. Generally, I would multiply the matrices in the net one after the other for a single image to get final input, but with BN, do I need to feed forward for all the images in the batch till the first layer, then normalise the values, then fwd pass these values till second layer, then normalise again, and so on?&#xA;Once I reach the final layer, should I backpropagate the error for the corresponding input-output pair and update the parameters immediately as fwd pass for all the images in the batch has been done already?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going by the way I have described, it seems to require a lot of parameter tracking throughout the batch.&#xA;It will be helpful if you can point out a better way to do it or anything that I have misunderstood so far. &lt;/p&gt;&#xA;" OwnerUserId="10967" LastEditorUserId="836" LastEditDate="2017-08-22T18:04:27.733" LastActivityDate="2020-12-15T13:01:22.710" Title="Implementing Batch normalisation in Neural network" Tags="|machine-learning|neural-network|backpropagation|batch-normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10742" PostTypeId="2" ParentId="6797" CreationDate="2016-03-16T15:57:38.733" Score="1" Body="&lt;p&gt;Tuning the hyperparameters of a model is still more of an art than a science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both R (caRet) and Python (sklearn) have frameworks for performing grid searches of hyperparameters using cross validation. You can define custom cost functions in these frameworks, which will allow you to directly assess your cross-validation results using your cost matrix. See the &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter&quot; rel=&quot;nofollow&quot;&gt;scikit-learn documentation&lt;/a&gt; for Python, or the &lt;a href=&quot;http://topepo.github.io/caret/training.html#metrics&quot; rel=&quot;nofollow&quot;&gt;caRet documentation&lt;/a&gt; for R. You will get much better results using your cost matrix directly to evaluate your models, instead of using F-1, precision, recall, accuracy, or any other metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other Guidance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to understand what the different hyperparameters actually mean in the context of each learning algorithm to help guide the parameter search, as computational limitations usually strictly limit the number of different models you can build total, limiting the size of the grid you can realistically search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With such a low percentage of negative examples in your set, you will want to ensure that you use stratified sampling for your cross-validation. This will ensure that each fold has the same percentage of negative examples as the base set. Otherwise, your cross-validation results will likely be biased by the non-uniform negative class distribution.&lt;/p&gt;&#xA;" OwnerUserId="17078" LastActivityDate="2016-03-16T15:57:38.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10743" PostTypeId="1" AcceptedAnswerId="10746" CreationDate="2016-03-16T16:08:56.073" Score="2" ViewCount="457" Body="&lt;p&gt;In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that I will not get frequencies by reconstructing the original matrix, but is the non-zeros after reconstruction valid? &lt;/p&gt;&#xA;" OwnerUserId="17101" LastActivityDate="2016-05-15T20:35:18.930" Title="In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?" Tags="|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10745" PostTypeId="1" AcceptedAnswerId="18380" CreationDate="2016-03-16T18:47:46.380" Score="12" ViewCount="6450" Body="&lt;p&gt;R's caret package works with 180 models. The author warns that some of the package may be intractably slow or less accurate than top-choice models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author is not wrong about this. I've tried to train Boruta and evtree models and had to give up after they ran &gt; 5 hours on a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author links to &lt;a href=&quot;https://github.com/szilard/benchm-ml&quot; rel=&quot;noreferrer&quot;&gt;a set of machine learning benchmarks&lt;/a&gt;, but those only cover performance of a small number of algorithms, comparing different implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some other resource I can turn to, for guidance on which of the 180 models are worth trying, and which will be very inaccurate or unreasonably slow?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2020-02-24T10:57:50.090" Title="Which of the 180 algorithms in R's caret package are feasible?" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10746" PostTypeId="2" ParentId="10743" CreationDate="2016-03-16T19:13:33.277" Score="1" Body="&lt;p&gt;It is possible to produce a corpus from the learned LDA parameters ($\theta$ and $\phi$) according to the generative model of LDA but it is not realistic to expect that you would recreate the original documents (in bag-of-words form). To be more specific, it is &lt;em&gt;possible&lt;/em&gt; - but highly improbable - that you would generate the bag-of-words documents corresponding to the input corpus.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-03-16T19:13:33.277" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10747" PostTypeId="2" ParentId="10739" CreationDate="2016-03-16T19:56:45.303" Score="5" Body="&lt;p&gt;K-means is based on the assumption that the data is &quot;translation invariant&quot; (more precisely: variance does, and k-means is variance minimization).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, it assumes that a difference of d=(x-y)^2 is of the &lt;em&gt;same&lt;/em&gt; importance everywhere. Because of this, &lt;strong&gt;k-means does not work on skewed data&lt;/strong&gt;. Furthermore, because of the square, it is &lt;strong&gt;sensitive to outliers and other extreme values&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For salaries and other monetary values, this usually does not hold. The difference between \$0 and \$1000 is massive, and not the same as a salary difference of \$100000 to \$101000. Salaries are usually rather skewed, and you often have some extreme values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Converting the &quot;user&quot; attribute to a numerical value is outright statistical nonsense&lt;/strong&gt;. What's variance worth in this attribute? &lt;strong&gt;K-means is for continuous numerical data only&lt;/strong&gt;, and converting data does not chnage the &lt;em&gt;nature&lt;/em&gt;, only the encoding - it's still inappropriate.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-03-16T19:56:45.303" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10748" PostTypeId="2" ParentId="10739" CreationDate="2016-03-17T01:54:02.977" Score="2" Body="&lt;p&gt;This is not a 'clustering' problem as much at is it an 'interval' problem since you only have 1 dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use an iterative process like &lt;a href=&quot;https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization&quot; rel=&quot;nofollow&quot;&gt;Jenk's natural break optimization&lt;/a&gt; in order to figure out how large to make your intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As other posters have said, do not user names as a clustering dimension unless you really think that variations in letters of a name are meaningful in some way (do you really think all the Dan's are paid similarly?). &lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-17T01:54:02.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10750" PostTypeId="1" CreationDate="2016-03-17T05:49:24.920" Score="2" ViewCount="64" Body="&lt;p&gt;Aside from questions about &lt;a href=&quot;https://datascience.stackexchange.com/questions/8583/how-to-visualize-data-of-a-multidimensional-dataset-timit&quot;&gt;How to visualize data of a multidimensional dataset (TIMIT)&lt;/a&gt;, the &lt;a href=&quot;https://datascience.stackexchange.com/questions/9038/purpose-of-visualizing-high-dimensional-data&quot;&gt;Purpose of visualizing high dimensional data?&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/4942/high-dimensional-data-what-are-useful-techniques-to-know&quot;&gt;High-dimensional data: What are useful techniques to know?&lt;/a&gt;, I am looking at tools for high-dimensional data (numerical only) visualization, possibly embedding some dimension reduction techniques (Sammon maps, t-SNE), that could be used by end-users. I have found for instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://hisee.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;HiSee&lt;/a&gt; (version 1.0.0, 2004-07-05),&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ggobi.org/downloads/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GGobi&lt;/a&gt; (version 2.1.10a, 2010-03-12),&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/datacratic/data-projector&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Projector&lt;/a&gt; (2014-07-15).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They do not seem to be heavily maintained. Are there more active or efficient developments to follow?&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-03-17T05:49:24.920" Title="Free/open interactive softwares/plugins for end-users' high-dimensional data visualization" Tags="|visualization|data|dimensionality-reduction|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10753" PostTypeId="1" AcceptedAnswerId="10776" CreationDate="2016-03-17T10:38:41.777" Score="4" ViewCount="3605" Body="&lt;p&gt;I work with neural networks (ConvNNs, DeepNNs, RNNs/LSTMs) for image segmentation and recognition and Genetic Algorithms for some optimization problems. Recently I started to learn some deep graph theory ideas (random graphs, chromatic numbers, graph coloring). I'm familiar with combinatorics at undergrad level. Are there any existing interesting applications and areas of research of graph theory and combinatorics in ML?  &lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="11097" LastEditDate="2016-03-17T10:52:37.037" LastActivityDate="2016-03-18T13:58:36.520" Title="Application of ideas from graph theory in machine learning" Tags="|neural-network|graphs|reference-request|" AnswerCount="1" CommentCount="0" ClosedDate="2017-01-06T17:56:28.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="10757" PostTypeId="2" ParentId="8709" CreationDate="2016-03-17T12:40:55.787" Score="0" Body="&lt;p&gt;NMF and PCA are very similar such that both are factoring a matrix M into WH. However, with NMF, you have the constraint that M, W and H are all nonnegative. In PCA, you can think of W and H as coefficients and eigenfaces then proceed to use the coefficients for facial recognition. In NMF, W and H are coefficients and face profiles (that is, parts of the face). You then proceed to use the coefficients for your facial recognition in the same manner as using PCA.&lt;/p&gt;&#xA;" OwnerUserId="14732" LastActivityDate="2016-03-17T12:40:55.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10758" PostTypeId="1" CreationDate="2016-03-17T13:56:10.293" Score="3" ViewCount="2551" Body="&lt;p&gt;Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model &lt;em&gt;stacking&lt;/em&gt; -- is a popular way of creating ensembles of Machine Learning models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = regression_predictions * .5 + tree_predictions * .5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While this approach is useful in a wide variety of use cases, a good example is Kaggle competitions. In these competitions you have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Labeled training data &lt;/li&gt;&#xA;&lt;li&gt;Unlabeled prediction (test) data&lt;/li&gt;&#xA;&lt;li&gt;A certain number of allowed submissions per day, often 5, wherein you submit your predictions and receive instant scoring based on a specified evaluation metric like RMSE&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is if there is an statistical or logical methodological framework that can guide the testing of various blends of your models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, can some algorithm or methodology be applied to the process of deciding how to alter the blend of predictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, such a framework would also guide the decision of when to add or remove a set predictions, though even a framework which assumes a given number of models would be highly useful.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-09-05T07:37:43.860" Title="Algorithmic approach to model blending" Tags="|machine-learning|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10762" PostTypeId="2" ParentId="10699" CreationDate="2016-03-17T16:31:16.180" Score="1" Body="&lt;p&gt;Problem solved! There was  mistake in my cost formula...lambda was not multiplied with both theta components due to a missing bracket! Resolved that and things working fine now. :-)&lt;/p&gt;&#xA;" OwnerUserId="17047" LastActivityDate="2016-03-17T16:31:16.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10763" PostTypeId="1" CreationDate="2016-03-17T17:48:05.420" Score="0" ViewCount="354" Body="&lt;p&gt;I am looking at this tutorial: &lt;a href=&quot;https://www.dataquest.io/mission/74/getting-started-with-kaggle&quot; rel=&quot;nofollow&quot;&gt;https://www.dataquest.io/mission/74/getting-started-with-kaggle&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is the code for linear regression to predict, based on some variables, the survival of the titanic passengers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Import the linear regression class&#xA;from sklearn.linear_model import LinearRegression&#xA;# Sklearn also has a helper that makes it easy to do cross validation&#xA;from sklearn.cross_validation import KFold&#xA;&#xA;# The columns we'll use to predict the target&#xA;predictors = [&quot;Pclass&quot;, &quot;Sex&quot;, &quot;Age&quot;, &quot;SibSp&quot;, &quot;Parch&quot;, &quot;Fare&quot;, &quot;Embarked&quot;]&#xA;&#xA;# Initialize our algorithm class&#xA;alg = LinearRegression()&#xA;# Generate cross validation folds for the titanic dataset.  It return the row indices corresponding to train and test.&#xA;# We set random_state to ensure we get the same splits every time we run this.&#xA;kf = KFold(titanic.shape[0], n_folds=3, random_state=1)&#xA;&#xA;predictions = []&#xA;for train, test in kf:&#xA;    # The predictors we're using the train the algorithm.  Note how we only take the rows in the train folds.&#xA;    train_predictors = (titanic[predictors].iloc[train])&#xA;    # The target we're using to train the algorithm.&#xA;    train_target = titanic[&quot;Survived&quot;].iloc[train]&#xA;    # Training the algorithm using the predictors and target.&#xA;    alg.fit(train_predictors, train_target)&#xA;    # We can now make predictions on the test fold&#xA;    test_predictions = alg.predict(titanic[predictors].iloc[test])&#xA;    predictions.append(test_predictions)&#xA;&#xA;import numpy as np&#xA;&#xA;# The predictions are in three separate numpy arrays.  Concatenate them into one.  &#xA;# We concatenate them on axis 0, as they only have one axis.&#xA;predictions = np.concatenate(predictions, axis=0)&#xA;&#xA;# Map predictions to outcomes (only possible outcomes are 1 and 0)&#xA;predictions[predictions &amp;gt; .5] = 1&#xA;predictions[predictions &amp;lt;=.5] = 0&#xA;&#xA;accuracy = sum(titanic[&quot;Survived&quot;] == predictions)/len(predictions)&#xA;&#xA;print (accuracy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is clear. For logistic regression is much easier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cross_validation&#xA;&#xA;# Initialize our algorithm&#xA;alg = LogisticRegression()&#xA;# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)&#xA;scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[&quot;Survived&quot;], cv=3)&#xA;# Take the mean of the scores (because we have one for each fold)&#xA;print(scores.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, why can't I do just, for LinearRegression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cross_validation&#xA;&#xA;# Initialize our algorithm&#xA;alg = LinearRegression()&#xA;# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)&#xA;scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[&quot;Survived&quot;], cv=3)&#xA;# Take the mean of the scores (because we have one for each fold)&#xA;print(scores.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me a totally wrong result. Why is that? What is wrong in the last snippet of code?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2016-06-15T22:12:17.300" Title="Why logistic regression example code does not port to linear regression example?" Tags="|python|logistic-regression|linear-regression|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10765" PostTypeId="1" CreationDate="2016-03-17T18:27:33.173" Score="2" ViewCount="307" Body="&lt;p&gt;I have a corpus of job descriptions and another corpus of CVs of applicants. I plan to implement a matching system using machine learning algorithms, to find top 5 or top 10 applicants for each job description. Should I store the data in a document oriented NoSQL db (MongoDB) or stick to SQL. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that the data I have is semi-structured at best, I feel a NoSQL db will offer more flexibility. I would appreciate opinions on this.&lt;/p&gt;&#xA;" OwnerUserId="13855" LastEditorUserId="13855" LastEditDate="2016-03-19T04:09:18.397" LastActivityDate="2016-03-19T04:09:18.397" Title="NoSQL vs SQL backend for semi structured data" Tags="|databases|sql|nosql|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10766" PostTypeId="2" ParentId="10763" CreationDate="2016-03-17T20:57:14.553" Score="1" Body="&lt;p&gt;I suspect that you are missing this mapping:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Map predictions to outcomes (only possible outcomes are 1 and 0)&#xA;predictions[predictions &amp;gt; .5] = 1&#xA;predictions[predictions &amp;lt;=.5] = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, what linear regression produces after a call to &lt;code&gt;predict&lt;/code&gt; is a set of numbers (scores), then we need to manually map it back to class label. &lt;code&gt;cross_validation.cross_val_score&lt;/code&gt; doesn't do the mapping for us, so what it ends up doing is comparing scores with class labels, which turns out to be bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;code&gt;predict&lt;/code&gt; of logistic regression gives us the class label (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression.predict&quot; rel=&quot;nofollow&quot;&gt;Reference&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-17T20:57:14.553" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10770" PostTypeId="1" CreationDate="2016-03-17T23:01:10.830" Score="1" ViewCount="109" Body="&lt;p&gt;Can we draw time series plot with only month and year in R? I don't have date variable in my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset looks like this-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;month  year     percentage&#xA;1      2012      34%&#xA;2      2012      55%&#xA;.&#xA;.&#xA;.&#xA;.&#xA;1     2013      15%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="13450" LastEditDate="2016-03-18T13:21:36.553" LastActivityDate="2016-04-17T18:24:06.693" Title="time series plot" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10771" PostTypeId="2" ParentId="896" CreationDate="2016-03-18T02:22:50.490" Score="1" Body="&lt;p&gt;Just to complement the question, regarding the overview of the algorithm, I found this slide to be very clear (but also failed to mention the table update mechanism):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/E9d76.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/E9d76.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from &lt;a href=&quot;http://scholar.google.com/citations?user=DxoenfgAAAAJ&amp;amp;hl=en&amp;amp;oi=ao&quot; rel=&quot;nofollow noreferrer&quot;&gt;Michael Collins&lt;/a&gt; given in his &lt;a href=&quot;https://class.coursera.org/nlangp-001&quot; rel=&quot;nofollow noreferrer&quot;&gt;MOOC on NLP&lt;/a&gt; (18 - 5 - The Brown Clustering Algorithm (Part 3) (9-18))&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-03-18T02:22:50.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10772" PostTypeId="1" AcceptedAnswerId="10778" CreationDate="2016-03-18T10:24:00.550" Score="1" ViewCount="54" Body="&lt;p&gt;I have data of intraday electricity consumptions (by half hours - 48 a day) over a year of 4000 households. Task is to establish baseline consumption of each of these households - possibly also differentiated on seasonality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way how to do this would be just taking the mean of the consumption signals. What would be more sophisticated method for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be very grateful for pointing out to methods I could look into.&lt;/p&gt;&#xA;" OwnerUserId="17149" LastEditorUserId="924" LastEditDate="2016-03-19T06:47:57.470" LastActivityDate="2016-03-19T06:47:57.470" Title="Identifying baseline consumption" Tags="|statistics|unsupervised-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10773" PostTypeId="1" AcceptedAnswerId="10775" CreationDate="2016-03-18T10:34:45.107" Score="19" ViewCount="86011" Body="&lt;p&gt;I am looking at this tutorial: &lt;a href=&quot;https://www.dataquest.io/mission/75/improving-your-submission&quot; rel=&quot;noreferrer&quot;&gt;https://www.dataquest.io/mission/75/improving-your-submission&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At section 8, finding the best features, it shows the following code.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.feature_selection import SelectKBest, f_classif&#xA;&#xA;predictors = [&quot;Pclass&quot;, &quot;Sex&quot;, &quot;Age&quot;, &quot;SibSp&quot;, &quot;Parch&quot;, &quot;Fare&quot;, &quot;Embarked&quot;, &quot;FamilySize&quot;, &quot;Title&quot;, &quot;FamilyId&quot;]&#xA;&#xA;# Perform feature selection&#xA;selector = SelectKBest(f_classif, k=5)&#xA;selector.fit(titanic[predictors], titanic[&quot;Survived&quot;])&#xA;&#xA;# Get the raw p-values for each feature, and transform from p-values into scores&#xA;scores = -np.log10(selector.pvalues_)&#xA;&#xA;# Plot the scores.  See how &quot;Pclass&quot;, &quot;Sex&quot;, &quot;Title&quot;, and &quot;Fare&quot; are the best?&#xA;plt.bar(range(len(predictors)), scores)&#xA;plt.xticks(range(len(predictors)), predictors, rotation='vertical')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is k=5 doing, since it is never used (the graph still lists all of the features, whether I use k=1 or k=&quot;all&quot;)?&#xA;How does it determine the best features, are they independent of the method one wants to use (whether logistic regression, random forests, or whatever)?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2021-03-11T20:04:14.150" Title="How does SelectKBest work?" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10774" PostTypeId="2" ParentId="10772" CreationDate="2016-03-18T10:37:46.340" Score="0" Body="&lt;p&gt;I don't think there is need for very complex methods here, I would take a look at several granularity levels and take means for these levels. One important thing is if all the households are in the same area/timezone. After establishing this I would first take the general mean and see per household how they compare to this, then do this for every halfhour segment of the day to get rid of time-of-day seasonality, do this for day of the week, month of the year etc. It really depends on what you want to know what to do with this data.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-03-18T10:37:46.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10775" PostTypeId="2" ParentId="10773" CreationDate="2016-03-18T12:42:19.223" Score="14" Body="&lt;p&gt;The SelectKBest class just scores the features using a function (in this case f_classif but could be others) and then &quot;removes all but the k highest scoring features&quot;. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So its kind of a wrapper, the important thing here is the function you use to score the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For other feature selection techniques in sklearn read: &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/feature_selection.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And yes, f_classif and chi2 are independent of the predictive method you use. &lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-03-18T12:42:19.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10776" PostTypeId="2" ParentId="10753" CreationDate="2016-03-18T13:58:36.520" Score="1" Body="&lt;p&gt;Graphs are a very flexible form of data representation, and therefore have been applied to machine learning in many different ways in the past. You can take a look to the papers that are submitted to specialized conferences like S+SSPR (The joint IAPR International Workshops on Structural and Syntactic Pattern Recognition and Statistical Techniques in Pattern Recognition) and GBR (Workshop on Graph-based Representations in Pattern Recognition) to start getting a good idea of potential applications. Some examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Within the Computer Vision field, graphs have been used to extract structure information that can later on be used on several applications, like for instance object recognition and detection, image segmentation and so on. &lt;/li&gt;&#xA;&lt;li&gt;Spectral clustering is an example of clustering method based on graph theory. It makes use of the eigenvalues of the similarity matrix to combine clustering and dimensionality reduction. &lt;/li&gt;&#xA;&lt;li&gt;Random walks may be used to predict and recommend links in social networks or to rank webpages by relevance. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-03-18T13:58:36.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10777" PostTypeId="2" ParentId="10770" CreationDate="2016-03-18T14:02:53.683" Score="1" Body="&lt;p&gt;I would convert it to a date column using paste and as.Date setting it to the first of the month.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- data.frame('month'=c(1,2,3,1,2), year=c(2012, 2012, 2012, 2013, 2013))&#xA;&#xA;&amp;gt; df&#xA;  month year&#xA;1     1 2012&#xA;2     2 2012&#xA;3     3 2012&#xA;4     1 2013&#xA;5     2 2013&#xA;&#xA;&amp;gt; df[, 'date_col'] &amp;lt;- as.Date(paste(1, df$month, df$year), '%d %m %Y')&#xA;&#xA;&amp;gt; df&#xA;  month year   date_col&#xA;1     1 2012 2012-01-01&#xA;2     2 2012 2012-02-01&#xA;3     3 2012 2012-03-01&#xA;4     1 2013 2013-01-01&#xA;5     2 2013 2013-02-01&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: If you don't want first of the month as a place holder, you can use &lt;code&gt;as.yearmon&lt;/code&gt; from the &lt;code&gt;zoo&lt;/code&gt; package&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; as.yearmon(paste(df$month, df$year), '%m %Y')&#xA;[1] &quot;Jan 2012&quot; &quot;Feb 2012&quot; &quot;Mar 2012&quot; &quot;Jan 2013&quot; &quot;Feb 2013&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10135" LastEditorUserId="10135" LastEditDate="2016-03-18T17:26:08.443" LastActivityDate="2016-03-18T17:26:08.443" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10778" PostTypeId="2" ParentId="10772" CreationDate="2016-03-18T17:49:26.493" Score="0" Body="&lt;p&gt;You can try and model your data as having trend/seasonal/cyclic components if you want something a little more sophisticated. Here is an intro &lt;a href=&quot;https://www.otexts.org/fpp/6/1&quot; rel=&quot;nofollow&quot;&gt;reference&lt;/a&gt; to get you started.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-18T17:49:26.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10779" PostTypeId="2" ParentId="10765" CreationDate="2016-03-18T18:48:55.093" Score="3" Body="&lt;p&gt;I would use SQL and create a set of structured fields that are common across all applications (name, school, years of experience, job they are applying for, etc.) with a field containing the raw application for you 'semi-structured' part of the data. You can always do something fancy with the raw application field, but if you want to get some summary statistics quickly then SQL is the proper route.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, thinking carefully about your schema up front will pay big dividends in the future. I would recommend NoSQL only if you really have no idea how to structure things and are pressed for time. Even getting a list of the keys in a collection requires &lt;a href=&quot;https://stackoverflow.com/questions/2298870/mongodb-get-names-of-all-keys-in-collection&quot;&gt;MapReduce&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-18T18:48:55.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10780" PostTypeId="1" AcceptedAnswerId="10781" CreationDate="2016-03-18T21:29:56.677" Score="1" ViewCount="293" Body="&lt;p&gt;I'm using sklearn/pandas/numpy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a labeled data set, where the potential outcomes are either True or False. However, the data set has a much higher proportion of True entries. When running through classifiers with k-fold (n=5) cross validation, this appears to bias the classifier towards just saying True.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using weights, I was able to adjust the sample data set I'm using to have a proportion closer to 1:1, like so (using a pandas csv):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;results = csv[['result']]&#xA;weights = np.where(results.as_matrix() == True,0.25,1).ravel()&#xA;csv_sample = csv.sample(n=60000, weights=weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the results are much more promising! However, I'm wondering if there's a way for me to do cross validation where the TRAINING set is adjusted in this manner, but the TEST set is closer to the actual proportion of data.&lt;/p&gt;&#xA;" OwnerUserId="17159" LastActivityDate="2016-03-19T00:26:16.793" Title="Sklearn: How to adjust data set proportion during training, but not testing" Tags="|scikit-learn|pandas|cross-validation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10781" PostTypeId="2" ParentId="10780" CreationDate="2016-03-19T00:26:16.793" Score="1" Body="&lt;p&gt;Try to use predictor option class_weight='balanced' or auto. It worked really well for me for &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;nofollow&quot;&gt;SGDClassifier&lt;/a&gt; in a similar situation.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-03-19T00:26:16.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10782" PostTypeId="2" ParentId="10368" CreationDate="2016-03-19T03:36:05.867" Score="3" Body="&lt;p&gt;This example trains an image and a partial caption to predict the next word in the caption. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: [, &quot;&amp;lt;BEGIN&amp;gt; The cat sat on the&quot;]&#xA;Output: &quot;mat&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice the model doesn't predict the entire output of the caption only the next word. To construct a new caption, you would have to predict multiple times for each word.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: [, &quot;&amp;lt;BEGIN&amp;gt;&quot;] # predict &quot;The&quot;&#xA;Input: [, &quot;&amp;lt;BEGIN&amp;gt; The&quot;] # predict &quot;cat&quot;&#xA;Input: [, &quot;&amp;lt;BEGIN&amp;gt; The cat&quot;] # predict &quot;sat&quot;&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To predict the entire sequence, I believe you need to use &lt;code&gt;TimeDistributedDense&lt;/code&gt; for the output layer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input: [, &quot;&amp;lt;BEGIN&amp;gt; The cat sat on the mat&quot;]&#xA;Output: &quot;The cat sat on the mat &amp;lt;END&amp;gt;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See this issue: &lt;a href=&quot;https://github.com/fchollet/keras/issues/1029&quot; rel=&quot;nofollow&quot;&gt;https://github.com/fchollet/keras/issues/1029&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="588" LastActivityDate="2016-03-19T03:36:05.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10783" PostTypeId="1" CreationDate="2016-03-19T09:48:01.243" Score="14" ViewCount="68825" Body="&lt;p&gt;I have two data frames df1 and df2 and I would like to merge them into a single data frame. It is as if df1 and df2 were created by splitting a single data frame down the center vertically, like tearing a piece of paper that contains a list in half so that half the columns go on one paper and half the columns go on the other. I would like to merge them back together. How do I do it?&lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2020-09-23T17:19:43.393" Title="How do I merge two data frames in Python Pandas?" Tags="|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10784" PostTypeId="2" ParentId="10783" CreationDate="2016-03-19T10:28:18.930" Score="7" Body="&lt;p&gt;Pandas has a built-in merge function. Please refer to the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;End-result should be something like this:&#xA;df_merged = pd.merge(df1, df2, left_on=['name_indexcolumn_df1_here'],&#xA;              right_on=['name_indexcolumn_df2_here'],&#xA;              how='inner')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9186" LastEditorUserId="16284" LastEditDate="2016-03-21T17:20:52.007" LastActivityDate="2016-03-21T17:20:52.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10785" PostTypeId="1" CreationDate="2016-03-19T10:43:03.593" Score="3" ViewCount="445" Body="&lt;p&gt;I'm currently involved in an aspect level sentiment analysis project and using Stanford CoreNLP toolkit to implement the system. My knowledge on these concepts is very limited and I'm seeking your help to clarify some things related to both machine learning and classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of sentences, which are travel reviews that needs to be categorized under different labels (AMBIANCE, SCENERY, COST, ENTERTAINMENT etc). This is done by checking if certain aspect terms related to the aforementioned categories are found in the sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to train a classifier (I'm using a Stanford classifier), to classify these sentences to respective categories and I have a train data set containing around 3000+ sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is, a sentence may contain aspect terms that belong to not just one, but multiple aspect categories. In that case, I want the classifier to classify them all at once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Review Sentence:&#xA;&#xA;  On the upside it was very **calm** there and good for **swimming** &#xA;&#xA;Categories&#xA;&#xA;  AMBIANCE, ENTERTAINMENT &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried to pre-process the training dataset in this manner, and trained a classifier. But when I tried to make it classify a sentence containing aspect terms belonging to two different categories, it only identified one category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training dataset was of the following format&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentence1    [tab]   category1&#xA;sentence2    [tab]   category1, category2&#xA;sentence3    [tab]   category2, category3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone please tell me if my approach is wrong? How can I achieve the desired output? I would greatly appreciate any help on this matter, as I'm currently stuck at this phase of my project.&lt;/p&gt;&#xA;" OwnerUserId="17165" LastEditorUserId="31513" LastEditDate="2017-05-10T04:00:10.160" LastActivityDate="2017-05-10T04:00:10.160" Title="How to implement multi class classifier for a set of sentences?" Tags="|machine-learning|classification|nlp|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10786" PostTypeId="2" ParentId="9826" CreationDate="2016-03-19T11:19:06.693" Score="11" Body="&lt;p&gt;In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains the semantic meaning of all the words in the context trained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we compare this to word2vec, each word in word2vec preserves its own semantic meaning. Thus summing up all the vectors or averaging them will result in a vector which could have all the semantics preserved. This is sensible, because when we add the vectors (transport+water) the result nearly equals ship or boat, which means summing the vectors sums up the semantics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before the paragraph vector paper got published, people used averaged word vectors as sentence vectors. To be honest, in my work these average vectors work better than document vectors. So, with these things in mind, in this way it could be compared.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-03-19T11:19:06.693" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10787" PostTypeId="1" AcceptedAnswerId="10788" CreationDate="2016-03-19T16:17:37.910" Score="1" ViewCount="433" Body="&lt;p&gt;I have a pandas dataframe df, and I want to show the histogram.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.hist(bins=100, label=&quot;myhist&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, for some reason I have lots of zeros in this df, so I only want to show the values between 1 and 100. I tried&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.hist(bins=(1,100), label=&quot;myhist&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but that gives a flat line which has nothing to do with the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do it?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2016-03-19T18:25:42.490" Title="Histogram of some values only" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10788" PostTypeId="2" ParentId="10787" CreationDate="2016-03-19T18:25:42.490" Score="1" Body="&lt;p&gt;Ok, after some digging around I found that I can pass a range = (1,100) and that does the trick.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2016-03-19T18:25:42.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10789" PostTypeId="1" CreationDate="2016-03-19T20:53:31.690" Score="3" ViewCount="603" Body="&lt;p&gt;I'd like to do normal machine learning with Python and Sklearn, but occasionally use the nice graphical widgets or Orange for presentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this possible?&lt;/p&gt;&#xA;" OwnerUserId="723" LastActivityDate="2018-05-10T07:49:55.653" Title="Can I run Orange widgets from normal Python scripts?" Tags="|python|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10790" PostTypeId="1" CreationDate="2016-03-20T00:08:52.193" Score="1" ViewCount="265" Body="&lt;p&gt;I have some input data which can belong to different outputs. I'm trying to determine if there's a pattern between the inputs and the output. I'm exploring using multi-label support vector machines in Accord and C#.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not sure if it's exactly what I want. My data is in tuples for the input, and a single int for the output, however depending on the more occurances between similar inputs and the same output, I'd like a stronger relationship to be shown, and an output that reflects this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would seem support vector machines give either a yes/no answer as in, &quot;yes these inputs corrolate to this class&quot;, and doesn't offer any ranking between other classes the inputs could belong to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What options do I have to explore ranking between multiple labels?&lt;/p&gt;&#xA;" OwnerUserId="2861" LastActivityDate="2016-03-20T00:08:52.193" Title="Multi-label ranking" Tags="|ranking|labels|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10791" PostTypeId="1" CreationDate="2016-03-20T05:04:16.053" Score="3" ViewCount="341" Body="&lt;p&gt;Suppose we had some data points $\{(x_i,y_i)\}$ where $x_i$ is a real number and $y_i$ is zero or one. We want to fit the logistic function to the data where the logistic function is $\hat{y}=\frac{1}{1+e^{-(\beta x +\beta_0)}}$. To do this we would like to choose $\beta_1$ and $\beta_0$ in order to minimize the residual sum of squares, $RSS = \sum_i (y_i-\hat{y}_i)^2$.[nevermind that cross-entropy is a better cost function.] I believe there is only a single minimum. How can I know? Does anyone know of a proof for this or a counter example?&lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2016-03-22T00:20:22.480" Title="How many minima does the residual sum of squares have for the logistic curve?" Tags="|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10795" PostTypeId="2" ParentId="10211" CreationDate="2016-03-20T08:40:07.487" Score="6" Body="&lt;p&gt;If you are seeking a working solution, I know of an API that supports many languages, including Russian: &lt;a href=&quot;https://indico.io/docs#sentiment&quot; rel=&quot;nofollow noreferrer&quot;&gt;indico.io Text Analysis sentiment()&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import indicoio&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.config.api_key = YOUR_API_KEY&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.sentiment(u&quot;Это круто, убивает!  Хочу.&quot;, language='ru')&#xA;0.6978093435482927&#xA;&amp;gt;&amp;gt;&amp;gt; indicoio.sentiment(u&quot;Ты кто такой?  Давай досвидания&quot;, language='ru')&#xA;0.13258737684773209&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the &lt;code&gt;language&lt;/code&gt; parameter is optional&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Obviously it's not a lib, but they offer a Python client and their free tier is generous enough.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: As of Q2 2018, Google and IBM sentiment analysis APIs still do not support Russian.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="12363" LastEditDate="2018-06-21T06:12:03.607" LastActivityDate="2018-06-21T06:12:03.607" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10796" PostTypeId="1" CreationDate="2016-03-20T09:06:00.657" Score="1" ViewCount="425" Body="&lt;p&gt;I use square loss to measure the difference between the output of the deconvolution network and the input image, use learning rate 0.01 and use SGD to optimize, code is written in tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_params = [[&quot;layer1&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 3, 64],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer2&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 64, 64],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer3&quot;,{&quot;type&quot;:&quot;max_pooling&quot;, &quot;ksize&quot;:[1, 2, 2, 1],  &quot;padding&quot;:&quot;same&quot;}], # ksize&quot;:[1, 2, 2, 1], &#xA;                         [&quot;layer4&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 64, 128], &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer5&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 128, 128],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer6&quot;,{&quot;type&quot;:&quot;max_pooling&quot;, &quot;ksize&quot;:[1, 2, 2, 1], &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer7&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 128, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer8&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer9&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256], &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer10&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer11&quot;,{&quot;type&quot;:&quot;max_pooling&quot;, &quot;ksize&quot;:[1, 2, 2, 1], &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer12&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 1024],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer13&quot;,{&quot;type&quot;:&quot;conv_relu&quot;, &quot;kernel&quot;:[1, 1, 1024, 1024], &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer14&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 1024],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer15&quot;,{&quot;type&quot;:&quot;demax_pooling&quot;}],&#xA;                         [&quot;layer16&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer17&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer18&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 256, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer19&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 128, 256],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer20&quot;,{&quot;type&quot;:&quot;demax_pooling&quot;}],&#xA;                         [&quot;layer21&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 128, 128],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer22&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 64, 128],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer23&quot;,{&quot;type&quot;:&quot;demax_pooling&quot;}],&#xA;                         [&quot;layer24&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 64, 64],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         [&quot;layer25&quot;,{&quot;type&quot;:&quot;deconv_relu&quot;, &quot;kernel&quot;:[3, 3, 3, 64],  &quot;padding&quot;:&quot;same&quot;}],&#xA;                         ]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but the loss did not go down, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2016-03-20 16:36:19.764441: step 0, loss = 54437.99 (24.1 examples/sec; 5.306 sec/batch)&#xA;2016-03-20 16:36:42.029972: step 10, loss = 54460.74 (65.3 examples/sec; 1.959 sec/batch)&#xA;2016-03-20 16:37:02.488069: step 20, loss = 54398.99 (61.1 examples/sec; 2.096 sec/batch)&#xA;2016-03-20 16:37:25.320226: step 30, loss = 54171.65 (61.0 examples/sec; 2.100 sec/batch)&#xA;2016-03-20 16:37:45.594397: step 40, loss = 54180.36 (62.6 examples/sec; 2.044 sec/batch)&#xA;2016-03-20 16:38:08.296269: step 50, loss = 54444.09 (63.4 examples/sec; 2.020 sec/batch)&#xA;2016-03-20 16:38:28.625820: step 60, loss = 54644.72 (61.0 examples/sec; 2.099 sec/batch)&#xA;2016-03-20 16:38:50.959914: step 70, loss = 54447.85 (61.6 examples/sec; 2.077 sec/batch)&#xA;2016-03-20 16:39:11.174660: step 80, loss = 54334.46 (61.7 examples/sec; 2.074 sec/batch)&#xA;2016-03-20 16:41:16.831728: step 140, loss = 54293.51 (67.0 examples/sec; 1.911 sec/batch)&#xA;2016-03-20 16:41:38.710691: step 150, loss = 54386.80 (59.9 examples/sec; 2.138 sec/batch)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to check where go wrong or maybe someone have seen this problem before&lt;/strong&gt;?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/o5xz1.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o5xz1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17176" LastActivityDate="2016-03-20T09:06:00.657" Title="Why does the square loss not decrease in deconvolution network?" Tags="|neural-network|deep-learning|tensorflow|loss-function|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10797" PostTypeId="2" ParentId="10789" CreationDate="2016-03-20T13:35:11.463" Score="1" Body="&lt;p&gt;I don't think Orange was ever intended to be used that way, but if you can convert your data into &lt;code&gt;Orange.data.Table&lt;/code&gt;, you might be able to instantiate the widgets you need with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the &lt;code&gt;if __name__ == '__main__'&lt;/code&gt; block at the bottom of most widgets for some example.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-03-20T13:35:11.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10799" PostTypeId="2" ParentId="2368" CreationDate="2016-03-20T14:05:48.093" Score="5" Body="&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/questions/2368/machine-learning-features-engineering-from-date-time-data/#2370&quot;&gt;Ben&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/2368/machine-learning-features-engineering-from-date-time-data/#2369&quot;&gt;Nar&lt;/a&gt; nicely explained, breaking down the date-time object into buckets of date and time parts would help detect seasonal trends, where the complete (and usually even worse - unique) date-time object would miss it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You didn't mention any specific machine learning algorithm you're interested in, but in case you're also interested with distance-based clustering, like k-means, I'd &lt;strong&gt;generalize&lt;/strong&gt; the date-time object into the &lt;strong&gt;unix-time format&lt;/strong&gt;.&#xA;This would allow for a simple numerical distance comparison for the algorithm, simply stating how far 2 date values are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example I'd generalize the date-only value 2014-05-05 to 1399248000 (the unix time representing the start of may the 5th 2014, UTC).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[One could argue that you can achieve that by bucketing the date-time into every possible date-time part.. but that would significantly increase your dataset dimensions. So, I'd suggest combining the unix-time, for distance measuring, and some of the date-time buckets]&lt;/p&gt;&#xA;" OwnerUserId="17177" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-03-20T14:05:48.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10800" PostTypeId="1" CreationDate="2016-03-20T14:27:08.700" Score="1" ViewCount="938" Body="&lt;p&gt;I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The game goes on for roughly 1 million steps. At each step, I have 36 &lt;em&gt;continuous&lt;/em&gt; features available (unknown rules), and 4 actions to choose from. Afterwards, the environment will tell me what the 36 features are for the next step, and what my score is. I've been creating and looking at the data, and it sometimes only becomes clear a move was good/bad a lot later (10 moves or so).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there are 2 things involved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I have to &quot;learn&quot; how, given an action, the world would change from pre_state to post_state, and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I have to learn to optimize the reward given the state and choosing from 4 actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was thinking to just record state_before, state_after, reward of a lot of random moves. I could perhaps just use 36 * 10 of the last moves as predictors. But then again, perhaps I should only be interested in the &lt;em&gt;difference between&lt;/em&gt; states?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem lies in the fact that I've looked at Markov Decision Processes, but they assume a discrete search space (whereas here it is continuous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be useful in trying to understand what &lt;em&gt;layers should be involved&lt;/em&gt; to solve this, and perhaps what the most logical way would be to &lt;em&gt;sample data&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really hope people can point me in the right direction. I'm willing to use any neural networks framework in Python basically.&lt;/p&gt;&#xA;" OwnerUserId="17178" LastActivityDate="2016-08-16T20:54:23.830" Title="Combining Neural Network with Reinforcement Learning in a Continuous Space" Tags="|neural-network|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10801" PostTypeId="2" ParentId="10168" CreationDate="2016-03-20T15:54:55.880" Score="1" Body="&lt;p&gt;Maybe have a look at &lt;a href=&quot;https://swarbrickjones.wordpress.com/2015/04/29/convolutional-autoencoders-in-pythontheanolasagne/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; post. You can do a convolution which produces an output of similar size and then &quot;unpool&quot; those feature maps.&lt;/p&gt;&#xA;" OwnerUserId="17181" LastActivityDate="2016-03-20T15:54:55.880" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10802" PostTypeId="1" AcceptedAnswerId="10820" CreationDate="2016-03-20T16:11:45.457" Score="15" ViewCount="5087" Body="&lt;p&gt;I understand from Hinton's paper that T-SNE does a good job in keeping local similarities and a decent job in preserving global structure (clusterization).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I'm not clear if points appearing closer in a 2D t-sne visualization can be assumed as &quot;more-similar&quot; data-points. I'm using data with 25 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, observing the image below, can I assume that blue datapoints are more similar to green ones, specifically to the biggest green-points cluster?. Or, asking differently, is it ok to assume that blue points are more similar to green one in the closest cluster, than to red ones in the other cluster? (disregarding green points in the red-ish cluster)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xAnH5.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xAnH5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When observing other examples, such as the ones presented at sci-kit learn Manifold learning it seems right to assume this, but I'm not sure if is correct statistically speaking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7L587.png&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7L587.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have calculated the distances from the original dataset manually (the mean pairwise euclidean distance) and the visualization actually represents a proportional spatial distance regarding the dataset. However, I would like to know if this is fairly acceptable to be expected from the original mathematical formulation of t-sne and not mere coincidence.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="5143" LastEditDate="2016-03-21T22:12:57.663" LastActivityDate="2016-09-18T01:58:19.257" Title="Can closer points be considered more similar in T-SNE visualization?" Tags="|visualization|dimensionality-reduction|tsne|manifold|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10803" PostTypeId="1" AcceptedAnswerId="10825" CreationDate="2016-03-20T19:18:18.573" Score="10" ViewCount="2929" Body="&lt;p&gt;I have read that distributional representation is based on distributional hypothesis that words occurring in similar context tends to have similar meanings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2Vec and Doc2Vec both are modeled according to this hypothesis. But, in the original paper, even they are titled as &lt;code&gt;Distributed representation of words and phrases&lt;/code&gt; and &lt;code&gt;Distributed representation of sentences and documents&lt;/code&gt;. So, are these algorithms based on distributional representation or distributed representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about other models such as LDA and LSA.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="843" LastEditDate="2016-03-24T19:33:09.697" LastActivityDate="2016-05-02T11:53:15.783" Title="Are Word2Vec and Doc2Vec both distributional representation or distributed representation?" Tags="|nlp|word-embeddings|terminology|word2vec|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10804" PostTypeId="1" AcceptedAnswerId="10808" CreationDate="2016-03-20T20:33:03.160" Score="1" ViewCount="215" Body="&lt;p&gt;I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not.&#xA;I have done some research and saw that A/B testing might be a way to do this but A/B tests can only be carried out having 2 groups (control and experiment), and in this case all I have is past and &quot;after the decision&quot; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can the statistics behind AB testing be used in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have is in the for of sales per day per article only for one company before and after the decision (only one decision).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sales for articles X and Y from the 2016-01-01 to 2016-02-01.&#xA;The decision was to lower discounts on articles A and B on 2016-01-15, so I want to know if sales decreased or not after this decision.&lt;/p&gt;&#xA;" OwnerUserId="15933" LastEditorUserId="6478" LastEditDate="2016-03-23T13:11:34.583" LastActivityDate="2016-03-23T13:11:34.583" Title="How to determine if a company decision was successful or not?" Tags="|statistics|time-series|experiments|ab-test|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10805" PostTypeId="1" CreationDate="2016-03-21T06:51:17.803" Score="15" ViewCount="21719" Body="&lt;p&gt;I just fitted a logistic curve to some fake data. I made the data essentially a step function. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = -------------++++++++++++++&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I look at the fitted curve, the slope is very small. The function that best minimizes the cost function, assuming cross entropy, is the step function. Why does it not look like a step function? Is there some regularization, L1 or L2, done by default?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cDRXN.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cDRXN.png&quot; alt=&quot;Logistic regression using scikit-learn&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="11097" LastEditDate="2016-03-21T07:26:39.427" LastActivityDate="2016-03-23T11:30:21.753" Title="Does scikit-learn use regularization by default?" Tags="|logistic-regression|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10806" PostTypeId="2" ParentId="10805" CreationDate="2016-03-21T07:34:08.130" Score="7" Body="&lt;p&gt;Yes, there is regularization by default. It appears to be L2 regularization with a constant of 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I played around with this and found out that L2 regularization with a constant of 1 gives me a fit that looks exactly like what sci-kit learn gives me without specifying regularization.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LogisticRegression    &#xA;model = LogisticRegression()&#xA;model.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is the same as&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = LogisticRegression(penalty=&quot;l2&quot;, C=1)&#xA;model.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I chose &lt;code&gt;C=10000&lt;/code&gt;, I got something that looked a lot more like step function. &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="816" LastEditDate="2016-03-23T11:29:34.890" LastActivityDate="2016-03-23T11:29:34.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10807" PostTypeId="2" ParentId="10805" CreationDate="2016-03-21T08:00:45.280" Score="12" Body="&lt;p&gt;Please take a look at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;. The first line shows the default parameters, which include &lt;code&gt;penalty='l2'&lt;/code&gt; and &lt;code&gt;C=1.0&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You actually cannot disable regularization completely, you can only regularize less... try setting &lt;code&gt;C=1e10&lt;/code&gt; for example.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="816" LastEditDate="2016-03-23T11:30:21.753" LastActivityDate="2016-03-23T11:30:21.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10808" PostTypeId="2" ParentId="10804" CreationDate="2016-03-21T10:55:58.397" Score="3" Body="&lt;p&gt;You could use time series approach to model the &quot;what if not&quot; scenario and compare it with your values after the new program introduction. Check the causal_impact packet from Google and you might also find helpful this tutorial for probabilistic programming. &lt;a href=&quot;http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb&quot; rel=&quot;nofollow&quot;&gt;http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-03-21T10:55:58.397" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10809" PostTypeId="1" AcceptedAnswerId="10819" CreationDate="2016-03-21T11:04:21.280" Score="2" ViewCount="1618" Body="&lt;p&gt;I'm working on an animal classification problem, with the data extracted from a video feed. The recording was made in a pen, so the problem is quite challenging with a dark background and many shadows: &lt;a href=&quot;https://i.stack.imgur.com/2B5BE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2B5BE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I tried scikit-image, but then someone helped me with an advanced tool called crf-rnn (&lt;a href=&quot;http://crfasrnn.torr.vision/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://crfasrnn.torr.vision/&lt;/a&gt;) that does a great job segmenting and labelling objects in an image. I did the following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import caffe&#xA;net = caffe.Segmenter(MODEL_FILE, PRETRAINED)&#xA;IMAGE_FILE = '0045_crop2.png'&#xA;input_image = caffe.io.load_image(IMAGE_FILE)&#xA;from PIL import Image as PILImage&#xA;image = PILImage.fromarray(np.uint8(input_image))&#xA;image = np.array(image)&#xA;mean_vec = [np.mean(image[:,:,vals]) for vals in range(image.shape[2])]&#xA;im = image[:, :, ::-1]&#xA;im = im - reshaped_mean_vec&#xA;cur_h, cur_w, cur_c = im.shape&#xA;pad_h = 750 - cur_h&#xA;pad_w = 750 - cur_w&#xA;print(pad_h, pad_w, &quot;999&quot;)&#xA;im = np.pad(im, pad_width=((0, max(pad_h,0)), (0, max(pad_w,0)), (0, 0)), mode = 'constant', constant_values = 255)&#xA;segmentation = net.predict([im])&#xA;segmentation2 = segmentation[0:cur_h, 0:cur_w]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting image segmentation is rather poor (although two cows are recognized correctly):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/eBEr4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eBEr4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use a trained crf-rnn (MODEL_FILE, PRETRAINED), which works well for other problems, but this one is harder. I would appreciate any suggestions on how to pre-process this sort of image to extract the shape of most cows. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2016-03-21T23:12:41.200" Title="Image Segmentation with a challenging background" Tags="|python|classification|neural-network|scikit-learn|image-classification|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10810" PostTypeId="1" CreationDate="2016-03-21T12:01:23.043" Score="16" ViewCount="26934" Body="&lt;p&gt;I would like to know how to match postal addresses when their format differ or when one of them is mispelled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've found different solutions but I think that they are quite old and not very efficient. I'm sure some better methods exist, so if you have references for me to read, I'm sure it is a subject that may interest several persons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution I found (examples are in R) :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Levenshtein distance, which equals the number of characters you have to insert, delete or change to transform one word into another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;agrep(&quot;acusait&quot;, c(&quot;accusait&quot;, &quot;abusait&quot;), max = 2, value = TRUE)&lt;/code&gt;&#xA;&lt;code&gt;## [1] &quot;accusait&quot; &quot;abusait&quot;&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The comparison of phonemes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;library(RecordLinkage)&lt;/code&gt;&#xA;&lt;code&gt;soundex(x&amp;lt;-c('accusait','acusait','abusait'))&lt;/code&gt;&#xA;&lt;code&gt;## [1] &quot;A223&quot; &quot;A223&quot; &quot;A123&quot;&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The use of a spelling corrector &lt;a href=&quot;http://norvig.com/spell-correct.html&quot; rel=&quot;noreferrer&quot;&gt;(eventually a bayesian one like Peter Norvig's)&lt;/a&gt;, but not very efficient on address I guess.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I thought about using the suggestions of Google suggest, but likewise, it is not very efficient on personal postal addresses. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can imagine using a machine learning supervised approach but you need to have stored the mispelled requests of users to do so which is not an option for me. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10275" LastEditorUserId="16284" LastEditDate="2016-03-21T16:41:32.997" LastActivityDate="2019-10-31T18:33:30.420" Title="How to do postal addresses fuzzy matching?" Tags="|text-mining|data-cleaning|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10811" PostTypeId="2" ParentId="10810" CreationDate="2016-03-21T14:02:04.250" Score="4" Body="&lt;p&gt;There are lots of clever ways to extend the Levenshtein distance to give a fuller picture. A brief intro to a pretty useful module (for python) called '&lt;a href=&quot;https://github.com/seatgeek/fuzzywuzzy&quot; rel=&quot;nofollow&quot;&gt;Fuzzy Wuzzy&lt;/a&gt;' is &lt;a href=&quot;http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; by the team at SeatGeek.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple things you can do is partial string similarity (if you have different length strings, say m &amp;amp; n with m &amp;lt; n), then you only match for m characters. You can also separate the string into tokens (individual words) and look at how sets of tokens match or arrange them alphabetically and order them.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-21T14:02:04.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10812" PostTypeId="2" ParentId="10810" CreationDate="2016-03-21T14:42:36.467" Score="10" Body="&lt;p&gt;As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for linking .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See for more information on the Jaro and Jaro-Winkler distance in &lt;a href=&quot;https://journal.r-project.org/archive/2014-1/loo.pdf&quot; rel=&quot;noreferrer&quot;&gt;this journal&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a comparison of different matching techniques, read &lt;a href=&quot;https://cs.anu.edu.au/people/Peter.Christen/publications/tr-cs-06-02.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-03-21T14:42:36.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10814" PostTypeId="2" ParentId="10741" CreationDate="2016-03-21T17:47:29.347" Score="1" Body="&lt;p&gt;Give the whole batch as an input, take the activated output of the first layer and pipe it the next until the final. . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;calculate the gradient decent direction (derivative of the cost function) for each row. U will have the error for each row of ur training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then add the errors and divide the result with number of the training set. Then finally update it weights. &#xA;I know from experience that tracking all these numbers is hard. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what language u use, but if it's Python, numpy will help u a a lot. For example, to add the errors, errors.sum(axis=0). X.dot(weight) to multiply ur entire dataset with ur weight.  So my suggestion is use some matrix library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I want to mention is batch processing is susceptible to local minima which u don't want...especially in complex functions. SDG is a nice algorithm. I'm not sure why u needed batch update. There are many interesting tweaks in NN, don't waste ur time on batch update, use SDG&lt;/p&gt;&#xA;" OwnerUserId="11141" LastActivityDate="2016-03-21T17:47:29.347" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10815" PostTypeId="2" ParentId="10803" CreationDate="2016-03-21T21:28:52.550" Score="5" Body="&lt;p&gt;Effectively, Word2Vec/Doc2Vec is based on &lt;code&gt;distributional hypothesis&lt;/code&gt; where the context for each word is its nearby words. Similarly, LSA takes the entire document as the context. Both techniques solve the &lt;code&gt;word embedding&lt;/code&gt; problem - embed words into a continuous vector space while keeping semantically related words close together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, LDA isn't made to solve the same problem. They deal with a different problem called &lt;code&gt;topic modeling&lt;/code&gt;, which is finding latent topics in a set of documents.&lt;/p&gt;&#xA;" OwnerUserId="14313" LastEditorUserId="14313" LastEditDate="2016-03-21T21:50:28.860" LastActivityDate="2016-03-21T21:50:28.860" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10816" PostTypeId="1" CreationDate="2016-03-21T21:32:25.067" Score="0" ViewCount="146" Body="&lt;p&gt;I need to write a SQL query where I need the data for the year 2014 from May-Dec and complete data (Jan-Dec) of the year 2015. Can anyone help me with this query?&lt;/p&gt;&#xA;" OwnerUserId="13417" LastActivityDate="2016-06-24T14:26:48.653" Title="Data manipulation in SQL" Tags="|sql|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10817" PostTypeId="2" ParentId="10816" CreationDate="2016-03-21T22:35:04.773" Score="0" Body="&lt;p&gt;One date col:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;SELECT columns FROM table WHERE date &gt;= '2015-05-01' AND date &amp;lt; '2016-1-1'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Separate month and year cols:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;SELECT columns, CAST(&#xA;        CAST(year AS VARCHAR(4)) +&#xA;        RIGHT('0' + CAST(month AS VARCHAR(2)), 2) +&#xA;        RIGHT('0' + CAST(day AS VARCHAR(2)), 2) &#xA;     AS DATETIME) as date FROM table WHERE date &gt;= '2015-05-01' AND date &amp;lt; '2016-1-1'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-22T17:18:52.110" LastActivityDate="2016-03-22T17:18:52.110" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10818" PostTypeId="2" ParentId="10783" CreationDate="2016-03-21T22:57:46.467" Score="4" Body="&lt;p&gt;If you split the DataFrame &quot;vertically&quot; then you have two DataFrames that with the same index. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the &lt;strong&gt;merge&lt;/strong&gt; function or the &lt;strong&gt;concat&lt;/strong&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;strong&gt;concat&lt;/strong&gt; with would be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pandas.DataFrame.concat([df1,df2], axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With &lt;strong&gt;merge&lt;/strong&gt; with would be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pandas.Dataframe.merge([df1,df2], left_index=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more complex merging options see the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/merging.html&quot; rel=&quot;nofollow&quot;&gt;Merge, join and concat pandas tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11121" LastActivityDate="2016-03-21T22:57:46.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10819" PostTypeId="2" ParentId="10809" CreationDate="2016-03-21T23:12:41.200" Score="1" Body="&lt;p&gt;It would be appreciated if you could explain precisely what your goal is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;you want to identify what animal is in your picture ?&lt;/li&gt;&#xA;&lt;li&gt;you want to count the number of animals ?&lt;/li&gt;&#xA;&lt;li&gt;you want to get the position of each animal in the picture ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In any case, I know that you can get some already trained neural nets from google or anywhere else. This neural net can be used with caffe as it is the case in this google deepdream stuff on github (look at ):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/google/deepdream/blob/master/dream.ipynb&quot; rel=&quot;nofollow&quot;&gt;https://github.com/google/deepdream/blob/master/dream.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, if you want to highlight or identify the positions of your animals, you'll find this article inspiring:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It explain how to reverse convolutional networks to identify what part of your image helped to recognize what is inside. The found projection get you something similar to your second picture (called a mask), but depending on the neural net you use, you can get better results. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-03-21T23:12:41.200" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="10820" PostTypeId="2" ParentId="10802" CreationDate="2016-03-21T23:52:34.010" Score="5" Body="&lt;p&gt;I would present t-SNE as a smart probabilistic adaptation of the Locally-linear embedding. In both cases, we attempt to project points from a high dimensional space to a small one. This projection is done by optimizing the conservation of local distances (directly with LLE, preproducing a probabilistic distribution and optimizing the KL-divergence with t-SNE). Then if your question is, does it keep global distances, the answer is no. It will depend on the &quot;shape&quot; of your data (if the distribution is smooth, then distances should be somehow conserved). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;t-SNE actually doesn't work well on the swiss roll (your &quot;S&quot; 3D image) and you can see that, in the 2D result, the very middle yellow points are generally closer to the red ones than the blue ones (they are perfectly centered in the 3D image).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An other good example of what t-SNE does is the clustering of handwritten digits. See the examples on this link:&lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;noreferrer&quot;&gt;https://lvdmaaten.github.io/tsne/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-03-21T23:52:34.010" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="10821" PostTypeId="2" ParentId="10791" CreationDate="2016-03-22T00:20:22.480" Score="2" Body="&lt;p&gt;Sigmoid itself is not a convex function (see &lt;a href=&quot;http://web.stanford.edu/class/ee364a/reviews/review2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;) and square loss based on sigmoid, such as $ \left( A - \frac{1}{1 + e^{-z}} \right)^2 $ is not convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply plotting the square loss function with A = 5 shows that it's convex for x &gt; 0 and concave, otherwise&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/v6gD8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/v6gD8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want mathematical proof, take the second derivative and you'll see that it not strictly positive nor negative.&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-03-22T00:20:22.480" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10822" PostTypeId="1" CreationDate="2016-03-22T05:34:44.123" Score="0" ViewCount="149" Body="&lt;p&gt;I have time series for product usage over an year on daily basis. Product usage exhibits seasonality i.e. it usage increase/decreases by more than the normal usage during that time. When i get the current day's data and its more than normal, i want to check if its exhibits seasonality or not and take some actions accordingly. I am thinking of using Double Exponential Smoothing to achieve this. Is this an right approach or is there an better approach to do it.&lt;/p&gt;&#xA;" OwnerUserId="17215" LastEditorUserId="17215" LastEditDate="2016-03-22T13:33:38.257" LastActivityDate="2016-04-23T03:56:16.847" Title="Time Series - Prediction" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10824" PostTypeId="1" CreationDate="2016-03-22T09:42:15.517" Score="4" ViewCount="171" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&#xA;I am trying to use Orange as to classify if a patient has TB based on their coughing sounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the dataset, there are say 100 patients and for each patient we have 10 coughs. For each cough, we have a full feature vector (170 features). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Giving Orange this dataset and training various learning algorithms is fairly straightforward, but the issue that I have is that Orange will consider each feature vector to be independent of another feature vector, which means it will consider every cough of each patient to be independent, and they aren't. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: Is there a way to tell orange that all 10 coughs of a patient belong to that patient, and when performing leave-one-out or cross validation methods, all the coughs from each patient should be excluded in each fold? &lt;/p&gt;&#xA;" OwnerUserId="17222" LastActivityDate="2019-12-04T15:02:45.353" Title="Is it possible to create feature groups in Orange?" Tags="|cross-validation|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10825" PostTypeId="2" ParentId="10803" CreationDate="2016-03-22T10:10:21.420" Score="3" Body="&lt;p&gt;The reply from Andrey Kutuzov via google groups felt satisfactory&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I would say that word2vec algorithms are based on both. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;When people say &lt;code&gt;distributional representation&lt;/code&gt;, they usually mean the &#xA;  linguistic aspect: meaning is context, know the word by its company and &#xA;  other famous quotes. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But when people say &lt;code&gt;distributed representation&lt;/code&gt;, it mostly doesn't have &#xA;  anything to do with linguistics. It is more about computer science &#xA;  aspect. If I understand Mikolov and other correctly, the word &#xA;  &lt;code&gt;distributed&lt;/code&gt; in their papers means that each single component of a &#xA;  vector representation does not have any meaning of its own. The &#xA;  interpretable features (for example, word contexts in case of word2vec) &#xA;  are hidden and &lt;code&gt;distributed&lt;/code&gt; among uninterpretable vector components: &#xA;  each component is responsible for several interpretable features, and &#xA;  each interpretable feature is bound to several components. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;So, word2vec (and doc2vec) uses distributed representations technically, &#xA;  as a way to represent lexical semantics. And at the same time it is &#xA;  conceptually based on distributional hypothesis: it works only because &#xA;  distributional hypothesis is true (word meanings do correlate with their &#xA;  typical contexts). &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;But of course often the terms &lt;code&gt;distributed&lt;/code&gt; and &lt;code&gt;distributional&lt;/code&gt; are &#xA;  used interchangeably, increasing misunderstanding :) &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="18352" LastEditDate="2016-05-02T11:53:15.783" LastActivityDate="2016-05-02T11:53:15.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10826" PostTypeId="2" ParentId="10804" CreationDate="2016-03-22T11:51:20.687" Score="1" Body="&lt;p&gt;You are trying to determine if some event/treatment (i.e. going on sale) has a significant effect on sales of some product_x. An A/B test is appropriate to answer this question if you have both test and control &#xA;groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given: (&lt;strong&gt;T&lt;/strong&gt;)est and (&lt;strong&gt;C&lt;/strong&gt;)ontrol groups, where &lt;strong&gt;O1&lt;/strong&gt; = observation 1, a baseline measurement of normalized sales (i.e. average sales of product_x over some pre-defined time period) and &lt;strong&gt;O2&lt;/strong&gt; = after treatment measure of normalized sales. Between the first and second observation/measurement, &lt;strong&gt;T&lt;/strong&gt; are given experimental treatment &lt;strong&gt;X&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical experimental design - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;T:  O1  X  O2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C:  O1  .  O2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the OP's description it sounds as if you don't have an adequate control group - although not ideal, it is possible to create control-like groups after the fact (sometimes you can use packages to help create matching control groups after the fact e.g., &lt;a href=&quot;https://cran.r-project.org/web/packages/optmatch/optmatch.pdf&quot; rel=&quot;nofollow&quot;&gt;{optmatch}&lt;/a&gt;). In your case, is it possible to get sales data from a similar store to the one you are studying, but one which did not have the sale on product_x? If so, this &lt;em&gt;may&lt;/em&gt; be adequate as a control group. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your description, however, it sounds like you have a pre-test post-design:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given: (&lt;strong&gt;T&lt;/strong&gt;)est group with treatment &lt;strong&gt;X&lt;/strong&gt; and normalized sales before and after the event&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-----time-----&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;T:  O1  X  O2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this design, the primary threat to validity is lack of an adequate control group to ensure that any observed effect is actually due to the experimental treatment (i.e. the Sale). For example, if you observe an increase in over all sales of product_x from the pre to the post-sale period, this finding is consistent with your original hypothesis (A Sale will increase overall sales of product_x), but it is also consistent with some sort of temporal effect that happens to correlate with sales (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Seasonality&quot; rel=&quot;nofollow&quot;&gt;seasonality effects&lt;/a&gt;, where product sales increase during certain time periods like the holiday season).&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-03-23T01:20:04.607" LastActivityDate="2016-03-23T01:20:04.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10828" PostTypeId="2" ParentId="5855" CreationDate="2016-03-22T13:36:03.117" Score="2" Body="&lt;p&gt;I found a number of interesting articles on mathematical analyses of gerrymandering. Hopefully some of these might give you some ideas on detecting / optimising against gerrymander.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udel.edu/johnmack/research/gerrymandering.pdf&quot; rel=&quot;nofollow&quot;&gt;Gerrymandering and Legislator Efficiency&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.maa.org/programs/maa-awards/writing-awards/gerrymandering-and-convexity&quot; rel=&quot;nofollow&quot;&gt;Gerrymander and Convexity&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rangevoting.org/GerryExamples.html&quot; rel=&quot;nofollow&quot;&gt;Recursive Algorithm with Examples&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://digital.lib.washington.edu/researchworks/bitstream/handle/1773/24334/Solbrig_washington_0250O_12376.pdf?sequence=1&amp;amp;isAllowed=y&quot; rel=&quot;nofollow&quot;&gt;Mary Solbrig's Automated District Model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-03-22T13:36:03.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10829" PostTypeId="1" AcceptedAnswerId="10838" CreationDate="2016-03-22T13:52:19.410" Score="4" ViewCount="2788" Body="&lt;p&gt;I have a dataset which looks like-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Variable value year Quarter Location&#xA;A 48.235 2011 Q1 North&#xA;B 65.444 2011 Q2 North&#xA;C 77.453 2011 Q3 North&#xA;D 44.678 2011 Q4 North&#xA;A 88.542 2012 Q1 South&#xA;B 66.566 2012 Q2 South&#xA;C 55.443 2012 Q3 South&#xA;D 78.990 2012 Q4 South&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anybody help me with a code to generate heatmap with more than 2 categorical variables? I need 'variable' and 'Location' to be on y axis and 'Quarter'&amp;amp;' year' to be on x axis. The shades of the data inside heat map should correspond to 'value'.  &lt;/p&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="16284" LastEditDate="2016-04-09T23:30:53.580" LastActivityDate="2016-04-09T23:30:53.580" Title="Heat maps in R with more than 2 categorical variables" Tags="|r|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10831" PostTypeId="2" ParentId="10494" CreationDate="2016-03-22T15:45:50.220" Score="1" Body="&lt;p&gt;I think your process is not really a cluster analysis problem, but instead a time series analysis problem. You have some process that is measuring a value ($Y$ variable) at different points in time ($X$ variable). I would suggest using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Control_chart&quot; rel=&quot;nofollow&quot;&gt;Shewhart Control Chart&lt;/a&gt; or something similar to understand the process that you have. You should also do analysis on the distribution to ensure it is normal, or use the CLT to enable the data to move towards normality.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-03-22T15:45:50.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10832" PostTypeId="2" ParentId="10225" CreationDate="2016-03-22T15:59:26.913" Score="3" Body="&lt;p&gt;If you want to approach this from a SQL perspective, then broadly I would identify any classification variables that cause different behaviour. Then perform something like the following on a number of analysis variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT ClassificationVar1, &#xA;    ClassificationVar2, &#xA;    MIN(AnalysisVar1) as Min_AnalysisVar1, &#xA;    MAX(AnalysisVar1) as Max_AnalysisVar1, &#xA;    MEAN(AnalysisVar1) as Mean_AnalysiVar1, &#xA;    STDEV(AnalysisVar1) as Std_AnalysisVar1,&#xA;    MIN(AnalysisVar2) as Min_AnalysisVar2, &#xA;    MAX(AnalysisVar2) as Max_AnalysisVar2, &#xA;    MEAN(AnalysisVar2) as Mean_AnalysiVar2, &#xA;    STDEV(AnalysisVar2) as Std_AnalysisVar2,&#xA;    etc.&#xA;    FROM YourDataFile&#xA;GROUP BY ClassificationVar1, ClassificationVar2&#xA;ORDER BY ClassificationVar1, ClassificationVar2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would perform this as a one-off exercise on say the most recent year worth of data, then from a speed perspective, I would run this as regularly as you need to, to flag up exceptional data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach, perhaps, which means learning new technologies is a HDFS/Spark then PIG/Python/R solution. But HDFS/Spark have some solutions that come out of the box to do log analysis. 500 million records is probably getting up into the reaches of performance problems with SQL, even with table partitioning and column indexing.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-03-22T15:59:26.913" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10833" PostTypeId="1" CreationDate="2016-03-22T16:46:28.493" Score="3" ViewCount="353" Body="&lt;p&gt;I'm using spikeslab for the first time, but don't understand what the output means. It was suggested to me that I use it to tell which variables my dependent variable is most correlated to, in a ranked order. &#xA;Particuarly, what is &quot;bma&quot; bma.scale&quot; &quot;gnet&quot; and &quot;gnet.scale&quot;? I also don't understand how to read the corresponding plot to the model.Thanks for any help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, this is one of the models I created using spikeslab, with its output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model2_ss &amp;lt;-spikeslab(Risk_Pct ~ Race&#xA;+                    +hispanic&#xA;+                    +Born_In_US&#xA;+                    +Highest_Education&#xA;+                    +Marital_Status&#xA;+                    , na.rm = TRUE, data = LabeledData)&#xA;&amp;gt; model2_ss&#xA;------------------------------------------------------------------- &#xA;Variable selection method     : AIC &#xA;Big p small n                 : FALSE &#xA;Screen variables              : FALSE &#xA;Fast processing               : TRUE &#xA;Sample size                   : 26 &#xA;No. predictors                : 5 &#xA;No. burn-in values            : 500 &#xA;No. sampled values            : 500 &#xA;Estimated mse                 : 0.4299 &#xA;Model size                    : 3 &#xA;&#xA;&#xA;---&amp;gt; Top variables:&#xA;                bma      gnet     bma.scale     gnet.scale&#xA;Marital_Status  0.516     0.516     0.319         0.319&#xA;Born_In_US      -0.469    -0.447    -0.440       -0.419&#xA;Race            0.458     0.421     0.926         0.851&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8BizP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8BizP.png&quot; alt=&quot;Plot of model2_ss&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17231" LastEditorUserId="471" LastEditDate="2016-03-25T18:08:26.127" LastActivityDate="2019-12-12T11:02:00.297" Title="Understanding Spikeslab Output" Tags="|r|bayesian-networks|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10834" PostTypeId="2" ParentId="10665" CreationDate="2016-03-22T16:50:40.443" Score="0" Body="&lt;p&gt;Why don't you use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Control_chart&quot; rel=&quot;nofollow&quot;&gt;Shewhart Control Chart&lt;/a&gt; (more modernly referred to as &lt;a href=&quot;https://en.wikipedia.org/wiki/Six_Sigma&quot; rel=&quot;nofollow&quot;&gt;6-sigma&lt;/a&gt;) with a moving variable length window to understand the process that you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you find your process variability decreases markedly, by having too many results within $±1σ$ then you have a &quot;process change&quot; and that might be defined as a plateau. You will need to decide how many observations you require to establish a plateau.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The as you find your variability increases and your $±3σ$, you know you again have a process change and need to recalculate your limits.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastEditorUserId="16284" LastEditDate="2016-03-23T11:04:56.223" LastActivityDate="2016-03-23T11:04:56.223" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10835" PostTypeId="2" ParentId="10816" CreationDate="2016-03-22T17:57:02.203" Score="1" Body="&lt;p&gt;Use following &lt;code&gt;WHERE&lt;/code&gt; predicate &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-- data (Jan-Dec) of the year 2015&#xA;year = 2015  OR&#xA;-- data for the year 2014 from May-Dec&#xA;year = 2014 and month in ('May','Jun','Jul','Aug','Sep','Oct','Nov','Dec')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Adjust the column names and/or litaral values as required.&#xA;If the month is stored as number you may use &lt;code&gt;month between 5 and 12&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-03-22T17:57:02.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10836" PostTypeId="1" AcceptedAnswerId="13396" CreationDate="2016-03-22T20:04:23.467" Score="40" ViewCount="20832" Body="&lt;p&gt;I am still confused about the difference between &lt;code&gt;Dense&lt;/code&gt; and &lt;code&gt;TimeDistributedDense&lt;/code&gt; of &lt;code&gt;Keras&lt;/code&gt; even though there are already some similar questions asked &lt;a href=&quot;https://github.com/fchollet/keras/issues/1029&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://github.com/fchollet/keras/issues/278&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. People are discussing a lot but no common-agreed conclusions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And even though, &lt;a href=&quot;https://github.com/fchollet/keras/issues/1029#issuecomment-157786333&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, @fchollet stated that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;TimeDistributedDense&lt;/code&gt; applies a same &lt;code&gt;Dense&lt;/code&gt; (fully-connected) operation to every timestep of a 3D tensor.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I still need detailed illustration about what exactly the difference between them. &lt;/p&gt;&#xA;" OwnerUserId="14007" LastActivityDate="2022-02-22T14:20:13.297" Title="The difference between `Dense` and `TimeDistributedDense` of `Keras`" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10837" PostTypeId="2" ParentId="8286" CreationDate="2016-03-22T20:54:32.573" Score="2" Body="&lt;p&gt;Feature Engineering is at the heart of Machine Learning and is rather laborious and time consuming. There have been various attempts at automating feature engineering in hopes of taking the human out of the loop. One specific implementation that does this for classification problems is &lt;a href=&quot;http://auto-sklearn.readthedocs.org/en/master/&quot; rel=&quot;nofollow&quot;&gt;auto-sklearn&lt;/a&gt;. It uses an optimization procedure called &lt;a href=&quot;http://www.cs.ubc.ca/labs/beta/Projects/SMAC/&quot; rel=&quot;nofollow&quot;&gt;SMAC&lt;/a&gt; under the hood to choose the appropriate set of transforms and algorithm (and algorithm parameters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that Trifacta offers a really easy to use tool for data transformation. It has a highly intuitive GUI that allows to set up transformation/ feature engineering maps. There is also a free trial version that can be used for reasonably sized problems. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2016-03-22T20:54:32.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10838" PostTypeId="2" ParentId="10829" CreationDate="2016-03-22T23:21:47.797" Score="2" Body="&lt;p&gt;Here is one way of visualizing the data you have presented. However, I have used some liberties and assumptions to create this plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, create a year_quarter variable that simply concatenates the year and quarter to present the time on the X axis. The following piece of code can do this in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;year_quarter = paste(dat$year, dat$Quarter, sep=&quot;-&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, the dataset you have will look like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Variable  value year Quarter Location year_quarter&#xA;1        A 48.235 2011      Q1    North      2011-Q1&#xA;2        B 65.444 2011      Q2    North      2011-Q2&#xA;3        C 77.453 2011      Q3    North      2011-Q3&#xA;4        D 44.678 2011      Q4    North      2011-Q4&#xA;5        A 88.542 2012      Q1    South      2012-Q1&#xA;6        B 66.566 2012      Q2    South      2012-Q2&#xA;7        C 55.443 2012      Q3    South      2012-Q3&#xA;8        D 78.990 2012      Q4    South      2012-Q4&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, using &lt;code&gt;ggplto2&lt;/code&gt;, you can create the plot such that the colour represents the value, the shape represents the Location and the size represents the Variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This simple one liner can help you produce such a plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;p = ggplot(dat, aes(x = year_quarter, y = value, colour = value)) + geom_point(aes(shape = Location,size = Variable))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how the output plot looks like: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gouOb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gouOb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you can also add &lt;code&gt;geom_line&lt;/code&gt; with &lt;code&gt;interaction&lt;/code&gt; if you would like the lines to be connected based on Location and Variable. &lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2016-03-22T23:21:47.797" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10839" PostTypeId="1" CreationDate="2016-03-23T03:25:53.170" Score="19" ViewCount="13939" Body="&lt;p&gt;I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences between them are?&lt;/p&gt;&#xA;" OwnerUserId="2504" LastEditorUserId="29169" LastEditDate="2019-06-08T03:34:47.823" LastActivityDate="2020-08-16T17:03:00.027" Title="What is difference between one hot encoding and leave one out encoding?" Tags="|machine-learning|data-mining|feature-selection|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="10840" PostTypeId="1" CreationDate="2016-03-23T06:51:16.860" Score="0" ViewCount="45" Body="&lt;p&gt;I am building a randoForest model on some data. To be able to tune the model faster, I was working with 10% sample. I have tuned it now and now I want to build the model on the whole (100% data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering, if I will have to increase the number of trees now that I am building the model on more training data points? Please let me know.&lt;/p&gt;&#xA;" OwnerUserId="12377" LastActivityDate="2016-05-30T14:54:45.920" Title="does random forest require more trees to be built if # rows increase in train data" Tags="|predictive-modeling|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10841" PostTypeId="1" AcceptedAnswerId="12380" CreationDate="2016-03-23T07:09:34.330" Score="1" ViewCount="450" Body="&lt;p&gt;I have a big list of spare parts with several parameters (material, weight, size, manufacturing complexity, ...). For some parts in this list, a price has either not been set or has to be adjusted in order to be in line with other parts. There are a few obvious and simple correlations in this dataset, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if material and complexity is the same, bigger parts are more expensive;&lt;/li&gt;&#xA;&lt;li&gt;if size and material is the same, more complex parts are more expensive;&lt;/li&gt;&#xA;&lt;li&gt;for equal size and complexity, more expensive material leads to higher price.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Trying to figure out all these rules by hand and sticking them together seems to be an endless endeavour, so I thought about training a neural network with the priced parts' parameters (input) and prices (output) and let it figure out the prices for the parts which don't have a price yet. The decisions of the NN could be supervised by an expert who knows the parts and could manually figure out a price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1) Is this a good idea in general?&lt;br&gt;&#xA;Question 2) If yes, what type of NN would be most suited for such a problem?&lt;/p&gt;&#xA;" OwnerUserId="17246" LastEditorUserId="17246" LastEditDate="2016-03-23T07:37:17.280" LastActivityDate="2016-06-24T05:11:39.740" Title="Pricing decisions using neural network" Tags="|neural-network|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10842" PostTypeId="1" AcceptedAnswerId="11871" CreationDate="2016-03-23T08:07:54.790" Score="0" ViewCount="352" Body="&lt;p&gt;I'm newbie to Data Science. I'm trying to understand how to correlate the position of an app in the app store, e.g. 1-10, to the number of backlinks, e.g. #1 = 250 links, #8 = 50 links, to the app store listing page in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've manage to correlate both rows entirely, which obviously gives me a nonsense figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone point me in the right direction to allow me to fill this knowledge gap. I'm not entirely sure of the correct terminology for this, making Googling it pretty impossible.&lt;/p&gt;&#xA;" OwnerUserId="17248" LastActivityDate="2016-06-21T17:48:51.127" Title="How/What Correlation for Position Ranking?" Tags="|python|pandas|correlation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10843" PostTypeId="2" ParentId="10839" CreationDate="2016-03-23T12:06:40.467" Score="20" Body="&lt;p&gt;They are probably using &amp;quot;leave one out encoding&amp;quot; to refer to Owen Zhang's strategy.&lt;/p&gt;&#xA;&lt;p&gt;From &lt;a href=&quot;https://www.kaggle.com/c/caterpillar-tube-pricing/forums/t/15748/strategies-to-encode-categorical-variables-with-many-categories&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, excluding the row itself. This gives you the advantage of having a one-column representation of the categorical while avoiding direct response leakage&lt;/p&gt;&#xA;&lt;p&gt;This picture expresses the idea well.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oCj1P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oCj1P.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17250" LastEditorUserId="98307" LastEditDate="2020-08-16T17:03:00.027" LastActivityDate="2020-08-16T17:03:00.027" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="10844" PostTypeId="1" AcceptedAnswerId="10863" CreationDate="2016-03-23T13:46:49.460" Score="10" ViewCount="6866" Body="&lt;p&gt;I'd like to test a new algorithm for &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;noreferrer&quot;&gt;collaborative filtering&lt;/a&gt;. A typical use case is to recommend movies based on the preferences of users similar to the specific user. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for collaborative filtering.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastActivityDate="2017-11-03T17:58:37.833" Title="Benchmark datasets for collaborative filtering" Tags="|dataset|recommender-system|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10845" PostTypeId="2" ParentId="10706" CreationDate="2016-03-23T13:51:55.687" Score="1" Body="&lt;p&gt;I think the following synthetic example explains what is going on:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.base import clone&#xA;from sklearn.svm import SVC&#xA;import numpy as np&#xA;from sklearn.datasets import make_blobs&#xA;&#xA;X, y = make_blobs(1000, centers=[[1, 1], [-1, -1], [1, -1]], cluster_std=0.8)&#xA;&#xA;models = [&#xA;    ('normal', SVC()),&#xA;    ('balanced', SVC(class_weight='balanced')),&#xA;]&#xA;&#xA;import matplotlib.pyplot as plot&#xA;plot.ioff()&#xA;&#xA;for i, (name, m) in enumerate(models):&#xA;    for k in np.unique(y):&#xA;        m = clone(m).fit(X, (y == k).astype(int))&#xA;&#xA;        xx, yy = np.meshgrid(np.arange(-3, 3, 0.1), np.arange(-3, 3, 0.1))&#xA;        z = m.predict(np.c_[xx.ravel(), yy.ravel()])&#xA;        z = z.reshape(xx.shape)&#xA;        colors = ['blue', 'green', 'red']&#xA;        plot.contour(xx, yy, z, colors=colors[k])&#xA;    plot.scatter(X[:, 0], X[:, 1], c=y)&#xA;    plot.title(name)&#xA;    plot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/REd4p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/REd4p.png&quot; alt=&quot;normal SVM&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/euDKR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/euDKR.png&quot; alt=&quot;balanced SVM&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The balanced SVM is clearly superior when considering each model individually, at least as measured by the F1 score, which punishes false positives and false negatives equally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But One-vs-Rest uses maximum score, which in this case is given by the distance to the decision hyperplane. It is not yet clear why this is worse in some cases, but clearly it is possible for one model to beat another for individual classes, but to lose for One-vs-Rest: how well the relative distances are modeled is what counts in One-vs-Rest.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-03-23T13:51:55.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10846" PostTypeId="2" ParentId="10824" CreationDate="2016-03-23T14:05:11.313" Score="1" Body="&lt;p&gt;This is not Orange-specific, but IIUC, you could preprocess your data (e.g. in Python or &lt;a href=&quot;https://www.libreoffice.org/discover/calc/&quot; rel=&quot;nofollow&quot;&gt;Excel&lt;/a&gt;) to have each of the 10 coughs pertaining to a patient on the same patients line. Thus you would have: 100 rows of patients with each row (10*170 + other patient data) attributes wide.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-03-23T14:05:11.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10847" PostTypeId="1" AcceptedAnswerId="14746" CreationDate="2016-03-23T14:18:50.757" Score="1" ViewCount="1421" Body="&lt;p&gt;I have something of a ground floor question I’d like to ask.  I’m looking at various options for recommendation engines using Spark.  I feel that I have a decent grasp of the basics of collaborative filtering and have been able to replicate the simple examples provided by the &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;Spark&lt;/a&gt; docs and &lt;a href=&quot;http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html&quot; rel=&quot;nofollow&quot;&gt;AMPlab&lt;/a&gt;.  However, I am seeking to investigate how to create an effective Product Recommendation Algorithm using Implicit Feedback where there is an abundance of user level data available.  Let’s say that I have a large dataset with individual user attributes such as a unique user ID, Product ID, user demographic data, user behavioral data, etc.  What this data set does not contain is an explicit rating that the user gave the product.  In the absence of that explicit feedback, is it still possible to train a Recommendation model using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ALS.trainImplicit(ratings, rank, numIterations, alpha=0.01)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I found this statement from the Spark website to be a bit nebulous: “If the rating matrix is derived from other source of information (i.e., it is inferred from other signals), you can use the trainImplicit method to get better results.”  Does this mean that the large table from my hypothetical example would need to be compacted or reduced to a singular value for the ‘rank’ argument in the trainImplicit function? I have looked everywhere I can think of, but cannot find an accessible example describing how this would be accomplished.  If anyone knows of a good resource, please let me know.  Lastly, if anyone provides a code example, please do so in Python since that is my preferred Spark API. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in Advance.  &lt;/p&gt;&#xA;" OwnerUserId="17256" LastActivityDate="2016-10-25T00:18:48.493" Title="Implicit Training Models in Spark MLlib?" Tags="|python|apache-spark|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10848" PostTypeId="1" CreationDate="2016-03-23T14:35:18.510" Score="3" ViewCount="2950" Body="&lt;p&gt;I have a confusion matrix with 7 classes and would like to represent the matrix in a graph. Something similar like a confusion wheel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mainly I need to show, the correct observations in each class and incorrect ones with other classes in a pie chart.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any python implementations would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rb1wq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rb1wq.png&quot; alt=&quot;confusion wheel&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;http://www.cvast.tuwien.ac.at/ConfusionAnalysis/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cvast.tuwien.ac.at/ConfusionAnalysis/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="55122" LastEditDate="2021-02-26T15:49:43.450" LastActivityDate="2021-02-26T15:49:43.450" Title="Python plot for confusion matrix similar to confusion wheel?" Tags="|python|visualization|multiclass-classification|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10849" PostTypeId="1" AcceptedAnswerId="10850" CreationDate="2016-03-23T14:51:53.353" Score="3" ViewCount="5676" Body="&lt;p&gt;I've been trying to find an answer to this question for a long time.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What are the relationships/differences between Bias, Variance and&#xA;Residuals?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I think I do understand Bias, Variance and Residuals as separate concepts. Correct me if I'm wrong - &lt;br&gt;&lt;br&gt;&#xA;&lt;b&gt;Bias&lt;/b&gt; is the difference between the average expected results from different runs of the model and the true values from data. &lt;br&gt;&lt;br&gt;&#xA;&lt;b&gt;Variance&lt;/b&gt; is the variability in the expected results (predictions) of a given data point between different runs of the model. &lt;br&gt;&lt;br&gt;&#xA;&lt;b&gt;Residual&lt;/b&gt; is the difference between the expected results from a model and the true values from data.&lt;br&gt; y - y^ &lt;br&gt;Residual seems somewhat similar to Bias. But is that my misunderstanding?&lt;br&gt;&lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;Please explain to me in plain words and also simple equations the relationships or differences between these three.&lt;/p&gt;&#xA;" OwnerUserId="2647" LastEditorUserId="8820" LastEditDate="2020-11-22T06:18:46.927" LastActivityDate="2020-11-22T06:18:46.927" Title="What are the relationships/differences between Bias, Variance and Residuals?" Tags="|machine-learning|predictive-modeling|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10850" PostTypeId="2" ParentId="10849" CreationDate="2016-03-23T15:23:30.583" Score="4" Body="&lt;p&gt;I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bias&lt;/strong&gt;, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Variance&lt;/strong&gt;, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the &quot;stability&quot; of a given algorithm/parameter set when exposed to different training and testing sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(N.B. This can be confusing because there is a specific definition of &quot;variance&quot; used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Residuals&lt;/strong&gt; are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.&lt;/p&gt;&#xA;" OwnerUserId="17078" LastActivityDate="2016-03-23T15:23:30.583" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10851" PostTypeId="2" ParentId="10848" CreationDate="2016-03-23T15:34:35.063" Score="4" Body="&lt;p&gt;I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, there really isn't a great way to visualize a large confusion matrix. The best ones I've seen are a grid-based plot with either color-coding or size-coding representing the value in each table cell. Funnily enough, the visualization in Fig. 1(a) of the link you included is significantly easier to read and more informative than the confusion wheel in Fig 1(b). A grid graph like the one in the link's Fig. 1(a) is what I would recommend.&lt;/p&gt;&#xA;" OwnerUserId="17078" LastActivityDate="2016-03-23T15:34:35.063" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10852" PostTypeId="2" ParentId="8982" CreationDate="2016-03-23T15:37:03.153" Score="2" Body="&lt;p&gt;There are several things that can be done, however as @Dawny33 mentioned, start with the problem you want solved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, If I was trying to solve: Which factors, influence the prices and how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple analysis could be to run a linear regression model (&lt;code&gt;lm&lt;/code&gt;) with &lt;code&gt;PriceSqFt&lt;/code&gt; as a dependent variable and others as regressors. You can then select top 10 or 15 or how so ever many you like and look at their relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may help you answer some simple questions like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;How do prices vary by dealers, by property type, by location, by size?&#xA;Does move in ready date make a difference to rent / prices?&#xA;Do specific kind of properties have later than usual ready dates?&#xA;Is there a negative or positive correlation between size and pricesqft?&#xA;Are certain dealers over or under priced compared to other identical&#xA;properties and location?  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also create a plots using &lt;code&gt;pairs&lt;/code&gt; or other functions to look at pairwise relationship for initial analysis and building on top of what you see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to take it a to the next level, you can create a prediction model to predict the price based on select parameters. &lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2016-03-23T15:37:03.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10853" PostTypeId="1" CreationDate="2016-03-23T16:07:03.883" Score="2" ViewCount="201" Body="&lt;p&gt;I ran a survey asking this question: 'What do you like the most about your favorite character in the game?' and I got 23K responses; most of them are phrases/sentences; what's the best way to analyze these results such as to find out what the main responses are? A simple word cloud may not do because it strips out the context by isolating words; but I'm open. Would love help. Didn't know how to attached the dataset (my first time on Stack-Ex) so [here][1] it is. Thanks in advance fellow coders/analysts.&lt;/p&gt;&#xA;" OwnerUserId="17239" LastActivityDate="2019-09-14T02:01:55.240" Title="Text Analysis to analyze qualitative (text based) Survey Results" Tags="|machine-learning|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10854" PostTypeId="2" ParentId="10849" CreationDate="2016-03-23T18:22:08.147" Score="4" Body="&lt;p&gt;A graphical explanation how I understood bias and variance from &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;noreferrer&quot;&gt;Andrew Ngs course&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/n6Tfr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/n6Tfr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-03-23T18:22:08.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10856" PostTypeId="1" CreationDate="2016-03-23T19:04:23.127" Score="5" ViewCount="503" Body="&lt;p&gt;I have a data set with a set of users and a history of documents they have read, all the documents have metadata attributes (think topic, country, author) associated with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to cluster the users based on their reading history per one of the metadata attributes associated with the documents they have clicked on. This attribute has 7 possible categorical values and I want to prove a hypothesis that there is a pattern to the users' reading habits and they can be divided into seven clusters. In other words, that users will often read documents based on one of the 7 possible values in the particular metadata category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone have any advice on how to do this especially in R, like specific packages? I realize that the standard k-means algorithm won't work well in this case since the data is categorical and not numeric.&lt;/p&gt;&#xA;" OwnerUserId="17267" LastActivityDate="2018-01-13T14:03:54.040" Title="Categorical Clustering of Users Reading Habits" Tags="|r|clustering|k-means|unsupervised-learning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10857" PostTypeId="1" AcceptedAnswerId="10937" CreationDate="2016-03-23T19:47:30.083" Score="10" ViewCount="46274" Body="&lt;p&gt;I need to scrape a table off of a webpage and put it into a pandas data frame. But I am not being able to do it. Let me first give you a hint of how the table is encoded into html document. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;tbody&amp;gt;&#xA;&amp;lt;tr&amp;gt;&#xA;&amp;lt;th colspan=&quot;2&quot;&amp;gt;United States Total&amp;lt;strong&amp;gt;**&amp;lt;/strong&amp;gt;&amp;lt;/th&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;strong&amp;gt;15,069.0&amp;lt;/strong&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;strong&amp;gt;14,575.0&amp;lt;/strong&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;strong&amp;gt;100.0&amp;lt;/strong&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;/tr&amp;gt;&#xA;&amp;lt;tr&amp;gt;&#xA;&amp;lt;th colspan=&quot;7&quot;&amp;gt;Arizona&amp;lt;/th&amp;gt;&#xA;&amp;lt;/tr&amp;gt;&#xA;&amp;lt;tr&amp;gt;&#xA;&amp;lt;td&amp;gt;Pinal Energy, LLC&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;Maricopa, AZ&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;50.0&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;50.0&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;NA&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;2012-07-01&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;2014-03&amp;lt;/td&amp;gt;&#xA;&amp;lt;/tr&amp;gt;&#xA;&amp;lt;tr&amp;gt;&#xA;&amp;lt;td colspan=&quot;2&quot;&amp;gt;&amp;lt;strong&amp;gt;Arizona Total&amp;lt;/strong&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;50.0&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;50.0&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;NA&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;td&amp;gt;&amp;lt;/td&amp;gt;&#xA;&amp;lt;/tr&amp;gt;&#xA;&amp;lt;tr&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The body of the table begins with &lt;code&gt;&amp;lt;tbody&amp;gt;....&amp;lt;/tbody&amp;gt;&lt;/code&gt;. Each &lt;code&gt;&amp;lt;tr&amp;gt;....&amp;lt;/tr&amp;gt;&lt;/code&gt; is a row of the table.Within each row, that is within each pair of &lt;code&gt;&amp;lt;tr&amp;gt;....&amp;lt;/tr&amp;gt;&lt;/code&gt;, each column is given by &lt;code&gt;&amp;lt;td&amp;gt;50.0&amp;lt;/td&amp;gt;&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my questions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How do I scrape it ? I am using &lt;code&gt;BeautifulSoup&lt;/code&gt; and &lt;code&gt;requests&lt;/code&gt; for this purpose as well as &lt;code&gt;pandas&lt;/code&gt; module. I tried the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;r = requests.get(url)&#xA;bs = BeautifulSoup(r.text)&#xA;info = bs.findALL('tr','td')&#xA;  ....&#xA;  ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it is giving me this error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-24-32d9483e2c59&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 bs = BeautifulSoup(r.text)&#xA;----&amp;gt; 2 info = bs.findALL('tr','td')&#xA;      3 #print bs&#xA;&#xA;TypeError: 'NoneType' object is not callable&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) I need to skip some of the rows based on the text in it. For example I don't want to read in the row in which the word 'Total' appears (as in&lt;code&gt;&amp;lt;th colspan=&quot;2&quot;&amp;gt;United States Total&amp;lt;strong&amp;gt;**&amp;lt;/strong&amp;gt;&amp;lt;/th&amp;gt;&lt;/code&gt;). How do I do that ? Although, it is not extremely important as I can get rid off it later, but skipping these rows while reading the data is ideally what I need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it is a long post, but if someone can help me with it, i would greatly appreciate it. Please let me know if more information is needed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks much.&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2019-07-06T21:17:56.320" Title="How to scrape a table from a webpage?" Tags="|python|pandas|scraping|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10859" PostTypeId="2" ParentId="8982" CreationDate="2016-03-23T20:55:47.693" Score="0" Body="&lt;p&gt;You already have ideas about potential questions. To get some more, you can try to test different types of visualizations, and check whether patterns appear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your inspiration, this &lt;a href=&quot;http://www.r2d3.us/visual-intro-to-machine-learning-part-1/&quot; rel=&quot;nofollow&quot;&gt;visual introduction to machine learning&lt;/a&gt; starts from a 7-dimension dataset of homes in San Francisco or in New York, and build different types of great graphics and predictions.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2016-03-23T20:55:47.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10860" PostTypeId="2" ParentId="10856" CreationDate="2016-03-23T20:57:26.877" Score="0" Body="&lt;p&gt;check out the &lt;a href=&quot;https://cran.r-project.org/web/packages/homals/homals.pdf&quot; rel=&quot;nofollow&quot;&gt;homals&lt;/a&gt; package in R, it might be a good starting point for you in terms of reducing the dimensionality of your dataset.  &lt;/p&gt;&#xA;" OwnerUserId="17256" LastActivityDate="2016-03-23T20:57:26.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10862" PostTypeId="2" ParentId="10841" CreationDate="2016-03-23T23:21:06.533" Score="1" Body="&lt;p&gt;Yes you can use a neural network, or any other regression-like algorithm for this task. There are lots of algorithms that you can use. A simple feed-forward network should suffice. Although, if you are not a machine learning expert (I'm assuming that you are not based on your question), I would suggest that you stick with something a little less exciting like OLS. It is easier to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you just need to spend some time doing a lot of feature engineering. So that you can feed not just raw variables but meaningful features to the model. For example the combination of features like size * complexity * material would give a lot of models to work with.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastEditorUserId="10024" LastEditDate="2016-05-24T21:02:09.130" LastActivityDate="2016-05-24T21:02:09.130" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10863" PostTypeId="2" ParentId="10844" CreationDate="2016-03-23T23:37:32.020" Score="8" Body="&lt;p&gt;The obvious answer would be the Netflix prize dataset, there is a lot of research into it and most CF algorithms have known scores in it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other available datasets that are usually used as benchmarks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://grouplens.org/datasets/movielens/&quot; rel=&quot;noreferrer&quot;&gt;Movie lens Dataset&lt;/a&gt;: a 20 million ratings dataset used for benchmarking CF algorithms;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://eigentaste.berkeley.edu/dataset/&quot; rel=&quot;noreferrer&quot;&gt;Jester Dataset&lt;/a&gt;: a joke recommendation dataset with more than 6 million ratings;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can find many more datasets in this &lt;a href=&quot;https://gist.github.com/entaroadun/1653794&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="11121" LastEditorUserId="11121" LastEditDate="2016-03-24T10:12:57.290" LastActivityDate="2016-03-24T10:12:57.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10864" PostTypeId="1" CreationDate="2016-03-24T00:58:53.053" Score="0" ViewCount="65" Body="&lt;p&gt;I have some data that I've noticed conforms to a sine wave. Details of the data is unknown. My task is to approximate it as closely as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From some experimentation in Excel, I noticed the data follows a sine wave, so attempted to plot in the base sine wave (orange) and am now trying to approximate the blue line. I believe this is a simple job of harmonics being added to the base sine wave, however; the quantity and their respective amplitude and harmonic number (2nd harmonic being 2 * f, 3rd harmonic being 3 * f etc) is what I now need to calculate. I'm also guessing the amplitude of the resulting wave is modified by possibly a sine wave too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there software which could automate this and approximate it through some analysis (Fourier analysis maybe) or possibly brute force, or is it something I have to write myself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dASAU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dASAU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2861" LastActivityDate="2016-03-24T01:05:53.173" Title="How to Ascertain Sine Wave and Harmonics" Tags="|data|parameter-estimation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10865" PostTypeId="2" ParentId="10864" CreationDate="2016-03-24T01:05:53.173" Score="1" Body="&lt;p&gt;You want to look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Spectral_density&quot; rel=&quot;nofollow&quot;&gt;spectral density&lt;/a&gt; to get an idea of the distribution of the frequency components. Check out the 'psd' package in r or the &lt;a href=&quot;https://www.mathworks.com/help/signal/ug/psd-estimate-using-fft.html&quot; rel=&quot;nofollow&quot;&gt;signal processing toolbox&lt;/a&gt; if you have access to Matlab.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-24T01:05:53.173" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10866" PostTypeId="2" ParentId="10822" CreationDate="2016-03-24T03:51:55.807" Score="3" Body="&lt;p&gt;A useful introductory reference is &lt;a href=&quot;https://www.otexts.org/fpp&quot; rel=&quot;nofollow&quot;&gt;Forecasting: Principles and Practice&lt;/a&gt; by Hyndman and Athanasopoulos. Depending on your data, you might be able to build a simple additive or multiplicative model with seasonal, trend, and cyclic components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides exponential smoothing, you can look at ARIMA models. Exponential smoothing is based on trend and seasonality while ARIMA is based on autocorrelations in your data.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-24T03:51:55.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10868" PostTypeId="1" CreationDate="2016-03-24T11:53:06.690" Score="1" ViewCount="370" Body="&lt;p&gt;I'm trying to get used to text mining using orange. I set up a very easy workflow loading a corpus (tab separated text file containing 10 entries), using a corpus viewer that shows my data correctly and a topic discovery item. But I always receive an error Message at 95% of topic discovery called:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Message: &quot;Error calling '%s' of '%s'&quot;&#xA;Arguments: ('set_data', 'Topic Discovery')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What do you think am I doing wrong here?&#xA;Thank you very much for your help in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stefan&lt;/p&gt;&#xA;" OwnerUserId="17293" LastEditorUserId="13413" LastEditDate="2016-04-24T13:08:46.667" LastActivityDate="2020-01-10T18:02:20.350" Title="Text Mining error" Tags="|text-mining|orange|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10869" PostTypeId="2" ParentId="1192" CreationDate="2016-03-24T11:57:40.420" Score="1" Body="&lt;p&gt;I don't know of an automated tool for this, but as a suggestion, you could look to using &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow&quot;&gt;feature selection&lt;/a&gt; or correlation of all variables in a particular dataset against all other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on the direction that you are wanting to take your analysis later, but the second method will enable you to see variables that are highly correlated, and so might be good candidates for checking that they are duplicates? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, not sure if you are aware, but there are a few R packages (&lt;a href=&quot;https://cran.r-project.org/web/packages/memisc/memisc.pdf&quot; rel=&quot;nofollow&quot;&gt;memisc&lt;/a&gt; is particularly popular) that enable you to read SPSS files directly and then work on the data directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry wont save you months of work though ... but might save you days ... &lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-03-24T11:57:40.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10870" PostTypeId="1" CreationDate="2016-03-24T15:25:20.030" Score="5" ViewCount="1677" Body="&lt;p&gt;Very new to Machine learning process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious as to how researchers, companies, academics get training data for their research. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do they pay some users to train data. If yes, how do they guarantee accuracy of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do graduate students (on PhD track) do it for their professor. (This is meant to be a joke)&lt;/p&gt;&#xA;" OwnerUserId="17302" LastActivityDate="2016-03-27T12:33:17.590" Title="How is training data generated" Tags="|machine-learning|supervised-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10871" PostTypeId="2" ParentId="10870" CreationDate="2016-03-24T15:45:59.883" Score="5" Body="&lt;p&gt;'Training' data is really just splitting data you have already collected into test or training sets. For example, if you want to build a classifier for handwritten numbers, you collect thousands of samples of handwritten numbers like the &lt;a href=&quot;https://en.wikipedia.org/wiki/MNIST_database&quot; rel=&quot;noreferrer&quot;&gt;MNIST database&lt;/a&gt;. When you think you have enough data to build a model, you then split it into train and test sets (usually by randomly assigning individual samples to one group or another at a specific ratio).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think where your confusion lies is in the idea of collecting a 'training' set first as if it's truly independent from the test set. When collecting handwritten numbers, the researchers did not say, well, we have 10,000 samples, let's build a model with 10,000 samples and then have it running for our future data sets that we have not collected yet - in fact, that strategy is particularly bad and can lead to &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;noreferrer&quot;&gt;overfitting&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you would do is take those 10,000 samples and split them - say 7,000 for training to build a model, 3,000 for testing said model - and maybe you would randomly build up many 7,000/3,000 models and take the average of the parameters for those models that you built. Then you can say our model predicts our test set with an accuracy of 97%, we think it will work well on data we have not yet collected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How you collect that initial data set is specific to the process you are trying to understand. Maybe it's clicks on a website, images from a satellite, or electrical recordings from an ensemble of neurons. Sometimes you pay money to collect data - like a census or survey or maybe even buying another company that collected a bunch of user data that you want - typically, data collection is an inherent process to what you are doing and you are using statistical methods to make models and inferences about your population of interest.&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-24T15:45:59.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10873" PostTypeId="2" ParentId="10868" CreationDate="2016-03-24T16:54:58.893" Score="1" Body="&lt;p&gt;One thing I've noticed about Orange is that anytime you get an error calling &lt;code&gt;set_data&lt;/code&gt;, there will be some sort of &lt;code&gt;ValueError&lt;/code&gt; or &lt;code&gt;FormatError&lt;/code&gt; that it is a child of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the rest of the errors and look for some type of value or formatting error, or post your full error log in a pastebin so I can take a look.&lt;/p&gt;&#xA;" OwnerUserId="3152" LastActivityDate="2016-03-24T16:54:58.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10875" PostTypeId="1" CreationDate="2016-03-24T17:06:34.277" Score="1" ViewCount="1326" Body="&lt;p&gt;I am working on 2 sample independent t-test. I have conducted analysis on test group vs control group and I have to write a report but I have few questions.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do we have to take out the outliers and then perform t-test?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once I perform t-test- can anybody explain the t-test output? The explanation should not be in terms of statistical terms but in such a way that non business person can also understand. I need simple explanation for confidence intervals and difference in means of the two samples.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What kind of charts can we draw to represent our results?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="29169" LastEditDate="2021-03-22T19:20:11.370" LastActivityDate="2021-03-22T19:20:11.370" Title="Can I conduct independent t-test when data is infested with outliers ? and how to interpret the t-statistics?" Tags="|statistics|statsmodels|inference|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10876" PostTypeId="1" CreationDate="2016-03-24T17:32:12.767" Score="7" ViewCount="6397" Body="&lt;p&gt;I've been working for a long time with an &lt;strong&gt;artificial neural network&lt;/strong&gt; algorithm (specifically, a &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/bromley-94.pdf&quot; rel=&quot;nofollow&quot;&gt;siamese neural network&lt;/a&gt;) that I implemented in Torch &amp;amp; Lua.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been studying and playing with many details of this algorithm (momentum alpha, number of minibatches, learning rate, gradient update iterations, dropout, 10-fold cross validation, etc), but I'm still facing the same old error: after training and during testing, my artificial neural network &lt;strong&gt;predicts almost every test element as positive&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I train my model with a training set, and then test it on a test set. Both sets contain 2,000 elements.&#xA;This is a typical &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;nofollow&quot;&gt;confusion matrix result&lt;/a&gt; I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;false negatives FN:       21&#xA;true positives TP:       179&#xA;&#xA;false positives FP:    1,747&#xA;true negatives TN:        53&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And these category values lead to the following rates:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f1_score = 0.16839          = 2*tp/(2*tp+fp+fn) [1: best] [0: worst]&#xA;accuracy = 0.116            = (tp+tn)/(tp+fn+fp+tn) [1: best] [0: worst]&#xA;recall = 0.9                = tp/(tp+fn) [1: best] [0: worst]&#xA;precision = 0.09            = tp/(fp+tp) [1: best] [0: worst]&#xA;specificity = 0.03          = tn/(fp+tn) [1: best] [0: worst]&#xA;fallout = 0.97              = fp/(fp+tn) [0: best] [1: worst]&#xA;false discovery rate = 0.91 = fp/(fp+tp) [0: best] [1: worst]&#xA;miss rate = 0.105           = fn/(fn+tp) [0: best] [1: worst]&#xA;MatthewsCC = -0.12008       = ((tp*tn)-(fp*fn))/math.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)) [+1: perfect correlation] [0: no better than random prediction] [-1: total disagreement]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can notice, the FN and TP results are very good, but the FP and TN results are very bad. &lt;strong&gt;False positives are too many&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;artificial neural network&lt;/strong&gt; thinks almost everything it processes is a positive element: it &lt;strong&gt;predicts the 96% of the elements as positives&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the weird thing is that the training set is &lt;strong&gt;artificially balanced towards negative values&lt;/strong&gt;: among the 2,000 elements, I'm choosing &lt;strong&gt;90% of negatives&lt;/strong&gt; and only &lt;strong&gt;10% of positives&lt;/strong&gt;.&#xA;I also am using the same proportion for the test set (90% of negatives and only 10% of positives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any idea about what is going on?&#xA;Why does my neural network predict so many positives?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: Thanks to all for the help. Here are some data and the main core of my code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some toy data (you can copy them into your Torch terminal):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;first_datasetTrain={};&#xA;&#xA;first_datasetTrain[1]=torch.Tensor{4, 5, 8, 10, 36, 0, 11, 22, 23, 44, 49, 35, 22, 6, 12, 16, 16, 4, 8, 10, 12, 14, 3, 30, 11, 12, 1, 3, 12, 16, 3, 4, 3, 6, 8, 0, 5, 12, 5, 4, 18, 10, 6, 8, 4, 7, 12, 7, 13, 3, 66, 61, 9, 51, 28, 7, 24, 8, 43, 40, 115, 2, 5, 11, 11, 41, 8, 2, 2, 2, 11, 6, 4, 4, 5, 6, 6, 9, 25, 16, 19, 13 };&#xA;&#xA;first_datasetTrain[2]=torch.Tensor{4, 5, 8, 6, 5, 0, 27, 8, 4, 8, 20, 11, 6, 7, 5, 22, 153, 3, 2, 6, 11, 28, 22, 37, 13, 5, 8, 17, 13, 8, 9, 0, 10, 14, 4, 30, 7, 17, 2, 5, 6, 9, 1, 18, 7, 3, 9, 2, 200, 7, 17, 16, 5, 5, 19, 7, 8, 8, 22, 11, 20, 0, 7, 3, 6, 7, 12, 7, 9, 9, 5, 5, 23, 2, 43, 13, 4, 10, 21, 9, 13, 15 };&#xA;&#xA;first_datasetTrain[3]=torch.Tensor{10, 4, 11, 16, 20, 0, 2, 10, 17, 10, 32, 30, 9, 11, 10, 11, 9, 8, 21, 21, 9, 16, 19, 13, 11, 16, 9, 12, 20, 14, 2, 9, 7, 13, 0, 17, 11, 26, 10, 11, 8, 2, 18, 16, 10, 10, 10, 7, 4, 11, 32, 20, 8, 19, 21, 3, 7, 26, 17, 19, 139, 5, 10, 11, 15, 13, 2, 3, 2, 4, 24, 11, 11, 1, 8, 11, 3, 7, 20, 29, 24, 13 };&#xA;&#xA;first_datasetTrain[4]=torch.Tensor{3, 6, 6, 10, 24, 0, 22, 3, 16, 7, 25, 13, 11, 24, 20, 14, 9, 7, 9, 10, 9, 15, 7, 49, 2, 13, 12, 4, 21, 7, 22, 14, 4, 12, 14, 13, 4, 12, 8, 8, 88, 88, 105, 87, 9, 35, 12, 16, 17, 18, 26, 12, 9, 23, 200, 13, 25, 12, 29, 28, 200, 10, 4, 17, 16, 10, 18, 3, 5, 2, 26, 11, 14, 3, 30, 4, 4, 0, 27, 25, 24, 18 };&#xA;&#xA;first_datasetTrain[5]=torch.Tensor{14, 18, 17, 14, 16, 0, 19, 10, 14, 6, 15, 21, 15, 5, 14, 22, 7, 14, 18, 107, 13, 18, 19, 22, 25, 11, 32, 46, 14, 26, 8, 20, 29, 64, 24, 14, 25, 20, 42, 7, 18, 12, 14, 32, 12, 20, 19, 17, 35, 14, 19, 12, 8, 18, 32, 13, 23, 35, 24, 14, 32, 9, 34, 39, 6, 10, 51, 19, 8, 23, 39, 13, 200, 6, 32, 21, 18, 3, 32, 21, 133, 38 };&#xA;&#xA;first_datasetTrain[6]=torch.Tensor{8, 5, 10, 9, 15, 0, 199, 23, 21, 15, 21, 17, 13, 16, 11, 34, 89, 7, 8, 16, 7, 19, 41, 61, 22, 28, 4, 44, 18, 17, 10, 9, 31, 16, 5, 23, 10, 11, 23, 6, 7, 5, 6, 6, 11, 3, 12, 16, 200, 17, 30, 10, 95, 32, 22, 6, 11, 41, 33, 24, 19, 11, 10, 13, 12, 21, 11, 1, 6, 10, 15, 6, 22, 3, 13, 29, 14, 2, 111, 24, 27, 15 };&#xA;&#xA;first_datasetTrain[7]=torch.Tensor{8, 15, 46, 200, 200, 0, 200, 200, 200, 200, 92, 200, 200, 90, 42, 38, 76, 55, 200, 75, 16, 91, 86, 148, 200, 200, 5, 19, 22, 164, 23, 57, 172, 57, 3, 31, 8, 17, 46, 78, 11, 14, 21, 21, 12, 25, 11, 17, 86, 8, 200, 200, 200, 200, 24, 14, 15, 24, 200, 173, 200, 7, 46, 57, 25, 200, 16, 7, 9, 11, 100, 22, 46, 6, 95, 200, 9, 0, 110, 27, 30, 30 };&#xA;&#xA;first_datasetTrain[8]=torch.Tensor{9, 9, 10, 34, 50, 0, 6, 27, 20, 29, 23, 21, 9, 19, 10, 16, 10, 6, 14, 16, 9, 20, 17, 33, 89, 78, 9, 8, 5, 10, 5, 5, 4, 8, 16, 8, 14, 13, 5, 3, 10, 17, 12, 15, 9, 3, 9, 16, 8, 7, 13, 14, 6, 21, 19, 13, 20, 19, 22, 22, 20, 7, 4, 7, 6, 28, 21, 3, 12, 4, 22, 6, 11, 3, 15, 20, 4, 2, 12, 7, 25, 10 };&#xA;&#xA;first_datasetTrain[9]=torch.Tensor{5, 7, 18, 77, 29, 0, 20, 21, 35, 53, 128, 42, 28, 104, 10, 23, 13, 11, 8, 12, 19, 26, 18, 33, 21, 19, 13, 11, 28, 87, 10, 10, 200, 35, 5, 11, 7, 13, 20, 53, 15, 7, 14, 14, 7, 13, 12, 9, 18, 10, 121, 116, 83, 72, 19, 14, 12, 8, 40, 39, 200, 12, 21, 19, 20, 25, 22, 9, 4, 6, 26, 2, 102, 2, 76, 12, 51, 3, 23, 15, 18, 29 };&#xA;&#xA;first_datasetTrain[10]=torch.Tensor{4, 14, 10, 10, 12, 0, 17, 7, 17, 17, 26, 21, 6, 12, 40, 22, 12, 1, 10, 20, 6, 24, 33, 38, 8, 22, 16, 9, 12, 9, 11, 3, 5, 22, 12, 24, 9, 22, 16, 5, 17, 9, 19, 22, 9, 7, 7, 14, 7, 9, 51, 17, 84, 48, 13, 2, 11, 45, 33, 55, 88, 5, 8, 15, 5, 9, 9, 10, 9, 6, 10, 6, 7, 4, 15, 7, 6, 6, 12, 26, 36, 13 };&#xA;&#xA;second_datasetTrain={};&#xA;&#xA;second_datasetTrain[1]=torch.Tensor{18, 16, 29, 7, 16, 0, 11, 11, 15, 11, 45, 15, 10, 9, 17, 23, 132, 43, 27, 24, 40, 22, 42, 31, 9, 9, 110, 53, 42, 90, 3, 40, 174, 23, 41, 22, 8, 30, 200, 13, 13, 11, 11, 8, 8, 19, 90, 13, 200, 9, 29, 13, 3, 30, 25, 10, 200, 17, 31, 9, 25, 14, 28, 10, 20, 9, 34, 6, 15, 30, 8, 3, 81, 44, 23, 12, 185, 3, 11, 15, 32, 19 };&#xA;&#xA;second_datasetTrain[2]=torch.Tensor{2, 6, 9, 12, 70, 0, 38, 23, 52, 54, 83, 60, 50, 129, 36, 12, 15, 17, 23, 13, 5, 45, 16, 98, 97, 13, 3, 7, 11, 26, 7, 2, 7, 13, 9, 3, 4, 9, 3, 6, 6, 7, 10, 13, 6, 5, 8, 10, 7, 11, 96, 57, 65, 177, 35, 5, 11, 17, 48, 179, 100, 7, 7, 12, 9, 21, 11, 5, 10, 6, 16, 8, 12, 2, 9, 7, 4, 3, 69, 13, 11, 7 };&#xA;&#xA;second_datasetTrain[3]=torch.Tensor{3, 6, 6, 10, 24, 0, 22, 3, 16, 7, 25, 13, 11, 24, 20, 14, 9, 7, 9, 10, 9, 15, 7, 49, 2, 13, 12, 4, 21, 7, 22, 14, 4, 12, 14, 13, 4, 12, 8, 8, 88, 88, 105, 87, 9, 35, 12, 16, 17, 18, 26, 12, 9, 23, 200, 13, 25, 12, 29, 28, 200, 10, 4, 17, 16, 10, 18, 3, 5, 2, 26, 11, 14, 3, 30, 4, 4, 0, 27, 25, 24, 18 };&#xA;&#xA;second_datasetTrain[4]=torch.Tensor{13, 6, 34, 155, 69, 0, 34, 44, 28, 57, 41, 45, 27, 4, 28, 29, 20, 12, 52, 28, 5, 18, 27, 29, 21, 31, 4, 7, 13, 107, 14, 16, 17, 13, 7, 23, 17, 37, 13, 29, 10, 19, 14, 13, 8, 26, 3, 10, 6, 11, 77, 85, 31, 90, 23, 27, 9, 28, 46, 34, 200, 20, 11, 23, 15, 200, 0, 4, 29, 4, 42, 3, 14, 2, 7, 15, 42, 5, 49, 12, 12, 17 };&#xA;&#xA;second_datasetTrain[5]=torch.Tensor{2, 11, 19, 27, 23, 0, 16, 11, 18, 13, 25, 18, 10, 14, 15, 40, 1, 9, 12, 21, 17, 20, 22, 25, 28, 19, 12, 25, 8, 18, 3, 15, 11, 24, 9, 16, 17, 21, 23, 9, 12, 13, 14, 25, 19, 21, 11, 8, 11, 13, 18, 10, 21, 24, 26, 5, 20, 33, 57, 25, 16, 8, 26, 15, 5, 9, 13, 9, 7, 13, 16, 11, 9, 4, 9, 21, 5, 8, 12, 22, 33, 10 };&#xA;&#xA;second_datasetTrain[6]=torch.Tensor{78, 13, 200, 200, 200, 0, 70, 200, 200, 200, 200, 200, 200, 18, 21, 27, 11, 12, 20, 58, 28, 18, 22, 119, 200, 200, 65, 54, 178, 200, 88, 95, 200, 200, 24, 47, 30, 26, 200, 109, 76, 85, 50, 65, 21, 200, 4, 36, 110, 30, 200, 200, 200, 200, 200, 101, 23, 23, 200, 200, 200, 19, 123, 36, 200, 86, 69, 6, 7, 76, 38, 21, 200, 1, 200, 44, 59, 6, 142, 30, 53, 200 };&#xA;&#xA;second_datasetTrain[7]=torch.Tensor{10, 5, 7, 12, 15, 0, 35, 18, 11, 11, 17, 14, 4, 9, 47, 77, 28, 33, 94, 61, 7, 37, 35, 40, 4, 21, 7, 17, 10, 25, 11, 15, 10, 20, 6, 59, 18, 16, 9, 26, 6, 10, 25, 23, 95, 13, 1, 14, 13, 11, 22, 5, 14, 20, 23, 11, 25, 33, 22, 30, 64, 7, 7, 27, 10, 14, 4, 7, 6, 4, 18, 15, 10, 4, 23, 71, 5, 3, 81, 41, 33, 13 };&#xA;&#xA;second_datasetTrain[8]=torch.Tensor{6, 10, 14, 81, 200, 0, 39, 141, 200, 200, 200, 200, 200, 10, 4, 23, 16, 11, 9, 37, 8, 22, 21, 74, 200, 195, 6, 15, 16, 30, 8, 5, 19, 19, 11, 71, 7, 12, 29, 6, 11, 14, 7, 8, 7, 17, 3, 12, 14, 7, 200, 200, 200, 200, 30, 5, 17, 24, 200, 155, 200, 4, 19, 25, 26, 39, 6, 11, 4, 7, 33, 9, 30, 1, 27, 10, 9, 16, 37, 8, 30, 19 };&#xA;&#xA;second_datasetTrain[9]=torch.Tensor{15, 2, 11, 160, 11, 0, 7, 9, 11, 33, 30, 14, 14, 12, 16, 18, 33, 16, 38, 12, 8, 16, 26, 21, 4, 16, 6, 11, 15, 6, 2, 4, 4, 14, 4, 12, 6, 8, 12, 9, 16, 5, 17, 13, 13, 11, 10, 3, 8, 3, 10, 8, 4, 34, 21, 6, 17, 27, 27, 25, 58, 7, 19, 10, 12, 12, 20, 4, 4, 6, 36, 5, 14, 4, 15, 12, 4, 3, 41, 11, 18, 11 };&#xA;&#xA;second_datasetTrain[10]=torch.Tensor{20, 1, 27, 187, 161, 0, 200, 95, 200, 200, 200, 200, 200, 8, 51, 34, 27, 33, 50, 41, 3, 34, 49, 200, 190, 146, 5, 15, 6, 108, 30, 67, 72, 13, 10, 11, 20, 20, 14, 11, 55, 44, 56, 43, 88, 52, 7, 15, 3, 9, 97, 145, 138, 200, 200, 5, 14, 54, 110, 190, 200, 6, 24, 18, 9, 132, 8, 3, 12, 4, 50, 9, 17, 2, 16, 6, 5, 5, 43, 55, 31, 22 };&#xA;&#xA;targetDatasetTrain={};&#xA;&#xA;targetDatasetTrain[1]={0};&#xA;&#xA;targetDatasetTrain[2]={1};&#xA;&#xA;targetDatasetTrain[3]={0};&#xA;&#xA;targetDatasetTrain[4]={1};&#xA;&#xA;targetDatasetTrain[5]={0};&#xA;&#xA;targetDatasetTrain[6]={1};&#xA;&#xA;targetDatasetTrain[7]={0};&#xA;&#xA;targetDatasetTrain[8]={1};&#xA;&#xA;targetDatasetTrain[9]={0};&#xA;&#xA;targetDatasetTrain[10]={1};&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here's the short version of my Code that implements a siamese neural network having two parallel neural networks (upper and lower) that process first_datasetTrain and second_datasetTrain, and then compares their hidden representation through the Cosine distance (Inspiration from: &lt;a href=&quot;https://github.com/torch/nn/blob/master/doc/table.md#nn.CosineDistance&quot; rel=&quot;nofollow&quot;&gt;https://github.com/torch/nn/blob/master/doc/table.md#nn.CosineDistance&lt;/a&gt; )&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require &quot;nn&quot;;&#xA;&#xA;-- Gradient update for the siamese neural network&#xA;function gradientUpdate(perceptron, dataset_vector, targetValue, learningRate, i, ite);&#xA;&#xA;   function dataset_vector:size() return #dataset_vector end&#xA;&#xA;   local predictionValue = perceptron:forward(dataset_vector)[1];&#xA;&#xA;   local plusChar = &quot;&quot;&#xA;   if targetValue == 1 then plusChar = &quot;+&quot;; end&#xA;&#xA;   local meanSquareError = math.pow(targetValue - predictionValue,2);&#xA;   io.write(&quot;(ite=&quot;..ite..&quot;) (ele=&quot;..i..&quot;) pred = &quot;..predictionValue..&quot; targetValue = &quot;..plusChar..&quot;&quot;..targetValue ..&quot; =&amp;gt; meanSquareError = &quot;..meanSquareError);&#xA;    io.flush();&#xA;&#xA;   if meanSquareError &amp;gt; 1 then &#xA;       io.write(&quot; LARGE MeanSquareError&quot;); &#xA;       io.flush();&#xA;       sys.sleep(0.1);&#xA;   end&#xA;   io.write(&quot;\n&quot;);&#xA;&#xA;&#xA;&#xA;    if predictionValue*targetValue &amp;lt; 1 then&#xA;          gradientWrtOutput = torch.Tensor({-targetValue});&#xA;      perceptron:zeroGradParameters();&#xA;      perceptron:backward(dataset_vector, gradientWrtOutput);&#xA;      perceptron:updateParameters(learningRate);&#xA;    end&#xA;&#xA;  return perceptron;&#xA;end&#xA;&#xA;&#xA;local dropOutFlag = true&#xA;local hiddenUnits = 4&#xA;local hiddenLayers = 4&#xA;&#xA;&#xA;&#xA;print('&amp;lt;siameseNeuralNetworkApplication_justTraining start&amp;gt;');&#xA;io.write(&quot;#first_datasetTrain = &quot;.. (#first_datasetTrain));&#xA;io.write(&quot; #second_datasetTrain = &quot;..(#second_datasetTrain));&#xA;io.write(&quot; #targetDatasetTrain = &quot;..(#targetDatasetTrain)..&quot;\n&quot;);&#xA;io.write(&quot; dropOutFlag = &quot;..tostring(dropOutFlag));&#xA;io.write(&quot; hiddenUnits = &quot;..hiddenUnits);&#xA;io.write(&quot; hiddenLayers = &quot;..hiddenLayers);&#xA;&#xA;&#xA;local input_number = (#(first_datasetTrain[1]))[1]; -- they are 6&#xA;local output_layer_number = input_number&#xA;&#xA;local trainDataset = {}&#xA;local targetDataset = {}&#xA;&#xA;print(&quot;Creatin\' the siamese neural network...&quot;);&#xA;print('hiddenUnits='..hiddenUnits..'\thiddenLayers='..hiddenLayers);&#xA;&#xA;-- imagine we have one network we are interested in, it is called &quot;perceptronUpper&quot;&#xA;local perceptronUpper= nn.Sequential()&#xA;&#xA;perceptronUpper:add(nn.Linear(input_number, hiddenUnits))&#xA;perceptronUpper:add(nn.Tanh())&#xA;--perceptronUpper:add(nn.ReLU())&#xA;if dropOutFlag==TRUE then perceptronUpper:add(nn.Dropout()) end&#xA;&#xA;for w=1, hiddenLayers do&#xA; perceptronUpper:add(nn.Linear(hiddenUnits,hiddenUnits))&#xA; perceptronUpper:add(nn.Tanh())&#xA; --perceptronUpper:add(nn.ReLU())&#xA; if dropOutFlag==TRUE then perceptronUpper:add(nn.Dropout()) end&#xA;end&#xA;&#xA;perceptronUpper:add(nn.Linear(hiddenUnits,output_layer_number))&#xA;perceptronUpper:add(nn.Tanh())&#xA;--perceptronUpper:add(nn.ReLU())&#xA;&#xA;&#xA;local perceptronLower = perceptronUpper:clone('weight', 'gradWeight') &#xA;&#xA;-- we make a parallel table that takes a pair of examples as input. they both go through the same (cloned) perceptron&#xA;-- ParallelTable is a container module that, in its forward() method, applies the i-th member module to the i-th input, and outputs a table of the set of outputs.&#xA;local parallel_table = nn.ParallelTable()&#xA;parallel_table:add(perceptronUpper)&#xA;parallel_table:add(perceptronLower)&#xA;&#xA;&#xA;&#xA;-- now we define our top level network that takes this parallel table and computes the cosine distance betweem&#xA;-- the pair of outputs&#xA;local generalPerceptron= nn.Sequential()&#xA;generalPerceptron:add(parallel_table)&#xA;generalPerceptron:add(nn.CosineDistance())&#xA;&#xA;&#xA;&#xA;MAX_ITERATIONS_CONST = 1000&#xA;LEARNING_RATE_CONST = 0.01&#xA;local max_iterations = MAX_ITERATIONS_CONST;&#xA;local learnRate = LEARNING_RATE_CONST;&#xA;&#xA;&#xA;for ite = 1, max_iterations do   &#xA;     for i=1, #first_datasetTrain do  &#xA;&#xA;&#xA;     trainDataset[i]={first_datasetTrain[i], second_datasetTrain[i]}&#xA;     collectgarbage();  &#xA;&#xA;     local currentTarget = 1&#xA;     if tonumber(targetDatasetTrain[i][1]) == 0 &#xA;    then currentTarget = -1; &#xA;     end&#xA;&#xA;     generalPerceptron = gradientUpdate(generalPerceptron, trainDataset[i], currentTarget, learnRate, i, ite);    &#xA;&#xA;     local predicted = generalPerceptron:forward(trainDataset[i])[1];  &#xA;     print(&quot;predicted = &quot;..predicted);&#xA;     end&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You just have to copy this code into a file say &lt;code&gt;siamese.lua&lt;/code&gt;, then open a Torch terminal, copy and paste the data file into the terminal, run &lt;code&gt;dofile(&quot;siamese.lua&quot;)&lt;/code&gt;, and everything should go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data have just 10 elements, but if you need more &lt;a href=&quot;http://bit.ly/1Y4ANk6&quot; rel=&quot;nofollow&quot;&gt;you can download this file&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be very appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="17305" LastEditorUserId="17305" LastEditDate="2016-04-01T21:47:30.027" LastActivityDate="2016-04-04T11:55:49.413" Title="Why is my artificial neural networks almost always predicting positive elements?" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="10877" PostTypeId="1" AcceptedAnswerId="10903" CreationDate="2016-03-24T18:26:46.147" Score="1" ViewCount="78" Body="&lt;p&gt;What would one use for manipulating data of the kind below ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Data is bio-markers of different globs&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;GlobA 3 4 5 .... &lt;/li&gt;&#xA;&lt;li&gt;GlobB 2 1 1 .... &lt;/li&gt;&#xA;&lt;li&gt;GlobC 3 2 1 ....&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;b) Manipulations are queries like:&lt;br&gt;&#xA;- show me the Globs where average of each Glob in file Efficiency is greater than 50%&lt;br&gt;&#xA;- or show me sorted Glob list where first and second sort criterion are xx and yy)&lt;br&gt;&#xA;- Construct a chart of globs that differ in criteria x by integer 3 (show me globs whose average over 5 runs for Structure and Efficiency differ from their nearest neighbors by 3)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, this data is stored in a 100MB Excel file that is painfully slow to load on the speediest computer our lab can afford.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, there would be some open source program that accepts csv files of this data and has ability for user to construct queries that can be stored in a library for easy pulling up, charting abilities would be great too.&lt;br&gt;&#xA;Here are 2 files (real data would be around 40 such files each file containing 20K rows):  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Efficiency File:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Glob,Run1,Run2,Run3,Run4,Run5   &#xA;SigX,6.2,4.8,2.4,4.32,5.59  &#xA;SigY,8.44,8.16,5.99,0.98,9.6   &#xA;SigZ,0.00,0.00,0.00,0.01,0.20     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Structure File:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Glob,Run1,Run2,Run3,Run4,Run5   &#xA;SigX,3.2,3.8,2.4,7.32,6.32  &#xA;SigY,2.4,5.16,6.99,0.98,9.6   &#xA;SigZ,1.02,0.00,2.23,0.01,0.20     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17308" LastEditorUserId="381" LastEditDate="2016-03-25T17:24:20.927" LastActivityDate="2016-03-26T00:18:12.877" Title="Creating queries on the fly and general manipulation for dataset of half a million data records" Tags="|visualization|data|csv|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10879" PostTypeId="2" ParentId="10870" CreationDate="2016-03-24T18:44:12.113" Score="4" Body="&lt;p&gt;Data typically exists. What typically does not exist, is ground truth (in the case of classification). Such ground truth is typically always collected manually and crowd sourcing plays an important role. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, think about Face recognition that Facebook does. Before automatic tagging was available, Facebook allowed users to manually add tags to create a set of labeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more general way of doing this is through Amazon's Mechanical Turk (Amazon's marketplace). See the &lt;a href=&quot;https://www.mturk.com/mturk/findhits?match=false&amp;amp;state=SDlIZ3RRRUwxVm5sNlV6U0kza1BHWUV0Q0hJPTIwMTYwMzI0MTE1NVVzZXIudHVya1NlY3VyZX50cnVlJQ--&quot; rel=&quot;nofollow&quot;&gt;tasks&lt;/a&gt; listed there. Some of them are clearly related to manual generation of labels that will later form the basis of a learning system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most research in academia is creating methods and you can demonstrate how well it works on existing datasets. However, when a new company is launching, for example, a fraud detection platform, they have to deal with manually labeling transactions as fraudulent or not. Sometimes, this is done when a report comes in from the customer and sometimes by manual eyeballing by analysts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you would imagine, there has been a lot of academic interest in understanding the &lt;a href=&quot;https://www.ischool.utexas.edu/~ml/papers/lease-hcomp11.pdf&quot; rel=&quot;nofollow&quot;&gt;quality of results obtained through crowd sourcing&lt;/a&gt; and continues to be an active area of research. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2016-03-25T00:36:45.193" LastActivityDate="2016-03-25T00:36:45.193" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10880" PostTypeId="1" AcceptedAnswerId="10901" CreationDate="2016-03-24T19:22:53.283" Score="3" ViewCount="314" Body="&lt;p&gt;I'm very new to data science in general, and have been tasked with a big challenge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My organization has a lot of documents that are all sorted on document type (not binary format, but a subjectively assigned type based on content, e.g. &quot;Contract&quot;, &quot;Receipt&quot;, &quot;Statement&quot;, etc...).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking assignment of these types is done upon receipt of the documents, and is not a challenge, though we would like to remove the human element of this categorization.  Similarly, there are times when there are special attributes that we would like to identify, like &quot;Statement showing use.&quot;  Thus far, this is entirely done by human intervention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a python programmer, and have been looking at tools to extract the text from these docs (all PDFs, all OCR'ed and searchable) and run analysis.  Research has led me to look at standard libraries like NLTK, scikit-learn and gensim.  But I'm struggling to identify what would be the best methodology for categorizing newly received documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My research is leading me down a few paths...one is creating a Tf-iDf vector model based on a sampling of current corpa and then creating a model for an incoming document's corpus and doing a naive bayes analysis against existing models to discern which category the incoming document belongs to based on highest probability.  Question 1: is this right?  If so question 2 becomes what is the right programmatic methodology for accomplishing this?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I bring this up at all is because most tutorials I find seem to lean toward a binary discernment of text corpa (positive vs negative, spam vs ham).  I did see scikit-learn has information on multi-label classification, but I'm not sure I'm going down the right road with it.  The word &quot;classification&quot; seems to have different meaning in document analysis than what I would want it to mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this question is too vague let me know and I can edit it to be more specific.&lt;/p&gt;&#xA;" OwnerUserId="17309" LastActivityDate="2016-03-25T19:29:23.097" Title="Document Categorization Problem" Tags="|python|scikit-learn|nltk|gensim|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10881" PostTypeId="2" ParentId="10803" CreationDate="2016-03-24T19:42:33.260" Score="2" Body="&lt;p&gt;&lt;em&gt;Turian, Joseph, Lev Ratinov, and Yoshua Bengio. &quot;&lt;a href=&quot;https://scholar.google.com/scholar?cluster=7872411552705927180&amp;amp;hl=en&amp;amp;as_sdt=0,5&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word representations: a simple and general method for semi-supervised learning&lt;/a&gt;.&quot; Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics, 2010.&lt;/em&gt; define distributional representations and distributed representations as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;A &lt;strong&gt;distributional word representation&lt;/strong&gt; is based upon a cooccurrence matrix $F$ of size $W×C$, where $W$ is the vocabulary size,&#xA;  each row $F_w$ is the initial representation of word $w$, and each column&#xA;  $F_c$ is some context. Sahlgren (2006) and Turney and Pantel (2010)&#xA;  describe a handful of possible design decisions in contructing $F$,&#xA;  including choice of context types (left window? right window? size of&#xA;  window?) and type of frequency count (raw? binary? tf-idf?). $F_w$ has&#xA;  dimensionality $W$, which can be too large to use $F_w$ as features for&#xA;  word w in a supervised model. One can map $F$ to matrix f of size W × d,&#xA;  where $d &amp;lt;&amp;lt;  C$, using some function g, where f = g(F). $F_w$ represents word&#xA;  $w$ as a vector with $d$ dimensions. The choice of $g$ is another design&#xA;  decision, although perhaps not as important as the statistics used to&#xA;  initially construct $F$.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;A &lt;strong&gt;distributed representation&lt;/strong&gt; is dense, low-dimensional, and real-valued. Distributed word representations are called word&#xA;  embeddings. Each dimension of the embedding represents a latent&#xA;  feature of the word, hopefully capturing useful syntactic and semantic&#xA;  properties. A distributed representation is compact, in the sense that&#xA;  it can represent an exponential number of clusters in the number of&#xA;  dimensions.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;FYI: &lt;a href=&quot;https://linguistics.stackexchange.com/q/8331/2680&quot;&gt;What's the difference between word vectors, word representations and vector embeddings?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:54:11.560" LastActivityDate="2016-03-24T19:42:33.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10882" PostTypeId="2" ParentId="10875" CreationDate="2016-03-24T20:37:32.393" Score="0" Body="&lt;p&gt;1) Maybe, remember that you are assuming a normal distributions, if you don't satisfy those assumptions you are not running a valid test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)You are testing whether or not the difference is zero, i.e. no difference=zero in my confidence interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3)Bar charts are the easiest to understand because you can see the difference. Box-plots provide more info but are for technical people only. &lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-03-24T20:37:32.393" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10883" PostTypeId="2" ParentId="10833" CreationDate="2016-03-24T23:19:52.510" Score="1" Body="&lt;p&gt;BMA is &quot;Bayesian Model Averaged&quot;. GNET is &quot;Generalized Elastic Net&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you tried reading the Ishwaran and Rao papers as mentioned in the documentation for &lt;code&gt;spikeslab&lt;/code&gt;?  There's an article in the R Journal as well that might be worth reading too: &lt;a href=&quot;https://journal.r-project.org/archive/2010-2/RJournal_2010-2_Ishwaran~et~al.pdf&quot; rel=&quot;nofollow&quot;&gt;https://journal.r-project.org/archive/2010-2/RJournal_2010-2_Ishwaran~et~al.pdf&lt;/a&gt; - no sense duplicating it all here.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-03-24T23:19:52.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10884" PostTypeId="1" AcceptedAnswerId="10889" CreationDate="2016-03-25T05:44:52.280" Score="0" ViewCount="262" Body="&lt;p&gt;I want to know how to start from scratch for Machine Learning. Also which language is best for implementing its algorithms or developing future applications based on it. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17315" LastActivityDate="2021-02-08T17:16:05.867" Title="From where should I start Machine Learning?" Tags="|machine-learning|programming|" AnswerCount="2" CommentCount="0" ClosedDate="2016-03-25T18:08:41.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="10885" PostTypeId="1" CreationDate="2016-03-25T06:04:10.987" Score="8" ViewCount="5473" Body="&lt;p&gt;I am a newbie when it comes to machine learning. I am trying to get hands on experience by analyzing different supervised learning algorithms using scikit-learn library of python. I am using the sentiment140 dataset of 1.6 million tweets for sentiment analysis using various of these algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if it is a stupid question, but I was wondering whether if it'd be possible to classify into three classes (positive, negative and neutral) when you've only trained over two classes (positive and negative). The &lt;a href=&quot;http://help.sentiment140.com/for-students&quot; rel=&quot;nofollow&quot;&gt;sentiment140 training set&lt;/a&gt; consists of only two classes (positive and negative) of 1.6M tweets in total but their test set consists of 500 tweets over three classes (positive, negative and neutral), so it got me thinking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is this possible? If yes, how do I proceed to identify the neutral tweets?&#xA;Intuitively, I can use to find the conditional probability of each classified tweet in the test set over each class (using predict_proba) and tell if it's neutral if it is below a certain threshold (say less than 0.7) for both the positive and negative classes. Is this the right way to go?&lt;/p&gt;&#xA;" OwnerUserId="17313" LastEditorUserId="31513" LastEditDate="2017-05-10T03:59:02.530" LastActivityDate="2017-07-21T13:46:54.233" Title="Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes" Tags="|machine-learning|nlp|sentiment-analysis|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10886" PostTypeId="2" ParentId="10884" CreationDate="2016-03-25T06:10:47.793" Score="2" Body="&lt;p&gt;You can start learning machine learning by attending the lectures of Andrew Ng at Coursera. This course helps you to develop basics as well as advanced concepts in machine learning.&#xA;Following is the link to the course:&#xA;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning by Andrew Ng&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Start learning Python for implementing algorithms related to machine learning.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2021-02-08T17:16:05.867" LastActivityDate="2021-02-08T17:16:05.867" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10888" PostTypeId="2" ParentId="548" CreationDate="2016-03-25T07:49:38.273" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;would LOVE to see a box written which collected case studies on feature engineering / extraction&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please advise if this helps &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Discretization of Time Series Data &#xA;&lt;a href=&quot;http://arxiv.org/ftp/q-bio/papers/0505/0505028.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/ftp/q-bio/papers/0505/0505028.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Optimizing Time Series Discretization for Knowledge Discovery&#xA;&lt;a href=&quot;https://www.uni-marburg.de/fb12/datenbionik/pdf/pubs/2005/moerchen05optimizing&quot; rel=&quot;noreferrer&quot;&gt;https://www.uni-marburg.de/fb12/datenbionik/pdf/pubs/2005/moerchen05optimizing&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Experiencing SAX: a Novel Symbolic Representation of Time Series&#xA;&lt;a href=&quot;http://cs.gmu.edu/~jessica/SAX_DAMI_preprint.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://cs.gmu.edu/~jessica/SAX_DAMI_preprint.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Indexing for Interactive Exploration of Big Data Series&#xA;&lt;a href=&quot;http://acs.ict.ac.cn/storage/slides/Indexing_for_Interactive_Exploration_of_Big_Data_Series.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://acs.ict.ac.cn/storage/slides/Indexing_for_Interactive_Exploration_of_Big_Data_Series.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Generalized Feature Extraction for Structural Pattern Recognition in&#xA;Time-series Data&#xA;&lt;a href=&quot;http://www.semanticscholar.org/paper/Generalized-Feature-Extraction-for-Structural-Olszewski-Maxion/7838bcd87bb6616e9fd3ffd92d4676a7082da34c&quot; rel=&quot;noreferrer&quot;&gt;http://www.semanticscholar.org/paper/Generalized-Feature-Extraction-for-Structural-Olszewski-Maxion/7838bcd87bb6616e9fd3ffd92d4676a7082da34c&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package&#xA;&lt;a href=&quot;https://cran.r-project.org/web/packages/dtw/vignettes/dtw.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://cran.r-project.org/web/packages/dtw/vignettes/dtw.pdf&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14542" LastEditorUserId="14542" LastEditDate="2016-03-26T14:34:31.200" LastActivityDate="2016-03-26T14:34:31.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10889" PostTypeId="2" ParentId="10884" CreationDate="2016-03-25T09:10:13.373" Score="1" Body="&lt;p&gt;I am a book person so I would recommend one of the following books:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Elements of Statistical Learning (Hastie and Tibshirani). &lt;/li&gt;&#xA;&lt;li&gt;Pattern Recognition and Machine Learning (Bishop). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first book is available as a free download from the authors' website. You may download and start reading it. You will get an idea about your deficiencies. If it's too difficult, then you need to improve your statistics and linear algebra skills. For Linear Algebra I recommend:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Linear Algebra and its Applications (David Lay). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For statistics I like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Discovering Statistics (Andy Fields).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Stay away from the recipe books if your aims are long-term. &lt;/p&gt;&#xA;" OwnerUserId="16774" LastActivityDate="2016-03-25T09:10:13.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10890" PostTypeId="2" ParentId="10636" CreationDate="2016-03-25T10:43:18.997" Score="0" Body="&lt;p&gt;This happens with me when I'm trying to train with no events for a given event specified in &lt;code&gt;engine.json&lt;/code&gt; .&#xA;Check your &lt;code&gt;engine.json&lt;/code&gt; to find which &lt;em&gt;eventNames&lt;/em&gt; you're expecting and make sure that you have imported at least one event for each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're following the template tutorial, you're importing your data with : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;python examples/import_handmade.py&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure to run this command before training your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some commands I use to debug issues with pio that could be helpful : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;pio export --appid {your_app_id} --output ./data.out&lt;/code&gt; Dumps all app events in JSON to data.out directory&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pio import --appid {your_app_id} --input ./data/test_data.json&lt;/code&gt; Imports events from test_data in JSON format&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pio status&lt;/code&gt; Lets you know if everything is running&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pio app data-delete {your_app_name}&lt;/code&gt; Deletes all data stored by app (Helpful if you want to debug a subset of data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In your case try exporting the data to make sure there's nothing missing then delete the data, import the new set, and train.&lt;/p&gt;&#xA;" OwnerUserId="17325" LastEditorUserId="17325" LastEditDate="2016-03-26T15:31:16.570" LastActivityDate="2016-03-26T15:31:16.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10891" PostTypeId="1" AcceptedAnswerId="10909" CreationDate="2016-03-25T11:33:05.390" Score="7" ViewCount="4000" Body="&lt;p&gt;I have salary data of several user (Python list).&lt;/p&gt;&#xA;&lt;p&gt;Now I am using KMeans to cluster them.&lt;/p&gt;&#xA;&lt;p&gt;Given this data, Is there a way to figure out the best value for 'K' automatically through program?&lt;/p&gt;&#xA;&lt;p&gt;I tried &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;silhouette analysis&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;take &lt;code&gt;k&lt;/code&gt; with highest &lt;code&gt;silhouette_avg&lt;/code&gt; score. Is this the best way?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="98307" LastEditDate="2020-08-04T01:31:12.577" LastActivityDate="2020-08-04T17:40:54.270" Title="Determinate K in K-Means Clustering" Tags="|python|clustering|k-means|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10892" PostTypeId="2" ParentId="10885" CreationDate="2016-03-25T13:42:51.997" Score="0" Body="&lt;p&gt;Say the results of one sample is $P(positive | sample) = 0.15$ and $P(negative | sample) = .05$, as opposed to just taking the higher probability group you can set a minimum threshold, say 0.3. Since both are below this you classify it as neutral&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-03-25T13:42:51.997" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10893" PostTypeId="1" AcceptedAnswerId="10894" CreationDate="2016-03-25T15:44:35.820" Score="1" ViewCount="1903" Body="&lt;p&gt;guys!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with a bunch of costumer-behavior features and the output being &quot;Churned&quot;/&quot;Not churned&quot;. I applied a simple Random Forest Classifier and got a nice performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this, I can predict whether or not a given user will churn. But I need to understand what are the patterns among churned users and patterns among non-churned users. How could I achieve that? (Where I could present something like &quot;Usually, users that churn do this, that and that&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: No need for a full explanation, I'd happy enough if you give me some directions to what to study so I can achieve this knowledge&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="17330" LastActivityDate="2016-03-25T16:20:07.660" Title="Understanding churn prediction model" Tags="|python|scikit-learn|random-forest|churn|" AnswerCount="1" CommentCount="1" ClosedDate="2016-03-25T18:08:11.127" ContentLicense="CC BY-SA 3.0" />
  <row Id="10894" PostTypeId="2" ParentId="10893" CreationDate="2016-03-25T16:20:07.660" Score="1" Body="&lt;p&gt;A couple good options would be to look at a &lt;a href=&quot;http://freakonometrics.hypotheses.org/19835&quot; rel=&quot;nofollow&quot;&gt;Feature/Variable importance plot&lt;/a&gt; for your RF model.  Alternatively, depending on the model you could try extracting a couple individual trees from the model and examining them.  However, these methods  wouldn't be definitive; i.e. determining what variables are strong predictors for churn does not mean that they have a causal impact on churn, and an individual tree may be biased and not representative of the aggregation output presented by the RF model. To determine causation, you could use these methods as a starting point to design a test.  &lt;/p&gt;&#xA;" OwnerUserId="17256" LastActivityDate="2016-03-25T16:20:07.660" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10895" PostTypeId="2" ParentId="10877" CreationDate="2016-03-25T16:25:09.883" Score="0" Body="&lt;p&gt;Checkout the R programming language, specifically the &lt;a href=&quot;https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html&quot; rel=&quot;nofollow&quot;&gt;dplyr&lt;/a&gt; and &lt;a href=&quot;http://docs.ggplot2.org/current/&quot; rel=&quot;nofollow&quot;&gt;ggplot2&lt;/a&gt; packages.  &lt;/p&gt;&#xA;" OwnerUserId="17256" LastActivityDate="2016-03-25T16:25:09.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10897" PostTypeId="2" ParentId="10877" CreationDate="2016-03-25T17:12:51.377" Score="0" Body="&lt;p&gt;Try QlikView. It's not open source but is free as a beer, does everything you described with awesome speed and is very convenient.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-03-25T17:12:51.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10898" PostTypeId="1" AcceptedAnswerId="10900" CreationDate="2016-03-25T17:34:06.167" Score="8" ViewCount="6079" Body="&lt;p&gt;I have a dataset with 40 million item, where each item is 400-dimension double vector. What I want to do is &lt;strong&gt;to find top-k (small k, about 3~10) most similar items to an arbitrary given input vector&lt;/strong&gt;. Similarity measure is cosine similarity, since this dataset is based on word2vec representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this data is so huge, so it couldn't fit at main memory (I'm now working in single machine). The goal I want to achieve is to find top-k similar items as fast as possible, with small memory(~5G) that could fit in RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions for this problem? I've already tried PCA, but I observed that this data projected in low-dimension wasn't working very well..&lt;/p&gt;&#xA;" OwnerUserId="17332" LastActivityDate="2020-08-17T15:55:22.420" Title="Method for finding top-k cosine similarity based closest item on large dataset" Tags="|data-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10899" PostTypeId="1" CreationDate="2016-03-25T18:10:45.370" Score="2" ViewCount="1036" Body="&lt;p&gt;I tried the random forest model in my research topic, but I met a problem during the validation phase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When, I used the final model of random forest to predict on an independent dataset, I received this message:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Type of predictors in new data do not match that of the training data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, to detect the different categories in my factors/variables, I used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;levels(Train$Aquifer.media)           levels(Test$Aquifer.media)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this factor &quot;Aquifer.media&quot;, I have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train dataset: &quot;Carbonates rocks&quot;  &quot;Crystalline rocks&quot;  &quot;Siliciclastic sedimentary rocks&quot;  &quot;Unconsolisated sediments rocks&quot;  &quot;Volcanic rocks&quot;&#xA;&#xA;Test Dataset: &quot;Crystalline rocks&quot;  &quot;Siliciclastic sedimentary rocks&quot;  &quot;Unconsolisated sediments rocks&quot;  &quot;Volcanic rocks&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I detected that predictors  were of different categories, I would like to know, how I can solve this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible  to delete some categories in the factors?&lt;/p&gt;&#xA;" OwnerUserId="17331" LastEditorUserId="31513" LastEditDate="2017-07-11T11:13:29.030" LastActivityDate="2017-07-11T11:13:29.030" Title="Random forest model in R - predictors and training data types mismatch" Tags="|r|random-forest|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10900" PostTypeId="2" ParentId="10898" CreationDate="2016-03-25T18:28:31.210" Score="5" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Locality sensitive hashing&lt;/a&gt; is a great tool for this problem.&lt;/p&gt;&#xA;&lt;p&gt;Pick &lt;em&gt;n&lt;/em&gt; random 400-dimensional vectors. (Be careful or not all directions will be chosen with equal probability; pick each dimension as a standard Gaussian.) Each really defines a hyperplane through the origin cutting your space in half. The sign of the dot product of any of these vectors with some new vector tells you which side of the hyperplane it's on. So computing &lt;em&gt;n&lt;/em&gt; dot products gives &lt;em&gt;n&lt;/em&gt; 0/1 bits, which make an n-bit hash.&lt;/p&gt;&#xA;&lt;p&gt;Any new vector hashing to the same value must be in the same small sliver of space from the origin. And those are exactly the vectors that have a high cosine similarity with each other since their mutual angles are very small. Likewise anything hashing to almost the same value -- differing in a few bits -- is likely to be nearby. So you can restrict your search for most-similar vectors to things within one or more buckets of hashed candidate vectors.&lt;/p&gt;&#xA;&lt;p&gt;It doesn't help directly with memory since you might need any particular bucket to satisfy a request. You also lose some accuracy since there is not a guarantee the most similar vectors lie in the buckets you examine (though it's likely, the more you examine). It lets you trade speed for accuracy mostly. However you may find you can get away with some caching scheme, where some buckets are rarely if ever accessed and so don't stay in memory.&lt;/p&gt;&#xA;&lt;p&gt;You can see an &lt;a href=&quot;https://github.com/OryxProject/oryx/blob/master/app/oryx-app-serving/src/main/java/com/cloudera/oryx/app/serving/als/model/LocalitySensitiveHash.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation of this in Oryx&lt;/a&gt;, which I think is pretty straightforward.&lt;/p&gt;&#xA;&lt;p&gt;Most of the complexity comes because it lets you specify a target percentage of vectors to evaluate, and works out the optimal hash size based on that and your machine's number of cores.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="98307" LastEditDate="2020-08-17T15:55:22.420" LastActivityDate="2020-08-17T15:55:22.420" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10901" PostTypeId="2" ParentId="10880" CreationDate="2016-03-25T19:29:23.097" Score="4" Body="&lt;p&gt;Except for the OCR part, the right bundle would be &lt;code&gt;pandas&lt;/code&gt; and &lt;code&gt;sklearn&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check this &lt;a href=&quot;https://gist.github.com/herokyar/5ffb653df69139a3e2b2&quot; rel=&quot;noreferrer&quot;&gt;ipython notebook&lt;/a&gt; which uses TfidfVectorizer and SVC Classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This classifier can make &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;noreferrer&quot;&gt;one-vs-one or one-vs-the-rest&lt;/a&gt; multiclass predictions, and if you use the &lt;code&gt;predict_proba&lt;/code&gt; method instead of &lt;code&gt;predict&lt;/code&gt;, you would have the confidence level of each category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for performances and you don't need prediction confidence levels, you should use &lt;code&gt;LinearSVC&lt;/code&gt; which is way faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sklearn is very well documented and you will find everything you need for text classification.&lt;/p&gt;&#xA;" OwnerUserId="17324" LastActivityDate="2016-03-25T19:29:23.097" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10903" PostTypeId="2" ParentId="10877" CreationDate="2016-03-26T00:09:52.487" Score="0" Body="&lt;p&gt;You can do this in pandas since your data set is small. For &quot;big&quot; data that does not fit in memory you would want to use a database; &lt;a href=&quot;https://en.wikipedia.org/wiki/PostgreSQL&quot; rel=&quot;nofollow&quot;&gt;PostgreSQL&lt;/a&gt; with the &lt;a href=&quot;https://en.wikipedia.org/wiki/PostGIS&quot; rel=&quot;nofollow&quot;&gt;PostGIS extension&lt;/a&gt; would be ideal, since it handles the nearest neighbor part, which is the most challenging aspect. Here are some sample queries, in python.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Show me the Globs where average of each Glob in file Efficiency is greater than 50%&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas&#xA;&#xA;efficiency = pandas.read_csv('efficiency.csv', sep=',', index_col=0)&#xA;structure = pandas.read_csv('structure.cv', sep=',', index_col=0)&#xA;&#xA;efficiency[efficiency.mean(1) &amp;gt; 0.5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Sort the Structure list descending on Run3, then ascending on Run2.&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure.sort_values(by=[&quot;Run3&quot;, &quot;Run2&quot;], ascending=[False, True])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;I'm not sure how you're defining the distance so I am unable to demonstrate the last part.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-26T00:18:12.877" LastActivityDate="2016-03-26T00:18:12.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10905" PostTypeId="1" AcceptedAnswerId="10908" CreationDate="2016-03-26T06:18:22.083" Score="1" ViewCount="108" Body="&lt;p&gt;I have sets of features of different nature(for example, 300 features from FFT-transform, 1000 categorical features and so on). However there are only 900 samples and I`m trying to select important features using Lasso. &#xA;So the question is: should I perform feature selection differentially on subsets of features or better firstly, concatenate features and only then perform feature selection? &lt;/p&gt;&#xA;" OwnerUserId="15693" LastActivityDate="2016-03-26T07:04:55.367" Title="Features selection: should I mix features?" Tags="|classification|scikit-learn|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10906" PostTypeId="2" ParentId="10870" CreationDate="2016-03-26T06:20:01.023" Score="0" Body="&lt;p&gt;Generating a training set requires an expert-domain knowledge, it can be very hard or it can be very easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example 1: Web document classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interested to classify a web-document, you'd have billions web pages on the Internet for you to download. The problem is not the amount of data (you just need a web crawler, therefore cheap), but how you process them into something representation that is more manageable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example 2: Disease classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Collecting disease data could be very expensive. Not only there could be legislation issues, you'd need a team of Phd specialists to analyze the data (very expensive). The experiment must also be sound statistically, for example, you'd have to consider covariate variables.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-03-26T06:20:01.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10907" PostTypeId="2" ParentId="10875" CreationDate="2016-03-26T06:31:03.637" Score="1" Body="&lt;p&gt;It's fine to do a t-test on unequal sample size, however, the power wouldn't be as good as equal sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1:) Yes or no. Impossible to say without plotting the outliers. What's more important, can you &lt;strong&gt;assume&lt;/strong&gt; your data be normally distributed? Have you checked the QQ-plot? Have you checked the histogram? Do they look like close to a normal distribution? While the t-test is robust against non-normal data as long as the sample size is sufficient large, your data shouldn't behave too far away from a normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you think about outliers, ask yourself the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How many outliers? If you have many, t-test is probably not appropriate.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Why&lt;/strong&gt; the outliers? If it's a random error (you're just unlucky), you could include it in the t-test. If it's a systematic error, stop the test, go back and check your data. &lt;/li&gt;&#xA;&lt;li&gt;How do you define the outliers? &lt;/li&gt;&#xA;&lt;li&gt;Do those outliers look symmetry? If so, you might assume your sample come from a normal population. You can check the skewness of your data. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You &lt;strong&gt;have&lt;/strong&gt; to try to understand those outliers to come with up a decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2:) You can just explain like &lt;em&gt;&quot;the probability of the difference in means is (or isn't) significant&quot;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3:) You should draw a box-plot for each group.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-03-26T06:39:56.293" LastActivityDate="2016-03-26T06:39:56.293" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10908" PostTypeId="2" ParentId="10905" CreationDate="2016-03-26T07:04:55.367" Score="1" Body="&lt;p&gt;Let LASSO pick the best ones. If the features are highly correlated and you want them picked as a group, add some L2 regularization too. This is called Elastic Net regularization, and it is a generalization of L1 and L2 regularization. Other than that, do not feel obliged to artificially group features.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-26T07:04:55.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10909" PostTypeId="2" ParentId="10891" CreationDate="2016-03-26T17:24:08.157" Score="6" Body="&lt;p&gt;Wang, Kaijun, Baijie Wang, and Liuqing Peng. &amp;quot;CVAP: Validation for cluster analyses.&amp;quot; Data Science Journal 0 (2009): 0904220071.:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;To measure the quality of clustering results, there are two kinds of&#xA;validity indices: external indices and internal indices.&lt;/p&gt;&#xA;&lt;p&gt;An external&#xA;index is a measure of agreement between two partitions where the first&#xA;partition is the a priori known clustering structure, and the second&#xA;results from the clustering procedure (Dudoit et al., 2002).&lt;/p&gt;&#xA;&lt;p&gt;Internal&#xA;indices are used to measure the goodness of a clustering structure&#xA;without external information (Tseng et al., 2005).&lt;/p&gt;&#xA;&lt;p&gt;For external indices, we evaluate the results of a clustering algorithm based on a known cluster structure of a data set (or cluster labels).&lt;/p&gt;&#xA;&lt;p&gt;For internal indices, we evaluate the results using quantities and features inherent in the data set. The optimal number of clusters is usually determined based on an internal validity index.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;(Dudoit et al., 2002): Dudoit, S. &amp;amp; Fridlyand, J. (2002) A prediction-based resampling method for estimating the number of clusters in a dataset. Genome Biology, 3(7): 0036.1-21.&lt;/p&gt;&#xA;&lt;p&gt;(Tseng et al., 2005): Thalamuthu, A, Mukhopadhyay, I, Zheng, X, &amp;amp; Tseng, G. C. (2006) Evaluation and comparison of gene clustering methods in microarray analysis. Bioinformatics, 22(19):2405-12.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;In your case, you need some internal indices since you have no a priori clustering structure. There exist tens of internal indices, like:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Silhouette index (implementation in &lt;a href=&quot;http://www.mathworks.com/help/stats/silhouette.html&quot; rel=&quot;noreferrer&quot;&gt;MATLAB&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Davies-Bouldin&lt;/li&gt;&#xA;&lt;li&gt;Calinski-Harabasz&lt;/li&gt;&#xA;&lt;li&gt;Dunn index (implementation in &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/27859-dunns-index&quot; rel=&quot;noreferrer&quot;&gt;MATLAB&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;R-squared index&lt;/li&gt;&#xA;&lt;li&gt;Hubert-Levin (C-index)&lt;/li&gt;&#xA;&lt;li&gt;Krzanowski-Lai index&lt;/li&gt;&#xA;&lt;li&gt;Hartigan index&lt;/li&gt;&#xA;&lt;li&gt;Root-mean-square standard deviation (RMSSTD) index&lt;/li&gt;&#xA;&lt;li&gt;Semi-partial R-squared (SPR) index&lt;/li&gt;&#xA;&lt;li&gt;Distance between two clusters (CD) index&lt;/li&gt;&#xA;&lt;li&gt;weighted inter-intra index&lt;/li&gt;&#xA;&lt;li&gt;Homogeneity index&lt;/li&gt;&#xA;&lt;li&gt;Separation index&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Each of them have pros and cons, but at least they'll give you a more formal basis for your comparison. The MATLAB toolbox &lt;a href=&quot;http://www.mathworks.com/matlabcentral/fileexchange/14620-cvap-cluster-validity-analysis-platform-cluster-analysis-and-validation-tool&quot; rel=&quot;noreferrer&quot;&gt;CVAP&lt;/a&gt; might be handy as it contains many internal validity indices.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;More information:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set&quot; rel=&quot;noreferrer&quot;&gt;Determining the number of clusters in a data set&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/76093/12359&quot;&gt;How to determine which method is the most reasonable clustering results?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/How-can-I-detect-if-my-dataset-is-clustered-or-unclustered-i-e-forming-one-single-cluster?share=1&quot; rel=&quot;noreferrer&quot;&gt;How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/How-can-we-choose-a-good-K-for-K-means-clustering?srid=3p9p&quot; rel=&quot;noreferrer&quot;&gt;How can we choose a &amp;quot;good&amp;quot; K for K-means clustering?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/q/15376075/395857&quot;&gt;Cluster analysis in R: determine the optimal number of clusters&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-03-26T17:24:08.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10910" PostTypeId="1" AcceptedAnswerId="10911" CreationDate="2016-03-26T20:18:02.517" Score="1" ViewCount="2049" Body="&lt;p&gt;I am implementing a Gaussian Naive Bayes classifier (so each feature is continuous and assumed to be coming from a Gaussian distribution).  When evaluating the probability of a feature value in the test set, if the value is sufficiently far away from the mean (e.g. the mean and s.d. on the training data is say 0 and 1 but the test value is 10^10) then there is underflow.  This is an issue because then the probability will be calculated as 0.0 so the log probability is undefined.  Is there a standard way of handling underflow in this case?&lt;/p&gt;&#xA;" OwnerUserId="17344" LastActivityDate="2016-03-26T20:59:26.260" Title="Handling underflow in a Gaussian Naive Bayes classifier" Tags="|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10911" PostTypeId="2" ParentId="10910" CreationDate="2016-03-26T20:59:26.260" Score="4" Body="&lt;p&gt;The standard answer is to work in log space, and manipulate the log of probabilities instead of probabilities, for exactly this reason. This classifier involves products of probabilities which just become sums of log probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You allude to that already, but the problem you suggest isn't a problem. Internally you don't calculate a probability and then take the log again. It stays in log space. So for very small P, log P is a large negative number while P itself may underflow to 0.0. But you may never need to evaluate P internally.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-03-26T20:59:26.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10912" PostTypeId="1" CreationDate="2016-03-26T21:54:20.927" Score="2" ViewCount="428" Body="&lt;p&gt;I know 'easy to use' is going to be subjective, so let me qualify the question a little. Is there a library or working scrap of code that I can essentially copy into my project, change the number of neurons in each layer, the number of layers, and the source of the inputs and then click run. I've written at least 15 of these myself, not a single one has worked properly - I need to see something working in a language I understand so I can get to grips with why I'm apparently a moron!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I can post some of my own code if anyone is interested, but it's lengthy and not too related to the question)&lt;/p&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="843" LastEditDate="2016-05-29T02:31:04.833" LastActivityDate="2021-07-30T05:38:20.987" Title="Is there an easy to use .net library for neural networks?" Tags="|neural-network|software-recommendation|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10913" PostTypeId="1" CreationDate="2016-03-27T05:50:49.920" Score="3" ViewCount="768" Body="&lt;p&gt;I'm using xgboost with the reg:logistic objective. As far as I understand, that means that I'm trying to optimize the log-likelihood loss function of my model, and that each boosting step would be in the gradient direction of this function (w.r.t all the examples I've given).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does changing the eval_parameter to auc affect any of the above? Or is it just affecting what's measured on my validation set examples every 10 steps of boosting?&lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2016-03-27T15:22:10.473" Title="does xgboost's eval_metric changes the loss function being optimized?" Tags="|xgboost|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10915" PostTypeId="2" ParentId="10870" CreationDate="2016-03-27T12:33:17.590" Score="0" Body="&lt;p&gt;I think I can make it a bit more clear and collect previous answers into one. You can think of four types of data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Type 1: Inherently labeled data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of this would be something like dataset for guess the next word in a sentence. As soon as you have the text corpus you can also get the target words for your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Type 2: Generated data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am not mistaken then MNIST database was generated by asking people write numbers. If you have a row of handwritten numbers that are all supposed to be fives then you do not have to pay another person afterwards to label them. This brings us to...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Type 3: Labeled data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Human expert has gone through and labeled the data by hand. Example of this would be doctors classifying the disease type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Type 4: Unlabeled data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anything that doesnt have a label.&lt;/p&gt;&#xA;" OwnerUserId="17355" LastActivityDate="2016-03-27T12:33:17.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10918" PostTypeId="2" ParentId="10913" CreationDate="2016-03-27T15:22:10.473" Score="3" Body="&lt;p&gt;Chaging &lt;code&gt;eval_metric&lt;/code&gt; does not affect the fitted model in any way. You have to interact with &lt;code&gt;objective&lt;/code&gt; to do this.&lt;/p&gt;&#xA;" OwnerUserId="17250" LastActivityDate="2016-03-27T15:22:10.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10919" PostTypeId="1" AcceptedAnswerId="10921" CreationDate="2016-03-27T15:50:52.957" Score="5" ViewCount="626" Body="&lt;p&gt;Im a 26 year old guy with an MBA and I work as an ERP system administrator.&#xA;I have been interested in the field of data science for a while now. I've always liked statistics and various analytical tasks.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would really like to try and work within this field, but it feels a bit overwhelming. People mention that you have to learn Python, R, SQL, Machine Learning, advanced algebra, data modeling, big data (hadoop etc), predictive analytics, various business intelligence tools, VBA, Matlab etc etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As of today, I have some SQL knowledge and have a general understanding of BI, big data, good excel skills etc. I am willing to learn some of the aforementioned areas, but I don't have the money or time to go back to full time university studies again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here is my question: is there any recognized &quot;light version&quot; of data scientist on the market? What are they usually called? What skills do they need to master? What should I learn in order to work with big data sets and analytics without having to study full time for another 5 years?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I live in Scandinavia so the job market is probably different here, but I thought it would be interesting to hear some answers.&lt;/p&gt;&#xA;" OwnerUserId="17359" LastEditorUserId="29169" LastEditDate="2019-06-07T23:27:51.690" LastActivityDate="2019-06-07T23:27:51.690" Title="Data scientist light?" Tags="|beginner|career|" AnswerCount="2" CommentCount="5" CommunityOwnedDate="2018-08-04T16:09:24.433" ContentLicense="CC BY-SA 4.0" />
  <row Id="10920" PostTypeId="1" AcceptedAnswerId="10922" CreationDate="2016-03-27T19:15:54.487" Score="3" ViewCount="230" Body="&lt;p&gt;In the game won by Lee Sedol, AlphaGo was apparently surprised by a brilliant and unexpected move from Lee Sedol.&#xA;After analysing the logs, the Deep Mind CEO said that AlphaGo had evaluated a 1/10000 probability for that specific move to be played by Lee Sedol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand here is : whatever the probability for a good move to be played, why taking the risk ? Why not instead expecting the opponent to always plays the best moves ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course it's always possible that you miss the best move your opponent could play when playing Monte Carlo to evaluate his possibilities, but here it seems that the move was founded. If AlphaGo knew that its strategy could be counter by such a move, why not choosing another strategy, where the worst case scenario would be less &quot;worse&quot;.&lt;/p&gt;&#xA;" OwnerUserId="17360" LastActivityDate="2016-03-28T11:58:26.977" Title="Why doesn't AlphaGo expect its opponent to play the best possible moves?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10921" PostTypeId="2" ParentId="10919" CreationDate="2016-03-27T19:56:37.637" Score="9" Body="&lt;p&gt;&lt;strong&gt;Business intelligence&lt;/strong&gt; is perfect for you; you already have the business background. If you want to become a bona fide data scientist brush up on your computer science, linear algebra, and statistics. I consider these the bare essentials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know about Scandinavia, but in the U.S., data science covers a broad spectrum of tasks ranging from full-time software development to full-time data analysis, often with domain expertise required in various niches, such as experimental design. You have to decide where your strengths and interests lie to pick a position on this spectrum, and prepare accordingly. Useful activities include participating in Kaggle competitions, and contributing to open source data science libraries.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-27T19:56:37.637" CommentCount="0" CommunityOwnedDate="2018-08-04T16:09:24.433" ContentLicense="CC BY-SA 3.0" />
  <row Id="10922" PostTypeId="2" ParentId="10920" CreationDate="2016-03-27T20:45:23.953" Score="5" Body="&lt;p&gt;It appears that AlphaGo did &lt;em&gt;not&lt;/em&gt; rate the move as a best possible move for Lee Sedol, just as one that was within its search space. To put into context the board is 19x19, so a 1 in 10000 chance of a move is much lower than chance of the square being picked at random. That likely makes the move that it &quot;found&quot; not worth exploring much deeper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to note too that the probabilities assigned to moves are equivalent to AlphaGo's rating for quality of that move - i.e. AlphaGo predicted that this was a bad choice for its opponent. Another way of saying this, is &quot;there is a probability p that this move is the best possible one, and therefore worth investigating further&quot;. There is no separate quality rating - AlphaGo does not model &quot;opponent's chance of making a move&quot; separately from &quot;opponent's chance of gaining the highest score from this position if he/she makes that move&quot;. There is just one probability covering both those meanings &lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it, AlphaGo rates the probabilities of &lt;em&gt;all possible moves&lt;/em&gt; at each game board state that it considers (starting with the current board), and employs the most search effort for deeper searches on the highest rated ones. I don't know the ratios or how many nodes are visited in a typical search, but expect that a 1 in 10000 rating would not have been explored in much detail if at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not surprising to see the probability calculation in the system logs, as the logs likely contain the ratings for &lt;em&gt;all&lt;/em&gt; legal next moves, as well as ratings for things that didn't actually happen in the game but AlphaGo considered in its deeper searches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also not surprising that AlphaGo failed to rate the move correctly. The neural network is not expected to be a perfect oracle that rates all moves perfectly (if it was, then there would be no need to search). In fact, the opposite could be said to be the case - it is surprising (and of course an amazing feat of engineering) just how good the predictions are, good enough to beat a world-class champion. This is not the same as solving the game though. Go remains &quot;unsolved&quot;, even if machines can beat humans, there is an unknown amount of additional room for better and better players - and in the immediate future that could be human or machine.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There are in fact two networks evaluating two different things - the &quot;policy network&quot; evaluates potential moves, and the output of that affects the Monte Carlo search. There is also a &quot;value network&quot; which assesses board states to score the end point of the search. It is the policy network that predicted the low probability of the move, which meant that the search had little or no chance of exploring game states past Lee Sedol's move (if it had, maybe the value network would of detected a poor end result from playing that through). In reinforcement learning, a &lt;em&gt;policy&lt;/em&gt; is set of rules, based on known state, that decide between actions that an agent can take.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-03-28T11:58:26.977" LastActivityDate="2016-03-28T11:58:26.977" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="10923" PostTypeId="2" ParentId="10912" CreationDate="2016-03-27T22:03:32.567" Score="2" Body="&lt;p&gt;I have no personal experience with it, but I came across &lt;a href=&quot;http://accord-framework.net/docs/html/N_Accord_Neuro.htm&quot; rel=&quot;nofollow&quot;&gt;Accord&lt;/a&gt;, which has limited support for neural networks. Honestly, .net does not have a machine learning community, so you might be better off using something else if you do not want to DIY.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-03-27T22:03:32.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10924" PostTypeId="2" ParentId="10919" CreationDate="2016-03-28T01:44:15.677" Score="3" Body="&lt;p&gt;The &lt;strong&gt;business analyst&lt;/strong&gt; flavour of data science is something you are nicely suited for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I have seen business analysts and business intelligence engineers in the industry, most of their work is centered around deriving insights from Excel sheets and writing SQL queries to dig out the appropriate data. They do write scripts, but that is generally for just the visualization purpose, and not for higher lever analytics like Machine Learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can also see a nice future for you in the &lt;strong&gt;financial analytics/quant&lt;/strong&gt; domain. It is also a domain where the learning curve is a bit steep, but totally worth it. &lt;a href=&quot;https://www.quora.com/How-do-I-become-a-financial-data-scientist/answer/Jalem-Raj-Rohit&quot; rel=&quot;nofollow&quot;&gt;Here is my answer&lt;/a&gt; on Quora about getting into the field of quant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you want to get up to speed with data science, then you have to slowly build up strong linear algebra skills, along with a very keen and valuable domain knowledge in whatever domain you'd be working with. The latter is often underrated, but from my (short but valuable) experience in the industry; I vouch for that fact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus resources:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/topic/Data-Science&quot; rel=&quot;nofollow&quot;&gt;Quora data science topic wiki&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.metacademy.org/roadmaps/&quot; rel=&quot;nofollow&quot;&gt;Metacademy learning paths&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ocw.mit.edu/courses/mathematics/&quot; rel=&quot;nofollow&quot;&gt;OCW math courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you're okay with starting with a full blast ground up learning marathon for data science, then this is the path I'd recommend:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Single Variable Calculus&lt;/li&gt;&#xA;&lt;li&gt;Multi Variable Calculus&lt;/li&gt;&#xA;&lt;li&gt;Differential Equations&lt;/li&gt;&#xA;&lt;li&gt;Linear Algebra&lt;/li&gt;&#xA;&lt;li&gt;Probability theory and combinatorics Basics&lt;/li&gt;&#xA;&lt;li&gt;Statistics&lt;/li&gt;&#xA;&lt;li&gt;Algorithms&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All the above courses are available in the OCW catalogue. If not, then you can find them in other MOOC aggregators too.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-03-28T02:34:32.747" LastActivityDate="2016-03-28T02:34:32.747" CommentCount="1" CommunityOwnedDate="2018-08-04T16:09:24.433" ContentLicense="CC BY-SA 3.0" />
  <row Id="10926" PostTypeId="1" CreationDate="2016-03-28T12:17:51.047" Score="11" ViewCount="4072" Body="&lt;p&gt;Given &lt;em&gt;difficult&lt;/em&gt; learning task (e.g high dimensionality, inherent data complexity) &lt;em&gt;Deep Neural Networks&lt;/em&gt; become hard to train.&#xA; To ease many of the problems one might:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normalize &amp;amp;&amp;amp; handpick &lt;em&gt;quality data&lt;/em&gt; &lt;/li&gt;&#xA;&lt;li&gt;choose a different &lt;em&gt;training algorithm&lt;/em&gt; (e.g RMSprop instead of Gradient Descent)&lt;/li&gt;&#xA;&lt;li&gt;pick a steeper gradient &lt;em&gt;Cost function&lt;/em&gt; (e.g Cross Entropy instead of MSE)&lt;/li&gt;&#xA;&lt;li&gt;Use different network structure (e.g Convolution layers instead of Feedforward)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better:&#xA;Glorot and Bengio (2010)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;for sigmoid units: sample a &lt;em&gt;Uniform(-r, r)&lt;/em&gt; with $r =&#xA;   \sqrt{\frac{6}{N_{in} + N_{out}}}$&lt;/li&gt;&#xA;&lt;li&gt;or hyperbolic tangent units: sample a &lt;em&gt;Uniform(-r, r)&lt;/em&gt; with $r =4&#xA;   \sqrt{\frac{6}{N_{in} + N_{out}}}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there any consistent way of initializing the weights better?&lt;/p&gt;&#xA;" OwnerUserId="17355" LastActivityDate="2016-03-28T16:14:41.793" Title="HOW TO: Deep Neural Network weight initialization" Tags="|machine-learning|neural-network|deep-learning|randomized-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10927" PostTypeId="1" CreationDate="2016-03-28T13:43:29.393" Score="2" ViewCount="308" Body="&lt;p&gt;I am using the Stanford NER to recognize each entity in a search text. Once I identify entities, I need to pass that entities to an algorithm which calculates score for each entity type (e.g. country, customer) using weights of each word. Currently my training data has word and answer like below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;country_training.tsv&#xA;    brazil country&#xA;    japan country&#xA;    &#xA;customer_training.tsv&#xA;    hyundai customer&#xA;    apple customer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How can I associate weight to each above training data like below so when I can get weight of each word too?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;country_training.tsv&#xA;    brazil country 1.5&#xA;    japan country 4.0&#xA;    &#xA;customer_training.tsv&#xA;    hyundai customer 4.0&#xA;    apple customer 2.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Please advise.&lt;/p&gt;&#xA;&lt;p&gt;Weights are input to NER to annotate each entity wiith their weights.&lt;/p&gt;&#xA;" OwnerUserId="17381" LastEditorUserId="29169" LastEditDate="2021-05-21T20:22:35.610" LastActivityDate="2021-05-21T20:22:35.610" Title="Stanford NER Training - Assign weight to each word" Tags="|nlp|language-model|stanford-nlp|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10928" PostTypeId="5" CreationDate="2016-03-28T16:11:22.967" Score="0" Body="&lt;p&gt;Stanford NLP is natural language processing research group at Stanford University. It is comprised of faculty, post-docs, programers, and students that all work towards helping computers better understand human language. This group provides and maintains a NLP toolkit called CoreNLP that allows for natural language processing in Java. The group also works across multiple departments at Stanford including the Linguistics and Computer Science departments. It is also a part of the Stanford AI Lab.&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2021-01-17T23:01:57.593" LastActivityDate="2021-01-17T23:01:57.593" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10929" PostTypeId="4" CreationDate="2016-03-28T16:11:22.967" Score="0" Body="Stanford NLP is natural language processing research group at Stanford University." OwnerUserId="843" LastEditorUserId="29169" LastEditDate="2021-01-17T22:13:08.903" LastActivityDate="2021-01-17T22:13:08.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="10930" PostTypeId="2" ParentId="10926" CreationDate="2016-03-28T16:14:41.793" Score="7" Body="&lt;p&gt;As far as I know the two formulas you gave are pretty much the standard initialization. I had done a &lt;a href=&quot;https://stats.stackexchange.com/a/186351/12359&quot;&gt;literature review&lt;/a&gt; a while ago, I copied it below if interested.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;[1] addresses the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, weights shouldn't be set to zeros in order to break the symmetry when backprogragating:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Biases can generally be initialized to zero but weights need to be initialized carefully to break the symmetry between hidden units of the same layer. Because different output units receive different gradient signals, this symmetry breaking issue does not concern the output weights (into the output units), which can therefore also be set to zero.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Some initialization strategies:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[2] and [3] recommend scaling by the inverse of the square root of the fan-in&lt;/li&gt;&#xA;&lt;li&gt;Glorot and Bengio (2010) and the Deep Learning Tutorials use a combination of the fan-in and fan-out:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;for sigmoid units: sample a Uniform(-r, r) with $r=\sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).&lt;/li&gt;&#xA;&lt;li&gt;for hyperbolic tangent units: sample a Uniform(-r, r) with $r=4 \sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;in the case of RBMs, a zero-mean Gaussian with a small standard deviation around 0.1 or 0.01 works well (Hinton, 2010) to initialize the weights.&lt;/li&gt;&#xA;&lt;li&gt;Orthogonal random matrix initialization, i.e. &lt;code&gt;W = np.random.randn(ndim, ndim); u, s, v = np.linalg.svd(W)&lt;/code&gt; then use &lt;code&gt;u&lt;/code&gt; as your initialization matrix.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, unsupervised pre-training may help in some situations:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An important choice is whether one should use&#xA;  unsupervised pre-training (and which unsupervised&#xA;  feature learning algorithm to use) in order&#xA;  to initialize parameters. In most settings&#xA;  we have found unsupervised pre-training to help&#xA;  and very rarely to hurt, but of course that&#xA;  implies additional training time and additional&#xA;  hyper-parameters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Some ANN libraries also have some interesting lists, e.g. &lt;a href=&quot;http://lasagne.readthedocs.org/en/latest/modules/init.html#module-lasagne.init&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lasagne&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Constant([val]) Initialize weights with constant value.&#xA;Normal([std, mean]) Sample initial weights from the Gaussian distribution.&#xA;Uniform([range, std, mean]) Sample initial weights from the uniform distribution.&#xA;Glorot(initializer[, gain, c01b])   Glorot weight initialization.&#xA;GlorotNormal([gain, c01b])  Glorot with weights sampled from the Normal distribution.&#xA;GlorotUniform([gain, c01b]) Glorot with weights sampled from the Uniform distribution.&#xA;He(initializer[, gain, c01b])   He weight initialization.&#xA;HeNormal([gain, c01b])  He initializer with weights sampled from the Normal distribution.&#xA;HeUniform([gain, c01b]) He initializer with weights sampled from the Uniform distribution.&#xA;Orthogonal([gain])  Intialize weights as Orthogonal matrix.&#xA;Sparse([sparsity, std]) Initialize weights as sparse matrix.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;[1] Bengio, Yoshua. &quot;&lt;a href=&quot;http://arxiv.org/abs/1206.5533&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical recommendations for gradient-based training of deep architectures.&lt;/a&gt;&quot; Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] LeCun, Y., Bottou, L., Orr, G. B., and Muller, K. (1998a). &lt;a href=&quot;https://scholar.google.com/scholar?cluster=1366432119558306638&amp;amp;hl=en&amp;amp;as_sdt=0,22&quot; rel=&quot;nofollow noreferrer&quot;&gt;Efficient backprop. In Neural Networks, Tricks of the Trade&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] Glorot, Xavier, and Yoshua Bengio. &quot;&lt;a href=&quot;https://scholar.google.com/scholar?cluster=17889055433985220047&amp;amp;hl=en&amp;amp;as_sdt=0,22&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding the difficulty of training deep feedforward neural networks&lt;/a&gt;.&quot; International conference on artificial intelligence and statistics. 2010.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-03-28T16:14:41.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10931" PostTypeId="1" CreationDate="2016-03-28T16:17:17.790" Score="2" ViewCount="240" Body="&lt;p&gt;I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of data I need. However, Google doesn't seem to provide access to historical data. There are several other companies that provide analytics based on traffic data to their clients but raw data is never provided. How would one go about obtaining this kind of data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: the kind of data I need would be spatio-temporal. Take a bunch of N sensor locations on an inner city road network and give me the driving speed of the cars passing through each one of them at T equally spaced time points during a day. So we would have N*T data points per day. The denser the locations the better. &#xA;If you look at Google Maps traffic overlay you can see traffic flow speeds for almost every small street. I think they take this data from GPS in driving cars as well as stationary sensors. Google doesn't provide historic data but maybe one could record this from the website directly?&lt;/p&gt;&#xA;" OwnerUserId="17385" LastEditorUserId="17451" LastEditDate="2016-03-31T02:24:18.700" LastActivityDate="2016-10-27T19:56:11.047" Title="How to go about obtaining data" Tags="|data-mining|bigdata|dataset|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="10932" PostTypeId="1" AcceptedAnswerId="11696" CreationDate="2016-03-28T16:40:25.020" Score="29" ViewCount="20447" Body="&lt;p&gt;I was reading a high level summary about Google's &lt;a href=&quot;http://googleresearch.blogspot.co.uk/2016/01/alphago-mastering-ancient-game-of-go.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AlphaGo&lt;/a&gt;, and I came across the terms &amp;quot;policy network&amp;quot; and &amp;quot;value network&amp;quot;. At a high level, I understand that the policy network is used to suggest moves and the value network is used to, &amp;quot;Reduce the depth of the search tree [and estimate] the winner in each position in place of searching all the way to the end of the game.&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;These two networks seem redundant to me. What is the policy network doing if it's not using the value network to prune its policies? It seems pretty clear that the value network is a deep learning neural network; is the policy network just a theoretical abstraction and not an actual neural network? The target variable for the value network seems to be win/loss. Is there a target variable for the policy network; if so, what is it? What is the policy network trying to optimize?&lt;/p&gt;&#xA;&lt;p&gt;The full pdf of Google's paper, published in Nature, can be found &lt;a href=&quot;https://vk.com/doc-44016343_437229031?dl=56ce06e325d42fbc72&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="98307" LastEditDate="2020-08-20T18:57:52.073" LastActivityDate="2022-07-26T13:12:08.783" Title="Difference between AlphaGo's policy network and value network" Tags="|machine-learning|reinforcement-learning|" AnswerCount="7" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="10933" PostTypeId="1" AcceptedAnswerId="10934" CreationDate="2016-03-28T17:38:49.710" Score="2" ViewCount="177" Body="&lt;p&gt;There is a classification problem(two classes). We have train data, for which we know class labels and we have test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine, that you have created model that with good accuracy(~95%) make predictions and we know that we are &lt;strong&gt;not overfitted.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we make prediction on test data, extract objects for which we sure in class label(for example, &lt;code&gt;predict_proba&lt;/code&gt; higher than 90%) and add this objects to train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this tactic make any sense?&lt;/p&gt;&#xA;" OwnerUserId="15693" LastActivityDate="2016-04-27T17:32:05.100" Title="Use test data as train: does it make sense?" Tags="|data-mining|classification|data-cleaning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10934" PostTypeId="2" ParentId="10933" CreationDate="2016-03-28T18:30:54.287" Score="4" Body="&lt;p&gt;This idea will most likely increase the bias in the model. Let's assume that the model has non-zero bias in the model. In this case, when it assumes its predictions to be true, without confirmation from an Oracle as in &lt;a href=&quot;https://en.wikipedia.org/wiki/Active_learning_(machine_learning)&quot; rel=&quot;nofollow&quot;&gt;active learning&lt;/a&gt;, the bias of the model increases. In common terms, if the model has some amount of bias in its predictions, and it uses its predictions to learn on, the bias in the model can only increase. This issue does not arise when there is 0 bias in the model to begin with, however, in that case, there is no need to learn any further!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this is a highly intuitive answer but I cannot think of an argument against the intuition :-) I will appreciate any feedback on this. &lt;/p&gt;&#xA;" OwnerUserId="847" LastEditorUserId="847" LastEditDate="2016-04-27T17:32:05.100" LastActivityDate="2016-04-27T17:32:05.100" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10935" PostTypeId="2" ParentId="10933" CreationDate="2016-03-28T20:45:47.943" Score="0" Body="&lt;p&gt;You haven't stated your reason for wanting to do this, nor what algorithm you are using, both of which will affect whether your proposed tactic &quot;makes sense&quot;. I'll give you a couple reasons why the tactic may be a Bad Idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, data are divided into training and testing data for a reason: so you have a sense for how the classifier will generalize (i.e., to estimate it's true classification accuracy). Once you start polluting your training data with the testing data, you should have less confidence in the classifier's generalized performance. On one hand, I would expect that adding high confidence testing observations to the training data would increase training accuracy. On the other hand, your testing data will now have a larger proportion of low confidence observations, which will reduce your testing accuracy. So how would you judge the ability of the classifier to generalize?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you add the high confidence test observations to your training set, you are teaching the classifier something it already knows. Depending on your classifier, there is a good chance that by doing this your are biasing the classifier farther away from the outliers (the 5% initially misclassified) and by doing so, reducing the true (general) classification accuracy. In fact, there are algorithms designed to do just the opposite of what you are proposing: they instead place more significance on the &lt;em&gt;misclassified&lt;/em&gt; (low probability) observations to improve the accuracy of the classifier. For an example of this, take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Boosting_(machine_learning)&quot; rel=&quot;nofollow&quot;&gt;boosting&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-03-28T20:45:47.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10937" PostTypeId="2" ParentId="10857" CreationDate="2016-03-28T20:55:38.237" Score="11" Body="&lt;p&gt;This will give you all the  values under &lt;code&gt;&amp;lt;tr&amp;gt;&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bs=BeautifulSoup(data, &quot;lxml&quot;)&#xA;table_body=bs.find('tbody')&#xA;rows = table_body.find_all('tr')&#xA;for row in rows:&#xA;    cols=row.find_all('td')&#xA;    cols=[x.text.strip() for x in cols]&#xA;    print cols&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17392" LastEditorUserId="381" LastEditDate="2016-03-28T21:01:07.263" LastActivityDate="2016-03-28T21:01:07.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10940" PostTypeId="2" ParentId="10932" CreationDate="2016-03-29T08:33:15.573" Score="0" Body="&lt;p&gt;From what I understand the difference is in the outputs. Where policy network outputs a probability distribution over the possible moves, the value network returns a real value that can be interpreted as the probability of winning given this board configuration. From there Monte-Carlo tree search is performed via taking top K moves from and then narrowing the search tree again by taking top K value network outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feel obligated to correct me if I am wrong.&lt;/p&gt;&#xA;" OwnerUserId="17355" LastActivityDate="2016-03-29T08:33:15.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10941" PostTypeId="2" ParentId="10665" CreationDate="2016-03-29T08:40:20.463" Score="0" Body="&lt;p&gt;I found a good solution for my problem here: &lt;a href=&quot;https://stats.stackexchange.com/a/201315/101744&quot;&gt;https://stats.stackexchange.com/a/201315/101744&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to everyone!&lt;/p&gt;&#xA;" OwnerUserId="15690" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-03-29T08:40:20.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10943" PostTypeId="1" AcceptedAnswerId="10945" CreationDate="2016-03-29T14:14:46.867" Score="36" ViewCount="26969" Body="&lt;p&gt;I'm trying to train a gradient boosting model over 50k examples with 100 numeric features. &lt;code&gt;XGBClassifier&lt;/code&gt; handles 500 trees within 43 seconds on my machine, while &lt;code&gt;GradientBoostingClassifier&lt;/code&gt; handles only 10 trees(!) in 1 minutes and 2 seconds :( I didn't bother trying to grow 500 trees as it will take hours. I'm using the same &lt;code&gt;learning_rate&lt;/code&gt; and &lt;code&gt;max_depth&lt;/code&gt; settings, see below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes XGBoost so much faster? Does it use some novel implementation for gradient boosting that sklearn guys do not know? Or is it &quot;cutting corners&quot; and growing shallower trees? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. I'm aware of this discussion: &lt;a href=&quot;https://www.kaggle.com/c/higgs-boson/forums/t/10335/xgboost-post-competition-survey&quot;&gt;https://www.kaggle.com/c/higgs-boson/forums/t/10335/xgboost-post-competition-survey&lt;/a&gt; but couldn't get the answer there... &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1,&#xA;gamma=0, learning_rate=0.05, max_delta_step=0, max_depth=10,&#xA;min_child_weight=1, missing=None, n_estimators=500, nthread=-1,&#xA;objective='binary:logistic', reg_alpha=0, reg_lambda=1,&#xA;scale_pos_weight=1, seed=0, silent=True, subsample=1)&#xA;&#xA;GradientBoostingClassifier(init=None, learning_rate=0.05, loss='deviance',&#xA;max_depth=10, max_features=None, max_leaf_nodes=None,&#xA;min_samples_leaf=1, min_samples_split=2,&#xA;min_weight_fraction_leaf=0.0, n_estimators=10,&#xA;presort='auto', random_state=None, subsample=1.0, verbose=0,&#xA;warm_start=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2016-04-03T09:19:28.203" Title="Why is xgboost so much faster than sklearn GradientBoostingClassifier?" Tags="|scikit-learn|xgboost|gbm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10944" PostTypeId="1" AcceptedAnswerId="10948" CreationDate="2016-03-29T15:56:46.513" Score="0" ViewCount="270" Body="&lt;p&gt;I'm new to AI and specifically NLP, I always love to study things as part of a random project I decide to build, right now I'm working on a chat bot and I was looking for an easy way(in terms of code) to extract the main topic/subject from a sentence(in English), the problem is that wherever I search, even if it's a simple code, it combines a full sentence &quot;meaning&quot;(device parts of speech), I don't like to implement something I don't understand so a small step for how to extract only the subject will be really good and can help my bot with some directions, Is there a simple code(don't care what language it written with) to extract, only main topic/s from a sentence(in English)?.&lt;/p&gt;&#xA;" OwnerUserId="17415" LastEditorUserId="381" LastEditDate="2016-03-29T19:28:05.997" LastActivityDate="2016-03-29T19:28:05.997" Title="New to ML and NLP. Is topic/subject extraction a good place to start?" Tags="|nlp|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10945" PostTypeId="2" ParentId="10943" CreationDate="2016-03-29T16:54:16.703" Score="28" Body="&lt;p&gt;Since you mention &quot;numeric&quot; features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that the biggest effect comes from the fact that XGBoost uses an approximation on the split points. If you have a continuous feature with 10000 possible splits, XGBoost consider only &quot;the best&quot; 300 splits by default (this is a simplification). This behavior is controlled by the &lt;code&gt;sketch_eps&lt;/code&gt; parameter, and you can read more about it &lt;a href=&quot;https://xgboost.readthedocs.org/en/latest/parameter.html#parameters-for-tree-booster&quot; rel=&quot;noreferrer&quot;&gt;in the doc&lt;/a&gt;. You can try lowering it and check the difference it makes. Since there is no mention of it in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn documentation&lt;/a&gt;, I guess it is not available. You can learn what XGBoost method is in the &lt;a href=&quot;http://arxiv.org/abs/1603.02754&quot; rel=&quot;noreferrer&quot;&gt;their paper (arxiv)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBoost also uses an approximation on the evaluation of such split points. I do not know by which criterion scikit learn is evaluating the splits, but it could explain the rest of the time difference.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Adressing Comments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the evaluation of split points&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However, what did you mean by &quot;XGBoost also uses an approximation on the evaluation of such split points&quot;? as far as I understand, for the evaluation they are using the exact reduction in the optimal objective function, as it appears in eq (7) in the paper.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In order to evaluate the split point, you would have to compute $L(y,H_{i-1}+h_i)$ where $L$ is the cost function, $y$ the target, $H_{i-1}$ the model built until now, and $h_i$ the current addition. Notice that this is not what XGBoost is doing; they are simplifying the cost function $L$ by a Taylor Expansion, which leads to a very simple function to compute. They have to compute the Gradient and the Hessian of $L$ with respect to $H_{i-1}$, and they can reuse those number for all potential splits at stage $i$, making the overral computation fast. You can check &lt;a href=&quot;https://stats.stackexchange.com/questions/202858/loss-function-approximation-with-taylor-expansion/204377&quot;&gt;Loss function Approximation With Taylor Expansion (CrossValidated Q/A)&lt;/a&gt; for more details, or the derivation in their paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that they have found a way to approximate $L(y,H_{i-1} + h_i)$ efficiently. If you were to evaluate $L$ fully, without insider knowledge allowing optimisation or avoidance or redundant computation, it would take more time per split. It this regard, it is an approximation. However, other gradient boosting implementations also use a proxy cost functions to evaluate the splits, and I do not know whether XGBoost approximation is quicker in this regards than the others.&lt;/p&gt;&#xA;" OwnerUserId="15501" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-03T09:19:28.203" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10946" PostTypeId="1" CreationDate="2016-03-29T18:57:08.210" Score="-1" ViewCount="334" Body="&lt;p&gt;For example, if I create two tables, both contain multiple kinds of data: numeric (integer), numeric (continuous), and factor (character) like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = c('red','yellow','blue','blue','red','blue','yellow','blue','red','red')&#xA;b = c(1,2,3,2,3,2,2,2,1,2)&#xA;c = c(1023,432.34,775.33,342.78,3241.45,1029,938,837.32, 739.43,649)&#xA;d = c(17313,23523.32,89790,98790.45,98792,498792,23984.87,29739,69198,917638.48)&#xA;data.t1 = data.frame(a=a, b=b, c=c, response=d)&#xA;&#xA;a = c('blue','blue','yellow','blue','red','red','yellow','red','blue','red')&#xA;b = c(2,1,1,3,2,1,3,1,3,2)&#xA;c = c(1775.33,8342.78,649,241.45,29,938,1083,4432.34, 3837.32, 2739.43)&#xA;d = c(27313,2423.32,8990,18790.45,27792,4982,2384.87,9739,6198,91638.48)&#xA;data.t2 = data.frame(a=a,b=b,c=c, response = d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I will have two tables like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data.t1&#xA;        a b       c  response&#xA;1     red 1 1023.00  17313.00&#xA;2  yellow 2  432.34  23523.32&#xA;3    blue 3  775.33  89790.00&#xA;4    blue 2  342.78  98790.45&#xA;5     red 3 3241.45  98792.00&#xA;6    blue 2 1029.00 498792.00&#xA;7  yellow 2  938.00  23984.87&#xA;8    blue 2  837.32  29739.00&#xA;9     red 1  739.43  69198.00&#xA;10    red 2  649.00 917638.48&#xA;&amp;gt; data.t2&#xA;        a b       c response&#xA;1    blue 2 1775.33 27313.00&#xA;2    blue 1 8342.78  2423.32&#xA;3  yellow 1  649.00  8990.00&#xA;4    blue 3  241.45 18790.45&#xA;5     red 2   29.00 27792.00&#xA;6     red 1  938.00  4982.00&#xA;7  yellow 3 1083.00  2384.87&#xA;8     red 1 4432.34  9739.00&#xA;9    blue 3 3837.32  6198.00&#xA;10    red 2 2739.43 91638.48&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;data.t1 is the data collected at time 1, and data.t2 is data collected at time 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so I want to know, which are the key parameter(s) that contributed the most to the change of the &quot;response&quot; var (or vars, if I can scale it that would be nice as well) from data.t1 to data.t2. for example, if the change in variable a &amp;amp; b contributes most to the increasing (or decreasing) trend in the response var from table at t1 to table at t2, i'd like the code to return var a &amp;amp; b. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;note:&lt;/strong&gt; &#xA;the data i created are completely random, so may not actually display a &quot;trending&quot; but this is more just for my illustration purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;added note:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rows correspond to each other here; ie row 6 of data.t1 corresponds to row 6 of data.t2, and I am interested in the change in response in that row &quot;caused by&quot; in some sense the change in the a, b and c variables&lt;/p&gt;&#xA;" OwnerUserId="17423" LastEditorUserId="16284" LastEditDate="2016-04-21T13:56:26.097" LastActivityDate="2016-04-21T13:56:26.097" Title="R how to find the top key parameters contribute to the response var change in two tables?" Tags="|r|glm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10948" PostTypeId="2" ParentId="10944" CreationDate="2016-03-29T19:07:20.880" Score="1" Body="&lt;p&gt;You could learn about &lt;a href=&quot;https://en.wikipedia.org/wiki/Word_embedding&quot; rel=&quot;nofollow&quot;&gt;word embeddings&lt;/a&gt;. These will provide you a natural path to topic models, and many other NLP tasks. Look up &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is better not to learn two new subjects (NLP and ML) at once. I would start with ML on its own; take &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;this class&lt;/a&gt; perhaps. A good place to start is &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;linear regression&lt;/a&gt;, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Binary_classification&quot; rel=&quot;nofollow&quot;&gt;binary classification&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-03-29T19:26:54.773" LastActivityDate="2016-03-29T19:26:54.773" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10950" PostTypeId="1" CreationDate="2016-03-29T22:46:48.213" Score="4" ViewCount="1679" Body="&lt;p&gt;I'd like to test a new algorithm for link prediction on graphs. I'm interested in  both weighted/unweighted and directed/undirected graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for link prediction.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastActivityDate="2016-06-28T07:50:36.380" Title="Link prediction benchmark datasets" Tags="|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10951" PostTypeId="1" CreationDate="2016-03-30T04:42:40.557" Score="1" ViewCount="2033" Body="&lt;p&gt;I have a description of a product say mobile in this form -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;5.5&quot; (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness &amp;amp; contrast ratio of 1000:1first phone with theatermax technology and dolby atmos surround sound over speakers13mp f2.2 primary camera with 5p lens element, dual led dual tone flash, 5mp front camera with 4p lens elementandroid v5.1 os with 1.3 ghz mediatek 6753 64-bit octa core processor, arm mali t720 gpu, 3 gb ddr3 ram, 16gb internal memory (expandable up to 128gb) and dual micro sim dual standby (4g+4g)3300mah lithium-ion battery with fast charger 2.0 providing talk-time of 29 hours&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to extract the important elements of the description and map it to keys like -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Display size - 5.5&lt;/li&gt;&#xA;&lt;li&gt;Display Resolution - 1920 x 1080 &lt;/li&gt;&#xA;&lt;li&gt;Sim type - Dual Sim&lt;/li&gt;&#xA;&lt;li&gt;Ram - 3gb &lt;/li&gt;&#xA;&lt;li&gt;Storage - 16gb.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have a dictionary that maps key values pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example - {Ram:[&quot;3gb&quot;,&quot;3 GB&quot;,&quot;4 gb&quot;],&quot;Sim type&quot;:[&quot;Dual&quot;,&quot;Micro&quot;,&quot;Nano&quot;]}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest how to do this. I am coding in python. How to proceed it with NLTK. Should I use ngrams??? Any useful examples that uses self tagging will be helpful. All examples i see uses already defined tags like noun,person,organisation etc. I want to give my own dictionary as corpus and train the model so that it extracts the relevant key values pairs from the text.&lt;/p&gt;&#xA;" OwnerUserId="13046" LastEditorUserId="13046" LastEditDate="2016-03-30T04:53:04.750" LastActivityDate="2017-09-26T13:35:33.773" Title="Extracting words belonging to a key from the text" Tags="|python|nlp|text-mining|nltk|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10952" PostTypeId="2" ParentId="2537" CreationDate="2016-03-30T06:58:54.947" Score="2" Body="&lt;p&gt;sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&quot; rel=&quot;nofollow&quot;&gt;GradientBoostingClassifier&lt;/a&gt; / &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html&quot; rel=&quot;nofollow&quot;&gt;GradientBoostingRegressor&lt;/a&gt; have a &lt;em&gt;max_features&lt;/em&gt; parameter and &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&quot; rel=&quot;nofollow&quot;&gt;XGBoost&lt;/a&gt; has &lt;em&gt;colsample_bylevel&lt;/em&gt; and &lt;em&gt;colsample_bytree&lt;/em&gt; parameters that control how many features are sampled for each tree / split.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-03-30T06:58:54.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10953" PostTypeId="2" ParentId="10950" CreationDate="2016-03-30T07:23:06.290" Score="1" Body="&lt;p&gt;A cursory Google search gave me:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://noesis.ikor.org/datasets/link-prediction&quot; rel=&quot;nofollow&quot;&gt;http://noesis.ikor.org/datasets/link-prediction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ial.eecs.ucf.edu/travian.php&quot; rel=&quot;nofollow&quot;&gt;http://ial.eecs.ucf.edu/travian.php&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://konect.uni-koblenz.de/networks/&quot; rel=&quot;nofollow&quot;&gt;http://konect.uni-koblenz.de/networks/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another way would be to create your own dataset by downloading say Github user's graph, defining a network by adding edges  between user A and user B if A follows B and so on. Then you can take snapshots of this graph at different time instances and see if you can use last &lt;code&gt;t&lt;/code&gt; snapshots to predict the edges in the &lt;code&gt;t+1&lt;/code&gt;th snapshot. Of course that would be a lot more extra effort and the links I posted may suffice. Please let me know if it helps.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-03-30T07:23:06.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10954" PostTypeId="1" AcceptedAnswerId="10990" CreationDate="2016-03-30T07:38:19.027" Score="3" ViewCount="131" Body="&lt;p&gt;I am looking for ideas on how to proceed with a situation. I have historical data of appointments for many users. I can (easily) predict their future behaviour (whether the next appointment will be positive or not)/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now need to build profiles, i.e., classify a user as &quot;good&quot; or &quot;bad&quot; user. So, I won't be predicting future appointments, but just say &quot;this user is a good/bad user&quot;, and then compare with, e.g., a monthly behaviour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas where I could look for information on how to proceed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize for my vague first question. I will try to be more clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have ~200k users, with their doctor appointment history. As demographic, I only have their age and gender. The rest of the variables relate to their appointment: time of the appointment, which doctor (service), day of the appointment, and so on. I also have whether they went or not to the appointment (show / no show).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assignment consist on classifying the users as &quot;good&quot; or &quot;bad&quot;, i.e., whether they will go or not to the appointment. I do not have to predict if they will go to the next appointment, just to have a list of the users classification. By doing this, if user A, whom I classified as &quot;good&quot;, calls for an appointment, I know he is &quot;good&quot; and he will make it to the appointment. I do not to take any action on him.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, I do not have to predict future appointments, or the behavior of a new user, just to classify the existing ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope now it is a bit more clear. If not, let me know. Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know perhaps the information I am giving is a bit vague, but I didn't know if to go into full detail or not. If you want me to share more info, just let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="15568" LastEditorUserId="15568" LastEditDate="2016-03-31T06:22:18.763" LastActivityDate="2016-04-01T11:15:34.903" Title="Building Customers/Patient Profiles" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10955" PostTypeId="2" ParentId="10951" CreationDate="2016-03-30T09:23:31.287" Score="1" Body="&lt;p&gt;You have two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use regular expressions to extract features of interest based on your dictionary. You may have mixed results, since, for example, dual sim cards will be micro or nano, so you will get two types from single description. However, that just says, that the dictionary isn't well structured.&lt;/li&gt;&#xA;&lt;li&gt;Train your own NER (named entity recognition) system. For that, first, you will need a data set which will consist of your descriptions &lt;strong&gt;and marked entities&lt;/strong&gt;. This would require manual labour, either yours or paid one one some crowd sourcing platform. When you will have a marked-up dataset, you could use NLTK to train a NER. Markup may be done in any way familiar to you, but, generally speaking, xml-like markup is the best choice since there are many ways to train a NER, and most of frameworks know how to parse xml-like tags:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example markup:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&amp;lt;screen_size&gt;5.5&quot;&amp;lt;/screen_size&gt;&lt;/strong&gt; (13.97cm) full hd &lt;strong&gt;&amp;lt;resolution&gt;1920 x 1080&amp;lt;/resolution&gt;&lt;/strong&gt; pixels ips screen gorilla glass display etc.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-03-30T09:23:31.287" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10956" PostTypeId="2" ParentId="10899" CreationDate="2016-03-30T10:21:17.850" Score="3" Body="&lt;p&gt;Your training set should be true representation of the entire population which is not true in your case. The levels in your train data set's media column has 4 factor levels which is 1 level less than the test data set's media column factor levels. Assuming you are using R, you can fix it with below code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;levels(TrainAquifier.media) &amp;lt;- levels(TestAquifier.media)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can find answer to similar question in stackoverflow &lt;a href=&quot;https://stackoverflow.com/questions/24829674/r-random-forest-error-type-of-predictors-in-new-data-do-not-match&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8465" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-03-30T10:28:07.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10957" PostTypeId="1" CreationDate="2016-03-30T11:14:33.680" Score="1" ViewCount="429" Body="&lt;p&gt;Trying to compare xts objects, wanted to ensure that Ret and Ret2 columns are the same. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    head(rts.fut)&#xA;            Open  High   Low Close Close.prev        Ret       Ret2&#xA;2015-01-05 78450 78450 72470 73100         NA         NA         NA&#xA;2015-01-06 73230 74900 71000 72760      73100 -0.4651163 -0.4651163&#xA;2015-01-08 72800 81380 72800 80190      72760 10.2116548 10.2116548&#xA;2015-01-09 80190 81050 74450 77330      80190 -3.5665295 -3.5665295&#xA;2015-01-12 77140 77140 73180 74490      77330 -3.6725721 -3.6725721&#xA;2015-01-13 74230 74230 70870 73470      74490 -1.3693113 -1.3693113&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt; head(rts.fut$Ret)==head(rts.fut$Ret2)&#xA;             Ret&#xA;2015-01-05    NA&#xA;2015-01-06 FALSE&#xA;2015-01-08 FALSE&#xA;2015-01-09 FALSE&#xA;2015-01-12 FALSE&#xA;2015-01-13 FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help will be highly appreciated,&lt;br&gt;&#xA;Thanks, &lt;/p&gt;&#xA;" OwnerUserId="17119" LastActivityDate="2017-05-25T00:25:44.560" Title="Xts object comparison" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10958" PostTypeId="2" ParentId="2668" CreationDate="2016-03-30T12:34:21.797" Score="0" Body="&lt;p&gt;Scala is suited for both large and small data science applications. Consider &lt;a href=&quot;http://mandar2812.github.io/DynaML&quot; rel=&quot;nofollow&quot;&gt;DynaML&lt;/a&gt; if you are interested to try a machine learning library which integrates well with Apache Spark. It is still in its infancy so to speak in terms of number of models offered, but it makes up for it by a broad and flexible machine learning API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take a look at some sample use cases consider (more where that came from)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mandar2812.github.io/posts/gp-abott-power-plant/&quot; rel=&quot;nofollow&quot;&gt;System Identification - Abott Power Plant&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: I am the author of DynaML&lt;/p&gt;&#xA;" OwnerUserId="17446" LastEditorUserId="17446" LastEditDate="2016-03-30T13:53:22.133" LastActivityDate="2016-03-30T13:53:22.133" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10959" PostTypeId="1" CreationDate="2016-03-30T12:38:48.880" Score="4" ViewCount="1400" Body="&lt;p&gt;To my understanding, a learning method is used to train a neural network on a dataset by training it's outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aware that the same neural network can be run backwards to generate data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to learn how to reverse a neural network's algorithm to generate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://www.youtube.com/watch?feature=player_detailpage&amp;amp;v=AyzOUbkUf3M#t=1395&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; video.&lt;/p&gt;&#xA;" OwnerUserId="17444" LastActivityDate="2017-01-10T23:08:03.627" Title="How do I produce generative neural networks?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10960" PostTypeId="1" AcceptedAnswerId="11162" CreationDate="2016-03-30T14:24:51.723" Score="1" ViewCount="83" Body="&lt;p&gt;I've got rather expansive development experience, but I'm new to data science. I have been trying to familiarize myself with the main concepts and deciding between R and Python to put my time into. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know you can write Python in R and vice versa, but since data visualization in a beautiful manner would be a very high priority for me in the future and both Python and R use additional packages and libraries for data representation, I'm not entirely sure whether it's possible to use the additional libraries when integrating code into one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would really appreciate it if someone could give me some advice regarding this and also provide me with some general tips about the best approaches, practices and tools when it comes to data visualization.&lt;/p&gt;&#xA;" OwnerUserId="17450" LastActivityDate="2016-04-12T12:24:35.793" Title="Need some tips regarding starting out with the field's specific programming languages, with a heavy focus on data visualization" Tags="|r|python|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10961" PostTypeId="2" ParentId="10957" CreationDate="2016-03-30T16:27:02.513" Score="1" Body="&lt;p&gt;This is not necessary related to &lt;code&gt;xts&lt;/code&gt;. You are comparing to floating point values. As pointer out in this &lt;a href=&quot;http://floating-point-gui.de/errors/comparison/&quot; rel=&quot;nofollow&quot;&gt;excellent resource&lt;/a&gt; these type of comparisons are dangerous. Best way to code it if you do not have &lt;code&gt;Inf&lt;/code&gt;, &lt;code&gt;-Inf&lt;/code&gt; or &lt;code&gt;NaN&lt;/code&gt; is by doing,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(abs(rts.fut$Ret - rts.fut$Ret2) / pmin(abs(rts.fut$Ret) + abs(rts.fut$Ret2),&#xA;  .Machine$double.eps)) &amp;lt; .Machine$double.eps&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-03-30T16:27:02.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10963" PostTypeId="2" ParentId="10951" CreationDate="2016-03-30T17:39:43.693" Score="1" Body="&lt;p&gt;I think using regexp after some pre-processing (removing stopwords, symbols or lowercasing, etc) is a good solution here. Training a custom NER is too much work.&lt;/p&gt;&#xA;" OwnerUserId="11007" LastEditorUserId="11007" LastEditDate="2017-09-26T13:35:33.773" LastActivityDate="2017-09-26T13:35:33.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10965" PostTypeId="2" ParentId="10679" CreationDate="2016-03-30T21:02:01.090" Score="5" Body="&lt;p&gt;You ideally want to use Reinforcement Learning in situations where there is delayed feedback and stochastic transitions in the environment. Although you could potentially apply RL, in your case, you might be better off with a Sequence to Sequence learning framework (&lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&lt;/a&gt;) since you have access to the entire sentence and there is no stochasticity involved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the topic of RL with Information Extraction, this might be of interest:&#xA;Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning (&lt;a href=&quot;http://arxiv.org/abs/1603.07954&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/abs/1603.07954&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="17462" LastActivityDate="2016-03-30T21:02:01.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10967" PostTypeId="1" AcceptedAnswerId="11009" CreationDate="2016-03-31T02:27:58.373" Score="4" ViewCount="2357" Body="&lt;p&gt;I have two questions on how to produce impulse responses using R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Impulse responses to a negative shock in the independent variable (money supply)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Impulse responses at 2 standard deviations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code I used to generate the impulse responses to a positive shock at 1 standard deviation is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m1 &amp;lt;- read.csv(&quot;m1.csv&quot;, header=T)&#xA;&#xA;m1&#xA;&#xA;varm1 &amp;lt;- VAR(m1, p=8, type=&quot;cons&quot;)&#xA;&#xA;irfm1 &amp;lt;- irf(varm1, impulse=&quot;m1&quot;, response= c(&quot;gdp&quot;), boot = FALSE)&#xA;&#xA;plot(irfm1)&#xA;&#xA;irfm1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17469" LastEditorUserId="11097" LastEditDate="2016-03-31T02:56:47.770" LastActivityDate="2022-02-08T07:42:28.047" Title="Impulse Response Function - Negative Shocks on R" Tags="|r|statistics|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10968" PostTypeId="2" ParentId="10931" CreationDate="2016-03-31T02:31:16.170" Score="0" Body="&lt;p&gt;There are a lot of companies, both startups and fortune 500 (such as siemens and GE), working on &quot;smart traffic lights&quot;. It might be worth trying to hunt down where they get their data from.&lt;/p&gt;&#xA;" OwnerUserId="17464" LastActivityDate="2016-03-31T02:31:16.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10969" PostTypeId="2" ParentId="10876" CreationDate="2016-03-31T03:49:45.470" Score="8" Body="&lt;p&gt;You included an important detail in your comment that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With balanced dataset (50% positives and 50% negatives) I have similar results.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think the issue is in the activation function of your neural network. If you used a symmetric activation function in your neural network then the FP and FN rates should be roughly the same in the balanced dataset case. You can test whether this is the source of your issue by flipping the signs of the labels on your data (i.e. relabel positive as negative and negative as positive and see what happens). If the FP and FN rates stay the same even when you switch what the dataset labels positive and negative, then this shows that the large false positive rate is not coming from the data but rather from the probability model itself, in which case check your activation function to see if they are biased towards labelling something as positive. &lt;/p&gt;&#xA;" OwnerDisplayName="user17471" LastActivityDate="2016-03-31T03:49:45.470" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10970" PostTypeId="1" AcceptedAnswerId="10971" CreationDate="2016-03-31T05:26:07.507" Score="4" ViewCount="2770" Body="&lt;p&gt;I have a large data set(460 Mb) which has a column - Log with 386551 rows.&#xA;I wish to use clustering and N-Gram approach to form word cloud. My code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(readr)&#xA;AMC &amp;lt;- read_csv(&quot;All Tickets.csv&quot;)&#xA;Desc &amp;lt;- AMC[,4]&#xA;&#xA;#Very large data hence breaking it down for creating corpus&#xA;#DataframeSource has been used insted of VectorSource is to be able to       handle the data&#xA;&#xA;library(tm)&#xA;docs_new &amp;lt;- data.frame(Desc)&#xA;&#xA;test1 &amp;lt;- docs_new[1:100000,]&#xA;test2 &amp;lt;- docs_new[100001:200000,]&#xA;test3 &amp;lt;- docs_new[200001:300000,]&#xA;test4 &amp;lt;- docs_new[300001:386551,]&#xA;test1 &amp;lt;- data.frame(test1)&#xA;test1 &amp;lt;- Corpus(DataframeSource(test1))&#xA;test2 &amp;lt;- data.frame(test2)&#xA;test2 &amp;lt;- Corpus(DataframeSource(test2))&#xA;test3 &amp;lt;- data.frame(test3)&#xA;test3 &amp;lt;- Corpus(DataframeSource(test3))&#xA;test4 &amp;lt;- data.frame(test4)&#xA;test4 &amp;lt;- Corpus(DataframeSource(test4))&#xA;&#xA;# attach all the corpus&#xA;docs_new &amp;lt;- c(test1,test2,test3,test4)&#xA;&#xA;docs_new &amp;lt;- tm_map(docs_new, tolower)&#xA;docs_new &amp;lt;- tm_map(docs_new, removePunctuation)&#xA;docs_new &amp;lt;- tm_map(docs_new, removeNumbers)&#xA;docs_new &amp;lt;- tm_map(docs_new, removeWords, stopwords(kind = &quot;en&quot;))&#xA;docs_new &amp;lt;- tm_map(docs_new, stripWhitespace)&#xA;docs_new &amp;lt;- tm_map(docs_new, stemDocument)&#xA;docs_new &amp;lt;- tm_map(docs_new, PlainTextDocument)&#xA;&#xA;#tokenizer for tdm with ngrams&#xA;library(RWeka)&#xA;options(mc.cores=1) &#xA;BigramTokenizer &amp;lt;- function(x) NGramTokenizer(x, Weka_control(min = 2, max =2))&#xA;tdm &amp;lt;- TermDocumentMatrix(docs_new, control = list(tokenize = BigramTokenizer))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is giving me results as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TermDocumentMatrix (terms: 1874071, documents: 386551)&amp;gt;&amp;gt;&#xA;Non-/sparse entries: 17313767/724406705354&#xA;Sparsity           : 100%&#xA;Maximal term length: 733&#xA;Weighting          : term frequency (tf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I then converted it to dgMatrix using following :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;Matrix&quot;)&#xA;mat &amp;lt;- sparseMatrix(i=tdm$i, j=tdm$j, x=tdm$v, dims=c(tdm$nrow, tdm$ncol))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While trying to use the following I am getting memory size error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;removeSparseTerms(tdm, 0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please suggest further as I am new to Text Analytics.&lt;/p&gt;&#xA;" OwnerUserId="17404" LastActivityDate="2018-11-21T02:46:10.730" Title="Text Mining on Large Dataset" Tags="|r|bigdata|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10971" PostTypeId="2" ParentId="10970" CreationDate="2016-03-31T06:00:06.177" Score="2" Body="&lt;p&gt;You are using R, and everything that you are currently working on will be held in memory hence the error. You just don't have enough of it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be better off creating the frequencies of the terms from the original splits as opposed to creating one big file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then after you have the frequencies adding them all together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I use this code to create my wordclouds.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;##Clean code for single column of a dataframe, in this case named Text&#xA;  x = alltweets&#xA;  tweets.text &amp;lt;- x$text&#xA;  tweets.text.cleaned &amp;lt;- gsub(&quot;@\\w+ *#&quot;, &quot;&quot;, tweets.text)&#xA;  tweets.text.cleaned &amp;lt;- gsub(&quot;(f|ht)tp(s?)://(.*)[.][a-z]+&quot;, &quot;&quot;, tweets.text.cleaned)&#xA;  tweets.text.cleaned &amp;lt;- gsub(&quot;[^0-9A-Za-z///' ]&quot;, &quot;&quot;, tweets.text.cleaned)&#xA;  tweets.text.corpus &amp;lt;- Corpus(VectorSource(tweets.text.cleaned))&#xA;  tweets.text.final &amp;lt;- tm_map (tweets.text.corpus, removePunctuation, mc.cores=1)&#xA;  tweets.text.final2 &amp;lt;- tm_map (tweets.text.final, content_transformer(tolower), mc.cores=1)&#xA;  tweets.text.final2 &amp;lt;- tm_map(tweets.text.final2, removeNumbers, mc.cores=1)&#xA;  tweets.text.final2 &amp;lt;- tm_map(tweets.text.final2, removePunctuation, mc.cores=1)&#xA;  tweets.text.final2 &amp;lt;- tm_map(tweets.text.final2,removeWords, stopwords(&quot;English&quot;), mc.cores=1)&#xA;  tweets.text.final2 = tm_map(tweets.text.final2, removeWords, c(&quot;amp&quot;, &quot;&amp;amp;&quot;))&#xA;#create corpus&#xA;   housing.tweets.corpus &amp;lt;- Corpus(VectorSource(tweets.text))&#xA;&#xA;#clean up by removing stop words&#xA;   housing.tweets.corpus &amp;lt;- tm_map(housing.tweets.corpus, function(x)removeWords(x,stopwords()))&#xA;&#xA;#install wordcloud if not already installed&#xA;   install.packages(&quot;wordcloud&quot;)&#xA;   library(&quot;wordcloud&quot;)&#xA;&#xA;#generate wordcloud&#xA;   wordcloud(housing.tweets.corpus,min.freq = 2, scale=c(7,0.5),colors=brewer.pal(8, &quot;Dark2&quot;),  random.color= TRUE, random.order = FALSE, max.words = 500)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17473" LastActivityDate="2016-03-31T06:00:06.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10972" PostTypeId="1" CreationDate="2016-03-31T07:07:29.200" Score="3" ViewCount="7845" Body="&lt;p&gt;I've thought about some, like predicting Concrete's strength by it's composition, but would like some ideas on the theme. One other thing I was thinking was traffic, how would I use machine learning to predict something related to streets and roads... Maybe something related to hydraulics on a building. I'm really beginning on machine learning so I'm not so good at knowing how can I use it. Although it's amazing. Thank's a lot guys.&lt;/p&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2016-04-28T09:42:18.657" Title="Would like to know some Machine Learning applications for Civil Engineering" Tags="|machine-learning|" AnswerCount="3" CommentCount="0" ClosedDate="2016-04-28T18:57:15.003" ContentLicense="CC BY-SA 3.0" />
  <row Id="10974" PostTypeId="1" CreationDate="2016-03-31T12:42:21.497" Score="4" ViewCount="378" Body="&lt;p&gt;Some methods related to manifold-learning are commonly stated as &lt;strong&gt;good-for-visualization&lt;/strong&gt;, such as T-SNE and self-organizing-maps (SOM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that when referring specifically to &quot;visualization&quot; means that the non-linear dimensionality reduction can provide good insights of data in its low-dimensional projection, but that most commonly this low-dimensional projection cannot be used in machine learning algorithms, since some of information of the high-dimensional structure is lost (roughly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, and here the question, If &quot;clusters&quot; are being observed in the visualization is it acceptable to apply a clustering algorithm to the low-dimensional transformed data and analyze the clusters or groups separately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I'm applying T-SNE to rather-high dimensional data (40 features) and obtaining this representation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/X8IgS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X8IgS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disregarding the colors you observe in the picture, I would like to apply a clustering algorithm and separate the data from the found clusters (let's say 6 or 7 clusters), and then analyze the characteristics of each cluster using the high-dimensional representation of each point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is in synthesis: using the low dimensional to find clusters, and analyze (exploration) each cluster separately using the high dimensional representation. If I'm not able to do this, I don't see the actual point of visualizing in the low-dimensional space, in a practical sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand T-SNE preserves well local structures and less accurately global structures, is this a drawback to do why I want? Is this low-dim clustering approach more suitable for other manifold learning methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;:&#xA;Probably a more direct way of asking this is: Can I use observed clusters in low dimensional representation to label or tag examples, and use these labels for discrimination using the original high dimensional representation?&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="31513" LastEditDate="2017-04-24T17:38:14.033" LastActivityDate="2017-04-24T17:38:14.033" Title="Can I apply Clustering algorithms to the result of Manifold Visualization Methods?" Tags="|clustering|visualization|dimensionality-reduction|tsne|manifold|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10976" PostTypeId="1" CreationDate="2016-03-31T13:56:33.917" Score="2" ViewCount="95" Body="&lt;p&gt;I have a dataset that includes all of the building permits that were issued for homes within a city. I have the data in a &quot;snapshot&quot; from a year ago (and potentially a few others, older snapshots) and today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Included in the data are things like home age and other attributes about the home. What I'm hoping to do is figure out what characteristics predict a future permit. For this, I was going to compare the year ago snapshot with the present snapshot to see which homes ended up getting permits (e.g., to remodel a bathroom) and determine if there's any correlation between characteristics that predict these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are the following: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;What form would I need the data to do this?&lt;/strong&gt; Assume I have all the characteristics in different &quot;columns&quot;. Most of it will be categorical or binary (true/false). Dates will obviously be in date format. I have the ability to add additional flags in the past or current data that might indicate, for example, a new plumbing permit was issued.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;What analysis would work best for this and any recommendations for tools/platforms to run this?&lt;/strong&gt; Assuming hundreds of thousands of entries would be available to analyze.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17486" LastEditorUserId="26596" LastEditDate="2016-12-01T13:31:29.417" LastActivityDate="2016-12-01T13:31:29.417" Title="what predictive analysis will work with this data set?" Tags="|predictive-modeling|random-forest|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10977" PostTypeId="1" CreationDate="2016-03-31T14:30:49.210" Score="5" ViewCount="1524" Body="&lt;p&gt;I have implemented a CNN with images as input and 101 classes as output.&#xA;I have applied mean subtraction and normalization to the input before giving it as input to the network.&lt;/p&gt;&#xA;&lt;p&gt;I have also experimented with multiple optimizers (ADAM &amp;amp; SGD) and also changed the learning rate (0.1,0.01,0,001) and also batch sizes (19,32,64,128,200).&#xA;But with all of these cases I had observed a similar pattern.&lt;/p&gt;&#xA;&lt;p&gt;I dont think the network is learning much and more importantly the accuracy oscillating between a few values.&lt;/p&gt;&#xA;&lt;p&gt;I had initially thought it was because of the batch size but I was wrong.&#xA;I have built and trained the exact same network with same inputs with caffe.  And achieved a 55% accuracy(check for working of  network model).&lt;/p&gt;&#xA;&lt;p&gt;My guess is the implementation went wrong somewhere but not sure where.&lt;/p&gt;&#xA;&lt;h1&gt;&lt;a href=&quot;http://pastebin.com/ySbaawbi&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mycode&lt;/a&gt;&lt;/h1&gt;&#xA;&lt;h1&gt;Output&lt;/h1&gt;&#xA;&lt;p&gt;This particular output is with lr=0.001 batch_size=200 and ADAM optimizer&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Iter 2000, Minibatch Loss= 12560.197266, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.025&#xA;Iter 4000, Minibatch Loss= 4587.021484, Training Accuracy= 0.00500&#xA;Testing Accuracy: 0.005&#xA;Iter 6000, Minibatch Loss= 2196.316406, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.02&#xA;Iter 8000, Minibatch Loss= 918.784119, Training Accuracy= 0.01500&#xA;Testing Accuracy: 0.0&#xA;Iter 10000, Minibatch Loss= 315.327911, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.0&#xA;Iter 12000, Minibatch Loss= 102.744026, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.01&#xA;Iter 14000, Minibatch Loss= 59.417763, Training Accuracy= 0.00500&#xA;Testing Accuracy: 0.005&#xA;Iter 16000, Minibatch Loss= 33.026432, Training Accuracy= 0.00500&#xA;Testing Accuracy: 0.02&#xA;Iter 18000, Minibatch Loss= 30.361868, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.005&#xA;Iter 20000, Minibatch Loss= 33.944790, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.02&#xA;Iter 22000, Minibatch Loss= 15.029477, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.02&#xA;Iter 24000, Minibatch Loss= 13.751340, Training Accuracy= 0.02500&#xA;Testing Accuracy: 0.02&#xA;Iter 26000, Minibatch Loss= 9.085027, Training Accuracy= 0.01500&#xA;Testing Accuracy: 0.015&#xA;Iter 28000, Minibatch Loss= 14.407433, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.02&#xA;Iter 30000, Minibatch Loss= 11.090684, Training Accuracy= 0.02500&#xA;Testing Accuracy: 0.02&#xA;Iter 32000, Minibatch Loss= 9.385777, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.03&#xA;Iter 34000, Minibatch Loss= 12.902111, Training Accuracy= 0.01500&#xA;Testing Accuracy: 0.02&#xA;Iter 36000, Minibatch Loss= 5.580408, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.005&#xA;Iter 38000, Minibatch Loss= 11.641799, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.03&#xA;Iter 40000, Minibatch Loss= 6.941562, Training Accuracy= 0.01500&#xA;Testing Accuracy: 0.02&#xA;Iter 42000, Minibatch Loss= 6.781301, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.0&#xA;Iter 44000, Minibatch Loss= 6.840708, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.01&#xA;Iter 46000, Minibatch Loss= 5.501908, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.03&#xA;Iter 48000, Minibatch Loss= 4.667253, Training Accuracy= 0.03000&#xA;Testing Accuracy: 0.015&#xA;Iter 50000, Minibatch Loss= 4.613351, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.01&#xA;Iter 52000, Minibatch Loss= 7.234328, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.01&#xA;Iter 54000, Minibatch Loss= 8.120478, Training Accuracy= 0.02500&#xA;Testing Accuracy: 0.01&#xA;Iter 56000, Minibatch Loss= 6.804164, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.01&#xA;Iter 58000, Minibatch Loss= 5.060484, Training Accuracy= 0.03000&#xA;Testing Accuracy: 0.02&#xA;Iter 60000, Minibatch Loss= 4.937673, Training Accuracy= 0.02000&#xA;Testing Accuracy: 0.015&#xA;Iter 62000, Minibatch Loss= 8.758166, Training Accuracy= 0.01000&#xA;Testing Accuracy: 0.02&#xA;Iter 64000, Minibatch Loss= 4.616434, Training Accuracy= 0.01500&#xA;Testing Accuracy: 0.025&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17487" LastEditorUserId="29169" LastEditDate="2021-02-16T15:50:37.017" LastActivityDate="2021-02-16T15:50:37.017" Title="Tensorflow oscillating Test and Train Accuracy?" Tags="|machine-learning|deep-learning|neural-network|tensorflow|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="10979" PostTypeId="2" ParentId="10972" CreationDate="2016-03-31T15:59:29.660" Score="-1" Body="&lt;p&gt;Machine Learning is not an answer for the question of Life, the Universe, and Everything or silver bullet but a very powerful tool which you should apply to a problem (and corresponding data). So, the short answer to your question is 'depends on data you have and problems you see'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, the best way would be taking any available introductory course into Data Science / Machine Learning, given you have at least some programming experience so that you could complete assignments: that would not address your question directly, but you will get your hands on some examples of data and will see how ML is used to tackle that data. After you will grow some understanding, you will see potential applications of ML when you see any particular blob of data.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-03-31T15:59:29.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10981" PostTypeId="1" AcceptedAnswerId="10984" CreationDate="2016-03-31T16:38:14.260" Score="3" ViewCount="111" Body="&lt;p&gt;For fun I want to design a convolutional neural net to recognize enemy NPCs in a first person shooter. I have captured 100 jpegs of the npcs as well as 100 jpegs of not-NPCs. I have successfully trained a really simple convNEt to identify NPCs. This was really easy because the game actually highlights the NPCs with a red marker to let humans identify them. Makes it SUPER easy for a machine learning algorthm to find them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Great , so now I can classify a screenshot of an NPC. The next step is to identify these in a data stream at 60 frames per second. We all know that the stupid little processors inside most cameras have a face detection algorithm that operates in real time. So my i7 with 2 NVIDIA gpus can do this no sweat. So now I have to grab the screen buffer, capture a screen shot, feed it to my conVnet, get the location of the NPC, and then move the mouse cursor to the center of that NPC. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any easy to follow tutorials of running a convolutional neural net on a data stream like this?&lt;/p&gt;&#xA;" OwnerUserId="14283" LastActivityDate="2018-02-10T09:15:52.867" Title="Designing a ConvNet to facilitate game playing" Tags="|neural-network|data-stream-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10982" PostTypeId="1" CreationDate="2016-03-31T17:06:02.793" Score="5" ViewCount="502" Body="&lt;p&gt;I have a dataframe 'd' like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   breaks counts.x counts.y&#xA;1     -20        1        1&#xA;2     -15        0        1&#xA;3     -10        0        5&#xA;4      -5        4       18&#xA;5       0       13       27&#xA;6       5       18       25&#xA;7      10        9       12&#xA;8      15        2        1&#xA;9      20        1        7&#xA;10     25       NA        0&#xA;11     30       NA        1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I try to melt using 'breaks'  as id variable it is giving me the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; d=melt(d,id=breaks)&#xA;Error in varnames[id.vars] : &#xA;  only 0's may be mixed with negative subscripts&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I resolve this?&#xA;I have to draw a ggplot using breaks on x axis and counts on y axis.&lt;/p&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="381" LastEditDate="2016-03-31T17:28:05.637" LastActivityDate="2018-06-22T16:47:28.310" Title="Error with negative bins while melting dataframe" Tags="|r|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="10984" PostTypeId="2" ParentId="10981" CreationDate="2016-03-31T21:01:06.177" Score="2" Body="&lt;p&gt;I've recently started using OpenCV's python implementation, and I found some good OpenCV tutorials on this website: &lt;a href=&quot;http://www.pyimagesearch.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.pyimagesearch.com/&lt;/a&gt; that I really liked. OpenCV allows you to do Haar Cascades for fast facial recognition (by default it doesn't use a convoluted neural network but an optimized implementation of Ada Boosting that evaluates frames in stages for faster processing). OpenCV converts each frame into a multidimensional numpy tensor/matrix that you can then feed into into your ML algorithm (e.g., in TensorFlow or some other library), although I think most people just use the built-in OpenCV face classifiers. In any case, I believe OpenCV can process up to 70 frames per second, so it should be fast enough for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper that invented Haar Cascades:&#xA;&lt;a href=&quot;https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The OpenCV documentation that further explains Haar Cascades:&#xA;&lt;a href=&quot;http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0&quot; rel=&quot;nofollow&quot;&gt;http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-03-31T21:39:47.107" LastActivityDate="2016-03-31T21:39:47.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10985" PostTypeId="1" CreationDate="2016-04-01T01:09:04.640" Score="2" ViewCount="103" Body="&lt;p&gt;I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've struggled to develop a concise definition of data literacy, but have determined it consists of four cornerstones: 1) &lt;strong&gt;statistical literacy&lt;/strong&gt;, 2) &lt;strong&gt;programming knowledge&lt;/strong&gt; (Python, R, etc.), 3) &lt;strong&gt;domain/ substantive knowledge&lt;/strong&gt;, and 4) &lt;strong&gt;visualization/ communication&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the latter half of my presentation, I want to provide a list of 10-12 critical questions data literate professionals ask, including:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Where does the data come from and how was it collected? How reliable or &quot;noisy&quot; is the data (e.g. measurement error, missingness, etc.)?&lt;/li&gt;&#xA;&lt;li&gt;What is the larger population we are inferring from the sample?&lt;/li&gt;&#xA;&lt;li&gt;What is the context of the data? What is the appropriate frame of reference or point of comparison?&lt;/li&gt;&#xA;&lt;li&gt;What are the inherent trade-offs (e.g. bias vs. variance) to the methods used to interpret this data?&lt;/li&gt;&#xA;&lt;li&gt;What's the level of uncertainty with a given model's predictions (e.g. standard errors, confidence intervals, etc.? How do I know when my inference is wrong? &lt;/li&gt;&#xA;&lt;li&gt;Does this sample of data help me answer the problem, puzzle, or question I’m actually interested in?&lt;/li&gt;&#xA;&lt;li&gt;Have your findings been replicated? If not, do you have the data and programming scripts so that another data scientist could attempt to replicate your findings? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any suggestions, comments, or critiques of this framework or particular questions would be much appreciated! Recommendations on good sources (videos, online, books) on data literacy for professionals would be helpful as well!&lt;/p&gt;&#xA;" OwnerUserId="17503" LastActivityDate="2016-04-01T01:09:04.640" Title="Core components of data literacy for working professionals" Tags="|bigdata|visualization|data|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10986" PostTypeId="2" ParentId="10728" CreationDate="2016-04-01T01:48:26.150" Score="3" Body="&lt;p&gt;You bring up a number of good questions here Ans. I will do my best to cover each of them in turn.  It isn't an exhaustive treatment but hopefully it helps...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. How to normalize the categories.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, assess whether your categorical variables can be considered zero variance (e.g. all records possessing one category only) or near zero variance (vast majority of records belonging to very few categories). Create a basic frequency distribution to identify this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it doesn't matter as much in Neural contexts per se, it is a good idea to consider filtering low variance variables from your model. Just be careful as eliminating near zero variance variables may have you throwing out the baby with the bath water.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've worked with Neural Nets before so you know that you need to convert categories to numeric values.  A good question to ask is whether a given categorical value is ordinal in nature (e.g. on a Likert scale of 1 to 5) and whether you want to preserve ordinality. This pushes you into an area such as that described by Pinto da Costa and Cardoso (&lt;a href=&quot;https://www.researchgate.net/publication/221112186_Classification_of_Ordinal_Data_Using_Neural_Networks&quot; rel=&quot;nofollow&quot;&gt;https://www.researchgate.net/publication/221112186_Classification_of_Ordinal_Data_Using_Neural_Networks&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. How to deal with missing data.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you are talking about missing continuous values, you will want to impute these numeric values based collectively on the values that are present across the entire attribute.  There are a number of approaches to use here but something important to keep in mind is dispersion.  In a basic sense, if your attribute is skewed by outliers, you will want to steer clear of a mean-based calculation and go with a median-based approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. How to deal with the fact that 'Defaults' are ~5% of the data.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't an immediate concern given that you have 2500 examples of what denotes a 'Default' (in a 50,000 dataset). A sane approach here would be to ensure that you are using a K-fold cross validation scheme (say with 10 folds) to ensure that you are truly randomizing training vs test.  This will help protect you against overfitting.  Again, this is pretty high level guidance, but it is prudent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extra Credit:&lt;/strong&gt;&#xA;I haven't gotten into other standard tactics such as normalization of your continuous attributes, but you may want to get up to speed there to better generalize your model (while also developing a deeper understanding of the dynamics at play in the data). This could point you to trying out other algos, etc.&lt;/p&gt;&#xA;" OwnerUserId="16827" LastActivityDate="2016-04-01T01:48:26.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10987" PostTypeId="1" CreationDate="2016-04-01T04:55:13.103" Score="1" ViewCount="372" Body="&lt;p&gt;I apologize that this has been asked and I feel that it may be obvious, but I am wondering exactly what the meaning of the numerical value below from clusterdump:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Top Terms:&#xA;   monkey       =&amp;gt;  0.8170868432876803&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I believe that to the be center of the centroid.  But if the term vectors were created with term frequencies, could one interpret this as the average occurrence of the &quot;monkey&quot; in the documents that are considered part of the cluster? In this case, &quot;monkey&quot; would appear in 82% of the docs in that cluster &lt;em&gt;or more likely that the average count of monkey is .82&lt;/em&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking further, I see words like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Top Terms: &#xA;     zebra    =&gt;   3.432595573440644&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is best to interpret this as the average count of &quot;zebra&quot; in the set of docs...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And given the radius values, one could consider that the range of percentages of &quot;monkey&quot;?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mahout seq2sparse -i out/sequenced \&#xA;    -o out/sparse-kmeans -wt TF --maxDFPercent 100 --namedVector&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mahout kmeans \&#xA;    -i out/sparse-kmeans/tf-vectors/ \&#xA;    -c out/kmeans-clusters \&#xA;    -o out/kmeans \&#xA;    -dm org.apache.mahout.common.distance.CosineDistanceMeasure \&#xA;    -x 10 -k $i -ow --clustering &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When one uses tf-idf weighting, it may be best to normalize the output weights by creating a proportion of evidence via Wi=Wi/sum(W) &#xA;Is that a good idea? (Some Python LDA libs do this.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://mahout.apache.org/users/clustering/cluster-dumper.html&quot; rel=&quot;nofollow&quot;&gt;https://mahout.apache.org/users/clustering/cluster-dumper.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8650" LastEditorUserId="8650" LastEditDate="2016-04-01T13:07:42.877" LastActivityDate="2016-04-01T13:07:42.877" Title="mahout clusterdump top terms meaning" Tags="|k-means|apache-mahout|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="10989" PostTypeId="2" ParentId="10982" CreationDate="2016-04-01T08:40:29.850" Score="1" Body="&lt;p&gt;I am sorry, is this the kind of ggplot you want to obtain ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jR6lN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jR6lN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes this is the code behind it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape)&#xA;library(ggplot2)&#xA;&#xA;d &amp;lt;- data.frame(breaks = c(-20, -15, -10, -5,  0, 5, 10, 15, 20, 25, 30),&#xA;                counts.x  = c(1, 0, 0 ,4 , 13, 18, 9, 2, 1, NA, NA),&#xA;                counts.y = c(1, 1, 5, 18, 27, 25, 12, 1, 7 , 0, 1))&#xA;&#xA;d &amp;lt;- melt(d, id = &quot;breaks&quot;)&#xA;&#xA;ggplot(d, aes(x = breaks, y = value)) + geom_point(aes(colour = variable)) +&#xA;       labs(title = &quot;Bins vs. Counts&quot;, x = &quot;Bins&quot;, y = &quot;Counts&quot;) +&#xA;       theme(plot.title = element_text(face = &quot;bold&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user16577" LastActivityDate="2016-04-01T08:40:29.850" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10990" PostTypeId="2" ParentId="10954" CreationDate="2016-04-01T11:15:34.903" Score="1" Body="&lt;p&gt;Maybe I'm being a bit too simplistic, but I would build a set of &lt;strong&gt;training&lt;/strong&gt; data that looks like this (Good=1 means patient showed up for appt and is &lt;strong&gt;good&lt;/strong&gt; by your definition, 0 = &lt;strong&gt;bad&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Recid, PatId, SurgeryId, DrId, DateAppt, TimeAppt, Gender, Age, Good&#xA;1, 1, 100, 10, 01jan16, 10:30, M, 31, 1&#xA;2, 1, 100, 12, 05jan16, 15:20, M, 31, 1&#xA;3, 1, 100, 10, 06mar16, 11:45, M, 31, 0&#xA;4, 2, 101, 15, 02Feb16, 12:35, F, 75, 1&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would then use one of the machine learning tools in R - there are a variety of them, to train a model of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then with another set of data, I would test the model you have just built to see how correct it is. If you don't have a second set of data, then randomly partition your original training set and only train with half of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some further suggestions to make your model more powerful, is create additional variables with the following information&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a Flag to indicate if a person has missed a previous appointment with any doctor.&lt;/li&gt;&#xA;&lt;li&gt;a Flag to indicate if a person has missed a previous appointment with the particular doctor they are going to be visiting - I know from person experience there are some doctors in my surgery that I have a preference to see.&lt;/li&gt;&#xA;&lt;li&gt;The number of days since the last appointment&lt;/li&gt;&#xA;&lt;li&gt;Day of week of appointment&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Sounds like a nice dataset to be working with!&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-01T11:15:34.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10991" PostTypeId="2" ParentId="8838" CreationDate="2016-04-01T11:56:31.933" Score="2" Body="&lt;p&gt;The &lt;code&gt;trim()&lt;/code&gt; function is part of the &lt;em&gt;SAS Language&lt;/em&gt;. Without the &lt;code&gt;%sysfunc()&lt;/code&gt; &lt;em&gt;macro function&lt;/em&gt;, &lt;code&gt;trim()&lt;/code&gt; can only be used within a data step or in a macro definition that gets called inside of a data step. You can't have it out in open code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if &lt;code&gt;%trim()&lt;/code&gt; existed, which it does not, it would work just fine the way you used it. But, there is no function named &lt;code&gt;%trim()&lt;/code&gt; that is defined in the SAS macro language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing what you want without &lt;code&gt;%sysfunc()&lt;/code&gt;, would have to go something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data _NULL_;&#xA;    call symput(&quot;x&quot;, trim(&quot;&amp;amp;num.&quot;));&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or if you want &lt;code&gt;x&lt;/code&gt; to be a data set variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%let expr= trim(&quot;&amp;amp;num.&quot;);&#xA;&#xA;DATA ds;&#xA;    x = &amp;amp;expr.;&#xA;RUN;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17511" LastActivityDate="2016-04-01T11:56:31.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10993" PostTypeId="2" ParentId="10946" CreationDate="2016-04-01T12:34:22.380" Score="0" Body="&lt;p&gt;Construct a new data set with differences in the numeric variables and new factors in the categorical data containing the first and second levels to represent change. Then fit a linear model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = cbind(data.t1, data.t2)&#xA;names(data)=c(&quot;a1&quot;,&quot;b1&quot;,&quot;c1&quot;,&quot;response1&quot;,&quot;a2&quot;,&quot;b2&quot;,&quot;c2&quot;,&quot;response2&quot;)&#xA;data$da = paste(data$a1,&quot;-&quot;,data$a2)&#xA;    data$db = data$b2 - data$b1&#xA;data$dc = data$c2 - data$c1&#xA;    data$dresponse = data$response2 - data$response1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So now the data columns we are interested in look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; data[,9:12]&#xA;                da db       dc dresponse&#xA;1       red - blue  1   752.33     10000&#xA;2    yellow - blue -1  7910.44    -21100&#xA;3    blue - yellow -2  -126.33    -80800&#xA;4      blue - blue  1  -101.33    -80000&#xA;5        red - red -1 -3212.45    -71000&#xA;6       blue - red -1   -91.00   -493810&#xA;7  yellow - yellow  1   145.00    -21600&#xA;8       blue - red -1  3595.02    -20000&#xA;9       red - blue  2  3097.89    -63000&#xA;10       red - red  0  2090.43   -826000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So now see how the &lt;em&gt;change&lt;/em&gt; in response from &lt;code&gt;data.t1&lt;/code&gt; to &lt;code&gt;data.t2&lt;/code&gt; depends on the &lt;em&gt;change&lt;/em&gt; in the (numeric or factor) variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; m = glm(dresponse ~ da + db + dc,data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;summary(m)&lt;/code&gt; will give you a coefficients table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;                    Estimate Std. Error t value Pr(&amp;gt;|t|)&#xA;(Intercept)        4.953e+05  8.904e+05   0.556    0.677&#xA;dablue - red      -1.395e+06  1.822e+06  -0.766    0.584&#xA;dablue - yellow   -1.713e+06  2.337e+06  -0.733    0.597&#xA;dared - blue       2.558e+05  6.113e+05   0.418    0.748&#xA;dared - red       -1.206e+06  1.209e+06  -0.998    0.501&#xA;dayellow - blue   -1.413e+06  2.666e+06  -0.530    0.690&#xA;dayellow - yellow  4.828e+04  6.543e+05   0.074    0.953&#xA;db                -5.712e+05  7.489e+05  -0.763    0.585&#xA;dc                 4.109e+01  1.541e+02   0.267    0.834&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which you can interpret in the usual way. The process you describe about choosing which variables are most influential sounds like you just want the variables with the largest point estimate, but you may want to do some standardisation. For example, in the above, a unit change in &lt;code&gt;b&lt;/code&gt; from 1 to 2, say, decreases the &lt;code&gt;dresponse&lt;/code&gt; by -570000, but a unit change in &lt;code&gt;c&lt;/code&gt; changes the response by +41.09 in the fitted model. But &lt;code&gt;c&lt;/code&gt; has a much bigger range, so perhaps you want to standardise your numeric variables to have mean 0 and sd=1 first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The factor variables have a similar interpretation - rows with &quot;yellow-yellow&quot; increase response by 48280 compared to the average change in response. I'm not sure how you standardise categorical variables, since they are just a bunch of step-changes...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistically none of these variables make any difference to the response (none of the t-values are big enough and there's no stars in the P column).&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-04-01T12:34:22.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10995" PostTypeId="1" AcceptedAnswerId="10996" CreationDate="2016-04-01T14:04:33.847" Score="8" ViewCount="1333" Body="&lt;p&gt;In a neural network architecture  can I use the sigmoid function in some layers and the tanh function in the others? Is it a good choice?&lt;/p&gt;&#xA;" OwnerUserId="17518" LastActivityDate="2016-04-01T14:13:31.737" Title="Neural Networks : Can I use both sigmoid and tanh as activation functions?" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="10996" PostTypeId="2" ParentId="10995" CreationDate="2016-04-01T14:13:31.737" Score="6" Body="&lt;p&gt;Yes you can. There are no hard rules against having different activation functions in any layer, and combining these two types should give no numerical difficulties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact it can be a good choice to have tanh in hidden layers and sigmoid on the last layer, if your goal is to predict membership of a single class or non-exclusive multiple class probabilities. The sigmoid output lends itself well to predicting an independent probability (using e.g. a logloss (aka cross-entropy) objective function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether or not it is better than using sigmoid on all layers will depend on other features of your network, the data and the problem you are trying solve. Usually the best way to find out which is better - at least in terms of accuracy - is to try out some variations and see which scores best on a cross-validation data set. In my experience, there often is a small difference between using tanh or sigmoid in the hidden layers.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-04-01T14:13:31.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10997" PostTypeId="1" CreationDate="2016-04-01T14:35:06.673" Score="16" ViewCount="5518" Body="&lt;p&gt;background:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in &lt;a href=&quot;http://arxiv.org/abs/1603.02754&quot; rel=&quot;noreferrer&quot;&gt;xgboost&lt;/a&gt; the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $g_i, h_i$ are first order and second order derivatives over our previous best estimation $\hat{y}$ (from iteration $t-1$): &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$g_i=d_{\hat{y}}l(y_i, \hat{y}) $&lt;/li&gt;&#xA;&lt;li&gt;$h_i=d^2_{\hat{y}}l(y_i, \hat{y}) $&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and $l$ is our loss function. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The question (finally):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When building $f_t$ and considering a specific feature $k$ in a specific split, they use the following heuristic to assess only some split candidates: They sort all examples by their $x_k$, pass over the sorted list and sum their second derivative $h_i$. They consider a split candidate only when the sum changes more than $\epsilon$. Why is that???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The explanation they give eludes me:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They claim we can rewrite the previous equation like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - g_i/h_i]^2 + constant$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I fail to follow the algebra - can you show why is it equal? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then they claim that &quot;this is exactly weighted squared loss with labels $gi/hi$ and weights $h_i$&quot; - a statement I agree with, but I don't understand how does it relate to the split candidate algorithm they are using...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and sorry if this is too long for this forum.  &lt;/p&gt;&#xA;" OwnerUserId="16050" LastEditorUserId="16050" LastEditDate="2016-04-03T15:34:14.403" LastActivityDate="2021-04-01T22:33:13.533" Title="Need help understanding xgboost's approximate split points proposal" Tags="|xgboost|gbm|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="10998" PostTypeId="1" CreationDate="2016-04-01T16:47:30.473" Score="2" ViewCount="646" Body="&lt;p&gt;I have a feature set with each feature having its own feature-set, and they are laid out in a SQL database with foreign key relationships. The target is defined on the top level feature-set and is a binary Win/Loss variable.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Target;&lt;br&gt;&#xA;  feature -&gt; sub-features-&gt;sub-features;&lt;br&gt;&#xA;  feature -&gt; sub-features;&lt;br&gt;     &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;All the classification algorithms expect a flat feature file as an input. What are the methods for flattening out such a structured dataset, while preserving the cross feature dependence on the sub-feature set for other features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you decide which sub-features to select? Currently I'm aggregating total Wins/Attempts and regressing (PCR) the individual sub features to see which ones have an influence over the final target, what would be a more generalized way of selecting these? And then combining these into a flattened feature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also appreciate if someone can identify the correct machine learning jargon for this exercise.&lt;/p&gt;&#xA;" OwnerUserId="17520" LastEditorUserId="17520" LastEditDate="2016-04-04T13:34:01.177" LastActivityDate="2016-04-04T13:34:01.177" Title="What are the strategies for feature engineering in a hierarchical/relational structured data?" Tags="|classification|regression|feature-engineering|hierarchical-data-format|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="10999" PostTypeId="2" ParentId="52" CreationDate="2016-04-02T06:24:27.867" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://openrefine.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenRefine&lt;/a&gt; (formerly Google Refine) is a great tool where you can interactively perform data cleanup and transformations. Your final output can be exported to a variety of standard formats (json, csv) or any custom format(eg: wiki).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exact duplicates can be handled in OpenRefine by sorting + using the &quot;blank down&quot; feature. &#xA;Also it has has approximate string (fuzzy) clustering feature where you can group similar looking duplicates such as company names or addresses. This feature alone makes it awesome.&lt;/p&gt;&#xA;" OwnerUserId="17531" LastEditorUserId="17531" LastEditDate="2017-01-23T06:27:10.313" LastActivityDate="2017-01-23T06:27:10.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11000" PostTypeId="2" ParentId="10544" CreationDate="2016-04-02T09:05:45.330" Score="1" Body="&lt;p&gt;you can use &quot;frbs&quot;, &quot;sets&quot; and fugeR packege for fuzzy logic model building.&lt;/p&gt;&#xA;" OwnerUserId="17533" LastActivityDate="2016-04-02T09:05:45.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11001" PostTypeId="2" ParentId="10932" CreationDate="2016-04-02T10:47:21.370" Score="8" Body="&lt;p&gt;I think the OP was confusing about AlphaGo with alpha-beta. In alpha-beta, you'd indeed use the policy network for helping with pruning, but not here. Again, there is no pruning as the algorithm relies on Monte-Carlo tree search (MCTS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone who thinks my answer is too long might skip to the summary section, where I state why the two networks are not redundant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the following example, I'll do some simplification to make my ideas easier to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have a position where there are two legal moves. The first move is a dead-lost for you, however, the second move gives you a winning advantage.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First move: forced loss for you&lt;/li&gt;&#xA;&lt;li&gt;Second move: forced win for you&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Evaluation network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume the evaluation network Google gives you is perfect. It can evaluate any leaf position in our example perfectly. We won't change our value network in the example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To simplify our example, let's assume our value network gives:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;-1000 for any leaf position which is a loss for you&lt;/li&gt;&#xA;&lt;li&gt;+1000 for any leaf position which is a win for you&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Policy network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume Google gives you two policy networks. The probabilities generated for our position is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Policy 1: 0.9 for move 1 and 0.1 for move 2&lt;/li&gt;&#xA;&lt;li&gt;Policy 2: 0.2 for move 1 and 0.8 for move 2.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that our first policy network gives &lt;strong&gt;incorrect&lt;/strong&gt; prior probability for our example. It gives 0.9 for move 1, which is a losing move. This is fine because not even Google could train a perfect policy network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Playing with the first policy network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AlphaGo needs to generate a simulation with Monte-Carlo, and it needs to choose move 1 or 2. Now, AlphaGo draws a uniform-distributed random variable, and it'll pick:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Move 1 if the random number is &amp;lt;= 0.9&lt;/li&gt;&#xA;&lt;li&gt;Move 2 if the random number is &gt; 0.9&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So AlphaGo is much more likely to pick the losing move to simulate (in our very first simulation). In our first simulation, we will also use the value network to get a score for the simulation. In the paper, it's:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kqpt7.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kqpt7.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This value would be -1000, because this simulation would lead to a loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, AlphaGo needs to generate the second simulation. Again, the first move would be much more likely to pick. But eventually, the second move would be pick because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Our prior probability for the second move is 0.1, not zero&lt;/li&gt;&#xA;&lt;li&gt;AlphaGo is encouraged to try moves which haven't been explored much. In the paper, this is done by this equation:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sdkQf.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sdkQf.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;N&lt;/code&gt; is the number of moves searched for the move and it's in the denominator. The more likely our first move is searched, the smaller the &lt;code&gt;u&lt;/code&gt; function is. Thus, the probability for selecting our second move improves because AlphaGo actually picks a move by this equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jZviV.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jZviV.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lw5cz.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lw5cz.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;strong&gt;key&lt;/strong&gt; equation. Please look at it carefully:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It has a term &lt;code&gt;P&lt;/code&gt; for the prior probability (given by the policy network)&lt;/li&gt;&#xA;&lt;li&gt;It has a term &lt;code&gt;Q&lt;/code&gt; for the evaluation scores (given by the value network)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, we know our second move will eventually be chosen. When it does happen, the value network gives a +1000. This will increase &lt;code&gt;Q&lt;/code&gt;, which makes the second move much more &lt;strong&gt;likely&lt;/strong&gt; be chosen in the later simulations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given enough simulations, the number of times the second move is chosen for simulation should be more than the number of times the first move is chosen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the move that AlphaGo decides to make is (quoted from the paper):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Once the search is complete, the algorithm chooses the most visited move from the root position.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Playing with the second policy network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our second policy network will need less iterations to pick move 2 because it's prior probability given by the policy network is correct in the first place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Remarks&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything here is very similar to &lt;code&gt;Bayesian&lt;/code&gt; analysis. We start off with some prior probability (given by the policy network), then we generate data to move the probability distirubtion (given by the value network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summaries&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Policy network is used to generate prior probabilities to guide what move the Monte-Carlo search should pick&lt;/li&gt;&#xA;&lt;li&gt;Value network is used to generate data to validate the policy network. If the policy network is bad, AlphaGo would need more computing resources to converge (if ever).&lt;/li&gt;&#xA;&lt;li&gt;You can think of it like Bayesian analysis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-04-05T01:28:56.440" LastActivityDate="2016-04-05T01:28:56.440" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11002" PostTypeId="1" CreationDate="2016-04-02T11:03:48.257" Score="1" ViewCount="598" Body="&lt;p&gt;If I understand correctly, the &lt;code&gt;most_similar&lt;/code&gt; function computes the cosine similarity of the vector with all other vectors and finds the closest one.  The vectors then viewed from a certain origin of (0,0,0,..0) for N dimensions (for N dimensional word vectors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there by any means that I could compute the angle of these vectors from the origin ?. &#xA;If I compute the similarity between the vector and the origin vector and take the inverse cosine, will the resultant angle be the angle of the vector on origin space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I get such an angle, on which plane does it lie as the vector is high dimensional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is that, when we try to reduce it to 2 dimensions and plot them, only magnitude of the vector is considered right, so the direction information is missing. I would like to plot it in a polar plot to view both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any such function in &lt;code&gt;gensim&lt;/code&gt;?.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-04-02T11:03:48.257" Title="Compute angle of vector in word2vec models" Tags="|dimensionality-reduction|word-embeddings|word2vec|gensim|cosine-distance|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11003" PostTypeId="1" AcceptedAnswerId="11007" CreationDate="2016-04-02T13:00:04.853" Score="2" ViewCount="513" Body="&lt;p&gt;I've got predicted results from two different types of neural networks. Now I would like to run significance testing on both of the results to prove that they do not have equal predictive accuracy. I've learnt that the only tool in the game for this is Diebold-Mariano test. What tool I can use to run this testing (Matlab? R?) &lt;/p&gt;&#xA;" OwnerUserId="17536" LastEditorUserId="17536" LastEditDate="2017-07-03T00:45:05.113" LastActivityDate="2017-07-03T00:45:05.113" Title="Using Diebold-Mariano test" Tags="|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11007" PostTypeId="2" ParentId="11003" CreationDate="2016-04-02T16:59:21.237" Score="1" Body="&lt;p&gt;So you want to do a Diebold-Mariano test eh? How about the Diebold-Mariano test &lt;code&gt;dm.test&lt;/code&gt; function in the &lt;code&gt;forecast&lt;/code&gt; package of R?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dm.test {forecast}&#xA;&#xA;Diebold-Mariano test for predictive accuracy&#xA;Package: &#xA;forecast&#xA;Version: &#xA;6.2&#xA;Description&#xA;&#xA;The Diebold-Mariano test compares the forecast accuracy of two forecast methods.&#xA;Usage&#xA;&#xA;dm.test(e1, e2, alternative=c(&quot;two.sided&quot;,&quot;less&quot;,&quot;greater&quot;),&#xA;    h=1, power=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Took me ten seconds to find)&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-04-02T16:59:21.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11009" PostTypeId="2" ParentId="10967" CreationDate="2016-04-03T02:40:17.170" Score="4" Body="&lt;p&gt;Here is a simple example that should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(vars)&#xA;data(&quot;Canada&quot;)&#xA;var.2c=VAR(Canada,p=2,type=&quot;const&quot;)&#xA;# 1 SD impulse reponse function&#xA;irf.rw.e=irf(var.2c,impulse=&quot;rw&quot;,response=c(&quot;e&quot;))&#xA;# gamma is the number of standard deviations for the irf&#xA;gamma=-0.25&#xA;irf.rw.e_gamma = irf.rw.e&#xA;n=length(irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$irf$&lt;/span&gt;rw)&#xA;for(i in 1:n){&#xA;    irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$irf$&lt;/span&gt;rw[i]   = irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$irf$&lt;/span&gt;rw[i]*gamma&#xA;    irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$Lower$&lt;/span&gt;rw[i] = irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$Lower$&lt;/span&gt;rw[i]*gamma&#xA;    irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$Upper$&lt;/span&gt;rw[i] = irf.rw.e_gamma&lt;span class=&quot;math-container&quot;&gt;$Upper$&lt;/span&gt;rw[i]*gamma&#xA;} &#xA;plot(irf.rw.e)&#xA;plot(irf.rw.e_gamma)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://r.789695.n4.nabble.com/Impulse-response-analysis-within-package-vars-td841596.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17535" LastEditorUserId="17535" LastEditDate="2020-02-28T11:41:31.347" LastActivityDate="2020-02-28T11:41:31.347" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="11010" PostTypeId="1" AcceptedAnswerId="11013" CreationDate="2016-04-03T08:11:14.220" Score="3" ViewCount="1259" Body="&lt;p&gt;I have a list of documents which look like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[&amp;quot;Display is flickering&amp;quot;]&#xA;[&amp;quot;Battery charger is broken&amp;quot;]&#xA;[&amp;quot;Hard disk is making noises&amp;quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;These text documents are just free text.&#xA;I have processed with tokenization, lemmatization, stop words removal, and now I want to assign tags based on a list of words. Example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;{&amp;quot;#display&amp;quot;:[&amp;quot;display&amp;quot;,&amp;quot;screen&amp;quot;,&amp;quot;lcd&amp;quot;,&amp;quot;led&amp;quot;]}&#xA;{&amp;quot;#battery&amp;quot;:[&amp;quot;battery&amp;quot;,&amp;quot;power cord&amp;quot;,&amp;quot;charger&amp;quot;,&amp;quot;drains&amp;quot;]}&#xA;{&amp;quot;#hard disk&amp;quot;:[&amp;quot;hard disk&amp;quot;,&amp;quot;performance&amp;quot;,&amp;quot;slow&amp;quot;]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;After text normalization I have:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[&amp;quot;Display is flickering&amp;quot;] -&amp;gt; [&amp;quot;display&amp;quot;,&amp;quot;flicker&amp;quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;What technique is recommended to compare document: [&amp;quot;display&amp;quot;,&amp;quot;flicker&amp;quot;]&#xA;with my dictionary of words and return which value matches the best?&#xA;In this case I would like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[&amp;quot;display&amp;quot;,&amp;quot;flicker&amp;quot;] = &amp;quot;#display&amp;quot;:&amp;quot;display&amp;quot;&#xA;[&amp;quot;battery&amp;quot;,&amp;quot;charger&amp;quot;,&amp;quot;broke&amp;quot;] = &amp;quot;#battery&amp;quot;:&amp;quot;charger&amp;quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Basically it compares Document A in tokens with a list B of other documents and return which document in list B with more common matches.&lt;/p&gt;&#xA;&lt;p&gt;I'm using &lt;a href=&quot;http://stevenloria.com/finding-important-words-in-a-document-using-tf-idf/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF&lt;/a&gt;, but want to know if there are other techniques, code samples to use.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="98307" LastEditDate="2020-08-04T06:28:08.780" LastActivityDate="2020-08-04T07:40:54.433" Title="Word analysis in Python" Tags="|python|nlp|nltk|cosine-distance|text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11012" PostTypeId="1" AcceptedAnswerId="11015" CreationDate="2016-04-03T15:15:55.967" Score="1" ViewCount="68" Body="&lt;p&gt;Let's say I have a dataset like this&lt;code&gt;X&amp;lt;-matrix(rnorm(30), nrow=100, ncol=6)&lt;/code&gt;. I am trying to find a way to apply the sum function to every four rows of column 3. This means I want to sum rows 1-4, rows 4-8 and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any functions such as apply or lapply I could use?&lt;/p&gt;&#xA;" OwnerUserId="10894" LastActivityDate="2016-04-03T18:54:35.620" Title="Apply function on every four rows" Tags="|r|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11013" PostTypeId="2" ParentId="11010" CreationDate="2016-04-03T15:44:04.663" Score="1" Body="&lt;p&gt;You can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Word_embedding&quot; rel=&quot;nofollow&quot;&gt;word embedding&lt;/a&gt; in order to compare whole phrases. I am aware about two models: Google's &lt;a href=&quot;http://deeplearning4j.org/word2vec&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt; and Stanford's &lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow&quot;&gt;GloVe&lt;/a&gt;. Now, word embedding works best with, well - words. However, you could concatenate every word in your phrase and re-train the models. Afterwards, you could calculate their similarity (say, with cosine similarity) and see how similar your whole phrases are semantically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="17564" LastActivityDate="2016-04-03T15:44:04.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11014" PostTypeId="1" CreationDate="2016-04-03T16:37:53.997" Score="8" ViewCount="1828" Body="&lt;p&gt;In binary classification problems it seems the F1 score is often used as a performance measure. As far as I've understood the idea is to find the best tradeoff between precision and recall. The formula for the F1 score is symmetric in  precision and recall. However, (and that's what bothers me) there is an asymmetry between precision and recall. While recall is a property of the classifier that is independent of prior probabilities, precision is a quantity that does depend on the prior probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me what's so special about the combination of precision and recall? Why don't we use precision (which is the &lt;em&gt;positive predictive value&lt;/em&gt;) and &lt;em&gt;negative predictive value&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="17565" LastActivityDate="2016-06-03T18:45:09.667" Title="Why are precision and recall used in the F1 score, rather than precision and NPV?" Tags="|classification|binary|scoring|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11015" PostTypeId="2" ParentId="11012" CreationDate="2016-04-03T18:54:35.620" Score="2" Body="&lt;p&gt;You could use &lt;code&gt;tapply&lt;/code&gt; or &lt;code&gt;aggregate&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1)&#xA;X &amp;lt;- matrix(rnorm(30), nrow=100, ncol=6)&#xA;id &amp;lt;- ceiling(seq_len(nrow(X))/4)&#xA;&#xA;tapply(X[, 3], id, sum)&#xA;#       1       2       3       4       5       6       7       8       9      10 &#xA;# -0.2137 -1.0629 -0.5030  0.2687  1.4961 -0.9343  2.0076  3.1162 -1.3511 -1.6868&#xA;&#xA;aggregate(X[, 3], by=list(id), FUN=sum)&#xA;#    Group.1       x&#xA;# 1        1 -0.2137&#xA;# 2        2 -1.0629&#xA;# 3        3 -0.5030&#xA;# 4        4  0.2687&#xA;# 5        5  1.4961&#xA;# 6        6 -0.9343&#xA;# 7        7  2.0076&#xA;# 8        8  3.1162&#xA;# ...&#xA;&#xA;# check results&#xA;sum(X[1:4, 3])&#xA;# [1] -0.2137&#xA;sum(X[5:8, 3])&#xA;# [1] -1.063&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2961" LastActivityDate="2016-04-03T18:54:35.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11016" PostTypeId="2" ParentId="10997" CreationDate="2016-04-03T19:03:05.183" Score="10" Body="&lt;p&gt;I won't go into details but the following should help you grasp the idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They use &lt;a href=&quot;https://en.wikipedia.org/wiki/Quantile&quot; rel=&quot;noreferrer&quot;&gt;Quantiles (Wikipedia)&lt;/a&gt; to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ (sorted), you can try the $10$-quantiles split points $\{x_{10}, x_{20}, \cdots, x_{90}\}$ and have a good approximation already. This is what the $\epsilon$ parameter is doing. They consider a split point when the split has $\sim \epsilon N$ more points under it than the last split point. If $\epsilon = 0.01$, you will end up with $\sim 100$ split points, being larger than $\{1\%, 2\%, ..., 99\%\}$ of the other points. They do not consider a new split when &quot;the sum changes more than $\epsilon$&quot; but when the number of points under the current point is larger by $\epsilon$ than the last one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you have a lot of continuous points very that are already well classified, it might be useless to split between them. You want to split the parts of your data set that are very wrong, the ones that are difficult to learn. To do so, they use weighted quantiles. This is where the weights play a role. The first $10$-quantile will not be the first point that is larger than $10\%$ of the points, but the first point that is larger than $10\%$ of the weights.&lt;/p&gt;&#xA;" OwnerUserId="15501" LastActivityDate="2016-04-03T19:03:05.183" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11017" PostTypeId="2" ParentId="10997" CreationDate="2016-04-03T19:50:15.297" Score="8" Body="&lt;p&gt;Just adding the algebraic part to @Winks answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second equation should have its sign reversed, as in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - (-g_i/h_i)]^2 + constant = &#xA;\sum_{i=1}^n\frac{1}{2}h_i[f_t^2(x_i) + 2\frac{f_t(x_i)g_i}{h_i} + (g_i/h_i)^2] = \sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i) + \frac{gi^2}{2h_i}]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last term is indeed constant: remember that the $g_i$ and $h_i$ are determined by the previous iteration, so they're constant when trying to set $f_t$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, now we can claim &quot;this is exactly weighted squared loss with labels $-gi/hi$ and weights $h_i$&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Credit goes to Yaron and Avi from my team for explaining me this.&lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2016-04-03T19:50:15.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11018" PostTypeId="1" CreationDate="2016-04-03T20:05:25.647" Score="1" ViewCount="80" Body="&lt;p&gt;I have county-level data from each of the 50 states, and each state suppresses data differently. I've been looking everywhere to find a way to do a tobit-type regression in R that can account for the different censoring levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found out that in STATA you can do this with cnreg, so I'm wondering if there's an equivalent package in r, or any way to do it, maybe with censReg? &lt;/p&gt;&#xA;" OwnerUserId="15154" LastActivityDate="2016-04-03T20:05:25.647" Title="R: tobit with a varying censoring level (cnreg equivalent)" Tags="|r|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11020" PostTypeId="2" ParentId="11014" CreationDate="2016-04-04T01:13:49.407" Score="4" Body="&lt;p&gt;F1 score weights precision and recall equally but there are easy generalizations to any case where you consider recall $\beta$ times more important than precision. See &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/F1_score&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F_\beta = (1 + \beta^2) \frac{precision \cdot recall}{\beta^2 precision + recall}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;F1 is just a harmonic mean. The simple mean doesn't quite make sense because precision and recall have the same numerators (true positive) but different denominators (test positive, condition positive). So only a harmonic mean makes sense. I don't know if there's more theory to it than that -- the simplest weighted mean that make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I get the gist of your point, which I paraphrase to refer to the fact that precision has &quot;test positive&quot; in the denominator so is quite sensitive to how much the classifier marks positive. For this reason you don't so often see, for example, precision-recall curves. You see ROC curves which are recall-specificity curves (true positive rate vs false positive rate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's closer to what you suggest but you're suggesting PPV vs NPV. Sure that could be valid depending on your use case, but I think the argument tends to cut the other way, to recall-specificity instead, not precision-NPV.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-04-04T01:13:49.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11021" PostTypeId="2" ParentId="10974" CreationDate="2016-04-04T03:27:46.440" Score="3" Body="&lt;p&gt;You can do anything you want in the low dimensional space, and can try to validate as well. By clustering the above, you are in effect assigning features/tags to your data points in higher dimensions. Remember, tSNE tries to preserve distances, so that points in high dimensions will remain close to each other in low dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind, don't forget that no two instances of tSNE will be the same, which means that your clustering centers will be different each time you run tSNE.&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-04-04T03:27:46.440" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11024" PostTypeId="1" CreationDate="2016-04-04T09:31:14.030" Score="27" ViewCount="17827" Body="&lt;p&gt;I am trying to understand how I can encode categorical variables using likelihood estimation, but have had little success so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="17318" LastEditorUserId="29169" LastEditDate="2019-06-08T03:50:21.487" LastActivityDate="2020-04-27T20:41:13.493" Title="Encoding categorical variables using likelihood estimation" Tags="|feature-engineering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11025" PostTypeId="1" CreationDate="2016-04-04T09:37:28.470" Score="6" ViewCount="8671" Body="&lt;p&gt;In particular, how much memory does a recurrent NN require as a function of the dataset size,number of nodes, etc., and how expensive is it to evaluate at runtime given a new test point?&lt;/p&gt;&#xA;" OwnerUserId="17583" LastActivityDate="2017-04-27T12:44:31.750" Title="What is the complexity of a recurrent neural network?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11026" PostTypeId="2" ParentId="10876" CreationDate="2016-04-04T11:55:34.253" Score="2" Body="&lt;p&gt;Of course you have wrong loss function in function gradientUpdate. You are using MSE, which should be used for regression problems. When you are doing classification you should use loss function for classification problems - this is old post about it: &lt;a href=&quot;https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/&quot; rel=&quot;nofollow&quot;&gt;https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also look at wiki: &lt;a href=&quot;https://en.wikipedia.org/wiki/Loss_functions_for_classification&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Loss_functions_for_classification&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10301" LastActivityDate="2016-04-04T11:55:34.253" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11027" PostTypeId="2" ParentId="10690" CreationDate="2016-04-04T14:24:09.340" Score="1" Body="&lt;p&gt;Interesting ideas on finger printing, but that isnt going to work here, as you don't have a definitive sound you are working with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me, this seems a bit more like an image processing problem. So instead of thinking how you could reduce dimensionality, could could try and combine your data together into &quot;images&quot; that digitally represent the sounds you are looking at. Then using standard image processing techniques. I say this waving my hands around vaguely, as I don't know of specific ones, but have enough understanding of edge detection, feature detection and applying map/reduce type methodologies which might be able to give templates for what you are after.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-04T14:24:09.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11029" PostTypeId="2" ParentId="11014" CreationDate="2016-04-04T17:34:57.313" Score="0" Body="&lt;p&gt;If it is purely a binary classification problem (class A vs. class B), then the benefit of the F-score is primarily for characterizing performance over an unbalanced data set (more instances of one class than the other) and your question/concern is more relevant. The Wikipedia page for &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F-score&lt;/a&gt; states&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;em&gt;Note, however, that the F-measures do not take the true negatives into account, and that measures such as the Phi coefficient, Matthews correlation coefficient, Informedness or Cohen's kappa may be preferable to assess the performance of a binary classifier.&lt;/em&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if the classifier is intended to be a &lt;em&gt;detector&lt;/em&gt;, one is usually more interested in performance with respect to the target class (Positive) than the non-target class (Negative). Furthermore, the target is often the one that is under-represented in the data set. In that context, I think it is more intuitive to want to know what fraction of targets are detected (recall) and how reliable/confident each detection is (precision). While knowing how good the detector is at not detecting non-targets (negative predictive value) can have value, it is not a very insightful quantity to deal with when trying to characterize the performance of a target detector with an imbalanced data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, the F-score tuning parameter ($\beta$) provides a more intuitive way to balance the importance of detecting all the targets (high recall) with the importance of having detections with high confidence (high precision). Note also that the F-score can be written in terms of Type I and Type II errors (see the Wikipedia link above).&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-04-04T17:34:57.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11031" PostTypeId="1" AcceptedAnswerId="11035" CreationDate="2016-04-04T20:44:08.073" Score="-2" ViewCount="883" Body="&lt;p&gt;Im currently in the last year, and I want to do a masters thesis on a topic that has NOSQL and Machine Learning or Business Intelligence. In my topic i want for defintely NOSQL, so I want to add a complementary topic (machine learning or business intelligence) to it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my research i know that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOSQL: provides a mechanism for storage and retrieval of data which is modeled in means other than the tabular relations used in relational databases. And there are currently 4 types (key/value, column, document and graph).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine Learning: is a subfield of computer science that evolved from the study of pattern recognition and computational learning theory in artificial intelligence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in short, for example can there be a thesis topic where i can evaluate which machine learning algorithm can be better implemented using data from a nosql database, or something like that. I would like to know your opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can't give NOSQL related to machine learning, also NOSQL related with Business Intelligence would be a great help aswell, since Business Intelligence &quot;is a set of techniques and tools for the acquisition and transformation of raw data into meaningful and useful information for business analysis purposes&quot;.&lt;/p&gt;&#xA;" OwnerUserId="17596" LastActivityDate="2016-04-05T01:22:33.970" Title="Any Master Thesis Topics related to NoSQL and Machine Learning or Business Intelligence?" Tags="|machine-learning|nosql|" AnswerCount="1" CommentCount="1" ClosedDate="2016-04-05T08:34:52.153" ContentLicense="CC BY-SA 3.0" />
  <row Id="11032" PostTypeId="1" CreationDate="2016-04-04T22:25:49.787" Score="3" ViewCount="1596" Body="&lt;p&gt;I have a dataset where a set of people donated for charity along with the dates of the donation. I have to find the probability of each donor donating in the next three months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data is available from August 2014 - February 2016. I have to predict the probability of each person donating for March-June 2016.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a snapshot of the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id  date    amount&#xA;1   13-08-14    2485&#xA;1   21-11-14    2105&#xA;1   17-09-15    1359&#xA;2   13-08-14    2542&#xA;2   20-04-15    1276&#xA;2   12-10-15    2694&#xA;3   20-11-14    3556&#xA;4   28-07-15    3383&#xA;5   13-08-14    1698&#xA;5   11-12-14    1725&#xA;5   09-06-15    1376&#xA;5   17-09-15    3230&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="16829" LastEditorUserId="16829" LastEditDate="2016-04-05T00:19:25.047" LastActivityDate="2016-04-07T19:00:15.733" Title="How to predict the probability of an event?" Tags="|predictive-modeling|probability|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11033" PostTypeId="2" ParentId="11032" CreationDate="2016-04-04T23:42:28.800" Score="-2" Body="&lt;p&gt;First you need to find in what kind of distribution is your data: linear, exponential, normal ... Etc. &#xA;After that you need to find the area size under the equation in which your event will occur. &#xA;It's all come down to what kind of distribution you are on.&lt;/p&gt;&#xA;" OwnerUserId="17601" LastActivityDate="2016-04-04T23:42:28.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11034" PostTypeId="1" CreationDate="2016-04-05T00:19:56.920" Score="2" ViewCount="4785" Body="&lt;p&gt;Some people say data scientists don't necessarily need to know real analysis and measure theory, but for others, real analysis and measure theory are very important for the undersdanding of kernel methods, stochastic processes etc. Should I learn real analysis and measure theory? I don't want to be too academic&lt;/p&gt;&#xA;" OwnerUserId="17602" LastEditorUserId="12527" LastEditDate="2016-04-05T07:38:18.330" LastActivityDate="2016-04-05T07:38:18.330" Title="Are real analysis and measure theory essential to learn for data science?" Tags="|machine-learning|career|probability|" AnswerCount="2" CommentCount="0" ClosedDate="2016-04-06T00:17:45.737" ContentLicense="CC BY-SA 3.0" />
  <row Id="11035" PostTypeId="2" ParentId="11031" CreationDate="2016-04-05T01:22:33.970" Score="3" Body="&lt;p&gt;The question is a bit broad and opinion-based for StackExchange, but I'll have a quick go anyway: I don't see good topics in this area. For academic machine learning, how you store your data is largely irrelevant, either because the research is on small data anyway and will be read into memory, or because the research is pretty theoretical to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are certainly interesting issues to explore here for distributed ML. However NoSQL stores would not in general be helpful. They specialize in random access and random updates to keyed data. ML generally needs high-throughput sequential access to data without updating it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BI + ML is too broad. Yes there are topics in there somewhere but hard to discuss at that level.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-04-05T01:22:33.970" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11036" PostTypeId="2" ParentId="11032" CreationDate="2016-04-05T01:24:52.890" Score="1" Body="&lt;p&gt;Please double-check if there's the only data you have got, because all you have is a single predictor &lt;code&gt;date&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is indeed your only data source, then you only have a single predictor, and your independent variable is continuous. Now, you should plot &lt;code&gt;date&lt;/code&gt; vs &lt;code&gt;amount&lt;/code&gt; and fit a single linear regression. Does the fitting look good? Only you can tell because we don't have the full data-set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's not a good fit, look at the plot and ask yourself does this look like a curve? If so, you might want to fit a spline curve or something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also check the autocorrelation. This makes sense because your data look like a time series (you'll need to check it yourself). If this is the case, you might want to consider MA and ARCH model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not possible for us to give you accurate advice because we don't know your data.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-04-05T01:34:02.340" LastActivityDate="2016-04-05T01:34:02.340" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11037" PostTypeId="2" ParentId="11034" CreationDate="2016-04-05T01:50:33.280" Score="2" Body="&lt;p&gt;Neither of real analysis or measure theory are necessary for data science. Most of the mathematics you need to know is at the undergraduate level and these courses should suffice: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculus &lt;/li&gt;&#xA;&lt;li&gt;Linear Algebra &lt;/li&gt;&#xA;&lt;li&gt;Probability and Statistics &lt;/li&gt;&#xA;&lt;li&gt;Optimization &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most of machine learning involved in data science (that deals with application vs creating novel machine learning algorithms), requires a linear combination of concepts learnt in the above mathematics courses. For example, you can understand enough kernel methods well enough if you understood linear algebra (inner products, matrices etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, data science has to do with practice and experimentation. A lot of key concepts become clearer only through experimentation and if you need to understand a new method, the best way is to read up specifically about it and then experiment with it. In this respect, I feel studying an entire branch of involved mathematics is not worth it if your goal is to understand a machine learning method -- you can always ask a Guru in such situations to explain what you're missing at a conceptual level :-)&lt;/p&gt;&#xA;" OwnerUserId="847" LastActivityDate="2016-04-05T01:50:33.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11039" PostTypeId="2" ParentId="11034" CreationDate="2016-04-05T02:54:24.767" Score="2" Body="&lt;p&gt;No, you don't need to understand measure theory and real analysis to do machine learning in data science. However, it'd be hard to for you to read academic papers (eg: kernel methods) if you don't have the knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you want to be a mathematician or wish to pursue a Phd, you really don't need to know too much about the theories. In fact, most of the engineers working in the field I know don't understand statistics at all. They simply use some framework, get some good results, then get paid. &lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-04-05T02:54:24.767" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11042" PostTypeId="1" CreationDate="2016-04-05T06:52:25.740" Score="3" ViewCount="142" Body="&lt;p&gt;Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$.  The probability of $x$ is: &#xA;\begin{equation}&#xA;P(x) = \sum_h P(x,h) = \sum_h \frac{e^{-E(x,h)}}{Z}.&#xA;\end{equation}&#xA;The goal is to maximize the probability of $x$ conditional on the parameters of the model $\theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element  $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood: &#xA;\begin{equation}&#xA;\left\lbrace \hat{\theta} \in \hat{\Theta} :  N^{-1} \sum_{x_i} \frac{\partial  \log p(x_i)} {\partial \theta} = 0 \right\rbrace &#xA; \end{equation} &#xA;and chose the one $\theta^\star \in \hat{\Theta} $ maximizing the empirical likelihood.  There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $\theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:&#xA;\begin{equation}&#xA; \theta_{i+1} = \theta_{i} - \eta_i \frac{\partial  \log p(x_i)}{\partial \theta_i}&#xA; \end{equation}&#xA;  The practice I learned in &lt;a href=&quot;http://dx.doi.org/10.1162/neco.2006.18.7.1527&quot; rel=&quot;nofollow&quot;&gt;Hinton et al. (2006)&lt;/a&gt; and in &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1390290&quot; rel=&quot;nofollow&quot;&gt;Tieleman (2008)&lt;/a&gt;   is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help! &lt;/p&gt;&#xA;" OwnerUserId="17614" LastEditorUserId="17614" LastEditDate="2016-04-05T08:31:08.757" LastActivityDate="2019-12-08T13:02:00.260" Title="Training the parameters of a Restricted Boltzman machine" Tags="|neural-network|optimization|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11043" PostTypeId="1" CreationDate="2016-04-05T09:03:15.113" Score="11" ViewCount="12824" Body="&lt;p&gt;Amazon's &lt;a href=&quot;https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/defining-the-voice-interface&quot; rel=&quot;noreferrer&quot;&gt;Alexa&lt;/a&gt;, Nuance's &lt;a href=&quot;https://developer.nuance.com/mix/nlu/docs/natural-language-understanding/About_Mix.nlu.html#developing-an-nlu-model&quot; rel=&quot;noreferrer&quot;&gt;Mix&lt;/a&gt; and Facebook's &lt;a href=&quot;https://wit.ai/docs/console/quickstart&quot; rel=&quot;noreferrer&quot;&gt;Wit.ai&lt;/a&gt; all use a similar system to specify how to convert a text command into an intent - i.e. something a computer would understand. I'm not sure what the &quot;official&quot; name for this is but I call it &quot;intent recognition&quot;. Basically a way to go from &quot;please set my lights to 50% brightness&quot; to &lt;code&gt;lights.setBrightness(0.50)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way they are specified is by having the developer provide a list of &quot;sample utterances&quot; which are associated with an intent, and optionally tagged with locations of &quot;entities&quot; (basically parameters). Here's an example from Wit.ai:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mJlpP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mJlpP.png&quot; alt=&quot;wit.ai example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;strong&gt;how do these systems work?&lt;/strong&gt; Since they are all very similar I assume there is some seminal work that they all use. Does anyone know what it is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly &lt;a href=&quot;https://www.houndify.com/&quot; rel=&quot;noreferrer&quot;&gt;Houndify&lt;/a&gt; uses a different system that is more like regexes: &lt;code&gt;[&quot;please&quot;] . (&quot;activate&quot; | &quot;enable&quot; | &quot;switch on&quot; | &quot;turn on&quot;) . [(&quot;the&quot; | &quot;my&quot;)] . (&quot;lights&quot; | &quot;lighting&quot;) . [&quot;please&quot;]&lt;/code&gt;. I assume that is integrated into the beam search of their voice recognition system, whereas Alexa, Wit.ai and Mix seem to have separate Speech-&gt;Text and Text-&gt;Intent systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; I found a starting point - &lt;a href=&quot;http://ccc.inaoep.mx/~villasen/bib/dmello-roman07.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;A Mechanism for Human - Robot Interaction through Informal Voice Commands&lt;/em&gt;&lt;/a&gt;. It uses something called Latent Semantic Analysis to compare utterances. I'm going to read up on that. At least it has given me a starting point in the citation network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2:&lt;/strong&gt; LSA is essentially comparing the words used (Bag of Words) in each paragraph of text. I don't see how it can work very well for this case as it totally loses the word order. Although maybe word order doesn't matter much for these kinds of commands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 3:&lt;/strong&gt; &lt;a href=&quot;http://people.ee.duke.edu/~lcarin/HiddenTopicMarkov.pdf&quot; rel=&quot;noreferrer&quot;&gt;Hidden Topic Markov Models&lt;/a&gt; look like they might be interesting.&lt;/p&gt;&#xA;" OwnerUserId="17618" LastEditorUserId="17618" LastEditDate="2016-04-05T14:40:42.573" LastActivityDate="2017-11-28T21:33:35.107" Title="How do &quot;intent recognisers&quot; work?" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11044" PostTypeId="2" ParentId="11032" CreationDate="2016-04-05T09:21:08.337" Score="-1" Body="&lt;p&gt;I think you could use time series modeling algorithm as @Student_T said. Also you can make window time to find relation between new donate and previous donate and you can use amount, may be people with high payment and low payment have different behavior. first of all you should change your data in a way that fill gaps. I mean you should add data about month that a person have not any payment. &#xA;after that you should make a table like this:&#xA;person_id / month(or day or week or 3month) / count of payment / count of payment last month/ sum of amount that paid last month/ Is paid last month?/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then you should find whether your filed is useful and independent or not. and try to add other filed. and then build your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;good luck.&lt;/p&gt;&#xA;" OwnerUserId="15569" LastActivityDate="2016-04-05T09:21:08.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11045" PostTypeId="1" CreationDate="2016-04-05T09:34:36.330" Score="2" ViewCount="1214" Body="&lt;p&gt;My dataset is the result of multiple joins from large transactional database stored in a MySQL database. There are some columns with text value and double values.  I am trying to do a first level sequential pattern mining on this dataset using SPMF.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is it possible to give words as input in SPMF? The documentation tells that only positive integers are allowed.&lt;/li&gt;&#xA;&lt;li&gt;If I want to give words or decimal numbers, which software/package shall I use?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="16284" LastEditDate="2016-04-09T23:31:07.237" LastActivityDate="2021-04-08T16:20:12.297" Title="Sequential pattern mining in SPMF: Giving words as input" Tags="|data-mining|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11047" PostTypeId="1" CreationDate="2016-04-05T15:47:19.137" Score="5" ViewCount="1417" Body="&lt;p&gt;I have some trouble understanding LSTM models in &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow&lt;/a&gt;.&#xA;For simplicity, let us consider the &lt;a href=&quot;https://github.com/tflearn/tflearn/blob/master/examples/nlp/lstm.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Example program&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use the &lt;a href=&quot;http://tflearn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tflearn&lt;/a&gt; as a wrapper as it does all the initialization and other higher level stuff automatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Till line number &lt;code&gt;42&lt;/code&gt; &lt;code&gt;net = tflearn.input_data([None, 200])&lt;/code&gt; it's pretty clear what happens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You load a dataset into variables and make it of standard lengths in this case 200. for both the input variables and also the 2 classes present in this case are converted to one hot vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to know here is how the LSTM takes the input and across how many samples does it predict the output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do these parameters indicate: &lt;code&gt;n_words=20000&lt;/code&gt; &amp;amp; &lt;code&gt;net = tflearn.embedding(net, input_dim=20000, output_dim=128)&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to &lt;a href=&quot;http://jeffdonahue.com/lrcn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;replicate&lt;/a&gt; the &lt;a href=&quot;http://arxiv.org/abs/1411.4389&quot; rel=&quot;nofollow noreferrer&quot;&gt;activity recognition dataset in the paper.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I would like to input a 4096 vector as input to the lstm and the idea is to take 16 of such vectors and then produce the classification result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the code would look like this but I don't know how the input to the LSTM should be given&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import division, print_function, absolute_import&#xA;&#xA;import tflearn&#xA;from tflearn.data_utils import to_categorical, pad_sequences&#xA;from tflearn.datasets import imdb&#xA;&#xA;train, val = something.load_data()&#xA;trainX, trainY = train #each X sample is a (16,4096) nd float64 &#xA;valX, valY = val #each Y is a one hot vector of 101 classes.&#xA;&#xA;net = tflearn.input_data([None, 16,4096])&#xA;net = tflearn.embedding(net, input_dim=4096, output_dim=256)&#xA;net = tflearn.lstm(net, 256)&#xA;net = tflearn.dropout(net, 0.5)&#xA;net = tflearn.lstm(net, 256)&#xA;net = tflearn.dropout(net, 0.5)&#xA;net = tflearn.fully_connected(net, 101, activation='softmax')&#xA;net = tflearn.regression(net, optimizer='adam',&#xA;                         loss='categorical_crossentropy')&#xA;&#xA;model = tflearn.DNN(net, clip_gradients=0., tensorboard_verbose=3)&#xA;model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,&#xA;          batch_size=128,n_epoch=2,snapshot_epoch=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17487" LastEditorUserId="26596" LastEditDate="2016-11-29T10:52:26.190" LastActivityDate="2018-05-13T11:01:07.357" Title="Understanding Tensorflow LSTM models?" Tags="|machine-learning|python|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11049" PostTypeId="1" CreationDate="2016-04-05T17:25:38.213" Score="5" ViewCount="4074" Body="&lt;p&gt;I'm looking at a diagram of map reduce where there is a map step, a shuffle step and then the reduce step. Why shuffle?&lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2016-04-07T06:02:35.083" Title="Why does map reduce have a shuffle step?" Tags="|map-reduce|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11050" PostTypeId="2" ParentId="11032" CreationDate="2016-04-05T17:25:45.673" Score="0" Body="&lt;p&gt;You can use Binary Logistic Regression for this analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to using Binary Logit, you'd have to spend some time preparing the data for this analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can create several RFM types of features from this data set. Examples: Number of donations, time between donations, time since the most recent donation, time since the first donation, average donation amount, the amount of first donation, the most recent donation amount, etc. (I can provide more examples, if needed.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your task is to predict the probability of donation during a four-month timeframe (Mar-Jun 2016), you can create those features (leading indicators) for each donor &lt;em&gt;as of&lt;/em&gt; the end of October 2015. All leading indicators would be created based off of timeframes prior to that cut-off point. Your &lt;strong&gt;observation window&lt;/strong&gt; is from Nov-2015 to Feb-2016. This is where your event flag (dependent variable) should come from: 1 if a donor donated (again) during the observation window, and 0 otherwise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to make this model generalizable, I'd recommend pulling several such cross-sections of your data (in addition to the October 2015 slice explained above,)&lt;/p&gt;&#xA;" OwnerUserId="17397" LastActivityDate="2016-04-05T17:25:45.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11052" PostTypeId="2" ParentId="11049" CreationDate="2016-04-05T17:48:24.513" Score="0" Body="&lt;p&gt;The shuffle step involves transferring data from the mappers to the reducers - this is necessary simply so that you have input to the reducers. You can find a more detailed answer &lt;a href=&quot;https://stackoverflow.com/questions/22141631/what-is-the-purpose-of-shuffling-and-sorting-phase-in-the-reducer-in-map-reduce&quot;&gt;at this Stack Overflow question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10462" LastActivityDate="2016-04-05T17:48:24.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11055" PostTypeId="2" ParentId="11025" CreationDate="2016-04-05T19:44:35.167" Score="1" Body="&lt;p&gt;I've found some time ago two interesting papers about recurrent neural networks and their complexity. I guess you can use those as a reference points at least:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://arxiv.org/pdf/1602.08210v2.pdf&quot; rel=&quot;nofollow&quot;&gt;Architectural Complexity Measures of Recurrent Neural Networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://papers.nips.cc/paper/822-bounds-on-the-complexity-of-recurrent-neural-network-implementations-of-finite-state-machines.pdf&quot; rel=&quot;nofollow&quot;&gt;Bounds on the complexity of recurrent neural network implementations of finite state machines &lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12855" LastActivityDate="2016-04-05T19:44:35.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11057" PostTypeId="1" CreationDate="2016-04-05T20:25:46.083" Score="4" ViewCount="2327" Body="&lt;p&gt;Following the raise of ensembling (e.g ensembling of xgboost learners) after its recurrent wins in Kaggle competitions, using it has become a tradition in machine learning. However, some argue that clustering is a form of ensembling where we apply clustering on a data set first, then applying a learning algorithm (to do classification or regression) on each of the resulting clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to understand how clustering is different from ensembling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that clustering can be similar to the idea of ensembling if we simply consider &quot;homogeneous ensembling&quot; where we train the same model on each of the clusters. But I am looking for potential limitations of clustering where using heterogeneous ensembles (different learns for all the data) is the &quot;only&quot; way to go about the problem.  &lt;/p&gt;&#xA;" OwnerUserId="9774" LastEditorUserId="9774" LastEditDate="2016-04-14T14:19:22.680" LastActivityDate="2019-08-29T09:25:40.053" Title="Ensembling vs clustering in machine learning" Tags="|machine-learning|clustering|ensemble-modeling|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11059" PostTypeId="1" AcceptedAnswerId="11064" CreationDate="2016-04-06T02:30:41.510" Score="2" ViewCount="2733" Body="&lt;p&gt;In a neural network, each neuron will have it's activation.&#xA;But what the activation mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it just mean nothing but a temporal value to produce the final result or it's has something to with our understanding to the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, considering a neural network to recognize a handwritten character from a picture, I wonder if it's possible that a neuron's activation represents the how the pixel matches the specific value in a small part of the picture?&lt;/p&gt;&#xA;" OwnerUserId="17644" LastActivityDate="2016-04-06T08:45:03.530" Title="What does the activation of a neuron mean?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11060" PostTypeId="1" CreationDate="2016-04-06T05:14:11.457" Score="18" ViewCount="6843" Body="&lt;p&gt;I have sparse features which are predictive, also I have some dense features which are also predictive. I need to combine these features together to improve the overall performance of the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the thing is when I try to combine these together, the dense features tend to dominate more over sparse features, hence giving only 1% improvement in AUC compared to model with only dense features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has somebody come across similar problems? Really appreciate the inputs, kind of stuck. I have already tried lot of different classifiers, combination of classifiers, feature transformations and processing with different algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for the help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already tried the suggestions which are given in the comments. What I have observed is, for almost 45% of the data, sparse features perform really well, I get the AUC of around 0.9 with only sparse features, but for the remaining ones dense features perform well with AUC of around 0.75. I kind of tried separating out these datasets, but I get the AUC of 0.6, so, I can't simply train a model and decide which features to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the code snippet, I have tried out so many things, that I am not sure what exactly to share :(&lt;/p&gt;&#xA;" OwnerUserId="17619" LastEditorUserId="17619" LastEditDate="2016-04-18T04:42:58.910" LastActivityDate="2020-07-23T13:23:55.593" Title="Merging sparse and dense data in machine learning to improve the performance" Tags="|machine-learning|classification|predictive-modeling|scikit-learn|supervised-learning|" AnswerCount="5" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11062" PostTypeId="1" CreationDate="2016-04-06T06:17:55.703" Score="2" ViewCount="232" Body="&lt;p&gt;I have time-series energy consumption data for a duration of one-month. The frequency of data is half-hourly. The features of dataset are&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;temperature - temperature value at particular time instant&lt;/li&gt;&#xA;&lt;li&gt;humidity -    humidity value at particular time instant&lt;/li&gt;&#xA;&lt;li&gt;timeofday -  this corresponds to 48 half-hourly durations of a day.&lt;/li&gt;&#xA;&lt;li&gt;weekday -  day of the week (value between 1 - 7)&lt;/li&gt;&#xA;&lt;li&gt;prevday1 -  energy consumption at the same time-instant on previous day&lt;/li&gt;&#xA;&lt;li&gt;prevday2 - energy consumption at the same time-instant on a day before the  previous day&lt;/li&gt;&#xA;&lt;li&gt;prev_instant1 - energy consumption at the previous time instant. For example, for 7 pm, previous_instant1 is 6:30 pm&lt;/li&gt;&#xA;&lt;li&gt;prev_2_hour - average energy consumption during the last two hours&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Using these variable I need to forecast power (energy) consumption. The &lt;code&gt;pairs&lt;/code&gt; plot of these variables is&lt;a href=&quot;https://i.stack.imgur.com/tosbW.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tosbW.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started with a linear model, but it seems that the model is performing very badly. I want to know is there any automatic technique, which can check all possible combinations of predictor variables and output the best fitted model. I can check all models (linear combinations, non-linear, splines etc) manually, but it will take huge time.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am forecasting in real-time at a half-hourly rate (very short term forecasting). Also, I use separate models corresponding to different half-hours of the day. For example, In a day we have 48 half-hours so corresponding to each half-hour I have a separate model. For the whole process, I use following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a training data set with above mentioned features. This training data set is of one month duration (30 days)&lt;/li&gt;&#xA;&lt;li&gt;In the next step, I forecast for the next day (testing day) using specific models for each of 48 half-hour durations&lt;/li&gt;&#xA;&lt;li&gt;At the end of the day, I retrain all my 48 models with recently updated data (recent historical 30 days)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Steps 2 and 3 continue till stopping condition. Stopping condition used is the no. of testing days. A screenshot of my training data is as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(training_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bp24V.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bp24V.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tail(training_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GhpPC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GhpPC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13291" LastEditorUserId="13291" LastEditDate="2016-04-06T08:10:28.607" LastActivityDate="2016-04-06T16:26:47.873" Title="Underlying model for prediction using different prediction variables" Tags="|r|predictive-modeling|regression|time-series|feature-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11063" PostTypeId="1" CreationDate="2016-04-06T08:12:42.710" Score="1" ViewCount="340" Body="&lt;p&gt;I have a dataset of reviews and I want to extract the features along with their opinion words in the reviews.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to extract features from my data using any Vector Space Model?(TF-IDF, Word2Vec, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit for sample data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Teacher&quot;, &quot;Subject&quot;, &quot;Feedback&quot;&#xA;&quot;Dr. Tayal&quot;,&quot;Discrete Mathematics&quot;,&quot;He has very good subject knowledge. He didn't take all the lectures. He teaches and explains concepts very well.&quot;&#xA;&quot;Ms Vibha&quot;,&quot;OOPS&quot;,&quot;Subject knowledge is poor, but she is regular with classes. She is unnecassrily strict and does not teach the subject nicely.&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="807" LastEditorUserId="807" LastEditDate="2016-04-09T11:21:34.737" LastActivityDate="2016-04-09T11:21:34.737" Title="Feature Extraction and Vector Space Model" Tags="|feature-extraction|word2vec|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11064" PostTypeId="2" ParentId="11059" CreationDate="2016-04-06T08:45:03.530" Score="6" Body="&lt;p&gt;The activation of a neuron is mathematically nothing but a function of its input. Consider a neural network with one hidden layer and one input vector $\mathbf{x}$. The input $a_j$ of neuron $j$ can be written as:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_j = w_j^T \mathbf{x} + b_j$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The activation of neuron $j$ is then a transformation $g: \mathbf{R} \rightarrow \mathbf{R}$ of the input. For exmaple, one can use the sigmoid activiation function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$sig(a_j) = \dfrac{1}{1 + \exp(-a_j)} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weights $w$ are chosen to minimize a loss function and not for the sake of their interpretation. Nevertheless, one can try to interpret the activation of a neuron as &lt;em&gt;internal representation of the input&lt;/em&gt;. One of the nicest examples I've seen comes from the &lt;a href=&quot;http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html&quot; rel=&quot;noreferrer&quot;&gt;Rummelhart et al (1986)&lt;/a&gt;, Figure 4. In that paper, two family trees were fed into a neural network. The family trees represented an Italian and an English speaking family comprising three generations. Among other things, when activating one name of the tree, the neuronal activities represented whether the person was from the English or Italian tree and which generation the person embodied.   &lt;/p&gt;&#xA;" OwnerUserId="17614" LastActivityDate="2016-04-06T08:45:03.530" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11065" PostTypeId="2" ParentId="11049" CreationDate="2016-04-06T09:13:37.380" Score="6" Body="&lt;p&gt;The shuffle step occurs to guarantee that the results from mapper which have the same key (of course, they may or may not be from the same mapper) will be send to the same reducer. So, the reducer can further reduce the result set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example, your WordCount mappers emit &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(hot, 1)&#xA;(cold, 1)&#xA;(hot, 1)&#xA;(rain, 1)&#xA;(cold, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the shuffle step will ensure that&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(hot, 1), (hot, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;get to the same reducer.&lt;br&gt;&#xA;So, you get your final answer &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(hot, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the same applied for (cold,1) and (rain,1)&lt;/p&gt;&#xA;" OwnerUserId="16872" LastActivityDate="2016-04-06T09:13:37.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11066" PostTypeId="2" ParentId="11057" CreationDate="2016-04-06T09:36:47.410" Score="2" Body="&lt;p&gt;As far as I understand your question, the difference between those two approaches lies in the field of limits you apply to the model. During ensemble learning, say xgboost, you will be training multiple boosted trees models, each of which would be trained on a (random) subset of features and (random) subset of species from your dataset. That way, you will get N classifiers, for example, each of which would have slightly different &lt;em&gt;experience&lt;/em&gt; about your problem, but every one of them would be trained on randomly limited (with no specific limit) chunk of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you apply clustering (for any reason, be it diversity of classifiers or reduction of dataset to fit it into memory) before training, you &lt;strong&gt;fist&lt;/strong&gt; rely on a clustering algorithm to find some sort of communities in your dataset and then you train separate classifiers for every cluster. How they are used in your example further, I am not completely sure about, but a concise way would be assignment of a particular cluster for a new data and using 'right' classifier for that instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a real-world example, take loan approval problem. In &lt;strong&gt;ensemble learning&lt;/strong&gt; you will have three persons sitting in the same office, talking to a constant flow of customers in need of money, and every credit approver would have similar experience, but will tend to look into particular data they are familiar with, like job, marital status or any other 'feature'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;clustering&lt;/strong&gt;, you will have three persons approving loans, but one will be working strictly with farmers, other will be looking into cases of recently fired people and third would be dealing with students in urge to buy a new smartphone. All three will be working on the same problem, but their experience will be focused on a subset of people and a student-handler wouldn't be a good choice to make a decision on a farmer loan since he has no experience with appraising their financial situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering approach makes more sense when you have a diverse data which would contradict itself in terms of training (load approval criterion would be different for gender-based, marital status based and so on, and married persons would be approved with higher probability than single once due to responsibilities and stuff even if their income would be lower than single person's) - but for every cluster you would have a single working model, other-cluster model would provide random results since it hasn't been trained on other-cluster data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ensemble learning works great in a way that you get a set of models with different 'experience' about your problem. Each classifier would have its own precision/recall/general performance metric, and if three classifiers with different feature and example set would agree on an outcome, &lt;em&gt;generally&lt;/em&gt; it is safe to say that if three classifiers say you it is an apple, it should be an apple (because, error rate of 1% for each classifier would give you 1%*1%*1% = 0.0001% chance that all three classifiers are incorrect).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The arithmetic of joint classifier error rate isn't that straightforward since usually ensemble sub-models (talking about xgboost here) aren't independent as they share some features, but general idea is like that. &lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-04-06T09:36:47.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11068" PostTypeId="2" ParentId="11057" CreationDate="2016-04-06T14:47:58.510" Score="6" Body="&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; Ensembling and clustering are completely unrelated techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ensembling:&lt;/strong&gt; Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate sets of data -- it's the models themselves that are different. Generally the more diverse the models the better. For example, an ensemble might comprise the following models: a support vector machine (SVM), a random forest, a gradient boosting model (GBM), logistic regression, k-nearest-neighbors, and a neural network. On Kaggle, I have also seen top competitors create ensembles comprising the same type of model (e.g., random forests) but with a variety of tuning parameters (e.g., one with 100 trees, another with 250 trees, etc). Clustering could be one of the models included in an ensemble. Ensembles are very effective (although extremely complex to operationalize) and can generally improve your error about 2-5%.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clustering:&lt;/strong&gt; Divide the data into mutually exclusive groups with the hope that your cluster-specific models can reduce error by specializing, except in the case of fuzzy clustering where the groups need not be mutually exclusive. At financial companies that build credit risk models, these cluster-specialized models are sometimes called &quot;sub-models&quot; in the sense that they predict the same target variable and therefore fit under an abstract/figurative parent model. This makes model risk reporting easier in the sense that it's simpler to refer to model 1 then model 1a, 1b, 1c, 1d. Clustering in this context is generally based on business knowledge and not a formal clustering technique such as k-means or other types of centroid-based clustering. For supervised learning tasks, formal clustering is generally considered a pretty poor technique because measuring the quality of the clusters is subjective. Your clusters can also vary dramatically depending on how you initialize the centers and how many clusters you specify. In practice, clustering is used much less frequently than ensembling.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="-1" LastEditDate="2019-08-29T09:25:40.053" LastActivityDate="2019-08-29T09:25:40.053" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11069" PostTypeId="1" CreationDate="2016-04-06T15:53:52.853" Score="4" ViewCount="556" Body="&lt;p&gt;I am taking a class in information retrieval. We learned that the index of a search engine has (possibly among other things):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A vocabulary mapping terms to their statistics (frequency, type, ...) and&lt;/li&gt;&#xA;&lt;li&gt;A posting list mapping terms to the documents were they are stored (with or without positions, fields, ...)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These are separate data structures. I understand why those information is needed and what for. But I don't understand why we want to keep them separate. Why can't we have one data structure that maps terms to statistics and documents?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently thinking it might be because the vocabulary would be much smaller and we could read it from memory. So we could use the statistics to remove certain query terms, which are likely not useful or to try to find misspellings in the query without having to touch the large posting list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct or is there another reason to keep vocabulary and posting list separate?&lt;/p&gt;&#xA;" OwnerUserId="17659" LastEditorUserId="17659" LastEditDate="2016-04-06T20:13:37.853" LastActivityDate="2019-12-18T10:01:05.157" Title="Why keep vocabulary and posting list separate in a search engine" Tags="|information-retrieval|search|indexing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11070" PostTypeId="2" ParentId="11057" CreationDate="2016-04-06T16:02:45.950" Score="1" Body="&lt;p&gt;It depends on how you &lt;em&gt;combine&lt;/em&gt; results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many ensemble techniques will either:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;train the same classifier, but on different parts of the data&lt;/li&gt;&#xA;&lt;li&gt;train different classifiers on the full data&lt;/li&gt;&#xA;&lt;li&gt;vary both: classifier and data subsets&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In either situation, you afterwards have to combine the results; usually with some form of majority voting. So if 2 classifiers return &quot;A&quot;, 1 classifier returns &quot;B&quot;, then the outcome is &quot;A&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a good outcome, every member needs to be better than random; to improve over the individual results they must not be too similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use clustering for the first approach (to get different parts of the data). But the problem is that these parts are not independent, and too biased. You would usually want each classifier to know &quot;a little bit of everything&quot;. By withholding some part of the data you prevent them from overfitting the same way. For this, random is usually best. If you do clustering, there is a chance you get 1 classifier that thinks everything is &quot;A&quot;, 1 that thinks everything is &quot;B&quot;, and 1 that thinks everything is &quot;C&quot;. You even encourage them to overfit! So you always get the result 1 A, 1 B, 1 C = no majority.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-04-06T16:02:45.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11071" PostTypeId="2" ParentId="11062" CreationDate="2016-04-06T16:26:47.873" Score="-1" Body="&lt;p&gt;Have had the same problem before.  Used Weka tool (&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.waikato.ac.nz/ml/weka/&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach:&lt;br&gt;&#xA;• convert time of day into a nominal (non-numeric), eg bins numbered 0-47&lt;br&gt;&#xA;• convert day of week into a nominal (non-numeric)&lt;br&gt;&#xA;• use M5Tree to construct a model&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rules in the tree will pop out things like &quot;if 0 &amp;lt; bin &amp;lt; 5:30am_bin&quot; then ... which discriminates start of day&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is better with more than 1 month of data.  There will only be 4 samples for certain days of the week, for example.  So more months would be better. And in such case again converting month to a nominal value (non-numeric) lets the rules pick up seasonal and monthly changes.&lt;/p&gt;&#xA;" OwnerUserId="17485" LastActivityDate="2016-04-06T16:26:47.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11072" PostTypeId="2" ParentId="10168" CreationDate="2016-04-06T16:34:11.170" Score="1" Body="&lt;p&gt;Things have changed in TensorFlow since this question was asked but here is a link to doing &lt;a href=&quot;https://github.com/loliverhennigh/All-Convnet-Autoencoder-Example&quot; rel=&quot;nofollow&quot;&gt;conv2d_transpose&lt;/a&gt;. I think thats what you are looking for&lt;/p&gt;&#xA;" OwnerUserId="17660" LastActivityDate="2016-04-06T16:34:11.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11073" PostTypeId="1" AcceptedAnswerId="11079" CreationDate="2016-04-06T18:40:17.237" Score="5" ViewCount="3808" Body="&lt;p&gt;I am learning cs231n Convolutional Neural Networks for Visual Recognition. The lecture notes introduce the concepts of width, height, depth. For example,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In CIFAR-10, images are only of size 32x32x3 (32 wide, 32 high, 3&#xA;  color channels)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, in another example,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a volume of size [55x55x96] has 96 depth slices, each of size [55x55]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does width 96 means? Does it mean 96 color channels? Why can we have more than 3 color channels?&lt;/p&gt;&#xA;" OwnerUserId="16310" LastEditorUserId="29169" LastEditDate="2021-02-16T15:50:22.590" LastActivityDate="2021-02-16T15:50:22.590" Title="What is the depth of an image in Convolutional Neural Network?" Tags="|neural-network|image-classification|convolutional-neural-network|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11074" PostTypeId="1" CreationDate="2016-04-06T19:29:55.927" Score="2" ViewCount="76" Body="&lt;p&gt;I am struggling to figure a way to determine if a person is 'no good' or 'good'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little bit about what I am trying to accomplish, I have a data set of a payment for a violation with a check or a bounced check on a given day, if there is a bounced check it means there is no such account. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they have a pattern of payments and bounces in chronological order then they are doing something fishy such as gaming the system. But, if they have payments, bounces, payments, and little to no bounces after that then the initial bounces could have been human entry errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use the data set to determine who are the 'no good' people that have a pattern, ex. paying, bouncing, paying, bouncing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of my data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;person     state date                           status  bounce_code   &#xA;A           NY  23-DEC-15 06.27.08.000000000 PM Paid    &#xA;A           NY  23-DEC-15 06.12.58.000000000 PM Paid    &#xA;A           NY  18-DEC-15 10.14.39.000000000 AM Return  R03&#xA;A           NY  15-DEC-15 04.16.58.000000000 PM Return  R03&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The original data set contains thousands of people and their history of payments and bounces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea is to analyze each person to see if they fit a &quot;mold&quot; (payment, bounce, payment, bounce) and if they do i will label them as a 'no good' person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if this is confusing and I am open to suggestions on how to approach this problem.&lt;/p&gt;&#xA;" OwnerUserId="17663" LastActivityDate="2016-04-06T20:51:29.517" Title="How to detect a no good person in data?" Tags="|classification|clustering|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11075" PostTypeId="2" ParentId="11074" CreationDate="2016-04-06T20:51:29.517" Score="2" Body="&lt;p&gt;I would start by looking at the distribution of &lt;strong&gt;bounce rates&lt;/strong&gt; for each individual. That distribution would provide some details around what could be considered a &quot;normal&quot; bounce rate, and where you would draw a line for an extreme bounce rate that implies a 'bad'. Additionally, you can also look at &lt;strong&gt;average&lt;/strong&gt; bounce rates (divide bounce rate by the tenure), and review these averages by tenure. This may reveal that different threshold values might be appropriate for customers with different tenure. For instance, if a customer is new (first month of activity) then even a moderate bounce rate could imply a 'bad'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional metrics -- like the ratio of bounce to valid payments, time between bounce -- could be useful to look at as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once a set of initial thresholds are identified, I would closely review some of the customers who are identified as 'bad' based on those thresholds to ensure that there are not too many false positives. The &lt;em&gt;business&lt;/em&gt; point of view (from those who would be using this information to make decisions) would be great to incorporate into this review.&lt;/p&gt;&#xA;" OwnerUserId="17397" LastActivityDate="2016-04-06T20:51:29.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11076" PostTypeId="1" CreationDate="2016-04-06T22:04:30.173" Score="4" ViewCount="1792" Body="&lt;p&gt;I am currently conducting some analysis using NTSB aviation accident database. There are cause statements for most of the aviation incidents in this dataset that describe the factors lead to such event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of my objectives here is to try to group the causes, and clustering seems to be a feasible way to solve this kind of problem. I performed the followings prior to the beginning of k-means clustering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Stop-word removal, that is, to remove some common functional words in the text&lt;/li&gt;&#xA;&lt;li&gt;Text stemming, that is, to remove a word's suffix, and if necessary, transform the term into its simplest form &lt;/li&gt;&#xA;&lt;li&gt;Vectorised the documents into TF-IDF vector to scale up the less-common but more-informative words and scale down highly-common but less-informative words&lt;/li&gt;&#xA;&lt;li&gt;Applied SVD to reduce the dimensionality of vector&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After these steps k-means clustering is applied to the vector. By using the events that occurred from Jan 1985 to Dec 1990 I get the following result with number of clusters &lt;code&gt;k = 3&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: I am using Python and sklearn to work on my analysis)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;... some output omitted ... &#xA;Clustering sparse data with KMeans(copy_x=True, init='k-means++', max_iter=100, n_clusters=3, n_init=1,&#xA;    n_jobs=1, precompute_distances='auto', random_state=None, tol=0.0001,&#xA;    verbose=True)&#xA;Initialization complete&#xA;Iteration  0, inertia 8449.657&#xA;Iteration  1, inertia 4640.331&#xA;Iteration  2, inertia 4590.204&#xA;Iteration  3, inertia 4562.378&#xA;Iteration  4, inertia 4554.392&#xA;Iteration  5, inertia 4548.837&#xA;Iteration  6, inertia 4541.422&#xA;Iteration  7, inertia 4538.966&#xA;Iteration  8, inertia 4538.545&#xA;Iteration  9, inertia 4538.392&#xA;Iteration 10, inertia 4538.328&#xA;Iteration 11, inertia 4538.310&#xA;Iteration 12, inertia 4538.290&#xA;Iteration 13, inertia 4538.280&#xA;Iteration 14, inertia 4538.275&#xA;Iteration 15, inertia 4538.271&#xA;Converged at iteration 15&#xA;&#xA;Silhouette Coefficient: 0.037&#xA;Top terms per cluster:&#xA;**Cluster 0: fuel engin power loss undetermin exhaust reason failur pilot land**&#xA;**Cluster 1: pilot failur factor land condit improp accid flight contribute inadequ**&#xA;**Cluster 2: control maintain pilot failur direct aircraft airspe stall land adequ**&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I generated a plot graph of the data as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yFrpO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yFrpO.png&quot; alt=&quot;Plot result of the k-means clustering&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result doesn't seem like make sense to me. I wonder why all of the clusters contain some common terms like &quot;pilot&quot; and &quot;failure&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility that I can think of (but I am not sure if it is valid in this case) is the documents with these common terms are actually located at the very centre of the the plot graph, therefore they can not be efficiently clustered into a right cluster. I believe this problem cannot be addressed by increasing the number of clusters, as I have just done it and this problem persists. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just want to know if there is any other factors that could cause the scenario that I am facing? Or more broadly, am I using the right clustering algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks StackExchange and Data Science Portal. &lt;/p&gt;&#xA;" OwnerUserId="17670" LastEditorUserId="17670" LastEditDate="2016-04-06T22:41:25.780" LastActivityDate="2016-04-07T10:31:09.933" Title="How to explain the outcome of k-means clustering?" Tags="|python|data-mining|clustering|scikit-learn|k-means|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11077" PostTypeId="1" CreationDate="2016-04-07T01:38:43.780" Score="6" ViewCount="7053" Body="&lt;p&gt;I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec on it, or is there a better way?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2018-06-19T22:48:41.180" Title="Using several documents with word2vec" Tags="|nlp|word-embeddings|word2vec|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11079" PostTypeId="2" ParentId="11073" CreationDate="2016-04-07T05:36:19.313" Score="6" Body="&lt;p&gt;It means that the number of filters (&lt;a href=&quot;https://stats.stackexchange.com/a/188216/12359&quot;&gt;a.k.a. kernels, or  feature detector&lt;/a&gt;) in the previous convolutional layer is 96. You may want to watch the &lt;a href=&quot;https://youtu.be/LxfUGhug-iQ?t=29m55s&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; of the lecture, and in particular &lt;a href=&quot;http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this slide&lt;/a&gt;, which mentions that a filter is applied to the full depth of your previous layer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AuqKy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AuqKy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-07T14:48:14.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11080" PostTypeId="2" ParentId="11049" CreationDate="2016-04-07T06:02:35.083" Score="5" Body="&lt;p&gt;The shuffle step is actually the &lt;strong&gt;most important step&lt;/strong&gt;. That is where the MapReduce system takes over and you get all the performance if it is well implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mappers and reducers run in parallel, and independent of each other. They usually do really stupid operations, such as in the word-count example. The shuffle phase is where all the heavy lifting occurs. &lt;strong&gt;All the data is rearranged&lt;/strong&gt; for the next step to run in parallel again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key contribution of MapReduce is that surprisingly many programs can be factored into a mapper, the predefined shuffle, and a reducer; and they will run fast as long as you optimize the shuffle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why you can extend with custom map and reduce functions, but not with a custom shuffle - that part needs to be written by experts, you can only modify the keys used by it. You can live with a &quot;just ok&quot; map, you can live with a &quot;just ok&quot; reduce, but you cannot have a &quot;just ok&quot; shuffle, it needs to be top notch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also where people &lt;em&gt;fail&lt;/em&gt; in writing good mapreduce. For example by producing too many copies before shuffling, or putting everything into the same key (shuffle does not make a lot of sense then anymore).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-04-07T06:02:35.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11081" PostTypeId="1" AcceptedAnswerId="11086" CreationDate="2016-04-07T09:03:49.453" Score="5" ViewCount="2433" Body="&lt;p&gt;While it is obviously clear that features can be ranked on basis of importance and many machine learning books give examples of random forests on how to do so, its not very clear on which occasions one should do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, for boosting methods, is there any reason why one should do feature selection. Wouldn't the boosting methods themselves eliminate the low importance feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't is just always better to add more features (if one didn't have the practical problem of time limitations).&lt;/p&gt;&#xA;" OwnerUserId="17571" LastActivityDate="2016-04-07T17:07:25.030" Title="Importance of feature selection for boosting methods" Tags="|feature-selection|feature-extraction|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11082" PostTypeId="2" ParentId="11076" CreationDate="2016-04-07T10:31:09.933" Score="3" Body="&lt;p&gt;I made some similar studies on process descriptions on computers. The main difficulty was to discard words that were not meaningful. Which is hard because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Some words appear in every clusters, but in different proportion. They should not be discarded, like &quot;pilot&quot;. It is normal that &quot;pilot&quot; appears often, but depending on the number of occurrences it might signify that the pilot made a decisive action for the accident.&lt;/li&gt;&#xA;&lt;li&gt;Some words appear very rarely, but they are the key words for the reason why the accident happened. Then it is very hard to find a way to give them more importance without adding direct prior knowledge to your clustering.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I could suggest is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First try different k value. Why did you chose 3 ?&lt;/li&gt;&#xA;&lt;li&gt;Try a different algorithm for 2D visualization. For example t-sne is good for reducing high dimension in 2. Look at the examples here &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/manifold/plot_lle_digits.html#example-manifold-plot-lle-digits-py&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/auto_examples/manifold/plot_lle_digits.html#example-manifold-plot-lle-digits-py&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Try working directly on your word occurrences to check whether your word differ between clusters. For example, you can compute the KL-divergence of each word for each cluster given their global distribution. See here &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;If you can't get good results, think of another way of vectorizing. For example word2vec is good at finding words that occurs often together and could be more suitable to your case.               &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-04-07T10:31:09.933" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11085" PostTypeId="2" ParentId="9939" CreationDate="2016-04-07T16:27:06.513" Score="-1" Body="&lt;p&gt;I do not know what industry you are working in but predicting unique users can be difficult at times because of factors outside of your control. For example if your company tries a new advertising campaign that targets a new segment it is hard to calculate a range because they are a new eco system interacting with your previously defined set of rules. &lt;/p&gt;&#xA;" OwnerUserId="17694" LastActivityDate="2016-04-07T16:27:06.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11086" PostTypeId="2" ParentId="11081" CreationDate="2016-04-07T17:07:25.030" Score="4" Body="&lt;p&gt;There is a difference between boosting and features selection. &#xA;It is very important to understand that the original boosting algorithm or bagging algorithm have been modified and augmented with many features selection and/or data sampling ( over/ down/ synthetic) to improve the accuracy. &#xA;Let us talk about the difference between bagging and boosting : &#xA;Booth of them are random subspace based algorithms, the difference is in bagging we used uniform distribution and all the samples have the same weight , in boosting we use non- uniform distribution, during the training the distribution will be modified and difficult samples will have higher probability. The second difference is the voting. In bagging it is average voting , in boosting it is an weighted voting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features selection algorithms try to find the best set of features that can separate the classes. But there is no explicit consideration for difficult or easy samples and what is the used training algorithm. &#xA;In boosting, the algorithm selects the feature that minimize the error , the error is the sum of prob &quot;weights&quot; of samples that miss classified, since the difficult samples have higher weights , the selected feature will be the one that better distinguish between the difficult samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;FE ( Features, data) --&gt; feature set&#xA;Boosting ( features, data, base learners type, initial distribution, difficult samples) --&gt; feature set &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-07T17:07:25.030" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11087" PostTypeId="2" ParentId="9939" CreationDate="2016-04-07T18:12:03.940" Score="0" Body="&lt;p&gt;You can't retrospectively do it just with counts of unique visitors per day. If you represent the unique users on each day by sets $A_1, A_2, \dots, A_n$, the union can be as small as $|A_1|$, if all sets are equal, or as large as $|A_1| + \dots + |A_n|$ if all sets a pairwise disjoint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you could estimate average number of days r per month a user visited the site (over the users that visited at least once), then there were exactly $(|A_1| + \dots + |A_n|)/r$ unique visitors that month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice proper way to do this, starting with new online data, is to use Hyperloglog or other memory-efficient approximate counting algorithms, as Marmite Bomber suggests.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="6550" LastEditDate="2016-04-07T18:22:01.810" LastActivityDate="2016-04-07T18:22:01.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11088" PostTypeId="1" AcceptedAnswerId="11093" CreationDate="2016-04-07T18:41:02.600" Score="6" ViewCount="11462" Body="&lt;p&gt;I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++.&#xA;I have not come across many instances (close to none, I did find libraries like Shark though) of implementation in C++.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it to do with the fact that C++ isn't as simple to use when it comes to manipulating large amount of data?&lt;/p&gt;&#xA;" OwnerUserId="17699" LastEditorUserId="17726" LastEditDate="2016-04-09T00:04:38.190" LastActivityDate="2019-03-02T16:19:12.490" Title="Why aren't languages like C, C++ used for data analytics instead of R, Python?" Tags="|predictive-modeling|data|c|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11089" PostTypeId="2" ParentId="9195" CreationDate="2016-04-07T18:46:25.913" Score="0" Body="&lt;p&gt;Definitely. The paper from the Creator himself, Geoffrey Hinton. &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&lt;/a&gt; read it. But I encourage u to see the difference by yourself implementing it. &lt;/p&gt;&#xA;" OwnerUserId="11141" LastActivityDate="2016-04-07T18:46:25.913" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11090" PostTypeId="2" ParentId="11032" CreationDate="2016-04-07T19:00:15.733" Score="0" Body="&lt;p&gt;You could try using the markov model!&#xA;(An illustration of which can be found &lt;a href=&quot;http://www.dummies.com/how-to/content/how-to-utilize-the-markov-model-in-predictive-anal.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you could detect patterns in the dataset by plotting it and then figure out which algorithm to use based on what is the degree of correlation and the nature of the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, what is the number of users you have? &#xA;You could group together data of each user and run the algorithm for the user that is being asked for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regression is the way to go if you want to know if a user will donate or not, to find the probability, try markov!&lt;/p&gt;&#xA;" OwnerUserId="17699" LastActivityDate="2016-04-07T19:00:15.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11091" PostTypeId="1" CreationDate="2016-04-07T19:15:04.710" Score="9" ViewCount="5013" Body="&lt;p&gt;Let's say there are two cars in an image. How can it detect these cars, given that it can detect single car in an image?&lt;/p&gt;&#xA;" OwnerUserId="11141" LastEditorUserId="29169" LastEditDate="2021-02-16T15:50:06.173" LastActivityDate="2021-02-16T15:50:06.173" Title="How does deep learning helps in detecting multiple objects in single image?" Tags="|deep-learning|convolutional-neural-network|object-recognition|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11092" PostTypeId="2" ParentId="11088" CreationDate="2016-04-07T19:23:47.103" Score="1" Body="&lt;p&gt;It's because you lose to much time configuring and building the code itself rather than solving the actual problem. Example... to load the data, you have to initialize memory in C, then you need to handle it later. In Python you just call a method to load it. Garbage collector will handle it for you later when you don't need it. This is a very simple scenario.&lt;/p&gt;&#xA;" OwnerUserId="11141" LastEditorUserId="676" LastEditDate="2016-04-09T00:04:35.113" LastActivityDate="2016-04-09T00:04:35.113" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11093" PostTypeId="2" ParentId="11088" CreationDate="2016-04-07T20:34:44.950" Score="7" Body="&lt;p&gt;Yes, you're correct -- it's that C and C++ are harder to use and are more burdened with boilerplate code that obfuscates your model building logic. When you build models, you have to iterate rapidly and frequently, often throwing away a lot of your code. Having to write boilerplate code each time substantially slows you down over the long run.  Using R's caret package or Python's scikit-learn library, I can train a model in just 5-10 lines of code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ecosystem also plays a big role. For example, Ruby is easy to use, but the community has never really seen a need for machine learning libraries to the extent that Python's community has. R is more widely used than Python (for stats and machine learning only) because of the strength of its ecosystem and its long history catering to that need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth pointing out that most of these R and Python libraries are written in low-level languages like C or Fortran for their speed. For example, I believe Google's TensorFlow is built with C, but to make things easier for end users, its API is in Python. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-04-09T20:18:38.237" LastActivityDate="2016-04-09T20:18:38.237" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11094" PostTypeId="2" ParentId="11088" CreationDate="2016-04-07T21:56:39.250" Score="3" Body="&lt;p&gt;&lt;strong&gt;Advantages of any modern interpreted language over C++.&lt;/strong&gt; &#xA;Like any tradeoff, these are advantages in some situations and disadvantages in others. Situations where you don't want these conveniences are becoming more rare, though, as hardware gets even faster and high-level language implementations get even more efficent.&#xA;No compile step. Write your code in my_program.py, then run it with python my_program.py.&#xA;No memory management. You don't have to explicitly allocate memory for new variables, and you don't have to explicitly free memory you're done with. The interpreter will allocate memory for you and free it when it's safe to do so.&#xA;High-level native data types. Strings, tuples, lists, sets, dictionaries, file objects and more are built-in. As an example, {&quot;x&quot;: &quot;y&quot;} defines a dictionary (hash table) with string &quot;x&quot; as a key and string &quot;y&quot; as its value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Specific advantages of Python:&lt;/strong&gt;&#xA;Especially clean, straightforward syntax. This is a major goal of the Python language. Programmers familiar with C and C++ will find the syntax familiar yet much simpler without all the braces and semicolons.&#xA;Duck typing. If an object supports .quack, go ahead and call .quack on it without worrying about that object's specific type.&#xA;Iterators, generators and comprehensions. To get the first character of every line in a file, you'd write:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;file = open(&quot;file.txt&quot;)&#xA;list_of_first_characters = [line[0] for line in file]&#xA;file.close()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This iterates over the file only once.&#xA;(These particular features are just the tip of the iceberg of simple built-in syntax for high-level language features. Check out decorators next if you're intrigued.)&#xA;Huge standard library. Just to pick some random examples, Python ships with several XML parsers, csv &amp;amp; zip file readers &amp;amp; writers, libraries for using pretty much every internet protocol and data type, etc.&#xA;Great support for building web apps. Along with Ruby and JavaScript, Python is very popular in the web development community. There are several mature frameworks and a supportive community to get you started.&lt;/p&gt;&#xA;" OwnerUserId="16302" LastEditorUserId="16284" LastEditDate="2016-04-09T23:30:47.893" LastActivityDate="2016-04-09T23:30:47.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11096" PostTypeId="1" AcceptedAnswerId="11650" CreationDate="2016-04-07T22:38:43.297" Score="8" ViewCount="693" Body="&lt;p&gt;I'm trying to analyze some data I have but there is a lot of inconsistencies in my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a SQL table that I'm trying to analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The table is a table of universities with the following structure: &lt;code&gt;name:string, city:string, state:string, country:string&lt;/code&gt;&#xA;Name is always present however city, state, country can be missing.&#xA;My main issue is that there are a ton of typos and different declination of a university name.&#xA;For example here are the declination of Standford Unversity I find when I do  &lt;code&gt;SELECT &quot;universities&quot;.* FROM &quot;perm_universities&quot; WHERE (name like '%stanford%')&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stanford university - stanford - ca - united states of america&#xA;the leland stanford junior university - stanford - ca - united states of america&#xA;leland stanford jr. university - stanford - ca - united states of america&#xA;stanford university graduate school of business - stanford - ca - united states of america&#xA;the leland stanford junior university (stanford university) - stanford - ca - united states of america&#xA;leland stanford junior university - stanford - ca - united states of america&#xA;stanford university - stanford -  -&#xA;leland stanford jr. university, graduate school of business - stanford - ca - united states of america&#xA;stanford law school - stanford - ca - united states of america&#xA;stanford - stanford - ca - united states of america&#xA;stanford university, graduate school of business - stanford - ca - united states of america&#xA;stanford graduate school of business - stanford - ca - united states of america&#xA;stanford univerity - stanford - ca - united states of america&#xA;stanford university (the leland stanford junior university) - stanford - ca - united states of america&#xA;the leland stanford jr. university - palo alto - ca - united states of america&#xA;leland stanford junior university, school of law - stanford - ca / n/a - united states of america&#xA;stanford universit - stanford - ca - united states of america&#xA;the leland stanford university - stanford - ca - united states of america&#xA;leland standford stanford junior university - stanford - ca - united states of america&#xA;stanford university - cambridge - ma - united states of america&#xA;the leland stanford junior university 'stanford university' - stanford - ca - united states of america&#xA;stanford university school of law - stanford - ca - united states of america&#xA;stanford univresity - stanford - ca - united states of america&#xA;the leland stanford jr. university (stanford university) - stanford - ca - united states of america&#xA;leeland stanford junior university - stanford - ca - united states of america&#xA;leland stanford junion university -  - ca - united states of america&#xA;leland stanford junior university (stanford university) - stanford - ca - united states of america&#xA;the leland stanford junior university - stanford -  -&#xA;stanford university - graduate school of business - stanford - ca - united states of america&#xA;graduate school of business, stanford university - stanford - ca - united states of america&#xA;stanford universoty - stanford - ca - united states of america&#xA;leland stanford junior university - stanford -  -&#xA;stanford univeristy - palo alto - ca - united states of america&#xA;leland stanford university - palo alto - ca - united states of america&#xA;stanford university - stanford - ca / n/a - united states of america&#xA;the leland stanford junior university, stanford university - stanford - ca - united states of america&#xA;the leland stanford junior university graduate school of business - stanford - ca - united states of america&#xA;stanford universtiy - stanford - ca - united states of america&#xA;stanford univerisity - stanford - ca - united states of america&#xA;stanford university - stanford - ct - united states of america&#xA;stanford law scool - stanford - ca - united states of america&#xA;mba: stanford university - stanford - ca - united states of america&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;They are all the same university, but some have typos, some have different names, some have no cities, some have the wrong cities, ... The data isn't great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm trying to fix it. How can I consolidate this data?&lt;/p&gt;&#xA;" OwnerUserId="17702" LastEditorUserId="16284" LastEditDate="2016-04-09T16:57:07.157" LastActivityDate="2016-05-09T06:27:31.647" Title="Fixing data inconsistencies" Tags="|data-cleaning|similarity|distance|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11097" PostTypeId="2" ParentId="11077" CreationDate="2016-04-07T22:58:49.107" Score="1" Body="&lt;p&gt;There is no need of concatenation. The iterator class can be changed accordingly so that the model could iterate over all the documents in a single folder.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;&#xA;class WordTrainer(object):&#xA;   def __init__(self, dir_name):&#xA;      self.dir_name = dir_name&#xA;   def __iter__(self):&#xA;      for idx,file_name in enumerate(os.listdir(self.dir_name)):&#xA;        for idxx,line in enumerate(open(os.path.join(self.dir_name, file_name),'r'):&#xA;            words = [word.lower() for word in line.split()]&#xA;            yield words&#xA;&#xA; patient_details = WordTrainer('/path/for/records/folder')&#xA; word_vector_model = gensim.models.Word2Vec(patient_details, size=100, window=8, min_count=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The iterator provides each data from the files line by line, rather than loading an entire document into the model. It is not a memory efficient approach when you have few documents that could fit into your memory.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2017-08-01T12:09:36.900" LastActivityDate="2017-08-01T12:09:36.900" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11099" PostTypeId="2" ParentId="10585" CreationDate="2016-04-08T07:36:55.980" Score="3" Body="&lt;p&gt;Suppose you are investigating if heart rate can predict if a person smokes. You measure bpm for 30x1m consecutive times, and ask if the person smokes in order to build your training model data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would contribute to a better predictor? 30 observations from a person who smokes, or 1 observation from 30 people who smoke? Given that one person's heart rate won't change much over 30m, it seems clear that you'd rather have 1 sample from 30 people than 30 samples from one person. 30 samples from one person are not worth as much as 1 sample from 30 people. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the samples from one person are not independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think if you put non-independent samples into a neural net then it won't affect the predictive power too much as long as the non-independence is similar across all your training data. In one extreme, if all smokers and non-smokers have the same heart rate over the 30m period, then all you've done is repeated your input data precisely 30 times and nothing will be changed (except it will take 30x as long to run...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if smokers' heart rates are constant, and non-smokers' vary, then you add 30 measurements of each smoker's heart rate to your model, and a bunch of random measurements correlating those rates with non-smoking. Your NN is very likely to predict anyone with one of those smokers' heart rates is a smoker. This is clearly wrong - those 30 measurements from each smoker are only worth one measurement, and putting them in a NN will train the network wrongly.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-04-08T07:36:55.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11100" PostTypeId="2" ParentId="11091" CreationDate="2016-04-08T09:42:30.860" Score="3" Body="&lt;p&gt;The question itself is not quite clear, since you don't state that you have a model that can detect one car per run for an image or you are just asking what tools, algorithms or frameworks to use to detect cars (or other objects) in an image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answering second variant, you should be using developed algorithms for object detection, which are either Haar Cascades (which are embedded into OpenCV and there are clear tutorials of how to train your custom object detector, for example, &lt;a href=&quot;http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html&quot; rel=&quot;nofollow&quot;&gt;banana tutorial&lt;/a&gt;) or CNNs, which are the choice for object detection using neural networks, personally, I enjoy working with &lt;a href=&quot;https://github.com/pjreddie/darknet&quot; rel=&quot;nofollow&quot;&gt;that implementation&lt;/a&gt; - simple and comprehensive code and amazing results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both approaches (Haar Cascades and CNNs) basically find patterns of mutually connected and co-located shapes that &lt;em&gt;describe&lt;/em&gt; your particular object (be it face, banana, car or UFO) and use these patterns to find objects on a new image. Mutual inclusion of detected objects (when borders of objects intersect or one is included by another) is used to find best match for every region.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-04-08T09:42:30.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11101" PostTypeId="2" ParentId="11091" CreationDate="2016-04-08T10:47:15.370" Score="12" Body="&lt;p&gt;Although many solutions in production systems still use a sliding window as described below in this answer, the field of computer vision is moving quickly. Recent advances in this field include &lt;a href=&quot;https://arxiv.org/abs/1506.01497&quot; rel=&quot;noreferrer&quot;&gt;R-CNN&lt;/a&gt; and &lt;a href=&quot;http://pjreddie.com/media/files/papers/yolo.pdf&quot; rel=&quot;noreferrer&quot;&gt;YOLO&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Detecting object matches in an image, when you already have an object classifier trained, is usually a matter of brute-force scanning through image patches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with the largest expected patch size. E.g. if your image is 1024 x 768, but always a distance shot of a road maybe you do not expect any car to take up more than 80 x 80 pixels in the image. So you take an 80x80 block of pixels from one corner of the image, and ask your classifier what chance there is a car in that corner. Then take the next patch - perhaps move by 20 pixels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat for all possible positions, and decide which patches are most likely to contain cars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, take a block size down (maybe 60 x 60, moving 15 pixels at a times) and repeat the same exercise again. Repeat this until you have hit the expected smallest block size for your goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eventually you will have a list of areas within the image, with the probability that each contains a car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overlapped blocks both with high probability are most likely the &lt;em&gt;same&lt;/em&gt; car, so the logic needs to have thresholds for merging blocks - usually taking the overlapped area with the highest probability score - and declaring there is only one car in that area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As usual with ML approaches, you will need to experiment with correct meta-params - in this case block sizes, step sizes and rules for merging/splitting areas - in order to get the most accurate results.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-08-28T09:07:45.330" LastActivityDate="2016-08-28T09:07:45.330" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11102" PostTypeId="1" CreationDate="2016-04-08T14:02:14.107" Score="1" ViewCount="57" Body="&lt;p&gt;I ran in a somewhat basic problem with Orange 3.3.1... but wonder if this is a bug or something I don't understand:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a Matrix, all with discrete values.&#xA;I select Columns, that goes ok&#xA;Select Rows... and use the « is one of » filter with 3 values out of 15 in a particular column.&#xA;But when I Scatter Plot. Values matching the filter seems to have been selected, but the X Axis still show X names, all of them,  that are not in « is one of ». On top of it, values have shifted to to wrong X name...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone has an idea???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="17712" LastActivityDate="2016-10-18T20:21:05.227" Title="Discrete Values Matrix: Select Rows and Scattered Plot problem" Tags="|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11104" PostTypeId="1" CreationDate="2016-04-08T16:22:37.437" Score="2" ViewCount="3897" Body="&lt;p&gt;I'm trying to figure out if I can identify styles of art using support vector machines and I don't quite understand what makes them good. Perhaps I can make the question more accurate. Support vector machines are basically a way of doing classification with a linear model. You find $\vec{\beta}$ and $\alpha$ such that $sgn(\beta \cdot x + \alpha)$ determines what the class is. So I believe that in order to do image classification, you need logic. It does not suffice to find the average color of an image and decide whether it is greater or smaller than a number to decide which number an image represent. You need logic. I believe you need intermediate logic, like comparing one part of the image to another part of an image. Kind of like a multilayered support vector machine, which now looks like a neural net. Is this intuition correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sincerely,&#xA;Still figuring it out. &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="16284" LastEditDate="2016-04-09T23:30:50.737" LastActivityDate="2022-07-16T16:24:04.537" Title="Why are support vector machines good at classifying images?" Tags="|svm|image-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11105" PostTypeId="1" CreationDate="2016-04-08T16:28:19.133" Score="2" ViewCount="1736" Body="&lt;p&gt;I'm currently pursuing a bachelor's degree in physics from a university in the UK. Most data science jobs here have a strong preference for people with PhDs in numerate degrees like physics and maths. I don't understand the point of spending 5 years of my life doing research in something I don't want to pursue a career in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm confused about what master's degree I should choose if I want to apply for a job straight after graduating. A physics degree kind of leaves me hanging in the middle in terms of skills. I have done some analysis for experimental data using Python, but only very basic stuff. A bit of predictive modelling and C++, lots of maths, but barely any statistics. So I could either get a degree in CS to get better at programming or get one in statistics. Which one would give me the skills most relevant to data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, would you recommend that I get a diploma/certification in machine learning?&lt;/p&gt;&#xA;" OwnerDisplayName="user17716" LastActivityDate="2016-04-08T18:33:47.533" Title="Data science after a physics degree?" Tags="|beginner|education|" AnswerCount="1" CommentCount="1" ClosedDate="2016-04-09T00:04:04.720" ContentLicense="CC BY-SA 3.0" />
  <row Id="11106" PostTypeId="2" ParentId="11105" CreationDate="2016-04-08T18:33:47.533" Score="0" Body="&lt;p&gt;I think the best thing for a beginner is some practice to let the theory sink in. (If you need book suggestions, do a search.) If you are weak on computer science, contribute to an open source project like scikit-learn. If you are weak on data analysis, compete in a Kaggle competition. Write a paper and/or a few blog posts to prove your skills. Do this now while you are in school. An undergraduate physics degree should be enough to get your foot in the door if you do the above. A degree will rarely get you a job by itself; you still have to pass the job interview, and that's why I recommend getting practice.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-04-08T18:33:47.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11108" PostTypeId="2" ParentId="11104" CreationDate="2016-04-08T19:38:05.110" Score="2" Body="&lt;p&gt;Support vector machines have one built-in &quot;layer&quot; that helps with having an interpretation of the data - the kernel. You could even use output from some other image classifier, including a neural network, as the kernel. E.g. you could measure how far apart two images are in &quot;classifier space&quot; from a trained neural network (maybe one trained against different targets from the ones you want to classify with the SVM)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also an implied &quot;layer&quot; in image feature construction.  Common image features are histograms of pixel values, histograms of pixel differences (edge detection), bags of &quot;visual words&quot; and there are many more possible transformations and statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically if you are using a classifier other than a deep neural network (which is supposed to discover the features automatically), then you will pre-process the image into a set of features. That pre-processing covers the majority of an approach being &quot;good at images&quot;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-04-08T19:38:05.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11109" PostTypeId="2" ParentId="9428" CreationDate="2016-04-08T21:11:05.903" Score="1" Body="&lt;p&gt;I believe that the command you are looking for is outreg2. For other output options check out &lt;a href=&quot;http://www.cpc.unc.edu/research/tools/data_analysis/statatutorial/misc/exporting_results&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-04-08T21:11:05.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11115" PostTypeId="1" AcceptedAnswerId="11116" CreationDate="2016-04-09T22:46:51.333" Score="6" ViewCount="5902" Body="&lt;p&gt;The standard setup when training a neural network seems to be to split the data into train and test sets, and keep running until the scores stop improving on the test set.  Now, the problem:  there is a certain amount of noise in the test scores, so the single best score may not correspond to the state of the network which is most likely to be best on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen a few papers point to a specific epoch or iteration in the training as being &quot;best by cross-validation&quot; but I have no idea how that is determined (and the papers do not provide any details).  The &quot;best by cross-validation&quot; point is not the one with the best test score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would one go about doing this type of cross validation?  Would it be by doing k-fold on the test set?  Okay, that gives k different test scores instead of one, but then what?&lt;/p&gt;&#xA;" OwnerUserId="26" LastEditorUserId="26" LastEditDate="2016-05-11T08:53:09.983" LastActivityDate="2016-05-11T08:53:09.983" Title="Cross validation when training neural network?" Tags="|neural-network|cross-validation|training|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11116" PostTypeId="2" ParentId="11115" CreationDate="2016-04-10T01:25:29.343" Score="5" Body="&lt;p&gt;I couldn't say what the authors refer to by&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;best by cross-validation&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but I'll mention a simple and general procedure that's out there:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are correct that analyzing &lt;strong&gt;one&lt;/strong&gt; estimate of the generalization performance using &lt;strong&gt;one&lt;/strong&gt; training and &lt;strong&gt;one&lt;/strong&gt; test set is quite simplistic. Cross-validation can help us understand how this performance varies across datasets, instead of wonder whether we got lucky/unlucky with our choice of train/test datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Split the whole dataset into &lt;em&gt;k&lt;/em&gt; folds (or partitions), and train/test the model &lt;em&gt;k&lt;/em&gt; times using different folds. When you're done, you can compute the mean performance and a variance that will be of utmost importance assessing confidence in the generalization performance estimate.&lt;/p&gt;&#xA;" OwnerUserId="17461" LastActivityDate="2016-04-10T01:25:29.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11117" PostTypeId="1" AcceptedAnswerId="11160" CreationDate="2016-04-10T04:19:59.357" Score="1" ViewCount="731" Body="&lt;p&gt;I want to create a data visualization on medical data i.e. patient's medical history, allergies one is having, any chronic patient's name must be highlighted, etc. And, there can be a separate visualisation for the medicines, which shows medicines' availability, their expiry date can be highlighted by using mouse_hover_on functionality.  It will be useful for the doctors and the medical staff as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problems:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bl.ocks.org/&quot; rel=&quot;nofollow&quot;&gt;bl.ocks.org&lt;/a&gt; has variety of base visualization models to work upon, but the schema of database I have (given by my mentor) is a bit confusing.&lt;/li&gt;&#xA;&lt;li&gt;Also, he won't provide us with the actual data of our college due to its confidential nature, so can we get dummy data on the net?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Experience:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have done work on &lt;a href=&quot;https://en.wikipedia.org/wiki/Weka_(machine_learning)&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; earlier to understand about data mining, and many different algorithms and classifiers.&lt;/li&gt;&#xA;&lt;li&gt;I have built a generic data visualization model (for hierarchical data) using the &lt;a href=&quot;http://d3js.org&quot; rel=&quot;nofollow&quot;&gt;D3.js&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;I have good working knowledge of Python. So, if anyone could please tell whether there is any hope for doing it(the data visualisation) in python.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10834" LastActivityDate="2016-04-12T08:26:27.097" Title="How to create visualisation on medical data?" Tags="|python|visualization|javascript|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11118" PostTypeId="1" AcceptedAnswerId="23897" CreationDate="2016-04-10T05:06:57.810" Score="13" ViewCount="595" Body="&lt;p&gt;I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The program AlphaGo has been built using, among other things (Monte-Carlo exploration of trees, etc.), neural networks which are trained from a huge database of human-played go games, and which are then reinforced by letting play versions of the program against itself many times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I wonder what would happen is we tried to build such a program without human database, i.e. starting with a basic program of Go just knowing rules and some method to explore trees, and letting play against itself to improve its neural network. Will we, after many games against itself, arrive at a program able to compete with or beat the best human players? And if so, how many games (in order of magnitude) would be needed for that? Or on the contrary, will such a program converge toward a much weaker player?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that the experiment has not been made, since AlphaGo is so recent. But the answer may nevertheless be obvious to a specialist. Otherwise any educated guess will interest me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can also ask the same question for &quot;simpler&quot; games. If we use roughly the same reinforcement-learning technics used for AlphaGo, but with &#xA;no use of human database, for a Chess program, would we eventually get&#xA;a program able to beat the best human? And if so, how fast? Has this been tried? Or if not for Chess, what about Checkers, or even simpler games? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot.   &lt;/p&gt;&#xA;" OwnerUserId="17752" LastActivityDate="2019-03-16T02:18:05.983" Title="AlphaGo (and other game programs using reinforcement-learning) without human database" Tags="|reinforcement-learning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11120" PostTypeId="2" ParentId="4963" CreationDate="2016-04-10T10:27:50.050" Score="2" Body="&lt;p&gt;Spark is based on Dryad research work &lt;/p&gt;&#xA;" OwnerUserId="17757" LastActivityDate="2016-04-10T10:27:50.050" CommentCount="0" CommunityOwnedDate="2016-04-10T10:27:50.050" ContentLicense="CC BY-SA 3.0" />
  <row Id="11121" PostTypeId="1" CreationDate="2016-04-10T10:38:12.783" Score="0" ViewCount="209" Body="&lt;p&gt;To be honest I find entire DS field so vague that I can't find actual roadmap of learning. I am actually a programmer for 10+ years and pretty much interested to get into &lt;code&gt;Data&lt;/code&gt; related field. My stats and math suck at this moment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is it Ok that I use my programming skills and jump right onto Machine Learning/Analysis tools like WEKA, Python Scikit etc and use their algorithm for understanding and keep studying/learning theory in parallel. Right now in terms of Data, I am already involved in:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data Processing(RDBMS)&lt;/li&gt;&#xA;&lt;li&gt;Data Scrapping(Python)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I am thinking the logical steps are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data Mining&lt;/li&gt;&#xA;&lt;li&gt;Data Analysis (sentiment analysis etc)&lt;/li&gt;&#xA;&lt;li&gt;Data Presentation/Visualization( D3 etc?)&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning Kit/Libraries&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have given myself a 6 month time to get into this field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="10879" LastActivityDate="2016-04-12T20:23:05.840" Title="A new comer in this field - How to start?" Tags="|bigdata|data|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11122" PostTypeId="1" CreationDate="2016-04-10T13:00:00.167" Score="4" ViewCount="189" Body="&lt;p&gt;I have a dataset of 30000 whole-genome sequence analysis. For each sample I have a text description that does not follow any fixed format.&#xA;What I want, is an annotation of each sample indicating the disease and the particular tissue that characterize the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not into text mining, so I don't know which tools I can use. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="11034" LastEditorUserId="16284" LastEditDate="2016-04-10T13:12:19.977" LastActivityDate="2017-05-30T14:50:16.093" Title="Automatic annotation of medical text data" Tags="|classification|text-mining|text|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11123" PostTypeId="2" ParentId="11118" CreationDate="2016-04-10T13:19:59.750" Score="8" Body="&lt;p&gt;As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states and actions covered by the RL algorithm are not simply the entire possible configuration of the game (Go has a huge complexity) but are based on a policy network and a value network, learned from real games and then improved by playing games AlphaGo vs AlphaGo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we might wonder if the training from real games is just a shortcut to save time or a necessary option to get such efficiency. I guess no one really know the answer, but we could state some assumptions. First, the human ability to promote good moves is due to much more complex intelligence than a simple neural net. For board games, it is a mix between memory, experience, logic and feelings. In this direction, I'm not sure the AlphaGo algorithm could build such a model without explicitly exploring a huge percentage of the entire configuration of the Go game (which is practically impossible). Current researches focus on building more complex representation of such a game, like relational RL or inductive logic learning. Then for simpler games (might be the case for chess but nothing sure), I would say that AlphaGo could retrieve similar techniques as humans by playing against itself, especially for openings (there are first only 10 moves available).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still it is only an opinion. But I'm quite sure that the key to answer your question resides in the RL approach that is nowadays still quite simple in term of knowledge. We are not really able to identify what makes us able to handle these games, and the best way we found until yet to defeat human is to roughly learn from him, and improve (a bit) the learned model with massive calculations.          &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-04-10T13:19:59.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11124" PostTypeId="2" ParentId="11104" CreationDate="2016-04-10T13:38:19.730" Score="2" Body="&lt;p&gt;To add some stuffs to what Neil Slater said:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First I'm not sure you got (as I read your question) the fact that SVMs are quite good, as long as you use them with a proper kernel. Which means that you must find a way of transforming your images such that the resulting input data are linearly separable in the sens of your formula. If the SVM algorithm is very simple, using kernel is nontrivial.&lt;/li&gt;&#xA;&lt;li&gt;Then the best approach nowadays for image classification is deep neural network. Not because they are magic but mostly because of the use of convolutional layers. Let say that for 10 000 neurons in a network, 100 will do what SVM do: classification. The rest of it just act as a kernel. In the case of images, since features to recognize are intricately local and global, the use of successive convolutional layers can decorticate your image to retrieve the specifications you need (in your example, you could recognize brushstroke as well as global shapes).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To sum up:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Shallow learning (SVM for example) work great but you have to design your own kernel, or treat your data before using them.&lt;/li&gt;&#xA;&lt;li&gt;Deep learning (convolutional networks for example) do the work for you, but are harder to set up and to interpret.       &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-04-10T13:38:19.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11125" PostTypeId="2" ParentId="758" CreationDate="2016-04-10T15:12:43.207" Score="6" Body="&lt;p&gt;I recently came across the following tool - &lt;a href=&quot;http://github.com/ctuning/ck&quot; rel=&quot;noreferrer&quot;&gt;http://github.com/ctuning/ck&lt;/a&gt; . It is already written in Python and seems to include what you need (my colleague is using it in the pilot project to automate image recognition). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pros: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;very small, portable and customizable &lt;/li&gt;&#xA;&lt;li&gt;includes web server to&#xA;distribute experiments and process them using predictive analytics&lt;/li&gt;&#xA;&lt;li&gt;has a cool usage example to crowdsource and reproduce compiler&#xA;optimization - &lt;a href=&quot;http://cknowledge.org/repo&quot; rel=&quot;noreferrer&quot;&gt;http://cknowledge.org/repo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a bit low level - you need to implement your own workflow from Python components shared via GitHub using JSON API or command line&lt;/li&gt;&#xA;&lt;li&gt;documentation is somewhat complex - I really hope that they will find time to update it soon.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17766" LastActivityDate="2016-04-10T15:12:43.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11126" PostTypeId="1" AcceptedAnswerId="11127" CreationDate="2016-04-10T17:28:10.910" Score="13" ViewCount="3788" Body="&lt;p&gt;I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for feedback on the car's surroundings. I'm using OpenCV to turn the video frames into tensors, and I'm using Google's TensorFlow to build a convoluted neural network to learn road boundaries and obstacles. My main question is, should I use supervised learning to teach the car to drive or should I provide objectives and penalties and do reinforcement learning (i.e, get to point B as fast as possible while not hitting anything and staying within the road boundaries)? Below is a list of the pros and cons that I've come up with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Supervised learning pros:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The inputs to the learning algorithm are pretty straightforward. The car learns to associate the video frame tensor and sensor distance readings with forward, backward, and angular wheel displacement&lt;/li&gt;&#xA;&lt;li&gt;I can more or less teach the car to drive exactly how I want (without overfitting, of course)&lt;/li&gt;&#xA;&lt;li&gt;I've done tons of supervised learning problems before, and this approach seems to comfortably fit my existing skill set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Supervised learning cons:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It's not clear how to teach speed, and the correct speed is pretty arbitrary as long as the car doesn't go so fast that it veers off the road. I suppose I could drive fast during training, but this seems like a crude approach. Maybe I could manually add in a constant variable during training that corresponds to the speed for that training session, and then when the learning algorithm is deployed, I set this variable according to the speed I want? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reinforcement learning pros:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If I build my car with the specific purpose of racing other people's self driving cars, reinforcement learning seems to be the natural way to tell my car to &quot;get there as fast as possible&quot; &lt;/li&gt;&#xA;&lt;li&gt;I've read that RL is sometimes used for autonomous drones, so in theory it should be easier in cars because I don't have to worry about up and down&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reinforcement learning cons:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I feel like reinforcement learning would require a lot of additional sensors, and frankly my foot-long car doesn't have that much space inside considering that it also needs to fit a battery, the Raspberry Pi, and a breadboard&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The car will behave very erratically at first, so much so that maybe it destroys itself. It might also take an unreasonably long time to learn (e.g., months or years)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;I can't incoporate explicit rules later on, e.g., stop at a toy red-light. With supervised learning, I could incorporate numerous SL algorithms (e.g., a Haar Cascade classifier for identifying stoplights) into a configurable rules engine that gets evaluated between each video frame. The rules engine would thus be able to override the driving SL algorithm if it saw a red stoplight even though the stoplight might not have been part of the training of the driving algorithm. RL seems too continous to do this (ie., stop only at the terminal state)&lt;/li&gt;&#xA;&lt;li&gt;I don't have a lot experience with applied reinforcement learning, although I definitely want to learn it regardless&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-20T15:56:33.747" Title="Supervised learning vs reinforcement learning for a simple self driving rc car" Tags="|reinforcement-learning|supervised-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11127" PostTypeId="2" ParentId="11126" CreationDate="2016-04-10T18:53:03.283" Score="13" Body="&lt;p&gt;I'd suggest you to try a hybrid approach:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First, &lt;a href=&quot;https://papers.nips.cc/paper/95-alvinn-an-autonomous-land-vehicle-in-a-neural-network&quot;&gt;train your car in supervised fashion by demonstration&lt;/a&gt;. Just&#xA;control it and use your commands as labels. This will let you get all&#xA;the pros of SL.&lt;/li&gt;&#xA;&lt;li&gt;Then, fine tune your neural net using reinforcement learning. You&#xA;don't need extra sensors for that: the rewards may be obtained from&#xA;distance sensors (larger distances = better) and from the speed&#xA;itself. This will give you the pros of RL and train your NN to the correct goal of driving fast while avoiding obstacles instead of the goal of imitating you.&lt;/li&gt;&#xA;&lt;li&gt;Combining both approaches will get you the pros of both SL and RL&#xA;while avoiding their cons. RL won't start from random behavior, just&#xA;small gradual deviations from what you tought the NN. A similar approach was applied successfuly by Google DeepMind with &lt;a href=&quot;http://www.nature.com/nature/journal/v529/n7587/full/nature16961.html&quot;&gt;AlphaGo&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;You can always include explicit rules on top of this. Implement them with high priority and call the NN only when there is no explicit rule for current situation. This is reminiscent of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Subsumption_architecture&quot;&gt;Subsumption Architecture&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9814" LastActivityDate="2016-04-10T18:53:03.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11128" PostTypeId="1" CreationDate="2016-04-10T20:24:40.857" Score="2" ViewCount="132" Body="&lt;p&gt;I've been trying to extend the LDA and wanted some help, direction and insight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can Author-Topic LDA be used as a document &quot;category&quot; model? The premise of the Author-Topic model is that multiple authors influence the Topic from which every word is derived. Can I instead treat &quot;authors&quot; as categories to which documents belong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to slightly alter this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://yaowu.co/docs/sigir12.pdf&quot; rel=&quot;nofollow&quot;&gt;http://yaowu.co/docs/sigir12.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://papers.nips.cc/paper/2994-modeling-general-and-specific-aspects-of-documents-with-a-probabilistic-topic-model.pdf&quot; rel=&quot;nofollow&quot;&gt;http://papers.nips.cc/paper/2994-modeling-general-and-specific-aspects-of-documents-with-a-probabilistic-topic-model.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with Author-Topic model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm trying to do is find out document-level word associations in the posting-data of a forum on a specialized site. For example, this site is about Candy - its members post about sweets, candy and toffee. Obviously there are categories of candies, then brands and then the products themselves. My reasoning is that if I use Author-Topic and treat the observed node &quot;author&quot; as observed-node &quot;candy categories&quot;, then I can introduce that candy category feature into recommender system down the pipeline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like the papers above suggest, a &quot;word stream&quot;, &lt;em&gt;switched&lt;/em&gt; by a multinomial 'x' decides which stream a word comes from (background, document or 'candy category').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think this makes sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, is there a book or resource that, very thoroughly and step-by-step explains the Collapsed Gibbs derivation for LDA?&lt;/p&gt;&#xA;" OwnerUserId="17776" LastActivityDate="2016-04-10T20:24:40.857" Title="Extending Author-Topic LDA" Tags="|topic-model|lda|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11129" PostTypeId="1" CreationDate="2016-04-11T03:57:40.190" Score="2" ViewCount="2812" Body="&lt;p&gt;I have a set of timeseries binary (boolean) data, with intervals of 1 day. Each day can either be 1 or 0 (true/false). What is the best way to forecast the next day/week's data based on the data I already have? &lt;/p&gt;&#xA;" OwnerUserId="17780" LastEditorUserId="11097" LastEditDate="2016-07-06T12:08:21.033" LastActivityDate="2018-06-24T12:53:25.140" Title="How to best forecast simple binary data?" Tags="|time-series|beginner|categorical-data|forecast|" AnswerCount="3" CommentCount="8" ClosedDate="2018-06-24T21:16:44.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="11130" PostTypeId="1" AcceptedAnswerId="11153" CreationDate="2016-04-11T05:05:11.047" Score="7" ViewCount="952" Body="&lt;p&gt;What methods exist for distance calculation in clustering? like Manhattan, Euclidean, etc.? &#xA;Plus, I don't know when I should use them. I always use Euclidean distance.&lt;/p&gt;&#xA;" OwnerUserId="15569" LastEditorUserId="67328" LastEditDate="2019-03-23T10:49:16.103" LastActivityDate="2019-03-23T10:49:16.103" Title="What methods exist for distance calculation in clustering? when we should use each of them?" Tags="|clustering|distance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11131" PostTypeId="1" CreationDate="2016-04-11T05:22:38.523" Score="5" ViewCount="381" Body="&lt;p&gt;Is it possible to identify the point in time where the cluster separation is at its most in a discrete time series clustering? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have 4 clusters of discrete time series and I want to pick a point in time where I can tell with the least bias which cluster it belongs to after a kmeans clustering, what other criteria than classification success can U use to identify my cluster separation performance?&lt;/p&gt;&#xA;" OwnerUserId="17781" LastEditorUserId="8878" LastEditDate="2017-01-12T08:47:45.210" LastActivityDate="2019-12-09T11:02:05.287" Title="Clustering efficiency in a discrete time-series" Tags="|clustering|time-series|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11133" PostTypeId="2" ParentId="11130" CreationDate="2016-04-11T06:12:06.100" Score="1" Body="&lt;p&gt;There are two method which are widely used for calculating distance in the domain of clustering. They are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Manhattan Distance&lt;/li&gt;&#xA;&lt;li&gt;Euclidean Distance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, there is no clear directive as to which of the above to select, so &lt;a href=&quot;https://stats.stackexchange.com/a/80387/84191&quot;&gt;this post&lt;/a&gt; might be helpful to you regarding the same. Generally, the distance metric depends on the problem statement and the type of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the euclidean and cosine distance are used when the data is dense and sparse respectively.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I always use euclidean distance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I wouldn't blame you for that. However, when calculating cartesian distance (like in the case of Recommender systems), the Euclidean distance is preferred.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-11T09:03:10.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11134" PostTypeId="1" CreationDate="2016-04-11T09:10:18.440" Score="12" ViewCount="7442" Body="&lt;p&gt;I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text or document.&lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;My problem is that there are too many features from a document. In my dataset, each document has more than 1000 tokens/words. To feed each example to a CNN, I convert each document into a matrix by using &lt;strong&gt;word2vec&lt;/strong&gt; or &lt;strong&gt;glove&lt;/strong&gt; resulting a big matrix. For each matrix, the height is the length of the document, and the width is the size of word embedding vector. My dataset has more than 9000 examples and it takes a lot of time to train the network (a whole week) which makes it difficult to fine-tune parameters.&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;Another feature extracting method is to use one-hot vector for each word, but this will create very sparse matrices. And of course, this method even takes more time to train than the previous method.&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;So is there a better method for extracting features without creating large input matrices?&#xA;&lt;br&gt; And how should we handle variable length of documents? Currently, I add special strings to make document have the same length, but I don't think it's a good solution.&lt;/p&gt;&#xA;" OwnerUserId="17784" LastEditorUserId="29169" LastEditDate="2021-02-16T15:49:42.590" LastActivityDate="2021-02-16T15:49:42.590" Title="Document classification using convolutional neural network" Tags="|neural-network|classification|text-mining|convolutional-neural-network|word2vec|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="11135" PostTypeId="2" ParentId="10637" CreationDate="2016-04-11T09:37:41.670" Score="1" Body="&lt;p&gt;I hope you do understand, that trying to predict some &lt;em&gt;abstract&lt;/em&gt; TF-IDF vector based on read article you are trying to teach a neural network to &lt;strong&gt;produce&lt;/strong&gt; a new article for a given user. It would be a major step in artificial intelligence if achieved, so keep us posted on your progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you would think of alternative approaches, I would suggest following steps to make your neural network to work:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You need a classification/clustering model for your articles (which can be based on your TF*IDF, or on some tags assigned to articles). The more classes you could squeeze from your model, the better would your resulting recommender be. On that step, you should be at least able to provide user with articles similar by topic.&lt;/li&gt;&#xA;&lt;li&gt;You will need at least two-hop data set of user reading current article and hopping to the next one which would be used as a training set for actual predictor. Here, you will train a neural network (or any other model) which will try to predict a next-article class (from step one) based on current article - that would be a recommender system.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Of course, you may, given you have data from (2), train a neural network that would answer the question of &quot;&lt;em&gt;If user reads &lt;strong&gt;article A&lt;/strong&gt;, will the next article contain &lt;strong&gt;word B&lt;/em&gt;&lt;/strong&gt;&quot;, but that would require you to have the size of output variables of your TF*IDF matrix (for any reasonable dataset that would be at least 20-30k words unless you work with books for toddlers) which makes your model overcomplicated (that would be, actually, a TF*IDF matrix size number of models for every word).&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-04-11T09:37:41.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11136" PostTypeId="1" AcceptedAnswerId="11138" CreationDate="2016-04-11T10:18:02.017" Score="10" ViewCount="582" Body="&lt;p&gt;In the past, when trying different machine learning algorithms in order to solve a problem, I used to write drown the set of approaches on a notebook, keeping details such as features, feature preprocessing, normalization, algorithms, algorithm parameters... therefore, building a hand-written logbook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, currently I'm concerned about using a 'more professional' tool, so that I can keep more details and even share it with other team-members, who are also able to stamp their approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great an automated and collaborative tool that keep track of the work done, considering details like: features, algorithms, algorithms parameter, data pre-process, data, metrics... beyond a collaborative Google Drive Spreadsheet for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are you solving this? How are you keeping track of the work done? What's your &lt;strong&gt;logbook&lt;/strong&gt; tool?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance.&lt;/p&gt;&#xA;" OwnerUserId="15762" LastActivityDate="2020-08-20T17:06:09.327" Title="Logbook: Machine Learning approaches" Tags="|machine-learning|algorithms|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11137" PostTypeId="2" ParentId="11121" CreationDate="2016-04-11T10:52:56.247" Score="1" Body="&lt;p&gt;I would suggest you taking a Machine Learning and Statistics (Coursera, Udacity etc.) course straight away, since you have a profound experience in programming, so mastering any library wouldn't be of a big issue. Understanding of theoretical approaches is way more important than a particular library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Main reason for that is the fact, that tutors of courses usually provide you with data big enough to understand the purpose of each algorithm/approach in the session/assignment so you will not only become proficient in some ML approach, but also understand which type of data can be then handled by a particular algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I suggest you select a particular field of data science / machine learning, since data science, machine learning, statistics and so on, are just powerful tools and their application in computer vision, statistical modelling, natural language processing, fraud detection and other fields share something in common, but focusing on one field will allow you to progress faster.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-04-11T10:52:56.247" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11138" PostTypeId="2" ParentId="11136" CreationDate="2016-04-11T11:11:56.190" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How are you solving this? How are you keeping track of the work done?&#xA;  What's your logbook tool?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This might not be the best approach. But, this is how my team does it. We believe that for pulling off an end-to-end data science experiment, proper conscience is very important. So, we use &lt;a href=&quot;https://slack.com/&quot;&gt;Slack&lt;/a&gt; for the same for our discussions and the meetings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to them, we have Rmd (R markdown) files for documenting the planning and the analysis parts.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-04-11T11:11:56.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11139" PostTypeId="2" ParentId="11136" CreationDate="2016-04-11T11:39:47.907" Score="3" Body="&lt;p&gt;Check &lt;a href=&quot;http://www.openml.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; out, looks like exactly what you need&lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="98307" LastEditDate="2020-08-20T17:06:09.327" LastActivityDate="2020-08-20T17:06:09.327" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="11141" PostTypeId="2" ParentId="11121" CreationDate="2016-04-11T13:04:06.620" Score="1" Body="&lt;p&gt;Given your programming exposure, You can jump into machine learning directly,&#xA;However, here are some resources that will help you in getting started with data science:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An edX course that will help you start out in data science :- &lt;a href=&quot;https://www.edx.org/course/analytics-edge-mitx-15-071x-2&quot; rel=&quot;nofollow&quot;&gt;The Analytics Edge&lt;/a&gt;&#xA;You could try &lt;a href=&quot;https://www.edx.org/course/machine-learning-data-science-analytics-columbiax-ds102x-0&quot; rel=&quot;nofollow&quot;&gt;Machine learning for data science&lt;/a&gt; at a later stage, there's a fabulous coursera course for the same as well.&#xA;Apart from that, 'Data science from scratch' is a good book to start with!&#xA;You might also want to read a few non-technical books explaining the applications of data science so that you could choose a field and focus on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'Data mining: Concepts and Techniques' is a great book to start learning data mining and then gradually you can go deeper(This book goes pretty deep although).&lt;/p&gt;&#xA;" OwnerUserId="17699" LastActivityDate="2016-04-11T13:04:06.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11143" PostTypeId="2" ParentId="11121" CreationDate="2016-04-11T13:41:11.920" Score="1" Body="&lt;p&gt;Jump right in by joining a Kaggle competition: &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/&lt;/a&gt;. You'll learn a lot by doing, and the most valuable skills are those that you can apply. The theory will come naturally from experience. Also, if you do well enough in Kaggle, you can win some serious money.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not Kaggle, I recommend finding some personal project that interests you and will keep you motivated enough to get through the tough parts. For example, I'm building a self driving remote control car so that I can learn more about convoluted neural nets. The car is cool, but the neural nets are difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, it will probably take you 3-4 years to be truly comfortable with machine learning. The 6 month estimate you've provided sounds good for one project though. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-04-11T17:44:45.377" LastActivityDate="2016-04-11T17:44:45.377" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11144" PostTypeId="2" ParentId="10521" CreationDate="2016-04-11T13:42:31.047" Score="1" Body="&lt;p&gt;Have a look on &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow&quot;&gt;conditional random field&lt;/a&gt;. Conditional random fields  are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2016-04-11T13:42:31.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11145" PostTypeId="2" ParentId="11121" CreationDate="2016-04-11T15:33:27.210" Score="0" Body="&lt;p&gt;According to me, learning data science involves weaving your way through these three:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Business domain knowledge&lt;/li&gt;&#xA;&lt;li&gt;Math&lt;/li&gt;&#xA;&lt;li&gt;Technology&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Going through each of these in detail:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Business domain knowledge - There is no 'given way' of going about gaining this. This will come to you as you solve a given problem and try to understand the business around it to be able to better understand the problem you are solving.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Math - I believe 'Elements of statistical learning' by Hastie et. all is a very good place to begin. The same authors have a less math intensive version of the same book called ' Introduction to statistical learning' with R.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Technology - This comes as a personal choice, how many tools and what kind of tools do you want to have under your skill set. R and python are most popular now, but you can see what suits you the best&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16704" LastActivityDate="2016-04-11T15:33:27.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11146" PostTypeId="1" AcceptedAnswerId="11147" CreationDate="2016-04-11T16:25:23.597" Score="2" ViewCount="111" Body="&lt;p&gt;Having multiple sets of categories from different listings sites (e.g. Yelp, yellowpages.com,  Google My Business...).  I want to figure out what category X on one site is on another site.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have hundreds of thousands of businesses and the categories on all the sites they are on, so we could see that &quot;Galbi Foo Restaurant&quot; is in category &quot;Restaurants &gt; Korean&quot; on one site and &quot;Restaurants&quot; on the other.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples category mappings that will have to happen:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Nail Salons = Manicure &amp;amp; Pedicure&lt;/li&gt;&#xA;&lt;li&gt;Eyelash Service = Visagist&lt;/li&gt;&#xA;&lt;li&gt;Tanning = Sunbed Salon&lt;/li&gt;&#xA;&lt;li&gt;Specialty Food = Grocery (Specialty Food child node doesn't exist)&lt;/li&gt;&#xA;&lt;li&gt;Diagnostic Imaging = Radiologist&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Where would I start to solve this?  It seems like a classification (logistic regression) problem.  But this ML stuff hasn't clicked with me yet, so I'm likely to drastically over or under complicate these things :). &lt;/p&gt;&#xA;" OwnerUserId="17795" LastEditorUserId="17795" LastEditDate="2016-04-12T02:32:16.500" LastActivityDate="2016-04-12T02:32:16.500" Title="How would I map categories between two similar but different sets of categories" Tags="|machine-learning|classification|beginner|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11147" PostTypeId="2" ParentId="11146" CreationDate="2016-04-11T17:05:12.603" Score="1" Body="&lt;p&gt;This sounds like a pretty standard supervised learning problem. In this case, your records would be businesses on site X and their actual category on site Z. Your predictors would be tags/categories for a particular business on site X, and your target variable, y (i.e., what you're trying to predict), would be the category on the other website. As far as the code goes, you have a variety of options depending on your preferred language. You could use the caret package in R, the scikit-learn library in Python, or the Weka library (maybe even Spark's ML lib because of its simplicity) in Java/Scala. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note, in your question I think you meant to say &quot;logistic regression&quot; instead of &quot;logical regression&quot;. You don't need to use logistic regression (although it wouldn't hurt). You could also try algorithms like Random Forests or Naive Bayes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also worth noting: your target variable will have many classes (ie every possible category for the site you're trying to predict), so don't get alarmed if it seems like there are a lot of classes. That's normal for a problem like the one you've described.   &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-04-11T17:05:12.603" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11148" PostTypeId="2" ParentId="11136" CreationDate="2016-04-11T20:03:53.147" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How are you solving this? How are you keeping track of the work done? What's your logbook tool?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For &lt;a href=&quot;http://arxiv.org/abs/1511.09030&quot; rel=&quot;nofollow&quot;&gt;my bachelors thesis&lt;/a&gt; (&lt;a href=&quot;http://www.martin-thoma.de/write-math/classify/&quot; rel=&quot;nofollow&quot;&gt;write-math.com&lt;/a&gt;) I wrote my own little toolkit to go through different models / preprocessing steps very fast. Each experiment had one configuration file (see &lt;a href=&quot;https://github.com/MartinThoma/hwr-experiments&quot; rel=&quot;nofollow&quot;&gt;hwr-experiments repository&lt;/a&gt;). For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data-source: feature-files/baseline-3-points&#xA;training: '{{nntoolkit}} train --epochs 1000 --learning-rate 0.1 --momentum 0.1 --print-errors --hook=''!detl&#xA;    test {{testing}},err=testresult_%e.txt'' {{training}} {{validation}}&#xA;    {{testing}} &amp;lt; {{src_model}} &amp;gt; {{target_model}} 2&amp;gt;&amp;gt; {{target_model}}.log'&#xA;model:&#xA;    type: mlp&#xA;    topology: 24:500:369&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The trained model is stored; it is pretty fast to get the evaluation results (e.g. accuracy, confuscation matrix).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-04-11T20:03:53.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11150" PostTypeId="2" ParentId="10258" CreationDate="2016-04-11T20:56:59.033" Score="3" Body="&lt;p&gt;Trifacta (&lt;a href=&quot;https://www.trifacta.com/&quot; rel=&quot;nofollow&quot;&gt;https://www.trifacta.com/&lt;/a&gt;) supposedly can do that (learn from some examples provided by the user. I have no interest in the company, but it comes from academic research that I'm familiar with (&lt;a href=&quot;http://vis.stanford.edu/wrangler/&quot; rel=&quot;nofollow&quot;&gt;http://vis.stanford.edu/wrangler/&lt;/a&gt;). Google had something free online (Google Refine) that can also do some intuitive things very simply, but I don't think it's as well developed.&#xA;If you try either product, please let us know about your experience!&lt;/p&gt;&#xA;" OwnerUserId="17800" LastActivityDate="2016-04-11T20:56:59.033" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11151" PostTypeId="1" AcceptedAnswerId="11152" CreationDate="2016-04-11T21:04:54.823" Score="0" ViewCount="293" Body="&lt;p&gt;I'm trying to analyze my training model, Google Prediction API provides analyze method to get insights for the model.&#xA;Currently I want to improve confidence levels, for my predictions, I haven't found any documentation in how to read this ConfusionMatrix, any insights will be great:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA; &quot;kind&quot;: &quot;prediction#analyze&quot;,&#xA; &quot;id&quot;: &quot;modelX&quot;,&#xA; &quot;selfLink&quot;: &quot;https://www.googleapis.com/prediction/v1.6/projects/projectX/trainedmodels/modelX/analyze&quot;,&#xA; &quot;dataDescription&quot;: {&#xA;  &quot;outputFeature&quot;: {&#xA;   &quot;text&quot;: [&#xA;    {&#xA;     &quot;value&quot;: &quot;labelA&quot;,&#xA;     &quot;count&quot;: &quot;681&quot;&#xA;    },&#xA;    {&#xA;     &quot;value&quot;: &quot;labelB&quot;,&#xA;     &quot;count&quot;: &quot;127&quot;&#xA;    },&#xA;    {&#xA;     &quot;value&quot;: &quot;labelC&quot;,&#xA;     &quot;count&quot;: &quot;814&quot;&#xA;    },&#xA;    {&#xA;     &quot;value&quot;: &quot;labelD&quot;,&#xA;     &quot;count&quot;: &quot;427&quot;&#xA;    }&#xA;   ]&#xA;  },&#xA;  &quot;features&quot;: [&#xA;   {&#xA;    &quot;index&quot;: &quot;0&quot;,&#xA;    &quot;text&quot;: {&#xA;     &quot;count&quot;: &quot;2049&quot;&#xA;    }&#xA;   }&#xA;  ]&#xA; },&#xA; &quot;modelDescription&quot;: {&#xA;  &quot;modelinfo&quot;: {&#xA;   &quot;kind&quot;: &quot;prediction#training&quot;&#xA;  },&#xA;  &quot;confusionMatrix&quot;: {&#xA;   &quot;labelA&quot;: {&#xA;    &quot;labelA&quot;: &quot;14.17&quot;,&#xA;    &quot;labelB&quot;: &quot;0.17&quot;,&#xA;    &quot;labelC&quot;: &quot;3.83&quot;,&#xA;    &quot;labelD&quot;: &quot;0.67&quot;&#xA;   },&#xA;   &quot;labelB&quot;: {&#xA;    &quot;labelA&quot;: &quot;0.50&quot;,&#xA;    &quot;labelB&quot;: &quot;2.00&quot;,&#xA;    &quot;labelC&quot;: &quot;1.33&quot;,&#xA;    &quot;labelD&quot;: &quot;0.00&quot;&#xA;   },&#xA;   &quot;labelC&quot;: {&#xA;    &quot;labelA&quot;: &quot;1.17&quot;,&#xA;    &quot;labelB&quot;: &quot;0.00&quot;,&#xA;    &quot;labelC&quot;: &quot;70.00&quot;,&#xA;    &quot;labelD&quot;: &quot;3.50&quot;&#xA;   },&#xA;   &quot;labelD&quot;: {&#xA;    &quot;labelA&quot;: &quot;1.17&quot;,&#xA;    &quot;labelB&quot;: &quot;0.17&quot;,&#xA;    &quot;labelC&quot;: &quot;4.17&quot;,&#xA;    &quot;labelD&quot;: &quot;12.17&quot;&#xA;   }&#xA;  },&#xA;  &quot;confusionMatrixRowTotals&quot;: {&#xA;   &quot;labelA&quot;: &quot;18.83&quot;,&#xA;   &quot;labelB&quot;: &quot;3.83&quot;,&#xA;   &quot;labelC&quot;: &quot;74.67&quot;,&#xA;   &quot;labelD&quot;: &quot;17.67&quot;&#xA;  }&#xA; }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2016-04-11T21:47:42.020" Title="Understanding ConfusionMatrix for Google Prediction API" Tags="|machine-learning|confusion-matrix|google-prediction-api|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11152" PostTypeId="2" ParentId="11151" CreationDate="2016-04-11T21:47:42.020" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://cloud.google.com/prediction/docs/reference/v1.6/trainedmodels/analyze&quot; rel=&quot;nofollow&quot;&gt;As explained in the documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This shows an estimate for how this model will do in predictions. This is first indexed by the true class label. For each true class label, this provides a pair {predicted_label, count}, where count is the estimated number of times the model will predict the predicted label given the true label.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are not sure what a confusion matrix, &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;nofollow&quot;&gt;see Wikipedia&lt;/a&gt;, where the &quot;actual class&quot; refers to the same thing as the &quot;true class&quot; in the Google documentation.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-04-11T21:47:42.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11153" PostTypeId="2" ParentId="11130" CreationDate="2016-04-11T22:55:44.767" Score="8" Body="&lt;p&gt;Well, there is a book called&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Deza, Michel Marie, and Elena Deza.&lt;br /&gt; &lt;strong&gt;Encyclopedia of distances&lt;/strong&gt;.&lt;br /&gt; Springer Berlin Heidelberg, 2009. ISBN 978-3-642-00233-5&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I guess that book answers your question better than I can...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose the distance function most appropriate for your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, on latitude and longitude, use a distance like Haversine. If you have enough CPU, you can use better approximations such as Vincenty's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On histograms, use a distribution-baes distance. Earth-movers (EMD), divergences, histogram intersection, quadratic form distances, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On binary data, for example Jaccard, Dice, or Hamming make a lot of sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On non-binary sparse data, such as text, various variants of tf-idf weights and cosine are popular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably the best tool to experiment with different distance functions and clustering is ELKI. It has &lt;a href=&quot;http://elki.dbs.ifi.lmu.de/wiki/DistanceFunctions&quot;&gt;many many distances&lt;/a&gt;, and many clustering algorithms that can be used with all of these distances (e.g. OPTICS). For example Canberra distance worked very well for me. That is probably what I would choose as &quot;default&quot;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2016-04-12T08:00:57.687" LastActivityDate="2016-04-12T08:00:57.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11154" PostTypeId="2" ParentId="11096" CreationDate="2016-04-11T23:03:49.410" Score="1" Body="&lt;p&gt;This is quite difficult to do without first structuring your dataset. There's a reason cleaned datasets cost thousands of dollars because they try to clean these issues for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can try is first creating a taxonomy system. First you give the general &quot;Stanford university&quot; an ID of &quot;1&quot;. Something like the &quot;Stanford graduate school of business&quot; would get ID &quot;1.2.5&quot;, where the new &quot;2&quot; refers to the graduate school division, &quot;5&quot; refers to the business school category. It really depends on what your ultimate goal is. In short, set up a list of ID's for possible subdivisions, &quot;Graduate, undergraduate, etc.&quot; and then further subdivisions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For locations, usually you can define a &quot;primary&quot; and &quot;secondary&quot; location, i.e. &quot;Palo Alto&quot; and &quot;Stanford&quot; , which you can determine by histogramming counts for each and selecting the top two. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For correcting misspellings, you could use google search API and exploit the &quot;Showing results for...&quot; which would give you the correct spelling. &lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-04-11T23:03:49.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11155" PostTypeId="2" ParentId="11091" CreationDate="2016-04-12T03:00:41.800" Score="10" Body="&lt;p&gt;I'd want to add @Neil_Slater's answer by sharing my application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my application, I want to train a model that can automatically load a chess position from a chess book like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3FqZY.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3FqZY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Before I did anything, I made sure I had a model that can accurately detect a chess piece.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RLFJY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RLFJY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was not a hard-problem because it was like training the MINST digits. I collected enough samples, randomly add some noise to those samples. My model was a 2-layer convolutional deep-learning.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Since chess board is always a square. I use square-detection available in OpenCV to give me a list of candidates. I would throw away any square that is too small, too large or not divisible by 64 (since there are 64 squares).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next, I'd crop the image to something like this:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6tlaO.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6tlaO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have &lt;strong&gt;another&lt;/strong&gt; multi-layer convolutional network to check each square in the board. The stride length is the dimension of the image divided by 8 (since there're eight squares in each dimension). The patch size is the same as the stride length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My pipework worked as I was able to &lt;strong&gt;combine&lt;/strong&gt; two different classifiers. I personally prefer to train two classifiers, as it'd be easier to train and verify than trying to put everything into a single model.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-04-12T03:12:48.237" LastActivityDate="2016-04-12T03:12:48.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11156" PostTypeId="2" ParentId="11060" CreationDate="2016-04-12T04:30:58.213" Score="1" Body="&lt;p&gt;The variable groups may be multicollinear or the conversion between sparse and dense might go wrong. Have you thought about using a voting classifier/ ensemble classification? &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/ensemble.html&lt;/a&gt;&#xA;That way you could deal with both above problems. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-04-12T04:30:58.213" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11157" PostTypeId="2" ParentId="11060" CreationDate="2016-04-12T04:44:04.533" Score="4" Body="&lt;p&gt;The best way to combine features is through ensemble methods.&#xA;Basically there are three different methods: bagging, boosting and stacking.&#xA;You can either use Adabbost augmented with feature selection (in this consider both sparse and dense features) or stacking based (random feature - random subspace) &#xA;I prefer the second option you can train a set of base learners ( decisions. Trees) by using random subsets and random feature ( keep training base learners until you cover the whole set of features) &#xA;The next step is to test the Training set to generate the meta data. Use this meta data to train a meta classifier. &#xA;The meta classifier will figure out which feature is more important and what kind of relationship should be utilized &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-12T04:44:04.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11158" PostTypeId="2" ParentId="11130" CreationDate="2016-04-12T04:49:29.277" Score="0" Body="&lt;p&gt;I wanna highlight that in addition to the well known distances:&#xA; Manhattan Distance&#xA;Euclidean Distance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Symmetric kl-d can be used when you are clustering distributions.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-12T04:49:29.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11159" PostTypeId="1" CreationDate="2016-04-12T06:40:47.977" Score="0" ViewCount="7955" Body="&lt;p&gt;I am doing a project to predict the outcome of a cricket match, I have the data that states which matches were won by whom for ODIs. [Espn data]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm could be used to predict the outcome of the coming matches?&#xA;Would Quadratic Regression be a good idea? Or would predicting based on probability algorithms such as the markov's algorithm is what is generally used?&#xA;Any other algorithm I should use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, I want to know which algorithm I should use, I will implement in C++ ultimately but I will do it in R or python first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a newbie in this field, hence pardon if the question sounds too stupid.&#xA;I have learnt regression so far in data analytics.&lt;/p&gt;&#xA;" OwnerUserId="17699" LastActivityDate="2019-06-18T11:22:04.730" Title="What algorithms can be used to predict the outcome of a cricket match?" Tags="|machine-learning|predictive-modeling|regression|linear-regression|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11160" PostTypeId="2" ParentId="11117" CreationDate="2016-04-12T08:26:27.097" Score="4" Body="&lt;p&gt;If you have good working knowledge in Python than you are good for it. &lt;a href=&quot;https://d3js.org/&quot; rel=&quot;nofollow&quot;&gt;D3.js&lt;/a&gt; actually has a Python comparable called &lt;a href=&quot;http://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Bokeh&lt;/a&gt;. You question implies a lot of ramifications and does explain fully what you are trying to achieve so let's go by parts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some 2D Scientific Plotting Python Libraries:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.org/&quot; rel=&quot;nofollow&quot;&gt;Matplotlib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ggplot.yhathq.com/&quot; rel=&quot;nofollow&quot;&gt;ggplot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Bokeh&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.enthought.com/projects/chaco/&quot; rel=&quot;nofollow&quot;&gt;Chaco&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.pyqtgraph.org/&quot; rel=&quot;nofollow&quot;&gt;pyQtGraph&lt;/a&gt; (This one has some interesting features in volume slicing if it suits you)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some UI development Python libraries:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.riverbankcomputing.com/software/pyqt/download5&quot; rel=&quot;nofollow&quot;&gt;PyQt5&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.riverbankcomputing.com/software/pyqt/download&quot; rel=&quot;nofollow&quot;&gt;PyQt4&lt;/a&gt; (there are significant changes to PyQt5, thus the suggestion)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://wiki.qt.io/PySide&quot; rel=&quot;nofollow&quot;&gt;PySide&lt;/a&gt; (also a Qt port)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.wxpython.org/&quot; rel=&quot;nofollow&quot;&gt;wxPython&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some Python numerical libraries:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Pandas&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;Numpy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;Scipy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;Scikit-learn&lt;/a&gt; (You can do some serious data analysis with this)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Using combinations of the above you can build some very powerful software solutions. Any of the UI libraries suggested has spreadsheet widgets. There are libraries to read &lt;a href=&quot;https://docs.python.org/2/library/csv.html&quot; rel=&quot;nofollow&quot;&gt;csv&lt;/a&gt; and &lt;a href=&quot;http://www.python-excel.org/&quot; rel=&quot;nofollow&quot;&gt;excel&lt;/a&gt;, among others (&lt;a href=&quot;https://docs.python.org/2/library/json.html&quot; rel=&quot;nofollow&quot;&gt;JSON&lt;/a&gt;, &lt;a href=&quot;http://docs.h5py.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;HDF5&lt;/a&gt;, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data problem can be resolved by either building your own randomized data or by using resources on the net like &lt;a href=&quot;http://www.healthdata.gov/dataset&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; or &lt;a href=&quot;https://www.nlm.nih.gov/hsrinfo/datasites.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17806" LastActivityDate="2016-04-12T08:26:27.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11162" PostTypeId="2" ParentId="10960" CreationDate="2016-04-12T08:57:46.800" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow&quot;&gt;R&lt;/a&gt; is a more compact, target oriented, package. Good if you want to focus on very specific tasks (generally scientific). &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt;, on the other hand, is a general purpose language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, and obviously this is a matter of opinion, if you are an experienced developer go for Python. You'll have far more choices in libraries and a far bigger potential to build big software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some examples of 2D scientific plotting libraries:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.org/&quot; rel=&quot;nofollow&quot;&gt;Matplotlib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bokeh.pydata.org/en/latest/docs/gallery.html&quot; rel=&quot;nofollow&quot;&gt;Bokeh&lt;/a&gt; (targeted D3.js)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.enthought.com/projects/chaco/&quot; rel=&quot;nofollow&quot;&gt;Chaco&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ggplot.yhathq.com/&quot; rel=&quot;nofollow&quot;&gt;ggplot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stanford.edu/~mwaskom/software/seaborn/&quot; rel=&quot;nofollow&quot;&gt;Seaborn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.pyqtgraph.org/&quot; rel=&quot;nofollow&quot;&gt;pyQtGraph&lt;/a&gt; (some significant 3D features)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some examples of 3D scientific plotting libraries&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://vispy.org/&quot; rel=&quot;nofollow&quot;&gt;Vispy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://code.enthought.com/projects/mayavi/&quot; rel=&quot;nofollow&quot;&gt;Mayavi&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.vtk.org/&quot; rel=&quot;nofollow&quot;&gt;VTK&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://glumpy.github.io/&quot; rel=&quot;nofollow&quot;&gt;Glumpy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some examples of libraries typically used in Data Science in Python:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Pandas&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;Numpy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;Scipy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;Scikit-learn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-image.org/&quot; rel=&quot;nofollow&quot;&gt;Scikit-image&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also check the list for other relevant &lt;a href=&quot;https://scikits.appspot.com/scikits&quot; rel=&quot;nofollow&quot;&gt;Scikit packages&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for starting software I would advise you to use any of the already prepared Python distributions that already come with a bunch of scientific libraries inside as well as software such as IDEs. Some examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://winpython.github.io/&quot; rel=&quot;nofollow&quot;&gt;WinPython&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://python-xy.github.io/&quot; rel=&quot;nofollow&quot;&gt;Python XY&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.continuum.io/why-anaconda&quot; rel=&quot;nofollow&quot;&gt;Anaconda&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.enthought.com/products/canopy/&quot; rel=&quot;nofollow&quot;&gt;Canopy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Personally I'm a user of WinPython due to being portable (former user of Python XY, both are great). In any case these distributions will greatly simplify the task of having your scientific Python environment (so to speak) prepared. You just need to code. One IDE known to be specially good for scientists is &lt;a href=&quot;https://github.com/spyder-ide/spyder/&quot; rel=&quot;nofollow&quot;&gt;Spyder&lt;/a&gt;. Yet these ones also will work:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.pydev.org/&quot; rel=&quot;nofollow&quot;&gt;PyDev&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.jetbrains.com/pycharm/&quot; rel=&quot;nofollow&quot;&gt;PyCharm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.wingware.com/&quot; rel=&quot;nofollow&quot;&gt;WingIDE&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://komodoide.com/features/&quot; rel=&quot;nofollow&quot;&gt;Komodo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.visualstudio.com/en-us/features/python-vs.aspx&quot; rel=&quot;nofollow&quot;&gt;Python Tools for Visual Studio&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As for data visualization tips you'll see that the most common functions in the libraries mentioned above are also the most widely used. For instance a library like Pandas let's you call plots directly from the object so there is already an intuitive approach to data visualization. A library like scikit-learn (check the site) already shows examples followed by data visualization of the results. I wouldn't be too concerned about this point. You'll learn just by roaming a bit on the libraries documentation (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html#sklearn.cluster.DBSCAN&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="17806" LastActivityDate="2016-04-12T08:57:46.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11163" PostTypeId="1" AcceptedAnswerId="11164" CreationDate="2016-04-12T10:10:06.887" Score="1" ViewCount="340" Body="&lt;p&gt;I am using kernel regression to build a prediction model. For the same, I am using &lt;a href=&quot;https://cran.r-project.org/web/packages/np/np.pdf&quot; rel=&quot;nofollow&quot;&gt;np&lt;/a&gt; package. It is working fine, but I observed in multiple runs on the same data, it produces different results. Why it results in diverse outputs on the same data? Is there any way to select the best run of the model? Here is the minimal &lt;code&gt;R&lt;/code&gt; code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(np)&#xA;   bw.all = npregbw(formula=power ~ temperature  &#xA;                                  + prevday1 + prevday2&#xA;                                  + prev_instant1 + prev_instant2 &#xA;                                  + prev_2_hour,&#xA;                regtype=&quot;ll&quot;,bwmethod=&quot;cv.aic&quot;, data=new_tr_dat)&#xA;   model.np &amp;lt;- npreg(bws=bw.all)&#xA;   summary(model.np) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using following data for my experiments:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        power temperature   prevday1   prevday2 prev_instant1 prev_instant2 prev_2_hour&#xA;1   220.59680          38         NA         NA      648.3621     1392.2186    848.7299&#xA;2   584.06867          38  220.59680         NA     1012.6853      250.1150    434.7129&#xA;3   206.39849          40  584.06867  220.59680      169.9380      105.5796    127.7294&#xA;4   177.05559          39  206.39849  584.06867      167.6312      229.3927    249.9871&#xA;5   165.71996          41  177.05559  206.39849      214.8291      248.5378    247.0262&#xA;6   184.02724          44  165.71996  177.05559      256.9970      314.3742    485.5184&#xA;7   187.70557          43  184.02724  165.71996      125.6160      213.9993    174.0830&#xA;8   916.78484          43  187.70557  184.02724      668.2840      217.3451    423.8285&#xA;9   185.98017          42  916.78484  187.70557      295.7329      331.6580   1227.0293&#xA;10  490.42294          42  185.98017  916.78484      241.6590      249.0523    255.3110&#xA;11  703.92694          39  490.42294  185.98017      806.5259     1515.1619   1140.4415&#xA;12 2038.91747          37  703.92694  490.42294      232.5541      582.5105    632.7118&#xA;13  208.66049          26 2038.91747  703.92694      210.5353      217.5053    221.3938&#xA;14  281.89860          37  208.66049 2038.91747      796.4336      256.4664    603.0781&#xA;15  425.72868          32  281.89860  208.66049      250.6069      187.1751    260.0573&#xA;16   86.77193          36  425.72868  281.89860      174.1249      179.6437    164.4359&#xA;17  218.06322          39   86.77193  425.72868      223.6548      316.2230    322.8536&#xA;18  258.89159          43  218.06322   86.77193      233.4561      372.5123    256.8588&#xA;19 1436.19980          40  258.89159  218.06322     1266.2630     1387.2287    791.7056&#xA;20  261.68520          42 1436.19980  258.89159      278.3378      230.5614    262.0084&#xA;21  225.34517          44  261.68520 1436.19980      211.3332      147.6705    196.8328&#xA;22  852.68835          44  225.34517  261.68520     1271.5826     1233.7158    991.7835&#xA;23 1729.79826          44  852.68835  225.34517      945.6528      298.0929    412.2199&#xA;24  464.58053          43 1729.79826  852.68835      182.6507      184.3031    203.5395&#xA;25  902.30950          45  464.58053 1729.79826      308.1398     1743.3495    642.4563&#xA;26  428.18792          45  902.30950  464.58053      205.1806      697.9208   1434.5425&#xA;27 1508.74739          43  428.18792  902.30950     1371.0550     2165.7173   1918.5236&#xA;28  355.01704          42 1508.74739  428.18792     1750.3907     1740.4654   1022.5056&#xA;29 3248.62618          43  355.01704 1508.74739      686.8528      360.0539    660.6378&#xA;30 1949.63937          44 3248.62618  355.01704      258.4627      217.2683    232.3818&#xA;31  725.25368          40 1949.63937 3248.62618     1406.3282     1714.6412   1375.2824&#xA;32  261.31252          32  725.25368 1949.63937      553.0443      275.6697    409.9598&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13291" LastActivityDate="2016-04-12T10:49:56.930" Title="Kernel regression results in diverse outputs" Tags="|r|predictive-modeling|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11164" PostTypeId="2" ParentId="11163" CreationDate="2016-04-12T10:28:57.313" Score="2" Body="&lt;p&gt;You have a very small number of observations (32?) and a non-trivial number of predictors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is known that the cross-validation function possesses multiple local minima/maxima. If you increase the number of multistarts to, say, nmulti=100 (add this option to your call to npregbw()), you ought to see that the same results occur on each invocation of the optimization process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note you can do this all in the call to npreg() and skip the bandwidth call for convenience (npreg() will call npregbw() automatically but accept the arguments intended for npregbw()). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you will get the same results if you restart R each time you run the routine (seeds are set automatically to ensure this).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- npreg(power ~ temperature  &#xA;                     + prevday1 &#xA;                     + prevday2&#xA;                     + prev_instant1 &#xA;                     + prev_instant2 &#xA;                     + prev_2_hour,&#xA;                regtype=&quot;ll&quot;,&#xA;                bwmethod=&quot;cv.aic&quot;, &#xA;                nmulti=100,&#xA;                data=new_tr_dat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To see whether things are stable with respect to the number of multistarts, look at the value of the cross-validation function and summary provided. Also, you can look at partial regression plots along with resampled variability bounds via plot().&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(model$bws)&#xA;plot(model,common.scale=FALSE,plot.errors.method=&quot;bootstrap&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, I note that your predictor `temperature' is discrete, so you might consider using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+ ordered(temperature)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(i.e. use a discrete support kernel). Doing so reveals that there is little signal in this model, but the same holds for a simple parametric model (adjusted r-squared is negative).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.lm &amp;lt;- lm(model$bws$formula)&#xA;summary(model.lm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="17809" LastEditorUserId="17809" LastEditDate="2016-04-12T10:49:56.930" LastActivityDate="2016-04-12T10:49:56.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11166" PostTypeId="1" AcceptedAnswerId="11172" CreationDate="2016-04-12T12:33:15.450" Score="-4" ViewCount="538" Body="&lt;p&gt;I am a Python developer but I want to become a data scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At its core what is the bare minimum I need to have to make this transition? I know it cannot simply be that I need to learn Numpy and Pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Thoughts:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am hoping to frame my question with the following three perspectives in mind and am trying to answer what is essentially needed for each category:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Technical: Analytics&lt;/li&gt;&#xA;&lt;li&gt;Technical: Computer Science&lt;/li&gt;&#xA;&lt;li&gt;Non-Technical: Soft Skills &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated. :) &lt;/p&gt;&#xA;" OwnerUserId="17814" LastEditorUserId="26596" LastEditDate="2016-12-05T19:57:21.460" LastActivityDate="2016-12-05T19:57:21.460" Title="What is most likely the bare minimum knowledge one has to have to become data scientist?" Tags="|python|bigdata|beginner|education|" AnswerCount="2" CommentCount="4" ClosedDate="2016-04-12T20:49:20.313" ContentLicense="CC BY-SA 3.0" />
  <row Id="11167" PostTypeId="1" AcceptedAnswerId="11219" CreationDate="2016-04-12T12:56:51.450" Score="2" ViewCount="361" Body="&lt;p&gt;I'm working on a problem where in I have some data sets about some power generating units. Each of these units have been activated to run in the past and while activation, some units went into some issues. I now have all these data and I would like to come up with some sort of Ranking for these generating units. The criteria for ranking would be pretty simple to start with. They are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Maximum number of times a particular generating unit was activated&lt;/li&gt;&#xA;&lt;li&gt;How many times did the generating unit ran into problems during activation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Later on I would expand on this ranking algorithm by adding more criteria. I will be using Apache Spark MLIB library and I can already see that there are quite a few algorithms already in place. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-guide.html&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/docs/latest/mllib-guide.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just not sure which algorithm would fit my purpose. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="17815" LastActivityDate="2016-04-15T00:24:40.087" Title="Algorithm Suggestion For a Specific Problem" Tags="|machine-learning|apache-spark|ranking|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11168" PostTypeId="2" ParentId="11166" CreationDate="2016-04-12T13:08:46.193" Score="6" Body="&lt;p&gt;Just a few thoughts which aren't covered in the &lt;a href=&quot;https://datascience.stackexchange.com/questions/9303/seeking-career-guidance/9304&quot;&gt;link&lt;/a&gt; I pasted above ... &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Big data != data science. If you are a data scientist you may or may not be using big data tools. Your question wasn't clear if you understand this or not, but the distinction is important.&lt;/li&gt;&#xA;&lt;li&gt;There are various careers that 'fit' in the data science spectrum. Instead of repeating them here, I wrote a short blog a while back on &lt;a href=&quot;http://www.consolidata.co.uk/business/building-your-first-data-analytics-team/&quot; rel=&quot;nofollow noreferrer&quot;&gt;building a data analytics team&lt;/a&gt;. People often broaden the term &lt;em&gt;data scientist&lt;/em&gt; to include most of the elements in the team I describe.&lt;/li&gt;&#xA;&lt;li&gt;Learn some maths - if you don't already. The analytics part of data science is very heavy in maths.&lt;/li&gt;&#xA;&lt;li&gt;Expand on your data analytics algorithms, to include some clustering and classification as they are generally applicable, even if you think you are just going to be dealing with timeseries data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16284" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-04-12T15:17:49.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11170" PostTypeId="2" ParentId="10683" CreationDate="2016-04-12T13:38:53.090" Score="1" Body="&lt;p&gt;Home field advantage is incredibly important in sports. If you always put the home team in the first column, your model will adjust the bias term accordingly. I've been doing this successfully for NBA and MLB models for several years now. Making the first team the home team also eliminates the need for an explicit home indicator variable and you'll be able to use &quot;half&quot; of the data like you've described. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For rare neutral games like the Superbowl, you would need to create a mirror image of the data. In other words, present two rows for each game (team-A,team-B;team-B,team-A). When it's time to score you model, you can just arbitrarily pick one of the two records because they represent mirror opposite of the same thing. I've done this before for March Madness basketball games (all neutral courts), and this year (2016) my March Madness model was in the top 10% in Kaggle, so the technique works. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Home field advantage is just one of the variables you'll need to make a good predictive model, but that's an answer for another question. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-04-12T14:02:19.887" LastActivityDate="2016-04-12T14:02:19.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11171" PostTypeId="1" AcceptedAnswerId="11996" CreationDate="2016-04-12T13:45:33.810" Score="11" ViewCount="30136" Body="&lt;p&gt;For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;patient, Attr1, Attr2, Attr3, .., Label&#xA;002      90.0   8.0    98.0 ...   ? ===&amp;gt; predict drug A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I calculate the confidence or probability of the prediction result of drug A?&lt;/p&gt;&#xA;" OwnerUserId="12866" LastEditorUserId="29169" LastEditDate="2019-06-08T03:38:26.603" LastActivityDate="2022-06-08T13:13:42.603" Title="Decision tree, how to understand or calculate the probability/confidence of prediction result" Tags="|decision-trees|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="11172" PostTypeId="2" ParentId="11166" CreationDate="2016-04-12T13:55:21.473" Score="4" Body="&lt;p&gt;&lt;strong&gt;From personal experience&lt;/strong&gt; (so take into consider that I might not be representative although I'm probably not that far away too) the people that approached me with a job offer for &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data Scientist&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;did so because:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1)&lt;/strong&gt; Considerable knowledge in one or more programming language typically used for data analysis. In my case &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2)&lt;/strong&gt; Knowledge in applied mathematics (usually they don't even care about the base field). You just have to know how to interpret data and take valid conclusions from it (as a starting point at least).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3)&lt;/strong&gt; Past experience with libraries such as &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;numpy&lt;/a&gt;, &lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow&quot;&gt;scipy&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt; (very relevant), &lt;a href=&quot;http://scikit-image.org/&quot; rel=&quot;nofollow&quot;&gt;scikit-image&lt;/a&gt; (if you are going to do image analysis also), &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;pandas&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4)&lt;/strong&gt; Past experience with data visualization libraries such as &lt;a href=&quot;http://matplotlib.org/&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt;, &lt;a href=&quot;https://stanford.edu/~mwaskom/software/seaborn/&quot; rel=&quot;nofollow&quot;&gt;seaborn&lt;/a&gt;, &lt;a href=&quot;http://code.enthought.com/projects/chaco/&quot; rel=&quot;nofollow&quot;&gt;Chaco&lt;/a&gt;, &lt;a href=&quot;http://ggplot.yhathq.com/&quot; rel=&quot;nofollow&quot;&gt;ggplot&lt;/a&gt;, &lt;a href=&quot;http://www.pyqtgraph.org/&quot; rel=&quot;nofollow&quot;&gt;pyQtGraph&lt;/a&gt;, &lt;a href=&quot;http://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Bokeh&lt;/a&gt;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5)&lt;/strong&gt; Knowledge about &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html#supervised-learning&quot; rel=&quot;nofollow&quot;&gt;regression techniques&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#clustering&quot; rel=&quot;nofollow&quot;&gt;clustering&lt;/a&gt;, and &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html#supervised-learning&quot; rel=&quot;nofollow&quot;&gt;classification&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;6)&lt;/strong&gt; Valid extras depending on the field are typical applied mathematics in &lt;a href=&quot;http://hpgl.github.io/hpgl/&quot; rel=&quot;nofollow&quot;&gt;space estimation&lt;/a&gt;, &lt;a href=&quot;http://scikit-image.org/docs/dev/auto_examples/&quot; rel=&quot;nofollow&quot;&gt;image analysis and processing&lt;/a&gt; and &lt;a href=&quot;http://opencv.org/&quot; rel=&quot;nofollow&quot;&gt;computer vision&lt;/a&gt;, &lt;a href=&quot;http://docs.enthought.com/mayavi/mayavi/&quot; rel=&quot;nofollow&quot;&gt;3D visualization&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;7)&lt;/strong&gt; If you already have experience in building &lt;a href=&quot;https://sourceforge.net/projects/geoms2/&quot; rel=&quot;nofollow&quot;&gt;scientific software solutions&lt;/a&gt; using those programming languages, it might be a great advantage.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With point &lt;strong&gt;7)&lt;/strong&gt; in mind you might consider looking at &lt;a href=&quot;https://www.riverbankcomputing.com/software/pyqt/download5&quot; rel=&quot;nofollow&quot;&gt;PyQt5&lt;/a&gt; and &lt;a href=&quot;http://www.wxpython.org/&quot; rel=&quot;nofollow&quot;&gt;wxPython&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;8)&lt;/strong&gt; Ideally you are also able to present your results to an assistance that is not necessarily made of scientists only (I advise lots of illustrations..., actually, now that I think about it, lots of illustrations even if it's only scientists). So this takes some skill into building appropriate diagrams and figures (see vector graphics software such as &lt;a href=&quot;https://inkscape.org/en/&quot; rel=&quot;nofollow&quot;&gt;Inkscape&lt;/a&gt;, together with plotting libraries it can make wonders).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;9)&lt;/strong&gt; Last but not least quite a bit of flexibility (this is common for scientific and development staff). Sometimes you need to change your technology and this takes some learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that my experience does not say much in terms of web development &lt;em&gt;per se&lt;/em&gt;. Mine is a scientific background with very little of web development so people that approach me, do so with this in mind. Other fields might request for different skills (and by the way, you don't need to be a web developer to deal with web data).&lt;/p&gt;&#xA;" OwnerUserId="17806" LastEditorUserId="17806" LastEditDate="2016-04-12T14:30:21.460" LastActivityDate="2016-04-12T14:30:21.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11174" PostTypeId="2" ParentId="11167" CreationDate="2016-04-12T14:14:59.920" Score="0" Body="&lt;p&gt;With a few exceptions, you can pretty much use any machine learning algorithm for your model. The beauty of most machine learning packages is that the interface for each model is mostly the same (although the tuning parameters will differ), and it takes just a few lines of code to try out each model. There is no reason you should artificially constrain yourself to trying certain models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some exceptions to this rule are algorithms that might only work for classification or only work for regression. It sounds like you're trying to predict a continuous target variable that you'll then use for ranking. If that's the case, then you won't be able to use an algorithm called Naive Bayes because it can only output probabilities. In other rare cases like Deep Learning models, the run time can be very long (hours or days) and in those cases you wouldn't want to use an algorithm like that unless you had a good reason to do so (e.g., face recognition in images). You should be able to use nearly every algorithm in MLlib though: gradient boosting, random forests, etc. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-04-12T14:27:57.677" LastActivityDate="2016-04-12T14:27:57.677" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="11175" PostTypeId="1" CreationDate="2016-04-12T14:53:45.657" Score="1" ViewCount="3626" Body="&lt;p&gt;I have a predictive model in IPython notebook. How can I create dll or exe from it? And is it possible to save in executable file already trained model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want a program with very simple syntax(X - file with objects, out -- file with answers):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;./model X out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know if is appropriate community for this question. If not, I'll delete it.&lt;/p&gt;&#xA;" OwnerUserId="15693" LastActivityDate="2016-04-12T15:16:36.240" Title="Technical question: how to build exe from python code?" Tags="|python|programming|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11176" PostTypeId="2" ParentId="11175" CreationDate="2016-04-12T15:14:44.823" Score="2" Body="&lt;p&gt;You can &lt;a href=&quot;https://ipython.org/ipython-doc/1/interactive/nbconvert.html&quot; rel=&quot;nofollow&quot;&gt;convert&lt;/a&gt; your ipython notebook to a variety of different formats. If I were you, I would convert it to a .py file, and organize your code to take in &lt;code&gt;sys.argv&lt;/code&gt; 'X' and 'out' for reading and writing your data. Then at the CLI, you can type&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;python ./model.py X out&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10135" LastActivityDate="2016-04-12T15:14:44.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11177" PostTypeId="2" ParentId="11175" CreationDate="2016-04-12T15:16:36.240" Score="3" Body="&lt;p&gt;You'll need to copy, or &lt;a href=&quot;https://ipython.org/ipython-doc/1/interactive/nbconvert.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;convert&lt;/a&gt;*, your IPython code (I think) into a Python script. After that you can use software such as &lt;a href=&quot;http://www.py2exe.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;py2exe&lt;/a&gt; to create a standalone application. Other hypothesis are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cx-freeze.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;cx_freeze&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.pyinstaller.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pyInstaller&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pythonhosted.org/py2app/&quot; rel=&quot;nofollow noreferrer&quot;&gt;py2App&lt;/a&gt; (Mac OsX)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's possible to save data inside the scripts (it will be compiled the same way as code as long as it is valid). Other possibility is to save in an external file and just load the data on execution. If the data is in a class check &lt;a href=&quot;https://docs.python.org/2/library/pickle.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pickle&lt;/a&gt; to save it to a file.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;as&lt;/em&gt; @dmb &lt;em&gt;suggested.&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;EDIT: Ah I forgot to mention the arguments. That is easily achievable. Check this &lt;a href=&quot;https://stackoverflow.com/questions/9497370/making-exe-file-from-python-that-uses-command-line-arguments&quot;&gt;Question&lt;/a&gt; in SO.&lt;/p&gt;&#xA;" OwnerUserId="17806" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-04-12T15:16:36.240" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11178" PostTypeId="1" CreationDate="2016-04-12T16:14:05.453" Score="1" ViewCount="542" Body="&lt;p&gt;I've created an application on Facebook's &lt;a href=&quot;http://wit.ai&quot; rel=&quot;nofollow&quot;&gt;wit.ai&lt;/a&gt;. In my application, I have a single intent with the following phases:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hello&#xA;Hi bud&#xA;Hi&#xA;Hey there&#xA;Hey&#xA;Hey bud !! How's it going ?&#xA;Hey bud !! Hoss it going ?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if I try the easy phase &lt;code&gt;Hello&lt;/code&gt;, I get a confidence score 0.535.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now if I  try the phase &lt;code&gt;shut up&lt;/code&gt;, I also get a confidence score 0.535.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; How does the Facebook wit.ai work? I don't understand why I would get the same confidence score in both cases.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-04-12T16:14:05.453" Title="NLP: wit.ai. How to use confidence score?" Tags="|nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11179" PostTypeId="1" CreationDate="2016-04-12T16:41:25.243" Score="1" ViewCount="96" Body="&lt;p&gt;I’ve just gotten the results back from a marketing campaign that consisted of testing new men’s and women’s apparel creative (test creative) against business as usual men’s and women’s creative (control creative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to test to see if the lift in response rate was significant between...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.The Men’s Test Vs Control Creatives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.The Women’s Test Vs Control Creatives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.All Test vs All Control&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what my data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name              Pannel    Sample_Size  RR_Numerator   RR_Dominator Res_Rate &#xA;Men’s In-Target     Test    236,119      1,345          236,119      0.57%&#xA;Men’s In-Target     Control 207,715      1,220          207,715      0.59%&#xA;Women’s In-Target   Test    364,966      3,915          364,966      1.07%&#xA;Women’s In-Target   Control 542,786      4,489          542,786      0.83%&#xA;Model Test          Test    601,123      5,538          601,123      0.92%&#xA;Business As Usual   Control 750,573      6,185          750,573      0.82%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.Is the appropriate course of action / decision mechanism to test significance a Chi-Square test ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.Should I be adjusting for the unequal Test vs Control sample sizes?&lt;/p&gt;&#xA;" OwnerUserId="12947" LastActivityDate="2016-04-12T16:41:25.243" Title="Campaign Hypothesis Testing: Is using chi-square test appropriate?" Tags="|statistics|ab-test|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11180" PostTypeId="1" CreationDate="2016-04-12T19:57:51.047" Score="1" ViewCount="544" Body="&lt;p&gt;If I have a convolutional network which compresses from 256x256x3 images to, say, 100 hidden continuous states, then back to 256x256x3 and then train it with stochastic gradient descent (where the output is a denoised input), is it reasonable to expect it to learn the same or similar features as a similarly architected network, but one that uses variational methods (reconsruction + latent loss) techniques?  Assuming they both have the same number of conv/max/norm steps between the input and the FC layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about if I have a really small dataset, say 10 examples, and want to overfit?  Will both of them arrive at similar representations at the hidden layer?&lt;/p&gt;&#xA;" OwnerUserId="16135" LastActivityDate="2016-04-12T19:57:51.047" Title="Behavioral Differences between Standard Autoencoder and Variational Autoencoder" Tags="|autoencoder|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11182" PostTypeId="1" CreationDate="2016-04-12T20:28:52.260" Score="4" ViewCount="234" Body="&lt;p&gt;New York City provides tens of gigs of data of taxi routes all over the city. What I'd like to do, is use this data (or some other method), to come up with an algorithm that can take a persons GPS data over a short span of time (say an hour), and answer the question: Is this person driving a cab?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm should work in any location, not just NYC. The idea is that I'd like to be able to determine patterns, that signal that a route being driven by a person is the type of route a person driving a cab would take.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like to write this in Ruby. But I am open to other suggestions, approaches, and implementations. Links to projects I should research, suggestions on languages to use, approaches to take, etc are all appreciated.&lt;/p&gt;&#xA;" OwnerUserId="17828" LastEditorUserId="16284" LastEditDate="2016-04-21T13:57:11.083" LastActivityDate="2016-04-21T13:57:11.083" Title="Using GPS signal, determine is this person driving a cab" Tags="|machine-learning|training|geospatial|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11183" PostTypeId="2" ParentId="11182" CreationDate="2016-04-12T20:37:56.123" Score="2" Body="&lt;p&gt;Take the taxi routes and combine them with civilian car routes to form a data set for classification. Using a map (say, from Google)&#xA; break down each route into a sequence of roads segments, from intersection to intersection. If you only have GPS traces this will involve spatio-temporal segmentation. (Intersections/terminuses are places where cars go but stop at, and the road segments are the places cars move through to get to from one intersection to another). Model these as categorical variables (road segment 1, 2, 3, etc); abstract out the physical location. Then train a classifier than accepts a sequence as input (e.g., a recurrent neural network). Use the time of departure as another feature, modeled as two real variables, $\cos(2*\pi*t/24), \sin(2*\pi*t/24)$, where $0&amp;lt;t&amp;lt;24$. If you have really accurate GPS information, I would also try to estimate the rate of lane crossing; taxi drivers are well known for their aggressive driving, and the rate of lane crossing would capture this well. Finally, use a pair of boolean variables to record whether the points of departure and destination are parking areas, if you can obtain this information. I imagine taxis will be more likely to start and stop at prohibited places.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rest is your usual hyper-parameter optimization black magic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to be able to do anything besides classify a given route as taxi-driven or not, please say so.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-04-15T15:44:49.777" LastActivityDate="2016-04-15T15:44:49.777" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11186" PostTypeId="1" CreationDate="2016-04-12T23:16:13.907" Score="3" ViewCount="3695" Body="&lt;p&gt;I'm using CNN, RNN and OpenCV to identify people and cars within images, once I identify several images I'm cropping them and dividing them in cars and people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to group all same-looking cars and all same-looking people. What would be the best approach to do so?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm recollecting these images from an IP camera, so I cannot base my code in colors, because the light is not the same when is day and night. Plus the cars aren't always looking at the same direction. I guess SVM would work nice, but I would need classify some data by hand first. I'm looking for something to make clusters of similar images without a supervising method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would something like k-means work?&lt;/p&gt;&#xA;" OwnerUserId="16712" LastEditorUserId="12876" LastEditDate="2016-04-13T06:19:49.767" LastActivityDate="2017-04-09T17:23:19.497" Title="Grouping similar data/images" Tags="|neural-network|svm|k-means|image-classification|object-recognition|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11187" PostTypeId="1" AcceptedAnswerId="11197" CreationDate="2016-04-13T01:20:26.753" Score="3" ViewCount="90" Body="&lt;p&gt;I'm specifically interested in tying doctors to their published papers. The key issue is that using name alone will result many collisions. I'm wondering what set of features I would need to reliably connect a doctor with a given published paper? Aside from weak features such as specialty, is there any database that has a link between doctor NPI and papers published?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://hhs.ddod.us/wiki/Use_Case_24:_Mapping_of_NPI_number_to_PubMed_ID#Solution&quot; rel=&quot;nofollow&quot;&gt;I've seen this on linking NPIs to authors in PubMed&lt;/a&gt; but it seems rather unreliable. &lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-04-13T14:55:16.203" Title="Connecting Authors with Published Papers" Tags="|scraping|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11188" PostTypeId="1" CreationDate="2016-04-13T05:38:28.643" Score="2" ViewCount="412" Body="&lt;p&gt;In order to build a classifier, I need to extract a few features from the data stored on a &lt;code&gt;MySQL database&lt;/code&gt;. I need to join multiple tables and it is taking a lot of time. I have joined 2 tables at one time and have got results in multiple cases. I need to combine them. Writing a script will be the best option?&#xA;How do people extract features from large relational databases? Am I missing something?&lt;/p&gt;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="116702" LastEditDate="2021-07-17T07:49:56.747" LastActivityDate="2024-01-17T13:04:20.563" Title="Feature extraction from relational database" Tags="|feature-extraction|databases|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="11189" PostTypeId="1" CreationDate="2016-04-13T06:57:10.043" Score="2" ViewCount="1855" Body="&lt;p&gt;I have used the keras example code of Image Captioning in that I have used the VGG pretrained model for extracting image features(4096) and for text part I have done indexing to the unique words and post zero padding according the max caption length(which is equal the length of biggest sentence in data) and for next words I created a numpy array of (number of example, vocabulary size) vocabulary size is equal to number of unique words in data. The next words is a 1's 0's matrix 1 means present of word in the sentence 0 absence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for predicting what should be given exactly in partial caption and in next words if we consider &quot;cat sat on mat&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in training data I am appending &quot;START&quot; and &quot;END&quot; tokens in the start and end of training captions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is my code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;correct me if I am doing wrong in code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers.core import Dense, Dropout, Activation, Flatten&#xA;from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D&#xA;from keras.optimizers import SGD, Adadelta, Adagrad&#xA;from keras.utils import np_utils, generic_utils&#xA;from keras.callbacks import EarlyStopping&#xA;from keras.layers.advanced_activations import PReLU, LeakyReLU&#xA;from keras.layers import Embedding,GRU,TimeDistributedDense,RepeatVector,Merge&#xA;from keras.preprocessing.text import one_hot&#xA;from keras.preprocessing import sequence&#xA;import cv2&#xA;import numpy as np&#xA;&#xA;max_caption_len = 21&#xA;vocab_size = 43&#xA;def VGG_16(weights_path=None):&#xA;    model = Sequential()&#xA;    model.add(ZeroPadding2D((1,1),input_shape=(3,224,224)))&#xA;    model.add(Convolution2D(64, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(64, 3, 3, activation='relu'))&#xA;    model.add(MaxPooling2D((2,2), strides=(2,2)))&#xA;&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(128, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(128, 3, 3, activation='relu'))&#xA;    model.add(MaxPooling2D((2,2), strides=(2,2)))&#xA;&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(256, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(256, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(256, 3, 3, activation='relu'))&#xA;    model.add(MaxPooling2D((2,2), strides=(2,2)))&#xA;&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(MaxPooling2D((2,2), strides=(2,2)))&#xA;&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(ZeroPadding2D((1,1)))&#xA;    model.add(Convolution2D(512, 3, 3, activation='relu'))&#xA;    model.add(MaxPooling2D((2,2), strides=(2,2)))&#xA;&#xA;    model.add(Flatten())&#xA;    model.add(Dense(4096, activation='relu'))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(4096, activation='relu'))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(1000, activation='softmax'))&#xA;&#xA;    if weights_path:&#xA;        model.load_weights(weights_path)&#xA;&#xA;    #Remove the last two layers to get the 4096D activations&#xA;    model.layers.pop()&#xA;    model.layers.pop()&#xA;&#xA;    return model&#xA;print &quot;VGG loading&quot;&#xA;image_model = VGG_16('vgg16_weights.h5')&#xA;image_model.trainable = False&#xA;print &quot;VGG loaded&quot;&#xA;# let's load the weights from a save file.&#xA;# image_model.load_weights('weight_file.h5')&#xA;&#xA;# next, let's define a RNN model that encodes sequences of words&#xA;# into sequences of 128-dimensional word vectors.&#xA;print &quot;Text model loading&quot;&#xA;language_model = Sequential()&#xA;language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))&#xA;language_model.add(GRU(output_dim=128, return_sequences=True))&#xA;language_model.add(TimeDistributedDense(128))&#xA;print &quot;Text model loaded&quot;&#xA;# let's repeat the image vector to turn it into a sequence.&#xA;print &quot;Repeat model loading&quot;&#xA;image_model.add(RepeatVector(max_caption_len))&#xA;print &quot;Repeat model loaded&quot;&#xA;# the output of both models will be tensors of shape (samples, max_caption_len, 128).&#xA;# let's concatenate these 2 vector sequences.&#xA;print &quot;Merging&quot;&#xA;model = Sequential()&#xA;model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1))&#xA;# let's encode this vector sequence into a single vector&#xA;model.add(GRU(256, return_sequences=False))&#xA;# which will be used to compute a probability&#xA;# distribution over what the next word in the caption should be!&#xA;model.add(Dense(vocab_size))&#xA;model.add(Activation('softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', optimizer='rmsprop')&#xA;print &quot;Merged&quot;&#xA;# &quot;images&quot; is a numpy float array of shape (nb_samples, nb_channels=3, width, height).&#xA;# &quot;captions&quot; is a numpy integer array of shape (nb_samples, max_caption_len)&#xA;# containing word index sequences representing partial captions.&#xA;# &quot;next_words&quot; is a numpy float array of shape (nb_samples, vocab_size)&#xA;# containing a categorical encoding (0s and 1s) of the next word in the corresponding&#xA;# partial caption.&#xA;print &quot;Data preprocessig&quot;&#xA;Texts = [&quot;START A girl is stretched out in shallow water END&quot;,&#xA;        &quot;START The two people stand by a body of water and in front of bushes in fall END&quot;,&#xA;        &quot;START A blonde horse and a blonde girl in a black sweatshirt are staring at a fire in a barrel END&quot;,&#xA;        &quot;START Children sit and watch the fish moving in the pond END&quot;,&#xA;        &quot;START A fisherman fishes at the bank of a foggy river END&quot;]&#xA;&#xA;Images = [&quot;667626_18933d713e.jpg&quot;,&#xA;         &quot;3637013_c675de7705.jpg&quot;,&#xA;         &quot;10815824_2997e03d76.jpg&quot;,&#xA;         &quot;12830823_87d2654e31.jpg&quot;,&#xA;         &quot;17273391_55cfc7d3d4.jpg&quot;]&#xA;images = []&#xA;for image in Images:&#xA;    img = cv2.imread(image)&#xA;    img.resize((3,224,224))&#xA;    images.append(img)&#xA;images = np.asarray(images)&#xA;&#xA;words = [txt.split() for txt in Texts]&#xA;unique = []&#xA;for word in words:&#xA;    unique.extend(word)&#xA;unique = list(set(unique))&#xA;word_index = {}&#xA;index_word = {}&#xA;for i,word in enumerate(unique):&#xA;    word_index[word] = i&#xA;    index_word[i] = word&#xA;&#xA;partial_captions = []&#xA;for text in Texts:&#xA;    one = [word_index[txt] for txt in text.split()]&#xA;    partial_captions.append(one)&#xA;&#xA;partial_captions = sequence.pad_sequences(partial_captions, maxlen=max_caption_len,padding='post')&#xA;next_words = np.zeros((5,vocab_size))&#xA;for i,text in enumerate(Texts):&#xA;    text = text.split()&#xA;    x = [word_index[txt] for txt in text]&#xA;    x = np.asarray(x)&#xA;    next_words[i,x] = 1&#xA;&#xA;print &quot;Data preprocessing done&quot;&#xA;model.fit([images, partial_captions], next_words, batch_size=1, nb_epoch=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this task, I am taking only 5 example to know the working of this model and also &lt;strong&gt;I want to know whether my approach is right or wrong&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I am going to use flickr8k dataset for the same.&lt;/p&gt;&#xA;" OwnerUserId="17841" LastEditorUserId="11097" LastEditDate="2016-04-13T07:03:39.233" LastActivityDate="2016-04-13T07:03:39.233" Title="Image Captioning example in keras Approach?" Tags="|machine-learning|deep-learning|keras|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11195" PostTypeId="2" ParentId="11060" CreationDate="2016-04-13T12:54:49.240" Score="6" Body="&lt;p&gt;This seems like a job for Principal Component Analysis. In Scikit is &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt; implemented well and it helped me many times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA, in a certain way, combines your features. By limiting the number of components, you fetch your model with noise-less data (in the best case). Because your model is as good as your data are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider below a simple example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.pipeline import Pipeline&#xA;pipe_rf = Pipeline([('pca', PCA(n_components=80)),&#xA;                    ('clf',RandomForestClassifier(n_estimators=100))])&#xA;pipe_rf.fit(X_train_s,y_train_s)&#xA;&#xA;pred = pipe_rf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why I picked 80? When I plot cumulative variance, I got this below, which tells me that with ~80 components, I reach almost all the variance. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/TeV7X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TeV7X.png&quot; alt=&quot;cumulative variance&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would say give it a try, use it in your models. It should help.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2017-12-29T12:56:52.327" LastActivityDate="2017-12-29T12:56:52.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11196" PostTypeId="2" ParentId="11130" CreationDate="2016-04-13T14:45:00.077" Score="0" Body="&lt;p&gt;Which distance function to use depends on the data geometry itself. In some cases you can plot you data and visualize then take decisions but in real world problems mostly it is not possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most clustering algorithm like Kmeans, as long as a distance function is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;metric&lt;/a&gt; you can use it. &#xA;There exists methods to learn a metric according to data geometry which you can use to cluster the data. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is &lt;a href=&quot;https://en.wikipedia.org/wiki/Similarity_learning#Metric_learning&quot; rel=&quot;nofollow&quot;&gt;metric learning&lt;/a&gt;?&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://researchers.lille.inria.fr/abellet/misc/metric_learning_tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;Introduction and Motivation&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Brian Kulis has a very nice &lt;a href=&quot;http://web.cse.ohio-state.edu/~kulis/pubs/ftml_metric_learning.pdf&quot; rel=&quot;nofollow&quot;&gt;survey paper&lt;/a&gt; on this.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Metric learning is closely related to dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using MATLAB check this &lt;a href=&quot;http://www.cs.cmu.edu/~liuy/distlearn.htm&quot; rel=&quot;nofollow&quot;&gt;toolkit&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="3570" LastEditorUserId="3570" LastEditDate="2016-04-13T14:55:12.240" LastActivityDate="2016-04-13T14:55:12.240" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11197" PostTypeId="2" ParentId="11187" CreationDate="2016-04-13T14:55:16.203" Score="4" Body="&lt;p&gt;It's not an easy question to answer even for groups with a lot of leverage such as &lt;a href=&quot;https://www.researchgate.net/home&quot;&gt;Research Gate&lt;/a&gt; (RG). RG has it's own (I assume proprietary) &lt;code&gt;author matching algorithm&lt;/code&gt; &lt;a href=&quot;http://www.scilogs.com/in_scientio_veritas/researchgate-heal-thyself-please/&quot;&gt;which has caused problems in the past&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They use the name of the author (in different combinations) to suggest &lt;a href=&quot;https://explore.researchgate.net/display/support/Authorship&quot;&gt;authorship to RG users&lt;/a&gt; (so has you said, it does causes a lot of problems). Every once in a while users that are not the author accept the suggestions and, from the portal point of view, gain the equivalent reputation. It's a serious business that requires quite a bit of R&amp;amp;D before making a decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said I can't answer with certainties only with reasonable possibilities. A few questions I would make (and hope to answer with a bit of data analysis):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the probability that an author with a confirmed publication in an identified journal will publish in that journal again? - &lt;strong&gt;Journal Name&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the probability that an author that has partnered with other authors will repeat the same co-authoring combination? &lt;strong&gt;Co-author Names&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the expected time story for publishing for each author? Authors rarely publish articles 20 years apart. Typically they publish more and more, or less and less. &lt;strong&gt;Time frame&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How frequently do authors change the institution they belong to? &lt;strong&gt;Institution Name&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the preferred keywords for a given author? &lt;strong&gt;Key Names&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the preferred citations made by a given author? &lt;strong&gt;Bibliography&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All of the questions above require quite a bit of &lt;a href=&quot;https://en.wikipedia.org/wiki/Text_mining&quot;&gt;Text Mining&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Category:String_matching_algorithms&quot;&gt;String Matching&lt;/a&gt;, as well as solid dataset to start your mining.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some publishers have their own &lt;a href=&quot;http://api.elsevier.com/documentation/SCIDIRSearchAPI.wadl&quot;&gt;API&lt;/a&gt; although I can't say much about permissiveness (never tried it myself). &lt;a href=&quot;https://www.researchgate.net/post/Is_there_any_Researchgate_API&quot;&gt;RG has been promising one for years&lt;/a&gt; but, as far as I know, it still does not &lt;a href=&quot;https://github.com/researchgate&quot;&gt;exist&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An unlikely thing I remember now is the inspiring story of &lt;a href=&quot;https://en.wikipedia.org/wiki/Aaron_Swartz&quot;&gt;Aaron Swartz&lt;/a&gt;. This activist, along with other persons, successfully managed to create large open archives for books and articles. Should that information still exist it might be worth your time to take a look &lt;a href=&quot;https://openlibrary.org/&quot;&gt;there&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you have a list of the journals you are considering (you've mentioned only &quot;doctors&quot; which is a bit vague) you can try and see with the publisher if they have any way of accessing their database.&lt;/p&gt;&#xA;" OwnerUserId="17806" LastActivityDate="2016-04-13T14:55:16.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11199" PostTypeId="1" AcceptedAnswerId="11243" CreationDate="2016-04-13T16:48:04.420" Score="2" ViewCount="373" Body="&lt;p&gt;I was wondering if it makes sense to apply clustering techniques on an aggregation of data, like, I have three different sources of data such as S1 S2 and S3 where each of these sources share some common columns but the majority are not shared. Does it make sense to group all the sources with all the columns in a big dataframe and apply clustering techniques whereas some records will only have null values for the columns that are not part of its corresponding service.&#xA;Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: &#xA;The input is basically logs coming from different services with different columns where some of them are shared between all services. The output should be a cluster of records representing a user having the same behavior.&lt;br&gt;&#xA;The logs are gathered by second corresponding of a user action. They are aggregated by hours for each user to derive other features (the granularity of seconds it according to me, too much).&lt;br&gt;&#xA;The goal, is to detect anormal behaviors.&lt;br&gt;&#xA;And the question is, can I regroup different dataframe having different features while some of them are shared and run a K-mean on it. Because my dataframe would look like this (s* for service, and common_ for common feature):  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-------------------------------------------------------------&#xA;s1_f1 | s1_f2 | s2_f1 | s2_f2 | s3_f1 | common_f1 | common_f2&#xA;  1       OK     NULL    NULL    NULL     midday      less&#xA;  2       OK     NULL    NULL    NULL     midnight    more&#xA; NULL    NULL     2       5      NULL     midday      less&#xA; NULL    NULL     8       9      NULL     morning     less&#xA; NULL    NULL    NULL    NULL    777      morning     more&#xA; NULL    NULL    NULL    NULL    888      night       more&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13915" LastEditorUserId="13915" LastEditDate="2016-04-14T21:46:24.777" LastActivityDate="2016-04-15T20:58:56.547" Title="Does it make sense to apply clustering on aggregation of data?" Tags="|clustering|feature-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11200" PostTypeId="2" ParentId="11060" CreationDate="2016-04-13T17:24:26.690" Score="1" Body="&lt;p&gt;In addition to some of the suggestions above, I would recommend using a &lt;strong&gt;&lt;em&gt;two-step modeling&lt;/em&gt;&lt;/strong&gt; approach. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use the sparse features first and develop the best model. &lt;/li&gt;&#xA;&lt;li&gt;Calculate the predicted probability from that model.&lt;/li&gt;&#xA;&lt;li&gt;Feed that probability estimate into the second model (as an input feature), which would incorporate the dense features. In other words, use all dense features &lt;em&gt;and&lt;/em&gt; the probability estimate for building the second model. &lt;/li&gt;&#xA;&lt;li&gt;The final classification will then be based on the second model. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17397" LastActivityDate="2016-04-13T17:24:26.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11201" PostTypeId="1" CreationDate="2016-04-13T19:18:29.360" Score="1" ViewCount="532" Body="&lt;p&gt;I love Weka for its visual data exploration features (it's great at quickly giving you a mile-high aerial view of the distributions of all your variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a 0.5gb dataset, and Weka is unable to load it (on a 64-bit Windows PC with 16gb of RAM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Weka fans out there, what's the closest 'visual data exploration' alternative for very large datasets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(FWIW - I've done all the various things that are recommended when Weka fails at loading a very large dataset, e.g., increasing the Java heap size, etc.)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Update: I tried RapidMiner Basic (it's open source and free), and that worked.&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="16928" LastEditDate="2016-04-16T00:10:46.643" LastActivityDate="2016-07-01T15:20:55.587" Title="Weka alternative - visual data exploration with a very large dataset" Tags="|visualization|weka|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11202" PostTypeId="1" AcceptedAnswerId="11228" CreationDate="2016-04-13T22:20:00.967" Score="1" ViewCount="153" Body="&lt;p&gt;I have a data set with 3 independent variables and 1 dependent variable. The dependent is play_golf, the independents are Humidity, Pending_Chores, Wind.&#xA;   I want to aggregate the probability of playing golf based on those rules of multiple trees. So again, play_golf (this is the dependent value), and there are three independent variables Humidity(High, Medium, Low), Pending_Chores(Taxes, None, Laundry, Car Maintenance) and Wind (High, Low).  A rule would be like (IF humidity = &quot;High&quot; AND pending_chores = &quot;None&quot; AND Wind = &quot;High&quot; THEN play_golf = 77% ).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking that random forest would weight the rules somehow on the collection of trees and give a probability.  But if that doesn't make sense, then can you just tell me how to get the decision rules with one tree and I will work from that. I believe am I talking about a decision tree vector? I'm not sure.&lt;/p&gt;&#xA;" OwnerUserId="16676" LastEditorUserId="924" LastEditDate="2016-04-30T11:49:44.630" LastActivityDate="2016-04-30T11:49:44.630" Title="Aggregating Decision Trees" Tags="|machine-learning|r|classification|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11203" PostTypeId="1" CreationDate="2016-04-14T08:25:52.707" Score="1" ViewCount="41" Body="&lt;p&gt;Let's say that I'm building a binary classifier for giving loans to consumers. But apart from the standard accuracy metrics I can use, I have a requirement from the business, which wants to be able to &quot;auto-approve&quot; 80% of the loans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My gut instinct is that this should be &lt;code&gt;1 - False Positive Rate&lt;/code&gt;, but I'm not quite sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="1127" LastActivityDate="2016-04-14T09:52:48.933" Title="Translating a Business Metric into an ML Accuracy Metric" Tags="|classification|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11204" PostTypeId="1" CreationDate="2016-04-14T08:28:44.620" Score="2" ViewCount="1924" Body="&lt;p&gt;I am quite new to machine learning, and the task I was given was to find attributes that will allow me to identify users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of data for training that contains roughly 48,000 instances, where every instance is labeled with user id.&#xA;The data are TCP sessions with data on destination, source, ttl, start time, endtime, destination IP addresses, and more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task is to find attributes so that in given a difference TCP flows in the same format form the same group of users, I will be able to labeled them with the appropriate user Id.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 10 users in the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what the best way to do it, I tried using weka to run number of machine learning algorithm, but I couldn't configure them right, for example I tried to use K-means but the results didn't fit with the labeled user id.&lt;/p&gt;&#xA;" OwnerUserId="17877" LastEditorUserId="16284" LastEditDate="2016-04-14T16:47:04.163" LastActivityDate="2016-06-26T07:22:42.967" Title="User identification using machine learning" Tags="|machine-learning|feature-selection|supervised-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11206" PostTypeId="2" ParentId="11203" CreationDate="2016-04-14T09:47:35.653" Score="1" Body="&lt;p&gt;The purpose of this algorithm appears to be about reducing cost. So a metric which includes the &lt;em&gt;financial risk&lt;/em&gt; associated with defaults should be preferred over one that simply measures number of defaults.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A weighted &lt;code&gt;false_positive * loss_on_default&lt;/code&gt; metric would seem to be ideal, where you set the probability cutoff of positive class (auto-approve) so that 80% of loans were approved regardless of absolute probability of the prediction. Conceptually this is similar to picking a point on the ROC curve and assessing the algorithm at that point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This metric could not be used directly as an objective function, you'd probably use want something like logloss weighted per sample by the loss_on_default in order to train a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did a quick search for similar metrics, and could not find anything (it is also not something I have done before, so do please test the idea carefully, it is mostly conjecture on my part). There are plenty of metrics in e.g. SciKit learn however that take an array &lt;code&gt;sample_weight&lt;/code&gt; that does something similar - e.g. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss&quot; rel=&quot;nofollow&quot;&gt;Sklearn's Zero-One loss&lt;/a&gt; is pretty close except it will include false negatives as well.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="11097" LastEditDate="2016-04-14T09:52:48.933" LastActivityDate="2016-04-14T09:52:48.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11208" PostTypeId="1" AcceptedAnswerId="11210" CreationDate="2016-04-14T11:43:48.733" Score="5" ViewCount="2612" Body="&lt;p&gt;Recently I stumbled upon a R package that works similar like caret, but I can't remember what it's name was and I can't find it. It seemed to be less well known, but at least as extensive. Someone able to help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; I search for this other package to widen my horizon. There is no specific other reason. This question is specific however, since I am not looking for a general recommendation, but I am looking for this specific package I have in mind. I will recognize the package once I read its functions and descriptions. I had searched for a couple of hours myself and did not find it by searching for &quot;package similar to caret&quot;. That is why I need help.&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="14909" LastEditDate="2016-04-14T12:40:29.417" LastActivityDate="2018-12-08T19:41:11.073" Title="Package that is similar to R's caret?" Tags="|r|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11209" PostTypeId="2" ParentId="11122" CreationDate="2016-04-14T12:13:21.787" Score="0" Body="&lt;p&gt;Is all the information you want for each sequence contained in the text that's attached to it? If so, just compare a list of diseases and a list of tissues against every text. Lists of diseases can be found e.g. on the &lt;a href=&quot;http://cdc.gov&quot; rel=&quot;nofollow&quot;&gt;CDC&lt;/a&gt; website.&lt;/p&gt;&#xA;" OwnerUserId="12492" LastActivityDate="2016-04-14T12:13:21.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11210" PostTypeId="2" ParentId="11208" CreationDate="2016-04-14T12:17:51.747" Score="10" Body="&lt;p&gt;&lt;strong&gt;MLR&lt;/strong&gt; is similar to &lt;strong&gt;caret&lt;/strong&gt;, MLR offers a high-level interface to various statistical and machine learning packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the package description:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Interface to a large number of classification and regression&#xA;  techniques, including machine-readable parameter descriptions. There&#xA;  is also an experimental extension for survival analysis, clustering&#xA;  and general, example-specific cost-sensitive learning. Generic&#xA;  resampling, including cross-validation, bootstrapping and subsampling.&#xA;  Hyperparameter tuning with modern optimization techniques, for single-&#xA;  and multi-objective problems. Filter and wrapper methods for feature&#xA;  selection. Extension of basic learners with additional operations&#xA;  common in machine learning, also allowing for easy nested resampling.&#xA;  Most operations can be parallelized.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please check the &lt;a href=&quot;https://cran.r-project.org/web/packages/mlr/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference&lt;/a&gt; and &lt;a href=&quot;https://mlr.mlr-org.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Homepage&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="29575" LastEditDate="2018-12-08T19:41:11.073" LastActivityDate="2018-12-08T19:41:11.073" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="11212" PostTypeId="1" CreationDate="2016-04-14T13:25:16.030" Score="4" ViewCount="185" Body="&lt;p&gt;a team has to create models that predict the cost of deploying a machine over time. This is a regression problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The team is further divided into two groups, A and B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group A puts lots of attention on selecting attributes and finding rules that would segment the training set into clusters that are more or less homogeneous, and then uses a linear model to create predictions within the cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group B does not do clustering first, but instead includes the same attributes that team A uses for clustering into a non-linear model (let's say an ensemble of random forests or gradient boosting machine).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results are similar (or slightly better using the non linear model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are results measured? mean squared error over a hold-out set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The explanation appears to be that, by definition, tree models segment the population using attributes, so that the segments are as homogeneous or pure as possible, given the attributes.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the work team A is doing to cluster the instances, the tree model is is also doing per se - because segmentation is embedded in tree models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does this explanation make sense?&lt;/strong&gt;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it correct to infer that the approach of group B is less demanding in terms of time? (i.e. the model finds the attributes to segment the data as opposed to selecting the attributes manually)&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9197" LastEditorUserId="9197" LastEditDate="2016-04-14T16:50:30.150" LastActivityDate="2020-02-07T20:01:30.183" Title="clustering plus linear model versus non linear (tree) model" Tags="|clustering|linear-regression|decision-trees|ensemble-modeling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11213" PostTypeId="2" ParentId="10557" CreationDate="2016-04-14T14:05:24.643" Score="0" Body="&lt;p&gt;If you are prepared to spend time learning how a new tool works, you can try &lt;a href=&quot;https://automl.github.io/auto-sklearn/master/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Auto-Sklearn&lt;/a&gt;. It does all that's needed to build an ML pipeline for you.&lt;/p&gt;&#xA;&lt;p&gt;Feature preprocessing, selection, model ensemble building and tuning through cross-validation. Depending on the amount of data you have, it may or may not be a faster way to a good prediction, but it is certainly a promising one.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="69263" LastEditDate="2023-02-22T16:12:50.650" LastActivityDate="2023-02-22T16:12:50.650" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11214" PostTypeId="1" AcceptedAnswerId="11224" CreationDate="2016-04-14T14:32:52.763" Score="1" ViewCount="620" Body="&lt;p&gt;I posted my question on stack overflow, but there someone suggested that I should try it here. What I'm doing now :) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OK, first to my data. I have a word-bi-gram frequency matrix (1100 x 100658, dtype=int), where the first 5 columns contain information about the document. So every row is a document and every column a word-bi-gram like (of-the, on-the, and-that,...). I want to visualize the data, but before I do that, I want to reduce the dimension. So I thought I do that with PCA from sklearn. First I set the column labels with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myPandaDataFrame.columns = word-bi-grams&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I deleted some doc-columns, because I want to see what kind of information I can get if I only look at the proficiency.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;del existing_df['SUBSET']&#xA;del existing_df['PROMPT']&#xA;del existing_df['L1']&#xA;del existing_df['ESSAYID']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I set the proficiency column to be the index with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myPandaDataFrame.columns.set_index(['PROFICIENCY'], inplace=True, drop=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then I did this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.decomposition import PCA&#xA;x = 500&#xA;pcax = PCA(n_components=x)&#xA;pcax.fit(myPandaDataFrame)&#xA;PCA(copy=True, n_components=x, whiten=False)&#xA;existing_2dx = pcax.transform(myPandaDataFrame)&#xA;existing_df_2dx = pandas.DataFrame(existing_2dx)&#xA;existing_df_2dx.index = myPandaDataFrame.index&#xA;existing_df_2dx.columns = ['PC{0}'.format(i) for i in range(x)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But with this implementation I can only set 1100 n_components as a maximum. This is the number of documents (rows). This makes me suspicious. I tried a couple of examples / tutorials, but I can't get it right. So I hope someone can help me find out what I'm doing wrong? If would also be very happy about a good example / tutorial that can help me with my problem. Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With best regards.&lt;/p&gt;&#xA;" OwnerUserId="17886" LastActivityDate="2016-04-15T08:16:18.840" Title="Sklearn and PCA. Why is max n_row == max n_components?" Tags="|python|scikit-learn|pandas|pca|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11215" PostTypeId="1" CreationDate="2016-04-14T15:32:43.077" Score="1" ViewCount="42" Body="&lt;p&gt;I would like to find the two furthest pages of a big websites (+5millions pages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(It's the degree of separation, the most links between the two pages)&lt;/p&gt;&#xA;" OwnerUserId="17888" LastEditorUserId="11097" LastEditDate="2016-04-14T16:07:01.877" LastActivityDate="2016-04-15T10:09:31.233" Title="Find the furthest pages of a website" Tags="|data-mining|" AnswerCount="2" CommentCount="4" ClosedDate="2016-04-14T16:46:26.833" ContentLicense="CC BY-SA 3.0" />
  <row Id="11216" PostTypeId="2" ParentId="11215" CreationDate="2016-04-14T16:05:24.687" Score="2" Body="&lt;p&gt;This is a classic &quot;graph database&quot; question. I would do the following steps:-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Traverse all the pages using a webcrawler, or if you can interrogate the website server directly that may be qukcker. &lt;/li&gt;&#xA;&lt;li&gt;Ensure you keep all links between pages, for later use.&lt;/li&gt;&#xA;&lt;li&gt;Put the web pages into a &lt;a href=&quot;http://neo4j.com/&quot; rel=&quot;nofollow&quot;&gt;neo4j&lt;/a&gt; database with connections between the pages&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This will allow you to be able to do the query of pages with the maximum number of steps, or pages most frequently linked to, etc.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastEditorUserId="16284" LastEditDate="2016-04-14T16:49:35.840" LastActivityDate="2016-04-14T16:49:35.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11217" PostTypeId="2" ParentId="11215" CreationDate="2016-04-14T16:23:47.573" Score="-1" Body="&lt;p&gt;Assume you have the data in a table containing a row for each link, storing the PAGE and the linked page (LINK_TO).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select * from page_link;&#xA;&#xA;PAGE LINK_TO&#xA;---- -------&#xA;A    A       &#xA;A    B       &#xA;A    B       &#xA;A    C       &#xA;B    C       &#xA;B    C       &#xA;B    C       &#xA;C    A  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Than simple group on the two columns and count the occurences. &#xA;This gives the number of links between any two linked pages, so you can see &lt;strong&gt;the most links between the two page&lt;/strong&gt;s - if this is realy what you want...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select PAGE, LINK_TO, count(*) &#xA;from page_link &#xA;group by PAGE, LINK_TO order by 3 desc;&#xA;&#xA;PAGE LINK_TO   COUNT(*)&#xA;---- ------- ----------&#xA;B    C                3 &#xA;A    B                2 &#xA;A    C                1 &#xA;A    A                1 &#xA;C    A                1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A few milions of records should not be a problem for any database of your chioce.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2016-04-15T10:09:31.233" LastActivityDate="2016-04-15T10:09:31.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11218" PostTypeId="1" AcceptedAnswerId="11354" CreationDate="2016-04-14T18:59:44.113" Score="3" ViewCount="3835" Body="&lt;p&gt;I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc.&#xA;I see such transitions for a number of independent data points, and I have the cumulative frequencies from all data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have:&#xA;$$&#xA;  Period  A   B  C           \newline&#xA;  1       4   4  2&#xA;  2       1   2  7 &#xA;  3       0   1  9&#xA;  4      10  0  0&#xA;  5       8   1  1&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I DO know the transitions from one state to another, for example from A-&gt;B, B-&gt;C so on and so forth. For example I know that from Period 1, (all A's went to C. Among the missing B's one went to A, and rest to C.)&#xA;I was thinking of what would be the best way to visually represent this time wise transitions from one state to another. I was thinking that there might be some better way than just having a transition matrix, maybe something that looks like a Markov Chain but which could accommodate all the 5 periods of transitions in a succinct way? &#xA;I myself work on a statistical software called STATA, which has limited graphical applications. IS there something on other software packages (R maybe?) which can help me in this?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sorry for the hack representation of the data matrix.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5044" LastEditorUserId="5044" LastEditDate="2016-04-19T18:48:32.240" LastActivityDate="2016-04-24T10:53:11.183" Title="Data visualization of frequencies of state transitions (possibly in R?)" Tags="|r|visualization|markov-process|stata|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11219" PostTypeId="2" ParentId="11167" CreationDate="2016-04-15T00:24:40.087" Score="2" Body="&lt;p&gt;You can use a clustering algorithm such as k-means to divide the generators into groups.  You never know what kind of groups you'll get until you try it.  Try and assess the character of each group of generators as you increase the number of clusters.  At some point you should find a meaningful division of generators.  The inputs to your k-means algorithm will be the criteria you mentioned in your post: the number of times it was activated, the number of activation problems, and so forth.  When you are finished, the group a generator belongs to is its ranking.  This method will not generate a ranking of 1-1000 if you have 1000 generators.  Rather it will give you, for example with k=3: a group of 243 outstanding generators, 320 average generators, and 446 terrible generators.   &lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-04-15T00:24:40.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11220" PostTypeId="1" CreationDate="2016-04-15T03:33:55.863" Score="8" ViewCount="18480" Body="&lt;p&gt;I am currently working on sentiment analysis using Python. I wanted to find whether reviews given for a movie is positive or negative based on sentiment analysis. I have found a training dataset as provided in &lt;a href=&quot;https://inclass.kaggle.com/c/si650winter11/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This dataset have reviews and a score with 1 indicating review is positive and 0 indicating movie review is negative, but it has less number of records. I have a test dataset which I will predicting based on training set. My test dataset has complex and long words for which my python ML model sometimes gives positive result for a negative reviews (returning result as 1 for negative review). I am looking for better dataset to train my model, so that my model can predict well. Could you please advise me any good/large and effective training dataset to use for this scenario? If you could share any links that would be great. The training data can be in format as 1 for positive reviews and 0 for negative reviews or even polarity like pos for positive reviews or neg for negative reviews.&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="98307" LastEditDate="2020-08-17T18:55:20.607" LastActivityDate="2022-01-12T23:37:01.760" Title="Training Dataset for Sentiment Analysis of Movie Reviews" Tags="|machine-learning|python|classification|data-mining|sentiment-analysis|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11221" PostTypeId="1" CreationDate="2016-04-15T06:40:50.733" Score="2" ViewCount="96" Body="&lt;p&gt;I have a response variable series which will be generated randomly in a fixed interval [0-100] base on every second,&#xA;and I want to detect the event when the new generated data is significantly greater than data of last second, and send alarm message to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I calculate the difference of response variable by 1 lag and divided by difference of time (slope), &#xA;than use bootstrapping to construct the 95% confidence interval of response's 90% percentile, if the new data is greater than Upper Limit, I define it as unusual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the quantity of data I have is extremely large, and these data update frequently.&#xA;So, it will be horrible to resampling and calculate the CI, even it could be a way to solve problem.&#xA;And it's looks like not necessarily to sampling because I have large sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any better way to do? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17899" LastEditorUserId="17899" LastEditDate="2016-04-15T07:57:40.497" LastActivityDate="2024-02-07T05:07:05.747" Title="Detect unusal slope increasing" Tags="|data-mining|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11222" PostTypeId="1" CreationDate="2016-04-15T07:03:52.583" Score="3" ViewCount="135" Body="&lt;p&gt;In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent,  Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By metaheuristics I mean methods such as simulated annealing, ant colony optimization, etc., which were developed to avoid getting stuck in a local minima.&lt;/p&gt;&#xA;" OwnerUserId="16424" LastActivityDate="2023-10-18T05:06:08.270" Title="In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?" Tags="|machine-learning|neural-network|deep-learning|optimization|backpropagation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11223" PostTypeId="2" ParentId="11222" CreationDate="2016-04-15T07:35:53.000" Score="6" Body="&lt;ol&gt;&#xA;&lt;li&gt;It would be a waste of information; the gradient is available, so use it and save time.&lt;/li&gt;&#xA;&lt;li&gt;There is reason to believe that the local optima are good; see, for example, &lt;a href=&quot;http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf&quot; rel=&quot;nofollow&quot;&gt;Choromanska et al.&lt;/a&gt; (&lt;a href=&quot;http://wikicoursenote.com/wiki/The_loss_surfaces_of_multilayer_networks_(Choromanska_et_al.)&quot; rel=&quot;nofollow&quot;&gt;notes&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by &lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;nofollow&quot;&gt;stopping early&lt;/a&gt;. Probably the best free lunch in machine learning.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-04-15T15:43:47.197" LastActivityDate="2016-04-15T15:43:47.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11224" PostTypeId="2" ParentId="11214" CreationDate="2016-04-15T08:16:18.840" Score="2" Body="&lt;p&gt;Given m rows of n columns, I think it's natural to think of the data as n-dimensional. However the inherent dimension d of the data may be lower; d &amp;lt;= n. d is the rank of the m x n matrix you could form from the data. The dimensionality of the data can be reduced to d with no loss of information, even. The same actually goes for rows, which is less intuitive but true; d &amp;lt;= m. So, it always makes sense to reduce dimensionality to something &amp;lt;= d since there's no loss; we typically reduce much further. This is why it won't let you reduce to more than the number of rows.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-04-15T08:16:18.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11225" PostTypeId="1" CreationDate="2016-04-15T11:34:49.060" Score="2" ViewCount="1376" Body="&lt;p&gt;I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the optimal size of training data? I understand that I should collect as much data as possible, however is there any empirical advice?&lt;/p&gt;&#xA;" OwnerUserId="15693" LastEditorUserId="17910" LastEditDate="2016-04-15T18:40:29.953" LastActivityDate="2016-07-14T15:47:54.210" Title="How big should training data be?" Tags="|data-mining|dataset|training|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11226" PostTypeId="2" ParentId="11225" CreationDate="2016-04-15T12:00:19.140" Score="1" Body="&lt;p&gt;The ratio between no of samples and features should be more than 10:1 to get sufficiently decent results. But this ratio also varies with applications. If training data is very less : make sure that dataset is balanced, try ensemble methods, careful feature selection, perform cross validation&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2016-04-15T12:00:19.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11227" PostTypeId="2" ParentId="11222" CreationDate="2016-04-15T12:09:17.077" Score="3" Body="&lt;p&gt;In a large neural network there can be millions of free parameters, and assessing the current heuristic means evaluating the network for a significant number of training records, assuming supervised or semi-supervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient-based methods can cope with this amount of parameters in a reasonable time. The gradient can be approximated for all parameters in a single pass through a subset of the training data, and a single step will update all of parameters at once. There is theoretical support showing gradient-based changes to weights should converge to a better solution (caveat being local minima). There is no theoretical result that shows that a set of weights with optimal values for &lt;em&gt;some&lt;/em&gt; of its parameters (in some local subset of all the weights) should be better at a task than a set of random weights. This puts search-based methods that look to build good local choices and combine them at a disadvantage in that we really don't know whether they have a chance of working for a specific problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Genetic algorithms have successfully been used to train small/medium networks, e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies&quot; rel=&quot;nofollow&quot;&gt;NEAT&lt;/a&gt;. This works well in small control systems with immediate feedback (although reinforcement learning may be a better choice in many cases), and also commonly seen in a-life simulations. But this approach does not scale up to complex networks, or supervised learning scenarios with large amounts of data. The cost of assessing the GA against the training data for each member of the population, and the size of population/number of assessments required for a search for a genome that might be several megabytes, makes GA for large networks computationally too expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simulated annealing cannot be used directly to search weights in a standard feed-forward network, since it needs a heuristic calculated (e.g. from a significant portion of training data) for every tiny &quot;move&quot; mode in parameter space - the cost of each move needs to be fast to compute. However, Boltzmann Machine training is closely related to simulated annealing, and &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;Restricted Boltzmann Machines&lt;/a&gt; can be used to pre-train deep neural networks, amongst other things, so there are cases where SA is sort-of used.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-04-15T12:09:17.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11228" PostTypeId="2" ParentId="11202" CreationDate="2016-04-15T12:17:39.860" Score="1" Body="&lt;p&gt;Have a look on HMM (Hidden Markov Model) also. A concrete example Of HMM is available in &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;wikipedia&lt;/a&gt;. Decision tree is better&#xA;in generalising and applying learned data in another context and a Markov-model is better to recall the exact&#xA;learned machine state.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2016-04-15T12:17:39.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11229" PostTypeId="2" ParentId="6201" CreationDate="2016-04-15T12:18:48.987" Score="0" Body="&lt;p&gt;Look at &lt;a href=&quot;http://openfootball.github.io/&quot; rel=&quot;nofollow&quot;&gt;http://openfootball.github.io/&lt;/a&gt; there are a lot of data for soccer South, North America, Europe and Asian is included&lt;/p&gt;&#xA;" OwnerUserId="17904" LastActivityDate="2016-04-15T12:18:48.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11230" PostTypeId="2" ParentId="11220" CreationDate="2016-04-15T12:23:42.523" Score="3" Body="&lt;p&gt;There are many datasets available.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.jhu.edu/%7Emdredze/datasets/sentiment/index2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-Domain Sentiment Dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Twitter sentiment&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://thinknook.com/wp-content/uploads/2012/09/Sentiment-Analysis-Dataset.zip&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sentiment Analysis Dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ai.stanford.edu/%7Eamaas/data/sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Large Movie Review Dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="98307" LastEditDate="2020-08-17T18:55:13.353" LastActivityDate="2020-08-17T18:55:13.353" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11232" PostTypeId="1" CreationDate="2016-04-15T14:16:12.037" Score="-1" ViewCount="60" Body="&lt;p&gt;Suppose you have the historical location data of users, you also know the locations of shops on the street, can you infer what a user is doing (e.g. taking a tube, being on a business trip, traveling, staying at home, shopping)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use any other obtainable data of the user, e.g. Twitter, Amazon, Expedia, Facebook...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an open question. Please give your plan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are doing local-based-service by profiling our users. It would be much appreciated if the admins can un-hold this question. We really need some informative inputs like the one provided by @Marcus D&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks so much!&lt;/p&gt;&#xA;" OwnerUserId="17908" LastEditorUserId="17908" LastEditDate="2016-04-17T00:34:05.020" LastActivityDate="2016-04-17T00:34:05.020" Title="How do I infer user activities from historical location data and other relevant data?" Tags="|bigdata|predictive-modeling|" AnswerCount="1" CommentCount="1" ClosedDate="2016-04-15T18:40:04.933" ContentLicense="CC BY-SA 3.0" />
  <row Id="11235" PostTypeId="2" ParentId="11232" CreationDate="2016-04-15T15:58:21.273" Score="3" Body="&lt;p&gt;It sounds a little like homework, your bio suggests otherwise, but geographic data is very interesting, so I thought I'd give it a bit of a comment!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have &lt;a href=&quot;https://www.google.com/maps/timeline&quot; rel=&quot;nofollow&quot;&gt;location tracking&lt;/a&gt; in Google turned on, which the data can be downloaded, you can determine many things. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to enrich your data with extra GPS data sets of locations of schools, shops, particular buildings, places of worship, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some for examples of what can be determined ... &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;During a 'work day' you can determine a 'normal place of work'. &lt;/li&gt;&#xA;&lt;li&gt;In the 'evening' (for most people) the gps locator will become static, this is a good indication of 'home'&lt;/li&gt;&#xA;&lt;li&gt;You can probably infer if a person has children if they go to a primary or secondary school at 9am and 3pm. You can guess at their ages depending on the type of school.&lt;/li&gt;&#xA;&lt;li&gt;By following the route between 'work' and 'home' you can determine, by the congruence of roads or rail networks, and the locations of bus stops / train stations, the mode of transport.&lt;/li&gt;&#xA;&lt;li&gt;If the data you have enriched is detailed enough, you can determine an individual's choice in lunch options.&lt;/li&gt;&#xA;&lt;li&gt;If there are periods of increased velocity, depending on the time of day, you can determine that they perform exercise, either by running or by pushbike. Further to that you might be able to determine the consistency of their fitness resolve by the frequency and duration of their exercise.&lt;/li&gt;&#xA;&lt;li&gt;You can start building a more detailed profile of the person, by looking at how often they eat out, if they attend church, if they go to sporting matches, what team they support, where they get their car serviced. All these things can give insight into an individual with regard to their lifestyle, habits, preferences, affiliations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-15T15:58:21.273" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11237" PostTypeId="1" CreationDate="2016-04-15T16:25:02.490" Score="1" ViewCount="63" Body="&lt;p&gt;If starting from scratch, what hardware and software would be ideal and under \$5k or \$10k for manipulating a data set of 10 to 20 million records without slowing to the crashing point or taking several hours to complete? Manipulation, if it matters, means multivariate regression (simple OLS with less than a dozen variables), basic graphs, and tables.&lt;/p&gt;&#xA;" OwnerUserId="17913" LastEditorUserId="12876" LastEditDate="2016-04-16T08:48:05.040" LastActivityDate="2016-04-16T10:45:25.797" Title="Soft and hardware for manipulating multi-million record data sets" Tags="|bigdata|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11238" PostTypeId="2" ParentId="11225" CreationDate="2016-04-15T16:45:12.097" Score="3" Body="&lt;p&gt;While the other answers aren't wrong, they don't touch anything about bioinformatics. I'll go into details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In bioinformatic, simply asking for the size of the training set makes no sense. You'll need to understand deeper about bioinformatic to answer this question. There is a good reference: &lt;a href=&quot;https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0758-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;An ensemble approach to accurately detect somatic mutations using SomaticSeq&lt;/a&gt; for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of the project is to build a gradient-boosting machine for classifying somatic mutation given an alignment file. This is very close to what you're doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you want to know the training size? You should ask yourself the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sequencing errors&lt;/li&gt;&#xA;&lt;li&gt;Reads&lt;/li&gt;&#xA;&lt;li&gt;Sequencing depth&lt;/li&gt;&#xA;&lt;li&gt;Sensitivity&lt;/li&gt;&#xA;&lt;li&gt;F1 score&lt;/li&gt;&#xA;&lt;li&gt;Allele frequency&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In an next-generation sequencing, your sequencing machine always give you errors. How reliable is your pipeline to detect those errors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How good are your reads for your problem? For example, if you're detecting disease related to structural variant, you might need more samples to compsenate for the short reads.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's your sequencing depth? In the paper I quote, they have like 30x depth. It'd be pointless to talk about the training size unless you know well about your depth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does your sequencing cover well your region of the genome you're interested in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sensitivity and F1 score. You can have good idea by drawing an F1 curve something like as in the paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ezTly.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ezTly.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's your expected allele frequency range? If you have something like close to 0.5 (ie: maximum heterozygosity), you'll probably be safe with a smaller sample size. However, if you're trying to detect a rare disease, something like &amp;lt;0.1 as in the diagram, you'll need to have more samples and most likely more features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Empirically, we generally use spike-in standards to measure the sensitivity. Once you draw the sensitivity against training size, you'll know at what training size to use achieve whatever sensitivity you're looking for. A typical tool would be ROC.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-04-15T16:45:12.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11240" PostTypeId="1" CreationDate="2016-04-15T18:01:05.170" Score="2" ViewCount="453" Body="&lt;p&gt;I have used K means clustering. In order to find the best value for K, I've looked at the changes of inertia value vs K and also changes of average Silhouette number vs K. The graph for inertia seems to indicate there are 5 clusters in the data. However, the average Silhouette number reaches a minimum at 5. So, how does one interpret this? If there are 5 clusters in the data, shouldn't the average Silhouette number have a maximum at 5 (not a minimum)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17916" LastEditorUserId="17916" LastEditDate="2016-04-15T21:51:39.027" LastActivityDate="2021-10-06T11:43:35.153" Title="Question about the Silhouette number for K means clustering" Tags="|clustering|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11241" PostTypeId="1" CreationDate="2016-04-15T18:15:10.900" Score="1" ViewCount="144" Body="&lt;p&gt;I've used K means to cluster my data. Before using K means, I had used StandardScaler on my data to standardize the data. Now, I'm wondering how can I show the clusters of the original data. Scikit-learn gives the labels on the standardized data but I want to have the labels on the original data and show the clusters of the original data on the graph.&lt;/p&gt;&#xA;" OwnerUserId="17916" LastEditorUserId="85045" LastEditDate="2021-03-23T13:25:46.347" LastActivityDate="2021-03-23T14:47:01.230" Title="Question about graphing the clusters in K means" Tags="|clustering|scikit-learn|k-means|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11242" PostTypeId="2" ParentId="11201" CreationDate="2016-04-15T20:38:51.640" Score="1" Body="&lt;p&gt;I tried Azure Machine Learning (by Microsoft) and it is very nice. It can easily handle large data sets and has similar to weka pipeline for ML tasks and visualisation. They offer it free for 30 days so you can try it. You can google it or follow this link &lt;a href=&quot;https://azure.microsoft.com/en-gb/services/machine-learning/&quot; rel=&quot;nofollow&quot;&gt;Azure ML&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14070" LastActivityDate="2016-04-15T20:38:51.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11243" PostTypeId="2" ParentId="11199" CreationDate="2016-04-15T20:58:56.547" Score="0" Body="&lt;p&gt;Since some features are missing for specific sources, the missing values are not missing-at-random but are systematically missing. In this situation, I'd advise against doing clustering on the combined data set with all available features. If missing values were occurring at random, you could have used some missing value imputation method before performing cluster analysis. However, since the values are systematically missing, imputation would be difficult to tackle. (You could try to predict those missing values, but I am afraid that will add a lot of unnecessary noise in the data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend choosing from one of these two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Perform clustering on the combined data set, &lt;em&gt;but&lt;/em&gt; use only those features that are non-missing across all sources. &lt;/li&gt;&#xA;&lt;li&gt;Perform three different cluster analysis, one for each source. This way, you can ensure that you are using as many features (information) as possible. The determination of &quot;abnormal&quot; behavior can then be determined within each source. This can be an added benefit since it would allow you to be more specific about why a use might be abnormal, as you have more features that can be used to explain this. The results can also be then summarized across all sources to create one consolidated report.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17397" LastActivityDate="2016-04-15T20:58:56.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11244" PostTypeId="1" AcceptedAnswerId="11369" CreationDate="2016-04-15T21:43:51.117" Score="4" ViewCount="916" Body="&lt;p&gt;I'm planning to use weather data to predict crime in Chicago.&#xA;I'm confused which to use as features: historical weather forecast data or historical actual weather data?&#xA;And what is the underlying logic here?&#xA;I'm assuming later only forecast data is available for prediction. Thank you for any input.&lt;/p&gt;&#xA;" OwnerUserId="14379" LastActivityDate="2016-04-22T15:21:38.733" Title="Use forecast weather data or actual weather data for prediction?" Tags="|predictive-modeling|feature-selection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11245" PostTypeId="2" ParentId="11201" CreationDate="2016-04-15T21:45:25.860" Score="1" Body="&lt;p&gt;First I wanted to propose &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange Data Mining Suite&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YnQh6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YnQh6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, I know almost for certain that there isn't a Windows version of 64bit NumPy available, which Orange uses, and thus your data probably won't fit in with the 32bit Windows release they provide out of the box. It may be installable if you can find your way around Python well enough, but it may just also prove being easier to switch onto some more productive platform (i.e. GNU/Linux and Macintosh OS X are supported). &lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="471" LastEditDate="2016-07-01T15:20:55.587" LastActivityDate="2016-07-01T15:20:55.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11246" PostTypeId="2" ParentId="11244" CreationDate="2016-04-15T22:00:33.837" Score="2" Body="&lt;p&gt;You could you &lt;strong&gt;both&lt;/strong&gt; forecast and actual weather, then see which provide better features for forecasting? Obviously there will be a lot of correlation between the forecast and the actual weather, but you could use feature selection to choose the most important?&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-15T22:00:33.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11247" PostTypeId="1" CreationDate="2016-04-16T03:52:54.283" Score="4" ViewCount="2035" Body="&lt;p&gt;I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From the input dataset, I am using a logic to remove stopwords and after that training my dataset to predict the result. As of now, my test results are predicted, but few results are predicted wrongly. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;This was a great film. Jack Halley is comical as usual and Bela Lugos is  hilarious even in his tiny role. I love this flick.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example, the above sentence is positive, but it is predicting as negative. Similar instance occurs to many other scenarios. If it has some sarcasm, then there are chances that it may predict the other way. But for few direct reviews, it is predicting the other way. &#xA;Could you suggest other ways to better my algorithm, so that it increases the accuracy of prediction. Any other efficient methods apart from removal of stopwords? I have been looking for various suggestions through internet but I am new to this topic and hence couldn't find an effective way to proceed. Any good suggestions would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="471" LastEditDate="2016-04-18T11:41:31.560" LastActivityDate="2022-01-13T19:11:28.090" Title="Sentiment Analysis of Movie Reviews using Python" Tags="|python|data-mining|bigdata|pandas|nltk|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11248" PostTypeId="1" CreationDate="2016-04-16T07:08:10.233" Score="1" ViewCount="628" Body="&lt;p&gt;I am trying to interpret the p-values for model selection. Here is a sample code taken from a book (&lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow noreferrer&quot;&gt;An Intro. to stat. Learning, page 290, by Gareth James et al.&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qo8f7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qo8f7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;￼&#xA;Null-hypothsis: Model &lt;code&gt;M1&lt;/code&gt; is sufficient to explain the data against the alternative hypothesis that a more complex model &lt;code&gt;M2&lt;/code&gt;is required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this statement, I think p-value should be high so that null-hypotheis can be rejected and a model &lt;code&gt;M2&lt;/code&gt; is selected. Keeping this in mind, I think we should select Model 5 as it is having highest p-value, but authors mention either of Model 3 or Model 4 is a good fit. I don't understand why Model 5 is not a good fit.&lt;/p&gt;&#xA;" OwnerUserId="13291" LastActivityDate="2016-06-16T14:19:42.393" Title="Model selection by Interpreting p-value of anova function" Tags="|machine-learning|r|statistics|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11249" PostTypeId="2" ParentId="11244" CreationDate="2016-04-16T07:44:29.023" Score="1" Body="&lt;p&gt;I'd suggest trying &lt;strong&gt;both&lt;/strong&gt; simultaneously. My thought is that certain weather conditions make for better situations to commit crime. If my assumption is true, then it is likely that criminals use this to their advantage and look at weather predictions. Don't laugh at me.  &lt;/p&gt;&#xA;" OwnerUserId="16591" LastActivityDate="2016-04-16T07:44:29.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11250" PostTypeId="2" ParentId="11237" CreationDate="2016-04-16T10:45:25.797" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use vowpal wabbit for building your regression or classification models. It'll train at the speed at which you can read your data &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Use pandas and dask for plotting graphs etc.&#xA;You can use an AWS i2.8xlarge or equivalent machine which will cost you ~ USD 6k for an year&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-04-16T10:45:25.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11252" PostTypeId="2" ParentId="11241" CreationDate="2016-04-16T13:49:57.043" Score="0" Body="&lt;p&gt;Option 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep and access the original data (e.g. by index) - recompute the means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apply the inverse transformation. StandardScaler is a linear transformation, so its reversible up to some loss of precision.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-04-16T13:49:57.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11253" PostTypeId="2" ParentId="11240" CreationDate="2016-04-16T13:53:22.297" Score="2" Body="&lt;p&gt;These are only &lt;em&gt;heuristics&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;These are not very reliable metrics, and often fail. They will not detect if you did not preprocess your data well.&lt;/p&gt;&#xA;&lt;p&gt;So instead of looking at some number, look at your data instead.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="33245" LastEditDate="2021-10-06T11:43:35.153" LastActivityDate="2021-10-06T11:43:35.153" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11254" PostTypeId="2" ParentId="11248" CreationDate="2016-04-16T14:49:43.280" Score="1" Body="&lt;p&gt;To understand why, we should try to understand what we're doing here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's start with the first p-value: &lt;code&gt;2e-16&lt;/code&gt;, what does that mean? This is the p-value under the null hypothesis that the linear model and the second polynomial model are statistically identical. We say that they are the same if the extra coefficient in the polynomial is statistically zero. This is exactly what the p-value is telling you. It reads like this: &lt;code&gt;There is 2e-16 probability that the null hypothesis of the linear model and the polynomial are identical&lt;/code&gt;. This is a very small probability, so you can conclude that they are not identical. This means, the second-order polynomial is a &lt;strong&gt;better&lt;/strong&gt; model than the linear model with a significance level of 5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(PS: a more correct statistical interpretation of the p-value relates to false-positive rejection, but let's not go into that deep)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, using the same logic, you can conclude the third-order polynomial fits &lt;strong&gt;better&lt;/strong&gt; than the second-order polynomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, the p-value comparing the third and fourth order polynomial is about 0.05. You can reject it or you don't want to reject it, this is up to you. But If was you, I'd simply fail to reject it because it's larger than 0.05.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the final p-value is about &lt;code&gt;0.37&lt;/code&gt; which is too high. This means although the fifth order polynomial fits better than the fourth order, the loss in your RSS is insufficient to justify the loss of the degree of freedom. Therefore, we say &lt;code&gt;the fifth order polynomial is statistically no better than the fourth-order&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;: It's &quot;bad&quot; to have a large p-value because you really want to reject a null hypothesis. Statistically, we do it to control the false positive rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: In your example, R uses the F-test to compare the two models.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-04-17T12:56:39.280" LastActivityDate="2016-04-17T12:56:39.280" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11255" PostTypeId="1" AcceptedAnswerId="11256" CreationDate="2016-04-16T15:35:41.177" Score="2" ViewCount="308" Body="&lt;p&gt;Why for logistic regression, with target values 0 or 1, it will not work to take the sum of the squares of the difference between target value and prediction, but rather:&#xA;$$&#xA;error({\bf w}) = -1/m * \sum_{i=1}^{m} [ y_i \ln (\sigma({x_i})) + (1-y_i) \ln (1 - \sigma({x_i} ) ]&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="9728" LastEditDate="2016-11-21T13:30:27.307" LastActivityDate="2016-11-21T13:30:27.307" Title="Why for logistic regression the error is given by [y ln(sigma(x)) + (1 − y) ln(1 − sigma(x)]" Tags="|logistic-regression|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11256" PostTypeId="2" ParentId="11255" CreationDate="2016-04-16T18:38:52.717" Score="4" Body="&lt;p&gt;This is the log-likelihood: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\log P(x; w) \equiv \log \prod_i P(x_i | w) = \sum_i \log P(x_i | w)$, where $P(x_i | w) \equiv \left\{ \begin{array}{rl}\sigma(x_i), &amp;amp; y_i =1 \\ 1 - \sigma(x_i), &amp;amp;y_i = 0\end{array} \right.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why the log-likelihood? When you have a probabilistic model, such as logistic regression, it's one way (the &lt;a href=&quot;https://en.wikipedia.org/wiki/Maximum_likelihood&quot; rel=&quot;nofollow&quot;&gt;MLE&lt;/a&gt;) of finding the parameters that fit best. Recall that in logistic regression we are, contrary to the name, trying to &lt;em&gt;classify&lt;/em&gt; rather than regress, and the MSE is a regression loss; it seeks to minimize the distance from a point, while &lt;a href=&quot;https://en.wikipedia.org/wiki/Loss_functions_for_classification&quot; rel=&quot;nofollow&quot;&gt;we wish to penalize being in the wrong &lt;em&gt;subspace&lt;/em&gt;&lt;/a&gt; (the parts that don't correspond to the correct class). If you squint a bit, you can see that the negative log-likelihood minimizes the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression&quot; rel=&quot;nofollow&quot;&gt;cross entropy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-04-16T19:13:02.627" LastActivityDate="2016-04-16T19:13:02.627" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11257" PostTypeId="1" CreationDate="2016-04-16T21:27:44.907" Score="3" ViewCount="140" Body="&lt;p&gt;I would appreciate your comments/help about a strategy I am applying in one of my analysis. In short, my case is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) My data have biological origin, collected in a period of 120s, from a&#xA; subject receiving, each time, one of possible three stimuli (response label 1&#xA; to 3), in a random manner, one stimulus per second (trial). Sampling &#xA; frequency is 256 Hz and 61 different sensors (input variables). So, my &#xA; dataset has 120x256 rows and 62 columns (1 response label + 61 input &#xA; variables);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) My goal is to identify if there is an underlying pattern for each stimulus.&#xA; For that, I would like to use deep learning neural networks to test my&#xA; hypothesis, but not in a conventional way (to predict the stimulus from a&#xA; single observation/row).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) My approach is to divide the whole dataset, after shuffling per row&#xA; (avoiding any time bias), in training and validation sets (50/50) and then to&#xA; run the deep learning algorithm. The division does not segregate trial events&#xA; (120), so each training/validation sets should contain data (rows) from the&#xA; same trial (but never the same row). If there is a dominant pattern per&#xA; stimulus, the validation confusion matrix error should be low. If there is a&#xA; dominant pattern per trial, the validation confusion matrix error should be&#xA; high. So, the validation confusion matrix error is my indicator of the&#xA; presence of a hidden pattern per stimulus;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any input you could provide me regarding the validity of my logic. I would like to emphasize that I am not trying to predict the stimulus based on row inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17933" LastActivityDate="2016-04-16T21:27:44.907" Title="Heterodox use of Deep Learning to find hidden patterns" Tags="|machine-learning|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11258" PostTypeId="2" ParentId="11247" CreationDate="2016-04-17T04:50:52.013" Score="1" Body="&lt;p&gt;I'm assuming that you are using bag of words, you can try adding bigrams and/or trigrams (or really any other arbitrary n-grams) to your vocabulary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also had a lot of success using latent dirichlet allocation to predict the distribution of topics for a particular sentence. Feeding that distribution into the naive bayes algorithm as features. It is something of a hack but it seems to work really well for the limited sentiment analysis that I have done. My guess is that the LDA does a decent job of picking up on sarcasm. It isn't full proof, but it might help you get better predictions. These are some of the things that I would try at the very least.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-04-17T04:50:52.013" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11260" PostTypeId="2" ParentId="11247" CreationDate="2016-04-17T16:43:57.390" Score="1" Body="&lt;p&gt;Mainly the accuracy depends upon pre-processing steps, features extracted and the learning model used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pre-processing steps normally includes removal of stop words and that is fine. Features extraction is of various methods. Word embeddings is gaining its popularity in NLP, due to its interesting characteristics of vectors generated. &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;Gensim&lt;/a&gt; provides a nice python library for word embeddings both word2vec as well as doc2vec models. For the detailed algorithm of how it works, read &lt;a href=&quot;https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt;, &lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow&quot;&gt;doc2vec&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are lot of learning models from naive bayes, svm to neural network models. The accuracy of it depends upon the dataset used and the features generated and so each models need to be tested under trial and error method. &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;sklearn&lt;/a&gt; provides a nice support for ML models.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-04-17T16:43:57.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11261" PostTypeId="1" CreationDate="2016-04-17T19:07:56.067" Score="2" ViewCount="9578" Body="&lt;p&gt;Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?&lt;/p&gt;&#xA;" OwnerUserId="17939" LastActivityDate="2020-02-19T18:29:20.333" Title="Linear Regression and learning rate" Tags="|linear-regression|" AnswerCount="3" CommentCount="1" ClosedDate="2021-01-26T18:51:44.433" ContentLicense="CC BY-SA 3.0" />
  <row Id="11265" PostTypeId="2" ParentId="11261" CreationDate="2016-04-18T01:35:25.253" Score="2" Body="&lt;p&gt;Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-04-18T01:35:25.253" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11266" PostTypeId="1" CreationDate="2016-04-18T03:41:21.950" Score="2" ViewCount="1191" Body="&lt;p&gt;I have a dataset of &lt;code&gt;&amp;lt;user, item&amp;gt;&lt;/code&gt; pairs where each entry records which user bought which item.&#xA;e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;u1, i1&amp;gt;&#xA;&amp;lt;u1, i4&amp;gt;&#xA;&amp;lt;u2, i2&amp;gt;&#xA;&amp;lt;u3, i2&amp;gt;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I created an encoded dataset with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;no_of_features = no_of_users + no_of_items&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and have set output variable y to 1, as each of those entries represented the user having bought that particular item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: All &lt;code&gt;y&lt;/code&gt; values are 1 in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The encoded dataset looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user1 user2 user3 .... item1 item2 item3 item4 .... y&#xA;&#xA;  1     0    0    ....   1     0     0     0   .... 1&#xA;  1     0    0    ....   0     0     0     1   .... 1&#xA;  0     1    0    ....   0     1     0     0   .... 1&#xA;  0     0    1    ....   0     1     0     0   .... 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to know how to use &lt;code&gt;fastFM&lt;/code&gt; to generate recommendations for cases &lt;code&gt;&amp;lt;x, y&amp;gt;&lt;/code&gt; for any user &lt;code&gt;x&lt;/code&gt; and any item &lt;code&gt;y&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly &lt;code&gt;regression&lt;/code&gt; is out of the picture here. So should I use the &lt;code&gt;classification&lt;/code&gt; or &lt;code&gt;ranking&lt;/code&gt; approach of &lt;code&gt;fastFM&lt;/code&gt;? Also how? e.g. If I use &lt;code&gt;classification&lt;/code&gt;, do I need to generate instances with &lt;code&gt;y=0&lt;/code&gt;? If I use &lt;code&gt;ranking&lt;/code&gt; approach, do I arrange items by the output of ranking approach and recommend those with higher values?&lt;/p&gt;&#xA;" OwnerUserId="17947" LastActivityDate="2016-04-18T03:41:21.950" Title="How to use Python's FastFM library (factorization machines) for recommendation tasks?" Tags="|python|recommender-system|scikit-learn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11267" PostTypeId="1" CreationDate="2016-04-18T03:50:41.830" Score="1" ViewCount="455" Body="&lt;p&gt;Suppose I am given a set of structured data. The data is known to be problematic, and I need to somehow &quot;score&quot; them on consistency. For example, I have the data as shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fieldA | fieldB | fieldC&#xA;-------+--------+-------&#xA;foo    | bar    | baz&#xA;fooo   | bar    | baz&#xA;foo    | bar    | lorem&#xA;..     | ..     | ..&#xA;lorem  | ipsum  | dolor&#xA;lorem  | upsum  | dolor&#xA;lorem  | ipsum  | baz&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So assume the first row is considered the correct entry because there are relatively more data in that combination compared to the records in second and third row. In the second row, the value for &lt;code&gt;fieldA&lt;/code&gt; should be &lt;code&gt;foo&lt;/code&gt; (inconsistent due to misspelling). Then in the third row, the value of &lt;code&gt;fieldC&lt;/code&gt; should be &lt;code&gt;baz&lt;/code&gt; as other entries in the dataset with similar values for &lt;code&gt;fieldA&lt;/code&gt; (&lt;code&gt;foo&lt;/code&gt;) and &lt;code&gt;fieldB&lt;/code&gt; (&lt;code&gt;bar&lt;/code&gt;) suggest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, in other part of the dataset, there's another combination that is relatively more common (&lt;code&gt;lorem&lt;/code&gt;, &lt;code&gt;ipsum&lt;/code&gt;, &lt;code&gt;dolor&lt;/code&gt;). So the problem in the following records are the same as the one mentioned before, just that the value combination is different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially dumped everything to a SQL database, and use statements with &lt;code&gt;GROUP BY&lt;/code&gt; to check consistency of fields values. So there will be 1 query for each field I want to check for consistency, and for each record.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT    fieldA, count(fieldA)&#xA;FROM      cache&#xA;WHERE     fieldB = 'bar' and fieldC = 'baz'&#xA;GROUP BY  fieldA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I could check if the value of &lt;code&gt;fieldA&lt;/code&gt; of a record is consistent with the rest by referring the record to the object below (processed result of the previous SQL query).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'foo': {'consistency': 0.99, 'count': 99, 'total': 100}&#xA; 'fooo': {'consistency': 0.01, 'count': 1, 'total': 100}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However it was very slow (dataset has about 2.2million records, and I am checking 4 fields, so making about 9mil queries), and would take half a day to complete. Then I replaced SQL storage to elasticsearch, and the processing time shrunk to about 5 hours, can it be made somehow faster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also just out of curiosity, am I re-inventing a wheel here? Is there an existing tool for this? Currently it is implemented in Python3, with elasticsearch.&lt;/p&gt;&#xA;" OwnerUserId="13562" LastActivityDate="2016-04-18T03:50:41.830" Title="Scoring consistency within dataset" Tags="|python|data-cleaning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11268" PostTypeId="2" ParentId="11060" CreationDate="2016-04-18T06:11:06.527" Score="1" Body="&lt;p&gt;Try PCA only on sparse features, and combine PCA output with dense features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you'll get dense set of (original) features + dense set of features (which were originally sparse).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;+1 for the question. Please update us with the results.&lt;/p&gt;&#xA;" OwnerUserId="5380" LastActivityDate="2016-04-18T06:11:06.527" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11270" PostTypeId="2" ParentId="11247" CreationDate="2016-04-18T12:02:34.730" Score="2" Body="&lt;p&gt;Given that exact case, I would assume that you are getting negative decision due to names, mentioned in the review (in your training dataset actors were more often met in negative reviews). You should, probably, remove all non-relevant words from reviews, and that includes not only stop words, but all person names (since they are less of a sentiment marker except, maybe, justin bieber, who is a really negative marker to anything :) ).&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-04-18T12:02:34.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11272" PostTypeId="1" CreationDate="2016-04-18T13:10:10.420" Score="3" ViewCount="1152" Body="&lt;p&gt;Boosting methods (such as the popular &lt;code&gt;xgboost&lt;/code&gt;) do not tend to overfit when we use many iterations - &lt;a href=&quot;http://www.cc.gatech.edu/~isbell/tutorials/boostingmargins.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Schapire and Freund&lt;/a&gt;. Are they also resistant to overfitting when we feed them with a large number of features (where some of the features are not very useful?) If so, is there a theoretic connection between these two desirable properties? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(related to &#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality&quot;&gt;https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="16050" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-19T13:31:27.153" Title="Is boosting resistant to overfitting for both number of iterations and number of features?" Tags="|feature-selection|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11273" PostTypeId="1" AcceptedAnswerId="11282" CreationDate="2016-04-18T13:29:07.543" Score="3" ViewCount="219" Body="&lt;p&gt;I would like to know if neo4j can be considered a graph processing platform, even though i know that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;neo4j: is a graph database management system developed by Neo Technology, Inc. Described by its developers as an ACID-compliant transactional database with native graph storage and processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;graph processing platform: is a platform used for processing graphs, by applying global algorithms towards large graphs (used more in OLAP scenario).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this paper &lt;a href=&quot;http://www.ds.ewi.tudelft.nl/~iosup/perf-eval-graph-proc14ipdps.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.ds.ewi.tudelft.nl/~iosup/perf-eval-graph-proc14ipdps.pdf&lt;/a&gt;, neo4j is benchmarked as a processing platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, if its correct to put neo4j in the group of processing platform like pregel and giraph.&lt;/p&gt;&#xA;" OwnerUserId="17596" LastActivityDate="2016-04-19T02:41:59.513" Title="Is it ideally correct to benchmark neo4j as graph processing platform?" Tags="|graphs|neo4j|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11274" PostTypeId="1" CreationDate="2016-04-18T14:07:45.893" Score="6" ViewCount="9389" Body="&lt;p&gt;Given there are two matrices of dimensionality 100x2 with absolute values ranging from -50 to +50. Is it possible to determine the kl-divergence by applying the entropy algorithm from scipy.stats to the two flattened vectors of size 200?&lt;/p&gt;&#xA;" OwnerUserId="17960" LastActivityDate="2016-04-27T00:24:51.427" Title="How to calculate KL-divergence between matrices" Tags="|scikit-learn|tsne|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11276" PostTypeId="2" ParentId="11274" CreationDate="2016-04-18T15:50:32.617" Score="2" Body="&lt;p&gt;I think you can. Just normalize both of the vectors to be sure they are distributions. Then you can apply the kl divergence .&#xA;Note the following:&#xA;- you need to use a very small value when calculating the kl-d to avoid division by zero. In other words , replace any zero value with ver small value &#xA;- kl-d is not a metric . Kl(AB) does not equal KL(BA) . If you are interested in it as a metric you have to use the symmetric kl = (Kl(AB) +KL(BA) )/2&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-18T15:50:32.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11278" PostTypeId="1" AcceptedAnswerId="11353" CreationDate="2016-04-18T19:11:51.947" Score="3" ViewCount="2007" Body="&lt;p&gt;I'm trying to choose an algorithm for filtering spam.&#xA;I found two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Create word dictionary for spam and not spam data.&#xA;Calculate average TF-IDF for each word and use cosine similarity for filtering.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or use word dictionary for training logistic regression model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Could you suggest what fit the best for my goal. &#xA;Maybe I should use some another algorithm.&lt;/p&gt;&#xA;" OwnerUserId="17971" LastActivityDate="2016-04-22T03:27:23.820" Title="Cosine similarity or logistic regression for spam filtering" Tags="|machine-learning|classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11281" PostTypeId="2" ParentId="11278" CreationDate="2016-04-18T22:42:56.007" Score="1" Body="&lt;p&gt;You have to make a prediction (after performing supervised learning), so cosine distance won't help since it's fundamentally used in unsupervised learning (clustering and distance).  A popular spam prediction approach was published in Springer textbook by Hastie &amp;amp; Tibshirani, which is on statistical learning.  You could probably find papers online on spam prediction.   &lt;/p&gt;&#xA;" OwnerDisplayName="user9086" LastActivityDate="2016-04-18T22:42:56.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11282" PostTypeId="2" ParentId="11273" CreationDate="2016-04-19T02:41:59.513" Score="1" Body="&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Emre has rightly pointed out, the Chief Scientist of the company himself has written a blog post claiming the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, Neo4j does a lot of computations which graph processing tools can do. In fact, it does graph traversals much faster than giraph, due to the Hadoop overhead and also as it stores the adjacent nodes in a doubly linked list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, it's not rare to confuse Neo4j with a graph processing platform (thus the claim in the paper) due to it's overlap with the features of a processing platform like Pregel and Giraph.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-04-19T02:41:59.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11284" PostTypeId="1" AcceptedAnswerId="11287" CreationDate="2016-04-19T05:59:02.653" Score="0" ViewCount="8945" Body="&lt;p&gt;In the example given for the max function for PySpark:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://spark.apache.org/docs/1.2.0/api/python/pyspark.html&quot; rel=&quot;nofollow&quot;&gt;Pyspark&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])&#xA;&amp;gt;&amp;gt;&amp;gt; rdd.max()&#xA;43.0&#xA;&amp;gt;&amp;gt;&amp;gt; rdd.max(key=str)&#xA;5.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Q1. How does it get evaluated to 5.0 when key=str ? Is it based on conversion to character/string type ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2. Is there any value the parameter &quot;key&quot; can take ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also found the function definition of &quot;max&quot; at this location&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/adobe-research/spark-gpu/blob/master/src/rdd.py&quot; rel=&quot;nofollow&quot;&gt;https://github.com/adobe-research/spark-gpu/blob/master/src/rdd.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def max(self, key=None):&#xA;        &quot;&quot;&quot;&#xA;        Find the maximum item in this RDD.&#xA;        :param key: A function used to generate key for comparing&#xA;        &amp;gt;&amp;gt;&amp;gt; rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])&#xA;        &amp;gt;&amp;gt;&amp;gt; rdd.max()&#xA;        43.0&#xA;        &amp;gt;&amp;gt;&amp;gt; rdd.max(key=str)&#xA;        5.0&#xA;        &quot;&quot;&quot;&#xA;        if key is None:&#xA;            return self.reduce(max)&#xA;        return self.reduce(lambda a, b: max(a, b, key=key))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15790" LastEditorUserId="21" LastEditDate="2016-05-10T13:56:19.977" LastActivityDate="2019-01-05T19:01:01.060" Title="key parameter in max function in Pyspark" Tags="|python|apache-spark|pyspark|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11285" PostTypeId="2" ParentId="6318" CreationDate="2016-04-19T06:40:33.497" Score="0" Body="&lt;p&gt;One of the best book I have came across is &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1783555130&quot; rel=&quot;nofollow&quot;&gt;Machine Learning in Python&lt;/a&gt; from Sebastian Raschka. Easy examples, step by step explanation and just right amount of math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The structure of book covers the whole process from data cleaning to ensembling and evaluation.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-04-19T06:40:33.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11286" PostTypeId="1" CreationDate="2016-04-19T07:01:35.470" Score="0" ViewCount="1609" Body="&lt;p&gt;I trained pre-trained ResNet-101 following caffe model(By Kaiming He github) with 800000 data for training and 200000 data for validation. After I train this model, I got 59% accuracy for 1st and accuracy-top5 is 82% with 30 epoch as seen below picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iqsxZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iqsxZ.png&quot; alt=&quot;Training process&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I tried predict some images with this model(net.forward()), the results always produce same probability like below even though I tried with other images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9S49y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9S49y.png&quot; alt=&quot;Produces same output&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First thing I thought was image preprocessing problem in predict step like subtracting mean values or adequating batch size corresponding with training step. But all of these step was correctly set up. &#xA;I checked all other questions having a same problem with me but couldn't find a solution. As the above picture(1) is showing, I assume the training process wasn't something wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I followed &quot;&lt;a href=&quot;http://nbviewer.jupyter.org/github/BVLC/caffe/blob/master/examples/00-classification.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image Classification and Filter Visualization(00-classification.ipynb)&lt;/a&gt;&quot; file provided from Caffe editing &lt;code&gt;model_def&lt;/code&gt; and &lt;code&gt;model_weights&lt;/code&gt; with my &lt;code&gt;model_def&lt;/code&gt; and &lt;code&gt;model_weights&lt;/code&gt; having a 30 epoch.&lt;/p&gt;&#xA;" OwnerUserId="17987" LastEditorUserId="31513" LastEditDate="2017-05-15T15:48:51.053" LastActivityDate="2018-04-12T13:09:15.707" Title="Caffe net.predict() , predict same probability" Tags="|machine-learning|neural-network|caffe|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11287" PostTypeId="2" ParentId="11284" CreationDate="2016-04-19T07:48:17.950" Score="3" Body="&lt;p&gt;You pass a function to the key parameter that it will virtually map your rows on to check for the maximum value. In this case you pass the str function which converts your floats to strings. Since '5.0' &gt; '14.0' due to the nature of string comparisons, this is returned. What is usually a more likely use is using the key parameter as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test = sc.parallelize([(1, 2), (4,3), (2,4)])&#xA;test.max(key = lambda x: -x[1])&#xA;(1,2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because of the - we sort descending and the x[1] means we use the second entry in our tuples as the key.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-04-19T14:02:08.067" LastActivityDate="2016-04-19T14:02:08.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11288" PostTypeId="1" CreationDate="2016-04-19T07:53:14.233" Score="0" ViewCount="142" Body="&lt;p&gt;It took me years before I could understand what was going on in a histogram. I had trouble understanding the tails, skewness, and concepts such as &quot;fat tails.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a lot of practice, histograms have become more intuitive for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe I have some sort of &quot;graphical dyslexia.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build an inituitive understanding of basic, common types of data visualizations such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;histograms&lt;/li&gt;&#xA;&lt;li&gt;scatter plots&lt;/li&gt;&#xA;&lt;li&gt;dotplots&lt;/li&gt;&#xA;&lt;li&gt;stem and leaf plots&lt;/li&gt;&#xA;&lt;li&gt;density plots&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By intuitive, I mean that when I glance at one of these plots, I want to figure out the gist of it within a few seconds (rather than stare at it for minutes without anything registering in my mind, &lt;strong&gt;&lt;em&gt;as is usually the case for me&lt;/em&gt;&lt;/strong&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;StackExchange: can you recommend to me books/resources that can help me with this?&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="16928" LastEditDate="2016-04-19T20:13:25.517" LastActivityDate="2016-04-19T20:13:25.517" Title="Graphical Dyslexia -- building intuition for common data visualizations?" Tags="|visualization|" AnswerCount="1" CommentCount="2" ClosedDate="2016-04-19T12:10:03.370" ContentLicense="CC BY-SA 3.0" />
  <row Id="11290" PostTypeId="2" ParentId="11288" CreationDate="2016-04-19T10:05:32.590" Score="4" Body="&lt;p&gt;There are two books that are our &quot;bibles&quot; in data visualisation.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Edward Tufte, &lt;a href=&quot;https://www.amazon.co.uk/The-Visual-Display-Quantitative-Information/dp/0961392142?ie=UTF8&amp;amp;qid=1461059964&amp;amp;ref_=la_B000APET3Y_1_1&amp;amp;s=books&amp;amp;sr=1-1&quot; rel=&quot;nofollow&quot;&gt;The Visual Display of Quantitative Information&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;David McCandless &lt;a href=&quot;https://www.amazon.co.uk/Information-Beautiful-New-David-McCandless-x/dp/0007492898/ref=sr_1_1?s=books&amp;amp;ie=UTF8&amp;amp;qid=1461060112&amp;amp;sr=1-1&amp;amp;keywords=data%20is%20beautiful&quot; rel=&quot;nofollow&quot;&gt;Information is Beautiful&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Tufte has a wealth of examples on good and bad visualisations. McCandless has some lovely, or even beautiful graphics and ways to display data.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-19T10:05:32.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11292" PostTypeId="1" CreationDate="2016-04-19T12:37:05.593" Score="3" ViewCount="3791" Body="&lt;p&gt;I have about 3,000,000 samples and each sample is described by a list of size about 20. Some elements in this list are categorical, for example name of cities, day of week, etc. (some categories have a large number of options, for example one category is url with more than 700,000 unique elements in my dataset!). Also some elements have real values, for example for time of day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is labelled (2 categories,) and I need to train a classifier for test data. I am inclined towards decision tree or random forest since they seem to be a good choice for this type of problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How do I pre-process categorical data? one-hot-encoding seem to be the right choice but given that some of my categories have huge number of possible values, one hot encoder will produce very long words! am I correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How do I combine data from different categories? For example data from category 'cities' with data from 'urls', since they have different lengths. Do I simply concatenate them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) How can I combine categorical data with real valued data, for example 'name of cities' with 'time of day' to produce one matrix that can then be passed to decision tree classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Are there any special normalisation, etc. that I have to do before passing data to classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I plan to use python and Scikit for this task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="17993" LastActivityDate="2016-04-22T05:35:15.567" Title="decision trees on mix of categorical and real value parameters" Tags="|bigdata|scikit-learn|decision-trees|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11293" PostTypeId="2" ParentId="11272" CreationDate="2016-04-19T13:31:27.153" Score="2" Body="&lt;p&gt;I can't speak from a theoretical perspective, but I can say that in practice overfitting with a large number of features is not an issue with gradient boosting. I work at a huge financial institution that has been doing credit risk modeling for decades, and for the past decade or so they have almost exclusively been doing their variable selection using gradient boosting. Gradient boosting (along with any tree-based method) can be used to find relative feature importances (based on how much error is reduced after each split). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statisticians where I work will often use gradient boosting to narrow down 2,000 or so features into a more manageable 10-15 by taking the top ranked features. These features are then fed into a logistic regression model, which is more interpretable than gradient boosting. Even on the hold-out partition, the gradient boosting algorithm built on the 2,000 features generally outperforms the logistic regression model. The gradient boosting model is thus used for variable selection and predictive performance benchmarking. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-04-19T13:31:27.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11295" PostTypeId="1" AcceptedAnswerId="11296" CreationDate="2016-04-19T16:25:01.727" Score="1" ViewCount="460" Body="&lt;p&gt;I'm building a neural net for classifying characters in pictures. The input can be any character a-z, lowercase and uppercase. I only care about classifying the characters, and not the case, so the neural net has an output vector of length 26; one for each character. It makes sense, intuitively, to have a hidden layer of size 26*2 just upstream of the output layer. It also makes intuitive sense for this layer not to be fully connected to the output layer, but instead having two and two hidden nodes connect to each output node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Does this make sense? I'm getting about 75 % success rate on a pretty hard data set with just one hidden layer, but I'm not certain on how to improve from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) If so, What activation function should I use from the hidden layer with 26*2 nodes to the output layer? Maybe I should use an OR function for this, since both the lowercase and the uppercase version of a character should output for a single character.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c) Would it be wiser to have 26*2 output nodes instead, and just combine lowercase and uppercase outputs after the neural net?&lt;/p&gt;&#xA;" OwnerUserId="18000" LastEditorUserId="18000" LastEditDate="2016-04-19T16:30:14.463" LastActivityDate="2016-04-19T16:59:26.347" Title="Character recognition neural net topology/design" Tags="|machine-learning|neural-network|deep-learning|theano|image-recognition|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11296" PostTypeId="2" ParentId="11295" CreationDate="2016-04-19T16:45:00.050" Score="0" Body="&lt;p&gt;Your design makes some sense, but there is no need to limit connections even if you expect to represent probabilities of upper/lower case separately, because they will interact usefully. E.g if the character could most likely be one of &lt;code&gt;o, O, Q, G&lt;/code&gt; then this might be useful information to choose the correct one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you went ahead, you would need to train this network without the final layer (so that it learns the representations you expect, not some other group of 52 features), then add the final layer later, with no need for special connection rules, just use existing ones. Initially you would training the new layer separately from the full output of the 52-class net i.e. probability values, not selected class. Then you would combine with the existing net and fine-tune the result by running a few more epochs with a low learning rate on the final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That all seems quite complex, and IMO unlikely to gain you much accuracy (although I am guessing, it could be great - so if you have time to explore ideas, you could still try). Personally I would not take your hidden layer idea further. The full 52-class version with simple logic to combine results is I think simpler. This is also not necessary, the neural net can learn to have two different-looking images be in the same class quite easily, provided you supply examples of them in training. However, it may give you useful insights into categorisation failures in training or testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear from the question, but if you are not already using convolutional neural network for lower layers, then you should do so. This will make the largest impact on your results by far.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-04-19T16:59:26.347" LastActivityDate="2016-04-19T16:59:26.347" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11297" PostTypeId="2" ParentId="11286" CreationDate="2016-04-19T17:04:56.817" Score="1" Body="&lt;p&gt;I solved this problem. The problem was use_global_stats setting in deploy.prototxt. In training step, use_global_stats has to be set as false because mean/var need's to be update. But when I predict using deploy.prototxt use_global_stats has to be set as true. &lt;/p&gt;&#xA;" OwnerUserId="17987" LastActivityDate="2016-04-19T17:04:56.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11298" PostTypeId="2" ParentId="10064" CreationDate="2016-04-19T18:23:21.590" Score="0" Body="&lt;h2&gt;Question 3&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://synergy.st-andrews.ac.uk/vannesmithlab/research/bayesian-networks/&quot; rel=&quot;nofollow&quot;&gt;synergy.st-andrews.ac.uk/vannesmithlab&lt;/a&gt; claims that the graphs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;G1 = o-&amp;gt;o-&amp;gt;o and&#xA;G2 = o&amp;lt;-o-&amp;gt;o&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;are in one equivalence class. According to that source, the models  represents exactly the same joint probability distribution.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-04-20T09:30:21.023" LastActivityDate="2016-04-20T09:30:21.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11299" PostTypeId="2" ParentId="11278" CreationDate="2016-04-19T19:00:56.657" Score="1" Body="&lt;p&gt;Logistic Regression is great for CTR and spam filtering (text data in general) thanks to the use of the hashing trick. Vowpal Wabbit has an optimized implementation of it in case you have GB of data. I guess neural networks will perform well too in these kind of tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit&quot; rel=&quot;nofollow&quot;&gt;https://github.com/JohnLangford/vowpal_wabbit&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One recommendation that applies to every other data science problem; the biggest improvements you can get come from creating new features. In your case: TF-IDF, n-grams, POS Tagging can be used to improve the classifier more than just choosing A or B algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Vowpal Wabbit can create n-grams and interactions automatically. Other types of pre-processing should be coded separately.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2016-04-19T19:00:56.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11301" PostTypeId="1" CreationDate="2016-04-19T21:42:41.477" Score="1" ViewCount="79" Body="&lt;p&gt;I'm wondering if there is literature or studies done on how to model organic attribution from paid user acquisition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the context is, on our mobile app, we have paid installs that we purchase and organic installs, but we believe organic installs mostly come from either featuring or referrals from current users. Thus, for every paid user we buy, we actually can attribute some organic installs to that paid user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem then is what this coefficient is. My first instinct is to use a granger like model regressing on the previous values of organic and paid installs with a dummy variable for featuring like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_t=\beta_0+\beta_1 y_{t-1}+ ... + \beta_q y_{t-q} + \gamma_1 x_{t-1} + ... + \gamma_q x_{t-q} +\delta_0 z_t+\varepsilon_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_t$ is organic installs, $x_t$ is paid installs, and $z_t$ is just a dummy variable indicating whether the game was featured on date $t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this model valid and what would be the best way to determine the order for this model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, would love to hear everyone's thoughts or suggestions if there's a well-defined model for this! &lt;/p&gt;&#xA;" OwnerUserId="15768" LastEditorUserId="15768" LastEditDate="2016-04-20T19:45:47.057" LastActivityDate="2016-04-20T19:45:47.057" Title="Organic vs Paid Attribution Model" Tags="|predictive-modeling|time-series|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11302" PostTypeId="2" ParentId="10064" CreationDate="2016-04-19T23:23:32.403" Score="8" Body="&lt;p&gt;TL;DR: sometimes you can make an equivalent Bayesian network by reversing arrows, and sometimes you can't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply reversing the direction of the arrows yields another directed graph, but that graph is not necessarily the graph of an equivalent Bayesian network, because the dependence relations represented by the reversed-arrow graph can be different from those represented by the original graph. If the reversed-arrow graph represents different dependence relations than the original, in some cases it's possible to create an equivalent Bayesian network by adding some more arrows to capture dependence relations which are missing in the reversed-arrow graph. But in some cases there is not an exactly equivalent Bayesian network. If you have to add some arrows in order to capture dependencies, you might end up with a graph which represents fewer independence relations and therefore fewer opportunities for simplifying the computations of posterior probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;code&gt;a -&amp;gt; b -&amp;gt; c&lt;/code&gt; represents the same dependencies and independencies as &lt;code&gt;a &amp;lt;- b &amp;lt;- c&lt;/code&gt;, and the same as &lt;code&gt;a &amp;lt;- b -&amp;gt; c&lt;/code&gt;, but not the same as &lt;code&gt;a -&amp;gt; b &amp;lt;- c&lt;/code&gt;. This last graph says that &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; are independent if &lt;code&gt;b&lt;/code&gt; is not observed, but &lt;code&gt;a &amp;lt;- b -&amp;gt; c&lt;/code&gt; says &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; are dependent in that case. We can add an edge directly from &lt;code&gt;a&lt;/code&gt; to &lt;code&gt;c&lt;/code&gt; to capture that, but then &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; being independent when &lt;code&gt;b&lt;/code&gt; is observed is not represented. That means that there is at least one factorization which we cannot exploit when computing posterior probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this stuff about dependence/independence, arrows and their reversals, etc., is covered in standard texts on Bayesian networks. I can dig out some references if you want. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian networks don't express causality. Judea Pearl, who did a lot of work on Bayesian networks, has also worked on what he calls causal networks (essentially Bayesian networks annotated with causal relations).&lt;/p&gt;&#xA;" OwnerUserId="18001" LastActivityDate="2016-04-19T23:23:32.403" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11304" PostTypeId="1" AcceptedAnswerId="11306" CreationDate="2016-04-20T00:41:59.873" Score="0" ViewCount="1306" Body="&lt;p&gt;In my clustering project, I need to customize the linkage function, so that after each cluster merging I can update the inter-cluster distance in my own way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm using scikit-learn AgglomerativeClustering, which seems not having this customizable feature. After a quick glance in scipy, no luck there either. Does anyone know any python hierarchical clustering toolkit that has customizable linkage?&lt;/p&gt;&#xA;" OwnerUserId="18011" LastEditorUserId="16284" LastEditDate="2016-04-20T11:12:31.030" LastActivityDate="2016-04-20T11:12:31.030" Title="Hierarchical Clustering customized Linkage function" Tags="|machine-learning|python|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11306" PostTypeId="2" ParentId="11304" CreationDate="2016-04-20T02:16:32.867" Score="0" Body="&lt;p&gt;Fork sklearn and implement it yourself! The linkage function is referenced in &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cluster/hierarchical.py&quot; rel=&quot;nofollow&quot;&gt;cluster/hierarchical.py&lt;/a&gt; as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cluster/hierarchical.py#L396&quot; rel=&quot;nofollow&quot;&gt;join_func = linkage_choices[linkage]&lt;/a&gt; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cluster/hierarchical.py#L520&quot; rel=&quot;nofollow&quot;&gt;coord_col = join_func(A[i], A[j], used_node, n_i, n_j)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you have time, &lt;a href=&quot;http://scikit-learn.org/stable/developers/contributing.html#contributing-code&quot; rel=&quot;nofollow&quot;&gt;polish your code&lt;/a&gt; and submit a pull request when you're done.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-04-20T02:16:32.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11308" PostTypeId="1" CreationDate="2016-04-20T03:12:14.353" Score="3" ViewCount="549" Body="&lt;p&gt;I'm attempting to build a model/suite of models to predict a binary target. The exact details of the models aren't important, but suffice to say that I've tried with half a dozen different types of models, with comparable results from all of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On looking at the predictions on various subsets of the training data, it appears that a certain subset of features is important for around 30% of the data, while a different subset is important for the remaining 70%. With the training data/holdout set. This separation is fairly easy to detect when the target is known (run a model with subset1, another with subset2, find the subset where one model does better than the other). Obviously, this is not possible with the test data, since the target is not known there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are clearly (at least) two regions in the data that are substantially different from each other, since a model trained on the whole dataset does worse than the combination of separate models trained on each of these regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the dataset seems spectacularly resistant to separation, with no major difference between principal components (linear and RBF kernel) and no clear/stable clusters after applying several different clustering algorithms (KMeans, agglomerative, mean shift). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any popular methods that allow separation/clustering of data that are resistant to &quot;normal&quot; methods? The end goal here is to find out which model to use on which rows when target is unknown.&lt;/p&gt;&#xA;" OwnerUserId="18012" LastEditorUserId="13727" LastEditDate="2016-12-21T21:10:45.500" LastActivityDate="2016-12-22T08:21:32.603" Title="Dealing with an apparently inseparable dataset" Tags="|machine-learning|domain-adaptation|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11309" PostTypeId="1" CreationDate="2016-04-20T05:18:21.833" Score="0" ViewCount="86" Body="&lt;p&gt;What kind of feature engineering techniques should one apply for longitudinal data comprising of individual binary time interval data about when an activity was done during the day(we have this data for every person in our database) with each interval being represented by the starting and end time ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought about pre classifying people into number of logical groups and then for each group dividing the entire day into slots of fixed interval and then in each slot finding:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;number of people in that sub group that started that activity in that slot&lt;/li&gt;&#xA;&lt;li&gt;number of people in that sub group that ended that activity in that slot&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have also thought about finding average number of time that each group spends in a day performing that activity, and repeat the above process for all days&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Do we have some existing research that address this kind of data?&lt;/p&gt;&#xA;" OwnerUserId="18014" LastEditorUserId="18014" LastEditDate="2016-04-20T05:23:51.060" LastActivityDate="2016-04-20T05:23:51.060" Title="binary longitudinal time series" Tags="|time-series|feature-engineering|binary|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11311" PostTypeId="2" ParentId="11304" CreationDate="2016-04-20T05:49:14.543" Score="0" Body="&lt;p&gt;It's not Python, but ELKI allows customizing linkages easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used this tutorial:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://elki.dbs.ifi.lmu.de/wiki/Tutorial/HierarchicalClustering#Addingadditionallinkagestrategies&quot; rel=&quot;nofollow&quot;&gt;http://elki.dbs.ifi.lmu.de/wiki/Tutorial/HierarchicalClustering#Addingadditionallinkagestrategies&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Lance-Williams-Uodate approach is very efficient. But I don't think you can easily add Minimax Linkage the same way. It may be much more expensive to do minimax linkage?&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-04-20T05:49:14.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11312" PostTypeId="2" ParentId="11218" CreationDate="2016-04-20T06:27:04.330" Score="0" Body="&lt;p&gt;I'm not sure if this is the type of analysis you are after, but you mention that the visual side is restricted in STATA. A colleague wrote a &lt;a href=&quot;http://www.consolidata.co.uk/technical/analysing-website-data-with-graph-databases/&quot; rel=&quot;nofollow&quot;&gt;blog&lt;/a&gt; that utilised neo4j to read web data into a graph database, and d3js to display the data graphically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise you don't have web data as such, but your data can be stored in a &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_database&quot; rel=&quot;nofollow&quot;&gt;graph database&lt;/a&gt;, but I guess when I was asking about what types of analysis you were planning on doing, I was asking were you needing a qualitative or quantitative direction. But it seems like you are still in the process of working that out. The nice thing with neo4j is that you can pull the data into R and do any sort of analytics you want on it.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-20T06:27:04.330" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11314" PostTypeId="1" AcceptedAnswerId="11376" CreationDate="2016-04-20T08:06:09.363" Score="5" ViewCount="3953" Body="&lt;p&gt;I have a question related to change detection. Application domain is robotics/planning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background/setting:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a sensor detecting distance from obstacle (ultrasonic / sonar sensor) at a specific position &lt;code&gt;(x, y, theta)&lt;/code&gt; in the environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It returns some reading at regular time intervals. Lets say the reading is &lt;code&gt;R&lt;/code&gt; and over a period of time it records &lt;code&gt;R+&lt;/code&gt; or &lt;code&gt;R-&lt;/code&gt; (&lt;code&gt;+/-&lt;/code&gt; means variation due to sensor inaccuracies).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 1:&#xA;I introduce an additional object between the sensor and the obstacle at a distance &lt;code&gt;D&lt;/code&gt; (&lt;code&gt;D &amp;lt; R&lt;/code&gt;) so that at the next instance &lt;code&gt;D&lt;/code&gt; is detected and returned&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 2: &#xA;I remove the original obstacle and now the next obstacle is &lt;code&gt;D'&lt;/code&gt; (&lt;code&gt;D' &amp;gt; R&lt;/code&gt;) and at the next instance &lt;code&gt;D'&lt;/code&gt; is returned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to exactly (or with high probability) say that a changed occurred NOW (when I add or remove an obstacle)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most change analysis algorithms consider a run length before change point and some data after change point and indicate the position change occurred. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But none I have read so far say change happened NOW; even the &quot;online&quot; algorithms seem to need some burn in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ultimate goal&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to implement a method that takes the data vector and return if the latest data point was a change point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A possible Solution/hack&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my work involves streaming data, this is the approach I am currently taking. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Read a window of data (for now, my window size is 20 values) from the end of the stream.&lt;/li&gt;&#xA;&lt;li&gt;Run bcp (from R) on this window. &lt;/li&gt;&#xA;&lt;li&gt;Check for the posterior probability of the change at location 18. (for all the runs i just had, the last value is NA, hence ignore that, and the data is zero indexed, (calling R from Python using rpy2), hence, the position turns out 18 for window size of 20.&lt;/li&gt;&#xA;&lt;li&gt;Set a threshold of 70% for the posterior probability (for now in my experimental setting this works fine, I may have to work on getting a proper threshold later)&lt;/li&gt;&#xA;&lt;li&gt;If the posterior probability at location 18 &gt; 70%, I return TRUE indicating the recent data point has a different mean, or &quot;change detected&quot;, else return FALSE.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This may not be the most efficient way of doing it, but it is doing its job for now. I am using this approach to carry my work forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will update the thread if I find a better approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks you all for the help!&lt;/p&gt;&#xA;" OwnerUserId="18019" LastEditorUserId="18019" LastEditDate="2016-05-02T10:11:20.140" LastActivityDate="2017-11-20T15:49:22.413" Title="change detection" Tags="|python|statistics|anomaly-detection|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11315" PostTypeId="1" CreationDate="2016-04-20T08:43:00.263" Score="2" ViewCount="96" Body="&lt;p&gt;I have traffic jam intensity of many location in different times of eabch day. So I have kind of 3D data, where the intensities in each location is 2D like an image. Then there is time as a third dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to predict traffic jam intensity at a give location and time.&#xA;The intensity is from 0.0 to 10.0. high the intensity means high traffic jam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example. Whats the jam intensity of X latitude and Y longitude on 3:45 PM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to feed that 3D data to my neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: both the time and location contextual data. The intensity of the traffic jam is behavioral. &lt;/p&gt;&#xA;" OwnerUserId="11141" LastEditorUserId="11141" LastEditDate="2016-04-20T10:00:03.370" LastActivityDate="2023-12-10T07:08:46.150" Title="How to model spatiotemporal data with ANN?" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11316" PostTypeId="1" AcceptedAnswerId="11323" CreationDate="2016-04-20T09:00:57.287" Score="0" ViewCount="74" Body="&lt;p&gt;I have to perform predictive model over the dataset $D$ (with 1000 obs).&#xA;From $D$, I extract 700 obs for training $(T)$ and 300 obs for validation $( V )$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to perform bootstrap or 10-fold cross validation sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is which of these sets should I use?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Divide $D$ in 10 subsets and alternate training and validation between them ? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Divide $T$ (the training subset) in 10 subsets and perform training/validation on those subsets? $V$ is used only for final validation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18020" LastEditorUserId="11097" LastEditDate="2016-04-20T10:50:10.650" LastActivityDate="2016-04-20T18:03:14.330" Title="Which observation to use when doing k-fold validation or boostrap?" Tags="|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11320" PostTypeId="1" CreationDate="2016-04-20T12:54:12.763" Score="8" ViewCount="11269" Body="&lt;p&gt;Given an original probability distribution &lt;em&gt;P&lt;/em&gt;, I want to measure how much an approximation &lt;em&gt;Q&lt;/em&gt; differs from the initial distribution. For that I calculate the KL-divergence via &lt;code&gt;scipy.stats.entropy&lt;/code&gt;, which returns infinity due to the large difference. However, as with time the approximation becomes better, I still want to quantify the divergence between the two sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is, is there any hack to avoid &lt;code&gt;inf&lt;/code&gt; values or should I circumvent the behaviour by using some other distance measure?&lt;/p&gt;&#xA;" OwnerDisplayName="user17988" LastActivityDate="2017-09-25T09:09:15.687" Title="KL-divergence returns infinity" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11321" PostTypeId="2" ParentId="11320" CreationDate="2016-04-20T15:30:30.217" Score="7" Body="&lt;p&gt;The infinity is due to divide by zero. &#xA;Just replace any zero value with a very small value.&#xA;This question is very common , if you search for it you will find many questions similar to it . &#xA;Remember that kl is not a metric &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-20T15:30:30.217" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11322" PostTypeId="1" AcceptedAnswerId="12503" CreationDate="2016-04-20T16:02:20.687" Score="3" ViewCount="272" Body="&lt;p&gt;I need to rank human skills depending on their similarity to the input skill. So if I enter &quot;Dutch language&quot;, I want the list like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.97 Dutch&#xA;0.86 Dutch lessons&#xA;0.55 Frisian&#xA;0.50 Flemish&#xA;0.27 German language&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a database of around 4500 human skills (ranging from &quot;programming in C&quot; to &quot;baking almond cake&quot;) with 600 manually categorized. I can already find corresponding article on BabelNet and pull domain, categories and related terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example skill with the data from BabelNet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name:&quot;photography&quot;&#xA;categories:&#xA;  0:&quot;Photography&quot;&#xA;  1:&quot;French_inventions&quot;&#xA;  2:&quot;Optics&quot;&#xA;  3:&quot;1822_introductions&quot;&#xA;manualCategory:&quot;art &amp;amp; music&quot;&#xA;domains:&#xA;   ART_ARCHITECTURE_AND_ARCHAEOLOGY:1&#xA;compounds:&#xA;  0:&quot;digital_photography&quot;&#xA;  1:&quot;landscape_photography&quot;&#xA;  2:&quot;photographic_developing&quot;&#xA;  3:&quot;motion_photography&quot;&#xA;  4:&quot;nature_photography&quot;&#xA;  ...&#xA;  48:&quot;photographic_plates&quot;&#xA;otherForms:&#xA;  0:&quot;still_photography&quot;&#xA;  1:&quot;photo&quot;&#xA;  2:&quot;photos&quot;&#xA;  3:&quot;photographed&quot;&#xA;  4:&quot;photographers&quot;&#xA;  ...&#xA;  20:&quot;Photographer&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can you suggest me the approach or at least steer in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="18028" LastEditorUserId="18028" LastEditDate="2016-04-21T10:42:38.767" LastActivityDate="2016-06-30T02:19:23.140" Title="Ranking skills depending on similarity" Tags="|machine-learning|classification|nlp|text-mining|similarity|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11323" PostTypeId="2" ParentId="11316" CreationDate="2016-04-20T16:52:46.047" Score="1" Body="&lt;p&gt;I recommend using the second option you presented.  I would use $T$ with 10-fold CV to select my modeling technique and optimal tuning parameters.  Take a look at what performed the best (&quot;best&quot; being the model that gives us the best error, but also doesn't have the error fluctuate too much from fold to fold).  After selecting a model, you can use the model on $V$ to get a realistic error rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I don't recommend the first option is: There are varying degrees of over fitting that can occur when going through model selection and model tuning, then using that same data to get an error rate.  CV is a great way to limit this overfitting and it gives us a sense of performance variance which is great, but a classic hold-out validation set is the gold standard for model performance.  In your case the first option might not be wrong (depends a lot on data/techniques), but if a hold-out validation set is available I would go for that.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-04-20T16:52:46.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11324" PostTypeId="2" ParentId="11316" CreationDate="2016-04-20T17:14:09.400" Score="0" Body="&lt;p&gt;I would prefer doing cross validation on this dataset because it removes over fitting of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to second question it's better to do a 10-fold cross validation on your training data set(T) by diving T into 10 subsets and tuning your model accordingly. Once you are happy with model like when your preferred accuracy is reached, then test it on validation dataset.&#xA;Correct me if i am wrong.&lt;/p&gt;&#xA;" OwnerUserId="13517" LastEditorUserId="16284" LastEditDate="2016-04-20T18:03:14.330" LastActivityDate="2016-04-20T18:03:14.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11325" PostTypeId="1" CreationDate="2016-04-20T17:42:09.053" Score="4" ViewCount="700" Body="&lt;p&gt;Using orange, I would like to be able to do real time analysis or visualizations on streaming data. I would appreciate any input on the matter! &lt;/p&gt;&#xA;" OwnerUserId="18033" LastEditorUserId="16284" LastEditDate="2016-04-21T13:54:48.587" LastActivityDate="2017-03-02T17:09:57.430" Title="In Orange, is it possible to analyze or visualize real time, streaming data?" Tags="|time-series|visualization|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11327" PostTypeId="1" AcceptedAnswerId="11339" CreationDate="2016-04-20T18:04:38.367" Score="2" ViewCount="564" Body="&lt;p&gt;I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for choosing max_features, min_samples, max_depth. Trees and learning rate come later. Will I get different results tuning the fraction versus the whole data set?&lt;/p&gt;&#xA;" OwnerUserId="18035" LastActivityDate="2016-04-21T13:11:05.140" Title="Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?" Tags="|scikit-learn|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11329" PostTypeId="1" CreationDate="2016-04-20T18:14:21.247" Score="3" ViewCount="600" Body="&lt;p&gt;Q-learning works with a 2D SxA table of Q values, where S is the current state and A is the action taken. Is there some model-based variant of Q-learning (or SARSA) that uses a 3D SxAxS' table to store Q values, where S' is the resulting state?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar table may be used to learn models in discrete environments, but in this case you don't store Q values, but counts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find anything like this, either because I don't know the correct terms to search for, or because it does not exist. It doesn't matter to me if it is useful or not, I just need to know if it exists, and if it exists, how is it called.&lt;/p&gt;&#xA;" OwnerUserId="9814" LastEditorUserId="9814" LastEditDate="2016-04-20T19:07:53.830" LastActivityDate="2016-05-12T16:42:47.513" Title="Is there some model-based variation of the Q-Learning algorithm which learns on a 3D SxAxS' table instead of a 2D SxA table?" Tags="|reinforcement-learning|q-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11330" PostTypeId="1" CreationDate="2016-04-20T20:55:10.390" Score="2" ViewCount="152" Body="&lt;p&gt;I'm using fitctree function from Matlab to fit a CART tree to my dataset (10'000 data points, 20 features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, how can I exactly prune it and to what level should the tree be pruned? I think I have to use the prune method but I'm confused by the 'MergeLeaves' and 'Prune' flag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, what values of 'MinLeafSize', 'MinParentSize' and 'MaxNumSplits' is reasonable to try?&lt;/p&gt;&#xA;" OwnerUserId="18040" LastActivityDate="2016-04-20T20:55:10.390" Title="Parameters for CART tree" Tags="|machine-learning|decision-trees|matlab|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11331" PostTypeId="1" AcceptedAnswerId="11333" CreationDate="2016-04-20T21:05:09.387" Score="3" ViewCount="132" Body="&lt;p&gt;I have a spreadsheet with thousands of records regarding support requests.&#xA;Case number, Issue description, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our goal is to classify these records in many categories in order to assign them the right priority.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Customer can't use pickup feature. &lt;/li&gt;&#xA;&lt;li&gt;Customer can't dial 911 or Long Distance numbers.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For item number 1, I have decided to use a category called Best Effort and for item 2, an Urgent category.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Customer can't use pickup feature, &lt;strong&gt;BEST_EFFORT&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Customer can't dial 911 or Long Distance numbers, &lt;strong&gt;URGENT&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm planning to setup a dictionary of words.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;best_effort = ['pickup','record','conference']&#xA;urgent = ['system is down','911', 'can't dial emergency','call center is down']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My goal is to use &lt;a href=&quot;https://janav.wordpress.com/2013/10/27/tf-idf-and-cosine-similarity/&quot; rel=&quot;nofollow&quot;&gt;TFIDF&lt;/a&gt; and then cosine similarity to find best match and category. Does it makes sense? Any better recommendation to classify this type of information?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="14591" LastEditDate="2016-05-09T08:28:01.657" LastActivityDate="2016-05-09T08:28:01.657" Title="How to classify support call texts?" Tags="|classification|dataset|nlp|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11332" PostTypeId="1" CreationDate="2016-04-20T22:27:30.793" Score="4" ViewCount="259" Body="&lt;p&gt;I sometimes want to benchmark Theano, either to compare different versions of Theano, or to compare two different computing environment with the same version of Theano. Is there any Theano code I could use to benchmark the main functions of Theano? I plan to use Theano only for artificial neural networks (RNN and CNN).&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-11-01T14:56:13.157" Title="Benchmarking Theano" Tags="|deep-learning|performance|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11333" PostTypeId="2" ParentId="11331" CreationDate="2016-04-20T23:24:38.503" Score="3" Body="&lt;p&gt;Rather than use an external dictionary of keywords that are indicative of target class, you may want to take your raw data (or rather a random subset of it), then hand-label your instances (assign each row a label, BEST_EFFORT or URGENT). This becomes your &lt;a href=&quot;https://datascience.stackexchange.com/questions/10531/create-training-data/10532#10532&quot;&gt;training data&lt;/a&gt; - each row of data can be transformed into a bag-of-words vector indicating the presence/absence of the word in that particular text. You can train a classifier on this data, for example a naive bayes classifier, which can then be tested on the held out unseen test data. The advantages of the  proposed approach are: (1) automated computation of features vs. hand created dictionary; (2) probabilistic/weighted indicators of class vs. binary dictionary indicators.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-04-20T23:24:38.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11334" PostTypeId="1" CreationDate="2016-04-21T05:04:57.383" Score="4" ViewCount="13264" Body="&lt;p&gt;I have a test dataset and train dataset as below. I have provided sample data with min records, but my data has more than 1000's of record. Here if you see &lt;code&gt;E&lt;/code&gt; is my target variable which I need to predict using an algorithm. It has only four categories like 1, 2, 3, 4. It can take only any of these values.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Training Dataset:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A    B    C    D    E&#xA;1    20   30   1    1&#xA;2    22   12   33   2&#xA;3    45   65   77   3&#xA;12   43   55   65   4&#xA;11   25   30   1    1&#xA;22   23   19   31   2&#xA;31   41   11   70   3&#xA;1    48   23   60   4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Dataset:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A    B    C    D    E&#xA;11   21   12   11&#xA;1    2    3    4&#xA;5    6    7    8 &#xA;99   87   65   34 &#xA;11   21   24   12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since &lt;code&gt;E&lt;/code&gt; has only 4 categories, I thought of predicting this using Multinomial Logistic Regression (1 vs Rest Logic). I am trying to implement it using Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the logic that we need to set these targets in a variable and use an algorithm to predict any of these values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output = [1,2,3,4]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I am stuck at a point on how to use it using python (sklearn) to loop through these values and what algorithm should I use to predict the output values? Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="18051" LastEditorUserId="29575" LastEditDate="2018-10-27T19:25:26.213" LastActivityDate="2018-10-27T19:25:26.213" Title="Python : How to use Multinomial Logistic Regression using SKlearn" Tags="|python|data-mining|logistic-regression|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11335" PostTypeId="2" ParentId="11329" CreationDate="2016-04-21T06:33:19.260" Score="0" Body="&lt;p&gt;You can take a look at fitted Q iteration (&lt;a href=&quot;https://papers.nips.cc/paper/3233-fitted-q-iteration-in-continuous-action-space-mdps.pdf&quot; rel=&quot;nofollow&quot;&gt;here is the pdf&lt;/a&gt;) which is a sort of model based Q learning. I am not sure if this is exactly what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="18019" LastActivityDate="2016-04-21T06:33:19.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11336" PostTypeId="2" ParentId="11334" CreationDate="2016-04-21T08:33:48.273" Score="4" Body="&lt;p&gt;Put the training data into two numpy arrays:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;# data from columns A - D&#xA;Xtrain = np.array([[1,    20,   30,   1],&#xA;                   [2,    22,   12,   33],&#xA;                   [3,    45,   65,   77],&#xA;                   [12,   43,   55,   65],&#xA;                   [11,   25,   30,   1],&#xA;                   [22,   23,   19,   31],&#xA;                   [31,   41,   11,   70],&#xA;                   [1,    48,   23,   60]])&#xA;&#xA;# data from column E&#xA;ytrain = np.array([1, 2, 3, 4, 1, 2, 3, 4])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then train a logistic regression model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LogisticRegression&#xA;&#xA;lr = LogisticRegression().fit(Xtrain, ytrain)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Make predictions (on the training data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;yhat = lr.predict(Xtrain)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;=&gt; results in &quot;1, 4, 3, 4, 1, 2, 3, 4&quot;.. so it's got 7 right and 1 wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate accuracy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import accuracy_score&#xA;&#xA;accuracy_score(ytrain, yhat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;=&gt; results in 87.5% accuracy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make predictions for new data, just create another numpy array containing your test data and call &lt;code&gt;lr.predict&lt;/code&gt; on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to look into parameter tuning to improve your score. For example the LogisticRegression class has some parameters that control regularization - tuning them with methods found in sklearn.grid_search might improve your score.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="47230" LastEditDate="2018-03-05T02:58:15.257" LastActivityDate="2018-03-05T02:58:15.257" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11337" PostTypeId="1" CreationDate="2016-04-21T09:47:36.737" Score="3" ViewCount="515" Body="&lt;p&gt;I'm trying to use &lt;strong&gt;rvest&lt;/strong&gt; to screen scrape headline news items from google and failing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having previously written a utility to screen scrape high level stats from DS.SE (not user info I have to say!), which runs successfully, I know that my technique works, but this produces nothing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using &lt;a href=&quot;http://selectorgadget.com/&quot; rel=&quot;nofollow&quot;&gt;SelectorGadget&lt;/a&gt; to find the screen element needed (&quot;.r a&quot;) and the code snippet below should read the headlines from the google page for &quot;education will&quot; search term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code (reduced down to bare bones) is:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library (rvest)&#xA;&#xA;url = &quot;https://www.google.co.uk/#q=%22education+value%22&quot;&#xA;url&#xA;&#xA;index_html &amp;lt;- read_html( url )&#xA;&#xA;headline_tmp &amp;lt;- index_html %&amp;gt;%&#xA;  html_nodes (&quot;.r a&quot;) %&amp;gt;%&#xA;  html_text()&#xA;headline_tmp &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Giving the output as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library (rvest)&#xA;&amp;gt; &#xA;&amp;gt; url = &quot;https://www.google.co.uk/#q=%22education+value%22&quot;&#xA;&amp;gt; url&#xA;[1] &quot;https://www.google.co.uk/#q=%22education+value%22&quot;&#xA;&amp;gt; &#xA;&amp;gt; index_html &amp;lt;- read_html( url )&#xA;&amp;gt; &#xA;&amp;gt; headline_tmp &amp;lt;- index_html %&amp;gt;%&#xA;+   html_nodes (&quot;.r a&quot;) %&amp;gt;%&#xA;+   html_text()&#xA;&amp;gt; headline_tmp&#xA;&amp;gt;character(0) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run with the stackexchange URL, suitably modified code for the different URL, would give a vector of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reading I've done about google stopping screen scrapers, suggests that they only stop people who abuse it and usually will do it with a Captcha or similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on a solution?&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-04-21T09:47:36.737" Title="Problem Screen Scraping Google Data" Tags="|r|scraping|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11338" PostTypeId="1" CreationDate="2016-04-21T12:19:21.837" Score="1" ViewCount="315" Body="&lt;p&gt;I have a dataset of 256 rows with 61 columns/variables. Each row should be considered a vector of dimension 61. If I randomly split it, by rows, in 2 groups, how could I prove that the 2 groups are similar? The origin of the data is biomedical and nonlinear approaches should be preferable.&lt;/p&gt;&#xA;" OwnerUserId="17933" LastActivityDate="2016-10-19T01:25:25.140" Title="Group similarity by high dimension vector comparison" Tags="|similarity|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11339" PostTypeId="2" ParentId="11327" CreationDate="2016-04-21T13:11:05.140" Score="2" Body="&lt;p&gt;You should never train or do grid search on your entire data set, since it will lead to overfitting and reduce the accuracy of your model on new data. What you have described is actually the ideal approach: do grid search / training on a subset of your data. Yes, your model will get different results vs if you used the entire set of data, but your model will be much stronger because of it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on why you would want to split up / sample your data, see this quesiton: &lt;a href=&quot;https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set&quot;&gt;https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-21T13:11:05.140" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11340" PostTypeId="1" CreationDate="2016-04-21T13:29:03.303" Score="3" ViewCount="247" Body="&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt; &#xA;I am working on a classification project. where I recommend items to customers based on their past purchase history.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&#xA;How will &quot;time leakage&quot; affect training? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&#xA;Let's say that I am trying to predict today's purchases for some customer. I train only on the previous history, without any knowledge of what will be purchased today. My features consist of a set of binary variables that for any given day in history could be thought of as either recommending (1) or not recommending (0) an item (essentially acting as &quot;dumb&quot; classifiers themselves). For example, one of these features could be whether or not an item is in the customer's top 10 list, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, is it acceptable to compute top 10 over all of history and based on this list create a feature that is applied to each day in history? Of course, this will cause the features from the first day in history to have knowledge of the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or would having a sliding window where feature creation that can only look into the past be more appropriate? &lt;/p&gt;&#xA;" OwnerUserId="17497" LastEditorUserId="14675" LastEditDate="2017-07-11T11:19:30.427" LastActivityDate="2017-07-11T12:06:44.393" Title="Classification on time series data" Tags="|classification|time-series|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11341" PostTypeId="2" ParentId="11338" CreationDate="2016-04-21T13:53:02.340" Score="1" Body="&lt;p&gt;You can't actually &lt;em&gt;prove&lt;/em&gt; that the two groups are similar but you can establish a confidence level/threshold. Furthermore, it is possible that the two groups &lt;em&gt;won't&lt;/em&gt; be similar (depending on your threshold for similarity) if, for example, only one of the two groups contains strong outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, you can make comparisons based on assumptions regarding the underlying data distribution. For example, if the data can be assumed to be distributed as a multivariate normal distribution, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution#Hotelling.27s_two-sample_T-squared_statistic&quot; rel=&quot;nofollow&quot;&gt;Hotelling's two-sample T-squared statistic&lt;/a&gt; (a multivariate generalization of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Student%27s_t-test&quot; rel=&quot;nofollow&quot;&gt;Student's t-test&lt;/a&gt;) to test your confidence interval.&lt;/p&gt;&#xA;" OwnerUserId="964" LastActivityDate="2016-04-21T13:53:02.340" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11342" PostTypeId="5" CreationDate="2016-04-21T14:24:50.217" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-04-21T14:24:50.217" LastActivityDate="2016-04-21T14:24:50.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11343" PostTypeId="4" CreationDate="2016-04-21T14:24:50.217" Score="0" Body="pertaining to the geographic location and characteristics of natural or constructed features and boundaries on, above, or below the earth's surface; esp. referring to data that is geographic and spatial in nature." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-07T11:01:37.647" LastActivityDate="2019-04-07T11:01:37.647" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11344" PostTypeId="2" ParentId="11244" CreationDate="2016-04-21T15:25:08.680" Score="0" Body="&lt;p&gt;You are better off using the forecasts. In deployment your model will be relying on weather forecasts, which are obviously not 100% accurate. Therefore, if you train your model on the actual weather you may artificially inflate your accuracy. Using the forecasts will give you a better estimate of the accuracy of your model moving forward. &lt;/p&gt;&#xA;" OwnerUserId="18068" LastActivityDate="2016-04-21T15:25:08.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11345" PostTypeId="2" ParentId="11102" CreationDate="2016-04-21T17:04:52.843" Score="1" Body="&lt;p&gt;I removed the paint widget. Works fine witout it. That was in the version i had the problem. The latest daily build i installed after worked fine with the color widget. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17712" LastActivityDate="2016-04-21T17:04:52.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11346" PostTypeId="1" CreationDate="2016-04-21T18:06:56.097" Score="0" ViewCount="3977" Body="&lt;p&gt;I've two text files which contains my data.&#xA;One text file on X axis another text file on Y axis&#xA;Then using scatter function from python I did the data visualization&#xA;After that, I used polyfit function from python to get the curve which will fit my data&#xA;In that polyfit function we need to write degree of the polynomial we want eg. 2 or 3&#xA;Now let's consider I got 4 such a equations of degrees 2,3,4,5 respectively.&#xA;Now here comes my problem.&#xA;Among all those equations I got, how can I select the best equation which fits my data. I want to use cross validation here. Any high level library function can be use. My language is Python. &lt;/p&gt;&#xA;" OwnerUserId="18072" LastActivityDate="2016-04-21T23:20:18.197" Title="Use of cross validation for Polynomial Regression" Tags="|machine-learning|python|regression|scikit-learn|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11347" PostTypeId="1" AcceptedAnswerId="11373" CreationDate="2016-04-21T18:49:50.497" Score="11" ViewCount="92695" Body="&lt;p&gt;Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset of transaction data for a retail outlet. Variables along with explanation are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;section: the section of the store, a str;&lt;/li&gt;&#xA;&lt;li&gt;prod_name: name of the product, a str;&lt;/li&gt;&#xA;&lt;li&gt;receipt: the number of the invoice, an int;&lt;/li&gt;&#xA;&lt;li&gt;cashier, the number of the cashier, an int;&lt;/li&gt;&#xA;&lt;li&gt;cost: the cost of the item, a float;&lt;/li&gt;&#xA;&lt;li&gt;date, in format MM/DD/YY, a str;&lt;/li&gt;&#xA;&lt;li&gt;time, in format HH:MM:SS, a str;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Receipt has the same value for all the products purchased in a single transaction, thus it can be used to determine the average number of purchases made in a single transaction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to go about this? I essentially want to use &lt;code&gt;groupby()&lt;/code&gt; to group the receipt variable by its own identical occurrences so that I can create a histogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working with the data in a pandas DataFrame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some sample data with header (prod_name is actually a hex number):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; section,prod_name,receipt,cashier,cost,date,time &#xA; electronics,b46f23e7,102856,5,70.50,05/20/15,9:08:20 &#xA; womenswear,74558d0d,102857,8,20.00,05/20/15,9:12:46 &#xA; womenswear,031f36b7,102857,8,30.00,05/20/15,9:12:47 &#xA; menswear,1d52cd9d,102858,3,65.00,05/20/15,9:08:20 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this sample set I would expect a histogram of receipt that shows two occurrences of receipt 102857 (since that person bought two items in one transaction) and one occurrence respectively of receipt 102856 and of receipt 102858. Note: my dataset is not huge, about 1 million rows.&lt;/p&gt;&#xA;" OwnerUserId="18058" LastEditorUserId="381" LastEditDate="2016-04-22T17:32:34.170" LastActivityDate="2020-07-31T15:36:13.737" Title="How to group identical values and count their frequency in Python?" Tags="|python|statistics|pandas|ipython|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11348" PostTypeId="2" ParentId="11346" CreationDate="2016-04-21T20:09:11.080" Score="2" Body="&lt;p&gt;I think that you want this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html#sklearn.cross_validation.KFold&quot; rel=&quot;nofollow&quot;&gt;K-fold&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want say MSE of each check out section 3.1.1 here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html#computing-cross-validated-metrics&quot; rel=&quot;nofollow&quot;&gt;cross validated metrics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-04-21T20:09:11.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11349" PostTypeId="2" ParentId="11347" CreationDate="2016-04-21T20:40:29.173" Score="3" Body="&lt;p&gt;I'm putting together some tutorials around data wrangling. Maybe my &lt;a href=&quot;https://github.com/Rblivingstone/datawrangling/blob/master/groupby.ipynb&quot; rel=&quot;nofollow&quot;&gt;jupyter notebook&lt;/a&gt; on github will help. I think that it is the key is modifying the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('male')['age'].mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('reciept')['prod_name'].count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To group by multiple variables this should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby(['reciept','date'])['reciept'].count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16538" LastEditorUserId="16538" LastEditDate="2016-04-22T21:50:25.760" LastActivityDate="2016-04-22T21:50:25.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11350" PostTypeId="2" ParentId="11218" CreationDate="2016-04-21T21:37:26.443" Score="2" Body="&lt;p&gt;If you have the data in the form of a table of transition counts:&#xA;$$&#xA;Transition Period 1 Period 2 Period 3 Period 4 \newline&#xA;A-&amp;gt;A           	0             0	           0           8&#xA;A-&amp;gt;B            0             0            0           1&#xA;A-&amp;gt;C            4             1            0           1&#xA;B-&amp;gt;A            1             0            1           0&#xA;B-&amp;gt;B            2             0            0           0&#xA;B-&amp;gt;C            1             1            0           0&#xA;C-&amp;gt;A            0             0            9           0&#xA;C-&amp;gt;B            0             0            0           0&#xA;C-&amp;gt;C            2             7            0           0&#xA;$$&#xA;Then a possible visualization is an area plot. The following chart was produceds in Excel (use Charts/Area button on the Insert ribbon). This chart accurately captures all transitions that occurred in each period. Shaded areas of different colors represent the relative frequencies of transitions by origin-destination pair.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VZ92c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VZ92c.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18077" LastEditorUserId="18077" LastEditDate="2016-04-22T01:44:17.593" LastActivityDate="2016-04-22T01:44:17.593" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11351" PostTypeId="2" ParentId="11274" CreationDate="2016-04-21T22:18:44.773" Score="0" Body="&lt;p&gt;Even though it is possible, it probably does not produce a meaningful measure. I would advice however to be careful with these points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A probability distribution should represent probabilities. You can not take any arbitrary vector and consider them as probabilities distributions. The key here is to have a sound explanation for why outcome X has a say .2 (arbitrary) probability. The most simple way of &quot;calculating&quot; probabilities is considering it is the frequency of occurrences in an experiment comprised of N trials.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What your dimensions represent? To flatten a matrix only means you now have a vector. Having a vector might be enough for you function to run, but might not give you a meaningful measure.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;stats.entropy normalizes your vector if it does not sums up to 1. Often you will want to use the same kind of normalization on your pipeline. Be sure to normalize it before passing to entropy() if necessary. Here is the normalization performed by entropy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pk = 1.0*pk / np.sum(pk, axis=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16001" LastEditorUserId="16001" LastEditDate="2016-04-27T00:24:51.427" LastActivityDate="2016-04-27T00:24:51.427" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11352" PostTypeId="2" ParentId="11346" CreationDate="2016-04-21T23:20:18.197" Score="1" Body="&lt;p&gt;If instead of Numpy's polyfit function, you use one of &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_polynomial_interpolation.html&quot; rel=&quot;nofollow&quot;&gt;Scikit's generalized linear models with polynomial features&lt;/a&gt;, you can then apply  &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html#sklearn.grid_search.GridSearchCV&quot; rel=&quot;nofollow&quot;&gt;GridSearch with Cross Validation&lt;/a&gt; and pass in degrees as a parameter. It will find the best model based on the input features (i.e. 2,3,4,5). &lt;/p&gt;&#xA;" OwnerUserId="17497" LastActivityDate="2016-04-21T23:20:18.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11353" PostTypeId="2" ParentId="11278" CreationDate="2016-04-22T03:27:23.820" Score="3" Body="&lt;p&gt;Use logistic regression which allows the weights to be learnt. By using cosine similarity you are forcing the weights to be the same for all features (assuming that you normalize the features first). This is putting unnecessary restrictions on the model.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-04-22T03:27:23.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11354" PostTypeId="2" ParentId="11218" CreationDate="2016-04-22T03:30:53.077" Score="3" Body="&lt;p&gt;How about a &lt;a href=&quot;https://bost.ocks.org/mike/sankey/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sankey diagram&lt;/a&gt; with time on the x-axis and flow width representing state transition frequency. Here is a SO discussion on implementing &lt;a href=&quot;https://stackoverflow.com/questions/9968433/sankey-diagrams-in-r&quot;&gt;Sankey diagrams in R&lt;/a&gt;.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/PGUZn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PGUZn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible R package is &lt;a href=&quot;https://cran.r-project.org/web/packages/riverplot/riverplot.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;{riverplot}&lt;/a&gt;... here is code showing the first transition in your data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(riverplot)&#xA;nodes &amp;lt;- as.character(sapply(1:2, FUN = function(n){paste0(LETTERS[1:3],n)}))&#xA;edges &amp;lt;- list(A1=list(C2=4), B1=list(A2=1,C2=1,B2=2), C1=list(C2=2))&#xA;r &amp;lt;- makeRiver( nodes, edges, node_xpos= c( 1,1,1 ,2,2,2),&#xA;                node_labels= c( A1= &quot;A&quot;, B1= &quot;B&quot;, C1= &quot;C&quot;, A2=&quot;A&quot;,B2=&quot;B&quot;,C2=&quot;C&quot; ))&#xA;plot( r )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Will produce this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/KCCN2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KCCN2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-04-24T10:53:11.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11355" PostTypeId="2" ParentId="10842" CreationDate="2016-04-22T03:57:22.630" Score="0" Body="&lt;p&gt;Sounds like you are looking for the &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson correlation coefficient&lt;/a&gt; between the two variables. You can compute that by using Scipy's &lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.pearsonr.html&quot; rel=&quot;nofollow&quot;&gt;pearsonr method&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17497" LastActivityDate="2016-04-22T03:57:22.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11356" PostTypeId="1" AcceptedAnswerId="11361" CreationDate="2016-04-22T04:27:45.507" Score="35" ViewCount="170731" Body="&lt;p&gt;I have 10 data frames &lt;code&gt;pyspark.sql.dataframe.DataFrame&lt;/code&gt;, obtained from &lt;code&gt;randomSplit&lt;/code&gt; as &lt;code&gt;(td1, td2, td3, td4, td5, td6, td7, td8, td9, td10) = td.randomSplit([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1], seed = 100)&lt;/code&gt; Now I want to join 9 &lt;code&gt;td&lt;/code&gt;'s into a single data frame, how should I do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already tried with &lt;code&gt;unionAll&lt;/code&gt;, but this function accepts only two arguments. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;td1_2 = td1.unionAll(td2) &#xA;# this is working fine&#xA;&#xA;td1_2_3 = td1.unionAll(td2, td3) &#xA;# error TypeError: unionAll() takes exactly 2 arguments (3 given)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any way to combine more than two data frames row-wise? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of doing this is that I am doing 10-fold Cross Validation manually without using PySpark &lt;code&gt;CrossValidator&lt;/code&gt; method, So taking 9 into training and 1 into test data and then I will repeat it for other combinations. &lt;/p&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="21" LastEditDate="2016-05-10T13:56:37.837" LastActivityDate="2021-02-09T04:26:45.363" Title="Merging multiple data frames row-wise in PySpark" Tags="|python|apache-spark|cross-validation|pyspark|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11358" PostTypeId="2" ParentId="11292" CreationDate="2016-04-22T05:19:40.880" Score="1" Body="&lt;p&gt;Unfortunately scikit-learn does not handle categorical features well - they must be encoded.  One-hot-encoding is great and can be implemented in sklearn very easily.  So if you have an array of feature columns, X, and a class label vector, y&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import OneHotEncoder&#xA;ohe = OneHotEncoder(categorical_features=[0])&#xA;X = ohe.fit_transform(X).toarray()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will perform the encoding for you (The categorical_features attribute gives the index of the feature you want encoded).  More to your point though, if your feature has many levels, one-hot encoding can leave you with a sparse and inefficient X array.  In this case it may be beneficial to do a count transform, possibly replacing each level with its corresponding log-odds ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://msdn.microsoft.com/en-us/library/azure/dn913056.aspx&quot; rel=&quot;nofollow&quot;&gt;Count Transforms&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link above does a more thorough job explaining it, but, in layman's terms, for each categorical feature you: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;count the number of times a level(category) belongs to class0 or&#xA;class1 &lt;/li&gt;&#xA;&lt;li&gt;add 0.5 to each count (this takes care of instances when a level belongs strictly to one class)&lt;/li&gt;&#xA;&lt;li&gt;calculate the probability that it belongs to either class.&lt;/li&gt;&#xA;&lt;li&gt;calculate the log of the odds ratio&lt;/li&gt;&#xA;&lt;li&gt;replace each level with its corresponding log odds ratio&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In this manner you get numerical values representing each of the feature's many levels without creating a dummy variable for each level.  Still, 20 variables is a lot, and you may want to use PCA for data compression and dimension reduction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the great things about tree based methods in general is that they do not require standardization.  That being said, from a data science perspective, I would train several different models (tree based methods, svms, logistic regression, knn, etc.), and see which one yields the best results.  All methods besides CART benefit from scaling, so try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import StandardScaler&#xA;stdsc = StandardScaler()&#xA;X_train_std = stdsc.fit_transform(X_train)&#xA;X_test_std = stdsc.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope that answers most of what you're asking.  I'm not sure why you would want to concatenate two very different variables like cities and urls, but if you encode them or do a count transform your X array should be just fine and you can leave them separated.  Also, you shouldn't have to worry about the length of a given variable - python is object oriented (if you're getting error messages complaining about 'length', you've likely just confused your indices).&lt;/p&gt;&#xA;" OwnerUserId="16936" LastEditorUserId="16936" LastEditDate="2016-04-22T05:35:15.567" LastActivityDate="2016-04-22T05:35:15.567" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11361" PostTypeId="2" ParentId="11356" CreationDate="2016-04-22T08:25:37.830" Score="59" Body="&lt;p&gt;Stolen from: &lt;a href=&quot;https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds&quot;&gt;https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outside of chaining unions this is the only way to do it for DataFrames.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from functools import reduce  # For Python 3.x&#xA;from pyspark.sql import DataFrame&#xA;&#xA;def unionAll(*dfs):&#xA;    return reduce(DataFrame.unionAll, dfs)&#xA;&#xA;unionAll(td2, td3, td4, td5, td6, td7, td8, td9, td10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What happens is that it takes all the objects that you passed as parameters and reduces them using unionAll (this reduce is from Python, not the Spark reduce although they work similarly) which eventually reduces it to one DataFrame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If instead of DataFrames they are normal RDDs you can pass a list of them to the union function of your SparkContext&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: For your purpose I propose a different method, since you would have to repeat this whole union 10 times for your different folds for crossvalidation, I would add labels for which fold a row belongs to and just filter your DataFrame for every fold based on the label&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-04-22T08:52:20.653" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11363" PostTypeId="2" ParentId="6268" CreationDate="2016-04-22T10:06:40.423" Score="16" Body="&lt;p&gt;This can be done using &lt;code&gt;StringIndexer&lt;/code&gt; in PySpark and the reverse using &lt;code&gt;IndexToString&lt;/code&gt; for reference please check this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.ml.feature import StringIndexer&#xA;&#xA;df = sqlContext.createDataFrame(&#xA;    [(0, &quot;a&quot;), (1, &quot;b&quot;), (2, &quot;c&quot;), (3, &quot;a&quot;), (4, &quot;a&quot;), (5, &quot;c&quot;)],&#xA;    [&quot;id&quot;, &quot;category&quot;])&#xA;indexer = StringIndexer(inputCol=&quot;category&quot;, outputCol=&quot;categoryIndex&quot;)&#xA;indexed = indexer.fit(df).transform(df)&#xA;indexed.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more details, please check the &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#stringindexer&quot; rel=&quot;noreferrer&quot;&gt;spark documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-04-22T10:06:40.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11364" PostTypeId="1" CreationDate="2016-04-22T11:06:08.113" Score="3" ViewCount="3003" Body="&lt;p&gt;I want to calculate the Mahalanobis distance between cluster $a$ and cluster $b$, each consisting from a set of multidimensional points. Assuming no correlation, calculating the distance between a random point $p$ and cluster $b$ can be achieved using the normalized Euclidean distance formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ d(p,b)=\sqrt{ \sum_{i=1}^d \frac{(p_i-b_i)^2}{\sigma_i^2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $d$ is the number of dimensions and $\sigma_i^2$ is the squared standard deviation for each dimension in cluster $b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to estimate the Mahalanobis distance between clusters $a$ and $b$. Should I assume that cluster $a$ is a single point (i.e. $a$'s centroid)? Or should I normalize using the standard deviation of both clusters? In the latter case that would translate in the following formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ d(a,b)=\sqrt{ \sum_{i=1}^d \frac{(a_i-b_i)^2}{(\sigma_i^a)^2 \cdot (\sigma_i^b)^2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\sigma_i^b$ is the standard deviation for dimension $i$ in cluster $b$ and $\sigma_i^a$ is the standard deviation for dimension $i$ in cluster $a$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="18092" LastEditorUserId="18092" LastEditDate="2016-04-22T12:54:12.920" LastActivityDate="2016-07-23T19:24:40.363" Title="Mahalanobis distance between two clusters" Tags="|clustering|distance|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11367" PostTypeId="1" AcceptedAnswerId="11370" CreationDate="2016-04-22T13:32:17.873" Score="5" ViewCount="1089" Body="&lt;p&gt;I'm currently working on a part-time project which involves predicting the likelihood of customers going to buy a product using data analytics. The company I'm interning with has given me a customer CSV file with all current customers and their attributes and needs to make a prediction model to classify whether prospects are feasible to pursue or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However since they have given me a list of all their successful customers or leads, in marketing terms, is it possible to train a model like K-means with PCA (and k-fold cross validation?) and get results? I have to train my model to fit a value, say 10, which I will add to the CSV, and further test it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using pandas. Another issue is that there is a lot of demographical data, but I managed to overcome it using &lt;code&gt;get_dummies()&lt;/code&gt;. The number of columns escalated from about 10 to 47, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just entering into the world of data analysis, hence I'm a bit clueless as to what path to take or whether what I'm doing is right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exact analysis is called Predictive Lead Scoring/Analysis, in marketing terminology.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I followed what @HonzaB did and, hence did get a decision tree. However, since I had 40 columns, it looks like this &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A8Lck.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A8Lck.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had to take a screenshot of it, as it was over 2 MB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously it's really big, and I have to prune the tree somehow, but I not sure how to do so on pandas. Also, is there any way that I can just generate the best characteristics as a text file or something that can be understood without the help of a data scientist? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read up on a question that is quite similar to what I need to do. &lt;a href=&quot;https://datascience.stackexchange.com/questions/1119/predictive-modeling-based-on-rfm-scoring-indicators&quot;&gt;Predictive modeling based on RFM scoring indicators&lt;/a&gt;. In it there is a link to a paper([Data Mining using RFM Analysis][3]) that talks about &lt;strong&gt;rule-based classification&lt;/strong&gt;. Ideally this is what I need to do, and what is most suitable to the company's need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if it's possible to do this on Python/pandas. Or is it possible to traverse the decision tree and generate the rules? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 3&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found another website &lt;a href=&quot;http://chrisstrelioff.ws/sandbox/2015/06/25/decision_trees_in_python_again_cross_validation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision trees in python again, cross-validation&lt;/a&gt; that uses &lt;strong&gt;cross validation&lt;/strong&gt; and &lt;strong&gt;hyperparameter optimisation&lt;/strong&gt; to get a better solution. Also they have included Python code to get readable code. It's a feasible solution, however it's quite complicated and I can't understand how it works. Will it work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;PS I solved the &quot;really big decision-tree&quot; problem from Edit 1, by reducing max-depth. I didn't know at all.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="18098" LastEditorDisplayName="user380" LastEditDate="2017-05-11T07:09:56.610" LastActivityDate="2017-05-11T07:09:56.610" Title="Prediction model for marketing to prospective customers (using pandas)" Tags="|predictive-modeling|scikit-learn|pandas|marketing|scoring|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11368" PostTypeId="2" ParentId="11364" CreationDate="2016-04-22T14:45:53.517" Score="1" Body="&lt;p&gt;---------Please check the edits also for this answer---------------------&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to me, its very application specific, and depends on what you want to do. I will prefer second approach in a generic application because if 2 clusters between whom we are calculating distance are having high standard deviation, should have small distance. &#xA;Another approach I can think of is a combination of the 2. Calculate the Mahalanobis distance between 2 centroids and decrease it by the sum of standard deviation of both the clusters. I thought about this idea because, when we calculate the distance between 2 circles, we calculate the distance between nearest pair of points from different circles. Now think of circumference of the circle centered by centroid of circle. and the rest is obvious :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As pointed out by @MarcusD in the comment, I will try to explain a bit:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I remarked &quot;very application specific&quot; because in some cases where our answer should remain same irrespective of the standard deviation of the data, then first approach will work better. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For reference check. Kevin murphy- ML A probabilistic approach Pg. 104, 2 Class LDA. The second approach is exactly this one. If you are not having this book, google Linear discriminant analysis.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;The approach I gave is not theoretically different from LDA, but it is somewhat easier to implement in cases where the number of clusters is less. &lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18102" LastEditorUserId="18102" LastEditDate="2016-04-24T01:46:41.787" LastActivityDate="2016-04-24T01:46:41.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11369" PostTypeId="2" ParentId="11244" CreationDate="2016-04-22T15:21:38.733" Score="2" Body="&lt;p&gt;Using historical weather data implicitely means that you trust meteorologists and weather forecasters to improve their model over time and you let them the full responsability of it; however once your own model deployed a bias in the forecast may create a bias in your model response. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using weather forecasts instead should give better results because your model will directly capture potential bias in the forecasts; however if the weather forecasters update their forecasting model, and if you miss this update, your model response may suffer from it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't use both historical weather data and weather forecasts in the same model; I would consider to build two models, one with historical data and one with weather forecasts, then go for historical data if the improvement of using weather forecasts is not significant. &lt;/p&gt;&#xA;" OwnerUserId="3317" LastActivityDate="2016-04-22T15:21:38.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11370" PostTypeId="2" ParentId="11367" CreationDate="2016-04-22T15:39:05.890" Score="2" Body="&lt;p&gt;First, I would ask the company if there is more information about the customer. You mentioned you have 10 original columns, which might not be enough to make a good prediction. Same goes for number of rows. Usually, more data, better the model, up to a certain limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, encode categorical features (demographical data in your case) is good thing to do. The increased number of columns dont have to bother you in your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the task itself, yes, it is doable. Start easy, simply check importance of each feature (I would let PCA for later), pick few models and test them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also consider train simple decision tree. Your results can be easily visualized in way the business people understand. As oposed to black-box methods as K-Means.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2016-04-22T15:45:08.170" LastActivityDate="2016-04-22T15:45:08.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11372" PostTypeId="1" AcceptedAnswerId="11377" CreationDate="2016-04-22T17:03:49.950" Score="9" ViewCount="2204" Body="&lt;p&gt;I think I understand Tufte's concept of a &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Big_Duck&quot; rel=&quot;noreferrer&quot;&gt;Duck&lt;/a&gt;&quot; -- A graphic that is taken over by decorative forms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I couldn't understand why he called this a duck (a &quot;superbly produced&quot; one at that). It seemed to me more functional than decorative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fKnTl.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fKnTl.jpg&quot; alt=&quot;Tufte - &amp;quot;superbly produced duck&amp;quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="13727" LastEditDate="2016-04-24T13:09:53.277" LastActivityDate="2019-06-16T17:18:13.250" Title="Why did Tufte call this a &quot;superbly produced duck&quot;?" Tags="|visualization|terminology|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11373" PostTypeId="2" ParentId="11347" CreationDate="2016-04-22T17:39:35.320" Score="16" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;From this sample set I would expect a histogram of receipt that shows&#xA;  two occurrences of receipt 102857 (since that person bought two items&#xA;  in one transaction) and one occurrence respectively of receipt 102856&#xA;  and of receipt 102858.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then you want:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;df.groupby('receipt').receipt.count()&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;receipt&#xA;102856    1&#xA;102857    2&#xA;102858    1&#xA;Name: receipt, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-04-22T17:39:35.320" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11374" PostTypeId="1" CreationDate="2016-04-22T19:39:49.347" Score="1" ViewCount="328" Body="&lt;p&gt;I have a dataset for which I am trying to predict target variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Col1    Col2    Col3    Col4    Col5    &#xA;  1      2       23      11     1&#xA;  2      22      12      14     1&#xA;  22     11      43      38     3&#xA;  14     22      25      19     3&#xA;  12     42      11      14     1&#xA;  22     11      43      38     3&#xA;  1      2       23      11     4&#xA;  2      22      12      14     2&#xA;  22     11      43      38     3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have provided a sample data, but mine has thousands of records distributed in a similar way. Here, Col1, Col2, Col3, Col4 are my features and Col5 is target variable. Hence prediction should be 1,2,3 or 4 as these are my values for target variable. I have tried using algorithms such as random forest, decision tree etc. for predictions. &#xA;Here if you see, values 1 and 3 are occurring more times as compared to 2 and 4. Hence while predicting, my model is more biased towards 1 and 3 whereas I am getting only less number of predictions for 2 and 4 (Got only 1 predicted for policy4 out of thousands of records when I saw the confusion matrix). I am planning to do feature extractions and try to see how my model behaves, but is there any way to make my model predict in a generalized way where it can generalize some values 2 and 4? I just read of undersampling and oversampling concepts, but I am beginner here, is there any good approach I could try? Should I split the training dataset based on Col5? I am implementing this in python using pandas.&lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2016-04-22T19:39:49.347" Title="Python: How to make model predict in a generalized manner using ML Algorithm" Tags="|machine-learning|python|data-mining|dataset|pandas|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11375" PostTypeId="1" CreationDate="2016-04-22T20:13:34.877" Score="2" ViewCount="948" Body="&lt;p&gt;I am trying to build an SVD-based recommender system. According to my understanding, the training data should only contain the users who buy at least &lt;em&gt;m&lt;/em&gt; items and the items which are bought by &lt;em&gt;n&lt;/em&gt; unique users. Otherwise, if I use all users and items (including low-frequency users and items), I think the training data will be noisy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, here is the problem: To build the training data for SVD, I first retrieve all users who buy at least &lt;em&gt;m&lt;/em&gt; items from the database. But there are some items bought by these users are not bought by at least &lt;em&gt;n&lt;/em&gt; users. But after further filtering out these low-frequency items from the training data, some of the remaining users will not have item sets which contain &lt;em&gt;m&lt;/em&gt; items because some items are deleted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel that I am not on the right track. How am I supposed to build the training data for SVD?&lt;/p&gt;&#xA;" OwnerUserId="10464" LastActivityDate="2016-04-23T13:11:57.693" Title="How to prepare the training data for SVD-based Recommendation?" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11376" PostTypeId="2" ParentId="11314" CreationDate="2016-04-22T20:36:28.043" Score="4" Body="&lt;p&gt;Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of $X$ changes. Now your observations $X_k,\dots,X_n$ are different in some way. You want to know what $k$ is based on your observations alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to detect the change, you have to have some idea of what 'before' might look like so you can have confidence that 'after' is really different. So, yes, &lt;em&gt;all&lt;/em&gt; change detection algorithms will use some run length before and after the true change to make a decision (edit: actually, you don't need run length before and after, you could just have an assumption about the data-generating process. Maybe you say its normal mean 0 variance 1 and your first observation is 5000, you don't need run length to know you're wrong somewhere). Anything else would be an even wilder kind of predicting the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the real concern might be the latency of signal detection. You'd like the sensor to detect it after just a few instances of the data after the true change point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, do you really need it to work &lt;em&gt;now&lt;/em&gt;? It seems reasonable to me that you're not interested in the number of data points, but the time it takes to gather them. If you have a sensor that updates 100,000 times a second, 100 data points isn't a huge deal.&lt;/p&gt;&#xA;" OwnerUserId="18111" LastEditorUserId="18111" LastEditDate="2017-03-01T19:22:43.050" LastActivityDate="2017-03-01T19:22:43.050" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11377" PostTypeId="2" ParentId="11372" CreationDate="2016-04-22T21:36:43.083" Score="7" Body="&lt;p&gt;If you assume &lt;strong&gt;duck&lt;/strong&gt; to mean &quot;irrelevant decorative elements&quot; there are a few things that strike me as likely: (1) the &quot;squares&quot; only roughly indicate location of the target area; (2) squares represent volume, which is incongruent when overlaid on 2d geography; (3) shading of mountains/geographic features doesn't add detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also problematic: comparison of h20 volume applied to crop types is difficult to discern using grids. Quick, which crop color uses more water?? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KNGvw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KNGvw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to &lt;em&gt;compare volume to crop type&lt;/em&gt; then you should use a plain column chart, where it is easy to distinguish relative heights. This is analogous to why bar/column charts are preferable to pie charts - humans are better at comparing heights/lengths vs area/volume. If, on the other hand, the point of the graph is to &lt;em&gt;compare crop type irrigation by area&lt;/em&gt;, you would want to display a column graph with regions next to each other, but grouped by crop type. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as a &lt;strong&gt;superbly produced&lt;/strong&gt; duck, my guess is that the Applied Irrigation Water  is a duck, but a superb duck, at least when compared to the monstrous duck on the preceding page of the Visual Display of Quantitative Information 2nd Ed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dMmKm.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dMmKm.jpg&quot; alt=&quot;horrible duck&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-04-24T11:46:19.713" LastActivityDate="2016-04-24T11:46:19.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11378" PostTypeId="2" ParentId="11375" CreationDate="2016-04-23T07:45:02.987" Score="2" Body="&lt;p&gt;I consider this answer as a general approach for the data preparation for CF algorithms and based on my experience. I do not know the specific case of Your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While preparing the data for CF I would take care about those things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;dataset size and sparsity&lt;/em&gt; - there are many CF algorithms and they are giving different results depending on dataset size (users and items) and its data density. Simple intuition: memory-based KNN methods will be suffering from data sparsity while on the same setup the MF can perform better. Good comparison is presented in this article: &lt;a href=&quot;http://arxiv.org/abs/1205.3193&quot; rel=&quot;nofollow&quot;&gt;A Comparative Study of Collaborative Filtering Algorithms&lt;/a&gt;. Moreover, CF methods can depends on the number of users and items differently. To reduce sparsity, You removing items/users which have lower number of ratings/click than selected threshold. This is iterative process. As it's said in the question, when You remove particular item/user it can happen that different user/item respectively can be below the threshold. For research purposes, I even have prepared the function which calculate items/user threshold in order to receive the given sparsity.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;outliers&lt;/em&gt; in the meaning of filtering &lt;em&gt;unusual&lt;/em&gt; users and items. Those, which can influence Your model greatly, but in a wrong way. In my cases (e-commerce website) it were cases like: really heavy users - other shops, bots (scrapping scripts) that was collecting data, internal traffic (blacklist IPs), bidding automation tools, funny offers which lands on a digg-like webpages. Someone can say that this can be removed by user/item bias. Yes, as soon as Your CF method is using it :-)&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;correct slicing&lt;/em&gt; of the dataset. For some reasons (e.g. computational) You have to divide You data on smaller parts. And this is reasonable. Divide and conquer approach. But in such cases before doing this, check if You are not loosing a lot of the information by such slicing. Here is the example: we are slicing offers by categories, categories are formed into a category tree, for this reason the same leaf can be part of few main categories. Thus, slicing by main categories can result in having only a part of the whole user-item interactions about viewed/bought together items in this leaf category.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those are only few tips from my experience.&#xA;But I hope this would help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any edits and sharing the results :-) &lt;/p&gt;&#xA;" OwnerUserId="9314" LastEditorUserId="9314" LastEditDate="2016-04-23T13:11:57.693" LastActivityDate="2016-04-23T13:11:57.693" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11379" PostTypeId="1" CreationDate="2016-04-23T09:23:50.473" Score="2" ViewCount="142" Body="&lt;p&gt;Suppose we use a decision tree to predict if a bank customer can pay back a credit. So it is a two class classification problem. Now we can make two mistakes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\alpha$ error: The customer can back the credit, but we predict he can't.&lt;/li&gt;&#xA;&lt;li&gt;$\beta$ error: The customer can't pay back the credit, but we predict he can.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now we know that $\beta$ errors are 123.4 times as expensive as $\alpha$ errors. But we only have a given set of data. In this set we have $n_1=10000$ customers who paid back the credit and $n_2 = 100$ customers who didn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can the training of the decision tree be adjusted to account for the fact that $\beta$ errors are more expensive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: This is a theoretical question to learn about decision trees. I know about other classifiers like neural networks and I know of ensembles. However, I only want to know about decision trees here.)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2019-10-07T16:50:15.083" Title="How can decision trees be tuned for non-symmetrical loss?" Tags="|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11381" PostTypeId="1" AcceptedAnswerId="97866" CreationDate="2016-04-23T15:02:47.757" Score="1" ViewCount="22" Body="&lt;p&gt;I am dealing with textual data. Imagine a certain number of statements (they could be descriptions), each one referring to a certain process, where each process ends in one f three categories: A, B, C.&lt;/p&gt;&#xA;&lt;p&gt;I would like to characterize the descriptions of type A, the ones of type B, and the ones of type C. One possible way is to focus on the terms contained in the description.&#xA;How can I represent them? One possibility is to compute a document term matrix and once each term is characterized by a vector, compute the distance among all the terms. Starting from these distances it is possible to find a minimum spannign tree connecting the terms and recostructing someway the concepts contained in the original descriptions. Doing that only for the description of type A, etc., we could reconstruct graphically what the questions of type A, etc. are about.&lt;/p&gt;&#xA;&lt;p&gt;The question is: once the concepts are reconstructed...is there other evidence that can be extracted by network analysis?&lt;/p&gt;&#xA;" OwnerUserId="10024" LastEditorUserId="29169" LastEditDate="2021-07-14T22:12:09.057" LastActivityDate="2021-07-14T22:12:09.057" Title="Network analysis applied on terms" Tags="|visualization|graphs|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11382" PostTypeId="2" ParentId="11314" CreationDate="2016-04-23T15:15:33.357" Score="1" Body="&lt;p&gt;You can calculate the confidence interval for the current distance R based on a sliding window approach and immediately signal if the next reading D falls out of that interval. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-04-23T15:15:33.357" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11384" PostTypeId="2" ParentId="11379" CreationDate="2016-04-23T17:55:44.007" Score="3" Body="&lt;p&gt;Theoretically, decision tree algorithms specify the feature as well as the threshold that maximize the separation between classes at each node. This can be done by solving optimization problem related to the entropy at the children nodes. &#xA;You can modify this optimization problem by including the miss- classification costs you have in the optimization problem so the algorithm will be biased toward the most expensive class &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-23T17:55:44.007" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11386" PostTypeId="1" CreationDate="2016-04-24T02:36:23.300" Score="2" ViewCount="21" Body="&lt;p&gt;In most of my projects, I come up with models and want to visualize how some property $x$ varies as a function of a subset of parameters $p_1$,$p_2$, .. etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'll often end up with figures of the &quot;parameter scan&quot; which look like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AzN3O.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AzN3O.png&quot; alt=&quot;example picture&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those are &lt;em&gt;very&lt;/em&gt; helpful for explaining a model / process / datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;problem&lt;/strong&gt; is: I put an inordinate amount of work into producing the data necessary to generate these figures. Most of it wasted on the pipeline itself. I often have to couple anything from 2 to 10 mismatched programs, deal with thousands of intermediary files, track the parameters in the file name along the way, etc.&#xA;It quickly lead to overwhelming bash scripts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any &lt;em&gt;principled&lt;/em&gt; way to go about managing a pipeline of mismatched programs, with parameter scans in mind?&#xA;Say a python module where the user can declare wrappers and how they interact together. Then the user just &quot;launch the pipeline&quot; on every point of the parameter space and recover the data in easily potable format array? &lt;/p&gt;&#xA;" OwnerUserId="18135" LastActivityDate="2021-10-22T14:24:54.640" Title="Reusable parameter scans wrapper" Tags="|tools|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11387" PostTypeId="1" CreationDate="2016-04-24T05:40:10.657" Score="1" ViewCount="419" Body="&lt;p&gt;Once the Model is built we want to check its performance, i did the following &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predicted it on training set.&lt;/li&gt;&#xA;&lt;li&gt;Compute confusion matrix and ROC curve on training set.&lt;/li&gt;&#xA;&lt;li&gt;Predicted on test set &lt;/li&gt;&#xA;&lt;li&gt;Computed Confusion Matrix and ROC Curve on test set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to know is it a right way and what more precise can be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Decision Tress  Random Forest SVM Both Linear Radial Naive Bayeian &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression and Neural Network.  Do all of them have different &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performance Measure&lt;/p&gt;&#xA;" OwnerUserId="18139" LastEditorUserId="18139" LastEditDate="2016-04-24T08:03:28.017" LastActivityDate="2020-03-04T04:53:48.797" Title="Testing Model Performance" Tags="|r|predictive-modeling|performance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11388" PostTypeId="2" ParentId="10168" CreationDate="2016-04-24T05:46:30.900" Score="0" Body="&lt;p&gt;Not sure if you're looking for filter weights in the deconvolutional layer to be tied to corresponding convolutional layer, but either is possible in Lasagne which runs on Theano.&#xA;An untied implementation of deconvolutional layer which outputs an image larger than its input: &lt;a href=&quot;https://groups.google.com/forum/?hl=en#!topic/lasagne-users/9H6-mmnkHX0&quot; rel=&quot;nofollow&quot;&gt;https://groups.google.com/forum/?hl=en#!topic/lasagne-users/9H6-mmnkHX0&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18140" LastActivityDate="2016-04-24T05:46:30.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11389" PostTypeId="2" ParentId="11387" CreationDate="2016-04-24T06:39:29.090" Score="2" Body="&lt;p&gt;You should provide more information about your problem. How large is your data set? Do you have hyper-parameters? What is your classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, you should split your entire data set into three parts: training, validation and testing sets. The testing set you should never use before the model is finally tuned. If you data set is not large enough, you should you use the cross-validation method to measure the error of your classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several metrics to assess a performance of your classifier: accuracy, precision and recall, Cohen's kappa, F1-score, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are working with python and scikit-learn, I would recommend to check &lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/mastering-machine-learning-scikit-learn&quot; rel=&quot;nofollow noreferrer&quot;&gt;this book&lt;/a&gt;. It is nicely written and explains basics of model assessments.&lt;/p&gt;&#xA;" OwnerUserId="14070" LastEditorUserId="29575" LastEditDate="2020-03-04T04:53:48.797" LastActivityDate="2020-03-04T04:53:48.797" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11390" PostTypeId="1" AcceptedAnswerId="11480" CreationDate="2016-04-24T06:55:37.417" Score="28" ViewCount="49815" Body="&lt;p&gt;I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By using sparse methods (LASSO, elastic net) we can remove several features to reduce the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is (theoretically): &lt;strong&gt;before&lt;/strong&gt; we use metrics to assess the model selection are there any empirical observations which relate the &lt;strong&gt;optimal&lt;/strong&gt; number of features to the number of observations? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: for a binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?&lt;/p&gt;&#xA;" OwnerUserId="14070" LastEditorUserId="14070" LastEditDate="2017-06-25T12:29:01.950" LastActivityDate="2021-03-11T19:23:26.167" Title="Any &quot;rules of thumb&quot; on number of features versus number of instances? (small data sets)" Tags="|feature-selection|model-selection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11391" PostTypeId="1" CreationDate="2016-04-24T08:08:09.937" Score="3" ViewCount="658" Body="&lt;p&gt;In Natural Language Processing and Computational Linguistic what methods are deemed as SOA for similar word extraction?&lt;/p&gt;&#xA;&lt;p&gt;Can anyone direct me to those resources?&lt;/p&gt;&#xA;" OwnerUserId="12378" LastEditorUserId="29169" LastEditDate="2021-07-14T22:12:33.270" LastActivityDate="2023-12-16T10:05:40.793" Title="What is the state of the art method for synonym detection?" Tags="|nlp|similarity|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11392" PostTypeId="2" ParentId="11390" CreationDate="2016-04-24T08:12:05.677" Score="9" Body="&lt;p&gt;From my own experience:In one case, I worked with a real database that is very small (300 images) with many classes, severe data imbalance problem and I ended up with using 9 features: SIFT, HOG, Shape context, SSIM, GM and 4 DNN-based features. In another case, I worked with very large database  (&amp;gt; 1 M images) and ended up with using only HOG feature. I think there is no direct relation between the number of instances and the number of features required to achieve high accuracy. BUT: the number of classes, the similarity between classes and variation within the same class (these three parameters) may affect the number of features. when having larger database with many classes and large similarity between classes and large variation within the same class you need more features to achieve high accuracy. REMEMBER: the quality of used features is more important than the number of used features.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="29169" LastEditDate="2021-03-11T19:23:26.167" LastActivityDate="2021-03-11T19:23:26.167" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="11393" PostTypeId="1" CreationDate="2016-04-24T08:35:17.450" Score="1" ViewCount="66" Body="&lt;p&gt;I'm doing some data prep on a dataset provided by a telecommunication company. There is a continuous variable that indicates how many months have passed since a customer renewed her contract. However, 20% of observations have missing values. I figured out that there are two reasons why a value might be missing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- a customer did not renew the contract &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- a customer is still on her first contract and therefore never had the chance to renew it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is to fill the missing values with respectively &quot;Not renewed&quot; and &quot;Not applicable&quot;, and then discretize the rest of the numerical values through a supervised algorithm (chi2, MDLP) so that the variable would turn into a categorical one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, for this variable I'd have some observations categorized through simple labeling, while others through a supervised discretization algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: is it correct to do so? If not, how should I handle the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see my comment below for additional information.&lt;/p&gt;&#xA;" OwnerUserId="17709" LastEditorUserId="17709" LastEditDate="2016-04-28T09:10:11.987" LastActivityDate="2016-04-28T09:10:11.987" Title="How to model this variable?" Tags="|dataset|data-cleaning|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11395" PostTypeId="1" AcceptedAnswerId="11397" CreationDate="2016-04-24T09:07:15.927" Score="3" ViewCount="75" Body="&lt;p&gt;I am facing a regression problem and I have one feature that has some relevant correlation with the output. The value of this feature will be fixed in ALL the predictions I will use this model for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I keep it in my model or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17484" LastActivityDate="2016-04-25T06:38:55.073" Title="Is it a good idea to train with a feature which value will be fixed in future predictions?" Tags="|feature-selection|feature-engineering|feature-construction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11397" PostTypeId="2" ParentId="11395" CreationDate="2016-04-24T09:15:26.380" Score="4" Body="&lt;p&gt;Given that in your training data this feature has different values and some predictive power, I think not keeping this feature would be a mistake (without looking into overfitting due to having too many features). You cannot just discard the feature from your training set if it does influence the target because then these would be from a different population than your predictions and it will be able to learn from the other features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extreme example where x_2 will always be 5 in the future:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_1  x_2  y&#xA;2    8    6&#xA;3    7    5&#xA;2.5  5    1.5&#xA;3    5    0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Just removing x_2 loses a lot of information and would create a significant bias towards higher targets.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-04-25T06:38:55.073" LastActivityDate="2016-04-25T06:38:55.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11398" PostTypeId="2" ParentId="11364" CreationDate="2016-04-24T16:27:31.917" Score="1" Body="&lt;p&gt;Mahalanobis distance depends on the covariance matrix, which is usually local to each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a distance of two clusters, the following two approaches stand out:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;the weighted average distance of each object to the other cluster, using the other clusters Mahalanobis distance. You could approximate this by using the distance of the centroid only. Maybe use the maximum of the two clusters to resolve asymmetry. This will likely &lt;em&gt;not&lt;/em&gt; be a metric.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;divergence measures, that measure the overlap of the two Gaussians, and not of the individual data points. I believe some of the divergence measures should be &lt;em&gt;metric&lt;/em&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-04-24T16:27:31.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11399" PostTypeId="1" CreationDate="2016-04-24T17:12:38.880" Score="5" ViewCount="552" Body="&lt;p&gt;Let's say I have a neural network that classifies data into A, B or C.&#xA;I've heard that it's bad to train the network with data from one class at a time, e.g. &lt;code&gt;A,A,A,...,B,B,B,...,C,C,C&lt;/code&gt;. Now, is there a problem with other patterns in the data? For example &lt;code&gt;A,B,C,A,B,C,A,B,C,...&lt;/code&gt; and so forth.&lt;/p&gt;&#xA;" OwnerUserId="18000" LastActivityDate="2016-05-04T12:29:35.110" Title="Training neural nets: is it important that the data is randomly sorted?" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11400" PostTypeId="1" AcceptedAnswerId="11401" CreationDate="2016-04-24T19:09:47.957" Score="3" ViewCount="2391" Body="&lt;p&gt;While reading some articles about Bayesian Networks, I came across many occurrences of Belief Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do both of these terms mean the same thing or is there any difference between Bayesian Networks and Belief Networks?&lt;/p&gt;&#xA;" OwnerUserId="10049" LastEditorUserId="29169" LastEditDate="2019-05-11T18:28:52.267" LastActivityDate="2019-05-11T18:28:52.267" Title="What is difference between Bayesian Networks and Belief Networks?" Tags="|machine-learning|neural-network|probability|bayesian-networks|difference|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11401" PostTypeId="2" ParentId="11400" CreationDate="2016-04-24T19:39:54.513" Score="6" Body="&lt;p&gt;Both are literally the same. A Belief network is the one, where we establish a &lt;strong&gt;belief&lt;/strong&gt; that certain event A will occur, given B. The network assumes the structure of a directed graph. The term &lt;strong&gt;Bayesian&lt;/strong&gt; was coined after the name of Thomas Bayes.&lt;/p&gt;&#xA;" OwnerUserId="18150" LastActivityDate="2016-04-24T19:39:54.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11402" PostTypeId="1" AcceptedAnswerId="11421" CreationDate="2016-04-25T01:26:50.463" Score="4" ViewCount="9989" Body="&lt;p&gt;I'm going to use ( RNN+Logisitic Regression ) to make sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I do preprocessing for text like remove stop words, punctuation and extract keywords by found nouns ?&lt;/p&gt;&#xA;" OwnerUserId="18152" LastEditorUserId="11097" LastEditDate="2016-04-25T01:46:53.773" LastActivityDate="2016-04-26T06:24:46.263" Title="Preprocessing text before use RNN" Tags="|nlp|logistic-regression|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11404" PostTypeId="1" CreationDate="2016-04-25T07:26:53.743" Score="11" ViewCount="28285" Body="&lt;p&gt;I have a dataset for which I am trying to predict target variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Col1    Col2    Col3    Col4    Col5    &#xA;  1      2       23      11     1&#xA;  2      22      12      14     1&#xA;  22     11      43      38     3&#xA;  14     22      25      19     3&#xA;  12     42      11      14     1&#xA;  22     11      43      38     2&#xA;  1      2       23      11     4&#xA;  2      22      12      14     2&#xA;  22     11      43      38     3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have provided a sample data, but mine has thousands of records distributed in a similar way. Here, Col1, Col2, Col3, Col4 are my features and Col5 is target variable. Hence prediction should be 1,2,3 or 4 as these are my values for target variable. I have tried using algorithms such as random forest, decision tree etc. for predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here if you see, values 1,2 and 3 are occurring more times as compared to 4. Hence while predicting, my model is more biased towards 1 2 and 3 whereas I am getting only less number of predictions for 4 (Got only 1 predicted for policy4 out of thousands of records when I saw the confusion matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to make my model generalize, I removed equal percentage of data that belongs to 1,2 and 3 value randomly. I grouped by each value in Col5 and then removed certain percentage, so that I brought down the number of records. Now I could see certain increase in percentage of accuracy and also reasonable increase in predictions for value 4 in confusion matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right approach to deal with (removing the data randomly from those groups on which the model is biased)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried for in-built python algorithms like Adaboost, GradientBoost techniques using sklearn. I read these algorithms are for handling imbalance class. But I couldnt succeed in improving my accuracy, rather by randomly removing the data, where I could see some improvements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this reduction is undersampling technique and is this the right approach for under-sampling? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there are any pre-defined packages in sklearn or any logic which I can implement in python to get this done, if my random removal is wrong? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I learnt about SMOTE technique, which deals with oversampling. Should I try this for value 4? And can we do this using any in-built packages in python? It would be great if someone helps me in this situation. &lt;/p&gt;&#xA;" OwnerUserId="5043" LastEditorUserId="11097" LastEditDate="2016-04-25T07:38:50.830" LastActivityDate="2020-08-01T15:43:18.380" Title="Python: Handling imbalance Classes in python Machine Learning" Tags="|machine-learning|python|data-mining|dataset|pandas|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11406" PostTypeId="2" ParentId="11404" CreationDate="2016-04-25T08:26:49.277" Score="1" Body="&lt;p&gt;It depends on the ensemble technique you want to use. The basic problem that you are working with multi-class data imbalance problem. Under sampling can be used efficiently in bagging as well as in boosting techniques. &#xA;SMOTE algorithm is very efficient in generating new samples.&#xA;Data imbalance problem has been widely studied in literature. &#xA;I recommend you to read about one of these algorithms: &#xA; SMOTE-Boost&#xA;SMOTE-Bagging&#xA;Rus-Boost&#xA;EusBoost&#xA;These are boosting /bagging techniques designed specifically for imbalance data problem.&#xA;Instead of SMOTE you can try ADA-SMOTE or Border-Line SMOTE.&#xA;I have used and modified the Border-Line SMOTE for multi-class and it is very efficient. &#xA;If your data base is very large and the problem is easy try : viola - jones classifier. I have used also with data imbalance problem and it is really efficient  &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="676" LastEditDate="2016-04-25T17:14:32.333" LastActivityDate="2016-04-25T17:14:32.333" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11407" PostTypeId="1" AcceptedAnswerId="11415" CreationDate="2016-04-25T08:31:35.797" Score="0" ViewCount="89" Body="&lt;p&gt;I have a dataset of transaction data for a retail outlet. I am using pandas and want to analyse revenue by day of the week, but there are unequal days in the dataset (i.e. an extra weekend). I have used &lt;code&gt;df.dt.dayofweek&lt;/code&gt; to create an integer value for the day of the week, and grouped the data by that integer value using &lt;code&gt;df.groupby([&quot;Day_int&quot;,]).sum()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a 'total' column at the end that I am interested in, but I want to create a a new column, something like 'adj_total' that applies a division operation of /3 to the days Monday to Friday and /4 to the days Saturday and Sunday. Is the best way to loop through the dataframe or is it better &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the df that I am working with (most of the column values are nonsensical, only total is of interest). Day_int is the group_by variable and should appear lower than the other columns.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Day_int Section Prod_name Cashier Date Time Receipt Total&#xA;0 ..................................................91341&#xA;1 ..................................................82262&#xA;2 ..................................................84145&#xA;3 ..................................................90115&#xA;4 ..................................................115497&#xA;5 ..................................................151971&#xA;6 ..................................................109210&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18058" LastActivityDate="2016-04-25T17:46:52.393" Title="How to implement accurate counts or sums when you have different numbers of days of the week?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11409" PostTypeId="2" ParentId="11404" CreationDate="2016-04-25T12:02:28.433" Score="4" Body="&lt;p&gt;Some of sklearn's algorithms have a parameter called &lt;em&gt;class_weight&lt;/em&gt; that you can set to &lt;em&gt;&quot;balanced&quot;&lt;/em&gt;. That way sklearn will adjust its class weights depending on the number of samples that you have of each class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the random forest classifier, try the following and see if it improves your score:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rf = RandomForestClassifier(class_weight=&quot;balanced&quot;) # also add your other parameters!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-04-25T12:02:28.433" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11410" PostTypeId="1" AcceptedAnswerId="11493" CreationDate="2016-04-25T12:33:31.813" Score="3" ViewCount="387" Body="&lt;p&gt;I am a bit unsure about optimizing a neural net with 3 or more layers. The input data is quite noisy and I seem to project the noise into the learning (strong bias in the data, 90% belong to one class out of five). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would like to get some feedback on the interpretation (I use 50/25/8/8/8neurons with dropout (keep_prob=0,9 after first hidden layer):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/w3tIr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w3tIr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/m0HH2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m0HH2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18120" LastActivityDate="2016-04-29T00:58:39.603" Title="Noisy behavior in deep learning feed-forward net" Tags="|deep-learning|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11412" PostTypeId="2" ParentId="11220" CreationDate="2016-04-25T14:14:35.827" Score="9" Body="&lt;p&gt;You can use the &lt;a href=&quot;https://sites.google.com/site/nquocdai/resources&quot; rel=&quot;noreferrer&quot;&gt;SAR14 dataset&lt;/a&gt; of 234K IMDb movie reviews. The construction of the SAR14 dataset is detailed in the paper &quot;&lt;a href=&quot;http://www.aclweb.org/anthology/W14-2621&quot; rel=&quot;noreferrer&quot;&gt;Sentiment Classification on Polarity Reviews: An Empirical Study Using Rating-based Features&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="18164" LastActivityDate="2016-04-25T14:14:35.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11413" PostTypeId="1" CreationDate="2016-04-25T14:47:48.097" Score="-1" ViewCount="569" Body="&lt;p&gt;How do we calculate monthly rolling average? I have monthly 2 years of data . I know that if it is 2 months rolling average ,we need to take the average of every 2 months . But since I need monthly rolling average,I am assuming that it would just be the mean of each month and prepare a chart having means of each month . Please correct me if I am wrong..&#xA;I need to show this in R.&lt;/p&gt;&#xA;" OwnerUserId="13417" LastActivityDate="2016-04-25T17:11:34.700" Title="Time series_Calculation of monthly rolling average" Tags="|r|time-series|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11414" PostTypeId="2" ParentId="11413" CreationDate="2016-04-25T17:11:34.700" Score="1" Body="&lt;p&gt;If the 24 monthly measurements are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; data = c(5,4,6,7,6,5,6,7,8,9,1,2,1,2,3,4,5,6,7,8,9,1,2,3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then the monthly rolling averages are....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ra_data = I(data)&#xA; ra_data&#xA; #  [1] 5 4 6 7 6 5 6 7 8 9 1 2 1 2 3 4 5 6 7 8 9 1 2 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-04-25T17:11:34.700" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11415" PostTypeId="2" ParentId="11407" CreationDate="2016-04-25T17:46:52.393" Score="1" Body="&lt;p&gt;I would add a column that is a 3 if it's a weekday and a 4 if it's not using an apply, something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['divide_by'] = df.apply(lambda x: 3 if x['Day_int']&amp;lt;5 else 4, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Assuming days are Monday 0 to Sunday 6. Then you can add the column as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['adj_total'] = df.apply(lambda x: x['Total']/x['divide_by'], axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can remove the divide_by column and you have the result&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-04-25T17:46:52.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11416" PostTypeId="1" CreationDate="2016-04-25T18:05:05.683" Score="1" ViewCount="215" Body="&lt;p&gt;I have a random created dataframe named &lt;code&gt;a&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  group marks  upd&#xA;1     a     2   up&#xA;2     b     3   up&#xA;3     a     4 down&#xA;4     b     5   up&#xA;5     a     6 down&#xA;6     c     7   up&#xA;7     a     8 down&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wanted to subset it with only &lt;code&gt;group='a'&lt;/code&gt; and marks in decreasing order of &lt;code&gt;a&lt;/code&gt;. I used this query but the marks are not arranged in decreasing order. What is the error in my code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code and output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a[a$group==&quot;a&quot; &amp;amp; a$marks[order(a$marks,decreasing=T)],]&#xA;  group marks  upd&#xA;1     a     2   up&#xA;3     a     4 down&#xA;5     a     6 down&#xA;7     a     8 down&#xA;&#xA;Warning message:&#xA;In a$group == &quot;a&quot; &amp;amp; order(a$marks, decreasing = TRUE)[1:3] :&#xA;  longer object length is not a multiple of shorter object length&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="15202" LastEditDate="2016-04-25T23:03:32.397" LastActivityDate="2016-04-26T16:51:54.703" Title="Subsetting and Ordering in R" Tags="|r|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11417" PostTypeId="1" CreationDate="2016-04-25T19:43:45.530" Score="0" ViewCount="932" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&lt;br&gt;&#xA;I am investigating time-series anomaly detection for industrial machine data, and have stumbled upon &lt;a href=&quot;https://www.ge.com/digital/predix&quot; rel=&quot;nofollow&quot;&gt;GE Predix&lt;/a&gt;. It seems like a promising tool, however, I am not familiar with their approaches for anomaly detection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;br&gt;&#xA;Does anyone have experience with &lt;a href=&quot;https://www.ge.com/digital/predix&quot; rel=&quot;nofollow&quot;&gt;GE Predix&lt;/a&gt;, in the anomaly detection space? How would one rate its anomaly detection capability? What technical approaches (algorithms) does it use? Any other literature about &lt;a href=&quot;https://www.ge.com/digital/predix&quot; rel=&quot;nofollow&quot;&gt;GE Predix&lt;/a&gt; that is openly available that I have missed (besides References below)? Are there any available benchmarks (e.g., versus &lt;a href=&quot;https://splunkbase.splunk.com/app/1306/#/overview&quot; rel=&quot;nofollow&quot;&gt;Splunk + Prelert&lt;/a&gt;)? Who are its competitors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;br&gt;&#xA;1) &lt;a href=&quot;https://www.predix.com/sites/default/files/predixplatformbrief-march_2016.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.predix.com/sites/default/files/predixplatformbrief-march_2016.pdf&lt;/a&gt;&lt;br&gt;&#xA;2) &lt;a href=&quot;https://www.predix.com/sites/default/files/GE-Software-Modernizing-Machine-to-Machine-Interactions.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.predix.com/sites/default/files/GE-Software-Modernizing-Machine-to-Machine-Interactions.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1406" LastActivityDate="2016-06-22T20:56:07.530" Title="GE Predix's machine learning (anomaly detection) capabilities" Tags="|machine-learning|time-series|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11418" PostTypeId="2" ParentId="11416" CreationDate="2016-04-25T22:57:39.923" Score="2" Body="&lt;p&gt;&lt;code&gt;a$group==&quot;a&quot;&lt;/code&gt; gives &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;code&gt;a$marks[order(a$marks,decreasing=T)]&lt;/code&gt; gives &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# [1] 8 7 6 5 4 3 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Combining those to using a logical and (&lt;code&gt;&amp;amp;&lt;/code&gt;) does not make much sense here. &lt;code&gt;TRUE &amp;amp; 8&lt;/code&gt; is &lt;code&gt;TRUE&lt;/code&gt; - the subsetting may accidentally work. But you want &lt;code&gt;order&lt;/code&gt; to look at your subset only, not at all rows in &lt;code&gt;a&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try for instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newa &amp;lt;- a[a$group==&quot;a&quot;, ] &#xA;newa[order(newa$marks, decreasing = TRUE), ]&#xA;#   group marks  upd&#xA;# 7     a     8 down&#xA;# 5     a     6 down&#xA;# 3     a     4 down&#xA;# 1     a     2   up&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or, more convenient and readable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;a %&amp;gt;% filter(group==&quot;a&quot;) %&amp;gt;% arrange(desc(marks))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15202" LastEditorUserId="15202" LastEditDate="2016-04-25T23:02:50.423" LastActivityDate="2016-04-25T23:02:50.423" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11419" PostTypeId="2" ParentId="11390" CreationDate="2016-04-25T23:30:29.077" Score="4" Body="&lt;p&gt;It depends... but of course that answer gets you nowhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He is some rule of thumb for model complexity:&#xA;&lt;a href=&quot;https://www.youtube.com/watch?v=Dc0sr0kdBVI&amp;amp;hd=1#t=57m20s&quot; rel=&quot;nofollow&quot;&gt;Learning from data - VC dimension&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Very roughly&quot; you need 10 data points for each model parameter. And number of model parameters can be similar to number of features.&lt;/p&gt;&#xA;" OwnerUserId="723" LastActivityDate="2016-04-25T23:30:29.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11420" PostTypeId="2" ParentId="10329" CreationDate="2016-04-25T23:51:32.383" Score="11" Body="&lt;p&gt;&lt;strong&gt;Yes, a laptop will work just fine for getting acquainted with some deep learning projects:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can pick a smallish deep learning problem and gain some tractable insight using a laptop so give it a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://deeplearning.net/software/theano/&quot; rel=&quot;noreferrer&quot;&gt;Theano&lt;/a&gt; project has a &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;noreferrer&quot;&gt;set of tutorials&lt;/a&gt; on digit recognition that I've played with and moded on a laptop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow&lt;/a&gt; also has a &lt;a href=&quot;https://www.tensorflow.org/versions/r0.8/tutorials/index.html&quot; rel=&quot;noreferrer&quot;&gt;set of tutorials&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I let some of the longer runs go overnight, but nothing was intractable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also consider availing yourself of &lt;a href=&quot;https://aws.amazon.com/&quot; rel=&quot;noreferrer&quot;&gt;AWS&lt;/a&gt; or one of the other cloud services.  For 20-30 dollars you can perform some of the bigger calculations in the cloud on some sort of &lt;a href=&quot;https://aws.amazon.com/ec2/&quot; rel=&quot;noreferrer&quot;&gt;elastic computing node&lt;/a&gt;. The secondary advantage is that you can also list AWS or other cloud services as skill on your resume also :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-04-26T17:44:24.430" LastActivityDate="2016-04-26T17:44:24.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11421" PostTypeId="2" ParentId="11402" CreationDate="2016-04-26T06:24:46.263" Score="7" Body="&lt;p&gt;Welcome to the Data Science forum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and xml contain tags and other data structures that provide more metadata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a high level the sentiment analysis (using bag of words) will involve 4 steps: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Step 1: Data Assembly&lt;/li&gt;&#xA;&lt;li&gt;Step 2: Data Processing&lt;/li&gt;&#xA;&lt;li&gt;Step 3: Data Exploration or Visualization&lt;/li&gt;&#xA;&lt;li&gt;Step 4: Model Building &amp;amp; Validation (train &amp;amp; test)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Lets understand different possible data preprocessing activities:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Convert text to lowercase&lt;/code&gt;&lt;/strong&gt; – This is to avoid distinguish between words simply on case.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Remove Number&lt;/code&gt;&lt;/strong&gt; – Numbers may or may not be relevant to our analyses. Usually it does not carry any importance in sentiment analysis&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Remove Punctuation&lt;/code&gt;&lt;/strong&gt; – Punctuation can provide grammatical context which supports understanding. For bag of words based sentiment analysis punctuation does not add value. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Remove English stop words&lt;/code&gt;&lt;/strong&gt; – Stop words are common words found in a language. Words like for, of, are etc are common stop words.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Remove Own stop words(if required)&lt;/code&gt;&lt;/strong&gt; – Along with English stop words, we could instead or in addition remove our own stop words. The choice of own stop word might depend on the domain of discourse, and might not become apparent until we’ve done some analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Strip white space&lt;/code&gt;&lt;/strong&gt; – Eliminate extra white spaces.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Stemming&lt;/code&gt;&lt;/strong&gt; – Transforms to root word. Stemming uses an algorithm that removes common word endings for English words, such as “es”, “ed” and “’s”. For example i.e., 1) “computer” &amp;amp; “computers” become “comput”&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Lemmatisation&lt;/code&gt;&lt;/strong&gt; – transform to dictionary base form i.e., “produce” &amp;amp; “produced” become “produce”&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;Sparse terms&lt;/code&gt;&lt;/strong&gt; – We are often not interested in infrequent terms in our documents. Such “sparse” terms should be removed from the document term matrix.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To give you more insight onto the steps involved, here are some example sentiment analysis using logistic regressions codes &lt;a href=&quot;https://github.com/srom/sentiment&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/srom/sentiment&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/jadianes/data-science-your-way/blob/master/04-sentiment-analysis/README.md&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/jadianes/data-science-your-way/blob/master/04-sentiment-analysis/README.md&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="8465" LastActivityDate="2016-04-26T06:24:46.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11424" PostTypeId="2" ParentId="11416" CreationDate="2016-04-26T08:20:26.560" Score="1" Body="&lt;p&gt;without Dplyr, you could use &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b&amp;lt;- a[a$group==&quot;a&quot;,]       &#xA;c&amp;lt;- b[b$marks[order(b$marks,decreasing=T),]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this because you are referring to the order of $b$ inherited from a, and you can not use the ranking of the items of the list a to indicate where the rows of b should go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need the first three items in there, take   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;head(c,3) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18179" LastEditorUserId="18179" LastEditDate="2016-04-26T16:51:54.703" LastActivityDate="2016-04-26T16:51:54.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11429" PostTypeId="1" AcceptedAnswerId="11433" CreationDate="2016-04-26T11:22:21.663" Score="1" ViewCount="1203" Body="&lt;p&gt;What criteria can be used to decide whether to develop a decision tree or one or more rules using &lt;a href=&quot;http://search.r-project.org/library/FSelector/html/oneR.html&quot; rel=&quot;nofollow&quot;&gt;oneR&lt;/a&gt; or jRip(contained in &lt;a href=&quot;https://cran.r-project.org/web/packages/RWeka/RWeka.pdf&quot; rel=&quot;nofollow&quot;&gt;RWeka&lt;/a&gt;). The similarities between these two algorithms (decision tree and rule) is very high making it a little confusing.&lt;/p&gt;&#xA;" OwnerUserId="17959" LastEditorUserId="17959" LastEditDate="2016-04-26T12:02:11.203" LastActivityDate="2016-04-26T13:18:58.393" Title="Decision tree or rule" Tags="|classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11430" PostTypeId="1" CreationDate="2016-04-26T11:28:59.653" Score="0" ViewCount="13704" Body="&lt;p&gt;I have made a 3x3 PCA matrix with &lt;code&gt;sklearn.decomposition PCA&lt;/code&gt; and plotted it to a matplotlib 3D scatter plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I annotate labels near the points/marker? Here is my code:&#xA;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from mpl_toolkits.mplot3d import axes3d&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib notebook&#xA;fig = plt.figure()&#xA;ax = fig.gca(projection='3d')&#xA;ax.scatter(&#xA;       existing_df_3dx['PC0'], existing_df_3dx['PC1'], existing_df_3dx['PC2'],  # data&#xA;       s=60                                       # marker size&#xA;       )&#xA;&#xA;plt.show() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, if you know of a better way to plot a 3D PCA, please post your code&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A solution provided from Stack Overflow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(len(data_df_3dx)):&#xA;    x, y, z = data_df_3dx.iloc[i]['PC1'], data_df_3dx.iloc[i]['PC2'], data_df_3dx.iloc[i]['PC3']&#xA;    ax.scatter(x, y, z)&#xA;    #now that you have the coordinates you can apply whatever text you need. I'm &#xA;    #assuming you want the index, but you could also pass a column name if needed&#xA;    ax.text(x, y, z, '{0}'.format(data_df_3dx.index[i]), size=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17886" LastEditorUserId="434" LastEditDate="2016-04-26T23:08:51.987" LastActivityDate="2017-04-20T00:34:35.980" Title="How to annotate labels in a 3D matplotlib scatter plot?" Tags="|python|scikit-learn|pca|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11432" PostTypeId="1" CreationDate="2016-04-26T12:58:44.133" Score="1" ViewCount="129" Body="&lt;p&gt;I am having a dataset which all the features are from 0 to 1(real numbers) and the output is 0 or 1(integer numbers). Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var1   var2   var3  output&#xA;0.01    0.1    0.7       1&#xA;0.01    0.1    0.7       1&#xA; 0.1    0.2    0.3       0&#xA; 0.2    0.4    0.4       0&#xA; 0.4    0.1    0.9       1    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which classification algorithm is recommended when the variables are 'normalized' from 0 to 1? Does SVM or logistic regression 'react' good in these types of data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have noticed that most of the people that are using SVM,NN or logistic regression when making feature scaling they use stardadization( &lt;em&gt;(value-mean)/std )&lt;/em&gt; ). Is there a reason not to rescale the values between from 0 to 1?&lt;/p&gt;&#xA;" OwnerUserId="14736" LastEditorUserId="18194" LastEditDate="2016-04-26T15:43:12.323" LastActivityDate="2016-04-26T15:43:12.323" Title="Choosing an algorithm with normalized data(Classification)" Tags="|machine-learning|classification|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11433" PostTypeId="2" ParentId="11429" CreationDate="2016-04-26T13:18:58.393" Score="0" Body="&lt;p&gt;JRip implements a propositional rule learner, “Repeated Incremental Pruning to Produce Error Reduction” &lt;a href=&quot;http://sci2s.ugr.es/keel/pdf/algorithm/congreso/slipper.pdf&quot; rel=&quot;nofollow&quot;&gt;(RIPPER)&lt;/a&gt;, as proposed by Cohen (1995) and OneR builds a simple &lt;a href=&quot;http://www.mlpack.org/papers/ds.pdf&quot; rel=&quot;nofollow&quot;&gt;1-R classifier&lt;/a&gt;, proposed by Holte (1993).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its hard to say which algorithm works better. Best approach is to compare different classification algorithms performance in terms of precision, recall, accuracy, f1 score, AUC, specificity and sensitivity on your train/test data set and pick the one that gives best results or use ensemble of top performing algorithms to build your final model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a good white paper doing similar exercise of comparing different classification algorithms including OneR and JRIP &lt;a href=&quot;https://www.researchgate.net/publication/273015702_Applying_Naive_bayes_BayesNet_PART_JRip_and_OneR_Algorithms_on_Hypothyroid_Database_for_Comparative_Analysis&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="8465" LastActivityDate="2016-04-26T13:18:58.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11434" PostTypeId="2" ParentId="10972" CreationDate="2016-04-26T13:33:42.197" Score="4" Body="&lt;p&gt;This is something I've experimented with substantially.  Predicting the compressive strength of concrete based on it's mix design is a great example, and fairly challenging - I used the concrete compressive strength dataset available at the UCI ML repository to help me better understand how to manipulate ML algorithms to get good predictive results.  &lt;a href=&quot;https://bigml.com/user/charleslparker/gallery/model/5081b654035d076153000004&quot; rel=&quot;nofollow&quot;&gt;Here's a guy&lt;/a&gt; who took the same data set and used BigML's tools to do an analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also used ML to predict payitem bid prices on highway construction contracts to help improve cost estimation.  For example, I set out to determine the unit price for a specific item, like guardrail, that contractors are likely to bid given a contract's location, time of year, total value, relevant cost indices, etc.  It took a lot of time and experimentation, but I achieved surprisingly good predictive results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your specific topics of interest, I imagine you'll have to just do a lot of searching and reviewing master's theses on related topics, taking time to learn the mathematical models and theories that they employ.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know where to get started, my learning path involved the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Learning whatever I could about neural networks and machine learning&#xA;theory (guys like Jeff Heaton proved helpful, here) &lt;/li&gt;&#xA;&lt;li&gt;Taking Andrew Ng's &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/home/info&quot; rel=&quot;nofollow&quot;&gt;Machine Learning course&lt;/a&gt; on Coursera.&lt;/li&gt;&#xA;&lt;li&gt;Downloading and getting to know the R scripting language&lt;/li&gt;&#xA;&lt;li&gt;Reading whitepapers / theses on applying ML to specific&#xA;applications.&lt;/li&gt;&#xA;&lt;li&gt;Developing a solid, rigorous understanding of basic statistics theory, especially linear regression.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There's a lot of different ways to approach the topic, though.  Websites like BigML.com are a good way to get your feet wet doing ML without having a rigorous mathematical / statistical understanding, but it'll only get you so far.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, you'll really need to devote time to getting your head wrapped around the math and programming in languages like R or Python.  It's also important to remember that while ML seems to favor complexity over simplicity (in many cases non-linear problems benefit from novel and sometimes non-intuitive approaches), sometimes the simplest approach is best.  It's really a question of how accurate the final model really needs to be.&lt;/p&gt;&#xA;" OwnerUserId="18189" LastActivityDate="2016-04-26T13:33:42.197" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11436" PostTypeId="2" ParentId="11432" CreationDate="2016-04-26T14:08:52.233" Score="1" Body="&lt;p&gt;Re-scaling or any other form of standardization/normalization is very useful when dealing with models that are trained with gradient descent: (SVM,NN, LogReg). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question explains the effect of normalization on the gradient rather well: &lt;a href=&quot;https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re&quot;&gt;https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision trees for example are invariant to linear transformations, therefore feature scaling, in theory, does not effect the model in any way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of your set-up, you have a binary classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend trying out first a linear classifier on your data. Logistic regressions is a good first choice. If you decide that LogReg is not working as well as you would like it to then you can move on to more complicated models. I would recommend using decision trees coupled with a gradient boosting model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In problems that are similar to what you described, specifically when there is no inherent structure in the data (as there is in image classification, speech recognition, etc.), gradient boosting models tend to outperform neural networks.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-04-26T14:08:52.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11439" PostTypeId="1" CreationDate="2016-04-26T15:51:51.710" Score="3" ViewCount="1240" Body="&lt;p&gt;I am working on making predictions about traffic trends in a big city. Traffic is very seasonnal, and depends on the time of the day (with peaks at 9am and 6pm), day of the week (Monday to Friday are alike compared to Sat and Sun), holidays (Xmas, school holidays etc). And I have 2 years (at hourly rate) of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow&quot;&gt;RandomForestRegressor&lt;/a&gt; to make a prediction on the year to come. Is that a correct algorithm for this class of problem?&lt;/p&gt;&#xA;" OwnerUserId="18195" LastEditorUserId="11097" LastEditDate="2016-04-26T16:00:27.097" LastActivityDate="2016-04-26T17:57:41.207" Title="Prediction algorithm for traffic" Tags="|machine-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11440" PostTypeId="1" CreationDate="2016-04-26T17:36:05.733" Score="1" ViewCount="1048" Body="&lt;p&gt;I have a data-set with two variable frequency values as given below &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col[0],col[1],col[2],col[3]&#xA;val1, 23, 34, key1&#xA;val1, 23, 56, key2&#xA;val1, 23, 54, kay3&#xA;val2, 34, 76, kay1&#xA;val2, 34, 87, key4&#xA;val3, 90, 23, key2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Real Datasets &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Reduction_Paris 957 228 #promos&#xA;Reduction_Paris 957 214 #promos&#xA;Reduction_Paris 957 181 #paris&#xA;Reduction_Paris 957 175 #promos&#xA;CNNsWorld   361 156 #breakingnews&#xA;CNNsWorld   361 156 #gamedev&#xA;CNNsWorld   361 156 #paris&#xA;CNNsWorld   361 156 #parisattack&#xA;FollowParisNews 705 131 #france&#xA;FollowParisNews 705 131 #news&#xA;FollowParisNews 705 131 #paris&#xA;Reduction_Paris 957 128 #promos&#xA;Reduction_Paris 957 128 #paris&#xA;FollowParisNews 705 128 #france&#xA;FollowParisNews 705 128 #news&#xA;FollowParisNews 705 128 #paris&#xA;FollowParisNews 705 125 #france&#xA;FollowParisNews 705 125 #news&#xA;FollowParisNews 705 125 #paris&#xA;Reduction_Paris 957 123 #paris&#xA;FollowParisNews 705 119 #france&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data set has more than 1000s of rows &lt;code&gt;val&lt;/code&gt; is a string variable and &lt;code&gt;key&lt;/code&gt; is also string. where as the &lt;code&gt;col[1]&lt;/code&gt; and &lt;code&gt;col[2]&lt;/code&gt; are the frequencies respectively of &lt;code&gt;val&lt;/code&gt; and &lt;code&gt;key&lt;/code&gt;. The &lt;code&gt;val&lt;/code&gt; and &lt;code&gt;key&lt;/code&gt; are connected to each other several values are connected to several keys.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to plot two kinds of graphs. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A graph to show pair inverse cumulative distribution function (ccdf). &lt;code&gt;col[1]&lt;/code&gt; and &lt;code&gt;col[2]&lt;/code&gt; are to be considered in this case. When it is only one value we can very well plot ccdf as individual but as a pair I have no idea how to get the distribution plots.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A connected graph to show the connection relationship between the &lt;code&gt;val col[0]&lt;/code&gt; and &lt;code&gt;key col[3]&lt;/code&gt;. I tried using &lt;code&gt;Cytoscape&lt;/code&gt; and chose key as node and val as edge but I do not get the graph that I am looking for. Any solution on how I can show the relationship between the two ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;May be something like this kind of solution I am looking for, Cluster of Keywords col[3] and the col[0] connected to them.Either of the one type of graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dg6sV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dg6sV.jpg&quot; alt=&quot;Image_2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tLzUK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tLzUK.png&quot; alt=&quot;Image_1&quot;&gt;&lt;/a&gt;&#xA;I am comfortable with python so any suggestion in that area will be helpful. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="3486" LastEditorUserId="3486" LastEditDate="2016-05-11T06:36:09.807" LastActivityDate="2016-05-11T06:36:09.807" Title="Multi Model data set Visualization Python" Tags="|python|bigdata|visualization|graphs|research|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11441" PostTypeId="2" ParentId="11439" CreationDate="2016-04-26T17:57:41.207" Score="3" Body="&lt;p&gt;I think the best way to deal with this problem is to use blending (although it is not a bad idea to start with random forest just to get sense about the problem you have). If you are not familiar with the word blending you can check this page:&#xA;&lt;a href=&quot;http://mlwave.com/kaggle-ensembling-guide/&quot; rel=&quot;nofollow&quot;&gt;http://mlwave.com/kaggle-ensembling-guide/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But let me explain what is the idea:&#xA;I believe that there is a kind of relation between the features you have: you can group all the features related to Weather together, The day, time ..etc together. Each set of features can be used to train a random forest. The output of these random forests can be combined by either linear or non linear regression blending techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the database you have is very large it is better to read more about blending techniques and do some experiments before going to your problem to make yourself familiar with it  &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-26T17:57:41.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11442" PostTypeId="1" CreationDate="2016-04-26T19:12:58.750" Score="2" ViewCount="396" Body="&lt;p&gt;I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions for a good book/blog/any source where I could learn about this. The challenges I see are - the sample size is usually one or 2 stores for the test so how do you do statistically significant testing with such a small sample; test stores or control stores are matched 1-1 but often not selected randomly. Any guidance in learning about the data science in this area is greatly appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!!&lt;/p&gt;&#xA;" OwnerUserId="18200" LastEditorUserId="25517" LastEditDate="2016-10-26T12:17:11.640" LastActivityDate="2016-10-26T12:17:11.640" Title="Retail Store Testing" Tags="|statistics|data|ab-test|marketing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11444" PostTypeId="2" ParentId="9424" CreationDate="2016-04-27T02:39:03.537" Score="4" Body="&lt;p&gt;I am working on Random Forest Classifier and this classifier has &lt;strong&gt;probability&lt;/strong&gt; attribute in prediction i.e if you get the summary of &lt;code&gt;predictions = model.transform(testData)&lt;/code&gt; as &lt;code&gt;print(predictions)&lt;/code&gt; in &lt;strong&gt;PySpark&lt;/strong&gt; you will get the probability of each labels, You can check the below code and output of the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql import DataFrame&#xA;from pyspark import SparkContext, SQLContext&#xA;from pyspark.ml import Pipeline&#xA;from pyspark.ml.classification import RandomForestClassifier&#xA;from pyspark.ml.feature import StringIndexer, VectorIndexer&#xA;from pyspark.ml.evaluation import MulticlassClassificationEvaluator&#xA;&#xA;# Split the data into training and test sets (30% held out for testing)&#xA;(trainingData, testData) = data.randomSplit([0.7, 0.3])&#xA;&#xA;# Train a Random Forest model.&#xA;rf = RandomForestClassifier(labelCol=&quot;label&quot;, featuresCol=&quot;features&quot;, numTrees=12,  maxDepth=10)&#xA;&#xA;# Chain RF in a Pipeline&#xA;pipeline = Pipeline(stages=[rf])&#xA;&#xA;# Train model.&#xA;model = pipeline.fit(trainingData)&#xA;&#xA;# Make predictions.&#xA;predictions = model.transform(testData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now your work start from here. try printing predictions and values of predictions&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DataFrame[label: double, features: vector, indexed: double, rawPrediction: vector, probability: vector, prediction: double]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, in the DataFrame you have the &lt;strong&gt;probability&lt;/strong&gt; which is the probability of each indexedLabel, Further I have checked it as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print predictions.show(3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----+--------------------+-------+--------------------+--------------------+----------+&#xA;|label|            features|indexed|       rawPrediction|         probability|prediction|&#xA;+-----+--------------------+-------+--------------------+--------------------+----------+&#xA;|  5.0|(2000,[141,260,50...|    0.0|[34.8672584923246...|[0.69734516984649...|       0.0|&#xA;|  5.0|(2000,[109,126,18...|    0.0|[34.6231572522266...|[0.69246314504453...|       0.0|&#xA;|  5.0|(2000,[185,306,34...|    0.0|[34.5016453103805...|[0.69003290620761...|       0.0|&#xA;+-----+--------------------+-------+--------------------+--------------------+----------+&#xA;only showing top 3 rows&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Only for probability column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print predictions.select('probability').take(2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Row(probability=DenseVector([0.6973, 0.1889, 0.0532, 0.0448, 0.0157])), Row(probability=DenseVector([0.6925, 0.1825, 0.0579, 0.0497, 0.0174]))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my case I have &lt;strong&gt;5&lt;/strong&gt; indexedLabels and so the &lt;strong&gt;probability vector length&lt;/strong&gt; is &lt;em&gt;5&lt;/em&gt;, Hope this will help you to get the &lt;em&gt;probability&lt;/em&gt; of each labels in your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: You will probably get the probability in &lt;strong&gt;Decision Tree&lt;/strong&gt;, &lt;strong&gt;Logistic regression&lt;/strong&gt;. Just try to get the summary of &lt;code&gt;model.transform(testData)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-04-27T02:39:03.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11445" PostTypeId="2" ParentId="11442" CreationDate="2016-04-27T02:53:54.177" Score="0" Body="&lt;p&gt;I think your confusion is that you only have two stores, but you might have more than 2 data points being generated from those two stores. For example, you might be interested in how much individual customers spend at the two stores. You can look at receipt totals from the stores. Then you would just run an independent t-test on the receipt totals data, because you have essentially two groups. Group A comes from store A's Reciepts and Group B comes from store B's receipts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this methodology you could look at any number of response variables.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-04-27T02:53:54.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11446" PostTypeId="1" CreationDate="2016-04-27T03:03:06.427" Score="1" ViewCount="1764" Body="&lt;p&gt;I am looking for algorithms on Anomaly detection for time series data. It is uni-variate analysis, considering single parameter (inlet pressure) of air compressor sensor data. The objective is to detect if there is any significant deviation in parameter value in real time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any one has worked on similar projects, please share your thoughts.&lt;/p&gt;&#xA;" OwnerUserId="18207" LastEditorUserId="29169" LastEditDate="2019-06-07T17:34:11.137" LastActivityDate="2019-06-07T17:34:11.137" Title="Anomaly detection in Time Series Data - Help Required" Tags="|machine-learning|predictive-modeling|time-series|anomaly-detection|" AnswerCount="1" CommentCount="3" ClosedDate="2016-04-27T08:18:03.407" ContentLicense="CC BY-SA 4.0" />
  <row Id="11447" PostTypeId="2" ParentId="11446" CreationDate="2016-04-27T05:12:58.970" Score="1" Body="&lt;p&gt;Welcome to DS forum. There are many methods available for anomaly detection of uni-variate, low dimension and high dimension data sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Supervised algorithms&lt;/strong&gt; can be used if the past anomalies are recorded with flag variables and there is significant examples of flagged anomalies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;strong&gt;unsupervised algorithms&lt;/strong&gt; the key aspect of anomaly is which subset of the larger data set is considered when looking for anomolies i.e., global vs local and point vs contextual&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qjJVf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qjJVf.png&quot; alt=&quot;Anomaly Types&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some useful links to get you started, &lt;a href=&quot;http://www.diva-portal.se/smash/get/diva2:621699/FULLTEXT01.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;anomaly detection in machine generated data&lt;/a&gt; and &lt;a href=&quot;http://www.autonlab.org/tutorials/biosurv.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;introductory overview of time-series based anomaly detection algorithms&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="8465" LastActivityDate="2016-04-27T05:12:58.970" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11449" PostTypeId="2" ParentId="6380" CreationDate="2016-04-27T12:37:37.697" Score="0" Body="&lt;p&gt;I always, decrease &lt;code&gt;mtry&lt;/code&gt; until error on train dataset increases, then I lower nodesize and depth until difference between error on train and dataset stops to decrease&lt;/p&gt;&#xA;" OwnerUserId="8237" LastActivityDate="2016-04-27T12:37:37.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11450" PostTypeId="1" CreationDate="2016-04-27T12:42:28.153" Score="2" ViewCount="314" Body="&lt;p&gt;I'm a SQLServer DBA and in the new version of this tool, there are new features to integrate &lt;strong&gt;R scripts&lt;/strong&gt; and use it easily with the DB objects. That sounds cool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to use that, we have to know a little bit about stats and data mining.&#xA;And it's quite hard to find friendly and readable documentation for newbies like me. So, I will ask for some best-practice about &lt;strong&gt;clustering&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with few hundreds records and with 5 features, both categorical and numerical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding categorical data, I have a few basic questions :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;how can I manage &lt;strong&gt;categorical&lt;/strong&gt; data ? what I understood from articles found is that I can transform categorical data in boolean and &quot;pivoting&quot; it. If a have a column with categorical data (for instance with values 'a' or 'b'), creating to new columns like column_a and column_b with value 0 or 1 ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;should I always &lt;strong&gt;center / scale&lt;/strong&gt; this new column ? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18221" LastEditorUserId="18221" LastEditDate="2016-04-28T11:36:44.850" LastActivityDate="2016-12-12T07:57:02.390" Title="Clustering : handling categorical data, should we pivot and scale?" Tags="|r|clustering|k-means|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11451" PostTypeId="2" ParentId="10377" CreationDate="2016-04-27T12:46:12.087" Score="1" Body="&lt;p&gt;Im a little late, but better late than never.  It looks like your line where you find the coefficients: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method=&quot;exhaustive&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should have &lt;code&gt;data=Mydata[train,]&lt;/code&gt; instead of &lt;code&gt;data=Mydata&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model had already seen your test samples, so the validation error is not an accurate assessment.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-04-27T12:46:12.087" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11452" PostTypeId="1" AcceptedAnswerId="41218" CreationDate="2016-04-27T13:34:34.087" Score="3" ViewCount="573" Body="&lt;p&gt;Are there random forest implementations which allows for choosing set of columns which will &lt;strong&gt;be always selected&lt;/strong&gt; for every tree in the forest ? or implementations allowing for specifying &lt;strong&gt;probabilities of choosing&lt;/strong&gt; each column ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both cases could be simulated by choosing variables (partially or fully randomly) and building tree containing all variables in new temporal dataset, then repeating this procedure until we obtain assumed number of trees and then merging trees into forest,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but there are drawbacks like &#xA;moving large number of data in scripting language until they are sent to low level implementation of forest for building tree or lack of merging procedures, which makes work with set of trees harder.&lt;/p&gt;&#xA;" OwnerUserId="8237" LastActivityDate="2018-11-14T15:04:36.670" Title="Random forest implementation with probability of choosing column or guarantee of choosing set of columns" Tags="|algorithms|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11454" PostTypeId="1" CreationDate="2016-04-27T14:52:40.670" Score="1" ViewCount="1322" Body="&lt;p&gt;I have binary classification method name Fclassifier I need to apply it in a multi-class classification problem, this classifier doesn't have any decision_function (or predict_proba) and its core only accept {1,-1} as class labels! I am trying to use &lt;strong&gt;OneVsRestClassifier&lt;/strong&gt; multi-classifier from sklearn so that I can tune the Fclassifier into multi-class classifier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fm = FClassification(n_iter=1000, init_stdev=0.1, rank=2, random_state=123,l2_reg_w=0, l2_reg_V=0, l2_reg=0, step_size=0.1)&#xA;OneVsRestClassifier(fm).fit(X_train,y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I am facing the following error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;RuntimeError: Cannot clone object FClassification(init_stdev=0.1,&#xA;  l2_reg=None, l2_reg_V=0, l2_reg_w=0,&#xA;           n_iter=1000, random_state=123, rank=2, step_size=0.1), as the constructor does not seem to set parameter l2_reg_V&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;do you think if it is the problem of &lt;strong&gt;Fclassifier&lt;/strong&gt; that can not satisfy the expectation of &lt;strong&gt;OneVsRestClassifier&lt;/strong&gt; and if I should define a customized &lt;strong&gt;predict_proba&lt;/strong&gt; which can fit my data to the classifier in order to solve the problem? or if the error may have other reasons and solutions?&lt;/p&gt;&#xA;" OwnerUserId="18203" LastActivityDate="2017-03-03T19:22:43.437" Title="using &quot;OneVsRestClassifier&quot; from sklearn in Python to tune a customized binary classification into a multi-class classification" Tags="|python|scikit-learn|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11456" PostTypeId="1" AcceptedAnswerId="11501" CreationDate="2016-04-27T17:12:40.743" Score="1" ViewCount="121" Body="&lt;p&gt;Dataset: I'm given the number of minutes individual customers use a product each day and am trying to cluster this data in order to find common usage patterns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: How can I format the data so that, for example, a power user with high levels of use for a year looks the same as a different power user who has only been able to use the device for a month before I ended data collection?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've turned each customer into an array where each cell is the number of minutes used that day. This array starts when the user first uses the product and ends after the user's first year of use. All entries in the cells must be double values (e.x. 200.0 minutes used) for the clustering model. I've considered either setting all cells/days after the last day of data collection to either -1.0 or NULL. Are either of these a valid approach? If not what would you suggest?&lt;/p&gt;&#xA;" OwnerUserId="18232" LastActivityDate="2016-04-29T07:51:43.363" Title="Prepping Data For Usage Clustering" Tags="|machine-learning|data-mining|clustering|algorithms|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11457" PostTypeId="1" CreationDate="2016-04-27T18:19:33.597" Score="4" ViewCount="933" Body="&lt;p&gt;I have some social network data in which I'd like to identify people who belong to a group of 5 friends.  These are users who are friends with 4 other people who all are also friends with each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the graph sense, this task involves identifying components of the graph where each node has an edge between every other node in the component and where the component contains at least 5 nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a well known algorithm to do this?  I was looking at measures like centrality, clustering coefficients, and strongly connected components but none of those are quite what I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="18235" LastActivityDate="2016-05-06T10:27:42.850" Title="Finding groups of friends in social network data" Tags="|graphs|social-network-analysis|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11459" PostTypeId="1" CreationDate="2016-04-27T19:12:34.287" Score="2" ViewCount="83" Body="&lt;p&gt;I have collected some data I need to analyze. The data is the result of a survey in which I asked approx. 180 sellers at a bazaar, how important a certain buyer's characteristic is in relation to their price setting on a scale from '1 = absolutely unimportant' to '10 = extremely important' (for instance, how important is a buyer's nationality in relation to the price a merchant is offering his goods).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now have analyzed my results and clustered the data, where it was necessary, but I do not know at which point a value gets statistically important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I separate the data into two groups, one for answers below 5 and one for above, or in three groups (1-3,4-7,8-10) and at how many percent is something statistically important? Is one characteristic significant important, if 80% of the answers were 8 and above, is it still if there are just 60% above?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Ok let me clearify my question: I do not want to compare two chracteristics to each other. I am using R and I created histograms with density lines and also cummulative density lines to present my data. The problem is, that I do not know how I can say wether  one particular characteristic is important or not. Do I need to focus on their means? And how does their standard deviation affect the interpretation?&#xA;I have conducted a survey in one particular bazaar with approx. 3600 merchants, meaning I have the answers of 5% of them.&lt;/p&gt;&#xA;" OwnerUserId="18240" LastEditorUserId="1330" LastEditDate="2022-01-12T19:34:47.233" LastActivityDate="2022-01-12T19:34:47.233" Title="How do you determine if a value is statistically significant?" Tags="|r|descriptive-statistics|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11460" PostTypeId="1" AcceptedAnswerId="11462" CreationDate="2016-04-27T19:21:19.720" Score="3" ViewCount="419" Body="&lt;p&gt;I am working on a project that takes in comments in a given Facebook page and determines the average happiness rating for it based on users of the page. My question is where would I find a formula, rating, or any other literature that would help me find a way to measure happiness using words and collections of words?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for something similar to cosine similarity I suppose, but not to find similar words, but to find the average positivity or happiness related to a word or collection of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure this is the correct place for this question, but it has to do with data, and big data within Facebook so I am hoping to either get a question or be directed to somewhere that I may be able to find my answer. Thanks in advance for your help.&lt;/p&gt;&#xA;" OwnerUserId="18239" LastActivityDate="2016-04-27T20:52:13.717" Title="Algorithm or formula to measure happiness?" Tags="|data-mining|data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11461" PostTypeId="2" ParentId="11457" CreationDate="2016-04-27T19:22:20.653" Score="5" Body="&lt;p&gt;&lt;strong&gt;Community Detection and Clique Percolation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Community_structure&quot; rel=&quot;nofollow&quot;&gt;community detection&lt;/a&gt; problem.  Here is a very detailed &lt;a href=&quot;http://www.uvm.edu/~pdodds/teaching/courses/2009-08UVM-300/docs/others/everything/fortunato2010a.pdf&quot; rel=&quot;nofollow&quot;&gt;review article&lt;/a&gt; surveying the state of the art.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Clique_percolation_method&quot; rel=&quot;nofollow&quot;&gt;Clique Percolation Method&lt;/a&gt; is also useful to explore as it pretty much solves what you may need to know.  You can also go through what &lt;a href=&quot;https://en.wikipedia.org/wiki/Matching_%28graph_theory%29&quot; rel=&quot;nofollow&quot;&gt;Matching&lt;/a&gt; is, and link the concept with &lt;a href=&quot;https://en.wikipedia.org/wiki/Blossom_algorithm&quot; rel=&quot;nofollow&quot;&gt;Blossom Algorithm&lt;/a&gt;. Though, this finds the maximum matching, you may slightly modify the algorithm to get the minimum matchings of 5 at least.&lt;/p&gt;&#xA;" OwnerUserId="18150" LastEditorUserId="6550" LastEditDate="2016-05-05T16:15:47.223" LastActivityDate="2016-05-05T16:15:47.223" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11462" PostTypeId="2" ParentId="11460" CreationDate="2016-04-27T19:42:21.550" Score="4" Body="&lt;p&gt;You should be looking towards Natural Language Processing, specifically at &lt;a href=&quot;http://nlp.stanford.edu/sentiment/&quot; rel=&quot;nofollow&quot;&gt;Sentiment Analysis&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link I provided is a good starting point for learning about sentiment analysis.  If this is what you are looking for, it is available as part of Stanford's &lt;a href=&quot;http://stanfordnlp.github.io/CoreNLP/&quot; rel=&quot;nofollow&quot;&gt;Core NLP&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2016-04-27T19:42:21.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11463" PostTypeId="2" ParentId="11460" CreationDate="2016-04-27T19:50:43.930" Score="1" Body="&lt;p&gt;These doesn't exist an absolute solution for the your question, but I can suggest you some techniques which may help.&#xA;&lt;br&gt;If I'm not wrong, then you're basically trying to interpret the English language into human emotions.&#xA;&lt;br&gt;There are complete theories written on the subject, and after doing a little bit of research, I've come up with two things that might prove to be handy in your case.&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Affective_computing&quot; rel=&quot;nofollow&quot;&gt;Affective Computing&lt;/a&gt;&#xA;&lt;br&gt;&#xA;The main idea is, that you collect the verbal data (as much as you can), and you apply machine learning algorithms (classification) on the verbal data to figure out whether the verbal data is related to &quot;happiness&quot;, or not. You can think of as many important features as you can. One feature can be, that you can assign a number to each word, like (happy=1, happier=3, happiest=10)...this is just an example. Another feature can be, that you try to figure out &quot;nots&quot; and &quot;negations&quot; in your sentence, because &quot;not happy&quot; will obviously get you to a negative value in your feature. You can think of as many similar linguistic features as you can. After having a reasonable number of calculated features, you can simply apply the algorithms mentioned, and you may eventually get to a point.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ace.cs.ohio.edu/~razvan/papers/icdm2003.pdf&quot; rel=&quot;nofollow&quot;&gt;Sentiment Analyzer&lt;/a&gt;&#xA;&lt;br&gt;&#xA;I just found this research paper, and it works with Natural Language Processing, and after processing the language, it analyzes the emotions of the person, or the source. You can go through the study by yourself.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18150" LastActivityDate="2016-04-27T19:50:43.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11464" PostTypeId="2" ParentId="11460" CreationDate="2016-04-27T20:52:13.717" Score="1" Body="&lt;p&gt;Why don't you empirically derive happiness based upon a sampling of the population of your texts (i.e. facebook comments). You can recruit reviewers (either &quot;naive&quot; participants or perhaps experts in psychology/emotion depending upon your needs). Give reviewers some sort of scale to rate texts on - either classify texts as positive, negative, or neutral OR have them rate texts on a likert type scale, e.g., on a scale of 1 to 10, with 10 being max, how happy do you think this text is?). By having multiple reviewers you can set thresholds for reliability thresholds (e.g., 2 out of 3 reviewers must agree on assignment/score; 2 reviewers come to consensus on disagreement, etc.). This data then becomes your labelled training data that you can use to score your remaining data.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-04-27T20:52:13.717" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11465" PostTypeId="2" ParentId="11457" CreationDate="2016-04-27T21:45:45.523" Score="1" Body="&lt;p&gt;Not so much an 'algorithm' but Neo4j allows you to query a graph network, and this type of question features quite heavily in their demos. Here is an example of the Cypher language syntax you might have to use in your social network case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MATCH (a:Person)-[:FRIENDS_WITH]-&amp;gt;(b:Person), (b)-[:FRIENDS_WITH]-&amp;gt;(c:Person) &#xA;WITH b.name AS Friends, count(c) as MutualFriends    &#xA;WHERE a.name = &quot;Ollie&quot; AND NOT (a)-[:FRIENDS_WITH]-&amp;gt;(c) AND MutualFriends &amp;gt; 5&#xA;RETURN Friends, MutualFriends&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code might not be perfect, but it's for show. This kind of query returns results very quickly with medium to large graphs, and especially so if you can load your data and use indexes to search by name. Depends if you want to start from scratch!&lt;/p&gt;&#xA;" OwnerUserId="16287" LastEditorUserId="16287" LastEditDate="2016-04-27T22:02:51.257" LastActivityDate="2016-04-27T22:02:51.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11466" PostTypeId="1" AcceptedAnswerId="11507" CreationDate="2016-04-27T22:36:32.490" Score="1" ViewCount="544" Body="&lt;p&gt;I have a N-way frequency table generated from a regression model fit. This is a reproducible example of such table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(&quot;CO2&quot;)&#xA;lm.fit = lm(uptake ~  Type + Treatment, data = CO2)&#xA;lm.fit$coefficients&#xA;&#xA;test = count(CO2, c('Type','Treatment'))&#xA;test$res = predict(lm.fit, newdata = test)&#xA;test$freq = NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to visualize &lt;code&gt;test&lt;/code&gt; as a decision tree with nodes as &lt;code&gt;Type&lt;/code&gt; and &lt;code&gt;Treatment&lt;/code&gt; and &lt;code&gt;res&lt;/code&gt; as leaves. I would interpret it as the path the regression model takes, leading to the final value for a particular segment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to generate a tree with &lt;code&gt;test&lt;/code&gt;. I am also open to other novel ways of visualizing these results. My original problem has many categorical variables, so I am looking for a customizable visualization, something from &lt;code&gt;party::ctree&lt;/code&gt; or &lt;code&gt;rattle::fancyRpartPlot&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18248" LastEditorUserId="31513" LastEditDate="2017-04-24T17:37:39.667" LastActivityDate="2017-04-24T17:37:39.667" Title="Visualizing N-way frequency table as a Decision Tree in R" Tags="|r|visualization|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11467" PostTypeId="1" CreationDate="2016-04-28T00:07:22.413" Score="9" ViewCount="15862" Body="&lt;p&gt;They look like the same thing to me but I'm not sure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: in retrospect, this was not a very good question. OLS refers to fitting a line to data and RSS is the cost function that OLS uses. It finds the parameters that gives the &lt;strong&gt;least&lt;/strong&gt; residual sum of &lt;strong&gt;square&lt;/strong&gt; errors. It is called &lt;strong&gt;ordinary&lt;/strong&gt; in OLS refers to the fact that we are doing a linear fit.  &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="12443" LastEditDate="2016-05-12T18:29:14.927" LastActivityDate="2017-07-19T07:41:01.513" Title="What is the difference between residual sum of squares and ordinary least squares?" Tags="|linear-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11468" PostTypeId="2" ParentId="11467" CreationDate="2016-04-28T00:35:01.250" Score="15" Body="&lt;p&gt;Here is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Residual_sum_of_squares&quot; rel=&quot;noreferrer&quot;&gt;definition from &lt;strong&gt;Wikipedia&lt;/a&gt;&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In statistics, the residual sum of squares (RSS) is the sum of the&#xA;  squares of residuals. It is a measure of the discrepancy between the&#xA;  data and an estimation model; Ordinary least squares (OLS) is a method&#xA;  for estimating the unknown parameters in a linear regression model,&#xA;  with the goal of minimizing the differences between the observed&#xA;  responses in some arbitrary dataset and the responses predicted by the&#xA;  linear approximation of the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So RSS is a &lt;strong&gt;measure&lt;/strong&gt; of how good the model approximates the data while OLS is a &lt;strong&gt;method&lt;/strong&gt; of constructing a good model.&lt;/p&gt;&#xA;" OwnerUserId="18249" LastEditorUserId="11097" LastEditDate="2016-04-28T04:58:43.547" LastActivityDate="2016-04-28T04:58:43.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11469" PostTypeId="1" AcceptedAnswerId="11477" CreationDate="2016-04-28T02:00:35.087" Score="3" ViewCount="2873" Body="&lt;p&gt;I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Watch  | Nothing | Interact&#xA;A: 327445 | 271602  | 744702&#xA;B: 376455 | 140737  | 818204  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to use the chisquared test bundled with scipy. I'm completely new to data science, but I believe this is the evaluation metric I would want.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scipy.stats.chisquare([ [327445, 271602, 744702], [376455, 140737, 818204] ])&#xA;Power_divergenceResult(statistic=array([  3412.38826538,  41532.93339946,   3456.72996585]), pvalue=array([ 0.,  0.,  0.]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This does not look like a valid result...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I even tried adding an &lt;code&gt;expected frequencies&lt;/code&gt; options, but without success. Maybe I'm missing something, either about evaluating this type of data or just not using scipy correctly. Can anyone help me?&lt;/p&gt;&#xA;" OwnerUserId="15141" LastEditorUserId="11097" LastEditDate="2016-04-28T05:00:26.920" LastActivityDate="2016-04-28T11:35:25.127" Title="How would I chi-squared test these simple results from A/B experiment?" Tags="|scikit-learn|ab-test|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11470" PostTypeId="1" CreationDate="2016-04-28T02:04:01.797" Score="2" ViewCount="192" Body="&lt;p&gt;I am using &lt;a href=&quot;https://github.com/ibayer/fastFM&quot; rel=&quot;nofollow&quot;&gt;fastFM&lt;/a&gt; to run a factorization model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just wanted to know if there is an approach to interpreting the model?&#xA;How do I extract the Factorization Model equivalent of Variable Importance?&lt;/p&gt;&#xA;" OwnerUserId="12545" LastEditorUserId="11097" LastEditDate="2016-04-28T06:03:49.573" LastActivityDate="2016-04-28T06:03:49.573" Title="interpreting Factorization Machines" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11471" PostTypeId="2" ParentId="11467" CreationDate="2016-04-28T03:53:29.847" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Ordinary least squares (OLS)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ordinary least squares (OLS) is the workhorse of statistics. It gives a way of taking complicated outcomes and explaining behaviour (such as trends) using linearity. The simplest application of OLS is fitting a line.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Residuals&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Residuals are the observable errors from the estimated coefficients. In a sense, the residuals are estimates of the errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's explain the things using &lt;code&gt;R&lt;/code&gt; code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First fit a ordinary least square line of diamond datasets in &lt;code&gt;UsingR&lt;/code&gt; library:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(UsingR)&#xA;data(&quot;diamond&quot;)&#xA;y &amp;lt;- diamond$price&#xA;	x &amp;lt;- diamond$carat&#xA;n &amp;lt;- length(y)&#xA;olsline &amp;lt;- lm(y ~ x)&#xA;plot(x, y,&#xA;     main =&quot;Odinary Least square line&quot;,&#xA;     xlab = &quot;Mass (carats)&quot;, &#xA;     ylab = &quot;Price (SIN $)&quot;, &#xA;     bg = &quot;lightblue&quot;, &#xA;     col = &quot;black&quot;, cex = 2, pch = 21,frame = FALSE)&#xA;abline(olsline, lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/v5ShX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/v5ShX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, Let's calculate the residual i.e residual sum of squares:&#xA;   In &lt;code&gt;R&lt;/code&gt; you can easily calculate the residual as &lt;code&gt;resid(olsline)&lt;/code&gt;, for visualisation let's calculate it manually:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# The residuals from R method&#xA;e &amp;lt;- resid(olsline)&#xA;## Obtain the residuals manually, get the predicated Ys first&#xA;yhat &amp;lt;- predict(olsline)&#xA;# The residuals are y -yhat, Let's check by comparing this with R's build in resid function&#xA;ce &amp;lt;- y - yhat&#xA;max(abs(e-ce))&#xA;## Let's do it again hard coding the calculation of Yhat&#xA;max(abs(e- (y - coef(olsline)[1] - coef(olsline)[2] * x)))&#xA;# Residuals arethe signed length of the red lines&#xA;plot(diamond$carat, diamond$price,&#xA;    main =&quot;Residuals sum of (actual Y - predicted Y)^2&quot;,&#xA;     xlab = &quot;Mass (carats)&quot;, &#xA;     ylab = &quot;Price (SIN $)&quot;, &#xA;     bg = &quot;lightblue&quot;, &#xA;     col = &quot;black&quot;, cex = 2, pch = 21,frame = FALSE)&#xA;abline(olsline, lwd = 2)&#xA;for (i in 1 : n) &#xA;  lines(c(x[i], x[i]), c(y[i], yhat[i]), col = &quot;red&quot; , lwd = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iNabM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iNabM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope these visualization will clear your doubts between &lt;strong&gt;RSS&lt;/strong&gt; &amp;amp; &lt;strong&gt;OLS&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="17116" LastEditDate="2016-04-28T05:39:15.917" LastActivityDate="2016-04-28T05:39:15.917" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11475" PostTypeId="1" CreationDate="2016-04-28T07:27:26.020" Score="5" ViewCount="579" Body="&lt;p&gt;I have a multi label image dataset having 5 labels. Each image can have more than one label at the same time. I am using a convolutional neural network to extract features and those extracted features I am giving to &lt;code&gt;RepeatVector&lt;/code&gt; layer to create 5 copies of extracted features and after &lt;code&gt;RepeatVector&lt;/code&gt; layer I have connected &lt;code&gt;TimeDistributed&lt;/code&gt; layer with &lt;code&gt;TimeDistributed(Dense(2))&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;y_train&lt;/code&gt; is a 3D array and its shape is (1600, 5, 2) and &lt;code&gt;x_train&lt;/code&gt; is an array of images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; x_train.shape&#xA;(1600, 3, 100, 100)&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; y_train.shape &#xA;(1600,5,2)&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; y_train[0] = &#xA;array([[0, 1],   # [0,1] = 1 label present and  [1,0] = 0 label abset&#xA;       [1, 0],&#xA;       [1, 0],&#xA;       [1, 0],&#xA;       [1, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_label(y):&#xA;  tmp = []&#xA;  d = {0:[1,0],1:[0,1]}   # 0 absent 1 present&#xA;  for i,value in enumerate(y):&#xA;    tmp.append( d[value] )&#xA;  return tmp&#xA;&#xA;&#xA;X,Y= get_data()&#xA;Y = Y.tolist()&#xA;y = []&#xA;for value in Y:&#xA;  y.append(get_label(value))&#xA;&#xA;Y = np.array(y,dtype=int)&#xA;&#xA;&#xA;x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size =0.2,random_state=100)&#xA;&#xA;&#xA;img_channels = 3&#xA;img_rows     = 100&#xA;img_cols     = 100&#xA;nb_classes   = 5 &#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))&#xA;model.add(Activation('relu'))&#xA;model.add(Convolution2D(32, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;&#xA;model.add(Convolution2D(64, 3, 3, border_mode='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(Convolution2D(64, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;&#xA;model.add(Flatten())&#xA;model.add(Dense(512))&#xA;model.add(Activation('relu'))&#xA;model.add(RepeatVector(nb_classes))&#xA;model.add(TimeDistributed(Dense(2)))&#xA;model.add(Activation('softmax'))&#xA;&#xA;# let's train the model using SGD + momentum (how original).&#xA;# opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-06)&#xA;opt = SGD(lr=0.01, momentum=0.0, decay=1e-6, nesterov=False)&#xA;model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy'])&#xA;model.fit(x_train,y_train,nb_epoch=10,batch_size=32,validation_data=(x_test,y_test),shuffle=True)&#xA;out = model.predict_classes(x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But after training, I get all zeros for the test set. Is this approach wrong?&lt;/p&gt;&#xA;" OwnerUserId="17579" LastEditorUserId="17579" LastEditDate="2019-05-20T07:05:16.817" LastActivityDate="2019-06-19T08:01:35.170" Title="Solving multi label image classification using TimeDistributed dense layer" Tags="|machine-learning|deep-learning|keras|image-classification|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11476" PostTypeId="2" ParentId="10972" CreationDate="2016-04-28T09:42:18.657" Score="2" Body="&lt;p&gt;You are touching some issues that border simulation, namely &lt;a href=&quot;https://en.wikipedia.org/wiki/Discrete_event_simulation&quot; rel=&quot;nofollow&quot;&gt;Discrete Event Simulation&lt;/a&gt; (&lt;a href=&quot;http://simpy.readthedocs.io/en/latest/&quot; rel=&quot;nofollow&quot;&gt;Simpy&lt;/a&gt;, &lt;a href=&quot;http://www.simul8.com/&quot; rel=&quot;nofollow&quot;&gt;Simul8&lt;/a&gt;). For instance &lt;a href=&quot;https://en.wikipedia.org/wiki/Traffic_simulation&quot; rel=&quot;nofollow&quot;&gt;Traffic Simulation&lt;/a&gt; is mainly done using this, and others, techniques with the purpose of either testing the consequences of a change to the system, or to conceptualize the system altogether. Some commons areas you might like to consider exploring are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mobility scenarios (traffic, etc.),&lt;/li&gt;&#xA;&lt;li&gt;Optimization of mining or construction operations,&lt;/li&gt;&#xA;&lt;li&gt;Consumption scenarios (water, electricity, etc.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are all fields that can be broadly put inside &lt;a href=&quot;https://en.wikipedia.org/wiki/Operations_research&quot; rel=&quot;nofollow&quot;&gt;Operations Research&lt;/a&gt; (&lt;a href=&quot;https://wiki.python.org/moin/PythonForOperationsResearch&quot; rel=&quot;nofollow&quot;&gt;Python for OP&lt;/a&gt;). In the event that you might want to tackle issues related to this, the first thing you need to decide is &quot;What Problem are you trying to solve?&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I need to see if a buildings water distribution will hold the requirements (simulating water use scenarios against water/pressure availability).&lt;/li&gt;&#xA;&lt;li&gt;I need to optimize the transportation and mobility in a construction site.&lt;/li&gt;&#xA;&lt;li&gt;I need to conceptualize the emergency exits plan against possible evacuation scenarios.&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Than, what information do you need to build your models? &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;Data analysis&lt;/a&gt;, &lt;a href=&quot;http://scikit-image.org/&quot; rel=&quot;nofollow&quot;&gt;image analysis&lt;/a&gt;, &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;data management and manipulation&lt;/a&gt;. What are my variables? What are my scenarios? Can I predict phenomena (it's easier to predict water usage peaks than earthquakes, for example)? Can I create a model using those predictions?&lt;/p&gt;&#xA;" OwnerUserId="17806" LastActivityDate="2016-04-28T09:42:18.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11477" PostTypeId="2" ParentId="11469" CreationDate="2016-04-28T10:45:22.737" Score="2" Body="&lt;p&gt;If what you're trying to answer is if the action taken by a user (watch, interact, nothing) is influenced by the group they are in (A or B) you can use the chi2 test of independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For that you can use the scipy.stats.chi2_contingency function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [327445, 271602, 744702]&#xA;b = [376455, 140737, 818204]&#xA;&#xA;chi2, pvalue, _, _ = scipy.stats.chi2_contingency([a, b])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case it returns a chi2 test statistic of 48376.48 and a p-value of 0.0, so the null hypothesis (&quot;action is independent of group&quot;) is rejected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use the scipy.stats.chisquare function to arrive at the same results, but other than with the chi2_contingency function you'll have to calculate the &quot;expected frequencies&quot; yourself. The data you have recorded are your observed frequencies:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;obs = np.array([a, b]).astype(float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Note that I converted the numbers to float, as the chisquare function will run into some weird integer overflows otherwise...!?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expected frequencies are calculated like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;exp = np.outer(obs.sum(axis=1), obs.sum(axis=0)) / obs.sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, calling&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;chi2, pvalue = scipy.stats.chisquare(obs.ravel(), exp.ravel(), ddof=sum(obs.shape)-2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returns the same chi2 test statistic and p-value as before.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="676" LastEditDate="2016-04-28T11:35:25.127" LastActivityDate="2016-04-28T11:35:25.127" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11478" PostTypeId="1" AcceptedAnswerId="11499" CreationDate="2016-04-28T10:50:38.520" Score="3" ViewCount="2759" Body="&lt;p&gt;I had asked a question regarding predictive analysis for marketing earlier. &lt;a href=&quot;http://Prediction%20model%20for%20marketing%20to%20prospective%20customers%20(using%20pandas)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prediction model for marketing to prospective customers (using pandas)&lt;/a&gt; Still have some doubts about it, but I have a doubt regarding the decision tree that I generated for the marketing data.&lt;/p&gt;&#xA;&lt;p&gt;My aim is to predict if a lead will be won or lost, depending on how they were made aware of the product etc. I have a bool variable &amp;quot;Won&amp;quot;, 0-sale had failed, 1-sale was made. Using a decision tree, I was able to generate a model, however, there are no leaves for cases that lead to Not Won.&lt;/p&gt;&#xA;&lt;p&gt;Is this normal? Ive seen examples of the iris data set where all 3 features were represented in the tree, adn, hence am wondering if the approach Ive taken is correct.&lt;/p&gt;&#xA;&lt;p&gt;In the dataset of 38000, there are roughly 1700 who have Won=1.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6SZYA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6SZYA.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Im using pandas and my parameters for DecisionTreeClassifier are:&lt;/p&gt;&#xA;&lt;p&gt;min_samples_split=2&#xA;min_samples_leaf=1&#xA;max_depth=3&lt;/p&gt;&#xA;&lt;p&gt;I got then using grid search for parameter optimisation. I got a mean val of 0.95. If I use a bigger depth, the tree becomes too big to analyse.&lt;/p&gt;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Since Mark said to post some code, I am doing so&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import DecisionTreeClassifier,export_graphviz&#xA;dt=DecisionTreeClassifier(min_samples_split=2,min_samples_leaf=1,max_depth=3)&#xA;dt.fit(x,y)&#xA;features=x.columns&#xA;#print(features)&#xA;with open(&amp;quot;dt.dot&amp;quot;, 'w') as f:&#xA;    export_graphviz(dt, out_file=f,feature_names=features,class_names=[&amp;quot;Won&amp;quot;,&amp;quot;Lost&amp;quot;])&#xA;command = [&amp;quot;dot&amp;quot;, &amp;quot;-Tpng&amp;quot;, &amp;quot;dt.dot&amp;quot;, &amp;quot;-o&amp;quot;, &amp;quot;hello.png&amp;quot;]&#xA;try:&#xA;    subprocess.check_call(command)&#xA;except:&#xA;    exit(&amp;quot;Could not run dot, ie graphviz, to produce visualization&amp;quot;)&#xA;&#xA;from sklearn.cross_validation import cross_val_score&#xA;scores=cross_val_score(dt,x,y,cv=10)&#xA;print(scores.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This is the main training code, all previous lines are just munging&lt;/p&gt;&#xA;&lt;p&gt;The cross validation score come to 0.95&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pwPL4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pwPL4.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here is the csv snapshot&lt;/p&gt;&#xA;&lt;p&gt;I use all values from &amp;quot;Won&amp;quot; onwards and am training for &amp;quot;&amp;quot;Won&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;There was a single column X, which had many categorical values (20), &amp;quot;Won&amp;quot; being one of them.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Known' 'Recycled' 'Engaged' 'Prospect/MQL' 'Intern Transfer' 'MQL'&#xA; 'Working' 'Opportunity' nan 'Current Customer' 'Vendor' 'Disqualified'&#xA; 'Converted' 'SAL' 'SQL' 'engaged' 'working' 'Won' 'Web Registration'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;'Inactive&lt;/p&gt;&#xA;&lt;p&gt;However, they were all uniformly distributed ie. from 37000, all had almost the same number of observations. I used get_dummies to transform to numerical values, and dropped all the columns except &amp;quot;Won&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;All the rest of the values value for things like designation, opp(money value scaled from 1-3) and other categories, which are all boolean&lt;/p&gt;&#xA;" OwnerUserId="18098" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-04-29T07:56:00.567" Title="Decision Tree generating leaves for only one case" Tags="|data-mining|predictive-modeling|visualization|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11480" PostTypeId="2" ParentId="11390" CreationDate="2016-04-28T11:54:31.753" Score="20" Body="&lt;p&gt;Multiple papers have opined that&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;only in rare cases is there a known distribution of the error as a function of the number of features and sample size.&lt;/p&gt;&#xA;&lt;p&gt;The error surface for a given set of instances, and features, is a function of the correlation (or lack of) between features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://bioinformatics.oxfordjournals.org/content/21/8/1509.full&quot; rel=&quot;noreferrer&quot;&gt;This paper&lt;/a&gt; suggests the following:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For uncorrelated features, the optimal feature size is &lt;span class=&quot;math-container&quot;&gt;$ N-1 $&lt;/span&gt; (where &lt;span class=&quot;math-container&quot;&gt;$ N $&lt;/span&gt; is sample size)&lt;/li&gt;&#xA;&lt;li&gt;As feature correlation increases, and the optimal feature size becomes proportional to &lt;span class=&quot;math-container&quot;&gt;$ \sqrt N $&lt;/span&gt; for highly correlated features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Another (empirical) approach that could be taken, is to draw the learning curves for different sample sizes from the same dataset, and use that to predict classifier performance at different sample sizes. Here's the &lt;a href=&quot;http://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-12-8#CR31_460&quot; rel=&quot;noreferrer&quot;&gt;link to the paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12786" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-04-28T11:54:31.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11484" PostTypeId="1" AcceptedAnswerId="11492" CreationDate="2016-04-28T15:32:05.913" Score="2" ViewCount="1040" Body="&lt;p&gt;I try to use k-means clusters (using SQLserver + R), and I was wondering how we could estimate a distance correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if we consider Euclidean distance form the center of the clusters, what happen if we have, for the same dataset, clusters of different sizes ? a &quot;normal&quot; point in a big cluster will have a distance higher than an &quot;outlier&quot; point in a little one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it relevant to center / scale euclidean distance on each cluster ? (and then consider outliers as the ones with the highest scaled distance)&lt;/li&gt;&#xA;&lt;li&gt;Are there other kind of distance to consider ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18221" LastActivityDate="2016-04-28T22:30:46.640" Title="How to evaluate distance in k-means clusters?" Tags="|k-means|distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11485" PostTypeId="1" AcceptedAnswerId="11488" CreationDate="2016-04-28T15:44:44.497" Score="9" ViewCount="38708" Body="&lt;p&gt;I try to use k-means clusters (using SQLserver + R), and it seems that my model is not stable : each time I run the &lt;strong&gt;k-means&lt;/strong&gt; algorithm, it finds different clusters. But if I set &lt;strong&gt;nstart&lt;/strong&gt; (in R k-means function) high enough (10 or more) it becomes stable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The default value for this parameter is 1 but it &lt;em&gt;seems&lt;/em&gt; that setting it to a higher value (25) is recommended (I think I saw somewhere in the documentation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm a bit confused...&#xA;Any advice ?&lt;/p&gt;&#xA;" OwnerUserId="18221" LastActivityDate="2017-12-23T10:45:39.030" Title="k-means in R, usage of nstart parameter?" Tags="|r|k-means|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11487" PostTypeId="1" AcceptedAnswerId="11490" CreationDate="2016-04-28T16:13:53.623" Score="2" ViewCount="2097" Body="&lt;p&gt;I am trying to reproduce the results of &lt;a href=&quot;http://ieeexplore.ieee.org/xpl/articleDetails.jsp%3Freload%3Dtrue%26arnumber%3D6199099%26contentType%3DConference%2BPublications&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper, but using python and the HMMlearn library instead of matlab. The paper describes a procedure for using HMM (Hidden Markov Model) in order to predict stock prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper details the use of a 4-state, 5-mixture Gaussian distribution as the model. The transition probabilities and the initial state probabilities are uniform, however the emission probabilities are determined based on the results of a k-means algorithm using the data-set of existing stock prices. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This latter part is where I have gotten stuck, the paper advises to use the means, variance and weight of each cluster returned from the k-means algorithm as the mean, variance and weight of each component of the mixture. As I understand it the mean of the cluster is simply the center of each centroid, however I'm not sure how you would obtain the variance or the weight. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;&#xA;Given a 3 dimensional dataset X (in the form &lt;code&gt;[[a, b, c], [d, e, f]...]&lt;/code&gt;) and using the k-means algorithm where k = 5 (k = number of mixture components), how would I determine the weight and variance of each cluster?&lt;/p&gt;&#xA;" OwnerUserId="18271" LastActivityDate="2016-04-28T19:32:09.847" Title="How do I obtain the weight and variance of a k-means cluster?" Tags="|python|clustering|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11488" PostTypeId="2" ParentId="11485" CreationDate="2016-04-28T16:16:16.337" Score="4" Body="&lt;p&gt;Stability of the clusters is highly dependent on your dataset, for clear cut cases running it multiple times is a waste of resources. I think that is the rationale behind the default value of 1. But I agree that for most smaller cases setting it much higher makes a lot of sense.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-04-28T16:16:16.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11489" PostTypeId="1" AcceptedAnswerId="11494" CreationDate="2016-04-28T17:20:13.760" Score="11" ViewCount="3604" Body="&lt;p&gt;What kind of features you will try to extract from list of words for future predicting, is it existing word or just mess of characters ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is description of task that I found &lt;a href=&quot;https://github.com/hola/challenge_word_classifier&quot;&gt;there&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;You have to write a program that can answer whether a given word is English. This would be easy — you'd just need to look the word up in the dictionary — but there is an important restriction: your program must not be larger than 64 KiB.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I thought it would be possible to use logistic regression for solving the problem. I don't have a lot of experience with data-mining but the task is interesting for me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17971" LastEditorUserId="6478" LastEditDate="2016-04-29T15:35:54.633" LastActivityDate="2016-05-26T20:41:49.980" Title="How to determine if character sequence is English word or noise" Tags="|machine-learning|nlp|text-mining|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11490" PostTypeId="2" ParentId="11487" CreationDate="2016-04-28T19:32:09.847" Score="1" Body="&lt;p&gt;It is valid to use the k-means to initialize the EM for Mixture of Gaussian modeling. As you said, the mean of each component will be the average of all samples belong to the same cluster (it depends on the used clustering algorithm, some times the centroid is not the average of the cluster but is one of the samples). for the weight you can use the following: the weight of cluster x = the number of samples belong to cluster x divided by the total number of samples. thus, the cluster with the highest number of samples is the cluster with the highest weight. for the variance: just find the variance of all samples belong to the same cluster.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-04-28T19:32:09.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11491" PostTypeId="2" ParentId="11442" CreationDate="2016-04-28T22:15:12.753" Score="1" Body="&lt;p&gt;&quot;The New Science of Retailing: How Analytics are Transforming the Supply&quot;, Book by Ananth Raman and Marshall Fisher has some very good information of data science applied to Retail. Although, they don't address your case directly, they do have a chapter on store level execution that might help, and another chapter on assortment planning which might have relevant information. &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2016-04-28T22:15:12.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11492" PostTypeId="2" ParentId="11484" CreationDate="2016-04-28T22:30:46.640" Score="5" Body="&lt;p&gt;&lt;strong&gt;There are several important points to keep in mind in considering your questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You should always &lt;a href=&quot;http://www.dataminingblog.com/standardization-vs-normalization/&quot; rel=&quot;noreferrer&quot;&gt;normalize or standardize your data&lt;/a&gt; before applying k-means clustering. This is true of most other clustering algorithms also. If you are clustering in more than one dimension, the distance metric is meaningless unless each dimension has the same weight, so normalization is essential.  Imagine clustering people by body weight and income. Without normalization the results will depend on whether you think in pounds and dollars, kilograms and pesos, or moles and euros. The lack of normalization introduces non-determinism.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Strictly speaking, the stability of the k-means algorithm has been &lt;a href=&quot;https://papers.nips.cc/paper/989-convergence-properties-of-the-k-means-algorithms.pdf&quot; rel=&quot;noreferrer&quot;&gt;shown for Euclidean distance metrics&lt;/a&gt; and there is no assurance of convergence with other distance metrics.  More practically, most sensible metrics attain convergence and its not much of an issue, but its worth putting that warning out there.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;k-means isn't a clustering algorithm that readily lends itself to statistical analysis within the cluster. Every point in the space is a member of one of the k clusters regardless of how much of an outlier the point is. There are other clustering methods that are more adept at finding and ignoring outliers.  &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html&quot; rel=&quot;noreferrer&quot;&gt;DBSCAN&lt;/a&gt; is one such algorithm that is very good and finding clusters and ignoring noise.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, answering your questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it relevant to center / scale euclidean distance on each cluster ? (and then consider outliers as the ones with the highest scaled distance)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can certainly do this. Combining k-means with outlier detection is certainly possible but is probably not the most elegant or efficient algorithm.  It kind of sounds like poor-mans's DBSCAN.  Euclidean distance works fine, but just do a second set of normalizations using the centroids and the standard deviation of the cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there other kind of distance to consider?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are lots of other metrics that are useful for many different reasons.  As stated, the k-means convergence proofs hold only for Euclidean distance. For outlier detection Euclidean seems best, but there may be cases where Cosine Similarity metrics could be useful. People may suggest L1 (Manhattan) distance metrics, but I find this is only useful when there is significant linear dependence in your data, which can be remedied with dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give it a try as Euclidean should work fine, but also take a look at clustering via DBSCAN, which has outlier detection built into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-04-28T22:30:46.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11493" PostTypeId="2" ParentId="11410" CreationDate="2016-04-29T00:58:39.603" Score="1" Body="&lt;p&gt;The overal logarithmic shape of your f1 score graph indicates that learning is effective and cost is heading towards some minimum. That's good. I'm assuming that the noise you're referring to is the instability of the graph after approximately 3k iterations: cost dropping and rising in a zig-zag manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This often hints at the learning rate being too large. Back propagation finds the right gradient but you take too big step and end up climbing rather than descending along the edge of the cost function. It's especially evident when a graph seems to oscillate around some middle value. You haven't mentioned what exact value of learning rate you're using but try to reduce it. A good starting point would be 0.01 but it depends on many factors so try to experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another issue might be a batch size: that is, how many examples contribute to the calculation of a gradient. If it's too large you might end up with an average gradient pointing in the wrong direction. And then even a small step (i.e., low learning rate) won't help. And it might again manifest itself in a zig-zag pattern. If batch size is one of the parameters try to decrease it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The least likely issue might be the encoding architecture of your network. And especially the modest number of 8 neurons on the last layers. In this case individual neurons might have a considerable impact on the final output. And even little adjustments resulting from a single step of back propagation could potentially flip the sign of that neuron's activation value, impacting the results of other examples. Try increasing the number of neurons on the last layers. I'd personally suggest trying an architecture of 50x50x50.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18282" LastActivityDate="2016-04-29T00:58:39.603" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11494" PostTypeId="2" ParentId="11489" CreationDate="2016-04-29T03:09:46.390" Score="13" Body="&lt;p&gt;During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ngrams&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a random sample of words from &lt;em&gt;words.txt&lt;/em&gt;. For each word in sample, extract every possible &lt;a href=&quot;https://en.wikipedia.org/wiki/Bigram&quot; rel=&quot;nofollow noreferrer&quot;&gt;bi-gram&lt;/a&gt; of letters. For example, the word &lt;strong&gt;strength&lt;/strong&gt; consists of these bi-grams: {&lt;strong&gt;st&lt;/strong&gt;, &lt;strong&gt;tr&lt;/strong&gt;, &lt;strong&gt;re&lt;/strong&gt;, &lt;strong&gt;en&lt;/strong&gt;, &lt;strong&gt;ng&lt;/strong&gt;, &lt;strong&gt;gt&lt;/strong&gt;, &lt;strong&gt;th&lt;/strong&gt;}. Group by bi-gram and compute the frequency of each bi-gram in your corpus. Now do the same thing for tri-grams, ... all the way up to n-grams. At this point you have a rough idea of the frequency distribution of how Roman letters combine to create English words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ngram + word boundaries&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do a proper analysis you should probably create tags to indicate n-grams at the start and end of a word, (&lt;strong&gt;dog&lt;/strong&gt; -&gt; {&lt;strong&gt;^d&lt;/strong&gt;, &lt;strong&gt;do&lt;/strong&gt;, &lt;strong&gt;og&lt;/strong&gt;, &lt;strong&gt;g^&lt;/strong&gt;}) - this would allow you to capture phonological/orthographic constraints that might otherwise be missed (e.g., the sequence &lt;strong&gt;ng&lt;/strong&gt; can never occur at the beginning of a native English word, thus the sequence &lt;strong&gt;^ng&lt;/strong&gt; is not permissible - one of the reasons why Vietnamese names like &lt;strong&gt;Nguyễn&lt;/strong&gt; are hard to pronounce for English speakers). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Call this collection of grams the &lt;em&gt;word_set&lt;/em&gt;. If you reverse sort by frequency, your most frequent grams will be at the top of the list -- these will reflect the most common sequences across English words. Below I show some (ugly) code using package &lt;a href=&quot;https://cran.r-project.org/web/packages/ngram/ngram.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;{ngram}&lt;/a&gt; to extract the letter ngrams from words then compute the gram frequencies:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#' Return orthographic n-grams for word&#xA;#' @param w character vector of length 1&#xA;#' @param n integer type of n-gram&#xA;#' @return character vector&#xA;#' &#xA;getGrams &amp;lt;- function(w, n = 2) {&#xA;  require(ngram)&#xA;  (w &amp;lt;- gsub(&quot;(^[A-Za-z])&quot;, &quot;^\\1&quot;, w))&#xA;  (w &amp;lt;- gsub(&quot;([A-Za-z]$)&quot;, &quot;\\1^&quot;, w))&#xA;&#xA;&#xA;  # for ngram processing must add spaces between letters&#xA;  (ww &amp;lt;- gsub(&quot;([A-Za-z^'])&quot;, &quot;\\1 \\2&quot;, w))&#xA;  w &amp;lt;- gsub(&quot;[ ]$&quot;, &quot;&quot;, ww)&#xA;&#xA;  ng &amp;lt;- ngram(w, n = n)&#xA;  grams &amp;lt;- get.ngrams(ng)&#xA;  out_grams &amp;lt;- sapply(grams, function(gram){return(gsub(&quot; &quot;, &quot;&quot;, gram))}) #remove spaces&#xA;  return(out_grams)&#xA;}&#xA;&#xA;words &amp;lt;- list(&quot;dog&quot;, &quot;log&quot;, &quot;bog&quot;, &quot;frog&quot;)&#xA;res &amp;lt;- sapply(words, FUN = getGrams)&#xA;grams &amp;lt;- unlist(as.vector(res))&#xA;table(grams)&#xA;&#xA;## ^b ^d ^f ^l bo do fr g^ lo og ro &#xA;##  1  1  1  1  1  1  1  4  1  4  1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your program will just take an incoming sequence of characters as input, break it into grams as previously discussed and compare to list of top grams. Obviously you will have to &lt;strong&gt;reduce your top &lt;em&gt;n picks&lt;/em&gt; to fit the program size requirement&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;consonants &amp;amp; vowels&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible feature or approach would be to look at consonant vowel sequences. Basically convert all words in consonant vowel strings (e.g., &lt;strong&gt;pancake&lt;/strong&gt; -&gt; &lt;strong&gt;CVCCVCV&lt;/strong&gt;) and follow the same strategy previously discussed. This program could probably be much smaller but it would suffer from accuracy because it abstracts phones into high-order units.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;nchar&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another useful feature will be string length, as the possibility for legitimate English words decreases as the number of characters increases.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;library(ggplot2)&#xA;&#xA;file_name &amp;lt;- &quot;words.txt&quot;&#xA;df &amp;lt;- read.csv(file_name, header = FALSE, stringsAsFactors = FALSE)&#xA;names(df) &amp;lt;- c(&quot;word&quot;)&#xA;&#xA;df$nchar &amp;lt;- sapply(df$word, nchar)&#xA;grouped &amp;lt;- dplyr::group_by(df, nchar)&#xA;res &amp;lt;- dplyr::summarize(grouped, count = n())&#xA;qplot(res$nchar, res$count, geom=&quot;path&quot;, &#xA;      xlab = &quot;Number of characters&quot;, &#xA;      ylab = &quot;Frequency&quot;, &#xA;      main = &quot;Distribution of English word lengths in words.txt&quot;,&#xA;      col=I(&quot;red&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QpJDv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QpJDv.png&quot; alt=&quot;distribution of words lengths&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Error Analysis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The type of errors produced by this type of machine should be &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonsense_word&quot; rel=&quot;nofollow noreferrer&quot;&gt;nonsense&lt;/a&gt; words - words that look like they should be English words but which aren't (e.g., &lt;strong&gt;ghjrtg&lt;/strong&gt; would be correctly rejected (true negative) but &lt;strong&gt;barkle&lt;/strong&gt; would incorrectly classified as an English word (false positive)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, &lt;strong&gt;zyzzyvas&lt;/strong&gt; would be incorrectly rejected (false negative), because &lt;strong&gt;zyzzyvas&lt;/strong&gt; is a real English word (at least according to &lt;a href=&quot;https://raw.githubusercontent.com/hola/challenge_word_classifier/master/words.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;words.txt&lt;/a&gt;), but its gram sequences are extremely rare and thus not likely to contribute much discriminatory power.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-05-26T20:41:49.980" LastActivityDate="2016-05-26T20:41:49.980" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11495" PostTypeId="1" CreationDate="2016-04-29T04:19:50.950" Score="3" ViewCount="131" Body="&lt;p&gt;As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am thinking about, to what extend we should deal with the data quality issues? A robust algorithm should tolerate with the data quality issues but if we do better data cleansing work we get outcome confidently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how to balance this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somebody please give some ideas!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="16756" LastActivityDate="2016-06-19T23:47:59.063" Title="About data cleansing, to what extent should we do our work?" Tags="|data-cleaning|data|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11496" PostTypeId="1" CreationDate="2016-04-29T05:56:38.443" Score="1" ViewCount="537" Body="&lt;p&gt;I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = gensim.models.HdpModel(corpus, id2word=corpus.id2word, chunksize=50000)&#xA;&#xA;&#xA;&#xA; File &quot;/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py&quot;, line 210, in __init__&#xA;    self.update(corpus)&#xA;  File &quot;/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py&quot;, line 245, in update&#xA;    self.update_chunk(chunk)&#xA;  File &quot;/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py&quot;, line 313, in update_chunk&#xA;    self.update_lambda(ss, word_list, opt_o)&#xA;  File &quot;/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py&quot;, line 415, in update_lambda&#xA;    rhot * self.m_D * sstats.m_var_beta_ss / sstats.m_chunksize&#xA;MemoryError&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have loaded my corpus using following statement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;corpus = gensim.corpora.MalletCorpus('chunk5000K_records.mallet')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the data which I used to load corpus has 5 million records. And this is working for me when I am loading only 50K records. So I have added chunksize option HdpModel but it is still giving me an error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know how I can solve this issue. And I am running this on High Performance Computing so I think there should be a solution to resolve this issue as this cluster has really big size memory and disk capacity.&lt;/p&gt;&#xA;" OwnerUserId="18275" LastEditorUserId="11097" LastEditDate="2016-04-29T05:59:25.077" LastActivityDate="2023-05-17T12:07:15.813" Title="Memory error - Hierarchical Dirichlet Process, HDP gensim" Tags="|python|nlp|topic-model|lda|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11497" PostTypeId="2" ParentId="11495" CreationDate="2016-04-29T06:09:58.410" Score="1" Body="&lt;p&gt;This all depends on the needs and the budget for the models. The first cleaning steps usually bring a decent increase in performance. The more steps you take the slower the improvements will increase in general. If you are doing something for yourself, cut it off at a certain point, if you are doing it for somebody else, ask them what they want and how much they are willing to pay for it. It's comparable to the 80/20 rule where the first 20% of the total work will help with 80% of the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On another note, your statement about robust algorithms should tolerate with data qualities, you have to be careful with this. If there are biases in your data (missing values are not random for example) they will not learn properly, no matter how robust your algorithm is. Spending time using domain knowledge to fix this properly will help a lot.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-04-29T06:09:58.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11498" PostTypeId="2" ParentId="11495" CreationDate="2016-04-29T06:55:59.553" Score="0" Body="&lt;p&gt;I would say do as much cleaning and possible. The phrase &quot;Garbage in, garbage out&quot; is here for a reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Missing values, different definitions, suspicious information...all must be cleaned before fetched into any model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, even without all of this, the model will produce something, but it most likely will be less than optimal.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-04-29T06:55:59.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11499" PostTypeId="2" ParentId="11478" CreationDate="2016-04-29T07:15:09.100" Score="1" Body="&lt;p&gt;You are predicting binary class: 0 - sale failed, 1 - sale succeed. In the decision tree, the information &lt;code&gt;value&lt;/code&gt; is sorted. So for example in the first node, you have 35011 predictions of 0 and 1785 predictions of class 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, in your code you have this: &lt;code&gt;class_names=[&quot;Won&quot;,&quot;Lost&quot;])&lt;/code&gt;. So you are telling your decision tree that name of class 0 is &quot;Won&quot; and name of class 1 is &quot;Lost&quot;. I assume it should be inverted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in fact, the model always predict 0 - sale failed. Which seems reasonable as you said that out of 38000 samples, only 1700 are class 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, your dataset is highly unbalanced. So when cross validation gives accuracy of 95%, it says nothing, because only if your model gives all prediction 0, you will have accuracy of...95%.&#xA;For those cases, use models which can put weights to classes (SVM, Logistic regression or Random Forest classifier in scikit have this possibility). Think about undersample your dataset. Or add more of class 1 - dig in your database or do it artificially (SMOTE),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plot confusion matrix and give CV different score measure - precision or recall, depends on your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last thing, don't expect that your decision tree with depth of 3 will perform well. It is nice to show to business, but it gives poor prediction.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2016-04-29T07:56:00.567" LastActivityDate="2016-04-29T07:56:00.567" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11500" PostTypeId="2" ParentId="11404" CreationDate="2016-04-29T07:41:11.180" Score="5" Body="&lt;p&gt;&lt;a href=&quot;http://vcmi.inescporto.pt/reproducible_research/ijcnn2016/ClassImbalance/&quot; rel=&quot;nofollow&quot;&gt;This paper&lt;/a&gt; suggests using &lt;strong&gt;ranking&lt;/strong&gt; (I wrote it). Instead of using, for instance, SVM directly, you would use RankSVM. Since rankers compare observation against observation, training is necessarily balanced. There are two &quot;buts&quot; however: training is much slower, and, in the end, what these models do is rank your observations from how likely they are to belong to one class to how likely they are to belong to another so you need to apply a threshold afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are going to use &lt;strong&gt;pre-processing&lt;/strong&gt; to fix your imbalance I would suggest you look into &lt;a href=&quot;http://homes.cs.washington.edu/~pedrod/papers/kdd99.pdf&quot; rel=&quot;nofollow&quot;&gt;MetaCost&lt;/a&gt;. This algorithm involves building a bagging of models and then changing the class priors to make them balanced based on the hard to predict cases. It is very elegant. The cool thing about methods like SMOTE is that by fabricating new observations, you might making small datasets more robust.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, even though I wrote some things on class imbalance, I am still skeptic that it is an important problem in the real world. I would think it is very uncommon that you have imbalance priors in your training set, but balanced priors in your real world data. Do you? What usually happens is that type I errors are different than type II errors and I would bet most people would be better off using a cost matrix, which most training methods accept or you can apply it by pre-processing using MetaCost or SMOTE. I think many times &quot;fixing imbalance&quot; is short to &quot;I do not want to bother thinking about the relative trade-off between type I and II errors.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I tried for in-built python algorithms like Adaboost, GradientBoost&#xA;  techniques using sklearn. I read these algorithms are for handling&#xA;  imbalance class.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;AdaBoost gives better results for class imbalance when you initialize the weight distribution with imbalance in mind. I can dig the thesis where I read this if you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, of course, those methods won't give good accuracies. Do you have class imbalance in both your training and your validation dataset? You should use metrics such as F1 score, or pass a cost matrix to the accuracy function. &quot;Fixing&quot; class imbalance is when your priors are different in your training and your validation cases.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-04-29T13:44:40.623" LastActivityDate="2016-04-29T13:44:40.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11501" PostTypeId="2" ParentId="11456" CreationDate="2016-04-29T07:51:43.363" Score="1" Body="&lt;p&gt;I believe your problem boils down to clustering time-series of &lt;em&gt;different&lt;/em&gt; lengths. According to your question, you want the longer time-series of a power user to be considered similar to time-series of similar pattern but much shorter.&lt;br&gt;&#xA;Therefore you should look into clustering techniques and distance metrics which allow for these properties. I don't know your language of choice but here are some of the many packages in &lt;code&gt;R&lt;/code&gt; that you might find interesting :&lt;br&gt;&#xA;- &lt;a href=&quot;https://en.wikipedia.org/wiki/Fr%C3%A9chet_distance&quot; rel=&quot;nofollow&quot;&gt;Fréchet distance&lt;/a&gt; - one of the packages offering this is &lt;a href=&quot;https://mran.revolutionanalytics.com/web/packages/kmlShape/kmlShape.pdf&quot; rel=&quot;nofollow&quot;&gt;kmlShape&lt;/a&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow&quot;&gt;Dynamic Time Warping&lt;/a&gt; included in base &lt;code&gt;R&lt;/code&gt;&lt;br&gt;&#xA;- &lt;a href=&quot;http://pubman.mpdl.mpg.de/pubman/item/escidoc:2222505/component/escidoc:2222571/id10000557.pdf&quot; rel=&quot;nofollow&quot;&gt;Permutation Distribution Clustering&lt;/a&gt; - package &lt;a href=&quot;http://pubman.mpdl.mpg.de/pubman/item/escidoc:2222505/component/escidoc:2222571/id10000557.pdf&quot; rel=&quot;nofollow&quot;&gt;pdc&lt;/a&gt;&lt;br&gt;&#xA;This would also solve your data formatting problem as to setting values to &lt;code&gt;-1&lt;/code&gt; or &lt;code&gt;NULL&lt;/code&gt; would not be needed anymore. hth. &lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2016-04-29T07:51:43.363" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11502" PostTypeId="2" ParentId="11495" CreationDate="2016-04-29T12:58:43.030" Score="2" Body="&lt;p&gt;This is a question that is very dependent upon the data in question. Assuming that you can train your models in a reasonable time, I would start by not cleaning the data at all, and seeing how well your model performs, and then cleaning it a bit and redoing the experiment and so on.&#xA;This is because it is possible to &lt;strong&gt;overclean&lt;/strong&gt; your data, essentially removing variations in the data which are actually useful to your modelling.&#xA;The amount of cleaning that you apply is yet another meta-variable in your model.&lt;/p&gt;&#xA;" OwnerUserId="17509" LastActivityDate="2016-04-29T12:58:43.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11503" PostTypeId="1" CreationDate="2016-04-29T13:21:35.720" Score="1" ViewCount="1048" Body="&lt;p&gt;Imagine having a large database coming from a super market chain with panel data, whose cross-section variables are the different products sold in each super-market. The frequency is daily, that is you have one value for each product sold in each super-market per day. &#xA;Beyond their own history, you have the time-series of other variables e.g. dummy variables that signal the existence of offers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your objective is to predict the number of goods sold for each product in each super-market.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which functions would you use in R to make a week-ahead multivariate forecast?&lt;/strong&gt; &#xA;Which approach would you suggest?&lt;/p&gt;&#xA;" OwnerUserId="18295" LastActivityDate="2016-09-27T14:59:03.093" Title="Multivariate prediction model using R" Tags="|r|predictive-modeling|forecast|" AnswerCount="1" CommentCount="6" ClosedDate="2016-09-28T13:23:47.280" ContentLicense="CC BY-SA 3.0" />
  <row Id="11507" PostTypeId="2" ParentId="11466" CreationDate="2016-04-29T15:27:32.590" Score="2" Body="&lt;p&gt;You could try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(data.tree)&#xA;test$pathString &amp;lt;- with(test, paste(&quot;lm&quot;, Type, Treatment, round(res, 2), sep=&quot;/&quot;))&#xA;(tree &amp;lt;- as.Node(test))&#xA;#             levelName&#xA;# 1  lm                &#xA;# 2   ¦--Quebec        &#xA;# 3   ¦   ¦--nonchilled&#xA;# 4   ¦   ¦   °--36.97 &#xA;# 5   ¦   °--chilled   &#xA;# 6   ¦       °--30.11 &#xA;# 7   °--Mississippi   &#xA;# 8       ¦--nonchilled&#xA;# 9       ¦   °--24.31 &#xA;# 10      °--chilled   &#xA;# 11          °--17.45 &#xA;plot(tree)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The plot uses the &lt;code&gt;DiagrammeR&lt;/code&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oTjry.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oTjry.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-04-29T15:27:32.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11508" PostTypeId="2" ParentId="10329" CreationDate="2016-04-29T15:41:45.560" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] is it hopeless to get good results in reasonable time without a powerful computer/cluster/GPU?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's not hopeless and you can, without doubt, gain lots of relevant experience with deep learning using the computer spec you mentioned. It will come down to your neural network architecture (number of layers and neurons), size of the dataset (number of inputs), nature of the data (inherent patterns), and implementation. And although you may need to limit yourself with those regards it won't prevent you from acquiring intuition and knowledge you're referring to. You'll easily experience problems of overfitting, influence of regularization, effects of pre-training, impact of different neuron types and architectures to name a few.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll give you a more concrete example. I've implemented a couple of deep learning algorithms (all CPU-based) in Julia and run them on a MacBook Air (similar to your spec). The code was not terribly optimised as neurons and layers were represented by actual data structures rather than a single giant matrix. So further performance improvements were possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a fully-connected network of 56x300x300x300x1 (56 inputs and approx 200k connections) and 250 training examples I was able to get 5k back propagation passes within a day. Often that was enough to overfit the data or perfectly fit the training set (but this will depend on your dataset and other aforementioned factors). If the data has strong patterns and less than 10k examples you often won't need that many iterations. It's not uncommon that few hundreds of pre-training and refinement iterations lead to good results. So yes, your laptop is good enough and you could run meaningful experiments that take several hours.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] which deep architectures would you recommend that I try to implement with my hardware, such that the following goal is achieved: Acquiring intuition and knowledge about how and when to use techniques that were introduced in the past 10 years and were essential to the uprising of deep nets.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'd suggest to pick smaller datasets with strong patterns. And I'd recommend to look into pre-training techniques such as auto-encoders because they often require fewer iterations to reach better results. Start with back propagation and build from there, try different architectures, neuron types, use regularization, auto-encoders, dropout, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also make sure to pick a performant language or library for your experiments.&lt;/p&gt;&#xA;" OwnerUserId="18282" LastActivityDate="2016-04-29T15:41:45.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11509" PostTypeId="1" CreationDate="2016-04-29T15:43:44.527" Score="9" ViewCount="743" Body="&lt;p&gt;I'm reading about &lt;a href=&quot;https://en.wikipedia.org/wiki/Reservoir_computing&quot;&gt;reservoir computing&lt;/a&gt; techniques like &lt;a href=&quot;http://www.scholarpedia.org/article/Echo_state_network&quot;&gt;Echo State Networks&lt;/a&gt; and &lt;a href=&quot;http://www.igi.tugraz.at/psfiles/189.pdf&quot;&gt;Liquid State Machines&lt;/a&gt;. Both of the methods involve feeding inputs to a population of randomly (or not) connected spiking neurons, and a relatively simple readout algorithm that produces the output (e.g. linear regression). The neuron population weights are either fixed, or trained via a Hebbian-like local activity rule like &lt;a href=&quot;http://www.scholarpedia.org/article/Spike-timing_dependent_plasticity&quot;&gt;STDP&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These techniques perform well when modelling multi-dimensional inputs that have significant temporal components. However, computing the spiking neuron membrane potentials involves differential equation integration and can be computationally expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there any examples of where the additional computational complexity of reservoir computing techniques is outweighed by gains in a prediction or classification task?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, are there any cases of SNN techniques outperforming comparably complex architectures based on RNNs, ANNs, SVMs, DNNs, CNNs, or other algorithms?&lt;/p&gt;&#xA;" OwnerUserId="18303" LastActivityDate="2018-06-07T06:21:58.483" Title="Is there any domain where Spiking Neural Networks outperform other algorithms (non-spiking)?" Tags="|machine-learning|classification|neural-network|deep-learning|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11510" PostTypeId="1" CreationDate="2016-04-29T15:44:13.527" Score="-1" ViewCount="232" Body="&lt;p&gt;About ElecMart&lt;/p&gt;&#xA;&lt;p&gt;ElecMart, as the name suggests is a supermarket for Electronics. They serve the needs of both, retail clients and various corporate clients. Customers not only get to see and feel a wide range of products, they also receive exciting discounts and excellent customer service. ElecMart started in 1999 and launched a customer loyalty program in 2003.&#xA;ElecMart aims to be largest Electronic superstore across the nation, but they have a big hurdle ahead!&lt;/p&gt;&#xA;&lt;p&gt;The problem - Where are the recurring buyers?&lt;/p&gt;&#xA;&lt;p&gt;The loyalty program is meant for customers who want to take benefit from repeat purchases and register at the time of purchase. They need to present the loyalty card at Point of Sale at time of purchase and the benefits are non-transferrable. Also corporate sales automatically get the benefits of the loyalty program.&#xA;In a recent benchmarking activity and market survey which ElecMart sponsored, it was found that the &amp;quot;Repeat purchase rate&amp;quot; i.e. customer who come again for purchases from these customers is very low compared to other competitors. Increasing sales to these customers is the only way to run a successful loyalty program.&lt;/p&gt;&#xA;&lt;p&gt;Data&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Transaction_ID&lt;/strong&gt;  Unique ID of the transaction. Can not be used for modeling&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Transaction_Date&lt;/strong&gt;    Date on which the transaction took place&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Store_ID&lt;/strong&gt;    ID of the store in which the purchase was made&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Number_of_EMI&lt;/strong&gt;   Number of EMI opted by the customer&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Purchased_in_Sale&lt;/strong&gt;   Whether the transacted item was under offer&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Var1 - Var3&lt;/strong&gt; Masked categorical variable&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Client_ID&lt;/strong&gt;   Unique ID of the customer making the purchase.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Gende&lt;/strong&gt;r  Gender of the customer&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;DOB&lt;/strong&gt; Date of Birth of the customer&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Referred_Friend&lt;/strong&gt; Whether the customer referred a friend at time of purchase&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Sales_Executive_ID&lt;/strong&gt;  ID of the Sales executive who helped the customer with&lt;/p&gt;&#xA;&lt;p&gt;the purchase&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;SALES_Executive_Category&lt;/strong&gt;    Category of the sales executive who helped the&lt;/p&gt;&#xA;&lt;p&gt;customer with the purchase&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Lead_Source_Category&lt;/strong&gt;    Channel from which the customer came to know about&lt;/p&gt;&#xA;&lt;p&gt;the store and made the purchase (last source attribution)&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Payment_Mode&lt;/strong&gt;    Mode of the payment for the transaction&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Product_Category&lt;/strong&gt;    Category of the product purchased&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Transaction_Amoun&lt;/strong&gt;t  Amount of the transaction&lt;/p&gt;&#xA;&lt;p&gt;**If You are expected to identify the probabilty of the each customer (in the&lt;/p&gt;&#xA;&lt;p&gt;loyalty program) making a purchase in next 12 months.**&lt;/p&gt;&#xA;&lt;p&gt;Kindly Guide Me which model will be suitable here and what dependent variable i&lt;/p&gt;&#xA;&lt;p&gt;should use?&lt;/p&gt;&#xA;" OwnerUserId="18139" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-05-02T13:15:27.480" Title="Understanding Prediction Model" Tags="|predictive-modeling|" AnswerCount="2" CommentCount="0" ClosedDate="2016-05-02T20:13:14.597" ContentLicense="CC BY-SA 3.0" />
  <row Id="11511" PostTypeId="1" AcceptedAnswerId="11512" CreationDate="2016-04-29T18:02:04.490" Score="2" ViewCount="932" Body="&lt;p&gt;I'm training a random forest, trying to predict market shares of future stores on geographical areas. I have many features for these areas, some of which tell similar but different things about one thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I know the total number of $accommodations$ in the area, and I also have 5 others columns which are all linked in the following way: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$main \space accommodations + secondary \space accommodations + holiday \space accommodations = houses + flats = accommodations$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the feeling that including them all in my model would be wrong... but including them might be important... Any hint on how I should handle this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be a good idea to include $accommodations$ as absolute value and include all the other five but as % (of $accommodations$) and not as absolute values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a similar fashion, I also have the total number of $households$ of the area, the $total \space income$ of the area, and the $average \space income$ of households in the area (so that $households * average \space income = total \space income$). I have the feeling using the average and not the total income would be a better idea, but how can I be sure I'm right ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I guess I could train three random forests using the average income only, the total income only, and both, and see how they perform on cross validation, but is there a rule of thumb that I should know of which can make me go faster ?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In case it's relevant, I'm using &lt;code&gt;R&lt;/code&gt; and the &lt;code&gt;randomForest&lt;/code&gt; package)&lt;/p&gt;&#xA;" OwnerUserId="18306" LastEditorUserId="29575" LastEditDate="2017-12-14T12:36:53.383" LastActivityDate="2017-12-14T12:36:53.383" Title="In random forest, what happens if I add features that are correlated?" Tags="|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11512" PostTypeId="2" ParentId="11511" CreationDate="2016-04-29T20:12:04.063" Score="6" Body="&lt;p&gt;Random forests don't suffer from correlated variables like linear regression models do. Random forests randomly pick from a subset of variables at each split (hence the &quot;random&quot; in &quot;random forests&quot;). This means that correlated variables are less likely to show up together when the trees are being trained. But even when correlated variables show up in the same random subset of variables, it's still not much of an issue because the variables aren't assigned coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlated variables are mostly an issue for linear models that try to hold all other variables constant when calculating coefficients during training. The variable selection process is much simpler for trees and tree-based algorithms like random forests and gradient boosting. When a random forest is being trained and a tree's split is being evaluated, the algorithm will simply pick whichever feature most reduces error on that particular split of the tree. Once a variable is picked, there is no coefficient, just a greater-than/less-than split point, so the problem of &quot;exploding coefficients&quot; doesn't apply. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-04-29T20:12:04.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11513" PostTypeId="2" ParentId="11510" CreationDate="2016-04-29T20:57:12.670" Score="2" Body="&lt;p&gt;If all you're trying to do is identify the most frequent/recurring buyers, then the answer should be as simple as putting the data in a relational database (or if it's a lot of data, then Hadoop for Spark) and doing a simple SQL query to get the top ranked customers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;select Client_ID, count(*) as count from&#xA;    (select distinct Client_ID, Transaction_Date from table) as table2&#xA;group by Client_ID order by count desc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I suspect that what you're really asking is how to get more recurring customers, which is a much more difficult and open ended question to answer. One thing you could do is build an explanatory linear regression model to see which types of product categories are most associated with your frequent customers and then focus your efforts on optimizing customers' experience with those products. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, I don't think a predictive model will help you here. From the raw data it should be clear who your most frequent/recurring customers are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Your comment below clarified your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Split your data into two parts: a training dataset and a validation data set. Create a binary dependent variable that corresponds to whether someone has made a transaction 12 months after some pre-specified date. Feed multiple combinations of predictor variables into a gradient boosting model (GBM). Gradient boosting will capture interactions between your variables and will enable you to get a quick proof of concept working because most gradient boosting implementations can handle missing variables, so minimal data cleaning will be necessary. I noticed in your profile that you use R. R has a great predictive modeling package called &quot;caret&quot;: &lt;a href=&quot;http://topepo.github.io/caret/index.html&quot; rel=&quot;nofollow&quot;&gt;http://topepo.github.io/caret/index.html&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-05-01T15:22:20.230" LastActivityDate="2016-05-01T15:22:20.230" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11516" PostTypeId="1" CreationDate="2016-04-29T22:28:59.540" Score="1" ViewCount="561" Body="&lt;p&gt;I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Primary Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the reason behind this outcome?  &lt;/li&gt;&#xA;&lt;li&gt;Should I dump every single&#xA;feature into one model, then see which feature is important ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18227" LastEditorUserId="26596" LastEditDate="2016-12-06T22:34:02.157" LastActivityDate="2019-04-29T10:02:00.337" Title="xgboost performance with predicted values as input" Tags="|beginner|feature-extraction|xgboost|feature-engineering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11518" PostTypeId="1" CreationDate="2016-04-30T06:11:50.040" Score="2" ViewCount="2168" Body="&lt;p&gt;I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to find possible outliers in all these dimensions. I used Z-score to detect outliers in single dimension to find which high cost is causing outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a next step I tried to find outliers with high cost and high profit and low discount.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came up with a formula of :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Zscore(cost) + Zscore(profit) - Zscore(discount)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;negative sign because I want to find outliers with low discount.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this approach meaningful to do? or is there any further proven way to achieve this?&lt;/p&gt;&#xA;" OwnerUserId="18315" LastEditorUserId="5177" LastEditDate="2016-07-02T15:36:32.883" LastActivityDate="2016-07-02T15:36:32.883" Title="Finding outliers in multiple dimensions" Tags="|data-mining|bigdata|algorithms|data|outlier|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11521" PostTypeId="2" ParentId="11518" CreationDate="2016-04-30T08:48:35.877" Score="0" Body="&lt;p&gt;Because  your   data is   normal distributed ( gaussian) you could  easy  try to implement   in your desired    language  this   alghoritm  from  coursera  mooc :&#xA;&lt;a href=&quot;https://class.coursera.org/ml-005/lecture/91&quot; rel=&quot;nofollow&quot;&gt;https://class.coursera.org/ml-005/lecture/91&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18321" LastActivityDate="2016-04-30T08:48:35.877" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11522" PostTypeId="2" ParentId="11516" CreationDate="2016-04-30T08:52:37.353" Score="1" Body="&lt;p&gt;You could try   the next thing:,   get some   top  k predictions made with   xgboost, and use  only those as features  and   feed them  into  an LogisticRegression model.&#xA;I am currious if that will help  in case  you want  to  give a  feedback   on my proposed solution&lt;/p&gt;&#xA;" OwnerUserId="18321" LastActivityDate="2016-04-30T08:52:37.353" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11523" PostTypeId="2" ParentId="11503" CreationDate="2016-04-30T08:57:53.717" Score="0" Body="&lt;p&gt;You could try also  R  &lt;a href=&quot;https://cran.r-project.org/web/packages/h2o/h2o.pdf&quot; rel=&quot;nofollow&quot;&gt;h2o&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You  have implementations  for   random forest,   gbm , linear models and also   deep  learning .&#xA;THe  advantage is  that    h2o is pretty   fast  compared  to   other  implementations&lt;/p&gt;&#xA;" OwnerUserId="18321" LastActivityDate="2016-04-30T08:57:53.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11524" PostTypeId="1" CreationDate="2016-04-30T15:01:32.823" Score="3" ViewCount="561" Body="&lt;p&gt;I am not native English speaker and often times I use Ozdic to find the correct word choice. It's good overall but its database is quite limited. I think a tool based on Word2Vec should be able to do a great job for finding collocations/cooccurance. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With skip-grams, given a window size of n words around a word w, word2vec predicts contextual words c; i.e. in the notation of probability p(c|w). Conversely, CBOW predicts the current word, given the context in the window, p(w|c).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am looking for a tool that takes a sentence with a blank spot as input and gives alternative options for the blank spot as an output.&lt;/p&gt;&#xA;" OwnerUserId="17972" LastActivityDate="2017-09-08T22:16:04.973" Title="Is there any advanced collocation tool based on NLP+DL?" Tags="|nlp|word2vec|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11525" PostTypeId="2" ParentId="10931" CreationDate="2016-04-30T15:33:43.033" Score="1" Body="&lt;p&gt;You should try contacting cities about getting the sensor data from their Traffic Controller Systems. These usually have several detectors per lane at intersections (they also control the traffic lights so you get data on those as well). Some of these data are property of the companies that supplies these system, in which case you'll need to contact the company. These companies are generally not keen on sharing the data with commercial parties, but generally are interested in sharing it with educational parties as outreach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know one public dataset of this kind of data, the city of Utrecht in the Netherlands provides a sample of &lt;a href=&quot;https://utrecht.dataplatform.nl/dataset/verkeer-tellingen-verkeerslichten-2015&quot; rel=&quot;nofollow&quot;&gt;traffic density data&lt;/a&gt;. Unfortunately the metadata is mostly in Dutch.&lt;/p&gt;&#xA;" OwnerUserId="16199" LastActivityDate="2016-04-30T15:33:43.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11526" PostTypeId="2" ParentId="8731" CreationDate="2016-04-30T15:38:58.827" Score="0" Body="&lt;p&gt;There are two requirements for the model of behaviour that you should use: (1) &quot;more realistic&quot; and (2) orthogonal to your algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) By realistic, let us assume that it means that the behaviour should reflect behaviours observed in other, broader contexts than the specific context of Amazon purchases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Orthogonal is more straightforward to understand. The modelled behaviour should not be driven by similarities between products. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple approach to fulfil these two requirements would come from the fact that purchasing behaviours are driven by socio-demographic features such as gender, age, location (e.g. urban area / rural) and economic constraints (income and price).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a set of users and a set of products. You can estimate the relationship between the socio-demographic variables and the demand for product using simple but careful regression techniques. If needed, you could use external sources of data to make assumptions concerning important missing variables such as income. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then if you are a vendor, the socio-demographic model would predict which groups is most likely to purchase your product. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that helps :) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ben&lt;/p&gt;&#xA;" OwnerUserId="18325" LastActivityDate="2016-04-30T15:38:58.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11527" PostTypeId="2" ParentId="6378" CreationDate="2016-04-30T18:29:22.903" Score="3" Body="&lt;p&gt;I am going to answer your questions one after another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First, what do the 44 degrees of freedom mean?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It simply means that the model you built is constructed by using 44 independent variables. For example a model that looks like y = a&lt;em&gt;x + b has 1 independent variable (i.e. a) and thus 1 degree of freedom. A model that looks like y= a&lt;/em&gt;x1 + b*x2 + c would have 2 independent variables (i.e. a and b) and thus 2 degrees of freedom. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second, what is the Multiple R-squared?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here for the purpose of interpretating it Multiple R-squared is equivalent to the (simple) R-squared you would have for a linear regression model with 1 degree of freedom. Multiple R-squared tells us the share of the observed variance that is explained by the model. For example if you have a Multiple R-squared of 0.79 it means that your model explains 79% of the observed variance in your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Third, what is the Adjusted R-squared and why do we need it?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several problems with Multiple R-squared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 1:&lt;/strong&gt; Every time you add a predictor to a model, the R-squared increases, even if due to chance alone. It never decreases. Consequently, a model with more independent variables (more degrees of freedom) may appear to have a better fit simply because it has more independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 2:&lt;/strong&gt; If a model has too many predictors and higher order polynomials, it begins to model the random noise in the data. This condition is known as overfitting the model and misleadingly high R-squared values and a lessened ability to make predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem 1 is caused by Problem 2. And this is where Adjusted R-squared is coming in handy. Adjusted R-squared is an attempt at fixing these problems by factoring in the number of independent variables. The adjusted R-squared tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oyBLe.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oyBLe.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;n is the number of data points you have,  &lt;/li&gt;&#xA;&lt;li&gt;and k is the number of independent variables used to explain their distribution, excluding the constant&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you add more and more useless variables to a model, adjusted r-squared will decrease. If you add more useful variables, adjusted r-squared will increase.&#xA;Adjusted R-squared  will always be less than or equal to R-squared . You only need R-squared when working with samples. In other words, R-squared  isn’t necessary when you have data from an entire population.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is an interesting series of articles which will help you understand how to use R-squared to interpret the results of your model even better. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.minitab.com/blog/adventures-in-statistics/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit&quot; rel=&quot;nofollow noreferrer&quot;&gt;Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit?&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.minitab.com/blog/adventures-in-statistics/multiple-regession-analysis-use-adjusted-r-squared-and-predicted-r-squared-to-include-the-correct-number-of-variables&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multiple Regression Analysis: Use Adjusted R-Squared and Predicted R-Squared to Include the Correct Number of Variables&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18328" LastEditorUserId="18328" LastEditDate="2016-04-30T18:46:48.067" LastActivityDate="2016-04-30T18:46:48.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11528" PostTypeId="2" ParentId="11509" CreationDate="2016-05-01T03:38:15.790" Score="9" Body="&lt;p&gt;My answer comes from experience more than from experiments or benchmarks published. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, Spiking Neural Networks do not outperform other algorithms in any task. There have been advances in robotics and reservoir computing but  reservoir computing algorithms are as good as other algorithms (like reinforcement learning) according to recent publications.  There are rumours that some companies are interested in these algorithms because they've hired a few reservoir computing researchers recently but these are only rumours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is one of the most recent publications detailing advances and limitations of reservoir computing in robotics&#xA;&lt;a href=&quot;https://www.cs.unm.edu/~afaust/MLPC15_proceedings/MLPC15_paper_Polydoros.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://www.cs.unm.edu/~afaust/MLPC15_proceedings/MLPC15_paper_Polydoros.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I began experimenting with Liquid State Machines in college using Wolfgang Maass' proposed architecture. It looked promising, specially the idea of inhibitory neurons forming part of the circuit. But in reality using these algorithms in real life data applications (language classification, image classification among others) was not enough to get close to the benchmarks like RNNs, ANNs, SVMs. Sometimes even vanilla Multilayer Neural Networks perform better than Liquid State Machines. My understanding is that these kind of models are good for robotics and other autonomous related tasks like sensors and autonomous navigation (but that was not my area of research) but not so much for other types of data. There are a few labs, mainly in Europe working with this algorithm but so far I haven't heard of many advances in this area in the past years. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do believe that brain inspired algorithms are the next big step in AI, and while many companies like Numenta and Deepmind are researching towards this direction, as of today there is still a lot of work to be done in order to have the next breakthrough in AI.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-05-12T01:52:19.123" LastActivityDate="2016-05-12T01:52:19.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11530" PostTypeId="2" ParentId="8731" CreationDate="2016-05-01T14:31:39.883" Score="1" Body="&lt;p&gt;&lt;strong&gt;How will you validate your algorithm?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than trying to answer the second question, consider that your answer to the first question might need revision...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methods have you used for validating learning methods throughout your data science class?  First you want to define a &lt;strong&gt;specific set&lt;/strong&gt; of numerical metrics to assess the success or failure of your model. Second, what methods can you use to create a &lt;strong&gt;very&lt;/strong&gt; realistic testing population (more realistic than modeling the population)?  The first hint I will give is that the &lt;a href=&quot;http://jmcauley.ucsd.edu/data/amazon/&quot; rel=&quot;nofollow&quot;&gt;Amazon Reviews Dataset&lt;/a&gt; is very large, so your data is very amendable to this method.  The second hint I will give is that this method is likely the one that you have used in 95% of the supervised learning problems that you have worked on in class...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps... I will edit this if need be based on comments added by the OP, but don't want to provide the solution right away in order to elicit some organic thought e.g. since this is a class problem and the Prof is also trying to help you come up with the right solution on your own.&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-05-01T14:31:39.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11531" PostTypeId="1" CreationDate="2016-05-01T14:57:08.907" Score="2" ViewCount="601" Body="&lt;p&gt;I have a problem with using neurolab python library: I'm trying to predict some time-series with help of Elman recurrent neural network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import neurolab as nl&#xA;import numpy as np&#xA;&#xA;# Create train samples&#xA;# x = np.linspace(-7, 7, 20)&#xA;x = [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 4, 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5, 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9, 6, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, 7, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9, 9, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9, 10, 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 11, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8, 11.9, 12, 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 12.7, 12.8, 12.9, 13, 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7, 13.8, 13.9, 14, 14.1, 14.2, 14.3, 14.4, 14.5, 14.6, 14.7, 14.8, 14.9, 15, 15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7, 15.8, 15.9, 16, 16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 16.7, 16.8, 16.9, 17, 17.1, 17.2, 17.3, 17.4, 17.5, 17.6, 17.7, 17.8, 17.9, 18, 18.1, 18.2, 18.3, 18.4, 18.5, 18.6, 18.7, 18.8, 18.9, 19, 19.1, 19.2, 19.3, 19.4, 19.5, 19.6, 19.7, 19.8, 19.9]&#xA;x=np.asarray(x)&#xA;y = [0.000, 0.296, 0.407, 0.488, 0.552, 0.607, 0.655, 0.697, 0.734, 0.769, 0.800, 0.829, 0.855, 0.880, 0.903, 0.925, 0.945, 0.964, 0.982, 0.998, 1.014, 1.029, 1.043, 1.057, 1.069, 1.081, 1.092, 1.103, 1.113, 1.123, 1.132, 1.141, 1.149, 1.157, 1.164, 1.171, 1.177, 1.184, 1.189, 1.195, 1.200, 1.205, 1.209, 1.214, 1.218, 1.221, 1.225, 1.228, 1.231, 1.234, 1.236, 1.238, 1.240, 1.242, 1.244, 1.245, 1.246, 1.247, 1.248, 1.249, 1.249, 1.250, 1.250, 1.250, 1.250, 1.250, 1.249, 1.248, 1.248, 1.247, 1.246, 1.245, 1.243, 1.242, 1.240, 1.239, 1.237, 1.235, 1.233, 1.231, 1.228, 1.226, 1.224, 1.221, 1.218, 1.215, 1.213, 1.210, 1.206, 1.203, 1.200, 1.197, 1.193, 1.190, 1.186, 1.182, 1.178, 1.174, 1.170, 1.166, 1.162, 1.158, 1.154, 1.149, 1.145, 1.140, 1.136, 1.131, 1.126, 1.122, 1.117, 1.112, 1.107, 1.102, 1.096, 1.091, 1.086, 1.081, 1.075, 1.070, 1.064, 1.059, 1.053, 1.047, 1.041, 1.036, 1.030, 1.024, 1.018, 1.012, 1.006, 0.999, 0.993, 0.987, 0.981, 0.974, 0.968, 0.961, 0.955, 0.948, 0.942, 0.935, 0.928, 0.922, 0.915, 0.908, 0.901, 0.894, 0.887, 0.880, 0.873, 0.866, 0.859, 0.852, 0.844, 0.837, 0.830, 0.822, 0.815, 0.807, 0.800, 0.792, 0.785, 0.777, 0.770, 0.762, 0.754, 0.747, 0.739, 0.731, 0.723, 0.715, 0.707, 0.699, 0.691, 0.683, 0.675, 0.667, 0.659, 0.651, 0.643, 0.634, 0.626, 0.618, 0.610, 0.601, 0.593, 0.584, 0.576, 0.567, 0.559, 0.550, 0.542, 0.533, 0.525, 0.516, 0.507, 0.498, 0.490, 0.481]&#xA;y=np.asarray(y)&#xA;sample = [20, 20.1, 20.2, 20.3, 20.4, 20.5, 20.6, 20.7, 20.8, 20.9, 21, 21.1, 21.2, 21.3, 21.4]&#xA;sample=np.asarray(sample)&#xA;&#xA;size = len(x)&#xA;&#xA;inp = x.reshape(size,1)&#xA;tar = y.reshape(size,1)&#xA;smp = sample.reshape(len(sample),1)&#xA;#print(inp)&#xA;print(tar)&#xA;# Create network with 2 layers and random initialized&#xA;#net = nl.net.newelm([[min(x), max(y)]],[5, 1]) # neurolab.net.newff(minmax, size, transf=None)&#xA;net = nl.net.newelm([[min(x), max(y)]], [16, 1], [nl.trans.TanSig(), nl.trans.PureLin()])&#xA;# Set initialized functions and init&#xA;net.layers[0].initf = nl.init.InitRand([-0.1, 0.1], 'wb')&#xA;net.layers[1].initf = nl.init.InitRand([-0.1, 0.1], 'wb')&#xA;net.init()&#xA;# Train network&#xA;error = net.train(inp, tar, epochs=1900, show=100, goal=0.0001)&#xA;&#xA;# Simulate network&#xA;out = net.sim(smp)&#xA;print(out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It works fine with only one input time series (input vector). But I need more than one, in fact, I do need five input vectors.&#xA;Example:&#xA;I'm going to predict 6 rows of &quot;to_be_predicted&quot; column. The data: &lt;a href=&quot;http://pastebin.com/7z1DeikJ&quot; rel=&quot;nofollow&quot;&gt;pastebin.com/7z1DeikJ&lt;/a&gt;. So columns &quot;usd&quot;, &quot;euro&quot;, &quot;GDP_bln&quot;, &quot;inflation&quot;, &quot;CPI&quot; are the inputs and &quot;to_be_predicted&quot; is a target in my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody know how to solve this issue? Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="18338" LastEditorUserId="18338" LastEditDate="2016-05-01T15:02:57.187" LastActivityDate="2016-05-01T15:02:57.187" Title="Is there any ability to use two ore more inputs for Elman recurrent neural network?" Tags="|python|neural-network|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11532" PostTypeId="2" ParentId="10108" CreationDate="2016-05-01T18:05:00.620" Score="0" Body="&lt;p&gt;I had similar problem for my business: parsing tons of mails with different formatting and some of them slightly changing over the time. I did not solve it, but I use the service from &lt;a href=&quot;http://www.scriptminer.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.scriptminer.com/&lt;/a&gt; which is apparently using machine learning tools to parse these sort of semi-structured messages.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2016-05-01T18:05:00.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11534" PostTypeId="1" CreationDate="2016-05-01T19:06:15.857" Score="1" ViewCount="52" Body="&lt;p&gt;I'm learning R and trying to do a comparison of MLB player salaries vs. WAR&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One dataset contains all the salaries and the other contains WAR information, and they have their variables labelled differently. For example, each of the team names in one set is an abbreviation (ATL, NYY, HOU vs. Braves, Yankees, Astros). With that example, I'm sure I could find a way to make ATL == Braves, but with all the players that doesn't seem so easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The players are labelled in one dataset as the first 5 letters of the last name, first two letters of the first name, and then two numbers to differentiate matches. The other set simply has the names. An example would be Bryce Harper vs. harpebr03.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's no apparent way to sort via alphabetical order and matching up each player's variable ID would take way too much time. Is there an easier way to link these differently named variables in R?&lt;/p&gt;&#xA;" OwnerUserId="18344" LastActivityDate="2016-05-02T03:51:34.273" Title="Combining data sets with different IDs?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11535" PostTypeId="1" CreationDate="2016-05-01T21:27:28.867" Score="1" ViewCount="44" Body="&lt;p&gt;Below is a list of labs around the country that offer genetic sequencing.  What would be the reasons why they each cost a different amount if the end result is that you get a 90GB hard drive with 3+ billion base pairs for your genome?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.scienceexchange.com/services/whole-genome-seq&quot; rel=&quot;nofollow&quot;&gt;https://www.scienceexchange.com/services/whole-genome-seq&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18349" LastEditorUserId="13413" LastEditDate="2016-06-06T08:12:09.600" LastActivityDate="2016-06-06T08:12:09.600" Title="Why is there such a wide range of prices for whole genome sequencing services?" Tags="|genetic|bioinformatics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11536" PostTypeId="1" CreationDate="2016-05-02T00:12:52.263" Score="1" ViewCount="584" Body="&lt;p&gt;I'm trying to predict the birth weight baby using polynomial regression model. First what I need know what degree polynomial should fit better to my data.&#xA;In order to do that I split my dataset in training set (70%) and Cross Validation set(30%) and then plots each error by degree polynomial. I did run my script 4 times selecting randomly the data but I get so different curves  each time as you can see&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HQQJm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HQQJm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know why this happens or What am I doing wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see my code below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;main script&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%========== Begin - constants declaration ==========% &#xA;x_training_percent = 0.7;&#xA;cv_set_percent = 0.3;&#xA;%========== End - constants declaration ==========% &#xA;&#xA;load('data/dataset.m');&#xA;[m, n] = size(X);% m: Number of examples, n: Number of features.&#xA;&#xA;%========== Begin - Getting traingin and CV sets ==========% &#xA;training_set_size = round(m * x_training_percent);&#xA;cv_set_size = round(m * cv_set_percent);&#xA;test_set_size = round(m * test_set_percent);&#xA;&#xA;random_order_idx = randperm(m);&#xA;&#xA;indexes = random_order_idx(1:training_set_size);&#xA;random_order_idx(1:training_set_size) = []; % Remove indexes were used &#xA;x_training_o = X(indexes, :); % X original training Set&#xA;y_training = y(indexes, :);   % y original training Set&#xA;&#xA;indexes = random_order_idx(1:cv_set_size);&#xA;random_order_idx(1:cv_set_size) = []; % Remove indexes were used &#xA;x_cv_o = X(indexes, :); % X original Cross Validation Set&#xA;y_cv = y(indexes, :);   % y original Cross Validation Set&#xA;&#xA;%========== End - Getting traingin and CV sets ==========% &#xA;&#xA;max_p = 10; % Max degree polynomial&#xA;&#xA;cv_error = zeros(max_p, 1);&#xA;training_error = zeros(max_p, 1);&#xA;&#xA;&#xA;for p = 1:max_p&#xA;&#xA;  % Processing training set&#xA;  x_training = x_training_o;&#xA;  x_training = polyFeatures(x_training, p); % Adding polynomial terms from 1 to p&#xA;  [x_training, mu, sigma] = featureNormalize(x_training);&#xA;  x_training = [ones(training_set_size, 1) x_training];&#xA;&#xA;  % Processing cross validation set&#xA;  x_cv = x_cv_o;&#xA;  x_cv = polyFeatures(x_cv, p); % Adding polynomial terms from 1 to p&#xA;  x_cv = bsxfun(@minus, x_cv, mu);&#xA;  x_cv = bsxfun(@rdivide, x_cv, sigma);&#xA;  x_cv = [ones(cv_set_size, 1) x_cv];&#xA;&#xA;  %========== Begin - Training ==========% &#xA;  lambda = 0&#xA;  theta = trainLinearReg(x_training, y_training, lambda);&#xA;  %========== End - Training ==========% &#xA;&#xA;  %========== Begin - Computing prediction errors with polinomial degree p ==========% &#xA;    predictions = x_training * theta; % Predictions with training set&#xA;    training_error(p, :) = (1 / (2 * training_set_size)) * sum((predictions - y_training) .^ 2);&#xA;&#xA;    cv_predictions = x_cv * theta; % Predictions with cross validation set&#xA;    cv_error(p, :) = (1 / (2 * cv_set_size)) * sum((cv_predictions - y_cv) .^ 2);&#xA;  %========== End - Computing prediction errors ==========% &#xA;&#xA;end&#xA;&#xA;&#xA;plot(1:p, training_error, 1:p, cv_error);&#xA;&#xA;title(sprintf('Learning curve for linear regression with lambda = %f', lambda));&#xA;legend('Train', 'Cross Validation')&#xA;xlabel('degree of Polynomial')&#xA;ylabel('Error')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;polyFeatures&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [X_poly] = polyFeatures(X, p)&#xA;%POLYFEATURES Maps X (1D vector) into the p-th power&#xA;%   [X_poly] = POLYFEATURES(X, p) takes a data matrix X (size m x 1) and&#xA;%   maps each example into its polynomial features where&#xA;%   X_poly(i, :) = [X(i) X(i).^2 X(i).^3 ...  X(i).^p];&#xA;%&#xA;&#xA;X_poly = X; % For p = 1&#xA;&#xA;for i = 2:p&#xA;    X_poly = [X_poly (X .^ i)];&#xA;end&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;featureNormalize&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [X_norm, mu, sigma] = featureNormalize(X)&#xA;%FEATURENORMALIZE Normalizes the features in X &#xA;%   FEATURENORMALIZE(X) returns a normalized version of X where&#xA;%   the mean value of each feature is 0 and the standard deviation&#xA;%   is 1. This is often a good preprocessing step to do when&#xA;%   working with learning algorithms.&#xA;&#xA;mu = mean(X);&#xA;X_norm = bsxfun(@minus, X, mu);&#xA;&#xA;sigma = std(X_norm);&#xA;X_norm = bsxfun(@rdivide, X_norm, sigma);&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;trainingLinearReg&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [theta] = trainLinearReg(X, y, lambda)&#xA;%TRAINLINEARREG Trains linear regression given a dataset (X, y) and a&#xA;%regularization parameter lambda&#xA;%   [theta] = TRAINLINEARREG (X, y, lambda) trains linear regression using&#xA;%   the dataset (X, y) and regularization parameter lambda. Returns the&#xA;%   trained parameters theta.&#xA;%&#xA;&#xA;% Initialize Theta&#xA;initial_theta = zeros(size(X, 2), 1); &#xA;&#xA;% Create &quot;short hand&quot; for the cost function to be minimized&#xA;costFunction = @(t) linearRegCostFunction(X, y, t, lambda);&#xA;&#xA;% Now, costFunction is a function that takes in only one argument&#xA;options = optimset('MaxIter', 400, 'GradObj', 'on');&#xA;&#xA;% Minimize using fmincg&#xA;theta = fmincg(costFunction, initial_theta, options);&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;linearRegCostFuncion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function [J, grad] = linearRegCostFunction(X, y, theta, lambda)&#xA;%LINEARREGCOSTFUNCTION Compute cost and gradient for regularized linear &#xA;%regression with multiple variables&#xA;%   [J, grad] = LINEARREGCOSTFUNCTION(X, y, theta, lambda) computes the &#xA;%   cost of using theta as the parameter for linear regression to fit the &#xA;%   data points in X and y. Returns the cost in J and the gradient in grad&#xA;&#xA;% Initialize some useful values&#xA;m = length(y); % number of training examples&#xA;&#xA;% You need to return the following variables correctly &#xA;J = 0;&#xA;grad = zeros(size(theta));&#xA;&#xA;hx = X * theta; % Prediction&#xA;&#xA;J = (1 / (2 * m)) * sum((hx - y) .^ 2) + (lambda / (2 * m)) * sum([ 0; theta(2:end, :) ] .^ 2);&#xA;&#xA;grad = (1 / m) * (hx - y)' * X + (lambda / m) * [ 0; theta(2:end, :) ]';&#xA;&#xA;grad = grad(:);&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me?&lt;/p&gt;&#xA;" OwnerUserId="18353" LastEditorUserId="18353" LastEditDate="2016-05-03T02:45:35.543" LastActivityDate="2024-01-28T01:03:40.423" Title="Trying determining degree polynomial for polynomical regression" Tags="|regression|linear-regression|octave|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11537" PostTypeId="2" ParentId="9406" CreationDate="2016-05-02T00:40:11.000" Score="0" Body="&lt;p&gt;If you slightly extend the question to include general gradient boosting techniques (in contrast with the special case of boosted regression trees), then the answer is yes. Gradient boosting has been successfully used as an alternative for variable selection. &#xA;A good example is &lt;a href=&quot;https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;mboost package&lt;/a&gt;. The key is that the class of base learners used for boosting consists of continuous models to start with.&#xA;This &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; describes typical classes of base learners as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The commonly used base-learner models can be classified into&#xA;  a three distinct categories: linear models, smooth models and&#xA;  decision trees. There is also a number of other models, such as&#xA;  markov random fields (Dietterich et al., 2004) or wavelets (Viola&#xA;  and Jones, 2001), but their application arises for relatively specific&#xA;  practical tasks.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Note that it particularly mentions wavelets. Trees and wavelets has been successfully combined before into &lt;a href=&quot;https://arxiv.org/pdf/1011.4615.pdf&quot; rel=&quot;nofollow&quot;&gt;tree-based wavelets.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18077" LastEditorUserId="18077" LastEditDate="2016-05-02T12:34:05.413" LastActivityDate="2016-05-02T12:34:05.413" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11538" PostTypeId="1" CreationDate="2016-05-02T01:20:11.540" Score="1" ViewCount="33" Body="&lt;p&gt;Hi i'd like to know a bit more about kNN-like approach implementations for classification problems, and specifically classification problems where we want to have a probability distribution as an output (to compute logloss like metrics for example)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In packages such as knn for R, a distance matrix is asked as an input. but thats highly inefficient for even medium sized data sets, as such matrix size is n^2 where n is the number of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but i can see an easy (empirical) approach where we hash all data into hypercubes, and return for the test set, the implied observed distribution of the train data in the corresponding hypercube it hashes into.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, such an approach has many flaws - it will suffer the curse of dimensionality, but might it not work in there is a lot of data, but not many variables (2,3,4,5 ?) for example ?? - if the data density is heterogeneous, there would be some hypercubes with little data, and others a lot of data. one might then like to zoom on some areas which are overpopulated....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the advantage of hashing is that its computationally efficient and more like linear time/memory than quadratic time/memory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think i'm discovering the moon, so what do you guys think about it ? if there already some algos/libraries basically doing that or much better, which are those ?&lt;/p&gt;&#xA;" OwnerUserId="18354" LastActivityDate="2016-05-02T01:20:11.540" Title="Nearest Neighbours variant approach" Tags="|machine-learning|r|classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11539" PostTypeId="2" ParentId="11534" CreationDate="2016-05-02T03:51:34.273" Score="2" Body="&lt;p&gt;Here is some code that deals with the second part of your question regarding matching names. (BTW including sample data always helps)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(dplyr)  ## for join&#xA;&#xA;# create two example dataframes &#xA;d1 &amp;lt;- data.frame(player_first_last=c(&quot;Joe Smith&quot;, &quot;Jack Johnson&quot;, &quot;A Jones&quot;, &quot;Slow Mo&quot;), d1_value=c(1,2,3,4))&#xA;d2 &amp;lt;- data.frame(player_last_first2=c(&quot;SmithJo&quot;,&quot;JohnsJa&quot;, &quot;MoSl&quot;), d2_value=c(11,12,13))&#xA;&#xA;# first two letters of first name using substr, and trimming whitespace&#xA;d1$player_first2 &amp;lt;- trimws(substr(d1$player_first, 1, 2 ))&#xA;&#xA;# last name - regex capturing everything after the first whitespace&#xA;# this is a &quot;regular expressions&quot; &#xA;d1$player_last &amp;lt;- gsub(&quot;.+\\s(.+)&quot;, &quot;\\1&quot;, d1$player_first_last)&#xA;&#xA;# concatenate last 5 and first 2 using paste0&#xA;d1$player_last_first2 &amp;lt;- paste0(substr(d1$player_last,1,5), d1$player_first2)&#xA;&#xA;# join on constructed name -- full_join in case there are differences&#xA;d3 &amp;lt;- full_join(d1, d2, by=&quot;player_last_first2&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results in  d3&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  player_first_last d1_value player_first2 player_last player_last_first2 d2_value&#xA;&#xA;1         Joe Smith        1            Jo       Smith            SmithJo        1&#xA;2      Jack Johnson        2            Ja     Johnson            JohnsJa        2&#xA;3           A Jones        3             A       Jones             JonesA       NA&#xA;4           Slow Mo        4            Sl          Mo               MoSl        3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18355" LastActivityDate="2016-05-02T03:51:34.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11540" PostTypeId="1" CreationDate="2016-05-02T05:46:43.717" Score="0" ViewCount="827" Body="&lt;p&gt;I have a dataset on broadband usage and bills for a set of customers for an year. For every metric, like Upload usage, I have 12 values corresponding to each month.&#xA;Below are the columns and the format.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;CustomerId,Upload_Jan,Upload_Feb...Upload_Dec,&#xA;  Download_Jan,Download_Feb....Download_Dec,&#xA;  bill_Jan....bill_Dec&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The values in all columns are continuous.&#xA;I want to see if there are some sequences appearing in the data frequently. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can we do sequential mining on continuous data set?&lt;/li&gt;&#xA;&lt;li&gt;Is it mandatory to bin each column to separate bins? How to decide the number of bins?&lt;/li&gt;&#xA;&lt;li&gt;What is the best way of pre processing the data set for tools like &lt;code&gt;SPMF&lt;/code&gt; which requires transactionized data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I looked at &lt;code&gt;TraMineR&lt;/code&gt; package in R and wrote a few scripts, but I could not make out much from the sequences generated.&lt;/p&gt;&#xA;" OwnerUserId="6514" LastEditorUserId="6514" LastEditDate="2016-05-04T04:54:30.333" LastActivityDate="2016-05-04T12:08:19.450" Title="Sequence pattern mining on continuous dataset" Tags="|r|sequential-pattern-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11543" PostTypeId="1" AcceptedAnswerId="11551" CreationDate="2016-05-02T11:27:55.597" Score="6" ViewCount="4160" Body="&lt;p&gt;I have a dependent variable measuring the net revenue. One of the major predictor affecting this is &quot;product&quot; i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million entries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Products are assigned a specific categorical value.&#xA;I feel that using dummy variables to represent the products would be apt however, there are 4481 levels of products. I do not know how to code so many levels in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.matrix(~ product, data=salesdata)&lt;/code&gt; &#xA; returns an error. (Needs 38.4GB of memory)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone guide me a little on how to code these categorical variables?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dependent:&lt;/strong&gt; Net revenue (quantitative)&#xA;&lt;strong&gt;Independent:&lt;/strong&gt; Product code (quantitative but treated as qualitative since values are nominal) &lt;/p&gt;&#xA;" OwnerUserId="18369" LastActivityDate="2016-05-25T23:16:18.603" Title="Dummy coding a column in R with multiple levels" Tags="|r|regression|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11544" PostTypeId="2" ParentId="11510" CreationDate="2016-05-02T13:15:27.480" Score="1" Body="&lt;p&gt;Please correct me, but from what I see in your data you do not have (yet) a dependent variable. To create a supervised learning dataset, you will need to construct a dependent variable yourself, considering, for instance, the timespan between dates that a customer made purchases. You can identify same customers by using their loyalty id. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, for each customer, if their (date sorted) purchase history contains intervals shorter than 12 months, then you can use them to create some positive training examples. All the rest are your negative examples. Cleanup some of the data that are non pertinent, e.g. ID of Client and Transaction, and you can use the rest to build your first classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to transform some of your independent variables into more useful representations, such as date of birth to age groups etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for models: well, you have a plethora of choice here. Start from linear classifiers, and check performance in cross-validation. If not satisfactory, you may try tree-based classifiers, support vector machines and neural nets.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2016-05-02T13:15:27.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11547" PostTypeId="1" AcceptedAnswerId="11548" CreationDate="2016-05-02T19:28:18.620" Score="1" ViewCount="327" Body="&lt;p&gt;I have a Classifier which I have trained and tested on a small dataset - receiving solid results, though I wish to improve them. If I understand correctly one way of doing so is to add more data to obtain a more precise classification rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing this should I add data to both the training set and the test set? or should I add only to the training set? or maybe I should I create new training and test sets from the 'new dataset'? (new = the old data + the new data).&lt;/p&gt;&#xA;" OwnerUserId="14411" LastEditorDisplayName="user380" LastEditDate="2017-07-11T07:56:58.837" LastActivityDate="2017-07-11T07:56:58.837" Title="How to improve an existing (trained) classifier?" Tags="|machine-learning|classification|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11548" PostTypeId="2" ParentId="11547" CreationDate="2016-05-02T20:45:33.763" Score="2" Body="&lt;p&gt;Adding more data does not always help. However, you can get an estimate if more data will help you by the following procedure: Make a plot. On the $x$-axis is the amount of training examples, starting at one example per class going to wherever you are currently. The $y$-axis shows the error. Now you should add two curves: Training and test error. For low $x$, the training error should be very low (almost 0) and the test error very high. With enough data, they should be &quot;about the same&quot;. By plotting those curves you can make an educated guess how much more data will give you how much improvement.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When doing this should I add data to both the training set and the test set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Depends on what you want to achieve. If only getting a better classifier, then you can only add it to the training set. However, if you're doing this in a scientific setting this might be more difficult. (I assume that your test set is of reasonable size).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to have a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;cross-validation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-02T20:45:33.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11549" PostTypeId="1" CreationDate="2016-05-02T23:25:27.700" Score="-5" ViewCount="503" Body="&lt;p&gt;I'm using &lt;code&gt;numpy.eye(N, M, dtype=int)&lt;/code&gt; to print a matrix whose diagonal elements are 1 and rest of the elements are 0. Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import sys&#xA;&#xA;line = sys.stdin.readline()&#xA;&#xA;N, M = map(int, line.split())&#xA;&#xA;print numpy.eye(N, M, dtype=int)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above runs successfully in Python 2. But in python 3 I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;File &quot;solution.py&quot;, line 7&#xA;    print numpy.eye( N , M , dtype =int )&#xA;              ^&#xA;SyntaxError: invalid syntax&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there an alternate package other than numpy in Python 3 through which I can accomplish this? Has the &lt;code&gt;eye&lt;/code&gt; function become obsolete with Python 3?&lt;/p&gt;&#xA;" OwnerUserId="10532" LastEditorUserId="29575" LastEditDate="2020-05-14T02:06:56.943" LastActivityDate="2020-05-14T02:08:09.193" Title="Numpy.eye() in Python 3" Tags="|python|data-mining|programming|" AnswerCount="1" CommentCount="1" ClosedDate="2020-05-14T13:51:13.517" ContentLicense="CC BY-SA 4.0" />
  <row Id="11550" PostTypeId="2" ParentId="11549" CreationDate="2016-05-02T23:50:09.997" Score="2" Body="&lt;p&gt;No, the &lt;code&gt;print&lt;/code&gt; function has changed; it needs to be treated as a proper function, with parentheses. Try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(numpy.eye(N, M, dtype=int))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastEditorUserId="29575" LastEditDate="2020-05-14T02:08:09.193" LastActivityDate="2020-05-14T02:08:09.193" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11551" PostTypeId="2" ParentId="11543" CreationDate="2016-05-03T03:52:02.507" Score="3" Body="&lt;p&gt;You can use either sparse matrices or feature hashing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sparse Matrix&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose that using a sparse matrix is the only choice. I suspect that this line of code will work. This uses the &lt;em&gt;Matrix&lt;/em&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sparseProducts &amp;lt;- sparse.model.matrix(~ product, data=salesdata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Take my example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sparseDiagonalMatrix &amp;lt;- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;each column represents a different factor, this will yield:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1  1 . . . . . . . . .&#xA;2  1 1 . . . . . . . .&#xA;3  1 . 1 . . . . . . .&#xA;4  1 . . 1 . . . . . .&#xA;5  1 . . . 1 . . . . .&#xA;6  1 . . . . 1 . . . .&#xA;7  1 . . . . . 1 . . .&#xA;8  1 . . . . . . 1 . .&#xA;9  1 . . . . . . . 1 .&#xA;10 1 . . . . . . . . 1&#xA;&#xA;&amp;gt; class(sparseDiagonalMatrix)&#xA;[1] &quot;dgCMatrix&quot;&#xA;attr(,&quot;package&quot;)&#xA;[1] &quot;Matrix&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;alternatively you can remove the intercept and have all zeros represent class 1&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sparseDiagonalMatrix &amp;lt;- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))[, -1, drop=FALSE]&#xA;&#xA;10 x 9 sparse Matrix of class &quot;dgCMatrix&quot;&#xA;V12 V13 V14 V15 V16 V17 V18 V19 V110&#xA;1    .   .   .   .   .   .   .   .    .&#xA;2    1   .   .   .   .   .   .   .    .&#xA;3    .   1   .   .   .   .   .   .    .&#xA;4    .   .   1   .   .   .   .   .    .&#xA;5    .   .   .   1   .   .   .   .    .&#xA;6    .   .   .   .   1   .   .   .    .&#xA;7    .   .   .   .   .   1   .   .    .&#xA;8    .   .   .   .   .   .   1   .    .&#xA;9    .   .   .   .   .   .   .   1    .&#xA;10   .   .   .   .   .   .   .   .    1&#xA;&#xA;&amp;gt; class(sparseDiagonalMatrix)&#xA;[1] &quot;dgCMatrix&quot;&#xA;attr(,&quot;package&quot;)&#xA;[1] &quot;Matrix&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You will need a package that supports sparse matrices for measuring the net revenue though. Fortunately, most modern mainstream packages support sparse matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Hashing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a great explanation of feature hashing in R (among other techniques) which is also an alternative, specially useful when you have hundreds of thousands or millions of multiple levels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://amunategui.github.io/feature-hashing/&quot; rel=&quot;nofollow&quot;&gt;https://amunategui.github.io/feature-hashing/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2016-05-03T03:58:50.813" LastActivityDate="2016-05-03T03:58:50.813" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11552" PostTypeId="2" ParentId="11547" CreationDate="2016-05-03T06:44:14.667" Score="4" Body="&lt;p&gt;In order to improve your classifier, you have few options.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Ensembling&lt;/strong&gt; - make a group of classifier and let them predict together. Stacking, blending, bagging, boosting. Choice is yours.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Hyper parameter tunning&lt;/strong&gt; - you did not mention your tool, but i suppose that in every solid one is option to search in parameter space to find the best combination&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Sampling&lt;/strong&gt; - you can try undersample or create new samples (SMOTE) to give your classifier more data of class which you'd like to predict&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature engineering&lt;/strong&gt; - get rig of noisy features and work only with those which have an effect on your prediction, consider PCA too&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Scaling&lt;/strong&gt; - normalize your data can improve the performance, some classifiers require this&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data quality&lt;/strong&gt; - missing values, coding categorical variables or suspicious values can influence performance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2016-05-03T17:34:23.703" LastActivityDate="2016-05-03T17:34:23.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11553" PostTypeId="1" AcceptedAnswerId="11578" CreationDate="2016-05-03T06:50:14.997" Score="1" ViewCount="3838" Body="&lt;p&gt;I have a twitter sentiment analysis using bag of word approach from the training set. Now i want to add weights to certain words so that they are considered more important than others. &lt;/p&gt;&#xA;" OwnerUserId="15905" LastActivityDate="2018-04-10T17:45:11.710" Title="How can i add weights in a bag of words model in text analysis?" Tags="|text-mining|social-network-analysis|feature-engineering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11554" PostTypeId="1" AcceptedAnswerId="11607" CreationDate="2016-05-03T08:39:04.420" Score="6" ViewCount="4444" Body="&lt;p&gt;I'm following a Linear Discriminant Analysis tutorial from &lt;a href=&quot;http://nbviewer.jupyter.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/linear_discriminant_analysis.ipynb#LDA-in-5-steps&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; for dimensionality reduction. After working through the tutorial (did the PCA part, too), I shortened the code using sklearn modules where applicable and verified it on the Iris data set (same code, same result), a synthetic data set (with &lt;code&gt;make_classification&lt;/code&gt;) and the sklearn-digits dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, then I tried the exact same code on a complete different (unfortunately non-public) data set that contains spectra recordings of two classes. The LDA crashes at the eigenvector verification part, where the $\lambda \mathbf{v}$ is supposed to be almost equal to $S_W^{-1} S_B \mathbf{v}$ (with $\lambda$ being the eigenvalue and $\mathbf{v}$ the corresponding eigenvector; $S_W$ and $S_B$ are the in/between-class scatter matrices). The first vector to be wrong seems at random positions, meaning each run it's a different vector that's causing this error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect it's related to rounding during calculations, since I get complex eigenvectors. For the PCA I just discarded the complex part (I think I read it somewhere in this forum), but this approach does not seem to work with LDA. Has anybody encountered similar problems or knows what's wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is my code for the analysis, which is more or less the same as in the tutorial. I'm using the manual approach, since I'm interested in how many linear discriminants are needed to describe my data. (I'm not sure how to do this with sklearn's LDA.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def LDAnalysis_manual(X, y):&#xA;    n_features = X.shape[1]&#xA;    n_classes = len(np.unique(y))&#xA;&#xA;    print(&quot;Mean vectors...&quot;)&#xA;    mean_vectors = []&#xA;    for cl in range(n_classes):&#xA;        mean_vectors.append(np.mean(X[y == cl], axis=0))&#xA;        # print(&quot;Mean vector class {}: {}&quot;.format(cl, mean_vectors[cl - 1]))&#xA;&#xA;    print(&quot;In-class scatter matrix...&quot;)&#xA;    S_W = np.zeros((n_features, n_features))&#xA;    for cl, mv in zip(range(1, n_classes), mean_vectors):&#xA;        class_sc_mat = np.zeros((n_features, n_features))  # each class' scatter matrix&#xA;        for row in X[y == cl]:&#xA;            row, mv = row.reshape(n_features, 1), mv.reshape(n_features, 1)  # column vectors&#xA;            class_sc_mat += (row - mv).dot((row - mv).T)&#xA;        S_W += class_sc_mat  # sum class scatter matrices&#xA;&#xA;    overall_mean = np.mean(X, axis=0)&#xA;    print(&quot;Between-class scatter matrix...&quot;)&#xA;    S_B = np.zeros((n_features, n_features))&#xA;&#xA;    for i, mean_vec in enumerate(mean_vectors):&#xA;        n = X[y == i + 1].shape[0]&#xA;        mean_vec = mean_vec.reshape(n_features, 1)  # make column vector&#xA;        overall_mean = overall_mean.reshape(n_features, 1)&#xA;        S_B += n * (mean_vec - overall_mean).dot((mean_vec - overall_mean).T)&#xA;&#xA;    eig_vals, eig_vecs = np.linalg.eig(np.linalg.inv(S_W).dot(S_B))&#xA;&#xA;    print(&quot;Eigenvector test&quot;)&#xA;    for i in range(len(eig_vals)):&#xA;        print(&quot;\r{:3}&quot;.format(i), end=&quot; &quot;)&#xA;        sys.stdout.flush()&#xA;&#xA;        eigv = eig_vecs[:, i].reshape(n_features, 1)&#xA;        np.testing.assert_array_almost_equal(np.linalg.inv(S_W).dot(S_B).dot(eigv).real,&#xA;                                             (eig_vals[i] * eigv).real,&#xA;                                             decimal=6, err_msg='', verbose=True)&#xA;    __log.debug(&quot;\nAll values ok.&quot;)&#xA;&#xA;    eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:, i]) for i in&#xA;                 range(len(eig_vals))]  # make list of value &amp;amp; vector tuples&#xA;    eig_pairs = sorted(eig_pairs, key=lambda k: k[0], reverse=True)  # Sort tuple-list from high to low&#xA;&#xA;    __log.info(&quot;\nEigenvalues (decending):&quot;)&#xA;    for i in eig_pairs:&#xA;        __log.info(i[0])&#xA;&#xA;    tot = sum(eig_vals)&#xA;    var_exp = [(i / tot) for i in sorted(eig_vals, reverse=True)]&#xA;    cum_var_exp = np.cumsum(var_exp)&#xA;    cum_var_exp = cum_var_exp.real&#xA;    plot(len(var_exp), var_exp, cum_var_exp)&#xA;&#xA;    idx_98 = next(idx for idx, val in enumerate(cum_var_exp) if val &amp;gt; .98)&#xA;    return idx_98 + 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14657" LastActivityDate="2018-03-03T15:19:23.247" Title="Varying results when calculating scatter matrices for LDA" Tags="|python|scikit-learn|discriminant-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11557" PostTypeId="1" AcceptedAnswerId="11572" CreationDate="2016-05-03T10:28:34.040" Score="2" ViewCount="69" Body="&lt;p&gt;I read the likelihood is defined in logistic regression as the probability&#xA;$$&#xA;L(w) = P(y|x, w) = \prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}&#xA;$$&#xA;and the log of the last equation is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand (independent probabilities)&#xA;$$&#xA;L(w) = P(y|x, w) = \prod P(y^i| x^i,w)&#xA;$$&#xA;and I understand that from &#xA;$$&#xA;L(w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}&#xA;$$&#xA;the log is (basic log properties)&#xA;$$&#xA;log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))&#xA;$$&#xA;However, how do I get &#xA;$$&#xA;\prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}&#xA;$$&#xA;This basically means that&#xA;$$&#xA;P(y| x,w) = (\sigma(z))^{y}(1-\sigma(z))^{(1-y)}&#xA;$$&#xA;and I just don't see that.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2016-05-03T17:34:26.390" Title="Equation for likelihood in logistic regression" Tags="|logistic-regression|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11558" PostTypeId="1" AcceptedAnswerId="11561" CreationDate="2016-05-03T10:58:06.977" Score="1" ViewCount="545" Body="&lt;p&gt;I want to plot ROC curves using R. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a label vector. The column names of prediction columns are &lt;code&gt;ccs&lt;/code&gt;,&lt;code&gt;badaI&lt;/code&gt;,&lt;code&gt;badaII&lt;/code&gt; and the column name of label vector is &lt;code&gt;value&lt;/code&gt;. I am using &lt;code&gt;ROCR&lt;/code&gt;library for this as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ROCR)&#xA;pred1 &amp;lt;- prediction(df$ccs,df$value)&#xA;roc &amp;lt;- performance(pred1,&quot;tpr&quot;,&quot;fpr&quot;);&#xA;pred2 &amp;lt;- prediction(df$badaI,df$value)&#xA;roc2  &amp;lt;- performance(pred2,&quot;tpr&quot;,&quot;fpr&quot;)&#xA;pred3 &amp;lt;- prediction(df$badaII,df$value)&#xA;roc3 &amp;lt;- performance(pred3,&quot;tpr&quot;,&quot;fpr&quot;)&#xA;auc&amp;lt;- performance(pred,&quot;auc&quot;)&#xA;auc = round(unlist(auc@y.values),2)&#xA;auc2&amp;lt;- performance(pred2,&quot;auc&quot;)&#xA;auc2 = round(unlist(auc2@y.values),2)&#xA;auc3&amp;lt;- performance(pred3,&quot;auc&quot;)&#xA;auc3 = round(unlist(auc3@y.values),2)&#xA;plot(roc,col=&quot;black&quot;,lty=1, lwd=4, cex.lab=1.5,axt=&quot;n&quot;)&#xA;axis(1,cex.axis=1.0);axis(2,cex.axis=1.0)&#xA;plot(roc2, add=TRUE,col=&quot;black&quot;,lty=3, lwd=4)&#xA;plot(roc3, add=TRUE,col=&quot;black&quot;,lty=2, lwd=4)&#xA;abline(0,1,col=&quot;gray60&quot;)&#xA;legend(0.3,0.30,c(paste0(&quot;CCS, &quot;,&quot;AUC = &quot;,auc),paste0(&quot;BADAI, &quot;,&quot;AUC = &quot;,auc2),paste0(&quot;BADAII, &quot;,&quot;AUC = &quot;,auc3)),&#xA;      lty=c(1,3,2), col=c('black','black','black'), lwd=4,cex=1.4,bty=&quot;n&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While using the above code,I am getting following plot:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/crCmk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/crCmk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have doubt as:&#xA; While looking at the data, It is obvious that &lt;code&gt;ccs&lt;/code&gt; and &lt;code&gt;badaII&lt;/code&gt; should have higher &lt;code&gt;AUC&lt;/code&gt; values than &lt;code&gt;badaI&lt;/code&gt;, but the results are somehow opposite. Can anyone help me in understanding why it is behaving like this?&#xA;The &lt;code&gt;dput&lt;/code&gt; of the data used, &lt;code&gt;df&lt;/code&gt; is: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(ccs = c(0.16, 0.04, 0.18, 0.09, 0.14, 0.14, 0.04, &#xA;0.04, 0.08, 0.76, 0.03, 0.03, 0.68, 0.06, 0.83, 0.15, 0.07, 0.02, &#xA;0.93, 0.22, 0.28, 0.11, 0.05, 0.01, 0.17, 0.15, 1, 0.13, 0.23, &#xA;0.44, 1), badaI = c(0.61, 0.11, 0.53, 0.79, 0.75, 0.82, 0.57, &#xA;0.67, 0.4, 0.95, 0.49, 0.61, 0.97, 0.52, 0.98, 0.7, 0.03, 0.18, &#xA;0.85, 0.94, 0.9, 0.77, 0, 0.37, 0.47, 0.88, 0.99, 0.55, 0.86, &#xA;0.96, 0.99), badaII = c(0.32, 0, 0.27, 0.12, 0.33, 0.12, 0.56, &#xA;0, 0.32, 0.18, 0.18, 0.11, 0.18, 0.54, 0.37, 0.33, 1, 0.39, 0.29, &#xA;0.11, 0.32, 0.53, 0.25, 0.21, 0.15, 0.16, 0.85, 0.31, 0.44, 1, &#xA;1), value = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, &#xA;0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1)), .Names = c(&quot;ccs&quot;, &#xA;&quot;badaI&quot;, &quot;badaII&quot;, &quot;value&quot;), row.names = c(NA, -31L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&#xA;I use below figure to further explain my intuition. This figure is drawn using the same predictor values.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/7AaNt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7AaNt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;code&gt;value&lt;/code&gt; column, observation with numbers 15, 27, 30, 31 have labels as &lt;code&gt;1&lt;/code&gt; and the remaining observations have &lt;code&gt;0&lt;/code&gt; value. While looking at above figure it is clear that &lt;code&gt;CCS&lt;/code&gt; or &lt;code&gt;badaII&lt;/code&gt;are best in discriminating the difference between 0 and 1 as compared to &lt;code&gt;badaI&lt;/code&gt;, which always provide higher predictor values.In other words, I mean with &lt;code&gt;badaI&lt;/code&gt;, it is difficult to predict &lt;code&gt;1&lt;/code&gt; as its values are higher for both 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to correlate my intuition with the ROC plot. @TBSRounder, I understood what you have mentioned, but I need to support the above figure with the ROC plot, which I find disappointing. Can anyone help me to correlate the above figure with the ROC plot?&lt;/p&gt;&#xA;" OwnerUserId="13291" LastEditorUserId="13291" LastEditDate="2016-05-03T13:17:25.143" LastActivityDate="2016-05-03T13:41:44.033" Title="ROC curves/AUC values as a performance metric" Tags="|machine-learning|r|classification|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11559" PostTypeId="2" ParentId="11540" CreationDate="2016-05-03T11:43:12.950" Score="1" Body="&lt;p&gt;The established procedure is called &lt;a href=&quot;https://www.quora.com/MLconf-2015-Seattle-How-does-the-symbolic-aggregate-approximation-SAX-technique-work&quot; rel=&quot;nofollow&quot;&gt;&lt;em&gt;symbolic aggregate approximation&lt;/em&gt; or SAX&lt;/a&gt; in which you first first do &lt;em&gt;piecewise aggregate approximation&lt;/em&gt; (PAA), namely send non-overlapping mean (boxcar) windows of width $w$ over the data, and then transform the obtained values into symbols by equal-frequency-splitting the resulting data distribution into $k$ letters of the alphabet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to determine appropriate $w$ and $k$? They are a tradeoff between dimensionality reduction and information loss. One way to adapt $w$ is to look at window' data points variance; if the variance is too large, reduce the window size, and vice versa. Another information-theoretic is &lt;a href=&quot;https://stats.stackexchange.com/questions/192430/determining-window-sizes-of-varying-length-sub-sequences-of-time-series-data-for&quot;&gt;MDL as explained in this question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you only have 12 values in a series, the windowing step may not be necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could try reshaping the data into three series:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Dd_1^{(c_1)}\dots{}d_{12}^{(c_1)}Dd_1^{(c_2)}\dots{}d_{12}^{(c_2)}D\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Uu_1^{(c_1)}\dots{}u_{12}^{(c_1)}Uu_1^{(c_2)}\dots{}u_{12}^{(c_2)}U\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Bb_1^{(c_1)}\dots{}b_{12}^{(c_1)}Bb_1^{(c_2)}\dots{}b_{12}^{(c_2)}B\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where D, U, and B are string literals (used as start markers) and $d_m^{(c_i)}$, $u_m^{(c_i)}$, and $b_m^{(c_i)}$ are the values of download, upload and billed, for customer $c_i$ in month $m$. Then you could use one of the frequent subsequence mining algorithms on those three sequences (separately) to find any prevaling patterns.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="15527" LastEditDate="2016-05-04T12:08:19.450" LastActivityDate="2016-05-04T12:08:19.450" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11561" PostTypeId="2" ParentId="11558" CreationDate="2016-05-03T11:58:06.600" Score="2" Body="&lt;p&gt;The results you posted are correct, I did a quick check with &lt;code&gt;library(pROC)&lt;/code&gt; and got the same thing.  The important feature for AUC/ROC is that the cutpoint of calling a sample &quot;1&quot; or &quot;0&quot; is not set at 0.5.  For each of your predictors, the range of the numbers do not matter at all.  What does matter is how often higher numbers are associated with true positive labels.  Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Method 1:[1, 3, 5, 6, 9, 14] (predictor)&#xA;Method 1:[0, 0, 1, 0 ,1 , 1] (true label)&#xA;AUC: 0.8889&#xA;&#xA;Method 2:[[0.01, 0.02, 200 , 250, 300, 1000000] (predictor)&#xA;Method 2:[0, 0, 1, 0 ,1 , 1] (true label)&#xA;AUC: 0.8889&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The AUC measure is just a measure of how often true &quot;1&quot; samples have a higher number than true &quot;0&quot; samples.  The predictor does not have to even be bounded by [0,1], it can be any range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your plot would be more informative if it was sorted.  Consider the below sorted plot (crude, sorry) for each of the three methods, with the &quot;1&quot; labels highlighted in blue.  badaI clearly has the closest grouping of &quot;1&quot;s towards the higher values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sHxra.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sHxra.png&quot; alt=&quot;Sorted Values by Method&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14913" LastEditorUserId="14913" LastEditDate="2016-05-03T13:41:44.033" LastActivityDate="2016-05-03T13:41:44.033" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11564" PostTypeId="1" AcceptedAnswerId="11573" CreationDate="2016-05-03T13:23:35.380" Score="10" ViewCount="45679" Body="&lt;p&gt;I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix shows that the prediction and recall is completely incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ive read that the next step is to add weights to my class; I have two of them &quot;Won&quot; and &quot;Lost&quot;. In Pandas, how am I supposed to assign weights to them? I know that there is a &quot;class_weights&quot; attribute, but I have no clue on how to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TKu86.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TKu86.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. My &quot;Won&quot; class is unbalanced, very small compared to the &quot;Lost&quot; one. I train by repeating the set of &quot;Won&quot;s twice and randomly sample an almost equal amount of &quot;Lost&quot;s. I've tried all sorts of combinations of the classes.&lt;/p&gt;&#xA;" OwnerUserId="18098" LastActivityDate="2018-07-05T17:23:29.387" Title="How does class_weights work in RandomForestClassifier" Tags="|python|classification|random-forest|pandas|class-imbalance|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11565" PostTypeId="1" CreationDate="2016-05-03T13:26:07.407" Score="1" ViewCount="711" Body="&lt;p&gt;I'm trying to predict response of customers on a marketing campaign.&#xA;As of now, I have data from one Marketing campaign and rfm data of my customers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some proportion of customers say 60%, received adverts. Roughly 10% responded&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the response data, i.e. whether purchased or not, on this marketing campaign I built a random forest using scikit-learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model performs on a hold out set really well. but the most influential variable is the boolean: CustomerHasBeenAdvertised&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use this model, to select customers for a future marketing campaign.&#xA;To obtain the &quot;purchasing probabilities&quot; of the customers under the condition of similar advertisement, I set the variable CustomerHasBeenAdvertised to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, on a data set with this side condition, all the forecasts are above 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this extra ordinary high value due to the variable importance? Or are there other explanations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is setting the variable CustomerHasBeenAdvertised to 1 the false approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, how could one handle the case: Customer bought without beeing  advertised?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should one simply neglect the information whether advertisement took place or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="8816" LastEditorUserId="8816" LastEditDate="2016-05-03T14:03:06.657" LastActivityDate="2016-05-03T14:11:40.617" Title="Predict buying behavior under the condition that a customer is advertised or not" Tags="|python|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11566" PostTypeId="2" ParentId="11565" CreationDate="2016-05-03T14:05:53.970" Score="3" Body="&lt;p&gt;It depends what you want to do. If you want a model that will predict who to target/not-target for a marketing campaign, then you want to train a model using only the people who were marketed to, which defines your two classes &quot;responders&quot; and &quot;nonresponders&quot;. Given an unseen customer, your classifier will then determine whether they are likely to respond if you send them a directed marketing advertisement. Take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Uplift_modelling&quot; rel=&quot;nofollow&quot;&gt;uplift marketing&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-05-03T14:05:53.970" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11567" PostTypeId="2" ParentId="11565" CreationDate="2016-05-03T14:11:40.617" Score="2" Body="&lt;p&gt;If the CustomerHasBeenAdvertised variable corresponds to whether the customer received an ad from your campaign, then I think the best approach would be to exclude all records from your training set that do not have CustomerHasBeenAdvertised set to 1 and then eliminate that variable as a predictor. This could still result in most forecasts being above 0.5, but that's not necessarily a bad thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One common approach to dealing with imbalanced class problems (ex: few people responding to an ad or few people going bankrupt on a loan) is to separate your predictions into deciles, and then only take action on the top few deciles of interest (in your case customers most likely to make a purchase). This means that the relative / rank-order value of the prediction is more important than the nominal value of the prediction. Normally this is sufficient because you probably don't want to advertise to all customers as that would be cost prohibitive. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-05-03T14:11:40.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11568" PostTypeId="1" CreationDate="2016-05-03T14:33:40.117" Score="5" ViewCount="2226" Body="&lt;p&gt;I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt; on each text is the right approach. I've read something about calculating and compare the cosine similarity on tf-idf, but I don't know how to further process the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saying &quot;almost equal&quot; means &quot;the order of the words matters&quot;, it' not just the same words in different order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about using MLlib from Apache Spark to do the job (or at least use a Scala library).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody head me to the right direction, or even better add a link to a tutorial to this page?&lt;/p&gt;&#xA;" OwnerUserId="18407" LastEditorUserId="6478" LastEditDate="2016-05-04T14:25:27.217" LastActivityDate="2020-02-27T06:30:14.527" Title="How can I group texts with similar content together?" Tags="|nlp|similarity|text|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11569" PostTypeId="1" CreationDate="2016-05-03T14:58:09.070" Score="3" ViewCount="183" Body="&lt;p&gt;Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NDmyl.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NDmyl.gif&quot; alt=&quot;example tree structure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, in this kind of hierarchical classification, if a sample is wrongly classified in an upper layer, it is less likely to be corrected again in lower layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: Is there any strategy to mitigate such hierarchical error propagation in tree-structured classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="18409" LastActivityDate="2016-05-03T14:58:09.070" Title="How to mitigate the hierarchical error propagation in tree-structured classification" Tags="|machine-learning|svm|multiclass-classification|hierarchical-data-format|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11570" PostTypeId="2" ParentId="11568" CreationDate="2016-05-03T15:02:51.107" Score="4" Body="&lt;p&gt;Well, after further googling I found the solution: MinHash or SimHash will do the job and I also found a tool implementing MinHash written in Scala on GitHub right at this &lt;a href=&quot;https://github.com/pmandera/duometer&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18407" LastEditorUserId="18407" LastEditDate="2016-05-06T13:37:21.770" LastActivityDate="2016-05-06T13:37:21.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11572" PostTypeId="2" ParentId="11557" CreationDate="2016-05-03T17:34:26.390" Score="1" Body="&lt;p&gt;By the definition of the logistic regression model $\mathrm P(y = 1 | x,w) = \sigma\left(\left&amp;lt;w, x\right&amp;gt;\right)$ thus $\mathrm P(y = 0 | x,w) = 1- \sigma\left(\left&amp;lt;w, x\right&amp;gt;\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to verify (by setting y=0/y=1) that this is equivalent to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathrm P(y | x,w) = \sigma\left(\left&amp;lt;w, x\right&amp;gt;\right)^y \left[ 1- \sigma \left( \left&amp;lt;w, x \right&amp;gt; \right) \right]^{1-y}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you understand the rest?&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-05-03T17:34:26.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11573" PostTypeId="2" ParentId="11564" CreationDate="2016-05-03T17:45:07.613" Score="13" Body="&lt;p&gt;Maybe try to encode your target values as binary. Then, this &lt;code&gt;class_weight={0:1,1:2}&lt;/code&gt; should do the job. Now, class 0 has weight 1 and class 1 has weight 2.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-05-03T17:45:07.613" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11574" PostTypeId="2" ParentId="8190" CreationDate="2016-05-03T17:59:43.767" Score="1" Body="&lt;p&gt;A Tensor Flow implementation&#xA;&lt;a href=&quot;https://github.com/geffy/tffm/&quot; rel=&quot;nofollow&quot;&gt;https://github.com/geffy/tffm/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It supports arbitrary order (&gt;=2) Factorization Machines, dense and sparse input. The interface is in Scikit-learn style.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://github.com/geffy/tffm/blob/master/example.ipynb&quot; rel=&quot;nofollow&quot;&gt;Jupyter notebook with examples&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18414" LastEditorUserId="18414" LastEditDate="2016-05-10T06:54:59.060" LastActivityDate="2016-05-10T06:54:59.060" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11575" PostTypeId="1" CreationDate="2016-05-03T18:32:17.520" Score="1" ViewCount="229" Body="&lt;p&gt;the below dataset is just top 8 rows of 100 rows dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;a&#xA;  group marks    upd&#xA;2     C     3     up&#xA;3     C     4   down&#xA;6     C     7     up&#xA;1     T     2     up&#xA;4     T     5     up&#xA;5     T     6   down&#xA;7     T     1 middle&#xA;8     T     0   down&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used unique_groups=unique(a$group) to get a list of unique group names to be sent into a function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I need a function that gives a subset of this dataset where it takes each group from unique_groups and checks if 'upd' is in ('down','middle') and gives me a dataset having only those records having upd ('down','middle') of that respective group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use do.call function to get the dataset that I want as-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;do.call(rbind,sapply(unique_groups,FUN=function(y)&#xA;  {&#xA;  h=a[a$group==x &amp;amp; a$upd %in% c('down','middle'),]&#xA;  h&#xA;}&#xA;, simplify=F))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wasnt getting the dataset that I want and it was giving this warning message:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;code&gt;==.default&lt;/code&gt;(a$group, x) :&#xA;  longer object length is not a multiple of shorter object length&lt;/p&gt;&#xA;" OwnerUserId="13417" LastEditorUserId="13417" LastEditDate="2016-05-03T21:54:56.857" LastActivityDate="2016-05-04T23:44:42.577" Title="sub-setting based on categorical variable in R" Tags="|r|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11576" PostTypeId="2" ParentId="11575" CreationDate="2016-05-03T19:59:32.780" Score="0" Body="&lt;p&gt;Try this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;new_dataframe&amp;lt;- subset(a,upd==&quot;down&quot; | upd==&quot;middle&quot;)&lt;/p&gt;&#xA;" OwnerUserId="18417" LastActivityDate="2016-05-03T19:59:32.780" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11578" PostTypeId="2" ParentId="11553" CreationDate="2016-05-03T20:18:51.183" Score="2" Body="&lt;p&gt;One possible solution is to introduce prior counts for words (higher counts for words that are more important) that could be added to the term-document matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative solution is to compute tf-idf features (weights that modify word counts based on frequency) and apply additional weighting to tf-idf with higher weights corresponding to important words.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2016-05-03T20:18:51.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11580" PostTypeId="1" AcceptedAnswerId="11582" CreationDate="2016-05-04T02:25:29.190" Score="1" ViewCount="193" Body="&lt;p&gt;I'm coming from Stata and struggling to get used to group_by in dplyr. Perhaps using group_by is the wrong approach, but if you know Stata, you'll understand why I'm trying to use it (I think).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to transform the data as shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WkgIy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WkgIy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, I'm trying to collapse each NAICS code into a single row, and to create columns for the 5 statistics under CONC and HHI for each NAICS code (the 4 values for CONC and one for HHI, ignoring the Xs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is &lt;a href=&quot;https://drive.google.com/folderview?id=0B8tQJCb7UMEhcE1VYmpkRXpVNUU&amp;amp;usp=sharing&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. The code I'm trying is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ManufMktConc &amp;lt;- read.csv(&quot;Manufacturing.csv&quot;,&#xA;                            stringsAsFactors = FALSE)&#xA;&#xA;ManufMktConc %&amp;lt;&amp;gt;% &#xA;  tbl_df %&amp;gt;% &#xA;  slice(-1) %&amp;gt;% &#xA;  filter(CONCENFI.display.label!=&quot;All companies&quot;) %&amp;gt;%&#xA;  select(-GEO.id,-GEO.id2,-GEO.display.label,-COMPANY,-RCPTOT,-YEAR.id)&#xA;&#xA;ManufMktConc_byInd &amp;lt;- group_by(ManufMktConc,NAICS.id) &#xA;&#xA;ManufMktConc_byInd %&amp;lt;&amp;gt;% &#xA;      arrange(CONCENFI.id) %&amp;gt;% &#xA;      mutate(FourFirm = ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id==&quot;856&quot;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That gives me the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ManufMktConc_byInd %&amp;lt;&amp;gt;% arrange(CONCENFI.id) %&amp;gt;% mutate(FourFirm=ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id==&quot;856&quot;])&#xA;Error: incompatible size (651), expecting 4 (the group size) or 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I can tell that I am asking R to stick the 651 instances of &lt;code&gt;ManufMktConc_byInd$CONCENFI.id == &quot;856&quot;&lt;/code&gt; into the group of 4. I guess this is where my understanding of group_by falls apart. Why isn't my logical statement only applied within each group?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="18429" LastActivityDate="2016-05-04T06:31:53.687" Title="group_by: How to Transform my Data" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11581" PostTypeId="1" CreationDate="2016-05-04T03:45:04.807" Score="1" ViewCount="304" Body="&lt;p&gt;I have music play data organized by the day on which each track was played, from March 1st, 2015 to August 30th, 2015. The data set contains count data for every day a song was played. I'd like to predict play counts for each track for an out-of-time window, from 2015-09-01 to 2015-09-30.&lt;/p&gt;&#xA;&lt;h3&gt;Details of the data&lt;/h3&gt;&#xA;&lt;p&gt;The data is from &lt;a href=&quot;https://tianchi.shuju.aliyun.com/competition/information.htm?spm=5176.100067.5678.2.FWsGgG&amp;amp;raceId=231531&amp;amp;_lang=en_US&quot; rel=&quot;nofollow noreferrer&quot;&gt;Alibaba's TianChi Competition&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Action Table (mars_tianchi_user_actions)&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yKhvO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yKhvO.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Any user’s action data for a song is indicated by a unique row&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Songs Table（mars_tianchi_songs）&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HjGhx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HjGhx.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Result Set&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;The participants need to predict the artists’ plays data in the following two months (20150901-20151030).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Participant’s Result Table (mars_tianchi_artist_plays_predict)&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/J9ymr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J9ymr.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;For song id &lt;code&gt;daa234f183aee2373d20987b247cd768&lt;/code&gt;(all the song ids are hash values). The play plot is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1r4GS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1r4GS.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;As you can see it, the variance of each day's play for this track is quite large.&lt;/p&gt;&#xA;&lt;p&gt;I also have data on who played each track, with each user ids represented as a hash value.&lt;/p&gt;&#xA;&lt;p&gt;I'd like some guidance on which analysis method I could use here. I am considering time series analysis.&lt;/p&gt;&#xA;" OwnerUserId="12866" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-01-23T02:01:32.420" Title="How should I process music play data" Tags="|machine-learning|data-mining|predictive-modeling|time-series|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11582" PostTypeId="2" ParentId="11580" CreationDate="2016-05-04T06:31:53.687" Score="1" Body="&lt;pre&gt;&lt;code&gt;ManufMktConc &amp;lt;- read.csv(&quot;Downloads/Manufacturing.csv&quot;,&#xA;                         stringsAsFactors = FALSE)&#xA;&#xA;ManufMktConc %&amp;lt;&amp;gt;% &#xA;  tbl_df %&amp;gt;% &#xA;  slice(-1) %&amp;gt;% &#xA;  filter(CONCENFI.display.label!=&quot;All companies&quot;) %&amp;gt;%&#xA;  select(-GEO.id,-GEO.id2,-GEO.display.label,-COMPANY,-RCPTOT,-YEAR.id)&#xA;&#xA;##  use melt and cast from reshape2 &#xA;require(reshape2)&#xA;&#xA;# melt makes the data set tall and thin using id variables and measure variables&#xA;# &#xA;ManufMktConc_molten &amp;lt;- as.data.frame(ManufMktConc) %&amp;gt;%&#xA;  melt(id.vars=1:4, measure.vars=5:6) %&amp;gt;%&#xA;  filter(value!=&quot;X&quot;)&#xA;ManufMktConc_molten[1:5,]&#xA;&#xA;    NAICS.id       NAICS.display.label CONCENFI.id CONCENFI.display.label  variable value&#xA;1      311        Food manufacturing         856    4 largest companies CCORCPPCT  16.3&#xA;2      311        Food manufacturing         857    8 largest companies CCORCPPCT  24.2&#xA;3      311        Food manufacturing         858   20 largest companies CCORCPPCT  38.4&#xA;4      311        Food manufacturing         859   50 largest companies CCORCPPCT  50.9&#xA;5     3111 Animal food manufacturing         856    4 largest companies CCORCPPCT  30.2&#xA;&#xA;&#xA;&#xA;# make a new column with the eventual column header. (Note this is different that your example)&#xA;ManufMktConc_molten$label &amp;lt;- paste0(ManufMktConc_molten$variable,&#xA;                                    trimws(substr(ManufMktConc_molten$CONCENFI.display.label,1,2)))&#xA;&#xA;# cast it into multiple columns (something like a pivot in Excel).&#xA;ManufMktConc_result  &amp;lt;- ManufMktConc_molten %&amp;gt;%&#xA;  cast(NAICS.id + NAICS.display.label ~ label) %&amp;gt;%&#xA;  select(1,2,4,6,3,5,7)     ## reorder columns&#xA;&#xA;ManufMktConc_result[1:5,]&#xA;&#xA;      NAICS.id             NAICS.display.label CCORCPPCT4 CCORCPPCT8 CCORCPPCT20 CCORCPPCT50 VSHERFI50&#xA;    1      311              Food manufacturing       16.3       24.2        38.4        50.9     110.7&#xA;    2     3111       Animal food manufacturing       30.2       40.7        57.8        71.5     368.6&#xA;    3    31111       Animal food manufacturing       30.2       40.7        57.8        71.5     368.6&#xA;    4   311111  Dog and cat food manufacturing       67.8       80.6        89.6        96.5    2019.4&#xA;    5   311119 Other animal food manufacturing       24.3       36.2        51.5        68.2     228.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18355" LastActivityDate="2016-05-04T06:31:53.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11586" PostTypeId="2" ParentId="11399" CreationDate="2016-05-04T12:29:35.110" Score="3" Body="&lt;p&gt;Yes it is important that the data is randomly ordered. You want to remove patterns of how you descent along the cost function because such patterns can increase the likelihood of overfitting. It's easiest to explain on &lt;code&gt;A,A,A, ... B,B,B ...&lt;/code&gt; pattern. Here the network is given an opportunity to adapt to peculiarities of the first training example before moving further. And the goal of training is not to adapt to peculiarities but to find features that generalize well. So it seems that a better idea is to expose the network to a different example in each iteration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most research works that I've seen simply randomize the order of examples in each epoch. But it seems sensible to think that a useful pattern might in fact exist. Perhaps one that focuses on maximizing the variety of features that the network is exposed to with each subsequent iteration.&lt;/p&gt;&#xA;" OwnerUserId="18282" LastActivityDate="2016-05-04T12:29:35.110" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11588" PostTypeId="2" ParentId="11575" CreationDate="2016-05-04T13:13:17.010" Score="2" Body="&lt;p&gt;This question seemed to be more suitable for general stackoverflow since it's a R question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using &lt;code&gt;data.table&lt;/code&gt;. The syntax is a little bit different from &lt;code&gt;data.frame&lt;/code&gt;, but the time spent on learning &lt;code&gt;data.table&lt;/code&gt; is well worth the effort. See more tutorials on &lt;a href=&quot;https://github.com/Rdatatable/data.table/wiki&quot; rel=&quot;nofollow&quot;&gt;wiki&lt;/a&gt;, and this good &lt;a href=&quot;https://s3.amazonaws.com/assets.datacamp.com/img/blog/data+table+cheat+sheet.pdf&quot; rel=&quot;nofollow&quot;&gt;sheet&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(data.table)&#xA;s = &quot;rowname    group   marks   upd&#xA;2   C   3   up&#xA;3   C   4   down&#xA;6   C   7   up&#xA;1   T   2   up&#xA;4   T   5   up&#xA;5   T   6   down&#xA;7   T   1   middle&#xA;8   T   0   down&quot;&#xA;# fread is the fastest csv reader&#xA;a = fread(s) &#xA;# the i in data.table[i,j] subset rows, empty j select all columns&#xA;upd_subset = a[upd %in% c('down','middle'),]&#xA;&#xA;&amp;gt; upd_subset&#xA;   rowname group marks    upd&#xA;1:       3     C     4   down&#xA;2:       5     T     6   down&#xA;3:       7     T     1 middle&#xA;4:       8     T     0   down&#xA;&#xA;&amp;gt; upd_subset_bygroups = split(upd_subset, upd_subset$group)&#xA;&amp;gt; upd_subset_bygroups&#xA;$C&#xA;   rowname group marks  upd&#xA;1:       3     C     4 down&#xA;&#xA;$T&#xA;   rowname group marks    upd&#xA;1:       5     T     6   down&#xA;2:       7     T     1 middle&#xA;3:       8     T     0   down&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18426" LastEditorUserId="18426" LastEditDate="2016-05-04T23:44:42.577" LastActivityDate="2016-05-04T23:44:42.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11589" PostTypeId="1" AcceptedAnswerId="21792" CreationDate="2016-05-04T14:55:36.070" Score="14" ViewCount="37080" Body="&lt;p&gt;It is a well known fact that a 1-layer network cannot predict the xor function, since it is not linearly separable. I attempted to create a 2-layer network, using the logistic sigmoid function and backprop, to predict xor. My network has 2 neurons (and one bias) on the input layer, 2 neurons and 1 bias in the hidden layer, and 1 output neuron.&#xA;To my surprise, this will not converge. if I add a new layer, so I have a 3-layer network with input (2+1), hidden1 (2+1), hidden2 (2+1), and output, it works. Also, if I keep a 2-layer network, but I increase the hidden layer size to 4 neurons + 1 bias, it also converges. Is there a reason why a 2-layer network with 3 or less hidden neurons won't be able to model the xor function?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2023-04-22T16:03:08.973" Title="Creating neural net for xor function" Tags="|neural-network|backpropagation|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11590" PostTypeId="2" ParentId="11589" CreationDate="2016-05-04T15:33:35.623" Score="10" Body="&lt;p&gt;A network with one hidden layer containing two neurons should be enough to separate the XOR problem. The first neuron acts as an OR gate and the second one as a NOT AND gate. Add both the neurons and if they pass the threshold it's positive. You can just use linear decision neurons for this with adjusting the biases for the thresholds. The inputs of the NOT AND gate should be negative for the 0/1 inputs. This picture should make it more clear, the values on the connections are the weights, the values in the neurons are the &lt;strong&gt;negatives&lt;/strong&gt; of the biases. E.g. for input1=0 / input2=1, hidden neuron 1 would be &lt;code&gt;sign function(0*1 + 1*1 - 0.5)&lt;/code&gt; == 1, meaning it passes the OR gate. The decision functions act as 0/1 decisions (or just the sign function works in this case too).&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nRZ6z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nRZ6z.png&quot; alt=&quot;XOR Neural Network&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Picture thanks to &lt;a href=&quot;https://blog.abhranil.net/2015/03/03/training-neural-networks-with-genetic-algorithms&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;Abhranil blog&amp;quot;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="148514" LastEditDate="2023-04-22T16:03:08.973" LastActivityDate="2023-04-22T16:03:08.973" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="11591" PostTypeId="1" CreationDate="2016-05-04T18:17:38.530" Score="10" ViewCount="3276" Body="&lt;p&gt;By definition, Relu is &lt;code&gt;max(0,f(x))&lt;/code&gt;. Then its gradient is defined as:&#xA;&lt;code&gt;1 if x &amp;gt; 0 and 0 if x &amp;lt; 0&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wouldn't this mean the gradient is always 0 (vanishes) when x &amp;lt; 0? Then why do we say Relu doesn't suffer from the gradient vanish problem? &lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="8432" LastEditDate="2017-08-14T05:12:12.123" LastActivityDate="2017-08-14T05:12:12.123" Title="Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x &lt; 0?" Tags="|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11592" PostTypeId="2" ParentId="11591" CreationDate="2016-05-04T18:40:20.577" Score="2" Body="&lt;p&gt;Vanishing means that it goes towards 0 but will never really be 0. Having gradients of 0 makes for very easy calculations, having gradients close to 0 means that there are changes, just very tiny ones which mean slow learning and numerical issues. 1 and 0 are two of the easiest numbers to calculate in these kind of optimization problems.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-04T18:40:20.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11593" PostTypeId="2" ParentId="11581" CreationDate="2016-05-04T19:10:55.783" Score="1" Body="&lt;p&gt;Interesting problem. You certainly could use time series methods for this, but you should consider whether you think there is face-validity to song plays following a temporal pattern. From your plot above, I don't see any structure in the variation, but perhaps there is something that an algorithm can pick up. A good starting point would be R's &lt;code&gt;auto.arima&lt;/code&gt; function for autoregressive integrated moving average (ARIMA). The ARIMA method is more fully described elsewhere (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;), but an interesting extension to this approach might be using exogenous predictors in your model. For example, does the presence or absence of rain affect the likelihood of some songs being played? My guess would be yes (I love listening to post-rock bands on rainy days), but perhaps that's not true of everyone. This extension to ARIMA is referred to as ARIMAX in the literature. A really nice breakdown of how to handle covariates in your type of analysis can be found on &lt;a href=&quot;http://robjhyndman.com/hyndsight/arimax/&quot; rel=&quot;nofollow&quot;&gt;Rob Hyndman's blog&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second thought is that, with sufficient computational resources, there's an interesting hidden markov model (&lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;HMM&lt;/a&gt;) problem here. You could potentially model the transition probabilities from one song to another. Presumably there's a high likelihood of transitioning from track 1 to track 2 of a given record, but sometimes people might switch to a different record after listening to a favorite track on an album. The extent to which this is true for different tracks could be a part of this analysis, and you could attempt to model the hidden underlying states that motivate a listener to stay within a record, or to move to a new one.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-05-04T19:10:55.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11596" PostTypeId="2" ParentId="11589" CreationDate="2016-05-04T21:18:20.697" Score="6" Body="&lt;p&gt;If you are using basic gradient descent (with no other optimisation, such as momentum), and a minimal network 2 inputs, 2 hidden neurons, 1 output neuron, then it is definitely possible to train it to learn XOR, but it can be quite tricky and unreliable.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You may need to adjust learning rate. Most usual mistake is to set it too high, so the network will oscillate or diverge instead of learn.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It can take a surprisingly large number of epochs to train the minimal network using batched or online gradient descent. Maybe several thousand epochs will be required.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;With such a low number of weights (only 6), sometimes random initialisation can create a combination that gets stuck easily. So you may need to try, check results and then re-start. I suggest you use a seeded random number generator for initialisation, and adjust the seed value if error values get stuck and do not improve.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-05-04T21:18:20.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11598" PostTypeId="1" CreationDate="2016-05-04T22:00:33.907" Score="1" ViewCount="615" Body="&lt;p&gt;Using the unit step activation function for the artificial neuron, determine a set of weights (and threshold value) that will produce the following classification: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1  x2  output&#xA;-0.2 0.5    0&#xA;0.2 -0.5    0&#xA;0.8 -0.8    1&#xA;0.8 0.8     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand the theory of unit step activation function (maybe beginner) but i am stumped as to how to calculate the weights and threshold values. I thought that it is specific to the problem and is provided as input. Please help&lt;/p&gt;&#xA;" OwnerUserId="18452" LastEditorUserId="381" LastEditDate="2016-05-05T06:07:02.623" LastActivityDate="2017-02-04T06:38:04.503" Title="Unit Step Activation Function" Tags="|machine-learning|neural-network|linear-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11599" PostTypeId="2" ParentId="10690" CreationDate="2016-05-04T22:03:07.247" Score="1" Body="&lt;p&gt;&lt;strong&gt;Just unroll the matrices into a large set of features:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all of the matrices are the same size, you can just unroll them into a large feature set e.g. a 20 x 20 matrix turns into 400 features.  You can do this with multiple different feature matrices.  Its up to the learning algorithm to infer each feature's meaning, so don't overthink the lack of human readablility. Take a look at &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html&quot; rel=&quot;nofollow&quot;&gt;tutorials on digit recognition in scikit learn&lt;/a&gt; and you will see that the &quot;image pixel matrices&quot; have been similarly unrolled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then employ &lt;code&gt;PCA&lt;/code&gt; or a &lt;a href=&quot;https://www.cs.utah.edu/~piyush/teaching/25-10-slides.pdf&quot; rel=&quot;nofollow&quot;&gt;nonlinear dimensionality reduction&lt;/a&gt; scheme to select a subset of the feature space.  Though it might go against your intuition, the dimensionality reduction will likely improve your classification algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accounting for different lengths of sounds is a difficult problem because normalization will alter the frequency and thus affect your Fourier Transformation. I suggest defining some sort of characteristic sound length and subsampling long sounds $n$ times and oversampling short sounds $n$ times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!  &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-05-04T22:03:07.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11600" PostTypeId="1" AcceptedAnswerId="11618" CreationDate="2016-05-05T12:25:35.440" Score="2" ViewCount="1106" Body="&lt;p&gt;I would like to run the association rule mining algorithm of the Orange library on a dataset that is stored in a PostgreSQL database. The table 'buildingset' contains the itemsets for each user, thus each record is related to a user, and each field is related to an item. The values are either 1 (smallint) or missing. The table has about 14,000 records and 31 fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I try to run the algorithm on this dataset, I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;ValueError Traceback (most recent call last):&#xA;    File &quot;/opt/orange/orange3/Orange/canvas/scheme/widgetsscheme.py&quot;, line 722, in process_signals_for_widget&#xA;      handler(*args)&#xA;    File &quot;/home/bdukai/.local/lib/python3.4/site-packages/orangecontrib/associate/widgets/owassociate.py&quot;, line 444, in set_data&#xA;      self.X = data.X&#xA;    File &quot;/opt/orange/orange3/Orange/data/sql/table.py&quot;, line 353, in X&#xA;      self.download_data(AUTO_DL_LIMIT)&#xA;    File &quot;/opt/orange/orange3/Orange/data/sql/table.py&quot;, line 333, in download_data&#xA;      raise ValueError(&quot;Too many rows to download the data into memory.&quot;)&#xA;  ValueError: Too many rows to download the data into memory.&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Thus is there any way overcome this limitation, without upgrading the hardware? &lt;/p&gt;&#xA;" OwnerUserId="15405" LastActivityDate="2018-06-05T09:54:45.643" Title="Overcome memory limitation when downloading from database into Orange" Tags="|python|data-mining|databases|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11601" PostTypeId="1" AcceptedAnswerId="11603" CreationDate="2016-05-05T12:54:28.863" Score="3" ViewCount="2737" Body="&lt;p&gt;I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the number of 'Won's' in my dataset is 0.05%. I've tried both oversampling and downsampling, but it hasn't worked. Even if I take an equal amount of 'Won's and 'Lost's', the model is not accurate for the rest of the 'Lost' values. I've also tried out weights, but it's not working well. Ideally I think I have to put a very high weight for 'Won'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Using RandomForestClassifier, with a confusion matrix to verify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not keen on trying out SMOTE, as I've heard it's tough in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I'm trying to look at it in a different way, and do anomaly detection for the 'Won' case, as it natural for the data to have so few 'Won' cases.  So, two questions&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this a correct approach?&lt;/li&gt;&#xA;&lt;li&gt;How to go about it using Python?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18098" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:42.010" LastActivityDate="2017-05-04T19:57:42.010" Title="Outlier detection for unbalanced classes" Tags="|python|class-imbalance|anomaly-detection|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11602" PostTypeId="2" ParentId="11601" CreationDate="2016-05-05T15:57:33.613" Score="1" Body="&lt;p&gt;Firstly, anamoly detection &lt;strong&gt;&lt;em&gt;does not look&lt;/em&gt;&lt;/strong&gt; to be the right approach for your use case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 1:&lt;/strong&gt; &#xA;There is a python module to perform under and over sampling with various techniques &lt;a href=&quot;https://github.com/fmfn/UnbalancedDataset&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. This package includes SMOTE implementation as well. Using this package try to balance event data in your sample before running RandomForest or any other classification algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach 2:&lt;/strong&gt; If the above does not work well, then firth's bias reduced logistic regression approach with penalized profile likelihood based confidence intervals for parameter estimates will suit your case. You can try L1 (Lasso) regularization on logistic regression from Python scikit library. Alternatively you can try R's 'logistf' package &lt;a href=&quot;http://www.r-bloggers.com/example-8-15-firth-logistic-regression/&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8465" LastActivityDate="2016-05-05T15:57:33.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11603" PostTypeId="2" ParentId="11601" CreationDate="2016-05-05T17:59:40.410" Score="4" Body="&lt;p&gt;you need to distinguish between these cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Data Imbalance&lt;/li&gt;&#xA;&lt;li&gt;Data Imbalance + Very few number of samples (minority class)&lt;/li&gt;&#xA;&lt;li&gt;Severe Data Imbalance + Very few number of samples (minority class)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;20:60   vs.  10:20    vs. 100:1000  vs. 10:100&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and these cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;similarities between different classes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;wide variations within the same class.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You need to understand to which of these cases your problem belong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you have very severe data imbalance + very few number of samples + wide variation within the majority class and similarities between different classes. regular oversampling or down sampling techniques will not help you as well as most of the synthetic oversampling techniques designed specifically to deal with the data imbalance but the assumption is to have enough number of samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to focus more on ensemble techniques that designed mainly to deal with data imbalance. &#xA;SMOTE-Boost&#xA;RUSBoost&#xA;SMOTEBagging&#xA;IIIVote&#xA;EasyEnsemble&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-05T17:59:40.410" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11604" PostTypeId="1" CreationDate="2016-05-05T18:02:57.473" Score="1" ViewCount="970" Body="&lt;p&gt;I have a model with 13 independent variables (2 of them are categorical variables) and 678 observations. All of the variables are significant.I'm planning to check the coefficients stability. My approach is to start from a model with one variable and find that coefficients. Then I’ll add another variable; however, I'm keeping the previous variable also. I’ll continue until that all variables are in the final model. I'm using the following R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;add.APV&amp;lt;-lm(HPV~log(APV),data=Mydata)&#xA;add1(add.APV,scope =~ AWD + Variety.bodyandchassis + Flexibility + Model.types + Year + log(CAC)+ Outsourcing + Platformstrategy + Hourly.total + Newownership + Vehicle.Launch + Segment.MD., test=&quot;F&quot;, k=2, trace=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I'm getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in factor.scope(attr(terms1, &quot;factors&quot;), list(add = attr(terms2,  : &#xA;  upper scope has term ‘NA’ not included in model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was wondering what the problem is. Any feedback would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="12867" LastEditorUserId="381" LastEditDate="2016-05-06T03:45:13.770" LastActivityDate="2019-05-11T07:02:55.643" Title="Checking regression coefficients' stability?" Tags="|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11607" PostTypeId="2" ParentId="11554" CreationDate="2016-05-05T19:33:00.193" Score="4" Body="&lt;ol&gt;&#xA;&lt;li&gt;The LDA crashes for the exact reason you suspected.  You have&#xA;complex eigenvalues.  If you use &lt;code&gt;np.linalg.eigh&lt;/code&gt;, which was&#xA;designed to decompose Hermetian matrices, you will always get real&#xA;eigenvalues.  &lt;code&gt;np.linalg.eig&lt;/code&gt; can decompose nonsymetric square&#xA;matrices, but, as you've suspected, it can produce complex&#xA;eigenvalues.  In short, &lt;code&gt;np.linalg.eigh&lt;/code&gt; is more stable, and I would suggest using it for both PCA and LDA.&lt;/li&gt;&#xA;&lt;li&gt;Dropping the complex part of the eigenvalues may have been acceptable in your specific example, but in practice it should be avoided.  Depending on the size of the complex part of the number, it can significantly change the result.  For example think of the multiplication of two complex conjugates.  $(3+ .1i)*(3-.1i)=9-.01=9.01$ comared to $9$ when droping the complex part is a relatively safe, but $(3-2i)*(3+2i)=13$ compared to $9$ is a significant miscalculation.  Using the above method for the eigen-decomposition will prevent this situation from arising.&lt;/li&gt;&#xA;&lt;li&gt;Remember that one of the assumptions of LDA is that the features are normally distributed and independent of each other.  Try running &lt;code&gt;print('Class label distribution: %s' % np.bincount(y_train)[1:])&lt;/code&gt;.  If the counts are not close to being equal, you've violated the first asumption of LDA, and the within-class scatter matrix must be scaled, in short divide each count by the number of class samples $N_i$.  By doing this it should be obvious that computing the normalized scatter matrix is the same as computing the covariance matrix $\Sigma_i$.  $$\Sigma_i=\frac{1}{N_i}S_W=\frac{1}{N_i}(x-m_i)(x-m_i)^T $$&lt;/li&gt;&#xA;&lt;li&gt;Make sure your scaling your features before you do your PCA/LDA.&lt;/li&gt;&#xA;&lt;li&gt;If the above doesn't fix your eigenvector verification step I suspect the problem is that the eigenvectors are scalled differently.  Remember from your linear algebra class that a single eigenvalue, $\lambda_i$, has infinitely many eigenvectors, each being a scalar multiple of the others.  $v_i=[1,2,3]$ and $v_i=[2,4,6]$ can both be the eigenvector of $\lambda_i$.  So while you may get different values when calculating values at any given step after the decomposition, the end result should be the same.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Below is a template I use for LDA data compression.  It assumes that you've split your data into a training and test set, the feature space has been properly scaled, and there are three classes in your label vector (you can adjust accordingly).  It plots the individual and cumulative &quot;discriminability&quot; of each linear discriminant and then relies on the &lt;code&gt;lda&lt;/code&gt; package in &lt;code&gt;sklearn&lt;/code&gt; to transform the feature space using the number of discriminants you intend on using (here I chose to use the first 2).  It also scales the within class scatter matrices by default.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;LINEAR DISCRIMINANT ANALYSIS&lt;/h1&gt;&#xA;&#xA;&lt;h1&gt;calculate mean vectors&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean_vecs = []&#xA;for label in range(1, 4):&#xA;    mean_vecs.append(np.mean(X_train_std[y_train==label], axis=0))&#xA;    print('MV %s: %s\n' %(label, mean_vecs[label-1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;calculate within-class scatter matrix&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = X_train_std.shape[1]&#xA;S_W = np.zeros((d, d))&#xA;for label, mv in zip(range(1, 4), mean_vecs):&#xA;    class_scatter = np.cov(X_train_std[y_train==label].T)&#xA;    S_W += class_scatter&#xA;print('Scaled within-class scatter matrix: %sx%s' % (S_W.shape[0], S_W.shape[1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;calculate between-class scatter matrix&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean_overall = np.mean(x_train_std, axis=0)&#xA;S_B = np.zeros((d, d))&#xA;for i, mean_vec in enumerate(mean_vec):&#xA;    n = X_train_std[y_train==i+1, :].shape[0]&#xA;    mean_vec = mean_vec.reshape(d, 1)&#xA;    mean_overall = mean_overall.reshape(d, 1)&#xA;    S_B += n * (mean_vec - mean_overall).dot((mean_vec - mean_overall).T)&#xA;print('Between-class scatter matrix: %sx%s' % (S_B.shape[0], S_B.shape[1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;eigen decomposition&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eigen_vals, eigen_vecs = np.linalg.eigh(np.linalg.inv(S_W).dot(S_B))&#xA;eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:,i]) for i in range(len(eigen_vals))]&#xA;eigen_pairs = sorted(eigen_pairs, key=lambda k: k[0], reverse=True)&#xA;print('Eigendecomposition: \nEigenvalues in decreasing order:\n')&#xA;for eigen_val in eigen_pairs:&#xA;    print(eigen_val[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;plot discriminablithy and select number of linear discriminants&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tot = sum(eigen_vals.real)&#xA;discr = [(i / tot) for i in sorted(eigen_vals.real, reverse=True)]&#xA;cum_discr = np.cumsum(discr)&#xA;&#xA;plt.bar(range(1, 14), discr, alpha=0.5, align='center',&#xA;        label='individual &quot;discriminability&quot;')&#xA;plt.step(range(1, 14), cum_discr, where='mid',&#xA;         label='cumulative &quot;discriminability&quot;')&#xA;plt.ylabel('&quot;discriminability&quot; ratio')&#xA;plt.xlabel('Linear Discriminants')&#xA;plt.ylim([-0.1, 1.1])&#xA;plt.legend(loc='best')&#xA;plt.tight_layout()&#xA;plt.show()&#xA;&#xA;from sklearn.lda import LDA&#xA;&#xA;lda = LDA(n_components=2)&#xA;x_train_lda = lda.fit_transform(X_train_std)&#xA;x_test_lda = lda.transform(X_test_std)&#xA;print('Features projected onto %d-dimensional LD subspace' % lda.n_components)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16936" LastEditorUserId="16936" LastEditDate="2016-05-05T23:08:57.197" LastActivityDate="2016-05-05T23:08:57.197" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11609" PostTypeId="1" CreationDate="2016-05-05T23:43:27.647" Score="3" ViewCount="25723" Body="&lt;p&gt;I have multiple zip files containing two types of files(A.csv &amp;amp; B.csv)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;/data/jan.zip --&gt; contains A.csv &amp;amp; B.csv&lt;br&gt;&#xA;/data/feb.zip --&gt; contains A.csv &amp;amp; B.csv&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to read the contents of all the A.csv files inside all the zip files using pyspark.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; textFile = sc.textFile(&quot;hdfs://&amp;lt;HDFS loc&amp;gt;/data/*.zip&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone tell me how to get the contents of A.csv files into an RDD?&lt;/p&gt;&#xA;" OwnerUserId="18477" LastEditorUserId="21" LastEditDate="2016-05-10T13:59:02.417" LastActivityDate="2017-02-15T23:48:16.643" Title="How to read contents of a CSV file inside zip file using spark (python)" Tags="|python|apache-hadoop|apache-spark|" AnswerCount="2" CommentCount="1" ClosedDate="2017-03-15T10:30:33.267" ContentLicense="CC BY-SA 3.0" />
  <row Id="11611" PostTypeId="1" AcceptedAnswerId="11612" CreationDate="2016-05-06T01:47:14.587" Score="2" ViewCount="333" Body="&lt;p&gt;The Wikipedia &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; on &lt;em&gt;Logistic Regression&lt;/em&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Logistic regression is an alternative to Fisher's 1936 method, linear&#xA;  discriminant analysis.  If the assumptions of linear discriminant&#xA;  analysis hold, application of Bayes' rule to reverse the conditioning&#xA;  results in the logistic model, so if linear discriminant assumptions&#xA;  are true, logistic regression assumptions must hold.  The converse is&#xA;  not true, so the logistic model has fewer assumptions than&#xA;  discriminant analysis and makes no assumption on the distribution of&#xA;  the independent variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Could someone help me to understand what the assumptions of linear discriminant are, an example of where they hold, and how application of Bayes' rule results in the &lt;em&gt;Logistic Model&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be correct to say that Logistic Regression is always the preferred choice?  Are there conditions where Linear Discriminant Analysis is to be preferred?&lt;/p&gt;&#xA;" OwnerUserId="16244" LastActivityDate="2016-05-06T02:22:05.400" Title="When to use Linear Discriminant Analysis or Logistic Regression" Tags="|logistic-regression|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11612" PostTypeId="2" ParentId="11611" CreationDate="2016-05-06T02:22:05.400" Score="2" Body="&lt;p&gt;This question was asked and answered at the Cross-Validate SE. The answer is a few years old but it is still useful. If you want you can &lt;a href=&quot;https://stats.stackexchange.com/questions/95247/logistic-regression-vs-lda-as-two-class-classifiers&quot;&gt;click here&lt;/a&gt; to find their answer.&lt;/p&gt;&#xA;" OwnerUserId="17959" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-05-06T02:22:05.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11613" PostTypeId="2" ParentId="11609" CreationDate="2016-05-06T03:49:25.060" Score="-1" Body="&lt;p&gt;&lt;a href=&quot;https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; website would help you out. In the topic called &lt;strong&gt;Writing a Spark Application&lt;/strong&gt;, they've described reading file contents from a zip folder. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="9966" LastActivityDate="2016-05-06T03:49:25.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11614" PostTypeId="1" CreationDate="2016-05-06T09:21:39.373" Score="-1" ViewCount="980" Body="&lt;p&gt;I have a file (access.log) from apache server which is my dataset. and I want to train a model on what is &quot;Normal&quot; and &quot;Abnormal&quot; in the file, so I manually added to some line a field that takes one of the two labels (normal | abnormal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I put &quot;Normal&quot; ==&gt; &quot;0&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &quot;Abnormal&quot; ==&gt; &quot;1&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I created a decision tree classifier &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Features = [[&quot;GET /robots.txt HTTP/1.1&quot;, 55],[&quot;POST /xmlrpc.php HTTP/1.0&quot;, 370]]&#xA;Labels = [1, 0]&#xA;clf = tree.DecisionTreeClassifier()&#xA;clf = clf.fit(Features, Labels)&#xA;print clf.predict([&quot;GET /robots.txt HTTP/1.1&quot;,55])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that I can't do it this way as I am working with String values so the compiler shows an error saying: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;could not convert string to float: GET /robots.txt HTTP/1.1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I really want to find a solution to manipulate the log file lines,  extract data for classifier, then train the model and predict the correct decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="18458" LastActivityDate="2016-05-06T10:10:45.853" Title="Reading from log file and train a model for prediction" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11615" PostTypeId="2" ParentId="11614" CreationDate="2016-05-06T10:10:45.853" Score="0" Body="&lt;p&gt;Decision Trees cannot deal with strings. To be able to learn from your log lines, you need to represent what is happening at the logs. You can use both categorical features and numerical features. This process is called feature engineering, which is part of preprocessing. I don't know what the number after yoru header represents but this could very well be a feature if it influences if it counts as 'Normal' or 'Abnormal'. I would extract a categorical feature for the type of request, like GET and POST. You could add a granular level that tries to represent what document the client requests. Just feeding strings to Machine Learning algorithms will not work (un)fortunately.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-06T10:10:45.853" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11617" PostTypeId="1" CreationDate="2016-05-06T14:13:53.787" Score="2" ViewCount="1406" Body="&lt;p&gt;i'm trying to use TensorFlow on a dataset with has a few Categorical variables. I've encoded them with dummies but it looks like its causing trouble and TF is complaining that the dataset is not dense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is the reason for the error something totally different ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to run a simple Neural Network Model with 1 hidden layer with stochastic gradient. The code was working when the input was numeric variables (images of digits from MNIST)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;-------------------------------------------------------------------------- ValueError                                Traceback (most recent call&#xA;  last)  in ()&#xA;       37     return(test_acc,round(l,5))&#xA;       38 &#xA;  ---&gt; 39 define_batch(0.005)&#xA;       40 run_batch()&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt; in define_batch(beta)&#xA;       11                                         shape=(batch_size, num_var))&#xA;       12       tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))&#xA;  ---&gt; 13       tf_valid_dataset = tf.constant(valid_dataset)&#xA;       14       tf_test_dataset = tf.constant(test_dataset)&#xA;       15 &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/ops/constant_op.pyc&#xA;  in constant(value, dtype, shape, name)&#xA;      159   tensor_value = attr_value_pb2.AttrValue()&#xA;      160   tensor_value.tensor.CopyFrom(&#xA;  --&gt; 161       tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))&#xA;      162   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)&#xA;      163   const_tensor = g.create_op(&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.pyc&#xA;  in make_tensor_proto(values, dtype, shape)&#xA;      320       nparray = np.array(values, dtype=np_dt)&#xA;      321       if list(nparray.shape) != _GetDenseDimensions(values):&#xA;  --&gt; 322         raise ValueError(&quot;Argument must be a dense tensor: %s&quot; % values)&#xA;      323     # python/numpy default float type is float64. We prefer float32 instead.&#xA;      324     if (nparray.dtype == np.float64) and dtype is None:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;ValueError: Argument must be a dense tensor:         Tuesday &#xA;  Wednesday  Thursday  Friday  Saturday  Sunday  CENTRAL  \ 736114&lt;br&gt;&#xA;  0.0        0.0       0.0     0.0       1.0     0.0      0.0    437148      0.0        0.0       1.0     0.0       0.0     0.0      0.0    605041      0.0        0.0       0.0     0.0       0.0     0.0      0.0    444608      0.0        0.0       0.0     0.0       1.0     0.0      0.0    695549      0.0        0.0       0.0     0.0       1.0     0.0      0.0    662807      0.0        0.0       0.0     1.0       0.0     0.0      0.0    238635      0.0        0.0       0.0     0.0       0.0     1.0      0.0    549524      0.0        0.0       0.0     1.0       0.0     0.0      0.0    705478      1.0        0.0       0.0     0.0       0.0     0.0      0.0    557716      0.0        0.0       0.0     1.0       0.0     0.0      0.0    41808       0.0        0.0       0.0     0.0       0.0     1.0      0.0    227235      1.0        0.0       0.0     0.0       0.0     0.0      0.0    848719      0.0        0.0       0.0     0.0       0.0     0.0      0.0    731202      0.0        0.0       0.0     0.0       1.0     0.0      0.0    467516      1.0        0.0       0.0     0.0       0.0     0.0      1.0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is an excerpt from the code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Adding regularization to the 1 hidden layer network&#xA;def define_batch(beta):&#xA;    batch_size = 128&#xA;    num_RELU =256&#xA;    graph1 = tf.Graph()&#xA;    with graph1.as_default():&#xA;&#xA;      # Input data. For the training data, we use a placeholder that will be fed&#xA;      # at run time with a training minibatch.&#xA;      tf_train_dataset = tf.placeholder(tf.float32,&#xA;                                        shape=(batch_size, num_var))&#xA;      tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))&#xA;      tf_valid_dataset = tf.constant(valid_dataset)&#xA;      tf_test_dataset = tf.constant(test_dataset)&#xA;&#xA;      # Variables.&#xA;      weights_RELU = tf.Variable(&#xA;        tf.truncated_normal([num_var, num_RELU]))&#xA;      biases_RELU = tf.Variable(tf.zeros([num_RELU]))&#xA;      weights_layer1 = tf.Variable(&#xA;        tf.truncated_normal([num_RELU, num_labels]))&#xA;      biases_layer1 = tf.Variable(tf.zeros([num_labels]))&#xA;&#xA;      # Training computation.&#xA;      logits_RELU = tf.matmul(tf_train_dataset, weights_RELU) + biases_RELU&#xA;      RELU_vec = tf.nn.relu(logits_RELU)&#xA;      logits_layer = tf.matmul(RELU_vec, weights_layer1) + biases_layer1                  &#xA;      # loss = tf.reduce_mean(&#xA;      #        tf.nn.softmax_cross_entropy_with_logits(logits_layer, tf_train_labels))&#xA;      cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits_layer, tf_train_labels,name=&quot;cross_entropy&quot;)&#xA;      l2reg = tf.reduce_sum(tf.square(weights_RELU))+tf.reduce_sum(tf.square(weights_layer1))&#xA;      beta = 0.005&#xA;      loss = tf.reduce_mean(cross_entropy+beta*l2reg)&#xA;&#xA;  # Optimizer.&#xA;      optimizer = tf.train.GradientDescentOptimizer(0.3).minimize(loss)&#xA;&#xA;      # Predictions for the training, validation, and test data.&#xA;      train_prediction = tf.nn.softmax(logits_layer)&#xA;      valid_prediction = tf.nn.softmax(&#xA;        tf.matmul(tf.nn.relu((tf.matmul(tf_valid_dataset, weights_RELU) + biases_RELU)),weights_layer1)+biases_layer1)&#xA;&#xA;      test_prediction =tf.nn.softmax(&#xA;        tf.matmul(tf.nn.relu((tf.matmul(tf_test_dataset, weights_RELU) + biases_RELU)),weights_layer1)+biases_layer1)&#xA;&#xA;import datetime&#xA;&#xA;startTime = datetime.datetime.now() &#xA;&#xA;num_steps = 301 # change to 3001&#xA;&#xA;def run_batch():&#xA;&#xA;    with tf.Session(graph=graph1) as session:&#xA;&#xA;      tf.initialize_all_variables().run()&#xA;      print(&quot;Initialized&quot;)&#xA;      for step in range(num_steps):&#xA;        # Pick an offset within the training data, which has been randomized.&#xA;        # Note: we could use better randomization across epochs.&#xA;        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)&#xA;        # Generate a minibatch. &#xA;        batch_data = train_dataset[offset:(offset + batch_size), :]&#xA;        batch_labels = train_labels[offset:(offset + batch_size), :]&#xA;        # Prepare a dictionary telling the session where to feed the minibatch.&#xA;        # The key of the dictionary is the placeholder node of the graph to be fed,&#xA;        # and the value is the numpy array to feed to it.&#xA;        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}&#xA;        _, l, predictions, logits = session.run(&#xA;          [optimizer, loss,train_prediction,logits_RELU], feed_dict=feed_dict)&#xA;        if (step % 500 == 0):&#xA;          print(&quot;Minibatch loss at step %d: %f&quot; % (step, l))&#xA;          print(&quot;Minibatch accuracy: %.1f%%&quot; % accuracy(predictions, batch_labels))&#xA;          print(&quot;Validation accuracy: %.1f%%&quot; % accuracy(&#xA;            valid_prediction.eval(), valid_labels))&#xA;      test_acc = accuracy(test_prediction.eval(), test_labels)&#xA;      print(&quot;Test accuracy: %.1f%%&quot; % test_acc)&#xA;&#xA;      print('loss=%s' % l)&#xA;    x = datetime.datetime.now() - startTime&#xA;    print(x)&#xA;    return(test_acc,round(l,5))&#xA;&#xA;define_batch(0.005)&#xA;run_batch()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: &#xA;@gdhal thanks for looking at it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_dataset&lt;/code&gt; is a pandas dataframe&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_dataset.columns&#xA;Index([u'Tuesday', u'Wednesday', u'Thursday', u'Friday', u'Saturday',&#xA;       u'Sunday', u'CENTRAL', u'INGLESIDE', u'MISSION', u'NORTHERN', u'PARK',&#xA;       u'RICHMOND', u'SOUTHERN', u'TARAVAL', u'TENDERLOIN', u' 3H - 4H',&#xA;       u' 5H - 6H', u' 7H - 8H', u' 9H - 10H', u'11H - 12H', u'13H - 14H',&#xA;       u'15H - 16H', u'17H - 18H', u'19H - 20H', u'21H - 22H', u'23H - 0H',&#xA;       u'Xnorm', u'Ynorm', u'Hournorm'],&#xA;      dtype='object')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;all the variables are dummies (taking 0 or 1 values) except the last 3 variables (Xnorm, Ynorm, and Hournorm) which are numerical values normalized to [0,1] interval. &lt;code&gt;valid_dataset&lt;/code&gt; and &lt;code&gt;test_dataset&lt;/code&gt; have the same format&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_labels&lt;/code&gt; is a pandas series&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_labels.describe()&#xA;&#xA;count            790184&#xA;unique               39&#xA;top       LARCENY/THEFT&#xA;freq             157434&#xA;Name: Category, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;valid_labels&lt;/code&gt;, and &lt;code&gt;test_labels&lt;/code&gt; have the same format&lt;/p&gt;&#xA;" OwnerUserId="18354" LastActivityDate="2016-05-06T14:13:53.787" Title="TensorFlow and Categorical variables" Tags="|neural-network|tensorflow|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11618" PostTypeId="2" ParentId="11600" CreationDate="2016-05-06T14:34:13.867" Score="0" Body="&lt;p&gt;You can use &quot;download data to local memory&quot; checkbox in SQL Table widget, it should allow you to work with up to 1000000 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8Vskn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8Vskn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18489" LastActivityDate="2016-05-06T14:34:13.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11619" PostTypeId="1" AcceptedAnswerId="11621" CreationDate="2016-05-06T14:36:20.190" Score="62" ViewCount="73755" Body="&lt;p&gt;I've been thinking about the Recurrent Neural Networks (RNN) and their varieties and Convolutional Neural Networks (CNN) and their varieties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would these two points be fair to say:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use CNNs to break a component (such as an image) into subcomponents (such as an object in an image, such as the outline of the object in the image, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Use RNNs to create combinations of subcomponents (image captioning, text generation, language translation, etc.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would appreciate if anyone wants to point out any inaccuracies in these statements.  My goal here is to get a more clearer foundation on the uses of CNNs and RNNs.&lt;/p&gt;&#xA;" OwnerUserId="16244" LastActivityDate="2020-11-26T05:58:01.833" Title="RNN vs CNN at a high level" Tags="|machine-learning|neural-network|beginner|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11620" PostTypeId="1" CreationDate="2016-05-06T16:59:20.173" Score="2" ViewCount="553" Body="&lt;p&gt;I am looking for the implementation of Recursive Auto Encoders (RAE) in tensor flow python. I want to model English sentence representations from a sequence to sequence neural network model. RAE is proven to be one of the best choice to represent sentences in recent machine learning approaches. I googled and didn't find any tensorflow Recursive Auto Encoders (RAE) implementation resource, please help. And if there is no implementation of RAE in tensorflow then please share the links of Theano.&lt;/p&gt;&#xA;" OwnerUserId="16964" LastEditorUserId="471" LastEditDate="2016-05-07T12:17:04.153" LastActivityDate="2016-05-07T12:17:04.153" Title="Is there any implementation of Recursive Auto Encoders in Tensorflow?" Tags="|machine-learning|python|neural-network|tensorflow|autoencoder|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11621" PostTypeId="2" ParentId="11619" CreationDate="2016-05-06T17:36:36.590" Score="46" Body="&lt;p&gt;A CNN will learn to recognize patterns across space.  So, as you say, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and then learn to combine these components to recognize larger structures (e.g., faces, objects, etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could say, in a very general way, that a RNN will similarly learn to recognize patterns across time.  So a RNN that is trained to translate text might learn that &quot;dog&quot; should be translated differently if preceded by the word &quot;hot&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mechanism by which the two kinds of NNs represent these patterns is different, however.  In the case of a CNN, you are looking for the &lt;em&gt;same&lt;/em&gt; patterns on all the different subfields of the image.  In the case of a RNN you are (in the simplest case) feeding the hidden layers from the previous step as an additional input into the next step.  While the RNN builds up memory in this process, it is not looking for the same patterns over different slices of time in the same way that a CNN is looking for the same patterns over different regions of space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should also note that when I say &quot;time&quot; and &quot;space&quot; here, it shouldn't be taken too literally.  You could run a RNN on a single image for image captioning, for instance, and the meaning of &quot;time&quot; would simply be the order in which different parts of the image are processed.  So objects initially processed will inform the captioning of later objects processed.&lt;/p&gt;&#xA;" OwnerUserId="18416" LastActivityDate="2016-05-06T17:36:36.590" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11622" PostTypeId="1" CreationDate="2016-05-06T20:15:43.663" Score="-1" ViewCount="734" Body="&lt;p&gt;I'm an R user trying to learn python. I normally create functions and do loops but I 've never created an object. I wonder if there's something that I'm really missing. &lt;strong&gt;Can you give a useful example of OOP for data science?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="14691" LastActivityDate="2016-05-06T20:37:24.360" Title="OOP for data science" Tags="|r|python|algorithms|" AnswerCount="1" CommentCount="2" ClosedDate="2016-05-06T21:30:46.097" ContentLicense="CC BY-SA 3.0" />
  <row Id="11623" PostTypeId="2" ParentId="11622" CreationDate="2016-05-06T20:37:24.360" Score="1" Body="&lt;p&gt;Here is a &quot;little&quot; piece of python code, which is actually making an &quot;object&quot; of &quot;Network&quot;, which was later used as a multi-layered neural network (sizes is actually a list of number of neurons for each layer). The code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Network(object):&#xA;&#xA;def __init__(self, sizes):&#xA;    self.num_layers = len(sizes)&#xA;    self.sizes = sizes&#xA;    self.biases = [np.random.randn(y, 1) for y in sizes[1:]]&#xA;    self.weights = [np.random.randn(y, x)&#xA;                    for x, y in zip(sizes[:-1], sizes[1:])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In out Network object, there are some &quot;member variables&quot;, and if you have a slight idea about neural nets, then the names of these members clearly tell the purpose of these member variables.&#xA;Hope it was clear enough.&#xA;P.S: __init__() is actually a function, which acts as a &quot;constructor&quot;, i-e, it automatically gets executed whenever the object is created anywhere.&lt;/p&gt;&#xA;" OwnerUserId="18150" LastActivityDate="2016-05-06T20:37:24.360" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11624" PostTypeId="2" ParentId="11535" CreationDate="2016-05-06T20:57:13.567" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Quality&lt;/li&gt;&#xA;&lt;li&gt;Certificates (is &quot;for research use only&quot; ok for you?)&lt;/li&gt;&#xA;&lt;li&gt;There is no &quot;common&quot; or &quot;average&quot; genome, some of the prices are for bacterial some for human(4,6 x 10 ^6 bp (E.coli) vs. 3.3 x 10^9 bp(human))&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18502" LastActivityDate="2016-05-06T20:57:13.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11625" PostTypeId="1" AcceptedAnswerId="11626" CreationDate="2016-05-07T04:55:58.673" Score="1" ViewCount="77" Body="&lt;p&gt;In an RBM, if we represent the weights learned by the hidden units, they show that the neural net is learning basic shapes. For example, for the mnist dataset, they learn features of the numbers they are trying to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regular feed-forward net with one hidden layer, I can train the network to recognize digits, and it works, but when I try to visualize the hidden layer weights, I only see noise, no distinguishable feature. Why is that? Hasn't the network learned to recognize the digits?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="434" LastEditDate="2016-05-07T05:25:47.920" LastActivityDate="2016-05-07T06:48:55.867" Title="Hidden neuron representation of weights" Tags="|neural-network|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11626" PostTypeId="2" ParentId="11625" CreationDate="2016-05-07T06:48:55.867" Score="1" Body="&lt;p&gt;It has learnt to recognize the digits, but it might have put too much weight on single pixels. Try to add different amounts of L2 regularization or dropout and compare the visualizations of the weights. Adding some kind of regularization should make the net rely less on single / independent pixels and more on the inherent structure of the digits, giving you smoother weights / visualization.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-05-07T06:48:55.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11627" PostTypeId="1" AcceptedAnswerId="12484" CreationDate="2016-05-07T11:00:37.657" Score="3" ViewCount="215" Body="&lt;p&gt;I have a dataset of categorical data, and I need to cluster it without knowing k. I know algos for clustering data without knowing the number of centroids, like G-mean, but none works for categorial data. &#xA;I think that DBSCAN is also bad because of density.&#xA;My dataset is typically a marketing dataset, each row is a customer, and each column corresponds to an attribute, like eyes color. So, I need to binarize this dataset before any algorithm. Any ideas??&lt;/p&gt;&#xA;" OwnerUserId="18509" LastEditorUserId="11097" LastEditDate="2016-05-07T13:32:18.683" LastActivityDate="2016-06-29T07:43:24.897" Title="How do i cluster binarized categorial data, without knowing the number of clusters?" Tags="|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11628" PostTypeId="1" CreationDate="2016-05-07T12:37:35.947" Score="7" ViewCount="2999" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/OPTICS_algorithm&quot; rel=&quot;noreferrer&quot;&gt;OPTICS clustering algorithm&lt;/a&gt; defines&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{core-dist}_{\varepsilon,MinPts}(p)=\begin{cases}\text{UNDEFINED} &amp;amp; \text{if } |N_\varepsilon(p)| &amp;lt; MinPts\\ MinPts\text{-th smallest distance to } N_\varepsilon(p) &amp;amp; \text{otherwise}\end{cases}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} &amp;amp; \text{if } |N_\varepsilon(p)| &amp;lt; MinPts\\ \max(\text{core-dist}_{\varepsilon,MinPts}(p), \text{dist}(p,o)) &amp;amp; \text{otherwise}\end{cases}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;why isn't it simply&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} &amp;amp; \text{if } |N_\varepsilon(p)| &amp;lt; MinPts\\ \text{dist}(p,o) &amp;amp; \text{otherwise}\end{cases}$$&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2019-05-05T07:07:21.870" Title="Why does OPTICS use the core-distance as a minimum for the reachability distance?" Tags="|clustering|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11629" PostTypeId="1" CreationDate="2016-05-07T15:00:51.933" Score="0" ViewCount="68" Body="&lt;p&gt;I have historical data with time (hour, day, etc.) and region code linked to number of events in the integer format. I was starting to build a model in R to predict number of events given time and region. I explored SVM in R but seems like it is a non-starter with categorical variables. Any other approach which might of use here. Thanks for your response.   &lt;/p&gt;&#xA;" OwnerUserId="18512" LastActivityDate="2016-05-07T20:28:59.293" Title="How to work with predictor variables which are all categorical but the target variable is an integer?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11630" PostTypeId="2" ParentId="11628" CreationDate="2016-05-07T17:17:37.730" Score="5" Body="&lt;p&gt;There are a few ways of looking at this. To make this easier I'm going to work describe things in terms of HDBSCAN*, a succesor to OPTICS. HDBSCAN* is essentially the same as OPTICS by removes the $\varepsilon$ parameter (or, if you like, fixes $\varepsilon = \infty$). HDBSCAN* also has a different approach for cluster extraction, but that's not overly relevant here. Thus under HDBSCAN* we define&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\text{core-dist}_\mathrm{MinPts}(p) = \text{Distance the the MinPts nearest neighbor of } p$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\text{mutual-reachability-dist}_\mathrm{MinPts}(o, p) = \\ \qquad\max(\text{core-dist}_\mathrm{MinPts}(p), \text{core-dist}_\mathrm{MinPts}(o), \mathrm{dist}(p,o))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;much the same as OPTICS, but now we don't require any undefined values (we also include the core-distance of $o$ to ensure that mutual reachability is a metric).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can view both OPTICS and HDBSCAN* as an extension of DBSCAN. DBSCAN takes two parameters MinPts and $\epsilon$ (note that this is a different use of epsilon than OPTICS -- if we focus on HDBSCAN* which has no such parameter we can avoid confusion: epsilon will refer only to the parameter used in DBSCAN). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HDBSCAN* generates a complete hierarchy of clusterings for a range of possible $\epsilon$ values, and thus for any fixed $\epsilon$ value you want the clustering at that level in the hierarchy to be the clustering DBSCAN would give for that $\epsilon$ value (OPTICS works similarly, but simply constrains the range of value that $\epsilon$ may take). To be in a cluster in DBSCAN you must be a core-point (I'm going to ignore border points for now); that is, the point must have at least MinPts other points within a ball of radius $\epsilon$ of itself. If a point isn't a core-point it is noise. The HDBSCAN* (and OPTICS*) reachability-distance captures this distinction by ensuring a point is not joined into a cluster until the DBSCAN $\epsilon$ value is such that the point is within the relevant distance of the other points in the cluster &lt;em&gt;and&lt;/em&gt; the point is a core-point at that DBSCAN $\epsilon$ value. The alternative reachability-distance you suggest covers the first part, but not the second: noise points would be included in clusters as long as they were close enough -- and by transitive chaining this could result in clusters merging together prematurely via noise points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative way of looking at this is by comparing with other algorithms like Robust Single Linkage (which also produces a cluster hierarchy). Robust Single Linkage aims to improve single linkage clustering by making it more robust to noise. In single linkage at each distance-scale $r$ you form a graph by joining points with an edge if they are within $r$ of each other, and then the clustering at level $r$ of the hierarchy is simply the connected components of that graph. The problem is that single linkage is very susceptible to noise -- a point or two in the wrong place can join clusters together when perhaps they shouldn't be. Robust Single Linkage attempts to fix this by requiring a point to have at least $k$ neighbors within $\alpha r$ of it  before adding any edges to it. This discounts sparse points until larger distance-scales and makes the approach more robust to noise. It can be demonstrated that this approach provides convergence to the level set tree of the PDF that generated the data -- i.e. it does a good job of fixing the noise issues! If we fix $\alpha = 1$ and $k = \mathrm{MinPts}$ then we recover HDBSCAN*; if we fix $r \leq \varepsilon$ then we get essentially OPTICS. Again we are using core-distance to discount noise, and make the clustering more noise robust.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastEditorUserId="18513" LastEditDate="2016-05-07T17:21:51.437" LastActivityDate="2016-05-07T17:21:51.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11631" PostTypeId="2" ParentId="11628" CreationDate="2016-05-07T19:21:18.497" Score="3" Body="&lt;p&gt;If you use $dist$ instead of $\text{core-dist}$, then you get &lt;strong&gt;single-linkage clustering&lt;/strong&gt; (hierarchical agglomerative clustering), probably the oldest clustering method out there. Same happens if you set $minPts=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, you pretty much lose any notion of &quot;density&quot; in the algorithm.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-05-07T19:21:18.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11632" PostTypeId="1" CreationDate="2016-05-07T19:30:24.900" Score="4" ViewCount="1472" Body="&lt;p&gt;I'm totally new in machine learning. The first confusing concept is subspace. In multi label classification we have to share the subspace. What does one mean by that shared sub spaces?&lt;/p&gt;&#xA;" OwnerUserId="18516" LastEditorUserId="8820" LastEditDate="2016-05-08T08:59:06.083" LastActivityDate="2017-06-02T19:23:38.253" Title="What is a subspace and what is a shared subspace?" Tags="|machine-learning|classification|statistics|multilabel-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11633" PostTypeId="2" ParentId="11629" CreationDate="2016-05-07T20:28:59.293" Score="1" Body="&lt;p&gt;I think you need to create a &lt;a href=&quot;https://en.wikipedia.org/wiki/Dummy_variable_(statistics)&quot; rel=&quot;nofollow&quot;&gt;dummy variables&lt;/a&gt; from your categorical variables. Almost all machine learning algorithms can't work with categorical variables.&lt;/p&gt;&#xA;" OwnerUserId="16541" LastActivityDate="2016-05-07T20:28:59.293" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11634" PostTypeId="2" ParentId="11077" CreationDate="2016-05-08T02:54:00.490" Score="0" Body="&lt;p&gt;I would suggest try &lt;code&gt;doc2vec&lt;/code&gt; instead of word2vec. &lt;code&gt;doc2vec&lt;/code&gt; not only gives you word embeddings as given by &lt;code&gt;word2vec&lt;/code&gt; but it also gives you document embeddings. Using the document embeddings you can also do operations like &quot;which patients have similar notes&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Needless to say, with &lt;code&gt;doc2vec&lt;/code&gt; you can feed one document at a time.  &lt;/p&gt;&#xA;" OwnerUserId="995" LastActivityDate="2016-05-08T02:54:00.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11635" PostTypeId="2" ParentId="11568" CreationDate="2016-05-08T02:58:35.357" Score="0" Body="&lt;p&gt;If order of words matter, then I would say try tf-idf but as phrases. Take 2-word or 3-words as features and generate tf-idf for them and do similarity matching. &lt;/p&gt;&#xA;" OwnerUserId="995" LastActivityDate="2016-05-08T02:58:35.357" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11636" PostTypeId="1" CreationDate="2016-05-08T04:15:59.083" Score="6" ViewCount="1659" Body="&lt;p&gt;I need to purchase some GPUs, which I plan to use for training and using some neural networks (most likely with Theano and Torch). Which GPU specifications should I pay attention to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;one should make sure that the VRAM is large enough for one's application&lt;/li&gt;&#xA;&lt;li&gt;the more teraflops, the faster programs running exclusively on the GPU will run&lt;/li&gt;&#xA;&lt;li&gt;the higher the &lt;a href=&quot;https://en.wikipedia.org/wiki/Thermal_design_power&quot; rel=&quot;nofollow noreferrer&quot;&gt;thermal design power (TDP)&lt;/a&gt;, the higher the electricity bill&lt;/li&gt;&#xA;&lt;li&gt;the &lt;a href=&quot;https://en.wikipedia.org/wiki/CUDA#Supported_GPUs&quot; rel=&quot;nofollow noreferrer&quot;&gt;compute capability&lt;/a&gt;, which &lt;a href=&quot;https://stackoverflow.com/q/33760192/395857&quot;&gt;determines which version of CUDA and cuDNN one can use&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What else matter, and what else doesn't matter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., is it reasonable to assume that the number of sockets, clock speed and the number of cores do not bring any additional useful information (since we already consider the number of Tflops)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jvD0Q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jvD0Q.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-11-17T19:31:39.943" Title="What GPU specifications matter when training and using neural networks?" Tags="|neural-network|deep-learning|gpu|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11637" PostTypeId="2" ParentId="11636" CreationDate="2016-05-08T05:11:40.143" Score="3" Body="&lt;p&gt;Besides the criteria listed look for&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of cores&lt;/li&gt;&#xA;&lt;li&gt;The ability to do low precision arithmetic&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In practice you are probably best off just getting the latest generation NVIDIA card (currently 10xx).&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-05-08T07:58:40.050" LastActivityDate="2016-05-08T07:58:40.050" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11638" PostTypeId="2" ParentId="11636" CreationDate="2016-05-08T05:48:52.040" Score="2" Body="&lt;p&gt;It depends on the expected size of neural network you are thinking about. &#xA;If the DNN consists multiple layers with large default input and many fully connected layers then you need a GPU with large memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Memory is the most important factor , without enough memory, you GPU is useless.&#xA;To get an idea about the size of any DNN try to think about the number and size of fully connected layers. Of course the number of cores is important but again memory is more important &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-08T05:48:52.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11639" PostTypeId="2" ParentId="11632" CreationDate="2016-05-08T06:21:18.560" Score="0" Body="&lt;p&gt;Let me first explain an easier concept, which is the subset? &#xA;Subset is related &quot; in general&quot; to ensemble classifiers. In ensemble classifier, you combine the outputs of multiple outputs of different base learners &quot; weak learners&quot;. One of the important keys is having diversity. That different base learners have different abilities. One of the easiest way to achieve diversity is by using subset learning. Each time you train a base learner you sample from the training data, since this sampling process is randomly, these base learner will not be the same and thus you support diversity. &#xA;What about subspace?&#xA;In subspace, you use all samples, but each time you select different set of dimensions. &#xA;For example, let us say that each sample is a vector 1*100 &#xA;The first base learner can be trained by using all samples but only the first 10 dimensions. So the first base learner will be trained by using samples where each is 1*10.&#xA;By choosing random number of dimensions and random dimensions you support diversity &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-08T06:21:18.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11640" PostTypeId="1" CreationDate="2016-05-08T08:05:42.787" Score="1" ViewCount="1406" Body="&lt;p&gt;We have a classic structure of an online shop database (products, customers, sales) and we want to implement a Frequently bought together feature. Our software is in ASP.NET and we do not know PHP to reverse engineer how this is being done in Magento.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And all we need is a simple Frequently bought together (not with discounts like Magento offers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that this is machine learning and one of the more common ways is Jaccard coefficient. Is that the recommended way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main question however is what the View I create in the DB should look like? What is the structure of the table that Jaccard coefficient and other relevant tests require so that used best?&lt;/p&gt;&#xA;" OwnerUserId="18524" LastEditorUserId="21" LastEditDate="2016-05-08T09:31:06.487" LastActivityDate="2016-05-09T04:18:31.373" Title="Implementing Frequently bought together using a DB" Tags="|machine-learning|databases|jaccard-coefficient|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11641" PostTypeId="2" ParentId="11632" CreationDate="2016-05-08T08:30:49.910" Score="1" Body="&lt;p&gt;I don't know what you mean by &quot;In multi label classification we have to share the subspace.&quot;, but the concept of subspaces is fairly easy to explain.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In mathematics, a space is a set (sometimes called a universe) with some added structure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://en.wikipedia.org/wiki/Space_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common spaces are $\mathbb{R}$ (the number line), $\mathbb{R}^2$ (the plane) and $\mathbb{R}^3$ (the space we often think in, with $x,y,z$ axes). Lets assume our main space is $\mathbb{R}^3$. Our data are datapoints in this space. However, we have to write a report. This report has to be printed on paper. Hence we project the data points from a 3-dimensional space to a two-dimensional subspace. So a subspace is just a part of the original space.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-08T08:30:49.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11642" PostTypeId="1" CreationDate="2016-05-08T12:38:25.180" Score="0" ViewCount="251" Body="&lt;p&gt;Suppose, I am building a hotel recommendation system, that learns user profile based on his/her interaction with the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two classes, &quot;Like&quot; and &quot;Dislike&quot;. For example, a user likes 5 &quot;5-star&quot; hotels, likes 3 &quot;1-star&quot; hotels and dislikes 10 &quot;1-star&quot; hotels. Now, a Naive Bayes Classifier will put a new 5 star hotel in the &quot;Like&quot; class and a new 1 star hotel in &quot;Dislike&quot; class. Now, what I am trying to achieve is a bit different, I don't want to classify, instead I want to extract features/attributes more closely related with the &quot;Like&quot; class. For example, I need &quot;5-star&quot; attribute from the &quot;Like&quot; class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, obviously, I have many more attributes, than the hotel star rating. What I want to know is that, can Naive Bayes be modified to extract features from a class. Or are there any specialized algorithms that are more suited for this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have data in this form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;user_id | hotel_id | hotel_rating | hotel_attr1 | .... | hotel_attr(n)| preference  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;1       1        5 star        some value   some value    like&lt;/pre&gt;&#xA;&#xA;&lt;pre&gt;1       2        1 star        some value   some value    dislike&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another thing, If I want to extract features such as &quot;5 star&quot; and &quot;1 star&quot; from a class, Is the data formatted above correct? or instead of hotel_rating as the feature, I need 5_star and 1_star as the features, instead of them being values of the hotel_rating feature?&lt;/p&gt;&#xA;" OwnerUserId="18529" LastEditorUserId="18529" LastEditDate="2016-05-08T13:55:13.063" LastActivityDate="2016-05-09T12:30:18.490" Title="is this a classification or clustering problem?" Tags="|machine-learning|classification|clustering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11644" PostTypeId="1" CreationDate="2016-05-08T14:41:04.507" Score="1" ViewCount="412" Body="&lt;p&gt;We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, etc. Specifically, we check for multicollinearity, for a numeric/continuous variable, VIF (Variance Inflation Factors) etc. If we find that there is multicollinearity, we then drop one of the highly correlated features.&lt;/p&gt;&#xA;&lt;p&gt;My question is: what can be done with categorical variables? I mean if two categorical variables are correlated/associated does it mean I have to drop it? I am not clear how to handle categorical variables like we handle correlations between continuous variables.&lt;/p&gt;&#xA;&lt;p&gt;What do we mean by two factor variables are correlated or dependent or independent? What if there is a collinearity? How do you identify that collinearity? How do you deal with it?&lt;/p&gt;&#xA;" OwnerUserId="18139" LastEditorUserId="91724" LastEditDate="2021-04-10T07:35:42.843" LastActivityDate="2021-08-16T07:08:00.687" Title="Association between Categorical Variables and regression" Tags="|regression|data|categorical-data|pearsons-correlation-coefficient|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11645" PostTypeId="2" ParentId="11642" CreationDate="2016-05-08T19:36:01.890" Score="1" Body="&lt;p&gt;As you have two classes, like and dislike, it's a supervised and not a clustering problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why don't you just try naive bayes and decision trees?&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-05-08T19:36:01.890" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11647" PostTypeId="2" ParentId="11642" CreationDate="2016-05-09T01:42:23.933" Score="2" Body="&lt;p&gt;To me, what you described, sounds very much like &lt;strong&gt;collaborative filtering&lt;/strong&gt;. In collaborative filtering, the features of the user(usually individual preferences) and features of the item(e.g. type and attribute of hotels) are trained collaboratively, gradually improving each other through iterations and converge to an optimal. Take a look at Andrew Ng's lecture, you can get a very quick idea of how it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, collaborative filtering is a relatively basic technique for providing recommendations. Decision tree based models are more generally used in real life recommendations by big companies. &lt;/p&gt;&#xA;" OwnerUserId="18538" LastActivityDate="2016-05-09T01:42:23.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11648" PostTypeId="2" ParentId="11640" CreationDate="2016-05-09T04:18:31.373" Score="2" Body="&lt;p&gt;It is called Association Rule mining. You can implement a very basic version of this algorithm easily which works fine for small  or mid size datasets. What this algorithm does is basically finding the most frequent items in all the transactions. In your example, you can find the frequent items purchased together by customers in your online store. I implemented one in .Net for a course it just took me 2 days to understand and code it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The format of the data really depends on the implementation, for instance, my algorithm reads data in a text file that each line of the file is a tuple of items:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1,2,3&#xA;4,3,5&#xA;6,1,4,5,1&#xA;7,6,5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where each line is a transaction, and the numbers can be the items purchased by a user at one checkout. The algorithm needs a number which is called the threshold. It means the algorithm finds the patterns of items that repeated X times (it can be a percentage) in all of the transactions.&#xA;The result can be a couple of rows that each is the combination of items that were bought together in a transaction and managed to satisfy the threshold.&lt;/p&gt;&#xA;" OwnerUserId="12535" LastActivityDate="2016-05-09T04:18:31.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11649" PostTypeId="2" ParentId="11636" CreationDate="2016-05-09T05:56:33.513" Score="1" Body="&lt;p&gt;I would add a few more parameters to consider:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Size of cache memory associated with each cuda core.&lt;/li&gt;&#xA;&lt;li&gt;Video memory throughput (Gb/s), higher is better.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If a particular GPU's memory falls short for your dataset, one alternative is to work on a sample of your training set. Although this may not be possible in some cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another alternative to deal with large datasets is if your're comfortable porting algorithms by writing your own CUDA kernels, then you could overcome memory limitations by breaking up your dataset horizontally and swapping each part in/out till the entire dataset is processed. It is quite cumbersome to implement the backpropagation in this manner, and I haven't tried implementing it in this manner on a GPU, so it would be interesting to get feedback on your progress.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-05-09T05:56:33.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11650" PostTypeId="2" ParentId="11096" CreationDate="2016-05-09T06:27:31.647" Score="3" Body="&lt;p&gt;Since this dataset is already organised in a table, you can leverage standard SQL functions to perform a large part of the cleanup.&#xA;A record seems to be composed of 4 fields, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;university name, city, state, country&#xA;stanford law school - stanford - ca - united states of america&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could follow these steps to get a cleaner representation of this dataset:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Starting with the highest level (country), find unique values use it to identify all similar sounding words by listing soundex matches with itself (build a join query of the table with itself).&lt;/li&gt;&#xA;&lt;li&gt;Use these suggested similar matches to fix all mistakes by updating the names.&lt;/li&gt;&#xA;&lt;li&gt;Continue in this manner till you've fixed all four fields.&lt;/li&gt;&#xA;&lt;li&gt;Identify missing states by using the city name to query the correct state from the rest of the table; if state and country are missing for &quot;leland stanford junior university&quot;, then use the city name &quot;stanford&quot; to join it with itself and get the state/country name from the other records in the table.&lt;/li&gt;&#xA;&lt;li&gt;For the university name, identify all abbreviations using grep to search for words ending in a dot character. Replace them with full expansions.&lt;/li&gt;&#xA;&lt;li&gt;Next, breakup the university name into individual words and dump these into a temporary table in a single column. De-duplicate the values in the column so it only contains unique values.&lt;/li&gt;&#xA;&lt;li&gt;Run the same soundex matching join query as in step 1 to identify similar sounding names, append these suggested similar names in a second column of the temp table.&lt;/li&gt;&#xA;&lt;li&gt;Manually do a quick sanity check of what you've obtained as suggestions and delete any invalid matches.&lt;/li&gt;&#xA;&lt;li&gt;Write a procedure to replace the words in each university name with suggested replacements, and you will have a much cleaner dataset.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-05-09T06:27:31.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11651" PostTypeId="2" ParentId="9990" CreationDate="2016-05-09T06:48:52.000" Score="1" Body="&lt;p&gt;Firstly, a common chemical battery discharging from 95% to 35% in 1 second is undergoing a very deep discharge and its discharge profile will be very different from a slow steady discharge, e.g. 95% to 35% within 10 hours. Secondly, inferring from a very small snippet of the charge process (2% change) may work only if you know exactly which charge profile is being exhibited by the battery during that charge cycle, so that you can identify at which point of the charge cycle is the battery right now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most chemical batteries, the current consumption by the load determines the discharge profile and hence the capacity at a given point of time in the future. Since this is usually non-linear, and the slope varies at different points of the discharge profile, a linear regression would not be a good fit. I would recommend trying out a polynomial regression fit on the discharge profile. Multivariate adaptive regression splines should also work well. Similarly, the rate of charging determines the charge profile/characteristic curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could fit multiple models, one each for each type of charge profile (e.g. deep discharge, slow discharge, etc.) and then select the appropriate model based on the current status of the battery, to predict the future state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand this process for a lead acid battery, refer to these -&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scubaengineer.com/documents/lead_acid_battery_charging_graphs.pdf&quot; rel=&quot;nofollow&quot;&gt;http://scubaengineer.com/documents/lead_acid_battery_charging_graphs.pdf&lt;/a&gt; and,&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://batteryuniversity.com/learn/article/how_to_measure_state_of_charge&quot; rel=&quot;nofollow&quot;&gt;http://batteryuniversity.com/learn/article/how_to_measure_state_of_charge&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-05-10T04:52:36.767" LastActivityDate="2016-05-10T04:52:36.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11652" PostTypeId="1" CreationDate="2016-05-09T07:50:09.217" Score="2" ViewCount="429" Body="&lt;p&gt;I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. given a long list of company names, how can I find descriptions for these companies. Here are my current techniques, which works ok, but not great:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using Google Search API to and fetch summary from the results page (Google normally gives the company website as the first result when searching by company name)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fetch all data from the company's website (includes an about-page and a lot of other less relevant data)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the FullContact Company API to get any descriptions the company has on its social profiles (works great if the company has an online precense)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18543" LastActivityDate="2016-05-09T07:50:09.217" Title="How can I find company descriptions for a long list of companies?" Tags="|data|scraping|crawling|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11653" PostTypeId="2" ParentId="11642" CreationDate="2016-05-09T12:30:18.490" Score="0" Body="&lt;p&gt;Try  this link &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&lt;/a&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is called Boruta algorithm will give u relevant feature selection , It is done &lt;/p&gt;&#xA;&#xA;&lt;p&gt;in r language, It kind of gives you best features required for your analysis..its &lt;/p&gt;&#xA;&#xA;&lt;p&gt;like wrapping around random Forest&lt;/p&gt;&#xA;" OwnerUserId="18139" LastActivityDate="2016-05-09T12:30:18.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11654" PostTypeId="1" AcceptedAnswerId="11658" CreationDate="2016-05-09T12:52:11.827" Score="1" ViewCount="2357" Body="&lt;p&gt;When preparing my data for funneling into the Microsoft Association Rules algorithm, I was not sure if I should group by data by Transaction and Item, or have a record for every instance of an item in a transaction. Does the algorithm care and add weight if an item appears 3 times in a transaction  Or is it just looking for the existence of an item with another item, regardless of how many are present?&lt;/p&gt;&#xA;" OwnerUserId="18552" LastActivityDate="2016-05-09T15:04:20.643" Title="Does the count of items in a transaction matter to apriori?" Tags="|data-mining|association-rules|market-basket-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11655" PostTypeId="1" AcceptedAnswerId="12411" CreationDate="2016-05-09T13:22:02.683" Score="2" ViewCount="1861" Body="&lt;p&gt;Could someone please explain how the EasyEnsemble algorithm works? Im using it for a prediction model for imbalanced minority class.&lt;/p&gt;&#xA;&lt;p&gt;Please don't refer me to this paper, as it makes no sense to me.&#xA;&lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=4&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwj47L32ic3MAhUEjo4KHaYIA3cQFggxMAM&amp;amp;url=http%3A%2F%2Fcse.seu.edu.cn%2Fpeople%2Fxyliu%2Fpublication%2Ftsmcb09.pdf&amp;amp;usg=AFQjCNEgDOnCfYdg7RFurxN-o5vc1HQMMA&amp;amp;sig2=BuJH2l9w1M7d7hPVClxEMA&amp;amp;bvm=bv.121421273,d.c2E&quot; rel=&quot;nofollow noreferrer&quot;&gt;EasyEnsemble and Feature Selection for Imbalance Data Sets&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Im using the algorithm in Pandas, with the UnbalancedDataset library which is on GitHub  &lt;a href=&quot;https://github.com/glemaitre/UnbalancedDataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;UnbalancedDataset&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I get an array of matrices as O/P, I don't know how to use this in the end, to train with random forests.&lt;/p&gt;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="18098" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-06-26T00:11:54.507" Title="EasyEnsemble explaination" Tags="|python|random-forest|pandas|class-imbalance|ensemble-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11656" PostTypeId="1" AcceptedAnswerId="11776" CreationDate="2016-05-09T14:18:45.320" Score="7" ViewCount="5855" Body="&lt;p&gt;I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am now getting most results as class 1 when I test my model. I just tried to test it on the training set and this is what I got:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Without SMOTE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QMcb8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QMcb8.png&quot; alt=&quot;Without SMOTE&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With SMOTE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/id65Z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/id65Z.png&quot; alt=&quot;With SMOTE&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also tried hyperparameter optimisation, but that hasn't worked&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Used SMOTE implementation in &lt;code&gt;pandas&lt;/code&gt; with &lt;code&gt;UnbalancedDataset&lt;/code&gt; library&lt;/p&gt;&#xA;" OwnerUserId="18098" LastEditorUserId="13450" LastEditDate="2016-06-14T08:19:47.907" LastActivityDate="2016-06-14T08:19:47.907" Title="Overfitting for minority class after SMOTE w/ random forests" Tags="|python|random-forest|pandas|class-imbalance|hyperparameter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11657" PostTypeId="1" CreationDate="2016-05-09T14:46:47.170" Score="0" ViewCount="16255" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=Cluster_analysis&amp;amp;oldid=718035863&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Association rule learning is a method for discovering interesting relations between variables in large databases.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=Association_rule_learning&amp;amp;oldid=714074922&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So both, clustering and association rule mining (ARM), are in the field of unsupervised machine learning. Clustering is about the data points, ARM is about finding relationships between the attributes of those datapoints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I wonder if there are more relationships. For example, given a clustering, can this enhance / simplify ARM or vice versa?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-09T22:45:12.623" Title="What is the relationship between clustering and association rule mining?" Tags="|clustering|association-rules|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11658" PostTypeId="2" ParentId="11654" CreationDate="2016-05-09T15:04:20.643" Score="2" Body="&lt;p&gt;No it is not important and highly recommended to remove the duplicate items and sort the items in lexicographical order in each transaction. This is to improve the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In association rule mining, an item is frequent iff it is repeated in multiple transactions not in a single transaction. This is why you don't need to have duplicate items in a each transaction.&lt;/p&gt;&#xA;" OwnerUserId="12535" LastActivityDate="2016-05-09T15:04:20.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11659" PostTypeId="2" ParentId="11657" CreationDate="2016-05-09T15:17:44.193" Score="2" Body="&lt;p&gt;First of all, the problem of finding frequent itemsets (ARM) differs from the similarity search (clustering). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think about market baskets, clustering tries to find objects that have a large fraction of their baskets in common, the absolute number of those objects (basket occurences) is not of interest.&#xA;With ARM we are interested in the absolute number of baskets that contain a particular set of items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CLustering:  Allocates objects in such a way that objects in the same group (called a cluster) are more similar (given a distance metric) to each other than to those in other groups (clusters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ARM: Given many baskets (could be actual supermarket baskets) find which items inside a basket predict another item in the basket.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-clustering-and-association-rule-mining&quot; rel=&quot;nofollow&quot;&gt;Sources&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16772" LastActivityDate="2016-05-09T15:17:44.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11660" PostTypeId="2" ParentId="11657" CreationDate="2016-05-09T16:27:59.770" Score="-1" Body="&lt;p&gt;The paper&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Marie Plasse et al: Combined use of association rules mining and clustering methods to find relevant links between binary rare attributes in a large data set. &lt;a href=&quot;https://cedric.cnam.fr/fichiers/RC1172.pdf&quot; rel=&quot;nofollow&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;combines both, clustering and association rule mining. They could improve ARM by association rule mining. From the abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A method to analyse links between binary attributes in a large sparse data set is proposed. Initially the variables are clustered to&#xA;  obtain homogeneous clusters of attributes. Association rules are then mined in each cluster. A graphical comparison of some rule&#xA;  relevancy indexes is presented. It is used to extract best rules depending on the application concerned. The proposed methodology&#xA;  is illustrated by an industrial application from the automotive industry with more than 80 000 vehicles each described by more than&#xA;  3000 rare attributes.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-09T16:27:59.770" CommentCount="0" CommunityOwnedDate="2016-05-09T16:27:59.770" ContentLicense="CC BY-SA 3.0" />
  <row Id="11661" PostTypeId="1" CreationDate="2016-05-09T16:38:09.957" Score="1" ViewCount="4742" Body="&lt;p&gt;Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.palisade.com/risk/&quot; rel=&quot;nofollow&quot;&gt;http://www.palisade.com/risk/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@RISK (pronounced “at risk”) performs risk analysisusing Monte Carlo simulation to show you many possible outcomes in your spreadsheet model—and tells you how likely they are to occur. It mathematically and objectively computes and tracks many different possible future scenarios, then tells you the probabilities and risks associated with each different one. This means you can judge which risks to take and which ones to avoid, allowing for the best decision making under uncertainty. (Taken from their website)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main purposes for it would be for the construction company I currently work for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Able to provide more accurate models for job costing, e.g. time to construct certain parts, associated costs, etc. &#xA;&#xA; Provide more accurate models in regard to advertising and marketing, allowing us to better understand and predict what provides the better return. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this isn't the right forum for this question, please tell me where it needs to be asked. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;" OwnerUserId="18557" LastEditorUserId="18557" LastEditDate="2016-05-09T17:44:04.173" LastActivityDate="2017-07-12T18:25:51.007" Title="@RISK Vs R. Monte Carlo simulation in R?" Tags="|r|predictive-modeling|data|databases|weighted-data|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11664" PostTypeId="2" ParentId="11661" CreationDate="2016-05-09T19:20:49.477" Score="2" Body="&lt;p&gt;Monte Carlo simulations are very easy in R. The simplest approach is to write your own scripts that carry out the steps you need for your simulations. To construct these scripts you will need to understand what you are simulating, that is what is the distribution of outcomes, and what are you measuring about those outcomes. If you understand what you are simulating than writing these scripts is easy, but will require programming in R. If you are new to R and/or to programming then it will take some effort to get up to speed. In the end, you will probably understand your data, simulations, and results better. There are packages for Monte Carlo in R, but these are not packages that will do the simulations for you and would require as significant learning of R to be able to use (as much as writing your own scripts). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest reading more about conducting Monte Carlo simulations in R. There are many resources online and offline, such as this &lt;a href=&quot;https://books.google.com/books?id=WIjMyiEiHCsC&amp;amp;printsec=frontcover&amp;amp;dq=monte%20carlo%20simulation%20in%20r&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwj80p-T2s3MAhVJqB4KHSJhCA0Q6AEIHTAA#v=onepage&amp;amp;q=monte%20carlo%20simulation%20in%20r&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;book&lt;/a&gt; or this &lt;a href=&quot;http://ditraglia.com/Econ103Public/Rtutorials/Rtutorial4.html&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="17771" LastActivityDate="2016-05-09T19:20:49.477" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11665" PostTypeId="1" AcceptedAnswerId="11848" CreationDate="2016-05-09T19:59:33.310" Score="2" ViewCount="409" Body="&lt;p&gt;So, I have this data set where each instance is made of past 20 samples of 2 variables. Labels are next five samples. So every instance looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Instance1_feature = [s1_var1 s2_var1 ... s20_var1 s1_var2 s2_var2 ... s20_var2]&#xA;Instance1_label = [s21_var1 s22_var2 ... s25_var5 s21_var2 s22_var2 ... s25_var2]&#xA;&#xA;&#xA;Instance2_feature = [s2_var1 s3_var1 ... s21_var1 s2_var2 s3_var2... s21_var2]&#xA;Instance2_label = [s22_var1 s23_var2 ... s26_var5 s22_var2 s23_var2 ... s26_var2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All the data are categorical. var1 takes value in {1,2,3,4,5}; var2 takes value in  {6,7,8,9,10}&#xA;I am familiar with machine learning. Hence I am looking for a way to develop a model to predict labels for a new instance. I have reasons to believe that by looking at the past samples, it is possible to predict future samples. Its like stock prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help, links to code samples will be highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; This is a dataset I collected while driving. I want to see if I can predict next 5 seconds of my driving, given I have past 20 seconds of data. That's why I created the feature vectors and labels like this. The values  {1,2,3,4,5} and {6,7,8,9,10} are categorical, as in the variables are split into segments.&lt;/p&gt;&#xA;" OwnerUserId="10108" LastEditorUserId="10108" LastEditDate="2016-05-20T18:54:38.360" LastActivityDate="2016-05-24T11:55:09.513" Title="Obtain a model where each feature vector is past few samples and labels are future few samples?" Tags="|machine-learning|python|classification|regression|matlab|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11668" PostTypeId="1" CreationDate="2016-05-09T21:49:47.930" Score="-1" ViewCount="703" Body="&lt;p&gt;I am learning about Probabilistic Topic Models by reading this &lt;a href=&quot;http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; by D. Blei, watching this &lt;a href=&quot;https://www.youtube.com/watch?v=zeX8ZSG5uGE&quot; rel=&quot;nofollow&quot;&gt;video&lt;/a&gt;, and doing this exercise &lt;a href=&quot;https://eight2late.wordpress.com/2015/09/29/a-gentle-introduction-to-topic-modeling-using-r/&quot; rel=&quot;nofollow&quot;&gt;A Gentle Introduction to Topic Modeling&lt;/a&gt; in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the topics in my corpus are defined (by the algorithm), will the &quot;LDA&quot; package in R allow me to find the documents for a specific topic that was modeled? Put differently, how can I check which documents contributed to the formation of a certain topic in the final model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice on this topic or how to better formulate my question is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="18047" LastActivityDate="2017-04-05T03:15:05.627" Title="Using Topic Models in R" Tags="|machine-learning|r|text-mining|topic-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11669" PostTypeId="2" ParentId="11668" CreationDate="2016-05-09T22:17:31.170" Score="0" Body="&lt;p&gt;In the R code under the link for  &lt;a href=&quot;https://eight2late.wordpress.com/2015/09/29/a-gentle-introduction-to-topic-modeling-using-r/&quot; rel=&quot;nofollow&quot;&gt;A Gentle Introduction to Topic Modeling&lt;/a&gt; there is a snippet that does just what I'm looking for - i.e. it attributes each document in the corpus to one of 5 theme topics.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#write out results&#xA;#docs to topics&#xA;ldaOut.topics &amp;lt;- as.matrix(topics(ldaOut))&#xA;write.csv(ldaOut.topics,file=paste(&quot;LDAGibbs&quot;,k,&quot;DocsToTopics.csv&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This snippet quantifies the probability with which each document is associated with a certain theme:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#probabilities associated with each topic assignment&#xA;topicProbabilities &amp;lt;- as.data.frame(ldaOut@gamma)&#xA;write.csv(topicProbabilities,file=paste(&quot;LDAGibbs&quot;,k,&quot;TopicProbabilities.csv&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18047" LastActivityDate="2016-05-09T22:17:31.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11671" PostTypeId="2" ParentId="11598" CreationDate="2016-05-10T01:02:29.917" Score="1" Body="&lt;p&gt;Here is what i did. Not sure if this is correct. Will have update next week. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I plotted these 4 points on a graph with x1 and x2 on x and y axis respectively. since its a unit step function, which essentially has 1 or 0 output - i tried to draw a linear regression line such that all &quot;0&quot; result points are on 1 side and &quot;1&quot;  result points are on other. This gave me the possible weights w1 and w2 and i constructed an equation  &lt;a href=&quot;https://i.stack.imgur.com/ApE48.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ApE48.png&quot; alt=&quot;graph for the 4 points&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;w1= 1.5 and w2 = 0.5 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then i made a table to check if the equation provided the actual results &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gpkAt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gpkAt.png&quot; alt=&quot;excel table to check the calculations&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i started adjusting the w1 and w2 to try to &quot;fit&quot; to training data but one data was just not working!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i could bring the error down at w1=1 and w2 =0.39&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then - the threshold value, T &#xA;it should be in such a way that the z = x1w1+x2w2 + T and y=0 if z&amp;lt;0 and y=1 if z&amp;lt;=0. The value of T should change the results of (0.2,-0.5 ) to fit but not change the result of other points. So i started at -0.4 and worked my way up to -0.44. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Final equation came out to be.&#xA;y = x1 + 0.39x2 -0.44&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This fits the current training data.&lt;/p&gt;&#xA;" OwnerUserId="18452" LastActivityDate="2016-05-10T01:02:29.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11672" PostTypeId="1" CreationDate="2016-05-10T04:02:10.363" Score="3" ViewCount="202" Body="&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt;: I have a huge dataset which is more than &lt;em&gt;200GBs&lt;/em&gt; in size. The dataset contains around 200 columns (predictors). The task at hand is to &lt;strong&gt;predict which product to promote to the customer so that he/she ultimately buys it and thus maximizes the revenue of the company&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a cross selling situation but I need to look at product recommendation as well as revenue maximization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand, revenue maximization would require a regression model. (&lt;em&gt;Correct me if I'm wrong&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure &lt;strong&gt;how to recommend products&lt;/strong&gt; since there are more than 2000 unique products. (Dummy coding would require tremendous amount of time and resources I feel).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the sheer size of the data, I am planning to use Python for handling the data. &lt;em&gt;(Suggestions about R also welcome)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;PS: Forgive me if the problem seems too basic, but I have just started learning&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[UPDATE]:&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The data is in long(narrow) format&lt;/li&gt;&#xA;&lt;li&gt;I can also use R in order to tackle this&lt;/li&gt;&#xA;&lt;li&gt;Products are identified by their unique product ID (2000+ unique product IDs)&lt;/li&gt;&#xA;&lt;li&gt;Headers:&#xA;&lt;code&gt;date | time | pid | cust_id | ... | amount | tax | net_revenue&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Net revenue: Continuous variable&lt;/li&gt;&#xA;&lt;li&gt;Product ID: Continuous but to be treated as nominal&lt;/li&gt;&#xA;&lt;li&gt;Cust_id: Continuous but to be treated as nominal&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18369" LastEditorUserId="18369" LastEditDate="2016-05-11T09:38:46.637" LastActivityDate="2016-05-25T22:26:24.620" Title="What type of a problem is this?" Tags="|machine-learning|r|python|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11673" PostTypeId="2" ParentId="11672" CreationDate="2016-05-10T04:31:27.887" Score="0" Body="&lt;p&gt;Your question right now is really vague. However, if I had to make a recommendation based on what you have written I would suggest doing a market basket analysis to determine which products are commonly purchased together. Once you understand which products are purchased together you can do a regression analysis of the combos found in the market basket analysis in order to predict revenue maximization &lt;/p&gt;&#xA;" OwnerUserId="17959" LastActivityDate="2016-05-10T04:31:27.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11674" PostTypeId="2" ParentId="11628" CreationDate="2016-05-10T08:33:47.557" Score="2" Body="&lt;p&gt;When using OPTICS, one wants a reachability-plot as output from which one can read the number of clusters depending on $\varepsilon$. If $\text{dist}$ was used instead of $\text{reachability-dist}$, then some points would have a lower distance in the reachability plot. This could lead to the wrong conclusion that there would be more clusters for some small $\varepsilon$.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-10T08:33:47.557" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11675" PostTypeId="2" ParentId="11115" CreationDate="2016-05-10T08:59:00.190" Score="2" Body="&lt;p&gt;Best by cross validation&#xA;The whole data can be divided into training and testing. &#xA;You can not touch the testing data set for any kind of training. Keep it away!! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the training data you can use it freely to specify the best classification model. But how to know if the current setting is the best one, you do cross validation by using only training data set, dividing the training data set into k folds. Based on this cross validation result you select the setting that gives the best result. &#xA;Best by cross validation &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2016-05-10T09:04:43.957" LastActivityDate="2016-05-10T09:04:43.957" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11676" PostTypeId="2" ParentId="11672" CreationDate="2016-05-10T09:18:41.727" Score="1" Body="&lt;p&gt;If I understand your question correctly, you need to recommend one product out of 2000. You want to pick the one with the highest revenue expectation. This can be decomposed in the probability of making the sale times the revenue given that is a sale. The revenue for a sale is known, the probability is what is needed. I assume you have historic data of which customers (with the given features) bought which of these products. Now you can train regression models to estimate the probability of a sale for a specific product. Logistic Regression and Neural Networks with a sigmoid activation functions are well suited for unbiased probability estimates, by optimizing the log-loss cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this amount of data I propose to start on a subset of your data and using Logistic Regression. Since this is a generalized linear model you will likely need to do some feature engineering to increase performance. Since you mentioned you use Python, scikit-learn has a lot of functions for Logistic Regression optimization and feature engineering so take a look at that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have trained 2000 models, for a new recommendation you would run the features through the models, get all the probabilities, multiply them with revenue corresponding to this product and pick the highest one.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-10T09:18:41.727" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11677" PostTypeId="2" ParentId="8847" CreationDate="2016-05-10T09:21:14.420" Score="1" Body="&lt;p&gt;As Jeremy Barnes and Jamesmf said, you can use any machine learning algorithms to deal with the problem. They are powerful and could identify the features automatically. You just need to feed the algorithm the correct training data. Since it is needed to work on images, convolution neural networks will be a better option for you .&lt;/p&gt;&#xA;&lt;p&gt;This is a good tutorial for learning about the convolution neural network. You could download the code also and could change according to your problem definition. But you need to learn python and theano library for the processing and you will get good &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorials&lt;/a&gt; for that too&lt;/p&gt;&#xA;" OwnerUserId="18577" LastEditorUserId="98307" LastEditDate="2020-08-03T10:25:09.250" LastActivityDate="2020-08-03T10:25:09.250" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11678" PostTypeId="1" AcceptedAnswerId="11679" CreationDate="2016-05-10T12:38:18.240" Score="11" ViewCount="6941" Body="&lt;p&gt;We use &lt;code&gt;.cache()&lt;/code&gt; on RDD for persistent caching of an dataset, My concern is when this cached will be expired?.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt = sc.parallelize([2, 3, 4, 5, 6])&#xA;dt.cache()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="18375" LastEditDate="2017-12-09T10:27:13.007" LastActivityDate="2019-05-21T07:15:20.433" Title="When does cache get expired for a RDD in pyspark?" Tags="|apache-spark|pyspark|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11679" PostTypeId="2" ParentId="11678" CreationDate="2016-05-10T13:01:07.573" Score="13" Body="&lt;p&gt;It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system. &lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-10T13:01:07.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11680" PostTypeId="2" ParentId="11678" CreationDate="2016-05-10T14:09:24.340" Score="6" Body="&lt;p&gt;In addition to Jan's answer, I would like to point out that serialized RDD &lt;strong&gt;storage(/caching) works much better than normal RDD caching for large datasets&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also helps optimize garbage collection, in case of large datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, from the spark docs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When your objects are still too large to efficiently store despite&#xA;  this tuning, a much simpler way to reduce memory usage is to store&#xA;  them in serialized form, using the serialized StorageLevels in the RDD&#xA;  persistence API, such as MEMORY_ONLY_SER. Spark will then store each&#xA;  RDD partition as one large byte array. The only downside of storing&#xA;  data in serialized form is slower access times, due to having to&#xA;  deserialize each object on the fly. We highly recommend using Kryo if&#xA;  you want to cache data in serialized form, as it leads to much smaller&#xA;  sizes than Java serialization (and certainly than raw Java objects).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-05-10T14:09:24.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11682" PostTypeId="1" CreationDate="2016-05-10T16:35:06.310" Score="2" ViewCount="595" Body="&lt;p&gt;I have a XML file has this structure (not exactly a tree though)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;posthistory&amp;gt;&#xA;&amp;lt;row Id=&quot;1&quot; PostHistoryTypeId=&quot;2&quot; PostId=&quot;1&quot; &#xA;RevisionGUID=&quot;689cb04a-8d2a-4fcb-b125-bce8b7012b88&quot; &#xA;CreationDate=&quot;2015-01-27T20:09:32.720&quot; UserId=&quot;4&quot; Text=&quot;I just got a &#xA;pound of microroasted, local coffee and am curious what the optimal &#xA;way to store it is (what temperature, humidity, etc)&quot; /&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using apache pig to extract just the &quot;Text&quot; part using this code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grunt&amp;gt; A = load 'hdfs:///parsingdemo/PostHistory.xml' using &#xA;org.apache.pig.piggybank.storage.XMLLoader('posthistory') as(x:chararray);&#xA;&#xA;grunt&amp;gt; result = foreach A generate XPath(x, 'posthistory/Text');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this returns &quot;()&quot; (null)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon examining the XML file, I learnt that my XML file should be in this format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;root&amp;gt;&#xA;  &amp;lt;child&amp;gt;&#xA;    &amp;lt;subchild&amp;gt;.....&amp;lt;/subchild&amp;gt;&#xA;  &amp;lt;/child&amp;gt;&#xA;&amp;lt;/root&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But my XML data file (stackoverflow data dump actually) is not in this format. Is there a way the tree structure can be imposed? what is wrong with my pig query?&lt;/p&gt;&#xA;" OwnerUserId="6443" LastActivityDate="2016-06-28T17:57:49.450" Title="unable to parse XML in pig" Tags="|data-cleaning|apache-hadoop|apache-pig|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11683" PostTypeId="1" CreationDate="2016-05-10T18:21:44.037" Score="3" ViewCount="106" Body="&lt;p&gt;I have a pretty basic question and I was hoping someone could help me. I’m not a math person and I’m fairly new to mahout so I’m looking for a poor’s man explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a typical order recommendation system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a database with around 699,445 orders.&#xA;These orders have items which were “purchased”.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran the following mahout command:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;mahout itemsimilarity --input /mnt/p1.csv --output ./output --similarityClassname SIMILARITY_LOGLIKELIHOOD --booleanData TRUE --threshold 0.9&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I decided to spot-check the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took the following line from the output file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;58331   120216  0.9705375406679205&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my input file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1540 orders  have product 58331&#xA;35 orders have product 120216&#xA;10 orders have both (58331 and 120216)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Putting in Ted Dunning term’s  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k_11 = 10   b_12 = 25&#xA;k_21 = 1531 k_22= 697889&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The similarity number generated by the loglikehood algorithm between 58331 and 120216 is 0.9705375406679205.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)  What does that mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)  Should I recommend 58331 when someone orders 120216 ? Should I recommend 120216 when someone orders 58331?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) How do I calculate the entropy used in the LLR formula?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thx a lot&lt;/p&gt;&#xA;" OwnerUserId="18588" LastEditorUserId="381" LastEditDate="2016-05-10T19:35:04.920" LastActivityDate="2016-05-10T19:35:04.920" Title="Interpretation of Similarity Number generated by LogLikehood in Mahout" Tags="|apache-mahout|recommender-system|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11684" PostTypeId="2" ParentId="5121" CreationDate="2016-05-11T03:11:09.933" Score="32" Body="&lt;p&gt;The &lt;a href=&quot;https://datascience.stackexchange.com/a/5128/29575&quot;&gt;answer from saq7&lt;/a&gt; is wrong, as well as not answering the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;∥A∥ means the &lt;span class=&quot;math-container&quot;&gt;$L2$&lt;/span&gt; norm of &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt;, i.e. the length of the vector in Euclidean&#xA;space, not the dimensionality of the vector &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt;.  In other words, you&#xA;don't count the 0 bits, you add up only the 1 bits and take the square root.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry I don't have a real answer as to when you should use which metric,&#xA;but I can't let the incorrect answer go unchallenged.&lt;/p&gt;&#xA;" OwnerUserId="18596" LastEditorUserId="29575" LastEditDate="2020-02-08T16:00:29.027" LastActivityDate="2020-02-08T16:00:29.027" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11688" PostTypeId="1" CreationDate="2016-05-11T15:06:55.627" Score="2" ViewCount="202" Body="&lt;p&gt;I have a bunch of test measurements data and a semi-empirical model that has 18 parameters which I have to find so that the model fits my data well. So far I've managed to find and optimise the parameters using Optimisation and Global Optimisation algorithms in MATLAB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to explore different approaches for the parameter estimation. I have read some papers where the approach with NNs is described. I am new to NNs and have no idea if this is even possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would create a two layer network with 18 input and output neurons. I am not sure what kind of transfer function would be appropriate for the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula I have to fit and find the parameters look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ y = D \sin(C \arctan(Bx - E(Bx - \arctan(Bx))))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $B, C, D, E$ are macro-coefficients and the micro-coefficients are used to express the variation of each of the primary coefficients with respect to some other data. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/8C5b8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8C5b8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how my data looks like. &#xA;How would you create network in MATLAB for this problem? Can you give me some hints and a push in the right direction to tackle this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="18609" LastActivityDate="2016-05-11T16:14:49.943" Title="Artificial Neural Networks and Efficient Parameter Optimization" Tags="|neural-network|supervised-learning|parameter-estimation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11689" PostTypeId="2" ParentId="11688" CreationDate="2016-05-11T16:09:49.440" Score="1" Body="&lt;p&gt;If you were fitting a large number of different models, and you had sequences of training data for each different model (with the params already known in those cases), then you might be able to use a recursive neural network (RNN) to provide param estimates for new data sequences. However, that does not seem to be your situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand your question, you have a mathematical model with some free parameters, and &lt;em&gt;one&lt;/em&gt; set of data that you would like to use to estimate those parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A neural network is not really of any use to you here. That is because the neural network would &lt;em&gt;replace&lt;/em&gt; your semi-empirical model with the NN, and the free parameters become the weights of the NN. The trained NN would not match your desired model, but would generically fit to the data. It could be used to predict more function outputs given the inputs - the NN might even do better than your preferred model, which would be an interesting result implying that your model is incomplete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often we don't know (or perhaps even care) about an underlying parsimonious model. In this case, ML techniques that fit params of a generic model to data using some objective function like squared error can be very useful tools. However, when we do have a good idea about a simple or explanatory  model, those same techniques are less useful, and instead we can use optimisers direct on the model - in some cases these could be the same optimisers used to train neural networks, provided the model is differentiable.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-11T16:14:49.943" LastActivityDate="2016-05-11T16:14:49.943" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11691" PostTypeId="1" AcceptedAnswerId="11722" CreationDate="2016-05-11T16:53:49.887" Score="1" ViewCount="778" Body="&lt;p&gt;I am thinking to install the quick start virtual box of &lt;a href=&quot;http://www.cloudera.com/&quot; rel=&quot;nofollow&quot;&gt;Cloudera&lt;/a&gt;, search on Cloudera &lt;a href=&quot;http://www.cloudera.com/downloads/quickstart_vms/5-7.html&quot; rel=&quot;nofollow&quot;&gt;download page&lt;/a&gt;, but unable to find the spark version in the latest Cloudera QuickStart virtual box, can any have knowledge of this, please let me know.  &lt;/p&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-05-13T02:05:23.950" Title="What version of spark in latest Cloudera QuickStart VirtualBox?" Tags="|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11692" PostTypeId="1" AcceptedAnswerId="11697" CreationDate="2016-05-11T17:51:46.200" Score="6" ViewCount="840" Body="&lt;p&gt;I'm using the scikit-learn gradient boosting classifier found &lt;a href=&quot;http://scikit-learn.org/dev/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I run the classifier on the same data without seeding the random number generator, I get different feature importances, and notably, different ones are 0 each time. If I just look at the features where the feature importance is &amp;lt; .005, most of the same ones appear, but there is a little variation there too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, does this have an impact other than computation time? If it were to hurt accuracy, I imagine that would be due to over-fitting. However, if the classifier already identifies the relatively unimportant features, then doesn't that take care of the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it does hurt accuracy, how should I select which features to get rid of?&lt;/p&gt;&#xA;" OwnerUserId="18618" LastActivityDate="2018-06-24T12:47:26.390" Title="Does using unimportant features hurt accuracy?" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11693" PostTypeId="1" AcceptedAnswerId="14225" CreationDate="2016-05-11T20:27:05.973" Score="8" ViewCount="1504" Body="&lt;p&gt;I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different variations (e.g &quot;Earth&quot; and &quot;earth&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only way I can think of is to take average of vectors for &quot;Earth&quot; and &quot;earth&quot; to create a single vector to represent the word. (Since the feature vector's dimensions are similar)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this an &quot;okay&quot; method? If it is not, what might be a good way to handle this issue? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Lowering all words in preprocessing is not an option for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; The info about whether or not feature dimensions are truly linear would also be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2:&lt;/strong&gt; Combining both answers from &lt;code&gt;patapouf_ai&lt;/code&gt; and &lt;code&gt;yazhi&lt;/code&gt; gave the best results. How are these combined? Weighted average improved the results but putting word frequencies through a scaled sigmoid function gave the best results, because using word frequencies in a linear manner gives them more importance than they bear.&lt;/p&gt;&#xA;" OwnerUserId="18619" LastEditorUserId="18619" LastEditDate="2018-11-08T08:35:24.543" LastActivityDate="2018-11-08T08:35:24.543" Title="Averaging two Word2vec vectors to obtain a unified representation for single word" Tags="|feature-extraction|word-embeddings|word2vec|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11695" PostTypeId="1" CreationDate="2016-05-11T22:26:32.840" Score="1" ViewCount="225" Body="&lt;p&gt;My problem has three categorical variables &lt;code&gt;C1&lt;/code&gt;,&lt;code&gt;C2&lt;/code&gt;, &lt;code&gt;C3&lt;/code&gt; and one continous variable &lt;code&gt;X&lt;/code&gt;, predicting a continuous outcome &lt;code&gt;Y&lt;/code&gt;. I can visualize the problem with the following reproducible code (apology for the badly written code):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(data.tree)&#xA;i = expand.grid(c(&quot;A&quot;,&quot;B&quot;),c(&quot;C&quot;,&quot;D&quot;),c(&quot;E&quot;,&quot;F&quot;))&#xA;i = i[order(i[,1],i[,2],i[,3]),]&#xA;i[,4] = c(1:8)&#xA;&#xA;t = expand.grid(c(&quot;A&quot;,&quot;B&quot;),c(&quot;C&quot;,&quot;D&quot;),c(&quot;E&quot;,&quot;F&quot;),seq(0,1,0.1))&#xA;t = t[order(t[,1],t[,2],t[,3]),]&#xA;t = join(t,i, by = c(&quot;Var1&quot;,&quot;Var2&quot;,&quot;Var3&quot;))&#xA;t$Var5 = runif(nrow(t))&#xA;t$pathString &amp;lt;- with(t, paste(&quot;Tree&quot;, Var1, Var2, Var3, V4, sep=&quot;/&quot;))&#xA;plot_tree &amp;lt;- as.Node(t)&#xA;&#xA;plot(plot_tree)&#xA;ggplot(data = t, aes(x=Var4, y=Var5)) + geom_line() + facet_grid(~V4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The tree categorical variables show 8 possible path combinations:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/udh16.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/udh16.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And within each path there is a distribution of &lt;code&gt;Y&lt;/code&gt; depending on a continuous variable &lt;code&gt;X&lt;/code&gt;:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dUozn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dUozn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I would like to bin both the continuous &lt;code&gt;Y&lt;/code&gt; and &lt;code&gt;X&lt;/code&gt; variables such that I am left with a more concise decision tree. There is method to this madness as in my actual problem, some categorical paths will become insignificant due to no movement in the predicted &lt;code&gt;Y&lt;/code&gt; beyond the established volatility threshold of 5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can manually obtain bins by brute force, but is there a binning algorithm that supports binning of both predicted and predictor variables? I have looked at &lt;code&gt;smbinning&lt;/code&gt; and my knowledge in this area of algorithms is limited. My actual problem has many categorical and continuous variables resulting in a more complex structure, but I would like to convert it to a decision tree result which will help me understand the significance and movement of variables better. &lt;/p&gt;&#xA;" OwnerUserId="18248" LastActivityDate="2016-05-11T22:26:32.840" Title="Binning of Continous Predictor and Predicted Variables" Tags="|r|data-mining|decision-trees|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11696" PostTypeId="2" ParentId="10932" CreationDate="2016-05-12T00:25:00.583" Score="14" Body="&lt;p&gt;In brief each net has a different purpose as you mentioned:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The value network was used at the leaf nodes to reduce the depth of the tree search.&lt;/li&gt;&#xA;&lt;li&gt;The policy network was used to reduce the breadth of the search from a node (guiding towards promising immediate actions).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, you can use value function methods to find an optimal policy or directly search in the policy space to optimize a parametrized policy function (of course there are pros and cons). You can use function approximators (e.g Deep Nets) in each case. I see that mainly you are confused about the policy net so I focus my answer into this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The policy net was first:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;trained to do the moves that most likely a human would do given a board state (so input is a board state and output is a histogram that shows the probability of each action given that state). The net can approximate the probability function underlying the mapping from states to actions. It is reasonable to think to start building your policy from available data after all. After supervised training using experts moves the policy net could play the game sufficient (although far from a Master's level). Simply, you attempted to capture the general pattern of action selection of professional players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it was trained in games with opponent itself, in order to optimize the previous-learned policy. This time its weights were updated using the REINFORCE algorithm. By doing this, you update the net parameters towards maximization of expected reward. Eventually you have a net that not only selects the actions like a professional player but also towards winning the game (However it cannot plan!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this step, they approximated the value function of a bit more noisy version of the learned policy, by regression (input is the state board and target the result of the game). You can use this network to affect the leaf node evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conceptually speaking, the policy net gives you a probability over actions, but this doesn't indicate that you will end up in a good, for winning the game, state. AlphaGo had some &quot;blind spots&quot; and during the tournament did some really bad moves but also one exceptional move that a human could never had thought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you can use your planning algorithm (MCTS) in combination with these nets. Why we took all these steps? Briefly, the simple MCTS without any &quot;intuition&quot; would have failed.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="29575" LastEditDate="2018-08-05T14:29:56.060" LastActivityDate="2018-08-05T14:29:56.060" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="11697" PostTypeId="2" ParentId="11692" CreationDate="2016-05-12T07:00:52.893" Score="11" Body="&lt;p&gt;I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YDDlt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YDDlt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From that image it looks like &quot;good-enough performance&quot; is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only when you zoom in a bit on the Y-axis you see this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A3raC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A3raC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question &quot;&lt;strong&gt;does this have an impact other than computation time?&lt;/strong&gt;&quot;: it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your second question &quot;&lt;strong&gt;If it does hurt accuracy, how should I select which features to get rid of?&lt;/strong&gt;&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Select the top n features&lt;/li&gt;&#xA;&lt;li&gt;Use a cross-validated grid search to train a GBC model on the top n features&lt;/li&gt;&#xA;&lt;li&gt;Repeat with the top n+1,... features&lt;/li&gt;&#xA;&lt;li&gt;Select the model / features with the best score&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How to select the top n features in step 1):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="676" LastEditDate="2016-05-12T07:19:51.670" LastActivityDate="2016-05-12T07:19:51.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11699" PostTypeId="1" AcceptedAnswerId="11703" CreationDate="2016-05-12T08:38:12.740" Score="111" ViewCount="83031" Body="&lt;p&gt;This is a small conceptual question that's been nagging me for a while: How can we back-propagate through a max-pooling layer in a neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across max-pooling layers while going through &lt;a href=&quot;https://github.com/soumith/cvpr2015/blob/master/Deep%20Learning%20with%20Torch.ipynb&quot; rel=&quot;noreferrer&quot;&gt;this tutorial&lt;/a&gt; for Torch 7's nn library. The library abstracts the gradient calculation and forward passes for each layer of a deep network. I don't understand how the gradient calculation is done for a max-pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that if you have an input ${z_i}^l$ going into neuron $i$ of layer $l$, then ${\delta_i}^l$ (defined as ${\delta_i}^l = \frac{\partial E}{\partial {z_i}^l}$) is given by:&#xA;$$&#xA;{\delta_i}^l = \theta^{'}({z_i}^l) \sum_{j} {\delta_j}^{l+1} w_{i,j}^{l,l+1}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a max-pooling layer would receive the ${\delta_j}^{l+1}$'s of the next layer as usual; but since the activation function for the max-pooling neurons takes in a vector of values (over which it maxes) as input, ${\delta_i}^{l}$ isn't a single number anymore, but a vector ($\theta^{'}({z_j}^l)$ would have to be replaced by $\nabla \theta(\left\{{z_j}^l\right\})$). Furthermore, $\theta$, being the max function, isn't differentiable with respect to it's inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So....how should it work out exactly?&lt;/p&gt;&#xA;" OwnerUserId="18632" LastEditorUserId="18632" LastEditDate="2016-05-13T05:09:57.890" LastActivityDate="2023-04-27T15:33:37.230" Title="Backprop Through Max-Pooling Layers?" Tags="|neural-network|backpropagation|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11700" PostTypeId="2" ParentId="11693" CreationDate="2016-05-12T08:45:58.727" Score="0" Body="&lt;p&gt;The words &quot;Earth&quot; and &quot;earth&quot; may have the same meaning, but according to word2vec algorithm, it derives the semantic information from the position of the words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus commonly, &quot;Earth&quot; will appear most often at the start of the sentence being a subject and &quot;earth&quot; will appear mostly in the object form at the end. So, the closest adjacent words may differ, but on overall both the sentences may contain the words such as &quot;pollution, climate, water, countries&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, I guess with a larger window size, it seems to preserve the same semantic information with a little changes where the &quot;Earth&quot; will have some subject information and &quot;earth&quot; will have object information. So, averaging wont affect much and seems to be a possible case. But with lower window size, there is a high probability that it could have different meanings. &lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-05-12T08:45:58.727" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11701" PostTypeId="1" AcceptedAnswerId="11707" CreationDate="2016-05-12T10:00:43.200" Score="2" ViewCount="1045" Body="&lt;p&gt;I am currently working on a prototype of an application that should be able to interact with user interfaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now every user interface has some common elements, like buttons, scrollbars, input fields etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use Machine Learning to &quot;interpret&quot; such user interfaces in a way, in which I later can input a user interface as an image for example, and let the prototype then &quot;try out&quot; the interface, meaning, clicking on buttons, using scrollbars inputting some text into input fields etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that this would have to be done using Image Recognition, since there are many different UIs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am specifically interested in Websites, Adobe Reader with an opened PDF (that in turn can be a form etc.), and Word with an opened Document (again this can contain forms etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my main question is if there is already some research going on in this field that I can use, or even an existing tool for parts of the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated :)&lt;/p&gt;&#xA;" OwnerUserId="18636" LastEditorUserId="68809" LastEditDate="2019-03-04T05:28:43.427" LastActivityDate="2019-03-04T05:28:43.427" Title="Interpret User Interfaces with Machine Learning" Tags="|tensorflow|image-recognition|learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11702" PostTypeId="2" ParentId="8841" CreationDate="2016-05-12T10:01:52.643" Score="1" Body="&lt;p&gt;SMOTE is not designed to work with severe data imbalance specially if you have wide variation within the minority class &#xA;Try borderline SMOTE&#xA;Or&#xA;SMoteBoosting &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-12T10:01:52.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11703" PostTypeId="2" ParentId="11699" CreationDate="2016-05-12T11:52:40.280" Score="111" Body="&lt;p&gt;There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that actually achieves the max. Thus, the gradient from the next layer is passed back to only that neuron which achieved the max. All other neurons get zero gradient.&lt;/p&gt;&#xA;&lt;p&gt;So in your example, &lt;span class=&quot;math-container&quot;&gt;$\delta_i^l$&lt;/span&gt; would be a vector of all zeros, except that the &lt;span class=&quot;math-container&quot;&gt;$i^{*^{th}}$&lt;/span&gt; location will get a values &lt;span class=&quot;math-container&quot;&gt;$\left\{\delta_j^{l+1}\right\}$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$i^* = argmax_{i} (z_i^l)$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="18637" LastEditorUserId="120060" LastEditDate="2021-07-09T11:36:59.007" LastActivityDate="2021-07-09T11:36:59.007" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="11704" PostTypeId="1" AcceptedAnswerId="11705" CreationDate="2016-05-12T13:41:11.543" Score="14" ViewCount="35120" Body="&lt;p&gt;I am a beginner to Keras and I have started with the MNIST example to understand how the library actually works. The code snippet of the MNIST problem in the Keras example folder is given as :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;np.random.seed(1337)  # for reproducibility&#xA;&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Activation, Flatten  &#xA;from keras.layers import Convolution2D, MaxPooling2D&#xA;from keras.utils import np_utils&#xA;&#xA;batch_size = 128&#xA;nb_classes = 10&#xA;nb_epoch = 12&#xA;&#xA;# input image dimensions&#xA;img_rows, img_cols = 28, 28&#xA;# number of convolutional filters to use&#xA;nb_filters = 32&#xA;# size of pooling area for max pooling&#xA;nb_pool = 2&#xA;# convolution kernel size&#xA;nb_conv = 3&#xA;&#xA;# the data, shuffled and split between train and test sets&#xA;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)&#xA;X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)&#xA;X_train = X_train.astype('float32')&#xA;X_test = X_test.astype('float32')&#xA;..........&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am unable to understand the reshape function here. What is it doing and why we have applied it?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="97640" LastEditDate="2021-07-14T04:34:30.250" LastActivityDate="2021-07-14T04:34:30.250" Title="Reshaping of data for deep learning using Keras" Tags="|python|neural-network|deep-learning|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11705" PostTypeId="2" ParentId="11704" CreationDate="2016-05-12T14:27:31.480" Score="11" Body="&lt;p&gt;&lt;code&gt;mnist.load_data()&lt;/code&gt; supplies the MNIST digits with structure &lt;code&gt;(nb_samples, 28, 28)&lt;/code&gt; i.e. with 2 dimensions per example representing a greyscale image 28x28.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Convolution2D layers in Keras however, are designed to work with 3 dimensions per example. They have 4-dimensional inputs and outputs. This covers colour images &lt;code&gt;(nb_samples, nb_channels, width, height)&lt;/code&gt;, but more importantly, it covers deeper layers of the network, where each example has become a set of feature maps i.e. &lt;code&gt;(nb_samples, nb_features, width, height)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The greyscale image for MNIST digits input would either need a different CNN layer design (or a param to the layer constructor to accept a different shape), or the design could simply use a standard CNN and you must explicitly express the examples as 1-channel images. The Keras team chose the latter approach, which needs the re-shape.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-12T14:32:47.230" LastActivityDate="2016-05-12T14:32:47.230" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11707" PostTypeId="2" ParentId="11701" CreationDate="2016-05-12T14:39:21.653" Score="1" Body="&lt;p&gt;I would try experimenting with recurrent neural networks: &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&lt;/a&gt;. Recurrent neural networks can output sequences of variable length given inputs of variable length. In your case a recurrent neural network might output a sequence like the following when given a user interface: click a button, select a field, type some text, hit enter. For another interface, the network might output only: click one button, click another button, and that's it. This would be useful for you because the sequence and length of actions from interface to interface might change a lot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also experiment with reinforcement learning and build an algorithm that has an objective (reach some final page in as few actions as possible). The algorithm would start by doing random things (like clicking the same button a bunch of times), and then gradually learn over time to take appropriate actions. If you go that route you could use deep learning and Monte Carlo Tree Search (MCTS) like what Alpha Go did. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In either case you're going to need a framework that can train an algorithm quickly because you're likely to have to go through a lot of iterations. TensorFlow (&lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow&quot;&gt;https://www.tensorflow.org/&lt;/a&gt;) is one option (I've started using it recently, and I like it a lot because of its easy of use). TensorFlow is capable of building both recurrent neural nets and deep neural nets. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-05-12T14:39:21.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11709" PostTypeId="2" ParentId="11118" CreationDate="2016-05-12T15:01:08.220" Score="9" Body="&lt;p&gt;The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, given the complexity of the game, it would be a difficult task to train an algorithm from scratch without prior knowledge. Thus, it is reasonable at the beginning to start building such a system by upgrading it to a Master level using knowledge acquired by humans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth noting that, although the human moves bias the action selection at the tree nodes (states), this prior has a decay factor. This means that increased visitations to a specific state, reduce the strength of the prior to encourage the algorithm to explore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current level of Mastery of AlphaGo is unknown how close or far it is to a human's way of playing (in the tournament it did one move that a human had almost zero probability to perform!- but equally did some really bad moves as well). Possibly it remains for all these questions to be answered by actually implementing the corresponding testing algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I owe to edit my answer as the &lt;a href=&quot;https://www.nature.com/nature/journal/v550/n7676/full/nature24270.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;recent paper of DeepMind&lt;/a&gt; answers your question. There were lots of advancements that came out from the whole previous experience with the first version of AlphaGo and it is really worth reading it.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="25180" LastEditDate="2019-03-16T02:18:05.983" LastActivityDate="2019-03-16T02:18:05.983" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11710" PostTypeId="2" ParentId="6216" CreationDate="2016-05-12T15:52:06.073" Score="2" Body="&lt;p&gt;If you still have the question I hope my answer helps you. From what I understand you want to fit a RL model to participant's data who did a decision making task. The whole process of fitting RL models to data is described in great detail in &lt;a href=&quot;http://www.princeton.edu/~ndaw/d10.pdf&quot; rel=&quot;nofollow&quot;&gt;Trial by Trial analysis using RL&lt;/a&gt; by Nathaniel Daw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly, what you observe (your data) is just actions and rewards at every time step. If the participant is using a RL model you should expect to update the values of his actions according to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_t(a_t)=Q_t(a_{t-1})+\alpha\cdot (r_t-Q_t(a_{t-1}))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and action selection with the Boltzmann function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(a_{t}|s_{t})=\frac{e^{\beta Q(s_{t},a_{t})}}{\sum_{a'}e^{\beta Q(s_{t},a')}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We prefer to use a stochastic policy as it simulates better human choices and it is straightforward to create a likelihood function. The likelihood function would be the product of all the probabilities $p(a_t|s_t)$. The model parameters are $[{\alpha,\beta}]$ and these can be inferred by standard Maximum Likelihood techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, our hypothesis (=model) can be summarized as: A subject generates a sequence of (observed) actions. To generate these actions we assumed that uses a RL model with some parameters. We want to fit the model to the data to infer these parameters and test the suitability of the model and the cognitive/psychological links of the values of the parameters. &lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2016-08-17T20:19:25.913" LastActivityDate="2016-08-17T20:19:25.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11711" PostTypeId="2" ParentId="11329" CreationDate="2016-05-12T16:42:47.513" Score="3" Body="&lt;p&gt;The concept of state-action values $Q$ is to denote how good is to be in a particular state and perform a particular action in terms of expected future reward. From what I understand from your question, you are interested in the problem of model uncertainty (uncertainty on the dynamics of the system). In other words, our artificial agent interacts within an unknown environment (transition dynamics $T(s,a,s')$ and reward dynamics $R(s,a)$ or $R(s,a,s')$ are unknown).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The framework you should take a look at is Bayesian Model-based RL. I outline an approach so you can have an idea:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Modelling Transitions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;First assume that we have uncertainty on the transitions of the environment $T(s,a,s')$.&#xA;To tackle this we will assume that our agent maintains a distribution over possible transitions. Without getting into the theoretical math, I will illustrate this by using a simple Dirchlet-Multinomial model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The states are sampled from a Multinomial likelihood $s'\sim Mult(p_{ss'}^{a})$ and we assume a prior over the transitions $p_{ss'}\sim Dir(\alpha)$, where $\alpha$ is set to $1/|\cal{S}|$, where is $\cal{S}$ is the state space.&#xA;The posterior over transitions will be also a Dirichlet because of the conjugacy of the likelihood and prior distributions. To update such a posterior you need to perform simple algebraic calculations and maintaining the counts of each transition.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The Algorithm&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The agent does &lt;strong&gt;two&lt;/strong&gt; processes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(Simulation or Planning phase) Samples an MDP from the posterior over transitions (in the first iteration you sample from the prior distribution). Having a 'fixed table of tranistions' solves the sampled MDP (e.g with Value iteration) and selects the optimum action.&lt;/li&gt;&#xA;&lt;li&gt;(Real-world interaction phase) In the real world, you observe the reward of your action and the new state of the environment. You update the posterior counts.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Eventually, and if you have chosen an appropriate distribution for your domain the agent will adapt to the unknown environment. Of course richer priors with non conjugacy will lead to MCMC sampling methods. I refer you to this paper to get an overview of the problem: &lt;a href=&quot;http://robotics.stanford.edu/~nir/Papers/DFA1.pdf&quot; rel=&quot;nofollow&quot;&gt;Model-based Bayesian Exploration&lt;/a&gt; and the quite advanced: &lt;a href=&quot;http://www.cs.mcgill.ca/~aguez/files/guez_jair2013.pdf&quot; rel=&quot;nofollow&quot;&gt;Bayes-Adaptive MDPs&lt;/a&gt; for further research and exploration.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2016-05-12T16:42:47.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11713" PostTypeId="1" CreationDate="2016-05-12T18:02:29.143" Score="1" ViewCount="303" Body="&lt;p&gt;In short: For naive Bayes and text classification, do you multiply a probability for each instance of a word in a document or once if the word occurs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more detail: The question is how to calculate naive Bayes for a document. We have a corpus of text from which we can calculate the frequency of documents with class $C$ having a word $w$. We then calculate the probability $P(w_1|C)...P(w_n|C)$ and choose the $C$ that maximizes this number. But I'm not sure whether we multiply a probability for each word whenever it occurs or just once if it occurs in the document? Like, if the text is &quot;$w_1 \ w_2 \ w_1$&quot;, is the estimated probability $P(w_1|C)^2 P(w_2|C)$ or is it $P(w_1|C)P(w_2|C)$? &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="12443" LastEditDate="2016-05-12T18:49:18.883" LastActivityDate="2016-05-17T10:30:33.680" Title="When classifying documents with naive Bayes, which probabilities do you multiply in case of repeated terms?" Tags="|nlp|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11716" PostTypeId="1" CreationDate="2016-05-12T19:21:37.730" Score="2" ViewCount="95" Body="&lt;p&gt;Suppose you have a classification task and accuracy is what you care about. Now an old system $s_1$ has an accuracy of $a(s_1) = 0.9$ and a new system $s_2$ has an accuracy of $a(s_2) = 0.99$. This is an absolute improvement of $a(s_2) - a(s_1) = 0.09$ percentage points and a relative improvement of $\frac{a(s_2)-a(s_1)}{a(s_1)} = \frac{0.09}{0.9} = 0.1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when you now try to get a system of $a(s_3) = 0.999$ this seems to be much more difficult, although it is only an absolute improvement of $0.009$ and a relative improvement of $0.00\overline{90}$. So neither the absolute nor the relative difference in accuracy seems to capture this well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a common other way to quantify how much better the system is?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-05-26T22:57:43.323" Title="What is a reasonable way to compare the improvement in accuracy?" Tags="|accuracy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11717" PostTypeId="1" AcceptedAnswerId="11725" CreationDate="2016-05-12T19:48:57.940" Score="1" ViewCount="61" Body="&lt;p&gt;I have a large dataset that contains product purchase history, like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;userID  productID   category        subcategory&#xA;123     ABC         Kitchen         Knives&#xA;123     BEA         Kitchen         Organization&#xA;233     ZZS         Electronics     Phones&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a first project, I'm looking to answer the question: &quot;What discrete groups of categories/subcategories do shoppers tend to shop in?&quot;. For example, we may find that shoppers who buy Monitors are highly likely to buy Keyboards and Mice as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any direction on getting started on a problem like this is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="18650" LastActivityDate="2016-05-13T05:05:02.543" Title="Determining correlated product categories using store purchase history" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11718" PostTypeId="2" ParentId="10634" CreationDate="2016-05-12T21:15:17.737" Score="1" Body="&lt;p&gt;Some reasons the network may not be getting better: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Learning rate is too high (jumps back and forth forever)&lt;/li&gt;&#xA;&lt;li&gt;Learning rate is too low (when divided by training set size, goes to zero)&lt;/li&gt;&#xA;&lt;li&gt;You're replacing the error signal with the weight gradient signal.  The reuse of error (based on the indentation) seems to be right on the same line as the inner loop.  I'm not sure if that's correct.  First, backprop your errors, THEN calculate the gradient based on the activation of the hidden neurons and delta sigmoid(error).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16135" LastActivityDate="2016-05-12T21:15:17.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11719" PostTypeId="2" ParentId="11591" CreationDate="2016-05-12T21:26:04.493" Score="6" Body="&lt;p&gt;You're mostly correct!  ReLU does have a problem with the gradient vanishing, but only on one side, so we call it something else: the 'dying ReLU problem'.  See this stack overflow response for more information: &lt;a href=&quot;https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks&quot;&gt;What is the &amp;quot;dying ReLU&amp;quot; problem in neural networks?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a small semantic difference.  Lots of functions (tanh and logistic/sigmoid) have derivatives very close to zero when you're outside the standard operating range.  This is the 'vanishing gradient' issue.  The worse you get, the harder it is to get back into the good zone.  ReLU &lt;strong&gt;doesn't&lt;/strong&gt; get worse the farther you are in the positive direction, so no vanishing gradient problem (on that side).  This asymmetry might be enough to justify calling it something different, but the ideas are quite similar.&lt;/p&gt;&#xA;" OwnerUserId="16135" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-05-12T21:26:04.493" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11720" PostTypeId="1" AcceptedAnswerId="11833" CreationDate="2016-05-12T21:30:33.763" Score="1" ViewCount="7071" Body="&lt;p&gt;So I recently came along kNN k nearest neighbour.&#xA;When looking at its disadvantages, most of the literature mentions it is costly, lazy, requires full training data plus depends on the value of k and has the issue of dimensionality because of the distance.&#xA;Other than that I have following hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- It ignores the fact that dimensions can be inter related and instead assumes they are independent (as we are just calculating distance)&#xA;2- Has the issue of normalization of data... if the data is not normalized distance can be biased towards a specific dimension&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will like to have a comprehensive analysis on the disadvantages of kNN apart from those mentioned above and if they are wrong then why.&lt;/p&gt;&#xA;" OwnerUserId="18653" LastActivityDate="2016-05-19T00:00:43.277" Title="kNN(classifier) - Disadvantages" Tags="|classification|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11721" PostTypeId="1" CreationDate="2016-05-12T22:57:31.797" Score="1" ViewCount="1090" Body="&lt;p&gt;I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (Married, Having young kids, etc.), financial status etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest some data sources having such information to train the model. &lt;/p&gt;&#xA;" OwnerUserId="10049" LastActivityDate="2017-01-09T10:15:16.607" Title="Where can I get a comprehensive criminal dataset?" Tags="|machine-learning|data-mining|predictive-modeling|dataset|data|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11722" PostTypeId="2" ParentId="11691" CreationDate="2016-05-13T02:05:23.950" Score="2" Body="&lt;p&gt;QuickStart VM latest 5.7 has spark version 1.6.0 by default. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/po32P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/po32P.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2016-05-13T02:05:23.950" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11723" PostTypeId="1" AcceptedAnswerId="11772" CreationDate="2016-05-13T03:38:51.757" Score="3" ViewCount="2433" Body="&lt;p&gt;Does anyone know of an algorithm that could be used to determine the next action to take to reach a desired state when trained on time-series data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a robot starts at a certain state, then takes an action to get to another state. This occurs continuously for many iterations (imagine the robot is randomly exploring a room). If the robot is at a specific starting state, and I desire the robot to end up in a different state, is there an algorithm that could recommend the best next action (or set of next actions) to take to reach that final desired state?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach I've tried is to use a neural network with the current state and the next state being the input and the action to get from the current state to the next state being the output. The network would know for a single state how to get to a next desired state that is one action away. The issue is, what if the desired state is many actions away?&lt;/p&gt;&#xA;" OwnerUserId="18660" LastActivityDate="2016-05-14T19:52:26.807" Title="Predicting next action to take to reach a final state" Tags="|machine-learning|neural-network|predictive-modeling|time-series|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11724" PostTypeId="2" ParentId="11717" CreationDate="2016-05-13T04:11:03.280" Score="0" Body="&lt;p&gt;This problem is similar to the twitter tag clustering problem. You can read literature around that for many ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My personal starting approach would be to construct a network of &lt;code&gt;category&lt;/code&gt; such that each unique category is a node and the strength between a pair of categories is determined by counting the number of customers who buy both products. Then you can look at some community detection algorithms in graph theory to form clusters of categories based on strength of pairs. Similar analysis can you repeated with &lt;code&gt;subcategory&lt;/code&gt; and creating a combined variable of &lt;code&gt;category-subcategory&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18248" LastActivityDate="2016-05-13T04:11:03.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11725" PostTypeId="2" ParentId="11717" CreationDate="2016-05-13T05:05:02.543" Score="1" Body="&lt;p&gt;This is classic &lt;strong&gt;market basket analysis&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is the weong tool, you want frequent item set mining and association rules instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Association_rule_learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-05-13T05:05:02.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11726" PostTypeId="1" CreationDate="2016-05-13T08:48:54.160" Score="1" ViewCount="2530" Body="&lt;p&gt;I am really worried How does RandomForest Algorithm Works internally,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Out of bag error&lt;/li&gt;&#xA;&lt;li&gt;Variable Importance&lt;/li&gt;&#xA;&lt;li&gt;Z Scores&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let me know all these three ? How to calculate it? What it controls?&lt;/p&gt;&#xA;" OwnerUserId="18139" LastActivityDate="2016-05-13T09:04:37.877" Title="Random Forest variable Importance Z Score" Tags="|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11728" PostTypeId="2" ParentId="11726" CreationDate="2016-05-13T09:04:37.877" Score="4" Body="&lt;p&gt;Please allow me to cite the &lt;a href=&quot;https://www.stat.berkeley.edu/%7Ebreiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;site&lt;/a&gt; of creators of the &lt;code&gt;RandomForest&lt;/code&gt; algorithm&lt;/p&gt;&#xA;&lt;p&gt;Out of bag error&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In random forests, there is no need for cross-validation or a separate test set to get an unbiased estimate of the test set error. It is estimated internally, during the run, as follows:&lt;/p&gt;&#xA;&lt;p&gt;Each tree is constructed using a different bootstrap sample from the original data. About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree.&lt;/p&gt;&#xA;&lt;p&gt;Put each case left out in the construction of the kth tree down the kth tree to get a classification. In this way, a test set classification is obtained for each case in about one-third of the trees. At the end of the run, take j to be the class that got most of the votes every time case n was oob. The proportion of times that j is not equal to the true class of n averaged over all cases is the &lt;strong&gt;oob error estimate&lt;/strong&gt;. This has proven to be unbiased in many tests.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Variable importance and Z-score&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In every tree grown in the forest, put down the oob cases and count the number of votes cast for the correct class. Now randomly permute the values of variable m in the oob cases and put these cases down the tree. Subtract the number of votes for the correct class in the variable-m-permuted oob data from the number of votes for the correct class in the untouched oob data. The average of this number over all trees in the forest is the &lt;strong&gt;raw importance score&lt;/strong&gt; for variable m.&lt;/p&gt;&#xA;&lt;p&gt;If the values of this score from tree to tree are independent, then the standard error can be computed by a standard computation. The correlations of these scores between trees have been computed for a number of data sets and proved to be quite low, therefore we compute standard errors in the classical way, divide the raw score by its standard error to get a z-score, ands assign a significance level to the &lt;strong&gt;z-score&lt;/strong&gt; assuming normality.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="17646" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-05-13T09:04:37.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11729" PostTypeId="1" AcceptedAnswerId="16198" CreationDate="2016-05-13T09:33:58.113" Score="0" ViewCount="1126" Body="&lt;p&gt;I am trying to recognize and classify the entity types based on the IOB/Sequence labeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to use nltk.ne_chunk() which is already trained to recognize named entities using their train set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is any way to train the algorithm using my on training set and labels using training data like the following,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(&#xA;send        O&#xA;sms         B-TASK &#xA;to          &#xA;8714349616  B-MOB&#xA;how         B-MSG&#xA;are         I-MSG&#xA;you         I-MSG&#xA;)&#xA;&#xA;(&#xA;sms        B-TASK        &#xA;how        B-MSG&#xA;are        I-MSG&#xA;you        I-MSG        &#xA;to         O&#xA;8714349616 B-MOB&#xA;)&#xA;etc..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is it possible using NLTK ? Any examples or Tutorials ?&lt;/p&gt;&#xA;" OwnerUserId="18665" LastActivityDate="2017-01-09T07:54:05.113" Title="How to train NLTK Sequence labeling algorithm for using custom labels/Train set?" Tags="|python|nlp|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11730" PostTypeId="1" CreationDate="2016-05-13T09:43:55.280" Score="8" ViewCount="31982" Body="&lt;p&gt;How do I scrape a website that basically looks like google with just a giant searchbar in the middle of the screen. From it you can search after various companies and their stats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a list of 1000 companies I want to get information about. I want some bot to search each company from my list in the search bar, open the specific company's info window and extract a certain company code that exist on each page for each company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any easy and (of course) legal way to do it?&lt;/p&gt;&#xA;" OwnerUserId="17359" LastEditorUserId="16284" LastEditDate="2016-05-13T12:40:00.507" LastActivityDate="2021-03-15T11:33:29.590" Title="How to scrape a website with a searchbar" Tags="|data-mining|scraping|crawling|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11731" PostTypeId="1" CreationDate="2016-05-13T09:59:21.247" Score="0" ViewCount="5479" Body="&lt;p&gt;I am running the following code with data traing with targetBuy as response &#xA;variable&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# load the library&#xA;library(caret)&#xA;library(klaR)&#xA;&#xA;# define training control&#xA;train_control &amp;lt;- trainControl(method = &quot;repeatedcv&quot;, number = 10, repeats = 3)&#xA;model &amp;lt;- train(TargetBuy ~ ., data = traing, trControl = train_control, method = &quot;nb&quot;)&#xA;print(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Getting the following error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in train(TargetBuy ~ ., data = traing, trControl =&#xA;  train_control,  :     unused arguments (data = traing, trControl =&#xA;  train_control, method = &quot;nb&quot;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18139" LastEditorUserId="18139" LastEditDate="2016-05-13T10:30:24.653" LastActivityDate="2016-05-13T11:32:31.037" Title="Error in Model with caret Package" Tags="|r|predictive-modeling|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11732" PostTypeId="2" ParentId="11731" CreationDate="2016-05-13T11:05:35.183" Score="1" Body="&lt;p&gt;Please read carefully &lt;code&gt;caret::train&lt;/code&gt; since S3 method for class &lt;code&gt;formula&lt;/code&gt; is slightly different than default S3 method for &lt;code&gt;caret::train&lt;/code&gt; (and takes different arguments)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should do the work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- caret::train(y = traing$TargetBuy, x = traing[, colnames(traing) != 'TargetBuy'], trControl = train_control, method = &quot;nb&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also use &lt;code&gt;model.matrix&lt;/code&gt;, but then you need to remember that it removes missing values. See &lt;a href=&quot;https://stackoverflow.com/questions/5616210/model-matrix-with-na-action-null&quot;&gt;this post for solution&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- caret::train(y = factor(traing$TargetBuy), x = model.matrix(TargetBuy ~ ., data = traing), trControl = train_control, method = &quot;nb&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17646" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-05-13T11:32:31.037" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11733" PostTypeId="2" ParentId="11730" CreationDate="2016-05-13T11:41:11.113" Score="6" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I would suggest reading about http query methods, specifically about&#xA;GET and POST. You can pass parameters with query and open directly&#xA;company page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://google.com/search?q=GET%20and%20POST&quot; rel=&quot;noreferrer&quot;&gt;http://google.com/search?q=GET+and+POST&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where (q=GET+and+POST) is a parameter.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once you have page you can parse it with your favorite library.&#xA;(for example &lt;a href=&quot;http://www.pythonforbeginners.com/beautifulsoup/beautifulsoup-4-python&quot; rel=&quot;noreferrer&quot;&gt;beautifulsoup&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;EXAMPLE:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Getting number of results from couple of google queries with python 3 and beautifulsoup:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from bs4 import BeautifulSoup&#xA;import urllib.request&#xA;&#xA;# List with google queries I want to make&#xA;desired_google_queries = ['Word' , 'lifdsst', 'yvou', 'should', 'load', 'from']&#xA;&#xA;for query in desired_google_queries:&#xA;    # Constracting http query&#xA;    url = 'http://google.com/search?q=' + query&#xA;    # For avoid 403-error using User-Agent&#xA;    req = urllib.request.Request(url, headers={'User-Agent' : &quot;Magic Browser&quot;})&#xA;    response = urllib.request.urlopen( req )&#xA;    html = response.read()&#xA;    # Parsing response&#xA;    soup = BeautifulSoup(html, 'html.parser')&#xA;    # Extracting number of results&#xA;    resultStats = soup.find(id=&quot;resultStats&quot;).string&#xA;    print(resultStats)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18676" LastEditorUserId="18676" LastEditDate="2016-05-13T13:20:07.487" LastActivityDate="2016-05-13T13:20:07.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11734" PostTypeId="2" ParentId="11730" CreationDate="2016-05-13T12:05:22.800" Score="2" Body="&lt;p&gt;I would suggest using a combination of rvest and rselenium, depending on the way the web page is set up.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/RSelenium/vignettes/basics.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rselenium&lt;/a&gt; to navigate the page (if needed)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/rvest/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rvest&lt;/a&gt; to scrape the data from the page&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16284" LastEditorUserId="31592" LastEditDate="2020-08-16T21:51:44.693" LastActivityDate="2020-08-16T21:51:44.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11735" PostTypeId="1" AcceptedAnswerId="11736" CreationDate="2016-05-13T12:43:16.063" Score="2" ViewCount="1942" Body="&lt;p&gt;Let's say you train a neural network on Input = 1/Output = 2; Input = 4/Output = 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you train a machine to recognize that it needs to multiply the input by 2, from scratch?&lt;/p&gt;&#xA;" OwnerUserId="18679" LastActivityDate="2016-05-13T12:57:47.663" Title="How would you teach multiplications to a neural network?" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11736" PostTypeId="2" ParentId="11735" CreationDate="2016-05-13T12:51:56.590" Score="1" Body="&lt;p&gt;You need to limit the network model to one which naturally would generalise your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a simple multiplication, this would be a single linear neuron. You can also train bias so that the network represents &lt;code&gt;y = Ax + B&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you take that very simple network give it just 2 or 3 examples to learn from, and train it using least squares error, it will quickly converge to the function you want. A single layer linear network like this can generalise to &lt;code&gt;y = Ax + B&lt;/code&gt; where x, y and B are vectors, and A is a matrix - i.e. it can solve simultaneous linear equations provided you supply enough examples to do so (usually number of examples equals number of dimensions of the vectors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is actually really trivial, and if you train such a simple network using any modern NN library, it will be very fast to learn your linear function. However, in the general case of having a specific non-linear function and wanting the network to learn it in a general sense, it is not possible. The network can be made to learn to generalise reasonably well near where you have provided example inputs and outputs, but will output incorrect results when you give it an &lt;code&gt;x&lt;/code&gt; value far away from the training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to note that the network does not learn the math formula for your function. It finds the best approximation given its own internal model. But by picking a network where the internal model is a good match to the function you want to learn, it is going to generalise well to it, and may be able to get it exactly if there is no noise in the examples and enough of them.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-13T12:57:47.663" LastActivityDate="2016-05-13T12:57:47.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11737" PostTypeId="2" ParentId="11723" CreationDate="2016-05-13T12:56:27.127" Score="0" Body="&lt;p&gt;You can't know the next best step unless you know the whole best path.&#xA;Your task is something similar to &lt;em&gt;TSP&lt;/em&gt; (traveling salesman problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how to find the best path?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably you can add penalties (not known) to each state. Then you can define your goal as path with minimal total penalty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For robot from your example floor will have 0 penalty and some obstacles &gt;0 penalty).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I know you can't avoid brute forcing through all possible paths unless your task have additional restrictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to approximate penalties with data you already have. (So, penalties is parameters in your machine learning task)&#xA;When you estimated penalties you can compute total penalty for each path, so you can find all paths by brute force and find &quot;best&quot; one.(with smallest penalty)&lt;/p&gt;&#xA;" OwnerUserId="18676" LastActivityDate="2016-05-13T12:56:27.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11738" PostTypeId="1" AcceptedAnswerId="11746" CreationDate="2016-05-13T13:02:38.443" Score="0" ViewCount="314" Body="&lt;p&gt;I'm working with the Convex Nonnegative Matrix Factorization Algorithm described in &lt;a href=&quot;http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4685898&amp;amp;tag=1&quot; rel=&quot;nofollow&quot;&gt;Ding, Li, Jordan 2008 (&quot;Convex and Semi-Nonnegative Matrix&#xA;Factorizations&quot;)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good initialization strategies make all the difference and using the described k-means clustering to get started works very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is a paper describing &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0031320307004359&quot; rel=&quot;nofollow&quot;&gt;NNDSVD (Boutsidis, Gallopoulos, 2007)&lt;/a&gt; to initialize &quot;traditional&quot; NMF Algos. I wanted to test this, to see if it improves my results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nonnegativity constraints for Convex-NMF are relaxed. X can have mixed sign data, where X ~ XWG', with factors W and G having only positive data.&#xA;I've implemented NNDSVD just like in the paper (in C++ w/ OpenCV), but since X has mixed sign data, the resulting W contains negative values as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone tried to adapt this initialization strategy? Are there any other recommended initialization strategies for Convex-NMF, aside from the ones mentioned in the Ding et al Paper (and random init)?&lt;/p&gt;&#xA;" OwnerUserId="18680" LastEditorUserId="18680" LastEditDate="2016-05-13T13:30:11.333" LastActivityDate="2016-05-13T14:25:47.430" Title="NNDSVD to initialize Convex-NMF" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11740" PostTypeId="1" AcceptedAnswerId="11781" CreationDate="2016-05-13T13:24:11.013" Score="5" ViewCount="10099" Body="&lt;p&gt;The mean of a population of binary values can be sampled with about 1000 samples at 95% confidence, and 3000 samples at 99% confidence.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming a binary classification problem, why is the 80/20% rule always used, and not the fact that with a few thousand samples the mean accuracy can be estimated with &gt; 95% confidence?&lt;/p&gt;&#xA;" OwnerUserId="18682" LastActivityDate="2021-12-19T16:20:17.860" Title="What is the minimum size of the test set?" Tags="|statistics|cross-validation|ab-test|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11742" PostTypeId="2" ParentId="11740" CreationDate="2016-05-13T13:54:57.307" Score="0" Body="&lt;p&gt;It is just a rule of thumb, the bigger your test set the more accurate your performance measure. In reality most people use k-fold cross validation to get a better performance estimate than a 80/20 split. The higher k, the lower variance your estimator will be. Even better would be take-1-out cross validation however this gets computationally very expensive. If there is stochastic behavior in your model, then an even better approach would be to repeat take-1-out cross validation multiple times due to training on the same set will yield a different model. Making a choice in this regard is simply a trade-off between the variance in your estimate and the computational costs of training your model.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-13T13:54:57.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11743" PostTypeId="1" CreationDate="2016-05-13T14:03:32.570" Score="1" ViewCount="259" Body="&lt;p&gt;I am trying to see how the fundamental formulas of backpropagation from this &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;online book&lt;/a&gt; work on a small example.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author lists these 4 fundamental equations (where I believe from the reading that the cost (or loss) function C as well as the derivative of the activation function $\sigma$ are general and work with any cost or activation function):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nHBMo.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nHBMo.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I take a small example network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wwQbw.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wwQbw.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and assume....&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;C = $\frac{1}{2}(a^3-y)^2$  (squared error)&lt;/li&gt;&#xA;&lt;li&gt;The activation (a) for each node is just the logistic function sigmoid&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From here we have the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$z^2_1=w^1_{11}x_1+w^1_{12}x_2+b_1$&lt;/li&gt;&#xA;&lt;li&gt;$z^2_2=w^1_{21}x_1+w^1_{22}x_2+b_2$&lt;/li&gt;&#xA;&lt;li&gt;$z^2_3=w^1_{31}x_1+w^1_{32}x_2+b_3$&lt;/li&gt;&#xA;&lt;li&gt;$a^2_1 =\sigma(z^2_1)$ &lt;/li&gt;&#xA;&lt;li&gt;$a^2_2 =\sigma(z^2_2)$  &lt;/li&gt;&#xA;&lt;li&gt;$a^2_3 =\sigma(z^2_3)$&lt;/li&gt;&#xA;&lt;li&gt;$z^3_1=w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3$&lt;/li&gt;&#xA;&lt;li&gt;$a^3=\sigma(z^3_1)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I want to calculate $\frac{\partial C}{\partial w^2_{11}}$ it seems from the book that I should calculate $a^1_1\delta^2_1$ where $\delta^2 = ((w^{2+1})^T\delta^{2+1})\odot\sigma'(z^2) $. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $W^3$ does not exist I assume it is ignored and we can calculate $\delta^2_1= \delta^{2+1}_1\odot\sigma'(z^2_1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\delta^3_1 = \frac{\partial C}{\partial z^3_1} = (a^3-y)\sigma'(z^3_1)$ &#xA;so $\frac{\partial C}{\partial w^2_{11}}= a^1_1*((a^3-y)\sigma'(z^3_1))*\sigma'(z^2_1) $. $a^1_1$ doesnt exist except if this would mean $x_1?$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if I just calculate $\frac{\partial C}{\partial w^2_{11}}$ I get $\frac{\partial }{\partial w^2_{11}}\frac{1}{2}([\sigma(w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3)]-y)^2=a^2_1(a^3_1-y)\sigma'(z^3_1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where am I going wrong?&lt;/p&gt;&#xA;" OwnerUserId="1138" LastEditorUserId="1138" LastEditDate="2016-05-13T14:47:50.007" LastActivityDate="2016-12-10T00:19:05.540" Title="Help Using Backprop Formulas from Deep Learning Book" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11744" PostTypeId="2" ParentId="11723" CreationDate="2016-05-13T14:04:34.463" Score="1" Body="&lt;p&gt;I think you need to use Reinforcement learning &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow&quot;&gt;Reinforcement learning&lt;/a&gt;. Pybrain library for python has an example similar with your task &lt;a href=&quot;http://pybrain.org/docs/tutorial/reinforcement-learning.html&quot; rel=&quot;nofollow&quot;&gt;robot explore maze&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16541" LastActivityDate="2016-05-13T14:04:34.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11745" PostTypeId="1" CreationDate="2016-05-13T14:22:56.333" Score="7" ViewCount="414" Body="&lt;p&gt;I am processing some data using a feedforward neural network in Keras. I have noticed that if I log transform the data, the model trains better, however the error metric on the transformed data doesn't necessarily reflect very well what's going on with the original dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment I am using Mean Average Percentage Error metric (mape), and a CV error of 3.8% on log transformed data set corresponds to about 25% when I reverse-transform the predictions and check against the original outputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better choice of cost/objective function for situations like this? &lt;/p&gt;&#xA;" OwnerUserId="18686" LastEditorUserId="8501" LastEditDate="2017-03-08T17:01:36.657" LastActivityDate="2020-05-29T03:16:51.610" Title="Choice of objective function for log transformed data for feedforward NN" Tags="|machine-learning|neural-network|keras|loss-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11746" PostTypeId="2" ParentId="11738" CreationDate="2016-05-13T14:25:47.430" Score="0" Body="&lt;p&gt;I've played before with a library for (&quot;classic&quot;) NMF from University of Vienna: &lt;a href=&quot;http://www.univie.ac.at/rlcta/software/&quot; rel=&quot;nofollow&quot;&gt;libNMF&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it wasn't useful for me at first, since they don't implement Convex-NMF, I had a look at their NNDSVD implementation.&#xA;They simply &lt;strong&gt;replace all negative values&lt;/strong&gt; and values smaller than the machine epsilon &lt;strong&gt;with zero&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried that, but it fails, since the update rules for Convex-NMF are such that I get a division by zero. Since they &lt;strong&gt;add 0.2 to all values&lt;/strong&gt; in their proposed initialization scheme (for &quot;smoothing&quot;), I just did the same here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works, the precision is worse than expected (a bit worse than k-means, which is my best approach for now), but I'll tweak it a bit more.&lt;/p&gt;&#xA;" OwnerUserId="18680" LastActivityDate="2016-05-13T14:25:47.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11747" PostTypeId="1" AcceptedAnswerId="16061" CreationDate="2016-05-13T14:39:51.430" Score="10" ViewCount="45550" Body="&lt;p&gt;Suppose I would like to train and test the MNIST dataset in Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The required data can be loaded as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.datasets import mnist&#xA;&#xA;digits_data = mnist.load_data()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any way in keras to split this data into three sets namely: &lt;code&gt;training_data&lt;/code&gt;, &lt;code&gt;test_data&lt;/code&gt;, and &lt;code&gt;cross_validation_data&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-28T13:57:49.397" LastActivityDate="2020-12-29T19:00:35.780" Title="Cross Validation in Keras" Tags="|machine-learning|python|deep-learning|keras|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11749" PostTypeId="2" ParentId="11740" CreationDate="2016-05-13T15:19:30.253" Score="2" Body="&lt;p&gt;Maybe some of your best predictors doesn't occur very often (sparse) and if you have 10 million rows, and only take 3000 samples you don't get any of those samples with that really important sparse outcomes. You could have zip code as a predictor, and with over 43,000 unique zip codes in the US associated with the 10 million samples, and your sample of 3,000 could get about 7% of the unique zip codes at best.  That would be tough to get an accurate measure of error with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for using a test set whose size is relative to the data (be it 20% or 30% holdout, or 10-fold cross validation) is to have a standard and more robust measure of error than just a fixed number of samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only reason why I think you would be tempted to use a few thousand samples as a test set, is to use more data in your model building procedure.  If that is the case, cross validation like 10-fold would be more tempting since it uses all of your data, and gives you an idea of the variability of your model performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd play around with some open binary data set that has 100,000 or so samples and try out your proposal yourself.  Run two simulations many times: One where you train 80% and test on 20% and record the miss-classification error each time, and another where you train on (n-5000) and test on 5000 and record the miss-classification error each time.  Take a look at the stored errors and compare their means and standard deviations.  my bet is the 20% will be much more stable.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-05-13T15:19:30.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11750" PostTypeId="1" CreationDate="2016-05-13T16:35:24.443" Score="1" ViewCount="3342" Body="&lt;p&gt;I am Running the following models&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Logistic regression&lt;/li&gt;&#xA;&lt;li&gt;Decision Trees&lt;/li&gt;&#xA;&lt;li&gt;SVM&lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes&lt;/li&gt;&#xA;&lt;li&gt;Random Forest&#xA;On the same data set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am using Caret package in r.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its My dream to plot Training error and Testing error curves, I know those plots&lt;/p&gt;&#xA;&#xA;&lt;p&gt;are pretty beautiful as well as other performance measure curves using caret &lt;/p&gt;&#xA;&#xA;&lt;p&gt;package, any help would be really appreciated.&lt;/p&gt;&#xA;" OwnerUserId="18139" LastActivityDate="2017-02-08T04:58:24.457" Title="Training And Testing Error Curves caret package in r" Tags="|machine-learning|r|predictive-modeling|performance|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11751" PostTypeId="1" AcceptedAnswerId="13590" CreationDate="2016-05-13T16:57:00.567" Score="4" ViewCount="9189" Body="&lt;p&gt;I am trying to find a package to construct a dashboard with interactive graphs (including widgets such as sliders) in python (mainly IPython notebook). I know there is plotly but I would like a fully open-source solution without constraints (i.e. having a public repository as with plotly without the subscription fee).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at &lt;a href=&quot;https://pypi.python.org/pypi/IPython-Dashboard&quot; rel=&quot;nofollow&quot; title=&quot;IPython Dashboard&quot;&gt;IPython Dashboard&lt;/a&gt; package but it is not compatible with python 3 (because of MySQL-python). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone had any luck with any other package?&lt;/p&gt;&#xA;" OwnerUserId="14487" LastEditorUserId="8878" LastEditDate="2017-05-24T07:52:23.933" LastActivityDate="2021-02-24T21:01:27.220" Title="Open-source interactive dashboard in Python" Tags="|python|visualization|jupyter|ipython|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11752" PostTypeId="1" AcceptedAnswerId="11762" CreationDate="2016-05-13T17:14:26.627" Score="0" ViewCount="107" Body="&lt;p&gt;I'm doing some ADA boosting with Decision stumps and in inducing a binary classifying decision stump, i'm finding both leaf nodes to have a positive value. Can this be the case? Is this possible?&lt;/p&gt;&#xA;" OwnerUserId="18691" LastActivityDate="2016-05-13T23:19:08.237" Title="Decision Stumps with same value leaf nodes" Tags="|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11753" PostTypeId="1" AcceptedAnswerId="11835" CreationDate="2016-05-13T18:29:47.550" Score="1" ViewCount="179" Body="&lt;p&gt;I have cloudera CDH5 running inside a virtual box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when I try to run :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mahout spark-itemsimilarity ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Unknown program 'spark-itemsimilarity' chosen.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do i have to install any additional package to run the spark-similarity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated !&lt;/p&gt;&#xA;" OwnerUserId="18588" LastActivityDate="2016-05-19T00:22:39.260" Title="Unknown program 'spark-itemsimilarity' chosen" Tags="|apache-spark|apache-mahout|recommender-system|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11754" PostTypeId="1" CreationDate="2016-05-13T18:58:48.893" Score="2" ViewCount="153" Body="&lt;p&gt;I have followed the tutorial &lt;a href=&quot;https://www.tensorflow.org/versions/r0.8/tutorials/recurrent/index.html#language_modeling&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/ptb_word_lm.py&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I failed to understnad the exact specific input and the output of the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how many words do you feed to the model and how many words do you expect at the end? How the data processing happens?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: if I have the sentence &quot;The quick brown fox jumps over the lazy dog.&quot;. This will be turned into vectors using word2vec. Then how are these given to the NN?&lt;/p&gt;&#xA;" OwnerUserId="18692" LastActivityDate="2016-05-13T18:58:48.893" Title="In plain English, how to descibe i/o of the TensorFlow for language modelling?" Tags="|deep-learning|tensorflow|language-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11755" PostTypeId="2" ParentId="11743" CreationDate="2016-05-13T19:02:52.497" Score="1" Body="&lt;p&gt;My mistake was silly. I did not notice in the book that the weight matrices are numbered as the layer they are going into..so my $w^2$ above should be $w^3$ and then the result from the book is $\frac{\partial }{\partial w^3_{11}}=a^2_1(a^3_1-y)\sigma'(z^3_1)$ exactly as calculated by just doing the derivatives.&lt;/p&gt;&#xA;" OwnerUserId="1138" LastActivityDate="2016-05-13T19:02:52.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11756" PostTypeId="1" CreationDate="2016-05-13T19:43:10.180" Score="0" ViewCount="401" Body="&lt;ol&gt;&#xA;&lt;li&gt;I have a dataset from survey results of Genetic Testing that looks like below:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gAQde.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gAQde.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details sample csv file here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/open?id=0BzFPqeNZO-6mM2VwZ1pRQ3NxYTA&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://drive.google.com/open?id=0BzFPqeNZO-6mM2VwZ1pRQ3NxYTA&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to create a model to predict the column “actionspost” (the extreme right column of the above file). The column currently holds comma separated values like “2, 4, 6”. Each numerical value is a coded action that a patient would take after knowing the results of the genetic test as per the survey filled by the patient. For example, 2 indicates: Change eating habits, 4 indicates: Getting members of his/her family tested and so on. Based on the features in the csv file above, I want to create a model that would help me predict the “actionspost” values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I thought, I could do this was break down comma separated values in “actionspost” to create duplicate rows for same customer Ids holding only one “actionspost” value at a time (For example: There would be 4 rows for customer ID: C00003 with first row having “actionspost” value of 2, second row with “actionspost” value 8, third with 9 and fourth with 10. Then create subset of the data set for each unique value of “actionspost”. This time, the dependent variable will not be actionspost but something like “isactionpost2?” (Yes =1 and No = 0). And train models for each of this subset for different values within the “actionspost” column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally don’t like the above method as it too cumbersome and in my opinion, not the optimal one too. I was wondering if there could be a better way to address such scenario? My end goal is to try to train different models like Decision Trees, Naïve Bayes, and Neural Nets and check which one leads to better predictability.&lt;/p&gt;&#xA;" OwnerUserId="17771" LastActivityDate="2016-05-27T22:32:17.260" Title="Prediction of a dependent variable having comma separated values" Tags="|predictive-modeling|categorical-data|multiclass-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11758" PostTypeId="1" CreationDate="2016-05-13T20:29:09.057" Score="4" ViewCount="357" Body="&lt;p&gt;I've got a dependent variable in R with about 11,000 class levels. There's ample data for modeling this many classes, however some statistical packages like &lt;code&gt;h2o&lt;/code&gt; only support up to 1,000 classes or so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the best options for classification models in R with this high number of class levels?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-05-13T23:38:31.637" Title="Response variable with very high number of class levels in R" Tags="|r|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11759" PostTypeId="2" ParentId="11758" CreationDate="2016-05-13T20:55:50.510" Score="5" Body="&lt;p&gt;Cluster your &lt;em&gt;n&lt;/em&gt; response classes into &lt;em&gt;m&lt;/em&gt; clusters, where &lt;code&gt;m &amp;lt; 1,000&lt;/code&gt;. With an incoming instance &lt;em&gt;i&lt;/em&gt; assign it to one of the &lt;em&gt;m&lt;/em&gt; clusters using your favorite classification algorithm. Once the instance has been assigned to a specific cluster, you just need to run it through a classifier trained on the classes that comprise that particular cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used this particular architecture to assign texts to one of n possible &quot;emotions&quot;, which I illustrate below. The incoming text was first scored with a binary classifier trained on positive &amp;amp; negative texts. Given the text's probability of being positive (as predicted by the classifier), the text was then routed to either a series of all positive or a series of all negative emotion classifiers where it was a assigned a specific emotion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kndgI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kndgI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This same sort of design should work your case. First you assign your instance a cluster. Given that assignment, send your instance to the next stage - where the final classification occurs. &lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-05-13T23:38:31.637" LastActivityDate="2016-05-13T23:38:31.637" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11760" PostTypeId="1" CreationDate="2016-05-13T21:56:11.860" Score="1" ViewCount="1679" Body="&lt;p&gt;I am studying (via simulation) a system that has several input parameters. The output of the system is influenced by the input parameters. My goal is to identify the parameters that have the most impact on the output of the system. I ran a large amount of simulations and for each one of them I changed the value of each parameters using a random algorithm. To identify the parameters that have the most impact on the output I was planning on doing a basic multivariate linear regression including all the parameters, however, it turned out to have a low R2 and low F-stat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working a little bit out of my element here so I am looking for suggestions regarding what I should investigate next to achieve my goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The context of the study is building energy efficiency and the main difficulty is that some input parameters have multiple effect on the system. For example: if I lower the total lighting power used in the simulation it will have for effect to decrease the cooling load of my building but if at the same time I was increasing the cooling efficiency of my cooling system the energy saved by increasing the efficiency would be less than if the lighting power was not decreased. In this example I would like to show which has more impact in regards to the energy consumption. However, in what I am trying to do I have a lot more input parameters.&lt;/p&gt;&#xA;" OwnerUserId="18696" LastEditorUserId="18696" LastEditDate="2016-05-16T18:03:39.433" LastActivityDate="2016-11-12T21:30:33.590" Title="Simulation - identify the input parameters that impact the most the output" Tags="|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11762" PostTypeId="2" ParentId="11752" CreationDate="2016-05-13T23:19:08.237" Score="1" Body="&lt;p&gt;What is the overall response rate? If it's low (even 15-20%) it may be difficult to find decision stumps that contain one leaf with &gt; 50% response! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could consider oversampling or changing cutoff probability, but I think if your using only 2 leaf trees, your model is bound to struggle.&lt;/p&gt;&#xA;" OwnerUserId="9700" LastActivityDate="2016-05-13T23:19:08.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11763" PostTypeId="1" AcceptedAnswerId="11766" CreationDate="2016-05-14T00:57:30.570" Score="0" ViewCount="184" Body="&lt;p&gt;For instance, why is it that it is more favourable for a weight of [0.25, 0.25, 0.25, 0.25] (for which the L2 penalty is 0.25) instead of simply [1, 0, 0, 0] (for which an L2 penalty is 1)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, both weights would give the same dot product when using W.T * X&lt;/p&gt;&#xA;" OwnerUserId="13527" LastActivityDate="2016-05-14T09:53:14.550" Title="Why does an L2 penalty prefer smaller and more diffuse weight vectors?" Tags="|machine-learning|svm|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11764" PostTypeId="2" ParentId="11756" CreationDate="2016-05-14T02:36:53.083" Score="0" Body="&lt;p&gt;The easiest approach I can think of would be to recode the actionspost values to a multidigit set of binary flags. So if you have 15 possible responses, and the subject either took an action or did not, this would be a 15 digit number with each digit representing an action that was taken (=1) or not (=0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would let you investigate individual actions, every observed combination of actions, as well as number of actions taken very easily without any row duplication.&lt;/p&gt;&#xA;" OwnerUserId="18703" LastActivityDate="2016-05-14T02:36:53.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11766" PostTypeId="2" ParentId="11763" CreationDate="2016-05-14T09:53:14.550" Score="2" Body="&lt;p&gt;You answered this in your question. &quot;Prefer&quot; means &quot;produces a smaller penalty&quot;, and you've identified that the penalty in the first case is smaller. Why would this be a good thing? It amounts to preferring an explanation based a bit on many features, rather than one based entirely on one features. That's often a good bet to avoid overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two weight vectors do not produce the same dot product with other vectors in general. If the vector X contained all identical values, they would.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the 4 input features were regularly identical or nearly so, then it means they're redundant, and you may prefer to use just 1 of the features (your second case), instead of a bit of each. In this case, an L1 penalty would at least be indifferent to the two, not penalize the second one.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-05-14T09:53:14.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11767" PostTypeId="2" ParentId="10032" CreationDate="2016-05-14T11:27:55.510" Score="1" Body="&lt;p&gt;Nothing in the components they used is novel. All approaches have been explored. Checking their references you will notice many researchers doing similar work. The novelty was the pipeline they followed and the combination of model-free and model-based Reinforcement Learning approaches. I will try to give you a non technical different perspective on what they captured.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model-free approaches usually attempt to approximate functions such as Value functions (representing how good it is to be in a particular state - board configuration - in terms of future reward) or parametrized policy functions (probabilities of selecting an action given a state. Briefly, your model gains some kind of '&lt;strong&gt;intuition&lt;/strong&gt;' on which moves are relative good - something similar to the intuition professional Go players have, when they declare that they do a move because it 'feels' good. This is very important at the early stage of the game when planning is inefficient to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model-based approaches attempt to simulate every single possible trajectory of the game in the form of a decision tree. Thus they are useful for &lt;strong&gt;planning&lt;/strong&gt; (before you actually make a move in the game you check and evaluate all possible contingencies and then you decide which move to take from your current position). The MCTS is such an algorithm, creates a decision tree over possible future courses of the game from the current board position, and evaluates these heuristics according to some criteria. The best algorithms in Go so far were based in this algorithm (and is considered as a RL algorithm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in terms of novelty, with few words: combination of planning and intuition, which means combination of MCTS algorithm with function approximators for evaluation of the simulated game trajectories. In this case they used very deep convolutional neural nets for the 'intuition' part. In addition to this, the whole model is data-driven as it was first trained on human expert moves (this could be useful in applications in many other domains apart from gaming). If you examine every single component, there is nothing novel...but the whole process to combine effectively all these elements and achieve Mastery in that complex domain is something novel. Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2016-05-14T11:27:55.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11769" PostTypeId="1" CreationDate="2016-05-14T14:20:16.333" Score="3" ViewCount="538" Body="&lt;p&gt;I have been working on a project as part of my master degree in participation with a firm. &#xA;I developed a predictive model in the past few months that is essentially a document classification model. The biggest limitation of the research and model is the lack of data available for training. I have a small data set of 300 documents, where as the features are in excess of 15000 terms (before feature selection).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How do we identify or estimate the number of data-points (documents) necessary to gain a level of confidence that will allow us to move forward with the testing stage into a production environment ? So how do I estimate the necessary dataset size required for the performance metrics to be viable enough to ensure it is a solid model?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there standards or metrics in the industry for when a model is sufficiently tested and should qualify for production?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I cannot find literature about this topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="18713" LastEditorUserId="18713" LastEditDate="2016-05-30T11:17:48.747" LastActivityDate="2016-06-01T17:02:58.460" Title="From development environment to production" Tags="|machine-learning|predictive-modeling|model-selection|software-development|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11770" PostTypeId="1" CreationDate="2016-05-14T16:37:57.953" Score="0" ViewCount="1298" Body="&lt;p&gt;Assuming I am having the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----+-----+-----+-----+-------+&#xA;| f1  | f2  | f3  | f4  | output|&#xA;+-----+-----+-----+-----+-------+&#xA;| 0.1 | 0.7 | 0.4 | 0.3 |  1    |&#xA;+-----+-----+-----+-----+-------+&#xA;| 0.4 | 0.1 | 0.2 | 0.5 |  0    |&#xA;+-----+-----+-----+-----+-------+&#xA;| 0.3 | 0.7 | 0.4 | 0.7 |  0    |&#xA;+-----+-----+-----+-----+-------+&#xA;| 0.1 | 0.2 | 0.4 | 0.7 |  1    |&#xA;+-----+-----+-----+-----+-------+&#xA;             .&#xA;             .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I am training the data with logistic regression or another classifier. Is there a way to say 'I want an example that outputs 1'? Or is there a way to 'formulate' the 'borders' of the labels?&lt;/p&gt;&#xA;" OwnerUserId="14736" LastActivityDate="2016-07-15T15:21:33.493" Title="Reverse classification model: Is it possible?" Tags="|classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11771" PostTypeId="1" CreationDate="2016-05-14T16:43:01.747" Score="1" ViewCount="824" Body="&lt;p&gt;I have scientific database with articles and coauthors.&#xA;using this database I am training word2vec model on co-authors.&lt;/p&gt;&#xA;&lt;p&gt;Use use case here is to disambiguate authors.&lt;/p&gt;&#xA;&lt;p&gt;I was wondering my approach here can be improved or any suggestions will greatly be appreciated.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/krishnakalyan3/NLP/blob/master/W2V.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Code&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16971" LastEditorUserId="98307" LastEditDate="2020-08-04T22:21:16.307" LastActivityDate="2024-02-06T01:02:25.353" Title="Training Word2Vec with names instead of sentences" Tags="|python|nlp|word-embeddings|word2vec|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11772" PostTypeId="2" ParentId="11723" CreationDate="2016-05-14T16:52:19.253" Score="2" Body="&lt;p&gt;The problem you've described can be formalized as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_decision_process&quot; rel=&quot;nofollow&quot;&gt;Markov decision process&lt;/a&gt;, the foundational problem of &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow&quot;&gt;reinforcement learning&lt;/a&gt;. In broad strokes, reinforcement learning is concerned with how agents (robot) in a given environment (room) out to take actions (movements from one state to another) to maximize some notion of reward. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formalizing your problem requires defining a few parts of an MDP model:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Set of states $S$&lt;/li&gt;&#xA;&lt;li&gt;Set of actions $A$&lt;/li&gt;&#xA;&lt;li&gt;A reward function $R(s)$, defining the reward of arriving in a given state. In your case, a simple scheme of $R(s) = \mathbb{1}(s = s_{goal})$ is one option.&lt;/li&gt;&#xA;&lt;li&gt;A transition function $T(s,a,s')$ giving the probability of winding up in state $s'$ having taken action $a$ from state $s$.(Note that you can model deterministic transitions by returning a value of $1$ for a single $s'$.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the problem goes on infinitely, you'll also need a discount factor $\gamma$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reinforcement learning, the term &lt;strong&gt;optimal policy&lt;/strong&gt; describes a function that returns the best action to take from a given state. I.e., this gives the recommendation you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you &lt;em&gt;know&lt;/em&gt; all of the model components described above—or can at least derive ones that match your problem—you can use a variety of planning algorithms to find the optimal policy, e.g. &lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node19.html&quot; rel=&quot;nofollow&quot;&gt;value iteration&lt;/a&gt; or &lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node20.html&quot; rel=&quot;nofollow&quot;&gt;policy iteration&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't know the rewards and transitions—say, for example, that the robot is seeking a sensor attached to a charging station and you don't know where it is or the size of the enclosing room—you'll likely need to explore algorithms that observe action-outcome pairs and try to learn optimal policy from these learning episodes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A full description of these is beyond the scope of your question, but a good place to start is Sutton &amp;amp; Barto's &lt;em&gt;Reinforcement Learning: An Introduction&lt;/em&gt;. An &lt;a href=&quot;https://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html&quot; rel=&quot;nofollow&quot;&gt;html version&lt;/a&gt; is freely available. Another resource is &lt;a href=&quot;https://www.udacity.com/course/reinforcement-learning--ud600&quot; rel=&quot;nofollow&quot;&gt;RL udacity course&lt;/a&gt; produced by Georgia Tech.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example, you may also want to research potential-based reward functions. Very loosely, one potential might be the robot's distance to goal state, and the reward would be based on changes to this potential value. (This is described in a &lt;a href=&quot;http://www.cs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; by Ng, Harada and Russell, and in unit 6 of the GA Tech course mentioned above.)&lt;/p&gt;&#xA;" OwnerUserId="1154" LastEditorUserId="1154" LastEditDate="2016-05-14T19:52:26.807" LastActivityDate="2016-05-14T19:52:26.807" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11773" PostTypeId="2" ParentId="11756" CreationDate="2016-05-14T19:04:57.263" Score="0" Body="&lt;p&gt;You are in the right direction. &#xA;Indeed, actions must be considered as distinct possible outcomes -&gt; you should create one model per type of action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of data management, what you propose may be dangerous, please directly think about the desired outcome: n different training datasets.&#xA;If you indicate which statistical environment you intend to use, we can propose best practices to code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, in such situation, trying to predict number of reactions or simply is there a reaction (yes/no) is also of interest.&lt;/p&gt;&#xA;" OwnerUserId="18722" LastActivityDate="2016-05-14T19:04:57.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11774" PostTypeId="1" AcceptedAnswerId="11778" CreationDate="2016-05-14T19:12:02.683" Score="4" ViewCount="1733" Body="&lt;p&gt; &#xA;Hello, I am very new to data science, machine learning, and stack overflow. Excuse me for being unclear or asking naive questions. &lt;br&gt;&#xA;&lt;br&gt; My question is as follows: &lt;br&gt;&lt;br&gt;&#xA;From any given document, I am trying to classify it according to the emotions it evokes in readers, using a neural network. However, I am having difficulties with feature selection. I'm thinking of using NLTK and RAKE to extract keywords, but I don't know how I can translate them into features. Should I hash the keywords for one feature? Or, should I find a dictionary of english words (i.e. Wordnet), and use every word in the dictionary as a feature.&#xA;&lt;/p&gt;&#xA;" OwnerUserId="18721" LastActivityDate="2016-05-16T20:46:18.833" Title="Which features do I select from text?" Tags="|machine-learning|classification|neural-network|text-mining|feature-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11775" PostTypeId="1" CreationDate="2016-05-14T22:13:12.383" Score="1" ViewCount="1149" Body="&lt;p&gt;I just started learning about machine learning recently and have a project where I have to develop a program for QR code localization so that a QR code can be detected and read at any angle of rotation. Development will be done in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plan is to gather various images of the QR codes at different angles with different backgrounds. From this I would like to create a dataset for training with neural networks and then testing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue that I'm having is that I can't seem to figure out a correct feature design for the dataset and how to identify the QR code from the images for feature processing. &#xA;Would I use ground-truth images or edge magnitude maps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help with this would be amazing?&#xA;Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="18726" LastActivityDate="2017-04-07T20:59:07.907" Title="Feature selection - QR code localization" Tags="|machine-learning|classification|feature-selection|feature-extraction|supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11776" PostTypeId="2" ParentId="11656" CreationDate="2016-05-15T00:00:19.817" Score="4" Body="&lt;p&gt;smote algorithm depends on the data set you have. &#xA;If you have severe data imbalance, like the one you have in your case smote algorithm may not be able to help if the variations within the minority class is very high and the similarities  between the two classes is very high.&#xA;But How to know if this is the case.&#xA;Try to duplicate samples from the minority class train a non linear svm and check the results if the classification accuracy is very low then this the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smote use knn to create new samples but if the variation within the minority class is very high then using smote will use samples that are not real neighbors even.&#xA;To be honest with you , there is no clear solution for this problem but i can suggest the followings: &#xA;1. Try borderline smote : it is a modified version of smote algorithm&#xA;2. Try smote boosting : it is a modified version of adaboost where adaboost algorithm is augmented with smote&#xA;3. If you can modify the smote boost to consider  borderline smote instead of smote&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-15T00:00:19.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11777" PostTypeId="1" CreationDate="2016-05-15T10:46:04.210" Score="8" ViewCount="8538" Body="&lt;p&gt;&lt;a href=&quot;http://arxiv.org/pdf/1409.0575v3.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://arxiv.org/pdf/1409.0575v3.pdf&lt;/a&gt; Table 2&#xA;says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally I'd like to avoid downloading the 138 GB just for this purpose as I otherwise don't need it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if anyone knew the exact numbers per class in the training set, i.e., how likely is each class in the training set? &lt;/p&gt;&#xA;" OwnerUserId="18734" LastEditorUserId="16284" LastEditDate="2016-05-15T14:35:32.873" LastActivityDate="2016-12-14T12:34:53.587" Title="What is the distribution of categories in imagenet training set (ILSVRC2012)" Tags="|dataset|image-classification|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11778" PostTypeId="2" ParentId="11774" CreationDate="2016-05-15T12:39:30.623" Score="4" Body="&lt;p&gt;Using NLTK in python you should first &lt;strong&gt;Tokenize&lt;/strong&gt; the sentences into words, even you can use &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#n-gram&quot; rel=&quot;nofollow&quot;&gt;Ngram&lt;/a&gt; for 2-Gram or 3-Gram bags of word, the reason I am suggesting &lt;strong&gt;N-Gram&lt;/strong&gt; is that let's suppose you have sentence like: &lt;code&gt;I am not happy with this product&lt;/code&gt;, then &lt;strong&gt;2-Gram&lt;/strong&gt; tokenize it as &lt;code&gt;['not happy', 'happy with', 'with this', 'this product']&lt;/code&gt; here &lt;code&gt;I&lt;/code&gt; and &lt;code&gt;am&lt;/code&gt; are assumed as &lt;code&gt;STOPWORDS&lt;/code&gt;. Using &lt;strong&gt;HashingTF&lt;/strong&gt; you can hash the sentence into a &lt;strong&gt;feature vector&lt;/strong&gt; as &lt;code&gt;['word position': frequency of word, ...]&lt;/code&gt; i.e &lt;strong&gt;highly sparse vectors&lt;/strong&gt;, For Hashing in PySpark check this &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#tf-idf-hashingtf-and-idf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here below python code will help you to &lt;strong&gt;tokenize&lt;/strong&gt; in bags of word&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import string&#xA;&#xA;from nltk.tokenize import word_tokenize&#xA;from nltk.corpus import stopwords&#xA;from nltk.stem.porter import PorterStemmer&#xA;&#xA;PUNCTUATION = set(string.punctuation)&#xA;STOPWORDS = set(stopwords.words('english'))&#xA;STEMMER = PorterStemmer()&#xA;&#xA;example = ['Hello Krishna Prasad, this is test file for spark testing',&#xA;            'Another episode of star',&#xA;            'There are far and away many stars'&#xA;            'A galloping horse using two coconuts'&#xA;            'My kingdom for a horse'&#xA;            'A long time ago in a galaxy far']&#xA;&#xA;def tokenize(text):&#xA;    tokens = word_tokenize(text)&#xA;    lowercased = [t.lower() for t in tokens]&#xA;    no_punctuation = []&#xA;    for word in lowercased:&#xA;        punct_removed = ''.join([letter for letter in word if not letter in PUNCTUATION])&#xA;        no_punctuation.append(punct_removed)&#xA;    no_stopwords = [w for w in no_punctuation if not w in STOPWORDS]&#xA;    stemmed = [STEMMER.stem(w) for w in no_stopwords]&#xA;    return [w for w in stemmed if w]&#xA;&#xA;tokenized_word = [tokenize(text) for text in example]&#xA;for word in tokenized_word:&#xA;    print word&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Out of the above code as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$python WordFrequencyHash.py&#xA;[u'hello', u'krishna', u'prasad', u'test', u'file', u'spark', u'test']&#xA;[u'anoth', u'episod', u'star']&#xA;[u'far', u'away', u'mani', u'starsa', u'gallop', u'hors', u'use', u'two', u'coconutsmi', u'kingdom', u'horsea', u'long', u'time', u'ago', u'galaxi', u'far']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also use &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#word2vec&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt; or &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#countvectorizer&quot; rel=&quot;nofollow&quot;&gt;countvectorizer&lt;/a&gt; for tokenization.&lt;/p&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-05-15T12:39:30.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11779" PostTypeId="1" CreationDate="2016-05-15T16:00:43.597" Score="-2" ViewCount="77" Body="&lt;p&gt;I run the following models..Decision tree, Logistic Regression,Naive Bayes, SVM and Random Forest ..I am given certain input features and  Respnse feature as whether a customer will buy the product or not in that context i am asking this question , i went through possibly all definitions available on internate but could not be able to properly figure out which measure i should consider as proper performance measure , so far as my models are concern Is there any proper meaning for the following concepts in data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Precision&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall&lt;/p&gt;&#xA;&#xA;&lt;p&gt;F measure&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an easy explanation of these concepts with example?&lt;/p&gt;&#xA;" OwnerUserId="18139" LastEditorUserId="18139" LastEditDate="2016-05-16T16:10:24.823" LastActivityDate="2016-05-16T16:10:24.823" Title="Performance Measures" Tags="|predictive-modeling|confusion-matrix|" AnswerCount="1" CommentCount="2" ClosedDate="2016-05-16T13:40:16.333" ContentLicense="CC BY-SA 3.0" />
  <row Id="11780" PostTypeId="2" ParentId="11779" CreationDate="2016-05-15T18:51:05.210" Score="5" Body="&lt;p&gt;Maybe not the answer you would like to hear, but I have to admit that I find wikipedia page on the topic definitively well done...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot;&gt;https://en.wikipedia.org/wiki/Precision_and_recall&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18722" LastActivityDate="2016-05-15T18:51:05.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11781" PostTypeId="2" ParentId="11740" CreationDate="2016-05-15T18:52:16.913" Score="3" Body="&lt;p&gt;This is a great question since it is illuminating to examination the best practices of both about traditional statistics and machine learning as they are brought together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Two Separate Rules/Best Practices -&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the two items that you mentioned should be examined separately and not conflated i.e. they should be carefully combined as you suggest in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Significance&lt;/strong&gt;: You have estimated that you want greater than 1000 cases to have statistical significance and would further benefit from greater than 3000 test cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cross validation&lt;/strong&gt;: Cross validation is performed by splitting the data (often 80% train-20% test) so that tests for bias (~underfitting) and variance (~overfitting) can be assessed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Combining significance with cross validation&lt;/strong&gt;: Now we know that we want to have significance in our tests so we want greater than 3000 records.  We also want to perform cross validation, so in order for both the testing and training data to return significant results we want both to have a minimum of 3000 records.  The best scenario for this would be to have 15,000 total records. This way, the data can be split 80/20 and the testing set is still significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume you only have 4000 records in your data set. In this case, I would opt to make my training data significant while allowing the testing set to drop to lower significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More rigor:&lt;/strong&gt; Everything above has been quite hand wavey and lacks statistical rigor. For this, I refer you to a couple of papers referenced in &lt;a href=&quot;https://stats.stackexchange.com/questions/45851/how-to-statistically-compare-the-performance-of-machine-learning-classifiers&quot;&gt;another Stack Exchange question&lt;/a&gt; -  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dietterich, &quot;Approximate Statistical Tests for Comparing&#xA;Supervised Classication Learning Algorithms&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=21D16E681FF16B0BA91D741A63806A31?doi=10.1.1.29.5194&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Salzberg, Data Mining and Knowledge Discovery, 1, 317–327 (1997), &quot;On Comparing Classifiers: Pitfalls to Avoid and a&#xA;Recommended Approach&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-05-15T18:52:16.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11782" PostTypeId="2" ParentId="11760" CreationDate="2016-05-15T18:54:52.213" Score="0" Body="&lt;p&gt;Sorry I can't comment yet - and this is not an answer: do you change all parameters at same time or do you have a strategy to change parameter per parameter (experiment)?&lt;/p&gt;&#xA;" OwnerUserId="18722" LastActivityDate="2016-05-15T18:54:52.213" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11783" PostTypeId="2" ParentId="11760" CreationDate="2016-05-15T19:20:42.263" Score="1" Body="&lt;p&gt;With my background on Data Analysis and Nonlinear Regression, your question seems to be very general and blurry. I guess what you are looking for is called Sensitivity Analysis in some contexts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to understand that parameters may have different orders of impact on different sub-spaces of variables (and Parameters) space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to generate multiple sub-models based on this sub-spaces, so in each sub-model only a few of your parameters have the main effect and the rest could be neglected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again you need to make your question a little more detailed. for example tell us why you want to do this analysis. because your final goal totally impacts the solution.&lt;/p&gt;&#xA;" OwnerUserId="5200" LastActivityDate="2016-05-15T19:20:42.263" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11785" PostTypeId="2" ParentId="11770" CreationDate="2016-05-15T20:01:20.713" Score="-1" Body="&lt;p&gt;Let s1 to s4 be the signs (+1 or -1) of the regression coefficients of f1 to f4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the predicted value for a data item with columns s1*Inf to s4*Inf will be (as near as precision allows it) 1. You go to the extreme of the parameter space in the right direction (as told by the coefficients signs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inverting the signs and doing the same should give you a data item that is predicted to be 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, code in R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create sample data set.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(12345)&#xA;d2 = data.frame(f1=runif(10), f2 = runif(10), f3 = runif(10),&#xA;      output=sample(c(0,1),10,TRUE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Fit model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m2 = glm(output~f1+f2+f3, data=d2, family=&quot;binomial&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Look at model parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coef(m2)&#xA;## (Intercept)          f1          f2          f3 &#xA;##   5.258817   -4.848013  -15.370838    4.890984 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This means as &lt;code&gt;f1&lt;/code&gt; and &lt;code&gt;f2&lt;/code&gt; increase, the fitted probability of zero increases because their signs are negative. As &lt;code&gt;f3&lt;/code&gt; increases, the fitted probability of one increases because it has a positive sign. So....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(m2, newdata=data.frame(f1=-Inf, f2=-Inf, f3=Inf), type=&quot;response&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be one, which it is, and:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(m2, newdata=data.frame(f1=Inf, f2=Inf, f3=-Inf), type=&quot;response&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be zero, which it is, within tolerance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Zeroes and ones in logistic regression responses are like plus or minus infinity in a linear regression, so if you think back to a simple X-Y fit, you get an infinite response to a linear regression when you go to +Inf on the X-axis if the slope is positive or -Inf on the X-axis if the slope is negative. Transform that to a logistic regression and then you get to +1 or 0 at the appropriate signed infinities of the covariates.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2016-05-16T11:37:00.587" LastActivityDate="2016-05-16T11:37:00.587" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11787" PostTypeId="2" ParentId="8521" CreationDate="2016-05-16T00:17:26.587" Score="0" Body="&lt;p&gt;SVM on so many training vectors will take a very long time and a lot of memory[&lt;a href=&quot;http://www.jmlr.org/papers/volume6/tsang05a/tsang05a.pdf&quot; rel=&quot;nofollow&quot;&gt;O(n^3) time and O(n^2) space&lt;/a&gt;, n is the no. of training vector]. You could use an SVM library with GPU speed up, that might help a little. As mentioned in earlier posts, features don't matter much for SVMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your C value? For some values of C, it takes extra long. You could try tuning it with a few vectors and grid search. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks learn well from such huge datasets, coupled with RBM it should give good results. Or you could try Random Forests/Decision Trees with Bagging/XGBoost or other &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble&quot; rel=&quot;nofollow&quot;&gt;ensemble methods&lt;/a&gt;. They are surprisingly good and can be super fast (relative to SVM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, this might be unimportant but I don't have enough reputation to comment- what sort of visualisation is this- PCA? If this doesn't give you a good idea, you could try &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.MDS.html#sklearn.manifold.MDS&quot; rel=&quot;nofollow&quot;&gt;MDS&lt;/a&gt;, maybe?&lt;/p&gt;&#xA;" OwnerUserId="16812" LastActivityDate="2016-05-16T00:17:26.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11788" PostTypeId="1" AcceptedAnswerId="11789" CreationDate="2016-05-16T11:36:14.850" Score="17" ViewCount="22739" Body="&lt;p&gt;I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples?&lt;/li&gt;&#xA;&lt;li&gt;The same question for 1,000 positive example and 100,000 negative.&lt;/li&gt;&#xA;&lt;li&gt;The same question for 10,000 positive and 1,000 negative.&lt;/li&gt;&#xA;&lt;li&gt;etc...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="29575" LastEditDate="2018-06-03T12:47:19.500" LastActivityDate="2018-06-03T12:47:19.500" Title="When should we consider a dataset as imbalanced?" Tags="|classification|dataset|sampling|class-imbalance|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="11789" PostTypeId="2" ParentId="11788" CreationDate="2016-05-16T12:58:55.243" Score="11" Body="&lt;p&gt;I think subsampling (downsampling) is a popular method to control class imbalance at the base level, meaning it fixes the root of the problem.  So for all of your examples, randomly selecting 1,000 of the majority of the class each time would work.  You could even play around with making 10 models (10 folds of 1,000 majority vs the 1,000 minority) so you will use your whole data set.  You can use this method, but again you're kind of throwing away 9,000 samples unless you try some ensemble methods.  Easy fix, but tough to get an optimal model based on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The degree to which you need to control for the class imbalance is based largely on your goal.  If you care about pure classification, then imbalance would affect the 50% probability cut off for most techniques, so I would consider downsampling.  If you only care about the order of the classifications (want positives generally more higher than negatives) and use a measure such as AUC, the class imbalance will only bias your probabilities, but the relative order should be decently stable for most techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is nice for class imbalance because as long as you have  &gt;500 of the minority class, the estimates of the parameters will be accurate enough and the only impact will be on the intercept, which can be corrected for if that is something you might want.  Logistic regression models the probabilities rather than just classes, so you can do more manual adjustments to suit your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of classification techniques also have a class weight argument that will help you focus on the minority class more.  It will penalize a miss classification of a true minority class, so your overall accucracy will suffer a little bit but you will start seeing more minority classes that are correctly classified.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-05-16T12:58:55.243" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11790" PostTypeId="2" ParentId="11774" CreationDate="2016-05-16T13:23:06.743" Score="1" Body="&lt;p&gt;A &lt;strong&gt;feature extractor&lt;/strong&gt; is just a function that returns the value of a feature given a target instance. For example, given in the input sentence &quot;&lt;em&gt;I friggin hate regex&lt;/em&gt;&quot;, you could run a tokenizer on it to break it into a list of words. You could then have a feature extractor function &quot;hasCurseWord(tokens)&quot; that returns true or false indicating presence of a curse word (you could have a dict of predefined curse words you compare to. Similarly you could write a feat extractor that numCurseWords that returns the count of curse words in a text. By analogy you can do the same thing with positive and negative words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in answer to your query, in addition to individual grams and perhaps phrasal ngrams (as mentioned by another answer), you will want to add custom feature extractors - e.g. curse words are good indicators of sentiment - to increase the accuracy of your sentiment classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what a feature extractor that counts developer-defined negative words might look like in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def featx_negative(tokens, args=None):&#xA;       #negative terms&#xA;    num_neg = 0&#xA;    num_neg += len([w for w in tokens if re.search('hate', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('stuck', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('smh', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('angry', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('mad', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('blow', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('trash', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('garbage', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('bad', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('worst', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('&amp;lt;&amp;lt;&amp;lt;', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('dead', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('die', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('boo', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('horrib', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('terrib', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('annoy', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('wrong', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('dump', w, re.IGNORECASE)])&#xA;    num_neg += len([w for w in tokens if re.search('mess', w, re.IGNORECASE)])&#xA;    features = {}&#xA;    features['Has(NEGATIVE)'] = True if num_neg &amp;gt; 0 else False&#xA;    return features&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above approach, is what might be called a dictionary-approach -- essentially each feature extractor compares an input sentence to an externally defined dictionary of terms. The limitations of this approach are that you must manually create the dictionaries themselves (or find some external data source of positive/negative words). In addition, the dictionaries are static (what happens when a new slang term comes into existence?) and they essentially weight terms equally. The alternative approach would be to use a machine-learning algorithm that gets trained on sentences pre-labelled by hand (as pos/neg). This approach allows you to weight your words according to their actual distribution (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf&lt;/a&gt;) or chi-sq. For further details, see this &lt;a href=&quot;https://datascience.stackexchange.com/questions/10531/create-training-data/10532#10532&quot;&gt;answer to creating training data&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-05-16T20:46:18.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11793" PostTypeId="1" AcceptedAnswerId="11800" CreationDate="2016-05-16T18:59:01.567" Score="2" ViewCount="1000" Body="&lt;p&gt;Most recommendation algorithms recommend new products to users. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you bought this you might like that&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But sometimes the item user is most likely to buy is an item that he bought sometime ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any algorithm appropriate for this use?&lt;/p&gt;&#xA;" OwnerUserId="4978" LastActivityDate="2016-05-17T06:16:34.333" Title="Recommendation model that can recommend already bought item" Tags="|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11794" PostTypeId="2" ParentId="11793" CreationDate="2016-05-16T20:25:41.513" Score="1" Body="&lt;p&gt;What you are actually looking for is a technique called &lt;strong&gt;Content-Based Filtering&lt;/strong&gt;&#xA;&lt;br&gt;&#xA;As Wikipedia defines it:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Another common approach when designing recommender systems is content-based filtering. Content-based filtering methods are based on a description of the item and a profile of the user’s preference. In a content-based recommender system, keywords are used to describe the items and a user profile is built to indicate the type of item this user likes. In other words, these algorithms try to recommend items that are similar to those that a user liked in the past (or is examining in the present). In particular, various candidate items are compared with items previously rated by the user and the best-matching items are recommended. This approach has its roots in information retrieval and information filtering research.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;More can be found &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system#Content-based_filtering&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18150" LastActivityDate="2016-05-16T20:25:41.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11797" PostTypeId="1" AcceptedAnswerId="11799" CreationDate="2016-05-17T01:37:04.153" Score="9" ViewCount="30496" Body="&lt;p&gt;I am new to python and stuck at a particular problem involving dataframes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0KGgA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0KGgA.png&quot; alt=&quot;Sample Image clipped from Spyder&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image has a sample column, however the data is not consistent. There are also some floats and NAN. I need these to be split across columns. That is each unique value becomes a column in the df.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insights?&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2019-01-22T13:49:05.290" Title="Split a list of values into columns of a dataframe?" Tags="|python|pandas|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11798" PostTypeId="1" CreationDate="2016-05-17T05:07:59.847" Score="2" ViewCount="1050" Body="&lt;p&gt;I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this &quot;underfitting&quot;, because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points of uncertainty (outputs around 0.5) are in yellow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the sigmoid as the activation function, which is simply &lt;code&gt;1/(1+((math.e) ** (-sum)))&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d=0&#xA;error = 0&#xA;while d &amp;lt; (tPoints): # while all the points in the list have not been feedforwarded&#xA;    output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1)&#xA;    output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1)&#xA;&#xA;    if output1 &amp;lt; 0.99:&#xA;        error += (output1)**2&#xA;    if output2 &amp;gt; 0.01:&#xA;        error += (1-output2)**2&#xA;    d+=1&#xA;error = error/tPoints&#xA;return error # return mean-squared error&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias):&#xA;&#xA;    inputList = n.getNeuronList()[0]&#xA;    hidden1 = n.getNeuronList()[1]&#xA;    hidden2 = n.getNeuronList()[2]&#xA;    outputs = n.getNeuronList()[3]&#xA;    bias = n.getNeuronList()[4]&#xA;&#xA;    FFInputs = []&#xA;    FFHidden1 = []&#xA;    FFHidden2 = []&#xA;    FFOutputs = []&#xA;&#xA;    for inp in inputList:&#xA;        FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs()))&#xA;    for hidden in hidden1:&#xA;        FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias[0]))&#xA;    for hidden in hidden2:&#xA;        FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias[1]))&#xA;    for out in outputs:&#xA;        FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias[2]))&#xA;&#xA;    return FFOutputs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PQVt0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PQVt0.png&quot; alt=&quot;fail&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Graph done with matplotlib.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?&lt;/p&gt;&#xA;" OwnerUserId="18778" LastEditorUserId="18778" LastEditDate="2016-05-18T04:11:16.000" LastActivityDate="2016-07-17T14:51:56.740" Title="Why does my Multilayer Perceptron only classify linearly?" Tags="|machine-learning|classification|neural-network|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="11799" PostTypeId="2" ParentId="11797" CreationDate="2016-05-17T06:08:12.007" Score="9" Body="&lt;p&gt;It looks like you're trying to &quot;featurize&quot; the genre column.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pandas.Series([('Adventure', 'Drama', 'Fantasy'), ('Comedy', 'Family'), ('Drama', 'Comedy', 'Romance'), (['Drama']), &#xA;                    (['Documentary']), ('Adventure', 'Biography', 'Drama', 'Thriller')]).apply(frozenset).to_frame(name='genre')&#xA;for genre in frozenset.union(*df.genre):&#xA;    df[genre] = df.apply(lambda _: int(genre in _.genre), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| row | genre                                   | Romance | Documentary | Thriller | Biography | Family | Drama | Comedy | Adventure | Fantasy |&#xA;|-----|-----------------------------------------|---------|-------------|----------|-----------|--------|-------|--------|-----------|---------|&#xA;| 0   | (Drama, Adventure, Fantasy)             | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 1         | 1       |&#xA;| 1   | (Comedy, Family)                        | 0       | 0           | 0        | 0         | 1      | 0     | 1      | 0         | 0       |&#xA;| 2   | (Drama, Comedy, Romance)                | 1       | 0           | 0        | 0         | 0      | 1     | 1      | 0         | 0       |&#xA;| 3   | (Drama)                                 | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 0         | 0       |&#xA;| 4   | (Documentary)                           | 0       | 1           | 0        | 0         | 0      | 0     | 0      | 0         | 0       |&#xA;| 5   | (Drama, Biography, Adventure, Thriller) | 0       | 0           | 1        | 1         | 0      | 1     | 0      | 1         | 0       |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-05-17T06:08:12.007" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11800" PostTypeId="2" ParentId="11793" CreationDate="2016-05-17T06:16:34.333" Score="2" Body="&lt;p&gt;Usually recommendation algorithms provides the confidence that a user will like an item. Items that the user already bought should get high confidence - the user bought them since he liked them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases filtering out item the user already bough is done in the application level, not in the algorithm level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So , you can use regular recommendation algorithms in order to know if the user will like the item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that you might face a different problem - will the user buy the item for the second/third time. &#xA;In order to cope with this problem, using some domain knowledge might be beneficial. If the products are usually bough many times (e.g., milk), just use the recommendation algorithm. If all the products have the same tendency to be bought few times, build a model for that (e.g., probability for buying another time given already buying x times) and combine the models.&#xA;If your products are very different from this aspect you might need to get into a lower level. Different domains require different solution but you might split the product by the buying behavior and train few recommendation systems, add the number of buying as a feature, etc. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-05-17T06:16:34.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11801" PostTypeId="2" ParentId="11747" CreationDate="2016-05-17T08:33:29.873" Score="2" Body="&lt;p&gt;Not in Keras. I normally just use sklearn's &lt;code&gt;train_test_split&lt;/code&gt; function:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;&#xA;train, test = train_test_split(data, train_size=0.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Keras also has &lt;a href=&quot;http://keras.io/scikit-learn-api/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn wrappers&lt;/a&gt; that might be useful later on.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="85045" LastEditDate="2020-12-29T19:00:35.780" LastActivityDate="2020-12-29T19:00:35.780" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="11802" PostTypeId="1" AcceptedAnswerId="11875" CreationDate="2016-05-17T08:52:42.160" Score="1" ViewCount="1579" Body="&lt;p&gt;In R I have &lt;code&gt;data&lt;/code&gt; and I want to make a regression analysis, finding a function that can fit the data.&#xA;So &lt;code&gt;head(data)&lt;/code&gt; gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;promotion   new_users&#xA;39.5              100&#xA;36.1               79&#xA; 0.0               18&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To find the optimal regression function that fitted data I plot the residuals of the regression model to see if the residuals are systematic close the zero line. But I do not know which transformation is the best one to use. Here I tried to do linear transformation, &lt;code&gt;sqrt&lt;/code&gt;transformation and finally log-transformation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm.linear = lm(formula= data$new_users ~ data$promotion )&#xA;plot(resid(lm.linear), col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aORT9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aORT9.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm.sqrt = lm(formula= data$new_users ~ sqrt(data$promotion) )&#xA;plot(resid(lm.sqrt), col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wNSIz.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wNSIz.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm.log = lm(formula= data$new_users ~ log(0.1+data$promotion ))&#xA;plot(resid(lm.log), col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vBByy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vBByy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I simply just plot the data and the fitted regression function I can't see which regression function fittest the data best because they are very similar.&#xA;Which transformation is the best one and is there another way to find out what is the best transformation ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see if I can use poission regression model I type&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=glm(data$new_users ~data$promotion, familiy=&quot;poisson&quot;, data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I use goodnes of fit to see if the model fits.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with(m1, cbind(res.deviance = deviance, df = df.residual,&#xA;  p = pchisq(deviance, df.residual, lower.tail=FALSE)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get a low p-value meaning that the model is a good fit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore say we want to compare two poison regression models (one poission regr. model from &lt;code&gt;data&lt;/code&gt; and another one from another dataset say &lt;code&gt;data_new&lt;/code&gt;) and see if there is a significantly difference between the two, how would one do that? I assume one could use anova test to solve this. &lt;/p&gt;&#xA;" OwnerUserId="18787" LastEditorUserId="18787" LastEditDate="2016-05-23T12:04:10.553" LastActivityDate="2018-09-08T19:27:08.663" Title="How to see which transformation is the best" Tags="|regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11803" PostTypeId="1" CreationDate="2016-05-17T10:16:02.623" Score="9" ViewCount="248" Body="&lt;p&gt;I want to add a recommendation feature to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Document_management_system&quot; rel=&quot;noreferrer&quot;&gt;document management system&lt;/a&gt;. It is a server on which most company documents are stored. Employees browse the web interface and click to download (or read online) the documents they want.&lt;br&gt;&#xA;&lt;strong&gt;Each employee only has access to a subset of all documents:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6HPbXm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6HPbXm.png&quot; alt=&quot;Employees only have access to a subset of all documents&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My goal&lt;/strong&gt;: Recommend to an employee the documents recently opened by their teammates, or the spreadsheet that serves as an annex to the document they just opened, or anything they might want to read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many recommendation engines for &lt;em&gt;publicly&lt;/em&gt; available data (all Netflix users can see all movies), but the situation here is special: Each employee only has the permission to a fraction of all documents, whereas in Netflix any user has access to all movies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;: Employee1 can read DocumentA but not DocumentB. Employee2 can read both and Employee3 cannot read any.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, I must not recommend to an employee documents to which she/he does not have access. Furthermore, I guess I should consider the popularity of a document only in the context of the employees who have access to it. To make things even more complicated, employees sometimes move from a project to another, which impacts the documents to which they have access.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a name for this kind of problem?&lt;/li&gt;&#xA;&lt;li&gt;Can it be reduced without precision/efficiency loss to a more common kind of problem?&lt;/li&gt;&#xA;&lt;li&gt;If not, what approach would work well for this kind of problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note: A Netflix-like recommendation engine is not good enough. A document with 50 views should be prominent if only 10 employees (including me) have access to it, but not prominent if 100000 employees have access to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;In case it is needed, here are a few data specifics: The average company has 1000 employees, about 10000 documents, an employee clicks about 5 documents per day. Each project has an average of 10 employees having access to it, and has about 100 documents. Each employee works on an average of 5 projects in parallel.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="18790" LastEditorUserId="18790" LastEditDate="2016-05-19T02:53:11.157" LastActivityDate="2016-08-11T15:54:48.163" Title="What recommendation engine for a situation where users can only see a fraction of all items?" Tags="|machine-learning|recommender-system|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11804" PostTypeId="2" ParentId="11713" CreationDate="2016-05-17T10:30:33.680" Score="1" Body="&lt;p&gt;You could use either of the two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Just count whether your documents in your corpus contain or not a word $w$ and then estimate the likelihoods of $P(w_1|C)...P(w_n|C)$, as you describe&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You could count how many instances of any word has each of the documents of your corpus, but then the estimation of the likelihoods becomes a bit more complicated than what you describe; in essence you need to use a multinomial distribution: $P(w_1|C=k) = \sum_{i=1}^{N}x_{it}z_{ik}/\sum_{s=1}^{|V|}\sum_{i=1}^{N}x_{is}z_{ik}$, where $N$ is the number of documents, $x_{it}$ in the numerator is the counts of word $w_1$ per document, $s$ in the denominator iterates the number of terms $|V|$ in your corpus and $x_{is}$ is the number times each term appears in each document, and finally $z_{ik}$ is 1 if the document belongs to class k, 0 otherwise, in both numerator in the denominator&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hopefully the short explanation makes sense, anyway you have a very complete and useful reference &lt;a href=&quot;http://www.inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/inf2b-learn-note07-2up.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; (the same notation in the formula above).&lt;/p&gt;&#xA;" OwnerUserId="9584" LastActivityDate="2016-05-17T10:30:33.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11805" PostTypeId="2" ParentId="11803" CreationDate="2016-05-17T10:43:12.000" Score="0" Body="&lt;p&gt;From your description, I would suggest that you look towards methods called &lt;a href=&quot;http://recommender-systems.org/collaborative-filtering/&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt;. Basically, you could treat any view/download of a document as a positive feedback for some item and then recommend such items for users looking into similar documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Filtering of hidden results should be done on per-user basis (you find all possible suggestions, but output only those that user can rights to see).&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-05-17T10:43:12.000" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11806" PostTypeId="2" ParentId="11788" CreationDate="2016-05-17T10:50:58.913" Score="4" Body="&lt;p&gt;Imbalance is not defined formally but a ratio of 1 to 10 is usually imbalanced enough to benefit from using balancing technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two type of imbalance, relative and absolute. In the relative the ratios between the majority and minority classes are imbalanced. In absolute you also have a small number of minority samples. The higher the imbalance ratio, the more likely you will reach absolute imbalance too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that straight forward subsampling is not an optimal way to cope with imbalanced dataset. That is because you should build a classifier that will perform well on your original dataset. For a technique for building classifiers on imbalanced datasets see &lt;a href=&quot;https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set/answer/Dan-Levin-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;For evaluating your classifier see &lt;a href=&quot;https://datascience.stackexchange.com/questions/810/should-i-go-for-a-balanced-dataset-or-a-representative-dataset/8628#8628&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-05-17T10:50:58.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11807" PostTypeId="2" ParentId="11802" CreationDate="2016-05-17T11:02:23.867" Score="0" Body="&lt;p&gt;The best transformation is often a subjective decision. So you need to makr your own choicr with a sound explanation of why. In my opinion it looks as though the log transformation leads to the strongest correlation.&lt;/p&gt;&#xA;" OwnerUserId="17959" LastActivityDate="2016-05-17T11:02:23.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11808" PostTypeId="2" ParentId="11802" CreationDate="2016-05-17T12:05:54.373" Score="2" Body="&lt;p&gt;There are two main reasons people typically transform their data.  Either to &lt;strong&gt;help meet assumptions&lt;/strong&gt; or to &lt;strong&gt;help boost predictive performance&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Help meet assumptions&lt;/strong&gt;&#xA;In your case of a linear regression,  you have several assumptions that you have to meet including things like linearity and residuals have a constant variance and appear independent of each other.  Transforming your response can help make your relationship more linear and help with your residual assumptions, while a transformation on your predictors primarily helps with those residuals.  Typically if you are doing inference, you need normal residuals (or large sample size for CLT to kick in) which you can assess with a qq-plot or a test for normality, but if you are doing predictions you don't have to worry about this.  Trying a few popular transformations (sqrt, square, log) is a good idea, as is the Box-Cox transformation (on predictors) which basically finds a transformation that maximizes the normal likelihood function of the residuals.  The box-cox goes through several power transformations in a systematic manner, and it primarily used to make the residuals normal, though it can help the other issues too.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Improve predictive power&lt;/strong&gt;&#xA;This is more of an open game, and I would be wary of over fitting.  See what works best and go with it (can be tough, might have to go with cross-validation if you're data isn't too big).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Your Case&lt;/strong&gt;&#xA;Usually your x-axis for those plots is the fitted values (you can plot a glm object, not sure about lm, so run through most of your basic diagnostic plots). just with those plots its also tough to assess normality, which is typically done with a qq-plot or a hisotgram of the residuals.  It looks like your response variable new_users is a count; maybe a poisson model would be another option.  &quot;best&quot; transformation is the one that helps you meet your assumptions as noted in (1), its more of finding the transformation that makes you feel most confident that your assumptions are met.  If you are looking to increase performance, $R^2$ will help indicate a helpful transformation, but again be wary of overfitting.  The power transformations e.g.$(x^{1/3}), log(x+0.1), x^{2}$ are usually a good route to go.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-05-17T12:05:54.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11809" PostTypeId="1" AcceptedAnswerId="11822" CreationDate="2016-05-17T13:12:49.063" Score="0" ViewCount="83" Body="&lt;p&gt;This is a bit of a stretch topic, but I find myself often browsing gear websites (like, &lt;a href=&quot;http://www.gearpatrol.com&quot; rel=&quot;nofollow&quot;&gt;Gear Patrol&lt;/a&gt; and &lt;a href=&quot;http://www.uncrate.com&quot; rel=&quot;nofollow&quot;&gt;Uncrate&lt;/a&gt;) for birthday and gift ideas.  I find it cumbersome to click through each and every article looking for items that are either within my price range or within my price range &amp;amp;&amp;amp; category of item that I'm looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that I can go to Amazon and set specific filters for items that I am looking for, but I enjoy receiving suggestions from websites that are known to give quality product recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a data mining / web scraping standpoint, does anyone know of any resources of tutorials online that provide guidance for similar products?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is R or Python better suited for web scraping applications?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insight into my idea would be greatly appreciated :)&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2016-05-18T09:00:12.697" Title="Data Mining Gear/Goods Websites for Specific Prices" Tags="|r|python|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11811" PostTypeId="2" ParentId="11665" CreationDate="2016-05-17T15:31:06.320" Score="1" Body="&lt;p&gt;This looks like almost an ideal use case for a Recurrent Neural Network (RNN) model. I say almost, as your output variable is 2 separate class vectors, that part is slightly fiddly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main advantage of using a RNN is that the model would be built on the assumption that &lt;code&gt;s1_var1&lt;/code&gt; is of the same type as &lt;code&gt;s2_var1&lt;/code&gt; etc. Whilst with other models you &lt;em&gt;could&lt;/em&gt; flatten the time series into a 10 x 20 = 200 long vector, and the output likewise into a 50 long vector, and treat the whole problem as a single step supervised learning, such a model would not include the same assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your problem, the RNN would have 10 inputs (corresponding to the 5 classes of each of 2 variables) and 10 outputs. You would train it by presenting your input sequence one sample at a time in the correct order 1 to 20, then continue running the network 5 more steps, comparing the outputs against expected. When predicting, you do similar, but then just read off the outputs as predictions, one at a time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few different Python libraries that support RNN models, and there are different internal choices (such as using LSTM layer versus GRU) which you might want to explore. I am currently learning the &lt;a href=&quot;http://keras.io/&quot; rel=&quot;nofollow&quot;&gt;Keras&lt;/a&gt; framework, which supports some options for RNNs. Probably the example code which &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/stateful_lstm.py&quot; rel=&quot;nofollow&quot;&gt;learns to predict an exponentially-decaying sine wave&lt;/a&gt; would be a good place to start trying to understand the Keras model - although that's a regression, it should give you some insight into how RNNs can be built for this kind of series-based data.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;An alternative to RNN here might be a &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow&quot;&gt;hidden markov model&lt;/a&gt; - if you have some insight into the internal state of the system, that could be a better option.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-17T15:39:40.440" LastActivityDate="2016-05-17T15:39:40.440" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11812" PostTypeId="1" AcceptedAnswerId="11901" CreationDate="2016-05-17T16:26:20.703" Score="4" ViewCount="3320" Body="&lt;p&gt;My problem is different from the common time series data problem. What I need to do is check if future time series data is in accord with previous time series data I already consider to be correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I need a one-class classifier applied to time series data, which have variable length (going from 110 to 125 points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know any lib that might help me with this task? I usually use Python+scikit-learn, but I'm open to use any other configuration.&lt;/p&gt;&#xA;" OwnerUserId="14153" LastActivityDate="2016-06-01T17:20:30.357" Title="One-class classifier for time series data classification" Tags="|classification|time-series|" AnswerCount="4" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="11813" PostTypeId="1" CreationDate="2016-05-17T16:49:37.667" Score="5" ViewCount="776" Body="&lt;p&gt;I have images of identity cards (manually taken so not of same size) and I need to extract the text in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used tesseract to predict bounding boxes for each letter and am successful to some extent but some letters are not bounded.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cgL10.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cgL10.png&quot; alt=&quot;http://i.imgur.com/USVKp6h.png&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, I have around 5000 bounding boxes in all images combined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train it so as to predict bounding boxes for remaining letters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After predicting the bounding boxes I will try to classify the image into characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is different from conventional machine learning problem where I donot have training and testing data separately.&lt;/p&gt;&#xA;" OwnerUserId="12613" LastEditorUserId="29169" LastEditDate="2021-02-16T15:49:16.290" LastActivityDate="2021-02-16T15:49:16.290" Title="How to predict on part of image after training on other part of image?" Tags="|deep-learning|neural-network|image-classification|convolutional-neural-network|computer-vision|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11814" PostTypeId="2" ParentId="11788" CreationDate="2016-05-17T19:12:29.980" Score="3" Body="&lt;p&gt;Data imbalance problem ??&#xA;In theory, it is only about numbers.&#xA;Even if the difference is 1 sample it is data imbalance &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practical, saying this is a data imbalance problem is controlled by three things:&#xA;1. The number and distribution of Samples you have &#xA;2. The variation within the same class &#xA;3. The similarities between different classes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last two points change how we consider our problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain this let me give an example:&#xA;Class A = 100 samples&#xA;Class B = 10 000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the variation within class B is very low &#xA;Then down sampling will be enough , there is no data imbalance problem &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the variation is very high within class b , then down sampling may lead to information loss &#xA;And it is dangerous to apply down sampling &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another point, having a lot of samples ( mainly for the minority class) will relax the data imbalance problem and make it easier to deal with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.   10 : 100.   Vs.   1000 : 10 000&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-05-17T19:12:29.980" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11816" PostTypeId="1" AcceptedAnswerId="11817" CreationDate="2016-05-18T03:19:58.333" Score="2" ViewCount="4138" Body="&lt;p&gt;&lt;strong&gt;I have install NLTK and its working fine with the following code, I running in pyspark shell&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from nltk.tokenize import word_tokenize&#xA;&amp;gt;&amp;gt;&amp;gt; text = &quot;Hello, this is testing of nltk in pyspark, mainly word_tokenize functions in nltk.tokenize, working fine with PySpark, please see the below example&quot;&#xA;&amp;gt;&amp;gt;&amp;gt; text&#xA;//'Hello, this is testing of nltk in pyspark, mainly word_tokenize functions in nltk.tokenize, working fine with PySpark, please see the below example'&#xA;&amp;gt;&amp;gt;&amp;gt; word_token  = word_tokenize(text)&#xA;&amp;gt;&amp;gt;&amp;gt; word_token&#xA;//['Hello', ',', 'this', 'is', 'testing', 'of', 'nltk', 'in', 'pyspark', ',', 'mainly', 'word_tokenize', 'functions', 'in', 'nltk.tokenize', ',', 'working', 'fine', 'with', 'PySpark', ',', 'please', 'see', 'the', 'below', 'example']&#xA;&amp;gt;&amp;gt;&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;When I try to run it using spark inbuild method &lt;code&gt;map&lt;/code&gt; it throwing error &lt;code&gt;ImportError: No module named nltk.tokenize&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from nltk.tokenize import word_tokenize&#xA;&amp;gt;&amp;gt;&amp;gt; rdd = sc.parallelize([&quot;This is first sentence for tokenization&quot;, &quot;second line, we need to tokenize using word_tokenize method in spark&quot;, &quot;similar sentence here&quot;])&#xA;&amp;gt;&amp;gt; rdd_tokens = rdd.map(lambda sentence : word_tokenize(sentence))&#xA;&amp;gt;&amp;gt; rdd_tokens&#xA;// PythonRDD[2] at RDD at PythonRDD.scala:43&#xA;&amp;gt;&amp;gt;&amp;gt; rdd_tokens.collect()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using spark version:1.6.1 and python version: 2.7.9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Fullstack errors&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from nltk.tokenize import word_tokenize&#xA;&amp;gt;&amp;gt;&amp;gt; rdd = sc.parallelize([&quot;This is first sentence for tokenization&quot;, &quot;second line, we need to tokenize using word_tokenize method in spark&quot;, &quot;similar sentence here&quot;])&#xA;&amp;gt;&amp;gt; rdd_tokens = rdd.map(lambda sentence : word_tokenize(sentence))&#xA;&amp;gt;&amp;gt; rdd_tokens&#xA;// PythonRDD[2] at RDD at PythonRDD.scala:43&#xA;&amp;gt;&amp;gt;&amp;gt; rdd_tokens.collect()&#xA;    16/05/17 17:06:48 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 2.0 (TID 16, spark-w-0.c.clean-feat-131014.internal): org.apache.spark.api.python.PythonException: Traceback (most recent call last):&#xA;      File &quot;/usr/lib/spark/python/pyspark/worker.py&quot;, line 98, in main&#xA;        command = pickleSer._read_with_length(infile)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 164, in _read_with_length&#xA;        return self.loads(obj)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 422, in loads&#xA;        return pickle.loads(obj)&#xA;    ImportError: No module named nltk.tokenize&#xA;&#xA;        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)&#xA;			at org.apache.spark.api.python.PythonRunner$$anon$1.&amp;lt;init&amp;gt;(PythonRDD.scala:207)&#xA;			at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)&#xA;			at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)&#xA;			at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)&#xA;			at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)&#xA;			at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)&#xA;			at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;			at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;        at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;    16/05/17 17:06:49 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 2.0 failed 4 times; aborting job&#xA;    16/05/17 17:06:49 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.3 in stage 2.0 (TID 23, spark-w-0.c.clean-feat-131014.internal): org.apache.spark.TaskKilledException&#xA;        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:204)&#xA;			at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;			at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;        at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;    Traceback (most recent call last):&#xA;      File &quot;&amp;lt;stdin&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;      File &quot;/usr/lib/spark/python/pyspark/rdd.py&quot;, line 771, in collect&#xA;        port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())&#xA;      File &quot;/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py&quot;, line 813, in __call__&#xA;      File &quot;/usr/lib/spark/python/pyspark/sql/utils.py&quot;, line 45, in deco&#xA;        return f(*a, **kw)&#xA;      File &quot;/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py&quot;, line 308, in get_return_value&#xA;    py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.&#xA;    : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 22, spark-w-0.c.clean-feat-131014.internal): org.apache.spark.api.python.PythonException: Traceback (most recent call last):&#xA;      File &quot;/usr/lib/spark/python/pyspark/worker.py&quot;, line 98, in main&#xA;        command = pickleSer._read_with_length(infile)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 164, in _read_with_length&#xA;        return self.loads(obj)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 422, in loads&#xA;        return pickle.loads(obj)&#xA;    ImportError: No module named nltk.tokenize&#xA;&#xA;        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)&#xA;			at org.apache.spark.api.python.PythonRunner$$anon$1.&amp;lt;init&amp;gt;(PythonRDD.scala:207)&#xA;			at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)&#xA;			at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)&#xA;			at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)&#xA;			at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)&#xA;			at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)&#xA;			at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;			at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;        at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;    Driver stacktrace:&#xA;        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)&#xA;			at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)&#xA;        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)&#xA;			at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)&#xA;			at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)&#xA;			at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)&#xA;			at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)&#xA;        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)&#xA;			at scala.Option.foreach(Option.scala:236)&#xA;			at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)&#xA;			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)&#xA;			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)&#xA;			at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)&#xA;			at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)&#xA;			at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)&#xA;			at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)&#xA;			at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)&#xA;			at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)&#xA;			at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)&#xA;			at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)&#xA;			at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)&#xA;        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)&#xA;			at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)&#xA;			at org.apache.spark.rdd.RDD.collect(RDD.scala:926)&#xA;			at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:405)&#xA;        at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)&#xA;        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&#xA;        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)&#xA;        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#xA;        at java.lang.reflect.Method.invoke(Method.java:498)&#xA;        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)&#xA;        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)&#xA;        at py4j.Gateway.invoke(Gateway.java:259)&#xA;        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)&#xA;        at py4j.commands.CallCommand.execute(CallCommand.java:79)&#xA;        at py4j.GatewayConnection.run(GatewayConnection.java:209)&#xA;        at java.lang.Thread.run(Thread.java:745)&#xA;    Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):&#xA;      File &quot;/usr/lib/spark/python/pyspark/worker.py&quot;, line 98, in main&#xA;        command = pickleSer._read_with_length(infile)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 164, in _read_with_length&#xA;        return self.loads(obj)&#xA;      File &quot;/usr/lib/spark/python/pyspark/serializers.py&quot;, line 422, in loads&#xA;        return pickle.loads(obj)&#xA;    ImportError: No module named nltk.tokenize&#xA;&#xA;        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)&#xA;			at org.apache.spark.api.python.PythonRunner$$anon$1.&amp;lt;init&amp;gt;(PythonRDD.scala:207)&#xA;			at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)&#xA;			at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)&#xA;			at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)&#xA;			at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)&#xA;			at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)&#xA;			at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;			at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)&#xA;        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)&#xA;        ... 1 more&#xA;&#xA;    &amp;gt;&amp;gt;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-05-18T06:05:18.657" Title="Unable to load NLTK in spark using PySpark" Tags="|apache-spark|nltk|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11817" PostTypeId="2" ParentId="11816" CreationDate="2016-05-18T06:05:18.657" Score="2" Body="&lt;p&gt;It looks like you installed it only on the driver/gateway and not on the nodes/workers itself. The test you ran in the shell is running it locally, once you map a function via your SparkContext it gets distributed to the workers which don't have NLTK installed.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-18T06:05:18.657" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11818" PostTypeId="2" ParentId="11730" CreationDate="2016-05-18T06:40:01.043" Score="0" Body="&lt;p&gt;Thanks guys but I found a program called Mozenda that even idiots like me understand :) you basically click on the searchbar, import an excellist of stuff you want to search and then just click on the datafield you want to extract.&lt;/p&gt;&#xA;" OwnerUserId="17359" LastActivityDate="2016-05-18T06:40:01.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11819" PostTypeId="2" ParentId="11798" CreationDate="2016-05-18T07:26:29.193" Score="1" Body="&lt;p&gt;Your error calculation looks wrong:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if output1 &amp;lt; 0.99:&#xA;    error += (output1)**2&#xA;if output2 &amp;gt; 0.01:&#xA;    error += (1-output2)**2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I am reading this correctly, output1 = 1.0 is a correct classification, since you treat any value above 0.99 as no error at all. However, you then measure the squared error as being &lt;code&gt;output1 ** 2&lt;/code&gt; which means the largest error is at 0.99. This is going to confuse the network, it will treat classifications above the line as perfect 1.0 if they score &gt; 0.99, or otherwise want to adjust weights to make the classification 0.0. The opposite is true for your scoring of points under the line. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to either&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Swap the conditionals (change &lt;code&gt;&amp;lt; 0.99&lt;/code&gt; to &lt;code&gt;&amp;gt; 0.01&lt;/code&gt; and vice-versa)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Replace &lt;code&gt;(output1)&lt;/code&gt; with &lt;code&gt;(1-output1)&lt;/code&gt; and &lt;code&gt;(1-output2)&lt;/code&gt; with &lt;code&gt;(output2)&lt;/code&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;in order to make the logic self-consistent. Which you choose depends on whether above the line points are positive class, or it is the negative ones. If above the line is positive you should use option 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, as comments have pointed out, you might be a little ambitious with your approach. I spot the following things that make your job harder:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Using genetic algorithm search. Whilst it is possible to train small NNs like yours using a GA, it is less efficient. GAs might be a good choice for control systems or for a-life scenarios, but are far behind cutting edge for supervised learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using mean squared error metric for classification. You should use &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy&quot; rel=&quot;nofollow&quot;&gt;logloss&lt;/a&gt;, which more heavily penalises bad guesses. With a mean squared error, there is a higher chance the network will settle for a few bad misclassifications if it also gets some other reasonable ones.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ambitious function to separate. A neural network should be able to classify your function, but that's a lot of curvature to learn for a starting problem. Reduce it to just a couple of cycles to start with should help you debug practical issues.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;No input normalisation. This may lead to saturated values in the network - close to 1.0 or 0.0, making it very hard to discriminate between good and bad weights. You don't show your weight initialisations - large initial weights can also cause this effect (although using a GA may actually help here, depending on how you are mutating weights).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-18T13:51:44.590" LastActivityDate="2016-05-18T13:51:44.590" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11820" PostTypeId="1" AcceptedAnswerId="11821" CreationDate="2016-05-18T07:52:49.857" Score="4" ViewCount="5089" Body="&lt;p&gt;yes, DBSCAN parameters, and in particular the parameter &lt;strong&gt;eps&lt;/strong&gt; (size of the epsilon neighborhood).&#xA;In the documentation we have a &quot;&lt;em&gt;Look for the knee in the plot&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fine, but it requires a visual analysis. And it doesn't really work if we want to make things automatic.&#xA;So, I was wondering if it was possible to find a good eps in a few lines of code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's imagine something like :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;evaluate kNN distance  &lt;/li&gt;&#xA;&lt;li&gt;sort these values&lt;/li&gt;&#xA;&lt;li&gt;scale them (so that&#xA;the values are always between 0 and 1)&lt;/li&gt;&#xA;&lt;li&gt;evaluate the derivative &lt;/li&gt;&#xA;&lt;li&gt;find the first point where derivative is higher than a certain&#xA;value, let's try with 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In R, it would look like (using iris dataset as in the DBSCAN documentation) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# evaluate kNN distance&#xA;dist &amp;lt;- dbscan::kNNdist(iris, 4)&#xA;&#xA;# order result&#xA;dist &amp;lt;- dist[order(dist)]&#xA;&#xA;# scale&#xA;dist &amp;lt;- dist / max(dist)&#xA;&#xA;# derivative&#xA;ddist &amp;lt;- diff(dist) / ( 1 / length(dist))&#xA;&#xA;# get first point where derivative is higher than 1&#xA;knee &amp;lt;- dist[length(ddist)- length(ddist[ddist &amp;gt; 1])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the result is &lt;strong&gt;0.536&lt;/strong&gt; which looks quite good.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/FJ0FQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FJ0FQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this approach relevant or totally nonsense ?&lt;/p&gt;&#xA;" OwnerUserId="18221" LastActivityDate="2019-07-30T06:20:03.590" Title="How can we evaluate DBSCAN parameters?" Tags="|r|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11821" PostTypeId="2" ParentId="11820" CreationDate="2016-05-18T08:20:48.650" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://www.dbs.informatik.uni-muenchen.de/Publikationen/Papers/OPTICS.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;OPTICS&lt;/a&gt; gets rid of $\varepsilon$, you might want to have a look at it. Especially the &lt;strong&gt;reachability plot&lt;/strong&gt; is a way to visualize what good choices of $\varepsilon$ in DBSCAN might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wikipedia (&lt;a href=&quot;https://en.wikipedia.org/wiki/OPTICS_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;) illustrates it pretty well. The image on the top left shows the data points, the image on the bottom left is the reachability plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TamFM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TamFM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $y$-axis are different values for $\varepsilon$, the valleys are the clusters. Each &quot;bar&quot; is for a single point, where the height of the bar is the minimal distance to the already printed points.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-05-18T11:27:10.647" LastActivityDate="2016-05-18T11:27:10.647" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11822" PostTypeId="2" ParentId="11809" CreationDate="2016-05-18T09:00:12.697" Score="0" Body="&lt;p&gt;I would look into the Python package BeautifulSoup. It parses HTML documents into a tree structure and allows for all kind of filtering and manipulation of the tree. I've taken a look at the Gear Patrol website and I would advice these steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scrape the front page looking for articles. By taking a look using a web inspector like the one in Chrome (Ctrl+Shift+I) you can look at the HTML structure. Looking there we can see that the links to the articles are in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;body &amp;gt; div:content &amp;gt; div:home-this-week &amp;gt; div:posts hfeed &amp;gt; article &amp;gt; div:body-wrap&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you look for all article tags and create a list of the corresponding URLs. You can use the link of the pagination to retrieve more article links or use their URL structure to make an educated guess where more articles are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we have a list of URLs, iterate over them and parse these pages in BeautifulSoup to retrieve the necessary information in a similar fashion. I advice taking breaks of a few seconds in between requests for new pages to not get banned from their server and not overly disrupt it. There are a lot of tutorials available how to do webscraping using BeautifulSoup and this site looks like an easy victim.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-18T09:00:12.697" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11823" PostTypeId="2" ParentId="11820" CreationDate="2016-05-18T09:39:33.997" Score="2" Body="&lt;p&gt;If you go for an automatic solution, you may as well just decide that 80% of your points should be core points, and &lt;strong&gt;choose the 80% quantile of the 4-NN distances&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the derivative etc. to formally define a &quot;knee&quot; is fragile, this will not work in my opinion. Because you cannot compare distances to ranks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Several heuristics for DBSCAN parameterization have been proposed over the last 20 years.&lt;/li&gt;&#xA;&lt;li&gt;Several enhancements of DBSCAN such as OPTICS and HDBSCAN* have been published, that get rid of the epsilon parameter (in favor of a graphical approach, e.g. OPTICS plots).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the end, &lt;strong&gt;having parameters is a feature, not a limitation&lt;/strong&gt;. Cluster analysis is not something to fully automate. It is an explorative method. You try it, change parameters, try it again, change parameters again, ... until you have &lt;em&gt;learned&lt;/em&gt; something about your data. Any method that does not allow you to reiterate is badly designed.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2019-07-30T06:20:03.590" LastActivityDate="2019-07-30T06:20:03.590" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11824" PostTypeId="2" ParentId="9588" CreationDate="2016-05-18T09:52:39.343" Score="3" Body="&lt;p&gt;&lt;code&gt;'RDD' object has no attribute 'select'&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that &lt;code&gt;test&lt;/code&gt; is in fact an RDD and not a dataframe (which you are assuming it to be). Either you convert it to a dataframe and then apply &lt;code&gt;select&lt;/code&gt; or do a &lt;code&gt;map&lt;/code&gt; operation over the RDD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if you need any help around this.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-05-18T09:52:39.343" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11825" PostTypeId="1" AcceptedAnswerId="28102" CreationDate="2016-05-18T10:54:15.507" Score="2" ViewCount="539" Body="&lt;p&gt;I want to write my thesis about Fraud Detection in ERP Databases. I'm looking for a Industry Standard Processs such as CRISP-DM for Data Mining Projects, in order to justify my approach in solving the issue of finding outliers/anomalies in the data set.&#xA;Is there any established Standard Process which would be more suitable for such an project?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="18744" LastActivityDate="2018-02-21T12:45:44.873" Title="Industry Standard Process for Fraud/Outlier/Anomaly Detection" Tags="|anomaly-detection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11826" PostTypeId="2" ParentId="9588" CreationDate="2016-05-18T11:11:41.173" Score="3" Body="&lt;p&gt;Assuming you have an RDD each row of which is of the form &lt;code&gt;(passenger_ID, passenger_name)&lt;/code&gt;, you can do &lt;code&gt;rdd.map(lambda x: x[0])&lt;/code&gt;. This is for a basic RDD&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use Spark sqlcontext there are functions to select by column name.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-05-18T11:11:41.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11828" PostTypeId="2" ParentId="11825" CreationDate="2016-05-18T12:34:05.083" Score="3" Body="&lt;p&gt;You should refer &lt;a href=&quot;http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf&quot; rel=&quot;nofollow&quot;&gt;this survey paper&lt;/a&gt; on Anomaly Detection (from University of Minnesota). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if this helps you.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-05-18T12:34:05.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11831" PostTypeId="2" ParentId="8043" CreationDate="2016-05-18T22:17:17.183" Score="-1" Body="&lt;p&gt;Here is a &lt;a href=&quot;https://github.com/paulknysh/blackbox&quot; rel=&quot;nofollow&quot;&gt;solution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A mathematical method is described in this &lt;a href=&quot;http://arxiv.org/pdf/1605.00998.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18840" LastActivityDate="2016-05-18T22:17:17.183" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11832" PostTypeId="1" CreationDate="2016-05-18T23:18:15.860" Score="2" ViewCount="139" Body="&lt;p&gt;How would someone go about using deep learning to classify sign language gestures? For example, suppose I had video files of many different gestures. For any given gesture, I might have many videos of it, and each video would be comprised of many frames. When trying to classify MNIST digits in images the dimensions of the input are comparatively simple: height, width, and RGB channels. How would gestures (multiple frames over time) be accounted for? Would time be a fourth dimension? What should the neural net's architecture look like so as not to overfit? Should I use something instead of a convoluted neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there are probably clever ways to hand-code predictors for sign language, but I'm more interested in how to architect the neural net and take advantage of the time component of the data (where there is value in the transition of video frames over time). Classifying gestures is a simplification of the actual problem I'm trying to solve, so I'm looking for an approach that would be generalizable to other types of problems where it might be necessary to look at many frames of the video to predict the target variable.   &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29169" LastEditDate="2021-02-16T15:49:04.663" LastActivityDate="2021-02-16T15:49:04.663" Title="Applying ConvNets to classify motion/video data" Tags="|deep-learning|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11833" PostTypeId="2" ParentId="11720" CreationDate="2016-05-19T00:00:43.277" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;It doesn't handle categorical variables very well&lt;/li&gt;&#xA;&lt;li&gt;It doesn't handle 'soft' boundaries - i.e. areas where some cases appear on either side of a boundary.&#xA;See also mathbabe here: &lt;a href=&quot;https://mathbabe.org/?s=nearest+neighbor&quot; rel=&quot;nofollow&quot;&gt;https://mathbabe.org/?s=nearest+neighbor&lt;/a&gt; - for extended criticism.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-05-19T00:00:43.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11834" PostTypeId="2" ParentId="11832" CreationDate="2016-05-19T00:02:55.207" Score="2" Body="&lt;p&gt;Signs are the visual equivalent of &quot;words&quot;, and just as words can be decomposed into smaller parts, so can signs be broken down into smaller meaningful/useful units. For example, in &lt;a href=&quot;https://en.wikipedia.org/wiki/American_Sign_Language&quot; rel=&quot;nofollow&quot;&gt;American Sign Language&lt;/a&gt; (ASL), signs are decomposed into &lt;em&gt;parameters&lt;/em&gt; such as &lt;strong&gt;handshape&lt;/strong&gt;, &lt;strong&gt;orientation&lt;/strong&gt;, &lt;strong&gt;movement&lt;/strong&gt;, &lt;strong&gt;location&lt;/strong&gt; on the body. Each parameter can take on a finite set of values, for example &lt;a href=&quot;http://www.lifeprint.com/asl101/pages-layout/handshapes.htm&quot; rel=&quot;nofollow&quot;&gt;handshapes&lt;/a&gt; can be closed fist, open hand, cocked index, etc. I would expect a neural network would need to learn these parameters and their values, while learning to ignore distractor and nonsense configurations. Perhaps initial training includes learning to differentiate between signs and non-signs. And just as there are different spoken language, there are different sign languages as well. Moreover, there exist distinct sign language dialects like &lt;a href=&quot;https://en.wikipedia.org/wiki/Black_American_Sign_Language&quot; rel=&quot;nofollow&quot;&gt;Black ASL&lt;/a&gt; that will need to be taken into account by your system.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-05-19T00:09:47.583" LastActivityDate="2016-05-19T00:09:47.583" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11835" PostTypeId="2" ParentId="11753" CreationDate="2016-05-19T00:22:39.260" Score="1" Body="&lt;p&gt;Spark support for Mahout came from &lt;a href=&quot;http://mahout.apache.org/release-notes/Apache-Mahout-0.10.0-Release-Notes.pdf&quot; rel=&quot;nofollow&quot;&gt;Mahout 0.10 release&lt;/a&gt; while you are using 0.9 release. So this should explain why you get the &lt;code&gt;unknown program&lt;/code&gt; error. I would suggest using a higher version of Mahout.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-05-19T00:22:39.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11836" PostTypeId="1" CreationDate="2016-05-19T00:56:13.510" Score="4" ViewCount="1791" Body="&lt;p&gt;I'm tackling a binary classification task using SVM implemented in python scikit-learn. Datasize is around 10,000 and the number of feature is 34.&lt;/p&gt;&#xA;&lt;p&gt;After finding nice parameter set (using &lt;code&gt;RandomizedSearchCV&lt;/code&gt; class), I evaluate the model by the cross validation. The result seems nice.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;criteria_list = [&amp;quot;precision&amp;quot;, &amp;quot;recall&amp;quot;, &amp;quot;f1&amp;quot;, &amp;quot;roc_auc&amp;quot;]&#xA;score_df = []&#xA;score_df2 = []&#xA;clf = svm.SVC(**random_search_clf.best_estimator_.get_params())&#xA;for crit in criteria_list:&#xA;    scores = cross_validation.cross_val_score(clf, X, y, cv=3, scoring=crit)&#xA;    score_df.append([&amp;quot;{} (±{})&amp;quot;.format(np.round(np.mean(scores),3), np.round(np.std(scores),4)), scores])&#xA;    score_df2.append([&amp;quot;{} (±{})&amp;quot;.format(np.round(np.mean(scores),3), np.round(np.std(scores),4))])&#xA;&#xA;pd.DataFrame(np.transpose(score_df2), columns=criteria_list, index=[&amp;quot;SVM&amp;quot;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AEKWY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AEKWY.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;My question is whether it is possible to &lt;strong&gt;find out which feature is effective to classify the test data&lt;/strong&gt;. I thought it's relating to &lt;em&gt;sensitivity analysis&lt;/em&gt;, but good answer cannot be shown by googling &amp;quot;sensitivity analysis + svm&amp;quot; or &amp;quot;sensitivity analysis + scikit learn&amp;quot;.&lt;/p&gt;&#xA;" OwnerUserId="11062" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-03-21T21:11:15.993" Title="Find effective feature on machine learning classification task with scikit-learn" Tags="|machine-learning|python|classification|scikit-learn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11838" PostTypeId="1" CreationDate="2016-05-19T07:29:00.693" Score="2" ViewCount="1173" Body="&lt;p&gt;&lt;a href=&quot;http://homepage.tudelft.nl/a9p19/papers/icpr_04_occ.pdf&quot; rel=&quot;nofollow&quot;&gt;Consistency based model selection&lt;/a&gt; does not perform well for many datasets for &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow&quot;&gt;One-Class Classification&lt;/a&gt; (OCC). So I am looking for some other model selection criteria. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, only one class of data (i.e. the positive class) is available, classification error is no longer a valid criterion. It is very challenging task to decide on the optimal parameters in OCC. Here, &lt;em&gt;model selection&lt;/em&gt; means the optimal parameter selection for the one-class classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest any additional methods &lt;em&gt;except&lt;/em&gt; consistency based model selection method.&lt;/p&gt;&#xA;" OwnerUserId="18818" LastEditorUserId="6478" LastEditDate="2016-05-21T13:52:06.430" LastActivityDate="2016-05-21T13:52:06.430" Title="How to perform model selection for One-Class Classification?" Tags="|machine-learning|classification|neural-network|predictive-modeling|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11839" PostTypeId="2" ParentId="8043" CreationDate="2016-05-19T07:52:27.720" Score="2" Body="&lt;p&gt;Since no one has mentioned &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian optimization&lt;/a&gt; yet: &quot;Bayesian optimization is a sequential design strategy for global optimization of black-box functions.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the bottom of that wikipedia page are links to several bayesian optimization libraries like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/HIPS/Spearmint&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spearmint&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/yelp/MOE&quot; rel=&quot;nofollow noreferrer&quot;&gt;MOE - Metric Optimization Engine&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://sigopt.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SigOpt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rmcantin.bitbucket.org/html/&quot; rel=&quot;nofollow noreferrer&quot;&gt;BayesOpt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://sheffieldml.github.io/GPyOpt/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GPyOpt&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/fmfn/BayesianOptimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;fmfn's bayesian optimization library (short &amp;amp; simple)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Besides libraries using bayesian optimization there are also libraries using TPE (trees of parzen estimators), like &lt;a href=&quot;http://jaberg.github.io/hyperopt/&quot; rel=&quot;nofollow noreferrer&quot;&gt;hyperopt&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an image of fmfn's bayesian optimizer optimizing a non-linear 2d function. As you can see, a point close to the global optimum is found after relatively few iterations and the system tests more points in more promising (red) areas than in less promising (blue) areas - making it more efficient than random search:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bqpKP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bqpKP.png&quot; alt=&quot;bayesian optimization of a 2d function&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.imgur.com/RbdUoU7.gifv&quot; rel=&quot;nofollow noreferrer&quot;&gt;(full animation)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-05-19T07:52:27.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11840" PostTypeId="1" AcceptedAnswerId="11841" CreationDate="2016-05-19T07:54:30.420" Score="7" ViewCount="17056" Body="&lt;p&gt;Keras has two border_mode for convolution2D, same and valid.&#xA;Could anyone explain what &quot;same&quot; does or point out some documentation? I could not find any document on the net (except people asking that it be implemented in theano as well).&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-16T15:48:54.227" LastActivityDate="2021-02-16T15:48:54.227" Title="border_mode for convolutional layers in keras" Tags="|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11841" PostTypeId="2" ParentId="11840" CreationDate="2016-05-19T08:34:53.090" Score="12" Body="&lt;p&gt;With border mode &quot;valid&quot; you get an output that is smaller than the input because the convolution is only computed where the input and the filter fully overlap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With border mode &quot;same&quot; you get an output that is the &quot;same&quot; size as the input. That means that the filter has to go outside the bounds of the input by &quot;filter size / 2&quot; - the area outside of the input is normally padded with zeros.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that some libraries also support the border mode &quot;full&quot; where the filter goes even further outside the bounds of the input - up to &quot;filter size - 1&quot;. This results in an output shape larger than the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a short explanation in.. numpy's convolve documentation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.convolve.html&quot;&gt;http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.convolve.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-05-19T08:34:53.090" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11842" PostTypeId="2" ParentId="11836" CreationDate="2016-05-19T11:53:56.347" Score="0" Body="&lt;p&gt;The effectiveness of features in your data depends on the &quot;information gain&quot; from that feature. The more the information gain, the better is the feature for your classification. I'm not exactly sure whether SVMs support such a technique to evaluate features, but you can look for the &lt;strong&gt;Decision Tree&lt;/strong&gt; classification method. It calculates the entropy of the features, which then helps in calculating the information gain. From those calculations, you can easily &lt;strong&gt;find out which feature is effective to classify the test data.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="18150" LastActivityDate="2016-05-19T11:53:56.347" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11843" PostTypeId="1" CreationDate="2016-05-19T13:45:22.880" Score="0" ViewCount="991" Body="&lt;p&gt;I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch size I can use for mini batch gradient descent? Would this difference be significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for answers.&lt;/p&gt;&#xA;" OwnerUserId="13527" LastActivityDate="2017-02-06T10:14:04.257" Title="GPU computing: how much VRAM do I need for mini batch gradient descent?" Tags="|parallel|gpu|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11844" PostTypeId="1" CreationDate="2016-05-20T03:37:45.700" Score="1" ViewCount="51" Body="&lt;p&gt;Suppose in a classification we have a dataset with many features and their class, we want to select some features using which we can construct a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We perform the cluster evaluation for the given dataset taking different features (or derived features) and find cluster purity and entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can these purity and entropy calculated for certain feature vector be used to predict the classification performance of the classifier constructed for those set of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For eg.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have set of features vector {A,B,C,D,E,F} and n no dataset. on selection of features there are many possible feature vectors I am considering two: X = {A,B,C,D} and Y = {A,E,F}. Now I perform cluster evaluation and find cluster purity for both X and Y. Can these purity(X) and purity(Y) predict which one is a better selection of feature for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this purity may be a lower bound for the performance of classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: The clustering used is assumed to be k-means clustering with k = no of classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;originally posted &lt;a href=&quot;https://stats.stackexchange.com/questions/213423/can-cluster-analysis-of-preclassified-items-gives-idea-about-the-classification&quot;&gt;here&lt;/a&gt;, but could not find the answer, so posting here too.&lt;/p&gt;&#xA;" OwnerUserId="14789" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-05-20T04:17:14.073" Title="Can cluster analysis of preclassified items gives idea about the classification performance?" Tags="|classification|clustering|feature-selection|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11847" PostTypeId="1" AcceptedAnswerId="11864" CreationDate="2016-05-20T10:00:31.263" Score="-1" ViewCount="523" Body="&lt;p&gt;For in calculating success in information retrieval, &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow&quot;&gt;precision and recall&lt;/a&gt; are fairly standard measurements, relating to accuracy of the results, and to what extent the results are comprehensive, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, recall values typically require that you know how many correct results there are in total (in order to be able to state to what extent these results have been returned). This is, of course, problematic, if you do not know how many correct results there are in the first place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But supposing you have &lt;strong&gt;x&lt;/strong&gt; number of results. You do not know how they relate to the absolute number of correct documents that could have been returned, but you do know how accurate these results are. You can increase this accuracy, but you will lose some correct results in the process. You could exclude every result from &lt;strong&gt;x&lt;/strong&gt; bar one, and ensure you have 100% accuracy, but clearly there has been a dramatic decrease in &lt;em&gt;relative recall&lt;/em&gt; through this action. But what is the technically correct term to be associated with this &quot;relative recall&quot;, when you do not have any idea what the absolute value of the recall in this environment is? Is there such a term?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EXAMPLE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance supposing you were developing an IR system to return articles relating to the Call of Duty video game franchise and it gives you the following results&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.theverge.com/2016/5/2/11564464/call-of-duty-infinite-warfare-release-date-first-trailer&quot; rel=&quot;nofollow&quot;&gt;http://www.theverge.com/2016/5/2/11564464/call-of-duty-infinite-warfare-release-date-first-trailer&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.theguardian.com/technology/2016/may/03/call-of-duty-infinite-warfare&quot; rel=&quot;nofollow&quot;&gt;https://www.theguardian.com/technology/2016/may/03/call-of-duty-infinite-warfare&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.asianage.com/editorial/beyond-call-duty-943&quot; rel=&quot;nofollow&quot;&gt;http://www.asianage.com/editorial/beyond-call-duty-943&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://europe.newsweek.com/chinas-military-attempts-attract-millennials-call-duty-inspired-recruitment-455425?rm=eu&quot; rel=&quot;nofollow&quot;&gt;http://europe.newsweek.com/chinas-military-attempts-attract-millennials-call-duty-inspired-recruitment-455425?rm=eu&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.breitbart.com/video/2016/05/16/call-duty-black-ops-iii-eclipse-multiplayer-trailer/&quot; rel=&quot;nofollow&quot;&gt;http://www.breitbart.com/video/2016/05/16/call-duty-black-ops-iii-eclipse-multiplayer-trailer/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.journalgazette.net/news/local/courts/County-residents-getting-bogus-jury-duty-calls-13146090&quot; rel=&quot;nofollow&quot;&gt;http://www.journalgazette.net/news/local/courts/County-residents-getting-bogus-jury-duty-calls-13146090&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.straitstimes.com/singapore/health/healthcare-workers-who-went-beyond-call-of-duty&quot; rel=&quot;nofollow&quot;&gt;http://www.straitstimes.com/singapore/health/healthcare-workers-who-went-beyond-call-of-duty&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now you don't know what the absolute number of correct articles are for the call of duty franchise: but you do know the number in the dataset your IR system has produced. Above represents a precision of 42.85% and a relative recall of 100%. A cut list of &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.theverge.com/2016/5/2/11564464/call-of-duty-infinite-warfare-release-date-first-trailer&quot; rel=&quot;nofollow&quot;&gt;http://www.theverge.com/2016/5/2/11564464/call-of-duty-infinite-warfare-release-date-first-trailer&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.theguardian.com/technology/2016/may/03/call-of-duty-infinite-warfare&quot; rel=&quot;nofollow&quot;&gt;https://www.theguardian.com/technology/2016/may/03/call-of-duty-infinite-warfare&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.asianage.com/editorial/beyond-call-duty-943&quot; rel=&quot;nofollow&quot;&gt;http://www.asianage.com/editorial/beyond-call-duty-943&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Has precision of 66.66%, but at the expense of one of the correct articles, thereby affecting &lt;em&gt;&quot;recall&quot;&lt;/em&gt;. So what is the correct term for this so-called recall?&lt;/p&gt;&#xA;" OwnerUserId="5290" LastEditorUserId="5290" LastEditDate="2016-05-20T13:06:52.127" LastActivityDate="2017-04-16T18:57:00.130" Title="Term for relative recall" Tags="|information-retrieval|terminology|confusion-matrix|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11848" PostTypeId="2" ParentId="11665" CreationDate="2016-05-20T10:16:34.880" Score="1" Body="&lt;p&gt;I'm not sure why your data set looks like this (having different steps of future result as a single label) so provide some info about the context of your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you can use any Neural Network, Fuzzy Network and so on. you just need to consider your labels as continuous variables and then in final step round you network outputs. You probably don't need any recurrent or dynamic network, because your data already has lots of delays in it which is possibly able to represent dynamic behavior of your system behind data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use Multi-Step ahead prediction instead. for example using your Instant1_feature predict or estimate [s21_var1 s21_var2]. then use a feature vector which looks like Instant2_feature but with your estimations of [s21_var1 s21_var2] replaced with real values of [s21_var1 s21_var2].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="5200" LastActivityDate="2016-05-20T10:16:34.880" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11851" PostTypeId="1" AcceptedAnswerId="11856" CreationDate="2016-05-20T16:53:38.323" Score="2" ViewCount="265" Body="&lt;p&gt;I am working on CNN, and I have some doubts. Let's assume I only want one feature map, just to make things easier. And let's suppose my image is grayscale, to make things even easier. So, let's say my image is (32,32) --grayscale, hence just a channel and we don't need to write it explicitly, and my filter is (3,3) --again, one feature map, so I won't bother writing 1.&#xA;I understand this will map to a (30,30) layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many parameters will I have? If I understand it correctly, I will have 9 weights and one bias, so a total of 10, because we map each (3,3) subregion using the same weights. Back-propagation will determine the best values for those weights and that will give me one feature map, or a filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, so good. What I don't understand is how does the training work? I need to keep the same weights and bias when moving across the image (that's why I only have 10 parameters), but won't those change when I do back-propagation? How can I apply back-propagation and keep the same values for the weights regardless of the subregion they are applied to?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-16T15:48:44.783" LastActivityDate="2021-02-16T15:48:44.783" Title="Trying to figure out how to set weights for convolutional networks" Tags="|deep-learning|convolutional-neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11852" PostTypeId="1" CreationDate="2016-05-20T16:54:42.143" Score="4" ViewCount="1751" Body="&lt;p&gt;Hey I am a little new to the whole &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;RBM&lt;/a&gt; entropy/energy training thing, having some trouble understanding and trying to figure whether it is worth the effort needed to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can't RBMs quite easily be trained with a standard &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_descent&quot; rel=&quot;nofollow&quot;&gt;gradient descent&lt;/a&gt; algorithm? Is the current way just much faster? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some sample Theano code of what I mean:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def rbm(inputs, shape, cost=0, last_cost=1):&#xA;    X = T.vector()                     # input&#xA;    W = theano.shared(np.zeros(shape)) # weights&#xA;&#xA;    b = theano.shared(np.zeros( shape[1] )) # bias to hidden&#xA;    B = theano.shared(np.zeros( shape[0] )) # bias to reconstruction&#xA;&#xA;    H = sigmoid(T.dot(X, W) + b) # hidden activation&#xA;    Y = T.dot(H, W.T) + B        # reconstruction&#xA;&#xA;    L = ((X - Y) ** 2).mean()                    # loss function&#xA;    G = T.grad(L, W), T.grad(L, b), T.grad(L, B) # gradients&#xA;&#xA;    a = .1                                                  # learning rate&#xA;    U = [(W, W - a*G[0]), (b, b - a*G[1]), (B, B - a*G[2])] # shared updates&#xA;&#xA;    train = theano.function([X], L, updates=U) # training function&#xA;&#xA;    while abs(last_cost-cost) &amp;gt; 0.01:&#xA;        last_cost, cost = cost, 0&#xA;        for input in inputs: cost += train(input)&#xA;&#xA;    return W.get_value(), b.get_value()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And yes this code is very crude.&lt;/p&gt;&#xA;" OwnerUserId="18886" LastEditorUserId="6478" LastEditDate="2016-05-22T16:20:10.867" LastActivityDate="2017-09-17T19:24:44.730" Title="Training Restricted Boltzmann Machines (RBMs) using gradient descent" Tags="|gradient-descent|theano|rbm|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11853" PostTypeId="1" AcceptedAnswerId="11870" CreationDate="2016-05-20T17:29:10.243" Score="18" ViewCount="52071" Body="&lt;p&gt;I am trying to figure out how many weights and biases are needed for CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a (3, 32, 32)-image and want to apply a (32, 5, 5)-filter.&#xA;For each feature map I have 5x5 weights, so I should have 3 x (5x5) x 32 parameters. Now I need to add the bias. I believe I only have (3 x (5x5) + 1) x 32 parameters, so is the bias the same across all colors (RGB)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? Do I keep the same bias for each image across its depth (in this case 3) while I use different weights? Why is that?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-16T15:48:29.797" LastActivityDate="2021-02-16T15:48:29.797" Title="Question about bias in Convolutional Networks" Tags="|deep-learning|convolutional-neural-network|backpropagation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11854" PostTypeId="1" CreationDate="2016-05-20T18:51:44.993" Score="2" ViewCount="656" Body="&lt;p&gt;Background: The basic set-up for &lt;strong&gt;non-negative matrix factorization (nmf)&lt;/strong&gt;, is that we take a matrix with non-negative elements, X and find two other non-negative matrices H and T such that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$||X - HT||$ is a minimum&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$X$ in this example is a matrix that represents a data set. Rows correspond to samples of data, and columns correspond to features. I understand that &lt;strong&gt;nmf&lt;/strong&gt; is used for feature reduction. My understanding is that you can use the latent features to train a model on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;strong&gt;suppose that I use nmf for feature reduction and then get a new row of data and want to find the reduced features for the new data. Do I have to run nmf again with an updated dataset and then retrain my model each time?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="12443" LastEditorUserId="10902" LastEditDate="2016-10-18T15:44:13.650" LastActivityDate="2016-10-18T15:44:13.650" Title="How do I get latent features for a new row of data when doing non-negative matrix factorization?" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11855" PostTypeId="1" CreationDate="2016-05-20T20:20:02.007" Score="2" ViewCount="1689" Body="&lt;p&gt;I'm trying to add Gower distance for my KNN implementation but I can't find out the formula for Gower distance. What is the formula for Gower distance?&lt;/p&gt;&#xA;" OwnerUserId="18888" LastActivityDate="2016-12-17T00:24:23.253" Title="Gower Distance Formula for KNN" Tags="|machine-learning|similarity|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11856" PostTypeId="2" ParentId="11851" CreationDate="2016-05-20T20:23:23.303" Score="2" Body="&lt;p&gt;You are right there are just 10 params in your example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For determining gradients, you just add up all the deltas from backpropagation in each location - i.e. you run backpropagation 30x30 = 900 times, for each position the 3x3 kernel is used, for every example in your batch (or just for one example if you are running most simple onine stochastic gradient descent), and for each position you add those delta values into a suitably-sized buffer (10 values for weight deltas, or 9 values for previous layer activation deltas). You will end up with one set of summed deltas matching your single 3x3 filter (plus a delta bias term). You then apply the summed version to update the weights of your single filter + bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note this is a general rule you can apply whenever multiple gradient sources from backpropagation can be applied to any parameter - they just add. This occurs in RNNs too, or in any structure where you can set an objective function for non-output neurons.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-20T22:09:45.593" LastActivityDate="2016-05-20T22:09:45.593" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="11857" PostTypeId="2" ParentId="11853" CreationDate="2016-05-20T21:04:52.937" Score="1" Body="&lt;p&gt;It is the property of CNNs that they use shared weights and biases(same weights and bias for all the hidden neurons in a layer) in order to detect the same feature. This leads to a more deep learning as compared to simple neural networks. You can read this out as a reference :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;nofollow&quot;&gt;http://deeplearning.net/tutorial/lenet.html&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap6.html#introducing_convolutional_networks&quot; rel=&quot;nofollow&quot;&gt;http://neuralnetworksanddeeplearning.com/chap6.html#introducing_convolutional_networks&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-05-20T21:04:52.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11858" PostTypeId="2" ParentId="11855" CreationDate="2016-05-20T21:06:57.300" Score="3" Body="&lt;p&gt;The formula is defined in section two of Gower's &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.412.4155&quot; rel=&quot;nofollow&quot;&gt;A general coefficient of similarity and some of its properties&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Abstract&lt;/strong&gt;&#xA;  A general coefficient measuring the similarity between two sampling units is defined. The matrix of similarities between all pairs of sample units is shown to be positive semidefinite (except possibly when there are missing values). This is important for the multidimensional Euclidean representation of the sample and also establishes some inequalities amongst the similarities relating three individuals. The definition is extended to cope with a hierarchy of characters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-05-20T21:38:05.407" LastActivityDate="2016-05-20T21:38:05.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11859" PostTypeId="2" ParentId="9228" CreationDate="2016-05-20T22:45:09.837" Score="10" Body="&lt;p&gt;Classifiers like &lt;em&gt;Decision Tree, Bayesian, Back-propagation, Support Vector Machine&lt;/em&gt; come under the category of &lt;strong&gt;&quot;Eager Learners&quot;&lt;/strong&gt;, because they first build a classification model on the &lt;em&gt;training dataset&lt;/em&gt; before being able to actually classify an [unseen] observation from &lt;em&gt;test dataset&lt;/em&gt;. The learned model is now &quot;eager&quot; (read hungry) to classify previously unseen observations, hence the name.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The KNN-based classifier, however, does not build any classification model. It directly learns from the training instances (observations). It starts processing data only after it is given a test observation to classify. Thus, KNN comes under the category of &lt;strong&gt;&quot;Lazy Learner&quot;&lt;/strong&gt; approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the above foundational differences, we can conclude the following:-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Since KNN performs on-the-spot learning, it requires frequent database lookups, hence, can be computationally expensive. Decision Tree Classifier does not require such lookups as it has in-memory classification model ready.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since KNN performs instance-based learning, a well-tuned K can model complex decision spaces having arbitrarily complicated decision boundaries, which are not easily modeled by other &quot;eager&quot; learners like Decision Trees.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Eager&quot; learners work in batches, modeling one group of training observations at a time. So they are not fit for incremental learning. But KNN naturally supports incremental learning (data streams) since it is an instance-based learner.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Further, KNN classifier gives test error rates closer to that of Bayesian classier (the gold standard). As quoted in &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;noreferrer&quot;&gt;ISLR&lt;/a&gt;:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Bayes error rate is analogous to the irreducible error&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="17187" LastActivityDate="2016-05-20T22:45:09.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11860" PostTypeId="2" ParentId="11854" CreationDate="2016-05-20T23:52:28.583" Score="2" Body="&lt;p&gt;Given a dataset represented by a data matrix $X$, we can solve $||X-HT||$ using gradient descent, setting coefficients equal to zero if they become negative. This gives us an $H$. If we get a new row of data $x_{new}$, then we can find the corresponding reduced features $w_{new}$(weights for the latent features) by solving the equation: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$argmin_{w_{new}} = ||x_{new}-w_{new}T||$.&lt;/p&gt;&#xA;" OwnerUserId="12443" LastActivityDate="2016-05-20T23:52:28.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11861" PostTypeId="2" ParentId="11838" CreationDate="2016-05-21T02:57:09.073" Score="3" Body="&lt;p&gt;Have you read Janssens' dissertation &quot;&lt;a href=&quot;http://jeroenjanssens.com/jeroenjanssens-thesis.pdf&quot; rel=&quot;nofollow&quot;&gt;Outlier Selection and One-Class Classification&lt;/a&gt;&quot;? He has a chapter on evaluation which may be of use. Have you thought about &lt;a href=&quot;http://www.inf.u-szeged.hu/~busarobi/PDFs/BaKoBu07&quot; rel=&quot;nofollow&quot;&gt;artificial generation&lt;/a&gt; of negative instances? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the fact that you don't really have labelled instances of the negative class. Without these instances there is no way of computing &lt;strong&gt;precision = TP/(TP+FP)&lt;/strong&gt; which relies upon the count of incorrectly accepted negative instances (&lt;strong&gt;FP&lt;/strong&gt;). This just leaves you with &lt;strong&gt;recall = TP/(TP + FN)&lt;/strong&gt; as a computable metric - which by itself is next to useless in evaluating classifier performance i.e. it's trivial to write a classifier that get's 100% recall, just predict the positive class for every instance in your test data... you will correctly identify every true positive instance, but you will also incorrectly predict every negative instance as a positive instance. What is typically required is a balance or &lt;a href=&quot;http://www.edrm.net/resources/glossaries/grossman-cormack/precision-recall-tradeoff&quot; rel=&quot;nofollow&quot;&gt;trade-off between precision and recall&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You didn't mention a particular problem domain in which you are working, but perhaps you can leverage this domain knowledge as a work-around to the problem of OCC evaluation. This particular problem I will relate is technically cast as a &lt;a href=&quot;https://www.cs.uic.edu/~liub/NSF/PSC-IIS-0307239.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;PU&lt;/strong&gt; Learning&lt;/a&gt; problem (learning from &lt;strong&gt;P&lt;/strong&gt;ositive and &lt;strong&gt;U&lt;/strong&gt;nlabelled instances). I'll try to describe some solutions that I explored while tackling this challenge - I'll give you fair warning that they are not entirely satisfactory solution, but I believe they are logically sound and somewhat defensible. From a purely pragmatic perspective, they are definitely better than the alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Implicit Negative Class&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scenario: imagine a membership-based incentive program, in which people can pay an annual fee to become a member of Foobar Inc. Members, but not non-members, are entitled buy &lt;em&gt;widgets&lt;/em&gt; at Foobar's retail outlets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are in charge of a marketing drive to increase Foobar's membership. You have limited funds and reach, so you need to carefully select people from the general population that you believe will most likely become members if you target them with a promotional campaign. In this scenario you have a database of known Foobar member's (the labelled, positive class data, which we label POS) and a database of known non-members from the general public. This is your unlabelled data that contains people who may or may not become members if targeted. All we have is &lt;strong&gt;POS&lt;/strong&gt; instances and &lt;strong&gt;UL&lt;/strong&gt; instances -- there are no labelled &lt;strong&gt;NEG&lt;/strong&gt; instances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending upon your particular problem, context, and domain, you may be able to transform the OCC/PU problem into a more classical problem with standard performance metrics. In the situation I just described, there are at least three ways of creating an &lt;em&gt;implicit negative class&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use the &lt;a href=&quot;https://www.cs.uic.edu/~liub/S-EM/unlabelled.pdf&quot; rel=&quot;nofollow&quot;&gt;spy technique&lt;/a&gt; to generate &quot;reliable&quot; negative instances&#xA;that can be placed in your test set; &lt;/li&gt;&#xA;&lt;li&gt;If historical campaign data is available you could identify those&#xA;individuals who were targeted multiple times for campaign&#xA;membership, but who rejected it every time -- these could be&#xA;&quot;consistently negative&quot; instances; &lt;/li&gt;&#xA;&lt;li&gt;If your known membership pool consists of members of varying&#xA;value to the business, you could divide members into good/bad, and&#xA;use the bad instances as a proxy for NEG instances.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One final area you may want to explore is evaluation in &lt;a href=&quot;http://recommender-systems.org/collaborative-filtering/&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt; research. Collaborative filtering on unary responses (e.g., Facebook &quot;like&quot;) &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;can be considered the matrix completion analog of the&#xA;  positive-unlabelled (PU) learning problem in classification and&#xA;  regression modelling. (&lt;a href=&quot;http://www.springer.com/us/book/9783319296579&quot; rel=&quot;nofollow&quot;&gt;Aggarwal, 2016&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Perhaps this field has developed some novel approaches to OCC evaluation that you can leverage. Without knowing the details of your particular case, however, its hard to provide a more detailed solution. &lt;strong&gt;Please update this thread with your eventual solution&lt;/strong&gt; -- &lt;em&gt;there is so much more research that needs to be done in this particular problem domain&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-05-21T13:36:33.020" LastActivityDate="2016-05-21T13:36:33.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11864" PostTypeId="2" ParentId="11847" CreationDate="2016-05-21T10:54:54.850" Score="0" Body="&lt;p&gt;In the x results obtained from the IR system, let TP, FP, FN, TN be the True Positives, False Positives, False Negatives and True Negatives of the trial, respectively.&#xA;When you take a random smaller-sized sample s of x observations (&lt;em&gt;cut list&lt;/em&gt; in your language), the values TP and FP decrease (by p and q) in their respective ratios. This is compensated by proportional increase (by p and q) in values FN and TN, conserving the actual number of binary classes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TP + FN = (TP - p) + (FN + p)&#xA;FP + TN = (FP - q) + (TN + q)&#xA;x = TP + FP, s = (TP - p) + (FP - q)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a good classifier, TP &gt;&gt;&gt; FP. So decrease in already minimal FP (in new sample), results in improved precision. Now your concept of &quot;relative recall&quot; comes into picture, which in this case can be expressed as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(TP - p)/(TP+FN + p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where TP acts as actual true observations (ground-truth) for the new sample.&#xA;Note: FN is not known because ground-truth labels are not available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The decrease in recall is caused by &lt;a href=&quot;https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Type_II_error&quot; rel=&quot;nofollow&quot;&gt;type-2 error&lt;/a&gt;, which is the FN rate (Not reporting a link as Call of Duty when it actually is), also denoted β (beta). So this reduction in recall (relative recall) is attributed to rise in False Negative Rate (miss rate), since&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Recall = 1 - FN Rate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17187" LastEditorUserId="6478" LastEditDate="2016-05-21T13:52:10.227" LastActivityDate="2016-05-21T13:52:10.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11865" PostTypeId="2" ParentId="10810" CreationDate="2016-05-21T14:55:19.013" Score="4" Body="&lt;p&gt;Another popular technique for detecting partial string matches (though typically at the document-level) is &lt;strong&gt;shingling&lt;/strong&gt;. In essence it is a moving-window approach that extracts out a set of n-grams for the target word/doc and compares them to the sets of n-grams for other words/docs via the &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow&quot;&gt;Jaccard coefficient&lt;/a&gt;. Manning and colleagues (2008) discuss &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html&quot; rel=&quot;nofollow&quot;&gt;near duplicates and shingling&lt;/a&gt; in the context of informational retrieval.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-05-21T14:55:19.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11868" PostTypeId="1" CreationDate="2016-05-21T19:21:47.717" Score="0" ViewCount="1766" Body="&lt;p&gt;In the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;official Keras example cifar10&lt;/a&gt; there is the following code to train a CNN using keras10. When I tried it, my neural net would not learn at all, I always get around a 10% acuracy, which is basicaly random guessing. If I change in the first dense layer the activation from &quot;reluat &quot; to &quot;sigmoid&quot;, then I get much better results. In fact, &lt;strong&gt;I thought that before softmax one had to have a sigmoid, not a relu&lt;/strong&gt;. However, this example appears all over the internet, so I am sure it is correct. What am I missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(P.S. I am training using a CPU, but that should not affect performance, just take longer, also, I stopped at around 20 epochs, but still with the sigmoid I got almost 70%, so that should not be the problem either)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'''Train a simple deep CNN on the CIFAR10 small images dataset.&#xA;GPU run command:&#xA;    THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python cifar10_cnn.py&#xA;It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs.&#xA;(it's still underfitting at that point, though).&#xA;Note: the data was pickled with Python 2, and some encoding issues might prevent you&#xA;from loading it in Python 3. You might have to load it in Python 2,&#xA;save it in a different format, load it in Python 3 and repickle it.&#xA;'''&#xA;&#xA;from __future__ import print_function&#xA;from keras.datasets import cifar10&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Activation, Flatten&#xA;from keras.layers import Convolution2D, MaxPooling2D&#xA;from keras.optimizers import SGD&#xA;from keras.utils import np_utils&#xA;&#xA;batch_size = 32&#xA;nb_classes = 10&#xA;nb_epoch = 200&#xA;data_augmentation = False&#xA;&#xA;# input image dimensions&#xA;img_rows, img_cols = 32, 32&#xA;# the CIFAR10 images are RGB&#xA;img_channels = 3&#xA;&#xA;# the data, shuffled and split between train and test sets&#xA;(X_train, y_train), (X_test, y_test) = cifar10.load_data()&#xA;print('X_train shape:', X_train.shape)&#xA;print(X_train.shape[0], 'train samples')&#xA;print(X_test.shape[0], 'test samples')&#xA;&#xA;# convert class vectors to binary class matrices&#xA;Y_train = np_utils.to_categorical(y_train, nb_classes)&#xA;Y_test = np_utils.to_categorical(y_test, nb_classes)&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(Convolution2D(32, 3, 3, border_mode='same',&#xA;                        input_shape=(img_channels, img_rows, img_cols)))&#xA;model.add(Activation('relu'))&#xA;model.add(Convolution2D(32, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;&#xA;model.add(Convolution2D(64, 3, 3, border_mode='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(Convolution2D(64, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;&#xA;model.add(Flatten())&#xA;model.add(Dense(512))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(nb_classes))&#xA;model.add(Activation('softmax'))&#xA;&#xA;# let's train the model using SGD + momentum (how original).&#xA;sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)&#xA;model.compile(loss='categorical_crossentropy',&#xA;              optimizer=sgd,&#xA;              metrics=['accuracy'])&#xA;&#xA;X_train = X_train.astype('float32')&#xA;X_test = X_test.astype('float32')&#xA;X_train /= 255&#xA;X_test /= 255&#xA;&#xA;if not data_augmentation:&#xA;    print('Not using data augmentation.')&#xA;    model.fit(X_train, Y_train,&#xA;              batch_size=batch_size,&#xA;              nb_epoch=nb_epoch,&#xA;              validation_data=(X_test, Y_test),&#xA;              shuffle=True)&#xA;else:&#xA;    print('Using real-time data augmentation.')&#xA;&#xA;    # this will do preprocessing and realtime data augmentation&#xA;    datagen = ImageDataGenerator(&#xA;        featurewise_center=False,  # set input mean to 0 over the dataset&#xA;        samplewise_center=False,  # set each sample mean to 0&#xA;        featurewise_std_normalization=False,  # divide inputs by std of the dataset&#xA;        samplewise_std_normalization=False,  # divide each input by its std&#xA;        zca_whitening=False,  # apply ZCA whitening&#xA;        rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)&#xA;        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)&#xA;        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)&#xA;        horizontal_flip=True,  # randomly flip images&#xA;        vertical_flip=False)  # randomly flip images&#xA;&#xA;    # compute quantities required for featurewise normalization&#xA;    # (std, mean, and principal components if ZCA whitening is applied)&#xA;    datagen.fit(X_train)&#xA;&#xA;    # fit the model on the batches generated by datagen.flow()&#xA;    model.fit_generator(datagen.flow(X_train, Y_train,&#xA;                        batch_size=batch_size),&#xA;                        samples_per_epoch=X_train.shape[0],&#xA;                        nb_epoch=nb_epoch,&#xA;                        validation_data=(X_test, Y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-16T15:46:51.073" LastActivityDate="2021-02-16T15:46:51.073" Title="cifar10 official keras example not giving expected accuracy, using sigmoid seems better than relu" Tags="|deep-learning|neural-network|keras|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11870" PostTypeId="2" ParentId="11853" CreationDate="2016-05-22T08:48:06.823" Score="12" Body="&lt;p&gt;Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a &quot;channel&quot; in the next layer, although they are usually visualised separately where the input is visualised with channels combined. Another way of thinking about this is that the separate RGB channels in the original image are 3 &quot;feature maps&quot; in the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't matter how many channels or features are in a previous layer, the output to each feature map in the next layer is a single value in that map. One output value corresponds to a single virtual neuron, needing one bias weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a CNN, as you explain in the question, the same weights (including bias weight) are shared at each point in the output feature map. So each feature map has its own bias weight as well as &lt;code&gt;previous_layer_num_features x kernel_width x kernel_height&lt;/code&gt; connection weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, your example resulting in &lt;code&gt;(3 x (5x5) + 1) x 32&lt;/code&gt; weights total for the first layer is correct for a CNN with first hidden layer processing RGB input into 32 separate feature maps.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;sup&gt;*&lt;/sup&gt; You may be getting confused by seeing visualisation of CNN &lt;em&gt;weights&lt;/em&gt; which can be separated into the colour channels that they operate on.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-22T09:38:03.370" LastActivityDate="2016-05-22T09:38:03.370" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11871" PostTypeId="2" ParentId="10842" CreationDate="2016-05-22T15:29:52.407" Score="0" Body="&lt;p&gt;Person correlation assumes data is coming from a Normal distribution and there is a linear relationship. An alternative is the Spearman correlation or Kendall's tau for ranked data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an edit, here are the links to how you would calculate the Spearman Correlation coefficient and Kendall's Tau, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.spearmanr.html&quot; rel=&quot;nofollow&quot;&gt;http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.spearmanr.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.kendalltau.html&quot; rel=&quot;nofollow&quot;&gt;http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.kendalltau.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck.&lt;/p&gt;&#xA;" OwnerUserId="16220" LastEditorUserId="16220" LastEditDate="2016-05-22T17:05:56.590" LastActivityDate="2016-05-22T17:05:56.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11873" PostTypeId="2" ParentId="9228" CreationDate="2016-05-22T16:38:20.250" Score="2" Body="&lt;p&gt;I would add that decision trees can be used for both classification and regression tasks. &lt;em&gt;DT on the other hand predicts a class&lt;/em&gt; in the accepted answer would be more specific by describing &lt;strong&gt;Classification trees&lt;/strong&gt; which is technically a subtype of the generic DT concept.&#xA;One reference(ignoring the bottom layers that discuss specific implementations):&lt;br /&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/bT2TN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bT2TN.png&quot; alt=&quot;types of decision trees&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;From &lt;a href=&quot;http://www.simafore.com/blog/bid/62482/2-main-differences-between-classification-and-regression-trees&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15445" LastEditorUserId="98307" LastEditDate="2020-08-16T13:02:16.363" LastActivityDate="2020-08-16T13:02:16.363" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="11874" PostTypeId="2" ParentId="11836" CreationDate="2016-05-22T17:06:38.560" Score="1" Body="&lt;p&gt;Example code of the nice suggestion from stmax above, with modification to use RandomForest and match the questions sample size and number of features, I hope that helps:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;&#xA;X, y = make_classification(n_samples=10000,&#xA;                           n_features=34,&#xA;                           n_informative=10,&#xA;                           n_redundant=0,&#xA;                           n_repeated=0,&#xA;                           n_classes=2,&#xA;                           random_state=0,&#xA;                           shuffle=False)&#xA;&#xA;forest = RandomForestClassifier(n_estimators=250,&#xA;                              random_state=0)&#xA;&#xA;forest.fit(X, y)&#xA;importances = forest.feature_importances_&#xA;std = np.std([tree.feature_importances_ for tree in forest.estimators_],&#xA;             axis=0)&#xA;indices = np.argsort(importances)[::-1]&#xA;&#xA;# Print the feature ranking&#xA;print(&quot;Feature ranking:&quot;)&#xA;&#xA;for f in range(X.shape[1]):&#xA;    print(&quot;%d. feature %d (%f)&quot; % (f + 1, indices[f], importances[indices[f]]))&#xA;&#xA;# Plot the feature importances of the forest&#xA;plt.figure(figsize=(20,10))&#xA;plt.title(&quot;Feature importances&quot;)&#xA;plt.bar(range(X.shape[1]), importances[indices],&#xA;       color=&quot;g&quot;, yerr=std[indices], align=&quot;center&quot;)&#xA;plt.xticks(range(X.shape[1]), indices,rotation=60)&#xA;plt.xlim([-1, X.shape[1]])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5eP8L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5eP8L.png&quot; alt=&quot;Example feature importance plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15445" LastActivityDate="2016-05-22T17:06:38.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11875" PostTypeId="2" ParentId="11802" CreationDate="2016-05-22T17:15:52.723" Score="2" Body="&lt;p&gt;If your dependent variable is discrete, you should be using &lt;code&gt;glm&lt;/code&gt; with a &lt;code&gt;poisson&lt;/code&gt; model. You can use &lt;code&gt;lm&lt;/code&gt; but you're obviously violating assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See example: &lt;a href=&quot;https://stats.idre.ucla.edu/r/dae/poisson-regression/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://stats.idre.ucla.edu/r/dae/poisson-regression/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16220" LastEditorUserId="29575" LastEditDate="2018-09-08T19:27:08.663" LastActivityDate="2018-09-08T19:27:08.663" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="11880" PostTypeId="1" AcceptedAnswerId="11887" CreationDate="2016-05-22T23:56:24.217" Score="9" ViewCount="671" Body="&lt;p&gt;Machine learning texts describing algorithms such as gradient boosting machines or neural networks often comment that these models are good at prediction, but this comes at the price of a loss of explainability or interpretability. Conversely, single decision trees and classical regression models are labelled as good at explanation, but giving a (relatively) poor prediction accuracy compared to more sophisticated models such as random forests or SVMs. Are there machine learning models commonly accepted as representing a good tradeoff between the two? Is there are any literature enumerating the characteristics of algorithms which allow them to be explainable? (This question was previously asked on cross-validated)&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-05-30T17:06:15.627" Title="Which, if any, machine learning algorithms are accepted as being a good tradeoff between explainability and prediction?" Tags="|machine-learning|predictive-modeling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11881" PostTypeId="2" ParentId="11880" CreationDate="2016-05-23T05:26:04.523" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there machine learning models commonly accepted as representing a&#xA;  good tradeoff between the two?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assume that by &lt;em&gt;being good at prediction&lt;/em&gt; you mean being able to fit nonlinearities present in the data while being fairly robust to overfitting.&#xA;The tradeoff between interpretability and being able to predict those nonlinearities depends on the data and question asked. There really is no free lunch in data science and no single algorithm can be considered to be the best for any set of data (and the same applies for interpretability).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general rule should be that the more algorithms you know the better it is for you as you can adopt to your specific needs more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to pick my favorite for classification task that I often use in business environment I would pick &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Elastic_net_regularization&quot; rel=&quot;nofollow&quot;&gt;elastic-net for logistic regression&lt;/a&gt;&lt;/strong&gt;. Despite strong assumption about the process that generates the data it can easily adopt to data thanks to the regularization term maintaining its interpretability from basic logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there are any literature enumerating the characteristics of&#xA;  algorithms which allow them to be explainable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would suggest you to pick a well written book that describes the commonly used machine learning algorithms and thier pros and cons in diffrent scenarios.&#xA;An example of such book can be &lt;em&gt;&lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf&quot; rel=&quot;nofollow&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;&lt;/em&gt; by  T. Hastie, R. Tibshirani and J. Friedman&lt;/p&gt;&#xA;" OwnerUserId="17646" LastEditorUserId="6478" LastEditDate="2016-05-23T07:14:15.873" LastActivityDate="2016-05-23T07:14:15.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11883" PostTypeId="1" CreationDate="2016-05-23T09:08:52.157" Score="0" ViewCount="44" Body="&lt;p&gt;for a current research project I would need file access traces of real world cloud storages. Unfortunately I was not able to find any yet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More precisely I need real world traces with at least the following information: filename, file size, access time, operation (put, post, get, delete). The kind of data (if the files are images, textdata or something else) does not matter. Furthermore, it is also ok if it is not a cloud storage, but for example an online image gallery, as long as all 4 operations (put, post, get, delete) are included its fine for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="18933" LastActivityDate="2016-10-20T14:00:23.340" Title="Daily access pattern of a real cloud storage" Tags="|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11885" PostTypeId="2" ParentId="11883" CreationDate="2016-05-23T11:12:25.013" Score="1" Body="&lt;p&gt;Try these links from &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow&quot;&gt;UCI Machine Learning Repository&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Anonymous+Microsoft+Web+Data&quot; rel=&quot;nofollow&quot;&gt;Anonymous Microsoft Web Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/MSNBC.com+Anonymous+Web+Data&quot; rel=&quot;nofollow&quot;&gt;MSNBC.com Anonymous Web Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-05-23T11:12:25.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11886" PostTypeId="1" AcceptedAnswerId="11892" CreationDate="2016-05-23T13:05:46.413" Score="0" ViewCount="179" Body="&lt;p&gt;I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What mathematics tools should I use?&lt;/p&gt;&#xA;" OwnerUserId="14478" LastEditorUserId="10024" LastEditDate="2016-05-23T14:58:38.713" LastActivityDate="2016-05-23T17:52:40.703" Title="Verification of trained system" Tags="|machine-learning|dataset|data|learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11887" PostTypeId="2" ParentId="11880" CreationDate="2016-05-23T13:32:54.177" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there are any literature enumerating the characteristics of algorithms which allow them to be explainable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The only literature I am aware of is the recent &lt;a href=&quot;https://arxiv.org/pdf/1602.04938v1.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; by Ribero, Singh and Guestrin. They first define explainability of a single prediction:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;By “explaining a prediction”, we mean presenting&#xA;  textual or visual artifacts that provide qualitative understanding&#xA;  of the relationship between the instance’s&#xA;  components (e.g. words in text, patches in an image)&#xA;  and the model’s prediction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The authors further elaborate on what this means for more concrete examples, and then use this notion to define the explainability of a model. Their objective is to try and so-to-speak add  explainability artificially to otherwise intransparent models, rather than comparing the explainability of existing methods. The paper may be helpful anyway, as tries to introduce a more precise terminology around the notion of &quot;explainability&quot;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there machine learning models commonly accepted as representing a good tradeoff between the two?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I agree with @Winter that elastic-net for (not only logistic) regression may be seen as an example for a good compromise between prediction accuracy and explainability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a different kind of application domain (time series), another class of methods also provides a good compromise: Bayesian Structural Time Series Modelling. It inherits explainability from classical structural time series modelling, and some flexibility from the Bayesian approach. Similar to logistic regression, the explainability is helped by regression equations used for the modelling. See &lt;a href=&quot;http://people.ischool.berkeley.edu/~hal/Papers/2013/pred-present-with-bsts.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; for a nice application in marketing and further references.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Related to the Bayesian context just mentioned, you may also want to look at probabilistic graphical models. Their explainability doesn't rely on regression equations, but on graphical ways of modelling; see &quot;Probabilistic Graphical Models: Principles and Techniques&quot; by Koller and Friedman for a great overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure whether we can refer to the Bayesian methods above as a &quot;generally accepted good trade-off&quot; though. They may not be sufficiently well-known for that, especially compared to the elastic net example.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2016-05-23T13:32:54.177" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11888" PostTypeId="2" ParentId="9917" CreationDate="2016-05-23T14:02:53.470" Score="1" Body="&lt;p&gt;If you've got enough information on the underlying distributions of your data, classical statistical theory on regression models is an obvious alternative. These models always come with nice derivations of error estimates. See Frank Harrell's &quot;Regression Modelling Strategies&quot; for a comprehensive overview.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2016-05-23T14:02:53.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11890" PostTypeId="1" AcceptedAnswerId="12033" CreationDate="2016-05-23T16:37:45.187" Score="0" ViewCount="1289" Body="&lt;p&gt;I am writing a TensorFlow program which is trying to categorize a heavily skewed dataset between two categories. One category is represented at 30x the rate of the other.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category 0: 800&#xA;Category 1: 30000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The label for each record is a one-hot vector of length 2, representing the two categories. Using a cross entropy cost equation doesn't work very well since a very high accuracy can be attained just by rating everything as category 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then made a balanced dataset that has the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category 0: 800&#xA;Category 1: 800&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This dataset has about 67% accuracy using softmax, but there is a real-world cost when falsely categorizing a category 1 item as category 0, which is blown way up when mistakenly categorizing many more items as category 0 mistakenly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to fix this by making the cost equation penalize false categorization into category 0 more than other penalties. However, my cost equation is not producing any different results than the original cross entropy equation. I think I may have programmed it in wrong, but I am unsure where the mistake is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory, the skewed equation will add 1 to the normal cost when the label for a record is [0,1] and the predicted value is [1,0], which represents a miscategorization of a category 1 item into category 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The skewed cost equation is in the condition where &lt;code&gt;FLAGS.cost == 'skewed'&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = tf.placeholder(tf.float32, shape=[None, data_length])&#xA;&#xA;W = tf.Variable(tf.zeros([data_length, 2]))&#xA;b = tf.Variable(tf.zeros([2]))&#xA;&#xA;y = tf.nn.softmax(tf.matmul(x, W) + b)&#xA;&#xA;y_ = tf.placeholder(tf.float32, shape=[None, 2])&#xA;&#xA;if FLAGS.cost == 'cross_entropy':&#xA;    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))&#xA;elif FLAGS.cost == 'skewed':&#xA;    positive_cutoff = tf.constant([0.5, 0.5])&#xA;    desired_tensor = tf.constant([False, True])&#xA;    true_tensor = tf.fill(tf.pack([2, tf.shape(y)[0], 2]), True)&#xA;    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))+tf.to_float(&#xA;        tf.reduce_all(&#xA;            tf.logical_and(&#xA;                tf.pack([&#xA;                    tf.logical_and(tf.greater_equal(y_, positive_cutoff), desired_tensor),&#xA;                    tf.logical_and(tf.less_equal(y, positive_cutoff), desired_tensor)&#xA;                ]),#shape [2,x,2]&#xA;                true_tensor#shape [2,x,2]&#xA;            )#shape [2,x,2]&#xA;        )#shape [1]&#xA;    )&#xA;&#xA;train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost)&#xA;&#xA;correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))&#xA;&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18951" LastActivityDate="2016-06-02T12:58:30.820" Title="Tensorflow skewed cost function" Tags="|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11891" PostTypeId="1" CreationDate="2016-05-23T16:51:53.530" Score="1" ViewCount="514" Body="&lt;p&gt;Given two features in a data set: &lt;code&gt;NFObs&lt;/code&gt; (Number of failed observations) and &lt;code&gt;NObs&lt;/code&gt; (Number of total observations), I'd like to better understand how I should represent these data for downstream use in binary classification problem using a decision tree model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should the features be left &quot;as-is&quot; &lt;strong&gt;or&lt;/strong&gt; substituted for a new feature &lt;code&gt;FailRate&lt;/code&gt; equivalent to &lt;code&gt;(NFObs / NObs)&lt;/code&gt;?  My concern with the rate/proportion idea is that it the model will consider an instance originally &lt;code&gt;1/1&lt;/code&gt; the same as an instance originally &lt;code&gt;500/500&lt;/code&gt;.  However, someone interpreting the results of the model would value an instance with 500 observations much more than an instance with 1 observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;NFObs &amp;lt; NObs&lt;/code&gt; and &lt;code&gt;NObs &amp;gt;= 1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem Description&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each instance contains:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A unique id &lt;code&gt;uid&lt;/code&gt;, representing some widget.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two independent, quantitative measurements &lt;code&gt;feat1&lt;/code&gt; and &lt;code&gt;feat2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;class&lt;/code&gt; is &lt;code&gt;1&lt;/code&gt; if an instance (observation) &quot;failed&quot; and &lt;code&gt;0&lt;/code&gt; if the instance &quot;passed&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;NFObs&lt;/code&gt; is a count of how many times the &lt;code&gt;uid&lt;/code&gt; has previously been reported &lt;code&gt;1&lt;/code&gt;. (stored in some legacy db without access to &lt;code&gt;feat1&lt;/code&gt; or &lt;code&gt;feat2&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;NObs&lt;/code&gt; is a count of how many times the &lt;code&gt;uid&lt;/code&gt; has previously been reported either &lt;code&gt;1&lt;/code&gt; or &lt;code&gt;0&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hypothesis is that a model can predict whether a widget will fail using historical fail rate and newly collected &lt;code&gt;feat1&lt;/code&gt; and &lt;code&gt;feat2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [2]: df.head()&#xA;Out[2]: &#xA;&#xA;class   uid  feat1  feat2   NFObs   NObs&#xA;   1    001   63     21      500    500&#xA;   0    002   31     29        1      1&#xA;   1    003   38     56        1      1&#xA;   0    004   33     86        5     20&#xA;   0    005    7     35       13     77&#xA;&#xA;In [3]: dfRate.head()&#xA;Out[3]:&#xA;&#xA;class   uid  feat1  feat2   FailRate&#xA;   1    001   63     21       1.0&#xA;   0    002   31     29       1.0&#xA;   1    003   38     56       1.0&#xA;   0    004   33     86       0.25&#xA;   0    005    7     35       0.16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any insight or suggested reading would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="16855" LastEditorUserId="16855" LastEditDate="2016-05-23T18:12:20.563" LastActivityDate="2016-05-23T18:12:20.563" Title="How to best represent rate or proportion as a feature?" Tags="|machine-learning|preprocessing|feature-engineering|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11892" PostTypeId="2" ParentId="11886" CreationDate="2016-05-23T17:52:40.703" Score="0" Body="&lt;p&gt;Did you also validate the data set with a separate test set so you can see how it performs with known data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This answer has a good run down on how to do that:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set&quot;&gt;https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also is a best practice to ensure that you haven't overfit the data. The easiest way to do that is check the accuracy against the data that you used to train the data.  If its not substantially better than the test data (this data should never be used to train a model) then you have not over fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, if you eventually find out the correct classification in your system, I would compare the accuracy there to the accuracy you experienced in the training and test environments to determine if its performing as intended. &lt;/p&gt;&#xA;" OwnerUserId="18869" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-05-23T17:52:40.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11893" PostTypeId="1" AcceptedAnswerId="12415" CreationDate="2016-05-23T20:33:17.557" Score="4" ViewCount="596" Body="&lt;p&gt;My app receives messages with a random number of bits at a random time. But two weeks ago I started to notice some almost regular patterns on the metrics of my app. I suspect they are some bots sending artificially generated data to my app. Specifically, I'm looking for sequential subsets of messages in a time series where messages has almost the same number of bits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read about some methods but they use data where time is not a random variable. I appreciate any help you can provide, including books, web pages, tutorials (in Python if possible), etc.&lt;/p&gt;&#xA;" OwnerUserId="18612" LastEditorUserId="18612" LastEditDate="2016-05-26T01:27:41.120" LastActivityDate="2016-07-26T11:04:22.087" Title="If a time series has random time events, how to detect patterns?" Tags="|python|time-series|anomaly-detection|sequential-pattern-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11896" PostTypeId="1" CreationDate="2016-05-24T08:04:07.577" Score="2" ViewCount="585" Body="&lt;p&gt;After studying some academic papers about neural network recognition, plus trying some of them out myself, I do understand how you can train a network to give you a certain output on a defined set of output neurons depending on the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I struggle to understand, how I can dynamically add output neurons. I've seen neural network applications, which where recognizing objects from images, however they set of classes to be recognized could always be increased. From my understanding this should not be possible, as the size of output neurons is fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume I want to build a application in which user can make a photograph, upload it to the database and use if subsequent for image recognition.&#xA;I see only two possible solutions:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Retraining&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Retrain the whole neural network every time some adds a new image. However training is very expensive and this would require a lot of resources so users can add all their objects.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&quot;Dead&quot; Outputs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;as suggested in &lt;a href=&quot;https://datascience.stackexchange.com/a/9551/18744&quot;&gt;https://datascience.stackexchange.com/a/9551/18744&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could add a lot of neurons to the output-layer which are always trained to 0, unless we need more outputs. Once a new object appears, we start training the new output. However I believe the performance of such procedure would suffer in recognition terms, as we leave important training feature such as randomization out.&lt;/p&gt;&#xA;" OwnerUserId="18744" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2022-01-11T22:40:39.057" Title="dynamic addition and variety of neural network outputs" Tags="|machine-learning|image-classification|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11897" PostTypeId="1" AcceptedAnswerId="11904" CreationDate="2016-05-24T08:15:24.837" Score="3" ViewCount="527" Body="&lt;p&gt;I am bit confused about validating data. What is this data mainly for?? Like I am seeing some tutorial and they have some training[I know it] images , they they have some validation images[donot know] and some testing[I know] images? So what is validating images mainly for??&lt;/p&gt;&#xA;" OwnerUserId="13068" LastActivityDate="2016-05-24T11:29:40.507" Title="Difference between Validation data and Testing data?" Tags="|machine-learning|data-mining|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11898" PostTypeId="1" CreationDate="2016-05-24T09:03:28.020" Score="6" ViewCount="6368" Body="&lt;p&gt;I came across a Machine Learning software which I won't name, that claims that completely automates the Feature Engineering process from any source without domain knowledge, implying pretty much that turns Data Scientists unnecessary for this step. Now I find that claim a bit suspicious since it goes against the common sense of what I learned Feature Engineering consists of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But considering that it might be a lack of knowledge from my part to what point we can automate the featuring engineering process? I can think of some general approaches like statistical methods (median,etc..) or several types of coding (binary, polynomial,etc..) for categorical variables, but could anyone give me an overview state of the art approach regarding this subject or point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="917" LastEditorUserId="917" LastEditDate="2016-05-24T09:49:42.923" LastActivityDate="2019-08-31T15:42:53.013" Title="Automatic Feature Engineering" Tags="|machine-learning|feature-engineering|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11899" PostTypeId="2" ParentId="11812" CreationDate="2016-05-24T10:01:41.447" Score="2" Body="&lt;p&gt;It sounds like 'novelty detection' is what you might be looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a scikit library for this. It generates a one-class model and predicts whether new observations fit into the one class or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reading, I would like to refer to this link:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html#novelty-detection&quot; rel=&quot;nofollow&quot;&gt;Novelty detection scikit-learn&lt;/a&gt;&#xA;There you can also find an example using a SVM classifier.&lt;/p&gt;&#xA;" OwnerUserId="9186" LastEditorUserId="9186" LastEditDate="2016-05-24T17:26:20.467" LastActivityDate="2016-05-24T17:26:20.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11901" PostTypeId="2" ParentId="11812" CreationDate="2016-05-24T10:57:09.307" Score="4" Body="&lt;p&gt;Apart from the approach @Rolf Schorpion mentioned, there are others.&#xA;For example, you could use a deep neural network, specifically, an auto-encoder (see &lt;a href=&quot;https://github.com/h2oai/h2o-training-book/blob/master/hands-on_training/anomaly_detection.md&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; for a tutorial).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there's an important catch to all purely &quot;data-driven&quot; approaches: if the figure of 30 time series you mention in the comments is a typical order of magnitude for your training set, the results will be more or less arbitrary. If you don't define &quot;accordance&quot; in any way, this is a classification problem with only positive training data which consists of 30 data points with more than 100 features each. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless your data is very, very special (e.g., all time series are identical), there is just too much freedom in the solution of this problem. Different algorithms (or different parameter sets for the same algorithm) will use this freedom in very different ways. So you will probably see very different solutions when you experiment with different methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you don't want to do more or less arbitrary experiments and choose from the solutions afterwards, you have to use a method in which you somehow define the meaning of &quot;in accordance&quot; in advance. You may try and use traditional time series techniques to find a common model that fits each of the time series well. You could then postulate that &quot;accordance&quot; is a good fit to this model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you might just do some exploratory analysis of your data to decide upon a simple rule which could be used to detect accordance. There's lots of possibilities, and without further details on the application it's difficult to decide which one is appropriate. Delegating the decision of what's &quot;in accordance&quot; to some algorithm doesn't seem to be a good choice in this case, though.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2016-05-24T10:57:09.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11902" PostTypeId="2" ParentId="11898" CreationDate="2016-05-24T10:57:47.440" Score="0" Body="&lt;p&gt;Almost anything can be automated but that doesn't mean it makes sense theoretically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Developing features takes insight and content knowledge from the field that you are studying in order to make sense. This is why even the best data scientists need to work with content experts to make theoretically sound models&lt;/p&gt;&#xA;" OwnerUserId="17959" LastEditorUserId="17959" LastEditDate="2016-05-24T11:11:32.437" LastActivityDate="2016-05-24T11:11:32.437" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11903" PostTypeId="2" ParentId="11812" CreationDate="2016-05-24T11:05:15.083" Score="2" Body="&lt;p&gt;Based on your description, a simple cross-correlation should do it. You are testing whether a series point in the future correlates with a series in the past which you consider correct. This measurement is given exactly by the cross-correlation between the future series and the past series -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other good thing about cross correlation is you can adjust the length of your series accordingly, which gives a lot of flexibility. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross correlation itself might not be enough for your system, as it only provides a measure of difference, you can still build a model for the actual &quot;classification&quot; - since your output are binary, anything simple from a threshold detector (i.e. if xcor &gt; value then 1 else 0) to a small neural network should both work fine. &lt;/p&gt;&#xA;" OwnerUserId="18969" LastActivityDate="2016-05-24T11:05:15.083" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11904" PostTypeId="2" ParentId="11897" CreationDate="2016-05-24T11:24:08.947" Score="4" Body="&lt;p&gt;There are two uses for the validation set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) Knowing when to stop training&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some models are trained iteratively - like neural nets. Sooner or later the model might start to overfit the training data. That's why you repeatedly measure the model's score on the validation set (like after each epoch) and you stop training once the score on the validation set starts degrading again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia on Overfitting&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5dDzL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5dDzL.png&quot; alt=&quot;Learning Curves - Training vs Validation Error&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Training error is shown in blue, validation error in red, both as a function of the number of training cycles. If the validation error increases (positive slope) while the training error steadily decreases (negative slope) then a situation of overfitting may have occurred. The best predictive and fitted model would be where the validation error has its global minimum.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) Parameter selection&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model needs some hyper-parameters to be set, like the learning rate, what optimizer to use, number and type of layers / neurons, activation functions, or even different algorithms like neural net vs SVM... you'll have to fiddle with these parameters, trying to find the ones that work best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that you train a model with each set of parameters and then evaluate each model using the validation set. Finally you select the model / the set of parameters that yielded the best score on the validation set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both of the above cases the model might have fit the data in the validation set, resulting in a biased (slightly too optimistic) score - which is why you evaluate the final model on the test-set before publishing its score.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="676" LastEditDate="2016-05-24T11:29:40.507" LastActivityDate="2016-05-24T11:29:40.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11905" PostTypeId="2" ParentId="11665" CreationDate="2016-05-24T11:35:42.480" Score="0" Body="&lt;p&gt;If you'd like to keep things straight and simple for a start, you may want to view this as a standard categorial classification problem. As a second step, you use multi-step-Prediction, as @eulerleibniz suggested. To render this approach feasible, you only predict the label for the next second (not for five) as a first step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may use a random forest or gradient boosted trees algorithm. Both need little parameter tuning, are reasonably well-behaved, and implementations are widely available (e.g., in Python and R). You can have a go at your problem with these algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a second step, you then repeat the whole thing, and combine your input data with the prediction just generated to predict the next label. And so forth, until you've got the next five labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're just a little bit lucky, the results are already good enough. If not, at least they provide a benchmark for more laborious approaches like RNN.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastEditorUserId="13213" LastEditDate="2016-05-24T11:55:09.513" LastActivityDate="2016-05-24T11:55:09.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11906" PostTypeId="2" ParentId="11898" CreationDate="2016-05-24T14:47:06.483" Score="1" Body="&lt;p&gt;Well, there is some serious research going on in this direction under the label of &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_learning&quot; rel=&quot;nofollow&quot;&gt;&quot;feature learning&quot;&lt;/a&gt;. But as far as I know, it's not mature enough yet to be packaged into a software tool that renders manual software engineering superfluous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are important successes being achieved in that direction. Modern image recognition with deep neural networks often relies on features that the deeper layers of the neural network compute themselves instead of hand-crafted features developed by humans. So it's not entirely science-fiction. If you want to learn more about the image recognition part of the topic, you may want to watch this beautiful &lt;a href=&quot;https://www.youtube.com/watch?v=n1ViNeWhC24&quot; rel=&quot;nofollow&quot;&gt;video lecture&lt;/a&gt; by Andrew Ng, one of the leading researchers in this field.&lt;/p&gt;&#xA;" OwnerUserId="13213" LastActivityDate="2016-05-24T14:47:06.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11907" PostTypeId="2" ParentId="11898" CreationDate="2016-05-24T17:13:12.937" Score="13" Body="&lt;p&gt;In my experience, when people claim to have an automated approach to feature engineering, they really mean &quot;feature generation&quot;, and what they're actually talking about is that they've built a deep neural network of some sort. To be fair, in a limited sense, this could be a true claim. Properly trained deep neural networks can handle any number of pairwise correlations between individual features or groups of features. That said, without a great deal of up-front data pre-processing tools that know how to intelligently handle different types of input data (e.g., free text, images, etc.), none of this would be possible. Bottom line, it takes a great deal of manual effort to do something automatically.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-05-24T17:13:12.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11908" PostTypeId="2" ParentId="10885" CreationDate="2016-05-24T20:37:19.980" Score="3" Body="&lt;p&gt;The quick (and not very satisfying) answer is &quot;it depends&quot; -- specifically it depends upon what your &lt;strong&gt;underlying conceptual model of human emotion&lt;/strong&gt; is and how it &lt;strong&gt;manifests in verbal/written behavior&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is &lt;em&gt;your&lt;/em&gt; characterization of neutrality in relation to positive and negative valence? Can documents be put on some sort of quantitative scale with neutral sandwiched between positive and negative? This position has linguistic support at least with &lt;strong&gt;simple phrases which express single valence states&lt;/strong&gt; (&lt;em&gt;the solution is&lt;/em&gt; {&lt;em&gt;great&lt;/em&gt; &gt; &lt;em&gt;good&lt;/em&gt; &gt; &lt;em&gt;acceptable&lt;/em&gt; &gt; &lt;em&gt;poor&lt;/em&gt; &gt; &lt;em&gt;horrible&lt;/em&gt;}). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @dmb poster suggested, if this is your conceptual model, then you might reasonablly argue that neutral falls between pos/neg and all you have to do is &lt;strong&gt;determine the optimal boundaries/cutoff&lt;/strong&gt;. Thus, you can assign neutral to a test case even if the classifier wasn't trained on neutral cases (though you do need some way of determining these cutoffs). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about more complex cases? What happens when you move from the level of the phrasal unit to larger sentential and discourse level units? How would you rate the sentence &quot;&lt;em&gt;I love apples but hate bananas&lt;/em&gt;&quot;? Do the positive and negative elements &lt;strong&gt;cancel&lt;/strong&gt; each other out to create a neutral? You can easily see how muddied this gets when we start to talk about &lt;strong&gt;real&lt;/strong&gt; human texts. In my opinion, I don't think it is particularly meaningful to talk about a document-level emotion score. Rather, I believe people express emotion/sentiment that is directed &lt;strong&gt;at&lt;/strong&gt; an individual objects &quot;I hate bananas&quot; -&gt; &lt;strong&gt;hate(subj, obj)&lt;/strong&gt; that get combined into discourse-level constructions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, yes, I think you could reasonable defend the use of outputting a neutral category if only pos/neg are used to train... BUT you will need to justify your use of particular cutoffs as well as consider how to deal with longer documents that express multiple (conflicting) emotions. My preferred course of action would be to have people read and label tweets as pos, neg, neutral, and mixed (then ensure inter-coder reliability) to create my training data... let the algorithm do the hard work of finding the cutoff values.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastActivityDate="2016-05-24T20:37:19.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11909" PostTypeId="1" AcceptedAnswerId="11910" CreationDate="2016-05-24T21:16:56.540" Score="0" ViewCount="38" Body="&lt;p&gt;I have data for a 100,000 people with different personality traits. Here would be sample data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Person  Trait-1   Trait-2   Trait-3 .....   Trait-N&#xA;John     1          0         1                1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need a model where for a new user when I see Trait-X, I need a prediction of the likelihood of all other Traits i.e. how likely is he to have any of the other traits. Can someone point me to a possible model I could use? I am a novice so don't know much about this space. &lt;/p&gt;&#xA;" OwnerUserId="18992" LastEditorUserId="381" LastEditDate="2016-05-24T21:29:37.550" LastActivityDate="2016-05-24T22:05:32.880" Title="Modeling and Predicting Co-Occuring Values" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11910" PostTypeId="2" ParentId="11909" CreationDate="2016-05-24T22:05:32.880" Score="0" Body="&lt;p&gt;I would recommend a &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow&quot;&gt;Restricted Boltzmann machine&lt;/a&gt;, which is a type of neural network that can model probability distributions. What you are trying to do is to estimate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Marginal_distribution&quot; rel=&quot;nofollow&quot;&gt;marginal distribution&lt;/a&gt; of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Missing_data&quot; rel=&quot;nofollow&quot;&gt;missing features&lt;/a&gt;. In a probabilistic model, this is done through &lt;em&gt;marginalization&lt;/em&gt;. The beauty of this method is that it provides the full distribution over the missing features rather than mere &lt;a href=&quot;https://en.wikipedia.org/wiki/Point_estimation&quot; rel=&quot;nofollow&quot;&gt;point estimates&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; &lt;a href=&quot;http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/&quot; rel=&quot;nofollow&quot;&gt;are&lt;/a&gt; two tutorials.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-05-24T22:05:32.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11911" PostTypeId="1" CreationDate="2016-05-24T23:44:46.577" Score="3" ViewCount="443" Body="&lt;p&gt;For simplicity, suppose we're looking at Yelp reviews of restaurants, and are trying to classify the restaurant by cuisine type (e.g. &quot;Italian, Japanese,&quot; etc.). Lets also assume our data already a cuisine type column that we can use for accuracy checking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of approaching this problem would be a supervised Latent Dirichlet Allocation approach, where the restaurant type is the response. In this way, topics are trained to be then used in a multinomial logistic regression to guess the cuisine type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the above approach far superior to instead running (unsupervised) LDA with plenty of topics, and then using something like XGBOOST to predict cuisine type? In other words, we run unsupervised LDA, and then project all reviews to vector distances to each topic, and then use these feature vectors to predict cuisine type?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that sLDA will try to pick topics that better characterize each category type but is the former really superior to the latter approach? The reason I ask this is because I don't know of any fast sLDA implementations out there.&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-05-24T23:44:46.577" Title="sLDA vs. LDA+Classifier" Tags="|nlp|supervised-learning|unsupervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11912" PostTypeId="1" AcceptedAnswerId="11922" CreationDate="2016-05-25T05:08:13.180" Score="11" ViewCount="3562" Body="&lt;p&gt;When training neural networks, there are at least 4 ways to regularize the network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;li&gt; L1 Regularization&#xA;&lt;li&gt; L2 Regularization&#xA;&lt;li&gt; Dropout&#xA;&lt;li&gt; Batch Normalization&lt;/p&gt;&#xA;&#xA;&lt;p&gt;plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how would one choose which of those regularization methods to use? Is there a more principled way than &quot;just try everything and see what works&quot;?&lt;/p&gt;&#xA;" OwnerUserId="6403" LastActivityDate="2022-11-18T17:10:14.267" Title="Choosing regularization method in neural networks" Tags="|neural-network|regularization|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11914" PostTypeId="1" AcceptedAnswerId="11917" CreationDate="2016-05-25T08:18:52.983" Score="0" ViewCount="919" Body="&lt;p&gt;I have a dataset with the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TimeStamp  | Action  | UserId&#xA;2015-02-05 | Action1 | XXX&#xA;2015-02-06 | Action2 | YYY&#xA;2015-02-07 | Action2 | XXX&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I try to forecast future &lt;code&gt;Actions&lt;/code&gt; for specific users based on the &lt;code&gt;Users&lt;/code&gt; history in the dataset. I started with ntstool in MATLAB, but it can't handle mixed data types or non-numerical values. Now I am looking for other methods to predict future &lt;code&gt;Actions&lt;/code&gt; and find periodic patterns in the records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to convert the values for &lt;code&gt;Actions&lt;/code&gt; to numeric values or is there a possibility to create a neural network with mixed datatypes on the input? Maybe in R?&lt;/p&gt;&#xA;" OwnerUserId="19003" LastActivityDate="2019-05-30T08:09:00.090" Title="Time Series Forecasting with Neural Network (mixed data types)" Tags="|r|neural-network|time-series|matlab|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11915" PostTypeId="1" AcceptedAnswerId="11916" CreationDate="2016-05-25T08:56:51.667" Score="1" ViewCount="172" Body="&lt;p&gt;In R I have &lt;code&gt;data&lt;/code&gt; where &lt;code&gt;head(data)&lt;/code&gt; gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;day   promotion   profit   new_users&#xA;1           105    45662          33&#xA;2            12    40662          13&#xA;3            44    46800          20&#xA;4           203    54102          46&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now &lt;code&gt;day&lt;/code&gt; is simply the day (and is in order). &lt;code&gt;promotion&lt;/code&gt; is simply the promotion-value for the day, the &lt;code&gt;profit&lt;/code&gt; is the profit that day and &lt;code&gt;new_users&lt;/code&gt; is the number of new users that day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want investigate the relationships between &lt;code&gt;promotion&lt;/code&gt; to &lt;code&gt;profit&lt;/code&gt; and &lt;code&gt;new_users&lt;/code&gt;. We see a clear positive correlation between &lt;code&gt;promotion&lt;/code&gt; and &lt;code&gt;profit&lt;/code&gt;, and there is also a positive correlation between &lt;code&gt;promotion&lt;/code&gt; and &lt;code&gt;new_users&lt;/code&gt;.&#xA;In R I simply test correlation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor.test(data$promotion, data$profit, method=&quot;kendall&quot;, alternative=&quot;greater&quot; )&#xA;cor.test(data$promotion, data$new_users, method=&quot;kendall&quot;, alternative=&quot;greater&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which both gives a low p-value, ie we have a positive correlation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find a point where where the increase of &lt;code&gt;promotion&lt;/code&gt; don't increase &lt;code&gt;profit&lt;/code&gt; or &lt;code&gt;new_users&lt;/code&gt; that must, ie a sweet spot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is 2 plots and the R code for these&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(data$promotion, data$profit, col=&quot;brown&quot;)&#xA;plot(data$promotion, data$new_users)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0nKdo.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0nKdo.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wxL7A.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wxL7A.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should this be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts where to make a regression model.&#xA;For the first one &quot;promotion vs. new_users&quot; one could use a poisons model because it's a count-process, so a model like this would be a good chose?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(formula= data$new_users ~ data$promotion, family=&quot;poisson&quot;, data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next what regression model should one chose for the next one. Is it fair to say that this regression model is a good chose ? (I use sqrt command)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm(formula=data$profit ~ sqrt(data$promotion) , data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or maybe it's not even necessary to use a regression model at all to find a sweet spot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have now looked at 'good' new users. For each &lt;code&gt;day&lt;/code&gt; we have a &lt;code&gt;promotion&lt;/code&gt; value and we have a &lt;code&gt;count&lt;/code&gt; value which is the number of new good users. This plot shows us the number of good new users we get for a promotion for each day. For example for promotion value 90 we have a day where we got 8 new good users and a day where we got 14 new good users.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the right approach to find a sweet spot for the use of promotion ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xfbBd.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xfbBd.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="18787" LastEditorUserId="18787" LastEditDate="2016-05-30T09:56:31.300" LastActivityDate="2016-05-30T09:56:31.300" Title="Chose the right regression analysis" Tags="|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11916" PostTypeId="2" ParentId="11915" CreationDate="2016-05-25T09:35:14.523" Score="1" Body="&lt;p&gt;Since I cannot comment because I don't have enough reputation, I will post this as an answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your goal is to &quot;find a point where the increase of promotion don't increase profit or new_users&quot;, I won't do a simple regression, since the regression will tell you that if you do more promotions, you will &lt;strong&gt;always inscrease&lt;/strong&gt; the profits. I would say that, in reality, the relationship between promotions and profits or new users is &lt;strong&gt;not linear&lt;/strong&gt;. Because the number of new users is limited and the promotions are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better model is to say that there is a &lt;strong&gt;optimal promotion&lt;/strong&gt; that will give you the best increase of profits and new users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(if you have a real business to optimise, I would introduce the &lt;strong&gt;Customer Lifetime Value&lt;/strong&gt; of new users. Because generally, the new users you get when doing huge promotions will not come back...)&lt;/p&gt;&#xA;" OwnerUserId="18968" LastActivityDate="2016-05-25T09:35:14.523" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11917" PostTypeId="2" ParentId="11914" CreationDate="2016-05-25T10:56:39.593" Score="1" Body="&lt;p&gt;I guess I answered the question in the comments, so here goes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most ML models cannot deal with categorical values. A common way to solve this is to use one-hot encoding, also known as dummy variables. For very possible value of your categorical variable you create a column which is 0 unless this row has this category, then it is 1. It is possible to remove one of the categories since it is a linear combination of the other dummy variables (if all are 0, the last one must be 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside of this method is that it increases the dimensionality of your feature space. If you have enough data to support this or not that many categories that is not a problem. There are other alternatives, like taking the average feature of every category and adding that to your features as opposed to the categorical feature.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-25T10:56:39.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11919" PostTypeId="1" AcceptedAnswerId="11923" CreationDate="2016-05-25T13:08:06.693" Score="16" ViewCount="721" Body="&lt;p&gt;It seems to have become axiomatic that an ensemble of learners leads to the best possible model results - and it is becoming far rarer, for example, for single models to win competitions such as Kaggle. Is there a theoretical explanation for why ensembles are so darn effective?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="18843" LastEditDate="2016-06-04T10:50:57.223" LastActivityDate="2016-06-04T10:50:57.223" Title="Why are ensembles so unreasonably effective" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11920" PostTypeId="2" ParentId="11912" CreationDate="2016-05-25T13:11:34.620" Score="-1" Body="&lt;p&gt;Look at those algorithmic choices as additional hyperparameters and optimize them the same way as you do for your other hyperparameters. Typically this will require more data though.&lt;/p&gt;&#xA;" OwnerUserId="14826" LastActivityDate="2016-05-25T13:11:34.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11921" PostTypeId="1" AcceptedAnswerId="11932" CreationDate="2016-05-25T13:30:36.233" Score="9" ViewCount="9508" Body="&lt;p&gt;I don't see a package for doing Convolutional Neural Networks in R. Has anyone implemented this kind of algorithm in R?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="29169" LastEditDate="2021-02-16T15:41:26.837" LastActivityDate="2021-02-16T15:41:26.837" Title="Convolutional Neural Networks in R" Tags="|r|convolutional-neural-network|software-recommendation|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11922" PostTypeId="2" ParentId="11912" CreationDate="2016-05-25T15:36:41.020" Score="9" Body="&lt;p&gt;There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, it may not matter hugely &lt;em&gt;which&lt;/em&gt; regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-05-25T15:36:41.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11923" PostTypeId="2" ParentId="11919" CreationDate="2016-05-25T15:52:42.623" Score="16" Body="&lt;p&gt;For a specific model you feed it data, choose the features, choose hyperparameters etcetera. Compared to the reality it makes a three types of mistakes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bias (due to too low model complexity, a sampling bias in your data)&lt;/li&gt;&#xA;&lt;li&gt;Variance (due to noise in your data, overfitting of your data)&lt;/li&gt;&#xA;&lt;li&gt;Randomness of the reality you are trying to predict (or lack of predictive features in your dataset)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ensembles average out a number of these models. The bias due to sampling bias will not be fixed for obvious reasons, it can fix some of the model complexity bias, however the variance mistakes that are made are very different over your different models. Especially low correlated models make very different mistakes in this areas, certain models perform well in certain parts of your feature space. By averaging out these models you reduce this variance quite a bit. This is why ensembles shine.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-25T15:52:42.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11925" PostTypeId="2" ParentId="11672" CreationDate="2016-05-25T22:26:24.620" Score="0" Body="&lt;p&gt;Depending on the volume and density of the information you have about your customers and their purchases, you may be able to run a collaborative filtering, meaning finding the cluster of similar customers and see what each cluster tend to buy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, for each individual you can recommend what she has not bought before and has a greater metric (revenue in this case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the business standpoint, there may be other metrics you may also want to take into account (net profit, time in stock, etc.)&lt;/p&gt;&#xA;" OwnerUserId="18893" LastActivityDate="2016-05-25T22:26:24.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11926" PostTypeId="2" ParentId="11915" CreationDate="2016-05-25T22:58:54.057" Score="1" Body="&lt;p&gt;Sorry to post it as an answer, because of points required for comments. I could not understand the concept of &lt;code&gt;promotion&lt;/code&gt; in your case. Possible scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Count of promotions are available to users (quantitative)&lt;/strong&gt; In this case, the details of those promotions would help a lot, i.e. having a binary column for each promotion.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Promotion IDs (categorical)&lt;/strong&gt;. In this case, we need to consider it as &lt;code&gt;factor&lt;/code&gt;. For example, &lt;code&gt;promotion&lt;/code&gt; equals 20 would mean &quot;30% off dairy products&quot; and &lt;code&gt;20&lt;/code&gt; doesn't mean anything on its own and is not more effective (in increasing &lt;code&gt;new users&lt;/code&gt; or &lt;code&gt;profit&lt;/code&gt;) than &lt;code&gt;19&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18893" LastActivityDate="2016-05-25T22:58:54.057" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11927" PostTypeId="2" ParentId="11543" CreationDate="2016-05-25T23:16:18.603" Score="2" Body="&lt;p&gt;For the most part, models built in R (for example, linear regression using &lt;code&gt;lm&lt;/code&gt; ) can handle the categorical data coded as &lt;code&gt;factor&lt;/code&gt; and do not need any dummy coding.  You just need to do this before passing data to &lt;code&gt;lm&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;salesdata$product &amp;lt;- factor(salesdata$product)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, depending on the model you are about to build, you might not need to create dummy variables.&lt;/p&gt;&#xA;" OwnerUserId="18893" LastActivityDate="2016-05-25T23:16:18.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11928" PostTypeId="1" AcceptedAnswerId="11933" CreationDate="2016-05-26T04:13:04.033" Score="95" ViewCount="440745" Body="&lt;p&gt;I got ValueError when predicting test data using a RandomForest model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = RandomForestClassifier(n_estimators=10, max_depth=6, n_jobs=1, verbose=2)&#xA;clf.fit(X_fit, y_fit)&#xA;&#xA;df_test.fillna(df_test.mean())&#xA;X_test = df_test.values  &#xA;y_pred = clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Input contains NaN, infinity or a value too large for dtype('float32').&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I find the bad values in the test dataset? Also, I do not want to drop these records, can I just replace them with the mean or median?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="118595" LastEditDate="2021-09-15T07:42:44.663" LastActivityDate="2023-03-04T11:04:59.527" Title="ValueError: Input contains NaN, infinity or a value too large for dtype('float32')" Tags="|python|scikit-learn|pandas|random-forest|python-3.x|" AnswerCount="10" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="11930" PostTypeId="1" AcceptedAnswerId="11938" CreationDate="2016-05-26T10:08:05.470" Score="10" ViewCount="4683" Body="&lt;p&gt;What is the dimension hopping problem in machine learning (occurring in convolutional neural networks and image recognition)? I have googled about it but all I get is information on the Physics of material shape deformation. It will be more helpful to me if some one explain it with an example related to machine learning. Can anyone help me out with this or point me toward resources that can?&lt;/p&gt;&#xA;" OwnerUserId="19040" LastEditorUserId="13413" LastEditDate="2016-05-26T16:45:02.693" LastActivityDate="2017-06-26T06:15:24.287" Title="Dimension-Hopping in Machine Learning" Tags="|machine-learning|image-recognition|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11931" PostTypeId="2" ParentId="11928" CreationDate="2016-05-26T10:35:09.087" Score="9" Body="&lt;p&gt;Assuming &lt;code&gt;X_test&lt;/code&gt; is a pandas dataframe, you can use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html#pandas.DataFrame.fillna&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;DataFrame.fillna&lt;/code&gt;&lt;/a&gt; to replace the NaN values with the mean:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test.fillna(X_test.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17329" LastActivityDate="2016-05-26T10:35:09.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11932" PostTypeId="2" ParentId="11921" CreationDate="2016-05-26T14:26:10.500" Score="3" Body="&lt;p&gt;I guess there is no package for cnn but you can write your own convolutional layer. mxnet or h2o will be useful for it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;check this out:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dmlc.ml/rstats/2015/11/03/training-deep-net-with-R.html&quot; rel=&quot;nofollow&quot;&gt;http://dmlc.ml/rstats/2015/11/03/training-deep-net-with-R.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17020" LastActivityDate="2016-05-26T14:26:10.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11933" PostTypeId="2" ParentId="11928" CreationDate="2016-05-26T15:19:06.000" Score="77" Body="&lt;p&gt;With &lt;code&gt;np.isnan(X)&lt;/code&gt; you get a boolean mask back with True for positions containing &lt;code&gt;NaN&lt;/code&gt;s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;np.where(np.isnan(X))&lt;/code&gt; you get back a tuple with i, j coordinates of &lt;code&gt;NaN&lt;/code&gt;s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, with &lt;code&gt;np.nan_to_num(X)&lt;/code&gt; you &quot;replace nan with zero and inf with finite numbers&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://scikit-learn.org/stable/modules/impute.html#impute&quot; rel=&quot;noreferrer&quot;&gt;sklearn.impute.SimpleImputer&lt;/a&gt; for mean / median imputation of missing values, or&lt;/li&gt;&#xA;&lt;li&gt;pandas' &lt;code&gt;pd.DataFrame(X).fillna()&lt;/code&gt;, if you need something other than filling it with zeros.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10574" LastEditorUserId="8820" LastEditDate="2019-07-17T18:42:24.030" LastActivityDate="2019-07-17T18:42:24.030" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="11934" PostTypeId="1" AcceptedAnswerId="11985" CreationDate="2016-05-26T15:19:28.537" Score="3" ViewCount="466" Body="&lt;p&gt;I want to study the impact of 26 parameters on one variable, and therefore determine the 3 or 4 which have to most influence on it. For that, I have constructed a 10 x 26 matrix: 26 parameters with 10 observations, and a vector of 10 arrays containing the variable I want to study with 10 observations. From this, how can I determine which are the 3 or 4 most influential variables on the parameters. I was considering using PCA, but I'm not sure that's the right approach. Can anyone give me some direction here?&lt;/p&gt;&#xA;" OwnerUserId="19051" LastEditorUserId="19051" LastEditDate="2016-05-27T14:12:49.583" LastActivityDate="2016-06-05T04:08:00.910" Title="What statistical method can be applied in my case?" Tags="|data-mining|statistics|pca|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11935" PostTypeId="1" AcceptedAnswerId="11952" CreationDate="2016-05-26T15:38:15.903" Score="0" ViewCount="2923" Body="&lt;p&gt;I have to filter a pandas data frame by matching a complex regular expression on a text index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data frame is multi level indexed, and contains more than 2 million records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I'm doing is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;identifiers = self._data.index.get_level_values('identifier')&#xA;filt = ... # an_array_of_np.bool_with_the_same_length_as_my_data&#xA;&#xA;pattern = ... # a complex regular expression, as a string&#xA;filt = filt &amp;amp; np.array(identifiers.str.contains(pat=pattern, case=False, regex=True), dtype=np.bool)&#xA;&#xA;... # other filterings&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the line beginning by &lt;code&gt;filt = filt &amp;amp;&lt;/code&gt; is very slow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if you have some ideas to make it faster. I guess that's because of the &lt;code&gt;identifiers.str.contains&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks @Emre&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not allowed to share those data, but the code below demonstrates the problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Step 0: 0:00:00.013527&lt;/li&gt;&#xA;&lt;li&gt;Step 1: 0:00:00.010127&lt;/li&gt;&#xA;&lt;li&gt;Step 2: 0:00:04.468114&lt;/li&gt;&#xA;&lt;li&gt;Step 3: 0:00:02.109594&lt;/li&gt;&#xA;&lt;li&gt;Step 4: 0:00:00.027437&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In fact, my feeling is that we apply the regular expression on all the values of the identifiers, while I would expect that the filter applies on the possible values of the index (lost of values are reused many times).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import datetime&#xA;&#xA;N = 2000000&#xA;N_DVC = 10000&#xA;&#xA;def getData():&#xA;    identifiers = np.random.choice(np.array([&#xA;        &quot;need&quot;, &quot;need: foo&quot;, &quot;need: bar&quot;, &quot;need: foo: bar&quot;, &quot;foo: need&quot;, &quot;bar: need&quot;,&#xA;        &quot;not: need&quot;, &quot;not: need: foo&quot;, &quot;not: need: bar&quot;, &quot;not: need: foo: bar&quot;, &quot;foo: need: not&quot;, &quot;bar: need: not&quot;,&#xA;        &quot;need ign&quot;, &quot;need: foo ign&quot;, &quot;need: bar ign&quot;, &quot;need: foo: bar ign&quot;, &quot;foo: need ign&quot;, &quot;bar: need ign&quot;,&#xA;        &quot;ign need&quot;, &quot;need: ign foo&quot;, &quot;need: ign bar&quot;, &quot;need: foo: ign bar&quot;,&#xA;    ]), N)&#xA;    devices = np.random.choice(np.arange(N_DVC))&#xA;    timestamps = np.random.choice(pd.date_range('1/1/2016 00:00:00', periods=60*60*24, freq='s'), N)&#xA;    x = np.random.rand(N)&#xA;    y = np.random.rand(N)&#xA;    data = pd.DataFrame({'identifier': identifiers, 'device': devices, 'timestamp': timestamps, 'x': x, 'y': y})&#xA;    data.set_index(['device', 'identifier', 'timestamp'], drop=True, inplace=True)&#xA;    return data&#xA;&#xA;def filterData(data):&#xA;    # I know those regular expressions are not perfect for the example,&#xA;    # but it mimics the real expressions I have&#xA;    rexpPlus = '^(?:[^\s]+:\s)*need(?:(?::\s[^\s]+)*:\s[^\s]+)?$'&#xA;    rexpMinus = '(?::\s)(?:(?:not)|(?:ign))(?::\s)'&#xA;&#xA;    tic = datetime.datetime.now()&#xA;    identifiers = data.index.get_level_values('identifier')&#xA;    print(&quot;- Step 0: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;    tic = datetime.datetime.now()&#xA;    filt = np.repeat(np.False_, data.shape[0])&#xA;    print(&quot;- Step 1: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;    tic = datetime.datetime.now()&#xA;    filt = filt | np.array(identifiers.str.contains(pat=rexpPlus, case=False, regex=True), dtype=np.bool)&#xA;    print(&quot;- Step 2: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;    tic = datetime.datetime.now()&#xA;    filt = filt &amp;amp; (~np.array(identifiers.str.contains(pat=rexpMinus, case=False, regex=True), dtype=np.bool))&#xA;    print(&quot;- Step 3: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;    tic = datetime.datetime.now()&#xA;    data = data.loc[filt, :]&#xA;    print(&quot;- Step 4: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;    return data&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;    filterData(getData())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3024" LastEditorUserId="3024" LastEditDate="2016-05-27T10:55:56.833" LastActivityDate="2016-07-26T12:04:21.913" Title="python pandas optimization: filtering on text index values" Tags="|python|pandas|performance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11936" PostTypeId="2" ParentId="334" CreationDate="2016-05-26T16:28:18.417" Score="2" Body="&lt;p&gt;I am almost done with Johns Hopkins Data Science Specialization on Coursera (A course and a capstone left to graduate). I will just give you the pros and cons of it, trying to keep it as objective as possible:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pros&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Structure around the learning process&lt;/li&gt;&#xA;&lt;li&gt;You'll build a portfolio over time&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cons&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Different backgrounds needed for different courses. The first few courses don't assume previous knowledge. It suddenly gets not easy to understand in the conceptual courses. (Statistical Inference, Regression Analysis)&lt;/li&gt;&#xA;&lt;li&gt;Taught by 3 professors. I think they are not on the same page about their potential audience and their abilities/needs/interests.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18893" LastActivityDate="2016-05-26T16:28:18.417" CommentCount="0" CommunityOwnedDate="2016-05-26T16:28:18.417" ContentLicense="CC BY-SA 3.0" />
  <row Id="11937" PostTypeId="2" ParentId="334" CreationDate="2016-05-26T17:23:11.353" Score="2" Body="&lt;p&gt;The best way to be successful at getting the job that you want it to show that you can do it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MOOCs that you mention will give you a good grounding in the basics and should be enough to get you started solving your own machine learning/data science problems. Try a Kaggle competition or two, that is a great way to improve your skills, and a decent grade there will be of interest to a potential employer. Publish your results on Github using something like an iPython Notebook, which will allow your work to be easily seen and judged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try an analysis on other public data sets, like &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Bike+Sharing+Dataset&quot; rel=&quot;nofollow&quot;&gt;the UCI Bike Sharing Dataset&lt;/a&gt;, or &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008&quot; rel=&quot;nofollow&quot;&gt;the UCI Diabetes Treatment Dataset&lt;/a&gt; those are lots of fun to try, and show that you are keen and willing to develop your skills.&lt;/p&gt;&#xA;" OwnerUserId="17509" LastActivityDate="2016-05-26T17:23:11.353" CommentCount="0" CommunityOwnedDate="2016-05-26T17:23:11.353" ContentLicense="CC BY-SA 3.0" />
  <row Id="11938" PostTypeId="2" ParentId="11930" CreationDate="2016-05-26T17:32:51.870" Score="7" Body="&lt;p&gt;Welcome to DataScience.SE! I'd never heard of this problem so I looked it up. It is explained on the third slide of &lt;a href=&quot;http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec5.pdf&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; presentation by Geoff Hinton:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;More things that make it hard to recognize objects&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;• Changes in viewpoint cause changes in images that standard learning&#xA;  methods cannot cope with.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;–  &lt;strong&gt;Information hops between input dimensions&lt;/strong&gt; (i.e. pixels)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;• Imagine a medical database in which the age of a patient sometimes&#xA;  hops to the input dimension that normally codes for weight!&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;–  To apply machine learning we would first want to eliminate this&#xA;  dimension-hopping.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In other words, it is about conceptual features migrating or hopping from one input feature dimension to another while still representing the same thing. One would like to be able to capture or extract the essence of the feature while being invariant to which input dimension it is encoded on.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-05-26T17:44:08.743" LastActivityDate="2016-05-26T17:44:08.743" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11939" PostTypeId="2" ParentId="11919" CreationDate="2016-05-26T17:42:22.650" Score="6" Body="&lt;p&gt;The selected answer is fantastic, but I would like to add two things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It has been observed that averaging human predictions gives better predictions than any individual prediction. This is known as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Wisdom_of_the_crowd&quot;&gt;wisdom of the crowd&lt;/a&gt;. Now, you could argue that it is because some people have different information, so you are effectively averaging information. But no, this is true even for tasks such as guessing the number of beans in a jar. I hypothesize it has to do with some of the reasons given above about data mining models.&lt;/li&gt;&#xA;&lt;li&gt;Some techniques such as the dropout method in neural networks (where in each iteration during training you use only a chunk of your neural network) give results similar to an ensemble of neural networks. The rationale is that you are effectively forcing nodes to do the same predictor work as the other nodes, effectively creating a meta-ensemble. I am saying this to make the point that we may be able to introduce some of the advantages of ensembles in traditional models.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-05-26T17:42:22.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11940" PostTypeId="1" CreationDate="2016-05-26T18:06:06.083" Score="8" ViewCount="945" Body="&lt;p&gt;I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled class$_{1}$, and 200 are labeled class$_{2}$.My goal is to used supervised learning to build a model that will classify new emails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, after a lot of pre-processing (parsing, removing stopwords, etc.), and trying typical algorithms (SVM, decision trees, etc.) on a document term matrix, my confusion matrix contained many false positives and false negatives, but only a few false negatives with SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how could I improve my results? Do I need to use oversampling,  or bi-gram feature representation? I guess the problem is that the topics of the two categories are really close.&lt;/p&gt;&#xA;" OwnerUserId="19054" LastEditorUserId="13413" LastEditDate="2016-05-27T02:29:52.843" LastActivityDate="2017-01-05T20:42:53.923" Title="Classifying Email in R" Tags="|machine-learning|r|text-mining|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11941" PostTypeId="2" ParentId="11934" CreationDate="2016-05-26T20:29:17.337" Score="4" Body="&lt;p&gt;&lt;strong&gt;Update: With this sample size you almost can't find any useful insight.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the ways to find one to one relationship is finding correlation coefficient of two random variables. Correlation is the statistical relationship between two random variables or attributes (in your case). This coefficient is a value between 1 and -1. If the value is close to 1 it means that there is a strong positive correlation between the two random variables. In other worlds as the value of one attributes increases, the value of the other attribute increases too. On the other hand, if the value is close to -1 it means that there is a strong negative correlation between them which means as the value of one attribute increases the value of the other attribute decreases. If the correlation value is close to zero it means that there is no significant correlation between the two random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you could use Python, you can load a CSV file into a pandas object and run the corr() method of the pandas object to get the correlation coefficient of two or more random variables. See the code below (I couldn't find the source of this code to mention)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from string import letters&#xA;import numpy as np&#xA;import pandas as pd&#xA;import seaborn as sns&#xA;import matplotlib.pyplot as plt&#xA;&#xA;sns.set(style=&quot;white&quot;)&#xA;&#xA;# Generate a large random dataset&#xA;rs = np.random.RandomState(33)&#xA;d = pd.DataFrame(data=rs.normal(size=(100, 26)),&#xA;                 columns=list(letters[:26]))&#xA;&#xA;# Compute the correlation matrix&#xA;corr = d.corr()&#xA;&#xA;# Generate a mask for the upper triangle&#xA;mask = np.zeros_like(corr, dtype=np.bool)&#xA;mask[np.triu_indices_from(mask)] = True&#xA;&#xA;# Set up the matplotlib figure&#xA;f, ax = plt.subplots(figsize=(11, 9))&#xA;&#xA;# Generate a custom diverging colormap&#xA;cmap = sns.diverging_palette(220, 10, as_cmap=True)&#xA;&#xA;# Draw the heatmap with the mask and correct aspect ratio&#xA;sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3,&#xA;            square=True, xticklabels=5, yticklabels=5,&#xA;            linewidths=.5, cbar_kws={&quot;shrink&quot;: .5}, ax=ax)&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dMxz9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dMxz9.png&quot; alt=&quot;Correlation Plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot you can see the strong or week correlation between all of the variables and then pick the one you find the most significant.&lt;/p&gt;&#xA;" OwnerUserId="12535" LastEditorUserId="12535" LastEditDate="2016-06-05T04:08:00.910" LastActivityDate="2016-06-05T04:08:00.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11942" PostTypeId="1" CreationDate="2016-05-26T20:39:25.157" Score="0" ViewCount="765" Body="&lt;p&gt;I have a corpus of data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Class   Tweet&#xA;-1  toxic phenol ingredient in vaccines  the detrimental effects of injected phenol  have not been fully evaluated   &#xA;-1  doctors give flu shots to pregnant women despite evidence of harm to fetus  &#xA;-1   hearus autism and the mmr vaccine  the most diabolical medical scandal of the century       cdcwhistleblower  vaxxed&#xA;-1  rt   how to naturally detox from mandatory vaccine injections     hearus  cdcwhistleblower&#xA;-1  rt   the removal of vaccine exemptions forces parents who know that vaccines injure and sicken their children into coerced ha   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'll be doing some supervised learning on the text.  I put the text through a pipeline as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;selector = SelectKBest(chi2, k = K)&#xA;&#xA;clf = SVC(kernel = 'linear')&#xA;&#xA;p = Pipeline([('vect', CountVectorizer()),&#xA;                    ('tfidf',TfidfTransformer() ),&#xA;                    ('feat',selector),&#xA;                    ('clf',clf)])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;SelectKBest&lt;/code&gt; will return &lt;code&gt;K&lt;/code&gt; of the best features.  I suppose then that those will be used to train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to predict new tweets on the same features?  Or can I just leave the pipeline as is and use CV to find the best K?&lt;/p&gt;&#xA;" OwnerUserId="9618" LastActivityDate="2016-11-22T23:41:41.363" Title="SelectKBest for text analytics" Tags="|python|nlp|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11943" PostTypeId="2" ParentId="11942" CreationDate="2016-05-26T21:00:32.610" Score="1" Body="&lt;p&gt;Your Pipeline is now a model that expects similar input as before. You can use a CV scheme to choose your K inside your Pipeline, after it has evaluated the options and chosen the correct K it will use that to predict. For actual predictions, enter a list of tweets and it will classify them, no need to manually specify which features to use. If you want to know which features your Pipeline has chosen you can look in the properties of your 'p' object.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-26T21:00:32.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11944" PostTypeId="2" ParentId="11716" CreationDate="2016-05-26T22:22:54.007" Score="1" Body="&lt;p&gt;In high accuracy regimes (&gt;0.9), I look at the error rate -- the complement of the accuracy -- or its reciprocal; the mean time between errors. This captures the intuition that an accuracy of 0.99 is ten times as &quot;good&quot; as an accuracy of 0.9.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-05-26T22:57:43.323" LastActivityDate="2016-05-26T22:57:43.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11945" PostTypeId="1" AcceptedAnswerId="11946" CreationDate="2016-05-27T06:16:56.320" Score="3" ViewCount="3383" Body="&lt;p&gt;I am reading a research paper which models a regression model where the returns are regressed on the number of ad exposures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the equation looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Returns = beta_1*nExp + beta_2*nExp^2$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nExp: The number of times the user looks at the ad.&#xA;&#xA;Returns: The revenue from the user&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is that: $beta_1$ is positive (and statistically significant), which means that the ad exposures has a $+ve$ relation with the revenue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, $beta_2$ is negative. And the research paper says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;term for the square of the number of exposures, which has a negative&#xA;  coefficient suggesting diminishing returns to ad exposure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What does it really mean?&lt;/strong&gt; &lt;strong&gt;What does it mean when the term's coefficient is $+ve$ but the coefficient of it's squared term in the equation is $-ve$?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-05-27T06:49:22.417" Title="What does it mean when the coefficient of the squared term is negative in regression?" Tags="|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11946" PostTypeId="2" ParentId="11945" CreationDate="2016-05-27T06:49:22.417" Score="8" Body="&lt;p&gt;In this context probably the plain English way to put it is that 'returns increase with additional exposure to the ad, but there is a tapering effect at the upper end of exposures. Looking at this picture of a parabola here (i.e a graph of y= -ax^2+bx+c):&lt;a href=&quot;https://i.stack.imgur.com/ibAHt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ibAHt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely your returns data are between 0 and the peak so you don't actually see  a decline, just a reduction in the rate of increase.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-05-27T06:49:22.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11949" PostTypeId="2" ParentId="11940" CreationDate="2016-05-27T07:29:15.960" Score="2" Body="&lt;p&gt;( All emails are or in french or in english )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pre-processing methods :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Merge &quot; Summary &quot; and &quot; Content &quot; which are the head of the email and the content&lt;/li&gt;&#xA;&lt;li&gt;Remove all emails adress&lt;/li&gt;&#xA;&lt;li&gt;Remove all &quot; From : Someone To : someone ... subject : something &quot;&lt;/li&gt;&#xA;&lt;li&gt;Remove all images included in the email&lt;/li&gt;&#xA;&lt;li&gt;Order the email according to their class&lt;/li&gt;&#xA;&lt;li&gt;Replace all french accent by no accent like é -&gt; e ; ê -&gt; e ...&lt;/li&gt;&#xA;&lt;li&gt;Put text to lower&lt;/li&gt;&#xA;&lt;li&gt;remove punctuation&lt;/li&gt;&#xA;&lt;li&gt;remove numbers&lt;/li&gt;&#xA;&lt;li&gt;strip white space&lt;/li&gt;&#xA;&lt;li&gt;remove some first name from a list and some specific words&lt;/li&gt;&#xA;&lt;li&gt;remove stop words in french and english&lt;/li&gt;&#xA;&lt;li&gt;stem document in french and english&lt;/li&gt;&#xA;&lt;li&gt;Remove term &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;--&gt; Then Document term matrix, with TF-IDF&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two classes are from a technical support, two categories &quot;complex&quot; and &quot;easy&quot;, &quot;complex&quot; are topics around finance ( in theory ) , &quot;simple&quot; problem with the software ( in theory ) but in practice they have a lot words in common.&#xA;And the other classes I don't take them into account I just focus on the two for now&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually the kind of classification algorithm I used is not so relevant because I tried with 5 algo and none gave good results &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example ( confusion matrix )&#xA;Decision tree: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Decision tree:           &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;pred: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;335 | 10&lt;br&gt;&#xA;59  | 12            &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SVM:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;331  | 1&lt;br&gt;&#xA;83   | 1                     &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Knn (n=10):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;330  | 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;83   | 1 &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Naive Bayes:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;1   |  83&lt;/p&gt;&#xA;&#xA;&lt;p&gt;12   |  320&lt;/p&gt;&#xA;" OwnerUserId="19065" LastEditorUserId="19065" LastEditDate="2016-05-27T07:52:46.847" LastActivityDate="2016-05-27T07:52:46.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11950" PostTypeId="1" CreationDate="2016-05-27T09:57:50.057" Score="2" ViewCount="475" Body="&lt;p&gt;I’m using Spark Scala for multiclass classification,  and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, Multinomial and Bernoulli are supported, where I would need to use Gaussian, and LogisticRegressionWithLBFGS would not be suitable either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that in Python, you can integrate sci-kit learn with Spark, but are there any options when using Scala?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interested to hear people’s thoughts on this.&lt;/p&gt;&#xA;" OwnerUserId="17794" LastActivityDate="2018-04-21T02:03:42.000" Title="Spark Scala alternative Machine Learning Library?" Tags="|time-series|apache-spark|multiclass-classification|scala|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11951" PostTypeId="1" AcceptedAnswerId="11955" CreationDate="2016-05-27T10:25:46.447" Score="2" ViewCount="1338" Body="&lt;p&gt;I'm building a neural network for data analysis, however I'm stuck on how many output neurons I need and what they should represent. The neural network tries to predict peoples choices for certain objects. There are 75 possible objects, but here's the catch, they choose 6 of them at a time. These six can (theoretically, though this won't happen realistically) be any combination of the 75 objects, in rare cases there might even be duplicates of a certain object in this set of six.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I naturally considered creating an output neuron for each possible set, but since that would lead to 75^6th output neurons (or more if we consider duplicates) I imagine that that would have disadvantages for the learning speed. Another option I considered is just taking the six highest ratest items as a set, but I'm unsure if this would work since the choice of the second, third, etc. objects depend on which earlier objects were chosen. I wonder if there are any better, faster or more accurate ways of doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;, some more information about the trainingdata:&#xA;The neural network will be used to analyse the ingame item choices in the computergame called league of legends. There players can choose up to six items each game. These six items are most likely based on some factors that change each game such as the ingame characters they and their enemies play, but also things such as what the other 5 items are they have. However, the reason for this may be universal to all games. For instance, the (likely) biggest reason why items depend on eachother is the multiplicativity of item stats, if you already have an item that makes your attacks hit harder then getting an item that allows you to attack more often is going to do more for you than getting an item that makes your attacks hit even harder. There are also some links between item choices though, while probably more limited, there is for instance an item that returns damage to the enemy when he attacks you, while it's not detrimental it is probably not a great idea to combine that with an item that lowers the amount of attacks that an enemy can do to you.&lt;/p&gt;&#xA;" OwnerUserId="19067" LastEditorUserId="19067" LastEditDate="2016-06-04T19:28:25.407" LastActivityDate="2016-06-30T13:33:59.640" Title="Using Neural Networks To Predict Sets" Tags="|neural-network|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="11952" PostTypeId="2" ParentId="11935" CreationDate="2016-05-27T10:58:48.910" Score="1" Body="&lt;p&gt;I've found a solution: instead of filtering through all the values, I compute a hash table (in fact a DataFrame) of the possible values and filter on those values:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Step 0: 0:00:00.014251&lt;/li&gt;&#xA;&lt;li&gt;Step 1: 0:00:00.010097&lt;/li&gt;&#xA;&lt;li&gt;Init fast filter: 0:00:00.353645&lt;/li&gt;&#xA;&lt;li&gt;Step 2: 0:00:00.027462&lt;/li&gt;&#xA;&lt;li&gt;Step 3: 0:00:00.002550&lt;/li&gt;&#xA;&lt;li&gt;Step 4: 0:00:00.027452&lt;/li&gt;&#xA;&lt;li&gt;Globally: 0:00:00.435839&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It runs faster because there are lots of duplicated values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a class which is doing that stuff. It works for indexes and normal columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;&#xA;class DataFrameFastFilterText(object):&#xA;    &quot;&quot;&quot;This class allows fast filtering of text column or index of a data frame&#xA;    Instead of filtering directy the values, it computes a hash of the values&#xA;    and applies the filtering to it.&#xA;    Then, it filters the values based on the hash mapping.&#xA;    This is faster than direct filtering if there are lots of duplicates values.&#xA;    Please note that missing values will be replaced by &quot;&quot; and every value will&#xA;    be converted to string.&#xA;    Input data won't be changed (working on a copy).&#xA;    &quot;&quot;&quot;&#xA;&#xA;    def __init__(self, data, column, isIndex):&#xA;        &quot;&quot;&quot;Constructor&#xA;        - data: pandas Data Frame&#xA;        - column: name of the column - the column can be an index&#xA;        - isIndex: True if the column is an index&#xA;        &quot;&quot;&quot;&#xA;        self._data = data.copy()  # working with a copy because we will make some changes&#xA;        self._column = column&#xA;        self._isIndex = isIndex&#xA;        self._hashedValues = None&#xA;        self._initHashAndData()&#xA;&#xA;    def _initHashAndData(self):&#xA;        &quot;&quot;&quot;Initialize the hash and transforms the data&#xA;        1. data: a. adds an order column&#xA;                 b. creates the column from the index if needed,&#xA;                 c. rename the column as 'value' and&#xA;                 d. drops all the other columns&#xA;                 e. fill na with &quot;&quot;&#xA;                 f. convert to strings&#xA;        2. hash: data frame of 'hashId', 'hashValue'&#xA;        3. data: a. sort by initial order&#xA;                 b. delete order&#xA;        &quot;&quot;&quot;&#xA;        self._data['order'] = np.arange(self._data.shape[0])&#xA;        if self._isIndex:&#xA;            self._data[self._column] = self._data.index.get_level_values(self._column)&#xA;        self._data = self._data.loc[:, [self._column, 'order']]&#xA;        self._data.rename(columns={self._column: 'value'}, inplace=True)&#xA;        self._data['value'] = self._data['value'].fillna(&quot;&quot;, inplace=False)&#xA;        self._data['value'] = self._data['value'].astype(str)&#xA;        self._hash = pd.DataFrame({'value': pd.Series(self._data['value'].unique())})&#xA;        self._hash['hashId'] = self._hash.index&#xA;        self._data = pd.merge(&#xA;            left=self._data,&#xA;            right=self._hash,&#xA;            on='value',&#xA;            how='inner',&#xA;        )&#xA;        self._data.sort_values(by='order', inplace=True)&#xA;        del self._data['order']&#xA;&#xA;    def getFilter(self, *args, **kwargs):&#xA;        &quot;&quot;&quot;Returns the filter as a boolean array. It doesn't applies it.&#xA;        - args: positional arguments to pass to pd.Series.str.contains&#xA;        - kwargs: named arguments to pass to pd.Series.str.contains&#xA;        &quot;&quot;&quot;&#xA;        assert &quot;na&quot; not kwargs.keys()  # useless in this context because all the NaN values are already replaced by empty strings&#xA;        h = self._hash.copy()&#xA;        h = h.loc[h['value'].str.contains(*args, **kwargs), :].copy()&#xA;        d = self._data.copy()&#xA;        d['keep'] = d['hashId'].isin(h['hashId'])&#xA;        return np.array(&#xA;            d['keep'],&#xA;            dtype=np.bool&#xA;        ).copy()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the example code is adapted as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;tic = datetime.datetime.now()&#xA;fastFilter = DataFrameFastFilterText(data, 'identifier')&#xA;print(&quot;- Init fast filter: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;tic = datetime.datetime.now()&#xA;filt = filt | fastFilter.getFilter(rexpPlus)&#xA;print(&quot;- Step 2: %s&quot; % str(datetime.datetime.now() - tic))&#xA;&#xA;tic = datetime.datetime.now()&#xA;filt = filt &amp;amp; (~fastFilter.getFilter(rexpMinus))&#xA;print(&quot;- Step 3: %s&quot; % str(datetime.datetime.now() - tic))&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2016-05-27T10:58:48.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11954" PostTypeId="2" ParentId="11950" CreationDate="2016-05-27T11:02:26.990" Score="1" Body="&lt;p&gt;The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively easy for any Machine Learning library in Scala (I do not use Scala, so I cannot help you with suggestions in that regard)&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-27T11:02:26.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11955" PostTypeId="2" ParentId="11951" CreationDate="2016-05-27T12:07:54.287" Score="9" Body="&lt;p&gt;The &lt;code&gt;75^6&lt;/code&gt; option is not only bad for speed, but it is a very difficult representation to train, because the NN doesn't &quot;understand&quot; that any of the output categories are related. You would need an immense amount of data to train such a network, because ideally you need at least a few examples in any category that you expect the network to predict. Unless you had literally billions of examples to train from, the chances are certain combinations will never occur in your training set, thus could never be predicted with any confidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I would probably use 75 outputs, one for each object representing the probability that it would be chosen. This is easy to create training data for, if you have training examples with the 6 favoured objects - just a 1 for the objects chosen and 0 for all others as a 75-wide label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For prediction, select the 6 objects with the highest probabilities. If these choices are part of a recommender system (i.e. may be presented to same person as being predicted for), then you can select items randomly using the outputs as weights. You may even find that this weighted Monte Carlo selection works well for predicting bulk user behaviour as well (e.g. for predictions fed into stock purchases). In addition, this stochastic approach &lt;em&gt;can&lt;/em&gt; be made to predict duplicates (but not accurately, except perhaps averaged over many predictions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sigmoid transfer function on the output layer is good for representing non-exclusive probability. The logloss objective function can be used to generate the error values and train the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to accurately predict duplicate choices out of the 6 items chosen, then you will need plenty of examples where duplicates happened and have some way to represent that in the output layer. For example, you could have double the number of output neurons, with two assigned to each object. The first probability would then be probability of selecting the item once, and the second probability would be for selecting it twice.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The question has since been updated, and it appears there are strong relationships between items making the choice of a set of items potentially very recipe-like. That may reduce the effectiveness of the ideas outlined above in this answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, using 75 outputs may still work better than other approaches, and is maybe the simplest setup, so I suggest still giving it a try, even if just to establish a benchmark for other ideas. This will work best when decisions are driven heavily by the feature data available, and when in practice there are lots of valid choices for combining items so there is a strong element of player preference. It will work less well if there is a large element of game mastery and logic in player decisions in order to combine items.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-06-05T06:40:13.513" LastActivityDate="2016-06-05T06:40:13.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11957" PostTypeId="2" ParentId="10310" CreationDate="2016-05-27T13:22:07.387" Score="4" Body="&lt;p&gt;There are many approaches you could take here. Assuming data overlap in such a way that the classes are still distinguishable from a human expert—human performance is generally regarded as the performance ceiling in supervised classification tasks—you can approach this from the classifier side of the feature side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I would consider the feature representation you're using for your task. A classifier is only as good as the information you give it, and there are many ways you can adjust how you're representing your input data. For example, if you're working in text classification, using &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow&quot;&gt;a unigram representation will typically lead to different performance as compared with a bigram representation&lt;/a&gt;. Similarly, there may be non-linear transformations you can apply to continuous-valued data that a classifier may not pick up on its own. You should consider the domain in which your problem exists and create a set of feature generation and feature selection cross-validation experiments using a hold-out training data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I would consider what alterations you can make on the classifier end. If your data are highly skewed (i.e., one class is much more prevalent that another), it may not make sense to use an algorithm like Naïve Bayes, at least not with out some sampling approach. If your problem isn't high-dimensional, I recommend generating a graphic that will let you explore the spatial breakdown by class. &lt;a href=&quot;http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;amp;doc=exercises/ex8/ex8.html&quot; rel=&quot;nofollow&quot;&gt;There are certain overlap patterns that linear classifiers like Support Vector Machines will not be able to classify without using a kernel function&lt;/a&gt;, such as a Radial Basis Function.&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-05-27T13:22:07.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11959" PostTypeId="2" ParentId="823" CreationDate="2016-05-27T13:43:34.510" Score="0" Body="&lt;p&gt;There don't seem to be any special ethics in data science - exploiting someone else's pain to make a buck is plain wrong. So is stealing their info to make a buck. It's not like being a doctor and deciciding whether to push the patient towards surgery with an outside chance of working when that's what they want but you know their spouse and kids just want to make make most of the time that they could have together. &#xA;So far these things seem little taught - but if anything is required it is assistance to recognise when people are being unfairly exploited.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="18843" LastEditDate="2016-05-27T14:02:34.900" LastActivityDate="2016-05-27T14:02:34.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11960" PostTypeId="1" CreationDate="2016-05-27T14:16:57.877" Score="2" ViewCount="185" Body="&lt;p&gt;Let's say that I am an email provider like Gmail. Let's assume that I have two categories of email-address: spammers and non-spammers. When my servers receive a mail I need to quickly check if that email-ID is in the spammers set and if yes I take some action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that each email-Id could be several bytes ( say 10 bytes each) and I might have 1 billion spammers so I need 10GB of RAM to just store the emails in main memory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I want to use only 1 GB RAM. To do this I'm now &lt;strong&gt;ready to accept an approximate answer&lt;/strong&gt;. In particular I'm ok with a non-spam email mistakenly flagged as belonging to the spam  set but not vice versa. How will I do it?&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="25180" LastEditDate="2018-05-06T22:11:59.783" LastActivityDate="2018-05-07T03:33:56.050" Title="Fast checking of set membership on large data" Tags="|classification|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11962" PostTypeId="1" AcceptedAnswerId="11963" CreationDate="2016-05-27T15:01:35.930" Score="9" ViewCount="13506" Body="&lt;p&gt;I'm writing my own training algorithm, but I don't know how to set the bias weight. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have I to set bias in any layer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Must the bias weight, be updated in every layer? &lt;/p&gt;&#xA;" OwnerUserId="19058" LastEditorUserId="29575" LastEditDate="2017-11-16T13:12:14.080" LastActivityDate="2017-11-16T13:12:14.080" Title="How to update bias and bias's weight using backpropagation algorithm" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11963" PostTypeId="2" ParentId="11962" CreationDate="2016-05-27T15:42:15.460" Score="11" Body="&lt;p&gt;There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are usually two different approaches taken when implementing bias. You can do one or the other:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;As a separate vector of bias weights for each layer, with different (slightly cut down) logic for calculating gradients.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As an additional column in the weights matrix, with a matching column of 1's added to input data (or previous layer outputs), so that the exact same code calculates bias weight gradients and updates as for connection weights.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In both cases, you only do backpropagation calculation from neuron activation deltas to the bias weight deltas, you don't need to calculate the &quot;activation&quot; delta for bias, because it is not something that can change, it is always 1.0. Also the bias does not contribute deltas back further to anything else.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-05-27T16:31:38.000" LastActivityDate="2016-05-27T16:31:38.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11964" PostTypeId="2" ParentId="6642" CreationDate="2016-05-27T16:21:48.643" Score="0" Body="&lt;p&gt;It appears that this can now be done to some degree with the AzureML Package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the next to last section, labeled &quot;Other examples of publishing web services&quot; on this page:&#xA;&lt;a href=&quot;https://htmlpreview.github.io/?https://github.com/RevolutionAnalytics/AzureML/blob/master/vignettes/getting_started.html&quot; rel=&quot;nofollow&quot;&gt;https://htmlpreview.github.io/?https://github.com/RevolutionAnalytics/AzureML/blob/master/vignettes/getting_started.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19083" LastActivityDate="2016-05-27T16:21:48.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11965" PostTypeId="2" ParentId="11960" CreationDate="2016-05-27T17:02:58.130" Score="5" Body="&lt;p&gt;Somebody already tagged this with &quot;bloom-filter&quot;.  I agree with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://en.wikipedia.org/wiki/Bloom_filter&quot; rel=&quot;noreferrer&quot;&gt;Bloom Filter&lt;/a&gt; is a hash-based technique that never results in false negatives, but trades off probability of false positives for saved space.  It meets your specific performance requirements - &quot;fewer than 10 bits per element are required for a 1% false positive probability, independent of the size or number of elements in the set (Bonomi et al. (2006)).&quot; (from the linked article)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main reason you wouldn't use a Bloom filter for this is that removing spammers is computationally expensive, requiring you to rebuild the filter or track your list of removed spammers separately.&lt;/p&gt;&#xA;" OwnerUserId="19085" LastActivityDate="2016-05-27T17:02:58.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11966" PostTypeId="2" ParentId="1050" CreationDate="2016-05-27T18:36:35.830" Score="2" Body="&lt;p&gt;This is an old post, but there is a subgraph package and accompanying book/documentation for doing this in R:&#xA;&lt;a href=&quot;https://www.csc.ncsu.edu/faculty/samatova/practical-graph-mining-with-R/PracticalGraphMiningWithR.html&quot; rel=&quot;nofollow&quot;&gt;https://www.csc.ncsu.edu/faculty/samatova/practical-graph-mining-with-R/PracticalGraphMiningWithR.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I personally don't get the connection between subgraph mining and label propagation in this case. SVD++ might be closer to what you're looking for (supported by GraphX of Spark, which I think also supports label propagation).&#xA;&lt;a href=&quot;http://spark.apache.org/graphx/&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/graphx/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19090" LastActivityDate="2016-05-27T18:36:35.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11967" PostTypeId="1" AcceptedAnswerId="11969" CreationDate="2016-05-27T22:26:54.557" Score="2" ViewCount="740" Body="&lt;p&gt;I keep seeing LeNet used to referring to a convolution network? I am wondering why LeNet is called LeNet? Is it the abbreviation of anything? Is there a difference between LeNet and convolutional neural network? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="29169" LastEditDate="2021-02-16T15:41:16.763" LastActivityDate="2021-02-16T15:41:16.763" Title="LeNet for Convolution network?" Tags="|deep-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11968" PostTypeId="2" ParentId="11756" CreationDate="2016-05-27T22:32:17.260" Score="1" Body="&lt;p&gt;Your problem is called multi-label classification. If you search for this term you will find a lot of literature. In short, these are the most common approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Treat each label independently and build a binary classifier per label (baseline approach, Eric Lecoutre's answer) &lt;/li&gt;&#xA;&lt;li&gt;Predict the power set, means create a target variable that represents all possible combinations and predict it using a multi-class classifier (Upper_Case's answer)&lt;/li&gt;&#xA;&lt;li&gt;Use a classifier that can deal with multi-labels inherently, e.g. a neural net with multiple output nodes and cross-entropy as loss function, or respective multi-label variants of decision trees or SVM. For example, scikit-learn supports multi-label implementation for decision trees.&lt;/li&gt;&#xA;&lt;li&gt;Classifier chains: Train a binary classifier per label, but use the predictions of one classifier as input to another classifier&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Depending on the nature of the data, one or the other approach will work best. Start with a binary classifier per label to get a baseline. If the labels are correlated (some labels frequently occur together), there is potential for improvement using one of the other methods. Using one of the existing implementations of multi-label classifiers is probably the easiest next step. The power set approach only makes sense if the number of possible combinations is reasonable. I am not sure if classifier chains are worth the effort, if you are not doing this for a Kaggle competition :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As always, setup a proper cross-validation strategy, define the evaluation metric (e.g. micro/macro f-measure or cross-entropy loss) so that you are able to identify the best model.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2016-05-27T22:32:17.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11969" PostTypeId="2" ParentId="11967" CreationDate="2016-05-28T00:24:06.610" Score="8" Body="&lt;p&gt;&lt;a href=&quot;http://yann.lecun.com/exdb/lenet/&quot;&gt;LeNet&lt;/a&gt; is a family (LeNet-1, LeNet-4, LeNet-5) of convolutional neural network designed by Yann LeCun et al. The name is a play on his name and the French language, where &quot;le&quot; means &quot;the&quot;, hence LeNet means &quot;the network&quot;. I believe it was originally devised to recognize handwritten numbers on checks (cheques). LeNet is only one early instance of a convolutional neural network; many others exist today.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-05-28T00:29:44.983" LastActivityDate="2016-05-28T00:29:44.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11970" PostTypeId="1" CreationDate="2016-05-28T00:55:13.630" Score="13" ViewCount="7007" Body="&lt;p&gt;The &lt;a href=&quot;https://matplotlib.org&quot; rel=&quot;noreferrer&quot;&gt;matplotlib&lt;/a&gt; library is very capable but lacks interactiveness, especially inside Jupyter Notebook.  I would like a good offline plotting tool like &lt;a href=&quot;https://plot.ly&quot; rel=&quot;noreferrer&quot;&gt;plot.ly&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="19094" LastEditorUserId="31513" LastEditDate="2017-08-09T07:08:31.443" LastActivityDate="2020-08-05T02:11:35.487" Title="How do I make an interactive PCA scatterplot in Python?" Tags="|python|visualization|pca|jupyter|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11971" PostTypeId="2" ParentId="11970" CreationDate="2016-05-28T00:55:13.630" Score="10" Body="&lt;p&gt;There is an awesome library called &lt;a href=&quot;http://mpld3.github.io/&quot; rel=&quot;nofollow&quot;&gt;MPLD3&lt;/a&gt; that generates interactive D3 plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This code produces an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to to mouseover the data point and see information about the original data. This functionality is very useful when doing exploratory data analysis.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;import pandas as pd&#xA;import seaborn as sb&#xA;import mpld3&#xA;from mpld3 import plugins&#xA;%matplotlib inline&#xA;&#xA;iris = sb.load_dataset('iris')&#xA;from sklearn.preprocessing import StandardScaler&#xA;X = pd.get_dummies(iris)&#xA;X_scal = StandardScaler().fit_transform(X)&#xA;&#xA;dim = 3&#xA;from sklearn.decomposition import PCA&#xA;pca = PCA(n_components = dim)&#xA;Y_sklearn = pca.fit_transform(X_scal)&#xA;&#xA;# Define some CSS to control our custom labels&#xA;css = &quot;&quot;&quot;&#xA;table&#xA;{&#xA;  border-collapse: collapse;&#xA;}&#xA;th&#xA;{&#xA;  color: #ffffff;&#xA;  background-color: #000000;&#xA;}&#xA;td&#xA;{&#xA;  background-color: #cccccc;&#xA;}&#xA;table, th, td&#xA;{&#xA;  font-family:Arial, Helvetica, sans-serif;&#xA;  border: 1px solid black;&#xA;  text-align: right;&#xA;}&#xA;&quot;&quot;&quot;&#xA;&#xA;fig, ax = plt.subplots(dim,dim, figsize=(6,6))&#xA;fig.subplots_adjust(hspace=.4, wspace=.4)&#xA;tooltip = [None]*dim&#xA;&#xA;N = 200&#xA;index = np.random.choice(range(Y_sklearn.shape[0]),size=N)&#xA;&#xA;for m in range(dim):&#xA;    for n in range(m+1):&#xA;        ax[m,n].grid(True, alpha=0.3)&#xA;        scatter = ax[m,n].scatter(Y_sklearn[index,m],Y_sklearn[index,n],alpha=.05)&#xA;&#xA;        labels = []&#xA;        for i in index:&#xA;            label = X.ix[[i], :].T.astype(int)&#xA;            label.columns = ['Row {0}'.format(X.index[i])]&#xA;            labels.append(str(label.to_html()))&#xA;&#xA;        ax[m,n].set_xlabel('Component ' + str(m) )&#xA;        ax[m,n].set_ylabel('Component ' + str(n) )&#xA;        #ax[m,n].set_title('HTML tooltips', size=20)&#xA;&#xA;        tooltip[m] = plugins.PointHTMLTooltip(scatter, labels,&#xA;                                           voffset=20, hoffset=20, css=css)&#xA;        plugins.connect(fig, tooltip[m])&#xA;&#xA;plugins.connect(fig, plugins.LinkedBrush(scatter))&#xA;test = mpld3.fig_to_html(fig=fig)&#xA;&#xA;with open(&quot;Output.html&quot;, &quot;w&quot;) as text_file:&#xA;    text_file.write(test)&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.scottlittle.org/2016/05/01/interactive-pca-scatterplots/&quot; rel=&quot;nofollow&quot;&gt;See it in action on my blog&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update [July 9, 2016]:&#xA;I just found out that Plot.ly has an offline mode and is now open source. It has a lot of the bells and whistles prepackaged, but MPLD3 may still be appropriate in some cases.&lt;/p&gt;&#xA;" OwnerUserId="19094" LastEditorUserId="19094" LastEditDate="2016-07-09T18:04:28.243" LastActivityDate="2016-07-09T18:04:28.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11972" PostTypeId="2" ParentId="10912" CreationDate="2016-05-28T01:27:44.200" Score="0" Body="&lt;p&gt;Take a look at the C# ENCOG framework:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.heatonresearch.com/encog/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.heatonresearch.com/encog/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/encog&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/encog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19095" LastEditorUserId="19095" LastEditDate="2017-02-13T16:40:17.833" LastActivityDate="2017-02-13T16:40:17.833" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11973" PostTypeId="1" AcceptedAnswerId="11984" CreationDate="2016-05-28T06:39:38.653" Score="0" ViewCount="1090" Body="&lt;p&gt;I have time-series data of 10-days for the same time interval as shown in below figure. Here it shows one-hour power consumption for 10 days. Data is sampled at 10 minutes rate.&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/tDklb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tDklb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to show this 10-day usage with a single baseline/representative curve. I can calculate baseline curve simply by taking mean/median of these 10 days data, but before that I need to answer following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The baseline should not represent the outlier (abnormal) usage days. Here in figure, we see that day 1 and 2 are following an unusual pattern from the rest of days. I think the usage of these days should not be used in baseline calculation. How should I exclude these days from baseline calculation automatically?&lt;/li&gt;&#xA;&lt;li&gt;How should I find the most similar usages out of these 10-day usages for my baseline calculation? I think the most/maximum similar usage days represent the days used for baseline calculation.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13291" LastActivityDate="2016-05-29T15:46:45.033" Title="Compute Baseline/Representative of Time-Series Data" Tags="|machine-learning|classification|data-mining|predictive-modeling|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11974" PostTypeId="1" CreationDate="2016-05-28T07:47:05.033" Score="3" ViewCount="402" Body="&lt;p&gt;I am not a stats/math expert by any stretch of the imagination, but have been trying some linear regression with census data and think I have run across a fundamental problem/obstacle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The response variable is a percent. The independent variables are percents by state. Here is an example of what I only know to refer to as  &quot;partitioned&quot; data. One of the groups of independent variables is &quot;lighting&quot; and has the following variables and values for one city (for example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;electricity - 40%&#xA;kerosene - 10%&#xA;candle - 25%&#xA;generator - 15%&#xA;solar - 5%&#xA;other - 5%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When added together, those six variables = 100%. There are only six choices and a household can only have one type of &quot;lighting&quot;. Using all six of these variables, causes my model to blow up producing coefficients that are off the charts and clearly something is wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insight on how to handle or at least still make some use of partitioned data with linear regression? Is there another name for these types of variables?&lt;/p&gt;&#xA;" OwnerUserId="19100" LastEditorUserId="11097" LastEditDate="2016-05-28T08:38:56.640" LastActivityDate="2016-05-31T14:52:48.417" Title="linear regression with &quot;partitioned&quot; data" Tags="|predictive-modeling|linear-regression|beginner|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11975" PostTypeId="1" CreationDate="2016-05-28T08:44:40.363" Score="4" ViewCount="608" Body="&lt;p&gt;A part of our system does a limited rule-based word labeling (kinda WSD). Basically it takes a dependency tree of a sentence, finds words or POS tags of interest and assigns to each a label/sense based on the word itself, it's POS tag, position in sentence and so forth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if ML methods in general and Tensorflow in particular can be used for the task, and if so which ones to try. I'm familiar with theoretical basics of ML, but never actually used it in work, so I'd grateful for any pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, our input is a ConLL representation of a sentence that we can turn into an matrix: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA; ['cat', 'NN', 'nsubj']&#xA; ['jumped', 'VBD', 'root']&#xA; ['over', 'IN', 'case']&#xA; ['fox', 'NN', 'nmod:over']&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;we can encode words and labels to numbers and pad sentence to max length of say 60 words, which gives the final input 60x3 matrix of integers. Our output then is a vector of 60 numbers with codes for labels for corresponding input words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far my research points to using bidirectional RNN for the task, but all tutorials I can find use them either for classification (as with MNIST) or finding the next best word as in the Tensorflow's RNN tutorial, whereas I want to label each word/input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is a bidirection RNN a good choice for the task? And if so, what should the composition be for it? How many steps and hidden layers would you recommend?&lt;/li&gt;&#xA;&lt;li&gt;If RNN isn't a good choice, what else would you recommend?&lt;/li&gt;&#xA;&lt;li&gt;Would using word embeddings add to the accuracy if our labeling doesn't care for word meanings/similarities?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="19101" LastEditorUserId="11097" LastEditDate="2016-05-28T08:47:30.287" LastActivityDate="2017-08-25T18:25:31.540" Title="Word labeling with Tensorflow" Tags="|neural-network|nlp|supervised-learning|word-embeddings|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11976" PostTypeId="2" ParentId="11974" CreationDate="2016-05-28T13:57:38.053" Score="1" Body="&lt;p&gt;It is due to the fact that the 6th predictor is a linear combination of the other 5. You can write:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;electricity = 100%-kerosene-candle-generator-solar-other&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Linear regression models do not handle linear combinations at all. However the solution is easy, since the 6th variable is embedded in the first 5, you can just drop a column since it is implied by the other 5.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-05-28T13:57:38.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11977" PostTypeId="1" CreationDate="2016-05-28T16:27:28.330" Score="2" ViewCount="561" Body="&lt;p&gt;I want to find outliers in power consumption in real-time, at hourly rate, i.e., at the end of the hour, I should say whether power consumption in current hour was outlier/anomalous or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach:&lt;/strong&gt;  Till now, I am done with following steps&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Say I want to find whether power usage between 9 AM to 10 AM was anomalous? For this, I first find the usage of past &lt;em&gt;n&lt;/em&gt; days during the same time interval, then I find the mean/median of all the previous usages&lt;/li&gt;&#xA;&lt;li&gt;Now, I have usage of the current day and the mean/median usage of previous &lt;em&gt;n&lt;/em&gt; days. Which statistical measure should I use to declare whether current day usage was anomalous or not?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Using above approach, for 24 hours of a specific (test) day and using past 10 days consumption, I have obtained results as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure interpretation: Black line represents usage of current hour of current day; red and blue lines represent mean and median of past 10 days for same time interval&#xA;&lt;a href=&quot;https://i.stack.imgur.com/DUO2f.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DUO2f.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/VhjN2.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VhjN2.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the visual inspection, I can say that the usage between 07:10  - 08:00  and between 22:10 - 23:00 is anomalous as there is big difference between actual and previous mean/median usage. I don't know which statistical measure should I use to point out such anomalous instances automatically, using the discussed approach.&lt;/p&gt;&#xA;" OwnerUserId="13291" LastEditorUserId="924" LastEditDate="2016-05-28T21:21:15.953" LastActivityDate="2016-06-02T17:25:23.797" Title="Find outliers in time-series data" Tags="|machine-learning|time-series|anomaly-detection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11978" PostTypeId="2" ParentId="11951" CreationDate="2016-05-29T01:59:14.987" Score="2" Body="&lt;p&gt;I don't think you can split the combinations and use it for prediction. E.g: I like fish n chips as my first choice but neither fish nor chips might be in my top 10 choices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better idea is to use what's called &quot;support&quot;. Do some preprocessing to find out which of the $(75)^6$ combinations occur at least say 100 times in your dataset. Only use these combinations for training and prediction. You want to identify generalizable patterns in people's choices not events which might have occurred just by chance.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-06-30T13:33:59.640" LastActivityDate="2016-06-30T13:33:59.640" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="11979" PostTypeId="1" CreationDate="2016-05-29T03:01:37.580" Score="2" ViewCount="303" Body="&lt;p&gt;I know that Torch supports multi-GPU computation on the same machine (&lt;a href=&quot;https://github.com/soumith/imagenet-multiGPU.torch&quot; rel=&quot;nofollow&quot;&gt;Example&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to perform multi-GPU, multi-machine computations with Torch?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-05-29T03:01:37.580" Title="Multi-GPU, multi-machine computation with Torch" Tags="|neural-network|deep-learning|distributed|torch|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11980" PostTypeId="1" CreationDate="2016-05-29T07:32:28.977" Score="1" ViewCount="517" Body="&lt;p&gt;I am looking for an interactive visualization. I have store classification based on their sales (sales buckets) for multiple quarters. I want to visualize &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The size of each bucket for every quarter&lt;/li&gt;&#xA;&lt;li&gt;The contribution towards&#xA;sales from each of the buckets in the previous quarter (essentially&#xA;capturing upgrade/downgrade of stores from one bucket to the other)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I tried Sankey diagrams but the issue is that it only helps me visualize the &lt;strong&gt;count&lt;/strong&gt; of stores moving (changing buckets) over time. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Edit 2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1NGk6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1NGk6.png&quot; alt=&quot;Sankey diagram showing movement of stores across buckets&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The numbers in the example below and the diagram might not match&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to capture the contribution of sales from previous bucket to the next as well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Q1 buckets|Q1 Sales  &#xA;0-5tr|1831  &#xA;5-10tr|561  &#xA;10-20tr|452  &#xA;20-50tr|400  &#xA;&#xA;Q2 buckets|Q2 Sales  &#xA;0-5tr|1600  &#xA;5-10tr|200  &#xA;10-20tr|50  &#xA;20-50tr|69  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, tr is an abbreviation for turnover&#xA;Similarly, I will have the number of stores in each bucket for both quarters. Although, I can visualize the movement of stores (using sankey) from one bucket to the other by creating a nXn matrix (n being the number of buckets) and each value i,j representing the number of stores moving from bucket i (in Q1) to bucket j (in Q2), I am not able to do so for its sales.&lt;/p&gt;&#xA;" OwnerUserId="15255" LastEditorUserId="11097" LastEditDate="2016-06-01T05:58:16.380" LastActivityDate="2016-06-01T05:58:16.380" Title="Visualization: Changing group memberships over time" Tags="|classification|time-series|visualization|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="11981" PostTypeId="1" CreationDate="2016-05-29T13:10:25.490" Score="1" ViewCount="171" Body="&lt;p&gt;I am working on clustering with DBSCAN but with a certain constraint: the points inside a cluster have to be not only near in a Euclidean distance way but also near in a geographic distance way. It is feasible if you use the pseudocode and work on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My issue however is about the proper metric on evaluating the clustering results. Applying the silhouette and c index metrics I get (as expected) results lower than the ones of clustering without constraints. A visual check shows that my clustering is well implemented but it seems as though that the interval of values of the metrics (for example [-1,1] for silhouette and [0,1] for C index) has changed. How can I verify that? I don't want a biased metric.&lt;/p&gt;&#xA;" OwnerUserId="16250" LastActivityDate="2016-05-29T13:10:25.490" Title="Clustering with restrictions - Silhouette and C index metrics" Tags="|clustering|dbscan|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11983" PostTypeId="2" ParentId="11682" CreationDate="2016-05-29T14:52:40.560" Score="0" Body="&lt;p&gt;Use regular expression.Following is a generic format&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; foreach A GENERATE FLATTEN(REGEX_EXTRACT_ALL(x,'&amp;lt;child&amp;gt;\\s*&amp;lt;subchild1&amp;gt;(.*)&amp;lt;/subchild1&amp;gt;\\s*&amp;lt;subchild2&amp;gt;(.*)&amp;lt;/subchild2&amp;gt;\\s*&amp;lt;/child&amp;gt;'));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18998" LastActivityDate="2016-05-29T14:52:40.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11984" PostTypeId="2" ParentId="11973" CreationDate="2016-05-29T15:46:45.033" Score="1" Body="&lt;p&gt;Ad 1. Assuming the measurements at any given time are normally distributed (they shape approximately a bell curve), you could use simple standard deviation to detect outliers. Specifically, for any given time, you can calculate the mean and standard error. Then you calculate the mean sans outliers by taking into account only the measurements that fall at most some pre-set distance from the mean (e.g. given normal distribution, &lt;a href=&quot;https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule&quot; rel=&quot;nofollow&quot;&gt;68% of measurements fall within one standard deviation from the mean&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pseudo-code example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Measurements at time t0 for all 10 days&#xA;t0 = np.array([0.1, 0.1, 1.4, .9, 1.25, 1.25, 1.5, 0.1, 0.3, 1.75])&#xA;&#xA;# Get mean and standard error&#xA;mean0, std0 = t0.mean(), t0.std()&#xA;&#xA;# Inliers are within one sigma from the mean&#xA;inliers = np.logical_and(mean0 - std0 &amp;lt; t0,&#xA;                         mean0 + std0 &amp;gt; t0)&#xA;# ==&amp;gt; [0, 0, 1, 1, 1, 1, 0, 0, 1, 0]&#xA;&#xA;# And the baseline mean at time t0 is&#xA;baseline0 = t0[inliers].mean()&#xA;# ==&amp;gt; 1.02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ad 2. You can find the most similar days to the baseline by using any appropriate distance measure (i.e. for time series: Euclidean or &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow&quot;&gt;dynamic time warping&lt;/a&gt;). The result, then, consists of those days where distance is the least.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-05-29T15:46:45.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11985" PostTypeId="2" ParentId="11934" CreationDate="2016-05-29T17:03:57.360" Score="5" Body="&lt;p&gt;There are a number of methods for this. Here's a list:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can build a Regression model and observe the p-values of the coefficients of each variable.&lt;/li&gt;&#xA;&lt;li&gt;Pearson Correlation&lt;/li&gt;&#xA;&lt;li&gt;Spearman Correlation&lt;/li&gt;&#xA;&lt;li&gt;Kendall Correlation&lt;/li&gt;&#xA;&lt;li&gt;Mutual Information&lt;/li&gt;&#xA;&lt;li&gt;RReliefF algorithm&lt;/li&gt;&#xA;&lt;li&gt;Decision trees&lt;/li&gt;&#xA;&lt;li&gt;Principal Component Analysis (which you have tried)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can search for other methods with these keywords:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;feature selection, variable importance, variable ranking, parameter selection&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are all almost the same, just different terminology in different fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the issue that you MUST pay attention here is &quot;10&quot; observations are almost nothing. It will be extremely difficult to trust any outcome, regardless of the method you use. &lt;/p&gt;&#xA;" OwnerUserId="17937" LastActivityDate="2016-05-29T17:03:57.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="11986" PostTypeId="1" CreationDate="2016-05-29T22:52:32.053" Score="0" ViewCount="287" Body="&lt;p&gt;I have may years of experience with database application development mainly focused on SQL Server and Oracle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to move into Data Science field and I my understanding is that I need a solid knowledge of Statistics and background - I wonder if that is correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder what are the important statistic topics that I must know as data science prerequisites?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am reading the following book:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics in a Nutshell 2nd Edition (&lt;a href=&quot;http://amzn.com/1449316824&quot; rel=&quot;nofollow&quot;&gt;http://amzn.com/1449316824&lt;/a&gt; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which of the following topics are important for data science or is there any other important topic that I must learn before I start Data Science?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Chapter 3 Inferential Statistics&lt;/li&gt;&#xA;&lt;li&gt;Chapter 4 Descriptive Statistics and Graphic Displays&lt;/li&gt;&#xA;&lt;li&gt;Chapter 6 The t-Test&lt;/li&gt;&#xA;&lt;li&gt;Chapter 7 The Pearson Correlation Coefficient&lt;/li&gt;&#xA;&lt;li&gt;Chapter 8 Introduction to Regression and ANOVA&lt;/li&gt;&#xA;&lt;li&gt;Analysis of Variance (ANOVA)&lt;/li&gt;&#xA;&lt;li&gt;Chapter 10 Multiple Linear Regression&lt;/li&gt;&#xA;&lt;li&gt;Chapter 11 Logistic, Multinomial, and Polynomial Regression&lt;/li&gt;&#xA;&lt;li&gt;Chapter 12 Factor Analysis, Cluster Analysis, and Discriminant Function Analysis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="19127" LastActivityDate="2016-05-30T00:07:18.817" Title="Data Scientist Prerequisites: What are the essential prerequisite statistics topics?" Tags="|statistics|beginner|" AnswerCount="1" CommentCount="1" ClosedDate="2016-05-30T14:16:52.897" ContentLicense="CC BY-SA 3.0" />
  <row Id="11987" PostTypeId="1" AcceptedAnswerId="11989" CreationDate="2016-05-29T23:43:05.540" Score="1" ViewCount="7332" Body="&lt;p&gt;I have a folder with 30,000 plus JSON file. A sample file with contents is posted below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;&quot;name&quot;: null, &quot;release_date_local&quot;: null, &quot;title&quot;: &quot;3 (2011)&quot;, &#xA;&quot;opening_weekend_take&quot;: 1234, &quot;year&quot;: 2011,&#xA;&quot;release_date_wide&quot;: &quot;2011-09-16&quot;, &quot;gross&quot;: 59954&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I need the data in a df in a structure as given below:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name relase_date_local title    opening_weekend_take year release_date gross&#xA;NA   NA                3 (2011) 1234                 2011 2011-09-16   5994&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is my code snippet to get all the files as a list:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;path = &quot;./Week1/jsonfiles&quot;&#xA;temp = list.files(path, pattern = &quot;*.json&quot;)&#xA;filename = paste(path, temp, sep = &quot;/&quot;)&#xA;movies = c()&#xA;for (i in filename){&#xA;  movie = fromJSON(file = i)&#xA;  movies = c(movies, movie)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please advise, how can I read all 30,000 files as rows of a df?&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2016-05-30T02:32:28.007" Title="How to read Several JSON files to a dataframe in R?" Tags="|r|dataset|data-cleaning|json|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="11988" PostTypeId="2" ParentId="11986" CreationDate="2016-05-30T00:07:18.817" Score="1" Body="&lt;p&gt;Everything on that list is is important material you should know. The sequence the topics are introduced looks to be chosen so that the material builds on itself as you move through (this effect is never totally perfect but seems generally right in this case), so I'd pretty much read the chapters as they appear.&#xA;The material listed is the most basic of basic statistics, so it is hard to make a case to omit something. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, while knowledge of these topics is important, it is very common (probably 'more common') for data scientists to involved in machine learning, which is a different approach to analysing data which gives better results for prediction. An overlapping term (not completely equivalent) term that is used is 'data mining' - searching on Amazon for these terms will give good results that can be used to pick a book that matches your background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that statistics and machine learning are both heavily dependent on maths, especially calculus, linear algebra and probability. The book you mention has been written for people with little background in these, but to go deeper you'll need to brush up or get new knowledge in these areas.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-05-30T00:07:18.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11989" PostTypeId="2" ParentId="11987" CreationDate="2016-05-30T02:32:28.007" Score="4" Body="&lt;p&gt;First, you can use the &lt;code&gt;full.names&lt;/code&gt; parameter to &lt;code&gt;list.files()&lt;/code&gt; to get &#xA;the full path added to each file.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;temp &amp;lt;- list.files(path, pattern=&quot;*.json&quot;, full.names=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, there are issues with the data since they contain &lt;code&gt;NULL&lt;/code&gt; values&#xA;which throws off a quick-and-dirty solution. So, we have to take each&#xA;list element and convert any &lt;code&gt;NULL&lt;/code&gt; to &lt;code&gt;NA&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, we can use the handy &lt;code&gt;purrr::map_df()&lt;/code&gt; to take the whole list&#xA;of lists and turn them into a &lt;code&gt;data.frame&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;movies &amp;lt;- purrr::map_df(temp, function(x) { &#xA;  purrr::map(jsonlite::fromJSON(x), function(y) ifelse(is.null(y), NA, y)) &#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="710" LastActivityDate="2016-05-30T02:32:28.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11990" PostTypeId="1" AcceptedAnswerId="12131" CreationDate="2016-05-30T04:48:24.397" Score="4" ViewCount="1472" Body="&lt;p&gt;Let's say I want to measure a medicine’s impact on height. So I randomly broke down my user base into two groups (control and experiment, obviously). I calculated the average height for both groups before experiment and found out there’s a 2 cm difference. Then, after the medicine is applied to the experiment group, I found a 4 cm difference.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I recall correctly, the t-test doesn’t handle selection bias, is there a test I can do to put the initial 2 cm difference between group into consideration?&lt;/p&gt;&#xA;" OwnerUserId="19132" LastEditorUserId="11097" LastEditDate="2016-05-30T06:02:18.657" LastActivityDate="2016-06-09T09:14:29.983" Title="A/B testing randomization step" Tags="|statistics|ab-test|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="11992" PostTypeId="2" ParentId="11518" CreationDate="2016-05-30T13:40:23.770" Score="0" Body="&lt;p&gt;Using Z-score can be ok if you're sure about what you're looking for. It can also be just a way to transform your data before using some ML on it. Be aware that Z-score applies to normally distributed data (which you say is not the case). Be also aware that looking for outliers in 3 dimensions is not as simple as looking 3 times for outliers in 1 dimension. You should plot your data in 3D, and try to find where might be the outliers.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, one-class SVMs are pretty good at anomaly/outliers detection. Take a look at the introduction &lt;a href=&quot;http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Also, any clustering algorithm might be helpful to get a better insight. &lt;a href=&quot;https://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow&quot;&gt;DBSCAN&lt;/a&gt;, for example, helps you to find clusters based on the density of the data.&lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-05-30T13:40:23.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11993" PostTypeId="1" CreationDate="2016-05-30T14:02:14.090" Score="4" ViewCount="245" Body="&lt;p&gt;I am using random forests, and, in my data, I have a lot of situations where $X_1$ is a bad predictor, $X_2$ is a bad predictor, but the joint distribution would make a good predictor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that $X1$, $X2$ and $Y$ are binary variables. We have that $P(X_1|Y)=0.5$ and $P(X_2|Y=y)=0.5$ for any $X_1,X_2,Y\in\{0,1\}$, but $P(X_1=1,X_2=1|Y=1)=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Practical example: I want to predict whether text1 is the same as text2. I have added as a feature &lt;code&gt;difference-in-usage-of-!&lt;/code&gt;. But, &lt;code&gt;difference-in-usage-of-!&lt;/code&gt; can be zero in two very different cases: when neither text has &lt;code&gt;!&lt;/code&gt; or when both use &lt;code&gt;!&lt;/code&gt; in the same numbers. So, I have added another feature: &lt;code&gt;both-texts-use-!&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem now is that the decision tree needs to be forward looking because it is only true that the text is the same if &lt;code&gt;difference-in-usage-of-!&lt;/code&gt; is very small and &lt;code&gt;both-texts-use-!&lt;/code&gt; is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I probably could and should combine these two features into a single, smarter feature (and suggestions are appreciated). But I have a lot of features such as this one, and I wonder whether the decision trees in my random forest, which are learnt using traditional local search are sub-optimal, relative to lookaheads kind of search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read in &lt;a href=&quot;https://arxiv.org/pdf/1407.7502.pdf&quot; rel=&quot;nofollow&quot;&gt;this thesis&lt;/a&gt; that,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As empirically investigated in&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.48.2462&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;&#xA;  [Murthy and Salzberg, 1995]&lt;/a&gt;, such an approach (using lookahead search) is however not only more computationally intensive, it is also not significantly better than greedily induced decision trees.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Comment in parenthesis is added by myself. The work &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.48.2462&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;being cited&lt;/a&gt; is another thesis, and it indeed says that, for instance,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The counter-intuitive result that lookahead search does not improve over greedy search would stand to reason if the latter itself constructs near-optimal trees.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Both of these works are old and, from what I read, do not delve much into it (but I have not read them fully). I wonder if anyone knows if there are features for which only some kind of lookahead search would work, or whether regular the regular local search algorithm is good even for those cases, and whatever thoughts you have about local vs lookahead searches.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2024-01-14T08:00:20.047" Title="How important is lookahead search in decision trees?" Tags="|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11994" PostTypeId="2" ParentId="9481" CreationDate="2016-05-30T14:31:37.593" Score="1" Body="&lt;p&gt;This is what your post looks like to me:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I have these data, what can I do with them?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is &quot;A LOT&quot;. What do you want to do exactly? what is the goal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the topic, I assume your goal is to find the relations between &quot;Keyword&quot;, &quot;Average Position&quot;, &quot;Clicks&quot;, &quot;Cost&quot;, ... with &quot;Impressions&quot;. If this is true, then you have multiple predictor variables and a target variable (Impressions) in which you would like to know the relation. This can be done through multiple statistical techniques such as: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pearson Correlation&lt;/li&gt;&#xA;&lt;li&gt;Spearman Correlation&lt;/li&gt;&#xA;&lt;li&gt;Kendall Correlation&lt;/li&gt;&#xA;&lt;li&gt;Mutual Information&lt;/li&gt;&#xA;&lt;li&gt;Simple regression (and checking importance of each predictor variable)&lt;/li&gt;&#xA;&lt;li&gt;RReliefF algorithm etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17937" LastActivityDate="2016-05-30T14:31:37.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11995" PostTypeId="2" ParentId="10840" CreationDate="2016-05-30T14:54:45.920" Score="1" Body="&lt;p&gt;The best way to look at this problem is to consider the extreme case: let's say you duplicate every row in your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you train a decision tree directly, there are certain hyperparameters you will want to change. For instance, in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier&quot; rel=&quot;nofollow&quot;&gt;sklearn&lt;/a&gt;, you will want to double hyperparameters like &lt;code&gt;min_samples_split&lt;/code&gt; or &lt;code&gt;min_samples_leaf&lt;/code&gt; because in each node you'll have double the observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to a random forest, usually, a random forest is learn by bootstrapping with resampling=True in the entire dataset. By doubling every observation, you make each observation just as likely to be picked. Even if you use a random forest which subsamples, doubling the dataset makes no difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you are not doubling your data. You are introducing new data which might change the distribution of your data. To the extent that it does, then it will mean that whatever hyperparameters you have previously used might no longer be optimal. So you should just try. But, to answer your question, &lt;strong&gt;no&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-05-30T14:54:45.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11996" PostTypeId="2" ParentId="11171" CreationDate="2016-05-30T15:01:55.780" Score="11" Body="&lt;p&gt;What data mining package do you use?&lt;/p&gt;&#xA;&lt;p&gt;In sklearn, the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.predict_proba&quot; rel=&quot;nofollow noreferrer&quot;&gt;DecisionTreeClassifier&lt;/a&gt; can give you probabilities, but you have to use things like &lt;code&gt;max_depth&lt;/code&gt; in order to truncate the tree. The probabilities that it returns is &lt;span class=&quot;math-container&quot;&gt;$P=n_A/(n_A+n_B)$&lt;/span&gt;, that is, the number of observations of class A that have been &amp;quot;captured&amp;quot; by that leaf over the entire number of observations captured by that leaf (during training). But again, you must prune or truncate your decision tree, because otherwise the decision tree grows until &lt;span class=&quot;math-container&quot;&gt;$n=1$&lt;/span&gt; in each leaf and so &lt;span class=&quot;math-container&quot;&gt;$P=1$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;That being said, I think you want to use something like a random forest. In a random forest, multiple decision trees are trained, by using different resamples of your data. In the end, probabilities can be calculated by the proportion of decision trees which vote for each class. This I think is a much more robust approach to estimate probabilities than using individual decision trees.&lt;/p&gt;&#xA;&lt;p&gt;But random forests are not interpretable, so if interpertability is a requirement, use the decision tree like I mentioned. You can use grid search to maximize the ROC AUC score by changing hyperparameters such as maximum depth to find whatever decision tree gives the most reliable probabilities.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2022-06-08T13:13:42.603" LastActivityDate="2022-06-08T13:13:42.603" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="11997" PostTypeId="2" ParentId="8598" CreationDate="2016-05-30T16:03:02.840" Score="1" Body="&lt;p&gt;I don't know how this is implemented in matlab. I know that some packages use cross validation to decide whether to grow the tree or not. Quite simply, they decide whether to grow based on that evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Notice what a lot of packages call pruning is: during training, they mark branches with some score criterium and then remove if the user wants afterwards.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I use sklearn which does not have this feature. So, I just do a grid search for several values of &lt;code&gt;max_depth&lt;/code&gt; and use whatever maximizes accuracy or whatever score I want.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-05-30T16:03:02.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11998" PostTypeId="2" ParentId="11919" CreationDate="2016-05-30T16:50:56.247" Score="6" Body="&lt;h2&gt;Ensembles win at prediction for theoretical and practical reasons.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There is a fundamental theory of optimal forecasting, if we mean predicting the next event in a sequence based on knowledge of previous events. Solomonoff prediction (Solomonoff 1964) is provably optimal in several senses, including that it “will learn to correctly predict any computable sequence with only the absolute minimum amount of data.” (Hutter, Legg &amp;amp; Vitanyi 2007)  A Solomonoff predictor weights all programs compatible with the existing data, according to the Kolmogorov complexity of the program and the probability that program assigns to the data so far, combining Epicurean (“keep all theories”) and Ockham (“prefer simple theories”) philosophies in a Bayesian framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimality properties of Solomonoff prediction explain the robust finding you refer to: averaging over models, sources, or experts improves predictions, and averaged predictions outperform even the best single predictor.  The various ensemble methods seen in practice can be seen as computable approximations to Solomonoff prediction -- and some like MML (Wallace 2005) explicitly explore the ties, though most do not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wallace (2005) notes that a Solomonoff predictor is not parsimonious – it keeps an infinite pool of models -- but most of the predictive power inevitably falls to a relatively small set of models. In some domains the single best model (or family of nearly indistinguishable models) may account for a large portion of the predictive power and outperform generic ensembles, but in complex domains with little theory most likely no single family captures the majority of the posterior probability, and therefore averaging over the plausible candidates should improve predictions. To win the Netflix prize, the Bellkor team blended over 450 models (Koren 2009).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Humans typically seek a single good explanation: in &quot;high-theory&quot; domains like physics, these work well. Indeed if they capture the underlying causal dynamics, they should be nearly unbeatable. But where the available theories do not closely fit the phenomena (say, film recommendation or geopolitics), single models will underperform: all are incomplete, so none should dominate. Thus the recent emphasis on ensembles (for machine learning) and Wisdom of the Crowds (for experts), and the success of programs like IARPA ACE and specifically the Good Judgment Project (Tetlock &amp;amp; Gardiner 2015).  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;M. Hutter, S. Legg, and P. Vitanyi, “Algorithmic probability,” Scholarpedia, vol. 2, 2007, p. 2572.&lt;/li&gt;&#xA;&lt;li&gt;Y. Koren, “The BellKor Solution to the Netflix Grand Prize,” 2009.&lt;/li&gt;&#xA;&lt;li&gt;Solomonoff, Ray (March 1964). &quot;A Formal Theory of Inductive Inference Part I&quot; (PDF). Information and Control 7 (1): 1–22. doi:10.1016/S0019-9958(64)90223-2.&lt;/li&gt;&#xA;&lt;li&gt;Solomonoff, Ray (June 1964). &quot;A Formal Theory of Inductive Inference Part II&quot; (PDF). Information and Control 7 (2): 224–254. doi:10.1016/S0019-9958(64)90131-7.&lt;/li&gt;&#xA;&lt;li&gt;P.E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know?, Princeton University Press, 2005.&lt;/li&gt;&#xA;&lt;li&gt;Tetlock, P. E., &amp;amp; Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. New York: Crown.&lt;/li&gt;&#xA;&lt;li&gt;C.S. Wallace, Statistical and Inductive Inference by Minimum Message Length, Springer-Verlag, 2005.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12417" LastActivityDate="2016-05-30T16:50:56.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="11999" PostTypeId="2" ParentId="11880" CreationDate="2016-05-30T17:06:15.627" Score="1" Body="&lt;p&gt;Possibly see &lt;a href=&quot;https://datascience.stackexchange.com/questions/11919/why-are-ensembles-so-unreasonably-effective/11998#11998&quot;&gt;my answer&lt;/a&gt; regarding the unreasonable effectiveness of ensembles, and the tradeoffs on explanation versus prediction.  Minimum Message Length (MML, Wallace 2005) gives a formal definition of explanation in terms of data compression, and motivates the expectation that explanations generally fit without overfitting, and good explanations generate good, generalizable predictions. But it also touches on the formal theory why ensembles will &lt;em&gt;predict&lt;/em&gt; better -- a result going back to (Solomonoff 1964) on optimal prediction and intrinsic to fully Bayesian approaches: integrate over the posterior distribution, don't just pick the mean, median, or mode.    &lt;/p&gt;&#xA;" OwnerUserId="12417" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-05-30T17:06:15.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12000" PostTypeId="2" ParentId="937" CreationDate="2016-05-30T18:45:38.523" Score="11" Body="&lt;p&gt;Sklearn DOES have a forward selection algorithm, although it isn't called that in scikit-learn.  The feature selection method called &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html&quot; rel=&quot;noreferrer&quot;&gt;F_regression&lt;/a&gt; in scikit-learn will sequentially include features that improve the model the most, until there are &lt;code&gt;K&lt;/code&gt; features in the model (K is an input).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It starts by regression the labels on each feature individually, and then observing which feature improved the model the most using the F-statistic.  Then it incorporates the winning feature into the model.  Then it iterates through the remaining features to find the next feature which improves the model the most, again using the F-statistic or F test.  It does this until there are K features in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that the remaining features that are correlated to features incorporated into the model will probably not be selected, since they do not correlate with the residuals (although they might correlate well with the labels).  This helps guard against multi-collinearity.&lt;/p&gt;&#xA;" OwnerUserId="10512" LastActivityDate="2016-05-30T18:45:38.523" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12001" PostTypeId="1" CreationDate="2016-05-30T19:31:13.520" Score="3" ViewCount="458" Body="&lt;p&gt;I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To describe inconsistency lets say we have a &lt;code&gt;country&lt;/code&gt; field and there can be multiple entries for &lt;strong&gt;USA&lt;/strong&gt;. e.g. &lt;em&gt;USA&lt;/em&gt;, &lt;em&gt;United States of America&lt;/em&gt;, &lt;em&gt;US&lt;/em&gt;, &lt;em&gt;US-50&lt;/em&gt;, &lt;em&gt;United States&lt;/em&gt;, &lt;em&gt;America&lt;/em&gt;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I normalize all these entries and &lt;em&gt;just say&lt;/em&gt; &lt;strong&gt;USA&lt;/strong&gt; for all entries. I can think of some set of full text search rules to match against using lucene or something. Is there any specific technical term for this problem?&lt;/p&gt;&#xA;" OwnerUserId="19147" LastEditorUserId="6478" LastEditDate="2016-05-30T23:50:34.850" LastActivityDate="2016-05-31T18:14:57.277" Title="How to fix inconsistent (variable spelling) categorical data and &quot;fill in&quot; missing data" Tags="|data-cleaning|categorical-data|missing-data|normalization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12002" PostTypeId="2" ParentId="11974" CreationDate="2016-05-31T14:52:48.417" Score="1" Body="&lt;p&gt;I think it depends on what algorithm you are using to optimize the loss. If you are using the normal equations to solve $X^{T}X^{-1}$ exactly you'll face problems because in the case of  perfectly correlated columns the inverse does not exist.&#xA;On the other hand if you are using numerical methods like gradient descent you might actually be OK. &lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-05-31T14:52:48.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12003" PostTypeId="1" AcceptedAnswerId="12006" CreationDate="2016-05-31T15:07:49.483" Score="9" ViewCount="638" Body="&lt;p&gt;I am about to start a job in which I will be working with large datasets and will be expected to find trends, etc...  I have found lots of resources on where to learn ML and other hard skills and feel that I am (semi) competent on this end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in knowing if there are specific soft skills that are helpful as a data scientist.  &lt;strong&gt;What are things you wish you knew starting out?&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While Kaggle is very useful when learning, it also presents clear objectives.  &lt;strong&gt;How do you handle being given a dataset, but no clear objective?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if this is too broad, I can think of more specific questions.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-12-05T10:16:59.843" Title="Tips for a new data scientist" Tags="|beginner|" AnswerCount="2" CommentCount="4" CommunityOwnedDate="2018-08-02T19:42:51.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="12004" PostTypeId="2" ParentId="10773" CreationDate="2016-05-31T15:16:26.130" Score="4" Body="&lt;p&gt;The k parameter is important if you use &lt;code&gt;selector.fit_transform()&lt;/code&gt;, which will return a new array where the feature set has been reduced to the best 'k'.&lt;/p&gt;&#xA;" OwnerUserId="19162" LastEditorUserId="29169" LastEditDate="2021-03-11T20:04:14.150" LastActivityDate="2021-03-11T20:04:14.150" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12005" PostTypeId="2" ParentId="8421" CreationDate="2016-05-31T16:18:48.583" Score="1" Body="&lt;p&gt;You could try using other metrics to measure interest. An example for an article would be &quot;time on page&quot;. If you measure scroll depth, even better. you could give a &quot;5&quot; rating if the user spent more than &quot;n&quot; seconds (where n is the time it takes to read the article, on average) or if the user scrolled all the way to the bottom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a &quot;3&quot; could be if the user didn't show a lot of interest in the article, for example. &lt;/p&gt;&#xA;" OwnerUserId="19167" LastActivityDate="2016-05-31T16:18:48.583" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12006" PostTypeId="2" ParentId="12003" CreationDate="2016-05-31T16:35:40.667" Score="10" Body="&lt;p&gt;I think there are a lot of important soft skills to consider in the Data Science domain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some of them:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Know for a fact what the goal is, spending a lot of time on data wrangling, models, visualization and reports when it was not all for the specific goal in mind is a waste. Communicating with less technical people is a skill in itself.&lt;/li&gt;&#xA;&lt;li&gt;Iterate repeatedly with the product owner. Keep making sure you are on the right path.&lt;/li&gt;&#xA;&lt;li&gt;If the data doesn't tell the story they thought/want tell them it is not the case, be clear in why this is happening, what biases might be playing a role etcetera. Do not apply all kinds of filters or keep changing parameters to get the desired results.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Regarding your second question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective has to be either gotten from the product owner explicitly or derived from a less mathematical objective. An example could be where you need to predict train arrivals based on some features. They want the model to predict as many times as possible within a 10-minute error range. This is relatively explicit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes it is less clear than that, they might say we need it as accurate as possible. Then you will have to decide what to optimize, in some cases, this will just be minimizing the MSE but in other cases, other things might make more sense for your case. Usually, this will be clear from the implicit objective and something that you will get better at with more experience. Both implicit and explicit objectives derive from clear communication with the product owner.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="26596" LastEditDate="2016-12-05T10:16:59.843" LastActivityDate="2016-12-05T10:16:59.843" CommentCount="3" CommunityOwnedDate="2018-08-02T19:42:51.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="12007" PostTypeId="2" ParentId="12001" CreationDate="2016-05-31T18:14:57.277" Score="2" Body="&lt;p&gt;I'm also new to Data Science, so my approach may be laughably naive.  But I work as a &quot;terminology analyst&quot; who prepares and maintains standardized vocabularies and code sets (which are meant to avoid the very problem you describe), so at least I'm describing things I've actually paid for.  In fact, when my program was first established, a slide presentation called &quot;The State of the State File&quot; was prepared describing a problem very similar to yours: The file used to populate the data entry pick list of states contained hundreds of different ways the 50 US states could be represented in text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question: I would call the name of the problem &quot;unstructured data&quot; or &quot;unstandardized data.&quot;  We call the process of fixing it &quot;data standardization.&quot;  The outcome of such standardization is a &quot;reference terminology,&quot; which is a list of standardized terms as well as synonyms (aka &quot;search terms&quot;) for each standard term.  Sometimes alphanumeric codes are assigned to each standardized term, resulting in a &quot;code set.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach my organization has taken to standardizing data is rudimentary and hardly qualifies as any sophisticated data science: we run a DISTINCT query to get all the different text strings used to represent concepts.  Then that list needs to be reviewed to see what different text strings represent the same thing.  If your list is too large to review each text string in the amount of time you have, then you need to get frequency data for the number of times each distinct text string occurs. Sort the list of terms from highest to lowest number of occurrences, and work your way down the list as far as you can in the time alotted.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sounds simplistic, but sometimes you need to make judgement calls about equality or subsumption.  For example, if one of your text strings is &quot;the continental united states&quot; (excluding Alaska and Hawaii) do you lump that under &quot;USA&quot; or does it get its own category?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your result would be a reference terminology of a finite number of standardized country names, and all the synonyms for each country.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One place to start is ISO-3166, a set of codes &quot;for the names of countries, dependent territories, special areas of geographical interest, and their principal subdivisions (e.g., provinces or states).&quot;  You may be able to match each of your text strings to one of these established codes.&lt;/p&gt;&#xA;" OwnerUserId="19170" LastActivityDate="2016-05-31T18:14:57.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12008" PostTypeId="2" ParentId="10982" CreationDate="2016-05-31T20:02:03.890" Score="1" Body="&lt;p&gt;Not a fix to your exact problem, but an alternative solution would be to try &lt;code&gt;library(tidyr)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyr)&#xA;d &amp;lt;- gather(d,&quot;variable&quot;,&quot;value&quot;,2:3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And it will give you columns 2:3 as rows.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-05-31T20:02:03.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12009" PostTypeId="2" ParentId="11970" CreationDate="2016-05-31T22:51:59.320" Score="3" Body="&lt;p&gt;I would prefer this to be a comment instead of an answer, as my intention is not to plug/advertise, but I am currently working on my thesis which may be of interest to you as it kind of does what you want. In reality it is a clustering visualization tool, but if you use k-means with k=1 you have an interactive plot where you can search for terms, select an area and see the content of each node, and other stuff. &lt;a href=&quot;https://github.com/Lilykos/clusterix&quot; rel=&quot;nofollow noreferrer&quot;&gt;Take a look&lt;/a&gt; and see if it works for you!&lt;/p&gt;&#xA;" OwnerUserId="19177" LastEditorUserId="98307" LastEditDate="2020-08-04T22:21:18.710" LastActivityDate="2020-08-04T22:21:18.710" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12010" PostTypeId="1" AcceptedAnswerId="12013" CreationDate="2016-05-31T23:12:11.407" Score="1" ViewCount="1132" Body="&lt;p&gt;Using the R package 'Deducer', I saved a graphic (chart) as an .eps file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can open the .eps file, it's just a bunch of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I re-generate the graphic (chart) from the .eps file (preferably using Deducer or R)?&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="16928" LastEditDate="2016-06-01T09:36:30.863" LastActivityDate="2016-06-01T09:36:30.863" Title="generate graph from .eps file (preferably using R)" Tags="|r|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12011" PostTypeId="2" ParentId="12003" CreationDate="2016-06-01T00:05:28.287" Score="7" Body="&lt;p&gt;&quot;How do you handle being given a dataset, but no clear objective?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the advice above, understand that it is essential to understand the goals of the business you are in, and of your immediate client. Frequently you will need to understand the specific problem that made them turn to data better than they do.&#xA; It is very highly common to be presented with data and an unclear objective from your internal or external client - it will be usually your task to supply a goal that can be achieved with the data and will solve the client's actual business problem. An amount of lateral thinking will be required to make the data outcome and business solution match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would summarise the above as 'defining the objective is too important (and possibly too difficult!) to be left to the client (alone)'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the machine learning context, CRISP-DM is a methodology which tries to solve this problem by iterating through a loop so that additional data understanding can be used in discussion with the client to better understand the original problem. So, for example, they may state a ill-defined goal, a second discussion after you've done some EDA will sharped it a little. When you later produce a model that works well, but isn't on quite the right target, you'll get closer to the real business goal again. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, don't be too disturbed by the fuzziness of the task. Expect to encounter a vaccuum, and fill it to your advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a slight sideways shift, but the six sigma methodology attempts to solve this problem in a different context with the DMAIC system (the 'D' standing for 'Define', in terms of the 'voice of the customer'), so it is probable that some tips can be gleaned in resources for the six sigma context (e.g. exercises you can do with a client that help them express what you want more clearly)&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="18843" LastEditDate="2016-06-01T00:36:34.543" LastActivityDate="2016-06-01T00:36:34.543" CommentCount="1" CommunityOwnedDate="2018-08-02T19:42:51.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="12013" PostTypeId="2" ParentId="12010" CreationDate="2016-06-01T07:12:16.893" Score="1" Body="&lt;p&gt;EPS is &quot;Encapsulated PostScript&quot;. Its meant for embedding like an image in documents, or sending to printers. You can view it with a PostScript document viewer, and there are free PostScript document viewers for Linux, Windows, and Mac OSs. Ghostview, Evince, etc etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So although you can view the graphic once you've got a PostScript document viewer, you cannot load it into R as if you had just plotted it. &lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-01T07:12:16.893" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12014" PostTypeId="1" CreationDate="2016-06-01T07:39:08.340" Score="1" ViewCount="242" Body="&lt;p&gt;I am trying to plot a CDF of 7 files I have. Each file looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&#xA;2&#xA;1.5&#xA;2.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on. The problem is about the sizes of the files, which are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;862M&#xA;1,8G&#xA;2,4G&#xA;18G&#xA;2,0G&#xA;1,8G&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have put up a simple R script which simply loads the files and plot them. When I generate fake files of smaller size, the script works fine. However, it is running since three days and it hasn't produced any output using the full files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The script is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)    &#xA;data &amp;lt;- read.table('file1.csv')&#xA;data$g = &quot;G1&quot;&#xA;data2 &amp;lt;- read.table('file2.csv')&#xA;data2$g = &quot;G2&quot;&#xA;data3 &amp;lt;- read.table('file3.csv')&#xA;data4$g = &quot;G3&quot;&#xA;data4 &amp;lt;- read.table('file4.csv')&#xA;data4$g = &quot;G4&quot;&#xA;data5 &amp;lt;- read.table('file5.csv')&#xA;data5$g = &quot;G5&quot;&#xA;data6 &amp;lt;- read.table('file56.csv')&#xA;data6$g = &quot;G6&quot;&#xA;data7 &amp;lt;- read.table('file7.csv')&#xA;data7$g = &quot;G7&quot;&#xA;&#xA;dftotal = rbind(data,data2)&#xA;dftotal = rbind(dftotal,data3)&#xA;dftotal = rbind(dftotal,data4)&#xA;dftotal = rbind(dftotal,data5)&#xA;dftotal = rbind(dftotal,data6)&#xA;dftotal = rbind(dftotal,data7)&#xA;&#xA;gp &amp;lt;- ggplot(data = dftotal, aes(x = V1), group = factor(g)) + stat_ecdf()&#xA;ggsave('cdf.eps',gp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone knows a more efficient way to do this?&lt;/p&gt;&#xA;" OwnerUserId="19185" LastActivityDate="2016-06-01T19:12:12.497" Title="CDF with a massive amount of data" Tags="|r|bigdata|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12015" PostTypeId="2" ParentId="12014" CreationDate="2016-06-01T09:19:30.807" Score="1" Body="&lt;p&gt;My estimate is that there are around 5.4b numbers in your dataset, all in the range of 0-50 according to your comment. I doubt there are much faster ways to do this on all the data than you are currently doing. However if you would just take every file and get random samples of 0.1-10% of each file and then combine these, you will still get an ecdf that is visually almost exactly the same while reducing memory issues and computing power necessary.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-01T09:19:30.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12016" PostTypeId="2" ParentId="6590" CreationDate="2016-06-01T10:05:00.677" Score="3" Body="&lt;p&gt;Personally, I like to divide feature selection in two:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;unsupervised feature selection&lt;/li&gt;&#xA;&lt;li&gt;supervised feature selection&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Unsupervised feature selection&lt;/strong&gt; are things like clustering or PCA where you select the least redundant range of features (or create features with little redundancy). &lt;strong&gt;Supervised feature selection&lt;/strong&gt; are things like Lasso where you select the features with most predictive power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally usually prefer what I call supervised feature selection. So, when using a linear regression, I would select features based on Lasso. Similar methods exist to induce sparseness in neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But indeed, I don't see how I would go about doing that in a method using kernels, so you are probably better off using what I call unsupervised feature selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; you also asked about regularization. I see regularization as helping mostly because we work with finite samples and so the training and testing distribution will always differ somewhat, and you want your model to not overfit. I am not sure it removes the need to avoid selecting features (if you indeed have too many). I think that selecting features (or creating a smaller subset of them) helps by making the features you do have more robust and avoid the model to learn from spurious correlations. So, regularization does help, but not sure that it is a complete alternative. But I haven't thought thoroughly enough about this.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-06-16T15:21:02.050" LastActivityDate="2016-06-16T15:21:02.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12017" PostTypeId="1" CreationDate="2016-06-01T14:15:50.537" Score="-2" ViewCount="494" Body="&lt;p&gt;I want to do comparison between different factors with box plots. One factor doesn't compare with it self. So I need a matrix of &lt;code&gt;boxplots&lt;/code&gt;  (wich I could get using &lt;code&gt;ggplot&lt;/code&gt;) but without the diagonal. The results is like this but with boxplots. I'm not restricted to &lt;code&gt;R&lt;/code&gt;. I may use &lt;code&gt;python&lt;/code&gt; if suited.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pairs(iris[,1:4])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zZ62Q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zZ62Q.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[EDIT]: here is the data &lt;a href=&quot;http://pastebin.com/ujvnhtNE&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pastebin.com/ujvnhtNE&lt;/a&gt; the labeln-labeln is not needed and I can easily get rid of it. I'd like a box-plot to denote a label vs. another with two shots in it&lt;/p&gt;&#xA;" OwnerUserId="19191" LastEditorUserId="19191" LastEditDate="2016-06-01T19:38:37.570" LastActivityDate="2017-05-01T05:54:50.893" Title="How to get 2D-Array of boxplots without the diagonal with R (python also accepted)?" Tags="|python|r|ggplot2|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12018" PostTypeId="1" AcceptedAnswerId="12021" CreationDate="2016-06-01T16:11:21.190" Score="4" ViewCount="17210" Body="&lt;p&gt;I am currently working on a dataset which contains a &lt;code&gt;name&lt;/code&gt; attribute, which stands for a person's first name. After reading the csv file with &lt;code&gt;read.csv&lt;/code&gt;, the variable is a &lt;code&gt;factor&lt;/code&gt; by default (&lt;code&gt;stringsAsFactors=TRUE&lt;/code&gt;) with ~10k levels. Since &lt;code&gt;name&lt;/code&gt; does not reflect any group membership, I am uncertain to leave it as &lt;code&gt;factor&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it necessary to convert &lt;code&gt;name&lt;/code&gt; to &lt;code&gt;character&lt;/code&gt;? Are there some advantages in doing (or not doing) this? Does it even matter? &lt;/p&gt;&#xA;" OwnerUserId="19113" LastEditorUserId="19113" LastEditDate="2016-06-01T17:16:45.043" LastActivityDate="2020-12-13T08:41:18.043" Title="When to choose character instead of factor in R?" Tags="|r|data-wrangling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12019" PostTypeId="2" ParentId="11769" CreationDate="2016-06-01T16:11:36.683" Score="0" Body="&lt;p&gt;I have run into this problem before, and while I'm still relatively new to the field, I have some thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In answer to your first question, if cross-validating your model yields consistent results, you possibly have enough data.  If new data is considerably throwing your model off, you likely do not have enough data.  There are a number of metrics that can measure the robustness of your model, I have mostly used AUC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an interesting post about more data vs. better algorithms:&#xA;&lt;a href=&quot;http://www.kdnuggets.com/2015/06/machine-learning-more-data-better-algorithms.html&quot; rel=&quot;nofollow&quot;&gt;http://www.kdnuggets.com/2015/06/machine-learning-more-data-better-algorithms.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-06-01T16:11:36.683" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12020" PostTypeId="1" AcceptedAnswerId="12030" CreationDate="2016-06-01T16:22:59.640" Score="2" ViewCount="2379" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GWBEt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GWBEt.png&quot; alt=&quot;Image from Hinton&amp;#39;s course video&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As seen in this image we calculate partial derivative of Error w.r.t output of the output neuron. Shouldn't it be normal derivative? Does not that particular error is determined by only that output?&lt;/p&gt;&#xA;" OwnerUserId="18619" LastActivityDate="2016-06-04T05:28:02.090" Title="Why do we calculate partial derivative of Error w.r.t output of Neural Network during backpropagation?" Tags="|neural-network|backpropagation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12021" PostTypeId="2" ParentId="12018" CreationDate="2016-06-01T16:50:03.700" Score="2" Body="&lt;p&gt;Factors are stored as numbers and a table of levels. If you have categorical data, storing it as a factor may save lots of memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have a vector of length 1,000 stored as character and the strings are all 100 characters long, it will take about 100,000 bytes. If you store it as a factor, it will take about 8,000 bytes plus the sum of the lengths of the different factors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Comparisons with factors should be quicker too because equality is tested by comparing the numbers, not the character values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of keeping it as character comes when you want to add new items, since you are now changing the levels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Store them as whatever makes the most sense for what the data represent. If &lt;code&gt;name&lt;/code&gt; is not categorical, and it sounds like it isn't, then use character.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-01T16:50:03.700" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12022" PostTypeId="2" ParentId="11769" CreationDate="2016-06-01T16:57:21.207" Score="4" Body="&lt;p&gt;There isn't a well established way of estimating the number of data points that you'll need. It's much more an art than a science. As you gain more experience, you'll learn some common sense lessons (in hindsight) along the way. For instance, you should never have more parameters than data points; if you're building random forests, you should not have more trees than data points. If you're doing deep learning, you should never have more neurons than data points (these are extreme examples). As a rule of thumb, try to avoid having 10 times more features than you have data points. 300 data points is very small, and you so you should probably limit yourself to linear models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Hobbes mentioned, cross validation and/or holdout sets are the proper way to judge the readiness of your model irrespective of the datapoint-to-feature ratio. If the error on the training set is 10% better than the error on the test/validation set, you're probably overfitting and your model is not fit for production. The 10% threshold is very much a rule of thumb. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also worth mentioning, once you feel comfortable that your model is fit for production, you should never deploy it and then just walk away. If your model has value, you should do ongoing post production monitoring. In the real world, your model's performance will inevitably degrade over time (regardless of how strong/accurate it was at the time of the model build) as the environment that your model was trained for evolves. This is certainly true for finance and insurance industries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Production monitoring can also be used to pull the plug if your model performs far worse than you anticipated. If you have a very low opinion of the potential stability of your model, deploy it silently (don't allow the predictions to go to downstream systems) and measure how your model would have done had you hypothetically deployed it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A last piece of advice: when you do predictions on new documents you'll nearly always come across words that you have not accounted for in your training set. You should find a way to build this awareness into your model. For example, you could choose to use only keywords as predictors or you could create indicator variables for new words. Either way, you should not expect the vocabulary size to remain the same. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-06-01T17:02:58.460" LastActivityDate="2016-06-01T17:02:58.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12023" PostTypeId="2" ParentId="11812" CreationDate="2016-06-01T17:20:30.357" Score="2" Body="&lt;p&gt;As you (OP) further clarified, you've got equipment sensor data.  Your first time-series was recorded when you knew the machine was in good operating condition.  Later, you sample another time series, and you want to know if anything has changed.  This is called anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You described the time-series as stochastic, meaning there is an element of randomness.  There are several interesting methods you can use to find the probability that the two series were generated by the same process (nothing has changed).  I'm afraid the answers I've read to your question so far would only work in special cases, and are perhaps too complicated for the job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general concept is that there is an underlying probability distribution (described by a 'probability density function' or PDF) which generates your time-series.  When the machine gets out of order, that PDF changes.  You test if the new distribution is different from the old one.  You should use a completely general approach which does not assume a particular form of the PDF (Gaussian, Poisson, Beta, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, you can always &lt;a href=&quot;https://stats.stackexchange.com/questions/30858/how-to-calculate-cumulative-distribution-in-r&quot;&gt;construct and graph the empirical cumulative distribution function&lt;/a&gt; or ECDF for each time-series and compare them visually.  This requires a qualitative judgement on what &quot;different&quot; looks like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it quantitative, a test from classical statistics is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;K-S test&lt;/a&gt;.  It's a one-liner in R.  The disadvantage of this test is, well, it's not Bayesian.  :)  There are &lt;a href=&quot;https://asaip.psu.edu/Articles/beware-the-kolmogorov-smirnov-test&quot; rel=&quot;nofollow noreferrer&quot;&gt;other issues&lt;/a&gt; too.  That being said, I use the K-S test all the time because it's so fast and easy.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many caveats here.  You'll probably want to use the first-differences (changes) of sensor data instead of the values themselves.  If you have more than a few sensors, you'll probably want to use dimensionality reduction such as principal component analysis (PCA) as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I much prefer a Bayesian A-B test.  This requires you to construct models of the PDFs to which your data are applied.   This generates a PDF which most likely explains your data.  In fact, it generates a PDF for each parameter in your PDF.  You look to see how much overlap you have in the parameters.  Not enough overlap means an anomaly.  To get started with Bayesian probability I recommend Python and &lt;a href=&quot;https://pymc-devs.github.io/pymc/README.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PYMC&lt;/a&gt; and the &lt;a href=&quot;https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Hackers book&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-06-01T17:20:30.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12025" PostTypeId="2" ParentId="12014" CreationDate="2016-06-01T19:12:12.497" Score="0" Body="&lt;p&gt;If the frequency per bin is all you need, use awk or write a program in C to calculate that. The execution shouldn't take more than a few tens of minutes at most.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-06-01T19:12:12.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12026" PostTypeId="1" AcceptedAnswerId="12027" CreationDate="2016-06-01T21:01:54.657" Score="1" ViewCount="35123" Body="&lt;p&gt;This is a snippet of the dataset I am currently working on:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sample&#xA;    name sex count&#xA;1  Maria   f    97&#xA;2 Thomas   m    12&#xA;3  Maria   m     5&#xA;4  Maria   f    97&#xA;5 Thomas   m     8&#xA;6  Maria   m     4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to sum up the counts grouped by &lt;code&gt;name&lt;/code&gt; and &lt;code&gt;sex&lt;/code&gt; to finally get this data.frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; result&#xA;    Maria Thomas&#xA;f   194      0&#xA;m     9     20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wrote a simple loop to iterate over the rows and sum up the counts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;result &amp;lt;- matrix(0, nrow=2, ncol=2)&#xA;colnames(result) &amp;lt;- unique(sample$name)&#xA;rownames(result) &amp;lt;- unique(sample$sex)&#xA;&#xA;for (i in 1:nrow(sample)) {&#xA;    sex &amp;lt;- as.character(sample[i,&quot;sex&quot;])&#xA;    name &amp;lt;- sample[i,&quot;name&quot;]&#xA;    count &amp;lt;- sample[i,&quot;count&quot;]&#xA;&#xA;    result[sex, name] &amp;lt;- result[sex, name] + count&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it suitable to do it this way? Are there any other ways to do it in a more elegant / shorter fashion? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already tried it with &lt;code&gt;aggregate&lt;/code&gt;, but the output is in a different format: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; aggregate(sample$count,by=list(sample$name,sample$sex),sum)&#xA;  Group.1 Group.2   x&#xA;1   Maria       m   9&#xA;2  Thomas       m  20&#xA;3   Maria       w 194&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19113" LastEditorUserId="19113" LastEditDate="2016-06-01T21:13:24.697" LastActivityDate="2016-06-03T16:33:16.607" Title="Sum up counts in a data.frame grouped by multiple variables" Tags="|r|data-wrangling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12027" PostTypeId="2" ParentId="12026" CreationDate="2016-06-02T00:35:12.750" Score="2" Body="&lt;p&gt;You can do this using the &lt;code&gt;xtabs&lt;/code&gt; function! Here's how I did it using your example data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Create example data...&#xA;name &amp;lt;- c(&quot;Maria&quot;, &quot;Thomas&quot;, &quot;Maria&quot;, &quot;Maria&quot;, &quot;Thomas&quot;, &quot;Maria&quot;)&#xA;sex &amp;lt;- c(&quot;f&quot;, &quot;m&quot;, &quot;m&quot;, &quot;f&quot;, &quot;m&quot;, &quot;m&quot;)&#xA;count &amp;lt;- c(97, 12, 5, 97, 8, 4)&#xA;data &amp;lt;- data.frame(&quot;name&quot;=name, &quot;sex&quot;=sex, &quot;count&quot;=count)&#xA;&#xA;# Create table...&#xA;xtabs(formula=count~name + sex, data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        sex&#xA;name       f   m&#xA;Maria    194   9&#xA;Thomas     0  20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-06-02T00:35:12.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12028" PostTypeId="2" ParentId="10508" CreationDate="2016-06-02T03:36:11.623" Score="0" Body="&lt;p&gt;You could represent your input with numerical values for each fruit, negative if the fruit is in basket 1, positive if it's in basket 2 and 0 otherwise.  This representation generalizes to larger baskets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The representation will cause some algorithms, such as logistic regression, to take advantage of the symmetry of the training set.  For other algorithms like k-nearest neighbor, you can mirror the training data - for each example, include another example with the identities of the groups reversed - which would let you use separate variables for each fruit-basketID pair.&lt;/p&gt;&#xA;" OwnerUserId="19085" LastEditorUserId="19085" LastEditDate="2016-06-02T03:49:51.590" LastActivityDate="2016-06-02T03:49:51.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12029" PostTypeId="2" ParentId="9195" CreationDate="2016-06-02T06:53:26.780" Score="3" Body="&lt;p&gt;Two points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dropout is also usually compared with neural networks ensembles. It seems it has some of the performance benefits of training and averaging several neural networks.&lt;/li&gt;&#xA;&lt;li&gt;Dropout is easier to calibrate than regularization. There is only one hyperparameter which is the dropout rate and people widely use 0.5 while training (and then 1.0 on evaluation of course :)), see e.g. this &lt;a href=&quot;https://www.tensorflow.org/versions/r0.8/tutorials/mnist/pros/index.html&quot; rel=&quot;nofollow&quot;&gt;TensorFlow example&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Anyhow, I am a little skeptical of neural networks empirical studies. There are just too many hyperparameters to fine tune, from the topology of the network to the gradient descent optimization procedure to activation functions and whatever it is you are testing like regularization. Then, the entire thing is stochastic and usually performance gains are so small that you can hardly statistical test for differences. Many authors do not even bother doing statistical testing. They just average cross-validation and declare whatever model had the highest decimal point gain to be the winner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find a study promoting dropout only to be contradicted by another promoting regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it all boils down to aesthetics preferences. Dropout IMHO sounds more biological plausible than regularization. It also seems easier to calibrate. So, I personally prefer it when using a framework like TensorFlow. If we have to use our own neural network, which we often do, we will use regularization because it was easier to implement.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-06-02T06:53:26.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12030" PostTypeId="2" ParentId="12020" CreationDate="2016-06-02T08:03:08.333" Score="2" Body="&lt;p&gt;The partial derivative is used precisely because it separates concerns about how the value is calculated (from all the other parameters and outputs in the network) from how the value affects the output. This is purely by definition so you can do the calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;normal&quot; derivative of the error function can be expressed as the sum of any &quot;complete&quot; set of independent partial derivatives. There are a few such sets possible - in a simple feed-forward network one for each layer plus one for all the weights combined. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in essence you &lt;em&gt;are&lt;/em&gt; calculating the &quot;normal&quot; derivative w.r.t. the network weights, but due to the nature of the problem this is done by calculating the partial derivatives in multiple steps.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Caveat: I am probably not using those maths terms 100% accurately. For instance, I am ignoring the role of the training data and treating it as a constant.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-06-02T08:03:08.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12031" PostTypeId="1" CreationDate="2016-06-02T08:40:23.567" Score="1" ViewCount="57" Body="&lt;p&gt;Lets say that you have panel data, of &lt;strong&gt;daily consumption of product p1 of 10000 individuals.&lt;/strong&gt; The panel data is on a daily basis for only one month, this means that $t=1,...,30$. &#xA;The question is how do it ? What kind of model I to use and why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My answer iss to have a two-level model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{i,t}=\beta_{i}x_{i,t}+\epsilon_{i,t}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_{i}=\gamma_{i}z_{i} + \eta_{i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x_{i,t}$ would contain my &lt;strong&gt;time variant&lt;/strong&gt; variables and $z_{i}$ will contain all my &lt;strong&gt;time invariant&lt;/strong&gt; variables. My argument was, that by modeling it like this, I believe that I take control of the time dependency between the $y_{i,t}$ and $y_{i,t-1}$ by letting the $\epsilon_{i,t}$ &lt;strong&gt;also depend on time&lt;/strong&gt; and also I capture the heterogeneity of the consumers. Where $\epsilon_{it}\sim N(0,\Sigma_{\epsilon})$ and $\eta_{i}\sim N(0,\sigma_{ \eta _{i}}^2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think ?&lt;/p&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2016-06-02T13:53:00.410" LastActivityDate="2016-06-02T13:53:00.410" Title="Forecasting daily usage of a prouct" Tags="|predictive-modeling|bigdata|regression|time-series|hierarchical-data-format|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12032" PostTypeId="1" AcceptedAnswerId="12140" CreationDate="2016-06-02T09:04:55.753" Score="6" ViewCount="206" Body="&lt;p&gt;In R I have &lt;code&gt;data&lt;/code&gt; where &lt;code&gt;head(data)&lt;/code&gt; gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;day   count   promotion&#xA;1        33        20.8&#xA;2        23        17.1  &#xA;3        19         1.6  &#xA;4        37        20.8    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now &lt;code&gt;day&lt;/code&gt; is simply the day (and is in order). &lt;code&gt;promotion&lt;/code&gt; is the promotion-value for the day. It is simply the number of times an advertisement has been on television. &lt;code&gt;count&lt;/code&gt; is the number of new users we got that day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to investigate the impact the promotion-value has on new users (&lt;code&gt;count&lt;/code&gt;). Since we have a count process I thought it would be best to make a poisson regression model.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=glm(formula= data$count ~ data$promotion, data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When we type &lt;code&gt;summary(model)&lt;/code&gt; we get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Coefficients:&#xA;           (Intercept)              good_users$promotion  &#xA;              13.40216                   0.24342&#xA;&#xA;Degrees of Freedom: 793 Total (i.e. Null);  792 Residual&#xA;Null Deviance:      9484 &#xA;Residual Deviance: 9325     AIC: 12680&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qKM7j.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qKM7j.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I plot the fitted values for the model &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;points(model$promotion, model$fitted, col=&quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;we get this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EHzyO.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EHzyO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is another plot that shows the same but where days with 0 promotion are removed. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/rxq9R.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rxq9R.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I chose my regression model (should I use lm instead of glm) or is the another better approach to solve this? Because the data is not pretty but more random like this what should one do ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updated&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finding the sweet spot&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have done the following for finding a sweet spot.&#xA;I divide &lt;code&gt;data&lt;/code&gt; into 10 groups. &lt;code&gt;group1&lt;/code&gt; is simply a subset where the promotion-value is within &lt;code&gt;1:10&lt;/code&gt;. &lt;code&gt;group2&lt;/code&gt; is data where the promotion-value is between &lt;code&gt;11:20&lt;/code&gt;, and so on for the other groups. So in R we have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group1 &amp;lt;- subset(data, data$promotion %in% 1:10)&#xA;group2 &amp;lt;- subset(data, data$promotion %in% 11:20)&#xA;group3 &amp;lt;- subset(data, data$promotion %in% 21:30)&#xA;...&#xA;group10 &amp;lt;- subset(data, data$promotion %in% 91:100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I can use &lt;code&gt;wilcox.test&lt;/code&gt; to test if there is a significantly difference between the groups by typing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wilcox.test(group2, group1, alternative=&quot;greater&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives a low p-value, ie &lt;code&gt;group2&lt;/code&gt; has significant higher &lt;code&gt;new_good_users&lt;/code&gt; than &lt;code&gt;group1&lt;/code&gt;. The same goes for  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wilcox.test(group3, group2, alternative=&quot;greater&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but for     &lt;code&gt;wilcox.test(group4, group3, alternative=&quot;greater&quot;)&lt;/code&gt;&#xA;I get a p-value at 0.20, ie there is no significant difference in &lt;code&gt;new_good_users&lt;/code&gt; between &lt;code&gt;group4&lt;/code&gt; and &lt;code&gt;group3&lt;/code&gt;. And the same goes for the rest of the group-pairs up to 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this must mean that if we increase &lt;code&gt;promotion&lt;/code&gt; in the first groups we have an increase in &lt;code&gt;new_good_users&lt;/code&gt; but in the last groups we do not have that increase. This means that we have a sweet spot at &lt;code&gt;group3&lt;/code&gt; where the promotion-value is &lt;code&gt;21:30&lt;/code&gt;. Is this not correct ? &lt;/p&gt;&#xA;" OwnerUserId="18787" LastEditorUserId="18787" LastEditDate="2016-06-10T08:29:18.393" LastActivityDate="2016-06-10T08:29:18.393" Title="Regression model for a count proces" Tags="|regression|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="12033" PostTypeId="2" ParentId="11890" CreationDate="2016-06-02T12:58:30.820" Score="0" Body="&lt;p&gt;The equation I ended up using is listed in the link below, provided by Emre:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/10471/linear-regression-with-non-symmetric-cost-function/10474&quot;&gt;Linear regression with non-symmetric cost function?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18951" LastActivityDate="2016-06-02T12:58:30.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12034" PostTypeId="2" ParentId="11977" CreationDate="2016-06-02T17:25:23.797" Score="1" Body="&lt;p&gt;I have been using one heuristic to detect outlier values that is very simple. Start by calculating an interval centered on the mean and with three standard deviations in radius. Next, calculate a second interval using Tukey's box and whiskers measures, placing the interval limits at the whiskers' limits. Finally, calculate the union of both intervals and use this new interval to detect your outliers: any observation outside this interval is a potential outlier. Please note that you may have to tweak both intervals to calibrate for your desired precision. Also, you may have to take into account possible trends and seasonalities in these intervals, depending on how the historic data behaves.&lt;/p&gt;&#xA;" OwnerUserId="3439" LastActivityDate="2016-06-02T17:25:23.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12037" PostTypeId="1" CreationDate="2016-06-03T02:25:25.057" Score="3" ViewCount="274" Body="&lt;p&gt;I currently have a large stream of data, with points such as HTTP request/response codes (200, 404, 500, etc.). Essentially, I want to perform anomaly detection for when too many signals that are NOT 200 are received. This means that the signal to be analyzed is dependent on groupings of data points (i.e., a single data point indicating that a 404 was sent is not good enough, only a cluster of 404's close together with respect to time means something).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good algorithm/methodology to approach this? I was previously thinking of having a moving window counter of the previous codes and basing it off of this, but I feel like the window size for example, is very subjective, and I am not sure how to tune that.&lt;/p&gt;&#xA;" OwnerUserId="19240" LastActivityDate="2016-06-05T14:03:50.887" Title="How do I approach grouped anomaly detection?" Tags="|machine-learning|algorithms|anomaly-detection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12038" PostTypeId="1" CreationDate="2016-06-03T02:52:50.410" Score="-6" ViewCount="1894" Body="&lt;p&gt;i am getting error when i try to run my script. I have stored my data in txt file formar&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;&amp;lt;pyshell#4&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    plt.scatter (x,y)&#xA;  File &quot;C:\Python27\lib\site-packages\matplotlib\pyplot.py&quot;, line 3251, in scatter&#xA;    edgecolors=edgecolors, data=data, **kwargs)&#xA;  File &quot;C:\Python27\lib\site-packages\matplotlib\__init__.py&quot;, line 1812, in inner&#xA;    return func(ax, *args, **kwargs)&#xA;  File &quot;C:\Python27\lib\site-packages\matplotlib\axes\_axes.py&quot;, line 3840, in scatter&#xA;    raise ValueError(&quot;x and y must be the same size&quot;)&#xA;ValueError: x and y must be the same size&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2104,3,399900&#xA;1600,3,329900&#xA;2400,3,369000&#xA;1416,2,232000&#xA;3000,4,539900&#xA;1985,4,299900&#xA;1534,3,314900&#xA;1427,3,198999&#xA;1380,3,212000&#xA;1494,3,242500&#xA;1940,4,239999&#xA;2000,3,347000&#xA;1890,3,329999&#xA;4478,5,699900&#xA;1268,3,259900&#xA;2300,4,449900&#xA;1320,2,299900&#xA;1236,3,199900&#xA;2609,4,499998&#xA;3031,4,599000&#xA;1767,3,252900&#xA;1888,2,255000&#xA;1604,3,242900&#xA;1962,4,259900&#xA;3890,3,573900&#xA;1100,3,249900&#xA;1458,3,464500&#xA;2526,3,469000&#xA;2200,3,475000&#xA;2637,3,299900&#xA;1839,2,349900&#xA;1000,1,169900&#xA;2040,4,314900&#xA;3137,3,579900&#xA;1811,4,285900&#xA;1437,3,249900&#xA;1239,3,229900&#xA;2132,4,345000&#xA;4215,4,549000&#xA;2162,4,287000&#xA;1664,2,368500&#xA;2238,3,329900&#xA;2567,4,314000&#xA;1200,3,299000&#xA;852,2,179900&#xA;1852,4,299900&#xA;1203,3,239500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy import stats&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;&#xA;variable_1 = open ('foo.txt')&#xA;variable_2 = variable_1.readlines ()&#xA;&#xA;x = []&#xA;y = []&#xA;&#xA;for line in variable_2 :&#xA;    line = line.replace ('\n',' ')&#xA;    vals = line.split (',')&#xA;    x.append (float (vals [0]))&#xA;    y.append (float (vals [1]))&#xA;&#xA;gradient, intercept, r_value, p_value, std_err = stats.linregress (x,y)&#xA;print gradient, intercept, r_value, p_value, std_err&#xA;plt.scatter (x,y)&#xA;plt.plot (x, (gradient*x + intercept))&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19242" LastEditorUserId="19242" LastEditDate="2016-06-04T12:43:39.357" LastActivityDate="2016-06-04T12:43:39.357" Title="i want to build a linear regression" Tags="|machine-learning|python|scikit-learn|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12039" PostTypeId="1" CreationDate="2016-06-03T03:00:04.193" Score="2" ViewCount="156" Body="&lt;p&gt;I have a series of houseprices that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    410000 &#xA;1    610000 &#xA;2    579950 &#xA;3    559950 &#xA;4    404950 &#xA;Name: price, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And using &lt;code&gt;plt.hist(df.price)&lt;/code&gt; I get the below histogram, which is not useful. What is causing this? Am I missing something here? I want a range of prices to be binned together and their frequencies to be plotted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/h6wii.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h6wii.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19241" LastActivityDate="2016-09-01T12:15:32.030" Title="Why is this series of house prices being thrown into one bin with plt.hist?" Tags="|python|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12040" PostTypeId="2" ParentId="2504" CreationDate="2016-06-03T06:37:15.063" Score="8" Body="&lt;p&gt;On the lines of what @Simon has already said:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Deep learning approaches have been particularly useful in solving problems in vision, speech and language modeling where feature engineering is tricky and takes a lot of effort.&lt;/li&gt;&#xA;&lt;li&gt;For your application that does not seem to be the case since you have well defined features and only feature interactions etc. are required.&lt;/li&gt;&#xA;&lt;li&gt;Given that deep learning models currently need a lot of computing resources and scientist time in coding stuff up I'd suggest opting for a non-deep learning approach.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For your problem the &lt;em&gt;effort vs benefit&lt;/em&gt; tradeoff does not seem to be in deep learning's favour. &lt;strong&gt;DL would be an overkill&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-06-03T06:37:15.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12041" PostTypeId="2" ParentId="12039" CreationDate="2016-06-03T08:32:34.050" Score="2" Body="&lt;p&gt;Can't reproduce:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; housing = pd.DataFrame([410000, 610000, 579950, 559950, 404950],&#xA;                           columns=['price'], dtype=int)&#xA;&amp;gt;&amp;gt;&amp;gt; housing.price&#xA;0    410000&#xA;1    610000&#xA;2    579950&#xA;3    559950&#xA;4    404950&#xA;Name: price, dtype: int64&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; plt.hist(housing.price)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vv7GR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vv7GR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Resolution&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;You have an outlier at 1e8 while most observations fall into 1e7 range.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="15527" LastEditDate="2016-06-03T10:15:10.857" LastActivityDate="2016-06-03T10:15:10.857" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12042" PostTypeId="1" CreationDate="2016-06-03T10:54:33.320" Score="5" ViewCount="4422" Body="&lt;p&gt;Google recently &lt;a href=&quot;https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html&quot; rel=&quot;nofollow&quot;&gt;announced the TPU custom chip&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They stated it is available in Google Cloud Platform, but only for their internal usage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When will I be able to use it? Will it be possible to buy this chip or will it be only accessible through their cloud?&lt;/p&gt;&#xA;" OwnerUserId="19172" LastEditorUserId="19172" LastEditDate="2016-06-06T13:52:41.863" LastActivityDate="2016-06-06T13:52:41.863" Title="Google TPU: when/how will it be available to me?" Tags="|deep-learning|google|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12043" PostTypeId="2" ParentId="11010" CreationDate="2016-06-03T12:22:56.153" Score="1" Body="&lt;p&gt;What you try to do is called multiclass and multilabel text classification. Check the tutorials &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="98307" LastEditDate="2020-08-04T07:40:54.433" LastActivityDate="2020-08-04T07:40:54.433" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12044" PostTypeId="2" ParentId="12042" CreationDate="2016-06-03T13:47:45.843" Score="4" Body="&lt;p&gt;Google has apparently said it would not be available for sale outside of Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.forbes.com/sites/tiriasresearch/2016/05/26/google-builds-its-first-chip-just-for-machine-learning/#58ab2bcb7a92&quot; rel=&quot;nofollow&quot;&gt;http://www.forbes.com/sites/tiriasresearch/2016/05/26/google-builds-its-first-chip-just-for-machine-learning/#58ab2bcb7a92&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google also did mention it would be available to developers for Tensorflow and Cloud machine learning but otherwise it has been light on details or when this will become available.&lt;/p&gt;&#xA;" OwnerUserId="16117" LastActivityDate="2016-06-03T13:47:45.843" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12045" PostTypeId="2" ParentId="9525" CreationDate="2016-06-03T14:47:20.060" Score="3" Body="&lt;p&gt;word2vec treats words as atoms. To get meaningful vectors for unknown words, you either have to&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;change what these atoms are, e.g. switch to letter n-grams as in jamesmf's answer, or&lt;/li&gt;&#xA;&lt;li&gt;use a different model that explicitly looks at what is inside your words, e.g. the &lt;a href=&quot;https://github.com/Leonard-Xu/CWE&quot; rel=&quot;nofollow noreferrer&quot;&gt;CWE model&lt;/a&gt; is easy to use.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="19255" LastEditorUserId="29169" LastEditDate="2021-02-18T20:48:07.957" LastActivityDate="2021-02-18T20:48:07.957" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12046" PostTypeId="2" ParentId="6508" CreationDate="2016-06-03T15:08:17.383" Score="7" Body="&lt;p&gt;Just posting a summary of above comments and some more thoughts so that this question is removed from &quot;unanswered questions&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image_doctor's comment is right that these graphs are typical for k-means. (I am not familiar with the &quot;Silhouette&quot; measure though.) The in-cluster variance is expected to go down continuously with increasing k. The elbow is where the curve bends the most. (Maybe think &quot;2nd derivative&quot; if you want something mathematical.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, it is best to pick k using the final task. Do not use statistical measures of your cluster to make your decision but use the end-to-end performance of your system to guide your choices. Only use the statistics as a starting point.&lt;/p&gt;&#xA;" OwnerUserId="19255" LastEditorUserId="19255" LastEditDate="2017-01-13T16:25:29.090" LastActivityDate="2017-01-13T16:25:29.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12047" PostTypeId="2" ParentId="11813" CreationDate="2016-06-03T16:09:04.427" Score="2" Body="&lt;p&gt;As you seem to plan to build your own character classifier, I assume that you are happy to do some programming but just want to avoid heavy graphics stuff. A practical approach would be to use tesseract's bounding boxes just as a guide. Look to the left and right for more characters. Knowing where the rows are and what height and width the characters are is a good basis. An extension of rows gives you the upper and lower boundaries of the new candidate boxes. To find the left and right boundaries, you can project the pixels to the baseline of the row and look for positions with a low number of black pixels. A second factor for the boundary decision can be the expected width, which should be somewhere between the width for &quot;I&quot; and &quot;W&quot;. A third factor can be the letter/number that your classifier recognises and how well this fits into the context (e.g. use character language model).&lt;/p&gt;&#xA;" OwnerUserId="19255" LastActivityDate="2016-06-03T16:09:04.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12048" PostTypeId="2" ParentId="11813" CreationDate="2016-06-03T16:13:30.917" Score="0" Body="&lt;p&gt;A simple solution might be to reduce the image resolution to hide details such as the small deformations such as in the &quot;O&quot; in &quot;model&quot;. Half the resolution should still be enough.&lt;/p&gt;&#xA;" OwnerUserId="19255" LastActivityDate="2016-06-03T16:13:30.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12049" PostTypeId="2" ParentId="12026" CreationDate="2016-06-03T16:33:16.607" Score="1" Body="&lt;p&gt;Using &lt;code&gt;data.table&lt;/code&gt; is also another option you can explore. Working with data.tables is more efficient when you do certain operations on your table. Its simple to use as well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(data.table) &#xA;DT &amp;lt;- data.table(data) &#xA;DT[ , .(Totalcount = sum(count)), by = .(name,sex)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;output&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     name sex Totalcount&#xA;1:  Maria   f        194&#xA;2: Thomas   m         20&#xA;3:  Maria   m          9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19257" LastActivityDate="2016-06-03T16:33:16.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12050" PostTypeId="1" AcceptedAnswerId="14343" CreationDate="2016-06-03T16:38:52.037" Score="8" ViewCount="9604" Body="&lt;p&gt;Golang is one of my favourite languages and I want to use it for a personal NLP/ML project. Is golang's ecosystem good and rich enough for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an alternative package for nltk in golang?&lt;/p&gt;&#xA;" OwnerUserId="19258" LastEditorUserId="29575" LastEditDate="2018-09-19T02:23:57.040" LastActivityDate="2018-09-19T02:23:57.040" Title="Is there an alternative to nltk in golang?" Tags="|nlp|nltk|software-recommendation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12051" PostTypeId="2" ParentId="12037" CreationDate="2016-06-03T17:29:21.223" Score="1" Body="&lt;p&gt;It seems that your solution does not have to include an &quot;anomaly detection&quot; in context of data analysis, but can more easily be solved with an &lt;strong&gt;&lt;em&gt;iteration&lt;/em&gt;&lt;/strong&gt; through the vector assuming a chronological order of the data (you did not mention any other feature so the problem is &lt;strong&gt;&lt;em&gt;1D&lt;/em&gt;&lt;/strong&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest the following approaches: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Iterate through the vector and if you find N &gt; threshold non-2xx responses in a row, mark starting index of a &quot;cluster&quot;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    vector&amp;lt;int&amp;gt; status, indices;&#xA;    int threshold = 10, counter = 0, flag = 0;&#xA;    status = read_data(&quot;data.csv&quot;);&#xA;    for(int i = 0; i &amp;lt; status.size(); i++) {&#xA;            if (status[i] &amp;lt; 200 &amp;amp;&amp;amp; status[i] &amp;gt; 299) {&#xA;                    if (++counter &amp;gt;= threshold &amp;amp;&amp;amp; !flag) {&#xA;                            indices.push_back(i - counter);&#xA;                            flag = 1;&#xA;                    }&#xA;            } else {&#xA;                    counter = 0;&#xA;                    flag = 0;&#xA;            }&#xA;    }&#xA;    return indices;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2) Once you cross over some non-2xx example, initialize the weight to, let's say 0.0 (zero). For every new negative example, subtract, let's say 0.2 and for a positive example add 0.05 or something and if you end up &lt;strong&gt;bellow -2.0 or above 2.0&lt;/strong&gt;, reset the procedure and in case that it's &amp;lt; -2.0, return the starting index. You can play around with given numbers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Some more complex strategy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But yes, your solution does not have to include a machinery, try to start from something &lt;strong&gt;&lt;em&gt;more simple&lt;/em&gt;&lt;/strong&gt; and it will lead you in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="19260" LastEditorUserId="19260" LastEditDate="2016-06-05T14:03:50.887" LastActivityDate="2016-06-05T14:03:50.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12052" PostTypeId="2" ParentId="12037" CreationDate="2016-06-03T21:24:30.143" Score="1" Body="&lt;p&gt;Look at the &lt;em&gt;rate&lt;/em&gt; instead of single requests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is probably a &quot;usual&quot; frequency of non-success codes. If you observe an unusual frequency, you have an event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a simple algorithm: Consider that you want to raise an alarm if you observe a rate much higher than, say 5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add some margin, will increase this to 0.1, and a threshold of about 10 errors &quot;too much&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a counter x, initially 0, updated using&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;x_{t+1} \leftarrow\begin{cases}&#xA;\max(0, x_t - 0.1) &amp;amp; \text{if request successful} \\&#xA;x_t + 1 &amp;amp; \text{if request causes an error}&#xA;\end{cases}&#xA;$$&#xA;then alert if this value reaches 10.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2016-06-03T21:30:52.703" LastActivityDate="2016-06-03T21:30:52.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12053" PostTypeId="1" CreationDate="2016-06-03T21:57:04.387" Score="5" ViewCount="895" Body="&lt;p&gt;I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/7NpyH.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7NpyH.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;datetime;location;title;description&#xA;2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...&#xA;2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. =&gt; assume the nearby words are a date, etc.). But this would require lots of work for every different press release.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What general techniques do you see to address such a big (and difficult topic) concern ? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there open-source projects dedicated to that?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some Python libraries that would make this task easier?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any general idea?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="19261" LastActivityDate="2019-01-29T08:01:49.147" Title="A few ideas to parse &quot;events&quot; from a text document" Tags="|data-mining|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12055" PostTypeId="1" CreationDate="2016-06-03T23:40:56.187" Score="0" ViewCount="372" Body="&lt;p&gt;How can one use &lt;a href=&quot;https://en.wikipedia.org/wiki/Unstructured_data&quot; rel=&quot;nofollow&quot;&gt;unstructured data&lt;/a&gt; for forecasting purposes? (by unstructured, I mean unstructured in the database sense). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a forecasting system that uses historical data and a set of additional explanatory features to forecast demand. In terms of databases, this is as structured as data can get. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard mentioned in several places that big data and unstructured data can be used to improve demand forecasts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I dig into the details, it seems that they always end up reducing the unstructured data to some structured format first, then feed it to whatever forecasting method they are using. They aren't using &quot;true unstructured data&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can one use true unstructured data in a forecasting algorithm? Any publications, references on this?  &lt;/p&gt;&#xA;" OwnerUserId="8517" LastEditorUserId="8517" LastEditDate="2016-06-13T17:51:56.053" LastActivityDate="2017-12-28T23:20:20.890" Title="How can one use unstructured data for forecasting?" Tags="|reference-request|forecast|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12057" PostTypeId="2" ParentId="11934" CreationDate="2016-06-04T01:26:46.420" Score="1" Body="&lt;p&gt;You can try using lasso which does L1 regularization on the weights and sets the irrelevant parameters to 0. Among the a subset of correlated parameters, there only one will be choose. This works particularly well if you suspect that there are multi-collinearity in your dataset. &lt;/p&gt;&#xA;" OwnerUserId="19263" LastActivityDate="2016-06-04T01:26:46.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12058" PostTypeId="2" ParentId="12020" CreationDate="2016-06-04T05:28:02.090" Score="1" Body="&lt;p&gt;I assume that by &quot;normal derivative&quot; you mean $\frac{dE}{dy_j}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That doesn't make sense here. When you compute the derivative $\frac{dY}{dX}$ you're asking for the limiting ratio $\frac{\Delta Y}{\Delta X}$ as you make $\Delta X$ small. This only makes sense if there actually is a limit, ie. if this ratio approaches a constant as you make $\Delta X$ small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case it is usually possible to find many different deltas we could apply to the weights, all of which produce the same delta on $y_j$. (In general, if there are N weights, there is an $(N-1)$-dimensional space of weight deltas, all of which have the same effect on $y_j$.) Many of these different weight deltas will have different effects on your final $E$ (via the other $y_i$'s). This means that there is no fixed limit that the ratio $\frac{\Delta E}{\Delta y_j}$ tends towards. Different weight deltas will result in wildly different values for this ratio. And so there is no &quot;normal derivative&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to find how a delta for each weight individually modifies the final $E$. That tells you how much to update them. This is precisely what the partial derivative is intended to tell you.&lt;/p&gt;&#xA;" OwnerUserId="19265" LastActivityDate="2016-06-04T05:28:02.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12059" PostTypeId="1" CreationDate="2016-06-04T05:36:50.047" Score="0" ViewCount="239" Body="&lt;p&gt;Another beginner question: I'm trying to do PCA on compositional data. In other words, all the variables in the group add up to 100%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've since learned on this forum that compositional data poses a problem for linear regression. Will compositional data also pose a unique problem when doing PCA?&lt;/p&gt;&#xA;" OwnerUserId="19266" LastActivityDate="2016-09-03T23:51:03.263" Title="Principal components analysis with compositional data" Tags="|predictive-modeling|beginner|dimensionality-reduction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12061" PostTypeId="1" AcceptedAnswerId="12107" CreationDate="2016-06-04T11:37:38.327" Score="1" ViewCount="69" Body="&lt;p&gt;Tom White's 'Hadoop: the Definitive Guide' has become a popular guide to the entire Hadoop ecosystem and earned a reputation as providing both a broad survey, as well as covering individual aspects of Hadoop in decent depth. Has anyone thus far attempted to provide the Spark equivalent?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-07T15:33:58.620" Title="Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'?" Tags="|bigdata|apache-hadoop|apache-spark|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12062" PostTypeId="2" ParentId="12038" CreationDate="2016-06-04T12:25:46.013" Score="1" Body="&lt;p&gt;Python does &lt;em&gt;not&lt;/em&gt; vectorise calculations like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot (x, (gradient*x + intercept))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Did you think this was R code? :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to loop over &lt;code&gt;x&lt;/code&gt; to compute the new values. Use an implicit loop:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot (x, [gradient*z + intercept for z in x])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-04T12:25:46.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12063" PostTypeId="1" CreationDate="2016-06-04T18:44:29.820" Score="0" ViewCount="203" Body="&lt;p&gt;I have been trying to find the correlation b/w the following type of temporal data for quite some time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DataSet: &lt;code&gt;A A B A B C B C B A B C X Y X Y Z X Y X Z Y Z A B A B C B&lt;/code&gt; (The actual dataset has 1000's of unique points)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to find some sort of similarity value b/w these points (my points I mean 'A', 'B', 'X', etc). For example similarity b/w 'A' and 'B' in above dataset is high so similarity('A', 'B') should return me a high value and similarity('A', 'X') should return a low value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate the similarity so that I can cluster the similar points. (For above dataset, possible good clusters are &lt;code&gt;{A, B, C}, {X, Y, Z}&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Till now I have tried the following approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I tried to find segments of data whose &lt;code&gt;correlation = len(segment)/unique(points in segment)&lt;/code&gt; value is high. If the value is high than that means in this segment the same type of points are coming back and forth and therefore highly correlated. The problem with this correlation formula is that it is not robust enough. For example &lt;code&gt;Dataset is: X Y X X X X X Y X&lt;/code&gt;. Now the correlation value for subsegment with starting index = 3 and ending index = 7 is higher (&lt;code&gt;correlation = 5/1&lt;/code&gt;) than the correlation value of the whole Dataset i.e 1 to end (&lt;code&gt;correlation = 9/2&lt;/code&gt;). Now due to such problems I have to do a lot of manual settings, like select a segment having more unique points if correlation diff b/w those segments is less than some manually selected value (tradeoff b/w correlation value and unique points), and this does not work when I have 1000's of unique points.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow&quot;&gt;Word2Vec&lt;/a&gt;: I ran this algorithm on the whole data as a single sentence (i.e merged the data in a single sentence separated by space). I don't exactly understand this algorithm but it seems to not work on a single sentence. It gives &gt; 0.99 similarity b/w almost all points. I don't know why is this not working since it is supposed to learn the similarity b/w words based on how they are occurring in a sentence. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How should I approach this clustering problem? Can I tweak word2vec to fit for my problem? A better mathematical formula? Any other way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I am doing something wrong.&lt;/p&gt;&#xA;" OwnerUserId="19278" LastActivityDate="2017-04-23T22:41:01.487" Title="Clustering of Temporal Data" Tags="|clustering|word2vec|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12067" PostTypeId="1" CreationDate="2016-06-05T03:01:24.253" Score="1" ViewCount="207" Body="&lt;p&gt;If I understand the Databricks philosophy correctly, Spark will soon be heavily moving toward dataframes, i.e. away from the usual map/reduce on RDDs. I was wondering if there are any good suggestions for online courses or books that introduce Spark from the dataframe point of view? I know Databricks has a good number of resources on dataframes but I would really like to see some more detailed courses.&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2016-08-05T10:14:19.073" Title="Spark DataFrame courses" Tags="|apache-spark|education|pyspark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12068" PostTypeId="1" AcceptedAnswerId="12071" CreationDate="2016-06-05T05:09:15.123" Score="4" ViewCount="96" Body="&lt;p&gt;Given: &#xA;Time series data collected from sensors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an unexpected gradual drop in the initial data when sensors are idle.&#xA;However, this drop is not so visible when sensors are active because the drop is masked by the actual measurements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Main question: How do I eliminate this drop from the sensor output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure if this is an accurate assumption, but let's assume that the drop is a convolution of an unknown system X (leak, sensor drift etc.) with the system being measured G.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am feeding this into a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I get an approximation to the unknown influence X (from initial data)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal: Apply (-X) to all other data collected from G before training the neural net and expect an increase in performance.&#xA;Should I expect an increase in performance?&lt;/p&gt;&#xA;" OwnerUserId="20287" LastActivityDate="2017-01-31T11:47:21.133" Title="Account for unknown error in time series data" Tags="|machine-learning|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12069" PostTypeId="1" AcceptedAnswerId="12070" CreationDate="2016-06-05T06:27:09.147" Score="1" ViewCount="664" Body="&lt;p&gt;How can we extract information from time-stamp variable for modelling? I have a variable with format mm-dd-yyyy hh:mm:ss I want to predict an outcome variable using time-stamp as input variable. I do not think i can directly use this column for modelling and will need to do some transformation before i can use this in model. Not sure what type of transformation i need to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have date field, you can create dummy variables of day o week and use that as input variable in model. Not sure how to proceed in the case of time-stamp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help?&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2016-06-05T06:56:02.810" Title="Time-stamp for linear model" Tags="|time-series|linear-regression|feature-extraction|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12070" PostTypeId="2" ParentId="12069" CreationDate="2016-06-05T06:56:02.810" Score="2" Body="&lt;p&gt;Welcome to Datascience.SE!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like you said, you can extract the day of the week. Also extract the hour of the day, then encode these two variables using sines and cosines with their respective periodicities (7 and 24). Also create a column for the UNIX/epoch time. If there are &quot;special days&quot;, such as holidays or sales, create boolean columns for them too.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-06-05T06:56:02.810" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12071" PostTypeId="2" ParentId="12068" CreationDate="2016-06-05T07:27:32.737" Score="1" Body="&lt;p&gt;I would set up a sensor as idle, purposefully to determine the decay in the signal. Then try an &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;nofollow&quot;&gt;ARIMA&lt;/a&gt; or similar model on the data collected to find out what an approximation of the decay function is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further to this you should determine if the model still fits if they decay is put into a &quot;stop/start&quot; mode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Afterwards you can apply a corrective function to remove or take into account the decay on your other data.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-06-05T07:27:32.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12072" PostTypeId="2" ParentId="12053" CreationDate="2016-06-05T08:55:17.790" Score="1" Body="&lt;p&gt;These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think your best bet is to use a natural language processing toolkit and do named entity recognition. See &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nltk.org/book/ch07.html&lt;/a&gt; &quot;5 Named entity recognition&quot;. Or see the recommendation here: &lt;a href=&quot;https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data&quot;&gt;https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like &quot;headline, enumitem1, enumitem2, ...&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.&lt;/p&gt;&#xA;" OwnerUserId="723" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-06-05T08:55:17.790" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12073" PostTypeId="1" AcceptedAnswerId="12074" CreationDate="2016-06-05T10:02:27.790" Score="3" ViewCount="1702" Body="&lt;p&gt;I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the two classes now. Then I will use a neural network to classify them into positive and negative classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with the dataset is that it posses a lot of redundant data (Retweets basically). Manually deleting them is not possible and I have tried to find a programmable solution via the Tweepy API but couldn't find any.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is do I need to get rid of these redundant records or should i leave them as is? &lt;/p&gt;&#xA;" OwnerUserId="20288" LastActivityDate="2016-06-05T12:18:19.293" Title="Redundancy - is it a big problem?" Tags="|machine-learning|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12074" PostTypeId="2" ParentId="12073" CreationDate="2016-06-05T10:26:21.580" Score="2" Body="&lt;p&gt;Assuming you want to learn sentiment this is a problem. What happens when you feed this to a Machine Learning algorithm is that it will give more weight to the tweets that are in there multiple times, while not learning more information. It's unlikely that a tweet that has been retweeted 10 times carries more significant information about sentiment than one that hasn't been retweeted. What you could do is add the number of times the tweet was in the set as a feature to your sentiment model, it's possible something can be learned from that fact (maybe positive tweets are retweeted more often), but you should keep it at 1 row for every distinct tweet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting rid of these redundant records should not be difficult programmatically though. I don't know what language you are using but if you only consider the body of the tweet (the content) you could iterate over your tweets, keep a list of all the unique bodies, combined with other meta information (like user, labeled sentiment) and if the content of the next tweet is already in there, just do not add it. Look for 'distinct' functionality as opposed to redundant, enough information out there.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-06-05T12:18:19.293" LastActivityDate="2016-06-05T12:18:19.293" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12075" PostTypeId="1" CreationDate="2016-06-05T12:14:20.250" Score="0" ViewCount="69" Body="&lt;p&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&lt;/a&gt;&#xA;&lt;a href=&quot;https://github.com/apache/spark/blob/master/data/mllib/als/test.data&quot; rel=&quot;nofollow&quot;&gt;https://github.com/apache/spark/blob/master/data/mllib/als/test.data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find missing value in a matrix based on past matrices&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I find that the test.data are not matrices, instead a long list of 3 columns&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I understand test.data and create my own data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which value represent missing value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and how to understand the result, which value represent result?&lt;/p&gt;&#xA;" OwnerUserId="20293" LastEditorUserId="16284" LastEditDate="2016-06-05T15:21:05.677" LastActivityDate="2016-06-05T15:21:05.677" Title="how to create input file for collaborative filtering" Tags="|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12076" PostTypeId="2" ParentId="12075" CreationDate="2016-06-05T12:20:45.637" Score="2" Body="&lt;p&gt;This data set is in a sparse format. Most ratings are unknown, which is a common scenario in recommender systems. This data format is much smaller due to leaving out all the unknowns. You can interpret this data as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user, item, rating&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What this means is that user U has rated item I with rating R. Every UxI combination not in your dataset is unknown and has to be predicted using collaborative filtering.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-05T12:20:45.637" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12077" PostTypeId="1" AcceptedAnswerId="12079" CreationDate="2016-06-05T14:17:03.993" Score="0" ViewCount="94" Body="&lt;p&gt;I am trying to parse the files using Stanford nlp in Spark in mapper function. How to set the number of mappers in Apache Spark? Please help me.&lt;/p&gt;&#xA;" OwnerUserId="9322" LastActivityDate="2017-10-14T11:37:35.530" Title="Apache Spark Question" Tags="|bigdata|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12078" PostTypeId="1" CreationDate="2016-06-05T14:59:44.547" Score="7" ViewCount="157359" Body="&lt;p&gt;I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      country          loan_amount          gender &#xA;1      Austia              175                F        &#xA;2      Austia              100                F        &#xA;3      Austia              825                M        &#xA;4      Austia              175                F        &#xA;5      Austia             1025                M        &#xA;6      Austia              225                F        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I need to group by countries and then for each country, I need to calculate loan percentage by gender in new columns, so that new columns will have male percentage of total loan amount for that country and female percentage of total loan amount for that country. I need to do two group_by function, first to group all countries together and after that group genders to calculate loan percent.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Total loan amount = 2525&#xA;     female_prcent = 175+100+175+225/2525 = 26.73&#xA;     male_percent = 825+1025/2525 = 73.26&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output should be as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      country            female_percent   male_percent &#xA;1      Austia              26.73            73.26        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to do this in R. I tried the below function, but my R session is not producing any result and it is terminating.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  df %&amp;gt;%&#xA;  group_by(country, gender) %&amp;gt;%&#xA;  summarise_each(funs(sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could someone help me in achieving this output? I think this can be achieved using dplyr function, but I am struck inbetween. &lt;/p&gt;&#xA;" OwnerUserId="5043" LastActivityDate="2023-07-11T01:03:31.463" Title="How to group by multiple columns in dataframe using R and do aggregate function" Tags="|r|dataset|aggregation|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12079" PostTypeId="2" ParentId="12077" CreationDate="2016-06-05T15:15:04.333" Score="1" Body="&lt;p&gt;It automatically determines the amount of mappers by the number of partitions your data is in. You can call getNumberPartitions on your data source (RDD/DataFrame) to see how much it is and use repartition for scaling this up or coalesce to scale this down (you can use repartition for this as well but this is slower). Repartitioning is expensive however and should be avoided when unneccesary.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-05T15:15:04.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12080" PostTypeId="2" ParentId="12078" CreationDate="2016-06-05T16:13:20.727" Score="4" Body="&lt;p&gt;I am sure there are better ways of doing it. Below is my simplistic take.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr); library(reshape2)&#xA;Summary &amp;lt;- df %&amp;gt;%&#xA;  group_by(country, gender) %&amp;gt;%&#xA;  summarise(Net = sum(loan_amount))&#xA;&#xA;  final &amp;lt;- recast(Summary, country~gender, id.var = c(&quot;country&quot;, &quot;gender&quot;))&#xA;  final &amp;lt;- mutate(final, F_percent = final$F/(final$F+final$M), M_percent = final$M/(final$F+final$M) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Naming the columns with better names and retaining or dropping certain columns should now be easy.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2016-06-05T16:13:20.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12081" PostTypeId="2" ParentId="12078" CreationDate="2016-06-05T16:55:24.383" Score="7" Body="&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;library(tidyr)     &#xA;df %&amp;gt;% group_by(country, gender) %&amp;gt;% &#xA;       summarise(total_loan_amount =sum(loan_amount)) %&amp;gt;% &#xA;       spread(gender, total_loan_amount) %&amp;gt;% &#xA;       ungroup() %&amp;gt;%&#xA;       transmute(country = country,  female_percent = F / (F+M), male_percent = M /(F+M))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;results in&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Source: local data frame [1 x 3]&#xA;&#xA;      country female_percent male_percent&#xA;       (fctr)          (dbl)        (dbl)&#xA;    1  Austia      0.2673267    0.7326733&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18510" LastEditorUserId="18510" LastEditDate="2016-06-05T17:03:27.493" LastActivityDate="2016-06-05T17:03:27.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12082" PostTypeId="1" AcceptedAnswerId="12083" CreationDate="2016-06-05T18:19:56.917" Score="1" ViewCount="140" Body="&lt;p&gt;I originally had a wide CSV dataset of about 18000 columns(and about 80 rows) that I am trying to read in R. It was stored in an Excel sheet,which unfortunately has a limit of only 16384 columns. Hence, taking the dimension I obtain: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dim(train_set)&#xA;[1]    83 16384&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e 1000+ columns are getting eaten up ,and this would badly affect the accuracy of the predictions. How can I read all the columns in R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your suggestions are much appreciated. Thanks a lot!&lt;/p&gt;&#xA;" OwnerUserId="20302" LastEditorUserId="10024" LastEditDate="2016-06-08T08:22:17.820" LastActivityDate="2016-06-08T08:22:17.820" Title="Reading a wide dataset in R" Tags="|r|predictive-modeling|csv|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12083" PostTypeId="2" ParentId="12082" CreationDate="2016-06-05T18:35:20.757" Score="1" Body="&lt;p&gt;Refer to my comment, I believe what you need is  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- read.csv(&quot;./yourpath/yourfile&quot;, sep = &quot;;&quot;, header = TRUE) # play around with the arguments per your file.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2016-06-05T18:35:20.757" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12084" PostTypeId="2" ParentId="12067" CreationDate="2016-06-05T19:51:25.830" Score="1" Body="&lt;p&gt;edX offers some spark courses. Try the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x?gclid=CjwKEAjwm8-6BRDgnb-Dk96UmRASJADbMycYmMKC3xWs_3jGV9pUd0k3gxIbQRHE2H4_hlXeNMw1BhoCqHjw_wcB&quot; rel=&quot;nofollow&quot;&gt;https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x?gclid=CjwKEAjwm8-6BRDgnb-Dk96UmRASJADbMycYmMKC3xWs_3jGV9pUd0k3gxIbQRHE2H4_hlXeNMw1BhoCqHjw_wcB&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe this is through a partnership with Databricks&lt;/p&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2016-06-05T19:51:25.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12088" PostTypeId="1" AcceptedAnswerId="12094" CreationDate="2016-06-06T07:48:52.353" Score="0" ViewCount="117" Body="&lt;p&gt;&lt;br&gt;&#xA;I am kind of new to the data mining subject but i need help to choose a learning algorithm for my application:&lt;br&gt;&#xA;The problem: identifying that a certain curve or data set belongs to a certain fault in a Component.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;My training data should be like this:&lt;br&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Motor Current Values:&lt;br&gt;&#xA;[0.5,0.6,...,0.4] -&gt; Fault in Komponent 1&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0.2,0.3,...,0.4] -&gt; Fault in Komponent 3&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1,0.7,...,0.4] -&gt; Fault in Komponent 2&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;br&gt;&#xA;.&lt;br&gt;&#xA;.&lt;br&gt;&#xA;[0.8,0.7,...,0.3] -&gt; Fault in Komponent 3&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And i was wondering if i should using a &lt;strong&gt;cluster analysis&lt;/strong&gt; (k-means and save centroid centers then compute the distance of each new entry then give a fuzzy estimation to where which cluster my data might belong to).&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Decision Tree&lt;/strong&gt; algorithm(entropy of the values and stuff), Or should i calculate a &lt;br&gt;distance between the Nominal data (Healthy(No fault) Curve) and the faulty &lt;br&gt;data and play on a more simple basic decision tree with thresholds ?&lt;br&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Proceed with a &lt;strong&gt;peak analysis&lt;/strong&gt; and count the number of peaks within my data and add it to my learning algorithm.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And when should i do a &lt;strong&gt;data pre-processing&lt;/strong&gt; like normalization ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an example of my entry parameters : &lt;br&gt;[Speed(mm/s), Motor Current Values(Ampere), DistanceToNominalState(Ampere)]&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what my data looks like.&lt;a href=&quot;https://i.stack.imgur.com/Eiwt8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Eiwt8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions ? &lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="20311" LastActivityDate="2016-06-06T13:44:17.817" Title="Choosing the correct learning algorithm" Tags="|machine-learning|python|data-mining|clustering|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12089" PostTypeId="2" ParentId="12067" CreationDate="2016-06-06T09:04:53.623" Score="0" Body="&lt;p&gt;Alex, you can check in this Coursera Big Data Course a chapter about Spark DataFrames in &quot;Week 5&quot;, it's an introduction but it explains how to setup PySpark for DataFrames, may be for your situation it would be a good start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/bigdata-analytics&quot; rel=&quot;nofollow&quot;&gt;Big Data Analytics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&lt;/p&gt;&#xA;" OwnerUserId="18973" LastActivityDate="2016-06-06T09:04:53.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12092" PostTypeId="1" CreationDate="2016-06-06T11:24:34.600" Score="3" ViewCount="518" Body="&lt;p&gt;Mondobrain proposes a &lt;a href=&quot;http://www.mondobrain.com/technology.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;big data&quot; technology&lt;/a&gt; with:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a new generation of algorithms based on A. Grothendieck scheme theory&#xA;  (Field Medal) that extract knowledge and rules from data without any&#xA;  model or distance, and that can explore every part of &#xA;  multi-dimensional spaces independently.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is behind this method that does not use &quot;model of distance&quot;? Are there relations methods like Topological Data Analysis (evoked at &lt;a href=&quot;https://math.stackexchange.com/questions/1592421/why-does-topology-rarely-come-up-outside-of-topology/1592463#1592463&quot;&gt;math stackexchange&lt;/a&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2016-12-18T20:49:42.073" Title="What is behind &quot;A. Grothendieck scheme theory&quot; in Mondobrain?" Tags="|bigdata|distance|methods|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12094" PostTypeId="2" ParentId="12088" CreationDate="2016-06-06T13:44:17.817" Score="0" Body="&lt;p&gt;It seems you have a data set for one component where the component suffered a fixed number of failure modes. You want to find out which data (let's assume continuous in time, so, what time) correspond to what failure mode. In other words, you are doing &quot;pattern recognition&quot; in your failure data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you thought of using Self-Organizing Maps (SOM)? They are a sub-branch of artificial neural networks and have great capability in such problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also consider that not all failure modes appear in shape of a &quot;peak&quot; value. So, only looking at peaks is not a very smart way. It most probably will cover most of the failures, though, there will be moments that you miss. SOM could take care of this too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data pre-processing is done before the analysis. Be careful in normalization. You could miss the peaks or valley points easily if you don't pay enough attention in normalization. Don't just use any code or normalization method you find online. Test and check it with you data. For instance, some normalization could make all negative values positive which a negative value may have an important meaning in your work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you have another variable called &quot;failure&quot;. Another approach I suggest is building a Neural Network (NN) model, which is very common. You have three input data that you mentioned, consider the &quot;failure&quot; variable as your target variable. Build the neural network and apply it to your data again. If the numbers of failures are little, the NN will be able to rebuild the normal behavior of your data (NN here is called a normal behavior model). When you apply it to your input data, the NN model will detect any deviation which is not expected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;MATLAB has a very good support for both of these approaches. &lt;/p&gt;&#xA;" OwnerUserId="17937" LastActivityDate="2016-06-06T13:44:17.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12096" PostTypeId="2" ParentId="12082" CreationDate="2016-06-06T15:44:16.317" Score="1" Body="&lt;p&gt;If the CSV really comes from Excel, I fear additional columns are lost... If you still have the source file (ie the text file with all the 18.000 columns), I recommend using &lt;code&gt;fread&lt;/code&gt; from high-performance package &lt;code&gt;data.table&lt;/code&gt;. Something else to learn but definitively worth for high volumes data...&#xA;Syntax similar to read.csv.&lt;/p&gt;&#xA;" OwnerUserId="18722" LastActivityDate="2016-06-06T15:44:16.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12097" PostTypeId="1" CreationDate="2016-06-06T15:49:09.420" Score="0" ViewCount="338" Body="&lt;p&gt;I have developed an RNN in matlab and now its time to test it. You can set your desired number of layers and nodes and it trains data in random chunks. I also included an annealing function for learning rate that decays gradually from 1 to 1e-5 to avoid overtraining of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like you to suggest me some simple ways to test my network in a way that i can get some useful feedback about its behavior from the results. I am a little concerned about convergence. I' ve set to do some testing of my own but would like to hear some suggestions from more experienced users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I try to simulate some 1st, 2nd, 3rd, ... grade function between (0-1) and see how it goes. I will also try to predict the next value of a time series based on previous values, as another testing method. What would be a good result from these tests? An accuracy of ??% and an mse of ?? ?&lt;/p&gt;&#xA;" OwnerUserId="20321" LastActivityDate="2016-08-17T15:31:06.930" Title="Please suggest some good ways to test my RNN" Tags="|rnn|training|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12099" PostTypeId="1" AcceptedAnswerId="12100" CreationDate="2016-06-07T01:42:41.457" Score="2" ViewCount="234" Body="&lt;p&gt;What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Table A:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ArmyVet_ID  Served_WW2  Served_KoreanWar    Served_VietnamWar&#xA;110001          1              0                    0&#xA;110002          1              0                    0&#xA;110004          0              1                    0&#xA;110005          0              1                    0&#xA;110009          0              0                    1&#xA;110010          0              0                    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Table B:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ArmyVet_ID    Served&#xA;110001          WW2&#xA;110002          WW2&#xA;110004          KoreanWar&#xA;110005          KoreanWar&#xA;110009          VietnamWar&#xA;110010          VietnamWar    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, the question of how to do the above conversion using R has been asked to death on SO. However, there seem to be way too many ways to do it. If anyone's figured out the absolutely best way to do it (quickest, easiest), I'd appreciate pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update after correct answer marked below: It turns out that Table A is called &quot;wide format&quot; and B is called &quot;long format&quot;.&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="16928" LastEditDate="2016-06-07T03:04:57.397" LastActivityDate="2019-03-15T17:51:38.080" Title="Technical name for this data wrangling process? Multiple columns into multi-factor single column" Tags="|r|dataset|data-cleaning|data-wrangling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12100" PostTypeId="2" ParentId="12099" CreationDate="2016-06-07T02:13:15.073" Score="2" Body="&lt;p&gt;It is usually called reshaping! For a great description of the process, see &lt;a href=&quot;http://www.statmethods.net/management/reshape.html&quot; rel=&quot;nofollow&quot;&gt;this walkthrough&lt;/a&gt;, or read up on Hadley Wickham's documentation for &lt;a href=&quot;https://github.com/hadley/reshape&quot; rel=&quot;nofollow&quot;&gt;the &lt;code&gt;reshape&lt;/code&gt; package&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2016-06-07T02:13:15.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12101" PostTypeId="1" AcceptedAnswerId="12118" CreationDate="2016-06-07T06:46:18.560" Score="7" ViewCount="15073" Body="&lt;p&gt;I'm just starting to investigate machine learning concepts, so I'm sorry if this question is very naive, but I'm hoping that it will be an easy one to answer!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a document matching algorithm that individually calculates a match for each field (0-1, with 0 = no match, 1 = 100% match), and applies a separate weight to each field match to be used in calculating an overall weighted average relevance &quot;score&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., given a document of 3 fields (d1-3) and an input query against each of the fields (q1-3), field matches are calculated for each pair (m1-3) and then weights (w1-3) are applied using a weighted average for a final relevance score: s = sum(mi x wi)/sum(wi).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this contrived example, perhaps we can simply say that a document is considered relevant if the score is above 0.5. I.e., there is either a &quot;relevant&quot; (0.5-1.0) or &quot;not relevant&quot; (0-0.5) outcome. But I don't want every field to have equal weight in determining the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is simply: What type of machine learning technique is &quot;best&quot; used to calculate the appropriate weights (w1-n), based on past, known results? Is this even an appropriate use of machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And secondly, if instead of a simple outcome of relevant and non-relevant, I actually want to rank the documents by relevancy, can this also be achieved using a machine learning technique?&lt;/p&gt;&#xA;" OwnerUserId="20338" LastActivityDate="2016-06-08T15:09:54.583" Title="Machine learning technique to calculate weighted average weights?" Tags="|machine-learning|weighted-data|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12104" PostTypeId="1" CreationDate="2016-06-07T10:57:25.210" Score="1" ViewCount="191" Body="&lt;p&gt;I created was database through commands (see lower)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Started Citespace, clicked DATA - IMPORT/EXPORT&#xA;Citespace-Built-in-database&#xA;Database-Connect to mySQL&#xA;Input directory - choosed directory with WOS files&#xA;Project - import data (WOS)​ &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now check:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;click Count records&#xA;Recieved:&#xA;projects  0&#xA;articles    0&#xA;authors  18020&#xA;refs   144499&#xA;keywords  64260&#xA;phrases    0&#xA;institutions  11695&#xA;verbs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so tables&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- articles&#xA;- bursts&#xA;- clusters&#xA;- phrases&#xA;- verbs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;are empty &#xA;and system see no projects&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CAN YOU help me to understand what i do wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;=== database schema&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CREATE DATABASE wos;&#xA;USE wos;&#xA;CREATE TABLE articles&#xA;(&#xA;    id int,&#xA;    uid VARCHAR(100),&#xA;    project VARCHAR(100),&#xA;    author VARCHAR(100),&#xA;    title VARCHAR(100),&#xA;    abstract VARCHAR(1500),&#xA;    source VARCHAR(200),&#xA;    j9 VARCHAR(100),&#xA;    volume VARCHAR(100),&#xA;    issue VARCHAR(100),&#xA;    bp VARCHAR(100),&#xA;    ep VARCHAR(100),&#xA;    page VARCHAR(100),&#xA;    dt VARCHAR(100),&#xA;    doi VARCHAR(100),&#xA;    year int,&#xA;    month int,&#xA;    date int,&#xA;    citations int,&#xA;    editor VARCHAR(100),&#xA;    tagged boolean&#xA;);&#xA;/*  TABLE articles &#xA;id(int), uid, project, author, title, abstract, source, j9, volume, issue, bp, ep, page, dt, doi, year(int),&#xA;month(int), date(int), citations(int), editor, tagged(boolean)&#xA;*/&#xA;CREATE TABLE authors&#xA;(&#xA;id int,&#xA;lastname VARCHAR(100),&#xA;firstname VARCHAR(100),&#xA;initials VARCHAR(100),&#xA;project VARCHAR(100),&#xA;uid VARCHAR(100),&#xA;pos VARCHAR(100)&#xA;);&#xA;/*  TABLE authors&#xA;id(int), lastname, firstname, initials, project, uid, pos */&#xA;CREATE TABLE refs&#xA;(&#xA;id int,&#xA;bibcode VARCHAR(100),&#xA;ref VARCHAR(100),&#xA;doi VARCHAR(100),&#xA;author VARCHAR(100),&#xA;year VARCHAR(100),&#xA;source VARCHAR(100),&#xA;volume VARCHAR(100),&#xA;page VARCHAR(100),&#xA;citer_uid VARCHAR(100),&#xA;project VARCHAR(100)&#xA;);&#xA;/*   TABLE refs&#xA;id(int), bibcode, ref, doi, author, year, source, volume, page, citer_uid, project */&#xA;CREATE TABLE keywords&#xA;(&#xA;id int,&#xA;keyword VARCHAR(100),&#xA;uid VARCHAR(100),&#xA;year VARCHAR(100),&#xA;project VARCHAR(100),&#xA;type VARCHAR(100)&#xA;);&#xA;/* TABLE keywords&#xA;id(int), keyword, uid, year, project, type */&#xA;CREATE TABLE phrases&#xA;(&#xA;id int,&#xA;phrase VARCHAR(100),&#xA;isTitlePhrase boolean,&#xA;project VARCHAR(100),&#xA;uid VARCHAR(100),&#xA;year int, &#xA;month int, &#xA;date int, &#xA;freq int&#xA;);&#xA;/*  TABLE phrases&#xA;id(int), phrase, isTitlePhrase(booelan), project, uid, year(int), month(int), date(int), freq(int)  */&#xA;CREATE TABLE verbs&#xA;(&#xA;id int,&#xA;project VARCHAR(100),&#xA;uid VARCHAR(100),&#xA;verb VARCHAR(100),&#xA;freq VARCHAR(100)&#xA;);&#xA;/*  TABLE verbs&#xA;id(int), project, uid, verb, freq  */&#xA;CREATE TABLE bursts&#xA;(&#xA;id int,&#xA;project VARCHAR(100),&#xA;term VARCHAR(100),&#xA;weight  double,&#xA;start int,&#xA;end int&#xA;);&#xA;/* TABLE bursts&#xA;id(int), project, term, weight(double), start(int), end(int) */&#xA;CREATE TABLE institutions&#xA;(&#xA;id int,&#xA;name VARCHAR(100),&#xA;country VARCHAR(100),&#xA;uid VARCHAR(100),&#xA;year int,&#xA;project VARCHAR(100)&#xA;);&#xA;/* TABLE institutions&#xA;id(int), name, country, uid, year(int), project */&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20342" LastEditorUserId="924" LastEditDate="2016-06-08T17:04:57.710" LastActivityDate="2017-03-13T15:51:49.090" Title="mysql could not recieve table from Citespace" Tags="|sql|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-15T10:30:59.000" ContentLicense="CC BY-SA 3.0" />
  <row Id="12106" PostTypeId="1" AcceptedAnswerId="12116" CreationDate="2016-06-07T15:32:25.020" Score="4" ViewCount="4060" Body="&lt;p&gt;I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like help objectively understanding the practical tradeoffs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The reason I am interested in Java&lt;/strong&gt; is that I think some of my production, frequently refreshed, forecasts and analyses at work would run much faster in Java (compared to R or Python) and by becoming more proficient in Java I would enable myself to work on interesting side projects, such as non-Data-Science apps I'd like to develop. Currently I've taken a couple of Java courses, but I need much more education and practice to master it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The reason I started considering learning Scala&lt;/strong&gt; is very similar -- I figured that the statistical/ML orientation would make it good for my work as a Data Scientist, and since it is based on Java I would be getting practice at work that helps me with my side-interest in Java, even though there are a few major differences such as &lt;em&gt;functional vs. imperative&lt;/em&gt; and &lt;em&gt;traits vs. interfaces&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like a lot of the advantages of Scala revolve around integration with &lt;strong&gt;Spark&lt;/strong&gt;. I am thinking that this should be the tipping point for my decision, because my team is not currently using Spark and I don't have a good enough reason to request it. However, I thought I should ask here so that I don't waste too much time if Scala is still a better choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purposes of this question &lt;strong&gt;please ignore alternatives such as Python, R, Julia, etc&lt;/strong&gt; (I've eliminated those from consideration for other reasons, such as already being sufficiently familiar with them for my use cases).&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-06-09T19:30:03.350" LastActivityDate="2019-11-05T19:31:18.493" Title="Scala vs Java if you're NOT going to use Spark?" Tags="|java|scala|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12107" PostTypeId="2" ParentId="12061" CreationDate="2016-06-07T15:33:58.620" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://shop.oreilly.com/product/0636920028512.do&quot; rel=&quot;nofollow&quot;&gt;Learning Spark: Lightning Fast Big Data Analytics&lt;/a&gt; is a fairly comprehensive book covering the core concepts as well as the higher level components involved in the Spark stack. This is the book recommended by Databricks for their Spark Developer Certification as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested more about the use cases built using Spark, I suggest &lt;a href=&quot;http://shop.oreilly.com/product/0636920035091.do?green=FDE26D1E-4F6A-5328-9E23-46715C3B56EB&amp;amp;intcmp=af-mybuy-0636920035091.IP&quot; rel=&quot;nofollow&quot;&gt;Advanced Analytics with Spark&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14391" LastActivityDate="2016-06-07T15:33:58.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12108" PostTypeId="1" AcceptedAnswerId="12189" CreationDate="2016-06-07T18:31:21.040" Score="2" ViewCount="418" Body="&lt;p&gt;Lately, I've been experimenting with Microsoft R Server(MRS) and SQL Server R Services. I've come across that MRS is built on architectural components like &lt;b&gt;ScaleR&lt;/b&gt; algorithms optimized for fast parallel execution, &lt;b&gt;DistributedR&lt;/b&gt; parallel computing framework for managing compute resources, and &lt;b&gt;ConnectR&lt;/b&gt; for versatile connections to data sources.&#xA;&lt;br&gt;&#xA;&lt;br&gt;&#xA;I've also learnt (and got confused) that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;b&gt;ScaleR&lt;/b&gt; algorithms are basically functions of the &lt;b&gt;RevoScaleR package&lt;/b&gt; built on the &lt;b&gt;DistributedR&lt;/b&gt; framework.&lt;/li&gt;&#xA;&lt;li&gt;&lt;b&gt;ConnectR&lt;/b&gt; component of the architecture is also a bunch of &lt;b&gt;RevoScaleR&lt;/b&gt; functions / adapters to access a variety of data sources.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is:&lt;br&gt;&#xA;What is &lt;b&gt;RevoScaleR&lt;/b&gt; in relation to &lt;b&gt;DistributedR&lt;/b&gt; and &lt;b&gt;ConnectR&lt;/b&gt;?&#xA;&lt;br&gt;&#xA;Where does &lt;b&gt;DeployR&lt;/b&gt; fit into all this?&#xA;&lt;br&gt;&#xA;&lt;br&gt;&#xA;In general, I want to know the high level differences between&lt;br&gt;&#xA;&lt;b&gt;RevoScaleR package&lt;/b&gt;&lt;br&gt;&#xA;&lt;b&gt;DistributedR&lt;/b&gt;&lt;br&gt;&#xA;&lt;b&gt;ScaleR&lt;/b&gt;&lt;br&gt;&#xA;&lt;b&gt;ConnectR&lt;/b&gt;&lt;br&gt;&#xA;&lt;b&gt;DeployR&lt;/b&gt;&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-06-14T11:41:17.283" Title="What is the difference between the 3 architectural components of Microsoft R Server?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12109" PostTypeId="1" AcceptedAnswerId="15123" CreationDate="2016-06-07T18:37:51.627" Score="6" ViewCount="4059" Body="&lt;p&gt;I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the dataset and need ideas on how to train word embeddings on such a large dataset which is a collection of 243 thousand full article documents. The research computing resource is timed so get short access to GPU node and so opted for Incremental model training: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Incremental Model training: One way to train on entire dataset is to use incremental model training that is train the model on one chunk of the data and save it. Later on pick up the same pre-trained model and start training on it next chunk of data. The problem that I am facing in this approach is that how do I maintain the vocabulary/dictionary of words. In word embedding methods dictionary/vocab plays an important role. We sweep over all documents and create vocab of words that have count greater than a minimum set frequency. Now, actually this vocab is a hash map which has index associated to each word and in training samples we replace words with their indices in the vocab for the simplicity in the model. In the case of incremental training how do I create dictionary incrementally? Do I have to create vocab/dictionary on entire documents initially and then train incrementally? Or is the a way to extend vocab also in incremental training? &lt;/li&gt;&#xA;&lt;li&gt;Another problem is memory limit on the size of the vocab data structure. I am implementing my model in Torch which is LUA based. So, LUA puts a limit on tables size, I cannot load vocab for entire documents in a single table. How to overcome such memory issues? &lt;/li&gt;&#xA;&lt;li&gt;Taking inspiration from Glove vectors. In their paper they say that they “We trained our model on five corpora of varying sizes: a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion to- kens; Gigaword 5 which has 4.3 billion tokens; the combination Gigaword5 + Wikipedia2014, which has 6 billion tokens; and on 42 billion tokens of web data, from Common Crawl5. We tokenize and lowercase each corpus with the Stanford tokenizer, build a vocabulary of the 400,000 most frequent words6, and then construct a matrix of co- occurrence counts X” . Any idea on how Glove vectors trained on such a big corpus and big vocabulary and how memory restrictions in their case might have got handled?  Paper reference - &lt;a href=&quot;http://nlp.stanford.edu/pubs/glove.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://nlp.stanford.edu/pubs/glove.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Any ideas on how to limit the size of the dataset for generating word embeddings? How would it affect the performance or coverage of word embeddings with the increase or decrease in number of documents? Is it a good idea to use sampling techniques to sample documents from dataset? If yes, then please suggest some of the sampling techniques.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="20358" LastActivityDate="2016-11-15T05:22:13.097" Title="How to handle Memory issues in training Word Embeddings on Large Datasets?" Tags="|dataset|deep-learning|word-embeddings|torch|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12110" PostTypeId="2" ParentId="6107" CreationDate="2016-06-07T20:15:32.383" Score="302" Body="&lt;p&gt;&lt;em&gt;Deconvolution layer&lt;/em&gt; is a very unfortunate name and should rather be called a &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/256#issuecomment-162257789&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;transposed convolutional layer&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually, for a transposed convolution with stride one and no padding, we just pad the original input (blue entries) with zeroes (white entries) (Figure 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YyCu2.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YyCu2.gif&quot; alt=&quot;Figure 1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of stride two and padding, the transposed convolution would look like this (Figure 2): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f2RiP.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f2RiP.gif&quot; alt=&quot;Figure 2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All credits for the great visualisations go to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Vincent Dumoulin, Francesco Visin - &lt;a href=&quot;https://arxiv.org/pdf/1603.07285v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;A guide to convolution arithmetic for deep learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find more visualisations of convolutional arithmetics &lt;a href=&quot;https://github.com/vdumoulin/conv_arithmetic&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14578" LastEditorUserId="14578" LastEditDate="2020-05-18T17:19:25.377" LastActivityDate="2020-05-18T17:19:25.377" CommentCount="17" ContentLicense="CC BY-SA 4.0" />
  <row Id="12111" PostTypeId="1" CreationDate="2016-06-07T20:22:44.327" Score="0" ViewCount="803" Body="&lt;p&gt;In most of the popular packages (rpart etc.), the decision tree split is based on information gain, entropy or Gini index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a popular decision tree package ( either 'R' or python) where the node split quality can be specified using chi-squared statistic?&lt;/p&gt;&#xA;" OwnerUserId="15790" LastEditorUserId="21" LastEditDate="2016-06-08T08:24:04.613" LastActivityDate="2016-06-08T08:24:04.613" Title="DecisionTree split quality using chi-squared statistic" Tags="|machine-learning|python|r|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12112" PostTypeId="2" ParentId="9317" CreationDate="2016-06-07T20:48:22.300" Score="0" Body="&lt;p&gt;I think @biswajit-banerjee comments (and others) should be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say, &lt;code&gt;How do I find the common values&lt;/code&gt;, i assume you are looking at either &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Values of A that exist in the other vector, or&lt;/li&gt;&#xA;&lt;li&gt;Values of B that exist in the other vector, or&lt;/li&gt;&#xA;&lt;li&gt;Unique values that exist in either vectors. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The third is tricky because you do not have vectors of equal lengths. Hence there is no index that could be mapped. So the option in that case should be the intersection of the unique values from each vector as @michaelg mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few ways depending on what you want to do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;list1 = c(&quot;item1&quot;, &quot;item2&quot;, &quot;item3&quot;, &quot;item1&quot;, &quot;item8&quot;, &quot;item2&quot;)&#xA;list2 = c(&quot;item8&quot;, &quot;item9&quot;, &quot;item10&quot;, &quot;item1&quot;)&#xA;&#xA;index &amp;lt;- match(list1,list2) # solve 1&#xA;result1 &amp;lt;- list2[na.omit(index)]&#xA;&#xA;index &amp;lt;- match(list2,list1) # solve 2&#xA;result2 &amp;lt;- list1[na.omit(index)]&#xA;&#xA;unique(result1) # or unique(result2) to solve 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output for each case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4 NA NA  4  1 NA # index&#xA;&quot;item1&quot; &quot;item1&quot; &quot;item8&quot; # list2[na.omit(index)]&#xA;&#xA;5 NA NA  1 # index&#xA;&quot;item8&quot; &quot;item1&quot; # list1[na.omit(index)]&#xA;&#xA;&quot;item1&quot; &quot;item8&quot; # result1&#xA;&quot;item8&quot; &quot;item1&quot; # result2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="10345" LastEditDate="2016-06-09T17:00:58.990" LastActivityDate="2016-06-09T17:00:58.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12113" PostTypeId="2" ParentId="12111" CreationDate="2016-06-08T00:56:17.253" Score="2" Body="&lt;p&gt;Base on another &lt;a href=&quot;https://stats.stackexchange.com/questions/130697/chi-squared-significance-test-for-stopping-criteria-in-decision-tree&quot;&gt;post about significance test for stopping decision tree&lt;/a&gt; in stats stackexchange website. CHAID, QUEST, GUIDE are three options. The &lt;a href=&quot;http://webcache.googleusercontent.com/search?q=cache:ql5q26bay9wJ:www.stat.wisc.edu/~loh/quest.html%20&amp;amp;cd=2&amp;amp;hl=en&amp;amp;ct=clnk&amp;amp;gl=us&quot; rel=&quot;nofollow noreferrer&quot;&gt;QUEST&lt;/a&gt; was provide in binary format, but not available now. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usage for an implementation of CHAID in R is expained &lt;a href=&quot;https://stackoverflow.com/questions/17310097/chaid-analysis-options-for-os-x-python-r&quot;&gt;here&lt;/a&gt;. &#xA;To install it on R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;partykit&quot;)&#xA;install.packages(&quot;CHAID&quot;, repos=&quot;http://R-Forge.R-project.org&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12237" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-06-08T01:01:24.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12114" PostTypeId="1" CreationDate="2016-06-08T06:35:50.503" Score="1" ViewCount="43" Body="&lt;p&gt;Does anyone have any ideas about what would be a good way to go about building a user Life Time Value model? I have a website with two kinds of revenue, pay per click and impression from advertisments. My initial instincts are telling me I will need to use a unsupervised clustering algorithm to segment individual users by some kind of rough LTV metric. and then maybe re define a revenue distribution for each cluster. Anyway, any helpful ideas would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="20368" LastActivityDate="2016-06-08T06:35:50.503" Title="how to build Life Time Value model" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12115" PostTypeId="1" CreationDate="2016-06-08T07:09:41.287" Score="6" ViewCount="3032" Body="&lt;p&gt;I have an event dataset from which I would like to detect recurring events (i.e: weekly, bi-weekly, monthly). The dataset contains:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Timestamp (date)&lt;/li&gt;&#xA;&lt;li&gt;Event type which can get any value (e.g: &lt;code&gt;purchase&lt;/code&gt;, &lt;code&gt;withdrawal&lt;/code&gt;, &lt;code&gt;inquiry&lt;/code&gt;, &lt;code&gt;bid&lt;/code&gt;, etc...).&lt;/li&gt;&#xA;&lt;li&gt;Amount in USD (can be bid or price for example)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My goal is to find and label recurring events and their frequency. Most of the time there's one event per day, but sometimes there are two or more events for the same day (can be with the same event type, but doesn't have to be). I should also be able to account for a change in recurrence (from bi-weekly to weekly or vice-versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you approach this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="20298" LastActivityDate="2016-09-08T16:55:01.447" Title="Recurring events - finding in a time series" Tags="|machine-learning|algorithms|time-series|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12116" PostTypeId="2" ParentId="12106" CreationDate="2016-06-08T08:33:24.177" Score="8" Body="&lt;p&gt;This is a bit off topic for this SE, or maybe opinion-based, but, I work in this field and I'd recommend Scala.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No I would not characterize Scala as a &quot;stats-oriented&quot; Java. I'd describe it as what you get if you asked 3 people to design &quot;Java 11&quot; and then used all of their ideas at once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Java 8 remains great, but Scala fully embraces just about all the good ideas from languages that you'd want, like type safety and closures and functional paradigms, and a more elaborate types/generics system, down to more syntactic sugar conveniences like case classes and lazy vals. Plus you get to run everything in the same JVM and interoperate with any Java libraries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The price is complexity: understanding all of Scala is a lot more difficult than all of Java. And some bits of Scala maybe were a bridge too far. And, the tooling ecosystem isn't great IMHO in comparison to the standard Java tools. But there again you can for example use Maven instead of SBT. But you can mostly avoid the parts of Scala that are complex if you don't need them. Designing Scala libraries takes a lot of skill and know-how; just developing run-of-the-mill code in Scala does not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a productivity perspective, once you're used to Scala, you'll be more productive. If you're an experienced Java dev, I actually think you'll appreciate Scala. I have not liked any other JVM language, which tend to be sort of one-issue languages that change lots of stuff for marginal gains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Certainly, for analytics, the existence of Spark argues for Scala. You'll use it eventually.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-06-08T08:33:24.177" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12117" PostTypeId="2" ParentId="12109" CreationDate="2016-06-08T08:56:15.117" Score="1" Body="&lt;p&gt;I'm not familiar with Torch, but since basically word2vec and doc2vec are considered, These models learn from each sentences and so there is no need to have all the sentences in the memory. You could iterate via each sentence in the corpora and let the model learn from each of the sentence. And that is probably how people train on huge corpora with or without high computation machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A short example in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class SentenceIterator(object):&#xA;   def __iter__(file_name):&#xA;      for line in open(file_name)&#xA;         yield line&#xA;&#xA;sentences = SentenceIterator(&quot;path/to/file&quot;)&#xA;for line in sentences:&#xA;    model.train(line)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this way, The memory is loaded with only one sentence at a time and when it is done the memory loads the next one. &#xA;For building the vocabulary, you can do the whole iterating via all the documents to build the vocab first and then train the data, depending upon the word-embedding functions implemented.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-06-08T08:56:15.117" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12118" PostTypeId="2" ParentId="12101" CreationDate="2016-06-08T09:31:45.987" Score="6" Body="&lt;p&gt;Yes it is definitely possible to calculate optimised weightings provided you have some training examples where you know the document fields, the query, and either the outcome (relevant/not-relevant) or the desired score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think your training feature set should be the query score in range [0.0,1.0] for each field of each example. The training label should be either relevance 0 or 1 for each example, or the relevance score that the example has.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you have a target score for each example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to determine the weights $W_i$ to use for each field $i$. Your calculated relevance score would be $\hat{y} = \sum_{i=1}^{N_{fields}} W_i * X_i$ where the caret signifies this is the estimate from your function and $N_{fields}$ is the number of field. Note I am ignoring your original idea of dividing by the sum of all $W_i$, because it makes things more complex. You can either add that term or force the sum to be equal to 1.0 if you wish (I am not going to show you how though, as this answer would get too long, and it probably won't help you much)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a target score and training data, the simplest approach is to find the weights which cause the lowest error when used with the training data. This is a very common goal in supervised learning. You need a &lt;a href=&quot;https://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow&quot;&gt;loss function&lt;/a&gt;. Having a target scalar value means you can use difference from target and a very common loss function for this kind of regression problem is the mean squared error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E = \frac{1}{N_{examples}} \sum_{j=1}^{N_{examples}} (\hat{y}_j - y_j)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\hat{y}_j$ is your calculated relevance score for example $j$ and $y_j$ is your training label for the same example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few different ways to solve for lowest $E$ with this loss function, and it is one of the simplest to solve. If you express your weights as a vector $W$ length $N_{fields}$ your example features as a matrix $X$ size $N_{examples} \times N_{fields}$ and the labels as a vector $Y$ length $N_{examples}$ then you can get an exact solution to minimise loss using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Least_squares#Linear_least_squares&quot; rel=&quot;nofollow&quot;&gt;linear least squares equation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W = (X^TX)^{-1}X^TY$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other approaches that work too - gradient descent or other function optimisers. You can look these up and see which you would prefer to use for your problem. Most programming languages will have a library with this already implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you will likely get scores greater than 1.0 or less than 0.0 from some document/query pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have to use a adjust the technique if you want to divide by total of all weights or want sum of all weights equal to 1 in your scoring system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you have a relevance 0 or 1 for each example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a classification problem, relevant or not are your two classes. This can still be made to work, but you will want to change how you calculate your weighted score and use &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow&quot;&gt;logistic regression&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your weighted score under logistic regression would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{y} = \frac{1}{1 + e^{-(b + \sum_{i=1}^{N_{fields}} W_i * X_i)}}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $b$ is a &lt;em&gt;bias&lt;/em&gt; term. This looks complicated, but really it is just the same as before but mapped by a sigmoid function to better represent class probabilities - the result is always between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look up solvers for logistic regression, and most stats or ML libraries will have functions ready to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Caveats&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have made a starting assumption that a simple combined relevance score will lead to a useful end result for your users performing search. This has led to simple linear models looking like a good solution. However, this may not be the case in practice, and you may need to re-visit that assumption. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-06-08T15:09:54.583" LastActivityDate="2016-06-08T15:09:54.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12123" PostTypeId="1" CreationDate="2016-06-08T12:58:38.227" Score="1" ViewCount="609" Body="&lt;h2&gt;Use Case&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Recommend similar items (restaurants) to diner.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Solution:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;We have used &lt;em&gt;apache spark MLlib ALS algorithm&lt;/em&gt;. Values for lambda and rank has been obtained by iterating through all permutations of combinations on selected range of ranks (9-20) and lamda(0.0001-3). We have kept iterations constant (10). In the end, we evaluated the rank-lamda for which the error is minimum  for our dataset and used it .&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note - Training set - booking matrix for time range say November&#xA;  2015-March 2016. Test Set/Cross Validation Set - booking matrix for&#xA;  the same diner set for time range April 2016 - May 2016. This is just&#xA;  example, we have followed 70/30 ratio.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Approaches followed:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Approach 1&lt;/em&gt; - &#xA;input data consists dinerID,restaurantID,rating. rating in this case is the number of time a restaurant has been booked. 0 rating if it has not been booked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Approach 2 -&lt;/em&gt; &#xA;Instead of number of times a restaurant has been booked, we have rating as 1 if diner has booked a restaurant. &#xA;Using Approach 2 as it had far less error rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Considerations/Tail cutting&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Only active diners and restaurants considered&lt;/li&gt;&#xA;&lt;li&gt;Diner who has booked more than 1 restaurant has been considered&lt;/li&gt;&#xA;&lt;li&gt;Restaurant which has been booked more than once is considered.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Implementation:&lt;/em&gt;&#xA;These 16 latent factors (rank - 16 in our case) for every restaurant/item is obtained and persisted in database. Since our dataset is not big (&lt;em&gt;~25k items&lt;/em&gt;), on runtime &lt;em&gt;euclidean distance&lt;/em&gt; for each is calculated and top 5 are recommended for each restaurant.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Problem:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Restaurants are city specific and so are diners. But while building model as well as creating input matrix, city or any other explicit factor is not taken. I believe spark should be identifying a pattern and one of the factor could be city as well.&#xA;Now, out of X number of cities we operate in, 70%-80% of our business is driven from a city A. A small number from rest 2-3 cities and very few from the rest.&#xA;Recommendation for city A is very good. Also, the recommendations for other 2-3 cities seem to be decent to normal. Other cities, honestly I have not evaluated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pain point:&lt;/strong&gt;&#xA;Since model is trained irrespective of city, I assume the 16 factor vector for each restaurant is a good metric to compare item item similarity. So, if my last booking is a restaurant abc in city A and next time I am in city B (I am new to this city btw) and looking for similar restaurants. What I have implemented is item-item for restaurant abc and restaurants in city B.&#xA;Whenever similar restaurant to any restaurant from city A(the major driver) is calculated for any other city, irrespective of which restaurant from city A has been selected, recommendations are same. However, there are slight variations if instead of city A there are comparisons between city b/c/d/e.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What could be the possible reasons to this?&lt;/li&gt;&#xA;&lt;li&gt;Is it because of heterogeneous/skewed data?&lt;/li&gt;&#xA;&lt;li&gt;Ways to improve recommendations?&lt;/li&gt;&#xA;&lt;li&gt;Can we work on individual latent factors (for example if it is identified that factor2 is city, factor 3 could be cuisine, then can we manipulate??)&lt;/li&gt;&#xA;&lt;li&gt;Any other way to improve or test model?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="20310" LastActivityDate="2016-06-08T12:58:38.227" Title="Spark MLlib recommendation - restaurant/ item similarity - issues/improvement" Tags="|machine-learning|recommender-system|apache-spark|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12124" PostTypeId="1" CreationDate="2016-06-08T13:10:05.593" Score="5" ViewCount="3025" Body="&lt;p&gt;I currently face an unsupervised learning task that is to be approaches using clustering. More specifically, it is a segementation task and hence there is some prior knowledge about a) the number of clusters and b) the rough content of each segment. &#xA;From literature this seems like a prime example, where &quot;constrained k-means&quot; would come into play and I'm really eager to try it out. Both of the above mentionend pieces of prior knowledge can be incorporated in &quot;constrained k-means&quot; as a) is represented by k and b) can be expressed in &quot;must-link constraints&quot;. (For a good overview over constrained k-means see Wagstaff, Cardie, Rogers, Schrödl, &amp;amp; others (2001).)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem now is that I can't find a suitable implementation in R on CRAN.&#xA;Programming it myself in R would be ok, based on the pseudocode, but my programming is most certainly not efficient enough, since we are talking about a somewhat large dataset (75.000 x 30).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I turn to this community hoping that someone has found an efficient implementation of &quot;constrained k-means&quot; in R or is willing to provide code that is somewhat efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clemens &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update 23.06.2016:&#xA;Thanks for your replies so far. In order to make the problem a little more succint please see a sample data file &lt;a href=&quot;https://drive.google.com/file/d/0B3gkU4YdPZivR01CQ3RLajR6UGc/view?usp=sharing&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. (Note: this is not an excerpt of my original data as it is highly sensitive, but an artificial dataset with the same characteristics (less rows though))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two examples of simple constraints:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;all observations that have either a 1 in v305 or a 1 in v306 must be linked (must-link-constraint)&lt;/li&gt;&#xA;&lt;li&gt;all observations that have a 1 in v258 and v262 must be linked (must-link-constraint)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Currently looking at your suggestions, Rahul. Thanks very much.&lt;/p&gt;&#xA;" OwnerUserId="20373" LastEditorUserId="20373" LastEditDate="2016-06-23T08:13:41.200" LastActivityDate="2022-11-01T17:04:55.633" Title="Constrained k-means algorithms in R (must-link constraints)" Tags="|r|k-means|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12125" PostTypeId="1" CreationDate="2016-06-08T13:37:20.413" Score="2" ViewCount="551" Body="&lt;p&gt;Below is text / code from &lt;a href=&quot;https://stackoverflow.com/questions/37692653/problems-with-theano-convnet-model-accuracy-in-wild-data&quot;&gt;a recent SO post of mine&lt;/a&gt; that I hope you could comment on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Keras/Theano, I successfully trained a model using the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras sample code for CIFAR10&lt;/a&gt;. It ran for about 2 hours on a MacPro with a GTX98-Ti GPU, with Theano (latest version) backend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All original parameters are unchanged and model reached ~81% accuracy. At the end of script the weights (weights.h5, 5MB on disk) and architecture (arch.json) were exported.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In another script I restored the model and weights, and tried to classify single images using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from skimage.data import imread&#xA;import os&#xA;from keras.models import model_from_json&#xA;from keras.optimizers import SGD&#xA;&#xA;model = model_from_json(open('./arch.json').read())&#xA;model.load_weights('./weights.h5')&#xA;sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)&#xA;model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])&#xA;&#xA;for root, dirs, files in os.walk('./input/singles/'):&#xA;    for idx, file in enumerate(files):&#xA;        if not file.startswith('.'):&#xA;            file_path = root + os.sep + file&#xA;            img = imread(file_path)&#xA;            img = fn_resize(img, 32) # this is a resize function in another file&#xA;                                     # I have plotted images at this point and they are&#xA;                                     # correctly sized smaller version of the original&#xA;                                     # normalized between 0-1 as in training&#xA;&#xA;            img = img.reshape((1, 3, 32, 32))&#xA;&#xA;            y_pred = int(model.predict_classes(img, 1))&#xA;            pred_labels = ['airplane', 'automobile', ' bird', 'cat', 'deer',&#xA;                           'dog', 'frog', 'horse', 'ship', 'truck']&#xA;            print(pred_labels[y_pred], &quot;prediction \\\&quot;, file, &quot;truth&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Essentially it restores the model and architecture, then loops through 10 images from the Net in a folder, resizing them to 32x32 pixels, reshaping the array, and forwarding the data to &lt;code&gt;model.predict_classes()&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was surprised to see the below output, in which none of the images was classified correctly. &lt;strong&gt;Has anyone done such an experiment and knows if there are issues with the model build or if my procedure is incorrect?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Using Theano backend.&#xA;Using gpu device 0: GeForce GTX 980 Ti (CNMeM is disabled, cuDNN 5005)&#xA;1/1 [==============================] - 0s&#xA;truck predicted \\\ airplane.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;airplane predicted \\\ bird.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;airplane predicted \\\ car.jpg truth&#xA;1/1 [==============================] - 0s&#xA;airplane predicted \\\ cat.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;frog predicted \\\ deer.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;ship predicted \\\ frog.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;automobile predicted \\\ horse.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;automobile predicted \\\ dog.tif truth&#xA;1/1 [==============================] - 0s&#xA;frog predicted \\\ ship.jpeg truth&#xA;1/1 [==============================] - 0s&#xA;airplane predicted \\\ truck.jpeg truth&#xA;&#xA;Process finished with exit code 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20374" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-06-08T17:17:09.333" Title="Why is this trained model misclassifying new cases?" Tags="|machine-learning|python|neural-network|deep-learning|theano|" AnswerCount="0" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="12126" PostTypeId="1" AcceptedAnswerId="12133" CreationDate="2016-06-08T15:27:22.430" Score="0" ViewCount="196" Body="&lt;p&gt;I'm working on a neural networks project right now and for that I'm reading a bunch of scientific papers, in a few of those the terms additive hidden nodes and radial basis functions are thrown around, but I seem to have trouble to get a clear explanation of the terms there and anywhere else on the internet. I seem to have gathered that these are classifications for neuron types but I would love to get a more clear intuitive explanation of the terms. Preferably one that doesn't require me to be very mathy to understand. I'm fairly new to neural networks so beyond sigmoid neurons and backpropagation algorithms I'm still a bit lost when it comes to the common termonology.&lt;/p&gt;&#xA;" OwnerUserId="19067" LastEditorUserId="9814" LastEditDate="2016-06-09T09:54:36.733" LastActivityDate="2016-06-09T10:30:16.297" Title="Looking for a rough explanation of additive hidden nodes and radial basis functions" Tags="|neural-network|terminology|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12127" PostTypeId="1" AcceptedAnswerId="12200" CreationDate="2016-06-09T00:02:39.277" Score="4" ViewCount="1635" Body="&lt;p&gt;I have event dataset in a factless table. It has list of events&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp-&amp;gt; event name -&amp;gt; node ( In network)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is always a sequence of event happening. So how do I start predicting future event based on past one and discover list of nodes that will be affected from past experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am programmer without knowledge of machine learning. I have installed spark,R and have dataset in oracle database.Is there any tutorial/algorithm that I can use to get started. I taught myself scala/R but have no idea on getting started. My dataset are huge i.e. more than 9billion rows for 3 months.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Node            Eventtime       alarmname&#xA;192.168.1.112   6/14/2016 19:41 Main power supply has a fault alarm&#xA;192.168.1.113   6/14/2016 19:23 Association path broken&#xA;192.168.1.113   6/14/2016 19:23 NA&#xA;192.168.1.113   6/14/2016 19:23 Association broken&#xA;192.168.1.112   6/14/2016 19:23 Mains Failure&#xA;192.168.1.112   6/14/2016 19:23 Mains Failure&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Additional Information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 98 nodes. I would like to predict:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i. No of node that has alarm or goes down when a single node goes down&#xA;e.g. if node A has alarm if in 1 month list of node that has alarm at the same period&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ii. The sequence of event occurrence i.e. if one node has mains failure then the next event would be node down.&lt;/p&gt;&#xA;" OwnerUserId="3230" LastEditorUserId="3230" LastEditDate="2016-06-14T23:44:38.053" LastActivityDate="2016-06-17T05:06:23.403" Title="How to start prediction from dataset?" Tags="|machine-learning|predictive-modeling|apache-spark|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12128" PostTypeId="1" CreationDate="2016-06-09T02:38:44.360" Score="5" ViewCount="176" Body="&lt;p&gt;are there any resources out there (book, blog, your own answer post etc.) that gives advise on modelling strategy of deep neural net?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to fit a neural net, I know how to change settings like number of hidden layers, activation function, dropout etc. I know how to use cross-validation to validate models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what I need is advise on the actual modelling process. i.e. given a dataset (which has been cleaned and explored), where do you start? What type of neural net do you train first? How do you then tune it?&lt;/p&gt;&#xA;" OwnerUserId="20385" LastActivityDate="2016-06-15T17:06:12.323" Title="Deep neural net modelling strategy" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12129" PostTypeId="1" AcceptedAnswerId="18877" CreationDate="2016-06-09T06:19:25.890" Score="3" ViewCount="2139" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/llM3T.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/llM3T.png&quot; alt=&quot;It describe formula of ward method i.e. increment of error after merging two cluster&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure describe formula of ward method for hierarchical or agglomarative clustering i.e. increment of total error after merging two cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How did they get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{n_A\cdot n_B}{n_A+n_B} ||\vec m_A-\vec m_B||^2 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from first line of formula as shown in figure?&lt;/p&gt;&#xA;" OwnerUserId="18818" LastEditorUserId="8878" LastEditDate="2017-05-11T07:48:24.670" LastActivityDate="2017-05-15T11:23:44.863" Title="Derivation of Ward's formula for agglomerative clustering" Tags="|machine-learning|data-mining|clustering|hierarchical-data-format|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12130" PostTypeId="2" ParentId="12106" CreationDate="2016-06-09T07:48:47.307" Score="3" Body="&lt;p&gt;The other side of the coin:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have an extensive experience with Scala; I have written approximately 10,000 lines of Scala code. However, consider that Scala code is often much shorter than its rough equivalent of 40,000 lines of Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On short I don't like Scala at all. I love it's goals, it's ideas but for production use I consider the implementation of those ideas too bold and sometimes even foolish. My belief is that Scala code is hard to write in a robust and crystal clear fashion. There are too many concepts which often overlaps by their side effects which can make your life really hard if you are not an expert. There is an opinion that one can avoid those problems by simply not using those constructs. I reject this opinion since if you want to avoid problematic things you would have to know that you do not understand completely those features, which would require an expert knowledge. My belief is that Scala language implementation is proper for an university where new concepts should be tested, which is exactly what happens with Scala. &#xA;There are many examples, I will give some reasons which can sustain what I say:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Java generics are incomplete mostly because of type erasure, everybody agrees on that. Scala &quot;solved&quot; that problems using a public API in at least three ways: &lt;strong&gt;Manifest&lt;/strong&gt;, &lt;strong&gt;TypeTag&lt;/strong&gt;, &lt;strong&gt;ClassTag&lt;/strong&gt;. In my opinion this means that the problem is simply not solved or if it's solved some of the solutions are wrong enough.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;converters&lt;/strong&gt; are an example of a feature which I consider problematic: one simply removes the strong connection between types with something soft, very easy to be missed. After I started to use a third party library where some courageous guy used plenty of them and my code started to behave weird. I typed something wrong which happened to be covered by a converter and spent hours debugging. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;parameters with default values&lt;/strong&gt;: one can define methods with parameters with default values; this is useful mostly because using named parameters you avoid writing many methods... until the moment when you discover that you cannot have two methods both with a default valued parameter with the same name. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now I don't want to flame a war, far from it. Scala has many wonderful ideas, Java is old and lacks many things. But for production I would always chose Java. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Later edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on Scala type erasure there is a short review &lt;a href=&quot;https://xiaalex.wordpress.com/2014/12/03/scala-type-erasure-manifest-and-typetag/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to clarify what I stated about parameter with values I just give an hypothetical example (so objections like Array and List implements some traversable interface does not count):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Plot {&#xA;  def histogram(x: Array[Double], bins: Int = 30): String = &quot;test1&quot;&#xA;  def histogram(x: List[Double], bins: Int = 30): String = &quot;test2&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above code does not compile: &lt;em&gt;error: in class Plot, multiple overloaded alternatives of method f define default arguments&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;type-erasure-manifest-and-typetag/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Another later edit&lt;/strong&gt;&#xA;I received an edit proposal and since I do not know how to message the contributor I will answer here. His idea is that the code is not right and it should look like the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def histogram(x: Array[Double], bins: Int = 30): String = {&#xA; var r = &quot;test1&quot;&#xA; return r;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As far as I know this is the same as above, since my version is a shortcut. The point is that the code does not compile because you have a parameter with default values  with the same name in two methods, not because methods are not declared correctly. Obviously, like in many other languages, one can find other ways to overpass this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole idea is that to me Scala looks like a too generous language, with many ways of doing many things, which as a side effect creates a complexity burden. In the context, having features like the one I mentioned, it makes things even harder. Obviously I do not own the truth, so I take the liberty to avoid the language complexity and concentrate my effort on solving real problems, and anybody else can have the liberty to solve its problems with this language. With all due respect.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2016-10-18T06:19:55.263" LastActivityDate="2016-10-18T06:19:55.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12131" PostTypeId="2" ParentId="11990" CreationDate="2016-06-09T09:14:29.983" Score="2" Body="&lt;p&gt;As far as I understand you simply don't have to worry about that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you have a sample split randomly in control and treatment. You measure something before treatment, and after, for the same individuals. Because you measure on the same individuals than you have paired measures: delta = before - after. You are interested to measure if the mean of delta for control sample is significantly different than the mean of delta for control sample. This is done with paired sample test known also as dependent test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume a normal distribution you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Student&amp;#39;s_t-test#Dependent_t-test_for_paired_samples&quot; rel=&quot;nofollow&quot;&gt;paired t test&lt;/a&gt;. If you can't reasonably assume the normal distribution than you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test&quot; rel=&quot;nofollow&quot;&gt;Wilcoxon signed rank test&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea of pairings in test is to eliminate the effect of confounders. What you computed can be the effect of a confounder, but if the sample was split randomly in control and treatment you should not worry about that, unless you have strong reasons to doubt the randomization procedure.  In the later case you perhaps should include confounders in the equation and take the route of random effects modeling.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-06-09T09:14:29.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12132" PostTypeId="1" CreationDate="2016-06-09T09:17:49.230" Score="1" ViewCount="354" Body="&lt;p&gt;I have a dataset of users, each user has has daily information about his activities (numerical values representing some measurements of his physical activities).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, each user in each day has a boolean value that represents if he/she took a particular action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset looks as follow&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+------+----------+----------+----------+-------+&#xA;|userID|      date| activity1| activity2| action|&#xA;+------+----------+----------+----------+-------+&#xA;| user1|2016-06-05|       5.3|         6|  false|&#xA;| user1|2016-06-04|       3.1|         8|   true|&#xA;| user1|2016-06-03|       2.0|        13|  false|&#xA;| user1|2016-06-02|       4.7|         1|  false|&#xA;| user1|2016-06-01|       1.3|         9|  false|&#xA;| user1|   ...ect.|       ...|       ...|    ...|&#xA;| user2|2016-06-05|       0.6|         5|   ture|&#xA;| user2|2016-06-04|       3.0|         5|  false|&#xA;| user2|2016-06-03|       0.0|         0|  false|&#xA;| user2|2016-06-02|       2.1|         3|  false|&#xA;| user2|2016-06-01|       6.3|         9|  false|&#xA;| user2|   ...ect.|       ...|       ...|    ...|&#xA;| user3|2016-06-05|       5.3|         0|  false|&#xA;| user3|2016-06-04|       5.3|        11|  false|&#xA;| user3|2016-06-03|       6.8|         5|  false|&#xA;| user3|2016-06-02|       4.9|         2|  false|&#xA;| user3|   ...ect.|       ...|       ...|    ...|&#xA;+------+----------+----------+----------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; that the dataset is not fixed, so one new row is added for each user on every new day. But the number of columns is fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Goal&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Build a model that predicts which user is likely to take the action in the near future (e.g. in any of the next 7 days).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach is to build feature vectors representing the activity values for each users over a period of time, and use the action column as a source of ground-truth. Then I feed the ground-truth and the feature vectors to a binary classification training algorithm (e.g. SVM or Random Forest) in order to generate a model able to predict if a user is likely to take the action or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started by the positive examples that are the users who took the action. To extract the feature vector of a positive example, I combined the activity values in the X (30 or 7 or 1) days preceding the action (the day of taking the action is included).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I moved to the negative examples, it gets less obvious, I am not sure &lt;strong&gt;how to select negative examples and how to extract features from them&lt;/strong&gt;. This has led me actually to re-question if &lt;strong&gt;my way of selecting positive examples and building my the feature vectors was correct&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to build the ground-truth of positive (users who did take the action) and negative (users who didn't take the action) examples?&lt;/li&gt;&#xA;&lt;li&gt;What is a negative example in this case? is it the user who didn't take the action in a fixed period of time? What if he didn't take the action in this fixed period, but he just took it right after?&lt;/li&gt;&#xA;&lt;li&gt;What are the possible approaches of selecting the ranges of dates to extract feature vectors from.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rational Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there more suitable approaches (other than classification) to solve this kind of problems?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="13891" LastEditDate="2016-06-09T13:48:20.640" LastActivityDate="2016-06-10T05:52:15.587" Title="Ground-truth and feature extraction for predictive modelling" Tags="|predictive-modeling|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12133" PostTypeId="2" ParentId="12126" CreationDate="2016-06-09T10:24:09.740" Score="2" Body="&lt;p&gt;Albeit not wrong, Huang seems to be the only person in the world to use the term &quot;additive hidden nodes&quot;. By this, he means that the neuron computes the sum of weighted inputs. In other words, the kind of neural network you're already used to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An RBF neuron, on the other hand, computes a distance (usually the Euclidean distance) from input to some center (which can be thought as the weights if you see them as a vector) and applies a &lt;code&gt;exp(-dist²)&lt;/code&gt; function in order to obtain a Gaussian activation. Thus, RBF neurons have maximum activation when the center/weights are equal to the inputs.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9u5fQ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9u5fQ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9814" LastEditorUserId="9814" LastEditDate="2016-06-09T10:30:16.297" LastActivityDate="2016-06-09T10:30:16.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12135" PostTypeId="1" CreationDate="2016-06-09T14:36:49.700" Score="2" ViewCount="339" Body="&lt;p&gt;I'm using the MLPRegressor in sklearn to train a network with approx 1000 inputs and a continuous output variable.  Essentially, the issue is one of image classification (1000 pixels) with the distribution of the pixels related to a continuous variable output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can generate an arbitrarily large training set, but run into memory problems running this on a single machine when loading arrays into numpy.  I've selected hyper parameters that give a reasonable result with a limited (5k) training set, but am trying to work out the best way to train the network with two orders of magnitude more data.  Is there a way to pass mini-batches to sklearn myself so that I can manage the memory and IO?  Is training and retraining the network helpful if I can setup a means to ensure I read batches for each training step in a randomised fashion? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing this is a common problem, but I'm struggling to find a sensible solution.&lt;/p&gt;&#xA;" OwnerUserId="20398" LastEditorUserId="11097" LastEditDate="2016-06-09T14:50:11.720" LastActivityDate="2024-03-08T19:07:10.840" Title="MLP batch iteration in python" Tags="|python|neural-network|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12137" PostTypeId="1" CreationDate="2016-06-09T14:56:26.750" Score="2" ViewCount="57" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xsIiU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xsIiU.png&quot; alt=&quot;k=5&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am conducting a network text analysis as part of a systematic literature review of social network analysis in computer-supported learning. My nodes are terms in the included studies, and the ties represent how close these terms are to each other in the text (window size of 10).  This is my first time conducting any sort of network analysis and I am unsure about interpreting my results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran a weighted clique percolation (k=5) on my data in the hopes of finding subcommunities of terms that could give me a quantitative overview of the content of my pool of studies. Instead of finding overlapping subcommunities, I found one large subcommunity subsuming smaller ones (k=5). What can I make out of this data then? Is it a result still work noting?&lt;/p&gt;&#xA;" OwnerUserId="20400" LastActivityDate="2016-06-09T14:56:26.750" Title="How do I interpret the results of weighted clique percolation if communities are subsuming (not overlapping)?" Tags="|social-network-analysis|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12138" PostTypeId="2" ParentId="12106" CreationDate="2016-06-09T15:09:51.520" Score="2" Body="&lt;p&gt;The two languages have pretty similar benefits since Scala can call Java libraries. So Java machine learning packages like Weka (&lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.waikato.ac.nz/ml/weka/&lt;/a&gt;), can in theory be easily used with Scala. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are minor pros and cons to each, however:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Java is a language that most software engineers with 5+ years of experience understand. If you go to a big financial institution and need to hand off your app to a legacy team, they'll likely know how to support Java a app but not a Scala app (at least where I work)&lt;/li&gt;&#xA;&lt;li&gt;Java is stable and doesn't often change, but Scala seems to change rapidly. This can make it more difficult to maintain in the long term&lt;/li&gt;&#xA;&lt;li&gt;Poorly written Java code might be verbose, but poorly written Scala code can be completely unintelligible (ex: what does ::: mean?). The creator of Python optimized Python for readability; it sometimes feels like the creator of Scala did the opposite&lt;/li&gt;&#xA;&lt;li&gt;You can write Scala code much faster than you can write Java code. Scala is ideal for temporary prototype code because you can see your idea come to life faster than you can with Java&lt;/li&gt;&#xA;&lt;li&gt;Spark is much easier to work with Scala than Java. The machine learning Spark libraries are decent enough that you might not need to use a different machine learning library like Weka. I've seen people build Scala models in Spark even on small datasets. You don't need to have a huge data set to use Spark. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary:&lt;/strong&gt; Go with Scala. Most data science work is prototyping, and Scala will help you work through prototypes faster. Spark ML is probably good enough for your needs, and Scala is much better for Spark than Java. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="-1" LastEditDate="2019-11-05T19:31:18.493" LastActivityDate="2019-11-05T19:31:18.493" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12139" PostTypeId="1" CreationDate="2016-06-09T15:30:07.473" Score="2" ViewCount="237" Body="&lt;p&gt;I am using a smartphone in my car to gather acceleration data (both longitudinal and lateral). Now, I want to classify my data in real-time based on the acceleration force applied through accelerating, breaking and maybe turning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a proper method to do this kind of online classification? &#xA;And also how does one determine whether a &quot;harsh/aggressive&quot; or just &quot;normal&quot; driving style?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/icjWx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/icjWx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15690" LastEditorUserId="15690" LastEditDate="2016-06-10T07:56:33.740" LastActivityDate="2016-06-14T15:05:56.183" Title="Finding a proper method for an online driving style classification using acceleration data" Tags="|r|classification|time-series|java|online-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12140" PostTypeId="2" ParentId="12032" CreationDate="2016-06-09T17:06:08.580" Score="6" Body="&lt;p&gt;I have to quote Tukey, perhaps the grandfather of data science:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The combination of some data and an aching desire for an answer does&#xA;  not ensure that a reasonable answer can be extracted from a given body&#xA;  of data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I see nothing wrong with your Poisson model. In fact its a pretty good fit to the data. The data is noisy. There is nothing you can do about it. Perhaps the noise if due to whatever else is on TV at the time, or the weather, or the phase of the moon. Whatever it is, its not in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you reasonably think the weather might be affecting your data, get the weather data and add it. If it decreases the log-likelihood enough for each degree of freedom then it's doing a good job and you leave it in. This is regression modelling 101. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course there's a zillion other things you can do. Scale the data by any old transformation you want. Fit a quadratic. A quartic. A quintic. A spline.  You could include the date and possible temporal correlation effects. But always bear in mind what Tukey was saying - if your data is noisy, you won't get anything much out of it. So it goes.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-09T17:06:08.580" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12141" PostTypeId="2" ParentId="9505" CreationDate="2016-06-09T19:57:40.110" Score="5" Body="&lt;p&gt;I agree with @Hoap. Your features might be low for the amount of training observations you have. Instead of excluding columns, see if you're missing more features. Feature Engineering, rather than Feature Selection. &lt;br&gt;&#xA;However, if you are looking for Feature Selection, then Azure ML has a Feature Selection Module with the option to specify how many features you'd like to keep.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some simple verifications to do before you jump into modeling: &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Visualize your dataset for any non-linear relationships. &lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;You could also perform a simple correlation analysis to check for multi-collinearity. &lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;I also think that normalizing all of your data between 0 to 1 for consistent comparable values between features would be helpful.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hopefully one of these will show some unexpected pattern in your data. I apologize if you've already performed these checks. Just wanted to put them out there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looks like you pretty much used every regression model in the Azure ML library. &lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-06-09T19:57:40.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12142" PostTypeId="2" ParentId="12128" CreationDate="2016-06-09T23:04:16.070" Score="2" Body="&lt;p&gt;I cannot recommend highly enough this &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow&quot;&gt;online book&lt;/a&gt; on neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tricky part about neural networks is the stuff you mentioned -- what value to use for the learning rate, what topology to use for the network, etc etc -- we call these things &lt;em&gt;hyperparameters&lt;/em&gt; to distinguish from &lt;em&gt;parameters&lt;/em&gt; which are estimated by the optimization process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This big &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html#how_to_choose_a_neural_network&amp;#39;s_hyper-parameters&quot; rel=&quot;nofollow&quot;&gt;section of the aforementioned book&lt;/a&gt; covers exactly that. I don't know if you'll find what you want though. The reason is because there is no silver bullet.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-06-14T13:25:04.777" LastActivityDate="2016-06-14T13:25:04.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12144" PostTypeId="2" ParentId="12139" CreationDate="2016-06-10T04:58:16.200" Score="1" Body="&lt;p&gt;&lt;strong&gt;Supervised vs. Unsupervised Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will first need to decide whether to approach this as a &lt;code&gt;supervised&lt;/code&gt; or &lt;code&gt;unsupervised learning&lt;/code&gt; problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;supervised learning&lt;/code&gt;, you would need to take some portion of the data and hand score it as harsh/aggressive and normal.  I suggest you &lt;strong&gt;not&lt;/strong&gt; go this route to start.  But if you do, I suggest using a &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;support vector classifier&lt;/code&gt;&lt;/a&gt; for your initial testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest approaching this as an &lt;code&gt;unsupervised learning&lt;/code&gt; problem. Assuming you tend to be a normal/safe driver you can view this behavior as normal and harsh/aggressive driving as an &lt;code&gt;anomaly&lt;/code&gt; or an &lt;code&gt;outlier&lt;/code&gt;.  This is known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Anomaly_detection&quot; rel=&quot;nofollow&quot;&gt;outlier detection&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Engineering&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Feature engineering&lt;/code&gt; will play a significant roll in this problem. Despite your attention to the longitudinal and lateral axes of the car, the phone will have arbitrary orientation within the car i.e. it will just be in your pocket.  Thus its probably best to factor out the direction somewhat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I would find the length of the acceleration vector (magnitude of acceleration without regard to its directional orientation).  Since you have three orthogonal axes, you can just use: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$||\mathbf{a}||=\mathbf{a}_x^2+\mathbf{a}_y^2+\mathbf{a}_z^2=a$$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you can also integrate the acceleration vector over time and find its magnitude to find the vehicle speed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$speed=||\mathbf{v}||=\mathbf{v}_x^2+\mathbf{v}_y^2+\mathbf{v}_z^2=v$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $$\mathbf{v}(t)=\int_{start-of-trip}^t\mathbf{a}(t)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The speed is very useful in determining the difference between different modes of acceleration e.g. driving around in a circle will yield constant acceleration, but also constant speed vs. holding down the gas pedal and driving straight, which could yield constant acceleration with changing speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important aspect of this method lies in the fact that integrals of your data will tend to damp noise, where as derivatives (differences) will tend to amplify noise.  &lt;strong&gt;Don't take derivatives of your data&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Outlier Detection&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point you can normalize the speed and acceleration-magnitude and find the mean and standard deviation of both values.  When your normalized speed and acceleration-magnitude lies outside of some z-value, say two standard deviations, then you can classify your driving as dangerous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget to think of speed and acceleration as orthogonal values and form a circular function out of the respective z-values for proper outlier detection.  You can &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;nofollow&quot;&gt;read about 2D or nD outlier detection&lt;/a&gt; and also &lt;a href=&quot;https://stat.ethz.ch/education/semesters/ss2012/ams/slides/v2.2.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember that the equation for a circle is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r^2=a^2+v^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can therefore solve for $r$ and set it equal to $2$ standard deviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, note that since variance is just the square of the standard deviation, you could alternatively think of this in terms of variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for two standard deviations, the driving is aggressive if:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sqrt{\big(\frac{v-\mu_v}{\sigma_v}\big)^2+\big(\frac{a-\mu_a}{\sigma_a}\big)^2} \gt2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The online learning component simply requires that you continuously update the means and standard deviations.  Though, I suggest that this be done in some sort of batch mode rather than continuously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-06-14T15:05:56.183" LastActivityDate="2016-06-14T15:05:56.183" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12145" PostTypeId="2" ParentId="12132" CreationDate="2016-06-10T05:52:15.587" Score="1" Body="&lt;p&gt;Your approach is a good one.&#xA;This way to extract features can lead to good results. &#xA;But before moving forward I recommend answering these two questions:&#xA;Is there is any correlation between the behavior of the user today and yesterday ... Etc &quot; temporal correlation/ temporal dependency&quot; &#xA;Is there is any kind of dependency between the users? &#xA;Answering these questions can measure the quality of your feature ? &#xA;Sometimes there is no need even to do calculations to answer these questions , if you tell us what kind of actions or activities you are looking at we may help in better way .&#xA;For example : if user 1 used his car today , there still a high prob to use it tomorrow, but if user 1 went to barbershop today there is a high probability that will not go tomorrow &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-06-10T05:52:15.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12146" PostTypeId="2" ParentId="19" CreationDate="2016-06-10T06:31:49.753" Score="5" Body="&lt;p&gt;There is special thing to graph algorithms, you original questions which makes then special, which is about he ability to partition the data essentially.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some things, like sorting numbers on an array it is not too difficult to partition the problem on the data structure into smaller disjunctive pieces, e.g. &lt;a href=&quot;http://www.drdobbs.com/parallel/parallel-in-place-merge-sort/240169094&quot; rel=&quot;noreferrer&quot;&gt;Here: Parallel in place merge sort&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For graph algorithms however there is the challenge that finding an optional partitioning on a given graphic metric is known to be $NP-hard$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So while 10GB of numbers to sort might be a very well approachable problem on a normal PC (You can just to in via dynamic programming and have very good predictability about the program flow), working with a 10GB graph data structure can already by challenging. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of specialized frameworks such as &lt;a href=&quot;https://spark.apache.org/graphx/&quot; rel=&quot;noreferrer&quot;&gt;GraphX&lt;/a&gt; using methods and special computing paradigms to somewhat circumvent the inherent challenges of graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your question briefly: &#xA;As mentioned before by others, when your data does not fit into main memory on a normal PC but you need all of it to answer your problem, is a good hint that your data is already somewhat big. The exact labeling though depends i think a bit on the data structure and question asked. &lt;/p&gt;&#xA;" OwnerDisplayName="user14073" LastActivityDate="2016-06-10T06:31:49.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12147" PostTypeId="1" CreationDate="2016-06-10T07:22:28.273" Score="3" ViewCount="2110" Body="&lt;p&gt;I am working on a certain insurance claims related data-set to classify newly acquired customers as either &lt;strong&gt;claim&lt;/strong&gt; or &lt;strong&gt;non-claim&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic problem with the training set is the extremely large imbalance in &lt;strong&gt;claim&lt;/strong&gt; and &lt;strong&gt;non-claim&lt;/strong&gt; profiles, with the claims amounting to just ~ 0.26% of the training set. Also, most claims are concentrated largely towards the final few years (data is sorted by date).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On applying Logistic Regression or even Random Forests, to train on 70% of the data, the test results were well below satisfactory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been looking at alternate models and I came across &lt;a href=&quot;http://fastml.com/what-is-better-gradient-boosted-trees-or-random-forest/&quot; rel=&quot;nofollow&quot;&gt;this blog post&lt;/a&gt;. A particular line that got my attention is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;GBM is better than rf_t. In the paper, the best classifier for two-class data sets was avNNet_t, with 83.0% accuracy&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although, no real clarification was given as to why that was. Can someone help me open this &quot;blackbox&quot;? Which model really works (in the case described above) and &lt;em&gt;why&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2016-06-10T11:03:51.120" LastActivityDate="2016-06-17T15:03:56.697" Title="Gradient Boosted Trees or Neural Networks Using Model Averaging?" Tags="|machine-learning|classification|neural-network|accuracy|gbm|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="12148" PostTypeId="1" AcceptedAnswerId="12374" CreationDate="2016-06-10T08:35:44.360" Score="9" ViewCount="5185" Body="&lt;p&gt;Applied Lasso to rank the features and got the following results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rank feature prob.&#xA;==================================&#xA;1       a     0.1825477951589229&#xA;2       b     0.07858498115577893&#xA;3       c     0.07041793111843796&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that the data set has 3 labels. The ranking of the features for the different labels are same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then applied random forest to the same data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rank feature score&#xA;===================================&#xA;1       b     0.17504808300002753&#xA;6       a     0.05132699243632827&#xA;8       c     0.041690685195283385&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that ranking are very different from the one produced by Lasso.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to interpret the difference? Does it imply the underlying model is inherently nonlinear?&lt;/p&gt;&#xA;" OwnerUserId="15624" LastActivityDate="2016-06-23T19:54:58.747" Title="feature importance via random forest and linear regression are different" Tags="|feature-selection|random-forest|linear-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12149" PostTypeId="1" CreationDate="2016-06-10T09:15:48.140" Score="1" ViewCount="401" Body="&lt;p&gt;If I am making nonsense, I beg some mercy. I am only 17 and is not exactly a college freshmen.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In Nate's Silvers methodology of predicting election result, he explained that each survey has a decay rate of 200 days. That means, if a new data weights  full, data that was published 200 days ago only weights the half of the new day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, it makes senses. A poll of Hillary v Trump D-5 of election is more accurate compared to Hillary v Trump D-200.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So I implemented the same thing with soccer result. I have like 12 years of data at my disposal, and using these data, I want to predict the soccer result--specifically to its score (ie.2-5,5-7,3-8) not just who wins and who loses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I treated this as a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my idea was to have a decay rate when evaluating the match. I mean, older matches are pretty much useless right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used Bayesian Theorem to predict the result. To get the prior and like hood probability, I used Gaussian and Poisson distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I used Poisson distribution because a research against Bundesliga showed that using Poisson distribution yields best match prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I used Gaussian distribution because it turns out that goal distribution is pretty much log-Gaussian (you have to log transform the data first to make it Gaussian.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plan was to train the decay rate using the train data, and later pick the best model using validation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to create a variable called decayrate = X, where the value of match occurs in a season is 1/X of the value of the match occurs in the next season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ex: I'm predicting a match from 2013/2014 with a decayrate = 1.5. A match from 2012/2013 weights full. A match from 2011/2012 weights = 1/1.5, A march from 2010/2011 weights 1/(1.5^2), and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find the correct decay rate, I find the logloss of each method when pitted against every decay rate from 1.0 ~ 2.0, multiple of 0.05. The decay rate with the lowest logloss is the optimal decay rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an experiment, I randomly pick 25% match from the pool of data as a train data 10 times--not as a retarded attempt to do cross validation--but to check if the optimal decay rate changes when pitted against different train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Gaussian distribution, the optimal decay rate is always either 1.20 or 1.25--that's close enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is when I used Poisson distribution, the optimal decay always changes as I throw in another random training. It can be 1, it can be 2, or it can 1.3. There's no pattern.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So my question is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Is there anything I can do when the data is not working with me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is there a technical jargon for this? I googled &lt;code&gt;time series&lt;/code&gt; but no result. I googled &lt;code&gt;decay rate data science&lt;/code&gt;, but what I got was exponential decay rate used for physics.&lt;/p&gt;&#xA;" OwnerUserId="20420" LastActivityDate="2016-06-10T09:15:48.140" Title="How do I find the correct decay rate when the data are not helping?" Tags="|data-mining|time-series|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12151" PostTypeId="2" ParentId="12115" CreationDate="2016-06-10T10:03:46.627" Score="2" Body="&lt;p&gt;I'd suggest to use ARIMA (autoregressive iterate moving average) as a way to detect the regularities in your time series. Take a look to the following link, in which you will be introduced to ARIMA by means of series of blog posts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ucanalytics.com/blogs/forecasting-time-series-analysis-manufacturing-case-study-example-part-1/&quot; rel=&quot;nofollow&quot;&gt;http://ucanalytics.com/blogs/forecasting-time-series-analysis-manufacturing-case-study-example-part-1/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-06-10T10:03:46.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12153" PostTypeId="2" ParentId="12115" CreationDate="2016-06-10T12:29:26.790" Score="1" Body="&lt;p&gt;To discover periodic behavior in time series data, you should plot and look into a &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat510/node/71&quot; rel=&quot;nofollow noreferrer&quot;&gt;periodogram&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Usn3f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Usn3f.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-06-10T12:29:26.790" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12154" PostTypeId="1" CreationDate="2016-06-10T15:48:25.987" Score="1" ViewCount="172" Body="&lt;p&gt;I am working with data using python. I tried to plot a histogram using &lt;code&gt;mathplotlib&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a few examples on how to analyze data to get more accurate graphs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I coded:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Call csvfile&#xA;csvdata = pandas.read_csv('data/education_data/' + filename + '.csv', low_memory=False, encoding='latin-1')&#xA;&#xA;# Convert amount field into int()&#xA;def str_to_int(mainList):&#xA;    for item in mainList:&#xA;        newList = [(int(item.replace('$', '').replace(',', '')) / (1000000)) for item in mainList]&#xA;    return newList&#xA;&#xA;# Convert str to int using str_to_int() function&#xA;csvdata['str_costo'] = str_to_int(csvdata['Cuantía'])&#xA;sigma = numpy.std(csvdata['str_costo'])&#xA;mean = numpy.mean(csvdata['str_costo'])&#xA;x = mean + sigma * csvdata['str_costo']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, I saw one step in particular:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In[0] x = mean + sigma * csvdata['str_costo']&#xA;Out[0] 1.371822e+08, 1.608953e+08, etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All values from &lt;code&gt;csvdata['str_costo']&lt;/code&gt; are integers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to interpret &lt;code&gt;x&lt;/code&gt; values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretically what they mean?&lt;/p&gt;&#xA;" OwnerUserId="17636" LastEditorUserId="17636" LastEditDate="2016-06-11T17:56:01.227" LastActivityDate="2016-06-11T17:56:01.227" Title="Interpreting statistical formulas using python (mu + sigma * data)" Tags="|python|bigdata|statistics|visualization|descriptive-statistics|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12155" PostTypeId="1" AcceptedAnswerId="12158" CreationDate="2016-06-10T17:55:24.757" Score="1" ViewCount="217" Body="&lt;p&gt;I have users in my database that I would like to match up or group togetter based on the content of there articles. I cant seem to find how this kind of problem is being solved today. Any advice will help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Available Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Each user's posts (anything written by them, like a blog).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Tags for each post (tags that the user gave to their post when they created it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Goal:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Match/Group users based on available data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Produce match percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Attempt:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I matched people based on the number of exact matches of their tags.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: user1 has [car,honda,sports], user2 has [car,food].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give a 33% match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can image this does not work very well. Most users have 20 tags but typically get a match percentage of 0% even if they are talking about similar things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tags that have a clear relationship like CAR and HONDA are NOT matched.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I match/Group users based on tags or the content of there articles?&lt;/p&gt;&#xA;" OwnerUserId="20437" LastActivityDate="2016-06-11T08:42:11.310" Title="Match users based on the content of their articles" Tags="|machine-learning|algorithms|recommender-system|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12156" PostTypeId="1" AcceptedAnswerId="12162" CreationDate="2016-06-10T18:12:00.957" Score="-1" ViewCount="109" Body="&lt;p&gt;Currently our company, has a special user forum.&#xA;The main forum is about specific topic: SIP protocol.&#xA;Im trying to understand what would be a good approach to classify the top 10 issues customers report in the forum, &#xA;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Installation, Crash, Media, Database, etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would be a good approach to start classifying the threads, currently each thread is in a CSV file.&#xA;I though of doing the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tokenization&lt;/li&gt;&#xA;&lt;li&gt;Remove stop words&lt;/li&gt;&#xA;&lt;li&gt;Extract top terms per thread&lt;/li&gt;&#xA;&lt;li&gt;Count all Top issues across all threads and display top words.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any suggestion is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2016-06-11T14:23:28.980" Title="Newsgroup classification" Tags="|classification|text-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12157" PostTypeId="2" ParentId="9696" CreationDate="2016-06-11T07:17:05.860" Score="2" Body="&lt;p&gt;An HMM doesn't really make sense (echoing what Dries said). If you want to use an HMM, you would have to justify it by asking &quot;Can Reddit posts be represented by a Markov process?&quot; I can't think of a way to make that sentence true and still take advantage of the features related to a popular post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the possible feature set: the time it was posted, the user posting it, the type of post (link / image / text), the subreddit, the number of subscribers to that subreddit, a score of positivity / negativity, number of words in the title etc. Don't count out these features.&lt;/p&gt;&#xA;" OwnerUserId="20445" LastActivityDate="2016-06-11T07:17:05.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12158" PostTypeId="2" ParentId="12155" CreationDate="2016-06-11T08:42:11.310" Score="1" Body="&lt;p&gt;One way could be to apply word-embeddings for semantic similarity checking. word2vec model generates feature vectors which could capture semantic similarity. For example, the closest vector to &lt;code&gt;car&lt;/code&gt; will be &lt;code&gt;honda, ferrari, vehicle, bike&lt;/code&gt;. Train a model using large amount of data from wikipedia dumps or the one released by google. It has fine quality vectors. &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow&quot;&gt;Gensim&lt;/a&gt; has a nice implementation of word2vec&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each blog articles, pre-process the data by removing the stop-words and stemming them. From the resulting words, collect the more frequent words. Do this for all the articles and check for the similarity among the frequent words in other articles as well. So that one article with frequent words &lt;code&gt;car, race, tournament, ferrari, F1&lt;/code&gt; will have more closer vectors in article with frequent words &lt;code&gt;bike, honda, racer&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or other way is to look for similar vectors in tags itself. It is good to play with it for sometime, so that you get to know which features work better for the dataset you have.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-06-11T08:42:11.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12159" PostTypeId="2" ParentId="11775" CreationDate="2016-06-11T12:50:22.830" Score="1" Body="&lt;p&gt;The idea would be to find  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Rotational independent and &#xA;2) Shift invariant features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply speaking the Fourier transform is a rotational and translational (shift) independent feature. &#xA;There are SIFT features used in computer vision that can extract such features. This is something you can try. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you are using QR codes,I think edge magnitude maps would be better than images because you could use image processing techniques to suppress the  noise in the background and highlight the edges on the QR code. You could even look at autoencoder and CNN related feature extraction. &lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2016-06-11T12:50:22.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12160" PostTypeId="1" CreationDate="2016-06-11T13:52:09.267" Score="2" ViewCount="2061" Body="&lt;p&gt;I have time series of several variables. Just in one specific case one variable is linear combination of the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict probability distribution (that is not only best estimate but estimates with probability of that happening) of future value of variables. I want to see when probability of small interval of possible outcome is high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A priory I don't know rules of the game how variables evolve and inter-depend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the tool to best do such prediction and how easy is it? Will scikit-learn do? Maybe neural networks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ADDED based on answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I've understood, time series theory data science assumes random walk, however I assume variables are at least partly moved by free will of players of the game under some a priory unknown to me constraints and goals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should statistical solution advised work in such case? Can data predict reversal of possible current trend?&lt;/p&gt;&#xA;" OwnerUserId="20448" LastEditorUserId="20448" LastEditDate="2016-06-11T16:37:23.163" LastActivityDate="2017-01-07T21:36:09.597" Title="predicting probability distribution for time series" Tags="|predictive-modeling|time-series|scikit-learn|probability|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12161" PostTypeId="2" ParentId="12160" CreationDate="2016-06-11T14:08:36.563" Score="-1" Body="&lt;p&gt;Yes, you can use scikit-learn for this use case. You may find &lt;a href=&quot;http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/&quot; rel=&quot;nofollow&quot;&gt;this tutorial&lt;/a&gt; useful.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-06-11T14:08:36.563" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12162" PostTypeId="2" ParentId="12156" CreationDate="2016-06-11T14:23:28.980" Score="1" Body="&lt;p&gt;Your use case boils down to categorizing news feed on an online forum and then finding out top-n categories. I would suggest you look at this &lt;a href=&quot;https://blog.monkeylearn.com/hacker-news-categorizer-with-monkeylearn/&quot; rel=&quot;nofollow&quot;&gt;Hacker News Categorizer developed by MonekyLearn&lt;/a&gt;. This way you can understand how to get started with such projects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS : I am &lt;strong&gt;not&lt;/strong&gt; affiliated with MonkeyLearn.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-06-11T14:23:28.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12163" PostTypeId="2" ParentId="9819" CreationDate="2016-06-11T17:55:44.787" Score="4" Body="&lt;p&gt;I looked &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, and found that the default value changed from 1 to 5. Apparently the authors believe that &lt;strong&gt;more epochs will improve the results&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot tell from experience, yet.&lt;/p&gt;&#xA;" OwnerUserId="5059" LastEditorUserId="29575" LastEditDate="2019-05-11T19:01:30.377" LastActivityDate="2019-05-11T19:01:30.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12164" PostTypeId="1" AcceptedAnswerId="12967" CreationDate="2016-06-11T18:24:31.133" Score="0" ViewCount="1460" Body="&lt;p&gt;Background:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images.  This works well, except when it doesn't.  Many docs are old, scanned images and what Tika extracts is gibberish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Spark on Hadoop and either ML or MLlib (haven't settled, though I like ML better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far getting best results from a pipeline using Naive Bayes that removes Stopwords, tokenizes and Countvectorizes features (no Tf-Idf).  Total bag-of-words approach.  Next best is using ML to tokenize, Tf with Idf and into LogisticregressionwithLBFGS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, the thought occurred to me that the model uses many docs that are junk.  Literally just strings of gibberish like &quot;mmmmmmmm aaannnammmmrrr hdhhdhhhhhjjj...&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't good, but since I'm operating at scale it's just what happened.  Certainly I could pick through 10,000 training docs and remove the bad examples, but there has to be an easier way.  Is there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The title of this question belies my brainstorm that there might be a way to discount, downweight or outright ignore tokens that aren't discernable by a dictionary.  Is there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Open to any and all advice or approaches to get better precision out of this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17309" LastActivityDate="2016-07-24T15:56:31.693" Title="How to down-weight non-words in text classification?" Tags="|classification|data-cleaning|apache-spark|scala|text|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12165" PostTypeId="1" AcceptedAnswerId="12170" CreationDate="2016-06-11T21:04:26.030" Score="2" ViewCount="71" Body="&lt;p&gt;I am scraping the prices of several products on different websites for the past couple of weeks. These prices are stored, plotted and e-mailed to me every day with an update about the price changes. Now I want to take it to the next level, and make a store selection based on several products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want, is pick a number of products, and calculate the most optimal shops where to buy these products, including shipping costs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I want to buy products A, B and C, at stores X and Y, with the following prices (so product A costs 2 at store X, and 3 at store Y):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   X Y&#xA;A  2 3&#xA;B  3 4&#xA;C  3 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Not taking into account shipping costs, I would buy A and B at store X, and C at store Y. However, if the shipping costs are too large for store Y, it might be better to buy everything at store X. Making things more complicated, if stores X and Y have shipping costs of 5, but Y is free when buying more then a specific amount, it might be more profitable to buy everything at Y.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right direction how to solve this? The idea for now is to include ~10 products at three stores. I though about linear optimisation, but I do not know how to include the shipping costs.&lt;/p&gt;&#xA;" OwnerUserId="12302" LastActivityDate="2016-06-12T07:35:48.987" Title="Optimisation strategy webstores with shipping costs" Tags="|python|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12167" PostTypeId="1" CreationDate="2016-06-12T00:04:18.160" Score="2" ViewCount="1820" Body="&lt;p&gt;I'm teaching myself Data mining and now struggling from this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to this table(&lt;a href=&quot;http://slidewiki.org/slide/22352&quot; rel=&quot;nofollow&quot;&gt;http://slidewiki.org/slide/22352&lt;/a&gt;),&#xA;if the attributes are Asymmetric, dissimilarity is calculated without &lt;strong&gt;&lt;em&gt;t&lt;/em&gt;&lt;/strong&gt; (Values which are 0 for both)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me give an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 objects, called Products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have 2 attributes, COLOR and SIZE.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;COLOR is either RED(1) and BLUE(0) (They are binary).&lt;/li&gt;&#xA;&lt;li&gt;SIZE is either BIG(1) or SMALL(0) (They are binary too).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When there are RED-BIG(1-1) &amp;amp; RED-SMALL(1-0) objects, d is 1/2, regardless of if the attributes are symmetric or asymmetric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if they are BLUE-SMALL(0-0) and BLUE/BIG(0-1), d is 1/2 for symmetric BUT 1/1 for asymmetric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the difference come from? In both cases, there is only one difference, BUT dissimilarity isn't the same.&lt;/p&gt;&#xA;" OwnerUserId="20454" LastEditorUserId="20454" LastEditDate="2016-06-12T03:53:52.957" LastActivityDate="2019-11-08T02:02:15.307" Title="Why does measuring for Asymmetric Binary Attributes exclude t (Values which are 0 for both)?" Tags="|data-mining|statistics|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12168" PostTypeId="1" AcceptedAnswerId="22385" CreationDate="2016-06-12T02:57:09.367" Score="2" ViewCount="68" Body="&lt;p&gt;I'm searching for generalized methods and libraries to extract quantitative relationships between concepts in natural language.  For instance&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Exam score greater than 90%&#xA;Exam scores of 90% or greater&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should extract to a structure like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Exam Score,&amp;gt;=,90% &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I began with a simple regexp / dictionary method, but it feels very primitive.&lt;/p&gt;&#xA;" OwnerUserId="14136" LastActivityDate="2017-08-19T23:50:13.840" Title="NLP Library or routine for Quantitative relationships" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12170" PostTypeId="2" ParentId="12165" CreationDate="2016-06-12T07:35:48.987" Score="2" Body="&lt;p&gt;I have done the opposite of your problem - I have written code to &lt;em&gt;implement&lt;/em&gt; shipping costs for e-commerce sites which runs on their sites.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shipping cost rules can be almost completely arbitrary logic. In the general case, you have no good choice but to implement them as logic that is assessed per order. That means your optimiser will have to run that logic for every combination of purchases it considers. Which makes this a planning/combinatorics problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may not be so bad - e.g. for purchasing 10 items across 5 stores you have $5^{10}$ combinations which is 9765625. That is possible by brute force.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For larger orders, or more choices of store, you may want to look at dynamic solutions to optimise cost, such as simulated annealing, genetic algorithms. Reinforcement learning run per order may also work.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-06-12T07:35:48.987" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12171" PostTypeId="1" CreationDate="2016-06-12T14:34:28.933" Score="1" ViewCount="224" Body="&lt;p&gt;I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, I would like to build a collection of texts with random 'sentences' and each one is a possible derivation (chosen randomly) from the grammar. I would like also to draw randomly the terminal symbols (the final words). I have looked into (python) nltk and have achieved some document generation, but the derivations chosen are not selected randomly.&lt;/p&gt;&#xA;" OwnerUserId="7966" LastActivityDate="2016-06-12T14:34:28.933" Title="Generating a text training dataset from a grammar" Tags="|dataset|text-mining|parsing|text|training|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12172" PostTypeId="1" CreationDate="2016-06-12T17:27:19.537" Score="8" ViewCount="1144" Body="&lt;p&gt;I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only with the rated fields or should we mark them 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://guidetodatamining.com/&quot;&gt;book&lt;/a&gt; I am following says to exclude the fields since it will give wrong interpretation in case of Euclidean and Pearson Correlation but in case of Cosine Similarity, it makes all the non-rated fields to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain why is it needed to make the non-rated fields to 0 only for Cosine and not others or is there a different way to do it.&#xA;(I know how making fields 0 in Euclidean and Pearson affect the output but not sure of cosine) &lt;/p&gt;&#xA;" OwnerUserId="20472" LastActivityDate="2018-01-05T19:16:35.650" Title="What should be the value of non-rated field when finding cosine similarity" Tags="|correlation|recommender-system|cosine-distance|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12173" PostTypeId="1" CreationDate="2016-06-12T20:52:43.283" Score="0" ViewCount="66" Body="&lt;p&gt;I would like to work out an estimate for some variable Xi using values of a bunch of other variables Xj, j in Y (I guess this should include Xi too) and how correlated they are to Xi.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I do weight(j)=correlation(i,j)/sum over Y of other correlations, then this sum can be zero but I feel that might not be the only problem. I could use the min-max remapping but then the most negatively correlated variables will have no or little effect, whereas in fact they should have a strong negative effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One obvious fix would be to ABS() the sum but I could not prove to myself that this is the correct thing to do. It may be possible that each correlation needs to ABSed in the sum (then it would resemble the way stdev is calculated - then do I need to square my correlations?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I simplified the problem I am trying to solve for this question but I am not sure if the details would affect the solution. There is a time dimension to it, and I am trying to use changes over each variable over time and their correlations to work out the estimate for the next change or a chain of changes I then multiply with the current value. (I don't know if this step is necessary or if proper normalisation of correlations would correct for this even if I used absolute values and not relative changes).&lt;/p&gt;&#xA;" OwnerUserId="20475" LastActivityDate="2016-10-11T05:28:40.673" Title="How to normalise estimate weighted on correlations (some will obviously be negative)" Tags="|correlation|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12174" PostTypeId="1" AcceptedAnswerId="12182" CreationDate="2016-06-12T22:14:22.790" Score="-2" ViewCount="2683" Body="&lt;p&gt;I am looking for a thesis to complete my master, I am interested in Predictive Analytics in marketing, HR, management or financial subject, using Data Mining Application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found a very interesting subject: &quot;Predicting customer churn using decision tree&quot; or either &quot;Predicting employee turnover using decision tree&quot;, I looked around very hard but unfortunately couldn't find any relevant dataset to download (&lt;a href=&quot;https://opendata.stackexchange.com/questions/8951/telecommunication-customer-churn-dataset&quot;&gt;Telecommunication Customer churn Dataset&lt;/a&gt; ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to work on a similar subject using &quot;Decision Tree Technique&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest some topics or project that would make for a good masters thesis subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="20477" LastEditorUserId="-1" LastEditDate="2017-04-13T12:57:29.190" LastActivityDate="2016-06-13T09:03:02.997" Title="Master thesis topics" Tags="|data-mining|predictive-modeling|bigdata|decision-trees|research|" AnswerCount="2" CommentCount="0" ClosedDate="2016-06-13T12:37:20.187" ContentLicense="CC BY-SA 3.0" />
  <row Id="12175" PostTypeId="2" ParentId="12174" CreationDate="2016-06-13T02:27:39.793" Score="0" Body="&lt;p&gt;First, talk to your thesis advisor before committing to a project.  They know better than I do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, just analyzing a new dataset using standard techniques doesn't make for a good masters thesis.  Your project is expected to use some sort of novel approach.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, I'd suggest that you start by reading up on existing decision tree techniques, learning why they work and what their flaws are, and try to find ways to overcome the flaws.  Then, once you have your improvement, it should be relatively easy to find a dataset to apply it to.&lt;/p&gt;&#xA;" OwnerUserId="19085" LastActivityDate="2016-06-13T02:27:39.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12176" PostTypeId="1" CreationDate="2016-06-13T02:37:39.823" Score="4" ViewCount="513" Body="&lt;p&gt;I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to use a method that is much faster than it is accurate, because of the size of the data-set and the fact that I don't have much to lose in a speed-accuracy tradeoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running &lt;code&gt;missForest()&lt;/code&gt; takes several hours while Hmisc's &lt;code&gt;impute()&lt;/code&gt; function gives unsatisfactory results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What functions in R might be useful in such (or similar) case?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xWvYy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xWvYy.png&quot; alt=&quot;&amp;lt;code&amp;gt;mice_plot&amp;lt;/code&amp;gt; output&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2016-06-13T13:30:47.557" LastActivityDate="2016-06-26T20:06:44.097" Title="Computationally Inexpensive Imputation Techniques in R" Tags="|machine-learning|r|efficiency|missing-data|data-imputation|" AnswerCount="1" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12177" PostTypeId="2" ParentId="12173" CreationDate="2016-06-13T03:31:06.993" Score="1" Body="&lt;p&gt;The appropriate method, if you have the data to use it, is linear regression.  Simply using correlations fails to take into account that the other variables are also correlated with each other; for example, any averaging method like the one you describe will overweight any duplicated variables.&lt;/p&gt;&#xA;" OwnerUserId="19085" LastActivityDate="2016-06-13T03:31:06.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12178" PostTypeId="2" ParentId="12172" CreationDate="2016-06-13T04:02:12.897" Score="1" Body="&lt;p&gt;Implementation-side, there is a good reason to make 0 correspond to not rated.  Since most users haven't rated most books, 0 will be the most common value and the cosine similarity function can use &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=3&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjq4q6ei6TNAhXM5IMKHbxLA0oQFggmMAI&amp;amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FSparse_matrix&amp;amp;usg=AFQjCNF91MbdixWuWFhhOnyJ-uE8l-ItZg&quot; rel=&quot;nofollow&quot;&gt;sparse matrices&lt;/a&gt; internally to speed up the computation.  The sparse matrix shortcut is the main reason why people use cosine similarity in the first place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, it will throw off your recommendations if the algorithm interprets a 0 to be closer to a 1 than a 2. You do need to intentionally ignore those, but you can easily do so inside your cosine similarity implementation rather than filtering the data before applying it.&lt;/p&gt;&#xA;" OwnerUserId="19085" LastActivityDate="2016-06-13T04:02:12.897" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12179" PostTypeId="1" CreationDate="2016-06-13T05:10:10.980" Score="1" ViewCount="116" Body="&lt;p&gt;I have a data set listing attributes of clients - a combination of ordinal, categorical and interval data. I am also given the 5 nearest neighbors within the data set for and have to essentially reverse engineer an algorithm for the nearest neighbors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the complexity of the data I'm quite lost as to how I can work out and then obtain parameters of a distance function. I apologize in advance for the vagueness of the question but I'm not even sure where to begin.&lt;/p&gt;&#xA;" OwnerUserId="20485" LastActivityDate="2017-01-03T00:06:49.883" Title="Develop classification model, given the k nearest neighbors" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12180" PostTypeId="1" AcceptedAnswerId="12181" CreationDate="2016-06-13T07:42:57.850" Score="8" ViewCount="1520" Body="&lt;p&gt;I want to extract parameters from an image using a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given an image of a brick wall the NN should extract the width and height of the bricks, the color and the roughness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can generate images for given parameters to train the NN and want to use it to extract the parameters from an actual image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked into CNNs. Can I perform this task with them? Do I need special learning algorithms to extract multiple parameters instead of classification? Are there any NNs that are designed for such tasks?&lt;/p&gt;&#xA;" OwnerUserId="20487" LastActivityDate="2016-06-13T12:00:53.633" Title="Using Neural Networks to extract multiple parameters from images" Tags="|neural-network|deep-learning|computer-vision|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12181" PostTypeId="2" ParentId="12180" CreationDate="2016-06-13T08:43:00.637" Score="8" Body="&lt;p&gt;A CNN could be a good choice for this task if you expect variation in the original image scale, rotation lighting etc, and also have a &lt;em&gt;lot&lt;/em&gt; of training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual CNN architecture is to have convolutional layers close to the input, and fully-connected layers in the output. Those fully-connected layers can have the output arranged for different classification or regression tasks as you see fit. Predicting the values of parameters describing the image is a regression task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want accurate measures of size, you may need to avoid using max pooling layers. Unfortunately, not using pooling will make your network larger and harder to train - you &lt;em&gt;might&lt;/em&gt; get away with strided convolution instead if that is a problem for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your input images are very simple and clear (because they are always computer generated), then other approaches may be more reliable. You may be able to reverse-engineer image production and derive simple rules such as identifying lines, corners, circles and other easy-to-filter image components, and make direct measurements. There may also be a middle ground in complexity where extracting this data as features and using it to train a simple NN (or other ML model) will have good performance.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-06-13T12:00:53.633" LastActivityDate="2016-06-13T12:00:53.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12182" PostTypeId="2" ParentId="12174" CreationDate="2016-06-13T09:03:02.997" Score="1" Body="&lt;p&gt;This is the approach I took:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find journals related to your field of studies&lt;/li&gt;&#xA;&lt;li&gt;Skim through the proceedings, see if there are titles that catch your interest&lt;/li&gt;&#xA;&lt;li&gt;Read the papers (carefully or globally) that seemed interesting&lt;/li&gt;&#xA;&lt;li&gt;Carefully consider the approaches and whatever future suggestions they present in their papers&lt;/li&gt;&#xA;&lt;li&gt;Think critically: What would you change? What do you want to find out? Don't limit yourself to data but rather orient from the perspective of research. Solutions for data might only become apparent when you know exactly &lt;strong&gt;what&lt;/strong&gt; you want to examine.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think this has advantages because these papers outline details regarding data as well -- perhaps you can use the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Present some papers and your idea to your prospective supervisor and he/she will make some suggestions. Researchers generally have a lot of knowledge about the possibilities and might even be curious about some things themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck! And enjoy.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2016-06-13T09:03:02.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12184" PostTypeId="1" CreationDate="2016-06-13T13:30:19.647" Score="1" ViewCount="82" Body="&lt;p&gt;How can I estimate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{i,t}=β_{i}x_{i,t}+ϵ_{i,t} \\&#xA;β_{i}=γ_{i}z_{i}+η_{i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in R ? Moreover, if I have splitted my data set to a train set which will contain 80% of all the i's and then I want to forecast the $y_{i,t}$ for the rest 20% i's  $\forall t$, how can I do it in R ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2018-07-19T11:51:00.687" Title="Estimating panel model in R" Tags="|r|regression|linear-regression|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12186" PostTypeId="1" CreationDate="2016-06-13T16:48:48.797" Score="8" ViewCount="2756" Body="&lt;p&gt;I have a large set of points (order of 10k points) formed by particle tracks (movement in the xy plane in time filmed by a camera, so 3d - 256x256px and ca 3k frames in my example set) and noise. These particles travel on approximately straight lines roughly (but only roughly) in the same direction, and so for the analysis of their trajectories I am trying to fit lines through the points. I tried to use Sequential RANSAC, but can't find a criterion to reliably single out false positives, as well as T- and J-Linkage, which were too slow and also not reliable enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an image of a part of the dataset with good and bad fits I got with sequential Ransac: &lt;a href=&quot;https://i.stack.imgur.com/8ua2o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8ua2o.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I'm using the centroids of the particle blobs here, blob sizes vary between 1 and about 20 pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that subsamples using for example only every 10th frame worked quite well too, so the data size to be processed can be reduced this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read a blog post about all the things neural networks can accomplish, and would like to ask you if this would be a feasible application for one before I start reading (I come from a non-maths background, so I would have to do quite a bit of reading)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or could you suggest a different method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addendum: Here is code for a Matlab function to generate a sample point cloud containing 30 parallel noisy lines, which I can't distinguish yet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function coords = generateSampleData()&#xA;coords = [];&#xA;for i = 1:30&#xA;    randOffset = i*2;&#xA;    coords = vertcat(coords, makeLine([100+randOffset 100 100], [200+randOffset 200 200], 150, 0.2));&#xA;end&#xA;&#xA;figure&#xA;scatter3(coords(:,1),coords(:,2),coords(:,3),'.')&#xA;&#xA;function linepts = makeLine(startpt, endpt, numpts, noiseOffset)&#xA;    dirvec = endpt - startpt;&#xA;    linepts = bsxfun( @plus, startpt, rand(numpts,1)*dirvec); % random points on line&#xA;    linepts = linepts + noiseOffset*randn(numpts,3); % add random offsets to points&#xA;end&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20509" LastEditorUserId="20509" LastEditDate="2016-07-05T11:42:14.727" LastActivityDate="2016-07-17T10:05:26.880" Title="Fitting lines through large point clouds" Tags="|machine-learning|clustering|regression|" AnswerCount="2" CommentCount="19" ContentLicense="CC BY-SA 3.0" />
  <row Id="12188" PostTypeId="1" CreationDate="2016-06-14T07:46:25.090" Score="5" ViewCount="984" Body="&lt;p&gt;I have unsupervised data (i.e this data doesn't have any target variable through which I can learn it's prior behaviour) it is a mix of continuous and categorical data. Now I want to classify the test data into three categories on basis of my unsupervised data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach I took is to first do the clustering of unsupervised data, use this categorised data as a base data for preparing a new  model that predicts on top of it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want know whether this approach correct or not or is there better way for classifying test set? Particular algorithm I need to follow for this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing this in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach is to modify the training set data so that this can be used to properly predict the test data. Here target variable is missing in train and test set.&lt;/p&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="20517" LastEditDate="2016-12-17T14:10:25.863" LastActivityDate="2017-10-13T20:58:56.347" Title="Classification technique for unsupervised data?" Tags="|machine-learning|r|unsupervised-learning|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12189" PostTypeId="2" ParentId="12108" CreationDate="2016-06-14T11:41:17.283" Score="1" Body="&lt;p&gt;RevoScaleR package &amp;amp; ScaleR are same (I think after MS acquisition its became scaleR) &#xA;A new file format especially designed for large files such as : &#xA;XDF File Format (block based data format)&#xA;RxDataSource (designed to support the use of external memory algorithms with .xdf files.)&#xA;PEMA  based Algorithms&#xA;External memory implementations of the statistical algorithms most commonly used with large data sets such as : rxSummary, rxCrossTabs, rxLinMod. rxKMeans, rxGlm etc.&#xA;An extensible programming framework that allows R programmers to write their own external memory algorithms that can take advantage of ScaleR Enterprise’s new Big Data capabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DistributedR for supporting big data framework&#xA;ConnectR It has all connectors to interact with other data sources&#xA;DeployR is used to integrate R with any application programming &lt;/p&gt;&#xA;" OwnerUserId="20525" LastActivityDate="2016-06-14T11:41:17.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12190" PostTypeId="1" AcceptedAnswerId="14159" CreationDate="2016-06-14T13:57:37.703" Score="2" ViewCount="224" Body="&lt;p&gt;I want to build a segmentation to substitute the existing RFM segmentation which is a basic segmentation based on the Recency, Frequency and Monetary values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The new segmentation will be used for two purposes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Rank customers per value&lt;/li&gt;&#xA;&lt;li&gt;Have homogeneous groups of customers, so there should be an interpretation and meaning for each group.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think this shouldn't be an unsupervised learning, but a mix of both supervised and unsupervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So What I did is that I defined a target variable that takes 1 if the customer is active in the next period and 0 if not. then I used a decision tree to predict this value, so that the leaves represent the clusters. the percentage of active customers in each group is used to rank these clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think that's a good solution and do you have other ideas ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance &lt;/p&gt;&#xA;" OwnerUserId="20392" LastEditorUserId="20392" LastEditDate="2016-06-15T15:27:29.503" LastActivityDate="2016-09-22T15:12:03.247" Title="Predicting and clustering at the same time?" Tags="|machine-learning|data-mining|clustering|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12191" PostTypeId="2" ParentId="8549" CreationDate="2016-06-14T14:18:39.453" Score="20" Body="&lt;p&gt;You can manage Spark memory limits programmatically (by the API). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As SparkContext is already available in your Notebook:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sc._conf.get('spark.driver.memory')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can set as well, but you have to shutdown the existing SparkContext first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conf = SparkConf().setAppName(&quot;App&quot;)&#xA;conf = (conf.setMaster('local[*]')&#xA;        .set('spark.executor.memory', '4G')&#xA;        .set('spark.driver.memory', '45G')&#xA;        .set('spark.driver.maxResultSize', '10G'))&#xA;sc = SparkContext(conf=conf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If your workload is the same for all analysis, then editing &lt;strong&gt;spark-defaults.conf&lt;/strong&gt; as cited above is the way to go.&lt;/p&gt;&#xA;" OwnerUserId="1368" LastActivityDate="2016-06-14T14:18:39.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12192" PostTypeId="2" ParentId="9416" CreationDate="2016-06-14T14:38:07.307" Score="1" Body="&lt;p&gt;If the features scale of measurement and/or number of categories vary, permutation accuracy is more reliable than the GINI index (Strob et al. 2007). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, better yet is to use Boruta all relevant variable selection prior to running random forests to determine which variables are relevant to the target property, and which are not. This allows you to trim your dataset to only useful, or relevant variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strobl, Carolin, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.” BMC Bioinformatics 8 (2007): 25. doi:10.1186/1471-2105-8-25.&lt;/p&gt;&#xA;" OwnerUserId="20529" LastActivityDate="2016-06-14T14:38:07.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12193" PostTypeId="2" ParentId="12128" CreationDate="2016-06-14T21:31:59.370" Score="0" Body="&lt;p&gt;There should be a distinction between the terms of &lt;em&gt;neural networks&lt;/em&gt; and &lt;em&gt;deep neural networks&lt;/em&gt;. I found this link: &lt;a href=&quot;https://www.quora.com/How-does-deep-learning-work-and-how-is-it-different-from-normal-neural-networks-applied-with-SVM&quot; rel=&quot;nofollow&quot;&gt;https://www.quora.com/How-does-deep-learning-work-and-how-is-it-different-from-normal-neural-networks-applied-with-SVM&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, there is no general rule and probably will never be one that says that you should use a certain type of network &quot;a priori&quot;. It depends on the task you are trying to solve, but you already knew that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose you are wondering what are the &lt;strong&gt;meanings&lt;/strong&gt; of the additional hidden layers, how to figure out how many of them should be there and how large they should be. Also, how the additional parameter &lt;strong&gt;affects&lt;/strong&gt; this network and how to think through your model when something doesn't work right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, it depends. There are lots of types of networks that are good for additional problems. For example, LSTMs are good when working with series of data. If you want to tune your LSTM to work well, you should go INTO it and figure out how EXACTLY does it work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend this playground site for neural networks:&#xA;&lt;a href=&quot;http://playground.tensorflow.org&quot; rel=&quot;nofollow&quot;&gt;http://playground.tensorflow.org&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also suggest &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/how_tos/graph_viz/index.html&quot; rel=&quot;nofollow&quot;&gt;https://www.tensorflow.org/versions/r0.9/how_tos/graph_viz/index.html&lt;/a&gt; to inspect your code's architecture. Additionally, try to explore the TensorFlow documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, I advise you to read. Read more. (&lt;a href=&quot;http://karpathy.github.io/&quot; rel=&quot;nofollow&quot;&gt;http://karpathy.github.io/&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="19260" LastEditorUserId="19260" LastEditDate="2016-06-15T17:06:12.323" LastActivityDate="2016-06-15T17:06:12.323" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12194" PostTypeId="1" CreationDate="2016-06-15T01:43:41.887" Score="1" ViewCount="81" Body="&lt;p&gt;From my machine learning class,it seems L1 norm regularization is the standard way to obtain sparse and probably better fit for machine learning problems. But from pybrain, it seems L1 norm regularization is not implemented, rather L2 regularization is implemented which is quite weird to me...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is implemented as followed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trainer = BackpropTrainer(net, ds, weightdecay=0.01)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it possible for me to implement L1 norm regularization in pybrain? Even if I need to modify the source code? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="20551" LastActivityDate="2016-06-15T01:43:41.887" Title="L1 regularization in pybrain" Tags="|machine-learning|regularization|pybrain|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12195" PostTypeId="1" AcceptedAnswerId="12198" CreationDate="2016-06-15T01:45:41.930" Score="3" ViewCount="631" Body="&lt;p&gt;Deep Learning is an excellent model for classification problem such as image recognition or object detection. Can we use deep learning for regression problems - Time Series prediction ? So if it can, how can we build structure of deep learning. I mean how to building layer to extract features from time series.&lt;/p&gt;&#xA;" OwnerUserId="12285" LastActivityDate="2016-06-15T13:02:04.223" Title="Deep Learning for Time series" Tags="|predictive-modeling|time-series|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12196" PostTypeId="2" ParentId="12124" CreationDate="2016-06-15T04:50:32.150" Score="0" Body="&lt;p&gt;For the given problem I would apply constrained Hierarchical clustering in R. That is more efficient in this case. If you like to apply Hierarchical clustering the package is &quot;rioja&quot; and the function that you can use is chclust().&lt;br&gt;&#xA;This is not constrained K-means implementation in R but this should solve your actual problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two types of constrained in chclust() function, you can set those with the help of method parameter in function. This will add the constraint on the basis of sum of square.You can read in detail about these constraint with simple search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This chclust function will produce the constraint object and you can check the proper number of group by using other functions in &quot;rioja&quot; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A practical implementation of same can be checked on below link:&#xA;&lt;a href=&quot;https://gist.github.com/benmarwick/7097120&quot; rel=&quot;nofollow&quot;&gt;https://gist.github.com/benmarwick/7097120&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="20517" LastEditDate="2016-06-22T18:19:05.967" LastActivityDate="2016-06-22T18:19:05.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12197" PostTypeId="2" ParentId="11450" CreationDate="2016-06-15T05:04:55.700" Score="1" Body="&lt;p&gt;You need to create a dissimilarity matrix first and then apply the clustering technique. Below link has many answer referring your point. Hope below will suffice for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/24540/clustering-of-mixed-type-data-with-r&quot;&gt;https://stats.stackexchange.com/questions/24540/clustering-of-mixed-type-data-with-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-06-15T05:04:55.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12198" PostTypeId="2" ParentId="12195" CreationDate="2016-06-15T05:27:16.510" Score="7" Body="&lt;p&gt;Recurrent neural networks (RNNs) can work with series as input or output or both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even a simple one-layer RNN is effectively &quot;deep&quot; because it has to solve similar problems as multi-layer non-recursive networks. That is because backpropagation logic in a RNN has to account for delay between input and target, which is solved by &lt;a href=&quot;https://en.wikipedia.org/wiki/Backpropagation_through_time&quot; rel=&quot;noreferrer&quot;&gt;backpropagation through time&lt;/a&gt; - essentially adding a layer to the network for every time step of delay between first input and last output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RNN architecture has become more sophisticated in recent years by using &quot;gating&quot; techniques such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Gated_recurrent_unit&quot; rel=&quot;noreferrer&quot;&gt;Gated Recurrent Units&lt;/a&gt; (GRU) or &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;noreferrer&quot;&gt;Long Short Term Memory&lt;/a&gt; (LSTM). These have multiple trainable params - 3 or 4 - per neuron, and the schematics are more complicated than feed-forward networks. They have been demonstrated as &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;noreferrer&quot;&gt;very effective in practice&lt;/a&gt;, so this extra complexity does seem to pay off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although you can research and implement RNNs yourself in a library like Theano or Tensor Flow, several neural network libraries already implement RNN architectures (e.g. &lt;a href=&quot;http://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt;, &lt;a href=&quot;https://github.com/jcjohnson/torch-rnn&quot; rel=&quot;noreferrer&quot;&gt;torch-rnn&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-06-15T13:02:04.223" LastActivityDate="2016-06-15T13:02:04.223" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12200" PostTypeId="2" ParentId="12127" CreationDate="2016-06-15T07:09:11.703" Score="6" Body="&lt;p&gt;The problem you are facing is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;noreferrer&quot;&gt;time series&lt;/a&gt; problem.&#xA;Your events are categorial which is a specific case (so most common techniques like &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;noreferrer&quot;&gt;arima&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Fourier_transform&quot; rel=&quot;noreferrer&quot;&gt;Fourier transform&lt;/a&gt; are irrelevant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before getting into the analysis, try to find out whether the events among nodes are independent. If they are independent, you can break them into sequences per node and analyze them. If they are not (e.g., &quot;Main power supply has a fault alarm&quot; on node x indicates the same event on node y) you should the combined sequence. Sometime even when the the sequence are dependent you can gain from using the per node sequence as extra data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You data set is quite large, which means that computation will take time. You data is probably noisy, so you will probably have some mistakes. Therefore, I recommend advancing in small steps from simple models to more complex ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with descriptive statistics, just to explore the data. How many events do you have? How common are they? What is the probability of the events that you try to predict? Can you remove some of the events as meaningless (e.g., by using domain knowledge)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you have domain knowledge that indicates that recent events are the important ones, I would have try predicting based on the n last events. Start with 1 and grow slowly since the number of combinations will grow very fast and the number of samples you will have for each combination will become small and might introduce errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incase that the important event are not recent, try to condition on these events in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases such simple model will help you get a bit above the baseline but not too much. Then you will need more complex models. I recommend using &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;noreferrer&quot;&gt;association rules&lt;/a&gt; that fit your case and have plenty of implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can further advance more but try these technique first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The techniques mentioned before will give you a model the will predict the probability that a node will be down, answering your question (ii). Running it on the sequence of the nodes will enable you to predict the number of nodes that will fail answering question (i).&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-06-15T07:09:11.703" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12202" PostTypeId="1" AcceptedAnswerId="12323" CreationDate="2016-06-15T08:34:35.197" Score="4" ViewCount="682" Body="&lt;p&gt;There is a data mining competition website called &lt;a href=&quot;https://numer.ai/&quot; rel=&quot;nofollow&quot;&gt;numer.ai&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Presumably, behind the website is a hedge fund which makes use of the predictions that people send. People within the 100th top places continuously make money, until the next dataset is revealed, and the competition resets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is that websites like Kaggle avoid overfitting by having a public and a private leaderboard. The private leaderboard is only revealed at the end of the competition and only then are prizes handed out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numerai says under &lt;a href=&quot;https://numer.ai/rules&quot; rel=&quot;nofollow&quot;&gt;rules&lt;/a&gt; that it uses the same approach. Quoting:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If models overfit the public leaderboard and perform poorly on the&#xA;  private leaderboard, those users will suffer penalties potentially&#xA;  eradicating all gains. This discourages overfitting.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What do they mean by &quot;penalties&quot; being applied? I know someone who is making money on that website by continuously playing the public leaderboard. Does this mean that if he does not withdraw his bitcoin he is in danger of losing what he has earned in case he is overfitting?&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="11097" LastEditDate="2016-06-15T09:53:23.573" LastActivityDate="2016-06-21T10:46:01.563" Title="numer.ai: how does their leaderboard system work?" Tags="|data-mining|competitions|overfitting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12206" PostTypeId="1" AcceptedAnswerId="12208" CreationDate="2016-06-15T11:30:33.413" Score="8" ViewCount="841" Body="&lt;p&gt;I have to think about a model to identify prospects (companies) that have a high chance of being converted into clients, and I'm looking for advice on what kind of model could be of use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The databases I will have are, as far as I know (I don't have them yet), the &lt;code&gt;list of current clients&lt;/code&gt; (in other words, &lt;code&gt;converted prospects&lt;/code&gt;) and their features (&lt;code&gt;size&lt;/code&gt;, &lt;code&gt;revenue&lt;/code&gt;, &lt;code&gt;age&lt;/code&gt;, &lt;code&gt;location&lt;/code&gt;, stuff like that), and a &lt;code&gt;list of prospects&lt;/code&gt; (that I have to score) and their features. However, I don't think I'll have a list of the companies that used to be prospects but for which the conversion to clients failed (if I had, I think I could have opted for a random forest. Of course I could still use a random forest, but I feel it would be a bad idea to run a random forest on the union of my two databases, and treat the clients as &lt;code&gt;converted&lt;/code&gt; and the prospects as &lt;code&gt;non-converted&lt;/code&gt;...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I need to find, in the list of prospects, those who look like the already existing clients. What kind of model can I use to do that ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm also thinking about things such as &quot;evaluating the value of the clients and apply this to the similar prospects&quot;, and &quot;evaluating the chance each prospect has of going out of business&quot; to further refine the value of my scoring, but it's kinda out of the scope of my question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="18306" LastEditorUserId="18306" LastEditDate="2016-12-18T14:39:40.623" LastActivityDate="2016-12-18T14:39:40.623" Title="Ideas for prospect scoring model" Tags="|machine-learning|predictive-modeling|supervised-learning|unsupervised-learning|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12207" PostTypeId="1" CreationDate="2016-06-15T11:38:50.670" Score="0" ViewCount="1426" Body="&lt;p&gt;I have a dataset of over 67 000 records, and I'm trying to run a k-means cluster analysis on that. Orange returns a memory error.&#xA;The data is in an excel file, but I also tried to load it from a csv. Omitting some records and downsizing it under 65k records also couldnt help.&#xA;Hierarchical clustering also doesnt work for the same reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any ideas on what to do here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error code:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MemoryError&lt;br&gt;&#xA;  Traceback (most recent&#xA;  call last):&lt;br&gt;&#xA;  File&#xA;  &quot;C:\Python34\lib\site-packages\Orange\widgets\gui.py&quot;, line 2220, in&#xA;  &#xA;      b.button = btn = button(b, master, label, callback=lambda: do_commit())   File&#xA;  &quot;C:\Python34\lib\site-packages\Orange\widgets\gui.py&quot;, line 2191, in&#xA;  do_commit&#xA;      commit()&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py&quot;,&#xA;  line 246, in run&#xA;      self.run_optimization()&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py&quot;,&#xA;  line 225, in run_optimization&#xA;      self.optimization_runs.append((k, kmeans(self.data)))&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\Orange\projection\base.py&quot;, line 28, in&#xA;  &lt;strong&gt;call&lt;/strong&gt;&#xA;      clf = self.fit(data.X, data.Y)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\Orange\clustering\kmeans.py&quot;, line 23,&#xA;  in fit&#xA;      proj = proj.fit(X, Y)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py&quot;, line 821,&#xA;  in fit&#xA;      n_jobs=self.n_jobs)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py&quot;, line 277,&#xA;  in k_means&#xA;      tol = _tolerance(X, tol)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py&quot;, line 162,&#xA;  in _tolerance&#xA;      variances = np.var(X, axis=0)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\numpy\core\fromnumeric.py&quot;, line 2938,&#xA;  in var&#xA;      keepdims=keepdims)&lt;br&gt;&#xA;  File &quot;C:\Python34\lib\site-packages\numpy\core_methods.py&quot;, line 102, in&#xA;  _var&#xA;      x = asanyarray(arr - arrmean)&#xA;  MemoryError&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="20565" LastEditorUserId="20565" LastEditDate="2016-06-16T06:43:01.687" LastActivityDate="2016-07-01T14:36:07.653" Title="Orange, K-means clustering, memory error" Tags="|data-mining|clustering|k-means|orange|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12208" PostTypeId="2" ParentId="12206" CreationDate="2016-06-15T12:03:56.500" Score="8" Body="&lt;p&gt;I faced almost exactly the same scenario a year and a half ago -- basically what you have is a variation of the &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-class classification&lt;/a&gt; (OCC) problem, specifically &lt;a href=&quot;https://www.cs.uic.edu/~liub/NSF/PSC-IIS-0307239.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PU-learning&lt;/a&gt; (learning from Positive and Unlabelled data). You have your known, labelled positive dataset (&lt;strong&gt;clients&lt;/strong&gt;) and an un-labelled dataset of &lt;strong&gt;prospects&lt;/strong&gt; (&lt;em&gt;some of which are client-like and some of which are not client-like&lt;/em&gt;). Your task is to identify the most client like of the prospects and target them... this hinges on the assumption that &lt;em&gt;prospects that look most like clients&lt;/em&gt; are &lt;strong&gt;more likely to convert&lt;/strong&gt; than &lt;em&gt;prospects that look less like clients&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach we settled upon used a procedure called the &lt;a href=&quot;https://www.cs.uic.edu/~liub/S-EM/unlabelled.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spy-technique&lt;/a&gt;. The basic idea is that you take a sample from your known positive class and inject them into your unlabelled set. You  then train a classifier on this combined data and then run the unlabelled set back through the trained classifier assigning each instance a probability of being a positive class member. The intuition is that the injected positives (&lt;em&gt;so-called spies&lt;/em&gt;) should behave similarly to the positive instances (as reflected by their posterior probabilities). By setting a threshold this allows you to extract reliable negative instances from the unlabelled set. Now, having both positive and negative labelled data, and you can build a classifier using any standard classification algorithm you choose. In essence, with the spy technique, you boot-strap your data to provide you with the needed negative instances for proper training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZlLJG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZlLJG.png&quot; alt=&quot;spy technique&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For starters you should look into the work of Li and Liu who have a number of papers exploring the topic of OCC and PU-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1lWd4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1lWd4.png&quot; alt=&quot;OCC and PU-learning Papers&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-06-16T13:57:48.310" LastActivityDate="2016-06-16T13:57:48.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12209" PostTypeId="2" ParentId="11770" CreationDate="2016-06-15T13:18:18.747" Score="2" Body="&lt;p&gt;As with anything, it depends. :) Your question will very much depend on the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is reminiscent of the broader debate about &lt;a href=&quot;https://stackoverflow.com/questions/879432/what-is-the-difference-between-a-generative-and-discriminative-algorithm&quot;&gt;discriminative versus generative models&lt;/a&gt;. Usually, we work with discriminative models. That is, you model $\text{P}(Y=1\,|\,X_1, X_2, \dots)$, where $Y$ are your classes and $X_i$ your observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is another class of models: generative models, like &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Restricted Boltzmann Machines&lt;/a&gt;, which model joint distributions, $\text{P}(Y,X_1, X_2, \dots)$. They can be used for classification because you can easily turn a joint probability into a conditional probability. These models usually are broader in scope, and usually do not perform as well, which is why they aren't used as much. But they do exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you are talking about discriminative models, like decision trees or the logistic regression, then what you want is how to &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_probability#Assuming_conditional_probability_is_of_similar_size_to_its_inverse&quot; rel=&quot;nofollow noreferrer&quot;&gt;invert this conditional probability&lt;/a&gt;, so that you have a distribution along $X_i$; that is, $\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Bayes's theorem, $\text{P}(Y=1\,|\,X_1=x_1, X_2=x_2, \dots)=\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y=1)\frac{\text{P}(Y=1)}{\text{P}(X_1=x_1, X_2=x_2, \dots)}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, yes, you can. Where is the catch?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X$ and $Y$ have to be discrete. It is very easy to create a probability density function from categorical variables by just getting conditional probabilities out of your model by just trying all combinations of $X$. And, once you have a probability density function, then you can just use a sampling technique to generate values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is less clear to me how you could make this work for continuous values, unless you are willing to use an approximation by discretizing your observations. I think an exact solution will depend very much on the discriminative model at hand. You cannot do this alone by getting conditional probabilities out of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if sampling is important for you, then you should just use a generative model.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2016-06-15T14:41:47.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12212" PostTypeId="2" ParentId="8871" CreationDate="2016-06-15T14:02:52.080" Score="0" Body="&lt;p&gt;I'm not sure how well this approach is going to work for you, but it worked for me in my scenario of propensity scoring. So, if it makes sense, try to relate your data to mine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a list of customers and their purchase behaviors. From those purchase behaviors I deduced association rules to determine which customer is likely to purchase in what product category (based on their previous purchases- associated with the previous purchases of the group). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also include recency and frequency of the product purchases into the model to decide whether or not to recommend a particular product/offer to the customer for a specific time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the confidence, support and lift metrics of those rules for each customer, I mapped them to highly likely to purchase in this category with confidence being their propensity score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, based on your data and the task you want to achieve, every model hass to customized a little. This is only an idea that worked for me.&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-06-15T14:02:52.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12213" PostTypeId="2" ParentId="8172" CreationDate="2016-06-15T14:11:22.097" Score="1" Body="&lt;p&gt;If you have the purchase data for all these customers, this could be one way to approach the problem: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could either cluster the customers into natural groups based on their most recent purchases or find association rules (the likelihood of the customer purchasing in a particular category based on their most recent purchases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how well this approach is going to work for you, but it worked for me in my scenario of propensity scoring. So, if it makes sense, try to relate your data to mine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a list of customers and their purchase behaviors. From those purchase behaviors I deduced association rules to determine which customer is likely to purchase in what product category (based on their previous purchases- associated with the previous purchases of the group). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also include recency and frequency of the product purchases into the model to decide whether or not to recommend a particular product/offer to the customer for a specific time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the confidence, support and lift metrics of those rules for each customer, I mapped them to highly likely to purchase in this category with confidence being their propensity score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if this works with your data.&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-06-15T14:11:22.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12215" PostTypeId="1" AcceptedAnswerId="30875" CreationDate="2016-06-15T15:44:45.850" Score="5" ViewCount="3377" Body="&lt;p&gt;Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would one split an image of unknown size and an unknown amount of digits into a series of 28x28px images to feed to this network? (The order of the digits must be obtainable.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/haT9e.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/haT9e.png&quot; alt=&quot;String of digits&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this be split up to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MgVJ0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MgVJ0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that there are not always 5 digits and the initial image is not always the same size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, my thoughts would be to create a secondary neural network. This neural network would output an (x, y) coordinate. This could be used to crop the image to 28x28px about this coordinate. However, this neural network would only be able to locate 1 digit at a time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, another idea is that a series of random crops could be performed and then all given to the number recognising neural network. However, this would create a high error rate and the number recognising neural network has no way of telling that there was no valid number given (Unless that output was added). But more importantly the order of the digits would/could be lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am struggling to find any resource explaining a possible solution. Google's house number recognition takes the entire image of digits and returns a value. This can be found in &lt;a href=&quot;https://www.technologyreview.com/s/523326/how-google-cracked-house-number-identification-in-street-view/&quot; rel=&quot;nofollow noreferrer&quot;&gt;How Google Cracked House Number Identification in Street View&lt;/a&gt; and &lt;a href=&quot;http://arxiv.org/abs/1312.6082&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20571" LastEditorUserId="20571" LastEditDate="2018-05-09T23:48:53.850" LastActivityDate="2018-05-09T23:48:53.850" Title="How would one separate digits for number recognition?" Tags="|neural-network|preprocessing|image-recognition|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12216" PostTypeId="2" ParentId="12215" CreationDate="2016-06-15T16:40:01.560" Score="3" Body="&lt;p&gt;In Image Processing, this task is known as localization. You basically want to localize each digit in the image and then use your &lt;em&gt;digit recognizer&lt;/em&gt; over the digits. A cursory google search for &lt;a href=&quot;https://www.google.com/search?q=digit+localization+in+images&quot; rel=&quot;nofollow&quot;&gt;digit localization in images&lt;/a&gt; gives me following papers which seem to be very helpful.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf&quot; rel=&quot;nofollow&quot;&gt;Reading Digits in Natural Images with Unsupervised Feature Learning&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://research.microsoft.com/en-us/um/people/manik/pubs%5CdeCampos09.pdf&quot; rel=&quot;nofollow&quot;&gt;Character Recognition In Natural Images&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-06-15T16:40:01.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12219" PostTypeId="1" AcceptedAnswerId="12229" CreationDate="2016-06-15T18:59:58.047" Score="10" ViewCount="7613" Body="&lt;p&gt;I'm currently studying papers about outlier detection using RNN's (Replicator Neural Networks) and wonder what is the particular difference to Autoencoders? RNN's seem to be treaded for many as the holy grail of outlier/anomaly detection, however the idea seems to be pretty old to, as autoencoders have been there for a long while.&lt;/p&gt;&#xA;" OwnerUserId="18744" LastActivityDate="2016-06-16T09:04:03.307" Title="Difference: Replicator Neural Network vs. Autoencoder" Tags="|neural-network|anomaly-detection|autoencoder|outlier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12220" PostTypeId="1" CreationDate="2016-06-15T19:32:42.153" Score="1" ViewCount="36" Body="&lt;p&gt;I want to develop a new autoencoder. Once the network itself is developed, the customer wants to train and deploy the it all by himself. Due to data protection he can't give me anymore information, as advising me about the usage in transactional ERP data and the number of inputs/outputs, which will be 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually I would start a project by analysing the data and checking the distribution by running histogram. The initial check on the data usually gives me insight about &lt;strong&gt;Skewness&lt;/strong&gt;, &lt;strong&gt;max-values&lt;/strong&gt;, &lt;strong&gt;min-values&lt;/strong&gt;, &lt;strong&gt;averages&lt;/strong&gt; and &lt;strong&gt;medians&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly the skewness worries me, as in my experiences neural-networks such as auto-encoders perform extremely bad on such inputs. Usually I would offset a strongly skewed data distribution by using logarithms, root functions or power transformation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if I can just us any of these &quot;Anti-Skewness&quot; methods just blindly, in order to offset &quot;bad-data&quot; but not worrying about destroying the performance of my network.&lt;/p&gt;&#xA;" OwnerUserId="18744" LastEditorUserId="18744" LastEditDate="2016-06-15T20:24:58.460" LastActivityDate="2016-06-15T20:24:58.460" Title="Optimizing the distribution on an unknown of input / Data Audit Pattern" Tags="|machine-learning|autoencoder|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12222" PostTypeId="1" CreationDate="2016-06-16T01:40:15.160" Score="1" ViewCount="561" Body="&lt;p&gt;I have noticed that when you make a small decision tree model, and then extend the model by creating an ensemble of trees around the same tree settings, the variable importance is diluted in the sense that the least important and most important variables become a lot more closer together. In some cases, there may be almost do distinction in importance.&#xA;Are there methods available to either mitigate this effect or to measure it, with a view to defining any tradeoff between understanding variable importance and overall accuracy?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-08-15T19:24:39.930" Title="Variable importance in ensemble models" Tags="|random-forest|ensemble-modeling|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12223" PostTypeId="1" AcceptedAnswerId="12224" CreationDate="2016-06-16T02:06:04.567" Score="1" ViewCount="118" Body="&lt;p&gt;I'm not sure if this question is appropriate for this forum, so excuse me if it's not (if not, any suggestions on where might be a better place would be very much appreciated). I'm currently an undergrad in a quantitative field, and for the summer, I've been given an opportunity to do a data project by the company I am working for. I'm not really sure where to start. I had a conversation with one of the business owners today, in order to get a better handle on how the business works, and what kind of data they have. We talked a little bit about what sorts of questions they have, and what sort of things would be nice to know. I guess that seems to be the main question: What questions to ask? My initial thoughts are to first just look at the data via traditional descriptive stats methods (histograms, scatter plots etc....), and maybe that creates some ideas. If anyone has some tips, or even some good links (yes, I have already Googled it quite a bit), I would be grateful. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="20582" LastEditorUserId="20582" LastEditDate="2016-06-16T02:11:33.067" LastActivityDate="2016-06-16T13:28:32.637" Title="First steps when analyzing a company's data" Tags="|data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12224" PostTypeId="2" ParentId="12223" CreationDate="2016-06-16T02:21:29.190" Score="2" Body="&lt;p&gt;The thing you need to understand as completely as possible is how they expect a data analysis to enable them to achieve their objective. They are a business, so their overall objective is likely related to maximising profit. However, there will be a more immediate objective underneath that heading. To maximise profit you can either reduce costs or increase sales. In turn, to increase sales you can increase the number of customers or increase the amount of sales to each customer etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question then turns on how you can use data science to perform one those objectives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, questions that can almost be answered with data science could be 'how do I better identify potential customers?' or 'how do increase existing custmers' spend?' These are still very high level questions, but they are the sort of questions that you need to have in mind as you start to do your descriptive stats etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bear in mind that this is an iterative process and it is completely normal to start off in a fuzzy sort of area. At this stage it is almost the case that having a question in mind is a McGuffin - it will kick things off, but it may not be the question you end up answering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CRISP-DM process is a process that has been built for data mining that discusses how to iteratively use results from analyses and models to increase your understanding of the customer's situation, and hence drive the development of a better business objective for use in a data science project.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-16T02:21:29.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12225" PostTypeId="1" CreationDate="2016-06-16T05:56:16.480" Score="5" ViewCount="3288" Body="&lt;p&gt;I want to create a domain specific QA system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on a project to create an QA system for a given text book (on a specific domain and answer all related questions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach I am considering for this is to create an Ontology/Knowledge base and then use this for answer retrieval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I extract data from documents and populate this to the KB ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The question will be in natural language, how can I use this question to query from the knowledge base ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this the best approach ? because domain knowledge is required here for creating ontologies. And there are n number of domains. So shall I need to create different ontologies for each domain ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can a single QA model can be used to cover different domains ? Is there any other Machine learning and Deep learning approaches can be used for this ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5177" LastEditDate="2017-01-07T12:34:05.857" LastActivityDate="2017-02-06T13:29:08.863" Title="Creating Domain specific Question Answering Systems" Tags="|machine-learning|python|nlp|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12226" PostTypeId="1" CreationDate="2016-06-16T06:24:25.003" Score="4" ViewCount="988" Body="&lt;p&gt;I wanted to perform Twitter sentiment analysis on Twitter tweets. When I googled it, I found a &lt;a href=&quot;http://nlp.stanford.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;stanfordNLP&lt;/em&gt;&lt;/a&gt; code of sentiment class. I used it for finding Twitter sentiment later I came to know &lt;a href=&quot;https://gate.ac.uk/wiki/twitter-postagger.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;TwitterGate&lt;/em&gt;&lt;/a&gt; model is made for Twitter tweets only. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I replaced the &lt;em&gt;stanfordNLP&lt;/em&gt; model with &lt;em&gt;TwitterGate&lt;/em&gt; model. I did not find any difference in the sentiment results. Both models give the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I tested the same tweets with &lt;a href=&quot;http://monkeylearn.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Monkeylearn&lt;/em&gt;&lt;/a&gt; API and &lt;a href=&quot;http://www.datumbox.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Datumbox&lt;/em&gt;&lt;/a&gt; API sentiment, I got confused with results. In NLP and twitterGate most tweets sentiment is given as negative, where the same tweets sentiment is given as neutral in &lt;em&gt;Monkeylearn&lt;/em&gt; &amp;amp; &lt;em&gt;Datumbox&lt;/em&gt; APIs code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I know which model is giving the correct sentiment? &lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="31513" LastEditDate="2017-05-10T08:05:17.437" LastActivityDate="2017-07-11T10:12:15.297" Title="How do I assess which sentiment classifier is best for my project?" Tags="|machine-learning|sentiment-analysis|stanford-nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12227" PostTypeId="2" ParentId="12226" CreationDate="2016-06-16T07:23:08.067" Score="5" Body="&lt;p&gt;A couple of important points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;em&gt;ground truth&lt;/em&gt; for sentiment data sets is established by people. The best you can hope for from any trained ML classifier is that it closely matches the opinion of those people when it predicts on new data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your different sentiment analysis tools could well vary in quality, because they will have been built with different technologies at different times, and with different training data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are a couple of ways I can think of that you could assess the classifiers for your purpose:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Test against your own labelled test data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to source this yourself to ensure it does not overlap with training data used by any of the models you want to compare (if it did overlap then it gives a big advantage to any model that trained using it). You will need to collect ground truth data on sentiment for all the texts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before you test, it is a good idea to pick a metric that you care about for your intended use. Accuracy is not the only metric, but it is probably fine for your purpose. You might choose other metrics if there are different costs associated with incorrect classifications - e.g. if assigning negative sentiment needs to be done cautiously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then run each classifier across your test set and calculate the metric. The classifier with the best metric is your best guess at the one you should use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a large enough test set, you can split it into parts and get a measure of the estimate error in your metric. This will help you see whether differences in the test metrics are significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be the best approach, especially if you can source data that you know matches your project goals. However, it would take a long time and a lot of effort.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are in a hurry, you could just search for and use someone else's dataset e.g I found these ones on a quick web search: &lt;a href=&quot;https://inclass.kaggle.com/c/si650winter11&quot; rel=&quot;noreferrer&quot;&gt;a Kaggle competition&lt;/a&gt; and &lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/&quot; rel=&quot;noreferrer&quot;&gt;a free sample hosted by Niek Sanders&lt;/a&gt; (I have no idea of the quality of these - you should sample them and see if the data is useful for you). There is a risk though that these data sets were used to train one of your classifiers, so it would give you a false high rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Read up on associated reports and papers&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of the sentiment analysers should explain the model class and data used to train it in documentation. Quite often you can find papers comparing the different approaches and quoting accuracy scores on a standard data set. If you are lucky, you will find enough comparisons that you get some sense of which algorithms are considered &quot;cutting edge&quot; and which are out-dated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Collect text that is classified differently by algorithms you are considering, and get feedback&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A variation of (1), you can do this at any stage, and it might work well in the context of a live product. For all the data that is classified, log how different classifiers respond to it, and where there is discrepancy, save it for later assessment. You will need some people to read and help classify the text (ideally without seeing what the classifiers thought), and you can collect metric data over time and use the classifier that does best once you have collected enough examples to make a clear decision.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Ultimately what matters is the results of using your classifier in a project. Just labelling data has no inherent purpose. It is the consequences of those labels that matter. So you need to be driven by results. If you use one classifier and get feedback that the product is not performing well, you may be able to trial another one. If your product is used by many people, and enough are giving feedback of quality, you could even A/B test a couple of classifiers in production.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-06-16T07:23:08.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12228" PostTypeId="2" ParentId="12226" CreationDate="2016-06-16T07:31:01.493" Score="4" Body="&lt;p&gt;You can classify a few of the tweets yourself, and compare afterwards which of the two algorithmic results is closer to your classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without more information we cannot tell what these algorithms were doing. It may well be that they were just using different thresholds internally: Algo 1 decided that everything &gt; 60% threshold is &quot;positive&quot;, all &amp;lt; 30% &quot;negative&quot;, all in between is &quot;neutral&quot;. Algo 2 may have used 75%/25%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, Twitter messages are not really well-suited for sentiment analysis. They are too short and grammatically too messy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might look for emoticons inside the tweet-text only , that might work if you have lots+lots of tweets.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-06-16T07:31:01.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12229" PostTypeId="2" ParentId="12219" CreationDate="2016-06-16T09:04:03.307" Score="8" Body="&lt;p&gt;Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main difference seems to be the way how the input is compressed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plain autoencoders&lt;/strong&gt; squeeze the input through a hidden layer that has fewer neurons than the input/output layers.. that way the network has to learn a compressed representation of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Replicator neural networks&lt;/strong&gt; squeeze the data through a hidden layer that uses a staircase-like activation function. The staircase-like activation function makes the network compress the data by assigning it to a certain number of clusters (depending on the number of neurons and number of steps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R6vvi.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R6vvi.png&quot; alt=&quot;staircase-like activation function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://www.inf.u-szeged.hu/~ggabor/publlist/2004-isnn.pdf&quot; rel=&quot;noreferrer&quot;&gt;Replicator Neural Networks for Outlier Modeling in&#xA;Segmental Speech Recognition&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;RNNs were originally introduced in the field of data compression [5].&#xA;  Hawkins et al. proposed it for outlier modeling [4]. In both papers a&#xA;  5-layer structure is recommended, with a linear output layer and &lt;strong&gt;a&#xA;  special staircase-like activation function in the middle layer&lt;/strong&gt; (see&#xA;  Fig. 2). &lt;strong&gt;The role of this activation function is to quantize the&#xA;  vector of middle hidden layer outputs into grid points and so arrange&#xA;  the data points into a number of clusters.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-06-16T09:04:03.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12230" PostTypeId="1" CreationDate="2016-06-16T10:14:26.183" Score="4" ViewCount="168" Body="&lt;p&gt;I'm working on a project and i need to discover topics existing in a social media data set. For instance, i wanna extract the topics existing on 200K tweets. Any one recommend to me any machine learning algorithms?&lt;/p&gt;&#xA;" OwnerUserId="20590" LastActivityDate="2018-04-10T17:33:25.820" Title="How can I discover topics in a social media data-set?" Tags="|machine-learning|nlp|social-network-analysis|topic-model|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12231" PostTypeId="2" ParentId="12104" CreationDate="2016-06-16T11:04:19.810" Score="0" Body="&lt;p&gt;The problem may be is that you created data schema manually. You can erase wos database and just use CiteSpace. It creates dataschema automatically. Hope it will be OK.&lt;/p&gt;&#xA;" OwnerUserId="20593" LastActivityDate="2016-06-16T11:04:19.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12232" PostTypeId="2" ParentId="12230" CreationDate="2016-06-16T11:24:09.187" Score="2" Body="&lt;p&gt;You can take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow&quot;&gt;Latent Dirichlet Allocation&lt;/a&gt;. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter handles and probably URLs) before feeding it to the algorithm. The only really important parameter that you need to give it is the number of topics. This will depend on your population (are these random tweets, or only tweets from a specific subgroup/hashtag?) and you need to compare some settings. What you can do is print the most important words per topic and see if they indeed do belong together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are different languages in your tweets you need to deal with that beforehand, maybe classify them on language and only keep the English ones for example.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-16T11:24:09.187" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12233" PostTypeId="1" CreationDate="2016-06-16T12:25:35.350" Score="2" ViewCount="1387" Body="&lt;p&gt;I am trying to scrape some data from a website with very little success. Basically there is a route overlaid on google maps and whenever you mouse over specific sections of the map (about 200 in all) it fetches 7 fields from a database and displays them on screen. Doing a single map manually would take about 30 minutes and be quite imprecise. There are about 10,000 map routes I want to scrape so this is not realistic to do it manually. Is there a relatively straightforward way of automating this process?&lt;/p&gt;&#xA;" OwnerUserId="20596" LastActivityDate="2016-06-16T18:04:37.333" Title="Scraping Mouse Over Generated Data" Tags="|scraping|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12234" PostTypeId="1" AcceptedAnswerId="12239" CreationDate="2016-06-16T12:54:34.747" Score="6" ViewCount="3204" Body="&lt;p&gt;I am currently working on a dataset to predict customer attrition based on past data and transactions of the customers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2,40,000 customers in total out of which around 1,77,000 customers are active(as of today) while the remaining ones are inactive (6300).&#xA;This is how sample headers look like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;custID|custAge|custGender|TQuantity|TVolume|TValue|TAmount|HolidayStatus|...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Overall, I have 40 predictors which include customer details, transaction details, item details etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data obviously has more active customers than inactive customers i.e. &lt;strong&gt;inactive customers form only 2.6% of the entire customer base&lt;/strong&gt;. Due to this, there are more transactions conducted by active customers(25million/32million) than by inactive (previously active) (6million/32million) ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite this, I created a logistic regression model using random data (&lt;code&gt;shuf -n 500000 data.csv&lt;/code&gt;). The model achieves 96.69% base accuracy in predicting when fed with random data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem:&lt;/strong&gt; How to make the model predict &lt;em&gt;with greater accuracy&lt;/em&gt; on such a biased dataset? or How do I sample the data more appropriately?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model prediction:&lt;/strong&gt; With 99.7% probablity, it predicts that the customer will be active whereas the customer is inactive&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Changing threshold won't help much&lt;/p&gt;&#xA;" OwnerUserId="18369" LastActivityDate="2016-06-16T17:58:05.033" Title="Logistic regression on biased data" Tags="|machine-learning|r|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12236" PostTypeId="2" ParentId="12223" CreationDate="2016-06-16T13:28:32.637" Score="0" Body="&lt;p&gt;I asked a similar question a couple weeks ago and got some good feedback.  One of the answers to my question is also from Robert de Graaf, here's the link:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/12003/tips-for-a-new-data-scientist&quot;&gt;Tips for a new data scientist&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the first task is often just collecting the data from various sources and cleaning it.  Based on my limited experience, I think that is a good place to start and can actually take quite a bit of time.  Once the data is organized, moving on to data visualization/exploration will help you get a feel for the data.  Kaggle has some good data visualization tutorials and I will post one if I find it.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-06-16T13:28:32.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12237" PostTypeId="1" CreationDate="2016-06-16T14:53:13.890" Score="7" ViewCount="25862" Body="&lt;p&gt;This is using python with Spark 1.6.1 and dataframes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have timestamps in UTC that I want to convert to local time, but a given row could be in any of several timezones. I have an 'offset' value (or alternately, the local timezone abbreviation. I can adjust all the timestamps to a single zone or with a single offset easily enough, but I can't figure out how to make the adjustment dependent on the 'offset' or 'tz' column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There appear to be 2 main ways of adjusting a timestamp:&#xA;using the 'INTERVAL' method, or using &lt;code&gt;pyspark.sql.from_utc_timestamp&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = [&#xA;  (&quot;2015-01-01 23:59:59&quot;, &quot;2015-01-02 00:01:02&quot;, 1, 300,&quot;MST&quot;),  &#xA;  (&quot;2015-01-02 23:00:00&quot;, &quot;2015-01-02 23:59:59&quot;, 2, 60, &quot;EST&quot;),  &#xA;  (&quot;2015-01-02 22:59:58&quot;, &quot;2015-01-02 23:59:59&quot;, 3, 120,&quot;EST&quot;),  &#xA;  (&quot;2015-03-02 15:59:58&quot;, &quot;2015-01-02 23:59:59&quot;, 4, 120,&quot;PST&quot;),  &#xA;  (&quot;2015-03-16 15:15:58&quot;, &quot;2015-01-02 23:59:59&quot;, 5, 120,&quot;PST&quot;),  &#xA;  (&quot;2015-10-02 18:59:58&quot;, &quot;2015-01-02 23:59:59&quot;, 4, 120,&quot;PST&quot;),  &#xA;  (&quot;2015-11-16 18:58:58&quot;, &quot;2015-01-02 23:59:59&quot;, 5, 120,&quot;PST&quot;),  &#xA;  (&quot;2015-03-02 15:59:58&quot;, &quot;2015-01-02 23:59:59&quot;, 4, 120,&quot;MST&quot;),  &#xA;  (&quot;2015-03-16 15:15:58&quot;, &quot;2015-01-02 23:59:59&quot;, 5, 120,&quot;MST&quot;),  &#xA;  (&quot;2015-10-02 18:59:58&quot;, &quot;2015-01-02 23:59:59&quot;, 4, 120,&quot;MST&quot;),  &#xA;  (&quot;2015-11-16 18:58:58&quot;, &quot;2015-01-02 23:59:59&quot;, 5, 120,&quot;MST&quot;),&#xA;  ...&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I realize the offset and tz columns aren't consistent - this isn't real data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = sqlCtx.createDataFrame(data, [&quot;start_time&quot;, &quot;end_time&quot;, &quot;id&quot;,&quot;offset&quot;,&quot;tz&quot;])  &#xA;from pyspark.sql import functions as F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;these two options both do what is expected:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.withColumn('testthis', F.from_utc_timestamp(df.start_time, &quot;PST&quot;)).show()&#xA;df.withColumn('testThat', df.start_time.cast(&quot;timestamp&quot;) - F.expr(&quot;INTERVAL 50 MINUTES&quot;)).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But if I try to replace the &quot;PST&quot; string with df.tz, or the &quot; 50 &quot; string with df.offset.cast('string'), I get a type error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;TypeError: 'Column' object is not callable&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've tried variations on this, but to no avail. &lt;/p&gt;&#xA;" OwnerUserId="20600" LastEditorUserId="381" LastEditDate="2016-08-02T22:59:30.937" LastActivityDate="2018-09-04T17:39:17.213" Title="Pyspark coverting timestamps from UTC to many timezones" Tags="|pyspark|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12238" PostTypeId="2" ParentId="12234" CreationDate="2016-06-16T16:34:47.907" Score="4" Body="&lt;p&gt;While there are well known techniques like down sampling for dealing with imbalanced classes (this is common in the finance industry where bankruptcies occur around only 1-3% of the time), I think in your case the model type will be much more important. If you are prioritizing explanatory power over predictive power or if you have a small data set, then logistic regression is fine. However, if you have a large dataset (which you have) and if you care more about predicting (which it sounds like you do) then you should pick a more advanced type of model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some good examples: random forest, gradient boosting, and support vector machines. Tree based methods likes gradient boosting and random forests are able to identify variable interactions that you would have discover manually / yourself if you went with logistic regression. These models are easily accessible in R (caret), Python (scikit-learn), or Java/Scala (Spark's ML lib or Weka). &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-16T16:34:47.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12239" PostTypeId="2" ParentId="12234" CreationDate="2016-06-16T17:58:05.033" Score="9" Body="&lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll start with some background to help you research the solution yourself and then will add some specifics.  What you refer to as &quot;biased data&quot; is more commonly known as &lt;code&gt;unbalanced classes&lt;/code&gt; in the data science world.  Also &quot;customer turnover&quot; is often referred to as &lt;code&gt;churn&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Metrics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As hoards of &lt;a href=&quot;https://class.coursera.org/ml-003/lecture&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ng'ian devotees&lt;/a&gt; will undoubtably point out you need to start by designing a set of metrics that work better with &lt;code&gt;unbalanced classes&lt;/code&gt; than &lt;code&gt;accuracy&lt;/code&gt;. Accuracy does a poor job in testing the quality of predictions for unbalanced classes e.g. a cancer test for a cancer that occurs in 0.05 % of the population is 99.95% accurate if it always predicts &quot;no cancer&quot;. I suggest using the &lt;a href=&quot;https://datascience.stackexchange.com/questions/6817/clasification-roc-curve-with-very-high-number-of-false-negatives/6822#6822&quot;&gt;&lt;code&gt;F1-score&lt;/code&gt; as the key metric in cross-validating your model&lt;/a&gt;.  The &lt;code&gt;F1-score&lt;/code&gt; is the harmonic mean of &lt;code&gt;precision&lt;/code&gt; and &lt;code&gt;recall&lt;/code&gt; and tends to work both for balanced and unbalanced classes.  There are other rations of harmonic mean, that could work in special cases, so &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;be aware of these&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.cornell.edu/courses/cs578/2003fa/performance_measures.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;There are other metrics you should learn about also&lt;/a&gt;. ROC-AUC is likely at the top of the list for other metrics you should understand and know about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model Selection and Cross-Validation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beginning a classification task with &lt;code&gt;Logistic Regression&lt;/code&gt; is a fantastic strategy. I make a point to always use a linear regression for regression tasks and a logistic regression for classification tasks. The linear model provides significant insight into the &lt;code&gt;feature importance&lt;/code&gt; and helps frame the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But following this initial survey you should move on to other, more sophisticated models. Many will give you a litany of things to try. You should perhaps focus on one or two and develop the model while paying very careful attention to &lt;code&gt;bias&lt;/code&gt; and &lt;code&gt;variance&lt;/code&gt; as you &lt;code&gt;cross-validate&lt;/code&gt; and test your model. A &lt;a href=&quot;http://scikit-learn.org/stable/modules/learning_curve.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;full &lt;code&gt;bias-variance decomposition&lt;/code&gt;&lt;/a&gt; may be unnecessary, once you develop better intuition, but is a great place for newbs to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest starting with an &lt;code&gt;SVM&lt;/code&gt; and also eventually trying a &lt;code&gt;random forest&lt;/code&gt; or &lt;code&gt;naive Bayes&lt;/code&gt; model as this will traverse several regimes of model types (analogy, decision trees, bagging, Bayesian).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finally... Unbalanced Classes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two typical methods for dealing with unbalanced classes. These include &lt;a href=&quot;http://www.marcoaltini.com/blog/dealing-with-imbalanced-data-undersampling-oversampling-and-proper-cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;oversampling the minority class&lt;/a&gt;, and fixing the model by &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;altering the hyperplane (SVM)&lt;/a&gt; or &lt;a href=&quot;http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;changing priors (Bayes)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are &lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;lots&lt;/a&gt; of &lt;a href=&quot;https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning&quot;&gt;summaries&lt;/a&gt; of this &lt;a href=&quot;http://pages.stern.nyu.edu/~fprovost/Papers/skew.PDF&quot; rel=&quot;nofollow noreferrer&quot;&gt;problem and solution&lt;/a&gt; if you search for &quot;&lt;code&gt;unbalanced classes&lt;/code&gt;&quot;. &lt;strong&gt;But&lt;/strong&gt;, it can still be a tricky problem despite all the literature.  Good luck...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-06-16T17:58:05.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12240" PostTypeId="2" ParentId="12233" CreationDate="2016-06-16T18:04:37.333" Score="3" Body="&lt;p&gt;I've been building web scrapers for over 5 years, and I have to say that a web scraper is rarely &quot;relatively straightforward&quot; simply because of how idiosyncratic each website is. It usually takes a minimum of 10 hours per site to code your scraper if you know what you're doing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whenever you have to interact with the page that you're scraping, I recommend Selenium. It's open source and works with most major languages, including Python, Java, and Scala. Moving a mouse around is possible, but I think in your case it might be easier to directly call the javascript that is triggered by the mouse movements. Your web scraper would iterate over all of the hoverable html elements and call the on-hover javascript on each element. However, the devil is in the details, and you're going to need to post a lot of questions on Stackoverflow and do a lot of Googling before you get to a final solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard that Google has an API for their maps capability. That might be a lot easier. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-16T18:04:37.333" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12241" PostTypeId="2" ParentId="12222" CreationDate="2016-06-16T18:26:31.173" Score="2" Body="&lt;p&gt;I've been using tree-based enesembling methods such as random forests and gradient boosting for several years now, and I have to say that I've never seen that behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some packages measure variable importance solely based on trees' final splits rather than candidate/surrogate splits, so if you have two important inputs that are correlated, but one is consistently a tiny bit better than the other, the less important input might never get selected as a final split and thus look as important as some of the less valuable inputs. However, this phenomenon is independent of the number of trees/ensembling, so I don't think it fully explains the behavior you're seeing.  &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-16T18:26:31.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12242" PostTypeId="1" AcceptedAnswerId="12625" CreationDate="2016-06-16T19:25:21.973" Score="2" ViewCount="121" Body="&lt;p&gt;Someone wants me to try to see if I can establish a workflow for them using PowerBI and Azure's ML, then show them how to use it. The idea is to create kind of a seemless experience for them, so they done have to worry about coding or dealing with ETL, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now I can access Azure but I have to go through some red tape to access the cloud version of PowerBI. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone happen to know if these 2 product services can be directly linked, such that the person I'm trying to help wouldn't have to worry about importing or exporting files (assuming I pipe the data into Azure's ML for them)?&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-07-06T05:39:15.323" Title="Does MS PowerBI directly connect with Azure's ML workbench?" Tags="|azure-ml|powerbi|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12243" PostTypeId="1" CreationDate="2016-06-17T01:18:57.753" Score="1" ViewCount="2269" Body="&lt;p&gt;I've seen both of these techniques be used for image search. One difference I can think of is that autoencoders don't rely on labeled data. I'm not sure, but it seems logical therefore that they can possibly generate more discriminatory dimensions for the final vector-representation, given that you're no longer bound by the classifications from the labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my particular problem, I have labeled data, which is why I'm stuck between the 2. This is for non-CNN autoencoders.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the advantages/disadvantages of using Autoencoders over CNNs for image search?&lt;/p&gt;&#xA;" OwnerUserId="20616" LastEditorUserId="29169" LastEditDate="2021-02-16T15:41:05.807" LastActivityDate="2021-02-16T15:41:05.807" Title="What are the advantages/disadvantages of using Autoencoders over CNNs for image search?" Tags="|image-classification|convolutional-neural-network|autoencoder|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12244" PostTypeId="2" ParentId="12127" CreationDate="2016-06-17T05:06:23.403" Score="2" Body="&lt;p&gt;There will be so many follow steps (based on results in step 1), for the objectives you mentioned. I am mentioning starting steps to take analysis to next level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first objective, you can calculate simple conditional probability for every node for some time window time frame. Here, you will get a view how each node is affecting other nodes. Also, explore Bayesian network on the data.&#xA;For the second requirement, as recommended by Dan Levin, association rules are good point to start. To simplify process, you can start with two main events (may be mains failure and association path broken) from the available events. Fix the RHS in association rules to the two main events as mentioned. LHS will be events occurred before RHS events (consider some time window). Now run association rules on the data. You will be able to find some precursors for the two events considered. You can find implementation of association rules in Spark in the following paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R-Apriori: An Efficient Apriori based Algorithm on Spark&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link: &lt;a href=&quot;http://www.iith.ac.in/~mkaul/papers/pikm09-rathee.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.iith.ac.in/~mkaul/papers/pikm09-rathee.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, to use association rule for prediction, considering the following points will help&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There must be a time lag between LHS and RHS, i.e. time gap between antecedent (if one node has mains failure) and consequent of the rule (next event would be node down)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Prediction rule must have relatively stable confidence with respect to the time frame determined by application domain i.e. run association rules on overall 3 months data, you will get some rules with some confidence. Now run association rules by month wise and see the confidence of same rules, if confidence is consistent, you can use those rules in prediction with confidence &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For more details refer: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://link.springer.com/chapter/10.1007%2F11548706_11#page-1&quot; rel=&quot;nofollow&quot;&gt;http://link.springer.com/chapter/10.1007%2F11548706_11#page-1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use association rules for objective 1 by considering node 1 in LHS and consider all events after nodel 1 event in RHS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!!!&lt;/p&gt;&#xA;" OwnerUserId="9828" LastActivityDate="2016-06-17T05:06:23.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12246" PostTypeId="1" CreationDate="2016-06-17T07:12:06.417" Score="2" ViewCount="312" Body="&lt;p&gt;Let's say we want to histogram a finite set of measurements of some quantity. It is straight forward to calculate the usual statistical quantities for our sample such as the mean and the variance. Let's assume we can clean up our data by identifying outliers and moving them into underflow and overflow bins and thus, define more or less optimal min and max values for the plotting range. But how would one decide on the number and the size of bins? I would like to know if there are methods to find the optimal binning for the cases with fixed and variable bin sizes.&lt;/p&gt;&#xA;" OwnerUserId="20576" LastActivityDate="2019-04-05T11:50:34.003" Title="Are there any good general techniques for binning/histogramming arbitrary data?" Tags="|data-mining|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12247" PostTypeId="2" ParentId="12246" CreationDate="2016-06-17T07:37:59.413" Score="2" Body="&lt;p&gt;I don't know if this what you want but here is a way to calculate the number of bins.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Count the number of data points in your dataset.&lt;/li&gt;&#xA;&lt;li&gt;Take the square root of the number of data points and round up to determine the initial number of bins required: &lt;span class=&quot;math-container&quot;&gt;$InitialNumberOfBins = \sqrt{NumberOfDataPoints}$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Divide the specification tolerance &lt;span class=&quot;math-container&quot;&gt;$Max-Min\ value$&lt;/span&gt; by initial number of bins: &lt;span class=&quot;math-container&quot;&gt;$FinalNumberOfBins = (Max-Min\ value) / InitialNumberOfBins$&lt;/span&gt;. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17959" LastEditorUserId="29575" LastEditDate="2019-04-05T11:50:34.003" LastActivityDate="2019-04-05T11:50:34.003" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="12248" PostTypeId="1" AcceptedAnswerId="12313" CreationDate="2016-06-17T08:18:24.967" Score="6" ViewCount="1428" Body="&lt;p&gt;Given several wikipedia articles on different movies.&#xA;What are the different approaches to implement a QA system to answer different quires related to movies. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dataset : Wikipedia articles&#xA;&#xA;Input   : natural language query, eg : who directed terminator ?&#xA;&#xA;Output  : The Terminator is a 1984 American science fiction action film directed by James Cameron&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Different approaches available are :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 IR Based approaches&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In IR based approaches the similarities between query and passaeges are considered and best suitable passages are selected as answer (as Google does)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Ontology based &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Ontology based prior domain knowledge is required about the domain. And documents need to be mapped accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 Machine learning based &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In ML based approaches large training set of documents is needed for training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the other cons and pros of each approaches? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which of these approaches suits the best for this use case ? Also is there any other methods ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="5091" LastEditDate="2016-06-20T08:20:27.477" LastActivityDate="2016-06-20T21:34:16.910" Title="Approaches for implementing Domain specific Question answering System" Tags="|machine-learning|python|neural-network|nlp|deep-learning|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12249" PostTypeId="2" ParentId="12230" CreationDate="2016-06-17T12:02:10.130" Score="1" Body="&lt;p&gt;A different direction (although not necessarily a better one) is to cluster the texts you receive, perhaps using an algorithm that doesn't require many input parameters like the number of clusters.&#xA;Note that not any text clustering algorithm will do - some are optimized for clustering much longer texts. This paper gives a survey of short text clustering methods:&#xA;&lt;a href=&quot;http://www.charuaggarwal.net/text-cluster.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.charuaggarwal.net/text-cluster.pdf&lt;/a&gt;&#xA;It's not the newest, but it's a starting point. From experience, I agree with @Jan van der Vegt that it's recommended at the very least to look at English separately from other languages.&lt;/p&gt;&#xA;" OwnerUserId="12492" LastActivityDate="2016-06-17T12:02:10.130" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12250" PostTypeId="1" AcceptedAnswerId="12254" CreationDate="2016-06-17T12:32:37.780" Score="2" ViewCount="663" Body="&lt;p&gt;I have a classification problem. I want to reduce number of features to 4 (I have 30). I'm wondering why I get better result in classification when I use correlation based feature selection(cfs) first and then employ pca in comparison with just employing pca (the latter one is worse than the first one). It also should be mentioned that data loss in the second approach (just pca) 0.2-variance cover:0.8- and in the first one is 0.4 -variance coverd: 0.6!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="12345" LastActivityDate="2016-06-19T09:08:03.927" Title="Feature Selection and PCA" Tags="|machine-learning|feature-selection|correlation|pca|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12251" PostTypeId="2" ParentId="12147" CreationDate="2016-06-17T15:03:56.697" Score="2" Body="&lt;p&gt;I believe in your case, predicting claim is more important than no claim. As you said you have you got 70% Accuracy on the training data, most of the time you might be doing wrong predictions in claim case because of less records, comparatively, what I would suggest is to make the data set balance or select a random balanced data set (20% each of claim and non-clan) and train a model using previous techniques you have applied and test it on the remaining data. If possible use different error measures with respect to your business case such as giving weights to the outcomes. If the accuracy is not improved, you can implement GBM techniques on this data. Most of the times GBM makes better predictions because it increases the randomness (white noise) in residuals by decreasing the similarity among residuals. You can apply many different models on this data and check if the accuracy is improved, eventually we should be able to understand the model to explain someone why they should use this model. Moreover, if you use feature engineered data with different models, there is a high probability that you will do better than the previous models. However, this depends on your business understanding.&lt;/p&gt;&#xA;" OwnerUserId="20626" LastActivityDate="2016-06-17T15:03:56.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12253" PostTypeId="1" CreationDate="2016-06-17T19:19:06.887" Score="15" ViewCount="2709" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;precision&lt;/strong&gt; is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know their meaning but I don't know why it is called &lt;strong&gt;recall&lt;/strong&gt;? I am not a native-speaker of English. I know recall means remember, then I don't know the relevance of this meaning to this concept! maybe &lt;strong&gt;coverage&lt;/strong&gt; was better because it shows how many instances were covered...or any other term. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover &lt;strong&gt;sensitivity&lt;/strong&gt; is also insensible to me!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please help me to associate these words to the concept and have a sense of them?&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="3436" LastEditDate="2021-04-09T04:51:54.097" LastActivityDate="2021-04-09T04:51:54.097" Title="Performance measure: Why is it called recall and sensitivity?" Tags="|performance|terminology|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12254" PostTypeId="2" ParentId="12250" CreationDate="2016-06-17T19:23:16.447" Score="4" Body="&lt;p&gt;PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a toy example. Lets say I want to predict stock prices. Say I'm given four predictors: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Year-over-year earnings growth (relevant)&lt;/li&gt;&#xA;&lt;li&gt;Percent chance of rain (irrelevant)&lt;/li&gt;&#xA;&lt;li&gt;Humidity (irrelevant)&lt;/li&gt;&#xA;&lt;li&gt;Temperature (irrelevant)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If I apply PCA to this data set, the first principle component would relate to weather since 75% of the predictors are weather related. Is this principle component relevant? It's not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two options you've highlighted boil down to using CFS or not using it. The option that uses CFS does better because it explicitly selects variables that have relevance to the target variable.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-17T19:23:16.447" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12255" PostTypeId="2" ParentId="12250" CreationDate="2016-06-17T19:59:41.223" Score="0" Body="&lt;p&gt;Correlated Variables should be removed from PCA, as the variables together tend to exaggerate the effect they are expressing.  CFS selects uncorrelated subsets of variables.&lt;/p&gt;&#xA;" OwnerUserId="16145" LastActivityDate="2016-06-17T19:59:41.223" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12256" PostTypeId="2" ParentId="12253" CreationDate="2016-06-17T20:09:19.663" Score="4" Body="&lt;p&gt;Recall means to bring back or remember. The terminology comes from information retrieval where it's usually being applied to a result set from a query. I suppose the sense of it is, how much of the set of right answers was retrieved by the query? how much of it was recalled? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know if &quot;coverage&quot; is better or not. The word &quot;sensitivity&quot; is also used to mean the same thing as &quot;recall&quot;. In any event, these are just the standard words for these ideas.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-06-17T20:09:19.663" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12259" PostTypeId="1" AcceptedAnswerId="12260" CreationDate="2016-06-17T23:30:03.107" Score="2" ViewCount="2649" Body="&lt;p&gt;Why exactly does features being dependent on each other, features having high correlation with one another, mean that they would be redundant? Also, does PCA help get rid of redundant/irrelevant features or do we have to get rid of redundant/irrelevant features before running PCA on our dataset?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastEditorUserId="2723" LastEditDate="2016-06-18T07:14:59.543" LastActivityDate="2016-06-18T07:14:59.543" Title="feature redundancy" Tags="|feature-selection|dimensionality-reduction|pca|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12260" PostTypeId="2" ParentId="12259" CreationDate="2016-06-17T23:49:01.263" Score="1" Body="&lt;p&gt;For the sake of training, features that are highly correlated offer little training &quot;value&quot; as the presence/state of one value can always (or almost always) be used to determine the presence/state of the other. If this is the case there's no reason to add both features as having both will have little impact on the predictions - if A &quot;on&quot; = B &quot;off&quot;, and A &quot;off&quot; = B &quot;on&quot;, then all states can be represented by just learning off either A or B. This is greatly simplified, but the same is true for other highly correlated values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA can help reduce features, but in any case, if you've identified redundant or highly correlated features that will be of little use in training, it probably makes sense to eliminate them right away and then use PCA, or other feature importance metrics that can be generated by training off your full dataset, to further optimize your training feature set.&lt;/p&gt;&#xA;" OwnerUserId="18039" LastActivityDate="2016-06-17T23:49:01.263" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12262" PostTypeId="2" ParentId="12259" CreationDate="2016-06-18T02:01:20.823" Score="1" Body="&lt;p&gt;Redundant features &lt;em&gt;can&lt;/em&gt; be features that are &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow&quot;&gt;multicolinear&lt;/a&gt; (i.e. highly correlated), but more importantly they're measuring the same thing without a unique contribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, age and income might be highly correlated, but in some analyses they still have a unique effect in your model and may have conceptual differences that you want to captured for interpretation. OTOH, age and birth date are purely redundant in most use cases I can think of (though there are always exceptions, such as if season of birth is important).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can PCA help reduce redundancy?&lt;/strong&gt; Sure. It's one of at least dozens of techniques you could use for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way you use PCA for feature selection is to look at the factor loading on the principal components and determine which correlated variables are measuring the same principal component then pick the top 1 or few variables to represent that latent variable, eliminating highly correlated non-distinct features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Should you eliminate redundant features before PCA?&lt;/strong&gt; If you're going to use the principal components for prediction rather than feature elimination, then yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do one round of feature analysis involving PCA or other techniques and a second round to create latent variables for your model if you want to do both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some additional tools for feature selection:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Minimum Redundancy Maximum Relevance&lt;/li&gt;&#xA;&lt;li&gt;Correlation Feature Selection&lt;/li&gt;&#xA;&lt;li&gt;Canonical Correlations analysis&lt;/li&gt;&#xA;&lt;li&gt;Factor Analysis&lt;/li&gt;&#xA;&lt;li&gt;Use of a covariance matrix&lt;/li&gt;&#xA;&lt;li&gt;Singular Value Decomposition&lt;/li&gt;&#xA;&lt;li&gt;Variance Inflation Factors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-06-18T04:55:14.743" LastActivityDate="2016-06-18T04:55:14.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12263" PostTypeId="1" AcceptedAnswerId="12264" CreationDate="2016-06-18T02:48:46.153" Score="1" ViewCount="64" Body="&lt;p&gt;is the best way to create the most accurate classifier to train a bunch of classifying algorithms like ANN, SVM, KNN, etc, and test different parameters to get optimal parameters for each classifier, and see which classifier has the least testing error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it better to use ensemble method and choose the &quot;majority&quot; decision of different kinds of trained classifiers?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastEditorUserId="20635" LastEditDate="2016-06-18T03:18:38.500" LastActivityDate="2016-06-18T03:25:48.367" Title="how to choose classifer" Tags="|classification|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12264" PostTypeId="2" ParentId="12263" CreationDate="2016-06-18T03:25:48.367" Score="1" Body="&lt;p&gt;It's usually not that clear cut; there's typically not one universally best approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, there are some prototyped ensemble approaches that are supposed to always be better than their underlying component algorithms, notably Erin LeDell's binary ensemble classifier for H2O. However, even in those cases you still need to optimize the first stage algorithms for the ensemble to be universally better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus if you're willing to spend a lot of extra time, let's say 2 weeks for an ensemble instead of the 1 week it might take you for your single-stage algorithm, then it's possible (especially for binary classification) to find an ensemble that will definitely be better than your single-stage classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this is rarely the case and the way you framed the question implies there's a choice between &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;building 1 really good single-stage model, selected from many candidate models (and by the way be sure to avoid overfitting while making those selections)  and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;throwing an ensemble at the problem without completing #1 above for each component of the ensemble (or completing #1 but not also optimizing the 2nd stage of the ensemble)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If that's the decision then -- while there's no 1 universally right answer -- I'd say that in the &lt;strong&gt;vast&lt;/strong&gt; majority of the cases it's better to stick with #1.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-06-18T03:25:48.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12265" PostTypeId="1" CreationDate="2016-06-18T07:44:12.953" Score="5" ViewCount="2432" Body="&lt;p&gt;I am working on a project for a company which needs to &lt;strong&gt;&lt;em&gt;categorize customer e-mails&lt;/em&gt;&lt;/strong&gt; regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are &lt;strong&gt;3000&lt;/strong&gt;. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is &lt;strong&gt;very imbalanced&lt;/strong&gt;, the frequency of &lt;strong&gt;top 3 categories combines upto 2000&lt;/strong&gt; and there are 2 categories with frequencies ranging from 40-80.&#xA;I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created a supervised learning model using &lt;strong&gt;&lt;em&gt;LinearSVC&lt;/em&gt;&lt;/strong&gt; which gave around &lt;strong&gt;78%&lt;/strong&gt; on validation dataset, after which I used &lt;strong&gt;&lt;em&gt;XGBoost classifier&lt;/em&gt;&lt;/strong&gt; which increased the accuracy to &lt;strong&gt;80%&lt;/strong&gt;. Both of these models use &lt;strong&gt;&lt;em&gt;Tfidf Vectors&lt;/em&gt;&lt;/strong&gt; with unigrams and bigrams of the pre-processed(&lt;em&gt;stop words removed and stemmed text&lt;/em&gt;) as input.&#xA;I have tried &lt;strong&gt;&lt;em&gt;SMOTE, TruncatedSVD and SelectKBest with chi2&lt;/em&gt;&lt;/strong&gt; but they decreased the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of Machine Learning Models I have used &lt;em&gt;Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers&lt;/em&gt;. I have even tried &lt;em&gt;Feed Forward Neural Networks(MLP) and Recurrent Neural Networks&lt;/em&gt; for the classification but still XGBoost gives the best accuracy. I have also used &lt;em&gt;Topical Modeling using LDA&lt;/em&gt;. I generated top words for each category and tried finding &lt;em&gt;cosine similarities&lt;/em&gt; of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is what kind of &lt;strong&gt;features&lt;/strong&gt; should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to &lt;strong&gt;context aware feature extraction&lt;/strong&gt; or somehow engineer the features(add more or enhance them) to &lt;strong&gt;increase accuracy&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; In the emails the customer ask for &lt;strong&gt;&lt;em&gt;multiple actionable items&lt;/em&gt;&lt;/strong&gt;, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually  wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried &lt;strong&gt;&lt;em&gt;POS-tagging&lt;/em&gt;&lt;/strong&gt; for this but due to bad quality of data, it is not giving good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PPS:&lt;/strong&gt; The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking forward to get some insights on how to change my approach and specifically the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank You&lt;/p&gt;&#xA;" OwnerUserId="12557" LastEditorUserId="13686" LastEditDate="2016-06-20T21:10:51.063" LastActivityDate="2016-06-20T21:10:51.063" Title="Categorizing Customer Emails" Tags="|machine-learning|nlp|feature-selection|feature-extraction|multilabel-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12266" PostTypeId="2" ParentId="12265" CreationDate="2016-06-18T10:52:31.567" Score="2" Body="&lt;p&gt;First of all, I think that your accuracy is already very high for text classification. I want to provide some ideas for additional features and approaches though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Topic models&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topic models such as Latent Dirichlet Allocation (LDA) are quite frequently used when studying text corpora. You already wrote that you used LDA for coming up with cosine similarities. What you could also do though is to use the latent topics as additional features for individual documents, and use the probabilistic assignments as values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sentiment analysis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you work with loans and injurences, it might also be useful to use sentiment features in your classification. For example, you could use a dictionary based approach on valence, arousal and dominance and include these features into your classification; you can find information in the relevant &lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/23404613&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;. Another approach would be to use &lt;a href=&quot;http://sentistrength.wlv.ac.uk/&quot; rel=&quot;nofollow&quot;&gt;SentiStrength&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Word2Vec&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recent text analysis approaches have heavily focused on neural network approaches such as &lt;a href=&quot;http://deeplearning4j.org/word2vec&quot; rel=&quot;nofollow&quot;&gt;Word2Vec&lt;/a&gt; that finds word embeddings. Then, a simple approach could be to average the vectors over all words in a document and use these as features. There are also approaches that use the vectors as features in &lt;a href=&quot;http://arxiv.org/abs/1408.5882&quot; rel=&quot;nofollow&quot;&gt;convolutional neural networks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;E-Mail features&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did you think about using further e-mail features, such as the domain, or simply the text length. Might be worth a try to look into that. Also, the domain of e-mail spam classification might give you some further insights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature/classifier tuning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bag of word/tfidf feature generation is sometimes quite sensitive. You might want to try to tune this by altering the parameters in some form of grid search optimization; for instance, sometimes it is useful to use minimum number of document thresholds for features or things like sublinear scaling. Also, you can try to combine your classifiers in an ensemble approach which might lead to higher accuracy.&lt;/p&gt;&#xA;" OwnerUserId="20642" LastActivityDate="2016-06-18T10:52:31.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12267" PostTypeId="1" CreationDate="2016-06-18T12:08:43.933" Score="-3" ViewCount="1795" Body="&lt;p&gt;I have 11 years experience on .net, should i learn data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 11 years experience on .net and getting paid rs 14L in india noida,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should my experience get consider in Data Science and get better opportunity.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should i learn data science or just learn new .net technology like Angular, node.js etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me out.&lt;/p&gt;&#xA;" OwnerUserId="20646" LastActivityDate="2019-06-10T15:48:38.773" Title="I have 11 years experience on .net, should i learn data science?" Tags="|data|.net|" AnswerCount="1" CommentCount="2" ClosedDate="2016-06-18T20:21:30.837" ContentLicense="CC BY-SA 3.0" />
  <row Id="12268" PostTypeId="2" ParentId="12265" CreationDate="2016-06-18T12:31:23.700" Score="3" Body="&lt;p&gt;(1) &lt;strong&gt;Data quality&lt;/strong&gt;. The &lt;em&gt;single best way to improve your accuracy&lt;/em&gt;. &lt;a href=&quot;https://en.wikipedia.org/wiki/Garbage_in,_garbage_out&quot; rel=&quot;nofollow noreferrer&quot;&gt;Garbage in garbage out&lt;/a&gt;. You already said your data is suspect. Some data was mis-classified; data only has single label, when multiple labels are possible. This is the biggie - this will improve your accuracy more than any other technique: improve the quality of your training data. One way to go about this. Recruit expert labellers (these could be workers in your own company who do this task, or they could be external people, e.g., &lt;a href=&quot;https://www.mturk.com/mturk/welcome&quot; rel=&quot;nofollow noreferrer&quot;&gt;mturkers&lt;/a&gt; who have been trained to do this labelling task). In general, the more labellers, the better. At a minimum I would have 3 labellers, label all &lt;em&gt;n&lt;/em&gt; emails in your train/test set, using either majority label wins, or require all-three consensus depending on your confidence needs. You will want to measure your &lt;a href=&quot;https://en.wikipedia.org/wiki/Inter-rater_reliability&quot; rel=&quot;nofollow noreferrer&quot;&gt;inter-rater reliability&lt;/a&gt; to ensure the integrity of your data using &lt;a href=&quot;https://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cohen's kappa&lt;/a&gt; or similar. Importantly, this data will become your &lt;a href=&quot;https://stats.stackexchange.com/questions/9159/what-is-the-meaning-of-a-gold-standard&quot;&gt;gold standard data&lt;/a&gt;. That is to say the average agreement upon labels provided by your original labellers is the &lt;strong&gt;ground truth&lt;/strong&gt; to which your algorithm attempts to approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, this approach to data quality is costly in terms of resources (time to label all the emails; cost of hiring mturkers, etc.). But depending upon your &lt;em&gt;specific&lt;/em&gt; needs it needn't be... basically it depends upon the needed level of &quot;expertise&quot; to do the task. Sometimes, it is safe to consider you (the researcher/developer) as the &quot;expert&quot; (e.g., you yourself said that some emails were mis-classified... how do you know that? You must have some sort of expert knowledge.). However, using a single labeller for the task makes your data less reliable due to individual processing differences introducing a bias into the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) &lt;strong&gt;Including stop words&lt;/strong&gt;. Yes, you read that correct &lt;em&gt;including&lt;/em&gt; stop words. Grammatical function words (e.g., the, and, of, etc.) can often be highly indicative of specific text types or genres. It is premature to eliminate stop words without first ensuring that they are an uninformative feature. Similarly, try &lt;strong&gt;including punctuation&lt;/strong&gt;, try &lt;strong&gt;including capitalization&lt;/strong&gt; features. Time and time again, I see &lt;strong&gt;people blindly pre-processing text without empirically determining whether these features are significant predictors of class&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &lt;strong&gt;Capture spelling variation&lt;/strong&gt;. As the OP stated, the original texts have many &quot;spelling mistakes.&quot; You can improve accuracy by reducing this variance - basically map variants to their canonical form before extracting your word features (e.g., &quot;colour&quot; -&gt; &quot;color&quot;). I often rely on &lt;a href=&quot;https://opendata.stackexchange.com/questions/7508/dataset-of-common-typos-in-english/7511#7511&quot;&gt;external corpora&lt;/a&gt; to rapidly create these spelling maps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) &lt;strong&gt;Balance your training data&lt;/strong&gt; - The OP mentions that his classes are severely imbalanced. Try &lt;strong&gt;over-&lt;/strong&gt; and/or &lt;strong&gt;under-sampling&lt;/strong&gt; to balance your classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5) &lt;strong&gt;Describe your problem&lt;/strong&gt;. It is also important to understand what the &lt;em&gt;underlying purpose&lt;/em&gt; of your classifier. If your classifier is intended to run on historical data, perhaps to flag incorrectly human-classified emails, then you would probably want to keep auto-response and correspondence thread info. On the other hand, if the goal is to classify &lt;em&gt;new customer emails&lt;/em&gt; so that they get properly routed, for example, then you probably want to remove these correspondence features from your training and test data.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:57:29.190" LastActivityDate="2016-06-18T13:52:38.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12270" PostTypeId="2" ParentId="11045" CreationDate="2016-06-18T14:49:03.843" Score="4" Body="&lt;p&gt;Thanks for using SPMF. I'm the founder of that library. Currently, SPMF does not support text files containing some text as input. Thus, users need to do some preprocessing to convert the text file to the format used by SPMF.  But in the future, it is my plan to add the feature of handling text files natively in SPMF.  It will likely be added in the next few weeks, depending on my schedule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, if you have any question about SPMF, you can also directly ask them on the &lt;a href=&quot;http://forum.ai-directory.com/list.php?5&quot; rel=&quot;nofollow&quot;&gt;forum&lt;/a&gt; on the &lt;a href=&quot;http://www.philippe-fournier-viger.com/spmf/&quot; rel=&quot;nofollow&quot;&gt;SPMF&lt;/a&gt; website. Then, I will answer you more quickly.&lt;/p&gt;&#xA;" OwnerUserId="20647" LastActivityDate="2016-06-18T14:49:03.843" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12271" PostTypeId="2" ParentId="12267" CreationDate="2016-06-18T18:54:04.810" Score="1" Body="&lt;p&gt;Let me try and break it down slightly and expand on what @Dawny33 has mentioned.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should you learn DS?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If ambiguous and open ended problem statements excite you, you should. If you love and continue to apply mathematical concepts beyond your college degree, you should. If you always question why before what and how, you should. Given your question however, I am tempted to say No, but it is solely up to you. What would you like and more importantly why would you like it?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Will your experience count for DS opportunity?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No. Can you build solutions that can be data products and feed some reporting and analytics using .Net? Yes. Being a DS myself, I work with folks who do exactly that. Are they part of my Job Family? No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In spite of its similarities in some areas, Data Science (and analysis) in general requires a different skill set than a pure developer. It among other things can also require lot of data exploration, wrangling, plotting and research on a day to day basis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you are really ready to change your job profile at your career stage, and bear the ever changing and dynamic nature of DS, stay put and invest time in technologies that have synergies with your existing skill set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;+1 for Angular as I know how heavily my employer is investing in it.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="29169" LastEditDate="2019-06-10T15:48:38.773" LastActivityDate="2019-06-10T15:48:38.773" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12272" PostTypeId="1" AcceptedAnswerId="12276" CreationDate="2016-06-18T21:14:57.997" Score="1" ViewCount="1461" Body="&lt;p&gt;Having a bit of a mind-blank at the moment and am looking for some advice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am extracting features from time series data for input into a classification algorithm, for example I'm extracting average and variance from inputX. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For input Y,  I have graphed the data and have seen that for class A, it can be seen that there is an upwards slope, and for class B, it can be seen that there is a downward slope, for class C there is no slope, the line is more or less straight. &#xA;For Feature Extraction, how can I best describe this? Would a calculation to get positive/negative slope be best? &lt;/p&gt;&#xA;" OwnerUserId="17794" LastActivityDate="2016-06-19T07:32:22.153" Title="Feature Extraction - calculate slope" Tags="|time-series|feature-extraction|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12274" PostTypeId="1" AcceptedAnswerId="12285" CreationDate="2016-06-19T05:20:31.223" Score="17" ViewCount="52368" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The model of linear regression is linear in parameters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this actually mean?&lt;/p&gt;&#xA;" OwnerUserId="20655" LastActivityDate="2020-07-06T12:29:32.547" Title="What does &quot;linear in parameters&quot; mean?" Tags="|regression|linear-regression|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12276" PostTypeId="2" ParentId="12272" CreationDate="2016-06-19T07:32:22.153" Score="1" Body="&lt;p&gt;There are several ways to do this, here are a couple of options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Calculate different lag values (difference between now and &lt;strong&gt;t&lt;/strong&gt; time units)&lt;/li&gt;&#xA;&lt;li&gt;Calculate a linear regression for different time windows and store the slope and the bias&lt;/li&gt;&#xA;&lt;li&gt;You can also involve higher order models to describe what is happening, if you think for example that the acceleration also matters you could use a 2nd or 3rd degree polynomial over the past couple of observations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-19T07:32:22.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12277" PostTypeId="2" ParentId="12274" CreationDate="2016-06-19T07:32:58.800" Score="5" Body="&lt;p&gt;It simply means that $Y = A X$ where $A$ are the parameters. The variables $X$ might contain nonlinear relationships; e.g., $X = [\alpha\; \alpha \beta\; \beta^2]^T$, yet $Y$ is a linear function of $X$.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-06-19T16:31:21.197" LastActivityDate="2016-06-19T16:31:21.197" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12278" PostTypeId="2" ParentId="12250" CreationDate="2016-06-19T09:08:03.927" Score="2" Body="&lt;p&gt;If you have a classification problem, you should you LDA instead of PCA. PCA ignores classes, whereas LDA is class-aware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your data is 2D and you use PCA in the following example, you get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ylm1X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ylm1X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So before PCA, the classes were perfectly linearly separable, but after PCA they are not separable at all. I'm not saying this happens in your case, but it could be.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-06-19T09:08:03.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12279" PostTypeId="1" CreationDate="2016-06-19T11:30:48.420" Score="0" ViewCount="80" Body="&lt;p&gt;There are many instances of quotes and texts where the true author may be disputed. Are there data science or machine learning techniques which may be useful to establish the authorship?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-19T11:30:48.420" Title="Can data science be applied to establishing the author of a text?" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12280" PostTypeId="2" ParentId="12279" CreationDate="2016-06-19T11:30:48.420" Score="1" Body="&lt;p&gt;Yes - this is an active area of study, especially famously with respect to disputed plays of Shakespeare. Features commonly extracted include frequency of key words, novel vocabularly, word length, and these are well adapted to being used as data mining features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here is a thesis that data mined Shakespeare's extant plays to test whether they could have been the work of a number of authors often touted as being the &quot;true Shakespeare&quot; -&gt; &lt;a href=&quot;http://www.cs.dartmouth.edu/~datamining/Final.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.cs.dartmouth.edu/~datamining/Final.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-19T11:30:48.420" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12281" PostTypeId="1" CreationDate="2016-06-19T11:52:41.187" Score="4" ViewCount="4919" Body="&lt;p&gt;I have a twitter data-set and I wanna extract their related topics. So, I decided to classify my Tweets into clusters using an unsupervised machine learning algorithm like k-means. This choice is made due the time consuming of the training process in the supervised approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as a first step after cleaning my tweets, I will extract features (eg. Hashtags...) from them, and enrich them with side information from knowledge bases (eg. Wikipedia). Secondly, they will be represented in a Vector space. Next, using k-means and for given K=6 clusters, my tweets already enriched will be classified into 6 clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don’t know how to identify automatically the topics related to these clusters. Is there any solutions?&lt;/p&gt;&#xA;" OwnerUserId="20590" LastEditorUserId="31513" LastEditDate="2017-04-26T16:24:29.990" LastActivityDate="2020-06-02T10:19:04.437" Title="How cluster a twitter data-set?" Tags="|machine-learning|data-mining|clustering|nlp|social-network-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12283" PostTypeId="2" ParentId="12186" CreationDate="2016-06-19T14:18:40.543" Score="2" Body="&lt;p&gt;I solved similar, though simpler, task with a brute force approach.&#xA;The simplification was in the assumption, that the line is a linear function (in my case even the coefficients and intercept were in some known range).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will therefore not solve your problem in general, where a particle can move orthogonal with axis x (i.e. it trace no function), but I post the solution as a possible inspiration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Take all combinations of two points A and B with A(x) &gt; B(x) +  constant (to avoid the symmetry and a high error while calculating the coefficient)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Calculate the &lt;em&gt;coefficient&lt;/em&gt; c and &lt;em&gt;intercept&lt;/em&gt; i of the line AB&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; A(y) = i + c * A(x)&#xA; B(y) = i + c * B(x)&#xA; A(y) - B(y) = c * (A(x) - B(x))&#xA; c = (A(y) - B(y)) / (A(x) - B(x))&#xA; i = A(y) - c * A(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;3) Round the coefficient and intercept (this should eliminate / lower the problems with errors caused by the points in a grid)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) For each intercept and coefficient calculate the number of points on this line&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Consider only lines with points above some threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple example see &lt;a href=&quot;http://www.db-nemec.com/knitr/linear_clustering_adv.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-06-19T14:18:40.543" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12284" PostTypeId="2" ParentId="12265" CreationDate="2016-06-19T14:25:36.233" Score="0" Body="&lt;p&gt;Based on what you describe, maybe your problem setting is a multi label setting. E.g: the customer's email might be about an enquiry about two aspects. Say a question about returns policy and price.&#xA;If that's indeed the case you should try to train multiple binary classifiers like &quot;email about price&quot;and &quot;email not about price&quot; instead of explicitly trading off the categories against each other. If you have $k$ different categories you'll end up with $k$ classifiers. Then while predicting on test data you can take the union of the labels output by the $k$ classifiers as the predicted label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see  &quot;PT4&quot; in &lt;a href=&quot;http://lpis.csd.auth.gr/publications/tsoumakas-ijdwm.pdf&quot; rel=&quot;nofollow&quot;&gt;Multi-Label Classification&lt;/a&gt; for more details.&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-06-19T17:39:07.393" LastActivityDate="2016-06-19T17:39:07.393" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12285" PostTypeId="2" ParentId="12274" CreationDate="2016-06-19T15:31:32.560" Score="19" Body="&lt;p&gt;Consider an equation of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x$'s are the variables and $\beta$'s are the parameters. Here, y is a linear function of $\beta$'s (linear in parameters) and also a linear function of $x$'s (linear in variables). If you change the equation to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, it is no longer linear in variables (because of the squared term) but it is still linear in parameters. And for (multiple) linear regression, that's all that matters because in the end, you are trying to find a set of $\beta$'s that minimizes a loss function. For that, you need to solve a system of &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#Derivation_of_the_normal_equations&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;linear&lt;/em&gt; equations&lt;/a&gt;. Given its nice properties, it has a closed form solution that makes our lives easier. Things get harder when you deal with nonlinear equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you are not dealing with a regression model but instead you have a mathematical programming problem: You are trying to minimize an objective function of the form $c^Tx$ subject to a set of constraints: $Ax \geq b$ and $x\geq0$. This is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_programming&quot; rel=&quot;noreferrer&quot;&gt;linear programming&lt;/a&gt; problem in the sense that it is linear in variables. Unlike the regression model, you are trying to find a set of $x$'s (variables) that satisfies the constraints and minimizes the objective function. This will also require you to solve systems of linear equations but here it will be linear in variables. Your parameters won't have any effect on that system of linear equations.&lt;/p&gt;&#xA;" OwnerUserId="8664" LastActivityDate="2016-06-19T15:31:32.560" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12287" PostTypeId="1" CreationDate="2016-06-19T18:15:01.367" Score="0" ViewCount="199" Body="&lt;p&gt;I'm new here - I hope this kind of question is legit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a background in mathematics - my field is probability theory, where I am quite strong (top 2% of University). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have little experience in programming but I guess I get things done once I know what is out there (i.e. what program languages can do for me). &#xA;I want to learn more about data science and dive into the field particular when it comes to programming. I'd also like to get going on manageable yet interesting data science related tasks (building some &quot;usefull&quot; statistics skill might be desirable, too).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know kaggle.com are there other places I could look into?&#xA;Any projects I might take on?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot. :)&lt;/p&gt;&#xA;" OwnerUserId="18985" LastActivityDate="2016-06-19T19:29:04.570" Title="How to dive into data science quickly?" Tags="|beginner|" AnswerCount="2" CommentCount="0" ClosedDate="2016-06-20T10:42:09.970" ContentLicense="CC BY-SA 3.0" />
  <row Id="12288" PostTypeId="2" ParentId="12287" CreationDate="2016-06-19T18:29:42.570" Score="1" Body="&lt;p&gt;If you're new to both machine learning and programming, try taking a look at this guy's tutorials: &lt;a href=&quot;http://www.pyimagesearch.com/&quot; rel=&quot;nofollow&quot;&gt;http://www.pyimagesearch.com/&lt;/a&gt;. I've been using them for a while, and I think they're great because of how visual, hands-on, and practical they are. For example, he has facial recognition and handwriting recognition lessons. I'm using what I learned in his tutorials to build a self driving remote control car (similar to this: &lt;a href=&quot;https://zhengludwig.wordpress.com/projects/self-driving-rc-car/&quot; rel=&quot;nofollow&quot;&gt;https://zhengludwig.wordpress.com/projects/self-driving-rc-car/&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-19T18:29:42.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12289" PostTypeId="1" CreationDate="2016-06-19T18:40:13.467" Score="2" ViewCount="47" Body="&lt;p&gt;Creating a boolean for each value of each discrete variable explodes the number of features. I have an idea for solving it but I wasn't able to find anything on this on the internet, so perhaps someone could either point me to a resource or let me know whether they think this is possible. And if not, maybe point to another solution (all I could find was binarisation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we have training vectors $X_i$ and similarly labels $y_i$, I propose we partition the training set by some variable $j$, so that for each $X_i$ for which $j_i$ equals some value $q$, $X_i$ will be in $P(q)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for each $q$ we calculate the mean $y_i$ for $P(q)$ and sort those averages, mapping each $q$ onto its index in the sorted list. This value is used as the feature for $j$. I suspect that this ought to be better than just giving each value a random index, as any remaining non-linearity will be hopefully fixed by the hidden layers. But it seems that these non-linear patterns will be easier to pick up if the function is somewhat quasi-monotonous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this could be improved, potentially instead of just finding the mean, another measure could work better and also for multiple discrete features, I am not sure if this process should be repeated for each variable separately or through a multi-dimensional partitioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main question is of course, would this process be helpful at all, and if not, is there anything similar that could solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="20475" LastEditorUserId="836" LastEditDate="2016-06-19T19:12:46.607" LastActivityDate="2016-06-23T09:42:51.517" Title="Instead of binarising discrete variables for NN algorithms, could I attempt to assign them a value by splitting into buckets and sorting?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12290" PostTypeId="2" ParentId="12287" CreationDate="2016-06-19T19:29:04.570" Score="1" Body="&lt;p&gt;I would check out the following 2 sites as really good starting points:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.becomingadatascientist.com&quot; rel=&quot;nofollow&quot;&gt;http://www.becomingadatascientist.com&lt;/a&gt; - Interviews, resources, challenges &amp;amp; forums&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://datasciencemasters.org&quot; rel=&quot;nofollow&quot;&gt;http://datasciencemasters.org&lt;/a&gt; - web resources for self-edification&lt;/p&gt;&#xA;" OwnerUserId="20667" LastActivityDate="2016-06-19T19:29:04.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12291" PostTypeId="2" ParentId="12281" CreationDate="2016-06-19T21:36:15.143" Score="4" Body="&lt;h2&gt;k-means is &lt;em&gt;very&lt;/em&gt; sensitive to noise&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;because it is designed as a least-squares approach. Noise deviations, when squared, become even larger.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Twitter is mostly noise&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Twitter is full of spam and nonsense tweets. These will be entirely unlike any other and thus have the largest deviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chances are you get one &quot;cluster&quot; that contains almost everything, and the other k-1 clusters consist of a few tweets with their duplicates. &lt;strong&gt;Clusters are not topics&lt;/strong&gt;. They are more likely to be &lt;em&gt;duplicates&lt;/em&gt; than topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An appropriate clustering algorithm for tweets should probably discard 90% of tweets and produce thousands of clusters. But it will rarely be better than finding all tweets in common - most tweets only have 2-3 usable words.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="77627" LastEditDate="2020-06-02T10:19:04.437" LastActivityDate="2020-06-02T10:19:04.437" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="12292" PostTypeId="2" ParentId="12289" CreationDate="2016-06-19T22:11:23.537" Score="1" Body="&lt;p&gt;Just taking the mean seems better, that way the distances between the groups is better represented than just indexing them by order. With the target you should be a bit careful not to include the current row for the mean, since for the test set and the actual predictions you cannot use that target either (for obvious reasons). To get even more information about the category you try to get rid of is to also take the variance or standard deviation of the target within this group and take the mean and variance of important numerical features for these categories. If you have multiple categories you can also take the expected value for features and targets conditioned on multiple categories at once. This is a nice compromise between keeping enough information on high cardinality while keeping the dimensionality low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example where the c is the categorical variable, n are numeric and y is your target. We want to get rid of c (because c could have 1000s of values).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xc_1 | xn_1 | xn_2 | y&#xA;A    | 2    | 4    | 1&#xA;A    | 1    | 2    | 1&#xA;A    | 4    | 3    | 0&#xA;B    | 3    | 5    | 0&#xA;B    | 5    | 4    | 1&#xA;B    | 6    | 6    | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To keep things a little more simple we will disregard the values of the current row, which is better anyway. For the first row we will look at all other A rows, take the averages (and maybe other statistics) of the numeric features and of y, we will do the same for B, then we get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xc_1 | xn_1 | xn_2 | xnc_1 | xnc_2 | yc_1 | y&#xA;A    | 2    | 4    | 2.5   | 2.5   | 0.5  | 1&#xA;A    | 1    | 2    | 3     | 3.5   | 0.5  | 1&#xA;A    | 4    | 3    | 1.5   | 3     | 1    | 0 &#xA;B    | 3    | 5    | 5.5   | 5     | 0.5  | 0 &#xA;B    | 5    | 4    | 4.5   | 5.5   | 0    | 1  &#xA;B    | 6    | 6    | 4     | 4.5   | 0.5  | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We have captured a lot of information about category 1, which we can now discard.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-06-23T09:42:51.517" LastActivityDate="2016-06-23T09:42:51.517" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12293" PostTypeId="2" ParentId="11495" CreationDate="2016-06-19T23:47:59.063" Score="0" Body="&lt;p&gt;Data cleaning can be a real pain and also can easily take you away from the core task. Nevertheless, it is one of the critical aspect and hence cannot be taken lightly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe you have the right idea because you mentioned that you are seeking &lt;code&gt;balance&lt;/code&gt;. I always think of data cleaning as a marginal transaction. Beyond a certain point, it is not worth the time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/14904/jan-van-der-vegt&quot;&gt;Jan van&lt;/a&gt; mentioned, it all depends on your needs. If your model could be very sensitive to certain features, its better to have them cleaned religiously. If what you are looking for are broader insights, even a general clean of important features can work. The trick is to really understand the tipping point which can come with experience and / or knowledge of the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is this approach that my team takes often and has worked so far.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Once you know what model fits your problem, throw in a feature with random values. &#xA;2. Model your data set with minimal / basic cleaning. &#xA;3. Plot or tabulate accuracy for all features.&#xA;4. Drop all features that perform equal or worse than the random feature.&#xA;5. Some times depending on the accuracy even drop features that perform slightly higher than random.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this, we are mostly able to get rid of unnecessary features that hog up tidying effort. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wash Rinse Repeat.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-06-19T23:47:59.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12295" PostTypeId="1" CreationDate="2016-06-20T02:21:14.963" Score="4" ViewCount="133" Body="&lt;p&gt;It is a real world use case. For example, a route from place A to place B can be different series of lat, lng points - the different trips though they are exactly the same sequence from Street x and then Road y then High way z. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The differences are the locations are reported in different time (for example each trip reports the location 1 minute) and the Vehicles appear in the different lane of the same street. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, do you have some ideas about how to compute the similarity of two different trips belong to the same route (kinds of mapping different trips to the same route ).   &lt;/p&gt;&#xA;" OwnerUserId="16756" LastActivityDate="2016-09-08T12:26:16.677" Title="If I have got different routes - a series of (lat, lng) points, how to get the similarity of different routes?" Tags="|similarity|usecase|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12296" PostTypeId="1" CreationDate="2016-06-20T05:25:45.467" Score="5" ViewCount="2005" Body="&lt;p&gt;Consider a data-frame similar to  the one shown (the actual data-frame is &lt;em&gt;much&lt;/em&gt; larger)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID EDUCATION   OCCUPATION      BINARY_VAR&#xA;1  Undergrad   Student              1&#xA;2  Grad        Business Owner       1&#xA;3  Undergrad   Unemployed           0&#xA;4  PhD         Other                1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The final objective is to apply &lt;a href=&quot;http://poseidon01.ssrn.com/delivery.php?ID=872119091124115016031102012127095087049073070064025021111113071102071087101116120000120032062006006027002113105107107064023000000029026049019127065083026090094023124018052042067103003071104125064074030007117079089123125110073115125085127122067113082104&amp;amp;EXT=pdf&quot; rel=&quot;nofollow&quot;&gt;PRIDIT&lt;/a&gt; scoring on individual profiles (&lt;code&gt;ID&lt;/code&gt;) based on discrete &quot;rank&quot; scores of the individual values in the cell. These ranks can be thought of as indicator variables which will be used to collectively rate any $ID_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, the ranks could signify the possibility of some $ID_i$ committing fraud:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 : Low&#xA;2 : Medium&#xA;3 : High&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The variable, &lt;code&gt;BINARY_VAR&lt;/code&gt; is something like a &quot;training variable&quot; or rather, a &quot;predictor variable&quot;, such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var = 0:$ Fraud&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Var = 1:$ Non-fraud&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By this reasoning, an unemployed Undergraduate would be a Rank 3 profile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to apply PRIDIT, I must first convert the non-numeric variables into scores or levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way it is currently being done is by applying correspondence analysis on each column against &lt;code&gt;BINARY_VAR&lt;/code&gt; and then calculating the distance of the column contribution scores from row contribution score for non-fraud. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Row and column scores look something like this (respectively):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            CONTR&#xA;0           1.654&#xA;1           98.346&#xA;------------------------------&#xA;                  CONTR&#xA;Undergraduate     2.803602e-04&#xA;Graduate          3.147824e+00&#xA;PhD               9.176451e+00&#xA;Other             1.179664e+01&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The obtained distance (supposedly) gives the required score for the level, which is written back to the data-frame as a rank (higher value resulting in a higher rank).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main concerns about this technique are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The data-frame is really large, and resources are limited - it is a computationally expensive method.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It involves a lot of steps, and the result of the scoring can not really be verified &lt;strong&gt;(can it?)&lt;/strong&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My questions are:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does this technique seem viable?&lt;/li&gt;&#xA;&lt;li&gt;What are betters way to assign &quot;ranks&quot; to non-numeric variables?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2016-06-21T14:06:42.830" LastActivityDate="2017-02-24T04:03:09.773" Title="Converting non-numeric data values into equivalent rank scores" Tags="|correlation|ranking|pca|scoring|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12297" PostTypeId="1" CreationDate="2016-06-20T08:25:20.267" Score="1" ViewCount="281" Body="&lt;p&gt;A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have sufficient data to train a RNN. I have discounted Markov decision process based mechanisms for the same reason. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any online learning algos that I can use to get coarse/approximate predicted sequences with only a small training set? &#xA;I have looked at Q-learning but it seems to be ideal for best path problems where the end goal is definite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: Adding more clarity on the type of data post the comments.&#xA;My data is video content consumption data. About 100 users consuming from a library of 1000 video titles. Intent is to exploit (if it exists) the likelihood of consuming content in a probable sequence. I currently have 300 - 400 such sequences spanning 3 videos each. &lt;/p&gt;&#xA;" OwnerUserId="12809" LastEditorUserId="12809" LastEditDate="2016-06-21T07:40:05.423" LastActivityDate="2016-06-21T07:40:05.423" Title="Online methods for sequence prediction" Tags="|machine-learning|markov-process|online-learning|rnn|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12298" PostTypeId="1" CreationDate="2016-06-20T08:26:35.540" Score="0" ViewCount="340" Body="&lt;p&gt;I have a number of features X and a target which is the revenue generated by this interaction. More often than not no revenue is generated, but if there is it can differ considerably, it has a big right tail. I've made a model to estimate the expected revenue generated by a new x. Due to the weird distribution of the revenues I have split the model up in two parts, first estimating the probability of revenue generation and then conditionally on the fact that somebody will book, predict the amount of $log($revenue$)$. The first model is trained on a dummy variable which indicates revenue and the second one is trained on a subset of the data which contains all interactions that generated revenue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I have these two models P (probability of generation) and R (expected revenue generated conditioned on &gt; 0) we can calculate the expected revenue for a new x using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = P(x)e^{R(x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done a reasonable amount of cross validation on both the models individually and they work decent, however when they are combined they severly underestimate the amount of revenue generated. I have a good enough number of rows for the dimensionality of my data and I'm using Logistic Regression for the probability (linearity assumption is somewhat reasonable, however unbiased probabilities are very important) and Gradient Boosting for revenue estimation (got the best results for the second portion).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do about the underestimation of the revenue? What are alternative unbiased probablity estimators, should I train a neural network with a hidden layer for the log-loss function? Why does the combined model underestimate? Is there a good alternative to do it in one model? Which would not increase small errors into huge errors due to the multiplication and exponent?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-20T11:56:15.590" Title="Estimating expected revenue generation" Tags="|machine-learning|logistic-regression|cross-validation|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12299" PostTypeId="1" AcceptedAnswerId="12300" CreationDate="2016-06-20T08:57:04.020" Score="3" ViewCount="3233" Body="&lt;p&gt;I wonder how can I give weight to my feature before employing PCA. I mean somehow weighted PCA. Because I know that one of the features is better than others and want to give importance to it in creating components (It is not possible to select only that feature. I should have others impact too)&lt;/p&gt;&#xA;" OwnerUserId="12345" LastActivityDate="2016-06-20T09:48:51.110" Title="How can give weight to feature before PCA" Tags="|machine-learning|pca|weighted-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12300" PostTypeId="2" ParentId="12299" CreationDate="2016-06-20T09:12:55.890" Score="4" Body="&lt;p&gt;After standardizing your data you can multiply the features with weights to assign weights before the principal component analysis. Giving higher weights means the variance within the feature goes up, which makes it more important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Standardizing (mean 0 and variance 1) is important for PCA because it is looking for a new orthogonal basis where the origin stays the same, so having your data centered around the origin is good. The first principal component is the direction with the most variance, by scaling a certain feature with a weight of more than 1 will increase the variance on this axis and thus give more weight to this axis, pulling the first principal component in it's direction.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-06-20T09:48:02.537" LastActivityDate="2016-06-20T09:48:02.537" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12301" PostTypeId="2" ParentId="12299" CreationDate="2016-06-20T09:48:51.110" Score="7" Body="&lt;p&gt;PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, components are created in directions with highest variance and the importance is measured by eigenvalues. So I can imagine you can somehow increase the variance (like Jan van der Vegt proposed). But that's a sorcery with very questionable output.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-06-20T09:48:51.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12303" PostTypeId="2" ParentId="12298" CreationDate="2016-06-20T11:56:15.590" Score="1" Body="&lt;p&gt;Why not just use quantile regressions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, in regressions, the coefficients are estimated for the average case: $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want, you can estimate your coefficients so that they tell you things like $P(R=y)&amp;gt;p$. You give a probability $p$, and the model gives you $y$. You can even build probability density function out of these estimation techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In quantile regressions, you solve this optimization problem for every quantile $\tau$ that you want, $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i&amp;lt; X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a more robust approach to modeling the revenue distribution. Usually people use quantile regressions because software is readily available, but they can be implemented for other models as well. I work in an institute that does exactly that.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-06-20T11:56:15.590" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12305" PostTypeId="1" AcceptedAnswerId="13617" CreationDate="2016-06-20T13:51:41.460" Score="6" ViewCount="1308" Body="&lt;p&gt;I have the following model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_{it}=\alpha + x'_{it}\beta_{i} + \epsilon_{it}, \text{ }&#xA;i=1,2,...,N, \text{ } t=1,2,...,T$ (1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\beta_{i}= z'_{i}\gamma+\eta_{i}$ (2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $\epsilon_{it} \sim N(0,\sigma_{\epsilon_{i}}^{2})$ and $\eta_{i} \sim N(0,\sigma_{\eta}^{2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also i consider the following prior specifications:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(\beta,\gamma) \propto 1 \\&#xA;p(\sigma_{\epsilon_{i}}^{2}) \propto \sigma_{\epsilon_{i}}^{-2} \\&#xA;p(\sigma_{eta}^{2}) \propto \sigma_{\eta}^{-2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which has the following likelihood function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(Y|\theta)=\prod_{i=1}^{N}\int_{\beta_{i}} \left( \prod_{t=1}^{T}\frac{1}{\sigma_{\epsilon_{i}}\sqrt{2\pi}}exp \left( -\frac{1}{2\sigma_{\epsilon_{i}}^{2}}(y_{it}-\alpha-x'_{it}\beta_{i})^{2}\right) \right) \times \frac{1}{\sigma_{\eta}\sqrt{2\pi}}exp \left( -\frac{1}{2{\sigma_{\eta}^{2}}}(\beta_{i}-z'_{i}\gamma)^{2} \right) d\beta_{i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to estimate the parameters $\theta=(\alpha,\gamma,\sigma_{\epsilon_{i}}^{2},\sigma_{\eta}^{2})$. So $\theta$ is the parameter vector. Only $\beta_{i}$ is random. So I consider $\beta_{i}$ as the latent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to estimate these by using the following Gibbs sampling schema:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample $\alpha$ given $\{\beta_{i}\}_{i=1}^{N},\gamma,\sigma_{\epsilon}^{2},\sigma_{\eta}^{2},Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample $\gamma$ given $\{\beta_{i}\}_{i=1}^{N},\alpha,\sigma_{\epsilon}^{2},\sigma_{\eta}^{2},Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample $\sigma_{\epsilon}^{2}$ given $\{\beta_{i}\}_{i=1}^{N},\gamma,\alpha,\sigma_{\eta}^{2},Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample $\sigma_{\eta}^{2}$ given $\{\beta_{i}\}_{i=1}^{N},\gamma,\alpha,\sigma_{\epsilon}^{2},Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample $\{\beta_{i}\}_{i=1}^{N}$ given $\gamma,\alpha,\sigma_{\epsilon}^{2},\sigma_{\eta}^{2},Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sample $\alpha$ I re-write (1) as $y_{it} - x'_{it}\beta_{i}=\alpha + \epsilon_{it}$, so the full conditional distribution of $\alpha$ is normal with mean $\hat{\alpha}=\frac{1}{T\times N} \sum_{i=1}^{N}\sum_{t=1}^{T} (y_{it} - x'_{it}\beta_{i})$ and variance $\frac{\sigma_{\epsilon}^{2}}{T \times N}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sample $\sigma_{\epsilon}^{2}$ I consider the regression model in (1) , hence it can be sampled from an inverted Gamma-2 distribution with parameter $ \sum_{i=1}^{N}\sum_{t=1}^{T} (y_{it} - \alpha - x'_{it}\beta_{i})^{2}$ with $N \times T$ degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sample $\gamma$ I consider the regression model in (2), so $\gamma$ can be sampled from multivariate normal distribution &#xA;with mean equal to the standard OLS estimators $\hat{\gamma}=\left(\sum_{i=1}^{N}z_{i}z'_{i}\right)^{-1}(z'_{i}\beta_{i})$ and covariance matrix equal to the standard OLS covariance matrix estimator $\sigma_{\eta}^{2}\left(\sum_{i=1}^{N}z_{i}z'_{i}\right)^{-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sample $\sigma_{\eta}^{2}$ again we consider the linear regression in &#xA;(2), hence $\sigma_{\eta}^{2}$ can be sampled from an inverted Gamma-2 distribution with parameter $\sum_{i=1}^{N}(\beta_{i}-z'_{i}\gamma)^{2}$ and $N$ degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, we can sample $\{\beta_{i}\}_{i=1}^{N}$ from a normal distribution with mean \ $\left(\sum_{t=1}^{T}x_{it}^{2}+\left(\frac{\sigma_{\epsilon}}{\sigma_{\eta}}\right)^{2}\right)^{-1}\left(\sum_{t=1}^{T}x_{it}y_{it}+\left(\frac{\sigma_{\epsilon}}{\sigma_{\eta}}\right)^{2}z'_{i}\gamma\right)$ and covariance matrix $\sigma_{\epsilon}^{2}\left(\sum_{t=1}^{T}x_{it}^{2}+\left(\frac{\sigma_{\epsilon}}{\sigma_{\eta}}\right)^{2}\right)^{-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this in R ? If you could please give me an example for instance how to sample $\alpha$ I would be  very grateful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks ! :)&lt;/p&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2016-06-20T14:35:22.027" LastActivityDate="2016-08-23T11:57:22.203" Title="Gibbs sampling in R" Tags="|r|regression|linear-regression|parameter-estimation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12306" PostTypeId="1" CreationDate="2016-06-20T14:02:37.407" Score="2" ViewCount="833" Body="&lt;p&gt;in Spark, there is a &lt;code&gt;RowMatrix.columnSimilarities()&lt;/code&gt; method (see &lt;a href=&quot;http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/linalg/distributed/RowMatrix.html#columnSimilarities()&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/linalg/distributed/RowMatrix.html#columnSimilarities()&lt;/a&gt;) that returns &quot;&lt;em&gt;An n x n sparse upper-triangular matrix of cosine similarities between columns of this matrix&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I read it? If I try to implement an example from &lt;a href=&quot;https://stackoverflow.com/a/1750187&quot;&gt;https://stackoverflow.com/a/1750187&lt;/a&gt; as following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;JavaRDD&amp;lt;Vector&amp;gt; rows = sc.parallelize(Arrays.asList(&#xA;    new DenseVector(new double[]{2, 1, 0, 2, 0, 1, 1, 1}),&#xA;    new DenseVector(new double[]{2, 1, 1, 1, 1, 0, 1, 1})&#xA;));&#xA;&#xA;RowMatrix mat = new RowMatrix(rows.rdd());  &#xA;List&amp;lt;Vector&amp;gt; sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();&#xA;for(Vector v: sims) {&#xA;    System.out.println(v);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(8,[6,7],[0.7071067811865475,0.7071067811865475])&#xA;(8,[1,2,3,4,5,6,7],[0.9999999999999998,0.7071067811865475,0.9486832980505137,0.7071067811865475,0.7071067811865475,0.9999999999999998,0.9999999999999998])&#xA;(8,[2,3,4,5,6,7],[0.7071067811865475,0.9486832980505137,0.7071067811865475,0.7071067811865475,0.9999999999999998,0.9999999999999998])&#xA;(8,[7],[0.9999999999999998])&#xA;(8,[4,5,6,7],[0.4472135954999579,0.8944271909999159,0.9486832980505137,0.9486832980505137])&#xA;(8,[6,7],[0.7071067811865475,0.7071067811865475])&#xA;(8,[3,4,6,7],[0.4472135954999579,1.0,0.7071067811865475,0.7071067811865475])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should I interpret it? How do I get the cosine angle 0.822 from this, as mentioned in the referenced StackOverflow post?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="20687" LastEditorUserId="62668" LastEditDate="2018-11-15T18:40:02.347" LastActivityDate="2018-11-15T18:40:02.347" Title="How to interpret upper-triangular matrix of cosine similarities" Tags="|apache-spark|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12307" PostTypeId="2" ParentId="12306" CreationDate="2016-06-20T15:32:06.813" Score="2" Body="&lt;p&gt;Solution is to transform the matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;JavaRDD&amp;lt;Vector&amp;gt; rows = jsc.parallelize(Arrays.asList(&#xA;    new DenseVector(new double[]{2, 2}),&#xA;    new DenseVector(new double[]{0, 1}),&#xA;    new DenseVector(new double[]{1, 1}),&#xA;    new DenseVector(new double[]{1, 0}),&#xA;    new DenseVector(new double[]{0, 1}),&#xA;    new DenseVector(new double[]{2, 1}),&#xA;    new DenseVector(new double[]{1, 1}),&#xA;    new DenseVector(new double[]{1, 1})&#xA;));&#xA;​&#xA;RowMatrix mat = new RowMatrix(rows.rdd());&#xA;List&amp;lt;Vector&amp;gt; sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();&#xA;for(Vector v: sims) {&#xA;  System.out.println(v); //(2,[1],[0.8215838362577492])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20687" LastActivityDate="2016-06-20T15:32:06.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12310" PostTypeId="1" AcceptedAnswerId="12311" CreationDate="2016-06-20T19:54:10.870" Score="1" ViewCount="309" Body="&lt;p&gt;I am working on a binary class classification problem. &#xA;Each sample is a vector 1x101, I have a lot of data samples more than 150k &#xA;I tried training a linear svm and a non-linear svm (RBF) &quot;zscore normalization is used in both cases&quot;. surprisingly, the linear does better than the svm (RBF).&#xA;I am trying to explain this by considering the following points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I beleive that the quality of my feature is not very good. &lt;/li&gt;&#xA;&lt;li&gt;I think the nonlinear case experinces a kind of overfitting.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;my question is how to explain this behaviour?!! does what I am thinking in make sense?!! I am thinking in using Adaboost to perform  the training, is it a good idea or not?&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-09-19T00:20:53.790" Title="how to explain the behaviour: linear svm does better than non-linear RBF" Tags="|machine-learning|data-mining|svm|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12311" PostTypeId="2" ParentId="12310" CreationDate="2016-06-20T20:54:36.597" Score="3" Body="&lt;p&gt;The quality of your features might actually be better than you think. If they provide linear separability, nonlinear kernels will overfit more readily than a linear kernel, leading to your result.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-06-20T20:54:36.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12312" PostTypeId="2" ParentId="12305" CreationDate="2016-06-20T21:34:13.277" Score="1" Body="&lt;p&gt;In terms of writing this in R, here is an example I found:&#xA;&lt;a href=&quot;http://www.stat.purdue.edu/~zhanghao/MAS/handout/gibbsBayesian.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.purdue.edu/~zhanghao/MAS/handout/gibbsBayesian.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could then iterate through different levels of α and select the level yielding the best results.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-06-20T21:34:13.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12313" PostTypeId="2" ParentId="12248" CreationDate="2016-06-20T21:34:16.910" Score="3" Body="&lt;p&gt;While the actual &quot;best system&quot; depends heavily on a number of factors including your goals and your resources, it is possible to discuss some general pros and cons to each system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. IR Based approaches:&lt;/strong&gt; Information retrieval approaches allow the algorithm to make effective use of what we might call &quot;explicit knowledge&quot; or hardcoded facts. They can work very well when the answer to the query exists in some form already and where the algorithm simply needs to find this answer amongst a set of all the other possible answers. Such systems typically rely on either an explicit dataset of possible answers or facts, or by treating the web as such a dataset and querying it in some way. IR systems are often the only reasonable choice when the answer cannot be generated from abstract principles. For example, if we built a system to answer trivia questions about movie stars, we would need to have explicit knowledge of which features are associated with which movie stars (i.e. Brad Pitt has brown hair and starred in &lt;em&gt;Fight Club&lt;/em&gt;.) No machine learning algorithm can be expected to generate this without the knowledge already present (although machine learning systems with access to such knowledge may serve as effective information retrieval systems.) Due to the external nature of the data with respect to the algorithm, IR systems tend to be very effective for large and rapidly changing data sets, which would otherwise require retraining of the machine learning model or manual reconfiguration of the ontology to handle. It is also worth noting that more complex information retrieval systems may include other systems such as NLP engines to translate the data (ex. natural language text) into a form usable by the IR system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Ontology based approaches:&lt;/strong&gt; Ontology methods take the most time to implement and are usually the most finicky. They require the iterative hypothesizing of a structured representation of the domain, testing of the representation, and modification of the representation based on discovered flaws. For complex real-world domains with many possible contingencies, this process can take a very long time and be quite frustrating. Because everything is hard-coded, even the best ontology-based systems display a relative degree of inflexibility compared to their machine learning and IR counterparts. Ontologies can be effective, however, when you do not have access to a data set for information retrieval or machine learning training, as the domain knowledge are hard-coded into ontology systems by hand. They can also be useful when the domain is relatively well-structured and where it is important to understand and plan every decision made by the system such as in automated attendant phone systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Machine learning based approaches:&lt;/strong&gt; As machine learning approaches extract patterns from data, they are relatively robust to subtleties present in complex domains that were not explicitly built into an algorithm, model, or ontology. They work best when a prediction needs to be made about which &quot;class&quot; or category a given input (or query in your case) belongs to, or when a quantitative prediction needs to be produced from data. For example, if I have a large collection of animal pictured labeled with the type of animal present in these images, I can train a machine learning model to predict which animal type is present in future images, as long as the model has seen enough pictures of this animal that it can extract the patterns in the image that correspond to the image likely representing a lion or a tiger or a bear, etc. This works because there are properties in the image itself that predict which animal it represents and these patterns can be extracted from labeled data (there are also machine learning algorithms that pick out patterns in unlabeled data called unsupervised algorithms, but these are not relevant for your domain of interest.)  Traditionally, machine learning systems are relatively poor at modeling domains where the answer depends not on recognizing a pattern, but having access to knowledge like in the movie stars example above. Machine learning systems are also great because they require relatively little manpower to implement. As you mentioned, they also require access to a dataset, which may need to be quite large depending on the complexity of the domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can &lt;em&gt;roughly&lt;/em&gt; summarize the advantages and disadvantages by ranking the methods according to a few criteria:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9seU8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9seU8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the best systems (ex: IBM Watson) use a hybrid approach, taking advantage of each method for its relative strengths and substituting other methods to address their weaknesses. Depending on the performance you want, a QA system can be built by a single person with some knowledge of any of the above, or may require a team of upwards of 100 engineers. &lt;/p&gt;&#xA;" OwnerUserId="20702" LastActivityDate="2016-06-20T21:34:16.910" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12314" PostTypeId="1" CreationDate="2016-06-20T22:43:25.173" Score="1" ViewCount="81" Body="&lt;p&gt;I have recently observed in many cases that taking mean of the cost function vs sum of cost function results in very different results (specifically in thenano)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many times sum of cost gives nan while taking mean resolves it. Why is this the case? Mean requires sum too, then what is the possible reason for this behaviour?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-06-20T22:43:25.173" Title="Mean vs Sum of cost in theano" Tags="|machine-learning|python|theano|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12315" PostTypeId="2" ParentId="10005" CreationDate="2016-06-20T23:25:07.707" Score="0" Body="&lt;p&gt;SVR work only in margin of training data but RBFNN must be optimized&#xA; weight&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RBF NN may be fall in local minima opposed to  SVR&lt;/p&gt;&#xA;" OwnerUserId="20703" LastActivityDate="2016-06-20T23:25:07.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12316" PostTypeId="2" ParentId="12253" CreationDate="2016-06-21T00:01:53.150" Score="5" Body="&lt;p&gt;I think the term &quot;sensitivity&quot; comes from the world of medical tests. A very sensitive test will test positive for most or all people who take the test and really have a disease, as well as for many people who don't. This corresponds to high recall, which means the query retrieves most or all of the relevant documents, as well as many that may not be relevant.&lt;/p&gt;&#xA;" OwnerUserId="20704" LastActivityDate="2016-06-21T00:01:53.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12318" PostTypeId="1" AcceptedAnswerId="12594" CreationDate="2016-06-21T06:02:19.990" Score="57" ViewCount="67704" Body="&lt;p&gt;I ran a xgboost model. I don't exactly know how to interpret the output of &lt;code&gt;xgb.importance&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the meaning of Gain, Cover, and Frequency and how do we interpret them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters &lt;a href=&quot;https://raw.githubusercontent.com/pommedeterresautee/xgboost/master/R-package/vignettes/discoverYourData.Rmd&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other parameters that can tell me more about feature importances?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;importance_matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           Feature         Gain        Cover    Frequence&#xA;  1:            xxx 2.276101e-01 0.0618490331 1.913283e-02&#xA;  2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01&#xA;  3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01&#xA;  4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01&#xA;  5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02&#xA;&#xA;214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05&#xA;215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05&#xA;216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05&#xA;217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05&#xA;218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user14204" LastEditorUserId="29169" LastEditorDisplayName="user14204" LastEditDate="2019-05-29T13:26:36.357" LastActivityDate="2019-05-29T13:26:36.357" Title="How to interpret the output of XGBoost importance?" Tags="|machine-learning|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12319" PostTypeId="1" AcceptedAnswerId="12610" CreationDate="2016-06-21T08:39:52.053" Score="2" ViewCount="198" Body="&lt;p&gt;I'm downloading Stackoverflow questions &amp;amp; answers for a specific tag using &lt;a href=&quot;https://api.stackexchange.com/docs&quot;&gt;Stackoverflow API&lt;/a&gt; and Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to perform document &lt;a href=&quot;http://brandonrose.org/clustering&quot; rel=&quot;nofollow&quot;&gt;clustering&lt;/a&gt; to find relevant terms across the documents and find similarity among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;curl --compressed -H &quot;Accept-Encoding: gzip&quot; -X GET 'http://api.stackexchange.com/2.2/questions?page_number=1&amp;amp;pagesize=25&amp;amp;order=desc&amp;amp;sort=activity&amp;amp;tagged=amp-html&amp;amp;site=stackoverflow&amp;amp;run=true'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the procedure I'm using with Python 2.7 requests library:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Request Question  via API&lt;/li&gt;&#xA;&lt;li&gt;Create Question object and assign API response to body property after using BeautifulSoap.&lt;/li&gt;&#xA;&lt;li&gt;Find associated answers in Question.&lt;/li&gt;&#xA;&lt;li&gt;Request Answers via API&lt;/li&gt;&#xA;&lt;li&gt;Create Answer array&lt;/li&gt;&#xA;&lt;li&gt;Assign Answer array to parent Question object.&lt;/li&gt;&#xA;&lt;li&gt;For each Answer object assign API response to body property after using BeautifulSoap&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;API question request:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;curl --compressed -H &quot;Accept-Encoding: gzip&quot; -X GET 'https://api.stackexchange.com/2.1/questions/37745529?site=stackoverflow&amp;amp;filter=withbody'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;API response example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;items&quot;:[{&quot;tags&quot;:[&quot;html&quot;,&quot;input&quot;,&quot;amp-html&quot;],&quot;owner&quot;:{&quot;reputation&quot;:314,&quot;user_id&quot;:5426326,&quot;user_type&quot;:&quot;registered&quot;,&quot;accept_rate&quot;:39,&quot;profile_image&quot;:&quot;https://www.gravatar.com/avatar/cbff9d2f96733be04cb022f8a724ad49?s=128&amp;amp;d=identicon&amp;amp;r=PG&amp;amp;f=1&quot;,&quot;display_name&quot;:&quot;Pranav Bilurkar&quot;,&quot;link&quot;:&quot;http://stackoverflow.com/users/5426326/pranav-bilurkar&quot;},&quot;is_answered&quot;:true,&quot;view_count&quot;:47,&quot;accepted_answer_id&quot;:37787460,&quot;answer_count&quot;:1,&quot;score&quot;:1,&quot;last_activity_date&quot;:1465850038,&quot;creation_date&quot;:1465553340,&quot;last_edit_date&quot;:1465797429,&quot;question_id&quot;:37745529,&quot;link&quot;:&quot;http://stackoverflow.com/questions/37745529/input-tags-elements-in-amp-html&quot;,&quot;title&quot;:&quot;Input Tags/Elements in AMP-html&quot;,&quot;body&quot;:&quot;&amp;lt;p&amp;gt;we are implementing our current &amp;lt;a href=\&quot;http://articles.mercola.com/sites/articles/archive/2099/12/31/alzheimers-disease-early-detection.aspx\&quot; rel=\&quot;nofollow\&quot;&amp;gt;site&amp;lt;/a&amp;gt; in &amp;lt;code&amp;gt;AMP version&amp;lt;/code&amp;gt; and i am new to this. I have below queries regarding &amp;lt;code&amp;gt;AMP&amp;lt;/code&amp;gt; -&amp;lt;/p&amp;gt;\n\n&amp;lt;ol&amp;gt;\n&amp;lt;li&amp;gt;How to get user &amp;lt;code&amp;gt;input&amp;lt;/code&amp;gt; from user in &amp;lt;code&amp;gt;AMP-HTML&amp;lt;/code&amp;gt;.&amp;lt;/li&amp;gt;\n&amp;lt;li&amp;gt;In the above mentioned site [&amp;lt;code&amp;gt;desktop version&amp;lt;/code&amp;gt;] we have &amp;lt;code&amp;gt;comment section&amp;lt;/code&amp;gt; at the bottom of the page. We need to implement the same functionality in AMP.&amp;lt;/li&amp;gt;\n&amp;lt;li&amp;gt;Are there any &amp;lt;code&amp;gt;websites&amp;lt;/code&amp;gt; which are build and developed in &amp;lt;code&amp;gt;AMP&amp;lt;/code&amp;gt;? If yes, i need links to check.&amp;lt;/li&amp;gt;\n&amp;lt;/ol&amp;gt;\n\n&amp;lt;p&amp;gt;Any suggestion and ideas related to above queries would be appreciated.\nThanks.&amp;lt;/p&amp;gt;\n&quot;}],&quot;has_more&quot;:false,&quot;quota_max&quot;:300,&quot;quota_remaining&quot;:298}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since questions and answers contain HTML, URLs, special characters, quotes, single quotes, commas, etc in body property. I'm looking to convert this text into structured data which can be represented as a single text chunk and be analyzed as tokens and then I use TF-IDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Details:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Objects: Question and Answers&#xA;Question object has an array of answers objects as properties.&#xA;Each Question and Answer objects contain a &lt;strong&gt;body&lt;/strong&gt; property which is a single string containing each text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the recommended way to store this information? &lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-06-21T08:46:08.117" LastActivityDate="2016-07-05T15:09:43.760" Title="Stackoverflow API Structure data storage" Tags="|machine-learning|data-mining|nlp|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12320" PostTypeId="1" AcceptedAnswerId="12345" CreationDate="2016-06-21T09:40:45.333" Score="5" ViewCount="898" Body="&lt;p&gt;I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get some snippet in python to find the probability of labels.&lt;/p&gt;&#xA;" OwnerUserId="9322" LastEditorUserId="9322" LastEditDate="2016-06-22T09:30:47.577" LastActivityDate="2016-06-24T02:08:13.867" Title="Machine Learning in Spark" Tags="|bigdata|apache-spark|pyspark|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12321" PostTypeId="1" AcceptedAnswerId="12346" CreationDate="2016-06-21T10:05:08.587" Score="240" ViewCount="350253" Body="&lt;p&gt;I do not understand the difference between the &lt;code&gt;fit&lt;/code&gt; and &lt;code&gt;fit_transform&lt;/code&gt; methods in scikit-learn. Can anybody explain simply why we might need to transform data?&lt;/p&gt;&#xA;&lt;p&gt;What does it mean, fitting a model on training data and transforming to test data? Does it mean, for example, converting categorical variables into numbers in training and transforming the new feature set onto test data?&lt;/p&gt;&#xA;" OwnerUserId="15064" LastEditorUserId="43000" LastEditDate="2021-07-08T08:09:09.163" LastActivityDate="2021-07-08T08:09:09.163" Title="What's the difference between fit and fit_transform in scikit-learn models?" Tags="|python|scikit-learn|" AnswerCount="10" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="12323" PostTypeId="2" ParentId="12202" CreationDate="2016-06-21T10:46:01.563" Score="3" Body="&lt;p&gt;I found the answer in the &lt;a href=&quot;https://blog.numer.ai/2016/06/16/New-Rules-and-Round-6-Winners&quot; rel=&quot;nofollow&quot;&gt;comments section&lt;/a&gt; of their blog:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Earnings listed on the public-score leaderboard are potential winnings. Actual winnings are determined by the private-score leaderboard. At the time of withdrawal and when new datasets are released, your actual winnings will be revealed to you.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-06-21T10:46:01.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12324" PostTypeId="2" ParentId="9869" CreationDate="2016-06-21T10:46:06.970" Score="1" Body="&lt;p&gt;Kernel based methods work by constructing a feature vector based on some distance metric of an input vector to one or more example vectors, which are determined during training.These are the prototypes in your case (also known as regressors in some contexts). They are essentially vectors in the space of the independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Radial Basis functions essentially &quot;convert&quot; the (Euclidean) distance between input vector and prototype to a similarity value, usually  [0,1]. Some of them, such as the Gaussian, have parameters such as the bandwidth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In RBF Networks, there are a number of different options of determining the prototypes, for instance by Clustering at the space of independent variables, or by Orthogonal Least Squares (references at the bottom). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like you correctly assumed, the first sentence is explaining that in many cases of training algorithms (the ones mentioned above included), the bandwidth of the RBF is constant and determined a-priori What is subject to determining is the actual location of the prototypes. In this context, the prototype would refer to the value composition in the input space, but &lt;em&gt;not&lt;/em&gt; to the radius (which is fixed anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not read the paper in question, but it seems that the authors are proposing some method to individually adjust the bandwidths of the RBFs. In this case, the radius may have been included as well into the notion of the prototype, but it is specific to the research in question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References&lt;/p&gt;&#xA;&#xA;&lt;p&gt;J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2016-06-21T10:46:06.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12325" PostTypeId="1" AcceptedAnswerId="12326" CreationDate="2016-06-21T12:23:37.307" Score="4" ViewCount="658" Body="&lt;p&gt;I'm using scikit-learn RandomForestClassifier for a classification problem. When taking a closer look at one of the trees I noticed that the number of samples at the root was 662, but there were 507 instances of the first class and 545 of the second. What's going on or did I understand something wrong? Is the number of samples actually the number of unique samples and since I used bootstrap aggregation there are many samples that were chosen multiple times?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xZpTY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xZpTY.png&quot; alt=&quot;Picture of the decision tree&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20714" LastActivityDate="2016-06-21T14:22:36.943" Title="Why is the number of samples smaller than the number of values in my decision tree?" Tags="|random-forest|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12326" PostTypeId="2" ParentId="12325" CreationDate="2016-06-21T14:22:36.943" Score="3" Body="&lt;p&gt;Yes, it seems to display unique samples, the others have been duplicated by the bootstrap sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's the 0.632 rule - when you have N items and you take a random sample of size N with replacement (as bootstrap does), you only get 63.2% of the samples from N, the rest are duplicates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That roughly matches what you've seen: 0.632 * (507+545) = 665 unique samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try it with some Python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;samples = np.arange(507 + 545)&#xA;&#xA;bootstrap_sample = np.random.choice(samples, size=len(samples), replace=True)&#xA;&#xA;print(len(np.unique(bootstrap_sample)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This always prints values closely around 665.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-06-21T14:22:36.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12327" PostTypeId="1" AcceptedAnswerId="12329" CreationDate="2016-06-21T15:43:25.803" Score="12" ViewCount="12013" Body="&lt;p&gt;I'm working on a non-profit where we try to help potential university applicants by matching them with alumni that want to share their experience/wisdom and, at the moment, it is happening manually. So I'll have two tables, one with students and one with alumni (they may have some features in common, but not necessarily all of them)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$\begin{array}{|l|c|c|} \text{Name} &amp;amp; \text{Gender} &amp;amp; \text{Height} \\ \hline \text{Kathy} &amp;amp; F &amp;amp; 165 \\ \hline \text{Tommy} &amp;amp; M &amp;amp; 182 \\ \hline \text{Ruth} &amp;amp; F &amp;amp; 163 \\ \hline ... &amp;amp; ... &amp;amp; ... \\ \end{array}$ $\begin{array}{|l|c|c|} \text{Name} &amp;amp; \text{Gender} &amp;amp; \text{Weight} \\ \hline \text{Miss Lucy} &amp;amp; F &amp;amp; 65 \\ \hline \text{Miss Geraldine} &amp;amp; F &amp;amp; 70 \\ \hline \text{Miss Emily} &amp;amp; F &amp;amp; 60 \\ \hline ... &amp;amp; ... &amp;amp; ... \\ \end{array}$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Currently, we are manually matching the members of table 1 with those in table 2. We will also collect information after the match (&quot;Was it a good match? Please rate it on a scale from 1 to 10&quot;). So it will look something like this:&#xA;$$&#xA;\begin{array}{|l|l|c|} &#xA;\text{Person #1} &amp;amp; \text{Person #2} &amp;amp; \text{Match?} \\ \hline&#xA;\text{Ruth} &amp;amp; \text{Miss Lucy} &amp;amp; N \\ \hline&#xA;\text{Tommy} &amp;amp; \text{Miss Emily} &amp;amp; Y \\ \hline&#xA;\text{Kathy} &amp;amp; \text{Miss Geraldine} &amp;amp; N \\ \hline&#xA;\text{Ruth} &amp;amp; \text{Miss Emily} &amp;amp; N \\ \hline&#xA;... &amp;amp; ... &amp;amp; ... \\ &#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use a learning algorithm for this process. I know a little bit of machine learning, but I am still very much a novice (so it's also an opportunity for me to learn more about it), but I can't wrap my head around how you would do this kind of supervised learning when you have two sets both of which have multiple features. What sort of matching algorithms are available to do this? (Also, I prefer to work in R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(By the way, I would be grateful if you could just point me in the right direction and I'll try to read about it and solve it myself. Also, I know how deeply frustrating it is to see questions that have already been answered -- if this is case, please don't hesitate to let me know without answering the question. I have already tried to search for various strings on Google and StackExchange, but mostly find lecture slides on graph theory that don't seem to be what I'm looking for (although it may just be because it's a bit over my head). Many thanks!)&lt;/p&gt;&#xA;" OwnerUserId="20719" LastActivityDate="2021-04-07T20:35:55.620" Title="Which supervised learning algorithms are available for matching?" Tags="|machine-learning|beginner|similarity|supervised-learning|recommender-system|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12328" PostTypeId="2" ParentId="9568" CreationDate="2016-06-21T15:50:44.970" Score="1" Body="&lt;p&gt;Would be beneficial to know the structure of the data and what you are wanting to do with the data. Do you really need a database at all? Drill will read from a filesystem/S3/etc? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But to answer your question with the information you provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pro of MongoDB over PostgreSQL would be in scaling and read performance (after a certain scale). If this is mainly a read only, or analytic database (primarily reads) then MongoDB would probably work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PostgreSQL has the advantage of ACID compliance and more than likely better performance if you don't go above the ~10's of millions of records (especially with JSONB, which supports indexes on the datatype).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing much about the data, you could simply have two columns within your postgresql JSON table (id, JSON_field) and then use the functions within PostgreSQL to query your JSON field. It is a little more involved in querying JSON data within PostgreSQL but after you get used to it, it is pretty simple.&lt;/p&gt;&#xA;" OwnerUserId="14829" LastActivityDate="2016-06-21T15:50:44.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12329" PostTypeId="2" ParentId="12327" CreationDate="2016-06-21T16:04:43.160" Score="5" Body="&lt;p&gt;You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not a perfect fit as you want just one item for each user and you don't have previous match data for each user. However you could investigate this idea a bit further. I'm applying these techniques to the recruitment problem, I'm matching users with job offers and I'm having some success. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to read a bit about recommender systems, to start I recommend chapter 9 of mining massive data sets, it's really introductory, but gives a good overview of the most common techniques. &lt;/p&gt;&#xA;" OwnerUserId="11121" LastActivityDate="2016-06-21T16:04:43.160" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12330" PostTypeId="1" CreationDate="2016-06-21T17:38:56.907" Score="1" ViewCount="1778" Body="&lt;p&gt;Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I'd like to group together all the articles that are speaking about the same thing. However :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I want my algorithm to define the topics by itself (in opposition to &quot;the user tells the existing topics and the algorithm assigns each article to the topics&quot;), &lt;/li&gt;&#xA;&lt;li&gt;I can't know the precise number topics there are (because, obviously, a new one has to be created each time something new happens),&lt;/li&gt;&#xA;&lt;li&gt;and, as we are talking about news article, the list of topics should be expanding in real time if something new happens and new articles talk about it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For simplicity's sake, let's assume all the articles are in the same language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been using TF-IDF for this as of now, because that's what I came up with on the top of my head when I knew nothing about topic modeling, but it's not really good at its job. I'm getting into topic modeling and discovering things such as Latent Dirichlet Allocation, Correlated Topic Model, graph of words etc. I'm currently reading through all I can on the subject, but I'm probably unaware of interesting algorithms that could fit my need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested : what approach would you take to tackle my problem ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="18306" LastEditorUserId="18306" LastEditDate="2017-07-07T08:36:13.003" LastActivityDate="2017-07-07T08:36:13.003" Title="Real time topic identification of news article" Tags="|nlp|text-mining|information-retrieval|topic-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12333" PostTypeId="2" ParentId="10856" CreationDate="2016-06-22T00:19:37.747" Score="1" Body="&lt;p&gt;Sounds like you are set to do &lt;a href=&quot;http://recommender-systems.org/collaborative-filtering/&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt; - you have &lt;strong&gt;readers&lt;/strong&gt; (users) and &lt;strong&gt;documents&lt;/strong&gt; (items) creating a unary-response matrix with 1 cells indicating the reader (row) read the given document (col). There are various types of recommender systems (e.g., &lt;strong&gt;user-based&lt;/strong&gt; and &lt;strong&gt;item-based&lt;/strong&gt; collaborative filters) and several of them can account for the meta-data associated with your documents (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system#Content-based_filtering&quot; rel=&quot;nofollow&quot;&gt;content-based systems&lt;/a&gt;). You should look into R package &lt;a href=&quot;https://cran.r-project.org/web/packages/recommenderlab/recommenderlab.pdf&quot; rel=&quot;nofollow&quot;&gt;{recommenderlab}&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-06-22T00:38:26.093" LastActivityDate="2016-06-22T00:38:26.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12334" PostTypeId="2" ParentId="10856" CreationDate="2016-06-22T00:26:50.860" Score="0" Body="&lt;p&gt;Given that your data is categorical, one approach that might work is latent class clustering. There is a huge literature out there on this set of models and an R module that is fairly easy to run. See &lt;em&gt;poLCA: An R Package for Polytomous Variable Latent Class Analysis&lt;/em&gt; by Linzer and Lewis published in the Journal of Statistical Software, June 2011, Volume 42, Issue 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;LCMs are a subset of finite mixture models and were first developed in the 50s by Paul Lazarsfeld, a Columbia sociologist. They have seen extensive development since and are probably most widely used for segmentations by marketing scientists. If you have $1,000 to spend, you can purchase a perpetual license for &lt;em&gt;Latent Gold&lt;/em&gt; from Statistical Innovations. In my opinion, LG is the single best tool for LCMs. &lt;/p&gt;&#xA;" OwnerUserId="9974" LastActivityDate="2016-06-22T00:26:50.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12337" PostTypeId="1" CreationDate="2016-06-22T05:34:00.267" Score="0" ViewCount="78" Body="&lt;p&gt;I have a bunch of sales data of similar products and I want to know which product might be the best popular one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for the same category of items, there are cheap ones and expensive ones, and the sales volume of the cheap ones are always higher than the expensive ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, we have some ceiling lamps which cost 45\$, and some are 200\$. If the sales volumes for the 45\$ ones and 200$ are 200 and 180 respectively, even the cheap one is sold more than the expensive one. We know that the expensive one is actually more popular then the cheap one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I model the popularity by this kind of information?&lt;/p&gt;&#xA;" OwnerUserId="20736" LastActivityDate="2016-08-21T09:54:28.963" Title="How to model product popularity by their prices and sales volume?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12339" PostTypeId="2" ParentId="12327" CreationDate="2016-06-22T07:16:13.460" Score="1" Body="&lt;p&gt;I would have separate the problem into two:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predicting whether a certain pair will be a good match.&lt;/li&gt;&#xA;&lt;li&gt;Matching the pairs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;First, lets discuss the prediction problem.&#xA;I think you should treat matching the pairs as a supervised learning problem and not as a recommendation problem.&#xA;As João Almeida wrote new student won't have any previous relations with alumni.&lt;br&gt;&#xA;Even the alumnus will have very few previous relations. &#xA;I would have add to each alumni some features based on aggregations (e.g., the number of past relations, the ratio of past good matches). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that you should build a dataset of the past pairs, using 'Match?' as the concept.&#xA;It is not clear whether you will be able to learn a good match rule, even if it exists.&#xA;I guess that your dataset is quite small. If the probability of a match is low, you might have imbalance problem. &#xA;As AN6U5 commented, height and weight are quite strange features to match students to alumnus.&#xA;Compute the relations between the features and the concept (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow&quot;&gt;mutual information&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;Pearson correlation&lt;/a&gt;) in order to see if you have useful features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second question, even if you can predict well if a pair will be a good match, you still have an algorithmic problem of which pair to use.&#xA;Consider a &quot;super alumni&quot; that will be a good match to any student. You wouldn't like to match it to a &quot;super student&quot; but to a student that it will be hard to match to other alumni.&#xA;Luckily, there matching algorithms that you can use.&#xA;Build a graph with the students and alumnus as nodes. Create an edge if you predict a good match and run a &lt;a href=&quot;https://en.wikipedia.org/wiki/Matching_(graph_theory)&quot; rel=&quot;nofollow&quot;&gt;matching algorithm&lt;/a&gt; upon it.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-06-22T07:16:13.460" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12340" PostTypeId="1" AcceptedAnswerId="12342" CreationDate="2016-06-22T07:45:55.347" Score="2" ViewCount="104" Body="&lt;p&gt;This is my first post here. I hope I can make myself clear. Right now I'm learning linear regression as part of an introduction class to machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After going over the steps in the simple regression formulae, I realized that I had been doing something similar in the past to construct lines. In Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = [{'x': 1, 'y': 2}, {'x': 5, 'y': 7}, {'x': 6, 'y': 8}]&#xA;coeff = sum([d['x'] / d['y'] for d in data]) / len(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, we're calculating the mean ratio between the variables, which we can use as a coefficient for constructing a line. Does this method have a name, and how does it relate to simple linear regression?&lt;/p&gt;&#xA;" OwnerUserId="20743" LastActivityDate="2016-06-22T08:16:18.687" Title="What is this formula, related to simple linear regression, called?" Tags="|machine-learning|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12341" PostTypeId="2" ParentId="12337" CreationDate="2016-06-22T08:07:29.373" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;We know that the expensive one is actually more popular then the cheap one&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What makes you think that? Apparently you have a reason for that, so start from there, find those attributes - is it high customer satisfaction? positive reviews? repurchase? low retention? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gather those attributes together, evaluate them on all products to see what are the popular ones. It'd help to have a proper definition of popularity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sales volume can help too, give it a weight to balance the price difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have all this in one table, you can start playing with prediction, clustering, trends etc.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-06-22T08:07:29.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12342" PostTypeId="2" ParentId="12340" CreationDate="2016-06-22T08:16:18.687" Score="1" Body="&lt;p&gt;I do not know other terms than the  average of inverse slope or the inverse of the harmonic mean of slopes. This is also the negative of the average slope of the perpendiculars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It gives you the inverse of the average slope of lines passing through $(0,0)$ and $(x_i,y_i)$. If the $(x,y)$ are almost aligned with $(0,0)$, this is an estimate of the inverse of the slope of a line passing through the points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear regression is quite different, as it involves cross-products of sums of $x$ or $y$.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastActivityDate="2016-06-22T08:16:18.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12343" PostTypeId="1" CreationDate="2016-06-22T09:25:26.287" Score="3" ViewCount="23254" Body="&lt;p&gt;I have DataFrame which has column with spanish regions, when I import the .csv as following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_csv('...',delimiter=';',decimal=',')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And print the content of column, I see this: Castilla y Le�n.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could not find a way how to fix it, but I am assuming there is smart way how to do it.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2021-05-12T13:46:27.160" Title="Pandas - read CSV with spanish characters" Tags="|python|pandas|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12344" PostTypeId="1" AcceptedAnswerId="12349" CreationDate="2016-06-22T11:13:57.487" Score="1" ViewCount="84" Body="&lt;p&gt;I want to build a classifier for my problem statement and for that I don't have data. So while doing data acquisition, what should be the minimum sample size? And would it be a good practice if I label each observation myself to build a valid data set? (I cannot automate the process of labeling observation to each class while doing data acquisition and manually doing that takes up lot of time)&lt;/p&gt;&#xA;" OwnerUserId="20748" LastEditorUserId="676" LastEditDate="2016-06-22T12:20:41.120" LastActivityDate="2016-06-22T14:19:35.127" Title="Characterisitcs of the data set for a binary classification problem" Tags="|machine-learning|classification|data-mining|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12345" PostTypeId="2" ParentId="12320" CreationDate="2016-06-22T11:26:12.013" Score="7" Body="&lt;p&gt;Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is &lt;strong&gt;model&lt;/strong&gt;  then  &lt;code&gt;model.transform(test)&lt;/code&gt; contains a node of &lt;strong&gt;probability&lt;/strong&gt;, for more details please check the below code, going to show you the probability node and others useful nodes also for &lt;strong&gt;iris&lt;/strong&gt; dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Partition dataset randomly into Training and Test sets. Set seed for reproducibility&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(trainingData, testData) = irisdf.randomSplit([0.7, 0.3], seed = 100)&#xA;&#xA;trainingData.cache()&#xA;testData.cache()&#xA;&#xA;print trainingData.count()&#xA;print testData.count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;103&#xA;47&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, we will use the VectorAssembler() to merge our feature columns into a single vector column, which we will be passing into our Naive Bayes model. Again, we will not transform the dataset just yet as we will be passing the VectorAssembler into our ML Pipeline.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.ml.feature import VectorAssembler&#xA;vecAssembler = VectorAssembler(inputCols=[&quot;SepalLength&quot;, &quot;SepalWidth&quot;, &quot;PetalLength&quot;, &quot;PetalWidth&quot;], outputCol=&quot;features&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;iris dataset&lt;/strong&gt;, it has three classes namely &lt;strong&gt;setosa&lt;/strong&gt;, &lt;strong&gt;versicolor&lt;/strong&gt; and &lt;strong&gt;virginica&lt;/strong&gt;. So let's create a &lt;strong&gt;Multiclass Naive Bayes Classifier&lt;/strong&gt; using pysaprk library &lt;code&gt;ml&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.ml.classification import NaiveBayes&#xA;from pyspark.ml import Pipeline&#xA;&#xA;# Train a NaiveBayes model&#xA;nb = NaiveBayes(smoothing=1.0, modelType=&quot;multinomial&quot;)&#xA;&#xA;# Chain labelIndexer, vecAssembler and NBmodel in a pipeline&#xA;pipeline = Pipeline(stages=[labelIndexer, vecAssembler, nb])&#xA;&#xA;# Run stages in pipeline and train model&#xA;model = pipeline.fit(trainingData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Analyse the created mode &lt;strong&gt;model&lt;/strong&gt;, from which we can make predictions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predictions = model.transform(testData)&#xA;# Display what results we can view&#xA;predictions.printSchema()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;root&#xA; |-- SepalLength: double (nullable = true)&#xA; |-- SepalWidth: double (nullable = true)&#xA; |-- PetalLength: double (nullable = true)&#xA; |-- PetalWidth: double (nullable = true)&#xA; |-- Species: string (nullable = true)&#xA; |-- label: double (nullable = true)&#xA; |-- features: vector (nullable = true)&#xA; |-- rawPrediction: vector (nullable = true)&#xA; |-- probability: vector (nullable = true)&#xA; |-- prediction: double (nullable = true)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also select a particular node to view for some dataset as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# DISPLAY Selected nodes only&#xA;display(predictions.select(&quot;label&quot;, &quot;prediction&quot;, &quot;probability&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Above will show you in tabular formate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-naive-bayes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;spark&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://spark.apache.org/docs/latest/ml-classification-regression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Models using pipeline&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://mike.seddon.ca/natural-language-processing-with-apache-spark-ml-and-amazon-reviews-part-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://mike.seddon.ca/natural-language-processing-with-apache-spark-ml-and-amazon-reviews-part-1/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/31028806/how-to-create-correct-data-frame-for-classification-in-spark-ml&quot;&gt;https://stackoverflow.com/questions/31028806/how-to-create-correct-data-frame-for-classification-in-spark-ml&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-06-24T02:08:13.867" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12346" PostTypeId="2" ParentId="12321" CreationDate="2016-06-22T12:18:02.977" Score="245" Body="&lt;p&gt;To &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;noreferrer&quot;&gt;center the data&lt;/a&gt; (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$x' = \frac{x-\mu}{\sigma}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;You do that on the training set of the data. But then you have to apply the same transformation to your test set (e.g. in cross-validation), or to newly obtained examples before forecasting. But you have to use the exact same two parameters &lt;span class=&quot;math-container&quot;&gt;$\mu$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; (values) that you used for centering the training set.&lt;/p&gt;&#xA;&lt;p&gt;Hence, every scikit-learn's transform's &lt;code&gt;fit()&lt;/code&gt; just calculates the parameters (e.g. &lt;span class=&quot;math-container&quot;&gt;$\mu$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; in case of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html&quot; rel=&quot;noreferrer&quot;&gt;StandardScaler&lt;/a&gt;) and saves them as an internal object's state. Afterwards, you can call its &lt;code&gt;transform()&lt;/code&gt; method to apply the transformation to any particular set of examples.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;fit_transform()&lt;/code&gt; joins these two steps and is used for the initial fitting of parameters on the training set &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;, while also returning the transformed &lt;span class=&quot;math-container&quot;&gt;$x'$&lt;/span&gt;. Internally, the transformer object just &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/6c3e17989a7d80c34f124365f2c436a3fdcb1497/sklearn/base.py#L659-L690&quot; rel=&quot;noreferrer&quot;&gt;calls first &lt;code&gt;fit()&lt;/code&gt; and then &lt;code&gt;transform()&lt;/code&gt;&lt;/a&gt; on the same data.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="29169" LastEditDate="2021-06-19T21:46:14.237" LastActivityDate="2021-06-19T21:46:14.237" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="12347" PostTypeId="1" AcceptedAnswerId="12353" CreationDate="2016-06-22T13:35:13.040" Score="3" ViewCount="190" Body="&lt;p&gt;A response variable (label) $B$ can either be $0$ or $1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the training set, $B_i = 1$ is an extremely rare event at only $0.26\%$ occurrences. Which makes the prediction of this label on a test data-set a difficult problem.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I used &lt;code&gt;SMOTE&lt;/code&gt; to sample from the training-set of some $1.55 \times 10^5$ rows to obtain a completely balanced set of $620$ rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;balanced.df &amp;lt;- SMOTE(B ~ ., df, perc.over = 100, perc.under = 200)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;randomForest&lt;/code&gt; with $1000$ trees was used to fit the model, as shown:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;randomForest.Fit &amp;lt;- randomForest(B ~ ., data = balanced.df, ntree = 1000)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For making a validation set, $2000$ rows were sampled at random &lt;em&gt;without repetition&lt;/em&gt; from the data-set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The actual frequencies of $B_i$ in the validation set are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   0    1 &#xA; 1998   2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And those in the predicted set are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   0    1 &#xA; 1836  164&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t8VfG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t8VfG.png&quot; alt=&quot;ROC Curve&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results seem promising, but perhaps a little too much. Also, it is essential that the percentage of False Positives are reduced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My questions are:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;How severely do you think the skew in data is affecting the validation results?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a point in validating again by creating an arbitrary data-set with bias, for example by selecting more $B_i = 1$ in the validation set?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What other metrics/validation techniques which reflect the &quot;accuracy*&quot; of prediction?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;*The term accuracy is used in a generic sense.&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2016-06-23T10:23:51.327" LastActivityDate="2016-06-23T10:23:51.327" Title="Understanding ROCs in imbalanced data-sets" Tags="|r|random-forest|performance|accuracy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12348" PostTypeId="1" CreationDate="2016-06-22T13:46:43.247" Score="2" ViewCount="127" Body="&lt;p&gt;I’m doing an “analytics” project in which I have to collect data from small businesses (SMEs) in South America and try to come up with some insights about what are their needs in general. The project’s coach basically wants to collect data from social media and business association websites and do some text analytic on it to see if we can gain some insights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, the idea of the project is that given a country and a sector, find what the companies in those sectors are up to and see what are their problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I've thought:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Find keywords in the twitter API and facebook API.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find News about SMEs in a given country and crawl the content. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find associations web sites. I dis this but the content here is useless most of the time. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The goal of this project is to gain some insights and see if some new products can be customized to these companies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, my question is: how would you solve a problem like this if you do not have any data? I was also thinking about getting some contact information from a sample of this companies and send them all a survey or something.&lt;/p&gt;&#xA;" OwnerUserId="10155" LastActivityDate="2016-06-24T20:47:07.597" Title="Data sources problem" Tags="|data-mining|data|beginner|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12349" PostTypeId="2" ParentId="12344" CreationDate="2016-06-22T14:19:35.127" Score="2" Body="&lt;p&gt;Unfortunately you're not going to be able to do much without at least 200-300 records. You're going to be limited to simple (i.e. mostly linear) models until your dataset expands to at least 1,000. Anything less than 1,000 will require very thorough cross validation, and if you're not careful you'll be at risk of building a model that easily overfits. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;@EricLecoutre makes a great point that you should use Amazon's Mechanical Turk. It usually costs just a penny or two per record and could save you a lot of time. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-22T14:19:35.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12351" PostTypeId="2" ParentId="12343" CreationDate="2016-06-22T16:17:12.307" Score="3" Body="&lt;p&gt;&lt;code&gt;read_csv&lt;/code&gt; has an optional argument called &lt;code&gt;encoding&lt;/code&gt; that deals with the way your characters are encoded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can give a try to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pandas.read_csv('...', delimiter = ';', decimal = ',', encoding = 'utf-8')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you have to check how your characters are encoded (It is one of &lt;a href=&quot;https://docs.python.org/3/library/codecs.html#standard-encodings&quot; rel=&quot;nofollow&quot;&gt;them&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read the doc of &lt;code&gt;read_csv&lt;/code&gt; &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20759" LastActivityDate="2016-06-22T16:17:12.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12352" PostTypeId="1" AcceptedAnswerId="12358" CreationDate="2016-06-22T17:26:53.710" Score="1" ViewCount="116" Body="&lt;p&gt;I recently started learning Hadoop,&#xA;I found this data set &lt;a href=&quot;http://stat-computing.org/dataexpo/2009/the-data.html&quot; rel=&quot;nofollow&quot;&gt;http://stat-computing.org/dataexpo/2009/the-data.html&lt;/a&gt; - (2009 data),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want some suggestions as what type of patterns or analysis can I do in Hadoop MapReduce, i just need something to get started with, If anyone has a better data set link which I can use for learning, help me here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The attributes are as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1   Year    1987-2008&#xA;2   Month   1-12&#xA;3   DayofMonth  1-31&#xA;4   DayOfWeek   1 (Monday) - 7 (Sunday)&#xA;5   DepTime actual departure time (local, hhmm)&#xA;6   CRSDepTime  scheduled departure time (local, hhmm)&#xA;7   ArrTime actual arrival time (local, hhmm)&#xA;8   CRSArrTime  scheduled arrival time (local, hhmm)&#xA;9   UniqueCarrier   unique carrier code&#xA;10  FlightNum   flight number&#xA;11  TailNum plane tail number&#xA;12  ActualElapsedTime   in minutes&#xA;13  CRSElapsedTime  in minutes&#xA;14  AirTime in minutes&#xA;15  ArrDelay    arrival delay, in minutes&#xA;16  DepDelay    departure delay, in minutes&#xA;17  Origin  origin IATA airport code&#xA;18  Dest    destination IATA airport code&#xA;19  Distance    in miles&#xA;20  TaxiIn  taxi in time, in minutes&#xA;21  TaxiOut taxi out time in minutes&#xA;22  Cancelled   was the flight cancelled?&#xA;23  CancellationCode    reason for cancellation (A = carrier, B = weather, C     = NAS, D = security)&#xA;24  Diverted    1 = yes, 0 = no&#xA;25  CarrierDelay    in minutes&#xA;26  WeatherDelay    in minutes&#xA;27  NASDelay    in minutes&#xA;28  SecurityDelay   in minutes&#xA;29  LateAircraftDelay   in minutes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="20766" LastActivityDate="2016-06-22T23:43:23.577" Title="Suggestions on what patterns/analysis to derive from Airlines Big Data" Tags="|apache-hadoop|map-reduce|" AnswerCount="1" CommentCount="0" ClosedDate="2016-06-23T09:45:06.213" ContentLicense="CC BY-SA 3.0" />
  <row Id="12353" PostTypeId="2" ParentId="12347" CreationDate="2016-06-22T17:50:06.040" Score="2" Body="&lt;p&gt;So your data-set of 155000 records has 403 records where B=1, and B=0 for the remaining 154597 records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try splitting your data-set into 2:1 training/test sets sampled by each class of B. After you've done this, then only for the training set use SMOTE to over-sample the records with B=1 along with under-sampling the B=0 training records to bring the class ratio to something like 4:1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over/under sampling for the test set is not required as it is supposed to mimic real world uncertainty to test your model's performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model's AUC will definitely get reduced since (as rightly pointed out by stmax), you've leaked test records into the training set by over sampling B=1 cases before splitting the train-test sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answers to each of your questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Yes, class imbalance does effect a random forest model's accuracy. How severely depends on the severity of the imbalance as well as the nature of the data-set itself.&lt;/li&gt;&#xA;&lt;li&gt;Yes, you are biasing the training by over sampling the minority class, but if these are large enough samples in the original set, then hopefully they are close representations to the entire population.&lt;/li&gt;&#xA;&lt;li&gt;I would recommend two metrics/techniques you could use here: Kappa Statistic (refer to &lt;a href=&quot;https://en.wikipedia.org/wiki/Cohen&amp;#39;s_kappa&quot; rel=&quot;nofollow&quot; title=&quot;this article&quot;&gt;this article&lt;/a&gt;) and the precision-recall curves to compare different models.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-06-22T17:50:06.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12354" PostTypeId="1" CreationDate="2016-06-22T18:09:12.363" Score="1" ViewCount="921" Body="&lt;p&gt;I am working with an organization to analyse their data residing in Mongodb and to look for any trends/patterns in the data. I am quite new to the professional field of Data Analysis but have a good background of Statistics and Data Mining (University coursework). I will be doing a proof of concept on the data to understand if the data the organization is gathering is good for Analytics and if no what enhancements should they include in their datasets to make it better for Analytics. I do have some predefined questions that I am planning to answer but looking strategically what would be a good way to approach this kind of problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have previously worked with some of the Kaggle datasets and projects in University where datasets were fixed and questions to be answered were given.&lt;/p&gt;&#xA;" OwnerUserId="20722" LastEditorUserId="20722" LastEditDate="2016-06-23T17:36:08.547" LastActivityDate="2016-06-23T17:36:08.547" Title="What is a good way to start Data Analysis of unknown dataset (JSON data)" Tags="|machine-learning|data-mining|data-cleaning|mongodb|json|" AnswerCount="2" CommentCount="1" ClosedDate="2016-06-23T09:45:38.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="12355" PostTypeId="1" AcceptedAnswerId="12360" CreationDate="2016-06-22T18:35:42.283" Score="2" ViewCount="2183" Body="&lt;p&gt;I build a classification model based on SVM and getting same results after running different kernels. Can you please let me know if is mistake ? also recall for all are identical. Thank you for help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding the location for the notebook and data.&#xA;&lt;a href=&quot;https://github.com/sergiubuciumas/ML/blob/master/HW1_SVM_completed.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVC repo with the notebook and data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lJpmt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lJpmt.png&quot; alt=&quot;Support Vector Classification scores &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/G5DVd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/G5DVd.png&quot; alt=&quot;Accuracy score&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lNu47.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lNu47.png&quot; alt=&quot;recall score&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="8752" LastEditDate="2016-06-22T20:07:30.127" LastActivityDate="2016-06-23T01:30:35.477" Title="Support Vector Classification kernels ‘linear’, ‘poly’, ‘rbf’ has all same score" Tags="|machine-learning|svm|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12356" PostTypeId="2" ParentId="11417" CreationDate="2016-06-22T20:56:07.530" Score="3" Body="&lt;p&gt;(full disclosure, I do work for GE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GE itself has been using their GE Predix Anomaly detection algorithms to establish a relationship between anomalies &amp;amp; machine failure for their factories, in an effort to establish a preventive maintenance framework. The algorithms have been heavily tested to catch any point anomalies from a machine data.  Go ahead and test it out for your datasets: The algorithms are available for registered developers (in catalog) to use in their applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technical approaches (algorithms): Univariate/Multi-variate anomaly detection algorithms.  It detect Shift anomalies , Unexpected trends in physical system behaviors, Anomalous patterns &amp;amp; detect changes in relationship among different variables. It also detects data with very high dimensions (&gt;100). Mann-Kendall, Mann-Whitney, Kernel density estimate etc are some of the key algorithms used.&lt;/p&gt;&#xA;" OwnerUserId="20780" LastActivityDate="2016-06-22T20:56:07.530" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12357" PostTypeId="2" ParentId="12354" CreationDate="2016-06-22T21:12:34.810" Score="1" Body="&lt;p&gt;@Emre makes a great point. You should probably ask the business what they're trying to accomplish because only then will you know which data matters. You don't want to waste time cleaning up data that has minimal importance to the business. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of the business objective though, it's worth pointing out that Mongo DB is a pretty poor choice for storing analytical data. There is no enforced schema, so each record might be in a completely different and unexpected format. You might have entire columns missing, maybe no columns at all, wrong data types, duplicates, and no way to get easy summary statistics like you would in a relational database. You're pretty much going to have to do a database dump of your MongoDB data and probably comb through your records semi-manually, looking for commonly occurring schemas and going from there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary: MongoDB is great for operational data when all you want to do a quick lookup. It's a poor choice when you want to do any sort of in depth analysis on it though. Relational databases are better for analytical queries because the data is structured and also because you can more easily enforce data quality. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-22T21:12:34.810" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12358" PostTypeId="2" ParentId="12352" CreationDate="2016-06-22T23:43:23.577" Score="2" Body="&lt;p&gt;There really is no wrong answer here, but I recommend predicting flight cancellations (#22) and/or delays (25-29), since this is how I often see this data set being used. It could also have practical significance to you if you should ever find yourself flying to or departing from one of the worst offending airports/airlines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if you have a choice (perhaps your employer requires it), but don't use Map Reduce -- it's incredibly difficult to learn/maintain, it's slow, and on top of that it has become obsolete. Use something like Spark's ML lib (&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-guide.html&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/docs/latest/mllib-guide.html&lt;/a&gt;). It's much easier to use and is much more current. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-22T23:43:23.577" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12360" PostTypeId="2" ParentId="12355" CreationDate="2016-06-23T01:24:16.243" Score="3" Body="&lt;p&gt;&lt;strong&gt;You will love the answer to this one...&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at your code and notice that you are calling the scoring function and each time you are passing in the exact same values i.e. they are all spitting out the lin_svc.score().  Try interweaving the four scoring calls below the four respective fit calls and you should see the desired variation in the result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# we create an instance of SVM and fit out data. We do not scale our&#xA;# data since we want to plot the support vectors&#xA;#rbf is for gaussian&#xA;C = 1.0  # SVM regularization parameter&#xA;&#xA;svc = svm.SVC(kernel='linear', C=C).fit(X_train, y_train)&#xA;print svc.score(X_test, y_test)&#xA;&#xA;rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X_train, y_train)&#xA;print rbf_svc.score(X_test, y_test)&#xA;&#xA;poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X_train, y_train)&#xA;print poly_svc.score(X_test, y_test)&#xA;&#xA;lin_svc = svm.LinearSVC(C=C).fit(X_train, y_train)&#xA;print lin_svc.score(X_test, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly, you are doing the same thing below also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-06-23T01:30:35.477" LastActivityDate="2016-06-23T01:30:35.477" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12362" PostTypeId="2" ParentId="12354" CreationDate="2016-06-23T07:45:16.903" Score="1" Body="&lt;p&gt;Whatever business is behind this data, the biggest challenge is to try extract fields/keys that will bring valuable information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could be done by browsing data, get the schema and try to find deviations. As mongo uses dynamic schema this will give you possibility to observe different documents type in one collection - and can be source on valuable/invaluable data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use robomongo  or mongodb compass to have some data visualisation and get a grip.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning mongo aggregation framework and/or map reduce syntax is a must in such case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy mining! &lt;/p&gt;&#xA;" OwnerUserId="20789" LastActivityDate="2016-06-23T07:45:16.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12363" PostTypeId="2" ParentId="10494" CreationDate="2016-06-23T10:48:33.470" Score="1" Body="&lt;p&gt;First, I'll summarize my understanding of your query and the graphs provided:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The 4 charts represent time on x-axis and value of a random variable on y-axis.&lt;/li&gt;&#xA;&lt;li&gt;You want to understand whether the underlying process can be decomposed into erratic components and periodic component, and while doing so be able to predict the value at the next point in time.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This appears to be a time-series problem, as noted by Marcus D.&#xA;I would suggest, you could try the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Try decomposing the time-series using Holt-Winters (try both additive and multiplicative models)&lt;/li&gt;&#xA;&lt;li&gt;Try transforming the variable by taking difference from previous value, log difference, etc. which may yield a variable better suited for such filtering approaches.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The seasonal and trend components should give you insights into any predictability of this process. Sometimes domain-specific hypothesis could give good leads for next steps for the analysis.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-06-23T10:48:33.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12365" PostTypeId="1" CreationDate="2016-06-23T13:18:32.647" Score="1" ViewCount="11847" Body="&lt;p&gt;I have a time-series. The index is weekly dates and the values are a certain indicator that I made. I think I understand how to apply KNN in this situation but I'm not sure how exactly to do it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I currently have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Set lookback period to 200 rows (which is 200 weeks)&#xA;2. Set the KNN value to 10 Nearest Neighbors&#xA;3. Start with the 201 st row&#xA;4. Get the previous 200 days&#xA;5. Do Knn.fit(x = prev_200_row, y = profit_after_each_row, neighbors = 10)&#xA;6. Do Knn.predict(current_row)&#xA;7. Save the prediction to a list&#xA;8. Go to the next row and repeat steps 4, 5, 6, 7, 8 until no more rows.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this is a correct use of the process? Eventually, I would like to compare different values for K (10 nearest neighbors or maybe 50 nearest neighbors).&#xA;I am going to build on this so I want to make sure that I am starting correctly. I am wondering if it is correct that the Knn.fit is being called after every row but it seems correct to me. &lt;/p&gt;&#xA;" OwnerUserId="20800" LastEditorUserId="924" LastEditDate="2016-07-05T19:36:58.997" LastActivityDate="2016-07-05T19:36:58.997" Title="What is the correct way to apply KNN to a time-series using a rolling window?" Tags="|machine-learning|classification|time-series|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12366" PostTypeId="1" AcceptedAnswerId="16518" CreationDate="2016-06-23T15:20:40.973" Score="0" ViewCount="2444" Body="&lt;p&gt;I'm learning orange and I'm looking to performe some statistical test on my data.&lt;br&gt;&#xA;The test I usually do are z-test, t-test, chi-square test and Anova.&#xA;Can I use a widget to do these or should I do a python custom script ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank in advance. &lt;/p&gt;&#xA;" OwnerUserId="14562" LastActivityDate="2017-01-25T08:32:46.303" Title="Do statistical test with orange between data" Tags="|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12367" PostTypeId="2" ParentId="12365" CreationDate="2016-06-23T15:36:00.563" Score="1" Body="&lt;p&gt;I never used KNN on time series. I didn't know it was possible before reading your question. But by googling it found &lt;a href=&quot;http://people.revoledu.com/kardi/tutorial/KNN/KNN_TimeSeries.htm&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; tutorial that feel pretty clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if understand you correctly what you do with KNeighborsRegressor, by repeating 4 - 5 - 6 you will based the later predictions on what you predicted for the previous row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not intended you can just predict every row following the 200 used for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="14562" LastActivityDate="2016-06-23T15:36:00.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12371" PostTypeId="2" ParentId="10225" CreationDate="2016-06-23T18:20:35.073" Score="4" Body="&lt;p&gt;If you need SQL code that runs various outlier detection methods against any arbitrary table, check &lt;a href=&quot;https://multidimensionalmayhem.wordpress.com/category/diy-data-mining/outlier-detection-with-sql-server/&quot; rel=&quot;nofollow&quot;&gt;out my series of articles and code samples&lt;/a&gt; geared towards SQL Server. I provide some preliminary code for Grubb's Tests, Z-Scores and Modified Z-Scores, Interquartile Range, Dixon's Q-Test, GESD, the Tietjen-Moore Test, Pierce's Criterion, Chauvenet's Criterion, the Modified Thompson Tau Test, Cook's Distance, Mahalanobis Distance, Benford's Law and various visual plots. Please note that I'm an amateur at this and that this is some of my older code, so I'm sure corrections will be needed; this at least provides a starting point so that you can run these tests against any database table you have permissions for. It's also in T-SQL, so you'll need to adjust the code for PL/SQL or whatever other brand of SQL you might be using, if you happen to be on some other platform like Oracle or MySql. This should get you started though. Just works out the kinks and set the stored procedures to run periodically on a schedule and you're good to go. Let me know if you have any feedback on the routines, I'm always looking to improve my SQL writing.&lt;/p&gt;&#xA;" OwnerUserId="18544" LastActivityDate="2016-06-23T18:20:35.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12372" PostTypeId="2" ParentId="11364" CreationDate="2016-06-23T18:27:30.893" Score="0" Body="&lt;p&gt;Why not compute the inter-cluster distance as the average Mahalanobis distance from each point in a cluster A to each point in cluster B? &lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-06-23T18:27:30.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12373" PostTypeId="2" ParentId="540" CreationDate="2016-06-23T19:07:50.020" Score="0" Body="&lt;p&gt;A post by Donne Martin will give you a good idea of performing data analysis on a dataset to make it a &lt;strong&gt;tidy dataset&lt;/strong&gt; in order to apply machine learning algorithms.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://nbviewer.jupyter.org/github/donnemartin/data-science-ipython-notebooks/blob/master/kaggle/titanic.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Machine Learning Competition: Predicting Titanic Survivors | Jupyter nbviewer&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also if you are a beginner I would definitely recommend&#xA;&lt;a href=&quot;http://www.kdnuggets.com/2015/11/seven-steps-machine-learning-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;7 Steps to Mastering Machine Learning With Python | KDnuggets&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20722" LastEditorUserId="85045" LastEditDate="2021-03-30T11:52:05.423" LastActivityDate="2021-03-30T11:52:05.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12374" PostTypeId="2" ParentId="12148" CreationDate="2016-06-23T19:49:20.357" Score="6" Body="&lt;p&gt;So your query is a comparison of linear regression vs. random forest's model-derived importance of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lasso finds linear regression model coefficients by applying regularization. A popular approach to rank a variable's importance in a linear regression model is to decompose $R^2$ into contributions attributed to each variable. But variable importance is not straightforward in linear regression due to correlations between variables. Refer to the document describing the PMD method (Feldman, 2005) in the references below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another popular approach is averaging over orderings (LMG, 1980). The LMG works like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Find the semi-partial correlation of each predictor in the model, e.g. for variable a we have: $SS_a/SS_{total}$. It implies how much would $R^2$ increase if variable $a$ were added to the model.&lt;/li&gt;&#xA;&lt;li&gt;Calculate this value for each variable for each order in which the variable gets introduced into the model, i.e. {$a,b,c$} ; {$b,a,c$} ; {$b,c,a$}&lt;/li&gt;&#xA;&lt;li&gt;Find the average of the semi-partial correlations for each of these orders. This is the average over orderings.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The random forest algorithm fits multiple trees, each tree in the forest is built by randomly selecting different features from the dataset. The nodes of each tree are built up by choosing and splitting to achieve maximum variance reduction. While predicting on the test dataset, the individual trees output is averaged to obtain the final output. Each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method by which the model is fit on the training data is very different for a linear regression model as compared to random forest model.  But both models don't contain any structural relationships between the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your query about non-linearity of the dependent variable: The lasso is essentially a linear model which will not be able to give good predictions for an underlying non-linear processes, as compared to tree based models. You should be able to check this by verifying the models performance over a set-aside test set, if the random forest performs better, the underlying process may be non-linear.  Alternatively, you could include variable interaction effects and higher order variables created using a, b, and c in the lasso model and verify if this model performs better as compared to a lasso with only a linear combination of a, b and c. If it does, then the underlying process might be non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Decision_tree_learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Random_forest&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.jstatsoft.org/article/view/v017i01&quot; rel=&quot;noreferrer&quot; title=&quot;Relative importance of Linear Regressors in R&quot;&gt;Relative importance of Linear Regressors in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.gtcenter.org/Archive/Conf05/Downloads/Conf/Feldman60.pdf&quot; rel=&quot;noreferrer&quot; title=&quot;Relative Importance and Value&quot;&gt;Relative Importance and Value, Barry Feldman (PMD method)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-06-23T19:54:58.747" LastActivityDate="2016-06-23T19:54:58.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12375" PostTypeId="1" AcceptedAnswerId="12383" CreationDate="2016-06-23T20:38:12.440" Score="3" ViewCount="12235" Body="&lt;p&gt;I have problem with loading data into R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fileUrl &amp;lt;- &quot;http://jadi.net/files/iran_it_status_1394_detail_data_jadi_net.tsv&quot;&#xA;&#xA;download.file(fileUrl , destfile=&quot;iran_it_status_1394_detail_data_jadi_net.tsv&quot;)&#xA;&#xA;dev &amp;lt;- read.delim(&quot;iran_it_status_1394_detail_data_jadi_net.tsv&quot;,&#xA;              header=TRUE,sep=&quot;\t&quot;,blank.lines.skip = TRUE,&#xA;              na.strings=&quot;&quot;,fileEncoding=&quot;UTF-8&quot;,&#xA;              stringsAsFactors=FALSE,skipNul = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I receive the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in read.table(file = file, header = header, sep = sep, quote = quote,  : &#xA;  no lines available in input&#xA;  In addition: Warning message:&#xA;  In read.table(file = file, header = header, sep = sep, quote = quote,  :&#xA;  invalid input found on input connection    'iran_it_status_1394_detail_data_jadi_net.tsv'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: The dataset has 1217 rows and 33 variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(data) &amp;lt;- c(&quot;timestamp&quot;,&quot;age&quot;,&quot;sex&quot;,&quot;birth_province&quot;,&quot;work_province&quot;,&quot;experience&quot;,&quot;education&quot;,&#xA;            &quot;certificate&quot;,&quot;learn&quot;,&quot;project&quot;,&quot;book&quot;,&quot;language&quot;,&quot;wish_language&quot;,&quot;db&quot;,&quot;desktop_os&quot;,&#xA;            &quot;wish_os&quot;,&quot;mobile&quot;,&quot;env&quot;,&quot;theme&quot;,&quot;src_ctrl&quot;,&quot;tab_space&quot;,&quot;drink&quot;,&quot;items&quot;,&quot;device&quot;,&quot;title&quot;,&#xA;            &quot;org_type&quot;,&quot;org_emp&quot;,&quot;income&quot;,&quot;perk&quot;,&quot;job_contract&quot;,&quot;job_type&quot;,&quot;hour_wage&quot;,&quot;happy&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for language variable I expect this output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data[1:3,&quot;language&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;C#, Javascript, R, SQL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Java, C#, Javascript, Objective C, Swift, SQL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C#, SQL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python Solutions are also welcome&lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="3151" LastEditDate="2016-06-30T14:31:20.173" LastActivityDate="2016-06-30T14:31:20.173" Title="problem loading data into R" Tags="|r|dataset|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12376" PostTypeId="2" ParentId="12375" CreationDate="2016-06-23T21:11:25.123" Score="-1" Body="&lt;p&gt;Try using either&lt;code&gt;try&lt;/code&gt; or &lt;code&gt;tryCatch&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See here &lt;code&gt;http://stackoverflow.com/questions/13613270/how-to-fix-the-error-in-r-of-no-lines-available-in-input&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="9079" LastActivityDate="2016-06-23T21:11:25.123" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12377" PostTypeId="1" CreationDate="2016-06-23T22:11:10.443" Score="2" ViewCount="10636" Body="&lt;p&gt;I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way in SPSS to find the correlation &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a)between two categorical variables &lt;br&gt;&#xA;b)between categorical and continuous variables?&lt;/p&gt;&#xA;" OwnerUserId="20813" LastEditorUserId="11097" LastEditDate="2016-06-24T01:01:07.353" LastActivityDate="2019-12-07T09:01:40.157" Title="Checking Correlation of Categorical variables in SPSS" Tags="|correlation|categorical-data|spss|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12378" PostTypeId="2" ParentId="12348" CreationDate="2016-06-23T23:47:08.980" Score="3" Body="&lt;p&gt;Your best bet is to conduct focus groups or conduct market research, and that will probably be very expensive. You are not likely to find anything useful on Twitter, Facebook, or the news. No amount of text mining will solve your problem. This is not the answer that you want to hear, but your project is not set up for success, and you should look for/propose a new project rather than potentially waste more of your valuable time on this project. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-06-23T23:47:08.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12379" PostTypeId="2" ParentId="12377" CreationDate="2016-06-24T01:58:43.427" Score="1" Body="&lt;p&gt;For testing the correlation between categorical variables, you can use:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;binomial test&lt;/strong&gt;: A one sample binomial test allows us to test whether the proportion of successes on a two-level categorical dependent variable significantly differs from a hypothesized value.  For example, using the hsb2 data file, say we wish to test whether the proportion of females (female) differs significantly from 50%, i.e., from .5.  We can do this as shown below.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;chi-square test&lt;/strong&gt;: A chi-square goodness of fit test allows us to test whether the observed proportions for a categorical variable differ from hypothesized proportions.  For example, let's suppose that we believe that the general population consists of 10% Hispanic, 10% Asian, 10% African American and 70% White folks.  We want to test whether the observed proportions from our sample differ significantly from these hypothesized proportions. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can find the code for doing above analysis from the link below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure about comparing numerical and categorical variables but one thing you can do is to transform the numerical variable into categories. For example, if it is age, then, you can transform it to: [0,10), [10,20), ... , [90,100), so that you can compare the correlation of age with other categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="14152" LastActivityDate="2016-06-24T01:58:43.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12380" PostTypeId="2" ParentId="10841" CreationDate="2016-06-24T05:11:39.740" Score="2" Body="&lt;p&gt;&lt;strong&gt;Question 1) Is this a good idea in general?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solving this problem as a supervised learning regression problem is a fantastic idea and is the type of solution that will greatly benefit your company since it will translate to other similar and dissimilar problems much more easily than deterministic methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However using a neural network to solve this supervised learning regression problem is probably a very bad idea. Neural networks can add great value to certain problems, are among the very best algorithms for complex problems where computing power is not an issue, and have fascinating implications for the future of machine learning and artificial intelligence. But... they can be very tricky to train, require a lot of computing power, require a lot of data, and perform poorly in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you scope the problem using linear regression and then try a support vector regressor (SVM, SVR) or naive Bayes regressor. The analogue methods in SVR work very well with limited data and provide surprisingly accurate results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2) If yes, what type of NN would be most suited for such a problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you must use a neural network then try playing with the problem.  Start with a feed forward neural network. Note that it will likely underperform an SVR.  Then think about moving toward a convolution neural network. Again, this is probably a very bad way to go. Try linear regression followed by other methods first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Moving forward&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation for either &lt;a href=&quot;http://scikit-learn.org/stable/index.html&quot; rel=&quot;nofollow&quot;&gt;Scikit-Learn&lt;/a&gt; in python, &lt;a href=&quot;http://www.h2o.ai/product/&quot; rel=&quot;nofollow&quot;&gt;H20&lt;/a&gt; in Java, or &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/weka/&quot; rel=&quot;nofollow&quot;&gt;Weka&lt;/a&gt; provide very shallow learning curves to jump on the merry-go-round and take a spin or two. Please make sure that you thoroughly understand cross validation and scoring metrics as this is essential to continued, adequate progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-06-24T05:11:39.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12381" PostTypeId="2" ParentId="8208" CreationDate="2016-06-24T05:31:44.860" Score="0" Body="&lt;p&gt;So let me clarify your query: you have trained a random forest model to classify a dataset into multiple classes, e.g. A, B, C, or D. Now, you want to understand for each different class label (e.g. A), which features contributed to it being classified as class A vs. not class A, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find the importance of a variable in random forest, each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.&#xA;this method gives importance for the entire multinomial classification, i.e. A vs. B/C/D, B vs. A/C/D, etc. and not just for one label. So you can't claim that feature1 is more important to identify class A, but feature 2 is more important to identify Class B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been using the random forest package implemented in R and it does not seem to provide any way to decipher the internal details on the one-vs-all categorization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're really interested in doing just that, then you should fit a set of random forest models each for classifying class A vs. the rest, class B vs. the rest, etc. The variable importance of these separate models would give you the feature importance for classifying each label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the same library to fit these models using scikit-learn.  You may refer to their help page here - &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-06-24T05:31:44.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12382" PostTypeId="1" AcceptedAnswerId="12434" CreationDate="2016-06-24T05:34:53.880" Score="5" ViewCount="5613" Body="&lt;p&gt;Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastEditorUserId="13727" LastEditDate="2016-06-27T07:30:36.853" LastActivityDate="2016-07-04T10:25:35.940" Title="feature selection techniques" Tags="|feature-selection|mutual-information|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12383" PostTypeId="2" ParentId="12375" CreationDate="2016-06-24T05:36:11.500" Score="2" Body="&lt;p&gt;I am able to load the data set like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; dev &amp;lt;- read.table(&quot;iran_it_status_1394_detail_data_jadi_net.tsv&quot;,&#xA;                   header=TRUE, sep=&quot;\t&quot;, blank.lines.skip = TRUE,&#xA;                   na.strings=&quot;&quot;,&#xA;                   stringsAsFactors=FALSE, skipNul = TRUE, fill=T, quote=&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the removal of the encoding (so that the function &quot;finds&quot; lines in the file), the fill attribute (to allow for a ragged table with empty cells), and the elimination of quotes (apparently there is a misquoted line somewhere near line 585).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This results in a table full of encoded characters - you'll need to know more about the source data to figure out how to work with it, but if you open the file up in a raw text editor (e.g.: Sublime text) you might get some clues:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xLG1L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xLG1L.png&quot; alt=&quot;Character encoding issues?&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14625" LastActivityDate="2016-06-24T05:36:11.500" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12384" PostTypeId="1" CreationDate="2016-06-24T06:22:00.617" Score="5" ViewCount="147" Body="&lt;p&gt;I have a dependency treebank including 100 sentences, which I divide into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I extract such rules from the training set, I can measure the percentage of rules (DS patterns) that cover the test set, suppose&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(10, 24%), (20, 34%), (30,40%), (40,44%), (50, 55%),(60, 58%), (70, 61%)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see as I increase the size of the training set, the coverage of extracted patterns increases! however its not linear!, I want to see how many data I need to reach 100% coverage?&#xA;I guess I can use a regression, but which regression? logarithmic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this related to 'learning curve'? if yes how can I use regression for a learning curve?&lt;/p&gt;&#xA;" OwnerUserId="3436" LastEditorUserId="3436" LastEditDate="2016-06-24T10:36:02.960" LastActivityDate="2018-10-20T14:00:34.967" Title="Estimating data set size for grammar extraction" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12386" PostTypeId="1" AcceptedAnswerId="12410" CreationDate="2016-06-24T07:43:40.490" Score="1" ViewCount="1029" Body="&lt;p&gt;I came across &lt;a href=&quot;http://matlab.izmiran.ru/help/toolbox/nnet/backpr11.html&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; link. In BFGS Quasi-Newton Method, a Hessian matrix is used in weight updation. Is there any resource where I can find how this hessian matrix was obtained along with a clear description of the process, as to why Hessian matrix has been taken? I could not understand the wiki article.&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2016-07-27T08:13:59.883" Title="How to form Hessian matrix in BFGS Quasi-Newton Method" Tags="|machine-learning|neural-network|deep-learning|optimization|supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12387" PostTypeId="2" ParentId="12382" CreationDate="2016-06-24T07:56:14.040" Score="4" Body="&lt;p&gt;The fact that a feature is redundant in the presence of another one, or is not informative enough to describe the target variable, is not necessarily a sign of that feature not being useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, it may be the case that such feature may be extremely informative when combined with another one, in spite of not being very useful when considered in isolation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, when applying feature selection methods, you should also consider combinations of features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, and as pointed out by another answer to this question, finding the best combination of features is a NP-complete problem. Therefore, applying feature selection to individual features may be a good approximation. However, I'd rather apply a greedy approach (see for instance &lt;a href=&quot;https://studentnet.cs.manchester.ac.uk/pgt/COMP61011/goodProjects/Shardlow.pdf&quot; rel=&quot;nofollow&quot;&gt;https://studentnet.cs.manchester.ac.uk/pgt/COMP61011/goodProjects/Shardlow.pdf&lt;/a&gt; for more information about the topic.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT to answer OP's comment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) The table below displays an extreme example of a feature that by itself is very informative, but in combination with others is totally redundant (feature_2). This is a regression problem in which we are trying to build a model to predict the &quot;output&quot; variable from &quot;feature_1&quot; and &quot;feature_2&quot;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| feature_1 | feature_2 | output |&#xA;|-----------|-----------|--------|&#xA;|         1 |         1 |    0.1 |      &#xA;|         2 |         2 |    0.2 |    &#xA;|         3 |         3 |    0.3 |     &#xA;|         4 |         4 |    0.4 |      &#xA;|         5 |         5 |    0.5 |    &#xA;|         6 |         6 |    0.6 |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;b) The example below shows an extreme example of a feature that may not be very informative by itself, but that is very informative together with another one (feature_2). &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| feature_1 | feature_2 | output |&#xA;|-----------|-----------|--------|&#xA;|         1 |         1 |    0.1 |      &#xA;|         2 |         2 |   0.25 |    &#xA;|         3 |         1 |    0.3 |     &#xA;|         4 |         2 |   0.45 |      &#xA;|         5 |         1 |    0.5 |    &#xA;|         6 |         2 |   0.65 |   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="2576" LastEditDate="2016-07-04T10:25:35.940" LastActivityDate="2016-07-04T10:25:35.940" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12388" PostTypeId="1" AcceptedAnswerId="12389" CreationDate="2016-06-24T08:43:11.603" Score="1" ViewCount="3754" Body="&lt;p&gt;&lt;br&gt;&#xA;I am trying to perform a classification using a decision tree classifier. I was wondering whether using a Feature reduction method is relevant for decision trees since they automatically use pruning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea would be to perform a loop from 5 to 15 parameter reduction and then compare the classification accuracy of each decision tree, and then conclude the optimal number of parameters for my classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="20311" LastActivityDate="2016-06-24T09:10:42.907" Title="Pruning and parameter reduction for decision trees" Tags="|python|classification|decision-trees|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12389" PostTypeId="2" ParentId="12388" CreationDate="2016-06-24T09:10:42.907" Score="1" Body="&lt;p&gt;Pruning and feature reduction are different things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pruning:&lt;/strong&gt; It basically compares the &lt;em&gt;purity&lt;/em&gt; of the two leafs separately, and, together. If the leafs together are purer, than the two leafs are pruned. Thus, the decision over the parameter(s) at the node is wiped off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have &lt;em&gt;N&lt;/em&gt; different parameters. You tree might be tall enough such that pruning has been used over all the parameters at different nodes. And in the same time, all these parameters might have been used in other nodes. If not, the decision tree will take the decision itself not to use this parameter - doesn't prevent from overfitting though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dimensionality Reduction:&lt;/strong&gt;&#xA;It you reduce the number of parameters, then these parameters would never appear in your tree, at any node. Whereas they might have been relevant at some point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are not uncompatible, and performing a dimensionality reduction may increase the accuracy of your task for a further classifier (as decision trees). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, decision trees are also used for dimensionality reduction: After being trained, one might scan the features importance within the decision tree, i.e. how much each feature is used to create a split a different nodes. based on this new knowledge, you can used only the most important features to train another classifier.&lt;/p&gt;&#xA;" OwnerUserId="20759" LastActivityDate="2016-06-24T09:10:42.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12390" PostTypeId="2" ParentId="12348" CreationDate="2016-06-24T11:23:23.083" Score="4" Body="&lt;p&gt;The World Bank runs &lt;a href=&quot;http://www.enterprisesurveys.org/&quot; rel=&quot;nofollow&quot;&gt;Enterprise Surveys&lt;/a&gt; that 'provide the world's most comprehensive company-level data in emerging markets and developing economies. Business data are available on 130,000 firms in 135 countries'. You can get access by requesting on the site subject to some confidentiality terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data are fairly comprehensive and would give you a good starting point to form hypotheses as which areas (access to credit, regulation, corruption, etc) matter most that you could then try to test by mining news or social media.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The World Bank also collects data on &lt;a href=&quot;http://data.worldbank.org/data-catalog/doing-business-database&quot; rel=&quot;nofollow&quot;&gt;Doing Business&lt;/a&gt; that, while not specific to small companies, highlight country-specific challenges. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plenty of research has been published around the two programs above. &lt;/p&gt;&#xA;" OwnerUserId="14678" LastActivityDate="2016-06-24T11:23:23.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12391" PostTypeId="1" CreationDate="2016-06-24T11:36:23.127" Score="1" ViewCount="55" Body="&lt;p&gt;I am working on e-commerce, how to rank smartphones in the same category. I want to calculate a weighted average of sentiment score from reviews posted by buyers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weights will be based on how many days before review was posted, as people might give more importance to recent reviews than old reviews. Now to find weight for previous day review, i want to understand impact of previous reviews on current review. i.e. impact of t-1, t-2, t-3 on t.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know for how many days should i aggregate review sentiment?&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2016-07-25T17:49:53.757" Title="impact of old reviews on new reviews" Tags="|feature-engineering|feature-scaling|feature-construction|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12392" PostTypeId="2" ParentId="12377" CreationDate="2016-06-24T12:56:23.933" Score="0" Body="&lt;p&gt;Another approach is to use the SPSSINC HETCOR extension command.  It calculates set of Pearson, polychoric, or polyserial correlations according the measurement levels of the variables.  This extension requires both the Python and R Essentials.&lt;/p&gt;&#xA;" OwnerUserId="20834" LastActivityDate="2016-06-24T12:56:23.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12393" PostTypeId="2" ParentId="12295" CreationDate="2016-06-24T13:11:42.620" Score="0" Body="&lt;p&gt;If I correctly understood your problem: you have a set of routes between the same two start and destination locations, and each route is described as a set of (lat, lon, time) tuples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to compute the distance between two routes, one possibility would be to apply a variant of the edit distance between strings. This algorithm computes the distance between two strings as the amount of string operations (edit, remove, add letters) required to transform the first string into the second one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You just have to come up with a set of &quot;route operations&quot;, and the cost of each of these operations (add point routes, remove point routes, replace one point route by another one, and so on).&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-06-24T13:11:42.620" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12394" PostTypeId="1" AcceptedAnswerId="12429" CreationDate="2016-06-24T14:17:05.060" Score="1" ViewCount="2856" Body="&lt;p&gt;I have a dataset that lists all zip codes in the U.S., their types (standard, po box, university, etc). I want to replace po box and university zip codes with the next closest standard zip code. I broke down the dataset by state so that R wouldn't have to make as many calculations. In theory, I would like to have standard zip codes in the first column and zip codes that need replacement in the first row, and have the distance between the two be the intersection value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                REP 1   REP 2   REP 3   REP 4&#xA;STD 1           0.215   0.152   0.025   0.124   &#xA;STD 2           0.365   0.410   0.074   0.234&#xA;STD 3           0.234   0.201   1.322   0.683&#xA;STD 4           0.543   0.282   0.483   0.094 &#xA;MINS            STD 1   STD 1   STD 2   STD 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where STD 1 is a standard zip code with its own latitude and longitude, and REP 1 is a zip code that needs to be replaced (is a university/po box zip) with its own latitude and longitude. I only have about 5 weeks of experience in R, so please bear with me if something doesn't quite make sense to me immediately. I have tried to do this in excel and having a sheet with close to 10,000 columns by 40,000 rows crashes every time that I try to calculate all of the distances because there are just too many calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a feeling that either the &lt;code&gt;apply()&lt;/code&gt; or &lt;code&gt;mapply()&lt;/code&gt; functions are needed here. I want to calculate the distances using a formula that considers the curvature of the earth, (euclidean, etc) like &lt;code&gt;dist()&lt;/code&gt; or the &lt;code&gt;geosphere&lt;/code&gt; package to maintain accuracy and be reproducible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is anything else that would be helpful to add on here, let me know and I'll upload it asap. Here is my R code for Alaska, the first state in alphabetical order. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AK&amp;lt;-subset(db,STAABBRV.x==&quot;AK&quot;)&#xA;AKPO&amp;lt;-subset(AK,ZipCodeType!=&quot;STANDARD&quot;,select=c(&quot;ZIP_CODE&quot;,&quot;ZipCodeType&quot;,&quot;Long&quot;,&quot;Lat&quot;))&#xA;AKPO&amp;lt;-within(AKPO,{IS_PO=ifelse(ZipCodeType!=&quot;STANDARD&quot;,1,0)})&#xA;AKSTANDARD&amp;lt;-subset(AK,ZipCodeType==&quot;STANDARD&quot;,select=c(&quot;ZIP_CODE&quot;,&quot;ZipCodeType&quot;,&quot;Long&quot;,&quot;Lat&quot;))&#xA;AKSTANDARD&amp;lt;-within(AKSTANDARD,{IS_PO=ifelse(ZipCodeType!=&quot;STANDARD&quot;,1,0)})&#xA;table&amp;lt;-rbind(AKSTANDARD,AKPO)&#xA;table$ZipCodeType&amp;lt;-NULL&#xA;rm(AK,AKPO,AKSTANDARD)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This sets up a table that has column names &quot;ZIP_CODE&quot;, &quot;Long&quot;, &quot;Lat&quot;, and &quot;IS_PO&quot;. &quot;IS_PO&quot; is a numerical indicator for whether or not the zip code is standard or po/university. 1 indicates that the zip code is a po/univ zip and 0 indicates a standard zip. I did this because some functions required that the data in the dataset be the same type (numerical).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some of my failed attempts at writing code to calculate the minimum distances.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lapply(bit::chunk(1, nrow(zipcode), 1e2), function(ridx) {&#xA;  merge(zipcode, zipcode[ridx[1]:ridx[2]], by = &quot;dum&quot;, allow.cartesian = T)[&#xA;    , dist := distGeo(matrix(c(longitude.x, latitude.x), ncol = 2), &#xA;                      matrix(c(longitude.y, latitude.y), ncol = 2))/1609.34 # meters to miles&#xA;    ][dist &amp;lt;= 5 # necessary distance treshold&#xA;      ][, dum := NULL]&#xA;}) %&amp;gt;% rbindlist -&amp;gt; zip_nearby_dt&#xA;&#xA;DOESITWORK&amp;lt;-apply(db, 1, function(x) spDistsN1(matrix(x[3:4], nrow=1),&#xA;                                    x[5:6],&#xA;                                    longlat=TRUE)) &#xA;&#xA;&#xA;&#xA;mins&amp;lt;-apply(Lat,1,function(x)return(array(which.min(x))))&#xA;mins&amp;lt;-data.frame(row=names(mins),col=mins) &#xA;Lat$mins&amp;lt;-apply(mins,1,FUN=function(x)return(paste(x[&quot;row&quot;],colnames(Lat[as.numeric(x[&quot;col&quot;])]),Lat[x[&quot;row&quot;],as.numeric(x[&quot;col&quot;])],sep=&quot;/&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20835" LastEditorUserId="20835" LastEditDate="2016-06-24T15:12:00.277" LastActivityDate="2016-06-27T00:42:21.860" Title="How do I find the minimum distance between zip codes in R?" Tags="|r|distance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12395" PostTypeId="2" ParentId="10816" CreationDate="2016-06-24T14:26:48.653" Score="0" Body="&lt;p&gt;Suppose your date column is named DateID, you want to select all of the columns in the table and you're using a table named in MyTable in the default dbo schema. You can use the following T-SQL query to do that, assuming you're on SQL Server. I added the Year and Month functions to the Select so you can inspect them and an optional ORDER BY clause, so the records will be sorted by the Year and Month. Just substitute the names of your schema, table and date column (you may need to tweak the code a little if you're using Oracle or MySQL or whatever). I hope that helps.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    SELECT Year(DateID), Month(DateID), * &#xA;    FROM dbo.MyTable&#xA;    WHERE Year(DateID) = '2015' OR (Year(DateID) = '2014' AND  Month(DateID) BETWEEN 5 and 12)&#xA;    ORDER BY  Year(DateID), Month(DateID)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18544" LastActivityDate="2016-06-24T14:26:48.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12396" PostTypeId="1" CreationDate="2016-06-24T14:43:44.603" Score="2" ViewCount="394" Body="&lt;p&gt;I am working on collaborative filtering using matrix factorization in MATLAB. I am using Gradient Descent for parameter learning. The cost function to optimize is :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ J = {\left\| I \odot (R - U V') \right\|}_{F}^{2} + \lambda_1 {\left\| U \right\|}_{F}^{2} + \lambda_2 {\left\| V \right\|}_{F}^{2} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following equations for updates :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \nabla J_{u(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') v_j $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \nabla J_{v(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') u_i $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dimensions of the matrices are :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$I  -&amp;gt; n * n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R -&amp;gt; n*n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$U -&amp;gt; n*k$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V -&amp;gt; n*k$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to come up with vectorized form of the gradient, so I have to loop over i's and j's explicitly, which is slowing down the code a lot. &lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="11097" LastEditDate="2017-09-13T11:21:14.563" LastActivityDate="2019-12-03T13:02:20.203" Title="Vectorizing equation in MATLAB" Tags="|recommender-system|gradient-descent|matlab|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12397" PostTypeId="2" ParentId="12396" CreationDate="2016-06-24T15:00:40.980" Score="1" Body="&lt;p&gt;Is your question how to vectorize the equations for updating?  You have a vector of i and j, and for each update you are summing over j.  Calculate the vectors of each of the components and then sum the vector over j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Providing actual code might help your question.  As is, it seems vague.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-06-24T15:00:40.980" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12398" PostTypeId="2" ParentId="12188" CreationDate="2016-06-24T16:00:37.280" Score="1" Body="&lt;p&gt;You have many options of algorithms to use for classification of unsupervised data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a very broad topic, but if you need a specific algo recommendation, try to see if &lt;a href=&quot;https://en.wikipedia.org/wiki/Self-organizing_map&quot; rel=&quot;nofollow&quot;&gt;self-organizing maps (SOM)&lt;/a&gt; may help with your specific problem. In R, try the &lt;code&gt;kohonen&lt;/code&gt; package. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-means is another popular clustering algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter which method you use, consider converting your categorical data to numerical data for clustering, as it may alleviate some of your mixed data-type woes. &lt;/p&gt;&#xA;" OwnerUserId="13974" LastActivityDate="2016-06-24T16:00:37.280" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12399" PostTypeId="2" ParentId="12188" CreationDate="2016-06-24T16:36:51.237" Score="1" Body="&lt;p&gt;It's really a broad topic but I think you are going on a right track. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I solved a similar problem couple of months back where I worked on classification of documents in multiple categories using Centroid based algorithm.&#xA;Here, I clustered training dataset using Spherical K-Means, and the resulting centroids of the clusters represented a category. Later while predicting a category of new document, I would compare the document with all the centroids and assign a category based of SSE.&lt;/p&gt;&#xA;" OwnerUserId="20722" LastActivityDate="2016-06-24T16:36:51.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12400" PostTypeId="2" ParentId="12382" CreationDate="2016-06-24T17:19:37.110" Score="4" Body="&lt;p&gt;As with many things, it depends. The specifics of the relationship of your variables to the domain they describe will dictate, and even then relationships may not be intuitive. Seemingly disparate features can have a significant effect when combined in what is known as feature extraction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Automated feature engineering techniques can help you decide what features are significant if you have the time and resources available, particularly when it comes to testing the impact of combined features.  Additionally, some methods have the benefit of embedded feature selection, wherein the algorithm itself tends to diminish the effect of insignificant variables, e.g.: lasso regression, regularized decision trees, random forests, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a nice primer: &lt;a href=&quot;http://machinelearningmastery.com/an-introduction-to-feature-selection/&quot; rel=&quot;nofollow&quot;&gt;http://machinelearningmastery.com/an-introduction-to-feature-selection/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14625" LastActivityDate="2016-06-24T17:19:37.110" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12401" PostTypeId="2" ParentId="12348" CreationDate="2016-06-24T20:47:07.597" Score="0" Body="&lt;p&gt;I would also look at the academic articles, if they have done such analysis before or they have related data. If the topic may not be related but you can find some related data and get in touch with authors. &lt;/p&gt;&#xA;" OwnerUserId="20841" LastActivityDate="2016-06-24T20:47:07.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12402" PostTypeId="1" CreationDate="2016-06-25T06:53:00.070" Score="6" ViewCount="3362" Body="&lt;p&gt;Why is mutual information symmetric, meaning why does I(A,B) = I(B,A)? Isnt the definition of mutual information, I(A,B), something like &quot;the reduction of entropy in A when given B&quot;? P(A|B) doesnt equal P(B|A) right?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastActivityDate="2020-11-25T23:22:26.547" Title="Is mutual information symmetric?" Tags="|dimensionality-reduction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12403" PostTypeId="1" AcceptedAnswerId="12493" CreationDate="2016-06-25T08:46:21.680" Score="3" ViewCount="2287" Body="&lt;p&gt;I'm using &lt;code&gt;xgboost&lt;/code&gt; for training a model on a data with extreme class imbalance. After referring from &lt;a href=&quot;https://stats.stackexchange.com/a/181615/119631&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After performing grid search and some manual settings, I found that the following parameters work the best for me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weight &amp;lt;- as.numeric(labels) * nrow(test) / length(labels)&#xA;upscale &amp;lt;- sum(weight * (labels == 1.0))&#xA;&#xA;xgb_params = list(&#xA;  objective = 'binary:logistic',                                               &#xA;  eta = 0.1,                                                                  &#xA;  max_depth = 4,                                                               &#xA;  eval_metric = 'auc',                                                         &#xA;  max_delta_step = 10,&#xA;  scale_pos_weight = upscale&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can the process of setting optimal hyperparameters for &lt;code&gt;xgboost&lt;/code&gt; be automated for best AUC? Please note that some of these parameters aren't supported by the &lt;code&gt;caret&lt;/code&gt; implementation of &lt;code&gt;xgboost&lt;/code&gt; but are very important for the model I have to design.&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-04T19:35:12.460" Title="How can the process of hypertuning of XGBoost parameters be automated?" Tags="|r|optimization|xgboost|hyperparameter|weighted-data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12404" PostTypeId="2" ParentId="12402" CreationDate="2016-06-25T12:52:36.507" Score="5" Body="&lt;p&gt;Given the definition for mutual information&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I(X;Y) = \sum_{y \in Y} \sum_{x \in X}&#xA;  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}&#xA;  \right) },$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it follows from rearrangement of the summands&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I(Y; X) =&#xA;  \sum_{x \in X} \sum_{y \in Y}&#xA;  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}&#xA;  \right) }.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence $I(X; Y) = I(Y; X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;Edit&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;$I(X;Y)$ measures the average reduction in uncertainty of $X$ that results from knowing $Y$&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you interpret $H(X) - H(X|Y)$ where $H(X)$ is the marginal entropy of $X$ and $H(X|Y)$ the conditional entropy of $X$ given $Y$ as the reduction in uncertainty, then it is equal to $H(Y) - H(Y|X)$ and $I(X; Y)$ (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information#Relation_to_other_quantities&quot; rel=&quot;nofollow&quot;&gt;Wikipedia&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="16213" LastEditorUserId="16213" LastEditDate="2016-06-25T21:15:09.300" LastActivityDate="2016-06-25T21:15:09.300" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="12406" PostTypeId="2" ParentId="12391" CreationDate="2016-06-25T17:47:25.070" Score="1" Body="&lt;p&gt;Welcome to DataScience.SE. Let's see if I've interpreted your question correctly:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;i want to understand impact of previous reviews on current review&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Keep track of which reviews a reviewer has read (randomly withhold some if you have to), and use this information together the rating of the new review to create a regressor to estimate the score of a new review given the age and score of selected previous reviews.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do similar regressions with the &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow&quot;&gt;topic&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;nofollow&quot;&gt;sentiment&lt;/a&gt;, if you use &lt;a href=&quot;https://en.wikipedia.org/wiki/Word_embedding&quot; rel=&quot;nofollow&quot;&gt;embeddings&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-06-25T17:47:25.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12409" PostTypeId="1" CreationDate="2016-06-25T21:52:13.020" Score="2" ViewCount="115" Body="&lt;p&gt;I have a dataset where I have a large number of addresses. The problem lies in the fact that many addresses although they are same but haven't been noted down in the same manner. So I want to find these addresses which are similar and clean my data from the duplicate data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard approach or algorithms that can help me? How should I go about this problem? &lt;/p&gt;&#xA;" OwnerUserId="20864" LastActivityDate="2016-07-29T22:07:27.710" Title="How to perform Cleaning of a very large set of addresses" Tags="|bigdata|data-cleaning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12410" PostTypeId="2" ParentId="12386" CreationDate="2016-06-25T22:40:39.043" Score="0" Body="&lt;p&gt;In optimization, you are looking for minima and maxima of a certain function &lt;strong&gt;f&lt;/strong&gt;. Equivalently, under certain derivative conditions, you are looking for the zeros of &lt;strong&gt;g&lt;/strong&gt;, the derivative of &lt;strong&gt;f&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The first equation of your link describes one step of the Newton Method - not BFGS. It involves the Hessian of &lt;strong&gt;f&lt;/strong&gt; which is not accessible in practice - it's ineficcient in terms of time and memory storage. Due to this fact, some have developed methods not to calculate the hessian directly.&lt;/p&gt;&#xA;&lt;p&gt;Thanks to Taylor expansion, you can write&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;g&lt;/strong&gt;(x) = &lt;strong&gt;g&lt;/strong&gt;(y) + &lt;strong&gt;g'&lt;/strong&gt;(y)(x-y) + &lt;strong&gt;Θ&lt;/strong&gt;(|x-y|²)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Replacing x by x(k+1) and y by x(k), and &lt;strong&gt;g'&lt;/strong&gt; by the Hessian of &lt;strong&gt;f&lt;/strong&gt; gives you (with the following convention : &lt;strong&gt;g&lt;/strong&gt;(k+1) = &lt;strong&gt;g&lt;/strong&gt;( x(k+1) )&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;g&lt;/strong&gt;(k+1) ≈ &lt;strong&gt;g&lt;/strong&gt;(k) + &lt;strong&gt;∇²f&lt;/strong&gt;(x(k+1)) ( x(k+1) - x(k) )&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;H&lt;/strong&gt;(k+1) (x(k+1) - x(k)) ≈ &lt;strong&gt;g&lt;/strong&gt;(k+1) - &lt;strong&gt;g&lt;/strong&gt;(k)&#xA;where &lt;strong&gt;H&lt;/strong&gt;(k) aims to approximate ∇²&lt;strong&gt;f&lt;/strong&gt;(x(k)). But instead of calculating it at each step, we'll try to &lt;em&gt;approximate&lt;/em&gt; it thanks to the previous &lt;strong&gt;H&lt;/strong&gt;(k-1)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In 1970, Broyden, Fletcher, Goldfard and Shamo proposed a rank 2 update of &lt;strong&gt;H&lt;/strong&gt;(k), ie&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;H&lt;/strong&gt;(k+1) = &lt;strong&gt;H&lt;/strong&gt;(k) + Term1 + Term2&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;where Term1 does not depend on &lt;strong&gt;H&lt;/strong&gt; and Term2 depends on &lt;strong&gt;H(k)&lt;/strong&gt;. You can find these terms &lt;a href=&quot;https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But this solves only the time issue, not the memory storage. As for that, some have developed &lt;a href=&quot;https://en.wikipedia.org/wiki/Limited-memory_BFGS&quot; rel=&quot;nofollow noreferrer&quot;&gt;Limited-memory BFGS&lt;/a&gt; (or L-BFGS) to limit the amount of memory used.&lt;/p&gt;&#xA;" OwnerUserId="20759" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-06-27T07:53:58.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12411" PostTypeId="2" ParentId="11655" CreationDate="2016-06-26T00:11:54.507" Score="3" Body="&lt;p&gt;The toolbox only manage the sampling so this is slightly different from the algorithm from the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What it does is the following: it creates several subset of data which are balanced. These subsets are created by randomly under-sampling the majority class. That is what you are getting from the toolbox.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To obtain what in the paper, you need to train an AdaBoost classifier for each subset. Thus, what you get is an ensembles of ensembles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that's help.&lt;/p&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2016-06-26T00:11:54.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12412" PostTypeId="2" ParentId="11204" CreationDate="2016-06-26T06:59:20.397" Score="1" Body="&lt;p&gt;From your question I'm interpreting that you've extracted data from a log which has TCP session related data, some of the fields you've mentioned are: source/dest IP address, TTL, start time, end time.&#xA;You're trying to segregate this network traffic to isolate the traffic belonging to 10 unique client machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several ways to approach this client traffic segregation problem. But first you need to identify which variables are going to be most meaningful under different circumstances, since the network setup is not described, I'll list the different possible scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Clients reaching the server without intervening firewall/proxy: In this case each client will have a unique source ip-address which should help segregate the traffic without further analysis.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clients behind a proxy which filters traffic and changes the ip-address to set a unique ip-address of the gateway for all the traffic coming to the server. In this case, you could use a combination of the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Segregate clients by TCP/IP packet attributes: Attributes such as TTL, SYN packet size, TCP window size, etc, can identify client OS. For example - Linux and Mac-OSX have default starting TTL values of 64, whereas windows has a default of 128, others like Cisco/Solaris 2.x have 255. So you can roughly segregate the traffic by OS type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Since the TTL is decremented by 1 for each router placed between the client and the server, so even if you have 2 windows machines, but placed in a network segment separated by one more router, then the TTL number will be one lesser than the other windows clients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c) Further identify unique sessions by the sequence number and acknowledgement number in the TCP headers. Each acknowledgement number of the destination server can be correlated by the seq number of subsequent packets from the client for that session. This will let you segregate users connecting to the same server at the same time e.g. 10 different sessions to stackexchange.com.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d) In case you can get further web-server log details, then attributes such as user-agent strings and HTTP cookies can help you identify a client.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e) In some cases, the random source port no. used by clients for the TCP session might overlap for different clients accessing different dest. ip-addresses at the same time, you can then safely assume these are traffic form two different clients.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This still leaves you with the problem of segregating traffic for different servers at different points of time. For this you could analyze the data in the following manner:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Identify a client using their ISN: There is a certain logic in generating random initial sequence numbers by each OS, and if you have a large sample of ISN nos available, then you could identify the attributes of the random number generator to pinpoint the client. This ISN generation process has been exploited for DoS spoofing attacks, and many OS's have strengthened it since then, but this may still provide some clues to identify machines uniquely.&lt;/li&gt;&#xA;&lt;li&gt;Analyse SSL handshakes to identify clients via their attributes such as supported cipher suites information exchanged during the SSL setup handshake, etc. See &lt;a href=&quot;http://link.springer.com/article/10.1186/s13635-016-0030-7&quot; rel=&quot;nofollow&quot; title=&quot;this paper&quot;&gt;this paper&lt;/a&gt; for details about this technique.&lt;/li&gt;&#xA;&lt;li&gt;The source port nos. are usually assigned in increasing order, and it is unlikely that every machine started at the same time and had the same TCP traffic volume; this helps identify clients uniquely.&lt;/li&gt;&#xA;&lt;li&gt;TCP/IP timestamps: Most OS's will increment the timestamps in TCP headers and this keeps increasing as long as the system is up. Since systems may be switched on at different times, it helps segregate clients even if they have exactly the same OS and browser versions. See &lt;a href=&quot;http://www.ietf.org/rfc/rfc1323.txt&quot; rel=&quot;nofollow&quot; title=&quot;RFC 1323&quot;&gt;this RFC&lt;/a&gt; for mode details.&lt;/li&gt;&#xA;&lt;li&gt;Some proxies my use extra HTTP headers (Via, X-Forwarded-For) to indicate the exact client hostname or address requesting the data. This provides a  straightforward way to identify the client, but &lt;strong&gt;&lt;em&gt;only if&lt;/em&gt;&lt;/strong&gt; you can access the HTTP header information from the packet.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After segregating traffic using all these means, then you may use network traffic time-series analysis to identify distinct users from the periodicity in their usage. For example, the repetitive pattern of a person who logs into work everyday at 8:15 AM and checks on some specific websites for 5 minutes can be identified and that client uniquely segregated from the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More hypothesis and directions to take can only be identified after exploring the dataset in detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you're satisfied with this preparatory work, you could then calculate and add all these features along with the other direct indicators to train a multinomial classifier for predicting these 10 user ids. You could use a random forest, LogitBoost or Neural Networks as starters. If a variable is not identified as important for the classification, then discard it to reduce the total number of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For new traffic, you would need to generate all these features and then use the final trained model to classify the traffic by user.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-06-26T07:22:42.967" LastActivityDate="2016-06-26T07:22:42.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12413" PostTypeId="1" AcceptedAnswerId="12414" CreationDate="2016-06-26T08:05:24.197" Score="8" ViewCount="2425" Body="&lt;p&gt;I am having a lot of trouble understanding this. Does it mean you should not use the cost function very often?&lt;/p&gt;&#xA;" OwnerUserId="20873" LastActivityDate="2019-04-04T15:59:24.090" Title="What does it mean when people say a cost function is something you want to minimize?" Tags="|machine-learning|beginner|linear-regression|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12414" PostTypeId="2" ParentId="12413" CreationDate="2016-06-26T08:59:42.380" Score="12" Body="&lt;p&gt;No, it means you are trying to find the inputs that make the output of the cost function the smallest. It doesn't mean that you should &quot;minimize&quot; use of it.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="21" LastEditDate="2019-04-04T15:59:24.090" LastActivityDate="2019-04-04T15:59:24.090" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12415" PostTypeId="2" ParentId="11893" CreationDate="2016-06-26T10:21:10.707" Score="0" Body="&lt;p&gt;As a first step, to segregate the messages that appear to be a bot, you could first try binning by message size. For example, if messages sent by bots are likely to be around 128 bytes to 140 bytes, assign these to a unique bin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, create a time series based on this bin. Try to decompose the time series using an additive or multiplicative method such as Holt Winters. A strong seasonal component would help you identify regular and repetitive messages which are being generated automatically.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-06-26T10:21:10.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12416" PostTypeId="2" ParentId="12413" CreationDate="2016-06-26T11:01:35.403" Score="1" Body="&lt;p&gt;Cost functions in the context of Machine Learning often calculate some kind of metric that signifies how well your model is performing. A common one is for example the mean squared error, where you look at all your test examples where you know the true value and the predicted, take the difference between that and square it. By minimizing this error (cost function) you assume your predictions will be better.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-06-26T11:01:35.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12418" PostTypeId="1" CreationDate="2016-06-26T12:38:51.567" Score="2" ViewCount="211" Body="&lt;p&gt;I am new in datamining. I decided to play with k-means clustering in r with simple аrtificial data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(101)    &#xA;x1 &amp;lt;- runif(100,36.6,37.5)&#xA;x2 &amp;lt;- runif(100,37.6,38.4)&#xA;x3 &amp;lt;- runif(100,38.5,40)&#xA;x&amp;lt;-c(50,150,250)&#xA;y&amp;lt;-c(37,38,39.25)&#xA;&#xA;centers&amp;lt;-c(37.05,38,39.25)&#xA;all &amp;lt;- c(x1,x2,x3)&#xA;c1&amp;lt;-kmeans(all,centers=centers,iter.max = 1000,nstart=1,algorithm=&quot;Lloyd&quot;)&#xA;plot(all,col = c1$cluster)&#xA;points(x,y,col=&quot;red&quot;,pch=19)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I generated three random data sets &lt;code&gt;x1,x2,x3&lt;/code&gt;. Then i taked center of each cluster like mean of each set and used k-means algorith of r. In result i get what you see in picture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i am making wrong? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why second cluster contains the part of third set? How to improve result? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why result is so bad instead sets is lineary separate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main question is how to improve clustreing result?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/B4gNa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B4gNa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20880" LastActivityDate="2017-04-08T03:27:18.093" Title="k-means clustering unclear result" Tags="|r|k-means|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12419" PostTypeId="2" ParentId="12413" CreationDate="2016-06-26T14:27:16.477" Score="3" Body="&lt;p&gt;A cost function is something you use to penalize high deviations from the expected results when compared to your actual predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of a cost function as a sign of how bad your prediction was. A high cost function value means the prediction was really off, hence, the focus on minimizing the cost function, thereby producing an accurate prediction model.&lt;/p&gt;&#xA;" OwnerDisplayName="user20882" LastActivityDate="2016-06-26T14:27:16.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12420" PostTypeId="2" ParentId="12418" CreationDate="2016-06-26T16:03:19.823" Score="3" Body="&lt;p&gt;A 1 dimensional data set is &quot;linearly separable&quot; at any set of points between the data points. Kmeans isn't a nearest-neighbour type of clustering algorithm, it just divides the space so that each point belongs to its nearest cluster centre, so your data could have huge gaps between the source clusters, but the algorithm would be &quot;wrong&quot; if the clusters had different ranges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you generate points from two clusters on the intervals (0,1) and (2,20), even though there's a gap between 1 and 2, the clustering algorithm will still assign a lot of the points from the higher source cluster to the lower cluster because they are nearer that cluster centre. Its distance to cluster centre for each points that counts, not distance to neighbours. Use another clustering algorithm (like nearest-neighbours) if you are expecting gaps between clusters.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-26T16:03:19.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12422" PostTypeId="1" CreationDate="2016-06-26T17:31:13.530" Score="4" ViewCount="260" Body="&lt;p&gt;I'm trying fit my data , but I couldn't fit it.Data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.50,0&#xA;0.75,0&#xA;1.00,0&#xA;1.25,0&#xA;1.50,0&#xA;1.75,0&#xA;1.75,1&#xA;2.00,0&#xA;2.25,1&#xA;2.50,0&#xA;2.75,1&#xA;3.00,0&#xA;3.25,1&#xA;3.50,0&#xA;4.00,1&#xA;4.25,1&#xA;4.50,1&#xA;4.75,1&#xA;5.00,1&#xA;5.50,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and my code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = np.loadtxt('dat', delimiter=',',dtype=None);&#xA;&#xA;x=data[:,0:1];&#xA;y=data[:,1].reshape(x.size/x[0].size,1);&#xA;a=np.ones(shape=(y.size,x[0].size+1));&#xA;a[:,1:2]=x;&#xA;q=np.ones(shape=(a.shape[1],1));&#xA;alpha=0.003&#xA;&#xA;for i in range(500000):&#xA;    h=1/(1+np.exp(-np.dot(a,q)))&#xA;    for j in range(q.size):&#xA;        q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);&#xA;plt.axis((-1,10,-1,5))&#xA;plt.plot(x,y,'x',x,h);&#xA;plt.show();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I tried different learning rates(alpha),tried different number of iterations but my fitting data is looking like this &lt;a href=&quot;https://i.stack.imgur.com/5ILNe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5ILNe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;but it's should looks like this &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression#/media/File:Exam_pass_logistic_curve.jpeg&quot; rel=&quot;nofollow noreferrer&quot;&gt;enter link description here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I missing? Is there any logical error or something like that? Thanks for your deal.&lt;/p&gt;&#xA;" OwnerUserId="20887" LastEditorUserId="14" LastEditDate="2016-10-15T07:46:06.973" LastActivityDate="2016-10-15T07:46:06.973" Title="Logistic Regression can't fit my data" Tags="|machine-learning|python|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12423" PostTypeId="1" CreationDate="2016-06-26T18:20:43.357" Score="3" ViewCount="1104" Body="&lt;p&gt;Inspired by Google's recent AlphaGo project, I've decided that as a fun personal challenge I'd like to use deep learning and convoluted neural networks to build an algorithm that can beat an ordinary chess program when the difficulty is set to expert. I'm awful at chess and generally can't beat a computer beyond easy/medium, so making something that's a lot smarter than I am (at chess, at least) sounds fascinating. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&#xA;&lt;em&gt;What should the target variable look like for predicting the next best chess move?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been building predictive models for a long time, but I can't wrap my head around this question. Some things that have me confused:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I believe the target variable for Go is simply the entire grid of spaces on the board. In chess there are a limited number of legal moves, which continuously diminish as pieces are removed from the board throughout the game. How do I represent a shrinking target space?&lt;/li&gt;&#xA;&lt;li&gt;Not all pieces have the same set of rules (ex: queen vs knight). Do I need to somehow explicitly code these rules or will deep learning automatically learn them?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I know that alpha-beta pruning and quasi-brute-force search&#xA;techniques have been shown to do well in chess, however, I&#xA;specifically want to use deep learning instead.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;AlphaGo started with supervised learning (SL) on documented expert human moves to make a policy network, among other things. In my case, I'm just trying to do the supervised learning part, and when I do that, the target variable is difficult for me to define. That's the root of my question. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-06-26T19:33:28.190" LastActivityDate="2016-07-28T19:29:14.420" Title="How to represent target variable for chess AI" Tags="|machine-learning|deep-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12429" PostTypeId="2" ParentId="12394" CreationDate="2016-06-26T19:50:57.450" Score="2" Body="&lt;p&gt;I think I have read your question correctly it looks like you need a nearest neighbor implementation. If you are unfamiliar with the concept you can find the wiki article here &lt;a href=&quot;https://en.wikipedia.org/wiki/Nearest_neighbor_search&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Nearest_neighbor_search&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I went ahead a wrote an example implementation you can use as a guide. Please note that this is a brute force method and not useful for big data sets. Once you have a grasp of the material I suggest checking out some libraries like RANN that have &quot;real&quot; implementations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read in some random test data and clean&#xA;   For this test let us assume that we want to find the &#xA;   closest AMERICAN city for each location&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coord_data = read.csv(&quot;~/Downloads/SalesJan2009.csv&quot;, stringsAsFactors = F)&#xA;coord_data$id = c(1:nrow(coord_data))&#xA;coord_data$is_usa = ifelse(coord_data$Country == &quot;United States&quot;, 1, 0)&#xA;coord_data = coord_data[ , c(&quot;id&quot;, &quot;Latitude&quot;, &quot;Longitude&quot;, &quot;is_usa&quot;)]&#xA;names(coord_data) = tolower(names(coord_data))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Define your distance function.&#xA;         Here we have geo-coordinates over long distance so Euclidean will not do.&#xA;         I am using the Law of Cosines to calculate great circle distance&#xA;        but Haversine and Vincenty should be considered given your needs.&#xA;        To learn more start here: &lt;a href=&quot;https://en.wikipedia.org/wiki/Great-circle_distance&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Great-circle_distance&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;greatCircleDistance = function(latAlpha, longAlpha, latBeta, longBeta, radius = 6371) {&#xA;  ## Function taken directly from Wikipedia&#xA;  ## Earth radius in km is default (6371)&#xA;  ## Long/Lats are in degrees so need helper function to convert to radians&#xA;  degreeToRadian = function(degree) (degree * pi / 180)&#xA;  deltaLong = degreeToRadian(longBeta) - degreeToRadian(longAlpha)&#xA;  sinLat = sin(degreeToRadian(latAlpha)) * sin(degreeToRadian(latBeta))&#xA;  cosLat = cos(degreeToRadian(latAlpha)) * cos(degreeToRadian(latBeta))&#xA;  ## acos is finicky with precision so we will assume if NA is thrown&#xA;  ## the argument was very close to 1 and therefore will return 0&#xA;  ## acos(1) == 0&#xA;  acosRaw = suppressWarnings(acos(sinLat + cosLat * cos(deltaLong)))&#xA;  acosSafe = ifelse(is.na(acosRaw), 0, acosRaw)&#xA;  acosSafe * radius&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Distance between Basildon, UK and Parkville, US&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;greatCircleDistance(coord_data$latitude[1],&#xA;coord_data$longitude[1],&#xA;coord_data$latitude[2],&#xA;coord_data$longitude[2])&#xA;&#xA; Returns [1] [1] 6929.351 km.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It matches Google's calc so we are good to go!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Brute Force Example:&#xA;     As you noticed in your Excel sheet this will blow up quickly as data set gets larger. There are much more efficient ways of implementing the search. One idea is to start with the the geo-data structure itself and write an R-Tree, but I'll leave that for you.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; bruteForceNearestNeighbor = function(geoData) {&#xA;      makeCoordinate = function(idx) {&#xA;        c(&quot;id&quot; = idx, &quot;latitude&quot; = geoData$latitude[idx], &quot;longitude&quot; = geoData$longitude[idx])&#xA;      }&#xA;      singleCoordMinDistance = function(coordinate, locations) {&#xA;        locationsUS = locations[locations$is_us == 1 &amp;amp; locations$id != coordinate[&quot;id&quot;], ]&#xA;        distances = mapply(greatCircleDistance,&#xA;              latAlpha = coordinate[&quot;latitude&quot;],&#xA;              longAlpha = coordinate[&quot;longitude&quot;],&#xA;              latBeta = locationsUS$latitude,&#xA;              longBeta = locationsUS$longitude)&#xA;        closestIndex = which(distances == min(distances))&#xA;        locations[closestIndex, &quot;id&quot;]&#xA;      }&#xA;      nearestNeighbors = vector(&quot;numeric&quot;, nrow(geoData))&#xA;      for ( i in 1:nrow(geoData) ) {&#xA;        coord = makeCoordinate(i)&#xA;        nearestNeighbors[i] = singleCoordMinDistance(coord, geoData)&#xA;      }&#xA;      nearestNeighbors&#xA;    }&#xA;&#xA;    coord_data$nearest_neighbor = bruteForceNearestNeighbor(coord_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14469" LastEditorUserId="14469" LastEditDate="2016-06-27T00:42:21.860" LastActivityDate="2016-06-27T00:42:21.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12430" PostTypeId="2" ParentId="12176" CreationDate="2016-06-26T20:06:44.097" Score="1" Body="&lt;p&gt;Take a look at the h20 package &lt;a href=&quot;https://cran.r-project.org/web/packages/h2o/h2o.pdf&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/h2o/h2o.pdf&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything is designed with parallelization in mind. I've had great success with many of their implementations, in R and Scala. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to do it in R and are going for pure speed I doubt you'll find something faster. &lt;/p&gt;&#xA;" OwnerUserId="14469" LastActivityDate="2016-06-26T20:06:44.097" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12433" PostTypeId="2" ParentId="12422" CreationDate="2016-06-26T23:06:05.187" Score="0" Body="&lt;p&gt;I don't write a lot of Python code but it appears you are hard coding alpha, which would explain your results. I suggest reading up on the math and intuition behind logistic regression, along with how model parameters are estimated in general.(&lt;a href=&quot;https://en.wikipedia.org/wiki/Maximum_likelihood_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Maximum_likelihood_estimation&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For guidance on implementation this article does a good job of going through the steps and has the benefit of being in Python: &lt;a href=&quot;http://aimotion.blogspot.com/2011/11/machine-learning-with-python-logistic.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://aimotion.blogspot.com/2011/11/machine-learning-with-python-logistic.html&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I highly suggest you learn how to write it from scratch first, you may want to check out R which has a very easy to use base implementation of logistic regression. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = read.csv(&quot;~/data.csv&quot;, header = T, stringsAsFactors = F)&#xA;glm_train = glm(classifier ~ value, data = df, family = &quot;binomial&quot;)&#xA;&#xA;library(ggplot2)&#xA;binomial_smooth = function(...) {&#xA;  geom_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;), ...)&#xA;}&#xA;ggplot(df, aes(x = value, y = classifier)) + geom_point() + binomial_smooth()&#xA;&#xA;summary(glm_train)&#xA;# Deviance Residuals: &#xA;#   Min        1Q    Median        3Q       Max  &#xA;# -1.70557  -0.57357  -0.04654   0.45470   1.82008  &#xA;# &#xA;# Coefficients:&#xA;#   Estimate Std. Error z value Pr(&amp;gt;|z|)  &#xA;# (Intercept)  -4.0777     1.7610  -2.316   0.0206 *&#xA;#   x             1.5046     0.6287   2.393   0.0167 *&#xA;#   ---&#xA;#   Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0i1YB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0i1YB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14469" LastActivityDate="2016-06-26T23:06:05.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12434" PostTypeId="2" ParentId="12382" CreationDate="2016-06-27T05:42:35.957" Score="6" Body="&lt;p&gt;Doing that is a very good idea. The problem is that doing that is very hard.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;noreferrer&quot;&gt;Feature selection&lt;/a&gt; is a &lt;a href=&quot;https://en.wikipedia.org/wiki/NP-completeness&quot; rel=&quot;noreferrer&quot;&gt;NP-complete&lt;/a&gt; problem.&#xA;The practical meaning as that we don't know any fast algorithm that can select only the needed feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the other direction, omitting features that don't have mutual information (MI) with the concept might cause you to throw the features you need most.&#xA;There are cases in which a single feature is useless but given more features it becomes important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, omitting features with low MI is OK. Many learning algorithms are using MI so they won't be able to use the omitted variables anyway.&#xA;As for the selection itself, there are many algorithms, usually heuristics or approximation algorithm that are quite handy.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-06-27T05:42:35.957" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12435" PostTypeId="2" ParentId="12422" CreationDate="2016-06-27T07:38:25.347" Score="4" Body="&lt;p&gt;The problem is in the following line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;(h-y)&lt;/code&gt; has shape &lt;code&gt;(20, 1)&lt;/code&gt;, &lt;code&gt;a[:,j]&lt;/code&gt; has shape &lt;code&gt;(20)&lt;/code&gt;, multiplying them results in a shape &lt;code&gt;(20, 20)&lt;/code&gt;, which is wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try &lt;code&gt;a[:,j:j+1]&lt;/code&gt; instead of &lt;code&gt;a[:,j]&lt;/code&gt; and it'll start working:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j:j+1]);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives the following plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uUXZR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uUXZR.png&quot; alt=&quot;Result&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could adjust the shapes of your variables a bit to make it easier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x=data[:,:1]&#xA;y=data[:,1]&#xA;a=np.hstack([np.ones((len(x), 1)), x])&#xA;q=np.ones(a.shape[1])&#xA;alpha=0.003&#xA;&#xA;for i in range(10000):&#xA;    h=1/(1+np.exp(-np.dot(a,q)))&#xA;    for j in range(len(q)):&#xA;        q[j]=q[j]-alpha*np.sum((h-y)*a[:,j])&#xA;&#xA;plt.plot(x,y,'x',x,h)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also note that in Python you don't need ; at the end of each line.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-06-27T07:38:25.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12436" PostTypeId="2" ParentId="12296" CreationDate="2016-06-27T09:06:34.997" Score="0" Body="&lt;p&gt;Sorry cannot comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1841804&quot; rel=&quot;nofollow&quot;&gt;PRIDIT&lt;/a&gt;, &quot; ordinal categorical variables with different categories of possible response values, or continuous variables &quot;, this model should be able to support 'continuous variables'(Rank), and 'categories of possible response values' (Non-numeric data). I don't know your implementation but this model should be able to support category variable.&#xA;In another word, you do not need to convert the non-numeric data to continuous variable to fit the model. Or you just need to assign a category to any integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this technique seem viable?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;This technique is to embed categories into a vector with one dimension.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pros: easy to show the relationship with a category and the prediction value&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Cons: not good if the results is depended on combinations of categories&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What are betters way to assign &quot;ranks&quot; to non-numeric variables?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I recommend keeping the category and category (one hot vector) is supported in many current models. It can prevent losing data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;PS, you may not use the whole data frame to do analysis, selecting a certain amount with random records always give accurate results. In you case, you may select a certain amount while the number of each unique category is equal.&lt;/p&gt;&#xA;" OwnerUserId="20907" LastActivityDate="2016-06-27T09:06:34.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12437" PostTypeId="1" CreationDate="2016-06-27T09:37:02.480" Score="5" ViewCount="3606" Body="&lt;p&gt;I'm using &lt;a href=&quot;http://stanfordnlp.github.io/CoreNLP/&quot; rel=&quot;nofollow&quot;&gt;StanfordNLP (java pakage)&lt;/a&gt; for twitter sentiment analysis. I want to extract keywords from the tweets, classify the keywords as positive, negative, or neutral, and use those counts to classify the polarity of the tweet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this be done in Java? Please share few examples. What is the most used algorithm for keyword extraction?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="6478" LastEditDate="2016-06-29T00:59:15.580" LastActivityDate="2016-06-29T00:59:15.580" Title="What algorithm is used to extract keywords from unstructured texts?" Tags="|machine-learning|nlp|java|stanford-nlp|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12439" PostTypeId="2" ParentId="12437" CreationDate="2016-06-27T11:36:34.017" Score="1" Body="&lt;p&gt;I'm sorry I'm not a Java user and I never worked with StanfordNLP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I do know that the &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity&quot; rel=&quot;nofollow&quot;&gt;gini impurity criterion&lt;/a&gt; with and without decision tree were successfully applied for text classification. Moreover these tools have the ability to let you easily understand which features ( i.e. words in your case) contribute to the decision. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that convolutional neural network are a lot apply to &lt;a href=&quot;http://www.aclweb.org/anthology/C14-1008&quot; rel=&quot;nofollow&quot;&gt;sentiment analysis&lt;/a&gt; but you will have a really hard time trying to understand the intern-behaviour of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is not really an answer but I hope it might help :)&lt;/p&gt;&#xA;" OwnerUserId="14562" LastActivityDate="2016-06-27T11:36:34.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12440" PostTypeId="2" ParentId="12413" CreationDate="2016-06-27T12:05:52.963" Score="1" Body="&lt;p&gt;Consider you have some data and you want to model a function that fits the data. This function should fit well and should not have error (ideally). How do I define this error? and voila here comes the cost function.&lt;/p&gt;&#xA;" OwnerUserId="16971" LastActivityDate="2016-06-27T12:05:52.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12441" PostTypeId="1" AcceptedAnswerId="12442" CreationDate="2016-06-27T12:07:08.097" Score="0" ViewCount="443" Body="&lt;p&gt;I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an obviously data is huge (close to 1.3 million rows and 20 variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to reduce this time required for running algorithms into fraction of the time it takes currently? Lets say some hours instead of days? I have heard from a friend who has computer science background that spark takes less time than stand alone R. Not sure if Spark can reduce the analysis time from multiple days to some hours. I am open to suggestions and solutions(preferably open source). Thoughts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure a solution for this must exist as R is pretty old and some genius would have found a way to solve this painful problem.&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2016-06-27T12:45:13.667" Title="How to reduce time R takes for model building" Tags="|r|apache-spark|parallel|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12442" PostTypeId="2" ParentId="12441" CreationDate="2016-06-27T12:45:13.667" Score="1" Body="&lt;p&gt;Depends on the models you are trying to run. Your data isn't that big. For example using a support vector model from the &lt;code&gt;kernlab&lt;/code&gt; package, you run into problems. Not every model is fast or has a fast implementation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without more information on what you are doing it is difficult to say what causes the bottleneck. But if you just want a speed boost in running models, have a look at the &lt;code&gt;xgboost&lt;/code&gt; package, the &lt;code&gt;h2o&lt;/code&gt; package (GLM, GBM, rf, deeplearning), &lt;code&gt;ranger&lt;/code&gt; for a faster implementation of a randomforest model. &lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-06-27T12:45:13.667" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12443" PostTypeId="1" CreationDate="2016-06-27T15:19:01.063" Score="2" ViewCount="731" Body="&lt;p&gt;Using RBMs to pre-train a deep net as in this example &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow&quot;&gt;RBM&lt;/a&gt;, the activation function is sigmoid and makes the math much easier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the implications after the initial weights are learned using sigmoid activation functions to switch to ReLU for the train phase?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose that using tanh in either phase (pre-train or train) and sigmoid or ReLU in the other would cause great problems, but since ReLU and sigmoid are similar for small values, would it still render the pre-train phase useless?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question can be more general in the sense of how much knowledge can you transfer from a neural network using sigmoid activation functions to one identical in structure, but using ReLU activation functions.&lt;/p&gt;&#xA;" OwnerUserId="20920" LastActivityDate="2018-07-15T08:39:29.293" Title="Pre-train using sigmoid and train using ReLU?" Tags="|rbm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12444" PostTypeId="1" CreationDate="2016-06-27T17:38:13.310" Score="1" ViewCount="482" Body="&lt;p&gt;I was trying to absolve one name entity resolution problem.&#xA;Suppose, I have two sentences,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cameron is the PM of England . He decided to step down .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cameron was trying to keep England in EU . Mr. Cameron could not do it .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two sentences have two different properties, first contains a pronoun, and second one alias against Cameron. My objective is to determine the entities(NE or otherwise) and link the entities appropriately, considering pronoun and alias are entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have divided the task in two phrases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first I am using one entity extraction algorithm, and in the second, trying to link them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use Supervised Learning algorithm for both of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used one Hidden Markov Model(HMM) to extract the entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had given user defined training set and extracted entities as,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cameron/PERS is/NA the/NA PM/EXE of/NA England/LOC ./NA He/PRN decided/NA to/NA step/NA down/NA ./NA&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cameron/PERS was/NA trying/NA to/NA keep/NA England/LOC in EU/ORG ./NA Mr./TIT Cameron/PERS could/NA not/NA do/NA it/NA ./NA&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using one standard Naive Bayes(NB) classifier to classify these two sentences belonging to two groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one being 'PRN-STRING' and second one 'ALIAS'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a new training set in NB classifier, and classifying the extracted sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is being labelled as 'PRN-STRING' and second 'ALIAS'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(i) Am I doing right? As much I read about entity linking I am going right, my friends and seniors also say so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(ii) Classification and extraction tasks are independent, but am I right to combine in the same code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my code I am tagging with HMM and after some processing of the output, pushing as input to the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: EXE-&gt; It is a User defined tag as Executive.&lt;/p&gt;&#xA;" OwnerUserId="20513" LastActivityDate="2016-06-27T17:38:13.310" Title="Name Entity Linking with Naive Bayes Classifier" Tags="|machine-learning|nlp|multiclass-classification|naive-bayes-classifier|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12445" PostTypeId="1" CreationDate="2016-06-27T18:28:34.753" Score="2" ViewCount="1098" Body="&lt;p&gt;I'm trying to do an indoor locationing system based on my RSSI signal on my routers, I'm sniffing my network so I know what's the RSSI of my phone related to my routers antennas (I have 5 antennas all over my house).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The iPhone is not always broadcasting probe requests so I only get know in-real time the RSSI signal related to the router the device is connected to and no the other routers. To make it simple to understand: I sometimes have the RSSI of a device related to the 5 routers and sometimes only to 3 and in the worst case escenario only to 1 so my data sometimes looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: -55, rssi4: unknown, rssi5: unknown&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;room: 1, device_id: 1, rssi1: -80, rssi2: -95, rssi3: -55, rssi4: -102, rssi5: -96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: unknown, rssi4: unknown, rssi5: unknown&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is room based, so I'm not triangulating. I am trying to forecast in which room I am inside the house just by looking on the behaviour of the signals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine Learning would be perfect, but if the data was always there and not some values missing sometimes. What kind of data algorithm should I use for this scenarios when sometimes I have all the data and sometimes just a few.&lt;/p&gt;&#xA;" OwnerUserId="16712" LastEditorUserId="16712" LastEditDate="2016-06-27T19:10:59.013" LastActivityDate="2016-06-30T20:02:42.260" Title="Machine Learning with sometimes missing data" Tags="|machine-learning|neural-network|clustering|k-means|unsupervised-learning|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12446" PostTypeId="2" ParentId="12423" CreationDate="2016-06-27T20:20:18.483" Score="1" Body="&lt;p&gt;See the following paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1509.01549&quot; rel=&quot;nofollow&quot;&gt;Giraffe: Using Deep Reinforcement Learning to Play Chess - Lai&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12241" LastEditorUserId="12241" LastEditDate="2016-06-27T22:31:58.523" LastActivityDate="2016-06-27T22:31:58.523" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12447" PostTypeId="2" ParentId="12445" CreationDate="2016-06-27T20:48:05.773" Score="4" Body="&lt;p&gt;There are couple of ways to deal with missing data.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Replace missing values by mean/median. If the missing values is very&#xA;less, then this method would be apt. Also depends on how skewed&#xA;your data is. &lt;/li&gt;&#xA;&lt;li&gt;Imputation. Build a linear regression model to predict&#xA;the missing values based on other parameters. KNN could also be used&#xA;to predict the missing value&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also there are other methods to deal with missing data such as mimicking the parameters, removing missing data, etc. &lt;/p&gt;&#xA;" OwnerUserId="20722" LastEditorUserId="20929" LastEditDate="2016-06-28T21:05:11.600" LastActivityDate="2016-06-28T21:05:11.600" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12449" PostTypeId="1" AcceptedAnswerId="12454" CreationDate="2016-06-28T00:33:59.380" Score="2" ViewCount="3263" Body="&lt;p&gt;SPSS Modeler has an implementation of QUEST, along with C&amp;amp;RT, C5.0 and CHAID. QUEST is relatively rarely covered in textbooks - what are its pros and cons compared to other decision tree algorithms? How does it make splits? Why is it (apparently) not in as widespread use as C&amp;amp;RT or C5.0?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-28T17:18:23.130" Title="How does QUEST compare to other decision tree algorithms?" Tags="|data-mining|predictive-modeling|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12450" PostTypeId="2" ParentId="12409" CreationDate="2016-06-28T01:27:28.437" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;Eliminate exact duplicates&lt;/li&gt;&#xA;&lt;li&gt;Do fuzzy matching of addresses to get a score of near match.&lt;/li&gt;&#xA;&lt;li&gt;Retain only one of the matched address and discard the rest&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2608" LastActivityDate="2016-06-28T01:27:28.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12451" PostTypeId="2" ParentId="12409" CreationDate="2016-06-28T01:58:24.833" Score="0" Body="&lt;p&gt;You can apply clustering algorithms such as KNN or Spherical KMeans. You might need to vectorize the text in addresses before applying the algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Breakdown by zipcode or city &lt;/li&gt;&#xA;&lt;li&gt;Create feature vectors of remaining address section (except zipcode/city/State) &lt;/li&gt;&#xA;&lt;li&gt;Apply clustering algorithm&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am not quite sure how important the address field is, but I would even try regular expressions here.&lt;/p&gt;&#xA;" OwnerUserId="20722" LastActivityDate="2016-06-28T01:58:24.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12452" PostTypeId="2" ParentId="12437" CreationDate="2016-06-28T04:37:28.763" Score="1" Body="&lt;p&gt;One easy solution is to prepare a emotion dictionary first.&#xA;Such dictionary can be found online easily such as &lt;a href=&quot;http://www.psychpage.com/learning/library/assess/feelings.html&quot; rel=&quot;nofollow&quot;&gt;http://www.psychpage.com/learning/library/assess/feelings.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple workflow will be as follow:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For each tweet, tokenize the tweet into a list of vocabularies&lt;/li&gt;&#xA;&lt;li&gt;For each list of vocabularies, count the (positive, neutral, negative) words&lt;/li&gt;&#xA;&lt;li&gt;Classify a tweet as (positive/ neutral/ negative) emotion by the counting&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Although this approach does not use any machine learning, the result is quite good. Another benefit is not need to prepare the labeled data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, we should keep checking any missing emotion vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The workflow of checking missing emotion&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For each tweet, stem and tokenize by StandordNLP (ex. &quot;Happy&quot; will converted to (&quot;HAPPY&quot;, &quot;JJ&quot;) where the tag started from JJ refer to adjective&lt;/li&gt;&#xA;&lt;li&gt;Filter the tokens to extract the absent words which are absent in the prepared missing emotion vocabulary and has tag started with &quot;JJ&quot;&lt;/li&gt;&#xA;&lt;li&gt;Do the word count on the missing adjective and sort by frequency in reverse order&lt;/li&gt;&#xA;&lt;li&gt;Manually add the missing adjectives if necessary&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="20907" LastActivityDate="2016-06-28T04:37:28.763" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12453" PostTypeId="1" AcceptedAnswerId="12472" CreationDate="2016-06-28T06:45:29.917" Score="0" ViewCount="43" Body="&lt;p&gt;I have data from a public data set in gridded form 2.5 degree x 2.5 degree(lat,lon). &#xA;Latitude goes from 90 N to -90 S. Longitude goes from 0 to 357.5. It is stored every 2.5 degrees and there are no intermediate values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My user will want to download data from this data set. However they are not restricted from entering only the values that the public data set possesses. My goal then is to convert the data they enter to the &quot;nearest&quot; latitude and longitude of the public data set. For example while data may exist at 5 N and 60 E(and the next pont is 7.5 N and 62.5E) they may enter 6 N and 61.3 E. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I map the user input to the nearest latitude and longitude ? Note - user input will be in the form of a rectangle - lat_min,lat_max, lon_min and lon_max. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a shot at this and I hope I can find a better algorithm through DS SE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lat_min = floor(lat_min/2.5) * 2.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lat_max = ceiling(lat_max/2.5) * 2.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lon_min = floor(lon_min/2.5) * 2.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lon_max = ceiling(lon_max/2.5) * 2.5&lt;/p&gt;&#xA;" OwnerDisplayName="user20941" LastActivityDate="2016-06-28T16:39:34.687" Title="Convert Lat /Lon of User input to Lat/Lon of Open Data" Tags="|dataset|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12454" PostTypeId="2" ParentId="12449" CreationDate="2016-06-28T07:00:52.263" Score="3" Body="&lt;p&gt;QUEST stands for Quick, Unbiased and Efficient Statistical Tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It uses ANOVA F and contingency table Chi Square tests to select variables for splitting. Variables with multiple classes are merged into two super-classes to get binary splits which are determined using QDA (Quadratic Discriminant analysis). The tree can be pruned using the CART algorithm. It can be used for both classification and regression tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quest first transforms categorical (symbolic) variables into continuous variables by assigning discriminant coordinates to categories of the predictor. Then it applies quadratic discriminant analysis (QDA) to determine the split point. Notice that QDA usually produces two cut-off points—choose the one that is closer to the sample mean of the first superclass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An advantage of the QUEST tree algorithm is that it is not biased in split-variable selection, unlike CART which is biased towards selecting split-variables which allow more splits, and those which have more missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure why it is not used as widely as CART or C5.0. It might be due to greater coverage of CART/C5.0 in literature than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Quest reference manual: &lt;a href=&quot;http://www.stat.wisc.edu/~loh/treeprogs/guide/guideman.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.wisc.edu/~loh/treeprogs/guide/guideman.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.wisc.edu/~loh/quest.html&quot; rel=&quot;nofollow&quot;&gt;http://www.stat.wisc.edu/~loh/quest.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-06-28T17:18:23.130" LastActivityDate="2016-06-28T17:18:23.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12455" PostTypeId="1" AcceptedAnswerId="12476" CreationDate="2016-06-28T08:17:05.813" Score="1" ViewCount="1527" Body="&lt;p&gt;I am searching for a feature selection algorithm which selects features that are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;relevant to discriminate groups of samples (for each sample a group label is provided)&lt;/li&gt;&#xA;&lt;li&gt;endowed with high variance across all the samples&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This should be applied to gene expression dataset, in which each sample has a group label, therefore it should be possible to select for each group a set of features to be checked against.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have now two candidates:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;selecting features by the feature importance result of a Random Forest classifier&lt;/li&gt;&#xA;&lt;li&gt;using the Minimum Redundancy Maximum Relevance (mRMR) algorithm&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I am unsure of which may be the best or if there are better candidates for this purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the algorithm is implemented in Python scikit-learn it would be a plus.&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2016-06-28T20:44:16.683" Title="Feature selection for gene expression dataset" Tags="|scikit-learn|feature-selection|bioinformatics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12457" PostTypeId="1" CreationDate="2016-06-28T10:07:06.240" Score="2" ViewCount="248" Body="&lt;p&gt;I'm trying to explain a count variable and a continious variable &gt; 0 with GLM, using R.&#xA;In order to improve the quality of the regression, I want to add some interactions that can be useful for the model.&#xA;As I'm a newbie in machine learning, I want to know if RF and GBM can help me to determine useful interactions.&#xA;I saw that interact.gbm can assess the relative strength of interaction effects in non-linear models. The question is :&#xA;Will it be &quot;mathematically&quot; correct to add variables with important strength of interaction in order to reduce MSE/Deviance ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you !&lt;/p&gt;&#xA;" OwnerUserId="20951" LastActivityDate="2017-04-24T22:09:06.927" Title="Searching interactions with RandomForest and/or GBM" Tags="|random-forest|gbm|glm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12458" PostTypeId="2" ParentId="12437" CreationDate="2016-06-28T10:19:13.460" Score="0" Body="&lt;p&gt;If I'd have to do that, I would use the &quot;term-frequency inverse-document-frequency&quot;: &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;tokenize tweets of your topic&lt;/li&gt;&#xA;&lt;li&gt;calculate the tf-idf&lt;/li&gt;&#xA;&lt;li&gt;take all words with a high score of tf-idf and train a NN with it (as supervised machine learning) - use bag-of-words to get a good representation&lt;/li&gt;&#xA;&lt;li&gt;use the NN on new tweets&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;BTW: I don't know how familiar you are with machine learning but it could be a good workflow.&lt;/p&gt;&#xA;" OwnerUserId="20952" LastActivityDate="2016-06-28T10:19:13.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12461" PostTypeId="1" CreationDate="2016-06-28T11:45:18.460" Score="1" ViewCount="41" Body="&lt;p&gt;I have a panel data set and I have created a model and finally I have obtained some density forecasts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, I run my model for the $y_{it}$ and i obtain predictions for $\hat{y_{i,t+1}}$ , $\hat{y_{i,t+2}}$ ,..., $\hat{y_{i,t+h}}$  etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $\hat{y_{i,t+1}}$ , $\hat{y_{i,t+2}}$ etc, &lt;strong&gt;all&lt;/strong&gt; of them follow a normal distribution, but with different mean and variance. so $\hat{y_{i,t+1}} \sim N(\mu_{1},\sigma_{1}^{2})$ , $\hat{y_{i,t+2}} \sim N(\mu_{2},\sigma_{2}^{2})$, ..., $\hat{y_{i,t+h}} \sim N(\mu_{h},\sigma_{h}^{2})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, what is their joint distribution ? And how can I obtain it ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: The $y_{it}$'s are the daily consumption of a product of individual $i$ at time $t$, so yes, they are highly correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help :)&lt;/p&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2016-06-29T14:02:02.723" LastActivityDate="2016-06-29T14:02:02.723" Title="Joint distribution of density forecasts" Tags="|predictive-modeling|forecast|prediction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12462" PostTypeId="2" ParentId="12437" CreationDate="2016-06-28T13:37:27.840" Score="4" Body="&lt;p&gt;The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., &lt;a href=&quot;https://datascience.stackexchange.com/questions/10885/twitter-sentiment-analysis-detecting-neutral-tweets-despite-training-on-only-po/11908#11908&quot;&gt;this post&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach I would suggest is a &lt;em&gt;key keyword&lt;/em&gt; approach advocated by Mike Scott (author of WordSmith Tools) and Chris Tribble (a computational linguist). As discussed &lt;a href=&quot;http://www.lexically.net/downloads/corpus_linguistics/PC%20Analysis%20of%20Key%20Words%20-%20and%20Key%20Key%20Words.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, the basic approach is to create two corpora, your &lt;strong&gt;target corpus&lt;/strong&gt; which is composed of texts sampled from the texts you are interested in, and a &lt;strong&gt;reference corpus&lt;/strong&gt;, which is typically a much larger corpus (which is often more general in content). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The procedure begins by computing word (or &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow noreferrer&quot;&gt;n-gram&lt;/a&gt;) frequencies for both corpora. During this process, if a word’s frequency in the target corpus is found to be statistically probable in comparison to the reference corpus (as computed by a chi-squared test and a user defined p-value), it is considered a keyword (Baker, 2004). According to Scott (2006), the process typically identifies three types of words as key:&#xA;proper nouns, words that characterize a text’s “aboutness,” and high frequency words that are indicators of style or genre. I discuss the approach in greater length in this 2007 &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S1475158507000343&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; where I use the method to extract the salient features of academic discourse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate with a concrete example, imagine your are interested in identifying the key topics surrounding a target brand (say, Pepsi Cola) as expressed in social media (e.g., twitter). Create two corpora: for your target corpora you create a search for &quot;pepsi&quot; and for the reference corpora you could search for competitor brands of soft drinks (&quot;coke&quot;, &quot;mountain dew&quot;, &quot;dr. pepper&quot;, etc,). When the keyword process has terminated you will be left with all the keywords/topics that differentiate pepsi from other soft drink brands (and as a bonus you will also identify negative keywords... words that occur statistically less frequent in the target corpus). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you might surmise, the results you get depend upon how the reference corpus is constructed. This, in my opinion, is a feature - as it gives the researcher much greater flexibility in hypothesis testing and data exploration.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-06-28T13:58:36.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12463" PostTypeId="1" CreationDate="2016-06-28T13:49:01.797" Score="2" ViewCount="1097" Body="&lt;p&gt;I'm doing some classification with a text corpus [professional emails], I've already done all the basic&#xA;pre processing steps (stemming, remove stop words, punctuation, low frequency words, words length ...)  which gives me a F1-score = 0.6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering what could be the next step(s) to improve my results, I thought of &quot;stacking&quot; (use multiple classifiers) and maybe removing words which are common to the different classes (4 differents types of texts) [ not sure about this solution ]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried with 15 differents classifier the best one is gradient boosting (gbm )with an F1-score=0.6002159.&#xA;The features have been extracted through a bag of words ( Document Term Matrix ) with Tf-Idf [I don't think taking into account the length of a text, punctuation or others is relevant] - [It is mainly professional emails]&lt;/p&gt;&#xA;" OwnerUserId="19065" LastEditorUserId="19065" LastEditDate="2016-06-28T14:00:18.537" LastActivityDate="2016-07-01T19:52:05.360" Title="Increase F1 score on a text corpus" Tags="|machine-learning|text-mining|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12466" PostTypeId="1" CreationDate="2016-06-28T14:41:09.853" Score="3" ViewCount="584" Body="&lt;p&gt;I am performing regression analysis on prices of product that we have purchased, based on size and other attributes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each time I run a regression, I will check the 20 with the largest error manually, and 90%+ of the time they will be odd buys like mentioned before, and for my purposes can be completely ignored. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been looking into cooks distance to remove these, however I'm not sure how to best set the threshold, or if there is a better method to use.&lt;/p&gt;&#xA;" OwnerUserId="20959" LastActivityDate="2016-06-29T07:20:38.193" Title="How can I identify and remove outliers in R" Tags="|machine-learning|r|linear-regression|outlier|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12467" PostTypeId="2" ParentId="12457" CreationDate="2016-06-28T14:43:19.547" Score="1" Body="&lt;p&gt;Several times it does happen that interactions among variables improve the bias of the model. This is especially true when the &lt;strong&gt;effect of one independent variable on the target depends on the values of other independent variables.&lt;/strong&gt;&#xA;I don't think there's anything &lt;em&gt;mathematically incorrect&lt;/em&gt; in doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g: let's say you are trying to predict revenue as a function of advertising. In this example it's reasonable to assume that the effect on revenue of one extra unit of advertising on Television  would depend on the existing level of advertising on Facebook (say). The true data generating function (if you had access to it) might be something like: $$Revenue=Ad_{TV}^{\beta_{1}}Ad_{Print}^{\beta_{1}}...$$ where $Ad_{TV}$ is the # advertisements shown on TV etc.&#xA;If you use provide the model the opportunity to handle such interaction terms you will be closer to modeling the true data generating function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However adding more feature increases the complexity (capacity) of the model and you might have to use regularization wisely to prevent over-fitting&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-06-28T14:50:10.687" LastActivityDate="2016-06-28T14:50:10.687" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12468" PostTypeId="1" AcceptedAnswerId="12502" CreationDate="2016-06-28T15:07:36.833" Score="0" ViewCount="112" Body="&lt;p&gt;I have a dataframe in R containing  a variable for programming languages. This is extracted from a multiple choice questionnaire in a survey. As a result, any programming language may fall into the beginning, middle or end of the whole string.&#xA;I added a binary variable for any programming language. Here is a sample of my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;language &amp;lt;- c(&#xA;&quot;Ruby, Assembly, R, Go, Rust&quot;,&#xA;&quot;Java, Javascript, Ruby, SQL&quot;,&#xA;&quot;Java, Ruby&quot;,&#xA;&quot;C, C++&quot;,&#xA;&quot;PHP, Javascript, Ruby, Assembly, Swift, R, Matlab, Go, Haskell&quot;,&#xA;&quot;R&quot;,&#xA;&quot;Perl, Javascript, R&quot;,&#xA;&quot;Javascript, Ruby, Bash&quot;,&#xA;&quot;Python, PHP, Javascript&quot;,&#xA;&quot;Java&quot;,&#xA;&quot;Java, C&quot;&#xA; )&#xA;&#xA;df &amp;lt;-as.data.frame(language,stringsAsFactors = FALSE)&#xA;df$python      &amp;lt;- 0&#xA;df$java        &amp;lt;- 0&#xA;df$c           &amp;lt;- 0&#xA;df$r           &amp;lt;- 0&#xA;df$d           &amp;lt;- 0&#xA;&#xA;for(trow in (1:nrow(data)))&#xA;{if (length (grep(pattern=&quot;Python&quot;,x=df[trow,&quot;language&quot;]))  &amp;gt; 0)    {df[trow,&quot;python&quot;] &amp;lt;- 1}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have problem with one-word languages such as C, R, D and Java . I made the variables for C#, C++ , Javascript, Ruby and others using the for loop but  I have problems with these four. What regular expression can I use so that it covers Java but not Javascript (in the beginning, middle and end of the string) and covers R but not Ruby and so on? Any other solution for making the binary variables are also welcome.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-06-30T00:34:49.577" Title="problem with regular expression" Tags="|r|data-cleaning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12469" PostTypeId="1" CreationDate="2016-06-28T15:51:34.707" Score="0" ViewCount="361" Body="&lt;p&gt;I'm open to suggestions on how to improve the title.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is this, but I think it's a more general problem. In my context, I have a lot of data which has location data (Lat/Lon) as well as various characteristics of my objects.  I'm interested in clustering &lt;em&gt;based on location&lt;/em&gt; in such a way that it controls for effects based on weather and other local effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I want is a clustering which, after trained, determines the cluster of a new object solely based on its GPS coordinates. However, I want the clusters not to just cluster based on locational clusters (there are lots of objects in this ball in Florida, this ball in California, etc.) but to make clusters so that within the clusters, objects are more similar than you would expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example, k-means would &lt;em&gt;not&lt;/em&gt; accomplish this; it would just identify pockets of objects based on location clustering, or it would require more information to determine the cluster of a new object (neither are desirable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way would be to run k-means a bunch of times, rate the results based on comparing homogeneity within clusters, and pick the best, but this seems both computationally annoying and unlikely to get good results (it would tend to identify circular clusters, rather than e.g. picking out things within 10 miles of a coast, or whatever turns out to be useful).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like someone has thought of this before and come up with something.  What sorts of algorithms (classical or not) might help with this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't mind re-implementing something on my own, and I don't mind stacking/etc. classifiers.  My dataset is fairly small for now, so even if there isn't a fast algorithm for this, it should be okay (even $O(n^2 \log(n))$ algorithms are doable, I've found).&lt;/p&gt;&#xA;" OwnerUserId="19011" LastActivityDate="2017-01-12T11:20:08.780" Title="Clustering based on partial information?" Tags="|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12471" PostTypeId="2" ParentId="12468" CreationDate="2016-06-28T16:03:39.593" Score="0" Body="&lt;p&gt;Your examples are well-formatted; this answer assumes that's the reality of your data, and not just the examples you've provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like you could split on spaces and strip commas, then just pick out languages based on exact equality (this would not have a problem for Java vs. JavaScript, or R vs. Ruby).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you prefer to use regular expressions, it seems like you could search for language+comma or language+(end of string).  E.g. for searching for familiarity with R, you could do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R[ ,$]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So it will match R if and only if R is followed by a space or comma (indicating the end of the language) or the end of the string (indicated by a dollar sign).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to be doubly safe (though this won't matter in your examples) you could even do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[, ^]R[ ,$]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which will only match R if both the beginning and the end are among the boundary of the string, a space, or a comma.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replace R with whatever the full name of your programming language is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should all work so long as both of the following are satisfied:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your languages are all spaced out by either commas or spaces or both.&lt;/li&gt;&#xA;&lt;li&gt;None of your languages include commas or spaces (and even then it should be fine unless they interact in funny ways, e.g. a language &quot;Go&quot; and another language &quot;Go Home&quot;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="19011" LastActivityDate="2016-06-28T16:03:39.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12472" PostTypeId="2" ParentId="12453" CreationDate="2016-06-28T16:39:34.687" Score="0" Body="&lt;p&gt;What's wrong with what you've suggested?  It seems fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've supplied four of possible points, essentially:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;(latmin, lonmin)&lt;/li&gt;&#xA;&lt;li&gt;(latmin, lonmax)&lt;/li&gt;&#xA;&lt;li&gt;(latmax, lonmin)&lt;/li&gt;&#xA;&lt;li&gt;(latmax, lonmax)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Clearly one of them is the closest grid point to the user input. Then just use whatever distance calculation you like (e.g. Haversine distance, or better measurements if you have access to them) and pick which of the four points is closest to what the user specified.  It'll be accurate and performant.&lt;/p&gt;&#xA;" OwnerUserId="19011" LastActivityDate="2016-06-28T16:39:34.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12473" PostTypeId="2" ParentId="11682" CreationDate="2016-06-28T17:57:49.450" Score="1" Body="&lt;p&gt;This XPath will look for a &lt;em&gt;tag&lt;/em&gt; called &lt;code&gt;&amp;lt;Text&amp;gt;&lt;/code&gt; inside a tag called &lt;code&gt;&amp;lt;posthistory&amp;gt;&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;XPath(x, 'posthistory/Text');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You want to find the &lt;code&gt;Text&lt;/code&gt; attribute of the &lt;code&gt;row&lt;/code&gt; tag in &lt;code&gt;posthistory&lt;/code&gt; tags.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An XPath something like this will do that: &lt;code&gt;/posthistory/row/@Text&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See example here: &lt;a href=&quot;http://www.xpathtester.com/xpath/bac9874ec344f9d8ebcfb250633aaf65&quot; rel=&quot;nofollow&quot;&gt;http://www.xpathtester.com/xpath/bac9874ec344f9d8ebcfb250633aaf65&lt;/a&gt; and click &quot;Test&quot; to see results set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learn up on XPath notation for more.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-06-28T17:57:49.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12474" PostTypeId="2" ParentId="8694" CreationDate="2016-06-28T18:05:32.680" Score="2" Body="&lt;p&gt;I would recommend using a combination of both options #1 and #2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could first try tuning your hyper-parameters to find out till what extent could you reduce the number of trees to a point where the random forest model's prediction starts deteriorating on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because changing the value of &lt;code&gt;mtry&lt;/code&gt;, the randomly selected number of features for a new tree, is the only meaningful hyper-parameter that should impact accuracy of the model. Since averaging converges as the no. of trees increases, the no. of trees could be reduced to a point where its performance is not impacted as much. Hence, you need to iterate and choose a a limit beyond which very small number of trees may not produce a strong enough ensemble. A random forest needs works best by using more base learners for reducing the variance by averaging each individual tree's output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear from your case whether you're using the Random Forest for a a classification or a regression problem. In case this is a classification problem, and if your data-set is imbalanced in terms of ratio of positive vs. negative classes; then you could reduce the size of the training set by under-sampling the majority class to bring it nearer to a 1:1 ratio. Since you have a large number of records, such class based sampling could improve accuracy as well as reduce data size for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, if you've got a fine tuned Random Forest with good performance, then you could also evaluate dropping features that are least important as determined by the algorithm on OOB samples. This would reduce the time taken to train the model.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-06-28T18:05:32.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12475" PostTypeId="1" CreationDate="2016-06-28T20:23:29.867" Score="6" ViewCount="2188" Body="&lt;p&gt;I am working on a project and currently experimenting cluster analysis. The dataset is mainly categorical variables and discrete numbers. Please pardon my poor programming skills as I am not very familiar with MathJax, but I will try to summarize the data set in words in the following three examples.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;imagine column 1 is participants names of course, from column2 - 5, each column's value range from 1 (least important) - 5 (most important). so in this case, column 2-8 only have discrete data. &lt;/li&gt;&#xA;&lt;li&gt;column 6 for example, since this is a multiple choice question. row 1 chose &quot;nice&quot; as an answer, however, row 2 chose &quot;poor&quot;. In this case, we have one column that contains multiple categorical answers.&lt;/li&gt;&#xA;&lt;li&gt;for column 7-9, this is another type of multiple choice questions. However this time, each column represents only one answer. column 7 only allows string values &quot;true&quot;, column 8 only allows string &quot;somewhat&quot;. So in this case, we have multiple columns that represent multiple answers of the same question. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to solve this? Thankful to any input! &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A2       A3       A4       A5       A6       A7         A8         A9&#xA;1        4        5        4        nice     true       somewhate  false&#xA;2        4        3        1        poor     true                  false&#xA;1        5        2        1        nice                somewhate    &#xA;3        2        1        5        nice     true                  false&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16679" LastEditorUserId="16679" LastEditDate="2016-06-29T21:27:53.790" LastActivityDate="2016-06-30T05:28:08.767" Title="Clusering based on categorical variables?" Tags="|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12476" PostTypeId="2" ParentId="12455" CreationDate="2016-06-28T20:44:16.683" Score="2" Body="&lt;p&gt;It would be helpful if you described your dataset more.  Gene expression datasets seem to often have very high dimensionality and Lasso regularized logistic regression is a popular method to approach this problem.  This paper takes it a little further and might help you out:&#xA;&lt;a href=&quot;http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-198&quot; rel=&quot;nofollow&quot;&gt;http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-198&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random forest can generally certainly provide a meaningful importance ranking, but it also depends on what your dataset looks like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mRMR sounds like it is specifically designed for identifying gene characteristics, so definitely give it a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's also Principle Component Analysis which is also used for gene expression data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lots of options, but your questions is not detailed enough to go any further, and providing code as a solution at this point isn't realistic. The documentation for Python scikit-learn has many good explanations and examples.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-06-28T20:44:16.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12477" PostTypeId="2" ParentId="12466" CreationDate="2016-06-28T23:42:56.220" Score="1" Body="&lt;p&gt;Cook's distance and the alternative method DFFITS are not strictly speaking methods to detect 'outliers' in the sense of purely anomalous values, rather they detect 'influential' points, where leaving the value in or out of the analysis noticeably changes the result. Hence, Cook's distance measures how much the beta values change when a specific observation is omitted, which is a different idea to detecting an outlier as such, although they will naturally often be the same points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the description of your problem, it is possible that detecting influential points is what you are actually trying to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To directly answer your question on the Cook's distance threshold, the F-statistic with p and n-p (where p is number of regressors, and n is observations) degrees of freedom is often used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outliers per se in regression are more often identified via residual analysis e.g. if the data point's corresponding residual is large in the context of the data set. This is often taken to be when the value lies more than three standard deviations away from the mean of the residuals (though note that in a sufficiently large data set, a well fitted model will still have some observations at this level)&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-28T23:42:56.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12479" PostTypeId="2" ParentId="12475" CreationDate="2016-06-29T02:25:33.693" Score="1" Body="&lt;p&gt;I assume your data set is sth like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A0 A1 A2 B0 B1 B2&#xA;1  0  0  1  1  0&#xA;0  1  0  0  0  1&#xA;0  1  0  0  1  1&#xA;      ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where A0 refer to the answer of question A, choice 0 and question A is a single choice question,&#xA;B is a question allowing multiple answers.&#xA;Each row represents one record.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this data format, each column can be considered as a dimension (if some columns can be quantified to a value, you may group them into one column, ex &quot;Like = 1, Neutral = 0, Dislike = -1&quot;) Then applying clustering algorithms, such as K-mean to cluster it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please tell if your data set format is different from my assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Updated on Jun 30:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to quantify the variables,&#xA;For column 2-5, the values are already quantified to [1, 5]&#xA;For column 5, it allows (&quot;poor&quot;/ &quot;nice&quot;). Although they are strings, they are representing the preference of different magnitude. They can also be converted to &quot;poor&quot; = 0, &quot;nice&quot; = 1.&#xA;For column 7-9, because it is a binary option, they can also be easily converted to ex. &quot;true&quot; = 0, &quot;&quot; = 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you will have a matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1    4    5    4    1    1    1    1&#xA;2    4    3    1    0    1    0    1&#xA;1    5    2    1    1    0    1    1&#xA;3    2    1    5    1    1    0    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Remark1: For binary options, such as column 9, only allows &quot;False&quot; or &quot;null&quot;, choosing &quot;False&quot; = 1 or 0 does not matter. In classification, only the distance matters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remark2: If you have a column allows different strings, such as &quot;apple&quot;, &quot;banana&quot;, &quot;orange&quot;, you can convert it to an one hot vector.&#xA;Ex.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A&#xA;apple&#xA;orange&#xA;orange&#xA;banana&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;can be converted to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A0   A1   A2&#xA;1    0    0&#xA;0    0    1&#xA;0    0    1&#xA;0    1    0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where A0, A1, A2 represent &quot;apple&quot;, &quot;banana&quot; and &quot;orange&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remark3: if there is a column allow storing multi answers, ex.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A&#xA;apple&#xA;apple orange&#xA;orange&#xA;orange banana&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;can be converted to&#xA;    A0   A1   A2&#xA;    1    0    0&#xA;    1    0    1&#xA;    0    0    1&#xA;    0    1    1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remark4: before applying some classification algorithms such as K-mean, it is better to normalize the magnitude of each dimension. For example, value of column 9 is [0, 1] can be normalized by times 5 to [0, 5]. The factor/ the range of the dimension reflects the importance of that factor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope my answer can help.&lt;/p&gt;&#xA;" OwnerUserId="20907" LastEditorUserId="20907" LastEditDate="2016-06-30T05:28:08.767" LastActivityDate="2016-06-30T05:28:08.767" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="12481" PostTypeId="2" ParentId="12466" CreationDate="2016-06-29T07:20:38.193" Score="0" Body="&lt;p&gt;You should use robust regression e.g. &lt;code&gt;lmrob&lt;/code&gt; from R-package &lt;code&gt;robustbase&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20981" LastActivityDate="2016-06-29T07:20:38.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12482" PostTypeId="1" AcceptedAnswerId="12487" CreationDate="2016-06-29T07:30:18.007" Score="0" ViewCount="44" Body="&lt;p&gt;Do we have a data mining/analysis term for &lt;a href=&quot;https://stackoverflow.com/questions/17058219/r-turn-row-values-into-column-names-and-count-its-assigned-values&quot;&gt;turning row values into column names and count its assigned values&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="12739" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-06-29T13:21:17.250" Title="Analytics term for turning row values into column names and count its assigned values" Tags="|terminology|data-wrangling|aggregation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12483" PostTypeId="2" ParentId="10091" CreationDate="2016-06-29T07:35:28.100" Score="1" Body="&lt;p&gt;There are certain feature extraction algorithms in opencv library. Some of them are SURF or SIFT, HOG in opencv.  Local Binary Pattern(LBP) in sklearn library in Python. One more technique is to create Bag of visual words. There also exists BOW class in opencv. To understand the concept of bag of visual words you can look for some of the research papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SURF in opencv Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;surf = cv2.SURF(400)&#xA;kp, des = surf.detectAndCompute(img,None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see the opencv &lt;a href=&quot;http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_surf_intro/py_surf_intro.html&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; for more details. &#xA;Similarly you can know about all other feature extraction methods. &lt;a href=&quot;https://gurus.pyimagesearch.com/lesson-sample-histogram-of-oriented-gradients-and-car-logo-recognition/?email=riken.rocks09@gmail.com#&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is one more blog regarding HOG feature extraction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you extract all your features from your training data-set images, you can use &lt;em&gt;svm&lt;/em&gt; library of &lt;em&gt;sklearn&lt;/em&gt; to train your classifier. There are plenty of resources out there that will help you start with ML in python.&lt;/p&gt;&#xA;" OwnerUserId="20982" LastActivityDate="2016-06-29T07:35:28.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12484" PostTypeId="2" ParentId="11627" CreationDate="2016-06-29T07:43:24.897" Score="1" Body="&lt;p&gt;For categorical data, robust hierarchical clustering algorithm ( ROCK) will work better that employs links and not distances when merging clusters, which improves quality of clusters of categorical data. Boolean and categorical are two types of attributes that are most suited in this algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ROCK is a static model that combines nearest neighbor, relocation, and hierarchical agglomerative methods. In this algorithm, cluster similarity is based on the number of points from different clusters that have neighbors in common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use CBA Package in R to perform the ROCK clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithm Steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data-----&gt;Draw Random Sample-----&gt;Cluster with Links-----&gt;Label Data in DIsk&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A random sample is drawn from the database&lt;/li&gt;&#xA;&lt;li&gt;A hierarchical clustering algorithm employing links is applied to the samples&lt;/li&gt;&#xA;&lt;li&gt;This means: Iteratively merge clusters Ci, Cj that maximise the goodness function&#xA;merge(point1,point2) = total number of crosslinks /expected number of crosslinks &#xA;Stop merging once there are no more links between clusters or the required number of clusters has been reached.&lt;/li&gt;&#xA;&lt;li&gt;Clusters involving only the sampled points are used to assign the remaining data points on disk to the appropriate clusters  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details with examples, refer the following links:&#xA;&lt;a href=&quot;https://www.cis.upenn.edu/~sudipto/mypapers/categorical.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.cis.upenn.edu/~sudipto/mypapers/categorical.pdf&lt;/a&gt;&#xA;&lt;a href=&quot;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/RockCluster&quot; rel=&quot;nofollow&quot;&gt;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/RockCluster&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9828" LastActivityDate="2016-06-29T07:43:24.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12487" PostTypeId="2" ParentId="12482" CreationDate="2016-06-29T12:34:34.543" Score="0" Body="&lt;p&gt;Aggregation.&#xA;See  &lt;a href=&quot;https://en.wikipedia.org/wiki/Aggregate_function&quot; rel=&quot;nofollow&quot;&gt;Aggregate function&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-06-29T12:34:34.543" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12488" PostTypeId="2" ParentId="12482" CreationDate="2016-06-29T13:21:17.250" Score="0" Body="&lt;p&gt;If you mean the part about counting values, that is indeed simply a form of &quot;aggregation&quot;. Just call it counting even.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you mean the part where the resulting row values become columns in the result table, then I'd say this is called &quot;pivoting&quot;, made famous by Excel's &lt;a href=&quot;https://en.wikipedia.org/wiki/Pivot_table&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Pivot_table&lt;/a&gt; though this comes up in RDBMSes in general.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-06-29T13:21:17.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12491" PostTypeId="1" AcceptedAnswerId="12675" CreationDate="2016-06-29T16:11:34.107" Score="9" ViewCount="3575" Body="&lt;p&gt;&lt;a href=&quot;https://numer.ai/&quot; rel=&quot;noreferrer&quot;&gt;Numer.ai&lt;/a&gt; has been around for a while now and there seem to be only few posts or other discussions about it on the web.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The system has changed from time to time and the set-up today is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;train (N=96K) and test (N=33K) data with 21 features with continuous values in [0,1] and a binary target.&lt;/li&gt;&#xA;&lt;li&gt;The data is clean (no missing values) and updated every 2 weeks. You can upload your predictions (on the test set) and see the log-loss. Part of the test data is even live data and you get paid for good predictions.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I would like to discuss:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the features are totally anonymous I think that there is not much feature engineering we can do. So my approach is very mechanical:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;inspired by &lt;a href=&quot;http://fastml.com/adversarial-validation-part-two/&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; I use a classification algorithm to filter out those training data which fit to my test data best.&lt;/li&gt;&#xA;&lt;li&gt;Figure out some nice preprocessing&lt;/li&gt;&#xA;&lt;li&gt;train nice classification algorithms &lt;/li&gt;&#xA;&lt;li&gt;build ensembles of them (stacking, ..).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The concrete question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning step 1: Do you have experience with such an approach? Let's say I order the probability of train samples to belong to test (usually below 0.5) and then I take the largest K probabilities. How would you choose K? I tried with 15K .. but mainly to have a small training data set in order to speed up training in step 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning step 2: The data is already on a 0,1 scale. If I apply any (PCA like) linear transformation then I would break this scale. What would you try in preprocessing if you have such numerical data and no idea that this actually is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am aware that because numer.ai pays people discussing this could help me make some money. But as this is public this would help anybody out there...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PPS:&#xA;Today's leaderboard has an interesting pattern:&#xA;The top two with logloss of 0.64xx, then number 3 with 0.66xx and then most of the predictors reach 0.6888x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus there seems to be a very small top field and lot of moderately successful guys (including me).&lt;/p&gt;&#xA;" OwnerUserId="20884" LastEditorUserId="21024" LastEditDate="2016-07-06T13:09:13.073" LastActivityDate="2016-07-11T20:28:49.173" Title="How to approach the numer.ai competition with anonymous scaled numerical predictors?" Tags="|machine-learning|deep-learning|cross-validation|preprocessing|competitions|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12493" PostTypeId="2" ParentId="12403" CreationDate="2016-06-29T18:11:07.497" Score="4" Body="&lt;p&gt;In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used the caret package extensively. &lt;a href=&quot;http://topepo.github.io/caret/modelList.html&quot; rel=&quot;nofollow&quot; title=&quot;here&quot;&gt;Here&lt;/a&gt; is a reference of the parameters supported by caret for tuning a xgboost model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To automatically select parameters using caret, do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First define a range of values of each parameter you would want caret to search. Define this in the tuning grid.&lt;/li&gt;&#xA;&lt;li&gt;Start model training using caret after specifying a measure to optimize, e.g. accuracy or Kappa statistic, etc.&lt;/li&gt;&#xA;&lt;li&gt;Plot or print the performance comparison for various parameter values, refine and repeat if required.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Refer to the caret guide &lt;a href=&quot;http://topepo.github.io/caret/&quot; rel=&quot;nofollow&quot; title=&quot;here&quot;&gt;here&lt;/a&gt; to get step-by-step instructions on using it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For handling class imbalance, I've found from my experience that adjusting weights is not as helpful as under-sampling majority class and over-sampling the minority class, or a combination of the two. However, it all depends on the size of data available and the case at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you need to tune some parameters which are not supported by caret, then, you could write your own iterative loop to train and test the model for different values of that parameter and then choose one that works best. I think most of the really relevant parameters have already been included in caret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would need to adjust these parameters in case the population itself changes over time. Or, the methods to gather data and their accuracy may change which could result in performance deterioration. You could run a simple check by comparing the performance of your model over the current dataset vs. a 6 month older dataset. If the performance is similar, then you may not need to update the model in the future.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-06-30T06:37:01.370" LastActivityDate="2016-06-30T06:37:01.370" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12494" PostTypeId="1" CreationDate="2016-06-29T19:50:30.070" Score="4" ViewCount="396" Body="&lt;p&gt;I'll set this question up as a simplified example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We hypothesize that there is a causal relationship between average gasoline prices and road traffic in a particular city.  The data cover the same time period at the same frequency (e.g.: weekly) and contain no missing values. We'll put aside the idea that there could be confounding variables at work, presume that we've made the time series stationary (via differencing), and determine there is no autocorrelation in the series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are reliable methods to determine that if a shock occurs in the price data that a corresponding shock occurs in the traffic data at some time lag?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked a ;little bit into Dynamic Time Warping, but I suspect that this is better suited to determining how to map one series to another rather than for determining causality.&lt;/p&gt;&#xA;" OwnerUserId="14625" LastActivityDate="2016-06-30T04:07:40.283" Title="Methods for Determining Possible Causation Between Two Time Series" Tags="|time-series|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12495" PostTypeId="2" ParentId="11518" CreationDate="2016-06-29T19:56:10.063" Score="2" Body="&lt;p&gt;Have you considered using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Mahalanobis_distance&quot; rel=&quot;nofollow&quot;&gt;Mahalanobis Distance&lt;/a&gt;? It's can be thought of as the magnitude of a multi-dimensional Z-score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I prefer to view the the Mahalanobis Distance is as the square root of the exponent of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Multivariate_normal_distribution&quot; rel=&quot;nofollow&quot;&gt;Multivariate Normal Distribution&lt;/a&gt;. This is similar to the Z-score, which is the exponent of the univarate normal distribution before the square is applied. A big difference between these two is the Z-score is signed, while the Mahalanobis Distance is unsigned, which doesn't matter for finding outliers anyway. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand you don't have normally distributed data, but sometimes pretending your data is normally distributed can have good results, so using the Mahalanobis Distance can be worth investigating. &lt;/p&gt;&#xA;" OwnerUserId="21006" LastEditorUserId="21006" LastEditDate="2016-06-30T15:20:27.917" LastActivityDate="2016-06-30T15:20:27.917" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12497" PostTypeId="2" ParentId="262" CreationDate="2016-06-29T20:47:42.270" Score="4" Body="&lt;p&gt;To add, check out &lt;a href=&quot;https://asdf-standard.readthedocs.io/en/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;ASDF&lt;/a&gt; in particular their paper &lt;a href=&quot;http://dx.doi.org/10.1016/j.ascom.2015.06.004&quot; rel=&quot;nofollow&quot;&gt;ASDF: A new data format for Astronomy&lt;/a&gt;; ASDF tries to improve upon HDF5 and the paper describes some downsides of HDF5 format.&lt;/p&gt;&#xA;" OwnerUserId="21009" LastActivityDate="2016-06-29T20:47:42.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12498" PostTypeId="1" CreationDate="2016-06-29T21:24:01.147" Score="4" ViewCount="262" Body="&lt;p&gt;I'm working with a database about internally displaced persons in Colombia. All data are absolutes values, so I calculate the rate per 1000 people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started to visualize all data using QGis. I choosed std dev, quantiles and jenks methods to determine the arrangement of values into different classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MmMmw.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MmMmw.jpg&quot; alt=&quot;Clusters&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I got a question in this proccess: what is the best arrangement of values when we have big gaps between them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a list of values I am working with.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Values = [290 161 154 133 126 126 118 112 112 103 102 102 101 100 96 96 92  87  87  86  85  84  84  80  79  79  76  73  71  70  70  69  65  60  59  58  57  57  56  55  54  53  53  53  53  52  51  51  50  50  50  50  49  49  49  49  49  48  47  47  47  46  45  44  44  44  44  43  42  42  41  41  40  40  40  40  40  39  39  39  38  38  38  38  37  37  37  37  37  37  37  36  36  35  35  35  35  35  34  34  34  32  32  32  32  32  31  31  31  31  31  31  31  31  31  31  30  30  30  30  30  30  30  30  29  29  29  29  29  29  29  29  28  28  28  28  28  27  27  27  27  27  26  26  26  26  26  26  26  26  26  26  25  25  25  25  25  24  24  24  24  24  23  23  23  23  23  23  23  23  23  23  22  22  22  22  22  22  21  21  21  21  21  21  21  21  21  21  21  20  20  20  20  20  20  20  19  19  19  19  19  19  19  19  19  19  19  19  19  18  18  18  18  18  18  18  18  18  18  17  17  17  17  17  17  17  17  17  17  17  17  17  16  16  16  16  16  16  16  16  16  16  16  16  16  16  16  15  15  15  15  15  15  15  15  15  15  15  15  15  15  14  14  14  14  14  14  14  14  14  14  14  13  13  13  13  13  13  13  13  13  13  13  13  13  13  13  13  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  12  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  11  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  10  9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   9   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   8   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   7   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   6   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   5   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   3   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions to understand better data clustering are appreciate!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17636" LastEditorUserId="17636" LastEditDate="2016-06-29T21:29:23.627" LastActivityDate="2017-09-23T20:27:53.933" Title="Choosing data clustering method to visualize data" Tags="|classification|clustering|visualization|data-cleaning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12499" PostTypeId="2" ParentId="12409" CreationDate="2016-06-29T21:54:50.260" Score="1" Body="&lt;p&gt;I guess you are in a situation that different fields of an address are mixed together. Try break the address text into Shingles and then try &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow&quot;&gt;Locality-sensitive Hashing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot; rel=&quot;nofollow&quot;&gt;Prof. Jeff Ullman's Text Book&lt;/a&gt; may help you put together all the techniques you need, starts with &lt;b&gt;Section 3.2&lt;/b&gt;&lt;/p&gt;&#xA;" OwnerUserId="21011" LastActivityDate="2016-06-29T21:54:50.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12500" PostTypeId="1" AcceptedAnswerId="19502" CreationDate="2016-06-29T22:53:05.027" Score="4" ViewCount="1222" Body="&lt;p&gt;What is a genetic algorithm, and what are its practical advantages over other algorithms? Is it similar to any commonly used machine learning algorithm like linear/logistic regression, neural networks, or tree-based methods like gradient boosting and random forests? I've heard it's based on &quot;mutated&quot; combinations of other models. Does this make it more like an ensemble?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2021-08-04T10:21:02.550" Title="Understanding genetic algorithms" Tags="|machine-learning|genetic-algorithms|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12501" PostTypeId="1" AcceptedAnswerId="12507" CreationDate="2016-06-30T00:26:19.793" Score="3" ViewCount="880" Body="&lt;p&gt;Articles which use the terms 'stacking' and 'Super Learner' often seem to use the terms interchangeably. Is the Super Learner algorithm a specific form of the more generic stacking concept, or is Super Learner essentially the same thing, and at some stage one of these terms is likely to become redundant?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-06-30T05:34:45.103" Title="Is SuperLearning actually different to stacking,or are they essentially the same thing?" Tags="|data-mining|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12502" PostTypeId="2" ParentId="12468" CreationDate="2016-06-30T00:34:49.577" Score="1" Body="&lt;p&gt;You could do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reshape2::recast(&#xA;  data = setNames(strsplit(language, &quot;, &quot;, T), language), &#xA;  formula = L1~value, &#xA;  fun.aggregate = length&#xA;)&#xA;#                                                                L1 Assembly Go R Ruby Rust Java Javascript SQL C C++ Haskell Matlab PHP Swift Perl Bash Python&#xA;# 1                                                          C, C++        0  0 0    0    0    0          0   0 1   1       0      0   0     0    0    0      0&#xA;# 2                                                            Java        0  0 0    0    0    1          0   0 0   0       0      0   0     0    0    0      0&#xA;# 3                                                         Java, C        0  0 0    0    0    1          0   0 1   0       0      0   0     0    0    0      0&#xA;# 4                                     Java, Javascript, Ruby, SQL        0  0 0    1    0    1          1   1 0   0       0      0   0     0    0    0      0&#xA;# 5                                                      Java, Ruby        0  0 0    1    0    1          0   0 0   0       0      0   0     0    0    0      0&#xA;# 6                                          Javascript, Ruby, Bash        0  0 0    1    0    0          1   0 0   0       0      0   0     0    0    1      0&#xA;# 7                                             Perl, Javascript, R        0  0 1    0    0    0          1   0 0   0       0      0   0     0    1    0      0&#xA;# 8  PHP, Javascript, Ruby, Assembly, Swift, R, Matlab, Go, Haskell        1  1 1    1    0    0          1   0 0   0       1      1   1     1    0    0      0&#xA;# 9                                         Python, PHP, Javascript        0  0 0    0    0    0          1   0 0   0       0      0   1     0    0    0      1&#xA;# 10                                                              R        0  0 1    0    0    0          0   0 0   0       0      0   0     0    0    0      0&#xA;# 11                                    Ruby, Assembly, R, Go, Rust        1  1 1    1    1    0          0   0 0   0       0      0   0     0    0    0      0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-06-30T00:34:49.577" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12503" PostTypeId="2" ParentId="11322" CreationDate="2016-06-30T02:19:23.140" Score="2" Body="&lt;p&gt;Pretty late but I'm surprised this wasn't answered more.  &quot;Cosine similarity&quot; is a great technique to try, though simply letting users search with a hard string and then ranking by popularity isn't so bad (e.g. &quot;dutch&quot; brings up everything with &quot;dutch&quot; in it, though I would discard mid-word matches, so &quot;ball&quot; wouldn't return &quot;football&quot;, but would return &quot;ball room dancing&quot;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say in any approach a main issue will be deduplicating previous (non-standardized) skills input by users that weren't quite standardized.  You could also try replacing the candidate skills with versions that have different synonyms substituted at search time, e.g. &quot;soccer coaching&quot; might be stored also as &quot;football coaching&quot; if most of your content is from Europeans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes extreme accuracy might not be the best goal, though... You may want to encourage users to explore new skills that they never knew existed!  Not sure what your needs are...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whatever you settle on, it might be worth building a semi-hand-crafted test set of queries and relevant results so that you can see if the performance is terrible (Google precision and recall in the context of search results).&lt;/p&gt;&#xA;" OwnerUserId="21018" LastActivityDate="2016-06-30T02:19:23.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12505" PostTypeId="2" ParentId="12500" CreationDate="2016-06-30T03:35:13.687" Score="2" Body="&lt;p&gt;For the machine learning algorithm you mentioned, regression and neural networks are formulated in optimization framework, and tree-based method is based on information gain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Genetic algorithm (GA) is a local search method. Given a value in the solution space, it will mutate to create several candidates. A criteria will be used to evaluate each candidates and only the few best ones will be kept for further mutation. It could usually be proven that GA could converge to the best solution if there is enough iteration. However, it is unfeasible practically. More discussion could be found &lt;a href=&quot;https://en.wikipedia.org/wiki/Genetic_algorithm&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for ensemble learning, it refers to (weighted) voting of multiple models. There are two normal ways, bagging or boosting. &lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2016-06-30T03:35:13.687" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12506" PostTypeId="2" ParentId="12494" CreationDate="2016-06-30T04:07:40.283" Score="4" Body="&lt;p&gt;I find two possible solutions: &lt;a href=&quot;https://en.wikipedia.org/wiki/Granger_causality&quot; rel=&quot;nofollow&quot;&gt;Granger Causality&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Convergent_cross_mapping&quot; rel=&quot;nofollow&quot;&gt;Convergent Cross Mapping (CCM)&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Granger Causality is based on the t-test of lagged value of first variable with the second variable. The limitation is that it could not eliminate the confounding effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CCM first build the manifold for the first variable X. Then for each time point, weights of neighbors are estimated, and used to estimate Y. Then correlation between estimated Y and Y are tested. &lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2016-06-30T04:07:40.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12507" PostTypeId="2" ParentId="12501" CreationDate="2016-06-30T05:34:45.103" Score="6" Body="&lt;p&gt;So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. &lt;a href=&quot;http://www.stat.berkeley.edu/~ledell/docs/dlab_ensembles.pdf&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; gives a better detailed description. Now, Ensemble Learning can be broadly divided into multiple types like &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Boosting&lt;/li&gt;&#xA;&lt;li&gt;Bagging&lt;/li&gt;&#xA;&lt;li&gt;Stacking / Super Learning&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Stacking contains a bunch of algorithms along with a learner to ensemble a group of base learners. Going deeper, the term &lt;a href=&quot;http://statistics.berkeley.edu/sites/default/files/tech-reports/367.pdf&quot; rel=&quot;noreferrer&quot;&gt;Stacking&lt;/a&gt; was used way before Super Learning. Later, when the algorithm was actually developed theoretically and made popular in 2007, it was given the name 'Super Learner'. &lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-06-30T05:34:45.103" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12508" PostTypeId="1" CreationDate="2016-06-30T06:26:58.523" Score="1" ViewCount="600" Body="&lt;p&gt;I am trying to understand following case:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;when I create new &lt;code&gt;xgbost&lt;/code&gt; DMatrix &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;xgX = xgb.DMatrix(X, label=Y, missing=np.nan)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;based on input data &lt;strong&gt;X&lt;/strong&gt;  with 64 features&#xA;&lt;a href=&quot;https://i.stack.imgur.com/JWjms.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JWjms.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I got the new DMatrix with 55 features&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6WVP3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6WVP3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What the magic is doing here? Any advise would be great!&lt;/p&gt;&#xA;" OwnerUserId="14684" LastActivityDate="2018-04-07T17:33:51.287" Title="Why initialization of Xgboost DMatrix reducec features number?" Tags="|machine-learning|python|scikit-learn|pandas|xgboost|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12509" PostTypeId="2" ParentId="12413" CreationDate="2016-06-30T07:31:43.997" Score="1" Body="&lt;p&gt;Minimize a (cost) function means that you want to find good values for its parameters. Good parameters means that the function can produce the best possible outcomes, namely the smallest ones, because small values mean &lt;em&gt;less errors&lt;/em&gt;. This is an optimization problem: the problem of finding the best solution from all possible solutions. (&lt;a href=&quot;http://www.internalpointers.com/post/linear-regression-one-variable&quot; rel=&quot;nofollow&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="14500" LastActivityDate="2016-06-30T07:31:43.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12511" PostTypeId="2" ParentId="2550" CreationDate="2016-06-30T09:37:49.973" Score="2" Body="&lt;p&gt;&lt;code&gt;Boosting&lt;/code&gt; - is the ensemble which tries to add new models that do well where previous models lack. &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html&quot; rel=&quot;nofollow&quot;&gt;Bagging&lt;/a&gt; in scikit lets you send the base classifier as the parameter. Go through 1.11 of the link to understand more!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But since, you already have in mind that SVM performs better &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html&quot; rel=&quot;nofollow&quot;&gt;Voting Classifier&lt;/a&gt; which is present in &lt;code&gt;sklearn.ensemble&lt;/code&gt; lets you give weights to the classifiers which you seem to be performing well. For instance in your question, SVM can be given more weight. It also has another parameter 'voting'. If &lt;code&gt;hard&lt;/code&gt;, uses predicted class labels for majority rule voting else if &lt;code&gt;soft&lt;/code&gt;, predicts the class label based on the argmax of the sums of the predicted probabilities.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-06-30T10:06:51.547" LastActivityDate="2016-06-30T10:06:51.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12512" PostTypeId="2" ParentId="12498" CreationDate="2016-06-30T09:50:09.513" Score="0" Body="&lt;p&gt;I believe that you should use &lt;strong&gt;Logarithmic Scale&lt;/strong&gt; for better clustering results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the photo on the following link.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Logarithmic_scale&quot; rel=&quot;nofollow&quot;&gt;wikimedia/Logarithmic_scale&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if that helped.&lt;/p&gt;&#xA;" OwnerUserId="20875" LastActivityDate="2016-06-30T09:50:09.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12514" PostTypeId="1" CreationDate="2016-06-30T11:05:59.543" Score="0" ViewCount="250" Body="&lt;p&gt;I wanted to work on twitter sentiment analysis.so before that I decided to collect some twitter data and label them on my own (pos,neg,neu). My doubt is that Should I clean the data before I label (i.e., removing RT,#,https,@ symbols) Or I can label them with out cleaning the Data? Does cleaning of data before labeling make any difference?   &lt;/p&gt;&#xA;" OwnerUserId="20585" LastActivityDate="2016-06-30T13:04:35.423" Title="Should I use a cleaned labeled data for sentiment analysis?" Tags="|machine-learning|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12516" PostTypeId="2" ParentId="12514" CreationDate="2016-06-30T12:14:55.937" Score="2" Body="&lt;p&gt;The OP asks &quot;Does cleaning of data before labeling make any difference?&quot; - &lt;strong&gt;that's an empirical question&lt;/strong&gt;... one that should be investigated by EDA of &lt;strong&gt;your&lt;/strong&gt; data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases twitter-specific conventions can be highly indicative of a specific class (e.g., &lt;strong&gt;http://&lt;/strong&gt; associated with spam/advertising tweets) or sentiment (e.g., &lt;strong&gt;:-)&lt;/strong&gt; emoticon associated with positive valence). Similarly, as I discuss in this &lt;a href=&quot;https://datascience.stackexchange.com/questions/12265/categorizing-customer-emails/12268#12268&quot;&gt;post&lt;/a&gt;, stop words can be great features to keep in certain types of text models. My support for the above answer is based on an (&lt;em&gt;unpublished&lt;/em&gt;) project I did that involved about 10 people hand-scoring 50k tweets stratified across major industries (food, travel, electronics, CPG, etc.) and brands (McDonalds, Southwest Airlines, iPhone, Tide PODs). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice, create feature extractors for any textual feature that you believe has a theoretical or logical justification for being good indicator of text polarity. Then empirically test each feature to determine whether it significantly improves classification accuracy. Keep those that are in your model; save the other feature extractors for a rainy day.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-06-30T13:04:35.423" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12517" PostTypeId="2" ParentId="8654" CreationDate="2016-06-30T14:45:58.973" Score="3" Body="&lt;p&gt;The OP reports that when a series of &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest&quot; rel=&quot;nofollow&quot;&gt;one-vs-rest&lt;/a&gt; classifiers are chained together in an ensemble from most accurate to least, the &lt;em&gt;overall&lt;/em&gt; predictive accuracy of the ensemble decreases compared to the unchained version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes perfect sense. Imagine a simpler case of 3 classes of data, &lt;strong&gt;A&lt;/strong&gt;, &lt;strong&gt;B&lt;/strong&gt;, &amp;amp; &lt;strong&gt;C&lt;/strong&gt; that are used to build the chain you describe: AvsBC, BvAC, and CvAB. Let's assume the order described is in most-to-least accurate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now run a single instance &lt;strong&gt;x&lt;/strong&gt; through this chain. Suppose classifier AvsBC assigns x a posterior probability Pr(A) = 0.51. Under this result the ensemble would presumably stop, and never explore the other options, and thus might miss out on higher posterior probability assignments (e.g., under BvAC you might get Pr(B) = 0.60). Only by running your instances through &lt;em&gt;all&lt;/em&gt; classifiers can you maximize your accuracy.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-06-30T14:52:35.393" LastActivityDate="2016-06-30T14:52:35.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12518" PostTypeId="1" CreationDate="2016-06-30T14:47:08.677" Score="2" ViewCount="64" Body="&lt;p&gt;I'm actually reading &lt;a href=&quot;http://deeplearning.net/tutorial/logreg.html#logreg&quot; rel=&quot;nofollow&quot;&gt;this tutorial&lt;/a&gt; about deepLearning and in particular about Logistic Regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't get why it first says to optimize logistic regression taking the max Probability and after using the Log loss function ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry you can explain me the point where him will use the Argmax and where Will use the Loss ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you would not need only 1 of this 2 ?&lt;/p&gt;&#xA;" OwnerUserId="21043" LastEditorUserId="11097" LastEditDate="2016-07-01T01:26:24.957" LastActivityDate="2016-07-01T01:26:24.957" Title="Two ways of optimize the same function?" Tags="|machine-learning|deep-learning|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12519" PostTypeId="1" AcceptedAnswerId="12527" CreationDate="2016-06-30T16:07:19.340" Score="0" ViewCount="91" Body="&lt;p&gt;I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(specifically STALE FILE HANDLEs for now in GPFS file system). I am categorizing the log files and clustering based on their instances per time interval. Since some messages are more predominant over the others in any given time frame than the others, i don’t want the clustering to bias towards the one with maximum variance.&lt;/p&gt;&#xA;" OwnerUserId="21045" LastEditorUserId="1330" LastEditDate="2021-10-20T22:41:02.090" LastActivityDate="2021-10-20T22:41:02.090" Title="How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?" Tags="|dataset|clustering|statistics|bigdata|hpc|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12520" PostTypeId="2" ParentId="12463" CreationDate="2016-06-30T16:12:57.760" Score="3" Body="&lt;p&gt;Have you tried without the Tf-Idf weighting? What about a bi-gram analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also review the pre processing, maybe without removing the low frequency words (wich the tf idf already does). Another idea is to remove one very high frequency word that appear on your corpora.&lt;/p&gt;&#xA;" OwnerUserId="21044" LastActivityDate="2016-06-30T16:12:57.760" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12521" PostTypeId="2" ParentId="11940" CreationDate="2016-06-30T16:23:17.233" Score="0" Body="&lt;p&gt;Since you're dealing with only 2 classes you can make a &lt;code&gt;commonality.cloud()&lt;/code&gt; from both classes (I use this function on R, I don't know about other languages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will show the words that are in common in &lt;code&gt;class1&lt;/code&gt; and &lt;code&gt;class2&lt;/code&gt;. These words maybe not helping the algorithm to distinguish the classes so you can remove these words and make some tests.&lt;/p&gt;&#xA;" OwnerUserId="21044" LastEditorUserId="21044" LastEditDate="2017-01-05T16:49:39.120" LastActivityDate="2017-01-05T16:49:39.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12522" PostTypeId="2" ParentId="9582" CreationDate="2016-06-30T16:53:00.253" Score="0" Body="&lt;p&gt;You need first to classify these comments into &lt;em&gt;pro&lt;/em&gt; and &lt;em&gt;anti&lt;/em&gt; immigrants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the next step should be analyzing your documents with text mining tools.&lt;/p&gt;&#xA;" OwnerUserId="21044" LastActivityDate="2016-06-30T16:53:00.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12523" PostTypeId="2" ParentId="12207" CreationDate="2016-06-30T16:53:51.237" Score="1" Body="&lt;p&gt;Only 32-bit Orange is currently &lt;a href=&quot;http://orange.biolab.si/download/windows/&quot; rel=&quot;nofollow&quot;&gt;available for Windows&lt;/a&gt; (the filename says win32, if you look). So your per-process RAM is limited to 4 GB. The exception says you ran out of RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use &lt;a href=&quot;https://www.continuum.io/downloads&quot; rel=&quot;nofollow&quot;&gt;64-bit Anaconda installer&lt;/a&gt; to get 64-bit Python and libraries on Windows, and then install Orange 3 there. &lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="471" LastEditDate="2016-07-01T14:36:07.653" LastActivityDate="2016-07-01T14:36:07.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12524" PostTypeId="2" ParentId="12518" CreationDate="2016-06-30T18:50:13.437" Score="2" Body="&lt;p&gt;The argmax is to get the class label prediction y_pred given a sample x. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# symbolic description of how to compute prediction as class whose &#xA; # probability is maximal&#xA; self.y_pred = T.argmax(self.p_y_given_x, axis=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Loss function is used as optimization objective function to solve the coefficients W and b based on training data. On the tutorial, the loss function is the reverse the likelihood.  &lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2016-06-30T18:50:13.437" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="12525" PostTypeId="2" ParentId="12445" CreationDate="2016-06-30T19:12:41.070" Score="0" Body="&lt;p&gt;Missing values in matlab would be replaced by value indicators.The following values, for instance, would be easily cleaned: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A period (.)&lt;/li&gt;&#xA;&lt;li&gt;NA&lt;/li&gt;&#xA;&lt;li&gt;NaN&lt;/li&gt;&#xA;&lt;li&gt;-99&lt;/li&gt;&#xA;&lt;li&gt;Empty cells&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20998" LastActivityDate="2016-06-30T19:12:41.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12526" PostTypeId="2" ParentId="12445" CreationDate="2016-06-30T20:02:42.260" Score="2" Body="&lt;p&gt;If only a small fraction of features is missing you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Imputation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;imputation&lt;/a&gt;. For more serious cases, you can use a probabilistic model such as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kriging&quot; rel=&quot;nofollow&quot;&gt;Gaussian process&lt;/a&gt;, which will let you marginalize the missing features. Or alternatively, train a neural network with &lt;a href=&quot;http://jmlr.org/papers/v15/srivastava14a.html&quot; rel=&quot;nofollow&quot;&gt;dropout regularization&lt;/a&gt;. When you have missing data, just &quot;drop out&quot; the missing connections for real. Since your network is already trained for missing features, it should work fine without them.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-06-30T20:02:42.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12527" PostTypeId="2" ParentId="12519" CreationDate="2016-06-30T20:30:50.383" Score="2" Body="&lt;p&gt;Its unclear what the OP is asking (so this response is somewhat general), but the table below illustrates common contexts and the transformations that are typical:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;sales, revenue, income, price&lt;/strong&gt; --&gt; &lt;code&gt;log(x)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;distance&lt;/strong&gt; --&gt; &lt;code&gt;1/x&lt;/code&gt;, &lt;code&gt;1/x^2&lt;/code&gt;, &lt;code&gt;log(x)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;market share, preference share&lt;/strong&gt; --&gt; &lt;code&gt;(e^x)/(1+e^x)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;right-tailed dist&lt;/strong&gt; --&gt; &lt;code&gt;sqrt(x)&lt;/code&gt;, &lt;code&gt;log(x)&lt;/code&gt; caution log(x&amp;lt;=0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;left-tailed dist&lt;/strong&gt; --&gt; &lt;code&gt;x^2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use John Tukey's &lt;strong&gt;three-point method&lt;/strong&gt; as discussed in this &lt;a href=&quot;https://stats.stackexchange.com/questions/35711/box-cox-like-transformation-for-independent-variables/35717#35717&quot;&gt;post&lt;/a&gt;.  When specific transformations don't work, use Box-Cox transformation. Use package &lt;strong&gt;car&lt;/strong&gt; to &lt;code&gt;lambda &amp;lt;- coef(powerTransform())&lt;/code&gt; to compute lambda and then call &lt;code&gt;bcPower()&lt;/code&gt; to transform. Consider Box-Cox transformations on all variables with skewed distributions before computing correlations or creating scatterplots.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-06-30T20:30:50.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12528" PostTypeId="1" CreationDate="2016-07-01T03:14:26.747" Score="0" ViewCount="283" Body="&lt;p&gt;Say I have a document and I want to assign all (or most) of the words to pre-assigned topics. So I could take a random selection of words and manually assign them to the appropriate topics, then I want an algorithm to assign the remaining words to the appropriate topics based on how closely they occur to the manually assigned phrases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an algorithm or technique that does this? I've looked into topic modelling and word2vec and they seem to assign 'topics' in some arbitrary space without any room for manual training and designation of topics.&lt;/p&gt;&#xA;" OwnerUserId="10393" LastActivityDate="2017-04-28T06:09:22.403" Title="Assign words to various topics" Tags="|machine-learning|nlp|word-embeddings|word2vec|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12530" PostTypeId="1" CreationDate="2016-07-01T09:14:41.140" Score="1" ViewCount="176" Body="&lt;p&gt;Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) on a single compute node how do we perform the multiplication?&lt;/p&gt;&#xA;" OwnerUserId="13686" LastActivityDate="2016-07-01T09:14:41.140" Title="How to multiply a &quot;fat and short&quot; matrix with a &quot;tall and thin&quot; matrix using MapReduce?" Tags="|apache-hadoop|apache-spark|map-reduce|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12531" PostTypeId="2" ParentId="2550" CreationDate="2016-07-01T09:27:40.830" Score="0" Body="&lt;p&gt;Yes, it is certainly possible to improve accuracy with SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I'd go for already prepared &lt;code&gt;ensemble&lt;/code&gt; algorithms like Random Forest or AdaBoost. They can incorporate all features and are strong predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, as mentioned from Hima Varsha answer, scikit-learn offers &lt;code&gt;voting_classifier&lt;/code&gt; to easily combine algorithms. But you can go a step further and do something what is called blending - You fit various algorithms on your data set and then, on their prediction, you fit another set of algorithms to produce final prediction. It can be easily implemented, see this &lt;a href=&quot;http://www.chioka.in/stacking-blending-and-stacked-generalization/&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-07-01T09:27:40.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12532" PostTypeId="1" AcceptedAnswerId="12533" CreationDate="2016-07-01T11:54:14.957" Score="64" ViewCount="78180" Body="&lt;p&gt;I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found out that one solution is to reduce the &lt;code&gt;batch_size&lt;/code&gt;. However, I am not sure if this parameter is only related to memory efficiency issues or if it will effect my results. As a matter of fact, I also noticed that &lt;code&gt;batch_size&lt;/code&gt; used in examples is usually as a power of two, which I don't understand either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't mind if my network takes longer to train, but I would like to know if reducing the &lt;code&gt;batch_size&lt;/code&gt; will decrease the quality of my predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="29169" LastEditDate="2019-06-09T16:50:44.093" LastActivityDate="2023-11-09T15:41:49.543" Title="Does batch_size in Keras have any effects in results' quality?" Tags="|deep-learning|keras|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12533" PostTypeId="2" ParentId="12532" CreationDate="2016-07-01T12:14:37.910" Score="77" Body="&lt;p&gt;After one and a half years, I come back to my answer because my previous answer was wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Batch size impacts learning significantly. What happens when you put a batch through your network is that you average the gradients. The concept is that if your batch size is big enough, this will provide a stable enough estimate of what the gradient of the full dataset would be. By taking samples from your dataset, you estimate the gradient while reducing computational cost significantly. The lower you go, the less accurate your esttimate will be, however in some cases these noisy gradients can actually help escape local minima. When it is too low, your network weights can just jump around if your data is noisy and it might be unable to learn or it converges very slowly, thus negatively impacting total computation time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another advantage of batching is for GPU computation, GPUs are very good at parallelizing the calculations that happen in neural networks if part of the computation is the same (for example, repeated matrix multiplication over the same weight matrix of your network). This means that a batch size of 16 will take less than twice the amount of a batch size of 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorDisplayName="user10640" LastEditDate="2020-01-07T22:02:16.177" LastActivityDate="2020-01-07T22:02:16.177" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="12534" PostTypeId="1" CreationDate="2016-07-01T13:08:13.480" Score="2" ViewCount="626" Body="&lt;p&gt;Suggestions of &lt;strong&gt;widely used and fast GPU libraries&lt;/strong&gt; that calculate the Euclidean distance of a vector from a matrix of vectors (of the same dimension) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preferably one that works on NVIDIA GPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Need it as a part of comparing solutions that evaluate Radial Basis Function Neural Networks.&lt;/p&gt;&#xA;" OwnerUserId="21075" LastEditorUserId="21075" LastEditDate="2016-07-02T10:07:40.120" LastActivityDate="2016-07-02T13:02:24.997" Title="Library to calculate Distance Metrics on GPU" Tags="|machine-learning|gpu|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12535" PostTypeId="1" CreationDate="2016-07-01T15:13:22.600" Score="3" ViewCount="554" Body="&lt;p&gt;Recently I've been doing some text mining and the stemming processing is taking a lot of time (I'm working with portuguese language).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found the &lt;code&gt;hunspell&lt;/code&gt; package, that provides spell checking, corrections and stemming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, a friend developed another stemming function that I've been studying (RSLP: Removedor de Sufixos da Língua Portuguesa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But both were made to work on a char vector, and I need to make them work with the &lt;code&gt;tm_map&lt;/code&gt; function that read a corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that to make them work, I need to split the documents on the corpus into single words, but I don't know how.&lt;/p&gt;&#xA;" OwnerUserId="21044" LastEditorUserId="21044" LastEditDate="2016-07-04T18:58:33.517" LastActivityDate="2016-07-04T18:58:33.517" Title="How to use hunspell_stem(or a ordinary function) with tm_map?" Tags="|r|text-mining|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12536" PostTypeId="1" AcceptedAnswerId="12537" CreationDate="2016-07-01T17:28:41.517" Score="0" ViewCount="45" Body="&lt;p&gt;I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. So this is a total of 57,000 choose 2 pairwise combinations. (This is a tiny example I know, but I have other databases with the same problem that are much larger.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on other analysis, I concluded that I do not want to consider people with birthdates of more than three years apart. So I can subset the database into overlapping cohorts, and then only do all pair-wise comparisons within each cohort. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy to show examples of where this cohort approach will reduce the number of pairs I need to compare. Here is one example with my data just based on the quintiles of the year of birth (and those with missing birthdays go into all cohorts).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(Min,1968]  : 13,453&#xA;[1962,1980] : 17,335&#xA;[1974,1988] : 21,188&#xA;[1982,1993] : 21,993&#xA;[1987,Max)  : 17,449&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which saves me around 0.7 billion comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this brings up two questions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;are the choosing the bins based on quantiles a good strategy, or is there anther strategy that works better?&lt;/li&gt;&#xA;&lt;li&gt;how many bins should I make?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="21079" LastActivityDate="2016-07-01T19:31:10.760" Title="How to optimize cohort sizes to reduce pair-wise comparisons?" Tags="|clustering|parallel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12537" PostTypeId="2" ParentId="12536" CreationDate="2016-07-01T18:20:26.217" Score="1" Body="&lt;p&gt;There is a better way if I understand your question correctly. Here is the algorithm I propose:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Initialize a 'window' list and a 'pairs' list&lt;/li&gt;&#xA;&lt;li&gt;Sort your data on birthday from old to young (or the other way around)&lt;/li&gt;&#xA;&lt;li&gt;Loop over your rows and keep track of all the rows that are still in the last three years since your current row. When you get to a new row, throw out rows that are now more than three years apart, add the current row together with the rows in your 'window' to your pairs set and add current row to your 'window'.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This means you only iterate over your main list once, and if you implement your 'window' list properly (like a linked list for example) you don't need to do a lot of looping in that regard either. You also get all the pairs only once as a by product. Plus you actually get all the pairs, while with your binning approach you get missing pairs around the bin edges (if you don't overlap).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the overlapping binning approach I think you should have bins of 6 years width and the borders shift 3 years each time, that way all true pairs are in at least 1 bin together, and there is the least amount of unwanted pairs. &lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-07-01T19:31:10.760" LastActivityDate="2016-07-01T19:31:10.760" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12538" PostTypeId="1" CreationDate="2016-07-01T18:51:04.507" Score="0" ViewCount="277" Body="&lt;p&gt;I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance knowledge on this subject can enlighten me. Here is where I am getting confused:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can a statistical model (such as linear regression model) built on Hadoop where data is distributed across many nodes, and where work is done only on subset of original data-set residing in each node have same or better accuracy than linear regression model built on single server where all data resides locally. Would the error-rate (accuracy) of a model build on hadoop be same, better or worse than model build on local system. I was thinking that since all data reside in local system, algorithm can be tuned to optimize error in a linear regression model. However, in case of Hadoop where data is distributed, what if only local optimization lead to drastically bad algorithm overall? Are the algorithms written separately for distributed system to take into account data distribution?&lt;/p&gt;&#xA;" OwnerUserId="21082" LastActivityDate="2016-10-29T22:47:21.730" Title="Predictive Analytics on distributed systems vs standalone system" Tags="|predictive-modeling|bigdata|apache-hadoop|accuracy|distributed|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12539" PostTypeId="1" CreationDate="2016-07-01T18:52:56.730" Score="4" ViewCount="739" Body="&lt;p&gt;I am quite new to predictive modelling but have knowledge of GIS, R, python, SQL, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently doing a project in work trying to predict when parking spaces will be available based on data received from a mobile phone application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 sql tables&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ParkingTickets&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ParkingTicketId  (Integer)&lt;/li&gt;&#xA;&lt;li&gt;ParkingAreaId    (Integer)&lt;/li&gt;&#xA;&lt;li&gt;Latitude (Integer)&lt;/li&gt;&#xA;&lt;li&gt;Longitude    (Integer)&lt;/li&gt;&#xA;&lt;li&gt;Location (Text)&lt;/li&gt;&#xA;&lt;li&gt;ParkingDate  (DateTime)&lt;/li&gt;&#xA;&lt;li&gt;DurationInMinutes    (Integer)&lt;/li&gt;&#xA;&lt;li&gt;ExpiryDate   (DateTime)&lt;/li&gt;&#xA;&lt;li&gt;Day (Text)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ParkingAreas&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ParkingAreaId (Integer)&lt;/li&gt;&#xA;&lt;li&gt;MaxSpaces (Integer)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Main assumption is that only app users can park in the 18 parking lots (both on-street and off-street lots (no multi store or underground)). I do not take in to account user preference, weather, events, etc., etc. This is purely presence and absence based on data at hand. I have researched techniques and was looking into using a birth/death model but struggling to find examples of it in use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help or pointers on models to use would be brilliant!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gTlUD.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gTlUD.jpg&quot; alt=&quot;Here is image of parking areas:&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ParkingTicketId    ParkingAreaId   Latitude    Longitude   Date    DurationInMinutes   ExpiryDate  Day&#xA;60465   302 42.56246869 -70.91313754    2014/03/07 16:36    5   2014/03/07 16:41    Friday&#xA;60466   302 42.57139883 -70.91906364    2014/03/07 16:36    23  2014/03/07 16:59    Friday&#xA;60467   302 42.54419925 -70.9417496 2014/03/07 16:36    24  2014/03/07 17:00    Friday&#xA;60472   302 42.57576595 -70.92876607    2014/03/07 16:36    16  2014/03/07 16:52    Friday&#xA;60477   302 42.55573294 -70.92912634    2014/03/07 16:36    9   2014/03/07 16:45    Friday&#xA;60479   302 42.55711998 -70.91200458    2014/03/07 16:36    19  2014/03/07 16:55    Friday&#xA;60480   302 42.58008043 -70.91559081    2014/03/07 16:37    5   2014/03/07 16:42    Friday&#xA;60485   302 42.55161223 -70.9240808 2014/03/07 16:37    21  2014/03/07 16:58    Friday&#xA;60492   302 42.58437849 -70.92764527    2014/03/07 16:37    6   2014/03/07 16:43    Friday&#xA;&#xA;ParkingAreaId   MaxSpaces&#xA;302 8&#xA;304 50&#xA;306 95&#xA;308 30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21080" LastActivityDate="2020-08-04T21:36:03.617" Title="Parking Prediction based on Mobile application" Tags="|machine-learning|python|r|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12540" PostTypeId="2" ParentId="12463" CreationDate="2016-07-01T19:52:05.360" Score="2" Body="&lt;p&gt;Feature engineering is usually the way to improve the performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible way to improve result is through supervised feature selection. I could recommend &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html&quot; rel=&quot;nofollow&quot;&gt;Chi-square feature selection&lt;/a&gt;, &lt;a href=&quot;http://www.mathworks.com/help/stats/stepwise.html&quot; rel=&quot;nofollow&quot;&gt;step-wise feature selection&lt;/a&gt;. As for a more complete survey on feature selection, you could refer to &lt;a href=&quot;http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf&quot; rel=&quot;nofollow&quot;&gt;Guyon, Isabelle, and André Elisseeff. &quot;An introduction to variable and feature selection.&quot; Journal of machine learning research 3.Mar (2003): 1157-1182.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2016-07-01T19:52:05.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12541" PostTypeId="1" CreationDate="2016-07-01T20:07:31.860" Score="2" ViewCount="241" Body="&lt;p&gt;There is a need to classify an audio stream in real-time - using one from thousands of the defined classes. Up to 5000 classes potentially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the best machine learning algorithm for this task? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;DTW? HMM? SVM? Some specific DSP algorithms?&lt;/p&gt;&#xA;" OwnerUserId="15315" LastActivityDate="2016-07-01T20:07:31.860" Title="Real-time classification of audio - thousands of classes" Tags="|classification|time-series|audio-recognition|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12542" PostTypeId="2" ParentId="12538" CreationDate="2016-07-01T20:39:43.713" Score="1" Body="&lt;p&gt;There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One general reason is that, even if the model building process were worse, building on more data could result in a better result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're asking why building many models on subsets of data could be better, but this is precisely why ensemble models like random forests can outperform single instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But lastly, it's not true that distributed algorithms always work by making an ensemble of small models, so the premise is not strictly true.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-07-01T20:39:43.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12543" PostTypeId="2" ParentId="12538" CreationDate="2016-07-01T20:41:24.353" Score="0" Body="&lt;p&gt;The answer to the question depends on the what is done on distributed environment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the goal is to estimate the mean of the data. Each machine can get one mean value and the number of samples, then a reducer machine could get a weighted average of the means, which is the true mean of all data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While for some algorithm to heuristically merge results from distributed machines, there is no guarantee on quality of the results. &lt;/p&gt;&#xA;" OwnerUserId="12237" LastActivityDate="2016-07-01T20:41:24.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12544" PostTypeId="2" ParentId="12538" CreationDate="2016-07-01T20:41:42.500" Score="0" Body="&lt;p&gt;The accuracy should be the same. Yes, the data is distributed, and yes on each node there is only a subset of the data, but that does not mean that the learning algorithm (for example gradient descent for linear regression) happens on only one node for only one of the subset/local copies of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One common technique is to perform gradient descent on each node's local subset of the data and then pass the updates back to a master node. The master aggregates incoming updates from across its workers and then sends out the updated weights to workers for continued gradient descent. When repeated numerous times, the error aggregated across all nodes eventually reaches a global minimum. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-01T20:41:42.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12546" PostTypeId="1" AcceptedAnswerId="12551" CreationDate="2016-07-01T20:50:31.827" Score="2" ViewCount="4290" Body="&lt;p&gt;I am making a recommendation system (kind of) and I have to recommend the item a user is most likely to buy in his next purchase. Doesn't matter if he already bought this item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this, I'm treating this problem as a multiclass-classification problem with 4000 categories (number of different items users can buy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Searching in Wikipedia I found &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt; and decided to use the &lt;strong&gt;One vs -rest&lt;/strong&gt; method. So I decided to train one random forest for each item using as covariates flags if the user bought each item before (so I have around 4000 covariates). Then I will decide a rule to decide the recommended item (something like the one which has the largest probability to be bought or the largest lift.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that it's taking too long to train (5 to 10 min per item): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; 5*4000&#xA;[1] 20000&#xA;&amp;gt; 20000/60&#xA;[1] 333.3333&#xA;&amp;gt; 333.3333/24&#xA;[1] 13.88889 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So in the best case it would take 2 weeks to train. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if the method i'm using is right, and if there's another faster method to achieve this.&lt;/p&gt;&#xA;" OwnerUserId="4978" LastEditorUserId="4978" LastEditDate="2016-07-06T11:55:40.227" LastActivityDate="2016-07-06T11:55:40.227" Title="Multiclass Classification with large number of categories" Tags="|machine-learning|recommender-system|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12547" PostTypeId="2" ParentId="12528" CreationDate="2016-07-01T20:54:25.937" Score="0" Body="&lt;p&gt;There are techniques like singular value decomposition (SDV) and principal components analysis (PCA) that when applied to many documents (or many sentences in your case if you have only one document) can produce vectors of words that occur together. These vectors would then represent the topics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVD and PCA are commonly used text analysis techniques that have been shown to be capable of identifying different sections/topics of newspapers (e.g., sports, finance, world, etc) when applied to many news articles. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-07-01T21:49:04.760" LastActivityDate="2016-07-01T21:49:04.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12548" PostTypeId="2" ParentId="12539" CreationDate="2016-07-01T21:23:33.420" Score="1" Body="&lt;p&gt;If this data is coming in real-time then you don't need a model -- simply check how many spots have an ExpiryDate greater than now (i.e. when you need to provide a prediction) and subtract this from the total capacity of spots.&lt;/p&gt;&#xA;&lt;p&gt;If the data is not coming in real-time, then you could use time of day and day of week as predictors. You might even want to make them into interaction terms. You would also need to decide how often you want to call your model and group your data so that each row represents how many tickets were active during that timeframe; this will define your target variable (what you're trying to predict).&lt;/p&gt;&#xA;&lt;p&gt;By the way, I think you're referring to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Proportional_hazards_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;survival model&lt;/a&gt;. I would recommend gradient boosting instead; it's much more powerful. Gradient boosting models (GBM) are part of caret in R and scikit-learn in Python by the way.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="98307" LastEditDate="2020-08-04T21:36:03.617" LastActivityDate="2020-08-04T21:36:03.617" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="12549" PostTypeId="1" AcceptedAnswerId="12550" CreationDate="2016-07-01T21:25:33.887" Score="3" ViewCount="298" Body="&lt;p&gt;I was investigating scikit-learn's implementation of the EM algorithm for fitting Gaussian Mixture Models and I was wondering how they did come up with using the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/mixture/gmm.py#L515&quot; rel=&quot;nofollow&quot;&gt;average log likelihood&lt;/a&gt; instead of the sum of the log likelihoods to test convergence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see that it should cause the algorithm to converge faster (given their default parameters), but where does that idea come from ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know if they based this part of the implementation on a specific paper or if they just came up with it and used it ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most explanations of the EM algorithm I have come across, they would have used log_likelihoods.sum() instead of log_likelihoods.mean().&lt;/p&gt;&#xA;" OwnerUserId="21086" LastEditorUserId="5177" LastEditDate="2016-07-03T12:30:18.200" LastActivityDate="2016-07-03T12:30:18.200" Title="Gaussian Mixture Models EM algorithm use average log likelihood to test convergence" Tags="|machine-learning|scikit-learn|gaussian|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12550" PostTypeId="2" ParentId="12549" CreationDate="2016-07-01T21:36:17.267" Score="4" Body="&lt;p&gt;It makes unit testing easier; invariant to the size of the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/4178&quot;&gt;the github discussion that led to the change&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-01T21:36:17.267" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12551" PostTypeId="2" ParentId="12546" CreationDate="2016-07-01T22:29:21.057" Score="2" Body="&lt;p&gt;You might have more luck with a Naive Bayes Classifier.  It can handle a large number of target classes, and is relatively fast to train, since you largely just calculate a bunch of univariate stats to plug into at prediction time.  It won't capture fancy interactions as much as a random forest though, so if you are concerned about &quot;they only buy shoelaces if they bought shoes but NOT shoeshine&quot; vs &quot;they often buy shoelaces if they've bought shoes&quot; then it may disappoint.  You may also want to incorporate a time component, but I'm not sure what you're doing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Association_rule_learning&lt;/a&gt; may also be relevant.&lt;/p&gt;&#xA;" OwnerUserId="21018" LastActivityDate="2016-07-01T22:29:21.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12552" PostTypeId="2" ParentId="6778" CreationDate="2016-07-01T23:06:04.880" Score="1" Body="&lt;p&gt;Late answer!  Sorry if I am repeating things you know...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure somebody has worked on this formally, but least in my short industry experience, it might just be easier to roll your own solution here.  I recently had to do this for classifying businesses (into restaurants, or dentists, or doctors, or amusement parks etc). Because calling a dentist a doctor was a relatively minor error compared to calling a doctor an amusement park, certainly all mistakes are not equal.  My colleague decided to make a different ROC curve for each true class, and call things a &quot;hit&quot; if the classification was right, and a &quot;miss&quot; if the classification was wrong, and so he could leverage typical scikit-learn stuff, but it still only told part of the picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With all that in mind, we can still be quantitative.  Perhaps you could decide on a misclassification matrix pentaly, where cell (i,j) is the penalty for calling an &quot;i&quot; a &quot;j&quot;, with 0 penalties on the diagonal for correct classifications.  Then you could at least compare models (at one particular set of thresholds) by their cumulative error or squared error etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's already a little tough striking a tradeoff between false positives and false negatives in a binary classification problem - and it depends very heavily on your subject matter and what risks you are willing to tolerate. Do you want to be able to say &quot;we aren't confident and therefore we won't guess&quot;?  This complicated things further for me, because we were trading off accuracy with automation, so &quot;not sures&quot; were also unfortunate, but in a different sense.&lt;/p&gt;&#xA;" OwnerUserId="21018" LastActivityDate="2016-07-01T23:06:04.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12553" PostTypeId="1" AcceptedAnswerId="12555" CreationDate="2016-07-02T02:49:50.237" Score="0" ViewCount="1105" Body="&lt;p&gt;I started using the below link to teach myself data science, with some mathematical knowledge (including calculus, linear algebra, stats/probability) but very little programming experience:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-should-I-learn-in-data-science-in-100-hours&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora - Roman Trusov's answer on how to learn Data Science in 100 hours&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The above link has the following starting advice:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;You will need an RDBMS to handle the data, so the first day would look like this:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Install and configure MySQL. Import the dump into the database&lt;/li&gt;&#xA;&lt;li&gt;Read SQL basics. Spend some time doing simple exercises to get the hang of manipulating the data...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;After a few days, I'm still stuck at this stage. I'd like to learn how to do data science, and was hoping that following the plan in the above link would achieve that quickly. But I'm starting to feel it is not working. Is debugging MySQL installations an important part of becoming a data scientist - so I should just get advice on that and try harder? If not, what can I do differently to achieve my goals of learning data science in similar time frame?&lt;/p&gt;&#xA;" OwnerUserId="21090" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-07-03T07:57:05.147" Title="Beginner in programming and data science with 100 hours to spend learning the basics" Tags="|beginner|" AnswerCount="1" CommentCount="7" ClosedDate="2016-07-02T09:39:58.813" ContentLicense="CC BY-SA 3.0" />
  <row Id="12554" PostTypeId="1" AcceptedAnswerId="12597" CreationDate="2016-07-02T07:30:13.713" Score="59" ViewCount="59819" Body="&lt;p&gt;I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, for example: a variable (feature) $A$ also appears as $\frac{B}{A}$ and $\frac{C}{A}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How (&lt;em&gt;If?&lt;/em&gt;) do Boosted Decision Trees handle multicollinearity? &lt;/li&gt;&#xA;&lt;li&gt;How would the existence of multicollinearity affect prediction if it is not handled?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From what I understand, the model is learning more than one tree and the final prediction is based on something like a &quot;weighted sum&quot; of the individual predictions. So if this is correct, then Boosted Decision Trees &lt;em&gt;should&lt;/em&gt; be able to handle co-dependence between variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, on a related note - how does the variable importance object in XGBoost work?&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="5177" LastEditDate="2016-07-02T13:16:51.630" LastActivityDate="2021-06-09T05:47:46.203" Title="Does XGBoost handle multicollinearity by itself?" Tags="|feature-selection|correlation|xgboost|gbm|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12555" PostTypeId="2" ParentId="12553" CreationDate="2016-07-02T08:03:48.717" Score="9" Body="&lt;p&gt;Being good at a subject does not automatically make someone a good teacher, and it looks like Roman's answer on Quora has fallen into a trap of thinking everything he knows is simple and could be picked up quickly. Also making the potential student attempt things outside of a beginner level - effectively just by research on the web following a 2 paragraph pointer - is going to make progress slow and frustrating. Despite no doubt good intentions, the advice there is likely to give a very poor learning experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are full structured tutorials available on free Massive Online Open Courses (MOOCs) available for learning data science topics. These have been put together by professionals who know how to teach, and the effort put into designing any of these courses, providing materials etc, totally dwarfs the effort that went into the advice on Quora.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ignore the Quora answer, and sign up for one or more of these MOOCs. You can find them at &lt;a href=&quot;https://www.coursera.org/&quot; rel=&quot;noreferrer&quot;&gt;Coursera&lt;/a&gt;, &lt;a href=&quot;https://www.udacity.com/&quot; rel=&quot;noreferrer&quot;&gt;Udacity&lt;/a&gt;, &lt;a href=&quot;https://www.edx.org/&quot; rel=&quot;noreferrer&quot;&gt;edX&lt;/a&gt; and other similar places. If you want to cram a lot of study into a short time, look for &quot;at your own pace&quot; courses where all the materials are available to you immediately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The almost canonical start point to test the waters would be &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;noreferrer&quot;&gt;Andrew Ng's Machine Learning course on Coursera&lt;/a&gt;. Machine Learning is one of the more immediately accessible and fun parts of data science, and the course goes into theory and practice with additional sections for beginners at the start covering necessary maths and programming. Following that entire course is probably around 100 hours total effort. You won't come out of it knowing data science, but you will gain useful practical skills, and get a real taster for the machine learning side of the subject.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-07-03T07:57:05.147" LastActivityDate="2016-07-03T07:57:05.147" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12556" PostTypeId="2" ParentId="12534" CreationDate="2016-07-02T13:02:24.997" Score="1" Body="&lt;p&gt;If you are using python, use CUDA tool-kit with cudamat library. You will be able to perform matrix operations on GPU using cudamat. &lt;/p&gt;&#xA;" OwnerUserId="20982" LastActivityDate="2016-07-02T13:02:24.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12557" PostTypeId="1" AcceptedAnswerId="12559" CreationDate="2016-07-02T16:28:58.853" Score="7" ViewCount="292" Body="&lt;p&gt;Why does it seem that it's difficult to find out how people in data science create measurable value?  All I find on the internet are buzzwords like data cleaning, visualization, and writing about the data.  This is equivalent of describing a landscaper as trimming trees and grass or saying an investment banker spreads M&amp;amp;A.  That is insufficient.  Let my type out an insufficient answer:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A data scientist uses database querying languages like SQL as well as statistical programming languages like R to design experiments with data and visualizing such experimental conclusions to decision-makers in the hopes of making better decisions and thus creating value for a company/cause.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How is this insufficient?  Well, how do we know that data scientists' recommendations actually create value?  If we can't measure the value of our analyses, then how can we determine if hundreds of hours of learning/working in the subject is meaningful?  Extreme example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;George is a data scientist working for Company X.  After hundreds of hours of data cleaning/experimentation, he concludes Decision A will benefit Company X.  George convinces a Product Manager to apply Decision A that ultimately increases the revenues of Company X by $0.01&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do we know that several years of work may not amount to basically nothing?&lt;/strong&gt;  Let me give you a depressing and &lt;strong&gt;real/plausible&lt;/strong&gt; example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Instead of becoming a data scientist, George becomes a portfolio manager, managing investment portfolios for clients.  After 20 years of management, George has a track record of -1% per annum, compared to an S&amp;amp;P 500 return of 8%.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the above example, George is a useless person and has lost value for people.  How do data scientists know that they aren't destroying value, and if we can't figure that out, what are the steps to avoid value destruction and create the most value for companies? &lt;/p&gt;&#xA;" OwnerUserId="21090" LastActivityDate="2016-07-02T18:25:35.517" Title="Measuring Value in Data Science?" Tags="|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12559" PostTypeId="2" ParentId="12557" CreationDate="2016-07-02T18:25:35.517" Score="4" Body="&lt;p&gt;I read your question as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;tl;dr. How do you know that a data scientist adds value to a company?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The job of a data scientist is to pick out the best data set, scrub it, fit a statistical model. And most importantly, a data scientist needs to know the problem statement and why is the analysis being done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider a DS team in an e-commerce company. Their primary (let's assume) is to build a highly effective recommender system, which needs to adapt itself to a lot of variables, like the user behaviour, the session-based behaviour, user segments, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if the recommender system is still not being able to get the company a considerable CTR or upsells/cross-sells, then maybe the team is not adding proper value to the company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a data scientist needs to have a keen knowledge of the business value which an experiment is expected to add to the company. The decisions made from the experiments should ultimately add value to a company (monetary or otherwise). If not, then just like any other employee, it means that the person have failed in the job.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-07-02T18:25:35.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12560" PostTypeId="5" CreationDate="2016-07-02T18:33:53.977" Score="0" Body="&lt;p&gt;An &lt;a href=&quot;https://en.wikipedia.org/wiki/Outlier&quot; rel=&quot;nofollow&quot;&gt;outlier&lt;/a&gt; is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, outliers are not necessarily bad or wrong, nor do they need to be removed from data for further analysis.  However, outliers (of which there can be more than one in any set of data) indicate that some data at least appear to differ from the bulk of the dataset, suggesting they should be individually examined and understood.  Also, some statistical procedures are sensitive to outliers: this means that removal of one or more outliers could substantially change the conclusions of those procedures.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-03T13:11:31.540" LastActivityDate="2016-07-03T13:11:31.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12561" PostTypeId="4" CreationDate="2016-07-02T18:33:53.977" Score="0" Body="For questions regarding outliers or unusual points in the data." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-03T13:09:57.877" LastActivityDate="2016-07-03T13:09:57.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12562" PostTypeId="5" CreationDate="2016-07-02T18:40:26.763" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;Scikit-Learn&lt;/a&gt; is an open source package which provides simple and efficient tools for data mining and data analysis. It is built on NumPy, SciPy, and matplotlib&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-03T13:09:53.733" LastActivityDate="2016-07-03T13:09:53.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12563" PostTypeId="4" CreationDate="2016-07-02T18:40:26.763" Score="0" Body="Regarding the Python data analysis and machine learning package, scikit-learn." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-03T13:11:34.477" LastActivityDate="2016-07-03T13:11:34.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12564" PostTypeId="1" AcceptedAnswerId="12566" CreationDate="2016-07-03T07:21:29.737" Score="2" ViewCount="144" Body="&lt;p&gt;I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Frame the problem&lt;/li&gt;&#xA;&lt;li&gt;Collect the data&lt;/li&gt;&#xA;&lt;li&gt;Clean the data&lt;/li&gt;&#xA;&lt;li&gt;Work on the data&lt;/li&gt;&#xA;&lt;li&gt;Report the results&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm struggling to connect the dots when comes to work on the data. I'm aware that step 4 is where the fun happens, but I don't know where to begin. What are the steps taken when you work on the data? Example: do I need to find the central tendency or the standard deviation? Is machine learning needed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ps: I know these are broad questions, so please answer it within your own domain expertise. &lt;/p&gt;&#xA;" OwnerUserId="18919" LastActivityDate="2016-07-03T09:56:19.353" Title="How do you define the steps to explore the data?" Tags="|machine-learning|statistics|visualization|data-wrangling|methods|" AnswerCount="2" CommentCount="1" ClosedDate="2016-07-03T13:12:05.150" ContentLicense="CC BY-SA 3.0" />
  <row Id="12565" PostTypeId="2" ParentId="12564" CreationDate="2016-07-03T08:47:43.217" Score="2" Body="&lt;p&gt;That is a very good framework of solving a question you have. According to me, it has multiple answers. I'll give you the one which I relate to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After cleaning the data or rather while cleaning it, we have to be clear about the task ahead and about our results. &#xA;The work on the data follows the below steps mostly:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Feature Detection&lt;/li&gt;&#xA;&lt;li&gt;Training using the above features(there are many Machine Learning/Deep learning models to do this) like Classification (this depends on the task)&lt;/li&gt;&#xA;&lt;li&gt;Check the trained model on validation test if needed and later on test set &#xA;The features depend on the dataset. Standard deviation or to find central tendency isn’t always a criteria. Machine Learning is needed to train on the dataset in most of the cases.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-03T08:47:43.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12566" PostTypeId="2" ParentId="12564" CreationDate="2016-07-03T09:56:19.353" Score="2" Body="&lt;p&gt;As far as working with data depends on one's education, expertise, goal and favorite tools, I would answer it within my narrow scope - and trying to keep your track.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Framing the problem&lt;/strong&gt; is an important starting point a lot of people neglect. Even-though it is only the beginning, this should result in first strategies to explore the data. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Translate &lt;em&gt;&quot;What I want to do&quot;&lt;/em&gt; to &lt;em&gt;&quot;What are the implicit information I need to have to achieve it&quot;&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Given the information you need, find your way to get it (by decomposing it into tasks and sub-tasks) and the corresponding data to extract it (specific task implies specific signal(s) : structured data, pictures, movies, sounds, texts ...) &lt;/li&gt;&#xA;&lt;li&gt;Along with 1. and 2., you should have a clearer idea of the data you'll deal with and thus the tools you might use (NLP, image processing, time-series, ...)&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Collecting the data&lt;/strong&gt; is now easier as it is a implied by the previous task. However, classify mentally your data in the following graph to know what to start with according to your personal trade-off: &lt;a href=&quot;https://i.stack.imgur.com/87vAS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/87vAS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Direct data are those that can be obtained easily. Indirect are those that require some pre-process (scrapping websites, cropping images, counting number of clicks, ...)&lt;/li&gt;&#xA;&lt;li&gt;The simplicity / complexity of use depends on the data : generally speaking, structured data within arrays are easier to deal with that images.&lt;/li&gt;&#xA;&lt;li&gt;The size of the dot is the reward obtained if you achieve to work on these data, regarding your whole project&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Exploring and Cleaning the data&lt;/strong&gt; : there are levels of complexity here. I usually start with standard processes to clean the data (mean/median for missing values, normalization and centering when needed, ...). Meanwhile, I start looking deeper into the data by getting histograms of values, evolution of the mean for time series, word frequency for texts, ... This is task specific but exploration is here to give you hints about your data. Once inspecting them, you should mature your cleaning process.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Working on the data&lt;/strong&gt; : As you said, here comes the fun part. You can choose your favorite tools, or start improving your skills by looking for new concepts ( as a future good data scientist ), to process your data. One reason you don't know what to start with may be that you went too fast over the previous dots - implying what you have to do is still unclear. Get back to them, write the process on a paper until you clearly identify the inputs and outputs you need. Again, generally speaking, it involves the following :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dimensionality reduction (especially for images) and feature design (one-hot encoder, floats or ints, ordinal or cardinal category, ...)&lt;/li&gt;&#xA;&lt;li&gt;Choose of your estimator / model by tuning the hyper-parameters&lt;/li&gt;&#xA;&lt;li&gt;Training with validation methods (cross-validation, Leave One Out, ...)&lt;/li&gt;&#xA;&lt;li&gt;Testing and improving your results&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Report the results&lt;/strong&gt;. Not as easy as it sounds, as mentioned &lt;a href=&quot;https://datascience.stackexchange.com/questions/12557/measuring-value-in-data-science&quot;&gt;here&lt;/a&gt;. If is it for your own, having a whole project, started from scratch, is a good reward. Moreover, you may remember your scores when testing the model and how you improve it (which hyperparameters, which model, ...). If it is a well-discussed subject, you can compare to top teams in the world on well-known datasets. Finally, if it is for an employer, I would recommend to start this discussion before getting into the subject - would same time and trouble.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20759" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-03T09:56:19.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12568" PostTypeId="1" AcceptedAnswerId="12641" CreationDate="2016-07-03T14:37:51.930" Score="4" ViewCount="2404" Body="&lt;p&gt;Using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm&quot; rel=&quot;nofollow&quot;&gt;Smith-Waterman-Gotoh algorithm&lt;/a&gt; I want to get an overall similarity percentage between two sequences. What would be the best way to do this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg. comparing strings &lt;code&gt;COELACANTH&lt;/code&gt; and &lt;code&gt;PELICAN&lt;/code&gt; &lt;a href=&quot;http://etutorials.org/Misc/blast/Part+II+Theory/Chapter+3.+Sequence+Alignment/3.2+Local+Alignment+Smith-Waterman/&quot; rel=&quot;nofollow&quot;&gt;in this example&lt;/a&gt; gives a score of 4 with alignment: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ELACAN&#xA;ELICAN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would I then go an determine the overall similarity percentage between &lt;code&gt;COELACANTH&lt;/code&gt; and &lt;code&gt;PELICAN&lt;/code&gt; based on this? &lt;/p&gt;&#xA;" OwnerUserId="21120" LastEditorUserId="21120" LastEditDate="2016-07-06T05:46:45.317" LastActivityDate="2016-07-06T23:28:21.950" Title="Smith-Waterman-Gotoh Algorithm - how to determine an overall similarity percentage" Tags="|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12570" PostTypeId="1" AcceptedAnswerId="12571" CreationDate="2016-07-03T19:06:39.343" Score="2" ViewCount="1066" Body="&lt;p&gt;I've found a few similar questions, but I am new to R and can't figure out how it applies to my specific problem. Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(rvest)&#xA;library(plyr)&#xA;library(stringr)&#xA;&#xA;#function passes in letter and extracts bold text from each page&#xA;fetch_current_players&amp;lt;-function(letter){&#xA;  url&amp;lt;-paste0(&quot;http://www.baseball-reference.com/players/&quot;, letter, &quot;/&quot;)&#xA;  urlHTML&amp;lt;-read_html(url)&#xA;  playerData&amp;lt;-html_nodes(urlHTML, &quot;b a&quot;)&#xA;  player&amp;lt;-html_text(playerData)&#xA;  player&#xA;}&#xA;&#xA;#list of letters to pass into function&#xA;atoz&amp;lt;-c(&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;d&quot;,&quot;e&quot;,&quot;f&quot;,&quot;g&quot;,&quot;h&quot;,&#xA;        &quot;i&quot;,&quot;j&quot;,&quot;k&quot;,&quot;l&quot;,&quot;m&quot;,&quot;n&quot;,&quot;o&quot;,&quot;p&quot;,&quot;q&quot;,&quot;r&quot;,&#xA;        &quot;s&quot;,&quot;t&quot;,&quot;u&quot;,&quot;v&quot;,&quot;w&quot;,&quot;x&quot;,&quot;y&quot;,&quot;z&quot;)&#xA;player_list&amp;lt;-ldply(atoz, fetch_current_players, .progress=&quot;text&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So what this code is trying to do is use the URL structure of this website to pass a list of the letter A through Z into my function to produce a list of names that are in bold. I think the problem is that each list of players it returns is of different lengths and that is producing an error as when I manually type in each letter into the function the function appears to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="21124" LastEditorUserId="471" LastEditDate="2016-08-30T15:53:58.120" LastActivityDate="2016-08-30T15:53:58.120" Title="&quot;Results do not have equal lengths&quot; using ldply in R package plyr" Tags="|r|scraping|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12571" PostTypeId="2" ParentId="12570" CreationDate="2016-07-03T19:41:22.530" Score="1" Body="&lt;p&gt;Here's a slightly modified version using some newer &quot;tidyverse&quot; packages:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(rvest) &#xA;library(purrr) # flatten/map/safely&#xA;library(dplyr) # progress bar&#xA;&#xA;# just in case there isn't a valid page&#xA;safe_read &amp;lt;- safely(read_html)&#xA;&#xA;fetch_current_players &amp;lt;- function(letter){&#xA;&#xA;  URL &amp;lt;- sprintf(&quot;http://www.baseball-reference.com/players/%s/&quot;, letter)&#xA;  pg &amp;lt;- safe_read(URL)&#xA;&#xA;  if (is.null(pg$result)) return(NULL)&#xA;&#xA;  player_data &amp;lt;- html_nodes(pg$result, &quot;b a&quot;)&#xA;&#xA;  html_text(player_data)&#xA;&#xA;}&#xA;&#xA;pb &amp;lt;- progress_estimated(length(letters))&#xA;player_list &amp;lt;- flatten_chr(map(letters, function(x) {&#xA;  pb$tick()$print()&#xA;  fetch_current_players(x)&#xA;}))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="710" LastActivityDate="2016-07-03T19:41:22.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12575" PostTypeId="1" AcceptedAnswerId="12583" CreationDate="2016-07-04T06:00:34.093" Score="23" ViewCount="86160" Body="&lt;p&gt;I'm looking for a Python library that helps me identify the similarity between two words or sentences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be doing Audio to Text conversion which will result in an English dictionary or non dictionary word(s) ( This could be a Person or Company name) After that, I need to compare it to a known word or words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Text to audio result: &lt;strong&gt;Thanks for calling America Expansion&lt;/strong&gt; &#xA;will be compared to &lt;strong&gt;American Express&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both sentences are somehow similar but not the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looks like I may need to look into how many chars they share. Any ideas will be great. Looks a functionality like Google search &quot;did you mean&quot; feature.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-07-04T08:41:19.707" LastActivityDate="2021-04-08T20:39:17.613" Title="Similarity between two words" Tags="|nlp|nltk|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12576" PostTypeId="5" CreationDate="2016-07-04T06:39:27.880" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-07-04T06:39:27.880" LastActivityDate="2016-07-04T06:39:27.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12577" PostTypeId="4" CreationDate="2016-07-04T06:39:27.880" Score="0" Body="For questions related to the eXtreme Gradient Boosting algorithm." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-04T09:14:59.470" LastActivityDate="2016-07-04T09:14:59.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12578" PostTypeId="5" CreationDate="2016-07-04T06:41:50.337" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-07-04T06:41:50.337" LastActivityDate="2016-07-04T06:41:50.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12579" PostTypeId="4" CreationDate="2016-07-04T06:41:50.337" Score="0" Body="For question regarding distance between distributions or variables, such as Euclidean distance between points in n-space." OwnerUserId="11097" LastEditorUserId="21" LastEditDate="2016-07-04T09:14:32.423" LastActivityDate="2016-07-04T09:14:32.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12580" PostTypeId="5" CreationDate="2016-07-04T06:43:34.233" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-07-04T06:43:34.233" LastActivityDate="2016-07-04T06:43:34.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12581" PostTypeId="4" CreationDate="2016-07-04T06:43:34.233" Score="0" Body="For questions regarding &quot;Convolutional Neural Networks&quot; (CNN)" OwnerUserId="11097" LastEditorUserId="21163" LastEditDate="2018-03-19T23:32:38.313" LastActivityDate="2018-03-19T23:32:38.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12582" PostTypeId="2" ParentId="12575" CreationDate="2016-07-04T07:43:55.030" Score="6" Body="&lt;p&gt;If your dictionary is not too big a common approach is to take the Levenshtein distance, which basically counts how many changes you have to make to get from one word to another. Changes include changing a character, removing a character or adding a character. An example from &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;lev(kitten, sitting) = 3&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;k&lt;/strong&gt; itten -&gt; &lt;strong&gt;s&lt;/strong&gt; itten&lt;/li&gt;&#xA;&lt;li&gt;sitt &lt;strong&gt;e&lt;/strong&gt; n -&gt; sitt &lt;strong&gt;i&lt;/strong&gt; n&lt;/li&gt;&#xA;&lt;li&gt;sittin -&gt; sittin &lt;strong&gt;g&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; are some Python implements on Wikibooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm to compute these distances is not cheap however. If you need to do this on a big scale there are ways to use cosine similarity on bi-gram vectors that are a lot faster and easy to distribute if you need to find matches for a lot of words at once. They are however only an approximation to this distance.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-04T07:43:55.030" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12583" PostTypeId="2" ParentId="12575" CreationDate="2016-07-04T08:26:56.493" Score="20" Body="&lt;p&gt;The closest would be like Jan has mentioned inhis answer, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;noreferrer&quot;&gt;Levenstein's distance&lt;/a&gt; (also popularly called the edit distance).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In information theory and computer science, the Levenshtein distance&#xA;  is a string metric for measuring the difference between two sequences.&#xA;  Informally, the Levenshtein distance between two words is the minimum&#xA;  number of single-character edits (i.e. insertions, deletions or&#xA;  substitutions) required to change one word into the other.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is a very commonly used metric for identifying similar words. Nltk already has an implementation for the edit distance metric, which can be invoked in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import nltk&#xA;nltk.edit_distance(&quot;humpty&quot;, &quot;dumpty&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above code would return &lt;code&gt;1&lt;/code&gt;, as only one letter is different between the two words.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-07-04T08:26:56.493" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12584" PostTypeId="1" CreationDate="2016-07-04T10:09:06.210" Score="3" ViewCount="222" Body="&lt;p&gt;I'm collecting Twitter tweets for sentiment analysis. I chose to use Multinomial Navies theorem for finding the sentiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found some examples of SVM theorem making use of stop words. My question is: can we apply stop words in Multinomial Cavies theorem or not?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="29169" LastEditDate="2019-06-07T06:26:25.167" LastActivityDate="2019-06-07T06:26:25.167" Title="Can we use Stop Words while using multinomial navies theorem?" Tags="|machine-learning|nlp|svm|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12585" PostTypeId="1" CreationDate="2016-07-04T11:11:42.970" Score="3" ViewCount="959" Body="&lt;p&gt;Hello I am building a python program for text spinning. I have read about GloVe, word2vec, doc2vec or text2vec. I understand that what they do is to represent each word as a semantic vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess that if I train a GloVe I could use it just for finding synonyms, or something more?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I have come up with one strategy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First I train a doc2vec model. The goal would be to build semantic vectors of phrases&lt;/li&gt;&#xA;&lt;li&gt;Then I use my original paragraph input. Word by word I change it randomly with a synonym (provided by nltk or other synonym db). Before finally spinning one word I compute the doc2vec for the changed paragraph and compute the distance with the original paragraph vectorized. I only change the synonym if the distance is small.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How does that sound to you?&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="17484" LastEditDate="2016-07-04T14:00:11.190" LastActivityDate="2018-06-11T08:40:34.637" Title="How can I use GloVe for text spinning?" Tags="|text-mining|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12588" PostTypeId="2" ParentId="12575" CreationDate="2016-07-04T14:02:30.740" Score="4" Body="&lt;p&gt;An old and well-known technique for comparison is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Soundex&quot; rel=&quot;nofollow&quot;&gt;Soundex&lt;/a&gt; algorithm.&#xA;The idea is to compare not the words themselves but approximations of how they are pronounced.  To what extent this actually improves the quality of the results I don't know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However it feels a bit strange to apply something like Soundex to results from a speech-to-text recognition engine.  First you throw away information about how the words are pronounced, then you try to add it back again.  It would be better to combine these two phases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I expect the state of the art technology in this area to do that, and be some form of adaptive classification, e.g. based on neural networks.  Google does return recent research on &lt;a href=&quot;https://www.google.nl/search?hl=en&amp;amp;q=peech%20recognition%20neural%20networks&quot; rel=&quot;nofollow&quot;&gt;Speech Recognition with Neural Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="7925" LastActivityDate="2016-07-04T14:02:30.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12589" PostTypeId="1" AcceptedAnswerId="12590" CreationDate="2016-07-04T14:05:05.727" Score="5" ViewCount="840" Body="&lt;p&gt;I am on a binary classification problem with the AUC metrics. I did a random split 70%, 30% for training and test sets.&#xA;My first attempts using random forest with default hyper-parameters gave me auc 0.85 on test set and 0.96 on training set. &#xA;So, the model overfits. But the score of 0.85 is good enough for my business.&#xA;I also did a 5-folds cross validation with the same model and same hyper-parameters and the test set results were consistently something between 0.84 and 0.86 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: can I believe on the score 0.85 and use this model in production? &lt;/p&gt;&#xA;" OwnerUserId="21145" LastActivityDate="2016-07-04T15:18:52.713" Title="Can we use a model that overfits?" Tags="|machine-learning|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12590" PostTypeId="2" ParentId="12589" CreationDate="2016-07-04T14:14:26.000" Score="4" Body="&lt;p&gt;Yes, if your 0.85 AUC is good enough for your use case this is a good enough model. The performance on the training set indicates how well your model knows the training set. This we don't really care about, it's just what the model tries to optimize. The performance on the test set is an indication on how well your model generalizes. This is what we care about, and your model gets to around 0.85 as an estimate for your generalization. Differences between training and testing are the norm and in this case it could be that you might get a better performance by adding stronger regularization but if 0.85 is good enough, go for it!&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-07-04T14:50:12.737" LastActivityDate="2016-07-04T14:50:12.737" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12591" PostTypeId="1" CreationDate="2016-07-04T14:20:03.507" Score="3" ViewCount="171" Body="&lt;p&gt;I'm trying to cluster an unknown set of data with a replicator neural network.&#xA;The number of clusters is determined by the number of neuron units in the middle layer, multiplied by the number of steps in our step function responsible for the middle layer activation.&#xA;The only information provided, is that the feature size vector will be 10. However there are no information about the number of possible clusters.&#xA;The final goal is to detect anomalies with the neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I approach this task?&#xA;I'm not sure how much neurons and steps I should be using. I believe I can limit the neurons to about 4, as it should be usually less than half the input neurons. However I'm not sure how the network is going to perform on a lot of steps.&#xA;Perhaps to many neurons won't hurt, as for my understand neural networks cluster, compared to k-means, independent from the number of clusters, while a number to small might damage the anomaly detection by limitation?&lt;/p&gt;&#xA;" OwnerUserId="18744" LastActivityDate="2016-07-04T14:20:03.507" Title="Clustering with Replicator Neural Network" Tags="|neural-network|anomaly-detection|clustering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12592" PostTypeId="2" ParentId="12589" CreationDate="2016-07-04T15:18:52.713" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My first attempts [...] gave me auc 0.85 on test set and 0.96 on training set. So, the model overfits.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not quite true. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See, (&lt;em&gt;almost&lt;/em&gt;) each estimator will have a better prediction score on the training data than on the testing data. It doesn't mean each estimator overfit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is normal though to habe a better score on the training set, as the estimator is &lt;em&gt;built&lt;/em&gt; on it, meaning its parameters are fitted thanks to it. However, your estimator can fit your training data more or less. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take your Random-Forest example. If the depth is too high, you'll fit way to much to the training data : &lt;em&gt;&lt;strong&gt;overfit&lt;/strong&gt;&lt;/em&gt;. If the depth is not high enough, it will be hard to generalize to other data : you &lt;em&gt;&lt;strong&gt;underfit&lt;/strong&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Underfitting : &lt;code&gt;0.96&lt;/code&gt; on train set &amp;amp; &lt;code&gt;0.82&lt;/code&gt; on test set&lt;/li&gt;&#xA;&lt;li&gt;Possible good fitting : &lt;code&gt;0.96&lt;/code&gt; on train set &amp;amp; &lt;code&gt;0.89&lt;/code&gt; on test set&lt;/li&gt;&#xA;&lt;li&gt;Overfitting : &lt;code&gt;0.96&lt;/code&gt; on train set &amp;amp; &lt;code&gt;0.75&lt;/code&gt; on test set&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As a good data-scientist, you want your model to fit the data enough to generalize well but not too much not to overfit. To control how your model generalize, one uses cross-validation techniques. The value you get is pretty-much what you will obtain with new value ± the variance associated to this crossvalidation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Using cross-validation too often on test data makes you, &lt;em&gt;in a way&lt;/em&gt;, learning this data as you choose them to maximize your test score. It can lead to a form of overfitting for future new data.&lt;/p&gt;&#xA;" OwnerUserId="20759" LastActivityDate="2016-07-04T15:18:52.713" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12593" PostTypeId="1" CreationDate="2016-07-04T15:19:21.973" Score="5" ViewCount="579" Body="&lt;p&gt;In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you apply outlier detection in your usual routine? By this I mean: among $N$ training data decide $K&amp;lt;&amp;lt;N$ data points to be outliers and discard them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are your experiences doing this? Which methods do you apply for numerical data.&lt;/p&gt;&#xA;" OwnerUserId="20884" LastActivityDate="2016-07-20T11:51:54.210" Title="Do you apply outlier detection of numerical data in practical applications?" Tags="|preprocessing|outlier|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12594" PostTypeId="2" ParentId="12318" CreationDate="2016-07-04T15:22:15.100" Score="57" Body="&lt;p&gt;From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The meaning of the importance data table is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The &lt;strong&gt;&lt;em&gt;Gain&lt;/em&gt;&lt;/strong&gt; implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;strong&gt;&lt;em&gt;Cover&lt;/em&gt;&lt;/strong&gt; metric means the relative number of observations related to this feature. For example, if you have 100 observations, 4 features and 3 trees, and suppose feature1 is used to decide the leaf node for 10, 5, and 2 observations in tree1, tree2 and tree3 respectively; then the metric will count cover for this feature as 10+5+2 = 17 observations. This will be calculated for all the 4 features and the cover will be 17 expressed as a percentage for all features' cover metrics.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;strong&gt;&lt;em&gt;Frequency&lt;/em&gt;&lt;/strong&gt; (/'Frequence') is the percentage representing the relative number of times a particular feature occurs in the trees of the model. In the above example, if feature1 occurred in 2 splits, 1 split and 3 splits in each of tree1, tree2 and tree3; then the weightage for feature1 will be 2+1+3 = 6. The frequency for feature1 is calculated as its percentage weight over weights of all features.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The Gain is the most relevant attribute to interpret the relative importance of each feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The measures are all relative and hence all sum up to one, an example from a fitted xgboost model in R is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sum(importance&lt;span class=&quot;math-container&quot;&gt;$Frequence)&#xA;[1] 1&#xA;&amp;gt; sum(importance$&lt;/span&gt;Cover)&#xA;[1] 1&#xA;&amp;gt; sum(importance$Gain)&#xA;[1] 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="8501" LastEditDate="2019-05-28T20:37:58.393" LastActivityDate="2019-05-28T20:37:58.393" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12595" PostTypeId="1" AcceptedAnswerId="12864" CreationDate="2016-07-04T15:27:06.597" Score="4" ViewCount="1038" Body="&lt;p&gt;In &lt;a href=&quot;http://arxiv.org/pdf/1606.07792v1.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; research paper, it is discussed how to combine deep learning with wide (shallow) learning to achieve both generalisation and the ability to learn correlation/association rules.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input vector of such network is an n-dimensional vector of features $\mathbf{x} = (x_1, x_2, x_3,\dots, x_n)$. While training, the following objective function is maximised:&#xA;$$  P(Y=1|\mathbf{x})   $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within the context of recommender systems (which is one of the main applications of this paper) how does one cope with the problem of missing data (unseen products)?&lt;/p&gt;&#xA;" OwnerUserId="18682" LastEditorUserId="18682" LastEditDate="2016-07-06T14:55:50.960" LastActivityDate="2016-07-19T08:29:59.030" Title="Recommendations and Missing Data in Deep Learning" Tags="|neural-network|deep-learning|recommender-system|regularization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12596" PostTypeId="1" CreationDate="2016-07-04T16:31:22.983" Score="5" ViewCount="7452" Body="&lt;p&gt;I have a classification problem, where I initially started with 100+ class labels. My intuition told me this is too many labels for a model to predict with good accuracy. I thought grouping the labels together will help the situation and it did, there is now only 19 class labels. Is 19 still too many class labels? What are the typical signs that there are too many labels for a model to predict and how can this be addressed, if there is too many class labels?&lt;/p&gt;&#xA;" OwnerUserId="20429" LastEditorUserId="13727" LastEditDate="2016-07-07T06:26:38.050" LastActivityDate="2021-03-27T14:10:10.913" Title="Signs there are too many class labels" Tags="|machine-learning|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12597" PostTypeId="2" ParentId="12554" CreationDate="2016-07-04T16:49:58.827" Score="63" Body="&lt;p&gt;Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. However, its a good practice to remove any redundant features from any dataset used for training, irrespective of the model's  algorithm. In your case since you're deriving new features, you could use this approach, evaluate each feature's importance and retain only the best features for your final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The importance matrix of an xgboost model is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees. The second column is the Gain metric which implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-07-04T16:49:58.827" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12598" PostTypeId="5" CreationDate="2016-07-05T02:16:43.087" Score="0" Body="&lt;p&gt;Julia is a programming language designed for scientific computing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has an LLVM based JIT compiler which helps it deliver fast performance for technical computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complete documentation is available &lt;a href=&quot;http://docs.julialang.org/en/release-0.4/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-05T13:57:45.850" LastActivityDate="2016-07-05T13:57:45.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12599" PostTypeId="4" CreationDate="2016-07-05T02:16:43.087" Score="0" Body="For questions regarding the Julia Programming language" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2016-07-05T13:57:48.993" LastActivityDate="2016-07-05T13:57:48.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12600" PostTypeId="1" AcceptedAnswerId="12601" CreationDate="2016-07-05T06:04:53.387" Score="0" ViewCount="135" Body="&lt;p&gt;I am required to complete a project on ML applications. I guess there is a lot of statistics in ML, not helpful for a non-maths background. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am getting too bogged down by notations. There are too many notations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I am trying to read about what Bernoulli trials are and I can't relate to it.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is a Bayesian setting and why is Bayesian thing everywhere?&lt;/strong&gt; What makes it such an omnipotent distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any theorems/theories that I must know ? Any book/notes/resources where I could learn about these stuff in a relatable way (I have very little maths background, but I can learn)?&lt;/p&gt;&#xA;" OwnerUserId="21162" LastEditorUserId="11097" LastEditDate="2016-07-05T06:07:46.877" LastActivityDate="2016-07-05T06:20:21.467" Title="Understanding Bernoulli Trials, Bayesian Setting" Tags="|machine-learning|beginner|bayesian|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12601" PostTypeId="2" ParentId="12600" CreationDate="2016-07-05T06:20:21.467" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What is a Bayesian setting and why is Bayesian thing everywhere?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In very simple terms:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bayesian is a statistical setting, where the likelihood of an event&#xA;  happening (called the posterior) depends on the prior trials or&#xA;  observations (called the prior(s)).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Bayesian networks is an extension of the above, forming a chain or a network of inferencing.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any theorems/theories that I must know?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For understanding the Bayesian paradigm, you need to know the Bayesian theorem/relation, which is basically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(\theta|d) = \dfrac{P(d|\theta)P(\theta)}{P(d)}.$$&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Any book/notes/resources where I could learn about these stuff in a&#xA;  relatable way (I have very little maths background, but I can learn)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would highly recommend &lt;a href=&quot;http://store.elsevier.com/Doing-Bayesian-Data-Analysis/John-Kruschke/isbn-9780124059160/&quot; rel=&quot;nofollow&quot;&gt;&quot;Doing Bayesian Analysis&quot; by John Krushke&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-07-05T06:20:21.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12602" PostTypeId="2" ParentId="455" CreationDate="2016-07-05T08:40:00.757" Score="1" Body="&lt;p&gt;Apart from the suggestions above, there is an extremely useful pdf - &lt;a href=&quot;http://www.icmc.usp.br/CMS/Arquivos/arquivos_enviados/BIBLIOTECA_113_RT_395.pdf&quot; rel=&quot;nofollow&quot;&gt;Benchmarking Text Collections for Classification and Clustering Tasks&lt;/a&gt; which contains various datasets along with the benchmarks in order to test our models. This includes 20ng Collection, Reuters and many of the above suggested datasets. I hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-05T08:40:00.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12604" PostTypeId="1" CreationDate="2016-07-05T10:24:17.933" Score="4" ViewCount="182" Body="&lt;p&gt;I have a dataset with 8 continuous variables describing the behavior of a machine:&lt;br&gt;&#xA;- input a,b,c for both left and right: &lt;code&gt;a_left&lt;/code&gt;, &lt;code&gt;b_left&lt;/code&gt;, &lt;code&gt;c_left&lt;/code&gt;, &lt;code&gt;a_right&lt;/code&gt;, &lt;code&gt;b_right&lt;/code&gt;, &lt;code&gt;c_right&lt;/code&gt;&lt;br&gt;&#xA;- output &lt;code&gt;o&lt;/code&gt;, non-linearly related to the inputs (and almost certainly also influenced by other factors)&lt;br&gt;&#xA;- timestamp &lt;code&gt;t&lt;/code&gt;&lt;br&gt;&#xA;I have some environmental data (e.g. temperature) as well but these are fairly constant and I suspect not of significant influence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Theory&lt;/strong&gt;:&lt;br&gt;&#xA;My theory is that due to performance degradation of the machine due to fouling, the value of &lt;code&gt;o&lt;/code&gt; will be lower for the same input values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Occasionally, the machine is cleaned to restore performance again. I do not know when and how well (i.e. to a &quot;perfect&quot; state vs. to some intermediate state between &quot;perfect&quot; and the previous state). The rate of degradation is also not known and most likely not constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, a change is one of the input variables has a delayed effect on the output due to inertia.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actions I have tried:&lt;br&gt;&#xA;- plotting the data for some different relations between the input and output&lt;br&gt;&#xA;- k-means clustering for some different relations between input and output (without timestamps)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;br&gt;&#xA;- Is it at all possible to verify my theory with this data?&lt;br&gt;&#xA;- If so, what techniques are most suited to be applied? I'd prefer anything in Matlab.&lt;/p&gt;&#xA;" OwnerUserId="20389" LastEditorUserId="11097" LastEditDate="2016-07-05T10:35:28.730" LastActivityDate="2017-03-05T05:25:56.807" Title="How to find a changing relation between continuous variables?" Tags="|machine-learning|matlab|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12606" PostTypeId="2" ParentId="11325" CreationDate="2016-07-05T13:40:59.317" Score="3" Body="&lt;p&gt;&quot;Realtime&quot; is a very big word, in many regards you would only call a system realtime when there is very little latency in a system, almost no latency. It would require specialized hardware and os, for a discussion of python and realtime see &lt;a href=&quot;https://stackoverflow.com/questions/7079864/real-time-operating-via-python&quot;&gt;https://stackoverflow.com/questions/7079864/real-time-operating-via-python&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you describe the requirement as &quot;continously (or frequently) updated&quot;, it looks possible to me to do some mining on streamed data with orange. I am adding my 5 ct, as I have just started looking into orange. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would need to create a widget that opens your input stream and propagates it in chunks to the subsequent widgets in the flow, i.e. when you want to build something like a &quot;tail -f&quot; on a file source, reads character by character until there is a linefeed in the stream. In that case it would release a signal with the line as data to the output.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Orange widgets are receiving their inputs as signals, &quot;widgets receive inputs at runtime with the designated handler method (specified in the OWWidget.inputs class member).&quot; This next widget would update on that signal and propagate on its own outputs. So the continued live analysis propagation of streaming data through orange widgets seems in general possible to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How close to or far from real time this would be depends on many factors like load, computing power, and memory. The choice of mining algorithms might also be limited by such factors. &lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-05T13:40:59.317" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12607" PostTypeId="1" CreationDate="2016-07-05T13:41:01.623" Score="7" ViewCount="8050" Body="&lt;p&gt;I am working on a multilabel text classification problem with 10 labels. The dataset is small, +- 7000 items and +-7500 labels in total. I am using python sci-kit learn and something strange came up in the results. As a baseline, I started out with using the countvectorizer and was actually planning on using the tfidf vectorizer which I thought would work better. But it doesn't.. with the countvectorizer I get a performance of a 0.1 higher f1score. (0.76 vs 0.65)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot wrap my head around why this could be the case? There are 10 categories and one is called miscellaneous. Especially this one gets a much lower performance with tfidf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know when or why tfidf could perform worse than count?&#xA;I need to formalize this for my thesis.&lt;/p&gt;&#xA;" OwnerUserId="21166" LastEditorUserId="15902" LastEditDate="2018-04-17T15:33:32.517" LastActivityDate="2023-01-24T04:57:05.470" Title="TF-IDF vectorizer doesn't work better than countvectorizer" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12608" PostTypeId="2" ParentId="12607" CreationDate="2016-07-05T14:43:19.190" Score="6" Body="&lt;p&gt;There are a few possibilities. First, there is some variability in performance. It could have been only by chance that countvectorizer performed better than tf-idf. Did you use cross validation (with how many folds)? Is the superior performance of the countvectorizer reliable? I would compare performance across folds to make sure countvectorizer consistently performs better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, if you find that countvectorizer reliably outperforms tf-idf on your dataset, then I would dig deeper into the words that are driving this effect. It may be that common words (words which will appear in multiple documents) are helpful in distinguishing between classes. There is substantial research that shows that use of some function words (e.g. first person singular pronouns, “I”) change depending on someone’s psychological state. Function words like pronouns are very common and would be down weighted in tf-idf, but given equal weight to rare words in countvectorizer. I’m not suggesting that first person singular pronouns in particular are driving your results, but it’s worth looking at what words are driving the effect. I would examine which words are important in both types of models, countvectorizer and tf-idf, and then think about whether the words that are most important for the countvectorizer make sense in the context of your text documents and labels. Also, are you removing stop words? You could also see how the models perform with and without stop words, which would be another way to test whether frequent words are actually helping you to distinguish between classes.&lt;/p&gt;&#xA;" OwnerUserId="17771" LastActivityDate="2016-07-05T14:43:19.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12609" PostTypeId="2" ParentId="10552" CreationDate="2016-07-05T14:43:26.893" Score="1" Body="&lt;p&gt;You could also try &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow&quot;&gt;tSNE&lt;/a&gt;, also available in &lt;a href=&quot;https://en.wikipedia.org/wiki/Scikit-learn&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;. It is a probabilistic dimensionality reduction technique, specifically designed for plotting data in two or three dimensions, while preserving the original distances between the samples as much as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to see how well your data is clustered. &lt;/p&gt;&#xA;" OwnerUserId="18682" LastActivityDate="2016-07-05T14:43:26.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12610" PostTypeId="2" ParentId="12319" CreationDate="2016-07-05T15:09:43.760" Score="1" Body="&lt;p&gt;What do you want to define as your document? You could define a document as a single question, each question-answer pair, or a question with all its answers. How you define a document depends on why you want to cluster documents and how you plan to use this information. From reading your question, my guess is that you want to define a question with all its answer as a document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, it sounds like you want to apply bag of words techniques to extract features (e.g. tf-idf). If so, you would first want to concatenate the question title, question body, and each answer body into one string object. Then I would suggest using a Python package like scikit learn or nltk to do the other preprocessing steps. For example in scikit learn you can then apply &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow&quot;&gt;TfidfVectorizer&lt;/a&gt; to convert a string to a vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is unrelated to your question, but it may be easier to use the &lt;a href=&quot;https://data.stackexchange.com/&quot; rel=&quot;nofollow&quot;&gt;data explorer&lt;/a&gt; instead of the StackOverflow API. The data explorer allows you to download questions and answers in a large batch and does not have a quota. Unless you need to collect data in real time, the data explorer may be more efficient. It is updated every week and you can use SQL queries to output data tables as csv. &lt;/p&gt;&#xA;" OwnerUserId="17771" LastActivityDate="2016-07-05T15:09:43.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12612" PostTypeId="2" ParentId="963" CreationDate="2016-07-05T15:46:12.133" Score="0" Body="&lt;p&gt;I found this &lt;a href=&quot;https://github.com/wooorm/emoji-emotion&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github repo&lt;/a&gt; useful (a good start).&#xA;List of emoji rated for valence with an integer between minus five (negative) and plus five (positive).&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/wooorm/emoji-emotion/blob/master/Support.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;See list&lt;/a&gt; of supported unicode-emojis.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Note that some emoji receive arguably confusing polarities, such as&#xA;stuck_out_tongue_closed_eyes (0), due to being used for both positive&#xA;and negative emotions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="21170" LastEditorUserId="98307" LastEditDate="2020-08-06T11:03:50.720" LastActivityDate="2020-08-06T11:03:50.720" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12614" PostTypeId="1" CreationDate="2016-07-05T18:45:49.577" Score="1" ViewCount="1251" Body="&lt;p&gt;I am trying to learn Theano and TensorFlow for building neural networks for NLP based tasks. Any suggestions as to when one should choose one over the other or what works better and when or is it just a personal choice of what one is comfortable with?&lt;/p&gt;&#xA;" OwnerUserId="20779" LastEditorUserId="31513" LastEditDate="2017-05-25T10:35:22.323" LastActivityDate="2017-05-25T10:35:22.323" Title="Theano vs Tensorflow for building Neural Networks for NLP tasks" Tags="|neural-network|deep-learning|nlp|tensorflow|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12615" PostTypeId="2" ParentId="12614" CreationDate="2016-07-05T19:07:56.743" Score="3" Body="&lt;p&gt;At the end this comes down to one's opinion and experience but I ll attempt to give you a decent answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theano has been around longer and you will be able to find more resources on it. Both to learn and people's published work on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand Tensorflow has been getting better and better since it came out. It has a &lt;a href=&quot;https://www.udacity.com/course/deep-learning--ud730&quot; rel=&quot;nofollow&quot;&gt;free online course&lt;/a&gt; available and Google even released an &lt;a href=&quot;https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html&quot; rel=&quot;nofollow&quot;&gt;open model&lt;/a&gt; for natural language understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also have to check what hardware you have available. Tensorflow does not run on as many hardware as Theano currently. (Although they are actively working on Windows and OpenCl support.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good comparison on various Deep Learning Frameworks can be found &lt;a href=&quot;https://github.com/zer0n/deepframeworks&quot; rel=&quot;nofollow&quot;&gt;there&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21175" LastEditorUserId="21175" LastEditDate="2016-07-06T14:19:12.460" LastActivityDate="2016-07-06T14:19:12.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12616" PostTypeId="2" ParentId="12596" CreationDate="2016-07-05T20:11:44.670" Score="1" Body="&lt;p&gt;There is no clear cut answer for this. It depends on the size of your dataset, the computing power you have at your disposal and the accuracy you get compared to the number of classes.&lt;/p&gt;&#xA;&lt;p&gt;Some signs or rules of a thumb are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The smaller the dataset (both in data points and feature space dimensionality) the less class labels you can manage successfully.&lt;/li&gt;&#xA;&lt;li&gt;How fast can you train your model for each class? If adding that for all the classes results in acceptable time you can keep them. This applies to cases where you are training a model to predict classes in a one vs all setting. K-means clustering on the other will not be significantly affected by increasing the number of classes.&lt;/li&gt;&#xA;&lt;li&gt;Can you get good accuracy from your dataset with all those classes? If not reduce them to decrease the complexity of your problem.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For each case the balance of the above factors may give a different answer.&lt;/p&gt;&#xA;&lt;p&gt;Last but not least the number of classes to keep is influenced by the specific domain that gave you the dataset. You should use domain knowledge in addition to machine learning methods to assess whether you have too many classes or not.&lt;/p&gt;&#xA;" OwnerUserId="21175" LastEditorUserId="21175" LastEditDate="2021-03-27T14:10:10.913" LastActivityDate="2021-03-27T14:10:10.913" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12617" PostTypeId="1" AcceptedAnswerId="13241" CreationDate="2016-07-05T20:16:52.280" Score="0" ViewCount="1026" Body="&lt;p&gt;I am new to machine learning and I am struck at one thing. Please help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am developing an app to sort images based on user preferences. So initially I have n data points(images) with m features each. Now when user create an album and add some images in it then my algorithm should find similar images from the gallery and add it to the same album. This should work even if user creates more albums. And not all images are needed to be sorted, only the relevant one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithms should I use to achieve this?&lt;/p&gt;&#xA;" OwnerUserId="21179" LastActivityDate="2019-11-04T01:01:54.287" Title="How to identify similar data points?" Tags="|machine-learning|classification|clustering|decision-trees|naive-bayes-classifier|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12618" PostTypeId="2" ParentId="12593" CreationDate="2016-07-05T20:48:00.003" Score="0" Body="&lt;p&gt;If you are dealing with continuous data that generally fits a normal distribution, you can apply the Z-score as a method of determining how many standard deviations the observed value is from the mean.  Depending on your objective, you could throw out large z-scores.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Standard_score&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boxplots are good for visualizing and identifying outliers, although just a boxplot can sometimes hide information about the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link to a similar conversation:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/8667/which-algorithms-or-methods-can-be-used-to-detect-an-outlier-from-this-data-set?rq=1&quot;&gt;Which algorithms or methods can be used to detect an outlier from this data set?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-05T20:48:00.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12619" PostTypeId="1" CreationDate="2016-07-05T21:25:30.557" Score="3" ViewCount="302" Body="&lt;p&gt;I understand that SAS has some data storage options in the form of &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Default Storage&lt;/strong&gt; tables or SAS datasets - in the SAS library.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Multi-dimensional Storage&lt;/strong&gt; cubes - in the OLAP Server.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Third Party Storage&lt;/strong&gt; - into external databases like SQL Server ,Oracle, DB2, Teradata etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Parallel Storage&lt;/strong&gt; - ?&#xA;&lt;br&gt;&lt;br&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;My Questions are:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Are any of these data storage options designed for moderately big data i.e. data over a million observations? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is this quantification at all affected by the machine that SAS is&#xA;running on?&lt;/li&gt;&#xA;&lt;li&gt;If yes,then let's imagine a computer with 4 cores and 16GB Memory.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I've worked with SAS tables before, but they were never larger than a couple hundred thousand observations.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;The Parallel Storage is supposed to be a high speed alternative for processing very large data sets. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are these data sets stored in the SAS Library / Server etc? &lt;/li&gt;&#xA;&lt;li&gt;Can Parallel Storage help in scaling to data over a million records?&#xA;&lt;br&gt;&lt;br&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, do these data storage options make SAS function like a Database Management System - like SQL Server?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2647" OwnerDisplayName="Minu" LastActivityDate="2016-07-05T21:52:28.237" Title="Are SAS Data Storage Options designed for Big Data?" Tags="|dataset|sas|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12620" PostTypeId="1" CreationDate="2016-07-05T23:46:42.083" Score="1" ViewCount="6970" Body="&lt;p&gt;I am trying to find the best way to extract information from bank statements. A bank transaction is not a natural text but still human readable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to extract a bunch of data if present like payment method, date, amount, vendor/customer name and even information like an order/invoice ID or the reason. I have sets of data that I can use for the training, for instance (100k+ vendors, payment methods words, etc). Also the solution must work with multi languages. The bank transactions could be in different languages (not only english).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is named entity recognition the best way to go?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is some sample data I can have in input:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dividend&lt;/li&gt;&#xA;&lt;li&gt;POS PURCHASE Non-PIN: RCI*RINGCENTRAL, INC.&lt;/li&gt;&#xA;&lt;li&gt;IMP.DEB.LEY 83652 - ALICUOTA GENERA&lt;/li&gt;&#xA;&lt;li&gt;DIRECTDEBIT EQUITY COMMERCE MERCH FEES 158713 62000329526385 CCD&lt;/li&gt;&#xA;&lt;li&gt;ADVANCE AUTO PARTS JACKSONVILLE&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help&lt;/p&gt;&#xA;" OwnerUserId="21184" LastActivityDate="2020-10-06T16:49:11.453" Title="What algorithm to use for extracting information from bank statements" Tags="|machine-learning|nlp|named-entity-recognition|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12622" PostTypeId="1" CreationDate="2016-07-06T04:49:45.610" Score="2" ViewCount="100" Body="&lt;p&gt;I am working on Twitter sentiment analysis. Now, I should prepare a training data set that fits for any kind of Twitter data that predicts the sentiment of a tweet as pos, neg, and neu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From googling, I found an air-line tweets training data set with needed labels (pos, neg, neu, and sentiment). When I checked the sentiment of tweets with that training data set, it gave me mixed results. I found that some positive tweets are given as negative and negative as positive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the method that I am following correct or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest your views for preparing a training data set that can predict any kind of the twitter data.     &lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="29169" LastEditDate="2019-06-07T16:35:30.110" LastActivityDate="2019-06-07T16:35:30.110" Title="Does the training set of one topic will be useful to predicate the sentiment for any other topic?" Tags="|machine-learning|classification|naive-bayes-classifier|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12624" PostTypeId="2" ParentId="12596" CreationDate="2016-07-06T05:28:46.560" Score="5" Body="&lt;p&gt;A class taxonomy should:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Serve the business needs &lt;/li&gt;&#xA;&lt;li&gt;Be learnable&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case and the boundary samples will have more influence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check the number of samples per label in your dataset. A class label with not enough samples will be hard to learn. If you have many class label, the ratio of some of them will also be low, introducing an imbalance learning problem which is also harder. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a cost matrix, you can evaluate the extra cost due to merging some classes. merge classes when the cost on not distinguishing among them is low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Building a taxonomy of class might can help working in many resolutions. Take for example animals, you can create a taxonomy of animals -&gt; reptiles -&gt; crocodiles and then try to distinguish reptiles from other animals or crocodiles from other reptiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many some cases you might get many class labels due to a Cartesian product of some dimensions. Using the animals examples again, you might be interested in dimension like &quot;is_flying?&quot;, &quot;is_carnivore?&quot; and learn them instead of the animal classes. These dimensions will probably be more balanced and can be learn with a certain degree on independence (reusing samples). The down side is the a flying carnivore will be a bird of prey and not necessarily a falcon.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-07-06T05:28:46.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12625" PostTypeId="2" ParentId="12242" CreationDate="2016-07-06T05:39:15.323" Score="2" Body="&lt;p&gt;Yes, they can connect natively. You can manage data and then put it all in different services like it is showed next. As you can see you can use SQL database, blob storate and also PowerBI.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/gbSIy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gbSIy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://azure.microsoft.com/en-gb/documentation/articles/stream-analytics-power-bi-dashboard/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; you can find a tutorial on how to do streaming analytics with Azure and PowerBI.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-07-06T05:39:15.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12626" PostTypeId="1" CreationDate="2016-07-06T06:16:26.047" Score="2" ViewCount="12357" Body="&lt;p&gt;What is the difference between the test and training data sets? As per blogs and papers I studied, what I understood is that we will have 100% data set that is divided into 2 sets (test data set is 30% and reaming 70% is training data set). I would like to know more points and use of differentiating the 100% data set to test and training data sets. &lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="29169" LastEditDate="2019-06-07T21:44:11.027" LastActivityDate="2019-06-07T21:44:11.027" Title="What is the difference between the training and testdata set?" Tags="|machine-learning|beginner|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12628" PostTypeId="2" ParentId="12626" CreationDate="2016-07-06T07:25:08.253" Score="5" Body="&lt;p&gt;In Machine Learning, we basically try to create a model to predict on the test data. So, we use the training data to fit the model and testing data to test it. The models generated are to predict the results unknown which is named as the test set. As you pointed out, the dataset is divided into train and test set in order to check accuracies, precisions by training and testing it on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The proportion to be divided is completely up to you and the task you face. It is not essential that 70% of the data has to be for training and rest for testing. It completely depends on the dataset being used and the task to be accomplished. For example, a simple dataset like &lt;a href=&quot;http://www.nltk.org/book/ch02.html&quot; rel=&quot;nofollow&quot;&gt;reuters&lt;/a&gt;. So, assume that we trained it on 50% data and tested it on rest 50%, the precision will be different from training it on 90% or so. This is mostly because in Machine Learning, the bigger the dataset to train is better. You can refer &lt;a href=&quot;http://arxiv.org/pdf/1604.00783v1.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper, which tells the precision values based on the dataset size. It now depends on you, what precision or accuracy you need to achieve based on your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This said so, how would you predict the results for which you do not have the answer? (The model is ultimately being trained to predict results for which we do not have the answer). I would like to add on about validation dataset here.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sets:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Training Set&lt;/strong&gt;: Here, you have the complete training dataset. You can extract features and train to fit a model and so on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Validation Set&lt;/strong&gt;: This is crucial to choose the right parameters for your estimator. We can divide the training set into train set and validation set. Based on the validation test results, the model can be trained(for instance, changing parameters, classifiers). This will help us get the most optimized model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Testing Set&lt;/strong&gt;: Here, once the model is obtained, you can predict using the model obtained on the training set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Refer &lt;a href=&quot;https://www.quora.com/What-is-a-training-data-set-test-data-set-in-machine-learning-What-are-the-rules-for-selecting-them&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; for more info.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-07-06T07:27:09.220" LastActivityDate="2016-07-06T07:27:09.220" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12629" PostTypeId="2" ParentId="12584" CreationDate="2016-07-06T08:54:56.850" Score="2" Body="&lt;p&gt;The basic task here I get is sentiment analysis of tweets. So for this, we can first extract features and then use a classifier. Stop words are those which are present for grammatical purposes but do not add any value/meaning to the sentence.&#xA;So, yes you can remove the stop words from the tweet, extract the features and then pass it to the classifier. You can get already existing stop words from&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import stopwords&#xA;stop_words = stopwords.words('english')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Removing the stop words doesn't always improve the accuracy. &#xA;Take a look at &lt;a href=&quot;http://streamhacker.com/2010/05/24/text-classification-sentiment-analysis-stopwords-collocations/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; as it is a very similar problem statement as yours. Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-06T08:54:56.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12630" PostTypeId="1" CreationDate="2016-07-06T08:59:59.330" Score="2" ViewCount="124" Body="&lt;p&gt;I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (&lt;a href=&quot;http://pythonhosted.org/libpgm/&quot; rel=&quot;nofollow&quot;&gt;http://pythonhosted.org/libpgm/&lt;/a&gt;) and found that 2TBN is ideal for my purpose, but libpgm does not support structure learning of 2TBN. BNFinder and ebdbNet cannot learn edges within time slices. Could you tell me any great libraries and/or software with which I can infer structures of 2TBN from data or something similar?&lt;/p&gt;&#xA;" OwnerDisplayName="user21193" LastEditorDisplayName="user21193" LastEditDate="2016-07-06T15:21:25.690" LastActivityDate="2016-07-06T15:21:25.690" Title="Good Libraries or Software for Temporal Bayesian Network Structure Learning?" Tags="|dataset|time-series|bayesian-networks|model-selection|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12631" PostTypeId="2" ParentId="12622" CreationDate="2016-07-06T09:19:15.473" Score="1" Body="&lt;p&gt;Since you have trained your model on air-line tweets, the model will learn the characteristics of the air-line tweets. There could be words used in air-line tweets which are positive and contain great weightage but never even used in other tweets leading to negative results. &#xA;I suggest you to try &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Get a dataset which contains all kinds of tweets which you desire essentially similar to what your task ahead is.&lt;/li&gt;&#xA;&lt;li&gt;Look into &lt;a href=&quot;ftp://ftp.cs.wisc.edu/machine-learning/shavlik-group/torrey.handbook09.pdf&quot; rel=&quot;nofollow&quot;&gt;Transfer learning&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-06T09:19:15.473" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="12633" PostTypeId="1" CreationDate="2016-07-06T12:48:57.757" Score="3" ViewCount="228" Body="&lt;p&gt;I've found many articles (&lt;a href=&quot;http://link.springer.com/chapter/10.1007%2F978-3-642-14715-9_5#page-1&quot; rel=&quot;nofollow&quot;&gt;like this one&lt;/a&gt;) on HAR that mention using the gyroscope and magnetometer because it improves the precision of HAR performed just with the accelerometer. However no one mentions how this data is processed. For acceleration usually I calculate the mean, standard deviation, variance, energy and frequency domain entropy; but since the magnetometer just shows the degrees from magnetic or true North and it's not an acceleration value, does it make sense to use the same features for the magnetic field values? should I instead use the speed of variation of the magnetic field and gyroscope values? I can't find any article that explains how to process this data.&lt;/p&gt;&#xA;" OwnerUserId="21197" LastActivityDate="2016-07-06T12:48:57.757" Title="Human Activity Recognition - How to process gyroscope and magnetometer data" Tags="|machine-learning|data-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12635" PostTypeId="1" CreationDate="2016-07-06T13:29:13.103" Score="2" ViewCount="161" Body="&lt;p&gt;I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose.  For smaller data sets (MB to GB), I use modules in Python, R, or just good ole scripting with pipes for stdin/stdout, often with a relational database for storage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: what tools are available on Linux to handle terabyte+ sized data sets in other fields, such as data science, finance, web analytics, etc?  Are there any standard references on them, such as &lt;em&gt;K&amp;amp;R&lt;/em&gt; for learning C?&lt;/p&gt;&#xA;" OwnerUserId="21198" LastActivityDate="2016-07-06T13:29:13.103" Title="Linux tools to handle large data sets (several TB)" Tags="|bigdata|dataset|tools|" AnswerCount="0" CommentCount="1" ClosedDate="2016-07-06T22:23:22.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="12637" PostTypeId="2" ParentId="12626" CreationDate="2016-07-06T16:39:09.817" Score="0" Body="&lt;p&gt;^ I completely agree with the answer above from Hima Varsha, however I wanted to add that sometimes there is different names for testing sets. A data science company I used to work for would use a training set, validation set, testing set, and sometimes a third testing set called a &quot;holdout set&quot;. I'm not sure if that was particular to the company I was working for, but you might see it crop up in literature or documentation in the future. Also, there is a third metric that almost always goes along with precision and recall called the &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F1 score&lt;/a&gt;. This is actually calculated from both precision and recall, but commonly used as a summary of both precision and recall. &lt;/p&gt;&#xA;" OwnerUserId="21206" LastActivityDate="2016-07-06T16:39:09.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12638" PostTypeId="1" CreationDate="2016-07-06T17:45:06.570" Score="3" ViewCount="81" Body="&lt;p&gt;I am trying to figure out an approach for calculating the probability of a renter making &gt; 1 booking. I would prefer to use Python for this project. I have the following columns ready:&#xA;State, Category (fun, adventure, etc.), returning_renter? (TRUE if &gt;= 1 booking).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the most efficient approach for this project? Are there any particular Python libraries you would recommend?  &lt;/p&gt;&#xA;" OwnerUserId="21208" LastEditorUserId="21" LastEditDate="2016-07-07T06:27:17.100" LastActivityDate="2016-07-07T16:42:40.343" Title="Predicting return renters" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12639" PostTypeId="2" ParentId="12638" CreationDate="2016-07-06T18:05:09.817" Score="0" Body="&lt;p&gt;The &quot;standard&quot; approach, when using python, would be data manipulation with the &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow&quot;&gt;numpy&lt;/a&gt; library. If you have more complext needs &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;pandas&lt;/a&gt; library is also a very good idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to train a predictive model I would highly recommend &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;. It offers a wide range of machine learning algorithms and they have a very good online documentation and examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your needs there may be more efficient approaches but as a rule of a thumb your implementation using the aforementioned libraries will be good enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not dealing with a big dataset I would suggest using &lt;a href=&quot;https://docs.python.org/2/library/pickle.html&quot; rel=&quot;nofollow&quot;&gt;pickle&lt;/a&gt; in order to save intermediate python objects instead of having to recalculate them again.&lt;/p&gt;&#xA;" OwnerUserId="21175" LastEditorUserId="21175" LastEditDate="2016-07-06T18:14:59.303" LastActivityDate="2016-07-06T18:14:59.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12640" PostTypeId="1" CreationDate="2016-07-06T19:53:21.203" Score="1" ViewCount="68" Body="&lt;p&gt;So, I've been building data collectors for the stock market. Basically&#xA;this is just to give myself a resource that I can query as much as I&#xA;want and not get dinged by over usage. (the learning experience is&#xA;also a plus)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm curious though if the change in the frequency of data will skew&#xA;the results of a regression model (to keep things simple &lt;code&gt;y = price&lt;/code&gt;&#xA;and &lt;code&gt;x = time_stamp&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem comes in when the amount of data is considered. I have&#xA;the collectors recording the change in the stocks once a minute. For&#xA;data in the past that happened before I was collecting, I load the daily&#xA;open and close prices and whatever other information companies are&#xA;willing to share for free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, will the much more detailed data that I'm collecting&#xA;each minute  throw off or enhance the regression models that can be&#xA;created by just looking at the daily open and close prices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would assume that it would enhance it but, I would like to know if that's not the case and that in order to get an accurate measurement I would need to just use the interval that applies universally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example chart to better explain what I mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gTc8Z.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gTc8Z.jpg&quot; alt=&quot;sample_plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21211" LastEditorUserId="21211" LastEditDate="2016-07-07T16:05:52.670" LastActivityDate="2016-07-07T16:05:52.670" Title="Does variation in data density over time affect regression models?" Tags="|machine-learning|statistics|regression|logistic-regression|linear-regression|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12641" PostTypeId="2" ParentId="12568" CreationDate="2016-07-06T23:28:21.950" Score="2" Body="&lt;p&gt;I don't know how to write math algebra like on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm&quot; rel=&quot;nofollow&quot;&gt;Wikipedia page for Smith-Waterman&lt;/a&gt;, so i'll use pseudo code. I found the logic at SimMetrics in the &lt;a href=&quot;https://github.com/Simmetrics/simmetrics/blob/master/simmetrics-core/src/main/java/org/simmetrics/metrics/SmithWatermanGotoh.java&quot; rel=&quot;nofollow&quot;&gt;SmithWatermanGotoh java code&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;str1 = PELICAN&#xA;str2 = COELACANTH&#xA;matchValue = 1 #in the comparisons below, when characters are equal, assign this value&#xA;mismatchValue = -2 #in the comparisons below, when characters are not equal, assign this value&#xA;gapValue = -0.5 #the gap penalty used in smith-waterman &#xA;&#xA;# get the maxDistance which is the smallest number of characters between str1 and str2 multiplied by &#xA;# the largest of matchValue and gapValue &#xA;maxDistance = min(length(str1), length(str2)) x max(matchValue, gapValue);&#xA;&#xA;# function to compare character at index aIndex of string a with character at index bIndex of string b &#xA;function compareCharacters(a, aIndex, b, bIndex, matchValue, mismatchValue) {&#xA;  if a[aIndex] === b[bIndex] &#xA;    return matchValue &#xA;  else &#xA;    return mismatchValue &#xA;}&#xA;&#xA;v0 = an array &#xA;v1 = an array &#xA;&#xA;lengthOfStr1 = number of characters in str1 &#xA;lengthOfStr2 = number of characters in str2 &#xA;&#xA;# do the smith waterman similarity measure (currentMax)&#xA;currentMax = v0[0] = max(0, gapValue, compareCharacters(str1, 0, str2, 0, matchValue, mismatchValue))&#xA;&#xA;for (j = 1; j &amp;lt; lengthOfStr2; j++) {&#xA;  v0[j] = max(0, v0[j - 1] + gapValue,&#xA;            compareCharacters(str1, 0, str2, j, matchValue, mismatchValue))&#xA;&#xA;  currentMax = max(currentMax, v0[j])&#xA;}&#xA;&#xA;for (i = 1; i &amp;lt; lengthOfStr1; i++) {&#xA;  v1[0] = max(0, v0[0] + gapValue, compareCharacters(str1, i, str2, 0, matchValue, mismatchValue))&#xA;&#xA;  currentMax = max(currentMax, v1[0])&#xA;&#xA;  for (j = 1; j &amp;lt; lengthOfStr2; j++) {&#xA;    v1[j] = max(0, v0[j] + gapValue, v1[j - 1] + gapValue,&#xA;            v0[j - 1] + compareCharacters(str1, i, str2, j, matchValue, mismatchValue))&#xA;&#xA;    currentMax = max(currentMax, v1[j])&#xA;  }&#xA;&#xA;  for (j = 0; j &amp;lt; lengthOfStr2; j++) {&#xA;    v0[j] = v1[j]&#xA;  }&#xA;}&#xA;&#xA;# calculate the overallSimilarity between the strings &#xA;overallSimilarity = currentMax / maxDistance #&amp;lt;- 0.4767 for COELACANTH vs PELICAN &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21120" LastActivityDate="2016-07-06T23:28:21.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12642" PostTypeId="2" ParentId="12604" CreationDate="2016-07-07T01:55:03.733" Score="0" Body="&lt;p&gt;There are tests for the presence of heteroscedasticity in stochastic processes available through Matlab, such as Engle's ARCH test. This is the only test I've used, there may be others available through Matlab as well.&lt;/p&gt;&#xA;" OwnerUserId="10786" LastActivityDate="2016-07-07T01:55:03.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12643" PostTypeId="1" CreationDate="2016-07-07T03:26:01.983" Score="1" ViewCount="116" Body="&lt;p&gt;I've been playing around with some data, and I've created a decently large similarity matrix (each column value for a row represents how similar the document corresponding to the column is to the document corresponding to the row) and I'm trying to scale this data down to 2 dimensions so I can visualize it on a scatter plot, where each point represents a document, and the closer two documents are, the closer they would be on the scatter plot. I've tried using Sammon Mapping to accomplish this, but the problem is it takes a significant amount of time to run. Any ideas? &lt;/p&gt;&#xA;" OwnerUserId="21220" LastActivityDate="2016-07-08T09:13:03.633" Title="A faster multidimensional scaling?" Tags="|dataset|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12644" PostTypeId="1" CreationDate="2016-07-07T05:46:38.567" Score="3" ViewCount="729" Body="&lt;p&gt;I want to try the skipgrams approach on my dataset. But I do not know how to vectorize it. For example, I have my cleaned document for which I got it's skipgrams. Now, how do I know vectorize it so that I can use it further for classification? I use sklearn for all the above purposes.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2018-07-12T06:40:05.507" Title="Vectorizing Skipgrams in sklearn" Tags="|machine-learning|classification|nlp|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12645" PostTypeId="1" AcceptedAnswerId="12646" CreationDate="2016-07-07T10:26:23.330" Score="31" ViewCount="156308" Body="&lt;p&gt;How can I get the number of missing value in each row in Pandas dataframe.&#xA;I would like to split dataframe to different dataframes which have same number of missing values in each row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion?&lt;/p&gt;&#xA;" OwnerUserId="15064" LastActivityDate="2020-06-14T13:15:03.997" Title="How to count the number of missing values in each row in Pandas dataframe?" Tags="|python|pandas|" AnswerCount="8" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12646" PostTypeId="2" ParentId="12645" CreationDate="2016-07-07T11:13:11.347" Score="26" Body="&lt;p&gt;You can apply a count over the rows like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_df.apply(lambda x: x.count(), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;test_df:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    A   B   C&#xA;0:  1   1   3&#xA;1:  2   nan nan&#xA;2:  nan nan nan&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0:  3&#xA;1:  1&#xA;2:  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can add the result as a column like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_df['full_count'] = test_df.apply(lambda x: x.count(), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    A   B   C   full_count&#xA;0:  1   1   3   3&#xA;1:  2   nan nan 1&#xA;2:  nan nan nan 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-07-07T11:18:34.297" LastActivityDate="2016-07-07T11:18:34.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12647" PostTypeId="2" ParentId="12604" CreationDate="2016-07-07T11:49:40.070" Score="1" Body="&lt;p&gt;Besides doing initial EDA and plotting dependent vs independent variables, it would seem prudent to try to first identify the &quot;maintenance&quot; points. According to your description of the problem, we can expect to see a decline in output &lt;em&gt;o&lt;/em&gt; punctuated by single spike-step indicating improved performance. In other words you should in general see a negative slope interspersed with very high positive slope. We see a similar type of spike behavior in EEG signal, where the &quot;blips&quot; or &quot;spikes&quot; correspond to unwanted artifacts (e.g., eye blinks). These can be identified relatively easily by running a sliding window over the data and looking for big changes in slope. The time segments marked in green below are identified as artifacts by monitoring for changes in slope. You can look into &lt;a href=&quot;http://erpinfo.org/erplab/erplab-documentation/documentation-archive-for-previous-versions/v1.x-documentation/erplab-manual/Artifact_Detection.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;EEGlab&lt;/strong&gt; and &lt;strong&gt;ERPlab&lt;/strong&gt;&lt;/a&gt; -- which are Matlab based -- to find out more about this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QSH0v.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QSH0v.jpg&quot; alt=&quot;EEG artifacts&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For feature engineering, because of the input-to-response lag, you will want to initially include variable lags. For example, for variable &lt;code&gt;a_left&lt;/code&gt; and a corresponding change in output o, you would want to add as features a_left_1, a_left_2, ... a_left_n, where each feature refers to the input value at some lagged time prior to the observed change in behavior.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-07-08T00:29:23.663" LastActivityDate="2016-07-08T00:29:23.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12648" PostTypeId="1" CreationDate="2016-07-07T13:47:49.720" Score="1" ViewCount="3041" Body="&lt;p&gt;Principal Component Analysis is a means to reduce the dimensionality of data, if I understand correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I have a 1000 sample point 12 dimensional matrix and reduce it to a 1000 sample point 2 dimensional one, then are the values of the sample points themselves changed in some way? or are simply 10 dimensions/columns thrown out and 2 of the original remain?&lt;/p&gt;&#xA;" OwnerUserId="21233" LastActivityDate="2022-09-12T02:11:00.180" Title="Does PCA change the values of the data?" Tags="|machine-learning|bigdata|feature-selection|dimensionality-reduction|pca|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12649" PostTypeId="1" AcceptedAnswerId="32078" CreationDate="2016-07-07T13:51:42.563" Score="21" ViewCount="18559" Body="&lt;p&gt;I'm trying to calculate the amount of memory needed by a GPU to train my model based on &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/#computational-considerations&quot; rel=&quot;nofollow noreferrer&quot;&gt;this notes&lt;/a&gt; from Andrej Karphaty.&lt;/p&gt;&#xA;&lt;p&gt;My network has &lt;strong&gt;532,752 activations&lt;/strong&gt; and &lt;strong&gt;19,072,984 parameters&lt;/strong&gt; (weights and biases). These are all 32 bit floats values, so each takes 4 bytes in memory.&lt;/p&gt;&#xA;&lt;p&gt;My input image is 180x50x1 (width x height x depth) = &lt;strong&gt;9,000&lt;/strong&gt; float 32 values. I don't use image augmentation, so I think the miscellaneous memory would be only related to the mini-batch size. I'm using a mini-batch size of 128 images.&lt;/p&gt;&#xA;&lt;p&gt;Based on Andrej's recommendation, I get the following memory sizes:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Activations:&lt;/strong&gt; 532,752*4/(1024^2) = &lt;strong&gt;2.03 MB&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Parameters:&lt;/strong&gt; 19,072,984*4/(1024^2) * 3 = &lt;strong&gt;218.27 MB&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Miscellaneous:&lt;/strong&gt; 128&lt;em&gt;9,000&lt;/em&gt;4/(1024^2) = &lt;strong&gt;4.39 MB&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;So the total memory to train this network would be &lt;strong&gt;224,69 MB&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I'm using TensorFlow and I think I'm missing something. I haven't run the training yet, but I'm pretty sure (based on past experiences) that the memory in use will be much higher than what I've calculated.&lt;/p&gt;&#xA;&lt;p&gt;If for each image in the mini-batch, TensorFlow keeps their gradients so it can normalize them later for a single weights/biases updates step, then I think the memory should take into account another 532,752 * 128 values (gradients for each image in the mini-batch). If that is the case, then I'd need more 260.13 MB to train this model with 128 images/mini-batch.&lt;/p&gt;&#xA;&lt;p&gt;Can you help me understand the memory considerations for training my deep learning model? Are the above considerations right?&lt;/p&gt;&#xA;" OwnerUserId="19172" LastEditorUserId="98307" LastEditDate="2020-08-02T10:28:54.973" LastActivityDate="2023-04-15T07:48:36.233" Title="How to calculate the mini-batch memory impact when training deep learning models?" Tags="|deep-learning|tensorflow|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12650" PostTypeId="2" ParentId="12648" CreationDate="2016-07-07T14:15:30.557" Score="2" Body="&lt;p&gt;Yes, the new 2 dimensional values will be a projection of original 12 dimensional points onto the two principle components (vectors). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;please refer to the first figure in this clear tutorial:&#xA;&lt;a href=&quot;http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/&quot; rel=&quot;nofollow&quot;&gt;http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21234" LastActivityDate="2016-07-07T14:15:30.557" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12651" PostTypeId="1" AcceptedAnswerId="12654" CreationDate="2016-07-07T16:35:20.943" Score="6" ViewCount="3909" Body="&lt;p&gt;&lt;code&gt;SelectKBest(f_classif, k)&lt;/code&gt;, where &lt;code&gt;k&lt;/code&gt; is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works.  A sample of how this works is below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = SelectKBest(f_classif, k)&#xA;model.fit_transform(X_train, Target_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The ANOVA F-value, as I understand it, does not require a categorical response.  (see &lt;code&gt;scipy.stats.f_oneway&lt;/code&gt;)  It is computing the value between the features. Why does &lt;code&gt;f_classif&lt;/code&gt; require the response?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does &lt;code&gt;SelectKBest&lt;/code&gt; actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="24000" LastEditDate="2017-08-02T09:09:20.967" LastActivityDate="2017-08-02T09:09:20.967" Title="How does SelectKBest() perform feature selection?" Tags="|machine-learning|feature-selection|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12652" PostTypeId="2" ParentId="12638" CreationDate="2016-07-07T16:42:40.343" Score="1" Body="&lt;p&gt;You could trivially prototype your solution in &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt; (it's a Python library that uses &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt; under the hood). It also has a neat GUI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But with only two features (&lt;code&gt;state&lt;/code&gt; and &lt;code&gt;renting_reason&lt;/code&gt;), I'm afraid your models might not be as precise as you'd want them to be, unless you have lots of examples to learn from.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-07-07T16:42:40.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12653" PostTypeId="2" ParentId="12225" CreationDate="2016-07-07T19:06:41.107" Score="1" Body="&lt;p&gt;This is broad question, but here is the general approach. There are several sub-problems involved in a typical QA system, and each may be solved using different approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Question Classification&lt;/strong&gt; - What type of question is the user asking? This can be posed as a classification problem, assuming you have labels. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Parsing the question&lt;/strong&gt; - Multiple NLP techniques would be required here&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Converting question to canonical form&lt;/strong&gt; - Every question would be converted into a canonical or structured format (which can be used to query). This is usually the most difficult part of the process. This is usually referred to as &lt;strong&gt;Semantic Parsing&lt;/strong&gt;. One approach is to get variants of phrases and then map it to a single canonical form using some similarity metric. You can start with &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwiCiubjheLNAhWBMY8KHYfwCEQQFgggMAA&amp;amp;url=http%3A%2F%2Fnlp.stanford.edu%2Fpubs%2Fwang-berant-liang-acl2015.pdf&amp;amp;usg=AFQjCNEPz2-wx4n5EJPXIvI-ee8vMah48A&amp;amp;bvm=bv.126130881,d.c2I&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; paper.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you can get from a natural language text to a canonical form, you are essentially done. Storing data and retrieval are trivial aspects which could be done in numerous ways. Coming to the question of domains, it is always going to be a trade-off between accuracy for one domain vs a general QA system (which is way too difficult to crack). A single model for every domain will most likely give you better results at the cost of extra effort.&lt;/p&gt;&#xA;" OwnerUserId="8819" LastActivityDate="2016-07-07T19:06:41.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12654" PostTypeId="2" ParentId="12651" CreationDate="2016-07-07T20:22:19.120" Score="2" Body="&lt;p&gt;Your question is really more about &lt;code&gt;f_classif&lt;/code&gt; than &lt;code&gt;SelectKBest&lt;/code&gt;. It's to &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/feature_selection/univariate_selection.py#L148&quot; rel=&quot;nofollow&quot;&gt;drop duplicate labels&lt;/a&gt;; note the &lt;code&gt;np.unique(y)&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])&#xA;args = [X[safe_mask(X, y == k)] for k in np.unique(y)]&#xA;return f_oneway(*args)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;f_oneway&lt;/code&gt; still only gets passed the feature matrix, but a subset of it.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-07T20:22:19.120" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12655" PostTypeId="1" AcceptedAnswerId="12656" CreationDate="2016-07-07T21:16:15.870" Score="0" ViewCount="52" Body="&lt;p&gt;Example: weather data. You know the location data, but you don't know the previous days'/weeks' temperatures and other weather conditions. How can you exploit these variables in your past data when you build a predictive model that attempts to forecast pretty far out into the future?&lt;/p&gt;&#xA;" OwnerUserId="6486" LastActivityDate="2016-07-07T22:09:17.110" Title="How to take advantage of variables whose values are available in the past but not in the future?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12656" PostTypeId="2" ParentId="12655" CreationDate="2016-07-07T22:09:17.110" Score="1" Body="&lt;p&gt;My generic answer to the title is to use the extra data for &lt;a href=&quot;https://en.wikipedia.org/wiki/Regularization_(mathematics)&quot; rel=&quot;nofollow&quot;&gt;regularization&lt;/a&gt; in &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_learning&quot; rel=&quot;nofollow&quot;&gt;representation learning&lt;/a&gt;; a transformation of your features into a space conducive to your main task: regression (prediction, forecasting). &lt;a href=&quot;http://dx.doi.org/10.1109/TPAMI.2013.50&quot; rel=&quot;nofollow&quot;&gt;Here's&lt;/a&gt; a survey [&lt;a href=&quot;http://www.cifasis-conicet.gov.ar/granitto/ECI2014/Bengio13.pdf&quot; rel=&quot;nofollow&quot;&gt;PDF&lt;/a&gt;].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your example, you could build a model that takes the delay of the target time from the present as an input, so you can predict arbitrarily far into the future, though it probably would not predict as well as a simple regressor that has a fixed horizon since it is trying to learn a more complex function.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-07T22:09:17.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12658" PostTypeId="1" AcceptedAnswerId="12660" CreationDate="2016-07-08T04:02:19.263" Score="2" ViewCount="939" Body="&lt;p&gt;After training a model using word2vec I'd now like to store the trained model with the word serving as a key and the vector as its value. However I'm not sure how I'll be able to implement in this way a k-nearest neighbour search. What would be the correct way to get around this?&lt;/p&gt;&#xA;" OwnerUserId="4953" LastActivityDate="2016-07-08T05:55:18.287" Title="word2vec -storing a word and its vector yet be able to efficiently run k-nearest neighbour" Tags="|databases|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12660" PostTypeId="2" ParentId="12658" CreationDate="2016-07-08T05:55:18.287" Score="2" Body="&lt;p&gt;A most popular way of obtaining the &lt;em&gt;approximate&lt;/em&gt; nearest neighbors is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing#LSH_algorithm_for_nearest_neighbor_search&quot; rel=&quot;nofollow&quot;&gt;locality sensitive hash&lt;/a&gt;. And &lt;a href=&quot;https://erikbern.com/2015/07/04/benchmark-of-approximate-nearest-neighbor-libraries/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; are some practical results. Then once you have the neighboring keys, it's straightforward to use a key-value store to retrieve the corresponding words.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-08T05:55:18.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12661" PostTypeId="1" CreationDate="2016-07-08T06:12:47.193" Score="4" ViewCount="104" Body="&lt;p&gt;I have a market transactions dataset including time stamps and goods as follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;John always buy milk and bread in Super Market. Besides that, he also buys some goods like the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;On Monday, John bought milk, bread {beer, chocolate}.&lt;/li&gt;&#xA;&lt;li&gt;On Tuesday, John bought milk, bread {potato}.&lt;/li&gt;&#xA;&lt;li&gt;On Wednesday, John bought milk, bread {chocolate, avocado, peanuts}.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can we answer the question: &quot;What will he buy on Thursdays?&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: He will buy {beer, avocado} besides milk and bread on Thursdays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which one should I use? Supervised or unsupervised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which model can I use to predict a set of goods in this case?&lt;/p&gt;&#xA;" OwnerUserId="21236" LastEditorUserId="21236" LastEditDate="2016-07-08T08:26:58.170" LastActivityDate="2016-07-19T07:42:53.293" Title="Neural Network for predict a set in a certain time stamps" Tags="|machine-learning|classification|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12662" PostTypeId="1" CreationDate="2016-07-08T08:05:55.527" Score="1" ViewCount="30" Body="&lt;p&gt;I'm trying to perform anomaly detection on the open data from citibike.&#xA;They are giving bikeshare trips for the past 30+ months, as well as monthly reports. In those reports they say how many bikes have been repaired each month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The samples I am building are a sample by day and by bike. I actually don't have label for those samples, since I don't know which bike has been repaired which day. But I know that by classifying each sample normal or anormal, I can sum the number of bikes that have been classified anormal during a month and compare that number to the monthly report.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know how one usually deal with it, or how this is called so I can read research paper on the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Exemple of samples :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bikeid, day,         feature1, feature2...&#xA;1,      2016-01-01,  0.6,      -0.2 &#xA;2,      2016-01-01,  0.5,      -0.8&#xA;1,      2016-01-02,  0.7,      -0.1&#xA;2,      2016-01-02,  0.9,       1&#xA;   ...&#xA;1,      2016-01-31, -0.32,     -0.45&#xA;2,      2016-01-31, -0.5,      -0.8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example of label: 3456 bicycle repairs in January. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the shape of the data is irrelevant, what is important is that the labels are not about one sample but a group of samples. &lt;/p&gt;&#xA;" OwnerUserId="21251" LastEditorUserId="21251" LastEditDate="2016-07-08T09:10:39.197" LastActivityDate="2016-07-08T09:10:39.197" Title="how to learn from unlabeled samples but labeled group of samples?" Tags="|classification|anomaly-detection|labels|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12663" PostTypeId="2" ParentId="12643" CreationDate="2016-07-08T09:13:03.633" Score="1" Body="&lt;p&gt;You may use a sampling based method, in which you get an approximation of the actual result by combining the partial results for samples extracted from your data. There are several methods based on sampling, but you could take a look to the following link as an example, so you can understand what I am talking about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pike.cs.ucla.edu/~weiwang/paper/CIMCV06.pdf&quot; rel=&quot;nofollow&quot;&gt;A Fast Approximation to Multidimensional Scaling&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-07-08T09:13:03.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12664" PostTypeId="2" ParentId="989" CreationDate="2016-07-08T10:10:54.817" Score="22" Body="&lt;p&gt;SVM solves an optimization problem of quadratic order.&lt;/p&gt;&#xA;&lt;p&gt;I do not have anything to add that has not been said here. I just want to post a link the sklearn page about &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVC&lt;/a&gt; which clarifies what is going on:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The implementation is based on libsvm. The fit time complexity is more&#xA;than quadratic with the number of samples which makes it hard to scale&#xA;to dataset with more than a couple of 10000 samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;If you do not want to use kernels, and a linear SVM suffices, there is &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LinearSVC&lt;/a&gt;. You'll have to normalize your data though, in case you're not doing so already, because it applies regularization to the intercept coefficient, which is not probably what you want. It means if your data average is far from zero, it will not be able to solve it satisfactorily.&lt;/p&gt;&#xA;&lt;p&gt;What you can also use is stochastic gradient descent to solve the optimization problem. Sklearn features &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SGDClassifier&lt;/a&gt;. You have to use &lt;code&gt;loss='epsilon_insensitive'&lt;/code&gt; to have similar results to linear SVM. See the documentation. I would only use gradient descent as a last resort though because it implies much tweaking of the hyperparameters in order to avoid getting stuck in local minima. Use &lt;code&gt;LinearSVC&lt;/code&gt; if you can.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2021-08-11T10:26:27.617" LastActivityDate="2021-08-11T10:26:27.617" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="12665" PostTypeId="2" ParentId="12645" CreationDate="2016-07-08T10:18:28.450" Score="1" Body="&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df = pd.DataFrame([[1, 2, np.nan],&#xA;...                    [np.nan, 3, 4],&#xA;...                    [1, 2,      3]])&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; df&#xA;    0  1   2&#xA;0   1  2 NaN&#xA;1 NaN  3   4&#xA;2   1  2   3&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; df.count(axis=1)&#xA;0    2&#xA;1    2&#xA;2    3&#xA;dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-07-08T10:18:28.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12666" PostTypeId="2" ParentId="12295" CreationDate="2016-07-08T10:34:56.933" Score="1" Body="&lt;p&gt;I would go about this problem using the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a poly-line for the baseline trip.&lt;/li&gt;&#xA;&lt;li&gt;Create a buffer zone (closed polygon) around this poly-line using a small radius, say 20 meters.&lt;/li&gt;&#xA;&lt;li&gt;Using the points in the second trip, calculate the fraction of points that lie outside the buffer zone.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A fraction of zero means that all points from the second trip lie inside the first trip's buffer zone, while a fraction of one means that all lie outside.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use this fraction as a measure of dissimilarity, where 1 means that both trips are completely dissimilar and 0 means completely similar (within the given buffer radius). For completeness, you might want to reverse the poly-line roles and calculate the &quot;reverse&quot; similarity. Your final similarity score could then be the product of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I usually implement my algorithms in C# using .NET and for these purposes I use Microsoft's System.Spatial NuGet package. Here you can find methods like STBuffer and STContains that will greatly help to implement this.&lt;/p&gt;&#xA;" OwnerUserId="3439" LastEditorUserId="3439" LastEditDate="2016-07-10T11:10:30.080" LastActivityDate="2016-07-10T11:10:30.080" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12667" PostTypeId="1" AcceptedAnswerId="12669" CreationDate="2016-07-08T11:50:14.750" Score="2" ViewCount="5598" Body="&lt;p&gt;If I have some data regarding the occurence of an event on a certain date and some other variables regarding it (think fe.: I have data on which dates it rained, and some addtitional data like temperature, atmospheric pressure etc.), which is the most appropriate model for predicting on which day the event is going to happen again? Or, to be more precise, I'd like to predict the frequency of said event, to know in how many days it's going to occur again. I mostly use Python with the numpy, sklearn libraries, and I'm interested which of its models fits my use case best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2017-08-18T10:24:55.817" Title="Date prediction - periodic recurrence" Tags="|machine-learning|python|scikit-learn|prediction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12669" PostTypeId="2" ParentId="12667" CreationDate="2016-07-08T13:10:48.290" Score="2" Body="&lt;p&gt;You should read that : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/&quot; rel=&quot;nofollow&quot;&gt;http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and take a look at that : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARIMA.html&quot; rel=&quot;nofollow&quot;&gt;http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARIMA.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;endog&lt;/code&gt; argument is your time serie and the &lt;code&gt;exog&lt;/code&gt; is your other data&lt;/p&gt;&#xA;" OwnerUserId="20945" LastActivityDate="2016-07-08T13:10:48.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12670" PostTypeId="1" CreationDate="2016-07-08T15:29:17.333" Score="2" ViewCount="76" Body="&lt;p&gt;I am trying to solve a sparsity-promoting optimization problem. It is well known that the L1 norm is a good surrogate to the L0 norm, and it is studied in (Candes et al, 2008: Enhancing sparsity by reweighted L1 minimization &lt;a href=&quot;https://arxiv.org/abs/0711.1612&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/0711.1612&lt;/a&gt;) that a better approximation can be obtained by successively reweighting the L1 norm penalization term, where the result of a step gives the new weights for the next iteration.&lt;/p&gt;&#xA;&lt;p&gt;So far I am implementing this directly in my code but I wonder if there is any software/optimization package that does this automatically.&lt;/p&gt;&#xA;" OwnerUserId="21260" LastEditorUserId="303" LastEditDate="2020-10-19T07:11:58.197" LastActivityDate="2024-01-03T02:00:45.083" Title="Software for reweighted L1 minmization?" Tags="|regression|feature-selection|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12671" PostTypeId="1" CreationDate="2016-07-08T15:53:33.720" Score="2" ViewCount="378" Body="&lt;p&gt;There's tons of material online but yet I can't reconcile the different definitions for recommender system methodologies / strategies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think we can identify several axes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.carleton.edu/cs_comps/0607/recommend/recommender/modelbased.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;memory&lt;/em&gt; vs &lt;em&gt;model&lt;/em&gt; based;&lt;/a&gt; &lt;em&gt;Model based&lt;/em&gt; models do not need to store historical data such as user ratings or proxies (implicit data) such as purchase behavior, views, etc. &lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;product based&lt;/em&gt; vs &lt;a href=&quot;http://guidetodatamining.com/assets/guideChapters/DataMining-ch3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;collaborative filtering&lt;/em&gt;&lt;/a&gt;, aka user-based filtering. &lt;em&gt;Collaborative filtering&lt;/em&gt; relies on user preferences data (implicit or explicit - explicit when users explicitly provide ratings, likes, stars). &lt;em&gt;Product based&lt;/em&gt; rely uniquely on product features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;collaborative filtering&lt;/em&gt; comprises &lt;a href=&quot;https://stackoverflow.com/questions/16372191/whats-difference-between-collaborative-filtering-item-based-recommendation-and&quot;&gt;&lt;em&gt;item based&lt;/em&gt;&lt;/a&gt;, which is totally based on user-item rankings, and &lt;em&gt;content-based recommendation&lt;/em&gt;, which exploits product features and user preferences (although &lt;a href=&quot;https://www.snet.tu-berlin.de/fileadmin/fg220/courses/SS11/snet-project/recommender-systems_asanov.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;other&lt;/a&gt; &lt;a href=&quot;http://ac.els-cdn.com/S0950705113001044/1-s2.0-S0950705113001044-main.pdf?_tid=99dd187a-473f-11e6-8d54-00000aab0f6c&amp;amp;acdnat=1468225097_64f218616122bd940356a774d632f761&quot; rel=&quot;nofollow noreferrer&quot;&gt;sources&lt;/a&gt; put content based filtering outside the family of collaborative filtering)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? Anything (big) missing?&lt;/p&gt;&#xA;" OwnerUserId="1244" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-11T08:52:56.307" Title="Taxonomy of recommender system methodologies" Tags="|recommender-system|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12673" PostTypeId="1" CreationDate="2016-07-08T18:36:51.597" Score="4" ViewCount="1683" Body="&lt;p&gt;&lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/mnsc.42.7.1082&quot; rel=&quot;nofollow&quot;&gt;Many&lt;/a&gt; &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1111/1467-9876.00109/abstract&quot; rel=&quot;nofollow&quot;&gt;have&lt;/a&gt; &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0377221703005484&quot; rel=&quot;nofollow&quot;&gt;shown&lt;/a&gt; the effectiveness of using neural networks for modeling time series data, and described the transformations required and limitations of such an approach. R's &lt;code&gt;forecast&lt;/code&gt; &lt;a href=&quot;https://cran.r-project.org/web/packages/forecast/forecast.pdf&quot; rel=&quot;nofollow&quot;&gt;package&lt;/a&gt; even implements one approach to this in the &lt;code&gt;nnetar&lt;/code&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my reading, all of these approaches are for modeling a single outcome variable based on its past observations, but I'm having trouble finding a description of a neural-network-based approach that also incorporates independent predictor variables (a sort of ARIMAx analogue for neural networks). I've found references to Nonlinear autoregressive exogenous models (NARX), which seem like they should be what I'm looking for, but all the reading I've been able to find talks more about using this approach for multi-step-ahead prediction of a univariate series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right direction on this? For bonus points, does anyone know of an implementation of what I'm looking for in R?&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2016-07-08T23:46:18.560" LastActivityDate="2016-07-09T15:26:53.453" Title="Neural Network Timeseries Modeling with Predictor Variables" Tags="|r|neural-network|time-series|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12674" PostTypeId="1" AcceptedAnswerId="12676" CreationDate="2016-07-08T20:06:04.873" Score="1" ViewCount="108" Body="&lt;p&gt;Consider one dependent variable 'Y' and 10 independent variables or features- X1, X2, X3, ... X10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to create a non-linear polynomial regression model such that-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Y ~ a1.X1^b1 + a2.X2^b2 + .... + a10.X10^b10&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering is there any algorithm that will determine best possible values for powers of independent variables that is values of b1, b2, ... b10 from data.&lt;/p&gt;&#xA;" OwnerUserId="10049" LastEditorUserId="10049" LastEditDate="2016-07-08T20:12:56.030" LastActivityDate="2016-07-08T20:38:14.760" Title="How to decide power of independent variables in case of non-linear polynomial regression?" Tags="|machine-learning|predictive-modeling|statistics|algorithms|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12675" PostTypeId="2" ParentId="12491" CreationDate="2016-07-08T20:33:07.527" Score="2" Body="&lt;p&gt;I've looked at the approach and I'd select K by trying a range, i.e. 5k, 10k, 15k etc and then exploring the range in which the best result falls, say the best is 15k then I might do 13, 14, 15, 16, 17 and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've not found any pre-processing to be effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answering the comment:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using LogisticRegression, SVM, Neural Networks, RandomForests, Multinomial NB, Extra Trees. All except Neural Networks using the implementations in sklearn. PyBrain for the NN.&lt;/p&gt;&#xA;" OwnerUserId="17585" LastEditorUserId="17585" LastEditDate="2016-07-11T20:28:49.173" LastActivityDate="2016-07-11T20:28:49.173" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12676" PostTypeId="2" ParentId="12674" CreationDate="2016-07-08T20:38:14.760" Score="1" Body="&lt;p&gt;If all you care about is the quality of predictions (as opposed to explanatory power), skip linear models altogether and use gradient boosted trees instead. Gradient boosting can generally learn polynomial splines with ease, and you don't have to manually make a bunch of polynomial predictors yourself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, gradient boosting is implemented in Python's scikit-learn library, R's caret library, and Java/Scala's Weka library. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-08T20:38:14.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12677" PostTypeId="1" CreationDate="2016-07-08T20:42:27.437" Score="2" ViewCount="538" Body="&lt;p&gt;Say I'm attempting to improve click-through rates on videos on my website. I've been reading the literature on contextual bandits and came across the Microsoft MWT &lt;a href=&quot;https://github.com/Microsoft/mwt-ds/wiki&quot; rel=&quot;nofollow&quot;&gt;white paper&lt;/a&gt;. I believe this is the right method for this case. However, I'm a bit confused on the details of the policy exploration and policy training (Section 3.2 and 3.3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question: Is there a difference between a policy and exploration policy? I want to say no, but want confirmation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second question pertains to learning a policy. As an example, say that I initially used an $\epsilon$-greedy exploration policy and collected user click-log data. The paper states that (offline) I want to find a policy out of all allowed policies that approximately maximizes the estimated expected reward (Inverse Propensity Scoring estimator)  $\mu_{ips}(\pi)$ for policy $\pi$ -- Eqn. 4 in Section 3.3. The policy chooses an action $a\in A$ (e.g. a video to serve to the user) given a context $x$ (e.g. user attributes and perhaps attributes of the article/video currently being watched) and an outcome (click or no-click) is observed. The authors propose to reduce the policy training to a cost-sensitive classification problem (e.g. logistic regression, decision trees, or neural nets) where each policy $\pi$ is viewed as a classifier (i.e. for context $x_{i}$, a given policy $\pi$ chooses action $a_{i}$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it's not clear to me what the classification task is. Does it mean that I can use something like a neural net whose output is the policy that minimizes the cost? Is this already performed in ML packages like &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit&quot; rel=&quot;nofollow&quot;&gt;VW&lt;/a&gt;? If so, how?   &lt;/p&gt;&#xA;" OwnerUserId="14150" LastEditorUserId="14150" LastEditDate="2016-07-09T17:30:08.487" LastActivityDate="2016-07-09T17:30:08.487" Title="How do I train a contextual bandit policy?" Tags="|machine-learning|optimization|online-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12679" PostTypeId="1" CreationDate="2016-07-08T22:27:54.193" Score="5" ViewCount="7411" Body="&lt;p&gt;I've been tearing my hair out for the last 3 days and I just can't get it to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently got a GTX 1070 and I went to a fresh install with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ubuntu 16.04/15.10/15.04&lt;/li&gt;&#xA;&lt;li&gt;PPA drivers nvidia-367&lt;/li&gt;&#xA;&lt;li&gt;Cuda 7.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Somehow every step of the way I couldn't get CUDA to detect my GPU despite following various instructions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://askubuntu.com/questions/693145/installing-cuda-7-5-toolkit-on-ubuntu-15-10&quot;&gt;https://askubuntu.com/questions/693145/installing-cuda-7-5-toolkit-on-ubuntu-15-10&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.pugetsystems.com/labs/hpc/GTX-1080-CUDA-performance-on-Linux-Ubuntu-16-04-preliminary-results-nbody-and-NAMD-803/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.pugetsystems.com/labs/hpc/GTX-1080-CUDA-performance-on-Linux-Ubuntu-16-04-preliminary-results-nbody-and-NAMD-803/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone been able to get 1070/1080 running CUDA and tensorflow correctly and can guide me thru this painful process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I can't do Ubuntu 14.04 as it doesn't register my wifi card, which then I can't do anything)&lt;/p&gt;&#xA;" OwnerUserId="21272" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:35.510" LastActivityDate="2018-04-24T19:42:35.720" Title="Has anyone been able to run Tensorflow with GTX 1070 GPU on Ubuntu 16.04/15.10/15.04?" Tags="|tensorflow|gpu|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12681" PostTypeId="1" AcceptedAnswerId="12682" CreationDate="2016-07-08T23:07:34.640" Score="2" ViewCount="183" Body="&lt;p&gt;As the title says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where does the random in Random Forests come from?&lt;/p&gt;&#xA;" OwnerUserId="21268" LastActivityDate="2016-07-08T23:38:12.790" Title="Where does the random in Random Forests come from?" Tags="|machine-learning|random-forest|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12682" PostTypeId="2" ParentId="12681" CreationDate="2016-07-08T23:38:12.790" Score="5" Body="&lt;p&gt;For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-08T23:38:12.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12683" PostTypeId="2" ParentId="12673" CreationDate="2016-07-09T00:09:55.393" Score="0" Body="&lt;p&gt;I haven't worked on many time series problems, so take my answer with a grain of salt. Having said that, it's not clear why you can't you just treat all predictors the same, in other words treat previous values of the univariate target series and independent predictors as regular predictors. I don't see what is special about the element of time that makes it especially difficult to train/learn like any other type of ordinary predictor. If time is truly important, then the neural network will demonstrate that importance by weighting your multivariate lagged variables accordingly during training. This sort of reasoning would apply to any type of model, not just neural networks. For example, the same could be said of gradient boosting. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-09T00:09:55.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12686" PostTypeId="2" ParentId="12673" CreationDate="2016-07-09T10:11:12.987" Score="2" Body="&lt;p&gt;I think this thesis &lt;a href=&quot;https://dspace.mit.edu/bitstream/handle/1721.1/99565/924315586-MIT.pdf?sequence=1&quot; rel=&quot;nofollow&quot;&gt;https://dspace.mit.edu/bitstream/handle/1721.1/99565/924315586-MIT.pdf?sequence=1&lt;/a&gt; is a good starting point for building a model for time series forecasting with so called leading indicators and different machine learning models. Basically you need just to training sets where your inputs are past values of the variable you wish to predict plus the additional indicators (&lt;em&gt;i.e. past_value_t-6, past_value_t-5 ..., past_value_t-1, additional_variable_1, additional_variable_2, ..., additional_variable_n&lt;/em&gt;) and the output is either single value in future (&lt;em&gt;t_1&lt;/em&gt;) or multiple values (&lt;em&gt;t_1, t_2, ... t_n&lt;/em&gt;), in case you wish to predict for example the next twelve moths' sales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible to use &lt;em&gt;recursive strategy&lt;/em&gt; to generate future values for your prediction variable. In this case you generate next value for the prediction value based on your previous predicted value. However, at least based on my experience the first mentioned &lt;em&gt;direct strategy&lt;/em&gt; is way more efficient.&lt;/p&gt;&#xA;" OwnerUserId="21281" LastEditorUserId="21281" LastEditDate="2016-07-09T15:26:53.453" LastActivityDate="2016-07-09T15:26:53.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12687" PostTypeId="1" AcceptedAnswerId="12691" CreationDate="2016-07-09T11:18:08.200" Score="4" ViewCount="3523" Body="&lt;p&gt;The precision is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{precision} = \frac{\text{true positive}}{\text{true positive} + \text{false positive}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any definition what this value should be if there is no positive classification (but of course positive elements)?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-07-09T19:37:26.120" Title="What is the definition of precision for no positive classifications?" Tags="|classification|definitions|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12689" PostTypeId="2" ParentId="12687" CreationDate="2016-07-09T19:14:52.153" Score="1" Body="&lt;p&gt;I have worked on several practical projects. In these projects we designed classifiers to deal with practical problems. Before I started working on practical I used to be very interested in : overall classification accuracy, sensitivity and specificity. In practical life, usually they care in two values: &#xA;Positive predictive value &quot; precision &quot; and positive percentage PP &quot;how many samples are predicted as positive&quot;.&#xA;The pp value measures the usefulness of this practical solution while the ppv value measures the quality of your solution. &#xA;Based on my understanding if the pp value is zero the ppv value is not defined and the solution itself is not valid &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2016-07-09T19:30:39.637" LastActivityDate="2016-07-09T19:30:39.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12691" PostTypeId="2" ParentId="12687" CreationDate="2016-07-09T19:37:26.120" Score="2" Body="&lt;p&gt;I just found that &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;sklearn.metrics.precision_score&lt;/code&gt;&lt;/a&gt; handles it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; sklearn.metrics.precision_score(y_true, y_pred)&#xA;/home/moose/.local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1074:&#xA;  UndefinedMetricWarning: Precision is ill-defined and being&#xA;    set to 0.0 due to no predicted samples.&#xA;    'precision', 'predicted', average, warn_for)&#xA;0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So they give a warning and set it to 0.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-07-09T19:37:26.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12692" PostTypeId="2" ParentId="155" CreationDate="2016-07-09T20:42:50.700" Score="6" Body="&lt;p&gt;I created a github repo for this. The datasets are not big, but are minimal examples meant to practice and explore predictive-modeling techniques which can then be extended to big datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ben519/MLPB&quot; rel=&quot;noreferrer&quot;&gt;Machine Learning Problem Bible (MLPB)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cool/unique thing about this repo is that every problem is tagged with tags like [multi-class], [unbalanced-data], [regression], etc. making it easy to find certain types of problems/datasets.&lt;/p&gt;&#xA;" OwnerUserId="18246" LastActivityDate="2016-07-09T20:42:50.700" CommentCount="0" CommunityOwnedDate="2016-07-09T20:42:50.700" ContentLicense="CC BY-SA 3.0" />
  <row Id="12693" PostTypeId="1" AcceptedAnswerId="13480" CreationDate="2016-07-09T22:05:47.747" Score="6" ViewCount="1119" Body="&lt;p&gt;I would like to train a convnet to do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Input is a set of single channel (from black to tones of grey to white) pictures with a given object, let's say cars.&lt;/li&gt;&#xA;&lt;li&gt;Target is, for every picture in the set, the same picture, however pixels are either black or white. The pixels corresponding to the car object are in white (i.e. intensity 255) and the pixels corresponding to the background are black (i.e. intensity 0).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After training I would like to feed the net with pictures of cars and I would like the prediction - the ideal prediction at least - to be a picture with pixels either black or white, where white corresponds to the object and black to the background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that the input layer is a 2D convolutional layer and the output layer is also a 2D convolutional layer, each one with as many neurons as pixels in the pictures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can anyone please explain what kind of network architecture would accomplish just that?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be either the architecture (in theory) or implemented in code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect to tweak it, but it would be nice not to start from scratch.&lt;/p&gt;&#xA;" OwnerUserId="9197" LastEditorUserId="29169" LastEditDate="2021-02-16T15:40:56.540" LastActivityDate="2021-02-16T15:40:56.540" Title="Is there a known convolutional net architecture to calculate object masks for images?" Tags="|deep-learning|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12694" PostTypeId="1" AcceptedAnswerId="12699" CreationDate="2016-07-09T22:36:15.500" Score="0" ViewCount="587" Body="&lt;p&gt;So much of what we export is in CSV and JSON files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any useful tools you know of that can automatically perform data analysis on flat file formats. For example &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Basic statistics if numeric types: avg, stdev, mode, median&lt;/li&gt;&#xA;&lt;li&gt;Column type and Cardinality detection&lt;/li&gt;&#xA;&lt;li&gt;Find relationships between columns, if column A is X, then column B is always Y&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any of these things would be useful, even if there'd be an intermediary step of just loading the flat file into some software...&lt;/p&gt;&#xA;" OwnerUserId="21294" LastActivityDate="2016-07-10T15:31:40.500" Title="Tools for ML on csv files and jsons" Tags="|machine-learning|data-mining|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12695" PostTypeId="1" AcceptedAnswerId="12736" CreationDate="2016-07-09T22:50:26.543" Score="4" ViewCount="3173" Body="&lt;p&gt;I have just &lt;em&gt;very recently&lt;/em&gt; started to develop an interest in machine learning, and I have a particular problem in mind that I would like to start to explore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to train a system to automatically classify various attributes of an item, based on what's in a string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a long list of various mutual funds, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Ticker  Fund Name&#xA;------  ---------&#xA;ABNAX   ABC Bond Fund, Inc: Bond Inflation Strategy&#xA;ALYSX   ABC Bond Fund, Inc: Credit Long/Short Portfolio; Advisor Class&#xA;AGRXX   DEF Bond Fund, Inc: Government Reserves Portfolio; Class 1 Shares&#xA;HIYYX   FGH Bond Fund, Inc: High Yield Portfolio; Advisor Class Shares&#xA;HIYAX   FGH Bond Fund, Inc: High Yield Portfolio; Class A Shares&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;… And so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a large data set that contains &quot;complete&quot; classifications, which have Fund Names similar to the ones above, and – in addition – a human has already given the training set items certain attributes. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AIISX   Allianz Funds Multi-Strategy Trust: AllianzGI International Small-Cap Fund; Class R6 Shares&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which will have the associated attributes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Strategy: Multi-Strategy&#xA;Geography: International&#xA;Capitalization: Small-Cap&#xA;Share class: R6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The challenge for the machine learning system will be to assign the right value to an attribute, when there are values &quot;competing&quot; on the same attribute. Let's say that a certain fund can have &lt;code&gt;Strategy: Long-Short&lt;/code&gt; and &lt;code&gt;Strategy: High Yield&lt;/code&gt; at the same time – and both terms are present in the &lt;em&gt;Fund name&lt;/em&gt;. The system should select the right one, based on exposure to historical bias present in the training data set.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am interested in getting a grasp of which machine learning methods and algorithms that would be able to &quot;learn&quot; how to classify an item, based on a large set of examples with human-classified attributes, as indicated above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a complete beginner to machine learning, except for some basic knowledge of statistics, so I would just like to be pointed in a general direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can/should this be accomplished with something like multiple regression, or are we looking at something else? Is some sort of natural language processing needed – or is basic keyword pattern recognition enough?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, which terminology or labeled area of expertise would summarize this problem description?&lt;/p&gt;&#xA;" OwnerUserId="21291" LastEditorUserId="21291" LastEditDate="2019-08-23T08:07:29.053" LastActivityDate="2019-08-23T08:07:29.053" Title="Machine Learning Identification and Classification, based on string contents: General advice" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12696" PostTypeId="2" ParentId="12693" CreationDate="2016-07-09T23:17:51.070" Score="3" Body="&lt;p&gt;this is a typical image segmentation problem. You need to find a continuous blob in the image which is the segment you are looking for. A well-known architecture for this problem is the U-net architecture, and this is the link to the paper: &lt;a href=&quot;http://arxiv.org/abs/1505.04597&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1505.04597&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The architecture is called u-net because it has a contracting path (where the size of the input matrix is reduced) and an expanding path (where the size is enlarged again so that the output is more or less similar in size to the input).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The contracting path resembles most closely the typical convnet architecture, and it repeats the following pattern: a) convolutional layers (in this case using a 3x3 kernel), b) a rectified linear unit (RELU) on the convolutional layer output, and c) a pooling layer where the image is halved in every dimension. As you go down the contracting path, the number of feature channels is doubled every time the input goes through a convolutional layer, but the size is reduced every time it goes through a pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expanding path is more or less symmetrical, with upsampling layers that increase (double) the size of the input, followed by a typical convolutional layer and a rectified linear unit (RELU). While you go up the expanding path, the number of feature channels is reduced in the convolutional layers, until you end up with a single matrix (array) that corresponds to the output image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I look forward to implementing this architecture in the next few days, and I am sure my understanding will increase as a result. If I find out anything additional worth posting, I will go ahead and post it.&lt;/p&gt;&#xA;" OwnerUserId="9197" LastEditorUserId="9197" LastEditDate="2016-07-10T21:46:36.220" LastActivityDate="2016-07-10T21:46:36.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12697" PostTypeId="1" CreationDate="2016-07-10T04:42:55.197" Score="3" ViewCount="3624" Body="&lt;p&gt;What is the standard or what method do you use to select part of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ex: Using random forest, I got the following feature importances:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a : 25.4884726&#xA;b : 17.2736393&#xA;c : 12.3493490&#xA;d : 8.9383737&#xA;e : 8.1083837&#xA;f : 6.8272717&#xA;g : 4.1999999&#xA;...&#xA;...&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For final prediction, you need to select only few features from the above. How do you decide the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;No. of features that you choose?&lt;/li&gt;&#xA;  &lt;li&gt;Does the score matter when you choose?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="21155" LastEditorUserId="21155" LastEditDate="2016-07-10T07:07:33.917" LastActivityDate="2017-03-23T07:27:40.717" Title="Feature Importance - How to choose the number of best features?" Tags="|feature-selection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12699" PostTypeId="2" ParentId="12694" CreationDate="2016-07-10T15:26:08.083" Score="1" Body="&lt;p&gt;In &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; you can load the csv file easily by using the method &lt;a href=&quot;http://www.cookbook-r.com/Data_input_and_output/Loading_data_from_a_file/&quot; rel=&quot;nofollow noreferrer&quot;&gt;read.csv&lt;/a&gt; and get the summary of the data using the method &lt;code&gt;summary&lt;/code&gt;, you will get the most of the basic statistics of each columns like &lt;code&gt;Min&lt;/code&gt;, &lt;code&gt;1st Qu&lt;/code&gt;, &lt;code&gt;Median&lt;/code&gt;, &lt;code&gt;Mean&lt;/code&gt;, &lt;code&gt;3rd Qu&lt;/code&gt;, &lt;code&gt;Max&lt;/code&gt; and &lt;code&gt;NA&lt;/code&gt; counts for numeric/integer rows and for string columns you will also get the some statics of count of different type of string if it has repeated multiple i.e factors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample code:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Loading the csv file and check the few datasets:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- read.csv(&quot;SampleData.csv&quot;)&#xA;&amp;gt; head(df)&#xA;  name age       Type&#xA;1    A  34     Active&#xA;2    B  56 Cancelled &#xA;3    C  12     Active&#xA;4    D  32 Cancelled &#xA;5   Z   34     Active&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Basic statistics if numeric types: avg, stdev, mode, median&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(df)&#xA; name        age               Type  &#xA; A :1   Min.   :12.0   Active    :3  &#xA; B :1   1st Qu.:32.0   Cancelled :2  &#xA; C :1   Median :34.0                 &#xA; D :1   Mean   :33.6                 &#xA; Z :1   3rd Qu.:34.0                 &#xA;        Max.   :56.0                 &#xA;&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Column type and Cardinality detection&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sapply(df, class)&#xA;   name       age      Type &#xA;  &quot;factor&quot; &quot;integer&quot;  &quot;factor&quot; &#xA;&amp;gt;&#xA;&amp;gt; nrow(df)&#xA;[1] 5&#xA;&amp;gt; ncol(df)&#xA;[1] 3&#xA;&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find relationships between columns, if column A is X, then column B is always Y&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For this you have to do some Basic calculation and you can go through this cookbook: &lt;a href=&quot;http://www.cookbook-r.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cookbook-r.com/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qEAqC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;I have attached Sample CSV Image here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="17116" LastEditDate="2016-07-10T15:31:40.500" LastActivityDate="2016-07-10T15:31:40.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12700" PostTypeId="2" ParentId="12697" CreationDate="2016-07-10T16:11:31.020" Score="-2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;No. of features that you choose?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Its depends on the number of classes you have to predict, let's explain this with example: consider we have 5 classes(labels) in our dataset and we choose only one feature, so tree has only one parent node and two leaf node which will accommodate only two classes and hence the accuracy decrease sharply, similarly if we choose only two features then we can only accommodate three classes and so one, so you have to consider more number of features as of classes and there after you have to test it with accuracy till there is no change in accuracy by adding extra features in the model. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does the score(Variable Importance) matter when you choose?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Yes, the score matter when deciding the features that you choose, since its depends on the Variable Importance of a feature is computed as the average decrease in model accuracy on the out of bag samples when the values of the respective feature are randomly permuted, so if you choose only the lower score variables for features then the accuracy sharply decreases.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17116" LastActivityDate="2016-07-10T16:11:31.020" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12701" PostTypeId="1" CreationDate="2016-07-10T17:05:41.613" Score="1" ViewCount="143" Body="&lt;p&gt;I'm trying to train a simple neural net but I'm unsure of the correct backprop value to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using a 2 layer NN (1 input, 1 hidden and 1 output layer), using ReLU activations. Here is my h2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;h1 = np.maximum(0, X.dot(W1) + b1.T)&#xA;h2 = h1.dot(W2) + b2.T&#xA;score = h2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, if I were to backprop my score with respect to the loss function (cross entropy loss), what should be the correct computation I should use? I'm under the impression that this should always be 1 or at least usually equals to 1 since the final output is linearly related to the loss? On this writing, it doesn't really make sense. But if that's not the case, then what would be the correct computation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the score has not been directly linked to the cross entropy loss function, as I have not exponentiated the score matrix and do the necessary additions. If I were to link it to the cross entropy loss, then I'd do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss_mat = np.exp(scores) / np.sum(np.exp(scores))&#xA;loss_mat = np.log(losses)&#xA;loss_mat[:,y] -= 1 #Minus one for the correct label? Why do this?&#xA;                   #read this as one of the solutions&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="13527" LastActivityDate="2016-07-10T17:05:41.613" Title="Backpropagation: how do you compute the gradient of the final output with respect to any loss function?" Tags="|machine-learning|python|classification|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12702" PostTypeId="1" AcceptedAnswerId="12705" CreationDate="2016-07-10T17:06:25.437" Score="0" ViewCount="432" Body="&lt;p&gt;I'm building a convoluted neural network to teach a toy car, powered by a Raspberry Pi, how to drive based on incoming streams of frames from a webcam mounted on top of the car. The top half of each image is irrelevant. What matters is the curvature of the road, and this is in the bottom half. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've generated a substantial amount of data (about 40k records) by driving the car around myself and recording what I do (commands are left, right, and straight) and what the frames are. However my trained ConvNets aren't giving me the performance I'd hoped for. My experimentation with one of the deployed models confirms that it is indeed tricked by changes in the top half of the streaming images.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple solution is to programmatically cut the frames in half so that the neural net only receives relevant portions. However, deep neural nets are praised for their ability to learn features with (ideally) zero human future transformations, so I want to avoid this approach. This project is meant to be a learning experience for me so I want learn how to architect the ConvNet more effectively. Each training session takes several hours to run, so rather than try everything under the sun, I'd figure I'd reach out to the community here to narrow my focus of exploration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thought I have is to put a fully connected (FC) layer in front so that subsequent layers effectively only convolve over the relevant portion of the image. I think this could potentially work if this up front FC layer learned to assign very small weights to pixels in the top half of the image. Could this work? Are there better architectures?  &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29169" LastEditDate="2021-02-16T15:40:45.843" LastActivityDate="2021-02-16T15:40:45.843" Title="How to architect ConvNet to ignore top half of image" Tags="|machine-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12703" PostTypeId="2" ParentId="12679" CreationDate="2016-07-10T17:51:15.017" Score="1" Body="&lt;p&gt;This helped me run on GTX 1080 on a 16.04 ubuntu machine (driver 367.27):&#xA;&lt;a href=&quot;http://yangcha.github.io/Tensorflow/&quot; rel=&quot;nofollow&quot;&gt;http://yangcha.github.io/Tensorflow/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It basically says install CUDA 8RC, CuDNN 5 and build TensorFlow from source by following the instructions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume the GTX 1070 should behave the same to GTX 1080 on that regard.&lt;/p&gt;&#xA;" OwnerUserId="21315" LastActivityDate="2016-07-10T17:51:15.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12704" PostTypeId="2" ParentId="155" CreationDate="2016-07-10T18:54:14.123" Score="6" Body="&lt;p&gt;Eurostats   &lt;a href=&quot;http://ec.europa.eu/eurostat&quot; rel=&quot;noreferrer&quot;&gt;http://ec.europa.eu/eurostat&lt;/a&gt; and European Central Bank &lt;a href=&quot;https://www.ecb.europa.eu/stats/html/index.en.html&quot; rel=&quot;noreferrer&quot;&gt;https://www.ecb.europa.eu/stats/html/index.en.html&lt;/a&gt; provide a great variety of datasets which I use quite often in my work projects. &lt;/p&gt;&#xA;" OwnerUserId="21281" LastActivityDate="2016-07-10T18:54:14.123" CommentCount="0" CommunityOwnedDate="2016-07-10T18:54:14.123" ContentLicense="CC BY-SA 3.0" />
  <row Id="12705" PostTypeId="2" ParentId="12702" CreationDate="2016-07-10T19:48:01.010" Score="1" Body="&lt;p&gt;You could be right that ignoring top part of image would benefit the CNN. However, there is very little point in trying to &lt;em&gt;architect&lt;/em&gt; this - if your premise that the CNN will ignore irrelevant details in the top half is correct, then that will occur anyway and there is no standard NN architecture that will help that other than disconnecting the top half of network, which is going to be logically exactly the same as programmatically slicing the image, with the disadvantage of storing and calculating with twice as many parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should either programatically cut the image in half or do nothing to the image and rely on the CNN's inherent ability to give low weights to irrelevant details. If you do the latter, you may be able to get around the learning of incorrect details in the top half by augmenting your data - e.g. add some noise to images*, especially in the irrelevant top half. Perhaps horizontally flipping a few images (and reverse relevant targets for the control class) might be another useful augmentation. Some augmentations could also be useful if you just take the lower half of the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* Noise should be something close to variations that could be seen when in use. E.g. slurring pixels left or right might be reasonable. Inserting &quot;static&quot; probably is not.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-07-10T20:09:02.047" LastActivityDate="2016-07-10T20:09:02.047" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12706" PostTypeId="1" AcceptedAnswerId="12713" CreationDate="2016-07-10T20:31:28.220" Score="3" ViewCount="4447" Body="&lt;p&gt;Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;simple&lt;/li&gt;&#xA;&lt;li&gt;more biologically plausible&lt;/li&gt;&#xA;&lt;li&gt;no gradient to vanish&lt;/li&gt;&#xA;&lt;li&gt;better performance&lt;/li&gt;&#xA;&lt;li&gt;sparsity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why should not we forget about sigmoid/tanh for deep neural networks?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastEditorUserId="61970" LastEditDate="2020-03-09T16:12:55.223" LastActivityDate="2020-03-09T16:42:32.313" Title="Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?" Tags="|neural-network|deep-learning|activation-function|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12707" PostTypeId="1" AcceptedAnswerId="12708" CreationDate="2016-07-10T22:01:32.933" Score="1" ViewCount="1007" Body="&lt;p&gt;Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'.&#xA;I have following independent variables or features- Age, Height, Gender, Income etc. I trying to predict Attitude using decision tree classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Attitude ~ Age + Height + Gender + Income (Decision Tree)&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am getting &gt;90% accuracy for the when tree depth is 15. As tree is dividing on continuous variables (i.e. Age, Income and Height) again and again to get leaf with pure classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this problem of overfitting? Should I convert the continuous variables into categorical variables (like range classes)? &lt;/p&gt;&#xA;" OwnerUserId="10049" LastActivityDate="2016-07-10T22:37:21.877" Title="Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?" Tags="|machine-learning|classification|random-forest|decision-trees|preprocessing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12708" PostTypeId="2" ParentId="12707" CreationDate="2016-07-10T22:37:21.877" Score="3" Body="&lt;p&gt;There is no need to split continuous variables because the tree already does that automatically. The only way you can test for overfitting is by either using a holdout set or by doing cross validation. If you are overfitting, changing a continuous variable to a categorical variable likely won't make a difference. If you get the sense that you're overfitting, you should reduce the depth of your tree. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-10T22:37:21.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12709" PostTypeId="2" ParentId="11803" CreationDate="2016-07-11T03:53:04.980" Score="1" Body="&lt;p&gt;I feel that you need to address two things separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, You need to have access control for the users in your system. You can have access tokens attached to each user and file. Filter the files database before you process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, Ranking the documents I would suggest have some weight for a document weight and user weight which relative to the current browsing user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I can think of document weight and user weight as follows but they can be much more complex as per your system-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DocumentWeight = Number of Views/ Number of Users can Access&#xA;UserWeight = ## Relative to browsing user- Users in similar project will have higher weights&#xA;&#xA;DocumentScore = Sum over all viewed users{DocumentWeight x UserWeight}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can rank the documents, this will statistically pull up the documents you need. I hope this will be some help.&lt;/p&gt;&#xA;" OwnerUserId="10049" LastActivityDate="2016-07-11T03:53:04.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12710" PostTypeId="1" CreationDate="2016-07-11T05:24:14.213" Score="2" ViewCount="582" Body="&lt;p&gt;This is a really simple example where my training data has a single feature vector (1,2,3) and an equivalent target vector (1,2,3).  I can get &lt;code&gt;xgboost&lt;/code&gt; to build a regression tree that perfectly fits the data, but when I use it to make predictions, the predictions against the training data, the predictions don't match the target.  What gives?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train &amp;lt;- matrix(c(1,2,3))&#xA;target &amp;lt;- matrix(c(1,2,3))&#xA;bst &amp;lt;- xgboost(data=train, nrounds=1, label=target, eval_metric=&quot;rmse&quot;, max.depth=3, min_child_weight=1, gamma=0, lambda=0, alpha=0)&#xA;predict(bst, train)  # returns 0.65 0.95 1.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;xgb.plot.tree(&quot;f1&quot;, model=bst)&lt;/code&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mYfoG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mYfoG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgb.model.dt.tree(&quot;f1&quot;, model=bst)&#xA;    ID Feature Split Yes  No Missing Quality Cover Tree Yes.Feature Yes.Cover Yes.Quality No.Feature No.Cover No.Quality&#xA;1: 0-0      f1   2.5 0-1 0-2     0-1    1.50     3    0          f1         2        0.50       Leaf        1       0.75&#xA;2: 0-1      f1   1.5 0-3 0-4     0-3    0.50     2    0        Leaf         1        0.15       Leaf        1       0.45&#xA;3: 0-2    Leaf    NA  NA  NA      NA    0.75     1    0          NA        NA          NA         NA       NA         NA&#xA;4: 0-3    Leaf    NA  NA  NA      NA    0.15     1    0          NA        NA          NA         NA       NA         NA&#xA;5: 0-4    Leaf    NA  NA  NA      NA    0.45     1    0          NA        NA          NA         NA       NA         NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18246" LastActivityDate="2016-07-11T05:24:14.213" Title="Why does xgboost give this unexpected result?" Tags="|r|xgboost|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12712" PostTypeId="1" CreationDate="2016-07-11T10:55:31.973" Score="2" ViewCount="590" Body="&lt;p&gt;An undirected graph represents a database where nodes of the graph represent   tables, edges represent the joiner columns.  There are 100 databases( it means, 100 undirected graphs). We have to build a clustering model which can cluster these graphs into four groups based on attributes of nodes( that is the list of column names), attributes of edges ( that is the joiner column) and overall structure of the graph database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we build the clustering models ?&lt;/p&gt;&#xA;" OwnerUserId="7702" LastActivityDate="2016-07-11T10:55:31.973" Title="graph database and its clustering" Tags="|data-mining|clustering|social-network-analysis|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12713" PostTypeId="2" ParentId="12706" CreationDate="2016-07-11T11:18:15.163" Score="4" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In certain scenarios having an activation with mean 0 is important (so tanh makes sense).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Sigmoid activation in the output layer is still important for classification&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In 95% of the cases ReLU is much better though.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-11T11:18:15.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12715" PostTypeId="2" ParentId="12706" CreationDate="2016-07-11T11:51:12.940" Score="3" Body="&lt;p&gt;Sigmoid helps in controlling the activation unlike ReLu which blows up it up. Sigmoids don't overfit as much. &#xA;Have a look at &lt;a href=&quot;https://github.com/domluna/relu-vs-sigmoid/blob/master/ReLU%20vs%20Sigmoid.ipynb&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still would ask you to start with ReLu for training as it performs better most of the time.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-11T11:51:12.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12717" PostTypeId="1" CreationDate="2016-07-11T14:34:50.903" Score="1" ViewCount="3110" Body="&lt;p&gt;I am wondering how to convert caffe reference model trained with ImageNet (color pics) for grayscale image to save memory and to speed up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The filters in the caffe convolution layers are different for RGB but this can be, for example, averaged to form one channel filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my data is grayscale and speed at test (prediction) is important for me, this may help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can of course convert my grayscale images in RGB representation to reuse pretrained convolution layers as they are. But I want to avoid it if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this be possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How exactly one should combine filters? (Average?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After filters converted, weights between conv layers need to be re-trained?&lt;/p&gt;&#xA;" OwnerUserId="21337" LastActivityDate="2018-11-07T11:45:02.420" Title="Convert filters pre-trained with ImageNet to grayscale?" Tags="|machine-learning|deep-learning|image-classification|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12718" PostTypeId="2" ParentId="12717" CreationDate="2016-07-11T14:56:01.493" Score="1" Body="&lt;p&gt;I doubt either of your proposed approaches will work, these convolutional layers learn all kind of relationships between the colors which will be non-existent in grayscale, upsampling it to RGB makes your input space very different from the input space that ImageNet was trained on and averaging these filters of the trained ImageNet model makes a whole lot of assumptions that are unlikely to hold. The second approach does seem more promising than the first, so if you want to test an approach I would go for the second one. It's possible that it is at least a better initialization than randomly. I think you will end up needing a lot of data and computation anyway, so it might be better to start off with your own network shape and start from scratch.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-11T14:56:01.493" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12719" PostTypeId="1" CreationDate="2016-07-11T15:07:50.947" Score="1" ViewCount="288" Body="&lt;p&gt;I am no stats expert so any assistance will be greatly appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m working on a post-campaign analysis to determine if any measured sales lift (sales lift per audience) is significant between test and control treatment groups.   From what I have been reading, I believe you could potentially use an Anova or Ancova to run a post-campaign analysis to determine if any measured sales lift is above or below a standard deviation or margin of error? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, does this make sense?  I’m never run an ANOVA is SAS is there any sample code one can share?  &lt;/p&gt;&#xA;" OwnerUserId="12947" LastActivityDate="2016-07-12T14:18:57.267" Title="A/B Test: Determine if a Lift in Measured Sales is Significant" Tags="|ab-test|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12720" PostTypeId="1" AcceptedAnswerId="15039" CreationDate="2016-07-11T15:31:12.807" Score="4" ViewCount="662" Body="&lt;p&gt;While trying to find how one place (Military High School) like a school is similar to another one (Military H School or Military High S), I used the LV algorithm to find the string distance and then converted them into a percentage on a calibrated scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;stringdist(data[i,1],data[i,2],method = &quot;lv&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though this is admissible, I still need a more precise way to determine the threshold. The goal is to set a threshold of acceptance for a variation of a name from the standardized name, so that say if the threshold of this word is less than say 85%, it would be rejected. This exercise is part of data pre-processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="21341" LastActivityDate="2016-11-10T08:22:41.263" Title="Text Mining - Similarity among words to determine thresholds" Tags="|r|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12721" PostTypeId="1" CreationDate="2016-07-11T16:45:21.020" Score="86" ViewCount="65844" Body="&lt;p&gt;The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values. So far I have come across two models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;noreferrer&quot;&gt;LSTM&lt;/a&gt; (long short term memory; a class of recurrent neural networks)&lt;/li&gt;&#xA;&lt;li&gt;ARIMA&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have tried both and read some articles on them. Now I am trying to get a better sense on how to compare the two. What I have found so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;LSTM works better if we are dealing with huge amount of data and enough training data is available, while ARIMA is better for smaller datasets (is this correct?)&lt;/li&gt;&#xA;&lt;li&gt;ARIMA requires a series of parameters &lt;code&gt;(p,q,d)&lt;/code&gt; which must be calculated based on data, while LSTM does not require setting such parameters. However, there are some hyperparameters we need to tune for LSTM.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; One major difference between the two that I noticed while reading a great article &lt;a href=&quot;http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, is that ARIMA could only perform well on stationary time series (where there is no seasonality, trend and etc.) and you need to take care of that if want to use ARIMA&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Other than the above-mentioned properties, I could not find any other points or facts which could help me toward selecting the best model. I would be really grateful if someone could help me finding articles, papers or other stuff (had no luck so far, only some general opinions here and there and nothing based on experiments.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to mention that originally I am dealing with streaming data, however for now I am using &lt;a href=&quot;https://github.com/numenta/NAB&quot; rel=&quot;noreferrer&quot;&gt;NAB datasets&lt;/a&gt; which includes 50 datasets with the maximum size of 20k data points.&lt;/p&gt;&#xA;" OwnerUserId="20544" LastEditorUserId="21024" LastEditDate="2016-07-22T09:53:07.673" LastActivityDate="2023-08-13T13:43:30.073" Title="Time series prediction using ARIMA vs LSTM" Tags="|time-series|deep-learning|rnn|prediction|" AnswerCount="8" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12722" PostTypeId="2" ParentId="12649" CreationDate="2016-07-11T17:11:09.057" Score="1" Body="&lt;p&gt;Alternatively, I think you can use any profiler library to analyze the memory and CPU usage by your program.&#xA;There are many python libraries which can give you snapshot of memory and CPU usage by particular thread or process at millisecond interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may run the part of your program you want to monitor in a different sub-process using &lt;a href=&quot;https://docs.python.org/2/library/subprocess.html#subprocess.Popen&quot; rel=&quot;nofollow&quot;&gt;popen&lt;/a&gt; and monitor it's memory and CPU usage using it's PID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pypi.python.org/pypi/psutil&quot; rel=&quot;nofollow&quot;&gt;psutil&lt;/a&gt; I find good for such work. Though there are many others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this will help. &lt;/p&gt;&#xA;" OwnerUserId="10049" LastActivityDate="2016-07-11T17:11:09.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12723" PostTypeId="1" CreationDate="2016-07-11T17:36:41.240" Score="4" ViewCount="999" Body="&lt;p&gt;I'm building convoluted neural networks with TensorFlow. I've got the latest Mac Pro, use all of my cores, and yet it still takes me several hours to train a single network. I need to do grid search on many different tuning parameters, so this is not ideal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is for a personal project, so I want to save money by using the cloud. I have used AWS extensively, and I like the idea of using their spot market, however I've heard that 1) Amazon's GPU instances aren't compatible out of the box with TensorFlow (AWS's GPUs have CUDA 3.0 and TensorFlow requires 3.5) and 2) Amazon's GPUs are a bit slow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does Microsoft's Azure or Google's Google Compute Engine have easy-to-use GPUs when using TensorFlow? How do their GPUs compare to those of Amazon in terms of the relevant neural-net-training tech specs, prices, and ease of use? Are there smaller cloud providers that you would recommend instead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, roughly how much faster can I expect training to take on one of these virtual GPUs compared to a top of the line Mac Pro?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-23T21:41:21.223" Title="Public cloud GPU support for TensorFlow" Tags="|machine-learning|tensorflow|gpu|" AnswerCount="2" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12724" PostTypeId="1" AcceptedAnswerId="12725" CreationDate="2016-07-11T18:21:58.313" Score="3" ViewCount="693" Body="&lt;p&gt;I am optimizing some loss function using Gradient Descent method. I am trying it with different learning rates, but the objective function's value is converging to same exact point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this means that I am getting stuck in a local minima?, because the loss function is non-convex so it is less likely that I would converge to a global minima.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-07-12T06:47:37.403" Title="DIfferent learning rates converging to same minima" Tags="|optimization|gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12725" PostTypeId="2" ParentId="12724" CreationDate="2016-07-11T19:43:19.593" Score="5" Body="&lt;p&gt;This is the expected behavior. Different learning rates should converge to the same minimum if you are starting at the same location. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're optimizing a neural network and you want to explore the loss surface, &lt;strong&gt;randomize the starting parameters&lt;/strong&gt;. If you always start your optimization algorithm from the same initial value, you will reach the same local extremum unless you really increase the step size and overshoot.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="9420" LastEditDate="2016-07-12T04:30:26.290" LastActivityDate="2016-07-12T04:30:26.290" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12726" PostTypeId="2" ParentId="12720" CreationDate="2016-07-11T19:45:11.297" Score="1" Body="&lt;p&gt;In order to optimize the string distance threshold for maximizing classification accuracy in your data, if you don't already have a &lt;strong&gt;gold standard&lt;/strong&gt; dataset, you will want to label some of your data. Depending on your situation, this could be as easy as you yourself hand-labeling string pairs as &lt;em&gt;same&lt;/em&gt; or &lt;em&gt;notsame&lt;/em&gt;. If you need more robust data, you would obviously want to use several labelers and use some sort of procedure to ensure high inter-coder reliability (I discuss this at length under &lt;strong&gt;data quality&lt;/strong&gt; in this &lt;a href=&quot;https://datascience.stackexchange.com/questions/12265/categorizing-customer-emails/12268#12268&quot;&gt;answer&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, compute string distance as before on this data. Create a vector of threshold values, e.g., &lt;code&gt;thresholds &amp;lt;- seq(from = 0, to = 20, by = 1)&lt;/code&gt;, then for each threshold compute your performance metrics on your labeled data set (e.g., accuracy, precision, recall). Choose the threshold that maximizes your desired metric.&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-12T13:30:01.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12727" PostTypeId="1" CreationDate="2016-07-11T21:02:40.633" Score="6" ViewCount="18903" Body="&lt;p&gt;I was trying to load the below weblogic domain log (application error log) into Spark dataframe. I created a RDD and converted the RDD into dataframe. I was able to load the data successfully for the first two rows because the records are not spread over to multiple lines. But, for the third row (highlighted in bold), the record is spread over multiple lines and Spark assumes the continuation of the last field on the next line as new record. Can anybody suggest any workaround so that the multi-line data can be considered as part of a single record? Any pointers would be really helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scala commands: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;case class error(error_log_id: String,procedure_name: String,section_name: String,sei_user_id: String,error_timestamp: String,error_number: String,error_message: String,error_stack: String,error_backtrace: String,java_thread_id: String) &#xA;val errorLog = sc.textFile(&quot;errorLog.csv&quot;).map(_.split(&quot;#&quot;)).map(p =&amp;gt; error(p(0),p(1),p(2),p(3),p(4),p(5),p(6),p(7),p(8),p(9))).toDF() &#xA;errorLog.registerTempTable(&quot;error_log&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Weblogic error log: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;||20160704||00:30:50,595||4affd321959f27e41467606650584||com.seic.common.presentation.controller.ProcessControllerAction||[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'||INFO ||Entered performPreCondition for event=evtLocationContext event = evtLocationContext &#xA;||20160704||00:30:50,607||4affd321959f27e41467606650584||com.seic.desktop.relationshipmgmt.presentation.helper.ContextTreeHelper||[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'||INFO ||ContextEntityWrapperTO found in Session &#xA;**||20160704||01:58:32,182||d3ac1cac46b457251467611729203||com.seic.common.presentation.filter.JSPTrackerFilter||[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'||INFO ||JSPTrackerFilter  :    User Id = 8000551,    User Firm Id = 1, JSP Name = /SEIDesktop/jsp/common/maintainSession.jsp    , Date &amp;amp; Timestamp = 7/4/16 1:58 AM    ,Browser Name &amp;amp; Version = Mozilla/5.0 ||20160704||01:58:32,294||396c0a8e2470e7a21467611910768||com.seic.dataservices.impl.InstrumentSearchDoImpl||[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'||ERROR||ExceptionFactoryMsg:  com.seic.dataservices.lib.DataServiceSqlException - Error - A SQL exception was encountered while processing this request. - EX13 -  - q02_Desktop_MS1#20160704015832294#13 - With Additional Info:  (Error Code:  20010 -  ) - Caused By (java.sql.SQLException: ORA-20010: PO_ID(1000) INSTRUMENT_ID(1004915) PRICE_DATE(01-JUL-16) does not exist in Table PO_SECURITY_PRICE.  Price data not found.. &#xA;ORA-06512: at &quot;xxx_xxx.ERROR_PKG&quot;, line 502 &#xA;ORA-06512: at &quot;xxx_xxx.xx_PRICING&quot;, line 6221 &#xA;ORA-06512: at line 1 &#xA;) &#xA;com.seic.dataservices.lib.DataServiceSqlException - Error - A SQL exception was encountered while processing this request. - EX13 -  - q02_Desktop_MS1#20160704015832294#13 - With Additional Info:  (Error Code:  20010 -  ) - Caused By (java.sql.SQLException: ORA-20010: PO_ID(1000) INSTRUMENT_ID(1004915) PRICE_DATE(01-JUL-16) does not exist in Table PO_SECURITY_PRICE.  Price data not found.. &#xA;ORA-06512: at &quot;xxx_xxx.ERROR_PKG&quot;, line 502 &#xA;ORA-06512: at &quot;xxx_xxx.xx_PRICING&quot;, line 6221 &#xA;ORA-06512: at line 1 &#xA;) &#xA;                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) &#xA;                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) &#xA;                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) &#xA;                at java.lang.reflect.Constructor.newInstance(Constructor.java:513) &#xA;                at com.seic.framework.exceptions.ExceptionFactory.getSeiException(ExceptionFactory.java:536) &#xA;                at com.seic.framework.exceptions.ExceptionFactory.createException(ExceptionFactory.java:269) &#xA;                at com.seic.framework.exceptions.ExceptionFactory.createException(ExceptionFactory.java:172) &#xA;                at com.seic.dataservices.dm.InstrumentSearchDo.Get(InstrumentSearchDo.java:3796) &#xA;                at com.seic.dataservices.impl.InstrumentSearchDoPkFilterHelper.doGet(InstrumentSearchDoPkFilterHelper.java:401) &#xA;                at com.seic.dataservices.lib.DefaultValuesDataServiceHelper.doGet(DefaultValuesDataServiceHelper.java:34) &#xA;                at com.seic.dataservices.lib.DataServiceImpl.getList(DataServiceImpl.java:81) &#xA;                at com.seic.dataservices.lib.DataServiceImpl.get(DataServiceImpl.java:127) &#xA;                at com.seic.dataservices.lib.DataServiceImpl.get(DataServiceImpl.java:144) &#xA;                at com.seic.ads.setup.SearchServices.searchInstruments(SearchServices.java:213) &#xA;                at com.seic.ads.ejb.ADSServicesEjbBean.searchInstruments(ADSServicesEjbBean.java:699) &#xA;                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) &#xA;                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) &#xA;                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) &#xA;                at java.lang.reflect.Method.invoke(Method.java:597) &#xA;                at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) &#xA;                at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)&#xA;                at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) &#xA;                at com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37) &#xA;                at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54) &#xA;                at com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) &#xA;                at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)&#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) &#xA;                at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)&#xA;                at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) &#xA;                at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) &#xA;                at com.sun.proxy.$Proxy2528.searchInstruments(Unknown Source) &#xA;                at com.seic.ads.ejb.ADSServicesEjbBean_otv51e_ADSServicesEjbImpl.searchInstruments(ADSServicesEjbBean_otv51e_ADSServicesEjbImpl.java:2975)&#xA;                at com.seic.ads.ejb.ADSServicesEjbBean_otv51e_ADSServicesEjbImpl_WLSkel.invoke(Unknown Source) &#xA;                at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174) &#xA;                at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345) &#xA;                at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259) &#xA;                at com.seic.ads.ejb.ADSServicesEjbBean_otv51e_ADSServicesEjbImpl_1033_WLStub.searchInstruments(Unknown Source) &#xA;                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) &#xA;                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) &#xA;                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) &#xA;                at java.lang.reflect.Method.invoke(Method.java:597) &#xA;                at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:84) &#xA;                at com.sun.proxy.$Proxy2527.searchInstruments(Unknown Source) &#xA;                at com.seic.ads.delegate.ADSServicesDelegate.searchInstruments(ADSServicesDelegate.java:490) &#xA;                at com.seic.desktop.ads.presentation.helper.AssetSearchHelper.getInstruments(AssetSearchHelper.java:423) &#xA;                at com.seic.desktop.ads.presentation.action.AssetSearchAction.handleSearchCriteriaEvents(AssetSearchAction.java:488) &#xA;                at com.seic.desktop.ads.presentation.action.AssetSearchAction.performPostCondition(AssetSearchAction.java:254) &#xA;                at com.seic.common.presentation.controller.ProcessControllerAction.handleEvents(ProcessControllerAction.java:78) &#xA;                at com.seic.common.presentation.controller.ProcessControllerActionBase.process(ProcessControllerActionBase.java:119) &#xA;                at com.seic.common.presentation.action.DesktopAction.execute(DesktopAction.java:357) &#xA;                at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58) &#xA;                at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67) &#xA;                at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51) &#xA;                at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191) &#xA;                at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:305) &#xA;                at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:191) &#xA;                at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283) &#xA;                at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) &#xA;                at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462) &#xA;                at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) &#xA;                at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) &#xA;                at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) &#xA;                at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) &#xA;                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300) &#xA;                at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) &#xA;                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) &#xA;                at com.seic.common.presentation.filter.UserContextFilter.doFilter(UserContextFilter.java:65) &#xA;                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) &#xA;                at com.seic.common.presentation.helper.InjectionAttackFilter.chainNext(InjectionAttackFilter.java:194) &#xA;                at com.seic.common.presentation.helper.InjectionAttackFilter.doFilter(InjectionAttackFilter.java:188) &#xA;                at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) &#xA;                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3684) &#xA;                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650) &#xA;                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) &#xA;                at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121) &#xA;                at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268) &#xA;                at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174) &#xA;                at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446) &#xA;                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) &#xA;                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173) &#xA;Caused by: java.sql.SQLException: ORA-20010: PO_ID(222) INSTRUMENT_ID(1111112) PRICE_DATE(01-JUL-16) does not exist in Table xxx.  Price data not found.. &#xA;ORA-06512: at &quot;Q02_EDB.ERROR_PKG&quot;, line 502 &#xA;ORA-06512: at &quot;Q02_EDB.IM_PRICING&quot;, line 6221 &#xA;ORA-06512: at line 1 &#xA;                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440) &#xA;                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) &#xA;                at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837) &#xA;                at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445) &#xA;                at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191) &#xA;                at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523) &#xA;                at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193) &#xA;                at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1073) &#xA;                at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:359) &#xA;                at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:263) &#xA;                at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source) &#xA;                at com.seic.dataservices.dm.InstrumentSearchDo.Get(InstrumentSearchDo.java:3551) &#xA;                ... 77 more &#xA;||20160704||01:58:32,294||396c0a8e2470e7a21467611910768||com.seic.dataservices.impl.InstrumentSearchDoImpl||[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'||ERROR||ExceptionFactoryMsg:  com.seic.dataservices.lib.DataServiceSqlException - Error - A SQL exception was encountered while processing this request. - EX13 -  - q02_Desktop_MS1#20160704015832294#13 - With Additional Info:  (Error Code:  20010 -  ) - Caused By (java.sql.SQLException: ORA-20010: PO_ID(1000) INSTRUMENT_ID(1004915) PRICE_DATE(01-JUL-16) does not exist in Table TABEL_NAME.  Price data not found.. &#xA;ORA-06512: at &quot;Qxx_xxx.ERROR_PKG&quot;, line 502 &#xA;ORA-06512: at &quot;Qxx_xxx.IM_PRICING&quot;, line 6221 &#xA;ORA-06512: at line 1 &#xA;)**&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21352" LastEditorUserId="381" LastEditDate="2016-07-12T17:56:18.860" LastActivityDate="2018-04-15T02:27:25.017" Title="Reading CSVs with new lines in fields with Spark" Tags="|apache-spark|csv|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12728" PostTypeId="1" AcceptedAnswerId="12739" CreationDate="2016-07-11T22:51:24.283" Score="3" ViewCount="1620" Body="&lt;p&gt;Calculating absolute values is much more efficient than calculating squares. Is there any advantage, then, to using the latter as a cost function over the former? Squares are easier to treat analytically, but in practice that doesn't matter.&lt;/p&gt;&#xA;" OwnerUserId="21353" LastActivityDate="2016-07-12T12:22:44.917" Title="Minimize absolute values of errors instead of squares" Tags="|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12729" PostTypeId="2" ParentId="12728" CreationDate="2016-07-12T04:46:02.370" Score="2" Body="&lt;p&gt;&lt;strong&gt;Both the square and the absolute value should work okay for gradient descent but the square will work better.  For calculus based methods, the absolute value method may be intractable.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using gradient descent methods then the square works very nicely in that it tends to form a U shape which takes big steps away from the minimum and smaller steps close to the minimum. Conversely, absolute values tend to form V shapes which take roughly the same step size. The result is that squares have better convergence properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, calculus based methods tend to take the derivative of a function and set it equal to zero.  Notice that U shapes tend to have well defined derivatives equal to zero at the minima, which works perfectly. By contrast, the V shape has an undefined derivative at its minima which tend to make conjugate gradient methods un-invertible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-07-12T04:46:02.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12730" PostTypeId="2" ParentId="12724" CreationDate="2016-07-12T06:31:50.520" Score="1" Body="&lt;p&gt;As you said, you are being stuck at a local minima mostly. Change the parameters as suggested above and try. A learning rate that is too large can hinder convergence and cause the loss function to fluctuate around the minimum or even to diverge actually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To help with starting point and to be specific to quadratic and cross-entropy cost, according to &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Micheal A.Nielson, &quot;neural Networks and Deep Learning,Determination Press,2015&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/poGMh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/poGMh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This might not work as suggested. Randomizing is a good try. &lt;a href=&quot;https://www.willamette.edu/~gorr/classes/cs449/momrate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is a good read.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="11097" LastEditDate="2016-07-12T06:47:37.403" LastActivityDate="2016-07-12T06:47:37.403" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12731" PostTypeId="2" ParentId="10695" CreationDate="2016-07-12T08:02:46.660" Score="5" Body="&lt;p&gt;Let us look at a sample code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt;from gensim.models import word2vec&#xA;&#xA;#let us train a sample model like yours&#xA;&amp;gt;&amp;gt;&amp;gt;sentences = [['first', 'sentence'], ['second', 'sentence']]&#xA;&amp;gt;&amp;gt;&amp;gt;model1 = word2vec.Word2Vec(sentences, min_count=1)&#xA;&#xA;#let this be the model from which you want to reset&#xA;&amp;gt;&amp;gt;&amp;gt;sentences = [['third', 'sentence'], ['fourth', 'sentence']]&#xA;&amp;gt;&amp;gt;&amp;gt;model2 = word2vec.Word2Vec(sentences, min_count=1)&#xA;&amp;gt;&amp;gt;&amp;gt;model1.reset_from(model2)&#xA;&amp;gt;&amp;gt;&amp;gt;model1.similarity('third','sentence')&#xA;-0.064622000988260417&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Hence, we observe that &lt;code&gt;model1&lt;/code&gt; is being reset by the &lt;code&gt;model2&lt;/code&gt; and hence the word, &lt;code&gt;'third'&lt;/code&gt; and &lt;code&gt;'sentence'&lt;/code&gt; are in it's vocabulary eventually giving its similarity. This is the basic use, you can also check &lt;code&gt;reset_weights()&lt;/code&gt; to reset the weights to untrained/initial state.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="98307" LastEditDate="2020-08-06T16:18:12.947" LastActivityDate="2020-08-06T16:18:12.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12732" PostTypeId="5" CreationDate="2016-07-12T08:11:44.097" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-07-12T08:11:44.097" LastActivityDate="2016-07-12T08:11:44.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12733" PostTypeId="4" CreationDate="2016-07-12T08:11:44.097" Score="0" Body="word2vec is a two layer neural network to process text. It takes words as an input and outputs a vector correspondingly. It uses a combination of Continuous Bag of Word and skipgram model implementation." OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-07-12T12:19:02.587" LastActivityDate="2016-07-12T12:19:02.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12734" PostTypeId="1" CreationDate="2016-07-12T08:32:41.217" Score="3" ViewCount="6088" Body="&lt;p&gt;The terms AUC and accuracy has been confusing for me.&lt;br&gt;&#xA;In which circumstance AUC rate and accuracy rate become exactly equal?&lt;/p&gt;&#xA;" OwnerUserId="21363" LastEditorUserId="11097" LastEditDate="2016-07-12T08:37:59.360" LastActivityDate="2019-05-27T12:55:05.493" Title="When can AUC and accuracy rate be equal?" Tags="|classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12735" PostTypeId="2" ParentId="12734" CreationDate="2016-07-12T09:00:54.487" Score="5" Body="&lt;p&gt;AUC (or most often AUROC = &quot;area under &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;noreferrer&quot;&gt;receiver operating characteristic&lt;/a&gt;&quot;) and accuracy are different measures, but used for same purpose - to objectively measure performance of a simple binary classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two measures can be equal at extreme values of 0 and 1 for perfect classifiers - or inverse perfect classifiers (you can just invert the output to get a perfect classifier). It might be possible for the values to be numerically equal at other times, but if so it would be a coincidence with no specific meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both these metrics can be used with a simple classifier that only outputs true or false values for class membership. However, the AUROC metric requires some kind of parameter you can vary in order to plot the ROC curve. Usually this is a threshold for classification, used against the class probability output of a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other possible metrics. For example, &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;noreferrer&quot;&gt;F1 score&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy&quot; rel=&quot;noreferrer&quot;&gt;cross entropy&lt;/a&gt;. The F1 score again has 1.0 for a perfect classifier and 0.0 for a bad classifier, but cross entropy scores lower for better classifiers - 0.0 for perfect and no upper limit for bad output. Again, the values of these might be equal to other metrics at some points, but if so it is not meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are comparing two classifiers for a particular task, then it is important to compare them using the same metric, on the same test data. The metric and test data you choose should relate to your original problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are reading other people's published results, and want to compare them, it is not really possible if one person has used AUC and the other accuracy.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-07-12T09:00:54.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12736" PostTypeId="2" ParentId="12695" CreationDate="2016-07-12T10:21:03.987" Score="1" Body="&lt;p&gt;If the content/information is lengthy, I'd suggest you to use some NLP tasks for starters. I would suggest you to use some basic NLP based preprocessing because it makes our model perform better. So, the basic feature extraction can be used for this. Example, using Porter &lt;a href=&quot;http://www.nltk.org/howto/stem.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stemmer&lt;/a&gt;, &lt;a href=&quot;http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lemmatizer&lt;/a&gt; to clean the data or removing stop words and then using ngrams for features seem to be a basic idea and a good start. There are various vectorizers which can be used to extract features the documents. For example, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; calculates the frequency of a word in a document and also frequency across documents. This can be more useful than a naive &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bag of words&lt;/a&gt; approach. Then, on top of this there are various classifiers which can be used like &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;OneVsRestClassifier&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#module-sklearn.multiclass&quot; rel=&quot;nofollow noreferrer&quot;&gt;others&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple approach could be selecting the input and target first. Select the parameters which are to be passed as input and the desired output. Then, decide to clean the input or not based on some NLP APIs(you can use &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;nltk&lt;/a&gt;). Then decide on a classifier. You can then predict the values. Test on &lt;a href=&quot;https://datascience.stackexchange.com/questions/12626/what-is-the-difference-between-the-training-and-testdata-set/12628#12628&quot;&gt;validation set&lt;/a&gt; and &#xA;try various classifiers for starters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for terminology, I can think of &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multiclass Classification&lt;/a&gt; only now.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-12T10:21:03.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12738" PostTypeId="1" AcceptedAnswerId="12787" CreationDate="2016-07-12T12:19:37.880" Score="2" ViewCount="852" Body="&lt;p&gt;Ripley's nnet package, for example, allows you to model count data using a multi nomial setting but is there a package which preserves the complete information relating to a count?&#xA;For example, whereas an ordinal multinomial model preserves the ordering of the integers that make up the count, a fully developed model of count data as a GLM such as Poisson or Negative Binomial Regression includes how large the integer counts are in relation to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another phrasing might be, 'What kind of models come closest to combining the advantages of neural networks, in terms of, as an example, easily modelling non-linearity in the predictors, and count data GLMs, which are good at taking into account that the data is in fact a count?'&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="18843" LastEditDate="2016-07-15T06:40:55.083" LastActivityDate="2017-04-05T02:19:08.527" Title="Is there an R package which uses neural networks to explicitly model count data?" Tags="|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12739" PostTypeId="2" ParentId="12728" CreationDate="2016-07-12T12:22:44.917" Score="2" Body="&lt;p&gt;In what term &quot;calculating absolute values is much more efficient than calculating squares&quot;? Compared to the complexity of any estimator/model used, I don't think it is significant - but I would be interested if anyone makes me wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, why do you think it doesn't matter in practice? Working with a smooth and convex function is more convenient (in terms of time and results) than not-convex function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually, you can choose whatever function to minimize you'd like; it is just a trade-off between : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which kind of value you want to penalize&lt;/li&gt;&#xA;&lt;li&gt;Complexity of the function to solve (mathematically speaking : local or global solution)&lt;/li&gt;&#xA;&lt;li&gt;Time consuming (related to the previous point)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Minimizing absolute values&lt;/strong&gt; :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With absolute value, you penalize the distance between &lt;em&gt;y&lt;/em&gt; and &lt;em&gt;f(x)&lt;/em&gt; linearly. Roughly speaking, you might end up with a lot of data that will look like outliers as long as &lt;em&gt;enough&lt;/em&gt; are well explained by your estimator &lt;em&gt;f&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, to minimize a function, one generally looks for the root(s) of its derivative. However, the derivative of &lt;em&gt;|x|&lt;/em&gt; is not smooth. You can work with subgradient and other more complex mathematical object that may result in a longer time process due to more calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Minimizing square values&lt;/strong&gt; :  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the distance between &lt;em&gt;y&lt;/em&gt; and &lt;em&gt;f(x)&lt;/em&gt; is more penalized. You'll tend to have less &lt;em&gt;outliers&lt;/em&gt; (relatively to &lt;em&gt;f(x)&lt;/em&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is interesting is that is a smooth function (i.e. a defined derivative) and convex (with a global minimum)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess people believe that the square of errors is a good trade-off.&lt;/p&gt;&#xA;" OwnerUserId="20759" LastActivityDate="2016-07-12T12:22:44.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12743" PostTypeId="1" AcceptedAnswerId="12756" CreationDate="2016-07-12T16:09:38.613" Score="1" ViewCount="3013" Body="&lt;p&gt;I have a matrix with several unordered categorical variables. Each row represents a type of individual. Each column represents the number of times each type of individual was found to be in that particular condition.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Type    coal    cobalt  concrete    copper  gold&#xA;A       12      0       0           19      5&#xA;B       5       0       0           11      0&#xA;C       4       2       0           14      1&#xA;D       1       3       15          0       1&#xA;E       0       20      2           1       9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is very simple: I want to know if there is a correlation between the type of the individual (A, B or C) with a particular condition (copper, gold, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which test should I use? If possible, I would like to get the answer by using R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21376" LastEditorUserId="21669" LastEditDate="2016-07-21T23:14:26.503" LastActivityDate="2016-07-21T23:14:26.503" Title="r - How to determine the correlation between unordered categorical variables and individuals?" Tags="|r|statistics|correlation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12744" PostTypeId="1" AcceptedAnswerId="12749" CreationDate="2016-07-12T17:16:03.147" Score="10" ViewCount="6247" Body="&lt;p&gt;ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. Backpropagation in first-order methods requires first-order derivative. Hence $x$ is derived to $1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if we use second-order methods, would ReLU's derivative be $0$? Because $x$ is derived to $1$ and is derived again to $0$. Would it be an error? For example, with Newton's method, you'll be dividing by $0$. (I don't really understand Hessian-free optimization, yet. IIRC, it's a matter of using an approximate Hessian instead of the real one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the effect of this $h''=0$? Can we still train the neural network with ReLU with second-order methods? Or would it be non-trainable/error (nan/infinity)?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For clarity, this is ReLU as $f(x)$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x) =$ \begin{array}{rcl}&#xA;	0 &amp;amp; \mbox{for} &amp;amp; x &amp;lt; 0\\&#xA;	x &amp;amp; \mbox{for} &amp;amp; x \ge 0\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f'(x) =$ \begin{array}{rcl}&#xA;	0 &amp;amp; \mbox{for} &amp;amp; x &amp;lt; 0\\&#xA;	1 &amp;amp; \mbox{for} &amp;amp; x \ge 0\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f''(x) = 0$&lt;/p&gt;&#xA;" OwnerUserId="9465" LastEditorUserId="2474" LastEditDate="2023-06-05T05:11:27.333" LastActivityDate="2023-06-05T05:11:27.333" Title="Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?" Tags="|neural-network|optimization|backpropagation|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12745" PostTypeId="1" CreationDate="2016-07-12T17:19:02.100" Score="3" ViewCount="78" Body="&lt;p&gt;So I'm hoping to ask professionals for their advice here. My university doesn't offer a &quot;data science&quot; major and I'm cobbling one together from different departments. So far I have a math major with stat emphasis but I'm having trouble finding technology use in this program so I was thinking computer science for science credits. I also have a choice of an Information Management minor or a Business Analytics minor, both from the business college. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, I'm wondering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Which areas of computer science should I focus on?&lt;/strong&gt; (I can only take 2 aside from the intro CSI and CSII course).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Which minor should I go for?&lt;/strong&gt; Analytics consists of applied econ analytics, marketing research, and quantitative analysis while the info minor is management, programming, busn intelligence (advanced excel), and &quot;big data&quot;. Right now I'm leaning towards the info one because it seems to have more software in it, including SQL.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(I should also mention that I'm located in Las Vegas and my university even offers probability and gambling statistics but that's in the hospitality college and not sure that would be worth the trouble. It sounds interesting though, and might be good experience?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advisors have no idea what I'm going for, partly b/c they're all in different departments, so I hoped someone from the real world job market might know a thing or two :-)&lt;/p&gt;&#xA;" OwnerUserId="21377" LastActivityDate="2016-11-10T20:38:29.727" Title="Help on designing data science curriculum from &quot;traditional&quot; offerings" Tags="|education|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12746" PostTypeId="2" ParentId="12745" CreationDate="2016-07-12T17:48:00.343" Score="1" Body="&lt;p&gt;Of the traditional majors, I think computer science + statistics is the most obvious. You can substitute applied mathematics or even physics for the statistics major, if you want. Always keep the computer science courses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the choice is between information management and business analytics, given that you're already studying maths, it's hard to say. They're both pretty light-weight, and will add little to your math major. I would not consider them a plus per se as a prospective employer reading your resume. Unless they're cheap, I'd take online classes and get some practice instead. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-12T17:48:00.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12747" PostTypeId="2" ParentId="12727" CreationDate="2016-07-12T17:55:40.813" Score="1" Body="&lt;p&gt;Try setting &lt;code&gt;parserLib&lt;/code&gt; to &lt;code&gt;univocity&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; val df = sqlContext.read.options(Map(&quot;parserLib&quot; -&amp;gt; &quot;univocity&quot;, &quot;header&quot; -&amp;gt; &quot;true&quot;)).format(&quot;csv&quot;).load(&quot;test.csv&quot;)&#xA;&amp;gt; df.collect.foreach(println)&#xA;&#xA;[hello&#xA;world,hi mom,and&#xA;dad]&#xA;[1,2&#xA;more,3]&#xA;[foo,bar,just&#xA;one]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;test.csv&lt;/code&gt; is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a,b,c&#xA;&quot;hello&#xA;world&quot;,hi mom,&quot;and&#xA;dad&quot;&#xA;1,&quot;2&#xA;more&quot;,3&#xA;foo,bar,&quot;just&#xA;one&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, replace the new line characters with something else before parsing, and back again afterwards.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-07-14T04:34:17.777" LastActivityDate="2016-07-14T04:34:17.777" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12748" PostTypeId="2" ParentId="8841" CreationDate="2016-07-12T18:14:36.910" Score="1" Body="&lt;p&gt;By experience, I would also consider to check the ROC and AUC. One might try to use under-sampling as well as other over-sampling methods. In R, you have this &lt;a href=&quot;https://cran.r-project.org/web/packages/unbalanced/unbalanced.pdf&quot; rel=&quot;nofollow&quot; title=&quot;toolbox&quot;&gt;toolbox&lt;/a&gt; that can provide you different options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also check this &lt;a href=&quot;https://www.researchgate.net/profile/Vicente_Garcia4/publication/220392058_On_the_effectiveness_of_preprocessing_methods_when_dealing_with_different_levels_of_class_imbalance/links/0a85e535fd0c5139c7000000.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; which provides a comparison between different methods and draw some insights when over- or under-sampling are preferable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would agree with jamesmf to check the discriminative power of your feature at first.&lt;/p&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2016-07-12T18:14:36.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12749" PostTypeId="2" ParentId="12744" CreationDate="2016-07-12T19:25:16.043" Score="6" Body="&lt;p&gt;Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work meaningfully on most other common transfer functions though - they cannot be minimised or maximised for finite inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you combine multiple ReLU functions with layers of matrix multiplications in a structure such as a neural network, and wish to minimise an objective function, the picture is more complicated. This combination does have stationary points. Even a single ReLU neuron and a mean square error objective will have different enough behaviour such that the second-order derivative of the single weight will vary and is not guaranteed to be 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nonlinearities when multiple layers combine is what creates a more interesting optimisation surface. This also means that it is harder to calculate useful second-order partial derivatives (or &lt;a href=&quot;https://en.wikipedia.org/wiki/Hessian_matrix&quot; rel=&quot;noreferrer&quot;&gt;Hessian matrix&lt;/a&gt;), it is not just a matter of taking second order derivatives of the transfer functions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that $\frac{d^2y}{dx^2} = 0$ for the transfer function will make some terms zero in the matrix (for the second order effect from same neuron activation), but the majority of terms in the Hessian are of the form $\frac{\partial^2E}{\partial x_i\partial x_j}$ where E is the objective and $x_i$, $x_j$ are &lt;em&gt;different&lt;/em&gt; parameters of the neural network. A fully-realised Hessian matrix will have $N^2$ terms where $N$ is number of parameters - with large neural networks having upwards of 1 million parameters, even with a simple calculation process and many terms being 0 (e.g. w.r.t. 2 weights in same layer) this may not be feasible to compute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are techniques to estimate effects of second-order derivatives used in some neural network optimisers. RMSProp can be viewed as roughly estimating second-order effects, for example. The &quot;Hessian-free&quot; optimisers more explicitly calculate the impact of this matrix.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-07-13T09:23:23.977" LastActivityDate="2016-07-13T09:23:23.977" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12750" PostTypeId="1" CreationDate="2016-07-12T19:28:54.177" Score="1" ViewCount="281" Body="&lt;p&gt;I have a dataset which has two class. It has 13 features. They are values which are sent from 13 sensors. Label is True or False. When I use mad outlier detection, when the label is false(really there are some outliers), it detects the outliers correctly. I mean for example when sensors 1 and 3 are modified which leads to having label &quot;False&quot;, it detects 1 and 3 exactly and don't raise wrong additional sensor's number or fewer than actual one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is when the the label is True and there is no outlier. It still outputs some sensors as outlier. Actually it raise false positive. This makes hard to know that this is really outlier or not.  How can I solve this problem and finally be sure that this result is outlier or not? I have tested combination of this method with some other outlier detection results. I also used clustering in combination with MAD but I have not reached to a systematic solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw, Maybe you ask when you have label why you use outlier detection. No, I should develop an unsupervised method and checking label is just to be sure about the method accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="12345" LastActivityDate="2016-07-12T19:28:54.177" Title="Outlier Detection" Tags="|machine-learning|clustering|anomaly-detection|outlier|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12751" PostTypeId="1" AcceptedAnswerId="12804" CreationDate="2016-07-12T20:18:41.973" Score="5" ViewCount="9104" Body="&lt;p&gt;I have been reading several papers, articles and blog posts about RNNs (LSTM specifically) and how we can use them to do time series prediction. In almost all examples and codes I have found, the problem is defined as finding the next &lt;code&gt;x&lt;/code&gt; values of a time series based on previous data. What I am trying to solve is the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assuming We have &lt;code&gt;t&lt;/code&gt; values of a time series, what would be its value at time &lt;code&gt;t+1&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So using different LSTM packages (deeplearning4j, keras, ...) that are out there, here is what I am doing right now:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a LSTM network and fit it to &lt;code&gt;t&lt;/code&gt; samples. My network has one input and one output. So as for input I will have the following patterns and I call them train data:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;t_1,t_2&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;t_2,t_3&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;t_3,t_4&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The next step is to use for example &lt;code&gt;t_4&lt;/code&gt; as input and expect &lt;code&gt;t_5&lt;/code&gt; as output then use &lt;code&gt;t_5&lt;/code&gt; as input and expect &lt;code&gt;t_6&lt;/code&gt; as output and so on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When done with prediction, I use &lt;code&gt;t_5,t_6&lt;/code&gt; to update my model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question: Is this the correct way of doing it? If yes, then I have no idea what does &lt;code&gt;batch_size&lt;/code&gt; mean and why it is useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: An alternative that comes to my mind is something similar to examples which generate a sequence of characters, one character at a time. In that case, &lt;code&gt;batch_size&lt;/code&gt; would be a series of numbers and I am expecting the next series with the same size and the one value that I'm looking for would be the last number in that series. I am not sure which of the above mentioned approaches are correct and would really appreciate any help in this regard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="20544" LastEditorUserId="20544" LastEditDate="2017-07-17T13:05:54.850" LastActivityDate="2017-07-17T13:05:54.850" Title="Using RNN (LSTM) for predicting one future value of a time series" Tags="|time-series|deep-learning|rnn|prediction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12752" PostTypeId="1" CreationDate="2016-07-12T20:22:19.207" Score="4" ViewCount="1951" Body="&lt;p&gt;&lt;br&gt;&#xA;I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the &lt;strong&gt;labeled/supervised LDA&lt;/strong&gt;. Can any one help me the steps of this task that I should follow. Otherwise, I will be thankful if you can provide me another python library to do so.&lt;/p&gt;&#xA;" OwnerUserId="21381" LastActivityDate="2020-08-07T08:41:49.977" Title="How can I run Labeled LDA over one textual document?" Tags="|nlp|supervised-learning|topic-model|lda|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12754" PostTypeId="1" CreationDate="2016-07-12T22:28:43.557" Score="3" ViewCount="3932" Body="&lt;p&gt;I have trained an ARIMA model on some 15 minute incremented time series data by using the statsmodels library. I would like to determine how anomalous the next 15 minute increment's data I observe is. I would then like to update the model with that data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can predict the range of future outcomes within an accepted error margin like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = arima_model.ARIMA(data.COUNTS,(p,d,q)).fit()&#xA;print model.forecast(alpha=.001)[2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That outputs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([[ 16.13395152,  48.47024783]]))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So you could say any number outside that range is anomalous. However, if my next observation were, for example, 50 that would presumably be less anomalous than an observation of 51. Which way do you recommend I use to determine exactly how anomalous an observation is using an ARIMA model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how should I go about updating my model with the new observation? &lt;/p&gt;&#xA;" OwnerUserId="21394" LastEditorUserId="21394" LastEditDate="2016-07-12T23:30:04.950" LastActivityDate="2016-10-13T12:01:04.543" Title="Anomaly Detection In Univariate Time Series Data Using ARIMA In Python With Updating" Tags="|python|time-series|anomaly-detection|outlier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12755" PostTypeId="1" AcceptedAnswerId="12784" CreationDate="2016-07-12T23:58:04.343" Score="0" ViewCount="333" Body="&lt;p&gt;I am looking for a library that implements a pairwise ranking algorithm. For example, if I have 200 writing samples from 100 people (two samples from each individual) and I want to identify which samples belong together (i.e., were written by the same person), what library could I use?&lt;/p&gt;&#xA;" OwnerUserId="21396" LastEditorUserId="21024" LastEditDate="2016-07-13T15:16:54.753" LastActivityDate="2016-07-14T20:11:51.483" Title="Where can I find a software library for pairwise matching (ideally, Python, R, Java)?" Tags="|machine-learning|python|r|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12756" PostTypeId="2" ParentId="12743" CreationDate="2016-07-13T00:05:00.900" Score="3" Body="&lt;p&gt;If you have data sets $X_1,\cdots,X_n$ and $Y_1,\cdots,Y_n$, then you can compute their correlation with the following formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Cor(X,Y) = \frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sqrt{\sum (X_i-\bar{X})^2\sum(Y_i-\bar{Y})^2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(where $\bar{X}$ denotes the average value of the $X_i$'s).  This is accomplished in $R$ with the following command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor(x,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That being said, it is unclear what two data sets you are trying to find the correlation for.  Finding the correlation between a type (A,B,C) and a condition (copper, gold, etc.) would not make any sense.  You could, however, find the correlation between two different types (A and B, for example), or between conditions (copper and gold).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: I think you might want to do a test for independence between categorical variables...if this is the case then &lt;a href=&quot;http://www.r-tutor.com/elementary-statistics/goodness-fit/chi-squared-test-independence&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; is what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="21395" LastEditorUserId="21395" LastEditDate="2016-07-13T00:17:45.857" LastActivityDate="2016-07-13T00:17:45.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12759" PostTypeId="1" AcceptedAnswerId="12762" CreationDate="2016-07-13T10:46:02.623" Score="2" ViewCount="158" Body="&lt;p&gt;Reading &lt;a href=&quot;http://arxiv.org/pdf/1606.06630v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;On Multiplicative Integration with Recurrent Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Despite of their varying characteristics, most of them(RNNs) share a common&#xA;  computational building block, described by the following equation:&#xA;  $φ(W_x + U_z + b)$, where $x ∈ R_n$ and $z ∈ R_m$ are state vectors coming&#xA;  from different information sources, $W ∈ R_{d×n}$ and $U ∈ R_{d×m}$ are&#xA;  state-to-state transition matrices, and $b$ is a bias vector.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Don't get what meaning of $W_x$ and $U_z$. I know that $W$ is typicaly for weights... what does this equation mean?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:52.663" LastActivityDate="2017-05-29T04:59:52.663" Title="LSTMs: what is $W_x$ &amp; $U_z$ in $φ(W_x + U_z + b)$?" Tags="|neural-network|deep-learning|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12761" PostTypeId="1" AcceptedAnswerId="12763" CreationDate="2016-07-13T11:03:54.740" Score="63" ViewCount="65603" Body="&lt;p&gt;So, I have not been able to find any literature on this subject but it seems like something worth giving a thought:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the best practices in model training and optimization if new observations are available?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there any way to determine the period/frequency of re-training a model before the predictions begin to degrade?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it over-fitting if the parameters are re-optimised for the aggregated data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that the learning may not &lt;em&gt;necessarily&lt;/em&gt; be online. One may wish to upgrade an existing model after observing significant variance in more recent predictions.&lt;/p&gt;&#xA;" OwnerUserId="13450" LastEditorUserId="13450" LastEditDate="2016-07-14T13:53:46.670" LastActivityDate="2023-05-31T09:49:20.983" Title="Should a model be re-trained if new observations are available?" Tags="|machine-learning|predictive-modeling|optimization|training|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12762" PostTypeId="2" ParentId="12759" CreationDate="2016-07-13T11:16:08.823" Score="4" Body="&lt;p&gt;If you look deeper in LSTMs or GRUs, we observe that the gates(input, output, cell or forget based on the RNN) are calculated using an equation like you specified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, according to &lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;nofollow&quot;&gt;deep learning tutorial of lstm&lt;/a&gt;, &lt;strong&gt;i&lt;sub&gt;t&lt;/sub&gt;=sigma(W&lt;sub&gt;i&lt;/sub&gt; x&lt;sub&gt;t&lt;/sub&gt; + U&lt;sub&gt;i&lt;/sub&gt; h&lt;sub&gt;t-1&lt;/sub&gt; + b&lt;sub&gt;i&lt;/sub&gt;)&lt;/strong&gt; In this, h is the hidden state vector and x is the input state vector as specified and W and U are the corresponding weights for the input gate i&lt;sub&gt;t&lt;/sub&gt;. SImilarly, there are gates for output and forget. So in the paper, they recall a gist of RNNs and sum it up as a general equation. It is a common computational block in RNNs despite their minor differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow&quot;&gt;Colah's blog&lt;/a&gt; or &lt;a href=&quot;http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/&quot; rel=&quot;nofollow&quot;&gt;wildml&lt;/a&gt;, I think they are one of the best to understand RNNs.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-07-13T12:01:38.787" LastActivityDate="2016-07-13T12:01:38.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12763" PostTypeId="2" ParentId="12761" CreationDate="2016-07-13T12:44:43.340" Score="32" Body="&lt;ol&gt;&#xA;&lt;li&gt;Once a model is trained and you get new data which can be used for training, you can load the previous model and train onto it. For example, you can save your model as a &lt;code&gt;.pickle&lt;/code&gt; file and load it and train further onto it when new data is available. Do note that for the model to predict correctly, &lt;code&gt;the new training data should have a similar distribution as the past data&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Predictions tend to degrade based on the dataset you are using. For example, if you are trying to train using twitter data and you have collected data regarding a product which is widely tweeted that day. But if you use use tweets after some days when that product is not even discussed, it might be biased. &lt;code&gt;The frequency will be dependent on dataset&lt;/code&gt; and there is no specific time to state as such. &lt;code&gt;If you observe that your new incoming data is deviating vastly, then it is a good practise to retrain the model&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Optimizing parameters on the aggregated data is not overfitting. Large data doesn't imply overfitting. Use cross validation to check for over-fitting.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-13T12:44:43.340" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="12765" PostTypeId="2" ParentId="12761" CreationDate="2016-07-13T16:45:22.977" Score="40" Body="&lt;p&gt;When new observations are available, there are three ways to retrain your model:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Online:&lt;/strong&gt; each time a new observation is available, you use this single data point to further train your model (e.g. load your current model and further train it by doing backpropagation with that single observation). With this method, your model learns in a sequential manner and sort of adapts locally to your data in that it will be more influenced by the recent observations than by older observations. This might be useful in situations where your model needs to dynamically adapt to new patterns in data. It is also useful when you are dealing with extremely large data sets for which training on all of it at once is impossible.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Offline:&lt;/strong&gt; you add the new observations to your already existing data set and entirely retrain your model on this new, bigger data set. This generally leads to a better global approximation of the target function and is very popular if you have a fixed data set, or if you don't have new observations to often. However it is unpractical for large data sets.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Batch/mini batch:&lt;/strong&gt; this is sort of a middle ground approach. With batch, you wait until you have a batch of $n$ new observations and then train your already existing model on this whole batch. It is not offline as you are not adding this batch to your preexisting data set and then retraining your model on it and it is not online as your are training your model on $n$ observations at once and not just a single one. So it's a bit of both :) &lt;strong&gt;Mini batch&lt;/strong&gt; is exactly the same except that the batch size is smaller so it tends towards online learning. Actually online learning is just batch with batch size 1 and offline is batch with batch size the size of the whole data set. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most models today will use &lt;strong&gt;batch/mini batch&lt;/strong&gt; and the choice for the size of the batch depends on your application and model. Choosing the right size batch is equivalent to choosing the right frequency with which to re-train your model. If your new observation have a low variance with your existing data, I'd suggest larger batches (256-512 maybe) and if on the contrary new observations tend to vary greatly with your existing data, use small batches (8-256). At the end of the day, batch size is kind of like another hyper-parameter which you need to tune and which is specific to your data&lt;/p&gt;&#xA;" OwnerUserId="21426" LastActivityDate="2016-07-13T16:45:22.977" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12768" PostTypeId="1" AcceptedAnswerId="12774" CreationDate="2016-07-13T21:36:50.657" Score="1" ViewCount="734" Body="&lt;p&gt;I am using SMOTE in Python to perform oversampling of the minor class in an unbalanced dataset. I would like to know the way SMOTE formats its output, that is, whether SMOTE concatenates the newly generated samples to the end of the input data and returns that as the output or whether the new synthetic data points are positioned randomly among the input data points. I'd appreciate your help.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastEditorUserId="26019" LastEditDate="2020-10-02T16:34:27.830" LastActivityDate="2020-10-02T16:34:27.830" Title="location of the resampled data from SMOTE" Tags="|python|scikit-learn|class-imbalance|smote|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12770" PostTypeId="2" ParentId="12723" CreationDate="2016-07-14T00:40:32.453" Score="0" Body="&lt;p&gt;Google does not currently have any public GPU or TPU offerings, but they might be able to help if you work for a large company. Contact them directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: Google reps, just today.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-14T00:40:32.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12771" PostTypeId="1" CreationDate="2016-07-14T04:40:24.610" Score="-1" ViewCount="639" Body="&lt;p&gt;I'm working with datumbox ml framework for sentiment analysis.&lt;a href=&quot;https://github.com/datumbox/datumbox-framework&quot; rel=&quot;nofollow&quot;&gt;Here the git link&lt;/a&gt;. I found a sentiment Ml code in code examples of frame work &lt;a href=&quot;https://github.com/datumbox/datumbox-framework-examples/blob/master/src/main/java/com/datumbox/examples/TextClassification.java&quot; rel=&quot;nofollow&quot;&gt;here is the code example link&lt;/a&gt;. I want to use same code for twitter-sentiment But in the code example,only two training datasets are used pos,neg But for Twitter sentiment analysis,I want to include Neutral sentiment also.I do have my own training datasets of pos &amp;amp; neg tweets but I don't any Neutral datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My doubt is : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Do I need to feed neutral training dataset for giving the neutral sentiment? &#xA;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Do I need to predict neutral sentiment based on positive &amp;amp; negative training sets.if I need go for predicting then I need an idea on how to do that?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="21024" LastEditDate="2016-07-14T04:59:31.427" LastActivityDate="2017-09-07T15:55:39.630" Title="Do we Need to Populate the Neutral DataSets for Sentiment analysis?" Tags="|machine-learning|classification|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12772" PostTypeId="2" ParentId="12768" CreationDate="2016-07-14T06:27:12.397" Score="2" Body="&lt;p&gt;SMOTE is an algorithm used to generate &quot; synthesize&quot; new samples from the real samples. &#xA;It selects randomly one of the k-nearest neighbors, find the distance between these two pints , synthesize new point by modifying the sample considering the distance and a random number between 0 and 1.&#xA;SMOTE algorithm does not use samples from majority class only samples from minority. &#xA;It synthesizes new samples&#xA;It expects a high density minor class with small variation within the class &#xA;Anything else is related to the implementation, many implementation return only the synthesized samples &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-07-14T06:27:12.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12773" PostTypeId="2" ParentId="12771" CreationDate="2016-07-14T06:30:10.533" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;The machine cannot return neutral sentiment if you do not feed it. If you just have positive or negative, the machine will not return neutral.&lt;/li&gt;&#xA;&lt;li&gt;Alternatively, maybe you can try other approaches. You can check the confidence with which it predicts positive or negative and if it's low, maybe you can assign it to neutral.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-14T06:30:10.533" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12774" PostTypeId="2" ParentId="12768" CreationDate="2016-07-14T10:14:52.013" Score="1" Body="&lt;p&gt;There is not that much package managing the under-/over-sampling in python. So if you are using &lt;a href=&quot;https://github.com/fmfn/imbalanced-learn&quot; rel=&quot;nofollow&quot;&gt;imbalanced-learn&lt;/a&gt;, it will return a numpy array which concatenate the original imbalanced set with the generated new samples in the minority class. &lt;/p&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2016-07-14T10:14:52.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12775" PostTypeId="1" AcceptedAnswerId="19567" CreationDate="2016-07-14T10:15:22.567" Score="3" ViewCount="1711" Body="&lt;p&gt;I am working on a dimensionality reduction algorithm for datasets with high feature to sample ratio. I need datasets with around 100000 features or more and 100-1000 samples with class labels to evaluate my algorithm's performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover the features should be real valued and not binary like in the Dorothea dataset.&lt;/p&gt;&#xA;" OwnerUserId="21449" LastActivityDate="2019-11-19T13:20:03.513" Title="Where can I find datasets with large number of features (100000) but less samples(100-1000)?" Tags="|classification|bigdata|dataset|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12776" PostTypeId="1" CreationDate="2016-07-14T11:36:34.623" Score="-1" ViewCount="835" Body="&lt;p&gt;In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well until you get a mix of categorical and continuous features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you mix the continuous and categorical features the continuous features completely dominate as they have so many more places to split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a known method on how to compare the feature importance of continuous and categorical features from this kind of measure? If not, what suggestions do you have on doing this?&lt;/p&gt;&#xA;" OwnerUserId="20995" LastActivityDate="2016-07-14T16:15:18.313" Title="Comparing Categorical and Continuous Features using Splits in GBM" Tags="|feature-selection|decision-trees|xgboost|feature-engineering|gbm|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="12777" PostTypeId="2" ParentId="12761" CreationDate="2016-07-14T13:12:29.367" Score="1" Body="&lt;p&gt;Your problem comes under the umbrella of Online Learning methods. Assuming a stream of data coming, you can use Stochastic Gradient Descent method to update your model parameters using that single example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your cost function is : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \min_\theta J(x,y,\theta) $ , &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta$ is parameter vector, then assuming a streaming data of form ($x^{i}, y^{i}$), you can update your parameter vector using SGD with the following update equation :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \theta^{t} = \theta^{t-1} - \nabla_\theta J(x^{i}, y^{i}) $. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essentially SGD with batch size 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is one other trick, you can adopt a window/buffer based method, where you buffer some examples from stream and treat it as batch and use batch SGD. In that case the update equation will become:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \theta^{t} = \theta^{t-1} - \sum_{i} \nabla_\theta J(x^{i}, y^{i}) $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essentially mini-batch SGD.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-07-14T13:12:29.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12778" PostTypeId="2" ParentId="11225" CreationDate="2016-07-14T15:47:54.210" Score="1" Body="&lt;p&gt;Primarily, the concern with data-size arises because of the error that it can cause to the overlying model. With that concept in mind, you might want to measure the Bias and Variance of the model with your current data, understand the fitness of your model, and then take it from there. &lt;br&gt;&lt;bR&gt;&#xA;Here's an article that I wrote that may help you do the same: &lt;a href=&quot;http://www.mariner-usa.com/blog/machine-learning-big-data/&quot; rel=&quot;nofollow&quot;&gt;Machine Learning: How Big Should Your Data Be?&lt;/a&gt; &lt;br&gt;&lt;br&gt;&#xA;This article talks about &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bias and Underfitting&lt;/li&gt;&#xA;&lt;li&gt;Variance and Overfitting&lt;/li&gt;&#xA;&lt;li&gt;Bias-Variance Trade-Off and Optimal Complexity&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It concludes that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;More sample data is always powerful, but is not always the right&#xA;answer. &lt;/li&gt;&#xA;&lt;li&gt;As you increase your sample data, make sure to adjust your parameter space to minimize error.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2016-07-14T15:47:54.210" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12779" PostTypeId="2" ParentId="12776" CreationDate="2016-07-14T16:15:18.313" Score="2" Body="&lt;p&gt;As pointed in the comments, there is a distinction to make between the importance of a feature for the model, i.e., how many times it is used, and the importance for the accuracy of the prediction. These two points might be unrelated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider that you want to predict a continuous target $y$ from two features, $x_1, x_2$, where the ground truth is $y = x_1 + x_2$. Assume that $x_1$ can either be $0$ or $100$ and $x_2$ is a continuous variable ranging from $0$ to $1$. Training a model that fits exactly your training set will have only one split for $x_1$ and as many splits on $x_2$ than distinct training values of $x_2$. In this case, you can make $x_2$ arbitrarily more important that $x_1$ by increasing the number of distinct training points, but $x_2$ is mostly irrelevant to the accuracy of the prediction when compared to $x_1$. This problem makes it impossible to judge the importance of a feature based on the number of time it is used, in the general case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think there is a definite answer on how to rank features with respect to the accuracy either, as the information contained in one feature can be a duplicate from another one, or be useful only if you have another feature to complement it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple method to approach this problem is the reverse of iterative feature selection, which you can read more on &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection#Subset_selection&quot; rel=&quot;nofollow&quot;&gt;Feature Selection (Wikipedia)&lt;/a&gt;. Given a model with 10 features, you can build 10 models with 9 features, removing a different one each time, and compute their loss on a validation set. If the 2nd model out of your 10 models performs the worst, then the 2nd feature is the most important. If the 9th model, without feature #9, is still as good as the full model, then feature #9 is not important, and you can use the difference in your loss metric to measure the importance to accuracy. However, it might be the case that feature #9 becomes useful if you add a new feature and that feature #2 can be built by a combination of other features, which you'd catch if you allowed your model to be more complex.&lt;/p&gt;&#xA;" OwnerUserId="15501" LastActivityDate="2016-07-14T16:15:18.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12780" PostTypeId="1" AcceptedAnswerId="13476" CreationDate="2016-07-14T18:52:58.407" Score="7" ViewCount="686" Body="&lt;p&gt;I work in physics. We have lots of experimental runs, with each run yielding a result, &lt;code&gt;y&lt;/code&gt; and some parameters that should predict the result, &lt;code&gt;x&lt;/code&gt;. Over time, we have found more and more parameters to record. So our data looks like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year 1 data: (2000 runs)&#xA;    parameters: x1,x2,x3                target: y&#xA;Year 2 data: (2000 runs)&#xA;    parameters: x1,x2,x3,x4,x5          target: y&#xA;Year 3 data: (2000 runs)&#xA;    parameters: x1,x2,x3,x4,x5,x6,x7    target: y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How does one build a regression model that incorporates the additional information we recorded, without throwing away what it &quot;learned&quot; about the older parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;just set &lt;code&gt;x4&lt;/code&gt;, &lt;code&gt;x5&lt;/code&gt;, etc to &lt;code&gt;0&lt;/code&gt; or &lt;code&gt;-1&lt;/code&gt; when I'm not using them? &lt;/li&gt;&#xA;&lt;li&gt;completely ignore &lt;code&gt;x4,x5,x6,x7&lt;/code&gt; and only use &lt;code&gt;x1,x2,x3&lt;/code&gt;? &lt;/li&gt;&#xA;&lt;li&gt;add another parameter that is simply the number of parameters? &lt;/li&gt;&#xA;&lt;li&gt;train separate models for each year, and combine them somehow? &lt;/li&gt;&#xA;&lt;li&gt;&quot;weight&quot; the parameters, so as to ignore them if I set the weight to 0?&lt;/li&gt;&#xA;&lt;li&gt;make three different models, using &lt;code&gt;x1,x2,x3&lt;/code&gt;, &lt;code&gt;x4,x5&lt;/code&gt;, and &lt;code&gt;x6,x7&lt;/code&gt; parameters, and then interpolate somehow?&lt;/li&gt;&#xA;&lt;li&gt;make a custom &quot;imputer&quot; to guestimate the missing parameters (using available parameters)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have tried imputation using mean and median, but neither works very well because the parameters are not independent, but rather fairly correlated.&lt;/p&gt;&#xA;" OwnerUserId="21462" LastEditorUserId="21462" LastEditDate="2018-11-03T22:40:09.327" LastActivityDate="2018-11-03T22:40:09.327" Title="Regression model with variable number of parameters in dataset?" Tags="|machine-learning|regression|linear-regression|parameter|hyperparameter|" AnswerCount="3" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="12781" PostTypeId="1" AcceptedAnswerId="12792" CreationDate="2016-07-14T19:19:52.450" Score="0" ViewCount="89" Body="&lt;p&gt;I have two datasets with information about companies and my task is to correlate (match) companies from dataset A to companies in dataset B. Datasets are from different sources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns in both datasets include fields such as &lt;code&gt;company_name&lt;/code&gt;, &lt;code&gt;country&lt;/code&gt;, &lt;code&gt;state&lt;/code&gt;, &lt;code&gt;city&lt;/code&gt;, &lt;code&gt;address&lt;/code&gt;, &lt;code&gt;zip&lt;/code&gt;.  All companies are in the US. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that even though we have the &lt;code&gt;company_name&lt;/code&gt; on both sides - the names in A aren't equal to the names in B. So for example on A you might have &lt;em&gt;Google&lt;/em&gt; and on B you might have &lt;em&gt;Google Inc.&lt;/em&gt; Another example is &lt;em&gt;Amazon&lt;/em&gt; and &lt;em&gt;Amazon LLC.&lt;/em&gt; etc, there are many different variations to that.&lt;br&gt;&#xA;These aren't typos, but just different representations of the same entity, one is more common and the other is more formal.&lt;br&gt;&#xA;The addresses themselves aren't always the same as well. Probably b/c a company might have more than one address. (at least large companies do)   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best approach to correlate (match) these entities b/w these two data sources?&lt;br&gt;&#xA;There are about 500k companies in each dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few ideas come to mind: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Soundex function on the company name (tried it, not great)&lt;/li&gt;&#xA;&lt;li&gt;Levenshtein distance b/w names of each two potential matches (didn't try it yet, but it is &lt;code&gt;O(n*m)&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Levenshtein b/w the concatenated values of company names state, city, etc (also &lt;code&gt;O(n*m)&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Geocode the address and build a function that takes into account the Levenshtein distance as well as geographical distance. &lt;/li&gt;&#xA;&lt;li&gt;Clean up the &quot;INC&quot;, &quot;LLC&quot; and all other extensions and run any of 1-4.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What's your take? Any other suggestions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21461" LastEditorUserId="21461" LastEditDate="2016-07-17T07:43:30.003" LastActivityDate="2016-07-17T07:43:30.003" Title="Correlating company entities between different data sources" Tags="|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12782" PostTypeId="2" ParentId="12775" CreationDate="2016-07-14T19:27:00.007" Score="2" Body="&lt;p&gt;If a synthetic dataset is an option, you can create it with sklearn in python:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;sklearn.datasets.make_classification&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Generate a random n-class classification problem.&#xA;This initially creates clusters of points normally distributed (std=1) about vertices of a 2 * class_sep-sided hypercube, and assigns an equal number of clusters to each class. It introduces interdependence between these features and adds various types of further noise to the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import make_classification&#xA; &#xA;n_classes = 5&#xA;n_samples = 100*n_classes&#xA;n_informative = 100&#xA; &#xA;X, y = make_classification(n_samples=n_samples, &#xA;                          n_features=n_features, &#xA;                          n_informative=n_informative,&#xA;                          n_classes=n_classes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;More:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.datasets.make_classification&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18156" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-07-14T19:27:00.007" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12784" PostTypeId="2" ParentId="12755" CreationDate="2016-07-14T20:11:51.483" Score="0" Body="&lt;p&gt;If you can transform those sentences into number vectors (e.g. into a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow&quot;&gt;bag of words&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf-idf&quot; rel=&quot;nofollow&quot;&gt;tf-idf&lt;/a&gt; representation), I guess you could use k-Means or hierarchical clustering functionality from &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt;, a GUI and machine learning library written in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also has an &lt;a href=&quot;https://github.com/biolab/orange3-text/&quot; rel=&quot;nofollow&quot;&gt;add-on for text mining&lt;/a&gt; specifically, but I cannot attest to it as I haven't tried it yet.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-07-14T20:11:51.483" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12785" PostTypeId="1" CreationDate="2016-07-15T05:40:38.260" Score="2" ViewCount="477" Body="&lt;p&gt;There are a million and one examples and tutorials on how to train up a neural network on the sample sets like the MNIST data and the CIFAR-10 data, but how does one go from the toy examples of recognising 200x200 clips each containing a single centred object to a real problem like finding CIFAR-10 category objects (the dog and the cat below) within a picture, like I presume Google does for their photo annotation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OkR4y.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OkR4y.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone describe how one might approach this leap from the classroom to the real world?&lt;/p&gt;&#xA;" OwnerUserId="21478" LastEditorUserId="21478" LastEditDate="2016-07-15T08:32:43.857" LastActivityDate="2016-07-15T08:32:43.857" Title="How would one use a network trained on CIFAR-10 to find things in a photo?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12786" PostTypeId="1" CreationDate="2016-07-15T06:40:06.330" Score="1" ViewCount="1448" Body="&lt;p&gt;I want to merge two CSV files with a common column using python pandas. With a 32 bit processor, after 2 gb memory, it will throw memory error. How can I do the same with multi processing or any other methods?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import gc&#xA;import pandas as pd&#xA;csv1_chunk = pd.read_csv('/home/subin/Desktop/a.txt',dtype=str, iterator=True, chunksize=1000)&#xA;csv1 = pd.concat(csv1_chunk, axis=1, ignore_index=True)&#xA;csv2_chunk = pd.read_csv('/home/subin/Desktop/b.txt',dtype=str, iterator=True, chunksize=1000)&#xA;csv2 = pd.concat(csv2_chunk, axis=1, ignore_index=True)&#xA;new_df = csv1[csv1[&quot;PROFILE_MSISDN&quot;].isin(csv2[&quot;L_MSISDN&quot;])]&#xA;new_df.to_csv(&quot;/home/subin/Desktop/apyb.txt&quot;, index=False)&#xA;gc.collect()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21481" LastEditorUserId="21" LastEditDate="2016-07-15T07:36:32.533" LastActivityDate="2016-07-15T07:36:32.533" Title="How to merge two csv files using multiprocessing with python pandas" Tags="|python|pandas|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12787" PostTypeId="2" ParentId="12738" CreationDate="2016-07-15T06:50:51.327" Score="1" Body="&lt;p&gt;I skimmed over a paper recently that aims to use Neural Networks as Poisson regression. The method they propose is basically a standard Multi-Layer Perceptron where they use a different loss function, namely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E = -\sum_{n=1}^N[-t_n + y_nlog(t_n)]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a version without regularization to prevent the overfitting, they use regular weight decay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They mention that they wrote it in R and Matlab but I don't have a clue if it's available online somewhere, but any neural network package where you can pass your own loss function should suffice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&amp;amp;%20Applic.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&amp;amp;%20Applic.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-15T06:50:51.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12788" PostTypeId="1" CreationDate="2016-07-15T06:51:35.517" Score="1" ViewCount="1189" Body="&lt;p&gt;I know there is &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;f1_score&lt;/a&gt; metric to get all types of F1 scores (micro, macro, and weighted); however, I would like to be able to print micro averaged F1 score using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification_report&lt;/a&gt; of scikit-learn. By default, it seems to be returning weighted micro averaged F1. But I would like the micro averaged F1 in the &lt;code&gt;classification_report&lt;/code&gt;. How do I do that?&lt;/p&gt;&#xA;&lt;p&gt;Also, I know the difference in the formula between weighted and microaverages, but what are the instances where one would be preferred over other? And, what information do they convey?&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="29169" LastEditDate="2021-08-08T17:16:32.203" LastActivityDate="2021-08-08T17:16:32.203" Title="Average F1 Scores scikit-learn" Tags="|python|scikit-learn|score|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12789" PostTypeId="2" ParentId="12785" CreationDate="2016-07-15T07:13:33.593" Score="1" Body="&lt;p&gt;This is a well defined problem called text spotting. There are numerous avenues to tackle this problem but most of the good ones are based on deep learning. Naively you would use a network like the one you trained on MNIST to slide over your input and see where it fires strongly to build up a string. This approach works reasonably well but to convolute this over your whole input image is extremely computationally expensive. A way that is actually used in practice is a two step process, first there is a network that is trained in localizing areas of interest, which are bounding boxes of parts where there might be texts and then use a more advanced network to grab the text. To my understanding this is also done in one network pass nowadays as opposed to querying on single patches. If you search for text spotting you will find a lot of nice papers and a thesis, mostly from the same guy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&amp;amp;%20Applic.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&amp;amp;%20Applic.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-15T07:13:33.593" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="12790" PostTypeId="1" CreationDate="2016-07-15T08:00:32.163" Score="2" ViewCount="307" Body="&lt;p&gt;&lt;strong&gt;I'm curious if anyone can point to some successful &lt;a href=&quot;https://en.wikipedia.org/wiki/Extract,_transform,_load&quot; rel=&quot;nofollow noreferrer&quot;&gt;extract, transform, load (ETL)&lt;/a&gt; automation libraries, papers, or use cases for somewhat inhomogenious data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be interested to see any existing libraries dealing with scalable ETL solutions.  Ideally these would be capable of ingesting 1-5 petabytes of data containing 50 billion records from 100 inhomogenious data sets in tens or hundreds of hours running on 4196 cores (256 I2.8xlarge AWS machines). I really do mean &lt;strong&gt;ideally&lt;/strong&gt;, as I would be interested to hear about a system with 10% of this functionality to help reduce our team's ETL load.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, I would be interested to see any books or review articles on the subject or high quality research papers. I have done a literature review and have only found lower quality conference proceedings with dubious claims.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen a few commercial products advertised, but again, these make dubious claims without much evidence of their efficacy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The datasets are rectangular and can take the form of fixed width files, CSV, TSV, and PSV. Number of fields range from 6 to 150 and contain mostly text based information about entities.  Cardinality is large for individual information (address), but smaller for specific details like car type (van, suv, sedan).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mappings from abbreviated data to human readable formats is commonly needed, as is transformation of records to &lt;a href=&quot;https://en.wikipedia.org/wiki/First_normal_form&quot; rel=&quot;nofollow noreferrer&quot;&gt;first-normal-form&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As is likely obvious to the cognoscenti, I am looking for techniques that move beyond deterministic methods, using some sort of semi-supervised or supervised learning model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is a tall order, but I'm curious to assess the state-of-the-art before embarking on some ETL automation tasks to help guide how far to set our sights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-12-14T05:02:26.843" LastActivityDate="2016-12-14T05:02:26.843" Title="Successful ETL Automation: Libraries, Review papers, Use Cases" Tags="|data-cleaning|normalization|etl|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12791" PostTypeId="2" ParentId="12754" CreationDate="2016-07-15T09:23:46.910" Score="1" Body="&lt;p&gt;Have you considered using a quantile forecast as your ARIMA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quantile forecast is when, instead of predicting the average, you predict quantiles. In your case, you could predict for a given day the probability that an observation is below the 0.5%-percentile and the 99.5%-percentile. That range would define the 1% of &quot;abnormal&quot; values for the next days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recapitulating, usually in regression you find the average value by minimizing $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could find the median value by minimizing $\min_\beta \sum_i|X_i\cdot\beta -y_i|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also find any given quantile $\tau$ by minimizing $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i&amp;lt; X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, these minimizations problems become more and more complicated. You can only estimate your $\beta$ for this last one using ADMM or gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think you'll find python tools for this. You'll find plenty of packages in R though. For instance, see this page:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://robjhyndman.com/hyndsight/quantile-forecasts-in-r/&quot; rel=&quot;nofollow&quot;&gt;http://robjhyndman.com/hyndsight/quantile-forecasts-in-r/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One last piece of advise: if you need this forecasting to interoperate with other python code, my recommendation is to do it via CSV files. Just have your python code do the pre-processing and generate a CSV file, then R doing its thing and generating another CSV file, then use this file for the post-processing in python, etc. Some people try to use things like rpy2, but I find it to be overengineering.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-07-15T09:23:46.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12792" PostTypeId="2" ParentId="12781" CreationDate="2016-07-15T09:39:08.870" Score="0" Body="&lt;p&gt;First of all I would clean up INC, LLC, BV etcetera from both the sources. After this there are a few options. Since Levenshtein is a metric you can use metric trees to search your space more efficiently (about O(n*log(m))). This will still be very slow so there are approximations available, for example the cosine similarity on bi-grams of the names. You can do this using matrix multiplication which is both very efficient and easily distributable. Instead of taking the highest similarity you could take the top-n and do further analysis on these, for example the real Levenshtein distance. The fact that you have additional information could be useful, you could add this to your similarity function in some way but this will be guess work. Most of these ideas I got from a PyData meetup that was recorded, a speaker from ING (a big bank) discusses the exact problem you have albeit on a bigger set with less additional information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=4ohTsblxOJs&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=4ohTsblxOJs&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-15T09:39:08.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12793" PostTypeId="1" AcceptedAnswerId="12798" CreationDate="2016-07-15T09:58:55.877" Score="0" ViewCount="1294" Body="&lt;p&gt;Sklearn has several functions for feature selection that lets the user determine the size of the chosen subset. An example of this is &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest&quot; rel=&quot;nofollow&quot;&gt;SelectKBest&lt;/a&gt; where the user determines the value of &quot;k&quot;, which is the number of top performing features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know what stopping criterion &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel&quot; rel=&quot;nofollow&quot;&gt;SelectFromModel&lt;/a&gt; uses when it selects a feature subset? The documentation mentiones a &quot;threshold&quot;-parameter that determines which features are important enough, and that this parameter is set to &quot;median&quot; OR &quot;mean&quot; by default.&lt;/p&gt;&#xA;" OwnerUserId="21486" LastActivityDate="2016-07-15T12:42:12.513" Title="Sklearn feature selection stopping criterion (SelectFromModel)" Tags="|machine-learning|feature-selection|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12794" PostTypeId="1" CreationDate="2016-07-15T10:03:34.893" Score="1" ViewCount="125" Body="&lt;p&gt;My &lt;strong&gt;goal&lt;/strong&gt; is to get a &lt;strong&gt;smartphone names&lt;/strong&gt; from Twitter. So this is what I followed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- I extracted 100K tweets using the keyword “smartphone”.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- I Applied LDA after applying ngram tokenization and cleaning.So, I got noisy results such as:(giveway, international, apple, iphone_6, samsung_s5,news…)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- I filtered the results using a smartphone list (iphone_6, samsung_s5,iphone_4s,…) extracted from DBpedia in order to remove the noise.&lt;br&gt;&lt;br&gt;&#xA;Is what I did supervised or unsupervised Machine learning?&lt;/p&gt;&#xA;" OwnerUserId="21487" LastActivityDate="2016-07-15T16:37:50.800" Title="Is what I did supervised or unsupervised Machine learning?" Tags="|machine-learning|supervised-learning|topic-model|unsupervised-learning|lda|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12795" PostTypeId="2" ParentId="12794" CreationDate="2016-07-15T10:26:39.710" Score="1" Body="&lt;p&gt;The basic idea behind supervised and unsupervised learning is:&#xA;In &lt;code&gt;supervised learning&lt;/code&gt;, one possesses a &lt;code&gt;training data set&lt;/code&gt; that includes the desired &lt;code&gt;target&lt;/code&gt; output feature. Examples of popular supervised learning algorithms are linear regression, and support vector classification etc.&#xA;In &lt;code&gt;unsupervised learning&lt;/code&gt;, we do not possess a training data set with the target output feature i.e. we do not know what our output or results look like. An example is clustering. There are really good &lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms&quot; rel=&quot;nofollow&quot;&gt;explanations&lt;/a&gt; out there about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LDA is an unsupervised learning algorithm and the process you described can be classified as unsupervised learning. The filtering step that you describe does not make the algorithm supervised because the target smartphones have not been directly correlated to the training data and therefore is only serving as a guide to restrict the cardinality of the target variable.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="9420" LastEditDate="2016-07-15T16:37:50.800" LastActivityDate="2016-07-15T16:37:50.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12797" PostTypeId="1" AcceptedAnswerId="13457" CreationDate="2016-07-15T10:43:58.690" Score="4" ViewCount="3735" Body="&lt;p&gt;I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier)&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Label  Feat1 Feat2 Feat3 Feat4&#xA; 1      1     1      2     2&#xA; 2      3     1      2     2 &#xA; 3      4     3      3     1     &#xA; 4      4     1      2     4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;page&lt;/a&gt; that Decision Tree does not support missing values.&lt;/p&gt;&#xA;&lt;p&gt;What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="17772" LastEditorUserId="85045" LastEditDate="2021-01-02T02:51:16.073" LastActivityDate="2021-01-02T02:51:16.073" Title="Scikit Learn Missing Data - Categorical values" Tags="|predictive-modeling|scikit-learn|categorical-data|multiclass-classification|missing-data|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12798" PostTypeId="2" ParentId="12793" CreationDate="2016-07-15T12:42:12.513" Score="1" Body="&lt;p&gt;Some regression/classification models can also calculate feature importances - for example &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;RandomForestClassifier&lt;/a&gt; models have a property &lt;strong&gt;feature_importances_&lt;/strong&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow&quot;&gt;LogisticRegression&lt;/a&gt; models have a &lt;strong&gt;coef_&lt;/strong&gt; property. There are many more models that can provide feature importances - but all of them either have a coef_ or feature_importances_ property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What SelectFromModel does is to check if the object you pass as &quot;estimator&quot; has one of these two properties and to return features where this property is higher than the specified threshold (or the mean/median).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if you pass a RandomForestClassifier to SelectFromModel, it will return all features where the random forest's feature_importances_ property is higher than the specified threshold. The same happens if you pass a LogisticRegression model, except that it'll compare the coef_ property with the threshold instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Selecting the best value for the threshold can be done using a grid- or randomized search.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-07-15T12:42:12.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12799" PostTypeId="1" CreationDate="2016-07-15T13:48:09.557" Score="23" ViewCount="45633" Body="&lt;p&gt;I am trying to run xgboost in scikit learn.  And I am only using Pandas to load the data into a dataframe. How am I supposed to use pandas df with xgboost?  I am confused by the DMatrix routine required to run the xgboost algorithm.&lt;/p&gt;&#xA;" OwnerUserId="10915" LastEditorUserId="29169" LastEditDate="2021-03-11T19:13:48.547" LastActivityDate="2021-03-11T19:14:27.893" Title="Pandas Dataframe to DMatrix" Tags="|scikit-learn|pandas|xgboost|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12800" PostTypeId="2" ParentId="12799" CreationDate="2016-07-15T14:12:34.423" Score="26" Body="&lt;p&gt;You can use the dataframe's &lt;code&gt;.values&lt;/code&gt; method to access raw data once you have manipulated the columns as you need them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = pd.read_csv(&quot;train.csv&quot;)&#xA;target = train['target']&#xA;train = train.drop(['ID','target'],axis=1)&#xA;test = pd.read_csv(&quot;test.csv&quot;)&#xA;test = test.drop(['ID'],axis=1)&#xA;&#xA;xgtrain = xgb.DMatrix(train.values, target.values)&#xA;xgtest = xgb.DMatrix(test.values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously you may need to change which columns you drop or use as the training target. The above was for a Kaggle competition, so there was no target data for &lt;code&gt;xgtest&lt;/code&gt; (it is held back by the organisers).&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-07-15T14:12:34.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12802" PostTypeId="2" ParentId="12797" CreationDate="2016-07-15T14:27:02.397" Score="0" Body="&lt;p&gt;In my opinion it's always better to deal yourself with missing data instead on relying on classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to deal with it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Drop missing observations&lt;/li&gt;&#xA;&lt;li&gt;Drop rows where all cells in that row is NA&lt;/li&gt;&#xA;&lt;li&gt;Fill in missing data with any one of random label&lt;/li&gt;&#xA;&lt;li&gt;Fill label with maximum frequency for missing attribute&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;PS: I suggest using python pandas library for the purpose of data cleanup. &lt;/p&gt;&#xA;" OwnerUserId="21497" LastActivityDate="2016-07-15T14:27:02.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12803" PostTypeId="2" ParentId="12751" CreationDate="2016-07-15T14:52:34.927" Score="2" Body="&lt;p&gt;I recently did a lot of reading and code writing on LSTMs so I'll try to pitch in and answer the question, though I'm not yet familiar with Theano Keras or deeplearning4j. From a quick scour of the Internet it seems that the meaning of batch_size is program-dependent, so it might be a good idea to check the online Help files. From what I gather though, it can refer to a couple of different things:&#xA;1) the count of cases fed into the training algorithm for processing before the next examination of inputs and/or outputs.&#xA;2) the number of iterations of the training algorithm over the last set of input before the next examination of inputs and/or outputs.&#xA;3) the size of sequences fed to the LSTM algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind it seems almost certain that Theano and deeplearning4j are referring to #1 and/or #2, given the following links. On the other hand, I've run across journal articles recently, where batch size referred to sequence size, so watch out for a potential mismatch between the software and academic terminology:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;☻ &lt;a href=&quot;https://stats.stackexchange.com/questions/223829/online-learning-in-lstm&quot;&gt;This post at CrossValidated&lt;/a&gt; indicates that in deeplearning4j and Keras the batch_size should be set to 1 for online learning. So in other words, batch_size controls the number of training cycles before the next input/output check. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;☻ In Keras, the batch_size apparently sets the number of training cycles to execute or # of cases to train on before checking the inputs and/or outputs again, &lt;a href=&quot;https://github.com/fchollet/keras/issues/1904&quot; rel=&quot;nofollow noreferrer&quot;&gt;according to this Github page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;☻ &lt;a href=&quot;https://github.com/Element-Research/rnn/issues/74&quot; rel=&quot;nofollow noreferrer&quot;&gt;This Github package&lt;/a&gt; also seems to use a different variable to denote sequence length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;☻ From Tony Kanh's answer at &lt;a href=&quot;https://stackoverflow.com/questions/36286594/predicting-the-next-word-using-the-lstm-ptb-model-tensorflow-example&quot;&gt;this StackOverflow page&lt;/a&gt;, in TensorFlow the batch_size determines the size of the output vector. This is probably identical to the number of items processed in each training cycles, since multiplying it by the num_steps and size (probably the sequence size?) determines the dimensions of the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;☻ For closely related neural nets like Bidirectionals, the batch size is apparently equivalent to the sequence size, not the number of cases fed into the training algorithm. See p. 5, Berglund, et al., &quot;Bidirectional Recurrent Neural Networks as Generative Models,&quot; available at &lt;a href=&quot;http://arxiv.org/abs/1504.01575&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Cornell University Library website without a paywall&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said I haven't used Theano and deeplearning4j yet, but I hope that at least provides a starting point to find the correct answer.&lt;/p&gt;&#xA;" OwnerUserId="18544" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-15T14:52:34.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12804" PostTypeId="2" ParentId="12751" CreationDate="2016-07-15T16:58:43.497" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;The way you are doing it is just fine. The idea in time series&#xA;prediction is to do regression basically. Probably what you have&#xA;seen other places in case of vector, it is about the size of the&#xA;input or basically it means feature vector. Now, assuming that you&#xA;have &lt;code&gt;t&lt;/code&gt; timesteps and you want to predict time &lt;code&gt;t+1&lt;/code&gt;, the best way&#xA;of doing it using either time series analysis methods or RNN models&#xA;like LSTM, is to train your model on data up to time &lt;code&gt;t&lt;/code&gt; to predict&#xA;&lt;code&gt;t+1&lt;/code&gt;. Then &lt;code&gt;t+1&lt;/code&gt; would be the input for the next prediction and so&#xA;on. There is a good example &lt;a href=&quot;https://stackoverflow.com/questions/25967922/pybrain-time-series-prediction-using-lstm-recurrent-nets&quot;&gt;here&lt;/a&gt;. It is based on LSTM using the pybrain framework.&lt;/li&gt;&#xA;&lt;li&gt;Regarding your question on &lt;code&gt;batch_size&lt;/code&gt;, at first you need to understand the difference between batch learning versus online learning. Batch size basically indicates a subset of samples that your algorithms is going to use in gradient descent optimization and it has nothing to do with the way you input the data or what you expect your output to be. For more information on that I suggest you read &lt;a href=&quot;https://www.kaggle.com/c/datasciencebowl/forums/t/12634/can-someone-explain-what-batch-size-is-doing-in-convolutional-nns/100344&quot; rel=&quot;nofollow noreferrer&quot;&gt;this kaggle&lt;/a&gt; post.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21500" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-15T16:58:43.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12805" PostTypeId="1" AcceptedAnswerId="12807" CreationDate="2016-07-15T17:36:20.600" Score="2" ViewCount="9312" Body="&lt;p&gt;Are there any concerns to normalizing data to be within the range 0 - 1 and mean centering the data as well? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it matter which comes first?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do one, is the other not required? &lt;/p&gt;&#xA;" OwnerUserId="21430" LastEditorUserId="29575" LastEditDate="2019-05-07T23:55:00.920" LastActivityDate="2019-05-07T23:55:00.920" Title="What does normalizing and mean centering data do?" Tags="|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12806" PostTypeId="1" CreationDate="2016-07-15T17:46:44.880" Score="2" ViewCount="2177" Body="&lt;p&gt;How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?&#xA;I can not seem to find it anywhere, even not from the author of LDA, M.Blei.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gensim and sklearn just work, but I want to know how to use the two matrices to construct the document topic-matrix (Spark MLLIB LDA only gives me the 2 matrices and not the document-topic matrix).&lt;/p&gt;&#xA;" OwnerUserId="21503" LastActivityDate="2024-01-20T17:02:30.270" Title="How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?" Tags="|python|apache-spark|lda|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12807" PostTypeId="2" ParentId="12805" CreationDate="2016-07-15T17:55:11.070" Score="2" Body="&lt;p&gt;If you don't center before you normalize, you don't take advantage of the full [-1,1] range if your input is non-negative. The combination of centering and normalization is called &lt;em&gt;standardization&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes one normalizes by the standard variation, and other times by just the range (max-min). The latter is called &lt;em&gt;feature scaling&lt;/em&gt;. The effect is much the same. Normalizing by the range is easier computationally. Normalizing by the standard deviation fixes the sample variance, which is nice from a statistical perspective. When using the standard deviation, the subtraction is usually against the sample mean rather than the minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several reasons for performing standardization. Sometimes we are interested in relative rather than absolute values. Standardization achieves invariance to these irrelevant differences. By explicitly preprocessing the data to reflect this disinterest, we relieve the model from having to learn it, allowing us to use a simpler one. Another reason is computational; it reduces the &lt;em&gt;condition number&lt;/em&gt; -- you can think of this as the skewness or niceness of the loss surface -- making optimization easier and faster.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="29575" LastEditDate="2019-05-07T23:54:22.200" LastActivityDate="2019-05-07T23:54:22.200" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="12808" PostTypeId="1" AcceptedAnswerId="12809" CreationDate="2016-07-15T18:17:00.940" Score="0" ViewCount="281" Body="&lt;p&gt;I wanted to know if there is any methodology in Deep/Machine learning, where given a set of input/output values, it can derive rules for the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I generate training input and output by $y=x^2$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i/p  |  o/p&#xA; 0       0&#xA; 2       4&#xA; .       .&#xA;1000   1000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It sort of generate rule like, $y=x*x$&lt;/p&gt;&#xA;" OwnerUserId="9003" LastActivityDate="2016-07-15T20:14:46.373" Title="Deep learning - rule generation" Tags="|machine-learning|deep-learning|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12809" PostTypeId="2" ParentId="12808" CreationDate="2016-07-15T19:17:11.230" Score="1" Body="&lt;p&gt;One way of stating what you are looking for is to find a simple mathematical model to explain your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing about neural networks is that (once they have more than 2 layers, and enough neurons total) they can in theory emulate any function, no matter how complex. This is useful for machine learning as often the function we want to predict is complex and cannot be expressed simply with a few operators. However, it is kind of the opposite of what you want - the neural network behaves like a &quot;black box&quot; and you don't get a simple function out, even if there is one driving the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try to fit a model (any model) to your data using very simple forms of regression, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;linear regression&lt;/a&gt;. So if you are reasonably sure that your system is a cubic equation $y= ax^3 + bx^2 +cx +d$ then you could create a table like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  bias   |   x  |  x*x  |  x*x*x  |     y&#xA;     1       0       0         0        0&#xA;     1       2       4         8        4&#xA;     1       3       9        27        9&#xA;     1       .         .        .       .&#xA;     1     100   1000000    10000   10000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then use a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;nofollow&quot;&gt;linear regression optimiser&lt;/a&gt; (sci-kit learn's SGD optimiser linked). With the above data this should quickly tell you $b=1, a,c,d=0$. But what it won't tell you is whether your model is the best possible or somehow &quot;correct&quot;. You can scan for more possible formulae by creating more columns - any function of any combination of inputs (if there is more than one) that could be feasible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;However&lt;/em&gt;, the more columns you add in this way, the more likely it is you will find an incorrect &lt;em&gt;overfit&lt;/em&gt; solution that matches all your data using a clever combination of parameters, but which is not a good general predictor. To address this, you will need to add regularisation - a simple L1 or L2 regularisation of the parameters will do (in the link I gave to scikit-learn, the &lt;code&gt;penalty&lt;/code&gt; argument can control this), which will penalise large parameters and help you home in on a simple formula if there is one.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-07-15T20:14:46.373" LastActivityDate="2016-07-15T20:14:46.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12810" PostTypeId="1" CreationDate="2016-07-15T19:58:34.780" Score="1" ViewCount="749" Body="&lt;p&gt;I have the following problem: I have a set of raw data entered manually. These data are evaluated by business rules and eventually generate integrity errors, mostly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then these errors are manually handled and the correction is performed manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best approach in machine learning to implement an algorithm that look at this historical corrections and suggest the correct approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: all data are structured and business rules are evaluated mostly at foreign keys restrictions.&lt;/p&gt;&#xA;" OwnerUserId="21508" LastEditorUserId="9420" LastEditDate="2016-07-15T21:17:46.073" LastActivityDate="2016-07-15T21:17:46.073" Title="What is the best machine learning approach for data error correction?" Tags="|machine-learning|" AnswerCount="0" CommentCount="1" ClosedDate="2021-07-13T17:08:19.583" ContentLicense="CC BY-SA 3.0" />
  <row Id="12812" PostTypeId="2" ParentId="758" CreationDate="2016-07-15T22:02:33.660" Score="7" Body="&lt;p&gt;There is an entire course that is devoted to reproducible research.&#xA;&lt;a href=&quot;https://www.coursera.org/learn/reproducible-research&quot; rel=&quot;noreferrer&quot;&gt;https://www.coursera.org/learn/reproducible-research&lt;/a&gt;&#xA;This course is based on R, but the underlying idea can be learnt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple way is to have an Ipython notebook and keep saving every dirty work you do, be it cleaning the data, exploratory analysis or building the model.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-15T22:02:33.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12813" PostTypeId="2" ParentId="1107" CreationDate="2016-07-15T22:10:08.333" Score="1" Body="&lt;p&gt;In addition to the answers posted here, if the number of positive examples are way too small when compared to the negative examples, then it comes close to being an anomaly detection problem where the positive examples are the anomalies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a whole range of methods for detecting anomalies ranging from using multivariate gaussian distribution to model all the points and then picking those that are 2 or 3 stds away from the mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another food for thought - I have seen  quite a few people who randomly sample the negative examples with more examples so that both the classes are same in number. It totally depends on the problem at hand, whether we want them to be balanced or not.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-15T22:10:08.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12814" PostTypeId="2" ParentId="22" CreationDate="2016-07-15T22:35:03.943" Score="7" Body="&lt;p&gt;It depends on your categorical variable being used. For ordinal variables, say like bad,average and good, it makes sense just to use one variable and have values 0,1,2 and distances make sense here(Avarage is closer to bad and good). However, if there is no order, you should ideally use one hot encoding as mentioned above.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-15T22:35:03.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12815" PostTypeId="2" ParentId="1002" CreationDate="2016-07-15T23:02:32.460" Score="2" Body="&lt;p&gt;Aleksandr has given a very thorough explanation, but briefly, these are the steps that are followed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extracting data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cleaning data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature extraction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Building models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inferring results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Publishing results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat steps 3,4,5 in loop till you get the right accuracy.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-15T23:02:32.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12816" PostTypeId="2" ParentId="3718" CreationDate="2016-07-15T23:19:12.977" Score="0" Body="&lt;p&gt;If you are using python, I would suggest using mpld3 which combines D3js javascript visualizations with matplotlib of python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The installation and usage is really simple and it has some cool plugins and interactive stuffs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mpld3.github.io/&quot; rel=&quot;nofollow&quot;&gt;http://mpld3.github.io/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-15T23:19:12.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12817" PostTypeId="2" ParentId="12469" CreationDate="2016-07-16T08:38:42.610" Score="1" Body="&lt;p&gt;You may need to treat this as an optimization problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without a formal notion of what &lt;em&gt;should&lt;/em&gt; be a cluster, you will not be getting far. A blackbox algorithm will not solve your problem, you need to open the box (understand the algorithm) and adjust it as necessary for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have this notion, you can try to match it to algorithms. For example, if you can &lt;em&gt;quantify&lt;/em&gt; similarity (e.g. if distance times attribute dissimilarity matches your desires) then you can try HAC. Or if you have thresholds (distance at most 100mi, attribute difference at most y) then you can run DBSCAN and merge neigh ors transitively into clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But don't expect anything automatic.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-07-16T08:38:42.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12818" PostTypeId="1" AcceptedAnswerId="12822" CreationDate="2016-07-16T09:44:06.663" Score="4" ViewCount="95" Body="&lt;p&gt;I have hourly power consumption data for 30 days. On representing, each day data using a separate line, I get a plot as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XM3sV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XM3sV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to highlight the days with abnormally high consumption (in other words, the outlier days). I think that the current plot is too much congested. Is there any other better representation to show the outlier days?&lt;/p&gt;&#xA;" OwnerUserId="13291" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:35.303" LastActivityDate="2017-05-04T19:57:35.303" Title="Represent outlier days" Tags="|time-series|visualization|outlier|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12819" PostTypeId="1" CreationDate="2016-07-16T14:09:38.790" Score="0" ViewCount="116" Body="&lt;p&gt;I have written the following SAS procedure to display Grades based on the salary:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Not Working:&lt;/strong&gt; This displays only 'I' for all observations irrespective of the else condition that says any salary less than 35000 should have a Grade as 'II'&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATA Test (Keep = FirstName LastName Salary Grade);&#xA;set orion.sales;&#xA;if salary &amp;lt; 35000 then grade = 'I';    /* Did not interpret 'I' */&#xA;else grade = 'II';    /* Did not interpret 'II' */&#xA;run;&#xA;&#xA;proc print data = test;&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Working:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DATA Test (Keep = FirstName LastName Salary Grade);&#xA;set orion.sales;&#xA;if salary &amp;lt; 35000 then grade = 'G1';    /* interprets 'G1' */&#xA;else grade = 'G2';    /* interprets 'G2' */&#xA;run;&#xA;&#xA;proc print data = test;&#xA;run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So basically, the SAS did not interpret the Roman letters or numbers. Does anyone has an idea of why this is happening or any workaround to display the Roman numbers or letters?&lt;/p&gt;&#xA;" OwnerUserId="13395" LastActivityDate="2016-09-06T18:56:37.790" Title="SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value" Tags="|dataset|sas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12820" PostTypeId="1" AcceptedAnswerId="12834" CreationDate="2016-07-16T19:40:02.737" Score="2" ViewCount="159" Body="&lt;p&gt;I have a pet project to figure out the birth year of a significant person in history. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm collecting a lot of data on other people with similar status during that time period. I have data such as education length, year married, year of child bearing, information about siblings and age difference between each child, marriages, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The age of this person is disputed between two years, one making the person very old, and the other making the person young. I want to regress the persons age. My first idea was to draw gaussians over each variable and see if one is more likely to be an outlier than the other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What way would you tackle this problem?&lt;/p&gt;&#xA;" OwnerUserId="13768" LastEditorUserId="13768" LastEditDate="2016-07-18T15:42:03.147" LastActivityDate="2016-07-18T15:42:03.147" Title="Predicting Age of Birth" Tags="|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12821" PostTypeId="1" AcceptedAnswerId="12824" CreationDate="2016-07-16T21:30:29.840" Score="2" ViewCount="2311" Body="&lt;p&gt;I am a newbie to data science. I have a 'short text' categorization problem where input variables are either unstructured texts (names, definition, description etc) or categorical. There is not much semantic to the fields as they are product names, territory name, sales order type etc. Issue is I do not have any sample data set from which I can derive training, test, validation set or divided it into k-fold for cross validation. So how should I generate sample data? I have about 20 target classes. I can classify some dataset using regex or lucene rule based matches and manually verify them and make sure each class have equal amount of samples. But I am open to other suggestion.&lt;/p&gt;&#xA;" OwnerUserId="19147" LastEditorUserId="19147" LastEditDate="2016-07-18T19:56:59.363" LastActivityDate="2016-07-18T20:02:24.643" Title="how to generate sample dataset for classification problem" Tags="|machine-learning|text-mining|multiclass-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12822" PostTypeId="2" ParentId="12818" CreationDate="2016-07-16T21:43:31.070" Score="4" Body="&lt;p&gt;One idea would be to plot the daily average power consumption in a bar plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ym3Ws.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ym3Ws.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a finer visualization of day-hour peaks, you can try to plot it in a matrix format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o6jUH.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o6jUH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18156" LastActivityDate="2016-07-16T21:43:31.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12823" PostTypeId="2" ParentId="11241" CreationDate="2016-07-16T22:32:36.363" Score="1" Body="&lt;p&gt;StandardScaler subtracts the mean from each variable and then divides it by the standard deviation. It's a common preprocessing step, certainly for k-means because this algorithm heavily depends on the scaling of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand correctly you want to visualize the original data and make use of the labels from k-means by doing so. You could either add the labels to the original data (assuming the order of the records did not change):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;original_with_label = numpy.concatenate(original, labels, axis = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you could transform the data back to its original scale:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transformed_back_to_original = scalar_fit.inverse_transform(transformed_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2016-07-16T22:32:36.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12824" PostTypeId="2" ParentId="12821" CreationDate="2016-07-17T04:51:15.673" Score="4" Body="&lt;p&gt;I know this isn't answering the question that you actually asked, but I suggest that you &lt;strong&gt;NOT&lt;/strong&gt; generate data for your 'short text' categorization problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone new to data science and machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you avail yourself of some &lt;a href=&quot;https://datascience.stackexchange.com/questions/155/publicly-available-datasets/259#259&quot;&gt;free and open data here&lt;/a&gt;, or an &lt;a href=&quot;https://opendata.stackexchange.com/&quot;&gt;entire stack exchange dedicated to open data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Real data will give you experience with less contrived problems and you can often try out similar ML models on very different data sets in order to gain experience with the variations and pitfalls of data science. Be creative... you might not find exactly what you want, but you can subset the data by projecting out a few text fields and using them to classify another field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your recent edit, it is now apparent that you are seeking to turn your unsupervised training data into supervised training data in order to train a supervised learning classification model.  The method that you suggested, &quot;classify some dataset using regex or lucene rule based matches and manually verify them&quot;, is a deterministic unsupervised method without the human verification step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Without the human verification of the target data&lt;/strong&gt; that you have created, I would &lt;strong&gt;not&lt;/strong&gt; consider feeding the derived targets back into the classification algortihm as your results will only be as strong as the derived target data and predictions will show similar errors. Instead, you should think about a semi-supervised learning method where you perhaps employ a clustering algorithm and then label the clusters with the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With human verification of the target data&lt;/strong&gt; that you have created, this will work just fine as training data for a classification model. The only issue is that this can become tedious. There are &lt;a href=&quot;https://www.mturk.com/mturk/welcome&quot; rel=&quot;nofollow noreferrer&quot;&gt;mechanical turks&lt;/a&gt; (humans paid to perform repetitive tasks) that you can hire to perhaps perform the labeling for you, which may be a more scalable option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:57:29.190" LastActivityDate="2016-07-18T20:02:24.643" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12828" PostTypeId="2" ParentId="12186" CreationDate="2016-07-17T09:57:11.643" Score="3" Body="&lt;p&gt;Based on the feedback and trying to find more effective approach I developed the following algorithm using a dedicated distance measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following steps are performed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Define a &lt;strong&gt;distance metric&lt;/strong&gt; returning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;zero&lt;/em&gt; - if the points do not belong to a line&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Euclidian distance of the points&lt;/em&gt; - if the points constitute a line according to the defined parameters, i.e.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;their distance is higher or equal than the &lt;em&gt;min_line_length&lt;/em&gt; and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;their distance is lower or equal than the &lt;em&gt;max_line_length&lt;/em&gt; and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the line consists of at least &lt;em&gt;min_line_points&lt;/em&gt; points with a distance lower that &lt;em&gt;line_width/2&lt;/em&gt; from the line                        &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;2) Calculate &lt;strong&gt;distance matrix&lt;/strong&gt; using this distance measure (use sample of the data for large data sets; adjust the line parameters accordingly)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Find the points A and B with &lt;strong&gt;maximum distance&lt;/strong&gt; - break to step 5) if the distance is zero&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that if the distance is higher than zero the points A and B are building a line based on our definition&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Get all points belonging to the &lt;strong&gt;line AB&lt;/strong&gt; and remove them from the distance matrix. Repeat the step 3) to find another line&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Check the coverage of the point with the selected lines, if substantial number of points remains uncovered, repeat the whole algorithm with adjusted line parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) In case that data sample was used - reassign all points to the lines and recalculate the boundary points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following parameters are used:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;line width&lt;/strong&gt; - line_width/2 is the allowed distance of the point from the ideal line =   &lt;code&gt;r line_width&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;minimum line length&lt;/strong&gt;  - points with shorter distance are not considered to belong to the same line =  &lt;code&gt;r min_line_length&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;maximum line length&lt;/strong&gt;  - points with longer distance are not considered to belong to the same line =  &lt;code&gt;r max_line_length&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;minimum points on a line&lt;/strong&gt; - lines with less points are ignored = &lt;code&gt;r min_line_points&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With &lt;a href=&quot;http://www.db-nemec.com/linear_clustering/linear_clustering_3d.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;your data&lt;/a&gt; (after some fiddling with parameters) I got a good result covering all 30 lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DtcVQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DtcVQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details can be found in the &lt;a href=&quot;http://www.db-nemec.com/linear_clustering/linear_clustering_3d.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;knitr script&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2016-07-17T10:05:26.880" LastActivityDate="2016-07-17T10:05:26.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12829" PostTypeId="2" ParentId="12818" CreationDate="2016-07-17T10:03:22.907" Score="1" Body="&lt;p&gt;Another approach would be to explore weekly or fortnightly patterns, as it is possible that your outliers may simply depend on such trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way could be also to plot on the x axis the day of the month, on the y axis the month of the year, and use different shades or colors to highlight the absolute or relative power consumption.&lt;/p&gt;&#xA;" OwnerUserId="17269" LastActivityDate="2016-07-17T10:03:22.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12830" PostTypeId="1" AcceptedAnswerId="12833" CreationDate="2016-07-17T13:23:22.600" Score="87" ViewCount="51387" Body="&lt;p&gt;I recently read &lt;a href=&quot;https://www.facebook.com/yann.lecun/posts/10152820758292143&quot; rel=&quot;noreferrer&quot;&gt;Yan LeCuns comment on 1x1 convolutions&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;In Convolutional Nets, there is no such thing as &amp;quot;fully-connected layers&amp;quot;. There are only convolution layers with 1x1 convolution kernels and a full connection table.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;It's a too-rarely-understood fact that ConvNets don't need to have a fixed-size input.&lt;/p&gt;&#xA;&lt;p&gt;You can train them on inputs that happen to produce a single output vector (with no spatial extent), and then apply them to larger images.&lt;/p&gt;&#xA;&lt;p&gt;Instead of a single output vector, you then get a spatial map of output vectors.  Each vector sees input windows at different locations on the input.&#xA;&lt;strong&gt;In that scenario, the &amp;quot;fully connected layers&amp;quot; really act as 1x1 convolutions.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I would like to see a simple example for this.&lt;/p&gt;&#xA;&lt;h2&gt;Example&lt;/h2&gt;&#xA;&lt;p&gt;Assume you have a fully connected network. It has only an input layer and an&#xA;output layer. The input layer has 3 nodes, the output layer has 2 nodes. This&#xA;network has &lt;span class=&quot;math-container&quot;&gt;$3 \cdot 2 = 6$&lt;/span&gt; parameters. To make it even more concrete, lets say you have a ReLU activation function in the output layer and the weight matrix&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\begin{align}&#xA;W &amp;amp;= &#xA;\begin{pmatrix}&#xA; 0 &amp;amp; 1 &amp;amp; 1\\&#xA; 2 &amp;amp; 3 &amp;amp; 5\\&#xA;\end{pmatrix} \in \mathbb{R}^{2 \times 3}\\&#xA;b &amp;amp;= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2&#xA;\end{align}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;So the network is &lt;span class=&quot;math-container&quot;&gt;$f(x) = ReLU(W \cdot x + b)$&lt;/span&gt; with &lt;span class=&quot;math-container&quot;&gt;$x \in \mathbb{R}^3$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;How would the convolutional layer have to look like to be the same? What does LeCun mean with &amp;quot;full connection table&amp;quot;?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I guess to get an equivalent CNN it would have to have exactly the same number of parameters. The MLP from above has &lt;span class=&quot;math-container&quot;&gt;$2 \cdot 3 + 2 = 8$&lt;/span&gt; parameters.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-12-31T18:18:43.523" LastActivityDate="2021-12-31T18:18:43.523" Title="How are 1x1 convolutions the same as a fully connected layer?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12833" PostTypeId="2" ParentId="12830" CreationDate="2016-07-17T21:41:03.787" Score="43" Body="&lt;h2&gt;Your Example&lt;/h2&gt;&#xA;&lt;p&gt;In your example we have 3 input and 2 output units. To apply convolutions, think of those units having shape: &lt;code&gt;[1,1,3]&lt;/code&gt; and &lt;code&gt;[1,1,2]&lt;/code&gt;, respectively. In CNN terms, we have &lt;code&gt;3&lt;/code&gt; input and &lt;code&gt;2&lt;/code&gt; output feature maps, each having spatial dimensions &lt;code&gt;1 x 1&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Applying an &lt;code&gt;n x n&lt;/code&gt; convolution to a layer with &lt;code&gt;k&lt;/code&gt; feature maps, requires you to have a kernel of shape &lt;code&gt;[n,n,k]&lt;/code&gt;. Hence the kernel of your &lt;code&gt;1x1&lt;/code&gt; convolutions have shape &lt;code&gt;[1, 1, 3]&lt;/code&gt;. You need &lt;code&gt;2&lt;/code&gt; of those kernels (or filters) to produce the &lt;code&gt;2&lt;/code&gt; output feature maps. Please Note: &lt;span class=&quot;math-container&quot;&gt;$1 \times 1$&lt;/span&gt; convolutions really are  &lt;span class=&quot;math-container&quot;&gt;$1 \times 1 \times \text{number of channels of the input}$&lt;/span&gt; convolutions. The last one is only rarely mentioned.&lt;/p&gt;&#xA;&lt;p&gt;Indeed if you choose as kernels and bias:&lt;br /&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\begin{align}&#xA;w_1 &amp;amp;= &#xA;\begin{pmatrix}&#xA; 0 &amp;amp; 1 &amp;amp; 1\\&#xA;\end{pmatrix} \in \mathbb{R}^{3}\\&#xA;w_2 &amp;amp;= &#xA;\begin{pmatrix}&#xA;2 &amp;amp; 3 &amp;amp; 5\\&#xA;\end{pmatrix} \in \mathbb{R}^{3}\\&#xA;b &amp;amp;= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2&#xA;\end{align}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The conv-layer will then compute &lt;span class=&quot;math-container&quot;&gt;$f(x) = ReLU\left(\begin{pmatrix}w_1 \cdot x\\ w_2 \cdot x\end{pmatrix} + \begin{pmatrix}b_1\\ b_2\end{pmatrix}\right)$&lt;/span&gt; with &lt;span class=&quot;math-container&quot;&gt;$x \in \mathbb{R}^3$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h2&gt;Transformation in real Code&lt;/h2&gt;&#xA;&lt;p&gt;For a real-life example, also have a look at my &lt;a href=&quot;https://github.com/MarvinTeichmann/tensorflow-fcn/blob/d04bc268ac6e84f03afc4332d7f54ecff22d1732/fcn32_vgg.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;vgg-fcn&lt;/a&gt; implementation. The Code provided in this file takes the VGG weights, but transforms every fully-connected layer into a convolutional layers. The resulting network yields the same output as &lt;code&gt;vgg&lt;/code&gt; when applied to input image of shape &lt;code&gt;[244,244,3]&lt;/code&gt;. (When applying both networks without padding).&lt;/p&gt;&#xA;&lt;p&gt;The transformed convolutional layers are introduced in the function &lt;code&gt;_fc_layer&lt;/code&gt; (line 145). They have kernel size &lt;code&gt;7x7&lt;/code&gt; for FC6 (which is maximal, as &lt;code&gt;pool5&lt;/code&gt; of VGG outputs a feature map of shape &lt;code&gt;[7,7, 512]&lt;/code&gt;. Layer &lt;code&gt;FC7&lt;/code&gt; and &lt;code&gt;FC8&lt;/code&gt; are implemented as &lt;code&gt;1x1&lt;/code&gt; convolution.&lt;/p&gt;&#xA;&lt;h2&gt;&amp;quot;Full Connection Table&amp;quot;&lt;/h2&gt;&#xA;&lt;p&gt;He might refer to a filter/kernel which has the same dimension as the input feature map. In both cases (Code and your Example) the spatial dimensions are maximal in the sense, that the spatial dimension of the filter is the same as the spatial dimension as the input.&lt;/p&gt;&#xA;" OwnerUserId="13662" LastEditorUserId="127754" LastEditDate="2021-12-29T20:43:55.570" LastActivityDate="2021-12-29T20:43:55.570" CommentCount="13" ContentLicense="CC BY-SA 4.0" />
  <row Id="12834" PostTypeId="2" ParentId="12820" CreationDate="2016-07-18T04:07:44.977" Score="4" Body="&lt;p&gt;It seems like you have a classical bayesian problem. You have some sort of prior distribution, a distribution over years of birth, your prior distribution is bimodal with peaks at the two years,  you can probably use a convolution of two normal distributions to model this variable. Then have it spit out a posterior distribution after you feed in some data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real problem that I have with this analysis is it seems your features aren't particularly good. It is true these vars might have information about birth year, for example for the 20th century the average age of first marriage has steadily been increasing. But I suspect that the signal is going to be fairly weak. Essentially, if I tell you that I got married at age 24, had my first child at 26, and that my older brother is 3 years older than me and my younger sister is 2 years younger than me, can you tell me in what year was I born, 1956 or 1989?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that without additional data this information that I provided would be completely useless, mostly because it is a very noisy signal. That information could apply equally to someone born in 1956 or 1989. It isn't very helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, what I am saying is that when you update your prior, it isn't going to change very much. (Your posterior distribution would look very similar to the prior distribution.) Instead of doing some mustache twirling over what is the right algorithm to crack this problem, I think a much more fruitful exercise would be to think up some better features.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2016-07-18T04:07:44.977" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12836" PostTypeId="1" AcceptedAnswerId="12837" CreationDate="2016-07-18T08:33:29.880" Score="0" ViewCount="191" Body="&lt;p&gt;I have recently confronted with a (at least for me) new kind of ML problem, where the output of the model should be a vector/matrix (depending on the interpretation, but there is no difference actually), not a scalar as usual. This is totally unknown for me. What kind of approach should one apply here? Are the &quot;usual&quot; (scalar-based) models applicable on this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Just for the sake of completeness, the problem is an image segmentation task where the model should decide first: if there a given pattern on the picture?, second: if so, where is it? - In latter case, it should define the borders of the subset pixels).&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-07-18T09:15:15.073" Title="Prediction with non-scalar output (label)" Tags="|machine-learning|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12837" PostTypeId="2" ParentId="12836" CreationDate="2016-07-18T09:15:15.073" Score="1" Body="&lt;p&gt;Neural Networks can have a vector or matrix as output layer, image segmentation is a well researched topic and deep learning (as most things concerning images) are the state-of-the-art. You will need (a lot of) training examples where the pattern is found, and where. This could be a bounding box, or per pixel if it is part of the pattern or not (this will generate a matrix equally sized to your input). To see if the pattern is found you could construct a second network that is just a binary classifier, or you could try to see if your pixel-based network will output almost only zeros in case of no pattern. In this case you will need negative examples as well.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-07-18T09:15:15.073" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12839" PostTypeId="1" CreationDate="2016-07-18T11:00:24.413" Score="0" ViewCount="1282" Body="&lt;p&gt;I am trying to create a association using apriori algorithm.The data contains around 33000 records.Below is the sample of the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id  code&#xA;1   19&#xA;1   58&#xA;1   111&#xA;2   19&#xA;2   111&#xA;2   167&#xA;3   12&#xA;3   79&#xA;3   85&#xA;4   96&#xA;5   19&#xA;6   58&#xA;7   12&#xA;7   18&#xA;7   40&#xA;7   48&#xA;7   85&#xA;7   86&#xA;7   135&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In R this data is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;structure(list(id = c(1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5, 6, 7, &#xA;7, 7, 7, 7, 7, 7), code = c(19, 58, 111, 19, 111, 167, 12, 79, &#xA;85, 96, 19, 58, 12, 18, 40, 48, 85, 86, 135)), .Names = c(&quot;id&quot;, &#xA;&quot;code&quot;), row.names = c(NA, -19L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using the following code I tried to build the association&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# creating a string of codes based on the id&#xA;library(dplyr)&#xA;Asso2 = Asso %&amp;gt;% &#xA;  group_by(patient_id) %&amp;gt;% &#xA;  summarise(hi = toString(hcc_id))&#xA;# converting into transactions&#xA;library(arules)&#xA;fact &amp;lt;- data.frame(lapply(Asso2,as.factor))&#xA;trans &amp;lt;- as(fact, 'transactions') &#xA;# Applying aprior&#xA;rules = apriori(trans, parameter = list(supp = 0.001, conf = 0.001,target = &quot;rules&quot;))&#xA;rules&#xA;inspect(rules)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting totally 96 rules like below with empty lhs and I trying to understand whether we cannot make any rules from this data or am I missing anything here. Since I am novice in this, I would like to get some help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#     lhs    rhs               support     confidence  lift&#xA;#  1  {}  =&amp;gt; {hi=19, 96, 108}  0.001021696 0.001021696 1   &#xA;#  2  {}  =&amp;gt; {hi=176}          0.001021696 0.001021696 1   &#xA;#  3  {}  =&amp;gt; {hi=88, 108}      0.001021696 0.001021696 1   &#xA;#  4  {}  =&amp;gt; {hi=72}           0.001051746 0.001051746 1   &#xA;#  5  {}  =&amp;gt; {hi=88, 96}       0.001051746 0.001051746 1   &#xA;#  6  {}  =&amp;gt; {hi=108, 112}     0.001081796 0.001081796 1   &#xA;#  7  {}  =&amp;gt; {hi=84}           0.001081796 0.001081796 1   &#xA;#  8  {}  =&amp;gt; {hi=100, 103}     0.001111846 0.001111846 1   &#xA;#  9  {}  =&amp;gt; {hi=18, 108, 111} 0.001111846 0.001111846 1  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18013" LastEditorUserId="12384" LastEditDate="2017-02-28T16:01:55.740" LastActivityDate="2017-02-28T16:01:55.740" Title="Association rule mining interpretation" Tags="|machine-learning|r|association-rules|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12840" PostTypeId="1" AcceptedAnswerId="12951" CreationDate="2016-07-18T12:16:34.387" Score="1" ViewCount="3135" Body="&lt;p&gt;I have 2 vectors $1000 \times 1$, lets call them $y_{1}$ and $y_{2}$. Each vector represents a normal distribution with certain mean and variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I plot the contour plot using the following R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- y1&#xA;y &amp;lt;- y2&#xA;s &amp;lt;- subplot(&#xA;  plot_ly(x = x, type = &quot;histogram&quot;, showlegend=FALSE),&#xA;  plotly_empty(),&#xA;  plot_ly(x = x, y = y, type = &quot;histogram2dcontour&quot;, showlegend=FALSE),&#xA;  plot_ly(y = y, type = &quot;histogram&quot;, showlegend=FALSE),&#xA;  nrows = 2, heights = c(0.2, 0.8), widths = c(0.8, 0.2),&#xA;  shareX = TRUE, shareY = TRUE, titleX = FALSE, titleY = FALSE&#xA;)&#xA;layout(s)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and i get the following plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cjALm.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cjALm.jpg&quot; alt=&quot;contour plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do now, is to plot a 3D contour plot (so that I can actually see the &quot;mountain&quot; that is created after plotting the histogram of $y_{1}$ against the histogram of $y_{2}$). So in the z-axis I would like to have the frequencies of the values. Any suggestions ?&lt;/p&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2016-07-18T12:33:40.433" LastActivityDate="2016-07-23T02:14:09.287" Title="3d contour plot in R" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12841" PostTypeId="1" AcceptedAnswerId="13015" CreationDate="2016-07-18T12:39:29.543" Score="1" ViewCount="96" Body="&lt;p&gt;Let $$X_1\sim N(\mu_{X_1},\sigma_{X_2}^2)$$ $$X_2\sim N(\mu_{X_2},&#xA;\sigma_{X_2}^2)$$&#xA;where $\mu_{X_2}=c+aX_1$. Also, I have data $D$ (with missing values on $X_1,X_2$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I update/estimate the parameters $\mu_{X_1},\sigma_{X_1},\mu_{X_2},a,c,\sigma_{X_2}$ using EM?i.e. what is the formula for updating $\sigma_{X_2}$?   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model is a conditional Gaussian, which is a conditional form of a bivariate Gaussian $(X_1,X_2)$ with mean vector $(\mu_1,\mu_2)^\top$ and covariance matrix &#xA;$$\left( \begin{matrix}&#xA;   \Sigma _{11} &amp;amp; \Sigma _{12}  \\&#xA;   \Sigma _{21} &amp;amp; \Sigma _{22}  \\&#xA;\end{matrix} \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a reference to convert bivariate Gaussian to conditional Gaussian: $$\mu_{2|1}=\mu_{2}+\Sigma_{21}\Sigma_{11}^{-1}(X_1-\mu_1)\quad,\quad \Sigma_{22|1}=\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}$$ which yields my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that when $X_1$ has different observations and $X_2$ is unobserved, the variance of $X_2$ remains unchanged.So how to update the $\sigma_{x_2}$?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initial setting for the model&#xA;$$X_1\sim N(5,7)\quad,\quad X_2\sim N(0.5X_1,8)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X_1:\operatorname{9\,\,4\,\,NA}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\quad X_2:\operatorname{NA\,\,NA\,\,3}$$&lt;/p&gt;&#xA;" OwnerUserId="21565" LastEditorUserId="21565" LastEditDate="2016-07-26T22:31:57.570" LastActivityDate="2016-07-27T10:20:01.633" Title="EM parameter estimation for conditional Gaussians" Tags="|parameter-estimation|gaussian|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12842" PostTypeId="2" ParentId="12819" CreationDate="2016-07-18T13:03:45.847" Score="2" Body="&lt;p&gt;SAS initializes the length and type of the variable, I.e. creates and assigns a position for it in the program data vector, at the first reference to the variable it sees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If (condition) then output='I'; defines it as length 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Workarounds&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your discovered solution of having the same lengths.&lt;/li&gt;&#xA;&lt;li&gt;The first reference to the variable being &lt;em&gt;LENGTH output $2;&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Assigning the longest length first. &lt;/li&gt;&#xA;&lt;li&gt;The first reference to the variable being &lt;em&gt;If 0 then output='zz';&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can confirm the issue, which also pops up when concatenating datasets, by looking at variable properties with proc contents or proc datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the documentation: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note:   When you create character variables, SAS determines the length of the variable from its first occurrence in the DATA step. Therefore, you must allow for the longest possible value in the first statement that mentions the variable. If you do not assign the longest value the first time the variable is assigned, then data can be truncated.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://support.sas.com/documentation/cdl/en/basess/58133/HTML/default/viewer.htm#a001336069.htm&quot; rel=&quot;nofollow&quot;&gt;http://support.sas.com/documentation/cdl/en/basess/58133/HTML/default/viewer.htm#a001336069.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data a;&#xA;    input firstname $ lastname $ salary;&#xA;    cards;&#xA;Billy Bob 20000&#xA;Bob Graham 50000&#xA;Graham Alice 30000&#xA;Alice Graham 40000&#xA;;&#xA;&#xA;proc print data=a; run;&#xA;&#xA;*Did not work;&#xA;DATA test;&#xA;set a;&#xA;if salary &amp;lt; 35000 then grade = 'I';&#xA;else grade = 'II';&#xA;run; proc print data = test; run;&#xA;&#xA;*Worked;&#xA;DATA test;&#xA;length grade $ 2;&#xA;set a;&#xA;if salary&amp;lt;35000 then grade = 'I';&#xA;else grade = 'II';&#xA;run; proc print data = test; run;&#xA;&#xA;*Worked;&#xA;DATA test;&#xA;set a;&#xA;if salary&amp;gt;=35000 then grade = 'II';&#xA;else grade = 'I';&#xA;run; proc print data = test; run;&#xA;&#xA;*Worked;&#xA;DATA test;&#xA;if 0 then grade='ZZ';&#xA;set a;&#xA;if salary&amp;lt;35000 then grade = 'I';&#xA;else grade = 'II';&#xA;run; proc print data = test; run;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, as I stated, you can run &lt;em&gt;PROC CONTENTS DATA=TEST; RUN;&lt;/em&gt; after each data step to confirm the data length of each column.&lt;/p&gt;&#xA;" OwnerUserId="21566" LastEditorUserId="21566" LastEditDate="2016-08-07T18:40:11.000" LastActivityDate="2016-08-07T18:40:11.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12843" PostTypeId="2" ParentId="12839" CreationDate="2016-07-18T13:12:12.737" Score="1" Body="&lt;p&gt;Yes, this basically means that no sensible rules can be built with the constrains of support and confidence you impose. Note that your &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning#Lift&quot; rel=&quot;nofollow&quot;&gt;lift&lt;/a&gt; is always trivially one. The solution to this is given in the &lt;a href=&quot;https://cran.r-project.org/web/packages/arules/arules.pdf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; of the package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Note: The default value in APparameter for minlen is 1.&#xA;This means that rules with only one item (i.e., an empty&#xA;antecedent/LHS) like {} =&amp;gt; {beer} will be created. These&#xA;rules mean that no matter what other items are involved&#xA;the item in the RHS will appear with the probability given&#xA;by the rule's confidence (which equals the support). If&#xA;you want to avoid these rules then use the argument&#xA;parameter=list(minlen=2).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-18T13:12:12.737" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12844" PostTypeId="1" CreationDate="2016-07-18T14:13:55.867" Score="1" ViewCount="140" Body="&lt;p&gt;Deep learning seems to be the new cool thing in AI/machine learning and it works well in many domains, but I want to know- what are the specific application areas where deep learning is not the best approach and what is the reason for that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has some evaluation been made? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a particular class of problems? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so- what solutions are superior in solving that task? &lt;/p&gt;&#xA;" OwnerUserId="8285" LastActivityDate="2016-07-18T19:40:48.017" Title="which application domains are not well suited to deep learning?" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12846" PostTypeId="1" AcceptedAnswerId="13832" CreationDate="2016-07-18T15:49:11.970" Score="5" ViewCount="3630" Body="&lt;p&gt;Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the python module scipy I could either use: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html&quot; rel=&quot;noreferrer&quot;&gt;http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or I could use: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mstats.normaltest.html&quot; rel=&quot;noreferrer&quot;&gt;http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mstats.normaltest.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example on the same feature I get these outputs: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scipy.stats.shapiro(data[:,0])&#xA;(0.9985173940658569, 0.77845299243927)&#xA;&#xA;&#xA;scipy.stats.mstats.normaltest(data[:,0])&#xA;NormaltestResult(statistic=1.492603328675163, pvalue=0.47411675723570479)&#xA;&#xA;&#xA;scipy.stats.anderson(data[:,0], dist='norm')&#xA;AndersonResult(statistic=0.17542490527580412, critical_values=array([ 0.573,  0.653,  0.783,  0.913,  1.086]), significance_level=array([ 15. ,  10. ,   5. ,   2.5,   1. ]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10584" LastEditorUserId="10584" LastEditDate="2016-07-18T16:08:19.470" LastActivityDate="2016-09-03T04:44:44.073" Title="How to decided which test of normality to use" Tags="|statistics|distributed|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12847" PostTypeId="1" CreationDate="2016-07-18T16:15:06.700" Score="1" ViewCount="426" Body="&lt;p&gt;I'm trying to make human activity recognition using the iOS accelerometer and gyroscope. The feature that I use are mean, variance, standard deviation, energy and cross-domain entropy. If I provide Weka with a test set of, let's say, 30% of the total data, it gives me an accuracy of 98.7%.&#xA;But when I implement the J48 decision tree in my application, the decision tree literally doesn't work and it doesn't classify activities correctly (the precision is near 0%), although I'm taking samples the same way that I take samples for the test set. The same data used for classification in the app gives incorrect results in the app but a 98% precision in weka. How is this possibile? Maybe I'm implementing the decision tree the wrong way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LYW9P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LYW9P.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I simply read the decision tree this way: if Var(Gravity.z) is less or equal than 47572.90768, then I check Energy(GravityMag) and if it's less or equal than 31250.004174, then I check Energy(UserAcc.y) and so on... since Weka assumes that if the difference between two numbers is smaller than 10e-6, I multiplied all numbers for one million so the precision goes up to 10e-12, and I double checked that I'm also correctly multiplying the values to be used by the in-app classifier so that couldn't be the problem. If the precision on the test set is high it couldn't be overfitting so I really don't know what to think.&lt;/p&gt;&#xA;" OwnerUserId="21197" LastActivityDate="2016-07-18T16:15:06.700" Title="Weka Decision Tree not working on real data" Tags="|machine-learning|classification|data-mining|weka|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12848" PostTypeId="2" ParentId="12743" CreationDate="2016-07-18T16:16:56.043" Score="0" Body="&lt;p&gt;I think it does not make much sense to look for a correlation here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wild guess: &#xA;This looks like the result of a cluster analysis to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read these &lt;code&gt;=&amp;gt;&lt;/code&gt; as 'corresponds best with'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cluster...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E =&amp;gt; cobalt or gold&#xA;D =&amp;gt; concrete&#xA;C =&amp;gt; copper&#xA;B =&amp;gt; copper&#xA;A =&amp;gt; copper or coal&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Difficult to tell without domain knowledge.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-07-18T16:16:56.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12849" PostTypeId="2" ParentId="10959" CreationDate="2016-07-18T16:51:28.790" Score="3" Body="&lt;p&gt;Feed-forward neural networks are &lt;a href=&quot;https://en.wikipedia.org/wiki/Discriminative_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;discriminative models&lt;/a&gt;, i.e. they model $P(y|x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;restricted Boltzmann machines&lt;/a&gt; look a lot like neural networks, and share some of the proprieties, and are &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;generative models&lt;/a&gt;, i.e. they model $P(y,x)$ --- which, of course, you can always turn into a $P(y|x)$ using Bayes' rule, but the opposite is not true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Libraries like Theano &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;have support for RBMs&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; since the writing of my reply, generative adversarial networks have been invented. (Actually, they were invented in 2014, but only in 2016 were they taken to their full potential.) In these networks, you feed noise to the neural network and it generates data samples. This is achieved by co-training it against a regular discriminative neural network. Both networks are trained at the same time: the discriminative one must classify each sample as being either true or synthetically created by the other neural network. And the generative neural network must fool this one, so they get better and better, and, in the end, you have both a discriminative and a generative model!&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2017-01-10T23:08:03.627" LastActivityDate="2017-01-10T23:08:03.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12850" PostTypeId="1" CreationDate="2016-07-18T16:55:05.967" Score="1" ViewCount="1074" Body="&lt;p&gt;I had a question regarding about a career as a data scientist.  Im pursuing a degree in business analytics with a minor computer science.  Could i still become a data scientist with a bachelors degree in business analytics. Or would i have to pursue a masters degree in data science in order to become a data scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I have a good programming background in data structures and algorithm in java, plus web development, python)&lt;/p&gt;&#xA;" OwnerUserId="21573" LastActivityDate="2016-07-19T03:39:40.507" Title="Can i pursue a career in data science with a business analytics degree?" Tags="|education|career|" AnswerCount="2" CommentCount="1" ClosedDate="2016-07-19T11:33:12.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="12851" PostTypeId="1" CreationDate="2016-07-18T17:08:17.237" Score="178" ViewCount="250352" Body="&lt;p&gt;When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are good / simple ways to visualize common architectures automatically?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="28175" LastEditDate="2018-01-22T12:01:18.113" LastActivityDate="2023-12-15T08:52:26.853" Title="How do you visualize neural network architectures?" Tags="|machine-learning|neural-network|deep-learning|visualization|" AnswerCount="21" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12852" PostTypeId="2" ParentId="11452" CreationDate="2016-07-18T17:22:07.817" Score="2" Body="&lt;p&gt;I know that neither xgboost, nor sklearn offer what you want. I have checked R, and didn't find it either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But random forests are easy models to implement, so you can just produce one by yourself:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, in python using sklearn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;from scipy.stats import mode&#xA;&#xA;&#xA;class MyRandomForest:&#xA;    def __init__(self, Pcol, Pobs, n_estimators=10):&#xA;        self.n_estimators = n_estimators&#xA;        self.Pcol = Pcol  # vector&#xA;        self.Pobs = Pobs  # scalar&#xA;&#xA;    def fit(self, X, y):&#xA;        self.classes_ = np.unique(y)&#xA;        assert len(self.Pcol) == X.shape[1]&#xA;        self.cols = []&#xA;        self.ms = []&#xA;        while True:&#xA;            j = np.random.rand(X.shape[1]) &amp;lt;= self.Pcol&#xA;            if not np.any(j):  # at least one column must be chosen!&#xA;                continue&#xA;            x = X[:, j]&#xA;            i = np.random.choice(range(len(X)), int(len(X)*self.Pobs), False)&#xA;            self.cols.append(j)&#xA;            self.ms.append(DecisionTreeClassifier().fit(x[i], y[i]))&#xA;            if len(self.ms) == self.n_estimators:&#xA;                break&#xA;        return self&#xA;&#xA;    def predict(self, X):&#xA;        yp = [m.predict(X[:, cs]) for cs, m in zip(self.cols, self.ms)]&#xA;        yp = mode(yp, 0)[0][-1]&#xA;        return yp&#xA;&#xA;if __name__ == '__main__':  # TEST&#xA;    from sklearn.datasets import load_iris&#xA;    from sklearn.cross_validation import StratifiedKFold&#xA;    from sklearn.metrics import accuracy_score&#xA;    iris = load_iris()&#xA;    X = iris.data&#xA;    y = iris.target&#xA;    for tr, ts in StratifiedKFold(y):&#xA;        m = MyRandomForest([0.5, 1, 0.2, 0.3], 1).fit(X[tr], y[tr])&#xA;        print(accuracy_score(y[ts], m.predict(X[ts])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are using Linux, you can use &lt;code&gt;multiprocessing&lt;/code&gt; to easily make it run in parallel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did was just:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;train a sequence of individual trees, using your probabilities&lt;/li&gt;&#xA;&lt;li&gt;save both the models and whichever columns were sampled for training because we want to use only those for evaluation (or we will get an error)&lt;/li&gt;&#xA;&lt;li&gt;using mode to get the most voted prediction&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: &lt;code&gt;Pobs&lt;/code&gt; is the fraction of observations to use. You may also want to change &lt;code&gt;np.random.choice(..., ..., False)&lt;/code&gt; to &lt;code&gt;np.random.choice(..., ..., True)&lt;/code&gt;, or make it configurable, to allow for bootstrap with resampling. Usually, random forests are trained using &lt;code&gt;Pobs=1&lt;/code&gt; and &lt;code&gt;resample=True&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-07-19T08:12:57.170" LastActivityDate="2016-07-19T08:12:57.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12853" PostTypeId="2" ParentId="12850" CreationDate="2016-07-18T17:32:31.150" Score="2" Body="&lt;p&gt;You can. Nobody is going to demand a degree, but they will expect you to pass the interviews. I would recommend creating a blog and do some Kaggle competitions to show off your skills. I would also recommend taking more serious maths courses from the statistics department.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-18T17:32:31.150" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12854" PostTypeId="1" AcceptedAnswerId="12857" CreationDate="2016-07-18T19:11:52.940" Score="3" ViewCount="439" Body="&lt;p&gt;How do I capture $y = x_1 x_2$ using a simple neural network with commonly used activation functions? I assume that I need at least one hidden layer. What mix of commonly activation functions should I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have used $max(0,x)$ and $tanh$ activation for the hidden layer, but the gradient descent diverges very quickly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some thoughts which may be useful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ (x_1 y_1)^2 = \exp(\log(x_1^2) + \log(x_2^2)) $$&lt;/p&gt;&#xA;" OwnerUserId="21576" LastActivityDate="2016-07-19T04:18:36.930" Title="What is the simplest neural network for the simplest nonlinear function $f(x,y) =xy$" Tags="|neural-network|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12855" PostTypeId="2" ParentId="12844" CreationDate="2016-07-18T19:40:48.017" Score="1" Body="&lt;p&gt;Deep learning is generally not well suited for domains that have only very small training datasets. This includes fields like medicine where, for example, patient data can only be obtained through expensive and time consuming clinical trials. Clinical trials will often contain only several hundreds records. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By design, deep learning models have many parameters (e.g., millions), and like with any machine learning algorithm, you're likely to see overfitting when the number of parameters greatly exceeds the number of training records. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-18T19:40:48.017" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12856" PostTypeId="2" ParentId="12851" CreationDate="2016-07-18T19:59:17.143" Score="52" Body="&lt;h2&gt;Tensorflow, Keras, MXNet, PyTorch&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If the neural network is given as a Tensorflow graph, then you can &lt;a href=&quot;https://www.tensorflow.org/guide/graph_viz&quot; rel=&quot;noreferrer&quot;&gt;visualize this graph with TensorBoard&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how the MNIST CNN looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zJHpV.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zJHpV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can add names / scopes (like &quot;dropout&quot;, &quot;softmax&quot;, &quot;fc1&quot;, &quot;conv1&quot;, &quot;conv2&quot;) yourself.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Interpretation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The following is only about the left graph. I ignore the 4 small graphs on the right half.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each box is a layer with parameters that can be learned. For inference, information flows from bottom to the top. Ellipses are layers which do not contain learned parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The color of the boxes does not have a meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure of the value of the dashed small boxes (&quot;gradients&quot;, &quot;Adam&quot;, &quot;save&quot;).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2019-09-23T08:55:47.800" LastActivityDate="2019-09-23T08:55:47.800" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="12857" PostTypeId="2" ParentId="12854" CreationDate="2016-07-18T21:01:19.523" Score="3" Body="&lt;p&gt;Probably you need to do one or more of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Decrease learning rate. Diverging loss is often a symptom of learning rate too high.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Increase number of hidden neurons. The output function will be the combination of many &quot;patches&quot; each created by a neuron that has learnt a different bias. The shape and quality of each patch is determined by the activation functions, but almost any nonlinear activation function used in an NN library should work to make a universal function approximator.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Normalise inputs. If you are training with high values for $x_1$ or $x_2$, this could make it harder to train unless you normalise your training data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For your purposes, it might be an idea to skip the need for normalisation by training with $x_1$ and $x_2$ in range -2.0 to 2.0. It doesn't change your goal much to do this, and removes one potential problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should note that a network trained against an open-ended function like this (where there are no logical bounds on the input params) will not learn to extrapolate. It never &quot;learns&quot; the function itself, but an approximation of it close to the supplied examples. When you supply  a $(x_1, x_2)$ input far from the training examples, the output is likely to completely mismatch your original function.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-07-18T21:24:18.937" LastActivityDate="2016-07-18T21:24:18.937" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12858" PostTypeId="2" ParentId="12743" CreationDate="2016-07-18T22:26:18.470" Score="0" Body="&lt;p&gt;I'm also not sure what you would like to compute - your exemplary data only has one observation for each type, which makes the computation of a co-variation aka correlation impossible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you‘re interested in prediction, for example the probability that a new observation with values for coal, cobalt, concrete, copper and gold will be of a certain type, linear discriminant analysis (LDA) might be what you're looking for. LDA is implemented in the &lt;a href=&quot;https://cran.r-project.org/web/packages/MASS/index.html&quot; rel=&quot;nofollow&quot;&gt;MASS package&lt;/a&gt; and nicely explained on r-bloggers: &lt;a href=&quot;http://www.r-bloggers.com/computing-and-visualizing-lda-in-r/&quot; rel=&quot;nofollow&quot;&gt;http://www.r-bloggers.com/computing-and-visualizing-lda-in-r/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, you'll need more observations per Type to use LDA.&lt;/p&gt;&#xA;" OwnerUserId="21580" LastActivityDate="2016-07-18T22:26:18.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12859" PostTypeId="2" ParentId="12851" CreationDate="2016-07-19T00:43:43.137" Score="21" Body="&lt;p&gt;In Caffe you can use  &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/python/caffe/draw.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;caffe/draw.py&lt;/a&gt; to draw the NetParameter protobuffer:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5Z1Cb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Z1Cb.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In Matlab, you can use  &lt;a href=&quot;http://www.mathworks.com/help/nnet/ref/view.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;view(net)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rPpfa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rPpfa.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://transcranial.github.io/keras-js/#/inception-v3&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras.js&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vEfTv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vEfTv.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also, see &lt;a href=&quot;https://redd.it/kmfx2d&quot; rel=&quot;nofollow noreferrer&quot;&gt;Can anyone recommend a Network Architecture visualization tool? (Reddit/self.MachineLearning)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2020-12-29T23:41:57.220" LastActivityDate="2020-12-29T23:41:57.220" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12861" PostTypeId="2" ParentId="12850" CreationDate="2016-07-19T03:33:25.413" Score="2" Body="&lt;p&gt;This depends a lot on you, your ability to apply your knowledge of statistics, computer science and mathematics to abstract problems and also your problem solving intuition and prowess. A great many jobs in the US require Masters or Ph.D.s, but there are some that do not. But add to this, your lack of a degree in a truly quantitative discipline and it might make things tough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though you have cited some background in statistics, being a data science requires that you have the mathematical maturity to digest research papers and text books to apply abstract concepts to the real world. This typically requires the experience that is common to a mathematics BS or a Masters or Ph.D. in a slightly less rigorous field like engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would certainly consider taking 2 years of calculus and enough linear algebra to get past matrix manipulation and really understand Fourier analysis, orthogonal vector spaces and Hilbert space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would advise you to seriously consider a Masters or Ph.D. program. If you don't want to do this, then try to take full advantage of your CSE minor to be a really strong programmer. You should be able to find a job in a smaller startup which will allow you to prove yourself over time. Be prepared that a lot of first time jobs may be Data Engineer positions, which require a lot of ETL and data munging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck in whatever you decide to pursue!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-07-19T03:39:40.507" LastActivityDate="2016-07-19T03:39:40.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12862" PostTypeId="2" ParentId="12854" CreationDate="2016-07-19T04:18:36.930" Score="1" Body="&lt;p&gt;To add onto the above answer, a simple feedforward network doesn't learn the function itself. But recently, Neural Turing Machines claim to learn the algorithm. Worth a shot!&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-07-19T04:18:36.930" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12863" PostTypeId="2" ParentId="12661" CreationDate="2016-07-19T07:42:53.293" Score="2" Body="&lt;p&gt;Since you have a transaction data set on your hands, we are talking clear-cut &lt;em&gt;supervised learning&lt;/em&gt; here. And while you could certainly try to use neural networks, I think you should start simpler and switch to more complex and costly algorithms once you see the simple ones failing (this is generally a good idea).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now form what you describe, you have some sort of time-series data, with a categorical quantity that you want to predict, for which you will want some sort of modified &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model&quot; rel=&quot;nofollow&quot;&gt;ARMA model&lt;/a&gt;. You could set up a logistic regression for each good you are interested in (i.e. beer, chocolate, potato, etc.) that will predict whether or not he will buy the product. The variables you feed into the model are a set of binary observations describing if he bought the product a day ago, a week ago, four weeks ago, etc.. Play around with those lags. The nice thing about logistic regression here is that you'll see right away what is the period of the purchases. If you see a large coefficient on the one-week-ago-variable, you know that he eats avocados weekly and so forth. Note that if something is purchased weekly, it's also purchased every four weeks, so you will have some correlation issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't get decent predictions, you could try different models, e.g. trees or ensemble methods such as random forests or boosted trees. If those don't work either, you probably have a time series without clear periodicity. In that case you could try setting up your variables differently, e.g. using the number of days since he bought the product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last word of advice: You might want to check correlation between purchases. Maybe he buys avocados, tomato, and onion together to make delicious guacamole. If he then has some onions leftover, he might be inclined to buy more avocado and so on.&lt;/p&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-19T07:42:53.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12864" PostTypeId="2" ParentId="12595" CreationDate="2016-07-19T08:29:59.030" Score="2" Body="&lt;p&gt;This depends on what you mean by 'unseen products'. If your definition is like in the paper&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[...] new  feature  combinations  that have  never  or  rarely &#xA;occurred  in the  past.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then you're thinking along the lines of algorithms like &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt;. Those see the tastes of people as missing values in a matrix and try to complete it using a low-dimensional representation. Now these can be overly generic, depending on how low you choose that dimension to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another, older, approach is something like association rules, which can be deducted using algorithms like &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;apriori&lt;/a&gt;, which however rely on the product combinations seen in the past, so they &lt;em&gt;won't ever&lt;/em&gt; be able recommend product combinations not seen in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the former set of algorithms can deal with what under the definition above would be called 'missing data', the latter cannot. Those however tend to give better recommendations. The paper you quote uses neural networks in what looks like a successful attempt to get the best of those two worlds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If hover you mean by 'missing data' completely new products, than you're out of luck using any of the algorithms described above. If you don't have meta-data about those products, it's near impossible to recommend anything. Just imagine you have a list of products A, B, and C and who bought them. Now you get product D. Who will buy it? Without any additional information on D, you're dead in the water.&lt;/p&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-19T08:29:59.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12865" PostTypeId="1" CreationDate="2016-07-19T08:53:36.117" Score="-6" ViewCount="3337" Body="&lt;p&gt;I am trying to find the sentiment of tweets using a NLP package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of a tweet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@SouthwestAir Fastest response all day. Hour on the phone: never got off&#xA;hold. Hour in line: never got to the Flight Booking Problems desk.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, NLP labels sentiment based on sentence. So, this tweet has three sentences with full-stops. Hence NLP gives me three different sentiment labels for each sentence of tweet. Now, how can I label entire tweet has positive, negative or neutral?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Neutral    @SouthwestAir Fastest response all day.&#xA;Negative    Hour on the phone: never got off hold.&#xA;Negative    Hour in line: never got to the Flight Booking Problems desk.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20497" LastEditorUserId="29169" LastEditDate="2019-06-09T23:23:06.790" LastActivityDate="2021-02-09T23:55:19.377" Title="How to label sentiment using NLP?" Tags="|machine-learning|nlp|sentiment-analysis|stanford-nlp|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12866" PostTypeId="1" CreationDate="2016-07-19T09:34:35.650" Score="3" ViewCount="238" Body="&lt;p&gt;Say I have $N$ categorical features $f_i$ $i\in(1,N)$ each of which of different alphabet size $n_i$. How can I efficiently optimize the hashing trick on that feature vector? Should I enumerate hash bit length for each feature independently (greedy)? Should i assume no hashing is necessary for features with small alphabet ($n_i$ is small)? What is the best practice strategy?&lt;/p&gt;&#xA;" OwnerUserId="21596" LastActivityDate="2019-12-12T02:02:10.360" Title="how to determine hashing bit length for multiple categorical features?" Tags="|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12867" PostTypeId="1" CreationDate="2016-07-19T09:36:23.997" Score="1" ViewCount="1636" Body="&lt;p&gt;This is more a conceptual question than related to the implementation on decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've a feature vector say &lt;code&gt;V1,V2,V3,target_variable&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the vector is &lt;code&gt;a,b,c,true&lt;/code&gt; then, using normal decision trees, we can classify the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if the variable V1 is a set say &lt;code&gt;{x,y,z},b,c,true&lt;/code&gt; how can I implement it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think of methods by using x,y,z as dimensions rather than categories but the problem is the number of categories are very huge in the order if millions. This solution won't scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any efficient ways to deal with this problem?&lt;/p&gt;&#xA;" OwnerUserId="18315" LastEditorUserId="21" LastEditDate="2016-07-19T21:32:13.877" LastActivityDate="2016-07-19T21:32:13.877" Title="Multiple categories within a variable in decision tree" Tags="|machine-learning|data-mining|bigdata|decision-trees|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12868" PostTypeId="2" ParentId="12865" CreationDate="2016-07-19T12:08:27.643" Score="1" Body="&lt;p&gt;This should have been a comment but I do not have the reputation.&lt;/p&gt;&#xA;&lt;p&gt;There are multiple ways:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Simplest Approach - Use textblob to find polarity and add the polarity of all sentences. If the overall polarity of tweet is greater than 0 , then it's positive and if less than zero , you can label it as negative&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use of lexicons-&#xA;One can use MQPA lexicon , to find the presence of negative and positive words and similarly , you can compute the overall polarity. &lt;a href=&quot;http://mpqa.cs.pitt.edu/lexicons/subj_lexicon/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MPQA lexicon&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15905" LastEditorUserId="29169" LastEditDate="2021-02-09T23:55:19.377" LastActivityDate="2021-02-09T23:55:19.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12869" PostTypeId="2" ParentId="12867" CreationDate="2016-07-19T15:13:56.063" Score="1" Body="&lt;p&gt;Your intuition that a categorical feature is treated as multiple dimensions (i.e., multiple features) is correct. Millions of categories will be problematic for tree-baed algorithms that search through all features/categories for each split. A random forest would be a good model to try for your project, since it samples the features. Most software implementations let you set the sample proportion, so you could set yours really low if need be to speed up the split searches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could feed your dataset into a Lasso or Elastic Net model to remove noisy variables and substantially reduce the quantity of categories under consideration. Lasso and Elastic Net have both been shown to do well on datasets where the number of features far exceeds the number of observations (e.g., dealing with genome data). &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-19T15:13:56.063" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12871" PostTypeId="1" CreationDate="2016-07-19T16:05:29.500" Score="0" ViewCount="1498" Body="&lt;p&gt;I am trying to build a multi-label classifier for suggesting tags on blog posts. The textual data is full of noise. The approach I have been following until now was a BOW approach with Tf-idf weighting. However I could only get an accuracy of around 0.35 on the test set using OnevsRest approach and a SVC. I then decided to eliminate the noise and I applied TextRank on my data to obtain a more meaningful summary and then applied the same BOW approach which however resulted in further loss of accuracy. I am quite new to machine learning and data science in general. Can someone please guide me with the approach? I am looking specifically on how to create my features and what else can I try.&#xA;I have 12 labels and around 1500 blog posts. I know it isn't much data, but the data-set is still well-balanced and won't be the primary reason for such low accuracy. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-11-22T17:14:05.017" Title="Multi-label Text Classification" Tags="|machine-learning|python|scikit-learn|multilabel-classification|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12872" PostTypeId="1" CreationDate="2016-07-19T21:54:34.713" Score="33" ViewCount="40279" Body="&lt;p&gt;What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Using the 840B word Common Crawl GloVe vectors with gensim:&#xA;&#xA;# 'hot' is closer to 'cold' than 'warm'&#xA;In [7]: model.similarity('hot', 'cold')&#xA;Out[7]: 0.59720456121072973&#xA;&#xA;In [8]: model.similarity('hot', 'warm')&#xA;Out[8]: 0.56784095376659627&#xA;&#xA;# Cold is much closer to 'hot' than 'popular'&#xA;In [9]: model.similarity('hot', 'popular')&#xA;Out[9]: 0.33708479049537632&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;NLTK's Wordnet methods appear to just give up:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [25]: print wn.synset('hot.a.01').path_similarity(wn.synset('warm.a.01'))&#xA;None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What are other options?&lt;/p&gt;&#xA;" OwnerUserId="6403" LastActivityDate="2020-08-06T13:36:56.023" Title="How can I get a measure of the semantic similarity of words?" Tags="|nlp|word-embeddings|word2vec|nltk|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12873" PostTypeId="2" ParentId="406" CreationDate="2016-07-20T00:00:55.760" Score="0" Body="&lt;p&gt;I would advice against using a neural network or equivalent as, I assume, you have got such a good prior based on your experience with the store (ie that there are probably day-to-day / seasonal trends and some level of smoothness) and I imagine a relatively small amount of data. A better option IMO would be to go for a kernel method such as a Gaussian Process or SVM. &lt;/p&gt;&#xA;" OwnerUserId="21619" LastActivityDate="2016-07-20T00:00:55.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12874" PostTypeId="1" CreationDate="2016-07-20T04:00:02.170" Score="4" ViewCount="970" Body="&lt;p&gt;I have been attempting to train a RNN on a set of time series data. The goal is to predict one of six categorical outputs. The input is given as 5 time steps of 14 inputs, six of which at one-hot attributes for the output. &#xA;There is an output at each time step, but the goal is to use previous recorded time spots and their human-assigned outputs to assign an output to the most recent event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confusingly the RNN is unable to learn that one of the inputs is in fact the output of the classification. This is simply a sanity check for me, but it seems that it may indicate a larger underlying problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is heavily imbalanced, 91%, 4%, 2%, 1%,&amp;lt;1%,&amp;lt;1%, but a cost function is being use to weight mis-classification inversely to it's make-up in the data set. Could the imbalance cause this issue? I'm using 60,000 training examples right now, is this not enough?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working off of this dynamic RNN model: &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py&quot; rel=&quot;nofollow&quot;&gt;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py&lt;/a&gt;.&#xA;Note the additional matrix multiplication for a hidden layer. Is this done correctly?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def dynamicRNN(x, seqlen, weights, biases):&#xA;&#xA;# Prepare data shape to match `rnn` function requirements&#xA;# Current data input shape: (batch_size, n_steps, n_input)&#xA;# Required shape: 'n_steps' tensors list of shape (batch_size, n_input)&#xA;&#xA;# Permuting batch_size and n_steps&#xA;x = tf.transpose(x, [1, 0, 2])&#xA;# Reshaping to (n_steps*batch_size, n_input)&#xA;x = tf.reshape(x, [-1,n_input])&#xA;x  = tf.matmul(x, weights['hidden'])+ biases['hidden']&#xA;# Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)&#xA;x = tf.split(0, n_steps, x)&#xA;&#xA;# Define a lstm cell with tensorflow&#xA;lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)&#xA;&#xA;# Get lstm cell output, providing 'sequence_length' will perform dynamic&#xA;# calculation.&#xA;outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32,&#xA;                            sequence_length=seqlen)&#xA;&#xA;# When performing dynamic calculation, we must retrieve the last&#xA;# dynamically computed output, i.e, if a sequence length is 10, we need&#xA;# to retrieve the 10th output.&#xA;# However TensorFlow doesn't support advanced indexing yet, so we build&#xA;# a custom op that for each sample in batch size, get its length and&#xA;# get the corresponding relevant output.&#xA;&#xA;# 'outputs' is a list of output at every timestep, we pack them in a Tensor&#xA;# and change back dimension to [batch_size, n_step, n_input]&#xA;outputs = tf.pack(outputs)&#xA;outputs = tf.transpose(outputs, [1, 0, 2])&#xA;&#xA;# Hack to build the indexing and retrieve the right output.&#xA;batch_size = tf.shape(outputs)[0]&#xA;# Start indices for each sample&#xA;index = tf.range(0, batch_size) * n_steps + (seqlen - 1)&#xA;# Indexing&#xA;outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)&#xA;&#xA;# Linear activation, using outputs computed above&#xA;return tf.matmul(outputs, weights['out']) + biases['out']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21464" LastEditorUserId="21464" LastEditDate="2016-07-20T16:48:34.173" LastActivityDate="2017-01-22T01:54:36.090" Title="Tensorflow RNN not learning when output included in training variables" Tags="|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12875" PostTypeId="1" CreationDate="2016-07-20T05:56:29.777" Score="1" ViewCount="96" Body="&lt;p&gt;Are there well-known automated methods for deriving a name for each topic obtained through topic modeling? for a specifically given problem at hand I will probably default to an algorithm on top an ontology, on top the topic modeling results. But I'm highly curious as to whether there's something that I am missing, in the general case.&lt;/p&gt;&#xA;" OwnerUserId="21625" LastActivityDate="2016-07-20T05:56:29.777" Title="automated topic modeling topic naming" Tags="|topic-model|" AnswerCount="0" CommentCount="3" ClosedDate="2016-07-20T08:02:54.053" ContentLicense="CC BY-SA 3.0" />
  <row Id="12876" PostTypeId="2" ParentId="12593" CreationDate="2016-07-20T08:14:33.747" Score="1" Body="&lt;p&gt;You'd have to think about where the outliers come from. Technical malfunction? Then they are probably safe to disregard. But when you write that your main task is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;to draw conclusions from the training data that can be generalized to future unseen data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;you have to keep in mind that your future data will contain outliers &lt;em&gt;as well&lt;/em&gt; and deal with them appropriately. Models like trees are usually somewhat robust against those while regression can go haywire. There are several well working methods like disregarding everything that is more than 4 standard deviations (or median deviations from the median) out or just disregarding the n-th top and bottom percentiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then there is the case where outliers are &lt;strong&gt;not&lt;/strong&gt; just flukes and can actually tell you something about the problem domain you are working on. If you work in fraud detection, the outliers are the points that interest you. If you work on optimizing network performance, the packets that don't arrive can be the interesting ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my advice is to not blindly throw away data out of principle but always think about what the data represent and why outliers may be present.&lt;/p&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-20T08:14:33.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12878" PostTypeId="1" AcceptedAnswerId="13117" CreationDate="2016-07-20T11:31:27.710" Score="1" ViewCount="1866" Body="&lt;p&gt;I am reading the original paper by Chawla and others for SMOTE. I am trying to understand how to generate these synthetic examples for over-sampling the minority class. The paper says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbor. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the idea, take your sample, the nearest neighbor, pick a random point in between, what I don't understand is how these nearest neighbors are defined.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="25849" LastEditDate="2020-10-02T18:45:41.490" LastActivityDate="2020-10-02T18:45:41.490" Title="How to find nearest neighbors in SMOTE" Tags="|dataset|class-imbalance|smote|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12879" PostTypeId="2" ParentId="12593" CreationDate="2016-07-20T11:44:18.407" Score="1" Body="&lt;p&gt;As far as practical applications are concerned :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a few:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Aircraft Engine Anomaly Detection&lt;/strong&gt;: Input Features can be heat generated by engine, vibrational intensity, fuel consumed etc etc. Here outliers can be sent for testing again and further decisions can be made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Fraud Detection&lt;/strong&gt; : Features can be features of users activities on a website. We can model probabilities from the data. Identify unusual behavior by checking probability less than certain fixed threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;strong&gt;Monitoring Computers in a data center&lt;/strong&gt; : features can be memory use, no of disk accesses, CPU load, network traffic etc. Abnormal behavior here can help predict future breakdowns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Anomaly Detection&lt;/strong&gt; is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it &lt;strong&gt;gaussian&lt;/strong&gt; using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example : replace original feature x -&gt; Log(x) or feature x -&gt; (x)^4/3 etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.&lt;strong&gt;F1-Score&lt;/strong&gt; is a pretty good metric to evaluate the performance of the algorithm.&lt;/p&gt;&#xA;" OwnerUserId="21414" LastEditorUserId="21414" LastEditDate="2016-07-20T11:51:54.210" LastActivityDate="2016-07-20T11:51:54.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12880" PostTypeId="1" AcceptedAnswerId="13678" CreationDate="2016-07-20T12:38:29.057" Score="1" ViewCount="363" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;In the following text, uncertainty refers to standard deviation.&lt;/em&gt;&#xA;  My methodology and notation was based mainly on the book &lt;a href=&quot;http://www.cambridge.org/us/academic/subjects/engineering/engineering-general-interest/students-guide-data-and-error-analysis?format=PB&quot; rel=&quot;nofollow&quot;&gt;&quot;A Student's Guide to Data and Error Analysis&quot;&lt;/a&gt; by H.J.C. Berendsen.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Summary:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a timeseries which I use in a few equations and get averages and uncertainties. I do that up to a point where I have to integrate the averages. So the question is, &lt;strong&gt;what happens to uncertainty when you integrate the averages?&lt;/strong&gt; My problem might arise from a mistake in methodology so &lt;strong&gt;I would appreciate any feedback on the followed process as well&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Details:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate the uncertainty of a number that is computed as part of an analysis code. While I can compute the average number, I encounter some issues with the uncertainty for which I would appreciate any advice from more experienced analysts.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My methodology was:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;From 30 simulations, I have obtained 30 timeseries files that each contain 100000 lines of a z-axis position-fluctuation of a molecule. &lt;strong&gt;Therefore I have 30 files files that each contains two vectors: t and z&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;For each position/file&lt;/strong&gt;, I use the z timeseries to calculate &#xA;the variance of the timeseries $var(z)$ and the variance of the mean of the timeseries $var(\langle z \rangle)$. &lt;em&gt;The latter is done with block averaging (b.a.)&lt;/em&gt;. Both these methods have a textbook defined, relative standard deviation:&#xA;$$ var(z) \, [units] \pm \sqrt{\frac{2}{n-1}} $$&#xA;$$ var(\langle z \rangle) \, [units] \pm \sigma_{&amp;lt;χ&amp;gt;}^2 $$ &#xA;where $ \sigma_{&amp;lt;χ&amp;gt;}^2 $ is the squared standard error ($ \sigma_{&amp;lt;χ&amp;gt;}=\frac{\sigma}{\sqrt{n}}$) computed from b.a.&#xA; The above are used as input in a diffusion equation:&#xA;$$ D(z)=\frac{var(z)}{\big[ \frac{n \cdot var(\langle z \rangle) }{var(z)} -1 \big] \frac{dt}{2}} $$ &#xA;where $n$ and $dt$ are constants. &lt;em&gt;For this step, I also used two failed alternative methods that I explain in the end.&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I used the above to calculate the average diffusion coefficient D for each of the 30 files. To find the uncertainty of each average, I combined the errors based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Propagation_of_uncertainty&quot; rel=&quot;nofollow&quot;&gt;error propagation algebra&lt;/a&gt; where I considered (as a first approximation) the covariance $\sigma_{AB}$ as 0. In the end, I obtained a new &quot;summary&quot; file with 30 rows and 4 columns, containing the average position, average Diffusion, absolute uncertainty and relative uncertainty:&#xA;$$  z  \,,  D(z)  \,, \pm \Delta D(z) \,, \pm \frac{\Delta D(z)}{ D(z) }$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From the same simulation, I also obtain a similar &quot;summary&quot; file that contains 30 rows and 3 columns with the average position, Gibbs free-energy and its absolute uncertainty (standard deviation):&#xA;$$  z \,,  \Delta G(z)  \,, \pm \Delta \Delta G(z) $$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I then combined both summary files to compute a resistance profile with the equation:&#xA;$$ R(z) = \frac{\exp \big[ {\frac{\Delta G(z)}{k_BT}}\big] }{D(z)} $$&#xA;where $k_BT$ was a constant. To combine $\Delta G(z)$ and $D(z)$ I first  I interpolated them in a common, dense z-grid. Therefore, I ended up with a file with 500 values of resistance together with the absolute and relative uncertainty:&#xA;$$  z \,,  R( z )  \,, \pm \Delta R(z) \,, \pm \frac{\Delta R(z)}{ R(z) }$$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To get the final result which is permeation, I integrated these 500 values, according to the equation:&#xA;$$ P=\frac{1}{\int_{z_{min}}^{z_{max}}R(z)dz} $$&#xA;and I obtained the average P by performing a trapezoidal approximation over the 500 $R(z)$ values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is, what happens to the uncertainty? How does it propagate through the integral?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Also, do you have a more appropriate alternative to my methodology?&lt;/strong&gt; I am trying to find some guidelines on how to do a &quot;correct&quot; data analysis but the available information is either too theoretical or too vague. &lt;strong&gt;For example, would it make any difference in this case to perform fitting on the $D(z)$, $\Delta G(z)$ and $R(z)$ profiles, before integration? If yes, how would I calculate the uncertainty for each and for the $P$?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for any help or advice! :D&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In step 2, I tried 2 more things:&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;&lt;p&gt;instead of calculating the uncertainties of $var(z)$ and $var(\langle z \rangle)$ from the textbook, I used bootstrapping to&#xA;  create 100 new timeseries of z, for each position/file. And&#xA;  then calculated 100 different diffusions which I used to calculate the&#xA;  average and s.d. The method was returning the same statistics for the&#xA;  timeseries (which was expected) but the final Diffusion was extremely&#xA;  oscillatory (was even negative), therefore I dropped the method.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;I tried to use block averaging for the calculation of D, but Python was refusing to do recursive block averaging (I might had a bug in my&#xA;  code as well). Anyway, I quickly abandoned the idea as well.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="21627" LastActivityDate="2016-09-28T17:27:21.683" Title="Uncertainty calculation through integration and correct analysis methodology" Tags="|dataset|statistics|time-series|error-handling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12881" PostTypeId="2" ParentId="12878" CreationDate="2016-07-20T14:44:37.480" Score="0" Body="&lt;p&gt;The nearest neighbor will be the sample with the smaller Euclidean distance.&lt;/p&gt;&#xA;" OwnerUserId="20912" LastActivityDate="2016-07-20T14:44:37.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12883" PostTypeId="1" CreationDate="2016-07-20T15:51:05.770" Score="4" ViewCount="9524" Body="&lt;p&gt;I'm a software engineer new to Machine Learning. I've read about basic non-supervised techniques like k-means and hierarchical clustering and now I'm trying to put them into practice with a basic problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I've got a lot of rows of data with each row looking something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;employeesRange&quot;: &quot;11-50&quot;, &#xA;&quot;category&quot;: &quot;Lobbying&quot;, &#xA;&quot;categoryGroup&quot;: &quot;Polication Action&quot;, &#xA;&quot;sector&quot;: &quot;Industrials&quot;, &#xA;&quot;subCategory&quot;: &quot;Direct&quot;, &#xA;&quot;tags&quot;: [ &quot;Progressive&quot;, &quot;Libertarian&quot;, &quot;PAC&quot; ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to analyze this data and look for patterns, i.e. clusters of information that frequently appear together. For example maybe it's a common pattern to find smaller groups of lobbyists 11-50 people for &quot;Progressive&quot;, &quot;Libertarian&quot; causes but &quot;Presidential&quot; or &quot;Local Politics&quot; groups tend to skew larger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe there's another link to be found between categories and sector, that kind of thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems a little harder then the examples I've read about because some of my data is a tag cloud (the &quot;Tags&quot; field) which is unstructured can contain multiple entries and the more structured fields of categories and employees range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems a good place to start is using hierarchical clustering (since I don't know k a-priori) maybe following the recommendations here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/23943391/how-to-cluster-users-based-on-tags&quot;&gt;https://stackoverflow.com/questions/23943391/how-to-cluster-users-based-on-tags&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few questions about this. First, is this a good approach? And is Mahout the best tool for this? Any obvious ways to simplify? And finally, how can I combine the tag clustering approach with the other more structured data in the rows?&lt;/p&gt;&#xA;" OwnerUserId="21638" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-11-09T22:37:35.533" Title="Best approach for this unsupervised clustering problem with categorical data?" Tags="|machine-learning|clustering|categorical-data|unsupervised-learning|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12884" PostTypeId="2" ParentId="12871" CreationDate="2016-07-20T17:24:05.033" Score="1" Body="&lt;p&gt;What you seem to want is to identify meaningful discrepancies between your blog posts in order to identify what category they fit in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's important for you to consider the classification problem in terms of human thinking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How would &lt;em&gt;humans&lt;/em&gt; decide which blog gets which tags? &lt;/li&gt;&#xA;&lt;li&gt;Can we distil this decision-making into meaningful features? In other words, does this decision-making depend on (linguistic) phenomena that can be found in the text? (it probably does)&lt;/li&gt;&#xA;&lt;li&gt;(How) can we extract these features from the text? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also, it might be the case that either the tag set you propose or the data set you have are too similar. This means that the tags might be too similar and that their occurrence depends on many similar feature values which makes it difficult for your training algorithm to extract discriminative features. The same thing happens when the data expresses too little variety.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a final note: This might well be the first attempt to solving this particular problem (on this data set and this set of labels), so the number you get is not necessarily &lt;em&gt;bad&lt;/em&gt; or &lt;em&gt;sub-par&lt;/em&gt;, because it might simply be a difficult task.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2016-07-20T17:24:05.033" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12885" PostTypeId="1" CreationDate="2016-07-20T17:45:08.400" Score="1" ViewCount="196" Body="&lt;p&gt;I'm looking at identifying trends within my data, particularly wondering if usage of my app on new sign ups is increasing over a week to week basis.  As we are constantly improving our product, I'd love to be able to identify a correlation between app usage &amp;amp; new feature releases.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have is split into weekly cohorts.  Each week I have the number of new sign ups, and how many of those accounts are still active (e.g. users are logging into the system).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have the following data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 1 - 10 new trials, 3 accounts active&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Week 2 - 15 new trials, 5 accounts active&lt;/p&gt;&#xA;&#xA;&lt;p&gt;week 3 - 4 new trials, 3 accounts active&lt;/p&gt;&#xA;&#xA;&lt;p&gt;week 4 - 20 new trials, 12 accounts active&lt;/p&gt;&#xA;&#xA;&lt;p&gt;week 5 - 17 new trials, 9 accounts active&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my current approach of analysis, week 3 looks amazing because 75% of accounts are still active ... In reality though, the number of new trials is extremely small compared to other weeks.  As a result, I don't feel like I'm accurately comparing apples to apples in a week to week comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way I can normalize the weekly data so that I am performing accurate analysis - or is percentage based really the best way to look at this data?      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty new to this, so any help is much appreciated.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="21642" LastActivityDate="2018-07-12T09:51:39.453" Title="normalize usage data" Tags="|correlation|normalization|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12886" PostTypeId="1" CreationDate="2016-07-20T18:39:53.130" Score="13" ViewCount="14378" Body="&lt;p&gt;I have a classification problem with highly imbalanced data. I have read that over and undersampling as well as changing the cost for underrepresented categorical outputs will lead to better fitting. Before this was done tensorflow would categorize each input as the majority group (and gain over 90% accuracy, as meaningless as that is). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have noticed that the log of the inverse percentage of each group has made the best multiplier that I have tried. Is there a more standard manipulation for the cost function? Is this implemented correctly? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import Counter&#xA;counts = Counter(category_train)&#xA;weightsArray =[]&#xA;for i in range(n_classes):&#xA;    weightsArray.append(math.log(category_train.shape[0]/max(counts[i],1))+1)&#xA;&#xA;class_weight = tf.constant(weightsArray)&#xA;weighted_logits = tf.mul(pred, class_weight)&#xA;cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(weighted_logits, y))&#xA;optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21464" LastActivityDate="2018-10-23T06:55:13.770" Title="Tensorflow Adjusting Cost Function for Imbalanced Data" Tags="|tensorflow|loss-function|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12889" PostTypeId="2" ParentId="11609" CreationDate="2016-07-20T20:12:58.010" Score="0" Body="&lt;p&gt;You need to ensure the package &lt;a href=&quot;https://github.com/databricks/spark-csv&quot; rel=&quot;nofollow&quot;&gt;spark-csv&lt;/a&gt; is loaded; e.g., by invoking the &lt;code&gt;spark-shell&lt;/code&gt; with the flag &lt;code&gt;--packages com.databricks:spark-csv_2.11:1.4.0&lt;/code&gt;. After that you can use &lt;code&gt;sc.textFile&lt;/code&gt; as you did, or &lt;code&gt;sqlContext.read.format(&quot;csv&quot;).load&lt;/code&gt;. You might need to use &lt;code&gt;csv.gz&lt;/code&gt; instead of just &lt;code&gt;zip&lt;/code&gt;; I don't know, I haven't tried.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-20T20:12:58.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12890" PostTypeId="2" ParentId="11338" CreationDate="2016-07-20T20:18:40.443" Score="0" Body="&lt;p&gt;There are other recent methods like Principal Difference Analyses specifically designed to address these sort of problems. Am not sure if the methd is available as an R package, you can get the concept/algorithm from the manuscript.  See &lt;a href=&quot;http://arxiv.org/abs/1510.08956&quot; rel=&quot;nofollow&quot;&gt;http://arxiv.org/abs/1510.08956&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21636" LastActivityDate="2016-07-20T20:18:40.443" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12891" PostTypeId="2" ParentId="12871" CreationDate="2016-07-20T20:23:39.910" Score="0" Body="&lt;p&gt;BOW approach with Tf-idf weighting seems like a good strategy however you need to improve your feature engineering strategy. I would recommend to include n-grams (bigrams, trigrams etc.), special charecters (exclamations), emoji unicodes etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.35 is a poor estimate than of a random accuracy (&amp;lt;=0.5). I think your model is definitely overfitting. I think your dataset may have &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1007741&quot; rel=&quot;nofollow&quot;&gt;class imbalance&lt;/a&gt; and may need to account for that. &lt;/p&gt;&#xA;" OwnerUserId="21636" LastActivityDate="2016-07-20T20:23:39.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12892" PostTypeId="2" ParentId="12883" CreationDate="2016-07-20T20:25:54.517" Score="-1" Body="&lt;p&gt;K-means is neat strategy for these sort of problems. However you could also explore recent methods including &lt;a href=&quot;https://cran.r-project.org/web/packages/TDA/vignettes/article.pdf&quot; rel=&quot;nofollow&quot;&gt;topological data analyses&lt;/a&gt; to see clusters of features/individuals. &lt;/p&gt;&#xA;" OwnerUserId="21636" LastActivityDate="2016-07-20T20:25:54.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12895" PostTypeId="1" CreationDate="2016-07-20T21:08:35.813" Score="3" ViewCount="1636" Body="&lt;p&gt;I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What data preprocessing steps should I perform. The dataset is&#xA;highly imbalanced (500 failures and ~40000 non-failures) &lt;/li&gt;&#xA;&lt;li&gt;What type of Machine Learning models should I take into consideration as data is highly imbalanced? &lt;/li&gt;&#xA;&lt;li&gt;Few days back, I read about Survival Analysis&#xA;and now I am in conundrum whether the problem would be of Survival&#xA;Analysis or Machine Learning?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am currently working with dataset provided by BackBlaze(&lt;a href=&quot;https://www.backblaze.com/b2/hard-drive-test-data.html&quot; rel=&quot;nofollow&quot;&gt;https://www.backblaze.com/b2/hard-drive-test-data.html&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be really great it I could get some direction :)&lt;/p&gt;&#xA;" OwnerUserId="20722" LastActivityDate="2016-08-23T05:41:59.467" Title="Machine Learning or Survival Analysis?" Tags="|machine-learning|classification|data-cleaning|preprocessing|survival-analysis|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12896" PostTypeId="2" ParentId="12743" CreationDate="2016-07-20T23:10:40.050" Score="0" Body="&lt;p&gt;I guess what you want to calculate is called mutual information. &quot;It measures the average reduction in uncertainty about x that results from learning the value of y; or vice versa, the average amount of information that x conveys about y&quot;, y is the person type and x is the vector of condition counts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.inference.phy.cam.ac.uk/mackay/itila/book.html&quot; rel=&quot;nofollow&quot;&gt;http://www.inference.phy.cam.ac.uk/mackay/itila/book.html&lt;/a&gt; chapter 8.1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the R package description  &lt;a href=&quot;https://cran.r-project.org/web/packages/entropy/entropy.pdf&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/entropy/entropy.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-07-20T23:10:40.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12897" PostTypeId="1" AcceptedAnswerId="12915" CreationDate="2016-07-21T04:04:10.140" Score="2" ViewCount="489" Body="&lt;p&gt;I am trying to build a recommendation system. My system is basically a ecommerce application where our customers answers a bunch of questions related to healthcare (their basic health related question). Based on their ansers, we recommend some product. This process of recommendationis based of conventional rule base aapproach. Think of it as bunch of if-else condition. Now I am playing around with some machine learning technique nd want to see if this way will add any value in our healthcare system. I am at the very starting point and can use any suggestion from you guys. The suggestion could aim towards following: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Any product that you feel which leverages ML technique with respect to health(&lt;strong&gt;considering HIPPA constraints&lt;/strong&gt;) &lt;/li&gt;&#xA;&lt;li&gt;Any product that you feel which leverages ML technique with respect to health &lt;/li&gt;&#xA;&lt;li&gt;What could be the first step towards building such a system. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21657" LastActivityDate="2016-07-21T17:26:43.127" Title="recommendation system for eCommerce healthcare portal suggestion" Tags="|machine-learning|data-mining|predictive-modeling|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12898" PostTypeId="1" CreationDate="2016-07-21T05:42:08.117" Score="2" ViewCount="228" Body="&lt;p&gt;I have been playing around the algorithm with tensorflow in &lt;a href=&quot;http://arxiv.org/abs/1508.06576&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&#xA;I tried to convert a photo to a Chinese ink and wash painring, but I got some strange patterns in the output picture(those in the top left corner).&#xA;&lt;a href=&quot;https://i.stack.imgur.com/f85MU.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f85MU.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&#xA;I totally followed the algorithm in the paper except that I add another loss term for total variation denoising.&#xA;I have some confusions about this:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In the paper it used the 16-layer vgg net. I am going to try some&#xA;other networks, such as googlenet or alexnet. But I was wondering&#xA;whether it would avoid these patterns if I use some other networks.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the paper, it used the gram matrix to represent the style&#xA;loss. But I don't know the reason here. I was also wondering if&#xA;there's any other way to stand for the style loss.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since I have used total variation denoising here, and the&#xA;result is not satisfactory. I was wondering if there're some other&#xA;ways of denoising to avoid these patterns.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;(I'm sorry that I can't&#xA;upload my content input image because I don't have enough reputation&#xA;in stack overflow as a new user. If you're interested, you can&#xA;search the google image with&amp;quot;Shanghai Jiaotong university&amp;quot; and find&#xA;the original image there.)&lt;/p&gt;&#xA;" OwnerUserId="21658" LastEditorUserId="98307" LastEditDate="2020-08-12T14:32:41.360" LastActivityDate="2020-08-12T14:32:41.360" Title="How to improve the neural art algorithm?" Tags="|machine-learning|neural-network|computer-vision|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12899" PostTypeId="2" ParentId="12752" CreationDate="2016-07-21T06:38:55.820" Score="1" Body="&lt;p&gt;A &lt;a href=&quot;https://github.com/strin/BayesPA&quot; rel=&quot;nofollow noreferrer&quot;&gt;python package&lt;/a&gt; that you can check. It's an online max-margin topic model.&lt;/p&gt;&#xA;" OwnerUserId="21662" LastEditorUserId="98307" LastEditDate="2020-08-07T08:41:49.977" LastActivityDate="2020-08-07T08:41:49.977" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12900" PostTypeId="1" AcceptedAnswerId="12903" CreationDate="2016-07-21T06:41:18.743" Score="7" ViewCount="468" Body="&lt;p&gt;I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, and from the interviews I had so far, I noticed that they don't fully understand what they want from a data scientist. They barely know what data science is. I explained them the &quot;standard&quot; data science workflow (Ask A Question, Get the Data, Explore the Data, Model the data and Communicate the data). But I don't think they got it. I want to set the proper expectations, so the company and I can agree on my job description, and what they can expect from me in one month, tree months, six and twelve months. So, how do you handle the expectations(specially the CEO) from a company that's new to data science?&lt;/p&gt;&#xA;" OwnerUserId="21659" LastEditorUserId="21669" LastEditDate="2016-07-21T23:14:12.063" LastActivityDate="2016-10-26T22:52:35.457" Title="How to handle the CEO expectations from a company that's new to data science?" Tags="|data|beginner|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12901" PostTypeId="2" ParentId="12900" CreationDate="2016-07-21T06:51:30.277" Score="1" Body="&lt;p&gt;Good question! I've faced this problem before, where the CEO had unrealistic expectations. Unfortunately I can't tell you how we worked it out, because it didn't work out. The good news is that you're not working for them yet, so you can just walk away. At the end of the day, it comes down to trust, and they have to convince you they trust you. Talk to the other employees to find out they are treated; that might tell you something.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-21T06:51:30.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12903" PostTypeId="2" ParentId="12900" CreationDate="2016-07-21T08:23:33.817" Score="3" Body="&lt;p&gt;I tell you what has worked for me: practical examples. They have probably already read about what data science is in general and which are the standard procedures. What they have not seen is someone in front of them explaining how innovative (and useful for business!!!) the data science really is. Follow the &quot;Say what you would say to your granny to tell her how cool your work is&quot; advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People want to know what you have actually done, indipendently from the algorithms or procedures you have used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="10024" LastActivityDate="2016-07-21T08:23:33.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12904" PostTypeId="2" ParentId="12872" CreationDate="2016-07-21T09:17:58.473" Score="10" Body="&lt;p&gt;In &lt;a href=&quot;http://www.onlinejournal.in/IJIRV2I5/268.pdf&quot; rel=&quot;noreferrer&quot;&gt;Text Analytic Tools for Semantic Similarity&lt;/a&gt;, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a matrix and sum together to find out the similarity between sentences. So, it might be a shot to check word similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in &lt;a href=&quot;http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00237&quot; rel=&quot;noreferrer&quot;&gt;SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation&lt;/a&gt;, they explain the difference between &lt;code&gt;association&lt;/code&gt; and &lt;code&gt;similarity&lt;/code&gt; which is probably the reason for your observation as well. For example, coffee and a cup. They are not similar but they are associative. So just considering similarity would give a different result. The authors suggest various models to estimate them.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-08-09T05:56:44.733" LastActivityDate="2016-08-09T05:56:44.733" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12905" PostTypeId="1" CreationDate="2016-07-21T09:48:54.850" Score="7" ViewCount="4127" Body="&lt;p&gt;I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any ways in the literature for doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already found some algorithms being able to extract a DFA from a recurrent neural network, but nothing about DFAs either as input or output of ANN.&lt;/p&gt;&#xA;" OwnerUserId="14955" LastEditorUserId="26562" LastEditDate="2017-11-06T10:38:48.623" LastActivityDate="2022-11-30T21:01:06.527" Title="Can finite state machines be encoded as input/output for a neural network?" Tags="|neural-network|classification|language-model|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="12906" PostTypeId="1" CreationDate="2016-07-21T11:34:30.967" Score="6" ViewCount="290" Body="&lt;p&gt;I've been trying to implement a simple convolutional neural network. But I've been stuck at this problem for over a week. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be specific, assume there are 3 layers in a convolutional pass, marked as &lt;code&gt;l-1, l, l+1 -layer&lt;/code&gt; respectively. And the &lt;code&gt;l-1&lt;/code&gt; layer is the input layer with shape &lt;code&gt;(num_channels, img_height, img_width)&lt;/code&gt;,the &lt;code&gt;l&lt;/code&gt; layer is the convolutional layer with kernel shape &lt;code&gt;(num_kernels, num_channels, filter_height, filter_width)&lt;/code&gt;, and the &lt;code&gt;l+1&lt;/code&gt; layer is the pooling layer with pooling size &lt;code&gt;(pool_height, pool_width)&lt;/code&gt;.  Note that in my simple implementation, the filters are 3-dimensional, and they have the same depth as with the input, i.e., they have the same number of channels as that of the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On this setting, I encountered a problem in the back-propagation phase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've studied many tutorials in neural networks, however, it seems that all of them stop with back-propagating the error to &lt;code&gt;l&lt;/code&gt; layer with the formular:&#xA;\begin{align}&#xA;   \delta_k^{(l)} = \text{upsample}\left((W_k^{(l)})^T \delta_k^{(l+1)}\right) \bullet f'(z_k^{(l)})&#xA;   \end{align}&#xA;The question is, how should I back-propagate error from the &lt;code&gt;l&lt;/code&gt; layer to &lt;code&gt;l-1&lt;/code&gt; layer, that is, calculate $\delta_k^{(l-1)}$, in order to chain several conv-pooling pass together?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In my implementation, there is a class called &lt;code&gt;ConvPoolLayer&lt;/code&gt; which has a &lt;code&gt;feedforward&lt;/code&gt; method and a &lt;code&gt;backprop&lt;/code&gt; method. The method &lt;code&gt;backprop&lt;/code&gt; needs to propagate the error from output to input. When I chain instants of &lt;code&gt;class ConvPoolLayer&lt;/code&gt; together, I just need train their &lt;code&gt;feedforward&lt;/code&gt; methods and &lt;code&gt;backprop&lt;/code&gt; methods together respectively. &lt;/p&gt;&#xA;" OwnerUserId="21676" LastEditorUserId="29169" LastEditDate="2021-02-16T15:40:33.820" LastActivityDate="2021-02-16T15:40:33.820" Title="how to propagate error from convolutional layer to previous layer？" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|backpropagation|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12907" PostTypeId="2" ParentId="12885" CreationDate="2016-07-21T12:51:39.930" Score="1" Body="&lt;p&gt;With this low number of sign ups I doubt you can profit from advanced statistics at this stage. Just chart your numbers in excel along the timeline and judge visually. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-07-21T12:51:39.930" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12909" PostTypeId="1" CreationDate="2016-07-21T14:12:39.533" Score="12" ViewCount="28371" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; definition does not quite apply since we are not always assuming an underlying distribution. So what is a model really? Can a Gradient Boosted Model (GBM) with specified hyperparameters be considered a model? Is a model a collection of rules?&lt;/p&gt;&#xA;" OwnerUserId="236" LastEditorUserId="29169" LastEditDate="2021-01-24T00:46:52.120" LastActivityDate="2021-01-24T01:38:32.657" Title="Definition of a model in machine learning" Tags="|machine-learning|" AnswerCount="8" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12910" PostTypeId="1" CreationDate="2016-07-21T14:46:31.317" Score="0" ViewCount="75" Body="&lt;p&gt;I am using &lt;a href=&quot;http://nlp.stanford.edu/software/classifier.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Standford Classifier&lt;/a&gt; for my project.&#xA;This project takes training data to tune the algorithm then test data to classify text inputs into categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the format for test and training data is tab-delimited text which means predictor &lt;code&gt;-TAB- input text&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The software prints out the output to &lt;code&gt;stdout&lt;/code&gt; (command line).&#xA;Is there anyway to output to a text file?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I searched the &lt;a href=&quot;http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/classify/ColumnDataClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;javadoc&lt;/a&gt; of the project site, and I found:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t2Ao9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t2Ao9.png&quot; alt=&quot;this one&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't know how to use this property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried &lt;code&gt;-csvoutput=%1%n%c&lt;/code&gt; on command line&#xA;But it gives me java null pointer exception error when I try to run it.&#xA;Any ideas?&#xA;Thank you so much.&lt;/p&gt;&#xA;" OwnerUserId="21687" LastEditorUserId="15620" LastEditDate="2017-04-20T09:20:49.427" LastActivityDate="2019-04-11T09:02:15.723" Title="How to print out to a file using Standford Classifier" Tags="|classification|nlp|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12911" PostTypeId="2" ParentId="12909" CreationDate="2016-07-21T15:46:12.127" Score="10" Body="&lt;p&gt;I was interested in the same question recently and came to the realization that there is no single definition of a &quot;model&quot; in machine learning. It's highly dependent on the sources you're consulting, which may be the documentation for a particular software program, the slang adopted by its user community, or the definitions used in published academic papers, which can vary widely from journal to journal. Moreover, I had to learn to keep in mind that such papers are written not just by specialists in machine learning, but by experts in other disciplines who have a need for applying machine learning techniques (such as imaging, various medical fields, etc.). Many of them do not explicitly define the term &quot;model,&quot; which is often used loosely. Here are just a couple of different definition of &quot;model&quot; I've seen in all this widely scattered literature:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Statistical models, particularly the stats related to probability distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Regression data and related statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Mathematical models as mentioned by Neil Slater above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• The data models used in machine learning, such as the columns involved, their data types, the data sources and other metadata. This is particularly tricky because there's nothing mathematical about this definition at all, unlike the first three I listed. For an example, see all the documentation for SQL Server &quot;mining models,&quot; which serve double-duty for machine learning purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Sometimes all of the definitions above are expanded to include machine learning structures built on top of the equations and the metadata, such as the specifications of neural nets. In other cases, these are considered separate entities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the above are sometimes mixed and matched together, depending on the source. I'm sure there are other definitions of &quot;model&quot; I've left off this list, which will complicate the matter even further. To deal with this ambiguity, I'm trying to train myself to divine the intentions of the author whenever they use the term &quot;model.&quot; Sometimes it's easy to determine based on the context or the field the author works in, but other times I have to read deeply into an article or documentation before figuring it out. I wish I could be more definitive about it, but it's really a naturally fuzzy term; there's never going to be a simple one-size-fits-all answer to this. I hope that helps. &lt;/p&gt;&#xA;" OwnerUserId="18544" LastActivityDate="2016-07-21T15:46:12.127" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12915" PostTypeId="2" ParentId="12897" CreationDate="2016-07-21T17:26:43.127" Score="0" Body="&lt;p&gt;Your recommendation system will be designed to tell the customer what product they should choose, however, this doesn't account for what products the customer likes.  A ML method could take all the input parameters from the recommendation system and provide a recommended product based on what similar users liked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's no specific ML technique for considering HIPPA constraints.  This would come into play more in the pre-processing stage.  For example, you might not be able to acquire specific birth dates and addresses, but maybe you can get an age range and a zip code (first 3 digits only).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is frequently used in healthcare for binary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are lots of tutorials online for building ML models.  Kaggle has a good tutorial for Python and R using titanic survival data.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-07-21T17:26:43.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12916" PostTypeId="2" ParentId="12851" CreationDate="2016-07-21T17:32:24.190" Score="10" Body="&lt;p&gt;In R, &lt;code&gt;nnet&lt;/code&gt; does not come with a plot function, but code for that is provided &lt;a href=&quot;https://beckmw.wordpress.com/2013/03/04/visualizing-neural-networks-from-the-nnet-package/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use the more recent and IMHO better package called &lt;code&gt;neuralnet&lt;/code&gt; which features a &lt;code&gt;plot.neuralnet&lt;/code&gt; function, so you can just do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(infert, package=&quot;datasets&quot;)&#xA;plot(neuralnet(case~parity+induced+spontaneous, infert))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yyftd.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yyftd.png&quot; alt=&quot;neuralnet&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;neuralnet&lt;/code&gt; is not used as much as &lt;code&gt;nnet&lt;/code&gt; because &lt;code&gt;nnet&lt;/code&gt; is much older and is shipped with r-cran. But &lt;code&gt;neuralnet&lt;/code&gt; has more training algorithms, including &lt;a href=&quot;https://en.wikipedia.org/wiki/Rprop&quot; rel=&quot;noreferrer&quot;&gt;resilient backpropagation&lt;/a&gt; which is lacking even in packages like Tensorflow, and is much more robust to hyperparameter choices, and has more features overall.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-07-21T17:32:24.190" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12918" PostTypeId="1" AcceptedAnswerId="13134" CreationDate="2016-07-21T18:34:09.517" Score="0" ViewCount="2795" Body="&lt;p&gt;I am unable to change number of bins when using geom_histogram() function. There is nothing in the documention, except some examples where binwidth=X is used (which is how R:ggplot2 uses it). However, this parameter seems to have no effect on the actual output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Version: ggplot 0.10.4, Python 3.5 64bit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;plt.figure()&#xA;dataframe['column'].plot.hist(bins=30)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;works as expected and draws 30 bins. Meanwhile: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from ggplot import *&#xA;gg = ggplot(dataframe, aes(x='column')) + geom_histogram(binwidth=300)&#xA;print(gg)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;does not change number of bins (I understand that binwidth sets interval for one bin), and have tried multiple values and/or datasets&lt;/p&gt;&#xA;" OwnerUserId="21692" LastActivityDate="2016-08-01T18:47:12.830" Title="Python: ggplot geom_histogram() binwidth or bin parameter missing?" Tags="|python|visualization|ggplot2|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12919" PostTypeId="1" AcceptedAnswerId="12921" CreationDate="2016-07-21T18:40:47.767" Score="11" ViewCount="20388" Body="&lt;p&gt;I am new to &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt; and I need to understand the capabilities and shortcomings of TensorFlow before I can use it. I know that it is a deep learning framework, but apart from that which other machine learning algorithms can we use with tensor flow. For example can we use SVMs or random forests using TensorFlow? (I know this sounds crazy)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I want to know which Machine Learning Algorithms are supported by TensorFlow. Is it just deep learning or something more?&lt;/p&gt;&#xA;" OwnerUserId="21691" LastEditorUserId="29575" LastEditDate="2018-02-07T19:06:16.250" LastActivityDate="2018-04-06T00:21:14.927" Title="Is TensorFlow a complete Machine Learning Library?" Tags="|machine-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12920" PostTypeId="2" ParentId="12919" CreationDate="2016-07-21T19:13:58.827" Score="7" Body="&lt;p&gt;TensorFlow is especially indicated for deep learning, i.e. neural networks with lots of layers and weird topologies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's it. It is an alternative to &lt;a href=&quot;http://deeplearning.net/software/theano/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano&lt;/a&gt;, but developed by Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both TensorFlow and Theano, you program symbolically. You define your neural network in the form of algeabreic operations (these nodes are multiplied by these weights and then a non-linear transformation is applied, bla bla bla), which internally are represented by a graph (which in the case of TensorFlow, but not Theano, you can actually see in order to debug your neural network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, TensorFlow (or Theano) offer optimization algorithms which do the heavy-work of figuring out what weights minimize whatever cost function you want to minimize. If your neural network is meant to solve a regression problem, you might want to minimize the sum of squared differences between the predicted values and the true values. TensorFlow does the heavy work of differentiating your cost function and all that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Forgot to mention that, of course, SVMs can be seen as a &lt;a href=&quot;https://stackoverflow.com/questions/8963937/svm-and-neural-network&quot;&gt;type of neural network&lt;/a&gt;, so obviously, you can train a SVM using TensorFlow optimization tools. But TensorFlow only comes with gradient descent-based optimizers which are a bit stupid to use to train a SVM unless you have lots of observations, since there are specific optimizers for SVM that do not get stuck in local minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, probably worth mention, that TensorFlow and Theano are pretty low-level frameworks. Most people use frameworks that are built on top of them, and are easier to use. I won't suggest here none, because that would generate its own discussion. See &lt;a href=&quot;https://datascience.stackexchange.com/questions/694/best-python-library-for-neural-networks/9240&quot;&gt;here suggestions&lt;/a&gt; for easy to use packages.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2018-04-06T00:21:14.927" LastActivityDate="2018-04-06T00:21:14.927" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12921" PostTypeId="2" ParentId="12919" CreationDate="2016-07-21T20:43:04.383" Score="15" Body="&lt;p&gt;This is a big oversimplification, but there are essentially two types of machine learning libraries available today: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Deep learning&lt;/strong&gt; (CNN,RNN, fully connected nets, linear models)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Everything else&lt;/strong&gt; (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The reason for this is that deep learning is much more computationally intensive than other more traditional training methods, and therefore requires intense specialization of the library (e.g., using a GPU and distributed capabilities). If you're using Python and are looking for a package with the greatest breadth of algorithms, try scikit-learn. In reality, if you want to use deep learning and more traditional methods you'll need to use more than one library. There is no &quot;complete&quot; package. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-07-21T20:43:04.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12922" PostTypeId="1" CreationDate="2016-07-21T21:25:53.747" Score="1" ViewCount="128" Body="&lt;p&gt;I'm trying to build a model of Machine Learning for the first time, and I'm looking for some guidelines.&lt;br&gt;&#xA;My final purpose is to identify customers who are planning to flight abroad.&lt;br&gt;&#xA;From my existing data, I identified customers who flew in the past, and I've flagged some potential variables (features) that might indicate that one is about to flight abroad.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;br&gt;&#xA;1. What techniques should I use, in order to choose the right features for my model?&lt;br&gt;&#xA;2. What approaches (models) would you use in such a problem?    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;because this is the first time I'm experiencing ML, I'll be happy to hear any further suggestions you have to give me.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your help would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="20594" LastActivityDate="2016-07-21T23:56:08.333" Title="Guidelines for Machine Learning model" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12923" PostTypeId="2" ParentId="829" CreationDate="2016-07-21T21:31:18.530" Score="1" Body="&lt;p&gt;Normally the GROUP/COGROUP is used to group the relation by some key.after you group the relation describe the grouped relation.you can find &#xA;EX:&#xA;describe grp;&#xA;grp: {group: chararray,A: {(name: chararray,session: chararray,gpa: float)}}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in the above result you can find &quot;group&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you want to perform some operation on grouped relation(grp) ,you should use the &quot;group&quot; not GROUP.&lt;/p&gt;&#xA;" OwnerUserId="21699" LastActivityDate="2016-07-21T21:31:18.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12924" PostTypeId="1" CreationDate="2016-07-21T21:40:14.060" Score="0" ViewCount="87" Body="&lt;p&gt;I am wondering if we have a &lt;strong&gt;&lt;code&gt;A= n*p&lt;/code&gt;&lt;/strong&gt; matrix of samples and we run a PC decomposition on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say the eigenvector matrix is &lt;strong&gt;E&lt;/strong&gt;, so the samples in the eigenvector space should be &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;B= A*(E)^-1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, I am wondering if there is any rule that the columns of B to be correlated? That is, the first PC loadings and the second PC loadings to be correlated?&#xA; Thanks! &lt;/p&gt;&#xA;" OwnerUserId="16380" LastEditorUserId="10902" LastEditDate="2016-11-20T08:47:14.413" LastActivityDate="2016-12-20T09:40:27.247" Title="Could principle component decomposed coordinates value be correlated to each other?" Tags="|correlation|dimensionality-reduction|pca|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12925" PostTypeId="2" ParentId="12922" CreationDate="2016-07-21T23:56:08.333" Score="2" Body="&lt;p&gt;&lt;strong&gt;Which algo you should use is up to debate and sometimes comes down to the person&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're going for accuracy then random forests are a great place to start (as mentioned in the comments) but personally I like logistic regressions as my starting place for this type of problem as they have decent accuracy and are comparatively explainable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if this is your first Machine Learning problem I'd recommend doing some wider reading first on how to formulate and approach machine learning problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It'll give you the perspective to decide which model to use on your own. Furthermore, you'll likely need to explain your model choices further down the line and using the &quot;some dude from stack overflow told me to use it&quot; excuse won't cut the mustard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Links:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/&quot; rel=&quot;nofollow&quot;&gt;How to approach a ML problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.kaggle.com/author/kevin-markham/&quot; rel=&quot;nofollow&quot;&gt;Intro to ML in Python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-07-21T23:56:08.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12926" PostTypeId="2" ParentId="12895" CreationDate="2016-07-22T00:11:43.153" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;You could think about rebalancing the dataset using undersampling of the majority class, oversampling the minority dataset or by applying &lt;a href=&quot;https://www.google.co.nz/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=3&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjQnZ_64oXOAhUCFpQKHXXZBkgQFggxMAI&amp;amp;url=https%3A%2F%2Fwww.jair.org%2Fmedia%2F953%2Flive-953-2037-jair.pdf&amp;amp;usg=AFQjCNGdcP-u3VG9cnA9_eCa7LU1LTFNdw&amp;amp;sig2=Veesh323ZlzWJV878xFV7Q&quot; rel=&quot;nofollow&quot;&gt;SMOTE&lt;/a&gt; to the dataset. &lt;/li&gt;&#xA;&lt;li&gt;If you've got the dataset more balanced then either Logistic Regression / Random Forests aren't a bad starting place. Random Forests I believe are better at handling unbalanced classification problems, but they might still have troubles with the degree of unbalance that you're talking about.&lt;/li&gt;&#xA;&lt;li&gt;Survival analysis generally has an aspect of time to the problem (i.e: when might a hard drive fail?) so if time is included in your data then you definitely could frame it as a survival analysis problem. However if time doesn't enter the problem then it might be easier to just analyse as a ML problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-07-22T00:11:43.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12927" PostTypeId="1" CreationDate="2016-07-21T22:53:02.157" Score="1" ViewCount="189" Body="&lt;p&gt;I hope this question is relevant to this community as Theano is increasingly a standard software to do machine learning.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;So I have been trying to follow the instructions here,&#xA;&lt;a href=&quot;http://deeplearning.net/software/theano/install_windows.html#install-windows&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/software/theano/install_windows.html#install-windows&lt;/a&gt; to install Theano on my Windows machine but as expected I am stuck with various parts.&lt;/p&gt;&#xA;&lt;p&gt;So I have created a folder called &amp;quot;Theano&amp;quot; and in that I have 3 folders which have installed ``Theano-0.8.2&amp;quot; , GCC and Anaconda.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I am following the &amp;quot;Alternative:Anaconda&amp;quot; route and there I am instructed to type into my Windows command prompt, &amp;quot;conda install mingw libpython&amp;quot;. But on doing that I get this message saying that &lt;code&gt;mingw&amp;quot; and &lt;/code&gt;python 3.5*&amp;quot; specifications are in conflict.&lt;/p&gt;&#xA;&lt;p&gt;What does this mean? What is ``mingw&amp;quot;!?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the &amp;quot;Configuring The Environment&amp;quot; section what is this &amp;quot;PATH&amp;quot; that is being talked about? Do I have to write this ``shell script&amp;quot;? (What is this?) Where does one write this and save this file?&lt;/p&gt;&#xA;&lt;p&gt;Can someone show a generic version of this script in which I can maybe fill in the blanks corresponding to how my folders are organized? (The .bat file that they have written seems to correspond to their file structure and its hardly clear as to which parts change!)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then further down one sees this section, &amp;quot;Configuring Theano&amp;quot; it asks one to open a &amp;quot;Python Shell&amp;quot; and run the command &amp;quot;python setup.py develop&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;So what is the &amp;quot;Python Shell&amp;quot;? Where do I find it?&#xA;Is it that black &amp;quot;Anaconda Prompt&amp;quot; terminal?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;To be clear now when I open my &amp;quot;Anaconda Prompt&amp;quot; window it says,&lt;/p&gt;&#xA;&lt;p&gt;&amp;lt; C:\Users(my_user_name)\Theano\Anaconda &amp;gt; C:\Users(my_user_name)&amp;gt;&lt;/p&gt;&#xA;&lt;p&gt;Seeing this I thought that Theano is now installed.&#xA;But when I open the Jupyter notebook and type &amp;quot;import theano&amp;quot;, I get an error!&lt;/p&gt;&#xA;" OwnerDisplayName="gradstudent" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-07-22T00:28:36.877" Title="About installing Theano" Tags="|machine-learning|visualization|data-mining|theano|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="12928" PostTypeId="1" AcceptedAnswerId="13116" CreationDate="2016-07-22T00:36:21.247" Score="1" ViewCount="1419" Body="&lt;p&gt;I have an unbalanced dataset X. &#xA;I split it between data and labels, then I standardize the data.&#xA;Then I use train_test_split to split between train and test data and I output the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to compare to see what I would get if I used smote to upsample the minority class, but I want to keep the comparison meaningful. To do that, I keep the same test data as before, and I add the new synthetic samples only to the train data. How should I handle the standardization? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Simply assume that since I used points from the already standardised data that the new synthetic samples will also be standardised? Therefore just add the synthetic samples to the train data and be done with it? (Without touching the test data)&lt;/li&gt;&#xA;&lt;li&gt;Create the synthetic samples from the pre-standardised data, standardise the synthetic samples and add them to the train data? (Without touching the test data)&lt;/li&gt;&#xA;&lt;li&gt;Create the synthetic samples from the pre-standardised data, add them to the train data, and standardise the whole set? (Without touching the test data)&lt;/li&gt;&#xA;&lt;li&gt;Or ... ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am getting very different results doing these three techniques, what is the best way to get a meaningful result and comparison?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="25849" LastEditDate="2020-10-02T18:45:32.277" LastActivityDate="2020-10-02T18:45:32.277" Title="SMOTE and standardisation" Tags="|python|dataset|class-imbalance|smote|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12929" PostTypeId="1" AcceptedAnswerId="12938" CreationDate="2016-07-22T04:04:06.767" Score="4" ViewCount="1036" Body="&lt;p&gt;Could someone give an example of the application of Tf-idf with sparse data (lots of zeros) in sklearn? I am not quite sure where to insert the weight of Tf-idf and how to rightly obtain the weight. Without knowing this fully, I would not be able to well use the tool for prediction. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="21242" LastActivityDate="2016-07-22T10:00:41.657" Title="Sklearn Linear Regression examples" Tags="|scikit-learn|linear-regression|weighted-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12930" PostTypeId="1" CreationDate="2016-07-22T04:09:52.630" Score="3" ViewCount="7611" Body="&lt;p&gt;I have a set of data which indicates purchase transaction of users (~1 million records). User can have more than 1 purchase across time. Data is spread over 6-7 months.&#xA;Attributes that I have are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;userid&lt;/li&gt;&#xA;&lt;li&gt;merchant &lt;/li&gt;&#xA;&lt;li&gt;amount&lt;/li&gt;&#xA;&lt;li&gt;day of purchase&lt;/li&gt;&#xA;&lt;li&gt;date of purchase&lt;/li&gt;&#xA;&lt;li&gt;time of purchase&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Goal&lt;/strong&gt; - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cluster users based on their purchase behaviour, so that I can target set of users (Promotions) based on their cluster behaviour.&#xA;Eg - I want to send promotions to user who execute XX behaviour&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have tried&lt;/strong&gt; - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried KMeans Clustering in Python Sklearn for 10,000 sample records. By using &quot;elbow method&quot; I have identified that 6-7 cluster are optimal. &#xA;Still, Clusters are not really helpful in identifying similar group of users. Clusters does not make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt; -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Am I doing this correctly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What would be the ideal algorithm for this type of use case. Should I be looking in Clustering or something like Recommendation algorithm (Group similar users)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Questions/Goal/Target group of users might change over time. How can we handle that?&lt;/p&gt;&#xA;" OwnerUserId="21707" LastActivityDate="2016-07-23T05:37:31.610" Title="Clustering users based on buying behaviour" Tags="|clustering|recommender-system|scikit-learn|k-means|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12931" PostTypeId="1" CreationDate="2016-07-22T05:49:15.560" Score="0" ViewCount="73" Body="&lt;p&gt;I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways are to measure the significance of the change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I changed one of my website's background from black to orange, this actually increased the site traffic by 17%.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I moved the check out button to a more obvious position on a different site, this specific site unfortunately experienced decreased traffic the day after the implementation by 8%.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have three questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I know the somewhat traditional method of measuring significance, basically run a t-test (look up the z-score in a table and see the probability of this happening by chance, and if its &amp;lt;5% then the change is significant, etc). But is there a more advanced (data-science-y) way of measuring the significance of the change? If it's too much to explain, pointing me to a general direction to look is greatly appreciated!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Speaking of a more data-science-y approach, I've been thinking about triple-exponential smoothing. I can break the day down by hour and build a model of usage by hour, then I can compare the actual hourly usages to the predicted usages and compare the two. Is this a more sophisticated way to go about this or am I just being dumb?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Right now I'm testing out features on different sites in order to minimized interference of the new features I'm implementing. If I implement both these changes to the same site, what are some of the ways I can separate and measure the significance of these two features (kind of similar to partial correlations where I take control of certain variables).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading! Please let me know your thoughts. Any advice would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="19132" LastActivityDate="2016-09-20T11:55:38.900" Title="Advance Methods of Understanding Significance of Customer Behaviors" Tags="|statistics|feature-extraction|ab-test|sequential-pattern-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12932" PostTypeId="2" ParentId="1028" CreationDate="2016-07-22T08:15:15.173" Score="1" Body="&lt;p&gt;&lt;strong&gt;STRUCTURED DATASET -&amp;gt; MISLEADING OOB ERRORS&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I've found interesting case of RF overfitting in my work practice. When data are structured RF overfits on OOB observations.&lt;/p&gt;&#xA;&lt;p&gt;Detail:&lt;/p&gt;&#xA;&lt;p&gt;I try to predict electricity prices on electricity spot market for each single hour (each row of dataset contain price and system parameters (load, capacities etc.) for that single hour).&lt;br /&gt;&#xA;Electricity prices are created in batches (24 prices created on electricity market in one fixing in one moment of time).&lt;br /&gt;&#xA;So OOB obs for each tree are random subsets of set of hours, but if you predict next 24 hours you do it all at once (in first moment you obtain all system parameters, then you predict 24 prices, then there is an fixing which produces those prices), so its easier to make OOB predictions, then for the whole next day. OOB obs are not contained in 24-hour blocks, but dispersed uniformly, as there is an autocorrelation of prediction errors its easier to predict price for single hour which is missing then for whole block of missing hours.&lt;/p&gt;&#xA;&lt;p&gt;easier to predict in case of error autocorrelation:&lt;br /&gt;&#xA;&lt;strong&gt;known, known, prediction, known, prediction&lt;/strong&gt; - OBB case&lt;br /&gt;&#xA;harder one:&lt;br /&gt;&#xA;&lt;strong&gt;known, known, known, prediction, prediction&lt;/strong&gt; - real world prediction case&lt;/p&gt;&#xA;&lt;p&gt;I hope it's interesting.&lt;/p&gt;&#xA;" OwnerUserId="8237" LastEditorUserId="29169" LastEditDate="2021-02-08T02:11:50.497" LastActivityDate="2021-02-08T02:11:50.497" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12933" PostTypeId="2" ParentId="12931" CreationDate="2016-07-22T08:51:19.840" Score="1" Body="&lt;p&gt;Welcome to DS stack exchange! Some of your questions/statements are not too clear, I'll try to answer to the best of my abilities but try to ask more precise questions in the future to get better answers (and avoid being down-voted).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To your points 1. and 2.: I would say it's highly unlikely that you'd be able to attribute a small fluctuation like 8% to just moving a button, moreover on such a short time span. And the question you should ask yourself is: &quot;Is this the &lt;em&gt;only&lt;/em&gt; thing that you've changed?&quot;. The fluctuation in point 2. could easily be attributed to effects like it being the weekend. Two things you'll want to look at to get you started are &lt;a href=&quot;https://en.wikipedia.org/wiki/A/B_testing&quot; rel=&quot;nofollow&quot;&gt;A/B testing&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model&quot; rel=&quot;nofollow&quot;&gt;ARMA models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use A/B testing to show &lt;em&gt;some&lt;/em&gt; of your customers e.g. a button on a new position or a new background color. Now check how likely they are to return vs. the batch of customers who saw the old design. If the new one is significantly better, then you can be confident that you should roll out the new design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set up an ARMA model to predict daily aggregate traffic. I'd advice against smoothing of any kind, it would likely be a bad idea. You first need to set a baseline which you do by comparing e.g. a month's worth of predictions with the actual counts. From this you can estimate the probability distribution of the errors you'd expect from your model. Now when you roll out new features/contents, you can see if the predicted traffic (which is based on data for the &lt;em&gt;old&lt;/em&gt; website) is significantly lower that the actual traffic. If so, your new feature/content is a success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-22T08:51:19.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12935" PostTypeId="2" ParentId="12924" CreationDate="2016-07-22T09:13:50.437" Score="0" Body="&lt;p&gt;They should be uncorrelated. Just think of a simple example, where you have two variables $x_1$ and $x_2$, which are 100% correlated. If you run PCA on those, what would be the outcome? Well, you'd get the first principal component $\tilde x_1$, which would be some linear combination of your original variables $x_1$ and $x_2$ absorbing all the fluctuations and a second component which is always zero.&lt;/p&gt;&#xA;" OwnerUserId="21567" LastActivityDate="2016-07-22T09:13:50.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12936" PostTypeId="2" ParentId="12924" CreationDate="2016-07-22T09:23:51.673" Score="1" Body="&lt;p&gt;PCs are calculated based on eigenvectors (e1 nad e2 in your case) of correlation or covariance matrix. Eigenvectors are orthogonal, so PCs are uncorrelated. If you would like to check the proof, you can find it on page 432 in &lt;em&gt;Applied Multivariate Statistical Analysis 6th edition&lt;/em&gt; written by Richard A. Johnson and Dean W. Wichern.&lt;/p&gt;&#xA;" OwnerUserId="21713" LastActivityDate="2016-07-22T09:23:51.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12937" PostTypeId="1" CreationDate="2016-07-22T09:30:22.507" Score="3" ViewCount="736" Body="&lt;p&gt;For some $8000$ customer profiles, in addition to a data-set, I have two kinds of scores available:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Type 1 Score&lt;/code&gt; ranges from $0$ to $1$ and gives the prediction probability of that profile belonging to a class. The distribution of this score is severely skewed, as shown&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dbWW3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dbWW3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Type 2 Score&lt;/code&gt; ranges from $-5$ to $5$ and is a risk score for the profile obtained using the &lt;a href=&quot;http://poseidon01.ssrn.com/delivery.php?ID=872119091124115016031102012127095087049073070064025021111113071102071087101116120000120032062006006027002113105107107064023000000029026049019127065083026090094023124018052042067103003071104125064074030007117079089123125110073115125085127122067113082104&amp;amp;EXT=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PRIDIT&lt;/a&gt; method on the actual data. The distribution of this score is fairly normal&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q3jsf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q3jsf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to bin the profiles into some $N$ bins using the aforementioned information. The bins should represent various degrees of risk associated with that profile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought was to cluster the profiles based on the two scores available to us. However no meaningful clusters were observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next idea was to sort the data on &lt;code&gt;Type 1 Score&lt;/code&gt; and then cut on the points where there's a sudden change in slope, which seemed like a good idea at first but here's what the plot for &lt;code&gt;Type 1 Score&lt;/code&gt; looks like ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XwZeT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XwZeT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am thinking now is to transform &lt;code&gt;Type 1 Score&lt;/code&gt; and &lt;code&gt;Type 2 Score&lt;/code&gt; together to give some pseudo-normal type of distribution, hoping that some kind of discretization &lt;em&gt;might&lt;/em&gt; give the correct bins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My questions are:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can I transform the two scores into a pseudo-normal distribution?&lt;/li&gt;&#xA;&lt;li&gt;What would be a better way to approach the above problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13450" LastActivityDate="2016-10-21T08:27:32.407" Title="Unsupervised binning of non-normal data" Tags="|machine-learning|clustering|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12938" PostTypeId="2" ParentId="12929" CreationDate="2016-07-22T10:00:41.657" Score="2" Body="&lt;p&gt;There is an application of tf-idf on the &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow&quot;&gt;sklearn website&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;sklearn handles sparse matrices for you, so I wouldn't worry about it too much:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Fortunately, most values in X will be zeros since for a given document less than a couple thousands of distinct words will be used. For this reason we say that bags of words are typically high-dimensional sparse datasets. We can save a lot of memory by only storing the non-zero parts of the feature vectors in memory.&#xA;  scipy.sparse matrices are data structures that do exactly this, and scikit-learn has built-in support for these structures.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Regarding your point about inserting the weight, I guess you have already performed tf-idf on your training corpus, but you don't know how to apply it to your test corpus? If so you could so as follows (taken from above link)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfTransformer&#xA;from sklearn.feature_extraction.text import CountVectorizer&#xA;&#xA;count_vect = CountVectorizer()&#xA;X_train_counts = count_vect.fit_transform(data)&#xA;&#xA;# Perform tf-idf&#xA;tfidf_transformer = TfidfTransformer()&#xA;X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)&#xA;&#xA;docs_new = ['God is love', 'OpenGL on the GPU is fast'] # New test documents&#xA;X_test_counts = count_vect.transform(docs_new) # Count vectorise the new documents&#xA;X_test_tfidf = tfidf_transformer.transform(X_new_counts) # transfrom the test counts &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20429" LastActivityDate="2016-07-22T10:00:41.657" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12939" PostTypeId="2" ParentId="12930" CreationDate="2016-07-22T14:48:51.067" Score="0" Body="&lt;p&gt;I think its not a problem of algo, you should do the job of understanding and describing clusters after you got them. What helps me is computing means of variables that are used in clustering and then putting their difference in some other logical ground (like animals or colors or some cartoon heroes), then try to put labels on the clusters that describes them best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, in your case, i should start with making some metrics out of your variables, like turn time of purchase into share of purchases for each customer made in day hours or evening, or converting a day to weekend/working day and so on.&lt;/p&gt;&#xA;" OwnerUserId="21722" LastActivityDate="2016-07-22T14:48:51.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12942" PostTypeId="2" ParentId="8099" CreationDate="2016-07-22T20:11:14.697" Score="1" Body="&lt;p&gt;A rule based classifier is generally suited more for this problem where most of your features are going to contain discrete values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Decision trees, Boosting, Random forests should do the job for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing you should always keep in mind is how you are going to evaluate your model. For fraud detection, make sure that False negative is eliminated completely.  A false positive is fine, but the opposite is dangerous.  &lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T20:11:14.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12944" PostTypeId="2" ParentId="454" CreationDate="2016-07-22T20:19:02.920" Score="0" Body="&lt;p&gt;As mentioned above, the best way is to repeatedly sample the majority class N times(sampling without replacement) and for each time, the size of negative class should be equal to the size of positive class. Now, N different classifiers can be trained and the average can be used to evaluate it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way is to use the technique of bootstrapping. This might introduce overfitting, but worth trying and then if neeeded can regularize the model to avoid overfitting.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T20:19:02.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12945" PostTypeId="2" ParentId="4942" CreationDate="2016-07-22T20:26:30.733" Score="0" Body="&lt;p&gt;Aleksander has given a very comprehensive answer but there are a few that are sued very widely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For dimensionality reduction, PCA is used.This, however, does only a linear transformation and for non-linear dimensionality reduction, Manifold learning is what you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Projecting a lower dimensional data to higher dimensions can be done using kernels. You usually do this, when your classifier is unable to find a linear plane of separation in current dimension but will be able to find a linear hyperplane that separates the classes in a higher dimension. Kernels are used widely in SVM's.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T20:26:30.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12946" PostTypeId="2" ParentId="265" CreationDate="2016-07-22T21:19:41.657" Score="3" Body="&lt;p&gt;A lot of people have stressed about what are the things that can be predicted in their answers. Now, with the fascination for deep learning, you could, for example, use RNN's(say LSTM) to predict outcomes for sports problems that are based on time. These are state of the art and beat traditional models hands down.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T21:19:41.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12947" PostTypeId="2" ParentId="52" CreationDate="2016-07-22T21:27:03.690" Score="2" Body="&lt;p&gt;There is an entire course devoted to this in Coursera. You might want to go over the techniques they mention and the important part is to know when to use what.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/data-cleaning&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/learn/data-cleaning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's always better to do it manually instead of automating since each dataset has its own problems and issues and not all steps are applicable to all the datasets.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T21:27:03.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12948" PostTypeId="2" ParentId="2368" CreationDate="2016-07-22T21:37:01.177" Score="1" Body="&lt;p&gt;Plot graphs with different variations of time against the outcome variable to see its impact. You could use  month, day, year as separate features and since month is a  categorical variable, you could try a box/whisker plot and see if there are any patterns. For numerical variables, you could use a scatter plot.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T21:37:01.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12949" PostTypeId="2" ParentId="61" CreationDate="2016-07-22T21:54:16.960" Score="0" Body="&lt;p&gt;There have been a lot of good explanations about overfitting.&#xA;Here are my thoughts. Overfitting happens when your variance is too high and bias is too low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have training data with you, which you divide into N parts. Now, if you train a model on each of the datasets, you will have N models. Now  find the mean model and then use the variance formula to compute how much each model varies from the mean. For overfitted models, this variance will be really high. This is because, each model would have estimated parameters which are very specific to the small dataset that we fed to it. Similarly, if you take the mean model and then find how much it is different from the original model that would have given the best accuracy, it wouldn't be very different at all. This signifies low bias. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find whether your model has overfitted or not, you could construct the plots mentioned in the previous posts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, to avoid overfitting you could regularize the model or use cross validation.&lt;/p&gt;&#xA;" OwnerUserId="21510" LastActivityDate="2016-07-22T21:54:16.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12950" PostTypeId="2" ParentId="12679" CreationDate="2016-07-22T22:28:59.180" Score="4" Body="&lt;p&gt;I was in the same boat, but now that I have figured it out, I have listed the steps for installing tensorflow 0.9 with cuda toolkit 8.0, cudnn 5.1, bazel 0.3 on Ubuntu 16.04 LTS here: &lt;a href=&quot;http://abhay.harpale.net/blog/machine-learning/deep-learning/getting-tensorflow-to-work-with-gpu-nvidia-gtx-1080-on-ubuntu-16-04-lts/&quot; rel=&quot;nofollow&quot;&gt;http://abhay.harpale.net/blog/machine-learning/deep-learning/getting-tensorflow-to-work-with-gpu-nvidia-gtx-1080-on-ubuntu-16-04-lts/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the gist &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Install NVidia Cuda Toolkit&lt;/li&gt;&#xA;&lt;li&gt;Install NVidia CuDNN&lt;/li&gt;&#xA;&lt;li&gt;Install Tensorflow dependencies such as swig, python-dev, numpy, python-wheel, zlib1g, g++&lt;/li&gt;&#xA;&lt;li&gt;Configure and build tensorflow using Bazel&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21733" LastEditorUserId="21733" LastEditDate="2016-07-25T04:55:30.710" LastActivityDate="2016-07-25T04:55:30.710" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12951" PostTypeId="2" ParentId="12840" CreationDate="2016-07-23T02:14:09.287" Score="1" Body="&lt;p&gt;This plot not use frequencies but kernel density:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;freqz &amp;lt;- with(data.frame(x,y), MASS::kde2d(x, y, n = 50))&#xA;with(freqz, plot_ly(x = x, y = y, z = z, type = &quot;surface&quot;)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/067d6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/067d6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21737" LastActivityDate="2016-07-23T02:14:09.287" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12953" PostTypeId="2" ParentId="12930" CreationDate="2016-07-23T05:37:31.610" Score="6" Body="&lt;p&gt;&lt;strong&gt;Big Picture:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform.  Some people may tend to buy things on payday (15th and 30th of the month), some people might like to shop on Sundays, some people might like sports stores, people who shop on Wednesdays could like to buy woman's clothes, midnight shoppers could tend to focus on &quot;as seen on TV&quot; products.  There might be signal there, but with only 6 features (really 5), it might be tough to find it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Normalization/Standardization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/6715/is-it-necessary-to-standardize-your-data-before-clustering&quot;&gt;Make sure you standardize your data &lt;strong&gt;before&lt;/strong&gt; using k-means clustering&lt;/a&gt;. Otherwise, your distance metric will loose rotational invariance and the clusters will become nonsense from feature to feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Engineering:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will want to do some feature engineering to help your clustering algorithm find signal in the data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Get rid of userid unless it has some meaningful information like it is a proxy for when they first signed up i.e. ask yourself whether this would help the algorithm cluster the data or provide noise to confuse the algorithm. The answer is likely the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Do you have the merchant name only? Have you one-hot encoded the merchant list? Can you turn the merchant name into one or more variables with lower cardinality like groupings for food, clothes, household, sporting, etc?  Can you find other information on merchants (other data), join it into your data and enhance this feature to make it much more useful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Amount can possibly be teased out a bit, but be conservative with this one. You could possibly flag purchases over $100 or $1000 as is_expensive and is_very_expensive.  Also possibly squaring the amount could account for buyer behavior around big ticket items.  This might just add noise, so be careful with this one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Day of week should be first turned into a numerical variable 1-7, but then I would also recommend &lt;a href=&quot;https://datascience.stackexchange.com/questions/8799/boundary-conditions-for-clustering&quot;&gt;turning cyclic variables into two spatial dimensions&lt;/a&gt;.  I explain how to do this &lt;a href=&quot;https://datascience.stackexchange.com/questions/8799/boundary-conditions-for-clustering&quot;&gt;here&lt;/a&gt;, and in addition to time, it can be applied to day-of-week also. I recommend keeping both the 1-7 format and the two dimensional spatial format.  Are there other days that you want to specifically highlight?  You could, for instance have an is_weekend variable that is 1 for Saturday and Sunday and 0 for other days. You could have another feature for is_boring_day that is 1 for Monday, Tuesday, and Wednesday and 0 for other days. Play with it, think outside the box... this can be a very useful feature but requires some coaxing at times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Date of purchase is pretty loaded in that you can pull out the months and turn them into cyclic variables.  You can also pull out the day of the month in a similar way.  You can create at payday flag for the 15th and 30th and create a feature like 1/(1+days_since_payday) since people might be more likely to buy on payday. You can add a seasons feature from the date. Finally, you can use raw date in days since your earliest piece of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) Similarly for time, you should &lt;a href=&quot;https://datascience.stackexchange.com/questions/8799/boundary-conditions-for-clustering&quot;&gt;turn this into a 2D spatial feature&lt;/a&gt;. This will make it easier to pull out the spending behavior of people who shop at certain times of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Clustering Algorithms&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;k-means can be a bit blunt at times.  I suggest you try at least two more clustering algorithms besides k-means. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#example-cluster-plot-dbscan-py&quot; rel=&quot;noreferrer&quot;&gt;DBSCAN&lt;/a&gt; is a very sophisticated clustering tool and works well for noisy data. It also benefits from not having to specify the number of clusters a priori. Experiment with changing the neighborhood size and how much noise is allowed to see how this effects your clustering. Try at least one more clustering type in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;noreferrer&quot;&gt;scikit arsenal&lt;/a&gt;... they are very easy to swap out once you do it once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Methods:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can play with trying to other supervised learning methods. Maybe try separating your data into training and testing sets and try predicting purchase amounts based on the other features.  Or try classifying the type of purchase based on the other information.  Maybe try training a &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html&quot; rel=&quot;noreferrer&quot;&gt;SVM for classification&lt;/a&gt; and a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html#sklearn.linear_model.BayesianRidge&quot; rel=&quot;noreferrer&quot;&gt;Bayesian-ridge regressor&lt;/a&gt; for predicting the purchase amount.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Handling changing user data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Changing data can be handled in a number of different ways.  The most obtuse is simply to retrain your model periodically, while either aggregating all of the data, or forgetting some of the older data due to a lack of relevance. The next step in sophistication is to use an &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dict_face_patches.html&quot; rel=&quot;noreferrer&quot;&gt;online learning algorithm&lt;/a&gt;, which continually updates the model based on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, just have fun with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-23T05:37:31.610" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12954" PostTypeId="1" CreationDate="2016-07-23T07:05:30.207" Score="0" ViewCount="313" Body="&lt;p&gt;I currently have a bunch of extracted news articles to perform news classification. However, the articles are unlabeled. There are about 160k articles therefore manually labeling them is impossible. I'm thinking of clustering the similar news articles together for easier labeling. Is this approach possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: Now I only want to separate business related news and non-business related news apart.&lt;/p&gt;&#xA;" OwnerUserId="20974" LastEditorUserId="20974" LastEditDate="2016-07-26T04:16:30.507" LastActivityDate="2018-04-03T19:30:34.007" Title="News clustering on unlabeled datasets" Tags="|python|clustering|unsupervised-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12955" PostTypeId="2" ParentId="12937" CreationDate="2016-07-23T07:29:31.713" Score="2" Body="&lt;p&gt;It often is appropriate to use quantile bins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. to get four bins, choose the smallest 25%, then up to the median, then the Q75, ...&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-07-23T07:29:31.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12956" PostTypeId="1" AcceptedAnswerId="19251" CreationDate="2016-07-23T09:46:42.783" Score="36" ViewCount="13086" Body="&lt;p&gt;So, recently there's a &lt;a href=&quot;https://arxiv.org/abs/1607.06450&quot;&gt;Layer Normalization&lt;/a&gt; paper. There's also &lt;a href=&quot;https://gist.github.com/udibr/7f46e790c9e342d75dcbd9b1deb9d940&quot;&gt;an implementation of it&lt;/a&gt; on Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I remember there are papers titled &lt;a href=&quot;https://arxiv.org/abs/1603.09025&quot;&gt;Recurrent Batch Normalization&lt;/a&gt; (Cooijmans, 2016) and &lt;a href=&quot;https://arxiv.org/abs/1510.01378&quot;&gt;Batch Normalized Recurrent Neural Networks&lt;/a&gt; (Laurent, 2015). What's the difference between those three?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is this Related Work section I don't understand:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the recurrent batch normalization layer to 0.1 makes significant difference in the final performance of the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In weight normalization, &lt;strong&gt;instead of the variance, the L2 norm of the incoming weights is used to normalize the summed inputs to a neuron&lt;/strong&gt;. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is &lt;strong&gt;not a re-parameterization of the original neural network. The layer normalized model, thus, has different invariance properties than the other methods&lt;/strong&gt;, that we will study in the following section&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9465" LastEditorUserId="836" LastEditDate="2017-08-22T18:05:41.210" LastActivityDate="2018-03-12T01:06:02.507" Title="Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?" Tags="|deep-learning|rnn|normalization|batch-normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12958" PostTypeId="1" AcceptedAnswerId="13106" CreationDate="2016-07-23T12:34:34.210" Score="2" ViewCount="209" Body="&lt;p&gt;I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I want to know what category the samples without A would most likely be in if they did have A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could build a classifier based on the data with A, and predict the data without A (this is the best route IMHO).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'm wondering, out of curiosity, if this method could also be a very, very, very rough way of doing something similar:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Cluster the data which has A into the same number of clusters as categories in A (in this case four).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check for an an association between the clusters and the categories in A (using a frequency table and chi-square test).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If there is an association, run the data without A through the clustering model to figure out what category of A it's most likely associated with (based on what cluster it is in).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thoughts? &lt;/p&gt;&#xA;" OwnerUserId="21690" LastEditorUserId="21690" LastEditDate="2016-07-24T12:39:55.597" LastActivityDate="2016-07-31T00:48:59.390" Title="Cluster analysis as an associative model?" Tags="|classification|clustering|statistics|missing-data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12959" PostTypeId="1" AcceptedAnswerId="14532" CreationDate="2016-07-23T21:02:17.010" Score="3" ViewCount="144" Body="&lt;p&gt;I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (depending on song length). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand hmm_train cannot be used for sequences of vectors and murphyk's toolbox might work. I tried modifying the demo file as follows to try a single song but I'm getting the error &lt;em&gt;'Attempted to access obsmat(:,61.6357); index must be a positive integer or logical..'&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;O = 5;&#xA;Q = 10;&#xA;&#xA;v=wavread('We are the champions');&#xA;coeff=melfcc(v);&#xA;data=coeff';&#xA;&#xA;% initial guess of parameters&#xA;prior1 = normalise(rand(Q,1));&#xA;transmat1 = mk_stochastic(rand(Q,Q));&#xA;obsmat1 = mk_stochastic(rand(Q,O));&#xA;&#xA;% improve guess of parameters using EM&#xA;[LL, prior2, transmat2, obsmat2] = dhmm_em(data, prior1, transmat1, obsmat1, 'max_iter', 5);&#xA;LL    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I haven't worked on this stuff before so I apologize in advance if my question comes across as really basic. It would be great if somebody could tell me where I'm going wrong. Also for multiple clips, should I simply append the matrices generated to the one I'm supplying above? Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: It turns out that the above hmm can only be applied to discrete values which is possibly why I was getting errors. It seems I have to use another function for 'gaussian output' instead, mhmm_em. I'll post another question to figure out its working. &lt;/p&gt;&#xA;" OwnerUserId="21749" LastEditorUserId="21749" LastEditDate="2016-07-24T09:59:19.820" LastActivityDate="2017-11-22T19:14:21.553" Title="Using dhmm_em to form the hmm of mfccs' from song clips" Tags="|machine-learning|markov-process|audio-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12960" PostTypeId="1" AcceptedAnswerId="12962" CreationDate="2016-07-23T22:33:04.860" Score="3" ViewCount="6086" Body="&lt;p&gt;I'm very new to the DS world, so please bear with my ignorance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to analyse user comments in Spanish. I have a somewhat small dataset (in the 100k's -- is that small?), and when I run the algorithm in a, let's say, naïve way (scikit-learn's default options +remove accents and no vocabulary / stop words) I get very high values for very common and low value words (such as spanish equivalents of &quot;to&quot;, &quot;at&quot;, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the most effective way to train the vectoriser on a 100k long corpus of ~200 char long docs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of using a larger corpus of Spanish text or looking for a stop word removing implementation for Spanish, but would love to have some expert advise before jumping into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21750" LastActivityDate="2016-07-25T08:31:47.323" Title="Choosing the right parameters to train a Tf-Idf vectoriser" Tags="|text-mining|feature-extraction|text|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12961" PostTypeId="1" CreationDate="2016-07-24T08:33:43.370" Score="1" ViewCount="317" Body="&lt;p&gt;I have a collection of 80k graphs, half of them are labeled 1 (they represent the cities of France) and the other half is labeled 0 (representing cities in the USA). My goal is to create a classifier that can predict whether a graph (representing a city), using a kernel on graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since nodes in graphs representing cities are generally of degree &amp;lt;&amp;lt; number of nodes, I decided to use the all connected 3,4,5 graphlet kernel presented in this article : &lt;a href=&quot;http://jmlr.csail.mit.edu/proceedings/papers/v5/shervashidze09a/shervashidze09a.pdf&quot; rel=&quot;nofollow&quot;&gt;Article about all connected graphlet kernel&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What this kernel does is simple : it represents each graph G by a vector F(G) which is the distribution of the all connected 3, 4 and 5 graphlets in the original graph. There are 29 different graphlets of size 3, 4 and 5 so every graph is now represented by a vector of length 29. The article then defines the kernel between two graphs G and G' by the inner product between the two vectors : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;code&gt;K(G,G') = &amp;lt;F(G),F(G')&amp;gt;&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that this kernel does not behave like others kernels we are used to work with : we specifically know the feature map and the Hilbert space associated to the kernel. The Hilbert space is R^29 and the feature map is F. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then in order to create the classifier I use SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here's the problem : if I use the feature map F as it is, then compute the gram matrix and apply SVM, all predictions on the test data are 0. That's pretty sad. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have been noticing is that 3 of the 29 graphlets are much more present than the others in the graphs. So for a typical city, 95% of the distribution is concentrated on theses 3 graphs, which means that when you compute the kernel (the inner product), the gram matrix have really similar values : the standard deviation of the matrix is 0.0003. Naturally, I though : &lt;strong&gt;I want to transform F so that the differences between graphs are clearer.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I computed the mean distribution M of 3,4,5 graphlets in my database, I derived the inverse mean distribution M' (I just inverted every element of M), and instead of using F, i used F* which takes a graph G, computes F(G), and then F*(G) by multiplying point by point F(G) and M'. That way, the differences between graphs when computing the kernel are more easy to see. Note that the kernel is still valid since it's still an inner product in R^29. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea how mathematically accurate is this intuition, but when I applied SVM with this other feature map F*, I got a 82% accuracy, and the AUC was 0.88. The ROC curves is beautiful too. So I must have done something right ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My actual question is why does SVM likes the feature map F* more than F ?&lt;/strong&gt; I mean I didn't add any information using the new feature map, I just modified the old one using my intuition. &lt;/p&gt;&#xA;" OwnerUserId="21755" LastActivityDate="2016-07-24T08:33:43.370" Title="Kernel on graphs and SVM : a weird interaction." Tags="|classification|svm|graphs|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12962" PostTypeId="2" ParentId="12960" CreationDate="2016-07-24T08:45:05.480" Score="0" Body="&lt;p&gt;There are no silver bullets. But here are some suggests:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a better stopwords vocabulary. If you still have words like &quot;to&quot; and &quot;at&quot;, then you are either not removing stopwords or using a lousy vocabulary. Try using the Spanish stopwords from &lt;code&gt;nltk&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import stopwords&#xA;stopwords.words('spanish')&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;code&gt;max_df &amp;lt; 1&lt;/code&gt;. This will truncate words that appear in more than that percentage number of documents.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The TF-IDF part that punishes common words (transversal to all documents) is the IDF part of TF-IDF, which means inverse document transform. Several functions may be used as your &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Inverse_document_frequency_2&quot; rel=&quot;nofollow&quot;&gt;IDF function&lt;/a&gt;. Usually, IDF=$\log\frac{\text{#documents}}{\text{#documents where word appears}}$ is used. You could try a more punitive IDF function. sklearn does not seem to allow to specify it, but you can use &lt;code&gt;nltk&lt;/code&gt; or &lt;code&gt;gensim&lt;/code&gt; or easily implement your own TF-IDF vectorization. It needs no more than five lines of code.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would try each of these suggestions in this order, and stop when it is good enough. It sounds like using a better stopwords vocabulary will be good enough for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I forgot to mention, but of course you can add more uninteresting words to the stopwords list, if you have the need to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to first evaluate the impact TF-IDF is having in your counts. I think this will work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m = TfidfVectorizer().fit(docs)&#xA;c = [m.transform([word]) for word in m.vocabulary_]&#xA;for i in np.argsort(c)[::-1][:20]:&#xA;    print(m.vocabulary_[i], c[i])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You may have to adjust the code. Try to use with IDF disabled and enabled, and other tweaks.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-07-25T08:31:47.323" LastActivityDate="2016-07-25T08:31:47.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12964" PostTypeId="1" AcceptedAnswerId="13162" CreationDate="2016-07-24T10:17:35.023" Score="38" ViewCount="48048" Body="&lt;p&gt;From Tensorflow code: &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn_cell.py&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow. RnnCell.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_units: int, The number of units in the LSTM cell.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can't understand what this means. What are the units of LSTM cell? Input, Output and Forget gates? Does this mean &quot;the number of units in the recurrent projection layer for Deep LSTM&quot;? Then why is it called &quot;number of units in the LSTM cell&quot;?   What is an LSTM cell and how is it different from an LSTM block, what is the minimal LSTM unit if not a cell?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastEditorUserId="70874" LastEditDate="2020-04-07T07:01:43.163" LastActivityDate="2020-04-07T07:01:43.163" Title="What is the meaning of &quot;The number of units in the LSTM cell&quot;?" Tags="|neural-network|tensorflow|rnn|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="12967" PostTypeId="2" ParentId="12164" CreationDate="2016-07-24T15:56:31.693" Score="1" Body="&lt;p&gt;Usually, those non-sensical words are not problematic because they appear in one or two documents, and people usually just filter words with such low frequencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you not able to do this? Or do your non-sensical words appear that much?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, your suggestion is what people use:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;outright ignore tokens that aren't discernable by a dictionary. Is there?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Exactly. There is no magic way to know if a word is English or not. What word processors do is to use a dictionary, as you yourself suggested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In python, before stemming, you could filter based on &lt;a href=&quot;https://pypi.python.org/pypi/pyenchant/&quot; rel=&quot;nofollow&quot;&gt;pyenchant&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import enchant&#xA;d = enchant.Dict(&quot;en_US&quot;)&#xA;d.check('hello')  # True&#xA;d.check('mmmmmmmm')  # False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I bet this would be good enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if there are a lot of false negatives, you could ask the dictionary for the most similar words and then apply a word distance measure. Here I will use &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow&quot;&gt;Levenshtein distance &lt;/a&gt; as &lt;a href=&quot;https://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python&quot; rel=&quot;nofollow&quot;&gt;implemented in python&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; r = d.suggest('helo')&#xA;&amp;gt;&amp;gt;&amp;gt; r&#xA;['hole', 'help', 'helot', 'hello', 'halo', 'hero', 'hell', 'held', 'helm', 'he lo', 'he-lo', 'heel', 'loathe', 'Helios', 'helicon']&#xA;&amp;gt;&amp;gt;&amp;gt; min([LD('helo', w) for w in r])&#xA;1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The distance will be higher for the non-sensical words you suggested:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; d.suggest('mmmmmmmm')&#xA;['mammogram', &quot;Mammon's&quot;]&#xA;&amp;gt;&amp;gt;&amp;gt; min([LD('mmmmmmmm', w) for w in d.suggest('mmmmmmmm')])&#xA;5&#xA;&amp;gt;&amp;gt;&amp;gt; min([LD('aaannnammmmrrr', w) for w in d.suggest('aaannnammmmrrr')])&#xA;12&#xA;&amp;gt;&amp;gt;&amp;gt; min([LD('hdhhdhhhhhjjj', w) for w in d.suggest('hdhhdhhhhhjjj')])&#xA;10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To recap:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;filter words that appear in less than 3 documents&lt;/li&gt;&#xA;&lt;li&gt;if that's not enough or it is not a possibility for your use-case, use a dictionary&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think this last suggestion of mine of using Levenshtein distance to the closest English words is overkill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your documents involve a lot of urban words or hip terms, you could also filter based on the number of Google results, instead of using a common dictionary. Your word &lt;a href=&quot;https://www.google.pt/webhp?sourceid=chrome-instant&amp;amp;ion=1&amp;amp;espv=2&amp;amp;ie=UTF-8&amp;amp;client=ubuntu#q=hdhhdhhhhhjjj&quot; rel=&quot;nofollow&quot;&gt;hdhhdhhhhhjjj&lt;/a&gt;, for instance, gives me only two Google results, one of which is this stackexchange. Data science is about being creative. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There probably are heuristics or statistics based on the number of consonants in a word, and letter combinations, which you could use to probabilistic shrink your documents' dictionary, but I wouldn't go there. It will be too much work and very brittle.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-07-24T15:56:31.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12971" PostTypeId="2" ParentId="12721" CreationDate="2016-07-24T21:24:57.843" Score="31" Body="&lt;p&gt;&lt;strong&gt;Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following might point you in the right direction but hopefully you'll get a few more answers with more depth in the arena of LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mention that you have tried both algorithms and that you are simply trying to figure out which one is better, which leads me to think you may be having more trouble with the data science process and cross validation than with the specifics of the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Time series in general:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series, in general, are difficult to forecast. If they were easy to forecast then all data scientists would be wealthy, having accurately forecast the value of all of the stocks. The reality is that hedge funds, on average, do not outperform the market and that time series forecasting is typically very poor and applies only to very short durations.  The main problems are that there is a lot of noise, there are many hidden influences, models are overly simplistic, influencers do not behave as we think they should, the interplay between linearity and nonlinearity is subtle and confusing, ... ad infinitum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ARIMA&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are incorrect in your assessment that ARIMA requires stationary time series to forecast on.  &lt;a href=&quot;http://www.slideshare.net/21_venkat/arima-26196965&quot; rel=&quot;noreferrer&quot;&gt;Non-seasonal ARIMA has three input values to help control for smoothing, stationarity, and forecasting&lt;/a&gt; ARIMA(p,d,q), where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;p is the number of autoregressive terms,&lt;/li&gt;&#xA;&lt;li&gt;d is the number of nonseasonal differences needed for stationarity, and&lt;/li&gt;&#xA;&lt;li&gt;q is the number of lagged forecast errors in the prediction equation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By contrast &lt;a href=&quot;http://people.duke.edu/~rnau/411arim.htm#pdq&quot; rel=&quot;noreferrer&quot;&gt;seasonal ARIMA has six input values&lt;/a&gt; ARIMA(p,d,q,P,D,Q), where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;P is the number of seasonal autoregressive terms, &lt;/li&gt;&#xA;&lt;li&gt;D is the number of seasonal differences, and &lt;/li&gt;&#xA;&lt;li&gt;Q is the number of seasonal moving-average&#xA;terms.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Subject to the qualifying statements above, I suggest playing with seasonal ARIMA to get a feel for the intricacies involved in smoothing, de-seasoning, de-trending, de-noiseing, and forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;LSTM&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know enough about LSTM to add much here. I will add that red flags tend to be raised when someone begins at data science exercise with deep learning. I suggest learning as much as you can using ARIMA and then applying some of your ARIMA expertise to help you learn LSTM. Neural networks can be a very powerful tool, but they:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;can take a long time to run,&lt;/li&gt;&#xA;&lt;li&gt;often require more data to train than other models, and&lt;/li&gt;&#xA;&lt;li&gt;have lots of input parameters to tune.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cross validation and comparing models:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series are fun in that all training data can usually be turned into supervised learning training sets.  Once can simply take a time series and roll back time. That is... pick a point in time and pretend that you don't have any additional data, then produce a forecast and see how well you did.  You can &lt;a href=&quot;https://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection&quot;&gt;march through the time series doing this $n$ times in order to get an assessment of the performance of your model&lt;/a&gt; and to compare models while taking the &lt;a href=&quot;http://www.stat.tugraz.at/AJS/ausg083+4/08304Kunst.pdf&quot; rel=&quot;noreferrer&quot;&gt;necessary precautions&lt;/a&gt; to &lt;a href=&quot;http://robjhyndman.com/hyndsight/tscvexample/&quot; rel=&quot;noreferrer&quot;&gt;prevent overfitting&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps and good luck!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-07-24T21:24:57.843" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="12972" PostTypeId="2" ParentId="9415" CreationDate="2016-07-24T21:46:46.360" Score="1" Body="&lt;p&gt;Wow I'm surprised that no one has given this one a go. I'm very very late to the party but I'll attempt an answer just for information's sake.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I attempt to answer your questions I'd like to say that any situation in which the number of features exceeds the number of observations by 50% there may be some instability in the results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I'd recommend is to attempt a number of feature selection approaches and compare the results&lt;/strong&gt;. Personally I'd start with a PCA to reduce the number of features such that n &gt; p and go from there. But that's just me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I'm not surprised by more estimates being zero at all. In the first lasso regression with this many features a large amount of these features are likely to be noise (no prediction power) meaning that any feature with a small capability of prediction would be selected in the first cut of variables. Running a secondary lasso regression might not be completely advisable (logically, can't really think of any theoretical issues at current) but &lt;strong&gt;it's likely that some of the stronger predictors are being uncovered on this second iteration&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The estimates in lasso regressions are quite difficult to interpret (apart from zero and non-zero values obviously), but what I'd recommend is to check whether you've scaled your features before running the lasso reg? As my understanding is that a lasso reg is a penalised version of the ridge regression, and the ridge regression needs the features to be scaled as otherwise features represented by larger numbers are given higher coefficients. If you have just ignore this, but &lt;strong&gt;if you want to interpret the estimates I'd switch to a Linear Reg/GLM not a lasso reg&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully this helps. &lt;/p&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-07-24T21:46:46.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12974" PostTypeId="1" AcceptedAnswerId="16507" CreationDate="2016-07-25T04:31:08.070" Score="4" ViewCount="3348" Body="&lt;p&gt;I'm computing and optimization of some Variables that are used on an external process, but I get the error &quot;No Gradient&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A heavily simplified (not tested) version of the code, but you can get the idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def external_process (myvar):&#xA;    subprocess.call(&quot;process.sh&quot;, myvar)&#xA;    with open('result.json', 'r') as f:&#xA;        result = json.load(data, f)&#xA;    return np.array(result[&quot;result&quot;])&#xA;&#xA;myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)&#xA;&#xA;loss = tf.reduce_sum( tf.py_func(external_process, [myvar], [tf.float32])[0] )&#xA;&#xA;optimizer = tf.train.AdamOptimizer(0.05)&#xA;train_step = optimizer.minimize(loss)&#xA;sess.run(train_step)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I saw this discussion but I don't fully understand it: &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/1095&quot; rel=&quot;nofollow&quot;&gt;https://github.com/tensorflow/tensorflow/issues/1095&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21775" LastActivityDate="2017-05-12T10:06:08.053" Title="Tensorflow - How to set gradient of an external process (py_func)?" Tags="|python|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12978" PostTypeId="1" CreationDate="2016-07-25T13:00:12.680" Score="1" ViewCount="218" Body="&lt;p&gt;After I calculated the similarities matrix, how do I get the neighbors, for example, consider the matrix of similarities between users, if I did not make any mistakes, the matrix must be symmetric with diagonal 1 considering pearson.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;u0: 1.0  -0.2  0.8  -0.6  0.2&#xA;u1:-0.2   1.0  0.7  -0.3  0.4&#xA;u2: 0.8   0.7  1.0   0.4  0.3&#xA;u3:-0.6  -0.3  0.4   1.0  0.8&#xA;u4: 0.2  -0.4  0.3   0.8  1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So if I want get 2 neighbors for u0, the result would be &lt;code&gt;[u2:0.8, u4:0.2]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't undertando why to do KNN and not a sort operation on its row?&lt;/p&gt;&#xA;" OwnerUserId="21787" LastActivityDate="2016-07-25T13:00:12.680" Title="KNN on collaborative filtering" Tags="|recommender-system|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12979" PostTypeId="1" CreationDate="2016-07-25T14:22:25.120" Score="0" ViewCount="481" Body="&lt;p&gt;I want to start   with  image processing (image classification).&#xA;I installed  opencv  and plan  to use it    with Python.&#xA;Being starter  at   this, I am looking   for references   regarding  trivial  issues like:&#xA;I just  saved some images  and  want to  get their pixel  values  as  features ,  hoewever   when I try loading it  in opencv,  I am    getting differently  shaped   3-dimmensional  objects.&#xA;Some   references   or suggestion regarding this   issue  will be highly  appreciated.&lt;/p&gt;&#xA;" OwnerUserId="18321" LastEditorUserId="13518" LastEditDate="2016-07-26T04:24:07.843" LastActivityDate="2016-11-23T13:55:37.067" Title="Features from images using opencv in Python" Tags="|python|feature-extraction|image-classification|image-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12980" PostTypeId="2" ParentId="12979" CreationDate="2016-07-25T14:53:52.953" Score="1" Body="&lt;p&gt;Your numpy arrays are 3-dimensional because they are colored images. The first dimension is the y-dimension, the second dimension is your x-dimension and the third dimension is the different color channels. What order they are in depends on your reader, but I think OpenCV uses BGR (check the documentation &lt;a href=&quot;http://docs.opencv.org/3.0-beta/doc/tutorials/tutorials.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;). A lot of the feature extraction methods only work on grayscale arrays which are mapped back to two dimensions, any of the well known image packages can do this for you (OpenCV, scikit-image).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="15412" LastEditDate="2016-11-23T09:23:33.690" LastActivityDate="2016-11-23T09:23:33.690" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="12981" PostTypeId="2" ParentId="12954" CreationDate="2016-07-25T16:15:40.833" Score="1" Body="&lt;p&gt;Simply clustering and then labeling all the texts in cluster will yield very noisy results (and you might learn your clustering similarity function rather that the &quot;real&quot; one). while there are some techniques that handle noisy labels (e.g- this NIPS paper: &lt;a href=&quot;http://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf&quot; rel=&quot;nofollow&quot;&gt;http://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf&lt;/a&gt;), there are several other options to consider:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;labeling a small amount of data (be sure to have label data from each category, assuming you know the categories), training a classifier, classifying new examples, adding the most certain one to the training set, than retraining the model and so on. For example, see this article: &lt;a href=&quot;http://link.springer.com/article/10.1023/A:1007692713085&quot; rel=&quot;nofollow&quot;&gt;http://link.springer.com/article/10.1023/A:1007692713085&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Active Learning can be useful in this scenario. E.g: &lt;a href=&quot;http://www.kamalnigam.com/papers/emactive-icml98.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.kamalnigam.com/papers/emactive-icml98.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Semi-supervised learning techniques. E.g: &lt;a href=&quot;http://mlg.eng.cam.ac.uk/zoubin/papers/zglactive.pdf&quot; rel=&quot;nofollow&quot;&gt;http://mlg.eng.cam.ac.uk/zoubin/papers/zglactive.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It is a common practice to combine more than one method. Also, LDA might be useful here.&lt;/p&gt;&#xA;" OwnerUserId="21790" LastEditorUserId="21790" LastEditDate="2016-07-26T07:38:49.493" LastActivityDate="2016-07-26T07:38:49.493" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12982" PostTypeId="2" ParentId="12958" CreationDate="2016-07-25T16:33:16.470" Score="1" Body="&lt;p&gt;I try to stay away from bounty problems as they seem to lead to one upmanship, but I can't resist this one as its an interesting question...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The upper classification method that your describe sounds spot on. Use that one!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bottom method has a number of issues:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you know that the clusters you get out would correspond to the categories of A? The answer is that you wouldn't. There are other data features and the clusters would be a linear superposition of the maximum separation between the various variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess you could standardize your data and then coax the clusters to conform to the separation along A by multiplying the A feature by a factor greater than 1 (~2-10), but this seems like more of a bastardization of clustering than a good process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main issue that I see with your method is that you are essentially employing a &lt;a href=&quot;http://www.cs.upc.edu/~bejar/amlt/material/06a-SemisupervisedClustering.pdf&quot; rel=&quot;nofollow&quot;&gt;semi-supervised learning methodology&lt;/a&gt;, but you don't need to because you have the data to solve this as a supervised learning problem. It would be a good methodology if you only had a handful of labeled A values, but since you have lots more data, you should harness the power of that data that you have. The supervised learning problem will fully harness the variance of your data and much of that will be thrown out in the semi-supervised learning methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps and good luck!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-07-25T16:33:16.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12984" PostTypeId="1" CreationDate="2016-07-25T18:55:53.813" Score="18" ViewCount="17906" Body="&lt;p&gt;Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine.&lt;/p&gt;&#xA;" OwnerUserId="2504" LastEditorUserId="29169" LastEditDate="2021-03-13T20:11:43.153" LastActivityDate="2023-01-05T11:52:11.490" Title="List of feature engineering techniques" Tags="|feature-selection|feature-extraction|feature-engineering|feature-construction|featurization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="12985" PostTypeId="2" ParentId="9823" CreationDate="2016-07-25T19:04:22.057" Score="3" Body="&lt;p&gt;I would suggest using &lt;a href=&quot;https://github.com/hyperopt/hyperopt&quot; rel=&quot;nofollow noreferrer&quot;&gt;hyperopt&lt;/a&gt; , which uses a kind of Bayesian Optimization for search optimal values of hyperparameters given the objective function. It is more intuitive to use than Spearmint.&lt;/p&gt;&#xA;&lt;p&gt;PS : There is a wrapper of hyperopt speifically for keras, &lt;a href=&quot;https://github.com/maxpumperla/hyperas&quot; rel=&quot;nofollow noreferrer&quot;&gt;hyperas&lt;/a&gt;. You can also use it.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="98307" LastEditDate="2020-08-03T14:25:33.807" LastActivityDate="2020-08-03T14:25:33.807" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12986" PostTypeId="1" CreationDate="2016-07-25T19:54:14.170" Score="3" ViewCount="3742" Body="&lt;p&gt;Nvidia is planing to add hardware support for int8 operations to their titan card and target deep learning. I am trying to understood how its useful and what types of network will benefit from this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that FP16 instead of FP32 is what should be useful for DL, but not sure how int8 could do. There are some research that you can train with full FP32 precision and then round it to one byte - but this does not speedup or reduce memory footprint for training. &lt;/p&gt;&#xA;" OwnerUserId="21799" LastActivityDate="2019-05-09T11:43:31.567" Title="How Int8 (byte) operations can be useful for deep learning?" Tags="|deep-learning|tensorflow|theano|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12987" PostTypeId="1" CreationDate="2016-07-25T20:20:58.883" Score="2" ViewCount="3041" Body="&lt;p&gt;i have 206 data frames named after country(FRANCE,GERMANY,HONGKONG...) with same 37 variables in these 206 files and another data frame name pf with 9 variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i want to do left join of these 206 data frames with PF data frame one by one selecting all variables and store in other data frame. left join is based on two condition( hcom_id=tnow_id and city=region). hcom_id and city are variables in 206 different data frames and tnow_id and region are in PF   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i am using sqldf package to join:&#xA;a&amp;lt;-sqldf(&quot;select * from FRANCE left join PF on (FRANCE.hcom_id=pf.tnow_id and FRANCE.city=pf.region)&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am struggling to automate it using loop rather than doing 206 times separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea please suggest &lt;/p&gt;&#xA;" OwnerUserId="10416" LastEditorUserId="10416" LastEditDate="2016-07-26T06:32:59.187" LastActivityDate="2017-07-26T19:48:40.380" Title="How to automate left join of multiple data frames with single data frame one by one in R" Tags="|r|programming|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12989" PostTypeId="2" ParentId="12874" CreationDate="2016-07-25T21:57:16.507" Score="1" Body="&lt;p&gt;Strangely, once the columns of each block of 5 data entries were normalized, with null rows included, the accuracy jumped to &gt;99%. Again the outcome is included in the data set but this seemed to correct the issue with this sanity check. &lt;/p&gt;&#xA;" OwnerUserId="21464" LastActivityDate="2016-07-25T21:57:16.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12990" PostTypeId="2" ParentId="12987" CreationDate="2016-07-25T22:42:15.913" Score="0" Body="&lt;p&gt;It is tough for me to tell you a precise way I would do this without looking at the data frames.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the way I might approach this is would involve the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Load all 206 data frame;&lt;/li&gt;&#xA;&lt;li&gt;Rename all variables such that they begin with the data frames name (&lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/grep.html&quot; rel=&quot;nofollow&quot;&gt;sub&lt;/a&gt; + an apply or loop might suffice here);&lt;/li&gt;&#xA;&lt;li&gt;Append all data frames columns wise (ugly, I know);&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;https://rpubs.com/bradleyboehmke/data_wrangling&quot; rel=&quot;nofollow&quot;&gt;tidyr&lt;/a&gt; to reshape the data and separate the date in such way that I would be abble to joing it with the PF data frame;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This last step is a bit vague, but that is how I would approach it.&lt;/p&gt;&#xA;" OwnerUserId="21804" LastActivityDate="2016-07-25T22:42:15.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12992" PostTypeId="1" CreationDate="2016-07-26T04:21:33.760" Score="6" ViewCount="662" Body="&lt;p&gt;Pedro Domingos in  &lt;a href=&quot;https://books.google.com.sg/books/about/The_Master_Algorithm.html&quot;&gt;&quot;The Master Algorithm&quot;&lt;/a&gt; listed five tribes of machine learning algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Symbolists &lt;/li&gt;&#xA;&lt;li&gt;Connectionists &lt;/li&gt;&#xA;&lt;li&gt;Evolutionaries &lt;/li&gt;&#xA;&lt;li&gt;Bayesians&lt;/li&gt;&#xA;&lt;li&gt;Analogizers &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which category do &lt;a href=&quot;https://www.coursera.org/learn/probabilistic-graphical-models&quot;&gt;probabilistic graphical models&lt;/a&gt; fall under?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From wikipedia (&lt;a href=&quot;https://en.wikipedia.org/wiki/Graphical_model&quot;&gt;https://en.wikipedia.org/wiki/Graphical_model&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A graphical model or probabilistic graphical model (PGM) is a&#xA;  probabilistic model for which a graph expresses the conditional&#xA;  dependence structure between random variables. They are commonly used&#xA;  in probability theory, statistics—particularly Bayesian statistics—and&#xA;  machine learning.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In that case, would it be &quot;Bayesian&quot;? What about something like &lt;strong&gt;Conditional Random Field&lt;/strong&gt;? Is that Bayesian as well?&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2021-02-08T16:35:15.073" Title="Which tribe does Probabilistic Graphical Models fall under?" Tags="|machine-learning|bayesian|graphical-model|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12993" PostTypeId="1" CreationDate="2016-07-26T04:50:27.877" Score="2" ViewCount="78" Body="&lt;p&gt;Assume I have a vector of displacements, when I calculate the derivative I obtain velocity. The problem is, there are so many noisy points and I am looking for one number as a velocity. What do you think is the best technique to obtain the velocity ? I am taking derivative and then take the mean, this gives me good result when there is not much noise however this technique is not really robust to noise also taking the median is not giving good result. Can anyone suggest a better technique to handle this problem?&lt;/p&gt;&#xA;" OwnerUserId="21810" LastEditorUserId="14904" LastEditDate="2016-07-26T08:17:22.933" LastActivityDate="2016-07-26T08:17:22.933" Title="extracting information in presence of noise" Tags="|data-mining|statistics|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12994" PostTypeId="2" ParentId="12992" CreationDate="2016-07-26T05:29:24.563" Score="2" Body="&lt;p&gt;Only Domingos knows for sure, since he invented this taxonomy, but I'd guess it would fall under &quot;connectionists&quot; (which he associates with neural networks), since graphs are all about connections (between random variables). Bayesians would be my second choice. CRFs are not natively Bayesian (you don't use priors or posteriors of the model parameters), but they can be &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/minka/papers/bcrf/Qi-Bayesian-CRF-AIstat05.pdf&quot; rel=&quot;nofollow&quot;&gt;augmented into one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-26T05:29:24.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12995" PostTypeId="2" ParentId="12992" CreationDate="2016-07-26T05:31:34.533" Score="2" Body="&lt;p&gt;Based on &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/minka/papers/bcrf/Qi-Bayesian-CRF-AIstat05.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; (Qi, Szummer, and Minka. 2005. Bayesian Conditional Random Fields):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this paper, we propose Bayesian Conditional Random Fields (BCRF), a novel Bayesian approach to training and inference for conditional random fields&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We can see that the original CRF model is presumably not Bayesian, since this paper's contribution is a novel Bayesian approach to the CRF model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on &lt;a href=&quot;http://www.salon.com/2015/10/10/an_algorithm_might_save_your_life_how_the_amazon_and_netflix_method_might_someday_cure_cancer/&quot; rel=&quot;nofollow&quot;&gt;this excerpt of the book&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Each of the five tribes of machine learning has its own master algorithm, a general-purpose learner that you can in principle use to discover knowledge from data in any domain. The symbolists’ master algorithm is inverse deduction, the connectionists’ is backpropagation, the evolutionaries’ is genetic programming, the Bayesians’ is Bayesian inference, and the analogizers’ is the support vector machine.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It seems that CRF can be put into connectionists (since CRF calculates optimizes a function based on the gradients) or analogizers (since CRF uses &quot;soft-examples&quot; calculated by softmax to separate the good output and bad outputs, as compared to the support vectors in (structured) SVM calculated by max. For example, &lt;a href=&quot;https://www.lti.cs.cmu.edu/sites/default/files/cmulti10008.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; shows that the difference CRF and SSVM are the addition of cost function and change from softmax to max)&lt;/p&gt;&#xA;" OwnerUserId="1177" LastActivityDate="2016-07-26T05:31:34.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="12996" PostTypeId="2" ParentId="12992" CreationDate="2016-07-26T06:34:31.073" Score="8" Body="&lt;p&gt;Probabilistic Graphical Models (PGMs) are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Connectionist: RBMs are PGMs and neural networks (&lt;a href=&quot;http://image.diku.dk/igel/paper/AItRBM-proof.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Bayesian: Bayes Networks are bayesian (&lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia article&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Symbolist: Markov Logic Networks (&lt;a href=&quot;http://link.springer.com/article/10.1007/s10994-006-5833-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Analogizers and Evolutionaries: &lt;a href=&quot;http://www.kdnuggets.com/2015/11/domingos-5-tribes-machine-learning-acm-webinar.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;According to Domingos&lt;/a&gt;, they are also in Markov Logic Networks.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;So the answer is that you can't simply categorize such a general technique as probabilistic graphical models in a single one of those categories.&lt;/p&gt;&#xA;&lt;p&gt;See also: &lt;a href=&quot;https://www.youtube.com/watch?v=E8rOVwKQ5-8&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;The Five Tribes of Machine Learning (And What You Can Learn from Each),&amp;quot; Pedro Domingos&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="85045" LastEditDate="2021-02-08T16:35:15.073" LastActivityDate="2021-02-08T16:35:15.073" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="12997" PostTypeId="2" ParentId="12993" CreationDate="2016-07-26T06:38:33.247" Score="1" Body="&lt;p&gt;Usually removing noise is a process depends on the type of noise. &#xA;Noises can be high or low frequency , you are using low frequency filters ( average and median) . &#xA;1. You did not describe the source of noise, we can not tell if it is a high frequency or low frequency . &#xA;2. Try to use a higher order low pass filter &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-07-26T06:38:33.247" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="12999" PostTypeId="1" AcceptedAnswerId="13001" CreationDate="2016-07-26T08:37:19.570" Score="1" ViewCount="3785" Body="&lt;p&gt;I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier.&lt;/p&gt;&#xA;&lt;p&gt;This is what my data looks like (pandas dataframe):&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Label  Feat1  Feat2  Feat3  Feat4&#xA;&#xA;0        1     3       2      1&#xA;1        0     1       1      2&#xA;2        2     2       1      1&#xA;3        3     3       2      3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have split the data into dataLabel and dataFeatures.&#xA;Prepared dataLabel using &lt;code&gt;dataLabel.ravel()&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;I need to discretize features so the classifiers treat them as being categorical not numerical.&lt;/p&gt;&#xA;&lt;p&gt;I’m trying to do this using &lt;code&gt;OneHotEncoder&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;enc = OneHotEncoder()&#xA;&#xA;enc.fit(dataFeatures)&#xA;chk = enc.transform(dataFeatures)&#xA;from sklearn.naive_bayes import MultinomialNB&#xA;&#xA;mnb = MultinomialNB()&#xA;&#xA;from sklearn import metrics&#xA;from sklearn.cross_validation import cross_val_score&#xA;scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I get this error: &lt;code&gt;bad input shape (64, 16)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;This is the shape of label and input:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;dataLabel.shape = 72&lt;/code&gt;&#xA;&lt;code&gt;chk.shape = 72,16&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;Why won't the classifier accept the onehotencoded features?&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: Adding how I got dataFeatures&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;dataFeatures = data[['Accpred', 'Gyrpred', 'Barpred', 'altpred']]&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Y = dataLabel.ravel()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="17772" LastEditorUserId="85045" LastEditDate="2021-01-05T16:04:22.570" LastActivityDate="2021-01-05T16:04:22.570" Title="Scikit Learn OneHotEncoded Features causing error in classifier" Tags="|python|scikit-learn|predictive-modeling|categorical-data|dataframe|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13000" PostTypeId="2" ParentId="12984" CreationDate="2016-07-26T11:05:06.483" Score="10" Body="&lt;p&gt;There is no definite source on how to do feature engineering. It is often dependent on the problem you are trying to solve. Some say it is more of an art than it is science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I would go through some of the high scoring kaggle kernels / winning solutions if available. Just head over to &lt;a href=&quot;https://www.kaggle.com&quot; rel=&quot;noreferrer&quot;&gt;kaggle&lt;/a&gt; and browse through the competitions. There is a lot of very useful material in there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the journal of machine learning research has a lot of papers about feature engineering. Just search on their site &lt;a href=&quot;http://www.jmlr.org/&quot; rel=&quot;noreferrer&quot;&gt;http://www.jmlr.org/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following links are useful and to long to paraphrase:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Some information about some best practices of feature engineering can&#xA;be found on Quora, see &lt;a href=&quot;https://www.quora.com/What-are-some-best-practices-in-Feature-Engineering&quot; rel=&quot;noreferrer&quot;&gt;this&#xA;link&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;At machine learning mastery there is &lt;a href=&quot;http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&quot; rel=&quot;noreferrer&quot;&gt;this&#xA;page&lt;/a&gt;&#xA;about feature engineering.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-07-26T11:05:06.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13001" PostTypeId="2" ParentId="12999" CreationDate="2016-07-26T13:04:10.740" Score="1" Body="&lt;pre&gt;&lt;code&gt;scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have your &lt;code&gt;Y&lt;/code&gt; and &lt;code&gt;chk&lt;/code&gt; switched. That's it. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The signature of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.cross_val_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;cross_val_score&lt;/code&gt;&lt;/a&gt; is &lt;code&gt;sklearn.cross_validation.cross_val_score(estimator, X, y)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;X&lt;/code&gt; is a matrix and &lt;code&gt;y&lt;/code&gt; is a 1D vector with your class labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;unlike in R, most (or all?) sklearn models do not support categorical variables. Most of the time, encoding your feature matrix &lt;code&gt;X&lt;/code&gt; into what is called &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot encoding&lt;/a&gt; is good enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that, in some models, this hack is not the same as true native categorical support, and the performance of the model will be worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Invert One-Hot Encoding&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sklearn does not seem to have an easy method to invert the one-hot encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not trivial how to do this. I found &lt;a href=&quot;https://stackoverflow.com/a/35898452/2680707&quot;&gt;this suggestion&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def inverse(enc, out, shape):&#xA;    return np.array([enc.active_features_[col] for col in out.sorted_indices().indices]).reshape(shape) - enc.feature_indices_[:-1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.preprocessing import OneHotEncoder&#xA;enc = OneHotEncoder()&#xA;X = np.array([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]))&#xA;Z = enc.fit_transform(X)&#xA;print(inverse(enc, Z, X.shape))&#xA;# [[0 0 3]&#xA;#  [1 1 0]&#xA;#  [0 2 1]&#xA;#  [1 0 2]]&#xA;print(X)&#xA;# [[0 0 3]&#xA;#  [1 1 0]&#xA;#  [0 2 1]&#xA;#  [1 0 2]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This only works when &lt;code&gt;HotOneEncoding(sparse=True)&lt;/code&gt; (default) because it uses scipy sparse matrix methods (this could be changed by making the code only use numpy methods), but this is probably what you want since working with a dense matrix will kill your memory anyhow&lt;/li&gt;&#xA;&lt;li&gt;I think this will only work if your variables are within the range [0,something] because you lose that information in the transformation (no work-around for this other than you using something like &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DictVectorizer&lt;/a&gt; which offers you more control over the transformation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-26T15:30:05.520" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13002" PostTypeId="1" AcceptedAnswerId="13004" CreationDate="2016-07-26T14:21:19.897" Score="8" ViewCount="10331" Body="&lt;p&gt;I have a jupyter notebook and I would like to perform several runs, the code I want to run depending on the results of the previous runs. I divided my notebook into several cells, and for each run I would like to select which cell should be executed and which one should not. Is there a functionality which looks like &quot;run all cells except those I explicitly deactivate&quot;?&lt;/p&gt;&#xA;" OwnerUserId="21825" LastActivityDate="2022-11-03T21:19:54.910" Title="Is it possible to (de)activate a specific set of cells in jupyter?" Tags="|jupyter|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13004" PostTypeId="2" ParentId="13002" CreationDate="2016-07-26T17:41:05.693" Score="9" Body="&lt;p&gt;Welcome to DataScience.SE! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not currently possible. You could change the cells to &lt;em&gt;Raw&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-26T17:41:05.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13005" PostTypeId="1" CreationDate="2016-07-26T17:54:46.293" Score="2" ViewCount="125" Body="&lt;p&gt;I want to know the individual contribution of each salesperson in terms of actual sales where the dataset I have aggregates sales data by sales &quot;team&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Month | # of Sales | SalesPerson1 | SalesPerson2 | SalesPerson3&#xA;1 | 25 | Fred | Susan | Tom&#xA;1 | 12 | John | Wayne | Connor&#xA;2 | 16 | Fred | John | Wayne&#xA;2 | 11 | Susan | Tom | Winona&#xA;3 | 34 | Connor | Susan | Tom&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to predict the individual contribution of any given salesperson based on the data above where the membership of any given sales team can change from one month to the next?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to say that for any given sales person, they close x # of sales per month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21829" LastActivityDate="2016-07-28T17:27:20.773" Title="How to determine individual contribution when data is aggregated by team" Tags="|machine-learning|probability|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13006" PostTypeId="2" ParentId="12900" CreationDate="2016-07-26T18:40:34.603" Score="4" Body="&lt;p&gt;I have not worked as a data scientist.  But managing expectations is something we've all done. Considering you're referring to the company CEO, tact is especially required.  There are three practices that have helped me a lot over the years:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Never say &quot;it can't be done&quot; right away.  Many, many, many times I have said that (as a programmer), only to think of a solution later on.  When I'm skeptical that something can't be done, I've learned to say, &quot;Can I think about it and get back to you?&quot;  That approach has kept egg off my face many-a-time.  It also gives me time to do some research.&lt;/li&gt;&#xA;&lt;li&gt;Even when something can't be done, focus on what you &lt;em&gt;can&lt;/em&gt; provide. If the requested solution is unrealistic for some reason, try to think of what is realistic and offer that.&lt;/li&gt;&#xA;&lt;li&gt;When discussing what can and cannot be realistically done, clients often don't care about the technical details (even though that's what excites you, and what you want to talk about).  Be ready to go into the technical details if asked, but start off with a very high level explanation.  Rather than focus on the negative details of why something cannot be done, describe the positive details of &lt;em&gt;what it would require&lt;/em&gt; to accomplish the task. That shows you have thought it through, and that you're trying to be team player rather than a road block.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="19170" LastActivityDate="2016-07-26T18:40:34.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13010" PostTypeId="1" CreationDate="2016-07-26T21:09:57.130" Score="1" ViewCount="3020" Body="&lt;p&gt;I want to know which language/packages performs better &amp;amp; faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and libraries performs the best on wrangling and cleaning Big datasets?&lt;/p&gt;&#xA;" OwnerUserId="20290" LastActivityDate="2016-07-29T22:41:30.470" Title="Which one is better performer on wrangling big data, R or Python?" Tags="|data-mining|bigdata|data-cleaning|data|data-wrangling|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13011" PostTypeId="1" CreationDate="2016-07-26T21:27:13.647" Score="4" ViewCount="475" Body="&lt;p&gt;I am interested in proposed base R/Python solutions (i.e. without relying on external packages/libraries) to the following problem:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You have a large dictionary containing millions of valid English&#xA;  words. You are given an input “word”, which may or may not be an&#xA;  anagram of one or more words in the dictionary. Find all possible&#xA;  valid words in the dictionary of which the input may be an anagram.&#xA;  Note, the input “word” may be meaningless. For example,&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;input: lloeh output: hello&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;input: cksli output: slick, licks&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Rather than use a brute force technique that computes all possible&#xA;  character combinations for each input target and compares them to a&#xA;  dictionary, you want to use a more efficient technique.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The approach I was considering was to pre-compute impossible string onsets in the target language. For example, in English, the onset sequences ^ck and ^ng are not permissable because they violate the orthographic and phonotactic constraints of the language. Given an off-line list of impossible_sequences at run time you would incrementally build possible &quot;words&quot; from the target. During each iteration the possible word is checked against the impossible_sequences list and if it matches the search&#xA;down that search branch is terminated. The following pseudo-code in R uses &lt;code&gt;for&lt;/code&gt; loops to illustrate my idea.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;target &amp;lt;- &quot;cksli&quot;&#xA;impossible_list &amp;lt;- c(&quot;ck&quot;, &quot;ng&quot;, &quot;rt&quot;, &quot;zz&quot;) # pre-computed not-possible English onsets&#xA;&#xA;for (i in 1:nchar(target)) {&#xA;  word &amp;lt;- substr(target, i, i)&#xA;  if (word %in% impossible_list) break&#xA;  for (j in 2:nchar(target)) {&#xA;    word &amp;lt;- paste0(word, substr(target, j, j))&#xA;    if (word %in% impossible_list) break&#xA;    for (k in 3:nchar(target)) {&#xA;      ### same logic&#xA;      ### goes deeper and deeper&#xA;    }&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(1) Are there any issues with the underlying logic of my proposal?&#xA;(2) Obviously nested for loops are inelegant - Could one do this recursively and if so, how?&#xA;(3) What other approaches should be considered?&lt;/p&gt;&#xA;" OwnerUserId="6478" LastEditorUserId="6478" LastEditDate="2016-07-26T22:47:31.397" LastActivityDate="2018-02-26T16:42:49.363" Title="Non-brute force approach to finding permissible English word anagrams" Tags="|python|r|nlp|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13012" PostTypeId="2" ParentId="13010" CreationDate="2016-07-26T21:48:22.307" Score="3" Body="&lt;p&gt;I think this may help you:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/41/is-the-r-language-suitable-for-big-data?rq=1&quot;&gt;Is the R language suitable for Big Data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it really depends on what you are comfortable with and what your objectives are.  In Python, I can execute SQL queries directly into Pandas data frames.  From there, data cleaning and visualizing (I like seaborn) is fairly straightforward.  I can easily perform matrix manipulations using Numpy, and the sklearn is a well documented ML package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, I still like R and there are definitely times when I will use R.  Learn both!&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-26T21:48:22.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13013" PostTypeId="2" ParentId="13010" CreationDate="2016-07-26T22:13:25.113" Score="3" Body="&lt;p&gt;It really depends on what you mean by 'big data'.  A truly big dataset cannot fit in memory, in which case local python and R really only work for smaller scale experimentation and prototyping.  For the purpose of data wrangling, you'll want a picture of the whole dataset by either slicing based on cuts, sampling, or aggregation.  In any case, you'll need to work on a distributed computational platform.  In my opinion, python has a big leg up in this regard, as its interface with Apache Spark is quite robust (whereas the R interface seems to be wanting).  With that said, if you are skillful with impala, pig, or hive, you can do partitioning in a distributed query language, and create data that can be examined locally.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-26T22:13:25.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13014" PostTypeId="2" ParentId="12958" CreationDate="2016-07-26T22:24:13.950" Score="1" Body="&lt;p&gt;Clustering techniques at high dimensionality tend to be unstable/inaccurate.  As such, if you are able to accurately classify one column based on others, then you are well suited with pursuing a dimensionality reduction technique such as Non-negative matrix factorization.  In this way, you can simply remove the column from all data (as the other columns hold the same information), and then cluster on the reduced space.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-26T22:24:13.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13015" PostTypeId="2" ParentId="12841" CreationDate="2016-07-26T22:24:16.940" Score="0" Body="&lt;p&gt;When $X_1$ is unobserved, at iteration $k=1$ of EM, the posterior mean value (when $X_2=3$) is $5.18$ by using an inference algorithm, i.e. Junction tree/kalman filter. &#xA;Then the sufficient statistics for $X_1$ is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s_1=\Sigma_{i=1}^nx_{i1}=9+4+5.18$ and $s_{11}=\Sigma_{i=1}^nx_{i1}^2=9^2+4^2+(5.18^2+\sigma_{11.2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\sigma_{11.2}$ is the posterior conditional variance of $X_1$ given $X_2$, when $X_1$ is unobserved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s_2=\Sigma_{i=1}^nx_{i2}$ and $s_{22}=\Sigma_{i=1}^nx_{i2}^2$. When $X_2$ is unobserved a term $\sigma_{22.1}$ is also added to $x_{i2}^2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So parameters $\mu_{X_1}$ and $\sigma_{X_1}$ are obtained. Similarly, posterior mean of unobserved $X_2$ is also obtained by using i.e. Junction tree inference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sufficient statistics for $s_{12}=\Sigma_{i=1}^nx_{i1}x_{i2}$, and $s_{21}=\Sigma_{i=1}^nx_{i2}x_{i1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mean vector is then: $\mu^{(k)}=(s_1/n, s_2/n)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Covariance matrix is: $\Sigma^{(k)}=$$\left( \begin{matrix}&#xA;   \Sigma _{11} &amp;amp; \Sigma _{12}  \\&#xA;   \Sigma _{21} &amp;amp; \Sigma _{22}  \\&#xA;\end{matrix} \right)$$=$$\left( \begin{matrix}&#xA;   s_{11}/n-(s_1/n)^2 &amp;amp; s_{21}/n-(s_2/n)(s_1/n)  \\&#xA;   s_{12}/n-(s_1/n)(s_2/n) &amp;amp; s_{22}/n-(s_2/n)^2  \\&#xA;\end{matrix} \right)$$&#xA; $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other parameters in the question are obtained. Conditional mean and conditional variance of $X_2|X_1$ is calculated by the formula in the reference of the question.&lt;/p&gt;&#xA;" OwnerUserId="21565" LastEditorUserId="21565" LastEditDate="2016-07-27T10:20:01.633" LastActivityDate="2016-07-27T10:20:01.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13016" PostTypeId="1" CreationDate="2016-07-26T23:22:12.340" Score="1" ViewCount="49" Body="&lt;p&gt;I am new at deep learning, but willing to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have 5 inputs: A,B,C,D,E(columns in CSV) that gives me 1 or 0.&#xA;A,B,C,D or E can be between 0 and 100.&#xA;For example, a combination of values of these letters will give me 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A  | B | C  | D | E&#xA;20, 61, 42, 90, 3 will give me 1&#xA;20, 31, 2, 61, 20 will give me 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Idea is this if A is up to 20 or less, B is up to 61 or less, C is 42 or less, D is &#xA;90 or more and E is 3 or more, it will give me 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem number one is next, I need to find boundaries(Max and Min values) for these numbers. I do not know which column should be Max or Min. So I need to find these boundaries that will give me the smallest mean error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem number two, this is more like next step, sometimes, some of the columns just do not apply to give 0 or 1, they are just relevant, so I need to find relevant columns, and set for example that column to be 100 or less, or 0 or more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Except writing python code to load CSV, and some research regarding Logistic and Linear regressions, and other algorithms like Nearest Neiber I am kinda stuck, need advice in which direction to go. Any sample math, the code would be amazing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope I have explain it good, really sorry if it is not, please let me know, and I will try to make it clearer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="21834" LastEditorUserId="21834" LastEditDate="2016-07-27T00:06:41.350" LastActivityDate="2016-07-27T00:06:41.350" Title="Find boundaries for a smallest mean error" Tags="|classification|neural-network|deep-learning|theano|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13017" PostTypeId="2" ParentId="13011" CreationDate="2016-07-27T01:49:31.893" Score="2" Body="&lt;p&gt;A simple solution is store the words in a dictionary (since you have to store them in some data structure) whose key is the character distribution; e.g., Counter(&quot;hello&quot;) = {h: 1, e: 1, l: 2, o: 1}. Querying the dictionary would give you the anagrams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For storing the immutable key (character distribution) can either use a tuple list, incurring a sort, or you can use a vector the length of your alphabet (26). So you can make a speed-space trade-off in the preparation stage; the look-ups are constant time, not counting the time it takes to calculate the character distribution of the query word. If you go the latter, fixed-width route, you can make another trade-off by hashing the key, since you know the cardinality of the input (number of unique words).&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-07-28T04:35:47.390" LastActivityDate="2016-07-28T04:35:47.390" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13018" PostTypeId="1" AcceptedAnswerId="13020" CreationDate="2016-07-27T03:44:44.910" Score="2" ViewCount="226" Body="&lt;p&gt;For the single variate data sets, we can use some straightforward methods, such as box plot or [5%, 95%] quantile to identify outliers. For multivariate data sets, are there any statistics that can be used to identify outliers?&lt;/p&gt;&#xA;" OwnerUserId="20978" LastActivityDate="2016-07-27T05:04:30.453" Title="statistics or robust statistics for identifying multivariate outliers" Tags="|machine-learning|data-mining|statistics|anomaly-detection|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13019" PostTypeId="2" ParentId="9885" CreationDate="2016-07-27T04:49:38.697" Score="4" Body="&lt;p&gt;Well, many months have been passed from this help request. I write this answer to my own request in order to share my experience. I learned &lt;code&gt;ggplot2&lt;/code&gt; and then &lt;code&gt;ggvis&lt;/code&gt; as well as &lt;code&gt;Shiny&lt;/code&gt;. &lt;code&gt;Shiny&lt;/code&gt; can work with both of them, but I found &lt;code&gt;ggvis&lt;/code&gt; more structured and lucid comparing to &lt;code&gt;ggplot2&lt;/code&gt;. Something which is expectable as the former is being developed based on the development experience of the latter. Moreover, &lt;code&gt;ggvis&lt;/code&gt; seems much faster than &lt;code&gt;ggplot2&lt;/code&gt; in quick graph/chart generation, it seems much more suitable for 'Shiny' and creating an applet. However, the negative point is the lack of many features in &lt;code&gt;ggvis&lt;/code&gt; since the package is under development and has not become matured yet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And many thanks to comments here. I have checked &lt;code&gt;Gephi&lt;/code&gt;, it wouldn't help. I created my applet which works really fine, however I finally reached to this point that I need to use more powerful tools such as &lt;code&gt;d3js&lt;/code&gt;, as one has already suggested. &lt;/p&gt;&#xA;" OwnerUserId="15604" LastActivityDate="2016-07-27T04:49:38.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13020" PostTypeId="2" ParentId="13018" CreationDate="2016-07-27T04:59:11.480" Score="2" Body="&lt;p&gt;Multivariate outlier detection can be quite tricky and even 2D data can be difficult to visually decipher at times.  You are spot-on in looking for robust statistical treatments analogous to 95% quantiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where as normally distributed data naturally aligns with the chi square distribution, the gold standard for robust statistics in n dimensions would be to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Mahalanobis_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Mahalanobis distances&lt;/code&gt;&lt;/a&gt; and then eliminate data beyond 95% or 99% quantiles in Mahalanobis space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plug and play capabilities are available in &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/covariance/plot_mahalanobis_distances.html#example-covariance-plot-mahalanobis-distances-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; and in &lt;a href=&quot;https://cran.r-project.org/web/packages/mvoutlier/mvoutlier.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an &lt;a href=&quot;http://www.dtic.mil/dtic/tr/fulltext/u2/a545834.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;excellent theoretical and practical treatment of the methodology&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NKBfd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NKBfd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is a &lt;a href=&quot;https://stat.ethz.ch/education/semesters/ss2012/ams/slides/v2.2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;big picture viewpoint with some heuristics&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally there is a very sophisticated treatments called &lt;code&gt;PCOUT&lt;/code&gt; for outlier detection that instead rely on principal component decomposition.  There is a &lt;a href=&quot;http://www.inside-r.org/packages/cran/mvoutlier/docs/pcout&quot; rel=&quot;nofollow noreferrer&quot;&gt;corresponding R package&lt;/a&gt;, but the theoretical treatment is behind a paywall:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1315052&quot; rel=&quot;nofollow noreferrer&quot;&gt;P. Filzmoser, R. Maronna, M. Werner. Outlier identification in high dimensions, Computational Statistics and Data Analysis, 52, 1694-1711, 2008&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="9420" LastEditDate="2016-07-27T05:04:30.453" LastActivityDate="2016-07-27T05:04:30.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13023" PostTypeId="2" ParentId="13011" CreationDate="2016-07-27T07:16:16.407" Score="1" Body="&lt;p&gt;I think, it would be hard to find a solution without external packages at all. Internally you can create an index, that can be used for efficient search, but I can't remember any &quot;pure-Python/R&quot; data structure with non-unique index search. Dictionary wouldn't fit, because all anagrams will have the same key, that is impossible for dict. And there is no native binary tree search implementation both in Python and R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a sketch of a possible solution, I would use. Let's create decimal hash to determine letters, a word consists of (Python):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def word2hash(word):&#xA;    alphabet = list(map(chr, range(ord('a'), ord('z')+1)))&#xA;    hashed_word = {k: 0 for k in alphabet}&#xA;    for letter in set(word.lower()):&#xA;        hashed_word[letter] = 1&#xA;    hashed_word = sorted(hashed_word.items(), key=lambda x: alphabet.index(x[0]))&#xA;    hashed_word = ''.join([str(x[1]) for x in hashed_word])&#xA;    return int(hashed_word, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function would produce a decimal hash, based on letters. This hash could be used for efficient binary tree search (BTS). But the collision is that it doesn't take into account double letters. E.g., both for 'hello' and 'ooohell' output would be the same: 2377728. Anyway, it's fast and robust to filter out most part of your dictionary. There are several options of implementing efficient search, based on this index:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SQL database: non-unique indexes supported out-of-the-box, but you need an SQL-interface library (&lt;a href=&quot;https://pypi.python.org/pypi/MySQL-python&quot; rel=&quot;nofollow&quot;&gt;python-mysqldb&lt;/a&gt;/&lt;a href=&quot;https://cran.r-project.org/web/packages/RMySQL/index.html&quot; rel=&quot;nofollow&quot;&gt;RMySQL&lt;/a&gt;/etc). As for me, this is the best option. Just put your dictionary with precomputed decimal indexes in a database, and query it as much as you want;&lt;/li&gt;&#xA;&lt;li&gt;Any Python/R package with BTS: (e.g., &lt;a href=&quot;https://cran.r-project.org/web/packages/data.tree/vignettes/data.tree.html&quot; rel=&quot;nofollow&quot;&gt;data.tree&lt;/a&gt;);&lt;/li&gt;&#xA;&lt;li&gt;Implement BTS on your own without external dependencies.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the second step let's use more precise tuple-hash. It's not so simple, as previous one, but it's input is much smaller:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def word2letterset(word):&#xA;    alphabet = list(map(chr, range(ord('a'), ord('z')+1)))&#xA;    letterset = {}&#xA;    for letter in list(word):&#xA;        try:&#xA;            letterset[letter] += 1&#xA;        except KeyError:&#xA;            letterset[letter] = 1&#xA;    return tuple(letterset.items())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This step could be fully implemented in pure Python/R with &quot;brute force&quot;: compute tuple-hash for an input &quot;word&quot;, and compare it with computed hashes for all the words, found on previous step.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-07-27T07:16:16.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13026" PostTypeId="1" CreationDate="2016-07-27T12:03:57.123" Score="-1" ViewCount="220" Body="&lt;p&gt;I am training a random forest model. I am wondering if it is safe (leakage?) to use on my training set the parameter &lt;strong&gt;average price of a car&lt;/strong&gt; calculated using all my data points. The issue is that some of these data are used to test my model as well. I try to give an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, the average for that car model is composed of 5 cars and it is 10,000 EUR. I used 3 of these cars to train my random forest. For each car, I provide (besides other parameters) the average price of all cars (10,000) and the price of that particular car (say 8,000) as input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my testing, I would still provide the price of all cars (10,000) and the price of each particular car (13,000) as input. I want to know if that's leakage, since the 13,000 (now on the testing set) was also used to obtain the 10,000 average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it is clear.&lt;/p&gt;&#xA;" OwnerUserId="21451" LastActivityDate="2017-04-06T22:17:10.847" Title="Avoiding leakage on my random forest?" Tags="|random-forest|parameter|experiments|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13027" PostTypeId="2" ParentId="1192" CreationDate="2016-07-27T12:24:55.347" Score="0" Body="&lt;p&gt;If you are talking about just making the names of the variables consistent across all of the data sets to make them easier to work with, you are dealing with what is known as &quot;Data Integration&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not familiar with SPSS. The base SAS language has data management features/options that lend to performing tricks specific to your data which can semi-automate this kind of process. SAS can also very easily import data from a SPSS data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specialty software exists for this kind of thing as well. So called data integration tools made by SAP and Informatica are the first that come to mind. I'm not sure if IBM makes anything similar. Might be worth your while to explore this route depending on your project budget.&lt;/p&gt;&#xA;" OwnerUserId="17511" LastActivityDate="2016-07-27T12:24:55.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13028" PostTypeId="1" CreationDate="2016-07-27T14:21:52.663" Score="1" ViewCount="582" Body="&lt;p&gt;I'm currently working on a project where we're building a data pipeline. We have spark setup and have generated models. Sadly, loading the model in Spark and querying it isn't fast enough for us. What's the most straightforward way of exporting a model and loading it into memory on a local server? I've researched PMML and some libraries and that appears to be one path. &lt;/p&gt;&#xA;" OwnerUserId="21853" LastActivityDate="2016-08-17T10:40:19.737" Title="Loading and querying a Spark machine learning model outside of Spark" Tags="|apache-spark|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13029" PostTypeId="1" CreationDate="2016-07-27T14:35:15.043" Score="7" ViewCount="14061" Body="&lt;p&gt;There are various algorithms for reinforcment learning (RL). One way to group them is by &quot;off-policy&quot; and &quot;on-policy&quot;. I've heard that SARSA is on-policy, while Q-Learning is off-policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think they work as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1CB4h.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1CB4h.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vJKzA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vJKzA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How exactly is &quot;on-policy RL&quot; and &quot;off-policy RL&quot; defined?&lt;/li&gt;&#xA;&lt;li&gt;What are the advantages / disadvantages of both?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-07-27T15:07:10.203" LastActivityDate="2016-12-22T16:12:05.983" Title="What are the advantages / disadvantages of off-policy RL vs on-policy RL?" Tags="|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13030" PostTypeId="1" CreationDate="2016-07-27T15:38:08.720" Score="4" ViewCount="2790" Body="&lt;p&gt;how much data does &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html&quot; rel=&quot;nofollow&quot;&gt;word2vec&lt;/a&gt; require?  Are there any public data sets that are useful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, could it be that 1000 newspaper articles are enough to use word2vec?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a word2vec tutorial from Kaggle that uses &lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors&quot; rel=&quot;nofollow&quot;&gt;50,000 movie reviews&lt;/a&gt;.  I am trying to understand the scale of the word2vec input.&lt;/p&gt;&#xA;" OwnerUserId="8956" LastActivityDate="2018-09-01T16:34:36.800" Title="how much text data is required for a meaningful use of word2vec" Tags="|machine-learning|neural-network|word2vec|language-model|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13031" PostTypeId="1" AcceptedAnswerId="13037" CreationDate="2016-07-27T15:42:23.953" Score="4" ViewCount="7183" Body="&lt;p&gt;I have learned regularization for linear and logistic regression but when I implement that algorithm to my code generally my estimates not changing.I mean,it looks like ineffective.I know,it's for overfitting.So if I use it in my code every time ,could it be a problem? or is this a good thing?&lt;/p&gt;&#xA;" OwnerUserId="20926" LastActivityDate="2021-01-31T09:44:21.730" Title="Should I use regularization every time?" Tags="|machine-learning|logistic-regression|linear-regression|regularization|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13032" PostTypeId="1" CreationDate="2016-07-27T15:46:50.317" Score="0" ViewCount="5795" Body="&lt;p&gt;I am trying to solve the following classification problem: one has a large (~2-10 GB) .csv file containing a data frame of various types. In particular, there are columns containing numeric floats as well as date/times, text strings and discrete parameters from a small set. There is also a binary parameter classifying each entry which one wants to predict in the future using the machine learning approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what would be the most appropriate tool for this job? After some research I settled on scikit-learn as the main option, but if there is other software radically better suited for the task, I'd be happy to switch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, is the random forest approach the best way to go here? I realise there are probably many factors influencing the answer to this question, but from playing around with a few tutorials and looking at the scikit-learn flowchart I am currently leaning towards random forests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming one uses random forests, how exactly one should prepare these data in order for the model to make sense of it? From what I understand, scikit-learn takes only numerical floats as inputs for the model. While I could discretise date/time and other discrete fields with something like OneHotEncoder, it is not immediately clear what to do with text fields since the processing, I suppose, is very different. (I have looked at the bag-of-words approach, but being someone new to the field I fell it a bit hard to grasp right away.) Is there a meaningful way of transforming text fields for use in the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, what is the best way of dealing with large datasets for this task? While for the start their size could be limited to something fitting inside runtime memory, ideally there should be a way to extend the training set to be very large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies for the vagueness of some questions. Since I am quite new to data science in general, I would appreciate if you could include actual working examples in your answers. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21854" LastActivityDate="2017-11-11T17:50:46.640" Title="Applying random forest model to a dataframe with multiple types of data" Tags="|scikit-learn|random-forest|" AnswerCount="2" CommentCount="2" ClosedDate="2017-11-15T16:26:52.600" ContentLicense="CC BY-SA 3.0" />
  <row Id="13033" PostTypeId="1" CreationDate="2016-07-27T16:10:29.003" Score="2" ViewCount="600" Body="&lt;p&gt;I have a very big set of high-dimensional, but sparse binary vectors. Each vector represents a &quot;one-hot-style&quot; n-gram sequence of words where each index of the words that occur in the n-gram is set to 1, all others to 0. E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0,0,0,0,0,1,0,0,0,0,1,0,0,0,...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;An efficient representation for a 5-gram:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[5,10,235,1253,5521]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is generate such n-gram representations from a large text corpus. Hence, this may easily result in a vocabulary size of hundreds of thousands (depending on potential pre-processing steps including stopword removal, stemming, etc.) and even more distinct n-grams (&lt;code&gt;m&lt;/code&gt;).&#xA;This could be represented as a matrix of size &lt;code&gt;m x vocab_size&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Subsequently, I would like to reduce the number of rows by clustering similar n-grams together while retaining the contained words. For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[5,7,10,12,15],[5,8,10,15,18] -&amp;gt; [5,7,8,10,12,15,18]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My naive approach would be to apply some traditional clustering algorithm in order to assign each entry in the initial n-gram set to a specific cluster and then perform a union over all the n-grams assigned to a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1st question: what would be an sensible algorithm for clustering sparse binary representations?&#xA;2nd question: how could this be done with a very big data set?&lt;/p&gt;&#xA;" OwnerUserId="21858" LastActivityDate="2016-08-07T07:44:52.967" Title="Cluster very many sparse binary vectors" Tags="|clustering|bigdata|binary|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13034" PostTypeId="2" ParentId="13033" CreationDate="2016-07-27T17:28:15.067" Score="1" Body="&lt;p&gt;A few points.  First, if you're doing text analytics, there are methods to reduce the space, e.g. stop word filtering and word stemming.  You can also consider training a word2vec model, such as that provided by gensim: &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow&quot;&gt;https://radimrehurek.com/gensim/models/word2vec.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The particular strength of this method will be to take the space from O(10^7) to O(10^2) (but the vectors will become dense).  From there, you can either try approximate nearest neighbor, or cosine similarity.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-27T17:28:15.067" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13035" PostTypeId="2" ParentId="13031" CreationDate="2016-07-27T17:33:30.410" Score="0" Body="&lt;p&gt;Regularization is useful in preventing overtraining, however, if your regularization parameter is too aggressive, then it will dominate the learning function, and not allow you to properly tune variables.  Try reducing the regularization parameter. &lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-27T17:33:30.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13036" PostTypeId="2" ParentId="13030" CreationDate="2016-07-27T17:39:57.363" Score="3" Body="&lt;p&gt;As word2vec is a neural network, it benefits from very large datasets.  The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences.  As they note, they get approximately the same results using bag of words and word2vec.  One thing which is of note, since the review data comes from the internet, the sentences are much more loosely structured than what you would encounter in a corpus of newspapers, which typically go through grammatical review.  A great dataset for training word2vec on structured language is the wikipedia dataset:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Wikipedia:Database_download&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Wikipedia:Database_download&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-27T17:39:57.363" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13037" PostTypeId="2" ParentId="13031" CreationDate="2016-07-27T17:45:58.510" Score="6" Body="&lt;p&gt;Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the process, so you will have to approximate with a flexible enough model. If the model is not flexible enough you will not need to regularize but you won't approximate well anyway. If you use a more flexible model, you will get closer on average (low bias) but you will have more variance, thus the need for increased regularization. In other words, the amount of regularization you need depends on the model. This is related to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff&quot; rel=&quot;noreferrer&quot;&gt;bias-variance trade-off&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Welcome to DataScience.SE. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-27T17:45:58.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13039" PostTypeId="2" ParentId="13010" CreationDate="2016-07-27T17:58:14.653" Score="3" Body="&lt;p&gt;Wrong question. Big data is not a question of this or that language, but cluster computing. For me it's implicit in the definition; if you can find a way to process your data on your laptop it just isn't big data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark is the de facto standard today for cluster computing. It comes with many of its own munging primitives, borrowed from numerous languages (think dataframes and functional programming), and bindings to them. Scala is the best language in terms of Spark API coverage, followed by python, then R. If you want to experiment with Spark you can rent managed instances from Google on &lt;a href=&quot;https://cloud.google.com/dataproc/&quot; rel=&quot;nofollow&quot;&gt;DataProc&lt;/a&gt; or &lt;a href=&quot;http://spark.apache.org/docs/latest/spark-standalone.html&quot; rel=&quot;nofollow&quot;&gt;spin up your own&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-07-27T18:22:07.443" LastActivityDate="2016-07-27T18:22:07.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13040" PostTypeId="1" AcceptedAnswerId="13103" CreationDate="2016-07-27T17:59:09.400" Score="0" ViewCount="170" Body="&lt;p&gt;I'm an undergrad interested in machine learning, and I'm playing around with some data in order to get a better understanding of the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;DATA&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working with the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0QCyG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0QCyG.png&quot; alt=&quot;Regression plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an idea of what you're looking at (sorry for the unlabeled axes) -- each data point corresponds to a basketball player's points per game (y-values) over a one-week period (x-values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;END GOAL&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict progress throughout the year, so in order to test/train, I start with the first two points, fit a regression model, and then predict the third point. I then add the actual value of the third point to the set, re-train, and predict the fourth point, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ISSUES&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the data is all over the place, and the predictions are just as messy (sometimes getting up to ~200 points per game, which is totally impossible).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tested different degrees of linear regression (quadratic, cubic, etc.) and degree=1 is always the best predictor because of how wonky the data is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IDEAS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought of the following ways to get more accurate predictions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Smoothing the data, maybe using a moving average or some variant&lt;/li&gt;&#xA;&lt;li&gt;Set an upper limit for predictions &lt;/li&gt;&#xA;&lt;li&gt;Non-linear regression &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But outside of smoothing the data, I'm not sure if the rest are even possible in a regression model (upper limit) or applicable to this situation (non-linear).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUESTION&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are any of the ideas I had above worth pursuing? If not, is there anything I should look into that might help me solve this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21862" LastActivityDate="2016-07-30T14:23:26.987" Title="How to move forward on Regression problem" Tags="|machine-learning|regression|linear-regression|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13041" PostTypeId="1" CreationDate="2016-07-27T22:59:16.890" Score="3" ViewCount="2401" Body="&lt;p&gt;I have a dataset and need to predict, out of 9 variables, which ones matter most to predict number 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first tried using the &lt;code&gt;selectKBest&lt;/code&gt;method from &lt;code&gt;sklearn.feature_selection&lt;/code&gt; and it looks like 4,5,6 and 8 are best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I tried to look at linear correlation using &lt;code&gt;pearsonr&lt;/code&gt; between each of the variables from 1 to 9 and number 10, and I get 2,4,5,8 are most correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I used &lt;code&gt;PCA&lt;/code&gt; from &lt;code&gt;sklearn.decomposition&lt;/code&gt; and calculate the &lt;code&gt;pca.explained_variance_&lt;/code&gt; I get these values&#xA;[  2.13128046e+01   3.44315766e-01   3.26052258e-01   2.67148345e-01&#xA;   1.85871921e-01   1.55241540e-01   1.31994073e-01   9.34982387e-02&#xA;   1.03297667e-02]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't it a problem that the first variable is so much higher than all the other ones? What does it mean? &lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2016-07-28T07:51:03.110" Title="Which variables matter most for prediction of another variable?" Tags="|python|predictive-modeling|pca|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13042" PostTypeId="2" ParentId="12974" CreationDate="2016-07-28T02:49:05.877" Score="1" Body="&lt;p&gt;Here &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/api_docs/python/framework.html&quot; rel=&quot;nofollow&quot;&gt;https://www.tensorflow.org/versions/r0.9/api_docs/python/framework.html&lt;/a&gt; (search &lt;code&gt;gradient_override_map&lt;/code&gt;) is an example on &lt;code&gt;gradient_override_map&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@tf.RegisterGradient(&quot;CustomSquare&quot;)&#xA;def _custom_square_grad(op, grad):&#xA;  # ...&#xA;&#xA;with tf.Graph().as_default() as g:&#xA;  c = tf.constant(5.0)&#xA;  s_1 = tf.square(c)  # Uses the default gradient for tf.square.&#xA;  with g.gradient_override_map({&quot;Square&quot;: &quot;CustomSquare&quot;}):&#xA;    s_2 = tf.square(s_2)  # Uses _custom_square_grad to compute the&#xA;                          # gradient of s_2.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, a possible solution could be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@tf.RegisterGradient(&quot;ExternalGradient&quot;)&#xA;def _custom_external_grad(unused_op, grad):&#xA;    # I don't know yet how to compute a gradient&#xA;    # From Tensorflow documentation:&#xA;    return grad, tf.neg(grad)&#xA;&#xA;def external_process (myvar):&#xA;    subprocess.call(&quot;process.sh&quot;, myvar)&#xA;    with open('result.json', 'r') as f:&#xA;        result = json.load(data, f)&#xA;    return np.array(result[&quot;result&quot;])&#xA;&#xA;myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)&#xA;&#xA;g = tf.get_default_graph()&#xA;with g.gradient_override_map({&quot;PyFunc&quot;: &quot;ExternalGradient&quot;}):&#xA;    external_data =  tf.py_func(external_process, [myvar], [tf.float32])[0]&#xA;&#xA;loss =  tf.reduce_sum(external_data)&#xA;&#xA;optimizer = tf.train.AdamOptimizer(0.05)&#xA;train_step = optimizer.minimize(loss)&#xA;sess.run(train_step)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21775" LastActivityDate="2016-07-28T02:49:05.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13044" PostTypeId="1" AcceptedAnswerId="13046" CreationDate="2016-07-28T05:52:37.533" Score="0" ViewCount="191" Body="&lt;p&gt;I'm not quite sure I understand the bag-of-visual-words representation, so I may misformulate my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm currently looking for is an open source library (possibly with python API). I give it pictures as input, and its output is a set of (sparse) features, so that I can perform my stuff base on this features. Idealy, I would like this piece of software to work without internet connection (so that I can work with it while in the plane).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I just learnt that facebook recently (summer 2016) released some of its image recognition code (namely &lt;a href=&quot;https://github.com/facebookresearch/multipathnet&quot; rel=&quot;nofollow&quot;&gt;multipathnet&lt;/a&gt;, &lt;a href=&quot;https://github.com/facebookresearch/deepmask&quot; rel=&quot;nofollow&quot;&gt;deepmask and sharpmask&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="21825" LastEditorUserId="21825" LastEditDate="2016-08-26T13:21:13.857" LastActivityDate="2016-08-26T13:21:13.857" Title="Is there an open source implementation for bag-of-visual words?" Tags="|image-classification|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13046" PostTypeId="2" ParentId="13044" CreationDate="2016-07-28T06:27:48.347" Score="3" Body="&lt;p&gt;There is one implementation of BoVW in openCV. You can find the documentation here :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.opencv.org/2.4/modules/features2d/doc/object_categorization.html&quot; rel=&quot;nofollow&quot;&gt;http://docs.opencv.org/2.4/modules/features2d/doc/object_categorization.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-07-28T06:27:48.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13047" PostTypeId="2" ParentId="13041" CreationDate="2016-07-28T07:48:00.430" Score="2" Body="&lt;p&gt;Let's use the following notation : $(x_1, ...x_9)$ are the first 9 variables out of which you try to predict $y$, your tenth variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will try to address what looks like a PCA misunderstanding, then give you some ways to predict which variables $ x_i $ matter most to predict $y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Principal Component Analysis&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The returned features you get from the PCA are not the original one! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The PCA transforms $(x_1, ..., x_9)$ into $(x'_1, ..., x'_9)$ where each $x'_i$ is a particular combination of the $(x_1, ..., x_9)$ such that for any $i, i'$, $x_i$ and $x_i'$ are linearly uncorrelated (ie orthogonal). Moreover, it is done such that each $x_i$ explains a &lt;em&gt;certain part&lt;/em&gt; of the training dataset variance. In fact, most of the implementation sort the $x'_1, ..., x'_9$ such that the first one is the one &lt;em&gt;explaining&lt;/em&gt; the variance the most, then the second, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus you end up having your first variable $x'_1$ explaining a lot your prediction $y$. But this $x'_i$ is a particular combinaison of $(x_1, ..., x_9)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Getting the feature importance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to assert which variable $x_i$ explains $y$ the best, one might use different ways. Just to mention that the feature importance is not absolute ; it relies on the technique (or estimator) you use to address this question. Here is a non exhaustive list of possibilities :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The linear correlation is a good start&lt;/li&gt;&#xA;&lt;li&gt;With the &lt;code&gt;RandomForestClassifier&lt;/code&gt; (or &lt;code&gt;RandomForestRegressor&lt;/code&gt; depending on $y$) of &lt;code&gt;sklearn.ensemble&lt;/code&gt;, you can use &lt;code&gt;feature_importances_&lt;/code&gt; method to get which one is used the most at tree nodes. Note : it works with any decision tree estimator.&lt;/li&gt;&#xA;&lt;li&gt;With standard linear or logistic regression (same with Lasso, Ridge, ...), you can check which variable has the higher coefficient (do not forget to normalize your input variables $(x_1, ..., x_9)$ ) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't know how it is done using neural networks but would be glad if anyone has an hint.&lt;/p&gt;&#xA;" OwnerUserId="20759" LastActivityDate="2016-07-28T07:48:00.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13048" PostTypeId="2" ParentId="13041" CreationDate="2016-07-28T07:51:03.110" Score="2" Body="&lt;p&gt;The output from PCA is not a value of each attribute. It is eigenvalues you get. So the first value is direction with the highest variance, or first component if you wish. This component is combination of attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No need to repeat what have been already written, check this &lt;a href=&quot;https://stackoverflow.com/questions/22984335/recovering-features-names-of-explained-variance-ration-in-pca-with-sklearn&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-07-28T07:51:03.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13049" PostTypeId="2" ParentId="12866" CreationDate="2016-07-28T08:02:46.313" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How can I efficiently optimize the hashing trick on that feature vector?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Use different size of hash, and hash function and see which work best.&#xA;One cannot tell in advance.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I enumerate hash bit length for each feature independently (greedy)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, this is NEVER a good practice. The hash should be common to all features.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I assume no hashing is necessary for features with small alphabet (ni is small)? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can always choose not to hash specific features. It implies you think these features are more important, and should avoid collision.&lt;/p&gt;&#xA;" OwnerUserId="21882" LastEditorUserId="29575" LastEditDate="2018-03-21T01:33:50.560" LastActivityDate="2018-03-21T01:33:50.560" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13050" PostTypeId="2" ParentId="12423" CreationDate="2016-07-28T10:30:12.537" Score="1" Body="&lt;p&gt;In a basic model, the target variable will be the score for a particular board position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let's have a closer look at the various parts of this sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For score you could take any number, but the Giraffe paper suggests taking the winning chances in a given board position ( for instance a number between -1 and 1 ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will first need to determine all possible (legal) moves. Let's say you find 42 candidate moves. Each move corresponds to one board position. Each board position has one score ( = the target variable ). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you will have to predict a score for 42 board positions, and, in a simplified model, take the move which results in the highest score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before training a model, you will have to assign a score to board positions in a train set, in order to build your model and be able to make predictions on new board positions.&#xA;So you will have to find a way to assign scores, either manually or by existing chess engines, or a combination of both. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the chess rules determine the possible legal moves, which is the first step and it's important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is more to it, but i hope this answers the question. This description will by itself not make for a high-ranking engine, but it might be a start for reasoning on how to build one.&lt;/p&gt;&#xA;" OwnerUserId="9186" LastEditorUserId="9186" LastEditDate="2016-07-28T19:29:14.420" LastActivityDate="2016-07-28T19:29:14.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13051" PostTypeId="1" AcceptedAnswerId="13056" CreationDate="2016-07-28T12:23:12.990" Score="1" ViewCount="105" Body="&lt;p&gt;I code a lot for web, games and some basic ML scripts. Now I would like to learn about data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/11172/21881&quot;&gt;This post&lt;/a&gt; is a good starting point but I would like some readings. I would like advices on books for a beginner (maths, tools, whatever).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found these ones from O'Reilly : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Machine Learning for Hackers, &lt;em&gt;By Drew Conway, John Myles White&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Agile Data Science, &lt;em&gt;By Russell Jurney&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;R Cookbook, &lt;em&gt;By Paul Teetor&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Sorry, my reputation is too low to let me post more than 1 link haha)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you recommand these readings ? Are there other must-reading books ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="21881" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-28T14:03:24.117" Title="From developper to data scientist" Tags="|beginner|books|" AnswerCount="1" CommentCount="0" ClosedDate="2016-07-28T23:40:42.370" ContentLicense="CC BY-SA 3.0" />
  <row Id="13053" PostTypeId="2" ParentId="13032" CreationDate="2016-07-28T13:37:21.320" Score="1" Body="&lt;p&gt;You can find everything you need on scikit documentation and tutorial pages. &#xA;Consider using a different algorithm using this map &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/tutorial/machine_learning_map/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For text variables use a vectorizer. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-07-28T13:37:21.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13054" PostTypeId="2" ParentId="12296" CreationDate="2016-07-28T13:45:30.093" Score="1" Body="&lt;p&gt;To me the approach looks overcomplicated. If you're not limited to that one algorithm use one-hot encoding and try out various classifiers. Many of them can predict probability which you can use to calculate the ranks. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastEditorUserId="15361" LastEditDate="2016-07-28T22:24:28.543" LastActivityDate="2016-07-28T22:24:28.543" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13055" PostTypeId="1" CreationDate="2016-07-28T13:55:51.490" Score="0" ViewCount="461" Body="&lt;p&gt;I have written my custom scorer object which is necessary for my problem and which I've called &quot;p_value_scoring_object&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the function sklearn.cross_validation.cross_val_score one of the parameters is &quot;scoring&quot;, which allows to use this scorer object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this option is not available for the score method of a classifier. Is sklearn just lacking that feature, or is there a way around it?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_iris&#xA;from sklearn.cross_validation import cross_val_score&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;clf = DecisionTreeClassifier(random_state=0)&#xA;iris = load_iris()&#xA;cross_val_score(clf, iris.data, iris.target, cv=10,scoring=p_value_scoring_object)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This works. However, this doesn't:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.fit(iris.data,iris.target)&#xA;clf.score(iris.data,iris.target,scoring=p_value_scoring_object)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21262" LastActivityDate="2016-11-25T19:18:33.630" Title="Using Scorer Object for Classifier Score Method for scikit-learn" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13056" PostTypeId="2" ParentId="13051" CreationDate="2016-07-28T14:03:24.117" Score="0" Body="&lt;p&gt;I suggest &lt;a href=&quot;https://github.com/open-source-society/data-science&quot; rel=&quot;nofollow&quot;&gt;this guide&lt;/a&gt; for a good introduction to the resources available to you to learn to do data science. You are starting at a good time, there are loads of excellent (and mostly free) resources to teach you this interesting field.&lt;/p&gt;&#xA;" OwnerUserId="17509" LastActivityDate="2016-07-28T14:03:24.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13057" PostTypeId="1" AcceptedAnswerId="13152" CreationDate="2016-07-28T14:46:27.583" Score="4" ViewCount="618" Body="&lt;p&gt;I have a dataset of about 1M observation and I had to predict a response that occurs only about 10.000 times (1%). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to train a random forest, but this takes a lot of time to train because the data is too large for my hardware. So I decided to take a sample, but an aleatory sample would be already too large to have a minimum quantity of response. (If  I take 10% aleatory, i would have only 1000 response)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I took a stratified sample. All responses and 10.000 aleatory non-responses, and trained my model in this dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now I need to rescale the probability so I have the real probability of the observation to be response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to simulate this problem with this code in R. Training a model in a balanced dataset and another one in the unbalanced data. But those models are not very correlated and I didn't find a good way to tranform the probabilities to the original unbalanced scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kA5e5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kA5e5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/publication/283349138_Calibrating_Probability_with_Undersampling_for_Unbalanced_Classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Found this is a good reference for future readers&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I found that for logistic regression I can do this by just changing the intercept this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat{\beta_0}  =  \hat{\beta_0^*} - log(\frac{\gamma_1}{\gamma_2})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $ \gamma_1 = Pr(Z=1|Y=1)$ and $ \gamma_2 = Pr(Z=1|Y=0)$. $Z$ is the an aleatory variable indicating if the observation is in the reduced dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was found in &lt;a href=&quot;https://www.ime.usp.br/~giapaula/texto_2013.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this book&lt;/a&gt; (in portuguese) page 216&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;simulate_data &amp;lt;- function(n){&#xA;  X &amp;lt;- data.frame(matrix(runif(n*20), ncol = 20))&#xA;  list(&#xA;    X = X,&#xA;    Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %&amp;gt;% pnorm(mean = 13)) %&amp;gt;% as.factor()  &#xA;  )&#xA;}&#xA;&#xA;balance &amp;lt;- function(X, Y){&#xA;  X &amp;lt;- rbind(&#xA;    X[Y == &quot;1&quot;,],&#xA;    X[Y == &quot;0&quot;,] %&amp;gt;% sample_n(length(Y[Y == &quot;1&quot;]))&#xA;    )&#xA;  return(list(&#xA;    X = X,&#xA;    Y = as.factor(c(rep(c(1,0), each = length(Y[Y == &quot;1&quot;]))))&#xA;  ))&#xA;}&#xA;&#xA;train &amp;lt;- simulate_data(100000)&#xA;&#xA;library(randomForest)&#xA;m_desb &amp;lt;- mean(train$Y == &quot;1&quot;)&#xA;modelo_desb &amp;lt;- randomForest(train$X,train$Y, ntree = 200, cutoff = c(1-m_desb, m_desb), nodesize = 30, mtry = 8)&#xA;bal &amp;lt;- balance(train$X, train$Y)&#xA;m_bal &amp;lt;- mean(bal$Y == &quot;1&quot;)&#xA;modelo_bal &amp;lt;- randomForest(bal$X,bal$Y, ntree = 100, cutoff = c(1-m_bal, m_bal), nodesize = 50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Code for plot&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;data.frame(&#xA;  unbalanced = predict(modelo_desb, newdata = test$X, type = &quot;prob&quot;)[,2],&#xA;  balanced = predict(modelo_bal, newdata = test$X, type = &quot;prob&quot;)[,2]&#xA;) %&amp;gt;%&#xA;  ggplot(aes(x = balanced, y = unbalanced)) +&#xA;  geom_point(size = 0.3) +&#xA;  xlim(0,1) +&#xA;  geom_smooth() +&#xA;  geom_hline(yintercept = m_desb, linetype = &quot;dashed&quot;) +&#xA;  geom_vline(xintercept = m_bal, linetype = &quot;dashed&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4978" LastEditorUserId="4978" LastEditDate="2016-08-09T14:29:11.777" LastActivityDate="2016-08-09T14:29:11.777" Title="Predict probability when model was trained in balanced dataset" Tags="|machine-learning|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13058" PostTypeId="1" CreationDate="2016-07-28T14:58:36.160" Score="-1" ViewCount="863" Body="&lt;p&gt;Have been trying to get a working database connection to OrientDB (v2.1.16) from my Java-based Spark (v2.0.0) code but had no luck so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only available connector specifically for OrientDB/Spark seems to only work with Scala code: &lt;a href=&quot;https://github.com/metreta/spark-orientdb-connector&quot; rel=&quot;nofollow&quot;&gt;https://github.com/metreta/spark-orientdb-connector&lt;/a&gt;&#xA;Since Java and Scala both execute on the JVM I imagine it should be possible to use it from Java but haven't seen how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OrientDB has a JDBC driver and I have also tried that route:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; SparkSession spark = SparkSession.builder().master(&quot;local&quot;).appName(&quot;My Spark App&quot;).getOrCreate();&#xA;&#xA; Map&amp;lt;String, String&amp;gt; options = new HashMap&amp;lt;String,String&amp;gt;();&#xA; options.put(&quot;driver&quot;, &quot;com.orientechnologies.orient.jdbc.OrientJdbcDriver&quot;);&#xA; options.put(&quot;url&quot;, &quot;jdbc:orient:remote:192.168.1.2/mydb&quot;);&#xA; options.put(&quot;dbtable&quot;, &quot;Customer&quot;);&#xA; options.put(&quot;user&quot;, &quot;user&quot;);&#xA; options.put(&quot;password&quot;, &quot;password&quot;);&#xA;&#xA; Dataset&amp;lt; org.apache.spark.sql.Row&amp;gt; df = spark.read().format(&quot;jdbc&quot;).options(options).load();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives an error on all queries I've tried (supplying Spark with the OrientDB JDBC jar in spark-submit):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Exception in thread &quot;main&quot; java.sql.SQLException: No current record&#xA;at com.orientechnologies.orient.jdbc.OrientJdbcResultSetMetaData.getCurrentRecord(OrientJdbcResultSetMetaData.java:202)&#xA;at com.orientechnologies.orient.jdbc.OrientJdbcResultSetMetaData.getColumnCount(OrientJdbcResultSetMetaData.java:84)&#xA;at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:130)&#xA;at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.&amp;lt;init&amp;gt;(JDBCRelation.scala:117)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Has anyone successfully accomplished this?&lt;/p&gt;&#xA;" OwnerUserId="21896" LastActivityDate="2017-08-27T09:38:57.533" Title="Connecting to OrientDB from Spark in Java" Tags="|apache-spark|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13059" PostTypeId="1" AcceptedAnswerId="13063" CreationDate="2016-07-28T15:15:45.510" Score="7" ViewCount="23351" Body="&lt;p&gt;I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-27T13:48:47.960" LastActivityDate="2020-12-27T13:48:47.960" Title="Merging large CSV files in pandas" Tags="|python|pandas|data|csv|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13060" PostTypeId="2" ParentId="2510" CreationDate="2016-07-28T17:08:46.557" Score="0" Body="&lt;p&gt;Data scientists are in high demand now and appear to be in the near future so looks like you're thinking in the right direction! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A recent McKinsey study predicts there will only be 200,000 data scientists to fill the 490,000 data science jobs by 2018.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the vast majority of data scientist now have an advanced degree in a quantitative field such as mathematics, computer science or econometrics, it is not &lt;em&gt;necessary&lt;/em&gt;.  One of my best friends is a leading data scientist at a Fortune 500 company without earning an advanced degree.  You can find his personal story here &lt;a href=&quot;http://bit.ly/2aA6PHk&quot; rel=&quot;nofollow&quot;&gt;http://bit.ly/2aA6PHk&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend checking out The Open Source Data Science Masters site at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://datasciencemasters.org/&quot; rel=&quot;nofollow&quot;&gt;http://datasciencemasters.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clare Corthell has done a masterful job of curating a list of free or inexpensive resources to learn the topics of data science, including statistics and data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck! &lt;/p&gt;&#xA;" OwnerUserId="21871" LastActivityDate="2016-07-28T17:08:46.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13061" PostTypeId="1" AcceptedAnswerId="13362" CreationDate="2016-07-28T17:12:29.933" Score="89" ViewCount="88202" Body="&lt;p&gt;I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used &lt;s&gt;, while the first layer uses He uniform initialization.&lt;/s&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked through ResNet paper and &quot;Delving Deep into Rectifiers&quot; paper (He initialization paper), but I haven't found any mention on normal init vs uniform init.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Batch Normalization allows us to use much higher learning rates and be less careful about initialization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In Batch Normalization paper's abstract, it is said that Batch Normalization allows us to be less careful about initialization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;s&gt;ResNet itself is still care on when to use normal init vs uniform init (rather than just go with the uniform init).&lt;/s&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When to use (He or Glorot) normal-distributed initialization over uniform initialization?&lt;/li&gt;&#xA;&lt;li&gt;What are normal-distributed initialization effects with Batch Normalization?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Notes aside:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It rhymes to use normal init with Batch Normalization, but I haven't found any paper to back this fact.&lt;/li&gt;&#xA;&lt;li&gt;I knew that ResNet uses He init over Glorot init because He init does better on a deep network.&lt;/li&gt;&#xA;&lt;li&gt;I've understood &lt;a href=&quot;http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization&quot; rel=&quot;noreferrer&quot;&gt;about Glorot init vs He init&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;My question is about Normal vs Uniform init.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9465" LastEditorUserId="8820" LastEditDate="2017-05-01T07:47:44.273" LastActivityDate="2023-09-16T04:05:43.500" Title="When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?" Tags="|neural-network|deep-learning|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13062" PostTypeId="2" ParentId="13005" CreationDate="2016-07-28T17:19:08.600" Score="1" Body="&lt;h2&gt;Two alternatives&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TrueSkill&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to try a variant of &lt;a href=&quot;https://www.microsoft.com/en-us/research/publication/trueskilltm-a-bayesian-skill-rating-system/&quot; rel=&quot;nofollow&quot; title=&quot;TrueSkill&quot;&gt;TrueSkill&lt;/a&gt;. As Wikipedia says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Factor graphs are used to &quot;pack up&quot; each team into $(\mu ,\sigma )$&#xA;  pairs on which the update formulas are run; the skill updates are then&#xA;  correctly distributed to each player.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's the &lt;a href=&quot;https://www.microsoft.com/en-us/research/publication/trueskilltm-a-bayesian-skill-rating-system/&quot; rel=&quot;nofollow&quot;&gt;TrueSkill paper&lt;/a&gt; with more details. From the abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The new system tracks the uncertainty about player skills, explicitly&#xA;  models draws, can deal with any number of competing entities and can&#xA;  infer individual skills from team results. Inference is performed by&#xA;  approximate message passing on a factor graph representation of the&#xA;  model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Simple Regression&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try to set this up as a multivariate regression problem. Inputs are tuples representing who was involved in the sale, outputs are # of sales. So something like this, assuming tuples are (Fred, Susan, Tom, John, Wayne, Conner, Winona):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1, 1, 1, 0, 0, 0, 0), 25&#xA;(0, 0, 0, 1, 1, 1, 0), 12&#xA;(1, 0, 0, 1, 1, 0, 0), 16&#xA;etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now each person's coefficient should be proportional to their contribution, assuming no interaction effects.&lt;/p&gt;&#xA;" OwnerUserId="6403" LastEditorUserId="6403" LastEditDate="2016-07-28T17:27:20.773" LastActivityDate="2016-07-28T17:27:20.773" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="13063" PostTypeId="2" ParentId="13059" CreationDate="2016-07-28T17:32:31.257" Score="4" Body="&lt;p&gt;No, there is not. You will have to use an alternative tool like &lt;a href=&quot;http://dask.pydata.org/en/latest/dataframe-groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;dask&lt;/a&gt;, &lt;a href=&quot;https://drill.apache.org/docs/using-sql-functions-clauses-and-joins/&quot; rel=&quot;nofollow noreferrer&quot;&gt;drill&lt;/a&gt;, &lt;a href=&quot;https://lab.getbase.com/pandarize-spark-dataframes/&quot; rel=&quot;nofollow noreferrer&quot;&gt;spark&lt;/a&gt;, or a good old fashioned relational database.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="30877" LastEditDate="2018-12-13T07:33:01.933" LastActivityDate="2018-12-13T07:33:01.933" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13064" PostTypeId="2" ParentId="13055" CreationDate="2016-07-28T17:53:26.340" Score="1" Body="&lt;p&gt;Why should it work? &lt;code&gt;scoring&lt;/code&gt; is an argument of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.cross_val_score.html#sklearn.cross_validation.cross_val_score&quot; rel=&quot;nofollow&quot;&gt;cross_val_score&lt;/a&gt;, not &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.score&quot; rel=&quot;nofollow&quot;&gt;score&lt;/a&gt;. The documentation says &lt;code&gt;score&lt;/code&gt; &quot;returns the mean accuracy on the given test data and labels.&quot; This behavior is not specific to &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;, but common to classifiers in sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you want &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics&quot; rel=&quot;nofollow&quot;&gt;sklearn.metrics&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-28T17:53:26.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13066" PostTypeId="1" CreationDate="2016-07-28T21:04:31.690" Score="0" ViewCount="541" Body="&lt;p&gt;I had a post on stackoverflow. As it is related to scikit-learn, I am hoping that I can obtain some assistance from data scientists in this forum. &lt;a href=&quot;https://stackoverflow.com/questions/38640815/python-running-out-of-memory&quot;&gt;https://stackoverflow.com/questions/38640815/python-running-out-of-memory&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, however, I would like to seek opinions on the reduction of memory use when one uses scikit-learn, as this can be a daily problem to deal with as a data scientist. Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="21242" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-05-10T14:32:05.087" Title="How to reduce pipeline footprint in scikit-learn?" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13067" PostTypeId="2" ParentId="13031" CreationDate="2016-07-29T02:11:13.847" Score="4" Body="&lt;p&gt;Adding a few more specifics to the previous two responses which both contain useful insight and perspective:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Regularization&lt;/code&gt; is used to control overfitting (more formally &lt;code&gt;high variance&lt;/code&gt;) scenarios. You should view any model as a careful balance of &lt;code&gt;bias&lt;/code&gt; and &lt;code&gt;variance&lt;/code&gt;. Thus a model that is unresponsive to regularization might be too underfit to begin with. Its tough to guess what is occurring without seeing your cross validation results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you cross validated your model? Are you using &lt;a href=&quot;https://datascience.stackexchange.com/questions/11014/why-are-precision-and-recall-used-in-the-f1-score-rather-than-precision-and-npv&quot;&gt;good metrics&lt;/a&gt; to &lt;a href=&quot;http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;assess your model&lt;/a&gt;? How do the the training metrics compare to the testing metrics as you adjust the regularization?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ideal model is slightly overfit and then regularization is applied to balance the bias and the variance. One can engage in a &lt;a href=&quot;https://datascience.stackexchange.com/questions/5268/how-to-detect-overfitting-of-a-stock-screener/5674#5674&quot;&gt;&lt;code&gt;bias-variance&lt;/code&gt; decomposition&lt;/a&gt;, but with a little more experience you can just compare the single metrics for the training and testing set and test the effect of increasing and decreasing the regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-07-29T14:37:12.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13068" PostTypeId="2" ParentId="11921" CreationDate="2016-07-29T03:38:03.397" Score="3" Body="&lt;p&gt;The following 2 packages are available in R for deep neural network training:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;darch&lt;/strong&gt;: Package for Deep Architectures and Restricted Boltzmann Machines.&#xA;The darch package is built on the basis of the code from G. E. Hinton and R. R. Salakhutdinov (available under Matlab Code for deep belief nets). This package is for generating neural networks with many layers (deep architectures), train them and fine tuning with common known training algorithms like backpropagation or conjugate gradients. Additionally, supervised fine-tuning can be enhanced with maxout and dropout, two recently developed techniques to improve fine-tuning for deep learning. CRAN link: &lt;a href=&quot;http://cran.um.ac.ir/web/packages/darch/index.html&quot; rel=&quot;nofollow&quot; title=&quot;http://cran.um.ac.ir/web/packages/darch/index.html&quot;&gt;http://cran.um.ac.ir/web/packages/darch/index.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;deepnet&lt;/strong&gt;: deep learning toolkit in R. Implement some deep learning architectures and neural network algorithms, including BP,RBM,DBN,Deep autoencoder and so on. CRAN link: &lt;a href=&quot;https://cran.r-project.org/web/packages/deepnet/index.html&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/deepnet/index.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-07-29T03:38:03.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13069" PostTypeId="1" AcceptedAnswerId="13075" CreationDate="2016-07-29T07:34:48.967" Score="1" ViewCount="5896" Body="&lt;p&gt;I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Hospital_ID   District_ID  Employee&#xA;Hospital 1    District 19   5 &#xA;Hospital 1    District 19   10&#xA;Hospital 1    District 19   6&#xA;Hospital 2    District 10   50&#xA;Hospital 2    District 10   51&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now I want to remove the duplicates but I want to replace the values in my original data frame by their mean so that it should look like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Hospital 1    District 19   7.0000&#xA;Hospital 2    District 10   50.5000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-27T00:47:44.497" LastActivityDate="2020-12-27T13:48:41.010" Title="Replacing column values in pandas" Tags="|python|pandas|dataframe|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13071" PostTypeId="2" ParentId="13069" CreationDate="2016-07-29T07:52:46.423" Score="2" Body="&lt;p&gt;What you want to do is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Aggregate_function&quot; rel=&quot;nofollow&quot;&gt;aggregation&lt;/a&gt;; deduplication or duplicate removal is something else.  I think the code self-explanatory:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.groupby(['Hospital_ID', 'District_ID']).mean()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-07-29T17:15:45.537" LastActivityDate="2016-07-29T17:15:45.537" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13072" PostTypeId="1" CreationDate="2016-07-29T08:06:43.490" Score="0" ViewCount="105" Body="&lt;p&gt;I'm interested in data science market. I was expecting that there would be a lot of companies who are making algorithms and models for companies like in kaggle competitions. But i struggle to find any. Practically every data science company is selling some automated software which makes algorithms/models by its own. So why is that so? Did i understand situation correctly? and if there are companies who make algorithms and models which are the leaders and best companies in that field. thanks p.s. sorry for my bad english&lt;/p&gt;&#xA;" OwnerUserId="21920" LastActivityDate="2016-07-29T16:16:51.770" Title="Data Science Companies" Tags="|bigdata|algorithms|" AnswerCount="2" CommentCount="3" ClosedDate="2016-07-29T21:33:20.703" ContentLicense="CC BY-SA 3.0" />
  <row Id="13073" PostTypeId="1" AcceptedAnswerId="13095" CreationDate="2016-07-29T08:34:50.377" Score="1" ViewCount="268" Body="&lt;p&gt;I have visualized a dataset in 2D after employing PCA. As 2D visualization shows in figure, there is a good separation between points (A, B). Now, I want to use a metric which can separate these points (between these 2 PC components not in main dataset) too. I mean have separation between these PCA components without visualization. I used some clustering methods but they raise false positives. I mean they miss cluster many points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as shown in histogram there is a gap between points A,B. Does this help in devising any metric? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be so grateful if you can introduce me any method and algorithm to be able to do separation between A and B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qkdvf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qkdvf.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oPUqh.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oPUqh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="31513" LastEditDate="2017-05-18T09:25:28.563" LastActivityDate="2017-05-18T09:25:28.563" Title="Looking for an algorithm that correctly clusters visually separable clusters" Tags="|clustering|algorithms|visualization|pca|distance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13074" PostTypeId="2" ParentId="13073" CreationDate="2016-07-29T09:08:21.237" Score="-1" Body="&lt;p&gt;Using k-means clustering algorithm on this dataset should work perfectly fine. You just have to pass the (n_samples, 2) matrix where element $(i,j)$ represents the j-th coordinate of sample i in the PCA to any k-means algorithm, and specify that you want 2 clusters, and Euclidean metric. &lt;/p&gt;&#xA;" OwnerUserId="21755" LastActivityDate="2016-07-29T09:08:21.237" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13075" PostTypeId="2" ParentId="13069" CreationDate="2016-07-29T11:08:20.230" Score="6" Body="&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/381/emre&quot;&gt;Emre&lt;/a&gt; already mentioned, you may use the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;groupby&lt;/a&gt; function. After that, you should apply &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;reset_index&lt;/a&gt; to move the MultiIndex to the columns:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame( [ ['Hospital 1', 'District 19', 5],&#xA;                     ['Hospital 1', 'District 19', 10],&#xA;                     ['Hospital 1', 'District 19', 6],&#xA;                     ['Hospital 2', 'District 10', 50],&#xA;                     ['Hospital 2', 'District 10', 51]], columns = ['Hospital_ID', 'District_ID', 'Employee'] )&#xA;&#xA;df = df.groupby( ['Hospital_ID', 'District_ID'] ).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;which gives you:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;  Hospital_ID  District_ID  Employee&#xA;0  Hospital 1  District 19       7.0&#xA;1  Hospital 2  District 10      50.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21815" LastEditorUserId="85045" LastEditDate="2020-12-27T13:48:41.010" LastActivityDate="2020-12-27T13:48:41.010" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13076" PostTypeId="2" ParentId="13072" CreationDate="2016-07-29T12:39:35.847" Score="0" Body="&lt;p&gt;In the market there is two types of companies operating in data sciences algorithm :&#xA;The First ones companies how's operating in the different sector and hiring data scientists to analyse or predict their business.&#xA;The second kind: is software provider how's customizing software for their clients for the specific business and you can find them by fields like Data analysis for pharmaceutical products, Statistical Forecast pharmaceutical products.&lt;/p&gt;&#xA;" OwnerUserId="21930" LastActivityDate="2016-07-29T12:39:35.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13078" PostTypeId="2" ParentId="5357" CreationDate="2016-07-29T13:38:21.467" Score="0" Body="&lt;p&gt;Take a look at &lt;a href=&quot;https://software.intel.com/en-us/intel-daal&quot; rel=&quot;nofollow&quot;&gt;Intel DAAL&lt;/a&gt; which is underway. It's highly optimised for Intel CPU architecture and supports distributed computations. &lt;/p&gt;&#xA;" OwnerUserId="21932" LastActivityDate="2016-07-29T13:38:21.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13081" PostTypeId="2" ParentId="5357" CreationDate="2016-07-29T14:21:25.633" Score="0" Body="&lt;p&gt;&lt;strong&gt;Scalable Machine Learning Solutions for Big Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will add my $.02 because there is a key area that seems not to have been addressed in all of the previous posts - &lt;strong&gt;machine learning on big data&lt;/strong&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For big data, scalability is key, and R is insufficient. Further, languages like Python and R are only useful for interfacing with scalable solutions which are usually written in other languages. I make this distinction not because I want to disparage those using them, but only because its so important for members of the data science community to understand what truly scalable machine learning solutions look like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do most of my work with &lt;strong&gt;big data on distributed memory clusters&lt;/strong&gt;. That is, I don't just use one 16 core machine (4 quad core processors on a single motherboard sharing the memory of that motherboard), I use a small cluster of 64 16 core machines. The requirements are very different for these distributed memory clusters than for shared memory environments and big data machine learning requires scalable solutions within distributed memory environments in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also use C and C++ everywhere within a proprietary database product. All of our high level stuff is handled in C++ and MPI, but the low level stuff that touches the data is all longs and C style character arrays to keep the product very very fast. The convenience of std strings are simply not worth the computational cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There not many C++ libraries available which offer distributed, scalable machine learning capabilities - &lt;a href=&quot;http://www.mlpack.org/&quot; rel=&quot;nofollow&quot;&gt;MLPACK&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are other scalable solutions with APIs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apache Spark has a scalable machine learning library called &lt;a href=&quot;http://spark.apache.org/docs/latest/ml-guide.html&quot; rel=&quot;nofollow&quot;&gt;MLib&lt;/a&gt; that you can interface with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow&quot;&gt;Tensorflow&lt;/a&gt; now has &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/how_tos/distributed/index.html&quot; rel=&quot;nofollow&quot;&gt;distributed tensorflow&lt;/a&gt; and has a &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/api_docs/cc/index.html&quot; rel=&quot;nofollow&quot;&gt;C++ api&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-07-29T14:21:25.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13082" PostTypeId="1" AcceptedAnswerId="13084" CreationDate="2016-07-29T15:08:01.010" Score="0" ViewCount="189" Body="&lt;p&gt;Until now I have implemented linear and logistic regression myself. I have not used any library other than &lt;code&gt;numpy&lt;/code&gt; and &lt;code&gt;matplotlib&lt;/code&gt;. But in the internet every example is solved using libraries such as &lt;code&gt;sklearn&lt;/code&gt;, &lt;code&gt;pandas&lt;/code&gt; and etc. My question is, which one is good to learn for machine learning, implementing algorithm yourself or using libraries (sklearn or ...)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="20926" LastEditorUserId="20544" LastEditDate="2016-07-29T21:33:37.513" LastActivityDate="2018-09-14T21:56:40.433" Title="To learn machine learning which one is good?" Tags="|machine-learning|" AnswerCount="3" CommentCount="1" ClosedDate="2018-09-15T02:59:56.123" ContentLicense="CC BY-SA 3.0" />
  <row Id="13083" PostTypeId="1" AcceptedAnswerId="13088" CreationDate="2016-07-29T15:12:44.180" Score="9" ViewCount="1133" Body="&lt;p&gt;I have used the t-SNE algorithm to visualize my high dimensional data. However, I was wondering if this is a practical method for inference?&lt;/p&gt;&#xA;" OwnerUserId="21936" LastEditorUserId="85045" LastEditDate="2020-12-26T12:53:10.713" LastActivityDate="2020-12-26T12:53:10.713" Title="Is t-SNE just for visualization?" Tags="|machine-learning|visualization|dimensionality-reduction|tsne|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13084" PostTypeId="2" ParentId="13082" CreationDate="2016-07-29T15:35:47.270" Score="4" Body="&lt;p&gt;I can think of the following pros and cons for each. As for learning to code your own machine learning algorithms such as logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You will definitely learn more about specific algorithms, their details, role of different parameters and etc. That is also a good practice of coding itself. Then you can validate your implementation by benchmarking it against other packages and implementations.&lt;/li&gt;&#xA;&lt;li&gt;You will have more freedom in controlling different aspects of your method. You can add functions and modules as you wish and do not necessary have to deal with predefined variables, methods and etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the other hand, implementing algorithms when it is not necessary and you can just use existing packages is like reinventing the wheel. It normally takes a lot of time and you have to verify your results for each one of them. Packages like &lt;code&gt;sklearn&lt;/code&gt; are popular because of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A group of people are working on those, constantly making them up to date, testing the  methods in different situations for different needs. That makes packages like &lt;code&gt;sklearn&lt;/code&gt; very dependable and usually scalable.&lt;/li&gt;&#xA;&lt;li&gt;If you have a question about them, there are tons of resources out there; documentation, forums, source code, communities like StackOverflow where thousands of people are eager to help you literally for any error you face while running your code. &lt;/li&gt;&#xA;&lt;li&gt;Another important feature is automated hyperparameters tuning. Most of machine learning algorithms have a series of hyperparameters that need to be optimized in order to achieve the best performance. Packages like &lt;code&gt;sklearn&lt;/code&gt; efficiently search for the optimal tuning parameters (or &quot;hyperparameters&quot;) for your machine learning model in order to maximize its performance.&lt;/li&gt;&#xA;&lt;li&gt;Still if you are interested in implementing machine learning algorithms and like the coding, you can always contribute to the existing packages. They usually have Github repositories where you can raise an issue, ask for a new feature or provide help improving them.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All in all, if you have enough time and you are keen to learn low level details about those models, go ahead and give implementation them a shot. That is certainly fun. However, if you need to get to the results as soon as possible and looking for a reliable package where a huge group of people both in industry and academia are already using, &lt;code&gt;sklearn&lt;/code&gt;, &lt;code&gt;pandas&lt;/code&gt; and others are you options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this is helpful and good luck.&lt;/p&gt;&#xA;" OwnerUserId="20544" LastEditorUserId="20544" LastEditDate="2016-07-29T16:10:59.767" LastActivityDate="2016-07-29T16:10:59.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13085" PostTypeId="2" ParentId="13072" CreationDate="2016-07-29T16:16:51.770" Score="2" Body="&lt;p&gt;Some of the biggest companies in the world are 'data science' companies; facebook, for instance, performs targeted advertising based on analyzing users preference.  In essence, it's the most highly trained recommendation engine ever, as it knows a great deal about it's consumers.  At it's heart, Google is an advertisement firm, and similarly leverages user information to provide targeted messaging to consumers.  The same can be said for Amazon, Twitter, and most likely pokemon go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're thinking of a more purely 'data science as a service' company, larger consultancy firms such as the Boston Consulting Group have always incorporated analytics in their services, and have certainly profited from the proliferation of data.  They don't say &quot;we do data science&quot;, because people care more about getting the right answer, than how the answer is obtained.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as Kaggle goes, while it's an awesome resource, and it provides cool tutorials and data sets, if you're a professional data scientist, then you don't work for free ;-)  If a billion dollar company can harvest even a 1% gain in efficiency by offering a $50K prize, they can recoup that money very quickly.  If you're only asset is intellectual property, you need to be cautious about giving it away.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-07-29T16:16:51.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13086" PostTypeId="2" ParentId="13082" CreationDate="2016-07-29T16:21:36.810" Score="0" Body="&lt;p&gt;I second @NeilSlater. If your purpose is to study and understand Machine Learning, implementing algorithms would definitely help. On the other had, if you want to implement a ML algorithm to solve a real-life problem (or in Data Science competitions), I would suggest you try out the machine learning libraries in python, spark, R or any other language. The reason as explained by @ahajib is that these libraries have efficient implementations of the popular Machine Learning algorithms. &lt;strong&gt;So unless there is a purpose, why re-invent the wheel&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="20722" LastActivityDate="2016-07-29T16:21:36.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13087" PostTypeId="1" CreationDate="2016-07-29T17:20:08.917" Score="-1" ViewCount="285" Body="&lt;p&gt;Association rules are a very common technique when you want to figure out which events happens together (like burger and bread mostly sell together). In marketing this technique is used to find out the complimentary products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a technique to extract the &quot;Substitute Products&quot; which is like Inverse Association rules to find out which events are less likely to happen together. Are there any algorithms or techniques available in Spark, R, Python, etc. for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, Amir&lt;/p&gt;&#xA;" OwnerUserId="21938" LastEditorUserId="20544" LastEditDate="2016-07-29T21:33:33.640" LastActivityDate="2017-08-24T21:21:37.353" Title="Inverse Association Rules" Tags="|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13088" PostTypeId="2" ParentId="13083" CreationDate="2016-07-29T17:38:58.207" Score="2" Body="&lt;p&gt;It's a dimensionality reduction algorithm. Inference is the problem of determining the parameters, or labels, that best fit the model for a given input once the model parameters have been learned, or estimated.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-07-29T17:38:58.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13089" PostTypeId="1" AcceptedAnswerId="13090" CreationDate="2016-07-29T17:40:39.687" Score="1" ViewCount="30398" Body="&lt;p&gt;Suppose I have a data frame like this :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Hospital_name    State    Employees    ......&#xA;Fortis           Delhi    5000         ......&#xA;AIIMS            Delhi    1000000      ......&#xA;SuperSpeciality  Chennai  1000         ......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now I want to use this data frame to build a machine learning model for predictive analysis. For that, I must convert the strings to float values. Also, some of these columns in Hospital_name and State contains 'NAN' values. In such a case, how should I prepare my data for building a model in Keras?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-26T11:23:50.737" LastActivityDate="2020-12-26T12:53:15.610" Title="String Values in a data frame in Pandas" Tags="|machine-learning|python|data-cleaning|pandas|keras|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13090" PostTypeId="2" ParentId="13089" CreationDate="2016-07-29T17:58:07.453" Score="4" Body="&lt;p&gt;To convert from string to float in pandas (assuming you want to convert Employees and you loaded the data frame with df), you can use:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df['Employees'].apply(lambda x:float(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;You have not given enough information about your input and expected output. So let us assume that hospital name or anything for that matter which is the input for your model is &lt;code&gt;nan&lt;/code&gt;. You would like to remove it from the dataset because extracting features from '&lt;code&gt;nan&lt;/code&gt;' wouldn't make sense. Apart from that, if they are just other peripheral features, then it might be alright. In that case, if you wish to convert them into blank, then use:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.replace(np.nan,' ', regex=True)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Else, if you wish to remove that frame, you can check for nan using &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/missing_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="85045" LastEditDate="2020-12-26T12:52:42.933" LastActivityDate="2020-12-26T12:52:42.933" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13091" PostTypeId="2" ParentId="13089" CreationDate="2016-07-29T18:05:52.163" Score="1" Body="&lt;p&gt;A more direct way of converting &lt;code&gt;Employees&lt;/code&gt; to &lt;code&gt;float&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.Employees = df.Employees.astype(float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;You didn't specify what you wanted to do with NaNs, but you can replace them with a different value (int or string) using:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df = df.fillna(value_to_fill)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;If you want to drop rows with &lt;code&gt;NaN&lt;/code&gt; in it, use:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df = df.dropna()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21940" LastEditorUserId="85045" LastEditDate="2020-12-26T12:52:50.517" LastActivityDate="2020-12-26T12:52:50.517" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13092" PostTypeId="2" ParentId="13089" CreationDate="2016-07-29T18:16:52.830" Score="2" Body="&lt;p&gt;The best way to deal with types is to specify it when ingesting the file:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;pandas.read_csv(file_name, dtype={&amp;quot;Employees&amp;quot;: float})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;What you do with the missing data in Keras is up to you. You can elaborate further as it actually depends on your plan.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="85045" LastEditDate="2020-12-26T12:53:15.610" LastActivityDate="2020-12-26T12:53:15.610" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13093" PostTypeId="2" ParentId="12987" CreationDate="2016-07-29T19:55:24.187" Score="0" Body="&lt;p&gt;Here is one possible solution to the problem (if the dataframes fit in memory):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Put the dataframes into a list, naming the list entries the dataframe names&lt;/li&gt;&#xA;&lt;li&gt;Wrap the sqldf command into a function with one input, names(list)&lt;/li&gt;&#xA;&lt;li&gt;Use lapply to apply the function to each dataframe in the list&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is an example function that would work. It takes an input dataframe and left joins columns from a dataframe PF.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sqldf_join_wrapper = function(df_title) {&#xA;&#xA;    temp_df = sqldf(&#xA;    paste0(&quot;SELECT distinct a.* &#xA;           FROM &quot;, df_title, &quot; a&#xA;           LEFT JOIN PF b&#xA;           ON (a.hcom_id = b.tnow_id and a.city = b.region)&quot;)&#xA;    )&#xA;&#xA;    return(temp_df)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could use this function in an lapply statement to left join the columns from PF into all 206 of your country dataframes at once (assuming they are in a list). However, you need to pass in the titles of the input dataframes to the sql string, and it seems like lapply makes it a bit difficult based on my trial and error. Here is a solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output_list = lapply(names(initial_list_of_dfs), sqldf_join_wrapper)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output list should then contain the initial dataframes.&lt;/p&gt;&#xA;" OwnerUserId="21943" LastEditorUserId="21943" LastEditDate="2016-07-29T20:30:57.250" LastActivityDate="2016-07-29T20:30:57.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13094" PostTypeId="2" ParentId="13089" CreationDate="2016-07-29T19:59:19.957" Score="1" Body="&lt;p&gt;I don't understand why you would map the strings to floats. I would suggest using one hot encoding to categorize the strings with a Boolean 1 or 0.&lt;/p&gt;&#xA;&lt;p&gt;In pandas this would be:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;s = pd.Series(list('abca'))&#xA;pd.get_dummies(s)&#xA;&#xA;   a  b  c&#xA;0  1  0  0&#xA;1  0  1  0&#xA;2  0  0  1&#xA;3  1  0  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;You can also add &lt;code&gt;pd.get_dummies(l, dummy_na=True)&lt;/code&gt; to deal with the NaN values.&lt;/p&gt;&#xA;" OwnerUserId="21936" LastEditorUserId="85045" LastEditDate="2020-12-26T12:52:47.163" LastActivityDate="2020-12-26T12:52:47.163" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13095" PostTypeId="2" ParentId="13073" CreationDate="2016-07-29T20:16:32.530" Score="1" Body="&lt;p&gt;With appropriate parameters, DBSCAN and single linkage hierarchical agglomerative clustering should work very well. Epsilon=0.2 or so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But why? You know the data, just &lt;strong&gt;use a threshold&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want an algorithm to &quot;confirm&quot; your desired outcome then you are using it wrong. Be honest: if you want your result to be &quot;if $F-factor-1 &gt; 1.5 then cluster1 else cluster2&quot;, then just say so, instead of attempting to find a clustering algorithm to fit to your desired solution!&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-07-29T20:16:32.530" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13096" PostTypeId="2" ParentId="13010" CreationDate="2016-07-29T22:41:30.470" Score="3" Body="&lt;p&gt;This is an interesting question. I also see people mentioning Spark as de-facto. Here are my two cents on the same&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Big Data:&#xA;The size of the data and the goal is the key here. Goal could be defined as&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Decrease computation time for data wrangling&lt;/li&gt;&#xA;&lt;li&gt;Efficient storage&lt;/li&gt;&#xA;&lt;li&gt;Handling multiple file formats coming from different sources&lt;/li&gt;&#xA;&lt;li&gt;Ability to provide data summarization so one can keep track of the data&#xA;being wrangled. Its very easy to make mistakes and handling missing &#xA;data the right way is more difficult than it seems.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R for BigData: A lot of wrangling can be done using R itself, if one &#xA;has been able to resolve the problem of storage. May NAS or our good &#xA;friend HDFS. The number of aggregators R provide is very detailed and &#xA;mature(less buggy). But, one problem with R is that the syntax is not &#xA;consistent and it becomes difficult to mange the code over a peroid of &#xA;time. Also, there is a SparkR interface if you decide to go down the &#xA;route of Spark.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;python for BigData: Pandas and Numpy provide a pretty good package for &#xA;handling data in an efficient way. Types can be guaranteed through&lt;br&gt;&#xA;type-casting and not just depending on duck-typing. There is a binding &#xA;with Spark called pyspark which could be useful if one decides to move &#xA;from python based data-structure to spark dataframes/RDDs.&lt;a href=&quot;https://books.google.com/books?id=XmeDCwAAQBAJ&amp;amp;pg=PR12&amp;amp;lpg=PR12&amp;amp;dq=scala%20library%20for%20data%20wrangling&amp;amp;source=bl&amp;amp;ots=zzoQzsBAyN&amp;amp;sig=uvxrV81VbQ3-QvKNIM49bC7EXrU&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwiNvKyz25nOAhWD8x4KHTu5BhsQ6AEIPzAE#v=onepage&amp;amp;q=scala%20library%20for%20data%20wrangling&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;data &#xA;wrangling in python&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Throwing in another one scala for BigData: Spark is natively written in&lt;br&gt;&#xA;Scala and is a good candidate for data wrangling and data modeling. &#xA;Unlike python and R it support functional paradigm as well OOP as a &#xA;first class citizens hence allowing the ease to write manageable code &#xA;with static typing. Twitter guys have a bunch of libraries to help one &#xA;in data-wrangling e.g. &lt;a href=&quot;https://github.com/twitter/algebird&quot; rel=&quot;nofollow&quot;&gt;algebird&lt;/a&gt;, &lt;a href=&quot;https://www.safaribooksonline.com/blog/2013/07/31/scala-collections-api-for-people-and-their-pets/&quot; rel=&quot;nofollow&quot;&gt;scala collection api&lt;/a&gt;, &#xA;&lt;a href=&quot;https://github.com/milessabin/shapeless&quot; rel=&quot;nofollow&quot;&gt;shapeless&lt;/a&gt;, &lt;a href=&quot;http://slick.lightbend.com/&quot; rel=&quot;nofollow&quot;&gt;slick&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And then as other have mentioned there are other offering such as Pig, Hive(is batch oriented). Checkout Apache Drill( May be that suits your need better ). And if you land up using Spark then Spark SQL is an option too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum it up, would suggest to start with python to be safe and then move to pyspark if you feel that the size is not manageable anymore and you need more sophisticated data-structure. And once you realize your needs may be use Scala for production code. Let me know if I could be of help or you need more information or guidance in anyway. Thanks&lt;/p&gt;&#xA;" OwnerUserId="5179" LastActivityDate="2016-07-29T22:41:30.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13099" PostTypeId="1" CreationDate="2016-07-30T03:15:50.823" Score="1" ViewCount="126" Body="&lt;p&gt;Below are the results of two different linear regressions.  The first only has an N of 10 while the second has an N of 43.  The first has a very high association between the dependent variable and the independent variables overall (Adj R2 = 0.948, P = 0.003).  The second one only has one variable in the model with a significant association with the dependent variable and a lower overall association (Adj R2 = 0.436, P &amp;lt; 0.001).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SENCE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SENCE.png&quot; alt=&quot;1st case&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ncch4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ncch4.png&quot; alt=&quot;2nd Case&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am trying to understand some of the reasons for these two sets of results.  The first seems spurious due to the small N.  I am still having trouble understanding how the second could have an adjusted R2 as high as it does considering only one variable has a significant association with the dependent.  What kinds of things might be going on here?&lt;/p&gt;&#xA;" OwnerUserId="15032" LastEditorUserId="11097" LastEditDate="2016-08-01T07:15:47.873" LastActivityDate="2016-08-01T07:15:47.873" Title="Interpreting Multiple Linear Regression" Tags="|linear-regression|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13100" PostTypeId="1" CreationDate="2016-07-30T06:20:44.920" Score="1" ViewCount="716" Body="&lt;p&gt;I have a huge dataset with 3 variables &lt;code&gt;Company_ID, Area_code, Product_ID&lt;/code&gt; each one of them is a categorical variable of levels &lt;code&gt;1500,50,15&lt;/code&gt; respectively,where &lt;code&gt;Product_ID&lt;/code&gt; is the product the &lt;code&gt;Company_ID&lt;/code&gt; is using.Each company can be located in more than one &lt;code&gt;area_code&lt;/code&gt; and can be using more than one product.Lets call this train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a combination of &lt;code&gt;Company_ID, Area_code, Product_ID&lt;/code&gt; ,&lt;br&gt;&#xA;&lt;code&gt;example- comp_025, area_012, p_10&lt;/code&gt; that is not present in train data,i have to predict the probability that a company &lt;code&gt;Company_ID&lt;/code&gt; located in area &lt;code&gt;Area_Code&lt;/code&gt; uses the product &lt;code&gt;Product_ID&lt;/code&gt;. So that i can recommend that product to company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a beginner and having difficulty in coming up with a model. Any input is highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="21929" LastEditorUserId="21929" LastEditDate="2016-07-30T06:36:33.630" LastActivityDate="2016-08-01T13:26:41.487" Title="predict the probability of buying a product" Tags="|machine-learning|r|predictive-modeling|regression|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13102" PostTypeId="2" ParentId="13087" CreationDate="2016-07-30T12:37:54.737" Score="0" Body="&lt;p&gt;FWIK from using it, the algorithm in &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow&quot;&gt;Orange&lt;/a&gt; add-on for &lt;a href=&quot;http://orange3-associate.readthedocs.org/&quot; rel=&quot;nofollow&quot;&gt;association rules mining&lt;/a&gt; (a Python package) works like this: if you have a dense table of transactions where a 1 in the table denotes that a product was bought and 0 that it was not, then the mined association rules also contain products that were explicitly not bought, e.g. &lt;code&gt;bread=1, burger=0 ==&amp;gt; milk=1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, however, the products table is passes as a sparse basket with non-empty values representing the product counts, then the generated rules don't contain any such negative (&lt;code&gt;burger=0&lt;/code&gt;) checks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-07-30T12:37:54.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13103" PostTypeId="2" ParentId="13040" CreationDate="2016-07-30T12:47:01.713" Score="0" Body="&lt;p&gt;You have little features (as in none hehe). Your data reminds me a lot of stock market predictions, where you know nothing about what you're studying except for price. Furthermore, price behaves chaotically, because if it was easily predictable, then everybody would be predicting it, thus removing its predictability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if you are serious about studying that data, you may want to look up literature on stock market prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggested in a comment that one approach could be to see this as a classification problem rather than a regression problem. This is in fact what many people do in the stock market. Instead of predicting actual price, people study whether the price will go up or down (and therefore whether they should buy or sell).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of what I mean, implemented in sklearn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.cross_validation import cross_val_score&#xA;from sklearn.linear_model import LogisticRegression&#xA;&#xA;# random signal&#xA;n = 100&#xA;t = np.arange(n)&#xA;x = np.random.rand(n)&#xA;plt.plot(t, x)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/waavw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/waavw.png&quot; alt=&quot;random signal&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# model ups and downs: convert to 0s (down) and 1s (up)&#xA;t = t[1:]&#xA;y = ((np.sign(x[1:] - x[:-1])+1)/2).astype(int)&#xA;plt.plot(t, y)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Gu42l.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Gu42l.png&quot; alt=&quot;ups and downs&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# one look-ahead model using the four previous observations&#xA;X = np.c_[y[:-4], y[1:-3], y[2:-2], y[3:-1]]&#xA;y = y[4:]&#xA;scores = cross_val_score(LogisticRegression(), X, y, cv=10, n_jobs=-1)&#xA;print('Acc: %.2f (%.2f)' % (np.mean(scores), np.std(scores)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Acc: 0.75 (0.17)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One advantage of this approach is that it gives you probabilities of going up or down. Remember, a logistic regression is a probabilistic model $P(y|x)$, and here we are taking $P(y|x)&amp;lt;0.5$ to mean down and $P(y|x)&amp;gt;0.5$ to mean up. But if the probability is close to 0.5, you could have your model say &quot;I don't know&quot;. This way you only take action when the model reports down or up with high confidence (close to 0 (down) or 1 (up)). A colleague of mine wrote &lt;a href=&quot;https://sigarra.up.pt/flup/pt/pub_geral.show_file?pi_gdoc_id=91387&quot; rel=&quot;nofollow noreferrer&quot;&gt;his thesis&lt;/a&gt; on this kind of thing. Are you trying to model sports in order to place bets? If so, this is a pretty simple but effective model of choosing when it is worthy to place bets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if the idea is to learn about data mining, then I would try another dataset.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sklearn comes with a bunch of &lt;a href=&quot;http://scikit-learn.org/stable/datasets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;nice datasets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;kaggle has a lot of 101 training competitions &lt;a href=&quot;https://www.kaggle.com/competitions&quot; rel=&quot;nofollow noreferrer&quot;&gt;rich in tutorials&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-07-30T14:23:26.987" LastActivityDate="2016-07-30T14:23:26.987" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13104" PostTypeId="1" AcceptedAnswerId="13139" CreationDate="2016-07-30T13:15:15.260" Score="4" ViewCount="6440" Body="&lt;p&gt;I am working on the Titanic dataset. So far my submission has 0.78 score using soft majority voting with logistic regression and random forest. As for the features, I used Pclass, Age, SibSp, Parch, Fare, Sex, Embarked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to further boost the score for this classification problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I tried is to add more classifiers for the majority voting, but it does not help, it even worthens the result. How do I understand this worthening effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your insight.&lt;/p&gt;&#xA;" OwnerUserId="21955" LastEditorUserId="21955" LastEditDate="2016-07-30T14:03:03.873" LastActivityDate="2016-08-05T07:35:54.750" Title="How to further improve the kaggle titanic submission accuracy?" Tags="|machine-learning|classification|beginner|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13105" PostTypeId="1" CreationDate="2016-07-30T22:20:40.047" Score="1" ViewCount="951" Body="&lt;p&gt;I'm building a simple feedforward neural network in Tensorflow and something seems to be broken.  I have adopted the basic structure from &lt;a href=&quot;http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/&quot; rel=&quot;nofollow&quot;&gt;http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/&lt;/a&gt; and have tried to adapt it to spit out 1 output unit instead.  The input is 10 numerically represented players ex:(5, 17, 19, 54, 110, 3, 112, 78, 60, 63), 5 on the 'North' team and 5 on the 'South' team, and the output is a 1 or 0 (if the 'North' team won or not).  3 hidden layers (200 parameters each).  With how this is set up, the algorithm trains to constantly predict 1 (North victory).  What can I fix that allows this to train to predict which team is going to have a higher chance of winning?  My expected outcome is ex: [.54] (slight edge to the north team) or [.40] (edge to the south team) etc.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;import sqlite3&#xA;&#xA;def get_matches(start, stop):&#xA;    connection = sqlite3.connect('5v5.db')&#xA;    cursor = connection.cursor()&#xA;&#xA;    result = cursor.execute('''SELECT * FROM matches WHERE rowid&amp;gt;=? AND rowid&amp;lt;=?''', (start, stop))&#xA;&#xA;    return result.fetchall()&#xA;&#xA;def init_weights(shape):&#xA;    return tf.Variable(tf.random_normal(shape, stddev=0.01))&#xA;&#xA;def model(X, w_h1, w_h2, w_h3, w_o):&#xA;    h1 = tf.nn.sigmoid(tf.matmul(X, w_h1))&#xA;    h2 = tf.nn.sigmoid(tf.matmul(h1, w_h2))&#xA;    h3 = tf.nn.sigmoid(tf.matmul(h2, w_h3))&#xA;    return tf.nn.sigmoid(tf.matmul(h3, w_o))&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;    TRAINING_GAMES = 2500 #number of games to train on&#xA;    TEST_GAMES = 500 #number of games to test the results on&#xA;&#xA;    NUM_DIGITS = 10 #input units&#xA;    FIRST_HIDDEN_LAYER = 200&#xA;    SECOND_HIDDEN_LAYER = 200&#xA;    THIRD_HIDDEN_LAYER = 200&#xA;    BATCH_SIZE = 150&#xA;&#xA;    training_matches = get_matches(1, TRAINING_GAMES)&#xA;    trX = np.array([match[1:11] for match in training_matches])&#xA;    trY = np.array([[match[11]] for match in training_matches])&#xA;&#xA;    X = tf.placeholder(&quot;float&quot;)&#xA;    Y = tf.placeholder(&quot;float&quot;)&#xA;&#xA;    w_h1 = init_weights([NUM_DIGITS, FIRST_HIDDEN_LAYER])&#xA;    w_h2 = init_weights([FIRST_HIDDEN_LAYER, SECOND_HIDDEN_LAYER])&#xA;    w_h3 = init_weights([SECOND_HIDDEN_LAYER, THIRD_HIDDEN_LAYER])&#xA;    w_o = init_weights([THIRD_HIDDEN_LAYER, 1])&#xA;&#xA;    py_x = model(X, w_h1, w_h2, w_h3, w_o)&#xA;&#xA;    cost = tf.square(Y - py_x)&#xA;    train_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost)&#xA;&#xA;    predict_op = py_x&#xA;&#xA;    with tf.Session() as sess:&#xA;        tf.initialize_all_variables().run()&#xA;&#xA;        for epoch in range(10000):&#xA;            p = np.random.permutation(range(len(trX)))&#xA;            trX, trY = trX[p], trY[p]&#xA;&#xA;            for start in range(0, len(trX), BATCH_SIZE):&#xA;                end = start + BATCH_SIZE&#xA;                sess.run(train_op, feed_dict={X: trX[start:end], Y: trY[start:end]})&#xA;&#xA;                print(epoch, sess.run(predict_op, feed_dict={X: trX, Y:trY}))&#xA;&#xA;        test_matches = get_matches(TRAINING_GAMES+1, TRAINING_GAMES+TEST_GAMES)&#xA;&#xA;        teX = np.array([match[1:11] for match in test_matches])&#xA;        teY = sess.run(predict_op, feed_dict={X: teX})&#xA;        print(teY)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21964" LastActivityDate="2016-08-01T08:00:41.117" Title="Simple feed forward neural network on Tensorflow question" Tags="|machine-learning|python|neural-network|tensorflow|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13106" PostTypeId="2" ParentId="12958" CreationDate="2016-07-31T00:48:59.390" Score="3" Body="&lt;p&gt;I think the question you're wrestling with is essentially this: Is there a way to use information that may be present in the data without A as part of your strategy for predicting A?  There is actually a name for the set of methods that do exactly that: &lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;nofollow&quot;&gt;semi-supervised learning&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While there are multiple techniques, a method analogous to what you suggest in your question would be something like the following.  Here I refer to the set of data with A as labeled and the rest as unlabeled.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply an unsupervised learning technique (such as clustering) to the full data set (both labelled and unlabelled).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Transform the labelled data based on the result of your unsupervised learning technique.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply a supervised learning technique with the transformed set (possibly combined with the original set) of labelled data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Generally you would want to apply a cross-validation strategy as well to detect overfitting.&lt;/p&gt;&#xA;" OwnerUserId="17346" LastActivityDate="2016-07-31T00:48:59.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13107" PostTypeId="1" AcceptedAnswerId="14881" CreationDate="2016-07-31T08:46:18.510" Score="1" ViewCount="528" Body="&lt;p&gt;So I'm creating a topics classifier where a document may be tagged for several different topics, let's say - A, B while actually the document belongs to A, B and C. &#xA;In the training stage I want the classifier to learn that the document belongs to A and B but I'm not sure about class C so I don't want it to learn that the document doesn't\does belong to class C. &#xA;Any ideas on how to implement such thing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about adding weights to the output labels (low weight means that there's no way the document belongs to the aspect, high weight means that the document belongs to the aspect for sure and mid weight means I'm not sure (so the penalty in this case will be lower). &lt;/p&gt;&#xA;" OwnerUserId="17443" LastActivityDate="2017-01-02T12:44:23.630" Title="Training multi-label classifier with low quality training set" Tags="|machine-learning|classification|deep-learning|multiclass-classification|multilabel-classification|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13109" PostTypeId="1" AcceptedAnswerId="13583" CreationDate="2016-07-31T13:08:30.500" Score="0" ViewCount="85" Body="&lt;p&gt;What kind of &lt;code&gt;Machine Learning&lt;/code&gt;/&lt;code&gt;Data Mining&lt;/code&gt; packages are available that can easily be scaled on a cluster. For instance &lt;a href=&quot;http://www.h2o.ai/&quot; rel=&quot;nofollow&quot; title=&quot;h2o&quot;&gt;H2O&lt;/a&gt; is one of them because it's running in Java so it can be easily extended to other machines providing parallelism. What other are popular? or even for general Data munging purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Language wise preferably Python or R. (if other please don't be discouraged to include)&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-08-20T21:08:13.717" Title="Adoptive(Machine wise) ML packages" Tags="|machine-learning|python|r|data-mining|data-wrangling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13114" PostTypeId="1" CreationDate="2016-07-31T21:04:38.343" Score="-1" ViewCount="135" Body="&lt;p&gt;I am trying to find the common topics between articles read using the respective tags attached to each article. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background of my mini project&lt;/strong&gt;:&#xA;The problem I am trying to solve involves looking at articles read by a group of readers who have searched the same keyword, in order to gain better understanding on the nature of content they are interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have understood, topic models are commonly used for topic extraction. I'd like some advice on whether this would be suitable for my problem, given that I already have a dataset that contains the tags ('topics') of the articles. Or would a simple probability model be more suitable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Illustration for simple probability model:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keyword searched: &quot;lifestyle&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Articles read by User 1: fashion, health, organic food, clean eating&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Articles read by User 2: fitness&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Articles read by User 3: recipes, diet plan, clean eating&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outcome: 25% clean eating, 12.5% diet plan etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, I hope my explanation isn't confusing!&lt;/p&gt;&#xA;" OwnerUserId="21980" LastEditorUserId="25180" LastEditDate="2017-05-30T08:46:11.720" LastActivityDate="2017-07-29T09:46:16.163" Title="Topic modelling or simple case of calculating probabilities?" Tags="|text-mining|topic-model|probability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13115" PostTypeId="1" CreationDate="2016-07-31T22:29:33.123" Score="1" ViewCount="117" Body="&lt;p&gt;In January, DeepMind published the &lt;a href=&quot;http://www.nature.com/doifinder/10.1038/nature16961&quot; rel=&quot;nofollow&quot;&gt;article&lt;/a&gt; (see &lt;a href=&quot;https://www.youtube.com/watch?v=SUbqykXVx0A&quot; rel=&quot;nofollow&quot;&gt;video&lt;/a&gt;) about its win against Fan Hui, which happend in October 2015. The article and other interviews say, it used 100.000 human games, and then 13.000.000 self-plays. In January, many commentators where sceptical whether AlphaGo can win against Lee Sedol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In March 2016 (5-6 months after Fan Hui's defeat), AlphaGo won against Lee Sedol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was searching through many documents, watched all games, read the papers, but could not find the answer to the question: &lt;em&gt;What did DeepMind do in the 5-6 months between the two games to improve the quality of AlphaGo to that level?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was it only further self-play? If so, are there some numbers of games that were played? Were there algorithmic advances?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to understand that. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="21982" LastEditorUserId="21982" LastEditDate="2016-08-01T07:14:39.607" LastActivityDate="2016-08-01T07:14:39.607" Title="What did DeepMind do with AlphaGo between the Fan Hui and Lee Sedol games?" Tags="|deep-learning|reinforcement-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13116" PostTypeId="2" ParentId="12928" CreationDate="2016-07-31T23:48:47.643" Score="2" Body="&lt;p&gt;Solution 1 and 3 look really similar but I would go for 1 when thinking about the programming paradigm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seeing the classification as a big pipeline using standardization, balancing, and classification, one would not like to re-standardize something in the balancing. Furthermore, the assumption made in 1 is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a practical way, you could use the pipeline offer by the &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/auto_examples/pipeline/plot_pipeline_classification.html#sphx-glr-auto-examples-pipeline-plot-pipeline-classification-py&quot; rel=&quot;nofollow&quot;&gt;imbalanced API&lt;/a&gt; to use SMOTE with a &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-scaler&quot; rel=&quot;nofollow&quot;&gt;scikit-learn normalizer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2016-07-31T23:48:47.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13117" PostTypeId="2" ParentId="12878" CreationDate="2016-07-31T23:58:53.817" Score="2" Body="&lt;p&gt;You need to compute the Euclidean distance between each point and sort theses distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, you can find an implementation in the &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn&quot; rel=&quot;nofollow&quot;&gt;imbalanced-learn&lt;/a&gt; toolbox. More precisely, you can see the different steps of the implementation that you mentioned: (i) &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn/blob/master/imblearn/over_sampling/smote.py#L303&quot; rel=&quot;nofollow&quot;&gt;fit a KNN&lt;/a&gt;, (ii) &lt;a href=&quot;http://s://github.com/scikit-learn-contrib/imbalanced-learn/blob/master/imblearn/over_sampling/smote.py#L307&quot;&gt;find the NN of each sample&lt;/a&gt;, (iii) &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn/blob/master/imblearn/over_sampling/smote.py#L231&quot; rel=&quot;nofollow&quot;&gt;generate a new samples&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2016-07-31T23:58:53.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13120" PostTypeId="1" CreationDate="2016-08-01T10:03:07.317" Score="12" ViewCount="5630" Body="&lt;p&gt;I'm working on Sentiment Analysis over tweets using word2vec as word representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet has different length and the classifier (RandomForest) needs all examples to be of the same size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, for every tweet I'm averaging the vectors of all of its words, to end up with a vector representation of the tweet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;My word2vec model represents each word as vectors of size 300.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have &lt;code&gt;Tweet1&lt;/code&gt; formed by 10 words, and &lt;code&gt;Tweet2&lt;/code&gt; formed by 5 words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what I do is, for &lt;code&gt;Tweet1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(v1_Tweet1 + v2_Tweet1 + ... +v10_Tweet1)/10 = v_Tweet1 #avg vector of 300 elements.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;Tweet2&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(v1_Tweet2 + v2_Tweet2 + ... +v5_Tweet1)/5 = v_Tweet2 #avg vector of 300 elements.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;*Being v1_TweetX the vector of the first word of the TweetX and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works 'fine' but I would like to know what other approaches do you take to overcome the different sizes on the train and text examples for the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="21992" LastEditorUserId="21992" LastEditDate="2016-08-01T10:18:20.133" LastActivityDate="2017-03-03T21:36:19.687" Title="How to overcome training example's different lengths when working with Word Embeddings (word2vec)" Tags="|word-embeddings|word2vec|sentiment-analysis|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13121" PostTypeId="2" ParentId="13120" CreationDate="2016-08-01T10:11:48.263" Score="1" Body="&lt;p&gt;Instead of averaging and getting a single vector for the tweet, you can instead get vectors for each word and for different length vector sizes, padding can be done with zeros.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-01T10:11:48.263" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13123" PostTypeId="1" CreationDate="2016-08-01T11:21:51.223" Score="14" ViewCount="111387" Body="&lt;p&gt;How can I import a .csv file into pyspark dataframes? I even tried to read csv file in Pandas and then convert it to a spark dataframe using createDataFrame, but it is still showing some error. Can someone guide me through this? Also, please tell me how can I import an xlsx file?&#xA;I'm trying to import csv content into pandas dataframes and then convert it into spark data frames, but it is showing the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Py4JJavaError&quot; An error occurred while calling o28.applySchemaToPythonRDD. : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark import SparkContext &#xA;from pyspark.sql import SQLContext &#xA;import pandas as pd &#xA;sqlc=SQLContext(sc) &#xA;df=pd.read_csv(r'D:\BestBuy\train.csv') &#xA;sdf=sqlc.createDataFrame(df) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21998" LastEditorUserId="29169" LastEditDate="2019-06-05T23:11:42.053" LastActivityDate="2019-06-05T23:11:42.053" Title="Import csv file contents into pyspark dataframes" Tags="|pyspark|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="13124" PostTypeId="1" CreationDate="2016-08-01T12:21:09.110" Score="4" ViewCount="334" Body="&lt;p&gt;I just started learning predictive modelling in R , however i do understand some terms below but i lack in making more interpretations ,  just want to know what a pro statistician or Data Scientist interpret from it . How does one look at it ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(model)&#xA;&#xA;Call:&#xA;lm(formula = comp$Minutes ~ comp$Units)&#xA;&#xA;Residuals:&#xA;    Min      1Q  Median      3Q     Max &#xA;-9.2318 -3.3415 -0.7143  4.7769  7.8033 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)    4.162      3.355    1.24    0.239    &#xA;comp$Units    15.509      0.505   30.71 8.92e-13 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 5.392 on 12 degrees of freedom&#xA;Multiple R-squared:  0.9874,    Adjusted R-squared:  0.9864 &#xA;F-statistic: 943.2 on 1 and 12 DF,  p-value: 8.916e-13&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21384" LastEditorUserId="9465" LastEditDate="2016-08-04T10:32:19.200" LastActivityDate="2018-04-12T22:13:31.723" Title="What does this linear regression summary tells us?" Tags="|r|predictive-modeling|linear-regression|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="13125" PostTypeId="2" ParentId="13100" CreationDate="2016-08-01T13:26:41.487" Score="1" Body="&lt;p&gt;Seems like Naive Bayes of &lt;code&gt;e1071&lt;/code&gt; package could help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the model &lt;code&gt;product_id ~ comp_id + area_id&lt;/code&gt; it will construct conditional probabilities that may reveal some pattern in your data. For specifics, however, some sample data is nescessary.&lt;/p&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-08-01T13:26:41.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13126" PostTypeId="1" CreationDate="2016-08-01T14:29:32.783" Score="3" ViewCount="3409" Body="&lt;p&gt;While going through the following &lt;a href=&quot;http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/&quot; rel=&quot;nofollow&quot;&gt;blog&lt;/a&gt; I came across the following code snippet&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import StratifiedKFold&#xA;eval_size = 0.10&#xA;kf = StratifiedKFold(y,round(1./eval_size))&#xA;train_indices, valid_indices = next(iter(kf))&#xA;X_train, y_train = X[train_indices], y[train_indices]&#xA;X_valid, y_valid = X[valid_indices], y[valid_indices]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I am unable to understand how it works. Can anyone please help me with an explanation? Thanks&lt;/p&gt;&#xA;" OwnerUserId="2504" LastActivityDate="2016-09-30T20:55:14.533" Title="Sklearn StratifiedKFold code explanation" Tags="|scikit-learn|sampling|kaggle|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13127" PostTypeId="2" ParentId="13057" CreationDate="2016-08-01T14:41:51.997" Score="1" Body="&lt;p&gt;Try using Platt's trick for probability learning. This is just a logistic regression of the model output to the real probability (see &lt;a href=&quot;https://www.csie.ntu.edu.tw/~cjlin/papers/plattprob.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt; )&lt;/p&gt;&#xA;" OwnerUserId="21596" LastActivityDate="2016-08-01T14:41:51.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13129" PostTypeId="1" CreationDate="2016-08-01T14:44:46.700" Score="3" ViewCount="210" Body="&lt;p&gt;The calibration graph is the predicted versus actual probability(see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html&lt;/a&gt;). Is it possible to optimize the linearity of that curve in terms of a loss function? Does the log-loss actually optimizes this curve in terms of KL divergence? &lt;/p&gt;&#xA;" OwnerUserId="21596" LastEditorUserId="55122" LastEditDate="2019-05-19T04:44:07.343" LastActivityDate="2020-01-17T06:00:17.920" Title="which loss function (if any) optimizes the calibration graph" Tags="|machine-learning|loss-function|probability|probability-calibration|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13131" PostTypeId="2" ParentId="13123" CreationDate="2016-08-01T16:24:03.550" Score="1" Body="&lt;p&gt;I have in my local directory a file 'temp.csv'.  From there, using a local instance I do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from pyspark import SQLContext&#xA;&amp;gt;&amp;gt;&amp;gt; from pyspark.sql import Row&#xA;&amp;gt;&amp;gt;&amp;gt; sql_c = SQLContext(sc)&#xA;&amp;gt;&amp;gt;&amp;gt; d0 = sc.textFile('./temp.csv')&#xA;&amp;gt;&amp;gt;&amp;gt; d0.collect()&#xA;[u'a,1,.2390', u'b,2,.4390', u'c,3,.2323']&#xA;&amp;gt;&amp;gt;&amp;gt; d1 = d0.map(lambda x: x.split(',')).map(lambda x: Row(label = x[0], number = int(x[1]), value = float(x[2])))&#xA;&amp;gt;&amp;gt;&amp;gt; d1.take(1)&#xA;[Row(label=u'a', number=1, value=0.239)]&#xA;&amp;gt;&amp;gt;&amp;gt; df = sql_c.createDataFrame(d1)&#xA;&amp;gt;&amp;gt;&amp;gt; df_cut = df[df.number&amp;gt;1]&#xA;&amp;gt;&amp;gt;&amp;gt; df_cut.select('label', 'value').collect()&#xA;[Row(label=u'b', value=0.439), Row(label=u'c', value=0.2323)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So d0 is the raw text file that we send off to a spark RDD.  In order for you to make a data frame, you want to break the csv apart, and to make every entry a Row type, as I do when creating d1.  The last step is to make the data frame from the RDD.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-08-01T16:24:03.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13132" PostTypeId="2" ParentId="13109" CreationDate="2016-08-01T16:34:06.410" Score="1" Body="&lt;p&gt;SPARKR, &lt;a href=&quot;http://spark.apache.org/docs/latest/sparkr.html&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/docs/latest/sparkr.html&lt;/a&gt;, is designed to support parallel processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional resources are summarized here:  &lt;a href=&quot;https://cran.r-project.org/web/views/HighPerformanceComputing.html&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/views/HighPerformanceComputing.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21951" LastEditorUserId="21951" LastEditDate="2016-08-01T22:18:13.850" LastActivityDate="2016-08-01T22:18:13.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13134" PostTypeId="2" ParentId="12918" CreationDate="2016-08-01T18:47:12.830" Score="-1" Body="&lt;p&gt;bins=10 parameter is actually added in latest version (ggplot 0.11) &lt;a href=&quot;https://github.com/yhat/ggplot/issues/513&quot; rel=&quot;nofollow&quot;&gt;https://github.com/yhat/ggplot/issues/513&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21692" LastActivityDate="2016-08-01T18:47:12.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13135" PostTypeId="2" ParentId="13126" CreationDate="2016-08-01T19:40:12.797" Score="2" Body="&lt;p&gt;A KFold split will take the data and split it however many times you designate.   StratifiedKFold is used in order to ensure that your training and validation datasets each contain the same percentage of classes (see sklearn documentation for more). The function StratifiedKFold takes two arguments, the array of labels (for binary classification this would be an array of 1's and 0's) and the number of folds.  They have designated the number of folds as 1./eval_size where eval_size = 0.10.  So this is a 10-fold validation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_indices, valid_indices = next(iter(kf))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This line derives the indices in order to split the data into the train/validation data sets.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we have the indices, we use those indices to actually split the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is interesting that they wrote next(iter(kf)) and then input the indices into the dataset when they could use sklearn.cross_validation.train_test_split.  train_test_split is just a wrapper for next(iter(kf)), but it is more readable and it's already a function in sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-08-01T19:40:12.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13136" PostTypeId="2" ParentId="13057" CreationDate="2016-08-01T20:21:02.290" Score="0" Body="&lt;p&gt;I believe for your case a better optimization metric like &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F1 score&lt;/a&gt; will work better since you have unbalanced labels. &lt;/p&gt;&#xA;" OwnerUserId="22008" LastActivityDate="2016-08-01T20:21:02.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13138" PostTypeId="1" AcceptedAnswerId="13179" CreationDate="2016-08-01T22:38:16.490" Score="19" ViewCount="17057" Body="&lt;p&gt;While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text &lt;strong&gt;word by word&lt;/strong&gt; and others &lt;strong&gt;character by character&lt;/strong&gt; without actually stating why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what is the difference between RNN models that predict text &lt;strong&gt;per-word&lt;/strong&gt; basis and the ones that predict text &lt;strong&gt;per-char&lt;/strong&gt; basis? Do word-based RNN require a bigger corpus size? Do char-based RNN generalize better? Maybe the only difference is input representation (one-hot encoding, word embeddings)? Which ones to choose for text generation? &lt;/p&gt;&#xA;" OwnerUserId="22012" LastActivityDate="2022-08-26T07:48:12.663" Title="What is the difference between word-based and char-based text generation RNNs?" Tags="|machine-learning|neural-network|nlp|rnn|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13139" PostTypeId="2" ParentId="13104" CreationDate="2016-08-02T02:52:09.177" Score="5" Body="&lt;p&gt;big question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok so here's a few things I'd look at if I was you..&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Have you tried any feature engineering ?(it sounds like you've just used the features in the training set but I can't be 100%)&lt;/li&gt;&#xA;&lt;li&gt;Random Forests should do pretty well, but maybe try xgboost too? It's quite good at everything on Kaggle. SVM's could be worth a go also if you're thinking of stacking/ensembling.&lt;/li&gt;&#xA;&lt;li&gt;Check out some of the tutorials around this competition. There's hundreds of them and most of them are great.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Links:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://trevorstephens.com/kaggle-titanic-tutorial/getting-started-with-r/&quot; rel=&quot;noreferrer&quot;&gt;R #1 (my favourite)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.numbrcrunch.com/blog/the-good-ol-titanic-kaggle-competition-pt-1&quot; rel=&quot;noreferrer&quot;&gt;R #2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ultravioletanalytics.com/2014/10/30/kaggle-titanic-competition-part-i-intro/&quot; rel=&quot;noreferrer&quot;&gt;Python #1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/agconti/kaggle-titanic&quot; rel=&quot;noreferrer&quot;&gt;Python #2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...Hopefully this helps&lt;/p&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-08-02T02:52:09.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13142" PostTypeId="2" ParentId="13114" CreationDate="2016-08-02T05:20:07.473" Score="0" Body="&lt;p&gt;I can think of multiple approaches. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can use a simple probability model, train the model and extract the top features in order to get the nature of the interested content.&lt;/li&gt;&#xA;&lt;li&gt;As you suggested, Topic Modeling also might be useful here. You can check &lt;a href=&quot;http://arxiv.org/abs/1604.00783&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; out and maybe use a variant of it based on your needs.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-02T05:20:07.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13143" PostTypeId="1" AcceptedAnswerId="13145" CreationDate="2016-08-02T08:29:03.307" Score="0" ViewCount="81" Body="&lt;p&gt;I am downloading the data set for the Kaggle competition on the titanic.&lt;/p&gt;&#xA;&lt;p&gt;If I use the following code :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;if (!file.exists(&amp;quot;data&amp;quot;)){&#xA;  dir.create(&amp;quot;data&amp;quot;)&#xA;}&#xA;fileUrl &amp;lt;- 'https://www.kaggle.com/c/titanic/download/train.csv'&#xA;&#xA;download.file(fileUrl, destfile='./data/train.csv') &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I get a 14kb file, however,&#xA;Paste &lt;a href=&quot;https://www.kaggle.com/c/titanic/download/train.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Url&lt;/a&gt; directly in your browser and you will download the correct file about 60kb.&lt;/p&gt;&#xA;" OwnerUserId="18811" LastEditorUserId="26019" LastEditDate="2020-08-14T09:15:02.550" LastActivityDate="2020-08-14T09:15:02.550" Title="Depending upon how I download, I get two different files" Tags="|r|" AnswerCount="1" CommentCount="0" ClosedDate="2016-08-02T16:42:05.060" ContentLicense="CC BY-SA 4.0" />
  <row Id="13144" PostTypeId="2" ParentId="13120" CreationDate="2016-08-02T09:37:32.030" Score="1" Body="&lt;p&gt;In my work, I have done the same way by averaging the word vectors. But there is another idea I wanted to try. It is with the help of POS tags.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First construct a most complicated sentence with all the POS tags as possible and set these POS tags as a template. For each sentence in the twitter corpus, POS tag all the words in it and apply those word vectors respective to the POS tags in the template. So, the unseen POS tags will have zeros.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: &lt;code&gt;NNP PDT DT NNS VB MD JJS CC PRP RBS&lt;/code&gt; is the template. Thus each position will contain 300-dimensional vector totally a 3000-dimensional vector. And if  the first tweet's POS tags are &lt;code&gt;NNP VB JJS PRP&lt;/code&gt;, then the word vectors are applied on these positions and have vectors on &lt;code&gt;NNP VB JJS PRP&lt;/code&gt; positions and 300-dimensional zero vectors on other positions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method not only solves the problem of representing the sentence by a single vector but also preserves the syntactic structure of the sentence by positioning in the right POS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ofcourse, there will be problems when there are more than one POS tags or jumbled positions of the tags. This is just one of the possibility.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-08-02T09:37:32.030" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13145" PostTypeId="2" ParentId="13143" CreationDate="2016-08-02T10:17:05.937" Score="0" Body="&lt;p&gt;This code works for sites where you don't need to be logged on. The Kaggle link only gives you the file when you are logged on to Kaggle. The file that is created with the code only contains html / javascript code of the Kaggle page. &lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-08-02T10:17:05.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13146" PostTypeId="2" ParentId="10508" CreationDate="2016-08-02T13:58:38.687" Score="1" Body="&lt;p&gt;I think, you need to look carefully through popular ranking algorithms. I would suggest to start from &lt;a href=&quot;https://en.wikipedia.org/wiki/Elo_rating_system&quot; rel=&quot;nofollow&quot;&gt;Elo&lt;/a&gt;, at a glance, it could help you to solve the problem. I worked only with &quot;classical&quot; &quot;1-vs.-1&quot; version. But here are some thoughts on your case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider every bundle as a team of two players; every customer's choice as a duel. The main issue here would be the &quot;teams randomness&quot;, as you have no constant team compositions. But you can try to solve it, using average ratings of team members. With the large number of duels and &quot;teams&quot; you will get enough data to predict the outcome for brand new mixtures of fruits.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-08-02T13:58:38.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13147" PostTypeId="1" CreationDate="2016-08-02T14:20:43.947" Score="2" ViewCount="48" Body="&lt;p&gt;I have data for a product rating survey, which requires the respondents to rate a product in five levels: Very Bad, Bad, Regular, Good and Very Good. This survey was applied to several communities of clients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After taking a look on the data, I observed that a lot of communities have an overwhelming &quot;Regular&quot; preference for the product. Later I found out that &quot;Regular&quot;, was the default response for the survey question. Apparently, in some communities user's just didn't bother and left all the questions in the default value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a proper way of identifying the communities that are indifferent to the survey, so they can be excluded? I was thinking in a simple threshold-based approach (If the community has more than 50% of non-default responses, it is not indifferent), but I was wondering if there is a more principled way.&lt;/p&gt;&#xA;" OwnerUserId="22030" LastActivityDate="2016-08-03T04:22:31.637" Title="Remove indiferent respondents in survey data" Tags="|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13148" PostTypeId="1" CreationDate="2016-08-02T15:25:09.123" Score="2" ViewCount="350" Body="&lt;p&gt;I'm a second year pure maths, applied maths and computer science student. I have taken up a research course and was given the topic to focus on facial recognition using deep learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have done a fair deal of reading on the topic (looking at older methods such as eigenfaces and fisherfaces all the way to newer methods such as deepface). I also feel comfortable with the maths and algorithms underlying ANNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering whether someone in the community could maybe point me in the direction in what I (as an undergraduate) could take a deeper look at in the category of facial recognition using deep learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought it might be interesting to pursue recognising faces given a few images of a face i.e. &amp;lt; 5. Maybe someone has a better suggestion?&lt;/p&gt;&#xA;" OwnerUserId="22034" LastActivityDate="2016-08-02T17:59:40.360" Title="Deep learning facial recognition research project" Tags="|machine-learning|neural-network|deep-learning|research|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13150" PostTypeId="1" CreationDate="2016-08-01T14:48:46.027" Score="1" ViewCount="161" Body="&lt;p&gt;I am searching for a feature selection algorithm able to select the minimum number (minimum redundancy) of relevant variables (maximum relevance) with respect to a categorical response variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found mRMR [1] but I am unsure on how it behaves with variable interactions (I think it works better on single variables than on their combinations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically I am searching for an algorithm that is able to retain all variable that are relevant when linearly combined one with another (e.g. summing/subtracting their value). I cannot combine them first and then run mRMR afterwards since the number of combinations is huge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Peng, Hanchuan, Fuhui Long, and Chris Ding. &quot;&lt;a href=&quot;http://home.penglab.com/papersall/docpdf/2005_TPAMI_FeaSel.pdf&quot; rel=&quot;nofollow&quot;&gt;Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy&lt;/a&gt;,&quot; IEEE Transactions on pattern analysis and machine intelligence 27.8 (2005): 1226-1238.&lt;/p&gt;&#xA;" OwnerUserId="133" OwnerDisplayName="fbrundu" LastEditorUserId="381" LastEditDate="2016-08-02T17:41:59.787" LastActivityDate="2016-08-02T17:41:59.787" Title="Feature selection with linear interaction between variables and correlation with categorical response variable" Tags="|feature-selection|dimensionality-reduction|mutual-information|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13151" PostTypeId="1" AcceptedAnswerId="30408" CreationDate="2016-08-02T15:47:47.503" Score="26" ViewCount="30540" Body="&lt;p&gt;Does the random forest implementation in scikit-learn use mean accuracy as its scoring method to estimate generalization error with out-of-bag samples? This is not mentioned in the documentation, but the &lt;code&gt;score()&lt;/code&gt; method reports the mean accuracy.&lt;/p&gt;&#xA;&lt;p&gt;I have a highly unbalanced dataset, and I am using AUC of ROC as my scoring metric in the grid search. Is there a way to tell the classifier to use the same scoring method on the OOB samples as well?&lt;/p&gt;&#xA;" OwnerUserId="21433" LastEditorUserId="29169" LastEditDate="2021-02-09T20:16:37.850" LastActivityDate="2021-02-09T20:16:37.850" Title="RandomForestClassifier OOB scoring method" Tags="|random-forest|scikit-learn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13152" PostTypeId="2" ParentId="13057" CreationDate="2016-08-02T17:14:35.800" Score="6" Body="&lt;p&gt;Daniel,&#xA;What you did goes under the name of &quot;oversampling.&quot; There is a sample of some &quot;real&quot; population, and you replace it with a sample from a &quot;manufactured&quot; population. The problem that makes sense in application is the estimation of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P_r(Y=1|X) = \text{probability of response=1 in the $\mathbf{real}$ population given the predictor $X$} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but by using an oversample you are estimating&#xA;$$P_m(Y=1|X) = \text{probability of response=1 in the $\mathbf{manufactured}$ population given the predictor $X$}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two probabilities are related. I'll worked the details. I'll pretend the predicor $X$ is discrete. If $X$ takes numerical values one has to replace some probabilities by probability densities.&#xA;$$\dots\dots\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To simplify the notation, let $\pi_1 = P_r(Y=1)$ and $\mu_1 = P_m(Y=1)$ be the probabilities of response in the real and manufactured populations, let &#xA;$$&#xA;L_r = &#xA;\frac{P_r(X=x|Y=1)}{P_r(X=x|Y=0)} = \frac{\frac{P_r(Y=1|X=x)}{P_r(Y=0|X=x)}}{\frac{\pi_1}{1-\pi_1} }$$&#xA;be the odds ratio of $Y=1$, i.e.: the ratio of the odds among cases with $X=x$ and the odds in the general $\mathbf{real}$ population. Finally, let $L_m$ be the corresponding ratio in the $\mathbf{manufactured}$ population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By Bayes' Theorem:&#xA;$$ P_r(Y=1|X=x) = \frac{P_r(Y=1,X=x)}{P_r(X=x)} =   \\&#xA;=\frac{P_r(X=x|Y=1)\space \pi_1}{P_r(X=x|Y=1)\space\pi_1 + P_r(X=x|Y=0)\space (1 - \pi_1)} = \\&#xA;=\frac{L_r\space \pi_1}{L_r\space\pi_1 + \space (1 - \pi_1)} \tag{1}&#xA;$$&#xA;In a similar way, we get an analogous result for the manufactured population:&#xA;$$&#xA;P_m(Y=1|X=x) = \frac{L_m\space \mu_1}{L_m\space\mu_1 + \space (1 - \mu_1)} \tag{2}&#xA;$$&#xA;Since the manufactured sample is a random sample, stratified by $Y$, the conditional distribution of X within responders is the same as in the real population. Same as for non responders, i.e.:&lt;br&gt;&#xA;$$&#xA; P_r(X=x|Y=j) = P_r(X=x|Y=j) $$&#xA;for $j=0,1.$ If the sample stratified by values of Y was anything other than random sample these would not be true.&#xA;It follows that $\boxed{ L_r = L_m }$. Next we solve for $L_m$ in terms of $P_m(Y=1|X)$ from (2) and replace in (1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\dots\dots\dots$$&#xA;$\mathbf{Digression}$:&#xA;Here is my easy way to carry the steps, without mess. Two non-zero vectors $\mathbf{v_1}$, $\mathbf{v_2}$ are parallel iff there is $\lambda \ne 0$ such that $\mathbf{v_1}=\lambda \mathbf{v_2}.$ Below I will use this idea, and I will not care about the exact value of $\lambda$, so I will be using &quot;$\lambda$&quot; as a short-hand for &quot;$\mathbf{\text{some non-zero mumber}}$.&quot;&#xA;$\mathbf{\text{End Digression}}$:&#xA;$$\dots\dots\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easy way to solve is to observed that for non-zero messy values of $\lambda$ (not the same in each occurrence!) one has:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{bmatrix}&#xA; P_r(Y=1|X) \\&#xA; 1 \\&#xA;\end{bmatrix} = \lambda &#xA;\begin{bmatrix}&#xA;       \pi_1 &amp;amp;0\\&#xA;      \pi_1      &amp;amp;1-\pi_1 &#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA;L_r\\&#xA;1 &#xA;\end{bmatrix} ,&#xA;$$&#xA;and&#xA;$$&#xA;\begin{bmatrix}&#xA; P_m(Y=1|X) \\&#xA; 1 \\&#xA;\end{bmatrix} = \lambda &#xA;\begin{bmatrix}&#xA;      \mu_1 &amp;amp;0 \\&#xA;      \mu_1      &amp;amp;1-\mu_1 &#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA;L_m\\&#xA;1 &#xA;\end{bmatrix}&#xA;. $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{bmatrix}&#xA;L_m\\&#xA;1 &#xA;\end{bmatrix}&#xA; = \lambda &#xA;\begin{bmatrix}&#xA;      \mu_1  &amp;amp;0 \\&#xA;      \mu_1      &amp;amp;1-\mu_1 &#xA;\end{bmatrix}^{-1} &#xA;\begin{bmatrix}&#xA; P_m(Y=1|X) \\&#xA; 1 \\&#xA;\end{bmatrix} ,&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and so (remember that here $\lambda$ stands for &quot;some non-zero number&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \space &#xA;\begin{bmatrix}&#xA;  P_r \\&#xA;   1     &#xA;\end{bmatrix} = \lambda &#xA;\begin{bmatrix}&#xA;       \pi_1 &amp;amp;0 \\&#xA;      \pi_1      &amp;amp;1-\pi_1 &#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA;L_r \\&#xA;1 &#xA;\end{bmatrix} = \\&#xA;\text{              }=\lambda &#xA;\begin{bmatrix}&#xA;      \pi_1      &amp;amp;0    \\&#xA;      \pi_1      &amp;amp;1-\pi_1 &#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA;       \mu_1     &amp;amp;0     \\&#xA;      \mu_1      &amp;amp;1-\mu_1 &#xA;\end{bmatrix}^{-1} &#xA;\begin{bmatrix}&#xA; P_m \\&#xA; 1 \\&#xA;\end{bmatrix} = \\&#xA;\text{              }=\lambda &#xA;\begin{bmatrix}&#xA;      \pi_1 (1- \mu_1)        &amp;amp;0 \\&#xA;      \pi_1  - \mu_1     &amp;amp; \mu_1 (1- \pi_1)&#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA; P_m \\&#xA; 1 \\&#xA;\end{bmatrix} = &#xA;\lambda &#xA;\begin{bmatrix}&#xA;      \pi_1  (1-\mu_1)  P_m \\&#xA;      (\pi_1  - \mu_1) \; P_m   +  \mu_1 (1- \pi_1)&#xA;\end{bmatrix}. &#xA;$&lt;br&gt;&#xA;Thus,&#xA;$$ P_r = \frac{\pi_1 (1- \mu_1) P_m}{(\pi_1 - \mu_1) \; P_m   +  \mu_1 (1- \pi_1) } $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\dots\dots\dots$$ &#xA;Example: Let's work the details of a Binomial model, &#xA;$$P_m(Y=1|X) = \frac{e^{\beta_0 + \beta X}}{1+e^{\beta_0 + \beta X}} $$&#xA;or in the &quot;where $\lambda$ is some non-zero scalar&quot; notation (I would not had digressed before if I did not had ulterior motive.. :) ):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA; \begin{bmatrix}&#xA; P_m \\&#xA; 1 \\&#xA;\end{bmatrix} = &#xA;\lambda &#xA;\begin{bmatrix}&#xA;e^{\beta_0 + \beta X} \\&#xA;1 + e^{\beta_0 + \beta X}&#xA;\end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the implied model in the real population?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \space &#xA;\begin{bmatrix}&#xA;     P_r \\&#xA;      1     &#xA;\end{bmatrix} = &#xA;\lambda &#xA;\begin{bmatrix}&#xA;      \pi_1 (1- \mu_1)   &amp;amp;0 \\&#xA;      \pi_1  - \mu_1     &amp;amp;\mu_1 (1- \pi_1)&#xA;\end{bmatrix} &#xA;\begin{bmatrix} &#xA;  P_m \\&#xA;   1 &#xA;\end{bmatrix} =&#xA;$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\;$&#xA;$&#xA;= \lambda &#xA;\begin{bmatrix}&#xA;      \pi_1 (1- \mu_1)   &amp;amp;0 \\&#xA;      \pi_1  - \mu_1     &amp;amp;\mu_1 (1- \pi_1)&#xA;\end{bmatrix} &#xA;\begin{bmatrix}&#xA;e^{\beta_0 + \beta X} \\&#xA;1 + e^{\beta_0 + \beta X}&#xA;\end{bmatrix}=&#xA;$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\;$&#xA;$&#xA;= \lambda &#xA;\begin{bmatrix}&#xA;      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} \\&#xA;      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} + \mu_1 (1- \pi_1)&#xA;\end{bmatrix} = &#xA;\lambda&#xA;\begin{bmatrix}&#xA;\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X} \\&#xA;1 + \frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X}&#xA;\end{bmatrix}&#xA;.$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we let $\tau = \ln(\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)})$, we can absorve this constant in the exponents to get:&#xA;$$ &#xA;\begin{bmatrix}&#xA;     P_r \\&#xA;      1     &#xA;\end{bmatrix} = &#xA;\lambda &#xA;\begin{bmatrix}&#xA;e^{\tau + \beta_0 + \beta X} \\&#xA;1 + e^{\tau + \beta_0 + \beta X}&#xA;\end{bmatrix}&#xA;.$$&lt;br&gt;&#xA;Taking the ratio and simplifying the non-zero constant in numerator and denominator we get that fitting a logistic model to the manufactured population results in an implied logistic model for the real population, $\mathbf{\text{with the same coefficients for X}}$ and with a difference in the constant (in the logistic model) given by:&#xA;$$ \beta_{real} = \tau + \beta_0 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\dots$$&#xA;Note that, according to your reference, the ratio of $\gamma_1 = Pr(Z=1|Y=1)$ and $\gamma_0 = Pr(Z=1|Y=0)$ should come up. Indeed:&lt;br&gt;&#xA;$$ \gamma_1 = Pr(Z=1|Y=1) = \frac{P(Z=1,Y=1)}{P(Y=1)} = \frac{P_r(Y=1|Z=1)P_r(Z=1)}{P_r(Y=1)} = \frac{P_m(Y=1)}{P_r(Y=1)} P_r(Z=1)= &#xA;\frac{\mu_1}{\pi_1}P_r(Z=1) $$&#xA;likewise (i.e. change Y to 1-Y),&#xA;$$ \gamma_0 = \frac{1-\mu_1}{1-\pi_1}P_r(Z=1) $$ &#xA;so &#xA;$$ \ln(\frac{\gamma_1}{\gamma_0}) = - \ln(\frac{\pi_1 (1-\mu_1)}{\mu_1 (1-\pi_1}) = \tau $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\dots\dots\dots$$&#xA;Notes for full disclosure: I worked with the probability model. When one works with finite samples the example above suggests two ways of estimating the coefficients:&#xA;* estimate coefficients using the sample from the real population&#xA;* estimate coefficients using the manufactored populations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It terns out that this two estimators are not the same (it is obvious if one consideres one estimator is based on more cases than the other). Both estimators are asymtopically consistent, but it can be shown the one based on the manufactored population is more biased (forgot the reference :( ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the data science space we are more concern with the quality of the predictions than the parameters of the parameters used to make those predictions, so as long as you check results properly (e.g.: using a testing set to build models and another to validate them), the bias in the parameters should not deter us from using oversampling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\dots\dots\dots$$&lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-02T17:14:35.800" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13153" PostTypeId="2" ParentId="13148" CreationDate="2016-08-02T17:59:40.360" Score="3" Body="&lt;p&gt;First, &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow&quot;&gt;tensorflow&lt;/a&gt; would be a great resource as it is gaining a userbase with exponential growth. Tensorflow has gpu support for single node calculations, and has recently added a &lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/how_tos/distributed/index.html&quot; rel=&quot;nofollow&quot;&gt;distributed&lt;/a&gt; version for larger calculations.  There are some tutorials on &lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/tutorials/deep_cnn/index.html&quot; rel=&quot;nofollow&quot;&gt;convolution ANN&lt;/a&gt;, which will probably be the direction you will go with a computer vision problem. Also, &lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/tensorflow-machine-learning-cookbook&quot; rel=&quot;nofollow&quot;&gt;some&lt;/a&gt; books &lt;a href=&quot;https://gumroad.com/l/ORjB&quot; rel=&quot;nofollow&quot;&gt;are&lt;/a&gt; coming &lt;a href=&quot;https://www.manning.com/books/machine-learning-with-tensorflow&quot; rel=&quot;nofollow&quot;&gt;soon&lt;/a&gt; and some are in alpha.  But enough of plugging a particular library!&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A perspective based facial recognition problem would be interesting if data were available e.g. with two perspectives on faces, do recognition on a third perspective.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A relative based facial recognition problem could be interesting, again, if data were available e.g. given two parents, try to find the child.  One could probably scrape facebook to get a reasonable data set for this one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Blending and aging faces and controlling for sex (hair length, facial hair, etc) for problems where a kid has been abducted and you have parents pictures and sibling pictures and the child has grown up.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A simple aging recognition problem could be interesting... given baby pictures, can you recognize the adult or vice versa.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just some thoughts, the possibilities are endless. I suggest starting with a very simple stub of a problem and then slowly adding sophistication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-08-02T17:59:40.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13154" PostTypeId="2" ParentId="13123" CreationDate="2016-08-02T20:39:20.117" Score="0" Body="&lt;p&gt;You can use the package &lt;a href=&quot;https://github.com/databricks/spark-csv&quot; rel=&quot;nofollow&quot;&gt;spark-csv&lt;/a&gt; by DataBricks that does a lot of things for you automatically, like taking care of the header, use escape characters, automatic schema inferring etcetera. Starting from Spark 2.0 there is an inbuilt function for dealing with CSVs.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-02T20:39:20.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13156" PostTypeId="2" ParentId="12237" CreationDate="2016-08-02T23:51:29.193" Score="4" Body="&lt;p&gt;You can use the SQL interface to get what you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df.selectExpr(&quot;from_utc_timestamp(start_time, tz) as testthis&quot;).show()&#xA;+--------------------+&#xA;|            testthis|&#xA;+--------------------+&#xA;|2015-01-01 16:59:...|&#xA;|2015-01-02 18:00:...|&#xA;|2015-01-02 17:59:...|&#xA;|2015-03-02 07:59:...|&#xA;|2015-03-16 08:15:...|&#xA;|2015-10-02 11:59:...|&#xA;|2015-11-16 10:58:...|&#xA;|2015-03-02 08:59:...|&#xA;|2015-03-16 08:15:...|&#xA;|2015-10-02 11:59:...|&#xA;|2015-11-16 11:58:...|&#xA;+--------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-02T23:51:29.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13157" PostTypeId="1" CreationDate="2016-08-03T01:02:30.837" Score="2" ViewCount="107" Body="&lt;p&gt;I am working on a hobby project where I have a data related to &lt;code&gt;financial trades(such as stock trades)&lt;/code&gt;. Now I want to analyse this data and extract possible suspicious patterns and trades. Since I am new to Data Science so I would love to get few suggestions regarding what are some effective algorithms(&lt;code&gt;machine learning&lt;/code&gt;) that can be relevant to such a scenario. One category that I think is using a possible &lt;code&gt;outlier detection&lt;/code&gt; algorithm but I am unable to determine which one to use. I would also appreciate few pointers related to what my approach should be.&lt;/p&gt;&#xA;" OwnerUserId="22045" LastEditorUserId="22045" LastEditDate="2016-08-03T02:17:52.327" LastActivityDate="2016-08-03T05:48:54.040" Title="Some algorithms and approach for identification of specific patterns?" Tags="|machine-learning|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13158" PostTypeId="1" CreationDate="2016-08-03T02:12:00.623" Score="1" ViewCount="398" Body="&lt;p&gt;I am doing experiments on the high-dimensional regression. However, it is hard to obtain the practical or synthetic high-dimensional data. I have checked on UCI website as well as some papers with related to this topic. Currently, I have not obtained satisfactory datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone give me some hints on how to search and obtain the high-dimensional data? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="9893" LastActivityDate="2016-08-03T17:07:11.590" Title="about high-dimensional regression data" Tags="|bigdata|dimensionality-reduction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13159" PostTypeId="2" ParentId="13158" CreationDate="2016-08-03T03:18:15.913" Score="1" Body="&lt;p&gt;Please check this website: &lt;strong&gt;&lt;a href=&quot;http://dataportals.org/&quot; rel=&quot;nofollow&quot;&gt;http://dataportals.org/&lt;/a&gt;&lt;/strong&gt;, its a collection of data portals from around the world. Might find here what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="21414" LastActivityDate="2016-08-03T03:18:15.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13160" PostTypeId="2" ParentId="13157" CreationDate="2016-08-03T03:24:07.223" Score="3" Body="&lt;p&gt;I suggest , go for Anomaly detection:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anomaly Detection is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it gaussian using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example : replace original feature x -&gt; Log(x) or feature x -&gt; (x)^4/3 etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.F1-Score is a pretty good metric to evaluate the performance of the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a proper hold on this topic , I also suggest go through  anomaly detection course videos by &lt;strong&gt;Andrew NG&lt;/strong&gt; in machine learning on Coursera. Free course and very nicely made.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The well known techniques for the same are:&#xA;Density-based techniques (k-nearest neighbor),Support vector Machines,Cluster Analysis Anomaly detection. People also use Fuzzy logic based techniques.&#xA;And for python I suggest use libraries like &lt;strong&gt;scikit-learn&lt;/strong&gt; and &lt;strong&gt;TensorFlow&lt;/strong&gt; , they have superb implementations of usable machine learning algorithms.Heres a video link from a googler,giving general overview for the same :&lt;a href=&quot;https://www.youtube.com/channel/UCwUtgW0JCPFjLbKx4VVr3bw&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/channel/UCwUtgW0JCPFjLbKx4VVr3bw&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21414" LastEditorUserId="21414" LastEditDate="2016-08-03T05:48:54.040" LastActivityDate="2016-08-03T05:48:54.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13161" PostTypeId="2" ParentId="13147" CreationDate="2016-08-03T04:22:31.637" Score="1" Body="&lt;p&gt;I'd recommend converting your responses to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Likert_scale&quot; rel=&quot;nofollow&quot;&gt;Likert Scale&lt;/a&gt; and going from there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a widely researched area so there should be plenty of research on how to best deal with 'so-so' responses.&lt;/p&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-08-03T04:22:31.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13162" PostTypeId="2" ParentId="12964" CreationDate="2016-08-03T07:57:57.797" Score="16" Body="&lt;p&gt;As the helpful comments in that function say,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The definition of cell in this package differs from the definition used in the&#xA;    literature. In the literature, cell refers to an object with a single scalar&#xA;    output. The definition in this package refers to a horizontal array of such&#xA;    units.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In essence, the layer will contain multiple parallel LSTM units, structurally identical but each eventually &quot;learning to remember&quot; some different thing.&lt;/p&gt;&#xA;" OwnerUserId="15518" LastActivityDate="2016-08-03T07:57:57.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13163" PostTypeId="1" AcceptedAnswerId="13164" CreationDate="2016-08-03T10:45:40.823" Score="2" ViewCount="1051" Body="&lt;p&gt;So I'm a little bit baffled. I have just started working with the Keras framework for Python (which is awesome by the way!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However just trying a few simple test of neural networks has got me a bit confused. I initially tried to classify the Iris data as it was a small, quick and simple dataset. However when I constructed a neural network for it (4 input dimensions, 8 node hidden layer, 3 node output layer for binary classification of the 3 classes). This however didn't produce any predictive powers at all (100 samples in the training set, a separate 50 in the test set. Both had been shuffled so as to include a good distribution of classes in each).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I thought I was doing something wrong, but I thought I'd give the network a quick test on the MNIST dataset just in case. So I used basically the exact same network (other than changing the input dimensions to 784, hidden nodes to 30, and output nodes to 10 for the binary encoded 0-9 output). And this worked perfectly! With a 97% accuracy rate and a 5% loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I'm not sure why the Iris dataset isn't playing ball, does anyone have any clues? I've tried changing the number of hidden layer nodes and also normalized the X input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an output I've produced when I've manually run some of the test set through the trained Iris model. As you can see it's basically producing a uniform random guess.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Target: [ 1.  0.  0.]  | Predicted: [[ 0.44635904  0.43874186  0.45729554]]&#xA;Target: [ 0.  0.  1.]  | Predicted: [[ 0.44618103  0.43869928  0.45735642]]&#xA;Target: [ 0.  1.  0.]  | Predicted: [[ 0.44612524  0.43863046  0.45729461]]&#xA;Target: [ 0.  0.  1.]  | Predicted: [[ 0.44617626  0.43870446  0.45736298]]&#xA;Target: [ 0.  0.  1.]  | Predicted: [[ 0.44613886  0.43865535  0.45731983]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here's my full code for the Iris MLP (the MNIST one is essentially the same)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import random&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Activation&#xA;from sklearn.datasets import load_iris&#xA;from sklearn import preprocessing&#xA;&#xA;# Model Layers Defnition&#xA;# Input layer (12 neurons), Output layer (1 neuron)&#xA;model = Sequential()&#xA;model.add(Dense(8, input_dim=4, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(3, init='uniform', activation='sigmoid'))&#xA;&#xA;&#xA;model.compile(optimizer='rmsprop',&#xA;              loss='binary_crossentropy',&#xA;              metrics=['accuracy'])&#xA;&#xA;# Iris has 150 samples, each sample has 4 dimensions&#xA;iris = load_iris()&#xA;iris_xy = zip(iris.data, iris.target)&#xA;random.shuffle(iris_xy) # Iris data is sequential in it's labels&#xA;iris_x, iris_y = zip(*iris_xy)&#xA;iris_x = preprocessing.normalize(np.array(iris_x))&#xA;iris_y = np.array(iris_y)&#xA;&#xA;# Encode decimal numbers to array&#xA;iris_y_enc = np.zeros(shape=(len(iris_y),3))&#xA;for i, y in enumerate(iris_y):&#xA;    iris_y_enc[i][y] = 1&#xA;&#xA;train_data = np.array(iris_x[:100]) # 100 samples for training&#xA;test_data  = np.array(iris_x[100:]) # 50 samples for testing&#xA;train_targets = np.array(iris_y_enc[:100])&#xA;test_targets  = np.array(iris_y_enc[100:])&#xA;&#xA;model.fit(train_data, train_targets, nb_epoch=10)&#xA;&#xA;#score = model.evaluate(test_data, test_targets)&#xA;&#xA;for test in zip(test_data, test_targets):&#xA;    prediction = model.predict(np.array(test[0:1]))&#xA;    print &quot;Target:&quot;, test[1], &quot; | Predicted:&quot;, prediction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3313" LastActivityDate="2019-09-04T11:48:51.297" Title="MLP on Iris Data not working but it does fine on MNIST - Keras" Tags="|python|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13164" PostTypeId="2" ParentId="13163" CreationDate="2016-08-03T13:15:11.220" Score="2" Body="&lt;p&gt;By default &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.preprocessing.normalize&lt;/a&gt; normalizes samples, not features. Replace sklearn.preprocessing.normalize with &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.preprocessing.scale&lt;/a&gt;. This will center and scale (to unit variance) every feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also give it more than 10 epochs. Here are learning curves (log loss) for 5000 epochs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9ME3N.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9ME3N.png&quot; alt=&quot;Learning Curves&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should end up with an accuracy of about 96%.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-08-03T13:15:11.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13165" PostTypeId="2" ParentId="13107" CreationDate="2016-08-03T13:24:09.160" Score="0" Body="&lt;p&gt;Multilabel classification can seem to be a tough one in nlp. Recently there have been many techniques developed for this purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1604.00783.pdf&quot; rel=&quot;nofollow&quot;&gt;ML-PA-LDA&lt;/a&gt; algorithm seems to be working well with multilabel approach. PA stands for presence absence where they consider the correlations because of absence of a class in a document. &lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-03T13:24:09.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13167" PostTypeId="1" AcceptedAnswerId="13168" CreationDate="2016-08-03T13:52:27.180" Score="0" ViewCount="62" Body="&lt;p&gt;I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, and quantity of the product sold, and I'm asked to predict the quantity of a product, given the client ID, and the product ID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, say the client IDs are in the range 10000 - 50000 and the product IDs are in the range 1-10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, for a moment that the client IDs were random integers from 1-1000000 and the product IDs were random integers from 1000-2000. This isn't supposed to make the slightest change in the results, is it? After all, IDs are mere tags. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But thinking data-wise, I've just bloated up two entire columns in my data to a higher scale, and these are two features as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do I think about this? How do I factor in features which are IDs? How do I 'normalize' them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I'm not being vague here. I just don't know a better way of phrasing this question. &lt;/p&gt;&#xA;" OwnerUserId="22059" LastActivityDate="2016-08-03T14:17:43.153" Title="How do I factor in features which are IDs?" Tags="|dataset|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13168" PostTypeId="2" ParentId="13167" CreationDate="2016-08-03T14:17:43.153" Score="2" Body="&lt;p&gt;These IDs should not be represented as numerical values to your model. If you would, your model thinks 2 and 3 are closer together than 2 and 2000, while it's just an ID, the number is just a name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some models can deal with them but then they need to be factors or categories (like decision trees). However most models cannot deal with categories at all, there are however numerous ways to solve this problem. The most used one is probably one-hot encoding, which means for every category in your feature you add a column, and you put a 1 if it's that category and a 0 otherwise. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID | target&#xA;1  | 0&#xA;1  | 1&#xA;2  | 3&#xA;3  | 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID_1 | ID_2 | ID_3 | target&#xA;1    | 0    | 0    | 0&#xA;1    | 0    | 0    | 1&#xA;0    | 1    | 0    | 3&#xA;0    | 0    | 1    | 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This work very well if you have few categories, however in the case of thousands of IDs this will increase your dimensionality too much. What you can do is collect statistics about the target and other features per group and join these onto your set and then remove your categories. This is what is usually done with a high number of categories. You have to be careful not to leak any information about your target into your features though (problem called label leaking).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-03T14:17:43.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13169" PostTypeId="1" AcceptedAnswerId="13247" CreationDate="2016-08-03T14:49:21.637" Score="2" ViewCount="321" Body="&lt;p&gt;My data (the doc-topics output from a MALLET topic model) has the following shape:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0   000cbac90fcc47efad081a929d74ab4c    0   0.3571185904395461  19  0.3113396339935042  4   0.12325835735304397 10  0.10409710001928904 8   0.04929547912982593 1   0.026833654459159112    2   0.01605333272832067 6   0.0048677134975701405   11  0.0019476615596546681   17  0.001908921062167065    14  0.0016877826048433426   5   5.256520178505095E-4    12  2.6155079718746636E-4   15  2.1354275518902175E-4   16  1.4885824861864537E-4   18  1.0362646846573807E-4   3   9.802082006786611E-5    9   9.641623707017035E-5    13  9.348016118039801E-5    7   5.0625647445916464E-5   &#xA;1   002d2e8fbf8f40399fb52c12fe6d79a1    2   0.4893657941273363  19  0.31683989601254264 5   0.14250187050440621 4   0.020578111489117222    8   0.012792452390528172    10  0.012232805681991418    11  9.110846196785881E-4    17  8.170410538631972E-4    12  6.349213666458032E-4    15  5.18380595356186E-4 0   4.750799583739948E-4    6   3.65757686209981E-4 16  3.613572723378424E-4    14  2.8022169390809563E-4   1   2.5184798205700335E-4   18  2.5155594892638147E-4   3   2.3794809156182638E-4   9   2.3405292457801712E-4   13  2.2692552394855585E-4   7   1.228950766326711E-4    &#xA;2   0046e05d3731491da4d9bab51d6ea36a    16  0.652945776661391   8   0.07953971245258269 0   0.06617734607073089 19  0.059407148715209045    4   0.02302855863782211 5   0.019674895033989195    11  0.014047819199510685    17  0.01259778154009113 2   0.012346407436534609    10  0.012058449719318621    12  0.009789716972385079    15  0.0079927997057698  6   0.005639539660456337    14  0.004320678460350115    1   0.0038831902561877263   18  0.0038786874597068794   3   0.0036688708127993316   9   0.003608812064842682    13  0.0034989161965088794   7   0.001894892943813293    &#xA;etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row (representing a document) has an index column, an id column, and 40 more columns that represent the 20 topics present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The shape I'm trying to get is one where each topic number is it's own column and rows contain just the index, id, and topic proportions, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i   id                                  0                   1                   2&#xA;0   000cbac90fcc47efad081a929d74ab4c    0.3571185904395461  0.2346339935042     0.1884010001928904&#xA;etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems like this could be accomplished with pivots, but the topic numbers being out of order (sorted by relevance in rows) makes this a head scratcher...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would one accomplish this?&lt;/p&gt;&#xA;" OwnerUserId="22061" LastActivityDate="2016-08-05T21:12:07.217" Title="How to get columns from unsorted rows in Pandas? (MALLET)" Tags="|python|nlp|pandas|topic-model|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13170" PostTypeId="1" CreationDate="2016-08-03T15:27:17.023" Score="1" ViewCount="88" Body="&lt;p&gt;I am searching for a score to compare two different classifications of the same observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html#sklearn.metrics.adjusted_rand_score&quot; rel=&quot;nofollow&quot;&gt;Adjusted Rand Index&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html&quot; rel=&quot;nofollow&quot;&gt;Adjusted Mutual Information&lt;/a&gt;, but the problem is that the first classification splits the observations in 4 classes, the second in 5 classes. If I understood these metrics correctly, the number of classes should be the same between the two classifications (only permutation of the labels is allowed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that, if the score is between 0 (no concordance) to 1 (full concordance) I may get a result like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# permutation is ok  &#xA;score( [1,2,3,4], [2,1,3,4] )  =  1&#xA;&#xA;# subset is ok&#xA;score( [1,2,3,4], [1,1,2,3] )  =  1&#xA;&#xA;# this should be less than 1 (last item in the second classification&#xA;# may be 1,2 but not 3)&#xA;score( [1,1,2,1], [1,2,3,3] )  &amp;lt;  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a similar score to evaluate concordance of two classifications with different number of classes?&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="133" LastEditDate="2016-08-03T15:41:53.493" LastActivityDate="2016-08-03T15:41:53.493" Title="Score for concordance of two classifications with different number of classes" Tags="|classification|multilabel-classification|score|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13171" PostTypeId="1" CreationDate="2016-08-03T16:16:23.313" Score="1" ViewCount="140" Body="&lt;p&gt;In R, &lt;code&gt;stem()&lt;/code&gt; produces the graph with &quot;stems&quot; in ascending order. Is there a code for obtaining them in descending order?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;stem (cars$dist)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The decimal point is 1 digit(s) to the right of the &lt;code&gt;|&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   0 | 14004678&#xA;   2 | 002466668822244466&#xA;   4 | 002668024466&#xA;   6 | 046806&#xA;   8 | 04523&#xA;  10 | &#xA;  12 | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The desired output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  12 | 0&#xA;  10 | &#xA;   8 | 04523&#xA;   6 | 046806&#xA;   4 | 002668024466&#xA;   2 | 002466668822244466&#xA;   0 | 14004678&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22065" LastEditorDisplayName="user8022" LastEditDate="2016-08-04T05:09:06.683" LastActivityDate="2016-10-15T16:22:50.653" Title="stem() output order in R" Tags="|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13172" PostTypeId="2" ParentId="13158" CreationDate="2016-08-03T17:07:11.590" Score="0" Body="&lt;p&gt;Here is the big list of data repositories, check one of them &lt;a href=&quot;http://www.kdnuggets.com/datasets/index.html&quot; rel=&quot;nofollow&quot;&gt;http://www.kdnuggets.com/datasets/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="22066" LastActivityDate="2016-08-03T17:07:11.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13173" PostTypeId="1" CreationDate="2016-08-03T17:34:12.867" Score="0" ViewCount="179" Body="&lt;p&gt;Is there any market for data science as service which provides service without selling software, just consulting, modelling and developing algorithms?&lt;/p&gt;&#xA;" OwnerUserId="22068" LastActivityDate="2016-09-05T16:25:56.570" Title="Is there any market for data science consulting" Tags="|bigdata|" AnswerCount="1" CommentCount="0" ClosedDate="2016-09-05T20:19:38.177" ContentLicense="CC BY-SA 3.0" />
  <row Id="13174" PostTypeId="2" ParentId="13058" CreationDate="2016-08-03T18:13:12.353" Score="0" Body="&lt;p&gt;Here's a Github project that seems to be for connecting Spark with OrientDB:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/metreta/spark-orientdb-connector/blob/master/spark-orientdb-connector/src/test/scala/com/metreta/spark/orientdb/connector/rdd/GraphRDDSpec.scala&quot; rel=&quot;nofollow&quot;&gt;https://github.com/metreta/spark-orientdb-connector/blob/master/spark-orientdb-connector/src/test/scala/com/metreta/spark/orientdb/connector/rdd/GraphRDDSpec.scala&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also just use the HTTP (REST) API with JSON formatted data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://orientdb.com/docs/2.1/OrientDB-REST.html&quot; rel=&quot;nofollow&quot;&gt;http://orientdb.com/docs/2.1/OrientDB-REST.html&lt;/a&gt; (note: not the latest version of the docs)&lt;/p&gt;&#xA;" OwnerUserId="22071" LastEditorUserId="22071" LastEditDate="2016-08-03T18:20:10.393" LastActivityDate="2016-08-03T18:20:10.393" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13175" PostTypeId="1" CreationDate="2016-08-03T19:15:41.397" Score="9" ViewCount="1692" Body="&lt;p&gt;Neural networks are trained to minimize some error function over the weights of the neural connections. In some applications, these weights could be considered intellectual property. Is there a way to encrypt these weights and still have an operational neural network?&lt;/p&gt;&#xA;&lt;p&gt;Some context: I'm trying to scale a neural network algorithm, but right now we're doing all the computations on a centralized server and it's getting bogged down. We can shift the computation to the client side, but we don't want someone to unpack the executable and obtain the weights of the network. Is there some way to distribute an &amp;quot;encrypted neural network&amp;quot; such that our IP is protected?&lt;/p&gt;&#xA;&lt;p&gt;To clarify: I am &lt;strong&gt;not&lt;/strong&gt; talking about &amp;quot;oblivious&amp;quot; neural networks that operate on encrypted data, I am talking about the weights of the neural network themselves.&lt;/p&gt;&#xA;&lt;p&gt;I'm fine with alternative obfuscation techniques.&lt;/p&gt;&#xA;&lt;p&gt;Edit: I found &lt;a href=&quot;http://clem.dii.unisi.it/%7Evipp/files/publications/S1687416107373439.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;, but it states&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Great attention&#xA;is paid to avoid any unnecessary disclosure of information,&#xA;so that at the end of the protocol the user only knows&#xA;the final NN output, whereas all internal computations are&#xA;kept secret. In this way, the possibility for a malevolent user&#xA;to provide a set of fake inputs properly selected to disclose&#xA;the network secrets is prevented. A solution is also sketched&#xA;that permits to obfuscate the network topology, however, a&#xA;deeper investigation in this direction is left for future research.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;indicating that this paper is on a related concept, but I'm looking for a resource where the topology is obfuscated.&lt;/p&gt;&#xA;" OwnerUserId="22075" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-01-30T23:04:50.497" Title="Neural network obfuscation" Tags="|neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13176" PostTypeId="2" ParentId="13175" CreationDate="2016-08-03T19:54:17.143" Score="0" Body="&lt;p&gt;My answer only applies to one niche, the database server market. If you're storing your weights in a SQL Server database, you can use Transparent Data Encryption (TDE). This will encrypt your whole database so that only authorized users can read its contents; it is also possible to encrypt individual columns to further restrict access, so that even authorized users can't read the contents of a table without first decrypting it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's &lt;a href=&quot;https://msdn.microsoft.com/en-us/library/bb934049.aspx&quot; rel=&quot;nofollow&quot;&gt;a link&lt;/a&gt; to some documentation on how to use TDE. It's actually really easy to use, but as with anything, there are a few potential &quot;gotchas&quot; when setting it up. For a short list of some of those I encountered, see my column &lt;a href=&quot;http://www.sqlservercentral.com/blogs/multidimensionalmayhem/2012/02/28/misadventures-in-tde-how-to-restore-an-encrypted-sql-server-database-completely-wrong/&quot; rel=&quot;nofollow&quot;&gt;Misadventures in TDE: How to Restore an Encrypted SQL Server Database Completely Wrong&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally you don't have to do any coding whatsoever in a front-end application (such as a GUI user access interface with spreadsheets, etc.) to use a TDE-enabled database; the user accessing the data just needs sufficient permissions. I know that .Net languages like VB and C# also have their own encryption commands you can leverage too if for some reason you needed to do encryption on the front-end too; I suspect other programming ecosystems like Java etc. have their own counterparts. I also fairly certain that SQL Server's competitors in the database server market, such as Oracle and MySQL, have their own versions of database encryption, but I'm not familiar with how they work. I firmly believe that in due time, a lot of neural net and other machine learning algorithms will end up running on database servers of this kind because of this and many other advantages. I'm pretty sure that Oracle etc. have free versions of their database akin to SQL Server's, which lets you store and operate on about 10 GB. If that's enough space for your neural nets, then you could just set them up on a dedicated server without having to pay for a database server license. Disclaimer: I'm not in any way connected with Microsoft or in a position to profit from it; I just know this one technology well. I use it to encrypt my own neural net weights, so I know from personal experience that this solution will work quite well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're merely storing the weights in files rather than a database you can probably make use of disk and file encryption software, such as TrueCrypt or EFS and BitLocker for Windows. I'm weak in networking so I can't comment on transport security across the Internet, although I suspect technologies like SSL etc. would do the job. I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="18544" LastActivityDate="2016-08-03T19:54:17.143" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13178" PostTypeId="1" AcceptedAnswerId="13221" CreationDate="2016-08-03T20:28:03.443" Score="13" ViewCount="30339" Body="&lt;p&gt;I have a data set with 20000 samples, each has 12 different features. Each sample is either in category 0 or 1. I want to train a neural network and a decision forest to categorize the samples so that I can compare the results and both techniques. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing I stumbled upon is the proper normalization of the data. One feature is in the range $[0,10^6]$, another one in $[30,40]$ and there is one feature that mostly takes the value 8 and sometimes 7. So as I read in different sources, proper normalization of the input data is crucial for neural networks. As I found out, there are many possible ways to normalize the data, for example:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Min-Max Normalization&lt;/strong&gt;: The input range is linearly transformed to the interval $[0,1]$ (or alternatively $[-1,1]$, does that matter?)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Z-Score Normalization&lt;/strong&gt;: The data is transformed to have zero mean and unit variance:&#xA;$$y_{new}=\frac{y_{old}-\text{mean}}{\sqrt{\text{Var}}}$$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which normalization should I choose? Is normalization also needed for decision forests? With Z-Score normalization, the different features of my test data do not lie in the same range. Could this be a problem? Should every feature normalized with the same algorithm, so that I decide either to use Min-Max for all features or Z-Score for all features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there combinations where the data is mapped to $[-1,1]$ and also has zero mean (which would imply a non-linear transformation of the data and hence a change in the variance and other features of the input data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel a bit lost because I can't find references which answer these questions.&lt;/p&gt;&#xA;" OwnerUserId="22080" LastActivityDate="2016-08-06T09:56:33.170" Title="How to normalize data for Neural Network and Decision Forest" Tags="|neural-network|decision-trees|normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13179" PostTypeId="2" ParentId="13138" CreationDate="2016-08-03T20:30:53.633" Score="19" Body="&lt;p&gt;Here is what I learnt recently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about &lt;strong&gt;word/char-based&lt;/strong&gt; text generation RNNs, we are asking about &lt;strong&gt;word/char-based&lt;/strong&gt; RNN language models (LM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Word-based&lt;/strong&gt; LMs display &lt;strong&gt;higher accuracy&lt;/strong&gt; and &lt;strong&gt;lower computational cost&lt;/strong&gt; than &lt;strong&gt;char-based&lt;/strong&gt; LMs. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1511.06303&quot; rel=&quot;noreferrer&quot;&gt;This drop of performance is &lt;strong&gt;unlikely&lt;/strong&gt; due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, we can say that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://arxiv.org/abs/1511.06303&quot; rel=&quot;noreferrer&quot;&gt;one of the &lt;strong&gt;fundamental differences&lt;/strong&gt; between the &lt;strong&gt;word level&lt;/strong&gt; and &lt;strong&gt;character level&lt;/strong&gt; models is in &lt;strong&gt;the number of parameters&lt;/strong&gt; the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, &lt;strong&gt;char-based&lt;/strong&gt; RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using &lt;strong&gt;word-based&lt;/strong&gt; RNN LMs to model such languages is difficult if possible at all and is not advised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above analysis makes sense especially when you look at the output text, generated by &lt;strong&gt;char-based&lt;/strong&gt; RNNs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While simple &lt;a href=&quot;http://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139&quot; rel=&quot;noreferrer&quot;&gt;char-based Maximum Likelihood LM&lt;/a&gt; with a 13-character window delivers this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;And when she made many solid bricks. He stacked them in piles and stomped her&#xA;  feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked&#xA;  her out.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward &lt;em&gt;conditional probability&lt;/em&gt; generates better texts than far more complex &lt;strong&gt;char-based&lt;/strong&gt; RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Char-based&lt;/strong&gt; RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while &lt;strong&gt;word-based&lt;/strong&gt; RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.&lt;/p&gt;&#xA;" OwnerUserId="22012" LastEditorUserId="22012" LastEditDate="2016-08-05T11:43:13.003" LastActivityDate="2016-08-05T11:43:13.003" CommentCount="15" ContentLicense="CC BY-SA 3.0" />
  <row Id="13180" PostTypeId="2" ParentId="13178" CreationDate="2016-08-03T21:02:02.537" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;There is no clear cut answer&lt;/strong&gt;. What I'd recommend would be to scale your data using different approaches and then use the same model to predict outcomes on your holdout set (RFs would work fine here). That should atleast show you which scaling approach is best in your prediction problem.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/8961586/do-i-need-to-normalize-or-scale-data-for-randomforest-r-package&quot;&gt;You don't need to scale your data for Random Forests&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;The individual ranges shouldn't be a problem as long as they are consistently scaled to begin with. This is just illustrating that there are differences between the variables, just on a more compact scale than before.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Yes&lt;/strong&gt;  - all your data should be scaled with the same approach. Otherwise values in your transformed dataset might relate not to the data itself, but the algo used for scaling.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hopefully this helps.&lt;/p&gt;&#xA;" OwnerUserId="21702" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-03T21:02:02.537" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13181" PostTypeId="1" CreationDate="2016-08-03T22:03:14.930" Score="18" ViewCount="26784" Body="&lt;p&gt;I'm starting a project where the task is to identify sneaker types from images.  I'm currently reading into &lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/tutorials/image_recognition/index.html&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt; and &lt;a href=&quot;https://github.com/karpathy/neuraltalk2&quot; rel=&quot;noreferrer&quot;&gt;Torch&lt;/a&gt; implementations. My question is: how many images per class are required to reach a reasonable classification performance?      &lt;/p&gt;&#xA;" OwnerUserId="14150" LastEditorUserId="29169" LastEditDate="2021-02-16T15:40:21.847" LastActivityDate="2021-02-16T15:40:21.847" Title="How many images per class are sufficient for training a CNN" Tags="|machine-learning|neural-network|image-classification|convolutional-neural-network|image-recognition|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13184" PostTypeId="2" ParentId="10093" CreationDate="2016-08-04T01:09:46.383" Score="4" Body="&lt;p&gt;OK, for a 95% confidence interval, you want to know how many standard deviations away from the mean your point estimate is (the &quot;z-score&quot;).  To get that, you take off the 5% &quot;tails&quot;.  Working in percentile form you have 100-95 which yields a value of 5, or 0.05 in decimal form.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Divide that in half to get 0.025 and then, in R, use the qnorm function to get the z-star (&quot;critical value&quot;).  Since you only care about one &quot;side&quot; of the curve (the values on either side are mirror images of each other) and you want a positive number, pass the argument lower.tail=FALSE.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in the end, it would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;qnorm(.025,lower.tail=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yielding a value of 1.959964&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You then plug that value into the equation for the margin of error to finish things up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to go the other direction, from a &quot;critical value&quot; to a probability, use the pnorm function.  Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pnorm(1.959964,lower.tail=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which will give you back 0.025&lt;/p&gt;&#xA;" OwnerUserId="6554" LastEditorUserId="95458" LastEditDate="2020-04-22T10:07:43.387" LastActivityDate="2020-04-22T10:07:43.387" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13185" PostTypeId="1" AcceptedAnswerId="13232" CreationDate="2016-08-04T01:28:45.307" Score="9" ViewCount="10640" Body="&lt;p&gt;If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (&lt;code&gt;GridSearchCV&lt;/code&gt;) finds the best hyperparameters, and the outter loop (&lt;code&gt;cross_val_score&lt;/code&gt;) evaluates the hyperparameter tuning algorithm. I then choose which tuning/model combo from the outer loop that minimizes &lt;code&gt;mse&lt;/code&gt; (I'm looking at regression classifier) for my final model test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read the questions/answers on nested-cross-validation, but haven't seen an example of a full pipeline that utilizes this. So, does my code below (please ignore the actual hyperparameter ranges - this is just for example) and thought process make sense?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import cross_val_score, train_test_split&#xA;from sklearn.grid_search import GridSearchCV&#xA;from sklearn.metrics import mean_squared_error&#xA;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn.svm import SVR&#xA;from sklearn.datasets import make_regression&#xA;&#xA;# create some regression data&#xA;X, y = make_regression(n_samples=1000, n_features=10)&#xA;params = [{'C':[0.01,0.05,0.1,1]},{'n_estimators':[10,100,1000]}]&#xA;&#xA;# setup models, variables&#xA;mean_score = []&#xA;models = [SVR(), RandomForestRegressor()]&#xA;&#xA;# split into train and test sets&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)&#xA;&#xA;# estimate performance of hyperparameter tuning and model algorithm pipeline&#xA;for idx, model in enumerate(models):&#xA;    clf = GridSearchCV(model, params[idx], scoring='mean_squared_error')&#xA;&#xA;    # this performs a nested CV in SKLearn&#xA;    score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')&#xA;&#xA;    # get the mean MSE across each fold&#xA;    mean_score.append(np.mean(score))&#xA;    print('Model:', model, 'MSE:', mean_score[-1])&#xA;&#xA;# estimate generalization performance of the best model selection technique&#xA;best_idx = mean_score.index(max(mean_score)) # because SKLearn flips MSE signs, max works OK here&#xA;best_model = models[best_idx]&#xA;&#xA;clf_final = GridSearchCV(best_model, params[best_idx])&#xA;clf_final.fit(X_train, y_train)&#xA;&#xA;y_pred = clf_final.predict(X_test)&#xA;rmse = np.sqrt(mean_squared_error(y_test, y_pred))&#xA;&#xA;print('Final Model': best_model, 'Final model RMSE:', rmse)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23090" LastActivityDate="2017-07-24T13:46:18.487" Title="Nested cross-validation and selecting the best regression model - is this the right SKLearn process?" Tags="|python|scikit-learn|cross-validation|model-selection|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13186" PostTypeId="1" AcceptedAnswerId="13209" CreationDate="2016-08-04T02:26:25.090" Score="2" ViewCount="6058" Body="&lt;p&gt;I found that PageRank algorithm depend heavily on the existence of edges, but the weights of edges give small effect. &#xA;For example, PageRank on triangle graph(A -&gt; B,  B -&gt; C,  C -&gt; A) is always (1/3, 1/3, 1/3) even if one of the weight is 0.0000001.&#xA;Is there any PageRank-like method work well on the weighted graph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="23091" LastActivityDate="2020-07-08T14:01:14.390" Title="Is there any PageRank-like method on weighted graph?" Tags="|model-evaluations|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13187" PostTypeId="2" ParentId="13186" CreationDate="2016-08-04T05:41:38.133" Score="2" Body="&lt;p&gt;As you mentioned, PageRank is effective to identify important nodes in the connected graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you need to think about what kinds of weights to add, and where you put the weight on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found several works adding weights on edges.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;inbound/outbound link weighted PageRank (Xing, Ghorbani 2004)&lt;/li&gt;&#xA;&lt;li&gt;AuthorRank considering link weights among the co-authorship links (Liu, Bollen, Nelson, and Sompel 2005)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And, some other works adding weights on nodes.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a credit-weighted PageRank (Radicchi, Fortunato, Makines, Vespignani 2009)&lt;/li&gt;&#xA;&lt;li&gt;a weighted PageRank with number of citation, number of publication (Ding 2011)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="22054" LastActivityDate="2016-08-04T05:41:38.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13188" PostTypeId="1" CreationDate="2016-08-04T08:25:33.450" Score="9" ViewCount="3549" Body="&lt;p&gt;I am reading this paper &quot;Sequence to Sequence Learning with Neural Networks&quot; &lt;a href=&quot;http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under &quot;2. The Model&quot; it says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The LSTM computes this conditional probability by first obtaining the fixed dimensional representation v of the input sequence (x1, . . . , xT ) given by the last hidden state of the LSTM, and then computing the probability of y1, . . . ,yT′ with a standard &lt;strong&gt;LSTM-LM&lt;/strong&gt; formulation whose initial hidden state is set to the representation v of x1, . . . , xT: &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know what an LSTM is, but what's an LSTM-LM? I've tried Googling it but can't find any good leads.&lt;/p&gt;&#xA;" OwnerUserId="23097" LastEditorUserId="23097" LastEditDate="2016-08-04T08:56:54.080" LastActivityDate="2018-09-18T12:40:01.363" Title="What's an LSTM-LM formulation?" Tags="|machine-learning|neural-network|nlp|rnn|machine-translation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13189" PostTypeId="2" ParentId="13188" CreationDate="2016-08-04T08:32:24.003" Score="12" Body="&lt;p&gt;The definition of a Language Model (LM) is a probability distribution over sequences of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple illustration of an LM is predicting the next word given the previous word(s).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I have a language model and some initial word(s):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I set my initial word to &lt;code&gt;My&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;My model predicts there is a high probability that &lt;code&gt;name&lt;/code&gt; appears after &lt;code&gt;My&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;By setting the initial words to  &lt;code&gt;My name&lt;/code&gt;, my model predicts there is a high probability that &lt;code&gt;is&lt;/code&gt; appears after &lt;code&gt;My name&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;So it's like: &lt;code&gt;My&lt;/code&gt; -&gt; &lt;code&gt;My name&lt;/code&gt; -&gt; &lt;code&gt;My name is&lt;/code&gt; -&gt; &lt;code&gt;My name is Tom&lt;/code&gt;, and so on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can think of the autocompletion on your smartphone keyboard. In fact, LM is the heart of autocompletions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;strong&gt;LSTM-LM&lt;/strong&gt; is simply using an LSTM (and softmax function) to predict the next word given your previous words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, Language Model is not limited to LSTM, other RNNs (GRU), or other structured models. In fact, you can also use feedforward networks with context/sliding/rolling window to predict the next word given your initial words.&lt;/p&gt;&#xA;" OwnerUserId="9465" LastEditorUserId="9465" LastEditDate="2017-10-15T14:41:57.020" LastActivityDate="2017-10-15T14:41:57.020" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13190" PostTypeId="2" ParentId="13171" CreationDate="2016-08-04T08:44:02.760" Score="2" Body="&lt;p&gt;Try to capture output, and modify it in a desired way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rev.stem &amp;lt;- function(data){&#xA;  out &amp;lt;- capture.output(stem(data))&#xA;  out &amp;lt;- c(out[1:3], rev(out[4:length(out)]))&#xA;  cat(out, sep=&quot;\n&quot;)&#xA;}&#xA;&#xA;rev.stem(cars$dist)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-08-04T08:44:02.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13191" PostTypeId="1" CreationDate="2016-08-04T09:07:31.277" Score="1" ViewCount="206" Body="&lt;p&gt;In this paper &lt;a href=&quot;http://www.aclweb.org/anthology/D13-1176&quot; rel=&quot;nofollow&quot;&gt;http://www.aclweb.org/anthology/D13-1176&lt;/a&gt;, in &quot;2. Framework&quot; it says &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We begin by describing the modelling framework underlying RCTMs. An RCTM estimates the probability $P(f \ | \ e)$ of a target sentence $f = f_1, ..., f_m$ being a translation of a source sentence $ e = e_1, ..., e_k$. Let us denote by $f_{i\ : \ j}$ the substring of words $ f_i, ..., f_j $. Using the following identity, &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;P(f \ | \ e) = \prod_{i=1}^m P(f_i \ | \ f_{1:i−1}, e)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;an RCTM estimates $P(f|e)$ by directly computing for each target position $i$ the conditional probability $P (f_i|f_{1:i-1}, e)$ of the target word $f_i$ occurring in the translation at position $i$, given the preceding target words $f_{1:i-1}$ and the source sentence $e$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I understand the identity that they used, but how is it implemented in terms of code? How can a module calculate such a probability?&lt;/p&gt;&#xA;" OwnerUserId="23097" LastActivityDate="2016-12-04T16:30:29.663" Title="How to compute conditional probability in code?" Tags="|machine-learning|nlp|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13194" PostTypeId="2" ParentId="13181" CreationDate="2016-08-04T14:03:41.417" Score="2" Body="&lt;p&gt;The best approach is to collect as much data as you comfortably can. Then get started with the project and make a data model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can evaluate your model to see if it has &lt;strong&gt;High Bias&lt;/strong&gt; or &lt;strong&gt;High Variance.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;High Variance&lt;/strong&gt; : In this situation you will see that  Cross-Validation error is higher than Training error after convergence.There is a significant gap if you plot the same against training data size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;High Bias&lt;/strong&gt;: In this situation Cross-Validation error is slightly higher than training error which itself is high when plotted against training data size.By plotting against training data size I mean ,you can input subsets of training data you have and keep incrementing subset size and plot errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you see your model has &lt;strong&gt;high variance(overfit), adding more data will usually help&lt;/strong&gt; in contrast to &lt;strong&gt;high bias(underfit) model where adding new training data doesn't help.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also per class you must try to get same number of images otherwise datasets can become &lt;strong&gt;skewed&lt;/strong&gt;(more of one kind).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I suggest if you are using &lt;strong&gt;TensorFlow&lt;/strong&gt; ,read more about &lt;strong&gt;GOOGLE's INCEPTION&lt;/strong&gt; Image Classifier. It is already trained classifier on google's image database and you can use it for your images, that way requirements for number of images comes down drastically. &lt;/p&gt;&#xA;" OwnerUserId="21414" LastEditorUserId="21414" LastEditDate="2016-08-04T14:12:31.100" LastActivityDate="2016-08-04T14:12:31.100" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13195" PostTypeId="1" CreationDate="2016-08-04T14:26:29.927" Score="1" ViewCount="4106" Body="&lt;p&gt;Can I do online learning with random forests? I have a few million datapoints and the classifier fails to finish on the cross validation step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can i break it up in chunks sequentially?&lt;br&gt;&#xA;Current code:    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()&#xA;&#xA;   print('Planting trees...')&#xA;   clf = RandomForestClassifier(&#xA;       n_estimators=50,&#xA;       max_depth=None,&#xA;       min_samples_split=1,&#xA;       random_state=0&#xA;   )&#xA;&#xA;   print('Growing trees...')&#xA;   classifier = clf.fit(X_train, y_train)&#xA;&#xA;   # see how we did&#xA;   print('Testing trees...')&#xA;   scores = cross_val_score(classifier, X_test, y_test)&#xA;   print(scores)&#xA;   print('accuracy: %d' % (scores.mean()))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can I change it to something like:     &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for chunk in df:&#xA;     clf.fit(...)&#xA;     cross_validate...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8643" LastEditorUserId="8643" LastEditDate="2016-08-04T14:41:58.817" LastActivityDate="2016-10-03T18:50:49.553" Title="Sklearn randomforest online learning" Tags="|random-forest|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13196" PostTypeId="1" AcceptedAnswerId="13199" CreationDate="2016-08-04T15:08:50.847" Score="1" ViewCount="4201" Body="&lt;p&gt;I got &lt;code&gt;tableA&lt;/code&gt;, which contains a numeric variable, &lt;code&gt;CdTaller&lt;/code&gt; (e.g. 1948675).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also got &lt;code&gt;tableB&lt;/code&gt;, which contains two variables: &lt;code&gt;Id&lt;/code&gt; (e.g. 2513978) and &lt;code&gt;Name&lt;/code&gt; (e.g. &quot;JOSE MANUEL PEREZ&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to add a new column to &lt;code&gt;tableA&lt;/code&gt;, &lt;code&gt;Name_Taller&lt;/code&gt;, which contains a name for those &lt;code&gt;CdTaller&lt;/code&gt; which correspond to the &lt;code&gt;Id&lt;/code&gt; variable of &lt;code&gt;tableB&lt;/code&gt;, so I do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tableA[, Name_Taller := tableB[Id==tableA\$CdTaller]$Name]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, for those &lt;code&gt;CdTaller&lt;/code&gt; of &lt;code&gt;tableA&lt;/code&gt; that are not in &lt;code&gt;tableB&lt;/code&gt;, R recycles the new variable &lt;code&gt;Name_Taller&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Warning message:&#xA;In \`[.data.table\`(tableA, , \`:=\`(Name, tableB[Id ==  :&#xA;  Supplied 867324 items to be assigned to 947871 items of column 'Name'&#xA;  (recycled leaving remainder of 80547 items).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So for the &lt;code&gt;tableA$CdTaller == '1320402'&lt;/code&gt;, which can't be found in &lt;code&gt;tableB&lt;/code&gt;, R still adds a &lt;code&gt;Name_Taller&lt;/code&gt; which actually corresponds to &lt;code&gt;CdTaller == '4430219'&lt;/code&gt;, because of this recycling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is: how can I avoid this recycling, so that R only adds a &lt;code&gt;NA&lt;/code&gt; or empty string to those rows containing a &lt;code&gt;CdTaller&lt;/code&gt; not found in &lt;code&gt;tableB&lt;/code&gt;? Is there any parameter in data table I can use for this purpose?&lt;/p&gt;&#xA;" OwnerUserId="2489" LastEditorUserId="2961" LastEditDate="2016-08-05T05:39:17.673" LastActivityDate="2016-08-05T05:39:17.673" Title="Avoid recycling when merging with R Data Table" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13197" PostTypeId="2" ParentId="11898" CreationDate="2016-08-04T15:38:32.653" Score="5" Body="&lt;p&gt;There are several methodologies to do that... For the tool you have been talking about (which I will not mention it's name with respect to you) all sorts of pre-coded functions are run at once, it's basically a vast rule engine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Easiest method is to run mini-trees with a random combination of variables (sounds like random forest)... Each tree that has some significant predictive power on classification is a variable and the nodes are it's categories... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) You can build auto-encoders (&lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Autoencoder&lt;/a&gt;), it's easy to do but hard to interpret, basically what deep learning does is automatic feature engineering, that's why it takes so much time to compute ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) You can do symbolic regression (&lt;a href=&quot;https://en.wikipedia.org/wiki/Symbolic_regression&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Symbolic_regression&lt;/a&gt;), possibly using a genetic optimization algorithm to select variables and some mathematical operators to come up with a nice formula that has some power of classification. So when you have some data on a company balance sheet it provides many formulas like EBITDA...&lt;/p&gt;&#xA;" OwnerUserId="23109" LastActivityDate="2016-08-04T15:38:32.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13198" PostTypeId="1" CreationDate="2016-08-04T15:48:51.653" Score="8" ViewCount="2070" Body="&lt;p&gt;For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks. I need to predict a probability distribution over one of the inputs of the next step. Most of my sequences are relatively short, with some big outliers in there. There is more than enough data so that should not be an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Keras for this task, using either LSTM or GRU layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My input exists of sequences of categorical input $x_A$, $x_B$ and a few numerical values $x_1$ to $x_{11}$. I'm trrying to predict $x_A$ of the last step that is not trained on. For the high cardinality categorical features I'm using an embedding layer to map them to a dense space and then merge the three different types of input into one layer. My issue is merging them together before going to the softmax layer to do the one-hot encoding prediction of the last step. I cannot seem to get the inputs to match. I pad the shorter sequences to 10 steps plus the label and truncate the longer sequences cutting off the start. I've attempted both the Graph and the Sequential interface.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def build_graph_model(n_A, n_B, max_length, batch_size):&#xA;    g = Graph()&#xA;    g.add_input(name='A', input_shape=(max_length, ), dtype='int')&#xA;    g.add_node(Embedding(n_A, 32, input_length=max_length, mask_zero=True), name='embedding_A', input='A')&#xA;    g.add_input(name='B', input_shape=(max_length, ), dtype='int')&#xA;    g.add_node(Embedding(n_B, 32, input_length=max_length, mask_zero=True), name='embedding_B', input='B')&#xA;    g.add_input(name='rest', input_shape=(max_length, 11))&#xA;    g.add_node(Dense(30, activation='relu'), name='rest_dense', input='rest')&#xA;    g.add_node(LSTM(100), name='lstm', inputs=['embedding_A', 'embedding_B', 'rest_dense'])&#xA;    g.add_node(Dense(n_dest, activation='softmax'), name='softm', input='lstm')&#xA;    g.add_output(name='output', input='softm')&#xA;    g.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives an assertion error for having the wrong input dimensions. If I remove the $x_1$ to $x_11$ thing completely it works. If I change the Dense layer into TimeDistributed(Dense) I still get the same error although I think that should work. Inserting the rest input directly into the merge doesn't work because the merge cannot be the first layer. I'm uncertain how to fix this issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: This guy asks the same question but much more concise (&lt;a href=&quot;https://groups.google.com/forum/#!topic/keras-users/zCuS4xdv5QY&quot; rel=&quot;noreferrer&quot;&gt;https://groups.google.com/forum/#!topic/keras-users/zCuS4xdv5QY&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-04T16:01:33.063" LastActivityDate="2020-04-26T23:05:05.593" Title="Recurrent neural network multiple types of input Keras" Tags="|python|deep-learning|rnn|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13199" PostTypeId="2" ParentId="13196" CreationDate="2016-08-04T15:56:44.777" Score="2" Body="&lt;p&gt;What you want to do is &lt;code&gt;join&lt;/code&gt;-like behaviour that is among others documented here: &#xA;&lt;a href=&quot;https://rstudio-pubs-static.s3.amazonaws.com/52230_5ae0d25125b544caab32f75f0360e775.html&quot; rel=&quot;nofollow&quot;&gt;joins&lt;/a&gt; and &lt;a href=&quot;https://jangorecki.github.io/blog/2015-12-11/Solve-common-R-problems-efficiently-with-data.table.html&quot; rel=&quot;nofollow&quot;&gt;others&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make things readable here a version with &lt;code&gt;dplyr&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ex1 &amp;lt;- data.frame(id = 1:10, name = replicate(10, paste(sample(letters[1:5], size = 2), collapse = &quot;&quot;)))&#xA;&#xA;ex2 &amp;lt;- data.frame(otherinfo = (1:5)+50, othername = replicate(5, paste(sample(letters[1:5], size = 2), collapse = &quot;&quot;)))&#xA;&#xA;dplyr::left_join(ex1, ex2, by = c(name = &quot;othername&quot;))&#xA;&#xA;   id name otherinfo&#xA;1   1   ac        NA&#xA;2   2   ac        NA&#xA;3   3   ab        NA&#xA;4   4   da        53&#xA;5   5   ab        NA&#xA;6   6   cb        NA&#xA;7   7   eb        NA&#xA;8   8   de        NA&#xA;9   9   ad        NA&#xA;10 10   ec        52&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-08-04T15:56:44.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13200" PostTypeId="1" AcceptedAnswerId="13201" CreationDate="2016-08-04T16:01:59.640" Score="3" ViewCount="1649" Body="&lt;p&gt;I am trying to implement k-means clustering algorithm, but I am confused about calculating the distance and update(move) cluster centroids. For example, let's say that I have 2 features. One of them is &lt;code&gt;weight={2,4,6,8,11,14,21}&lt;/code&gt; and the other one is &lt;code&gt;height={4,6,7,8,9,12,14}&lt;/code&gt;. So, in the coordinate system my points are &lt;code&gt;x1={2,4},x2={4,6},x3={6,7}&lt;/code&gt; and so on. Then, I initialize the cluster centroids randomly, doesn't matter how many there are for now, but they have coordinates too. Let's say &lt;code&gt;μ1={4,2}&lt;/code&gt;. At this point, I understand how do I calculate distance with Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/73EAw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/73EAw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code for calculating distance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_distance(x1,x2,s1,s2):&#xA;      return np.sqrt(np.power(s1-x1,2)+np.power(s2-x2,2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I get a distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question is how cluster assignment step(first step in loop) will know which centroid assign to c(i).I mean,am I supposed to look at each centroid for understand which sample(x(i)) is close to it and then I should assign centroid to c(i), right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second question, let's say I got distances and I have c(1,2..,n) array now. Second step in algorithm which is called move(update) centroid step, we are calculate μ. According to formula, this μ is the average of points assigned to clusters so for example μ1=[x(3) + x(4) + x(6)] / 3. However, here our μ was a point in coordinate system, right? I mean, μ1 was {4,2}. How can this be possible? It's a point not a variable. It has coordinates. Well, if it will become a variable let's say μ1=5, how can I subtract ||x(i)-μ|| then? x is a coordinate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My last question is very simple. For this example, I have two features weight and height. What is the maximum number of features that we can use in k-mean? Is it possible to use k-mean algorithms for many many features? For instance, my first feature is height and the second is weight and the third is width fourth and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope, I explained my problem clearly. If not, sorry for the bad English. I think these three questions are independent questions, so you can answer one of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="20926" LastEditorUserId="29169" LastEditDate="2019-06-07T16:41:28.710" LastActivityDate="2019-06-07T16:41:28.710" Title="K-means Clustering algorithm problems" Tags="|machine-learning|python|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13201" PostTypeId="2" ParentId="13200" CreationDate="2016-08-04T16:33:51.797" Score="4" Body="&lt;p&gt;Wikipedia says: &quot;Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think in your case, this is translatable to: $c_i$ is assigned to the closest centroid by euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your second question, the centroid should $\mu$ should have the same number of dimensions as each training point $x_i$. They are both points in the co-ordinate system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a high number of features with K-means, for example, text analytics might reduce a corpora of news articles to 10,000+ dimensions. Depending on the package you use these might be represented as a sparse matrix.&lt;/p&gt;&#xA;" OwnerUserId="23110" LastActivityDate="2016-08-04T16:33:51.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13203" PostTypeId="2" ParentId="13195" CreationDate="2016-08-04T17:57:24.640" Score="2" Body="&lt;p&gt;There's nothing out of the box that will do true online learning.  In order for a scikit-learn algorithm to support online learning it must provide the &lt;code&gt;partial_fit&lt;/code&gt; function, which &lt;code&gt;RandomForestClassifier&lt;/code&gt; does not.  I think the code you have given will just refit the entire forest on the subset of data it is currently looking at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One alternative you could try is to initialize &lt;code&gt;RandomForestClassifier&lt;/code&gt; with &lt;code&gt;warm_start&lt;/code&gt; set to True.  Then on each subsequent call to &lt;code&gt;fit&lt;/code&gt;, the random forest will add a new set of trees to the ensemble.  You may have to reduce the number of estimators in the ensemble because the number of estimators you end up with will be the number you initialized with times the number of chunks you loop over. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This won't behave exactly the same way as if you were to train the entire ensemble in a truly online way, but it may be close enough for your purposes.&lt;/p&gt;&#xA;" OwnerUserId="18416" LastActivityDate="2016-08-04T17:57:24.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13205" PostTypeId="1" AcceptedAnswerId="13207" CreationDate="2016-08-04T19:49:24.620" Score="5" ViewCount="1948" Body="&lt;p&gt;I am going through tensor-flow tutorial and noticed that they use one-hot encoding in regression &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/learn/wide_n_deep_tutorial.py&quot; rel=&quot;noreferrer&quot;&gt;tensorflow&lt;/a&gt;. I don't fully understand how it works. Let us take oversimplified case of ordinary least square regression. Assume we have y = [1,2,3] and x = [cat, dog, mouse]. Converting to one hot vector we get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat = [0,0,1]&#xA;dog = [0,1,0]&#xA;mouse = [1,0,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how does regression equation looks now? Is it multivariate regression now?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = alpha + beta*x_1 + beta*x_2 + beta*x_3,&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where x_1, x_2, x_3 are coordinates of one-hot vector?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I am interested more in mechanics of this set up, not so much meaning.&lt;/p&gt;&#xA;" OwnerUserId="14999" LastEditorUserId="14999" LastEditDate="2016-08-04T19:55:39.690" LastActivityDate="2016-08-06T22:23:22.430" Title="One-hot encoding" Tags="|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13206" PostTypeId="1" CreationDate="2016-08-04T19:54:09.393" Score="2" ViewCount="35" Body="&lt;p&gt;When I plot my data into bins, there is a frequency of data points per bin, which I can plot with a histogram. Based on this probability density function, I would like to find the maximum likelihood estimation of various parameters associated with this pdf. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I infer the number of &quot;peaks&quot; of this pdf? Is this possible using a Gaussian Mixture Model (let's say using Expectation Maximization)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe in this case, we are inferring the location and variance of each &quot;peak&quot; using Gaussian clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, how does one infer the number of clusters which model the data best? &lt;/p&gt;&#xA;" OwnerUserId="23120" LastEditorUserId="21702" LastEditDate="2016-08-04T20:55:29.980" LastActivityDate="2016-08-04T20:55:29.980" Title="With EM algorithm, can you infer the location and variance of each &quot;peak&quot; in a pdf? Gaussian Mixture Models?" Tags="|unsupervised-learning|parameter-estimation|expectation-maximization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13207" PostTypeId="2" ParentId="13205" CreationDate="2016-08-04T20:00:32.377" Score="3" Body="&lt;p&gt;Yes you turn it into three different variables, however this is not called multivariate regression, that indicates multiple output variables, not inputs. (Thanks to Simon for correcting me)&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-06T22:23:22.430" LastActivityDate="2016-08-06T22:23:22.430" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13208" PostTypeId="2" ParentId="13120" CreationDate="2016-08-04T20:12:59.260" Score="4" Body="&lt;p&gt;Two very different suggestions here to avoid averging the vectors:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use Word Mover's Distance (&lt;a href=&quot;https://github.com/mkusner/wmd&quot; rel=&quot;nofollow&quot;&gt;https://github.com/mkusner/wmd&lt;/a&gt;) to compute distance between the tweets (not sure how well it would work on short texts like tweets, I still need to try that myself...)&lt;/li&gt;&#xA;&lt;li&gt;Cluster the word vectors themselves (using e.g. kmeans), then for each tweet create a vector with k entries (one for each cluster) that encodes whether it contains words belonging to that cluster. I think I saw this in a Kaggle tutorial on word2vec, will be happy for the link if you find it!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23122" LastActivityDate="2016-08-04T20:12:59.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13209" PostTypeId="2" ParentId="13186" CreationDate="2016-08-04T21:06:23.670" Score="4" Body="&lt;p&gt;Weighted versions of Pagerank do exist, and it is easy to &lt;em&gt;incorporate&lt;/em&gt; edge weights into the PageRank algorithm (just multiply each edge's probability by its weight vector, and then normalize to make the edge probabilities to add up to 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difficult question is how to set these weights. Backstrom and Leskovec propose an algorithm to learn these weights in their WSDM paper on &lt;a href=&quot;https://cs.stanford.edu/people/jure/pubs/linkpred-wsdm11.pdf&quot; rel=&quot;nofollow&quot;&gt;Supervised Random Walks&lt;/a&gt; which they propose for use in link prediction and link recommendation.&lt;/p&gt;&#xA;" OwnerUserId="13505" LastActivityDate="2016-08-04T21:06:23.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13211" PostTypeId="2" ParentId="10552" CreationDate="2016-08-04T22:25:37.160" Score="1" Body="&lt;p&gt;&lt;strong&gt;I just want to chime in and highlight that using clustering to segment your data before modelling might be a bad idea.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a high level, clustering is a two/three dimensional visualisation of similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have two/three dimensions to your dataset you're probably OK using clustering to segment your data as all the possible relationships are being visualised by the clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if you have &lt;strong&gt;more than three variables&lt;/strong&gt; than this you won't be able to visualise all the possible relationships meaning that using the 2D/3D clustering might find patterns that are not there. See &lt;a href=&quot;http://www-users.cs.umn.edu/~kumar/papers/high_dim_clustering_19.pdf&quot; rel=&quot;nofollow&quot;&gt;Steinbach, Ertöz, and Kumar&lt;/a&gt; for more information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said you &lt;strong&gt;could use dimensionality reduction techniques to reduce the dimensionality&lt;/strong&gt; of the dataset to two/three dimensions and then cluster your population- but this is situational. &#xA;By reducing the dimensionality of your dataset to get cluster-able results you might be throwing away information that would be useful when it comes to modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also: I'd recommend using a more formalised metric for deciding the adequacy of your clustering &lt;a href=&quot;http://www.math.canterbury.ac.nz/~r.vale/Clustering.pdf&quot; rel=&quot;nofollow&quot;&gt;see here&lt;/a&gt; - maybe you're using too many/not enough clusters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've only answered your half of your question so now I'll attempt the second part: when in doubt, Random Forests are your best friend. Use them as your baseline prediction and you'll be getting great accuracy out of the box.&lt;/p&gt;&#xA;" OwnerUserId="21702" LastActivityDate="2016-08-04T22:25:37.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13212" PostTypeId="1" CreationDate="2016-08-05T00:41:17.617" Score="7" ViewCount="1068" Body="&lt;p&gt;I have to solve a time series model that can take one of two shapes. It can probably take more but here are the two I'm going to ask about. If you have other ideas they are of course welcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First Possible Model -&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X(Dependent variable 'Spending') = X(lag1)...X(lagN) + X(Dummy&#xA;  variable when US) + X(Dummy Variable when Mexico) ... + Error term&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Or Make a separate model for each Country like -&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X(Total Spending in the US only) = X(lag1)...X(lagN) + Error term&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;X(Total Spending in the Mexico only) = X(lag1)...X(lagN) + Error term&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;… for Each country&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Mathematically I can't decide what approach is better. I will use an &lt;code&gt;F-Statistic, dickey fuller statistic&lt;/code&gt; to check the auto regression for stationarity and then compare the two models but I wanted to see what others thought of the theory and if you should ever include the dummy variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking more for an answer that includes mathematical reasoning.&lt;/p&gt;&#xA;" OwnerUserId="12861" LastEditorDisplayName="user380" LastEditDate="2017-05-04T19:57:52.887" LastActivityDate="2017-08-02T22:10:10.763" Title="Time Series Machine Learning Feature Selection Problem" Tags="|python|r|statistics|time-series|regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13213" PostTypeId="1" AcceptedAnswerId="13261" CreationDate="2016-08-05T01:32:49.753" Score="0" ViewCount="75" Body="&lt;p&gt;To be more specific, loss reserving models in actuarial science, such as the chain ladder method, can be expressed as GLMs. I have developed a predictive model using neural nets which takes into account some aspects of the insured (it is an individual risk model). Can the output of this model be safely used as an input to the insurance company's existing loss reserve model?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2016-08-06T22:16:12.357" Title="Are there pitfalls in using the output of machine learning model, such as a neural net as the input to a traditional GLM or similar?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13214" PostTypeId="1" AcceptedAnswerId="13312" CreationDate="2016-08-05T02:50:18.050" Score="4" ViewCount="5782" Body="&lt;p&gt;I can't work out how to get the moving annual sum from this data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;                          revenue&#xA;&amp;gt; txdate   2014-01-31     2014-02-28      2014-03-31     2014-04-30     ....&#xA;&amp;gt; user_id&#xA;&amp;gt; 1            0              10             165             0&#xA;&amp;gt; 2          265             265             200           250&#xA;&amp;gt; 3          770             985            1235           900&#xA;&amp;gt; ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Previously I would have tried something like this and adjusted until it worked:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby(level='practice_id').apply(lambda x: pd.rolling_sum(x, 12))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but it's deprecated and I'm not getting my head around the 0.18 changes to rolling despite reading the docs, and I'm not sure that the shape of the data is helpful (it's close to what needs to be inserted in a db table). The original data format is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; txdate          user_id        tx_amount&#xA;&amp;gt; 2014-01-01         2               5&#xA;&amp;gt; 2014-01-02         2               5&#xA;&amp;gt; 2014-01-02         3              30&#xA;&amp;gt; 2014-01-03         3              15&#xA;&amp;gt; 2014-01-02         2              10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I reshaped with the following cmd:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df.set_index('txdate').groupby('user_id').resample('M').agg({'revenue': np.sum})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm thinking I might need to reverse the order of operations.&lt;/p&gt;&#xA;" OwnerUserId="23126" LastActivityDate="2016-08-09T00:08:44.577" Title="How can I calculate a rolling window sum in pandas across this MultiIndex dataframe?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13215" PostTypeId="1" CreationDate="2016-08-05T03:01:04.613" Score="2" ViewCount="73" Body="&lt;p&gt;I am trying to obtain raw data for (violent) crime rates of a US/Canadian city (any city would do), but I need the data to be granular and raw. All I could find is either interpretations, summary data or useless editorials. I'm trying to do analysis, and I need day-by-day (detailed-level, granular) data that shows the number of crimes recorded per day. Does anyone have any good sources/suggestions for finding this? Thank you immensely! &lt;/p&gt;&#xA;" OwnerUserId="23127" LastActivityDate="2016-08-05T03:39:03.230" Title="Difficulties of getting raw data" Tags="|data-mining|data|open-source|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13216" PostTypeId="1" AcceptedAnswerId="17905" CreationDate="2016-08-05T03:36:04.553" Score="44" ViewCount="42164" Body="&lt;p&gt;I read about NCE (a form of candidate sampling) from these two sources:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/extras/candidate_sampling.pdf&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow writeup&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v9/gutmann10a/gutmann10a.pdf&quot; rel=&quot;noreferrer&quot;&gt;Original Paper&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Can someone help me with the following:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A simple explanation of how NCE works (I found the above difficult to parse and get an understanding of, so something intuitive that leads to the math presented there would be great)&lt;/li&gt;&#xA;&lt;li&gt;After point 1 above, a naturally intuitive description of how this is different from Negative Sampling. I can see that there's a slight change in the formula but could not understand the math. I do have an intuitive understanding of negative sampling in the context of &lt;code&gt;word2vec&lt;/code&gt; - we randomly choose some samples from the vocabulary &lt;code&gt;V&lt;/code&gt; and update only those because &lt;code&gt;|V|&lt;/code&gt; is large and this offers a speedup. Please correct if wrong.&lt;/li&gt;&#xA;&lt;li&gt;When to use which one and how is that decided? It would be great if you could include examples(possibly easy to understand applications)&lt;/li&gt;&#xA;&lt;li&gt;Is NCE better than Negative Sampling? Better in what manner?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="793" LastEditorUserId="21968" LastEditDate="2021-06-22T22:39:25.180" LastActivityDate="2023-12-10T14:57:24.837" Title="Intuitive explanation of Noise Contrastive Estimation (NCE) loss?" Tags="|deep-learning|tensorflow|word-embeddings|sampling|loss-function|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13217" PostTypeId="2" ParentId="13215" CreationDate="2016-08-05T03:39:03.230" Score="1" Body="&lt;p&gt;I found some sites providing open data.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can find crime data by year, by city in &lt;a href=&quot;http://us-city.census.okfn.org/dataset/crime-stats&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&#xA;Data contains incident number, crime type, date, time, location.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://catalog.data.gov/dataset?tags=crime&quot; rel=&quot;nofollow&quot;&gt;This site&lt;/a&gt; also provides crime data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I hope this help.&lt;/p&gt;&#xA;" OwnerUserId="22054" LastActivityDate="2016-08-05T03:39:03.230" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13221" PostTypeId="2" ParentId="13178" CreationDate="2016-08-05T05:55:09.460" Score="25" Body="&lt;p&gt;I disagree with the other comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, I see no need to normalize data for &lt;strong&gt;decision trees&lt;/strong&gt;. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X&amp;gt;x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Random forests&lt;/strong&gt; are just a bunch of decision trees, so it doesn't change this rationale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Neural networks&lt;/strong&gt; are a different story. First of all, in terms of prediction, it makes no difference. The neural network can easily counter your normalization since it just scales the weights and changes the bias. The big problem is in the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use an algorithm like &lt;a href=&quot;https://en.wikipedia.org/wiki/Rprop&quot; rel=&quot;noreferrer&quot;&gt;resilient backpropagation&lt;/a&gt; to estimate the weights of the neural network, then it makes &lt;em&gt;no&lt;/em&gt; difference. The reason is because it uses the sign of the gradient, not its magnitude, when changing the weights in the direction of whatever minimizes your error. This is the default algorithm for the &lt;code&gt;neuralnet&lt;/code&gt; package in R, by the way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When does it make a difference? When you are using traditional backpropagation with sigmoid activation functions, it can &lt;strong&gt;saturate&lt;/strong&gt; the sigmoid derivative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the sigmoid function (green) and its derivative (blue):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OuyeO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OuyeO.png&quot; alt=&quot;sigmoid&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens if you do not normalize your data is that your data is multiplied by the random weights and you get things like $s'(9999)=0$. The derivative of the sigmoid is (approximately) zero and the training process does not move along. The neural network that you end up with is just a neural network with random weights (there is no training).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this help us to know what the best normalization function is? But of course! First of all, it is crucial to use a normalization that centers your data because most implementation initialize bias at zero. I would normalize between -0.5 and 0.5, $\frac{X-\min{X}}{\max{X}-\min{X}}-0.5$. But &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;noreferrer&quot;&gt;standard score&lt;/a&gt; is also good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual normalization is not very crucial because it only influences the initial iterations of the optimization process. As long as it is centered and most of your data is below 1, then it might mean you have to use slightly less or more iterations to get the same result. But the result will be the same, as long as you avoid the saturation problem I mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is something not here discussed which is &lt;strong&gt;regularization&lt;/strong&gt;. If you use regularization in your objective function, the way you normalize your data &lt;strong&gt;will&lt;/strong&gt; affect the resulting model. I'm assuming your are already familiar with this. If you know that one variable is more prone to cause overfitting, your normalization of the data should take this into account. This is of course completely independent of neural networks being used.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-08-06T09:56:33.170" LastActivityDate="2016-08-06T09:56:33.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13222" PostTypeId="2" ParentId="12179" CreationDate="2016-08-05T07:28:29.353" Score="0" Body="&lt;p&gt;Okay, I didn't really understand your question but I am still taking a shot. Firstly, It depends on the fact that how separable your data is if classification is the goal. So, I would suggest doing some visualisations to get a sense of your data. You can use&lt;a href=&quot;http://matplotlib.org/&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt; if you are using python.&lt;br&gt;&#xA;Interval data is going to be an issue, but depending on the data, you can make an assumption to use the middle value or better yet, see if you can create more features out of the interval, such as minimum value, maximum value etc. Now, since you have been gifted with the neighbors already, you can plot them and see how similar are they before you use any function to calculate the similarity. If they are indeed neighbors you will most likely observe a relationship between them.&#xA;Since your goal is classification, and I am assuming you have no labels so you are looking at unsupervised classification. Clustering should be the first course of action, you can play around with the distance metrics, in &lt;a href=&quot;http://scikit-learn.org/stable/modules/neighbors.html&quot; rel=&quot;nofollow&quot;&gt;Nearest Neighbors&lt;/a&gt; and through a bit of trial and error you will probably arrive at the distance function you are looking at.&#xA;If you are only looking to figure out the distance function, it would be better if you take a small chunk of data and try some pairwise experiments with the data and the neighbors you have been given. This will be simpler and faster. You can check out the link I mentioned before to calculate pairwise distances as well.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-08-05T07:28:29.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13223" PostTypeId="2" ParentId="13104" CreationDate="2016-08-05T07:35:54.750" Score="3" Body="&lt;p&gt;Okay, I am currently at 0.81340 in the competiton. And I will just clear out certain things. I would suggest that you try feature engineering before you go for ensemble methods. There are actually quite a decent tutorials as mentioned before. &#xA;One can actually score upto at least 0.82 just relying on feature engineering and a ten-fold cross validated RandomForest. &#xA;Certain things for you to ponder at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Look at the Age, what other information does it give you.&lt;/li&gt;&#xA;&lt;li&gt;Do SibSp and Parch actually represent different things?&lt;/li&gt;&#xA;&lt;li&gt;Can you get something out of the Name of the passengers?  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All the Best.&lt;br&gt;&#xA;Cheers.  &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-08-05T07:35:54.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13225" PostTypeId="2" ParentId="12617" CreationDate="2016-08-05T07:45:44.987" Score="0" Body="&lt;p&gt;You could use K-NearestNeighbors to do this. &lt;a href=&quot;http://scikit-learn.org/stable/modules/neighbors.html&quot; rel=&quot;nofollow&quot;&gt;Nearest Neighbors&lt;/a&gt; is what you are probably you are looking for. You can try out different distance metrics. &#xA;Cheers.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-08-05T07:45:44.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13226" PostTypeId="2" ParentId="13185" CreationDate="2016-08-05T07:45:52.390" Score="2" Body="&lt;p&gt;You do not need&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# this performs a nested CV in SKLearn&#xA;score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;GridSearchCV&lt;/code&gt; does this for you. To get intuition of the grid-search process, try to use  &lt;code&gt;GridSearchCV(... , verbose=3)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To extract scores for each fold see &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html#example-model-selection-grid-search-digits-py&quot; rel=&quot;nofollow&quot;&gt;this example in scikit-learn documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2016-08-05T07:45:52.390" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13227" PostTypeId="1" AcceptedAnswerId="13252" CreationDate="2016-08-05T08:14:46.610" Score="2" ViewCount="212" Body="&lt;p&gt;I am using the forward feature selection algorithm from MATLAB. The code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; X=combine_6_non;&#xA; y=target;&#xA; c = cvpartition(y,'k',10);&#xA; opts = statset('display','iter');&#xA; [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The function fun is as follows:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function err = fun(XT,yT,Xt,yt)&#xA;  model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);&#xA;  err = sum(svmclassify(model, Xt) ~= yt);&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="11097" LastEditDate="2016-08-05T08:17:36.383" LastActivityDate="2021-01-05T20:38:18.063" Title="Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them?" Tags="|machine-learning|feature-selection|svm|feature-extraction|matlab|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13228" PostTypeId="2" ParentId="13205" CreationDate="2016-08-05T09:02:41.500" Score="3" Body="&lt;p&gt;Yes, that is the standard approach to convert categorical variables for fitting a model. In this case it would be used to train a neural network.&#xA;So each category of a categorical variable is represented as a separate vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you do not need to do this for binary variables such as Male/Female as the presence of one category implies absence of the other category, so instead of using a variable such as &lt;code&gt;Gender = Male/Female&lt;/code&gt;; you could convert it into a variable called &lt;code&gt;is_female = 0/1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this dataset is used to fit a regression model, the proper nomenclature should be multiple linear regression.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-08-05T09:02:41.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13229" PostTypeId="1" CreationDate="2016-08-05T09:10:52.093" Score="5" ViewCount="383" Body="&lt;p&gt;I want to use feature selection and observation subsampling on my data, for several reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;feature selection for the usual motivations (reduce noise, decrease running time, etc.)&lt;/li&gt;&#xA;&lt;li&gt;observation subsampling because I have strongly imbalanced data, and I want not to introduce bias towards the most prevalent class in downstream classifiers&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is: is there a specific order in which I should do feature and observation selection? E.g. first feature selection then subsampling?&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="8501" LastEditDate="2016-12-13T09:18:25.287" LastActivityDate="2016-12-13T09:18:25.287" Title="Is there a particular order in which to do feature selection and sampling?" Tags="|feature-selection|dimensionality-reduction|class-imbalance|sampling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13231" PostTypeId="1" CreationDate="2016-08-05T09:47:48.430" Score="8" ViewCount="13591" Body="&lt;p&gt;&lt;strong&gt;Image classification&lt;/strong&gt; is the task of assigning one of $n$ previously known labels to a given image. For example, you know that you will be given a couple of photos and each single image has exactly one of $\{cat, dog, car, stone\}$ in it. The algorithm should say what the photo shows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benchmark dataset for image classification is &lt;strong&gt;ImageNet&lt;/strong&gt;; especiall thy &lt;strong&gt;large scale visual recognition challenge (LSVRC)&lt;/strong&gt;. It has exactly 1000 classes and a huge amount of training data (I think there is a down-sampled version with about 250px x 250px images, but many images seem to be from Flicker).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This challenge is typically solved with CNNs (or other neural networks).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any paper which tries an approach which does not use neural networks in LSVRC?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify the question: Of course, there are other classification algorithms like $k$ nearest neighbors or SVMs. However, I doubt they work at all for that many classes / that much data. At least for $k$-NNs I'm sure that prediction would be extremely slow; for SVMs I guess both fitting and prediction would be much to slow (?).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-08-05T09:57:51.680" LastActivityDate="2016-08-05T15:49:43.343" Title="Are there any image classification algorithms which are not neural networks?" Tags="|image-classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13232" PostTypeId="2" ParentId="13185" CreationDate="2016-08-05T10:35:52.660" Score="9" Body="&lt;p&gt;Yours is not an example of nested cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a score, it does not output a model like in your code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be an example of nested cross validation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;from sklearn.cross_validation import KFold&#xA;from sklearn.metrics import mean_squared_error&#xA;from sklearn.grid_search import GridSearchCV&#xA;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn.svm import SVR&#xA;import numpy as np&#xA;&#xA;params = [{'C': [0.01, 0.05, 0.1, 1]}, {'n_estimators': [10, 100, 1000]}]&#xA;models = [SVR(), RandomForestRegressor()]&#xA;&#xA;df = load_boston()&#xA;X = df['data']&#xA;y = df['target']&#xA;&#xA;cv = [[] for _ in range(len(models))]&#xA;for tr, ts in KFold(len(X)):&#xA;    for i, (model, param) in enumerate(zip(models, params)):&#xA;        best_m = GridSearchCV(model, param)&#xA;        best_m.fit(X[tr], y[tr])&#xA;        s = mean_squared_error(y[ts], best_m.predict(X[ts]))&#xA;        cv[i].append(s)&#xA;print(np.mean(cv, 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By the way, a couple of thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I see no purpose to grid search for &lt;code&gt;n_estimators&lt;/code&gt; for your random forest. Obviously, the more, the merrier. Things like &lt;code&gt;max_depth&lt;/code&gt; is the kind of regularization that you want to optimize. The error for the nested CV of &lt;code&gt;RandomForest&lt;/code&gt; was much higher because you did not optimize for the right hyperparameters, not necessarily because it is a worse model.&lt;/li&gt;&#xA;&lt;li&gt;You might also want to try gradient boosting trees.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-08-05T10:35:52.660" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13237" PostTypeId="2" ParentId="8278" CreationDate="2016-08-05T12:40:11.870" Score="2" Body="&lt;p&gt;Adaptive learning with covariate shift-detection for motor imagery-based brain–computer interface&#xA;&lt;a href=&quot;http://link.springer.com/article/10.1007/s00500-015-1937-5&quot; rel=&quot;nofollow&quot;&gt;http://link.springer.com/article/10.1007/s00500-015-1937-5&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments (&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0031320314002878&quot; rel=&quot;nofollow&quot;&gt;http://www.sciencedirect.com/science/article/pii/S0031320314002878&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="23139" LastActivityDate="2016-08-05T12:40:11.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13238" PostTypeId="1" CreationDate="2016-08-05T13:03:48.977" Score="2" ViewCount="10925" Body="&lt;p&gt;I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame.&#xA;I am using the below code :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql import SQLContext&#xA;sqlc=SQLContext(sc)&#xA;df=sc.textFile(r'D:\Home\train.csv')&#xA;df=sqlc.createDataFrame(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but it is showing error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Can not infer schema for type: &amp;lt;class 'str'&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First 2 rows of df are :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['&quot;id&quot;,&quot;product_uid&quot;,&quot;product_title&quot;,&quot;search_term&quot;,&quot;relevance&quot;',&#xA; '2,100001,&quot;Simpson Strong-Tie 12-Gauge Angle&quot;,&quot;angle bracket&quot;,3']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think the first row is creating this problem. Moreover I want to create data frame which stores the values from 2nd row to last.(Not the first row because it will be the header). How can I achieve this ? I've searched for it but could not find any solution. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="23138" LastEditorUserId="20544" LastEditDate="2016-08-05T18:48:49.707" LastActivityDate="2016-08-06T17:19:11.320" Title="Converting RDD to spark data frames in python and then accessing a particular values of columns" Tags="|pyspark|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13239" PostTypeId="2" ParentId="13231" CreationDate="2016-08-05T13:13:42.643" Score="-1" Body="&lt;p&gt;There are many algorithms that can be used to perform classifications ( many to the point that it is difficult to mention all of them ) &#xA;I suggest you to have a look at this &#xA;&lt;a href=&quot;http://dlib.net/ml_guide.svg&quot; rel=&quot;nofollow&quot;&gt;http://dlib.net/ml_guide.svg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Making the decision which algorithm to use is a function of the problem you are working with, mainly:&#xA;1. The number of classes&#xA;2. The number of samples&#xA;3. The variations within classes and similarities between classes&#xA;4. Data imbalance&#xA;5. The dimension of your feature &#xA;And many other parameters &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, CNN is very popular for two reasons:&#xA;They can lead to high performance in very challenging problems and they are general solutions in the context that you need to understand their architectures the strategies and tricks to perform training only, after that you do not need to change anything , no parameters to play with . &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-08-05T13:13:42.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13241" PostTypeId="2" ParentId="12617" CreationDate="2016-08-05T15:18:25.183" Score="1" Body="&lt;p&gt;Well, thats a question that institutes at univiersities, among others, are trying to solve since several decades and where millions of money currencies are spent in each country every year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Short answer: for now, nothing &lt;em&gt;general&lt;/em&gt; is known to perform good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long answer: it depends completely on your image domain. Do you want to recognize suns in a picture? That possible up to some false discovery rates. Do you want recognize familiar faces? Possible, see automated passport controls at airports. Emotion recognition? Good luck with that! Grouping by persons? Possible, but statues will be included. etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something really general:  Grouping by colors? Totally viable! But will not mean much. Something that includes some hint to the structure? Well, the easiest way is to compute color histograms and use them as feature vectors for (like suggested) nearest neighbor calculation - (watch out for normalization, color space, and other sources of information distortion)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basic answer: take some Computer Vision courses either online (e.g. &lt;a href=&quot;https://youtu.be/skaQfPQFSyY?list=PL7v9EfkjLswLfjcI-qia-Z-e3ntl9l6vp&quot; rel=&quot;nofollow&quot;&gt;https://youtu.be/skaQfPQFSyY?list=PL7v9EfkjLswLfjcI-qia-Z-e3ntl9l6vp&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also an intersting TED talk to that topic &lt;a href=&quot;https://youtu.be/40riCqvRoMs&quot; rel=&quot;nofollow&quot;&gt;https://youtu.be/40riCqvRoMs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other side: don't let yourself down! Understand the #basics of data processing and break the structures! Develop that algorithm we are all hoping to use some day!&lt;/p&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-08-05T15:18:25.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13242" PostTypeId="2" ParentId="13238" CreationDate="2016-08-05T15:41:35.160" Score="1" Body="&lt;p&gt;To read a csv file to spark dataframe you should use spark-csv. &#xA;&lt;a href=&quot;https://github.com/databricks/spark-csv&quot; rel=&quot;nofollow&quot;&gt;https://github.com/databricks/spark-csv&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to use spark csv&lt;/strong&gt;&#xA;If you are using pyspark directly from the terminal. Instead of calling &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$SPARKHOME/bin/pyspark&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have to use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$SPARKHOME/bin/pyspark --packages com.databricks:spark-csv_2.11:1.4.0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then use the code above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using ipython + findspark, you'll have to modify your PYSPARK_SUBMIT_ARGS (before starting ipython)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;export PYSPARK_SUBMIT_ARGS=--master local[4] --packages &quot;com.databricks:spark-csv_2.11:1.4.0&quot; pyspark-shell &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21145" LastEditorUserId="21145" LastEditDate="2016-08-06T17:19:11.320" LastActivityDate="2016-08-06T17:19:11.320" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13243" PostTypeId="2" ParentId="13231" CreationDate="2016-08-05T15:44:32.667" Score="4" Body="&lt;p&gt;Part of the problem with answering this question is there are actually two questions.  The first:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Are there any image classification algorithms which are not neural networks?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, lots. But now the actually question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Is there any paper which tries an approach which does not use neural networks in LSVRC?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your question, you rule out methods such as kNN and SVM because of speed. Bag of Words is one method used to solve this problem.  MATLAB has a good demonstration (&lt;a href=&quot;http://www.mathworks.com/help/vision/examples/image-category-classification-using-bag-of-features.html&quot; rel=&quot;nofollow&quot;&gt;http://www.mathworks.com/help/vision/examples/image-category-classification-using-bag-of-features.html&lt;/a&gt;).  But BoW incorporates k-means clustering, so that may not fit your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some other interesting image classification methods such as texture analysis. TA is being researched as a way to classify malevolence of disease in medical images (such as tumors).  Here is a commonly referenced paper:&#xA;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701316/&quot; rel=&quot;nofollow&quot;&gt;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701316/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an overview of image classification:&#xA;&lt;a href=&quot;http://www.tandfonline.com/doi/full/10.1080/01431160600746456&quot; rel=&quot;nofollow&quot;&gt;http://www.tandfonline.com/doi/full/10.1080/01431160600746456&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="19161" LastEditDate="2016-08-05T15:49:43.343" LastActivityDate="2016-08-05T15:49:43.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13244" PostTypeId="1" CreationDate="2016-08-05T16:01:59.317" Score="2" ViewCount="828" Body="&lt;p&gt;I'm trying to train neural networks to extract parameters out of input data.&#xA;In particular determining the width, height and color of bricks (see attached pictures as example inputs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already tried CNNs operating on the raw images and NNs operating on extracted features ( 2 describing edges and a color histogram ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both approaches seem to learn the mean of the training data. The color is mostly gray and the width and height between the two extremes. The output does not depend on the input as the NN always outputs the &quot;same&quot; values (only very small fluctuations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm operating on generated data I have a perfect correlation between input and desired output and infinite training data. The used loss function is mean squared error (also tried the absolute error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain why this is happening or how to avoid this training behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the library &lt;a href=&quot;https://github.com/nyanp/tiny-cnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;tiny-cnn&lt;/a&gt;. &#xA;My current approach (using features) uses 5 fully connected layers, decreasing layer size from 1200 to 17(amount of parameters).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    net &amp;lt;&amp;lt; fully_connected_layer&amp;lt;sigmoid&amp;gt;(1200, 600)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;sigmoid&amp;gt;(600, 300)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;sigmoid&amp;gt;(300, 150)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;sigmoid&amp;gt;(150, 60)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;identity&amp;gt;(60, 17);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've test several CNN structures, always ending in 2 fully connected layers. I tried different kernel sizes from 3 to 11. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one of the last I tried:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net &amp;lt;&amp;lt; convolutional_layer&amp;lt;tan_h&amp;gt;(100, 100, 5, 3, 4)&#xA;    &amp;lt;&amp;lt; average_pooling_layer&amp;lt;tan_h&amp;gt;(96, 96, 4, 2)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;tan_h&amp;gt;(48, 48, 7, 4, 8)&#xA;    &amp;lt;&amp;lt; average_pooling_layer&amp;lt;tan_h&amp;gt;(42, 42, 8, 2)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;tan_h&amp;gt;(21, 21, 7, 8, 16)&#xA;    &amp;lt;&amp;lt; average_pooling_layer&amp;lt;tan_h&amp;gt;(15, 15, 16, 3)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;tan_h&amp;gt;(5, 5, 5, 16, 32)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;relu&amp;gt;(32, 64)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;identity&amp;gt;(64, size);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;5-conv, average, 7-conv, average, 7-conv, average(3x3), 5-conv, full, full&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've mainly used RMSProp and adagrad with standard learning rates but also tried higher and lower rates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example inputs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BOv4D.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BOv4D.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6UJHH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6UJHH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/CITUN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CITUN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20487" LastEditorUserId="836" LastEditDate="2016-08-09T12:22:46.237" LastActivityDate="2016-09-08T15:26:44.087" Title="Neural Net for regression task on images only learning mean of training data" Tags="|neural-network|regression|image-recognition|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="13246" PostTypeId="2" ParentId="12179" CreationDate="2016-08-05T20:36:36.350" Score="1" Body="&lt;p&gt;If I understand you correctly, based on the data and the given nearest neighbors, you need to decide what distance calculation was used to determine the nearest neighbors.  It seems unlikely that you are being asked to create an altogether new distance measure, rather explore the different accepted distance measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is correct, it sounds like all you need to do is run the nearest neighbors recursively with different distance calculations and compare the results to the provided nearest neighbors to determine which measure created the 5 nearest neighbors you were given.&lt;/p&gt;&#xA;" OwnerUserId="23159" LastActivityDate="2016-08-05T20:36:36.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13247" PostTypeId="2" ParentId="13169" CreationDate="2016-08-05T21:12:07.217" Score="0" Body="&lt;p&gt;So one way to do it is to use python slice operators to grab every other value in the line and zip them (along with the filename) into 3-tuples, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = []&#xA;malletOutput = open('doc-topics','r').readlines()&#xA;&#xA;for line in malletOutput:&#xA;    line = line.split('\t')[1:-1] # slicing out useless leading index and trailing \n&#xA;    _id = line[0]&#xA;&#xA;    tIndicies = map(int,line[1::2])&#xA;    tVals = map(float, line[2::2])&#xA;    topics = sorted(zip(tIndicies, tVals))&#xA;    topics = [t + tuple([_id]) for t in topics] # add _id&#xA;    for t in topics:&#xA;        data.append(t)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you have a list of 3-tuples which you can add to a dataframe. A simple pivot from there results in the desired shape:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame(data)&#xA;df = df.pivot(index=2,columns=0,values=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22061" LastActivityDate="2016-08-05T21:12:07.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13248" PostTypeId="1" CreationDate="2016-08-05T21:26:58.590" Score="1" ViewCount="160" Body="&lt;p&gt;I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's price for Product A in London, but not New York. I am looking for guidance on how to impute these values given the data that I already have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are trends to be exploited in the data. Such as: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Geographical trends&lt;/strong&gt;: i.e, London and New York are similar markets as measured by the median price of a given product, but markets like Cairo and Johannesburg tend to be much more expensive (though in varying degrees for different products)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Product Trends&lt;/strong&gt;: Within a given market, for a given company, products are priced hierarchically, although the relationship is nonlinear. For example, if the products are &quot;Good&quot;, &quot;Better&quot; and &quot;Best&quot;, the price of &quot;Better&quot; might be 2X&quot;Good&quot;, and the price of &quot;Best&quot; might be 2.5X &quot;Good&quot;, though these relationships are not strictly held.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Company Trends&lt;/strong&gt;: Certain companies tend to have higher prices than others in most markets, but again, these relationships are not strictly held, and may differ for certain subsets of products or markets where a company has a competitive advantage.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I describe the method that I have tried so far below, though I feel it is overly simplistic. I am looking for suggestions as to how to better capture the complexity of my data and obtain a reasonable accurate imputation for the missing values. Any advice is appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method I have tried so far is to estimate coefficients for each category. For example, holding Company and Product constant, what is the average price multiple of Market B over Market A? Market C over Market A? Then, if I know I have a price for (Company X, product &quot;Good&quot;) in markets B and C, I'll multiply them by those average multiples in order to obtain a &quot;best guess based on market&quot;. Then I repeat that holding constant Company and Market, and Market and Product. At the end, I'm left with three &quot;best guesses&quot;. Then the issue is converting my three best guesses into a single guess, because, based on some testing of this method, the true price may lie either between or outside these three best guesses. This is the histogram of errors based on a training sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uYw7Q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uYw7Q.png&quot; alt=&quot;Histogram: estimate as % of actual&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17594" LastActivityDate="2016-08-17T16:20:37.997" Title="Method for predicting price based on Geographical market, Product, and Company" Tags="|predictive-modeling|categorical-data|data-imputation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13249" PostTypeId="1" CreationDate="2016-08-05T23:02:21.727" Score="4" ViewCount="3219" Body="&lt;p&gt;I am having trouble in understanding the generalized likelihood ratio test (GLRT). Can anyone explain what it is to me, or point me toward an easy-to-understand reference? Is it a supervised or unsupervised method? How GLRT relates to Bayesian approaches?&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="12345" LastEditDate="2016-08-05T23:13:37.133" LastActivityDate="2023-04-21T07:02:02.687" Title="generalized likelihood ratio test (GLRT)" Tags="|machine-learning|supervised-learning|unsupervised-learning|parameter-estimation|bayesian|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13250" PostTypeId="1" AcceptedAnswerId="13345" CreationDate="2016-08-06T03:06:58.170" Score="3" ViewCount="524" Body="&lt;p&gt;In continuation to &lt;a href=&quot;https://datascience.stackexchange.com/questions/13227/different-runs-of-feature-selection-algorithm-giving-different-set-of-selected-f?noredirect=1#comment14523_13227&quot;&gt;this&lt;/a&gt; question, I have a conceptual question. If I am not using the 'nfeatures' then I am getting 7 features from a feature set of 515 features. If I use the 'nnfeatures' tag (this tag specifies the minimum number of features I want it to display) with a value of 10, then I am getting those 7 faetures along with 3 new one. Now I am unable to decide how many features should I take. Is it okay to take 7 features out of 515 features? Will any ML algorithm be able to capture the characteristics of the data with just these 7 features? &#xA;&lt;br&gt;&lt;br&gt;&#xA;Another question is, a criterion value is given when the sequentialfs algorithm is applied in MATLAB. What is this criterion function? From &lt;a href=&quot;https://datascience.stackexchange.com/questions/13227/different-runs-of-feature-selection-algorithm-giving-different-set-of-selected-f?noredirect=1#comment14523_13227&quot;&gt;this&lt;/a&gt; page I came to know that a set of features are chosen whose mean criterion value is the lowest. So does it mean that only those features are chosen with the minimum criterion value? Also, if this is the case, then how does the algorithm decide how many features it should select?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-08-10T05:06:31.597" Title="How many features should I take out of 515 features?" Tags="|machine-learning|feature-selection|feature-extraction|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13251" PostTypeId="2" ParentId="13249" CreationDate="2016-08-06T04:43:32.283" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Likelihood-ratio_test&quot; rel=&quot;nofollow&quot;&gt;Likelihood-ratio tests&lt;/a&gt; are a mainstay of &lt;a href=&quot;https://en.wikipedia.org/wiki/Frequentist_inference&quot; rel=&quot;nofollow&quot;&gt;classical&lt;/a&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow&quot;&gt;hypothesis testing&lt;/a&gt;. The idea is to form the likelihoods of the two hypotheses under consideration, and choose the one with the highest likelihood if their &lt;em&gt;ratio&lt;/em&gt; is sufficiently large. Hypotheses come in two flavors: simple, and composite. Simple tests are those for which the hypothesis uniquely defines the distribution; e.g., the mean is that, or the variance is that. If it is not simple, it is composite; e.g., the mean is &lt;em&gt;not&lt;/em&gt; equal to something, or the variance is &lt;em&gt;less than&lt;/em&gt; something. &lt;em&gt;Generalized&lt;/em&gt; likelihood ratio tests apply to composite hypotheses, and the goal is to find the distributions out of all possible options in the hypothesis space that maximize the likelihoods, and consider their ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Bayesian statistics one considers the &lt;a href=&quot;https://en.wikipedia.org/wiki/Posterior_probability&quot; rel=&quot;nofollow&quot;&gt;posterior probabilities&lt;/a&gt; instead of the likehoods, and the corresponding ratio is termed the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayes_factor&quot; rel=&quot;nofollow&quot;&gt;Bayes factor&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supervised/unsupervised learning is something else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you consult a textbook for worked examples; I think my explanation does not do the subject justice.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-08-06T04:48:49.547" LastActivityDate="2016-08-06T04:48:49.547" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13252" PostTypeId="2" ParentId="13227" CreationDate="2016-08-06T09:34:40.460" Score="0" Body="&lt;p&gt;You can set a &lt;code&gt;seed&lt;/code&gt; or a &lt;code&gt;random state&lt;/code&gt; for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done by &lt;code&gt;set_seed(...)&lt;/code&gt; in R, and adding the &lt;code&gt;random_state = ...&lt;/code&gt; in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-08-06T09:34:40.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13253" PostTypeId="2" ParentId="13191" CreationDate="2016-08-06T11:59:37.267" Score="1" Body="&lt;p&gt;For this specific paper, the conditional distribution is calculated as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(f_i = v |f_{1:i-1} ) = \frac{\exp(o_{i,v})}{\sum_{v=1}^V\exp(o_{i,v})} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on page 1702&lt;/p&gt;&#xA;" OwnerUserId="23097" LastActivityDate="2016-08-06T11:59:37.267" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13254" PostTypeId="2" ParentId="11632" CreationDate="2016-08-06T12:17:06.817" Score="0" Body="&lt;p&gt;Perhaps you mean this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are doing multi-label classification, so each of your examples might be associated with multiple labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the image, any example in the labelled 'shared subspace' could be labelled as A and B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uBmVt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uBmVt.png&quot; alt=&quot;Shared subspace&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23110" LastActivityDate="2016-08-06T12:17:06.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13255" PostTypeId="1" CreationDate="2016-08-06T14:39:04.060" Score="1" ViewCount="264" Body="&lt;p&gt;I came across &lt;a href=&quot;http://blog.datadive.net/selecting-good-features-part-iii-random-forests/&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt; which talks about feature selections in random forest&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It introduces two methods&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean decrease impurity&lt;/li&gt;&#xA;&lt;li&gt;Mean decrease accuracy&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to know if there are more methods and when should one method be preferred. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also is it correct to say that less feature for random forest always gives worse result and the only gain is in computation speed?&lt;/p&gt;&#xA;" OwnerUserId="21955" LastEditorUserId="11097" LastEditDate="2016-08-06T17:29:23.637" LastActivityDate="2016-08-06T17:29:23.637" Title="how does feature selection for random forest help" Tags="|feature-selection|random-forest|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13256" PostTypeId="2" ParentId="13248" CreationDate="2016-08-06T18:19:14.557" Score="1" Body="&lt;p&gt;Your method looks interesting. I will borrow it when I have the chance. Let me make a couple of suggestions (&quot;suggestions&quot;, not &quot;prescriptions&quot;):  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The price multiple is usually refer as &quot;index.&quot; Consider indexing against an over-all weighted average, instead of doing the pairwise indexing. A weighted average index will make comparisons against a more stable denominator. To circumvent the missing values in the index, you can use the indexed lagged one unit of time, use it with today's values, and make updates (replacing today NA's with predictions) to be used tomorrow.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the past I tried the following: if the value of a quantity $X$ at time $t$ is not available, use $\hat X(t) = \text{most recent observed value of $X$}$. It did not work very well in my case for what I was trying to do, but if the dynamics of the quantities you are tracking is not too wild, it might work for you. You may combine both ideas, using 2) to update the baseline of the index in 1).  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Remember that there are no &quot;best predictions&quot;. There are only &quot;reasonable prediction methods&quot;, and &quot;useful predictions.&quot; My suggestion of using some average as baline for the index is borne in from the fact that averages are more stable than single observations.    &lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-06T18:19:14.557" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13257" PostTypeId="2" ParentId="13212" CreationDate="2016-08-06T18:43:06.953" Score="0" Body="&lt;p&gt;In the first model, there are &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1 (the constant) + N (the lagged variables) + C (dummy for countries )&#xA;  - 1(because Dummy_US + Dummy_Mx + ... = 1) = N + C&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;coefficients to be fitted, here C = #countries.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second model, there are &lt;code&gt;C*(N+1) coefficients&lt;/code&gt; to be fitted.&#xA;You should use the second approach if the variance of the error terms are very different. If not, you can get the C*(N+1) parameters by fitting the model&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X(Dependent variable 'Spending') = X(lag1) + ...+ X(lagN) +     +&#xA;  X(lag1)*X(Dummy US) + ...+ X(lagN)*X(Dummy US) + X(Dummy US)&lt;br&gt;&#xA;  + X(lag1)*X(Dummy Mx) + ...+ X(lagN)*X(Dummy Mx) + X(Dummy Mx)    ... + Error term&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(drop the dummy for one country, which will be the reference county)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model involves &lt;code&gt;C*(N+1) coefficients&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The coefficients for the reference country are:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; constant  &#xA; coeff of X(lag1)&#xA;...&#xA; coef of X(lagN)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The coefficients to forecast values for another country, for example the US, are:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; constant + coeff of X(Dummy US)&#xA; coeff of X(lag1) + ceoff of X(lag1)*X(Dummy US)&#xA;...&#xA; coef of X(lagN) + ceoff of X(lagN)*X(Dummy US)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7763" LastEditorUserId="10902" LastEditDate="2016-11-06T13:43:13.520" LastActivityDate="2016-11-06T13:43:13.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13258" PostTypeId="1" CreationDate="2016-08-06T19:02:05.787" Score="2" ViewCount="974" Body="&lt;p&gt;I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling to find the data for the purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please give me some pointers on where to find such kind of data or what kind of features are considered for such model, so that I can try to simulate the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance. &lt;/p&gt;&#xA;" OwnerUserId="23129" LastEditorUserId="29169" LastEditDate="2019-09-08T17:37:20.650" LastActivityDate="2019-10-08T21:01:25.420" Title="Multi-touch Attribution Model" Tags="|machine-learning|r|data-mining|data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13259" PostTypeId="1" AcceptedAnswerId="13263" CreationDate="2016-08-06T19:24:54.217" Score="0" ViewCount="139" Body="&lt;p&gt;I'm wondering that is there any features that can help in differentiating the following two images. I mean differentiating in related numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D5LEo.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D5LEo.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7DyUB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7DyUB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="94454" LastEditDate="2020-04-13T15:13:40.437" LastActivityDate="2020-04-13T15:13:40.437" Title="Which features can help to differentiate these two density?" Tags="|machine-learning|statistics|feature-extraction|feature-engineering|probability|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13260" PostTypeId="1" AcceptedAnswerId="13264" CreationDate="2016-08-06T19:46:13.653" Score="1" ViewCount="2327" Body="&lt;p&gt;What methods could someone use to find out what products are most frequently grouped with each other per order? Are there applications that can make achieving this goal easier?&lt;/p&gt;&#xA;" OwnerUserId="23182" LastActivityDate="2016-08-07T04:52:23.793" Title="How do companies like Amazon track what products are most frequently bought together?" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ClosedDate="2016-08-07T16:52:55.707" ContentLicense="CC BY-SA 3.0" />
  <row Id="13261" PostTypeId="2" ParentId="13213" CreationDate="2016-08-06T22:16:12.357" Score="1" Body="&lt;p&gt;Theoretically there is no problem. I've seen tree models put as predictors in logistic models. NN as input into a GLM moodel makes sense. The ultimate decition should be made based on the predictability of the NN.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to mind a few issues:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;model mantenance and deployment. The NN model would probably have more parameters than your vanilla GLM deployment might be more involved. You could live it frozen and never update it, and let the GLM model use the NN score as long as it adds to the GLM prediction.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;interpretability. There might be managers whose acountability is to review risk, and they may not be comfortable with a NN. In that case feeding the NN results into a GLM might make it more acceptable, much as credit scores are used in some risk models. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-06T22:16:12.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13262" PostTypeId="2" ParentId="13259" CreationDate="2016-08-06T22:22:59.080" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There must be some moments that distinguish these two, so keep on looking at higher powers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Kolmogorov-Smirnov statistic should work in this case.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-06T22:22:59.080" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13263" PostTypeId="2" ParentId="13259" CreationDate="2016-08-07T01:32:07.030" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Regarding two probability distributions $A,B$ on the line, that have densities, the following are equivalent:   &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$ A = B$ (i.e.: given any Borel set $\mathscr S$, the probability assign to $\mathscr S$ by A equals the probability assign to $\mathscr S$ by B)&lt;/li&gt;&#xA;&lt;li&gt;$A(-\infty,a] = B(-\infty,a] $, for any $a \in \mathbb R$, i.e.: $A,B$ have the same cumulative distribution function &lt;/li&gt;&#xA;&lt;li&gt;$A,B$ have the same densities.&lt;br&gt;&#xA;In the case of your question, the two densities are clearly different, so the two probabilities are different.  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;From your question, it seems you want something you can measure (perhaps on sample from these distributions). There are infinitely many quantities that you can measure that when apply to $A$ and $B$ produce different results. The statement above gives some hints for a possible solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your graphs, calling $A$ the first pdf, and $B$ the second:&#xA;$$ F_A(0.35) = A(-\infty, 0.35] = \text{ probability that an observation from the first distribution is &amp;lt; 0.35 } = 0 $$&#xA;since the density is 0 to the left of 0.35. On the other hand, &#xA;$$ F_B(0.35) = B(-\infty, 0.35] = \text{ probability that an observation from the second distribution is &amp;lt; 0.35 } &amp;gt; 0 $$&#xA;since the density is positive from -0.2 to 0.35.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can a little to the right of 0.35 and find a number $a_0$ such that $0 &amp;lt; F_A(a_0) &amp;lt; F_B(a_0) $.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Being less than $a_0$ is something one can measure on samples.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $A_1, .., A_n$ be an iid sample from the first distribution, and let $B_1,...,B_m$ be an iid sample from the second distribution. Then $x_j = I(A_j \le a_0)$ is a measure (Note: I am using the notation $I(S)$ = indicator function of a set $S$, or, in more typical notation in the context of probabilitie, $I(S)$ = indicator function of event $S$ = 1 if $S$ is true, 0 if $S$ is false).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, one can measure $y_j = I(B_j \le a_0).$   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\{x_j\}$ are iid Bernoulli, with $P(x_j=1) = P(A_j \le a_0) = F_A(a_0).$&lt;br&gt;&#xA;$ \frac{1}{n} \sum_1^n x_j $ is an estimator of $F_A(a_0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\{y_j\}$ are iid Bernoulli, with $P(y_j=1) = P(B_j \le a_0) = F_B(a_0).$&lt;br&gt;&#xA;$ \frac{1}{m} \sum_1^m y_j $ is an estimator of $F_B(a_0)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now one can run a rest of the hyposesis that the tw means are the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before proceeding, does this construction answer your question?&lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-07T01:32:07.030" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13264" PostTypeId="2" ParentId="13260" CreationDate="2016-08-07T04:52:23.793" Score="3" Body="&lt;p&gt;This is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow&quot;&gt;Association Learning&lt;/a&gt;. Quoting Wikpedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Association rule learning is a method for discovering interesting&#xA;  relations between variables in large databases. It is intended to&#xA;  identify strong rules discovered in databases using some measures of&#xA;  interestingness&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, your question asks about how companies like Amazon and Walmart know what products are frequently bought together. So, that is done by the &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow&quot;&gt;Apriori algorithm&lt;/a&gt;&lt;/strong&gt;, which falls under the class of Association Learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, quoting Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Apriori is an algorithm for frequent item set mining and association&#xA;  rule learning over transactional databases. It proceeds by identifying&#xA;  the frequent individual items in the database and extending them to&#xA;  larger and larger item sets as long as those item sets appear&#xA;  sufficiently often in the database.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This algorithm generates rules which show what items are bought along with which items, depending on the support and confidence thresholds. (As you didn't ask about the particulars, I would skip the details. Pl check wikipedia for the finer details and the algorithm itself).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;arules&lt;/code&gt; package of R implements the apriori algorithm. Unfortunately, there is no fast implementation of the same in Python.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-08-07T04:52:23.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13266" PostTypeId="1" AcceptedAnswerId="15146" CreationDate="2016-08-07T06:42:47.370" Score="6" ViewCount="5083" Body="&lt;p&gt;I was going through this &lt;a href=&quot;https://www.youtube.com/watch?v=kVfhpn0EuhQ&quot; rel=&quot;nofollow&quot;&gt;Udacity video&lt;/a&gt; (part of the ML nano-degree), where the math behind maximizing the length between the data clouds of two classes is done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the line closest to the +1 class cloud would be: $w^T x_1 + b = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, the line closest to the -1 class cloud would be: $w^T x_2 + b = -1$  ($x_1$ and $x_2$ being points of both the lines respectively.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for getting the distance between both of them (the lines above), the professors subtract both of the equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then, while finding ($x_1-x_2$), they divide both the sides with the norm of w ($||w||$), which starts at 2:24 minutes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From then, I didn't understand the math in the video. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why did the professors divide the sides by the norm?  What do the norm specify/imply?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please explain the math from after 2:24?&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="23283" LastEditDate="2016-08-11T09:37:17.723" LastActivityDate="2018-10-20T11:38:51.890" Title="Understanding the math behind SVM" Tags="|machine-learning|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13267" PostTypeId="2" ParentId="13266" CreationDate="2016-08-07T07:31:03.603" Score="3" Body="&lt;p&gt;Without any context, division by the norm of a vector can usually be interpreted as only retaining the direction of a vector while ignoring its magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it turns out, retaining only direction is especially important when projecting a vector onto another vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, it turns out that finding the shortest distance between two lines is also based on projections. In this case, however, the goal is to find a perpendicular line between the two lines, which is actually the original line less its projected part on the other line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More here: &lt;a href=&quot;https://en.wikipedia.org/wiki/Vector_projection&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Vector_projection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23187" LastActivityDate="2016-08-07T07:31:03.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13268" PostTypeId="2" ParentId="13033" CreationDate="2016-08-07T07:44:52.967" Score="3" Body="&lt;p&gt;I don't think clustering is the right approach here. You want to merge them automatically, but clustering is too unpredictable for this; you need to manually examine clustering results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you look at &lt;strong&gt;near duplicate detection&lt;/strong&gt; instead. You want to merge documents that have a high overlap. There are some interesting - and scalable - algorithma for that. In particular, minhash (and later work) is worth looking at.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-08-07T07:44:52.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13271" PostTypeId="1" CreationDate="2016-08-07T19:39:17.753" Score="0" ViewCount="430" Body="&lt;p&gt;I am trying to train a neural network with the lasagne module in Python. I do not want a fully connected network as defined by lasagne.layers.DenseLayer. Instead, I would like to fix some of the weight parameters to zero. Does anyone know how to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest solution I have found is something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = lasagne.layers.get_all_params(network, trainable=True)&#xA;layer1.params[layer1.W].remove(&quot;trainable&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, this fixes the entire set of weight parameters to their initial values. How can I fix only a subset of these weights?&lt;/p&gt;&#xA;" OwnerUserId="23195" LastEditorUserId="11097" LastEditDate="2016-08-08T01:57:46.040" LastActivityDate="2016-08-08T16:49:15.967" Title="Neural network with fixed weights in Python lasagne" Tags="|python|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13272" PostTypeId="1" AcceptedAnswerId="13274" CreationDate="2016-08-07T22:05:11.457" Score="1" ViewCount="1285" Body="&lt;p&gt;Have access to a dataset with hundreds of variables and millions of cases (American Community Survey).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Need to identify a small, manageable set of Independent Variables (&lt;strong&gt;IVs&lt;/strong&gt;) to use for Multiple Regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to do this, of course, would be to use applicable theories to identify the IVs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was wondering how I could use a data-driven (data-mining?) approach as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use a Decision Tree to identify impactful (candidate? relevant?) IVs?&lt;/li&gt;&#xA;&lt;li&gt;And then use &lt;strong&gt;&lt;em&gt;these&lt;/em&gt;&lt;/strong&gt; as the IVs in the Multiple Regression?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;(Seem to remember reading once, in passing, that this approach to variable reduction is permitted.)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tried searching on Google for articles that clarify the above, but the search terms are such that I keep getting hits to articles that compare Decision Trees and Multiple Regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you know of articles and research papers that describe how to do the above, please leave links below. Also, I would welcome your own original suggestions on how to proceed.&lt;/p&gt;&#xA;" OwnerUserId="16928" LastEditorUserId="16928" LastEditDate="2016-08-08T00:11:29.107" LastActivityDate="2016-08-08T02:26:04.763" Title="Using Decision Tree methodology to identify Independent Variables for Multiple Regression" Tags="|regression|linear-regression|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13273" PostTypeId="1" CreationDate="2016-08-07T22:37:35.173" Score="2" ViewCount="17871" Body="&lt;p&gt;I have a dataset with categorical features. I want to segment the data using clustering techniques. What could be the possible choices for this scenariogiven the fact that data has categorical features. Is there any variation of k-means which can be used here.&lt;/p&gt;&#xA;" OwnerUserId="23198" LastActivityDate="2016-08-08T21:12:42.117" Title="Clustering categorical data" Tags="|data-mining|" AnswerCount="4" CommentCount="3" ClosedDate="2016-08-09T10:05:47.307" ContentLicense="CC BY-SA 3.0" />
  <row Id="13274" PostTypeId="2" ParentId="13272" CreationDate="2016-08-08T02:26:04.763" Score="1" Body="&lt;p&gt;Decision trees are useful for determining nested/interactive relationships between combinations of IVs and a DV. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model you specified, a multiple regression, presupposes a relationship between the IVs and the DV (e.g. linear). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you are aware, these models are different. So using a decision tree coupled with some importance measure to find predictive variables won't necessarily provide you with an optimal set of IVs in a regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, it can be a helpful exercise to inform you of non-linear relationships or interaction terms that could be predictive, and which may not be captured by specifying a model such as a multiple regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were you, I wouldn't solely rely on using decision trees to determine a  set of IVs for a regression model. I would investigate penalized regression methods such as LASSO or ridge regression to help take you from a reduced candidate set of IVs to your final IVs. In addition, you might want to explore associative metrics related to your model specification that might be useful in exploring the relationships in your data, such as information values, chi-square tests, correlations, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may be helpful:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/47367/decision-tree-as-variable-selection-for-logistic-regression&quot;&gt;https://stats.stackexchange.com/questions/47367/decision-tree-as-variable-selection-for-logistic-regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15807" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-08T02:26:04.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13275" PostTypeId="2" ParentId="13173" CreationDate="2016-08-08T02:46:44.513" Score="1" Body="&lt;p&gt;Yes. If there wasn't a market, the largest consulting (&quot;Big 3&quot;) and audit (&quot;Big 4&quot;) firms wouldn't offer and continue to be expanding their advisory services in data &amp;amp; analytics. The same could be said for the thousands of independent consultants who offer such services (such as those who can be found on &lt;a href=&quot;https://www.upwork.com/&quot; rel=&quot;nofollow&quot;&gt;www.upwork.com&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="15807" LastActivityDate="2016-08-08T02:46:44.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13276" PostTypeId="2" ParentId="13273" CreationDate="2016-08-08T03:04:57.623" Score="1" Body="&lt;p&gt;Not enough reputation to comment...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any insight on whether your categorical variables exhibit some ordering? Or are they nominal? Is it possible to impose an ordering on your variables such that it is intuitive?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem comes down to choosing an appropriate distance metric. Or rather, what defines 'similarity' to you. There is a variant of the k-means algorithm called k-modes that you may want to explore. The last link below provides more information on this categorical clustering method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In absence of knowing more about your data, these links might be useful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/56479/cluster-analysis-on-ordinal-data-likert-scale&quot;&gt;https://stats.stackexchange.com/questions/56479/cluster-analysis-on-ordinal-data-likert-scale&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/28170/clustering-a-dataset-with-both-discrete-and-continuous-variables&quot;&gt;https://stats.stackexchange.com/questions/28170/clustering-a-dataset-with-both-discrete-and-continuous-variables&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;K-Means clustering for mixed numeric and categorical data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15807" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-08-08T03:04:57.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13277" PostTypeId="1" AcceptedAnswerId="13281" CreationDate="2016-08-08T04:32:35.520" Score="4" ViewCount="1396" Body="&lt;p&gt;I have applied the sequential forward selection to my dataset having 214 samples and 515 features (2 class problem). The feature selection algorithm has selected 8 features. Now I have applied the svm (MATLAB) on these 8 features. I have also tried to see the performance after adding more features. The table given below gives the correct rate of the algorithm (training data set) along with the feature set used. The result obtained is:&lt;br&gt;&lt;br&gt;&#xA;8 features = 0.9392&lt;br&gt;&#xA;10 features = 0.9439&lt;br&gt;&#xA;12 features = 0.9672&lt;br&gt;&#xA;14 features = 0.9672&lt;br&gt;&#xA;16 features = 0.9626 &lt;br&gt;&#xA;18 features = 0.9766&lt;br&gt;&#xA;20 features = 0.9672 &lt;br&gt;&#xA;&lt;br&gt;&lt;br&gt;&#xA;As visible, the accuracy seems to increase. Is it because of over fitting? Should I use the default feature set as given by the sequentialfs function of Matlab or should I force it to deliver more features to get more accuracy?&lt;br&gt;&lt;a href=&quot;https://i.stack.imgur.com/DcMXn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DcMXn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I have uploaded the validation training and testing performance (70-15-15). Now can you tell me if my data is being over-fitted or not?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2016-08-08T07:25:24.820" LastActivityDate="2016-08-08T13:49:04.907" Title="Is there a problem of over fitting in my dataset?" Tags="|feature-selection|svm|matlab|overfitting|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13279" PostTypeId="1" AcceptedAnswerId="13488" CreationDate="2016-08-08T05:45:31.677" Score="3" ViewCount="1229" Body="&lt;p&gt;I have a book Recommendation System project and have a huge data set of feature vectors. What is the best solution for in memory computation? I mean, &lt;strong&gt;the program should&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;calculate the cosine Similarity among Database Books&lt;/li&gt;&#xA;&lt;li&gt;be REAL TIME, because it's an interactive business site.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I've already thought about big data analysis tools and management such as Spark, Hadoop, ... . However, I am really new to Spark Techs and therefore not sure whether it's practical or not. I am really mixed up how can it be helpful? I have studied spark's documentation in java but it confused me more about how spark can be helpful? &lt;/p&gt;&#xA;" OwnerUserId="23201" LastEditorUserId="29169" LastEditDate="2019-06-07T16:57:17.333" LastActivityDate="2019-06-07T16:57:17.333" Title="Solution for in Time/Space Complexity challenge in Recommendation System?" Tags="|apache-spark|recommender-system|word2vec|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13280" PostTypeId="2" ParentId="13279" CreationDate="2016-08-08T06:27:12.120" Score="1" Body="&lt;p&gt;For in-memory computation wouldn't it be simpler to start with Python or R and one of their machine learning libraries? If the data does not fit into RAM, you can load only a part of it (for example, load only p fraction of vectors by loading each vector with probability p at random). Once you analyse your data, and understand which methods work best, you can scale up your system and rewrite your code in one of these distributed computation tools (or perhaps you find that you don't need this at all).&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2016-08-08T06:27:12.120" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13281" PostTypeId="2" ParentId="13277" CreationDate="2016-08-08T07:19:27.930" Score="5" Body="&lt;p&gt;It is not possible to tell whether a machine learning algorithm is overfitting based purely on the training set accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could be right, that using more features with a small data set increases sampling error and reduces the generalisation of the SVM model you are building. It is a valid concern, but you cannot say that for sure with only this worry and the training accuracy to look at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual solution to this is to keep some data aside to test your model. When you see a high training accuracy, but a low test accuracy, that is a classic sign of over-fitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often you are searching for the best hyper-parameters to your model. In your case you are trying to discover the best number of features to use. When you start to do that, you will need to make multiple tests in order to pick the best hyper-parameter values. At that point, a single test set becomes weaker measure of true generalisation (because you have had several attempts and picked best value - just by selection process you will tend to over-estimate the generalisation). So it is common practice to split the data three ways - training set, cross-validation set and test set. The cross-validation set is used to check accuracy as you change the parameters of your model, you pick the best results and then finally use the test set to measure accuracy of your best model. A common split ratio for this purpose is 60/20/20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking a pragmatic approach when using the train/cv/test split, it matters less that you are over or under fitting than simply getting the best result you can with your data and model class. You can use the feedback on whether you are over-fitting (high training accuracy, low cv accuracy) in order to change model parameters  - increase regularisation when you are over-fitting for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When there are a small number of examples, as in your case, then the cv accuracy measure is going to vary a lot depending on which items are in the cv set. This makes it hard to pick best hyper-params, because it may just be noise in the data that makes one choice better than another. To reduce the impact of this, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation&quot; rel=&quot;nofollow&quot;&gt;k-fold cross-validation&lt;/a&gt; - splitting your train/cv data multiple times and taking an average measure of the accuracy (or whatever metric you want to maximise).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In your confusion matrices, there is no evidence of over-fitting. A training accuracy of 100%* and testing accuracy of 93.8% are suggestive of some degree of over-fit, but the sample size is too low to read anything into it. You should bear in mind that balance between over- and under- fit is very narrow and most models will do one or the other to some degree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;* A training accuracy of 100% is nearly always &lt;em&gt;suggestive&lt;/em&gt; of overfit. Matched with e.g. 99% test accuracy, you may not be too concerned. The question is at worst &quot;could I do better by increasing regularisation a little&quot;? However, matched with ~60% test accuracy it is clear you have actually overfit - even then you might be forced to accept the situation if that's the best you could achieve after trying many different hyperparameter values (including some attempts with increased regularisation).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-08-08T13:49:04.907" LastActivityDate="2016-08-08T13:49:04.907" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="13283" PostTypeId="2" ParentId="13273" CreationDate="2016-08-08T09:03:32.437" Score="0" Body="&lt;p&gt;I don't really see a reason why simple K-Means clustering shouldn't work. If you convert your categorical data into integers (or encode to binary where one column is equal to one category, so called &quot;one-hot encoding&quot;), you can then fetch it into the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you can compare the cluster between each other by, lets say, calculate the mode to see the differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as dmanuge mentioned, playing with different metric can be helpful. But I'd go for this after the simple K-Means.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-08-08T09:03:32.437" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13284" PostTypeId="1" CreationDate="2016-08-08T09:29:38.910" Score="6" ViewCount="1277" Body="&lt;p&gt;I'm bit confused with the names of sentiment-analysis and twitter-sentiment-analysis.I want to do sentiment analysis for facebook fan page posts.so I decide to use an api but I did not found any api which gives sentiment analysis for facebook Posts.Every api service is made twitter.I found datumbox api which gives results for sentiment-analysis and twitter-sentiment analysis &lt;a href=&quot;http://www.datumbox.com/machine-learning-api/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for facebook sentiment analysis, do I need to go with sentiment-analysis api or twitter sentiment-analysis.(I know Name itself suggests sentiment is for twitter tweets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is their any api that provides sentiment analysis for facebook posts data ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;facebook trained datasets with the classes of POS,Neg,Neu?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="8501" LastEditDate="2016-09-09T09:52:32.673" LastActivityDate="2019-01-14T10:00:47.660" Title="What is the difference between sentiment analysis (e.g. on Facebook) and Twitter sentiment analysis?" Tags="|machine-learning|classification|sentiment-analysis|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13285" PostTypeId="2" ParentId="13284" CreationDate="2016-08-08T09:38:38.433" Score="1" Body="&lt;p&gt;Twitter sentiment analysis as the name suggests is ideal for Twitter data. They are different because for sentiment analysis of Twitter data, we can handpick some features for example, &lt;code&gt;emoticons&lt;/code&gt; can be tagged earlier with their sentiment and also &lt;code&gt;hashtags&lt;/code&gt; can be understood. Whereas a generic sentiment analysis mostly doesn't include these. Also, for twitter dataset the data is hardly a &lt;code&gt;sentence&lt;/code&gt; of two that is 160 characters for which the model might be different from others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know of any API specific for facebook data, but if your dataset is small you can give Twitter API a shot else if generic sentiment analysis might do a good option.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-08T09:38:38.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13286" PostTypeId="1" AcceptedAnswerId="13316" CreationDate="2016-08-08T09:57:46.317" Score="1" ViewCount="812" Body="&lt;p&gt;I am using the neural network toolbox of Matlab to train a network. Now my code is as follows:&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = xdata.';&#xA;t = target1';&#xA;% Create a Pattern Recognition Network&#xA;hiddenLayerSize = 10;&#xA;net = patternnet(hiddenLayerSize);&#xA;net.input.processFcns = {'removeconstantrows','mapminmax'};&#xA;net.output.processFcns = {'removeconstantrows','mapminmax'};&#xA;net.layers{2}.transferFcn = 'softmax';&#xA;net.divideFcn = 'dividerand';  % Divide data randomly&#xA;net.divideMode = 'sample';  % Divide up every sample&#xA;net.divideParam.trainRatio = 60/100;&#xA;net.divideParam.valRatio = 20/100;&#xA;net.divideParam.testRatio = 20/100;&#xA;net.trainFcn = 'trainscg';  % Scaled conjugate gradient&#xA;net.performFcn = 'mse';&#xA;net.performParam.regularization = 0.5;&#xA;%net.performParam.normalization = 0.01;&#xA;net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...&#xA;  'plotregression', 'plotfit', 'plotconfusion'};&#xA;&#xA;% Train the Network&#xA;[net,tr] = train(net,x,t);&#xA;% Test the Network&#xA;y = net(x);&#xA;e = gsubtract(t,y);&#xA;tind = vec2ind(t);&#xA;yind = vec2ind(y);&#xA;percentErrors = sum(tind ~= yind)/numel(tind);&#xA;performance = perform(net,t,y)&#xA;% Recalculate Training, Validation and Test Performance&#xA;trainTargets = t .* tr.trainMask{1};&#xA;valTargets = t  .* tr.valMask{1};&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;Now I am supposed to get different accuracies with different run of data since the sampling (division of dataset into train test and validation set) is random. But I am getting a constant accuracy (89.7%). The variable 'xdata' contains only those features selected by a feature selection algorithms. Is there any reason why my accuracy value is constant?&lt;br&gt;&lt;br&gt;&#xA;I have trained an SVM too with the same dataset. There too I am getting the a stale accuracy even with multiple run (94%)&lt;br&gt;&lt;br&gt;&#xA;The output y contains 2 values. What do those values signify? &lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2016-08-09T05:55:25.483" LastActivityDate="2016-08-09T06:07:14.130" Title="Accuracy value constant even after different runs" Tags="|neural-network|supervised-learning|performance|matlab|accuracy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13288" PostTypeId="1" AcceptedAnswerId="13311" CreationDate="2016-08-08T10:17:53.103" Score="0" ViewCount="113" Body="&lt;p&gt;Suppose I have a dataset which I want to train using Neural network and SVM. Is it possible that with my dataset after training, the Neural network is overfit while the SVM is not? Like can a dataset be overfit for one training algorithms and not be overfit for another training algorithm? Is it even possible? Or if it is not overfit for one training algorithm, can be assume that it will not be overfit for other training algorithms too?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2021-02-10T14:58:29.803" Title="Does the phenomenon of over-fitting of data varies with training algorithms?" Tags="|neural-network|svm|supervised-learning|training|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13292" PostTypeId="2" ParentId="13288" CreationDate="2016-08-08T12:09:40.340" Score="1" Body="&lt;p&gt;e.g. linear SVM is less prone to overfitting than non-linear SVM, especially when you have a large number of features (comparing to your training samples)&lt;/p&gt;&#xA;&lt;p&gt;But yeah in general the more complex a model is the more prone it is to overfitting, e.g., &lt;a href=&quot;http://arxiv.org/abs/1412.1897&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2454" LastEditorUserId="85045" LastEditDate="2021-02-10T14:58:29.803" LastActivityDate="2021-02-10T14:58:29.803" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13294" PostTypeId="1" CreationDate="2016-08-08T12:36:45.153" Score="1" ViewCount="63" Body="&lt;p&gt;I'm analyzing a single large table (about 50GB 16M rows, 15 columns perhaps more) currently using PostgreSQL 9.4.&#xA;Most of my queries doesn't involve more tables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Queries run time are very large. I'm comparing it to similar queries in Tableau.&#xA;I know this comparison is not fair, Tableau extract was designed for this usage.&#xA;Nevertheless i think that i'm not the only one looking for such a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i'm looking for is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Database (probably relational)&lt;/li&gt;&#xA;&lt;li&gt;Analytics oriented (no updates\deletes are required)&lt;/li&gt;&#xA;&lt;li&gt;Fast (like using a tableau extract)&lt;/li&gt;&#xA;&lt;li&gt;Supports SQL queries&lt;/li&gt;&#xA;&lt;li&gt;Preferably open-source&lt;/li&gt;&#xA;&lt;li&gt;Will not require much optimization (i don't want to be responsible for indexes creation, like in Tableau extract creation) &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To illustrate: if i had the ability to connect to a Tableau extract locally like to a localhost database and run SQL quires on it, this will be great.&lt;/p&gt;&#xA;" OwnerUserId="23211" LastActivityDate="2016-08-08T12:36:45.153" Title="Analytics oriented database" Tags="|performance|databases|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13295" PostTypeId="1" CreationDate="2016-08-08T13:01:19.127" Score="14" ViewCount="310" Body="&lt;p&gt;I have text documents which contain mainly lists of Items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each Item is a group of several token from different types: FirstName, LastName, BirthDate, PhoneNumber, City, Occupation, etc.&#xA;A token is a group of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Items can lie on several lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Items from a document have about the same token syntax, but they don't necessarily have to be exactly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They may be some more/less tokens between Items, as well as within Items.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FirstName LastName BirthDate PhoneNumber&#xA;Occupation City&#xA;FirstName LastName BirthDate PhoneNumber PhoneNumber&#xA;Occupation City&#xA;FirstName LastName BirthDate PhoneNumber&#xA;Occupation UnrecognizedToken&#xA;FirstName LastName PhoneNumber&#xA;Occupation City&#xA;FirstName LastName BirthDate PhoneNumber&#xA;City Occupation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The goal is to identify the used grammar, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Occupation City&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and in the end identify all the Items, even thought they don't exactly match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to stay short and readable, let's use instead some aliases A, B, C, D, ... to designate those token types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A B C&#xA;D F&#xA;A B C&#xA;D E F&#xA;F&#xA;A B C&#xA;D E E F&#xA;A C B&#xA;D E F&#xA;A B D C&#xA;D E F&#xA;A B C&#xA;D E F G&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here we can see that the Item syntax is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A B C&#xA;D E F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;because it is the one that matches the best the sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The syntax (token types and orders) can vary a lot from one document to another.&#xA;e.g. another document may have that list&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;D A&#xA;D A&#xA;D&#xA;D A&#xA;B&#xA;D A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The goal is to figure out that syntax &lt;em&gt;without prior knowledge of it&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From now, a new line is considered as a token as well.&#xA;A document can then be represented as a 1-dimension sequence of tokens:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here the repeated sequence would be &lt;code&gt;A B C B&lt;/code&gt; because it is the token that creates the least conflicts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's complexify it a bit.&#xA;From now each token has no determined type.&#xA;In the real world, we are not always 100% sure of some token's type. Instead, we give it a probability of having a certain type.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  A 0.2    A 0.0    A 0.1&#xA;  B 0.5    B 0.5    B 0.9     etc.&#xA;  C 0.0    C 0.0    C 0.0&#xA;  D 0.3    D 0.5    D 0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is an abstract graphic of what I want to achieve:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/e2njr.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Solution considered A: Convolution of a patch of tokens&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This solution consists in applying a convolution with several patches of tokens, and take the one that creates the least conflicts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hard part here is to find potential patches to roll along the observe sequence.&#xA;Few ideas for this one, but nothing very satisfying:&lt;/p&gt;&#xA;&#xA;Build a Markov model of the transition between tokens&#xA;&#xA;&lt;p&gt;Drawback: since a Markov model has no memory, we will lose the orders of transition.&#xA;e.g. If the repeated sequence is &lt;code&gt;A B C B D&lt;/code&gt;, we lose the fact that A-&gt;B occurs before C-&gt;B.&lt;/p&gt;&#xA;&#xA;Build a suffix tree&#xA;&#xA;&lt;p&gt;This seems to be used extensively in biology in order to analyse nucleobases (GTAC) in DNA/RNA.&#xA;Drawback: Suffix trees are good for exact matching of exact tokens (e.g. characters). We have neither exact sequences, nor exact tokens.&lt;/p&gt;&#xA;&#xA;Brute force&#xA;&#xA;&lt;p&gt;Try every combination of every size. Could actually work, but would take some (long (long)) time.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Solution considered B: Build a table of Levenshtein distances of suffixes&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The intuition is that there may exist some local minima of distance when computing the distance from every suffix to every suffix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distance function is the Levenshtein distance, but we will be able to customize it in the future in order to take in account the probability of being of a certain types, instead of having a fixed type for each token.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to stay simple in that demonstration, we will use fixed-type tokens, and use the classic Levenshtein to compute the distance between tokens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. Let's have the input sequence &lt;code&gt;ABCGDEFGH ABCDEFGH ABCDNEFGH&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We compute the distance of every suffix with every suffix (cropped to be of equal size):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i = 0 to sequence.lengh&#xA;  for j = i to sequence.lengh&#xA;    # Create the suffixes&#xA;    suffixA = sequence.substr(i)&#xA;    suffixB = sequence.substr(j)&#xA;    # Make the suffixes the same size&#xA;    chunkLen = Math.min(suffixA.length, suffixB.length)&#xA;    suffixA = suffixA.substr(0, chunkLen)&#xA;    suffixB = suffixB.substr(0, chunkLen)&#xA;    # Compute the distance&#xA;    distance[i][j] = LevenshteinDistance(suffixA, suffixB)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We get e.g. the following result (white is small distance, black is big):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/VFTBZ.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, it's obvious that any suffix compared to itself will have a null distance. But we are not interested by suffix (exactly or partially) matching itself, so we crop that part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ypYa8.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the suffixes are cropped to the same size, comparing long string will always yield a bigger distance than comparing smaller strings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to compensate that by a smooth penalty starting from the right (+P), fading out linearly to the left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure yet how to choose a good penalty function that would fit all cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we apply a (+P=6) penalty on the extreme right, fading out to 0 to the left.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/ccFge.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can clearly see 2 clear diagonal lines emerge. There are 3 Items (Item1, Item2, Item3) in that sequence. The longest line represents the matching between Item1 vs Item2 and Item2 vs Item3. The second longest represents the matching between Item1 vs Item3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am not sure on the best way to exploit that data. Is it as simple as taking the highest diagonal lines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's compute the average value of the diagonal line that start from each token.&#xA;We can see the result on the following picture (the vector below the matrix) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/PXoP7.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are clearly 3 local minima, that match the beginning of each Item.&#xA;Looks fantastic!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's add some more imperfections in the sequence:&#xA;&lt;code&gt;ABCGDEFGH ABCDEFGH TROLL ABCDEFGH&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/D2oYi.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly now, our vector of diagonal averages is messed up, and we cannot exploit it anymore...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My assumption is that this could be solved by a customized distance function (instead of Levenshtein), where the insertion of a whole block may not be so much penalized. That is what I am not sure of.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Conclusion&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;None of the explored convolution-based solutions seem to fit our problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Levenshtein-distance-based solution seems promising, especially because it is compatible with probability-based-type tokens. But I am not sure yet about how to exploit the results of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be very grateful if you have experience in a related field, and a couple of good hints to give us, or other techniques to explore.&#xA;Thank you very much in advance.&lt;/p&gt;&#xA;" OwnerUserId="23213" LastEditorUserId="25180" LastEditDate="2019-03-15T21:36:38.463" LastActivityDate="2019-03-15T21:36:38.463" Title="Recognize a grammar in a sequence of fuzzy tokens" Tags="|data-mining|clustering|text-mining|time-series|correlation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13297" PostTypeId="1" CreationDate="2016-08-08T14:55:05.983" Score="2" ViewCount="362" Body="&lt;p&gt;I have a question about basic understanding of how item-item collaborative filtering of &quot;Graphlab&quot; library works.&#xA;I run this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f = graphlab.SFrame({'user_id': ['0', '0', '0', '1', '1', '2', '2'],&#xA;                     'item_id': ['a', 'b', 'c', 'a', 'b', 'b', 'c'],&#xA;                     'rating': [ 1,   3,   2,   5,   4,   1,   4]})&#xA;m = graphlab.item_similarity_recommender.create(f, target=&quot;rating&quot;, similarity_type='cosine')&#xA;&#xA;print m.get_similar_items(verbose=True)&#xA;print m.recommend()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;The output is:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wshrq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wshrq.png&quot; alt=&quot;output&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The matrix form of data:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mYr2Z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mYr2Z.png&quot; alt=&quot;matrix&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the model, the recommendation for (1,c) is 1.096 but according to my calculation it is 4.16!&#xA;My calculation: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1,c) = (sim(c,a)*5 + sim(c,b)*4)/(sim(c,a) + sim(c,b))&#xA;      = (0.0877*5 + 0.4385*4)/(0.0877+0.4385)&#xA;      = 4.16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please help, what am I missing?&lt;/p&gt;&#xA;" OwnerUserId="23216" LastEditorUserId="11097" LastEditDate="2016-08-08T15:36:13.200" LastActivityDate="2016-11-29T03:08:27.287" Title="wrong prediction from graphlab.recommender.item_similarity_recommender" Tags="|machine-learning|predictive-modeling|recommender-system|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13298" PostTypeId="1" AcceptedAnswerId="13299" CreationDate="2016-08-08T15:37:18.030" Score="0" ViewCount="186" Body="&lt;p&gt;I am giving a presentation on Data Science, and I want to talk about the idea that data that is not &quot;big&quot; enough is a big barrier for Machiene Learning. Looking online, there are concepts like overfitting and underfitting, but I am more looking to talk about data that, even if fitted optimally, would still not actually be a good model for the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good term to use for this?&lt;/p&gt;&#xA;" OwnerUserId="21182" LastActivityDate="2016-08-08T16:44:23.860" Title="What is the term for data that is too sparse to represent the underlying data model?" Tags="|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13299" PostTypeId="2" ParentId="13298" CreationDate="2016-08-08T16:44:23.860" Score="2" Body="&lt;p&gt;Small sample size is probably the concept you're looking for. A common failure in statistics is trying to draw conclusions from data that isn't &quot;big&quot; enough to accurately represent the underlying distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth noting that &quot;small data&quot; increases your chances of overfitting - which is another way of saying that your model is weak to outliers and noise. It's also worth noting that bootstrapping, simulation, and duplication might help (but not always).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may be some other niche term for this, but if you relate it to basic stats there's a high probability everyone in the room will understand what you're talking about.&lt;/p&gt;&#xA;" OwnerUserId="12575" LastActivityDate="2016-08-08T16:44:23.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13300" PostTypeId="2" ParentId="13271" CreationDate="2016-08-08T16:49:15.967" Score="1" Body="&lt;p&gt;I'm not sure what your intention is by setting the weights to zero. Have you looked at dropout layers?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l_hid1 = lasagne.layers.DenseLayer(num_units=200)&#xA;&#xA;l_hid1_drop = lasagne.layers.DropoutLayer(l_hid1, p=0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should drop 50% of your data from the l_hid1 layer.&lt;/p&gt;&#xA;" OwnerUserId="21182" LastActivityDate="2016-08-08T16:49:15.967" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13301" PostTypeId="1" AcceptedAnswerId="14526" CreationDate="2016-08-08T17:08:44.627" Score="3" ViewCount="1212" Body="&lt;p&gt;The &quot;kernel&quot; $k(x,y) = e^{-\|x-y\|} $ is used in the context of SVM. Here $x,y \in \mathbb R^n$ and $\|.\|$ is the Euclidean norm.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a Hilbert space H and a map $\varphi:\mathbb R^n \rightarrow H$ such that $k(x,y) = &amp;lt;\varphi(x),\varphi(y)&amp;gt; $ for all $x,y \in \mathbb R^n$?&lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-10-14T11:44:29.763" Title="Does the &quot;laplacian&quot; kernel used in the SVM context come from a Hilbert space inner product?" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13302" PostTypeId="2" ParentId="13123" CreationDate="2016-08-08T17:27:14.953" Score="15" Body="&lt;p&gt;&quot;How can I import a .csv file into pyspark dataframes ?&quot; &#xA;-- there are many ways to do this; the simplest would be to start up pyspark with Databrick's spark-csv module. You can do this by starting pyspark with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pyspark --packages com.databricks:spark-csv_2.10:1.4.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then you can follow the following steps:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql import SQLContext&#xA;sqlContext = SQLContext(sc)&#xA;&#xA;df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The other method would be to read in the text file as an rdd using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;myrdd = sc.textFile(&quot;yourfile.csv&quot;).map(lambda line: line.split(&quot;,&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then transform your data so that every item is in the correct format for the schema (i.e. Ints, Strings, Floats, etc.). You'll want to then use &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from pyspark.sql import Row&#xA;&amp;gt;&amp;gt;&amp;gt; Person = Row('name', 'age')&#xA;&amp;gt;&amp;gt;&amp;gt; person = rdd.map(lambda r: Person(*r))&#xA;&amp;gt;&amp;gt;&amp;gt; df2 = sqlContext.createDataFrame(person)&#xA;&amp;gt;&amp;gt;&amp;gt; df2.collect()&#xA;[Row(name=u'Alice', age=1)]&#xA;&amp;gt;&amp;gt;&amp;gt; from pyspark.sql.types import *&#xA;&amp;gt;&amp;gt;&amp;gt; schema = StructType([&#xA;...    StructField(&quot;name&quot;, StringType(), True),&#xA;...    StructField(&quot;age&quot;, IntegerType(), True)])&#xA;&amp;gt;&amp;gt;&amp;gt; df3 = sqlContext.createDataFrame(rdd, schema)&#xA;&amp;gt;&amp;gt;&amp;gt; df3.collect()&#xA;[Row(name=u'Alice', age=1)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;http://spark.apache.org/docs/1.6.1/api/python/pyspark.sql.html#pyspark.sql.Row&quot; rel=&quot;noreferrer&quot;&gt;http://spark.apache.org/docs/1.6.1/api/python/pyspark.sql.html#pyspark.sql.Row&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Also, please tell me how can I import xlsx file?&quot; -- Excel files are not used in &quot;Big Data&quot;; Spark is meant to be used with large files or databases. If you have an Excel file that is 50GB in size, then you're doing things wrong. Excel wouldn't even be able to open a file that size; from my experience, anything above 20MB and Excel dies.&lt;/p&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2016-08-08T17:27:14.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13304" PostTypeId="1" CreationDate="2016-08-08T18:17:42.123" Score="5" ViewCount="762" Body="&lt;p&gt;After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.&lt;/p&gt;&#xA;&lt;p&gt;AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)&lt;/p&gt;&#xA;&lt;p&gt;As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12&lt;/p&gt;&#xA;&lt;p&gt;Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1KeDo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1KeDo.png&quot; alt=&quot;Results of AA testing&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;My question : I would like some pointers of people with more experience than me in data science to help me to :&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;- Understand if I am forgetting to check anything on my side to explain this discrepancy&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;The exact process that is used from our server to the final client is :&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;we send every x hours a list of a few hundred people to Responsys&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;then the list is filtered to remove undeliverable people, and each branch is sent the exact same email&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;During launch, for each email recipient:&lt;/p&gt;&#xA;&lt;p&gt;a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()&lt;/p&gt;&#xA;&lt;p&gt;b. decide the bucket this random number is in&lt;/p&gt;&#xA;&lt;p&gt;c. send the campaign in that bucket to the recipient&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="21473" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-03-08T22:16:38.993" Title="AB testing : When AA testing doesn't work" Tags="|statistics|ab-test|experiments|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13305" PostTypeId="2" ParentId="13273" CreationDate="2016-08-08T20:12:41.010" Score="0" Body="&lt;p&gt;Your approach may depend on the number of features and the number of categories in each feature that you are trying to include in your model.  I've used dummy variables to convert categorical data into numerical data and then used the dummy variables to do K-means clustering with some success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a small example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+----+----+----+&#xA;| ID | F1 | F2 |&#xA;+----+----+----+&#xA;|  1 | a  | x  |&#xA;|  2 | d  | w  |&#xA;|  3 | f  | x  |&#xA;+----+----+----+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Create a column for each category of each feature.  For each record, the value of the dummy variable field is 1 only in the dummy variable field that corresponds to the initial feature value.  The rest are 0.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+----+------+------+------+------+------+&#xA;| ID | F1_a | F1_d | F1_f | F2_w | F2_x |&#xA;+----+------+------+------+------+------+&#xA;|  1 |    1 |    0 |    0 |    0 |    1 |&#xA;|  2 |    0 |    1 |    0 |    1 |    0 |&#xA;|  3 |    0 |    0 |    1 |    0 |    1 |&#xA;+----+------+------+------+------+------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you're working with Pandas in Python, &lt;code&gt;pandas.get_dummies()&lt;/code&gt; can &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow&quot;&gt;generate the dummy variables for you.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes, you could have so many categories it would been unreasonable to try and create a dummy variable for each one.  For my problem, it was acceptable to only include in my model dummy variables for categories that occurred most frequently (e.g. Top 15 categories), but you'll have to decide whether or not that's appropriate for your problem.&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-08-08T20:12:41.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13306" PostTypeId="1" CreationDate="2016-08-08T20:51:42.173" Score="4" ViewCount="602" Body="&lt;p&gt;I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results.&lt;/p&gt;&#xA;&lt;p&gt;This is apparently the approach that Turi uses in their explain function which outputs something like this:&lt;/p&gt;&#xA;&lt;p&gt;Turi's explain function:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/69z2J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/69z2J.png&quot; alt=&quot;image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;from their page &lt;a href=&quot;https://turi.com/learn/userguide/churn_prediction/using-a-trained-model.html#explaining-predictions&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I know that for random forests you can average the contribution of each feature in every tree as seen in the TreeInterpreter python package, but this seems to be a different method since it is focused on exact splits and one decision tree.&lt;/p&gt;&#xA;&lt;p&gt;Does anyone know anything more about this method for interpreting gradient boosted trees?&lt;/p&gt;&#xA;" OwnerUserId="15768" LastEditorUserId="85045" LastEditDate="2021-03-23T13:25:53.073" LastActivityDate="2021-03-23T13:25:53.073" Title="Fit Decision Tree to Gradient Boosted Trees for Interpretability" Tags="|python|classification|decision-trees|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13307" PostTypeId="2" ParentId="13273" CreationDate="2016-08-08T21:12:42.117" Score="5" Body="&lt;p&gt;k-means is not a good choice, because it is designed for &lt;em&gt;continuous&lt;/em&gt; variables. It is a least-squares problem definition - a deviation of 2.0 is 4x as bad as a deviation of 1.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On binary data (such as one-hot encoded categorical data), this notion of squared deviations is not very appropriate. In particular, the cluster centroids are not binary vectors anymore!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;question you should ask first is: &quot;what &lt;em&gt;is&lt;/em&gt; a cluster&quot;&lt;/strong&gt;. Don't just hope an algorithm works. Choose (or build!) and algorithm that solves &lt;em&gt;your&lt;/em&gt; problem, not someone else's!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On categorical data, &lt;strong&gt;frequent itemsets&lt;/strong&gt; are usually the much better concept of a cluster than the centroid concept of k-means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-08-08T21:12:42.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13309" PostTypeId="2" ParentId="13306" CreationDate="2016-08-08T22:54:29.113" Score="4" Body="&lt;p&gt;Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual produced by the predictions from the earlier trees. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;explain&lt;/code&gt; method shows for each prediction (i.e. record) why a particular decision was made. This results in a set of decisions. Each tree makes several decisions. The decisions of all trees are stacked to get the set of rules that justifies a single decision. This information is specific for the gradient boosting model &lt;strong&gt;and&lt;/strong&gt; the specific record.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know more about the gradient boosting model, you may want to consider feature importance. Feature importance expresses how important the features are. You can use &lt;code&gt;print model.get_feature_importance()&lt;/code&gt; for this. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2016-08-08T22:54:29.113" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="13311" PostTypeId="2" ParentId="13288" CreationDate="2016-08-08T23:21:22.317" Score="3" Body="&lt;p&gt;A model is over-fitting if it makes good predictions on a test set but bad predictions on new data. This is generally a good indication that the used model is too complex. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complexity of the model is often quantified as the number of free parameters. These are the parameters that need to be set in order to fit the data. More parameters allow more flexibility in what can be expressed but also increase the change of over-fitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The type of model also restricts what functions can be learned. For example, linear models can only learn linear functions (really).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple answer to your question is yes. For example, a SVM with little parameters might fit data well while a ANN with many parameters might over-fit on the same data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A fair comparison however compares models with the same number of parameters. But the answer remains yes. One model might be better suited than another to fit the intrinsic structure of the data. For example, on the same data, a decision tree with 100 nodes might over-fit, a linear model might under-fit and a ANN might work perfectly. It all depends on the underlying structure that you want to model. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2016-08-08T23:21:22.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13312" PostTypeId="2" ParentId="13214" CreationDate="2016-08-09T00:08:44.577" Score="2" Body="&lt;p&gt;If anyone else comes looking, this was my solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# find last column&#xA;last_column = df.shape[1]-1&#xA;&#xA;# grab the previous 11 columns (also works if there aren't that many in the df)&#xA;wanted = df.iloc[:, (last_column-11):last_column]&#xA;&#xA;# calculate the rounded moving annual total &#xA;mat_calc = round(wanted.sum(axis=1)/len(wanted.columns), 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Probably not the most pandastic solution, but it works well.&lt;/p&gt;&#xA;" OwnerUserId="23126" LastActivityDate="2016-08-09T00:08:44.577" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13314" PostTypeId="1" CreationDate="2016-08-09T04:57:46.100" Score="0" ViewCount="5127" Body="&lt;p&gt;I've created my own neural network in python. I am tracking the performance of each run. What are some things I can investigate when my performance results are inconsistent? When I compare my network results against my expect results (during cross-validation) I can get everything from 0.46 to 0.51 all the way up to 0.56 and various numbers in between. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a sign of bad data? Not enough training epochs? Need for a different learning rate?&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2021-02-08T17:16:15.377" Title="Causes of inconsistent results with neural network?" Tags="|machine-learning|neural-network|cross-validation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13315" PostTypeId="1" AcceptedAnswerId="13317" CreationDate="2016-08-09T05:45:59.030" Score="2" ViewCount="148" Body="&lt;p&gt;While tuning the SVM classification model in Matlab, I came across the rng function in matlab in which seed (stabilizes the random shuffling of the data in the algorithm) is changed. When the function called is rng(1) then I am getting one accuracy value (99%). When it is changed to rng(2) then I am getting another value (57%). So there is a huge change in accuracy as visible. What does this mean? Am I training it wrong? &#xA;&lt;br&gt;&lt;br&gt;&#xA;The train and test set correct rate (in %) that I am getting with different runs without changing rng are(train,test)&lt;br&gt;&#xA;(96, 82.8)&lt;br&gt;&#xA;(94.6, 95.3)&lt;br&gt;&#xA;(96, 85.9)&lt;br&gt;&#xA;(96, 90)&lt;br&gt;&#xA;(95, 95)&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2016-08-09T07:08:24.310" LastActivityDate="2016-08-09T07:08:24.310" Title="Different accuracy for different rng values" Tags="|classification|svm|supervised-learning|matlab|accuracy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13316" PostTypeId="2" ParentId="13286" CreationDate="2016-08-09T05:53:49.657" Score="1" Body="&lt;p&gt;I see that you have set a Random Number Generator(rng) seed in the following line in your code: &lt;code&gt;rng(1)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this splits the data in the same way no matter whichever run. So, that is the reason why you are getting the same error values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try removing the line. Then, the data shall be slit randomly (as there is no seed now). You shall get different error then, depending on how the split is done (which is, randomly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/222279/should-random-forests-based-on-same-data-but-different-random-seeds-be-compared#comment420513_222279&quot;&gt;A model which generalizes well should be robust to the choice of seed&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-09T06:07:14.130" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13317" PostTypeId="2" ParentId="13315" CreationDate="2016-08-09T06:06:18.437" Score="0" Body="&lt;p&gt;The training errors in this dataset has a huge difference (99% vs 57%). So, maybe the one with the &lt;code&gt;rng(1)&lt;/code&gt; split has overfitted your dataset.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So there is a huge change in accuracy as visible. What does this mean?&#xA;  Am I training it wrong?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The huge change might be due to overfitting. (Also, judge the model through validation curves, and then fit a model which balances the bias-variance plot.)&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-08-09T06:06:18.437" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13319" PostTypeId="2" ParentId="19" CreationDate="2016-08-09T06:35:31.727" Score="2" Body="&lt;p&gt;&quot;Big data&quot; is literally just a lot of data. While it's more of a marketing term than anything, the implication is usually that you have so much data that you can't analyze all of the data at once because the amount of memory (RAM) it would take to hold the data in memory to process and analyze it is greater than the amount of available memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that analyses usually have to be done on random segments of data, which allows models to be built to compare against other parts of the data.&lt;/p&gt;&#xA;" OwnerUserId="23241" LastActivityDate="2016-08-09T06:35:31.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13320" PostTypeId="2" ParentId="11730" CreationDate="2016-08-09T06:45:30.587" Score="0" Body="&lt;p&gt;I think the easiest way to do that is by using some machine, some bots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently I found one called &lt;a href=&quot;http://www.octoparse.com/&quot; rel=&quot;nofollow&quot;&gt;Octoparse&lt;/a&gt; and the solution goes like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Drop an “Loop Item” into the Workflow Designer in the bot.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then select a “loop mode” &gt; Choose “text list”&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Enter the terms you want to search in the search bar.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next, click on the search box. Choose “Enter text value”.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Drag “Enter text value” into the “Loop Item” box so that the program will loop to enter the keywords, and automatically search them in the search box.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then select “Use current loop text to fill the text box”. Then click &quot;save&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next, capture the term entered. Click the search box and select “Extract value of this item”.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The search item you just captured will be added to the extracted result.&#xA;8. Click search button &gt; choose “Click an item”.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;9&quot;&gt;&#xA;&lt;li&gt;The information I want is on the detail page. So I need to create a list of item to get into that page. Click on the title &gt; Select Create a list a item &gt; Add current item to the list &gt; Continue to edit the list.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Click on the send title &gt; Add current item to the list again. Then click “loop”.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may check &lt;a href=&quot;http://www.octoparse.com/tutorial/how-to-capture-the-search-terms-entered-and-the-output/&quot; rel=&quot;nofollow&quot;&gt;the link&lt;/a&gt; to see if that's what you want.&lt;/p&gt;&#xA;" OwnerUserId="23241" LastActivityDate="2016-08-09T06:45:30.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13321" PostTypeId="2" ParentId="13284" CreationDate="2016-08-09T07:17:05.053" Score="1" Body="&lt;p&gt;I didn't find any api for facebook post either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm doing a project to keep an eye on our competitor's facebook information update. My manager asked for report of analysis of one of our competitors.I think that's sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I started to look for some &lt;a href=&quot;http://www.octoparse.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;website scraping tool&lt;/a&gt; to collect post from their posts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people said that facebook has strict anti-bot measure. So I was just trying. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that bot worked. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the result I got. ( I bulk collected what people are saying in their facebook group. )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't find any api that provides sentiment analysis for facebook posts data, I think that will be a great solution. Scraping post is a little tricky.&#xA;To locate the right place of the data you need to write XPath in the bot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YYoP1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YYoP1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23246" LastActivityDate="2016-08-09T07:17:05.053" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13323" PostTypeId="1" AcceptedAnswerId="13349" CreationDate="2016-08-09T09:28:26.697" Score="9" ViewCount="3086" Body="&lt;p&gt;I am searching for a theoretical or experimental estimation of the lower bound for the number of trees in a Random Forest classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I usually test different combinations and select the one that (using cross-validation) provides the median best result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I think that there may be a lower bound on the number of trees to use, given $m$ observations and $n$ features, to reduce the variance contribution to the error. Is there some test or reference I may check to correctly tune my classifier?&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2023-02-25T05:52:37.337" Title="Minimum number of trees for Random Forest classifier" Tags="|random-forest|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13324" PostTypeId="1" AcceptedAnswerId="13325" CreationDate="2016-08-09T10:14:41.780" Score="3" ViewCount="435" Body="&lt;p&gt;What would you guys identify as the most important areas of machine learning and data science in the future? What kind of business do you see companies focused on ML doing in the future?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give some background to the question, I'm asking this as someone hugely interested in ML at a conceptual level but with no real experience on it save for some online courses. My intention is to learn the necessary math, coding, etc. and as I'm very entrepreneurial person, eventually found my own ML company. I've found that the field is vast and thus I feel one of the biggest challenges is to select a specific area (I guess method would be a better word here) to focus on and build expertise.&lt;/p&gt;&#xA;" OwnerUserId="23250" LastActivityDate="2016-08-09T10:23:58.533" Title="What are the 'hottest' future areas of Machine Learning and Data Science?" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ClosedDate="2016-08-09T15:09:02.753" ContentLicense="CC BY-SA 3.0" />
  <row Id="13325" PostTypeId="2" ParentId="13324" CreationDate="2016-08-09T10:23:58.533" Score="1" Body="&lt;p&gt;There is a brilliant answer by Yann LeCunn recently on a Quora session. &#xA;&lt;a href=&quot;https://www.quora.com/What-are-some-recent-and-potentially-upcoming-breakthroughs-in-deep-learning&quot; rel=&quot;nofollow&quot;&gt;What are some recent and potentially upcoming breakthroughs in deep learning?&lt;/a&gt; As he quotes, adversarial training proposed by Ian Goodfellow is one of the hottest area. Apart from that, I think Memory Networks, RNNs are widely used right now and can solve various problems up ahead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to what companies might do, I see a lot of text analytic companies. Considering the amount of consumer data we have now, there is a great need to be able to understand and analyze it. Computer Vision using ML or Deep learning is fast catching up. Using cameras for face detection, object tracking, fraud detection, tracking number plates is a huge field for companies to handle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I think &lt;a href=&quot;https://www.quora.com/What%E2%80%99s-your-advice-for-undergraduate-student-who-aspires-to-be-a-research-scientist-in-deep-learning-or-related-field-one-day&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; might help you to prepare. To catch up with the overlooked developments, refer &lt;a href=&quot;https://www.quora.com/What-are-some-exciting-but-overlooked-developments-in-ML-research&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; which might give you an insight about what can develop in the future.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-09T10:23:58.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13326" PostTypeId="2" ParentId="13244" CreationDate="2016-08-09T11:44:10.663" Score="3" Body="&lt;p&gt;I took a deeper look at this problem with some example data (that the OP provided) and some simplifications. I just predicted brick height param as opposed to the 17 example params in the question. I used the Keras library in Python to explore a few different architectures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I replicated the problem, simple CNNs were predicting identical values for all heights, and the best loss (around 0.0053 on the sample data) was when this predicted the mean very precisely for all inputs. For some reason the network could not find the visual structure that is obviously in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some experimentation, I had a network that functioned as intended. It was surprisingly deep for such a simple problem. My best guess at the reason why is that the combined overlap of all the convolutions needs to be of the same size order as the features being detected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some important details:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Check that the order of the dimensions of the images when read with the image library you are using match the expected structure that the CNN is working on. If the dimensions are in a different order, this will not cause any error, but it scrambles the interpretation used by the CNN and prevents it learning anything but the mean.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Avoid using pooling layers, as they obscure pixel locations of features from further layers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check that your parameters are not ambiguous. The two &quot;brick width&quot; parameters seem interchangeable for instance. That makes it much harder for the network to predict them, and it is likely to output a rough mean for the example even if you avoid the problems with architecture.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The problem is harder to find a working architecture for than I first thought. I am not entirely sure why that is the case, but am guessing it is to do with the size of the features that need to be isolated in order for the network to &quot;measure&quot; them. Eventual working architectures were quite deep e.g. 10 convolutional layers, despite the pixel-level structures being quite obvious and clear in the training data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/neilslater/40201a6c63b4462e6c6e458bab60d0b4&quot; rel=&quot;nofollow&quot;&gt;Here is a gist of my solution in Python/Keras.&lt;/a&gt; Summary of architecture in the gist: 4 times Conv2d layers with 8 feature maps and 5x5 kernel. Dropout 25%. 4 more times Conv2d layers with 16 feature maps and 5x5 kernel. Dropout 50%. Fully connected layer with 256 outputs. Dropout 50%. Fully connected later with 128 outputs. Dropout 50%. Single output neuron. All layers use ReLU activation, except output which is linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example re-sizes images to 50x50 for speed. However, using 100x100 images works fine. I added a couple of convolutional layers to it for that (one each for the 8 features and 16 features section), and get a reasonable loss of ~0.00025 on a validation set. It looks possible to get a lower loss still with more training examples, more epochs and perhaps more feature maps. Many of the predictions were very good, close to the actual value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, the worst errors were for short bricks predicting close to a 2:1 or 3:2 ratio with the ground truth, implying that the network had found correct structure but was being confused by the periodic nature of the bricks. I think this is something that could be resolved with more training examples (and perhaps more features in higher layers) and it should be possible to get much lower loss metric on brick height.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that the corresponding tiny-cnn structure looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(100, 100, 5, 3, 8)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(96, 96, 5, 8, 8)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(92, 92, 5, 8, 8)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(88, 88, 5, 8, 8)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(84, 84, 5, 8, 8)&#xA;    &amp;lt;&amp;lt; dropout(80 * 80 * 8, 0.25)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(80, 80, 5, 8, 16)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(76, 76, 5, 16, 16)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(72, 72, 5, 16, 16)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(68, 68, 5, 16, 16)&#xA;    &amp;lt;&amp;lt; convolutional_layer&amp;lt;relu&amp;gt;(64, 64, 5, 16, 16)&#xA;    &amp;lt;&amp;lt; dropout(60 * 60 * 16, 0.5)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;relu&amp;gt;(57600, 256)&#xA;    &amp;lt;&amp;lt; dropout(256, 0.5)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;relu&amp;gt;(256, 128)&#xA;    &amp;lt;&amp;lt; dropout(128, 0.5)&#xA;    &amp;lt;&amp;lt; fully_connected_layer&amp;lt;identity&amp;gt;(128, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-08-09T14:42:43.787" LastActivityDate="2016-08-09T14:42:43.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13327" PostTypeId="1" AcceptedAnswerId="13332" CreationDate="2016-08-09T12:57:47.013" Score="7" ViewCount="4703" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/pdf/1607.01759v2.pdf&quot; rel=&quot;noreferrer&quot;&gt;Bag of Tricks for Efficient Text Classification&lt;/a&gt; paper which is popular right now, he calculates prec@1 for the datasets in the experimentation segment. What does that mean?&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-10T16:29:09.290" Title="What does Prec@1 in fastText mean?" Tags="|machine-learning|nlp|ab-test|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13329" PostTypeId="2" ParentId="13248" CreationDate="2016-08-09T14:41:33.153" Score="0" Body="&lt;p&gt;I would train multiple machine learning models based on information that is available. First of all I would standardize the data per product, so that the average is 0. That way you can better compare different price categories. I would add the mean price known to your set however, standardized over your whole training data, because I can imagine policies being different for different price groups. Now you can train different models for different products. The more features you use, the less data you have so this is a trade-off that you have to test. Here is an example of what I'm talking about (non-standardized):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  Price1 Price2 Price3 Feat1 Feat2&#xA;1   1      2      NA     3     8&#xA;2   4      NA     4      6     9&#xA;3   2      NA     3      5     3&#xA;4   NA     2      NA     3     4&#xA;5   5      1      NA     4     5&#xA;6   NA     3      5      2     3&#xA;7   8      5      7      4     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now to impute the missing Price1 values you could use:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Price1 = F(Price2, Price3, Feat1, Feat2) using only 1 row (ID: 7)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Price1 = F(Price2, Feat1, Feat2) using 3 rows (IDs: 1, 5, 7)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Price1 = F(Feat1, Feat2) using 5 rows (IDs: 1, 2, 3, 5, 7)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this very short example the third one is probably better, but with more data using more features will lead to better results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compare imputation models you could per price remove a number of known entities, train on the rest and compare the output to the known price. You can use a crossvalidation scheme to get all the prices in there once if the process is fully automated. I would do this at the standardized prices so it's about the relative mistakes and not absolute.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-09T14:41:33.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13331" PostTypeId="1" CreationDate="2016-08-09T15:49:12.570" Score="3" ViewCount="317" Body="&lt;p&gt;Normally when I train a random forest to classify observations into multi-class buckets the objective is to correctly predict which bucket an observation will fall into based on historical (training) data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I want to train the random forest to classify observations on some other criteria, such as profit maximization. Is this possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Here's an example:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that we have data on Dropbox subscriptions over the past 1 year. Some Dropbox leads (potential customers) had 1 of 3 possible coupons, other leads did not have a coupon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this scenario I want to determine which (if any) coupon I should should offer to a lead to maximize net revenue, considering their likelihood of purchasing a subscription, their predicted retention (# of months they will continue their subscription) and the effect of the coupon on purchase price. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretically some leads who are likely to purchase a subscription and be retained will not need a coupon to do so. Others might produce higher net revenue from a coupon for &quot;$5 off per month&quot; or &quot;first month free&quot;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I presume the dependent variable should still be coupon type, i.e.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = No Coupon, Coupon A, Coupon B, Coupon C, Coupon D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it possible to make the random forest work in this way? You can consider this question language-agnostic, though if I have a choice I will try to do this in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that in the case of eXtreme Gradient Boosting in either R or Python I can specify a custom objective function.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-08-10T03:07:55.673" LastActivityDate="2016-08-10T03:07:55.673" Title="Random Forest where objective is not to replicate past classifications" Tags="|machine-learning|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13332" PostTypeId="2" ParentId="13327" CreationDate="2016-08-09T16:59:24.757" Score="8" Body="&lt;p&gt;Its &quot;Precision at 1&quot;, or how often the highest ranked document is relevant:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ir-ratio.blogspot.co.uk/2012/03/precision-at-1-and-reciprocal-rank.html&quot; rel=&quot;noreferrer&quot;&gt;http://ir-ratio.blogspot.co.uk/2012/03/precision-at-1-and-reciprocal-rank.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you are looking for items about monkeys. Your query engine queries documents for &quot;monkeys&quot; and ranks by relevance. If the highest ranked document is indeed about monkeys, then that's a win for your query algorithm. But if the highest ranked document is ranked 1 because it has the text &quot;Enough of your monkey business&quot; then its a loss, because that's not really about monkeys. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat over a bunch of search terms. The Precision-at-one is then the number of wins over the total number of search terms tried.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2016-08-10T16:29:09.290" LastActivityDate="2016-08-10T16:29:09.290" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13333" PostTypeId="1" CreationDate="2016-08-09T17:53:07.593" Score="3" ViewCount="279" Body="&lt;p&gt;What is a general approach or planning when it comes to selecting the number of hidden layers in a neural network? Are there datasets/industries/classification v. linear that would help in choosing the number of hidden layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mostly deal with small, financial datasets (less than 10,000 records) and want to put more planning into the number of layers that I use. What should I be taking into account?&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2016-08-09T18:06:04.913" Title="Considerations for multiple hidden layers in neural networks?" Tags="|python|r|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13334" PostTypeId="2" ParentId="13333" CreationDate="2016-08-09T18:06:04.913" Score="2" Body="&lt;p&gt;You generally won't know ahead of time how many layers to use for a particular project, so you'll have to try a variety (1-5 layers, etc). However, one key constraint that will provide an upper bound is that you don't want so many layers that your model starts to overfit. 10,000 records is a very small dataset, so I probably wouldn't try more than 3 hidden layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some problems (e.g., object recognition in images) can have more layers depending on the design of the network. For example, using convolutions on images reduces overfitting, so you could have more layers. However, convolutions probably won't work on financial data. Convolutions work best on spatial data. Lastly, the number of layers is independent of whether the output is classification vs regression. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-09T18:06:04.913" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13335" PostTypeId="1" AcceptedAnswerId="46155" CreationDate="2016-08-09T20:07:15.700" Score="9" ViewCount="5737" Body="&lt;p&gt;I have a large number of images that I need to classify for training a clustering algorithm, and I would like to do so offline (the data is proprietary). Basically, I'd like to build a desktop survey tool that enables me to rapidly place each image into one or two categories. Ideally, the tool would:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Search in a pre-specified desktop folder for an image;&lt;/li&gt;&#xA;&lt;li&gt;Display the image and a static list of categories, allowing me to click on one;&lt;/li&gt;&#xA;&lt;li&gt;Upon clicking, record the category associated with the image;&lt;/li&gt;&#xA;&lt;li&gt;Store the image filename and associated category in a dataset somewhere;&lt;/li&gt;&#xA;&lt;li&gt;Display the next untagged image in the folder and repeat the process.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there an easy way to build this kind of tool in Python, or some other pre-built utility that I could use for free offline? &lt;/p&gt;&#xA;" OwnerUserId="23262" LastActivityDate="2019-02-24T18:31:43.917" Title="Build a tool for manually classifying training data images" Tags="|machine-learning|image-classification|training|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13336" PostTypeId="2" ParentId="13331" CreationDate="2016-08-09T20:16:08.437" Score="1" Body="&lt;p&gt;Your target variable should always reflect what you're trying to optimize (maximize or reduce), so if you want to maximize revenue, you should make revenue your target variable. I would use the coupons as independent variables. When you want to see whether to offer a coupon or not for a particular customer, feed a new record into your model for every permutation of coupon / no coupon and see which has the highest prediction. There might be other/better ways of doing this, but I think this should get what you're looking for. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-08-09T20:30:26.340" LastActivityDate="2016-08-09T20:30:26.340" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13337" PostTypeId="1" AcceptedAnswerId="13344" CreationDate="2016-08-09T20:23:01.200" Score="1" ViewCount="103" Body="&lt;p&gt;I used a training data set to train both a random forest and a neural network (one hidden layer). Then I compared how both systems perform on a test data set. Interestingly, both turned out to have about the same prediction probabilities. The forest classified 86% of the data correctly into category 0 or 1, the network achieved 85%. As there is 75% of the data in category 0, I am not too happy with the result and hoped for something better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then analyzed, which data sets were classified incorrectly. It turned out that there is a large overlap: 86% percent of the data, which was classified incorrectly by the forest, was also classified wrongly by the network. I then compared the probabilities which were attributed to those test samples by the forest and the network. Also the probabilities were almost always comparable so that when the forest was sure that sample &lt;code&gt;A&lt;/code&gt; belongs to category 0, then also the network was sure that 0 is the right category. Also, if the probabilities of the forest were around 50% for both categories, also the network was &quot;unsure&quot; which category was correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is such an outcome somehow obvious? Is there an explanation? &lt;/p&gt;&#xA;" OwnerUserId="22080" LastActivityDate="2016-08-10T04:42:50.980" Title="Similar output from random forest and neural network" Tags="|neural-network|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13339" PostTypeId="2" ParentId="13335" CreationDate="2016-08-09T20:55:23.937" Score="3" Body="&lt;p&gt;I would recommend building your own database-backed Web app, since you have proprietary data and few (only two?) classes. I would create tables for the images, users, and labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;user: (id, name)&lt;br&gt;&#xA;image: (id, url)&lt;br&gt;&#xA;label: (user.id, image.id, time, class)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The label class can be an enum. If you don't want to let users rate the same image multiple times you can drop the time column, and set the first two ID columns as the compound primary key.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you've never dealt with Web applications and databases it will seem complicated, but it is easy once you get the hang of it. &lt;a href=&quot;https://blog.pythonanywhere.com/121/&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; is a tutorial. The benefit of this approach is persistence; you can turn off your computer and the start where you left off thanks to the database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simpler alternative is to collect all your data in one session using GUI components such as &lt;a href=&quot;https://github.com/ipython/ipywidgets&quot; rel=&quot;nofollow&quot;&gt;ipywidgets for jupyter&lt;/a&gt;, and writing the labels to a file. With this approach you do not get persistence.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-08-09T21:07:48.587" LastActivityDate="2016-08-09T21:07:48.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13340" PostTypeId="2" ParentId="13304" CreationDate="2016-08-09T21:04:12.297" Score="4" Body="&lt;p&gt;I 'solved' the issue with Oracle's support, I still post an answer here because some of what I encountered can be useful :  According to Oracle's support, this bug in AB testing programs was known for the last 9 month, but corrected only this week-end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my problem is supposedly solved (I will re-launch a test and see if results are now consistent), but I still have some pointers if someone encounters the same problem I did :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Make sure this is not a statistical artifact : test your hypothesis on several independant periods. I used a Fischer test because it is exact for a binomial framework, it is what you should use if your population/conversion rate is too small for a normal approximation&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Respect hypothesis testing framework : start your test after you formulated your hypothesis&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Get a very very low p-value : No one will listen to your complaint about faulty AB testing if you come up with a 3% p-value, and they would be right. A bayesian analysis probably points to a false positive. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Make sure the random numbers that are used to split the populations are actually random (and not, say a 50/50 split)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Imagine all the possibilities that your reporting might be wrong. I thought I had imagined them all, but if you read below the real reason behind the bug, you will find that faulty reporting can be hidden very far away in parts of the environment you don't control or even have a view in.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have the actual answer from Oracle that describes the bug that happened, in case someone has the exact same problem or wants to know how this kind of bug happens : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For programs triggered by event, their engine does not have the capability to associate the different open and clicks events to the correct email. In the case several emails have been sent through the same program, Responsys assumes the event concerns the earliest email that has been sent through this program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, the program could have been sent several times during the period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the period where I was not splitting my emails through two branches, the name of the campaign I used was already A  (the same than one of my branches in my testing period)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see where this is going : Every customer opening any email was marked as having read the &quot;A&quot; email, as long as they had been into the program in the 3 months before I started my AB testing, which caused a faulty reporting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So actually both branches were sent to the correct customers, and it is just a reporting problem that comes from an approximative algorithm on Responsys side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The workaround I will try now is simply to create two brand new &quot;campaign&quot; objects before starting a test rather than using a campaign that already existed.&lt;/p&gt;&#xA;" OwnerUserId="21473" LastEditorUserId="21473" LastEditDate="2016-08-10T21:55:24.097" LastActivityDate="2016-08-10T21:55:24.097" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13341" PostTypeId="1" AcceptedAnswerId="13342" CreationDate="2016-08-09T21:09:48.223" Score="4" ViewCount="1539" Body="&lt;p&gt;With typical machine learning you would usually use a training data-set to create a model of some kind, and a testing data-set to then test the newly created model. For something like linear regression after the model is created with the training data you now have an equation that you would use to predict the outcome of the set of features in the testing data. You would then take the prediction that the model returned and compare that to the actual data in the testing set. How would a validation set be used here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With nearest neighbor you would use the training data to create an n-dimensional space that has all the features of the training set. You would then use this space to classify the features in the testing data. Again you would compare these predictions to the actual value of the data. How would a validation set help here as well?&lt;/p&gt;&#xA;" OwnerUserId="23264" LastActivityDate="2016-08-09T21:21:37.563" Title="How exactly does a validation data-set work work in machine learning?" Tags="|machine-learning|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13342" PostTypeId="2" ParentId="13341" CreationDate="2016-08-09T21:21:37.563" Score="4" Body="&lt;p&gt;Machine learning models output some sort of function; for example, a decision tree is a series of comparisons that results in a leaf node, and that leaf node has some associated value. This decision tree predicts survival chance on the Titanic, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7OqZ2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7OqZ2.png&quot; alt=&quot;Titanic decision tree from Wikipedia&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Image by Steven Milborrow, from &lt;a href=&quot;https://commons.wikimedia.org/wiki/File:CART_tree_titanic_survivors.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be used exactly like the equation generated by linear regression. You feed the new data points into the model, you get predictions out, and then you compare those predictions to the actual values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The comparison you do is applying a &quot;cost function,&quot; and what cost function you use is determined by your application. For example, linear regression can be done with different cost functions; typically, people use &quot;ordinary least squares&quot; which tries to minimize mean squared error, but other possibilities exist, like minimizing the absolute value of the error. When the result is a percentage, then something like a log probability loss function is typically a better choice than mean squared difference between the actual and predicted probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even k-nearest neighbors generates a function; it splits the space into regions where the k-nearest neighbors are the same set, and then has a flat value for that region. It ends up very similar to the result generated by a decision-tree based method.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There's also a terminology point here: typically, people talk about the training set and the test set, or the training set, the validation set, and the test set. The point of the validation set is different from that of the test set; the test set is used to determine out-of-sample error (that is, how much the model has overfit and what its real world error is likely to be like) whereas the validation set is used to determine what hyperparameters minimize the expected test error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could fit ten different models on 80% of the data, determine which one has the lowest error on another 10% of the data, and then finally estimate real-world error on the last 10% of the data.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2016-08-09T21:21:37.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13343" PostTypeId="2" ParentId="13248" CreationDate="2016-08-10T01:58:00.840" Score="2" Body="&lt;p&gt;It may be usefull to put what you are doing in a formal context, that way you can look at some standard solutions to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have obsvations of a (random) variable X (price), together with some explanatory variables L, C, P (L=&quot;location&quot; = city, C=company, P=price). So your data consists of cuadruples $(C_i, L_i, P_i, X_i)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are assuming a $\mathbf{multiplicative}$ model:&lt;br&gt;&#xA;$$ E(X|C,L,P) = \lambda \space f_C \space g_L \space h_P .$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model fits nicely in the context of generalized linear models, since:&#xA;$$ \ln (E(X|C,L,P)) = \ln(\lambda) + \ln( f_C) + \ln( g_L) + \ln( h_P) .$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are fitting the model by a variation of iterative scaling. I don't know if there is a probabilistic model that corresponds to that way of fitting the models. Both the multiplicative model and iterative scaling have long traditions, so the answer might be known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, once in the context of generalized linear models, you can fit the parameters making some assumptions on the error. Since $X$ counts the number of monetary units per item, you can start by trying a Poisson model (and a Poisson model with overdisperssion). It is also worth trying other GLM's. &lt;/p&gt;&#xA;" OwnerUserId="7763" LastEditorUserId="7763" LastEditDate="2016-08-17T16:20:37.997" LastActivityDate="2016-08-17T16:20:37.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13344" PostTypeId="2" ParentId="13337" CreationDate="2016-08-10T04:42:50.980" Score="2" Body="&lt;p&gt;I think that your neural network is probably learning the same features as your Random Forest. I do not think it is obvious because it is hard to say what the neural network is learning, but in this case based on the your model it might be working similar.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-10T04:42:50.980" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13345" PostTypeId="2" ParentId="13250" CreationDate="2016-08-10T05:06:31.597" Score="0" Body="&lt;p&gt;&lt;strong&gt;Disclaimer:&lt;/strong&gt;  I do not know enough about MATLAB to answer the full question but I'll have a go at answering the rest&lt;/p&gt;&#xA;&lt;h1&gt;1 - How many features should you take?&lt;/h1&gt;&#xA;&lt;p&gt;If this is a predictive problem you're trying to solve, how about you try and assess the accuracy of your predictions by using features sets of 7 and 10 respectively? You might find that there is very little difference in predictive power so taking 7 might be OK.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;NB:&lt;/strong&gt; I am quite surprised that your feature selection algo selected such a small proportion of your available features - you're right to want to dig a little deeper on what's happening under the hood..&lt;/p&gt;&#xA;&lt;h1&gt;2 - Will any ML algorithm be able to capture the characteristics of the data with just these 7 features?&lt;/h1&gt;&#xA;&lt;p&gt;If your feature selection algo is working fine then there's all the likelihood that a ML would be able to characterise the underlying structure of your dataset relatively easily.&lt;/p&gt;&#xA;&lt;p&gt;There's a number of problems out there that have a large amount of explanatory variables but are best characterised by a select few (for instance GDP is usually relatively well explained by what it was last quarter, with all other related variables having small impacts on prediction accuracy).&lt;/p&gt;&#xA;&lt;h1&gt;3 - MATLAB QUESTIONS: TBC&lt;/h1&gt;&#xA;&lt;p&gt;Sorry I can't answer all your questions - maybe someone will come by with MATLAB knowledge and we can roll our answers together. Otherwise you might be better splitting this into two questions to get more detailed answers on the MATLAB algo.&lt;/p&gt;&#xA;" OwnerUserId="21702" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-08-10T05:06:31.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13346" PostTypeId="1" AcceptedAnswerId="13353" CreationDate="2016-08-10T05:07:28.147" Score="0" ViewCount="112" Body="&lt;p&gt;I have a saved canvas I created in Orange 2.7. I just installed Orange 3 and opening the old canvas gives multiple &quot;UnknownWidgetDefinition&quot; errors. I gather that the naming/location of some widgets has changed between versions. Is there a way to import the old canvas?&lt;/p&gt;&#xA;" OwnerUserId="23270" LastActivityDate="2016-08-10T12:10:49.407" Title="Import Orange 2.7 canvas in Orange 3" Tags="|orange|" AnswerCount="1" CommentCount="0" ClosedDate="2016-08-11T18:56:43.817" ContentLicense="CC BY-SA 3.0" />
  <row Id="13347" PostTypeId="1" AcceptedAnswerId="13356" CreationDate="2016-08-10T05:43:41.613" Score="11" ViewCount="20568" Body="&lt;p&gt;I have a DataFrame with IDF of certain words computed.&#xA;For example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(10,[0,1,2,3,4,5],[0.413734499590671,0.4244680552337798,0.4761400657781007, 1.4004620708967006,0.37876590175292424,0.48374466516332])&#xA;&#xA;&#xA;&#xA; .... and so on&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now give a query Q, I can calculate the TF-IDF of this query.&#xA;How do I calculate the cosine similarity of the query with all documents in the dataframe (there are close to million documents)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could do it manually in a map-reduce job by using the vector multiplication&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cosine Similarity (Q, document) =  Dot product(Q, dodcument) / ||Q|| * ||document||&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but surely Spark ML must natively support calculating cosine similarity of a text?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words given a search Query how do I find the closest cosines of document TF-IDF from the  DataFrame?&lt;/p&gt;&#xA;" OwnerUserId="23245" LastActivityDate="2016-08-10T15:06:45.783" Title="Calculate cosine similarity in Apache Spark" Tags="|machine-learning|nlp|apache-spark|cosine-distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13348" PostTypeId="1" CreationDate="2016-08-10T08:31:02.127" Score="1" ViewCount="73" Body="&lt;p&gt;I have a script that collects memory usage data from a device running a Linux stack. I collect samples roughly every 200ms and write the measurement to a csv file with a timestamp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the &lt;code&gt;date&lt;/code&gt; binary on the target platform only reports the time in seconds, so I have multiple samples for each distinct timestamp. Some seconds contain more data points than others because the sample rate is affected by what the device is doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to spread my data points smoothly over a millisecond scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could take a naive approach and just take each second in the data as a separate series and average across it based on the number of data points there. However, this would not really smooth the seriet. It might approximate it well enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to know is how to do this properly. In particular, is there an algorithm or open source library (javascript perfered)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be an optimization problem with the following constraints:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The total deltas of successive timestamps should be minimized&lt;/li&gt;&#xA;&lt;li&gt;Only the millisecond parts of the timestamps should change (they shouldn't move to another second&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23277" LastEditorUserId="1330" LastEditDate="2017-11-26T03:12:49.830" LastActivityDate="2017-11-26T03:12:49.830" Title="How do I convert a series of timestamps in seconds to milliseconds in order to distribute them smoothly" Tags="|time-series|optimization|preprocessing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13349" PostTypeId="2" ParentId="13323" CreationDate="2016-08-10T09:51:11.357" Score="7" Body="&lt;p&gt;This is not necessarily an answer to your question. Just general thoughts about cross-validating the number of decision trees within a random forest.&lt;/p&gt;&#xA;&lt;p&gt;I see a lot of people in &lt;a href=&quot;/questions/tagged/kaggle&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;kaggle&amp;#39;&quot; aria-label=&quot;show questions tagged &amp;#39;kaggle&amp;#39;&quot; rel=&quot;tag&quot; aria-labelledby=&quot;tag-kaggle-tooltip-container&quot;&gt;kaggle&lt;/a&gt; and &lt;a href=&quot;/questions/tagged/stackexchange&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;stackexchange&amp;#39;&quot; aria-label=&quot;show questions tagged &amp;#39;stackexchange&amp;#39;&quot; rel=&quot;tag&quot; aria-labelledby=&quot;tag-stackexchange-tooltip-container&quot;&gt;stackexchange&lt;/a&gt; cross-validating the number of trees in a random forest. I have also asked a couple of colleagues, and they tell me it is important to cross-validate them to avoid overfitting.&lt;/p&gt;&#xA;&lt;p&gt;This never made sense to me. Since each decision tree is trained independently, adding more decision trees should just make your ensemble more and more robust.&lt;/p&gt;&#xA;&lt;p&gt;(This is different from gradient boosting trees, which are a particular case of ada boosting, and therefore there is potential for overfitting since each decision tree is trained to weight residuals more heavily.)&lt;/p&gt;&#xA;&lt;p&gt;I did a simple experiment:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;from sklearn.datasets import load_digits&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.grid_search import GridSearchCV&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;plt.ioff()&#xA;&#xA;df = load_digits()&#xA;X = df['data']&#xA;y = df['target']&#xA;&#xA;cv = GridSearchCV(&#xA;    RandomForestClassifier(max_depth=4),&#xA;    {'n_estimators': np.linspace(10, 1000, 20, dtype=int)},&#xA;    'accuracy',&#xA;    n_jobs=-1,&#xA;    refit=False,&#xA;    cv=50,&#xA;    verbose=1)&#xA;cv.fit(X, y)&#xA;&#xA;scores = np.asarray([s[1] for s in cv.grid_scores_])&#xA;trees = np.asarray([s[0]['n_estimators'] for s in cv.grid_scores_])&#xA;o = np.argsort(trees)&#xA;scores = scores[o]&#xA;trees = trees[o]&#xA;plt.clf()&#xA;plt.plot(trees, scores)&#xA;plt.xlabel('n_estimators')&#xA;plt.ylabel('accuracy')&#xA;plt.savefig('trees.png')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IluRe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IluRe.png&quot; alt=&quot;performance&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am not saying you are committing this fallacy about thinking more trees can cause overfitting. You clearly are not since you have asked for a lower bound. This is just something that has been bugging me for a while, and I think it is important to keep it in mind.&lt;/p&gt;&#xA;&lt;p&gt;(Addendum: &lt;a href=&quot;http://statweb.stanford.edu/%7Etibs/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elements of Statistical Learning&lt;/a&gt; discusses this on page 596, and is in agreement on this with me.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;em&gt;&amp;quot;It is certainly true that increasing B [B=number of trees] does not cause the random forest sequence to overfit&amp;quot;&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The author does make the observation that &lt;em&gt;&amp;quot;this limit can overfit the data&amp;quot;&lt;/em&gt;. In other words, since &lt;strong&gt;other&lt;/strong&gt; hyperparameters may lead to overfitting, creating a robust model does not rescue you from overfitting. You have to pay attention when cross-validating your other hyperparameters.)&lt;/p&gt;&#xA;&lt;p&gt;To answer your question, adding decision trees will always be beneficial to your ensemble. It will always make it more and more robust. But, of course, it is dubious whether the marginal 0.00000001 reduction in variance is worth the computational time.&lt;/p&gt;&#xA;&lt;p&gt;Your question, therefore, as I understand, is whether you can somehow calculate or estimate the number of decision trees to reduce the error variance to below a certain threshold.&lt;/p&gt;&#xA;&lt;p&gt;I very much doubt it. We do not have clear answers for many broad questions in data mining, much less specific questions like that. As Leo Breiman (the author of random forests) wrote, &lt;a href=&quot;http://www.math.snu.ac.kr/%7Ehichoi/machinelearning/(Breiman)%20Statistical%20Modeling--The%20Two%20Cultures.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;there are two cultures in statistical modeling&lt;/a&gt;, and the random forest is the type of model that he says that has few assumptions but is also very data-specific. That is why he says we cannot resort to hypothesis testing; we have to go with brute-force cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="69263" LastEditDate="2023-02-25T05:52:37.337" LastActivityDate="2023-02-25T05:52:37.337" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13350" PostTypeId="1" AcceptedAnswerId="13351" CreationDate="2016-08-10T09:52:04.133" Score="7" ViewCount="21100" Body="&lt;p&gt;I was trying to implement a regression model in Keras, but am unable to figure out how to calculate the score of my model, i.e., how well it performed on my dataset.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.wrappers.scikit_learn import KerasRegressor&#xA;from sklearn.cross_validation import cross_val_score, KFold&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.pipeline import Pipeline&#xA;&#xA;&#xA;## Load the dataset&#xA;dataframe = pd.read_csv(&amp;quot;housing.csv&amp;quot;, delim_whitespace=True,header=None)&#xA;dataset = dataframe.values&#xA;X_train = dataset[:400,0:13]&#xA;Y_train = dataset[:400,13]&#xA;X_test = dataset[401:,0:13]&#xA;Y_test = dataset[401:,13]&#xA;&#xA;##define base model&#xA;def base_model():&#xA;     model = Sequential()&#xA;     model.add(Dense(14, input_dim=13, init='normal', activation='relu'))&#xA;     model.add(Dense(7, init='normal', activation='relu'))&#xA;     model.add(Dense(1, init='normal'))&#xA;     model.compile(loss='mean_squared_error', optimizer = 'adam')&#xA;     return model&#xA;&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;&#xA;scale = StandardScaler()&#xA;X_train = scale.fit_transform(X_train)&#xA;X_test = scale.fit_transform(X_test)&#xA;&#xA;clf = KerasRegressor(build_fn=base_model, nb_epoch=100, batch_size=5,verbose=0)&#xA;&#xA;clf.fit(X_test,Y_test)&#xA;res = clf.predict(X_test)&#xA;&#xA;## line below throws an error&#xA;clf.score(Y_test,res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Please tell me how can I get the score for my model and what mistake am I doing in the above code.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-26T00:12:14.343" LastActivityDate="2020-12-26T00:12:14.343" Title="Regression in Keras" Tags="|machine-learning|python|deep-learning|regression|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13351" PostTypeId="2" ParentId="13350" CreationDate="2016-08-10T10:38:00.850" Score="7" Body="&lt;p&gt;The syntax is not exact, you should pass the features &lt;code&gt;X_test&lt;/code&gt; and the true labels &lt;code&gt;Y_test&lt;/code&gt; to &lt;code&gt;clt.score&lt;/code&gt; (the method performs the prediction on itself, no need to do it explicitly).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score = clf.score(X_test, Y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also use other metrics available in the &lt;code&gt;metrics&lt;/code&gt; module of sklearn. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import mean_squared_error&#xA;score = mean_squared_error(Y_test, clf.predict(X_test))&#xA;&#xA;from sklearn.metrics import mean_absolute_error&#xA;score = mean_absolute_error(Y_test, clf.predict(X_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Just some other remarks on your code that are not directly related to the question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;you should not call &lt;code&gt;clf.fit&lt;/code&gt; on the test data, you should instead fit on the training data and use the test set to compute the score to check the generalization of your model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;you should fit &lt;code&gt;StandardScaler&lt;/code&gt; only on the training data and use &lt;code&gt;X_test = scale.transform(X_test)&lt;/code&gt; to apply the same transformation on the test set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23283" LastEditorUserId="836" LastEditDate="2016-08-11T06:55:32.850" LastActivityDate="2016-08-11T06:55:32.850" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13352" PostTypeId="1" CreationDate="2016-08-10T11:17:44.787" Score="2" ViewCount="330" Body="&lt;p&gt;I have to detect price and product name from given text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where should I look to if I am new to this topic ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know all product names. It can be anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Example 1&lt;/b&gt;&#xA;&lt;code&gt;&#xA;Hello Denys, thank you for purchasing beer. Enjoy!&#xA;&lt;/code&gt;&#xA;&lt;b&gt;Example 2&lt;/b&gt;&#xA;&lt;code&gt;&#xA;Hello Denys, here is your peanuts.&#xA;&lt;/code&gt;&#xA;Products are - &lt;b&gt;beer, peanuts&lt;/b&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I'm familiar with Ruby and found this library - &lt;a href=&quot;https://github.com/jekyll/classifier-reborn&quot; rel=&quot;nofollow&quot;&gt;https://github.com/jekyll/classifier-reborn&lt;/a&gt;. Is it applicable to this task ?&lt;/p&gt;&#xA;" OwnerUserId="23285" LastEditorUserId="23285" LastEditDate="2016-08-13T06:04:35.200" LastActivityDate="2016-08-13T06:04:35.200" Title="How to detect product name from the bill text?" Tags="|nlp|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13353" PostTypeId="2" ParentId="13346" CreationDate="2016-08-10T12:10:49.407" Score="0" Body="&lt;p&gt;Not only the naming has changed, also the individual widget settings and layouts are completely different. Some Orange 2 widgets are not even available in Orange 3 yet, or have been consolidated into other ones. I'm afraid Orange 2 workflows are just not compatible with Orange 3.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-08-10T12:10:49.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13354" PostTypeId="1" CreationDate="2016-08-10T14:06:12.223" Score="3" ViewCount="433" Body="&lt;p&gt;I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority feature in given row/column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to approach missing data? Why can't the classifier just ignore the missing feature, and rely on the known features? Why is imputation necessary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using scikit learn, and am trying to feed in NaN to a classification model (naive bayes, logistic regression, decision tree, random forest), to see what happens. &lt;/p&gt;&#xA;" OwnerUserId="17772" LastActivityDate="2017-02-06T18:29:19.317" Title="Missing Categorical Features - no imputation" Tags="|predictive-modeling|scikit-learn|categorical-data|missing-data|data-imputation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13355" PostTypeId="2" ParentId="13354" CreationDate="2016-08-10T14:23:33.663" Score="2" Body="&lt;p&gt;Certain models are able to deal with missing values 'naturally', like certain tree based models. Most models however are just a mathematical function which is shaped after the training data. A very easy example would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = \alpha x_1 + log(2x_2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you do if one of them is NaN? The function is undefined and no prediction can be made. By imputing the values you make a reasonable guess where this sample lies on the data manifold.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-10T14:23:33.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13356" PostTypeId="2" ParentId="13347" CreationDate="2016-08-10T15:06:45.783" Score="8" Body="&lt;p&gt;There's a related example to your problem in the Spark repo &lt;a href=&quot;https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/CosineSimilarity.scala&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.  The strategy is to represent the documents as a &lt;a href=&quot;https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.linalg.distributed.RowMatrix&quot; rel=&quot;noreferrer&quot;&gt;RowMatrix&lt;/a&gt; and then use its columnSimilarities() method.  That will get you a matrix of all the cosine similarities.  Extract the row which corresponds to your query document and sort.  That will give the indices of the most-similar documents.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your application, all of this work can be done pre-query.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-08-10T15:06:45.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13357" PostTypeId="1" CreationDate="2016-08-10T15:50:19.340" Score="1" ViewCount="127" Body="&lt;p&gt;We have a deformable mirror controlled by 40 actuators that have input voltages from -1V to 1V.&#xA;Prior to hitting the mirrors, a pulse of light enters a diffraction grating and then the light is spread over the mirror by wavelength. Each actuator applies a difference phase shift to each wavelength. In the end, the light gets reconverged and read by a CCD. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our goal is to maximize the integral of the spectrum read by the CCD with respect to the 40 variable voltages. This optimization will be done in matlab. I mostly just need to know what type of problem this is...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The program will do something like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Measure a spectrum and integrate&lt;/li&gt;&#xA;&lt;li&gt;Input the new spectrum to the optimization toolbox. &lt;/li&gt;&#xA;&lt;li&gt;Output a new set of voltages to try&lt;/li&gt;&#xA;&lt;li&gt;Go back to 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There is likely multiple local maxima and the positions will change from day to day as our setup is always changing. Any help on where to look for more information would be much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="23301" LastActivityDate="2016-11-14T21:12:36.380" Title="What type of optimization problem is this?" Tags="|optimization|matlab|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13358" PostTypeId="1" AcceptedAnswerId="13388" CreationDate="2016-08-10T17:11:59.087" Score="3" ViewCount="331" Body="&lt;p&gt;I have implemented PCA algorithm and I understood it very well but still I have some questions. My code is below and it's very simple implementation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;x = np.loadtxt('CCPP', delimiter=',')&#xA;row, column = x.shape&#xA;&#xA;# Mean normalization&#xA;for i in range(column):&#xA;    x[:,i] = (x[:,i] - x[:,i].mean()) / (x[:,i].max() - x[:,i].min())&#xA;&#xA;sigma = x.transpose().dot(x) / row&#xA;u, s, v = np.linalg.svd(sigma, 0)&#xA;z = x.dot(u[:,:3]) ## new features&#xA;&#xA;new_x = z.dot(u[:,:3].transpose()) ##reconstruction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see above my sigma variable is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;x.transpose().dot(x) / row&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's giving to me an nxn matrix (n is number of features). But sigma's formula is $$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why there is a summation symbol in formula? I mean, if I use this sigma formulation then sigma is going to be a number, not a matrix. I have to get nxn matrix, right? So is my sigma implementation correct? or am I missing something about the formula?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we are reconstructing X (at the bottom in the code), should new_x equal to my first X? I mean, I reduced dimension of the data set, then I reconstructed it, original dataset and reconstructed dataset must be the same, right? This is my second question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Third Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This one is easy. Should I use data compression for each of my dataset which has 1000, 100000 or more features? I mean, can I always use it? Is it a good choice to use it every time?&lt;/p&gt;&#xA;" OwnerUserId="20926" LastEditorUserId="23283" LastEditDate="2016-08-11T07:10:01.160" LastActivityDate="2016-08-12T09:33:49.477" Title="PCA algorithm problems - Python" Tags="|machine-learning|python|pca|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13359" PostTypeId="1" CreationDate="2016-08-10T19:50:41.730" Score="2" ViewCount="2211" Body="&lt;p&gt;Problem : There are several events (eventA, eventB,....) represented by waveforms. For each event there are several csv files (eventA1.csv, eventA2.csv,...eventAn.csv) having the points(x,y) from which the corresponding waveform can be generated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using these waveforms as training data, waveform segments on (test data) larger data should be identified i.e. the interval during which an event occurs is needed to be labelled on the test data (See image for reference).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What would be the best strategy, tools and algorithms to address this problem?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If classification model is to be generated then,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The points are just to describe the nature of the waveform i.e. the test waveform will not be on same points but with similar nature e.g. the concavity, convexity, slope etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Different events in training graphs are represented using different number of points i.e. event A can be determined by 20 points, while event B has 200 points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I create feature vectors to include all these things for classification? &lt;/li&gt;&#xA;&lt;li&gt;How would the test data be analysed, as it has large number of points on the large waveform i.e. how to segment it to generate features and give it to classification model?  &lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tMJBM.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tMJBM.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23304" LastEditorUserId="8878" LastEditDate="2016-08-11T17:33:51.050" LastActivityDate="2016-08-11T17:55:13.487" Title="Identifying Waveform Segments Using Training Waveforms" Tags="|machine-learning|classification|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13360" PostTypeId="1" AcceptedAnswerId="13361" CreationDate="2016-08-10T20:34:02.433" Score="1" ViewCount="235" Body="&lt;p&gt;From some neural net article I read that if you scale up the Neural Net architecture the differnce in different local minimas in the loss surface diminishes. Essentially all local minimas become equivalent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is the case then why do different variants of SGD (like Adagrad, ADAM etc.) works better than plain SGD? I believe reason for using these variants of SGD is to solve &quot;bad&quot; local minima problem, but if all local minimas are more or less the same then what is objective of using these variants?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-08-10T21:13:11.317" Title="Reason for better performance of variants of SGD when local minimas of Neural Nets are equivalent?" Tags="|neural-network|deep-learning|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13361" PostTypeId="2" ParentId="13360" CreationDate="2016-08-10T21:13:11.317" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I believe reason for using these variants of SGD is to solve &quot;bad&quot; local minima problem&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is not accurate. The variants are mostly about accelerating steps of gradient descent when faced with shallow or rapidly changing gradients, or with gradients that need adaptive learning rates because some parts of the network get stronger gradient signals than others. They do this by weighting or adjusting the step sizes in each dimension, based on additional knowledge to the current gradients, such as the history of previous gradients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shapes like saddle points or curving gullies, in the cost function &quot;landscape&quot; can cause difficulties for basic SGD. Take a saddle point as an example - there is no &quot;bad&quot; minima, a saddle point can be quite high up in a cost function. But the gradient values can be very low, even if they pick up again if you can take steps away from the saddle point. The trouble for SGD is that using just the gradient is likely to make updates oscillate up and down the steep parts of the saddle, and not move in the shallow direction away from the saddle point. Other difficult shapes can cause similar problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a visualisation of the difference in behaviours of some of the optimisers at a saddle point, &lt;a href=&quot;http://sebastianruder.com/content/images/2016/01/saddle_point_evaluation_optimizers.gif&quot; rel=&quot;nofollow&quot;&gt;take a look at this animation&lt;/a&gt; (I'd like to find a creative-commons variant of this and include in the answer), and &lt;a href=&quot;http://sebastianruder.com/optimizing-gradient-descent/&quot; rel=&quot;nofollow&quot;&gt;a blog that references it&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, for deep learning network, there is a problem that gradients can &quot;explode&quot; or &quot;vanish&quot; as you work back through the layers. Using ReLU activation or similar can help with that, but is not always possible (think of RNNs which need sigmoid activations inside LSTM modules). An optimiser like RMSprop deals with this by normalising gradients used by the weight update steps based on recent history. Whilst SGD could more easily get stuck and fail to update the lower layer weights (weights closer to the input) - either not updating them much at all, or taking too large steps.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-08-10T21:13:11.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13362" PostTypeId="2" ParentId="13061" CreationDate="2016-08-10T23:20:53.210" Score="66" Body="&lt;p&gt;The normal vs uniform init seem to be rather unclear in fact.&lt;/p&gt;&#xA;&lt;p&gt;If we refer solely on the &lt;a href=&quot;https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Glorot&lt;/a&gt;'s and &lt;a href=&quot;https://openaccess.thecvf.com/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;He&lt;/a&gt;'s initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly considering the type of the distribution. As such, their theorical conclusions hold for any type of distribution of the determined variance. In fact, in the Glorot paper, a uniform distribution is used whereas in the He paper it is a gaussian one that is chosen. The only &amp;quot;explaination&amp;quot; given for this choice in the He paper is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Recent deep CNNs are mostly initialized by random weights drawn from Gaussian distributions&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;with a reference to &lt;a href=&quot;https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;AlexNet paper&lt;/a&gt;. It was indeed released a little later than Glorot's initialization but however there is no justificaion in it of the use of a normal distribution.&lt;/p&gt;&#xA;&lt;p&gt;In fact, in a &lt;a href=&quot;https://github.com/fchollet/keras/issues/52&quot; rel=&quot;nofollow noreferrer&quot;&gt;discussion on Keras issues tracker&lt;/a&gt;, they also seem to be a little confused and basically it could only be a matter of preference... (i.e. hypotetically Bengio would prefer uniform distribution whereas Hinton would prefer normal ones...) One the discussion, there is a small benchmark comparing Glorot initialization using a uniform and a gaussian distribution. In the end, it seems that the uniform wins but it is not really clear.&lt;/p&gt;&#xA;&lt;p&gt;In the original &lt;a href=&quot;https://arxiv.org/pdf/1512.03385v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ResNet paper&lt;/a&gt;, it only says they used a gaussian He init for all the layers, I was not able to find where it is written that they used a uniform He init for the first layer. (maybe you could share a reference to this?)&lt;/p&gt;&#xA;&lt;p&gt;As for the use of gaussian init with Batch Normalization, well, with BN the optimization process is less sensitive to initialization thus it is just a convention I would say.&lt;/p&gt;&#xA;" OwnerUserId="23283" LastEditorUserId="154169" LastEditDate="2023-09-16T04:05:43.500" LastActivityDate="2023-09-16T04:05:43.500" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13363" PostTypeId="1" CreationDate="2016-08-11T08:11:00.057" Score="0" ViewCount="176" Body="&lt;p&gt;I have the following model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a table with events, each row is event, each event has a timestamp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to try and learn how many event happen per day of week per hour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to work with the naive bayes learner (with binning), but got bad &#xA;results, I think because of the binning, the numbers does have meaning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming there is a relation between day and hour to the amount of events, some simple formula/tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What learner should i use that can work with numbers and not nominal values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks, &lt;/p&gt;&#xA;" OwnerUserId="23278" LastEditorUserId="23278" LastEditDate="2016-08-16T11:17:52.473" LastActivityDate="2016-09-15T13:08:31.020" Title="KNIME model suggestion" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13364" PostTypeId="2" ParentId="9373" CreationDate="2016-08-11T09:28:16.413" Score="1" Body="&lt;p&gt;For your info, Pylearn2 is now deprecated. You can check it's alternatives i.e. platforms that are build on top of theano, which are: lasagne, blocks and keras.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-08-11T09:28:16.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13365" PostTypeId="2" ParentId="13363" CreationDate="2016-08-11T09:30:44.953" Score="1" Body="&lt;p&gt;You can check this whitepapers for time series prediction with KNIME.&#xA;&lt;a href=&quot;https://www.knime.org/files/knime_bigdata_energy_timeseries_whitepaper.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.knime.org/files/knime_bigdata_energy_timeseries_whitepaper.pdf&lt;/a&gt;&#xA;&lt;a href=&quot;https://www.knime.org/files/Anomaly_Detection_Time_Series_final.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.knime.org/files/Anomaly_Detection_Time_Series_final.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure though if you are trying to predict how many events will happen say in the next hour or just count how many events have happened per hour in the past. In this last case, just use a GroupBy node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if this helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- Rosaria&lt;/p&gt;&#xA;" OwnerUserId="23317" LastActivityDate="2016-08-11T09:30:44.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13366" PostTypeId="2" ParentId="670" CreationDate="2016-08-11T09:33:39.037" Score="0" Body="&lt;p&gt;Check this video on YouTube &lt;a href=&quot;https://www.youtube.com/watch?v=RHsO10q7e2Y&quot; rel=&quot;nofollow&quot;&gt;https://www.youtube.com/watch?v=RHsO10q7e2Y&lt;/a&gt;&#xA;It is about a basic model (you can still optimize it) for churn prediction.&#xA;uses all records available (churned and not churned).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it only introduces sub-sampling when evaluating performances.&#xA;You can also introduce sub-sampling in the training set, depending on the machine learning algorithm you are using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rosaria&lt;/p&gt;&#xA;" OwnerUserId="23317" LastActivityDate="2016-08-11T09:33:39.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13367" PostTypeId="2" ParentId="5750" CreationDate="2016-08-11T09:36:21.693" Score="0" Body="&lt;p&gt;You can try some form of correlation or you can try a dimensionality reduction in a prediction model. In this case what is left are the important features for the prediction problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://www.knime.org/files/knime_seventechniquesdatadimreduction.pdf&quot; rel=&quot;nofollow&quot;&gt;https://www.knime.org/files/knime_seventechniquesdatadimreduction.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- Rosaria&lt;/p&gt;&#xA;" OwnerUserId="23317" LastActivityDate="2016-08-11T09:36:21.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13368" PostTypeId="2" ParentId="5504" CreationDate="2016-08-11T09:39:56.553" Score="-1" Body="&lt;p&gt;To compare clustering results from different clustering techniques you can use the Entropy Scorer node in KNIME (www.knime.org) &#xA;THis node overlaps and compares results from different techniques. The quality measure at the end tells you how similar the clusterization is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- Rosaria&lt;/p&gt;&#xA;" OwnerUserId="23317" LastActivityDate="2016-08-11T09:39:56.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13370" PostTypeId="2" ParentId="11803" CreationDate="2016-08-11T15:54:48.163" Score="0" Body="&lt;p&gt;Take a look into &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/book.pdf&quot; rel=&quot;nofollow&quot;&gt;Mining of Massive Data Sets&lt;/a&gt; pp. 328 which eventually will lead you to SVD that is commonly used in recommender systems.&lt;/p&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-08-11T15:54:48.163" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13371" PostTypeId="2" ParentId="10634" CreationDate="2016-08-11T16:07:42.897" Score="1" Body="&lt;p&gt;Here is a very nice blog post of how to do it in &lt;code&gt;R&lt;/code&gt; from scratch: &lt;a href=&quot;http://www.parallelr.com/r-deep-neural-network-from-scratch/&quot; rel=&quot;nofollow&quot;&gt;ParallelR NNs&lt;/a&gt;. Follow some links suggested in this post; they are enlightening. &#xA;Furthermore, additional posts tackle how you can improve on computation time by using parallel computation on CPUs and GPUs.&lt;/p&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-08-11T16:07:42.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13372" PostTypeId="1" CreationDate="2016-08-11T17:43:03.470" Score="3" ViewCount="104" Body="&lt;p&gt;I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 1: I am just starting out in data science and wanted to know if there is such a thing as a library that will read through text and automatically classify words or sentences as being positive/negative/neutral. Is there such a thing that exists, perhaps in R or Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 2: From what I can tell, there isn't such a library, and I would have to do that myself by hand. It appears that the best way to analyze sentiment is to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a) put the words in a corpus&lt;/li&gt;&#xA;&lt;li&gt;b) do things like word stemming and&#xA;clean up the text&lt;/li&gt;&#xA;&lt;li&gt;c) download a dictionary of predetermined&#xA;sentiment &lt;/li&gt;&#xA;&lt;li&gt;d) run a program that vectorizes the text&lt;/li&gt;&#xA;&lt;li&gt;e) compare the text from my corpus to the dictionary&lt;/li&gt;&#xA;&lt;li&gt;f) ..... unsure what after this step.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Are these steps accurate, or am I totally off-base?&lt;/p&gt;&#xA;" OwnerUserId="23327" LastActivityDate="2020-08-04T08:39:42.040" Title="How to automatically find the sentiment?" Tags="|machine-learning|python|data-mining|nlp|nltk|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13373" PostTypeId="1" CreationDate="2016-08-11T17:48:38.837" Score="8" ViewCount="795" Body="&lt;p&gt;Lets say a Model was trained on date $dt1$ using the available labeled data, split into training and test i.e $train_{dt1}$, $test_{dt1}$. This model is then deployed in production and makes predictions on new incoming data. Some $X$ days pass, and there is bunch of labelled  data that is collected in between $dt1$ and $dt1 + X$ days, lets call it $Data_x$.  In my current approach, I take  random samples out of  $DATA_x$ (take for e.g 80/20 split) ,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&#xA;$80\%$ of $DATA_x$ = $train_x$ (new data used to fine-tune the existing model trained on $dt1$)&#xA;$20\%$ of $DATA_x$ = $test_x$ (new data added to the $test_{dt1}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This process of fine-tuning repeated as time passes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By doing this I get an ever expanding test set, as well as I prevent retraining the whole model (essentially I can throw away the old data as the model has learnt from it). The new model generated is just a fine-tuned version of the old one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some questions, regarding this approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are there any obvious drawbacks in doing this?&lt;/li&gt;&#xA;&lt;li&gt;Would the model ever need to be completely retrained (forgetting everything that was learnt before, and training the model with new train/test splits ) after some time or can the approach I described above continue indefinitely ? &lt;/li&gt;&#xA;&lt;li&gt;What should be the condition for swapping the existing deployed model with the newly fine-tuned model ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12858" LastActivityDate="2016-08-13T10:26:30.437" Title="Machine Learning models in production environment" Tags="|machine-learning|cross-validation|model-selection|data-product|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13374" PostTypeId="2" ParentId="13359" CreationDate="2016-08-11T17:55:13.487" Score="2" Body="&lt;h2&gt;Statistical Data Analysis&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I would start with the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Statistically analyse your training events and see the mean and the variance of length and width and other properties to have an overall image of what they are. &lt;/li&gt;&#xA;&lt;li&gt;For each event take the mean and the standard deviation.&lt;/li&gt;&#xA;&lt;li&gt;Walk through the test data and compute the error between test signal and learned events and plot it.&lt;/li&gt;&#xA;&lt;li&gt;At the end for each event type, you will have an error curve whit many local minimums.&lt;/li&gt;&#xA;&lt;li&gt;Choose a threshold based on standard deviation of training events and detect minimums behind that threshold as events.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Pattern Recognition&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;More advanced option is to &lt;strong&gt;learn&lt;/strong&gt; a model to classify events. At the beginning I should say the last question is theoretically wrong because you do not &lt;strong&gt;generate&lt;/strong&gt; features from test data but only &lt;strong&gt;compute&lt;/strong&gt; them. Features have been extracted (generated) from training data. I assume you mean how to feed the model with test data for classification according to its size. Well, I'd say you can segment it to chunk which may lead to losing some events because of cutting points but if test data is large enough that would not be a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that time-series are a special kind of data and there are methods specifically for them so the classic feature extraction and learning may lead you to an ocean of confusing methods. I suggest to search the term &lt;strong&gt;Time-series Mining&lt;/strong&gt; instead to see more direct solutions. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Time-Series Mining&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The very first approach I proposed was kind of this! There are many more algorithms for detecting similarity based on two time-series mostly based on Euclidean distance. For example look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Longest_common_subsequence_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;LCSS&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;DTW&lt;/a&gt;. These methods are what you are looking for and both are advanced version of my first suggested algorithm. Please note that for all of them normalizing your training and test data plays an important role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://datascience.stackexchange.com/a/6117/8878&quot;&gt;this&lt;/a&gt; answer as well however that is about unsupervised segmentation of time-series.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-08-11T17:55:13.487" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13375" PostTypeId="2" ParentId="13373" CreationDate="2016-08-11T18:00:56.937" Score="5" Body="&lt;p&gt;I think this is a good approach in general. However:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Fine-tuning your model (online learning) depends a lot on the algorithm and model how well this works. Depending on your algorithm it might be wise to retrain the whole thing&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your sample space might change overtime. If you have enough data maybe retraining every few days/weeks/months over only the last year worth of data might be better. If your old samples don't represent the current situation as well having them included might hurt your performance more than the extra samples help&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The biggest condition is if it's tested and how much downtime it involves, but in general swapping more times is better, and this can be automated&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-11T18:00:56.937" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13376" PostTypeId="1" AcceptedAnswerId="13392" CreationDate="2016-08-11T19:31:25.963" Score="1" ViewCount="11850" Body="&lt;p&gt;I was building a model for a classification problem in Keras for which I used the KerasClassifier, the wrapper scikit-learn. Below is the code for the same.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.metrics import accuracy_score,roc_auc_score&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense,Dropout&#xA;from keras.wrappers.scikit_learn import KerasClassifier&#xA;from sklearn.grid_search import GridSearchCV&#xA;&#xA;&#xA;# In[3]:&#xA;&#xA;def cleanPeople(people):&#xA;    people = people.drop(['date'],axis=1)&#xA;&#xA;    people['people_id'] = people['people_id'].apply(lambda x : x.split('_')[1])&#xA;    people['people_id'] = pd.to_numeric(people['people_id']).astype(int)&#xA;&#xA;    fields = list(people.columns)&#xA;    cat_data = fields[1:11]&#xA;    bool_data = fields[11:]&#xA;&#xA;    for data in cat_data:&#xA;        people[data] = people[data].fillna('type 0')&#xA;        people[data] = people[data].apply(lambda x: x.split(' ')[1])&#xA;        people[data] = pd.to_numeric(people[data]).astype(int)&#xA;&#xA;    for data in bool_data:&#xA;        people[data] = pd.to_numeric(people[data]).astype(int)&#xA;&#xA;    return people&#xA;&#xA;&#xA;# In[4]:&#xA;&#xA;def cleanAct(data, train=False):&#xA;    data = data.drop(['date'],axis = 1)&#xA;    if train:&#xA;        data = data.drop(['outcome'],axis=1)&#xA;&#xA;    data['people_id'] = data['people_id'].apply(lambda x : x.split('_')[1])&#xA;    data['people_id'] = pd.to_numeric(data['people_id']).astype(int)&#xA;&#xA;    data['activity_id'] = data['activity_id'].apply(lambda x: x.split('_')[1])&#xA;    data['activity_id'] = pd.to_numeric(data['activity_id']).astype(int)&#xA;&#xA;    fields = list(data.columns)&#xA;    cat_data = fields[2:13]&#xA;&#xA;    for column in cat_data:&#xA;        data[column] = data[column].fillna('type 0')&#xA;        data[column] = data[column].apply(lambda x : x.split(' ')[1])&#xA;        data[column] = pd.to_numeric(data[column]).astype(int)&#xA;&#xA;    return data    &#xA;&#xA;&#xA;# In[5]:&#xA;&#xA;people = pd.read_csv(&quot;people.csv&quot;)&#xA;people = cleanPeople(people)&#xA;&#xA;act_train = pd.read_csv(&quot;act_train.csv&quot;)&#xA;act_train_cleaned = cleanAct(act_train,train=True)&#xA;&#xA;act_test = pd.read_csv(&quot;act_test.csv&quot;)&#xA;act_test_cleaned = cleanAct(act_test)&#xA;&#xA;&#xA;# In[6]:&#xA;&#xA;train = act_train_cleaned.merge(people,on='people_id', how='left')&#xA;test = act_test_cleaned.merge(people, on='people_id', how='left')&#xA;&#xA;&#xA;# In[8]:&#xA;&#xA;output = act_train['outcome']&#xA;X_train, X_test, y_train, y_test = train_test_split(train,output, test_size=0.2, random_state =7)&#xA;input_len = len(X_train)&#xA;print(input_len)&#xA;&#xA;&#xA;# In[9]:&#xA;&#xA;def base_model(optimizer='rmsprop', init='normal', dropout_rate =0.0):&#xA;    model = Sequential()&#xA;&#xA;    model.add(Dense(100, input_dim = input_len, activation='relu', init=init))&#xA;    model.add(Dropout(dropout_rate))&#xA;    model.add(Dense(50, activation = 'relu', init = init))&#xA;    model.add(Dropout(dropout_rate))&#xA;    model.add(Dense(10, activation = 'relu', init = init))&#xA;    model.add(Dropout(dropout_rate))&#xA;    model.add(Dense(1, activation = 'sigmoid', init = init))&#xA;&#xA;    model.compile(loss = 'binary_crossentropy', optimizer = optimizer, metrics =['accuracy'])&#xA;    return model&#xA;&#xA;&#xA;# In[10]:&#xA;&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;model = KerasClassifier(build_fn = base_model)&#xA;&#xA;&#xA;# In[12]:&#xA;&#xA;#grid_parameters&#xA;optimizers = ['rmsprop', 'adam']&#xA;init = ['normal', 'uniform']&#xA;dropout_rate = [0.0, 0.2, 0.5]&#xA;epochs = [100, 150, 200]&#xA;batches = [10,20,30]&#xA;param_grid = dict(optimizer = optimizers, init=init, dropout_rate = dropout_rate, nb_epoch = epochs, batch_size=batches)&#xA;&#xA;&#xA;# In[ ]:&#xA;&#xA;validator = GridSearchCV(estimator=model, param_grid= param_grid)&#xA;validator.fit(X_train, y_train)&#xA;&#xA;print(validator.best_score_)&#xA;print(validator.best_params_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following code thrown this error when I ran it on my workstation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;../src/script.py&quot;, line 137, in &#xA;    model.fit(X_train, y_train)&#xA;  File &quot;/opt/conda/lib/python3.5/site-packages/Keras-1.0.6-py3.5.egg/keras/wrappers/scikit_learn.py&quot;, line 148, in fit&#xA;    history = self.model.fit(X, y, **fit_args)&#xA;  File &quot;/opt/conda/lib/python3.5/site-packages/Keras-1.0.6-py3.5.egg/keras/models.py&quot;, line 429, in fit&#xA;    sample_weight=sample_weight)&#xA;  File &quot;/opt/conda/lib/python3.5/site-packages/Keras-1.0.6-py3.5.egg/keras/engine/training.py&quot;, line 1036, in fit&#xA;    batch_size=batch_size)&#xA;  File &quot;/opt/conda/lib/python3.5/site-packages/Keras-1.0.6-py3.5.egg/keras/engine/training.py&quot;, line 963, in _standardize_user_data&#xA;    exception_prefix='model input')&#xA;  File &quot;/opt/conda/lib/python3.5/site-packages/Keras-1.0.6-py3.5.egg/keras/engine/training.py&quot;, line 108, in standardize_input_data&#xA;    str(array.shape))&#xA;Exception: Error when checking model input: expected dense_input_1 to have shape (None, 1757832) but got array with shape (1757832, 52)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I trained the model in scikit-learn, there was no such error. Please help!&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-08-12T13:18:32.027" Title="Error in model.fit() method in Keras" Tags="|machine-learning|python|deep-learning|scikit-learn|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13377" PostTypeId="1" AcceptedAnswerId="14827" CreationDate="2016-08-11T20:25:16.350" Score="4" ViewCount="1115" Body="&lt;p&gt;I'm trying to create radar plots for players's profiling. I got this solution from this &lt;a href=&quot;https://datascience.stackexchange.com/questions/6084/how-do-i-create-a-complex-radar-chart&quot;&gt;previous question&lt;/a&gt;, but the graphic is not correctly centered and in fact there is a label that ends outside the plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0t8uV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0t8uV.png&quot; alt=&quot;Flaw Plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns  # improves plot aesthetics&#xA;&#xA;&#xA;def _invert(x, limits):&#xA;    &quot;&quot;&quot;inverts a value x on a scale from&#xA;    limits[0] to limits[1]&quot;&quot;&quot;&#xA;    return limits[1] - (x - limits[0])&#xA;&#xA;&#xA;def _scale_data(data, ranges):&#xA;    &quot;&quot;&quot;scales data[1:] to ranges[0],&#xA;    inverts if the scale is reversed&quot;&quot;&quot;&#xA;    for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;        assert (y1 &amp;lt;= d &amp;lt;= y2) or (y2 &amp;lt;= d &amp;lt;= y1)&#xA;    x1, x2 = ranges[0]&#xA;    d = data[0]&#xA;    if x1 &amp;gt; x2:&#xA;        d = _invert(d, (x1, x2))&#xA;        x1, x2 = x2, x1&#xA;    sdata = [d]&#xA;    for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;        if y1 &amp;gt; y2:&#xA;            d = _invert(d, (y1, y2))&#xA;            y1, y2 = y2, y1&#xA;        sdata.append((d-y1) / (y2-y1) * (x2 - x1) + x1)&#xA;    return sdata&#xA;&#xA;&#xA;class ComplexRadar():&#xA;    def __init__(self, fig, variables, ranges,&#xA;                 n_ordinate_levels=6):&#xA;        import ipdb; ipdb.set_trace()&#xA;        angles = np.arange(0, 360, 360./len(variables))&#xA;        axes = [fig.add_axes([0.1, 0.1, 0.9, 0.9], polar=True,&#xA;                label=&quot;axes{}&quot;.format(i))&#xA;                for i in range(len(variables))]&#xA;        l, text = axes[0].set_thetagrids(angles,&#xA;                                         labels=variables)&#xA;        [txt.set_rotation(angle-90) for txt, angle&#xA;         in zip(text, angles)]&#xA;        for ax in axes[1:]:&#xA;            ax.patch.set_visible(False)&#xA;            ax.grid(&quot;off&quot;)&#xA;            ax.xaxis.set_visible(False)&#xA;        for i, ax in enumerate(axes):&#xA;            grid = np.linspace(*ranges[i], num=n_ordinate_levels)&#xA;            gridlabel = [&quot;{}&quot;.format(round(x, 2))&#xA;                         for x in grid]&#xA;            if ranges[i][0] &amp;gt; ranges[i][1]:&#xA;                grid = grid[::-1]  # hack to invert grid. gridlabels aren't reversed&#xA;            gridlabel[0] = &quot;&quot;  # clean up origin&#xA;            ax.set_rgrids(grid, labels=gridlabel,&#xA;                          angle=angles[i])&#xA;            # ax.spines[&quot;polar&quot;].set_visible(False)&#xA;            ax.set_ylim(*ranges[i])&#xA;        # variables for plotting&#xA;        self.angle = np.deg2rad(np.r_[angles, angles[0]])&#xA;        self.ranges = ranges&#xA;        self.ax = axes[0]&#xA;&#xA;    def plot(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;&#xA;    def fill(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;&#xA;&#xA;def data2():&#xA;    metrics = ('Shots', 'Goal Conversion %', 'Assists', 'Passing %', 'Int+Tackles', 'Disposessed')&#xA;    firmino = (3.4, 15.7, 0.3, 82.3, 2.6, 1.7)&#xA;    ranges = [(1, 5), (4.5, 22.5), (0.1, 0.4), (65, 85), (1, 4.5), (1, 3)]&#xA;    fig = plt.figure(figsize=(12, 9))&#xA;    radar = ComplexRadar(fig, metrics, ranges)&#xA;    radar.plot(firmino)&#xA;    radar.fill(firmino, alpha=0.2)&#xA;    plt.savefig('firmino.png')&#xA;&#xA;if __name__ == '__main__':&#xA;    data2()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23330" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-10-30T00:24:19.547" Title="Python Matplotlib Radar not correctly centered" Tags="|python|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13378" PostTypeId="2" ParentId="13372" CreationDate="2016-08-11T21:20:07.483" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a tutorial, that should help you get started.&lt;/p&gt;&#xA;" OwnerUserId="12858" LastEditorUserId="98307" LastEditDate="2020-08-04T07:41:29.917" LastActivityDate="2020-08-04T07:41:29.917" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13379" PostTypeId="2" ParentId="8841" CreationDate="2016-08-11T21:25:35.500" Score="2" Body="&lt;p&gt;In my experience, giving weights to observations (if the algorithm in use supports it) generally works better for highly imbalanced  classification problems. Since, your are using RandomForests I would suggest you to try that.&lt;/p&gt;&#xA;" OwnerUserId="12858" LastActivityDate="2016-08-11T21:25:35.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13380" PostTypeId="2" ParentId="9850" CreationDate="2016-08-11T22:14:23.460" Score="11" Body="&lt;p&gt;To answer your question on Cross entropy, you'll notice that both of what you have mentioned are the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-\frac{1}{n} \sum(y\_train * \log(y\_output) + (1 - y\_train) \cdot \log(1 - y\_output))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that you mentioned is simply the binary cross entropy loss where you assume that $y\_train$ is a 0/1 scalar and that $y\_output$ is again a scalar indicating the probability of the output being 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other equation you mentioned is a more generic variant of that extending to multiple classes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;-tf.reduce_sum(y_train * tf.log(y_output))&lt;/code&gt;&#xA;is the same thing as writing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-\sum_n train\_prob \cdot \log (out\_prob)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the summation is over the multiple classes and the probabilities are for each class. Clearly in the binary case it is the exact same thing as what was mentioned earlier. The $n$ term is omitted as it doesn't contribute in any way to the loss minimization as it is a constant.&lt;/p&gt;&#xA;" OwnerUserId="23332" LastActivityDate="2016-08-11T22:14:23.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13382" PostTypeId="5" CreationDate="2016-08-12T00:43:12.410" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2016-08-12T00:43:12.410" LastActivityDate="2016-08-12T00:43:12.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13383" PostTypeId="4" CreationDate="2016-08-12T00:43:12.410" Score="0" Body="The CNMeM library is a simple library to help deep learning frameworks manage CUDA memory.&#xD;&#xA;&#xD;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2016-08-13T08:40:23.467" LastActivityDate="2016-08-13T08:40:23.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13385" PostTypeId="1" CreationDate="2016-08-12T06:48:33.883" Score="1" ViewCount="1133" Body="&lt;p&gt;Currently, I have a bunch of extracted news articles. I would like to determine whether a particular news article is related to a particular company. For example, &quot;Apple stock dropped by 15%&quot; should be related to Apple company. Besides that, &quot;Tim cook involves in sexual harassment case&quot; should be also related to Apple company. Can recommend me some strategies to accomplish this?&lt;/p&gt;&#xA;" OwnerUserId="20974" LastActivityDate="2018-04-03T19:00:18.437" Title="News Classification" Tags="|machine-learning|python|classification|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13386" PostTypeId="2" ParentId="13385" CreationDate="2016-08-12T06:57:16.147" Score="1" Body="&lt;p&gt;Have a look at 20ng dataset and it's classification techniques. It is a collection of news articles which are divided into 20 classes. It is not exactly same as yours, but similar. The second line you mentioned about Tim Cook might be a difficult sentence for classification, so I'd suggest you to have a good training dataset before you start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simple beginning, you can try text cleaning and tokenizations. Refer to &lt;a href=&quot;https://datascience.stackexchange.com/questions/12695/machine-learning-identification-and-classification-based-on-string-contents-ge/12736#12736&quot;&gt;this&lt;/a&gt;, one previous answer already provided which has relevant links and further information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For classification techniques further on themes, I found that &lt;a href=&quot;http://anthology.aclweb.org/N/N16/N16-1061.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; one gives a really good performance on 20ng&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-08-12T06:57:16.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13387" PostTypeId="1" AcceptedAnswerId="13389" CreationDate="2016-08-12T06:57:56.820" Score="1" ViewCount="3849" Body="&lt;p&gt;I would like to get more understanding of deep learning. Browsing the web I find applications in speech recognition and hand-written digits. However I would be interested to get some guidance on how to apply this in the classical setting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;binary classifier&lt;/li&gt;&#xA;&lt;li&gt;numerical features (each sample is a numerical vector of $K$ entries, no 2D pixels or such).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am doing my own experiments choosing learning rates, number of hidden neurons and so on, but I would be happy to see an application by somebody more experienced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The software that I use offers weight initialziation using Restricted Boltzmann Machines (RBMs). I wonder whether this is useful in this context and whether the other special techniques that one encounters in the literature (convolutional NN) are useful here to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody share a blog post, a paper or personal experience?&lt;/p&gt;&#xA;" OwnerUserId="20884" LastActivityDate="2018-10-16T15:04:20.627" Title="Example of binary classifier with numerical features using deep learning" Tags="|classification|deep-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13388" PostTypeId="2" ParentId="13358" CreationDate="2016-08-12T08:06:16.383" Score="2" Body="&lt;p&gt;&lt;strong&gt;Regarding First Question&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above formula, if I'm not wrong, x is a matrix of elements. So what the formula wants from you, is to sum all dot products of every line with it's transpose. This will give you scalar.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.array([1, 2, 3, 4])&#xA;res = x.dot(x.transpose())&#xA;# res = 30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So my sugestion would be to change that line of code to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(row):&#xA;    sigma += x[i].transpose().dot(x[i])&#xA;sigma = sigma/row&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because you reduced the dimensionality, the x_new matrix will not be the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Third Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When to use the PCA is a thing of domain problem. The point of dimensionality reduction is, to get new data set, which will not be as hard to process, but it will loose some information. So if you are your &quot;result&quot;/&quot;time to process&quot; is good, I don't think you should use it.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditDate="2016-08-12T08:12:26.460" LastActivityDate="2016-08-12T08:12:26.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13389" PostTypeId="2" ParentId="13387" CreationDate="2016-08-12T08:53:26.853" Score="3" Body="&lt;p&gt;I used Binary classification for sentiment analysis of texts. I converted sentences into vectors by taking appropriate vectorizer and classified using OneVsRest classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On another approach, my words were converted into vectors and there, I used a CNN based approach to classify. Both when tested on my datasets were giving comparable results as of now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have vectors, there are already really good approaches available for binary classification which you can try. &lt;a href=&quot;http://arxiv.org/pdf/1509.03891v1.pdf&quot; rel=&quot;nofollow&quot;&gt;On Binary Classification with Single–Layer&#xA;Convolutional Neural Networks&lt;/a&gt; is a good read for you for classification using CNNs for starters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/&quot; rel=&quot;nofollow&quot;&gt;This&lt;/a&gt; is one of the first blogs I read to gain more knowledge about this and doesn't require much of pre-requisites to understand(I am assuming you know the basics about convolution and Neural Networks). &lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-08-12T08:53:26.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13390" PostTypeId="1" AcceptedAnswerId="13395" CreationDate="2016-08-12T09:22:36.677" Score="2" ViewCount="113" Body="&lt;p&gt;I am reading the programme outline of &lt;a href=&quot;http://www.studiegids.science.ru.nl/2015/en/science/prospectus/Datascience_master/info/44423/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; two-year MSc in Data Science and I found that it has no deep learning content (as in many other european ones). I am no expert but as far as I've seen I think that DL is going to be a heavy weight of ML algorithms for a long time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think it is a bad idea to take a strong focus on classical models (e.g. bayesian) instead of teaching DL in a Data Science/ML MSc?&lt;/p&gt;&#xA;" OwnerUserId="17484" LastActivityDate="2016-08-12T16:46:01.777" Title="Is deep learning a must in a Data Science MSc programme?" Tags="|deep-learning|education|bayesian|" AnswerCount="1" CommentCount="1" ClosedDate="2016-08-13T08:40:40.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="13391" PostTypeId="2" ParentId="13358" CreationDate="2016-08-12T09:33:49.477" Score="3" Body="&lt;h3&gt;First question: computing $\Sigma$&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Actually, you perform the same computation but using a matrix operation instead of a scalar operation. You might be mislead by your notation of the feature matrix $X$ that you write $x$ because in fact&#xA;$$X = (x_j^{(i)})_{i,j} =\Big(x^{(1)} ... x^{(i)} ... x^{(n)}\Big)^T$$&#xA;that is to say, $i$-th row of the matrix $X$ contains $x^{(i)}$ the $i$-th sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, when you want to compute the empirical covariance matrix $\Sigma$ (which is indeed a $n \times n$ matrix), you have:&#xA;$$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T = \frac{1}{n} X^T X$$&#xA;you can check that this is exactly the same computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your code, you are actually computing directly $\Sigma$ using the matrix operation (i.e. $\Sigma = \frac{1}{n} X^T X$) and this does give you a $n \times n$ matrix. So, both your implementation and the formula is correct.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Second question: reconstructing $X$&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Your new feature matrix $Z$ is a $n \times 3$ matrix according to your code. In fact, as your code does not show the original size of the feature space, we will see the two cases here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$X$ is also a $n \times 3$ matrix, then you do not perform any kind of dimension reduction and you should have $X_{new} = X$ (at least in theory, in practice you might have a very small numerical approximation error but it will basically be the same)&lt;/li&gt;&#xA;&lt;li&gt;$X$ is a $n \times d$ matrix with $d &amp;gt; 3$, then you do perform a dimension reduction and in this case you get rid of some information contained in the original data and in this case $X_{new} \neq X$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In your case, I guess that you do have $d &amp;gt; 3$ and so you're in the second situation.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Third question: when to apply PCA&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Well that depends in fact...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all PCA performs an SVD which can become very expensive if you have a lot of features. In fact, there are several way to compute PCA, your way is the closer to the mathematical theory but in practice computing an SVD directly on $X$ avoids computing $X^T X$ which is expensive and allows to retrieve the same space. But in any case, in some situation, it can be expensive and time consuming and thus not practical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As PCA sorts the new feature space according to the value of the eigenvalues, i.e. of the variance of the new directions, removing the last direction might in some cases remove some noise which can help. However, at the same time, it can throw away valuable discrimative information, this is why other methods such as LDA can be interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to summarize, the answer to the question is no, it is not a good choice to use it everytime, it depends of your problem.&lt;/p&gt;&#xA;" OwnerUserId="23283" LastActivityDate="2016-08-12T09:33:49.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13392" PostTypeId="2" ParentId="13376" CreationDate="2016-08-12T13:18:32.027" Score="1" Body="&lt;p&gt;In your &lt;code&gt;base_model&lt;/code&gt; function, the &lt;code&gt;input_dim&lt;/code&gt; parameter of the first &lt;code&gt;Dense&lt;/code&gt; layer should be equal to the number of features and not to the number of samples, i.e. you should have &lt;code&gt;input_dim=X_train.shape[1]&lt;/code&gt; instead of &lt;code&gt;input_dim=len(X_train)&lt;/code&gt; (which is equal to &lt;code&gt;X_train.shape[0]&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="23283" LastActivityDate="2016-08-12T13:18:32.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13393" PostTypeId="1" CreationDate="2016-08-12T15:35:21.833" Score="0" ViewCount="69" Body="&lt;p&gt;I have two images, let's say old and new. In the old one there are 1:M objects, in the new one 1:N. I need to label the objects in the new one based on some metric (yes, I realize object tracking is a whole complicated area, but I need some   entry point right now). So I've obtained a metric based on the distance to the objects in the previous image, sort of 1-NN. I've also obtained a different metric, also sort of 1-NN, based on pixel intensity (color). Of course, I want to use both of these features somehow. For example, based on the distance metric object 3 in the new is closest to object 5 in the old, and based on the pixel intensity metric - closer to object 9 in the old image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to combine these two? Perhaps some kernel, but I'm not sure. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorDisplayName="user380" LastEditDate="2017-05-30T14:50:20.397" LastActivityDate="2017-05-30T14:50:20.397" Title="Classification with two 1-NN classifiers" Tags="|classification|image-classification|sampling|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13394" PostTypeId="2" ParentId="6964" CreationDate="2016-08-12T16:30:04.117" Score="8" Body="&lt;p&gt;Have a look at the rnn package (full disclosure, I am the author). It implements a multilayer RNN, GRU, and LSTM directly in R, i.e. not an underlying C++ library, so you should also be able to read the code and understand what is going on.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages('rnn')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The CRAN version is quite up to date, but the &lt;a href=&quot;https://github.com/bquast/rnn&quot; rel=&quot;noreferrer&quot;&gt;GitHub version&lt;/a&gt; is bleeding edge and can be installed using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if (!require('devtools')) install.packages('devtools')&#xA;devtools::install_github('bquast/rnn')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="700" LastEditorUserId="700" LastEditDate="2016-10-18T09:25:19.447" LastActivityDate="2016-10-18T09:25:19.447" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13395" PostTypeId="2" ParentId="13390" CreationDate="2016-08-12T16:46:01.777" Score="3" Body="&lt;p&gt;No, it's not problematic. Most data scientists do not need or use deep learning. Deep learning is very popular right now, but that does not mean it's widely used. Deep learning can lead to substantial overfitting on small to medium datasets (I'm arbitrarily going to say that means less than 2 GB), which are the sizes that most people have. Deep learning is primarily used for object recognition in images, or text/speech models. If you're not doing either of these two things, you probably don't need to use DL. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-12T16:46:01.777" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13396" PostTypeId="2" ParentId="10836" CreationDate="2016-08-12T17:26:34.503" Score="53" Body="&lt;p&gt;Let's say you have time-series data with $N$ rows and $700$ columns which you want to feed to a &lt;code&gt;SimpleRNN(200, return_sequence=True)&lt;/code&gt; layer in Keras. Before you feed that to the RNN, you need to reshape the previous data to a 3D tensor. So it becomes a $N \times 700 \times 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png&quot; alt=&quot;unrolled RNN&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sup&gt;The image is taken from &lt;a href=&quot;https://colah.github.io/posts/2015-08-Understanding-LSTMs&quot; rel=&quot;noreferrer&quot;&gt;https://colah.github.io/posts/2015-08-Understanding-LSTMs&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In RNN, your columns (the &quot;700 columns&quot;) is the timesteps of RNN. Your data is processed from $t=1 \  to \ 700$. After feeding the data to the RNN, now it have 700 outputs which are $h_1$ to $h_{700}$, not $h_1$ to $h_{200}$. Remember that now the shape of your data is $N \times 700 \times 200$ which is &lt;strong&gt;samples (the rows) x timesteps (the columns) x channels&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then, when you apply a &lt;code&gt;TimeDistributedDense&lt;/code&gt;, you're applying a &lt;code&gt;Dense&lt;/code&gt; layer on each timestep, which means you're applying a &lt;code&gt;Dense&lt;/code&gt; layer on each $h_1$, $h_2$,...,$h_t$ respectively. Which means: actually you're  applying the fully-connected operation on each of its channels (the &quot;200&quot; one) respectively, from $h_1$ to $h_{700}$. The 1st &quot;$1 \times 1 \times 200$&quot; until the 700th &quot;$1 \times 1 \times 200$&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are we doing this? Because you don't want to flatten the RNN output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not flattening the RNN output? Because you want to keep each timestep values separate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why keep each timestep values separate? Because: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you're only want to interacting the values between its own timestep&lt;/li&gt;&#xA;&lt;li&gt;you don't want to have a random interaction between different timesteps and channels.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9465" LastEditorUserId="-1" LastEditDate="2017-03-10T09:42:41.930" LastActivityDate="2017-01-15T09:16:57.527" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13397" PostTypeId="1" CreationDate="2016-08-12T18:35:57.757" Score="2" ViewCount="1736" Body="&lt;p&gt;I want to extract named entities from a text but I don't know whether that comes under classification or not.if it comes under classification then how to prepare classes for recognizing entities in a text.(does the collection of few org names and person names and using them as my training dataset makes the solution?)&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="22008" LastEditDate="2016-08-13T08:40:20.873" LastActivityDate="2016-08-13T10:00:33.930" Title="does entity recognition comes under classification problem?" Tags="|classification|nlp|svm|named-entity-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13398" PostTypeId="2" ParentId="13397" CreationDate="2016-08-12T20:28:02.627" Score="2" Body="&lt;p&gt;Simply put, Named Entity Recognition (NER) is a multi-class structured prediction (classification) problem, so you have a sequence of words and you want to label each one most of the time with these labels ( start-of-a-person-name, continue-of-a-person-name, start-of-an-org-name, continue-of-an-org-name, start-of-a-location-name, continue-of-a-location-name, other). Note that these are not the only classes, classes with different granularity can also be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to solve this problem with SVM you can use StructSVM or other  variations of SVM for structure prediction. Though the common baseline for this task uses Maximum Entropy (Maxent)(log-linear) models.&lt;/p&gt;&#xA;" OwnerUserId="22008" LastActivityDate="2016-08-12T20:28:02.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13399" PostTypeId="2" ParentId="13393" CreationDate="2016-08-12T22:41:53.567" Score="1" Body="&lt;p&gt;There are multiple solutions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do linear scalarizing. E.g. If you have $d_0$ and $d_1$ then you could do:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$$D = w_1 d_1 + w_2 d_2$$&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;You could use the Pareto optimal front. The Pareto optimal front is the set of all distance pairs where one of the distances cannot be dominated without deteriorating the other one. This solution is probably not that well suited for your problem.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you have a training set of linked object you could learn a model that predicts whether or not the objects are the same based on the distances. E.g. A logistic regression with the two distances as input. You could use a kernel or other non-linear tools to increase performance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;More ideas on &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-objective_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;this wiki about multi-objective optimization&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorDisplayName="user380" LastEditDate="2017-05-30T14:50:11.253" LastActivityDate="2017-05-30T14:50:11.253" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13400" PostTypeId="2" ParentId="11122" CreationDate="2016-08-12T23:12:43.273" Score="2" Body="&lt;p&gt;You could use linear regression on the genome sequence to predict the occurrence of words in the description. More specifically:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;a href=&quot;https://stats.stackexchange.com/questions/115049/why-do-we-need-to-dummy-code-categorical-variables&quot;&gt;dummy variables&lt;/a&gt; to encode the genome sequence.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;a href=&quot;https://en.wikipedia.org/wiki/Stemming&quot; rel=&quot;nofollow noreferrer&quot;&gt;stemming&lt;/a&gt; to make different conjugations of the same word the same.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag-of-words&lt;/a&gt; representation to represent the words.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a scaling of the word counts $w_i$ like $\log(w_i+1)$ or the more advanced &lt;a href=&quot;http://www.tfidf.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since you have a lot of independent variables (maybe more than the number of records?) you should use some &lt;a href=&quot;https://en.wikipedia.org/wiki/Elastic_net_regularization&quot; rel=&quot;nofollow noreferrer&quot;&gt;regularization&lt;/a&gt; of the model. Lasso would be a good choice if you want a sparse model, use ridge regression if you want to put a zero prior on the coefficients.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This method you can use to predict which words are typical for a gene sequence and, hence, characterize the sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use the intermediate result of the linear model to see what tissue is important for the prediction. The important ones are the dummy variables that are &quot;on&quot; and have high coefficients. Because you have multiple outputs you could simplify this by only using the coefficients of the top-n likely words.&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorDisplayName="user380" LastEditDate="2017-05-30T14:50:16.093" LastActivityDate="2017-05-30T14:50:16.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13401" PostTypeId="2" ParentId="13397" CreationDate="2016-08-13T10:00:33.930" Score="1" Body="&lt;p&gt;Named entity recognition can be seen as a multi class classification problem. A large data set will be required to train a model(preferably bayesian) for recognising different named entities. You can use word embedding (like google word2vec) to for preparing your training set. Also, if you can try IBM Bluemix AlchemyAPI for named entity extraction if you want to get your job done.  &lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2016-08-13T10:00:33.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13403" PostTypeId="2" ParentId="13373" CreationDate="2016-08-13T10:26:30.437" Score="0" Body="&lt;p&gt;It mainly depends on the kind of learning your ml algorithm does. &#xA;For&#xA;Offline learning: re training the whole thing is wise as some algorithm require your full data to generate better assumption.&#xA;Online learning : Your model can be fine tuned to the recent or latest data with update in model as the data arrives.&lt;/p&gt;&#xA;" OwnerUserId="21072" LastActivityDate="2016-08-13T10:26:30.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13405" PostTypeId="1" AcceptedAnswerId="13406" CreationDate="2016-08-13T11:38:23.527" Score="9" ViewCount="16312" Body="&lt;p&gt;I have been reading a paper recently on &lt;a href=&quot;https://arxiv.org/abs/1505.00387&quot; rel=&quot;nofollow noreferrer&quot;&gt;Highway Neural Networks&lt;/a&gt; and found the following:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$y=H(x,W_H)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$H$&lt;/span&gt; is usually an &lt;strong&gt;affine transform&lt;/strong&gt; followed by a non-linear activation function, but in general it may take other forms.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;After Googling about &lt;strong&gt;affine transform&lt;/strong&gt; I can't say I fully understand what it means. Can somebody please elaborate?&lt;/p&gt;&#xA;" OwnerUserId="22012" LastEditorUserId="29169" LastEditDate="2021-11-19T22:36:37.850" LastActivityDate="2022-04-09T23:58:00.440" Title="What is affine transformation in regard to neural networks?" Tags="|machine-learning|neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13406" PostTypeId="2" ParentId="13405" CreationDate="2016-08-13T12:06:12.837" Score="8" Body="&lt;p&gt;It is a linear transformation. For example, lines that were parallel before the transformation are still parallel. Scaling, rotation, reflection etcetera. With regard to neural networks, it is usually just the input matrix multiplied by the weight matrix.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="29169" LastEditDate="2021-11-19T22:37:38.247" LastActivityDate="2021-11-19T22:37:38.247" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13407" PostTypeId="1" AcceptedAnswerId="13413" CreationDate="2016-08-13T16:07:48.723" Score="1" ViewCount="1198" Body="&lt;p&gt;I was searching for latent class logit model for conjoint analysis. i found a paper which has equations for &lt;a href=&quot;http://wak2.web.rice.edu/bio/My%20Reprints/Concomitantvariablelatentclassmodels.pdf&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have co-workers who knows how to decipher the meaning of these equations and write these algorithms in any language, from scratch. One of my co-worker told me that he understood what is written in this paper and can write the algorithm in R. My co-workers mostly have masters degree in engineering. I have done graduation in accounting and fortunately landed in this job of analytics. I am very keen to learn this art/science of deciphering equations in research paper.  Can anyone suggest which discipline is it or lets say any recommendation of online course or books that can help me learn this? I will be indebted for life.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="13100" LastEditDate="2016-08-13T17:30:40.440" LastActivityDate="2016-08-15T13:23:50.670" Title="How to understand equations in research papers?" Tags="|career|reference-request|self-study|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13409" PostTypeId="1" AcceptedAnswerId="13527" CreationDate="2016-08-13T17:18:57.950" Score="0" ViewCount="3042" Body="&lt;p&gt;I have a model which has many categorical variables. For each categorical variable there are many levels, like 50~. But not all of them have significant counts. I got these counts using the function &lt;code&gt;value_counts()&lt;/code&gt; in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A                 50 &#xA;B                 38&#xA;C                 26&#xA;D                 18&#xA;E                 10&#xA;...&#xA;T                 1&#xA;X                 1&#xA;Z                 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I change the levels with count (say) less than 5 to a new level &quot;others&quot;?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for x in data.class:&#xA;    if x.value_counts() &amp;lt;30:&#xA;        x = &quot;others&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23208" LastEditorUserId="381" LastEditDate="2016-08-13T22:35:48.043" LastActivityDate="2016-08-18T19:12:31.513" Title="How to assign a new level to many levels of a categorical variable" Tags="|python|categorical-data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13410" PostTypeId="1" AcceptedAnswerId="13449" CreationDate="2016-08-13T17:58:19.430" Score="10" ViewCount="15631" Body="&lt;p&gt;I am trying to build a model in scikit-learn. I used &lt;code&gt;RandomForestClassifier&lt;/code&gt; as my method for classification. In order to improve the score and efficiency of my model, I thought about using GridSearchCV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.metrics import accuracy_score,roc_auc_score&#xA;from sklearn.grid_search import GridSearchCV&#xA;&#xA;..................................... ## code for cleaning data&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(train,output, test_size=0.2, random_state =7)&#xA;&#xA;&#xA;# In[18]:&#xA;&#xA;clf = RandomForestClassifier(n_estimators =100)&#xA;param_grid = {'max_depth' : [None, 10,20],&#xA;              'max_features' : ['auto',None],&#xA;              'n_estimators' :[100,200,300],&#xA;              'random_state': 7}&#xA;## This line is throwing the error shown below&#xA;validator = GridSearchCV(clf, param_grid= param_grid) &#xA;vaildiator.fit(X_train,y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error being thrown by my code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError     Traceback (most recent call &#xA;&#xA;last)&#xA;&amp;lt;ipython-input-22-3711af477b0c&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      3          &quot;max_depth&quot; : [5,10,50],&#xA;      4          &quot;random_state&quot; : 7}&#xA;----&amp;gt; 5 grid = GridSearchCV(clf, param_grid=param, n_jobs=1)&#xA;      6 grid.fit(X_train,y_train)&#xA;&#xA;C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in __init__(self, estimator, param_grid, scoring, fit_params, n_jobs, iid, refit, cv, verbose, pre_dispatch, error_score)&#xA;    785             refit, cv, verbose, pre_dispatch, error_score)&#xA;    786         self.param_grid = param_grid&#xA;--&amp;gt; 787         _check_param_grid(param_grid)&#xA;    788 &#xA;    789     def fit(self, X, y=None):&#xA;&#xA;C:\Anaconda3\envs\DeepLearning\lib\site-packages\sklearn\grid_search.py in _check_param_grid(param_grid)&#xA;    326             check = [isinstance(v, k) for k in (list, tuple, np.ndarray)]&#xA;    327             if True not in check:&#xA;--&amp;gt; 328                 raise ValueError(&quot;Parameter values should be a list.&quot;)&#xA;    329 &#xA;    330             if len(v) == 0:&#xA;&#xA;ValueError: Parameter values should be a list.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please help me in figuring out the above error and why is this happening?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-26T00:12:23.143" LastActivityDate="2020-12-26T00:12:25.823" Title="Parameters in GridSearchCV in scikit-learn" Tags="|machine-learning|python|scikit-learn|random-forest|gridsearchcv|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13411" PostTypeId="1" AcceptedAnswerId="13412" CreationDate="2016-08-13T19:04:41.793" Score="0" ViewCount="70" Body="&lt;p&gt;I have a regression problem and I am in doubt about how I can calculate RMSE in my life-cycle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I deal with time-series and for every prediction, I want to look N points in the future. It is apparent how to calculate RMSE for a single iteration. My question is how to calculate RMSE for N predictions of N points to get a meaningful prediction performance metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess, I can average RMSE of all iterations though as I said I am not sure at all if this would reflect actual performance.&lt;/p&gt;&#xA;" OwnerUserId="23383" LastActivityDate="2016-08-13T20:04:24.380" Title="Cross-validation strategy" Tags="|cross-validation|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13412" PostTypeId="2" ParentId="13411" CreationDate="2016-08-13T20:04:24.380" Score="0" Body="&lt;p&gt;The natural choice would be the total squared error across the N predicted values, averaged across all examples. This is the simple extension of mean squared error from the univariate case. If you're using multivariate linear regression, this is in fact what you want to optimize in order to get the maximum likelihood estimate of the parameters as well.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-08-13T20:04:24.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13413" PostTypeId="2" ParentId="13407" CreationDate="2016-08-13T21:30:23.603" Score="1" Body="&lt;p&gt;Study multiclass logistic regression; it's similar and has many tutorials. Good books include Larry Wasserman's &lt;em&gt;All of Statistics&lt;/em&gt;, and &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow&quot;&gt;An Introduction to Statistical Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way to understand research papers is simply to read more of them. Whenever you encounter something you don't understand follow the references or look it in one of the aforementioned books. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-13T21:30:23.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13414" PostTypeId="2" ParentId="13410" CreationDate="2016-08-13T21:34:56.557" Score="6" Body="&lt;p&gt;I would say that you have to remove &lt;code&gt;random_state&lt;/code&gt; from the parameter grid. That, or put something like [7, X] which will work but that doesn't make sense I think. If you want to use fixed &lt;code&gt;random_state = 7&lt;/code&gt;, you should write it when you instantiate the estimator just as another hyperparameter (next to &lt;code&gt;n_estimators&lt;/code&gt;).&lt;/p&gt;&#xA;&lt;p&gt;I can't test it right now but I'd say that's the problem.&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="85045" LastEditDate="2020-12-26T00:12:25.823" LastActivityDate="2020-12-26T00:12:25.823" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13415" PostTypeId="1" CreationDate="2016-08-14T04:18:59.357" Score="0" ViewCount="3005" Body="&lt;p&gt;I am new to time series analysis and I am trying out some codes seen in some tutorials. I am using stock prices as the time series data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(quantmod)&#xA;aapl=getSymbols(&quot;AAPL&quot;,from=&quot;2015-01-01&quot;,auto.assign=F)&#xA;is.ts(aapl)&#xA;ts.aapl=as.ts(aapl)&#xA;is.ts(ts.aapl)&#xA;x=Cl(ts.aapl)&#xA;head(x)&#xA;plot(decompose(s))&#xA;plot(stl(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But its showing these errors&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; plot(decompose(x))&#xA;Error in decompose(x) : time series has no or less than 2 periods&#xA;&amp;gt; plot(stl(x))&#xA;Error in stl(x) : series is not periodic or has less than two periods&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is the error is due to its lack of seasonality and trend?&lt;/p&gt;&#xA;" OwnerUserId="21471" LastActivityDate="2016-08-17T20:42:36.400" Title="Error while using decompose and stl functions in R" Tags="|r|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13417" PostTypeId="1" CreationDate="2016-08-14T09:31:25.423" Score="-1" ViewCount="461" Body="&lt;p&gt;What are good films for teaching data-driven decision making? In one of my classes, I asked my students to watch Moneyball at home and we discussed it in the class. It was successful and they learned much more than they could by reading a chapter of a textbook.&#xA;Are there films with similar theme suitable for my purpose?&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-08-16T08:12:34.763" Title="What are good films for teaching data-driven decision making? (like Moneyball)" Tags="|beginner|reference-request|" AnswerCount="3" CommentCount="2" CommunityOwnedDate="2016-08-16T09:36:59.137" ContentLicense="CC BY-SA 3.0" />
  <row Id="13418" PostTypeId="2" ParentId="13417" CreationDate="2016-08-14T15:15:15.270" Score="0" Body="&lt;p&gt;You could argue that 'The big short' is data driven in the sense that a bunch of guys analyse the markets/data and use it to profit by going against preconceived ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could check out &lt;a href=&quot;http://www.qedcat.com/moviemath/&quot; rel=&quot;nofollow&quot;&gt;The Mathematical Movie Database&lt;/a&gt; for other ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Id also recommend Nate Silver's book 'The Signal and the Noise: The Art and Science of Prediction' which gives some great insight on applying statistics and data analysis to real world situations.&lt;/p&gt;&#xA;" OwnerUserId="5144" LastEditorUserId="5144" LastEditDate="2016-08-14T15:31:31.670" LastActivityDate="2016-08-14T15:31:31.670" CommentCount="0" CommunityOwnedDate="2016-08-16T09:36:59.137" ContentLicense="CC BY-SA 3.0" />
  <row Id="13419" PostTypeId="1" CreationDate="2016-08-14T18:03:15.310" Score="1" ViewCount="189" Body="&lt;p&gt;I have a training dataframe &lt;code&gt;dfTrain&lt;/code&gt; and the output of &lt;code&gt;dfTrain.head()&lt;/code&gt; is shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    C0 C1                              C2   C3 C4  C5               C6  &#xA;0   1  73                 Not in universe   0   0   0  Not in universe&#xA;1   2  58  Self-employed-not incorporated   4  34   0  Not in universe&#xA;2   3  18                 Not in universe   0   0   0      High school&#xA;3   4   9                 Not in universe   0   0   0  Not in universe&#xA;4   5  10                 Not in universe   0   0   0  Not in universe&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are total 38 features and they are both &lt;code&gt;categorical&lt;/code&gt; and &lt;code&gt;numerical&lt;/code&gt;. Ignoring &lt;code&gt;C1&lt;/code&gt; and scaling numerical features, I am trying to build a &lt;strong&gt;Logistic Regression&lt;/strong&gt; model. Since, the dataframe has &lt;code&gt;categorical&lt;/code&gt; features, I am creating another dataframe which has dummy variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = pd.get_dummies(dfTrain)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The shape of &lt;code&gt;X&lt;/code&gt; now has 160 features which is much more than that of &lt;code&gt;dfTrain&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I pass &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; (where &lt;code&gt;y&lt;/code&gt; is target variable) to &lt;strong&gt;Logistic Regression Classifier&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;modelLogistic = LogisticRegression(C=10**-2, class_weight = 'balanced')&#xA;&#xA;modelLogistic.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The reason to use &lt;code&gt;class_weight = 'balanced'&lt;/code&gt; is that there are 17 classes in &lt;code&gt;y&lt;/code&gt; and highly imbalanced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is my approach correct? Am I missing anything?&lt;/p&gt;&#xA;" OwnerUserId="23403" LastEditorUserId="28951" LastEditDate="2017-02-14T16:32:20.310" LastActivityDate="2017-04-15T17:25:58.673" Title="Python: how to handle categorial values in dataset to build models" Tags="|predictive-modeling|scikit-learn|logistic-regression|categorical-data|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13420" PostTypeId="2" ParentId="12797" CreationDate="2016-08-14T18:09:38.357" Score="0" Body="&lt;p&gt;It's always a good practice to perform data cleansing before actually building a model and applying some algorithm on it. In order to data cleansing like handling missing values, &amp;quot;pandas&amp;quot; library is highly preferred.&#xA;Here is the &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; to the &amp;quot;pandas&amp;quot; latest version, and &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/missing_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;that&lt;/a&gt; for &amp;quot;Working with missing values&amp;quot; reference in pandas.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2021-01-02T02:51:12.863" LastActivityDate="2021-01-02T02:51:12.863" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13421" PostTypeId="2" ParentId="13419" CreationDate="2016-08-14T18:24:34.680" Score="1" Body="&lt;p&gt;Yes, your approach seems to be right. The only thing that I would like to point out is that while it is desirable to convert categorical features to dummies but if you don't have enough memory then you can even consider to factorize your categorical variables. You can read about it more &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.factorize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;Also, just make sure that after converting to dummies, the number of features are reasonable compared to the number of training samples. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="15412" LastEditDate="2017-02-11T07:33:19.677" LastActivityDate="2017-02-11T07:33:19.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13422" PostTypeId="1" AcceptedAnswerId="13426" CreationDate="2016-08-14T21:01:09.887" Score="3" ViewCount="961" Body="&lt;p&gt;I am working on a binary classification problem. My data contains 100K samples from two different sources. When I perform the training and testing on data from the first source I can achieve classification accuracy up to 98% and when perform training and testing on the data from the second source, I can achieve up to 99%. The problem is when mix both of them, the classification accuracy goes down to 89%. Any idea how to perform the training to achieve high accuracy. Knowing that one of my features is related to the source&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="13727" LastEditDate="2016-08-15T21:12:38.490" LastActivityDate="2017-07-16T22:23:13.167" Title="Training data from different sources" Tags="|machine-learning|classification|data-mining|bigdata|domain-adaptation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13423" PostTypeId="2" ParentId="6964" CreationDate="2016-08-14T21:26:11.937" Score="3" Body="&lt;p&gt;You might want to take a look at &lt;a href=&quot;https://github.com/dmlc/mxnet&quot; rel=&quot;nofollow&quot;&gt;mxnet&lt;/a&gt;. It is a distributed library for deep learning. It supports C++, python, scala and R. There are many examples with R. &lt;a href=&quot;https://github.com/dmlc/mxnet/blob/master/R-package/vignettes/CharRnnModel.Rmd&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; you have an example of LSTM in R with this library.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-08-14T21:26:11.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13424" PostTypeId="2" ParentId="6420" CreationDate="2016-08-14T21:54:13.610" Score="1" Body="&lt;p&gt;I don't think theano have spark support yet, however there are at least 2 deep learning libraries that are good with that. The first one is &lt;a href=&quot;https://github.com/dmlc/mxnet&quot; rel=&quot;nofollow&quot;&gt;mxnet&lt;/a&gt;, they have support for spark, R, python and C++. Another option is &lt;a href=&quot;https://github.com/deeplearning4j/deeplearning4j&quot; rel=&quot;nofollow&quot;&gt;deeplearning4j&lt;/a&gt;, made in Java with direct access to Spark.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-08-14T21:54:13.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13426" PostTypeId="2" ParentId="13422" CreationDate="2016-08-15T06:56:40.547" Score="3" Body="&lt;p&gt;It seems that you have a &lt;a href=&quot;https://en.wikipedia.org/wiki/Domain_adaptation&quot; rel=&quot;nofollow&quot;&gt;domain adaptation&lt;/a&gt; problem. The samples from the two sources behaves differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest reading &lt;a href=&quot;http://www.umiacs.umd.edu/~hal/docs/daume07easyadapt.pdf&quot; rel=&quot;nofollow&quot;&gt;Frustratingly Easy Domain Adaptation&lt;/a&gt;. As the name hints, the solution is easy , popular (800 citation until now) and a good survey of other directions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the classifier that you run on the entire dataset was train on it. &#xA;How well does the classifiers trained on the single sources perform on the other sources?&#xA;How many of the samples belong to the first source?&#xA;Will you have an indication at production of the source of the sample?&#xA;The answer to these question might open more directions. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-08-15T06:56:40.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13427" PostTypeId="2" ParentId="6420" CreationDate="2016-08-15T09:13:38.040" Score="1" Body="&lt;p&gt;I don't think there is any support for Map-Reduce or Spark for theano but if you want to run it in a distributed setting you can refer to &lt;a href=&quot;http://arxiv.org/abs/1605.08325.&quot; rel=&quot;nofollow noreferrer&quot;&gt;this technical report&lt;/a&gt; on using theano with MPI.&lt;/p&gt;&#xA;&lt;p&gt;PS : Authors of the above mentioned report have released the code for the same. You can find it &lt;a href=&quot;https://github.com/uoguelph-mlrg/Theano-MPI&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="98307" LastEditDate="2020-08-13T13:56:54.723" LastActivityDate="2020-08-13T13:56:54.723" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13428" PostTypeId="1" AcceptedAnswerId="13430" CreationDate="2016-08-15T09:23:10.203" Score="11" ViewCount="12685" Body="&lt;p&gt;I have learned that Keras has a functionality to &quot;merge&quot; two models according to the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Merge&#xA;&#xA;left_branch = Sequential()&#xA;left_branch.add(Dense(32, input_dim=784))&#xA;&#xA;right_branch = Sequential()&#xA;right_branch.add(Dense(32, input_dim=784))&#xA;&#xA;merged = Merge([left_branch, right_branch], mode='concat')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the point in mergint NNs, in which situations is it useful? Is it a kind of ensemble modelling? What is the difference between the several &quot;modes&quot; (concat, avg, dot etc...) in the sense of performance?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-08-16T15:01:41.440" Title="What is the significance of model merging in Keras?" Tags="|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13429" PostTypeId="1" CreationDate="2016-08-15T09:23:25.137" Score="3" ViewCount="2936" Body="&lt;p&gt;I am referring to seq2seq model's example code in keras (&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py&quot; rel=&quot;nofollow&quot;&gt;https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py&lt;/a&gt;). The model is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))&#xA;&#xA;model.add(RepeatVector(DIGITS + 1))&#xA;for _ in range(LAYERS):&#xA;    model.add(RNN(HIDDEN_SIZE, return_sequences=True))&#xA;&#xA;model.add(TimeDistributed(Dense(len(chars))))&#xA;model.add(Activation('softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this model we are passing the encoded input vector from encoder's last state to each time step in the decoder. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we are not passing any other input to the decoder except the encoded input vector, but in all seq2seq models we pass output sequence also (time delayed) with the encoded input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is this a valid seq2seq model? To my surprise it works well. How does this works?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2018-02-15T09:50:08.383" Title="Principle behind seq2seq model's example in keras?" Tags="|neural-network|deep-learning|keras|sequence|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13430" PostTypeId="2" ParentId="13428" CreationDate="2016-08-15T11:26:51.303" Score="14" Body="&lt;p&gt;It is used for several reasons, basically it's used to join multiple networks together. A good example would be where you have two types of input, for example tags and an image. You could build a network that for example has:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;IMAGE -&gt; Conv -&gt; Max Pooling -&gt; Conv -&gt; Max Pooling -&gt; Dense&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TAG -&gt; Embedding -&gt; Dense layer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To combine these networks into one prediction and train them together you could merge these Dense layers before the final classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Networks where you have multiple inputs are the most 'obvious' use of them, here is a picture that combines words with images inside a RNN, the Multimodal part is where the two inputs are merged:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nKKzB.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nKKzB.png&quot; alt=&quot;Multimodal Neural Network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example is Google's Inception layer where you have different convolutions that are added back together before getting to the next layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To feed multiple inputs to Keras you can pass a list of arrays. In the word/image example you would have two lists:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_input_image = [image1, image2, image3]&#xA;x_input_word = ['Feline', 'Dog', 'TV']&#xA;y_output = [1, 0, 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can fit as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(x=[x_input_image, x_input_word], y=y_output]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-16T15:01:41.440" LastActivityDate="2016-08-16T15:01:41.440" CommentCount="17" ContentLicense="CC BY-SA 3.0" />
  <row Id="13432" PostTypeId="2" ParentId="13407" CreationDate="2016-08-15T13:23:50.670" Score="1" Body="&lt;p&gt;Take the Machine Learning class on Coursera:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/learn/machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's free without a certificate.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-08-15T13:23:50.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13433" PostTypeId="1" AcceptedAnswerId="13434" CreationDate="2016-08-15T14:53:23.173" Score="1" ViewCount="77" Body="&lt;p&gt;Say you have 12 K active customers every month in your platform in August 2016, how can you determine at what rate do you need to grow every month to achieve a certain total (say, 1.5 million) by next year, August 2017. Further, how can you generate different growth scenarios? Say, grow aggressively during the first quarter and then plateau. Are there any time series or optimization tools related to this kind of problem?&lt;/p&gt;&#xA;" OwnerUserId="23417" LastActivityDate="2016-08-15T17:09:45.967" Title="How can you determine the growth rate you need to achieve a certain customer base?" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13434" PostTypeId="2" ParentId="13433" CreationDate="2016-08-15T15:02:02.267" Score="2" Body="&lt;p&gt;A growth rate is an exponential growth, you multiply your userbase by a certain fraction, which is above 1 in case of growth, else it would be decline. This effect compounds because the amount of users is bigger while being multiplied by the same multiplier. Let's call this multiplier &lt;strong&gt;a&lt;/strong&gt;. We apply this 12 times, once for every month. We are interested in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$12000 \cdot a^{12} = 1500000$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This equals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a^{12} = 125$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can take the 12th root on both sides:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a = 1.495$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means you would need to grow approximately 50% every month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For growing faster first and then plateauing you can do similar equations, although a bit more difficult. You have to express the growth at the first level in terms of the other level because else there are two unknowns which make for infinite solutions. Let's say the first 3 months the growth &lt;strong&gt;b&lt;/strong&gt; is five times as high as the growth &lt;strong&gt;a&lt;/strong&gt; last 9 months. This means $b = 5(a-1)+1$. This leads to the following equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we have three months of growth rate &lt;strong&gt;b&lt;/strong&gt;, which means after three months our userbase is $12000\cdot b^3$ and after that we have nine months of growth &lt;strong&gt;a&lt;/strong&gt; which leads to the final user base being $12000\cdot b^3\cdot a^9$. Since $b=5(a-1)+1$ this comes down to $12000\cdot (5(a-1)+1)^3\cdot a^9$, which according to Wolfram Alpha comes down to &lt;strong&gt;a&lt;/strong&gt; being around 1.2785 which means &lt;strong&gt;b&lt;/strong&gt; is around 2.3925. This means the first three months a growth of 139% and then nine months of 27.85% growth.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-15T17:09:45.967" LastActivityDate="2016-08-15T17:09:45.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13436" PostTypeId="1" AcceptedAnswerId="13440" CreationDate="2016-08-15T16:47:59.603" Score="2" ViewCount="396" Body="&lt;p&gt;I have the problem that I got a huge source data file which is showing text for all variable values instead of numerical IDs. So for example, I would like to have the variable gender coded as 1 and 2 instead of &quot;female&quot; and &quot;male&quot; written out. And equally the same for 200 other variables of which some have up to hundreds of distinct variable values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, doing this manually is not really an option here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anybody please point me to a solution or hint within R, SPSS or Python how I can assign numerical IDs to each distinct variable value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought this would be a problem other people face more commonly as well, but I have found nothing of this kind at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any help!&lt;/p&gt;&#xA;" OwnerUserId="23422" LastActivityDate="2016-08-16T07:39:14.497" Title="Assigning numerical IDs to variable values in a data file" Tags="|python|r|spss|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13437" PostTypeId="2" ParentId="13436" CreationDate="2016-08-15T17:20:49.477" Score="3" Body="&lt;p&gt;You can use the Python sklearn preprocessing &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html#sklearn.preprocessing.LabelEncoder&quot; rel=&quot;nofollow&quot;&gt;LabelEncoder&lt;/a&gt;. Here is some example code from this page with my comments:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Make a Label Encoder instance&#xA;le = preprocessing.LabelEncoder()&#xA;&#xA;# Show it the data it has to encode, so your column&#xA;le.fit([&quot;paris&quot;, &quot;paris&quot;, &quot;tokyo&quot;, &quot;amsterdam&quot;])&#xA;&#xA;# Get a ordered list of all classes it found&#xA;list(le.classes_)&#xA;&#xA;# Transform a column/list&#xA;le.transform([&quot;tokyo&quot;, &quot;tokyo&quot;, &quot;paris&quot;]) &#xA;&#xA;# Transform encoding back to original&#xA;list(le.inverse_transform([2, 2, 1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-15T17:20:49.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13438" PostTypeId="1" CreationDate="2016-08-15T18:13:56.187" Score="7" ViewCount="10437" Body="&lt;p&gt;I have recently downloaded the small image net dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://image-net.org/small/download.php&quot; rel=&quot;noreferrer&quot;&gt;http://image-net.org/small/download.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The archives contain a lot of images, but no other files. How do I know which label the images have?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2022-06-16T20:05:44.260" Title="Where can I get labels for small ImageNet?" Tags="|dataset|image-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13440" PostTypeId="2" ParentId="13436" CreationDate="2016-08-16T04:33:44.280" Score="0" Body="&lt;p&gt;SPSS has an AUTORECODE command which will do the whole job with one command. for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AUTORECODE vr1 to vr100 /into Kvr1 to Kvr100/PRINT.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will take text variables vr1 to vr100 and recode them into new numerical variables Kvr1 to Kvr100 in which each textual category in the old variable is now automatically numbered in the new variable, with the textual category now used as a value label.&lt;br&gt;&#xA;The &lt;code&gt;PRINT&lt;/code&gt; sub-command will show you in the output window a list of all the number codes chosen for text categories in each variable.&lt;br&gt;&#xA;Please note - using the &lt;code&gt;TO&lt;/code&gt; convention (as in &quot;vr1 to vr100&quot;) only works when the variables are consecutively ordered in the file. If they are not, you have to name them separately.&lt;/p&gt;&#xA;" OwnerUserId="18022" LastEditorUserId="18022" LastEditDate="2016-08-16T06:37:16.017" LastActivityDate="2016-08-16T06:37:16.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13442" PostTypeId="2" ParentId="13417" CreationDate="2016-08-16T07:17:55.027" Score="1" Body="&lt;p&gt;So while reading an article I came across this list.I have almost watched all them and I think you should too.I hope this helps &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2015/11/7-watch-documentaries-statistics-machine-learning/&quot; rel=&quot;nofollow&quot;&gt;https://www.analyticsvidhya.com/blog/2015/11/7-watch-documentaries-statistics-machine-learning/&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="23438" LastActivityDate="2016-08-16T07:17:55.027" CommentCount="0" CommunityOwnedDate="2016-08-16T09:36:59.137" ContentLicense="CC BY-SA 3.0" />
  <row Id="13443" PostTypeId="2" ParentId="13436" CreationDate="2016-08-16T07:39:14.497" Score="2" Body="&lt;p&gt;In R you turn your categorical value into a factor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dfr$id = as.numeric(factor(dfr$mycolumn))&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-08-16T07:39:14.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13445" PostTypeId="1" AcceptedAnswerId="13472" CreationDate="2016-08-16T08:04:51.537" Score="7" ViewCount="3571" Body="&lt;p&gt;I am interested in clustering multivariate N time series of T'values' each(different lengths) using python. Each variable have many trends and values which are simultaneously numeric and nominal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample &lt;span class=&quot;math-container&quot;&gt;$T_{i}$&lt;/span&gt; in the dataset has the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TimeStamp       | Sensor0 | Sensor1| Sensor2&#xA;2015-02-05 11:30|&amp;lt;Min     | On     | off&#xA;2015-02-05 11:31|&amp;lt;Min     | on     | off &#xA;2015-02-05 11:32| Action2 | 10     | 0.0001  &#xA;2015-02-07 11:33| Action2 | 10     | 0.00012 &#xA;2015-02-07 11:34| Action2 | 10     | 0.00012 &#xA;2015-02-07 11:35| Action2 | 20     | 0.00015 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another sample &lt;span class=&quot;math-container&quot;&gt;$T_{j}$&lt;/span&gt; in the dataset has the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TimeStamp       | Sensor0 | Sensor1| Sensor2&#xA;2015-10-05 11:30| Action2 | 11     | off&#xA;2015-10-05 11:31| Action1 | 11     | off &#xA;2015-10-05 11:32| Action2 | NAN    | 0.0001  &#xA;2015-10-07 11:33| Action3 | NAN    | 0.00012 &#xA;2015-10-07 11:34| &amp;lt;Min    | 10     | 0.00012 &#xA;2015-10-07 11:35| &amp;lt;Min    | 15     | on &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the missing values (not numeric), they were not collected by the sensors so my idea was to replace them by minimum values., given that all values are strictly positive.&#xA;Otherwise, they would be considered as missing values. In which case the problem would be of finding a similiraty measure that can compare missing values (off,on..) and numeric values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if there is a similarity / distance measure already exist in the litterature to compare such multivariate timeseries, with hetergonuos lengths, and whether this kind of problem has already been formulated in the papers, books or else for R and python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your advice.&lt;/p&gt;&#xA;" OwnerUserId="23440" LastEditorUserId="29169" LastEditDate="2019-06-07T23:43:50.977" LastActivityDate="2019-06-07T23:43:50.977" Title="Similarity measure for multivariate time series with heterogeous length and content" Tags="|clustering|time-series|similarity|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="13446" PostTypeId="1" CreationDate="2016-08-16T08:09:18.083" Score="4" ViewCount="283" Body="&lt;p&gt;I study Neural Networks and I pretty much understand the logic of the structure with layers, activation functions and connections. But a fundamental question is not clear: how does one put together the actual structure from the several possible combinations? It seems me that most people just combine these quite randomly without any proper reason (or with brute force &quot;grid search&quot; in better cases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read several tutorials but nobody explained the reason of choosing &quot;sigmoind&quot; activation function over &quot;tanh&quot; for instance, not to mention which one to pick in case of different activation functions in different layers. I stress that I understand the working of these functions by themselves, but the logic of the order of them is quite a puzzle.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-08-17T03:02:08.540" Title="Guiding principle of Neural Network structure building" Tags="|neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13447" PostTypeId="2" ParentId="13417" CreationDate="2016-08-16T08:12:34.763" Score="0" Body="&lt;p&gt;Maybe you can find something about Sports Analytics in the Online Archive of the &quot;11mm - International Football Film Festival Berlin&quot;. There are documentaries, short films, feature films. I cannot recommend any, because I haven't seen any of those. Maybe it still helps and you can find something from the descriptions and trailers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.11-mm.de/index.php/de/ueber-uns/film-archiv&quot; rel=&quot;nofollow&quot;&gt;http://www.11-mm.de/index.php/de/ueber-uns/film-archiv&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the ~15-minute &lt;a href=&quot;https://www.google.com/search?name=f&amp;amp;hl=en&amp;amp;q=ted%20talks%20hans%20rosling&quot; rel=&quot;nofollow&quot;&gt;TED talks by Hans Rosling&lt;/a&gt;, on population growth, global health and economic development are very insightful. Watch the one where he swallows a sword at the end, surprising the audience.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-08-16T08:12:34.763" CommentCount="0" CommunityOwnedDate="2016-08-16T09:36:59.137" ContentLicense="CC BY-SA 3.0" />
  <row Id="13448" PostTypeId="2" ParentId="13446" CreationDate="2016-08-16T08:26:36.833" Score="1" Body="&lt;p&gt;You would use a sigmoid if you want your activations to be between zero and 1 and tanh between -1 and 1, this is important for the final layer, but for a hidden layer there is not much difference between them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an article on how to build a neural network in a more systematic way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic principle is to start with only one layer, and make it bigger and bigger until you see satisfactory results, if you don't, add another layer and start over. Oh and make sure you use batch normalization, and don't use sigmoids or tanh's, use PreLUs or ELUs:&#xA;&lt;a href=&quot;https://www.linkedin.com/pulse/keras-neural-networks-win-nvidia-titan-x-abhishek-thakur?trk=prof-post&quot; rel=&quot;nofollow&quot;&gt;https://www.linkedin.com/pulse/keras-neural-networks-win-nvidia-titan-x-abhishek-thakur?trk=prof-post&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sigmoids:&lt;/strong&gt; use for output layer which are probabilities with logloss loss function, and for gates in RNNs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tanh:&lt;/strong&gt; use for updating the state in RNNs (linear-like activations might make it unstable), or for an output layer with variables between -1 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rectifiers:&lt;/strong&gt; use for activations in hidden layers of feed-forward network. &lt;/p&gt;&#xA;" OwnerUserId="23416" LastEditorUserId="23416" LastEditDate="2016-08-16T12:59:29.253" LastActivityDate="2016-08-16T12:59:29.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13449" PostTypeId="2" ParentId="13410" CreationDate="2016-08-16T08:27:26.160" Score="13" Body="&lt;p&gt;Well the error message is quite clear. GridSearchCV accepts only lists. Therefore &lt;code&gt;'random_state': [7]}&lt;/code&gt; will solve the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However when you have only one value with this parameter, it makes more sense put it directly into the classifier as you did with &lt;code&gt;n_estimators&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-08-16T08:27:26.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13450" PostTypeId="1" AcceptedAnswerId="13451" CreationDate="2016-08-16T09:16:24.810" Score="3" ViewCount="1540" Body="&lt;p&gt;After downloading the imagenet urls (&lt;a href=&quot;http://image-net.org/download-imageurls&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;), I see that it is a single 1.1 GB text file which starts like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n00004475_6590    http://farm4.static.flickr.com/3175/2737866473_7958dc8760.jpg&#xA;n00004475_15899    http://farm4.static.flickr.com/3276/2875184020_9944005d0d.jpg&#xA;n00004475_32312    http://farm3.static.flickr.com/2531/4094333885_e8462a8338.jpg&#xA;n00004475_35466    http://farm4.static.flickr.com/3289/2809605169_8efe2b8f27.jpg&#xA;n00004475_39382    http://2.bp.blogspot.com/_SrRTF97Kbfo/SUqT9y-qTVI/AAAAAAAABmg/saRXhruwS6M/s400/bARADEI.jpg&#xA;n00004475_41022    http://fortunaweb.com.ar/wp-content/uploads/2009/10/Caroline-Atkinson-FMI.jpg&#xA;n00004475_42770    http://farm4.static.flickr.com/3488/4051378654_238ca94313.jpg&#xA;n00004475_54295    http://farm4.static.flickr.com/3368/3198142470_6eb0be5f32.jpg&#xA;n00005787_13    http://www.powercai.net/Photo/UploadPhotos/200503/20050307172201492.jpg&#xA;n00005787_32    http://www.web07.cn/uploads/Photo/c101122/12Z3Y54RZ-22027.jpg&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure &lt;code&gt;n00004475_6590&lt;/code&gt; is the class of the image. How can I look up what this means (in natural language)?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-09-15T10:08:22.007" Title="How can I look up classes of ImageNet?" Tags="|dataset|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13451" PostTypeId="2" ParentId="13450" CreationDate="2016-08-16T09:20:48.403" Score="1" Body="&lt;p&gt;The first part before the underscore can be entered in this URL:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n00004475&quot; rel=&quot;nofollow&quot;&gt;http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n00004475&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-08-16T09:20:48.403" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13452" PostTypeId="2" ParentId="11777" CreationDate="2016-08-16T09:50:29.857" Score="11" Body="&lt;p&gt;I couldn't find an URL text file for the ILSVRC2012 training set, but for complete imagenet you can download the URLs only as a text file: &lt;a href=&quot;http://image-net.org/download&quot; rel=&quot;noreferrer&quot;&gt;http://image-net.org/download&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote the following script to get a feeling for the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;&quot;&quot;&quot;Analyze the distribution of classes in ImageNet.&quot;&quot;&quot;&#xA;&#xA;classes = {}&#xA;images = 0&#xA;&#xA;with open(&quot;fall11_urls.txt&quot;) as f:&#xA;    for i, line in enumerate(f):&#xA;        label, _ = line.split(&quot;\t&quot;, 1)&#xA;        wnid, _ = label.split(&quot;_&quot;)&#xA;        if wnid in classes:&#xA;            classes[wnid] += 1&#xA;        else:&#xA;            classes[wnid] = 1&#xA;        images += 1&#xA;&#xA;# Output&#xA;print(&quot;Classes: %i&quot; % len(classes))&#xA;print(&quot;Images: %i&quot; % images)&#xA;&#xA;class_counts = [count for _, count in classes.items()]&#xA;import matplotlib.pyplot as plt&#xA;plt.hist(class_counts, bins=range(max(class_counts)))&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gave:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Classes: 21841&#xA;Images: 14197122&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8iqYO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8iqYO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classes which have less than 100 examples are pretty much useless. Lets remove them from the plot. Also increase the bin size to 25:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;&quot;&quot;&quot;Analyze the distribution of classes in ImageNet.&quot;&quot;&quot;&#xA;&#xA;classes = {}&#xA;images = 0&#xA;&#xA;with open(&quot;fall11_urls.txt&quot;) as f:&#xA;    for i, line in enumerate(f):&#xA;        label, _ = line.split(&quot;\t&quot;, 1)&#xA;        wnid, _ = label.split(&quot;_&quot;)&#xA;        if wnid in classes:&#xA;            classes[wnid] += 1&#xA;        else:&#xA;            classes[wnid] = 1&#xA;        images += 1&#xA;&#xA;# Output&#xA;print(&quot;Classes: %i&quot; % len(classes))&#xA;print(&quot;Images: %i&quot; % images)&#xA;&#xA;class_counts = [count for _, count in classes.items()]&#xA;import matplotlib.pyplot as plt&#xA;plt.title('ImageNet class distribution')&#xA;plt.xlabel('Amount of available images')&#xA;plt.ylabel('Number of classes')&#xA;min_examples = 100&#xA;bin_size = 25&#xA;plt.hist(class_counts, bins=range(min_examples, max(class_counts), bin_size))&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Jm4ft.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Jm4ft.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or with seaborn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;sns.distplot(class_counts, kde=True, rug=False);&#xA;sns.plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0lxqi.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0lxqi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Top 10&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The top 10 classes with most data are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;top10 = sorted(classes.items(), key=lambda n: n[1], reverse=True)[:10]&#xA;for class_label, count in top10:&#xA;    print(&quot;%s:\t%i&quot; % (class_label, count))&#xA;&#xA;n02094433:    3047 (Yorkshire terrier)&#xA;n02086240:    2563 (Shih-Tzu)&#xA;n01882714:    2469 (koala bear, kangaroo bear, native bear, )&#xA;n02087394:    2449 (Rhodesian ridgeback)&#xA;n02100735:    2426 (English setter)&#xA;n00483313:    2410 (singles)&#xA;n02279972:    2386 (monarch butterfly, Danaus plexippus)&#xA;n09428293:    2382 (seashore)&#xA;n02138441:    2341 (meerkat)&#xA;n02100583:    2334 (vizsla, Hungarian pointer)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using &lt;a href=&quot;http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n02094433&quot; rel=&quot;noreferrer&quot;&gt;http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n02094433&lt;/a&gt; you can look the names up.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-08-16T10:22:55.323" LastActivityDate="2016-08-16T10:22:55.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13455" PostTypeId="1" AcceptedAnswerId="13467" CreationDate="2016-08-16T12:18:44.480" Score="13" ViewCount="15697" Body="&lt;p&gt;I have built some implementations using &lt;code&gt;NetworkX&lt;/code&gt;(graph Python module) native algorithms in which I output some attributes which I use them for classification purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to scale it to a distributed environment. I have seen many approaches like &lt;code&gt;neo4j&lt;/code&gt;, &lt;code&gt;Graphx&lt;/code&gt;, &lt;code&gt;GraphLab&lt;/code&gt;. However, I am quite new to this, thus I want to ask, which of them would be easy to locally apply graph algorithms (ex. node centrality measures), preferably using Python. To be more specific, which available option is closer related to &lt;code&gt;NetworkX&lt;/code&gt; (easy installation, premade functions/algorithms, ML wise)?&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2023-05-15T08:28:16.173" Title="Large Graphs: NetworkX distributed alternative" Tags="|machine-learning|graphs|distributed|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13456" PostTypeId="2" ParentId="6395" CreationDate="2016-08-16T13:05:06.970" Score="4" Body="&lt;p&gt;It is not so much which activation function which you use which determines which loss funtion you should use, but rather what the interpretation you have of the output is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the output is supposed to be a probability, then log-loss is the way to go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the output is a generic value then mean squared error is default way to go. So for example, if your output was a grey scale pixel with grey-scale labelled by a number from 0 to 1, it might make sense to use a sigmoid activation function with a mean squared error objective function. &lt;/p&gt;&#xA;" OwnerUserId="23416" LastActivityDate="2016-08-16T13:05:06.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13457" PostTypeId="2" ParentId="12797" CreationDate="2016-08-16T13:30:20.750" Score="4" Body="&lt;p&gt;We would need more information on the prediction problem and the features to be able to give something more precise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, I am surprised no answer so far included all possible options since they aren't that many:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;get rid&lt;/strong&gt; of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much information otherwise&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;replace&lt;/strong&gt; NAs with some value like -1 --- this depends on the classifier you use; if your classifier supports categorical variables, you can create a new category for those NAs for example. In some continuous variables, sometimes there are some values that make sense (for instance, in text mining classification, if you have a &lt;code&gt;title-length&lt;/code&gt; feature but you have no title, it might make sense to replace with &lt;code&gt;title-length=0&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;fill up&lt;/strong&gt; the missing data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This last point encompasses too many things:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;4&quot;&gt;&#xA;&lt;li&gt;replace NAs with the median (this is the usual lazy approach; sklearn &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html&quot; rel=&quot;nofollow&quot;&gt;has a class&lt;/a&gt; for this)&lt;/li&gt;&#xA;&lt;li&gt;if time series, replace with an average of the previous and following values -- in pandas, this can be done using &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html&quot; rel=&quot;nofollow&quot;&gt;DataFrame.resample()&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;use the $k$ closest neighbors. build a KNN model using the other variables and then do the average of those neighbors (if you use euclidean distance, you probably should normalize first). I never seen this done, but you probably could try predicting the missing NAs using another model as well.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But all this depends very much on what you are doing. For instance, if you have performed clustering analysis and you know your data is made up of clusters, you could use the median within each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly other solutions could include things like multimodal or multiview models. These are recent techniques that can cope with missing modalities, and you can see a feature, or subset of features, as a modality. For instance, you could build a different classifier for various subsets of your features (using the complete cases in each of those subsets) and then build another classifier on top of that to merge those probabilities. I would only try these techniques if &lt;em&gt;most&lt;/em&gt; of your data is missing. There are more advanced deep learning versions of this using autoencoders.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-08-16T13:30:20.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13458" PostTypeId="2" ParentId="13446" CreationDate="2016-08-16T13:35:09.690" Score="0" Body="&lt;p&gt;You're right that most of the structure has to come from trial and error on your dataset. However, there are tips that can save you time, although that's not always the case. Rectified linear units are popular these days because they're more resistant to the vanishing gradient problem compared to sigmoid activation functions. I don't see a big difference between a sigmoid and tanh activation function though. Another helpful tip is to use convolutions for 2D (ex image) datasets. You also don't want to add more layers than the size of your dataset can handle (you might overfit). Ultimately these are all just suggestions. No one will be able to say for sure, ahead of time, what will work on your particular problem. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-08-16T16:44:53.190" LastActivityDate="2016-08-16T16:44:53.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13460" PostTypeId="2" ParentId="13446" CreationDate="2016-08-16T14:35:23.503" Score="0" Body="&lt;p&gt;Theory is still way behind practice in neural nets. But a classic &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Yann LeCun has some helpful guidance about architectural choices including activations and scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those coming by this question who might not know: tanh is just a shifted and scaled sigmoid; see &lt;a href=&quot;https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function&quot;&gt;this question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23450" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-16T14:35:23.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13461" PostTypeId="1" CreationDate="2016-08-16T14:38:24.193" Score="25" ViewCount="65733" Body="&lt;p&gt;When I request Keras to apply prediction with a fitted model to a new dataset without label like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model1.predict_classes(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it works fine. But when I try to make prediction for only one row, it fails:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model1.predict_classes(X_test[10])&#xA;&#xA;Exception: Error when checking : expected dense_input_6 to have shape (None, 784) but got array with shape (784, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wonder, why?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2020-08-05T13:58:20.647" Title="How can I get prediction for only one instance in Keras?" Tags="|neural-network|keras|" AnswerCount="9" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13462" PostTypeId="2" ParentId="13461" CreationDate="2016-08-16T15:09:48.510" Score="2" Body="&lt;p&gt;You should pass a list with just 1 example, I can't test right now but this should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model1.predict_classes([X_test[10]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-16T15:09:48.510" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13463" PostTypeId="2" ParentId="13461" CreationDate="2016-08-16T15:11:38.793" Score="11" Body="&lt;p&gt;&lt;code&gt;predict_classes&lt;/code&gt; is expecting a 2D array of shape &lt;code&gt;(num_instances, features)&lt;/code&gt;, like &lt;code&gt;X_test&lt;/code&gt; is. But indexing a single instance as in &lt;code&gt;X_test[10]&lt;/code&gt; returns a 1D array of shape &lt;code&gt;(features,)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add back the extra axis, you can use &lt;a href=&quot;http://docs.scipy.org/doc/numpy/reference/generated/numpy.expand_dims.html#numpy.expand_dims&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;np.expand_dims&lt;/code&gt;&lt;/a&gt;&lt;code&gt;(X_test[10], axis=0)&lt;/code&gt;, or &lt;code&gt;X_test[10][np.newaxis,:]&lt;/code&gt;, or don't get rid of it in the first place (e.g., by using &lt;code&gt;X_test[10:11]&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="23450" LastActivityDate="2016-08-16T15:11:38.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13464" PostTypeId="2" ParentId="13171" CreationDate="2016-08-16T15:28:45.947" Score="0" Body="&lt;p&gt;Thanks a lot, Sobach, for your solution works nicely for the stem function, and it also works for the stem.leaf.backback function (in the aplpack package). These two functions do not offer an option for showing descending values in the plot. Small numbers at the lower end, and bigger numbers at the upper end is probably the best way in this exploratory data analysis tool.&lt;/p&gt;&#xA;" OwnerUserId="22065" LastActivityDate="2016-08-16T15:28:45.947" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13465" PostTypeId="1" AcceptedAnswerId="13466" CreationDate="2016-08-16T15:38:29.150" Score="3" ViewCount="7756" Body="&lt;p&gt;I believe I have a simple if not trivial question. I have a background in statistics and I tend to use Stata and R quite a bit. I am interested in learning Python. I used it for a while now and recently came into contact with &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow&quot;&gt;scikit-learn&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to reproduce a simple example. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import tree&#xA;features = [[140, 1],[130, 1], [150, 0], [170, 0]]&#xA;labels = [0 , 0 , 1 , 1]&#xA;clf = tree.DecisionTreeClassifier()&#xA;clf = clf.fit(features, labels)&#xA;print clf.predict([[150, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the tiny script tries to predict  - by the means of a decision tree - wether a object with the properties &lt;code&gt;[150, 0]&lt;/code&gt; is likely a type &lt;code&gt;1&lt;/code&gt; or &lt;code&gt;0&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run the script and get following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;File &quot;clf_decision_tree.py&quot;, line 6&#xA;    print clf.predict([[150, 0]])&#xA;            ^&#xA;SyntaxError: invalid syntax&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I realy don't get what is wrong... Can you help me out? Best /R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I not sure if Cross Validated or Stackoverflow are better places to ask. Let me know. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="23452" LastEditorUserId="23452" LastEditDate="2016-08-16T15:45:39.293" LastActivityDate="2017-08-14T10:21:16.280" Title="First steps with Python and scikit-learn" Tags="|python|scikit-learn|decision-trees|error-handling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13466" PostTypeId="2" ParentId="13465" CreationDate="2016-08-16T16:10:00.213" Score="10" Body="&lt;p&gt;In python 3 the &lt;code&gt;print&lt;/code&gt; function &lt;strong&gt;must&lt;/strong&gt; have parenthesis, so &lt;code&gt;print(clf.predict([[150, 0]]))&lt;/code&gt; will work&lt;/p&gt;&#xA;" OwnerUserId="17484" LastActivityDate="2016-08-16T16:10:00.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13467" PostTypeId="2" ParentId="13455" CreationDate="2016-08-16T16:53:19.397" Score="9" Body="&lt;p&gt;Good , old and unsolved question! Distributed processing of large graphs as far as I know (speaking as a graph guy) has 2 different approaches, with the knowledge of Big Data frameworks or without it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://snap.stanford.edu/&quot; rel=&quot;noreferrer&quot;&gt;SNAP&lt;/a&gt; library from Jure Leskovec group at  Stanford which is originally in C++ but also has a Python API (please check if you need to use C++ API or Python does the job you want to do). Using snap you can do many things on massive networks without any special knowledge of Big Data technologies. So I would say the easiest one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Apache Graphx is wonderful only if you have experience in Scala because there is no Python thing for that. It comes with a large stack of built in algorithms including centrality measures. So the second easiest in case you know Scala.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long time ago when I looked at GraphLab it was commercial. Now I see it goes open source so maybe you know better than me but from my out-dated knowledge I remember that it does not support a wide range of algorithms and if you need an algorithm which is not there it might get complicated to implement. On the other hand it uses Python which is cool. After all please check it again as my knowledge is for 3 years ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are familiar with Big Data frameworks and working with them, &lt;a href=&quot;http://giraph.apache.org/quick_start.html&quot; rel=&quot;noreferrer&quot;&gt;Giraph&lt;/a&gt; and &lt;a href=&quot;https://github.com/dbs-leipzig/gradoop&quot; rel=&quot;noreferrer&quot;&gt;Gradoop&lt;/a&gt; are 2 great options. Both do fantastic jobs but you need to know some Big Data architecture e.g. working with a hadoop platform.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;PS&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;1) I have used simple NetworkX and multiprocessing to distributedly process DBLP network with 400,000 nodes and it worked well, so you need to know HOW BIG your graph is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) After all, I think SNAP library is a handy thing.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2016-08-16T16:53:19.397" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13468" PostTypeId="2" ParentId="12780" CreationDate="2016-08-16T17:18:09.133" Score="1" Body="&lt;p&gt;If the old variables and the new variables are highly correlated then you could do a more advanced form of imputation and make a model for each new input that predicts the new input given the old inputs. This model would probably be pretty effective good at predicting the new inputs because, as you said, there is a strong correlation among the inputs. Then you would split up all of your data across the years so that you have an equal proportion of old records and new records in your training, validation, and test sets. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-16T17:18:09.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13469" PostTypeId="1" CreationDate="2016-08-16T18:09:00.380" Score="3" ViewCount="382" Body="&lt;p&gt;I've been learning about neural networks and I'm curious: Are there any other layer types like convolution and fully connected layers? I know there are things like max pooling or various regularization layers, but I'm mainly interested in more variations on fully connected layers (like how convolution layers do the same thing as fully connected layers but take the shape of data into account). It just seems strange that we wouldn't need anything except straight matrix multiplication or matrix multiplication on smaller chunks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do these exist? If not, why not?&lt;/p&gt;&#xA;" OwnerUserId="23457" LastActivityDate="2016-08-16T19:15:12.930" Title="Are there more layer types like convolution layers and fully connected layers?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13470" PostTypeId="2" ParentId="13469" CreationDate="2016-08-16T18:20:22.993" Score="3" Body="&lt;p&gt;Recurrent neural networks have different types of layers, Long Short Term Memory layers for example. Neural Turing Machines are even wilder. Another example is a non-square convolution layer, I was looking at writing a CNN reinforcement learning program that could learn how to play Abalone, which is played on a hexagon board, where similar patterns appear over the board in hexagon patches, so that is also a possibility.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-16T18:20:22.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13471" PostTypeId="2" ParentId="13357" CreationDate="2016-08-16T18:33:07.650" Score="1" Body="&lt;p&gt;It sounds like a &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow&quot;&gt;reinforcement learning&lt;/a&gt; problem to me: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It's a high-dimensional input space with non-linear interactions&lt;/li&gt;&#xA;&lt;li&gt;The only way to get information from the system is to use it&lt;/li&gt;&#xA;&lt;li&gt;Its behavior is time-dependent given all the intricacies of laser/optics setups&lt;/li&gt;&#xA;&lt;li&gt;There is interplay between short-term and long-term rewards (multiple local maxima vs. a global max.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can search &quot;reinforcement learning matlab&quot; and &quot;markov decision process&quot; for some available projects and tutorials.  &lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-08-16T18:33:07.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13472" PostTypeId="2" ParentId="13445" CreationDate="2016-08-16T18:42:13.900" Score="2" Body="&lt;p&gt;Try this recent paper: &lt;a href=&quot;http://jmlr.org/papers/v17/khaleghi16a.html&quot; rel=&quot;nofollow&quot;&gt;Consistent Algorithms for Clustering Time Series&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question is very much a current research topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an older but excellent paper which talks about the fundamentals: &lt;a href=&quot;https://www.semanticscholar.org/paper/Generalized-Feature-Extraction-for-Structural-Olszewski-Maxion/7838bcd87bb6616e9fd3ffd92d4676a7082da34c/pdf&quot; rel=&quot;nofollow&quot;&gt;Generalized Feature Extraction for Structural Pattern Recognition in Time-series Data&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2016-09-01T14:58:38.313" LastActivityDate="2016-09-01T14:58:38.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13473" PostTypeId="2" ParentId="13469" CreationDate="2016-08-16T19:15:12.930" Score="4" Body="&lt;p&gt;Jan van der Vegt did a good job of mentioning some of the main irregular layer types, but I'll toss in a couple of others for the sake of completeness. I know from experience that layers in Teuvo Kohonen's Self-Organizing Maps can be fully connected, without requiring a square or rectangular matrix. I've gotten excellent results by creating topologies that match the underlying problems, which may feature jagged and possibly multidimensional shapes, then operating on them with sets instead of matrices. That's one reason I prefer using set-based languages like SQL to matrix math when coding neural nets. Another interesting neural net type worth exploring is Neural Gas, which expands across the space of data points to fit the natural topology of a problem, whatever that may be. It is easier to express in terms of manifolds than ordinary space. In a sense, it can be viewed as a generalization and abstraction of Kohonen nets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Resources on Kohonen nets are easy to find, but the literature on Neural Gas is still scarce. Here are some worthwhile links I've read to Neural Gas and assorted variants:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Martinetz and Schultern's &lt;a href=&quot;http://ftp.ks.uiuc.edu/Publications/Papers/PDF/MART91B/MART91B.pdf&quot; rel=&quot;noreferrer&quot;&gt;original article on Neural Gas&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• The &lt;a href=&quot;https://en.wikipedia.org/wiki/Neural_gas&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia article on Neural Gas&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• A &lt;a href=&quot;http://www.booru.net/download/MasterThesisProj.pdf&quot; rel=&quot;noreferrer&quot;&gt;Master's Thesis on Growing Neural Gas&lt;/a&gt;, which dynamically creates the topology&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• &lt;a href=&quot;https://papers.nips.cc/paper/893-a-growing-neural-gas-network-learns-topologies.pdf&quot; rel=&quot;noreferrer&quot;&gt;A Growing Neural Gas Network Learns Topologies&lt;/a&gt;, a research paper by Bernd Fritzke&lt;/p&gt;&#xA;" OwnerUserId="18544" LastActivityDate="2016-08-16T19:15:12.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13475" PostTypeId="2" ParentId="10800" CreationDate="2016-08-16T20:54:23.830" Score="4" Body="&lt;p&gt;I might be a bit late for answering but hope it helps!I assume that you are familiar with RL so I will omit lots of details (please if you are still interested comment so I can help you).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Neural Networks and RL:&lt;/strong&gt; You have two options. The first one is to use a network which you will have as input your feature vectors (states) and output probability of each action. This is called a policy network and you can find a very detailed tutorial with Python Code in order to implement it by &lt;a href=&quot;http://karpathy.github.io/2016/05/31/rl/&quot; rel=&quot;nofollow&quot;&gt;A. Karpathy&lt;/a&gt;.&#xA;Your second option is to use the Q-Network approach. Your input will be again the same but the output will be values of your Q function for each action you have ($Q(a_i)$). You will use the Q-learning  equation $Q(s_t,a_t)=Q(s_t,a_t) +\alpha[r_t+\gamma \max _a'Q(s_{t+1},a')-Q(s_t,a_t)]$. The details of the implementation can be found in the paper of &lt;a href=&quot;http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html&quot; rel=&quot;nofollow&quot;&gt;V. Mnih&lt;/a&gt;. Also do not worry about the delayed rewards as the discount factor $\gamma will &quot;help&quot; your agent to be affected by the future rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to calculate your states, I would suggest you to create a simulation of the environment and a step function. You don't mention what kind of game you are dealing with but the general idea is that the step function will take as input your current state and current action and output the next state and reward (don't mind about the continuous space of your features as you can discretize it - you can use Kalman filters or other models to have a better state estimation as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice would be to choose your approach (Policy net or Q-net) and read the blog or the paper, create a simulation of the environment and a step function for your game. You can find tons of implementations of the Deep Q-net although I would suggest you to start with a very simple network so you don't get in trouble by tuning the Deep net.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2016-08-16T20:54:23.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13476" PostTypeId="2" ParentId="12780" CreationDate="2016-08-16T21:24:47.610" Score="2" Body="&lt;p&gt;One simple idea, no imputation needed: build a model using the parameters have always existed, then each time a new set of parameters gets added, use them to model the &lt;em&gt;residual&lt;/em&gt; of the previous model. Then you can sum the contributions of all the models that apply to the data you happen to have. (If effects tend to multiply rather than add, you could do this in log space.)&lt;/p&gt;&#xA;" OwnerUserId="23450" LastActivityDate="2016-08-16T21:24:47.610" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13477" PostTypeId="1" CreationDate="2016-08-16T21:38:19.747" Score="1" ViewCount="193" Body="&lt;p&gt;I'd like to render a neural network in Python to make sure it is set up properly.  I'd like to verify that it has the correct # of nodes, correct edge configuration, and perhaps even correct edge weights after training.  I'd like to visually debug simple examples by printing them to a PNG/JPG but do not know any libraries that do this.  Is there a library for drawing a neural network for visual debugging?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Such a library would of course make assumptions on the data types/structures - e.g., assuming the neural network comes from Theano, TensorFlow, or some other library.  I'm flexible on the assumptions made.)&lt;/p&gt;&#xA;" OwnerUserId="1097" LastActivityDate="2016-08-16T21:38:19.747" Title="Are there any libraries for drawing a neural network in Python?" Tags="|python|neural-network|" AnswerCount="0" CommentCount="1" ClosedDate="2016-08-19T08:59:59.493" ContentLicense="CC BY-SA 3.0" />
  <row Id="13478" PostTypeId="2" ParentId="12780" CreationDate="2016-08-16T21:52:57.437" Score="0" Body="&lt;p&gt;I would multiply-impute the values for x4, x5, x6, and x7. For the number of imputations, look at the whole dataset and compute the % of fields missing and round up to the nearest integer. Don't use mean- or median-imputation, use PROC MI in SAS or the equivalent. I would guess that because your data are monotone-missing, you could use a MONOTONE statement. This is probably the most conservative approach because you open yourself up to bias when excluding information--whether it be variables or observations.&lt;/p&gt;&#xA;" OwnerUserId="23462" LastActivityDate="2016-08-16T21:52:57.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13480" PostTypeId="2" ParentId="12693" CreationDate="2016-08-17T02:57:05.220" Score="5" Body="&lt;p&gt;I'm surprised nobody mention &lt;a href=&quot;https://arxiv.org/abs/1411.4038&quot; rel=&quot;nofollow noreferrer&quot;&gt;fully convolutional neural networks (FCNs) for semantic segmentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are inspired by the original AlexNet style convnets that ended with one or two densely connected layers and softmax classifier. But FCNs dispense with the dense layers and stay &lt;em&gt;fully convolutional&lt;/em&gt; all the way to the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/k4WlS.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k4WlS.jpg&quot; alt=&quot;from the linked article https://arxiv.org/abs/1411.4038&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the basic principle. Take AlexNet, or VGG or something like that. But instead of using the parameters in the classifier to compute a scalar for each category, use them to compute a whole array (i.e. image) using a  &lt;code&gt;1x1xNUM_CATEGORIES&lt;/code&gt; convolution. The output will be &lt;code&gt;NUM_CATEGORIES&lt;/code&gt; feature maps, each representing a coarse-grained &quot;heat map&quot; for that category. A map of dogness, a map of catness. It can be sharpened by including information from earlier layers with &quot;skip connections&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Just one further bit of good news: the authors of that paper provide implementations of their nets in &lt;a href=&quot;https://github.com/shelhamer/fcn.berkeleyvision.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe's Model Zoo&lt;/a&gt;. Tweak away!&lt;/p&gt;&#xA;" OwnerUserId="21273" LastEditorUserId="21273" LastEditDate="2016-08-20T23:48:10.090" LastActivityDate="2016-08-20T23:48:10.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13481" PostTypeId="2" ParentId="13446" CreationDate="2016-08-17T03:02:08.540" Score="0" Body="&lt;p&gt;There is a good deal of trial-and-error when building NNs, but here's a bit of semi-structured advice on the matter.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;training networks in general:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In addition to the &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf&quot; rel=&quot;nofollow&quot;&gt;LeCun paper&lt;/a&gt; mentioned here, Geoffry Hinton has a great guide to the overall process &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, which i suggest you read. You can gain some intuition about batch training sizes as well, yet another thing to consider when training NNs.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Hidden layers&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Regarding hidden layers on non-recurrent/non-convolutional networks with $n$ features, I've found that a single hidden layer of size $x$ somewhere in the range of $n &amp;gt; x &amp;gt; \log(n)$ works well.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;ReLUs&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Using a ReLU in the hidden layer helps with sparsity and to combat vanishing gradients. ReLUs have an output of $[0, \infty]$ as it is $f(x) = max(0, x)$ where $x$ is the unit's input, where sigmoid units have outputs in the range $[0, 1]$. There is significantly more biological plausibility for ReLUs, as they better approximate a common neuroscience model for neuronal firing, the leaky-integrate-and-fire model. They are slightly more efficient computationally (the only operations are compare, add,and  multiply). For even more detail about ReLUs, read &lt;a href=&quot;http://jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf&quot; rel=&quot;nofollow&quot;&gt;Glorot, Bordes, and Bengio&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11098" LastActivityDate="2016-08-17T03:02:08.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13482" PostTypeId="2" ParentId="13258" CreationDate="2016-08-17T03:19:26.550" Score="1" Body="&lt;p&gt;We have recently completed one such exercise using Markov chains. The data that we used for this project was as follows:&#xA;For a selected set of users, for a 6 month time period their complete campaign history. Format as follows:&#xA;Userid, campaign-id, campaign-date, response tag, response value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The response tag is a binary tag indicating responded or no. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our markov chain approach did not really care about the time of campaign, it only used the order and getting null state vs success. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What approach are you using in your model? I'd be happy to compare notes. While I'd not be able to share the data, we could run your code on our data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We had real campaign history data for a client, hence didn't have to simulate it.&lt;/p&gt;&#xA;" OwnerUserId="23469" LastActivityDate="2016-08-17T03:19:26.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13483" PostTypeId="1" AcceptedAnswerId="13485" CreationDate="2016-08-17T03:31:55.937" Score="0" ViewCount="465" Body="&lt;p&gt;What are good resources for data science applications in finance/banking ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know about the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Trends in finance/banking area &lt;/li&gt;&#xA;&lt;li&gt;What creative companies do in finance/banking area with data science&lt;/li&gt;&#xA;&lt;li&gt;Major open data sources in finance/banking in Canada&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I subscribed to several data science news but none of them focus on finance/banking. I need daily/weekly news and reports for my internship. This is my job and I will be evaluated based on this so it is very important for me to get enough information in time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions? Thank you very much!&lt;/p&gt;&#xA;" OwnerUserId="23470" LastEditorUserId="3151" LastEditDate="2016-08-17T10:41:34.647" LastActivityDate="2016-08-17T15:01:03.617" Title="Resources for data science applications in finance/banking" Tags="|finance|" AnswerCount="1" CommentCount="0" ClosedDate="2016-08-17T10:41:16.850" ContentLicense="CC BY-SA 3.0" />
  <row Id="13485" PostTypeId="2" ParentId="13483" CreationDate="2016-08-17T06:22:26.357" Score="1" Body="&lt;p&gt;Take a look at &lt;a href=&quot;http://www.oreilly.com/data/free/&quot; rel=&quot;nofollow&quot;&gt;O'Reilly free Ebooks&lt;/a&gt; There are a couple of resources for banking/finance/fintech.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some sites from which you can work on for learning more about predictive modelling in this sector:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.) &lt;a href=&quot;https://inclass.kaggle.com/c/name-that-loan-open/data&quot; rel=&quot;nofollow&quot;&gt;https://inclass.kaggle.com/c/name-that-loan-open/data&lt;/a&gt; - Kaggle dataset   for prediction of interest rates&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.) &lt;a href=&quot;https://datahack.analyticsvidhya.com/contest/practice-problem-loan-prediction-iii/&quot; rel=&quot;nofollow&quot;&gt;https://datahack.analyticsvidhya.com/contest/practice-problem-loan-prediction-iii/&lt;/a&gt; - Dataset on Analytics Vidhya for loan prediction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.) There is also a twitter handle with the name @StatCan_eng posting datasets related to Canada specifically.  &lt;/p&gt;&#xA;" OwnerUserId="3151" LastEditorUserId="23380" LastEditDate="2016-08-17T15:01:03.617" LastActivityDate="2016-08-17T15:01:03.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13487" PostTypeId="1" CreationDate="2016-08-17T07:59:27.757" Score="0" ViewCount="332" Body="&lt;p&gt;I have a database of multivariate time series that I want to cluster in order to find natural grouping of features. I am thinking of taking each cluster points and perform an ARIMA analysis on its points. Within those clusters, what should be the best way to infer those groupings in the original mutltidimentional feature space? &#xA;a sample of my time series is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TimeStamp       | Sensor0 | Sensor1| Sensor2&#xA;2015-02-05 11:30|&amp;lt;Min     | On     | off&#xA;2015-02-05 11:31|&amp;lt;Min     | on     | off &#xA;2015-02-05 11:32| Action2 | 10     | 0.0001  &#xA;2015-02-07 11:33| Action2 | 10     | 0.00012 &#xA;2015-02-07 11:34| Action2 | 10     | 0.00012 &#xA;2015-02-07 11:35| Action2 | 20     | 0.00015 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have roughly 10 separate groupings and I don't think performing different ARIMA calculations would be the best efficient method. Any thoughts or ideas are also welcome!&lt;/p&gt;&#xA;" OwnerUserId="23440" LastEditorUserId="23440" LastEditDate="2016-08-17T09:29:07.273" LastActivityDate="2016-08-17T18:09:40.523" Title="arima analysis after clustering time series" Tags="|clustering|time-series|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13488" PostTypeId="2" ParentId="13279" CreationDate="2016-08-17T08:40:21.443" Score="2" Body="&lt;p&gt;Apache Spark is a great solution for such problems.&lt;/p&gt;&#xA;&lt;p&gt;But, first let's be clear about the definition of &lt;strong&gt;real time processing&lt;/strong&gt;. It's the type of processing that must guarantee response within specified time which on an interactive business site is actually very low. You can read about those kind of specifications in &lt;a href=&quot;https://stackoverflow.com/questions/164175/what-is-considered-a-good-response-time-for-a-dynamic-personalized-web-applicat&quot;&gt;this answer&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Spark doesn't provide such luxury in predicting under 0.1 sec and I'm citing&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Excerpt from Chapter 5 in my book Usability Engineering, from 1993:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;0.1 second&lt;/strong&gt; is about the limit for having the user feel that the system is &lt;strong&gt;reacting instantaneously&lt;/strong&gt;, meaning that no special feedback is necessary except to display the result.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Having an &lt;strong&gt;interactive business site&lt;/strong&gt; on which you'd want to display predictions doesn't mean that your predictions has to be in real time.&lt;/p&gt;&#xA;&lt;p&gt;So the obvious is, actually, the following :&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; &lt;em&gt;Now that I have computed recommendations for my users, what should I do ?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A:&lt;/strong&gt; Let's define a &lt;strong&gt;serving layer&lt;/strong&gt; for our systems fast enough to query when recommendations are needed.&lt;/p&gt;&#xA;&lt;p&gt;It can be anything fast enough to answer your calls e.g Elasticsearch, Solr, HBase, Redis. Whatever flavor suits you.&lt;/p&gt;&#xA;&lt;p&gt;On other hands, well&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; &lt;em&gt;I don't want my system to be static, I need to recompute my predictions every T hours/days/etc&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A:&lt;/strong&gt; Spark can do a scheduled job perfectly here. (a simple cron would do)&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; &lt;em&gt;But when do I retrain my recommender system ?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A:&lt;/strong&gt; I would say it actually depends on so many stuff a bit too broad to discuss here. You can read about the topic &lt;a href=&quot;https://www.quora.com/How-often-do-big-companies-refit-their-learning-algorithms-to-new-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; if you wish.&lt;/p&gt;&#xA;&lt;p&gt;Ok, so we defined now our &lt;strong&gt;batch layer&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; &lt;em&gt;And what about data coming in real time, through Kafka, Rabbit, etc. ?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A:&lt;/strong&gt; This is actually when it can get more complicated, because the method that you'll use to compute distances, approximations, new recommendations will depends on what type of recommender systems you are building and what technologies you are using.&lt;/p&gt;&#xA;&lt;p&gt;Spark streaming can fit very well to apply &amp;quot;simple&amp;quot; computations on &amp;quot;window&amp;quot; based micro batches. This can be our &lt;strong&gt;speed layer&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;To conclude, all of the above defines what is called a &lt;a href=&quot;http://lambda-architecture.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;lambda architecture&lt;/a&gt;. And one of the best framework that follows this design is &lt;a href=&quot;https://github.com/OryxProject/oryx&quot; rel=&quot;nofollow noreferrer&quot;&gt;Oryx&lt;/a&gt; (personal opinion).&#xA;It's quite interesting, you ought taking a look at it.&lt;/p&gt;&#xA;&lt;p&gt;I also believe that it's quite possible to have a RT set-up for a recommendation system without the &lt;strong&gt;speed layer&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I hope that this answers your question.&lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-08-17T15:12:18.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13489" PostTypeId="1" CreationDate="2016-08-17T08:57:48.920" Score="0" ViewCount="3594" Body="&lt;p&gt;I have a data.frame which has multiple time series in it, in the following manner:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;01.01.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1&#xA;01.02.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1&#xA;01.03.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1&#xA;01.01.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2&#xA;01.02.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2&#xA;01.03.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Meaning: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is one timeframe which is the same for every time series&lt;/li&gt;&#xA;&lt;li&gt;There are multiple (thousands) of time series within that time &lt;/li&gt;&#xA;&lt;li&gt;All are made up by the same type of values (columns) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to cluster this dataset in order to find the time series objects similar to each other. &#xA;I want to find those time series(s) which changed the same with respect to the values mentioned. I'm obviously interested in the measured distance. But using a simple hierarchical clustering algorithm is not cutting it since I need to tell the (whatever clustering algorithm I end up using) what is a time series and what not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using R for this. If there might be a better option with Python, I'm totally open to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any kind of help!&lt;/p&gt;&#xA;" OwnerUserId="23477" LastActivityDate="2019-10-31T18:02:30.867" Title="How to cluster multiple time-series from one data frame" Tags="|python|r|clustering|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13490" PostTypeId="1" AcceptedAnswerId="13496" CreationDate="2016-08-17T09:35:45.110" Score="263" ViewCount="428840" Body="&lt;p&gt;I know that there is a possibility in Keras with the &lt;code&gt;class_weights&lt;/code&gt; parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to provide one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, in this case the appropriate praxis is simply to weight up the minority class proportionally to its underrepresentation?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="109650" LastEditDate="2020-12-31T11:08:43.710" LastActivityDate="2021-07-15T16:24:48.873" Title="How to set class weights for imbalanced classes in Keras?" Tags="|deep-learning|classification|keras|weighted-data|" AnswerCount="10" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13492" PostTypeId="2" ParentId="4977" CreationDate="2016-08-17T10:06:24.210" Score="0" Body="&lt;p&gt;Flink is an upcoming good equivalent to Spark (or even better) as it process each tuples unlike micro batch concept of spark streaming.&lt;/p&gt;&#xA;" OwnerUserId="21072" LastActivityDate="2016-08-17T10:06:24.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13493" PostTypeId="2" ParentId="4977" CreationDate="2016-08-17T10:33:16.200" Score="1" Body="&lt;p&gt;Just to throw an extra option in, &lt;a href=&quot;https://azure.microsoft.com/en-gb/services/stream-analytics/&quot; rel=&quot;nofollow&quot;&gt;Microsoft Azure's Stream Analytics&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as a comparison of the different technologies and their relative performance, those stats seem difficult to get, but there is a useful comparison of a few technologies in &lt;a href=&quot;https://www.infoq.com/articles/stream-processing-hadoop&quot; rel=&quot;nofollow&quot;&gt;this article&lt;/a&gt;, though doesn't mention Azure Stream Analytics. I think the difficulty is that the architectures are so different, that is makes simple metrics hard to produce. What is needed is some more robust measurement tools that can be applied to give a few metrics for various operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think that Gartner would have something to say, but the nearest I could find was their &lt;a href=&quot;http://www.kdnuggets.com/2016/02/gartner-2016-mq-analytics-platforms-gainers-losers.html&quot; rel=&quot;nofollow&quot;&gt;BI and Analytics quadrant&lt;/a&gt;, which only obliquely address stream analytics.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-08-17T10:33:16.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13494" PostTypeId="2" ParentId="307" CreationDate="2016-08-17T10:35:14.380" Score="1" Body="&lt;p&gt;Take a look at &lt;a href=&quot;http://www.oreilly.com/data/free/archive.html&quot; rel=&quot;nofollow&quot;&gt;O'Reilly free data reports&lt;/a&gt; . You can find reports on Banking and Fintech, Sports, Fashion, Music, Health, Oil and Gas and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that McKinsey report mentioned earlier is a classic report and a must-read.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-08-17T10:35:14.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13495" PostTypeId="2" ParentId="13028" CreationDate="2016-08-17T10:40:19.737" Score="1" Body="&lt;p&gt;If PMML supports the model you're trying to express, it's probably the best option. Spark has some partial support for exporting as PMML. Look into JPMML + OpenScoring to score the PMML in an embedded instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are loads of tools in this space that implement something like this, but they're typically entwined with their own formats or model management system.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-08-17T10:40:19.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13496" PostTypeId="2" ParentId="13490" CreationDate="2016-08-17T10:49:51.700" Score="221" Body="&lt;p&gt;If you are talking about the regular case, where your network produces only one output, then your assumption is correct. In order to force your algorithm to treat every instance of &lt;strong&gt;class 1&lt;/strong&gt; as 50 instances of &lt;strong&gt;class 0&lt;/strong&gt; you have to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Define a dictionary with your labels and their associated weights&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class_weight = {0: 1.,&#xA;                1: 50.,&#xA;                2: 2.}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Feed the dictionary as a parameter:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(X_train, Y_train, nb_epoch=5, batch_size=32, class_weight=class_weight)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;&quot;treat every instance of &lt;strong&gt;class 1&lt;/strong&gt; as 50 instances of &lt;strong&gt;class 0&lt;/strong&gt;&quot; means that in your loss function you assign higher value to these instances.&#xA;Hence, the loss becomes a weighted average, where the weight of each sample is specified by &lt;strong&gt;class_weight&lt;/strong&gt; and its corresponding class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://keras.io/models/sequential/&quot; rel=&quot;noreferrer&quot;&gt;Keras docs&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;class_weight&lt;/strong&gt;: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="23478" LastEditorUserId="43000" LastEditDate="2020-04-11T13:33:02.133" LastActivityDate="2020-04-11T13:33:02.133" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="13497" PostTypeId="2" ParentId="12097" CreationDate="2016-08-17T13:20:35.430" Score="1" Body="&lt;p&gt;When I built my first vanilla RNN from scratch, I was very curious about it's correctness. So, I trained it on a portion of a sine curve and tried to generate the part of the curve that comes next. And consequently, I plotted mine and the actual to visualise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(P.S.: &lt;a href=&quot;http://nbviewer.jupyter.org/github/rajarsheem/playing-with-RNNs/blob/fd394d6b381643025dd65ec547ba7221fd20e765/int_seq.ipynb&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt; is my code!)&lt;/p&gt;&#xA;" OwnerUserId="16915" LastEditorUserId="16915" LastEditDate="2016-08-17T15:31:06.930" LastActivityDate="2016-08-17T15:31:06.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13499" PostTypeId="2" ParentId="13489" CreationDate="2016-08-17T16:27:30.830" Score="3" Body="&lt;p&gt;For an applied solution to your problem, I highly recommend reading the following: &lt;em&gt;TSclust: An R Package for Time Series Clustering&lt;/em&gt; which can be found &lt;a href=&quot;https://www.jstatsoft.org/article/view/v062i01&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It presents the &lt;code&gt;TSclust&lt;/code&gt; package in R and provides code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that time series data is special, and cannot be treated like other data. So typical clustering techniques are not appropriate.&lt;/p&gt;&#xA;" OwnerUserId="16220" LastEditorUserId="29169" LastEditDate="2019-10-31T18:02:30.867" LastActivityDate="2019-10-31T18:02:30.867" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13500" PostTypeId="1" CreationDate="2016-08-17T16:59:26.300" Score="2" ViewCount="2901" Body="&lt;p&gt;I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any reference to documentation about the linear boosts in XGBoost will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Linear boosts can be implemented in XGBoost by setting the 'booster' parameter to 'gblinear'. See:&#xA;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/&quot; rel=&quot;nofollow&quot;&gt;http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/&lt;/a&gt;&#xA;for useful information on linear boosting. Note that I am not speaking about the objective function (which can be also linear) but about the boosts themselve.&lt;/p&gt;&#xA;" OwnerUserId="23492" LastEditorUserId="23492" LastEditDate="2016-08-18T12:12:48.173" LastActivityDate="2016-08-20T10:29:54.103" Title="How does xgboost work with linear booster?" Tags="|python|linear-regression|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13501" PostTypeId="2" ParentId="13279" CreationDate="2016-08-17T18:01:51.160" Score="1" Body="&lt;p&gt;The best solution is not to compute the cosine similarity (in order to recommend an item) but to &lt;a href=&quot;https://papers.nips.cc/paper/5329-asymmetric-lsh-alsh-for-sublinear-time-maximum-inner-product-search-mips.pdf&quot; rel=&quot;nofollow&quot;&gt;transform&lt;/a&gt; &lt;a href=&quot;http://www.www2015.it/documents/proceedings/companion/p1263.pdf&quot; rel=&quot;nofollow&quot;&gt;your&lt;/a&gt; &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Shen_Learning_Binary_Codes_ICCV_2015_paper.pdf&quot; rel=&quot;nofollow&quot;&gt;features&lt;/a&gt; so that you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Nearest_neighbor_search#Approximate_nearest_neighbor&quot; rel=&quot;nofollow&quot;&gt;approximate similarity search&lt;/a&gt;, for which &lt;a href=&quot;https://github.com/erikbern/ann-benchmarks&quot; rel=&quot;nofollow&quot;&gt;many fast options&lt;/a&gt; exist. In other words, precalculate to expedite queries. (This means you will have found a co-embedding for your users and items.)&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-17T18:01:51.160" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13502" PostTypeId="2" ParentId="13487" CreationDate="2016-08-17T18:09:40.523" Score="1" Body="&lt;p&gt;Here's a reference for your problem. If you read the first two pages, you'll get a hint at what features to use for time series clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.jstatsoft.org/article/view/v062i01&quot; rel=&quot;nofollow&quot;&gt;https://www.jstatsoft.org/article/view/v062i01&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2016-08-17T18:09:40.523" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13503" PostTypeId="2" ParentId="13409" CreationDate="2016-08-17T18:49:28.163" Score="0" Body="&lt;p&gt;Say your data is in a DataFrame &lt;code&gt;df&lt;/code&gt;, and the column of interest is named &lt;code&gt;column&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;treshold = 5 #or whatever value makes sense to you&#xA;&#xA;def set_levels(df):&#xA;&#xA;    for i df.column.unique():&#xA;       if len(df.loc[df.column == i]) &amp;lt; treshold:&#xA;          df.loc[df.column == i] = 'other'&#xA;&#xA;    return df&#xA;&#xA;new_df = set_levels(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23495" LastEditorUserId="23495" LastEditDate="2016-08-18T19:12:31.513" LastActivityDate="2016-08-18T19:12:31.513" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13504" PostTypeId="1" CreationDate="2016-08-17T19:08:17.143" Score="5" ViewCount="654" Body="&lt;p&gt;I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my experience with ML/data science is through Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have insight as to skills that employers will expect me to add before I get a job? I've been looking at postings and it seems like most of the things that I'm not as sharp with are: SQL, Hadoop, Python, and R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are Python and R both necessary, or would that be redundant?&#xA;Is SQL and Hadoop something employers will overlook? Because quite frankly I find learning both to be unbearably boring. &#xA;Lastly, am I missing anything from this list of skills?&lt;/p&gt;&#xA;" OwnerUserId="23496" LastActivityDate="2017-03-20T01:31:13.360" Title="Skills that school doesn't teach you" Tags="|machine-learning|python|r|data|apache-hadoop|" AnswerCount="8" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13505" PostTypeId="2" ParentId="10322" CreationDate="2016-08-17T19:22:46.013" Score="3" Body="&lt;p&gt;You can use &lt;a href=&quot;http://bokeh.pydata.org/en/latest/docs/user_guide/charts.html#grouping&quot; rel=&quot;nofollow noreferrer&quot;&gt;grouping&lt;/a&gt; in the Bokeh high-level bar chart if you first melt your Pandas dataframe.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from bokeh.plotting import figure, show&#xA;&#xA;# Use output_notebook if you are using an IPython or Jupyter notebook&#xA;from bokeh.io import output_notebook&#xA;output_notebook()&#xA;&#xA;# Get your data into the dataframe&#xA;df = pd.read_csv(&quot;data.csv&quot;)&#xA;&#xA;# Create a &quot;melted&quot; version of your dataframe&#xA;melted_df = pd.melt(df, id_vars=['year'], value_vars=['Recalled', 'Recovered'])&#xA;&#xA;melted_df.head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the format of your melted dataframe:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+---+------+----------+--------+&#xA;|   | year | variable | value  |&#xA;+---+------+----------+--------+&#xA;| 0 | 1994 | Recalled | 11.472 |&#xA;| 1 | 1995 | Recalled |  11.81 |&#xA;| 2 | 1996 | Recalled | 10.632 |&#xA;| 3 | 1997 | Recalled | 13.857 |&#xA;| 4 | 1998 | Recalled | 13.861 |&#xA;+---+------+----------+--------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then just use your melted dataframe as your data in the Bokeh bar chart:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p = Bar(melted_df, label=&quot;year&quot;, values=&quot;value&quot;, group=&quot;variable&quot;, legend=&quot;top_left&quot;, ylabel='Values')&#xA;show(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MFGaF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MFGaF.png&quot; alt=&quot;Bar chart&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-08-17T19:22:46.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13506" PostTypeId="2" ParentId="13504" CreationDate="2016-08-17T19:26:42.530" Score="3" Body="&lt;p&gt;Based on my own experience and in reading what others have written, SQL is one of those skills employers look for, perhaps even assume that you have along with some of the basic skills of communication and teamwork.  The main reason is that lots of data are stored in a relational database with SQL being the primary way to extract that data to get it into your models.&lt;/p&gt;&#xA;" OwnerUserId="23420" LastEditorUserId="23420" LastEditDate="2016-08-17T19:31:55.687" LastActivityDate="2016-08-17T19:31:55.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13507" PostTypeId="2" ParentId="13504" CreationDate="2016-08-17T20:21:15.370" Score="1" Body="&lt;p&gt;Hadoop and SQL are things that you will pick up reasonably quickly and in my experience are far from necessary for a lot of jobs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would focus on Python or on R, not on both. A lot of employers are Python and R and allow you to choose yourself as long as it's on one or both. I feel like Python is growing faster in the Data Science community than R and is in my opinion a much more sophisticated language and has a wider support for things that help with data science but are not directly related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As jab already mentioned, I think soft skills are very important, although difficult to learn without real working experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best thing to prepare in my opinion is just by doing some project(s) and figuring things out along the way. Join a Kaggle competition or find some personal project. I spend hours and hours working on those and I keep coming across problems that I haven't faced before, requiring me to read papers, implementing some new ideas, trying things out. This also allows you to build up a github portfolio to show off some cool projects you have been working on.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-17T20:21:15.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13508" PostTypeId="2" ParentId="5192" CreationDate="2016-08-17T20:22:51.343" Score="2" Body="&lt;p&gt;Go to pubmed. &#xA;&lt;a href=&quot;http://www.ncbi.nlm.nih.gov/pubmed/&quot; rel=&quot;nofollow&quot;&gt;http://www.ncbi.nlm.nih.gov/pubmed/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;typing the key words of your interest, you will able to pull all the public health research papers with the keywords chronologically from all the journals.   &lt;/p&gt;&#xA;" OwnerUserId="23498" LastActivityDate="2016-08-17T20:22:51.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13509" PostTypeId="2" ParentId="13415" CreationDate="2016-08-17T20:42:36.400" Score="1" Body="&lt;p&gt;The error is likely due to the fact that your &lt;code&gt;x&lt;/code&gt; is an irregular time series: there are temporal gaps, e.g. no data on 2015-01-03, 2015-01-04. You will have to implement your own version of &lt;code&gt;decompose&lt;/code&gt;, e.g. following this approach:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/12623027/how-to-analyse-irregular-time-series-in-r&quot;&gt;https://stackoverflow.com/questions/12623027/how-to-analyse-irregular-time-series-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23234" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-17T20:42:36.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13513" PostTypeId="1" AcceptedAnswerId="13514" CreationDate="2016-08-18T05:05:45.470" Score="57" ViewCount="13783" Body="&lt;p&gt;I see a many times in job description for data scientist asking for Python/Java experience and disregard R. Below is a personal email I received from chief data scientist of a company I applied for through linkedin.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X, Thanks for connecting and expressing interest. You do have good&#xA;  Analytics Skills. However,  all our data scientists must have  good&#xA;  programming skills in Java/Python as we are a internet/mobile&#xA;  organisation and everything we do is online.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While I respect the decision of the chief data scientist, I am unable to get a clear picture as to what are the tasks that Python can do that R cannot do. Can anyone care to elaborate? I am actually keen to learn Python/Java, provided I get a bit more detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I found an interesting discussion on Quora.&#xA;&lt;a href=&quot;https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists&quot;&gt;Why is Python a language of choice for data scientists?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit2: Blog from Udacity on &lt;a href=&quot;http://blog.udacity.com/2016/04/languages-and-libraries-for-machine-learning.html&quot;&gt;Languages and Libraries for Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="13100" LastEditDate="2016-08-26T03:01:05.177" LastActivityDate="2017-07-09T17:22:02.853" Title="Why do internet companies prefer Java/Python for data scientist job?" Tags="|beginner|tools|career|reference-request|" AnswerCount="8" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="13514" PostTypeId="2" ParentId="13513" CreationDate="2016-08-18T06:14:06.393" Score="67" Body="&lt;p&gt;So you can integrate with the rest of the code base. It seems your company uses a mix of Java and python. What are you going to do if a little corner of the site needs machine learning; pass the data around with a database, or a cache, drop to R, and so on? Why not just do it all in the same language? It's faster, cleaner, and easier to maintain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Know any online companies that run solely on R? Neither do I...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that said Java is the last language I'd do data science in.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-18T06:14:06.393" CommentCount="15" ContentLicense="CC BY-SA 3.0" />
  <row Id="13515" PostTypeId="2" ParentId="13504" CreationDate="2016-08-18T06:48:00.050" Score="0" Body="&lt;p&gt;Python is more suited for a 'real' data scientist work whose work needs to be productized and used in a software or website. We need to remember that ultimately whatever we do needs to be productized. Although R has Shiny library that can be used for web hosting, we need to remember that a website will not have only machine learning component, it will have many other aspects, which cannot be built in R, but needs a full fledged software development language. I sent my resume to senior level data scientist for job application and the response i received is in the linked question.&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/13513/why-do-internet-companies-prefer-java-python-for-data-scientist-job/&quot;&gt;Why do internet companies prefer Java/Python for data scientist job?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-08-18T06:48:00.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13516" PostTypeId="2" ParentId="13504" CreationDate="2016-08-18T07:10:14.237" Score="2" Body="&lt;p&gt;I believe technology is cheap and science is expensive. You can learn R, Python, SQL and Hadoop pretty fast (considering that you know programming) but learning statistics, machine learning and the methodology of working with data is difficult and takes time. (which you know based on your background)&#xA;In my eyes, go and apply for jobs with self-confidence. In the meanwhile, consider learning SQL and Python. They are necessary for jobs in industry.&lt;/p&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-08-18T07:10:14.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13517" PostTypeId="1" AcceptedAnswerId="13524" CreationDate="2016-08-18T08:11:23.383" Score="5" ViewCount="253" Body="&lt;p&gt;I am trying to understand regularisation for logistic regression currently, but I am not sure I get it. I understand the issue of overfitting when there are relatively too many features, and I get that you would like to limit the impact of some of these superfluous features, but you are doing in regularisation is impacting all features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this brings me to my next point, which is that by putting a limit on weights, you can still construct the same hyperplane. As long as you have the same ratio between the weights $w_i$ for $i \in \{1,2,..,n\}$, the &quot;angle&quot; of the hyperplane would still be the same, and then you can simply &quot;adjust&quot; it's position by the intercept $w_0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many lecturers and authors use the complex polynomial example with additional terms $f(x_i)$ that make the decision boundary non-linear to illustrate that using regularisation will make this boundary more linear, and thus less prone to overfitting, but then again, you could plot $f(x_i)$ on a separate axis, making decision boundary a hyperplane, and then you could fit the exact same decision boundary with smaller weights, as long as the ratios are conserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, if you can make the exact same hypothesis with smaller sum of squares of weights $\sum w_i^2$ (that is with smaller regularisation term), what is the point of regularisation? Or in other words, as there obviously exists empirical evidence showing that regularisation works, how does it work? Is there a good proof for it, or some good (and less vague) intuition?&lt;/p&gt;&#xA;" OwnerUserId="23511" LastActivityDate="2016-08-19T10:40:13.237" Title="Good explanation for why regularisation works" Tags="|logistic-regression|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13519" PostTypeId="1" AcceptedAnswerId="13520" CreationDate="2016-08-18T09:04:47.370" Score="1" ViewCount="639" Body="&lt;p&gt;I have 2 matrices, A (1000x21) and B (1000x7). Matrix A has individuals(=1000) in the rows and their consumption in 21 days at the columns. Matrix B has the &lt;strong&gt;SAME&lt;/strong&gt; individuals(=1000) in the rows and some weights for each day of the week(=7) in the columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to have in the end is 1000 (with the dimension of 2x21) matrices  (one for each individual), lets call them $X_{i}$. In the first row of each $X_{i}$ I would like to have the consumption of the individual $i$ each of the 21 days (this will come from matrix A), and at the second row of the $X_{i}$ I would like to have the respective weight of that day (this will come from matrix B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So matrix A looks like $[cons_{1,1} \ cons_{1,2} \ ... \ cons_{1,21} \\&#xA;                        \ cons_{2,1} \ cons_{2,2} \ ... \ cons_{2,21} \\&#xA;                                     . \\&#xA;                        . \\&#xA;                         . \\&#xA;                       \ cons_{1000,1} \ cons_{1000,2} \ ... \ cons_{1000,21}] $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matrix B looks like $[weight_{1,1} \ weight_{1,2} \ ... \ weight_{1,7} \\ \ weight_{2,1} \ weight_{2,2} \ ... \ weight_{2,7} \\ . \\ . \\ . \\ \ weight_{1000,1} \ weight_{1000,2} \ ... \ weight_{1000,7}]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I would like the matrix $X_{i}$ to be like $[cons_{i,1} \ cons_{i,2} \ ... \ cons_{i,21} \\ \ weight_{i,1} \ weight_{i,2} \ ... weight_{i,k}]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to do this in R in a loop ?&lt;/p&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2016-08-18T10:31:00.713" Title="Create multiple matrices from 2 bigger ones in R" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13520" PostTypeId="2" ParentId="13519" CreationDate="2016-08-18T09:43:29.663" Score="2" Body="&lt;p&gt;This will extract the rows from matrix &lt;strong&gt;A&lt;/strong&gt; and &lt;strong&gt;B&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to get them into a list (recommend)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;single&amp;lt;-lapply(c(1:1000), function(x) matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To put them all in diagonal matrix with &lt;em&gt;adiag&lt;/em&gt; from &lt;strong&gt;magic&lt;/strong&gt; package&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d&amp;lt;-adiag(single[[1]])&#xA;&#xA;for(i in 2:1000){&#xA;  d&amp;lt;-adiag(d,single[[i]])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I could make it work without the loop (anyone any suggestions?)&lt;/p&gt;&#xA;" OwnerUserId="21990" LastEditorUserId="21990" LastEditDate="2016-08-18T10:31:00.713" LastActivityDate="2016-08-18T10:31:00.713" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="13521" PostTypeId="2" ParentId="13500" CreationDate="2016-08-18T10:54:36.680" Score="1" Body="&lt;p&gt;Refer to the authors' paper on xgboost here - &lt;a href=&quot;http://arxiv.org/pdf/1603.02754v1.pdf&quot; rel=&quot;nofollow&quot; title=&quot;XGBoost: A Scalable Tree Boosting System&quot;&gt;XGBoost: A Scalable Tree Boosting System&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Equation 3 on page 2 mentions that in each step all the predictors are used to greedily fit the next additive tree.&#xA;However, since column sub-sampling is also employed to prevent over-fitting, not all features may actually be used for each step, see section 2.3 of the same paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, the specific answer to your query is that the algorithm uses all features to fit the residual.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-08-18T10:54:36.680" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13522" PostTypeId="2" ParentId="13513" CreationDate="2016-08-18T11:25:45.870" Score="26" Body="&lt;p&gt;There may be a lot of reasons like: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Workforce flexibility: One Java / Python programmers can be moved to other tasks or projects easily.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Candidates availability: there are plenty of Java / Python programmers. You do not want to introduce a new programming language to later find out that there are no qualified workers or they are just too expensive. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Integration and ETL: Sometimes getting the data with the right quality is the hardest part of the project. So it is natural to use the same language as the rest of the systems.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Business model definition: Most business rules and business models are already written in this languages. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Just keeping things simple. It is already hard enough to be up-to-date with the technologies. A diverse base of language can be chaotic. R for this, Ruby for that, Scala, Clojure, F#, Swift, Dart... They may need different servers, different pathes, a hell to administer. All have their own IDEs with tools and plugins (not always free).  See some Uncle Bob's points about &lt;a href=&quot;http://blog.cleancoder.com/uncle-bob/2016/05/21/BlueNoYellow.html&quot;&gt;languages choice&lt;/a&gt; and &lt;a href=&quot;http://blog.cleancoder.com/uncle-bob/2016/07/27/TheChurn.html&quot;&gt;new technologies&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So even if you have a 5% - 15% productivity advantage using R for the specific task, they may prefer a tool that just does the job even if not in the most efficient way. &lt;/p&gt;&#xA;" OwnerUserId="23521" LastEditorUserId="2544" LastEditDate="2016-08-18T15:31:22.117" LastActivityDate="2016-08-18T15:31:22.117" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13523" PostTypeId="1" CreationDate="2016-08-18T11:31:41.423" Score="2" ViewCount="922" Body="&lt;p&gt;I am using Gaussian Process Regressor to fit data for a Bayesian Optimiser. This is a relevant part of my Python code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.gaussian_process import GaussianProcessRegressor&#xA;from sklearn.gaussian_process.kernels import (RBF, Matern, RationalQuadratic,&#xA;                                          ExpSineSquared, DotProduct,&#xA;                                          ConstantKernel)&#xA;kernel = Matern(length_scale=lenSc,nu=2.5)&#xA;gp = GaussianProcessRegressor(kernel=kernel)&#xA;l2=10&#xA;Xi=np.concatenate((nprand.uniform(10.0,30.0,[l2,6]),nprand.uniform(-0.3,0.3,[l2,42]),nprand.uniform(0.0,30.0,[l2,1]),nprand.uniform(-30.0,30.0,[l2,1]),nprand.uniform(0.0,0.3,[l2,6]),nprand.uniform(5.0,20.0,[l2,6])),axis=1)&#xA;yi=map(lambda x: f1(x), Xi)&#xA;gp.fit(Xi, yi)&#xA;while(1):&#xA;   #Some code to get the new values- cc and nv&#xA;   Xi=numpy.row_stack((Xi,cc))&#xA;   yi=np.concatenate((yi,[nv]))&#xA;   gp.fit(Xi, yi)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each iteration of the Bayesian Optimiser, I add a new element to $Xi$ and $yi$ and fit $gp$, the Gaussian Process, again. As the size of $Xi$ and $yi$ reach over 200, the time taken to fit the data becomes noticeable, this decreases the overall efficiency of my Bayesian Optimiser. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Bayesian Optimiser has to make around 1000 iterations, each iteration needs to fit the data all over again. So, I was wondering if there's a way to incrementally fit the data? If that's not already being done by &lt;a href=&quot;http://scikit-learn.org/dev/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html#sklearn.gaussian_process.GaussianProcessRegressor&quot; rel=&quot;nofollow&quot;&gt;sklearn's GaussianProcessRegressor&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="16812" LastActivityDate="2016-08-18T20:06:38.107" Title="Can Gaussian Process be fit incrementally?" Tags="|machine-learning|scikit-learn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13524" PostTypeId="2" ParentId="13517" CreationDate="2016-08-18T11:34:28.660" Score="3" Body="&lt;p&gt;The &quot;same angle hyperplane&quot; does not have the same cost. It is the same decision boundary as you describe it, but perpendicular distances to it are larger wrt the norm of the weights. In effect with higher weights in the same ratio (i.e. without any regularisation effect), the classifier will be more confident in all of its decisions. That means the classifier will be more sensitive to getting as many observations as possible in the training set on the &quot;right&quot; side of the boundary. In turn this makes it sensitive to noise in the observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your estimated probability for being in the positive class is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(y=1|X) = \frac{1}{1+e^{-W^TX}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This includes $w_0$ and fixed value 1 for $x_0$. If you take the midpoint, decision line where $W^TX$ is zero (and the output is at the threshold value 0.5), that defines your decision hyperplane in $X$ space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $W$ has the same factors but a larger norm with $w_0$ compensating to make the same hyperplane, then $X$ values that were on the decision hyperplane still give threshold value of 0.5. However, $X$ values away from the hyperplane will deviate more strongly. If instead of 0 you had $W^TX=1.0$ and doubled the weights keeping the same hyperplane, you would get $W^TX=2.0$ for that example. Which changes your confidence from 0.73 to 0.88.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual cost function without regularisation for logistic regression with example vectors $X_j$ and targets $y_j$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J = - \sum_{\forall j} y_jlog(\frac{1}{1+e^{-W^TX_j}}) + (1 -y_j)(1  - log(\frac{1}{1+e^{-W^TX_j}}))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost is more sensitive to distances from the hyperplane for larger weight values. Looking your example for the imaginary item (with 0.73 or 0.88 confidence), when the categorisation is correct (i.e. y=1) the score would improve by 0.19 for that example if the weights doubled. When the categorisation is wrong (y=0) then the score would worsen by 0.81. In other words for &lt;em&gt;higher&lt;/em&gt; weights, with same weight ratio, the same miscategorisations are punished more than correct categorisations are rewarded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When training, weights will converge to specific balanced weight vector for the minimum cost, &lt;strong&gt;not&lt;/strong&gt; to a specific ratio that forms a &quot;best decision hyperplane&quot;. That's because the hyperplane does not correspond to a single value of the cost function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can demonstrate this effect. Train a logistic regression classifier - without any regularisation to show it has nothing to do with that. Take the weight vector and multiply by some factor e.g. 0.5.Then re-train starting with those weights. You will end up with the same weights as before. The cost function minimum clearly defines &lt;em&gt;specific&lt;/em&gt; weight values, not a ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you add regularisation, that changes the cost and how the weights will converge. Higher regularisation in effect makes the classifier prefer a boundary with lower confidence in all its predictions, it penalises &quot;near misses&quot; less badly because the weights are forced down where possible. When viewed as a hyperplane, the boundary will likely be different.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-08-19T10:40:13.237" LastActivityDate="2016-08-19T10:40:13.237" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13525" PostTypeId="1" AcceptedAnswerId="13531" CreationDate="2016-08-18T11:35:25.867" Score="3" ViewCount="4554" Body="&lt;p&gt;I was reading &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/B00PUM14EY&quot; rel=&quot;nofollow&quot;&gt;Modern Optimization with R (Use R!)&lt;/a&gt; and wondering if a book like this exists in Python too? To be precise something that covers stochastic gradient descent and other advanced optimization techniques. Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="13100" LastEditDate="2016-08-18T12:13:12.273" LastActivityDate="2022-08-15T12:02:09.390" Title="Is there any book for modern optimization in Python?" Tags="|beginner|tools|career|reference-request|books|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13526" PostTypeId="2" ParentId="13513" CreationDate="2016-08-18T14:02:30.057" Score="14" Body="&lt;p&gt;It is in general true that for purely data science and statistics exercises R offers the best and fastest (especially if using the &lt;code&gt;data.table&lt;/code&gt; package) tools and methods, that otherwise would be heavier to implement in Python (I assume by Python we all mean Pandas, though). Most data scientists do in fact use R to perform their models and calculations, or just to see how data behave.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the exercise is complete it is time to make it available to the rest of the people who have to use it (i. e. to deploy); to this aim it is oftentimes preferred to submit the code in Python for two main reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Most architectures are written in Python or are Python-friendly, therefore it would be easier to implement models natively written in that language.&lt;/li&gt;&#xA;&lt;li&gt;R syntax and grammar is extremely complicated. I myself strongly favour R other than anything else but have to however admit that the syntax is not really straightforward and has a very picked learning curve.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The above said, it is still true that one can easily translate R code into any other language, provided methods, libraries and packages are available (in Python most of them are, so that is no problem at all). Plenty of infrastructures and databases support underlying R code, hence portability is not really a problem, especially if one just has to submit the results of the calculations (to that extend, nobody really sees the underlying code anyway). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Java is of almost no use for the pure data science itself (although the Stanford University has a collection of machine learning NLP libraries written in Java, as far as I remember - but please check). The only reason why it can be required is just that the rest of the company uses it to big extents and they do not want to replace it with something new.&lt;/p&gt;&#xA;" OwnerUserId="10372" LastActivityDate="2016-08-18T14:02:30.057" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13527" PostTypeId="2" ParentId="13409" CreationDate="2016-08-18T15:09:27.833" Score="0" Body="&lt;p&gt;Using the notation you gave (i.e. &lt;code&gt;data&lt;/code&gt; is the dataframe and &lt;code&gt;data.class&lt;/code&gt; is the categorical variable you want to process), this can be done this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def cut_levels(x, threshold, new_value):&#xA;    value_counts = x.value_counts()&#xA;    labels = value_counts.index[value_counts &amp;lt; threshold]&#xA;    x[np.in1d(x, labels)] = new_value&#xA;&#xA;cut_levels(data.class, 30, 'others')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this modifies the original data, thus if you want to keep it, do a copy of it before or change the code to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def cut_levels(x, threshold, new_value):&#xA;    x = x.copy()&#xA;    value_counts = x.value_counts()&#xA;    labels = value_counts.index[value_counts &amp;lt; threshold]&#xA;    x[np.in1d(x, labels)] = new_value&#xA;    return x&#xA;&#xA;new_class = cut_levels(data.class, 30, 'others')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23283" LastActivityDate="2016-08-18T15:09:27.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13529" PostTypeId="1" AcceptedAnswerId="13532" CreationDate="2016-08-18T15:56:46.767" Score="2" ViewCount="458" Body="&lt;p&gt;For better understand of neural networks I started implementation of Multi Layer Perceptron. For now I'm implemented single Perceptron that resolve XOR problem. From this point I want start build MLP but I'm not sure if I correctly&#xA; understand MLP structure. Assume I have data instance with 3 attributes and 2 classes and 5 input perceptrons. Should I give every perceptron same input and every output of perceptron should go to every hidden layer perceptron's input? I attach neural net pic.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Dot5a.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Dot5a.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23537" LastActivityDate="2016-08-18T20:00:35.733" Title="Input for individual perceptron in input layer in MLP" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13531" PostTypeId="2" ParentId="13525" CreationDate="2016-08-18T17:58:27.360" Score="3" Body="&lt;p&gt;You should be able to translate code written in one language -- even pseudo-code -- to another, so I see no reason to avoid books for R. If you want one specifically for python, there's &lt;em&gt;Machine Learning in Action&lt;/em&gt; by Peter Harrington.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of scikit-learn's core committers is a releasing a book in October: &lt;a href=&quot;http://shop.oreilly.com/product/0636920030515.do&quot; rel=&quot;nofollow&quot;&gt;Introduction to Machine Learning with Python: A Guide for Data Scientists&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-18T17:58:27.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13532" PostTypeId="2" ParentId="13529" CreationDate="2016-08-18T20:00:35.733" Score="2" Body="&lt;p&gt;With MLPs you can follow a set of simple rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The number of neurons in your input layer equals the number of features / data instances you have.&lt;/li&gt;&#xA;&lt;li&gt;A single neuron takes only one input (one feature)&lt;/li&gt;&#xA;&lt;li&gt;The output from one neuron can go to multiple neurons in the next layer which may or may not be fully connected.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;May the force be with you ;)&lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2016-08-18T20:00:35.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13533" PostTypeId="2" ParentId="13523" CreationDate="2016-08-18T20:06:38.107" Score="4" Body="&lt;p&gt;I think with GPR, the computational cost increases on an order of N^3, so the amount of time with more data increases drastically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think the current version of sklearn's GPR implements the kind of incremental fit you are expecting. With larger data and iterations, I suggest implementing your own version on a GPU or somethings since GPR implementation is not that difficult.&lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2016-08-18T20:06:38.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13535" PostTypeId="1" CreationDate="2016-08-18T20:35:24.683" Score="1" ViewCount="771" Body="&lt;p&gt;I am looking to perform clustering on bitcoin addresses within the blockchain. I have generated a graph structure of the blockchain with a source address, destination address, value of transaction, in-degree, out-degree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think misunderstanding lies in the fact that this is directed graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could this information be represented as input to the k-means clustering algorithm? Should nodes and their corresponding features be the input to k-means alone and should the directed (source/destination) element be removed from the input data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;source address | destination address | transaction value | timestamp | in-degree | out-degree | pagerank | tags&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;source and destination&lt;/code&gt; addresses are just a public key of a sender and receiver of the bitcoin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;transaction value&lt;/code&gt; would be the number of bitcoins you would want to send.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;in-degree&lt;/code&gt; would show the number of links (transactions in this case) pointing in to a given node&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;out-degree&lt;/code&gt; would show the number of links (transactions in this case) pointing out of a given node&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pagerank&lt;/code&gt; is a number showing the importance of a node by the number of other important nodes linking to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tags&lt;/code&gt; will be string based descriptions of an addresses known activity. For example, some gambling companies publish their addresses so we can associate transactions involving this address as a gambling transaction.&lt;/p&gt;&#xA;" OwnerUserId="15961" LastEditorUserId="15961" LastEditDate="2016-08-19T10:52:00.160" LastActivityDate="2018-12-06T12:49:05.577" Title="Clustering bitcoin addresses with k-means - how would one prepare input" Tags="|clustering|k-means|preprocessing|" AnswerCount="0" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="13536" PostTypeId="2" ParentId="9945" CreationDate="2016-08-18T22:03:50.100" Score="1" Body="&lt;p&gt;If you use SVM, you can try the &lt;a href=&quot;https://github.com/Danko-Lab/Rgtsvm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rgtsvm&lt;/a&gt; package for GPU which is backwards compatible with the e1071 implementation.&lt;/p&gt;&#xA;" OwnerUserId="23549" LastEditorUserId="29169" LastEditDate="2021-03-11T20:15:33.507" LastActivityDate="2021-03-11T20:15:33.507" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13538" PostTypeId="1" CreationDate="2016-08-19T06:57:32.600" Score="1" ViewCount="52" Body="&lt;p&gt;The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source) &lt;a href=&quot;https://i.stack.imgur.com/lXIfk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lXIfk.png&quot; alt=&quot;Data obtained from sensor at different temperature.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on re-arrangement (temp in deg C)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oymoZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oymoZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case temperature and frequency are independent variables and humidity is dependent on both of them. Humidity seems directly proportional to frequency if analyzed for humidity Vs Frequency and temperature seems inversely proportional to frequency while analyzed Temp Vs Frequency. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to be able to make an equation which gives Humidity upon solving with known temperature and frequency. For example if I input &lt;strong&gt;F = 8179&lt;/strong&gt; and &lt;strong&gt;T = 20.5&lt;/strong&gt;, &lt;strong&gt;H&lt;/strong&gt; should be near to &lt;strong&gt;42%&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried &lt;code&gt;polyfit&lt;/code&gt; and &lt;code&gt;interp2&lt;/code&gt; from MATLAB for frequency vs humidity and got much errors and tried least square method manually but ended getting nothing near. Specifically I do not have idea on modeling for 3 Variable equation. What may be the way for modeling 3 variable equations upon this small sets of data. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="23559" LastActivityDate="2016-08-19T08:48:22.347" Title="Modeling 3 variable equation" Tags="|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13540" PostTypeId="2" ParentId="13538" CreationDate="2016-08-19T08:48:22.347" Score="2" Body="&lt;p&gt;You want to fit the model $H = \beta_0 + \beta_1 T + \beta_2 H + \epsilon$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard way to enter the data to fit this problem is to enter 3 columns, one for T, one for F, one for H.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;T    F     H&lt;br&gt;&#xA;20  8179    40&lt;br&gt;&#xA;30  8044    40&lt;br&gt;&#xA;40  7923    40&lt;br&gt;&#xA;20  8343    60&lt;br&gt;&#xA;30  8250    60&lt;br&gt;&#xA;40  8130    60&lt;br&gt;&#xA;20  8528    80&lt;br&gt;&#xA;30  8430    80&lt;br&gt;&#xA;40  8320    80   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, the model can be fitted with the command:&lt;br&gt;&#xA;lm(H~T+F)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without checking the appropiatemes of the model, the result is:&lt;br&gt;&#xA;Coefficients:&lt;br&gt;&#xA;(Intercept)   $\;\;\;\;\;$         T     $\;\;\;\;\;\;\;\;\;\;\;\;\;$      F&lt;br&gt;&#xA;  -845.4072 $\;\;\;\;\;$    1.1911   $\;\;\;\;\;$    0.1056&lt;br&gt;&#xA;with Rsqr= 0.9958.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When F = 8179 and T = 20.5, the predicted value of H = -845.4072 + 1.1911*20.5 + 0.1056*8179 =  42.39791&lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2016-08-19T08:48:22.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13541" PostTypeId="2" ParentId="13504" CreationDate="2016-08-19T08:53:06.060" Score="2" Body="&lt;p&gt;A large component of data science work in industry is data wrangling. It is quite important to have some basic understanding of data storage systems as you will often have to extract the data you need yourself (unless you work for a large company). Hadoop may not be a necessary skill, but knowing something of relational data stores (SQL) and object storage (No-SQL) will be very useful. A person often needs to be able to process data quickly too, so you will need to know something of optimisations such as indexing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work in python and R (as do most practitioners I know personally), but find that python is easier to deploy in a production environment. Much of the work relies on libraries these days, so it useful to know the language's library landscape (when it comes to project timelines, familiarity with your tools reduces the pressure on yourself and your team immensely). It is quite common that people experiment in jupyter/ipython notebooks and make their code available online (e.g. &lt;a href=&quot;http://www.kdnuggets.com/2016/04/top-10-ipython-nb-tutorials.html&quot; rel=&quot;nofollow&quot;&gt;KDNuggets post&lt;/a&gt;). We often use notebooks at work and commit them to the code-base to provide the empirical backing for the solution. I would recommend you find some cool notebooks that interface with a database, and see if you can run their code (it is python typically). At least this way you will start to get a feel for more of the grunt work associated with the tools (considering that you already have the theoretical background).&lt;/p&gt;&#xA;" OwnerUserId="13821" LastActivityDate="2016-08-19T08:53:06.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13542" PostTypeId="1" AcceptedAnswerId="13546" CreationDate="2016-08-19T09:07:39.483" Score="0" ViewCount="48" Body="&lt;p&gt;To measure the performance of a classification algorithms on a dataset that has an attribute for class type, I divide my dataset to training and test samples and then create a &lt;code&gt;confusion matrix&lt;/code&gt; for &lt;code&gt;False Positive&lt;/code&gt;, &lt;code&gt;False Negative&lt;/code&gt;, &lt;code&gt;True Positive&lt;/code&gt; and &lt;code&gt;True Negative&lt;/code&gt; samples. Hence there is a class type attribute(e.g&lt;code&gt;Yes&lt;/code&gt; or &lt;code&gt;No&lt;/code&gt; ), the confusion matrix is pretty easy to calculate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that I have a dataset that lacks a class type attribute and all samples are of the class &lt;code&gt;Yes&lt;/code&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I measure the performance of different classification algorithms using these kind of datasets?&lt;/p&gt;&#xA;" OwnerUserId="23564" LastActivityDate="2016-08-19T10:36:29.293" Title="Measuring performance of different classifiers without class type in data" Tags="|classification|performance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13543" PostTypeId="2" ParentId="13513" CreationDate="2016-08-19T09:34:23.790" Score="-2" Body="&lt;p&gt;The tools in Python are just better than R. Ther R community is pretty stagnant while the Python community is evolving really quick. Especially in tools for Data Science.&lt;br&gt;&#xA;Also Python works way easier with everything around it. You can easily scrape the web, connect to databases and so on. That makes prototyping really fast.&lt;br&gt;&#xA;And if you have a working prototype and care to make it faster or integrate it into the company workflow, it gets usually reimplemented in Java.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R has a few neat tools and visualization but it is not that great to build new stuff in it.&lt;/p&gt;&#xA;" OwnerUserId="23565" LastActivityDate="2016-08-19T09:34:23.790" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13545" PostTypeId="2" ParentId="13504" CreationDate="2016-08-19T10:31:04.027" Score="2" Body="&lt;p&gt;If you find learning SQL and Hadoop unbearably boring, you should not be looking for a data scientist job. Anyhow, feel free to skip Hadoop. There are lots of deployments of Hadoop, but they are being phased out with more modern tech, for example Spark, which is also what most companies use for new deployments. I hope you find learning Spark a bit more bearable. The Edx online courses on ML with Spark are actually very nice.&lt;/p&gt;&#xA;" OwnerUserId="23562" LastActivityDate="2016-08-19T10:31:04.027" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13546" PostTypeId="2" ParentId="13542" CreationDate="2016-08-19T10:36:29.293" Score="1" Body="&lt;p&gt;Normal classification will not work in this case, classifiers learn functions that are able to seperate the training examples. In your case you only have one class which will not work for all the classical examples. There is a machine learning task called One-class classification, see this &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page&lt;/a&gt;. In your case PU Learning (Positive and Unlabeled) seems most appropriate. Evaluating the performance is extremely difficult without having some negative examples as well however. See &lt;a href=&quot;https://stats.stackexchange.com/questions/223956/how-are-performance-measures-affected-in-pu-learning&quot;&gt;this question and answer&lt;/a&gt; on the Stats stackexchange.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-19T10:36:29.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13547" PostTypeId="2" ParentId="13513" CreationDate="2016-08-19T10:56:09.163" Score="0" Body="&lt;p&gt;My point of view as a general purpose programmer with a tiny bit of R experience:&#xA;R is excellent for data science, but it's geared towards people manually interpreting data. If you want to use the results for something automated, you have to interface with something else, and that something else will be hard to do in a problem specific language like R. Can you do a web site in R? :)&#xA;On the other hand, python does have ready made libraries for data sciency stuff and is a general purpose programming language that doesn't get in the way of your doing anything else with it.&#xA;As for Java, it's good for large programming projects with hundreds of thousands to millions of lines of code. If the data science part needs to interface with that, it may make sense to do everything in Java then.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random whine: Why do I have to sign in to each StackExchange site separately?&lt;/p&gt;&#xA;" OwnerUserId="23568" LastActivityDate="2016-08-19T10:56:09.163" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13548" PostTypeId="1" CreationDate="2016-08-19T11:06:25.533" Score="0" ViewCount="933" Body="&lt;p&gt;As per my idea unsupervised learning clustering algorithms like k-means, HCA is used for recommendations stuff. I just wanted to know what are advanced algorithms used for this type of work in social network (fb,twitter) &amp;amp; e-commerce websites. Are classification algorithms used for this kind of work?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="20585" LastEditDate="2016-09-08T03:58:16.223" LastActivityDate="2016-09-08T03:58:16.223" Title="what ml algorithms are used for friend suggestions or item recommendations?" Tags="|machine-learning|classification|data-mining|clustering|k-means|" AnswerCount="1" CommentCount="0" ClosedDate="2016-08-19T22:20:35.290" ContentLicense="CC BY-SA 3.0" />
  <row Id="13549" PostTypeId="1" CreationDate="2016-08-19T12:00:17.440" Score="0" ViewCount="569" Body="&lt;p&gt;I'm using KMeans of scikit-learn for clustering of forums.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The attribute inertia_ of model gives the within class inertia but how can i get the between class inertia? In other word how can I verify the quality of my clustering?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="23571" LastEditorUserId="25218" LastEditDate="2016-11-18T14:27:17.770" LastActivityDate="2016-11-18T14:27:17.770" Title="Kmeans: Between class intertia" Tags="|python|clustering|scikit-learn|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13550" PostTypeId="1" AcceptedAnswerId="13551" CreationDate="2016-08-19T12:16:40.093" Score="2" ViewCount="21501" Body="&lt;p&gt;I am learning python and trying myself out at mashine learning. I am reproducing a super simple example - based on the infamous iris dataset. Here it goes: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;iris = datasets.load_iris()&#xA;&#xA;X = iris.data&#xA;y = iris.target&#xA;&#xA;from sklearn.cross_validation import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .5)&#xA;&#xA;from sklearn import tree&#xA;nordan_tree = tree.DecisionTreeClassifier()&#xA;&#xA;nordan_tree.fit(X_train, y_train)&#xA;&#xA;from sklearn.metrics import accuracy_score &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following error message: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;tree3.py&quot;, line 17, in &amp;lt;module&amp;gt;&#xA;    print(accuracy_score(y_test, predictions))&#xA;NameError: name 'predictions' is not defined&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't get it. As far as I understand, &lt;code&gt;predictions&lt;/code&gt; is the vector containing all the predictions produced with  &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;? What am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="23452" LastActivityDate="2018-08-30T15:26:17.690" Title="Problems with accuracy.score sklearn" Tags="|python|scikit-learn|error-handling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13551" PostTypeId="2" ParentId="13550" CreationDate="2016-08-19T12:20:07.457" Score="2" Body="&lt;p&gt;You have not defined the variable &lt;code&gt;predictions&lt;/code&gt; anywhere. You will need to get them from your classifier somehow. You have fit your &lt;code&gt;nordan_tree&lt;/code&gt; on your training data, now you can use the fitted &lt;code&gt;nordan_tree&lt;/code&gt; to generate the predictions, for example like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predictions = nordan_tree.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then your line of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(accuracy_score(y_test, predictions))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should work.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-19T12:20:07.457" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13553" PostTypeId="1" CreationDate="2016-08-19T13:41:52.247" Score="1" ViewCount="2042" Body="&lt;p&gt;I have a dataset like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; id color  body  eyes&#xA; 1  A  blue  slim green&#xA; 2  B black   fat  blue&#xA; 3  A black  slim black&#xA; 4  C green  slim  blue&#xA; 5  D black medim black&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;whereas each id represents an individual with his individual physical characteristics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reproducible:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; structure(list(id = structure(c(1L, 2L, 1L, 3L, 4L), .Label = c(&quot;A&quot;, &#xA;&quot;B&quot;, &quot;C&quot;, &quot;D&quot;), class = &quot;factor&quot;), color = structure(c(2L, 1L, &#xA;1L, 3L, 1L), .Label = c(&quot;black&quot;, &quot;blue&quot;, &quot;green&quot;), class = &quot;factor&quot;), &#xA;body = structure(c(3L, 1L, 3L, 3L, 2L), .Label = c(&quot;fat&quot;, &#xA;&quot;medim&quot;, &quot;slim&quot;), class = &quot;factor&quot;), eyes = structure(c(3L, &#xA;2L, 1L, 2L, 1L), .Label = c(&quot;black&quot;, &quot;blue&quot;, &quot;green&quot;), class = &quot;factor&quot;)),&#xA; .Names = c(&quot;id&quot;, &#xA;&quot;color&quot;, &quot;body&quot;, &quot;eyes&quot;), class = &quot;data.frame&quot;, row.names = c(NA, &#xA;-5L))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then number of the characteristics is fixed (color: blue/black/green, body: slim/fat/medium, eyes: green/blue/black).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim is to cluster those individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My conceptual question regards the approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A simple correlation could be a first step. A question could be: how the combination of these characteristic is likely to appear in groups of individuals?&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;A more complicated approach. Maybe k-means clustering. How can address this given that these are categorical variables? should I convert them into dummies?&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm new to this kind of analysis and any hint/reference to the implementation in R is highly appreciated!&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="23574" LastEditorUserId="8501" LastEditDate="2016-11-18T14:27:38.310" LastActivityDate="2016-12-18T20:11:10.357" Title="How to implement clustering on categorical feature vectors in R?" Tags="|r|clustering|k-means|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13554" PostTypeId="2" ParentId="13513" CreationDate="2016-08-19T13:48:22.923" Score="7" Body="&lt;p&gt;I've seen quite a few companies using the title Data Scientist for &quot;Data Engineer&quot; type roles. Particularly in the big data space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the company is using Hadoop or a distributed framework like Spark to do it's analytics in then Java or Python (or probably Scala) would be the languages that would make the most sense .   &lt;/p&gt;&#xA;" OwnerUserId="23578" LastActivityDate="2016-08-19T13:48:22.923" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13555" PostTypeId="2" ParentId="13513" CreationDate="2016-08-19T13:49:35.683" Score="4" Body="&lt;p&gt;&lt;strong&gt;Java&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd have to disagree with the other posters on the java question. There are certain noSQL databases (like hadoop) that one needs to &lt;a href=&quot;https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html&quot; rel=&quot;nofollow&quot;&gt;write mapreduce jobs in java&lt;/a&gt;. Now you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Apache_Hive&quot; rel=&quot;nofollow&quot;&gt;HIVE&lt;/a&gt; to achieve much the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The python / R debate continues. Both are extensible languages, so potentially both could have the same ability to process. I only know R and my python knowledge is quite superficial. Speaking as a small business owner, you want to not have too many tools in your business otherwise there will be a general lack of depth in them, and difficulty supporting them. I think it will come down to depth of tool knowledge in the team. If the team is focused on python, then hiring another python data scientist is going to make sense as they can engage with the existing code base and historic experiment code.&lt;/p&gt;&#xA;" OwnerUserId="16284" LastActivityDate="2016-08-19T13:49:35.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13557" PostTypeId="1" AcceptedAnswerId="13562" CreationDate="2016-08-19T14:53:27.307" Score="3" ViewCount="4124" Body="&lt;p&gt;I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the code of RMSLE used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def evalerror(preds, dtrain): # this is compatible with DMatrix&#xA;    labels = dtrain.get_label()&#xA;    assert len(preds) == len(labels)&#xA;    labels = labels.tolist()&#xA;    preds = preds.tolist()&#xA;    terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]&#xA;    return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the parameters of XGBoost used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }&#xA;bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this is the evaluation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0] eval-error:0.836219 train-error:0.835095&#xA;Multiple eval metrics have been passed: 'train-error' will be used for early stopping.&#xA;&#xA;Will train until train-error hasn't improved in 20 rounds.&#xA;[1] eval-error:0.809301 train-error:0.806747&#xA;[2] eval-error:0.792647 train-error:0.78908&#xA;[3] eval-error:0.803355 train-error:0.798805&#xA;[4] eval-error:0.803261 train-error:0.79835&#xA;[5] eval-error:0.809352 train-error:0.804283&#xA;[6] eval-error:0.810453 train-error:0.805126&#xA;[7] eval-error:0.811059 train-error:0.805646&#xA;[8] eval-error:0.815261 train-error:0.809722&#xA;[9] eval-error:0.820237 train-error:0.814521&#xA;[10]    eval-error:0.823378 train-error:0.817408&#xA;[11]    eval-error:0.824981 train-error:0.81868&#xA;[12]    eval-error:0.826607 train-error:0.820176&#xA;[13]    eval-error:0.827813 train-error:0.821358&#xA;[14]    eval-error:0.827625 train-error:0.821007&#xA;[15]    eval-error:0.823347 train-error:0.816547&#xA;[16]    eval-error:0.824362 train-error:0.81752&#xA;[17]    eval-error:0.82529  train-error:0.818321&#xA;[18]    eval-error:0.824621 train-error:0.817463&#xA;[19]    eval-error:0.824103 train-error:0.816766&#xA;[20]    eval-error:0.814759 train-error:0.807234&#xA;[21]    eval-error:0.807961 train-error:0.800186&#xA;[22]    eval-error:0.808398 train-error:0.800246&#xA;Stopping. Best iteration:&#xA;[2] eval-error:0.792647 train-error:0.78908&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It may be the case that I need to adjust my objective function to this evaluation metric? &lt;/p&gt;&#xA;" OwnerUserId="23492" LastEditorUserId="23492" LastEditDate="2016-08-20T21:02:43.277" LastActivityDate="2016-08-20T21:02:43.277" Title="XGBoost increase the error when changing evaluation function" Tags="|xgboost|model-evaluations|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13558" PostTypeId="2" ParentId="13553" CreationDate="2016-08-19T17:13:29.703" Score="0" Body="&lt;p&gt;To start with, an initial brute force method would be to perform a mere count of similar occurrences to have a flavour of which combinations of features appear how many times. This can be easily one using the &lt;code&gt;.N&lt;/code&gt; operator provided by the &lt;code&gt;data.table&lt;/code&gt; package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;counts &amp;lt;- df[, .N, by = c(&quot;first_column&quot;, &quot;second_column&quot;)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once so, you could calculate the correlation between categorical variables using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Test_of_independence&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pearson's chi-squared&lt;/a&gt; test of independence (available in &lt;code&gt;R&lt;/code&gt; as &lt;code&gt;chisq.test(x,y, ...)&lt;/code&gt;) that is an initial standard approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to perform the clustering you have to introduce a distance method that allows you to decide how close two points (whose coordinates are the categorical variables at hand) are. There are plenty of methods to assign distances to nominal variables, as in your case, each of which adapts to the use case one is dealing with (dummy variables assignments, dimensional reduction or simply similarity measures, i. e. the variables are either the same or they are not, the former case corresponding to distance 0, the latter to distance 1). There is a good working example in &lt;a href=&quot;https://stackoverflow.com/a/2983763/5017267&quot;&gt;this other answer&lt;/a&gt;. After having introduced a suitable distance you can perform the $k$-means clustering or any other type of clustering that most suits you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very precise walkthrough is provided in &lt;a href=&quot;https://datascience.stackexchange.com/a/898/10372&quot;&gt;this detailed answer&lt;/a&gt; (with examples and references).&lt;/p&gt;&#xA;" OwnerUserId="10372" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-19T17:13:29.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13559" PostTypeId="2" ParentId="13504" CreationDate="2016-08-19T17:56:42.163" Score="1" Body="&lt;h2&gt;SQL / Big Data Tools&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Getting and cleaning data is a central part of a data scientist's job. My team (data science at a mid-to-large internet company) regularly deals with data wrangling at scale -- combining terabytes of browsing history with a multitude of other data sources, structured and unstructured, to investigate problems. SQL and Spark/Hive/Hadoop are nearly daily parts of our workflow. We filter new candidates heavily on their ability to use SQL intelligently and favor those who have worked with big data technologies in production environments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note, distributed processing/storage is a fascinating research area and the mechanics behind databases are pretty cool. Perhaps you can make it a bit more interesting for yourself by not just focusing on the semantics of SQL, but also some under-the-hood bits, like multi-pass algorithms for joins and so forth. You could read the &lt;a href=&quot;http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf&quot; rel=&quot;nofollow&quot;&gt;Amazon DynamoDB paper&lt;/a&gt; or &lt;a href=&quot;http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf&quot; rel=&quot;nofollow&quot;&gt;Google's BigTable paper&lt;/a&gt; for an intro to big, distributed databases. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Programming environment&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Our team is python heavy, though we are using Spark more and more. It's got a great ecosystem for working with big data and has great machine-learning support via scikit-learn and the like. We often prototype in jupyter notebooks and scale solutions by building python apis that run our jobs or handle incoming requests. Other data science teams here use R. Pick one, do a few projects in it, and do a few projects in the other. You'll figure out which syntax and style works best for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to train yourself not to code like a grad student anymore, either. :) &lt;/p&gt;&#xA;" OwnerUserId="11098" LastActivityDate="2016-08-19T17:56:42.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13560" PostTypeId="2" ParentId="13548" CreationDate="2016-08-19T18:08:27.260" Score="2" Body="&lt;p&gt;Welcome to DataScience.SE! The best way to get up to speed on a problem is to read the latest surveys. Here are some I found with a quick search:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://paper.sciencenet.cn/upload/news/file/2012/10/20121071511108170.pdf&quot; rel=&quot;nofollow&quot;&gt;Recommender Systems&lt;/a&gt; (&lt;a href=&quot;http://doc.rero.ch/record/30535&quot; rel=&quot;nofollow&quot;&gt;alternative link&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://romisatriawahono.net/lecture/rm/survey/information%20retrieval/Bobadilla%20-%20Recommender%20Systems%20-%202013.pdf&quot; rel=&quot;nofollow&quot;&gt;Recommender systems survey&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0957417412002825&quot; rel=&quot;nofollow&quot;&gt;A literature review and classification of recommender systems research&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://icdic.org/media/ckeditor_uploads/buaa/2013/08/17/Link%20Prediction%20in%20Complex%20Networks_A%20Survey.pdf&quot; rel=&quot;nofollow&quot;&gt;Link prediction in complex networks: A survey&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.semanticscholar.org/paper/A-Survey-of-Link-Prediction-in-Social-Networks-Hasan-Zaki/e7d30fefe1b99c21813873f976e46d03dc82b4fc/pdf&quot; rel=&quot;nofollow&quot;&gt;A survey of link prediction in social networks&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find more by searching for &quot;recommender systems&quot; or &quot;link prediction&quot;.&#xA;If you have any specific questions after reading them we can help but there is no way we can do justice to a broad question like the one in your title when the articles take dozens of pages to answer the same question.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-08-19T18:35:46.527" LastActivityDate="2016-08-19T18:35:46.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13562" PostTypeId="2" ParentId="13557" CreationDate="2016-08-19T21:27:34.457" Score="6" Body="&lt;p&gt;If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use &lt;code&gt;reg:linear&lt;/code&gt; as objective (which is the default) and &lt;code&gt;rmse&lt;/code&gt; as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dtrain = DMatrix(X_train, label=np.log1p(y_train))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;np.log1p(x)&lt;/code&gt; is equal to &lt;code&gt;np.log(x+1)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you want to make your prediction in the original space, you will need to compute the inverse transform of &lt;code&gt;np.log1p&lt;/code&gt;, that is to say &lt;code&gt;np.expm1&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predictions = np.expm1(bst.predict(dtest))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.&lt;/p&gt;&#xA;" OwnerUserId="23283" LastEditorUserId="23283" LastEditDate="2016-08-20T07:17:52.860" LastActivityDate="2016-08-20T07:17:52.860" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13563" PostTypeId="2" ParentId="13549" CreationDate="2016-08-19T21:42:19.450" Score="1" Body="&lt;p&gt;Total variance = within-class variance + between-class variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. if you compute the total variance once, you can get the between class inertia simply by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;between-class variance = total variance - within-class variance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-08-19T21:42:19.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13564" PostTypeId="1" CreationDate="2016-08-19T22:28:33.060" Score="2" ViewCount="137" Body="&lt;p&gt;I have two columns in my dataset:  &lt;code&gt;Health&lt;/code&gt; and &lt;code&gt;Weight&lt;/code&gt;,  both being of numeric type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Health&amp;lt;-number of days when health is not good,&#xA;    Weight&amp;lt;-weight &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All I want to check if there  is a relationship between Health and Weight. In other words, does a &lt;code&gt;Weight&lt;/code&gt; increase yield an increase of the number of days  where &lt;code&gt;Health&lt;/code&gt; is not good, or the  converse?  I just want to check the relation between those two columns in dataset by plotting some graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here my sample dataset: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| Health     | Weight      | &#xA;|:-----------|------------:|&#xA;| 0          |      30     |        &#xA;| 3          |      63     |        &#xA;| 2          |      31     |          &#xA;| 10         |      169    |            &#xA;| 1          |      9      |    &#xA;|0           |     139     |   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21392" LastEditorUserId="471" LastEditDate="2016-08-30T15:53:35.570" LastActivityDate="2023-06-15T23:09:27.737" Title="How do I explain the relationship between Health and Weight?" Tags="|r|dataset|visualization|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13567" PostTypeId="1" AcceptedAnswerId="13575" CreationDate="2016-08-20T06:51:26.563" Score="33" ViewCount="27364" Body="&lt;p&gt;I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform normalization on the other features but how do I approach latitude/longitude features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: This is a problem to predict agriculture yield. I would think lat/long is very important since locations can be vital in prediction and hence the dilemma.&lt;/p&gt;&#xA;" OwnerUserId="23600" LastEditorUserId="23600" LastEditDate="2016-08-20T18:56:59.550" LastActivityDate="2017-03-02T19:08:25.743" Title="Ways to deal with longitude/latitude feature" Tags="|machine-learning|python|feature-engineering|feature-scaling|normalization|" AnswerCount="1" CommentCount="7" ClosedDate="2016-08-20T17:20:30.537" ContentLicense="CC BY-SA 3.0" />
  <row Id="13568" PostTypeId="2" ParentId="13557" CreationDate="2016-08-20T08:42:30.973" Score="3" Body="&lt;p&gt;While using RMSLE, you should pay attention to the point that the metric penalises the under predicted values more than the over predicted values, hence driving the the model towards high bias. This might be changing the weights of a model and hence increasing error after second iteration.&#xA;&lt;a href=&quot;https://www.kaggle.com/wiki/RootMeanSquaredLogarithmicError&quot; rel=&quot;nofollow&quot;&gt;you can go to the following link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23380" LastEditorUserId="23380" LastEditDate="2016-08-20T10:59:32.190" LastActivityDate="2016-08-20T10:59:32.190" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13570" PostTypeId="2" ParentId="13564" CreationDate="2016-08-20T09:26:49.980" Score="0" Body="&lt;p&gt;You could perform unsupervised clustering on the data(k-means), this will give you relationships like weights of the people whose health is not good for a particular number of days.&lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2016-08-20T09:26:49.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13571" PostTypeId="2" ParentId="11921" CreationDate="2016-08-20T09:48:04.110" Score="2" Body="&lt;p&gt;I think mxnet is one of the best options if you code in R. They have an &lt;a href=&quot;https://github.com/dmlc/mxnet/tree/master/R-package&quot; rel=&quot;nofollow&quot;&gt;R wrapper&lt;/a&gt; but the core is in C++.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have several examples in the web. One of them is the &lt;a href=&quot;https://github.com/dmlc/mxnet/blob/master/R-package/vignettes/mnistCompetition.Rmd&quot; rel=&quot;nofollow&quot;&gt;character recognition&lt;/a&gt; with MNIST database. They have support for multi-gpus and also for Spark.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-08-20T09:48:04.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13572" PostTypeId="2" ParentId="13564" CreationDate="2016-08-20T10:28:16.340" Score="2" Body="&lt;p&gt;I second Arun Aniyan's answer. Look at how the two features are related to each other by computing  Pearson's correlation coefficient. Another option would be to visualize your data by plotting a scatterplot&lt;/p&gt;&#xA;" OwnerUserId="14360" LastActivityDate="2016-08-20T10:28:16.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13573" PostTypeId="1" AcceptedAnswerId="13577" CreationDate="2016-08-20T11:01:09.573" Score="1" ViewCount="1047" Body="&lt;p&gt;In Keras, the &lt;code&gt;border_mode = 'valid'&lt;/code&gt; doesn't zero pad the input. Thus, we subsequently get an output feature map that is not the same size as the input. Likewise, setting &lt;code&gt;border_mode = 'same'&lt;/code&gt; gives output feature maps as the input. My question is if we set &lt;code&gt;border_mode = 'same'&lt;/code&gt;, should we also perform zero padding using the &lt;code&gt;zero_padding&lt;/code&gt; function?&lt;/p&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="8820" LastEditDate="2016-08-24T09:29:04.480" LastActivityDate="2016-08-24T09:29:04.480" Title="Similarity between bordermode and zero padding in Keras" Tags="|machine-learning|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13574" PostTypeId="1" CreationDate="2016-08-20T11:08:40.737" Score="1" ViewCount="120" Body="&lt;p&gt;I have a data set of persons with attribute job that have 300 different nominal value.&#xA;attribute job have a lot of noise. I decide to cluster my data base on other attribute (other feature of person) that related to their job. I hope I reach to clusters with high correlation to job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to know, how much I were successful. In my opinion, more correlation between job and the clusters means more success. I found chi-squared as a method to check correlation between two nominal attributes. &#xA;But the result is very low p-value for all clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any method to evaluate a clustering with a predefined class? &lt;/p&gt;&#xA;" OwnerUserId="15569" LastActivityDate="2016-08-21T20:57:11.547" Title="How to evaluate clusters base on an attribute of the dataset?" Tags="|clustering|model-evaluations|" AnswerCount="1" CommentCount="1" ClosedDate="2016-08-22T09:38:58.380" ContentLicense="CC BY-SA 3.0" />
  <row Id="13575" PostTypeId="2" ParentId="13567" CreationDate="2016-08-20T14:09:22.793" Score="39" Body="&lt;p&gt;Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points in these 3 dimensions are also close in reality. Depending on the use case you can disregard the changes in height and map them to a perfect sphere. These features can then be standardized properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify (summarised from the comments):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = cos(lat) * cos(lon)&#xA;y = cos(lat) * sin(lon), &#xA;z = sin(lat) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="29238" LastEditDate="2017-03-02T19:08:25.743" LastActivityDate="2017-03-02T19:08:25.743" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13576" PostTypeId="1" CreationDate="2016-08-20T14:11:33.757" Score="3" ViewCount="1047" Body="&lt;p&gt;Looking for assistance kick-starting a new machine learning scenario. In this case I need to pair one entity (ex. person) with a group of entities (ex. other people) given a history of matching patterns over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Assume we have people who like other people with a combination of traits such as height, weight, education level, etc... This notation of 'like' is bi-directional when calculating probabilities. When a new person arrives I'd like to isolate a group of people (ranked by probability) to recommend them too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case the input would be a single person with all their traits and the output would be a list of people with ranked probabilities of matching. There seems to be two main sets of data&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Known people  &lt;/li&gt;&#xA;&lt;li&gt;Traits known people have liked or not with the traits of people they have liked or not&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am relatively new so not sure which types of training model to use. For example, is this something that can be accomplished most reasonable with a linear regression model, binary classification, boost-tree's, matchbox, etc, etc, etc..? Possibly I need a combination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the Azure Machine Learning studio which has lots of options but I am just not sure where to start in this scenario. Any suggestions or pointers would be appreciated in helping &lt;/p&gt;&#xA;" OwnerUserId="23611" LastActivityDate="2017-03-18T22:20:13.137" Title="Selecting the right algorithm for match probability prediction" Tags="|machine-learning|classification|probability|scoring|azure-ml|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13577" PostTypeId="2" ParentId="13573" CreationDate="2016-08-20T16:45:59.940" Score="1" Body="&lt;p&gt;I think I found the answer to this here &lt;a href=&quot;https://github.com/fchollet/keras/issues/1984&quot; rel=&quot;nofollow&quot;&gt;https://github.com/fchollet/keras/issues/1984&lt;/a&gt;. If the stride is 1, border_mode = 'same' does the job of padding to ensure that the output feature maps are the same size as the input. &lt;/p&gt;&#xA;" OwnerUserId="14360" LastActivityDate="2016-08-20T16:45:59.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13578" PostTypeId="2" ParentId="13387" CreationDate="2016-08-20T16:52:12.980" Score="1" Body="&lt;p&gt;Deep learning is a fancy thing now in ML since it has been outperforming other ML algorithms in many respects. Convolutional neural networks is one of the methods to implement Deep learning and it is highly applicable to different data types such as images, signals (time series) and text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mainly use CNN for images and signals. In my application I have done binary and multiclass classification. The theoretical background to both is the same. Depending on the problem and data at hand you may want to break your multiclass classification to simple binary ones and the combine them at a later stage. These are things that comprise model selection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying deep (neural) networks to a feature dataset can be thought in two aspects. One is applying to text data after vectorizing as explained by @HimaVarsha.  I have not worked with text data, but my applications has mainly coming from images ,signals or catalogues (astronomy catalogues). I have tried using catalogue data which is basically numerical features (2nd aspect) in deep neural networks but did not get significant improvement in results as  made the network more deep -- it can be either the problem with my design or maybe the features itself. (Also this method is not often considered a way of learning in deep learning, because deep learning is more of learning directly from the data. But it's another philosophical question)&#xA;With convolutional neural networks I have got nice results with deeper layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the blog link in the first answer I would also suggest &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;. &lt;a href=&quot;https://blog.keras.io/&quot; rel=&quot;nofollow&quot;&gt;Keras blog&lt;/a&gt; also recommended if you would like to look into CNNs.&lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2016-08-20T16:52:12.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13579" PostTypeId="2" ParentId="13576" CreationDate="2016-08-20T16:56:03.840" Score="1" Body="&lt;p&gt;It seems a variant of Movie Recommendation problem. Take Netflix for example, they have  user ratings for different type of movies; in your case users are telling what attributes they like. When a new user joins and rates the things he/she likes, you will find the people with similar interests, again with Netflix analogy it means recommending movies based on people with same tastes as the new user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read on &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow&quot;&gt;Collaborative filtering&lt;/a&gt;, which is used for building recommendation systems. The reason I drew the Netflix/Movie Recommendation analogy is because you can find a lot of literature on it.&lt;/p&gt;&#xA;" OwnerUserId="12858" LastActivityDate="2016-08-20T16:56:03.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13580" PostTypeId="1" AcceptedAnswerId="13582" CreationDate="2016-08-20T19:19:05.623" Score="0" ViewCount="548" Body="&lt;p&gt;I know how to split the dataset into train and test sets using &lt;code&gt;train_test_split&lt;/code&gt; but is there any way that I can split the dataset into three different sets, i.e., &amp;quot;Train set&amp;quot;, &amp;quot;Test set&amp;quot; and &amp;quot;Validation Set&amp;quot;. An example should be enough.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-26T00:12:16.983" LastActivityDate="2020-12-26T00:12:19.707" Title="Splitting data in scikit-learn" Tags="|machine-learning|python|scikit-learn|dataset|cross-validation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13581" PostTypeId="1" CreationDate="2016-08-20T20:24:15.910" Score="2" ViewCount="1906" Body="&lt;p&gt;I am working on a forecasting problem and came across this issue. How do I forecast sales of a brand new product? For example, a product has been introduced in the store and the store would like to know how the sales will look like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I figured this would be similar to cold start problem in recommender systems. But still I would like to know if there are any standard approaches for such problems.&lt;/p&gt;&#xA;" OwnerUserId="23403" LastActivityDate="2019-02-14T00:01:45.330" Title="How to use ML to forecast sales of a brand new product" Tags="|machine-learning|predictive-modeling|recommender-system|forecast|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13582" PostTypeId="2" ParentId="13580" CreationDate="2016-08-20T20:56:40.073" Score="2" Body="&lt;p&gt;&lt;code&gt;train_test_split&lt;/code&gt; is just a utility function around &lt;code&gt;ShuffleSplit&lt;/code&gt;, which on its turn just randomly assigns each sample to either &lt;code&gt;train&lt;/code&gt; or &lt;code&gt;test&lt;/code&gt;, taking the desired probability into account.&#xA;You can do that however you'd like, and there's no real reason to use that specific function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its not too hard to come up with some code that does that for three values or N values, if you rather avoid calling &lt;code&gt;train_test_split&lt;/code&gt; twice.&lt;/p&gt;&#xA;" OwnerUserId="13714" LastActivityDate="2016-08-20T20:56:40.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13583" PostTypeId="2" ParentId="13109" CreationDate="2016-08-20T21:08:13.717" Score="1" Body="&lt;p&gt;For python (which is my language of preference) I can suggest one of two alternatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, &lt;a href=&quot;http://spark.apache.org/docs/latest/api/python/&quot; rel=&quot;nofollow&quot;&gt;PySpark&lt;/a&gt;. As part of the Spark collection of utilities it is very mature and designed to work with hundreds and thousands of nodes. It is extremely stable and has decent API, especially when empowered with &lt;a href=&quot;http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html&quot; rel=&quot;nofollow&quot;&gt;Spark-MLlib&lt;/a&gt;. Spark has loads of users and resources and comes highly recommended, Nearly the entire industry uses that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If thousands of nodes is not your biggest concern, and you're familiar with Pandas, you might want to consider &lt;a href=&quot;http://dask.pydata.org&quot; rel=&quot;nofollow&quot;&gt;Dask&lt;/a&gt; for a native python solution, together with sklearn. This is a bit &quot;risky&quot; from a corporate's standpoint but I recommend giving it a try.&lt;/p&gt;&#xA;" OwnerUserId="13714" LastActivityDate="2016-08-20T21:08:13.717" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13584" PostTypeId="1" AcceptedAnswerId="13622" CreationDate="2016-08-20T21:57:59.340" Score="3" ViewCount="170" Body="&lt;p&gt;I've built an autonomous sailing robot (&lt;a href=&quot;https://github.com/kolosy/ArduSailor&quot; rel=&quot;nofollow&quot;&gt;https://github.com/kolosy/ArduSailor&lt;/a&gt;). Turns out, the problem of piloting it is fairly complex, and my procedural approach to solving it hasn't worked well (or at all). I think that an ML-based approach may be better, and I'm trying to figure out the right algorithm to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm viewing it as an optimization problem of sorts - I've got a small set of parameters:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Position (lat, lon) &lt;/li&gt;&#xA;&lt;li&gt;Orientation (in 9 DOF)&lt;/li&gt;&#xA;&lt;li&gt;Speed&lt;/li&gt;&#xA;&lt;li&gt;Wind speed &amp;amp; direction&lt;/li&gt;&#xA;&lt;li&gt;Distance to waypoint&lt;/li&gt;&#xA;&lt;li&gt;Heading to waypoint&lt;/li&gt;&#xA;&lt;li&gt;Sail position (basically winch orientation, a single value between 0 and 180)&lt;/li&gt;&#xA;&lt;li&gt;Rudder position (same as above)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I'm thinking about this right, I need to vary my rudder and winch over time in response to my current position, orientation and wind direction to minimize the difference between my orientation and the heading to the waypoint, and minimize the distance to the waypoint. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach right now, is to train an ANN (&lt;a href=&quot;http://robotics.hobbizine.com/arduinoann.html&quot; rel=&quot;nofollow&quot;&gt;using this code&lt;/a&gt;) by recording a manual run through my course. Is this the right approach and algorithm? Is there a better / more suitable way of thinking about this?&lt;/p&gt;&#xA;" OwnerUserId="23628" LastEditorUserId="836" LastEditDate="2017-01-20T20:07:37.587" LastActivityDate="2017-06-19T23:05:14.077" Title="Algorithm selection for controlling a model vehicle" Tags="|machine-learning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13587" PostTypeId="1" AcceptedAnswerId="14224" CreationDate="2016-08-21T09:44:54.333" Score="16" ViewCount="5965" Body="&lt;p&gt;I have a small sub-question to &lt;a href=&quot;https://datascience.stackexchange.com/questions/11699/backprop-through-max-pooling-layers?newreg=a19724e3c92a4b71ad72b7128751da1c&quot;&gt;this question&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that when back-propagating through a max pooling layer the gradient is routed back in a way that the neuron in the previous layer which was selected as max gets all the gradient. What I'm not a 100% sure of is how the gradient in the next layer gets routed back to the pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the first question is if I have a pooling layer connected to a fully connected layer - like the image below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hJUT8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hJUT8.png&quot; alt=&quot;example1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When computing the gradient for the cyan &quot;neuron&quot; of the pooling layer do I sum all the gradients from the FC layer neurons? If this is correct then every &quot;neuron&quot; of the pooling layer has the same gradient?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if the first neuron of FC layer has a gradient of 2, second has a gradient of 3, and third a gradient of 6. What are the gradients of the blue and purple &quot;neurons&quot; in the pooling layer and why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then? See the example below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7N4nM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7N4nM.png&quot; alt=&quot;example2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the topmost rightmost &quot;neuron&quot; of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please do not post a bunch of equations and tell me that my answer is right in there because I've been trying to wrap my head around equations and I still don't understand it perfectly that's why I'm asking this question in a simple way.&lt;/p&gt;&#xA;" OwnerUserId="23640" LastEditorUserId="29169" LastEditDate="2021-02-16T15:40:11.510" LastActivityDate="2021-02-16T15:40:11.510" Title="Back-propagation through max pooling layers" Tags="|machine-learning|neural-network|convolutional-neural-network|backpropagation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13588" PostTypeId="2" ParentId="10256" CreationDate="2016-08-21T14:39:57.857" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;http://datamarket.azure.com/dataset/bing/search&quot; rel=&quot;nofollow&quot;&gt;Bing Search API&lt;/a&gt; mentions that it not be available after Dec 31st 2016. &lt;/p&gt;&#xA;" OwnerUserId="12786" LastActivityDate="2016-08-21T14:39:57.857" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13589" PostTypeId="1" AcceptedAnswerId="13594" CreationDate="2016-08-21T19:09:33.840" Score="1" ViewCount="80" Body="&lt;p&gt;I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python. A typical data set is about 250GB (raw), and requires at least 4 different preprocessing steps before analysis. The finished product typically ends up taking up about 1TB. The goal of my package is to allow the user to pick and choose which existing package to use for each step before running the analysis, and then the program will execute it without further user intervention. Since the goal is to integrate these different packages, written in different languages, I decided to write the program in bash to make it easy to call the actual analysis scripts no matter what language they are written in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The program is starting to come along, but it is getting very complex because of the various idiosyncratic expectations and conventions of each analysis package. I realize bash may not be the most suitable language for complex tasks, but I like that it's easy to call scripts in different languages from there, and that it's relatively simple. The program also does a lot of file handling, which bash is good at. On the other hand, I hear it's also very slow, and it gets clunky when things get more complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if bash is the best choice for this task. Does anyone have suggestions for other languages, or combinations of languages, that might be better suited to my needs? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should note that I am a self-taught programmer and this is my first real programming challenge. I am mostly familiar with bash, matlab, R, and a little bit of python, but I'd like to learning new things too (C maybe?). Also, this is all going to run on unix. &lt;/p&gt;&#xA;" OwnerDisplayName="user23593" LastActivityDate="2016-08-22T11:57:45.070" Title="Which programming languages should I use to integrate different analysis software packages?" Tags="|dataset|programming|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13590" PostTypeId="2" ParentId="11751" CreationDate="2016-08-21T20:30:22.847" Score="5" Body="&lt;p&gt;Have a look at &lt;a href=&quot;https://ipywidgets.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;IPyWidgets&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful.&lt;/p&gt;&#xA;&lt;p&gt;You define a function:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def f(x):&#xA;return x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And you connect it to a widget using the &lt;em&gt;Interact()&lt;/em&gt; function.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;interact(f, x=10);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This generates a widget (a slider in this case) automatically and updates your function with new input when you interact with the widget. The type of widget is determined from the default value you specify (i.e. a checkbox for a boolean, a textbox for a string, etc.)&lt;/p&gt;&#xA;" OwnerUserId="23664" LastEditorUserId="25180" LastEditDate="2021-02-24T21:01:27.220" LastActivityDate="2021-02-24T21:01:27.220" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13591" PostTypeId="2" ParentId="13574" CreationDate="2016-08-21T20:57:11.547" Score="0" Body="&lt;p&gt;Use &lt;em&gt;external&lt;/em&gt; cluster evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, use ARI and NMI.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-08-21T20:57:11.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13592" PostTypeId="1" CreationDate="2016-08-22T08:57:33.617" Score="1" ViewCount="462" Body="&lt;p&gt;I found a script on &lt;a href=&quot;https://www.kaggle.com/c/titanic&quot; rel=&quot;nofollow&quot;&gt;Kaggle's titanic competion&lt;/a&gt; in wich the creator convert Multi-values single feature (namely Pclass = {1,2,3}) to 3 binary features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the pro/cons of such a conversion? Does it really improve something, or is it worse? Or doesn't change anything?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite taking longer to compute, I also worry that it could introduce a bias in the model.&lt;/p&gt;&#xA;" OwnerUserId="23679" LastActivityDate="2016-10-21T12:38:37.800" Title="Multiple binary dummy features Vs Multi-values single feature" Tags="|machine-learning|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13593" PostTypeId="1" CreationDate="2016-08-22T11:34:21.027" Score="1" ViewCount="1008" Body="&lt;p&gt;I'm working with the orange data mining tool. In most of the examples data  taken from files or google sheets links but in the blog &lt;a href=&quot;http://blog.biolab.si/2015/10/19/sql-for-orange/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; they given that we can take data from postgresql database tables also but when follow the steps of blog I'm getting error.I attached the screen shots of error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zuCbn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zuCbn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yTm4z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yTm4z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need help in solving this error.&lt;/p&gt;&#xA;" OwnerUserId="20585" LastActivityDate="2024-01-07T14:00:45.123" Title="Database missing extensions:tym_system_time,quantile orange widget error" Tags="|orange|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13594" PostTypeId="2" ParentId="13589" CreationDate="2016-08-22T11:57:45.070" Score="0" Body="&lt;p&gt;If you are mostly stitching up together calls into other software, like unix utilities (awk, grep, sed ...), python, and matlab scripts, bash &lt;em&gt;is just fine&lt;/em&gt; or possibly even best for the job to construct simple pipelines and workflows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy in bash to read user input, store it in variables, then launch other software depending on the set variables. It's perfectly fast enough for that, and nothing else gets any easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you, however, were to use bash for preprocessing itself, like looping through files line by line, packing and unpacking tab-separated values into arrays etc., that would be excruciatingly slow and not recommended.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-08-22T11:57:45.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13595" PostTypeId="2" ParentId="13592" CreationDate="2016-08-22T12:03:43.200" Score="1" Body="&lt;p&gt;As @K3 suggested, as @Sean Owen told me the name (One Hot Encoding), I can provide at least some elements to answer myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As found on sklearn &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; : &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...]integer representation can not be used directly with scikit-learn&#xA;  estimators, as these expect continuous input, and would interpret the&#xA;  categories as being ordered, which is often not desired.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is implemented in sklearn as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow&quot;&gt;OneHotEncoder&lt;/a&gt; and in Pandas as &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow&quot;&gt;get_dummies&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="23679" LastActivityDate="2016-08-22T12:03:43.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13596" PostTypeId="1" AcceptedAnswerId="13600" CreationDate="2016-08-22T13:54:43.567" Score="5" ViewCount="9597" Body="&lt;p&gt;I have a database with around 30,000 pictures. All of them are a different object. They are all from a certain perspective, the pictures itself are the same size but the objects vary in size. I want to build a system that you can query with a new picture, and that will return it's nearest neighbor, given that it is similar enough. The queried images will look relatively similar to the originals, there might be some horizontal and or vertical translations, a bit different lightning and sometimes there will be a sticker on a different place. Some queried objects will not be in the set, and that needs to be returned as well. What are good techniques to try and what would be the downside? Getting multiple pictures of each object is infeasible. Here's some ideas, I'm wondering if there is more to try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Euclidean distance on the raw data (very sensitive but fast)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use traditional keypoint matching, linear matching is very slow unfortunately&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use (denoising) autoencoder for lower dimensional feature representation, linear match on this encoded space (smallest Euclidean distance, at least faster linear search)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Learn siamese network for linear matching (don't know how fast this works but seems slow too)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Learn deep binary autoencoder onto 28 bits which allows for very quick narrowing of the search space to do one of the previous methods, by using these bits as memory mapping to a list of candidate solutions&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any other ideas?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-22T14:34:19.140" LastActivityDate="2016-08-22T16:34:56.770" Title="What techniques to use for image matching" Tags="|deep-learning|object-recognition|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13597" PostTypeId="1" CreationDate="2016-08-22T14:03:25.350" Score="10" ViewCount="2864" Body="&lt;p&gt;Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction given that there are some features missing in the test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that it is common practice to impute missing data, but why do this when Naïve Bayes should be able to make a prediction given that there are some features missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this be implemented in sci-kit learn? I tried a test set with less features, and got a ValueError as the shapes are not aligned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So theoretically this is possible, but is it possible in scikit learn?&lt;/p&gt;&#xA;" OwnerUserId="17772" LastActivityDate="2017-06-19T00:45:47.050" Title="Naive Bayes Should generate prediction given missing features (scikit learn)" Tags="|scikit-learn|naive-bayes-classifier|missing-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13599" PostTypeId="1" AcceptedAnswerId="13609" CreationDate="2016-08-22T15:10:32.340" Score="3" ViewCount="151" Body="&lt;p&gt;To date, I have done several ad-hoc text clustering projects which use combinations of topic modeling, k-means, and other algorithms. Basically, the point of these projects was to produce themes for different events based on associated text. The themes were named manually after the appropriate levels of clustering were determined, and are now stored in a csv in the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;event_id    majortheme    minortheme         majortheme_id    minortheme_id&#xA;12          Job Failure   TWS Issue          1                major1minor1&#xA;14          Job Failure   TWS Issue          1                major1minor1&#xA;15          Job Failure   Job Abend          1                major1minor2&#xA;16          Access Issue  Unable to Login    2                major2minor1&#xA;17          Access Issue  Unable to Connect  2                major2minor2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to transition from clustering to classification (from descriptive to prescriptive analytics), i.e. being able to take new events (with new &lt;code&gt;event_id&lt;/code&gt;s) and classify them based on previous clusterings. This would be sort of an iterative training dataset, as new classifications would be added to previous clusterings, after verifying that the model hasn't gone completely awry. Using Python, what is the best approach to implement this sort of classification pipeline? Is it as simple as saving my initial clustering results and then just using that data as a training set going forward? Then saving the test predictions to the original training dataset, and so on?&lt;/p&gt;&#xA;" OwnerUserId="13087" LastActivityDate="2016-08-23T08:13:37.030" Title="Transition from clustering to classification?" Tags="|classification|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13600" PostTypeId="2" ParentId="13596" CreationDate="2016-08-22T16:11:57.927" Score="8" Body="&lt;p&gt;Hashing is the way to go if you want fast -- constant time -- retrieval of nearest neighbors. Here's a recent example using neural networks to learn a binary hash:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_cvpr_workshops_2015/W03/papers/Lin_Deep_Learning_of_2015_CVPR_paper.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deep Learning of Binary Hash Codes for Fast Image Retrieval&lt;/a&gt; (&lt;a href=&quot;https://github.com/kevinlin311tw/caffe-cvprw15&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;) (&lt;a href=&quot;http://www.csie.ntu.edu.tw/~r01944012/deepworkshop-slide.pdf&quot; rel=&quot;noreferrer&quot;&gt;slides&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to avoid doing all-pairs computations like correlations.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-08-22T16:34:56.770" LastActivityDate="2016-08-22T16:34:56.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13601" PostTypeId="1" CreationDate="2016-08-22T19:11:14.897" Score="2" ViewCount="2171" Body="&lt;p&gt;I have a time series containing stock price data.&lt;/p&gt;&#xA;&lt;p&gt;I would like to calculate the Money Flow Index (MFI) for each row.&lt;/p&gt;&#xA;&lt;p&gt;Given that the MFI, uses the previous approx. 14 rows to calculate the current MFI, what would be the best approach to doing this?&lt;/p&gt;&#xA;&lt;p&gt;The below calculates the current &lt;a href=&quot;http://www.investopedia.com/terms/m/mfi.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;MFI&lt;/a&gt; for a given DataFrame, but I would like to implement this for each row in the data frame&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Typical Price =(high price + low price + closing price) / 3&#xA;tp=(hst['High']+hst['Low']+hst['Close'])/3    &#xA;tp=tp.to_frame(name='Price')&#xA;&#xA;# Raw money flow = typical price x volume&#xA;tp['Raw']=tp['Price'] * hst['Volume']&#xA;&#xA;# Identify flow (upwards or downwards)&#xA;tp['Direction'] = np.where(np.nan_to_num(tp['Raw'].shift(1))&amp;gt;tp['Raw'], 'up', 'down')&#xA;mfr=0&#xA;# Money flow ratio = (14-day Positive Money Flow) / (14-day Negative Money Flow)&#xA;if tp.loc[tp['Direction']=='down']['Price'].sum(axis=0) != 0:&#xA;    mfr=tp.loc[tp['Direction']=='up']['Price'].sum(axis=0) / tp.loc[tp['Direction']=='down']['Price'].sum(axis=0)&#xA;&#xA;# MFI = 100 - 100 / (1 + money flow ratio)&#xA;100-(100/(1+ mfr))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I managed to work this using the &lt;code&gt;'apply'&lt;/code&gt; function, but this function only allows for 1 argument, whilst I would need to pass more than 1 argument to compute the above.&lt;/p&gt;&#xA;" OwnerUserId="16104" LastEditorUserId="98307" LastEditDate="2020-08-04T08:39:30.643" LastActivityDate="2020-08-04T08:39:30.643" Title="Pandas Data Frame: Calculating custom moving average" Tags="|python|time-series|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13602" PostTypeId="2" ParentId="13601" CreationDate="2016-08-22T19:40:49.747" Score="1" Body="&lt;p&gt;Worked with the below to add a column that calculates the above function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;quote['MFITEST']=quote['Date'].apply(lambda x: calc_mfi(quote,14,x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16104" LastActivityDate="2016-08-22T19:40:49.747" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13603" PostTypeId="2" ParentId="13597" CreationDate="2016-08-22T20:22:52.297" Score="4" Body="&lt;p&gt;Your question is sensible. The way in which posterior probability is calculated in the classical Naive Bayes classifier (in sklearn) is like  summation of the conditional probabilities of the all the features in the dataset. Even though the features are treated as conditionally independent, to learn the classification probability all the features are always used in this setup. Once the model has been learned you still all those features to calculate the posterior for a new observation. The conditional independence is just an assumption that is taken to make the statistics and math obey the rules and work.&lt;/p&gt;&#xA;&lt;p&gt;But slightly modifying the way in which the posterior is calculated you can use Bayesian approach to make predictions even with the absence of certain features. Using Bayesian approach to make predictions in the absence of certain features is still an ongoing work. You may want to have a look at this &lt;a href=&quot;https://arxiv.org/pdf/1211.3607.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; in which Bayesian approach is applied to astronomy to do classification with missing values.&lt;/p&gt;&#xA;&lt;p&gt;Naive Bayes can be easily implemented in python as it is a simple calculation. The sklearn Naive Bayes does not support predictions with missing values and can be tricky to implement to what you are thinking. Maybe writing your own piece of code would be better :)&lt;/p&gt;&#xA;" OwnerUserId="18465" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-08-22T20:22:52.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13605" PostTypeId="1" AcceptedAnswerId="13608" CreationDate="2016-08-23T03:21:36.840" Score="0" ViewCount="165" Body="&lt;p&gt;I can build ARIMA model with regressor to forecast monthly shipment for one product. but I have 300 products and each of them needs a monthly forecast.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;my question is, instead of building 300 models, is there another approach in R to deal with the goal I am trying to accomplish? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you!&lt;/p&gt;&#xA;" OwnerUserId="23699" LastActivityDate="2016-08-23T06:48:34.040" Title="forecast monthly shipment (time series) for 300 products individually in R" Tags="|r|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13606" PostTypeId="2" ParentId="12895" CreationDate="2016-08-23T05:41:59.467" Score="3" Body="&lt;p&gt;Some algorithms, such as SVM or Logistical regression, have possibility to add a weight to certain class, therefore fix the unbalanced issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This really sounds like a job for Survival analysis, which is especially designed to answer questions like &quot;When machine X fail&quot; or &quot;Which attribute influence the most the failure&quot;. You can simply start by plotting the Kaplan-Meier curve and then further stratify it by some attribute. Then you can try Cox regression model - it is useful to see the influence of an attribute on survival - the hazard ratio.&#xA;But don't forget to verify the assumptions (functional form and proportional hazard).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R the Survival analysis is implemented very well, so don't be affraid.&#xA;There is simple and short &lt;a href=&quot;https://rpubs.com/daspringate/survival&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; which might help.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-08-23T05:41:59.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13607" PostTypeId="1" CreationDate="2016-08-23T06:19:59.023" Score="14" ViewCount="39976" Body="&lt;p&gt;I am trying to implement &lt;a href=&quot;http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0137036&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; paper on a set of medical images. I am doing it in Keras. The network essentially consists of 4 conv and max-pool layers followed by a fully connected layer and soft max classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, I have followed the architecture mentioned in the paper. However, the validation loss and accuracy just remain flat throughout. The accuracy seems to be fixed at ~57.5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help on where I might be going wrong would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Activation, Dropout, Dense, Flatten  &#xA;from keras.layers import Convolution2D, MaxPooling2D&#xA;from keras.optimizers import SGD&#xA;from keras.utils import np_utils&#xA;from PIL import Image&#xA;import numpy as np&#xA;from sklearn.utils import shuffle&#xA;from sklearn.cross_validation import train_test_split&#xA;import theano&#xA;import os&#xA;import glob as glob&#xA;import cv2&#xA;from matplotlib import pyplot as plt&#xA;&#xA;nb_classes = 2&#xA;img_rows, img_cols = 100,100&#xA;img_channels = 3&#xA;&#xA;&#xA;#################### DATA DIRECTORY SETTING######################&#xA;&#xA;data = '/home/raghuram/Desktop/data'&#xA;os.chdir(data)&#xA;file_list = os.listdir(data)&#xA;##################################################################&#xA;&#xA;## Test lines&#xA;#I = cv2.imread(file_list[1000])&#xA;#print np.shape(I)&#xA;####&#xA;non_responder_file_list = glob.glob('0_*FLAIR_*.png')&#xA;responder_file_list = glob.glob('1_*FLAIR_*.png')&#xA;print len(non_responder_file_list),len(responder_file_list)&#xA;&#xA;labels = np.ones((len(file_list)),dtype = int)&#xA;labels[0:len(non_responder_file_list)] = 0&#xA;immatrix = np.array([np.array(cv2.imread(data+'/'+image)).flatten() for image in file_list])&#xA;#img = immatrix[1000].reshape(100,100,3)&#xA;#plt.imshow(img,cmap = 'gray')&#xA;&#xA;&#xA;data,Label = shuffle(immatrix,labels, random_state=2)&#xA;train_data = [data,Label]&#xA;X,y = (train_data[0],train_data[1])&#xA;# Also need to look at how to preserve spatial extent in the conv network&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=4)&#xA;X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols)&#xA;X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols)&#xA;X_train = X_train.astype('float32')&#xA;X_test = X_test.astype('float32')&#xA;&#xA;X_train /= 255&#xA;X_test /= 255&#xA;&#xA;Y_train = np_utils.to_categorical(y_train, nb_classes)&#xA;Y_test = np_utils.to_categorical(y_test, nb_classes)&#xA;&#xA;model = Sequential()&#xA;&#xA;## First conv layer and its activation followed by the max-pool layer#&#xA;model.add(Convolution2D(16,5,5, border_mode = 'valid', subsample = (1,1), init = 'glorot_normal',input_shape = (3,100,100))) # Glorot normal is similar to Xavier initialization&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size = (2,2),strides = None))&#xA;# Output is 48x48&#xA;&#xA;print 'First layer setup'&#xA;###########################Second conv layer#################################&#xA;model.add(Convolution2D(32,3,3,border_mode = 'same', subsample = (1,1),init = 'glorot_normal'))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.6))&#xA;model.add(MaxPooling2D(pool_size = (2,2),strides = None))&#xA;#############################################################################&#xA;&#xA;print ' Second layer setup'&#xA;# Output is 2x24&#xA;&#xA;##########################Third conv layer###################################&#xA;model.add(Convolution2D(64,3,3, border_mode = 'same', subsample = (1,1), init = 'glorot_normal'))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.6))&#xA;model.add(MaxPooling2D(pool_size = (2,2),strides = None))&#xA;#############################################################################&#xA;# Output is 12x12&#xA;&#xA;print ' Third layer setup'&#xA;###############################Fourth conv layer#############################&#xA;model.add(Convolution2D(128,3,3, border_mode = 'same', subsample = (1,1), init = 'glorot_normal'))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.6))&#xA;model.add(MaxPooling2D(pool_size = (2,2),strides = None))&#xA;############################################################################# &#xA;&#xA;print 'Fourth layer setup'&#xA;&#xA;# Output is 6x6x128&#xA;# Create the FC layer of size 128x6x6#&#xA;model.add(Flatten()) &#xA;model.add(Dense(2,init = 'glorot_normal',input_dim = 128*6*6))&#xA;model.add(Dropout(0.6))&#xA;model.add(Activation('softmax'))&#xA;&#xA;print 'Setting up fully connected layer'&#xA;print 'Now compiling the network'&#xA;sgd = SGD(lr=0.01, decay=1e-4, momentum=0.6, nesterov=True)&#xA;model.compile(loss = 'mse',optimizer = 'sgd', metrics=['accuracy'])&#xA;&#xA;# Fit the network to the data#&#xA;print 'Network setup successfully. Now fitting the network to the data'&#xA;model. fit(X_train,Y_train,batch_size = 100, nb_epoch = 20, validation_split = None,verbose = 1)&#xA;print 'Testing'&#xA;loss,accuracy = model.evaluate(X_test,Y_test,batch_size = 32,verbose = 1)&#xA;print &quot;Test fraction correct (Accuracy) = {:.2f}&quot;.format(accuracy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="29169" LastEditDate="2019-06-08T03:17:23.550" LastActivityDate="2019-06-08T03:17:23.550" Title="Validation loss and accuracy remain constant" Tags="|machine-learning|python|deep-learning|keras|" AnswerCount="2" CommentCount="15" ContentLicense="CC BY-SA 4.0" />
  <row Id="13608" PostTypeId="2" ParentId="13605" CreationDate="2016-08-23T06:48:34.040" Score="1" Body="&lt;p&gt;It is difficult to answer this question with the limited information provided here. Nonetheless, I have following recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a (data backed) reason to believe that you do not need 300 different models? For example, are the shipments of different products correlated? Alternatively, can you instead model the problem at higher levels like product categories? If either case, you can take a look at hierarchical time series modeling. &lt;a href=&quot;http://robjhyndman.com/papers/hierarchical/&quot; rel=&quot;nofollow&quot;&gt;This journal article by Rob Hyndman, author of forecast package in R&lt;/a&gt; can get you started on hierarchical time series modeling.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2016-08-23T06:48:34.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13609" PostTypeId="2" ParentId="13599" CreationDate="2016-08-23T08:13:37.030" Score="1" Body="&lt;p&gt;In order to build a model to make predictions you need a labeled training set, that is, a training set in which each training example is assigned a class label. Training sets are usually labeled by human experts that use their domain knowledge to manually classify the examples in the training set. You have already done that, as described in your first paragraph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, sometimes this process is expensive. In order to decrease cost sometimes &lt;strong&gt;semi-supervised learning is applied&lt;/strong&gt;. In semi-supervised learning, and under some assumptions, a small amount of labeled, together with a large amount of unlabeled data, are used to build a predictive model. The unlabeled data are assigned labels during the training process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems to match your idea in your last paragraph, and it is already implemented in &lt;a href=&quot;http://scikit-learn.org/stable/modules/label_propagation.html&quot; rel=&quot;nofollow&quot;&gt;Python&lt;/a&gt;. I think that if you found clear clusters in your data it may work very well. Worth a try at least. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-08-23T08:13:37.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13610" PostTypeId="1" CreationDate="2016-08-23T08:58:00.587" Score="3" ViewCount="6692" Body="&lt;p&gt;Should one apply dimensionality reduction methods to the data set before &lt;em&gt;or&lt;/em&gt; after train-test splitting?&#xA;Anyway, in case of training a model with preprocessing by dim-red, one should apply the same dim-red to the future instances to be predicted, right? But how could you reduce the dimensionality of one single instance in the same way?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21825" LastEditDate="2016-08-23T12:04:07.897" LastActivityDate="2016-08-23T13:51:23.713" Title="Dimension Reduction - After or Before Train-Test Split" Tags="|dimensionality-reduction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13611" PostTypeId="1" CreationDate="2016-08-23T09:46:42.260" Score="1" ViewCount="454" Body="&lt;p&gt;I was much impressed by the news-explorer view of IBM.It shows a network view with on searching a keyword and it explores the relational edges with a graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here it is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EZcLW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EZcLW.png&quot; alt=&quot;screen shot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it uses the k-means clustering in background.I too want to do the same for twitter data.I have tweets and users and mentions i want show same a network relation between them.can any one suggest me any graphical data tool which shows the network just like IBM by taking data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;using clustering algorithms like k-means or any other make my work done?&#xA;or do i need to go for any other approach ?&lt;/p&gt;&#xA;" OwnerUserId="20585" LastEditorUserId="381" LastEditDate="2016-08-24T18:01:03.960" LastActivityDate="2016-09-23T18:40:32.303" Title="How to create a social network like IBM's Watson News Explorer?" Tags="|machine-learning|k-means|unsupervised-learning|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13612" PostTypeId="2" ParentId="13593" CreationDate="2016-08-23T09:51:29.200" Score="0" Body="&lt;p&gt;This is not really a proper place to ask questions about software issues, but rather about data mining concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'll answer your question.&#xA;Your database is missing both extensions as specified in the error message. &lt;a href=&quot;https://github.com/biolab/orange3/wiki/Installation-of-PostgreSQL-extensions&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt; are some brief instructions.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2016-08-23T09:51:29.200" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13613" PostTypeId="2" ParentId="13610" CreationDate="2016-08-23T10:02:00.820" Score="4" Body="&lt;p&gt;It depends on the type of algortithm you use for dimensionality reductions. In case you use PCA, you should build your PCA on your train set. Then you need to set your principal components to transform your points in test set into the same space. This way you can then use train and test set in the same reducted space.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2016-08-23T10:02:00.820" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13615" PostTypeId="2" ParentId="13610" CreationDate="2016-08-23T11:50:51.960" Score="3" Body="&lt;p&gt;You should always use just your training data and then apply the same transformation on your test data. This is a much fairer representation of how your model will perform when real new unseen samples are fed to your model. The point of a test set is trying to estimate generalization on unseen data, using your test data leaks information into your training set. Plus, if you cannot apply the transformation to your test data, how will you apply it to new data?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-23T13:51:23.713" LastActivityDate="2016-08-23T13:51:23.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13617" PostTypeId="2" ParentId="12305" CreationDate="2016-08-23T11:57:22.203" Score="1" Body="&lt;p&gt;Here is the code I wrote to answer my question. It might not be the most efficient one but it works. Sharing is caring :)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;niter&amp;lt;-10000&#xA;N&amp;lt;-nrow(Y)&#xA;&#xA;T&amp;lt;-10 # I will take into consideration until t=10 to estimate my parametres and then I will forecast the rest values t=11,12,... etc&#xA;result &amp;lt;- matrix(0,nrow = niter, ncol = 3) #Here I store 3 parameters: alpha, sigma_epsilon, sigma_eta. I do not store gamma here&#xA;result_betas&amp;lt;-matrix(0,nrow=niter,ncol=N) #the betas&#xA;result_gammas&amp;lt;-matrix(0,nrow=niter,ncol=nrow(Z)) #here I store the gammas&#xA;&#xA;alpha&amp;lt;-0&#xA;gamma&amp;lt;-0&#xA;&#xA;&#xA;sigma_epsilon_squared&amp;lt;-1&#xA;sigma_eta_squared&amp;lt;-1&#xA;beta&amp;lt;-matrix(0, nrow=N, ncol=1)&#xA;beta2&amp;lt;-0&#xA;&#xA;&#xA;&#xA;for (i in 1:niter) &#xA;{&#xA;  mean_alpha&amp;lt;-0&#xA;  for (obs in 1:N)&#xA;  {&#xA;    for (time in 1:T)&#xA;    {&#xA;      mean_alpha&amp;lt;-mean_alpha+Y[obs,time] - beta[obs]*X[obs,time]&#xA;    }&#xA;  }&#xA;  alpha&amp;lt;-rnorm(1, mean = (1/(T*N))* mean_alpha, sd = sqrt((sigma_epsilon_squared)/(T*N)))&#xA;&#xA;&#xA;  param_sigma_epsilon_squared&amp;lt;-0&#xA;  for (obs in 1:N)&#xA;  {&#xA;    for (time in 1:T)&#xA;    {&#xA;      param_sigma_epsilon_squared&amp;lt;-param_sigma_epsilon_squared + (Y[obs,time] - alpha - beta[obs]*X[obs,time])^2&#xA;    }&#xA;  }&#xA;  sigma_epsilon_squared &amp;lt;- rinvgamma(1, (N*T)/2, param_sigma_epsilon_squared/2 )&#xA;&#xA;&#xA;  mean_gamma&amp;lt;-0 #this is a vector&#xA;  mean_gamma1&amp;lt;-0 #this is a matrix&#xA;  mean_gamma2&amp;lt;-0&#xA;  sigma_gamma&amp;lt;-0&#xA;  for(obs in 1:N)&#xA;  {&#xA;    mean_gamma1&amp;lt;-mean_gamma1 + Z[,obs]%*%t(Z[,obs]) #this is (k x k)&#xA;    mean_gamma2&amp;lt;-mean_gamma2 + Z[,obs]*beta[obs]&#xA;&#xA;    #mean_gamma&amp;lt;-mean_gamma + 1/(Z[obs,]%*%t(Z[obs,])) * (Z[obs,]*beta[obs,1])&#xA;    #sigma_gamma&amp;lt;-sigma_gamma+1/(Z[obs,]%*%t(Z[obs,]))&#xA;  }&#xA;  mean_gamma&amp;lt;-solve(mean_gamma1)%*%mean_gamma2&#xA;  sigma_gamma&amp;lt;-solve(mean_gamma1)&#xA;  gamma&amp;lt;-mvrnorm(n = 1, mu=mean_gamma, Sigma=sqrt(sigma_eta_squared)*sigma_gamma)&#xA;&#xA;  result_gammas[i,]&amp;lt;-gamma&#xA;&#xA;  mean_sigma_eta_squared&amp;lt;-0&#xA;  for (obs in 1:N)&#xA;  {&#xA;    mean_sigma_eta_squared&amp;lt;- mean_sigma_eta_squared + (beta[obs] - t(Z[,obs])%*%gamma)^2&#xA;  }&#xA;  sigma_eta_squared&amp;lt;-rinvgamma(1, N/2, mean_sigma_eta_squared/2 )&#xA;&#xA;  result[i,]&amp;lt;-c(alpha,sigma_epsilon_squared,sigma_eta_squared)&#xA;&#xA;&#xA;&#xA;&#xA;  mean_beta&amp;lt;-0&#xA;  mean_beta1&amp;lt;-0 #this is scalar&#xA;  mean_beta2&amp;lt;-0 #this is scalar&#xA;  sigma_beta&amp;lt;-0&#xA;  for (obs in 1:N)&#xA;  {&#xA;    for (time in 1:T)&#xA;    {&#xA;      mean_beta1&amp;lt;-mean_beta1 + X[obs,time]^2 + (sigma_epsilon_squared/sigma_eta_squared)&#xA;      mean_beta2&amp;lt;-mean_beta2 + X[obs,time]*Y[obs,time] + (sigma_epsilon_squared/sigma_eta_squared)*(t(Z[,obs])%*%gamma)&#xA;&#xA;&#xA;    }&#xA;    mean_beta&amp;lt;-(1/mean_beta1)*mean_beta2&#xA;    sigma_beta&amp;lt;-1/mean_beta1&#xA;    beta2&amp;lt;-rnorm(1, mean = mean_beta, sd = sqrt((sigma_epsilon_squared)*sigma_beta))&#xA;&#xA;    beta[obs]&amp;lt;-beta2&#xA;    result_betas[i,obs]&amp;lt;-beta2&#xA;  }&#xA;  if(i%%10==0)&#xA;  {&#xA;    print(i)&#xA;  }&#xA;&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2016-08-23T11:57:22.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13618" PostTypeId="1" CreationDate="2016-08-23T13:23:40.167" Score="1" ViewCount="1159" Body="&lt;p&gt;I am using Hunspell to spellcheck and stem the words in my documents to reduct dimensionality. For spellchecking Hunspell works great with the default en_US dictionary by SCOWL (and friends), but not so for stemming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is that the dictionary is pretty inconsistent. (e.g. some words have their plurals in the dictionary, but some others doesn't, so it's inconsistent if the stem of a word in plural will be in plural or not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hunspell is definitely able to stem english words consistently, but it needs an other dictionary that is more &quot;correct&quot; morphologically.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What alternative dictionaries can I use to improve the quality of stemming English text with Hunspell?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If there are no such dictionaries, what other tools can I use for stemming? EDIT: note that I'm using python, which can be hooked up with most libraries and command-line tools easily, but R or Matlab specific tools might not fit in my current ecosystem that well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23713" LastEditorUserId="23713" LastEditDate="2016-08-23T14:54:47.283" LastActivityDate="2016-08-23T14:54:47.283" Title="Alternative Hunspell dictionary for stemming" Tags="|nlp|text-mining|dimensionality-reduction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13619" PostTypeId="1" AcceptedAnswerId="13627" CreationDate="2016-08-23T13:52:54.293" Score="0" ViewCount="247" Body="&lt;p&gt;I've a dataset with following schema:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Customer_ID - Unique ID &#xA;Product - ID of purchased product&#xA;Department - ID of the department that sells the product&#xA;Product_Type - The purchased product type&#xA;Date - The date of purchase&#xA;Quantity - The number of units purchased&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to do a link analysis project to analyze some consumption patterns of the products and answer the following questions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;If product B is purchased then customer will also take product A&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I will use Scala/Python to make the link analysis over the datasets but the  examples I have seen are dataset with direct links to the project of &quot;Flight Data&quot; in the schema is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID&#xA;Origin&#xA;Destination&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: there I need to prepare my dataset to make the Link Analysis (there exists some best practices to do this?) or can I analyze the dataset with that structure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks! Sorry my inexperience on this topic!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2017-12-28T23:58:12.850" Title="Need to prepare the data to Link Analysis project?" Tags="|data-mining|bigdata|dataset|apache-spark|social-network-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13620" PostTypeId="2" ParentId="13618" CreationDate="2016-08-23T14:11:47.560" Score="3" Body="&lt;p&gt;I can't comment on your question (my reputation isn't high enough), but I've got a few clarifying questions to ask: Why do you want to stem words?  Text classification?  Sentiment analysis?  Something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used both the 'tm' package and the 'RTextTools' package for stemming when doing text classification.  Both have some built-in functions for stemming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether they would be useful to you or not depends on why you want to stem words ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;FWIW, here are some links related to those packages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;tm: &lt;a href=&quot;https://cran.r-project.org/web/packages/tm/index.html&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/tm/index.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;RTextTools: &lt;a href=&quot;http://www.rtexttools.com/documentation.html&quot; rel=&quot;nofollow&quot;&gt;http://www.rtexttools.com/documentation.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23420" LastEditorUserId="23420" LastEditDate="2016-08-23T14:24:07.137" LastActivityDate="2016-08-23T14:24:07.137" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13621" PostTypeId="2" ParentId="6188" CreationDate="2016-08-23T14:23:58.547" Score="0" Body="&lt;p&gt;Would be good to have a look at your data to make a suggestion. If you need to recommend mp3 based on user previous choice then it would make sense to identify what mp3 (genres, length, authors, etc.) are most liked by the user (and what is least), of course if you have such a data for your mp3. Then knowing your user`s preferences you may apply different collaborative approaches.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/iabdrashitov/Recommendation_engine_yelp/blob/master/Yelp_Recommendation_Engine.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is how I did it with yelp data.&lt;/p&gt;&#xA;&lt;p&gt;it is not related to mp3, however could work similarly with mp3 in terms of identifying users preferences and recommending mp3 based on that. Hope this will help&lt;/p&gt;&#xA;" OwnerUserId="23716" LastEditorUserId="98307" LastEditDate="2020-08-17T18:55:06.123" LastActivityDate="2020-08-17T18:55:06.123" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13622" PostTypeId="2" ParentId="13584" CreationDate="2016-08-23T16:27:28.097" Score="0" Body="&lt;p&gt;Please, do not use ANN, because what you are looking for actually is SLAM method. It is a set of algorithms working together to localize your ship in an environment and mapping it simultaneously.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find an introduction on the Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what autonomous cars are using.&lt;/p&gt;&#xA;" OwnerUserId="23721" LastActivityDate="2016-08-23T16:27:28.097" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13623" PostTypeId="2" ParentId="13619" CreationDate="2016-08-23T16:52:24.597" Score="0" Body="&lt;p&gt;When referencing code examples built to analyze phenomena that differs from what I am studying, I find it best to learn as much as I can about the procedures themselves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, I would look at the procedures defined in the example code you have found, run a few tests on a subset of your own data to make sure you know how the procedures are functioning, and then, if you are satisfied with the results, proceed to use the code to in your analysis (making sure to cite code sources where required or applicable). &lt;/p&gt;&#xA;" OwnerUserId="23701" LastActivityDate="2016-08-23T16:52:24.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13624" PostTypeId="2" ParentId="12188" CreationDate="2016-08-23T17:55:42.147" Score="1" Body="&lt;p&gt;Michael is right - K-means clustering could possibly work for you, but K-means isn't designed to handle categorical variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have too many categories, then you could choose to represent them as dummy variables.  Here's a &lt;a href=&quot;https://datascience.stackexchange.com/a/13305/23230&quot;&gt;link to a post&lt;/a&gt; where I explain dummy variables in Python Pandas.  I also found a &lt;a href=&quot;https://stackoverflow.com/a/11952708/5066140&quot;&gt;Stack Overflow answer&lt;/a&gt; that explains how to create dummy variables in R.&lt;/p&gt;&#xA;" OwnerUserId="23230" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-23T17:55:42.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13627" PostTypeId="2" ParentId="13619" CreationDate="2016-08-23T19:10:22.257" Score="0" Body="&lt;p&gt;One option is that you could make a bipartite graph from your TLOG and then implement some link analysis. Depending on the requirements that you need (&lt;em&gt;volume of the data&lt;/em&gt;) there are different frameworks that you can use. One which is quite popular for not so large data is &lt;code&gt;networkx&lt;/code&gt; where (&lt;a href=&quot;https://networkx.github.io/documentation/networkx-1.9.1/reference/algorithms.html&quot; rel=&quot;nofollow&quot;&gt;manual&lt;/a&gt;) you can find already implemented algorithms for link analysis and link prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe, the community would be able to help you more if you try to be more specific about what kind of link analysis you want and what kind of problem you try to solve (&lt;em&gt;does it has to do probably with Supervised or Unsupervised learning&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-08-23T19:10:22.257" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13629" PostTypeId="1" CreationDate="2016-08-23T20:20:48.710" Score="6" ViewCount="228" Body="&lt;p&gt;I'm building a TensorFlow convoluted neural network that isn't getting the accuracy that I hoped for. So I figured I would visualize the learned weights to see where the network might be stumbling. As a benchmark I started visualizing the weights for a totally different project, &lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/index.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;abc&quot;&gt;Google's MNIST convnet example&lt;/a&gt;, which has very high accuracy (99.2%). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assumed that a very accurate model would have intuitive weights, but in reality I got weights that looked completely random. Other people seem to get similarly &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/908&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;random results&quot;&gt;random results&lt;/a&gt;. See martin-gorner's comment with visuals on Mar 18. His are results are similar to mine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More Googling shows that other people see essentially &lt;a href=&quot;https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;123&quot;&gt;random behavior&lt;/a&gt; for non-MNIST datasets. Is this common? If so, it would seem that visualizing weights is a fruitless exercise unlikely to bring any helpful insights to the modeler. Is this true? If not, why do so many people use &lt;a href=&quot;https://stats.stackexchange.com/questions/146413/why-convolutional-neural-networks-belong-to-deep-learning&quot; title=&quot;a1&quot;&gt;this visual&lt;/a&gt; (clear edges in the first layer, and composite images in subsequent layers) when describing how convnets work? &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;a&quot;&gt;Stanford's course material&lt;/a&gt; also seems to at least show edges (you have to scroll down about half of the page) when visualizing the weights. I'm not sure what to think. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29169" LastEditDate="2021-02-16T15:39:21.523" LastActivityDate="2021-02-16T15:39:21.523" Title="Highly accurate convnets appear to have random-looking visualized weights?" Tags="|machine-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13630" PostTypeId="2" ParentId="12723" CreationDate="2016-08-23T21:41:21.223" Score="2" Body="&lt;p&gt;Microsoft announced a couple of weeks ago virtual machines in Azure with &lt;a href=&quot;https://azure.microsoft.com/en-us/blog/azure-n-series-preview-availability/?utm_content=buffer2720b&amp;amp;utm_medium=social&amp;amp;utm_source=twitter.com&amp;amp;utm_campaign=buffer&quot; rel=&quot;nofollow&quot;&gt;GPUs&lt;/a&gt;. They use K-80 NVIDIA cards. The biggest machine has 4 GPUs and 224 GB of ram. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good to play with deep learning :-)&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-08-23T21:41:21.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13631" PostTypeId="1" AcceptedAnswerId="14005" CreationDate="2016-08-24T04:44:18.730" Score="0" ViewCount="134" Body="&lt;p&gt;In powerBi, a column is created with below formula, and it is working good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;activedays = COUNTAX(RELATEDTABLE(ActivesInactiveData),ActivesInactiveData[Surface Date])&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I want to add a filter some thing like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ActivesInactiveData[Topic]&amp;lt;&gt; null&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please advice on how to do so, most of the examples on web are using on relatedtable etc - where relation is already build, could you please advice/direct to me what I could find an example of adding filter to relatedtable or filter data from a unrelated table based on field like ID column etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here there is a relation user ID in current table has more than 1 rows in ActivesInactiveData table.&lt;/p&gt;&#xA;" OwnerUserId="23731" LastActivityDate="2016-10-14T23:18:28.757" Title="DistinctCount for relatedtable with additional filter" Tags="|powerbi|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13632" PostTypeId="2" ParentId="13611" CreationDate="2016-08-24T05:05:05.787" Score="1" Body="&lt;p&gt;They've created a graph from the news articles, topics, and named entities (locations, persons, companies, organizations). There are a lot of things going on here, but k-means is not one of them. If I had to do this I would use a named entity recognition (NER) and document embeddings (doc2vec, etc.). Once you have the embeddings, and the edges (thanks to NER), use a graph layout algorithm like force direction. If the graph is too dense, thin out the weaker edges. If all that sounds Chinese to you, start by reading about &quot;named entity recognition&quot;, and &quot;word embeddings&quot;. The idea is to attach a number (or rather, a vector) to everything from a word to a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Twitter is a different beast. The textual content, such as it is, won't play nice with these embedding algorithms, but you have hashtags and strong social signals; mentions, retweets, and follows. This is pretty complicated, too, so I'll leave you with a relevant paper: &lt;a href=&quot;http://users.cecs.anu.edu.au/~Chen.Changyou/nips_workshop2013.pdf&quot; rel=&quot;nofollow&quot;&gt;Twitter-Network Topic Model: A Full Bayesian Treatment for Social Network and Text Modeling&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-08-24T05:05:05.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13633" PostTypeId="1" AcceptedAnswerId="13635" CreationDate="2016-08-24T07:41:53.840" Score="1" ViewCount="80" Body="&lt;p&gt;I used the Neural Network Toolbox in matlab to train my data. I used four training algorithms, Scaled Conjugate Gradient (SCG), Gradient Descent with momentum and adaptive learning back-propagation (GDX), Resilient Back Propagation (RBP) and Broyden-Fletcher-Goldfarb-Shanno quasi-Newton back propagation (BFG). I have fixed the seeds at different points and obtained the accuracy. This is what I get:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/kqxny.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kqxny.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;br&gt;The first column contains the size of the feature set. I have added features and increased the size of feature set to analyse the performance. &lt;br&gt;&lt;br&gt;Initially I have ranked the features and then taken the top 8 feature as one set, the top 16 feature as the next set and so on. The first number before the '-' , is the performance of the algorithm on the training set, the second number after the '-' is the accuracy of the testing set. The train and test set has been divided into 60 and 20 respectively. The other 20 is the validation set. The learning algorithms each have been run with the same seed values to fix the accuracy.  &lt;br&gt;&lt;br&gt;I have fixed the seed to obtain each of the results btw, Like I have used rng(1), rng(10), rng(158), rng(250) and averaged the results to obtain one single pair of train-test accuracy, and I have done this for each pairs. &lt;br&gt;&lt;br&gt;As you can see I am getting the same accuracy for all feature set size for each of the individual training algorithm. The same data shows perturbation in SVM chen I change the set size. What does this mean?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2016-08-24T09:19:38.647" LastActivityDate="2016-08-24T09:19:38.647" Title="Query regarding neural network model" Tags="|neural-network|supervised-learning|matlab|accuracy|training|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13635" PostTypeId="2" ParentId="13633" CreationDate="2016-08-24T09:15:29.653" Score="1" Body="&lt;p&gt;To debug this case, I suggest you try the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Reduce the features step-by-step until you end up with using just 1 feature and see whether the accuracy changes or not.&lt;/li&gt;&#xA;&lt;li&gt;Add a sine-wave and a random noise to the feature set and see whether it effects any of these optimization algorithms.&lt;/li&gt;&#xA;&lt;li&gt;Re-evaluate how you selected or derived these features, check if these are highly correlated.&lt;/li&gt;&#xA;&lt;li&gt;Are your classification targets highly imbalanced? If so, then under/over sample them to achieve a more balanced training set. Then check the performance of you algorithm after training over this balanced dataset.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As already highlighted by Jan van der Vegt, its extremely odd that changing the no of features from 8 to 40 has no impact on test set accuracy.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-08-24T09:15:29.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13636" PostTypeId="1" AcceptedAnswerId="13637" CreationDate="2016-08-24T11:24:33.353" Score="5" ViewCount="6546" Body="&lt;p&gt;In case of image classification with 0-255 pixel values of integer type, is it necessary/recommended to convert the values into float with Neural Networks? This conversion seems me unnecessary (moreover, float slows down matrix operations) but several implementations follow this practice. Why?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-08-24T11:48:03.880" Title="Neural Network data type conversion - float from int?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13637" PostTypeId="2" ParentId="13636" CreationDate="2016-08-24T11:48:03.880" Score="7" Body="&lt;p&gt;Technically with most languages you could pass in integer features for the input layer, since the weights will be floats, and multiplying a float by an integer will give you a float. Also, you don't usually care about partial derivatives of the input data, so it doesn't matter that the values are discrete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For all weights and neuron activations, if you are using a method based on backpropagation for training updates, then you need a data type that approximates real numbers, so that you can apply fractional updates based on differentiation. Best weight values are often going to be fractional, non-whole numbers. Non-linearities such as sigmoid are also going to output floats. So after the input layer you have matrices of float values anyway. There is not much speed advantage multiplying integer matrix with float one (possibly even slightly slower, depending on type casting mechanism). So the input may as well be float.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In addition, for efficient training, the neural network inputs should be normalised to a specific roughly unit range (-1.0 to 1.0) or to mean 0, standard deviation 1.0. Both of these require float representation. If you have input data in 0-255 range - float or not - you will usually find the network will learn less effectively.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are exceptions to these rules in some architectures and learning algorithms, where perhaps an integer-based input would work, but for the most common NN types, including MLP and &quot;deep&quot; feed-forward networks, it is simpler and easier to use float data type.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-08-24T11:48:03.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13641" PostTypeId="1" CreationDate="2016-08-24T14:48:07.077" Score="4" ViewCount="1099" Body="&lt;p&gt;My target variable is boolean and has 0.001% of NO records and remaining YES records.&#xA;Can anyone suggest algorithms which are best for anomaly detection in R when there is severe class imbalance.&lt;/p&gt;&#xA;" OwnerUserId="13517" LastActivityDate="2018-01-23T14:24:15.203" Title="classification algorithms for anomaly detection when there is class imbalance" Tags="|r|classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13643" PostTypeId="2" ParentId="12059" CreationDate="2016-08-24T18:03:06.650" Score="2" Body="&lt;p&gt;I'm in the same situation as you.  PCA seems to have some difficulties for compositional data.  I've listed some papers below that have been helpful to me.  I'm still trying to understand everything, but these papers have a good overview I think.  Or googling &quot;Compositional Data&quot; or &quot;Compositional Data Principal Component Analysis&quot; returns a lot of other papers that are freely available and informative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal component analysis of compositional data J. Aitchinson&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Concise Guide to Compositional Data Analysis J. Aitchinson&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Principal components analysis for compositional data with outliers P. Filzmoser, et al.&lt;/p&gt;&#xA;" OwnerUserId="23752" LastActivityDate="2016-08-24T18:03:06.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13645" PostTypeId="1" CreationDate="2016-08-24T18:57:12.030" Score="6" ViewCount="4120" Body="&lt;p&gt;I am looking to perform k-means on my dataset which contains a large number of 0 values.&lt;/p&gt;&#xA;&lt;p&gt;The last value you see is different to the others, that is simply the sum of transactions, not related to the categorical frequency count.&lt;/p&gt;&#xA;&lt;h3&gt;Example:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 189200.579626]&#xA;[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06556796]&#xA;[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]&#xA;[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Each feature is a frequency count of how many times the associated id is seen within a category.&lt;/p&gt;&#xA;&lt;p&gt;When I run k-means, I see that most of the data is clustered in one cluster.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;+------------+--------+----------------------+&#xA;| cluster_id |  size  | sum_squared_distance |&#xA;+------------+--------+----------------------+&#xA;|     0      | 659187 |    0.999997057952    |&#xA;|     1      |   3    |    1.33333326876     |&#xA;|     2      |   1    |         0.0          |&#xA;|     3      |   3    |    0.666666716337    |&#xA;|     4      |   1    |         0.0          |&#xA;|     5      |   1    |         0.0          |&#xA;|     6      |   1    |         0.0          |&#xA;|     7      |   1    |         0.0          |&#xA;|     8      |   1    |         0.0          |&#xA;|     9      |   11   |    2.72727286816     |&#xA;+------------+--------+----------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am assuming that this is because the majority of the dataset has not been seen in a feature category and therefore has a value of 0.&lt;/p&gt;&#xA;&lt;p&gt;What is the best way to overcome this, drop rows where a &lt;code&gt;0&lt;/code&gt; is seen across each category? Are these rows meaningless to clustering?&lt;/p&gt;&#xA;" OwnerUserId="15961" LastEditorUserId="29575" LastEditDate="2020-08-04T13:26:11.640" LastActivityDate="2020-08-04T13:26:11.640" Title="k-means clustering data with large number of meaningless values" Tags="|machine-learning|python|clustering|algorithms|k-means|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13646" PostTypeId="1" CreationDate="2016-08-24T20:05:23.737" Score="1" ViewCount="151" Body="&lt;p&gt;This is not covered very well in their &lt;a href=&quot;http://www.cs.cmu.edu/~15780/AlphaGo.nature16961.pdf&quot; rel=&quot;nofollow&quot;&gt;AlphaGo paper&lt;/a&gt; but I assume that their policy network has a softmax output layer with a node for all the positions on the board, including illegal ones (the one with a stone on the board already). Maybe the illegal ones are basically zero already, I'm uncertain, but I assume they manually put them to zero and then normalize the probabilities. Please correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To dip my toes into using neural networks for game AI I decided to work on a Backgammon player. I have only implemented a 1D convolutional value network on the board and merged the convolutional layers with the bar and the bearing states. I let random players play a bunch of games, train the model on those game states and let the model play against itself a number of times, tune the model further and iterate this step a number of times. The results are promising so I was looking to add a tree search algorithm similar to AlphaGo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where my question comes from. I would like to use a policy network to guide the exploration through the tree, but I don't know how to represent it. If you don't take the current state in mind, you would have a huge amount of potential states in your last layer of which more than 99.99% would be illegal. If you do keep the current state in mind I don't see how I could train a network on this. Is there any way of building a policy network for Backgammon?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative method would be to use the value network on the potential states that you are able to go to (the network outputs a probability of winning for the active player) and transform those probabilities into a new distribution for exploration, taking in mind exploration versus exploitation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: My current representation of actions is the state it would transfer into by doing that action, which means the number of actions over all possible games is enormous.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-08-25T08:25:37.737" LastActivityDate="2016-08-25T08:25:37.737" Title="Policy network AlphaGo and transferring to other domains" Tags="|neural-network|deep-learning|reinforcement-learning|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13647" PostTypeId="2" ParentId="13645" CreationDate="2016-08-24T20:54:18.197" Score="4" Body="&lt;p&gt;One thing you could do is apply some dimensional reduction algorithm (such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt;) so you can get the columns with high variance, then run k-means on that data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I suggest against using k-means in sparse matrices like yours. Anony-Mousse's answer &lt;a href=&quot;https://stackoverflow.com/questions/18063087/is-k-means-for-clustering-data-with-many-zero-values&quot;&gt;here&lt;/a&gt; explains it well.&lt;/p&gt;&#xA;" OwnerUserId="7739" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-24T20:54:18.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13648" PostTypeId="2" ParentId="13645" CreationDate="2016-08-24T20:59:57.320" Score="1" Body="&lt;p&gt;welcome to DataScienceSO!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess from a logical standpoint a very sparse dataset (heaps of 0's or missing values) will likely create similar clusterings just simply due to the lack of information for a large amount of the observations. So you're right in your assumption. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically these rows are not useless as they represent a valid clustering, but in a business sense they essentially are as they communicate no information about potential groupings of observations going forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NB: K-means is influenced by differences in scale which may be causing some troubles&lt;/strong&gt;. Whatever statistical package will definitely have a scaling function that forces the variable to have mean zero and sd of one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a problem because k-means doesn't understand differences in units. For instance if variable A is measured in meters and variable B in km, A = 1000 would be seen as greater than B = 1 even though they are equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I'd take a second look at the number of clusters you have. It may be too many for the amount of real information you have. See &lt;a href=&quot;https://stackoverflow.com/questions/18042290/implementing-the-elbow-method-for-finding-the-optimum-number-of-clusters-for-k-m&quot;&gt;this post for more information&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21702" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-24T20:59:57.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13649" PostTypeId="1" CreationDate="2016-08-24T21:19:02.870" Score="0" ViewCount="4657" Body="&lt;p&gt;Can anyone help with plotting k-means results from GraphLab k-means tool. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sf = gl.SFrame(df_only_tags)&#xA;K = int(sqrt(sf.num_rows() / 2.0))&#xA;&#xA;kmeans_model = gl.kmeans.create(sf, num_clusters=6)&#xA;kmeans_model.summary()&#xA;&#xA;print(kmeans_model['cluster_info'][['cluster_id', 'size', 'sum_squared_distance']])&#xA;print(kmeans_model['cluster_id'].head())&#xA;kmeans_model.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am able to print my results just fine, but looking for the best way to represent my results visually:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+------------+-------+----------------------+&#xA;| cluster_id |  size | sum_squared_distance |&#xA;+------------+-------+----------------------+&#xA;|     0      | 10594 |     1034.9124542     |&#xA;|     1      |  312  |    42.9715532884     |&#xA;|     2      |   2   |    84.9999923706     |&#xA;|     3      |   1   |         0.0          |&#xA;|     4      |   98  |    213.489804491     |&#xA;|     5      |   1   |         0.0          |&#xA;+------------+-------+----------------------+&#xA;[6 rows x 3 columns]&#xA;&#xA;+--------+------------+-----------------+&#xA;| row_id | cluster_id |     distance    |&#xA;+--------+------------+-----------------+&#xA;|   0    |     0      | 0.0553383231163 |&#xA;|   1    |     0      | 0.0553383231163 |&#xA;|   2    |     0      | 0.0553383231163 |&#xA;|   3    |     0      | 0.0553383231163 |&#xA;|   4    |     0      | 0.0553383231163 |&#xA;|   5    |     0      | 0.0553383231163 |&#xA;|   6    |     0      | 0.0553383231163 |&#xA;|   7    |     0      | 0.0553383231163 |&#xA;|   8    |     0      | 0.0553383231163 |&#xA;|   9    |     0      | 0.0553383231163 |&#xA;+--------+------------+-----------------+&#xA;[10 rows x 3 columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as always, much appreciated &lt;/p&gt;&#xA;" OwnerUserId="15961" LastEditorUserId="15961" LastEditDate="2016-08-24T21:43:58.330" LastActivityDate="2020-08-04T09:32:45.313" Title="Plotting k-means output - python" Tags="|machine-learning|python|visualization|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13652" PostTypeId="2" ParentId="13649" CreationDate="2016-08-25T00:29:42.747" Score="1" Body="&lt;p&gt;Usually, I'd follow the approach outlined &lt;a href=&quot;https://stackoverflow.com/questions/7881994/matplotlib-how-to-change-data-points-color-based-on-some-variable&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In your case, you'll be setting the point colors based on the cluster ID.&lt;/p&gt;&#xA;&lt;p&gt;However, the cluster sizes are so imbalanced that it could be hard to see anything except for cluster 0. Since all of the distances in the header are identical, perhaps you're modeling duplicate points? If you can drop the duplicates before plotting the chart will probably be easier to read. Alternately, you could scale the point sizes by the number of duplicates.&lt;/p&gt;&#xA;" OwnerUserId="23761" LastEditorUserId="98307" LastEditDate="2020-08-04T09:32:45.313" LastActivityDate="2020-08-04T09:32:45.313" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13654" PostTypeId="1" CreationDate="2016-08-25T09:49:55.190" Score="1" ViewCount="560" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5tqyt.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5tqyt.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train silhouette images to figure out the original image&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm planning to use Tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I've thought about using Automatic colorization, but I have no idea if it works well since what it only has is just an outline of the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to achieve this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="23777" LastActivityDate="2016-08-25T09:49:55.190" Title="deep learning - how to train silhouette image?" Tags="|machine-learning|deep-learning|image-classification|tensorflow|image-recognition|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13656" PostTypeId="2" ParentId="13581" CreationDate="2016-08-25T10:38:01.050" Score="2" Body="&lt;p&gt;Without any additional data description, i would try out something along these lines: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Cluster the &lt;em&gt;shapes&lt;/em&gt; of the sales curves using methods for longitudinal data clustering (K-means for longitudinal data / frechet distance...)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Classify your products by using the features that you have as predictors and the cluster label as response (CHAID trees / random forests...)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Predict class of new product will result in a probable future sales curve ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It will highly depend on the quality of the classifier though, but I think it's worth a try. hope this helps, and let us know what did you end up doing - will be interesting to hear.&lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2016-08-25T10:38:01.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13657" PostTypeId="2" ParentId="10542" CreationDate="2016-08-25T10:53:48.513" Score="1" Body="&lt;p&gt;I would go for &lt;a href=&quot;http://rmarkdown.rstudio.com/developer_parameterized_reports.html&quot; rel=&quot;nofollow&quot;&gt;parametrised Rmarkdown reports&lt;/a&gt;. I think it adresses all your needs, it's simple to use and maintain, offers great flexibility, you can embed all kinds of things and it offers different output formats.&lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2016-08-25T10:53:48.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13658" PostTypeId="1" AcceptedAnswerId="13795" CreationDate="2016-08-25T10:57:42.487" Score="4" ViewCount="1167" Body="&lt;p&gt;I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am using csv files to save the data. Everybody supports csv. But csv does not support type information. This makes it sometimes cumbersome to load a csv file because I must check all the column types. This seems unnecessary because the SQL database already specifies the types of the columns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if there is a broadly accepted file format to save data that also specifies the type of the columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am working with SQL databases, FME ETL'ing and R but I think this is an issue for every data tranfer. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="85045" LastEditDate="2021-02-10T17:50:04.897" LastActivityDate="2021-02-10T17:50:04.897" Title="What is the most used format to save data with type information" Tags="|data|databases|data-formats|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13659" PostTypeId="2" ParentId="10542" CreationDate="2016-08-25T12:02:05.377" Score="1" Body="&lt;p&gt;Another alternative is to use Jupyter notebooks, if it supports kernels of your programming language. It allows for normal markdown, LaTeX, output shown in the web document and interactivity. You can share the true, runnable document if it's hosted on a normal server or a HTML dump for easy sharing.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-25T12:02:05.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13660" PostTypeId="1" CreationDate="2016-08-25T12:21:23.103" Score="3" ViewCount="2662" Body="&lt;p&gt;I'm training a corpus consisting of 200000 reviews into positive and negative reviews using a Naive Bayes model, and I noticed that performing TF-IDF actually reduced the accuracy (while testing on test set of 50000 reviews) by about 2%. So I was wondering if TF-IDF has any underlying assumptions on the data or model that it works with, i.e. any cases where accuracy is reduced by the use of it?&lt;/p&gt;&#xA;" OwnerUserId="22059" LastActivityDate="2016-12-13T02:06:41.187" Title="In general, when does TF-IDF reduce accuracy?" Tags="|classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13663" PostTypeId="1" CreationDate="2016-08-25T13:20:18.243" Score="12" ViewCount="9257" Body="&lt;p&gt;I'm a bit confused by the coexistence of Loss and Accuracy metrics in Neural Networks. Both are supposed to render the &quot;exactness&quot; of the comparison of &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\hat{y}$&lt;/span&gt;, aren't they? So isn't the application of the two redundant in the training epochs? Moreover, why aren't they correlate?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="84891" LastEditDate="2019-12-16T18:36:03.393" LastActivityDate="2019-12-16T18:36:03.393" Title="Neural Networks - Loss and Accuracy correlation" Tags="|neural-network|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13664" PostTypeId="2" ParentId="13663" CreationDate="2016-08-25T13:33:03.483" Score="0" Body="&lt;p&gt;Yes, they both measure the exactness of y and y_hat and yes they're usually correlated. Sometimes the loss function might not be accuracy but you're still interested in measuring the accuracy even though you're not optimizing it directly. Google's TensorFlow MNIST example minimizes/optimizes cross entropy loss but displays accuracy to the user when reporting results, and this is completely fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes you don't want to optimize accuracy directly. For example, if you have serious class imbalance, your model will maximize accuracy by simply always picking the most common class, but this would not be a useful model. In this case entropy / log-loss would be a better loss function to optimize.  &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-25T13:33:03.483" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13666" PostTypeId="2" ParentId="13658" CreationDate="2016-08-25T13:45:26.037" Score="2" Body="&lt;p&gt;Parquet and Avro both support data types (strings, integers, floats, etc). These are the primary file types used for &quot;big data&quot; projects, although your data doesn't have to be big. Apache Spark is able to read both with ease. Having said that, I'm a big fan of keeping data in a SQL database (e.g., MySQL or Postgres) because that is what they are built for. If you can't re-use the database you're pulling from, could you make your own database locally or on a separate server? I would try using a relational database until your data exceeds 50 GB (an arbitrarily &quot;somewhat large&quot; size), and then I would use Avro or Parquet. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-25T13:45:26.037" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13667" PostTypeId="2" ParentId="13663" CreationDate="2016-08-25T13:50:18.167" Score="11" Body="&lt;p&gt;Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) you use a loss function that behaves similarly to proxy the true metric. In case of binary classification you would use a sigmoid at the end and a log loss to approximate accuracy. They are highly correlated.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-25T13:50:18.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13668" PostTypeId="2" ParentId="13581" CreationDate="2016-08-25T14:13:07.410" Score="2" Body="&lt;p&gt;I will try below steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find similar existing products: Clustering the brand new product with existing products.&lt;/li&gt;&#xA;&lt;li&gt;Interpolate the predicted sales for this brand new product: Since we already found a cluster of existing products which are similar to this brand new product, we can apply predicting algorithm on the existing products and interpolate the predicted sales for the brand new product.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this will help you.&lt;/p&gt;&#xA;" OwnerUserId="1003" LastActivityDate="2016-08-25T14:13:07.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13669" PostTypeId="1" CreationDate="2016-08-25T14:49:06.843" Score="22" ViewCount="47193" Body="&lt;p&gt;I'm currently working/prototyping into a &lt;em&gt;Jupyter&lt;/em&gt; notebook. I want to run some of my code on a standalone &lt;em&gt;iPython&lt;/em&gt; shell. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I export my &lt;em&gt;iPython&lt;/em&gt; code (file --&gt; download as) and then execute it in my &lt;em&gt;iPython&lt;/em&gt; (with %run). It works, but I would like to export only one cell or set of cells. So, that I can run only what I modified in my &lt;em&gt;Jupyter&lt;/em&gt; notebook.&lt;/p&gt;&#xA;" OwnerUserId="21825" LastEditorUserId="40853" LastEditDate="2017-12-07T14:54:10.737" LastActivityDate="2021-02-17T01:02:58.350" Title="How to export one cell of a jupyter notebook?" Tags="|ipython|jupyter|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13671" PostTypeId="1" AcceptedAnswerId="13674" CreationDate="2016-08-25T17:45:41.980" Score="1" ViewCount="641" Body="&lt;p&gt;I'm using &lt;code&gt;agnes&lt;/code&gt; to group terms that frequently appear with one another in a set of documents. I get a dendrogram which verifies that things are working as expected. From this, I'd like to to retrieve the cluster of each member. I'm using cutree to cut the tree at a specified height, but it is returning one extra cluster, I'm not sure why.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;length(flex75$order.lab)&#xA;[1] 132&#xA;length(cutree(flex75, h = 1.5))&#xA;[1] 133&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It occurs for varying heights, even up to &lt;code&gt;h = max(flex75$height)&lt;/code&gt; and all clusters = 1. So I cannot bind the labels to their clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found the issue to be when I reorder &lt;code&gt;flex75$height&lt;/code&gt; to be in ascending order (I was previously receiving an error that: the 'height' component of 'tree' is not sorted (increasingly)). I was also reordering the &lt;code&gt;$order&lt;/code&gt; and &lt;code&gt;$order.lab&lt;/code&gt; components using the &lt;code&gt;$height&lt;/code&gt; order, and turns out that &lt;code&gt;height&lt;/code&gt; is of length n-1 and &lt;code&gt;order&lt;/code&gt;/&lt;code&gt;order.lab&lt;/code&gt; is of length n. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question now is: What is the relationship between the agnes object's &lt;code&gt;$height&lt;/code&gt; vector, and the &lt;code&gt;$order&lt;/code&gt;/&lt;code&gt;$order.lab&lt;/code&gt; vectors? Can I permute the heights, as I have, and feed that object into cutree? Should any of the object's other properties be correspondingly changed? &lt;/p&gt;&#xA;" OwnerUserId="23367" LastEditorUserId="11097" LastEditDate="2016-08-26T02:41:20.877" LastActivityDate="2016-08-26T02:41:20.877" Title="Feeding R agnes object into cutree" Tags="|r|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13673" PostTypeId="1" CreationDate="2016-08-25T19:36:28.910" Score="2" ViewCount="3589" Body="&lt;p&gt;I am trying to use &lt;code&gt;Logistic Regression&lt;/code&gt; to classify the datasets which has &lt;em&gt;Sparse Vector&lt;/em&gt; in feature vector:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Case 1&lt;/strong&gt;: I tried using the pipeline of ML in MLLIB as follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# used libraries&#xA;from pyspark.ml.feature import HashingTF&#xA;from pyspark.ml import Pipeline&#xA;from pyspark.ml.classification import LogisticRegression&#xA;&#xA;print(type(trainingData)) # for checking only&#xA;print(trainingData.take(2)) # To see the details of dataset&#xA;lr = LogisticRegression(labelCol=&quot;label&quot;, featuresCol=&quot;features&quot;, maxIter=maximumIteration,     regParam=re&#xA;gParamValue)&#xA;pipeline = Pipeline(stages=[lr])&#xA;# Train model&#xA;model = pipeline.fit(trainingData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Got the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;class 'pyspark.sql.dataframe.DataFrame'&amp;gt;&#xA;[Row(label=2.0, features=SparseVector(2000, {51: 1.0, 160: 1.0, 341: 1.0, 417: 1.0, 561: 1.0, 656: 1.0, 863: 1.0, 939: 1.0, 1021: 1.0, 1324: 1.0, 1433: 1.0, 1573: 1.0, 1604: 1.0, 1720: 1.0})), Row(label=3.0, features=SparseVector(2000, {24: 1.0, 51: 2.0, 119: 1.0, 167: 1.0, 182: 1.0, 190: 1.0, 195: 1.0, 285: 1.0, 432: 1.0, 539: 1.0, 571: 1.0, 630: 1.0, 638: 1.0, 656: 1.0, 660: 2.0, 751: 1.0, 785: 1.0, 794: 1.0, 801: 1.0, 823: 1.0, 893: 1.0, 900: 1.0, 915: 1.0, 956: 1.0, 966: 1.0, 1025: 1.0, 1029: 1.0, 1035: 1.0, 1038: 1.0, 1093: 1.0, 1115: 2.0, 1147: 1.0, 1206: 1.0, 1252: 1.0, 1261: 1.0, 1262: 1.0, 1268: 1.0, 1304: 1.0, 1351: 1.0, 1378: 1.0, 1423: 1.0, 1437: 1.0, 1441: 1.0, 1530: 1.0, 1534: 1.0, 1556: 1.0, 1562: 1.0, 1604: 1.0, 1711: 1.0, 1737: 1.0, 1750: 1.0, 1776: 1.0, 1858: 1.0, 1865: 1.0, 1923: 1.0, 1926: 1.0, 1959: 1.0, 1999: 1.0}))]&#xA;16/08/25 19:14:07 ERROR org.apache.spark.ml.classification.LogisticRegression: Currently, LogisticRegression with E&#xA;lasticNet in ML package only supports binary classification. Found 5 in the input dataset.&#xA;Traceback (most recent call last):&#xA;  File &quot;/home/LR/test.py&quot;, line 260, in &amp;lt;module&amp;gt;&#xA;    accuracy = TrainLRCModel(trainData, testData)&#xA;  File &quot;/home/LR/test.py&quot;, line 211, in TrainLRCModel&#xA;    model = pipeline.fit(trainingData)&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py&quot;, line 69, in fit&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py&quot;, line 213, in _fit&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py&quot;, line 69, in fit&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py&quot;, line 133, in _fit&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py&quot;, line 130, in _fit_java&#xA;  File &quot;/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py&quot;, line 813, in __call__&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py&quot;, line 45, in deco&#xA;  File &quot;/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py&quot;, line 308, in get_return_value&#xA;py4j.protocol.Py4JJavaError: An error occurred while calling o207.fit.&#xA;: org.apache.spark.SparkException: Currently, LogisticRegression with ElasticNet in ML package only supports binary&#xA; classification. Found 5 in the input dataset.&#xA;        at org.apache.spark.ml.classification.LogisticRegression.train(LogisticRegression.scala:290)&#xA;        at org.apache.spark.ml.classification.LogisticRegression.train(LogisticRegression.scala:159)&#xA;        at org.apache.spark.ml.Predictor.fit(Predictor.scala:90)&#xA;        at org.apache.spark.ml.Predictor.fit(Predictor.scala:71)&#xA;        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&#xA;        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)&#xA;        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#xA;        at java.lang.reflect.Method.invoke(Method.java:498)&#xA;        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)&#xA;        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)&#xA;        at py4j.Gateway.invoke(Gateway.java:259)&#xA;        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)&#xA;        at py4j.commands.CallCommand.execute(CallCommand.java:79)&#xA;        at py4j.GatewayConnection.run(GatewayConnection.java:209)&#xA;        at java.lang.Thread.run(Thread.java:745)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Case 2&lt;/strong&gt;: I search the possible alternate solution of above one and got that &lt;code&gt;LogisticRegressionWithLBFGS&lt;/code&gt; will work on multi-class classificaton, I tried as follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#used library&#xA;from pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel, LogisticRegressionWithSGD&#xA;print(type(trainingData)) # for checking only&#xA;print(trainingData.take(2)) # to see the dataset&#xA;model = LogisticRegressionWithLBFGS.train(trainingData, numClasses=5)&#xA;print(type(model))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Got the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;class 'pyspark.sql.dataframe.DataFrame'&amp;gt;&#xA;[Row(label=3.0, features=SparseVector(2000, {24: 1.0, 51: 2.0, 119: 1.0, 167: 1.0, 182: 1.0, 190: 1.0, 195: 1.0, 28&#xA;5: 1.0, 432: 1.0, 539: 1.0, 571: 1.0, 630: 1.0, 638: 1.0, 656: 1.0, 660: 2.0, 751: 1.0, 785: 1.0, 794: 1.0, 801: 1.&#xA;0, 823: 1.0, 893: 1.0, 900: 1.0, 915: 1.0, 956: 1.0, 966: 1.0, 1025: 1.0, 1029: 1.0, 1035: 1.0, 1038: 1.0, 1093: 1.&#xA;0, 1115: 2.0, 1147: 1.0, 1206: 1.0, 1252: 1.0, 1261: 1.0, 1262: 1.0, 1268: 1.0, 1304: 1.0, 1351: 1.0, 1378: 1.0, 14&#xA;23: 1.0, 1437: 1.0, 1441: 1.0, 1530: 1.0, 1534: 1.0, 1556: 1.0, 1562: 1.0, 1604: 1.0, 1711: 1.0, 1737: 1.0, 1750: 1&#xA;.0, 1776: 1.0, 1858: 1.0, 1865: 1.0, 1923: 1.0, 1926: 1.0, 1959: 1.0, 1999: 1.0})), Row(label=5.0, features=SparseV&#xA;ector(2000, {103: 1.0, 310: 1.0, 601: 1.0, 817: 1.0, 866: 1.0, 940: 1.0, 1023: 1.0, 1118: 1.0, 1339: 1.0, 1447: 1.0&#xA;, 1634: 1.0, 1776: 1.0}))]&#xA;Traceback (most recent call last):&#xA;  File &quot;/home/LR/test.py&quot;, line 260, in &amp;lt;module&amp;gt;&#xA;    accuracy = TrainLRCModel(trainData, testData)&#xA;  File &quot;/home/LR/test.py&quot;, line 230, in TrainLRCModel&#xA;    model = LogisticRegressionWithLBFGS.train(trainingData, numClasses=5)&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/mllib/classification.py&quot;, line 382, in train&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/mllib/regression.py&quot;, line 206, in _regression_train_wrapper&#xA;TypeError: data should be an RDD of LabeledPoint, but got &amp;lt;class 'pyspark.sql.types.Row'&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again I tried to convert the dataset into RDD of Labeled Point as follow i.e case 3:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Case 3&lt;/strong&gt;: Converted the dataset into RDD of Labeled Point so that I can use &lt;code&gt;LogisticRegressionWithLBFGS&lt;/code&gt; as follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    #used library&#xA;    from pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel, LogisticRegressionWithSGD&#xA;    from pyspark.mllib.regression import LabeledPoint&#xA;&#xA;    print(type(trainingData)) # For checking only&#xA;    print(trainingData.take(2)) # To see the datasets&#xA;    trainingData = trainingData.map(lambda row:[LabeledPoint(row.label,row.features)])&#xA;    print('type of trainingData')&#xA;    print(type(trainingData))&#xA;    print(trainingData.take(2))&#xA;    model = LogisticRegressionWithLBFGS.train(trainingData, numClasses=5)&#xA;    print(type(model))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Got the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;class 'pyspark.sql.dataframe.DataFrame'&amp;gt;&#xA;[Row(label=2.0, features=SparseVector(2000, {51: 1.0, 160: 1.0, 341: 1.0, 417: 1.0, 561: 1.0, 656: 1.0, 863: 1.0, 9&#xA;39: 1.0, 1021: 1.0, 1324: 1.0, 1433: 1.0, 1573: 1.0, 1604: 1.0, 1720: 1.0})), Row(label=3.0, features=SparseVector(&#xA;2000, {24: 1.0, 51: 2.0, 119: 1.0, 167: 1.0, 182: 1.0, 190: 1.0, 195: 1.0, 285: 1.0, 432: 1.0, 539: 1.0, 571: 1.0, &#xA;630: 1.0, 638: 1.0, 656: 1.0, 660: 2.0, 751: 1.0, 785: 1.0, 794: 1.0, 801: 1.0, 823: 1.0, 893: 1.0, 900: 1.0, 915: &#xA;1.0, 956: 1.0, 966: 1.0, 1025: 1.0, 1029: 1.0, 1035: 1.0, 1038: 1.0, 1093: 1.0, 1115: 2.0, 1147: 1.0, 1206: 1.0, 12&#xA;52: 1.0, 1261: 1.0, 1262: 1.0, 1268: 1.0, 1304: 1.0, 1351: 1.0, 1378: 1.0, 1423: 1.0, 1437: 1.0, 1441: 1.0, 1530: 1&#xA;.0, 1534: 1.0, 1556: 1.0, 1562: 1.0, 1604: 1.0, 1711: 1.0, 1737: 1.0, 1750: 1.0, 1776: 1.0, 1858: 1.0, 1865: 1.0, 1&#xA;923: 1.0, 1926: 1.0, 1959: 1.0, 1999: 1.0}))]&#xA;type of trainingData&#xA;&amp;lt;class 'pyspark.rdd.PipelinedRDD'&amp;gt;&#xA;[[LabeledPoint(2.0, (2000,[51,160,341,417,561,656,863,939,1021,1324,1433,1573,1604,1720],[1.0,1.0,1.0,1.0,1.0,1.0,1&#xA;.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]))], [LabeledPoint(3.0, (2000,[24,51,119,167,182,190,195,285,432,539,571,630,638,656&#xA;,660,751,785,794,801,823,893,900,915,956,966,1025,1029,1035,1038,1093,1115,1147,1206,1252,1261,1262,1268,1304,1351,&#xA;1378,1423,1437,1441,1530,1534,1556,1562,1604,1711,1737,1750,1776,1858,1865,1923,1926,1959,1999],[1.0,2.0,1.0,1.0,1.&#xA;0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0,1&#xA;.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]))]]&#xA;Traceback (most recent call last):&#xA;  File &quot;/home/LR/test.py&quot;, line 260, in &amp;lt;module&amp;gt;&#xA;    accuracy = TrainLRCModel(trainData, testData)&#xA;  File &quot;/home/LR/test.py&quot;, line 230, in TrainLRCModel&#xA;    model = LogisticRegressionWithLBFGS.train(trainingData, numClasses=5)&#xA;  File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/mllib/classification.py&quot;, line 381, in train&#xA;AttributeError: 'list' object has no attribute 'features'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone please suggest where I am missing something, I wanted to use the Logistic Regression in PySpark and classify the multi-class classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am using spark version &lt;code&gt;version 1.6.2&lt;/code&gt; and python version &lt;code&gt;Python 2.7.9&lt;/code&gt; on google cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanking you in advance for you kind help.&lt;/p&gt;&#xA;" OwnerUserId="17116" LastEditorUserId="17116" LastEditDate="2016-08-27T10:50:33.207" LastActivityDate="2019-07-12T08:06:25.827" Title="Logistic Regression as multiclass classification using PySpark and issues" Tags="|machine-learning|logistic-regression|multiclass-classification|pyspark|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13674" PostTypeId="2" ParentId="13671" CreationDate="2016-08-25T20:05:00.673" Score="0" Body="&lt;p&gt;I found an answer in a related post &lt;a href=&quot;https://stackoverflow.com/questions/9457473/strange-error-of-hierarchical-clustering-in-r&quot;&gt;here&lt;/a&gt;. Another user was getting a similar error about ordering &lt;code&gt;height&lt;/code&gt;, but theirs came with the suggestion to apply &lt;code&gt;as.hclust()&lt;/code&gt; first. I converted the &lt;code&gt;agnes&lt;/code&gt; object to an &lt;code&gt;hclust&lt;/code&gt; and passed that directly into &lt;code&gt;cutree&lt;/code&gt;. That seemed to solve the problem. &lt;/p&gt;&#xA;" OwnerUserId="23367" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-08-25T20:05:00.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13675" PostTypeId="1" CreationDate="2016-08-25T20:37:49.077" Score="1" ViewCount="261" Body="&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;x = np.linspace(0,10,20)&#xA;train_x = np.array([[i] for i in x])&#xA;train_y = np.sin(x)&#xA;&#xA;regressor = tf.contrib.learn.DNNRegressor(hidden_units=[10,20, 10])&#xA;regressor.fit(x = train_x, y = train_y, steps = 2000)&#xA;predictions = regressor.predict(x = train_x)&#xA;&#xA;plt.plot(x, train_y)&#xA;plt.plot(x, predictions)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It generates the following error message/warning:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/usr/bin/python3.4 /home/ttt/Dropbox/Programming/Python/Tensor_flow/one_dim_case.py&#xA;/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/array_ops.py:1197: &#xA;VisibleDeprecationWarning: converting an array with ndim &amp;gt; 0 to an &#xA;index will result in an error in the future&#xA;  result_shape.insert(dim, 1)&#xA;&#xA;Process finished with exit code 0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could you help me to understand what is wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An additional remark, with so many hidden layers and neurons and number of iterations, I really surprised with poor &lt;code&gt;sin&lt;/code&gt; approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. This is a toy example to help to understand tensorflow&lt;/p&gt;&#xA;" OwnerUserId="14999" LastEditorUserId="14999" LastEditDate="2016-08-25T20:55:15.173" LastActivityDate="2016-08-25T20:55:15.173" Title="Tensor flow error - conversion and peformance" Tags="|python|tensorflow|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13676" PostTypeId="1" CreationDate="2016-08-25T20:39:31.660" Score="5" ViewCount="3469" Body="&lt;p&gt;When it comes to convolutional neural networks there are normally many papers recommending different strategies. I have heard people say that it is an absolute must to add padding to the images before a convolution, otherwise to much spatial information is lost. On the other hand they are happy to use pooling, normally max-pooling, to reduce the size of the images. I guess the thought here is that max pooling reduces the spatial information but also reduces the sensitivity to relative positions, so it is a trade-off?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have heard other people saying that zero-padding does not keep more information, just more empty data. This is because by adding zeros you will not get a reaction from your kernel anyway when part of the information is missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine that zero-padding works if you have big kernels with &quot;scrap values&quot; in the edges and the source of activation centered in a smaller region of the kernel?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be happy to read some papers about the effect of down-sampling using pooling contra not using padding, but I can't find much about it. Any good recommendations or thoughts? &#xA;&lt;a href=&quot;https://i.stack.imgur.com/4smY9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4smY9.png&quot; alt=&quot;Figure: Spatial down-sampling using convolution contra pooling (Researchgate)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure: Spatial down-sampling using convolution contra pooling (Researchgate)&lt;/p&gt;&#xA;" OwnerUserId="23790" LastEditorUserId="29169" LastEditDate="2021-02-16T15:39:10.753" LastActivityDate="2021-02-16T15:39:10.753" Title="Max-pooling vs. zero padding: Loosing spatial information" Tags="|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13677" PostTypeId="1" CreationDate="2016-08-25T21:28:41.357" Score="2" ViewCount="36" Body="&lt;p&gt;I have a set of user transactions with a service and I suspect that there is a strong correlation between how users rate the service and how likely they are to use the service again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have chosen to represent user ratings - for each user in the dataset - as a number [1-5] which is the average of all ratings the given user has given in the past. There is more to it in regards to how to represent trends etc, but this is not in context for the problem at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My issue is that there is a high number of users that have never left any ratings at all and I am not sure how to deal with these users, in terms of finding the right value for the rating-related features for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have tried so far is to represent ratings in a different way, shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UID    rating_high    rating_medium    rating_low&#xA;  A              2                0             1&#xA;  B              0                0             0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where each of the predictors represent the count of ratings - in each rating category - by each of the users in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above case, &lt;code&gt;user A&lt;/code&gt; has left a good rating in 2 occasions, a low rating in one occasion and has never left a 'medium' rating. User B has never left any ratings for the service at all, therefore, he is assigned a count of &lt;code&gt;0&lt;/code&gt; for all 'rating categories'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on the 1-5 rating scale, I class all ratings below &lt;code&gt;3&lt;/code&gt; as &lt;code&gt;rating_low&lt;/code&gt;, all those equal to &lt;code&gt;3&lt;/code&gt; as &lt;code&gt;rating_medium&lt;/code&gt; and the ones above &lt;code&gt;3&lt;/code&gt; as &lt;code&gt;rating_high&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not managed to find any other way to represent this data, but I would very much rather not drop this part of the dataset as I believe it caries valuable semantics for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to best deal with this issue are most appreciated!&lt;/p&gt;&#xA;" OwnerUserId="17783" LastActivityDate="2016-08-25T21:28:41.357" Title="User ratings and conditional values" Tags="|machine-learning|dataset|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13678" PostTypeId="2" ParentId="12880" CreationDate="2016-08-25T21:31:03.837" Score="1" Body="&lt;p&gt;I found an answer to my question which in my opinion is correct (or at least has a point). Nevertheless it would be nice to have a validation from someone that might know more on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the average is calculated with a trapezoidal (which in reality is a summation) then the standard deviation can also be calculated by uncertainty propagation algebra for addition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Trapezoidal_rule&quot; rel=&quot;nofollow&quot;&gt;Wikipedia for trapezoidal rule&lt;/a&gt;: &#xA;$$ \int_c^d f(x) dx \approx (d-c) \left[ \frac{f(c)+f(d)}{2} \right] = \left( \frac{d-c}{2} \right) f(c) + \left( \frac{d-c}{2} \right) f(d)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which will follow the &lt;a href=&quot;https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulas&quot; rel=&quot;nofollow&quot;&gt;2nd uncertainty propagation rule from Wikipedia&lt;/a&gt;:&#xA;$$ f=aA+bB$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$ a = b = \frac{d-c}{2}$&lt;/li&gt;&#xA;&lt;li&gt;$ A = f(c) $&lt;/li&gt;&#xA;&lt;li&gt;$ B = f(d) $&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In terms of Python3 code that translates to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;# FUNCTIONS&#xA;def uncAddition(A=1.,B=1.,sigmaA=0.,sigmaB=0.,a=1.,b=1.,sigmaAB=0.):&#xA;    f =  (a * A) + (b * B)&#xA;    sigmaf = np.sqrt( (a * sigmaA)**2 \&#xA;                    + (b * sigmaB)**2 \&#xA;                    + (2 * a * b * sigmaAB) )&#xA;    return f, sigmaf&#xA;&#xA;&#xA;# MAIN&#xA;# Data: is an nx2 array that contains n number of f values &#xA;#           and their absolute standard deviation&#xA;# x:    is the independent variable of the f values&#xA;# Obviously x and f should have the same number of rows&#xA;&#xA;integralAve=0.&#xA;integralStD=0.&#xA;for i in range(1,len(data[:,0])):&#xA;    trapzPartAve,trapzPartStd = uncAddition(A=f[i-1,0],&#xA;                                            sigmaA=f[i-1,1],&#xA;                                            a=(x[i]-x[i-1])/2.0,&#xA;                                            B=f[i,0],&#xA;                                            sigmaB=f[i,1],&#xA;                                            b=(x[i]-x[i-1])/2.0) &#xA;    integralAve += trapzPartAve&#xA;    integralStd += trapzPartStd&#xA;&#xA;print(integralAve,integralStd)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21627" LastEditorUserId="21627" LastEditDate="2016-09-28T17:27:21.683" LastActivityDate="2016-09-28T17:27:21.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13679" PostTypeId="1" AcceptedAnswerId="13752" CreationDate="2016-08-26T00:11:40.687" Score="3" ViewCount="104" Body="&lt;p&gt;Suppose I am processing 3rd party vendor error log files that I am unable to change the schema from the source. And I am trying to predict a label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once logs are collected and I have all I need, in varying log formats, with each missing random information compared to the others.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-------------------------------------------------------&#xA;| vendor | variable_one | variable_two |variable_three|&#xA;-------------------------------------------------------&#xA;|    1   |       -      |&quot;some string&quot; |     NULL     |&#xA;-------------------------------------------------------&#xA;|    3   |      GET     |&quot;some string&quot; |     1270     |&#xA;-------------------------------------------------------&#xA;|  ...   |     POST     |    NULL      |      760     |&#xA;-------------------------------------------------------  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even after all the preprocessing and data transformations to change labels to Binary, and Categorical values and such. After cleaning, feature selection and training. Having a model trained, evaluated, and tested. &lt;em&gt;(label not depicted)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-------------------------------------------------------&#xA;| vendor | variable_one | variable_two |variable_three|&#xA;-------------------------------------------------------&#xA;|    1   |      GET     |&quot;some string&quot; |        0     |&#xA;-------------------------------------------------------&#xA;|    3   |      GET     |&quot;some string&quot; |     1270     |&#xA;-------------------------------------------------------&#xA;|  ...   |     POST     |     &quot;&quot;       |      760     |&#xA;-------------------------------------------------------  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;The questions are:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;When I go back to send my vendors logs to through the model, I am still sending it through with, potentially, missing values from what was altered, or selected as a feature for the model.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the model then useless?&lt;/li&gt;&#xA;&lt;li&gt;Do the transformations made in data cleansing need to be applied to each log entry that comes through, somewhere upstream?&lt;/li&gt;&#xA;&lt;li&gt;Assuming the model only knows &quot;nice pretty&quot; cleaned data, and I send log entries with missing values. Is the model making assumptions or less accurate prediction?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23800" LastEditorUserId="23800" LastEditDate="2016-12-22T01:02:21.810" LastActivityDate="2017-03-22T02:36:20.410" Title="Putting dirty data back into a model" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13680" PostTypeId="2" ParentId="1159" CreationDate="2016-08-26T03:35:43.640" Score="1" Body="&lt;p&gt;We implemented SVD to a larger data set using PySpark. We also compared consistency across different packages. Here is the &lt;a href=&quot;https://blog.dominodatalab.com/pca-on-very-large-neuroimaging-datasets-using-pyspark/&quot; rel=&quot;nofollow&quot;&gt;link.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23802" LastActivityDate="2016-08-26T03:35:43.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13682" PostTypeId="1" AcceptedAnswerId="13684" CreationDate="2016-08-26T06:01:44.333" Score="1" ViewCount="209" Body="&lt;p&gt;I am new to ds and stats in general. I read numerous article to understand logistic regression. I got some idea why it works and how it fits to scattered plot of 1s and 0s when target variable is binary. however one piece of puzzle I still don't understand is how come someone derived that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ln(p/1-p) = B0 + B1X1 + ..&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see all articles assume that this is the link function and then go on talking about how it solves our regression problem for binary variable. but how this link function came about. &lt;/p&gt;&#xA;" OwnerUserId="19147" LastActivityDate="2016-08-26T18:03:17.857" Title="logistic regression - why exponent (log ratio) is linear" Tags="|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13684" PostTypeId="2" ParentId="13682" CreationDate="2016-08-26T07:14:31.230" Score="4" Body="&lt;p&gt;p is a probability so it is strictly between 0 and 1. So ln(p/(1-p)) is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for p = 0: ln(0/1) = -Inf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for p = 1: ln(1/0) = +Inf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now you've rescaled the probability to +/- Inf. In the GLM framework you can use practically any function that scales probability to +/-Inf (see any GLM textbook, or &lt;a href=&quot;https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models&quot;&gt;https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How did it come about? Well, mathematicians and statisticians realised they needed a function with the above properties, had a think, came up with a few, decided on the ones that had tractable asymptotic properties, explored how they worked with real data and decided the sensible ones were logit (as above), probit (see references) and a few others. Of course you should always test your model assumptions against your data, and choice of link is just another one of those assumptions, just like assuming linearity in a covariate.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-08-26T07:14:31.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13685" PostTypeId="1" CreationDate="2016-08-26T07:40:03.340" Score="0" ViewCount="3241" Body="&lt;p&gt;I have two lists:&#xA;The first one includes sentences and the second one includes parts of speech (POS) tags. I want to use POS tags as a filter and find phrases according to a pattern such as 'JJ JJ NNS' (adjective+adjective+noun)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do that?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;&#xA;key_list= [' techniques from nonlinear analysis and partial differential equations form the basis for these studies .', &#xA; ' in this case , center-manifold theory can be applied to study the ordinary differential equations which replace the nonlinear \&#xA;partial differential equations for the wave motion .', &#xA;' differential equations are cool .']&#xA;&#xA;value_list = ['NNS IN JJ NN CC JJ JJ NNS VBP DT NN IN DT NNS .',&#xA; ' IN DT NN , JJ NN MD VB VBN TO VB DT JJ NN NNS WDT VB DT JJ JJ JJ NNS IN DT NN NN .', &#xA; ' JJ NNS VBP JJ .']&#xA;&#xA;pattern= re.compile('JJ JJ NNS',re.M|re.DOTALL)&#xA;&#xA;for value in value_list:&#xA;    value_output = pattern.findall(value)&#xA;    print(value_output)&#xA;&#xA;# I want  partial differential equations and ordinary differential equations&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3151" LastActivityDate="2016-08-26T16:25:07.717" Title="Filter phrases based on correspondent POS tags" Tags="|python|text-mining|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13687" PostTypeId="1" AcceptedAnswerId="13688" CreationDate="2016-08-26T08:12:11.207" Score="-1" ViewCount="63" Body="&lt;p&gt;In convolutional neutral network, the weights are shared within a feature map. What about two different feature map? How to make them different (so that we don't learn the same thing again).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; What exactly in the training algorithm to make it so that the weights are different across &lt;em&gt;different feature maps&lt;/em&gt;. For example, if I define 2 feature maps, does the network guarantee that the weights are different in feature map A and feature map B?&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-08-26T08:40:49.523" LastActivityDate="2016-08-26T09:17:33.267" Title="Shared weights in convolutional neutral network" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13688" PostTypeId="2" ParentId="13687" CreationDate="2016-08-26T09:15:50.780" Score="0" Body="&lt;p&gt;By randomizing your initial weights, the gradients will flow differently through the network. If all the feature maps would converge to the same weight set, there is much less to learn and your loss will be higher than learning different features. Let's say they start very, very close to each other, then making them more different from eachother makes the network more expressive. Due to the nature of backpropagation these weights will diverge as opposed to converge (in general).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-26T09:15:50.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13692" PostTypeId="1" CreationDate="2016-08-26T10:45:55.480" Score="1" ViewCount="334" Body="&lt;p&gt;I am using KNIME MLP Neural Network learner (If you are not familiarized with KNIME think of that like a package which implements Neural Network to a set of data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is you can tune the number of hidden layers, the number of iterations and neurons per layers, so here comes the first question: How to know a priori values for these parameters? For instance, for Random Forest 500 trees is supposed to be a standard to test its performance in the first stages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, I know in general add more layers and/or neurons does not ensure any enhance in the prediction, sometimes even the opposite happen. Someone could give some theoric insights respect this topic; why by adding more layers the prediction might get deteriorated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sergi&lt;/p&gt;&#xA;" OwnerUserId="23567" LastActivityDate="2016-08-26T12:05:56.860" Title="Neural Network - Adjust number of hidden layers and neurons" Tags="|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13693" PostTypeId="2" ParentId="13692" CreationDate="2016-08-26T12:05:56.860" Score="1" Body="&lt;p&gt;Sorry for not knowing exact number for the first question, but my point is to start from the most simple model, rather than any random parameter. Then use a grid search for the parameter by validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second question, the error of a model comes from two parts, variance and bias. Bias comes from lack of complexity of your model. For example, there will always be a large bias when using linear model to fit the distance v.s. the shooting angle of a canon. This can be overcome by using a more complex model,  adding layers to the NN for example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And variances comes from being too sensitive to the data, even to the noise in the data. When there is a high variance due to using a too complex model, it is called over-fitting. You may take a look at this page&lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;(over-fitting on wiki)&lt;/a&gt;. And using more layers/neurons is using a more complex model, which means the model may suffer from high variance. This can be overcome by lower the model complexity(which may increase the bias) or adding training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically if you got constant number of data, then you have to find somewhere where both bias and variance is not too high. &lt;/p&gt;&#xA;" OwnerUserId="23601" LastActivityDate="2016-08-26T12:05:56.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13694" PostTypeId="1" CreationDate="2016-08-26T13:13:06.470" Score="0" ViewCount="292" Body="&lt;p&gt;I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Purchase_ID Product_ID  Purchase_Value&#xA;    1       2           4.5&#xA;    1       3           1.2 &#xA;    2       3           1.4 &#xA;    2       1           3.5&#xA;    2       2           7.3&#xA;    3       2           0.5&#xA;    3       3           1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I want to conclude is that:&#xA;&quot;Every people that by Product_ID 2 also buy 3&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone knows If exists any code algorithm available to use in Spark Mllib? I already search on internet but I didn't found anything...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone can help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-09-25T21:26:53.020" Title="Spark algorithm to make a link analysis" Tags="|apache-spark|apache-hadoop|association-rules|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13695" PostTypeId="1" CreationDate="2016-08-26T14:16:14.003" Score="3" ViewCount="591" Body="&lt;p&gt;I have millions of user ratings on about 2k products. I want to use Machine Learning to analyse these ratings and recommend products to users based on other users ratings of the same and different products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. user A likes 1,2 &amp;amp; 3, user B likes 1&amp;amp;2 so probably likes 3 based on A's ratings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this possible to do through AWS ML? I'm looking at AWS ML as I'm a developer, not a data scientist and looking to keep this simple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been playing with this all day and think I have figured it out, but just looking for some further advise / guidance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my experimenting so far, the following format of dataset is what AWS needs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CustomerID, Prod1Rating, Prod2Rating, Prod3Rating,.....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then I would create a model for each product with the target on each being the row containing that products ratings. To make a recommendations engine from this I then just need to loop through every product for a user and ask each model the score.&lt;/p&gt;&#xA;" OwnerUserId="23818" LastActivityDate="2022-08-11T08:21:16.433" Title="Using AWS ML to recommend products" Tags="|machine-learning|recommender-system|aws|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13696" PostTypeId="1" AcceptedAnswerId="13698" CreationDate="2016-08-26T15:29:46.940" Score="3" ViewCount="2272" Body="&lt;p&gt;I supposed that the output layer should have certain kind of activation function (preferably linear or tanh) for &lt;strong&gt;regression&lt;/strong&gt;, but I recently read that in case of regression this is not necessary. This seems me reasonable, after all, the node(s) of the output layer produce(s) numeric values themselves. Which solution is better: with or without activation func?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-08-26T16:16:42.763" Title="Lack of activation function in output layer at regression?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13697" PostTypeId="2" ParentId="13695" CreationDate="2016-08-26T15:59:57.723" Score="2" Body="&lt;p&gt;This is a classical example of collaborative filtering, in particular you're talking about user-based collaborative filtering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think Amazon ML supports collaborative filtering (&lt;a href=&quot;https://forums.aws.amazon.com/thread.jspa?messageID=712158&quot; rel=&quot;nofollow&quot;&gt;https://forums.aws.amazon.com/thread.jspa?messageID=712158&lt;/a&gt;), but you can do it in R and run your R code on the cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy in R, take a look at the &lt;code&gt;recommenderlab&lt;/code&gt; package. The vignette has an example with the MovieLens movie rating data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also use the Microsoft Azure ML framework as it has better support for recommender systems. Google &quot;Azure recommender systems&quot; will give you useful links.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-08-26T15:59:57.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13698" PostTypeId="2" ParentId="13696" CreationDate="2016-08-26T16:08:43.000" Score="3" Body="&lt;p&gt;Activation &quot;linear&quot; is identical to &quot;no activation function&quot;. The term &quot;linear output layer&quot; also means precisely &quot;the last layer has no activation function&quot;. Whether you use one or the other term might be down to how your NN library implements it. You may also see it described either way around in documents, but it is exactly the same thing mathematically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a^{out}_j = b^{out}_j + \sum_{i=1}^{N^{hidden}} W_{ij}a^{hidden}_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $a$ values are activation, $b$ are biases, $W$ is weight matrix.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a regression problem with a mean squared error objective, this is the most common approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing stopping you using other activation functions. They might help you if they match the target variable distribution. About the only rule is that your network output should be able to cover possible values of the target variable. So if the target variable is always between -1.0 and 1.0, with higher density around 0.0, perhaps tanh could also work for you.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-08-26T16:16:42.763" LastActivityDate="2016-08-26T16:16:42.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13699" PostTypeId="2" ParentId="13685" CreationDate="2016-08-26T16:25:07.717" Score="3" Body="&lt;p&gt; Hey, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's how i would solve this.&#xA;The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it is easier to find the pattern in the tag list and get the matching words.&lt;/p&gt; &#xA;&#xA;&lt;pre&gt;&lt;code&gt;words = [word for key in key_list for word in key.split()]  &#xA;tags = [tag for value in value_list for tag in value.split()]  &#xA;&#xA;pattern = ['JJ', 'JJ', 'NNS']&#xA;matches = []&#xA;&#xA;for i in range(len(tags)):&#xA;    if tags[i:i+len(pattern)] == pattern:&#xA;        matches.append(words[i:i+len(pattern)])&#xA;&#xA;matches = [' '.join(match) for match in matches]&#xA;print(matches)&#xA;&#xA;# result : ['partial differential equations', 'partial differential equations']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, if I'm not mistaken your pos-tagger marked &quot;&lt;b&gt;differential&lt;/b&gt;&quot; as a noun (NN) in &quot;&lt;b&gt;ordinary differential equations&lt;/b&gt;&quot;. Nltk tagger does the job and it is really fast to implement but it is not perfect.&lt;/p&gt; &#xA;&#xA;&lt;p&gt;You can try using another tagger, I personally use the Stanford one for nlp processing.&lt;/p&gt;&#xA;" OwnerUserId="21719" LastActivityDate="2016-08-26T16:25:07.717" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13700" PostTypeId="2" ParentId="13682" CreationDate="2016-08-26T18:03:17.857" Score="2" Body="&lt;p&gt;One intuitive answer as to why &lt;code&gt;logistic function&lt;/code&gt; comes up is to look at &lt;code&gt;Logistic Regression&lt;/code&gt; from the angle of generative model, which results in &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_discriminant_analysis&quot; rel=&quot;nofollow&quot;&gt;Linear Discriminant Analysis&lt;/a&gt; model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the idea is that instead of directly modeling the &lt;code&gt;likelihood&lt;/code&gt; $p(y|x)$ like in &lt;code&gt;logistic regression&lt;/code&gt;. You model the &lt;code&gt;class-conditional&lt;/code&gt; $p(x|y)$ and $p(y)$, then derive the output $p(y)$ via Bayes' rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(y|x) = \frac{p(x|y) p(y)}{p(x)} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out that if you model the input by a Gaussian distribution or any distribution in the exponential family (with the same dispersion parameter for two classes), then your likelihood is a &lt;code&gt;logistic function&lt;/code&gt; of $x$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(y = 1 | x) = \frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1) + p(x|y=0)p(y=0)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &#xA;$$ p(x|y=1) = \mathcal{N(\mu_1, \Sigma)} $$ &#xA;$$ p(x|y=1) = \mathcal{N(\mu_0, \Sigma)} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;after some simplification&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(y=1|x) = \frac{1}{1 + exp(-w^Tx - b)} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $ w = \Sigma^{-1}(\mu_0 - \mu_1) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That explains why the &lt;code&gt;logistic function&lt;/code&gt; turns up&lt;/p&gt;&#xA;" OwnerUserId="14313" LastActivityDate="2016-08-26T18:03:17.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13701" PostTypeId="1" AcceptedAnswerId="14426" CreationDate="2016-08-26T18:03:38.557" Score="0" ViewCount="2093" Body="&lt;p&gt;I'm trying to do a algorithm in Spark Mllib. I'm trying to do a Market Basket Analysis. And I've as a data set this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  ID bigint,&#xA;  Chain int,&#xA;  Dept int,&#xA;  Category int,&#xA;  Company bigint,&#xA;  Brand bigint,&#xA;  Date string,&#xA;  Product_Size double,&#xA;  Product_Measure string,&#xA;  Purchase_Quantity int,&#xA;   Purchase_Amount double&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code that I found is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.fpm.FPGrowth&#xA;import org.apache.spark.rdd.RDD&#xA;import sys.process._&#xA;&#xA;val data = sc.textFile(&quot;/user/admin/retail/marketbaskets/part-r-00000&quot;)&#xA;&#xA;val transactions: RDD[Array[String]] = data.map(s =&amp;gt; s.trim.split(','))&#xA;&#xA;val fpg = new FPGrowth()&#xA;  .setMinSupport(0.007)&#xA;  .setNumPartitions(10)&#xA;val model = fpg.run(transactions)&#xA;&#xA;model.freqItemsets.collect().foreach { itemset =&amp;gt;&#xA;  println(itemset.items.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;) + &quot;, &quot; + itemset.freq)&#xA;}&#xA;&#xA;val minConfidence = 0.8&#xA;model.generateAssociationRules(minConfidence).collect().foreach { rule =&amp;gt;&#xA;  println(&#xA;    rule.antecedent.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;)&#xA;      + &quot; =&amp;gt; &quot; + rule.consequent .mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;)&#xA;      + &quot;, &quot; + rule.confidence)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However when I submit this line of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val model = fpg.run(transactions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the fowllowing error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val model = fpg.run(transactions)&#xA;16/08/26 10:56:09 WARN fpm.FPGrowth: Input data is not cached.&#xA;16/08/26 10:56:21 ERROR executor.Executor: Exception in task 0.0 in stage 10.0 (TID 7)&#xA;org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(13873775, 4, 99, 9909, 102113020, 15704, 2012-03-19:00, 6.25, OZ, 4, 11.96).&#xA;    at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:143)&#xA;	at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:140)&#xA;	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)&#xA;    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)&#xA;	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)&#xA;	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)&#xA;	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)&#xA;	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)&#xA;	at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)&#xA;	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)&#xA;	at java.lang.Thread.run(Thread.java:745)&#xA;16/08/26 10:56:21 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 10.0 (TID 7, localhost): org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(13873775, 4, 99, 9909, 102113020, 15704, 2012-03-19:00, 6.25, OZ, 4, 11.96).&#xA;	at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:143)&#xA;	at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:140)&#xA;    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)&#xA;	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)&#xA;	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)&#xA;	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)&#xA;	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)&#xA;	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)&#xA;	at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)&#xA;    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)&#xA;    at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;16/08/26 10:56:21 ERROR scheduler.TaskSetManager: Task 0 in stage 10.0 failed 1 times; aborting job&#xA;org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 10.0 failed 1 times, most recent failure: Lost task 0.0 in stage 10.0 (TID 7, localhost): org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(13873775, 4, 99, 9909, 102113020, 15704, 2012-03-19:00, 6.25, OZ, 4, 11.96).&#xA;    at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:143)&#xA;	at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:140)&#xA;	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)&#xA;    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)&#xA;    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)&#xA;    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)&#xA;    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)&#xA;    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)&#xA;    at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)&#xA;    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)&#xA;    at java.lang.Thread.run(Thread.java:745)&#xA;&#xA;Driver stacktrace:&#xA;    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)&#xA;	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)&#xA;    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)&#xA;	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)&#xA;	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)&#xA;	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)&#xA;	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)&#xA;    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)&#xA;	at scala.Option.foreach(Option.scala:236)&#xA;	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)&#xA;	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)&#xA;	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)&#xA;	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)&#xA;	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)&#xA;	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)&#xA;	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1843)&#xA;	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1856)&#xA;	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1869)&#xA;	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1940)&#xA;	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)&#xA;	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)&#xA;    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)&#xA;	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)&#xA;	at org.apache.spark.rdd.RDD.collect(RDD.scala:926)&#xA;	at org.apache.spark.mllib.fpm.FPGrowth.genFreqItems(FPGrowth.scala:149)&#xA;	at org.apache.spark.mllib.fpm.FPGrowth.run(FPGrowth.scala:118)&#xA;	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:38)&#xA;	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:43)&#xA;	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:45)&#xA;    at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:47)&#xA;	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:49)&#xA;	at $iwC$$iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:51)&#xA;	at $iwC$$iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:53)&#xA;    at $iwC$$iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:55)&#xA;	at $iwC$$iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:57)&#xA;	at $iwC.&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:59)&#xA;	at &amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:61)&#xA;	at .&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:65)&#xA;	at .&amp;lt;clinit&amp;gt;(&amp;lt;console&amp;gt;)&#xA;	at .&amp;lt;init&amp;gt;(&amp;lt;console&amp;gt;:7)&#xA;	at .&amp;lt;clinit&amp;gt;(&amp;lt;console&amp;gt;)&#xA;	at $print(&amp;lt;console&amp;gt;)&#xA;	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&#xA;	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)&#xA;	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#xA;	at java.lang.reflect.Method.invoke(Method.java:606)&#xA;	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)&#xA;	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)&#xA;	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)&#xA;	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)&#xA;	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)&#xA;	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)&#xA;	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)&#xA;	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)&#xA;	at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)&#xA;	at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)&#xA;	at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)&#xA;    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)&#xA;	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)&#xA;	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)&#xA;    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)&#xA;	at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)&#xA;	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)&#xA;	at org.apache.spark.repl.Main$.main(Main.scala:31)&#xA;    at org.apache.spark.repl.Main.main(Main.scala)&#xA;    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&#xA;    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)&#xA;    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&#xA;    at java.lang.reflect.Method.invoke(Method.java:606)&#xA;    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)&#xA;	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)&#xA;	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)&#xA;    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)&#xA;	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)&#xA;Caused by: org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(13873775, 4, 99, 9909, 102113020, 15704, 2012-03-19:00, 6.25, OZ, 4, 11.96).&#xA;	at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:143)&#xA;    at org.apache.spark.mllib.fpm.FPGrowth$$anonfun$1.apply(FPGrowth.scala:140)&#xA;	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)&#xA;	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)&#xA;    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)&#xA;    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)&#xA;    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)&#xA;    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)&#xA;    at org.apache.spark.scheduler.Task.run(Task.scala:89)&#xA;    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)&#xA;	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)&#xA;	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)&#xA;    at java.lang.Thread.run(Thread.java:745)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I already try to use this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val model = fpg.run(transactions.values.map(_.toArray))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it still gives me an error. I also see in the Hive if this error was a duplicate case but it isn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone knows how can I solve this error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23811" LastActivityDate="2016-10-09T08:49:45.707" Title="Items in a transaction must be unique but got WrappedArray" Tags="|apache-spark|scala|market-basket-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13702" PostTypeId="2" ParentId="13694" CreationDate="2016-08-26T20:33:13.727" Score="1" Body="&lt;p&gt;This sounds like a classic frequent pattern mining problem, where you're trying to find sets of items that are frequently found together within users' sets of purchases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start here: &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html&quot; rel=&quot;nofollow&quot;&gt;https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-08-26T20:33:13.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13706" PostTypeId="1" CreationDate="2016-08-27T08:44:45.513" Score="0" ViewCount="125" Body="&lt;p&gt;Suppose I write a program in Keras for MNIST data set. I used TensorFlow as my backend in Keras. Is it possible in any way that I can extract the backend code used for TensorFlow that has been used by Keras. if yes, please provide an example.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T15:13:58.823" LastActivityDate="2020-12-25T15:13:58.823" Title="Extracting the code from Keras" Tags="|machine-learning|deep-learning|keras|tensorflow|code|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13707" PostTypeId="1" AcceptedAnswerId="13708" CreationDate="2016-08-27T06:03:34.000" Score="1" ViewCount="75" Body="&lt;p&gt;I have to develop a video event detection tool in ticketing counter.The tool must take photographs of persons who jumb over gate without taking tickets.I have a set of videos in which people jumbing over gate.So with that data in hand How can i implement a Video event detection tool?I am new to video analytics.I don't know where to start.Where can i find some good tutorial about supervised video event detection.? &lt;/p&gt;&#xA;" OwnerUserId="23847" OwnerDisplayName="Ceem" LastActivityDate="2016-08-27T19:26:56.553" Title="Which Algorithm or combination of Algorithms to use to develop supervised Video Event detection?" Tags="|machine-learning|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13708" PostTypeId="2" ParentId="13707" CreationDate="2016-08-27T19:26:56.553" Score="2" Body="&lt;p&gt;I recommend using a 3D convoluted neural network. Traditional 2D convoluted neural networks excel at identifying objects in images, and 3D convolution is used for identifying objects or movements also across multiple frames (i.e., time). &lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/api_docs/python/nn.html#conv3d&quot; rel=&quot;nofollow&quot; title=&quot;tensorflow&quot;&gt;Tensorflow&lt;/a&gt; offers this capability. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-08-27T19:26:56.553" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13710" PostTypeId="1" CreationDate="2016-08-27T20:43:47.423" Score="0" ViewCount="1142" Body="&lt;p&gt;I have a large number of variables (2000) and want to visualize them using Python. Also, I would like to see the summary statistics of these variables. It seems that drawing box plots for all of these variables would not be feasible. What are some methods that could be used to visualize this data?&lt;/p&gt;&#xA;" OwnerUserId="23845" LastActivityDate="2020-08-04T09:33:06.660" Title="Visualizing a large number of variables in python" Tags="|python|visualization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13711" PostTypeId="1" CreationDate="2016-08-28T10:07:15.910" Score="2" ViewCount="2215" Body="&lt;p&gt;I'm rather new to Word2Vec, having started working on it about a week ago.&lt;/p&gt;&#xA;&lt;p&gt;My question is this: Is there a way to obtain frequently occurring phrases in a large document using Word2Vec along with a score to denote the 'frequency'?&lt;/p&gt;&#xA;" OwnerUserId="23850" LastEditorUserId="85045" LastEditDate="2020-12-23T17:29:09.617" LastActivityDate="2020-12-23T17:29:18.173" Title="Obtaining frequently occurring phrases using Word2Vec" Tags="|machine-learning|nlp|word2vec|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13712" PostTypeId="1" CreationDate="2016-08-28T10:36:02.307" Score="0" ViewCount="1054" Body="&lt;p&gt;I Have a general question - how to preprocess data in Orange. E.g. I want to do something with each row value of some feature. How can I do this visually? And how can I do this via Python Script?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have only some ideas how to do this via python script:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minimum = min(list(map(lambda row: row[0], in_data))).value&#xA;&#xA;from copy import deepcopy&#xA;out_data = deepcopy(in_data)&#xA;&#xA;for row in out_data:&#xA;    row[0].value = row[0].value - minimum&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it fails, because seems like the Value object is immutable... &#xA;Is there any way to create my own Value object with new data, and then do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out_data = list(map(lambda row: Value(row[0].variable, [row[0].value - minimum]) + row[1:], in_data))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am also interested in a &quot;Visual widget-based&quot; alternative, if it exists...&lt;/p&gt;&#xA;" OwnerUserId="23851" LastActivityDate="2016-08-29T16:03:29.657" Title="How to preprocess data in a data table? (e.g. change a value in each row for some feature)" Tags="|orange|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13713" PostTypeId="1" CreationDate="2016-08-28T11:01:06.473" Score="2" ViewCount="149" Body="&lt;pre&gt;&lt;code&gt;var digits = [&quot;0&quot;, &quot;1&quot;, &quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;, &quot;6&quot;, &quot;7&quot;, &quot;8&quot;, &quot;9&quot;];&#xA;var numbers = [digits[0], digits[1]...digits[9], digits[1] + digits[0], digits[1] + digits[1]...];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would you use to build a model that can &lt;code&gt;predictNextNumber(19)=20&lt;/code&gt; by understanding how counting works?&lt;/p&gt;&#xA;" OwnerUserId="23853" LastEditorDisplayName="user8022" LastEditDate="2016-08-29T09:20:07.273" LastActivityDate="2016-08-29T09:20:07.273" Title="How would you teach a machine how to count?" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13714" PostTypeId="2" ParentId="13713" CreationDate="2016-08-28T11:55:02.967" Score="0" Body="&lt;p&gt;Feed enough examples into a learning algorithm. In pseudocode:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L = learn_plus_one()&#xA;L.teach(&quot;12&quot;,&quot;13&quot;)&#xA;L.teach(&quot;3&quot;,&quot;4&quot;)&#xA;L.teach(&quot;99&quot;,&quot;100&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;after being taught enough examples, it should figure out that 73 plus one is 74, even though it really knows nothing about addition, its just discovered the pattern from the examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what does &lt;code&gt;learn_plus_one&lt;/code&gt; look like? Well that could be a neural network, or any other machine learning system really. Your question is quite vague so I cant be specific.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-08-28T11:55:02.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13716" PostTypeId="1" CreationDate="2016-08-28T12:15:11.720" Score="-1" ViewCount="158" Body="&lt;p&gt;Subj.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the following in Scatter Plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     .&#xA;&#xA;.  .   .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to get something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     .&#xA;    / \&#xA;.__.   .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;?&lt;/p&gt;&#xA;" OwnerUserId="23851" LastActivityDate="2016-10-28T17:15:56.013" Title="How to connect dots with lines in Scatter Plot or some other Plot?" Tags="|orange|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13717" PostTypeId="2" ParentId="13712" CreationDate="2016-08-28T12:18:55.437" Score="0" Body="&lt;p&gt;Something like this worked for my case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minimum = min(list(map(lambda row: row[0].value, in_data)))&#xA;&#xA;from copy import deepcopy&#xA;out_data = deepcopy(in_data)&#xA;&#xA;import Orange&#xA;for i in range(len(out_data)):&#xA;    out_data[i,0] = Orange.data.Value(out_data[i,0].variable, out_data[i,0].value - minimum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Interesting is that in order to plot the out_data, I needed additional &quot;Select Columns&quot; widget before &quot;Scatter Plot&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition I couldn't combine two out_data (one from one source and one from another) on one Scatter Plot... But this seems to be an other question...&lt;/p&gt;&#xA;" OwnerUserId="23851" LastActivityDate="2016-08-28T12:18:55.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13718" PostTypeId="2" ParentId="13713" CreationDate="2016-08-28T15:27:26.233" Score="3" Body="&lt;p&gt;I suggest using &lt;a href=&quot;https://en.wikipedia.org/wiki/Supervised_learning&quot; rel=&quot;nofollow&quot;&gt;supervised learning&lt;/a&gt; and employing a linear model: &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow&quot;&gt;linear regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a perfectly linear system (y=x+1), so linear regression will work just fine i.e. perfectly. Further, you have an infinite amount of data you can use to train the system, so it should be easy to train ;-)  I jest... I think that two data points will be sufficient, again, since it is perfectly linear!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pedagogically, the triviality of this extremely simple linear system gets a little more interesting when you try analogue based methods like support vector machine (SVM) - which should also be able to provide a perfect result, decision trees or random forests, and even naive Bayes regressors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though its useful to start learning with very simple systems, I suggest quickly moving on to something more complex, i.e. don't get too stuck in your own head for the trivial linear model. Don't forget that the power of data science and machine learning lies in its statistical nature, so try to find a test case that includes some statistical variation in the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-08-28T15:27:26.233" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13719" PostTypeId="1" CreationDate="2016-08-28T20:25:55.997" Score="2" ViewCount="1086" Body="&lt;p&gt;&lt;a href=&quot;http://image-net.org/challenges/LSVRC/2016/index#comp&quot; rel=&quot;nofollow&quot;&gt;Here&lt;/a&gt;'s the description about the data usage for &lt;code&gt;ILSVRC 2016&lt;/code&gt; of &lt;code&gt;ImageNet&lt;/code&gt;. I've interpreted it with the table as follows,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data          Image        Label    Changed    Release_Time&#xA;-----------------------------------------------------------------&#xA;training      1,200,000    1,000    no         always public  &#xA;validation       50,000    1,000    some       upon registration&#xA;test            100,000        0    some       final evaluation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is specific about the &lt;code&gt;test data&lt;/code&gt;. Is it chosen from the &lt;strong&gt;same&lt;/strong&gt; set of 150,000 images since &lt;code&gt;ILSVRC 2012&lt;/code&gt;, although randomly ? If so, most of the &lt;code&gt;test data&lt;/code&gt; has been made public before the final evaluation phase of the challenge, how the result could remain fair enough ?&lt;/p&gt;&#xA;" OwnerUserId="23862" LastActivityDate="2024-02-20T08:01:50.203" Title="Test data from ImageNet" Tags="|machine-learning|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13722" PostTypeId="1" CreationDate="2016-08-29T06:17:24.637" Score="1" ViewCount="7324" Body="&lt;p&gt;I'm trying to assign a vector to every word in my sentence.&#xA;Some words are not recognised even though very similar words are - for example: &#xA;&lt;code&gt;Going&lt;/code&gt;, &lt;code&gt;gone&lt;/code&gt;, &lt;code&gt;go&lt;/code&gt; are recognised while &lt;code&gt;goes&lt;/code&gt; isn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I go about assigning any logical values to the word goes or any word of this like? Please notice that I don't know in advance what words won't be recognised.&lt;/p&gt;&#xA;" OwnerUserId="4953" LastActivityDate="2016-12-27T15:54:41.523" Title="Word2Vec how to treat words that aren't in the vocabulary" Tags="|word2vec|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13723" PostTypeId="1" CreationDate="2016-08-29T06:40:36.440" Score="2" ViewCount="221" Body="&lt;p&gt;One of the most crucial skills of a data scientist is not only to be able to build an accurate predictive algorithm but to suggest a set of actions based on that to enhance the goal ratios.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have build a churn rate predictor for a retail APP which accuracy after testing is about 90%. So now that I know with high accuracy which users will churn out of the APP in the next days, what can I do to try to retain them. Thinking in the overall APP environment the only idea that comes to my mind is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Push notifications. CRM, use a recommender system to send push notifications based on users profile and micro-moments.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23567" LastEditorUserId="29575" LastEditDate="2017-11-15T08:51:20.457" LastActivityDate="2017-11-15T08:51:20.457" Title="Your Prediction Model works. What to do after?" Tags="|prediction|churn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13724" PostTypeId="2" ParentId="13723" CreationDate="2016-08-29T07:47:27.270" Score="2" Body="&lt;p&gt;You could do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find out the most important predictors used by your model; the methods used to identify importance depend on the model used.&lt;/li&gt;&#xA;&lt;li&gt;For each of the major pedictors try to find out how does each predictor segment the customer population of will-churn vs. will-not-churn. That would be the starting point for framing hypothesis of why these set of customers are churning. Generate reports/info-graphics showing how these predictors impact the tendency to churn. Sometimes predictors are not causes but side-effects of the hidden cause showing up elsewhere. So be open to skepticism about the predictors.&lt;/li&gt;&#xA;&lt;li&gt;Involve the business/users and have brainstorming sessions to go through each of these hypothesis and evolve the most practical and economic actions that could be taken to reduce churn. Before you get into a discussion, plan for a sufficient amount of time, come up with your best thought actions which you feel will address the situation and have all the details gleaned from your analysis available for sharing in the session.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is an art more than a science and your ability to understand the domain as well as good communication will help you achieve this goal.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-08-29T07:47:27.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13726" PostTypeId="1" AcceptedAnswerId="13737" CreationDate="2016-08-29T09:21:29.577" Score="2" ViewCount="187" Body="&lt;p&gt;Suppose I have a dataset of size(10000, 45). One of the features in the dataset is activity_type in which the values vary from 1 to 15 as shown below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_csv('actTrain.csv')&#xA;df['activity_type'].head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The output of the above code is as:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;0    1&#xA;1    1&#xA;2    2&#xA;3    1&#xA;4    3&#xA;Name: activity_type, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Will encoding the activity_type in the above code using OneHotEncoder in sklearn improve the model in anyway? Is it necessary to encode that feature? And if yes, which one should I choose : LabelEncoder or OneHotEnocder?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T10:07:39.987" LastActivityDate="2020-12-25T10:07:39.987" Title="Encoding features in sklearn" Tags="|machine-learning|scikit-learn|feature-engineering|encoding|one-hot-encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13727" PostTypeId="1" CreationDate="2016-08-29T09:24:02.927" Score="0" ViewCount="2059" Body="&lt;p&gt;I try to build a NN classifier on the well-known MNIST image database with Sklearn's Grid Search according the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = KerasClassifier(build_fn=create_model, verbose=1)&#xA;param_grid = dict(batch_size=[10, 50, 100, 250], nb_epoch=[10, 50, 100])&#xA;grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)&#xA;grid_result = grid.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;create_model is a function that builds the Neural Network Model. The fitting (last row) gives a long error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;JoblibTypeError                           Traceback (most recent call last)&#xA;&amp;lt;ipython-input-109-bcb5d85cabad&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      2 param_grid = dict(optimizer=['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'])&#xA;      3 grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=10)&#xA;----&amp;gt; 4 grid_result = grid.fit(X_train, Y_train)&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/grid_search.pyc in fit(self, X, y)&#xA;    802 &#xA;    803         &quot;&quot;&quot;&#xA;--&amp;gt; 804         return self._fit(X, y, ParameterGrid(self.param_grid))&#xA;    805 &#xA;    806 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)&#xA;    551                                     self.fit_params, return_parameters=True,&#xA;    552                                     error_score=self.error_score)&#xA;--&amp;gt; 553                 for parameters in parameter_iterable&#xA;    554                 for train, test in cv)&#xA;    555 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)&#xA;    808                 # consumption.&#xA;    809                 self._iterating = False&#xA;--&amp;gt; 810             self.retrieve()&#xA;    811             # Make sure that we get a last message telling us we are done&#xA;    812             elapsed_time = time.time() - self._start_time&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in retrieve(self)&#xA;    755                     # a working pool as they expect.&#xA;    756                     self._initialize_pool()&#xA;--&amp;gt; 757                 raise exception&#xA;    758 &#xA;    759     def __call__(self, iterable):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[very many rows here]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...........................................................................&#xA;    /home/buda/anaconda2/lib/python2.7/site-packages/keras/wrappers/scikit_learn.py in fit(self=&amp;lt;keras.wrappers.scikit_learn.KerasClassifier object&amp;gt;, X=memmap([[ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       ....,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32), y=memmap([[ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       ....,  0.,  0., ...,  0.,  1.,  0.]], dtype=float32), **kwargs={})&#xA;        132             self.model = self.__call__(**self.filter_sk_params(self.__call__))&#xA;        133         elif not isinstance(self.build_fn, types.FunctionType):&#xA;        134             self.model = self.build_fn(&#xA;        135                 **self.filter_sk_params(self.build_fn.__call__))&#xA;        136         else:&#xA;    --&amp;gt; 137             self.model = self.build_fn(**self.filter_sk_params(self.build_fn))&#xA;            self.model = undefined&#xA;            self.build_fn = &amp;lt;function create_model&amp;gt;&#xA;            self.filter_sk_params = &amp;lt;bound method KerasClassifier.filter_sk_params o...as.wrappers.scikit_learn.KerasClassifier object&amp;gt;&amp;gt;&#xA;        138 &#xA;        139         loss_name = self.model.loss&#xA;        140         if hasattr(loss_name, '__name__'):&#xA;        141             loss_name = loss_name.__name__&#xA;&#xA;    ...........................................................................&#xA;    /home/buda/Projects/Kaggle/Nerve/&amp;lt;ipython-input-62-14daa2be96d1&amp;gt; in create_model(optimizer='SGD')&#xA;          1 &#xA;          2 np.random.seed(100)&#xA;          3 &#xA;          4 def create_model(optimizer=&quot;SGD&quot;):&#xA;    ----&amp;gt; 5     model=Sequential()&#xA;          6     model.add(Dense(input_dim=784, init=&quot;uniform&quot;, activation=&quot;relu&quot;))&#xA;          7     model.add(Dense(output_dim=10, activation=&quot;sigmoid&quot;))&#xA;          8 &#xA;          9     model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])&#xA;         10     return model&#xA;&#xA;    TypeError: __init__() takes at least 2 arguments (4 given)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What can be the error here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The full code is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;&#xA;X_train=X_train.reshape(X_train.shape[0], X_train.shape[1]*X_train.shape[2]).astype(&quot;float32&quot;)&#xA;X_test=X_test.reshape(X_test.shape[0], X_test.shape[1]*X_test.shape[2]).astype(&quot;float32&quot;)&#xA;&#xA;Y_train=np_utils.to_categorical(y_train,10).astype(&quot;float32&quot;)&#xA;Y_test=np_utils.to_categorical(y_test,10).astype(&quot;float32&quot;)&#xA;&#xA;np.random.seed(100)&#xA;&#xA;def create_model(optimizer=&quot;SGD&quot;):&#xA;    model=Sequential()&#xA;    model.add(Dense(input_dim=784, init=&quot;uniform&quot;, activation=&quot;relu&quot;))&#xA;    model.add(Dense(output_dim=10, activation=&quot;sigmoid&quot;))&#xA;&#xA;    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])&#xA;    return model&#xA;&#xA;model = KerasClassifier(build_fn=create_model, nb_epoch=50, batch_size=10, verbose=0)&#xA;param_grid = dict(optimizer=['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'])&#xA;grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=10)&#xA;grid_result = grid.fit(X_train, Y_train, param_grid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With n_jobs=1 in GridSearchCV:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-110-538ac4724f14&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      2 param_grid = dict(optimizer=['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'])&#xA;      3 grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1, cv=10)&#xA;----&amp;gt; 4 grid_result = grid.fit(X_train, Y_train)&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/grid_search.pyc in fit(self, X, y)&#xA;    802 &#xA;    803         &quot;&quot;&quot;&#xA;--&amp;gt; 804         return self._fit(X, y, ParameterGrid(self.param_grid))&#xA;    805 &#xA;    806 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)&#xA;    551                                     self.fit_params, return_parameters=True,&#xA;    552                                     error_score=self.error_score)&#xA;--&amp;gt; 553                 for parameters in parameter_iterable&#xA;    554                 for train, test in cv)&#xA;    555 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)&#xA;    798             # was dispatched. In particular this covers the edge&#xA;    799             # case of Parallel used with an exhausted iterator.&#xA;--&amp;gt; 800             while self.dispatch_one_batch(iterator):&#xA;    801                 self._iterating = True&#xA;    802             else:&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)&#xA;    656                 return False&#xA;    657             else:&#xA;--&amp;gt; 658                 self._dispatch(tasks)&#xA;    659                 return True&#xA;    660 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in _dispatch(self, batch)&#xA;    564 &#xA;    565         if self._pool is None:&#xA;--&amp;gt; 566             job = ImmediateComputeBatch(batch)&#xA;    567             self._jobs.append(job)&#xA;    568             self.n_dispatched_batches += 1&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __init__(self, batch)&#xA;    178         # Don't delay the application, to avoid keeping the input&#xA;    179         # arguments in memory&#xA;--&amp;gt; 180         self.results = batch()&#xA;    181 &#xA;    182     def get(self):&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self)&#xA;     70 &#xA;     71     def __call__(self):&#xA;---&amp;gt; 72         return [func(*args, **kwargs) for func, args, kwargs in self.items]&#xA;     73 &#xA;     74     def __len__(self):&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/sklearn/cross_validation.pyc in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, error_score)&#xA;   1529             estimator.fit(X_train, **fit_params)&#xA;   1530         else:&#xA;-&amp;gt; 1531             estimator.fit(X_train, y_train, **fit_params)&#xA;   1532 &#xA;   1533     except Exception as e:&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/keras/wrappers/scikit_learn.pyc in fit(self, X, y, **kwargs)&#xA;    135                 **self.filter_sk_params(self.build_fn.__call__))&#xA;    136         else:&#xA;--&amp;gt; 137             self.model = self.build_fn(**self.filter_sk_params(self.build_fn))&#xA;    138 &#xA;    139         loss_name = self.model.loss&#xA;&#xA;&amp;lt;ipython-input-62-14daa2be96d1&amp;gt; in create_model(optimizer)&#xA;      3 def create_model(optimizer=&quot;SGD&quot;):&#xA;      4     model=Sequential()&#xA;----&amp;gt; 5     model.add(Dense(input_dim=784, init=&quot;uniform&quot;, activation=&quot;relu&quot;))&#xA;      6     model.add(Dense(output_dim=10, activation=&quot;sigmoid&quot;))&#xA;      7 &#xA;&#xA;TypeError: __init__() takes at least 2 arguments (4 given)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21560" LastEditDate="2016-08-30T14:27:09.857" LastActivityDate="2017-07-14T14:05:37.603" Title="Fitting error with Neural Network Grid Search in Keras" Tags="|scikit-learn|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13728" PostTypeId="2" ParentId="13722" CreationDate="2016-08-29T11:55:20.717" Score="1" Body="&lt;p&gt;Generally for words which are not found in the vocabulary, a zero vector is assigned to them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other hacks which I could think of are :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Try different morphologies of the word (ex. morphology of goes will give go and you can get vector of go to initialize goes).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try finding similar word from wordnet. If you are able to find embedding of similar word from wordnet, initialize original word's embedding with this word's embedding.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-08-29T11:55:20.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13730" PostTypeId="1" AcceptedAnswerId="13733" CreationDate="2016-08-29T12:49:26.950" Score="1" ViewCount="1646" Body="&lt;p&gt;Imagine that I've the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Customer_ID    Product_Desc&#xA;    1   Jeans&#xA;    1   T-Shirt&#xA;    1   Food&#xA;    2   Jeans&#xA;    2   Food&#xA;    2   Nightdress&#xA;    2   T-Shirt&#xA;    2   Hat&#xA;    3   Jeans&#xA;    3   Food&#xA;    4   Food&#xA;    4   Water&#xA;    5   Water&#xA;    5   Food&#xA;    5   Beer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to make the consumer behaviour and predicte what products are associated. For do that I think that will a good strategy make the relationships first and then count the occurrences (don't know if anyone have a better idea).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first step is to conclude this relationships:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Jeans-T-Shirt-Food&#xA;Jeans-Food-Nightdress-T-Shirt-Hat&#xA;Jeans-Food&#xA;Food-Water&#xA;Water-Food-Beer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can do this? With Apache PIG or with Spark? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!!!&lt;/p&gt;&#xA;" OwnerUserId="23811" LastEditorUserId="471" LastEditDate="2016-10-12T11:22:55.157" LastActivityDate="2017-12-21T15:20:53.063" Title="Market Basket Analysis - Data Modelling" Tags="|apache-spark|scala|market-basket-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13731" PostTypeId="1" AcceptedAnswerId="14383" CreationDate="2016-08-29T14:15:22.463" Score="0" ViewCount="1607" Body="&lt;p&gt;I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset contains transactions of a supermarket over a year and I will have to predict which products are associated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The requirements for my project are:&#xA;- We must visualize the evolution of the model (Learnable Evolution Model)&#xA;- Must be done using Hadoop tool (+ Spark)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already look for multiple source codes/tutoriais to learn how can I do this but I can't find a machine learning implementation project for find patterns using Spark...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone have an idea how can I find some good tutorials to learn this skills for me to do my master thesis algorithm?&lt;/p&gt;&#xA;" OwnerUserId="23714" LastEditorUserId="12876" LastEditDate="2016-10-05T21:44:15.647" LastActivityDate="2016-10-05T21:44:15.647" Title="Machine Learning model to find items that are frequently bought together using Hadoop Spark" Tags="|machine-learning|data-mining|apache-spark|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13733" PostTypeId="2" ParentId="13730" CreationDate="2016-08-29T15:52:30.053" Score="0" Body="&lt;p&gt;You may use groupByKey or combineByKey in Spark&lt;/p&gt;&#xA;" OwnerUserId="23601" LastActivityDate="2016-08-29T15:52:30.053" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13734" PostTypeId="2" ParentId="13716" CreationDate="2016-08-29T15:58:00.820" Score="1" Body="&lt;p&gt;Perhaps have a look at MDS (multi-dimensional scaling) widget, Line Chart widget (from Timeseries add-on), or Network Explorer widget (from Networks add-on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/c5JeU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/c5JeU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-08-29T15:58:00.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13735" PostTypeId="2" ParentId="13712" CreationDate="2016-08-29T16:03:29.657" Score="2" Body="&lt;p&gt;To scale the data, you can use &lt;a href=&quot;http://docs.orange.biolab.si/3/visual-programming/widgets/data/preprocess.html&quot; rel=&quot;nofollow&quot;&gt;Preprocess&lt;/a&gt; widget. To construct new features, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_feature = old_feature - some_values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can use &lt;a href=&quot;http://docs.orange.biolab.si/3/visual-programming/widgets/data/featureconstructor.html&quot; rel=&quot;nofollow&quot;&gt;Feature Constructor&lt;/a&gt; widget.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-08-29T16:03:29.657" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13737" PostTypeId="2" ParentId="13726" CreationDate="2016-08-29T17:55:35.177" Score="2" Body="&lt;p&gt;&lt;code&gt;LabelEncoder&lt;/code&gt; converts strings to integers, but you have integers already. Thus, LabelEncoder will not help you anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wenn you are using your column with integers as it is, &lt;code&gt;sklearn&lt;/code&gt; treats it as numbers. This means, for example, that distance between 1 and 2 is 1, distance between 1 and 4 is 3. Can you say the same about your activities (if you know the meaning of the integers)? What is the pairwise distances between, for example, &quot;exercise&quot;, &quot;work&quot;, &quot;rest&quot;, &quot;leasure&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you think, that the pairwise distance between any pair of activities is 1, because those are just different activities, then &lt;code&gt;OneHotEncoder&lt;/code&gt; is your choice.  &lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2016-08-29T17:55:35.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13738" PostTypeId="2" ParentId="13673" CreationDate="2016-08-29T18:16:10.953" Score="1" Body="&lt;p&gt;Try omitting the &lt;code&gt;[]&lt;/code&gt; so that you do &lt;em&gt;not&lt;/em&gt; create python list&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trainingData = trainingData.map(lambda row: LabeledPoint(row.label,row.features))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2016-08-29T18:16:10.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13739" PostTypeId="2" ParentId="13727" CreationDate="2016-08-29T19:02:00.033" Score="1" Body="&lt;p&gt;The last line in your code is the result of above error. This is because the 'fit' method takes only two arguments i.e the data and the labels. In order to get rid of the above error, modify your code as following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid_result = grid.fit(X_train,Y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After that you can perform various operations on your classifier such as :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;best_model = grid_result.best_estimator_.model&#xA;best_param = grid_result.best_params_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-08-29T19:02:00.033" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="13740" PostTypeId="1" AcceptedAnswerId="13742" CreationDate="2016-08-30T05:35:42.890" Score="1" ViewCount="3067" Body="&lt;p&gt;I have a dataset size of ~500000 with input dimension 46.&#xA;I am trying to use Pybrain to train the network but the training is extremely slow for the whole dataset. Using batches of 50000 data points, each batch takes more than 2 hours for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are caveats of optimizing the network design so that the training is faster?&lt;/p&gt;&#xA;" OwnerUserId="23905" LastActivityDate="2016-09-02T05:15:46.293" Title="How to train neural networks with large sized data sets?" Tags="|machine-learning|neural-network|pybrain|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13741" PostTypeId="2" ParentId="13710" CreationDate="2016-08-30T05:41:23.437" Score="1" Body="&lt;p&gt;You can load these variables to pandas dataframe and use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;dataframe.describe()&lt;/code&gt;&lt;/a&gt; for summary statistics.&lt;/p&gt;&#xA;" OwnerUserId="23905" LastEditorUserId="98307" LastEditDate="2020-08-04T09:33:06.660" LastActivityDate="2020-08-04T09:33:06.660" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13742" PostTypeId="2" ParentId="13740" CreationDate="2016-08-30T06:01:33.643" Score="2" Body="&lt;p&gt;Here are some of the things that influence your training speed:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Number of weights in your network&lt;/li&gt;&#xA;&lt;li&gt;Speed of your CPU&lt;/li&gt;&#xA;&lt;li&gt;Package you are using (mostly engine it is working on, in PyLearn this is Theano)&lt;/li&gt;&#xA;&lt;li&gt;If all your data fits in memory or you are reading from disk in between batches&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With regards to network design the only thing you can really do is make the network more shallow to reduce the number of weights. To reduce the number of epochs there might be other options like adding residual connections but that will not decrease the training time of 1 epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without more information it is unclear where the bottleneck is, but 20 hours for one epoch seems a bit high. The easiest and biggest improvement you will be able to get is to use a good GPU, which should be possible using pylearn since it is built on top of Theano.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-08-30T06:01:33.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13743" PostTypeId="2" ParentId="13740" CreationDate="2016-08-30T06:12:44.430" Score="0" Body="&lt;p&gt;2 hours for a batch is too long even with a CPU. How many layers you have? How many neutrons you have? Have you tried a single layer? Try to reduce the complexity of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to train a softmax regression, just to get an lower bound for your computation power. If you find your machine struggle even with softmax regression, you'll need to upgrade your hardware.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-08-30T06:12:44.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13744" PostTypeId="1" CreationDate="2016-08-30T07:07:26.147" Score="2" ViewCount="857" Body="&lt;p&gt;I have 1 million rows with 20 attributes to do hierarchical clustering. When I want to build a distance matrix on this data by &lt;code&gt;dist()&lt;/code&gt; in R, it says that it needs 5 TB memory. I have these approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Reduce the number of rows by sampling&lt;/li&gt;&#xA;&lt;li&gt;Change the method of clustering&lt;/li&gt;&#xA;&lt;li&gt;?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, do you suggest another approach?&#xA;And I have an idea, I think if I reduce the accuracy of the values and then doing &quot;group by&quot;, then I can remove duplicated rows and have a new column with the count of duplicates for each row. &#xA;Is there any R package that can do hierarchical clustering with these data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;group by&quot;: count number of duplicated rows and add a column that say how many times this row was duplicated in source.&lt;/p&gt;&#xA;" OwnerUserId="15569" LastEditorUserId="25157" LastEditDate="2016-12-14T20:11:50.590" LastActivityDate="2019-10-05T14:02:40.447" Title="Clustering big data by reducing data accuracy?" Tags="|r|clustering|bigdata|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13746" PostTypeId="1" CreationDate="2016-08-30T08:52:40.127" Score="16" ViewCount="34888" Body="&lt;p&gt;I tried to define a custom metric fuction (F1-Score) in Keras (Tensorflow backend) according to the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def f1_score(tags, predicted):&#xA;&#xA;    tags = set(tags)&#xA;    predicted = set(predicted)&#xA;&#xA;    tp = len(tags &amp;amp; predicted)&#xA;    fp = len(predicted) - tp &#xA;    fn = len(tags) - tp&#xA;&#xA;    if tp&amp;gt;0:&#xA;        precision=float(tp)/(tp+fp)&#xA;        recall=float(tp)/(tp+fn)&#xA;        return 2*((precision*recall)/(precision+recall))&#xA;    else:&#xA;        return 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far, so good, but when I try to apply it in model compilation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model1.compile(loss=&quot;binary_crossentropy&quot;, optimizer=Adam(), metrics=[f1_score])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it gives error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError                                 Traceback (most recent call last)&#xA;&amp;lt;ipython-input-85-4eca4def003f&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      5 model1.add(Dense(output_dim=10, activation=&quot;sigmoid&quot;))&#xA;      6 &#xA;----&amp;gt; 7 model1.compile(loss=&quot;binary_crossentropy&quot;, optimizer=Adam(), metrics=[f1_score])&#xA;      8 &#xA;      9 h=model1.fit(X_train, Y_train, batch_size=500, nb_epoch=5, verbose=True, validation_split=0.1)&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/keras/models.pyc in compile(self, optimizer, loss, metrics, sample_weight_mode, **kwargs)&#xA;    522                            metrics=metrics,&#xA;    523                            sample_weight_mode=sample_weight_mode,&#xA;--&amp;gt; 524                            **kwargs)&#xA;    525         self.optimizer = self.model.optimizer&#xA;    526         self.loss = self.model.loss&#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)&#xA;    664                 else:&#xA;    665                     metric_fn = metrics_module.get(metric)&#xA;--&amp;gt; 666                     self.metrics_tensors.append(metric_fn(y_true, y_pred))&#xA;    667                     if len(self.output_names) == 1:&#xA;    668                         self.metrics_names.append(metric_fn.__name__)&#xA;&#xA;&amp;lt;ipython-input-84-b8a5752b6d55&amp;gt; in f1_score(tags, predicted)&#xA;      4     #tf.convert_to_tensor(img.eval())&#xA;      5 &#xA;----&amp;gt; 6     tags = set(tags)&#xA;      7     predicted = set(predicted)&#xA;      8 &#xA;&#xA;/home/buda/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in __iter__(self)&#xA;    493       TypeError: when invoked.&#xA;    494     &quot;&quot;&quot;&#xA;--&amp;gt; 495     raise TypeError(&quot;'Tensor' object is not iterable.&quot;)&#xA;    496 &#xA;    497   def __bool__(self):&#xA;&#xA;TypeError: 'Tensor' object is not iterable.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the problem here? The fact that my f1_score function inputs are not Tensorflow arrays? If so, where/how can I convert them correctly?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21560" LastEditDate="2016-08-30T08:59:48.500" LastActivityDate="2020-08-02T10:29:00.353" Title="How to define a custom performance metric in Keras?" Tags="|tensorflow|keras|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13747" PostTypeId="2" ParentId="12987" CreationDate="2016-08-30T09:06:40.910" Score="1" Body="&lt;p&gt;If your data resides in clean form inside a database already:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;write an sql statement that includes a series of UNION statements for your 206 country tables. You can use R or any other programming language to create this string.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;save this series of 206 UNIONs as a view (maybe as an indexed view / materialized view), or as  a new table, or as a stored-procedure that creates a temp-table on your database server&lt;/p&gt;&#xA;&#xA;&lt;p&gt;perform one SQL join on the DB server&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fetch the data batch-wise&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-08-30T09:06:40.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13749" PostTypeId="1" AcceptedAnswerId="13777" CreationDate="2016-08-30T10:43:32.070" Score="0" ViewCount="2552" Body="&lt;p&gt;I've a data set with this fields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Transaction_ID&#xA;Customer_ID&#xA;Department&#xA;Product_ID&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I'm trying to obtain a tuple with the products associated with each customer transaction. Like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Transaction_1 -&amp;gt; Product_ID 1, Product_ID 2, Product_ID 3&#xA;Transaction_2 -&amp;gt; Product_ID 1, Product_ID 2, Product_ID 4&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've this code: But It not return the dataset as I want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;case class transactions (Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)&#xA;&#xA;def csvToMyClass(line: String) = {&#xA;    val split = line.split(',')&#xA;    transactions(split(0),split(1),split(2),split(3))&#xA;}&#xA;&#xA;val  csv = sc.textFile(&quot;FILE&quot;).map(csvToMyClass)&#xA;csv.take(10)&#xA;csv.saveAsTextFile(&quot;PATH/output.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I obtain the list of products associated group by Transaction_ID??&lt;/p&gt;&#xA;" OwnerUserId="23811" LastEditorUserId="5177" LastEditDate="2016-08-31T09:36:00.907" LastActivityDate="2016-08-31T10:01:00.580" Title="Apply GroupByKey to identify the Frequently Products Purchase Together" Tags="|apache-spark|scala|association-rules|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13750" PostTypeId="1" CreationDate="2016-08-30T11:32:46.447" Score="10" ViewCount="62269" Body="&lt;p&gt;I use Keras-Tensorflow combo installed with CPU option (it was said to be more robust), but now I'd like to try it with GPU-version. Is there a convenient way to switch? Or shall I re-install fully Tensorflow? Is the GPU version reliable?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-03-28T10:59:45.113" Title="Switching Keras backend Tensorflow to GPU" Tags="|tensorflow|keras|gpu|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13752" PostTypeId="2" ParentId="13679" CreationDate="2016-08-30T11:40:17.787" Score="1" Body="&lt;p&gt;This question has no general answer; it depends on the specifics of both the statistical method in question and the implementation of that method in whatever language you used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if you used a logistic GLM, then a missing value is an error since data records must be complete. Presumably you would have included a default value, and you will have to manually apply that if you want those records to be scored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you were using gradient boosted trees (gbm), then they &lt;em&gt;can&lt;/em&gt; cope with missing values, but do so in an implementation-defined way. The answers would be different in the &lt;code&gt;gbm&lt;/code&gt; and &lt;code&gt;xgboost&lt;/code&gt; R packages, for example.&lt;/p&gt;&#xA;" OwnerUserId="23911" LastActivityDate="2016-08-30T11:40:17.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13753" PostTypeId="1" CreationDate="2016-08-30T11:54:15.897" Score="2" ViewCount="2144" Body="&lt;p&gt;What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?&lt;/p&gt;&#xA;" OwnerUserId="23912" LastActivityDate="2017-02-10T11:41:19.440" Title="Significant difference in testing vs cross validation accuracy" Tags="|python|cross-validation|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13754" PostTypeId="1" CreationDate="2016-08-05T07:39:52.900" Score="14" ViewCount="47444" Body="&lt;p&gt;I am using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn Random Forest Classifier&lt;/a&gt; and I want to plot the feature importance such as in &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;noreferrer&quot;&gt;this example&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However my result is completely different, in the sense that feature importance standard deviation is almost always bigger than feature importance itself (see attached image).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xjhXa.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xjhXa.png&quot; alt=&quot;feature importance&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to have such kind of behaviour, or am I doing some mistakes when plotting it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;import pandas as pd&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;&#xA;clf = RandomForestClassifier()&#xA;clf.fit(predictors.values, outcome.values.ravel())&#xA;&#xA;importance = clf.feature_importances_&#xA;importance = pd.DataFrame(importance, index=predictors.columns, &#xA;                          columns=[&quot;Importance&quot;])&#xA;&#xA;importance[&quot;Std&quot;] = np.std([tree.feature_importances_&#xA;                            for tree in clf.estimators_], axis=0)&#xA;&#xA;x = range(importance.shape[0])&#xA;y = importance.ix[:, 0]&#xA;yerr = importance.ix[:, 1]&#xA;&#xA;plt.bar(x, y, yerr=yerr, align=&quot;center&quot;)&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="133" OwnerDisplayName="fbrundu" LastActivityDate="2017-02-07T09:52:35.657" Title="Feature importance with scikit-learn Random Forest shows very high Standard Deviation" Tags="|python|random-forest|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13755" PostTypeId="2" ParentId="13754" CreationDate="2016-08-05T10:54:52.013" Score="3" Body="&lt;p&gt;You are using RandomForest with the default number of trees, which is 10.&#xA;For around 30 features this is too few. Therefore standard deviation is large. Try at least 100 or even 1000 trees, like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = RandomForestClassifier(n_estimators=1000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a more refined analysis you can also check how large the correlation between your features is.&lt;/p&gt;&#xA;" OwnerUserId="9085" OwnerDisplayName="lanenok" LastActivityDate="2016-08-05T10:54:52.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13756" PostTypeId="1" CreationDate="2016-08-30T12:26:07.027" Score="3" ViewCount="891" Body="&lt;p&gt;I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the proper term to address this problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have settled for the term &lt;a href=&quot;https://en.wikipedia.org/wiki/Domain_adaptation&quot; rel=&quot;nofollow noreferrer&quot;&gt;domain adaptation&lt;/a&gt;, shortly described as a field in machine learning which aims to learn from a certain data distribution in order to predict data coming from a different (but related) target distribution. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="14372" LastEditDate="2016-12-25T14:00:29.977" LastActivityDate="2017-07-16T21:04:57.627" Title="Discrepancy between training set and real-world data set: domain adaptation?" Tags="|machine-learning|predictive-modeling|dataset|domain-adaptation|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13758" PostTypeId="2" ParentId="9945" CreationDate="2016-08-30T13:26:20.773" Score="1" Body="&lt;p&gt;A good library for machine learning with GPUs is &lt;a href=&quot;https://github.com/dmlc/mxnet&quot; rel=&quot;nofollow&quot;&gt;mxnet&lt;/a&gt;. The package is mostly deep learning though, so if you are looking for specific machine learning algorithms you might not find them there. However they have a good set of deep learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="14994" LastActivityDate="2016-08-30T13:26:20.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13759" PostTypeId="1" AcceptedAnswerId="14542" CreationDate="2016-08-30T13:54:40.950" Score="1" ViewCount="25280" Body="&lt;p&gt;I've installed the R package &lt;code&gt;tm&lt;/code&gt;. Now I am trying to load it, but it's giving me this error and warning everytime, I try. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The package is in this location- &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;‘C:/Users/Google/Documents/R/win-library/3.3’&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error Message--&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : &#xA;  there is no package called ‘slam’&#xA;In addition: Warning message:&#xA;package ‘tm’ was built under R version 3.3.1 &#xA;Error: package or namespace load failed for ‘tm’&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="2723" LastEditDate="2016-08-31T11:57:58.540" LastActivityDate="2018-01-31T08:47:24.947" Title="Getting Error in RStudio while loading a package- 'tm'" Tags="|r|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13760" PostTypeId="1" CreationDate="2016-08-30T13:55:24.413" Score="4" ViewCount="324" Body="&lt;p&gt;Someone I know has been using the Flesch-Kincaide readability score algorithm to try to gauge how well students write as a predictor of their academic success. They've been mostly unsuccessful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While taking a UCSD Coursera course I recently learned about this algorithm and I believe that their lack of success is just a basic misunderstanding of how the algorithm works. A low score does not imply poor writing -- anything written for college level reading ability, such as U.S. law and classic literature, receives a low readability score. High readability scores imply more simplistic writing, but not that the writing is more or less correct. So it's not really measuring much in terms of proficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that is to ask this -- &lt;strong&gt;is there some other algorithm or metric to gauge writing quality, which would be more useful? Something that perhaps measures proper grammar and/or punctuation?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been and will continue to search for this myself, but if anyone has any knowledge they can share it would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-08-30T14:07:07.013" LastActivityDate="2017-02-20T18:14:12.083" Title="Alternative to Flesch for a readability score algorithm" Tags="|text-mining|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13761" PostTypeId="2" ParentId="13759" CreationDate="2016-08-30T13:58:43.130" Score="3" Body="&lt;p&gt;This is a dependency error. You're just missing one of the packages that &lt;code&gt;tm&lt;/code&gt; assumes you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution is to simply directly install that package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;slam&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;another is to use a package manager that takes care of this for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;pacman&quot;)&#xA;pacman::p_load(tm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-08-30T13:58:43.130" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13762" PostTypeId="1" CreationDate="2016-08-30T14:28:51.950" Score="0" ViewCount="3202" Body="&lt;p&gt;I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;mail_id&#xA;DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=&#xA;BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=&#xA;BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=&#xA;EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=&#xA;K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=&#xA;UGATDXARg7jMEInKH7oXgty2nwxnwD2l0OW/8Nsa0MI=&#xA;qE9zgWiITYA97RUiN4X/t9IVWLViLz+lKijaYegyBiQ=&#xA;BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=&#xA;4+EEK8RbNYwuFCHznY9XSRCV4Yek60bHVgnP3jtjjzk=&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;After doing a lot of analysis on my data, I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T10:07:47.410" LastActivityDate="2020-12-25T10:07:47.410" Title="Using scikit-learn FeatureHasher" Tags="|machine-learning|scikit-learn|feature-engineering|feature-extraction|hashing-trick|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="13763" PostTypeId="1" AcceptedAnswerId="13765" CreationDate="2016-08-30T14:34:59.243" Score="2" ViewCount="2132" Body="&lt;p&gt;I'd install an R package &lt;code&gt;twitterR&lt;/code&gt; in my RStudio for accessing Twitter data for a particular handle. But whenever I do try to access the Twitter data it's showing me an unexpected message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In doRppAPICall(“search/tweets”, n, params = params, retryOnRateLimit = retryOnRateLimit, :&#xA;100 tweets were requested but the API can only return 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried different parameters, handles and functionalities but still getting the same issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my code for the particular functionality--&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_tweets &amp;lt;- searchTwitter('@jabhij', geocode='20.593684, 78.96288, 2000km', n = 100)&#xA;my_tweets_df &amp;lt;- do.call(&quot;rbind&quot;, laaply(my_tweets, as.data.frame))&#xA;View(my_tweets_df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on that will be very helpful.&#xA;Cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="2723" LastEditDate="2016-08-31T11:58:02.730" LastActivityDate="2018-10-02T02:47:29.643" Title="Unexpected Error Message in RStudio; while using 'twitterR' Package" Tags="|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13765" PostTypeId="2" ParentId="13763" CreationDate="2016-08-30T15:06:32.720" Score="1" Body="&lt;p&gt;It's because of the whitespace in the parameter for &lt;code&gt;geocode&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_tweets &amp;lt;- searchTwitter('@jabhij', geocode=&quot;20.593684,78.96288,2000km&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2016-08-30T15:06:32.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13766" PostTypeId="2" ParentId="10478" CreationDate="2016-08-30T15:13:18.680" Score="4" Body="&lt;p&gt;R's &lt;code&gt;gather()&lt;/code&gt; essentially goes from wide to long. So, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;check &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/reshaping.html&quot; rel=&quot;nofollow&quot;&gt;pandas page&lt;/a&gt; for how to use &lt;code&gt;pandas.wide_to_long()&lt;/code&gt;, &lt;/li&gt;&#xA;&lt;li&gt;check &lt;a href=&quot;http://connor-johnson.com/2014/08/28/tidyr-and-pandas-gather-and-melt/&quot; rel=&quot;nofollow&quot;&gt;this blog&lt;/a&gt; for a discussion on getting an elegant &lt;code&gt;gather&lt;/code&gt;-like function in Python.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1406" LastActivityDate="2016-08-30T15:13:18.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13767" PostTypeId="1" CreationDate="2016-08-30T16:09:05.360" Score="1" ViewCount="4146" Body="&lt;p&gt;I am trying to use LSTM neural networks in order to make a song composer. Basically this is based of a text generator (tries to predict the next character after looking at a sequence of characters) but instead of characters, it tried to predict notes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Structure of the midi file that serves as the input (Y-axis is the pitch or note value while X-axis is time):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Xi510.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xi510.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the predicted note values:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Em4BF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Em4BF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/hX8qW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hX8qW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I set an epoch of 50, but it seems that the LSTM's loss rate does not decrease, most of the time its loss rate does not improve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lzJjL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lzJjL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect this is because there is an overwhelming number of a particular note (in this case, note value 65) which makes the LSTM lazy during training phase and predict 65 each and every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like this is a common problem among LSTMs and time-series based learning algorithms. How would I solve a problem like this? If what I mentioned is not the problem, then what is the problem and how do I solve that?&lt;/p&gt;&#xA;" OwnerUserId="10523" LastEditorUserId="10523" LastEditDate="2016-08-30T20:30:39.207" LastActivityDate="2017-01-28T19:03:22.980" Title="How to deal with situation where LSTM fails to learn (constantly makes the same incorrect prediction)" Tags="|machine-learning|neural-network|deep-learning|time-series|keras|" AnswerCount="2" CommentCount="1" ClosedDate="2017-01-31T10:23:19.510" ContentLicense="CC BY-SA 3.0" />
  <row Id="13768" PostTypeId="1" CreationDate="2016-08-30T16:55:58.527" Score="0" ViewCount="532" Body="&lt;p&gt;Is it necessary to learn Python from square one if your ultimate goal is to use it for data science? I have used R a lot for data science. But could I learn Python sufficiently well just by looking at other data science problems and their Pythonic approaches?&lt;/p&gt;&#xA;" OwnerUserId="23921" LastActivityDate="2022-04-28T16:56:18.100" Title="Learning Python for Data Science" Tags="|python|" AnswerCount="4" CommentCount="2" ClosedDate="2022-04-28T18:12:21.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="13770" PostTypeId="2" ParentId="13768" CreationDate="2016-08-30T20:03:35.310" Score="4" Body="&lt;p&gt;Both Python and R are extensively used in the field of data science.Though both have their advantages and disadvantages, python has many libraries that are awesome to use for data science purpose. For example, scikit-learn, Keras, Theano, XGBoost, Matplotlib,etc. These make lives so easy for data science.&lt;/p&gt;&#xA;&lt;p&gt;Yes, analyzing and understanding the code of others can give you a lit of information but you cannot learn until you don't get your hands dirty with some python code. In order to learn how to use python in data science, &lt;a href=&quot;https://www.dataquest.io/subject/learning-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt; provides an excellent tutorial.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="98307" LastEditDate="2020-08-04T09:32:40.343" LastActivityDate="2020-08-04T09:32:40.343" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13771" PostTypeId="2" ParentId="13768" CreationDate="2016-08-30T20:29:34.973" Score="2" Body="&lt;p&gt;Python is a very intuitive language in terms of syntax. The language is designed keeping readability in mind thus is a very powerful and efficient language with a flat learning curve. R is a language with much stronger data analysis toolkit and support and a much steeper learning curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pandas package in Python implements the R data frame and should be a good place to start. &lt;/p&gt;&#xA;" OwnerUserId="23905" LastActivityDate="2016-08-30T20:29:34.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13772" PostTypeId="2" ParentId="13767" CreationDate="2016-08-30T20:33:26.823" Score="0" Body="&lt;p&gt;Try using a recurrent neural network and use a RProp minus trainer without weight backtracking (aka RProp-, cf. [Igel&amp;amp;Huesken, Neurocomputing 50, 2003]) and without ponderation, ie. all training samples have the same weight.&lt;/p&gt;&#xA;" OwnerUserId="23905" LastActivityDate="2016-08-30T20:33:26.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13774" PostTypeId="1" AcceptedAnswerId="13775" CreationDate="2016-08-31T06:46:47.193" Score="0" ViewCount="193" Body="&lt;p&gt;is there big difference between data Science , big Data and database? i am confused in these three can anyone help me to out of this confusion?&lt;/p&gt;&#xA;" OwnerUserId="23940" LastActivityDate="2016-08-31T07:17:49.977" Title="is there big difference between data Science , big Data and database?" Tags="|bigdata|databases|" AnswerCount="1" CommentCount="6" ClosedDate="2016-08-31T11:57:31.537" ContentLicense="CC BY-SA 3.0" />
  <row Id="13775" PostTypeId="2" ParentId="13774" CreationDate="2016-08-31T07:17:49.977" Score="4" Body="&lt;p&gt;Well, they are absolutely different things but that are somehow linked. I gonna go through each of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Base&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of a data base (DB from now) like a computer which only purpose is to store data accesable to be read. By data, and focusing only in SQL-like DB, I mean basically tables of information like and excel file with columns and rows. You can think of a SQL DB like an ecosystem of excel tables which share some common field. So basically a DB is a hardware infrastructure which allows to write and read a given amount of information within it (in the very beggining they were plain computers, of course with the rise of internet specialized hardware appeared). You can build your own DB in your personal computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Big Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot; An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed ... for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.&quot; P.S.Laplace&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Laplace did not think over it very deeply before formulate his sentence, obviusly if god had come to him and give what he wanted soon or later he would have realized all that information was indeed useless for him. Where could him store all that? From where should him start to read? What can he do with such amount of information he could never end to compute? In case he could read everything what should he calculate first?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These all are questions Big Data tries to answer and find a solution to. Big data appeared together with the huge websites internet gave birth, such as Amazon or Google. At some point they need to store so many information that it was imposible to store it in a single computer, not even in a big one, so they need to use a set of computers for which previous standard technologies for DB didnt work anymore. This fact was also the seed for No-SQL DB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More about Big Data and non-sql here: &#xA;&lt;a href=&quot;http://www.kdnuggets.com/2016/07/seven-steps-understanding-nosql-databases.html&quot; rel=&quot;nofollow&quot;&gt;http://www.kdnuggets.com/2016/07/seven-steps-understanding-nosql-databases.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Science&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally Data Science is a statistical science, which aim is to extract order out of chaos as any other science, however meanwhile the rest of the sciences are focused in a single &quot;narrow&quot; piece of knowledge, such as biology, chemistry, etc; data science is on the other hand multidisciplinar science that could face problems from a broaden origin. Examples would be marketing or business oriented, cosmology, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So data science uses mathematical and computer science algorithms to provide some useful information out of a disordered set of data. And here is where the link with Big Data comes, actually is in the question is set before: What can he do with such amount of information he could never end to compute? So Data Science and Big data are a very usual marriage in most IT companies nowadays and in more specific fields day after day. But, data science is apply a set of mathematical algorithms to data (like apply a calculation in an excel file to create a new row) and big data is the technology to have a huge amount of excel files (I use the word &quot;excel files&quot; here is just to make it easier to grasp).&lt;/p&gt;&#xA;" OwnerUserId="23567" LastActivityDate="2016-08-31T07:17:49.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13776" PostTypeId="1" CreationDate="2016-08-31T07:54:21.690" Score="1" ViewCount="123" Body="&lt;p&gt;I am trying to get some data out of R snippet to Azure Python 3 Jupyter notebook (hosting is available for free at &lt;a href=&quot;http://notebooks.azure.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://notebooks.azure.com&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried the following in Python 3 notebook:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%load_ext rmagic&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then tried to embed R: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%%R -o x&#xA;x &amp;lt;- 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I get Python error: &lt;code&gt;name 'x' is not defined&lt;/code&gt; – see the picture below. What is the right way to embed R into Python 3 and exchange data using Azure Notebooks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nogin.info/R2Py.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Azure Notebooks http://nogin.info/R2Py.png&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23945" LastEditorUserId="23945" LastEditDate="2016-08-31T15:07:35.900" LastActivityDate="2016-09-01T09:15:29.140" Title="How to use rmagic in Azure Notebooks?" Tags="|python|r|jupyter|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13777" PostTypeId="2" ParentId="13749" CreationDate="2016-08-31T10:01:00.580" Score="0" Body="&lt;p&gt;Since you are using Spark 1.6, I'd rather do these kind of transformations with DataFrame as it's much easier to manipulate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll need to use SQLContext implicits for this :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sqlContext.implicits._ // not need in spark-shell&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, let's create some dummy data just to follow the code snippet that you have provided :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; val data = sc.parallelize(Seq(&quot;1,2,3,4&quot;, &quot;2,3,4,5&quot;, &quot;1,3,4,5&quot;, &quot;1,6,6,7&quot;))&#xA;// data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[13] at parallelize at &amp;lt;console&amp;gt;:27&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Back to your code, we define a case class &lt;code&gt;Transactions&lt;/code&gt; and the CSV converter : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; case class Transactions(Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)&#xA;// defined class Transactions&#xA;&#xA;scala&amp;gt; def csvToMyClass(line: String) = { val split = line.split(','); Transactions(split(0), split(1), split(2), split(3)) }&#xA;// csvToMyClass: (line: String)Transactions&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We will use implicits now to convert our data into a DataFrame after converting it to Transactions :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; val df = data.map(csvToMyClass).toDF(&quot;Transaction_ID&quot;, &quot;Customer_ID&quot;, &quot;Department&quot;, &quot;Product_ID&quot;)&#xA;// df: org.apache.spark.sql.DataFrame = [Transaction_ID: string, Customer_ID: string, Department: string, Product_ID: string]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's take a look at the DataFrame :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; df.show&#xA;// +--------------+-----------+----------+----------+&#xA;// |Transaction_ID|Customer_ID|Department|Product_ID|&#xA;// +--------------+-----------+----------+----------+&#xA;// |             1|          2|         3|         4|&#xA;// |             2|          3|         4|         5|&#xA;// |             1|          3|         4|         5|&#xA;// |             1|          6|         6|         7|&#xA;// +--------------+-----------+----------+----------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now all we have to do is a simple group by and perform a collect_list aggregation on the first dataframe : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; val df2 = df.groupBy(&quot;Transaction_ID&quot;).agg(collect_list($&quot;Product_ID&quot;))&#xA;// df2: org.apache.spark.sql.DataFrame = [Transaction_ID: string, collect_list(Product_ID): array&amp;lt;string&amp;gt;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can check the content of our new DataFrame &lt;code&gt;df2&lt;/code&gt; :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; df.groupBy(&quot;Transaction_ID&quot;).agg(collect_list($&quot;Product_ID&quot;)).show&#xA;// +--------------+------------------------+&#xA;// |Transaction_ID|collect_list(Product_ID)|&#xA;// +--------------+------------------------+&#xA;// |             1|               [4, 5, 7]|&#xA;// |             2|                     [5]|&#xA;// +--------------+------------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope that this answers your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; If you wish to know what's the difference between RDD and DataFrames, I advice you to read Databrick's blog entry about it &lt;a href=&quot;https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2016-08-31T10:01:00.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13778" PostTypeId="1" CreationDate="2016-08-31T11:15:32.410" Score="1" ViewCount="266" Body="&lt;p&gt;My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to have a shared global model, unfortunately. Periodically I will update each device model with the new data from the device - so maybe once per day I will load an additional daily batch of readings and use the properties of stateful LSTM training to update the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of these models is very small, containing at most 10-20 thousand readings in their whole history, and fitting into memory on even a modest GPU. Training from scratch is relatively quick, and updating a single batch should be even more so, as typically there are only 48 new readings/day (30 minute intervals), but going as high as 1440 readings/day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is - &lt;strong&gt;what is an optimal architecture for handling all of these models and updates?&lt;/strong&gt; They are all independent and small, so I don't see a need for anything distributed, but there are so many of them that I am unsure how to proceed. I am thinking that perhaps using something like an AWS 'large' GPU cluster, with each GPU loading a serialized model from the DB, updating with a new batch of data, and writing back could work - however, I suspect that having to repeatedly compile the models will be a blocker and that being able to do only 4 at a time is woefully insufficient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, it may be possible to do the 'initial' training on some kind of powerful GPU rig, and then do the batch updates on CPU's, which would make it easier to run many instances in parallel. Is this an example of where Spark might be useful? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice is appreciated - including 'your premise is absurd, why would you do this!' This is only one of many ways available to do the anomaly detection, so if the architecture implementation is unfeasible, I will abandon the LSTM's altogether and try something simpler.&lt;/p&gt;&#xA;" OwnerUserId="23920" LastEditorUserId="2723" LastEditDate="2016-08-31T13:52:28.297" LastActivityDate="2016-08-31T13:52:28.297" Title="Scalable training/updating of many small LSTM models" Tags="|machine-learning|apache-spark|scalability|parallel|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13779" PostTypeId="2" ParentId="13756" CreationDate="2016-08-31T13:34:58.240" Score="3" Body="&lt;p&gt;You may be looking for &lt;a href=&quot;https://en.wikipedia.org/wiki/Sampling_bias&quot; rel=&quot;nofollow&quot;&gt;sampling bias&lt;/a&gt;. Also the other case (where training set does in fact represent the real world data-set well) is generally known as representative sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2016-08-31T13:34:58.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13782" PostTypeId="1" CreationDate="2016-08-31T19:30:25.727" Score="2" ViewCount="322" Body="&lt;p&gt;I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;user_id                                           mail_id       &#xA;AR+tMy3H/E+Re8Id20zUIz+amJkv6KU12o+BrgIDin0=      DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=&#xA;1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=&#xA;OEfFUcsTAGInCfsHuLZuIgdSNtuNsg8EdfN98VUZVTs=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=   &#xA;1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=&#xA;CYRcuV0cR0algMZJ1N6+3uKcqi8iu+6tJNzmBbmgN7o=      K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;After a detailed analysis of my data, I figured out that I cannot drop these two columns from my data frame as they are too important for prediction. I can hash these two features but there is one more interesting thing. There are only 2,000 types of user_ids and mail_ids. So doing one hot encoding can help a lot. My question is that if I convert this into one hot encoding using 'get_dummies' method in pandas with &lt;code&gt;sparse = True&lt;/code&gt;, will it be memory efficient or is there any other efficient way to do it?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T10:08:00.883" LastActivityDate="2021-10-14T11:46:38.660" Title="Feature Engineering" Tags="|dataset|pandas|feature-extraction|categorical-data|feature-engineering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="13783" PostTypeId="2" ParentId="13753" CreationDate="2016-08-31T22:18:04.847" Score="0" Body="&lt;p&gt;There can be many reasons for this thing but in most of the cases I have observed one common reason. When you split your data using train_test_split or any other method, it is important to note that the column on which you are splitting the data is significant for splitting considering both the training and testing set. For example if I have a 'time' field in my data and I have split the data into training and testing sets on this column such that there is no value in this column of the test set which matches with any of the values in the same columns in the train set. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-08-31T22:18:04.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13784" PostTypeId="1" CreationDate="2016-08-31T23:46:17.130" Score="0" ViewCount="787" Body="&lt;p&gt;Not sure if this is a right forum. Please advise if not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently employed as a Software Engineer and am fairly happy. Except, I'd like to make more money. I hear data science is the hottest field these days but wasn't sure if it paid more. I was contemplating to get a certificate like &lt;a href=&quot;https://www.udacity.com/course/data-analyst-nanodegree--nd002&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand in the beginning it might not be more than what I currently make (~100K) as I am new to the field. However, in few years (let's say about 3 years) is it common to make 150K+? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a MS in Computer Science, and was good at math classes. So I feel confident in being able to transition.&lt;/p&gt;&#xA;" OwnerUserId="23971" LastActivityDate="2016-09-02T19:00:23.913" Title="transition to data scientist for more money" Tags="|career|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13787" PostTypeId="2" ParentId="13784" CreationDate="2016-09-01T02:14:57.930" Score="1" Body="&lt;p&gt;You can make plenty of money as a software engineer. It depends on how good you are, and how scarce your skills are. If you transition to data science, you'll be starting from close to scratch, and will be paid accordingly. I don't know where you live, but you can currently make $150K in Silicon Valley as a data scientist or a software engineer. Search for salary surveys in your area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A certificate won't get you hired -- passing the interview will -- but it will help you get past HR. Maybe you should take the classes just to see if you enjoy it.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-01T02:14:57.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13791" PostTypeId="2" ParentId="13784" CreationDate="2016-09-01T06:04:44.967" Score="4" Body="&lt;p&gt;Data scientist won't actually make you better paid. You might have seen 150k+ for some data scientists, but those people usually have a Phd and they can get similar salary even they do something else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data science is a very big field; you can work in the field as a software engineer. But don't expect training a gradient boosting model; you'll more likely be asked to process Hadloop big data application, optimization Python workflow etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, a data science certificate is useless. If your next position is a software engineering in data science, you don't need a certificate; you'll just need have a good proven record that you can deal with data. If you next position will be mathematical modelling, you'll need something like a Phd.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-09-01T06:43:21.380" LastActivityDate="2016-09-01T06:43:21.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13792" PostTypeId="1" CreationDate="2016-09-01T07:36:40.657" Score="6" ViewCount="1686" Body="&lt;p&gt;One of the methods to address a classification predictive analysis on an imbalanced set consist on undersample the majority class (others approaches consist on: undersample the majority class, synthesize new minority classes...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So assume hereafter we use any of those solutions and then we train an algorithm with the new generated data set. Will this trained algorithm be useful to predict further data from this system which is in general imbalanced?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or to make it more concrete, is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set? Or both should be generarly either balanced or imbalanced?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sergi&lt;/p&gt;&#xA;" OwnerUserId="23567" LastActivityDate="2016-10-01T15:51:30.017" Title="Balanced Train set to predict Imbalanced Prediction set" Tags="|classification|predictive-modeling|class-imbalance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13793" PostTypeId="2" ParentId="13792" CreationDate="2016-09-01T07:51:04.260" Score="1" Body="&lt;p&gt;It's possible. Consider KNN. When you train a model, it essentially &quot;remembers&quot; the training set. Now, when you use the model to predict an unbalanced data set, the model simply read off from the memory, so there's no problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, imagine you train a model with an uniformly distributed age group from 10 to 80. Now you have a test set where everybody is 70-80 years old. KNN simply find out the neighbors for your old people (say 60-80 old people in your training data) and then compute a weighted average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a balance training set to predict a imbalanced test set is not super challenging. The other way around is much more challenging.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-09-01T07:51:04.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13795" PostTypeId="2" ParentId="13658" CreationDate="2016-09-01T08:38:22.307" Score="1" Body="&lt;p&gt;I think it depends on your requirements. Read/Write, Sparse/Nonsparse,...? There are many alternatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really common is SQLite, the &quot;&lt;a href=&quot;https://www.sqlite.org/mostdeployed.html&quot; rel=&quot;nofollow&quot;&gt;most widely deployed and used database engine&lt;/a&gt;&quot;, a small relational database, these days used behind-the-scenes by many open source and commercial software packages with data storage needs (e.g., Adobe Lightroom, Mozilla Firefox).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the top of my head:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you work with R and python:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://github.com/wesm/feather&quot; rel=&quot;nofollow&quot;&gt;feather&lt;/a&gt; software was designed for fast data-frame serialization. It is currently available for R and python. Two R and Python authorities designed it in a collaboration. It's built on top of &quot;Apache Arrow&quot; and/or &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Protocol_Buffers&quot; rel=&quot;nofollow&quot;&gt;protocol buffers&lt;/a&gt;&quot;, it's &lt;a href=&quot;https://blog.dominodatalab.com/the-r-data-i-o-shootout/&quot; rel=&quot;nofollow&quot;&gt;fast&lt;/a&gt; for reading, but it's in alpha state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some serialization formats available from the XML community.&#xA;You can store complex webs of objects in these formats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is JSON and JSON-schema.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your tables are sparse, there is, for instance, &quot;sparse ARFF&quot; format (in little use, though). There must be others (&lt;a href=&quot;https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats&quot; rel=&quot;nofollow&quot;&gt;I have to  look this up myself&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2016-09-01T08:53:26.090" LastActivityDate="2016-09-01T08:53:26.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13796" PostTypeId="1" CreationDate="2016-09-01T09:33:04.853" Score="0" ViewCount="98" Body="&lt;p&gt;In my application, I want to have clusters whose diameters are bounded by some fixed number. Also, the number of clusters in the data is unknown and, therefore, the clusters must be discovered without a 'k' parameter.&lt;/p&gt;&#xA;&lt;p&gt;What method should I use?&lt;/p&gt;&#xA;" OwnerUserId="23850" LastEditorUserId="85045" LastEditDate="2020-12-23T17:29:12.100" LastActivityDate="2020-12-23T17:29:12.100" Title="Clusters with bounded diameter" Tags="|machine-learning|clustering|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13797" PostTypeId="2" ParentId="13792" CreationDate="2016-09-01T09:44:03.170" Score="1" Body="&lt;p&gt;Class balancing is necessary when the loss function you are minimising during training is not the same as the metric you are using for evaluation. The answer to the question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;depends on the choice of loss function and measure of &quot;effectiveness&quot; (the evaluation metric). If both classes are given equal importance in evaluation (eg. ROC AUC), and not in the loss function, then balancing will lead to increased performance.&lt;/p&gt;&#xA;" OwnerUserId="12972" LastActivityDate="2016-09-01T09:44:03.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13799" PostTypeId="2" ParentId="13796" CreationDate="2016-09-01T10:59:31.880" Score="0" Body="&lt;p&gt;Start with &lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow&quot;&gt;Hierarchical clustering&lt;/a&gt;.&#xA;That will give you an hierarchical grouping of the datapoint, matching each one to the closest to it.&#xA;One of the benefits of such clustering is that you can decide afterwards what is the number of clusters (k) that you want and then get the actual clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case there is and additional constraint of the diameter. Hence, you should verify that the maximal distance between the items in the cluster is bellow your bound. &#xA;Since that in hierarchical clustering we match the closest items first, when you divide into clusters you should check that the distance between the current item and the rest of the items in the clusters is bellow your bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can doing that while building the hierarchy, saving computation time.&#xA;However, you can also do it after the hierarchy was constructed and this way use libraries that already implemented these algorithms and you will only need to add the diameter bound on your own. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-09-01T10:59:31.880" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13801" PostTypeId="2" ParentId="13792" CreationDate="2016-09-01T14:03:32.357" Score="0" Body="&lt;p&gt;The answer to this question is very related to the practical problem you are working with. I spent three years modifying SMOTE algorithm to use it with one problem and the next two years trying to design a biased classifier to deal with another problem. In fact, in academia we care a lot about average accuracy comparing to overall accuracy. Here it comes the point that you should deal with the imbalance data problem. But in practical, in many cases you will find that they care about overall accuracy and not average accuracy. When the cost of missing any of the samples is the same then it makes since. If you do not have any consideration,then the question is: can I maximize both average and overall accuracy? &#xA;This depends on the features you are using. &#xA;In some cases, they cost of missing the minority samples is higher, then you need to design a biased classifier.&#xA;The answer to your question:&#xA;If balancing the data can maximize both of average and overall accuracy, it is better to do it. If balancing the data can not maximize both of them , then you need to think about the problem requirements &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-09-01T14:03:32.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13802" PostTypeId="2" ParentId="13796" CreationDate="2016-09-01T15:35:56.213" Score="1" Body="&lt;p&gt;Hierarchical clustering, with &lt;strong&gt;complete linkage&lt;/strong&gt; will find clusters with a maximum pairwise distance i.e. diameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need two parameters:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Distance function&lt;/li&gt;&#xA;&lt;li&gt;Maximum distance = height where to cut the tree&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-09-01T15:35:56.213" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="13803" PostTypeId="1" AcceptedAnswerId="13805" CreationDate="2016-09-01T16:02:57.677" Score="0" ViewCount="947" Body="&lt;p&gt;Imagine that I've a field called date in this format: &quot;yyyy-mm-dd&quot; and I want to convert to number like &quot;yyymmdd&quot;. For that I'm trying to use this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data_ID = FOREACH File GENERATE CONCAT((chararray)SUBSTRING(Date,0,4),(chararray)SUBSTRING(Date,6,2),(chararray)SUBSTRING(Date,9,2));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I'm getting a list of nulls...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone knows what I'm doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thnaks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-09-01T17:10:43.930" Title="Convert date into number - Apache PIG" Tags="|apache-pig|etl|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13804" PostTypeId="2" ParentId="9582" CreationDate="2016-09-01T16:55:22.240" Score="1" Body="&lt;p&gt;I agree with @thebiro - you could start by classifying a sample of the comments as being either opposed to or in support of the policy.  If you only need basic for/against classification, you can do binary classification (e.g. against = 0, for = 1).  If you need specify the degree to which a statement is for or against the policy, then you could define a scale that indicates the degree to which a comment supports the policy (e.g. strongly against = -3, strongly for = 3).  Once you have chosen your scale and manually classified your sample comments, you have a dataset that you can use to train a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have your training data, you need to come up with a numerical representation of each comment.  There are too many possible approaches to enumerate here, but some basic concepts are bag of words and word vectors(aka word embeddings).  This &lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words&quot; rel=&quot;nofollow&quot;&gt;Kaggle tutorial&lt;/a&gt; may help explain those concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you need to train a model that takes the numerical representation of each comment as an input and outputs a sentiment score (0 or 1 for binary or a number on your scale).  You have a lot of choices for the type of model you use.  If you are working in Python, you can use one of the supervised learning methods in Scikit-learn.  Scikit-learn also has a &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/statistical_inference/supervised_learning.html&quot; rel=&quot;nofollow&quot;&gt;tutorial&lt;/a&gt; on working with text.  You train your model on the sample you manually classified (Something like &lt;code&gt;model.fit(training_data_inputs)&lt;/code&gt; in Scikit-learn) and then predict the output for the rest of your dataset (&lt;code&gt;model.predict(test_data_inputs)&lt;/code&gt; in Scikit-learn). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people are also using neural networks for sentiment analysis.  Keras is a great Python library for building neural networks and has sentiment analysis examples available on Github.&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-09-01T16:55:22.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13805" PostTypeId="2" ParentId="13803" CreationDate="2016-09-01T17:10:43.930" Score="0" Body="&lt;p&gt;I don't use PIG, but from looking online I think you should be able to use the Replace function built into PIG.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming your Date field is already a string:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data_ID = FOREACH File GENERATE(int) REPLACE(Date,'-','');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the reference if it's helpful: &lt;a href=&quot;https://pig.apache.org/docs/r0.9.1/func.html#replace&quot; rel=&quot;nofollow&quot;&gt;https://pig.apache.org/docs/r0.9.1/func.html#replace&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-09-01T17:10:43.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13807" PostTypeId="1" CreationDate="2016-09-02T00:28:27.893" Score="1" ViewCount="956" Body="&lt;p&gt;I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. Ultimately I would like to have all of this data placed into a .CVS file. I have read &lt;em&gt;Learning Python the Hard Way&lt;/em&gt; as well as a good portion of &lt;em&gt;Automating the hard stuff with Python&lt;/em&gt;, however I am still quite a novice. I have referenced the following tutorials while trying to build this script:&#xA;&lt;a href=&quot;http://doc.scrapy.org/en/latest/intro/tutorial.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;the documentation for Scrapy&lt;/a&gt; &amp;amp; &lt;a href=&quot;http://mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post by Michael Herman&lt;/a&gt;. I feel as though my code simply has a dumb syntax error but I'm too ignorant of what to do to fix it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code for the spider:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scrapy.contrib.spiders import CrawlSpider, Rule&#xA;from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor&#xA;from scrapy.selector import HtmlXPathSelector&#xA;from craigslist_sample.items import CraigslistSampleItem&#xA;&#xA;class MySpider(CrawlSpider):&#xA;name = &quot;craigs&quot;&#xA;allowed_domains = [&quot;sfbay.craigslist.org&quot;]&#xA;start_urls = [&quot;http://salem.craigslist.org/search/jjj&quot;]&#xA;#Initially grab all of the urls up to where craigslist allows&#xA;#In this case, it's 2400 &#xA;for i in range(1, 24):&#xA;    start_urls.append(base_url + &quot;s=&quot; + str(i) + &quot;00&amp;amp;&quot;)&#xA;&#xA;rules = (&#xA;    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class=&quot;button next&quot;]',)), callback=&quot;parse_items&quot;, follow= True),&#xA;)&#xA;&#xA;def parse_items(self, response):&#xA;    hxs = HtmlXPathSelector(response)&#xA;    titles = hxs.xpath('//span[@class=&quot;pl&quot;]')&#xA;    items = []&#xA;    for titles in titles:&#xA;        item = CraigslistSampleItem()&#xA;        item[&quot;title&quot;] = titles.xpath(&quot;a/text()&quot;).extract()&#xA;        item[&quot;link&quot;] = titles.xpath(&quot;a/@href&quot;).extract()&#xA;        item[&quot;email&quot;] = titles.xpath(&quot;@anonemail&quot;).extract()&#xA;        items.append(item)&#xA;    return(items)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following is the exact error I am receiving:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*File &quot;C:\Users\newfa\Documents\scripts\craigslist_sample\craigslist_sample\spiders\test.py&quot;, line 27&#xA;item[&quot;email&quot;] = titles.xpath(&quot;@anonemail&quot;).extract()&#xA;^&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;IndentationError: unexpected indent*&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do understand that indentation is very important in Python, however I keep trying different methods of indentation and have tried several &quot;beautify&quot; code methods to try and get it right, which leads me to believe that it may be some other error. &lt;/p&gt;&#xA;" OwnerUserId="23997" LastEditorDisplayName="user380" LastEditDate="2017-07-11T11:10:06.273" LastActivityDate="2017-07-11T11:10:06.273" Title="Trouble with Scrapy and Python: ever present unexpected indent" Tags="|python|dataset|scraping|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13808" PostTypeId="2" ParentId="13807" CreationDate="2016-09-02T00:42:31.213" Score="1" Body="&lt;p&gt;After moments more of work, I was capable of solving my own problem. The error turned out to be a wild 'tab' mark that I had lost in the code there. Also, my 'allowed_domains' section was wonky. This is all corrected in the code below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from scrapy.contrib.spiders import CrawlSpider, Rule&#xA; from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor&#xA; from scrapy.selector import HtmlXPathSelector&#xA; from craigslist_sample.items import CraigslistSampleItem&#xA;&#xA;class MySpider(CrawlSpider):&#xA;name = &quot;test&quot;&#xA;allowed_domains = [&quot;salem.craigslist.org&quot;]&#xA;start_urls = [&quot;http://salem.craigslist.org/search/jjj&quot;]&#xA;&#xA;&#xA;rules = (&#xA;    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class=&quot;button next&quot;]',)), callback=&quot;parse_items&quot;, follow= True),&#xA;)&#xA;&#xA;def parse_items(self, response):&#xA;    hxs = HtmlXPathSelector(response)&#xA;    titles = hxs.xpath('//span[@class=&quot;pl&quot;]')&#xA;    items = []&#xA;    for titles in titles:&#xA;       item = CraigslistSampleItem()&#xA;       item[&quot;title&quot;] = titles.xpath(&quot;a/text()&quot;).extract()&#xA;       item[&quot;link&quot;] = titles.xpath(&quot;a/@href&quot;).extract()&#xA;       item[&quot;email&quot;] = titles.xpath(&quot;A/@anonemail&quot;).extract()&#xA;       items.append(item)&#xA;    return(items)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I am still not managing to acquire the email, phone number, or description data - however as these problems are not directly related to the question I posed its fair to say that it is answered.&lt;/p&gt;&#xA;" OwnerUserId="23997" LastActivityDate="2016-09-02T00:42:31.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13812" PostTypeId="2" ParentId="13740" CreationDate="2016-09-02T05:15:46.293" Score="1" Body="&lt;p&gt;Use some data set analysis (like Principal Component Analysis) trying to figure out the underlying structure of your data set. You could probably reduce the dimension without losing too much information.&lt;/p&gt;&#xA;" OwnerUserId="24002" LastActivityDate="2016-09-02T05:15:46.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13814" PostTypeId="1" CreationDate="2016-09-02T05:40:20.323" Score="5" ViewCount="314" Body="&lt;p&gt;Is there literature on how big a net needs to be to learn an arbitrary classification problem where the input is a n-bit integer and the output is one of k categories? I'm interested in both a theoretical bound and the size needed in practice with current training and initialization techniques. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be specific, the neural net is allowed to see all possible inputs and their respective outputs, so I will not hide anything from the training set, as the function is not assumed to have any structure whatsoever.&lt;/p&gt;&#xA;" OwnerUserId="24003" LastEditorUserId="24003" LastEditDate="2016-09-04T07:34:52.757" LastActivityDate="2016-09-04T07:34:52.757" Title="How big does a Neural Net need to be to learn anything" Tags="|neural-network|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13815" PostTypeId="1" CreationDate="2016-09-02T06:39:33.483" Score="3" ViewCount="221" Body="&lt;p&gt;So I'm training a classification task which takes as input some description of the state of a 2x2x2 rubix cube and outputs the optimal move to take. And a potential problem I noticed is that in many states more than one move is optimal. In particular, only about 46% of states have only one optimal move, and 24% have 2, 12% have three, etc. So I have a few choices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The options I thought of were&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;have each data point choose an optimal move at random&lt;/li&gt;&#xA;&lt;li&gt;Do cross-entropy minimization with the data point containing the same probability for each optimal move. i.e (0.5,0.5,0,0,0,0,0,0,0) if the first two are optimal&lt;/li&gt;&#xA;&lt;li&gt;Discard states with more than one optimal move (this seems really bad)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What is the standard practice? Also, is there a difference between 1 and 2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If necessary you may assume that I'm using a neural network model&lt;/p&gt;&#xA;" OwnerUserId="24003" LastActivityDate="2016-09-02T07:33:14.493" Title="How to deal with item belonging to more than one category" Tags="|classification|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13816" PostTypeId="2" ParentId="13815" CreationDate="2016-09-02T07:04:14.737" Score="1" Body="&lt;p&gt;There are multiple ways you can do it. The first two options you propose would be valid but I'm not sure if the log-loss allows you to use probabilities as target as opposed to just binary labels, but if that is the case they will work for what you want. I have another suggestion. Let's say there are at any point &lt;strong&gt;n&lt;/strong&gt; possible moves available (In case of a 2x2x2 there are three dimensions and in every dimension 4 moves I think, so that would be 12 options) you could train 12 classifiers or in the case of a neural networks just twelve sigmoids in the output layer. Then you can represent every move as a probability, represent the training targets as a vector of 1s and 0s with 1 as being an optimal move and then during classification pick the one with the highest probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An extension of your first option would be to add all the optimal moves as state. Say move 2 and 4 are optimal you could just add the same state twice with different labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: That said, I think your second option is the best one if the math works out (I'm uncertain about this and don't have time to figure it out right now) but the others will work as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2: According to Neil Slater the second option will work if it's implemented properly and I think that is the cleanest solution so I would go for that if possible, otherwise I would go for either go for the multiple samples for multiple optimal moves or the 12-way binary classification.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-09-02T07:33:14.493" LastActivityDate="2016-09-02T07:33:14.493" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13818" PostTypeId="1" AcceptedAnswerId="13819" CreationDate="2016-09-02T11:05:02.417" Score="3" ViewCount="11472" Body="&lt;p&gt;How Can I save a DataFrame as a CSV in HDFS?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying with this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupBy(&quot;Product_ID&quot;).agg(collect_list($&quot;Stock&quot;)).saveAsTextFile(&quot;PATH/results.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23714" LastEditorUserId="8432" LastEditDate="2017-06-05T22:17:58.540" LastActivityDate="2018-09-12T15:22:02.360" Title="value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame" Tags="|apache-spark|scala|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13819" PostTypeId="2" ParentId="13818" CreationDate="2016-09-02T11:07:16.607" Score="3" Body="&lt;p&gt;This is not standard part of the API of DataFrames. You can either map it to a RDD, join the row entries to a string and save that or the more flexible way is to use the DataBricks spark-csv package that can be found &lt;a href=&quot;https://github.com/databricks/spark-csv&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it's just one column you can map it to a RDD and just call &lt;code&gt;.saveAsTextFile(filename)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-09-02T11:52:43.800" LastActivityDate="2016-09-02T11:52:43.800" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13820" PostTypeId="1" CreationDate="2016-09-02T11:15:19.617" Score="1" ViewCount="9808" Body="&lt;p&gt;I installed orange version 3 yesterday. I defined variables in excel file as per instruction given in documentation. When I used simple plots  it works fine but in case of classifier it's not working. Error message: data has no target variable. But I already set and saved the target variables. What will be the solution? &#xA;Here is my data structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ENZYME TYPE AGE GROUP   CONTROL 2 YEAR EXP  Relative increase in mean (in %)after 2 years of Exposure w.r.t control 4 YEAR EXP  Relative increase in mean (in %)after 4 years of Exposure w.r.t control 6 YEAR EXP  Relative increase in mean (in %)after 6 years of Exposure w.r.t control 8 YEAR EXP  Relative increase in mean (in %)after 8years of Exposure w.r.t control  10 YEAR EXP Relative increase in mean (in %)after 10 years of Exposure w.r.t control&#xA;d   d   c   c   c   c   c   c   c   c   c   c   c&#xA;class   class                                           &#xA;SGOT    19 TO 26 YEARS  24.12   34.03478261 41% 38.11818182 58% 39.88333333 65% 49.56666667 105%    50.7    110%&#xA;SGOT    27 TO 34 YEARS  24.64   35.46666667 44% 38.25   55% 44.91428571 82% 49.7625 102%    50.88333333 107%&#xA;SGOT    35 TO 42 YEARS  25.36   37.6    48% 39  54% 43.375  71% 53.36666667 110%    55  117%&#xA;SGOT    43 TO 50 YEARS  28.9    37.76666667 31% 41.16666667 42% 44.7    55% 55.36666667 92% 56.96666667 97%&#xA;SGOT    51 TO 58 YEARS  31.36   42.96666667 37% 41.36666667 32% 45.33333333 45% 58.25   86% 59.23333333 89%&#xA;SGPT    19 TO 26 YEARS  21.8067 27.9826 28% 26.1909 20% 33.8222 55% 42  93% 34.2571 57%&#xA;SGPT    27 TO 34 YEARS  21.04   34.2167 63% 28.975  38% 38  81% 43.075  105%    42.1833 100%&#xA;SGPT    35 TO 42 YEARS  22.56   36.3    61% 40.9333 81% 39.825  77% 45.6667 102%    42.8    90%&#xA;SGPT    43 TO 50 YEARS  23.58   42.8667 82% 42.7667 81% 40.5333 72% 46.3    96% 43.2333 83%&#xA;SGPT    51 TO 58 YEARS  24.16   45.1333 87% 44.5333 84% 48.4333 100%    47.6433 97% 46.5    92%&#xA;GGT 19 TO 26 YEARS  13.3967 27.6935 107%    32.6818 144%    40.2167 200%    33.9667 154%    33.8286 153%&#xA;GGT 27 TO 34 YEARS  13.5    31.45   133%    41.225  205%    46.1429 242%    37.375  177%    37.2333 176%&#xA;GGT 35 TO 42 YEARS  15.66   37.8333 142%    43.9667 181%    47.35   202%    45.6667 192%    37.5667 140%&#xA;GGT 43 TO 50 YEARS  22.94   38.3667 67% 48.2    110%    47.9333 109%    46.4333 102%    39.7333 73%&#xA;GGT 51 TO 58 YEARS  24  38.9667 62% 50.6667 111%    49.1667 105%    47.11   96% 40.0667 67%&#xA;ALP 19 TO 26 YEARS  151.86  257.2587    69% 260.3727    71% 270.3722    78% 274.5333    81% 296.0571    95%&#xA;ALP 27 TO 34 YEARS  152.46  218.75  43% 246.15  61% 257.0714    69% 273.1   79% 287.0667    88%&#xA;ALP 35 TO 42 YEARS  153.58  235.9   54% 249.1333    62% 267.175 74% 276.9333    80% 300.6333    96%&#xA;ALP 43 TO 50 YEARS  152.8   265.9333    74% 266.0333    74% 287.3333    88% 286.4667    87% 286.2667    87%&#xA;ALP 51 TO 58 YEARS  153.14  235.0333    53% 254.9   66% 281.0667    84% 291.3767    90% 293 91%&#xA;CPK 19 TO 26 YEARS  92.0667 109.0435    18% 148.3636    61% 146.3889    59% 141.1667    53% 157.4286    71%&#xA;CPK 27 TO 34 YEARS  87.4    143.6667    64% 151.75  74% 147.5714    69% 164.25  88% 162.3333    86%&#xA;CPK 35 TO 42 YEARS  96.4    149 55% 163.3333    69% 169.5   76% 173 79% 172.3333    79%&#xA;CPK 43 TO 50 YEARS  93.6    173 85% 175.3333    87% 173 85% 177.6667    90% 181.3333    94%&#xA;CPK 51 TO 58 YEARS  89  144.6667    63% 155.6667    75% 176.3333    98% 155.7767    75% 177 99%&#xA;LDH 19 TO 26 YEARS  198.5   279.6957    41% 299.0455    51% 291.6667    47% 281.5   42% 296.5714    49%&#xA;LDH 27 TO 34 YEARS  172.6   272.5   58% 296 71% 276 60% 265 54% 288.3333    67%&#xA;LDH 35 TO 42 YEARS  137.4   246 79% 285.6667    108%    278 102%    283.3333    106%    306.3333    123%&#xA;LDH 43 TO 50 YEARS  204.4   290.3333    42% 292.6667    43% 291.6667    43% 288.3333    41% 322.3333    58%&#xA;LDH 51 TO 58 YEARS  138.2   262 90% 284.6667    106%    323 134%    375.6667    172%    324.6667    135%&#xA;BCHE    19 TO 26 YEARS  25833.967   8313.7609   -68%    6551    -75%    6289.4444   -76%    5670    -78%    5042.7143   -80%&#xA;BCHE    27 TO 34 YEARS  25271.8 10165.5 -60%    9682.25 -62%    7886.5714   -69%    6763.5  -73%    5960.5  -76%&#xA;BCHE    35 TO 42 YEARS  23864.2 19860.6667  -17%    16186   -32%    15793   -34%    15402.3333  -35%    23498   -2%&#xA;BCHE    43 TO 50 YEARS  21286   15834.6667  -26%    15279.6667  -28%    14917   -30%    14114.6667  -34%    13457   -37%&#xA;BCHE    51 TO 58 YEARS  19063   9336.6667   -51%    6752    -65%    4395    -77%    2474    -87%    2258.6667   -88%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24007" LastEditorUserId="15527" LastEditDate="2016-09-03T21:56:08.113" LastActivityDate="2016-09-03T21:56:08.113" Title="Target variable problem :- Classifier" Tags="|data|orange|" AnswerCount="1" CommentCount="0" ClosedDate="2016-09-03T21:31:06.730" ContentLicense="CC BY-SA 3.0" />
  <row Id="13821" PostTypeId="2" ParentId="13814" CreationDate="2016-09-02T12:48:52.943" Score="9" Body="&lt;p&gt;The size of the Neural Network is usually treated as a hyper-parameter. You need to estimate it from a separate cross-validation set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Neural nets there are no theoretical proven bounds (there are some papers here and there, but they are not applicable in practice). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to use the good old &quot;cross-validation&quot; method to search for the best fit value of the size of NN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. : There is a paper from Yoshua Bengio with all trick and traits for training such machines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1206.5533&quot;&gt;https://arxiv.org/abs/1206.5533&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-09-02T12:48:52.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13822" PostTypeId="1" CreationDate="2016-09-02T14:57:29.057" Score="1" ViewCount="1103" Body="&lt;p&gt;I am currently on a project where we have people place sticky notes on a X-Y axis to plot their beliefs on certain topics. These sticky notes can be of different colors to reflect different levels of their beliefs. What we are then left with is a picture that we take of the various plots to base our findings on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am hoping to avoid plotting each point individually in a data frame based on the picture. I am also relatively new to utilizing machine learning, specifically in relation to image recognition. I have done a couple of random forest projects, but nothing image processing related. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is this, using R, is it possible to use a machine learning package to read these pictures and create a data frame of X-Y coordinates of these sticky notes plots?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open to all suggestions. Thanks for any and all help and let me know if you need any clarification.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Edit:&lt;/em&gt;&lt;/strong&gt; Here is an example of what one of the images might look like. Unfortunately, I could only quickly find an X-axis only graph, but we do X and Y-axis graphs as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ANrFo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ANrFo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23915" LastEditorUserId="23915" LastEditDate="2016-09-02T15:15:36.533" LastActivityDate="2016-09-06T11:15:22.583" Title="Machine Learning to Get X, Y Coordinates From Picture of Plot" Tags="|machine-learning|r|image-recognition|" AnswerCount="1" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13823" PostTypeId="1" AcceptedAnswerId="15522" CreationDate="2016-09-02T15:31:28.160" Score="1" ViewCount="286" Body="&lt;p&gt;In general, what sort of &lt;a href=&quot;https://en.wikipedia.org/wiki/Supervised_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;supervised algorithms&lt;/a&gt; and techniques should I use on data that has the following charactersitcs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;2 potential classification labels?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;3-5 potential classification labels?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;6-10 potential classification labels?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;10-50 potential classification labels?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;50 or more potential classification labels?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Primary Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What algorithms learn most effectively at these different tiers of total possible class labels? &lt;/li&gt;&#xA;&lt;li&gt;What algorithms generally make the best predictions with the smallest amount of data at each of these tiers? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastEditorUserId="26596" LastEditDate="2016-12-05T19:58:06.863" LastActivityDate="2016-12-05T19:58:06.863" Title="What is the best strategy to use on data with many classification labels?" Tags="|machine-learning|classification|algorithms|beginner|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13826" PostTypeId="2" ParentId="13784" CreationDate="2016-09-02T19:00:23.913" Score="2" Body="&lt;p&gt;Let me talk only about my own attempt to transition from Software Engineering to Machine Learning area. First of all - it is definitely doable without becoming a Phd. I made my NLP stuff working pretty well using 3rd party libs after auditing some of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/learn/machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/specializations/jhu-data-science&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specializations/jhu-data-science&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/specializations/statistics&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specializations/statistics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the salaries - this article could probably answer your question and help with salary negotiations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.linkedin.com/pulse/how-much-does-data-scientist-make-ill-show-you-vin-vashishta&quot; rel=&quot;nofollow&quot;&gt;https://www.linkedin.com/pulse/how-much-does-data-scientist-make-ill-show-you-vin-vashishta&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="23945" LastActivityDate="2016-09-02T19:00:23.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13828" PostTypeId="1" AcceptedAnswerId="13835" CreationDate="2016-09-02T19:36:40.647" Score="11" ViewCount="41173" Body="&lt;p&gt;I have a data set which has continuous independent variables and a continuous dependent variable. To predict the dependent variable using the independent variables, I've run an ensemble of regression models and tried to compare them against each other. Here are the results for reference:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nbpzM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nbpzM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can interpret what the &lt;code&gt;R-squared value / Coefficient of determination&lt;/code&gt; for each of those models means. However, I can't understand what the &lt;code&gt;Negative Log Likelihood&lt;/code&gt; means. Especially, why is it &lt;em&gt;Infinity&lt;/em&gt; for Linear Regression and Boosted Decision Tree, and &lt;em&gt;a finite value&lt;/em&gt; for a Decision Forest Regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit:&lt;br&gt;&#xA;Data Description: The data that went into these three models is all continuous independent variables and a continuous dependent variable. There are a total of 542 observations and 26 variables. &lt;br&gt;&#xA;These 542 variables are split 70 - 30 to get training and testing datasets. Therefore, the training dataset has 379 observations and 26 variables; the testing dataset has 163 observations and 26 variables.&#xA;No missing data.&#xA;&lt;/b&gt;&lt;br&gt;&lt;br&gt;&#xA;&lt;b&gt;Edit 2 Possible Explanation - (&lt;a href=&quot;https://gallery.cortanaintelligence.com/Experiment/Compare-Regressors-5&quot; rel=&quot;noreferrer&quot;&gt;click here&lt;/a&gt;): Apparently, Linear Regression and Boosted Trees in Azure ML don't calculate the Negative Log-Likelihood metric - and that could be the reason that NLL is infinity or undefined in both cases.&lt;/b&gt;&lt;/p&gt;&#xA;" OwnerUserId="2647" LastEditorUserId="2647" LastEditDate="2016-09-06T12:58:29.733" LastActivityDate="2018-12-18T03:12:27.920" Title="What does Negative Log Likelihood mean?" Tags="|regression|ensemble-modeling|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13829" PostTypeId="1" CreationDate="2016-09-02T20:46:55.187" Score="0" ViewCount="105" Body="&lt;p&gt;I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just replace null values with current or most recent time stamp. Are there any standard techniques followed ?&lt;/p&gt;&#xA;" OwnerUserId="24019" LastActivityDate="2016-09-06T11:00:29.817" Title="How should one deal with null values in time stamp data types?" Tags="|data|" AnswerCount="2" CommentCount="1" ClosedDate="2016-09-07T07:09:44.983" ContentLicense="CC BY-SA 3.0" />
  <row Id="13830" PostTypeId="2" ParentId="13829" CreationDate="2016-09-02T21:59:52.393" Score="-1" Body="&lt;p&gt;It is not clear from your question what your use case is exactly. But for cases like what you mentioned, it is usually more helpful if you generate time series out of each feature. For example, if you have user's login and logout time, you can generate a time series where it is 1 when the user is logged in and 0 when the user is logged out. &#xA;But again, tell us more about your use case first. &lt;/p&gt;&#xA;" OwnerUserId="8149" LastActivityDate="2016-09-02T21:59:52.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13831" PostTypeId="2" ParentId="13820" CreationDate="2016-09-02T23:42:11.370" Score="2" Body="&lt;p&gt;According to the message I assume that the class variable is not set correctly. Can you try with &lt;a href=&quot;https://dl.dropboxusercontent.com/u/6942875/Workbook1.xlsx&quot; rel=&quot;nofollow noreferrer&quot;&gt;this excel file&lt;/a&gt;? I just removed percentage signs and renamed variables otherwise this is your data. First I loaded this as it is and then set the class variable in Orange. You can do this inside File widget by clicking on &lt;code&gt;feature&lt;/code&gt; string in the third column of the row of your target feature and select &lt;code&gt;target&lt;/code&gt; in dropdown. Then click apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Bo1iY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Bo1iY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make sure that target is set correctly connect it to Table widget and in the Info box you should see &lt;code&gt;Continuous target variable&lt;/code&gt;. After that I experimented with Test &amp;amp; Score and Linear Regression and all went well. Also, I suggest that you set only one target variable at a time as Orange might have problems with multi target prediction.&lt;/p&gt;&#xA;" OwnerUserId="15685" LastActivityDate="2016-09-02T23:42:11.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13832" PostTypeId="2" ParentId="12846" CreationDate="2016-09-03T04:44:44.073" Score="2" Body="&lt;p&gt;Mathematical detail of the two different tests can be found in two papers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]. &lt;a href=&quot;http://sci2s.ugr.es/keel/pdf/algorithm/articulo/shapiro1965.pdf&quot; rel=&quot;nofollow&quot;&gt;http://sci2s.ugr.es/keel/pdf/algorithm/articulo/shapiro1965.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]. &lt;a href=&quot;http://www.jstor.org/stable/2334522?seq=1#page_scan_tab_contents&quot; rel=&quot;nofollow&quot;&gt;http://www.jstor.org/stable/2334522?seq=1#page_scan_tab_contents&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, both methods are based on hypothesis testing. But they are using different test statistics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shapiro-Wilk test has many good analytical properties and was designed to apply to data with sample size less than $50$. However, when the sample size becomes greater, Shapiro-Walk test might be unreliable. As I quote from the second paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Shapiro and Wilk did not extend their test beyond samples of size 50. A number of reasons indicate that it is best not to make such an extension. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Essentially, that is why people invented the second normality test. As I quote from the second paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We present a new test of normality applicable for samples of size 50 or larger which possesses the desirable omnibus property&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="24024" LastActivityDate="2016-09-03T04:44:44.073" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13835" PostTypeId="2" ParentId="13828" CreationDate="2016-09-03T10:45:07.647" Score="7" Body="&lt;p&gt;Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now additive. If you don't understand what I've said, just remember the higher the value it is, the more likely your model fits the model. Google for &lt;code&gt;maximum likelihood estimation&lt;/code&gt; if you're interested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, your input data is bad. You should give your model a proper data set. While I don't have your data set, we can take a look at the likelihood function for linear regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KTkck.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KTkck.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will get infinity if the likelihood function is zero or undefined (that's because log(0) is invalid). Look at the equation, most likely your &lt;strong&gt;sample standard deviation&lt;/strong&gt; is &lt;strong&gt;zero&lt;/strong&gt;. If it's zero, the last term will be undefined. Have you given a data set that you copied and pasted the same data over rows?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boosted trees should also be undefined if your sample deviation is zero. However, decision tree is estimated based on impurity and won't crash here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;: please check and double check your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; I think you just had a bug. Linear regression will always give you something here. Have you fitted the models in R with the same dataset? –&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="64200" LastEditDate="2018-12-17T23:41:07.013" LastActivityDate="2018-12-17T23:41:07.013" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="13836" PostTypeId="1" AcceptedAnswerId="13842" CreationDate="2016-09-03T13:18:48.170" Score="0" ViewCount="1137" Body="&lt;p&gt;I'm trying to find some outliers on my database using HIVE and I'm using Standard Deviation technique. My query is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT ID&#xA;FROM data&#xA;WHERE ID &amp;lt; (AVG(ID) + STDDEV(ID))&#xA;  AND ID &amp;gt; (AVG(ID) - STDDEV(ID));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run this code I'm getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 3:12 Not yet supported place for UDAF 'AVG'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to solve this problem? Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23811" LastActivityDate="2016-09-03T19:50:54.390" Title="Find outliers in Hive - SemanticException" Tags="|statistics|data-cleaning|hive|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13839" PostTypeId="2" ParentId="13814" CreationDate="2016-09-03T19:03:56.757" Score="3" Body="&lt;p&gt;Let's look at some specific tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input is the bytecode of a program, as integers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's use a program that computes prime numbers. I'm pretty sure you can write such a program in a reasonably small number of bytes. Let's add another integer, the number of primes we want to find.&#xA;The output would be whether the second highest bit of the prime is 0 or 1 (i.e. if the prime begins with 0...010 or 0...011) Just two classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, let the input be a program, and the output whether the program halts, or not (cf. &quot;halting problem&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure, not even the largest neural network can predict the output reliably of these two examples even if you bound the input length. So &lt;strong&gt;there is no guarantee of being able to learn anything with just enough nodes&lt;/strong&gt;. what you can probably guarantee id the ability to overfit. Given N training examples (that have consistent labels), you can build a network with x nodes that perfectly memoizes the training data (but that is unable to generalize, so it is overfitted to the training data).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-09-03T19:03:56.757" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13842" PostTypeId="2" ParentId="13836" CreationDate="2016-09-03T19:50:54.390" Score="2" Body="&lt;p&gt;Seems like Hive doesn't let you use &lt;code&gt;avg&lt;/code&gt; in a where clause. You can solve this with a subquery.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT id&#xA;FROM &#xA;    (SELECT id, AVG(id) as avg_id, STDDEV(id) as stddev_id FROM data)&#xA;WHERE id &amp;lt; avg_id + stddev_id AND id &amp;gt; avg_id - stddev_id&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17250" LastActivityDate="2016-09-03T19:50:54.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13843" PostTypeId="2" ParentId="12679" CreationDate="2016-09-03T20:33:14.797" Score="1" Body="&lt;p&gt;This took me quite a while to get just right but here is what I did&#xA;    Download Ubuntu 16.04 .iso&#xA;    Download unetbootin&#xA;    Boot from drive to install ubuntu use 3rd parter drivers to avoid wireless issues&#xA;    Download the driver for the NVIDIA 1070 card 367.27&#xA;    Hit ctrl-alt-f1 to open a virtual terminal&#xA;    sudo service lightdm stop&#xA;    cd ~/Downloads&#xA;    sudo chmod 755 &quot;name of driver&quot;.run&#xA;    sudo ./&quot;name of driver&quot;.run&#xA;    cd&#xA;    reboot&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Download CUDA 8.0 and patch 1&#xA;Hit ctrl-alt-f1 to open a virtual terminal&#xA;sudo service lightdm stop&#xA;cd ~/Downloads&#xA;sudo chmod 755 &quot;name of CUDA installer&quot;.run&#xA;sudo ./&quot;name of CUDA installer&quot;.run --override&#xA;*Do not install the driver since we already did&#xA;sudo chmod 755 &quot;name of CUDA installer&quot;.run&#xA;sudo ./&quot;name of CUDA installer&quot;.run --override&#xA;sudo chmod 755 &quot;name of CUDA patch&quot;.run&#xA;sudo ./&quot;name of CUDA patch&quot;.run&#xA;cd&#xA;reboot&#xA;&#xA;Download cuDNN 5.1&#xA;cd ~/Downoads&#xA;&#xA;tar xvzf cudnn-8.0-linux-x64-v5.1.tgz&#xA;sudo cp cuda/include/cudnn.h /usr/local/cuda-8.0/include&#xA;sudo cp cuda/lib64/libcudnn* /usr/local/cuda-8.0/lib64&#xA;sudo chmod a+r /usr/local/cuda-8.0/include/cudnn.h /usr/local/cuda-8.0/lib64/libcudnn*&#xA;&#xA;Run and add the following to bash file&#xA;&#xA;export LD_LIBRARY_PATH=&quot;$LD_LIBRARY_PATH:/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64&quot;&#xA;export CUDA_HOME=/usr/local/cuda-8.0&#xA;&#xA;&#xA;sudo apt-get install python-pip python-dev&#xA;&#xA;sudo apt-get install git&#xA;&#xA;&#xA;$ git clone https://github.com/tensorflow/tensorflow&#xA;&#xA;sudo apt-get install openjdk-8-jdk&#xA;&#xA;&#xA;sudo apt-get install pkg-config zip g++ zlib1g-dev unzip&#xA;&#xA;Download bazel-0.3.1 for linux&#xA;&#xA;&#xA; chmod +x bazel-version-installer-os.sh&#xA; ./bazel-version-installer-os.sh --user&#xA;&#xA;sudo apt-get install python-numpy swig python-dev python-wheel&#xA;&#xA;cd ~/tensorflow&#xA;&#xA;./configure&#xA;&#xA;use CUDA 8.0 and cudnn 5.1.5&#xA;&#xA;bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer&#xA;&#xA;&#xA;bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu&#xA;&#xA;bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package&#xA;&#xA;bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg&#xA;&#xA;sudo pip install /tmp/tensorflow_pkg/tensorflow-0.10.0rc0-py2-none-any.whl&#xA;&#xA;&#xA;cd tensorflow/models/image/mnist&#xA;python convolutional.py&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24040" LastActivityDate="2016-09-03T20:33:14.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13844" PostTypeId="1" AcceptedAnswerId="13860" CreationDate="2016-09-03T20:45:41.177" Score="-2" ViewCount="868" Body="&lt;p&gt;I'm thinking about becoming a data analyst, and I'm wondering if programming knowledge is a must for this. I'm fairly strong with maths, but I have very little programming experience. Do all data analysts have strong programming skills (R, SAS, SQL, Python, etc.), or can it vary depending on the type of data analyst you are?&lt;/p&gt;&#xA;" OwnerUserId="24032" LastActivityDate="2016-09-05T07:29:33.823" Title="Do I need a strong programming background to become a data analyst?" Tags="|career|programming|" AnswerCount="2" CommentCount="0" ClosedDate="2016-09-05T13:27:58.483" ContentLicense="CC BY-SA 3.0" />
  <row Id="13845" PostTypeId="2" ParentId="13844" CreationDate="2016-09-03T20:54:20.037" Score="0" Body="&lt;p&gt;Well, it is quite hard to do data science without using the computer. Unless you can do Random Forests by hand :) But if you mean raw programming, then anything you will be using (in 'everyday' DS) is already implemented in R, Python etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the skills, I think R is the best choice - very intuitive, made to be easy to use, although at the expense of speed. Python is becoming the new standard but is script language and takes little more to learn. There is also Eviews for instance that is not really programming at all and is straightforward to use.&lt;/p&gt;&#xA;" OwnerUserId="21990" LastActivityDate="2016-09-03T20:54:20.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13846" PostTypeId="2" ParentId="12059" CreationDate="2016-09-03T23:51:03.263" Score="0" Body="&lt;p&gt;The R package robcompositions has a function for PCA on compositional data. Details can be found here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rpackages.ianhowson.com/cran/robCompositions/man/pcaCoDa.html&quot; rel=&quot;nofollow&quot;&gt;http://rpackages.ianhowson.com/cran/robCompositions/man/pcaCoDa.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20297" LastActivityDate="2016-09-03T23:51:03.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13850" PostTypeId="1" CreationDate="2016-09-04T15:54:13.877" Score="2" ViewCount="56" Body="&lt;p&gt;I have a target function, which heavily depends on vector prefix:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 2&lt;/strong&gt;, 0, 0, 1, 2, 4) -&amp;gt; &lt;strong&gt;A&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 2&lt;/strong&gt;, 0, 55, 1, 99, 1) -&amp;gt; &lt;strong&gt;A&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 2&lt;/strong&gt;, 124, 55, 1, 99, 71) -&amp;gt; &lt;strong&gt;A&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 3&lt;/strong&gt;, -5, 0, 1, 2, 4) -&amp;gt; &lt;strong&gt;B&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 3&lt;/strong&gt;, 12, 55, 1, 99, 1) -&amp;gt; &lt;strong&gt;B&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 3, 124&lt;/strong&gt;, 55, 1, 99, 71) -&amp;gt; &lt;strong&gt;C&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;strong&gt;1, 3, 124&lt;/strong&gt;, 21, 10, 195, 11) -&amp;gt; &lt;strong&gt;C&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Prefix length may vary.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;pre-condition:&lt;/strong&gt; a training set of vectors and their target values&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;target:&lt;/strong&gt; to be able to evaluate target value on arbitrary vector&lt;/p&gt;&#xA;&lt;p&gt;I have tried HMM, but it doesn't seem to work well in this case. What algorithm should I use instead?&lt;/p&gt;&#xA;" OwnerUserId="16162" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-09-04T20:34:17.907" Title="ML on dataset with heavy dependency on prefix" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13854" PostTypeId="1" CreationDate="2016-09-04T19:01:31.330" Score="1" ViewCount="585" Body="&lt;p&gt;During comparison estimates/ coefficients in 'R'&amp;amp; Spark Mllib of Logistic regression, It has been observed that estimates are not same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On further investigation, I found that R &amp;amp; Mllib has different implementations for Logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;R's glm is return a maximum likelihood estimate of the model while&#xA;  Spark's LogisticRegressionWithLBFGS is return a regularized model&#xA;  estimate.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Maximum likelihood estimates seems more efficient, as per literature available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious to know, why Spark Mllib developers has not chosen 'Maximum likelihood estimation' technique?&lt;/p&gt;&#xA;" OwnerUserId="24053" LastActivityDate="2016-09-05T05:56:30.343" Title="Why Logistic regression into Spark Mllib does not use Maximum likelihood estimation?" Tags="|machine-learning|r|apache-spark|logistic-regression|" AnswerCount="1" CommentCount="1" ClosedDate="2016-09-07T07:08:34.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="13855" PostTypeId="2" ParentId="13850" CreationDate="2016-09-04T20:34:17.907" Score="2" Body="&lt;p&gt;Consider LSTMs. They're a type of recurrent neural network that are generally very good at sequence prediction tasks and have the ability to learn very long range dependencies like you describe. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;Andrej Karpathy gives a tremendously great intro to RNN/LSTM on his blog.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although going as far as a neural network opens up a whole new range of problems, such as long training time and finding the right architecture and gradient descent parameters. If your problem is very dependent on only a handful of terms then you may be able to get adequate results with a tabular multinomial regression approach and some smart encoding of your sequence. Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y prefix1 prefix2 prefix3 has1 has2 has3  ...&#xA;A       1       2       0    1    1    0  ...&#xA;B       1       3      -5    1    0    1  ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17250" LastActivityDate="2016-09-04T20:34:17.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13858" PostTypeId="2" ParentId="13854" CreationDate="2016-09-05T05:56:30.343" Score="3" Body="&lt;p&gt;I think you're confused. Maximum likelihood is a general technique for giving the most likely parameter estimation. There is no closed solution for maximum likelihood for logistic regression, so both R and Spark must numerically estimate it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How exactly the likelihood is estimated could be slightly different and might depends on the implementation. For example, Spark prefers to add regularization, R might like to use something else (you'll need to check the documentation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;: Both R and Spark try to estimate ML for you. Please check the R documentation on how R do it.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-09-05T05:56:30.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13859" PostTypeId="1" AcceptedAnswerId="13863" CreationDate="2016-09-05T07:17:57.893" Score="2" ViewCount="101" Body="&lt;p&gt;I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, where you would need to get a consensus from the ensemble at the end? Or is dropout run over one single network?&lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="15533" LastEditDate="2016-09-05T08:37:55.843" LastActivityDate="2016-09-05T13:47:56.827" Title="Does dropout require multiple passes of the same data set, as a sort of ensemble method?" Tags="|neural-network|deep-learning|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13860" PostTypeId="2" ParentId="13844" CreationDate="2016-09-05T07:29:33.823" Score="0" Body="&lt;p&gt;I would say you don't need to already have lots of programming experience, but being generally mathematically and computer-literate is important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you've literally never programmed a computer before, then dig up a basic online R or python tutorial. As to which one, depends on the industry you'll be working in. Tech companies tend to use python, elsewhere R might be slightly more prevalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are various &quot;drag-and-drop&quot; software &quot;solutions&quot; that don't in theory need programming, but in most real-life applications you'll find that there's functionality you need that they don't have, or you need to pre- or post-process your data in some way, and you'll have to resort to R/python/SAS/... to get that sorted out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll find that you are most likely able to learn the coding skills you need on the job, as long as you have just a little experience with writing code.&lt;/p&gt;&#xA;" OwnerUserId="23911" LastActivityDate="2016-09-05T07:29:33.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13861" PostTypeId="2" ParentId="10758" CreationDate="2016-09-05T07:37:43.860" Score="1" Body="&lt;p&gt;This is normally called &quot;Ensemble learning&quot;: &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow&quot;&gt;wiki link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian Model Combination is a popular modern technique which usually delivers better results than Bayesian Model averaging, on which it was based. The latter had a tendency to give almost all the weight to one model.&lt;/p&gt;&#xA;" OwnerUserId="23911" LastActivityDate="2016-09-05T07:37:43.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13863" PostTypeId="2" ParentId="13859" CreationDate="2016-09-05T13:47:56.827" Score="4" Body="&lt;p&gt;Dropout is applied over one network. Sometimes (like with non-dropout networks) you will run your data through it multiple times before it converged and this number will be a bit higher on average with dropout but it is one network. Per layer you have a dropout probability during training and during testing/prediction you use the full network without dropout by multiplying the weights by (1-p) where p is the dropout probability. This is because during training (1-p) of the nodes are actually used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why this is related to ensembles is that every training instance is basically trained on a different network, by randomly dropping out nodes, forcing it to learn specific things using different nodes because it will not have all the nodes available at all times. It is not a traditional ensemble in that you combine multiple networks, just during training it acts a bit like it.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-05T13:47:56.827" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13864" PostTypeId="1" CreationDate="2016-09-05T15:52:30.737" Score="9" ViewCount="24556" Body="&lt;p&gt;I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification challenges. I only found &lt;a href=&quot;http://www.image-net.org/download-imageurls&quot; rel=&quot;noreferrer&quot;&gt;http://www.image-net.org/download-imageurls&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is the data used for the ImageNet ILSVRC 2012 (or later) classification challenge?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-09-05T20:36:26.137" Title="How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?" Tags="|dataset|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13867" PostTypeId="2" ParentId="13864" CreationDate="2016-09-05T20:36:26.137" Score="13" Body="&lt;p&gt;You can use the site &lt;a href=&quot;http://academictorrents.com/browse.php?search=imagenet&quot; rel=&quot;noreferrer&quot;&gt;academic torrents&lt;/a&gt;. The imagenet data you are looking for is shared there.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-05T20:36:26.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13868" PostTypeId="1" AcceptedAnswerId="13869" CreationDate="2016-09-05T22:26:27.087" Score="-3" ViewCount="1103" Body="&lt;p&gt;The code below measures precision and recall and F-measure (&lt;a href=&quot;http://streamhacker.com/2010/05/17/text-classification-sentiment-analysis-precision-recall/&quot; rel=&quot;nofollow&quot;&gt;source&lt;/a&gt;). How can I measure AUC?&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import collections&#xA;import nltk.metrics&#xA;from nltk.classify import NaiveBayesClassifier&#xA;from nltk.corpus import movie_reviews&#xA;&#xA;def word_feats(words):&#xA;    return dict([(word, True) for word in words])&#xA;&#xA;negids = movie_reviews.fileids('neg')&#xA;posids = movie_reviews.fileids('pos')&#xA;&#xA;negfeats = [(word_feats(movie_reviews.words(fileids=[f])), 'neg') for f in negids]&#xA;posfeats = [(word_feats(movie_reviews.words(fileids=[f])), 'pos') for f in posids]&#xA;&#xA;negcutoff = len(negfeats)*3/4&#xA;poscutoff = len(posfeats)*3/4&#xA;&#xA;trainfeats = negfeats[:negcutoff] + posfeats[:poscutoff]&#xA;testfeats = negfeats[negcutoff:] + posfeats[poscutoff:]&#xA;print 'train on %d instances, test on %d instances' % (len(trainfeats), len(testfeats))&#xA;&#xA;classifier = NaiveBayesClassifier.train(trainfeats)&#xA;refsets = collections.defaultdict(set)&#xA;testsets = collections.defaultdict(set)&#xA;&#xA;for i, (feats, label) in enumerate(testfeats):&#xA;    refsets[label].add(i)&#xA;    observed = classifier.classify(feats)&#xA;    testsets[observed].add(i)&#xA;&#xA;print 'pos precision:', nltk.metrics.precision(refsets['pos'], testsets['pos'])&#xA;print 'pos recall:', nltk.metrics.recall(refsets['pos'], testsets['pos'])&#xA;print 'pos F-measure:', nltk.metrics.f_measure(refsets['pos'], testsets['pos'])&#xA;print 'neg precision:', nltk.metrics.precision(refsets['neg'], testsets['neg'])&#xA;print 'neg recall:', nltk.metrics.recall(refsets['neg'], testsets['neg'])&#xA;print 'neg F-measure:', nltk.metrics.f_measure(refsets['neg'], testsets['neg'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24095" LastEditorUserId="24000" LastEditDate="2016-09-06T10:03:41.020" LastActivityDate="2016-09-06T10:03:41.020" Title="I need to measure Performance : AUC for this code of NLTK and skLearn" Tags="|python|scikit-learn|nltk|" AnswerCount="1" CommentCount="2" ClosedDate="2016-09-07T07:09:23.430" ContentLicense="CC BY-SA 3.0" />
  <row Id="13869" PostTypeId="2" ParentId="13868" CreationDate="2016-09-06T05:39:23.120" Score="3" Body="&lt;p&gt;It is unclear if you are requesting AUC of ROC or Precision-Recall curve. However, instead of storing the indices of examples in sets, you can store the labels in lists and use sklearn's &lt;code&gt;auc&lt;/code&gt; function after running &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;precision_recall_curve&lt;/code&gt;&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;roc_curve&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import precision_recall_curve&#xA;from sklearn.metrics import roc_curve&#xA;from sklearn.metrics import auc&#xA;&#xA;def label2int(label):&#xA;    if label == 'pos':&#xA;        return 1&#xA;    else:&#xA;        return 0&#xA;&#xA;y_true, y_score = [], []&#xA;&#xA;for i, (feats, label_true) in enumerate(testfeats):&#xA;    label_predicted = classifier.classify(feats)&#xA;    y_true.append(label2int(label_true))&#xA;    y_score.append(label2int(label_predicted))&#xA;&#xA;# Precision-Recall AUC&#xA;precision, recall, _ = precision_recall_curve(y_true, y_score, pos_label=1)&#xA;pr_auc = auc(recall, precision)&#xA;print &quot;Precision-Recall AUC: %.2f&quot; % pr_auc&#xA;# ROC AUC&#xA;fpr, tpr, _ = roc_curve(y_true, y_score, pos_label=1)&#xA;roc_auc = auc(fpr, tpr)&#xA;print &quot;ROC AUC: %.2f&quot; % roc_auc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Precision-Recall AUC: 0.82&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;ROC AUC: 0.73&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2016-09-06T08:31:26.697" LastActivityDate="2016-09-06T08:31:26.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13870" PostTypeId="2" ParentId="13829" CreationDate="2016-09-06T11:00:29.817" Score="1" Body="&lt;p&gt;Leave those time stamps null in order to indicate that you do not possess the time of those events. Everything else depends on the logic of your application.&lt;/p&gt;&#xA;" OwnerUserId="23910" LastActivityDate="2016-09-06T11:00:29.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13871" PostTypeId="2" ParentId="13822" CreationDate="2016-09-06T11:15:22.583" Score="2" Body="&lt;p&gt;Learning means you have examples of a complex behavior and can learn the dependency of the behavior on some parameters implicitly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your picture, one can see the dependency rather explicitly and the dependency is rather simple: A big black disk for every sticky-note.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Take picture data (red, green, blue and each 0..255) and for each pixel calculate total squared difference to a black disk of exactly the same diameter as the black dots in the picture (sum differences for each color). This means you compute the difference pixelwise and then square it pixelwise and then add up over the area of the disk.&lt;/li&gt;&#xA;&lt;li&gt;Find local minima of these differences below a threshold.&lt;/li&gt;&#xA;&lt;li&gt;The positions of these local minima are the desired positions.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Unfortunately I'm not good enough in R but I believe it can be done in a few lines if loops exist in R.&lt;/p&gt;&#xA;" OwnerUserId="23910" LastActivityDate="2016-09-06T11:15:22.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13874" PostTypeId="1" AcceptedAnswerId="13875" CreationDate="2016-09-06T13:00:52.300" Score="0" ViewCount="80" Body="&lt;p&gt;I have collected dataset with two class labels and used the SVM Method to classify the dataset, and this is the results. Does this appear suspicious or not?&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;scikit-learn classifiers with SVM SVC train on 114859 instances, test on 49227 instances&#xA;Excution (Training) Time:  9.82799983025&lt;/p&gt;&#xA;&lt;p&gt;Excution (Testing) Time:  3.75&lt;/p&gt;&#xA;&lt;p&gt;accuracy: 0.999837487558&lt;/p&gt;&#xA;&lt;p&gt;Precision-Recall AUC: 1.00&lt;/p&gt;&#xA;&lt;p&gt;ROC AUC: 1.00&lt;/p&gt;&#xA;&lt;p&gt;pos precision: 0.999822253822&lt;/p&gt;&#xA;&lt;p&gt;pos recall: 1.0&lt;/p&gt;&#xA;&lt;p&gt;pos F-measure: 0.999911119012&lt;/p&gt;&#xA;&lt;p&gt;neg precision: 1.0&lt;/p&gt;&#xA;&lt;p&gt;neg recall: 0.998107404779&lt;/p&gt;&#xA;&lt;p&gt;neg F-measure: 0.999052806062&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="24095" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-09-06T14:35:33.520" Title="What do this Classification evaluation results mean to you? Do they are suspicious or not?" Tags="|performance|model-evaluations|confusion-matrix|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13875" PostTypeId="2" ParentId="13874" CreationDate="2016-09-06T14:04:03.063" Score="1" Body="&lt;p&gt;The statistics here are obviously very good, in fact too good for any practical data set. Your model is almost perfect... Unfortunately, it's practically useless and I'll explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In machine learning, if you see something like this you know you are in &lt;strong&gt;trouble&lt;/strong&gt;. That can happen if there are problems with your data workflow. For example, you might have removed all outliers that you shouldn't, or you actually used a subset of your training data for the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's fine if you're just toying SVM, but you'll never encounter something like this in real life.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-09-06T14:35:33.520" LastActivityDate="2016-09-06T14:35:33.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13876" PostTypeId="2" ParentId="13874" CreationDate="2016-09-06T14:28:54.443" Score="0" Body="&lt;p&gt;In the dark I would say that it is suspicious. However this really depends on the problem you are solving. Certain datasets will have a lot of clear examples, a very easy task to solve etcetera.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could have happened is that during the preprocessing somehow your targets leaked into your features. Some more advanced preprocessing techniques like aggregating numerical features and the target per category and appending those to your features allow for such mistakes and will lead to extremely high accuracy. Any information that is not available during new predictions should not be used in your training set.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-06T14:28:54.443" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13878" PostTypeId="1" CreationDate="2016-09-06T15:24:00.467" Score="3" ViewCount="607" Body="&lt;p&gt;I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response variable to the 100 or so variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that it is often difficult to compare two or more datasets and attribute the response variable to the 100 or so variables, due the large number of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do people do when faced with such a problem? What are some common analyses for doing this kind of work? It's preferable if the solution can be explained to people with a non-technical background. &lt;/p&gt;&#xA;" OwnerUserId="24109" LastEditorUserId="24111" LastEditDate="2016-09-07T01:39:22.183" LastActivityDate="2024-03-18T10:03:50.453" Title="How to quantitatively compare two or more complex data sets" Tags="|dataset|data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13880" PostTypeId="1" CreationDate="2016-09-06T15:58:56.180" Score="1" ViewCount="97" Body="&lt;p&gt;This image looks pretty familiar to anyone getting acquainted with neural networks, and on first glance it makes a lot of intuitive sense.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HAQVn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HAQVn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But on second, third, fourth, etc glance, some questions pop up! I hope I'm not the only one with these concerns.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 3 circles representing Input nodes. Does this mean that the data sent into the NN has to be separated into multiple batches? I don't hear much talk of that, or any kind of emphasis / reasoning. Why don't we send the data through the network all at once? Can't the video card handle it? (Or is that the reason why right there, that the card can't handle it?).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 4 circles representing Hidden nodes. Each previous node points to&lt;br&gt;&#xA;a Hidden one. The way this is show, it is confusing:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that data coming in is somehow merged together? I see 3 arrows all coalescing together into the next layer. That's what it looks like visually.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some order, or sequence here? There's no indication of this process happening in some order. It just looks like a mess of arrows pointing from Input to Hidden. Shouldn't be be concerned with the order in which these activations flow? After all, we are looking at a network graph.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 circles representing Output nodes. Is this showing that a NN can output 2 different sets of data?? Why do we want multiple outputs? We started with 3 blocks of data, now we have 2. How are 2 separate blocks of data going to help us make a binary choice on, say, image classification?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more I look at this image, the more I dislike it. Yet I see it everywhere, in many introductory / intermediate discussions of NN. I'm hoping there is a reason why such an unclear image is so popular. &lt;/p&gt;&#xA;" OwnerUserId="15533" LastActivityDate="2016-09-06T16:24:00.720" Title="Graphs demonstrating the structure of neural networks are very unclear" Tags="|neural-network|graphical-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13881" PostTypeId="2" ParentId="13880" CreationDate="2016-09-06T16:24:00.720" Score="2" Body="&lt;p&gt;You are making one mistake which cascades on towards other mistakes. The multiple inputs are different features of the same sample. Let's say we want to regress house prices based on some features of a house. These three nodes could represent the rating of the neighborhood, the surface area and the number of bedrooms for example. Then you take a weighted sum to every of the hidden nodes. Every hidden node takes different weights over these features. The order you mention is per layer, since we need the output of the previous layer as input for the next one, these dependencies force this order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to output, you can have a combined loss. In case of binary classification you would probably just have a sigmoid activation at the end, however if you would have multiclass classification you would have a softmax layer at the end, one output for every potential class. There are a lot of other things you could want to output, for example a bounding box so you would need coordinates of where the network thinks the box is, or even per pixel if it's part of the object or not, then you would have a &lt;strong&gt;m&lt;/strong&gt; x &lt;strong&gt;n&lt;/strong&gt; grid as output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: All the things you mentioned about the GPU etcetera, that is way too far for what this image represents. It's one network for one sample. I do agree it would have been better to just have one output node because most more basic examples in supervised learning have just 1 target.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-06T16:24:00.720" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13882" PostTypeId="1" CreationDate="2016-09-06T16:52:04.713" Score="5" ViewCount="1144" Body="&lt;p&gt;I'm very confuse with the term Data Engineer and Data Scientist. There are lot of jobs available for both roles in current market with almost same technical skills requirement. Are they same or different ? Which role is recommended for someone from coding background (C++, Java, Python) along with RDBMS knowledge ?&lt;/p&gt;&#xA;" OwnerUserId="23673" LastActivityDate="2018-05-18T07:48:50.607" Title="Difference between Data Engineer and Data Scientist" Tags="|career|definitions|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13883" PostTypeId="2" ParentId="13882" CreationDate="2016-09-06T18:29:36.300" Score="4" Body="&lt;p&gt;Data engineering is infrastructure work; maintaining &quot;big data&quot; pipelines from ingestion to output. Today you might be expected to know things like SQL, Hadoop, Spark, Docker, and AWS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data science is an umbrella term, so it can mean a lot of things, including data engineering. But it can also mean pure data analysis without any production work. It really depends on who's using the term; read the job description and ask the company for details.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-06T18:29:36.300" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13892" PostTypeId="1" AcceptedAnswerId="13952" CreationDate="2016-09-07T12:56:32.213" Score="0" ViewCount="361" Body="&lt;p&gt;I've the following dataset: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;**Strenght  Movie1  Movie2** &#xA;  23            2        3 &#xA;  80            1        2  &#xA;  10            4        3 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to create a graph with the relationships between movies having the first column as the strenght of the relationship. How can I do this using R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-09-12T13:52:47.137" Title="Network Analysis using R" Tags="|r|social-network-analysis|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13894" PostTypeId="1" AcceptedAnswerId="22559" CreationDate="2016-09-07T15:14:56.833" Score="21" ViewCount="78236" Body="&lt;p&gt;In the &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;noreferrer&quot;&gt;Keras blog on training convnets from scratch&lt;/a&gt;, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I think not). If there was a separate test folder on similar lines as the train and validation folders, how do we get a confusion matrix for the test data. I know that we have to use scikit learn or some other package to do this, but how do I get something along the lines of class wise probabilities for test data? I am hoping to use this for the confusion matrix.&lt;/p&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="20372" LastEditDate="2017-08-24T22:03:17.023" LastActivityDate="2020-08-20T23:10:27.930" Title="How to get predictions with predict_generator on streaming test data in Keras?" Tags="|machine-learning|python|deep-learning|keras|confusion-matrix|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13895" PostTypeId="2" ParentId="13894" CreationDate="2016-09-07T16:25:29.697" Score="0" Body="&lt;p&gt;For confusion matrix you have to use sklearn package. I don't think Keras can provide a confusion matrix. For predicting values on the test set, simply call the model.predict() method to generate predictions for the test set. The type of output values depends on your model type i.e. either discrete or probabilities. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-07T16:25:29.697" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13897" PostTypeId="1" AcceptedAnswerId="13898" CreationDate="2016-09-07T18:32:26.410" Score="1" ViewCount="1180" Body="&lt;p&gt;I have a corpus on which I want to perform sentiment analysis using LSTM and word embeddings. I have converted the words in the documents to word vectors using Word2Vec. My question is how to input these word vectors as input to Keras? I don't want to use the embeddings provided by Keras.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T13:49:55.247" LastActivityDate="2020-12-25T13:49:55.247" Title="Word vectors as input" Tags="|deep-learning|keras|word-embeddings|word2vec|sentiment-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13898" PostTypeId="2" ParentId="13897" CreationDate="2016-09-07T19:32:37.990" Score="2" Body="&lt;p&gt;You can just skip the Embedding layer and use a normal input layer with &lt;strong&gt;n&lt;/strong&gt; input nodes where &lt;strong&gt;n&lt;/strong&gt; is the dimensions of your word2vec embeddings. The rest is the same as you would with an embedding layer, just pass a sequence of &lt;strong&gt;n&lt;/strong&gt; dimensional vectors as the input, potentially padded or truncated depending on your model.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-07T19:32:37.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13900" PostTypeId="2" ParentId="13897" CreationDate="2016-09-07T20:15:34.130" Score="1" Body="&lt;p&gt;Keras does not provide pre-trained word embeddings out of the box.  Do you need to avoid using the Embedding layer entirely?  If not, you can use that layer as an input for a matrix of pre-trained word vectors.  You can load in your pre-trained embeddings as the initial layer weights using the &lt;code&gt;weights&lt;/code&gt; property of the layer.  If you set &lt;code&gt;trainable=False&lt;/code&gt; then your word embeddings will not shift when you run your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a snippet of code from an &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/pretrained_word_embeddings.py#L116&quot; rel=&quot;nofollow&quot;&gt;example using pre-trained word vectors&lt;/a&gt; from the Keras Github repo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;embedding_layer = Embedding(nb_words + 1,&#xA;                        EMBEDDING_DIM,&#xA;                        weights=[embedding_matrix],&#xA;                        input_length=MAX_SEQUENCE_LENGTH,&#xA;                        trainable=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you do need to avoid the Embedding layer entirely, then @Jan's probably gives you what you need.&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-09-07T20:15:34.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13901" PostTypeId="1" AcceptedAnswerId="13903" CreationDate="2016-09-07T22:40:00.723" Score="12" ViewCount="6171" Body="&lt;p&gt;I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine learning tasks with Big Datasets (like 100s GB or TB). Appreciate if fellow data scientists can share their experience. Here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Obviously, very large datasets take longer time to train (can be days or weeks). Many times we need to train various models (SVM, Neural Network, etc.) to compare and find better performance model. I suspect, in industry projects, we want the results as quick as possible but produce the best performance. Are there any tips for reducing the training &amp;amp; testing time? If you recommend subsetting the dataset, I will be interested to learn how best to subset the dataset to cover all or majority of scenarios from the dataset.&lt;/li&gt;&#xA;&lt;li&gt;We know that performing cross validation is better as it may reduce over-fitting. However, cross validation also takes time to train and the model trained with cross validation may not be implemented straight (speaking from python sklearn experience: I need to train the model with dataset again after cross validation testing for it to be implemented). Do you normally do cross validation in your big data projects or getting by with the train-test split?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Appreciate the feedback.  &lt;/p&gt;&#xA;" OwnerUserId="24153" LastActivityDate="2020-06-04T16:25:36.850" Title="Machine Learning Best Practices for Big Dataset" Tags="|machine-learning|predictive-modeling|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13902" PostTypeId="2" ParentId="13901" CreationDate="2016-09-08T03:42:28.493" Score="5" Body="&lt;p&gt;The question is, how much data does it take to saturate your model? To determine this you can plot &lt;a href=&quot;http://www.astroml.org/sklearn_tutorial/practical.html#learning-curves&quot; rel=&quot;noreferrer&quot;&gt;learning curves&lt;/a&gt; with varying amounts of data, perhaps scaling up/down in size by a constant factor. If training on all the data is not feasible, the learning curve can help you make an informed trade-off. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model saturation also comes into play with cross-validation. If you are not near saturation, using a small number of folds will give you pessimistic performance figures because you will be training your model on less data than it can make use of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally and conversely, you can use a more complex model instead of &quot;resizing&quot; the data to suit the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Welcome to DataScience.SE.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-08T03:42:28.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13903" PostTypeId="2" ParentId="13901" CreationDate="2016-09-08T05:30:02.793" Score="12" Body="&lt;p&gt;I'll list some practices I've found useful, hope this helps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end consumer is willing to wait, you may need to reset their expectations, or get faster hardware/software to build the model; but do not skip cross validation. Plotting learning curves and cross-validation are effective steps to help guide us so we recognize and correct mistakes earlier in the process. I've experienced instances when a simple train-test set does not reveal any problems until I run cross-fold validations and find a large variance in the performance of the algorithm on different folds.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Before sizing up a dataset, eliminate the records with missing values of key variables and outliers, columns of highly correlated variables, and near zero variance variables. This will give you a much better estimate of the real usable dataset. Sometimes you may end up with only a fraction of the available dataset that can actually be used to build a model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When sizing up a dataset for building a model, it is easier to estimate the computing resources if you enumerate the dataset in rows and columns and memory size of the final numeric matrix. Since every machine learning algorithm is ultimately going to convert the dataset into a numeric matrix, enumerating the dataset size in terms of GBs/TBs of raw input data (which may be mostly strings/textual nominal variables/etc.) is often misleading and the dataset may appear to be more daunting and gigantic to work with than it is.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once you know (or estimate) the final usable size of your dataset, check if you have a suitable machine to be able to load that into memory and train the model. If your dataset size is smaller than memory available/usable by the software, then you need not worry about the size any longer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the dataset size is larger than the memory available to train a model, then you could try these approaches (starting from the simplest ones first):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use a machine with more memory: If you're using a cloud service provider then the simplest approach could be just to provision more memory and continue building the model as usual. For physical machines, try to procure additional RAM, its price continues to reduce and if your dataset is going to remain this big or grow bigger over time, then it is a good investment.&lt;/li&gt;&#xA;&lt;li&gt;Add nodes to the cluster: For Hadoop and Spark based cluster computing deployments, training on a larger data-set is as easy as adding more machines to the cluster.&lt;/li&gt;&#xA;&lt;li&gt;Quite often classification tasks require training on data with highly imbalanced classes, the ratio of positive to negative classes could sometimes be as large as 1:1000 or more. A straightforward method to improve accuracy in these cases is to either over-sample the minority class or under-sample the majority class, or do both together. If you have a large dataset, under-sampling the majority class is a very good option which will improve your algorithm's accuracy as well as reduce training time.&lt;/li&gt;&#xA;&lt;li&gt;Build an ensemble: Split the dataset randomly and train several base learners on each part, then combine these to get the final prediction. This would most effectively make use of the large dataset and produce a more accurate model. But you need to spend more time to carefully build the ensemble and keep clear of the usual pitfalls of ensemble building.&lt;/li&gt;&#xA;&lt;li&gt;If you're using an ensemble, train many single-thread models in parallel. Almost all ML software provide features to train multiple models on different cores or separate nodes altogether.&lt;/li&gt;&#xA;&lt;li&gt;Evaluate multiple different algorithms on the time taken to train them for your specific dataset vs. their accuracy. While there is no universal answer, but I've found when using noisy data, SVMs take much longer time to train than carefully built ensemble of regularized regression models, but may be only slightly more accurate in performance; and a well built neural network may take a very long time to train as compared to a CART tree, but perform significantly more accurately that the tree.&lt;/li&gt;&#xA;&lt;li&gt;To reduce time taken to build the model, try to automate as much of the process as you can. A few hours spent automating a complex error-prone manual task may save your team a hundred hours later in the project.&lt;/li&gt;&#xA;&lt;li&gt;If available, use those algorithm implementations which use parallel processing, sparse matrices and cache aware computing, these reduce processing time significantly. For example, use xgboost instead of a single-core implementation of GBM.&lt;/li&gt;&#xA;&lt;li&gt;If nothing else works, train the model on a smaller dataset; as Emre has suggested in his answer, use learning curves to fix the smallest sample size required for training the model, adding more training records than this size does not improve model accuracy noticeably. Here is a good article which explores this situation - &lt;a href=&quot;http://www.cs.columbia.edu/~vondrick/largetrain.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;largetrain.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="58341" LastEditDate="2020-06-04T16:25:36.850" LastActivityDate="2020-06-04T16:25:36.850" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="13904" PostTypeId="1" AcceptedAnswerId="14059" CreationDate="2016-09-08T06:12:15.983" Score="1" ViewCount="192" Body="&lt;p&gt;Can I please get direction on what is wrong in the code? All forecasts provide output except the ones listed bellow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so basically doesn't pick &lt;code&gt;['UserDefinedData4']&lt;/code&gt; and &lt;code&gt;['ItemId']&lt;/code&gt; separate ... only when both are  in &lt;code&gt;group_level&lt;/code&gt; ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this link is with the program:&#xA;&lt;a href=&quot;https://github.com/sergiubuciumas/issue&quot; rel=&quot;nofollow&quot;&gt;program location&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;elif forecast_type == 4:&#xA;    # forecast weekly order quantity at item level with the dummy variable 'Quarter' using AR(8)&#xA;    mape = forecast(dataset, 'W', ['ItemId'], 'Quarter', 8, output_file)&#xA;elif forecast_type == 5:&#xA;    # forecast weekly order quantity at item level with the dummy variable 'Month' using AR(8)&#xA;    mape = forecast(dataset, 'W', ['ItemId'], 'Month', 8, output_file)&#xA;elif forecast_type == 6:&#xA;    # forecast monthly order quantity at item level with the dummy variable 'Quarter' using AR(2)&#xA;    mape = forecast(dataset, 'M', ['ItemId'], 'Quarter', 2, output_file)&#xA;elif forecast_type == 7:&#xA;    # forecast weekly order quantity at customer level with the dummy variable 'Month' using AR(8)&#xA;    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Month', 8, output_file)&#xA;elif forecast_type == 8:&#xA;    # forecast weekly order quantity at customer level with the dummy variable 'Quarter' using AR(8)&#xA;    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Quarter', 8, output_file)&#xA;elif forecast_type == 9:&#xA;    # forecast monthly order quantity at customer level with the dummy variable 'Quarter' using AR(2)&#xA;    mape = forecast(dataset, 'M', ['UserDefinedData4'], 'Quarter', 2, output_file)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you for help. &lt;/p&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="8752" LastEditDate="2016-09-08T06:51:41.647" LastActivityDate="2016-09-17T23:38:38.877" Title="Autoregressive Moving Average ARMA with statsmodels" Tags="|predictive-modeling|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13906" PostTypeId="1" AcceptedAnswerId="42080" CreationDate="2016-09-08T06:46:30.973" Score="5" ViewCount="4018" Body="&lt;p&gt;TensorFlow's &lt;code&gt;conv2d()&lt;/code&gt; operation lets you choose between &lt;code&gt;&quot;VALID&quot;&lt;/code&gt; (without padding) and &lt;code&gt;&quot;SAME&quot;&lt;/code&gt; (with zero-padding). I suppose all other frameworks let you do the same. I'm trying to understand the pros and cons of zero padding: when would you want to use it, or not to use it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, my understanding is that if the filter size is large relative to the input image size, then without zero padding the output image will be much smaller, and after a few layers you will be left with just a few pixels. So to maintain a reasonably sized output, you need zero-padding + stride 1. Is this the main reason for using zero-padding? Is it preferable to avoid it when you can, for example when the filter size is small relative to the input image size?&lt;/p&gt;&#xA;" OwnerUserId="17907" LastActivityDate="2022-11-14T20:02:17.097" Title="What are the pros and cons of zero padding in a convolution layer?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13907" PostTypeId="1" AcceptedAnswerId="13916" CreationDate="2016-09-08T08:51:08.337" Score="2" ViewCount="1336" Body="&lt;p&gt;I am trying to learn topics distribution for each document in a corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100)  and when I try to infer vectors for each document I am getting uniform distribution over them. This is highly unlikely since documents are of different topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevant code snippet is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#input : scipy sparse term-doc matrix (no_terms * no_docs)&#xA;&#xA;corpus = gensim.matutils.Sparse2Corpus(term_doc)&#xA;&#xA;lda = gensim.models.LdaModel(corpus, 100)&#xA;&#xA;vec_gen = lda[corpus]&#xA;&#xA;vecs = [vec for vec in vec_gen]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now for each vector in vecs I am getting same probability for each topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point out where I am going wrong?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2020-03-17T23:16:52.677" Title="Getting uniform distribution over topics from gensim's LDA?" Tags="|python|text-mining|lda|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13909" PostTypeId="1" AcceptedAnswerId="14003" CreationDate="2016-09-08T20:29:31.337" Score="1" ViewCount="1652" Body="&lt;p&gt;I tried using kind=&quot;box&quot; but am not sure if that works. For the example I tried, I got blank subfigures in the non-diagonal parts of the PairGrid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function I am trying to use is &lt;a href=&quot;https://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.pairplot.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;seaborn.pairplot&lt;/code&gt;&lt;/a&gt;. As mentioned in the link, by default, this function will create a grid of Axes such that each variable in the &lt;code&gt;data&lt;/code&gt; parameter will be shared in the y-axis across a single row and in the x-axis across a single column. The diagonal Axes are treated differently, drawing a plot to show the univariate distribution of the data for the variable in that column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a way to make sure that the &lt;code&gt;kind&lt;/code&gt; parameter leads to a box plot (&lt;a href=&quot;https://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.boxplot.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;seaborn.boxplot&lt;/code&gt;&lt;/a&gt;) for the non-diagonal parts of the PairGrid.&lt;/p&gt;&#xA;" OwnerUserId="24114" LastEditorUserId="24114" LastEditDate="2016-09-13T01:26:47.703" LastActivityDate="2020-08-04T07:40:39.340" Title="Python Seaborn: How can one obtain a box plot (box and whisker plot) for the non-diagonal variable pairs in a pairplot()?" Tags="|python|visualization|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13910" PostTypeId="1" CreationDate="2016-09-08T23:14:59.327" Score="8" ViewCount="670" Body="&lt;p&gt;I was watching some &lt;a href=&quot;https://www.youtube.com/watch?v=jajksuQW4mc&quot; rel=&quot;noreferrer&quot;&gt;videos online&lt;/a&gt; about convolutional networks, and the speaker was discussing the concept of running a filter over an image.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He said, and it is also shown in the image below, that &quot;Depth = Semantic representation&quot;.&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/L8joT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L8joT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sounds nice, but I'm not sure exactly what is so special about reducing the spatial dimension of the image in favor of increased depth? What makes depth more important than any other dimension? A dimension is just dimension, right?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I have a hunch that the reason is going to have something to do with translation independence...&lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="29169" LastEditDate="2021-02-16T15:39:00.070" LastActivityDate="2021-02-16T15:39:00.070" Title="Why does &quot;Depth = Semantic representation&quot; in convolutional neural networks?" Tags="|neural-network|convolutional-neural-network|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13912" PostTypeId="1" AcceptedAnswerId="14470" CreationDate="2016-09-09T05:55:01.660" Score="2" ViewCount="957" Body="&lt;p&gt;While attending to the Coursera's Machine Learning Course, I figured out that I could use a database from the company I work for (~50MM records) to do some linear regression experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But one of the steps involved on proposing this experiment, is to define the technology stack required for this task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my understanding the following tasks should be covered:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Read raw data and store it on an non-production database&lt;/li&gt;&#xA;&lt;li&gt;Transform data to a &quot;regression friendly&quot; format&lt;/li&gt;&#xA;&lt;li&gt;Store the transformed data on an intermediate database&lt;/li&gt;&#xA;&lt;li&gt;Compute the actual regression&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For #1 I can take some paths, like doing a custom .NET or Java program, or even use an ETL process (this is more to copy data to somewhere else and don't mess with production database).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On #2 the funny part begins: should I consider a specialized tool for a &amp;lt;100MM records database? If so, what would you suggest for transforming this data into a matrix-like representation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe #3 is dependant on the #4: I see lots of samples (eg.: in R, or Matlab/Octave) based on text or csv files. Are these the standard formats for these computations? Or should I read from a database&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For #4, from what I could understanding using R is the way to go, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, should I consider a multi-gig multi-processors server, or considering it's an experiment in which spending some hours of computation is not a big issue, a 4GB machine will do the job?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that this question may be considered too broad, but I really would like to hear from you about what should I consider for it, and even if I am missing something (or going to a totally wrong path).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the data, you can consider it like the house pricing in Boston: it's a 30 features (columns) dataset, used to predict the value for one of these columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(question originally posted on Stack Overflow)&lt;/p&gt;&#xA;" OwnerUserId="24192" LastActivityDate="2016-11-11T11:01:43.823" Title="Technology stack for linear regression on (not so) large dataset" Tags="|machine-learning|r|linear-regression|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13913" PostTypeId="2" ParentId="12225" CreationDate="2016-09-09T06:13:38.203" Score="1" Body="&lt;p&gt;You should check the Allen AI competition on kaggle: &lt;a href=&quot;https://www.kaggle.com/c/the-allen-ai-science-challenge&quot; rel=&quot;nofollow&quot;&gt;https://www.kaggle.com/c/the-allen-ai-science-challenge&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, the typical approach people took there was similar to what you're suggesting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;build a knowledge base from domain specific articles and/or Wikipedia&lt;/li&gt;&#xA;&lt;li&gt;index these articles with lucene or other IR system &lt;/li&gt;&#xA;&lt;li&gt;for each question/answer pair retrieve most relevant articles and use them as features&lt;/li&gt;&#xA;&lt;li&gt;build a classifier using these features for classifying  if the answer is correct or incorrect&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-09-09T06:13:38.203" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13914" PostTypeId="2" ParentId="8190" CreationDate="2016-09-09T11:29:16.607" Score="2" Body="&lt;p&gt;I have recently started &lt;a href=&quot;http://contrib.scikit-learn.org/polylearn/&quot; rel=&quot;nofollow&quot;&gt;polylearn&lt;/a&gt;, a scikit-learn-contrib package for efficient implementation of factorization machines and polynomial networks, in Python with Cython. Polylearn currently coordinate descent solvers for 2nd and 3rd order factorization machines, and for arbitrary order polynomial networks, using the approach described in the papers cited on the polylearn website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently working on an implementation of arbitrary higher order FMs using SG and CD style algorithms, using the approach from &lt;a href=&quot;http://arxiv.org/abs/1607.07195&quot; rel=&quot;nofollow&quot;&gt;this recent paper&lt;/a&gt;, I should have it ready within a week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Polylearn has not had a release yet, so I would love to hear any feedback on the API and functionality, in order to round things off before the 0.1 release!&lt;/p&gt;&#xA;" OwnerUserId="24201" LastActivityDate="2016-09-09T11:29:16.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13915" PostTypeId="2" ParentId="9424" CreationDate="2016-09-09T11:53:49.333" Score="1" Body="&lt;p&gt;To get all probabilities instead of all classes instead of just the labeled class, there is no explicit method till now (Spark 2.0) in Spark MLlib or ML. But you can extend the Logistic Regression class from the MLlib source code to get those probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample code snippet can be found in this &lt;a href=&quot;https://stackoverflow.com/a/36238801/3027482&quot;&gt;answer&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="10904" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-09-09T11:53:49.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13916" PostTypeId="2" ParentId="13907" CreationDate="2016-09-09T12:32:37.537" Score="1" Body="&lt;p&gt;I solved this issue. There is a parameter for minimum probability in gensim's LDA which is set to 0.01 by default. So topics with prob. &amp;lt; 0.01 are pruned from output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I set min. prob to a very low value the results had all topics and their corresponding probability.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-09-09T12:32:37.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13917" PostTypeId="2" ParentId="9488" CreationDate="2016-09-09T14:37:07.543" Score="23" Body="&lt;p&gt;On Python you have a nice scikit-learn wrapper, so you can write just like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xgb&#xA;exgb_classifier = xgb.XGBClassifier()&#xA;exgb_classifier.fit(X, y, sample_weight=sample_weights_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More information you can receive from this:&#xA;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier.fit&quot; rel=&quot;noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier.fit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19220" LastEditorUserId="19220" LastEditDate="2017-07-18T12:21:52.893" LastActivityDate="2017-07-18T12:21:52.893" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13918" PostTypeId="2" ParentId="13910" CreationDate="2016-09-09T16:33:23.980" Score="5" Body="&lt;p&gt;The point of the layer depth and gradual pyramidal reduction is to build up a hierarchy of spatially invariant representations, each more complex than those of the prior levels. For example, at the lowest level, a convolutional may be able to pick out noteworthy arrangements of pixels; at the next level, it can condense these into particular spots, basic shapes, edges, etc.; then at higher levels it can recognize increasingly large and complex objects.  I'll borrow an example from &lt;a href=&quot;http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA482820&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gerod M. Bonhoff's thesis&lt;/a&gt;&lt;a href=&quot;http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA482820&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt; on Hawkins' Hierarchical Temporal Memory (HTM), which is a closely related concept, one that also makes use of receptive regions to build invariant representations. At higher levels, the filtering process allows a convolutional or HTM to assemble individual lines and shapes into objects like &quot;dog's tail&quot; or &quot;dog's head&quot;; at the next stage they can be recognized as a &quot;dog&quot; or perhaps a particular variant, like &quot;German shepherd.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is made possible not only by the stacking of multiple layers, but the divisions of neurons within them into separate receptive regions. The receptive regions mimic actual neuronal &quot;cell assemblies&quot; and cortical columns that learn to fire together in groups; this enables clustering around particular types of objects, while the additional layers allow them to be related together into objects of increasing sophistication. The decrease in spatial dimensions in the example you cited reflects the narrowing of the receptive regions as we move up the pyramid; the third dimension (i.e. depth within a layer, as opposed to the depth of the layers) increases in tandem so that we can provide a wider choice of spatially invariant representations to select from at each stage i.e., each filter in the output volume depth-dimension learns to look at something different. If we simply narrowed the pyramid at each stage along every dimension, eventually we'd be left with just a narrow range of objects to choose from; taken far enough, it might just leave us with a single node at the top reflecting a single yes-no choice between &quot;is this a dog or not?&quot; This more flexible design allows us to choose more combinations of the previous layer's spatially invariant representations. I believe that this also enables a convolutional net to take into account various orientation problems, including translation independence, by adding more cell assemblies/columns to deal with each reorientation of an invariant representation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this excellent tutorial at github&lt;/a&gt; explains, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;First, the depth of the output volume is a hyperparameter: it corresponds to the number of filters we would like to use, each&#xA;  learning to look for something different in the input. For example, if&#xA;  the first Convolutional Layer takes as input the raw image, then&#xA;  different neurons along the depth dimension may activate in presence&#xA;  of various oriented edged, or blobs of color. We will refer to a set&#xA;  of neurons that are all looking at the same region of the input as a&#xA;  depth column (some people also prefer the term fibre).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This type of design is inspired by various biologically plausible structures found in actual organisms, such as the eyes of cats. If what I've said here isn't clear enough to answer your question, I can add a lot of further detail, including more examples, some based on actual organs of that kind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA482820&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;See pp. 26-27, 36 76 Bonhoff, Gerod M., Using Hierarchical Temporal Memory for Detecting Anomalous Network Activity. Thesis delivered March, 2008 to the faculty of the Air Force Institute of Technology at Wright-Patterson Air Force Base, Ohio.  &lt;/p&gt;&#xA;" OwnerUserId="18544" LastEditorUserId="-1" LastEditDate="2017-11-03T10:47:00.867" LastActivityDate="2017-11-03T10:47:00.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13919" PostTypeId="1" CreationDate="2016-09-09T16:47:54.007" Score="1" ViewCount="436" Body="&lt;p&gt;Reading the &lt;a href=&quot;https://arxiv.org/pdf/1603.05027v3.pdf&quot; rel=&quot;nofollow&quot;&gt;ResNet paper&lt;/a&gt;, paragraph 3.3:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The convolutional layers mostly have 3×3 filters and follow two simple&#xA;  design rules: (i) for the same output feature map size, the layers&#xA;  have the same number of filters; and (ii) if the feature map size is&#xA;  halved, the number of filters is doubled so as to preserve the time&#xA;  complexity per layer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The network uses convolutional layers with stride 2 to halve the feature map size, but this divides both the width and the height of the feature maps, so the total area is divided by 4, not by 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am guessing the time complexity per layer is halved every time the feature maps are halved. Did I miss something?&lt;/p&gt;&#xA;" OwnerUserId="17907" LastEditorUserId="29169" LastEditDate="2021-02-16T15:38:50.467" LastActivityDate="2021-02-16T15:38:50.467" Title="Do all layers have the same computational complexity in a ResNet?" Tags="|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13920" PostTypeId="1" CreationDate="2016-09-09T18:00:08.380" Score="2" ViewCount="5320" Body="&lt;p&gt;I am using Keras to classify images. I am following the &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow&quot;&gt;Keras blog&lt;/a&gt;. The accuracy from  predict_generator is not matching with the accuracy obtained from the confusion matrix, which I am computing using the scikit-learn package. I have included the relevant snippet of the code below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras.models import Sequential&#xA;from keras.layers import Convolution2D, MaxPooling2D&#xA;from keras.layers import Activation, Dropout, Flatten, Dense&#xA;import numpy as np&#xA;import theano&#xA;from sklearn.metrics import classification_report, confusion_matrix&#xA;y_actual = np.ones((nb_test_samples),dtype = int)&#xA;y_actual[0:2817] = 0&#xA;&#xA;train_datagen = ImageDataGenerator(&#xA;             featurewise_std_normalization=False,&#xA;             samplewise_std_normalization=False,&#xA;     rescale = 1./255)&#xA;&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;&#xA;train_generator = train_datagen.flow_from_directory(&#xA;                    train_data_dir,&#xA;                    target_size = (img_width,img_height),&#xA;                    batch_size = 32,&#xA;                    class_mode = 'binary')&#xA;&#xA;test_generator = test_datagen.flow_from_directory(&#xA;            test_data_dir,&#xA;                    target_size = (img_width,img_height),&#xA;                    batch_size = 32,&#xA;            class_mode = 'binary',&#xA;            shuffle = False         )&#xA;&#xA;model.fit_generator(&#xA;                    train_generator,&#xA;                    samples_per_epoch = nb_train_samples,&#xA;                    nb_epoch = nb_epoch,&#xA;                    validation_data = test_generator,&#xA;                    nb_val_samples = nb_test_samples)&#xA;&#xA;score =   model.evaluate_generator(&#xA;                      test_generator,&#xA;                      4938)&#xA;print &quot;Test fraction correct (Accuracy) = {:.2f}&quot;.format(score[1])&#xA;prediction = model.predict_generator(test_generator,nb_test_samples)&#xA;&#xA;for i in xrange(0,len(prediction)):&#xA;    if prediction[i]&amp;lt;0.5:&#xA;       prediction[i] = 0&#xA;    else:&#xA;       prediction[i] = 1&#xA;&#xA;#y_predicted = test_generator.classes&#xA;print np.sum(prediction)&#xA;CM = confusion_matrix(y_actual,prediction)&#xA;print CM &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I use y_predicted, I get a perfectly diagonal confusion matrix, when the console output shows an accuracy of 70% which doesn't make any sense at all. What is that I am doing wrong? &lt;/p&gt;&#xA;" OwnerUserId="14360" LastActivityDate="2022-06-10T05:34:54.150" Title="Accuracy doesn't match in Keras" Tags="|deep-learning|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13923" PostTypeId="1" AcceptedAnswerId="13924" CreationDate="2016-09-09T19:32:42.193" Score="12" ViewCount="7552" Body="&lt;p&gt;I am trying to do sentiment analysis. In order to convert the words to word vectors, I am using Word2Vec model. Suppose I have all the sentences in a list named 'sentences' and I am passing these sentences to word2vec as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = word2vec.Word2Vec(sentences, workers=4 , min_count=40, size=300,   window=5, sample=1e-3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Since I am noob to word vectors, I have two doubts:&lt;/p&gt;&#xA;&lt;p&gt;1- Setting the number of features to 300 defines the features of a word vector. But what these features signify? If each word in this model is represented by a 1x300 numpy array, then what do these 300 features signify for that word?&lt;/p&gt;&#xA;&lt;p&gt;2- What does downsampling as represented by 'sample' parameter in the above model do in actual?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-24T21:25:09.273" LastActivityDate="2020-12-25T15:14:06.083" Title="Features of word vectors in Word2Vec" Tags="|machine-learning|word-embeddings|word2vec|nlp|sentiment-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="13924" PostTypeId="2" ParentId="13923" CreationDate="2016-09-10T09:24:18.460" Score="10" Body="&lt;p&gt;1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These features are learnt by the neural network as this is unsupervised method. Each vector has several set of semantic characteristics.&#xA;For instance, let's take the classical example, &lt;code&gt;V(King) -V(man) + V(Women) ~ V(Queen)&lt;/code&gt; and each word represented by 300-d vector. &lt;code&gt;V(King)&lt;/code&gt; will have semantic characteristics of Royality, kingdom, masculinity, human in the vector in a certain order. &lt;code&gt;V(man)&lt;/code&gt; will have masculinity, human, work in a certain order. Thus when &lt;code&gt;V(King)-V(Man)&lt;/code&gt; is done, masculinity,human characteristics will get nullified and when added with &lt;code&gt;V(Women)&lt;/code&gt; which having femininity, human characteristics will be added thus resulting in a vector much similar to &lt;code&gt;V(Queen)&lt;/code&gt;. The interesting thing is, these characteristics are encoded in the vector in a certain order so that numerical computations such as addition, subtraction works perfectly. This is due to the nature of unsupervised learning method in neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- There are two approximation algorithms. &lt;code&gt;Hierarchical softmax&lt;/code&gt; and &lt;code&gt;negative sampling&lt;/code&gt;. When the sample parameter is given, it takes negative sampling. In case of hierarchical softmax, for each word vector its context words are given positive outputs and all other words in vocabulary are given negative outputs. The issue of time complexity is resolved by negative sampling. As in negative sampling, rather than the whole vocabulary, only a sampled part of vocabulary is given negative outputs and the vectors are trained which is so much faster than former method.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2017-07-10T07:44:14.780" LastActivityDate="2017-07-10T07:44:14.780" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="13925" PostTypeId="1" CreationDate="2016-09-10T11:41:32.493" Score="1" ViewCount="1126" Body="&lt;pre&gt;&lt;code&gt;install.packages('RTextTools')&#xA;library(RTextTools)&#xA;&#xA;data(&quot;NYTimes&quot;, package='RTextTools')&#xA;head(NYTimes)&#xA;&#xA;timesMat &amp;lt;- create_matrix(NYTimes$Title)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The package is installed and I can view the &quot;NYTimes&quot; data set but the &lt;code&gt;create_matrix&lt;/code&gt; call throws an error that it can't find the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more thing: call to &lt;code&gt;library(RTextTools)&lt;/code&gt; gives the following error messages:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck =&#xA;  vI[[i]]) :    there is no package called ‘slam’ Error: package or&#xA;  namespace load failed for ‘RTextTools’&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any help appreciated...&lt;/p&gt;&#xA;" OwnerUserId="24222" LastEditorUserId="2961" LastEditDate="2016-09-10T16:44:45.563" LastActivityDate="2016-09-10T16:44:45.563" Title="Package RTextTools loaded but facing Error: could not find function &quot;create_matrix&quot;" Tags="|r|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13926" PostTypeId="2" ParentId="9819" CreationDate="2016-09-10T18:03:35.480" Score="7" Body="&lt;p&gt;Increasing the number of epochs usually benefits the quality of the word representations. In experiments I have performed where the goal was to use the word embeddings as features for text classification setting the epochs to 15 instead of 5, increased the performance.  &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2016-09-10T18:03:35.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13930" PostTypeId="2" ParentId="13912" CreationDate="2016-09-11T08:24:06.873" Score="1" Body="&lt;p&gt;It's easier if your entire dataset fits into RAM. &#xA;So check how large it is in GB and get enough RAM. Multiprocessor probably won't help much. So try to get highest frequency proc with just a few cores. &#xA;You could extract your data into a csv and use R or Scikit-learn for modelling. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2016-09-11T08:24:06.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13933" PostTypeId="1" CreationDate="2016-09-11T17:38:19.650" Score="3" ViewCount="2645" Body="&lt;p&gt;I have a data set which has around 1000 samples and are divided in 4 groups - A, B ,C , D. The problem I am facing is that there are very high number of data sample which have B and C s output. They outweigh other two by factor of 3:1. Due to this most of the classification problems are producing very inefficient results with results from other two classes being mapped to A and B quite often . Is there any way to deal with it?&lt;/p&gt;&#xA;" OwnerUserId="24235" LastEditorUserId="13727" LastEditDate="2016-09-13T13:40:09.977" LastActivityDate="2016-11-12T14:24:19.023" Title="How to deal with classification problem where labels are non uniformly distributed?" Tags="|classification|multiclass-classification|class-imbalance|multilabel-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13934" PostTypeId="1" AcceptedAnswerId="13935" CreationDate="2016-09-11T19:35:19.467" Score="0" ViewCount="601" Body="&lt;p&gt;I have training data in form of pair of documents with an associated label - {doc1, doc2, label}. Label is defined as function of pair of documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to build a model which can predict the label given two new documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to try different representation of document (instead of common ones say TF-IDF). Can I use vectors (topic distribution) from LDA as features for a classifier? &lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-09-11T20:57:40.910" Title="Classifier on top of LDA topic vectors?" Tags="|text-mining|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13935" PostTypeId="2" ParentId="13934" CreationDate="2016-09-11T20:57:40.910" Score="0" Body="&lt;p&gt;Yes, that is a reasonable approach. Also try neural network based representations such as doc2vec. I suppose you know how to do the classification part?&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-11T20:57:40.910" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="13936" PostTypeId="2" ParentId="13933" CreationDate="2016-09-11T21:11:53.283" Score="2" Body="&lt;p&gt;Welcome to DataScience.SE! You need to ensure that your training distribution is similar to your test distribution in order to get the best results. This can be done through &lt;a href=&quot;https://en.wikipedia.org/wiki/Stratified_sampling&quot; rel=&quot;nofollow&quot;&gt;stratified sampling&lt;/a&gt;. It might be that there is no problem, and that the classifier is simply allocating its resources to best classify the majority case, though I'm a bit confused by your description because you say that B and C dominate and the &quot;other two classes&quot; (A and D) are &quot;mapped to A and B quite often&quot;. One option is to use more discriminating features, if you can think of any. Another is to partition the large classes into (B becomes the union of $B_i$ and likewise for C) if applicable. This can help if the shape of the class is complex by breaking it up into learnable pieces. It helps to plot your inputs, colored by class, to understand which classes overlap or have complex shapes. Finally, you can modify your loss function if getting certain classes wrong is more important than others. &lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-11T21:11:53.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13937" PostTypeId="2" ParentId="371" CreationDate="2016-09-11T21:24:35.287" Score="1" Body="&lt;p&gt;Turning simple questions into answers is not &lt;em&gt;simple&lt;/em&gt; whatsoever.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first technology to do this as broadly across technology and accurately will be a big winner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are some out there, filling in the gaps with &quot;answering questions&quot; with Artificial Intelligence (e.g. IBM Watson, and Amazon Alexa). This requires solving the language complexities related to the data in question, what's in the data stores, and what are nouns, verbs and pronouns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Microsoft ventured here with &lt;a href=&quot;https://dba.stackexchange.com/questions/12681/why-did-microsoft-kill-english-query-on-sql-server/12795&quot;&gt;English Query&lt;/a&gt; but, then stopped. &lt;a href=&quot;http://kueri.me/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kueri.me&lt;/a&gt; is a Python based platform doing about the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Structured Query Languages (SQL) and the like, SOQL, MDX, Hive, Impala and the newer takes on old fashioned SQL. Have not yet replaced much of anything, all these pieces are small fixes to the grander &quot;End Goal&quot; and that lies in Artificial Intelligence (AI), specifically, Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question being:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Can the computer, figure out what you want.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not yet. It takes Linguists, Mathematicians, Engineers and more to all contribute their piece of the pie so we can enjoy some of that sweet Artificial Intelligent and Machine Learned cake.&lt;/p&gt;&#xA;" OwnerUserId="23800" LastEditorUserId="-1" LastEditDate="2017-04-13T12:43:06.617" LastActivityDate="2016-09-11T21:24:35.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13941" PostTypeId="2" ParentId="4992" CreationDate="2016-09-12T03:51:16.957" Score="0" Body="&lt;p&gt;I would have just commented but I can't. I've been looking for a practical treatment of amplification in LSH and what you've presented makes a lot of sense. From what I gather, the primary hash function is $$h(x,v)=\left\{ \begin{array}{ll}0 &amp;amp; \mbox{if }\textrm{sgn}(x\cdot v) &amp;lt; 0\\ 1 &amp;amp; \mbox{else}\end{array} \right.$$ for some random vector $v$, after the AND this becomes $h'(x,i) = (h(x,v_{i+1}),...,h(x,v_{i+r}))$, and finally after the OR, $h''(x,j) =f(h'(x,rj),j)$ or $$h''(x,y) =\left\{ \begin{array}{ll}1 &amp;amp; \mbox{if }h''(x,j) = h''(y,j) \mbox{ for any }j \in [0,b)\\ 0 &amp;amp; \mbox{else}\end{array} \right.$$ Now you can AND/OR using $h''(x,y)$ as you describe. You would then just be choosing candidates based on AND/OR logical statements; you're not really hashing anymore. At this point to continue hashing, you'd need a mapping $\hat{h}:S\to S'$ of the bins such that each vector only appears once in $S'$, but doing so will also likely introduce false positives and/or negatives.  One idea for a hash is the minimum of $h''(x,j)$  for all $j$ (or the minimum across all $j$ and all directly and indirectly associated $y$). Both clearly would introduce bias. I might try one of these, though I'm not sure the hashes from one random AND/OR will be meaningful the next time around. But considering a uniform distribution of random $v$ and large number of replications, maybe?&lt;/p&gt;&#xA;" OwnerUserId="24251" LastActivityDate="2016-09-12T03:51:16.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13942" PostTypeId="1" CreationDate="2016-09-12T04:52:53.607" Score="2" ViewCount="333" Body="&lt;p&gt;I know there are questions asking for &quot;best neural network library in Python&quot; but I haven't found any (besides the old deprecated ones like PyBrain) which allow algorithms like NEAT and MultiNEAT and simple non-gradient/differentiation optimization algorithms like PSO to be applied. I'm working on a project involving NEAT in PyBrain but PyBrain is old and as far as I can tell very very slow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any more recent/in-progress projects in Python which allow custom neural network topologies (for example, Tensorflow and Keras seem to only allow fully-connected recurrent networks, I can't think of a way to, for example, modify those networks so they match up with a network generated by NEAT). &lt;/p&gt;&#xA;" OwnerUserId="24253" LastActivityDate="2016-09-12T04:52:53.607" Title="Python Neural Network Library - With Dynamic Topologies" Tags="|python|neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13944" PostTypeId="2" ParentId="13933" CreationDate="2016-09-12T07:01:13.627" Score="2" Body="&lt;p&gt;You can try merging the groups A and D which will provide more significant boundaries to the model. Apart from this you can try to sample the given data to form a new data so as to form a uniform distribution of all the groups.&lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2016-09-12T07:01:13.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13945" PostTypeId="2" ParentId="13912" CreationDate="2016-09-12T07:12:51.787" Score="0" Body="&lt;p&gt;You can use R or Python for 100 million records with the conventional regression libraries. You will require around 16GB of RAM according to my experience, may be more than that!! A quadcore processor will be fine while running algorithms and during pre-processing steps. It would be better to store the transformed data to an immediate database.  &lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2016-09-12T07:12:51.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13948" PostTypeId="1" CreationDate="2016-09-12T12:23:36.417" Score="3" ViewCount="180" Body="&lt;p&gt;I try to evaluate several NA imputation methods with supervised approach: I clone my original data frame with no NAs, artifically insert NAs into the resulting Data Frame and apply imputations to the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'd like to evaluate the imputations by comparing the imputed new DFs with the original one. I wonder what would be the best metod; is there any distance method for instance that I could apply to the original/imputed DF pairs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(My DF contains only numeric data, but a solution that handles factor variables as well would be especially handy).&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-02-05T02:44:11.470" Title="Check Imputation Efficiency - How To Compare Data Frames?" Tags="|data-imputation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13949" PostTypeId="1" CreationDate="2016-09-12T13:01:16.533" Score="3" ViewCount="2318" Body="&lt;p&gt;I am using &lt;a href=&quot;https://github.com/yoonkim/CNN_sentence&quot; rel=&quot;nofollow&quot;&gt;CNN for Sentence Classification&lt;/a&gt; code by Yoonkim. This is used for text classification. I noticed that he uses softmax layer and negative log likelihood error. This is optimal for single label multiclass classification. I now have a dataset which is for multilabel-multiclass classification. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are the ideal activation methods and error functions to use? &lt;/li&gt;&#xA;&lt;li&gt;I felt that, using softmax layer will give the sum of probabilities as 1 but in this case since it belongs to different labels, at each instance we might have high probability for each class. So I believe that Softmax is not an ideal option. Am I right?&lt;/li&gt;&#xA;&lt;li&gt;I find data for images but nothing prominent for text. Can anyone refer me to some source?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-09-12T13:28:05.117" Title="Activation method and Loss function for multilabel multiclass classification" Tags="|nlp|deep-learning|multiclass-classification|multilabel-classification|loss-function|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13950" PostTypeId="2" ParentId="13949" CreationDate="2016-09-12T13:28:05.117" Score="2" Body="&lt;p&gt;Assuming all the labels have the same importance you can have a sigmoid for every class at the output. For each of the classes it will ask, is this class part of this sentence or not? The loss is just the sum of the individual log losses for the outputs. If some labels are more important you can scale them accordingly in your loss function.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-12T13:28:05.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13951" PostTypeId="2" ParentId="13948" CreationDate="2016-09-12T13:40:39.287" Score="0" Body="&lt;p&gt;It's not necessarily a distance but what you could do is the following schema, for one column of potential NAs. Let's call that one $x_n$, the rest are knowns in any case. We will have three options, keep the full original (best case scenario), impute the artificially added NAs in multiple ways (realistic scenario) and drop the whole column (pessimistic scenario). Now we know how well the best case scenario works and what happens if we just drop the whole column. We just measure what we want to optimize, MSE for example. Our imputation methods will normally have a performance between best case and worst case. Then you could quantify your imputation method by putting it on a line between worst and best case.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-12T13:40:39.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13952" PostTypeId="2" ParentId="13892" CreationDate="2016-09-12T13:52:47.137" Score="2" Body="&lt;p&gt;Try this R code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(igraph)&#xA;dfr &amp;lt;- data.frame(idMovie1=c(2,1,4), idMovie2=c(3, 2, 3), strength=c(23,80,10))&#xA;&#xA;igr &amp;lt;- igraph::graph.data.frame(dfr)&#xA;plot(x = igr,&#xA;     edge.curved=FALSE, edge.width=log(edge_attr(igr)$strength), edge.label=edge_attr(igr)$strength,&#xA;     main=&quot;Graph of Movie Strengths&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Works only for small daasets. visualizations get ugly quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TjExc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TjExc.png&quot; alt=&quot;igraph&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-09-12T13:52:47.137" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13957" PostTypeId="2" ParentId="5345" CreationDate="2016-09-12T20:11:43.537" Score="12" Body="&lt;p&gt;You may try using R with Jupyter notebook. It requires installation of jupyter R kernel, IRkernel which will allow you to open a new jupyter notebook with option to choose &lt;code&gt;R&lt;/code&gt; instead of default &lt;code&gt;python&lt;/code&gt; kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://www.continuum.io/blog/developer/jupyter-and-conda-r&quot; rel=&quot;noreferrer&quot;&gt;https://www.continuum.io/blog/developer/jupyter-and-conda-r&lt;/a&gt; and &lt;a href=&quot;https://irkernel.github.io/installation/&quot; rel=&quot;noreferrer&quot;&gt;https://irkernel.github.io/installation/&lt;/a&gt; for installation steps.&lt;/p&gt;&#xA;" OwnerUserId="24275" LastActivityDate="2016-09-12T20:11:43.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13959" PostTypeId="2" ParentId="8681" CreationDate="2016-09-12T23:45:08.363" Score="11" Body="&lt;p&gt;Taking a stab:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I wasn't able to find an implementation of Gower Distance in Python when I searched for it about 4-5 months back. So I came up with my own implementation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.neighbors import DistanceMetric&#xA;&#xA;&#xA;def gower_distance(X):&#xA;    &quot;&quot;&quot;&#xA;    This function expects a pandas dataframe as input&#xA;    The data frame is to contain the features along the columns. Based on these features a&#xA;    distance matrix will be returned which will contain the pairwise gower distance between the rows&#xA;    All variables of object type will be treated as nominal variables and the others will be treated as &#xA;    numeric variables.&#xA;    Distance metrics used for:&#xA;    Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)&#xA;    Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)&#xA;    &quot;&quot;&quot;&#xA;    individual_variable_distances = []&#xA;&#xA;    for i in range(X.shape[1]):&#xA;        feature = X.iloc[:,[i]]&#xA;        if feature.dtypes[0] == np.object:&#xA;            feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))&#xA;        else:&#xA;            feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / np.ptp(feature.values)&#xA;&#xA;        individual_variable_distances.append(feature_dist)&#xA;&#xA;    return np.array(individual_variable_distances).mean(0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The link to the same piece of code: &lt;a href=&quot;https://github.com/matchado/Misc/blob/master/gower_dist.py&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/matchado/Misc/blob/master/gower_dist.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to the clustering technique, I haven't used the ones you've mentioned. But I've used hierarchical clustering in R along with gower distance with success in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking into the clustering techniques available in scikit learn, Agglomerative Clustering seems to fit the bill. &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Once you've assigned cluster labels to each row of your data, for each cluster look into the distribution of the features (summary stats for continuous variables &amp;amp; frequency distributions for categorical variables). This is easier to analyze visually if your number of features are manageable (&amp;lt;20 maybe?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But since you have 100+ features, I suggest a more organized approach. Create a matrix with cluster labels in the columns and the summary stat of the features in the rows (I suggest using median for continuous variable and percentage occurrence of most frequent value in cluster for categorical variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might look something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;╔═══════════════════════╦═══════════╦═══════════╦════╦═══════════╗&#xA;║        Feature        ║ Cluster 1 ║ Cluster 2 ║ …  ║ Cluster N ║&#xA;╠═══════════════════════╬═══════════╬═══════════╬════╬═══════════╣&#xA;║ Numeric feature 1     ║ 15        ║ 37        ║ .. ║ 1         ║&#xA;║ Numeric feature 2     ║ 34        ║ 56        ║ …  ║ 56        ║&#xA;║ Categorical feature 1 ║ 47%       ║ 87%       ║ …  ║ 25%       ║&#xA;║ …                     ║ …         ║ …         ║ …  ║ …         ║&#xA;║ Categorical feature N ║ 25%       ║ 91%       ║ …  ║ 11%       ║&#xA;║ Numeric feature N     ║ 0.2       ║ 0.7       ║ …  ║ 0.5       ║&#xA;╚═══════════════════════╩═══════════╩═══════════╩════╩═══════════╝&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20723" LastActivityDate="2016-09-12T23:45:08.363" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13960" PostTypeId="1" AcceptedAnswerId="14001" CreationDate="2016-09-13T03:23:24.483" Score="15" ViewCount="14351" Body="&lt;p&gt;When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastEditorUserId="24000" LastEditDate="2016-09-13T13:40:02.500" LastActivityDate="2016-10-24T14:47:16.297" Title="How to choose a classifier after cross-validation?" Tags="|machine-learning|cross-validation|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13962" PostTypeId="2" ParentId="13960" CreationDate="2016-09-13T07:15:08.387" Score="1" Body="&lt;p&gt;So let us assume you have training out of which you are using 80% as training and rest 20% as validation data. We can train on the 80% and test on the remaining 20% but it is possible that the 20% we took is not in resemblance with the actual testing data and might perform bad latter. So, in order to prevent this we can use k-fold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let us say you have different models and want to know which performs better with your dataset, k-fold cross validation works great. You can know the validation errors on the k-validation performances and choose the better model based on that. This is generally the purpose for k-fold cross validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to just one model and if you are checking with k-fold cross-validation, you can get an approximate of errors of test data, but &lt;code&gt;when you are actually training it finally, you can use the complete training data&lt;/code&gt;.(Because it is assumed here that the whole data will together perform better than a part of it.It might not be the case sometimes, but this is the general assumption.)&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-09-13T07:15:08.387" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="13964" PostTypeId="2" ParentId="13960" CreationDate="2016-09-13T07:42:04.157" Score="8" Body="&lt;p&gt;No. You don't select any of the k classifiers built during k-fold cross-validation. First of all, the purpose of cross-validation is not to come up with a predictive model, but to evaluate how accurately a predictive model will perform in practice. Second of all, for the sake of argument, let's say you were to use k-fold cross-validation with k=10 to find out which one of three different classification algorithms would be the most suitable in solving a given classification problem. In that case, the data is randomly split into k parts of equal size. One of the parts is reserved for testing and the rest k-1 parts will be used for training. The cross-validation process is repeated k (fold) times so that on every iteration different part is used for testing. After running the cross-validation you look at the results from each fold and wonder which classification algorithm (not any of the trained models!) is the most suitable. You don't want to choose the algorithm that has the highest test accuracy on one of the 10 iterations, because maybe it just happened randomly that the test data on that particular iteration contained very easy examples, which then lead to high test accuracy. What you want to do, is to choose the algorithm which produced the best accuracy &lt;strong&gt;averaged over all k folds&lt;/strong&gt;. Now that you have chosen the algorithm, you can train it using your whole training data and start making predictions in the wild.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is beyond the scope of this question, but you should also optimize model's hyperparameters (if any) to get the most out of the selected algorithm. People usually perform hyperparameter optimization using cross-validation. &lt;/p&gt;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2016-09-13T20:47:48.513" LastActivityDate="2016-09-13T20:47:48.513" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="13965" PostTypeId="1" CreationDate="2016-09-13T08:23:18.290" Score="2" ViewCount="96" Body="&lt;p&gt;I am curious to find out what is the algorithm behind Google's ActivityRecognitionAPI on the Android platform to find out what is the current state of the user (e.g., i the user us walking, running or laying) and how it works in a nutshell? Thank you in advance! &lt;/p&gt;&#xA;" OwnerUserId="24288" LastActivityDate="2016-09-16T18:38:32.300" Title="What algorithgm Android's ActivityRecognitionAPI uses?" Tags="|machine-learning|data-mining|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13966" PostTypeId="1" AcceptedAnswerId="23921" CreationDate="2016-09-13T09:09:11.553" Score="2" ViewCount="2680" Body="&lt;p&gt;I am currently discretizing my training set in R with discretize from the bnlearn package.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(bnlearn) &#xA;discretize(train, method = &quot;quantile&quot;, breaks = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The lower bound of one interval and the upper bound of the other interval are the minimum and the maximum of the respective column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Executing the same command on the test set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;discretize(test, method = &quot;quantile&quot;, breaks = 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will result in a different discretization, as the minimum and the maximum will likely be different on the test set. Is it possible in R (with another library or command) to transfer a discretization from a training set to a test set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparantly this is easy to do in Weka and Orange, however, I would prefer to do this in R (not using RWeka).&lt;/p&gt;&#xA;" OwnerUserId="24290" LastActivityDate="2017-10-20T07:24:18.797" Title="Using Discretization from Training Set on Test Set in R" Tags="|machine-learning|r|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13967" PostTypeId="1" CreationDate="2016-09-13T10:21:50.330" Score="1" ViewCount="1240" Body="&lt;p&gt;Other than just filling in with the mean of a feature, what other methods are there which can work well? I am trying to decide whether or not to use a denoising-autoencoder or just impute with the mean or any other method which can perform well enough which is quickly implementable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also is it necessary to make any assumptions about the underlying distribution? The denosing-autoencoder seems attractive here because you do not need to make any explicit assumptions.&lt;/p&gt;&#xA;" OwnerUserId="24050" LastEditorUserId="29169" LastEditDate="2021-01-14T18:55:27.890" LastActivityDate="2021-01-14T18:55:27.890" Title="What would be the best way to impute data?" Tags="|autoencoder|data-imputation|theory|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13968" PostTypeId="1" AcceptedAnswerId="13969" CreationDate="2016-09-13T13:15:01.727" Score="2" ViewCount="586" Body="&lt;p&gt;I'm using Spark's Machine Learning Library, and features are categorical. The features are strings, and Spark's MLlib (like many other machine learning libraries) does not accept Strings as inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normal procedure for overcoming this is to convert Strings to integers, and then encode these integers (using a onehotencoder for example), because converting to integers implies that there is an ordering between features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is - do categorical features always need to be encoded? In what situation could integers be used instead of encoding?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Logistic Regression and Naive Bayes. &#xA;When using integers as features I get an 84% accuracy, when these integers are encoded I get an 82% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it necessary to encode?&lt;/p&gt;&#xA;" OwnerUserId="17772" LastActivityDate="2016-09-14T04:58:29.943" Title="Do categorical features always need to be encoded?" Tags="|predictive-modeling|apache-spark|categorical-data|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13969" PostTypeId="2" ParentId="13968" CreationDate="2016-09-13T13:26:30.757" Score="6" Body="&lt;p&gt;You have partly answered this question yourself (&quot;because converting to integers implies that there is an ordering between features&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will just clarify the terminology a bit more.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat)&lt;/li&gt;&#xA;&lt;li&gt;Ordinal data: information has categories with natural ordering defined between them (annual income scale defined in terms of categories \$40000,   \$40000 - $80000).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the variable is of type ordinal, you can replace it with integers and proceed with the algorithm. If it is categorical, it should be converted as well as encoded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastEditorUserId="23204" LastEditDate="2016-09-14T04:58:29.943" LastActivityDate="2016-09-14T04:58:29.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13970" PostTypeId="1" AcceptedAnswerId="13974" CreationDate="2016-09-13T14:10:49.220" Score="8" ViewCount="1885" Body="&lt;p&gt;while taking an online course on machine learning by Andrew Ng on coursera, I came across a topic called &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;overfitting&lt;/a&gt;. I know it can occur when gradient descent is used in linear or logistic regression but can it occur when Advanced Optimization algorithms such as &quot;Conjugate gradient&quot;, &quot;BFGS&quot;, and &quot;L-BFGS&quot; are used ?&lt;/p&gt;&#xA;" OwnerUserId="24077" LastEditorUserId="20495" LastEditDate="2016-09-14T09:57:03.333" LastActivityDate="2016-09-14T09:57:03.333" Title="Can overfitting occur in Advanced Optimization algorithms?" Tags="|machine-learning|regression|optimization|gradient-descent|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13971" PostTypeId="1" AcceptedAnswerId="13972" CreationDate="2016-09-13T14:15:15.360" Score="1" ViewCount="8108" Body="&lt;p&gt;I understand that one should standardize and normalize the test data (or any &quot;unlabeled&quot; data) with the training mean and sd. How can I implement this in R language? Is there a kind of &quot;fitting&quot; to the training set and a kind of applying to the test data?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2021-04-28T13:21:39.200" Title="Standardization/Normalization test data in R" Tags="|r|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13972" PostTypeId="2" ParentId="13971" CreationDate="2016-09-13T14:25:52.193" Score="2" Body="&lt;p&gt;Check out the &lt;code&gt;preProcess&lt;/code&gt; function from the &lt;code&gt;caret&lt;/code&gt; library.  You can choose the parameters you want to scale/center the training data, and it also saves the transformations it makes so then you can normalize the test set with the same specifications that you normalized the training set with.  Could go something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;library(caret)&#xA;trainData &amp;lt;- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))&#xA;testData &amp;lt;- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))&#xA;normParam &amp;lt;- preProcess(trainData)&#xA;norm.testData &amp;lt;- predict(normParam, testData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;now your &lt;code&gt;norm.testData&lt;/code&gt; is scaled and centered according to the training data set parameters.&lt;/p&gt;&#xA;&lt;p&gt;Another way to do this without using caret:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;## set up data&#xA;trainData &amp;lt;- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))&#xA;testData &amp;lt;- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))&#xA;&#xA;## find mean and sd column-wise of training data&#xA;trainMean &amp;lt;- apply(trainData,2,mean)&#xA;trainSd &amp;lt;- apply(trainData,2,sd)&#xA;&#xA;## centered&#xA;sweep(trainData, 2L, trainMean) # using the default &amp;quot;-&amp;quot; to subtract mean column-wise   &#xA;## centered AND scaled&#xA;norm2.testData &amp;lt;- sweep(sweep(testData, 2L, trainMean), 2, trainSd, &amp;quot;/&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14913" LastEditorUserId="-1" LastEditDate="2021-04-28T13:21:39.200" LastActivityDate="2021-04-28T13:21:39.200" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="13973" PostTypeId="2" ParentId="13970" CreationDate="2016-09-13T15:16:55.497" Score="4" Body="&lt;p&gt;Overfitting is generally a result of the data and structure of your model.  The 'advanced' algorithms you mention have specific uses that may or may not out perform other methods depending on your objectives and your data.  Here is a source for some further reading:&#xA;&lt;a href=&quot;http://papers.nips.cc/paper/1895-overfitting-in-neural-nets-backpropagation-conjugate-gradient-and-early-stopping.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://papers.nips.cc/paper/1895-overfitting-in-neural-nets-backpropagation-conjugate-gradient-and-early-stopping.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-09-13T15:16:55.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13974" PostTypeId="2" ParentId="13970" CreationDate="2016-09-13T15:53:14.123" Score="11" Body="&lt;p&gt;There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, and that minimum shouldn't change regardless of the flavor of gradient descent that you're using (unless you're using regularization), so all of the methods you've listed would overfit (or underfit) equally. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moving from linear models to more complex models, like deep learning, you're even more at risk of seeing overfitting. I've had plenty of convoluted neural networks that badly overfit, even though convolution is supposed to reduce the chance of overfitting substantially by sharing weights. In summary, there is no silver bullet for overfitting, regardless of model family or optimization technique.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-09-13T15:53:14.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13975" PostTypeId="2" ParentId="13967" CreationDate="2016-09-13T15:59:51.477" Score="2" Body="&lt;p&gt;You can treat the missing feature as the target variable of a sub-problem and create a classifier (e.g., a linear model, SVM, etc) for it. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-09-13T15:59:51.477" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13976" PostTypeId="1" AcceptedAnswerId="13980" CreationDate="2016-09-13T16:47:07.230" Score="1" ViewCount="87" Body="&lt;p&gt;What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it? &#xA;How to use the schema for such transformation, how to deal with different data types like facts and dimensions for instance. What is the criteria for data separation into machines, indexes or unique keys?&lt;/p&gt;&#xA;" OwnerUserId="16251" LastEditorUserId="16251" LastEditDate="2016-09-13T21:26:42.467" LastActivityDate="2016-09-13T21:26:42.467" Title="Transformation from Datawarehouse into Big Data structure" Tags="|bigdata|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13977" PostTypeId="1" CreationDate="2016-09-13T17:37:39.583" Score="11" ViewCount="2527" Body="&lt;p&gt;When working with a neural network with more than one output, what is generally advised as the best strategy for early-stopping the training process? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that I am currently monitoring the net validation loss (validation loss from n different output neurons added together), and have noticed my training early-stopping before convergence in one or more of my losses, I am curious as to what the best practices in this situation are. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold. &lt;/li&gt;&#xA;&lt;li&gt;How would one implement this without risking overtraining some output nodes while preventing others from under-training? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am a newcomer to this field, if you hadn't guessed already so please bear with me if this question is not extremely well put together!&lt;/p&gt;&#xA;" OwnerUserId="24300" LastEditorUserId="71442" LastEditDate="2020-04-08T07:05:15.880" LastActivityDate="2020-04-08T07:05:15.880" Title="Earlystopping in multi-output deep learning" Tags="|deep-learning|classification|keras|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13978" PostTypeId="1" AcceptedAnswerId="13979" CreationDate="2016-09-13T17:59:32.950" Score="1" ViewCount="158" Body="&lt;p&gt;I am a student with a master degree in biostatistics. I am interested in data science. I know SAS and R. No experience with python. May I ask for your experts' advice on how to teach myself data science from zero, please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="24303" LastActivityDate="2016-09-13T18:32:41.663" Title="Any guidance for new beginners interested in data science" Tags="|learning|" AnswerCount="1" CommentCount="2" ClosedDate="2016-09-13T21:20:51.047" ContentLicense="CC BY-SA 3.0" />
  <row Id="13979" PostTypeId="2" ParentId="13978" CreationDate="2016-09-13T18:19:25.490" Score="0" Body="&lt;p&gt;In order to learn data science you should know about main tools and ideas in the data scientist's toolbox. &#xA;Two steps which most newbee's follow are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first is a conceptual introduction to the ideas behind turning data into actionable knowledge which involves following steps: &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Getting and Cleaning Data,Data Analysis,Reproducible Research&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Statistical Inference, using Regression Models and Machine Learning&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;The second is a practical introduction to the tools that will be used in the program like version control, markdown, git, GitHub, R, and RStudio.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would advice you to complete a general course on data science from any website or tutorial you find easy for example I personally like this course: &lt;a href=&quot;https://www.coursera.org/specializations/jhu-data-science&quot; rel=&quot;nofollow&quot;&gt;https://www.coursera.org/specializations/jhu-data-science&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then start building some projects of your own for example in my free time I build a lot of small data science projects playing with any available data like US census, petroleum data etc. For example see following link for one of my many data science projects: &#xA;&lt;a href=&quot;https://arjun-chaudhary.shinyapps.io/Data_Analytics/&quot; rel=&quot;nofollow&quot;&gt;https://arjun-chaudhary.shinyapps.io/Data_Analytics/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18673" LastEditorUserId="18673" LastEditDate="2016-09-13T18:32:41.663" LastActivityDate="2016-09-13T18:32:41.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13980" PostTypeId="2" ParentId="13976" CreationDate="2016-09-13T18:47:05.333" Score="1" Body="&lt;p&gt;Based on your comments transitioning big data sets from on-premises systems to a cloud-based system is cumbersome and fraught with challenges. However, you can use Amazon RedShift:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Amazon Redshift is a fully managed, petabyte-scale data warehouse&#xA;  service in the cloud. You can start with just a few hundred gigabytes&#xA;  of data and scale to a petabyte or more.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first step is to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster. &lt;/li&gt;&#xA;&lt;li&gt;After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;OR&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use a Hadoop environment as the landing zone to pull in data from&#xA;various sources, process it, and transfer the processed data to the&#xA;existing data warehouse or other repositories.&lt;/li&gt;&#xA;&lt;li&gt;Explore scenarios of different ways to implement a landing zone. Learn about the architecture of the zone and the tools and techniques for integrating it with various environments.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WYLgR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WYLgR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18673" LastEditorUserId="18673" LastEditDate="2016-09-13T19:04:18.937" LastActivityDate="2016-09-13T19:04:18.937" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="13981" PostTypeId="2" ParentId="8384" CreationDate="2016-09-13T19:21:06.823" Score="0" Body="&lt;p&gt;IBM data scientists break big data into four dimensions: volume, variety, velocity and veracity. This infographic explains and gives examples of each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ibmbigdatahub.com/infographic/extracting-business-value-4-vs-big-data&quot; rel=&quot;nofollow&quot;&gt;http://www.ibmbigdatahub.com/infographic/extracting-business-value-4-vs-big-data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And also this one picture beautifully summarizes whole Big Data and above two answers with one example covering all the parameters.&#xA;&lt;a href=&quot;http://www.ibmbigdatahub.com/infographic/four-vs-big-data&quot; rel=&quot;nofollow&quot;&gt;http://www.ibmbigdatahub.com/infographic/four-vs-big-data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18673" LastActivityDate="2016-09-13T19:21:06.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13982" PostTypeId="1" CreationDate="2016-09-13T21:21:11.037" Score="2" ViewCount="62" Body="&lt;p&gt;I have a list of ranked items and the factors that the group used to create the rankings (like college rankings: 1, 2, 3...), and I'd like to reverse engineer their methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I tried...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I took the standard score of each factor (factor value - mean /&#xA;standard deviation).&lt;/li&gt;&#xA;&lt;li&gt;I assigned coefficients to each factor and added up all these products to create a score.&lt;/li&gt;&#xA;&lt;li&gt;I used the scores to rank the items and compared these rankings to the real rankings, trying to minimize the difference.&lt;/li&gt;&#xA;&lt;li&gt;Rinse and repeat by changing the coefficients.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is a crude method and I am very new to this.  I was wondering what the best way is to figure out how this group weighted the factors to create the rankings.&lt;/p&gt;&#xA;" OwnerUserId="24308" LastActivityDate="2016-10-14T15:18:13.200" Title="Given rankings and the factors that determined the rankings, what is the best method to reverse engineer the rankings?" Tags="|regression|ranking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13984" PostTypeId="1" CreationDate="2016-09-13T23:51:04.063" Score="0" ViewCount="65" Body="&lt;p&gt;Suppose i want to have 80% training data and 20% testing data. How do i choose which 80% of the data to use for training? Should it be completely random? Like what if there is a class label with 2 clusters and the 80% training data we randomly choose only captures one of the clusters?&lt;/p&gt;&#xA;" OwnerUserId="20635" LastActivityDate="2016-10-14T02:11:16.607" Title="Picking training data" Tags="|machine-learning|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13985" PostTypeId="2" ParentId="13984" CreationDate="2016-09-14T01:40:13.807" Score="2" Body="&lt;p&gt;In order to validation you can choose different methods such as &lt;code&gt;k-fold cross validation&lt;/code&gt;, &lt;code&gt;repeated splits&lt;/code&gt; and &lt;code&gt;Bootstrapping&lt;/code&gt;. It seems what you aim to do is 2 fold cross validation and yes it should be random. You should be careful, and it seems your training data and testing data are very different.In this case, you;ll not have a model with a good performance.Try to find a way and make your training data look more like the data on which you want to implement your model. In this case you might want try 10 fold cross validation (for instance)&lt;/p&gt;&#xA;" OwnerUserId="17610" LastActivityDate="2016-09-14T01:40:13.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13986" PostTypeId="2" ParentId="13966" CreationDate="2016-09-14T05:09:54.120" Score="3" Body="&lt;p&gt;You need to save the breaks/endpoints when you perform the bucketing/discretization on the training set as a (named) vector. The same breaks/endpoints can be then re-used on test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The I am giving below sample code using cut function in base R to keep the answer more generally applicable (rather than giving answer specific to bnlearn package). Note the way output of quantile function is saved in vector breaks_to_use in line 3 below and re-used while applying cut on test data. You can (hopefully) do the same using the breaks argument in discretize function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_data &amp;lt;- runif(100)&#xA;test_data &amp;lt;- runif(100)&#xA;breaks_to_use &amp;lt;- quantile(training_data, seq(0, 1, 0.25))&#xA;discretized_training_data &amp;lt;- cut(training_data, breaks = breaks_to_use)&#xA;discretized_test_data &amp;lt;- cut(test_data, breaks = breaks_to_use)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2016-09-14T05:09:54.120" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="13987" PostTypeId="2" ParentId="13967" CreationDate="2016-09-14T06:38:02.673" Score="0" Body="&lt;p&gt;Let's say that we have a perfect algorithm for imputing data.&#xA;You give him a dataset with missing data for some features and it predicts them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had had such an data imputation algorithm, I would have use it as a classifier.&#xA;The fact the we can reduce classification to data imputation means that dat imputation is as hard as classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Rayn said, you can indeed do so. &#xA;Imputation algorithm could have been evaluated as classification algorithm but this is not very common. One of the reason is that the rule by which you hide data from the imputation algorithm is subjective and might effect the results significantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In whatever method you use, you assume that the data behave in someway and missing due to some rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also an important question, since errors in the imputation (you will probably have such), will effect the process down the road.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I prefer to avoid imputation and let the prediction algorithm cope with the missing data. There are many such algorithms. &#xA;A different approach is to use few methods of imputation, choose few simple classifiers (so their complexity won't have a too large influence on the results) and compare the predications.&#xA;In many cases, there is no big difference. &#xA;When there is a big difference, of course that you will prefer to proceed into a more detailed analysis using the winner.&#xA;However, before that try to understand why it has a big advantage. That might lead to interesting insights about your data.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-09-14T06:38:02.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13988" PostTypeId="1" CreationDate="2016-09-14T11:55:24.817" Score="1" ViewCount="163" Body="&lt;p&gt;I'm working on an image classification problem where each test image (query image) is compared with a set of candidate images (of which several images are considered as &quot;correct answers&quot; or &quot;relevant&quot;). From this comparison, a similarity measure is computed. The candidates are sorted decreasingly based on their similarity to the query image. So, basically, we are generating a ranking such as the ones generated by search engines. To evaluate such systems, it's typical to use mean average precision (MAP) and precision-vs-recall plots. However, I came across something with which I'm not familiar, which is the rank-vs-identification plot. From its name, I assume that this plot shows the identification rate at each rank. My problem is how to compute the chance curve for such a plot. Specifically, I have the following piece of code, which was extracted from the code used to generated Figure 3 of the following paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://crcv.ucf.edu/papers/cvpr2014/Resemblance_CVPR14.pdf&quot; rel=&quot;nofollow&quot;&gt;http://crcv.ucf.edu/papers/cvpr2014/Resemblance_CVPR14.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This code is for the chance curve only. I am hoping that someone would help me understand how the chance plot was generated and what the variables x and xMax stand for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;%%A snippet of Matlab code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fSize = 15;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;% Compute Chance&#xA;    xMax = 54;&#xA;     ranks = [1 2 3 4 5:5:xMax]';&#xA;     x = 8;% x value in the equation.&#xA;     c = ones(length(ranks),1); % the equation that calculates the chance     value &#xA;  for r = 1 : length(ranks)&#xA;    if ranks(r) &gt;= xMax; break; end&#xA;%     c(r) = 1 - (nchoosek(xMax-1,ranks(r)) / nchoosek(xMax,ranks(r)));&#xA;%     c(r) = 1 - (xMax - ranks(r))/xMax;&#xA;    c(r) = 1 - newc(xMax-ranks(r),x) / newc(xMax,x);&#xA;end&lt;/p&gt;&#xA;&#xA;&lt;p&gt;leg1 = {'Chance'};&#xA;figure(1);&#xA;semilogx(ranks,c,'linewidth',3,'color',[1,0,0]); set(gca,'FontSize',fSize);&#xA;hold on;&#xA; psAll = zeros(length(ranks));&#xA; figure(1); legend(leg1,'Location','SouthEast','FontSize',fSize); grid; xlim([1 1000])&#xA; xlabel('Rank','FontSize',fSize,'FontWeight','bold'); ylabel('Identification Rate','FontSize',fSize,'FontWeight','bold');&#xA;  xMaxStr = num2str(xMax);&#xA;   set(gca,'XTick',[1 10 100 1000]);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;% newc function used in the above piece of code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;function res = newc(n,x)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;res = n - x + 1 : 1 : n;&#xA;res = prod(res);&lt;/p&gt;&#xA;" OwnerUserId="24274" LastActivityDate="2016-09-14T11:55:24.817" Title="Chance Curve in Accuracy-vs-Rank Plots in matlab" Tags="|machine-learning|matlab|information-retrieval|computer-vision|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13990" PostTypeId="1" AcceptedAnswerId="13999" CreationDate="2016-09-14T12:49:48.277" Score="0" ViewCount="6537" Body="&lt;p&gt;I have a very large &lt;code&gt;skewed&lt;/code&gt; training set where every feature's &lt;code&gt;data-points&lt;/code&gt; are very similar ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, following is some part of the training data :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;93.65034,94.50283,94.6677,94.20174,94.93986,95.21071,1&#xA;94.13783,94.61797,94.50526,95.66091,95.99478,95.12608,1&#xA;94.0238,93.95445,94.77115,94.65469,95.08566,94.97906,1&#xA;94.36343,94.32839,95.33167,95.24738,94.57213,95.05634,1&#xA;94.65813,94.65246,94.64984,95.29596,95.14167,95.39941,1&#xA;95.50876,94.45346,95.23837,95.26877,94.84924,94.8021,0&#xA;94.5774,93.92291,94.96261,95.40926,95.97659,95.17691,0&#xA;93.76617,94.27253,94.38002,94.28448,94.19957,94.98924,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the last column is the &lt;code&gt;class-label&lt;/code&gt; - only 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This only a part of the dataset, but the actual dataset contains about &lt;code&gt;95%&lt;/code&gt; of samples with &lt;code&gt;class-label&lt;/code&gt; being &lt;code&gt;1&lt;/code&gt;, and the rest with &lt;code&gt;class-label&lt;/code&gt; being &lt;code&gt;0&lt;/code&gt;, despite the fact that more or less all the samples are very much similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest an appropriate &lt;code&gt;classifier&lt;/code&gt; (using &lt;code&gt;scikit-learn&lt;/code&gt;) as well.&lt;/p&gt;&#xA;" OwnerUserId="24246" LastEditorUserId="11097" LastEditDate="2016-09-14T14:21:59.820" LastActivityDate="2017-02-03T06:46:22.567" Title="How to deal with a skewed data-set having all the samples almost similar?" Tags="|machine-learning|classification|bigdata|scikit-learn|random-forest|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13992" PostTypeId="1" AcceptedAnswerId="14008" CreationDate="2016-09-14T12:56:58.240" Score="3" ViewCount="2424" Body="&lt;pre&gt;&lt;code&gt;import org.apache.spark.SparkContext&#xA;import org.apache.spark.SparkConf&#xA;import org.apache.spark.sql.cassandra.CassandraSQLContext&#xA;object Test {&#xA;  val sparkConf = new SparkConf(true).set(&quot;spark.cassandra.connection.host&quot;, &amp;lt;Cassandra Server IP&amp;gt;)&#xA;  val sc = new SparkContext(sparkConf)&#xA;  val cassandraSQLContext = new CassandraSQLContext(sc)&#xA;  val numberAsString = cassandraSQLContext.sql(&quot;select * from testing.test&quot;).first().getAs[Int](&quot;number&quot;).toString()&#xA;  val testRDD = sc.parallelize(List(0, 0))&#xA;  val newRDD = testRDD.map { x =&amp;gt; numberAsString }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code that I have written in Spark. I was expecting it to work properly because i am evaluating the numberAsString value and then using it in the map function but it gives me task not serializable error. I am running the job in local mode.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QUG6u.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QUG6u.png&quot; alt=&quot;Error on spark-shell&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m2gsE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m2gsE.png&quot; alt=&quot;Error on spark-shell&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24157" LastEditorUserId="11097" LastEditDate="2016-09-14T14:21:16.193" LastActivityDate="2016-09-15T07:00:20.307" Title="Task not serializable Error" Tags="|bigdata|apache-spark|scala|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13994" PostTypeId="2" ParentId="13992" CreationDate="2016-09-14T13:27:32.017" Score="0" Body="&lt;p&gt;You should broadcast your variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.SparkContext&#xA;import org.apache.spark.SparkConf&#xA;import org.apache.spark.sql.cassandra.CassandraSQLContext&#xA;object Test {&#xA;  val sparkConf = new SparkConf(true).set(&quot;spark.cassandra.connection.host&quot;, &amp;lt;Cassandra Server IP&amp;gt;)&#xA;  val sc = new SparkContext(sparkConf)&#xA;  val cassandraSQLContext = new CassandraSQLContext(sc)&#xA;  val numberAsString = cassandraSQLContext.sql(&quot;select * from testing.test&quot;).first().getAs[Int](&quot;number&quot;).toString()&#xA;  val numberAsStringBC = sc.broadcast(numberAsString)&#xA;  val testRDD = sc.parallelize(List(0, 0))&#xA;  val newRDD = testRDD.map { x =&amp;gt; numberAsStringBC.value }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24320" LastActivityDate="2016-09-14T13:27:32.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="13995" PostTypeId="1" AcceptedAnswerId="14014" CreationDate="2016-09-14T13:37:38.083" Score="0" ViewCount="848" Body="&lt;p&gt;I have the following data frame with one categorical and two numerical columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    V1  V2  V3&#xA;1   A   1   3&#xA;2   A   3   5&#xA;3   B   3   3&#xA;4   C   2   3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have turned this into the following dummy variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    V1.1  V1.2  V1.3 V2  V3&#xA;1   1     0     0    1   3&#xA;2   1     0     0    3   5&#xA;3   0     1     0    3   3&#xA;4   0     0     1    2   3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to apply clustering to this latter set. I guess that I could get better results if I downweight the dummy variable columns (proportionally to their number) because with equal weights the distance based clusters will be distorted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how could I get the following weight vector from the new data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.33, 0.33, 0.33, 1, 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-09-17T11:56:22.900" Title="Calculate feature weight vector for one-hot-encoded data frame in R" Tags="|r|clustering|weighted-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13996" PostTypeId="2" ParentId="13744" CreationDate="2016-09-14T13:57:16.953" Score="0" Body="&lt;p&gt;See Steve Mosher's blog post where he solves this problem: &lt;a href=&quot;https://stevemosher.wordpress.com/2012/04/12/nick-stokes-distance-code-now-with-big-memory/&quot; rel=&quot;nofollow&quot;&gt;&quot;Nick Stokes Distance code, now with Big Memory&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23144" LastActivityDate="2016-09-14T13:57:16.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="13998" PostTypeId="2" ParentId="13982" CreationDate="2016-09-14T14:14:37.377" Score="1" Body="&lt;p&gt;Depending on the complexity of the ranking scheme, this might be more or less difficult, but if you have all the inputs and the outputs, it is very possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start by plotting your data and try to see if you can see a pattern, or any obvious correlation, even though not necessarily linear. Do that for each variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also look at the distribution of your variables (what you call factors) for each of your rank, and see how they differ.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a better understanding on the relationship between the dependent variable (rank) and the independent variables (factors), then you can either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Come up with the model right there&lt;/li&gt;&#xA;&lt;li&gt;Apply a machine learning algorithm (linear/polynomial regression, decision trees, etc.) which will learn the model's parameters for you. Which one you chose will depend on your analysis.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="24320" LastActivityDate="2016-09-14T14:14:37.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="13999" PostTypeId="2" ParentId="13990" CreationDate="2016-09-14T16:50:20.563" Score="2" Body="&lt;p&gt;This is the so-called &quot;Class Imbalance Problem&quot;. Fortunately, there are a number of possible approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably easiest is going to be to use the &lt;code&gt;class_weight&lt;/code&gt; option present in a number of scikit-learn's classifiers. For example, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LogisticRegression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can either set this option to &lt;code&gt;&quot;balanced&quot;&lt;/code&gt; (or &lt;code&gt;&quot;auto&quot;&lt;/code&gt; if you're using an older version of sklearn), in which case it automatically adjusts weights inversely proportional to class frequency (i.e. it tries to compensate for the class imbalance); or you can manually set the weights for each class label using a dict, e.g. &lt;code&gt;{0:1, 1: 20}&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a worked example using an SVM, see &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See here for more information: &lt;a href=&quot;https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning&quot;&gt;https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23581" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-09-14T18:31:29.073" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14000" PostTypeId="2" ParentId="13992" CreationDate="2016-09-14T18:47:28.413" Score="3" Body="&lt;p&gt;Looking at &lt;a href=&quot;https://stackoverflow.com/questions/27579474/serialization-exception-on-spark&quot;&gt;another question&lt;/a&gt; on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the &lt;code&gt;SparkContext&lt;/code&gt; -- which is not serializable -- then an error is thrown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps this is happening to you?&lt;/p&gt;&#xA;" OwnerUserId="23581" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-09-14T18:47:28.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14001" PostTypeId="2" ParentId="13960" CreationDate="2016-09-14T19:09:53.597" Score="16" Body="&lt;p&gt;You do cross-validation when you want to do any of these two things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Model Selection&lt;/li&gt;&#xA;&lt;li&gt;Error Estimation of a Model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Model selection can come in different scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Selecting one algorithm vs others for a particular problem/dataset&lt;/li&gt;&#xA;&lt;li&gt;Selecting hyper-parameters of a particular algorithm for a particular problem/dataset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(please notice that if you are both selecting an algorithm - better to call it model - and also doing hyper-parameters search, you need to do Nested Cross Validation . &lt;a href=&quot;https://stackoverflow.com/questions/31632094/nested-cross-validation-really-necessary&quot;&gt;Is Nested-CV really necessary?&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-validation ensures up to some degree that the error estimate is the closest possible as generalization error for that model (although this is very hard to approximate). When observing the average error among folds you can have a good projection of the expected error for a model built on the full dataset. Also is importance to observe the variance of the prediction, this is, how much the error varies from fold to fold. If the variation is too high (considerably different values) then the model will tend to be unstable. Bootstrapping is the other method providing good approximation in this sense. I suggest to read carefully the section 7 on &quot;Elements of Statistical Learning&quot; Book, freely available at: &lt;a href=&quot;http://statweb.stanford.edu/~tibs/ElemStatLearn/&quot; rel=&quot;noreferrer&quot;&gt;ELS-Standford&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it has been mentioned before you must not take the built model in none of the folds. Instead, you have to rebuild the model with the full dataset (the one that was split into folds). If you have a separated test set, you can use it to try this final model, obtaining a similar (and must surely higher) error than the one obtained by CV. You should, however, rely on the estimated error given by the CV procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After performing CV with different models (algorithm combination, etc) chose the one that performed better regarding error and its variance among folds. You will need to rebuild the model with the whole dataset. Here comes a common confusion in terms: we commongly refer to model selection, thinking that the model is the ready-to-predict model built on data, but in this case it refers to the combination of algorithm+preprocesing procedures you apply. So, to obtain the actual model you need for making predictions/classification you need to build it using the winner combination on the whole dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last thing to note is that if you are applying any kind of preprocessing the uses the class information (feature selection, LDA dimensionality reduction, etc) this must be performed in every fold, and not previously on data. This is a critical aspect. Should do the same thing if you are applying preprocessing methods that involve direct information of data (PCA, normalization, standardization, etc). You can, however, apply preprocessing that is not depend from data (deleting a variable following expert opinion, but this is kinda obvious). This video can help you in that direction: &lt;a href=&quot;https://www.youtube.com/watch?v=S06JpVoNaA0&quot; rel=&quot;noreferrer&quot;&gt;CV the right and the wrong way&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, a final nice explanation regarding the subject: &lt;a href=&quot;https://stats.stackexchange.com/questions/2306/feature-selection-for-final-model-when-performing-cross-validation-in-machine&quot;&gt;CV and model selection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-24T14:47:16.297" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="14002" PostTypeId="1" AcceptedAnswerId="14144" CreationDate="2016-09-14T19:10:31.557" Score="2" ViewCount="312" Body="&lt;p&gt;I'm in the process of developing a new spark-based ARIMA(X) tool, and have reached the point where I need to know whether my coefficient estimates and forecasts are sensible. I can compare my results to R on the same data set, but, as my implementation is distributed while R's is in-memory, I think it's reasonable to assume that there will be some slight differences in our estimates. What I don't know is how different is too different for the coefficient estimates and forecasts? Is there a standard approach to evaluating to reasonableness of a new implementation of time series analytics?&lt;/p&gt;&#xA;" OwnerUserId="13413" LastEditorUserId="13413" LastEditDate="2016-09-15T18:14:31.820" LastActivityDate="2016-09-21T16:37:23.447" Title="ARIMA(X) Validation" Tags="|r|time-series|apache-spark|performance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14003" PostTypeId="2" ParentId="13909" CreationDate="2016-09-14T20:40:29.397" Score="0" Body="&lt;p&gt;This wouldn't be possible in seaborn without changing the &lt;a href=&quot;https://github.com/mwaskom/seaborn/blob/master/seaborn/linearmodels.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;pairplot is designed with only two options for the non-diagonal plots: scatter or reg.  This can be seen in the source code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;if kind == &amp;quot;scatter&amp;quot;:&#xA;    plot_kws.setdefault(&amp;quot;edgecolor&amp;quot;, &amp;quot;white&amp;quot;)&#xA;    plotter(plt.scatter, **plot_kws)&#xA;elif kind == &amp;quot;reg&amp;quot;:&#xA;    plotter(regplot, **plot_kws)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;One option would be copy the source code and create a function similar to pairplot, but with the ability to plot boxplots.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="98307" LastEditDate="2020-08-04T07:40:39.340" LastActivityDate="2020-08-04T07:40:39.340" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14004" PostTypeId="2" ParentId="758" CreationDate="2016-09-14T21:54:54.973" Score="8" Body="&lt;p&gt;Unfortunately, I do not have enough reputation points to answer to the post by Plank, so have to answer to the whole thread - sorry about that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am actually the developer of the open-source Collective Knowledge Framework mentioned above. It attempts to simplify sharing of artifacts and experimental workflows as reusable and reproducible Python components with unified JSON API and JSON meta shared via GitHub. They can also be connected to predictive analytics with the same unified JSON API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have just released the new version V1.8.1 and provided extensive documentation so hopefully it will be easier to understand the concepts now: &#xA;&lt;a href=&quot;http://github.com/ctuning/ck/wiki&quot; rel=&quot;noreferrer&quot;&gt;http://github.com/ctuning/ck/wiki&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We now have many academic and industrial projects based on this framework, so you may check one of them - crowdsourcing program optimization across mobile devices provided by volunteers in a reproducible way: &lt;a href=&quot;http://cknowledge.org/repo&quot; rel=&quot;noreferrer&quot;&gt;http://cknowledge.org/repo&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also keep track of various resources related to reproducible science here: &lt;a href=&quot;https://github.com/ctuning/ck/wiki/Enabling-open-science&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/ctuning/ck/wiki/Enabling-open-science&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I am primarily focusing on making computer systems' research reproducible, I had interesting chats with colleagues from other domains and seems like they have very similar problems. So, I will be very happy if our framework can be of any help to other communities! If you have any questions or suggestions, feel free to get in touch!&lt;/p&gt;&#xA;" OwnerUserId="17768" LastEditorUserId="17768" LastEditDate="2016-09-15T09:05:35.867" LastActivityDate="2016-09-15T09:05:35.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14005" PostTypeId="2" ParentId="13631" CreationDate="2016-09-14T23:09:07.197" Score="0" Body="&lt;p&gt;I would wrap your expression in a CALCULATE function, so something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;activedays = CALCULATE ( &#xA;    COUNTAX ( RELATEDTABLE ( ActivesInactiveData ) , ActivesInactiveData[Surface Date] ) &#xA;    , ISBLANK ( ActivesInactiveData[Topic] ) = FALSE )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I assumed [Topic] is text, so NULL will converted by DAX to BLANK().&lt;/p&gt;&#xA;" OwnerUserId="24333" LastActivityDate="2016-09-14T23:09:07.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14007" PostTypeId="1" CreationDate="2016-09-15T05:53:00.727" Score="6" ViewCount="3952" Body="&lt;p&gt;is it at the end of every tree? or only after all trees are build? I tried to think in both ways but didn't get a clear picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we focus more the part &quot;the loss function is applied between training models&quot; I am trying to understand how is Gradient Descent is applied to minimize the objective function function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;slide no 36 in &lt;a href=&quot;http://www.saedsayad.com/docs/xgboost.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.saedsayad.com/docs/xgboost.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data points reach to a leaf will be assigned a weight. The weight is&#xA;  the prediction&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;how is the weight assigned ? how is the weight calculated ? why predict the weight again ? Can someone shed some light on this?&lt;/p&gt;&#xA;" OwnerDisplayName="user14204" LastEditorDisplayName="user14204" LastEditDate="2016-09-21T06:03:01.617" LastActivityDate="2019-02-15T12:41:05.897" Title="When is Gradient Descent invoked on the objective function while running XGboost?" Tags="|xgboost|gradient-descent|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14008" PostTypeId="2" ParentId="13992" CreationDate="2016-09-15T07:00:20.307" Score="3" Body="&lt;p&gt;I changed my code to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.SparkContext&#xA;import org.apache.spark.SparkConf&#xA;import org.apache.spark.sql.cassandra.CassandraSQLContext&#xA;import org.apache.spark.rdd.RDD&#xA;object Test2 {&#xA;  def calculate(numberAsString: String, testRDD: RDD[Int]): RDD[String] = {&#xA;    val newRDD = testRDD.map { x =&amp;gt; numberAsString }&#xA;    newRDD&#xA;  }&#xA;}&#xA;&#xA;object Test {&#xA;  val sparkConf = new SparkConf(true).set(&quot;spark.cassandra.connection.host&quot;, &amp;lt;Cassandra Server IP&amp;gt;)&#xA;  val sc = new SparkContext(sparkConf)&#xA;  val cassandraSQLContext = new CassandraSQLContext(sc)&#xA;  val numberAsString = cassandraSQLContext.sql(&quot;select * from hdfc.test&quot;).first().getAs[Int](&quot;number&quot;).toString()&#xA;  val testRDD = sc.parallelize(List(0, 0))&#xA;  val newRDD = Test2.calculate(numberAsString, testRDD)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now when it will go to serialize the containing class then it will get serialized.&lt;/p&gt;&#xA;" OwnerUserId="24157" LastActivityDate="2016-09-15T07:00:20.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14011" PostTypeId="1" CreationDate="2016-09-15T09:27:02.430" Score="2" ViewCount="692" Body="&lt;p&gt;I have m( around O(millions) ) of rows of type&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp | val | ind1 | ind2 | ind3 | .... k entries &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My task is to predict the value of &quot;val&quot; for any future given time. How should I model this as a machine learning problem. Note that we want the value of &quot;val&quot; given the past data without knowing the current value of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ind1 | ind2 | ind3 | .... etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;of the same row. This is different in that sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j&gt;i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;About data&lt;/strong&gt; I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe &quot;val&quot; is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have m( around O(millions) ) of rows of type&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp | val | ind1 | ind2 | ind3 | .... k entries &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My task is to predict the value of &quot;val&quot; for any future given time. How should I model this as a machine learning problem. Note that we want the value of &quot;val&quot; given the past data without knowing the current value of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ind1 | ind2 | ind3 | .... etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;of the same row. This is different in that sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j&gt;i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;About data&lt;/strong&gt; I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe &quot;val&quot; is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And please give any idea, How will system learn iteratively as new data rows are added. I do not want to do the whole training again and again.&lt;/p&gt;&#xA;" OwnerUserId="24346" LastEditorUserId="8501" LastEditDate="2016-11-25T08:12:11.827" LastActivityDate="2016-11-25T08:12:11.827" Title="Feature Selection, Machine learning and Time Series analysis, for large financial timeseries" Tags="|machine-learning|algorithms|feature-selection|feature-extraction|tensorflow|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14012" PostTypeId="1" AcceptedAnswerId="14049" CreationDate="2016-09-15T10:11:30.967" Score="1" ViewCount="95" Body="&lt;p&gt;I have a dataset of Social Media Post and want to predict the number of &quot;thumbs up&quot; it will receive over time.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+---------+----------------+-----------+----------------+-----+-------+&#xA;| Post_id | Timestamp      | Follows   | Comments_count | ... | Likes |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;| 01      | 12-04-16 14:00 | 34        | 4              |     | 23    |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;| 01      | 12-04-16 14:35 | 35        | 7              |     | 34    |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;|         | ...            |           |                |     |       |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;| 02      | 12-04-16 14:02 | 134       | 5              |     | 36    |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;| 02      | 12-04-16 14:45 | 136       | 23             |     | 123   |&#xA;+---------+----------------+-----------+----------------+-----+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The likes amount over Time looks like f(x) = sqrt(x)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach is to create a multivariable polynomial regression for each post and somehow ensemble/average them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this a good approach? Which ensemble technique is appropriate?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="24225" LastEditorUserId="24225" LastEditDate="2016-09-15T14:41:47.043" LastActivityDate="2016-09-17T02:14:09.660" Title="Predictive Modeling of Multiple Items" Tags="|time-series|regression|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14014" PostTypeId="2" ParentId="13995" CreationDate="2016-09-15T11:55:44.320" Score="1" Body="&lt;p&gt;The information you need is in the &lt;code&gt;assign&lt;/code&gt; attribute on the matrix returned by &lt;code&gt;model.matrix&lt;/code&gt;. Something like this seems to work for me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data &amp;lt;- data.frame(V1 = factor(c(&quot;A&quot;, &quot;A&quot;, &quot;B&quot;, &quot;C&quot;)), V2 = c(1, 3, 3, 2), V3 = c(3, 5, 3, 3))&#xA;model &amp;lt;- model.matrix(~ . + 0, data = data)&#xA;assignments &amp;lt;- attr(model, &quot;assign&quot;)&#xA;counts &amp;lt;- table(assignments)&#xA;weights &amp;lt;- sapply(assignments, function (x) { 1 / counts[[x]] })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;weights&lt;/code&gt; is now (0.3333333, 0.3333333, 0.3333333, 1.0000000, 1.0000000).&lt;/p&gt;&#xA;" OwnerUserId="13296" LastActivityDate="2016-09-15T11:55:44.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14016" PostTypeId="2" ParentId="14012" CreationDate="2016-09-15T14:52:24.067" Score="0" Body="&lt;p&gt;You could pick a few time windows at specific times since the post and try to regress on that. Since it is count data the obvious choice would be to model the outcome as a Poisson counting process. There are multiple models that are able to do this natively, some generalized linear models but also Neural Networks with a certain loss function. Another option is to model the fraction of followers that will like the post. This is likely easier to generalize however a small portion of the posts goes 'viral' beyond their own followers. This will be tricky to model anyway and you could clip these instances to a fraction of 1.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-09-15T14:52:24.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14020" PostTypeId="1" AcceptedAnswerId="14382" CreationDate="2016-09-15T16:59:23.737" Score="0" ViewCount="4860" Body="&lt;p&gt;I need to set a couple of variables in my Jupyter Notebook where I have a pre-existing sparkContext and sqlContext and I am doing it wrong. If I don't include an&lt;br&gt;&#xA;    sc.stop()&#xA;, I get an error that I am trying to instantiate a second context. If I do include it, I get an error that I am trying to call methods on a stopped context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone tell me the correct way to set these variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark import SparkConf, SparkContext&#xA;from pyspark.sql import HiveContext&#xA;from pyspark.sql import SQLContext&#xA;conf = (SparkConf()&#xA;       .set(&quot;spark.yarn.executor.memoryOverhead&quot;, &quot;4096&quot;)&#xA;       .set(&quot;spark.kryoserializer.buffer.max.mb&quot;, &quot;1024&quot;))&#xA;&#xA;sc.stop()&#xA;sc = SparkContext(conf=conf)&#xA;sqlContext = SQLContext.getOrCreate(sc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23388" LastEditorUserId="23388" LastEditDate="2016-09-15T19:26:34.767" LastActivityDate="2016-10-05T17:38:26.367" Title="Correct way to set Spark variables in jupyter notebook" Tags="|pyspark|jupyter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14021" PostTypeId="1" AcceptedAnswerId="14023" CreationDate="2016-09-15T17:08:55.587" Score="1" ViewCount="7855" Body="&lt;p&gt;I have a csv file with around 130 columns and 6000 rows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is the best way to import them into python, so that I can later use them in a classification algorithm(columns are the labels and rows are individual samples)&lt;/p&gt;&#xA;" OwnerUserId="24355" LastActivityDate="2017-07-16T06:44:57.383" Title="importing csv data in python" Tags="|machine-learning|python|dataset|csv|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14022" PostTypeId="2" ParentId="14021" CreationDate="2016-09-15T17:16:02.900" Score="13" Body="&lt;p&gt;For small data, I think &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html&quot;&gt;pandas.read_csv&lt;/a&gt; is the way to go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &quot;medium&quot; data, I recommend &lt;a href=&quot;http://dask.pydata.org/en/latest/dataframe-create.html&quot;&gt;dask.read_csv&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for big data, I recommend &lt;a href=&quot;http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader&quot;&gt;spark.read.csv&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-09-15T17:25:10.220" LastActivityDate="2016-09-15T17:25:10.220" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14023" PostTypeId="2" ParentId="14021" CreationDate="2016-09-15T17:16:11.780" Score="4" Body="&lt;p&gt;Use pandas library: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd &#xA;pd.read_csv('foo.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pandas identify the headers automatically and is a great tool for data wrangling. &lt;br/&gt;&#xA;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/10min.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;10 Minutes intro to pandas&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24225" LastEditorUserId="24225" LastEditDate="2017-07-16T06:44:57.383" LastActivityDate="2017-07-16T06:44:57.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14024" PostTypeId="2" ParentId="14021" CreationDate="2016-09-15T21:11:00.140" Score="0" Body="&lt;p&gt;You can also SFrame. First &lt;a href=&quot;https://turi.com/download/install-graphlab-create.html&quot; rel=&quot;nofollow&quot;&gt;install graph lab&lt;/a&gt;. Then: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import graphlab as gl&#xA;data = gl.SFrame.read_csv('data.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you're 'hardcore' you can use &lt;a href=&quot;https://docs.python.org/2/library/csv.html&quot; rel=&quot;nofollow&quot;&gt;python's basic csv reader&lt;/a&gt;, but then you will have to write loops to manage the data yourself, so why bother reinvent the wheel, just use pandas or Frame.&lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2016-09-15T21:11:00.140" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14025" PostTypeId="1" AcceptedAnswerId="14035" CreationDate="2016-09-15T21:45:21.687" Score="4" ViewCount="8111" Body="&lt;p&gt;I am curious if ordinal features are treated differently from categorical features in decision tree, I am interested in both cases where target is categorical or continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a difference, could you anybody point to good source with explanation and any packages (R or Python) supporting it?&lt;/p&gt;&#xA;" OwnerUserId="14999" LastActivityDate="2016-09-16T11:01:34.750" Title="Ordinal feature in decision tree" Tags="|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14027" PostTypeId="1" AcceptedAnswerId="14033" CreationDate="2016-09-15T22:48:11.367" Score="8" ViewCount="17868" Body="&lt;p&gt;I am going over the Udacity tutorial on Neural Networks.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a diagram from the tutorial:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nCRdr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nCRdr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes this a '2 layer neural network'?  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I was &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/&quot; rel=&quot;noreferrer&quot;&gt;under the impression&lt;/a&gt; that the first layer, the actual input, should be considered a layer and included in the count.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This screenshot shows 2 matrix multiplies and 1 layer of ReLu's. To me this looks like 3 layers. &lt;strong&gt;There are arrows pointing from one to another, indicating they are separate.&lt;/strong&gt; Include the input layer, and this looks like a 4 layer NN.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="15533" LastEditDate="2016-09-16T06:09:32.637" LastActivityDate="2021-08-06T21:39:45.533" Title="Counting the number of layers in a neural network" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14028" PostTypeId="1" AcceptedAnswerId="14030" CreationDate="2016-09-16T00:51:07.420" Score="7" ViewCount="5185" Body="&lt;p&gt;On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me make the question more clear.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take this classic image:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QkT6H.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QkT6H.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.toptal.com/machine-learning/an-introduction-to-deep-learning-from-perceptrons-to-deep-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text under the image above&lt;/a&gt; says, &quot;A neural network is really just a composition of perceptrons, connected in different ways.&quot; They all look connected in the exact same way to me.&lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="29169" LastEditDate="2019-06-09T15:43:14.240" LastActivityDate="2023-04-20T09:51:50.067" Title="What is the purpose of multiple neurons in a hidden layer?" Tags="|machine-learning|neural-network|activation-function|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="14029" PostTypeId="2" ParentId="14028" CreationDate="2016-09-16T01:40:32.893" Score="2" Body="&lt;p&gt;I think the key here is that when in training, having same weights (and same neurons in the hidden layer) will not lead to optimal solution, it is the different weights that will lead to lower difference between actual and predicted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I coded up this really dumb Neural Network (as a learning exercise). Maybe it could be of some help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;class NeuralNetwork(object):&#xA;    def __init__(self, X, Y, hidden_layer_dim):&#xA;        self.X = X / np.max(X)&#xA;        self.Y = Y / np.max(Y) # Used for training&#xA;        self.hidden_layer_dim = hidden_layer_dim&#xA;&#xA;    def initialize_weights(self):&#xA;        self.w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))&#xA;        self.w2 = np.random.normal(0,1, self.hidden_layer_dim)&#xA;&#xA;    def forward(self, xi):&#xA;        &quot;&quot;&quot;&#xA;        x1 is 2d array&#xA;        &quot;&quot;&quot;&#xA;        # This method is also used for training &#xA;        xi = xi / np.max(xi)&#xA;        z2 = np.dot(xi, self.w1)&#xA;        a2 = sigmoid(z2)&#xA;        z3 = np.dot(a2, self.w2)&#xA;        y_hat = sigmoid(z3)&#xA;        return y_hat&#xA;&#xA;    def dump_train(self, n_iterations):&#xA;        min_mse = np.inf&#xA;        for i in range(n_iterations):&#xA;            w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))&#xA;            w2 = np.random.normal(0,1, self.hidden_layer_dim)&#xA;&#xA;            z2 = np.dot(self.X, w1)&#xA;            a2 = sigmoid(z2)&#xA;            z3 = np.dot(a2, w2)&#xA;            y_hat = sigmoid(z3)&#xA;&#xA;            diff = self.Y - y_hat&#xA;            mse = np.dot(diff, diff)&#xA;&#xA;            if mse &amp;lt; min_mse:&#xA;                min_mse = mse&#xA;                print(&quot;min_mse: {}, iteration: {}&quot;.format(mse, i))&#xA;                self.w1 = w1&#xA;                self.w2 = w2&#xA;&#xA;def sigmoid(a):&#xA;    return 1 / (1 + np.e ** (-a))&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;    my_x = np.array([[8,5], [7,5], [8,4],[8,1], [4, 3], [5,2], [4,2]], dtype=np.float)&#xA;    my_y = np.array([100, 90, 88, 60, 50, 45, 40], dtype=np.float)&#xA;&#xA;    NN = NeuralNetwork(my_x, my_y, hidden_layer_dim=3)&#xA;    NN.initialize_weights()&#xA;    NN.dump_train(100000)&#xA;&#xA;    new_x = [[8,4], [7,1], [3,3]]&#xA;&#xA;    y_hat = NN.forward(new_x)&#xA;&#xA;    print(&quot;prediction: {}&quot;.format(y_hat))&#xA;    print(&quot;weight 1: {}&quot;.format(NN.w1))&#xA;    print(&quot;weight 2: {}&quot;.format(NN.w2)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weight 1: [[-0.13787113 -1.30913914  0.64624687]&#xA; [-1.76733779  0.77449265  1.61122177]]&#xA;weight 2: [-1.42489674 -1.94360005  2.56365303]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The weights are different.&lt;/p&gt;&#xA;" OwnerUserId="3070" LastEditorUserId="3070" LastEditDate="2018-12-29T21:24:36.907" LastActivityDate="2018-12-29T21:24:36.907" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14030" PostTypeId="2" ParentId="14028" CreationDate="2016-09-16T02:10:41.833" Score="10" Body="&lt;p&gt;To explain using the sample neural network you have provided:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Purpose of the multiple inputs: Each input represents a feature of the input dataset.&lt;/li&gt;&#xA;&lt;li&gt;Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.&lt;/li&gt;&#xA;&lt;li&gt;Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way they will learn different &quot;weights&quot; and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edited to answer additional questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The only reason the neurons aren't redundant is because they've all been &quot;trained&quot; with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.&lt;/li&gt;&#xA;&lt;li&gt;The outputs from the Orange neurons (use your diagram as an example), are &quot;squashed&quot; by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-09-16T03:52:53.223" LastActivityDate="2016-09-16T03:52:53.223" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14031" PostTypeId="1" AcceptedAnswerId="14034" CreationDate="2016-09-16T03:37:10.137" Score="1" ViewCount="680" Body="&lt;p&gt;As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;a href=&quot;https://datascience.stackexchange.com/questions/14028/what-is-the-purpose-of-multiple-neurons-in-a-hidden-layer&quot;&gt;this answer&lt;/a&gt; says:  &quot;To learn non-linear decision boundaries when classifying the output, multiple neurons are required.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9CLai.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9CLai.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a multiply, add, ReLu, multiply and add, all in series.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From studying math, I know that a composite function made out of linear functions is itself linear.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do you coax non-linearity out of multiple linear functions?&lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-09-16T07:33:52.690" Title="How do multiple linear neurons together allow for nonlinearity in a neural network?" Tags="|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14032" PostTypeId="2" ParentId="14027" CreationDate="2016-09-16T05:14:21.720" Score="9" Body="&lt;p&gt;This is a 2-layer network because it has a single hidden layer and an output layer. We don't count the first layer. When we say 3 layers, we actually mean 2 hidden layers and an output layer. Perhaps this helps you?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; We don't count the input layer because there's no parameter (bias + weights). In actual implementation, it's not implemented. Netural network framework simply &quot;connects&quot; the input features to the first adjustable layer (eg: hidden layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CuVdi.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CuVdi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-09-16T05:38:37.337" LastActivityDate="2016-09-16T05:38:37.337" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14033" PostTypeId="2" ParentId="14027" CreationDate="2016-09-16T05:35:18.323" Score="5" Body="&lt;p&gt;Input layer is a layer, it's not wrong to say that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when calculating the depth of a deep neural network, we only consider the layers that have tunable weights.&lt;/p&gt;&#xA;" OwnerUserId="21895" LastEditorUserId="21895" LastEditDate="2016-09-16T07:03:19.333" LastActivityDate="2016-09-16T07:03:19.333" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14034" PostTypeId="2" ParentId="14031" CreationDate="2016-09-16T07:33:52.690" Score="3" Body="&lt;p&gt;The phase&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&quot;To learn non-linear decision boundaries when classifying the output, multiple neurons are required.&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is &lt;strong&gt;NOT&lt;/strong&gt; correct. More precisely, it should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&quot;To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function.&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand why, imagine you have a network with many layers and nodes (&lt;strong&gt;multiple neurons&lt;/strong&gt; in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-09-16T07:33:52.690" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14035" PostTypeId="2" ParentId="14025" CreationDate="2016-09-16T08:01:36.283" Score="2" Body="&lt;p&gt;As per my knowledge, it doesn't matter for a decision tree model whether the features are ordinal or categorical. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Decision trees describe patterns by using a list of attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a more detailed explanation, I am providing here some links which you will find helpful for such queries.&lt;br&gt;&#xA;&lt;a href=&quot;http://www.ibm.com/support/knowledgecenter/SS3RA7_17.0.0/clementine/nodes_treebuilding.html&quot; rel=&quot;nofollow&quot;&gt;http://www.ibm.com/support/knowledgecenter/SS3RA7_17.0.0/clementine/nodes_treebuilding.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/tree.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.ryerson.ca/~rmichon/mkt700/SPSS/Creating%20Decision%20Trees.htm&quot; rel=&quot;nofollow&quot;&gt;http://www.ryerson.ca/~rmichon/mkt700/SPSS/Creating%20Decision%20Trees.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-16T08:01:36.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14036" PostTypeId="1" CreationDate="2016-09-16T10:02:28.657" Score="1" ViewCount="75" Body="&lt;p&gt;I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( &lt;strong&gt;&lt;em&gt;total size: 400MB-800MB&lt;/em&gt;&lt;/strong&gt; ). To decrease the time consumed for calculations involved (feature-feature), I want  to &lt;strong&gt;&lt;em&gt;divide data in some 4-5 parts and execute all parts parallely.&lt;/em&gt;&lt;/strong&gt;&#xA;Since data is not huge in size, I am avoiding to use Hadoop.&#xA;What can be the &lt;strong&gt;&lt;em&gt;option for this parallelization&lt;/em&gt;&lt;/strong&gt;.. ( multi-threading , GPU or anything else) ??&lt;/p&gt;&#xA;" OwnerUserId="24386" LastActivityDate="2016-09-16T10:02:28.657" Title="Parallel processing for feature selection in microarray dataset" Tags="|machine-learning|data-mining|feature-selection|distributed|parallel|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14037" PostTypeId="2" ParentId="13967" CreationDate="2016-09-16T10:15:50.660" Score="0" Body="&lt;p&gt;There is not such thing as &quot;the best way to impute data&quot;. The best method will always depend on your specific application and model. Just remember the No Free Lunch Theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways in which you can impute values. You can ignore the rows containing missing values, you can impute values based on the other rows (mean, using classification, etc.), or you can even replace the missing values by a constant. Which method to use will depend on how much effort you want to put into the imputation algorithm and the final results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As usual, I'd start by using a simple method (the average, as you suggested), and increase the complexity of the imputation only if the results are not satisfactory enough. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-09-16T10:15:50.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14038" PostTypeId="2" ParentId="14025" CreationDate="2016-09-16T11:01:34.750" Score="2" Body="&lt;p&gt;Ordinal data and Categorical data are treated exactly the same in GBM. However, the results will most likely be different if you have ordinal data formatted as categorical. This would also be true of categorical data formatted as ordinal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ordinal data as Categorical data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us consider the following example. You are predicting weights of humans. You have the mapping &lt;code&gt;{'S': 0, 'XL': 1, 'L': 2, 'M': 3}&lt;/code&gt; for a feature of t-shirt size. The GBM does its splitting down to the last branch and has yet to split on the t-shirt size feature. At this point it tries to split on t-shirt size but finds no gain as a single split can only group &lt;code&gt;['S', 'XL']&lt;/code&gt; and &lt;code&gt;['M', 'L']&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we did encode ordinally then we would get &lt;code&gt;{'S'=0, 'M'=1, 'L'=2, 'XL=3'}&lt;/code&gt; as our mapping. Thus the final split would result in &lt;code&gt;['S', 'M']&lt;/code&gt; and &lt;code&gt;['L', 'XL']&lt;/code&gt; and would certainly result in a gain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I understand that this is a very contrived example and that tuning the parameters correctly would eventually split as such&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        ['S', 'XL', 'L', 'M']             &#xA;           /            \                 No Gain, maybe loss (-2)&#xA;     ['S', 'XL']      ['M', 'L']&#xA;       /       \         /      \         Big gain (+15)&#xA;    ['S']   ['XL']     ['M']   ['L']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why though, would you leave this up to tricky parameter tuning, longer training of models and potential loss of information when you can provide the GBM with &lt;em&gt;known&lt;/em&gt; information. It makes sense to encode ordinality in this scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Categorical Data as Ordinal&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say we now have employment status as categorical data for weight prediction. We label this as follows &lt;code&gt;{'unemployed': 0, 'employed': 1, 'self_employed':2, 'education':3}&lt;/code&gt;. We can assume over a large dataset that the 4 differences will has little effect on weight. However, if the dataset is sufficiently large there is a good chance that there will be a branch where the ordinality appear to have some relationship to weight. The model would then learn this and split accordingly. This would cause overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case if we represent this data &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Employment_Status &#xA;0&#xA;2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as follows (Binary Encoding) this relationship would not be learnt as they are being considered as different features. ES stands for Employment Status.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ES_unemployed    ES_employed    ES_self_employed    ES_education&#xA;1                0              0                   0&#xA;0                0              1                   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again, this is a contrived example. In this case you could overcome the overfitting in conventional methods such as through cross validation but this is extremely time consuming. IMO it would be well worth doing the correct feature engineering first and treating the categorical features as categorical and ordinal as ordinal.&lt;/p&gt;&#xA;" OwnerUserId="20995" LastActivityDate="2016-09-16T11:01:34.750" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14039" PostTypeId="1" AcceptedAnswerId="14041" CreationDate="2016-09-16T14:01:09.883" Score="21" ViewCount="25241" Body="&lt;p&gt;Can anyone recommend a tool to quickly label several hundred images as an input for classification?&lt;/p&gt;&#xA;&lt;p&gt;I have ~500 microscopy images of cells. I would like to assign categories such as &lt;code&gt;'healthy'&lt;/code&gt;, &lt;code&gt;'dead'&lt;/code&gt;, &lt;code&gt;'sick'&lt;/code&gt; manually for a training set and save those to a csv file.&lt;/p&gt;&#xA;&lt;p&gt;Basically, the same as described in &lt;a href=&quot;https://datascience.stackexchange.com/questions/13335/build-a-tool-for-manually-classifying-training-data-images&quot;&gt;this&lt;/a&gt; question, except I do not have proprietary images, so maybe that opens up additional possibilities?&lt;/p&gt;&#xA;" OwnerUserId="24392" LastEditorUserId="29169" LastEditDate="2021-03-11T19:09:15.510" LastActivityDate="2021-03-11T19:09:15.510" Title="Tool to label images for classification" Tags="|machine-learning|image-classification|training|" AnswerCount="6" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="14041" PostTypeId="2" ParentId="14039" CreationDate="2016-09-16T14:50:06.953" Score="9" Body="&lt;p&gt;I just hacked together a very basic helper in python&#xA;it requires that all images are stored in a pyton list &lt;code&gt;allImages&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;category=[]&#xA;plt.ion()&#xA;&#xA;for i,image in enumerate(allImages):&#xA;    plt.imshow(image)&#xA;    plt.pause(0.05)&#xA;    category.append(raw_input('category: '))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24392" LastActivityDate="2016-09-16T14:50:06.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14042" PostTypeId="1" CreationDate="2016-09-16T19:26:36.247" Score="2" ViewCount="119" Body="&lt;p&gt;I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage in dropdown and finally upload a text file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I read this file in my notebook? Do I need to create a connection ? &lt;/p&gt;&#xA;" OwnerUserId="24404" LastEditorUserId="1330" LastEditDate="2021-01-06T14:29:20.263" LastActivityDate="2021-01-06T14:29:20.263" Title="How to read data in DSX from Data hub object storage container?" Tags="|dataset|data-cleaning|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14043" PostTypeId="2" ParentId="11852" CreationDate="2016-09-16T20:11:40.297" Score="1" Body="&lt;p&gt;What you're trying to do is to minimize the difference between input and its reconstruction. They may be completely different values and RBM may be working fine! It's due to the fact that Restricted Boltzmann Machine models some probabilistic distribution. Learning rule is to find values of weights to model the distribution of the data. You may clamp a data (visible units) and see a reconstruction, but since hidden units are stochastic units, they would have many values. Even when you set your network free and let the units have some values according to stochastic units rule, after some time you can get a sample from RBM, which should be from your data distribution. &lt;/p&gt;&#xA;" OwnerUserId="24405" LastActivityDate="2016-09-16T20:11:40.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14045" PostTypeId="1" CreationDate="2016-09-16T22:38:09.860" Score="2" ViewCount="663" Body="&lt;p&gt;What are the advantages vs disadvantages of cross validation types? Like k-fold, leave one out, etc.&lt;/p&gt;&#xA;" OwnerUserId="20635" LastActivityDate="2016-09-17T04:36:39.387" Title="Cross validation techniques" Tags="|machine-learning|cross-validation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14046" PostTypeId="1" AcceptedAnswerId="14047" CreationDate="2016-09-16T23:17:05.873" Score="1" ViewCount="6927" Body="&lt;p&gt;I was working through a tutorial on the titanic disaster from Kaggle and I'm getting different results depending on the details of how I use &lt;code&gt;cross_validation.cross_val_score&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I call it like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[&quot;Survived&quot;], cv=3)&#xA;&#xA;print(scores.mean())&#xA;&#xA;0.801346801347&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get a different set of scores than if I call it like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kf = KFold(titanic.shape[0], n_folds=3, random_state=1)&#xA;&#xA;scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic[&quot;Survived&quot;], cv=kf)&#xA;&#xA;print(scores.mean())&#xA;&#xA;0.785634118967&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These numbers are close, but different enough to be significant. As far as I understand, both code snippets are asking for a 3-fold cross validation strategy. Can anyone explain what is going on under the hood of the second example which is leading to the slightly lower score?&lt;/p&gt;&#xA;" OwnerUserId="24411" LastEditorUserId="29169" LastEditDate="2019-06-07T06:05:15.317" LastActivityDate="2020-08-18T11:30:08.403" Title="sklearn.cross_validation.cross_val_score &quot;cv&quot; parameter question" Tags="|python|scikit-learn|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14047" PostTypeId="2" ParentId="14046" CreationDate="2016-09-17T00:40:14.633" Score="5" Body="&lt;p&gt;From the sklearn &lt;a href=&quot;https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt; for &lt;code&gt;cross_val_score&lt;/code&gt;'s cv argument :&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;For int/None inputs, if the estimator is a classifier and y is either&#xA;binary or multiclass, StratifiedKFold is used. In all other cases,&#xA;KFold is used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I believe that in the first case, &lt;code&gt;StratifiedKFold&lt;/code&gt; is being used as the default. In the second case, you are explicitly passing a &lt;code&gt;KFold&lt;/code&gt; generator.&lt;/p&gt;&#xA;&lt;p&gt;The difference between the two is also documented in the &lt;a href=&quot;https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;KFold&lt;/code&gt; divides all the samples in &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt; groups of samples, called folds (if&#xA;&lt;span class=&quot;math-container&quot;&gt;$k = n$&lt;/span&gt;, this is equivalent to the &lt;em&gt;Leave One Out&lt;/em&gt; strategy), of equal&#xA;sizes (if possible).&lt;/p&gt;&#xA;&lt;p&gt;[...]&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;StratifiedKFold&lt;/code&gt; is a variation of &lt;code&gt;k-fold&lt;/code&gt; which returns&#xA;&lt;code&gt;stratified&lt;/code&gt; folds: each set contains approximately the same&#xA;percentage of samples of each target class as the complete set.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This difference in folds is what is causing the difference in scores.&lt;/p&gt;&#xA;&lt;p&gt;As a side note, I noticed that you are passing a &lt;code&gt;random_state&lt;/code&gt; argument to the &lt;code&gt;KFold&lt;/code&gt; object. However, you should note that this seed is only used if you also set &lt;code&gt;KFold&lt;/code&gt;'s shuffle parameter to &lt;code&gt;True&lt;/code&gt;, which by default is &lt;code&gt;False&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24412" LastEditorUserId="25638" LastEditDate="2020-08-18T11:30:08.403" LastActivityDate="2020-08-18T11:30:08.403" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14048" PostTypeId="2" ParentId="14011" CreationDate="2016-09-17T01:09:07.337" Score="1" Body="&lt;p&gt;Your model should be based on what your data represents.  If you want meaningful results, you should get some idea of what your data is representing before trying to answer this question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, you could use your &lt;strong&gt;ind&lt;/strong&gt;# columns to predict the values of your &lt;strong&gt;ind&lt;/strong&gt;#${_j}$ rows, then use the &lt;strong&gt;ind&lt;/strong&gt;#${_j}$s to predict the &lt;strong&gt;val&lt;/strong&gt;${_j}$.  But this would only make sense if there is a predictive relationship between the &lt;strong&gt;ind&lt;/strong&gt;#${_j}$s.  And you would definite want to have some idea of what that relationship might be in order to choose an appropriate algorithm for each particular column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, the same would apply for $j_{th}$ row &lt;strong&gt;val&lt;/strong&gt;${_j}$ prediction.  At the very least you would want to  make an educated guess about the relationship between your &lt;strong&gt;ind&lt;/strong&gt;#${_j}$s and your &lt;strong&gt;val&lt;/strong&gt;${_j}$ column so that an appropriate algorithm could be chosen.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As another example, you may be able to use the previous row as a predictor for the next value in the time series. i.e. &lt;strong&gt;ind&lt;/strong&gt;#${_{j-1}}$s  =&gt; &lt;strong&gt;val&lt;/strong&gt;${_j}$.  But again, this only makes sense if you are able to make reasonable assumptions about the relationship of the data, because you ultimately need to choose the algorithm you use to make the predictions.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-17T01:09:07.337" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14049" PostTypeId="2" ParentId="14012" CreationDate="2016-09-17T01:55:50.073" Score="1" Body="&lt;p&gt;Overall classification is generally better when the decision rules of each component classifier differ and provide complimentary information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question becomes: Can you set up your component classifiers so that their decision rules are different and compliment one another based on the feature space? e.g. Does Post 1 have a significantly different feature space than Post 2?  etc. If so, the ensemble approach should be beneficial.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which technique?  If you can highly train each classifier and make it an expert in different regions of the feature space, try models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;mixture model&lt;/li&gt;&#xA;&lt;li&gt;mixture distribution&lt;/li&gt;&#xA;&lt;li&gt;gating subsystem&lt;/li&gt;&#xA;&lt;li&gt;winner take all.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="24409" LastEditorUserId="24409" LastEditDate="2016-09-17T02:14:09.660" LastActivityDate="2016-09-17T02:14:09.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14050" PostTypeId="2" ParentId="14045" CreationDate="2016-09-17T04:36:39.387" Score="4" Body="&lt;p&gt;As you probably know by now, cross-validation is a method of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;partition dataset into 'train set' and 'test set'&lt;/li&gt;&#xA;&lt;li&gt;fit model to train set, get the prediction error on test&lt;/li&gt;&#xA;&lt;li&gt;repeat (1-2) in a bunch of different, 'pre-defined' ways, storing the prediction error each time.&lt;/li&gt;&#xA;&lt;li&gt;average all the prediction errors to understand how your model will behave in the wild!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All the 'magic' of different cross-validation (CV) types happens at (3), basically. Say your dataset has $n$ observations.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;leave $p$ out CV separates your dataset into a train set of size $n-p$ and a test of $p$, gets the prediction error, and repeats this process for every single possible $p$-sized subset of the dataset.&lt;/li&gt;&#xA;&lt;li&gt;leave $1$ out CV is a special case of leave $p$ out where $p=1$&lt;/li&gt;&#xA;&lt;li&gt;$k$-fold cross CV partitions your dataset into $k$ equal sized partitions, trains on the first $k-1$ partitions and tests on the last partition, and gets the prediction error. The result at the end is ten $k$ prediction errors to average over. (Note that $k=n$ is the same as leave $1$ out CV.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As for disadvantages and advantages, it's basically just a trade-off between exhaustiveness and speed. And I guess it depends on your dataset. The upper limit of exhaustiveness is basically doing leave $p$ out CV for all $p=\{1,2,...,n\}$. Probably depends too on the linearity of your dataset. In my experience, I haven't had trouble using $k$-fold cross CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;Reference&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-09-17T04:36:39.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14052" PostTypeId="2" ParentId="13995" CreationDate="2016-09-17T11:56:22.900" Score="0" Body="&lt;p&gt;Actually a more reasonable weight vector would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0.5, 0.5, 0.5, 1, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why - because any two rows will be different in exactly 0 or 2 columns. With this weight, the one-hot encoded attribute yields a distance contribution of exactly 1 or exactly 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way is to encode your data using these weights instead of 0 and 1 rather than applying the weights in later calculations.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-09-17T11:56:22.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14053" PostTypeId="1" AcceptedAnswerId="14155" CreationDate="2016-09-17T12:48:37.273" Score="0" ViewCount="5112" Body="&lt;p&gt;Just as an example I'm using iris data. I have a histogram, which is nice, but the distribution is weird, and I'd like to see what the bars represent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a histogram:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3HaKX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3HaKX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I'd like to see the high/low that each bar represents. For instance, the first bar has a frequency of 34, but a width of something in the range of the lowest third of 0.0-0.5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously there won't be a width of 0.0, so what is the width range? How can I show it on the histogram?&lt;/p&gt;&#xA;" OwnerUserId="24420" LastEditorUserId="471" LastEditDate="2016-09-17T16:07:18.880" LastActivityDate="2016-10-22T18:54:19.453" Title="How to see actual numbers in a hist on R?" Tags="|r|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14055" PostTypeId="2" ParentId="14053" CreationDate="2016-09-17T13:46:44.177" Score="0" Body="&lt;p&gt;There are other ways to do this but:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;&#xA;data_frame(val=iris$Petal.Width,&#xA;           bin=cut(iris$Petal.Width, include.lowest=TRUE, # by default include.lowest defaults to TRUE in hist() &#xA;                   breaks=hist(iris$Petal.Width, plot=FALSE)$breaks)) %&amp;gt;% &#xA;  group_by(bin) %&amp;gt;% &#xA;  do(as.data.frame(setNames(as.list(range(.$val)), c(&quot;low&quot;, &quot;high&quot;))))&#xA;## Source: local data frame [12 x 3]&#xA;## Groups: bin [12]&#xA;## &#xA;##          bin   low  high&#xA;##       &amp;lt;fctr&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;&#xA;## 1    [0,0.2]   0.1   0.2&#xA;## 2  (0.2,0.4]   0.3   0.4&#xA;## 3  (0.4,0.6]   0.5   0.6&#xA;## 4    (0.8,1]   1.0   1.0&#xA;## 5    (1,1.2]   1.1   1.2&#xA;## 6  (1.2,1.4]   1.3   1.4&#xA;## 7  (1.4,1.6]   1.5   1.6&#xA;## 8  (1.6,1.8]   1.7   1.8&#xA;## 9    (1.8,2]   1.9   2.0&#xA;## 10   (2,2.2]   2.1   2.2&#xA;## 11 (2.2,2.4]   2.3   2.4&#xA;## 12 (2.4,2.6]   2.5   2.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="710" LastActivityDate="2016-09-17T13:46:44.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14059" PostTypeId="2" ParentId="13904" CreationDate="2016-09-17T22:33:37.577" Score="0" Body="&lt;p&gt;Found the issue and I will post the answer in case anybody will get this issue with ARMA. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0), exog= dataset_period[dummy_variable_period]).fit()&#xA;dataset_period['predict_'+str(col[1])+'_'+str(col[2])] = model.predict()&#xA;dataset_period['ape_'+str(col[1])+'_'+str(col[2])] = abs(  (dataset_period[col] - dataset_period['predict_'+str(col[1])+'_'+str(col[2])]) / dataset_period[col])&#xA;valid_ape = dataset_period['ape_'+str(col[1])+'_'+str(col[2])][dataset_period['ape_'+str(col[1])+'_'+str(col[2])] &amp;lt; 50]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;issue was with the parameters. the model accepted 2 parameters for &lt;strong&gt;&lt;em&gt;group_level&lt;/em&gt;&lt;/strong&gt; and I was basically providing 1 so in this case automatically become &lt;em&gt;zero and raise error&lt;/em&gt;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;solved with creating a new &lt;strong&gt;&lt;em&gt;def&lt;/em&gt;&lt;/strong&gt; and the model will use single column as the parameter for &lt;strong&gt;&lt;em&gt;group_level&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0),exog= dataset_period[dummy_variable_period]).fit()&#xA;dataset_period['predict_'+str(col[1])] = model.predict()&#xA;dataset_period['ape_'+str(col[1])] = abs((dataset_period[col] - dataset_period['predict_'+str(col[1])]) / dataset_period[col])&#xA;valid_ape = dataset_period['ape_'+str(col[1])][dataset_period['ape_'+str(col[1])] &amp;lt; 50]`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8752" LastEditorUserId="8752" LastEditDate="2016-09-17T23:38:38.877" LastActivityDate="2016-09-17T23:38:38.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14061" PostTypeId="2" ParentId="14002" CreationDate="2016-09-17T22:49:56.173" Score="0" Body="&lt;p&gt;What some people do is some kind of 'ordered' k-fold cross validation. (Check &lt;a href=&quot;http://robjhyndman.com/hyndsight/tscvexample/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;, for instance.) You partition the data into $k$ partitions. Then, you fit the model on the partition 1, test on partition 2, get prediction error. Then, you fit the model on the union of partitions 1 and 2, test on partition 3, get prediction error. So on so forth, then average the prediction errors.&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-09-17T22:49:56.173" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14063" PostTypeId="1" AcceptedAnswerId="14101" CreationDate="2016-09-18T09:19:43.033" Score="1" ViewCount="118" Body="&lt;p&gt;I provide an offline library of Music to my users.&#xA;My goal is to understand what my users are looking for, which means translate raw user searches to: Music Artists, Songs, Albums and then add music to the company library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the suggested clustering algorithms to group common short sentences into a single entity. Example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Taylor Swift Shake it Off&lt;/li&gt;&#xA;&lt;li&gt;Taylor Swift Shaek it Off &lt;/li&gt;&#xA;&lt;li&gt;Shake it off &lt;/li&gt;&#xA;&lt;li&gt;Twylor Swift Shake it Off&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I tried &lt;a href=&quot;http://www.artfact-online.fr/blog/blog-post/6&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; example and works fine for a specific number of clusters (K) where K &amp;lt; N | K &amp;lt;= N. But since searches are unpredictable need to find a way to automate the number of clusters: my goal is to cluster 2 or more similar items and let alone single searches in independent clusters, example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cluster 1:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Taylor Swift Shake it Off&lt;/li&gt;&#xA;&lt;li&gt;Taylor Swift Shaek it Off &lt;/li&gt;&#xA;&lt;li&gt;Shake it off &lt;/li&gt;&#xA;&lt;li&gt;Twylor Swift Shake it Off&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cluster 2&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Avicci&lt;/li&gt;&#xA;&lt;li&gt;Avicii Ibiza&lt;/li&gt;&#xA;&lt;li&gt;Avicci Electro House&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cluster 3&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Juan Gabriel&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cluster 4&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adele&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Cluster 5&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Britney Spears&lt;/li&gt;&#xA;&lt;li&gt;Britney Spears VMA 2016&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-09-18T09:28:45.573" LastActivityDate="2016-09-20T04:20:17.587" Title="Music corpus sentence level clustering" Tags="|clustering|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14064" PostTypeId="1" CreationDate="2016-09-18T11:14:27.453" Score="2" ViewCount="1732" Body="&lt;p&gt;I've got a collection of historic log files for a process that repeats daily, roughly the same each day. &#xA;I want an NN that, from the moment that the daily process starts, estimates what time the whole process will finally complete. &#xA;As the daily process progresses, the estimate should become more accurate and converge on the actual completion time as soon as possible.&#xA;A degree of confidence in the prediction would be useful too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The daily process is a collection of mostly serial subprocesses. Mostly serial because some subprocesses have no impact on the end time ( critical path vs non ).&#xA;There are other pattens hidden in the data. Weekends take longer than weekdays. In the real world, maybe others too.&#xA;There is a small amount of noise in the data, subprocess duration can be affected by completely external factors. &#xA;The performance of one given day has little / no expected correlation with the next day, although there may be long term trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've had a little success with a simple RNN but results have been underwhelming, even after trying a lot of hyperparameters.&#xA;Best results so far have been from one-hot encoding and &quot;cross joining&quot; to expand each day into multiple samples and padding them to the same maximum dimensions...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Xsample1.1 = [Event1, ]              Y = EventN&#xA;Xsample1.2 = [Event1, Event2, ]      Y = EventN&#xA;Xsample1.N = [Event1, ..., EventN ]  Y = EventN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've encoded weekday as one-hot variables and normalized datetimes to one universal day in preprocessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the examples I find are for predicting the next N samples, sliding forward, not a fixed end sample.&#xA;I'm using ipython3, pandas and keras and I've generated a &lt;a href=&quot;https://github.com/gregn610/ttl/tree/master/data/population_v2.1&quot; rel=&quot;nofollow noreferrer&quot;&gt;test population &lt;/a&gt; using some simple logic I'd expect an NN to easily handle. YHat for each file is the last datetime. The .debug files capture how the prediction should evolve but that information is not available from the training data and so is not used in the NN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a time series or regression problem, what type of NN should I use and how should i structure the input data for best results.&lt;/p&gt;&#xA;" OwnerUserId="24439" LastEditorUserId="24439" LastEditDate="2017-12-14T12:57:31.573" LastActivityDate="2017-12-14T12:57:31.573" Title="Time series prediction without sliding window" Tags="|python|keras|rnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14065" PostTypeId="1" AcceptedAnswerId="14068" CreationDate="2016-09-18T12:28:11.107" Score="0" ViewCount="6655" Body="&lt;p&gt;This is fairly straight forward but i am unable to do it. My data frame has a id variable which is repeating. For same id I want to replace the NAs in other continuous variable(rating and sur) with their corresponding mean. can anyone pls suggest&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID rating Sur&#xA;101 60 0.7687&#xA;101 78 NA&#xA;101 NA 0.765&#xA;102 60 NA&#xA;102 NA 0.654 &#xA;102 75 0.435&#xA;103 NA 0.576&#xA;103 68 0.875&#xA;103 70 NA &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10416" LastEditorUserId="10517" LastEditDate="2016-09-18T18:52:18.337" LastActivityDate="2017-01-10T16:55:42.600" Title="Imputing missing values by mean by id column in R" Tags="|r|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14068" PostTypeId="2" ParentId="14065" CreationDate="2016-09-18T16:14:21.100" Score="1" Body="&lt;p&gt;If you want the mean, you could use dplyr syntax:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = structure(list(ID = c(101L, 101L, 101L, 102L, 102L, 102L, 103L,103L, 103L), &#xA;                    rating = c(60L, 78L, NA, 60L, NA, 75L, NA, 68L, 70L), &#xA;                    Sur = c(0.7687, NA, 0.765, NA, 0.654, 0.435, 0.576, 0.875, NA)), &#xA;               .Names = c(&quot;ID&quot;, &quot;rating&quot;, &quot;Sur&quot;), class = &quot;data.frame&quot;, row.names = c(NA,-9L))&#xA;&#xA;&#xA;library(dplyr)&#xA;df %&amp;gt;% &#xA;  group_by(ID) %&amp;gt;% &#xA;  mutate(rating = ifelse(is.na(rating), mean(rating, na.rm = TRUE), rating),&#xA;         Sur = ifelse(is.na(Sur), mean(Sur, na.rm = TRUE), Sur))&#xA;&#xA;&#xA;     ID rating     Sur&#xA;  &amp;lt;int&amp;gt;  &amp;lt;dbl&amp;gt;   &amp;lt;dbl&amp;gt;&#xA;1   101   60.0 0.76870&#xA;2   101   78.0 0.76685&#xA;3   101   69.0 0.76500&#xA;4   102   60.0 0.54450&#xA;5   102   67.5 0.65400&#xA;6   102   75.0 0.43500&#xA;7   103   69.0 0.57600&#xA;8   103   68.0 0.87500&#xA;9   103   70.0 0.72550&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for other imputations you can look at packages like mi, mice, caret, amelia or simputation. And I'm probably forgetting a lot of others. Personally I like the simplicity of simputation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example median imputation is just one line:&#xA;&lt;code&gt;impute_median(df, rating + Sur ~ ID)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-09-18T16:14:21.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14069" PostTypeId="1" AcceptedAnswerId="15223" CreationDate="2016-09-18T16:45:15.647" Score="12" ViewCount="28714" Body="&lt;p&gt;I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it with LabelEncoder from scikit-learn. The problem is there are too many of them, and I do not want to convert them manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be an easy way to automate this process.&lt;/p&gt;&#xA;" OwnerUserId="14999" LastActivityDate="2018-06-13T22:51:18.667" Title="Mass convert categorical columns in Pandas (not one-hot encoding)" Tags="|scikit-learn|pandas|categorical-data|labels|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14071" PostTypeId="1" AcceptedAnswerId="14105" CreationDate="2016-09-18T19:55:38.020" Score="0" ViewCount="60" Body="&lt;p&gt;I'm using orange 3 and when I Score plots using the scatter plot widget how do I view the numerical score associated with the ordered list of plots. Did that feature get removed with the upgrade from 2.7 to 3? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, &lt;/p&gt;&#xA;" OwnerUserId="24445" LastActivityDate="2016-09-20T08:02:15.580" Title="Score for Ranked plots using scatter plot widget" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14072" PostTypeId="1" AcceptedAnswerId="14080" CreationDate="2016-09-18T21:20:50.413" Score="0" ViewCount="6839" Body="&lt;p&gt;I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I return only the details of the student that have positive grade (make the join) but not using SQL Context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val all_students = sc.textFile(&quot;/user/cloudera/Data&quot;);&#xA;&#xA;case class Students(Customer_ID:String,Name:String,Age:String);&#xA;&#xA;def MyClass(line: String) = {&#xA;val split = line.split(',');&#xA;Students(split(0),split(1),split(2))&#xA;}&#xA;&#xA;val df = all_students.map(MyClass).toDF(&quot;Customer_ID&quot;,&quot;Name&quot;, &quot;Age&quot;).select(&quot;Customer_ID&quot;,&quot;Name&quot;, &quot;Age&quot;);&#xA;&#xA;val students_positive_grande = sc.textFile(&quot;/user/cloudera/Data&quot;);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I make the join between this datasets? I want to join the &quot;Customer_ID&quot; with the first column of the second column...&lt;/p&gt;&#xA;" OwnerUserId="23811" LastActivityDate="2016-09-19T07:04:48.347" Title="Join two dataframes - Spark Mllib" Tags="|apache-spark|scala|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14073" PostTypeId="1" CreationDate="2016-09-19T00:22:02.257" Score="0" ViewCount="336" Body="&lt;p&gt;I've implemented simple Widrow-Hopf perceptron and k-means clustering and compare results against MNIST data set. I didn't expect great results because of linear nature of these algorithms.  WH perceptron ended up with ~70% accuracy and k-means with ~50% (I have unit tests and compare with pure random chance 1/10 I suppose they do work somehow correct).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I gave &quot;a hint&quot; to k-means by set up initial centroids around different digits, so it can converge faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose that k-means showed this result because digits are pretty much similar (6 is like 8 and 8 is like 9). Then I assumed that generalizations ability of WH perceptron is better than k-means. Did anyone see any article/book about this topic? I want to understand results from a rigor mathematical perspective.&lt;/p&gt;&#xA;" OwnerUserId="24453" LastEditorUserId="14675" LastEditDate="2017-07-11T11:18:08.163" LastActivityDate="2017-07-11T11:18:08.163" Title="k-means and perceptron generalization ability" Tags="|neural-network|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14075" PostTypeId="2" ParentId="14073" CreationDate="2016-09-19T00:53:26.877" Score="-1" Body="&lt;p&gt;Comparing K-means and perceptrons doesn't make sense, they are different types of algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-means doesn't have a generalization ability at all, since it is an &lt;a href=&quot;https://en.wikipedia.org/wiki/Unsupervised_learning&quot; rel=&quot;nofollow&quot;&gt;unsupervised&lt;/a&gt; algorithm, and &lt;a href=&quot;https://en.wikipedia.org/wiki/Generalization_error&quot; rel=&quot;nofollow&quot;&gt;generalization is a property of supervised learning algorithms&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are changing it to behave like &lt;a href=&quot;https://en.wikipedia.org/wiki/Supervised_learning&quot; rel=&quot;nofollow&quot;&gt;a supervised algorithm&lt;/a&gt;, than it will have poor generalization ability compared to the perceptron, because K-means will have a tendency to &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;overfit&lt;/a&gt; the data since it is forced to assign a cluster to each of its data points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to compare perceptron's performance to other algorithms, you should compare it to k-NN (K nearest neighbors), Random Forests, or Support Vector Machines. &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2016-09-19T00:53:26.877" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14076" PostTypeId="2" ParentId="14064" CreationDate="2016-09-19T01:46:29.430" Score="0" Body="&lt;p&gt;How big is N ? If it's not a huge number (if it's huge you need to use NN with online learning to achieve same results without inverting a matrix), how about to use simple linear regression and linear algebra?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short you will have N models for every stage, when new event comes you reevaluate your model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you will use linear algebra, all that you need is to estimate projection matrix (every line of this matrix is your events chain on step k [1:N]): $$ P=A(A^TA)^{-1}A^T $$&lt;/p&gt;&#xA;" OwnerUserId="24453" LastActivityDate="2016-09-19T01:46:29.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14078" PostTypeId="1" AcceptedAnswerId="14085" CreationDate="2016-09-19T05:42:56.340" Score="0" ViewCount="807" Body="&lt;p&gt;In case of Bagged Logistic regression, people suggest more the bags better will be results.&#xA;There should be some threshold for more. Is there any technique available which can suggest no. of bags for the type of data available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ex. &#xA;&lt;strong&gt;Details about data&lt;/strong&gt; &#xA;No. of features - 6 &#xA;No. of records - 1 M\&#xA;Bag size - 20%&#xA;No. of Bags - ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For above mentioned details, Is there any way to devise no. of bags?&lt;/p&gt;&#xA;" OwnerUserId="24053" LastActivityDate="2016-09-21T02:51:51.767" Title="Is there any technique, which can decide no. of bags for Bagged Logistic regression?" Tags="|machine-learning|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14080" PostTypeId="2" ParentId="14072" CreationDate="2016-09-19T07:04:48.347" Score="1" Body="&lt;p&gt;Use this syntax:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val joinedDF = students_positive_grande.as('a).join(&#xA;                          df.as('b),&#xA;                          $&quot;a.Customer_ID&quot; === $&quot;b.Customer_ID&quot;)&#xA;&#xA;joinedDF.select($&quot;a.Customer_ID&quot;, $&quot;b.Customer_ID&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-09-19T07:04:48.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14083" PostTypeId="1" CreationDate="2016-09-19T08:45:26.780" Score="2" ViewCount="80" Body="&lt;p&gt;I observed that my GPU's memory is being consumed but the Utilisation stays 0. Because of this, my model is taking forever to load. I have tweaked &lt;a href=&quot;https://github.com/yoonkim/CNN_sentence&quot; rel=&quot;nofollow noreferrer&quot;&gt;this code&lt;/a&gt; to handle multilabel data. The only changes being-using y as a matrix instead of vector and different methods of calculation of cost and errors(Sorry that I am not able to provide the actual code). We can observe that the theano function train_model is taking a lot of time. I am a beginnner at Theano and also this method internally seems to having cascading variables and methods as well which is making me heard to debug. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What might be the possible reasons for such a scenario? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What debug methods should I use for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/h3AlI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h3AlI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="31513" LastEditDate="2017-05-28T09:08:34.027" LastActivityDate="2017-05-28T09:08:34.027" Title="GPU Utilisation Issues" Tags="|machine-learning|deep-learning|theano|multilabel-classification|gpu|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14084" PostTypeId="1" CreationDate="2016-09-19T09:00:04.510" Score="1" ViewCount="1948" Body="&lt;p&gt;About the dataset&#xA;I have a training dataset of&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;129 columns(last column being the classes, i.e., y values)&lt;/li&gt;&#xA;&lt;li&gt;6068 rows&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have to train some algo to do binary classification. The data set has &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;701 examples of type A&lt;/li&gt;&#xA;&lt;li&gt;5367 examples of type B&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The test set consists of 1398 examples. Here is the accuracy I got for various algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;voting ensemble -&gt; 0.73963&lt;/li&gt;&#xA;&lt;li&gt;stochastic gradient boosting -&gt; 0.77682&lt;/li&gt;&#xA;&lt;li&gt;Adaboost -&gt; 0.75107&lt;/li&gt;&#xA;&lt;li&gt;bagging classifier(Decision tree) -&gt; 0.76538&lt;/li&gt;&#xA;&lt;li&gt;Random Forests -&gt; 0.75250&lt;/li&gt;&#xA;&lt;li&gt;Extra trees -&gt; 0.75393&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All the above results are from kaggle so they probably are just from half the test set. &#xA;The above methods were implemented using scikit-learn library in python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please suggest ways to improve accuracy, may be by things like dimensionality reduction or better algorithms. Also, please provide sample code in possible.&lt;/p&gt;&#xA;" OwnerUserId="24355" LastActivityDate="2020-08-04T10:54:11.497" Title="Binary classification: best ways to pre-procees the data" Tags="|machine-learning|python|classification|preprocessing|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14085" PostTypeId="2" ParentId="14078" CreationDate="2016-09-19T10:02:03.047" Score="0" Body="&lt;p&gt;From the start I have to state that I am not aware about any paper regarding number of bags for bagged logistic regression. Therefore my assertions apply generally, for any bagged ensemble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short answer is no, and I do not think that if possible to be constructed. There are a couple of reasons that I will describe above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first reason is that it depends on the complexity of the joint probability you want to estimate. Technically any model you build for prediction purposes aims to estimate a conditional probability space on output variable giving the joint input variables. Doing bagging puts the problem if the estimations from selected samples does cover the that relation in all places. To exemplify, you can have one input categorical variable with 2 levels or with 10 levels. I think more estimations are needed for the later case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second reason if the sample itself is representative. Often in practice the sample is not purely a random sample. It contains various interdependencies which can lead to problems. For example if you collect data from multiple countries and you have more samples from some countries simply because of costs, but not related with the phenomena you study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third reason is that 20%. A typical bootstrap sample have the same size as original. Reducing the size is interesting because of performance gain or because of induced artificially variance which makes the samples &quot;more independent&quot;. However estimating what is the effect of decreasing the sample size is again hard to estimate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourth factor could be also the signal to noise ratio. In other words if you data is noisy enough you need more samples to surface to interesting signal in a stable way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since finding the answer to your question is hard, and knowing it is close to similar with knowing the true model, there are things which can be follow in order to find a proper number of bags. What I recommend would be to build repeatedly models with increasing number of bags and see the error estimates. Usually when those estimates have a low variance and adding bags does improve the performance significantly than you can assume you need no more bags and bagging with your chosen model is not able to capture any other structure in your data.  &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2016-09-19T10:02:03.047" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14087" PostTypeId="2" ParentId="14084" CreationDate="2016-09-19T11:20:59.813" Score="1" Body="&lt;p&gt;A better clarity about the kind of problem that you are trying to solve will be really useful. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I any case the first thing I will suggest, is please go through your features and try to understand the relationship with the response. All tree based methods are highly prone to overfitting and hence they don't generalise well.&#xA;If you come to the conclusion that some variables show a linear relationship and some show a non-linear then you might be better off using Linear methods such as LogisticRegression or LinearDiscriminantClassifier(if your features show an indication that they are sampled from a normal distribution). Obviously you can use PCA or regularization methods if you are certain that some features are highly correlated and offer no prediction value. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Secondly, if you have done all the above then you are probably done with the feature selection step. The next approach is feature creation. See if certain variables are non-linear in nature. Then try adding higher degree polynomial terms. Try diagnosing interaction effects between features, try to include features for those interactions. If there are any features you think are redundant to the problem. Think again. See if you can extract any information out of them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Kindly tag the link of the problem here.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-09-19T11:20:59.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14088" PostTypeId="1" CreationDate="2016-09-19T11:35:00.540" Score="0" ViewCount="1548" Body="&lt;p&gt;I've been using Keras to do some timeseries predictive neural nets.&#xA;One thing that's had me staggered is how I was getting an almost 99% accuracy rate on very very noisy and uncorrelated data (I was looking for accuracy of around 50% to be good!)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I've now found, is that after I trained my model on the training set, I was casually running &lt;code&gt;model.evaluate(x_test, y_test)&lt;/code&gt; before then running a manual backtest on the data. The manual backtest on x_test and comparing it to the error in y_test then showed almost perfect accuracy...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I then correct in assuming that Keras when using the &lt;code&gt;evaluate()&lt;/code&gt; function in Keras the original model is actually updated with the testing data as well? This seems a bit strange and counterintuitive to me?&lt;/p&gt;&#xA;" OwnerUserId="3313" LastActivityDate="2021-06-16T19:48:28.133" Title="Keras' Evaluate function training model on test set" Tags="|neural-network|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14090" PostTypeId="1" CreationDate="2016-09-19T12:21:42.723" Score="3" ViewCount="334" Body="&lt;p&gt;I am trying to validate a classifier using two different training and testing datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The feature I am considering is a feature constructed doing the fold-change between two original features, i.e. $log2(feat_A / feat_b)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that, across the two different datasets, I see the same distribution of fold-change across groups but with different values. See, in the images attached, the values for each group (each box) in the different datasets (each plot).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LmjZ4m.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LmjZ4m.png&quot; alt=&quot;First&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/1Bz3Xm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1Bz3Xm.png&quot; alt=&quot;Second&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What may be causing the differences in fold-change? In this case the fold-change is different also in sign, which means that while in one dataset $feat_a$ is greater than $feat_b$, the opposite is true for the other dataset. However, the pairwise relation between the two groups (the two boxes in each plot) is maintained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about different normalization procedures on each feature across different dataset, which may explain the shift of the distribution of the fold-change. What other problems may be in these datasets?&lt;/p&gt;&#xA;" OwnerUserId="133" LastActivityDate="2019-12-05T22:01:55.223" Title="Shifted feature distribution across different datasets" Tags="|classification|dataset|distribution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14091" PostTypeId="2" ParentId="14084" CreationDate="2016-09-19T15:00:19.970" Score="1" Body="&lt;p&gt;There are two steps to this problem: feature selection/ dimensionality reduction and selecting the predictive model.  Selecting the 'best' features to use in the model will often improve the accuracy, and there are quite a few methods you can use.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When you are working with continuous data, you can use a regularization method such as &lt;strong&gt;Lasso&lt;/strong&gt; or &lt;strong&gt;ElasticNet&lt;/strong&gt; to select the best features for the model. These are both available in sklearn, but they do require some parameter tuning to find the right hyperparameters to get the best results.&lt;/li&gt;&#xA;&lt;li&gt;You can also take a look at &lt;strong&gt;sklearn.feature_selection&lt;/strong&gt;.  These are more statistical approaches to select features, generaly based on their p-values.  This also only works with continuous data.&lt;/li&gt;&#xA;&lt;li&gt;Another approach is calculating the Gini importance from decision tree models, such as RandomForest.  This is useful when you have continuous and categorical data.  &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is an example.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;After you have selected the best features, you want to choose the right model for binary classification.  The go-to model in this case is logistic regression.  There are multiple hyperparameters in sklearn.linear_model.LogisticRegression and in order to get the best results, you may have to perform some grid searches to find the right parameters.&lt;/p&gt;&#xA;&lt;p&gt;Here are some additional resources that may prove helpful:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.nycdatascience.com/student-works/bnp-claims/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.nycdatascience.com/student-works/bnp-claims/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.kaggle.com/2016/07/21/approaching-almost-any-machine-learning-problem-abhishek-thakur/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://insidebigdata.com/2014/10/22/ask-data-scientist-bias-vs-variance-tradeoff/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://insidebigdata.com/2014/10/22/ask-data-scientist-bias-vs-variance-tradeoff/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;I added the last link, discussing bias vs. variance tradeoff, because I think it is important to understand when testing your models.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="98307" LastEditDate="2020-08-04T10:54:11.497" LastActivityDate="2020-08-04T10:54:11.497" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14092" PostTypeId="1" CreationDate="2016-09-19T16:20:31.417" Score="3" ViewCount="3135" Body="&lt;p&gt;I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain subset of the data, trying different model parameters and checking the prediction performance with the scikit-learn package. Still I need a final model based on the model parameters from those random sampling results, but obviously I cannot build a model on top of the full dataset. What can I do with this problem? I am a newbie of machine learning, any suggestions are welcome. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21795" LastActivityDate="2016-12-09T01:07:13.043" Title="Predictive modeling on big data set that can't fit into memory" Tags="|machine-learning|python|scikit-learn|decision-trees|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14093" PostTypeId="2" ParentId="14092" CreationDate="2016-09-19T17:05:48.773" Score="2" Body="&lt;p&gt;In general, more training examples means improvement in learning but you can also get a very good (and nearby to the optimal score) if you just fit a good algorithm on a subset of your data set that has enough training examples. Here are a few things you can do in your current case :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Take a subset of the data say about 4-5GB. The only thing you need to consider is that your target labels should be nearly stratified in the subset other wise the model will perform poorly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply PCA on your data and try to minimize the number of features. There may be features that are just redundant in your subset of data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply the algorithm, best suited for your dataset, on your current subset. You may need to do a bit of research for that.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And above all, I suggest one thing. Training such a large dataset on a local machine can be too much of pain. So, it's better if you deploy your model on AWS or Google's Machine Learning platform provided by them on cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it helps!!&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-19T17:05:48.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14094" PostTypeId="1" AcceptedAnswerId="14099" CreationDate="2016-09-19T18:09:52.350" Score="0" ViewCount="1010" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If no, what are the other criteria. Please elaborate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What should be minimum value of AUC to select a model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24468" LastActivityDate="2016-09-20T03:13:20.663" Title="Is ROC and AUC the only criteria for choosing a model?" Tags="|model-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14095" PostTypeId="2" ParentId="14094" CreationDate="2016-09-19T18:30:30.010" Score="3" Body="&lt;p&gt;No, area under receiver operating characteristic (AUROC) is just one metric amongst very many possibilities, even assuming you just want to pick a standard approach. There are too many to list in a simple Stack Exchange answer. You can take a look at this list extracted from &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics&quot; rel=&quot;nofollow&quot;&gt;scikit learn documentantion on metrics&lt;/a&gt; for example, which is not by any means exhaustive:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Accuracy classification score.&#xA;Area Under the Curve (AUC) using the trapezoidal rule&#xA;Average precision (AP) from prediction scores&#xA;The Brier score.&#xA;The F1 score, also known as balanced F-score or F-measure&#xA;The F-beta score&#xA;The average Hamming loss.&#xA;Average hinge loss (non-regularized)&#xA;Jaccard similarity coefficient score&#xA;Log loss, aka logistic loss or cross-entropy loss.&#xA;The Matthews correlation coefficient (MCC) for binary classes&#xA;Area Under the Curve (AUC) from prediction scores&#xA;Zero-one classification loss.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What you should choose depends entirely on your model class and goals for the learning work. AUROC is a good choice for a binary classifier when you have different business costs associated with false positives and false negatives. That is because it gives you a sense of how well the classifier can be tuned to be more or less sensitive, and get the best outcomes by changing the class threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no minimum AUC or other metric required to select a model in practice. It depends on performance of existing solutions (including non-ML ones) and what the costs vs benefits would be of using the model. Clearly a poorly-performing model (e.g. with AUROC 0.5, no better than random guessing on a balanced data set) is unlikely to gain much from being implemented in a production environment, and you would want to take a new look at the original problem and see if was reasonable to expect anything from ML at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To decide which metric to use, you need to define some goal for the eventual model. An ideal metric is one that can be simply converted to terms that the end users will care about. Failing that, one that can be compared to known results from other ML work in the subject area (domain) of the problem.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-09-19T18:30:30.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14096" PostTypeId="2" ParentId="14092" CreationDate="2016-09-19T19:18:01.867" Score="0" Body="&lt;p&gt;You can also use some out of core libraries &lt;a href=&quot;https://turi.com/products/create/&quot; rel=&quot;nofollow&quot;&gt;like GraphLab&lt;/a&gt; (watch out though, GraphLab is free only for educational purposes). GraphLab works the same way as Scikit-learn but can run 'out of core' (meaning it isn't limited by the amount of memory you have). &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2016-09-19T19:18:01.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14098" PostTypeId="2" ParentId="14092" CreationDate="2016-09-19T21:23:20.780" Score="3" Body="&lt;p&gt;Whether you're using this for a hobby or for your job, I would recommend EMR on Amazon Web Services. The name is anachronistic (it's no longer just for map reduce), but EMR enables you to quickly stand up an Apache Spark cluster of several machines. Spark comes with machine learning libraries for building trees, and can scale to however many machines you need (to the point that you no longer face memory constraints). EMR is cheap because you're charged hourly and don't have to buy your own hardware. Apache Spark is great for your resume, and it's also a practical skill because it's generally the first tool most data scientists turn to when working with data sets that are too large for a single machine. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-09-19T21:23:20.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14099" PostTypeId="2" ParentId="14094" CreationDate="2016-09-20T00:51:05.540" Score="1" Body="&lt;p&gt;Not at all, and AUC is not a particularly well-respected measure of model performance. It performs particularly poorly for rare events, since the usual criterion for &quot;best fit&quot; is obtained for sensitivities between .7 and .9 where the specificities will typically be also .7 to .9 for evan a good model. The choice of a proper &quot;best&quot; criterion will very much depend on the type of outcome, the frequency of the outcomes of interest and any weighting or costs associated with them. Naive application of the AUC will have no consideration of the cost of false positives or false positives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should read up on calibration and cross-validation. This is quite a broad topic, but the statisticians have been at it for a while and there is a lot to be learned by searching an affiliated site, CrossValidated.com: &lt;a href=&quot;https://stats.stackexchange.com/search?q=best+fit+cross-validation&quot;&gt;https://stats.stackexchange.com/search?q=best+fit+cross-validation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The minimum AUC would of course be 0.5, which is what you would get from pure chance. There is no established minimum. Even teh p &amp;lt; 0.05 is an aribitrary boundary, not at all established by a theory. You need to decide how much to value good versus bad decisions from the task at hand.&lt;/p&gt;&#xA;" OwnerUserId="24478" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-09-20T03:13:20.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14100" PostTypeId="2" ParentId="14090" CreationDate="2016-09-20T04:07:07.107" Score="1" Body="&lt;p&gt;It'd be easier to understand if the plots were labeled. Nonetheless, I think I understand the question, and perhaps it's just sampling error. Why do you have two different pairs of train and test data? Are they from different sources? In that case, you probably don't want to be using the same classifier on both. Also, what do you mean 'validate' using two different train and test sets? Classifiers are trained on a train set, and validated on a test set, what do you mean you're validating on two different train/test pairs?&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-09-20T04:07:07.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14101" PostTypeId="2" ParentId="14063" CreationDate="2016-09-20T04:20:17.587" Score="3" Body="&lt;p&gt;I would suggest hierarchical clustering. It's unsupervised, and you don't need to predefine number of clusters. How it works (for the bottom up version) is each sentence (or object) is initialized as its own cluster. At each iteration of the algorithm, the two clusters of the smallest intra-cluster distance are joined, all the way until there's a 'root' where everything is one cluster. The result of this is a big dendrogram, and you can cut the dendrogram at whatever point you want to define clusters. Or you can just inspect it. Here's a nice &lt;a href=&quot;http://people.revoledu.com/kardi/tutorial/Clustering/Numerical%20Example.htm&quot; rel=&quot;nofollow&quot;&gt;numerical example&lt;/a&gt; of the alg in action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should detect the same types of clusters you're already finding, and you wouldn't need to redefine a distance metric. And, I anticipate you'll get the same amount of class separation you seem to already be getting; in fact, the nice thing about the dendrogram from hierarchical clustering is that it illustrates class separation nicely. it's 'hclust' in R, not sure for Python.&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-09-20T04:20:17.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14102" PostTypeId="1" AcceptedAnswerId="14123" CreationDate="2016-09-20T05:40:10.897" Score="6" ViewCount="5840" Body="&lt;p&gt;OK this is my first time in ML and for starter I am implementing Naive Bayes. I have Cricket(sports) data in which I have to check whether the team will win or lost based on Toss Won|Lost and Bat First|Second. Below is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.naive_bayes import GaussianNB&#xA;import numpy as np&#xA;&#xA;&quot;&quot;&quot;&#xA;    Labels : Lost, Draw, Won [-1,0,1]&#xA;    Features&#xA;    ==========&#xA;        Toss(Lost,Won) = [-1,1]&#xA;        Bat(First, Second) = [-1,1]&#xA;&quot;&quot;&quot;&#xA;#Based on Existing Data Features are:&#xA;features = np.array([[-1, 1],[-1, 1]])&#xA;labels = np.array([0,1])&#xA;# Create a Gaussian Classifier&#xA;model = GaussianNB()&#xA;&#xA;# Train the model using the training sets&#xA;model.fit(features, labels)&#xA;&#xA;# Predict Output&#xA;predicted = model.predict([[1,0]])&#xA;print(predicted)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On running this I get error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:393: RuntimeWarning: divide by zero encountered in log&#xA;[0]&#xA;  n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))&#xA;/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: divide by zero encountered in true_divide&#xA;  (self.sigma_[i, :]), 1)&#xA;/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: invalid value encountered in subtract&#xA;  (self.sigma_[i, :]), 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code given &lt;a href=&quot;https://github.com/kadnan/PakistanEnglanTestMatches&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10879" LastEditorUserId="10879" LastEditDate="2016-09-22T04:26:22.217" LastActivityDate="2018-01-30T21:27:37.267" Title="Naive Bayes: Divide by Zero error" Tags="|python|scikit-learn|naive-bayes-classifier|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14103" PostTypeId="2" ParentId="8722" CreationDate="2016-09-20T06:07:28.097" Score="1" Body="&lt;p&gt;This is a general ML answer, not about Spark 1.5.1, but perhaps it can help you find a way to the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem is multi-label as well as multi-class (e.g. &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/modules/multiclass.html&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some algorithms have multi-label versions, and will deal with the problem internally so you don't have to. If you don't have a multi-label version of your algorithm, you can either change to an algorithm that does, or roll your own multi-label solution around your algorithm, for example using binary classifiers for each label. (This typically leads to worse performance as there are many complications and possible optimisations to take into account).&lt;/p&gt;&#xA;" OwnerUserId="24384" LastActivityDate="2016-09-20T06:07:28.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14104" PostTypeId="2" ParentId="14092" CreationDate="2016-09-20T07:00:27.880" Score="1" Body="&lt;p&gt;You need to use an algorithm that does not load all examples in memory at the same time. Use Stochastic Gradient Descent (a good implementation is Vowpal Wabbit).&lt;/p&gt;&#xA;" OwnerUserId="13686" LastEditorUserId="13686" LastEditDate="2016-09-21T18:49:09.037" LastActivityDate="2016-09-21T18:49:09.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14105" PostTypeId="2" ParentId="14071" CreationDate="2016-09-20T08:02:15.580" Score="0" Body="&lt;p&gt;Hm, what was the meaning of this score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See. This is why it was removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(As I recall, it was the average probability assigned to the correct class by the k-nearest neighbour classifier on the projection -- or something similar. The number is useful for ranking projections, but doesn't have any meaningfyl interpretation or absolute scale, like &gt;0.70 is good, &gt;0.90 is excellent. Hence there's no point in showing it.)&lt;/p&gt;&#xA;" OwnerUserId="16578" LastActivityDate="2016-09-20T08:02:15.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14106" PostTypeId="2" ParentId="14088" CreationDate="2016-09-20T08:26:46.000" Score="1" Body="&lt;p&gt;Keras doesn't update your model with testing data. &#xA;It might be that your labels have been provided wrong in the test data, check the model.predict_classes() to get the output of your classes and crosscheck them with your actual output manually by picking random subset. &lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-09-20T08:26:46.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14108" PostTypeId="2" ParentId="14065" CreationDate="2016-09-20T09:31:52.850" Score="1" Body="&lt;p&gt;You can use for loops for find mean and replacing them with NA for unique ids&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in unique(df$ID)){&#xA;  df$rating[df$ID==i] &amp;lt;- ifelse(is.na(df$rating[df$ID==i]), mean(df$rating[df$ID==i],na.rm=T), df$rating)&#xA;  df$Sur[df$ID==i] &amp;lt;- ifelse(is.na(df$Sur[df$ID==i]), mean(df$Sur[df$ID==i],na.rm=T), df$Sur)&#xA;}&#xA;&#xA; ID rating     Sur&#xA;1 101   60.0 0.76870&#xA;2 101   78.0 0.76685&#xA;3 101   69.0 0.76500&#xA;4 102   60.0 0.54450&#xA;5 102   67.5 0.76685&#xA;6 102   69.0 0.76500&#xA;7 103   69.0 0.76870&#xA;8 103   78.0 0.76685&#xA;9 103   69.0 0.72550&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23675" LastActivityDate="2016-09-20T09:31:52.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14109" PostTypeId="2" ParentId="14007" CreationDate="2016-09-20T10:36:20.490" Score="2" Body="&lt;p&gt;I thought of throwing some historical perspective into this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The initial algorithm is called &lt;strong&gt;AdaBoost&lt;/strong&gt;. Unlike most of machine learning, this algorithm has its genesis in hard &lt;em&gt;theory&lt;/em&gt;. The authors were trying to answer the following theoretical question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it possible somehow to combine weak models and create a very accurate predictor?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Weak models are models that are hardly better than chance. Think about this question. It is quite a philosophical question... And, as it happens, it can be answered by hard mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They actually demonstrated that yes. Their paper is actually worth reading. They have since published a lot of interesting papers from different perspectives, for instance using game theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The AdaBoost algorithm is &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost#Example_algorithm_.28Discrete_AdaBoost.29&quot; rel=&quot;nofollow&quot;&gt;also very simple&lt;/a&gt; and worth checking because it is the basis of xgboost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AdaBoost works with any model. The only restriction is that the model supports giving weights to each observation. This is because it trains models sequentially by increasing (or decreasing) the weight associated to each observation to make that observation more important (or less important).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cool thing is that the algorithm is very unlikely to overfit. But notice that this is an empirical claim, not a theoretical one. The algorithm &lt;em&gt;can&lt;/em&gt; overfit, but it has been found (empirically) to be very resilient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This algorithm has been expanded into &lt;strong&gt;Gradient Boosting&lt;/strong&gt;, which &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow&quot;&gt;has more flexible loss functions&lt;/a&gt;. This algorithm is an AdaBoost adaption by the same guy that invented random forests. They are not concerned about theoretical questions like the authors of AdaBoost, they just want to make the algorithm more flexible and efficient. For instance, they also update the weights using all previous weak models, not just the last trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;xgboost&lt;/strong&gt; is just an intelligent implementation of Gradient Boosting. You can see the basic algorithm &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting#Algorithm&quot; rel=&quot;nofollow&quot;&gt;in wikipedia&lt;/a&gt;. Each decision tree is trained sequentially and weights are computed based on the errors from the current ensemble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can hear you say:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What? xgboost trains trees in sequence? How is xgboost so fast then?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This website &lt;a href=&quot;http://zhanpengfang.github.io/418home.html&quot; rel=&quot;nofollow&quot;&gt;explains it very well&lt;/a&gt;. While trees are trained sequentially, each individual decision tree is trained in parallel by using highly creative techniques. Most importantly, nodes in the same depth do not depend in each other, so you can paralelize them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long story short: the loss function is applied between training models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to making predictions, then the entire ensemble can be used in parallel.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-09-20T10:36:20.490" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14110" PostTypeId="1" AcceptedAnswerId="14113" CreationDate="2016-09-20T11:26:54.070" Score="3" ViewCount="309" Body="&lt;p&gt;If the both do the same thing then which give us better accuracy?&lt;/p&gt;&#xA;" OwnerUserId="24493" LastActivityDate="2016-09-20T11:55:08.467" Title="Do logistic regression and softmax regression do the same thing?" Tags="|machine-learning|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14112" PostTypeId="1" CreationDate="2016-09-20T11:48:56.400" Score="1" ViewCount="426" Body="&lt;p&gt;Before anybody asks please let me confirm this question is related to trading technical analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is quite simple, but I lack the scientific background required to solve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data comes as an array of float values equally spaced temporally. Some local minimums (same apply for maximums) tend to repeat over time (although not at the very same value, but in a rather tight range), because of human psychology: if some price level was important in the past, stopping the price to go below, it will tend to resurface a few times, blocking again the price from going below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm interested in is to find the most significant levels over a given past period (say 5000 time intervals). Basically the &quot;importance&quot; of a level is determined by the number of times it was tested (meaning the number of times the price tried go past that level and failed); however, a level tested three times at relatively large time intervals (like once each month for a daily chart) is more significant than a level tested three times in a short period (like within a week for a daily chart).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, data is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data comes as an array of float values&lt;/li&gt;&#xA;&lt;li&gt;data points are equally spaced temporally (however, time is of little to no importance)&lt;/li&gt;&#xA;&lt;li&gt;some local extremes are more important than others and tend to repeat over time&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can somebody help me with some Python code to extract important levels? Or at least point me to some theory/frameworks to learn? I know about pandas/numpy but I lack the math background required to a least identify what could be used of these...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DWQoU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DWQoU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24494" LastEditorUserId="381" LastEditDate="2016-09-26T17:30:23.577" LastActivityDate="2016-09-26T23:42:24.997" Title="Finding levels in an array of floats" Tags="|python|programming|finance|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14113" PostTypeId="2" ParentId="14110" CreationDate="2016-09-20T11:55:08.467" Score="7" Body="&lt;p&gt;There is a key difference:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Softmax regression provides class probabilities for mutually exclusive classes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Logistic regression treats class membership for each class separately. Classes do not need to be mutually exclusive.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The two are equivalent for a scenario with two mutually exclusive classes - e.g. a &quot;positive&quot; and &quot;negative&quot; class - where softmax would have two outputs summing to 1, and logistic regression would have one output giving probability of the &quot;positive&quot; class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not certain of the results comparing logistic regression one-vs-all (taking max output) with softmax regression on the same multi-class problem. I would expect the performance to be quite similar. Neither model copes well with non-linear relationships between input and target classes.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-09-20T11:55:08.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14114" PostTypeId="2" ParentId="13645" CreationDate="2016-09-20T14:10:47.640" Score="1" Body="&lt;p&gt;I agree with Calpis: first reduce number of dimensions. But instead of PCA (which is designed for multivariate normal data) use matrix factorization. SVD or NMF.&lt;/p&gt;&#xA;" OwnerUserId="14211" LastActivityDate="2016-09-20T14:10:47.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14118" PostTypeId="1" AcceptedAnswerId="14119" CreationDate="2016-09-20T21:27:12.427" Score="4" ViewCount="9642" Body="&lt;p&gt;I wonder when to use linear regression with stochastic or batch gradient descent to minimize the cost function vs when to use normal equations? The algorithms using gradient descent are iterative, so they might take more time to run, as opposed to the normal equation solution, which is a closed form equation. But it does use matrices to store the training data. Does this mean gradient solutions require more processing power, but using the normal equation method requires more memory because of the matrices? Which method is optimal in what scenario?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2016-09-21T00:46:01.750" Title="Linear regression - LMS with gradient descent vs normal equations" Tags="|machine-learning|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14119" PostTypeId="2" ParentId="14118" CreationDate="2016-09-21T00:46:01.750" Score="11" Body="&lt;p&gt;Andrew Ng answers this question succinctly in his &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/2DKxQ/normal-equation%22%22&quot; rel=&quot;noreferrer&quot;&gt;Coursera lecture about the normal equation&lt;/a&gt;.  I will summarize.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have m training examples and n features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disadvantages of gradient descent:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you need to choose the learning rate, so you may need to run the algorithm at least a few times to figure that out.&lt;/li&gt;&#xA;&lt;li&gt;it needs many more iterations, so, that could make it slower&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Compared to the normal equation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you don't need to choose any learning rate &lt;/li&gt;&#xA;&lt;li&gt;you don't need to iterate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disadvantages of the normal equation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Normal Equation is computationally expensive when you have a very large number of features ( n features ), because you will ultimately need to take the inverse of a n x n matrix in order to solve for the parameters data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Compared to gradient descent:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it will be reasonably efficient and will do something acceptable when you have a very large number ( millions ) of features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So if n is large then use gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If n is relatively small ( on the order of a hundred ~ ten thousand ), then the normal equation&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-21T00:46:01.750" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14120" PostTypeId="2" ParentId="14078" CreationDate="2016-09-21T02:51:51.767" Score="0" Body="&lt;p&gt;I think you can put something together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking about component classifiers, overall classification is better when the decision rules of the component classifiers differ and provide complementary information.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, one methodology you might try for suggesting the number of bags is to create an arbitrary statistic that indicates which parts of your feature space is covered by a particular bag.  You could stop adding bags when you reach a satisfactory level of coverage of your entire feature space: when you have expert classifiers for each region of your feature space.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your bag size could be found heuristically by starting with a roughly equal number of patterns in each bag ( n1 ). Increase n1 to a value half way between its current value and the maximum if you find that n1 is to low.  Conversely decrease n1 to between the current value and the minimum if you find it is to high.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-21T02:51:51.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14122" PostTypeId="1" CreationDate="2016-09-21T07:55:38.103" Score="14" ViewCount="13062" Body="&lt;p&gt;The idea of applying filters to do something like identify edges, is a pretty cool idea.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you can take an image of a 7. With some filters, you can end up with transformed images that emphasize different characteristics of the original image. The original 7:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/K4WkJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K4WkJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can be experienced by the network as:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L02xC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L02xC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice how each image has extracted a different edge of the original 7.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is all great, but then, say the next layer in your network is a Max Pooling layer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, generally, doesn't this seem a little bit like overkill? We just were very careful and deliberate with identifying edges using filters -- now, we no longer care about any of that, since we've blasted the hell out of the pixel values! Please correct me if I'm wrong, but we went from 25 X 25 to 2 X 2! Why not just go straight to Max Pooling then, won't we end up with basically the same thing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an extension the my question, I can't help but wonder what would happen if, coincidentally, each of the 4 squares all just happen to have a pixel with the same max value. Surely this isn't a rare case, right? Suddenly all your training images look the exact same.&lt;/p&gt;&#xA;" OwnerUserId="15533" LastEditorUserId="15533" LastEditDate="2016-09-21T08:20:39.300" LastActivityDate="2021-03-13T20:01:38.280" Title="Why convolute if Max Pooling is just going to downsample the image anyway?" Tags="|neural-network|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14123" PostTypeId="2" ParentId="14102" CreationDate="2016-09-21T08:06:54.730" Score="5" Body="&lt;p&gt;Although I haven't verified it, a first glance at the features and training set you used shows an obvious problem. You have just two data samples with the exact same features while you give it different labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although other types of models might not break and just give both samples an equal probability of being 0 or 1, something in the Naive Bayes classifier internal calculations probably requires there to be at least some difference in differently labeled samples (not an unreasonable assumption). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't confirm this for myself as I'd have to go deeper into the source code to actually check the scikit implementation so perhaps someone else more familiar with it can check.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I just tested this out by changing one line in your code to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;features = np.array([[-1, 1],[-1, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It worked.&lt;/p&gt;&#xA;" OwnerUserId="24412" LastEditorUserId="24412" LastEditDate="2017-07-04T03:49:32.720" LastActivityDate="2017-07-04T03:49:32.720" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14124" PostTypeId="1" CreationDate="2016-09-21T08:08:19.270" Score="15" ViewCount="6506" Body="&lt;p&gt;I have a dataset including a set of customers in different cities of California, time of calling for each customer, and the status of call (True if customer answers the call and False if customer does not answer). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to find an appropriate time of calling for future customers such that the probability of answering the call is high. So, what is the best strategy for this problem? Should I consider it as a classification problem which the hours (0,1,2,... 23) are the classes? Or should I consider it as a regression task which the time is a continuous variable? How can I make sure that the probability of answering the call will be high?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated. It also would be great if you refer me to similar problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a snapshot of the data. &lt;img src=&quot;https://i.stack.imgur.com/geF7x.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="24526" LastEditorUserId="21" LastEditDate="2016-09-22T12:35:08.090" LastActivityDate="2021-03-12T18:01:35.083" Title="Predict the best time of call" Tags="|machine-learning|classification|regression|scikit-learn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14125" PostTypeId="1" CreationDate="2016-09-21T08:30:26.603" Score="0" ViewCount="1623" Body="&lt;p&gt;I'm reading up on how to clean/munge/wrangle data sets in order to run machine learning algorithms on them. Lots of info on how to do the actual wrangling, but a practical detail seems to be glossed over: storage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is quite simple: &lt;strong&gt;which is the go-to technology to store/retrieve a large data set&lt;/strong&gt; in order to run algorithms on it in the most convenient/efficient way possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing the language in which the algorithms are written is not all that relevant here.&lt;/p&gt;&#xA;" OwnerUserId="24489" LastActivityDate="2016-09-21T12:54:36.550" Title="Storing large data sets for python machine learning algorithm consumption" Tags="|machine-learning|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14126" PostTypeId="2" ParentId="11134" CreationDate="2016-09-21T08:37:35.810" Score="8" Body="&lt;p&gt;You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;noreferrer&quot;&gt;Doc2vec&lt;/a&gt; is one way to do this (each sentence would be a &quot;document&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to use Doc2vec, one way to create the sentence vectors would be to average the word vectors for each sentence, giving you a single vector of the same width for each sentence. This may not be as precise as some methods available through Doc2Vec but I have used it with considerable success for topic modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way once you have your sentence vectors, line them up in sequence for each document like you're already doing for your word vectors and run then through your model. Because the sequence length for each document is shorter, your model should train more quickly than with word vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, this method could work when scaled up or down to meet your accuracy and speed needs. (e.g. if your CNN still trains too slowly with sentence vectors, you can create paragraph vectors instead).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to handle documents of different length is through padding. Your document sequences should all be equal in length to your longest document. So if your longest document is 400 sentences then all document sequences will be 400 vectors in length. Documents shorter than the max length would be padded with vectors filled with zeros.&lt;/p&gt;&#xA;" OwnerUserId="23230" LastActivityDate="2016-09-21T08:37:35.810" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14127" PostTypeId="1" AcceptedAnswerId="14138" CreationDate="2016-09-21T08:53:52.417" Score="2" ViewCount="638" Body="&lt;p&gt;Could any one help me know about different approaches, methods or algorithms to build a model only with positive responses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume we have a set of customers with a 'positive' behaviour. We want to find customers with a similar profile in the database and that are more likely to have this positive behaviour in the future. This can't be modeled with 0/1 target variable because there's not a clear definition of a 'negative' behaviour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like a look-alike modeling. However I think that look-alike modeling assumes knowing the response (0/1) for the given set of people (treatment group) and finding similar people in the control group to predict their responses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance of your response!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MK&lt;/p&gt;&#xA;" OwnerUserId="20392" LastActivityDate="2016-09-21T11:50:53.640" Title="model with only positive responses" Tags="|machine-learning|classification|clustering|predictive-modeling|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14128" PostTypeId="1" CreationDate="2016-09-21T09:07:36.787" Score="1" ViewCount="76" Body="&lt;p&gt;I am developing a predictive model in sports vertical. As the game will progress, my model will predict the &lt;strong&gt;winning probability&lt;/strong&gt; of each team playing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I am facing is to what metric would be appropriate to quantify accuracy for the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have in mind to quantify accuracy is as follows :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider &lt;code&gt;n&lt;/code&gt; matches for which my model predicts the winning probability of Team A at time instance &lt;code&gt;t1, t2 ... tn&lt;/code&gt;. For each match, I will quantify the loss as the hinge loss at that time. i.e. if &lt;strong&gt;probability&lt;/strong&gt; greater than &lt;strong&gt;threshold&lt;/strong&gt; and team_A actually won, then accurate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the game progresses, model gets new info and I can expect the model performance to improve, so I thought about giving weights to loss function acc. to &lt;strong&gt;Time of prediction&lt;/strong&gt; also. i.e. if model predicts wrongly at la&lt;strong&gt;ter point of the game&lt;/strong&gt; then it should be &lt;strong&gt;penalized heavily&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think it can be better. Any leads on the apt. loss function for this is highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hoping to learn something great from the answers.  &lt;/p&gt;&#xA;" OwnerUserId="24530" LastEditorUserId="21024" LastEditDate="2016-09-21T12:56:58.643" LastActivityDate="2016-09-21T12:56:58.643" Title="Right choice of accuracy metric or loss function" Tags="|machine-learning|deep-learning|loss-function|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14129" PostTypeId="2" ParentId="14122" CreationDate="2016-09-21T09:24:21.663" Score="6" Body="&lt;p&gt;We cannot go directly from input layer to max pooling because of the &lt;code&gt;convolution layer&lt;/code&gt; in between. The reason for convolution is to extract features. Max pooling down-samples the features that have been extracted. If you think there are features which are missing because of the direct jump from a large matrix to a max pooling layer, you can add more layers of convolution in between till you seem satisfied with a size and then do max pooling onto it so that it is not an overkill.&lt;/p&gt;&#xA;&lt;p&gt;Max pooling, which is a form of down-sampling is used to identify the most important features. But average pooling and various other techniques can also be used.&#xA;I normally work with text and not images. For me, the values are not normally all same. But if they are too, it wouldn't make much difference because it just picks the largest value.&lt;/p&gt;&#xA;&lt;p&gt;A very good understanding from &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;wiki&lt;/a&gt; -&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The intuition is that once a feature has been found, its exact&#xA;location isn't as important as its rough location relative to other&#xA;features. The function of the pooling layer is to progressively reduce&#xA;the spatial size of the representation to reduce the amount of&#xA;parameters and computation in the network, and hence to also control&#xA;overfitting. It is common to periodically insert a pooling layer&#xA;in-between successive conv layers in a CNN architecture. The pooling&#xA;operation provides a form of translation invariance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="29169" LastEditDate="2021-03-13T20:01:38.280" LastActivityDate="2021-03-13T20:01:38.280" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="14131" PostTypeId="2" ParentId="14125" CreationDate="2016-09-21T09:53:02.903" Score="1" Body="&lt;p&gt;There are a lot of technologies out there to handle. These are the most popular ones I know of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For moderate data to operate in chunks- Pandas -  pd.read_csv('train.csv', chunksize=chunksize)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For larger data - dask, Hadoop with R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer &lt;a href=&quot;https://stackoverflow.com/questions/11055502/recommended-package-for-very-large-dataset-processing-and-machine-learning-in-r&quot;&gt;this&lt;/a&gt; for various other suggestions.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-09-21T09:53:02.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14132" PostTypeId="2" ParentId="14124" CreationDate="2016-09-21T10:03:19.133" Score="4" Body="&lt;p&gt;You could try the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Divide up the day into various parts - early-morning, morning, noon, afternoon, evening, late evening, night, etc.&lt;/li&gt;&#xA;&lt;li&gt;Assign time boundaries to each part of the day, e.g. noon could be 12 pm to 1 pm.&lt;/li&gt;&#xA;&lt;li&gt;Create 3 new labels - &quot;part of the day to call the customer&quot;, for each positive case (status of call=true) assign it the corresponding label (morning/noon/evening). These labels will be in one-hot encoded format e.g. prefer_morning=0/1, prefer_noon, prefer_evening, etc. &lt;/li&gt;&#xA;&lt;li&gt;Build 3 models to predict whether the lead prefers morning/noon/or evening time of the day for a call to be successful.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Additionally, I recommend adding additional features such as occupation, gender, etc. since the features listed in the table (city, etc.) are too ambiguous and do not give much information to differentiate among customers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDITED as per suggestion in comments:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using the model, each lead would get classified as prefers_morning=yes/no, prefers_noon=yes/no and prefers_evening=yes/no. Based on the time of the day, for example in the morning, the call center agent (or software) could pick up and call leads classified in the morning preference set. When its noon, the call software picks up form the noon preferred list, and so on.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2016-11-30T05:46:27.243" LastActivityDate="2016-11-30T05:46:27.243" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14133" PostTypeId="2" ParentId="14122" CreationDate="2016-09-21T11:00:04.463" Score="17" Body="&lt;p&gt;Max pooling doesn't down-sample the image. It down-samples the features (such as edges) that you have just extracted. Which means you get more approximately where those edges or other features are. Often this is just what the network needs for generalisation - in order to classify it doesn't need to know there is a vertical edge running from 10,5 to 10,20, but that there is an approximately vertical edge about 1/3 from left edge about 2/3 height of the image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These rougher categories of features inherently cover more variations in the input image for very little cost, and the reduction in size of the feature map is a nice side effect too, making the network faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this to work well, you still need to extract features to start with, which max pooling does not do, so the convolutional layer is necessary. You should find you can down-sample the original image (to 14x14) instead of using the first max-pooling layer, and you will still get pretty reasonable accuracy. How much pooling to do, and where to add those layers is yet another hyper-parameter problem when building a deep neural network.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-09-21T11:00:04.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14134" PostTypeId="2" ParentId="155" CreationDate="2016-09-21T11:10:08.190" Score="3" Body="&lt;p&gt;3 datasets from &lt;a href=&quot;https://www.jc-bingo.com/about&quot; rel=&quot;nofollow&quot;&gt;https://www.jc-bingo.com/about&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;visitor-interests.csv Aggregated visitor interests compiled based on&#xA;1 week web access logs. Includes visitor IP address, user-agent&#xA;string, visitor country, accessed page languages and topics. 19,926&#xA;records, 2.9 Mb.&lt;/li&gt;&#xA;&lt;li&gt;user-agents.csv Real visitor user agents ordered by popularity. 4,826 records, 716 Kb.&lt;/li&gt;&#xA;&lt;li&gt;bots.csv Robot IP addresses and user-agent strings extracted from web access logs. 1,293 records, 122 Kb.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="24534" LastActivityDate="2016-09-21T11:10:08.190" CommentCount="0" CommunityOwnedDate="2016-09-21T11:10:08.190" ContentLicense="CC BY-SA 3.0" />
  <row Id="14135" PostTypeId="1" CreationDate="2016-09-21T11:14:27.000" Score="-1" ViewCount="38" Body="&lt;p&gt;There is continuous event, for example someone is inside the room. He can walk in once a day for 10 hours, or 10 times a day for 30 min. About this event, we get one time reports. There is always report when the person enters the room, leaves and but also multiple reports when he is in the room. These reports are indistinguishable. We can also ask the person &quot;are you in the room&quot; and get sure answer, but we don’t want to do it too often to not get him irritated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the question is:&#xA;Now we want to know, based on these reports, is the person in the room? How to model something like this, to minimize number of manual questions asked? &lt;/p&gt;&#xA;" OwnerUserId="24536" LastEditorUserId="924" LastEditDate="2016-09-21T19:46:35.920" LastActivityDate="2016-09-21T19:46:35.920" Title="Predicting continuous event based on on time reports about this event" Tags="|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14136" PostTypeId="1" CreationDate="2016-09-21T11:16:29.840" Score="1" ViewCount="1172" Body="&lt;p&gt;I am working in emotion analysis of tweets. I collected close to .6 million tweets. I dont want to label them manually, instead i coded a bunch of complex rules to arrive at their label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I use these algo generated labels for my ML solutions, I want to check/analyze how good is this labeling. Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code. &lt;/p&gt;&#xA;" OwnerUserId="13457" LastActivityDate="2016-09-21T12:32:23.100" Title="Labeling Dataset Algorithmically" Tags="|machine-learning|labels|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14137" PostTypeId="2" ParentId="14136" CreationDate="2016-09-21T11:37:52.380" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. You can treat your automated labelling as a model in its own right (it essentially is an &quot;expert&quot; model, that takes in additional features). Collect ground truth data with known accurate labels, and use a metric such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Accuracy_and_precision&quot; rel=&quot;nofollow&quot;&gt;accuracy&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow&quot;&gt;AUROC&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow&quot;&gt;F1 score&lt;/a&gt; to determine how well your expert model works. Usually you should pick one metric that makes sense to you based on the problem you are trying to solve. Most statistics libraries will allow you to assess these metrics on arbitrary data, so it should be possible to find the relevant part of the API and feed it your synthetic labels to compare with ground truth data you have collected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But wait! That is exactly what you would need to do in order to train and assess an ML model. About the best you can hope is that you will need less ground truth data to assess your expert than you would to train the ML version from scratch. For complex ML models such as deep neural networks, this could well be true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you may hit a limitation from this approach. Say that you assess from ground truth data that your expert is 85% accurate. You will not be able to use it to train an ML solution that is better than 85% accurate, because the ML will learn to copy the expert's predictions as best it can.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing to watch out for - if you analyse why the expert got some values wrong and then adjust it to give correct answers for those labels, you will need to remove the labels you used to do this from your test data - and ideally collect more, or take more from some unused pool, to replace them. Otherwise you will fit your expert to the test data and get incorrect reporting of how well the expert generalises to unseen data. Same reasoning means you should not use tweets and labels that you analysed when developing your expert model in order to test it.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-09-21T12:32:23.100" LastActivityDate="2016-09-21T12:32:23.100" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14138" PostTypeId="2" ParentId="14127" CreationDate="2016-09-21T11:50:53.640" Score="1" Body="&lt;p&gt;This sounds like a &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-class or unary&lt;/a&gt; classification problem where you can build a &lt;em&gt;model of normality&lt;/em&gt; using the class that you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;em&gt;novelty and outlier detection&lt;/em&gt; documentation for &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit learn&lt;/a&gt;. They discuss the one-class SVM there, which attempts to model the decision boundary given the observations that you have. The example that is implemented is pretty straight forward. Sadly, this implementation does not provide you with a probabilistic measure (although, there are &lt;a href=&quot;http://ieeexplore.ieee.org/document/6786486/&quot; rel=&quot;nofollow noreferrer&quot;&gt;papers&lt;/a&gt; on how to add this functionality).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can consider an auto-associative neural network (replicator network). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a nice discussion on time-series anomaly detection &lt;a href=&quot;https://datascience.stackexchange.com/questions/6547/open-source-anomaly-detection-in-python&quot;&gt;here&lt;/a&gt; on DS.&lt;/p&gt;&#xA;" OwnerUserId="13821" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-09-21T11:50:53.640" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14140" PostTypeId="2" ParentId="14125" CreationDate="2016-09-21T12:54:36.550" Score="1" Body="&lt;p&gt;Currently &lt;code&gt;Apache Hadoop&lt;/code&gt; is one of the popular technologies to store data and &lt;code&gt;Apache Spark&lt;/code&gt; is a very popular computational engine to compute/munge large datasets.&lt;/p&gt;&#xA;" OwnerUserId="16971" LastActivityDate="2016-09-21T12:54:36.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14141" PostTypeId="2" ParentId="14135" CreationDate="2016-09-21T13:26:08.203" Score="0" Body="&lt;p&gt;You should develop your algorithm based on the reports; then only query the person when the algorithm returns its least confident results.  Because you would expect that the most valuable patterns would be near the decision boundaries.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-21T13:26:08.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14142" PostTypeId="1" CreationDate="2016-09-21T14:53:26.113" Score="1" ViewCount="10069" Body="&lt;p&gt;The Hopkins statistic, is a statistic which gives a value which indicates the cluster tendency, in other words: how well the data can be clustered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the value is between {0.01, ...,0.3}, the data is regularly spaced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the value is around 0.5, it is random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the value is between {0.7, ..., 0.99}, it has a high tendency to cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question about my implementation of the Hopkins statistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it correct? If so, other people can use it :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X is the data with shape (n,m).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = len(vars) # columns&#xA;n = len(X) # rows&#xA;m = int(0.1 * n) # heuristic from article [1]&#xA;&#xA;from sklearn.neighbors import NearestNeighbors&#xA;nbrs = NearestNeighbors(n_neighbors=1, algorithm='brute').fit(X)&#xA;&#xA;from random import sample&#xA;rand_X = sample(range(0, n, 1), m)&#xA;&#xA;ujd = []&#xA;wjd = []&#xA;for j in range(0, m):&#xA;    u_dist, _ = nbrs.kneighbors(np.random.normal(size=(1, d)).reshape(1, -1), 2, return_distance=True)&#xA;    ujd.append(u_dist[0][1])&#xA;    w_dist, _ = nbrs.kneighbors(X[rand_X[j]].reshape(1, -1), 2, return_distance=True)&#xA;    wjd.append(w_dist[0][1])&#xA;&#xA;H = sum(ujd) / (sum(ujd) + sum(wjd))&#xA;print H&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any recommendations are much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Validating Clusters using the Hopkins Statistic from IEEE 2004.&lt;/p&gt;&#xA;" OwnerUserId="21503" LastActivityDate="2023-05-15T07:04:45.320" Title="Cluster tendency using Hopkins statistic implementation in Python" Tags="|machine-learning|python|clustering|statistics|unsupervised-learning|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14143" PostTypeId="2" ParentId="13229" CreationDate="2016-09-21T15:54:49.040" Score="0" Body="&lt;p&gt;Here is how I would think about this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I select my features first, I then need a large enough subsample so that the variance of the model is not raised counter productively.  That raises a few important questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can I get a large enough subsample that meets my unbiased criteria?&lt;/li&gt;&#xA;&lt;li&gt;Can I get a small enough subsample that meets my efficiency constraints?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If I select my subsamples first, I then need features that are useful for this subsample and I ultimately need to cover the whole feature space.  That raises a few questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can I get an appropriately sized feature space for my subsample size?  At the very least I would want a feature space that is an expert in some specific region, with the intent to cover the whole feature space with subsequent subsamples and feature selection.&lt;/li&gt;&#xA;&lt;li&gt;How many subsamples can I process while staying in my efficiency constraints?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would choose my methodology based on the positively answered questions above.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-21T15:54:49.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14144" PostTypeId="2" ParentId="14002" CreationDate="2016-09-21T16:37:23.447" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a standard approach to evaluating [the] reasonableness of a new implementation of time series analytics?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, there is. A way to validate your &lt;em&gt;build-from-scratch&lt;/em&gt; is to &lt;strong&gt;simulate&lt;/strong&gt; an ARIMAX timeseries from a data generating process (DGP) of your choice (see e.g. &lt;a href=&quot;http://robjhyndman.com/hyndsight/arimax/&quot; rel=&quot;nofollow&quot;&gt;http://robjhyndman.com/hyndsight/arimax/&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Call your choice of DGP-parameters $\theta$, then: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Draw a sample from the DGP. &lt;/li&gt;&#xA;&lt;li&gt;Estimate the model's parameters: $\hat{\theta}$. Make forecasts: $\hat{y}_{T+h|T}$.&lt;/li&gt;&#xA;&lt;li&gt;Assess $|| \hat{\theta} - \theta||_{\text{a metric}}$ and $||\hat{y}_{T+h|T} - y_{T+h}||_{\text{a metric}}$. &lt;/li&gt;&#xA;&lt;li&gt;Repeat &lt;code&gt;1 to 3&lt;/code&gt; a couple of times, also for different DGPs and sample sizes.&lt;/li&gt;&#xA;&lt;li&gt;You either increase your confidence in the build, or find weak points that need improvement.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A sensible metric could be squared loss. Personally, I like to use the &quot;eye-ball metric&quot; for the forecast residuals, their distribution could be assessed by way of histogram.&lt;/p&gt;&#xA;" OwnerUserId="24550" LastActivityDate="2016-09-21T16:37:23.447" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14145" PostTypeId="1" AcceptedAnswerId="14158" CreationDate="2016-09-21T16:55:06.187" Score="2" ViewCount="1520" Body="&lt;p&gt;when training a VAE, typically one samples from the latent distribution using the reparametrization trick using a fairly large minibatch size (&gt;100) in the decoder/generator half of the VAE. I'm assuming this minibatch size allows the network to 'smooth' out the error and allows us to avoid having to repeatedly sample from the latent space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm interested in online scenarios where you are training the VAE on streaming data as it arrives, so the batch size would be 1. In this case, it can take the VAE a long time to converge because the error is highly volatile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any way to avoid this issue in practice?&lt;/strong&gt; I am unsure what will happen if I have to repeatedly sample from the latent distribution and then take the mean of those samples (or something) - aside from obvious performance concerns. The other alternative is to wait for enough samples to arrive that I can train them in a larger batch, but even in this case I wouldn't be able to wait for 100+ samples to arrive.&lt;/p&gt;&#xA;" OwnerUserId="23920" LastEditorUserId="23920" LastEditDate="2016-09-21T17:04:22.830" LastActivityDate="2016-09-22T12:37:28.120" Title="Online Variational Autoencoder" Tags="|machine-learning|unsupervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14146" PostTypeId="1" CreationDate="2016-09-21T19:31:50.930" Score="1" ViewCount="1737" Body="&lt;p&gt;I have a list of E-mails, and I'm trying to find &amp;quot;similar&amp;quot; E-mails.&lt;/p&gt;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;DanB@gmail.com&lt;/p&gt;&#xA;&lt;p&gt;DanB1@gmail.com&lt;/p&gt;&#xA;&lt;p&gt;Dan12B@yahoo.com&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;will be &amp;quot;similar&amp;quot; for me.&lt;/p&gt;&#xA;&lt;p&gt;I'll be happy to hear if you are fimiliar with a method which can be applied here.&lt;/p&gt;&#xA;&lt;p&gt;P.S. my available tools are Teradata (SQL) or R.&lt;/p&gt;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="20594" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-09-21T21:11:24.923" Title="Text similarities" Tags="|r|text-mining|text|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14148" PostTypeId="2" ParentId="14146" CreationDate="2016-09-21T20:41:41.003" Score="3" Body="&lt;p&gt;Since your question is super-huge and no one is going to do this job for you, here's a starting point for you to at least let you see how the various string comparison algorithms work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(stringdist)&#xA;library(stringi)&#xA;library(purrr)&#xA;&#xA;emails &amp;lt;- c(&quot;ed@microsoft.com&quot;, &quot;DanB@gmail.com&quot;, &quot;DanB1@gmail.com&quot;, &quot;Dan12B@yahoo.com&quot;,&#xA;            &quot;fred@outlook.com&quot;, &quot;bill@juno.com&quot;, &quot;Danb21@aol.com&quot;, &quot;ed@tumblr.com&quot;)&#xA;&#xA;naive_regex &amp;lt;- stri_match_all_regex(emails, &quot;([[:print:]]+)@(.*)&quot;)&#xA;&#xA;addressees &amp;lt;- map_chr(naive_regex, 2)&#xA;&#xA;methods &amp;lt;- c(&quot;osa&quot;, &quot;lv&quot;, &quot;dl&quot;, &quot;hamming&quot;, &quot;lcs&quot;, &quot;qgram&quot;, &quot;cosine&quot;, &quot;jaccard&quot;, &quot;jw&quot;, &quot;soundex&quot;)&#xA;&#xA;map(seq_along(addressees), function(i) {&#xA;  map_df(methods, function(method) {&#xA;    c(addressees[i] ,method, stringdist(addressees[i], addressees, method=method)) %&amp;gt;% &#xA;      as.list() %&amp;gt;% &#xA;      setNames(c(&quot;looking_for&quot;, &quot;method&quot;, addressees))&#xA;  })&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is too long to put here but it's a list of comparison matrices like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## [[2]]&#xA;## # A tibble: 10 × 9&#xA;##    looking_for  method    ed  DanB              DanB1            Dan12B  fred  bill            Danb21&#xA;##          &amp;lt;chr&amp;gt;   &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt;              &amp;lt;chr&amp;gt;             &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt; &amp;lt;chr&amp;gt;             &amp;lt;chr&amp;gt;&#xA;## 1         DanB     osa     4     0                  1                 2     4     4                 3&#xA;## 2         DanB      lv     4     0                  1                 2     4     4                 3&#xA;## 3         DanB      dl     4     0                  1                 2     4     4                 3&#xA;## 4         DanB hamming   Inf     0                Inf               Inf     4     4               Inf&#xA;## 5         DanB     lcs     6     0                  1                 2     8     8                 4&#xA;## 6         DanB   qgram     6     0                  1                 2     8     8                 4&#xA;## 7         DanB  cosine     1     0  0.105572809000084 0.183503419072274     1     1 0.387627564304205&#xA;## 8         DanB jaccard     1     0                0.2 0.333333333333333     1     1 0.571428571428571&#xA;## 9         DanB      jw     1     0 0.0666666666666668 0.111111111111111     1     1              0.25&#xA;## 10        DanB soundex     1     0                  0                 0     1     1                 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once you have an idea of acceptable tolerances, you can then create an algorithm to match =&gt; remove / match =&gt; remove, etc.&lt;/p&gt;&#xA;" OwnerUserId="710" LastActivityDate="2016-09-21T20:41:41.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14149" PostTypeId="2" ParentId="14146" CreationDate="2016-09-21T21:11:24.923" Score="1" Body="&lt;p&gt;You could try a very simple algorithm: &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Levenshtein_distance&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/stringdist/stringdist.pdf&quot; rel=&quot;nofollow&quot;&gt;https://cran.r-project.org/web/packages/stringdist/stringdist.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After splitting the string by the '@'&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-09-21T21:11:24.923" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14150" PostTypeId="2" ParentId="14092" CreationDate="2016-09-21T22:01:31.013" Score="3" Body="&lt;p&gt;Since you're using Python and scikit-learn you could have a look at one of the following online learning algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;sklearn.naive_bayes.MultinomialNB&lt;/li&gt;&#xA;&lt;li&gt;sklearn.naive_bayes.BernoulliNB&lt;/li&gt;&#xA;&lt;li&gt;sklearn.linear_model.Perceptron&lt;/li&gt;&#xA;&lt;li&gt;sklearn.linear_model.PassiveAggressiveClassifier&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;sklearn.linear_model.SGDClassifier&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;sklearn.linear_model.PassiveAggressiveRegressor&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;sklearn.linear_model.SGDRegressor&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All of these &lt;a href=&quot;https://en.wikipedia.org/wiki/Online_machine_learning#Stochastic_gradient_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;online learning&lt;/a&gt; algorithms (in particular with SGD) allow for streaming the data through memory one entry at a time. Your memory would be more than sufficient for this approach. In scikit-learn this is implemented via the &lt;a href=&quot;https://stackoverflow.com/questions/24617356/sklearn-sgdclassifier-partial-fit&quot;&gt;partial_fit()&lt;/a&gt; method. More on this &lt;a href=&quot;http://scikit-learn.org/0.17/modules/scaling_strategies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;out-of-core approach&lt;/a&gt; can be found in the scikit-learn user guide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want stick to tree based algorithms, you can have a look at the xgboost package which also allows for streaming data through memory. However, this approach is a little more involved because it only accepts data in the LIBSVM format in order to parse it in the memory cache preserved for &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost&lt;/a&gt;. Also, it doesn't allow for parameter tuning, since xgboost works on numpy objects and converting from LIBSVM to numpy dumps the data from the cache to the main memory and therefore doesn't scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use the &lt;a href=&quot;https://community.cloud.databricks.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Databricks Community Edition&lt;/a&gt;&#xA;which let's you spin up Spark clusters (of limitied size for the &lt;strong&gt;free&lt;/strong&gt; edition) where you can run pyspark or plain python scripts.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-09T01:07:13.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14152" PostTypeId="1" CreationDate="2016-09-22T04:32:54.260" Score="4" ViewCount="8651" Body="&lt;p&gt;I am dealing with a dataset that has a total of 81 records. Out of them I divided it to 54 records for the training and the rest for testing. I noticed a few things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Accuracy is low for given test data set (~ 30%).&lt;/li&gt;&#xA;&lt;li&gt;Accuracy remains same no matter whether I use SVM or Naive Bayes.&lt;/li&gt;&#xA;&lt;li&gt;If I test against 2-3 values accuracy shoots up to 75%.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is this normal or I am doing something wrong?&lt;/p&gt;&#xA;" OwnerUserId="10879" LastEditorUserId="29169" LastEditDate="2019-06-08T03:10:06.997" LastActivityDate="2019-06-08T03:10:06.997" Title="Does dataset training and test size affect algorithm?" Tags="|machine-learning|svm|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14153" PostTypeId="2" ParentId="14152" CreationDate="2016-09-22T07:01:38.100" Score="6" Body="&lt;p&gt;It is completely normal in some circumstances. If you consider the learning problem from a statistical perspective, learning is done by trying to estimate the conditional estimate of your output variable given input variables. When you are doing learning you basically have to main components: sample data from training and the learning algorithm, and you use both to build an estimation of the aforementioned conditional probability. Now consider the relation of each component with the desired estimated probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Training sample&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training data is only a sample from all the possible data. You do not have all the possible data since if you would have it you would not need to do learning, right? So a sample is just a fraction of the data. This sample could cover more or less the whole domain of the possible data. An immediate thought is that a bigger sample is better than a smaller one. As an extreme case to illustrate the idea is to have only one observation. But the size of the sample is not the whole story. It is possible that the probability you want to estimate is very complex. The complexity of that space requires also more data than a simple probability space. To give an example consider the problem of learning the mean of the heights of males from USA. Having a small random sample often would be enough if you assume a normal distribution. One the other hand take the example of learning to recognize speech. Since pronouncing is not the same against individuals you would need a pretty big sample to unveil the useful relation needed to predict and throw away things which are not useful in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model/learning algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not any model is appropriate to express any true model of the signal you learn. For example if your true signal resembles a second degree polynomial would be impossible to fit properly with a linear model, other than some particular cases. So there is a complexity in the model too, which could be compared with the complexity of the conditional probability you want to estimate during learning. Even if you know the true model and you learning model is able to describe it, sometimes you need more data to make the model able to fit. As another example consider a random forest which estimates a simple line. Random forest is very flexible since the underlying statistical model is basically a constant given a local region in the input space. Considering that, even if the model would be capable of estimating the true model, since the model is a local approximation, you need plenty of data in every useful region of the space, to learn a good approximation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We considered the individual component comparisons with probability you want to estimate. But you have both. This makes even harder to understand what is happening behind the walls of learning. Thus there are no universal ways of telling when data is enough for a given model. So this is why is often the case that one algorithm can have different accuracies for different training set size and for different algorithms the required training set size to have approximate equal accuracies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I tried often in practice is given a learning model to check how accuracy evolves as the learning sample size increase. If your training sample is large enough you have typically a region in a small sample size where error is large due to not enough data. After that you would have a region where errors stays approximately the same and this perhaps is where your model do it's best. Also is possible for even larger sample sizes to have a decrease in performance for some models, maybe do to inability of the model to describe what is there or due to overfitting or some other cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can trace this kind of behavior using some sort of bootstrap validation, where the sample used for training is taken repeatedly for increasing lengths. Something like 20 samples with 5% size, 20 samples with 10% size, and so on. This kind of information can give you an idea between the relation of the sample size with your learning problem for a given algorithm.&lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="108" LastEditDate="2016-09-23T07:06:41.623" LastActivityDate="2016-09-23T07:06:41.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14154" PostTypeId="1" CreationDate="2016-09-22T08:34:17.947" Score="3" ViewCount="553" Body="&lt;p&gt;I am trying to perform a hashing trick and then a random forest with scala. I have the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val documents: RDD[Seq[String]] = sc.textFile(&quot;hdfs:///tmp/new_cromosoma12v2.csv&quot;).map(_.split(&quot;,&quot;).toSeq)&#xA;&#xA;val hashingTF = new HashingTF()&#xA;val tf: RDD[Vector] = hashingTF.transform(documents)&#xA;&#xA;val splits = tf.randomSplit(Array(0.7, 0.3))&#xA;val (trainingData, testData) = (splits(0), splits(1))&#xA;&#xA;val numClasses = 3&#xA;val categoricalFeaturesInfo = Map[Int, Int]()&#xA;val numTrees = 10 &#xA;val featureSubsetStrategy = &quot;auto&quot; &#xA;val impurity = &quot;gini&quot;&#xA;val maxDepth = 8&#xA;val maxBins = 32&#xA;&#xA;**val trainingData2=LabeledPoint(1.0,trainingData.collect())**&#xA;&#xA;val model = RandomForest.trainClassifier(trainingData2, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I have the error &lt;/p&gt;&#xA;&#xA;&lt;p&gt;found   : Array[org.apache.spark.mllib.linalg.Vector]&#xA; required: org.apache.spark.mllib.linalg.Vector&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in the bold line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know how I can solve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Laia&lt;/p&gt;&#xA;" OwnerUserId="8710" LastEditorUserId="676" LastEditDate="2016-09-22T14:38:52.017" LastActivityDate="2016-10-10T12:57:37.750" Title="Hashing trick with random forest in scala" Tags="|random-forest|apache-spark|scala|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14155" PostTypeId="2" ParentId="14053" CreationDate="2016-09-22T09:41:08.770" Score="2" Body="&lt;p&gt;Hope this may help you&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(iris$Petal.Width,labels=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vCkxv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vCkxv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23675" LastActivityDate="2016-09-22T09:41:08.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14156" PostTypeId="1" AcceptedAnswerId="14814" CreationDate="2016-09-22T10:18:02.220" Score="2" ViewCount="899" Body="&lt;p&gt;I'm trying to create a classifier in which there is less &quot;manual&quot; work for the user. For less manual work I mean that &lt;em&gt;there won't be an initial phase of manual labeling of a training set&lt;/em&gt;, like in Machine Learning (supervised)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is composed by instances that are really different by class. They are documents in which there are orders for specified products for different clients. And every client got its template.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Client A]&#xA;Image&#xA;Date: xxx  Order: &#xA;Products:&#xA;Table&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Client B]&#xA;Date: xxx&#xA;Order&#xA;Image&#xA;Products:&#xA;table&#xA;Image&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I'm doing the classification doing a simple check on every document, for the presence of a specified feature, that's manually identified by a user (by area and using edit distance)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The classes are really different&lt;/em&gt; (in some cases), and trying an unsupervised classifier like an &lt;em&gt;agglomerative clustering&lt;/em&gt; the classes are split really well. After that, using measures like &lt;em&gt;TF/ICF&lt;/em&gt; often the features (in my case I use tokenized and normalized text as features) which got the greater values are the ones that are used in my manual classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The criterions that I use for stop the clustering iteration are different ( I got different configuration) like max distance, or max number of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, I thought that when the clusters will be created an user at the end will label each cluster identifying the class by the best TF/ICF (term frequency, inverse cluster frequency) features found in each cluster. And after that the clusters will be used like &quot;classifier&quot;. I know that this approach will lead to worse classification, but It's not a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that when two classes are really similar (I got classes in which the difference is only the customer code, for example) they are really difficult to split. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea on how approach this problem?&#xA;And, there is a way in which my algorithm can find out if there is a &quot;new class&quot; in the flux?&lt;/p&gt;&#xA;" OwnerUserId="15883" LastEditorUserId="15883" LastEditDate="2016-09-22T10:32:53.037" LastActivityDate="2016-10-29T11:57:25.960" Title="Unsupervised Classification for documents" Tags="|classification|clustering|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14158" PostTypeId="2" ParentId="14145" CreationDate="2016-09-22T12:37:28.120" Score="1" Body="&lt;p&gt;The fact that pieces of data arrive one by one does not define the minibatch size. You can have a buffer of size N (with either FIFO or with any other eviction policy that suits the statistical properties needed) and sample minibatches of size M out it every time you want to update the autoencoder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the buffer eviction policy and the sampling strategy, this may also help to avoid local autocorrelation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: this answer was originally a comment to the OP's question.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-09-22T12:37:28.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14159" PostTypeId="2" ParentId="12190" CreationDate="2016-09-22T14:55:06.843" Score="3" Body="&lt;p&gt;Your solution seems interesting. However, it implies that the customers’ value is assessed through whether they are active or not in next period. It might be interesting to look at the purchase by customer for, say future 8 weeks (or future 6 weeks or 12 weeks for that matter – the time frame depends a lot on the particularity of the business). This measurement reflects whether customers are active or not in the next period and also quantifies their 'activity’. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a somewhat similar project I implemented the following solution:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Start by randomly selecting one breakpoint for each of the three dimensions (RFM); by breakpoint I mean a random value among the unique values of Recency, another randomly selected value for Frequency, and another one for Monetary.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Draw some initial segments based on these three breakpoints by selecting groups of customers that are relatively similar in terms of RFM according to the selected breakpoints. As there are 3 breakpoints (one per variable), you will get 2^3 segments because each variable will be broken down in 2 parts leading to 8 segments in total. In other words, you will have:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Segm 1: Recency &amp;lt; breakpoint_Recency &amp;amp; Frequency &amp;lt; breakpoint_Frequency &amp;amp; Monetary &amp;lt; breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 2: Recency &amp;lt; breakpoint_Recency &amp;amp; Frequency &gt;= breakpoint_Frequency &amp;amp; Monetary &amp;lt; breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 3: Recency &amp;lt; breakpoint_Recency &amp;amp; Frequency &amp;lt; breakpoint_Frequency &amp;amp; Monetary &gt;= breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 4: Recency &amp;lt; breakpoint_Recency &amp;amp; Frequency &gt;= breakpoint_Frequency &amp;amp; Monetary &gt;= breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 5: Recency &gt;= breakpoint_Recency &amp;amp; Frequency &amp;lt; breakpoint_Frequency &amp;amp; Monetary &amp;lt; breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 6: Recency &gt;= breakpoint_Recency &amp;amp; Frequency &gt;= breakpoint_Frequency &amp;amp; Monetary &amp;lt; breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 7: Recency &gt;= breakpoint_Recency &amp;amp; Frequency &amp;lt; breakpoint_Frequency &amp;amp; Monetary &gt;= breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Segm 8: Recency &gt;= breakpoint_Recency &amp;amp; Frequency &gt;= breakpoint_Frequency &amp;amp; Monetary &gt;= breakpoint_Monetary&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the mean of future 8 weeks purchases for each of the resulted segments&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then, of course, this segmentation was random and is not the optimal one. To get the optimal segmentation you then implement a Genetic Algorithm to re-do the previous procedure and explore the space for better breakpoints. By this you will likely get the optimal segmentation of homogeneous customers in terms of their purchasing behavior (as characterized by RFM) – optimal with respect to their value (as you decide to define their value).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By the way, the fitness function for the GA aims to maximize the mean of future 8 weeks purchases per segment. Or you can also set your own objective on how to measure the value of customers and define the fitness function accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps. Good luck!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: This solution allows you to find 'the homogenous groups of customers' and 'rank segments per value'. I am not sure if this is what you meant by 'rank customers per value'. In my view supervised clustering defeats the idea of individual ranking (but maybe I am wrong). &#xA;Moreover, customers change behaviors from one period to the other and it is perhaps better to cluster them in terms of purchasing behavior, not to rank individuals at a point in time. In this way, when a customer changes behavior (in terms of RFM), we know where s/he went (i.e. in which cluster), and thus we can assess whether s/he 'improved' or not. In the latter case, marketing people can take action to incentivize the concerned customer to move back into the better off segment.&lt;/p&gt;&#xA;" OwnerUserId="24528" LastEditorUserId="24528" LastEditDate="2016-09-22T15:12:03.247" LastActivityDate="2016-09-22T15:12:03.247" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14160" PostTypeId="2" ParentId="14154" CreationDate="2016-09-22T15:15:38.240" Score="4" Body="&lt;p&gt;Let's look at the error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&quot;found&quot; is the type of object that it came across, &quot;required&quot; is the type of the object that the function accepts. The types look mostly the same ( org.apache.spark.mllib.linalg.Vector ) but the first is Array[X] and the second is X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why? Because trainingData.collect() gives you the RDD as an array of objects in the array.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you probably want to do instead is &lt;code&gt;map&lt;/code&gt;, to turn each vector into a LabeledPoint, instead of trying to turn the whole RDD into a single LabeledPoint.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastActivityDate="2016-09-22T15:15:38.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14161" PostTypeId="1" AcceptedAnswerId="14200" CreationDate="2016-09-22T15:15:44.010" Score="2" ViewCount="246" Body="&lt;p&gt;How can it be that a &lt;code&gt;xgboost.cv&lt;/code&gt; cross-validation operation where n-folds are evaluated is quicker than a single &lt;code&gt;XGBoostClassifier.fit(X,y)&lt;/code&gt; of the xgboost.sklearn API?&lt;/p&gt;&#xA;" OwnerUserId="9053" LastActivityDate="2016-09-24T18:55:11.757" Title="xgboost speed difference per API" Tags="|classification|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14162" PostTypeId="1" CreationDate="2016-09-22T15:33:14.370" Score="4" ViewCount="1019" Body="&lt;p&gt;From my research, a recommendation system are a subclass of information filtering system that seek to predict the &quot;rating&quot; or &quot;preference&quot; that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. Any suggestions are welcomed.&lt;/p&gt;&#xA;" OwnerUserId="17596" LastActivityDate="2016-10-23T07:02:56.153" Title="How can conclusions be drawn from recommendation systems evaluation?" Tags="|predictive-modeling|recommender-system|information-retrieval|model-evaluations|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14163" PostTypeId="2" ParentId="14162" CreationDate="2016-09-22T20:33:01.827" Score="4" Body="&lt;p&gt;&quot;Good&quot;, I think, is based on the state of the art at the moment.  So I would look at respected models from industry leaders and use their  reported accuracies as a base line for what is &quot;good&quot;: since it comes down to what is possible. &lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-09-22T20:33:01.827" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14164" PostTypeId="2" ParentId="14162" CreationDate="2016-09-22T20:42:45.690" Score="2" Body="&lt;p&gt;You've already mentioned a few metrics and I guess they work, at least from a algorithmic point of view. However, I think a lot of the initial validation would really has to be done manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work on a similar problem where we are developing a model for a very new domain. We also have the additional problem of not having a lot of labelled data. We validated our results by having a lot of meetings and reviews with subject matter experts and we're also considering several options to get crowdsourced labels as a means to validate.&lt;/p&gt;&#xA;" OwnerUserId="8206" LastActivityDate="2016-09-22T20:42:45.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14165" PostTypeId="1" AcceptedAnswerId="14176" CreationDate="2016-09-22T21:52:17.627" Score="0" ViewCount="1135" Body="&lt;p&gt;Is there a way to take a set of data that consists of discrete values and predict a continuous value? Take for instance data that looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample matrix of jewel data&#xA; color |  size | shape &#xA;['red' ,'large','square']&#xA;['blue','small','circle'] &#xA;['blue','small','square']&#xA;&#xA;sample array of price labels&#xA;[9.99, 7.00, 6.37]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can I do &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/tree/plot_tree_regression.html&quot; rel=&quot;nofollow&quot;&gt;Decision Tree Regression&lt;/a&gt; on this to predict the price of a jewel with a given set of features? What if some of the data is continuous? Also is there any way I can/should pre-process the categorical data other than onehot encoding?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2016-09-23T17:27:01.093" Title="Can I use categorical data and Decision Trees to regress a continuous variable?" Tags="|machine-learning|python|regression|decision-trees|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14167" PostTypeId="1" CreationDate="2016-09-22T22:23:42.240" Score="1" ViewCount="70" Body="&lt;p&gt;Can a linear regression model with a single input variable x with x not significant but the model has a high R2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this exist? If yes, what are the reasons?&lt;/p&gt;&#xA;" OwnerUserId="20810" LastActivityDate="2016-11-22T11:33:58.900" Title="Insignificant input variable but high R2" Tags="|regression|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14168" PostTypeId="2" ParentId="14124" CreationDate="2016-09-22T22:36:34.247" Score="7" Body="&lt;p&gt;You might actually encounter problems if you model this as a regression problem without a suitable transformation. For instance, we know that most calls are probably answered during daytime and less during night time and early morning. A linear regression would have difficulty because the relationship is likely curvilinear, not linear. For the same reason, treating this as a classification task with logistic regression would also be problematic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested by other respondents, reclassifying your data into time periods will help, and I'd suggest you try something like a decision tree or random forest first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That all said, this might be a case for simple descriptive statistics. If you plot the proportion answered calls by time of day (split by city or any other demographic), is there a clear &lt;em&gt;best&lt;/em&gt; time?  If so, why complicate things with a model?&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2016-09-22T22:36:34.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14171" PostTypeId="2" ParentId="14152" CreationDate="2016-09-23T08:38:41.433" Score="4" Body="&lt;p&gt;Your training and test errors are affected by the size of the training. Take a look to this plot, usually known as a learning curve:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B7GI9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B7GI9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, we compute the training score and the test score (cross validation score) of a Naive Bayes model as we increase the number of examples in the training dataset. The higher the score is, the better the model is performing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A larger training set decreases the score because it is more difficult for the learning algorithm to learn a model that correctly represents all the training data. However, as we increase the size of the training set, the test score also increases, due to an increase in the model's ability to generalise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, both lines in the plot reach a limit on the right size. This means that in order to learn properly, an algorithm requires enough data, just enough for us to get to the right side of this plot. Once we reach that asymptote, we can not improve the test score by using more training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, notice that this plot only refers to the size of the training set, and not to the size of the test set, but inferring from it, I may assume that when you only use 2-3 examples in your test set you don't have enough test data to determine your test score. You may even get completely different results for other 2-3 different test examples. Therefore, I would assume that the 30% test error of your larger test set is a most feasible value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that, in order to increase the performance of your model, you will have to look for other reasons: are your features well defined? Do you have enough training data? &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-09-23T08:38:41.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14172" PostTypeId="1" CreationDate="2016-09-23T08:39:58.347" Score="6" ViewCount="1823" Body="&lt;p&gt;Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$MI(X,Y) = \int_Y \int_X&#xA;                 p_z(x,y) \log{ \left(\frac{p_z(x,y)}{p_x(x)\,p_y(y)}&#xA;                              \right) }$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $p_x$ and $p_y$ are the marginal distributions of X and Y:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_x(x) = \int_Yp_z(x,y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_y(y) = \int_Xp_z(x,y)$.&lt;/p&gt;&#xA;" OwnerUserId="23416" LastEditorUserId="23416" LastEditDate="2016-09-27T08:53:04.667" LastActivityDate="2016-09-27T08:53:04.667" Title="How to estimate the mutual information numerically?" Tags="|numerical|mutual-information|distribution|estimators|information-theory|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14173" PostTypeId="2" ParentId="14167" CreationDate="2016-09-23T10:42:56.777" Score="2" Body="&lt;p&gt;This can happen, but only with very small amounts of data. An example (in R) with three data points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(0,1,2)&#xA;y &amp;lt;- c(-0.1,0.7,1.2)&#xA;summary(lm(y ~ x))&#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)  &#xA;(Intercept)  -0.0500     0.1118  -0.447   0.7323  &#xA;x             0.6500     0.0866   7.506   0.0843 .&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 0.1225 on 1 degrees of freedom&#xA;Multiple R-squared:  0.9826,    Adjusted R-squared:  0.9651 &#xA;F-statistic: 56.33 on 1 and 1 DF,  p-value: 0.08432&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice the R-squared is very high, but the &lt;code&gt;x&lt;/code&gt; parameter is not 95%-significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With lots of data, one cannot have a very high R-squared parameter without the corresponding parameter being significant.&lt;/p&gt;&#xA;" OwnerUserId="23911" LastActivityDate="2016-09-23T10:42:56.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14174" PostTypeId="1" CreationDate="2016-09-23T13:53:49.707" Score="1" ViewCount="823" Body="&lt;p&gt;I have implemented two type of clustering in python, using SciPy: one with hierarchical approach, and the other one with k-means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each cases I have used as input a two dimensional array X (documents, binary features) of 628 * 492.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Hierarchical:&#xA;&#xA;Z = linkage(X, 'ward')&#xA;clusters = fcluster(Z, max_k, criterion='maxclust').tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#k-means:&#xA;&#xA;clusters = kmeans2(X, k, iter_, minit=&quot;points&quot;)[1].tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the first case (hierarchical one) I got an array of size 628 (an array which the i-th element is the i-th document associate cluster) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second one I got an array of size 627. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got also this warning in the k-means execution (using different values for k):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/usr/lib/python3/dist-packages/scipy/cluster/vq.py:600: &#xA;UserWarning: One of the clusters is empty. &#xA;Re-run kmean with a different initialization.&#xA;warnings.warn(&quot;One of the clusters is empty. &quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The cluster empty is the one associate to the missing document? &#xA;If the document missing would be in the &quot;middle&quot; of the list this situation wouldn't led to incoherent results? (list indexing wrong)&lt;/p&gt;&#xA;" OwnerUserId="15883" LastEditorUserId="15883" LastEditDate="2016-09-24T13:18:01.763" LastActivityDate="2016-09-24T13:18:01.763" Title="k-means python implementation: unclear\wrong result" Tags="|python|clustering|k-means|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14175" PostTypeId="2" ParentId="9480" CreationDate="2016-09-23T15:52:05.490" Score="1" Body="&lt;p&gt;I have had to perform that type of operation on so-called &lt;code&gt;.DAT&lt;/code&gt; files. You need a  good &lt;strong&gt;hexadecimal editor&lt;/strong&gt;, that allows you to display &lt;code&gt;char&lt;/code&gt;, &lt;code&gt;binary&lt;/code&gt; and &lt;code&gt;hex&lt;/code&gt; contents from binary, and some &lt;strong&gt;patience&lt;/strong&gt;. And probably a digit-probe friend you can talk with, to help decipher the riddle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seek global headers and local ones, file ID, dates and try to locate numbers or ascii code from your database. With that, you can infer field storage length with a relative ease, as least as long they are not exotic, like multiple of 4 bits. Sometimes, there are dummy bits to fill blanks. If the file is somehow compacted (one chunk on 3 bits, the next on 17 bits), it can become quite complicated. Imagine you are a private investigator,  challenge a geek friend. Performing such a task with another person can make it funnier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used &lt;a href=&quot;https://mh-nexus.de/en/hxd/&quot; rel=&quot;nofollow noreferrer&quot;&gt;HxD&lt;/a&gt; and &lt;a href=&quot;https://hexed.it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;HexEd.It&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="12527" LastEditDate="2018-03-30T14:41:58.127" LastActivityDate="2018-03-30T14:41:58.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14176" PostTypeId="2" ParentId="14165" CreationDate="2016-09-23T17:27:01.093" Score="1" Body="&lt;p&gt;Yes, most software implementations of trees will allow you to predict a continuous target variable with all binary predictors. This is because the predictors are only used as splits, and the prediction comes from the average value at a given terminal node. The predictions will not be truly continuous across all terminal nodes in the same way that linear regression is continuous, but in practice, this is generally not a problem. If your tree is under-fitting (not continuous enough) you can always add more terminal nodes. Also, one-hot encoding should be sufficient. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-09-23T17:27:01.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14177" PostTypeId="2" ParentId="14172" CreationDate="2016-09-23T19:13:09.640" Score="4" Body="&lt;p&gt;&lt;strong&gt;Binning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that fall within each bin to obtain a density. So, the calculation would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;    \sum_{b_x \in Bins_x}&#xA;    \sum_{b_y \in Bins_y} &#xA;    \frac {\#(b_x, b_y)} N \log \frac {\frac {\#(b_x, b_y)} N} &#xA;      {{\frac {\#b_x} N} {\frac {\#b_y} N}}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\#(b_x b_y)$ is the number of samples where $X \in b_x$ and $Y \in b_y$, $\#b_y$ is the number of samples where $Y \in b_y$ and $\#b_x$ is the number of samples where $X \in b_x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Entropy and Density Estimation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method is to note that the mutual information can be represented as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;    I(X, Y) = H(X) + H(Y) - H(X,Y)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so if you can estimate entropy you can estimate information. Now, in order to find the expectation of a function over your data you can just use the plugin estimator and do&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;    E[g(X)] = {\frac 1 n} \sum_i g(X_i)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem here is that we want to estimate the function $\log p(x)$ where $p$ depends on our data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So one can use an estimate of the density of $p$ for each point and then use the plugin estimator. Kernel density estimators are one approach and nearest neighbor estimators are another. Both methods are non-parametric.&lt;/p&gt;&#xA;" OwnerUserId="2724" LastEditorUserId="2724" LastEditDate="2016-09-23T19:19:40.747" LastActivityDate="2016-09-23T19:19:40.747" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14179" PostTypeId="1" AcceptedAnswerId="14182" CreationDate="2016-09-23T22:48:05.937" Score="0" ViewCount="590" Body="&lt;p&gt;I have reduced the data set to only the columns I need:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| yearID | POS | PO | A | E |&#xA;|--------|:---:|:--:|:-:|:-:|&#xA;|  1871  | SS  | 0.0|3.0|1.0|&#xA;|  1871  | 2B  |30.0|1.0|0.0|&#xA;|  ...   |  .. | ...|...|...|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;source: &lt;a href=&quot;http://seanlahman.com/files/database/baseballdatabank-master_2016-03-02.zip&quot; rel=&quot;nofollow&quot;&gt;Sean Lahmans 2015 Baseball Data set&lt;/a&gt; Using the Fielding.csv file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate the Fielding Percentage:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values = df['E'] / (df['PO'] + df['A'] + df['E'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where there are multiple records for each 'yearID'. I am not sure if I need to transpose, apply a function or map one. Additionaly, in what order should I be moving the pieces around in.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.loc[:,('C')] = middle_infielders.PO + middle_infielders.A + data.E&#xA;data.loc[:,('FP')] = 1 - (data.E / data.C)&#xA;&#xA;| yearID | POS | PO | A | E |  C  |  FP  |&#xA;|--------|:---:|:--:|:-:|:-:|     |      |&#xA;|  1871  | SS  | 0.0|3.0|1.0| 123 | .960 |&#xA;|  1871  | 2B  |30.0|1.0|0.0|  12 | .452 |&#xA;|  ...   |  .. | ...|...|...| ... |  ... |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like it in this form to plot a line graph:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| yearID |  SS  |  2B  |&#xA;|--------|:----:|:----:|&#xA;|  1871  |0.3745|0.1245|&#xA;|  1872  |0.8940|0.3366|&#xA;|  ...   |  ... | ...  |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;End result: One mean for each 'POS' (SS and 2B) each year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expecting pivoting to aggregate the values='FP' from the statement:&lt;br&gt;&#xA;&lt;code&gt;data.pivot(index='yearID', columns='POS', values='FP')&lt;/code&gt;&lt;br&gt;&#xA;But, I get the error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;ValueError: Index contains duplicate entries, cannot reshape.&quot; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should I apply a Lambda to calculate the Fielding Percentage ('FP') I want in the values instead of pre-calculating it?&lt;/p&gt;&#xA;" OwnerUserId="23800" LastEditorUserId="23800" LastEditDate="2016-10-01T00:04:32.443" LastActivityDate="2017-03-31T08:32:25.907" Title="Pivot DataFrame while calculating new values" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14181" PostTypeId="1" CreationDate="2016-09-23T23:27:29.433" Score="2" ViewCount="2832" Body="&lt;p&gt;I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Which normalization function to use and when? I am currently using the z score.&lt;/li&gt;&#xA;&lt;li&gt;If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?&lt;/li&gt;&#xA;&lt;li&gt;After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24629" LastEditorUserId="31513" LastEditDate="2017-05-04T19:57:18.963" LastActivityDate="2017-05-04T19:57:18.963" Title="Dealing with outliers and z-scores" Tags="|data-mining|dataset|data|outlier|" AnswerCount="2" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14182" PostTypeId="2" ParentId="14179" CreationDate="2016-09-24T04:10:42.687" Score="1" Body="&lt;p&gt;The magic word is &quot;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/reshaping.html&quot; rel=&quot;nofollow&quot;&gt;pivoting&lt;/a&gt;&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;records = [(1871,&quot;SS&quot;,0.0,3.0,1.0), (1871,&quot;2B&quot;,30.0,1.0,0.0)]&#xA;pandas.DataFrame.from_records(records, columns=(&quot;yearID&quot;, &quot;POS&quot;, &quot;PO&quot;, &quot;A&quot;, &quot;E&quot;)&#xA;     ).assign(result = df.apply(lambda x: x['E']/(x['PO']+x['A']+x['E']), 1)&#xA;     ).pivot(index='yearID', columns='POS', values='result')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| POS    | 2B | SS   |&#xA;|--------|----|------|&#xA;| yearID |    |      |&#xA;| 1871   | 0  | 0.25 |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I went to the trouble of looking at your file now that you linked to it, and the problem is that you have removed to much information, so your index is not unique. Indices always have to be unique, so either you add that information back, or you preprocess the data before pivoting such that the duplication is resolved. I chose to retain the extra columns; I don't know if this is what you want, but it should help you understand what it takes to make pivoting work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import read_csv&#xA;&#xA;field_cols = (&quot;playerID&quot;, &quot;yearID&quot;, &quot;teamID&quot;, &quot;POS&quot;, &quot;PO&quot;, &quot;A&quot;, &quot;E&quot;)&#xA;df = read_csv('Fielding.csv', usecols=field_cols).dropna(&#xA;             ).query('not PO == A == E == 0')&#xA;&#xA;df.assign(ID = df[['playerID', 'yearID', 'teamID']].apply(tuple, 1), &#xA;          FP = df.apply(lambda x: 1-x['E']/(x['PO']+x['A']+x['E']), 1)&#xA;          ).drop(set(df.columns) - {'POS'}, 1).drop_duplicates('ID'&#xA;          ).pivot(index='ID', values='FP', columns='POS')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| POS                    | 1B  | 2B  | 3B  | C   | CF  | LF  | OF  | P        | RF  | SS  |&#xA;|------------------------|-----|-----|-----|-----|-----|-----|-----|----------|-----|-----|&#xA;| ID                     |     |     |     |     |     |     |     |          |     |     |&#xA;| (aardsda01, 2006, CHN) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |&#xA;| (aardsda01, 2007, CHA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.857143 | NaN | NaN |&#xA;| (aardsda01, 2008, BOS) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |&#xA;| (aardsda01, 2009, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |&#xA;| (aardsda01, 2010, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.833333 | NaN | NaN |&#xA;  ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-10-01T18:29:32.270" LastActivityDate="2016-10-01T18:29:32.270" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14183" PostTypeId="2" ParentId="14181" CreationDate="2016-09-24T05:11:13.040" Score="0" Body="&lt;p&gt;Which normalization function to use and when? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Normalization_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Normalization (statistics)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which outlier technique should I use and when? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/16709/how-to-deal-with-outliers&quot;&gt;How to deal with outliers?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to devise a function of the z scores? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not able to give any specific answer, but my strategy would be to try and find meaningful relationships with the features.  I would then try and highlight those relationships with the statistics.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-09-24T05:11:13.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14184" PostTypeId="1" AcceptedAnswerId="14197" CreationDate="2016-09-24T08:40:12.640" Score="3" ViewCount="1032" Body="&lt;p&gt;Apologies if this question is not a suitable format. I am a novice in data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a database of species observation data consisting of ~16 million records.&#xA;Each record consists of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;latitude&lt;/li&gt;&#xA;&lt;li&gt;longitude&lt;/li&gt;&#xA;&lt;li&gt;date&lt;/li&gt;&#xA;&lt;li&gt;time&lt;/li&gt;&#xA;&lt;li&gt;species observed (that's species singular, not plural)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This data has been manually vetted by experts, so there is an additional field for each species in a record that classifies the observation as either valid or invalid (or more accurately speaking, likely correct/likely incorrect)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am exploring the idea of training a neural network on this data to automatically classify new records as being valid or invalid (&quot;invalid&quot; data will be flagged for manual expert review.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vast majority of records are classified as 'valid', so my worry is that there isn't much information to train the model on what constitutes 'invalid'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a good predictor of whether a record is valid is, informally speaking, &quot;are there other records of this species close by (spatially and/or temporally)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure where to start with formulating a neural network for this problem. E.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inputs: latitude, longitude, date, time, species&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output: validity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inputs: latitude, longitude, date, time&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outputs: one output for each known species indicating validity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like the idea of this second model as I can input a time and location and get out a list of likely species.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So my concrete questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this sounds like an application suitable for a neural network?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If so, where might I start with formulating a model for my problem? Or can someone point me in a good direction to learn more about this topic.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24635" LastActivityDate="2016-09-24T19:07:03.680" Title="Is a neural network suitable for this application" Tags="|neural-network|beginner|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14187" PostTypeId="1" AcceptedAnswerId="14194" CreationDate="2016-09-24T11:24:50.800" Score="49" ViewCount="56669" Body="&lt;p&gt;I have noticed that such terms as model &lt;strong&gt;hyperparameter&lt;/strong&gt; and model &lt;strong&gt;parameter&lt;/strong&gt; have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. Consider a machine learning model, an SVM/NN/NB based classificator or image recognizer, just anything that first springs to mind. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the &lt;strong&gt;hyperparameters&lt;/strong&gt; and &lt;strong&gt;parameters&lt;/strong&gt; of the model?&lt;br&gt;&#xA;Give your examples please.&lt;/p&gt;&#xA;" OwnerUserId="22012" LastEditorUserId="22012" LastEditDate="2016-09-27T08:36:30.687" LastActivityDate="2022-08-11T11:20:12.103" Title="What is the difference between model hyperparameters and model parameters?" Tags="|machine-learning|parameter|hyperparameter|language-model|" AnswerCount="7" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14193" PostTypeId="1" CreationDate="2016-09-24T13:16:15.190" Score="2" ViewCount="282" Body="&lt;p&gt;I'm using a neural network to analyze item choices made by players in a computer game. In the game players can choose between 0 and 7 items. Right now I'm struggling with how I can evaluate my data.&lt;/p&gt;&#xA;&lt;p&gt;Tensorflow provides a &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/top_k&quot; rel=&quot;nofollow noreferrer&quot;&gt;nice method&lt;/a&gt; for getting the k highest values.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;tf.nn.top_k(input, k=1, sorted=True, name=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The input here would be the prediction by my neural network. The output I get by applying top_k to the prediction would then be compared to applying top_k to the correct_output (the one the neural net should have had) and by doing this multiple times and averaging I get the accuracy that I want to have. The problem I'm running into is that k should depend on the amount of 1's in the correct_output. I am lost as to how I can achieve this.&lt;/p&gt;&#xA;&lt;p&gt;edit: correct output (if I understand how this works correctly) should already be a tensor. It is loaded from a .pickle file and at the beginning of the code it is prepared as follows:&#xA;correct_output = tf.placeholder('float')&lt;/p&gt;&#xA;&lt;p&gt;As to what it looks like: it is simply a list of given length of 1's and 0's&lt;/p&gt;&#xA;" OwnerUserId="19067" LastEditorUserId="98307" LastEditDate="2020-08-12T15:32:29.467" LastActivityDate="2020-08-12T15:32:29.467" Title="Using tensorflow to test a variable amount of correct labels" Tags="|neural-network|tensorflow|model-evaluations|multilabel-classification|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="14194" PostTypeId="2" ParentId="14187" CreationDate="2016-09-24T13:50:41.380" Score="36" Body="&lt;p&gt;Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. However, 'parameters' is a more general term. When you say 'passing the parameters to the model', it generally means a combination of hyperparameters along with some other parameters that are not directly related to your estimator but are required for your model.&lt;/p&gt;&#xA;&lt;p&gt;For example, suppose you are building a SVM classifier in sklearn:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;X = [[0, 0], [1, 1]]&#xA;y = [0, 1]&#xA;clf = svm.SVC(C =0.01, kernel ='rbf', random_state=33)&#xA;clf.fit(X, y) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In the above code an instance of SVM is your estimator for your model for which the hyperparameters, in this case, are &lt;code&gt;C&lt;/code&gt; and &lt;code&gt;kernel&lt;/code&gt;. But your model has another parameter which is not a hyperparameter and that is &lt;code&gt;random_state&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="-1" LastEditDate="2022-02-01T05:51:53.183" LastActivityDate="2022-02-01T05:51:53.183" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14195" PostTypeId="2" ParentId="14184" CreationDate="2016-09-24T13:56:28.270" Score="1" Body="&lt;p&gt;You may or may not apply neural network to your problem. Neural network will give predictions for sure but it can't be said how much efficient will it be for this problem. You have to code and check it for yourself. Also, if you have all the above data in tabular form and you know the labels(which I guess is &lt;code&gt;valid/inavlid&lt;/code&gt;), I will say try it with xgboost. Neural networks are exceptional for unsupervised learning but in the case of supervised learning, there may be a model which may outperform a neural network. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-24T13:56:28.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14197" PostTypeId="2" ParentId="14184" CreationDate="2016-09-24T16:50:28.893" Score="1" Body="&lt;p&gt;Before deciding on the model, I would recommend to re-formulate the dataset to best suit your problem. You could approach this problem as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Since the output you're trying to predict is validity of the observation, keep &quot;validity&quot; = True/False, or, 1/0 as the target variable.&lt;/li&gt;&#xA;&lt;li&gt;One of the parameters is a categorical variable &quot;species&quot;, and I'm expecting this to have a high cardinality. Since there are approximately 8.7 million species on earth, if you used this variable in a model it could possibly expand into 8.7 million individual columns (in on-hot encoding form). Even a conservative estimate of 100,000 species makes it nonviable to be used as it is. So you need a way to convert this species information to a fewer features.&lt;/li&gt;&#xA;&lt;li&gt;One approach you could try out is to create geographical clusters for each species (using only valid marked records), then identify the nearest center and max/avg./quartile measures of distance from their cluster center for each species. Do this for each quarter of the year separately to account for seasonal changes. Next, add this information back to the main dataset to indicate for each record - all the geographical centers of that species cluster. In the next step, for each record find the nearest cluster center and calculate this particular observation's distance from its cluster center. Then calculate the ratio of its distance from cluster center vs. max distance and vs. avg distance from that cluster's center. Use this metric instead of the geospatial coordinates and species identifier.&lt;/li&gt;&#xA;&lt;li&gt;Another approach could be to add additional features such as the climate of each location and average historic temperature at that location during the time of the year when the observation was taken. This is because some animals may migrate north/south based on the seasons and so if a species' location was found valid in the summer, it may be impossible to find it in the same location in winter due to it being unable to survive the cold weather. If you combine this with #3 above, it would enrich the observations significantly.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After doing this extensive hard work, you should do some exploratory analysis and plot subsets of this data to better understand it. By visualizing the data, sometimes we're able to figure out best course of action more quickly than without visualizing the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you may explore different machine-learning algorithms to fit a model to this refined data. I would recommend trying out other algorithms such as logistic regression, SVM, ridge-regression, random forests and gradient boosting machines in addition to neural networks and then select the best performing one. Most machine learning suites/frameworks implement these, so it should not be difficult to find out how to apply these to your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks are fine to try out, but as with all algorithms you need to be careful about the usual pitfalls such as:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Avoid over-fitting the model to training data: to avoid this use regularization and keep cross checking accuracy with an independent held-out validation set.&lt;/li&gt;&#xA;&lt;li&gt;Use cross-validation (10-fold) and repeat several times to get good estimates of the model's performance metrics on new data.&lt;/li&gt;&#xA;&lt;li&gt;Since the data is highly class imbalanced (many valid records but few invalid records by proportion), use a performance metric other than simple true positive accuracy. Try using F1 score, precision (of identifying invalid records), Kappa metric, etc.&lt;/li&gt;&#xA;&lt;li&gt;Due to high class imbalance, it would help if you either over-sampled the minority class (invalid) or under-sampled the majority class (valid), or did both together. This will improve the model's ability to classify mode precisely.&lt;/li&gt;&#xA;&lt;li&gt;Adjust the hyper-parameters such as learning rate and hidden layers/no. of units for best model performance.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-09-24T16:50:28.893" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14198" PostTypeId="1" CreationDate="2016-09-24T17:42:04.227" Score="1" ViewCount="2891" Body="&lt;p&gt;I'm using sklearn's LogisticRegression with &lt;code&gt;penaly=l1&lt;/code&gt; (lasso regularization, as opposed to ridge regularization l2). Lasso is causing the optimization function to do implicit feature selection by setting some of the feature weights to zero (as opposed to ridge regularization, which will preserve all features with some non zero weight). Is it possible to extract this weight information (feature selection) from sklearn somehow?&lt;/p&gt;&#xA;" OwnerUserId="23480" LastActivityDate="2016-10-24T18:39:04.287" Title="Feature selection with L1 regularization on sklearn's LogisticRegression" Tags="|scikit-learn|feature-selection|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14199" PostTypeId="2" ParentId="14198" CreationDate="2016-09-24T18:04:58.627" Score="2" Body="&lt;p&gt;Yes, as demonstrated in the documentation: &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_path.html&quot; rel=&quot;nofollow&quot;&gt;http://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_path.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Welcome to DataScience.SE&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-24T18:04:58.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14200" PostTypeId="2" ParentId="14161" CreationDate="2016-09-24T18:55:11.757" Score="1" Body="&lt;p&gt;I believe this is the answer: &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/651&quot; rel=&quot;nofollow&quot;&gt;https://github.com/dmlc/xgboost/issues/651&lt;/a&gt; &#xA;the sklearn api uses &lt;code&gt;n_estimators= 100&lt;/code&gt; as default whereas xgb.train is using &lt;code&gt;n_boost_rounds=10&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As both refer to the same parameter this could explain the huge difference.&lt;/p&gt;&#xA;" OwnerUserId="9053" LastActivityDate="2016-09-24T18:55:11.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14201" PostTypeId="2" ParentId="14184" CreationDate="2016-09-24T19:07:03.680" Score="1" Body="&lt;p&gt;The basics of machine learning is this.  Is there a pattern to the data?  If so then there is a formula which describes the pattern.  If the formula is known, use the formula.  If there is a pattern but the formula describing the pattern is unknown, you can use machine learning to determine the formula which best approximates it.  &lt;/p&gt;&#xA;" OwnerUserId="24649" LastActivityDate="2016-09-24T19:07:03.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14202" PostTypeId="1" CreationDate="2016-09-25T11:12:17.030" Score="0" ViewCount="72" Body="&lt;p&gt;I am wondering if someone here can recommend some good data science certified online courses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple google research brings hundreds of online academy and I am lost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is been one year since I am working in this field and I want to validate my knowledge and acquire more.&lt;/p&gt;&#xA;" OwnerUserId="24657" LastActivityDate="2016-09-25T13:21:24.067" Title="Suggestions for good data science certified online courses" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2016-09-25T13:33:04.953" ContentLicense="CC BY-SA 3.0" />
  <row Id="14203" PostTypeId="1" CreationDate="2016-09-25T11:14:17.810" Score="-2" ViewCount="2475" Body="&lt;p&gt;I am currently working on the data set from this &lt;a href=&quot;https://github.com/smarthi/UnivOfWashington-Machine-Learning/tree/master/MLFoundations/week4/people_wiki.gl&quot; rel=&quot;nofollow&quot;&gt;link&lt;/a&gt;. But I am unable to read these files from Pandas? Has anyone tried to play with such files?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.read_csv(&quot;m_4549381c276b46c6.0000&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I get the following error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23403" LastEditorUserId="924" LastEditDate="2016-09-26T05:42:02.003" LastActivityDate="2020-08-04T10:54:22.190" Title="Pandas: how to read certain file type in pandas" Tags="|python|pandas|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14204" PostTypeId="2" ParentId="14203" CreationDate="2016-09-25T11:57:31.390" Score="0" Body="&lt;p&gt;I don't know this particular dataset, but try adding&#xA;&lt;code&gt;encoding='utf-8'&lt;/code&gt; to the &lt;code&gt;read_csv&lt;/code&gt; call.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_csv(&amp;quot;m_4549381c276b46c6.0000&amp;quot;,encoding='utf-8')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23122" LastEditorUserId="98307" LastEditDate="2020-08-04T10:54:22.190" LastActivityDate="2020-08-04T10:54:22.190" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14205" PostTypeId="1" CreationDate="2016-09-25T13:05:36.487" Score="1" ViewCount="289" Body="&lt;p&gt;I am try to recreate Karpathy's great tutorial on &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow&quot;&gt;RNN character-level text generation&lt;/a&gt; using TensorFlow - Promise to contribute the code back to the community once it works. Meanwhile, my code runs, but the model does not converge. It stays stuck on the same level of error and emits rubbish. I suspect that the code is correct but my hyper-parameters are off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone previously trained a text-generating RNN successfully, I would be glad to get some hints on the parameters below. The input I use currently is NLTK Bible corpus, about 4MB of text, but I can use something else. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;number of steps, i.e. sequence length&lt;/li&gt;&#xA;&lt;li&gt;batch size&lt;/li&gt;&#xA;&lt;li&gt;number of hidden units&lt;/li&gt;&#xA;&lt;li&gt;number of layers&lt;/li&gt;&#xA;&lt;li&gt;Optimizer and optimization parameters - I'm using Adagrad with decaying learning rate, trying to mimick the original script, but it is not a must&lt;/li&gt;&#xA;&lt;li&gt;Any other tricks for the network architecture or optimization&lt;/li&gt;&#xA;&lt;li&gt;Most important, the number of epochs (full scans of the data) needed before the model start emitting something readable&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="846" LastActivityDate="2016-09-25T13:05:36.487" Title="Hyper-parameters for character-level RNN models" Tags="|nlp|rnn|text-generation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14206" PostTypeId="2" ParentId="14203" CreationDate="2016-09-25T13:17:43.673" Score="0" Body="&lt;p&gt;This is most likely a database file. You cannot directly open it in pandas. You can use a software in order to convert it first to a csv or something that can be opened in pandas. One such application is &lt;a href=&quot;http://dbfconv.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="98307" LastEditDate="2020-08-04T09:44:12.947" LastActivityDate="2020-08-04T09:44:12.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14207" PostTypeId="2" ParentId="14202" CreationDate="2016-09-25T13:21:24.067" Score="1" Body="&lt;p&gt;There are many but some of the resources are awesome. One such resource is &lt;a href=&quot;https://www.udacity.com/&quot; rel=&quot;nofollow&quot;&gt;Udacity&lt;/a&gt;. This is probably the best resource I have encountered yet. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-25T13:21:24.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14208" PostTypeId="1" AcceptedAnswerId="14210" CreationDate="2016-09-25T13:59:04.353" Score="1" ViewCount="9539" Body="&lt;p&gt;I'm trying to reproduce an algorithm designed in a paper. And everything is going well except one thing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It says &lt;code&gt;we considered the lengths zero-meaned accelerometer vectors and created a feature for the mean and standard deviation of this value.&lt;/code&gt; and I do not understand what is it &lt;code&gt;zero-meaned&lt;/code&gt; vectors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;-0.6946377  12.680544  0.50395286&#xA;5.012288    11.264028  0.95342433&#xA;4.903325    10.882658  -0.08172209&#xA;-0.61291564 18.496431  3.0237172&#xA;-1.1849703  12.108489  7.205164&#xA;1.3756552   -2.4925237 -6.510526&#xA;-0.61291564 10.56939   5.706926&#xA;-0.50395286 13.947236  7.0553403&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can any body help me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found only this information &lt;a href=&quot;https://www.quora.com/What-does-it-mean-when-a-vector-is-zero-mean&quot; rel=&quot;nofollow&quot;&gt;https://www.quora.com/What-does-it-mean-when-a-vector-is-zero-mean&lt;/a&gt; but I'm not sure about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="12789" LastActivityDate="2016-09-25T16:12:04.567" Title="What does &quot;zero-meaned vector&quot; mean" Tags="|machine-learning|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14210" PostTypeId="2" ParentId="14208" CreationDate="2016-09-25T16:10:21.680" Score="3" Body="&lt;p&gt;&quot;Zero-meaned&quot; means the vector has been transformed so that its mean is 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, you would do this by subtracting the mean of each column from that column. (This is for dimensional as well as algorithmic reasons; you don't want to subtract a person's weight from their height.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like here they're actually talking about the &lt;em&gt;row&lt;/em&gt; mean--that is, $(-0.6946377, 12.680544,  0.50395286)$ would be transformed to $(-4.857924, 8.5172577, -3.65933344, 4.1632863, 7.40047)$, where the first three are the original features minus the row mean, the fourth is the row mean, and the fifth is the standard deviation of the original features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would make sense if the three have the same units (if they're all accelerations at the same scale, this works), and so you want a separate measure of how much it's being accelerated at all and how much it's being accelerated in a particular direction.&lt;/p&gt;&#xA;" OwnerUserId="12876" LastEditorUserId="12876" LastEditDate="2016-09-25T16:12:04.567" LastActivityDate="2016-09-25T16:12:04.567" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14211" PostTypeId="2" ParentId="14208" CreationDate="2016-09-25T16:10:36.937" Score="2" Body="&lt;p&gt;Mean centering is one of many related techniques to preprocess data for downstream analysis in multivariate methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might sound odd at first, but it means exactly what it says: the vector has a mean of zero. In pseudocode, &lt;code&gt;(sum(vector) / len(vector)) == 0&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In multivariate data, this typically is applied along each column in a dataset so each column can be more easily compared to another within a similar range of data. After mean centering, each row only includes how it differs from the average sample from that variable in the original data. Typically, samples are also scaled to have unit variance as well, allowing you to more readily compare the data across continuous variables with different ranges. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you had a dataset of patients with variables &lt;code&gt;height, weight, age, household_income&lt;/code&gt;, despite each variable being a continuous value each of these variables will be in different range. Height might be between 60 -- 75 inches, weight between 100 -- 300 lbs, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do all this? &#xA;Removing the mean and standardizing the variance will help downstream methods not 'learn' the mean and variance of your data, making it easier to find relationships between variables. Many assume that your data is centered / scaled / normalized in some way and will behave poorly if you don't do so.  &lt;/p&gt;&#xA;" OwnerUserId="11098" LastActivityDate="2016-09-25T16:10:36.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14212" PostTypeId="2" ParentId="13629" CreationDate="2016-09-25T16:35:11.533" Score="2" Body="&lt;p&gt;The weights learned have no straightforward interpretability and i would not interpret those to be &quot;random&quot;. What exactly do you mean by &quot;random&quot;? They seem to be learning very basic patterns that might reasonably extract and combine into more complex patterns as you deeper in the network, which is exactly how the networks are supposed to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from the Keras blog post you linked, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;blockquote&gt;&#xA;      &lt;p&gt;A remarkable observation: a lot of these filters are identical, but rotated by some non-random factor (typically 90 degrees). This means that we could potentially compress the number of filters used in a convnet by a large factor by finding a way to make the convolution filters rotation-invariant. I can see a few ways this could be achieved --it's an interesting research direction.&lt;/p&gt;&#xA;    &lt;/blockquote&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The stanford page shows a great bit of what the network seems to be learning at an early layer -- combinations of edges and color blobs. &#xA;&lt;img src=&quot;https://cs231n.github.io/assets/cnn/weights.jpeg&quot; alt=&quot;img&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps this is more philosophical in nature, but why would we expect the layers to be meaningful or easy to interpret for humans? The network's huge number of weights are learned by optimizing a loss function via gradient descent. It can be difficult to interpret how far less complicated non-linear models &quot;learn&quot; or weight their coefficients; I'd expect these models would be (and are!) wildly difficult to interpret. The fact that we can see any structure in the various visualizations is cool, but they probably won't help us understand why a model isn't working all that well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your problem, how many classes do you have and what type of accuracy are you hoping for? MNIST is an almost trivial dataset since the classes are very well separated. Some of the real-life machine vision tasks I deal with at work with good somewhat clean classes can get 80-90%~ accuracy, but errors are made in photos that humans might call something else (e.g., a photo of a studio apartment's living room in which the kitchen is clearly visible might be labeled as a kitchen and not a living room, but this is mostly okay). Defining what is a &quot;good enough&quot; accuracy is an issue on its own -- that's more than enough for what we're doing at the moment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I missed the point of your question, please let me know. &lt;/p&gt;&#xA;" OwnerUserId="11098" LastEditorUserId="-1" LastEditDate="2017-03-10T09:42:41.947" LastActivityDate="2016-09-25T16:43:16.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14214" PostTypeId="2" ParentId="14203" CreationDate="2016-09-25T18:41:03.213" Score="1" Body="&lt;p&gt;It's an SFrame, not a CSV; you need GraphLab, not pandas. They show you how to load it in the &lt;a href=&quot;https://github.com/smarthi/UnivOfWashington-Machine-Learning/blob/master/MLFoundations/week4/document-retrieval.ipynb&quot; rel=&quot;nofollow&quot;&gt;attendant notebook&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;graphlab.SFrame(path)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-09-25T18:41:03.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14216" PostTypeId="1" CreationDate="2016-09-25T21:06:24.960" Score="4" ViewCount="14107" Body="&lt;p&gt;I have a skewed distribution that looks like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/09kmt.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/09kmt.jpg&quot; alt=&quot;picture of a skewed distribution&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as the order of values remains the same. I'm doing this to experiment if different distributions change the behavior of my ML models.&lt;/p&gt;&#xA;&lt;p&gt;I'm working with Python/NumPy/Pandas/scikit-learn.&lt;/p&gt;&#xA;&lt;p&gt;Edit: I should clarify that I have a lot of features and I'm looking to automatically transform all feature distributions. I was able to find a reasonable transformation for a single feature with a lot of experimentation, but it doesn't generalize to other features:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;normalize(np.log(0.30 + original))&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;** here would be image &lt;code&gt;i.stack.imgur.com/uzorK.jpg&lt;/code&gt; but I don't have enough rep to post more than 2 images **&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;normalize(np.log(0.17 + another_feature_distribution))&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IAWJo.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IAWJo.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In this image the purple bars represent the original distribution of another feature, green bars represent the transformed distribution. No matter how much I tweak the constant, I don't get the high green bar on the left extreme to disappear. Also, I don't have time to manually find a formula for each feature. Not sure if these are bell-shaped enough anyway?&lt;/p&gt;&#xA;" OwnerUserId="23480" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-01-29T15:11:31.170" Title="Transform a skewed distribution into a Gaussian distribution" Tags="|gaussian|distribution|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14217" PostTypeId="2" ParentId="14216" CreationDate="2016-09-25T21:19:26.893" Score="1" Body="&lt;p&gt;You can do a log transformation on your data with the help of numpy log functionality as shown below :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;log_data = np.log(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will transform the data into a normal distribution. Moreover, you can also try Box-Cox transformation which calculates the best power transformation of the data that reduces skewness although a simpler approach which can work in most cases would be applying the natural logarithm. More details about Box-Cox transformation can be found &lt;a href=&quot;http://scipy.github.io/devdocs/generated/scipy.stats.boxcox.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/questions/30018875/how-to-apply-box-cox-transformation-in-python&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-09-25T21:19:26.893" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14220" PostTypeId="1" CreationDate="2016-09-26T06:12:26.650" Score="5" ViewCount="1999" Body="&lt;p&gt;I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html#example-model-selection-plot-learning-curve-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;learning_curve&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dtK01m.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dtK01m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is observed that the accuracy of training dataset decreases but the accuracy of validation dataset increases. I am not able to justify this behavior. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normally, as training dataset increases, the training accuracy is supposed to increase right?&lt;/li&gt;&#xA;&lt;li&gt;Also, assuming that the dataset is very noisy and hence training accuracy is decreasing as dataset size increases. But this doesn't explain why validation accuracy increases because noise is supposed to affect that too.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-09-26T10:01:24.887" LastActivityDate="2016-09-26T10:01:24.887" Title="Training and cross validation error curves" Tags="|machine-learning|dataset|cross-validation|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14221" PostTypeId="2" ParentId="14220" CreationDate="2016-09-26T07:55:32.747" Score="3" Body="&lt;p&gt;I think what you're seeing is normal behaviour:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With only few samples (like 2000) it's easy for a model to (over)fit the data - but it doesn't generalize well. So you get high training accuracy, but the model might not work well with new data (i.e. low validation/test accuracy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you add more samples (like 9000) it becomes harder for the model to fit the data - so you get a lower training accuracy, but the model will work better with new data (i.e. validation/test accuracy starts to rise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;As the training dataset increases, the &lt;strong&gt;training accuracy&lt;/strong&gt; is supposed to &lt;strong&gt;decrease&lt;/strong&gt; because more data is harder to fit well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As the training dataset increases, the &lt;strong&gt;validation/test accuracy&lt;/strong&gt; is supposed to &lt;strong&gt;increase&lt;/strong&gt; as well since less overfitting means better generalization.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Andrew Ng has a video about &lt;a href=&quot;https://www.youtube.com/watch?v=g4XluwGYPaA&quot; rel=&quot;nofollow&quot;&gt;learning curves&lt;/a&gt;. Note that he plots the &lt;strong&gt;error&lt;/strong&gt; on the y axis, you have the &lt;strong&gt;accuracy&lt;/strong&gt; on the y axis.. so the y axis is flipped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also take a look at the second half of the video. It explains high bias and high variance problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model seems to have high variance (due to the big &quot;gap&quot; between the two curves) - it's still too complex for the small amount of data you've got. Either getting more data or using a simpler model (or more regularization on the same model) might improve the results.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="676" LastEditDate="2016-09-26T08:26:44.340" LastActivityDate="2016-09-26T08:26:44.340" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14222" PostTypeId="1" CreationDate="2016-09-26T09:52:12.410" Score="4" ViewCount="241" Body="&lt;p&gt;First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, techniques etc are used to tackle similar problems).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have different lists of strings which contain one or two useful pieces of information I would like to extract. In the following is an example I need to extract the bold and italic part from each line. This is just an example though, eventually I will need to end up with a process I can apply to different lists with different context. Here's a small sample from a list of 500:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;50&quot; &lt;em&gt;Sony&lt;/em&gt; &lt;strong&gt;KDL 50W756CSAEP&lt;/strong&gt; Smart LED Full HD&lt;/li&gt;&#xA;&lt;li&gt;55&quot; &lt;em&gt;Samsung&lt;/em&gt; &lt;strong&gt;UE55JU6400&lt;/strong&gt; Smart LED HD&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;LG&lt;/em&gt; &lt;strong&gt;55LF652V&lt;/strong&gt; 55&quot; SMART 3D FULL HD&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;HITACHI&lt;/em&gt; &lt;strong&gt;55HGW69&lt;/strong&gt; 55'' LED ULTRA SMART WIFI&lt;/li&gt;&#xA;&lt;li&gt;TV 65&quot; &lt;em&gt;SAMSUNG&lt;/em&gt; &lt;strong&gt;UE65KS7500&lt;/strong&gt; 4K LED Smart&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In my full list I have already manually extracted the &lt;em&gt;brand&lt;/em&gt; and &lt;strong&gt;model&lt;/strong&gt;. So what I need now is a way to automate the process for a new list containing more brands and models. I thought I could go about this heuristically but since I am not just doing this for this type of data it won't work well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can someone give me some suggestions on a good way to go about it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24682" LastActivityDate="2017-09-20T22:57:01.780" Title="Extract 2 pieces of information from a string - what to use?" Tags="|text-mining|information-retrieval|training|text|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14223" PostTypeId="1" CreationDate="2016-09-26T10:10:34.397" Score="2" ViewCount="598" Body="&lt;p&gt;In continuation with a &lt;a href=&quot;https://datascience.stackexchange.com/questions/14220/training-and-cross-validation-error-curves&quot;&gt;question&lt;/a&gt; already asked, I tried the same curve on a different dataset I found. My model is a simple Logistic regression curve with OnevsRest Classifier. But the graph I obtained this time was different.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/F4abN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F4abN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be a possible reason for this? As explained in the other answer, if it takes less data to fit better, shouldn't the accuracy decrease here as well?&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-09-27T10:10:40.387" Title="Directly proportional Trend in training and cross validation curves" Tags="|machine-learning|data|cross-validation|accuracy|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14224" PostTypeId="2" ParentId="13587" CreationDate="2016-09-26T14:45:10.923" Score="6" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If this is correct then every &quot;neuron&quot; of the pooling layer has the same gradient?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No.&lt;/strong&gt; It depends on the weights and activation function. And most typically the weights are different from the first neuron of the pooling layer to the FC layer as from the second layer of the pooling layer to the FC layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So typically you will have a situation like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ FC_i = f(\sum_j W_{ij} P_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $FC_i$ is the ith neuron in the fully connected layer, $P_j$ is the jth neuron in the pooling layer and $f$ is the activation function and $W$ the weigths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that the gradient with respect to P_j is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(P_j) = \sum_i grad(FC_i) f^\prime W_{ij}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is different for j=0 or j=1 because the W is different. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It does not make a difference what type of layer it is connected to. It is the same equation all the time. Sum of all the gradients on the next layer multiplied by how the output of those neurons is affected by the neuron on the previous layer. The difference between FC and Convolution is that in FC all neurons in the next layer will provide a contribution (even if maybe small) but in Convolution most neurons in the next layer are not at all affected by the neuron in the previous layer so their contribution is exactly zero. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For the topmost rightmost &quot;neuron&quot; of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Right.&lt;/strong&gt; Plus also the gradient of any other neurons on that convolution layer which take as input the topmost rightmost neuron of the pooling layer. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Add them.&lt;/strong&gt; Because of the chain rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Max Pooling&lt;/strong&gt;&#xA;Up to the this point, the fact that it was max pool was totally irrelevant as you can see. Max pooling is just the that the activation function on that layer is $max$. So this means that the gradients for the previous layers $grad(PR_j)$ are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_j) = \sum_i grad(P_i) f^\prime W_{ij}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now $f = id$ for the max neuron and $f = 0$ for all other neurons, so $f^\prime = 1$ for the max neuron in the previous layer and $f^\prime = 0$ for all other neurons. So:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_{max neuron}) = \sum_i grad(P_i) W_{i\ {max\ neuron}}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_{others}) = 0.$&lt;/p&gt;&#xA;" OwnerUserId="23416" LastEditorUserId="23416" LastEditDate="2016-09-26T14:54:02.863" LastActivityDate="2016-09-26T14:54:02.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14225" PostTypeId="2" ParentId="11693" CreationDate="2016-09-26T15:04:11.560" Score="2" Body="&lt;p&gt;Just averaging them might not be good because that would assume that they they have the same weight, and that is probably not the case if the capitalized and uncapitilized version appear with very different frequencies in your training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An incremental improvement would be to average them proportionally to their frequency in the corpus. So say Earth appears 159 times and earth 1239 times do something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;v(Earth &amp;amp; earth ) = 159/(159+1239) * v(Earth) + 1239/(159+1239) *v(earth).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vectors are supposed to encode semantics linearly, so this should give you a resonable approximation. &lt;/p&gt;&#xA;" OwnerUserId="23416" LastActivityDate="2016-09-26T15:04:11.560" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14226" PostTypeId="1" AcceptedAnswerId="14227" CreationDate="2016-09-26T17:33:37.590" Score="2" ViewCount="100" Body="&lt;p&gt;If we have MLP then we can easily compute the gradient for each parameters, by computing the gradient recursively begin with the last layer of the network, but suppose I have neural network that consist of different type of layer for instance Input-&gt;convolution layer-&gt;ReLu-&gt;max pooling-&gt;fully connected layer-&gt;siftmax layer,  how do I compute the gradient for each parameters ?&lt;/p&gt;&#xA;" OwnerUserId="24696" LastActivityDate="2016-09-26T19:29:16.063" Title="How to train neural network that has different kind of layers" Tags="|machine-learning|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14227" PostTypeId="2" ParentId="14226" CreationDate="2016-09-26T19:21:48.093" Score="1" Body="&lt;p&gt;The different layers you describe can all have gradients calculated using the same back propagation equations as for a simpler MLP. It is still the same recursive process, but it is altered by the parameters of each layer in turn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some details worth noting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you want to &lt;em&gt;understand&lt;/em&gt; the correct formula to use, you will need to study the &lt;a href=&quot;https://theclevermachine.wordpress.com/2014/09/06/derivation-error-backpropagation-gradient-descent-for-neural-networks/&quot; rel=&quot;nofollow&quot;&gt;equations of back propagation&lt;/a&gt; using the chain rule (note I have picked one example worked through, there are plenty to choose from - including &lt;a href=&quot;https://github.com/neilslater/ru_ne_ne/blob/master/tutorials/BACKPROP.md&quot; rel=&quot;nofollow&quot;&gt;some notes I made myself&lt;/a&gt; for a now defunct software project).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When feed-forward values overlap (e.g. convolutional) or are selected (e.g. dropout, max pooling), then the combinations are usually logically simple and easy to understand:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For overlapped and combined weights, such as with convolution, then gradients simply add. When you back propagate the the gradients from each feature &quot;pixel&quot; in a higher layer, they add into the gradients for the shared weights in the kernel, and also add into the gradients for the feature map &quot;pixels&quot; in the layer below (in each case before starting calculation, you might create an all-zero matrix to sum up the final gradients into).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For a selection mechanisms, such as the max pooling layer, you only backprop the gradient to the selected output neuron in the previous layer. The others do not affect the output, so by definition increasing or decreasing their value has no effect - they have a gradient of 0 for the example being calculated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the case of a feed-forward network, each layer's processing is independent from the next, so you only have a complex rule to follow if you have a complex layer. You can write the back propagation equations down so that they relate gradients in one layer to the already-calculated gradients in the layer above (and ultimately to the loss function evaluated in the output layer). It doesn't directly matter what the activation function was in the output layer after you backpropagate the gradient from it - at that point the only difference is numeric, the equations relating deeper layer gradients to each other do not depend on the output at all.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, if you want to just use a neural network library, you don't need to worry much about this, it is usually just done for you. All the standard activation functions and layer architectures are covered by existing code. It is only when creating your own implementations from scratch, or when making use of unusual functions or structure, that you might need to go as far as deriving the values directly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-09-26T19:29:16.063" LastActivityDate="2016-09-26T19:29:16.063" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14228" PostTypeId="1" CreationDate="2016-09-26T23:35:01.490" Score="1" ViewCount="833" Body="&lt;p&gt;I am trying to understand the input for the PrefixSpan model (pyspark). I thought in the input RDD, each row should be a list. However, as I looked into the source code, each row is actually a list of list. Why does it need a list of list instead of just list? Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala&quot; rel=&quot;nofollow&quot;&gt;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; def trainPrefixSpanModel(&#xA;      data: JavaRDD[java.util.ArrayList[java.util.ArrayList[Any]]],&#xA;      minSupport: Double,&#xA;      maxPatternLength: Int,&#xA;      localProjDBSize: Int ): PrefixSpanModelWrapper = {...}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2016-11-24T15:36:43.317" Title="PrefixSpan model input RDD format" Tags="|data-mining|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14229" PostTypeId="2" ParentId="14112" CreationDate="2016-09-26T23:42:24.997" Score="1" Body="&lt;p&gt;You could define a window (distance between red and blue line) and slide it from top to bottom over the y-axis and then count the local minima and maxima in that slice. One way with &lt;code&gt;scipy&lt;/code&gt; is &lt;code&gt;argrelextrema(x, np.less)&lt;/code&gt; for minima and &lt;code&gt;argrelextrema(x, np.greater)&lt;/code&gt; for maxima which gives you the indices of the extrema. With &lt;code&gt;len(argrelextrema(x, np.less))&lt;/code&gt; you would get the number of minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each slice would be a level and the results of counting the minima and maxima the corresponding importance. The size of the window is up to you. If your data is very granular you might want to consider smoothing it first (e.g. with &lt;code&gt;numpy.convolve()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-09-26T23:42:24.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14231" PostTypeId="1" AcceptedAnswerId="14232" CreationDate="2016-09-27T03:57:39.810" Score="1" ViewCount="402" Body="&lt;p&gt;When I read about convolutional neural network from the internet, like this &lt;a href=&quot;http://jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/&quot; rel=&quot;nofollow&quot;&gt;one&lt;/a&gt;, mostly I found that discrete convolution operator is defined as follow:&#xA;$$C=I*F$$&#xA;$$C(x,y)={\sum_{a=0}^{k-1} }{\sum_{b=0}^{k-1}}I(x-a,y-b)F(a,b)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the size of $F$ is $k\times k$, Suppose that the size of F is $3\times 3$ and size of $I$ is $9\times 9$, then $C(1,1)=\dots +F(2,2)I(1-2,1-2)+\dots$ that is does not make sense when $I$ has negative index, how do I compute matrix $C$ ? does we change the way the matrix indexed?&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="24696" LastEditDate="2016-09-27T04:55:35.317" LastActivityDate="2016-09-27T07:04:51.870" Title="Convolution operator yields negative index of matrix" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14232" PostTypeId="2" ParentId="14231" CreationDate="2016-09-27T06:16:53.467" Score="2" Body="&lt;p&gt;There are two approaches that can be taken:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Only use valid indices. Matrix C will then be smaller than matrix I, in your example it would be a 7x7 matrix (9 - 3 + 1 = 7). You may see this in neural network libraries as a convolution working with &quot;valid&quot; border.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use synthetic values, usually just 0, for out-of-bounds indices in matrix I in order to calculate C. This produces the same size output as input, so some neural network libraries note this with &quot;same&quot; border.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your formula suggests &quot;same&quot; border is being used*, because it would be unusual to work with a matrix C starting from an offset corner. If you do use &quot;same&quot; borders, it is important to normalise your input image data pixels to mean 0, otherwise the synthetic border will appear as a strong edge to the kernels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are developing your own CNN code, I would suggest using &quot;valid&quot; border mode, because it is simpler, and a more common approach. It will work just fine for image classification tasks.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Actually I think the formula you have posted is over-simplified or even wrong. I would expect to see the term $I(x+a,y+b)$ so that the top left index of C stays as (1,1) for a &quot;valid&quot; border. For a &quot;same&quot; border, I would expect to see a centered kernel.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-09-27T07:04:51.870" LastActivityDate="2016-09-27T07:04:51.870" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="14233" PostTypeId="1" CreationDate="2016-09-27T08:21:47.867" Score="0" ViewCount="627" Body="&lt;p&gt;I have read quite a lot about discretization techinques, measuring WoE and IV etc., but the basic question - when is it worth to use binning and when isn't - is not entirely clear. On the one hand, the result of discretization seems quite &quot;pretty&quot; and more easy to handle for me, but on the other hand it is always emphasized that binning always causes information loss. Could you provide some clues when should one use this technique?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-11-26T13:37:20.580" Title="When to apply binning?" Tags="|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14234" PostTypeId="2" ParentId="14187" CreationDate="2016-09-27T08:43:52.017" Score="33" Body="&lt;p&gt;In addition to the answer above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Model parameters&lt;/em&gt; are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: word frequency, sentence length, noun or verb distribution per sentence, the number of specific character n-grams per word, lexical diversity, etc. &lt;em&gt;Model parameters&lt;/em&gt; differ for each experiment and depend on the type of data and task at hand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Model hyperparameters&lt;/em&gt;, on the other hand, are common for similar models and cannot be learnt during training but are set beforehand. A typical set of hyperparameters for NN include the number and size of the hidden layers, weight initialization scheme, learning rate and its decay, dropout and gradient clipping threshold, etc. &lt;/p&gt;&#xA;" OwnerUserId="22012" LastActivityDate="2016-09-27T08:43:52.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14235" PostTypeId="2" ParentId="11699" CreationDate="2016-09-27T09:08:15.150" Score="7" Body="&lt;p&gt;&lt;strong&gt;Max Pooling&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So suppose you have a layer P which comes on top of a layer PR. Then the forward pass will be something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P_i = f(\sum_j W_{ij} PR_j)$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P_i$ is the activation of the ith neuron of the layer P, f is the activation function and W are the weights. So if you derive that, by the chain rule you get that the gradients flow as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_j) = \sum_i grad(P_i) f^\prime W_{ij}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now, if you have max pooling,  $f = id$ for the max neuron and $f = 0$ for all other neurons, so $f^\prime = 1$ for the max neuron in the previous layer and $f^\prime = 0$ for all other neurons. So:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_{max\ neuron}) = \sum_i grad(P_i) W_{i\ {max\ neuron}}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$grad(PR_{others}) = 0.$&lt;/p&gt;&#xA;" OwnerUserId="23416" LastActivityDate="2016-09-27T09:08:15.150" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14236" PostTypeId="2" ParentId="14223" CreationDate="2016-09-27T10:10:40.387" Score="1" Body="&lt;p&gt;These learning curves seem unusual - the training accuracy should start high and get lower as more samples are added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that your data is sequential (it has some kind of time dependency). sklearn's learning_curve function does not seem to shuffle the data (should it?), so the training accuracy can change/increase once new structures appear in the data over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a notebook trying to reproduce the effect:&#xA;&lt;a href=&quot;https://gist.github.com/stmax82/79b744877b0a482f8739d372c4777e0d&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/stmax82/79b744877b0a482f8739d372c4777e0d&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two images from the notebook:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning curves on the &lt;strong&gt;shuffled data&lt;/strong&gt; look like that (as expected):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/I5n35.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/I5n35.png&quot; alt=&quot;learning curves on shuffled data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the learning curves on the &lt;strong&gt;original / sequential data&lt;/strong&gt; look like that (unusual because training accuracy is rising over time):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lCAZf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lCAZf.png&quot; alt=&quot;learning curves on original data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's just my try at explaining your learning curves. It might be something completely different... try shuffling your data to find out.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-09-27T10:10:40.387" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14237" PostTypeId="1" CreationDate="2016-09-27T10:39:24.570" Score="1" ViewCount="5475" Body="&lt;p&gt;I was going through a paper related to feature selection wherein I constantly came across the term Laplacian Scores which I was not able to understand. Can anyone explain their importance in feature selection?&lt;/p&gt;&#xA;" OwnerUserId="24235" LastActivityDate="2019-11-24T15:25:38.890" Title="What are Laplacian scores and how do they effect feature selection?" Tags="|feature-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14238" PostTypeId="2" ParentId="14233" CreationDate="2016-09-27T11:59:08.250" Score="1" Body="&lt;p&gt;Mostly this is avoided if reasonably possible in my experience, but:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One scenario where you may want to consider binning is if you are doing modeling with a logistic model.  The coefficients indicate that per unit increase of the IV, the log odds of the dependent variable will increase by ___.  The important distinction is that we are assuming a linear relationship in the logit by saying &quot;per unit increase, we see ___ increase&quot;, when in fact it may not be linear.  To fix this, one possible solution is two bin/discretize your IV to make the relationship more linear, and this interpretable.  This is only important if you are interested in interpretability, not just prediction.  There are also other techniques to account for this that may be preferable (like &lt;a href=&quot;https://en.wikipedia.org/wiki/Generalized_additive_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;generalized additive models&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another scenario is for certain kinds of &quot;big&quot; data where the signals can shift a little bit on an axis due to measurement/machine error, and binning techniques allow you to make sure that the correct signal is matched up closely to other runs where the signal may be in a slightly different position. like below:&#xA;you may bin 65,000 x-axis values into 3200 x-axis values and average their appropriate y-values.  This reduces dimensionality and accounts for the minor shifts in x-axis position that can occur.  e.g. bin 1324 will likely capture the same signal across all other runs, where if you specificed bin 23,950, it might correspond to bin 24,000 on another run of the machine/technique, and your columns wouldn't match up correctly between samples.  In the example below, by binning you are making sure each respective compound is correctly captured, considering slight x-axis variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.unirioja.es/gsoe/espectro.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://www.unirioja.es/gsoe/espectro.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least, among several other factors seen &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/questions/41227/justification-for-low-high-or-tertiary-splits-in-anova/41233#41233&quot;&gt;here&lt;/a&gt;, you are purposefully dropping information for the model, which means you will lose power, as you mentioned.  Also, when you are deciding a cutpoint to (for example) categorize a continuous variable as high vs low, how do you choose the cutpoint?  Can you defend your choice, or is it somewhat arbitrary?  Did you choose multiple cutpoints to see which one favored your results and thus committed the tempting garden of forking paths/multiple testing?&lt;/p&gt;&#xA;" OwnerUserId="14913" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-09-27T11:59:08.250" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14239" PostTypeId="2" ParentId="14237" CreationDate="2016-09-27T12:59:06.537" Score="2" Body="&lt;p&gt;I don't think it can be explained any better than the original paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://papers.nips.cc/paper/2909-laplacian-score-for-feature-selection.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://papers.nips.cc/paper/2909-laplacian-score-for-feature-selection.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="29575" LastEditDate="2019-11-24T15:25:38.890" LastActivityDate="2019-11-24T15:25:38.890" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14242" PostTypeId="2" ParentId="11069" CreationDate="2016-09-27T15:47:05.657" Score="1" Body="&lt;p&gt;It has many reasons to be (performance, design, storage, compression, evaluation of data structures). The principal reason is that all structures are verified in the practice, but you can make you own data structure and show a new mode to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even the google has a paper for verify that he works fine, if you has your own data structure and design, I suggest you choose a database with the correct size for your experiment, search for the precision and recall values for your own collection and make it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other reasons that you can see is that the information has different requirements about performance, compression, storage and hardware.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost of gigabyte for main memory is very different between RAM and HD, when you have a lot of servers your strategy of cost reduction is a reason of improve low cost storage and no high hardware requirements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's clear when you have a collection with Terabytes of data, many cultures or many countries. (google has a paper about his cluster).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think simple, a collection with 50GB is a small collection, but how much cost a server with 60GB of RAM and how much cost a server with 50GB of HD?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is clear and the it's reflects in the data structures (B-tree is good for secondary memory and simple hastables are good and fast in main memory).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you decide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A old good reference is the chapter 5 of Managing Gigabytes (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1558605703&quot; rel=&quot;nofollow&quot;&gt;https://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703/ref=sr_1_1?s=books&amp;amp;ie=UTF8&amp;amp;qid=1474990622&amp;amp;sr=1-1&amp;amp;keywords=managing+gigabytes&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chapter shows a table with different results for many data structures and types of memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The google's paper shows the mode for verify your implementation and design:&#xA;&lt;a href=&quot;http://infolab.stanford.edu/~backrub/google.html&quot; rel=&quot;nofollow&quot;&gt;http://infolab.stanford.edu/~backrub/google.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about the cluster and jobs:&#xA;&lt;a href=&quot;http://static.googleusercontent.com/media/research.google.com/pt-BR//pubs/archive/43438.pdf&quot; rel=&quot;nofollow&quot;&gt;http://static.googleusercontent.com/media/research.google.com/pt-BR//pubs/archive/43438.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24218" LastActivityDate="2016-09-27T15:47:05.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14243" PostTypeId="2" ParentId="12986" CreationDate="2016-09-27T19:34:28.293" Score="2" Body="&lt;p&gt;Actually, recently people have been trying much lower precision in neural nets: 1-2-5 scheme (1 bit weights, 2 bit activations, and 5 bit gradients) seems to work well for easy datasets (MNIST and CIFAR-10). However, on ImageNet the results are significantly lower than those with full-precision (16 or 32 bit).&#xA;To achieve state of the art, convnets don't need more than 16 bits for training, but current RNNs might need more. For inference, on ImageNet, 4-5 bit weights (stochastically rounded from full precision) should be enough.   &lt;/p&gt;&#xA;" OwnerUserId="24724" LastActivityDate="2016-09-27T19:34:28.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14244" PostTypeId="1" CreationDate="2016-09-27T20:00:47.313" Score="0" ViewCount="460" Body="&lt;p&gt;For the simple AND learning with a perceptron, it is required to have two inputs x1 and x2 and one target data y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UqV7n.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UqV7n.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most AND examples (such as in the book &quot;Fundamentals of neural network-fausett&quot;) have this network topology in the figure above, that is 2 inputs(x1,x2) and 1 output(o). My question is, how do we give the network the target data (y) if there is only two inputs(x1,x2)?&lt;/p&gt;&#xA;" OwnerUserId="24728" LastActivityDate="2016-09-28T14:16:26.107" Title="neural network - target data" Tags="|classification|neural-network|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14245" PostTypeId="1" CreationDate="2016-09-27T20:46:14.797" Score="0" ViewCount="367" Body="&lt;p&gt;I want to compare which technique has higher memory utilization while training on the same dataset&lt;/p&gt;&#xA;" OwnerUserId="2504" LastActivityDate="2016-09-28T00:22:30.130" Title="Does anyone have memory utilization benchmark for random forest and xgboost?" Tags="|machine-learning|data-mining|random-forest|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14248" PostTypeId="2" ParentId="14245" CreationDate="2016-09-28T00:22:30.130" Score="2" Body="&lt;p&gt;Try to use &lt;a href=&quot;https://pypi.python.org/pypi/memory_profiler&quot; rel=&quot;nofollow&quot;&gt;memory_profiler&lt;/a&gt;. &#xA;All what you need to do is to decorate your training function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;@profile&#xA;def train_xgb():&#xA;   gb = xgb.XGBClassifier(learning_rate=0.1, n_estimators=100, subsample=0.8, max_depth=6)&#xA;   gb.fit(new_trainX,new_trainY)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21988" LastActivityDate="2016-09-28T00:22:30.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14249" PostTypeId="1" CreationDate="2016-09-28T00:31:38.757" Score="1" ViewCount="151" Body="&lt;p&gt;I've started interviewing for data science roles and I've noticed that a lot of companies ask traditional computer science questions. I have no formal CS education. In fact, I was a business major in college which is pretty much the opposite. I don't know anything about traditional CS algorithms like binary search, bubble sort, or dynamic programming. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a very strong background in machine learning / data science but pretty much everything I know is through years of Googling, doing interesting ML side projects, Kaggle, and generally learning things the hard way. I know that a data scientist needs to be able to code. Ironically I'm a professional software engineer and I code all day. I just don't have a theoretical background. Do I need to read something like &lt;a href=&quot;http://bayanbox.ir/view/4177858657730907268/introduction-to-algorithms-3rd-edition.pdf&quot; rel=&quot;nofollow&quot; title=&quot;asdf&quot;&gt;Introduction to Algorithms&lt;/a&gt; and brush up on CS undergrad work before I start applying to data science jobs in earnest? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Please note:&lt;/strong&gt; I know that there are a lot of other questions on here about what to study to become a good data scientist, but this question is subtly different. I personally disagree that knowledge of traditional CS algorithms is relevant for most day-to-day data science work and that knowing these things makes you a better data scientist.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most machine learning algorithms (gradient boosting, random forests, linear models, SVMs, neural nets, etc) are available in easy-to-use libraries like caret (R); scikit-learn, and TensorFlow (Python); or H2O, and MLlib (Scala/Java). There are also easy-to-use tools, like Spark, to make these algorithms scale. I feel that understanding how the learning algorithms (like gradient descent) work is relevant, but I don't agree that recursion or dynamic programming is relevant. Am I wrong? Should I be more open minded?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-09-28T00:53:14.787" LastActivityDate="2016-09-28T00:55:07.103" Title="How prevalent are traditional CS questions in data science interviews?" Tags="|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14250" PostTypeId="2" ParentId="14249" CreationDate="2016-09-28T00:41:51.907" Score="2" Body="&lt;p&gt;It depends on whether you're going to write production code or even pseudo-code that will be implemented by others. If so, yes you need computer science skills. If you're going to merely analyze data, you can technically get away without it, but some companies (e.g., Facebook) expect all technical employees to know computer science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computer science is useful even to pure analysts because it will help you implement algorithms not covered by your libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I make use of recursion and dynamic programming, for example, but I wrote production code. That's my deliverable, after analysis and prototyping.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-09-28T00:55:07.103" LastActivityDate="2016-09-28T00:55:07.103" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14251" PostTypeId="1" CreationDate="2016-09-28T05:57:38.303" Score="2" ViewCount="197" Body="&lt;p&gt;I have a dataset for a supervised learning task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row is a vector with a value of pixmap value in a range [0,255] of gray colormap, each vector is labeled with a character. I have to assign each vector with a character.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some methods that I can try to pre-process the data to gain better accuracy?&lt;/p&gt;&#xA;" OwnerUserId="24740" LastEditorUserId="26596" LastEditDate="2016-12-05T19:56:33.490" LastActivityDate="2017-12-01T10:13:58.930" Title="What are some method for pre-processing data in OCR?" Tags="|supervised-learning|preprocessing|methods|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14252" PostTypeId="1" AcceptedAnswerId="14270" CreationDate="2016-09-28T06:48:41.113" Score="6" ViewCount="5634" Body="&lt;p&gt;So, I have a dataset with daily operating conditions for different machines and a flag saying if it failed or not.&#xA;Here is a snapshot of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t2AZ8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t2AZ8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use survival analysis or any other algorithm to calculate when the machine is expected to fail in the future?&#xA;What I understand is that I can use survival package in R, but I am not able to use it for a time series data.&lt;/p&gt;&#xA;" OwnerUserId="24742" LastActivityDate="2016-09-29T07:15:59.763" Title="How to use survival analysis for predictive maintenance for time series data?" Tags="|predictive-modeling|time-series|survival-analysis|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14254" PostTypeId="2" ParentId="5941" CreationDate="2016-09-28T12:26:50.450" Score="0" Body="&lt;p&gt;You may encounter this problem of identical keywords when the index of the documents in the corpus is wrong, i.e. the list of occurrences vector has bad document ids (for instance the same document id for all documents):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;id_doc_0&amp;gt;, &amp;lt;vector of occurrences for doc 0&amp;gt;&#xA;&amp;lt;id_doc_1&amp;gt;, &amp;lt;vector of occurrences for doc 1&amp;gt;&#xA;...&#xA;&amp;lt;id_doc_1&amp;gt;, &amp;lt;vector of occurrences for doc 1&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4784" LastActivityDate="2016-09-28T12:26:50.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14257" PostTypeId="2" ParentId="14244" CreationDate="2016-09-28T14:16:26.107" Score="2" Body="&lt;p&gt;The target data is the desired output that happens to be in the real word. This information is known to you already(given the problem is supervised learning problem). So, for an 'AND' gate the target value is the vector stored in y on the right hand side as shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1    x2      y(desired output)&#xA;0     0          0&#xA;0     1          0&#xA;1     0          0&#xA;1     1          1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So this vector y = [0,0,0,1] is your target to these input types that you feed into the network as labels corresponding to your above input data.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-09-28T14:16:26.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14258" PostTypeId="2" ParentId="13722" CreationDate="2016-09-28T14:27:19.407" Score="3" Body="&lt;p&gt;The words &lt;code&gt;Going, Gone, Goes&lt;/code&gt; are considered to be similar in only one context i.e. they all have the same root word &lt;code&gt;Go&lt;/code&gt;. This is known as Stemming/Lemmatization, both of which are special cases of normalization process. So, in order to assign some value to these words, you have to first lemmatize them. In order to get more details about this, I am attaching two links which can be helpful in your case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/771918/how-do-i-do-word-stemming-or-lemmatization&quot;&gt;https://stackoverflow.com/questions/771918/how-do-i-do-word-stemming-or-lemmatization&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-09-28T14:27:19.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14260" PostTypeId="1" CreationDate="2016-09-28T17:56:14.807" Score="1" ViewCount="28" Body="&lt;p&gt;I'm running an experiment where I need to collect and analyse participants' browsing and search histories. The design of the experiment is similar to an &quot;instrumented user panel&quot;, described here:&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.8971&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.8971&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the classic case, participants must install some kind of logger on their computers, which collects and sends browsing data to the researcher behind the scenes. Finding such tools is where I get stuck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could, of course, just ask my participants to export their browsing histories and send them to me every night, but I was hoping there would be something more automated out there, with potential bells and whistles such as annotations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Please let me know if this is the wrong Stack Exchange; I must be really struggling with finding the right keywords for this...&lt;/p&gt;&#xA;" OwnerUserId="24759" LastActivityDate="2021-07-10T18:10:22.440" Title="What tools are out there to collect participants' browsing and/or search data as part of an experiment?" Tags="|tools|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14261" PostTypeId="1" AcceptedAnswerId="14263" CreationDate="2016-09-28T18:25:04.097" Score="1" ViewCount="376" Body="&lt;p&gt;Thanks in advance for reading my question!&lt;/p&gt;&#xA;&lt;p&gt;I've been using CNNs to classify text using Keras and TF. My data is strings &amp;quot;I read the news&amp;quot; or &amp;quot;I read machine learning news&amp;quot; and my labels are tags: Data Science, Reporter, Child...&lt;/p&gt;&#xA;&lt;p&gt;My issues is that each text can have multiple labels attached to it. How should I construct my target such that I can capture each of those multiple outputs?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Description, Tag&#xA;&amp;quot;I read the news&amp;quot;, Child&#xA;&amp;quot;I read the news&amp;quot;, Reporter&#xA;&amp;quot;I read machine learning news&amp;quot;, Data Science&#xA;&amp;quot;I read machine learning news&amp;quot;, Reporter&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24760" LastEditorUserId="29169" LastEditDate="2021-02-16T15:37:02.187" LastActivityDate="2021-02-16T15:37:02.187" Title="CNN tagging such that each input could have multiple tags" Tags="|machine-learning|keras|tensorflow|convolutional-neural-network|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14262" PostTypeId="2" ParentId="10340" CreationDate="2016-09-28T18:37:44.687" Score="2" Body="&lt;p&gt;&lt;code&gt;pymc&lt;/code&gt; will not provide you pretty sklearn-style &lt;code&gt;.predict&lt;/code&gt; method for this case, however you can do it on your own. The idea is simple enough: you should draw coefficients for the classifier using pymc, and after it use them for the classifier itself manually. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see a very basic example at &lt;a href=&quot;https://healthyalgorithms.com/2012/03/05/mcmc-in-python-a-random-effects-logistic-regression-example/&quot; rel=&quot;nofollow&quot;&gt;this blogpost&lt;/a&gt; or more complicated case at &lt;a href=&quot;http://pymc-devs.github.io/pymc3/notebooks/GLM-logistic.html&quot; rel=&quot;nofollow&quot;&gt;pymc3 documentation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24761" LastActivityDate="2016-09-28T18:37:44.687" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14263" PostTypeId="2" ParentId="14261" CreationDate="2016-09-28T19:38:35.913" Score="-1" Body="&lt;p&gt;CNN's (generally) use softmax as activation function at the last layer, which gives a probability distribution over all possible labels. You use some kind of loss function to optimize the CNN to generate &quot;right&quot; label for a novel sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Loss function used in such case is cross-entropy loss or KL-divergence which measures how close two distributions are. For multiclass classification (different from your case, where each input example can have only one label), desired distribution is (0, 0, .... , 1, ..., 0) where 1 appears at the position of actual label. Loss function penalizes the distribution which is &quot;far&quot; from this distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your case you can encode your output as (0, 0, 1, 0, ... 1, ..,) so on where you put ones at positions of the labels which are associated with your input and use KL-divergence as loss function and optimize your CNN to  minimize the loss.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="13518" LastEditDate="2016-09-28T19:45:03.227" LastActivityDate="2016-09-28T19:45:03.227" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14264" PostTypeId="1" CreationDate="2016-09-28T21:29:33.877" Score="-2" ViewCount="1563" Body="&lt;p&gt;I'm working with government infrastructure public tenders data and want to build a predictive model. I'm want to train a model to predict if a company will or will not participate in a public tender. I have the companies list with several attributes associated to them; company location, avg. historical bid amount, core business, etc. and also the bids they have participated in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the problem is, that even though I want my output labels to be binary (1=will participate, 0=will not participate), each tender also has associated attributes to it, like govt department of the tender (healthcare, highways etc.), $ amount of the tender, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to predict if a company with attributes A,B,C will participate in a tender with attributes X,Y,Z. How would you build this model and which algorithms you think are best?&lt;/p&gt;&#xA;" OwnerUserId="24767" LastActivityDate="2017-04-27T16:03:30.630" Title="Predictive Model with sort-of-non-binary labels" Tags="|machine-learning|python|dataset|scikit-learn|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14266" PostTypeId="1" AcceptedAnswerId="14622" CreationDate="2016-09-28T23:31:37.300" Score="1" ViewCount="568" Body="&lt;p&gt;I'm new in data analysis area and didn't have very strong sattistical background...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to use dimensional reduction algorithms such as PCA because I'm still collecting data, and there are too many columns, I hope to remove some columns first &lt;strong&gt;before&lt;/strong&gt; linking different sources of data together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Playing with PCA and machine learning models will be &lt;strong&gt;after&lt;/strong&gt; linking those sources of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, at this &quot;before&quot; stage, I'm trying to use R &lt;code&gt;library caret&lt;/code&gt;, &lt;code&gt;findCorrelation()&lt;/code&gt; method, but I could not find how does this method work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is, I should use this method &lt;strong&gt;after&lt;/strong&gt; data cleaning (such as, deal with missing data, outliers, data imbalance) &lt;strong&gt;or it doesn't matter to do it before or after&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="23470" LastActivityDate="2016-10-18T23:04:35.420" Title="Would it make a difference to clean the data before using R findCorrelation()?" Tags="|r|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14267" PostTypeId="2" ParentId="14264" CreationDate="2016-09-29T00:13:53.080" Score="0" Body="&lt;p&gt;The first step you need to produce the input data. I suggest use python pandas library and then you need to label your data. It's mean you need to assign class (Whether the company participated or not) to your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I would suggest to use some of the sklearn models. As a baseline you could try kNN and then Gradient Boosted Classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the model need to be understandable by someone (your supervisor or Prime Minister)? In this case I suggest to use DecisionTrees (easy to interpret). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea how hard your problem is. You didn't show us the distributions of the input features grouped by type. &lt;/p&gt;&#xA;" OwnerUserId="21988" LastActivityDate="2016-09-29T00:13:53.080" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14270" PostTypeId="2" ParentId="14252" CreationDate="2016-09-29T07:15:59.763" Score="6" Body="&lt;p&gt;Whenever your task includes something like &quot;...when XY will fail...&quot;, i'd say go for survival analysis first, it is easy and fast and it will give you overview of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your data you can either turn them into intervals to be able to plot survival curves, or proceed directly to Cox regression, which can work with continuous data and will yield the hazard ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can start with Kaplan-Meier curve (as bonus there are Confidence intervals):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;km &amp;lt;- survfit(Surv(datetime, Failed) ~ 1,conf.int=0.90, conf.type=&quot;log-log&quot;, data=Dataset)&#xA;summary(km)&#xA;plot(km, xlab=&quot;month&quot;, ylab=&quot;estimated S(t)&quot;, main=&quot;Kaplan-Meier with log-log, C.I.=90%&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The curve will look somewhat like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ko6C0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ko6C0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further you can split the curve to see if any of your parameters have different influence. You can do that by simply replace the &lt;code&gt;~1&lt;/code&gt; with something like &lt;code&gt;~AttributeX&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you should get this kind of plot:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Xh2Vl.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xh2Vl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course R will give you various tests and p-values as well, such as Log-Rank to verify whether the influence is significant (&lt;code&gt;survdiff&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can proceed to Cox Regression, which will tell you what is the hazard ratio (=whether the attribute influences the hazard positively or negatively and to what extent). It looks like this in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cox&amp;lt;-coxph(Surv(datetime, Failed)~AttributeX, data=pbc)&#xA;summary(cox)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is good practice to verify the assumptions - proportional hazard and functional form (again R will give you p-values or you can plot the residuals - Martingale or Schoenfeld).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested to know &lt;strong&gt;WHEN&lt;/strong&gt; the event will occur, search for Accelerated Failure Time models, which will give you the parametric survival time distribution, where you can simply put the time and obtain the probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wei&amp;lt;-survreg(Surv(datetime, Failed)~ AttributeA + AttributeB + AttributeC,data=Dataset) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are more possible distribution, you can check which one fits the best your data. I have never done the prediction, but there is a function &lt;code&gt;predict&lt;/code&gt; which is described in &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/survival/html/predict.survreg.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt; or there are already similar questions with answers on Crossvalidated, such as &lt;a href=&quot;https://stats.stackexchange.com/questions/127471/how-to-make-prediction-in-survival-analysis-using-r&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-09-29T07:15:59.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14273" PostTypeId="1" AcceptedAnswerId="14277" CreationDate="2016-09-29T10:07:14.980" Score="4" ViewCount="144528" Body="&lt;p&gt;I have a factor variable in my data frame with  values where in the original CSV &quot;NA&quot; was intended to mean simply &quot;None&quot;, not missing data. Hence I want replace every  value in the given column with &quot;None&quot; factor value. I tried this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF$col[is.na(DF$col)] &amp;lt;- &quot;None&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but this throws the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Warning message:&#xA;In `[&amp;lt;-.factor`(`*tmp*`, is.na(DF$col), value = c(NA, NA,  :&#xA;  invalid factor level, NA generated&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess this is because originally there is no &quot;None&quot; factor level in the column, but is it the true reason? If so, how could I add a new &quot;None&quot; level to the factor?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In case you would ask why didn't I convert NAs into &quot;None&quot; in the &lt;code&gt;read.csv&lt;/code&gt; phase: in other columns NA really does mean missing data).&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2019-07-06T05:39:24.010" Title="How to replace NA values with another value in factors in R?" Tags="|r|missing-data|" AnswerCount="5" CommentCount="1" ClosedDate="2018-12-15T05:14:15.637" ContentLicense="CC BY-SA 3.0" />
  <row Id="14275" PostTypeId="2" ParentId="14222" CreationDate="2016-09-29T10:35:30.313" Score="1" Body="&lt;p&gt;Maybe you can use Python with &lt;a href=&quot;https://docs.python.org/2/tutorial/datastructures.html#dictionaries&quot; rel=&quot;nofollow&quot;&gt;dictionaries&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can set a group of word in the dictionary that could be filled each time with new words you find.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To find new word (if the pattern remains the same as your example) you&#xA;  can grab the &quot;brand&quot; (Samsung) and the next word will be the model&#xA;  (UE65KS7500).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a good &lt;a href=&quot;http://textminingonline.com/getting-started-with-keyword-extraction&quot; rel=&quot;nofollow&quot;&gt;resource&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17957" LastActivityDate="2016-09-29T10:35:30.313" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14276" PostTypeId="2" ParentId="14156" CreationDate="2016-09-29T10:43:31.693" Score="0" Body="&lt;p&gt;I can't comment 'cause of lack of reputation.&#xA;Have you to use only &quot;Agglomerative clustering&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's better K-Means clustering for your usage. You can detect few difference with k-Means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If need to use the &quot;Agglomerative clustering&quot; you shoul tweak the dissimilarity measure.&lt;/p&gt;&#xA;" OwnerUserId="17957" LastActivityDate="2016-09-29T10:43:31.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14277" PostTypeId="2" ParentId="14273" CreationDate="2016-09-29T11:02:15.480" Score="6" Body="&lt;p&gt;You need to add &quot;None&quot; to the factor level and refactor the column DF$col. I added an example script using the iris dataset. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- iris&#xA;&#xA;# set 20 Species to NA&#xA;set.seed(1234)&#xA;s &amp;lt;- sample(nrow(df), 20)&#xA;df$Species[s] &amp;lt;- NA&#xA;&#xA;# Get levels and add &quot;None&quot;&#xA;levels &amp;lt;- levels(df$Species)&#xA;levels[length(levels) + 1] &amp;lt;- &quot;None&quot;&#xA;&#xA;# refactor Species to include &quot;None&quot; as a factor level&#xA;# and replace NA with &quot;None&quot;&#xA;df$Species &amp;lt;- factor(df$Species, levels = levels)&#xA;df$Species[is.na(df$Species)] &amp;lt;- &quot;None&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-09-29T11:02:15.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14279" PostTypeId="1" CreationDate="2016-09-29T12:21:28.973" Score="-1" ViewCount="1970" Body="&lt;p&gt;I have a data frame with both numeric and categorical columns and I want to apply &lt;code&gt;table&lt;/code&gt; function to all categorical columns (print out count of level occurances per column). Is there an elegant solution to this, perhaps with &lt;code&gt;lapply&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-09-29T13:22:32.077" Title="How to apply table to categorical columns in R?" Tags="|r|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14280" PostTypeId="2" ParentId="14279" CreationDate="2016-09-29T13:02:04.130" Score="0" Body="&lt;p&gt;Get the &lt;em&gt;factor&lt;/em&gt; class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;data.class &amp;lt;- sapply(df, class)&#xA;col &amp;lt;- data.class[data.class==&quot;factor&quot;]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apply &lt;em&gt;table&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;with(df, sapply(df[,which(names(df) %in% names(col))], table))&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="24528" LastEditorUserId="24528" LastEditDate="2016-09-29T13:22:32.077" LastActivityDate="2016-09-29T13:22:32.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14281" PostTypeId="2" ParentId="14279" CreationDate="2016-09-29T13:13:47.920" Score="0" Body="&lt;p&gt;I have figured out the solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Filter out categorical columns&#xA;cat_DF &amp;lt;- DF[, sapply(DF, is.categorical)]&#xA;&#xA;#Apply table to the filtered columns&#xA;lapply(cat_DF, function(x) table(x, useNA=&quot;ifany&quot;)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is a named list with the respective categorical distributions.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-09-29T13:13:47.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14282" PostTypeId="1" CreationDate="2016-09-29T15:16:07.503" Score="17" ViewCount="1664" Body="&lt;p&gt;I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most often single models are visualized with a decision tree or &lt;a href=&quot;http://guizzetti.ca/blogs/lenny/wp-content/uploads/2012/04/3state_markov.jpg&quot; rel=&quot;noreferrer&quot;&gt;transition state diagram&lt;/a&gt;. However, with multiple different models what are potential visualizations that could communicate the transition states that differentiate each model?   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen other people use a &lt;a href=&quot;http://www.bmj.com/highwire/markup/587855/expansion?width=1000&amp;amp;height=500&amp;amp;iframe=true&amp;amp;postprocessors=highwire_figures%2Chighwire_math&quot; rel=&quot;noreferrer&quot;&gt;table&lt;/a&gt; to depict different models and the transition states.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For clarity, I am not referring to a transition probabilities chart but a method of communicating the differences between multiple models.   &lt;/p&gt;&#xA;" OwnerUserId="15343" LastEditorUserId="71218" LastEditDate="2019-05-14T16:51:49.547" LastActivityDate="2020-03-05T07:22:02.370" Title="Visualization of multiple Markov models" Tags="|python|r|visualization|markov-process|simulation|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="14283" PostTypeId="1" AcceptedAnswerId="14305" CreationDate="2016-09-29T16:13:53.433" Score="0" ViewCount="340" Body="&lt;p&gt;I am using the Association Rules algorithm using this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html&quot; rel=&quot;nofollow&quot;&gt;http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="24790" LastActivityDate="2016-10-01T09:16:49.527" Title="Cut-Off using Frequent Pattern Mining - Spark Mllib" Tags="|data-mining|apache-spark|association-rules|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14287" PostTypeId="1" AcceptedAnswerId="14340" CreationDate="2016-09-30T04:07:20.720" Score="1" ViewCount="72" Body="&lt;p&gt;Morning,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of data of which I am positive an event (target data / the event I want to predict going forward) occurred within a two week time frame, however I am unsure when it happened within this time frame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can get daily or more frequent feature data, but the target data only appears in 2 week gaps or in some cases 4 week gaps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I use an average of any feature during the timeframe or gather the feature data on the same day as when the target data is available but is there a more appropriate / better way? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the data for machine learning purposes.&lt;/p&gt;&#xA;" OwnerUserId="24259" LastActivityDate="2016-10-03T21:46:56.423" Title="Postitive event but unsure when it occurred in time" Tags="|data-cleaning|data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14288" PostTypeId="1" CreationDate="2016-09-30T04:45:32.147" Score="3" ViewCount="2659" Body="&lt;p&gt;Thought maybe somebody here could help us solve a mystery (&lt;a href=&quot;https://github.com/dmlc/xgboost/issues/1623&quot; rel=&quot;nofollow&quot;&gt;https://github.com/dmlc/xgboost/issues/1623&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are trying to build a xgboost prediction function in R for a model that was trained in Python and the results don't match. See below for an example of how to reproduce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fairly new to xgboost, particularly using it across languages, so may be missing something obvious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Steps to reproduce:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Download this model file: &lt;a href=&quot;http://ml.stat.purdue.edu/hafen/WTKG.model&quot; rel=&quot;nofollow&quot;&gt;http://ml.stat.purdue.edu/hafen/WTKG.model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Run this script in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(xgboost)&#xA;mod &amp;lt;- xgb.load(&quot;WTKG.model&quot;)&#xA;x &amp;lt;- c(91, 9, 9, NA, NA, 273, 20, 170, NA, NA, 14, 14, 0,&#xA;  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,&#xA;  0.95490834613, 1, 90, 0.95490834613, 1, 90,&#xA;  0.93087973569, 357, -266, 0.93087973569, 357, -266,&#xA;  0.95490834613, NA, 0.93087973569, NA, NA, NA)&#xA;d &amp;lt;- xgb.DMatrix(matrix(x, nrow = 1), missing = NA)&#xA;predict(mod, d)&#xA;# [1] 0.6483372&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(3) Run this script in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import xgboost as xgb&#xA;&#xA;bst = xgb.Booster({'nthread': 4})&#xA;bst.load_model('WTKG.model')&#xA;x = [91, 9, 9, np.nan, np.nan, 273, 20, 170, np.nan, np.nan, 14, 14, 0,&#xA;  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,&#xA;  0.95490834613, 1, 90, 0.95490834613, 1, 90,&#xA;  0.93087973569, 357, -266, 0.93087973569, 357, -266,&#xA;  0.95490834613, np.nan, 0.93087973569, np.nan, np.nan, np.nan]&#xA;d = xgb.DMatrix(data=[x], missing=np.nan)&#xA;bst.predict(d)[0]&#xA;# 1.3775804&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24813" LastActivityDate="2016-12-31T23:17:31.200" Title="xgboost Predictions from R and Python don't match" Tags="|xgboost|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14289" PostTypeId="1" CreationDate="2016-09-30T05:28:14.723" Score="-1" ViewCount="845" Body="&lt;p&gt;Can we find the spam tweets and posts in twitter and facebook?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, How to Identify them.Is there any labelled dataset available as spam and non-spam for spam detection of tweets and posts of twitter,facebook?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what algorithm is used for detection of spam for Japanese &amp;amp; Chinese languages?   &lt;/p&gt;&#xA;" OwnerUserId="20585" LastActivityDate="2017-10-28T05:35:11.907" Title="Does Twitter and Facebook has spam tweets and posts ? Can I get any example dataset for finding spam?" Tags="|machine-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14290" PostTypeId="1" CreationDate="2016-09-30T08:56:03.210" Score="2" ViewCount="3729" Body="&lt;p&gt;I have read the following about MDS in a book:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;using MDS requires an understanding of the individual feature's units;&#xA;  maybe we are using features that cannot be compared using the&#xA;  Euclidean metric. For instance, a categorical variable, even when&#xA;  encoded as an integer (0= circle, 1= star, 2= triangle, and so on),&#xA;  cannot be compared using Euclidean (is circle closer to star than to&#xA;  triangle?).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I accept the statement above, but it raises a few question about the application of MDA:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Given the fact that many conventional data sets contain categorical&#xA;features, does it mean that MDA cannot fit to these sets?&lt;/li&gt;&#xA;&lt;li&gt;Maybe it would be a solution to change the distance measure type (e. g.&#xA;&quot;Euclidean&quot;) to other, but &lt;code&gt;Sklearn&lt;/code&gt; has no other built-in option,&#xA;not to mention R, where &lt;code&gt;cmdscale&lt;/code&gt; has no option at all to specify&#xA;distance type. How to change this feature in general?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;An additional question: I have read that PCA is a kind of MDS (or vice versa), apart from the fact that the former focuses on variance, the latter on keeping distance. Am I right that the two &quot;converges&quot; somehow (in case of visualization with the two first component, for instance)?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-12-29T13:41:47.223" Title="Multidimensional Scaling with Categorical Data" Tags="|categorical-data|dimensionality-reduction|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14291" PostTypeId="1" CreationDate="2016-09-30T09:44:38.730" Score="0" ViewCount="1138" Body="&lt;p&gt;I'm novice to deep learning and sorry if these questions may look very basic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;******************* First Question *******************&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to predict the next k-characters given some initial text input, where the algorithm is learnt on a piece of text (character sequence).&#xA;I've done some work on predicting only 1-character at a time (using LSTM and Keras wrapper), but my question is how to generate k-characters and not only one. Of course I can generate k-characters recursively using only 1-character prediction algorithm, but I'm wondering if this problem (input text to k-characters) is well known and there exist a state-of-the-art solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I learnt from reading relevant topics, is that this problem is (similar) to &quot;sequence-to-sequence&quot; learning and of course there are many solutions and implementations available (theano, tensorflow, etc).&#xA;Am I correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, given a string of letters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seq_1 = 'aaaaabababcbcbbababbbdcbbbaaabbbaad...'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the algorithm should predict (given a randomly picked input sequence):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;aaabbb -&amp;gt; aad&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(the last 9 characters from the aforementioned example  )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;******************* Second Question *******************&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know, that neural network algorithms are very &quot;data hungry&quot; and in order to make meaningful learning and prediction accuracy, one has to provide a lot of data (compared to shallow learning methods)&#xA;If I have only relatively short strings of letters (50-100 characters), does it make any sense to train seq2seq algorithm (LSTMs network) on such short sequences, but with many bootstrapped random instances? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any theoretical (or some rules of thumb) bounds on the size of the training set and the reliability of the trained algorithm? (overfitting)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite motivated by the &lt;a href=&quot;http://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow&quot;&gt;example&lt;/a&gt; of learning temporal patterns in alphabet with LSTM RNNs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author got very nice results, by random sampling of sequences of varied length and showing them to LSTM RNN. The idea was to randomly generate a relatively large (~1000 instances) data set and then train the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone comment on this way (bootstrapping) of creating a training set from a small sequence? Is it a standard approach? Are there any statistical flaws?&#xA;Many thanks!!&lt;/p&gt;&#xA;" OwnerUserId="14070" LastActivityDate="2018-07-31T20:50:43.493" Title="Learning character sequences and predicting sequences" Tags="|rnn|sequential-pattern-mining|text-generation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14292" PostTypeId="2" ParentId="14288" CreationDate="2016-09-30T10:56:49.783" Score="-1" Body="&lt;p&gt;I think you can get your answer from &lt;a href=&quot;https://www.kaggle.com/c/caterpillar-tube-pricing/forums/t/15885/xgboost-different-between-python-and-r&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2016-09-30T10:56:49.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14293" PostTypeId="1" CreationDate="2016-09-30T11:53:22.803" Score="0" ViewCount="868" Body="&lt;p&gt;I try to apply &lt;strong&gt;non-linear dimension reduction&lt;/strong&gt; in R. As usual in machine learning I have a large data set (100 K rows). I tried the packages&#xA;&lt;code&gt;RDRToolbox&lt;/code&gt; and &lt;code&gt;vegan&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library( RDRToolbox )    &#xA;res.lle = LLE(as.matrix(temp), dim = 3 , k = 10)&#xA;res.iso = Isomap(as.matrix(temp), dim = 3 , k = 10)&#xA;&#xA;library( vegan )&#xA;isomap(dist(temp), ndim=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first algoritms use the distance matrix internally and crash as they want to allocate 32GB. The second approach needs the distance matrix and crashes too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any tricks to do non-linear dimension reduction either with these algorithms? Or are there packages that do not use the raw and full distance matrix? Such approaches have to crash in my setting on my machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only algorithm that works for me is &lt;code&gt;Rtsne&lt;/code&gt; that appearantly does not use the whole distance matrix. &lt;code&gt;largeVis&lt;/code&gt; was too slow so far to let it finish.&lt;/p&gt;&#xA;" OwnerUserId="20884" LastActivityDate="2016-09-30T12:41:52.747" Title="Dimension reduction techniques in R that do not use the full distance matrix" Tags="|machine-learning|r|dimensionality-reduction|distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14294" PostTypeId="2" ParentId="14290" CreationDate="2016-09-30T12:11:45.080" Score="2" Body="&lt;p&gt;MDS only requires a distance matrix which stores the distance between each pair of data examples. How to compute that distance depends on the kind of data you are dealing with. If you only have numerical (real valued) features, you can use the Euclidean distance, but that is not always the case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if you have both numerical and categorical variables, you may apply any metric for mixed data, like for instance &lt;a href=&quot;https://pdfs.semanticscholar.org/38e1/4cac5b37ad1fd20c513e3cc64e9bd37df90a.pdf&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your second question, I need further clarification. What do you mean by &quot;converging? Are you asking if both algorithms converge to the same solution? Is that is your question, the answer is no, because the aim of both algorithms is different. MDS projects your data onto a 2D plane by trying to keep relative distances, whether PCA projects data by focusing on those &quot;directions&quot; in which there is more variability. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2016-09-30T12:11:45.080" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14296" PostTypeId="2" ParentId="14293" CreationDate="2016-09-30T12:41:52.747" Score="3" Body="&lt;p&gt;You can try using an auto-encoder which also is a non-linear dimension reduction technique.  It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose.  It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values as &quot;good&quot; (lowest MSE) as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YjbEJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YjbEJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case the input has 5 variables, and it reduces it down to 2 variables using non-linear transformations.  Once you get it down to the reduced variables (the light blue nodes) it then tries using non-lienar transformations to recreate those original five inputs using a similar nn structure.  The right-five red nodes are then compared to the left-five original inputs to see how much information is encoded in those 2 middle light blue reduced variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found the h2o implementation is pretty easy to use, but finding the correct parameters (number of layers, nodes, learning rate, etc) is still tough to select depending on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.r-bloggers.com/a-little-h2o-deeplearning-experiment-on-the-mnist-data-set/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Heres&lt;/a&gt; a little blog post that goes through the h2o autoencoder with the MNIST data set.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-09-30T12:41:52.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14297" PostTypeId="2" ParentId="14289" CreationDate="2016-09-30T13:38:28.550" Score="1" Body="&lt;p&gt;First on methods and techniques: there are a lot of language-independent spam detection algorithms. The main task is feature engineering: one need to construct features, that will detect behavioral differences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;temporal features (posting hour or minute entropy/autoregression, or even more complicated patterns),&lt;/li&gt;&#xA;&lt;li&gt;social media profile characteristics,&lt;/li&gt;&#xA;&lt;li&gt;other data, that can be gathered through network API (user's client id, activity patterns, friends attributes, etc).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is a kind of &quot;classical&quot; paper on Twitter bot detection: &lt;a href=&quot;https://www.eecis.udel.edu/~hnw/paper/acsac10.pdf&quot; rel=&quot;nofollow&quot;&gt;Who is Tweeting on Twitter: Human, Bot, or Cyborg?&lt;/a&gt;. It's already 6 years old, but could be a good starting point. Bots are being detected through their posting activity patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can use texts not even trying to &quot;understand&quot; (or parse) them. Just try shingle-based methods, locality-sensitive hashes (LSH). I advise to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Nilsimsa_Hash&quot; rel=&quot;nofollow&quot;&gt;Nilsimsa&lt;/a&gt;, but only because that is the only LSH, I used myself. And AFAIK, initially it was built exactly for spam detection. But you can try to search for any other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And on the datasets: I don't know publicly available Facebook and Twitter spam datasets, but here is a dataset with YouTube comments, hope it will be usefull:&#xA;&lt;a href=&quot;http://www.icwsm.org/2013/datasets/datasets/&quot; rel=&quot;nofollow&quot;&gt;Network Analysis of Recurring YouTube Spam Campaigns&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-09-30T13:38:28.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14300" PostTypeId="1" AcceptedAnswerId="14540" CreationDate="2016-09-30T14:58:16.003" Score="4" ViewCount="4831" Body="&lt;p&gt;Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ger = c(2,4,1,3,2,4,4,1,2,3)&#xA;phi = c(3,4,1,2,2,3,3,2,2,2)&#xA;mat = c(1,3,2,4,1,2,2,4,3,1)&#xA;phy = c(2,2,2,5,2,2,3,4,3,3)&#xA;A = cbind(ger,phi,mat,phy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I combine everything to a matrix and scale the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;As = scale(A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I perform a &lt;code&gt;summary&lt;/code&gt;  on the PCA:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;summary(princomp(As), loadings = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which returns the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Importance of components:&#xA;                       Comp.1    Comp.2     Comp.3     Comp.4&#xA;Standard deviation     1.3257523 1.1657791 0.59600603 0.35793402&#xA;Proportion of Variance 0.4882275 0.3775114 0.09867311 0.03558799&#xA;Cumulative Proportion  0.4882275 0.8657389 0.96441201 1.00000000&#xA;&#xA;Loadings:&#xA;     Comp.1 Comp.2 Comp.3 Comp.4&#xA;ger  0.496 -0.502  0.519  0.482&#xA;phi  0.548 -0.443 -0.423 -0.570&#xA;mat  -0.430 -0.572 -0.546  0.435&#xA;phy  -0.518 -0.474  0.503 -0.503&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a few hints for the first component (based on the loadings):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a high positive correlation between german and philosophy and there is also a high positive correlation between math and physics.&lt;/li&gt;&#xA;&lt;li&gt;Who is good in language (german and philosophy) is often worse in MINT (math and physics) and the other way around.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And an idea about the second one, which I cannot interpret:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It's a weighted arithmetic mean over all four variables.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I have no idea how to interpret the &lt;code&gt;Comp. 2&lt;/code&gt;, &lt;code&gt;Comp. 3&lt;/code&gt; and &lt;code&gt;Comp. 4&lt;/code&gt; based on the loadings. Especially because all values of &lt;code&gt;Comp. 2&lt;/code&gt; are all negative, or have the same orientation. Can someone help me? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="15946" LastEditorUserId="14" LastEditDate="2016-10-15T12:03:46.130" LastActivityDate="2016-10-15T12:03:46.130" Title="How to interpret the loading values of a pca?" Tags="|r|dataset|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14301" PostTypeId="2" ParentId="13756" CreationDate="2016-09-30T15:21:05.187" Score="0" Body="&lt;p&gt;Extrapolation?  Happens a lot when your data distributions change over time, so a system that is well modeled in the training set wont know how to deal with values that are not in a similar range.  More of a general term, so it might be what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also has different effects depending on the technique you use.  Something like random forests is not very good at extrapolation, where others like logistic regression can still perform OK.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-09-30T15:21:05.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14302" PostTypeId="2" ParentId="13754" CreationDate="2016-09-30T18:56:02.290" Score="0" Body="&lt;p&gt;There could be multiple reasons. The number of trees and the depth can change your results.  If your model doesn't perform well after selecting the parameters (cross-validation etc.), it's probably because your features are not very predictive, so they get picked almost &quot;randomly&quot; which leads to high standard deviations from tree to tree. &#xA;But there are other possibilities, e.g. it could also be that your features are highly correlated. A little more information would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-09-30T18:56:02.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14304" PostTypeId="2" ParentId="14251" CreationDate="2016-10-01T05:51:36.960" Score="2" Body="&lt;p&gt;You can do &lt;em&gt;image tresholding&lt;/em&gt;, normalize the data first by dividing each elements of the vector by 255, then do thresholding, for each $x_i \in \mathbb{x}$ $x_i=1$ if $threshold\le x_i$ where $\mathbb{x}$ is your data vector, and $x_i$ is the element of vector, you can choose the threshold based on experiment or personaly I choose 0.5. By doing image thresholding we reduce the noise of the image, hence its increase the accuracy&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="24696" LastEditDate="2016-10-04T14:46:42.807" LastActivityDate="2016-10-04T14:46:42.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14305" PostTypeId="2" ParentId="14283" CreationDate="2016-10-01T09:16:49.527" Score="0" Body="&lt;p&gt;Like @SeanOwen pointed out its called &lt;code&gt;support&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;spark.mllib’s FP-growth implementation takes it as a hyper-parameter under &lt;code&gt;minSupport&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Usage:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.fpm.FPGrowth&#xA;&#xA;val transactions: RDD[Array[String]] = ???&#xA;&#xA;val fpg = new FPGrowth().setMinSupport(0.2)&#xA;&#xA;val model = fpg.run(transactions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2016-10-01T09:16:49.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14312" PostTypeId="1" CreationDate="2016-10-02T09:31:27.363" Score="1" ViewCount="3495" Body="&lt;p&gt;I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QvFFP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QvFFP.png&quot; alt=&quot;table snippet here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I have to forecast sales for year 4 from the sales record of each product over 3 years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;N.A is present where the particular product was not sold in that year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.Which algorithm should I use to forecast the sales for year 4? I have heard about ARIMA and xgboost being used for time series data. Can you please help me with that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.Now I am being given a new product which was not sold before. I am being asked to predict its sales for year 4, using the data for sales of other products:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mQ4Ao.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mQ4Ao.png&quot; alt=&quot;table snippet here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should be my approach for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me out here. Thank you for your time and support in advance. I am sorry if my documentation or question is poor.&lt;/p&gt;&#xA;" OwnerUserId="24872" LastActivityDate="2018-08-04T14:04:26.170" Title="Forecasting sales of next year using sales of past years?" Tags="|r|data|xgboost|forecast|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14313" PostTypeId="1" AcceptedAnswerId="14319" CreationDate="2016-10-02T09:34:33.733" Score="2" ViewCount="189" Body="&lt;p&gt;I'm totally naive to &lt;em&gt;Data Science&lt;/em&gt; - that is, the relatively new, somewhat hyped-field that is so popular at the moment. But I'm not naive to data ... as a scientist and researcher I've worked with all sorts in different roles in the past.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm in the lamentable position of having dug a lot of shallow holes, using different software systems and different types of data, and not really being professionally competent in anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is&lt;/strong&gt;, if I want to &quot;get up to speed&quot; with data science, and perhaps leverage the different experiences I've had, how do I approach it? Ideally I'd like to make my research skills marketable - that is, become a data scientist of sorts, but with a greater emphasis on the research/reporting side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming I'm coming from scratch but have demonstrated capacity - I say that because, for example, I've used R before for some projects but after a break of a year or so I need to relearn it every time... Where do I start; how do I unify all these bits and pieces? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what claim can I make to work in this field? (I've worked on all sorts of data, from gigabytes of climate data and earth science, to health registers to longitudinal surveys ... but &lt;em&gt;none&lt;/em&gt; of it under the moniker of a data scientist).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Specifically, what tool(s) do I learn and what theory do I need to grasp?&lt;/strong&gt; (Keeping in mind that all my coding and statistical competencies are mostly self-taught.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike &lt;a href=&quot;https://datascience.stackexchange.com/questions/10919/data-scientist-light?rq=1&quot;&gt;this (fascinating) question&lt;/a&gt;, I don't have a business background and don't necessarily want to move towards the business analyst path - I still want to play with psychical (earth science) or social data. Neither do I want to work on the data management side so much - I want databases and coding to be a means, not an end. And finally, I'm not inclined much towards the theory and mathematics. Perhaps the best way to summarise my inclination and position is that I don't want to become a data science expert, but want to be able to become an expert in given subjects &lt;em&gt;through data science&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My inclination is perhaps to concentrate on something like Python, and use it to exploit R and other functionality?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tools I've used in the past (in order of exposure) -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SAS (for statistics and research, not the warehouse side)&lt;/li&gt;&#xA;&lt;li&gt;VBA/VB6/Excel/Access (data manipulation, reporting)&lt;/li&gt;&#xA;&lt;li&gt;GIS (ArcGIS for analysis/research, not database management)&lt;/li&gt;&#xA;&lt;li&gt;R (stats...)&lt;/li&gt;&#xA;&lt;li&gt;Some HTML/JS&lt;/li&gt;&#xA;&lt;li&gt;Some Python&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One thing I find is that my existing competencies don't provide me with a useful tool for bringing together different data and getting it into the state I want for analysis (ETL I suppose?), hence the inclination to re-learn Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your thoughts!&lt;/p&gt;&#xA;" OwnerUserId="24871" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-10-02T18:12:38.977" Title="Approaching data science" Tags="|beginner|" AnswerCount="1" CommentCount="0" ClosedDate="2016-10-02T21:00:25.200" ContentLicense="CC BY-SA 3.0" />
  <row Id="14314" PostTypeId="1" CreationDate="2016-10-02T13:13:15.027" Score="2" ViewCount="1662" Body="&lt;p&gt;Based on my research a recommendation system are a subclass of information filtering system that seek to predict the &quot;rating&quot; or &quot;preference&quot; that a user would give to an item. And I'm currently developing a collaborative filtering recommendation system, and basically is recommends the top 'n' items to a user (I've used the user-item algorithm). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So on top of that, I will try to evaluate my recommendation system using the movie lens dataset, and basically from my research typical evaluation measures for top-N recommendation are normalized discounted cumulative gain (NDCG) and precision/recall. So my question is how can I evaluate using these metrics (or if you have any other suggestion for another metric) using the movie lens dataset (since it's a rating of users to items).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. Any suggestions are welcome.&lt;/p&gt;&#xA;" OwnerUserId="17596" LastEditorUserId="17596" LastEditDate="2016-10-02T17:10:09.827" LastActivityDate="2017-01-01T18:18:47.337" Title="how to evaluate top n recommendation system with movie lens dataset?" Tags="|recommender-system|information-retrieval|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14319" PostTypeId="2" ParentId="14313" CreationDate="2016-10-02T18:12:38.977" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;https://www.udacity.com/course/data-analyst-nanodegree--nd002&quot; rel=&quot;nofollow&quot;&gt;Udacity Data Analyst Nanodegree&lt;/a&gt; gives a very gentle introduction using Python and R for mostly exploratory Data Analysis which I think is what you are looking for. (It's not tailored towards business analysis. The courses are free and you can just take the ones you think are interesting. In your case, I would skip the ones about data visualization with javascript and data wrangling with MongoDB.) THey provide a bunch of resources to go from there. Of course there are plenty of other online courses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are more into books you could check out Tukey's &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201076160&quot; rel=&quot;nofollow&quot;&gt;Exploratory Data Analysis&lt;/a&gt;. A classic that is still highly relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of tools, the most commonly used are R and Python and I would focus on them for now.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-10-02T18:12:38.977" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14322" PostTypeId="2" ParentId="14288" CreationDate="2016-10-02T21:57:25.427" Score="3" Body="&lt;p&gt;Solution for posterity: In Python, xgboost requires np.array as input. So this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = xgb.DMatrix(data=[x], missing=np.nan)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = xgb.DMatrix(data=np.array([x]), missing=np.nan)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24813" LastActivityDate="2016-10-02T21:57:25.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14324" PostTypeId="1" AcceptedAnswerId="14328" CreationDate="2016-10-03T03:06:22.523" Score="2" ViewCount="7291" Body="&lt;p&gt;I have a dataset in following format:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Movie ID |  Actors | Director | language | ReleaseYear | Genre&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1  |    Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran  | Vidhu Vinod Chopra  | hin | 1994  |  Drama;Romance;Patriotic&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As you can see the columns Actors and Genre has multiple values. I need to perform cluster analysis on this dataset. I don't know what would the best way to deal with such data. I am thinking of two possible solution(don't know if it is a correct approach for this type of problem)--&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split the column, say, 'Actors' into multiple columns e.g. Actors 1, Actor2,..., and then perform the cluster anylysis.&lt;/li&gt;&#xA;&lt;li&gt;Row split i.e. convert a single row of data into multiple rows each row for one value of columns like 'Actors' and 'Genre'.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please suggest me the best way to deal with such data for cluster analysis.&lt;/p&gt;&#xA;" OwnerUserId="24886" LastActivityDate="2019-03-11T16:43:57.403" Title="Handling a feature containing multiple values" Tags="|machine-learning|clustering|dataset|data-cleaning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14325" PostTypeId="2" ParentId="14324" CreationDate="2016-10-03T05:12:16.853" Score="1" Body="&lt;p&gt;You should create multiple new columns - one column per actor and then set it to 1 if respective actor is present in the movie, 0 otherwise. Same for genres. It is called one hot encoding and ensures that data is treated as categorical instead of continuos.&lt;/p&gt;&#xA;" OwnerUserId="21692" LastActivityDate="2016-10-03T05:12:16.853" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14326" PostTypeId="1" CreationDate="2016-10-03T08:12:44.983" Score="1" ViewCount="1862" Body="&lt;p&gt;I am trying to compute Silhouette with k-means. However I have the value really close to 0 and the clusters are very clearly separated. Do you know where can be the problem? This is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_samples, n_features = data.shape&#xA;n_digits=2&#xA;labels=data[:,-1]&#xA;data=data[:,:-1]&#xA;sample_size = 300&#xA;&#xA;print(&quot;n_digits: %d, \t n_samples %d, \t n_features %d&quot;&#xA;      % (n_digits, n_samples, n_features))&#xA;print(79 * '_')&#xA;print('% 9s' % 'init'&#xA;      '    time  inertia    homo   compl  v-meas     ARI AMI  silhouette')&#xA;&#xA;def bench_k_means(estimator, name, data):&#xA;    t0 = time()&#xA;    estimator.fit(data)&#xA;    print('% 9s   %.2fs    %i   %.3f   %.3f   %.3f   %.3f   %.3f    %.3f'&#xA;          % (name, (time() - t0), estimator.inertia_,&#xA;             metrics.homogeneity_score(labels, estimator.labels_),&#xA;             metrics.completeness_score(labels, estimator.labels_),&#xA;             metrics.v_measure_score(labels, estimator.labels_),&#xA;             metrics.adjusted_rand_score(labels, estimator.labels_),&#xA;             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),&#xA;             metrics.silhouette_score(data, estimator.labels_,&#xA;                                      metric='euclidean',&#xA;                                      sample_size=sample_size)))         &#xA;&#xA;pca = PCA(n_components=n_digits).fit(data)&#xA;bench_k_means(KMeans(init=pca.components_, n_clusters=n_digits, n_init=1),&#xA;              name=&quot;PCA-based&quot;,&#xA;              data=data)&#xA;&#xA;print(79 * '_')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The obtained Silhouette is 0.052 and this &lt;a href=&quot;https://i.stack.imgur.com/Le6Rh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Le6Rh.png&quot; alt=&quot;Silhouette&quot;&gt;&lt;/a&gt; is the obtained k-means clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Laia&lt;/p&gt;&#xA;" OwnerUserId="8710" LastEditorUserId="20330" LastEditDate="2017-03-10T10:50:26.323" LastActivityDate="2017-03-10T10:50:26.323" Title="Silhouette calculation in k-means" Tags="|python|clustering|scikit-learn|k-means|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14328" PostTypeId="2" ParentId="14324" CreationDate="2016-10-03T08:23:12.417" Score="3" Body="&lt;p&gt;You need to rethink your approach. Rather than &quot;what can I code to make things work&quot;, you need to ask &quot;what is the right thing to do, and how can I implement this&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is &lt;em&gt;hard&lt;/em&gt; (easy to run, hard to get good results).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering non-continuous data is even harder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why people create a lot of false results here is because you have infinitely many way to weight different values and attributes, and essentially &lt;strong&gt;you can get almost any result you want&lt;/strong&gt; just by playing around with parameters. Don't let yourself be lured by the common hack of &quot;one hot encoding&quot; everything; it just shows that people don't (want to) &lt;em&gt;know&lt;/em&gt; what they are doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a reliable result, you need to be very clear on your assumptions, such as &quot;I assume all actors are equally representative, and the overlap as measured by Jaccard is a good indicator of similarity&quot; (I would disagree here, actors should be weighted).&#xA;Then you need to do the same thing with genres. Here it is even more questionable hos to compute similarity of genres.&#xA;And after that, you need to combine all these different similarities into one. That is probably the hardest step, and will involve even more weighting parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, I'd say: whatever you do, the clustering is &lt;em&gt;not&lt;/em&gt; statistically sound. It's too many parameters chosen without even a good reason (yet a proof) that this way is better than another way of choosing them.&#xA;In particular, you will never know if there is an even better clustering with different parameters.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="-1" LastEditDate="2019-03-11T16:43:57.403" LastActivityDate="2019-03-11T16:43:57.403" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14329" PostTypeId="1" AcceptedAnswerId="26189" CreationDate="2016-10-03T08:36:22.390" Score="1" ViewCount="123" Body="&lt;p&gt;Documentaion for Spatial Convolution define it as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nInputPlane: The number of expected input planes in the image given into forward().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;nOutputPlane: The number of output planes the convolution layer will produce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have any experience with torch but i guess i have used a similar function in keras&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Convolution2D(64, 3, 3, border_mode='same', input_shape=(3, 256, 256))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which takes as input the shape of the image that is 256*256 in rgb.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read usage of Spatial Convolution in torch as below but unable to figure out what does the nInputPlane and nOutputPlane paramter correspond to?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;local convLayer = nn.SpatialConvolutionMM(384, 384, 1, 1, 1, 1, 0, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the code above what does these 384,384 represent ?&lt;/p&gt;&#xA;" OwnerUserId="24891" LastEditorUserId="29169" LastEditDate="2021-02-16T15:36:39.813" LastActivityDate="2021-02-16T15:36:39.813" Title="Unable to figure out nInputPlane in SpatialConvolution in torch?" Tags="|convolutional-neural-network|torch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14331" PostTypeId="1" AcceptedAnswerId="14334" CreationDate="2016-10-03T12:40:49.883" Score="0" ViewCount="2471" Body="&lt;p&gt;As per the references &#xA;1.   &lt;a href=&quot;http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning&quot; rel=&quot;nofollow&quot;&gt;http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning&lt;/a&gt; &#xA;2.   &lt;a href=&quot;http://www.listendata.com/2016/01/cluster-analysis-with-r.html&quot; rel=&quot;nofollow&quot;&gt;http://www.listendata.com/2016/01/cluster-analysis-with-r.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Hopkins statistics value close to '0' is also clusterable. Is this correct? Or the value should always be close to '1' for clusterable?&lt;/p&gt;&#xA;" OwnerUserId="24899" LastActivityDate="2018-09-04T02:14:27.573" Title="Clustering tendency using hopkin's statistics" Tags="|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14334" PostTypeId="2" ParentId="14331" CreationDate="2016-10-03T17:04:52.943" Score="1" Body="&lt;p&gt;I don't think the Hopkins statistic is useful for this purpose &lt;em&gt;at all&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is essentially a test for a uniform distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But not having a uniform distribution does not mean the data is suitable for cluster analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example a single Gaussian distribution (unimodal!) will score high on this test, but the data doesn't have multiple clusters but all points are from the same distribution.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-10-03T17:04:52.943" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14335" PostTypeId="2" ParentId="13181" CreationDate="2016-10-03T17:09:17.430" Score="4" Body="&lt;p&gt;From  &lt;a href=&quot;https://stats.stackexchange.com/q/226672/12359&quot;&gt;How few training examples is too few when training a neural network?&lt;/a&gt; on CV:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It really depends on your dataset, and network architecture. One rule of thumb I have read (2) was a few thousand samples per class for the neural network to start to perform very well. In practice, people try and see. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A good way to roughly assess to what extent it could be beneficial to have more training samples is to plot the performance of the neural network based against the size of the training set, e.g. from (1):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N2RHk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N2RHk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(1) Dernoncourt, Franck, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. &quot;&lt;a href=&quot;https://arxiv.org/abs/1606.03475&quot; rel=&quot;nofollow noreferrer&quot;&gt;De-identification of Patient Notes with Recurrent Neural Networks&lt;/a&gt;&quot; arXiv preprint arXiv:1606.03475 (2016).&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(2) Cireşan, Dan C., Ueli Meier, and Jürgen Schmidhuber. &quot;Transfer learning for Latin and Chinese characters with deep neural networks.&quot; In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-6. IEEE, 2012. &lt;a href=&quot;https://scholar.google.com/scholar?cluster=7452424507909578812&amp;amp;hl=en&amp;amp;as_sdt=0,22&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://scholar.google.com/scholar?cluster=7452424507909578812&amp;amp;hl=en&amp;amp;as_sdt=0,22&lt;/a&gt; ; &lt;a href=&quot;http://people.idsia.ch/~ciresan/data/ijcnn2012_v9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://people.idsia.ch/~ciresan/data/ijcnn2012_v9.pdf&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For classification tasks with a &lt;strong&gt;few thousand samples per&#xA;  class&lt;/strong&gt;, the benefit of (unsupervised or supervised) pretraining is not easy to demonstrate. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-10-03T17:09:17.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14336" PostTypeId="1" CreationDate="2016-10-03T18:01:30.123" Score="1" ViewCount="384" Body="&lt;p&gt;New to stats and R and was wondering if anyone can help me understand the output of the following code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pwr.t2n.test(n1=7561, n2=11531, d=NULL, sig.level = 0.05, power = 0.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t test power calculation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         n1 = 7561&#xA;         n2 = 11531&#xA;          d = 0.04148028&#xA;  sig.level = 0.05&#xA;      power = 0.8&#xA;alternative = two.sided&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I was going through the documentation it says d value of 0.2, 0.4, 0.8 corresponds to small, medium and large.. However, I get d as 0.04 when I run the above? What does it mean? Does it mean that I will be able to see and effect size of 0.0414 when I perform my tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand when I run the below code i.e. setting d = 0.8&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pwr.t2n.test(n1=7561, n2=11531, d=0.8, sig.level = 0.05)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of power is 1 meaning it will be able to find the Type II error&#xA;    t test power calculation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         n1 = 7561&#xA;         n2 = 11531&#xA;          d = 0.8&#xA;  sig.level = 0.05&#xA;      power = 1&#xA;alternative = two.sided&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting confused as I thought I will not be able to get a power of 1 for such high d value as my previous line gave a small value for d when power was 0.8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone pls. explain and clear my understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24795" LastActivityDate="2023-04-17T20:04:30.547" Title="Understand output of pwr.t2n.test in R" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14337" PostTypeId="1" AcceptedAnswerId="14901" CreationDate="2016-10-03T18:27:34.087" Score="2" ViewCount="4956" Body="&lt;p&gt;Calculating similarity between two users is rather straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User A = {7,3,2,4,1}&#xA;User B = {4,1,9,7,5}&#xA;&#xA;Products in common = {1,4,7}&#xA;Union of products = {1,2,3,4,5,7,9}&#xA;&#xA;Hence the Jaccard similarity: 3/7 = 0.429&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However it is not clear to me how to calculate similarity between two products.&#xA;Let's say I want to calculate similarity between products 7 and 1 from previous example, how can one achieve that?&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-11-03T06:05:49.007" Title="Jaccard similarity between two items" Tags="|similarity|jaccard-coefficient|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14338" PostTypeId="2" ParentId="14337" CreationDate="2016-10-03T20:14:10.960" Score="0" Body="&lt;p&gt;There are various ways to do it. One way is item-item collaborative filtering. Say you have 100 users and 100 songs which will be in this case a n[# of users] X 101 [user and songs column] matrix. Also, each user has liked x out of the 100 songs &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, disregard the user column and create a 100 x 100 song matrix. Now, for each song i.e. column you calculate the cosine similarity with other 99 songs (this will be cosine of song1 with n rows with song2 [n rows] and so on). Finally, for each song you will get a similarity value. Basically, songs which a lot of users like  with other songs will have a greater value of similarity. However, going ahead with the assumption that a listener usually likes certain kind of song, a user may like only 10 songs (similar kind) and this way we have inputs from n other users. Finally, we will be able to calculate similarity for each songs based on the inputs from n users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then sort and recommend the similarity score with highest value for each song to the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="24795" LastActivityDate="2016-10-03T20:14:10.960" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14339" PostTypeId="1" CreationDate="2016-10-03T20:53:14.283" Score="1" ViewCount="34" Body="&lt;p&gt;I devised a classification algorithm which is useful for specific complicated distribution of classes of data. The method works good on the artificial data which cannot be solved by typical algorithms in the literature. But yet i couldn't find any real data to evaluate the algorithm on real problems.&#xA;Also there is mathematical support for the basis of the approach and the way it works. &#xA;So i was wondering if an algorithm with good mathematical support but no evaluation on real data has any chance to get accepted in computer science conferences like &lt;strong&gt;SDM&lt;/strong&gt; or &lt;strong&gt;CVPR&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="24837" LastActivityDate="2016-10-03T20:53:14.283" Title="Can an algorithm tested only on artificial data be accepted in a high rank conference?" Tags="|algorithms|model-evaluations|" AnswerCount="0" CommentCount="0" ClosedDate="2018-08-19T20:24:02.573" ContentLicense="CC BY-SA 3.0" />
  <row Id="14340" PostTypeId="2" ParentId="14287" CreationDate="2016-10-03T21:46:56.423" Score="1" Body="&lt;p&gt;This is referred to as &quot;interval censoring&quot;: that is to say, you know the event occurred within an interval, but not exactly where in that interval. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think there has been a lot of attention in the ML community for interval censored data (or any, really). However, there has been a reasonable amount in the statistical community, myself included. As such, I've written the &lt;code&gt;R&lt;/code&gt; package &lt;code&gt;icenReg&lt;/code&gt; for interval censored regression models. Regression models certainly can be used as ML tool, although these don't yet have all the bells and whistles that are more typical for ML problems (i.e. no penalized regression a la elastic net etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;code&gt;icenReg&lt;/code&gt; at least contains a tool for generalized cross validation, although it is hidden from the public. It can be extracted by &lt;code&gt;icenReg:::icenReg_cv&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13005" LastActivityDate="2016-10-03T21:46:56.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14341" PostTypeId="1" AcceptedAnswerId="14384" CreationDate="2016-10-03T23:55:43.650" Score="-1" ViewCount="596" Body="&lt;p&gt;Every week I get a group of sentences (~1000) each of them may be similar. Example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;metallica hard wired&lt;/li&gt;&#xA;&lt;li&gt;metallica hardwire&lt;/li&gt;&#xA;&lt;li&gt;metallica hardwired&lt;/li&gt;&#xA;&lt;li&gt;metallica hard wire&lt;/li&gt;&#xA;&lt;li&gt;hardwired metallica&lt;/li&gt;&#xA;&lt;li&gt;hardwire metallica&lt;/li&gt;&#xA;&lt;li&gt;hardwire&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm using Cosine similarity to find common documents and group them.&#xA;I have realized that similar docs: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;metallica hardwire and metallica hardwired&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;return ~0.5 similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hardwired metallica and metallica hardwire&lt;/p&gt;&#xA;&#xA;&lt;p&gt;return ~0.433&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other docs with more words return higher values. (Im using &lt;strong&gt;cosine_similarity&lt;/strong&gt; from &lt;strong&gt;sklearn.metrics.pairwise&lt;/strong&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I iterate over each document and get the similarity among all docs, after that I extract the highest values. (cosine similarity &gt; 0.55)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far is working fine but there are cases in which I can't find similar sentences unless I reduce my coefficient, doing so it may associate other values to non-related items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know what is the best technique to group common sentences from a list of sentences. Not sure if that would be semantic similarity.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2016-10-04T01:10:20.913" LastActivityDate="2016-10-05T18:24:23.843" Title="Sentence similarity" Tags="|clustering|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14342" PostTypeId="1" AcceptedAnswerId="15491" CreationDate="2016-10-04T00:56:17.293" Score="2" ViewCount="2178" Body="&lt;p&gt;I am using Convolutional Neural Networks (CNN) and I just want to ask if the way I split my training/validation/testing set is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a total of 55 subjects. I plan to split them into 80–10–10 for training (45 subjects), validation (unseen 5 subjects), testing (unseen 6 subjects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should the validation set consists of unseen subjects as well? Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read that using N-Fold cross-validation, the whole training set (instances) are shuffled then split into N-folds and the model is trained and averaged N times. However, in the case for Neural Networks or CNN, we don't use cross-validation since it is very computationally expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all. On the other hand, if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN but when tested with the unseen testing set, my top-1 accuracy is around 15-16% only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="24913" LastEditorUserId="29169" LastEditDate="2021-02-16T15:36:29.930" LastActivityDate="2021-02-16T15:36:29.930" Title="Training Validation Testing set split for facial expression dataset" Tags="|machine-learning|neural-network|cross-validation|computer-vision|convolutional-neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14343" PostTypeId="2" ParentId="12050" CreationDate="2016-10-04T04:48:00.380" Score="2" Body="&lt;p&gt;&lt;strong&gt;No. Not yet&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no single package in Golang, which acts as versatile as nltk for NLP. However, there are several packages which aim to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a compiled list of such packages: &lt;a href=&quot;https://github.com/gopherds/resources/blob/master/tooling/README.md#nlp&quot; rel=&quot;nofollow&quot;&gt;https://github.com/gopherds/resources/blob/master/tooling/README.md#nlp&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2016-10-04T04:48:00.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14344" PostTypeId="2" ParentId="14326" CreationDate="2016-10-04T05:02:27.323" Score="2" Body="&lt;p&gt;Since the points in each cluster are fairly spread out &lt;em&gt;along the direction of the separating line&lt;/em&gt;, the average distance within a cluster is probably not that much smaller (as you might hope) than the average distance to points in the other cluster. A small positive value could be plausible. (Maybe look at the distribution of the points along the second principal component. A big spread could be the problem.)&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-10-04T05:02:27.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14345" PostTypeId="2" ParentId="14342" CreationDate="2016-10-04T07:35:40.560" Score="2" Body="&lt;p&gt;Validation set implies it is obtained by taking a part of training dataset. It cannot be unseen(I am assuming by unseen, you mean whose labels you do not know or you want to check on) subjects because then we will not be able to validate(check accuracy/precision to test our model) in the first place. I think what you are doing is probably taking validation set from training dataset which you are using for training as well leading to overfitting. Let me be elaborate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume you have 100 documents. Use 80 for training and rest 20 for validation. DO NOT use these 20 documents for training. If you use it, you will lead to overfitting(I think this is probably the answer for your overfitting). Now, the other test dataset(unseen documents) can be tested on.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-04T07:35:40.560" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14346" PostTypeId="1" CreationDate="2016-10-04T08:50:03.107" Score="6" ViewCount="900" Body="&lt;p&gt;I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is it, what is the main difference from the other two (or from the Unsupervised type, specifically), with some good example? Is it a promising alternative worth to explore or just some hyped niche curiosity?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2020-12-29T17:14:23.507" Title="What is Reinforcement Learning?" Tags="|reinforcement-learning|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14348" PostTypeId="2" ParentId="14346" CreationDate="2016-10-04T10:03:25.217" Score="6" Body="&lt;p&gt;Reinforcement Learning uses a simple logic of learning in which the network tries to learn from the feedback it obtains. This tries to optimise the overall reward in the long run instead of the current reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://reinforcementlearning.ai-depot.com/&quot;&gt;This&lt;/a&gt; is one of the best platform to read about it. It also contains some useful links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As stated by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot;&gt;wiki&lt;/a&gt;,&#xA;The basic reinforcement learning model consists of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a set of environment states S;&lt;/li&gt;&#xA;&lt;li&gt;a set of actions A;&lt;/li&gt;&#xA;&lt;li&gt;rules of transitioning between states;&lt;/li&gt;&#xA;&lt;li&gt;rules that determine the scalar immediate reward of a transition; and&lt;/li&gt;&#xA;&lt;li&gt;rules that describe what the agent observes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition. In many works, the agent is also assumed to observe the current environmental state, in which case we talk about full observability, whereas in the opposing case we talk about partial observability. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-04T10:03:25.217" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14349" PostTypeId="1" AcceptedAnswerId="14359" CreationDate="2016-10-04T11:05:24.647" Score="22" ViewCount="20093" Body="&lt;p&gt;I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's reasonable that one differentiates between logical and linear type functions, depending on the desired binary/continuous output but what is the advantage of sigmoid function over the simple linear one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ReLU is especially difficult to understand for me, for instance: what is the point to use a function that behaves like linear in case of positive inputs but is &quot;flat&quot; in case of negatives? What is the intuition behind this? Or is it just a simple trial-error thing, nothing more?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="34176" LastEditDate="2018-11-18T23:29:54.467" LastActivityDate="2018-11-18T23:29:54.467" Title="Difference of Activation Functions in Neural Networks in general" Tags="|neural-network|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14350" PostTypeId="2" ParentId="10711" CreationDate="2016-10-04T12:28:33.777" Score="2" Body="&lt;p&gt;I think what you are looking for is multilabel classification.I used mlr package in R ; I followed &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/devel/html/multilabel/index.html#multilabel-classification&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; link to achieve it.&lt;/p&gt;&#xA;" OwnerUserId="24575" LastEditorUserId="21024" LastEditDate="2016-10-05T13:19:35.070" LastActivityDate="2016-10-05T13:19:35.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14351" PostTypeId="1" CreationDate="2016-10-04T13:06:23.260" Score="2" ViewCount="280" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Spiking_neural_network&quot; rel=&quot;nofollow&quot;&gt;Spiking Neural Networks&lt;/a&gt; are said to be the NN's third generation. Feed-Forward NN are the first. What is the second Generation? Stacked NN?&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="843" LastEditDate="2016-10-04T15:31:11.700" LastActivityDate="2017-01-02T17:04:53.747" Title="Are stacked NN the second generation of NN?" Tags="|neural-network|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14352" PostTypeId="1" AcceptedAnswerId="14387" CreationDate="2016-10-04T13:13:15.930" Score="14" ViewCount="875" Body="&lt;p&gt;It is said in &lt;a href=&quot;https://en.wikipedia.org/wiki/Deep_learning#Deep_neural_network_architectures&quot;&gt;Wikipedia&lt;/a&gt; and &lt;a href=&quot;http://deeplearning4j.org/neuralnet-overview.html#concept&quot;&gt;deeplearning4j&lt;/a&gt; that Deep-learning NN (DLNN) are NN that have &gt;1 hidden layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These kind of NN were standard at university for me, while DLNN are very hyped right now. &lt;strong&gt;Been there, done that - what's the big deal?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I heard also that stacked NN are considered deep-learning. &lt;strong&gt;How is deep-learning really defined?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My background of NN is mostly from university, not from jobs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;studied applications of NN in industry&lt;/li&gt;&#xA;&lt;li&gt;had about 5 courses on artif. intel. &amp;amp; mach. learn. - though maybe 2 of them on NN&lt;/li&gt;&#xA;&lt;li&gt;used NN for small, simple project on image recognition - used 3 layer feed-forward NN&lt;/li&gt;&#xA;&lt;li&gt;did not do real research (as in doctor thesis) on them&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="8820" LastEditDate="2017-05-01T07:47:19.577" LastActivityDate="2017-05-01T07:47:19.577" Title="How are deep-learning NNs different now (2016) from the ones I studied just 4 years ago (2012)?" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14354" PostTypeId="2" ParentId="14352" CreationDate="2016-10-04T13:38:44.633" Score="0" Body="&lt;p&gt;Deep learning (and in particular deep convoluted neural networks) is very popular right now because of recent, substantial improvements in various machine vision / object detection challenges. Deep convents in particular have been used to reduce ImageNet classification errors from 28% in 2010 to 7% in 2014. Deep learning is certainly overhyped but that shouldn't detract from the fact that there have been meaningful improvements in the field of machine vision. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-10-04T13:38:44.633" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14355" PostTypeId="1" AcceptedAnswerId="14366" CreationDate="2016-10-04T13:54:26.537" Score="0" ViewCount="954" Body="&lt;p&gt;I started tinkering with sklearn kmeans last night out of curiosity with the goal of clustering users into groups to see what kind of user groups I can derive.  I am lost when it comes to plotting the results as most examples have nice (x,y) coordinates.  For example, the iris data set has pedal width and pedal length.  From my experimentation, I don't seem to have anything that displays very nice.  Is this assumption correct / does anyone have tips, pointers, learning resources that they could offer?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import pprint&#xA;import numpy as np&#xA;from sklearn.preprocessing import normalize&#xA;from sklearn.preprocessing import LabelEncoder&#xA;from sklearn.cluster import KMeans&#xA;from collections import defaultdict&#xA;import matplotlib.pyplot as plt&#xA;from matplotlib import style&#xA;style.use('ggplot')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I normalized the data as it had a wide variance...again, not sure if this is a correct assumption to make&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array(normalize(data, axis=0, copy=False))&#xA;&#xA;kmeans = KMeans(n_clusters=3)&#xA;pred = kmeans.fit_predict(X)&#xA;labels = kmeans.labels_&#xA;cent = kmeans.cluster_centers_&#xA;&#xA;plt.scatter(X[:, [4]], X[:, [6]])&#xA;plt.scatter(cent[:, [4]], cent[:, [6]], marker=&quot;x&quot;, s=150, linewidths=5, zorder=10)&#xA;plt.ylabel('Count')&#xA;plt.xlabel('Department')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any pointers are appreciated, I will include sample data below.  Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;emp_type,title,work_country,director_userid,dept_name,business_unit_name,UserCNT&#xA;0,9,7,29,20,2,2&#xA;0,13,7,8,14,6,5&#xA;0,4,3,56,29,8,3&#xA;0,15,3,36,32,2,3&#xA;0,4,3,32,16,2,0&#xA;0,4,1,40,13,6,0&#xA;0,4,3,62,12,4,1&#xA;0,13,7,61,5,13,4&#xA;2,1,3,70,35,15,2&#xA;0,4,3,64,4,13,0&#xA;2,1,3,43,43,2,0&#xA;0,13,7,50,17,16,0&#xA;2,1,3,31,26,2,1&#xA;2,1,3,65,58,17,0&#xA;0,4,3,57,63,12,0&#xA;2,1,6,7,45,18,2&#xA;2,1,3,43,42,2,0&#xA;1,1,7,65,58,17,0&#xA;2,1,3,32,16,2,0&#xA;2,1,3,29,20,2,0&#xA;0,4,0,50,17,16,2&#xA;0,5,3,20,23,9,0&#xA;0,9,3,32,16,2,2&#xA;0,4,3,5,51,12,0&#xA;2,1,7,51,53,7,0&#xA;0,13,7,37,55,12,0&#xA;2,1,4,19,62,13,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example clustering using entire data set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yPe9e.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yPe9e.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24930" LastEditorUserId="24930" LastEditDate="2016-10-04T17:53:47.880" LastActivityDate="2016-10-05T16:51:29.967" Title="Quick start using python and sklearn kmeans?" Tags="|python|scikit-learn|k-means|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14356" PostTypeId="1" AcceptedAnswerId="14361" CreationDate="2016-10-04T14:32:19.823" Score="6" ViewCount="32104" Body="&lt;p&gt;When examining my dataset with a binary target (y) variable I wonder if a correlation matrix is useful to determine predictive power of each variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My predictors (X) contain some numeric and some factor variables.&lt;/p&gt;&#xA;" OwnerUserId="9053" LastActivityDate="2018-03-18T14:45:10.767" Title="Is a correlation matrix meaningful for a binary classification task?" Tags="|classification|correlation|binary|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14357" PostTypeId="2" ParentId="14352" CreationDate="2016-10-04T15:24:27.737" Score="0" Body="&lt;p&gt;A similar question was asked on CV: &lt;a href=&quot;https://stats.stackexchange.com/q/234615/12359&quot;&gt;difference between neural network and deep learning&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Deep learning = deep artificial neural networks + &lt;a href=&quot;https://en.wikipedia.org/wiki/Deep_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;other kind of deep&#xA;  models&lt;/a&gt;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Deep artificial neural networks = artificial neural networks with more&#xA;  than 1 layer. (see &lt;a href=&quot;https://stats.stackexchange.com/q/229619/12359&quot;&gt;minimum number of layers in a deep neural&#xA;  network&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;FYI: &lt;a href=&quot;https://stats.stackexchange.com/q/86800/12359&quot;&gt;How true is this slide on deep learning claiming that all improvements from 1980s are only due to much more data and much faster computers?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-10-04T15:24:27.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14358" PostTypeId="2" ParentId="14351" CreationDate="2016-10-04T15:31:02.323" Score="1" Body="&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kumar, Satish. &lt;a href=&quot;https://books.google.com/books?id=y67YnH4kEMsC&amp;amp;pg=PA578&amp;amp;lpg=PA578&amp;amp;dq=%22second+generation+of+neural+network+models%22&amp;amp;source=bl&amp;amp;ots=_frZdlnbTp&amp;amp;sig=UV1yMvV9eVt5iOGp_U-1yB7jdXA&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwizz-HiusHPAhXk44MKHSvoB38Q6AEINTAE#v=onepage&amp;amp;q&amp;amp;f=false&quot; rel=&quot;nofollow&quot;&gt;Neural networks: a classroom approach&lt;/a&gt;. Tata McGraw-Hill Education, 2004.:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The first generation of neural network models employed McCulloch-Pitts TLN type neuron. They were essentially switching networks. The second generation of neural network models employed a smooth sigmoids. The sigmoidal function was motivated by the increase in firing rate of a neuron with the net input, which is saturates at a certain maximum frequency. The third generation of neural network models will use spiking neurons which encode information in spike trains.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-10-04T15:31:02.323" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14359" PostTypeId="2" ParentId="14349" CreationDate="2016-10-04T15:33:24.363" Score="19" Body="&lt;p&gt;A similar question was asked on CV: &lt;a href=&quot;https://stats.stackexchange.com/q/115258/12359&quot;&gt;Comprehensive list of activation functions in neural networks with pros/cons&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I copy below one of the answers:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One such a list, though not much exhaustive:&#xA;  &lt;a href=&quot;http://cs231n.github.io/neural-networks-1/&quot; rel=&quot;noreferrer&quot;&gt;http://cs231n.github.io/neural-networks-1/&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;h3&gt;Commonly used activation functions&lt;/h3&gt;&#xA;  &#xA;  &lt;p&gt;Every activation function (or &lt;em&gt;non-linearity&lt;/em&gt;) takes a single number&#xA;  and performs a certain fixed mathematical operation on it. There are&#xA;  several activation functions you may encounter in practice:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UOzlm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UOzlm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/Mg9s9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Mg9s9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;           &lt;b&gt;Left:&lt;/b&gt; Sigmoid non-linearity&#xA;  squashes real numbers to range between [0,1] &lt;b&gt;Right:&lt;/b&gt; The tanh&#xA;  non-linearity squashes real numbers to range between [-1,1].&#xA;  &#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Sigmoid.&lt;/strong&gt; The sigmoid non-linearity has the mathematical form $\sigma(x) = 1 / (1 + e^{-x})$ and is shown in the image above on&#xA;  the left. As alluded to in the previous section, it takes a&#xA;  real-valued number and &quot;squashes&quot; it into range between 0 and 1. In&#xA;  particular, large negative numbers become 0 and large positive numbers&#xA;  become 1. The sigmoid function has seen frequent use historically&#xA;  since it has a nice interpretation as the firing rate of a neuron:&#xA;  from not firing at all (0) to fully-saturated firing at an assumed&#xA;  maximum frequency (1). In practice, the sigmoid non-linearity has&#xA;  recently fallen out of favor and it is rarely ever used. It has two&#xA;  major drawbacks: &lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;em&gt;Sigmoids saturate and kill gradients&lt;/em&gt;. A very undesirable property of the sigmoid neuron is that when the neuron's activation&#xA;  saturates at either tail of 0 or 1, the gradient at these regions is&#xA;  almost zero. Recall that during backpropagation, this (local) gradient&#xA;  will be multiplied to the gradient of this gate's output for the whole&#xA;  objective. Therefore, if the local gradient is very small, it will&#xA;  effectively &quot;kill&quot; the gradient and almost no signal will flow through&#xA;  the neuron to its weights and recursively to its data. Additionally,&#xA;  one must pay extra caution when initializing the weights of sigmoid&#xA;  neurons to prevent saturation. For example, if the initial weights are&#xA;  too large then most neurons would become saturated and the network&#xA;  will barely learn.&lt;/li&gt;&#xA;  &lt;li&gt;&lt;em&gt;Sigmoid outputs are not zero-centered&lt;/em&gt;. This is undesirable since neurons in later layers of processing in a Neural Network (more on&#xA;  this soon) would be receiving data that is not zero-centered. This has&#xA;  implications on the dynamics during gradient descent, because if the&#xA;  data coming into a neuron is always positive (e.g. $x &amp;gt; 0$&#xA;  elementwise in $f = w^Tx + b$)), then the gradient on the weights&#xA;  $w$ will during backpropagation become either all be positive, or&#xA;  all negative (depending on the gradient of the whole expression&#xA;  $f$). This could introduce undesirable zig-zagging dynamics in the&#xA;  gradient updates for the weights. However, notice that once these&#xA;  gradients are added up across a batch of data the final update for the&#xA;  weights can have variable signs, somewhat mitigating this issue.&#xA;  Therefore, this is an inconvenience but it has less severe&#xA;  consequences compared to the saturated activation problem above.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Tanh.&lt;/strong&gt; The tanh non-linearity is shown on the image above on the right. It squashes a real-valued number to the range [-1, 1]. Like the&#xA;  sigmoid neuron, its activations saturate, but unlike the sigmoid&#xA;  neuron its output is zero-centered. Therefore, in practice the &lt;em&gt;tanh&#xA;  non-linearity is always preferred to the sigmoid nonlinearity.&lt;/em&gt; Also&#xA;  note that the tanh neuron is simply a scaled sigmoid neuron, in&#xA;  particular the following holds: $ \tanh(x) = 2 \sigma(2x) -1  $.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bAe0Y.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bAe0Y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/DZnAP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DZnAP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;           &lt;b&gt;Left:&lt;/b&gt; Rectified Linear&#xA;  Unit (ReLU) activation function, which is zero when x &amp;lt 0 and then&#xA;  linear with slope 1 when x &amp;gt 0. &lt;b&gt;Right:&lt;/b&gt; A plot from Krizhevsky&#xA;  et al. (pdf) paper indicating the 6x improvement in convergence&#xA;  with the ReLU unit compared to the tanh unit. &#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;ReLU.&lt;/strong&gt; The Rectified Linear Unit has become very popular in the last few years. It computes the function $f(x) = \max(0, x)$. In&#xA;  other words, the activation is simply thresholded at zero (see image&#xA;  above on the left). There are several pros and cons to using the&#xA;  ReLUs: &lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;(+) It was found to greatly accelerate (e.g. a factor of 6 in &lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf&quot; rel=&quot;noreferrer&quot;&gt;Krizhevsky et&#xA;  al.&lt;/a&gt;) the&#xA;  convergence of stochastic gradient descent compared to the&#xA;  sigmoid/tanh functions. It is argued that this is due to its linear,&#xA;  non-saturating form.&lt;/li&gt;&#xA;  &lt;li&gt;(+) Compared to tanh/sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply&#xA;  thresholding a matrix of activations at zero.&lt;/li&gt;&#xA;  &lt;li&gt;(-) Unfortunately, ReLU units can be fragile during training and can &quot;die&quot;. For example, a large gradient flowing through a ReLU neuron&#xA;  could cause the weights to update in such a way that the neuron will&#xA;  never activate on any datapoint again. If this happens, then the&#xA;  gradient flowing through the unit will forever be zero from that point&#xA;  on. That is, the ReLU units can irreversibly die during training since&#xA;  they can get knocked off the data manifold. For example, you may find&#xA;  that as much as 40% of your network can be &quot;dead&quot; (i.e. neurons that&#xA;  never activate across the entire training dataset) if the learning&#xA;  rate is set too high. With a proper setting of the learning rate this&#xA;  is less frequently an issue.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Leaky ReLU.&lt;/strong&gt; Leaky ReLUs are one attempt to fix the &quot;dying ReLU&quot; problem. Instead of the function being zero when x &amp;lt; 0, a leaky ReLU will instead have a small negative slope (of 0.01, or so). That is, the function computes $f(x) = \mathbb{1}(x &amp;lt; 0) (\alpha x) + \mathbb{1}(x&amp;gt;=0) (x) $ where $\alpha$ is a small constant. Some people report success with this form of activation function, but the results are not always consistent. The slope in the negative region can also be made into a parameter of each neuron, as seen in PReLU neurons, introduced in &lt;a href=&quot;http://arxiv.org/abs/1502.01852&quot; rel=&quot;noreferrer&quot;&gt;Delving Deep into Rectifiers&lt;/a&gt;, by Kaiming He et al., 2015. However, the consistency of the benefit across tasks is presently unclear.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1BX7l.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1BX7l.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Maxout&lt;/strong&gt;. Other types of units have been proposed that do not have the functional form $f(w^Tx + b)$ where a non-linearity is applied&#xA;  on the dot product between the weights and the data. One relatively&#xA;  popular choice is the Maxout neuron (introduced recently by&#xA;  &lt;a href=&quot;http://www-etud.iro.umontreal.ca/~goodfeli/maxout.html&quot; rel=&quot;noreferrer&quot;&gt;Goodfellow et&#xA;  al.&lt;/a&gt;) that&#xA;  generalizes the ReLU and its leaky version. The Maxout neuron computes&#xA;  the function $\max(w_1^Tx+b_1, w_2^Tx + b_2)$. Notice that both&#xA;  ReLU and Leaky ReLU are a special case of this form (for example, for&#xA;  ReLU we have $w_1, b_1 = 0$). The Maxout neuron therefore enjoys&#xA;  all the benefits of a ReLU unit (linear regime of operation, no&#xA;  saturation) and does not have its drawbacks (dying ReLU). However,&#xA;  unlike the ReLU neurons it doubles the number of parameters for every&#xA;  single neuron, leading to a high total number of parameters.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;This concludes our discussion of the most common types of neurons and&#xA;  their activation functions. As a last comment, it is very rare to mix&#xA;  and match different types of neurons in the same network, even though&#xA;  there is no fundamental problem with doing so.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;TLDR&lt;/strong&gt;: &quot;&lt;em&gt;What neuron type should I use?&lt;/em&gt;&quot; Use the ReLU non-linearity, be careful with your learning rates and possibly&#xA;  monitor the fraction of &quot;dead&quot; units in a network. If this concerns&#xA;  you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but&#xA;  expect it to work worse than ReLU/Maxout.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;License:&lt;BR/&gt;&lt;BR/&gt;&lt;BR/&gt;&#xA;  The MIT License (MIT)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Copyright (c) 2015 Andrej Karpathy&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Permission is hereby granted, free of charge, to any person obtaining&#xA;  a copy of this software and associated documentation files (the&#xA;  &quot;Software&quot;), to deal in the Software without restriction, including&#xA;  without limitation the rights to use, copy, modify, merge, publish,&#xA;  distribute, sublicense, and/or sell copies of the Software, and to&#xA;  permit persons to whom the Software is furnished to do so, subject to&#xA;  the following conditions:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The above copyright notice and this permission notice shall be&#xA;  included in all copies or substantial portions of the Software.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;THE SOFTWARE IS PROVIDED &quot;AS IS&quot;, WITHOUT WARRANTY OF ANY KIND,&#xA;  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF&#xA;  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.&#xA;  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY&#xA;  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,&#xA;  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE&#xA;  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.*&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-10-04T15:33:24.363" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14360" PostTypeId="2" ParentId="14356" CreationDate="2016-10-04T16:27:05.217" Score="3" Body="&lt;p&gt;It depends. Suppose you have a number of features, say 20, for your binary classification task. It might be the case that out of these 20 features some features are highly correlated. That may introduce some sort of redundant features in your feature space, so you may start to figure out which features to drop and still achieve a good result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases, the task still be a binary classification task and none of the features may be correlated and in that case you would like to incorporate all the features for training your model and making predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using Python then in order to find out which features are correlated and by how much, it is always useful to plot a scatter matrix using pandas which shows how each feature is correlated to other features. The same thing can be viewed even more clearly by plotting the features on a 'heatmap' provided by the seaborn library.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="-1" LastEditDate="2018-03-18T14:45:10.767" LastActivityDate="2018-03-18T14:45:10.767" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14361" PostTypeId="2" ParentId="14356" CreationDate="2016-10-04T17:40:46.053" Score="10" Body="&lt;p&gt;Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are quite a few answers on stats exchange covering this topic - &lt;a href=&quot;https://stats.stackexchange.com/questions/119835/correlation-between-a-nominal-iv-and-a-continuous-dv-variable&quot;&gt;this&lt;/a&gt; or &lt;a href=&quot;https://stats.stackexchange.com/questions/102778/correlations-between-continuous-and-categorical-nominal-variables&quot;&gt;this&lt;/a&gt; for example.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-10-04T17:40:46.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14362" PostTypeId="2" ParentId="14346" CreationDate="2016-10-04T19:09:37.707" Score="1" Body="&lt;p&gt;Hima's answer does a good job summarizing the outline and purpose of reinforcement learning. If you are interested in taking a deeper look I'd recommend &lt;a href=&quot;https://webdocs.cs.ualberta.ca/~sutton/book/bookdraft2016sep.pdf&quot; rel=&quot;nofollow&quot;&gt;this currently free book&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does a great job walking you from a basic reinforcement learning definition through various solutions to dealing with modern challenges.&lt;/p&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2016-10-04T19:09:37.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14363" PostTypeId="2" ParentId="11043" CreationDate="2016-10-04T19:26:12.847" Score="4" Body="&lt;p&gt;Although not directly answering your question, you may be interested in the field of automated &lt;a href=&quot;https://en.wikipedia.org/wiki/Question_answering&quot; rel=&quot;nofollow&quot;&gt;question answering&lt;/a&gt;. To answer natural language text questions they must first be understood, which overlaps with your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good resource is the &lt;a href=&quot;https://web.stanford.edu/~jurafsky/NLPCourseraSlides.html&quot; rel=&quot;nofollow&quot;&gt;course by Jurafsky and Manning&lt;/a&gt;. Particularly the sections on semantics and question answering may help with what you are looking for. There are accompanying lecture videos available on youtube &lt;a href=&quot;https://www.youtube.com/watch?v=nfoudtpBV68&amp;amp;list=PL6397E4B26D00A269&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2016-10-04T19:26:12.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14366" PostTypeId="2" ParentId="14355" CreationDate="2016-10-05T03:30:20.497" Score="0" Body="&lt;p&gt;This has been answered in other places e.g. &lt;a href=&quot;https://stackoverflow.com/a/14110321&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could run &lt;a href=&quot;http://scikit-learn.org/stable/modules/decomposition.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Principal Component Analysis&lt;/a&gt; (or other &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;dimensionality reduction techniques&lt;/a&gt;) and plot the cluster for the first two principal components. &lt;/li&gt;&#xA;&lt;li&gt;You could plot the results for two variables at a time.&lt;/li&gt;&#xA;&lt;li&gt;You could encode third or fourth variables using standard visualization techniques like color coding, symbols or facetting. &lt;/li&gt;&#xA;&lt;li&gt;There are ways to visualize the quality of the fit e.g. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#sphx-glr-auto-examples-cluster-plot-kmeans-silhouette-analysis-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;silhouette analysis&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#The_Elbow_Method&quot; rel=&quot;nofollow noreferrer&quot;&gt;elbow test for determining the number of cluster&lt;/a&gt; etc.&lt;/li&gt;&#xA;&lt;li&gt;Have a quick look at this &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data/24#24&quot;&gt;link&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-05T16:51:29.967" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14368" PostTypeId="1" AcceptedAnswerId="14379" CreationDate="2016-10-05T03:52:38.577" Score="1" ViewCount="1469" Body="&lt;p&gt;I am looking for benchmarks based on neural networks libraries (Theano/TensorFlow/Torch/Caffe/…) to compare the performance between different GPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/jcjohnson/cnn-benchmarks&quot; rel=&quot;nofollow&quot;&gt;https://github.com/jcjohnson/cnn-benchmarks&lt;/a&gt; (CNN in Torch)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/jcjohnson/neural-style#speed&quot; rel=&quot;nofollow&quot;&gt;https://github.com/jcjohnson/neural-style#speed&lt;/a&gt; (CNN in Torch)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/glample/rnn-benchmarks&quot; rel=&quot;nofollow&quot;&gt;https://github.com/glample/rnn-benchmarks&lt;/a&gt; (vanilla RNNs and LSTM in Theano, TensorFlow, and Torch)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What are some other benchmark codes?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-06-01T10:19:36.793" Title="Benchmarks based on neural networks libraries to compare the performance between different GPUs" Tags="|performance|library|gpu|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14369" PostTypeId="1" CreationDate="2016-10-05T04:46:00.443" Score="4" ViewCount="4937" Body="&lt;p&gt;I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?&lt;/p&gt;&#xA;" OwnerUserId="24946" LastActivityDate="2021-11-29T08:31:39.277" Title="How to bootstrap the AUC on a data-set with 50,000 entries?" Tags="|machine-learning|classification|statistics|regression|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14370" PostTypeId="1" CreationDate="2016-10-05T05:44:40.657" Score="0" ViewCount="295" Body="&lt;p&gt;In neural nets, the back-propagation algorithm is 'based on' gradient descent, optimizing over a cost function $C(w,b)$ of the weights and biases of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conjugate gradient descent has way faster convergence than standard gradient descent (and stochastic gradient descent). Is there a similar back-propagation algorithm based off of conjugate gradient descent?&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-10-05T09:35:43.967" Title="Neural Nets: Backpropogation from Conjugate Gradient Descent" Tags="|neural-network|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14371" PostTypeId="2" ParentId="13314" CreationDate="2016-10-05T06:40:50.760" Score="0" Body="&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;: It depends on how distinguishable your classes are. Like cats vs dogs is quite easy, while disgust expression vs angry expression is quite hard.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Epochs&lt;/strong&gt;: Over-fit your model and do early stopping. To handle over-fitting, increase regularization or decrease the complexity of model by decreasing the number of hidden neurons / layers. See what's best for you given the time and performance.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Learning rate&lt;/strong&gt;: Start with the largest learning rate that allows the model to learn. From there, experiment by dividing that learning rate by 10 or 2, observe the results, see which learning rate allows your model to converge faster and leads to higher accuracy.&lt;/p&gt;&#xA;&lt;p&gt;There are still a lot of things to consider to achieve the maximum performance such as learning rate decay, gradient descent optimization algorithms, weight initializations, etc.&lt;/p&gt;&#xA;&lt;p&gt;More observations = more productivity. Be mindful of the time and speed in choosing the best hyperparameters.&lt;/p&gt;&#xA;&lt;p&gt;To know more, I suggest reading this by Michael Nielsen - &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks and Deep Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24913" LastEditorUserId="85045" LastEditDate="2021-02-08T17:16:15.377" LastActivityDate="2021-02-08T17:16:15.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14372" PostTypeId="2" ParentId="13314" CreationDate="2016-10-05T08:07:12.723" Score="2" Body="&lt;p&gt;Before briefing about the other things, I would like to suggest you something good. It is always better to set a seed number in machine learning and deep learning problems to get a reproducible result and by that I mean that every time when you try to run your model anywhere, it can produce the same best result which you got on your machine while experimenting. So, you should always set &lt;code&gt;numpy.random.seed(*anynumber*)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how to compare your result to your expected result? There are a number of parameters that you can look upon during training your model. Some of these are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Validation accuracy&lt;/strong&gt;: It is the one of the most important parameter to look upon for evaluating your model's performance. This gives you a fair idea if your model is overfitting or not. More the validation accuracy, the better is the model.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Validation loss:&lt;/strong&gt; If your model is really good, then validation loss must decrease, at least initially until your best epoch hasn't been obtained. Lower the validation loss, better the model.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimizers:&lt;/strong&gt; There are a number of optimizers like 'rmsprop', 'sgd', etc. which can be used for different types of problems but not all give the same performance. You should grid-search for a number of optimizers for your model if you are not specific about which is best for your model.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Number of training epochs:&lt;/strong&gt; Depending upon the size of dataset, no. of training epochs differ but if you are not giving enough training to your model, then your model performance will suffer.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Training data:&lt;/strong&gt; If you think your model is fine, but you are not achieving expected results, then you can try to train your model on a large dataset. NN require large dataset for achieving awesome performance.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-10-05T08:07:12.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14374" PostTypeId="2" ParentId="14370" CreationDate="2016-10-05T09:35:43.967" Score="1" Body="&lt;p&gt;Yes,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see the article &lt;a href=&quot;https://scholar.google.com/scholar?cluster=7055976448135115973&amp;amp;hl=de&amp;amp;as_sdt=0,5&quot; rel=&quot;nofollow&quot;&gt;Optimization in an error backpropagation neural network environment with a performance test on a spectral pattern classification problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21995" LastActivityDate="2016-10-05T09:35:43.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14376" PostTypeId="2" ParentId="14369" CreationDate="2016-10-05T12:19:10.237" Score="5" Body="&lt;p&gt;Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times.  You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set.  Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample.  This helps extract some information on the natural variability of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) samples 1:50,000 with replacement.  You should find about 31,500 unique records in this new sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) calculate the AUC&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4)  Now you have 10,000 AUCs.  If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval.  The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on.  Now in hindsight you can see your single AUC measure was a little optimistically high.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-10-05T12:19:10.237" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14377" PostTypeId="1" CreationDate="2016-10-05T13:10:32.023" Score="6" ViewCount="11658" Body="&lt;p&gt;I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Choose loss based on your problem at hand. I use default one - &lt;strong&gt;deviance&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;li&gt;Pick n_estimators as large as (computationally) possible (e.g. 600).&lt;/li&gt;&#xA;&lt;li&gt;Tune max_depth, learning_rate, min_samples_leaf, and max_features via&#xA;grid search. &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Increase n_estimators even more and tune learning_rate&#xA;again holding the other parameters fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scikit-learn provides a convenient API for hyperparameter tuning and grid search.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let's look at python code the code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(new_features, target, random_state=42,train_size=0.1 )&#xA;gb_grid_params = {'learning_rate': [0.1, 0.05, 0.02, 0.01],&#xA;              'max_depth': [4, 6, 8],&#xA;              'min_samples_leaf': [20, 50,100,150],&#xA;              #'max_features': [1.0, 0.3, 0.1] &#xA;              }&#xA;print(gb_grid_params)&#xA;&#xA;gb_gs = GradientBoostingClassifier(n_estimators = 600)&#xA;&#xA;clf = grid_search.GridSearchCV(gb_gs,&#xA;                               gb_grid_params,&#xA;                               cv=2,&#xA;                               scoring='roc_auc',&#xA;                               verbose = 3, &#xA;                               n_jobs=10);&#xA;clf.fit(train_gs_X, train_gs_Y);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I obtain the parameters values I cross validate modes to check overfitting. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scores = cross_validation.cross_val_score(gb,&#xA;                                          all_data, target,&#xA;                                          scoring=&quot;roc_auc&quot;,&#xA;                                          n_jobs=6,&#xA;                                          cv=3);&#xA;&quot;Accuracy: %0.5f (+/- %0.5f)&quot;%(scores.mean(), scores.std())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is my approach sufficient? &#xA;Is it a correct way to tune Boosted Decision Trees hyperparameters? &#xA;Do you have any idea how can I improve my tuning procedure?&#xA;I know there exist method like Gaussian Process, which is faster, I mean can find the optimal hyperparameters configuration in less steps but it is not issue. I want to increase the performance measured as a ROC auc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second issue is how to deal with unbalanced trees? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two ideas: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use same number of signal and background (or whatever you call it) events. The problem with this approach is to skipp huge number of possibly useful events. &lt;/li&gt;&#xA;&lt;li&gt;Use DecisionTree parameters &lt;strong&gt;class_weight&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;See my code below: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;signal_event_no = counts = data[target == 1].count()[0]&#xA;background_event_no = counts = data[target == 0].count()[0]&#xA;ratio_background_to_signal = float(background_event_no)/signal_event_no&#xA;ratio_background_to_signal = numpy.round(ratio_background_to_signal, 3)&#xA;train_X, test_X, train_Y, test_Y = train_test_split(new_features, target, random_state=42,train_size=0.5 )              &#xA;gb6 = GradientBoostingClassifier( n_estimators=400, learning_rate=0.2,&#xA;   class_weight=ratio_background_to_signal, max_depth=6)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any other ideas? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last but not the least. How can I change the hyperparameters tuning procedure in respect to xgboost? What hyperparrameters should I take care? Is it the same set as for Gradient Boosted classifier? &lt;/p&gt;&#xA;" OwnerUserId="21988" LastEditorUserId="21988" LastEditDate="2016-10-05T19:16:38.217" LastActivityDate="2016-10-05T19:16:38.217" Title="Tuning Gradient Boosted Classifier's hyperparametrs and balancing it" Tags="|python|scikit-learn|decision-trees|xgboost|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14379" PostTypeId="2" ParentId="14368" CreationDate="2016-10-05T14:15:01.177" Score="0" Body="&lt;p&gt;I found a few more benchmarks with the help of &lt;a href=&quot;https://redd.it/55xr9r&quot; rel=&quot;nofollow&quot;&gt;reddit&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow&quot;&gt;https://github.com/soumith/convnet-benchmarks&lt;/a&gt;: feedforward and convolution layers in many different libraries. (&lt;a href=&quot;https://github.com/soumith/convnet-benchmarks/issues/101&quot; rel=&quot;nofollow&quot;&gt;overview&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/DeepMark/deepmark&quot; rel=&quot;nofollow&quot;&gt;https://github.com/DeepMark/deepmark&lt;/a&gt; : LSTM and CNN, torch only so far.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/baidu-research/DeepBench&quot; rel=&quot;nofollow&quot;&gt;https://github.com/baidu-research/DeepBench&lt;/a&gt; : &quot;DeepBench attempts to answer the question, &quot;Which hardware provides the best performance on the basic operations used for training deep neural networks?&quot; (another page on it: &lt;a href=&quot;https://svail.github.io/DeepBench&quot; rel=&quot;nofollow&quot;&gt;https://svail.github.io/DeepBench&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-10-05T14:15:01.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14382" PostTypeId="2" ParentId="14020" CreationDate="2016-10-05T17:25:10.313" Score="1" Body="&lt;p&gt;When you run Spark in the shell the SparkConf object is already created for you. As stated in the &lt;a href=&quot;https://spark.apache.org/docs/latest/api/scala/#org.apache.spark.SparkConf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; once a SparkConf object is passed to Spark, it  can no longer be modified by the user. So stopping it and creating a new one is actually the right way to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this should now be possible for Spark 2.0 and higher.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2016-10-05T17:38:26.367" LastActivityDate="2016-10-05T17:38:26.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14383" PostTypeId="2" ParentId="13731" CreationDate="2016-10-05T17:30:31.283" Score="1" Body="&lt;p&gt;I don't know about learnable evolution models, but Spark ML has a module for &lt;a href=&quot;https://spark.apache.org/docs/2.0.0-preview/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow&quot;&gt;collaborative filtering&lt;/a&gt;. This can be used to find objects that are similar/often bought together etc. &lt;a href=&quot;https://blogs.aws.amazon.com/bigdata/post/Tx6J5RM20WPG5V/Building-a-Recommendation-Engine-with-Spark-ML-on-Amazon-EMR-using-Zeppelin&quot; rel=&quot;nofollow&quot;&gt;Amazon uses it&lt;/a&gt; to show you the &quot;People who bought this also bought these items&quot; feature on their website. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-10-05T17:30:31.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14384" PostTypeId="2" ParentId="14341" CreationDate="2016-10-05T18:24:23.843" Score="1" Body="&lt;p&gt;Cosine is &lt;em&gt;only&lt;/em&gt; good for long documents. &lt;code&gt;example document&lt;/code&gt; and &lt;code&gt;exampl docunemt&lt;/code&gt; have 0 cosine similarity. Similarly, &lt;code&gt;hard wired&lt;/code&gt; and &lt;code&gt;hardwired&lt;/code&gt; are completely dissimilar for cosine. Because it is based on counting the number of &lt;em&gt;identical&lt;/em&gt; words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want letter-based similarity, consider levenshtein. But you will likely need to go to something complex n-gram based to also detect &lt;code&gt;wordA wordB&lt;/code&gt; and &lt;code&gt;wordB wordA&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-10-05T18:24:23.843" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14385" PostTypeId="1" AcceptedAnswerId="14388" CreationDate="2016-10-05T19:03:58.390" Score="2" ViewCount="3305" Body="&lt;p&gt;I have a dataset with election results and crime rates per city. For each variable I have an absolute value &lt;em&gt;(i.e. Total votes, Total crimes)&lt;/em&gt; and a relative value &lt;em&gt;(i.e. Percentage shares of votes)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to calculate the correlation coefficient for some variables, but in the process I had a question about what value I need to use, if relative values or absolute values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I calculated &lt;code&gt;z score&lt;/code&gt; for absolute values and then I calculated the correlation using excel. I also used &lt;code&gt;pandas.DataFrame.corr()&lt;/code&gt; and &lt;code&gt;pearsonr&lt;/code&gt; from &lt;code&gt;scipy.stats.stats&lt;/code&gt; in python, in order to corroborate results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I use absolute values I will get a positive correlation between &lt;strong&gt;&lt;em&gt;candidate 1&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;candidate 2&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = df['Abs Cand 1'].tolist()&#xA;y = df['Abs Cand 2'].tolist()&#xA;&#xA;print (pearsonr(x,y))&#xA;(0.95209664861187004, 0.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if I use relative ones I will get a negative correlation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = df['Rel Cand 1'].tolist()&#xA;y = df['Rel Cand 2'].tolist()&#xA;&#xA;print (pearsonr(x,y))&#xA;(-0.99704737036262991, 0.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was confused when I saw both results, and now I need some orientation to understand those differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="17636" LastActivityDate="2016-10-05T21:05:34.127" Title="Pearson correlation method using absolute values and relative values" Tags="|python|statistics|correlation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14386" PostTypeId="2" ParentId="14312" CreationDate="2016-10-05T19:13:53.407" Score="-2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;To generate a forecast more accurately for the next year given past 3 years of history, you will need more granular data. At least monthly sales, so that the forecast can capture trend and/seasonality. With three data points, you will at best be able to generate a moving average forecast. Which isn't very good. If you have more granular data, use the forecast package in R. The forecast() function will pick the best method based on the time series pattern. You don't need to worry about stationarity or heteroscedasticity either.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For new products, you have two options. Explore diffusion curves such as Bass. Or, base the forecast curve on previous new product launches if there are shared attributes with existing products. If it's a brand new product line, evaluate market trends to generate the forecast. New product forecasting is a very difficult problem as such. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24964" LastActivityDate="2016-10-05T19:13:53.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14387" PostTypeId="2" ParentId="14352" CreationDate="2016-10-05T19:53:19.497" Score="26" Body="&lt;p&gt;You are right in that the basic concept of a deep NN hasn't changed since 2012.  But there have been a variety of improvements to the ways in which deep NNs are trained that have made them qualitatively more powerful.  There are also a wider variety of architectures available today.  I've listed some developments since 2012, grouped by training improvements and architecture improvements: &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Improvements to training deep NNs&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Hardware&lt;/strong&gt;: The most obvious change is just the inexorable progression of Moore's law.  There is more computing power available today.  Cloud computing also makes it easy for people to train large NNs without needing to buy a huge rig.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Software&lt;/strong&gt;: The open source software for deep learning is really enormously improved from 2012.  Back in 2012 there was Theano, maybe Caffe as well.  I'm sure there are some others, too.  But today we also have TensorFlow, Torch, Paddle, and CNTK, all of which are supported by large tech companies.  This is closely related to the hardware bullet point since many of these platforms make it easy to train on GPUs, which drastically speeds up training time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Activation functions&lt;/strong&gt;: The use of ReLU activation functions is probably more widespread these days, which makes training very deep networks easier.  On the research side, there is a wider variety of activation functions being studied, including &lt;a href=&quot;http://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf&quot; rel=&quot;noreferrer&quot;&gt;leaky ReLU&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/1502.01852&quot; rel=&quot;noreferrer&quot;&gt;parametric ReLU&lt;/a&gt;, and &lt;a href=&quot;https://arxiv.org/abs/1302.4389&quot; rel=&quot;noreferrer&quot;&gt;maxout units&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Optimization algorithms&lt;/strong&gt;: There are more optimization algorithms around today.  &lt;a href=&quot;http://jmlr.org/papers/v12/duchi11a.html&quot; rel=&quot;noreferrer&quot;&gt;Adagrad&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/abs/1212.5701&quot; rel=&quot;noreferrer&quot;&gt;Adadelta&lt;/a&gt; just been introduced in 2011 and 2012, respectively.  But we now also have the &lt;a href=&quot;https://arxiv.org/abs/1412.6980&quot; rel=&quot;noreferrer&quot;&gt;Adam optimizer&lt;/a&gt; and it's become a very popular choice.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Dropout&lt;/strong&gt;: In the past few years, &lt;a href=&quot;http://www.jmlr.org/papers/volume15/srivastava14a.old/source/srivastava14a.pdf&quot; rel=&quot;noreferrer&quot;&gt;dropout&lt;/a&gt; has become a standard tool for regularization when training neural networks.  Dropout is a computationally inexpensive form of ensembling for NNs.  In general, a set of models trained on random samples of the dataset will outperform a single model trained on the entire dataset.  This is difficult to do explicitly for NNs because they are so expensive to train.  But a similar effect can be approximated just by randomly &quot;turning off&quot; neurons on each step.  Different subgraphs in the NN end up getting trained on different data sets, and thereby learn different things.  Like ensembling, this tends to make the overall NN more robust to overfitting.  Dropout is a simple technique that seems to improve performance in almost every case, so it's now used de rigueur.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Batch normalization&lt;/strong&gt;: It's been known for a while that NNs train best on data that is normalized --- i.e., there is zero mean and unit variance.  In a very deep network, as the data passes through each layer, the inputs will be transformed and will generally drift to a distribution that lacks this nice, normalized property.  This makes learning in these deeper layers more difficult because, from its perspective, its inputs do not have zero mean and unit variance.  The mean could be very large and the variance could be very small.  &lt;a href=&quot;https://arxiv.org/abs/1502.03167v3&quot; rel=&quot;noreferrer&quot;&gt;Batch normalization&lt;/a&gt; addresses this by transforming the inputs to a layer to have zero mean and unit variance.  This seems to be enormously effective in training very deep NNs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Theory&lt;/strong&gt;: Up until very recently, it was thought that the reason deep NNs are hard to train is that the optimization algorithms get stuck in local minima and have trouble getting out and finding global minima.  In the last four years there have been a number of studies that seem to indicate that this intuition was wrong (e.g., &lt;a href=&quot;https://arxiv.org/abs/1412.6544&quot; rel=&quot;noreferrer&quot;&gt;Goodfellow et al. 2014&lt;/a&gt;).  In the very high dimensional parameter space of a deep NN, local minima tend not to be that much worse than global minima.  The problem is actually that when training, the NN can find itself on a long, wide plateau.  Furthermore, these plateaus can end abruptly in a steep cliff.  If the NN takes small steps, it takes a very long time to learn.  But if the steps are too large, it meets a huge gradient when it runs into the cliff, which undoes all the earlier work.  (This can be avoided with gradient clipping, another post-2012 innovation.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;New architectures&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Residual networks&lt;/strong&gt;: Researchers have been able to train incredibly deep networks (more than 1000 layers!) using &lt;a href=&quot;https://arxiv.org/abs/1605.07146&quot; rel=&quot;noreferrer&quot;&gt;residual networks&lt;/a&gt;.  The idea here is that each layer receives not only the output from the previous layer, but also the original input as well.  If trained properly, this encourages each layer to learn something different from the previous layers, so that each additional layer adds information.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Wide and deep networks&lt;/strong&gt;: Wide, shallow networks have a tendency to simply memorize the mapping between their inputs and their outputs.  Deep networks generalize much better.  Usually you want good generalization, but there are some situations, like recommendation systems, in which simple memorization without generalization is important, too.  In these cases you want to provide good, substantive solutions when a user makes a general query, but very precise solutions when the user makes a very specific query.  &lt;a href=&quot;https://arxiv.org/abs/1606.07792&quot; rel=&quot;noreferrer&quot;&gt;Wide and deep networks&lt;/a&gt; are able to fulfill this task nicely.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Neural turing machine&lt;/strong&gt;: A shortcoming of traditional recurrent NNs (whether they be the standard RNN or something more sophisticated like an LSTM) is that their memory is somewhat &quot;intuitive&quot;.  They manage to remember past inputs by saving the hidden layer activations they produce into the future.  However, sometimes it makes more sense to explicitly store some data.  (This might be the difference between writing a phone number down on a piece of paper vs. remembering that the number had around 7 digits and there were a couple of 3s in there and maybe a dash somewhere in the middle.)  The &lt;a href=&quot;https://arxiv.org/abs/1410.5401&quot; rel=&quot;noreferrer&quot;&gt;neural Turing machine&lt;/a&gt; is a way to try to address this issue.  The idea is that the network can learn to explicitly commit certain facts to a memory bank.  This is not straightforward to do because backprop algorithms require differentiable functions, but committing a datum to a memory address is an inherently discrete operation.  Consequently, neural Turing machines get around this by committing a little bit of data to a distribution of different memory addresses.  These architectures don't seem to work super well yet, but the idea is very important.  Some variant of these will probably become widespread in the future.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Generative adversarial networks&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/abs/1406.2661&quot; rel=&quot;noreferrer&quot;&gt;GANs&lt;/a&gt; are a very exciting idea that seems to be seeing a lot of practical use already.  The idea here is to train two NNs simultaneously: one that tries to generate samples from the underlying probability distribution (a generator), and one that tries to distinguish between real data points and the fake data points generated by the generator (a discriminator).  So, for example, if your dataset is a collection of &lt;a href=&quot;https://arxiv.org/abs/1511.06434v2&quot; rel=&quot;noreferrer&quot;&gt;pictures of bedrooms&lt;/a&gt;, the generator will try to make its own pictures of bedrooms, and the discriminator will try to figure out if it's looking at real pictures of bedrooms or fake pictures of bedrooms.  In the end, you have two very useful NNs: one that is really good at classifying images as being bedrooms or not bedrooms, and one that is really good at generating realistic images of bedrooms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18416" LastEditorUserId="18416" LastEditDate="2016-10-17T17:19:31.277" LastActivityDate="2016-10-17T17:19:31.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14388" PostTypeId="2" ParentId="14385" CreationDate="2016-10-05T21:05:34.127" Score="2" Body="&lt;p&gt;In general, the correlation coefficient is &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Mathematical_properties&quot; rel=&quot;nofollow&quot;&gt;&quot;invariant to separate changes in location and scale in the two variables&quot;&lt;/a&gt;. In particular, you can mix relative with absolute values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, that only works if you scale the variables globally. You can't scale every individual data point (here on a city level). If this was a county wide election, you could scale the city values by the county population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it sounds like your crime rates are on a per city level. In this case you should scale the votes on a city level as well to make them comparable. This will change the correlation coefficient and give a different result than with absolute values. I think using percentages is more intuitive in your case. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-10-05T21:05:34.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14389" PostTypeId="1" CreationDate="2016-10-05T23:22:07.690" Score="1" ViewCount="892" Body="&lt;p&gt;I have training data consisting of a time series of numerical values (e.g. a user activity metric on a website over a period of 100 days).&#xA;I also have some categorical attributes of the user (demographics, type of browser, location, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a good approach to predict for a new user (say on day 50) the future value of the activity metric, either as a numeric value or as a class (low, medium, high)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could think of the following approaches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. Perform time series regression or classification separately for most (or all) combinations of the initial categorical features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B. Extract 4 to 10 different numerical and/or categorical features from time series (e.g. high, low, average, weekend levels, weekend to weekday ratio, etc), then performing a random forrest classification or regression on the entire set of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C. Something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="24938" LastActivityDate="2016-10-06T02:33:38.210" Title="Predicting a numerical value based on past values and categorical attributes" Tags="|classification|predictive-modeling|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14391" PostTypeId="2" ParentId="14389" CreationDate="2016-10-06T02:33:38.210" Score="3" Body="&lt;p&gt;Without having seen the data I would say that activity sensors may have a strong seasonal behavior. If that is the case time series analysis could give you some good results. Some basic before you head of in that direction, make sure to make your time series stationary before you use it in ARIMA. See for a good tutorial &lt;a href=&quot;http://www.statmethods.net/advstats/timeseries.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to forecast a categorical variable you will have to make sure to create sufficient covariates based on historical values. You could then proceed with a Random Forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end I would recommend you to consider how well you should be able to explain the outcome of the model. ARIMAX models are more tractable than Random Forest since the latter use bagging and averaging. So if you need to explain why your model gives a certain forecast, ARIMAX will probably be easier to explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-06T02:33:38.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14393" PostTypeId="1" CreationDate="2016-10-06T04:06:41.703" Score="2" ViewCount="1492" Body="&lt;p&gt;I am running a Q-learning algorithm with a finite time horizon.  Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Q-learning#Initial_conditions_.28Q0.29&quot; rel=&quot;nofollow&quot;&gt;Wikipedia on Initial Q Values&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24968" LastEditorUserId="31513" LastEditDate="2017-05-09T18:11:32.547" LastActivityDate="2017-05-09T18:11:32.547" Title="Initial Q-values in Q-Learning" Tags="|machine-learning|reinforcement-learning|q-learning|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14394" PostTypeId="1" CreationDate="2016-10-06T10:03:15.077" Score="3" ViewCount="1279" Body="&lt;p&gt;I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold.  My desired output is one ROC curve with a corresponding AUC value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One method (taken from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;) is to take the mean false positivity rates (fpr) and true positivity rates (tpr) over all folds and plot the overall ROC curve using the mean tpr and fpr values. Then compute the AUC using the mean-ROC curve. However, this method does not work well when the dataset is small. Without a long explanation, my classification is a diagnosis that uses many samples for one diagnosis and thus reduces the predictions per fold to around 3-5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alternative method is to save the probabilities of each prediction in every fold and then construct a ROC curve after k-Fold CV and compute the AUC using this ROC curve. However, this would mean that various models, trained on different datasets are combined into one ROC curve. I don't know if this is an issue? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the industry standard for model evaluation reporting when using ROC and AUC combined with k-Fold Cross validation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;-feel free to edit my question. &lt;/p&gt;&#xA;" OwnerUserId="17222" LastActivityDate="2016-10-06T10:03:15.077" Title="How to represent ROC curve when using Cross-Validation" Tags="|machine-learning|cross-validation|model-evaluations|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14396" PostTypeId="1" CreationDate="2016-10-06T11:48:16.920" Score="22" ViewCount="7625" Body="&lt;p&gt;I have found other requests for references here. In particular in:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/5860/where-to-start-which-books&quot;&gt;Where to start, which books&lt;/a&gt;&#xA;and&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/313/books-about-the-science-in-data-science/407#407&quot;&gt;Books about the &amp;quot;Science&amp;quot; in Data Science?&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I have given a glance to:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Artificial Intelligence: A Modern Approach (Russel &amp;amp; Norvig)&lt;/li&gt;&#xA;&lt;li&gt;Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)&lt;/li&gt;&#xA;&lt;li&gt;Learning From Data (Abu-Mostafa et al.)&lt;/li&gt;&#xA;&lt;li&gt;Introduction to Statistical Learning (James et al.)&lt;/li&gt;&#xA;&lt;li&gt;Elements of Statistical Learning (Hastie et al.)&lt;/li&gt;&#xA;&lt;li&gt;Pattern Recognition and Machine Learning (Bishop)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Now it is difficult to evaluate if they would fit my needs because only a few pages are generally available online. However my first impression is that they do not. In the appendices of &lt;em&gt;Artificial Intelligence: A Modern Approach&lt;/em&gt; I can read:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Mathematicians define a vector as a member of a vector space, but we will use a more concrete definition: a vector is an ordered sequence of values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This is exactly the kind of approach I am &lt;strong&gt;not&lt;/strong&gt; looking for.&lt;/p&gt;&#xA;&lt;p&gt;I'm looking for a book which assumes the reader has a good understanding in set theory, abstract algebra, measure and probability theory, statistics, topology, calculus, graph theory, complexity theory, etc and &lt;strong&gt;a preference for formal and axiomatic explanations&lt;/strong&gt; rather than lengthy and so-called &amp;quot;intuitive&amp;quot; approaches based on basic mathematical objects and examples. Furthermore I don't want something that looks like a recipe book from the very beginning. I want a book that formalizes the abstract and common shape of all data science methods as well as their common aim first. Only after that it can start to explain the different categories by explicitly stating which further hypotheses each category is assuming and which cases/problems/domains they are known to handle efficiently or not.&lt;/p&gt;&#xA;&lt;p&gt;At last, to be clear, I have no problem with being shown concrete examples and their treatment via a specific programming language for example. I just want this to come second as an illustration for the conceptual explanation, not as a substitute.&lt;/p&gt;&#xA;" OwnerUserId="24980" LastEditorUserId="24980" LastEditDate="2023-12-24T10:00:54.770" LastActivityDate="2023-12-24T10:00:54.770" Title="Data science / machine learning books for mathematicians" Tags="|reference-request|books|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="14399" PostTypeId="1" CreationDate="2016-10-06T13:26:57.993" Score="1" ViewCount="78" Body="&lt;p&gt;I just picked my topic for my econometrics course thesis. I decided to create an algorithm that forecasts NBA players' performance. My inspiration is Silver Nate's PECOTA and CARMELO algorithms, but I have no empirical experience with Data Science techniques, so I am trying to avoid any misleading transitions from theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is going to compare the performance of a current player with other historical players and then forecast his performance based on the of his &quot;neighbors&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I'm going to pick all player statistics per season. Then, I'll create indexes to establish informative dimensions for kNN algorithm (eg. Offense, Defense, Physics). As next step I thought to use naïve Bayesian algorithm to find the likelihood of having the same performance with his neighbors and then pick the weighted (depended on likelihood) sum of the neighbors' stats as a forecast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Empirically speaking, do you see any bias on this methodology?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="24984" LastEditorUserId="24984" LastEditDate="2016-10-06T20:16:33.310" LastActivityDate="2016-10-06T20:16:33.310" Title="Econometrics Thesis methodology Suggestions?" Tags="|machine-learning|time-series|data|naive-bayes-classifier|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14400" PostTypeId="2" ParentId="14312" CreationDate="2016-10-06T13:51:16.397" Score="1" Body="&lt;p&gt;As others have mentioned, the more information the better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, assuming that you need to explain the forecast to other, non-technical individuals, I would recommend an &lt;a href=&quot;https://en.wikipedia.org/wiki/EWMA_chart&quot; rel=&quot;nofollow&quot;&gt;Exponential Weighted Moving Average&lt;/a&gt;. In short, the EWMA will give higher weight to more recent historical sales. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've attached some &lt;a href=&quot;http://training-course-material.com/training/R_-_Forecasting#Triple_exponential_smoothing&quot; rel=&quot;nofollow&quot;&gt;R examples&lt;/a&gt; to highlight exponential smoothing&lt;/p&gt;&#xA;" OwnerUserId="24945" LastActivityDate="2016-10-06T13:51:16.397" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14401" PostTypeId="2" ParentId="14396" CreationDate="2016-10-06T14:49:07.383" Score="12" Body="&lt;p&gt;First of all, data only comes in so many forms that it might make sense to stick to a more &quot;concrete definition&quot;. Data Science is necessarily practical. But here are a few other books with a more theoretical grounding. Others will certainly know many more...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387946187&quot; rel=&quot;noreferrer&quot;&gt;A probabilistic theory of pattern recognition&lt;/a&gt; by Devroye, Györfi, Lugosi&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262111934&quot; rel=&quot;noreferrer&quot;&gt;An Introduction to Computational Learning Theory&lt;/a&gt; by Kearns&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0521642981&quot; rel=&quot;noreferrer&quot;&gt;Information theory, inference and learning algorithms&lt;/a&gt; by MacKay&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262018020&quot; rel=&quot;noreferrer&quot;&gt;Machine learning: a probabilistic perspective&lt;/a&gt; by Murphy&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1107057132&quot; rel=&quot;noreferrer&quot;&gt;Understanding machine learning&lt;/a&gt; by Shalev-Shwartz, Ben-David&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, research in machine learning is mostly found in journals and papers. It will be hard to find one or a few books that cover everything you want to know.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-10-06T14:49:07.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14402" PostTypeId="1" AcceptedAnswerId="14430" CreationDate="2016-10-06T15:10:40.077" Score="1" ViewCount="232" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5Uccim.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Uccim.jpg&quot; alt=&quot;Single Threshold&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Jlrm5m.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Jlrm5m.jpg&quot; alt=&quot;Multiple Threshold&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image on the left shows a standard ROC curve formed by sweeping a single threshold and recording the corresponding True Positive Rate (TPR) and False Positive Rate (FPR). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image on the right shows my problem setup where there are 3 parameters, and for each, we have only 2 choices. Together, it produces 8 points as depicted on the graph. In practice, I intend to have thousands of possible combinations of 100s of parameters, but the concept remains the same in this down-scaled case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I intend to find 2 things here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Determine the optimum parameter(s) for the given data&lt;/li&gt;&#xA;&lt;li&gt;Provide an overall performance score for all combinations of parameters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In the case of the ROC curve on the left, this is done easily using the following methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Optimal parameter: Maximal difference of TPR and FPR with a cost component (I believe it is called the J-statistic?)&lt;/li&gt;&#xA;&lt;li&gt;Overall performance: Area under the curve (the shaded portion in the graph)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, for my case in the image on the right, I do not know if the methods I have chosen are the standard principled methods that are normally used. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Optimal parameter set: Same maximal difference of TPR and FPR&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Parameter score = TPR - FPR * cost_ratio&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Overall performance: Average of all &quot;parameter scores&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have found a lot of reference material for the ROC curve with a single threshold and while there are other techniques available to determine the performance, the ones mentioned in this question is definitely considered a standard approach. I found no such reading material for the scenario presented on the right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottomline, the question here is two-fold: (1) Provide methods to evaluate the optimal parameter set and overall performance in my problem scenario, (2) Provide reference that claims the suggested methods to be a standard approach for the given scenario.&lt;/p&gt;&#xA;" OwnerUserId="18983" LastActivityDate="2016-10-09T13:59:33.077" Title="Optimal parameter estimation for a classifier with multiple parameters" Tags="|machine-learning|optimization|performance|parameter-estimation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14403" PostTypeId="1" CreationDate="2016-10-06T16:23:31.170" Score="8" ViewCount="8392" Body="&lt;p&gt;I wonder why training RNNs typically doesn't use 100% of the GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I run this &lt;a href=&quot;https://github.com/glample/rnn-benchmarks/tree/master/theano&quot; rel=&quot;noreferrer&quot;&gt;RNN benchmark&lt;/a&gt; on a Maxwell Titan X on Ubuntu 14.04.4 LTS x64, the GPU utilization is below 90%:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VbkoS.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VbkoS.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benchmark was launched using the command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;python rnn.py -n 'fastlstm' -l 1024 -s 30 -b 128&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I diagnose   what the bottleneck is?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-10-12T00:36:39.640" Title="Why doesn't training RNNs use 100% of the GPU?" Tags="|performance|theano|rnn|gpu|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14405" PostTypeId="1" AcceptedAnswerId="14408" CreationDate="2016-10-06T20:23:07.963" Score="1" ViewCount="1054" Body="&lt;p&gt;I'm trying to determine number of clusters for k-means using &lt;code&gt;sklearn.metrics.silhouette_score&lt;/code&gt;. I have computed it for &lt;code&gt;range(2,50)&lt;/code&gt; clusters. How to interpret this? What number of clusters should I choose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Suw3W.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Suw3W.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24994" LastActivityDate="2016-10-07T15:40:06.103" Title="How to interpret silouette coefficient?" Tags="|clustering|scikit-learn|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14406" PostTypeId="1" AcceptedAnswerId="14409" CreationDate="2016-10-06T21:27:28.460" Score="10" ViewCount="6804" Body="&lt;p&gt;I have a dataset in following structure inserted in a CSV file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Banana  Water   Rice&#xA;Rice    Water&#xA;Bread   Banana  Juice&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row indicates a collection of items that were purchased together.  For example, the first row denotes that the items &lt;code&gt;Banana&lt;/code&gt;, &lt;code&gt;Water&lt;/code&gt;, and &lt;code&gt;Rice&lt;/code&gt; were purchased together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to create a visualization like the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zAs5c.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zAs5c.png&quot; alt=&quot;example visualization&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is basically a grid chart but I need some tool (maybe Python or R) that  can read the input structure and produce a chart like the above as output.&lt;/p&gt;&#xA;" OwnerUserId="23714" LastEditorUserId="23714" LastEditDate="2016-10-15T23:03:43.210" LastActivityDate="2023-01-24T04:42:04.663" Title="Visualizing items frequently purchased together" Tags="|python|r|data-mining|visualization|association-rules|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14408" PostTypeId="2" ParentId="14405" CreationDate="2016-10-06T23:45:04.223" Score="2" Body="&lt;p&gt;They are all bad. A good Silhouette would be 0.7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try other clustering algorithms instead.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-10-06T23:45:04.223" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14409" PostTypeId="2" ParentId="14406" CreationDate="2016-10-06T23:54:42.783" Score="6" Body="&lt;p&gt;I think what you probably want is a discrete version of a heat map.  For example, see below.  The red colors indicate the most commonly purchased together, while green cells are never purchased together.&#xA; &lt;a href=&quot;https://i.stack.imgur.com/nPNmn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nPNmn.png&quot; alt=&quot;heat map&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is actually fairly easy to put together with Pandas DataFrames and matplotlib.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from pandas import DataFrame&#xA;import matplotlib&#xA;matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)&#xA;import matplotlib.pyplot as plt&#xA;&#xA;####&#xA;# Get data into a data frame&#xA;####&#xA;data = [&#xA;  ['Banana', 'Water', 'Rice'],&#xA;  ['Rice', 'Water'],&#xA;  ['Bread', 'Banana', 'Juice'],&#xA;]&#xA;&#xA;# Convert the input into a 2D dictionary&#xA;freqMap = {}&#xA;for line in data:&#xA;  for item in line:&#xA;    if not item in freqMap:&#xA;      freqMap[item] = {}&#xA;&#xA;    for other_item in line:&#xA;      if not other_item in freqMap:&#xA;        freqMap[other_item] = {}&#xA;&#xA;      freqMap[item][other_item] = freqMap[item].get(other_item, 0) + 1&#xA;      freqMap[other_item][item] = freqMap[other_item].get(item, 0) + 1&#xA;&#xA;df = DataFrame(freqMap).T.fillna(0)&#xA;print (df)&#xA;&#xA;#####&#xA;# Create the plot&#xA;#####&#xA;plt.pcolormesh(df, edgecolors='black')&#xA;plt.yticks(np.arange(0.5, len(df.index), 1), df.index)&#xA;plt.xticks(np.arange(0.5, len(df.columns), 1), df.columns)&#xA;plt.savefig('plot.png')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9121" LastActivityDate="2016-10-06T23:54:42.783" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14412" PostTypeId="1" CreationDate="2016-10-07T06:16:40.917" Score="11" ViewCount="1141" Body="&lt;p&gt;I have the network &lt;a href=&quot;https://i.stack.imgur.com/4wUjg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;architecture&lt;/a&gt; from the paper &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwj9htujhMjPAhWMP48KHeplDPYQFgggMAA&amp;amp;url=http%3A%2F%2Fusers.eecs.northwestern.edu%2F~jwa368%2Fpdfs%2Fdeep_ranking.pdf&amp;amp;usg=AFQjCNHgIMYAfMbLvfNk3MkKB5WvKXOfTw&amp;amp;sig2=YhsrJcxUtSGz8O1dmsurTA&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;learning fine-grained image similarity with deep ranking&quot;&lt;/a&gt; and I am unable to figure out how the output from the three parallel network is merged using the linear embedding layer.&#xA;The only information given on this layer, in the paper is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Finally, we normalize the embeddings from the three parts, and combine them with a linear embedding layer. The dimension of the embedding is 4096.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me in figuring out what exactly does the author mean when he is talking about this layer?&lt;/p&gt;&#xA;" OwnerUserId="24891" LastEditorUserId="31245" LastEditDate="2018-11-23T12:25:25.103" LastActivityDate="2019-06-10T15:03:10.363" Title="Unable to figure out the linear embedding layer in the convolutional neural network?" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="14413" PostTypeId="2" ParentId="14406" CreationDate="2016-10-07T06:31:22.793" Score="3" Body="&lt;p&gt;For &lt;code&gt;R&lt;/code&gt;, you can use library &lt;code&gt;ArulesViz&lt;/code&gt;. There is nice &lt;a href=&quot;https://cran.r-project.org/web/packages/arulesViz/vignettes/arulesViz.pdf&quot; rel=&quot;nofollow&quot;&gt;documentation&lt;/a&gt; and on the page 12, there is example how to create this kind of visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for that is as simple as this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(rules, method=&quot;grouped&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-10-07T06:31:22.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14415" PostTypeId="1" AcceptedAnswerId="14742" CreationDate="2016-10-07T08:10:51.287" Score="39" ViewCount="56601" Body="&lt;p&gt;How does Keras calculate accuracy from the classwise probabilities? Say, for example we have 100 samples in the test set which can belong to one of two classes. We also have a list of the classwise probabilites. What threshold does Keras use to assign a sample to either of the two classes?&lt;/p&gt;&#xA;" OwnerUserId="14360" LastActivityDate="2021-02-08T02:03:49.750" Title="How does Keras calculate accuracy?" Tags="|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14417" PostTypeId="2" ParentId="14405" CreationDate="2016-10-07T15:40:06.103" Score="0" Body="&lt;p&gt;Silhouette measures BOTH the separation between clusters AND cohesion in respective clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively speaking, it is the difference between separation B (average distance between each point and all points of its nearest cluster) and cohesion A (average distance between each point and all other points in its cluster) divided by max(A,B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a value between -1 and 1, the higher the better (negative value means that the point is more closer to the nearest cluster than to its own, which is quite a problem).&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-10-07T15:40:06.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14418" PostTypeId="1" CreationDate="2016-10-07T17:02:41.657" Score="1" ViewCount="61" Body="&lt;p&gt;Check this phrase in this online book:&#xA;&lt;a href=&quot;https://hyp.is/GsHrloyvEeaYQ093X053Wg/neuralnetworksanddeeplearning.com/chap2.html&quot; rel=&quot;nofollow&quot;&gt;https://hyp.is/GsHrloyvEeaYQ093X053Wg/neuralnetworksanddeeplearning.com/chap2.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It refers that the quadratic cost function (RSS) can be written as an average for each individual training example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so if the cost of a single training example is Cx = (1/2) * ||y-a||^2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can have C = (1/n) * Σ(Cx)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you give an example of an error / cost function that cannot be written as an average?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2021-07-10T17:55:56.343" Title="How could have a loss/cost/error function that cannot be written as an average?" Tags="|neural-network|loss-function|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14419" PostTypeId="2" ParentId="14406" CreationDate="2016-10-07T23:23:36.173" Score="0" Body="&lt;p&gt;With &lt;a href=&quot;http://reference.wolfram.com/language/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wolfram Language&lt;/a&gt; in &lt;a href=&quot;http://www.wolfram.com/mathematica/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mathematica&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = {{&quot;Banana&quot;, &quot;Water&quot;, &quot;Rice&quot;},&#xA;        {&quot;Rice&quot;, &quot;Water&quot;},&#xA;        {&quot;Bread&quot;, &quot;Banana&quot;, &quot;Juice&quot;}};&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Get pairwise counts.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;counts = Sort /@ Flatten[Subsets[#, {2}] &amp;amp; /@ data, 1] // Tally&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;{{{&quot;Banana&quot;, &quot;Water&quot;}, 1}, {{&quot;Banana&quot;, &quot;Rice&quot;}, 1}, &#xA; {{&quot;Rice&quot;, &quot;Water&quot;}, 2}, {{&quot;Banana&quot;, &quot;Bread&quot;}, 1}, &#xA; {{&quot;Bread&quot;, &quot;Juice&quot;}, 1}, {{&quot;Banana&quot;, &quot;Juice&quot;}, 1}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Get indices for named ticks.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;indices = Thread[# -&amp;gt; Range[Length@#]] &amp;amp;@Sort@DeleteDuplicates@Flatten[data]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;Banana&quot; -&amp;gt; 1, &quot;Bread&quot; -&amp;gt; 2, &quot;Juice&quot; -&amp;gt; 3, &quot;Rice&quot; -&amp;gt; 4, &quot;Water&quot; -&amp;gt; 5}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Plot with &lt;code&gt;MatrixPlot&lt;/code&gt; using &lt;code&gt;SparseArray&lt;/code&gt;. Could also use &lt;code&gt;ArrayPlot&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MatrixPlot[&#xA; SparseArray[Rule @@@ counts /. indices, ConstantArray[Length@indices, 2]],&#xA; FrameTicks -&amp;gt; With[{t = {#2, #1} &amp;amp; @@@ indices}, {{t, None}, {t, None}}],&#xA; PlotLegends -&amp;gt; Automatic&#xA; ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wI8Z5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wI8Z5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that it is upper-triangular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="10814" LastActivityDate="2016-10-07T23:23:36.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14420" PostTypeId="2" ParentId="6950" CreationDate="2016-10-08T03:18:50.820" Score="-1" Body="&lt;p&gt;If you're not sure which label ROCR took as +ve then check str(pred_obj@labels) and the greater one shown is considered +ve; if you want to change that then use label.ordering argument supplying a vector containing -ve and +ve labels while creating prediction object&lt;/p&gt;&#xA;" OwnerUserId="25026" LastEditorUserId="25026" LastEditDate="2016-10-08T04:22:42.267" LastActivityDate="2016-10-08T04:22:42.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14421" PostTypeId="1" AcceptedAnswerId="15063" CreationDate="2016-10-08T13:42:09.100" Score="1" ViewCount="195" Body="&lt;p&gt;I intend to make a classifier using the feature map obtained from a CNN. Can someone suggest how I can do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it work if I first train the CNN using +ve and -ve samples (and hence obtain the weights), and then every time I need to classify an image, I apply the conv and pooling layers to obtain the feature map? The problem I find in this, is that the image I want to classify, may not have a similar feature map, and hence I wouldn't be able to find the distance correctly. As the order of the features may by different in the layer.&lt;/p&gt;&#xA;" OwnerUserId="25033" LastEditorUserId="29169" LastEditDate="2021-02-16T15:36:18.913" LastActivityDate="2021-02-16T15:36:18.913" Title="How can I create a classifier using the feature map of a CNN?" Tags="|deep-learning|neural-network|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14422" PostTypeId="1" AcceptedAnswerId="14498" CreationDate="2016-10-08T16:23:39.617" Score="2" ViewCount="1342" Body="&lt;p&gt;I am confused about the input vector in LSTM model, the data I am using is the text data, e.g. 1,000 sentences. I have two questions about the LSTM input layer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.If I would tokenize those sentences into the vectors (we can call it sentence vectors), is there a way in Keras to make sentence vectors given a document? Should be word level, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.The second question is the 3D Tensor type in LSTM. I have 1,000 sentences (samples) and time_step would be 1 if I want to LSTM read one document at each time step, is that correct? The last one is the input dimension, this input dimension is the word dimension (100) in each sentence or how many word observed in each time step (10)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus the LSTM tensor should be (1000, 1, 10) or (1000, 1, 100)&lt;/p&gt;&#xA;" OwnerUserId="23865" LastActivityDate="2016-10-13T10:36:32.760" Title="LSTM input in Keras" Tags="|keras|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14424" PostTypeId="2" ParentId="8909" CreationDate="2016-10-08T22:22:53.753" Score="3" Body="&lt;p&gt;As of Oct 2016 there's also a &lt;a href=&quot;https://github.com/malmaud/TensorFlow.jl&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow wrapper for Julia&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="98307" LastEditDate="2020-08-05T13:58:11.387" LastActivityDate="2020-08-05T13:58:11.387" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14425" PostTypeId="2" ParentId="14403" CreationDate="2016-10-08T22:35:49.707" Score="5" Body="&lt;p&gt;I get about this same utilization rate when I train models using Tensorflow. The reason is pretty clear in my case, I'm manually choosing a random batch of samples and calling the optimization for each batch separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That means that each batch of data is in main memory, it's then copied into GPU memory where the rest of the model is, then forward/back propagation and update is performed in-gpu, then execution is handed back to my code where I grab another batch and call optimize on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a faster way to do that if you spend a few hours setting up Tensorflow to do batch loading in parallel from pre-prepared TF records. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize you may or may not be using tensorflow under keras, but since my experience tends to produce very similar utilization numbers, I'm going out on a limb by suggesting that there's a reasonably likely causal link to draw from these correlations. If your framework is loading each batch from main memory into the GPU without the added efficiency/complexity of asynchronous loading (which the GPU itself can handle), then this &lt;em&gt;would&lt;/em&gt; be an expected result.&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="24792" LastEditDate="2016-10-12T00:36:39.640" LastActivityDate="2016-10-12T00:36:39.640" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14426" PostTypeId="2" ParentId="13701" CreationDate="2016-10-09T08:49:45.707" Score="0" Body="&lt;p&gt;&lt;code&gt;org.apache.spark.SparkException: Items in a transaction must be unique but got WrappedArray(13873775, 4, 99, 9909, 102113020, 15704, 2012-03-19:00, 6.25, OZ, 4, 11.96).&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error pretty much tells you the problem. A basket is a set of &lt;em&gt;unique&lt;/em&gt; items, and 4 occurs twice here. That's not valid input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However you have a much bigger problem here. You are parsing a line of text that clearly is just a description of one item, and interpreting it as a bunch of item IDs. You have a date, weight, units, etc. This is completely invalid as input.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-10-09T08:49:45.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14427" PostTypeId="2" ParentId="9433" CreationDate="2016-10-09T11:26:52.143" Score="1" Body="&lt;p&gt;Scala:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def weightedSampleWithReplacement[T](data: Array[T], &#xA;                                     weights: Array[Double], &#xA;                                     n: Int, &#xA;                                     random: Random): Array[T] = {&#xA;  val cumWeights = weights.scanLeft(0.0)(_ + _)&#xA;  val cumProbs = cumWeights.map(_ / cumWeights.last)&#xA;  Array.fill(n) {&#xA;    val r = random.nextDouble()&#xA;    data(cumProbs.indexWhere(r &amp;lt; _) - 1)&#xA;  }&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Spark has an &lt;code&gt;RDD.sample()&lt;/code&gt; method that can sample without replacement, though not with weights. You could probably adapt that method along the lines above to do this however.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-10-09T11:26:52.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14428" PostTypeId="1" AcceptedAnswerId="14462" CreationDate="2016-10-09T12:17:22.307" Score="8" ViewCount="3881" Body="&lt;p&gt;What metrics can be used for evaluating text clustering models? I used &lt;code&gt;tf-idf&lt;/code&gt; + &lt;code&gt;k-means&lt;/code&gt;, &lt;code&gt;tf-idf&lt;/code&gt; + &lt;code&gt;hierarchical clustering&lt;/code&gt;, &lt;code&gt;doc2vec&lt;/code&gt; + &lt;code&gt;k-means (metric is cosine similarity)&lt;/code&gt;, &lt;code&gt;doc2vec&lt;/code&gt; + &lt;code&gt;hierarchical clustering (metric is cosine similarity)&lt;/code&gt;. &#xA;How to decide which model is the best?&lt;/p&gt;&#xA;" OwnerUserId="24994" LastActivityDate="2023-10-14T12:12:00.690" Title="How evaluate text clustering?" Tags="|machine-learning|clustering|text-mining|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14429" PostTypeId="1" AcceptedAnswerId="14663" CreationDate="2016-10-09T13:23:56.780" Score="12" ViewCount="13996" Body="&lt;p&gt;I am making a simple neural network using tensorflow, with data I collected myself, however, it's not cooperating :P I have run into an error that I can't fix or find the fix for and I would love your help.&lt;/p&gt;&#xA;&lt;p&gt;The errormessage:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;TypeError: Fetch argument 2861.6152 of 2861.6152 has invalid type &amp;lt;class 'numpy.float32'&amp;gt;, must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The error refers to the following line in my code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have already figured out that the error doesn't occur when I comment out the following lines in my code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;prediction = neural_network_model(champion_data)&#xA;cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(prediction, item_data))&#xA;optimizer = tf.train.AdamOptimizer().minimize(cost)&#xA;_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So somewhere one of these lines is getting something that doesn't look exactly like it expects it to look. I have already tried the obvious (removing the &lt;code&gt;np.array()&lt;/code&gt; from batch_input and batch_output or replacing it by &lt;code&gt;list()&lt;/code&gt;) but that doesn't solve the problem. My current hypothesis is that the output of &lt;code&gt;neural_network_model(champion_data)&lt;/code&gt; is somehow of the wrong shape or type, however I'm not sure how to test that or how to solve it if that turns out to be the case.&lt;/p&gt;&#xA;&lt;p&gt;The full code can be found &lt;a href=&quot;https://gist.github.com/HasseIona/4bcaf9f95ae828e056d5210a2ea07f88&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Edit: I have verified that the champion data that are input into the &lt;code&gt;neural_network_model&lt;/code&gt;, the prediction and the cost are all tensors. I've been trying to solve the problem using the hypothesis that the problem somehow lies in the &lt;code&gt;feed_dict = {}&lt;/code&gt; part of the code, but not getting anywhere so far&lt;/p&gt;&#xA;" OwnerUserId="19067" LastEditorUserId="98307" LastEditDate="2020-08-02T10:29:09.177" LastActivityDate="2020-08-02T10:29:09.177" Title="Tensorflow neural network TypeError: Fetch argument has invalid type" Tags="|machine-learning|python|neural-network|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14430" PostTypeId="2" ParentId="14402" CreationDate="2016-10-09T13:59:33.077" Score="0" Body="&lt;p&gt;The answer depends a lot on what you want to achieve:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Please future reader's of your machine learning paper OR&lt;/li&gt;&#xA;&lt;li&gt;Solve a use-case that matters to some business&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In first case, you have to scan this audience's publications, see what they use, and use the same. There isn't a perfect metric, since actually it depends on the business' cost-ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second case, all you need is the cost-matrix and the result follows naturally. Optimizing anything which isn't the cost-matrix is a rather academical exercise. The cost-matrix has one parameter and depends really on the business. No reference is going to solve this decision for you. The discussion between performance metrics is missing the point. There is only one answer once you manage to squeeze out a cost-matrix from your business stakeholders. In a business anything can be estimated in terms of money. It only gets tricky, if it more complex than a single class weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answers are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Parameter performance: calculate cost; Overall performance: take max(), or to be more precise correct for testing multiple hypotheses, too&lt;/li&gt;&#xA;&lt;li&gt;There isn't any, unless you find one which proves that optimizing a metric optimizes for all possible cost matrices simultaneously.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In that sense, ROCAUC is actually a pretty meaningless artifical metric, despite being popular. There is an interpretation which relies on the fitted model's probabilities being correctly estimated, but if you want to get real, only cost matrices matter. Model the real world &lt;em&gt;impact&lt;/em&gt; of you model and you'll get answers!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I personally consider TPR-FPR the best bet in absence of a defined cost matrix.&lt;/p&gt;&#xA;" OwnerUserId="723" LastActivityDate="2016-10-09T13:59:33.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14432" PostTypeId="1" CreationDate="2016-10-09T23:16:24.600" Score="6" ViewCount="13487" Body="&lt;p&gt;I am using convolutional neural networks (via Keras) as my model for facial expression recognition (55 subjects). My data set is quite hard and around 450k with 7 classes. I have balanced my training set per subject and per class label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I implemented a very simple CNN architecture (with real-time data augmentation):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Convolution2D(32, 3, 3, border_mode=borderMode, init=initialization,  input_shape=(48, 48, 3)))&#xA;model.add(BatchNormalization())&#xA;model.add(PReLU())&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&#xA;model.add(Flatten())&#xA;model.add(Dense(256))&#xA;model.add(BatchNormalization())&#xA;model.add(PReLU())&#xA;model.add(Dropout(0.5))&#xA;&#xA;model.add(Dense(nb_output))&#xA;model.add(Activation('softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After first epoch, my training loss decreases constantly while validation loss increases. Could overfitting happen that soon? Or is there a problem with my data being confusing? Should I also balance my testing set?&lt;/p&gt;&#xA;" OwnerUserId="24913" LastEditorUserId="29169" LastEditDate="2021-02-16T15:36:09.727" LastActivityDate="2021-02-16T15:36:09.727" Title="Overfitting after first epoch" Tags="|machine-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14433" PostTypeId="1" AcceptedAnswerId="14445" CreationDate="2016-10-10T00:06:31.593" Score="6" ViewCount="8159" Body="&lt;p&gt;In decision tree classifier most of the algorithms use &lt;em&gt;Information gain&lt;/em&gt; as spiting criterion. We select the feature with maximum information gain to split on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that using accuracy instead of information gain is simpler approach. Is there any scenario where accuracy doesn't work and information gain does?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain what are the advantages of using Information gain over accuracy as splitting criterion?  &lt;/p&gt;&#xA;" OwnerUserId="10049" LastActivityDate="2016-10-11T17:59:46.160" Title="Why we use information gain over accuracy as splitting criterion in decision tree?" Tags="|machine-learning|classification|decision-trees|information-theory|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14434" PostTypeId="1" AcceptedAnswerId="14443" CreationDate="2016-10-10T03:27:49.467" Score="2" ViewCount="198" Body="&lt;p&gt;I am not sure how to formulate this problem clearly into a machine learning task yet. So hope you guys can chime in and give me some help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem :&lt;/strong&gt; To predict whether someone will pick up their phone during office hours in &lt;strong&gt;week n+2&lt;/strong&gt; by looking at customer's behaviour in &lt;strong&gt;week  n.&lt;/strong&gt; &lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data    :&lt;/strong&gt; I have calling records for about 3 months, which is aggregated on customer level. The various attributes include, num of calls, duration of calls, time of calls, amount of data traffic. But of course, these main attributes are further split into at about 20 attributes.&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current Approach (Very Manual)&lt;/strong&gt; : I look at data at week n+2 and get the group of guys who picked up the phone during office hours (duration of calls &gt; 5s and time of call). This is the target group, T.&lt;br/&gt;&#xA;I look at data at week n and manually try all possible combinations of the attributes to get as close to T as possible. But trying manually seems tiring after some time. The baseline is of course using the same conditions as at week n+2. But the whole idea will be to increase this number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question : Is there any way I can transform this dataset so that I would be able to do accomplish it as a machine learning task ?&lt;/p&gt;&#xA;" OwnerUserId="18819" LastActivityDate="2020-01-12T02:03:15.410" Title="Period Predictive Model" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14435" PostTypeId="1" AcceptedAnswerId="14441" CreationDate="2016-10-10T06:17:43.313" Score="12" ViewCount="24243" Body="&lt;p&gt;I need to get the probability for each point in my data set. The idea is to compute distance matrix (first column contsins distances to first cluster, second column conteins distances to second cluster and etc). The closest point has probability = 1, the most distant has probability = 0.&#xA;The problem is linear function (like MinMaxScaller) have output where almost all points have almost the same probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to choose nonlinearity for this task? How to automatizate this process on python? For example for the most closest point &lt;code&gt;p=1&lt;/code&gt;, for the most distant point that belongs to cluster &lt;code&gt;p=0.5&lt;/code&gt;, for the most distant point p is almols 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you can propose another methods for computing this probability.&lt;/p&gt;&#xA;" OwnerUserId="24994" LastEditorUserId="24994" LastEditDate="2016-10-10T06:45:32.233" LastActivityDate="2019-07-16T05:32:36.193" Title="How to get the probability of belonging to clusters for k-means?" Tags="|python|clustering|k-means|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14439" PostTypeId="2" ParentId="14428" CreationDate="2016-10-10T09:52:01.513" Score="0" Body="&lt;p&gt;A clustering quality measure would be very nice to have. Unfortunately, that measure is hard to calculate -- probably AI-hard. You are trying to reduce a very complex thing to a single number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is AI-hard, then you could ask people to rate the clusterings somehow. It's not ideal, and won't scale but you will have a  single number that represents something close to what you want. &lt;/p&gt;&#xA;" OwnerUserId="25070" LastActivityDate="2016-10-10T09:52:01.513" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14441" PostTypeId="2" ParentId="14435" CreationDate="2016-10-10T12:43:12.837" Score="14" Body="&lt;p&gt;Let us briefly talk about a probabilistic generalisation of &lt;em&gt;k&lt;/em&gt;-means: the &lt;a href=&quot;https://en.wikipedia.org/wiki/Mixture_model&quot;&gt;Gaussian Mixture Model&lt;/a&gt; (GMM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;em&gt;k&lt;/em&gt;-means, you carry out the following procedure:&lt;br&gt;&#xA;- specify &lt;em&gt;k&lt;/em&gt; centroids, initialising their coordinates randomly&lt;br&gt;&#xA;- calculate the distance of each data point to each centroid&lt;br&gt;&#xA;- assign each data point to its nearest centroid&lt;br&gt;&#xA;- update the coordinates of the centroid to the mean of all points   assigned to it&lt;br&gt;&#xA;- iterate until convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a GMM, you carry out the following procedure:&lt;br&gt;&#xA;- specify &lt;em&gt;k&lt;/em&gt; multivariate Gaussians (termed &lt;em&gt;components&lt;/em&gt;), initialising their mean and variance randomly&lt;br&gt;&#xA;- calculate the probability of each data point being produced by each component (sometimes termed the &lt;em&gt;responsibility&lt;/em&gt; each component takes for the data point)&lt;br&gt;&#xA;- assign each data point to the component it belongs to with the highest probability&lt;br&gt;&#xA;- update the mean and variance of the component to the mean and variance of all data points assigned to it&lt;br&gt;&#xA;- iterate until convergence  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may notice the similarity between these two procedures. In fact, &lt;em&gt;k&lt;/em&gt;-means is a GMM with fixed-variance components. Under a GMM, the probabilities (I think) you're looking for are the responsibilities each component takes for each data point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a &lt;a href=&quot;http://scikit-learn.org/stable/modules/mixture.html&quot;&gt;scikit-learn implementation of GMM&lt;/a&gt; available if you wanted to look into that, but I'm guessing you just want a quick way to amend your existing code, in which case, if you're happy to assume your clusters are fixed-variance Gaussians, you could transform your distance matrix element-wise as $y = e^{-x}$ (giving you an exponential fall-off), and then calculating the softmax over your columns (normalising your distribution so $P(Y=1) + P(Y=2) + ... + P(Y=k) = 1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth pointing out that the assumption your clusters are fixed-variance Gaussians isn't necessarily valid. If your dimensions have wildly different scales, this may produce strange results, as dimensions with smaller-magnitude units will appear more &quot;probable&quot;. Standardising your data before running your clustering procedure should remedy this.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastEditorUserId="24895" LastEditDate="2016-10-10T16:07:58.040" LastActivityDate="2016-10-10T16:07:58.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14442" PostTypeId="2" ParentId="14154" CreationDate="2016-10-10T12:57:37.750" Score="0" Body="&lt;p&gt;&lt;code&gt;LabeledPoint&lt;/code&gt; expects: &lt;code&gt;LabeledPoint(label: Double, features: Vector)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when you execute &lt;code&gt;val trainingData2=LabeledPoint(1.0,trainingData.collect())&lt;/code&gt; you're actually getting all the Rows in your trainingData set so you will have &lt;code&gt;Array(Row(???), Row(???), ???)&lt;/code&gt; but what you need is to apply &lt;code&gt;def transformToLabeledPoint(vector: Vector) = LabeledPoint(1.0, vector)&lt;/code&gt;, you can either apply a map on your trainingData or create a UDF and apply it to your trainingData, for instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.ml.feature.SQLTransformer&#xA;import org.apache.spark.mllib.regression.LabeledPoint&#xA;import org.apache.spark.mllib.linalg.Vectors&#xA;&#xA;val df = spark.createDataFrame(Seq((1.0,100.0),(2.0,200.0))).toDF(&quot;id&quot;, &quot;value&quot;)&#xA;&#xA;spark.udf.register(&quot;setLabel&quot;, (label: Double, x: Seq[Double]) =&amp;gt; LabeledPoint(label, Vectors.dense(x.toArray)))&#xA;&#xA;val labeledTrainingData = new SQLTransformer()&#xA;    .setStatement(&quot;SELECT setLabel(1.0, array(*)) AS points FROM __THIS__&quot;)&#xA;    .transform(df)&#xA;val labeledRDD = labeledTrainingData.rdd.map(_(1).asInstanceOf[LabeledPoint])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4966" LastActivityDate="2016-10-10T12:57:37.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14443" PostTypeId="2" ParentId="14434" CreationDate="2016-10-10T13:28:32.473" Score="2" Body="&lt;p&gt;You can try to build some kind of &quot;sliding window table&quot;. Let's say you have following attributes: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;call duration (x1)&lt;/li&gt;&#xA;&lt;li&gt;time of call (x2)&lt;/li&gt;&#xA;&lt;li&gt;picked the phone (x3)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let's further assume that you have data from past 3 weeks, which allows us to set following table. The rows contain individual calls, the columns the attributes. The appendix _1 tells us the time. So for example x1_1 is call duration previous week, x2_2 is time of call two weeks before etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;client | x1_1 | x2_1 | x3_1 | x1_2 | x2_2 | x3_2 | ... | x3_3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can train your model using historical data, where x3_3 is last week. Then, you will feed the model with current data (_3 is current week*) and try to predict x3_3 - whether the customer will pick the phone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*I am assuming that you know who you are going to call, hence you have _3 attributes, but you don' know yet whether they respond or not&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aim is to give the model the opportunity to learn the time dependencies - maybe time of call week before together with call duration strongly correlate with future chance of picking up the phone again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can also help is perform feature selection. The assumption is that some attributes are strongly correlated with others, whereas others are not. You can simply use x1_1, x2_1 and see the relationship to x3_1. But I'd suggest recalculate these often as the preferences might change in time.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-10-10T13:28:32.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14445" PostTypeId="2" ParentId="14433" CreationDate="2016-10-10T20:10:09.540" Score="4" Body="&lt;p&gt;Decision trees are generally prone to over-fitting and accuracy doesn't generalize well to unseen data. One advantage of information gain is that -- due to the factor $-p*log(p)$ in the entropy definition -- leafs with a small number of instances are assigned less weight ($lim_{p \rightarrow 0^{+} } p*log(p) = 0$) and it favors dividing data into bigger but homogeneous groups. This approach is usually more stable and also chooses the most impactful features close to the root of the tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Accuracy is usually problematic with unbalanced data. Consider this toy example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Weather Wind    Outcome&#xA;Sunny   Weak    YES&#xA;Sunny   Weak    YES&#xA;Rainy   Weak    YES&#xA;Cloudy  Medium  YES&#xA;Rainy   Medium  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;Rainy   Strong  NO&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Weather and Wind both produce only one incorrect label hence have the same accuracy of 16/17. However, given this data, we would assume that weak winds (75% YES) are more predictive for a positive outcome than sunny weather (50% YES). That is, wind teaches us more about both outcomes. Since there are only few data points for positive outcomes we favor wind over weather, because wind is more predictive on the smaller label set which we would hope to give us a rule that is more robust to new data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The entropy of the outcome is  $ -4/17*log_2(4/17)-14/17*log_2(14/17)) =0.72$. The entropy for weather and outcome is $14/17*(-1/14*log_2(1/14)-13/14*log_2(13/14)) = 0.31$ which leads to an information gain of $0.41$. Similarly, wind gives a higher information gain of $0.6$.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2016-10-11T17:59:46.160" LastActivityDate="2016-10-11T17:59:46.160" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14446" PostTypeId="1" CreationDate="2016-10-10T21:30:06.607" Score="3" ViewCount="269" Body="&lt;p&gt;I'm just getting started with DSX. I have two files in my Object Storage that I want to read in my Jupyter notebook: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Warehousing-data.json &lt;/li&gt;&#xA;&lt;li&gt;Warehousing-sales_data- nominal_scenario.json&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have the credentials. Now I need the calling sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what have on my local copy of the notebook.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;resource_path= &quot;C:\\...\\warehousing\\&quot;&#xA;Warehousing_data_json = &quot;Warehousing-data.json&quot;&#xA;Warehousing_sales_data_nominal_scenario_json = &quot;Warehousing-sales_data- nominal_scenario.json&quot;&#xA;&#xA;resp = client.execute(input= [&#xA;    {'name': Warehousing_data_json, &#xA;     'filename': resource_path + Warehousing_data_json},&#xA;    {'name': Warehousing_sales_data_nominal_scenario_json,&#xA;     'filename': resource_path + Warehousing_sales_data_nominal_scenario_json}])&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the equivalent when the files are located in Object Storage?&lt;/p&gt;&#xA;" OwnerUserId="25085" LastActivityDate="2019-12-30T14:00:24.813" Title="Reading file from Object Storage in IBM DSX" Tags="|jupyter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14450" PostTypeId="1" CreationDate="2016-10-11T11:47:35.340" Score="0" ViewCount="47" Body="&lt;p&gt;I have a collection of podcasts, in particular the &quot;Talkingmachines&quot; Podcasts (&quot;Human conversation about machine learning&quot;), see &lt;a href=&quot;http://www.thetalkingmachines.com/blog/&quot; rel=&quot;nofollow&quot;&gt;http://www.thetalkingmachines.com/blog/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 22 episodes, each 1 hour long. MP3 files are available on my local disk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now  when I want to search the audio files for the episode where they talk about, say, the kmeans++ algorithm , what should I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I want to search for that episode where a male voice (Ryan) talks about kmeans++ in the first half of the podcast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, what should I do? Is there a desktop tool available that can do this search? Should I upload the files to some webservice that can do audio recognition?&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-10-11T20:47:09.083" Title="How to search collection of podcasts (.mp3 files)?" Tags="|search|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14451" PostTypeId="1" CreationDate="2016-10-11T12:00:33.760" Score="0" ViewCount="147" Body="&lt;p&gt;So I want to use fminunc(fun, X0) in Matlab on my function fun. The problem is that fun is a function of three parameters:(X, y, X0), all of which are vectors. What I want to do is to find the minimum, given the parameters X, y. I.e. the vector X0 which minimises the function for some given values X, y. How would I go about doing this?&lt;/p&gt;&#xA;" OwnerUserId="23511" LastActivityDate="2016-10-24T11:22:49.877" Title="How to use fminunc in Matlab on only some specified parameters of a function" Tags="|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14452" PostTypeId="2" ParentId="594" CreationDate="2016-10-11T12:03:51.227" Score="0" Body="&lt;p&gt;I am not working at Google,  but According to this blog post &lt;a href=&quot;https://research.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html&quot; rel=&quot;nofollow&quot;&gt;&quot;Graph-powered Machine Learning at Google&quot;&lt;/a&gt; from Oct 2016, they use an in-house software called &quot;Google Expander&quot;, and &quot;in conjuction with neural networks&quot; it performs semisupervised learning based on graph-theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blog post says, this &lt;a href=&quot;https://arxiv.orghttps://arxiv.org/abs/1512.01752&quot; rel=&quot;nofollow&quot;&gt;arXiv paper, Large Scale Distributed Semi-Supervised Learning Using Streaming Approximation&lt;/a&gt; from 2015 gives a longer technical answer to the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1606.04870&quot; rel=&quot;nofollow&quot;&gt;2016 arXiv paper, on smart gmail response&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-10-11T12:03:51.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14453" PostTypeId="2" ParentId="14451" CreationDate="2016-10-11T12:57:08.180" Score="1" Body="&lt;p&gt;What we have here is a function fun(X, y, X0) and fminunc(fun,X0). You know X, and y, lets call them X* and y*, so we could replace fun(...) by fun*(X0)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% function to be minized&#xA;fun = @(X, y, X0) (dot(X,y) + X0)^2   &#xA;X_star = [1,2,3]&#xA;y_star = [4,5,6]&#xA;$ same function but with X and y filled in&#xA;fun_star = @(X0) fun(X_star, y_star, X0) &#xA;X0_star = 0&#xA;fminunc(fun_star, X0) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25104" LastEditorUserId="25104" LastEditDate="2016-10-24T11:22:49.877" LastActivityDate="2016-10-24T11:22:49.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14454" PostTypeId="2" ParentId="12790" CreationDate="2016-10-11T12:59:11.087" Score="1" Body="&lt;p&gt;I don't think you'll find anything that checks all of your requirements, but here are some things to look at:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automated ETL mapping: There is a tool called &lt;a href=&quot;https://github.com/usc-isi-i2/Web-Karma&quot; rel=&quot;nofollow&quot;&gt;Karma&lt;/a&gt; started by a team at USC's Information Sciences Institute. It learns from your ETL mappings and helps automate future mappings. It's the only open source tool I'm aware of that helps automate the ETL process, but I would be very interested if there are others out there.&lt;/li&gt;&#xA;&lt;li&gt;Large scale ETL: There are many many tools you could look at for the scalability you are looking for. I can personally recommend looking at &lt;a href=&quot;http://storm.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Storm&lt;/a&gt; and &lt;a href=&quot;http://spark.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Spark&lt;/a&gt;. Storm is excellent for stringing a connection of processing steps together that, given enough resources, can compute in near real time on streaming data. Not too dissimilar Spark has a &lt;a href=&quot;http://spark.apache.org/streaming/&quot; rel=&quot;nofollow&quot;&gt;streaming component&lt;/a&gt; with a similar use case, but standard Spark may fit your needs better if the data you are needing to ETL is a fixed set to be processed once.&lt;/li&gt;&#xA;&lt;li&gt;Data storage: You may also need to consider where all this data will live during the ETL lifetime. You may need something like &lt;a href=&quot;https://kafka.apache.org/&quot; rel=&quot;nofollow&quot;&gt;Kafka&lt;/a&gt; to deal with large streams of data. Or maybe &lt;a href=&quot;http://hadoop.apache.org/&quot; rel=&quot;nofollow&quot;&gt;HDFS&lt;/a&gt; to store a static collection of files.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2016-10-11T12:59:11.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14455" PostTypeId="1" CreationDate="2016-10-11T13:03:49.220" Score="2" ViewCount="693" Body="&lt;p&gt;Basically I'm developing a recommendation system using a graph database (specifically neo4j), and I want to apply recommendation algorithms. Since i'm using a graph database, I can see the recommendation problem as a graph problem, and intuitively i can use graph based algorithms for the recommendation system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my research, recommendation systems are a subclass of information filtering system that seek to predict the &quot;rating&quot; or &quot;preference&quot; that a user would give to an item. And there exists basically two types, collaborative filtering and content based. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done a research on the algoritms, and i found some interesting ones:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Weighted Bipartite Graph algorithm&lt;/li&gt;&#xA;&lt;li&gt;Energy Spread Activation&lt;/li&gt;&#xA;&lt;li&gt;Union Colors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is simple, which other graph algorithms exists that can be used for graph based recommendation system? Or if I use a graph database for recommendation system, the algorithm doesn't necessary need to be a graph based?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. Any suggestions are welcomed.&lt;/p&gt;&#xA;" OwnerUserId="17596" LastActivityDate="2016-11-10T13:38:51.780" Title="Which algorithms should I use for recommendation system using a graph database?" Tags="|predictive-modeling|algorithms|graphs|recommender-system|neo4j|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14456" PostTypeId="2" ParentId="14455" CreationDate="2016-10-11T13:18:54.087" Score="2" Body="&lt;p&gt;A biadjacency matrix of a bipartite graph admits &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_decomposition&quot; rel=&quot;nofollow&quot;&gt;matrix factorisation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For $m$ items and $n$ users, the biadjacency matrix is an $m \times n$ matrix which can be factorised into two lower-rank factor matrices of sizes $m \times k$ and $k \times n$ respectively. This provides a lower-dimensional representation of how users' preferences vary over products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model also has the property that the inner product of the $i$th row and $j$th column of the two factor matrices provide an approximation of user $j$'s rating of item $i$. In other words, the product of the two matrices is an approximation of the original ratings matrix.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-11T13:18:54.087" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14457" PostTypeId="1" AcceptedAnswerId="14459" CreationDate="2016-10-11T14:21:45.293" Score="1" ViewCount="260" Body="&lt;p&gt;I'm currently working on a project involving the prediction of tenancy lengths. I've so far managed to get to a point where I've processed the data and pruned my Random Forest model (via sklearn in Python) to the following accuracy levels (in days):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train MAE: 131&#xA;Train R^2: 0.906&#xA;Test MAE: 259 (using cross-validation)&#xA;Test R^2: 0.651&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pS7Mk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pS7Mk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the model is decent for the industry, there's more performance to squeeze out of it. It currently overestimates results and has poor accuracy on the test data imo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to further develop a Neural Network approach, as my initial implementation of an MLP Regressor seems promising:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train MAE: 301&#xA;Train R^2: 0.582&#xA;Test MAE: 338 (using cross-validation)&#xA;Test R^2: 0.522&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R8fn3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R8fn3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I improve my results for the prediction (using Python) other than using GridSearch to play around with the MLPRegression function in sklearn? Are there any other models that could be useful in this situation? (I have tried also decision trees, gradient boosting)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it is relevant, my dataset contains ~5000 entries since 2008 onwards of individual tenancies, containing: tenancy dates, rent, repair costs, property information and replacements, client information, etc., currently at 41 variables.&lt;/p&gt;&#xA;" OwnerUserId="24830" LastActivityDate="2016-10-11T16:26:35.483" Title="Regression in Predicting Tenancy Lengths" Tags="|python|neural-network|predictive-modeling|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14458" PostTypeId="1" AcceptedAnswerId="14525" CreationDate="2016-10-11T15:02:43.010" Score="0" ViewCount="409" Body="&lt;p&gt;I have this dataset (just a sample):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;product1,product2,product3&#xA;product1,product4&#xA;product1,product2&#xA;product4,product3,product1,product2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The products are grouped by transaction. I want to create some data visualization using this dataset but I don't any tool or any type of visualization that allows creating some visualization with this structure...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone can suggest me an option?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel desperate because they can not find anything that suits this data structure&lt;/p&gt;&#xA;" OwnerUserId="23811" LastEditorUserId="471" LastEditDate="2016-10-11T19:51:36.587" LastActivityDate="2016-10-15T14:17:17.447" Title="Create data visualization for unstructured data - Basket Market Analysis" Tags="|visualization|association-rules|market-basket-analysis|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14459" PostTypeId="2" ParentId="14457" CreationDate="2016-10-11T16:26:35.483" Score="1" Body="&lt;p&gt;You may have done this already, but if your target values are positive integers, it could be worth transforming your output layer in such a way that constrains it to take an appropriate range of values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tenancy length values presumably have some recognisable distribution that might fruitfully be modelled by a known statistical process (Poisson process, survival analysis, etc.), and so instead of using your NN to predict the tenancy length directly, you could use the NN to parametrise a relevant probability distribution, and make a point estimate of that distribution in your output layer.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-11T16:26:35.483" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14461" PostTypeId="2" ParentId="14446" CreationDate="2016-10-11T17:49:20.190" Score="1" Body="&lt;p&gt;Found a sample &quot;Load data from different sources&quot; at &lt;a href=&quot;https://apsportal.ibm.com/exchange/public/entry/view/07db16e78e1722931e27c074ebe343ae&quot; rel=&quot;nofollow&quot;&gt;https://apsportal.ibm.com/exchange/public/entry/view/07db16e78e1722931e27c074ebe343ae&lt;/a&gt;&#xA;It works.&lt;/p&gt;&#xA;" OwnerUserId="25085" LastActivityDate="2016-10-11T17:49:20.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14462" PostTypeId="2" ParentId="14428" CreationDate="2016-10-11T18:02:02.343" Score="7" Body="&lt;p&gt;Check out &lt;a href=&quot;https://www.stat.washington.edu/raftery/Research/PDF/fraley1998.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;.  It also addresses question of how many clusters to use.  The R package &lt;a href=&quot;https://cran.r-project.org/web/packages/mclust/index.html&quot; rel=&quot;noreferrer&quot;&gt;mclust&lt;/a&gt; has a routine which will try different cluster models/number of clusters and plot the Bayesian inference criterion (BIC). (great vignette &lt;a href=&quot;https://cran.r-project.org/web/packages/mclust/vignettes/mclust.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;).  It's a general method, meaning, something you can do without being domain/data specific.  (It's always good to be domain-specific if you have the time and data.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chart is from the &lt;a href=&quot;https://cran.r-project.org/web/packages/mclust/vignettes/mclust.html&quot; rel=&quot;noreferrer&quot;&gt;vignette&lt;/a&gt; by Lucca Scrucca.  MClust tries 14 different clustering algorithms (represented by the different symbols), increasing the number of clusters from 1 to some default value.  It's finds the BIC each time.  Highest BIC is usually the best choice.  You could apply this methodology to your own stable of clustering algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CDQ4L.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CDQ4L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2016-10-11T18:15:48.650" LastActivityDate="2016-10-11T18:15:48.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14463" PostTypeId="2" ParentId="13754" CreationDate="2016-10-11T20:26:41.373" Score="1" Body="&lt;p&gt;Try &lt;code&gt;clf = RandomForestClassifier(max_features=None)&lt;/code&gt;. The &lt;code&gt;max_features&lt;/code&gt; param defaults to &lt;code&gt;'auto'&lt;/code&gt; which is equivalent to &lt;code&gt;sqrt(n_features)&lt;/code&gt;. &lt;code&gt;max_features&lt;/code&gt; is described as &quot;The number of features to consider when looking for the best split.&quot; Only looking at a small number of features at any point in the decision tree means the importance of a single feature may vary widely across many tree. So, don't look at a random subset, just look at all features at every level of the tree.&lt;/p&gt;&#xA;" OwnerUserId="25115" LastActivityDate="2016-10-11T20:26:41.373" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14464" PostTypeId="2" ParentId="14450" CreationDate="2016-10-11T20:47:09.083" Score="1" Body="&lt;p&gt;Transcribe then topic model the recordings. This will let you know which podcast talks about the subject of interest, then you can search the transcript. If you really need it to pinpoint the moment when the subject was discussed, you'll need a temporal topic model &lt;a href=&quot;http://www.idiap.ch/resource/research/probamod-v1/&quot; rel=&quot;nofollow&quot;&gt;like so&lt;/a&gt; but that would add a good bit of complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google has a &lt;a href=&quot;https://cloud.google.com/speech/&quot; rel=&quot;nofollow&quot;&gt;good service&lt;/a&gt; for the first part. You can use an open source library like gensim for the second part.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-10-11T20:47:09.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14465" PostTypeId="1" CreationDate="2016-10-12T02:02:39.550" Score="1" ViewCount="822" Body="&lt;p&gt;When we calculate week over week change for a metric like &quot;Monthly active users&quot;, we often refer to a calculation of SUM(This week) - SUM(Last week)/SUM(Last week), which means we will run total of this week's number first, and then compare to the total from last week. &#xA;There is another situation when we can have change any date comparing 7 days ago(week), So each date during a week will have a change value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any best practice for this &quot;week over week&quot; concept. I've also heard that someone is using &quot;weekly change&quot; referring to the similar thing. Is there any difference? Which calculation I should chose?&lt;/p&gt;&#xA;" OwnerUserId="25129" LastEditorUserId="8501" LastEditDate="2016-12-13T09:19:00.383" LastActivityDate="2016-12-13T09:19:00.383" Title="Best practice for computing week-over-week change in MAU metric" Tags="|bigdata|visualization|tableau|excel|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14466" PostTypeId="1" CreationDate="2016-10-12T06:52:35.030" Score="-1" ViewCount="307" Body="&lt;p&gt;I am using nnetar package for time series forecasting in R.But each time when I run the model, I am getting different output. i guess this is related to different random weight initialization at each time, but I don't know how to set it as constant.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ts=nnetar(Time_Series,P=3,lambda = 0,decay=0.001,maxit=150)&#xA;fs = forecast(ts, h=12)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any pointers would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="25136" LastActivityDate="2020-11-27T23:47:45.663" Title="nnetar giving different output each time" Tags="|r|neural-network|time-series|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14467" PostTypeId="1" AcceptedAnswerId="14468" CreationDate="2016-10-12T07:25:55.760" Score="5" ViewCount="9192" Body="&lt;p&gt;In this &lt;a href=&quot;https://github.com/jocicmarko/ultrasound-nerve-segmentation/blob/master/train.py&quot; rel=&quot;nofollow&quot;&gt;code &lt;/a&gt;, line 13 is commented as Theano dimension ordering mode. What does this mean? &lt;/p&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="39153" LastEditDate="2017-09-12T23:48:48.227" LastActivityDate="2017-09-12T23:48:48.227" Title="What does Theano dimension ordering mean?" Tags="|python|keras|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14468" PostTypeId="2" ParentId="14467" CreationDate="2016-10-12T07:40:47.483" Score="11" Body="&lt;p&gt;Let's say you're working with 128x128 pixel RGB images (that's 128x128 pixels with 3 color channels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you put such an image into a numpy array you can either store it with a shape of (128, 128, 3) or with a shape of (3, 128, 128).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dimension ordering specifies if the color channel comes first (as with theano / &quot;th&quot;) or if it comes last (as with tensorflow / &quot;tf&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code you've posted contains the following line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = Input((1, img_rows, img_cols))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It specifies the shape of the input as (1, img_rows, img_cols) - i.e. there's one color channel (gray scale) and it comes first. That's why it requires Theano's dimension ordering &quot;th&quot;.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-10-12T07:40:47.483" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14470" PostTypeId="2" ParentId="13912" CreationDate="2016-10-12T10:07:21.580" Score="3" Body="&lt;p&gt;Since no one has mentioned RAM-efficient methods yet:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of loading everything into RAM, you can use online/out-of-core learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python's &lt;a href=&quot;http://scikit-learn.org&quot; rel=&quot;nofollow&quot;&gt;Scikit-Learn&lt;/a&gt; for example has the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;nofollow&quot;&gt;SGDClassifier&lt;/a&gt; class. Set its loss function to &quot;log&quot; and you get logistic regression. Using the partial_fit function you can feed it small batches of data that you read straight from the database (or from some CSV file,...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot; rel=&quot;nofollow&quot;&gt;Vowpal Wabbit&lt;/a&gt; might also be worth a try. Its made for out-of-core learning - hardly uses any RAM and you won't find anything that's much faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use Python's &lt;a href=&quot;http://keras.io/&quot; rel=&quot;nofollow&quot;&gt;Keras&lt;/a&gt; library to build a neural network (or in the simplest case just logistic regression), which you can also feed with small batches of data instead of loading everything into RAM. Compared to the other two recommendations a neural network could also learn non-linear dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides that, try to start with fewer samples - plot the learning curves with 10k, 100k, 1M samples and see if 100M samples are even necessary to get a good score.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-10-12T10:07:21.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14471" PostTypeId="2" ParentId="12172" CreationDate="2016-10-12T10:28:08.073" Score="0" Body="&lt;p&gt;Consider how &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity#Definition&quot; rel=&quot;nofollow&quot;&gt;cosine similarity is calculated&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity takes the dot product of two real vectors, and divides this value by the product of their magnitudes. By the Euclidean dot product identity, this is equal to the cosine of the angle between the two vectors. The upshot of this is a value between 1 and -1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the value is 1, those vectors are pointing in exactly the same direction. When the value is -1, the vectors are pointing in exactly the opposite direction (one is the negation of the other). When the value is 0, the vectors are perpendicular to one another; in other words, when the value is zero, these two vectors are as unalike in the feature space as it is possible to get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dot product is the sum of all the element-wise products of your two vectors. The bigger those numbers, the more they contribute to the cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, take any feature in your vector. The fifth, say. If you set this to zero in one of your vectors, the fifth element in the element-wise product of the two vectors will also be zero, regardless of its value in the other vector. When you sum up all these element-wise products, the fifth element will not have any impact on the summation. As a result, setting a value in your feature vector to zero means it doesn't make any contribution to the cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why setting a value to zero in a feature vector is equivalent to not including the feature in the calculation of cosine similarity, and does not does not distort cosine similarity.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastEditorUserId="24895" LastEditDate="2016-10-12T10:35:07.563" LastActivityDate="2016-10-12T10:35:07.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14474" PostTypeId="1" CreationDate="2016-10-12T14:07:28.050" Score="1" ViewCount="678" Body="&lt;p&gt;I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apply the &quot;usual&quot; preparation for modeling (dummy variables, normalization, PCA etc., of course in the necessary cases) to the &lt;em&gt;training data&lt;/em&gt;. So far, so good. But when I get the to-be-classified &lt;em&gt;new data&lt;/em&gt; to make prediction the model constructed above, it's evident that I must apply these preparatory steps to this new data set as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the problem arises here, because if I simply apply the preparatory steps for my new data in turn again, these doesn't take into consideration the &lt;em&gt;characteristics of the training data&lt;/em&gt;. So, if I convert the new data factors into dummies, then it takes only the existing factor levels in the new data into account; if I min-max normalize the new data, it will be normalized according its own min-max values, disregarding the values in the training data; and if I use PCA, then the resulting components from the new data will be totally independent of the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So essentially my point is that applying the same conversions separately to the train set and the new data set(s) (which could be only one observation as well), then the two resulting transformed sets will have nothing in common, so the prediction will be baseless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found some traces that in some cases there is some &quot;learning&quot; step in the training phase in these transformations as well and apply this &quot;knowledge&quot; to new data (caret and Sklearn, for instance, with &quot;predict&quot; could transform to the new data with characteristics learned from the training data), but generally speaking this inconsistency remains unmentioned otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the correct practice here?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-10-01T22:59:33.880" Title="Preprocessing to-be-predicted data in ML with R - &quot;learn&quot; and &quot;apply&quot; features" Tags="|machine-learning|r|preprocessing|prediction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14476" PostTypeId="2" ParentId="14474" CreationDate="2016-10-12T14:35:06.557" Score="2" Body="&lt;p&gt;In short, the correct practice is to use the scaling/preprocessing parameters you found and used on your training set, and apply them to your test set.  the motivation for this is that you should be able to deal with someone coming to you with one new observation they want a prediction on.  You will be able to preprocess that single observation using your training set preprocessing parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ill go through some of the issues you brought up:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dummy Variables:&lt;/strong&gt;&#xA;This might entail you using &lt;code&gt;setdiff()&lt;/code&gt; in R to find what columns are in your training set but not in your test set, then make a column in your test set with a column of zeros.  This is the simplest way to do it, but it is very data/purpose dependent.  You may also want to change this or look for better options depending on what modeling technique you're using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Min/Max Normalize:&lt;/strong&gt;&#xA;When you scale your training set, save the min and max of each column, and scale your test set using those parameters (NOT the min/max of your test set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PCA:&lt;/strong&gt;&#xA;When you perform PCA on your training set, you are transform your data based on rotations that were calculated on your training set.  You want to perform those pre-calculated rotations on your test set.  Check out &lt;code&gt;?predict.prcomp&lt;/code&gt;.  This will show you how to apply your rotations from your training set onto your test set.&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2016-10-12T14:35:06.557" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14477" PostTypeId="1" CreationDate="2016-10-12T14:58:57.470" Score="2" ViewCount="3679" Body="&lt;p&gt;I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters of the different clustering algorithms is it possible to run some type of grid search on these algorithms in order to find the best set that optimized Silhouette score. I found this PR on Github &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/pull/6160&quot; rel=&quot;nofollow&quot;&gt;https://github.com/scikit-learn/scikit-learn/pull/6160&lt;/a&gt; but it looks like work on it died off.&lt;/p&gt;&#xA;" OwnerUserId="25153" LastActivityDate="2016-10-12T14:58:57.470" Title="Grid Search on Unsupervised Sklearn Clustering?" Tags="|clustering|scikit-learn|unsupervised-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14478" PostTypeId="2" ParentId="14421" CreationDate="2016-10-12T15:00:41.450" Score="0" Body="&lt;p&gt;You can use restricted boltman machine to obtain feature map, and then build a CNN with this architecture : conv layer -&gt; pooling-&gt;conv layer pooling-&gt; ....-&gt;fully connected layer.   Add conv layer and pooling layer until the matrix size reduced into $1\times 1$, then treat them as input for your fully connected layer, then train the fully connected layer &lt;/p&gt;&#xA;" OwnerUserId="24696" LastActivityDate="2016-10-12T15:00:41.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14479" PostTypeId="1" CreationDate="2016-10-12T15:12:02.777" Score="3" ViewCount="661" Body="&lt;p&gt;Just landed on a Entity Resolution (Matching) project.&#xA;I am working in python and I was wondering if there's an off-the-shelf implementation of string harmonization functions, including:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;removal of trailing spaces&lt;/li&gt;&#xA;&lt;li&gt;removal of multiple spaces&lt;/li&gt;&#xA;&lt;li&gt;removal of special characters (eg. ü)&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All this is easily done with string and Regex operations but I do not want to reinvent the wheel and tap into something already ready (and perhaps learn from that, shall I need to move on to something custom later).&#xA;Are there readily available libs for that? Tried &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow&quot;&gt;nltk&lt;/a&gt;, which suggests the use of regex expression, &lt;a href=&quot;https://textblob.readthedocs.io/en/dev/&quot; rel=&quot;nofollow&quot;&gt;textblob&lt;/a&gt;, and googling around but found nothing really helpful so far - probabily because I did not get to use the right keyword.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, it would be useful to have a function that replaces special characters from foreign languages with morphological equivalents - for example &quot;ß&quot; in German with &quot;ss&quot; (&quot;straße&quot; -&gt; &quot;strasse&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: clarification&lt;/strong&gt;&#xA;In my question, I am not really specifying the problem at hand. This is because what I am trying to get to is really a &quot;grammar&quot; that would let me and the other ppl in my team define workflows of harmonization/canonicalisation operations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am at the moment Working on this as a chain of functions f: string -&gt; string. Whether some steps are part of it, it will depend on the problem at hand. Workflow will be likely tested empirically (we'll pick the workflow that leads to the best performing algo)&lt;/p&gt;&#xA;" OwnerUserId="1244" LastEditorUserId="1244" LastEditDate="2016-10-13T13:21:05.380" LastActivityDate="2016-10-13T21:48:02.053" Title="String harmonization in Python" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14480" PostTypeId="1" AcceptedAnswerId="14481" CreationDate="2016-10-12T15:29:59.463" Score="0" ViewCount="361" Body="&lt;p&gt;From &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#sphx-glr-auto-examples-manifold-plot-compare-methods-py&quot; rel=&quot;nofollow&quot;&gt;sklearn docs&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that the purpose of the MDS is to find a low-dimensional representation of the data (here 2D) in which the distances respect well the distances in the original high-dimensional space, unlike other manifold-learning algorithms, it does not seeks an isotropic representation of the data in the low-dimensional space.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone elaborate, in layman's terms, what the distinction is?&lt;/p&gt;&#xA;" OwnerUserId="23480" LastActivityDate="2016-10-12T16:27:38.360" Title="Difference between MDS and other manifold learning algorithms" Tags="|dimensionality-reduction|manifold|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14481" PostTypeId="2" ParentId="14480" CreationDate="2016-10-12T16:27:38.360" Score="0" Body="&lt;p&gt;The images in the link you provide, of the severed sphere and its lower-dimensional representations, go some way towards explaining the difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The severed sphere is a set of points in a three-dimensional space, but we want a two-dimensional representation of it. The objective of manifold-learning is (shockingly) to find a manifold: a subset of that three-dimensional space which (a) closely fits all the points that make up the severed sphere, and (b) can be described with a two-dimensional coordinate system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at some of the other lower-dimensional representations of the severed sphere, it's like they're taking it and flattening it out into a rectangle so it'll fit in two dimensions. It's taking the severed sphere and figuring out a new coordinate system that maps as closely as possible onto all the points that make up the severed sphere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MDS lower-dimensional representation, though, is more like a shadow that the severed sphere casts on a wall. Rather than finding a new coordinate system that closely fits the sphere, it's just &quot;forgetting&quot; whichever of the dimensions it thinks it can most afford to lose while maintaining the same distance to and from all the points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good analogy would be maps of the earth. A good map of the earth makes a new coordinate system that fits a sphere onto a 2D surface. To do this it has to distort the relative distances between places, but you end up with effective 2D coordinates that relate well to places on the globe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of doing this, you could just take two photos of the earth from above the north and south pole and glue them back to back. You'd still have a 2D representation of the earth, but it doesn't work so well as a coordinate system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't to say that MDS is &quot;bad&quot;. It's just doing something different. You probably wouldn't use MDS for dimensionality reduction prior to carrying out some sort of statistical procedure, but if you're trying to produce a graphic that gives some idea of how close multidimensional points are to one another, it might be a good choice.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-12T16:27:38.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14482" PostTypeId="1" AcceptedAnswerId="14511" CreationDate="2016-10-12T17:43:55.443" Score="3" ViewCount="434" Body="&lt;p&gt;Is there a standard term for data that are not missing? I.e. is it called non-missing, present, or something else?&lt;/p&gt;&#xA;" OwnerUserId="24763" LastEditorUserId="83275" LastEditDate="2021-03-12T05:46:56.923" LastActivityDate="2021-03-12T05:46:56.923" Title="Data that's not missing is called...?" Tags="|data-cleaning|terminology|missing-data|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="14484" PostTypeId="1" CreationDate="2016-10-12T20:06:52.870" Score="-1" ViewCount="4562" Body="&lt;p&gt;I am completely new to data science and this is a homework assignment so apologies beforehand.  I have some raw data on restaurants that contain their categories (e.g. &quot;Pizza&quot;, &quot;Italian&quot;) and their coordinates. I'm to cluster them based on their coordinates for closeness and their categories for similarity using K-means. So far I've decided to vectorize the data from dictionary format into a NumPy/SciPy representation used by scikit-learn estimators. Something of this format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['category=Pizza', 'category=Bars', 'cateogry=Italian', 'latitude', 'longitude']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I'm having trouble scaling the vectors, since spatial coordinates and restaurant categories have different units of scale and there may be 20 different categories for a restaurant and only two features for latitude and longitude.  I've attempted to use the skilearn preprocessing library but it is not providing meaningful clusters.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="25161" LastActivityDate="2016-10-13T20:33:24.417" Title="clustering on multiple features and applying k-means" Tags="|python|clustering|scikit-learn|k-means|feature-scaling|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14485" PostTypeId="2" ParentId="11356" CreationDate="2016-10-12T20:42:56.857" Score="4" Body="&lt;p&gt;How about using recursion?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def union_all(dfs):&#xA;    if len(dfs) &amp;gt; 1:&#xA;        return dfs[0].unionAll(union_all(dfs[1:]))&#xA;    else:&#xA;        return dfs[0]&#xA;&#xA;td = union_all([td1, td2, td3, td4, td5, td6, td7, td8, td9, td10])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25162" LastActivityDate="2016-10-12T20:42:56.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14487" PostTypeId="1" CreationDate="2016-10-12T20:59:42.260" Score="1" ViewCount="706" Body="&lt;p&gt;I have a dataset that comprises of sentences and corresponding multi-labels (e.g. a sentence can belong to multiple labels). Using a combination of Convolutional Neural Networks and Recurrent Neural Nets on language models (Word2Vec) I'm able to achieve a good accuracy. However, it's /too/ good at modelling the output, in the sense that a lot of labels are arguably wrong and thus the output too. This means that the evaluation (even with regularization and dropout) gives a wrong impression, since I have no ground truth. Cleaning up the labels would be prohibitively expensive. So I'm left to explore &quot;denoising&quot; the labels somehow. I've looked at things like &lt;a href=&quot;http://www.ee.cuhk.edu.hk/~xgwang/papers/xiaoXYHWcvpr15.pdf&quot; rel=&quot;nofollow&quot;&gt;&quot;Learning from Massive Noisy Labeled Data for Image Classification&quot;&lt;/a&gt;, however they assume to learn some sort of noise covariace matrix on the outputs, which I'm not sure how to do in Keras. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone dealt with the problem of noisy labels in a mutli-label text classification setting before (ideally using Keras or similar) and has good ideas on how to learn a robust model with noisy labels? &lt;/p&gt;&#xA;" OwnerUserId="25163" LastActivityDate="2016-10-12T20:59:42.260" Title="Dealing with noisy training labels in text classification using deep learning" Tags="|deep-learning|supervised-learning|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14489" PostTypeId="1" CreationDate="2016-10-13T00:22:14.670" Score="1" ViewCount="2944" Body="&lt;p&gt;i have a question about the tutorial of tensorflow to train the mnist database&#xA;how do i create my own batch without using next_batch() , the idea is to train with a batch of 50 ,then 100 and so but it has to be in order&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is the code&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))&#xA;train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)&#xA;correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;sess.run(tf.initialize_all_variables())&#xA;for i in range(20000):&#xA;    batch = mnist.train.next_batch(50)#revisar aqui&#xA;    if i%100 == 0:&#xA;        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})&#xA;        print(&quot;step %d, training accuracy %g&quot;%(i, train_accuracy))&#xA;    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25167" LastEditorUserId="25284" LastEditDate="2016-10-16T21:32:45.453" LastActivityDate="2016-11-15T22:28:27.600" Title="tensorflow mnist tutorial" Tags="|machine-learning|python|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14490" PostTypeId="2" ParentId="13711" CreationDate="2016-10-13T06:10:17.320" Score="5" Body="&lt;p&gt;You may use &lt;a href=&quot;https://radimrehurek.com/gensim/models/phrases.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt; phrase vectorizer module available in Python.&lt;/p&gt;&#xA;&lt;p&gt;You need to give threshold value which is some sort of pmi of words. The higher this value less are the number of phrases the default is 10. You can play around with this value to get results for your data.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;phrase_threshold = 1&lt;/p&gt;&#xA;&lt;p&gt;bigram = Phrases(sentences,threshold=phrase_threshold)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This is based on this based on the skipgram &lt;a href=&quot;https://papers.nips.cc/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Tomas Mikolov.&lt;/p&gt;&#xA;" OwnerUserId="25174" LastEditorUserId="85045" LastEditDate="2020-12-23T17:29:18.173" LastActivityDate="2020-12-23T17:29:18.173" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14491" PostTypeId="2" ParentId="13372" CreationDate="2016-10-13T06:13:15.740" Score="0" Body="&lt;p&gt;Part 1: There are pretrained state of the art models trained on their corpus which could be used for &lt;a href=&quot;http://nlp.stanford.edu/sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sentiment analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Part 2: You may always train your data on their model rather than creating your own model.&lt;/p&gt;&#xA;" OwnerUserId="25174" LastEditorUserId="98307" LastEditDate="2020-08-04T08:39:42.040" LastActivityDate="2020-08-04T08:39:42.040" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14492" PostTypeId="2" ParentId="12872" CreationDate="2016-10-13T06:27:23.743" Score="19" Body="&lt;p&gt;Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg&#xA;The weather in California was _____ .&#xA;The blank could be filled by both hot and cold hence the similarity would be higher. This concept is called Paradigmatic relations.&lt;/p&gt;&#xA;&lt;p&gt;If you are interested to capture relations such as hypernyms, hyponyms, synonyms, antonym you would have to use any wordnet based similarity measure. There are many similarity measures based on wordnet. You may check &lt;a href=&quot;http://ws4jdemo.appspot.com/&quot; rel=&quot;noreferrer&quot;&gt;this link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25174" LastEditorUserId="98307" LastEditDate="2020-08-06T13:36:56.023" LastActivityDate="2020-08-06T13:36:56.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14493" PostTypeId="1" AcceptedAnswerId="14606" CreationDate="2016-10-13T08:26:45.253" Score="2" ViewCount="5592" Body="&lt;p&gt;I'm sorry, if I ask my question by the wrong community. &#xA;What is a difference between &lt;a href=&quot;https://spark.apache.org/docs/2.0.0/ml-guide.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spark ML&lt;/a&gt; and &lt;a href=&quot;https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/ml/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flink ML&lt;/a&gt; and between Spark and Flink in general? The both projects are the projects of Apache, I would like to know why Foundation has two similar projects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I have found some interesting article &lt;a href=&quot;http://www.kdnuggets.com/2015/11/fast-big-data-apache-flink-spark-streaming.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fast Big Data: Apache Flink vs Apache Spark for Streaming Data&lt;/a&gt;&#xA;It has answers on my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PPS This question has been already discussed by Stack Overflow community. The topic is &lt;a href=&quot;https://stackoverflow.com/questions/28082581/what-is-the-difference-between-apache-spark-and-apache-flink&quot;&gt;What is the difference between Apache Spark and Apache Flink&lt;/a&gt;&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="3281" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-20T07:14:57.317" Title="Apache Spark ML vs Flink ML" Tags="|machine-learning|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14494" PostTypeId="1" AcceptedAnswerId="14508" CreationDate="2016-10-13T08:29:20.457" Score="2" ViewCount="122" Body="&lt;p&gt;I am new to ML and looking to learn with some project. I have a medical imaging dataset where an image (image is a time series of an object so multiple images) has been looked at by radiologist and they have graded it on a scale of 1-5 for some pathology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to basically use this to predict the pathology on new images. I am guessing there are multiple approaches one could take to do so. Could someone point me to some methods I could try (simple to more advanced) as I would like to also learn about them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another issue is that different images are of different sizes. Is this usually a problem for these approaches? I could try something where I can register them so that they are of the same size.&lt;/p&gt;&#xA;" OwnerUserId="8337" LastEditorUserId="14560" LastEditDate="2016-10-13T14:53:24.290" LastActivityDate="2016-10-13T16:24:26.237" Title="Supervised learning with tagged images" Tags="|machine-learning|classification|supervised-learning|computer-vision|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14495" PostTypeId="1" CreationDate="2016-10-13T09:00:25.460" Score="2" ViewCount="120" Body="&lt;p&gt;I have millions of lines of statements containing both subjective (like I prefer the red skirt) and objective (Washington was born on February 22, 1732) statements or opinions. How can I separate them? Not manually.&lt;/p&gt;&#xA;&lt;p&gt;By &amp;quot;objective&amp;quot;, I mean if the object or predicative of the sentence changed, it would be in conflict with the truth. For illustration, in &amp;quot;She prefer red skirt.&amp;quot;, if we change &amp;quot;red&amp;quot; or &amp;quot;skirt&amp;quot;, the (new) statement would remain right, which can not apply to statements like &amp;quot;Washington was born on February 22, 2016&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;The statements which should be pushed out:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Washington was born on February 22, 1732&lt;/li&gt;&#xA;&lt;li&gt;Washington was born on February 21, 1731&lt;/li&gt;&#xA;&lt;li&gt;Obama was born on February 22, 1732&lt;/li&gt;&#xA;&lt;li&gt;Nobody A was born on February 22, 2016&lt;/li&gt;&#xA;&lt;li&gt;Red is blue&lt;/li&gt;&#xA;&lt;li&gt;Color is a day&lt;/li&gt;&#xA;&lt;li&gt;I'm a robot&lt;/li&gt;&#xA;&lt;li&gt;You are a person&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Statements which should be kept:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I like coffee&lt;/li&gt;&#xA;&lt;li&gt;Red is my favorite color&lt;/li&gt;&#xA;&lt;li&gt;I hate him&lt;/li&gt;&#xA;&lt;li&gt;You are awesome&lt;/li&gt;&#xA;&lt;li&gt;You are clever&lt;/li&gt;&#xA;&lt;li&gt;I was born years ago&lt;/li&gt;&#xA;&lt;li&gt;You are very old&lt;/li&gt;&#xA;&lt;li&gt;Tomorrow will be better&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9191" LastEditorUserId="85045" LastEditDate="2020-12-12T04:51:53.580" LastActivityDate="2020-12-12T19:03:04.980" Title="Method to filter out objective statements (or say facts)" Tags="|classification|data-cleaning|" AnswerCount="2" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="14498" PostTypeId="2" ParentId="14422" CreationDate="2016-10-13T10:36:32.760" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For LSTM, the documents should be at word level. Hence, sentence vectors are not that useful for a document but word vectors are. You can use an embedding layer if you want do it though.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;in the 3D tensor, the first dimension is number of sentences. so 1000 is correct. The second one is the number of time_steps which is the number of words for each sentence. The third one is the word vector dimension. Hence, taking your numerical example, the input dimension of the LSTM will be (1000, 10, 100).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-13T10:36:32.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14500" PostTypeId="2" ParentId="13920" CreationDate="2016-10-13T11:47:04.153" Score="2" Body="&lt;p&gt;The score method used in keras &lt;strong&gt;&lt;code&gt;does not&lt;/code&gt;&lt;/strong&gt; calculate accuracy like the sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;accuracy_score&lt;/code&gt;&lt;/a&gt; method. Check the internal implementation of &lt;a href=&quot;https://keras.io/api/models/model_training_apis/#evaluate-method&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;evaluate&lt;/code&gt;&lt;/a&gt; method to understand more. If you want to calculate accuracy, I suggest you to use sklearn's &lt;code&gt;accuracy_score&lt;/code&gt; by getting the predictions or manually calculate if it is easier for you.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="24000" LastEditDate="2022-06-10T05:34:54.150" LastActivityDate="2022-06-10T05:34:54.150" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14502" PostTypeId="2" ParentId="14479" CreationDate="2016-10-13T13:07:09.777" Score="2" Body="&lt;p&gt;Note: this is more frequently called &lt;em&gt;canonicalisation&lt;/em&gt; than &lt;em&gt;harmonisation&lt;/em&gt;.  &lt;em&gt;URLify&lt;/em&gt; and &lt;em&gt;URL-safe&lt;/em&gt; are also terms that can help you in your search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;whitespace&lt;/strong&gt;&lt;br&gt;&#xA;Combining the built-in functions &lt;code&gt;string.split&lt;/code&gt; + &lt;code&gt;string.join&lt;/code&gt; is a solid way to canonicalise intermediate and trailing whitespace, including tabs, carriage returns and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;non-ASCII chars&lt;/strong&gt;&lt;br&gt;&#xA;ASCIIification is actually somewhat contrary to canonicalisation.  That is, you should think hard about whether you really want to be converting 'München' to 'muenchen', rather than 'muenchen' to proper 'München'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want to go in that direction, you should give more details on the use case and the desired conversions.  Here are some examples of possible input:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Where is Münsterstraße?&lt;br&gt;&#xA;  ısparta'daki istanbul'dakı ve gülenciler!&lt;br&gt;&#xA;  Isparta'daki İstanbul'dakı ve Gülenciler!&lt;br&gt;&#xA;  ¿Cómo está?&lt;br&gt;&#xA;  Это вообще фарс...&lt;br&gt;&#xA;  Биће белајааа, чорба&lt;br&gt;&#xA;  Biće belajaaa, čorba&lt;br&gt;&#xA;  You can now resume your résumé.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Converting to ASCII will be lossy and there are some decisions to make, often dependent on the source and target language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you define the likely input, scope and desired behaviour then others can make recommendations.&lt;/p&gt;&#xA;" OwnerUserId="12363" LastEditorUserId="12363" LastEditDate="2016-10-13T21:48:02.053" LastActivityDate="2016-10-13T21:48:02.053" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14503" PostTypeId="1" AcceptedAnswerId="14506" CreationDate="2016-10-13T13:56:25.537" Score="4" ViewCount="3734" Body="&lt;p&gt;I am currently playing with tensorflow, but can't seem to get a hold whether it usefull for my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to create a neural network, that is capable of mapping input to output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way things are progression now, i haven't a single example in which this has been done, all types of problem tensorflow seem to solve are classification task, and not this kind of mapping problem... Is tensorflow able to do so?, do I have to use a different neural network framework to do so?, And if it is available could somebody show some code (not the MNIST example)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently trying to make an neural network that takes in samples of audio files, and generates MFCC features from the samples. The MFCC features can be calculated manually, which I have done, so I know what the I output i seek is.  MFCC feature is a feature vector is different real values numbers.  I cannot be classified as class A or Class B, or doing so would greatly reduce the accuracy of the output, as you are fitting the output to predetermined &quot;bins&quot;...&lt;/p&gt;&#xA;" OwnerUserId="25188" LastEditorUserId="21024" LastEditDate="2016-10-14T12:28:13.823" LastActivityDate="2016-10-14T12:28:13.823" Title="Is it possible using tensorflow to create a neural network that maps a certain input to a certain output?" Tags="|classification|neural-network|tensorflow|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14504" PostTypeId="2" ParentId="14503" CreationDate="2016-10-13T14:15:20.163" Score="1" Body="&lt;p&gt;What you propose is certainly possible, you would just have &lt;strong&gt;n&lt;/strong&gt; nodes in your output layer, with &lt;strong&gt;n&lt;/strong&gt; being the size of your MFCC feature vector, and you need to define a loss function that determines what mistakes it is currently making. If all the features have a similar scale and similar importance you could use the Mean Squared Error but you can get creative here. In principle this is not very different from multiclass classification with a softmax layer at the end. In that case you have &lt;strong&gt;c&lt;/strong&gt; output nodes after a softmax layer and a cross-entropy loss function, in your case you have &lt;strong&gt;n&lt;/strong&gt; nodes in your output layer after for example a fully connected layer with a MSE loss function. Feed it a number of examples and it should do the task reasonable if the data is represented well. Although I'm uncertain why you would want to use a ML approach if this is calculatable, you will lose accuracy and I'm uncertain how much speed you win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very easy example where TensorFlow is used for a regression task and not a classification task is linear regression with gradient descent, which you can see here: &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb&quot; rel=&quot;nofollow&quot;&gt;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2016-10-13T14:46:25.827" LastActivityDate="2016-10-13T14:46:25.827" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14506" PostTypeId="2" ParentId="14503" CreationDate="2016-10-13T15:23:23.653" Score="3" Body="&lt;p&gt;From your description, it seems that you are facing a &lt;strong&gt;regression&lt;/strong&gt; problem, because you want your output to be certain values. This is different from &lt;strong&gt;classification&lt;/strong&gt; problems, that have as output the probability of the input belonging to certain class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to using neural networks for regression is that the output layer should have no activation, that is, it should be a linear layer. As pointed out by @JanvanderVegt, a common loss function for regression problems is the mean square error (MSE) between the current output and the MFCC features you computed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you google &quot;tensorflow regression example&quot; you can find dozens of complete examples, like &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt; or &lt;a href=&quot;https://github.com/nlintz/TensorFlow-Tutorials/blob/master/01_linear_regression.py&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-10-13T15:23:23.653" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14508" PostTypeId="2" ParentId="14494" CreationDate="2016-10-13T16:24:26.237" Score="2" Body="&lt;p&gt;I have worked on similar projects (using medical images such as PET to predict outcomes).  A method being used more and more for predicting cancer treatment outcomes is texture analysis: &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/21321270&quot; rel=&quot;nofollow&quot;&gt;https://www.ncbi.nlm.nih.gov/pubmed/21321270&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method of texture analysis uses wavelet transforms: &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3505569/&quot; rel=&quot;nofollow&quot;&gt;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3505569/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After deriving the texture features, you could then use those in a ML model for making predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A benefit to texture analysis is that it is mostly independent of the size of the target.  It is however, more dependent on resolution. If your images are produced on different scanners, you may need to resample in order to normalize the images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are fewer studies about deep learning for medical imaging, but it definitely has some exciting potential.  The question is whether or not a CNN can pick up subtle differences.  From what I've seen (I'll look for a reference), CNN hasn't outperformed any other method yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe add a few more details so we can get a better idea of what your objective is?&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-10-13T16:24:26.237" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14511" PostTypeId="2" ParentId="14482" CreationDate="2016-10-13T20:00:58.597" Score="4" Body="&lt;p&gt;Depends on content, but I would probably go for &quot;observed&quot; (vs. &quot;unobserved&quot;). A suitable direct antonym of &quot;missing&quot; might be &quot;extant&quot;.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-13T20:00:58.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14512" PostTypeId="2" ParentId="14495" CreationDate="2016-10-13T20:25:19.633" Score="1" Body="&lt;p&gt;There isn't really an &quot;objective&quot; statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example: what if he wasn't born that day? Same statement, but is it still objective?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may assume that &quot;1 + 1 = 2&quot; is an objective statement. But what if I'm doing binary math, and 1+1=0 then? So even that is subjective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would a machine tell apart these things, where philosophers will disagree with each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All you can do is provide training data examples of &quot;your&quot; (&lt;em&gt;subjective&lt;/em&gt;) idea of objectiveness.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-10-13T20:25:19.633" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14513" PostTypeId="2" ParentId="14484" CreationDate="2016-10-13T20:33:24.417" Score="1" Body="&lt;p&gt;k-means - as the name &lt;em&gt;means&lt;/em&gt; suggeyts - needs to be able to compile meaningful centers using the &lt;em&gt;mean&lt;/em&gt;. Now this only works for &lt;strong&gt;continuous numerical variables&lt;/strong&gt;.&#xA;The mean minimizes squared deviations, but not geographics distance. So in short:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do not use k-means on latitude,longitude,category data&lt;/strong&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results will be nonsense. Earth is a sphere (not a plane), and 1 degree north is not 1 degree east, and not 1 pizza or 1 &quot;chinese food&quot;. Do not do this.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-10-13T20:33:24.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14514" PostTypeId="2" ParentId="9917" CreationDate="2016-10-13T21:23:05.850" Score="1" Body="&lt;p&gt;Bayesian methods (e.g. Markov Chain Monte Carlo, Variational Bayes) may indeed be a good fit, but you may explore their combination with other approaches, like &lt;strong&gt;bayesian deep learning&lt;/strong&gt; (e.g. &lt;a href=&quot;http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learning/&quot; rel=&quot;nofollow&quot;&gt;this&lt;/a&gt;), which can add up learning intermediate representations of the information.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-10-13T21:23:05.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14515" PostTypeId="2" ParentId="14465" CreationDate="2016-10-13T22:25:09.373" Score="1" Body="&lt;p&gt;Both of these options have value if interpreted properly. You are trying to answer  the question (I assume): 'are we doing better this week than last week?'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what industry you're in or how variable (or volatile) your metric is, but I'd advise caution in interpreting changes week to week. I might suggest that alongside your weekly (or whatever) metrics, you include some moving average. That way, you capture the overall trend while tempering your reaction to occasional ups and downs.  &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2016-10-13T22:25:09.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14516" PostTypeId="2" ParentId="13660" CreationDate="2016-10-14T01:18:50.043" Score="4" Body="&lt;p&gt;The IDF part of TF-IDF gives less weight to a word if it occurs in a large fraction of the documents in your corpus.  However, this doesn't necessarily mean that the word is unimportant for distinguishing your two classes.  A word which is common in your corpus, but which also occurs substantially more often in one class than the other, could very well be quite valuable in distinguishing the classes.  This can especially be true if your set is not balanced between the two classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For example:&lt;/strong&gt;&#xA;Suppose that 95% of negative reviews contained the term &quot;boring&quot;, while only 5% of the positive reviews contained &quot;boring&quot;.  If this occurred in a balanced set of negative and positive reviews, that means that half the total reviews would contain &quot;boring&quot;, and this might be one of the terms whose weight is more strongly suppressed by IDF, despite the fact that it's obviously a valuable term for distinguishing the classes.  If instead 90% of the reviews are negative (with &quot;boring&quot; again occurring in 95% of negative reviews and 5% of positive ones), then a full 86% of your documents contained the term, and its weight might be highly suppressed by IDF despite its obvious importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, the problem is that TF-IDF is agnostic to the class labels in your training set.  One alternative to TF-IDF for text classification that doesn't have this flaw is Bi-Normal Separation (BNS), see for example this paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.hpl.hp.com/techreports/2007/HPL-2007-32R1.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;Forman, G.  BNS Feature Scaling: An Improved Representation over TF-IDF for SVM Text Classification.  Hewlett-Packard Labs Tech Report HPL-2007-32R1, 2007.&lt;/code&gt;&lt;/a&gt;(link to PDF)&lt;/p&gt;&#xA;" OwnerUserId="14" LastEditorUserId="14" LastEditDate="2016-10-14T01:31:05.303" LastActivityDate="2016-10-14T01:31:05.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14518" PostTypeId="1" AcceptedAnswerId="14549" CreationDate="2016-10-14T07:51:38.177" Score="0" ViewCount="63" Body="&lt;p&gt;I am working with a data set containing the abundance of 100+ species at 50+ locations over 20+ time-points. I want to simulate equivalent data sets with varying degrees of community turnover i.e. abundant species staying abundant/rare staying rare --&gt; abundant becoming rare and vice versa while maintaining the species abundance distribution in the original data set. I have seen mentions of resampling simulation techniques using bootstrapping where the desired difference between &quot;template&quot; and simulated data sets can be specified in terms of the proportion of all species that were present in both assemblages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if that is the best way forward to get the desired simulated data sets and if anyone could point me towards a tutorial on how to do this in R/give me some tips themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="25219" LastActivityDate="2016-10-15T15:03:04.900" Title="Simulating datasets of species abundances with varying degrees of turnover in R" Tags="|r|simulation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14519" PostTypeId="1" AcceptedAnswerId="14522" CreationDate="2016-10-14T08:42:58.667" Score="1" ViewCount="210" Body="&lt;p&gt;I'm trying to solve a multivariate regression problem similar to PLS regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem can be described as a connectivity analysis problem where we have two regions with unknown unidirectional connections(many-to-many) and given a set of input region patterns and output region patterns, we want to infer the underlying connections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically, the problem can be formulated as below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = BX \qquad$  where $Y \in \mathbb{R}_+^{M\times N}$, $X \in \mathbb{R}_+^{L\times N}$, and $B \in \mathbb{R}_+^{M\times L}$ with $L &amp;gt; M &amp;gt;&amp;gt; N$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The column of $X$ and $Y$, will be a vectorized version of 2D image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this would result in highly underdetermined system, I do have some prior knowledge about the pattern in input/output regions that I can incorporate in the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a model/idea that I can use in situation like this?  &lt;/p&gt;&#xA;" OwnerUserId="25093" LastActivityDate="2016-10-14T10:31:18.800" Title="Multitask multivariate regression?" Tags="|machine-learning|clustering|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14521" PostTypeId="2" ParentId="13759" CreationDate="2016-10-14T09:23:50.683" Score="0" Body="&lt;p&gt;This is related to the fact that the 'slam' package currently available on CRAN is dependent on a recent version of R (≥ 3.3.1). &#xA;Updating R will solve the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those on Ubuntu wishing to keep the R version included in the repository, it is possible to install 'slam' as a package with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sudo apt-get install r-cran-slam&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25222" LastActivityDate="2016-10-14T09:23:50.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14522" PostTypeId="2" ParentId="14519" CreationDate="2016-10-14T10:31:18.800" Score="0" Body="&lt;p&gt;You may want to model your problem using &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_linear_regression&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;bayesian regression&lt;/strong&gt;&lt;/a&gt;, which may allow you to introduce your prior knowledge in the form of &lt;strong&gt;priors&lt;/strong&gt; (&lt;em&gt;a priori&lt;/em&gt; distributions) of the model parameters. They would also allow you to model latent variables that govern the dynamics of the interactions (and impose priors on them as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The specific approach may be based on sampling (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo&quot; rel=&quot;nofollow&quot;&gt;Markov Chain Monte Carlo&lt;/a&gt;) or optimization (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Variational_Bayesian_methods&quot; rel=&quot;nofollow&quot;&gt;Variational Bayes&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the most popular bayesian frameworks is &lt;a href=&quot;http://mc-stan.org/&quot; rel=&quot;nofollow&quot;&gt;Stan&lt;/a&gt;, which has bindings to R (&lt;a href=&quot;http://mc-stan.org/interfaces/rstan&quot; rel=&quot;nofollow&quot;&gt;rstan&lt;/a&gt;) and python (&lt;a href=&quot;http://mc-stan.org/interfaces/pystan&quot; rel=&quot;nofollow&quot;&gt;pystan&lt;/a&gt;). In R there are other alternatives such a &lt;a href=&quot;http://www.mrc-bsu.cam.ac.uk/software/bugs/&quot; rel=&quot;nofollow&quot;&gt;BUGS&lt;/a&gt; and &lt;a href=&quot;http://mcmc-jags.sourceforge.net/&quot; rel=&quot;nofollow&quot;&gt;JAGS&lt;/a&gt;. In the python realm other options are &lt;a href=&quot;https://github.com/pymc-devs/pymc3&quot; rel=&quot;nofollow&quot;&gt;PyMC&lt;/a&gt; (which is also pretty popular) or &lt;a href=&quot;https://github.com/blei-lab/edward&quot; rel=&quot;nofollow&quot;&gt;Edward&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-10-14T10:31:18.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14523" PostTypeId="2" ParentId="9863" CreationDate="2016-10-14T10:52:56.127" Score="1" Body="&lt;p&gt;I'd suggest padding your sequences to all be the same length. Obviously this isn't ideal as you end up wasting time processing fake data but it does solve the problem.&lt;/p&gt;&#xA;" OwnerUserId="25226" LastActivityDate="2016-10-14T10:52:56.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14524" PostTypeId="1" CreationDate="2016-10-14T11:00:46.320" Score="2" ViewCount="110" Body="&lt;p&gt;In Hintons talk &quot;&lt;a href=&quot;https://web.archive.org/web/20150929075327/http://d1baxxa0joomi3.cloudfront.net/69bf620e48d60761118dc4328f5d6e9c/basic.mp4&quot; rel=&quot;nofollow&quot;&gt;What's wrong about convolutional nets&lt;/a&gt;&quot; (Late 2016 or early 2015, I guess) he talks about capsules to make a modular CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any publicly available implementation or papers about this idea?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:35:52.020" LastActivityDate="2021-02-16T15:35:52.020" Title="Is there any public implementation / publication with Hintons capsules idea?" Tags="|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14525" PostTypeId="2" ParentId="14458" CreationDate="2016-10-14T11:08:06.390" Score="1" Body="&lt;p&gt;Considering data stored in CSV format like below without headers you can use below R code to plot simple bar chart. It will plot occurrences of transactions grouped by unique transactions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;product1,product2,product3&#xA;product1,product4&#xA;product1,product2&#xA;product4,product3,product1,product2&#xA;product1,product2,product3&#xA;product1,product4&#xA;product1,product4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;R Code -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transactions &amp;lt;- read.csv(&quot;filepath/transactions.csv&quot;, header = FALSE)&#xA;transactions$V1&#xA;plot(transactions$V1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Bar Chart-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s4Z9m.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s4Z9m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25223" LastActivityDate="2016-10-14T11:08:06.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14526" PostTypeId="2" ParentId="13301" CreationDate="2016-10-14T11:44:29.763" Score="2" Body="&lt;p&gt;Yes, this is guaranteed by the &lt;strong&gt;Moore–Aronszajn theorem&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$K(\mathbf{x},\mathbf{y}) = e^{-\|\mathbf{x} - \mathbf{y}\|}$ is a positive definite kernel.  This means it is a symmetric function satisfying $$\sum_{i,j=1}^{n} c_i c_j K(\mathbf{x_i},\mathbf{x_j}) \ge 0$$ for all $n \in \mathbb{N}$, all $\mathbf{x_1},\dotsb ,\mathbf{x_n} \in \mathbb{R}^n$, and all $c_1, \dotsb , c_n \in \mathbb{R}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Moore–Aronszajn theorem says that for each such function there exists a unique &lt;strong&gt;reproducing kernel Hilbert space&lt;/strong&gt; with reproducing kernel $K(\mathbf{x},\mathbf{y})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, there must be a unique Hilbert space $H$ of functions $f: \mathbb{R}^n \to \mathbb{R}$ with a mapping $\phi: \mathbb{R}^n \to H$ such that $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ for all $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$, and such that $K(\mathbf{x},\mathbf{y})$ satisfies the &lt;strong&gt;reproducing property&lt;/strong&gt;: $$\langle f, K(\cdot, \mathbf{x}) \rangle = f(\mathbf{x}) \qquad \forall \mathbf{x} \in \mathbb{R}^n, \enspace \forall f \in H$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the reproducing kernel Hilbert space is unique for a given positive definite kernel $K(\mathbf{x}, \mathbf{y})$, there can be other mappings $\phi$ to other Hilbert spaces which also satisfy $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ but without the reproducing property.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-14T11:44:29.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14527" PostTypeId="1" AcceptedAnswerId="16631" CreationDate="2016-10-14T11:50:40.087" Score="6" ViewCount="14083" Body="&lt;p&gt;When using the python / sklearn API of xgboost are the probabilities obtained via the &lt;code&gt;predict_proba&lt;/code&gt; method &quot;real probabilities&quot; or do I have to use &lt;code&gt;logit:raw&lt;/code&gt;and manually calculate the sigmoid function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to experiment with different cutoff points. Currently using &lt;code&gt;binary:lgistic&lt;/code&gt;via the &lt;code&gt;sklearn:XGBClassifier&lt;/code&gt; the probabilities returned from the &lt;code&gt;prob_a&lt;/code&gt; method rather resemble 2 classes and not a continuous function where changing the cut-off point impacts the final scoring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right way to obtain probabilities for experimenting with the cutoff value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xERKI.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xERKI.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9053" LastActivityDate="2017-01-30T23:21:06.327" Title="Xgboost predict probabilities" Tags="|predictive-modeling|scikit-learn|xgboost|probability|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14528" PostTypeId="1" CreationDate="2016-10-14T16:19:41.233" Score="1" ViewCount="1321" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; In recent months, &lt;a href=&quot;https://www.elastic.co/blog/welcome-prelert-to-the-elastic-team&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elastic has purchased Prelert&lt;/a&gt; and will actively incorporate it into the Elastic stack (and also discontinue the Splunk Anomaly Detective App!). I am trying to understand the theoretical methodology and process framework for their predictive analytics approach to time series anomaly detection so I can be an effective and efficient user of the Prelert tool. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; What topics to explore, and what is the general process flow/pipeline? How would one start to learn and understand the building blocks of the big concepts involved. I understand from a few posts online (&lt;a href=&quot;http://info.prelert.com/how-automated-anomaly-detection-works&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;,&lt;a href=&quot;http://www.prelert.com/resources/Anomaly_Detection_In_APM_Data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;) a bit about how it works, but want to understand the major process items involved. At a high level, I get that there is some training set that fits a distribution via unsupervised ML, and that distribution is used to assess probability of future events - but need to get into the sequential processing blocks involved (for example, data read step, followed by sending it to a distribution-matching step, then some unsupervised ML step where a distribution is selected/adjusted, etc). I understand specifics may be difficult, so guesses and hypothetical ideas from people with experience in this arena would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note 1:&lt;/strong&gt; I am &lt;strong&gt;&lt;em&gt;not&lt;/em&gt;&lt;/strong&gt; interested in the Prelert's &quot;secret sauce&quot; and the exact proprietary algorithms that they have developed - this would be unethical and I do not want to recreate their product, I just want to understand how to better use it. I am interested in an explanation of the main concepts implemented at a high level but in more detail than the posts (&lt;a href=&quot;http://info.prelert.com/how-automated-anomaly-detection-works&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;,&lt;a href=&quot;http://www.prelert.com/resources/Anomaly_Detection_In_APM_Data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;) I found. Example post of the explanation could be in the form: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;'distribution is populated via X concept/algorithm and then followed&#xA;by selection step by Y machine learning concept/algorithm. Possible&#xA;concepts to study are A,B,C. Key search words for literature searches&#xA;are Q,W,E. Some references are N,M,L.'&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note 2:&lt;/strong&gt; I have posted on a &lt;a href=&quot;https://datascience.stackexchange.com/questions/6547/open-source-anomaly-detection-in-python&quot;&gt;related topic before&lt;/a&gt;, but perhaps this is an interesting side question for a similar audience. &lt;/p&gt;&#xA;" OwnerUserId="1406" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-10-14T16:19:41.233" Title="How does Elastic's Prelert (formerly Splunk Anomaly Detective App) work?" Tags="|machine-learning|data-mining|predictive-modeling|time-series|anomaly-detection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14529" PostTypeId="2" ParentId="5549" CreationDate="2016-10-14T16:25:27.033" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://www-users.cs.umn.edu/~banerjee/papers/09/anomaly.pdf&quot; rel=&quot;nofollow&quot;&gt;Chandola et al's survey&lt;/a&gt; is probably the best and most-widely cited survey in the anomaly detection field. &lt;/p&gt;&#xA;" OwnerUserId="1406" LastActivityDate="2016-10-14T16:25:27.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14530" PostTypeId="1" AcceptedAnswerId="17547" CreationDate="2016-10-14T16:44:23.177" Score="0" ViewCount="1061" Body="&lt;p&gt;Can anyone explain the purpose of &quot;numDecimalPlaces&quot; parameter in J48 WEKA classifier ? Its default value is 2 and its description is given as follows: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The number of decimal places to be used for the output of numbers in&#xA;  the model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, I can't understand how it going to affect the results. &#xA;For example, if my feature values are something like this (0.00123 or 0.00234 etc). Does it means that the trained model will only consider these values up to 2 decimal points ? The documentation also says: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the documentation, its mentioned in the summary fields section:                Field Summary&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Fields inherited from class weka.classifiers.AbstractClassifier&#xA;  BATCH_SIZE_DEFAULT, NUM_DECIMAL_PLACES_DEFAULT&lt;br&gt;&#xA;  The number of decimal places used when printing numbers in the model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It means that changing this decimal point should change the decimal points in the printed output of the trained classifier in WEKA explorer. However, when I try to change it (1,2,3,4,5 etc decimal points), it doesn't affect the number of decimals in the decision tree conditional statements in the printed output. &#xA;A snapshot: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k1 &amp;lt;= 0.12544&#xA;|   k4 &amp;lt;= 4.504&#xA;|   |   k22 &amp;lt;= 0.24044: Class1 (179.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="25183" LastActivityDate="2017-03-13T08:54:03.533" Title="What does the “numDecimalPlaces” in J48 classifier do in WEKA?" Tags="|machine-learning|classification|decision-trees|weka|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14531" PostTypeId="2" ParentId="8560" CreationDate="2016-10-14T20:04:02.820" Score="1" Body="&lt;p&gt;Actually, for binary classification with HMMs you have 2 choices:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Either you have many examples for both Class A and B. Then you can train one HMM from your samples of Class A and one from your samples from Class B. When you want to classify a new sample, you compute the likelihood of this sample with respect to the 2 HMMs and choose the one with the highest likelihood.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the case one label is way more frequent than the other (say Class B appears for every 100 samples of Class A), or in the case one category is broader than the other (have more variance, i.e. Class A is an expected behavior of a process and Class B is all the cases that are abnormal), then we might want to train a unique HMM for Class A. For each new sample, we compute the likelihood to the model, and if it falls below a pre-defined threshold (that can be chosen with respect to some validation sample, or in the worst case, from the training samples themselves) we assign this sample to Class B. If the likelihood is above the threshold, we assign the sample to Class A.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25157" LastActivityDate="2016-10-14T20:04:02.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14532" PostTypeId="2" ParentId="12959" CreationDate="2016-10-14T20:14:17.463" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Edit: It turns out that the above hmm can only be applied to discrete&#xA;  values which is possibly why I was getting errors. It seems I have to&#xA;  use another function for 'gaussian output' instead, mhmm_em. I'll post&#xA;  another question to figure out its working.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In order to clarify this a bit.&#xA;The functions you were trying to use are indeed defined for discrete data represented by probability mass functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When working with continuous data and modeling them with an HMM, one needs to decide what probability distribution will best represent the data. When the data has an unbounded support, the Gaussian is often a good first choice, and most machine leaning toolboxes have built-in functions for the training of these HMMs. However, an HMM can be trained with other probability distributions such as the Dirichlet distribution (if your data is proportional), or the Student-t, Weibull distributions,... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that the distribution you choose is a modeling hypothesis that you do with respect to the data you have. If the hypothesis does not make sense, the results might not be accurate. Unfortunately, the use of these other distributions mostly rely on you to either, derive and code the needed equations (however, in some cases the equations are available in some scientific papers).&lt;/p&gt;&#xA;" OwnerUserId="25157" LastEditorUserId="25157" LastEditDate="2016-11-07T14:43:16.803" LastActivityDate="2016-11-07T14:43:16.803" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14533" PostTypeId="1" CreationDate="2016-10-14T20:41:56.590" Score="1" ViewCount="1503" Body="&lt;p&gt;In Keras, I would like to train a network with binary weights in the manner of &lt;a href=&quot;https://arxiv.org/abs/1602.02830&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coubariaux, et al.&lt;/a&gt;, but I cannot figure out where the quantization (binarization) should occur within the code.&lt;/p&gt;&#xA;&lt;p&gt;A core aspect of the training method is this:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;At the beginning of each batch during training, the stored real (e.g., int32) weights are converted to binary values (either by rounding or in a stochastic/probabilistic manner) and stored separately from the real-valued weights.&lt;/li&gt;&#xA;&lt;li&gt;Binary-valued weights are used in the forward pass, to compute activations.&lt;/li&gt;&#xA;&lt;li&gt;Real-valued weights are used in the backward pass, to compute the gradients.&lt;/li&gt;&#xA;&lt;li&gt;Weight updates are applied to the real-valued weights, not the binary ones.&lt;/li&gt;&#xA;&lt;li&gt;Binary-valued weights are not altered until the next batch, at which time they are recomputed by binarizing the recently updated real-valued weights.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;If I do the binarization in &lt;code&gt;Layer.call()&lt;/code&gt;, I believe it will occur for every forward pass (for every sample), but it should only occur once per batch.&lt;/p&gt;&#xA;&lt;p&gt;If I do the binarization in a &lt;code&gt;Callback.on_batch_begin()&lt;/code&gt;, I don't think I can specify the use of binary weights for the forward pass, and real-valued weights for gradient computation.&lt;/p&gt;&#xA;&lt;p&gt;Any suggestions?  Unfortunately, my Python knowledge is not deep, so it's been somewhat challenging for me to understand the flow of the code.&lt;/p&gt;&#xA;&lt;p&gt;**Note that when I talk about binary values, I do not mean 1-bit (e.g., 8 binary values placed into an int8).  The binary-valued weights can still be represented by int32, float32, whatever.&lt;/p&gt;&#xA;" OwnerUserId="25243" LastEditorUserId="29169" LastEditDate="2021-07-10T20:23:32.037" LastActivityDate="2023-12-16T02:05:50.707" Title="How to quantize weights in forward pass during training in Keras?" Tags="|neural-network|keras|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14534" PostTypeId="1" AcceptedAnswerId="14536" CreationDate="2016-10-14T23:38:07.803" Score="3" ViewCount="604" Body="&lt;p&gt;I'm new to data science but trying to get better&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I have an attribute and plotting its histogram&#xA;&lt;a href=&quot;https://i.stack.imgur.com/PuZZW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PuZZW.png&quot; alt=&quot;histogram&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I know so far such a distribution is imbalanced and my goal is to equalize things a little bit right?&#xA;Again from what I know, I have to transform this attribute to be more suitable for linear regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it obvious (to someone more experienced than me) which kind of transformation is applicable in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note that this is an attribute and not my target, this is not what I am trying to predict. This is one of the attributes to be used for predicting&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2019-12-16T16:31:33.953" Title="How to transform an imbalanced attribute to make it more suitable for linear regression?" Tags="|dataset|linear-regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14535" PostTypeId="1" CreationDate="2016-10-15T04:58:49.067" Score="-3" ViewCount="111" Body="&lt;p&gt;I have a 3GB dataset but my computer only has 4GB ram. Is it possible to run some simple analyses in R-studio (like anova, use some built in packages), etc. with that one 1GB of additional ram? &#xA;If it is impossible to run, is there any solutions to this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know before anything crashes and I lose my work... &#xA;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="24467" LastActivityDate="2016-10-15T14:14:05.603" Title="Do i have enough ram for the file analysis?" Tags="|dataset|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14536" PostTypeId="2" ParentId="14534" CreationDate="2016-10-15T08:44:19.023" Score="4" Body="&lt;p&gt;You have multiple options and you may choose the best by seeing the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First one is to use a guess. For positive-only values the log-transform is a hot candidate to make sense (maybe correct for small values here to avoid exceedingly large negative transformed values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Log-transform is natural if percentage increases have a particular real-world meaning. This is often the case for financial data. Why is log a hot candidate? You probably know that when there are addititve real world effects, the normal distribution often appears. Now, when you have multiplicative effects, you get the &lt;a href=&quot;https://en.wikipedia.org/wiki/Log-normal_distribution&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Log-normal_distribution&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other common transformations are power transforms, where you take some power of the values. I don't think there are many more which are very common. Theoretically, your perfect transform would make the noise on the linear regression Gaussian, but no-one can tell what that would be and most likely reality isn't perfectly linear anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A transform is more interesting when the transformed values follow a Gaussian distribution. But that is just a guess and in the end only final performance evaluation can tell more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a second option, be aware that you can force the transformed values to be any distribution you want. For example, if you take ranks of the values you get a uniform transformed distribution. You could even force it to be a Gaussian by a suitable mapping. However, in your case, this will lose the interesting bump on the right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think these are the most common options. &lt;em&gt;In data science, nothing is ever obvious and most of the time you can only decide by performance evaluation (cross-validation with the whole model)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set up a performance test (cross-validation; not on final test set though if you like a fair, final evaluation) and try all of the following&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try untransformed. It might already be what has most information.&lt;/li&gt;&#xA;&lt;li&gt;Try log-transform (while adding a small offset if you have some very small values)&lt;/li&gt;&#xA;&lt;li&gt;Try power-transform if you like&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="723" LastActivityDate="2016-10-15T08:44:19.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14537" PostTypeId="2" ParentId="14535" CreationDate="2016-10-15T09:29:33.847" Score="2" Body="&lt;p&gt;This depends on several factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Whether your operating system is 32 or 64 bits. If its a 32-bit operating system, most certainly you will not be able to do everything at the same time in RAM.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The version of R. There is a historic limit on vectors (and so on matrices also) length of 2^31-1. However, R 3.0.0 incorporated &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/LongVectors.html&quot; rel=&quot;nofollow&quot;&gt;long vectors&lt;/a&gt; which allow for larger vectors.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The specific processing you are doing and the amount of memory it uses (e.g. temporary buffers), and the memory access pattern it follows.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The amount of time you are willing to wait. You operating system's &lt;a href=&quot;https://en.wikipedia.org/wiki/Virtual_memory&quot; rel=&quot;nofollow&quot;&gt;virtual memory&lt;/a&gt; mechanisms will almost certainly enable your processing to take place, but swapping to disk may take time if your swap partition is on a magnetic drive (it may get much better if it is on an SSD drive). The amount of swap space is also relevant here in the case it is not enough to fit the needs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Summing up...it depends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, &lt;strong&gt;what you can do to avoid losing your work&lt;/strong&gt; is saving your R objects to file by means of &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/save.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;save&lt;/code&gt;&lt;/a&gt; so that you are able to restore it later by means of &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/load.html&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;load&lt;/code&gt;&lt;/a&gt; in the case anything goes wrong. After saving your data, just launch your processing without fear, you will be able to retry as many times as you want.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-10-15T09:29:33.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14538" PostTypeId="1" CreationDate="2016-10-15T09:37:32.517" Score="1" ViewCount="275" Body="&lt;p&gt;I have downloaded and built octave library and it works fine . But I cannot call function minimizers like &lt;code&gt;fminunc()&lt;/code&gt; , &lt;code&gt;fmingc()&lt;/code&gt; etc to minimize my functions for performing logistic regression or using it in neural networks . Can these functions be accessed from c++ ? If yes, then how ?&lt;/p&gt;&#xA;" OwnerUserId="24077" LastActivityDate="2016-11-14T10:58:02.440" Title="How can I use octave function minimizers in c++?" Tags="|machine-learning|neural-network|logistic-regression|optimization|octave|" AnswerCount="1" CommentCount="1" ClosedDate="2016-11-14T14:59:37.560" ContentLicense="CC BY-SA 3.0" />
  <row Id="14539" PostTypeId="2" ParentId="14538" CreationDate="2016-10-15T10:08:12.010" Score="1" Body="&lt;p&gt;You can use a native library in C++ for optimization. Check out this one :&#xA;&lt;a href=&quot;http://dlib.net/optimization.html&quot; rel=&quot;nofollow&quot;&gt;dlib&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-10-15T10:08:12.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14540" PostTypeId="2" ParentId="14300" CreationDate="2016-10-15T10:36:53.213" Score="2" Body="&lt;p&gt;The columns of your &lt;code&gt;loadings&lt;/code&gt; matrix are a basis of orthonormal eigenvectors.  This is an important concept from linear algebra, and well worth learning about in detail if you're not familiar.  But for the purposes of this answer it can be understood as defining a system of coordinates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each student, we can define a point in a four-dimensional space (specifically, in $\mathbb{R}^4$) which represents their grades (after centering and normalization).  Or to put it another way, you can imagine the set of all students' grades as a scatterplot in four dimensions, with four perpendicular axes.  We can orient these axes in various directions (just as we can in two or three dimensions).  The most obvious choice is to have one axis for each subject, so the axis which is collinear with the unit vector pointing from the origin to the point $(1,0,0,0)$ represents their grade in German, and likewise the axis which is collinear with the vector $(0,1,0,0)$ represents their grade in Philosophy, the axis which is collinear with the vector $(0,0,1,0)$ represents their grade in Math, and the axis which is collinear with the vector $(0,0,0,1)$ represents their grade in Physics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there's no reason to expect that the direction in which our scatterplot is most spread out (the direction of greatest variance in the data) will align with one of these axes.  PCA picks out a new set of axes so that one axis aligns with the direction of greatest variance, and another aligns with the direction of the greatest remaining variance after the first direction is projected out, and so forth.  The unit vectors (expressed in the original coordinate system) which point along these new axes are the columns in your &lt;code&gt;loadings&lt;/code&gt; matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of this particular example, the loading vector for the first principal component is along an axis that basically expresses whether they're better at Math and Physics, or better at German and Philosophy.  The loading vector for the second principal component is along an axis that basically expresses how good or bad a student they are over all (hence all the components of the vector have the same sign and similar magnitude).  You wondered about the negative sign on all four components - if you're familiar with eigenvectors you'll know that changing all components of the vector by an overall sign is irrelevant.  Basically, it's the same as just swapping which end of the axis we call positive and which we call negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this case the first two loading vectors are fairly close to what many of us might have expected to see.  But even in this fairly intuitive example, you shouldn't be surprised that the loading vectors for the later principal components don't seem as obvious to you.  That's because these are only addressing the variance that &lt;em&gt;remains&lt;/em&gt; after we project out the variance that's explained by the first two factors.  We all probably know that students who are good at Physics tend to be good at Math, but how many of us know (for example) if, after controlling for how good they are at Physics, the ones who are also better at Philosophy than German will be better at Math?  These subtler effects will be less obvious to a causal observer than the dominant effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get to the loading vector for the fourth principal component (out of four), you really don't need to wonder at all about why it has the particular value that it has.  In fact, this vector was entirely determined by the previous three (up to the irrelevant overall sign) .  This can be understood by remembering that PCA picked out four perpendicular axes in a four-dimensional space - once the first three are specified, there's only one remaining possible choice that's perpendicular to all of them.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-15T10:36:53.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14541" PostTypeId="1" CreationDate="2016-10-15T13:25:37.450" Score="0" ViewCount="4643" Body="&lt;p&gt;I'd import &lt;strong&gt;&lt;em&gt;'tweepy'&lt;/em&gt;&lt;/strong&gt; package for accessing twitter data for analysis and visualization in Python. Afterwards I did setup oauth authentication and everything else but on the next step, which is STREAMING TWEETS for me am using MyStreamListener. This causing me a &lt;strong&gt;&lt;em&gt;Name Error&lt;/em&gt;&lt;/strong&gt;--&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NameError: name 'MyStreamListener' is not defined&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here's my code-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# API Authentication &#xA;&#xA;# Importing TWEEPY Library&#xA;import tweepy&#xA;&#xA;# Store OAuth authentication credentials in relevant variables&#xA;access_token = &quot;provided&quot;&#xA;access_token_secret = &quot;provided&quot;&#xA;consumer_key = &quot;provided&quot;&#xA;consumer_secret = &quot;provided&quot;&#xA;&#xA;# Pass OAuth details to tweepy's OAuth handler&#xA;auth = tweepy.OAuthHandler(consumer_key, consumer_secret)&#xA;auth.set_access_token(access_token, access_token_secret)&#xA;&#xA;# Streaming Tweets  &#xA;&#xA;# Initialize Stream listener&#xA;l = MyStreamListener()&#xA;&#xA;# Create you Stream object with authentication&#xA;stream = tweepy.Stream(auth, l)&#xA;&#xA;# Filter Twitter Streams to capture data by the keywords:&#xA;stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What should I do now? Any thoughts?&#xA;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2017-02-12T15:29:43.847" Title="NameError: name 'MyStreamListener' is not defined" Tags="|python|visualization|social-network-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14542" PostTypeId="2" ParentId="13759" CreationDate="2016-10-15T13:28:29.280" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It worked after I &lt;strong&gt;&lt;em&gt;reinstalled the R&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;.&#xA;So I'll suggest to try doing this if anyone facing the same issue. Two of my mates also solved this issue by following the same approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-15T13:28:29.280" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14543" PostTypeId="2" ParentId="14541" CreationDate="2016-10-15T13:44:51.140" Score="1" Body="&lt;p&gt;You need to override StreamListener() class. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# API Authentication &#xA;&#xA;# Importing TWEEPY Library&#xA;import tweepy&#xA;&#xA;# Store OAuth authentication credentials in relevant variables&#xA;access_token = &quot;provided&quot;&#xA;access_token_secret = &quot;provided&quot;&#xA;consumer_key = &quot;provided&quot;&#xA;consumer_secret = &quot;provided&quot;&#xA;&#xA;# Pass OAuth details to tweepy's OAuth handler&#xA;auth = tweepy.OAuthHandler(consumer_key, consumer_secret)&#xA;auth.set_access_token(access_token, access_token_secret)&#xA;&#xA;# Streaming Tweets&#xA;#override tweepy.StreamListener to add logic to on_status&#xA;class MyStreamListener(tweepy.StreamListener):&#xA;    def on_status(self, status):&#xA;        print(status.text)&#xA;&#xA;# Initialize Stream listener&#xA;l = MyStreamListener()&#xA;&#xA;# Create you Stream object with authentication&#xA;stream = tweepy.Stream(auth, l)&#xA;&#xA;# Filter Twitter Streams to capture data by the keywords:&#xA;stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25223" LastActivityDate="2016-10-15T13:44:51.140" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14544" PostTypeId="2" ParentId="14535" CreationDate="2016-10-15T14:14:05.603" Score="1" Body="&lt;p&gt;You can try &lt;em&gt;ff&lt;/em&gt; or &lt;em&gt;bigmemory&lt;/em&gt; package to handle large datasets. &lt;/p&gt;&#xA;" OwnerUserId="25223" LastActivityDate="2016-10-15T14:14:05.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14545" PostTypeId="2" ParentId="14458" CreationDate="2016-10-15T14:17:17.447" Score="0" Body="&lt;p&gt;You can use &lt;strong&gt;&lt;em&gt;&lt;a href=&quot;https://plot.ly/&quot; rel=&quot;nofollow&quot;&gt;plotly&lt;/a&gt;&lt;/em&gt;&lt;/strong&gt; for making interactive visualization and charts. You can also give a try to &lt;strong&gt;&lt;em&gt;Linkurious&lt;/em&gt;&lt;/strong&gt;.&#xA;Hope it helps.&#xA;Cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-15T14:17:17.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14546" PostTypeId="2" ParentId="9473" CreationDate="2016-10-15T14:30:12.527" Score="3" Body="&lt;p&gt;take a look at these blogs- &lt;a href=&quot;http://www.parallelr.com/r-deep-neural-network-from-scratch/&quot; rel=&quot;nofollow&quot;&gt;R for Deep Learning - 1&lt;/a&gt; and &lt;a href=&quot;http://www.parallelr.com/r-dnn-parallel-acceleration/&quot; rel=&quot;nofollow&quot;&gt;R for Deep Learning - 2&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-15T14:30:12.527" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14548" PostTypeId="1" AcceptedAnswerId="14556" CreationDate="2016-10-15T14:52:03.983" Score="0" ViewCount="2992" Body="&lt;p&gt;I am working on a group project for my capstone course and we have been tasked with creating a sentiment analysis tool with Python business logic and (L/W)AMP everything else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have good feedback for every part of our project plan except for feature extraction. One of our advisors insists that we should have ~15 different kinds of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently we only use unigrams and are having a hard time finding others that are practical to implement with our small set of data (~50 items) and within our time limit (~2 weeks to fully implement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are feature extraction techniques that are useful for sentiment analysis and work on smaller datasets? They should be able to be implemented quickly or already exist in a Python library.&lt;/p&gt;&#xA;" OwnerUserId="14112" LastActivityDate="2016-10-16T06:46:54.647" Title="Feature extraction for sentiment analysis" Tags="|python|nlp|feature-extraction|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14549" PostTypeId="2" ParentId="14518" CreationDate="2016-10-15T15:03:04.900" Score="0" Body="&lt;p&gt;Without any example of your data or distributions it is a hard question to answer. Without further info I think there are two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;resample your data using stratified sampling. The R &lt;a href=&quot;https://cran.r-project.org/web/packages/sampling/sampling.pdf&quot; rel=&quot;nofollow&quot;&gt;sample&lt;/a&gt; package can be of good help here.&lt;/li&gt;&#xA;&lt;li&gt;generate a multivariate distribution, as far as I know there are no packages that will directly do this for you. Depending on the distributions of your data, creating the right multivariate distribution which includes the right correlations may be very hard. In some cases you may even need to use copulas (this is especially necessary if you are interested in tail events, correlation will not be sufficient to create dependence of these events). Note that this topic can be a separate PhD thesis on itself.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-15T15:03:04.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14550" PostTypeId="2" ParentId="9802" CreationDate="2016-10-15T22:54:42.727" Score="5" Body="&lt;p&gt;There is no difference in the algorithm, they are the same. When you use  &lt;code&gt;reg:logistic&lt;/code&gt; the default evaluation metric is &lt;code&gt;rmse&lt;/code&gt;, but you can change this parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;error rate&lt;/code&gt; and the &lt;code&gt;rmse&lt;/code&gt; may differ depending on the distribution of your output, as the &lt;code&gt;error rate&lt;/code&gt; uses a limit of $0.5$ if you have the output values concentrated in $0$ or $1$ it will be much smaller than &lt;code&gt;rmse&lt;/code&gt;, even though its correlated metric the model can be very different, the application will depend on your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To evaluate performance of logistic regression models is usual to calculate auc, logloss, accuracy, pseudo-$R^2$ and &lt;a href=&quot;https://www.r-bloggers.com/evaluating-logistic-regression-models/&quot; rel=&quot;noreferrer&quot;&gt;some others&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="25238" LastActivityDate="2016-10-15T22:54:42.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14552" PostTypeId="1" AcceptedAnswerId="14723" CreationDate="2016-10-16T03:32:22.690" Score="1" ViewCount="3656" Body="&lt;p&gt;I am computing confidence intervals for random forests using the package available here: &lt;a href=&quot;http://contrib.scikit-learn.org/forest-confidence-interval/auto_examples/plot_mpg.html#sphx-glr-auto-examples-plot-mpg-py&quot; rel=&quot;nofollow&quot;&gt;http://contrib.scikit-learn.org/forest-confidence-interval/auto_examples/plot_mpg.html#sphx-glr-auto-examples-plot-mpg-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to plot the average MPG and associated confidence interval. I can compute the average MPG but how do I compute the associated confidence interval?&lt;/p&gt;&#xA;" OwnerUserId="12985" LastActivityDate="2016-12-22T16:28:24.350" Title="Computing confidence interval of average output from random forest in scikit learn" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14553" PostTypeId="2" ParentId="904" CreationDate="2016-10-16T06:01:33.283" Score="2" Body="&lt;p&gt;I did find &lt;a href=&quot;http://rmarkdown.rstudio.com/flexdashboard/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flex Dashboard&lt;/a&gt; option pretty cool, interactive and easy to use. Give it a try maybe you would too.&lt;/p&gt;&#xA;&lt;p&gt;Meanwhile, you can also try these as well.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://plot.ly/dashboard/create/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plotly Dashboard&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://rstudio.github.io/shinydashboard/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Shiney Dashboard&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope it helps!&#xA;&lt;em&gt;(Update me if you get something even better than that.)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-16T06:01:33.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14554" PostTypeId="2" ParentId="2480" CreationDate="2016-10-16T06:21:39.837" Score="1" Body="&lt;p&gt;Take a look at this interactive chart gallery- &lt;a href=&quot;https://github.com/d3/d3/wiki/gallery&quot; rel=&quot;nofollow&quot;&gt;Visual Index&lt;/a&gt; simple to use and works just fine with any type of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from this you can also give a try to any of these examples.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bl.ocks.org/jose187/4733747&quot; rel=&quot;nofollow&quot;&gt;D3 Network Graph&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/network-visualization-part-6-d3-and-r-networkd3/&quot; rel=&quot;nofollow&quot;&gt;R Bloggers&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-16T06:21:39.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14555" PostTypeId="2" ParentId="1241" CreationDate="2016-10-16T06:26:57.983" Score="0" Body="&lt;p&gt;There is a perfect solution to this. You can try any of these tools for achieving the particular functionality&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://giphy.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Giphy&lt;/a&gt;&#xA;&lt;em&gt;(Simple and open source)&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://linkurio.us/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Linkurious&lt;/a&gt; &lt;em&gt;(More interactive visualization but paid. Trial availble)&lt;/em&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-16T06:26:57.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14556" PostTypeId="2" ParentId="14548" CreationDate="2016-10-16T06:46:54.647" Score="1" Body="&lt;p&gt;Have a look at these papers-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://semanticsimilarity.files.wordpress.com/2011/09/10-1109tkde-2006-130-authors-accepted-version.pdf&quot; rel=&quot;nofollow&quot;&gt;Semantic Similarity - 1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://semanticsimilarity.files.wordpress.com/2011/09/using-a-slim-function-word-classifier-to-recognise-instruction-dialogue-acts-10-1007978-3-642-22000-5_4.pdf&quot; rel=&quot;nofollow&quot;&gt;Semantic Similarity - 2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They came up with a good solution. Have a look maybe you will find something of your use in it. Plus try exploring these too&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pypi.python.org/pypi/vaderSentiment&quot; rel=&quot;nofollow&quot;&gt;vaderSentiment 0.5&lt;/a&gt;&lt;br&gt;&#xA;(VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://text-processing.com/demo/sentiment/&quot; rel=&quot;nofollow&quot;&gt;Demo- Sentiment Analysis with Python&lt;/a&gt;&lt;br&gt;&#xA;(This is a demonstration of sentiment analysis using a NLTK 2.0.4 powered text classification process. It can tell you whether it thinks the text you enter below expresses positive sentiment, negative sentiment, or if it's neutral. Using hierarchical classification, neutrality is determined first, and sentiment polarity is determined second, but only if the text is not neutral.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-16T06:46:54.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14559" PostTypeId="1" AcceptedAnswerId="14561" CreationDate="2016-10-16T09:50:02.800" Score="2" ViewCount="14602" Body="&lt;p&gt;I'm trying to create some charts using Python. I've this dataset in a CSV file:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Banana  Water   Rice&#xA;Rice    Water&#xA;Bread   Banana  Juice &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And I've this code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; import numpy as np&#xA; from pandas import DataFrame&#xA; import matplotlib&#xA; matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)&#xA; import matplotlib.pyplot as plt&#xA;    &#xA; data = DataFrame.read_csv(&amp;quot;test.csv&amp;quot;, index_col=1, skiprows=1).T.to_dict()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But when I execute the code I'm getting the following error:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;AttributeError: type object 'DataFrame' has no attribute 'read_csv'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How can I solve this problem?&lt;/p&gt;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastEditorUserId="97080" LastEditDate="2023-01-17T17:58:17.573" LastActivityDate="2023-01-17T17:58:31.503" Title="AttributeError: type object 'DataFrame' has no attribute 'read_csv'" Tags="|python|visualization|pandas|csv|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14561" PostTypeId="2" ParentId="14559" CreationDate="2016-10-16T12:05:28.160" Score="8" Body="&lt;p&gt;&lt;code&gt;read_csv()&lt;/code&gt; is not available on DataFrame. to read csvs using pandas:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;data = pd.read_csv(&amp;quot;file_name&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;If you check &lt;code&gt;type(data)&lt;/code&gt;, it will be pandas DataFrame.&lt;/p&gt;&#xA;" OwnerUserId="25223" LastEditorUserId="97080" LastEditDate="2023-01-17T17:58:27.767" LastActivityDate="2023-01-17T17:58:27.767" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14565" PostTypeId="2" ParentId="14489" CreationDate="2016-10-16T15:12:17.380" Score="1" Body="&lt;h2&gt;In a nutshell&lt;/h2&gt;&#xA;&lt;p&gt;If you have a look at what &lt;code&gt;mnist.train&lt;/code&gt; is you'll find there are two numpy arrays in it:&#xA;&lt;code&gt;mnist.train.images&lt;/code&gt; (shape &lt;code&gt;(55000, 784)&lt;/code&gt;)and &lt;code&gt;mnist.train.labels&lt;/code&gt; (shape &lt;code&gt;(55000, 10)&lt;/code&gt;).Then all you have to do is iterate on these.&lt;/p&gt;&#xA;&lt;h2&gt;Example&lt;/h2&gt;&#xA;&lt;p&gt;This example is voluntarily a bit complicated so that it is robust to whatever sizes you chose but keeps the idea of fixing a number of batches as primary information on how much you want to train on the dataset :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;X_train = mnist.train.images&#xA;Y_train = mnist.train.labels&#xA;number_of_examples = X_train.shape[0]&#xA;batch_size = 1000&#xA;number_of_batches = 200&#xA;epoch = 0&#xA;&#xA;for i in range(number_of_batches):&#xA;    &#xA;&#xA;    j = (i - epoch) * batch_size % number_of_examples&#xA;    k = (i - epoch + 1) * batch_size % number_of_examples&#xA;&#xA;    if (k &amp;lt; j):&#xA;        k = number_of_examples&#xA;        batch_x = X_train[j:number_of_examples, :]&#xA;        batch_y = Y_train[j:number_of_examples, :]&#xA;&#xA;        print('Shuffling data to retrain on dataset')&#xA;        data = numpy.concatenate((X_train, Y_train), axis=1)&#xA;        np.random.shuffle(data)&#xA;&#xA;        # redefine X_train and Y_train from shuffled dataset&#xA;        X_train = data[:, :784]&#xA;        Y_train = data[:, 784:794]&#xA;&#xA;        epoch = i + 1&#xA;    else:&#xA;        batch_x = X_train[j:k, :]&#xA;        batch_y = Y_train[j:k, :]&#xA;&#xA;    # [...]&#xA;    train_step.run(feed_dict={x: batch_x, y_: batch_y, keep_prob: 0.5})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;code&gt;j&lt;/code&gt; is the row of the dataset which will be the batch's first row&#xA;&lt;code&gt;k&lt;/code&gt; is the last one, so &lt;code&gt;j-k=batch_size&lt;/code&gt; examples per batch, as expected.&#xA;Except if you've reached the end of the dataset (&lt;code&gt;k&amp;lt;j&lt;/code&gt;)&#xA;in which case you just go to its end and then shuffle the&#xA;dataset so that batches won't be the same.&lt;/p&gt;&#xA;&lt;p&gt;You count &lt;code&gt;i - epoch&lt;/code&gt; and not just &lt;code&gt;i&lt;/code&gt; so that you're sure you'll always&#xA;start at the 1st row of the dataset even when you've already gone&#xA;completely through it.&#xA;For instance if &lt;code&gt;number_of_examples = 55555&lt;/code&gt;, &lt;code&gt;batch_size = 1000&lt;/code&gt; and&#xA;&lt;code&gt;number_of_batches = 200&lt;/code&gt;:  &lt;code&gt;i = 56&lt;/code&gt; is the first time you go again&#xA;through the (shuffled) dataset but &lt;code&gt;56 * 1000 % 55555 = 445&lt;/code&gt; so you'd&#xA;start at row 445 and not 0.&lt;/p&gt;&#xA;&lt;p&gt;Also you HAVE to concatenate, otherwise the pairs (example, label) will be lost.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;i, j ,k % Whaaat?&lt;/h3&gt;&#xA;&lt;p&gt;Run the toy code below to see how the iteration index behaves:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;number_of_examples = 55555&#xA;batch_size = 1000&#xA;number_of_batches = 200&#xA;epoch = 0&#xA;&#xA;for i in range(number_of_batches):&#xA;&#xA;    j = (i - epoch) * batch_size % number_of_examples&#xA;    k = (i - epoch + 1) * batch_size % number_of_examples&#xA;&#xA;    if (k &amp;lt; j):&#xA;        print('Overlap : ', j, number_of_examples)&#xA;        epoch = i + 1&#xA;    else:&#xA;        print(i, '   ', j, k)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25284" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-16T15:12:17.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14568" PostTypeId="1" AcceptedAnswerId="15162" CreationDate="2016-10-16T18:36:36.450" Score="14" ViewCount="10393" Body="&lt;p&gt;At the moment I'm playing with Restricted Boltzmann Machines and since I'm at it I would like try to classify handwritten digits with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model I created is now a quite fancy generative model but I don't know how to go further with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;https://www.quora.com/How-can-the-Restricted-Boltzmann-Machine-be-used-in-classification-problems&quot; rel=&quot;noreferrer&quot;&gt;this article&lt;/a&gt; the author say, that after creating a good generative model, one &quot;&lt;em&gt;then trains a discriminative classifier (i.e., linear classifier, Support Vector Machine) on top of the RBM using the labelled samples&lt;/em&gt;&quot; and furtherly states &quot;&lt;em&gt;since you propagate the data vectors to the hidden units of the RBM model to get hidden unit vectors, or a higher-level representation of the data&lt;/em&gt;&quot;. The problem is that I'm not sure if I get that right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that mean all I have to do is propagate the input to the hidden units and there I have my RBM feature for classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody explain this process to me?&lt;/p&gt;&#xA;" OwnerUserId="25289" LastActivityDate="2018-10-11T03:03:58.630" Title="How to use RBM for classification?" Tags="|classification|rbm|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14571" PostTypeId="2" ParentId="14559" CreationDate="2016-10-17T02:02:06.553" Score="1" Body="&lt;p&gt;For reading data from csv you can use &lt;code&gt;pd.read_csv&lt;/code&gt; or &lt;code&gt;pd.read_table&lt;/code&gt;.&#xA;Example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;data = pd.read_csv(&amp;quot;file_name&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25301" LastEditorUserId="97080" LastEditDate="2023-01-17T17:58:31.503" LastActivityDate="2023-01-17T17:58:31.503" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14572" PostTypeId="2" ParentId="14346" CreationDate="2016-10-17T06:06:32.523" Score="1" Body="&lt;p&gt;Reinforcement learning is the intersection of machine learning, decisions &amp;amp; control, and behavioral psychology. The intersection can be approached from all the three sides. Let me give you a short description from every angle-&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Machine Learning&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;From the ML perspective, RL is the paradigm of learning to control. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about how you learned to cycle or play a sport. These learning tasks are not supervised - no one tells you the correct move to make in a board position, or exactly the amount of angle to lean sideways to balance the cycle. They are also not completely unsupervised since some feedback is observed - whether you won or lost the game after a sequence of moves, how frequently do you fall from cycle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Thus, RL is learning to make good decisions from partial evaluative feedback.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Control &amp; Decision theory&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In control theory (and AI planning), perfect knowledge about the world is assumed, and the objective is to find the best way to behave. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for many problems knowledge about the world is not perfect. Hence, exploring the world could increase our knowledge and eventually help us make better decisions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;RL is balancing the exploration-exploitation trade-off in sequential decision-making problems.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Behavioral Psychology&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The simplified goal of behavioral psychology is to explain &lt;strong&gt;&lt;em&gt;why, when&lt;/em&gt;&lt;/strong&gt;, and &lt;strong&gt;&lt;em&gt;how&lt;/em&gt;&lt;/strong&gt; humans make decisions. We consider humans as rational agents, and hence psychology is also to some extent trying to explain rational behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can study the biological principles of how opinions are formed, which have close connections to temporal difference learning and eligibility traces. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;RL is the paradigm to explain how humans form opinions and learn to make good decisions with experience.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This was a short description from every important perspective. For a detail description, kindly go through these --&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://reinforcementlearning.ai-depot.com/&quot; rel=&quot;nofollow&quot;&gt;Reinforcement Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://karpathy.github.io/2016/05/31/rl/&quot; rel=&quot;nofollow&quot;&gt;Karpathy - Github RL&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Bayes/pomdp.html&quot; rel=&quot;nofollow&quot;&gt;CS- UBC : RL&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps! &lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="836" LastEditDate="2016-10-17T09:18:22.117" LastActivityDate="2016-10-17T09:18:22.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14574" PostTypeId="1" CreationDate="2016-10-17T08:45:51.210" Score="0" ViewCount="29" Body="&lt;p&gt;I have a data table called td and it contains 14k obs and 5k variables.&#xA;When I run the command &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;td[,fv_mc2:=1]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also get the whole data table as output in my console, which floods my console. Any ideas how to fix it ? (It might also be a setting in R, but I am not aware of it.&lt;/p&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2016-10-17T10:26:46.013" Title="How not to see the output in the console when adding a new column in a table?" Tags="|dataset|data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14575" PostTypeId="1" CreationDate="2016-10-17T09:25:14.800" Score="1" ViewCount="403" Body="&lt;p&gt;I'm having issues with fitting a Random Forest model to a completely new dataset. I'm trying to predict tenancy lengths for current tenants. I have a dataset with tenancy information since 2008, with both tenants that have finished their contract and others which are still in their properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps undertaken so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Remove the tenancies that are still on going. This is to actually get target values for &quot;Tenancy Length&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Find a suitable model to train and test on the finished tenancies. I'm happy with a Random Forest, with MAE of ~35 on train and ~70 on test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Create a new dataframe in python that only has the previously excluded information (tenancies that have not yet finished) in order to apply the .predict method on these. I've also combined the previous 'train' and 'test' into a single train, as there is no further need to split that data up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;df = finished tenancies;&#xA;currentDf = ongoing tenancies&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Select the features and target variable from finished tenancies data&#xA;X, y = df.iloc[:, 5:].values, df.iloc[:, 4].values&#xA;clf = RandomForestRegressor() #with relevant params&#xA;clf.fit(X, y)&#xA;yPred= clf.predict(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This results in a 'new' training set consisting of all the finished tenancies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) I move on and predict my current tenancies:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X2 = currentDF.iloc[:, 5:].values&#xA;y2Pred = clf.predict(X2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While I cannot create a residual plot for this (as I have no previously known information regarding targets), I've noticed that 'Tenancy Length' strongly correlates with 'Tenancy Start', which was not the case in my trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7Zzq1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7Zzq1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qy6Ah.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qy6Ah.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any idea what's going on? Why would something like this be happening? I can't for the life of me figure out if I'm doing something wrong, yet I doubt it's just coincidence for some reason.&lt;/p&gt;&#xA;" OwnerUserId="24830" LastActivityDate="2016-10-17T09:25:14.800" Title="Sklearn Random Forest Prediction Correlation Issue" Tags="|python|predictive-modeling|random-forest|scikit-learn|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14576" PostTypeId="1" CreationDate="2016-10-17T09:44:22.853" Score="4" ViewCount="957" Body="&lt;p&gt;1&#xA;down vote&#xA;favorite&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I am trying to find the similarity between two images using skimage - SSIM. The code block will be as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from skimage.measure import compare_ssim as ssim&#xA;from skimage import io&#xA;from skimage.transform import resize&#xA;&#xA;a = io.imread(&quot;http://ecx.images-amazon.com/images/I/51PV4Dd8wAL._AC_UL246_SR190,246_.jpg&quot;,as_grey=False,dtype=&quot;float64&quot;)&#xA;b = io.imread(&quot;http://ecx.images-amazon.com/images/I/914ZHE6JolL._UY500_.jpg&quot;,as_grey=False,dtype=&quot;float64&quot;)&#xA;&#xA;a = resize(a,b.shape)&#xA;ssim(a, b,multichannel=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got the similarity score as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.21009350738786017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although both the images are same where one images in slightly oriented , I am getting very really low scores for this comparisons. Am I missing any preprocessing steps here before I compute compare images. If so what are the things I should consider before comparing images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks In Advance !!&lt;/p&gt;&#xA;" OwnerUserId="25308" LastActivityDate="2024-03-03T21:01:40.153" Title="What preprocessing steps to be followed before image comparison?" Tags="|python|scikit-learn|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14577" PostTypeId="2" ParentId="14574" CreationDate="2016-10-17T10:02:42.123" Score="2" Body="&lt;p&gt;R by default returns the updated dataframe to console.You can assign it to variable by avoiding it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;td &amp;lt;- td[,fv_mc2:=1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By this way you can avoid the console of 14k data.&lt;/p&gt;&#xA;" OwnerUserId="25309" LastActivityDate="2016-10-17T10:02:42.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14579" PostTypeId="2" ParentId="9325" CreationDate="2016-10-17T10:30:42.837" Score="0" Body="&lt;p&gt;Sklearn has a method for it using which you can compute confusion matrix for multi class.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cross_validation&#xA;confusion_matrix(original, Predicted)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25309" LastActivityDate="2016-10-17T10:30:42.837" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14581" PostTypeId="1" CreationDate="2016-10-17T11:47:45.340" Score="180" ViewCount="182720" Body="&lt;p&gt;The key difference between a GRU and an LSTM is that a GRU has two gates (&lt;em&gt;reset&lt;/em&gt; and &lt;em&gt;update&lt;/em&gt; gates) whereas an LSTM has three gates (namely &lt;em&gt;input&lt;/em&gt;, &lt;em&gt;output&lt;/em&gt; and &lt;em&gt;forget&lt;/em&gt; gates). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM? &lt;/p&gt;&#xA;" OwnerUserId="23710" LastEditorUserId="34269" LastEditDate="2019-12-08T10:16:28.140" LastActivityDate="2022-08-26T22:08:35.557" Title="When to use GRU over LSTM?" Tags="|neural-network|deep-learning|lstm|gru|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14582" PostTypeId="2" ParentId="14581" CreationDate="2016-10-17T12:13:11.073" Score="15" Body="&lt;p&gt;This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;GRU exposes the complete memory unlike LSTM, so applications which&#xA;that acts as advantage might be helpful. Also, adding onto why to use&#xA;GRU - it is computationally easier than LSTM since it has only 2&#xA;gates and if it's performance is on par with LSTM, then why not?&lt;/li&gt;&#xA;&lt;li&gt;This &lt;a href=&quot;https://arxiv.org/pdf/1412.3555v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; demonstrates excellently with graphs the superiority&#xA;of gated networks over a simple RNN but clearly mentions that it&#xA;cannot conclude which of the either are better. So, if you are&#xA;confused as to which to use as your model, I'd suggest you to train&#xA;both and then get the better of them.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-10-17T13:35:36.567" LastActivityDate="2016-10-17T13:35:36.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14583" PostTypeId="1" AcceptedAnswerId="14584" CreationDate="2016-10-17T12:53:33.997" Score="3" ViewCount="2004" Body="&lt;p&gt;I've used Hyperopt in Python, but I'm looking for a package with similar capabilities in R. Does a package like this exist?&lt;/p&gt;&#xA;" OwnerUserId="21908" LastActivityDate="2019-11-01T18:49:29.873" Title="Is there a Hyperopt equivalent for optimization in R?" Tags="|machine-learning|python|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14584" PostTypeId="2" ParentId="14583" CreationDate="2016-10-17T13:18:19.387" Score="3" Body="&lt;p&gt;Give a try to &lt;code&gt;DEoptim()&lt;/code&gt;. This package might solve your problem. For the documentation and more information visit-&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.jstatsoft.org/v40/i06/paper&quot; rel=&quot;nofollow noreferrer&quot;&gt;Documentation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/DEoptim/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRAN-R : DEoptim()&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-17T13:18:19.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14585" PostTypeId="2" ParentId="14581" CreationDate="2016-10-17T13:33:26.927" Score="119" Body="&lt;p&gt;GRUs and LSTMs utilize different approaches toward gating information to prevent the vanishing gradient problem. Here are the main points comparing the two:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The GRU unit controls the flow of information like the LSTM unit, but without having to use a &lt;em&gt;&lt;strong&gt;memory unit&lt;/strong&gt;&lt;/em&gt;. It just exposes the full hidden content without any control.&lt;/li&gt;&#xA;&lt;li&gt;GRUs are relatively new, and in my experience, their performance is on par with LSTMs,  but computationally &lt;em&gt;&lt;strong&gt;more efficient&lt;/strong&gt;&lt;/em&gt; (&lt;em&gt;as pointed out, they have a less complex structure&lt;/em&gt;). For that reason, we are seeing it being used more and more.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For a detailed description, you can explore this research paper on &lt;a href=&quot;https://arxiv.org/pdf/1412.3555v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;Arxiv&lt;/a&gt;. The paper explains all this brilliantly.&lt;/p&gt;&#xA;&lt;p&gt;You can also explore these blogs for a better idea:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/&quot; rel=&quot;noreferrer&quot;&gt;WildML&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;Colah - Github&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope that helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorDisplayName="user133478" LastEditDate="2022-08-26T22:08:35.557" LastActivityDate="2022-08-26T22:08:35.557" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="14586" PostTypeId="1" CreationDate="2016-10-17T13:41:34.120" Score="1" ViewCount="271" Body="&lt;p&gt;What do you think will be the future of deep learning? A lot of people talk about deep learning and I can see that it provides various possibilities. For instance, &lt;a href=&quot;https://datascience.stackexchange.com/questions/1253/why-are-nlp-and-machine-learning-communities-interested-in-deep-learning&quot;&gt;this question&lt;/a&gt; gives a nice explanation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the future of deep learning? Will it become the main branch of machine learning or will the other branches of machine learning remain relevant as well? And what about different fields of research? Will it be more suitable for certain applications (like natural language processing) and less suitable for some other types of applications? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd love to hear your views on this, both from a practical as from a scientific point of view. Any practical experience or research that can help to resolve this question? More specifically: should we introduce deep learning in our company?&lt;/p&gt;&#xA;" OwnerUserId="18460" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-10-18T10:50:08.153" Title="Future of deep learning (compared to traditional machine learning)" Tags="|machine-learning|nlp|deep-learning|tensorflow|" AnswerCount="1" CommentCount="4" ClosedDate="2016-10-17T18:00:30.173" ContentLicense="CC BY-SA 3.0" />
  <row Id="14587" PostTypeId="2" ParentId="14586" CreationDate="2016-10-17T14:19:34.900" Score="1" Body="&lt;p&gt;That's very interesting topic. So thumbs up from my side. Now coming to the point. Deep learning may be hot now but some variants of it or something new all together may emerge later. Let me point out the reasons I feel deep learning maybe getting old or will get.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;h2&gt;Slow Learner&lt;/h2&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Slow learner as in, it converges to an optimal solution slowly but with GPU acceleration the training speed can be improved dramatically. The slowness is affected by the learning rate.&lt;/p&gt;&#xA;&lt;p&gt;Adjusting the learning rate affects the reliability of the resulting deep neural net.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;h2&gt;Huge Training Data Requirement&lt;/h2&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Deep learning requires very huge training data to achieve good performance. The presence of a huge number of parameters to adjust requires some huge example sets. The fact that deep learning requires such a large number of training examples makes it &lt;em&gt;&lt;strong&gt;dull&lt;/strong&gt;&lt;/em&gt; in some way, despite such huge training set requirement deep neural nets do have error rates of about 10%.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;h2&gt;Overfitting&lt;/h2&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Deep learning tends to overfit easily but with the new dropout algorithm, this problem can be avoided but with some consequences such as an increase in error rates.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;h2&gt;Poor Initial State&lt;/h2&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Deep neural nets have parameters that need to be initialized. The most used method is random initialization, this results in neural nets with very poor initial states. Compare this to a mammalian brain, the brain is born with some rigid instincts such as basic survival behavioral patterns.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;h2&gt;Sensory Data Transformations&lt;/h2&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For example in visual object recognition tasks, images undergo various geometric and photometric transformations that need to be modeled by a recognition system to rectify new image observations. Deep learning as used today does not take such transformations into account, this is one of the reasons why deep neural nets still suffer from relatively high error rates (relative to a human).&lt;/p&gt;&#xA;&lt;p&gt;Some other reason can be the slow transformation, algorithms mayb emerge together in future, strcture and the approaches. These are some pin points from my side. You can also go through these posts, they're also on it and to the point-&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdnuggets.com/2016/01/deep-learning-2016-beyond.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;KDnuggets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.huffingtonpost.in/entry/the-past-and-future-of-de_b_10005550&quot; rel=&quot;nofollow noreferrer&quot;&gt;Huffington Post&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-17T14:19:34.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14588" PostTypeId="1" AcceptedAnswerId="14598" CreationDate="2016-10-17T14:23:21.787" Score="2" ViewCount="166" Body="&lt;p&gt;I want to check how correlated two time series are, but they don't have the same cardinality. They have different number of data points because the timestamps the data are collected are different. &lt;a href=&quot;https://stackoverflow.com/questions/40050436/how-to-interpolate-and-check-correlation-of-rdds-with-differing-cardinality-in-s&quot;&gt;Available libraries I have found&lt;/a&gt; require that the cardinality be the same. Therefore, I would like to ask if there is a library, an algorithm I can implement myself, or some advice you can give me to approach the problem. Thank you &lt;/p&gt;&#xA;" OwnerUserId="13625" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-18T01:57:56.473" Title="How to interpolate and check correlation of two time series with differing cardinality" Tags="|time-series|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14590" PostTypeId="1" AcceptedAnswerId="15172" CreationDate="2016-10-17T14:59:40.163" Score="3" ViewCount="462" Body="&lt;p&gt;I'm doing an online course and one of the assignment says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To avoid parameters from exploding or becoming highly correlated, it is helpful to augment&#xA;  our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without&#xA;  constraining their direction, and often leads to classifiers with better generalization ability.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Exactly what does &quot;augment our cost function with a Gaussian prior&quot; mean, and how could I go about doing that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried searching on google but didn't find anything helpful.&lt;/p&gt;&#xA;" OwnerUserId="25321" LastActivityDate="2017-08-10T15:29:19.513" Title="Augmenting the Cost Function with a Gaussian Prior" Tags="|machine-learning|statistics|rnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14591" PostTypeId="2" ParentId="14590" CreationDate="2016-10-17T15:09:16.010" Score="-1" Body="&lt;p&gt;It was hard to look for anything based on the information. Still, I found this interesting paper- &lt;a href=&quot;https://papers.nips.cc/paper/3324-augmented-functional-time-series-representation-and-forecasting-with-gaussian-processes.pdf&quot; rel=&quot;nofollow&quot;&gt;Augmented Functional Time Series Representation and Forecasting with Gaussian Processes&lt;/a&gt; on the web.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just explore it. Hope it will help you upto some extent. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-17T15:09:16.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14596" PostTypeId="1" CreationDate="2016-10-17T17:30:04.263" Score="1" ViewCount="40" Body="&lt;p&gt;Given a sequence of events, I am trying to predict a label for events and I hypothesize that the last time this event occurred with a label $i$ has a bearing on when the next time the label $i$ appears. Is it possible to include a covariate which depends on the label? For example my probability for predicting label $i$ would be $P(x=i|y^i)$ and that for predicting label $j$ would be $P(x=j|y^j)$. This would violate the assumption that $y$ are the independent variables in the model.&lt;/p&gt;&#xA;" OwnerUserId="13549" LastActivityDate="2016-10-17T17:30:04.263" Title="Label Dependent Covariate in Logistic Regression" Tags="|machine-learning|logistic-regression|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14597" PostTypeId="2" ParentId="10143" CreationDate="2016-10-17T17:46:23.747" Score="4" Body="&lt;p&gt;I don't think there are best practices in this developing area yet but, in addition to cookiecutter, there were some interesting ideas shown in a &lt;a href=&quot;http://isaacslavitt.com/2016/07/20/data-science-is-software-talk/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; at the SciPy 2016 conference.&lt;/p&gt;&#xA;&lt;p&gt;Personally, I try to minimize the number of sub-folders in a project unless I have a really good way to distinguish them and have good reason to keep them separate. Bad organization is nearly as bad as no organization. I think best practice might depend on use case -- not every project needs the same amount of boilerplate.&lt;/p&gt;&#xA;" OwnerUserId="13977" LastEditorUserId="98307" LastEditDate="2020-08-03T15:11:48.000" LastActivityDate="2020-08-03T15:11:48.000" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14598" PostTypeId="2" ParentId="14588" CreationDate="2016-10-18T01:57:56.473" Score="0" Body="&lt;p&gt;I understand that your time series are unevenly spaced. In this case, why not simply use a library like &lt;a href=&quot;http://traces.readthedocs.io/en/latest/&quot; rel=&quot;nofollow&quot;&gt;traces&lt;/a&gt; and transform them to &lt;a href=&quot;http://traces.readthedocs.io/en/latest/examples.html#transform-to-evenly-spaced&quot; rel=&quot;nofollow&quot;&gt;evenly spaced time series&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-10-18T01:57:56.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14599" PostTypeId="1" AcceptedAnswerId="14601" CreationDate="2016-10-18T02:48:46.287" Score="1" ViewCount="1315" Body="&lt;p&gt;When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations. &#xA;Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow&quot;&gt;the inertia_ after KMeans fitting&lt;/a&gt; my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.&lt;/p&gt;&#xA;" OwnerUserId="25213" LastActivityDate="2016-10-18T03:40:17.277" Title="How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans" Tags="|scikit-learn|k-means|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14601" PostTypeId="2" ParentId="14599" CreationDate="2016-10-18T03:40:17.277" Score="1" Body="&lt;p&gt;It sounds like you are grappling with large data set sizes, for which I first suggest switching to &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html#sklearn.cluster.MiniBatchKMeans&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;mini-batch k-means&lt;/code&gt;&lt;/a&gt;. Mini-batch scales better so will be less frustrating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding apriori estimates of the &lt;code&gt;inertia_&lt;/code&gt;, I suggest using a sample data set to approximate the &lt;code&gt;inertia_&lt;/code&gt; with appropriate margins of error. But, mini-batch may just preclude your need for apriori &lt;code&gt;inertia_&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2016-10-18T03:40:17.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14602" PostTypeId="1" CreationDate="2016-10-18T05:03:05.973" Score="0" ViewCount="500" Body="&lt;p&gt;I used different models to train on the test set and to predict on the test set. The commonality is that all models underestimate the true values on the test set. Which steps should I take to understand why?&lt;/p&gt;&#xA;" OwnerUserId="25337" LastActivityDate="2017-01-16T07:31:50.137" Title="Models systematically underestimate values on the test set, why?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14603" PostTypeId="2" ParentId="14396" CreationDate="2016-10-18T05:11:11.957" Score="3" Body="&lt;p&gt;Hastie et al is at the mathematical level you require - being written by statistics academics with strong mathematical pedigree (Hastie is currently a mathematics professor, for example) - and the complete text is available for free online via the authors' website. It is probably about the best general survey of machine learning for people with mathematical and statistical background at the graduate student level. That said, it is still a survey, and individual topics will require follow up elsewhere, though useful recommended reading is provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bishop also assumes a reasonable degree of mathematical maturity, although the table of contents may make the content appear more simple than it is, for example by listing a review of probability distributions including the Gaussian as Chapter 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Russel &amp;amp; Norvig isn't about machine learning or data science, but rather the wider field of artificial intelligence, in which it includes machine learning as smallish subset, and data science effectively not at all. For example, it discusses a number of different kinds of systems of pre-programmed AI approaches - the exact opposite of machine learning. It is interesting if you want to understand the wider world of automation but will do little to help you understand ML.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="18843" LastEditDate="2016-10-18T10:39:53.893" LastActivityDate="2016-10-18T10:39:53.893" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14605" PostTypeId="2" ParentId="14602" CreationDate="2016-10-18T06:11:56.770" Score="1" Body="&lt;p&gt;Welcome to data science SO. It seems your model has a bias towards a lower estimate of the true values. This can happen for a variety of reasons. Say you are trying to fit a straight line on data that was generated by a second degree polynomial. The straight line fit will have a bias. Typically bias happens because the model you are trying to build is insufficient able to capture the dynamics of the data you are trying to fit it on. I understand this is very generic but without more info about what model you used or any plot of the test set it is hard to help you more.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-18T06:11:56.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14606" PostTypeId="2" ParentId="14493" CreationDate="2016-10-18T06:44:36.377" Score="4" Body="&lt;p&gt;Both Spark and Flink are designed to process data in batch or stream over distributed environment. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Flink primarily being defined as its ability to process streaming data in real time and being considered as good option for processing low data latency data and high fault tolerance on distributed systems on a large scale.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;However, Spark is utilized predominantly in the case of batch processing over distributed environment.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Spark do have streaming API on the other hand but Flink is getting more traction for executing this need.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Flink can be utilized on local JVM,  standalone, Yarn and cloud similarly Spark can be deployed over local, standalone, Yarn, Mesos.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are many more difference between these two. I found some really interesting answers for your question on Quora, have a look-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-are-the-differences-between-Apache-Spark-and-Apache-Flink&quot; rel=&quot;nofollow&quot;&gt;Differences between Apache Spark and Apache Flink - 1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-Apache-Flink-and-Apache-Spark&quot; rel=&quot;nofollow&quot;&gt;Difference between Apache Flink and Apache Spark - 2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-is-difference-between-Hadoop-spark-flink&quot; rel=&quot;nofollow&quot;&gt;Difference between Hadoop, spark, flink&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="10902" LastEditDate="2016-10-18T12:36:51.390" LastActivityDate="2016-10-18T12:36:51.390" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14607" PostTypeId="2" ParentId="11854" CreationDate="2016-10-18T10:01:29.187" Score="1" Body="&lt;p&gt;@sebastianspiegel is correct and you should accept their answer. That said, there's something to be added in terms of how you might want to practically go about achieving this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most NMF libraries give you the option of providing a pre-trained factor/basis matrix, as well as some flag to indicate that you want the algorithm to omit the update step on that matrix. This allows you to take a pre-trained $T$ matrix, and fit a new $H$ matrix to it based on your new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of nice benefits to this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) This factorisation should be much, much quicker to run because your $X$ matrix is now a $m \times 1$ matrix, rather than $m \times n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The features from your previous factorisation should be the same features in the current one, (i.e. if you're factorising movie ratings, and your first factorisation's fifth feature was &quot;scary movies&quot;, the feature spat out by this procedure should also be &quot;scary movies&quot;)&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-18T10:01:29.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14609" PostTypeId="2" ParentId="12839" CreationDate="2016-10-18T12:34:03.713" Score="1" Body="&lt;p&gt;I am not completely fluent in R either and I remember I had troubles converting the data into transactions as well. What worked for me was this (I had same structure as you):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;read.csv(&quot;C:/Users/ ... /data.csv&quot;, header = TRUE, sep=&quot;;&quot;)&#xA;dat &amp;lt;- split(dat$product, dat$ID)&#xA;trans &amp;lt;- as(dat, &quot;transactions&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The package &lt;code&gt;arules&lt;/code&gt; also contains tool from convert the .csv into transactions &lt;code&gt;read.transactions(...)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As was stated in previous answers, you can play with length, support or confidence. I'd also propose to try change the sides as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#What am I likely to buy BEFORE buying XY Product(rhs=&quot;XY&quot;)&#xA;rulesB &amp;lt;- apriori(data=trans, parameter=list(supp=0.006,conf=0.1),&#xA;                 appearance = list(default=&quot;lhs&quot;,rhs=&quot;22&quot;),&#xA;                 control = list(verbose=F))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#What am I likely to buy AFTER buying XY Product(lhs=&quot;XY&quot;)&#xA;rulesA &amp;lt;- apriori(data=trans, parameter=list(supp=0.006,conf=0.1),&#xA;                 appearance = list(default=&quot;rhs&quot;,lhs=&quot;22&quot;),&#xA;                 control = list(verbose=F))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, there is great introduction to Association rules in R &lt;a href=&quot;http://www.salemmarafi.com/code/market-basket-analysis-with-r/&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2016-10-18T12:34:03.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14610" PostTypeId="1" AcceptedAnswerId="14611" CreationDate="2016-10-18T12:48:37.677" Score="1" ViewCount="718" Body="&lt;p&gt;I trained a word2vec from the gensim package. Even though I pass a word in the model.train() method, it doesnt appear in the model's vocab.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can such a case arise?&lt;/li&gt;&#xA;&lt;li&gt;Why does it happen so?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-18T13:00:06.420" Title="Word2vec - KeyError" Tags="|machine-learning|word2vec|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14611" PostTypeId="2" ParentId="14610" CreationDate="2016-10-18T13:00:06.420" Score="1" Body="&lt;p&gt;The reason behind this is the default value for min_count is 5 in word2vec. Since my words have very less frequency, they are not being added to the vocabulary.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-18T13:00:06.420" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14612" PostTypeId="2" ParentId="11930" CreationDate="2016-10-18T14:22:46.680" Score="7" Body="&lt;p&gt;As far as I understand the issue is the following:&#xA;In image recognition the inputs to your network could be the pixels (grayscale or only 1 and 0 for black and white). If you want to, e.g. recognize handwritten numbers it is very difficult to only work with such values as you never know where exactly the number (i.e. the black values) will be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is pixel 140 black or 142 black? In both cases it could well be a three. In the age/weight example these inputs are well defined. Feature 2 is weight. Feature 3 is age. These &quot;dimensions&quot; shouldnt &quot;hop&quot; in your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: In your picture training the &quot;threes&quot; or &quot;cars&quot; or &quot;houses&quot; have to be recognized independent of their location in the picture, i.e. the pixel values, i.e. the feature/input vector, i.e. the dimensions as opposed to clearly defined inputs such as patient data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you solve this in image recognition? You use additional tricks, e.g. convolution.&lt;/p&gt;&#xA;" OwnerUserId="25349" LastActivityDate="2016-10-18T14:22:46.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14614" PostTypeId="1" AcceptedAnswerId="14623" CreationDate="2016-10-18T16:38:56.227" Score="0" ViewCount="359" Body="&lt;p&gt;Essentially, this is my data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X Class    Sex   Age Survived Freq&#xA;1   1st   Male Child       No    0&#xA;2   2nd   Male Child       No    0&#xA;3   3rd   Male Child       No   35&#xA;4  Crew   Male Child       No    0&#xA;5   1st Female Child       No    0&#xA;6   2nd Female Child       No    0&#xA;7   3rd Female Child       No   17&#xA;8  Crew Female Child       No    0&#xA;9   1st   Male Adult       No  118&#xA;10  2nd   Male Adult       No  154&#xA;11  3rd   Male Adult       No  387&#xA;12  Crew   Male Adult       No  670&#xA;13  1st Female Adult       No    4&#xA;14  2nd Female Adult       No   13&#xA;15  3rd Female Adult       No   89&#xA;16  Crew Female Adult       No    3&#xA;17  1st   Male Child      Yes    5&#xA;18  2nd   Male Child      Yes   11&#xA;19  3rd   Male Child      Yes   13&#xA;20  Crew  Male Child      Yes    0&#xA;21  1st Female Child      Yes    1&#xA;22  2nd Female Child      Yes   13&#xA;23  3rd Female Child      Yes   14&#xA;24  Crew Female Child      Yes    0&#xA;25  1st   Male Adult      Yes   57&#xA;26  2nd   Male Adult      Yes   14&#xA;27  3rd   Male Adult      Yes   75&#xA;28  Crew  Male Adult      Yes  192&#xA;29  1st Female Adult      Yes  140&#xA;30  2nd Female Adult      Yes   80&#xA;31  3rd Female Adult      Yes   76&#xA;32  Crew Female Adult      Yes   20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If there were no frequency but only single valued data, then I know how to invoke &lt;code&gt;rpart&lt;/code&gt; to construct a decision tree for me. How to do it considering the frequency of each class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a beginner in R. Thanks&lt;/p&gt;&#xA;" OwnerUserId="25353" LastActivityDate="2016-10-18T23:23:39.030" Title="How to construct a Decision tree in R where the training data has a frequency associated with each class" Tags="|r|classification|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14615" PostTypeId="1" AcceptedAnswerId="14616" CreationDate="2016-10-18T17:54:14.177" Score="4" ViewCount="3364" Body="&lt;p&gt;I read some blog articles recently. One mentions that you could not imagine high dimensional space as 2d or 3d as distance between any 2 points in high dimensional space tends to be similar, which means 'dense'. However in the &lt;a href=&quot;http://www.cs.toronto.edu/~hinton/absps/tsne.pdf&quot; rel=&quot;nofollow&quot;&gt;t-SNE paper&lt;/a&gt;, it says high dimensional space tends to be sparse such that you have to employ special dimensionality reduction techniques to visualize in 2d or 3d space in meaningful way. So how to reconciliate these 2 different views?&lt;/p&gt;&#xA;" OwnerUserId="25347" LastEditorUserId="8820" LastEditDate="2016-10-19T18:31:39.613" LastActivityDate="2018-06-18T11:22:49.460" Title="High dimensional space is dense or sparse?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14616" PostTypeId="2" ParentId="14615" CreationDate="2016-10-18T18:14:16.027" Score="9" Body="&lt;p&gt;Data in a high dimensional space tends to be sparser than in lower dimensions.  There are various ways to quantify this, but one way of thinking that may help your intuition is to start by imagining points spread uniformly at random in a three dimensional box.  Now flatten the box into a square, pushing two opposite sides together so all the points lie on a single plane.  Do you see that the average distance between a point and its nearest neighbor is now smaller?  Now flatten the square into a line segment.  Do you see that the average distance between a point and its neighbors is now smaller still?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no conflict between this and saying that the average distance between any 2 points in the high dimensional space tends to be similar.  The latter statement doesn't imply density.  The real number line is dense (it has no gaps), and yet the distance between points ranges from 0 to infinity.  The point is that the higher the dimension of your space, the more likely the points are to lie near the edges of the space rather than the center.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, consider the dimensions we can actually see.  Consider a circle with &lt;code&gt;radius=1&lt;/code&gt;, inscribed in a square with sides of &lt;code&gt;length=2&lt;/code&gt;.  The circle occupies $\pi / 4$ of the square's area, about 78.5%.  Now consider a sphere of &lt;code&gt;radius=1&lt;/code&gt; inscribed in a cube with sides of &lt;code&gt;length=2&lt;/code&gt;.  The sphere occupies $\pi / 6$ of the cube's volume, about 52.4%.  As you see in this example, the odds of a randomly placed point lying close to the center (with close to the center in this case meaning within the circle or the sphere) is lower as the dimension increases.  Points are more likely to be in the corners.  This is why in high dimensions the distance between the points tends to be similar - because randomly placed points tend to be close to the edges of the region.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-18T18:14:16.027" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14618" PostTypeId="1" AcceptedAnswerId="14621" CreationDate="2016-10-18T18:49:57.537" Score="1" ViewCount="366" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WsV0W.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WsV0W.png&quot; alt=&quot;distribution of datapoints&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to make a linear regression model for the sale price of a house based on many variables (based on the data from this Kaggle challenge &lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/house-prices-advanced-regression-techniques&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution above is the 2nd-floor size in square feet and the y-axis is the sale price. It shows a clear linearity, except for the fact that homes without a 2nd floor clearly are not for sale for 0 dollars. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have many variables like this that have some threshold either upper or lower that has a large distribution of response for a single value. If I simply exclude these values then the intercept for this curve will be through the origin. Should I let that be the case and assume that the $0 price tag will be corrected by the other variables in my regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to treat/fit data such as this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="25357" LastActivityDate="2016-10-19T05:58:16.153" Title="Multivariate linear regression accounting for threshold / data cleaning" Tags="|data-cleaning|linear-regression|kaggle|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14619" PostTypeId="1" AcceptedAnswerId="14620" CreationDate="2016-10-18T19:24:46.373" Score="2" ViewCount="246" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QLPdJ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QLPdJ.jpg&quot; alt=&quot;Fuel behavior&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hi guys.&#xA;I'm not a data analyst and I need some direction in this. I'm looking for a way to know the events of the fuel during a range of time, could be a day or a month, etc. If the consumption was like the picture above would be easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the data I have is like this picture.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/JaRbb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JaRbb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to take the fueling and the possible thefs, that considering that I have false data, that could go up or down. I can see that those abrupt changes are fake because it always comes to the original value, the main problem is that I don't know how long they are gonna stay fake. Could be just one point or n points of bad points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can I detect these events (fueling and possible theft) whitout counting the errors as events?.&#xA;What algorithms or formulas can I apply here?&lt;/p&gt;&#xA;" OwnerUserId="25352" LastEditorUserId="12384" LastEditDate="2016-10-18T19:52:46.683" LastActivityDate="2016-10-20T05:26:42.813" Title="How can I detect events on fuel tank" Tags="|time-series|data-cleaning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14620" PostTypeId="2" ParentId="14619" CreationDate="2016-10-18T19:56:41.937" Score="3" Body="&lt;p&gt;Welcome to SO! It looks like you have a time series problem. Typically the first step when dealing with time series is to consider the difference. Let us define $f(t)$ as the fuel level at time $t$. You would want to calculate $diff_{\text{fuel}}(t) = f(t) - f(t - 1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this step you will likely see that the spikes that you identify as bad data are outliers. You could detect these by for example looking at all the data below and above your 2.5 percentile or 5 percentile. Typically this requires some careful analysis work to ensure that you do not delete too much. Once you have identified what the best workable percentile is you can could &lt;a href=&quot;https://en.wikipedia.org/wiki/Winsorizing&quot; rel=&quot;nofollow&quot;&gt;Winsorize&lt;/a&gt; your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, you would look at the data points in the bottom of your resulting distribution. These will likely be the points you identify as theft.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-18T19:56:41.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14621" PostTypeId="2" ParentId="14618" CreationDate="2016-10-18T21:10:10.983" Score="0" Body="&lt;p&gt;You could try including &quot;Does not have a 2nd floor&quot; as a separate categorical variable, which you could encode as 1 or 0 in your linear model.  Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;price ~ no_second_floor + area_of_second_floor + [Other Variables]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Event without the other variables, it's already going to be able to fit the data better than &lt;code&gt;price ~ area_of_second_floor&lt;/code&gt; because, instead of forcing the price of the single-floor houses to $0, it would be able to fit it to the average price of all the single-floor houses in your data set - which is the best you can do for those houses until you add other variables besides &lt;code&gt;area_of_second_floor&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Actually, it wouldn't be forced to $0 if you were to include a constant term in your model, but the constant term that best lets you fit the linear portion of the data still probably isn't what you want for the single-floor houses, some of which - as you see - are quite expensive).&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-18T21:10:10.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14622" PostTypeId="2" ParentId="14266" CreationDate="2016-10-18T23:04:35.420" Score="3" Body="&lt;p&gt;In most cases, you will want to run findCorrelation after cleaning. The reason being the cleaning process is either removing unwanted data points (that you deem irrelevant anyway) or computing some transformation that you'll want findCorrelation to be aware of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, that said, I've often run it twice, once before cleaning and once after if my transformation could alter the correlation structure of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fortunately, since you're using caret, it is a simple matter to estimate your models both ways, then compare the ultimate differences in accuracy.  Unfortunately, the correct answer to many questions in ML is 'do it both ways then compare' &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2016-10-18T23:04:35.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14623" PostTypeId="2" ParentId="14614" CreationDate="2016-10-18T23:23:39.030" Score="1" Body="&lt;p&gt;To use the column named &lt;code&gt;Freq&lt;/code&gt; as your case weights, you can call &lt;code&gt;rpart&lt;/code&gt; with the argument &lt;code&gt;weights=Freq&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-18T23:23:39.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14624" PostTypeId="1" AcceptedAnswerId="14633" CreationDate="2016-10-19T02:59:11.717" Score="2" ViewCount="4559" Body="&lt;p&gt;Does anyone know how the python sklearn random forest implementation handles continuous variables in the fitting process? I'm curious to know if it does any sort of binning (and if so, how it does the binning), or if a continuous variable is just treated as a categorical variable? I'm hoping it's not the latter...thanks! Also, I'd be open to using some R implementation if anyone knows about that.&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-10-19T10:54:48.607" Title="sklearn random forest and fitting with continuous features" Tags="|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14626" PostTypeId="2" ParentId="14618" CreationDate="2016-10-19T05:58:16.153" Score="0" Body="&lt;p&gt;Every regression analysis should include a residual analysis as a further check on the adequacy of the chosen regression model. &#xA;A plot of all residuals on the y-axis vs. the predicted values on the x-axis, called &lt;strong&gt;a residual vs. ﬁt plot&lt;/strong&gt;, is a good way to check the linearity and equal variance assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question, I believe if you did this EDA (Exploratory Data Analysis) you would find that including your single floor data does indeed violate the linearity that you have to assume if you are making a linear regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get around your intercept at $0 dollars issue, try subtracting the mean of your two story houses from each data point (centering the data ).  This should give you  meaningful results at the intercept.  Your new intercept should be the mean of your two story houses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you do your EDA with the residuals, if you find your data is not linear try doing some transformation of the variables to get you to a linear data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used the following as a reference to answer your question.  I encourage you to check it out: especially the section on transforming variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.cmu.edu/~hseltman/309/Book/chapter9.pdf&quot; rel=&quot;nofollow&quot;&gt;Chapter 9 Simple Linear Regression - CMU Statistics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-19T05:58:16.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14627" PostTypeId="1" CreationDate="2016-10-19T06:45:22.967" Score="-1" ViewCount="95" Body="&lt;p&gt;I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is Cloudera the best environment? If not what is the alternative?&lt;/li&gt;&#xA;&lt;li&gt;My next step is to connect tableau to the hadoop environment and visualize it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Kindly help me out.&lt;/p&gt;&#xA;" OwnerUserId="25366" LastEditorUserId="29169" LastEditDate="2019-06-10T18:31:40.537" LastActivityDate="2019-06-10T18:31:40.537" Title="Connecting Twitter API to a Big Data Environment?" Tags="|bigdata|visualization|apache-hadoop|tableau|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="14628" PostTypeId="1" AcceptedAnswerId="14635" CreationDate="2016-10-19T06:54:26.097" Score="0" ViewCount="575" Body="&lt;p&gt;I have successfully built a logistic regression prediction model based on data set that is complete and clean, i.e., there is no missing values and the data is consistent.&#xA;Now, for deploying the model and testing it for online use, there is missing values in the inputs, i.e., not all inputs are available to predict the target value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way to deal with this?&lt;/p&gt;&#xA;" OwnerUserId="4968" LastActivityDate="2016-10-19T23:39:27.740" Title="Deploying the prediction model under missing values for test data" Tags="|machine-learning|predictive-modeling|missing-data|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14629" PostTypeId="2" ParentId="14628" CreationDate="2016-10-19T07:16:08.607" Score="1" Body="&lt;p&gt;In statistics coping with missing values is often done by imputation: &lt;a href=&quot;https://en.wikipedia.org/wiki/Imputation_(statistics)&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Imputation_(statistics)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and whole books have been written about it. Suggest you start reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One method, multiple imputation, works by creating a number of new complete data items by replacing the missing value with some values sampled from a distribution. You then predict from these new data items, and that gives you a set of predictions and variances from which you can compute a pooled prediction and variance. This variance will be larger than that from a complete item because of the variance introduced by the sampling. The increase will depend on how influential the missing variable is to the model and what distribution you put on the missing item. For example, if you have missing age, and your data should be from a population between 16 and 60, you'd sample age from the population distribution a number of times, do the predictions, and pool them according to the multiple imputation methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course you have to know if your missing data are missing at random, or perhaps biased missing (maybe more women over 40 don't give their age). Lots of interesting complications that will only come to light if you have a careful think about your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, as I say, whole books. And you should probably try the statistics stack exchange site too. Its not really data science much.&lt;/p&gt;&#xA;" OwnerUserId="471" LastEditorUserId="471" LastEditDate="2016-10-19T07:24:20.333" LastActivityDate="2016-10-19T07:24:20.333" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14630" PostTypeId="2" ParentId="14627" CreationDate="2016-10-19T07:22:12.573" Score="0" Body="&lt;p&gt;There are three main contenders &lt;strong&gt;Cloudera, MapR&lt;/strong&gt;, and &lt;strong&gt;Hortonworks&lt;/strong&gt;. But among all these 3 Cloudera and MapR are better solutions if money is not an issue and you'd like to get some advanced features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cloudera has &lt;strong&gt;Impala&lt;/strong&gt; which may be a killer feature for some. MapR has some unique HA attributes and uses a &lt;strong&gt;NFS compatible filesystem&lt;/strong&gt;. MapR is supposedly the fastest Hadoop solution in the market. You can find more information on this on &lt;a href=&quot;https://www.quora.com/Which-is-the-best-distribution-of-Hadoop-Is-Cloudera-the-clear-leader-in-this-space&quot; rel=&quot;nofollow&quot;&gt;Quora&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for your 2nd question, go through these articles.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.predie.com/2016/02/21/tableau-on-top-of-hadoop/&quot; rel=&quot;nofollow&quot;&gt;Tableau on Top of Hadoop&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://kb.tableau.com/articles/knowledgebase/hadoop-hive-connection&quot; rel=&quot;nofollow&quot;&gt;Connecting to Hadoop Hive&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://kb.tableau.com/articles/knowledgebase/administering-hadoop-hive&quot; rel=&quot;nofollow&quot;&gt;Administering Hadoop and Hive for Tableau Connectivity&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-19T07:22:12.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14631" PostTypeId="2" ParentId="14628" CreationDate="2016-10-19T07:29:11.537" Score="1" Body="&lt;pre&gt;&lt;code&gt;Missing values need to be treated, you can remove missing if are very  &#xA;small &amp;lt; 10% or you have large dataset. Some  statistical software like SAS will &#xA;exclude missing values if is not imputed/treated. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also you can't predict anything  with &lt;strong&gt;NA&lt;/strong&gt;: usually missing or coded data can be meaningful or mindless so you need to find out what you need to do with the missing data. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In Conclusion: missing data should be fixed because in case if is coded than you will have wrong predictions or  statistical software will drop from dataset/model automatically. Assuming that you already build this logistic model than you know how you treated the data for model building and next step if figure out if you prepare the data to include imputed missing data for model or drop from your dataset/input pipeline. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2016-10-19T07:29:11.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14633" PostTypeId="2" ParentId="14624" CreationDate="2016-10-19T10:54:48.607" Score="2" Body="&lt;p&gt;To understand how a random forest treats continuous data it is imperative to understand how a random forest works. At the base of the random forest algorithm lays a tree construction. The default in sklearn is to split a tree based on the Gini coefficient (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow&quot;&gt;sklearn documentation&lt;/a&gt;). This type of tree algorithm is referred to as CART trees. You can change the &lt;code&gt;criterion&lt;/code&gt; to &lt;code&gt;entropy&lt;/code&gt; to select ID3 and C4.5 trees. Without going to deep into the maths, the tree algorithm will seek to split the tree based on a cutoff that leads to the lowest Gini coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The random forest algorithm will build a large number of deep trees on your data and average over all the trained trees to give you the final prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your requirements in terms of data size and necessity for parallelization I can highly recommend H2O. It is an open source machine learning software suite with APIs in Python and R. Their random forest implementation is very fast and leads to models with a higher AUC (see &lt;a href=&quot;https://github.com/szilard/benchm-ml&quot; rel=&quot;nofollow&quot;&gt;this page&lt;/a&gt; for a good comparison between different ML libraries).&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-19T10:54:48.607" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14635" PostTypeId="2" ParentId="14628" CreationDate="2016-10-19T14:17:59.870" Score="2" Body="&lt;p&gt;I can think of three ways to deal with the problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Treat &quot;missing value&quot; as another feature&lt;/strong&gt;: Imagine you have a feature like &quot;date of graduation&quot;. One possible (likely?) reason why this value is missing might be that the person did not graduate. So you could build a model which as a binary feature &quot;graduation date is available&quot; and the actual graduation date as another feature.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Predict the missing values&lt;/strong&gt;: If data is missing because of your lack of knowledge of it (in contrast to the first point), then you might think about trying to predict the missing value. You could  also add a feature which encodes the certainty of the predicted value being correct.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Skip the feature&lt;/strong&gt;: If it is missing very often and if it doesn't add much value to your prediction, you might simply want to remove it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-10-19T18:36:22.727" LastActivityDate="2016-10-19T18:36:22.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14636" PostTypeId="1" CreationDate="2016-10-19T14:18:52.377" Score="0" ViewCount="146" Body="&lt;p&gt;Much of the vocabulary here is new to me so forgive me if I misspeak. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm attempting to find an approach to a very simple classification problem. I have a set of description such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;House&lt;/strong&gt; : Building with several walls some windows and a roof &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Car&lt;/strong&gt; : Vehicle with four wheels, a steering wheel, and an engine&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a third party giving a description of what they:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My House&lt;/strong&gt; : Red walls, a balcony, a roof and a great view&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give these loose natural language descriptions what would be a good approach to classifying &quot;My House&quot; as a &quot;House&quot;. Would a simple word counting approach be applicable, perhaps with some sort of emitting dictionary {&quot;the&quot;, &quot;a&quot;, etc}. Any alternative algorithms to consider? &lt;/p&gt;&#xA;" OwnerUserId="25383" LastActivityDate="2017-01-17T17:42:04.383" Title="Approach to Text Classification Problem" Tags="|classification|text-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14637" PostTypeId="2" ParentId="14636" CreationDate="2016-10-19T15:54:05.000" Score="3" Body="&lt;p&gt;It looks like a classical email classification problem: Spam or Ham.&#xA;In the following I will use some specific slang, do not worry, look it up in Google, if you will see specific terms you do not know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prerequisite: ideally your dataset will be &quot;balanced&quot;, having, say, 50% cars and 50% houses, could be 40/60 as well. Problems will arise, when one class will be say &amp;lt; 10%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So called &quot;bag of words&quot; can be seen as a start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your dataset consist of two columns, say, &quot;description&quot; and &quot;label&quot;&lt;/li&gt;&#xA;&lt;li&gt;Label has only two values car or house&lt;/li&gt;&#xA;&lt;li&gt;Tokenize your decription column (single words)&lt;/li&gt;&#xA;&lt;li&gt;Cleanse the desc column: remove stop words (like and, the , a - as they do not have any value), remove punctuation, possibly stem the words, remove numbers&lt;/li&gt;&#xA;&lt;li&gt;calculate document-term-matrix, possibly use TF-IDF&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;By the end of this operation you will have a dataset with a label column (car or house) and long number of word (or even n-gram, e.g. &quot;great_view&quot; is a bi-gram) columns containing binary values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Label; vehicle, balcony, wheel, ..., ...., great_view&#xA;Car,   1,        0,      1, ....., ...., ..., 0&#xA;House, 0,        1,      0, ..., ...,         1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then use naive bayes or logistic regression as a start to train your model.&#xA;Pre-process every new description as above and use your trained model to assign a probability to be a &quot;car&quot; or a &quot;house&quot;, check the confusion matrix, maybe adjust the threshold .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything I described can be done e.g. in R or Python.&#xA;In R use the text mining package &quot;tm&quot;. &lt;/p&gt;&#xA;" OwnerUserId="25337" LastActivityDate="2016-10-19T15:54:05.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14638" PostTypeId="2" ParentId="14636" CreationDate="2016-10-19T15:59:25.680" Score="0" Body="&lt;p&gt;I think that word counting is a good start.  I believe you can adapt spam filtering techniques to your task. i.e If a spam filter finds a group of words is spam or not spam, your filter would classify a group of words as a description of a house and not car or a car and not a house. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Wikipedia:&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&quot; rel=&quot;nofollow&quot;&gt;Naive Bayes spam filtering&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Naive Bayes classifiers are a popular statistical technique of e-mail filtering. They typically use bag of words features to identify spam e-mail, an approach commonly used in text classification.&#xA;Naive Bayes classifiers work by correlating the use of tokens (typically words, or sometimes other things), with spam and non-spam e-mails and then using Bayes' theorem to calculate a probability that an email is or is not spam.&#xA;Naive Bayes spam filtering is a baseline technique for dealing with spam that can tailor itself to the email needs of individual users and give low false positive spam detection rates that are generally acceptable to users. It is one of the oldest ways of doing spam filtering, with roots in the 1990s.&quot;&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-19T15:59:25.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14639" PostTypeId="1" CreationDate="2016-10-19T17:10:11.757" Score="12" ViewCount="9602" Body="&lt;p&gt;I have a generic matrix A which is symmetric, positive definite and sparsely populated (it's also quite big, say composed of tens to hundreds thousands rows). I would like to have a neural network learn how to find a matrix B that resembles the inverse of A (ie: that multiplied by A will yeld a minimal spread of the spectral radius) as accurately and as fast as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there are some AI branches that are focused on this kind of problems or if not how could I approach it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the classical algorithms used to do this, but I would just like to have a look at the possibilities offered by machine learning on the subject.&lt;/p&gt;&#xA;" OwnerUserId="25394" LastActivityDate="2016-10-19T21:18:52.683" Title="Finding the inverse of a matrix with neural networks" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14641" PostTypeId="2" ParentId="14639" CreationDate="2016-10-19T17:46:18.910" Score="10" Body="&lt;p&gt;In general this problem falls under the umbrella of &quot;structured prediction&quot; since you are trying to estimate a number of things that are related by virtue of being embedded in a PSD matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of estimating the inverse in one swoop, I'd pick an appropriate algorithm, say inversion by eigendecomposition, estimate the components (eigenvalues and eigenvectors), then piece them together. Here's one paper that shows how: &lt;a href=&quot;https://www.researchgate.net/profile/Zhang_Yi32/publication/222949356_Neural_networks_based_approach_for_computing_eigenvectors_and_eigenvalues_of_symmetric_matrix/links/5523b8350cf27b5dc37962fc.pdf&quot; rel=&quot;noreferrer&quot;&gt;Neural networks based approach for computing eigenvectors and eigenvalues of symmetric matrix&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can investigate approximation algorithms such as the one elaborated in &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.48.6081&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;noreferrer&quot;&gt;Approximating the inverse of a symmetric positive definite matrix&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Welcome to DataScience.SE&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-10-19T21:18:52.683" LastActivityDate="2016-10-19T21:18:52.683" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14644" PostTypeId="2" ParentId="14615" CreationDate="2016-10-19T20:32:39.820" Score="1" Body="&lt;p&gt;I would like to elaborate a bit on the fact that data in high dimensional spaces is sparser.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, we think of euclidean spaces. This means if we have points $p_1, p_2 \in \mathbb{R}^n$ we say their distance is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d(p_1, p_2) = \sqrt{\sum_{i=1}^n {\left (p_1^{(i)} - p_2^{(i)} \right )}^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we take the square root of the sum of the component-wise squared difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now think of two points in a unit hypercube in $\mathbb{R}^n$ (hence $[0, 1]^n$). The maximum distance two points can have in this hypercube is when they are on the diagonal. Hence the distance is $\sqrt{n}$, meaning the higher the dimension is, the farer away can points be in the unit hypercube.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://math.stackexchange.com/q/1976250/6876&quot;&gt;What is the maximum distance of k points in an n-dimensional hypercube?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Martin Thoma: &lt;a href=&quot;https://martin-thoma.com/average-distance-of-points/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Curse of Dimensionality&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-06-18T11:22:49.460" LastActivityDate="2018-06-18T11:22:49.460" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="14645" PostTypeId="1" AcceptedAnswerId="14646" CreationDate="2016-10-19T21:22:43.257" Score="22" ViewCount="91652" Body="&lt;p&gt;I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately convert it to a column of datetime data by adding the timestamp column series to a series that consists entirely of datetime values for 1970-1-1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to convert a series of &lt;em&gt;strings&lt;/em&gt; to datetime data (pandas.to_datetime), but I can't find or come up with any solution to convert the entire column of &lt;em&gt;ints&lt;/em&gt; to datetime data OR to timestamp data.&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2022-03-05T00:14:36.223" Title="Convert a pandas column of int to timestamp datatype" Tags="|python|time-series|data-cleaning|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14646" PostTypeId="2" ParentId="14645" CreationDate="2016-10-19T21:41:57.990" Score="29" Body="&lt;p&gt;You can specify the unit of a &lt;a href=&quot;https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;pandas.to_datetime&lt;/code&gt;&lt;/a&gt; call.&lt;/p&gt;&#xA;&lt;p&gt;Stolen from &lt;a href=&quot;https://stackoverflow.com/questions/19231871/convert-unix-time-to-readable-date-in-pandas-dataframe&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# assuming `df` is your data frame and `date` is your column of timestamps&#xA;&#xA;df['date'] = pandas.to_datetime(df['date'], unit='s')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Should work with integer datatypes, which makes sense if the unit is seconds since the epoch.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastEditorUserId="112944" LastEditDate="2021-11-01T01:26:24.123" LastActivityDate="2021-11-01T01:26:24.123" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="14647" PostTypeId="2" ParentId="12885" CreationDate="2016-10-19T23:15:59.383" Score="1" Body="&lt;p&gt;Try Normalizing the trails and active accounts data with a mean or median.  The resulting statistics will show week 3 as negative values which can be interpreted as a comparison to all the data.  See the Excel graphs below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cij1S.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cij1S.png&quot; alt=&quot;mean normal&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dqESC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dqESC.png&quot; alt=&quot;mean median combo&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As opposed to the data that has not been normalized.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/RKCrt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RKCrt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-19T23:15:59.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14648" PostTypeId="1" CreationDate="2016-10-19T23:22:22.527" Score="14" ViewCount="57388" Body="&lt;p&gt;Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example for the pyspark dataframe:&#xA;$$&#xA;\begin{array}{c|lcr}&#xA;&amp;amp; \text{c1} &amp;amp; \text{c2} &amp;amp; \text{c3} \\&#xA;\hline&#xA;1 &amp;amp; 0.04 &amp;amp; 1 &amp;amp; 1.35 \\&#xA;2 &amp;amp; -1 &amp;amp; null &amp;amp; -1.2 \\&#xA;3 &amp;amp; null &amp;amp; 1.2 &amp;amp; null&#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{array}{c|lcr}&#xA;&amp;amp; \text{c1} &amp;amp; \text{c2} &amp;amp; \text{c3} \\&#xA;\hline&#xA;1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 \\&#xA;2 &amp;amp; 1 &amp;amp; null &amp;amp; 1 \\&#xA;3 &amp;amp; null &amp;amp; 1 &amp;amp; null&#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="25406" LastEditorUserId="25406" LastEditDate="2016-10-20T08:02:32.023" LastActivityDate="2018-04-17T01:49:01.537" Title="Replace all numeric values in a pyspark dataframe by a constant value" Tags="|python|apache-spark|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14649" PostTypeId="2" ParentId="14628" CreationDate="2016-10-19T23:39:27.740" Score="0" Body="&lt;p&gt;So that it is close as possible to the real data, go back an rebuild your logistic regression model and simulate the missing values you are finding in the real data.  After all, the model is supposed to represent the real circumstance. This approach has a few advantages over the imputing approaches.  The foremost is that you can accurately gauge which and how much missing data your model can stand up to.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-19T23:39:27.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14650" PostTypeId="2" ParentId="9588" CreationDate="2016-10-20T02:25:42.330" Score="4" Body="&lt;p&gt;You could try the following,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testPassengerID = test.select('PassengerID').rdd&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this would select the column &lt;code&gt;PassengerID&lt;/code&gt; and convert it into a &lt;code&gt;rdd&lt;/code&gt; &lt;/p&gt;&#xA;" OwnerUserId="25409" LastEditorUserId="12384" LastEditDate="2016-10-20T09:24:51.650" LastActivityDate="2016-10-20T09:24:51.650" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14651" PostTypeId="2" ParentId="14619" CreationDate="2016-10-20T05:26:42.813" Score="1" Body="&lt;p&gt;So as my comment suggested, I think there is a relatively easy way to solve this.  That is with a Fourier transform.  As I now recall, removing part of a time based signal is easy when representing the signal in the frequency domain.  Because all that is necessary is to apply a filter to the signal to remove unwanted components of the signal.  In your case those spikes (abrupt changes) will look like high frequency signals in the frequency domain.  So in theory, you should be able to apply a low pass filter to the transformed data to get rid of the spikes ( fakes ), then transform the filtered signal back into the time domain. Then just like that you have gotten rid of all those abrupt changes in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To reiterate.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Transform existing signal (your data) into the frequency domain, using some&#xA;variation of the Fourier transform.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply a low pass filter to the transformed signal (your transformed&#xA;data).  Note you will need to pick a suitable cutoff frequency for&#xA;your low pass filter.  This should be relatively simple as most of&#xA;your data is changing very slowly. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply the inverse of the Fourier transform to your transformed data&#xA;    to bring it back to the time domain.  Your spike should be gone in&#xA;    your data now.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is a very simple process and should only take about 3~4 lines of code in any language with a DSP library.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-20T05:26:42.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14652" PostTypeId="2" ParentId="14648" CreationDate="2016-10-20T07:00:41.413" Score="8" Body="&lt;p&gt;As per your problem, I think it might be easier to use &lt;strong&gt;lit&lt;/strong&gt;. Try this-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql.functions import lit&#xA;new_df = df.withColumn('column_name', lit(1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-20T07:00:41.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14654" PostTypeId="2" ParentId="8278" CreationDate="2016-10-20T07:44:49.913" Score="2" Body="&lt;p&gt;Here is a simple procedure you can use:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;learn a classifier to distinguish between train/test data (using regular X features)&lt;/li&gt;&#xA;&lt;li&gt;compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Matthews_correlation_coefficient&quot; rel=&quot;nofollow&quot;&gt;phi correlation coefficient&lt;/a&gt; to estimate the quality of the classifier = the separability of the train/test data&lt;/li&gt;&#xA;&lt;li&gt;set a threshold (e.g. .2) above which you can claim there is a covariate shift (and start looking as corrections)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="6619" LastActivityDate="2016-10-20T07:44:49.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14655" PostTypeId="1" AcceptedAnswerId="14657" CreationDate="2016-10-20T07:56:19.977" Score="1" ViewCount="918" Body="&lt;p&gt;We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to the same event (data from light sensor for instance). I know about boosting methods but they don't give result we want to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any known method to do this ? &lt;/p&gt;&#xA;" OwnerUserId="25416" LastEditorUserId="14904" LastEditDate="2016-10-20T08:53:13.570" LastActivityDate="2017-11-01T20:29:24.193" Title="How to combine two CART decision trees learned in same type of data?" Tags="|machine-learning|classification|data-mining|decision-trees|distributed|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14656" PostTypeId="1" CreationDate="2016-10-20T08:29:21.550" Score="1" ViewCount="118" Body="&lt;p&gt;I used word2vec on a customer review dataset provided by my company and ran it on my local machine - Mac OS Sierra - 10.12.1(Beta) with 8gb RAM. It is working fine. But I wanted to fasten up the process and henceforth tried to run it on an AWS Instance with 16gb RAM. But when I tried to run it there, the program keeps crashing with memory issues or segmentation faults. I tried to provide 16gb of swap memory more. Even that memory gets full and the program crashes. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Am I missing to do something here?&lt;/li&gt;&#xA;&lt;li&gt;Why can't the memory be handled by the server like my MAC?&lt;/li&gt;&#xA;&lt;li&gt;Can I do something so that the program successfully runs on my server too?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note: The code and the dataset is the same on both local machine and the server.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-10-20T08:29:21.550" Title="Word2Vec running on Local machine(Mac OS) but not on AWS Instance" Tags="|machine-learning|word2vec|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14657" PostTypeId="2" ParentId="14655" CreationDate="2016-10-20T09:20:59.750" Score="3" Body="&lt;p&gt;You mention two decision trees. Traversing a decision tree is very cheap, so running a feature instance through multiple trees is very fast, you could just take all the decision trees from the data centers and average the outcome, maybe weigh it by the (crossvalidated) strength of the models. Random Forests are powerful models that also combine decision trees in this way, except that they are done on random subsets of the features (and in some cases also random subsets of the data).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="-1" LastEditDate="2017-11-01T20:29:24.193" LastActivityDate="2017-11-01T20:29:24.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14658" PostTypeId="1" AcceptedAnswerId="14660" CreationDate="2016-10-20T09:21:33.970" Score="1" ViewCount="434" Body="&lt;p&gt;Let's imagine I have a dataset of movie reviews with annotated sentiment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-1 means negative&#xA; 0 means neutral&#xA;+1 means positive&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I see a lot of people trying to do classification to try to answer those types of problems, but shouldn't regression be used instead? To me using regression would allow the system to model that there is a transition between labels, e.g. 0 is in between. Any thoughts on this?&lt;/p&gt;&#xA;" OwnerUserId="25418" LastActivityDate="2016-10-20T10:03:04.157" Title="Use regression instead of classification for hard labeled ranking datasets" Tags="|classification|regression|ranking|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14659" PostTypeId="2" ParentId="14658" CreationDate="2016-10-20T09:46:51.387" Score="3" Body="&lt;p&gt;An issue with regression is that the distance between negative, neutral and positive are not necessarily the same. This transition/uncertainty you mention can be modeled by using a probability distribution over the classes as opposed to a hard argmax. If you use Neural Networks for example the last softmax layer gives you this information for free (For example: 0.1 on Negative, 0.6 Neutral and 0.3 Positive).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-10-20T09:46:51.387" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14660" PostTypeId="2" ParentId="14658" CreationDate="2016-10-20T10:03:04.157" Score="1" Body="&lt;p&gt;This is Ordinal Regression &lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_regression&quot; rel=&quot;nofollow&quot;&gt;https://en.wikipedia.org/wiki/Ordinal_regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quote from Wikipedia:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In statistics, ordinal regression (also called &quot;ordinal&#xA;  classification&quot;) is a type of regression analysis used for predicting&#xA;  an ordinal variable, i.e. a variable whose value exists on an&#xA;  arbitrary scale where only the relative ordering between different&#xA;  values is significant.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Examples are the ranking system you describe or any question with categorical but ordered answers often seen in surveys (&quot;always&quot;, &quot;sometimes&quot;, &quot;never&quot;).&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-10-20T10:03:04.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14661" PostTypeId="2" ParentId="13711" CreationDate="2016-10-20T11:11:05.297" Score="1" Body="&lt;p&gt;Choose the implementation according to the need. In this scenario, &lt;code&gt;tf-idf&lt;/code&gt; does a much better work than word2vec. &lt;code&gt;tf-idf&lt;/code&gt; provides the importance of a word in a document by considering the relative frequency with other documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the frequently occuring words could also occur frequently in other documents too. In &lt;code&gt;tf-idf&lt;/code&gt; method, more weights are given to words which occur much more frequently in one document compared to others. For more reading of &lt;a href=&quot;http://www.tfidf.com/&quot; rel=&quot;nofollow&quot;&gt;tfidf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-10-20T11:11:05.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14662" PostTypeId="1" CreationDate="2016-10-20T12:07:52.907" Score="3" ViewCount="2675" Body="&lt;p&gt;I want to apply a CNN over documents. I have tf-idf vectors of documents with me (one vector per document). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is 1D CNN applicable in this case? The reason I am asking this question is that I have read 1D convolutions being applied to sequence of vectors and I have sequence of scalars (tf-idf vector) as input in this case, so is it applicable? &lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="29169" LastEditDate="2021-02-16T15:35:37.847" LastActivityDate="2021-02-16T15:35:37.847" Title="Applying convolutional neural network over text documents using 1-D tf-idf feature vectors" Tags="|deep-learning|neural-network|text-mining|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="14663" PostTypeId="2" ParentId="14429" CreationDate="2016-10-20T14:20:47.200" Score="17" Body="&lt;p&gt;The problem lay in using the name 'cost' on two occasions, the problem was solved by changing this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_, c = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This way the name of the variable 'c' doesn't clash anymore with the [optimizer, cost] part of the code.&lt;/p&gt;&#xA;" OwnerUserId="19067" LastActivityDate="2016-10-20T14:20:47.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14664" PostTypeId="1" AcceptedAnswerId="14665" CreationDate="2016-10-20T18:46:47.740" Score="9" ViewCount="8843" Body="&lt;p&gt;Is it possible to create a neural network which provides a consistent output given that the input can be in different length vectors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently in a situation where I have sampled a lot of audio files, which are of different length, and have to train a neural network provides me the desired output given a certain input.  I am trying to create a regression network that can generate MFCC feature, given samples of an audio file, which are of different length, which makes different numbered input. &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorDisplayName="user380" LastEditDate="2017-07-13T15:49:26.603" LastActivityDate="2021-03-22T22:33:30.290" Title="Neural network with flexible number of inputs?" Tags="|neural-network|regression|tensorflow|supervised-learning|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14665" PostTypeId="2" ParentId="14664" CreationDate="2016-10-20T19:01:49.137" Score="11" Body="&lt;p&gt;Yes this is possible by treating the audio as a sequence into a &lt;a href=&quot;https://deeplearning4j.konduit.ai/getting-started/tutorials/recurrent-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;Recurrent Neural Network (RNN)&lt;/a&gt;. You can train a RNN against a target that is correct at the end of a sequence, or even to predict another sequence offset from the input.&lt;/p&gt;&#xA;&lt;p&gt;Do note however that there is &lt;a href=&quot;https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;a bit to learn about options that go into the construction and training of a RNN&lt;/a&gt;, that you will not already have studied whilst looking at simpler layered feed-forward networks. Modern RNNs make use of layer designs which include memory gates - the two most popular architectures are LSTM and GRU, and these add more trainable parameters into each layer as the memory gates need to learn weights in addition to the weights between and within the layer.&lt;/p&gt;&#xA;&lt;p&gt;RNNs are used extensively to predict from audio sequences that have already been processed in MFCC or similar feature sets, because they can handle sequenced data as input and/or output, and this is a desirable feature when dealing with variable length data such as &lt;a href=&quot;https://arxiv.org/abs/1402.1128&quot; rel=&quot;nofollow noreferrer&quot;&gt;spoken word&lt;/a&gt;, music etc.&lt;/p&gt;&#xA;&lt;p&gt;Some other things worth noting:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;RNNs can work well for &lt;em&gt;sequences&lt;/em&gt; of data that are variable length, and where there is a well-defined dimension over which the sequences evolve. But they are less well adapted for variable-sized sets of features where there is no clear order or sequence.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RNNs can get state-of-the-art results for signal processing, NLP and related tasks, but only when there is a very large amount of training data. Other, simpler, models can work just as well or better if there is less data.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For the specific problem of generating MFCCs from raw audio samples: Whilst it should be possible to create a RNN that predicts MFCC features from raw audio, this might take some effort and experimentation to get right, and could take a lot of processing power to make an RNN powerful enough to cope with very long sequences at normal audio sample rates. Whilst creating MFCC from raw audio using the standard approach starting with FFT will be a lot simpler, and is guaranteed to be accurate.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="113718" LastEditDate="2021-03-22T22:33:30.290" LastActivityDate="2021-03-22T22:33:30.290" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="14666" PostTypeId="1" AcceptedAnswerId="14667" CreationDate="2016-10-20T19:32:00.293" Score="6" ViewCount="2031" Body="&lt;p&gt;Based on a training set we applied a simple Linear Regression on some attributes that all were numeric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have more attributes in terms of categories and of course we applied one-hot-encoding to transform the categories to binary attributes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take for example this simple python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8, test_size=0.2)&#xA;&#xA;model = LinearRegression(normalize=True).fit(X_train, y_train)&#xA;&#xA;printErrorMetrics(trueTargets=y_test, predictions=model.predict(X_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When the table X has only the original numeric attributes the scores from the printErrorMetrics function (RMSE, etc.) are all good enough&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We were expecting better results after adding the one-hot-encoded categories but the results are so worse that the method does not seem to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are we missing anything?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do we need to preprocess the data after adding the one-hot-encoded columns/attributes?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="25452" LastEditDate="2016-10-21T14:29:48.193" LastActivityDate="2016-10-21T14:29:48.193" Title="Why after adding categorical data the Linear Regression fails?" Tags="|python|linear-regression|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14667" PostTypeId="2" ParentId="14666" CreationDate="2016-10-20T20:03:17.130" Score="7" Body="&lt;p&gt;One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = LinearRegression(fit_intercept=False, normalize=True).fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will avoid the dummy variable trap:&#xA;&lt;a href=&quot;http://www.algosome.com/articles/dummy-variable-trap-regression.html&quot;&gt;http://www.algosome.com/articles/dummy-variable-trap-regression.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also use dummy encoding to avoid this problem:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-10-20T20:03:17.130" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14668" PostTypeId="1" CreationDate="2016-10-20T20:15:50.510" Score="1" ViewCount="283" Body="&lt;p&gt;I have a basic understanding of javascript, and know hardly any other programming language. Am I bound to face some issues in the field of neural networks and machine learning because of this? Should I learn something else for the sake of avoiding some inherent weaknesses of the language? I am most worried about the capacity of javascript to handle data, rather than its possibilities regarding the textual implementation of the algorithms per se...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="25394" LastActivityDate="2016-10-20T21:48:18.777" Title="Limits of Javascript on the implementation of AI algorithms" Tags="|machine-learning|neural-network|javascript|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14669" PostTypeId="1" CreationDate="2016-10-20T20:21:36.350" Score="1" ViewCount="70" Body="&lt;p&gt;We survey our members on a regular basis on various topics, but with always the same question (basically, do you like it? yes or no), and we classify them based on what is replied, and how many participated in the survey.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we send the email, we would like to be able to know at what point in time we can estimate that the final result (how many people will fill, what % of yes) is likely to be the one we have now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We did process the previous surveys, and we can see a pattern where after 6 or 12 hours, the result is similar to the final one (ie when we wait more than a week after the mailing). Obviously, a survey sent friday at 8pm takes longer to reach a conclusive stage than the ones sent on tue at 9am. They are other factors too, for instance if our mail provider experienced delay delivering, then of course we need to wait longer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to know how long to wait (3, 6, 12... hours?) before we get a solid prediction of the result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an algorithm that can predict the result at any point in time and give an estimate on how likely the estimate is correct (p value like)?&lt;/p&gt;&#xA;" OwnerUserId="25432" LastEditorUserId="39253" LastEditDate="2017-09-15T21:14:57.753" LastActivityDate="2017-09-15T21:14:57.753" Title="Predict the outcome of a new time series (mailing)" Tags="|time-series|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14670" PostTypeId="2" ParentId="14668" CreationDate="2016-10-20T21:48:18.777" Score="3" Body="&lt;p&gt;First of all, learning other programming languages is &lt;em&gt;always&lt;/em&gt; a good thing.  There are many reasons for this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages makes it easier for you to learn &lt;em&gt;other&lt;/em&gt; new languages in the future&lt;/strong&gt; - and sooner or later in your career you will almost certainly want to or need to.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages helps you understand the language you're already working in.&lt;/strong&gt;  If I told you JavaScript is a dynamically-typed language, would that mean anything to you?  Probably not so much if you've never used a statically-typed language.  If I told you it has first-class functions, would that mean anything to you?  It's like telling someone &quot;night is dark&quot; when they've never seen daylight.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages teaches you about ideas from other languages that may eventually make it into your language.&lt;/strong&gt; ECMAScript 6 added &quot;arrow functions&quot; to JavaScript, these are also known as lambdas and would already be familiar to you if you've used C# (from version 3 onwards), Java (from version 8 onwards), Scala, Haskell, or many other languages.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages teaches you different ways to think about code.&lt;/strong&gt; And this can only make you a better coder.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages increases your employment opportunities.&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning new languages is fun!&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In addition to these, and other benefits I've surely omitted, there's this which is directly relevant to your question about machine learning in particular.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;7&quot;&gt;&#xA;&lt;li&gt;&lt;strong&gt;Different languages have different libraries and different communities of users that use them for different purposes.&lt;/strong&gt;  Most people doing machine learning aren't working primarily in JavaScript.  Some are working in Python, some are working in R, some are working with Spark using Scala or Python, etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you already know JavaScript, then learning Python shouldn't be too hard for you, and it has some great machine learning libraries like scikit-learn, with lots of great resources online to help you learn.  So I'd recommend starting there.  But I also recommend that you eventually branch out and learn some languages that are further outside your comfort zone.  &lt;/p&gt;&#xA;" OwnerUserId="14" LastActivityDate="2016-10-20T21:48:18.777" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14671" PostTypeId="2" ParentId="9022" CreationDate="2016-10-20T21:55:29.733" Score="0" Body="&lt;p&gt;You can also do something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;patterns = df [['Total','Date']]&#xA;patterns = purchase_patterns.set_index(&amp;quot;Date&amp;quot;)&#xA;resample = patterns.resample ('D' , how = sum)&#xA;&#xA;#to extract the last items of the list&#xA;&#xA;last_7 = resample[-7:]&#xA;&#xA;# and to get the total&#xA;last_7 = resample[-7:].sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;A reference for data slicing is &lt;a href=&quot;http://chris.friedline.net/2015-12-15-rutgers/lessons/python2/02-index-slice-subset.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="25435" LastEditorUserId="98307" LastEditDate="2020-08-03T13:41:42.427" LastActivityDate="2020-08-03T13:41:42.427" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14673" PostTypeId="1" CreationDate="2016-10-21T03:24:56.373" Score="2" ViewCount="40" Body="&lt;p&gt;Popular counting sketches(loglog, hyperloglog, etc) feature natural union operations. Are there any known counting sketches that feature natural intersection operations? &lt;/p&gt;&#xA;" OwnerUserId="25439" LastEditorUserId="25439" LastEditDate="2016-10-21T04:21:34.213" LastActivityDate="2016-10-21T04:21:34.213" Title="Is there a counting sketch optimized for intersections?" Tags="|statistics|probability|data-stream-mining|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14674" PostTypeId="2" ParentId="14669" CreationDate="2016-10-21T04:16:06.620" Score="1" Body="&lt;p&gt;Think about this problem in two parts.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 1) &#xA;How many samples do you need in order to have the desired confidence in the Null hypothesis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 2) How long does it take to get that many samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 1 can be answered with A-priori power analysis.  Use this technique to determine the sample size needed to get the p-value desired.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part 2 can be addressed with an algorithm applied to the data you have collected thus far.  Your predictors in this case would be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column 1: the probability that your statistic of interest has leveled off (is not changing significantly with additional input).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column 2: date/day/time start. You probably will need to codify this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column 3: number of samples at finish.  These should be the number which you determined from your prior analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your response/output in this case would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Column 4: the date/day/time finish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course would need to do some data analysis to determine the appropriate algorithm to use on this data set.  The end product should be a function that takes as input a target probability, date/day/time start and a target sample size, while outputting a date/day/time finish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this at least gives you some ideas on how to proceed.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-21T04:16:06.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14675" PostTypeId="2" ParentId="10550" CreationDate="2016-10-21T07:25:49.957" Score="1" Body="&lt;p&gt;As per your problem and the description, I'll suggest you to try &lt;strong&gt;Text Mining - Bag of Words&lt;/strong&gt; approach. I did something similar using the same approach and it was really helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details about the approach are &lt;a href=&quot;http://gerardnico.com/wiki/natural_language/bag_of_word&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give it a try. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-21T07:25:49.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14676" PostTypeId="1" CreationDate="2016-10-21T08:38:57.870" Score="4" ViewCount="2339" Body="&lt;p&gt;I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the example in the sklearn documentation, I was wondering why the LDA model is fit on a TF array, while the NMF model is fit on a TF-IDF array. Is there a precise reason for this choice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the example: &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, any tips about how to find the best parameters (number of iterations, number of topics...) for fitting my models is well accepted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="25448" LastActivityDate="2020-02-01T03:02:35.317" Title="scikit-learn - Should I fit model with TF or TF-IDF?" Tags="|machine-learning|python|topic-model|lda|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14677" PostTypeId="1" CreationDate="2016-10-21T08:42:44.307" Score="1" ViewCount="4319" Body="&lt;p&gt;The company I'm working for runs social network sites, and we're classifying messages sent one-to-one as spam or not spam. The issue is that it's been trained exclusively on German training data, since most of the networks we operate are local to Germany. Soon we will need to support multiple languages for this, since the one trained on only German will often think messages in other languages are spam as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd prefer not to have to maintain different training data for different languages if possible, so my question is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What's a reasonable approach to doing text classification for multiple languages?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A spam message in our cases usually contains obfuscated links and subtle references to other websites. Example of a message that should be classified as spam:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;hi funnyguy kennst du mich noch ? konnte hier kein bilder hoch laden komm mal bitte zu ( somethingelse.com) mein nik ist ( 19theusername91 ) mal sehen ob du mich noch kennst :)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Somegirl&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The current approach is the normal stemming, TF-IDF and LSA for pre-processing, then a two-level classifier: an ensemble of normal classifiers that's used as input for a linear classifier that will make the final decision.&lt;/p&gt;&#xA;" OwnerUserId="23238" LastActivityDate="2022-09-29T02:27:18.733" Title="How to approach Multilingual Text Classification?" Tags="|classification|nlp|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14678" PostTypeId="2" ParentId="14677" CreationDate="2016-10-21T09:51:55.170" Score="2" Body="&lt;p&gt;I suggest to acquire some multilingual training and test data and do experiments whether it is better to train on mixed languages or doing language detection first and use monolingual models. The fact that you use some NLT techniques (like the stemming mentioned) suggests the latter path.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2016-10-21T09:51:55.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14680" PostTypeId="1" AcceptedAnswerId="14682" CreationDate="2016-10-21T10:19:08.580" Score="1" ViewCount="273" Body="&lt;p&gt;I just read Bishop's book &lt;em&gt;Pattern Recognition and Machine Learning&lt;/em&gt;. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the notation is somehow akward: suppose I want to compute $a_2$, then&#xA; $$a_2=w_{21}z_1+w_{22}z_2+\dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then does  $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="-1" LastEditDate="2016-10-23T11:21:53.050" LastActivityDate="2016-10-23T11:21:53.050" Title="Does the Bishop book imply that a neuron feeds to itself in chapter 5.3?" Tags="|machine-learning|backpropagation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14681" PostTypeId="2" ParentId="14680" CreationDate="2016-10-21T10:34:53.880" Score="2" Body="&lt;p&gt;The output of the neuron is computed as the activation function applied to the sum directly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z_3 = h(w_{21}z_1 + w_{22}z_2) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have a look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron&quot; rel=&quot;nofollow&quot;&gt;perceptron wikipage&lt;/a&gt;, where there are explanations and ilustrations of this, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron#/media/File:Perceptron.svg&quot; rel=&quot;nofollow&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2016-10-21T10:50:39.223" LastActivityDate="2016-10-21T10:50:39.223" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14682" PostTypeId="2" ParentId="14680" CreationDate="2016-10-21T10:36:17.850" Score="3" Body="&lt;p&gt;The equations are only working for a given layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to generalize, you need to rewrite them as, for example :&#xA;$$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-10-21T10:36:17.850" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14685" PostTypeId="1" CreationDate="2016-10-21T15:33:24.360" Score="0" ViewCount="143" Body="&lt;p&gt;I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Trump Facebook Posts&lt;/strong&gt;&#xA;&lt;a href=&quot;https://data.world/dave/trumps-facebook-statuses&quot; rel=&quot;nofollow&quot;&gt;https://data.world/dave/trumps-facebook-statuses&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Trump Tweets&lt;/strong&gt;&#xA;&lt;a href=&quot;https://data.world/dave/trump-tweets&quot; rel=&quot;nofollow&quot;&gt;https://data.world/dave/trump-tweets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally I would like to be able to count the distinct instances that a word or phrase (like 'crooked' or 'massive' or 'the best' appeared across both datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The table name for the Facebook Post dataset is &lt;code&gt;TrumpFacebook&lt;/code&gt; and for the Trump Tweets is &lt;code&gt;trump_tweets&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="14011" LastEditorUserId="14011" LastEditDate="2016-10-21T15:47:52.070" LastActivityDate="2016-10-22T15:24:16.600" Title="How do I write a SQL (or dwSQL) query across these two different datasets?" Tags="|data-mining|dataset|data|sql|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14686" PostTypeId="1" AcceptedAnswerId="14731" CreationDate="2016-10-21T15:48:03.547" Score="1" ViewCount="56" Body="&lt;p&gt;I am hoping that there is some existing software for what I want to accomplish, as I'm not a big fan of reinventing the wheel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I would like a software package that can serve as a workflow that integrates network analysis, predictive analytics, and performance evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, information about a criminal network along with other relevant data would be used to predict monetary flows across the network.  Law enforcement case data combined with information about network monetary flows would direct resource use in disrupting flows across the network.  These law enforcement performance metrics along with metrics related to criminal networks would be used for strategic decision making.  It would also be ideal to have dashboards showing some of the metrics of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have identified some potential software candidates: Dataiku DSS, RapidMiner, KNIME, Orange Data Mining, Watson, and Statistica.  I am not too familiar with them and I wonder if something else may immediately come to mind for those more familiar with these applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any direction is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Ted  &lt;/p&gt;&#xA;" OwnerUserId="25459" LastActivityDate="2016-10-24T07:03:32.070" Title="software for workflow integrating network analysis, predictive analytics, and performance metrics" Tags="|predictive-modeling|statistics|social-network-analysis|software-recommendation|" AnswerCount="1" CommentCount="0" ClosedDate="2016-10-24T11:44:56.660" ContentLicense="CC BY-SA 3.0" />
  <row Id="14688" PostTypeId="1" CreationDate="2016-10-21T16:58:18.680" Score="1" ViewCount="1583" Body="&lt;p&gt;is there any reason why the validation mean squared error output from Keras is always very similar to 1? Thank you. All of my training results looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;155/155 [==============================] - 0s - loss: 6062.6136 - mean_absolute_error: 0.8344 - mean_squared_error: 1.0271 - val_loss: 0.8252 - val_mean_absolute_error: 0.8252 - val_mean_squared_error: 1.0164&#xA;Epoch 29/1000&#xA;155/155 [==============================] - 0s - loss: 5870.5280 - mean_absolute_error: 0.8324 - mean_squared_error: 1.0211 - val_loss: 0.8246 - val_mean_absolute_error: 0.8246 - val_mean_squared_error: 1.0130&#xA;Epoch 30/1000&#xA;155/155 [==============================] - 0s - loss: 5668.5083 - mean_absolute_error: 0.8311 - mean_squared_error: 1.0134 - val_loss: 0.8244 - val_mean_absolute_error: 0.8244 - val_mean_squared_error: 1.0106&#xA;Epoch 31/1000&#xA;155/155 [==============================] - 0s - loss: 5530.8119 - mean_absolute_error: 0.8288 - mean_squared_error: 1.0115 - val_loss: 0.8243 - val_mean_absolute_error: 0.8243 - val_mean_squared_error: 1.0089&#xA;Epoch 32/1000&#xA;155/155 [==============================] - 0s - loss: 5222.6773 - mean_absolute_error: 0.8283 - mean_squared_error: 1.0119 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0071&#xA;Epoch 33/1000&#xA;155/155 [==============================] - 0s - loss: 5090.0273 - mean_absolute_error: 0.8273 - mean_squared_error: 1.0078 - val_loss: 0.8247 - val_mean_absolute_error: 0.8247 - val_mean_squared_error: 1.0060&#xA;Epoch 34/1000&#xA;155/155 [==============================] - 0s - loss: 4878.2420 - mean_absolute_error: 0.8272 - mean_squared_error: 1.0093 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0046&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;note: I have standardized my input and output with sklearn standardization:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import preprocessing&#xA;    X_scaler = preprocessing.StandardScaler().fit(X_list_total)&#xA;    X_list_total_standardized = X_scaler.transform(X_list_total)&#xA;    Y_scaler = preprocessing.StandardScaler().fit(Y_list_total)&#xA;    Y_list_total_standardized = Y_scaler.transform(Y_list_total)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does it just mean that there is nothing to learn from the data at all?&lt;/p&gt;&#xA;" OwnerUserId="20534" LastEditorUserId="20534" LastEditDate="2016-10-21T22:10:15.140" LastActivityDate="2016-11-21T11:53:43.523" Title="keras validation mean squared error always similar to 1" Tags="|machine-learning|scikit-learn|tensorflow|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14689" PostTypeId="2" ParentId="14432" CreationDate="2016-10-21T17:00:45.030" Score="3" Body="&lt;p&gt;Balancing the training set makes sense, there is no need to balance the test set. Anyway, you should not be watching overfitting in such a simple model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is interesting to validate that the training set and validation set variables are exactly the same, and if you did not do any transformation in a set and for some reason did not in the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand you are doing holdout crossvalidation, try k-fold crossvalidation or bootstrap can help.&lt;/p&gt;&#xA;" OwnerUserId="25238" LastEditorUserId="25238" LastEditDate="2016-10-21T17:06:07.047" LastActivityDate="2016-10-21T17:06:07.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14690" PostTypeId="1" AcceptedAnswerId="14824" CreationDate="2016-10-21T17:29:49.993" Score="3" ViewCount="9640" Body="&lt;p&gt;I have a data frame consisting of some continuous data features. I did a &lt;em&gt;kde&lt;/em&gt; plot of the features using seaborn kdeplot functionality which gave me a plot as shown below :&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dgl0w.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dgl0w.png&quot; alt=&quot;Seaborn kdeplot&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;How do I interpret this visualization in order to check for things like skew in the data points, etc.?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-25T15:13:47.763" LastActivityDate="2020-12-25T15:13:47.763" Title="Checking for skewness in data" Tags="|machine-learning|python|data-cleaning|visualization|seaborn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="14691" PostTypeId="1" AcceptedAnswerId="14700" CreationDate="2016-10-21T17:37:17.087" Score="0" ViewCount="224" Body="&lt;p&gt;I am interested in a canned or built-in network analysis tool.  Wondering if this is possible with Orange.&lt;/p&gt;&#xA;" OwnerUserId="25459" LastActivityDate="2016-10-22T04:11:22.737" Title="Is there a network analysis tool built into Orange?" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14693" PostTypeId="1" AcceptedAnswerId="14697" CreationDate="2016-10-21T19:38:22.763" Score="3" ViewCount="59746" Body="&lt;p&gt;I have in mind that &lt;em&gt;R-squared&lt;/em&gt; is the explained variance of the response by the predictors. But i'd like to know how the adjusted value is computed ? and if the concept has any change from the original.&lt;/p&gt;&#xA;" OwnerUserId="25043" LastActivityDate="2016-10-22T00:42:53.847" Title="What is the difference of R-squared and adjusted R-squared?" Tags="|statistics|linear-regression|parameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14694" PostTypeId="1" CreationDate="2016-10-21T19:43:13.030" Score="0" ViewCount="295" Body="&lt;p&gt;I have power consumption and the corresponding temperature data for one year, that is, for several weeks (week1, week2,..., week48) as one month has 4 weeks. I want to forecast power consumption for a month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forecasting power consumption means forecasting 4 weeks(a month has 4 weeks). I am using Ms excel to obtain the pearson correlation coefficient. The power consumption data and temperature data have a strong correlation of 0.8 for week1 to week48, that is, they have a good correlation for one year. One year means over a large time interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But over a small time interval, for eg, correlation b/w power consump and temperature for 12 weeks (3 months) is weak (0.2). I want to use a neural network with window size of 3 to represent 12 weeks(3 months) to forecast the next 4 weeks(next month).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: since the power data and temperature data has good correlation(0.8) for 48 weeks(1 year) but weak correlation(0.2) for 12 weeks(3 months), is it possible to use a neural network with window size of 3 as mentioned above to predict the power consumption for the next month with input variables power data and temperature data?&lt;/p&gt;&#xA;" OwnerUserId="25467" LastActivityDate="2016-11-21T05:02:38.833" Title="correlation coefficient between weather data and power consumption" Tags="|neural-network|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14697" PostTypeId="2" ParentId="14693" CreationDate="2016-10-22T00:42:53.847" Score="8" Body="&lt;p&gt;A google search for r-squared adjusted yielded several easy to follow explanations.  I am going to paste a few directly from such results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statisticshowto.com/adjusted-r2/&quot; rel=&quot;noreferrer&quot;&gt;Meaning of Adjusted R2&lt;/a&gt;&#xA;Both R2 and the adjusted R2 give you an idea of how many data points fall within the line of the regression equation. However, there is one main difference between R2 and the adjusted R2: R2 assumes that every single variable explains the variation in the dependent variable. The adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HMn8f.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HMn8f.png&quot; alt=&quot;Rsquared adjusted&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.minitab.com/blog/adventures-in-statistics/multiple-regession-analysis-use-adjusted-r-squared-and-predicted-r-squared-to-include-the-correct-number-of-variables&quot; rel=&quot;noreferrer&quot;&gt;What Is the Adjusted R-squared?&lt;/a&gt;&#xA;The adjusted R-squared compares the explanatory power of regression models that contain different numbers of predictors.&#xA;Suppose you compare a five-predictor model with a higher R-squared to a one-predictor model. Does the five predictor model have a higher R-squared because it’s better? Or is the R-squared higher because it has more predictors? Simply compare the adjusted R-squared values to find out!&#xA;The adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance. The adjusted R-squared can be negative, but it’s usually not.  It is always lower than the R-squared.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-10-22T00:42:53.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14698" PostTypeId="2" ParentId="14677" CreationDate="2016-10-22T01:34:35.423" Score="0" Body="&lt;p&gt;Some discriminative features like presence of URLs, frequency of proper punctuation and spelling mistakes translate easily. For semantic features you can use multilingual word embeddings, so your content can be treated by the same classifier, regardless of language. My educated guess is that you should be able to detect most spam without going this far.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-10-22T01:34:35.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14699" PostTypeId="2" ParentId="14694" CreationDate="2016-10-22T04:03:52.300" Score="1" Body="&lt;p&gt;I am not very sure how effective NN would be for this problem. The way I see it is that you have 48 entries in a time series and now you are trying to predict the next 4. Keeping aside the correlations for a second, it seems like there is very little data to &quot;learn&quot; from. I mean the power consumption is a seasonal thing (much like temperature) but you have only 1 years data so it should be difficult for NN to capture this seasonality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to raise another point about the way you are using correlations. If you are taking correlations using the actual temperature value then I do not think that to be a good idea. Think of it in this way, I would be using a lot of power at both very high and very low temperature. So instead of using the raw temperature values, I should normalize them with respect to some &quot;comfortable&quot; temperature value by taking the absolute difference between the actual temperature and the &quot;comfortable&quot; temperature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, let us say that &quot;comfortable&quot; temperature is 25C, then both 5C and 45C would be normalized to 20 and this looks more plausible as my power consumption should be high in both the cases. This might also explain the poor correlation that you observe for 3 months of data. What should be the &quot;comfortable&quot; temperature is an entirely different story altogether. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if this line of thought makes sense.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-10-22T04:03:52.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14700" PostTypeId="2" ParentId="14691" CreationDate="2016-10-22T04:11:22.737" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/biolab/orange3-network&quot; rel=&quot;nofollow&quot;&gt;This Networks add-on for Orange data mining suite&lt;/a&gt; should help you. If you are open to using other solutions, I would recommend &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow&quot;&gt;networkx&lt;/a&gt; Python library.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-10-22T04:11:22.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14701" PostTypeId="2" ParentId="14688" CreationDate="2016-10-22T04:18:40.143" Score="1" Body="&lt;p&gt;It could be very close to 1 (or for that matter, changing very slowly) for many reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Maybe your learning rate is very small.&lt;/li&gt;&#xA;&lt;li&gt;Maybe the network has reached its learning capacity. Since we do not know anything about the network architecture, we can't rule this out.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This does not mean there is nothing to learn from the data. In the worst case, there is nothing to learn from data, with the given architecture,  hyperparameters and the time you are willing to let the network learn.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-10-22T04:18:40.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14702" PostTypeId="1" CreationDate="2016-10-22T05:25:59.573" Score="2" ViewCount="199" Body="&lt;p&gt;I have hourly temperature and power consumption data of several days of a month. The pattern is almost similar across days like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/sOyQ5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sOyQ5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this data I want to predict the usage of a coming day. I have features : 1) hour of the day 2) temperature; and the response variable, power. Looking at the data, I believe I should fit three separate models and not a single model&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First model for data from Midnight to 10 AM as usage remains almost constant during this time, and temperature does not vary too much&lt;/li&gt;&#xA;&lt;li&gt;Second model for data from 11AM till 6 PM. This portion follows a sharp increase and then almost constant usage&lt;/li&gt;&#xA;&lt;li&gt;Third model for data from 7 PM till midnight. This portion shows constant decrease in power&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To follow this intuition, I used three models accordingly and later combined predictions from these to output sequence of 24 numbers for coming day. Formula for each of these models is:&#xA;&lt;code&gt;lm(power ~ time_hour + temperature, data = xxx)&lt;/code&gt;, but each model is trained with the data corresponding to specific time duration of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of dividing the data manually and using three separate models, &lt;strong&gt;is there any other existing technique which will take care of our intuition and does not need manual division of data or creation of separate models.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;During my search, I found that I can use GAM (Generalized Additive Models) and I came up with following formula&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(splines) &#xA;lm(power ~ ns(time_hour, knots =(9, 18)) + temperture, data = xxx)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using above formula I think that I am placing knots at 9 AM and at 6 PM. Right?&#xA;I don't know how should I enforce knots in temperature feature exactly at these specific times, so that knots of &lt;code&gt;temperature&lt;/code&gt; and &lt;code&gt;time_hour&lt;/code&gt; will sync.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The above plots were plotted using following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dframe &amp;lt;- structure(list(time_hour = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, &#xA;12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24), temperature = c(22.5, &#xA;24, 23.5, 20.5, 22.5, 22.5, 19.5, 23.5, 23, 20.5, 26.5, 28.5, &#xA;30, 32, 33.5, 33, 30.5, 30, 29.5, 29, 28, 27, 28, 28.5), power = c(97.04319, &#xA;95.7225, 88.59191, 88.34882, 90.17179, 88.82062, 87.73833, 89.36342, &#xA;85.31775, 91.1292, 116.79035, 149.58614, 172.32438, 171.27931, &#xA;159.53858, 162.03544, 170.78468, 164.0275, 155.86717, 135.77197, &#xA;133.01235, 116.29253, 100.87483, 97.84942)), .Names = c(&quot;time_hour&quot;, &#xA;&quot;temperature&quot;, &quot;power&quot;), row.names = c(NA, -24L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and minimal code used is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;par(mfrow= c(1,2))&#xA;plot(dframe$time_hour ,dframe$power,type=&quot;l&quot;,xlab = &quot;Hour of day&quot;, ylab = &quot;power&quot; )&#xA;plot(dframe$time_hour ,dframe$temperature,type=&quot;l&quot;,xlab = &quot;Hour of day&quot;, ylab = &quot;Temperature&quot; )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13291" LastActivityDate="2016-10-22T21:29:37.557" Title="Different models for different time durations of a day" Tags="|predictive-modeling|time-series|regression|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14704" PostTypeId="2" ParentId="14053" CreationDate="2016-10-22T12:07:15.787" Score="1" Body="&lt;h2&gt;Solution 1&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;To include the number of counts, you can just set &lt;code&gt;labels=TRUE.&lt;/code&gt; The example below is just slightly adapted from one on the  &lt;code&gt;hist()&lt;/code&gt; help page:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hist(iris$Petal.Width, labels = TRUE, ylim=c(0, 45))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Solution 2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Adding percentages is a bit more involved as of now. You can easily do that by directly manipulate the object returned by a call to &lt;code&gt;hist()&lt;/code&gt; as-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;histPercent &amp;lt;- function(x, ...) {&#xA;   H &amp;lt;- hist(x, plot = FALSE)&#xA;   H$density &amp;lt;- with(H, 100 * density* diff(breaks)[1])&#xA;   labs &amp;lt;- paste(round(H$density), &quot;%&quot;, sep=&quot;&quot;)&#xA;   plot(H, freq = FALSE, labels = labs, ylim=c(0, 1.08*max(H$density)),...)&#xA;}&#xA;&#xA;histPercent(iris$Petal.Width)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="10902" LastEditDate="2016-10-22T18:54:19.453" LastActivityDate="2016-10-22T18:54:19.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14705" PostTypeId="2" ParentId="14685" CreationDate="2016-10-22T14:55:29.143" Score="1" Body="&lt;p&gt;Have you tried something like this in SAS?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PROC SQL;&#xA; CREATE TABLE TRUMP_FB_TWEETS AS&#xA; SELECT *&#xA; FROM TRUMP_FB&#xA; UNION CORRESPONDING ALL&#xA; SELECT *&#xA; FROM TRUMP_TWEETS;&#xA;QUIT;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Source:&lt;a href=&quot;http://www2.sas.com/proceedings/sugi25/25/cc/25p109.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www2.sas.com/proceedings/sugi25/25/cc/25p109.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16688" LastActivityDate="2016-10-22T14:55:29.143" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14709" PostTypeId="2" ParentId="14702" CreationDate="2016-10-22T17:25:06.163" Score="1" Body="&lt;p&gt;This is a typical time series problem. The first step is to make sure your time series is stationary, see &lt;a href=&quot;https://www.otexts.org/fpp/8/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for a good explanation why this is necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we first do,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dframe$power_diff &amp;lt;- c(NA, diff(dframe$power))&#xA;plot(dframe$power_diff, type = &quot;l&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gH0Le.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gH0Le.png&quot; alt=&quot;powerdiff&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next we want to consider an &lt;a href=&quot;http://robjhyndman.com/hyndsight/arimax/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ARIMAX model&lt;/a&gt;. As part of this exercise we want to know whether there is a dependency in the time series, a so called auto regressive term. We can research this using,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;acf(na.omit(dframe$power_diff))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VzaJg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VzaJg.png&quot; alt=&quot;acf_power_diff&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this plot we see that there is a positive auto regressive term at lag 1 at a significance level of 95%, we conclude this based on the observation that the second line, which is lag 1, is above the blue dotted line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next we want to fit a ARIMAX model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Convert data to ts object&#xA;power_data &amp;lt;- ts(data = dframe$power_diff[-1], frequency = 8766, start = c(2016, 1, 1))&#xA;temp_data &amp;lt;- ts(data = dframe$temp_diff[-1], frequency = 8766, start = c(2016, 1, 1))&#xA;&#xA;# Build the model.&#xA;m &amp;lt;- Arima(power_data, order = c(1, 0, 0), xreg = temp_data)&#xA;&#xA;# Let's see how our fit is.&#xA;plot(power_data)&#xA;lines(fitted(m), col = &quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives this result,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/p2GIP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/p2GIP.png&quot; alt=&quot;forecast&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can play with the ARIMA(p, i, q) terms. We should leave $p=1$ since we found that the AR term is significant at lag 1. If we add a MA(1) we find a model that has a &lt;code&gt;MAPE=260.46&lt;/code&gt; versus a &lt;code&gt;MAPE=272.62&lt;/code&gt; for our initial model. So you can create the best model using,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m2 &amp;lt;- Arima(power_data, order = c(1, 1, 0), xreg = temp_data)&#xA;lines(fitted(m2), col = 'red')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Will lead to,&#xA;&lt;a href=&quot;https://i.stack.imgur.com/FRkhl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FRkhl.png&quot; alt=&quot;arimas&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course we can automate the above step with &lt;code&gt;auto.arima()&lt;/code&gt;. This function will search for the best &lt;code&gt;arima&lt;/code&gt; model,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m3 &amp;lt;- auto.arima(power_data, xreg = temp_data)&#xA;lines(fitted(m3), col = 'green')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which will give you,&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ShMLW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ShMLW.png&quot; alt=&quot;auto_arima&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You see that each model has a poor fit in the spike. To improve this you could introduce a dummy variable,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xdata &amp;lt;- ts(data = as.matrix(cbind(temp_data, dframe$power_diff[-1] &amp;gt; 20)), frequency = 8766, start =c(2016,1,1))&#xA;m_dummy &amp;lt;- auto.arima(power_data, xreg = xdata)&#xA;lines(fitted(m_dummy), col = 'orange')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives us the following result,&#xA;&lt;a href=&quot;https://i.stack.imgur.com/UC7M1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UC7M1.png&quot; alt=&quot;dummy&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We see that this model gives us a much better result. So if you can relate dummy to a point in time you could create a model that has a good fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According Occam's Razor you should try to keep your models simple. So I would suggest to build a single model for all your time slots instead of three separate models as you suggested in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can turn this back to the original time series with &lt;code&gt;cumsum&lt;/code&gt;. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2016-10-22T21:29:37.557" LastActivityDate="2016-10-22T21:29:37.557" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14710" PostTypeId="1" CreationDate="2016-10-22T23:19:47.387" Score="1" ViewCount="11094" Body="&lt;p&gt;I have following type of data. But I would like to have graph with X-axis as Year and add all the sales added up in that year on Y- axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Data frame name is salesdata. I have nearly 8000 rows of data. and the column headers are in the picture below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JuDQz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JuDQz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="22013" LastActivityDate="2017-04-21T16:12:39.220" Title="How to summarize(add) a column according to same year and plot in R?" Tags="|r|visualization|ggplot2|rstudio|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14712" PostTypeId="2" ParentId="14162" CreationDate="2016-10-23T01:44:13.543" Score="-1" Body="&lt;p&gt;There are numerous papers on evaluating recommender systems. See, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Evaluating collaborative filtering recommender systems&lt;/li&gt;&#xA;&lt;li&gt;Evaluating Recommendation Systems&lt;/li&gt;&#xA;&lt;li&gt;Evaluating Recommender Behavior For New Users&lt;/li&gt;&#xA;&lt;li&gt;Evaluating recommender systems from the user’s perspective: survey of the state of the art&lt;/li&gt;&#xA;&lt;li&gt;Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-10-23T01:44:13.543" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14715" PostTypeId="2" ParentId="14710" CreationDate="2016-10-23T06:49:45.667" Score="0" Body="&lt;h3&gt;Step 1: Summarizing&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You can use the &lt;code&gt;dplyr&lt;/code&gt; package for that purpose, as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;x %&amp;gt;% &#xA;  group_by(Order_Date) %&amp;gt;% &#xA;  summarise(Sales = sum(Sales))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OR, this approach will work as well,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x %&amp;gt;% &#xA;  group_by(Order_Date) %&amp;gt;% &#xA;  summarise_each(funs(sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Step 2: Plot&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Once you're done with it, use &lt;code&gt;ggplot&lt;/code&gt; to draw a plot of your interest, just use the respective function. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Additional Resources&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;For more details visit these blogs-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/plot-weekly-or-monthly-totals-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plot Weekly or Monthly Totals in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.magesblog.com/2012/06/transforming-subsets-of-data-in-r-with.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Transforming subsets of data in R with by, ddply and data.table&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/10349096/group-data-and-plot-multiple-lines&quot;&gt;Group data and plot multiple lines&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-23T06:49:45.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14716" PostTypeId="2" ParentId="14162" CreationDate="2016-10-23T07:02:56.153" Score="3" Body="&lt;p&gt;First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold.&lt;/p&gt;&#xA;&lt;p&gt;However, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better, you can make theoretical claims on levels of the RMSE by knowing what is expected from your DV in your field of research. Keep in mind that you can always normalize the RMSE.&lt;/p&gt;&#xA;&lt;h3&gt;Additional Info&lt;/h3&gt;&#xA;&lt;p&gt;Decision support accuracy metrics that are popularly used are &lt;strong&gt;Reversal rate, Weighted errors, Receiver Operating Characteristics (ROC) and Precision Recall Curve (PRC), Precision, Recall and F-measure.&lt;/strong&gt; These metrics help users in selecting items that are of very high quality out of the available set of items.&lt;/p&gt;&#xA;&lt;p&gt;The metrics view prediction procedure as a binary operation which distinguishes good items from those items that are not good. &lt;strong&gt;ROC curves&lt;/strong&gt; are very successful when performing comprehensive assessments of the performance of some specific algorithms.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Precision&lt;/strong&gt; is the fraction of recommended items that is actually relevant to the user, while &lt;strong&gt;Recall&lt;/strong&gt; can be defined as the fraction of relevant items that are also part of the set of recommended items. They are computed as --&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Precision (P) = Correctly Recommended Items / Total Recommended Items&lt;/p&gt;&#xA;&lt;p&gt;Recall (R) = Correctly Recommended Items / Total Useful Recommended Items&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;F-measure&lt;/strong&gt; defined below helps to simplify precision and recall into a single metric. The resulting value makes comparison between algorithms and across data sets very simple and straightforward.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;F-measure = 2PR / P + R&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Coverage has to do with the percentage of items and users that a recommender system can provide predictions. Prediction may be practically impossible to make if no users or few users rated an item. Coverage can be reduced by defining small neighborhood sizes.&lt;/p&gt;&#xA;&lt;p&gt;That's all. Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-23T07:02:56.153" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14718" PostTypeId="1" CreationDate="2016-10-23T07:59:40.220" Score="-3" ViewCount="443" Body="&lt;p&gt;While importing keras (from &lt;code&gt;keras.models&lt;/code&gt; import Sequential), I am getting the following error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;&#xA; File &quot;/home/afzal/Deep cnn/cnn_deeplearningpgm/keras_cnn_new3.py&quot;,&#xA; line 2, in &amp;lt;module&amp;gt;&#xA;     import keras   File &quot;/usr/local/lib/python2.7/dist-packages/Keras-1.1.0-py2.7.egg/keras/__init__.py&quot;,&#xA; line 2, in &amp;lt;module&amp;gt;&#xA;     from . import backend   File &quot;/usr/local/lib/python2.7/dist-packages/Keras-1.1.0-py2.7.egg/keras/backend/__init__.py&quot;,&#xA; line 29, in &amp;lt;module&amp;gt;&#xA;     _config = json.load(open(_config_path))   File &quot;/usr/lib/python2.7/json/__init__.py&quot;, line 291, in load&#xA;     **kw)   File &quot;/usr/lib/python2.7/json/__init__.py&quot;, line 339, in loads&#xA;     return _default_decoder.decode(s)   File &quot;/usr/lib/python2.7/json/decoder.py&quot;, line 364, in decode&#xA;     obj, end = self.raw_decode(s, idx=_w(s, 0).end())   File &quot;/usr/lib/python2.7/json/decoder.py&quot;, line 380, in raw_decode&#xA;     obj, end = self.scan_once(s, idx) ValueError: Expecting object: line 5 column 25 (char 108)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have installed theano. Could anyone help me solve this?&lt;/p&gt;&#xA;" OwnerUserId="25501" LastEditorUserId="29169" LastEditDate="2019-06-10T15:47:32.693" LastActivityDate="2019-06-10T15:47:32.693" Title="Keras Import Error" Tags="|keras|theano|json|" AnswerCount="1" CommentCount="0" ClosedDate="2021-09-02T14:31:36.153" ContentLicense="CC BY-SA 4.0" />
  <row Id="14719" PostTypeId="2" ParentId="14718" CreationDate="2016-10-23T11:17:46.697" Score="1" Body="&lt;p&gt;Looks like your &lt;code&gt;~/.keras/keras.json&lt;/code&gt; is either missing or incorrect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try deleting &lt;code&gt;~/.keras&lt;/code&gt; and running &lt;code&gt;import keras&lt;/code&gt;, this should re-create blank config. You may need to edit it to set backend to Theano.&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2016-10-23T11:17:46.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14721" PostTypeId="1" CreationDate="2016-10-23T13:33:51.017" Score="2" ViewCount="6779" Body="&lt;p&gt;I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: I have a dataset which contains information about a person. This can be gender, weight, age, height etc. Let's assume there are information missing of age and height for a specific person. How can this information be predicted based on the information I have in my dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read about PCA-methods, but I would like to get an overview of pros and cons of methodologies and also an overview of recent research (good working algorithms for the given dataset/where to start reading for developing an algorithm and solving the given problem). &lt;/p&gt;&#xA;" OwnerUserId="25508" LastEditorUserId="8501" LastEditDate="2016-10-24T11:46:25.623" LastActivityDate="2016-11-23T18:52:13.957" Title="Methodologies for predicting missing data" Tags="|predictive-modeling|missing-data|data-imputation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14723" PostTypeId="2" ParentId="14552" CreationDate="2016-10-23T15:30:28.423" Score="2" Body="&lt;p&gt;In the link that you post there is a portion that describes how to compute and plot errorbars. These bars are the confidence intervals. If you prefer to have this plotted as bands you could follow &lt;a href=&quot;http://blog.datadive.net/prediction-intervals-for-random-forests/&quot; rel=&quot;nofollow&quot;&gt;this example&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-23T15:30:28.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14725" PostTypeId="1" AcceptedAnswerId="14726" CreationDate="2016-10-23T18:39:58.357" Score="3" ViewCount="292" Body="&lt;p&gt;I am doing Tensorflow tutorial, getting what TF is. But I am confused about what neural network should I use in my work. I am looking at Single Layer Neural Network, CNN, RNN, and LSTM RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-----------------------What I'm going to do is...-----------------------&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a sensor which measures something and represents the result in 2 boolean ways. Here, they are Blue and Red, like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9J4nj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9J4nj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the sensor gives result values every 5minutes. If we pile up the values for each color, we can see some patterns: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/a26XX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a26XX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;number inside each circle represents the sequence of result values given from sensor. &#xA;(for example, 107 was given right after 106) when you see from 122 to 138, you can see decalcomanie-like pattern. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict the next result value, before the sensor imparts the result, with &lt;strong&gt;probability&lt;/strong&gt;. &#xA;Machine has to know what the next will be, based on patterns from past results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I may do supervised learning using past results. But I'm not sure which neural network or method is suitable. Thinking that this work needs pattern using past results (have to see context), and memorize past results, maybe LSTM RNN (long-short term memory recurrent neural network) would be suitable one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you tell me which one is suitable for this work?&lt;/p&gt;&#xA;" OwnerUserId="25515" LastEditorUserId="25515" LastEditDate="2016-10-23T18:47:58.527" LastActivityDate="2017-04-21T22:20:52.677" Title="In Tensorflow, What kind of neural network should I use?" Tags="|neural-network|tensorflow|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14726" PostTypeId="2" ParentId="14725" CreationDate="2016-10-23T19:18:11.603" Score="6" Body="&lt;p&gt;Sure, you can use an RNN. I would create two features for the past $k$ run lengths, as well as the length of the current run; e.g., just before t=150, the current run would be length 2 (red), and previous three runs would be (1,1,1) for red and (1,1,5) for blue. The intuition here is that the run lengths seem to follow some sort of exponential distribution, and you want to help the model estimate the scale parameter by feeding it samples of the length. You could additionally encode the past k events as a bit string, with 1 representing red and 0 representing blue. You have a classification problem, so you should use classification loss like the cross-entropy, and a softmax output layer to get your probabilities.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-10-23T19:18:11.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14727" PostTypeId="1" CreationDate="2016-10-23T23:11:23.583" Score="1" ViewCount="2715" Body="&lt;p&gt;I am currently trying to implemenent a RNN network for regression purposes, such that I can map a known input to a known output. &#xA;The problem in my case is that the input does not have static length, as it consist of samples of audio files, and the audio files have different length. But the output length is always consistent, and has length 14. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was due to this inconsistency of the input vector, I decided to use RNN in the first place. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using tensorflow at the moment, and it seems like that there is no pretty way of handling this issue. I tried a hack solution based on this &lt;a href=&quot;https://danijar.com/variable-sequence-lengths-in-tensorflow/&quot; rel=&quot;nofollow&quot;&gt;post&lt;/a&gt; somehow ended with the same issue as in the begining. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my implementation: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;from tensorflow.models.rnn import rnn_cell&#xA;from tensorflow.models.rnn import rnn&#xA;import numpy as np&#xA;import librosa&#xA;import glob&#xA;import matplotlib.pyplot as plt&#xA;from os import listdir&#xA;from os.path import isfile, join&#xA;import os&#xA;from os import walk&#xA;from os.path import splitext&#xA;from os.path import join&#xA;import time&#xA;rng = np.random&#xA;np.set_printoptions(threshold=np.nan)&#xA;import functools&#xA;&#xA;start_time = time.time()&#xA;&#xA;print &quot;Preprocessing&quot;&#xA;&#xA;def lazy_property(function):&#xA;    attribute = '_' + function.__name__&#xA;&#xA;    @property&#xA;    @functools.wraps(function)&#xA;    def wrapper(self):&#xA;        if not hasattr(self, attribute):&#xA;            setattr(self, attribute, function(self))&#xA;        return getattr(self, attribute)&#xA;    return wrapper&#xA;## Class definition ##&#xA;class VariableSequenceLabelling:&#xA;&#xA;    def __init__(self, data, target, num_hidden=200, num_layers=3):&#xA;        self.data = data&#xA;        self.target = target&#xA;        self._num_hidden = num_hidden&#xA;        self._num_layers = num_layers&#xA;        self.prediction&#xA;        self.error&#xA;        self.optimize&#xA;&#xA;    @lazy_property&#xA;    def length(self):&#xA;        used = tf.sign(tf.reduce_max(tf.abs(self.data), reduction_indices=2))&#xA;        length = tf.reduce_sum(used, reduction_indices=1)&#xA;        length = tf.cast(length, tf.int32)&#xA;        return length&#xA;&#xA;    @lazy_property&#xA;    def prediction(self):&#xA;        # Recurrent network.&#xA;        output, _ = tf.nn.dynamic_rnn(&#xA;             rnn_cell.GRUCell(self._num_hidden),&#xA;            self.data,&#xA;            dtype=tf.float32,&#xA;            sequence_length=self.length,&#xA;        )&#xA;        # Softmax layer.&#xA;        max_length = int(self.target.get_shape()[1])&#xA;        num_classes = int(self.target.get_shape()[2])&#xA;        weight, bias = self._weight_and_bias(self._num_hidden, num_classes)&#xA;        # Flatten to apply same weights to all time steps.&#xA;        output = tf.reshape(output, [-1, self._num_hidden])&#xA;        prediction = tf.nn.softmax(tf.matmul(output, weight) + bias)&#xA;        prediction = tf.reshape(prediction, [-1, max_length, num_classes])&#xA;        return prediction&#xA;&#xA;    @lazy_property&#xA;    def cost(self):&#xA;        # Compute cross entropy for each frame.&#xA;        cross_entropy = self.target * tf.log(self.prediction)&#xA;        cross_entropy = -tf.reduce_sum(cross_entropy, reduction_indices=2)&#xA;        mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))&#xA;        cross_entropy *= mask&#xA;        # Average over actual sequence lengths.&#xA;        cross_entropy = tf.reduce_sum(cross_entropy, reduction_indices=1)&#xA;        cross_entropy /= tf.cast(self.length, tf.float32)&#xA;        return tf.reduce_mean(cross_entropy)&#xA;&#xA;    @lazy_property&#xA;    def optimize(self):&#xA;        learning_rate = 0.0003&#xA;        optimizer = tf.train.AdamOptimizer(learning_rate)&#xA;        return optimizer.minimize(self.cost)&#xA;&#xA;    @lazy_property&#xA;    def error(self):&#xA;        mistakes = tf.not_equal(&#xA;            tf.argmax(self.target, 2), tf.argmax(self.prediction, 2))&#xA;        mistakes = tf.cast(mistakes, tf.float32)&#xA;        mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))&#xA;        mistakes *= mask&#xA;        # Average over actual sequence lengths.&#xA;        mistakes = tf.reduce_sum(mistakes, reduction_indices=1)&#xA;        mistakes /= tf.cast(self.length, tf.float32)&#xA;        return tf.reduce_mean(mistakes)&#xA;&#xA;    @staticmethod&#xA;    def _weight_and_bias(in_size, out_size):&#xA;        weight = tf.truncated_normal([in_size, out_size], stddev=0.01)&#xA;        bias = tf.constant(0.1, shape=[out_size])&#xA;        return tf.Variable(weight), tf.Variable(bias)&#xA;&#xA;#######################&#xA;#Converting file to .wav from .sph file format... God dammit!!!&#xA;&#xA;#with open(train_filelist, 'r') as train_filelist, open(test_filelist, 'r') as test_filelist:&#xA;    #train_mylist = train_filelist.read().splitlines()&#xA;    #test_mylist = test_filelist.read().splitlines()&#xA;    #for line in train_mylist:&#xA;        #new_line = ' '.join(reversed(line))&#xA;        #index_start = new_line.find('h')&#xA;        #index_end = new_line.find('/')&#xA;        #edited_line = ''.join(reversed(new_line[index_start+5:index_end])).strip().replace(&quot; &quot;,&quot;&quot;)&#xA;        #new_file = edited_line + 'wav'&#xA;        #os.system(line + ' &amp;gt;&amp;gt; ' + dnn_train + new_file)&#xA;    #for line in test_mylist:&#xA;        #new_line = ' '.join(reversed(line))&#xA;        #index_start = new_line.find('h')&#xA;        #index_end = new_line.find('/')&#xA;        #edited_line = ''.join(reversed(new_line[index_start+5:index_end])).strip().replace(&quot; &quot;,&quot;&quot;)&#xA;        #new_file = edited_line + 'wav'&#xA;        #os.system(line + ' &amp;gt;&amp;gt; ' + dnn_test + new_file)&#xA;&#xA;&#xA;path_train =  &quot;/home/JoeS/kaldi-trunk/egs/start/s5/data/train&quot;&#xA;path_test =  &quot;/home/JoeS/kaldi-trunk/egs/start/s5/data/test&quot;&#xA;dnn_train = &quot;/home/JoeS/kaldi-trunk/dnn/train/&quot;&#xA;dnn_test = &quot;/home/JoeS/kaldi-trunk/dnn/test/&quot;&#xA;dnn = &quot;/home/JoeS/kaldi-trunk/dnn/&quot;&#xA;path  = &quot;/home/JoeS/kaldi-trunk/egs/start/s5/data/&quot;&#xA;MFCC_dir = &quot;/home/JoeS/kaldi-trunk/egs/start/s5/mfcc/raw_mfcc_train.txt&quot;&#xA;&#xA;train_filelist = path_train+&quot;/wav_train.txt&quot;&#xA;test_filelist = path_test+&quot;/wav_test.txt&quot;&#xA;&#xA;os.chdir(path)&#xA;&#xA;def binify(number):&#xA;    divider = (36471330-10533580)/6&#xA;    if number &amp;gt;= divider*0 and number &amp;lt; divider*1:&#xA;        return 1&#xA;    if number &amp;gt;= divider*1 and number &amp;lt; divider*2:&#xA;        return 2&#xA;    if number &amp;gt;= divider*2 and number &amp;lt; divider*3:&#xA;        return 3&#xA;    if number &amp;gt;= divider*3 and number &amp;lt; divider*4:&#xA;        return 4&#xA;    if number &amp;gt;= divider*5 and number &amp;lt; divider*6:&#xA;        return 5&#xA;    if number &amp;gt;= divider*6:&#xA;        return 6&#xA;&#xA;def find_all(a_str, sub):&#xA;    start = 0&#xA;    while True:&#xA;        start = a_str.find(sub, start)&#xA;        if start == -1: return&#xA;        yield start&#xA;        start += len(sub) # use start += 1 to find overlapping matches&#xA;&#xA;def load_sound_files(file_paths ,  names_input, data_input):&#xA;    raw_sounds = []&#xA;    names_output = []&#xA;    data_output = []&#xA;    class_output = []&#xA;    for fp in file_paths:&#xA;        X,sr = librosa.load(fp)&#xA;        raw_sounds.append(X)&#xA;        index = list(find_all(fp,'-'))&#xA;        input_index = names_input.index(fp[index[1]+1:index[2]])&#xA;        names_output.append(names_input[input_index])&#xA;        data_output.append(data_input[input_index])&#xA;        class_output.append(binify(data_input[input_index][0]))&#xA;    return raw_sounds, names_output, data_output, class_output&#xA;&#xA;def generate_list_of_names_data(file_path):&#xA;    # Proprocess&#xA;    # extract name and data&#xA;    name = []&#xA;    data = []&#xA;    with open(MFCC_dir) as mfcc_feature_list:&#xA;        content = [x.strip('\n') for x in mfcc_feature_list.readlines()] # remove endlines&#xA;        start_index_data = 0&#xA;        end_index_data = 2&#xA;        for number in range(0,42):&#xA;            start = list(find_all(content[start_index_data],'['))[0]&#xA;            end = list(find_all(content[end_index_data],']'))[0]&#xA;            end_name = list(find_all(content[start_index_data],' '))[0]&#xA;            substring_data = content[start_index_data][start+1 :]+content[end_index_data][: end]&#xA;            substring_name = content[start_index_data][:end_name]&#xA;            arr = np.array(substring_data.split(), dtype = float)&#xA;            data.append(arr)&#xA;            name.append(substring_name)&#xA;            start_index_data = start_index_data + +3&#xA;            end_index_data = end_index_data +3&#xA;    return name, data&#xA;&#xA;files_train_path = [dnn_train+f for f in listdir(dnn_train) if isfile(join(dnn_train, f))]&#xA;files_test_path = [dnn_test+f for f in listdir(dnn_test) if isfile(join(dnn_test, f))]&#xA;&#xA;files_train_name = [f for f in listdir(dnn_train) if isfile(join(dnn_train, f))]&#xA;files_test_name = [f for f in listdir(dnn_test) if isfile(join(dnn_test, f))]&#xA;&#xA;os.chdir(dnn_train)&#xA;&#xA;train_name,train_data = generate_list_of_names_data(files_train_path)&#xA;train_data, train_names, train_output_data, train_class_output = load_sound_files(files_train_path,train_name,train_data)&#xA;&#xA;max_length = 0 ## Used for variable sequence input&#xA;&#xA;for element in train_data:&#xA;    if element.size &amp;gt; max_length:&#xA;        max_length = element.size&#xA;&#xA;NUM_EXAMPLES = len(train_data)/2&#xA;&#xA;test_data = train_data[NUM_EXAMPLES:]&#xA;test_output = train_output_data[NUM_EXAMPLES:]&#xA;&#xA;train_data = train_data[:NUM_EXAMPLES]&#xA;train_output = train_output_data[:NUM_EXAMPLES]&#xA;print(&quot;--- %s seconds ---&quot; % (time.time() - start_time))&#xA;##-------------------MAIN----------------------------##&#xA;&#xA;if __name__ == '__main__':&#xA;    data = tf.placeholder(tf.float32, [None, max_length, 1])&#xA;    target = tf.placeholder(tf.float32, [None, 14, 1])&#xA;    model = VariableSequenceLabelling(data, target)&#xA;    sess = tf.Session()&#xA;    sess.run(tf.initialize_all_variables())&#xA;    for epoch in range(10):&#xA;        for sample_set in range(100):&#xA;            batch_train = train_data[sample_set]&#xA;            batch_target = train_output[sample_set]&#xA;            sess.run(model.optimize, {data: batch_train, target: batch_target})&#xA;        test_set = test_data[epoch]&#xA;        test_set_output = test_output[epoch]&#xA;        error = sess.run(model.error, {data: test_set, target: test_set_output})&#xA;        print('Epoch {:2d} error {:3.1f}%'.format(epoch + 1, 100 * error))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error message I am receiving is that &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;tensorflow_datapreprocess_mfcc_extraction_rnn.py&quot;, line 239, in &amp;lt;module&amp;gt;&#xA;    sess.run(model.optimize, {data: batch_train, target: batch_target})&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 340, in run&#xA;    run_metadata_ptr)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py&quot;, line 553, in _run&#xA;    % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))&#xA;ValueError: Cannot feed value of shape (63945,) for Tensor u'Placeholder:0', which has shape '(?, 138915, 1)'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess debugging is more a stackoverflow question than a data science question, so instead of asking for help solving my issue regarding the code, would I like to know whether there is other framework which do natively support this, such I don't need to hack my solution.  I added my code such you know the structure of my input and output data.  It would be very appreciated if I could keep the structure. &lt;/p&gt;&#xA;" OwnerUserId="25188" LastActivityDate="2016-10-24T01:27:15.950" Title="Tensorflow RNN network with variable sequence length?" Tags="|python|neural-network|regression|tensorflow|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14728" PostTypeId="2" ParentId="14727" CreationDate="2016-10-24T01:27:15.950" Score="3" Body="&lt;p&gt;Since tensors can be only of a fixed size, you have to zero-pad your sequences (usually from the left) to the maximum occurring length with something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;max_len = max([len(x) for x in sequences])&#xA;sequences = [np.pad(x, (max_len - len(x), 0), 'constant', constant_values = (0.,0.)) for x in sequences]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2016-10-24T01:27:15.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14730" PostTypeId="1" AcceptedAnswerId="14768" CreationDate="2016-10-24T05:34:37.707" Score="0" ViewCount="751" Body="&lt;p&gt;I'm trying to train a classifier to classify text from a chat between 2 users so later on I can predict who of the two users is more likely to say X sentence/word. To get there I mined the text from the chat log and ended up with two arrays of words, &lt;code&gt;UserA_words&lt;/code&gt; and &lt;code&gt;UserB_words&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which classifier should I use fot this purpose and what structure the training data should have? I've researched for the bag of words structure but dont know exactly how to train a classifier with data in that format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify this last point, for now I have the data in a dict like &lt;code&gt;{&quot;hello&quot;:34, &quot;how&quot;:12}&lt;/code&gt; and so on, being the terms word:frequency of each user. As far as I know, there is no way to use this two dicts as a classifier fit input. So, how do I transform this 2 dicts into an array that I can use to train a classifier (let's say I want to use a gaussian Naive Bayes just for the sake of the example)&lt;/p&gt;&#xA;" OwnerUserId="25526" LastEditorUserId="8501" LastEditDate="2016-10-27T09:34:24.390" LastActivityDate="2016-10-27T09:34:24.390" Title="Which approach for user classification on chat text (classifier, representation, features)?" Tags="|machine-learning|python|classification|feature-engineering|text|" AnswerCount="1" CommentCount="12" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14731" PostTypeId="2" ParentId="14686" CreationDate="2016-10-24T07:03:32.070" Score="0" Body="&lt;p&gt;In Orange, you can do something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hkZNu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hkZNu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This takes the network, which is already containing class you'd like to predict, then training (or testing) the learner in Test &amp;amp; Score and evaluating it in Confusion Matrix. Then you can see misclassifications directly in the network graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a bunch of other learners and evaluation methods available. A big plus is also interactive data exploration (see how you can input wrongly classified data into Network Explorer?). However, there's no dashboard available yet. We make do with opening several windows side by side.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's just my 2¢ on Orange. I suggest you to at least try all of them and see which one works best for you. :)&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2016-10-24T07:03:32.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14732" PostTypeId="2" ParentId="13641" CreationDate="2016-10-24T08:22:29.730" Score="1" Body="&lt;p&gt;Here is a pretty detailed summary on &lt;a href=&quot;http://www.svds.com/learning-imbalanced-classes/&quot; rel=&quot;nofollow&quot;&gt;Handling Imbalanced Classes&lt;/a&gt;. Since you mentioned R, you might want to take a look at &lt;a href=&quot;https://cran.r-project.org/web/packages/unbalanced/unbalanced.pdf&quot; rel=&quot;nofollow&quot;&gt;&lt;code&gt;unbalanced&lt;/code&gt;&lt;/a&gt; package. &lt;/p&gt;&#xA;" OwnerUserId="1131" LastActivityDate="2016-10-24T08:22:29.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14734" PostTypeId="1" AcceptedAnswerId="14738" CreationDate="2016-10-24T13:13:44.127" Score="11" ViewCount="10457" Body="&lt;p&gt;One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your classifier and a subset of the ranked features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your features are &lt;code&gt;A, B, C, D, E&lt;/code&gt;, and if they are ranked as follow &lt;code&gt;D,B,C,E,A&lt;/code&gt;, then you calculate the accuracy using &lt;code&gt;D&lt;/code&gt;, then &lt;code&gt;D, B&lt;/code&gt; then &lt;code&gt;D, B, C&lt;/code&gt;, then &lt;code&gt;D, B, C, E&lt;/code&gt;... until your accuracy starts decreasing. Once it starts decreasing, you stop adding features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TmGB0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TmGB0.png&quot; alt=&quot;Example1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In example1 (above), you would pick features &lt;code&gt;F, C, D, A&lt;/code&gt; and drop the other features as they decrease your accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That methodology assumes that adding more features to your model increases the accuracy of your classifier until a certain point after which adding additional features decreases the accuracy (as seen in example 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nrCAT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nrCAT.png&quot; alt=&quot;Example2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a scenario such as this one, how do you pick your features? Do you only pick &lt;code&gt;F&lt;/code&gt; and drop the rest? Do you have any idea why the accuracy would decrease and then increase?&lt;/p&gt;&#xA;" OwnerUserId="25537" LastEditorUserId="15527" LastEditDate="2016-10-24T14:45:54.630" LastActivityDate="2016-10-25T00:22:15.073" Title="Feature selection and classification accuracy relation" Tags="|machine-learning|feature-selection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14736" PostTypeId="2" ParentId="14734" CreationDate="2016-10-24T14:35:52.557" Score="1" Body="&lt;p&gt;&lt;strong&gt;About the specific question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should not expect a specific behavior (increase and then decrease of accuracy) while you select subset of features, since this will be totally dependent on the problem (and each model)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you calculate the variable importance of features, you are taking into account the contribution of all the the features at the same time. Once you select a subset of features and build a new model, you will get a different representation or modelling of the problem (which does not take into account the other features - informative or not -). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you want to select the best number of features. This will also depend from your problem and the characteristics or conditions you need to fulfill. If you actually need to have the fewer features possible while optimizing prediction accuracy, you can select the lowest number of features that achieves the lowest error... and, if you have different cases with very similar errors, then pick a threshold, observe the top cases whose pairwise difference of errors is lower than the threshold, and select one (for example the one with lower number of features - since the errors are marginally the same -).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Consider Recursive Feature Elimination&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The method you are using might not be the most stable approach. You should consider trying something like recursive feature elimination (RFE), a wrapper method where you build the classifier, rank all the features, remove the worst and rebuild the model on the remaining features. Then you repeat the method again. This will tend to be more stable...and you should expect different ranking everytime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Variance is also a critical factor&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond the actual error (or accuracy) the model is giving you with each subset, you should consider to build each model through a cross-validation procedure and take into account both the mean error of the folds, and the standard deviation of these errors. If the standard deviation is high, then the selected subset of features is not stable, and will tend to vary plenty when testing with unseen data. This is important to evaluate the expected generalization capabilities of the model, and could be helpful for deciding between models (built with different subsets).&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2016-10-24T14:35:52.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14737" PostTypeId="2" ParentId="14734" CreationDate="2016-10-24T14:57:16.737" Score="1" Body="&lt;p&gt;You need to remove both redundant and irrelevant features from your data set. It can be seen that there are irrelevant and redundant features in your data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend you to look at the minimum Redundancy Maximum Relevance Feature Selection(MRMR) algorithm. It is a very popular and powerful filter before your train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible, but this will lead to more complex model.&lt;/p&gt;&#xA;" OwnerUserId="25543" LastActivityDate="2016-10-24T14:57:16.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14738" PostTypeId="2" ParentId="14734" CreationDate="2016-10-24T15:04:58.467" Score="6" Body="&lt;p&gt;feature selection involves several approaches just like methods for machine learning. Idea is to keep most relevant but not redundant feature for predictive model that can yield optimal accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I can not see which method you are using for feature selection but assuming that you are not taking account of multivariate nature of feature dependency. Say you have N features, likely reason that your model accuracy drops after n top feature(s) but improves by adding n+k (where n &amp;lt; k &amp;lt; N when features are in descending order based on information gain) is due to inter-dependency (more relevance and less redundancy) of top n and k features. Univariate feature selection does not necessarily get optimal model accuracy when features are inter-dependent and not mutually exclusive. From philosophical point of view, set of optimal features is analogous to a quote by Aristotle: &quot;The whole is greater than the sum of its parts&quot;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For optimal feature selection, I often is Caret package in &lt;code&gt;R&lt;/code&gt; language where one may do feature selection using &lt;a href=&quot;http://machinelearningmastery.com/feature-selection-with-the-caret-r-package/&quot; rel=&quot;noreferrer&quot;&gt;recursive feature elimination&lt;/a&gt; (RFE) among several other approaches. There is also a package called &lt;a href=&quot;https://cran.r-project.org/web/packages/mRMRe/index.html&quot; rel=&quot;noreferrer&quot;&gt;mRMRe&lt;/a&gt; to do feature selection based on maximum relevance, minimal redundancy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&lt;br&gt;&#xA;Samir&lt;/p&gt;&#xA;" OwnerUserId="24275" LastActivityDate="2016-10-24T15:04:58.467" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14740" PostTypeId="2" ParentId="14721" CreationDate="2016-10-24T16:55:22.643" Score="3" Body="&lt;p&gt;As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Mean/ Mode/ Median Imputation&lt;/strong&gt;: Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the data set to assist in estimating the missing values. Mean / Mode / Median imputation is one of the most frequently used methods. It consists of replacing the missing data for a given attribute by the mean or median (quantitative attribute) or mode (qualitative attribute) of all known values of that variable. This can further be classified as generalized and similar case imputation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Prediction Model:&lt;/strong&gt;  Prediction model is one of the sophisticated method for handling missing data. Here, we create a predictive model to estimate values that will substitute the missing data.  In this case, we divide our data set into two sets: One set with no missing values for the variable and another one with missing values. First data set become training data set of the model while second data set with missing values is test data set and variable with missing values is treated as target variable. Next, we create a model to predict target variable based on other attributes of the training data set and populate missing values of test data set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;KNN(k-nearest neighbor) Imputation:&lt;/strong&gt; In this method of imputation, the missing values of an attribute are imputed using the given number of attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-10-24T16:55:22.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14741" PostTypeId="1" CreationDate="2016-10-24T18:00:35.610" Score="1" ViewCount="140" Body="&lt;p&gt;I have a biased set of samples going into a binary classification sklearn pipeline, &lt;em&gt;white&lt;/em&gt; and &lt;em&gt;black&lt;/em&gt; samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy for me to fetch as many &lt;em&gt;black&lt;/em&gt; samples as required, while &lt;em&gt;whites&lt;/em&gt; are a bit harder to get.&#xA;I suspect the &lt;em&gt;black&lt;/em&gt; set is unbalanced and there for I have an incentive to preprocess as many &lt;em&gt;black&lt;/em&gt; samples as my hardware permits, and pick the most representative samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The feature set (after dropping completely invariant features) is about 28k features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best approach I came up with is to cluster my &lt;em&gt;black&lt;/em&gt; samples to a percentage of my white samples. For example say I have 500k &lt;em&gt;white&lt;/em&gt; samples, I would then cluster the &lt;em&gt;black&lt;/em&gt; samples to 100k clusters and pick five samples (randomly?) out of each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I have a transformer that will cluster the samples and mask out the unneeded samples in sklearn? Is there something ready for such tasks or should I build my own transformer based on the clustering algorithm?&lt;/li&gt;&#xA;&lt;li&gt;Can you point at any potential issue with what I plan to do? I think one potential risk here is not clustering the &lt;em&gt;white&lt;/em&gt; samples, but since I'm not getting enough of those anyway...&lt;/li&gt;&#xA;&lt;li&gt;What clustering algorithms might be a good fit? I'm looking a different clustering algorithms that accept the number of clusters as parameter right now I plan to experiment with most and see which ones work best, but a direction will be appreciated. &lt;/li&gt;&#xA;&lt;li&gt;Additionally, I'd like to read suggestions about other approaches to sample subsampling in my case.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13714" LastEditorUserId="381" LastEditDate="2016-10-24T18:37:32.580" LastActivityDate="2016-11-01T11:47:48.060" Title="Sample selection through clustering" Tags="|clustering|scikit-learn|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14742" PostTypeId="2" ParentId="14415" CreationDate="2016-10-24T18:37:05.820" Score="34" Body="&lt;p&gt;For binary classification, the code for accuracy metric is:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;K.mean(K.equal(y_true, K.round(y_pred)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;which suggests that 0.5 is the threshold to distinguish between classes. &lt;em&gt;y_true&lt;/em&gt; should of course be 1-hots in this case.&lt;/p&gt;&#xA;&lt;p&gt;It's a bit different for categorical classification:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;K.mean(K.equal(K.argmax(y_true, axis=-1), K.argmax(y_pred, axis=-1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;which means &amp;quot;how often predictions have maximum in the same spot as true values&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;There is also an option for top-k categorical accuracy, which is similar to one above, but calculates how often target class is within the top-k predictions.&lt;/p&gt;&#xA;" OwnerUserId="25504" LastEditorUserId="29169" LastEditDate="2021-02-08T02:03:49.750" LastActivityDate="2021-02-08T02:03:49.750" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="14743" PostTypeId="2" ParentId="14721" CreationDate="2016-10-24T18:51:01.010" Score="1" Body="&lt;p&gt;Ideally, there is no such approach or tool which can assure you that.. But as you asked there are surely pros-cons of every approach/tool. They're several approaches that you can follow. However, the best approach always depends on your goal and these factors--&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Types of &lt;code&gt;Missing Values&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Type of &lt;code&gt;Data&lt;/code&gt; you have.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Bias&lt;/code&gt; in the analyses should be minimized. &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Available Information&lt;/code&gt; must be maximized (for the researcher). &lt;/li&gt;&#xA;&lt;li&gt;It must give reasonable estimates of &lt;code&gt;Variability&lt;/code&gt; and &lt;code&gt;Error&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So keep these factors and your goals in mind before approaching to any. For more details, I'll suggest you to go through these blog posts--&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/How-do-you-handle-missing-data-statistics-What-imputation-techniques-do-you-recommend-or-follow&quot; rel=&quot;nofollow&quot;&gt;How do you handle missing data(statistics)? What imputation techniques do you recommend or follow?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.theanalysisfactor.com/missing-data-two-recommended-solutions/&quot; rel=&quot;nofollow&quot;&gt;Two Recommended Solutions for Missing Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.statisticssolutions.com/multiple-imputation-for-missing-data/&quot; rel=&quot;nofollow&quot;&gt;Multiple Imputation for Missing Data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope it helps! &lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-24T18:51:01.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14744" PostTypeId="1" CreationDate="2016-10-24T21:13:33.707" Score="12" ViewCount="4378" Body="&lt;h2&gt;The question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;How do I predict the rating for a new user in an ALS model trained in  Spark? (New = not seen during training time)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I'm following the official Spark ALS tutorial here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html&quot;&gt;http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm able to build a good recommender with a decent MSE but I'm struggling with how to input new data to the model. The tutorial changes the first user's ratings prior to training, but this is really a hack. They give the following hint:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;9.2. Augmenting matrix factors: &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In this tutorial, we add your ratings to the training set. A better way to get the recommendations for you&#xA;  is training a matrix factorization model first and then augmenting the&#xA;  model using your ratings. If this sounds interesting to you, you can&#xA;  take a look at the implementation of MatrixFactorizationModel and see&#xA;  how to update the model for new users and new movies.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The implementation does not help me at all though. Ideally, I'm looking for something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;predictions = model.predictAllNew(newinput)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But not such method exists. I could go and modify the original RDD, but I think that would require me to retrain the model, so that wouldn't be an ideal solution either. Surely there must be a more elegant way?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Where I am right now:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I think I need to find the latent representation of the new vector. According to the &lt;a href=&quot;http://yifanhu.net/PUB/cf.pdf&quot;&gt;original paper&lt;/a&gt; we can compute this like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_u = (Y^T C^u Y + \lambda I)^{-1} Y^T C^u p(u)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I calculate using the values in the paper, it doesn't match the values from the model. I fix alpha and the regularization parameter, but I think the MLLIB implentation has a different $C^u$ implementation. It's defined &lt;a href=&quot;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala&quot;&gt;here&lt;/a&gt; (see line 1304), but not being adept at Scala, this is very hard to reverse engineer for me ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current attempt:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;V = model.productFeatures().map(lambda x: (x[1])).collect() #product latent matrix Y&#xA;&#xA;Cui =  alpha * np.abs(newinput)&#xA;Cui =  (1. + Cui) / (Cui)&#xA;Cui[np.where(newinput == 0)] = 0&#xA;Cui = np.diag(Cui)&#xA;&#xA;lambdaI = len(np.where(newinput!=0)) * regularization_parameter * np.eye(np.shape(V)[1]) #&#xA;term   = np.dot(np.dot(Vt,Cui),V)+lambdaI&#xA;term   = np.dot(np.linalg.inv(term),Vt)&#xA;term   = np.dot(term,Cui)&#xA;term   = np.dot(term,newinput)&#xA;latentinput = term&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this doesn't match.&lt;/p&gt;&#xA;" OwnerUserId="5257" LastEditorUserId="5257" LastEditDate="2016-10-25T09:45:40.553" LastActivityDate="2016-10-25T10:35:35.470" Title="Spark ALS: recommending for new users" Tags="|apache-spark|recommender-system|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14745" PostTypeId="2" ParentId="10542" CreationDate="2016-10-24T21:50:36.160" Score="0" Body="&lt;p&gt;You should look at the &lt;a href=&quot;https://osf.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open Science Framework&lt;/a&gt;:Here is a &lt;a href=&quot;https://www.youtube.com/watch?v=G2VH8wlN_48&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introductory YouTube video&lt;/a&gt; This Open Source Project addresses all your concerns.&lt;/p&gt;&#xA;&lt;p&gt;Structured projects:&lt;/p&gt;&#xA;&lt;p&gt;Keep all your files, data, and protocols in one centralized location. No more trawling emails to find files or scrambling to recover from lost data. Secure Cloud&lt;/p&gt;&#xA;&lt;p&gt;Control access:&lt;/p&gt;&#xA;&lt;p&gt;You control which parts of your project are public or private making it easy to collaborate with the worldwide community or just your team. Project-level Permissions&lt;/p&gt;&#xA;&lt;p&gt;Respect for your workflow:&lt;/p&gt;&#xA;&lt;p&gt;Connect your favorite third party services directly to the Open Science Framework. 3rd Party Integrations&lt;/p&gt;&#xA;" OwnerUserId="24409" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-10-24T21:50:36.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14746" PostTypeId="2" ParentId="10847" CreationDate="2016-10-25T00:18:48.493" Score="0" Body="&lt;p&gt;Implicit feedback simply means that given a ratings-dataset, the zeroes are treated as &quot;nothing was rated&quot; as opposed to &quot;the user rated this as a zero&quot;. The algorithm they use is based on the one in &lt;a href=&quot;http://yifanhu.net/PUB/cf.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt;. The code is pretty difficult to read but available &lt;a href=&quot;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, if you do have binary product choice variables (i.e., user A chose product B, but not the actual ratings), the implicit training could also work as is, even if was not originally designed for it.&lt;/p&gt;&#xA;" OwnerUserId="5257" LastActivityDate="2016-10-25T00:18:48.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14747" PostTypeId="2" ParentId="14734" CreationDate="2016-10-25T00:22:15.073" Score="1" Body="&lt;p&gt;Generally there are three classes of feature selection algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Filter methods&lt;/strong&gt; which analyze the intrinsic properties of the data and assign a score to each feature, not involving any model.  Some examples are fold change, student t-test.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Wrapper methods&lt;/strong&gt; which different subsets of features are selected through the specific algorithms. We then fit into the classification or regression model to evaluate each selection and pick the one with best fitness value. Some examples are Genetic Algorithm for feature selection, Monte Carlo optimization for feature selection, forward/backward stepwise selection.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Embedded methods&lt;/strong&gt; which allows the model itself to pick the features having best contribution to the fitness of the model. Typical ones are LASSO, ridge regression.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is a great article in details of &lt;a href=&quot;http://machinelearningmastery.com/an-introduction-to-feature-selection/&quot; rel=&quot;nofollow&quot;&gt;introduction to feature selection&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23983" LastActivityDate="2016-10-25T00:22:15.073" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14750" PostTypeId="2" ParentId="14744" CreationDate="2016-10-25T10:35:35.470" Score="10" Body="&lt;p&gt;Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, once you have any data, &lt;em&gt;and&lt;/em&gt; you can rebuild the model to incorporate the user, you can make recommendations. You can do that in Spark but you already know that. This will take too long if you need to add information about new users at runtime. The technique you want is called &quot;fold-in&quot;, used to figure out what the new user vector is (approximately) given the items the user interacts with. It's just some linear algebra and does follow from the equation you give.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I dug out an old slide which might help:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RGhq2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RGhq2.png&quot; alt=&quot;ALS Fold-In&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;Cu&quot; is not really different. I added an 'extension' to handle the case of negative input, but it's the same for positive input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an implementation of fold-in, although I think it's going to be too dense to be of much value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/OryxProject/oryx/blob/2c01d496f93f2825a076eb7fe492aa39a5290aa6/app/oryx-app-common/src/main/java/com/cloudera/oryx/app/als/ALSUtils.java#L74&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/OryxProject/oryx/blob/2c01d496f93f2825a076eb7fe492aa39a5290aa6/app/oryx-app-common/src/main/java/com/cloudera/oryx/app/als/ALSUtils.java#L74&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the new user vector implied by a user-item interaction is fairly easy linear algebra. The tricky part I've found is deciding how much to weight it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that is a push in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-10-25T10:35:35.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14752" PostTypeId="1" AcceptedAnswerId="14802" CreationDate="2016-10-25T11:22:44.400" Score="1" ViewCount="408" Body="&lt;p&gt;Has anyone tried to install Torch on top of Gentoo Linux Distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm very familiar with this distribution, but I'm totally new at torch. Googling a little, I found that Gentoo is not officially supported by torch (take a look at the install-deps script, line 142: &lt;a href=&quot;https://github.com/torch/distro/blob/master/install-deps&quot; rel=&quot;nofollow&quot;&gt;https://github.com/torch/distro/blob/master/install-deps&lt;/a&gt;), but some users have successfully installed it, even if the corresponding tutorial is a little bit aged (&lt;a href=&quot;http://spotofdata.com/installing-torch-gentoo/&quot; rel=&quot;nofollow&quot;&gt;http://spotofdata.com/installing-torch-gentoo/&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if someone has tried the tandem gentoo/torch and, in that case, if her/he would suggest it in spite of the more traditional ones (arch/torch or ubuntu/torch) and if there are some issues/recommendations in using it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21686" LastActivityDate="2016-10-29T22:10:34.240" Title="Torch on Gentoo" Tags="|torch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14755" PostTypeId="1" CreationDate="2016-10-25T14:57:24.440" Score="6" ViewCount="2828" Body="&lt;p&gt;Logistic regression is a part in a simulation pipeline that I use for some scenario analysis. The dataset that this is based on is not small but relatively noisy, and only one explanatory variable/feature. Of course I can say something about this uncertainty using frequentist or Bayesian methods but I would like to use this in the sequential simulation step as well, to get a fairer final estimate. What I'm planning on doing should work but is somewhat computationally expensive and I would like to know if someone has a better idea.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Approximate distribution of the coefficients of logistic regression with MCMC&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now simulate this &lt;strong&gt;n&lt;/strong&gt; times:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sample one of the two coefficients from this distribution&lt;/li&gt;&#xA;&lt;li&gt;Fix this coefficient and use MCMC again to get approximated conditional distribution over second coefficient (could I get this in a better way? This seems highly correlated with the first coefficient)&lt;/li&gt;&#xA;&lt;li&gt;Do the rest of the simulation run using this sampled model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I feel like there should be a better way but I'm uncertain.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-17T14:38:42.753" Title="Modeling uncertainty from Logistic Regression" Tags="|logistic-regression|simulation|bayesian|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14756" PostTypeId="2" ParentId="12063" CreationDate="2016-10-25T20:31:24.380" Score="1" Body="&lt;p&gt;Have a look at &lt;a href=&quot;https://github.com/phanein/deepwalk&quot; rel=&quot;nofollow&quot;&gt;Deep Walk&lt;/a&gt;.  It's essentially word-to-vec applied to &quot;sentences&quot; created from a graph (nodes and edges).  If you can express your data as a graph, you can apply it.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-10-25T20:31:24.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14757" PostTypeId="1" AcceptedAnswerId="14758" CreationDate="2016-10-26T01:50:35.687" Score="3" ViewCount="6252" Body="&lt;p&gt;I'm computing cost in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = tf.nn.softmax_cross_entropy_with_logits(y, y_) &#xA;cost = tf.reduce_mean(cross_entropy); &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the first cost, I am getting 0.693147, which is to be expected on a binary classification when parameters/weights are initialized to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using one_hot labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, after completing a training epoch using stochastic gradient descent I am finding a cost of greater than 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this to be expected?&lt;/p&gt;&#xA;" OwnerUserId="25583" LastActivityDate="2016-10-26T03:37:48.727" Title="Cost of greater than 1, is there an error?" Tags="|machine-learning|tensorflow|gradient-descent|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14758" PostTypeId="2" ParentId="14757" CreationDate="2016-10-26T03:37:48.727" Score="3" Body="&lt;p&gt;The following piece of code does essentially what TF's &lt;code&gt;softmax_cross_entropy_with_logits&lt;/code&gt; functions does (crossentropy on softmaxed &lt;code&gt;y_&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import scipy as sp&#xA;import numpy as np&#xA;&#xA;def softmax(x):&#xA;    e_x = np.exp(x - np.max(x))&#xA;    return e_x / e_x.sum(axis=0)&#xA;&#xA;def crossentropy(true, pred):    &#xA;    epsilon = 1e-15&#xA;&#xA;    pred = sp.maximum(epsilon, pred)&#xA;    pred = sp.minimum(1-epsilon, pred)&#xA;&#xA;    ll = -sum(&#xA;        true * sp.log(pred) + \&#xA;            sp.subtract(1,true) * \&#xA;            sp.log(sp.subtract(1, pred))&#xA;    ) / len(true)&#xA;&#xA;    return ll&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;==&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;true = [1., 0.]&#xA;pred = [5.0, 0.5]&#xA;&#xA;true = softmax(true)&#xA;pred = softmax(pred)&#xA;&#xA;print true&#xA;print pred&#xA;&#xA;print crossentropy(true, pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;==&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.73105858  0.26894142]&#xA;[ 0.98901306  0.01098694]&#xA;1.22128414101&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see there is no reason why crossentropy on binary classification cannot be &gt; 1 and it's not hard to come up with such example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;** Crossentropy above is calculated as in &lt;a href=&quot;https://www.kaggle.com/wiki/LogarithmicLoss&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/wiki/LogarithmicLoss&lt;/a&gt;, softmax as in &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Softmax_function&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPD: there is a great explanation of what it means when logloss is &gt; 1 at SO: &lt;a href=&quot;https://stackoverflow.com/a/35015188/1166478&quot;&gt;https://stackoverflow.com/a/35015188/1166478&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25504" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-26T03:37:48.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14760" PostTypeId="1" AcceptedAnswerId="14764" CreationDate="2016-10-26T08:24:08.103" Score="3" ViewCount="249" Body="&lt;p&gt;I have a weight function f that outputs a numeric weighting for a sample s. I also have an ordered set of samples S where the weight of each sample s in set S varies greatly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I create n splitting points so that each split is weighted approximately the same? What kind of methodology, algorithms or models could I use to achieve this?&lt;/p&gt;&#xA;" OwnerUserId="25590" LastActivityDate="2016-10-26T12:08:27.073" Title="How to find splits in data so that each split has equal weighting according to function f" Tags="|machine-learning|data-mining|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14761" PostTypeId="1" CreationDate="2016-10-26T10:52:50.327" Score="3" ViewCount="221" Body="&lt;p&gt;I observed that &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier.fit&quot; rel=&quot;nofollow&quot;&gt;Adaboost&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html&quot; rel=&quot;nofollow&quot;&gt;Bagging&lt;/a&gt; ensemble classifiers present in sklearn only work for single label training data. How do I use these for multilabel data?&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-05-23T10:49:55.750" Title="Ensemble Techniques for multilabel data" Tags="|machine-learning|scikit-learn|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14764" PostTypeId="2" ParentId="14760" CreationDate="2016-10-26T12:08:27.073" Score="2" Body="&lt;p&gt;A classic optimization problem! You can use Linear Programming/Optimization to find a good split. Every of the &lt;strong&gt;n&lt;/strong&gt; samples &lt;strong&gt;s&lt;/strong&gt; $\in S$ has weight &lt;strong&gt;f(s)&lt;/strong&gt; and we want to divide them into &lt;strong&gt;m&lt;/strong&gt; folds. You can use a trick to linearize the L1 objective or you can use Quadratic Programming for a L2 objective function. The Quadratic Programming model is easier to define in this case. Let's define $x_{ij}$ as a binary decision to put sample &lt;strong&gt;i&lt;/strong&gt; in fold &lt;strong&gt;j&lt;/strong&gt; and &lt;strong&gt;$\mu$&lt;/strong&gt; is the ideal, mean weight per fold. Then this is our objective function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;min $\sum_{j=1}^m(\sum_{i=1}^nx_{ij}f(i)-\mu)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the following constraints:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{j=1}^mx_{ij}=1$ for all $i\in \{1..n\}$ to ensure exactly one assignment per sample&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_{ij} \in \{0, 1\}$ to turn it into binary decision varibales&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the size of your dataset and the solver you use this can be a heavy optimization, but there are a lot of greedy heuristics that will get you close fairly fast.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-10-26T12:08:27.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14765" PostTypeId="1" AcceptedAnswerId="14766" CreationDate="2016-10-26T16:41:08.893" Score="1" ViewCount="22516" Body="&lt;p&gt;I tried creating a &lt;strong&gt;SVM Classifier&lt;/strong&gt;, as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Create a SVM Classifier&#xA;model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,&#xA;            decision_function_shape=None, degree=3, gamma='auto', kernel='linear',&#xA;            max_iter=-1, probability=True, random_state=None, shrinking=True,&#xA;            tol=0.001, verbose=False&#xA;            )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Using &lt;strong&gt;Python 2.7&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But getting this error--&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;TypeError: &lt;strong&gt;init&lt;/strong&gt;() got an unexpected keyword argument&#xA;  'decision_function_shape'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on that? How to sort it out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; &gt;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My &lt;code&gt;sklearn&lt;/code&gt; version is &lt;code&gt;0.16.1&lt;/code&gt;. I tried to install the update but it's kept on saying- &lt;strong&gt;No matching distribution found for the upgrade&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="10902" LastEditDate="2016-10-27T12:46:30.240" LastActivityDate="2016-10-27T13:02:23.303" Title="Python TypeError: __init__() got an unexpected keyword argument 'decision_function_shape'" Tags="|python|scikit-learn|svm|" AnswerCount="1" CommentCount="0" ClosedDate="2018-12-20T00:34:27.680" ContentLicense="CC BY-SA 3.0" />
  <row Id="14766" PostTypeId="2" ParentId="14765" CreationDate="2016-10-26T17:37:49.993" Score="1" Body="&lt;p&gt;Your snippet is almost exactly the same as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;scikit-learn's example&lt;/a&gt; (except for kernel='rbf' and probability=False) and &lt;strong&gt;works fine&lt;/strong&gt; under version 0.18, provided the needed imports are present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: the version of scikit learn used is 0.16.1, and in that version, SVC did not have as many arguments as in 0.18, as per &lt;a href=&quot;http://scikit-learn.org/0.16/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow&quot;&gt;the docs&lt;/a&gt;. You should therefore use something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3,&#xA;            gamma=0.0, kernel='linear', max_iter=-1, probability=True,&#xA;            random_state=None, shrinking=True, tol=0.001, verbose=False)&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For upgrading scikit-learn to 0.18, (assuming you use pip) do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;pip install scikit-learn==0.18 --force-reinstall&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2016-10-27T13:02:23.303" LastActivityDate="2016-10-27T13:02:23.303" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14768" PostTypeId="2" ParentId="14730" CreationDate="2016-10-26T19:27:26.587" Score="0" Body="&lt;p&gt;You're asking what ML representation you should use for user-classification of chat text. User-classification is not the usual text-processing task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;It's not strictly necessary to semantically understand what the user is saying, only how they're saying it&lt;/strong&gt;; so we look for telltale features indicative of a specific user. And we don't necessarily need to use, or solely rely on, the usual text-processing representations like bag-of-words, word-counts, TFIDF and word-vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some features which are predictive of the user:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;character length, word length, sentence length of each comment&lt;/li&gt;&#xA;&lt;li&gt;typing speed (esp. if you have timestamps in seconds)&lt;/li&gt;&#xA;&lt;li&gt;ratio of punctuation (e.g. 17 punctuation symbols in 80 chars = 17/80)&lt;/li&gt;&#xA;&lt;li&gt;ratio of capitalization&lt;/li&gt;&#xA;&lt;li&gt;ratio of numerals&lt;/li&gt;&#xA;&lt;li&gt;ratio of whitespace&lt;/li&gt;&#xA;&lt;li&gt;character n-grams (and notice these can pick up e.g. &lt;em&gt;l0ser, f##k, :-)&lt;/em&gt; )&lt;/li&gt;&#xA;&lt;li&gt;use of Unicode (emojis, symbols e.g. stars)&lt;/li&gt;&#xA;&lt;li&gt;ratio of specific punctuation (e.g. how many '.', '!', '?', '*', '#' )&lt;/li&gt;&#xA;&lt;li&gt;word-counts, esp. anything statistically anomalous, foreign, slang, insults&lt;/li&gt;&#xA;&lt;li&gt;anything else you can think of that seems predictive for these two users, e.g. number of misspelled words per sentence (may be actual typos, or come from predictive swiping on a cellphone)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="8501" LastEditDate="2016-10-26T19:36:00.870" LastActivityDate="2016-10-26T19:36:00.870" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14769" PostTypeId="1" CreationDate="2016-10-26T20:12:08.230" Score="1" ViewCount="43" Body="&lt;p&gt;I want to estimate how fast my model learn. Mb how many learn steps my model need to do to achieve specific error rate(precision/ recall/ MSE/ etc.). Any standard approach for measuring this?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastActivityDate="2016-10-26T21:48:04.387" Title="Is there standard parameters that characterize how fast neural network learns to achieve specific error rate?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14770" PostTypeId="2" ParentId="694" CreationDate="2016-10-26T20:40:07.630" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/Microsoft/CNTK&quot; rel=&quot;nofollow noreferrer&quot;&gt;Microsoft Cognition Toolkit (previously known as CNTK)&lt;/a&gt; has a  &lt;a href=&quot;https://github.com/Microsoft/CNTK/wiki/CNTK-Library-API&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python API&lt;/a&gt;. Amongst other things, it is supposed to be good for &lt;a href=&quot;http://blogs.microsoft.com/next/2016/01/25/microsoft-releases-cntk-its-open-source-deep-learning-toolkit-on-github/#sm.000ombkcw1atgezsv4p1gzebefljy&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-GPU&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tVFJt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tVFJt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples and tutorials can be found on &lt;a href=&quot;https://github.com/Microsoft/CNTK/tree/master/bindings/python&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Microsoft/CNTK/tree/master/bindings/python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-10-26T20:40:07.630" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="14772" PostTypeId="2" ParentId="14769" CreationDate="2016-10-26T21:48:04.387" Score="1" Body="&lt;p&gt;Sometimes you can estimate that by smoothing out your &lt;em&gt;val_loss&lt;/em&gt; curve and hoping that your training is repeatable and stable, but seems that word &lt;em&gt;&quot;unpredictable&quot;&lt;/em&gt; was invented specifically to describe loss curves. :) Especially if you're using some sort of dynamic data generation or custom cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at few species at &lt;a href=&quot;http://lossfunctions.tumblr.com/&quot; rel=&quot;nofollow&quot;&gt;http://lossfunctions.tumblr.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2016-10-26T21:48:04.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14773" PostTypeId="2" ParentId="12900" CreationDate="2016-10-26T22:52:35.457" Score="2" Body="&lt;p&gt;Wow, you must be taking the job I just left behind!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our CEO (and full exec team) was used to Wired article headlines and viewed any test with &amp;lt;30% conversion boost as &quot;failure&quot;.  My team was expected to do needlessly complex analyses because one of their friends had forwarded a click-bait article about &quot;growth hacking&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First, find a baseline.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ask during the interview process what they hope you'll accomplish in the first 3-6 months.  If they have a solid list of deliverables, that's a good sign.  If that list is something you're comfortable with even better.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ask &quot;why?&quot; a lot.  In the interview/early days you're understandably naive, so you can say &lt;code&gt;oh, why is that important?&lt;/code&gt; and it's interpreted as &lt;code&gt;I'm trying to learn about our business.&lt;/code&gt;  If their assumptions show some thought and understanding, even if you don't fully agree, that's a great sign.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ask how those projects will help with their overall goal.  What is their overall goal?  &lt;code&gt;We want to be a $1B company&lt;/code&gt; is a terrible goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if your manager-to-be has no good answers to those questions: walk.  There are plenty of places hiring.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second, build a plan.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have their list of ideas and some reasons they're important.  Put together a rough draft plan of how you'd attack them.  Don't get too in the weeds, just list the strategic goal, the data you'll need, the method you'll start with, what you hope to find and a rough time estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Ok, you're disappointed with mobile conversion.  I'll want to implement on-site &#xA;tracking (eg Segment.io) to start logging a variety of parameters &#xA;(eg browser, IP locale) as well as getting views on visitor flow.  I'll use a &#xA;basic SQL client (eg Mode) to segment users and see if the problem is confined to &#xA;any one group (eg Android users).  From there, I'll work with the product team to&#xA;size/prioritize possible split tests, which we'll analyze manually at first.  &#xA;&#xA;It'll take a few days to get Segment up and verified, ~1-2 weeks for data&#xA;collection, and a full week to do a deep dive.  The earliest we could start &#xA;experimenting would be two weeks, but realistically after the better part of &#xA;a month.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You're just riffing to see their reaction.  Are they nodding reverently like you're speaking Latin, it sounds impressive but they don't follow?  Are they pushing for faster turnaround and guaranteed results?  Are they asking insightful questions (eg Can you really get an integration up in 2 days?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they're engaged and interested, then you've found a good partner.  Otherwise consider how much you're willing to invest on education/management.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finally, assess how aligned others are&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Talk to the engineering team and ask what they've worked on in the past year.  Push for how they source and prioritize projects.  You'll likely be in a very similar boat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consistency or thrashing?  Evidence-based or HiPPO decision making?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR: Go with genuine interest&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I've been most successful when I screen employers for curiosity over performance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everyone wants to do well in business, but the people I've loved doing data work for are ones who are eager to learn what's going on with their business and how to please customers better.  The ones I've dreaded working for want to brag about KPIs and stock prices.  The former will be patient as you get your sea-legs in a new org.&lt;/p&gt;&#xA;" OwnerUserId="25611" LastActivityDate="2016-10-26T22:52:35.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14774" PostTypeId="1" AcceptedAnswerId="14785" CreationDate="2016-10-27T00:18:08.207" Score="14" ViewCount="39032" Body="&lt;p&gt;&lt;a href=&quot;https://community.modeanalytics.com/gallery/geographic-heat-map/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mode Analytics&lt;/a&gt; has a nice heatmap feature, but it is not conducive to comparing maps (only one per report).&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QhA1I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QhA1I.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;What they do allow is data to be pulled easily into a wrapped python notebook.  And then any image in python can easily be added to a report.&lt;/p&gt;&#xA;&lt;p&gt;So my question is: how do I recreate a heatmap on an actual map in Python?  I've checked out follium and plotly, but neither seem to have similar functionality.&lt;/p&gt;&#xA;" OwnerUserId="25611" LastEditorUserId="29169" LastEditDate="2021-03-13T20:16:52.410" LastActivityDate="2021-03-13T20:18:03.603" Title="Heatmap on a map in Python" Tags="|python|visualization|geospatial|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="14776" PostTypeId="1" CreationDate="2016-10-27T08:24:35.253" Score="3" ViewCount="173" Body="&lt;p&gt;I am a R&amp;amp;D operator which is in charge to conduct an evaluation study for implementing a monitoring system for our process of production. I generalized what the software must do but I don't know which is the approach/algorithm to perform what I designed.&lt;br&gt;&#xA;Here's the process description:&lt;br&gt;&#xA;we produce brake hoses. The hoses are produced in a continuous process. It starts from (1) extruding a plastic tube which get wrapped in reels of about 1000mt. Then (2) these 1000mt of extruded tube get braided with a textile or steel wire. The last phase (3) is coating the braided tube from phase (2) with an extrusor with a &quot;T&quot; head. At the end we get batches of brake hoses in reels of about 1000mt.&#xA;The variables in this process are detected by sensors and in order are:&lt;br&gt;&#xA;(1)-extruder speed - temperature - vacuum pressure&lt;br&gt;&#xA;(1)-diameter of the extruded tube&lt;br&gt;&#xA;(1) human operator&lt;br&gt;&#xA;(2) speed of the braiding machines&lt;br&gt;&#xA;(2) temperature and humidity of the braiding room&lt;br&gt;&#xA;(2) human operator of the braiding machine&lt;br&gt;&#xA;(3) coating extruder speed - temperature - vacuum pressure&lt;br&gt;&#xA;(3) diameter of the coated hose (which the brake hose = the process OUTPUT)&lt;br&gt;&#xA;(3) human operator&lt;br&gt;&#xA;(3) tests performed by the laboratory onto the brake hose&lt;br&gt;&#xA;This process is in the time-domain as the three phases are in chronological order and the process is a linear process (it starts from an extrusion of the inner tube and ends with its coating).&lt;br&gt;&#xA;The operators of the laboratory &lt;em&gt;will perform tests on the finished product&lt;/em&gt; (the process &lt;strong&gt;output&lt;/strong&gt;) which is identified by a univoque batch code. The software must correlate these data and the diameter detected in the phase (3) to the sensors data of the process of &lt;em&gt;that&lt;/em&gt; batch of hose.&lt;br&gt;&#xA;Everytime we input data into the system the software must learn to recognize patterns and find &lt;em&gt;correlations&lt;/em&gt; among the variables.&lt;br&gt;&#xA;&lt;strong&gt;For example&lt;/strong&gt;: last year we had a leakage problem on a hose which wasn't determined and disappeared after one month without significant actions made onto the process. If the software had already been implemented we expected a result from its analysis like this (simplifying in a spoken language):&lt;br&gt;&#xA;-&quot;I detected that the batch n°3 had a leakage in the test of &quot;pressure holding&quot; same as batch n°2 and batch n°1. In all these batches I detected the coexistence of these three factors:&lt;br&gt;&#xA;1) the presence of the operator John Doe which stopped the braiding machine 2 times&lt;br&gt;&#xA;2) the humidity of the braiding room was 2% over the average level of the past 3 months&lt;br&gt;&#xA;3) there plastic grains for the extrusion of the inner tube (phase 1) has dried only for 20 minutes instead of 40.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, the software next time the process shows the same conditions listed above &lt;strong&gt;&lt;em&gt;warns&lt;/em&gt;&lt;/strong&gt; us that the same defect might happen.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which kind of algorithm (supervised-reinforced-unsupervised) is best suited for this task and which kind of approach (decision three-neural network?&#xA;As I'm making a study to be submitted to the top management I would like to be well prepared and also don't depend to what a software house says it is the best solution because they tend to be &quot;smart&quot; when somebody is ignorant on an argument.&lt;br&gt;&#xA;Thank you&lt;br&gt;&#xA;Alessio Cazzaniga  &lt;/p&gt;&#xA;" OwnerUserId="25618" LastEditorUserId="44990" LastEditDate="2018-11-07T14:07:52.500" LastActivityDate="2018-11-07T14:07:52.500" Title="Righteous data analysis approach for this industrial process monitoring" Tags="|machine-learning|data-mining|" AnswerCount="2" CommentCount="5" ClosedDate="2016-10-28T14:03:28.593" ContentLicense="CC BY-SA 4.0" />
  <row Id="14777" PostTypeId="1" CreationDate="2016-10-27T08:45:58.393" Score="0" ViewCount="593" Body="&lt;p&gt;I want to do a text classification problem for which I want to train use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html&quot; rel=&quot;nofollow&quot;&gt;Adaboost&lt;/a&gt; Classifier from sklearn using a Keras estimator. I do know how to use the Keras wrappers for using sklearn functions. So that is not the problem. I have a 3D input data (number_sentences, number_words, features) and when I checked the source code of Adaboost, it shows that X can be only 1D or 2D. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why is that so? &lt;/li&gt;&#xA;&lt;li&gt;Is there any other way for me to use Adaboost for 3D    input data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2016-10-27T09:12:36.017" LastActivityDate="2017-08-23T15:12:53.543" Title="Adaboost for 3D Input data" Tags="|machine-learning|scikit-learn|optimization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14778" PostTypeId="2" ParentId="14777" CreationDate="2016-10-27T11:27:18.147" Score="1" Body="&lt;p&gt;Adaboost doesn't care about the order of the features, so no matter how many dimensions your samples have, you can just flatten them. Flattening them gives you a 2D X with shape (n_samples, n_features) which you can feed to the Adaboost classifier.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2016-10-27T11:27:18.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14780" PostTypeId="1" AcceptedAnswerId="14783" CreationDate="2016-10-27T13:54:29.657" Score="2" ViewCount="180" Body="&lt;p&gt;Does anyone could help me to understand what the autoencoders means? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we expect is that the outputs are equal to the inputs, then why we need to do that? It doesn't make any sense to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find some interpretation that it learned how to reconstruct the input data, does that mean that we could just pick some of the pixels from the origin picture and then reconstruct the whole original picture? If so, it still makes no sense to me, because the reconstructing part of the model is from the hidden layer to the output layer, we cannot just put the selected data into the hidden layer, cuz the input data of the hidden layer are combinations of the whole raw data from the input layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="25623" LastActivityDate="2016-10-27T19:23:46.580" Title="Does anyone could help me to understand what is the autoencoders?" Tags="|machine-learning|autoencoder|" AnswerCount="2" CommentCount="1" ClosedDate="2016-10-28T14:03:37.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="14781" PostTypeId="2" ParentId="14776" CreationDate="2016-10-27T13:57:32.577" Score="1" Body="&lt;p&gt;I think a random forest would be appropriate in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have annotated historical data (records of all the variables you described as well as if the tube passed or failed), you could build a random forest, determine variable importance, and link both the categorical data (human operator) and continuous data (temperature, pressure...) to the tube outcome.  A benefit of random forests is that they can be more easily described in terms of a decision tree.  R has some nice visualization tools for decision trees, and in my experience, management has an easier time accepting decision trees then other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you need more detail and I can edit this response to include some sources on random forest, or more in depth explanation.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-10-27T13:57:32.577" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14783" PostTypeId="2" ParentId="14780" CreationDate="2016-10-27T15:04:41.953" Score="4" Body="&lt;p&gt;Autoencoders are a neural network solution to the problem of &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction&quot;&gt;dimensionality reduction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, if your data includes people's height, weight, trouser leg measurement and shoe size, we'd expect there to be some underlying &lt;em&gt;size&lt;/em&gt; dimension which would capture much of the variance of these variables. If you're familiar with Principal Component Analysis (PCA), this is another example of a dimensionality reduction technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Autoencoders attempt to capture a lower-dimensional representation of their data by having a hidden &quot;bottleneck&quot; layer which is much smaller than the dimensionality of the data. The idea is to train a neural network which throws away as much of its dimensionality as possible and can still reconstruct the original data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have an autoencoder which performs well, by observing the activations at the bottleneck layer, it's possible to see how an individual example scores in each of the reduced dimensions. This may allow us to begin to make sense of what each of the dimensions represents. One can then use these activations to score new examples on this set of reduced dimensions.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-10-27T15:04:41.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14784" PostTypeId="2" ParentId="14662" CreationDate="2016-10-27T15:14:41.093" Score="1" Body="&lt;p&gt;In TFIDF representation &quot;neighborhood&quot; is not interpretable. The 1st number in a document vector has just as much relation to the 2nd and to the 100th.&#xA;While in case of a standard time series or sequence analysis &quot;neighborhood&quot; is interpretable.&#xA;On the other hand, if you could re-order the TFIDF representation in a way, that makes sense from the analysis perspective (eg: ordering based on the &quot;sentiment&quot; of the word), maybe 1D CNN would show some good results.&lt;/p&gt;&#xA;" OwnerUserId="14211" LastActivityDate="2016-10-27T15:14:41.093" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14785" PostTypeId="2" ParentId="14774" CreationDate="2016-10-27T18:54:47.660" Score="11" Body="&lt;p&gt;Looks like the &lt;a href=&quot;https://github.com/pbugnion/gmaps&quot; rel=&quot;nofollow noreferrer&quot;&gt;gmaps&lt;/a&gt; package is what you're looking for.&lt;/p&gt;&#xA;&lt;p&gt;You can do things like this with it:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SZFWL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SZFWL.png&quot; alt=&quot;gmap&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25504" LastEditorUserId="29169" LastEditDate="2021-03-13T20:18:03.603" LastActivityDate="2021-03-13T20:18:03.603" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14786" PostTypeId="1" AcceptedAnswerId="14804" CreationDate="2016-10-27T18:59:21.527" Score="4" ViewCount="177" Body="&lt;p&gt;I was checking a kernel written in python from the Bosch kaggle competition (&lt;a href=&quot;https://www.kaggle.com/happycube/bosch-production-line-performance/scirpus-extreme-bayes-faron-36312&quot; rel=&quot;nofollow&quot;&gt;kaggle link to python kernel&lt;/a&gt;) and I came across with a weird (at least to me) way to fill Nan values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The train data is split into two halves and then some kind of average is computed in one half by using the non-Nan values of a field along with the target value and then fill Nan values on the other half with these computed values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, when training the model after filling Nan values, the model is only trained on the half data where the Nan values have been replaced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is, why would you split the data into two halves to compute on one and then fill the other?&#xA;Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? &#xA;Is this procedure prone to overfitting if you perform this operation on all the train data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advanced.&lt;/p&gt;&#xA;" OwnerUserId="21194" LastEditorUserId="21194" LastEditDate="2016-10-28T08:50:20.623" LastActivityDate="2016-10-28T20:38:52.740" Title="Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half?" Tags="|machine-learning|data-cleaning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14787" PostTypeId="2" ParentId="14780" CreationDate="2016-10-27T19:23:46.580" Score="2" Body="&lt;p&gt;Let me insert 2¢...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking &quot;autoencoding&quot; is a lossy compression technique, although not very useful due to being data-specific (autoencoder trained on &lt;em&gt;cats&lt;/em&gt; not very useful for &lt;em&gt;cars&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice it is used for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;data denoising&lt;/li&gt;&#xA;&lt;li&gt;dimensionality reduction&lt;/li&gt;&#xA;&lt;li&gt;unsupervised pre-training of feature extracting parts of more complex networks&lt;/li&gt;&#xA;&lt;li&gt;and... generating new unseen samples from seen ones (!)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The latter is possible with a variety called Variational Autoencoders (VAE), where you impose some constraints on the compressed representation being learned that force it to represent a set of variables that model the probability distribution that represents input data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words in VAE each &quot;bottleneck&quot; variable represents something meaningful about the input (think &quot;color of a cat&quot;). Thus changing this representation produces meaningful output on decoder part of AE.&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2016-10-27T19:23:46.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14788" PostTypeId="1" CreationDate="2016-10-27T20:25:55.397" Score="1" ViewCount="2211" Body="&lt;p&gt;I have a time series dataset with 200 data points. I have decomposed it using the function below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat2 = ts(dat1, frequency = 4)&#xA;decomposeDat = decompose(dat2, &quot;multiplicative&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get 4 components: trend, seasonal, cyclic and irregularity. But when I check if there is seasonality present in the dataset with frequency &quot;4&quot;, Rstudio says that there is no seasonality for this frequency. The check is performed with the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat2 = ts(dat1, frequency = 4)&#xA;fit &amp;lt;- tbats(dat2)&#xA;seasonal &amp;lt;- !is.null(fit$seasonal)&#xA;seasonal&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;seasonal returns FALSE meaning that there is no seasonality with frequency 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain, why can I decompose it into a seasonality component when no seasonality is present from the check mentioned above?&lt;/p&gt;&#xA;" OwnerUserId="25626" LastEditorUserId="12384" LastEditDate="2017-02-28T09:22:24.400" LastActivityDate="2017-06-28T12:08:02.470" Title="R - time series decomposition without detection of seasonality" Tags="|r|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14789" PostTypeId="2" ParentId="14786" CreationDate="2016-10-27T22:42:11.770" Score="0" Body="&lt;p&gt;The reason is you (obviously) cannot feed NaNs to your model, so you absolutely have to clean them up. Looks like the idea here is – by filling them with something other than zeros may help the learning by introducing some excessive information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be actually interesting first to run some sort of seq2seq RNN to try to predict NaNs from non-NaNs and then explore how it affects learning rate/accuracy...&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2016-10-27T22:42:11.770" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14793" PostTypeId="1" CreationDate="2016-10-28T04:41:24.640" Score="2" ViewCount="1589" Body="&lt;ol&gt;&#xA;&lt;li&gt;I'm trying to create a Sentiment Analysis algorithm for a custom data (government dept specifc data) and not like any other social media data etc. The data exists but I need to categorise the data as positive or negative.  &lt;/li&gt;&#xA;&lt;li&gt;My requirement is to classify the test data as positive or negative using a sentiment analysis algorithm. &lt;/li&gt;&#xA;&lt;li&gt;The big challenge for me is to take the similar set of government data which I can get. But I need to prepare the training data set. For this I need to classify the training data set as positive or negative. &lt;/li&gt;&#xA;&lt;li&gt;On what basis do I need to categorise the data set as positive or negative. The reason I'm asking this question is that preparing a good training data set is very important to improve the accuracy of my sentiment analysis algorithm .&lt;/li&gt;&#xA;&lt;li&gt;I have come up with my sentiment analysis algorithm but really need to prepare a robust set of training data(classified as positive or negative). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Experts - I'm a newbie to the machine learning area and I need advice from the researchers and experts out there. &lt;/p&gt;&#xA;" OwnerUserId="25631" LastEditorUserId="85045" LastEditDate="2021-02-10T17:49:57.297" LastActivityDate="2021-02-10T17:49:57.297" Title="How to create a training set and classify them as positive or negative" Tags="|nlp|sentiment-analysis|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14795" PostTypeId="1" CreationDate="2016-10-28T07:05:13.633" Score="3" ViewCount="116" Body="&lt;p&gt;In a employee attrition analysis with a table having rowwise data for a (employee like Id, name, Date_Join Date_Relieving Dept Role etc)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eID eName    Joining Releiving Dept Married Experience  &#xA;123 John Doe 10Oct15 12Oct16   HR   No      12  &#xA;234 Jen Doee 01jan16 -NA-      HR   No      11         (ie she is available)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can run regression on this data to find the beta coefficients&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eID eName    Joining Releiving Dept Married Experience  &#xA;123 John Doe 10Oct15 12Oct16   HR   No      12  &#xA;234 Jen Doee 01jan16 -NA-      HR   No      11  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I've seen other approach too.. &#xA;where employee have multiple entries depending on their difference between joining date and current month or relieving month(say Employee A joined in Jan and Left in Dec so he'll have 12 entries updating corresponding columns like experience and marriage etc)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eID eName    Dept Married Experience  &#xA;123 John Doe HR   No      0  &#xA;123 John Doe HR   No      1  &#xA;123 John Doe HR   Yes     2  &#xA;123 John Doe HR   Yes     3  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;can someone tell what differentiate two approaches.. and what are the benefits of this second approach.&lt;/p&gt;&#xA;" OwnerUserId="6558" LastEditorUserId="27432" LastEditDate="2017-01-12T17:49:03.690" LastActivityDate="2017-04-12T19:31:12.960" Title="Prediction approach on unique data or progressive data" Tags="|predictive-modeling|regression|prediction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14796" PostTypeId="2" ParentId="14776" CreationDate="2016-10-28T07:54:16.493" Score="2" Body="&lt;p&gt;The problem you are looking at is a common researched problem in the field of reliability engineering. If I understand you example correctly your goal is to gain an understanding in the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What manufacturing conditions cause certain parts (in your situation hoses) to fail sooner than others?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each variable you mention you will have to establish whether these are constant over the life time of a part or whether they change. Here I measure time as the time that elapses once the part is used by your client in production. In reliability engineering the variables that do not change over time e.g. human operator of the manufacturing process, production batch, etc. are called &lt;em&gt;time independent covariates&lt;/em&gt; the ones that change over time, e.g. amount of pressure put on a hose in production (if you have a sensor that reads that kind of information at your client site), are called &lt;em&gt;time dependent covariates&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have to transform your data such that it becomes censored survival data. This type of data means that every dependent variable is &lt;em&gt;amount of time in operation from production&lt;/em&gt;. Since some of your hoses will likely never have a defect they will leave your observation period. Note that you do need to take these points into consideration too. When you combine all these data points you get something called &lt;a href=&quot;https://en.wikipedia.org/wiki/Censoring_(statistics)&quot; rel=&quot;nofollow&quot;&gt;right censored survival data&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the right censored survival data and the covariates together you can start to build a &lt;a href=&quot;https://en.wikipedia.org/wiki/Accelerated_failure_time_model&quot; rel=&quot;nofollow&quot;&gt;accelerated failure time model&lt;/a&gt;. Note that it is recommended to first start with a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator&quot; rel=&quot;nofollow&quot;&gt;Kaplan Meier estimate&lt;/a&gt; of your survival rates before diving into AFT, but I just want to give you some pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A book that I can highly recommend on reliability engineering is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/047097981X&quot; rel=&quot;nofollow&quot;&gt;Practical Reliability Engineering&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0965306232&quot; rel=&quot;nofollow&quot;&gt;The New Weibull Handbook&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both don't dive deep into AFT but they do explain the fundamentals of physical systems and how this links to statistics. From experience I know that not many people are working on this field and that you can have a lot of wrong pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of bringing a model like this in production will heavily depend on the quality of your data. Typically sensor data can be of poor quality, moreover, user input tends to be even of worse quality. An article that gives you an idea of what kind of organizational challenges you may face can be found &lt;a href=&quot;http://performance.ey.com/wp-content/uploads/downloads/2016/05/EY-Performance-Products.pdf&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;. Expect a project like this to take several months to years.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-10-28T07:54:16.493" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14801" PostTypeId="1" CreationDate="2016-10-28T14:01:32.517" Score="4" ViewCount="4661" Body="&lt;p&gt;I am trying to visualise a transition from one state (work in origin country) and another (work in destination country). I think transition plot is perfect but I can't get how does it work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data frame is called (Cimad) and my variables are EMPLOIPRE (origin) and EMPLOIFR (destination). Both are factors, the first count 20 levels, the second 15 levels (is it a problem, do I have to make them all match?) and I have 400 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope you'll be able to help me because I am completely lost !&#xA;Thank you in advance&#xA;Caroline.&lt;/p&gt;&#xA;" OwnerUserId="25636" LastEditorUserId="23675" LastEditDate="2016-11-15T18:42:43.843" LastActivityDate="2017-03-15T20:03:15.517" Title="Transition plot in R how does it work?" Tags="|r|visualization|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14802" PostTypeId="2" ParentId="14752" CreationDate="2016-10-28T15:23:48.567" Score="1" Body="&lt;p&gt;I recently wanted to do the same thing, like yesterday, which is how I came across your post. You might also try &lt;a href=&quot;https://groups.google.com/forum/#!topic/torch7/mtNfCLlQNSU&quot; rel=&quot;nofollow&quot;&gt;this link&lt;/a&gt; that has a modified installation script for gentoo. I haven't gotten very far and know almost nothing about lua, but I have verified that after running the installation script I can load and run some basic torch scripts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[edit: fixed typo]&lt;/p&gt;&#xA;" OwnerUserId="25637" LastEditorUserId="25637" LastEditDate="2016-10-29T22:10:34.240" LastActivityDate="2016-10-29T22:10:34.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14804" PostTypeId="2" ParentId="14786" CreationDate="2016-10-28T20:38:52.740" Score="1" Body="&lt;p&gt;You basically answered your own question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? Is this procedure prone to overfitting if you perform this operation on all the train data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. You have already used the target values and the first half of the training set. In other words, the filled-out NaN values have information about the target values in them, so if you were to perform any cross-validation for performance estimation, you would end up with an over-estimation of your performance metric. It's best practice to leave that used part out of any further training steps to avoid information leakage.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastActivityDate="2016-10-28T20:38:52.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14805" PostTypeId="1" AcceptedAnswerId="25060" CreationDate="2016-10-28T21:47:16.713" Score="3" ViewCount="809" Body="&lt;p&gt;&lt;em&gt;Data augmentation&lt;/em&gt; techniques for image data and audio data (eg speech recognition) have proven successful and are now common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there libraries or techniques for augmenting &lt;em&gt;text&lt;/em&gt; data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;in: 'How are you?'&lt;br&gt;&#xA;  out: ['how are you?', 'HOW ARE YOU?', 'hwo are y ou?', 'How're you?', 'how r u', ...]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2020-07-03T20:05:36.230" Title="Are there libraries or techniques for 'noisifying' text data?" Tags="|machine-learning|nlp|deep-learning|data-cleaning|data-augmentation|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14806" PostTypeId="2" ParentId="14801" CreationDate="2016-10-28T23:44:02.140" Score="1" Body="&lt;p&gt;Fwiw, here's an example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create example data&#xA;set.seed(1)&#xA;Cimad &amp;lt;- data.frame(&#xA;  EMPLOIPRE = as.factor(sample(letters[1:15], 400, T)),&#xA;  EMPLOIFR = as.factor(sample(letters[1:20], 400, T))&#xA;)&#xA;&#xA;# create adjacency/transition matrix&#xA;labs &amp;lt;- unique(unlist(Cimad))&#xA;m &amp;lt;- table(factor(Cimad[,1], lev=labs), factor(Cimad[,2], lev=labs))&#xA;m &amp;lt;- m / rowSums(m)&#xA;&#xA;# plot it&#xA;library(igraph)&#xA;g &amp;lt;- graph_from_adjacency_matrix(m, weighted = &quot;prob&quot;)&#xA;E(g)$prob &amp;lt;- ifelse(is.nan(E(g)$prob), NA, E(g)$prob)&#xA;plot(g, edge.label = round(E(g)$prob, 2), edge.arrow.size = .25, edge.label.cex = .5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kcSj0.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kcSj0.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, if you want to export it into a PDF (e.g. for zoom):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pdf(tf &amp;lt;- tempfile(fileext = &quot;.pdf&quot;), width = 15, height = 15)&#xA;plot(g, edge.label = round(E(g)$prob, 2), edge.arrow.size = .25, edge.label.cex = .5)&#xA;dev.off()&#xA;shell.exec(tf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-10-28T23:44:02.140" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14807" PostTypeId="1" AcceptedAnswerId="14808" CreationDate="2016-10-29T00:16:56.327" Score="3" ViewCount="611" Body="&lt;p&gt;Does it make sense to use the &lt;code&gt;train&lt;/code&gt;, &lt;code&gt;test&lt;/code&gt;, and &lt;code&gt;validation&lt;/code&gt; model using the Association Rules Technique?&lt;/p&gt;&#xA;" OwnerUserId="24790" LastEditorUserId="21" LastEditDate="2016-10-29T12:45:15.937" LastActivityDate="2016-10-29T12:45:15.937" Title="Association Rules - Data Mining - Train and Test approach?" Tags="|data-mining|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14808" PostTypeId="2" ParentId="14807" CreationDate="2016-10-29T00:46:20.030" Score="3" Body="&lt;p&gt;Finding association rules is an unsupervised learning task (or exploratory task). You don't actually know which rules you want to find before you actually find them, so there's nothing to test against. Validating against a separated instance set is usually done on supervised learning tasks such as classification and regression.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2016-10-29T00:46:20.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14811" PostTypeId="2" ParentId="13710" CreationDate="2016-10-29T06:59:18.993" Score="0" Body="&lt;p&gt;If I get you right, then you can use any of the below libraries. These are simple to use and you can customize them as per your requirement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The libraries are-- &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://matplotlib.org/gallery.html&quot; rel=&quot;nofollow&quot;&gt;matplotlib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bokeh.pydata.org/en/latest/docs/gallery.html&quot; rel=&quot;nofollow&quot;&gt;Bokeh&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://seaborn.pydata.org/&quot; rel=&quot;nofollow&quot;&gt;Seaborn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ggplot.yhathq.com/&quot; rel=&quot;nofollow&quot;&gt;ggplot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.pygal.org/en/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;pygal&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://plot.ly/python/&quot; rel=&quot;nofollow&quot;&gt;Plotly&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/andrea-cuttone/geoplotlib&quot; rel=&quot;nofollow&quot;&gt;geoplotlib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/dgrtwo/gleam&quot; rel=&quot;nofollow&quot;&gt;Gleam&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/ResidentMario/missingno&quot; rel=&quot;nofollow&quot;&gt;missingno&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://leather.readthedocs.io/en/latest/index.html&quot; rel=&quot;nofollow&quot;&gt;Leather&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can use any but as per your data I would recommend you to go with &lt;code&gt;Plotly , Seaborn or Bokeh&lt;/code&gt;. For chart-codes and description you can try- &lt;a href=&quot;http://lisacharlotterost.github.io/2016/05/17/one-chart-code/&quot; rel=&quot;nofollow&quot;&gt;One Chart, Twelve Charting Libraries&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need any other help on this update me. I would help to help. Cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-29T06:59:18.993" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14812" PostTypeId="1" CreationDate="2016-10-29T08:52:18.763" Score="1" ViewCount="4691" Body="&lt;p&gt;I'm executing this code from &lt;a href=&quot;https://www.kaggle.com/danijelk/allstate-claims-severity/keras-starter-with-bagging-lb-1120-596&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/danijelk/allstate-claims-severity/keras-starter-with-bagging-lb-1120-596&lt;/a&gt; on a nvidia geforce 960M.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;''' &#xA;Author: Danijel Kivaranovic &#xA;Title: Neural network (Keras) with sparse data&#xA;'''&#xA;&#xA;## import libraries&#xA;import numpy as np&#xA;np.random.seed(123)&#xA;&#xA;import pandas as pd&#xA;import subprocess&#xA;from scipy.sparse import csr_matrix, hstack&#xA;from sklearn.metrics import mean_absolute_error&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.cross_validation import KFold&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Activation&#xA;from keras.layers.advanced_activations import PReLU&#xA;&#xA;## Batch generators ##################################################################################################################################&#xA;&#xA;def batch_generator(X, y, batch_size, shuffle):&#xA;    #chenglong code for fiting from generator (https://www.kaggle.com/c/talkingdata-mobile-user-demographics/forums/t/22567/neural-network-for-sparse-matrices)&#xA;    number_of_batches = np.ceil(X.shape[0]/batch_size)&#xA;    counter = 0&#xA;    sample_index = np.arange(X.shape[0])&#xA;    if shuffle:&#xA;        np.random.shuffle(sample_index)&#xA;    while True:&#xA;        batch_index = sample_index[batch_size*counter:batch_size*(counter+1)]&#xA;        X_batch = X[batch_index,:].toarray()&#xA;        y_batch = y[batch_index]&#xA;        counter += 1&#xA;        yield X_batch, y_batch&#xA;        if (counter == number_of_batches):&#xA;            if shuffle:&#xA;                np.random.shuffle(sample_index)&#xA;            counter = 0&#xA;&#xA;def batch_generatorp(X, batch_size, shuffle):&#xA;    number_of_batches = X.shape[0] / np.ceil(X.shape[0]/batch_size)&#xA;    counter = 0&#xA;    sample_index = np.arange(X.shape[0])&#xA;    while True:&#xA;        batch_index = sample_index[batch_size * counter:batch_size * (counter + 1)]&#xA;        X_batch = X[batch_index, :].toarray()&#xA;        counter += 1&#xA;        yield X_batch&#xA;        if (counter == number_of_batches):&#xA;            counter = 0&#xA;&#xA;########################################################################################################################################################&#xA;&#xA;## read data&#xA;train = pd.read_csv('../input/train.csv')&#xA;test = pd.read_csv('../input/test.csv')&#xA;&#xA;## set test loss to NaN&#xA;test['loss'] = np.nan&#xA;&#xA;## response and IDs&#xA;y = train['loss'].values&#xA;id_train = train['id'].values&#xA;id_test = test['id'].values&#xA;&#xA;## stack train test&#xA;ntrain = train.shape[0]&#xA;tr_te = pd.concat((train, test), axis = 0)&#xA;&#xA;## Preprocessing and transforming to sparse data&#xA;sparse_data = []&#xA;&#xA;f_cat = [f for f in tr_te.columns if 'cat' in f]&#xA;for f in f_cat:&#xA;    dummy = pd.get_dummies(tr_te[f].astype('category'))&#xA;    tmp = csr_matrix(dummy)&#xA;    sparse_data.append(tmp)&#xA;&#xA;f_num = [f for f in tr_te.columns if 'cont' in f]&#xA;scaler = StandardScaler()&#xA;tmp = csr_matrix(scaler.fit_transform(tr_te[f_num]))&#xA;sparse_data.append(tmp)&#xA;&#xA;del(tr_te, train, test)&#xA;&#xA;## sparse train and test data&#xA;xtr_te = hstack(sparse_data, format = 'csr')&#xA;xtrain = xtr_te[:ntrain, :]&#xA;xtest = xtr_te[ntrain:, :]&#xA;&#xA;print('Dim train', xtrain.shape)&#xA;print('Dim test', xtest.shape)&#xA;&#xA;del(xtr_te, sparse_data, tmp)&#xA;&#xA;## neural net&#xA;def nn_model():&#xA;    model = Sequential()&#xA;    model.add(Dense(400, input_dim = xtrain.shape[1], init = 'he_normal'))&#xA;    model.add(PReLU())&#xA;    model.add(Dropout(0.4))&#xA;    model.add(Dense(200, init = 'he_normal'))&#xA;    model.add(PReLU())&#xA;    model.add(Dropout(0.2))&#xA;    model.add(Dense(1, init = 'he_normal'))&#xA;    model.compile(loss = 'mae', optimizer = 'adadelta')&#xA;    return(model)&#xA;&#xA;## cv-folds&#xA;nfolds = 5&#xA;folds = KFold(len(y), n_folds = nfolds, shuffle = True, random_state = 111)&#xA;&#xA;## train models&#xA;i = 0&#xA;nbags = 5&#xA;nepochs = 55&#xA;pred_oob = np.zeros(xtrain.shape[0])&#xA;pred_test = np.zeros(xtest.shape[0])&#xA;&#xA;for (inTr, inTe) in folds:&#xA;    xtr = xtrain[inTr]&#xA;    ytr = y[inTr]&#xA;    xte = xtrain[inTe]&#xA;    yte = y[inTe]&#xA;    pred = np.zeros(xte.shape[0])&#xA;    for j in range(nbags):&#xA;        model = nn_model()&#xA;        fit = model.fit_generator(generator = batch_generator(xtr, ytr, 128, True),&#xA;                                  nb_epoch = nepochs,&#xA;                                  samples_per_epoch = xtr.shape[0],&#xA;                                  verbose = 0)&#xA;        pred += model.predict_generator(generator = batch_generatorp(xte, 800, False), val_samples = xte.shape[0])[:,0]&#xA;        pred_test += model.predict_generator(generator = batch_generatorp(xtest, 800, False), val_samples = xtest.shape[0])[:,0]&#xA;    pred /= nbags&#xA;    pred_oob[inTe] = pred&#xA;    score = mean_absolute_error(yte, pred)&#xA;    i += 1&#xA;    print('Fold ', i, '- MAE:', score)&#xA;&#xA;print('Total - MAE:', mean_absolute_error(y, pred_oob))&#xA;&#xA;## train predictions&#xA;df = pd.DataFrame({'id': id_train, 'loss': pred_oob})&#xA;df.to_csv('preds_oob.csv', index = False)&#xA;&#xA;## test predictions&#xA;pred_test /= (nfolds*nbags)&#xA;df = pd.DataFrame({'id': id_test, 'loss': pred_test})&#xA;df.to_csv('submission_keras.csv', index = False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I noticed that this code only sets the seed for numpy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I executed this code twice using a CPU, I would get the same results due to np.random.seed(123) on line #9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I have executed it twice using a GPU obtaining two different results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow allows us to set a seed, used even on the GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Keras I haven't found this feature. Googling it, I've found that the dropout function admits a seed parameter. But I have not found a global gpu seed. Neither seed parameter on model.fit_generator, nor Dense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to have the same results when code is executed twice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to set a seed on the GPU for TensorFlow using Keras?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could this code be deterministic on a GPU?&lt;/p&gt;&#xA;" OwnerUserId="23837" LastEditorUserId="31513" LastEditDate="2017-05-28T16:35:57.373" LastActivityDate="2020-06-18T00:33:25.600" Title="Making Keras + Tensorflow code execution deterministic on a GPU" Tags="|deep-learning|tensorflow|keras|gpu|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14813" PostTypeId="2" ParentId="14805" CreationDate="2016-10-29T09:55:00.450" Score="0" Body="&lt;p&gt;You can code certain simple rules like the ones you have mentioned in the question. Additionally, you can use knowledge bases like &lt;a href=&quot;https://www.google.com/webhp?sourceid=chrome-instant&amp;amp;ion=1&amp;amp;espv=2&amp;amp;ie=UTF-8#q=Freebase&quot; rel=&quot;nofollow&quot;&gt;Freebase&lt;/a&gt; and &lt;a href=&quot;https://wordnet.princeton.edu/&quot; rel=&quot;nofollow&quot;&gt;WordNet&lt;/a&gt; to enrich your language model. Note that this will not necessarily &quot;noisify&quot; your data but would have effect similar to the effect on data augmentation on say images for downstream tasks.&lt;/p&gt;&#xA;" OwnerUserId="75" LastActivityDate="2016-10-29T09:55:00.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14814" PostTypeId="2" ParentId="14156" CreationDate="2016-10-29T11:57:25.960" Score="1" Body="&lt;p&gt;If you have a good amount of instances for every class, you can try using a density-based approach for clustering, with algorithms like DBSCAN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can label at least some of the documents, you can use semi-supervised learning. Usually, when SSL is used for clustering, you need to specify &quot;cannot link&quot; and &quot;must link&quot; constraints for some pairs of instances, which is basically labeling some instances. One algorithm that follows this approach is HMRF-KMeans (Hidden Markov Random Fields K-Means).&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2016-10-29T11:57:25.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14815" PostTypeId="2" ParentId="9839" CreationDate="2016-10-29T12:29:31.937" Score="1" Body="&lt;p&gt;As I can see you're searching for a case study. Learning from data is useful because you can learn both the conditional probability distributions (CPDs) and the structure of the network. The best resource I've read is a freely-available Microsoft tech report. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at the &lt;a href=&quot;http://research.microsoft.com/en-us/um/people/heckerman/tutorial.pdf&quot; rel=&quot;nofollow&quot;&gt;paper&lt;/a&gt;, maybe it can help you upto some extend. I'll suggest you to start with &lt;code&gt;Chapter 7&lt;/code&gt; of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-10-29T12:29:31.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14816" PostTypeId="1" AcceptedAnswerId="14818" CreationDate="2016-10-29T14:27:28.897" Score="0" ViewCount="259" Body="&lt;p&gt;I didn't found any good resource that explains me very well what type of information I can extract from a Matrix based Visualization like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2P18w.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2P18w.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do't uderstand what RHS and LHS represents&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone can explain me the meaning of this chart?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-10-29T15:07:54.467" Title="Matrix based Visualization Meaning - Assocation Rules" Tags="|visualization|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14817" PostTypeId="1" AcceptedAnswerId="14821" CreationDate="2016-10-29T14:28:51.317" Score="2" ViewCount="5847" Body="&lt;p&gt;I have a Pandas DataFrame structured like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    user_id movie_id    rating&#xA;0   1       1193        5&#xA;1   2       1193        5&#xA;2   12      1193        4&#xA;3   15      1193        4&#xA;4   17      1193        5&#xA;5   18      1193        4&#xA;6   19      1193        5&#xA;7   24      1193        5&#xA;8   28      1193        3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row corresponds to a &lt;code&gt;rating&lt;/code&gt; event performed by the user &lt;code&gt;user_id&lt;/code&gt; for the movie &lt;code&gt;movie_id&lt;/code&gt;. For instance, the first row says that user &lt;code&gt;1&lt;/code&gt; rated the movie &lt;code&gt;1193&lt;/code&gt; with a rating of &lt;code&gt;5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This data comes from the &lt;a href=&quot;http://grouplens.org/datasets/movielens/&quot; rel=&quot;nofollow&quot;&gt;MovieLens&lt;/a&gt; project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My goal is to find all the users who satisfy these two conditions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rated movie &lt;code&gt;588&lt;/code&gt; with a rating of &lt;code&gt;5&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;rated movie &lt;code&gt;3578&lt;/code&gt; with a rating of &lt;code&gt;3&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I came up with two filtered DataFrame objects for each of the above conditions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ratings_588_5 = data[(data.movie_id == 588) &amp;amp; (data.rating == 5]&#xA;ratings_3578_3 = data[(data.movie_id == 3578) &amp;amp; (data.rating == 3)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which result in, respectively:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; ratings_588_5&#xA;user_id movie_id    rating&#xA;438     588         5&#xA;758     588         5&#xA;913     588         5&#xA;1024    588         5&#xA;1214    588         5&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; ratings_3578_3&#xA;user_id movie_id    rating&#xA;45      3578        3&#xA;321     3578        3&#xA;467     3578        3&#xA;758     3578        3&#xA;1024    3578        3&#xA;1381    3578        3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In Pandas, how can I compute a &lt;code&gt;list&lt;/code&gt; of all &lt;code&gt;user_id&lt;/code&gt; which appear in both DataFrames?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example, the result I want to obtain is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[758, 1024]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25640" LastActivityDate="2016-11-08T13:05:57.193" Title="Pandas - Get feature values which appear in two distinct dataframes" Tags="|python|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14818" PostTypeId="2" ParentId="14816" CreationDate="2016-10-29T15:07:54.467" Score="0" Body="&lt;p&gt;An association rule is (usually) of the format &lt;code&gt;X -&amp;gt; Y&lt;/code&gt;, meaning that if X happens then Y is likely to happen. In the traditional example of supermarket, a possible rule would be something like &lt;code&gt;Bread -&amp;gt; Butter&lt;/code&gt;. In this case, X is called the antecedent, or LHS (left-hand side), and Y is the consequent, or RHS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This specific chart has the number of items in the LHS as it's horizontal axis, and the number of items in the RHS as the vertical axis. The colors represents the metrics of support and confidence, as can be seen in the legend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example information that can be read from this chart is that the support is higher for lower number of items in the LHS, because the left side of the chart looks red-ish and the right side looks blue-ish, matching our intuition (more items in the LHS is more restrictive and thus less likely to happen). The confidence doesn't seem to be related to any of the axes in particular.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2016-10-29T15:07:54.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14819" PostTypeId="2" ParentId="14817" CreationDate="2016-10-29T15:17:46.793" Score="2" Body="&lt;p&gt;One possible way is to convert the user id's to plain old &lt;a href=&quot;https://docs.python.org/3/library/stdtypes.html#set&quot; rel=&quot;nofollow&quot;&gt;Python sets&lt;/a&gt; and check the intersection:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;users_rating_588_5 = set(ratings_588_5['user_id'])&#xA;users_rating_3578_3 = set(ratings_3578_3['user_id'])&#xA;users_matching = users_rating_588_5.intersection(users_rating_3578_3)&#xA;print(users_matching) # {1024, 758}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2016-10-29T15:17:46.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14821" PostTypeId="2" ParentId="14817" CreationDate="2016-10-29T15:29:21.653" Score="6" Body="&lt;p&gt;you can use &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.intersect1d.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;numpy.intersect1d()&lt;/a&gt; method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [277]: np.intersect1d(a.user_id, b.user_id).tolist()&#xA;Out[277]: [758, 1024]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;a href=&quot;https://github.com/pandas-dev/pandas/blob/master/pandas/core/common.py#L347&quot; rel=&quot;nofollow noreferrer&quot;&gt;pd.core.common.intersection()&lt;/a&gt; method, but it seems to be slow (at least on my notebook for &lt;code&gt;aa&lt;/code&gt; and &lt;code&gt;bb&lt;/code&gt; DataFrames [see setup below...]):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [307]: pd.core.common.intersection(a.user_id, b.user_id).tolist()&#xA;Out[307]: [1024, 758]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Timing&lt;/strong&gt; for &lt;code&gt;aa&lt;/code&gt; DF (50K rows) and &lt;code&gt;bb&lt;/code&gt; DF (60K rows):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [294]: aa = pd.concat([a] * 10**4, ignore_index=True)&#xA;&#xA;In [295]: bb = pd.concat([b] * 10**4, ignore_index=True)&#xA;&#xA;In [296]: aa.shape&#xA;Out[296]: (50000, 3)&#xA;&#xA;In [297]: bb.shape&#xA;Out[297]: (60000, 3)&#xA;&#xA;In [298]: %timeit aa.ix[aa.user_id.isin(bb.user_id),'user_id'].tolist()&#xA;10 loops, best of 3: 41.8 ms per loop&#xA;&#xA;In [299]: %timeit np.intersect1d(aa.user_id, bb.user_id).tolist()&#xA;100 loops, best of 3: 5.36 ms per loop&#xA;&#xA;In [300]: %timeit pd.merge(aa, bb, on='user_id').user_id.tolist()&#xA;...&#xA;skipped&#xA;...&#xA;MemoryError:&#xA;&#xA;In [308]: %timeit pd.core.common.intersection(aa.user_id, bb.user_id).tolist()&#xA;10 loops, best of 3: 52.8 ms per loop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PS &lt;a href=&quot;https://stackoverflow.com/a/40320353/5741205&quot;&gt;original answer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-29T15:29:21.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14824" PostTypeId="2" ParentId="14690" CreationDate="2016-10-29T20:45:01.183" Score="5" Body="&lt;p&gt;IIUC you can use [DataFrame.hist()] method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib&#xA;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;&#xA;matplotlib.style.use('ggplot')&#xA;&#xA;df = pd.DataFrame(np.random.randint(0,10,(20,4)),columns=list('abcd'))&#xA;&#xA;df.hist(alpha=0.5, figsize=(16, 10))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VBQue.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VBQue.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [44]: df&#xA;Out[44]:&#xA;    a  b  c  d&#xA;0   3  0  2  5&#xA;1   8  7  6  6&#xA;2   6  4  5  7&#xA;3   4  4  0  6&#xA;4   5  6  0  2&#xA;5   0  0  4  8&#xA;6   7  6  7  4&#xA;7   7  6  6  2&#xA;8   6  5  9  4&#xA;9   6  3  6  9&#xA;10  7  9  7  6&#xA;11  9  3  5  6&#xA;12  9  4  7  0&#xA;13  2  8  8  8&#xA;14  0  8  4  7&#xA;15  1  5  2  4&#xA;16  2  6  6  4&#xA;17  0  3  8  1&#xA;18  4  1  0  4&#xA;19  4  4  6  8&#xA;&#xA;In [45]: df.skew()&#xA;Out[45]:&#xA;a   -0.154849&#xA;b   -0.239881&#xA;c   -0.660912&#xA;d   -0.376480&#xA;dtype: float64&#xA;&#xA;In [46]: df.describe()&#xA;Out[46]:&#xA;               a          b          c          d&#xA;count  20.000000  20.000000  20.000000  20.000000&#xA;mean    4.500000   4.600000   4.900000   5.050000&#xA;std     2.964705   2.521487   2.770142   2.502105&#xA;min     0.000000   0.000000   0.000000   0.000000&#xA;25%     2.000000   3.000000   3.500000   4.000000&#xA;50%     4.500000   4.500000   6.000000   5.500000&#xA;75%     7.000000   6.000000   7.000000   7.000000&#xA;max     9.000000   9.000000   9.000000   9.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="37657" LastEditDate="2019-12-21T07:28:03.970" LastActivityDate="2019-12-21T07:28:03.970" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14825" PostTypeId="1" CreationDate="2016-10-29T22:39:32.167" Score="1" ViewCount="154" Body="&lt;p&gt;I've only seen examples that use (various types of) neural networks for both the discriminative and generative model. Is it not sound to use say, a logistic regression model for the discriminative model instead? Do I have to use a neural net instead of say, a clustering algorithm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the &quot;generate novel samples using one technique, then discriminate between novel and original samples using a different technique, then reinforce anything learned that makes discrimination more difficult&quot; general theory behind generative adversarial networks &lt;em&gt;have&lt;/em&gt; to be used with neural networks? Or is that just because most uses of GANs today involve image data, which seems to be particularly conducive to neural network analysis?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastEditorUserId="29169" LastEditDate="2021-02-16T15:35:24.687" LastActivityDate="2021-02-16T15:35:24.687" Title="Do I have to use neural networks in a generative adversarial network?" Tags="|machine-learning|neural-network|image-classification|convolutional-neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14826" PostTypeId="1" CreationDate="2016-10-29T23:18:33.057" Score="1" ViewCount="173" Body="&lt;p&gt;I want to predict using a multi layer perceptron. My data have 3 attributes x,y,z. My window size (input) is 3. That is, using 3 sets of values for x,y,z to predict next set of x,y,z. What should be my number of input neurons? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should it be 6 input neurons for 3 sets of values of x,y,z?&lt;br&gt;&#xA;or should it be 3 input neurons and each neuron has 3 inputs for x,y,z?&lt;/p&gt;&#xA;" OwnerUserId="25665" LastActivityDate="2016-10-30T21:25:45.570" Title="neural network number of input neuron" Tags="|neural-network|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14827" PostTypeId="2" ParentId="13377" CreationDate="2016-10-30T00:24:19.547" Score="1" Body="&lt;p&gt;Seems to me that &lt;code&gt;fig.add_axes()&lt;/code&gt; is miscalculating the dimensions here, essentially doubling the &quot;left&quot; and &quot;bottom&quot; values, not sure why. So you should use 0.8 for &quot;right&quot; and &quot;top&quot; values, as in: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;axes = [fig.add_axes([0.1, 0.1, .8, .8], polar=True,&#xA;            label=&quot;axes{}&quot;.format(i))&#xA;            for i in range(len(variables))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14558" LastActivityDate="2016-10-30T00:24:19.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14829" PostTypeId="1" AcceptedAnswerId="14830" CreationDate="2016-10-30T10:03:53.620" Score="0" ViewCount="1641" Body="&lt;p&gt;Imagine that I've this dataset (just a sample)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A   B   C&#xA;1   23  1000&#xA;2   52  5000&#xA;3   12  500&#xA;4   10  450&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to assign each row to a clustering based on C value. Like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A   B   C   CLUSTER&#xA;1   23  1000    2&#xA;2   52  5000    1&#xA;3   12  500     3&#xA;4   10  450     3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For that I'm using K-Means algorithm using Spark:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}&#xA;import org.apache.spark.mllib.linalg.Vectors&#xA;&#xA;val data = sc.textFile(&quot;/user/cloudera/TESTE1&quot;)&#xA;val parsedData = data.map(s =&amp;gt; Vectors.dense(s.split(',').map(_.toDouble))).cache()&#xA;&#xA;val numClusters = 4&#xA;val numIterations = 20&#xA;val clusters = KMeans.train(parsedData, numClusters, numIterations)&#xA;&#xA;val WSSSE = clusters.computeCost(parsedData)&#xA;println(&quot;Within Set Sum of Squared Errors = &quot; + WSSSE)&#xA;&#xA;clusters.save(sc, &quot;/user/cloudera/KMeansModel&quot;)&#xA;val sameModel = KMeansModel.load(sc, &quot;/user/cloudera/KMeansModel&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this script extracts me a .gz.parquet file and when I try to see what type of information this file contains, using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sqlContext.read.parquet(&quot;/user/cloudera/KMeansModel/data/part-r-00000-f551ea29-54db-45be-8cba-d06a97d6d9f2.gz.parquet&quot;).show&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm getting this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+---+--------------------+&#xA;| id|               point|&#xA;+---+--------------------+&#xA;|  0|[9.39601519885208...|&#xA;|  1|[9.80112351958380...|&#xA;|  2|[9.63822872186722...|&#xA;|  3|[9.44194658832542...|&#xA;+---+--------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I get the table that I put above? Basically I just want to extract the same fields and add a column with the cluster calculated by K-Mwans to each row...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-10-30T18:00:02.157" Title="SPARK RDD - Clustering - K-Means" Tags="|clustering|apache-spark|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14830" PostTypeId="2" ParentId="14829" CreationDate="2016-10-30T14:01:31.463" Score="1" Body="&lt;p&gt;This is well answered in this earlier question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/q/31447141/1060350&quot;&gt;https://stackoverflow.com/q/31447141/1060350&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beware that &lt;strong&gt;Spark k-means is slow&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data fits into main memory (i.e. a few gigabyte, which means billions of vectors!) then other tools such as ELKI that don't have the cluster overhead will be much faster. Use spark only for preprocessing the data, if you e.g. have several TB of jsons, and you need to first extract the numbers out of the JSONs, then here is where Spark shines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once your data is then vectors, use ELKI instead, it's much faster.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-30T18:00:02.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14831" PostTypeId="1" CreationDate="2016-10-30T16:37:03.810" Score="1" ViewCount="3506" Body="&lt;p&gt;I have run a logistic regression using scikit learn in python.  I know want to output the results to put into a csv and then load into Tableau.  To do that I need to combine the y_test, y_actual, and X_test data.  I was wondering if there's a way to output the y_test, y_actual, and X_test data?  I know I can use to_csv, but when I try that I can extract each of these data and concat together, but I'm afraid its not matching correctly since there's no identifying to join on.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, is there a way to keep a unique id (uid) with the logistic regression so then it's easy to see what the regression predicts for a specific person?&lt;/p&gt;&#xA;" OwnerUserId="25681" LastActivityDate="2016-10-30T21:54:15.777" Title="Output data from scikit learn logistic regression" Tags="|python|scikit-learn|logistic-regression|csv|tableau|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14833" PostTypeId="2" ParentId="14826" CreationDate="2016-10-30T18:31:53.850" Score="1" Body="&lt;p&gt;My understanding of the question is that you are using a basic MLP, feed-forward neural network, to work with data that is a 3-dimensional vector which you are calling $(x,y,z)$ - although note that most notation you read would call the whole observation $x$ which would be a vector $(x_1, x_2, x_3)$. You want to accept an input of 1st, 2nd, 3rd observations of this data (a window on some possibly longer series), and predict the 4th observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the given architecture, the prediction is made immediately on the supplied input - a MLP has no &quot;memory&quot; for previous inputs - so you will need 3 x 3 = 9 inputs to represent each (x,y,z) in the series. For a MLP it doesn't really matter how you arrange this input - e.g. you could have $(x^{(1)}, y^{(1)}, z^{(1)},x^{(2)}, y^{(2)}, z^{(2)},x^{(3)}, y^{(3)}, z^{(3)})$ or $(x^{(1)}, x^{(2)}, x^{(3)},y^{(1)}, y^{(2)}, y^{(3)},z^{(1)}, z^{(2)}, z^{(3)})$ or any other arrangement as long as you are consistent for every training example and prediction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't usually want or need to somehow combine the x, y and z values to create a pre-processed layer with 3 inputs, or create a special layer that combines $x^{(1)}, x^{(2)}, x^{(3)}$. You &lt;em&gt;might&lt;/em&gt; do that if your understanding of how the sequence worked allowed you to come up with features that modelled something important. However, the more usual approach is to allow the neural network to figure out how the multiple inputs combine from the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to look at alternative architecture of Recurrent Neural Network (RNN), which has more options for dealing with predictions of series. For a RNN, you would have 3 input neurons, and you would run the network 3 times in sequence and read the prediction after running it the third time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have noticed correctly that a MLP has no &quot;understanding&quot; internally that your x in (x,y,z) at step 1 is the same type of variable as the x in steps 2 and 3. The MLP would have 9 separate inputs and would ignore any direct relations, or even that there is a sequence being predicted. Whilst in a RNN, both those details are built into the model as core assumptions. In some cases that could be an important advantage to the RNN. But it is still worth trying the simpler MLP first and testing whether its predictions are good enough for purpose.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-10-30T21:25:45.570" LastActivityDate="2016-10-30T21:25:45.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14834" PostTypeId="1" CreationDate="2016-10-30T19:28:26.220" Score="0" ViewCount="747" Body="&lt;p&gt;I have a complex physical system that depends on many continuous input parameters (let's say 10 important ones) and produces an output that I can boil down to one continuous figure of merit. I want to optimize this system and find the set of parameters that produces the best output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenges are the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The system is sufficiently complicated that I cannot predict the behavior well enough to build a useful model. I have to treat it as a black box and do all tests with the actual system.&lt;/li&gt;&#xA;&lt;li&gt;The input parameters are generally correlated. So, optimizing the parameters individually is not enough.&lt;/li&gt;&#xA;&lt;li&gt;The system is slow to react. When changing the parameters it takes some time until an equilibrium state is reached again, depending also on the magnitude of the change.&lt;/li&gt;&#xA;&lt;li&gt;The system is noisy. Even when not touching the system the output figure of merit will fluctuate strongly. I want to optimize the long-term average of the figure of merit, but in order to get a good estimate of that I have to measure for some time (many minutes if I wanted to bring the noise down to a negligible level).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the system is relatively well behaved when it comes to the shape of the function to optimize:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I know that I'm already close to the global optimum and there are probably no local optima to get stuck in.&lt;/li&gt;&#xA;&lt;li&gt;I expect no sudden changes of the figure of merit. The gradients vary by not more than maybe two orders of magnitude over the permissible parameter space.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; What is a good technique to apply to this problem? Naïve approaches like a simple grid search are prohibitively time-intensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like there will be a trade-off between long averaging and dealing with noisy data. So I would either need an algorithm that makes as few queries as possible, or one that's very robust to noise.&lt;/p&gt;&#xA;" OwnerUserId="25488" LastEditorUserId="25488" LastEditDate="2016-10-30T19:46:53.027" LastActivityDate="2016-10-30T20:49:49.057" Title="Which optimization technique to use with noisy and slow black-box system?" Tags="|algorithms|optimization|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="14835" PostTypeId="2" ParentId="14834" CreationDate="2016-10-30T20:43:40.063" Score="1" Body="&lt;p&gt;Are you a petroleum engineer by any chance? This sounds very like the sort of optimization you have to perform in oilfield waterflood operations; you have to allocate your limited budget of injection water and/or lift gas to maximize oil production, with the additional complication that the optimal set point changes as the reservoir depletes and/or wells get drilled or abandoned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The oilfield case has the useful feature that the overall objective function (field oil flowrate) is a simple sum of well flow rates. Each production well is generally only sensitive (in the short term!) to a small local subset of the inputs (gas lift rate in the production well, and water injection rate in near-offset water injection wells).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would generalise the solution employed in the oilfield as follows: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Regularly perturb the inputs&lt;/li&gt;&#xA;&lt;li&gt;Measure the affected &lt;em&gt;local&lt;/em&gt; outputs&lt;/li&gt;&#xA;&lt;li&gt;Maintain a simple linear or low-order polynomial model of the system and its parts&lt;/li&gt;&#xA;&lt;li&gt;Use the model to optimize day to day&lt;/li&gt;&#xA;&lt;li&gt;Keep good records to ensure that the model is up to date with reality&lt;/li&gt;&#xA;&lt;li&gt;Monitor local and global model fidelity to decide what to perturb and recalibrate next&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This works because the small perturbations that you keep on making (for model calibration purposes) are of short duration and local effect, so they don't take the overall system very far away from the optimum. The trade-off between keeping the system optimized, and &quot;de-optimizing&quot; it to calibrate the model, works out favorably. The system reacts on a timescale of days to months, and the measurements generally take a day or two to accomplish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try Googling &lt;em&gt;waterflood optimization&lt;/em&gt; and &lt;em&gt;gaslift optimization&lt;/em&gt; for some more detailed pointers, although a lot of the trade literature is paywalled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking forward to learning what approaches have been taken in other fields with this kind of problem.&lt;/p&gt;&#xA;" OwnerUserId="25678" LastEditorUserId="25678" LastEditDate="2016-10-30T20:49:49.057" LastActivityDate="2016-10-30T20:49:49.057" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14836" PostTypeId="2" ParentId="14788" CreationDate="2016-10-30T21:01:13.660" Score="0" Body="&lt;p&gt;From my knowledge it decomposes the time series to a seasonal component as well since you specify it, that the seasonality is 4. Moreover, you could check the seasonality with 4 using the &lt;a href=&quot;https://github.com/trnnick/TStools&quot; rel=&quot;nofollow&quot;&gt;TStools&lt;/a&gt; (seasplot() function), it will provide you with a p-value of how significant is that seasonality, which seems more informative than a TRUE or a FALSE.&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-10-30T21:01:13.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14837" PostTypeId="2" ParentId="14831" CreationDate="2016-10-30T21:42:45.350" Score="2" Body="&lt;p&gt;Each line is treated independently at prediction, so you can be sure that the data is kept in the same order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, you can keep your data in a pandas dataframe. Here's a short working example:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;X, y = make_classification(n_samples=100, n_features=10)&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)&#xA;&#xA;clf = LogisticRegression()&#xA;clf.fit(X_train, y_train)&#xA;&#xA;data_test = pd.DataFrame(data=X_test, columns=['f{}'.format(i) for i in range(1, 11)])&#xA;data_test['y_test'] = y_test&#xA;data_test['y_pred'] = clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14089" LastEditorUserId="14089" LastEditDate="2016-10-30T21:54:15.777" LastActivityDate="2016-10-30T21:54:15.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14838" PostTypeId="1" CreationDate="2016-10-30T22:28:13.740" Score="1" ViewCount="417" Body="&lt;p&gt;Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does one do this? Is it something to do with perplexity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would also be glad if someone posted a reference to a reasonable Python implementation of the suggested method [a code snippet or a link to nltk/spacy/whatever method].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="14048" LastActivityDate="2017-04-07T22:15:28.980" Title="Given one language ngram model, how do I compare likelihoods of two texts of different length?" Tags="|nlp|nltk|language-model|sequence|text-generation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14840" PostTypeId="2" ParentId="13756" CreationDate="2016-10-30T23:39:00.517" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow&quot;&gt;Overfitting?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This happens when you make a model too specific to the training set, so that it performs very well on that particular training data, but then it is not able to generalize to other data (&quot;real world data&quot;) and so performs poorly in reality.&lt;/p&gt;&#xA;" OwnerUserId="25687" LastActivityDate="2016-10-30T23:39:00.517" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14843" PostTypeId="1" AcceptedAnswerId="31858" CreationDate="2016-10-31T06:23:25.297" Score="1" ViewCount="238" Body="&lt;p&gt;I am trying to train a &lt;code&gt;neural network&lt;/code&gt; for recognizing handwritten letters from A to J . I have a training set of size 200000 . Each training set is a list of  784 pixel values. My neural net has &lt;code&gt;input layer of size 784&lt;/code&gt; , &lt;code&gt;hidden layer of size 50&lt;/code&gt; and &lt;code&gt;output layer of size 10&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;strong&gt;fmin_cg minimization&lt;/strong&gt; function of &lt;code&gt;scipy&lt;/code&gt; library of python. The problem I am facing is that each iteration is taking a lot of time.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first iteration took almost 7-10 minutes. &lt;/li&gt;&#xA;&lt;li&gt;The second iteration took 20 minutes.&lt;/li&gt;&#xA;&lt;li&gt;Third is still running. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This might be due to my outdated computer with only 2 gb of memory and a slow processor but I have previously trained a neural net with the &lt;code&gt;training set of size 5000&lt;/code&gt; , &lt;code&gt;input layer size if 400&lt;/code&gt; , &lt;code&gt;hidden layer size 25&lt;/code&gt; and the &lt;code&gt;output layer of size 10&lt;/code&gt; . This neural net recognized handwritten digits and it was an exercise problem of coursera course on machine learning by Andrew Ng . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So  yes I know that the current neural network should take more time to train than the previous one as the training set , input layer, and hidden layer are all much larger than previous neural net but still, I think it's taking a lot of time . Why is it so slow ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it normal for the neural network of this size ? Or should I use other faster optimization algorithm ? Is there a way to measure time complexity of neural networks ?&lt;/p&gt;&#xA;" OwnerUserId="24077" LastEditorUserId="24077" LastEditDate="2016-10-31T06:36:32.977" LastActivityDate="2018-05-19T18:18:08.207" Title="Time complexity of function minimizers for neural networks" Tags="|machine-learning|neural-network|deep-learning|optimization|training|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14845" PostTypeId="1" CreationDate="2016-10-31T07:33:34.623" Score="1" ViewCount="185" Body="&lt;p&gt;I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as I can see it is quite &quot;primitive&quot; in comparison with TF and Keras (there are only feedforward types and hidden layer/output layer activation functions are hard-wired), there is no wide variety of optimizers, cost functions, architecture types etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any competitive alternative for large scale neural networks? Of course, I could use Amazon with very powerful, GPU-driven machines but these are not parallel frameworks either...&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-09-27T12:14:58.217" Title="How to Scaling Out Artifical Neural Networks?" Tags="|neural-network|apache-hadoop|distributed|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14846" PostTypeId="1" CreationDate="2016-10-31T08:19:15.067" Score="1" ViewCount="151" Body="&lt;p&gt;Trying to improve my chat App:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using previous (pre-processed) chat interactions from my domain, I have built a tool that offers the user 5 possible utterances to a given chat context, for example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Raw&lt;/strong&gt;: &quot;Hi John.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;: hi [[USER_NAME]]&lt;br&gt;&#xA;&lt;strong&gt;Utterances&lt;/strong&gt; : [Hi ,Hello , How are you, Hi there, Hello again]&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Of Course the results are not always relevant, for example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Raw&lt;/strong&gt;: &quot;Hi John. How are you? I am fine, are you in the office?&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;: hi [[USER_NAME]] how are you i am fine are you in the office&lt;br&gt;&#xA;&lt;strong&gt;Utterances&lt;/strong&gt; : [Yes, No , &lt;strong&gt;&lt;em&gt;Hi&lt;/em&gt;&lt;/strong&gt; , Yes i am, &lt;strong&gt;&lt;em&gt;How are you&lt;/em&gt;&lt;/strong&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;a href=&quot;https://www.elastic.co/products/elasticsearch&quot; rel=&quot;nofollow&quot;&gt;Elasticsearch&lt;/a&gt; with TF/IDF similarity model and an index structured like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;  &quot;_index&quot;: &quot;engagements&quot;,&#xA;  &quot;_type&quot;: &quot;context&quot;,&#xA;  &quot;_id&quot;: &quot;48&quot;,&#xA;  &quot;_score&quot;: 1,&#xA;  &quot;_source&quot;: {&#xA;    &quot;context&quot;: &quot;hi [[USER_NAME]] how are you i am fine are you in the office&quot;,&#xA;    &quot;utterance&quot;: &quot;Yes I am&quot;&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt; I know for sure that for the context &quot;hi [[USER_NAME]] how are you i am fine are you in the office&quot; the utterance &quot;Yes I am&quot; is relevant, however &quot;Yes&quot; , &quot;No&quot; are relevant too because they appeared on a similar context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to use this &lt;a href=&quot;https://www.youtube.com/watch?v=_0Z8aGwjLYo&quot; rel=&quot;nofollow&quot;&gt;excellent video&lt;/a&gt;, as a starting point&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q:&lt;/strong&gt; How can I measure precision and recall, if all I know (from my raw data) is just one true utterance?&lt;/p&gt;&#xA;" OwnerUserId="25696" LastEditorDisplayName="user120539" LastEditDate="2021-07-12T15:53:09.277" LastActivityDate="2021-07-12T15:53:09.277" Title="Measuring precision and recall" Tags="|classification|nlp|data-mining|chatbot|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14847" PostTypeId="1" CreationDate="2016-10-31T12:14:04.133" Score="11" ViewCount="16079" Body="&lt;p&gt;I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form &quot;Action|Animation|Comedy|Family|Fantasy&quot;. I want to encode them using pandas.get_dummies() but since the columns have multiple values, how to deal with such conditions?&#xA;      Additinal information on below link(question moved from stackoverflow)&#xA;&lt;a href=&quot;https://stackoverflow.com/q/40331558/4028904&quot;&gt;https://stackoverflow.com/q/40331558/4028904&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25698" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2018-07-19T01:01:30.133" Title="Multiple Categorical values for a single feature how to convert them to binary using python" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14849" PostTypeId="1" CreationDate="2016-10-31T13:21:05.700" Score="2" ViewCount="136" Body="&lt;h2&gt;Background&lt;/h2&gt;&#xA;&lt;p&gt;I have a massive text block of poorly written statements, some facts, some opinions. It's all scraped from various public internet resources and all in english.&lt;/p&gt;&#xA;&lt;p&gt;I am now using nltk in python to try to work out some basic statements from this massive text block.&lt;/p&gt;&#xA;&lt;p&gt;My goal is not to capture all statements, false positives are not too bad either. I would simply like to get a few probably true and simple assertions from this text block.&lt;/p&gt;&#xA;&lt;h3&gt;Examples&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Bob is married to Lisa since 1983.&lt;/p&gt;&#xA;&lt;p&gt;Lisa's cat Snowball died two years ago.&lt;/p&gt;&#xA;&lt;p&gt;Bob and Lisa has a 30 year old son named George.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;From these simple sentences, I would like to parse out some simple facts, like:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;&amp;quot;Bob is married&amp;quot;, &amp;quot;Lisa is married&amp;quot;, &amp;quot;George is Lisa's son&amp;quot;, &amp;quot;George is Bobs son&amp;quot;, &amp;quot;Snowball is a cat&amp;quot;&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;It does not need to be exactly this.&lt;/p&gt;&#xA;&lt;h1&gt;Actual question&lt;/h1&gt;&#xA;&lt;p&gt;What is this kind of processing called? (Except for that it is a subcategory of Natural Language Processing.)&lt;/p&gt;&#xA;&lt;p&gt;What are the simpler and the harder approaches to this problem?&lt;/p&gt;&#xA;&lt;p&gt;How well has this problem been solved by public and proprietary algorithms?&lt;/p&gt;&#xA;" OwnerUserId="9270" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-01T14:08:46.910" Title="Creating statements from a raw text" Tags="|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14851" PostTypeId="2" ParentId="14847" CreationDate="2016-10-31T13:42:34.480" Score="12" Body="&lt;p&gt;I'm starting with the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;data = pd.DataFrame({'title': ['Avatar', 'Pirates', 'Spectre', 'Batman'],&#xA;                 'genres': ['Action|Adventure|Fantasy|Sci-Fi',&#xA;                            'Action|Adventure|Fantasy',&#xA;                            'Action|Adventure|Thriller',&#xA;                            'Action|Thriller']},&#xA;                columns=['title', 'genres'])&#xA;&#xA;&#xA;     title                           genres&#xA;0   Avatar  Action|Adventure|Fantasy|Sci-Fi&#xA;1  Pirates         Action|Adventure|Fantasy&#xA;2  Spectre        Action|Adventure|Thriller&#xA;3   Batman                  Action|Thriller&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First, you want to have your data in a structure pairing titles with one genre at a time, multiple rows per title. You can get it in a series like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cleaned = data.set_index('title').genres.str.split('|', expand=True).stack()&#xA;&#xA;&#xA;title&#xA;Avatar   0       Action&#xA;         1    Adventure&#xA;         2      Fantasy&#xA;         3       Sci-Fi&#xA;Pirates  0       Action&#xA;         1    Adventure&#xA;         2      Fantasy&#xA;Spectre  0       Action&#xA;         1    Adventure&#xA;         2     Thriller&#xA;Batman   0       Action&#xA;         1     Thriller&#xA;dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(There's an extra index level that we don't want, but we'll get rid of it soon.) &lt;code&gt;get_dummies&lt;/code&gt; will now work, but it only works on one row at a time, so we need to re-aggregate the titles:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.get_dummies(cleaned, prefix='g').groupby(level=0).sum()&#xA;&#xA;&#xA;         g_Action  g_Adventure  g_Fantasy  g_Sci-Fi  g_Thriller&#xA;title&#xA;Avatar        1.0          1.0        1.0       1.0         0.0&#xA;Batman        1.0          0.0        0.0       0.0         1.0&#xA;Pirates       1.0          1.0        1.0       0.0         0.0&#xA;Spectre       1.0          1.0        0.0       0.0         1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25419" LastEditorUserId="25419" LastEditDate="2016-11-01T19:04:10.937" LastActivityDate="2016-11-01T19:04:10.937" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14853" PostTypeId="2" ParentId="14801" CreationDate="2016-10-31T13:59:31.750" Score="0" Body="&lt;p&gt;if you're looking for something like this--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s4jYS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s4jYS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then these resources can be very useful for you. Have a look-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/Gmisc/vignettes/transitionPlot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to transitionPlot&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/visualizing-transitions-with-the-transitionplot-function/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Visualizing transitions with the transitionPlot function&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Apart from these, there is an interesting &lt;a href=&quot;https://stackoverflow.com/questions/16112162/graph-flow-chart-of-transition-from-states&quot;&gt;post&lt;/a&gt; on stack overflow as well. I hope this will help. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-10-31T13:59:31.750" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14854" PostTypeId="2" ParentId="14801" CreationDate="2016-10-31T14:32:03.860" Score="2" Body="&lt;p&gt;The problem with Sankey diagram in your case is that you will duplicate all the nodes (on the left, and on the right), that is messy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Networks are better, but nodes are not well structured.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest to try &lt;a href=&quot;https://en.wikipedia.org/wiki/Chord_diagram&quot; rel=&quot;nofollow noreferrer&quot;&gt;chord diagrams&lt;/a&gt; for such data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ID0EJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ID0EJ.png&quot; alt=&quot;chord example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the greatest examples of chord diagram was build with d3.js, and based on migration flows dataset. Here is an &lt;a href=&quot;https://github.com/mattflor/chorddiag&quot; rel=&quot;nofollow noreferrer&quot;&gt;R wrapper for d3.js&lt;/a&gt; chords. Or you can try to reproduce &lt;a href=&quot;https://gjabel.wordpress.com/2014/03/28/circular-migration-flow-plots-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this manual&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-10-31T14:32:03.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14860" PostTypeId="2" ParentId="14845" CreationDate="2016-10-31T18:58:02.223" Score="1" Body="&lt;p&gt;You might take a look over TensorFrames, a Databricks library which allow running TensorFlow code on top of Apache Spark : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/databricks/tensorframes&quot; rel=&quot;nofollow&quot;&gt;https://github.com/databricks/tensorframes&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25718" LastActivityDate="2016-10-31T18:58:02.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14861" PostTypeId="1" CreationDate="2016-11-01T07:13:34.667" Score="1" ViewCount="283" Body="&lt;h1&gt;This is what I am trying to accomplish:&lt;/h1&gt;&#xA;&lt;p&gt;Predict &lt;strong&gt;Y&lt;/strong&gt; based on &lt;strong&gt;X&lt;/strong&gt;,that is &lt;code&gt;Y ~ X&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;p&gt;1.&lt;strong&gt;&lt;code&gt;X = {n, x_1, x_2, x_3, ...}&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a vector of factors&lt;/li&gt;&#xA;&lt;li&gt;size unchanged: &lt;code&gt;X.size() = DIM_X&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;2.&lt;strong&gt;Y = {}&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;also a vector&lt;/li&gt;&#xA;&lt;li&gt;size changing according to &lt;code&gt;n&lt;/code&gt; in &lt;strong&gt;X&lt;/strong&gt;: &lt;code&gt;Y.size() = 4 * n&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1&gt;So, can I do this with NNs ? is there a NN-Model I can refer to ? Or Is there other approaches can solve this ？&lt;/h1&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Sample data:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;1 0.1 109 123.7 0 0 512 512&#xA;2 0.8 201 88.7 384 216 384 246 0 0 1024 768&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In the above, first 4 elements each row represents &lt;strong&gt;X&lt;/strong&gt;, what's left is &lt;strong&gt;Y&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Appreciate your help.&lt;/p&gt;&#xA;" OwnerUserId="24005" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-11T15:51:54.967" Title="Is there a NN-Model which has multiple outputs?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14862" PostTypeId="1" CreationDate="2016-11-01T09:20:22.320" Score="2" ViewCount="6278" Body="&lt;p&gt;I would like to build an recommendation engine using &lt;code&gt;spark's Mlib itemsimilarity&lt;/code&gt; as mentioned here &lt;a href=&quot;https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html&quot; rel=&quot;nofollow&quot;&gt;&lt;strong&gt;LINK&lt;/strong&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it seems spark do not have this algorithm any more and some forums suggesting me to use &lt;code&gt;ALS&lt;/code&gt;, but please clarify is item similarity removed from spark or any other alter natives?&#xA; I want to index my indicators into search engine and I find Sparks Item similarity best suits for me. Please advice.&lt;/p&gt;&#xA;" OwnerUserId="25734" LastActivityDate="2016-11-02T06:00:22.877" Title="spark item similarity recommendation" Tags="|apache-spark|apache-hadoop|recommender-system|pyspark|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14863" PostTypeId="2" ParentId="14741" CreationDate="2016-11-01T11:47:48.060" Score="3" Body="&lt;p&gt;Clustering will be much too expensive for your purpose (most are O(n^2), and the good ones like HAC may even be O(n^3) - you won't be able to run them on 300k instances). Also beware of the prerequisites of the various algorithms - they may not be applicable to your data, or the results may be as bad as random! Also, clustering algorithms may behave rather unpredictable, and will profuce &lt;em&gt;unbalanced&lt;/em&gt; data sets. It would not be unusual if 99% of your data ends up in the same cluster, while other clusters are almost empty (or even empty!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would consider first to &lt;strong&gt;remove all duplicates and maybe near-duplicate records&lt;/strong&gt; (ignoring identifiers, of course). These algorithms are much more scalable and tolerate more data types. That may already be enough to achieve your goal of data reduction.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-11-01T11:47:48.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14864" PostTypeId="1" CreationDate="2016-11-01T12:03:03.020" Score="4" ViewCount="3786" Body="&lt;p&gt;I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely &lt;em&gt;country&lt;/em&gt; and &lt;em&gt;product type&lt;/em&gt;. To achieve this, I have two proposals:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Build a separate tree for each combination of &lt;em&gt;country&lt;/em&gt; and &lt;em&gt;product type&lt;/em&gt; and use subsets of the data accordingly and pass on to respective tree for prediction. Seen &lt;a href=&quot;https://stackoverflow.com/questions/34588248/how-to-combine-2-decision-trees&quot;&gt;here&lt;/a&gt; in comments. I have 88 levels in &lt;em&gt;country&lt;/em&gt; and 3 levels in &lt;em&gt;product type&lt;/em&gt; so it will generate 264 trees.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Build a basic tree with two variables namely &lt;em&gt;country&lt;/em&gt; and &lt;em&gt;product type&lt;/em&gt; with appropriate cp value to generate all combination as leaf nodes (264). Build a second tree with rest all variables and stack tree one upon tree two as a single decision tree.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I do not think the first one is the right way to do it. Also, I am stuck on how to stack the trees in second approach, even if it is not the right way would love to know how to achieve this.&lt;/p&gt;&#xA;&lt;p&gt;Please guide me on how to approach this problem.&lt;/p&gt;&#xA;" OwnerUserId="25741" LastEditorUserId="29169" LastEditDate="2021-12-20T22:44:25.947" LastActivityDate="2024-02-04T10:04:15.517" Title="Fix first two levels of decision tree?" Tags="|machine-learning|r|predictive-modeling|decision-trees|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="14865" PostTypeId="1" AcceptedAnswerId="14867" CreationDate="2016-11-01T13:54:48.413" Score="2" ViewCount="211" Body="&lt;p&gt;Suppose I have a datawarehouse (DWH) and now I would like to add many other bigdata sources of information most of them are not structured. I still keep the DWH with no architectural change. The only thing I do is to enrich the bigdata with the data resides in the DWH using connectivity. Can the DWH defined as datalake in this case?&lt;/p&gt;&#xA;" OwnerUserId="16251" OwnerDisplayName="Avi" LastEditorUserId="16251" LastEditDate="2016-11-01T14:00:17.280" LastActivityDate="2016-11-01T14:53:10.560" Title="Is datawarehouse considered as datalake in big data environnment?" Tags="|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14866" PostTypeId="2" ParentId="14849" CreationDate="2016-11-01T14:08:46.910" Score="1" Body="&lt;p&gt;You may want to read about the task of &lt;strong&gt;Relation Extraction&lt;/strong&gt;. The most common scenario is to extract triplets (entity1, relation, entity2). Usually entity1 and entity2 are given and the list of possible relations is known. Common methods use a classifier, others can be rule-based.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your example from the sentence &quot;Bob is married to Lisa since 1983.&quot; the system can extract (Bob, is_married, Lisa)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A variant is &lt;strong&gt;Open Information Extraction&lt;/strong&gt; which tries to extract relations but with no information specified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There exist multiple projects on the subject such as &lt;strong&gt;DeepDive&lt;/strong&gt;, &lt;strong&gt;IEPY&lt;/strong&gt; or &lt;strong&gt;openie-standalone&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://www.cs.cmu.edu/~nbach/papers/A-survey-on-Relation-Extraction-Slides.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this presentation&lt;/a&gt; or &lt;a href=&quot;http://www.cs.cmu.edu/~nbach/papers/A-survey-on-Relation-Extraction.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this review&lt;/a&gt; give an overview of the task.&lt;/p&gt;&#xA;" OwnerUserId="12558" LastActivityDate="2016-11-01T14:08:46.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14867" PostTypeId="2" ParentId="14865" CreationDate="2016-11-01T14:53:10.560" Score="2" Body="&lt;p&gt;Check out this article: &lt;a href=&quot;http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looks like the answer is yes, your DWH can be defined as datalake because you are adding unstructured data.  Here is an excerpt:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;“A data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed.”&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-11-01T14:53:10.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14868" PostTypeId="2" ParentId="12620" CreationDate="2016-11-01T15:57:14.897" Score="2" Body="&lt;p&gt;Am currently working on something in this domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rough process I am currently following is -&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Extract data from PDFs (ubiquitous version of Bank Statements nowadays) into more usable formats. Currently,converting them to TXT files first as an intermediate step.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Generically, bank statements (from a specific bank) tend to be structured in the same format. Hence, while converting the TXT to CSV, you can structure the algo such that it knows what to pick up - based on a rough analysis of the TXT file.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use 2 different data frames to store the statements content - one df keeping track of the transactional data (storing 3 values - Date, Description and Amount) and the other to keep track of the metadata (storing 2 values - Key and Value)&lt;/li&gt;&#xA;&lt;li&gt;You can determine the rough structure of the TXT file by going through a few TXTs. e.g. the transactions generally begin after a specific element's occurrence in every bank statement. Ignore the page footers / headers that recur in case of multi-page transactions. Also, another observation would be that all data in a specific column of the transactions (the transactions are read in as a table by my program [link given below]) appear together. &lt;/li&gt;&#xA;&lt;li&gt;Doing the above enables the transaction and meta data to a relatively structured and processable format &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Perform analysis on the Description column or the entire transactional df you have thus generated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can use the following code to convert a given PDF to a txt file -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import unicode_literals&#xA;import os&#xA;import sys  &#xA;reload(sys)  &#xA;sys.setdefaultencoding('utf8')&#xA;from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter&#xA;from pdfminer.pdfpage import PDFPage&#xA;from pdfminer.converter import XMLConverter, HTMLConverter, TextConverter&#xA;from pdfminer.layout import LAParams&#xA;from cStringIO import StringIO&#xA;&#xA;def pdfparser(data):&#xA;    dest = data[:-3]+&quot;txt&quot;    &#xA;    print &quot;\n\n\n\n\n&quot;,dest    &#xA;    fp = file(data, 'rb')&#xA;    rsrcmgr = PDFResourceManager()&#xA;    retstr = StringIO()&#xA;    codec = 'utf-8'&#xA;    laparams = LAParams()&#xA;    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)&#xA;    # Create a PDF interpreter object.&#xA;    interpreter = PDFPageInterpreter(rsrcmgr, device)&#xA;    # Process each page contained in the document.&#xA;    for page in PDFPage.get_pages(fp):&#xA;        interpreter.process_page(page)&#xA;        data =  retstr.getvalue()&#xA;        data = unicode(data,'utf-8', errors='ignore')&#xA;    #print data&#xA;    # write data to a file&#xA;    print data&#xA;    with open(dest, &quot;w&quot;) as f:&#xA;        f.write(data)&#xA;&#xA;# set the working directory&#xA;path = &quot;F:/banking/payments/&quot;&#xA;os.chdir(path)&#xA;fls=os.listdir(path)&#xA;&#xA;for x in fls:&#xA;    if x[-3:]==&quot;pdf&quot;:&#xA;        pdfparser(path+x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25742" LastEditorUserId="25742" LastEditDate="2016-11-02T06:22:57.720" LastActivityDate="2016-11-02T06:22:57.720" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14869" PostTypeId="2" ParentId="14862" CreationDate="2016-11-01T16:39:17.860" Score="2" Body="&lt;p&gt;There is &lt;code&gt;spark-itemsimilarity&lt;/code&gt; command line tool that is based on Spark and Mahout. (It is not a library you import inside a Spark application.) &lt;a href=&quot;https://www.packtpub.com/books/content/how-to-build-a-recommender-by-running-mahout-on-spark&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is an explanation of how it works.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-11-01T16:39:17.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14871" PostTypeId="1" AcceptedAnswerId="14903" CreationDate="2016-11-01T19:07:37.377" Score="4" ViewCount="3265" Body="&lt;p&gt;I am using &lt;strong&gt;spark ml IDF estimator/model (TF-IDF)&lt;/strong&gt; to convert text features into vectors before passing it to the classification algorithm.&#xA;Here's the process:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Datasets:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Full sample data (labeled) &amp;lt;br&amp;gt;&#xA;Training (labeled)&amp;lt;br&amp;gt;&#xA;Test (labeled)&amp;lt;br&amp;gt;&#xA;Unseen (non-labeled)&amp;lt;br&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is my current workflow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Fit IDF model (idf-1) on full Sample data&amp;lt;br&amp;gt;&#xA;Apply(Transform) idf-1 on full sample data&amp;lt;br&amp;gt;&#xA;Split data set into Training and Test data&amp;lt;br&amp;gt;&#xA;Fit ML model on Training data&amp;lt;br&amp;gt;&#xA;Apply(Transform) model on Test data&amp;lt;br&amp;gt;&#xA;Apply(Transform) idf-1 on Unseen data&amp;lt;br&amp;gt;&#xA;Apply(Transform) model on Unseen data&amp;lt;br&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for &lt;code&gt;|D|&lt;/code&gt; and &lt;code&gt;DF|t, D|&lt;/code&gt; while use new &lt;code&gt;TF|t, D|&lt;/code&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?&lt;/p&gt;&#xA;" OwnerUserId="19147" LastEditorUserId="10902" LastEditDate="2016-11-02T14:23:24.843" LastActivityDate="2016-11-03T10:01:22.757" Title="Spark MLLib - how to re-use TF-IDF model" Tags="|machine-learning|classification|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14872" PostTypeId="1" CreationDate="2016-11-01T21:04:23.883" Score="2" ViewCount="482" Body="&lt;p&gt;In the past few years there has been an explosion in the quantity and quality of available data visualization tools in R. I've long ago stopped using &lt;code&gt;plot&lt;/code&gt; base graphics in favor of the wonderful &lt;code&gt;ggplot2&lt;/code&gt; package, and I'm considering going for interactivity now, perhaps using one of the JS-integration packages, and outputting HTML. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm often in the business of creating long HTML report pages with dozens of graphics in there, and I care about a number of factors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;load-time for the graphics generated cannot be too long.&lt;/li&gt;&#xA;&lt;li&gt;should work decently with Shiny and Rmarkdown (ideally flexdashboard).&lt;/li&gt;&#xA;&lt;li&gt;should have a non-insane syntax for tweaking options (i.e no 20 line JS code inserts to change legend location).&lt;/li&gt;&#xA;&lt;li&gt;should be relatively well-documented.&lt;/li&gt;&#xA;&lt;li&gt;should be actively maintained. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What are your go-to packages for interactive data viz?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="5297" LastEditorUserId="5297" LastEditDate="2016-11-01T21:17:05.380" LastActivityDate="2016-11-02T07:12:32.743" Title="High performance R libraries for interactive graphics" Tags="|r|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14873" PostTypeId="2" ParentId="14872" CreationDate="2016-11-01T21:04:23.883" Score="2" Body="&lt;p&gt;My current go-to are some graphics from Rstudio's &lt;code&gt;flexdashboard&lt;/code&gt; and a set of relatively well-maintained packages: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://rstudio.github.io/leaflet/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Leaflet&lt;/a&gt;&lt;/strong&gt;, a library for creating dynamic maps that support panning and&#xA;  zooming, with various annotations like markers, polygons, and popups.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://rstudio.github.io/dygraphs&quot; rel=&quot;nofollow noreferrer&quot;&gt;dygraphs&lt;/a&gt;&lt;/strong&gt;, which provides rich facilities for charting time-series data&#xA;  and includes support for many interactive features including&#xA;  series/point highlighting, zooming, and panning.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://plot.ly/r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plotly&lt;/a&gt;&lt;/strong&gt;, which via its ggplotly interface allows you to easily&#xA;  translate your ggplot2 graphics to an interactive web-based version.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://hafen.github.io/rbokeh&quot; rel=&quot;nofollow noreferrer&quot;&gt;rbokeh&lt;/a&gt;&lt;/strong&gt;, an interface to Bokeh, a powerful declarative Bokeh framework&#xA;  for creating web-based plots.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://jkunst.com/highcharter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Highcharter&lt;/a&gt;&lt;/strong&gt;, a rich R interface to the popular Highcharts JavaScript&#xA;  graphics library.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;http://rmarkdown.rstudio.com/flexdashboard/using.html#components&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flexdashboard Components&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5297" LastActivityDate="2016-11-01T21:04:23.883" CommentCount="0" CommunityOwnedDate="2016-11-01T21:04:23.883" ContentLicense="CC BY-SA 3.0" />
  <row Id="14874" PostTypeId="1" AcceptedAnswerId="14906" CreationDate="2016-11-02T00:00:05.793" Score="0" ViewCount="642" Body="&lt;p&gt;I have a set of 10,000 integers, and another set of 100.&#xA;The integers in the first set are mapped to integers in the second set according to some rules (not mathematical rules, think of these values as codes for naming certain items, it is some categorical mapping).&#xA;The mapping is not necessarily 100 to 1, in some case I may have just 30 or so integers from the first set mapped to an integer in the second set, in other cases 300, but on average of course it is 100 to 1.&#xA;Using sklearn, I created a decision tree that was able to get over 99% accuracy, as I would expect.&#xA;When I tried logistic regression, though, accuracy was just 45%. The training sample is about 100,000 example, so, it should be enough to learn.&#xA;What is going on?&#xA;Is there something inherently different in the logistic regression method that I am missing?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2016-11-03T12:39:11.063" LastActivityDate="2016-11-06T16:07:34.500" Title="Contrasting logistic regression vs decision tree performance in specific example" Tags="|python|scikit-learn|logistic-regression|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14875" PostTypeId="1" AcceptedAnswerId="14878" CreationDate="2016-11-02T01:36:25.663" Score="4" ViewCount="4181" Body="&lt;p&gt;I am trying to collect some data for &lt;strong&gt;ML&lt;/strong&gt;, specifically for training a &lt;strong&gt;neural network model&lt;/strong&gt;, and I don't know how big the data set is enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is there a rule of thumb on how many data of dimension &lt;code&gt;DIM&lt;/code&gt; should one collect for training a NN-Model ? For example, it depends on the number of features or the kind of NN-Model or something else ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="24005" LastActivityDate="2016-11-02T05:23:04.473" Title="Data set size versus data dimension, is there a rule of thumb?" Tags="|machine-learning|neural-network|data-mining|dataset|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14876" PostTypeId="2" ParentId="14793" CreationDate="2016-11-02T05:03:43.007" Score="0" Body="&lt;p&gt;You can use SentiWordNet for classifying your data. SentiWordNet assigns to each synset of WordNet three sentiment scores: positivity, negativity, objectivity.&lt;/p&gt;&#xA;" OwnerUserId="25293" LastActivityDate="2016-11-02T05:03:43.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14877" PostTypeId="1" CreationDate="2016-11-02T05:05:13.200" Score="2" ViewCount="640" Body="&lt;p&gt;I have a user-generated list of proper names.  For the sake of conversation, imagine it's pet names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This list will have many variations of the same name: misspellings, alternative spellings, and even random use of punctuation or spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Fluffy&#xA;Fluffy the Dog&#xA;Flufy Dog&#xA;Fllufy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can get a count of each spelling to figure out that, say, Fluffy is the most common at 2,000 occurrences while the others are all &amp;lt;100.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to condense using the Levenshein distance package (&lt;code&gt;python-Levenshtein&lt;/code&gt;) which had mixed success.  I would assign a name to a &quot;more likely&quot; name if it's Levenshtein distance was &amp;lt; 3 and the &quot;more likely&quot; name had a higher count.  When I manually search a few known common names, I can manually weed out the Levenshein winners from the losers, as well as include others Levenshein missed in my list (eg &lt;code&gt;Fluffy the Dog&lt;/code&gt; and &lt;code&gt;Fluffy&lt;/code&gt; have a huge distance of 8).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there better techniques I should be trying?&lt;/p&gt;&#xA;" OwnerUserId="25611" LastActivityDate="2016-11-02T23:16:00.780" Title="Collapse a list to most common spellings" Tags="|python|text-mining|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14878" PostTypeId="2" ParentId="14875" CreationDate="2016-11-02T05:23:04.473" Score="4" Body="&lt;p&gt;In &lt;a href=&quot;https://www.youtube.com/watch?v=Dc0sr0kdBVI&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule of thumb, size of dataset should be at-least about 10x it's dimension and should be independent of the model used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;a href=&quot;http://www.cbcb.umd.edu/~salzberg/docs/murthy_thesis/survey/node16.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; link has summaries from some of the relevant papers, viz.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For a finite sized data with little or no a priori information, the ratio of the sample size to dimensionality must be as large as possible to suppress optimistically biased evaluations of the performance of the classifier.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This says, the ratio of size of dataset (sample) to dimension should be as large as possible to reduce classifier bias towards a particular class.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The ratio of the sample size to dimensionality should vary inversely&#xA;  proportional to the amount of available knowledge about the class&#xA;  conditional densities.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In a classifier setting, the more knowledge we have for each class' probability density, lesser can be the sample-size to dimension ratio. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In simpler terms, we can say we should be including as much data as possible and if we are not able to do that, include as much information as possible in the small dataset itself, because for any model to work we need to feed it with high variance dataset.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2016-11-02T05:23:04.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14879" PostTypeId="2" ParentId="14862" CreationDate="2016-11-02T06:00:22.877" Score="7" Body="&lt;p&gt;For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element  &lt;img src=&quot;https://latex.codecogs.com/gif.latex?r_%7Bij%7D&quot; alt=&quot;r_{ij}&quot;&gt;  of the matrix is the magnitude  of similarity between &lt;img src=&quot;https://latex.codecogs.com/gif.latex?item_%7Bi%7D&quot; alt=&quot;itemi&quot;&gt; and &lt;img src=&quot;https://latex.codecogs.com/gif.latex?item_%7Bj%7D&quot; alt=&quot;itemj&quot;&gt;. This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In spark this can be done by:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a &lt;code&gt;k X n&lt;/code&gt; matrix, where n items are described as k-dimensioned vectors. For representing each item as a &lt;code&gt;k&lt;/code&gt; dimension vector, you can use &lt;code&gt;ALS&lt;/code&gt; which represents each entity in a latent factor space. The dimension of this space (&lt;code&gt;k&lt;/code&gt;) can be chosen by you. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This k X n matrix can be represented as &lt;code&gt;RDD[Vector]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Convert this k X n matrix to &lt;a href=&quot;http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.linalg.distributed.RowMatrix&quot; rel=&quot;noreferrer&quot;&gt;RowMatrix&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;columnSimilarities()&lt;/code&gt; function to get a &lt;code&gt;n X n&lt;/code&gt; matrix of similarities between n items.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finally , whenever an itemi is being viewed, you can recommend other &lt;code&gt;m&lt;/code&gt; items obtained by sorting the items in row i by the decreasing value of similarities and picking top &lt;code&gt;m&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details can be found &lt;a href=&quot;https://databricks.com/blog/2014/10/20/efficient-similarity-algorithm-now-in-spark-twitter.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastEditorUserId="-1" LastEditDate="2017-03-10T09:42:41.977" LastActivityDate="2016-11-02T06:00:22.877" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14880" PostTypeId="1" AcceptedAnswerId="14966" CreationDate="2016-11-02T06:46:41.560" Score="2" ViewCount="1329" Body="&lt;p&gt;I have a time series of data points. Then I am given a future timestamp and  I have to predict the value for the data point. For simplicity, you can assume that the timestamp is bounded i.e. for e.g. query can be of at most 1 hr in the future. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is different from tradition train and predict models. Here you will be given the time as query input apart from past data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am training different models for each minute(yeah, 60 models: lots of waiting time ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if there is something available for this specific task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; To get the view about the data, you can assume a simple time series of real numbers, and I have to use history to predict for any general time(within next 1 hr).&lt;/p&gt;&#xA;" OwnerUserId="24346" LastEditorUserId="24346" LastEditDate="2016-11-06T15:11:53.820" LastActivityDate="2016-12-28T17:00:49.127" Title="Series prediction for any given time" Tags="|machine-learning|deep-learning|time-series|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14881" PostTypeId="2" ParentId="13107" CreationDate="2016-11-02T07:08:25.323" Score="1" Body="&lt;p&gt;You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 &amp;lt; c &amp;lt; 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-11-02T07:26:46.213" LastActivityDate="2016-11-02T07:26:46.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14883" PostTypeId="1" AcceptedAnswerId="14933" CreationDate="2016-11-02T08:15:53.687" Score="13" ViewCount="3435" Body="&lt;p&gt;I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Bot : Hi, what is your name?&#xA;User: My name is Edwin.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wish to extract the name Edwin from the sentence. However, the user can response in different ways such as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User: Edwin is my name.&#xA;User: I am Edwin.&#xA;User: Edwin. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm tried to rely on the dependency relations between words but the result does not do well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea on what technique I could use to tackle this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[UPDATED]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tested with named entity recognition together with part of speech tagger and parser. I found out that most model is trained in a way that the first character of the entity for the person name or the proper noun must be upper case. This may be true for normal document, but it is irrelevant for a chatbot. E.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User: my name is edwin.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Most NER failed to recognize this.&lt;/p&gt;&#xA;" OwnerUserId="20974" LastEditorUserId="20974" LastEditDate="2016-11-07T10:14:20.850" LastActivityDate="2021-01-05T16:28:43.830" Title="Extract information from sentence" Tags="|python|nlp|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14884" PostTypeId="1" CreationDate="2016-11-02T09:06:28.167" Score="1" ViewCount="195" Body="&lt;p&gt;I considered gather_nd, but it seemed that gather_nd need a index like [[0,..],[1,..],[2,..]] to get value from every row, and argmax just generated the 1-dimension indexes tensor, it seems that I have to run the argmax, get the narray, use for-loop to add the range index and then feed that into the graph again. I want to know is there any method that can directly get the result?&lt;/p&gt;&#xA;" OwnerUserId="25772" LastEditorUserId="25772" LastEditDate="2016-11-02T10:24:08.560" LastActivityDate="2016-11-02T10:24:08.560" Title="How to get value of a matrix from every row with the index generated by argmax in Tensorflow?" Tags="|tensorflow|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14885" PostTypeId="1" CreationDate="2016-11-02T09:44:46.437" Score="2" ViewCount="409" Body="&lt;p&gt;I installed Hadoop, Mahout and Spark. I am able to see the Hadoop and Spark MasterWebUI. Moreover, I can also run the following command, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[hadoop@muildevcel01 mahout]$ bin/mahout&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, we I try running the spark-shell I run in the problem stated below,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[hadoop@muildevcel01 mahout]$ bin/mahout spark-shell&#xA;&#xA;Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/spark/repl/SparkILoop&#xA;        at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)&#xA;Caused by: java.lang.ClassNotFoundException: org.apache.spark.repl.SparkILoop&#xA;        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)&#xA;        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)&#xA;        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)&#xA;        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)&#xA;        ... 1 more&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Any suggestions how I could resolve my problem?&lt;/p&gt;&#xA;" OwnerUserId="25734" LastEditorUserId="12384" LastEditDate="2016-11-02T13:31:11.770" LastActivityDate="2017-07-16T04:47:39.857" Title="Mahout Spark shell not working" Tags="|apache-spark|apache-hadoop|apache-mahout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14886" PostTypeId="2" ParentId="14883" CreationDate="2016-11-02T09:56:07.633" Score="5" Body="&lt;p&gt;You should use Named Entity Recognition, for example from NLTK. You can find a usage example &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It would work for your described case quite well.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastEditorUserId="85045" LastEditDate="2021-01-05T16:28:43.830" LastActivityDate="2021-01-05T16:28:43.830" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14887" PostTypeId="2" ParentId="14864" CreationDate="2016-11-02T11:47:41.007" Score="0" Body="&lt;p&gt;Depending which tree algorithm you want to use you could manually construct the two first levels of the tree. You can just follow the pseudo code explained for example &lt;a href=&quot;https://en.wikipedia.org/wiki/C4.5_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here for the C4.5 tree&lt;/a&gt;. Once you have done this you can remove the two features from the data set and create trees for the remaining part of the tree. If you want to create a &lt;code&gt;rpart&lt;/code&gt; object you would be required to take some parts of the source and this may be a bit more demanding. Depending on what tree algorithm you use you will just have a binary split at both levels so you will only need to build 4 separate trees and not 264. Note that you may not have the optimal decision tree since after stepping through the first two levels, the &lt;em&gt;country&lt;/em&gt; and &lt;em&gt;product type&lt;/em&gt; may still be variables that cause a split. But without seeing the data is impossible to tell. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note, it may be valuable to explain the business that &lt;em&gt;country&lt;/em&gt; and &lt;em&gt;product type&lt;/em&gt; are not the most sensible variables to have in the top of the decision tree. Sometimes it is better to educate the end users than to force machine learning to do something inaccurate. In my experience end users prefer to have a correct solution than a solution that works because people have a gut feeling that it should be in a certain way.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-11-02T11:47:41.007" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14893" PostTypeId="2" ParentId="8655" CreationDate="2016-11-02T20:51:09.287" Score="2" Body="&lt;p&gt;I assume that the ones from &lt;a href=&quot;http://joedsm.altervista.org/pythontreekernels.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; are Moschitti's version, as the author is associated with Moschitti in some papers. I'm unable to confirm, though.&lt;/p&gt;&#xA;" OwnerUserId="25792" LastEditorUserId="98307" LastEditDate="2020-08-03T10:26:29.840" LastActivityDate="2020-08-03T10:26:29.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14895" PostTypeId="2" ParentId="14877" CreationDate="2016-11-02T23:16:00.780" Score="2" Body="&lt;p&gt;Credit to @Emre for setting me on this path with n-grams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I discovered the library &lt;code&gt;fuzzywuzzy&lt;/code&gt; which is well-built to solve this exact problem.  I'll give a brief synopsis, but a well-written &lt;a href=&quot;http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;tutorial&quot;&gt;tutorial&lt;/a&gt; and &lt;a href=&quot;https://pypi.python.org/pypi/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;documentation&quot;&gt;documentation&lt;/a&gt; made this pretty straight-forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a DataFrame where names are the index and there is a count column.&lt;br&gt;&#xA;I loop through the index, finding high-similarity strings and then mapping the name to the name (including self) with the highest count.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names = df.index&#xA;&#xA;#iterate through list find the top 10 (arbitrary) matching strings &amp;gt; 90 score, pop off the highest-volume order one&#xA;def find_top_match(name):&#xA;    my_matches = process.extract(name, names, limit=10, scorer=fuzz.token_set_ratio)&#xA;    my_matches = [t[0] for t in my_matches if t[1] &amp;gt;= 90]&#xA;    top_match = df.loc[my_matches].sort_values(by='count', ascending=False).index[0]&#xA;    return(top_match)&#xA;&#xA;top_matches_set = []&#xA;&#xA;for c in df.index:&#xA;    returned_match = find_top_match(c)&#xA;    top_matches_set.append(returned_match)&#xA;&#xA;df['real_name'] = top_matches_set&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25611" LastActivityDate="2016-11-02T23:16:00.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14899" PostTypeId="1" CreationDate="2016-11-03T03:10:24.893" Score="198" ViewCount="363970" Body="&lt;p&gt;I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zyIUI.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zyIUI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CHuCF.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CHuCF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="843" LastEditDate="2018-10-25T07:14:53.910" LastActivityDate="2020-05-12T19:03:48.143" Title="How to draw Deep learning network architecture diagrams?" Tags="|machine-learning|neural-network|deep-learning|svm|software-recommendation|" AnswerCount="6" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14900" PostTypeId="2" ParentId="14899" CreationDate="2016-11-03T03:54:03.780" Score="19" Body="&lt;ul&gt;&#xA;&lt;li&gt;For automated drawing, see &lt;a href=&quot;https://datascience.stackexchange.com/q/12851/843&quot;&gt;How do you visualize neural network architectures?&lt;/a&gt;, &lt;a href=&quot;https://softwarerecs.stackexchange.com/q/28169/903&quot;&gt;https://softwarerecs.stackexchange.com/q/28169/903&lt;/a&gt; and  &lt;a href=&quot;https://softwarerecs.stackexchange.com/q/47841/903&quot;&gt;https://softwarerecs.stackexchange.com/q/47841/903&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;For manual drawing, see &lt;a href=&quot;https://redd.it/574usi&quot; rel=&quot;noreferrer&quot;&gt;https://redd.it/574usi&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-12-29T23:52:12.190" LastActivityDate="2017-12-29T23:52:12.190" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14901" PostTypeId="2" ParentId="14337" CreationDate="2016-11-03T06:05:49.007" Score="1" Body="&lt;p&gt;In any commerce setting, the concept of item similarity is very not straightforward. Two users usually buying same kinds of products can be considered as similar, but we cannot say the same about two items bought by same user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two different concepts of item similarity for recommendation purposes. One is, if the two items are physically similar, for example : &lt;em&gt;Blue Reebok Shoes&lt;/em&gt; and &lt;em&gt;Red Reebok Shoes&lt;/em&gt;, and other is if they have functional dependency on each other, for example: &lt;em&gt;Reebok Shoes&lt;/em&gt; and &lt;em&gt;Reebok Socks&lt;/em&gt;. &#xA;For finding physically similar items, one can create a dictionary of attributes defining the product and do Jaccard similarity on those attributes.&#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Item A = {color: Blue, size: 10, material: Cotton, brand: Reebok}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Item B = {color: Red, size: 10, material: Cotton, brand: Reebok}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the intersection of sets would be number of attributes that match up, i.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Intersection(A,B) = {size, material, brand}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Union(A,B) = {color, size, material, brand}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Jaccard Index = 3/4 = 0.75&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For finding behaviorally dependent items, one proxy that is generally used is more the two items are bought together in same session, more dependent they are for each other's functioning, thus more valuable recommendation. For this setting, one can create a matrix of products a user buys in a single session. For &lt;code&gt;m&lt;/code&gt; users and &lt;code&gt;n&lt;/code&gt; products it would be sparse &lt;code&gt;m X n&lt;/code&gt; matrix. If we read the same matrix column wise, that would be set of users who bought the item in a particular session. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Item A = {Ua, Ub, Uc}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Item B = {Ub, Ud}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Jaccard Index = 1/4 = 0.25&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2016-11-03T06:05:49.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14902" PostTypeId="1" CreationDate="2016-11-03T07:42:26.983" Score="0" ViewCount="141" Body="&lt;p&gt;Lets say we have weights(theta1 and theta2) of neural net as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta1 =[1, 2, 3]&#xA;theta2= [4, 5, 6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we unroll these weights into a single dimension array in matlab/octave ,we get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta = [theta1(:);theta2(:)]&#xA;%theta = [1, 4, 2, 5, 3, 6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we unroll these weights in a slight different way , for example consider this python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta = np.array(theta1,theta2)&#xA;  theta = theta.ravel()&#xA;  #theta = [1,2,3,4,5,6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have implemented a neural network using gradient descent in octave and it worked fine but the same implementation is not working in python(% accuracy = 10%). The only thing different the python implementation is the way the weights are unrolled. &#xA;So, does the performance of neural network really depend on the way the weights are unrolled ?&lt;/p&gt;&#xA;" OwnerUserId="24077" LastActivityDate="2016-11-03T10:09:25.377" Title="Does the performance of neural networks depend on the method used to unroll weights ?" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14903" PostTypeId="2" ParentId="14871" CreationDate="2016-11-03T10:01:22.757" Score="1" Body="&lt;p&gt;tf-idf will learn a vocabulary, idf, and some will also learn stop words (based on min_df, max_df, max_features). Read over &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn's TfidfVectorizer&lt;/a&gt; and you can see the attributes that the fit method will set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you expose a trained tf-idf to new data it will transform that data into a vector of the same size as your original data using the vocabulary to construct Term_Counts which are then converted into your tf vector. The value in this is that you can use another model to predict an outcome based on the tf_idf as each new document will have the same size tf_idf vector as the documents you used to train the model. Otherwise you couldn't use it to make a prediction! For example with a naive bayes classifying tfidf:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tfidf = TfidfVectorizer()&#xA;X = tfidf.fit_transform(X_train)&#xA;nb = MultinomialNB()&#xA;nb.fit(X, y_train)&#xA;&#xA;# When you receive a new document&#xA;X = tfidf.transform(new_doc)&#xA;prediction = nb.predict_proba(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I don't think you would want to refit the model. If you want some kind of continuous real-time update consider implementing a bayesian update&lt;/p&gt;&#xA;" OwnerUserId="25759" LastActivityDate="2016-11-03T10:01:22.757" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14904" PostTypeId="2" ParentId="14902" CreationDate="2016-11-03T10:09:25.377" Score="1" Body="&lt;p&gt;I don't really understand your notation and don't have a lot of experience with unrolling, but from my understanding there can only be one way of transforming your 2 weights (theta1 &amp;amp; theta2) to a single one (theta). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you adapt your python code to something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta = np.array(theta1,theta2)&#xA;theta = np.ravel(theta, order='F')&#xA;#theta = [1, 4, 2, 5, 3, 6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does it work now? If so, you can't just choose how to &quot;unroll&quot; your weights.&lt;/p&gt;&#xA;" OwnerUserId="25104" LastActivityDate="2016-11-03T10:09:25.377" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14905" PostTypeId="1" AcceptedAnswerId="17272" CreationDate="2016-11-03T15:10:25.880" Score="0" ViewCount="307" Body="&lt;p&gt;I am building lost sales estimation model for out of stock days etc. using XGBoost. I am using simple logic of training model on data of normal days with ample inventory (when sales and demand are same) and then using trained model to predict demand on out of stock days. For model building I am splitting the normal days data into train and test datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am getting a peculiar problem of very highly overestimated sales values of out of stock days. prediction for both train and test days are fine but only out of stock days prediction gives problem. Any tips what might be going wrong and how I can debug the problem in gradient tree type of model.&lt;/p&gt;&#xA;" OwnerUserId="12792" LastActivityDate="2017-03-01T09:08:30.757" Title="High overestimation on prediction data" Tags="|machine-learning|r|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14906" PostTypeId="2" ParentId="14874" CreationDate="2016-11-03T15:10:35.257" Score="0" Body="&lt;p&gt;A decision tree is designed to make many branches leading to any number of categorical outcomes.  Logistic regression in it's simplest form, however, takes a continuous variable and decides where to apply a threshold in order to model a binary response.  In your case, a decision tree makes sense because you are working with data that has no overall mathematical model, if I understand you correctly.  Logistic regression is going to struggle with deciding between 100 classes with no underlying pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest reviewing the math behind logistic regression in depth in order to understand the limitations.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2016-11-03T15:10:35.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14907" PostTypeId="2" ParentId="13120" CreationDate="2016-11-03T15:16:42.600" Score="1" Body="&lt;p&gt;I can think of a few possibilities that might fit your usecase:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use the &lt;a href=&quot;http://hunch.net/~jl/projects/hash_reps/&quot; rel=&quot;nofollow noreferrer&quot;&gt;hasshing trick&lt;/a&gt; to turn arbitrary length vectors into constant length.&lt;/li&gt;&#xA;&lt;li&gt;Trade out word vectors for &lt;a href=&quot;http://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paragraph vectors&lt;/a&gt;. &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gensim&lt;/a&gt; and &lt;a href=&quot;https://deeplearning4j.org/doc2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deeplearning4j&lt;/a&gt; both have implementations you can look at.&lt;/li&gt;&#xA;&lt;li&gt;Check out the &lt;a href=&quot;https://github.com/MaxwellRebo/awesome-2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;awesome-2vec&lt;/a&gt; list which links to a &lt;a href=&quot;https://github.com/bdhingra/tweet2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;tweet2vec&lt;/a&gt; implementation!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8085" LastActivityDate="2016-11-03T15:16:42.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14908" PostTypeId="1" AcceptedAnswerId="14927" CreationDate="2016-11-03T16:25:03.730" Score="1" ViewCount="57" Body="&lt;p&gt;I am trying to do some clustering. I have a dataset that is very&#xA;sparse - with the majority of features only occurring in a single vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a list of our features:&#xA;&lt;a href=&quot;https://gist.github.com/scrooloose/5963725dc88e5d15d74dcae522bebf82&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/scrooloose/5963725dc88e5d15d74dcae522bebf82&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for any suggestions/hints/pointers as to how we can merge some of&#xA;these isolated features together. This should hopefully make my clustering&#xA;experiments more successful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, from a manual inspection of the data, I can see this group of&#xA;features that could be all merged into a feature like &quot;health&quot; or perhaps&#xA;&quot;mental health&quot; + &quot;general health&quot; or similar.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;618: Mental Health Research&#xA;619: Mental disorder&#xA;1616: mental health&#xA;1617: mental illness&#xA;1618: men’s health&#xA;410: Genital wart&#xA;402: Genomic Medicine&#xA;476: Hygiene&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another example is this set of features that could be merged into something&#xA;like &quot;education&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;536: Kiir Primary School&#xA;591: Makonzi Boarding School&#xA;609: Mathematics&#xA;670: New York University&#xA;300: Education&#xA;301: Educational psychology&#xA;349: Female education&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any thoughts would be very welcome, thanks :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note: These features are keywords as returned from alchemy&#xA;(&lt;a href=&quot;http://www.alchemyapi.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.alchemyapi.com/&lt;/a&gt;). Resulting from keyword searches for a set of&#xA;URLS. The intention is to cluster the URLs (and hence they companies they&#xA;represent) by these keywords.&lt;/p&gt;&#xA;" OwnerUserId="25817" LastActivityDate="2016-11-05T12:25:37.743" Title="Help reducing a set of features" Tags="|clustering|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14909" PostTypeId="2" ParentId="14251" CreationDate="2016-11-03T16:52:46.353" Score="1" Body="&lt;p&gt;It could be advantageous to set the background level to 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your background is a constant value (a), the best thing to do first is to subtract this value. This way the value won't be transmitted through the network (or whatever you are using). Secondly, it is best to normalize your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And lastly, search the web since &quot;preprocessing data&quot; will probably give you all the information you need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learn how to search the web ;)&lt;/p&gt;&#xA;" OwnerUserId="25104" LastEditorUserId="26596" LastEditDate="2016-12-05T19:56:38.187" LastActivityDate="2016-12-05T19:56:38.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14913" PostTypeId="1" CreationDate="2016-11-03T20:32:25.533" Score="3" ViewCount="2019" Body="&lt;p&gt;I am working on my first ever ds problem on classification. I have got end to end process but not much visualization except bar charts and pie charts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to see how well classes are distinguished in sample dataset itself to find anomaly in sample data or to see how many outliers sample have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see many nice scatterplot clustering done in many articles, pappers and books which are mostly based on numerical features. Something like following taken from scikit learn : &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/datasets/plot_random_dataset.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/datasets/plot_random_dataset.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SBWN1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SBWN1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, most of those are for numeric features. How can I draw similar plot for textual features? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I can do it is converting all features into vectors; plot all the indexes(token indexes) on x-axis and their respective metric (TF or TF-IDF) on y-axis? But then there are so many features!?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how can I analyze raw sample data in such clustering format without transforming them into vectors. I can do tokenization and some basic normalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example  my sample data is &#xA;text, category&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;chiense restarant near me&quot;, chinesefoodlover&#xA;&quot;japanese food near me&quot;, japanesefoodlover&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can break it down to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[chiense, restarant, near, me], chinesefoodlover&#xA;[japanese, food, near, me], japanesefoodlover&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But then how to draw clusters of classes from there.&lt;/p&gt;&#xA;" OwnerUserId="19147" LastEditorUserId="19147" LastEditDate="2016-11-03T20:44:16.973" LastActivityDate="2017-12-29T22:01:10.423" Title="How to do scatterplot Visualization of Text Classification" Tags="|classification|text-mining|visualization|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14914" PostTypeId="1" AcceptedAnswerId="65491" CreationDate="2016-11-04T00:58:10.750" Score="1" ViewCount="898" Body="&lt;p&gt;We have 4000 features and we are applying Principal Component Analysis to reduce them a small number of features from 20 to 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are performing linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both training and validation errors are worse than without any use of PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However we notice that the training error is not lower and actually has become slightly larger than the validation error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something expected?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variance ratio of the first principal components is high enough in comparison to the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we are noticing is that if you use all of the 4000 principal components then the gap between training error and validation is larger and as you repeat the process and you reduce the principal components to 1000, 200, 100, 10, etc. then the training error comes closer to the validation error (even though both are being increased as expected)&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="24885" LastEditDate="2016-11-04T12:32:35.867" LastActivityDate="2019-12-26T22:11:52.073" Title="Why training error is larger than validation error after PCA?" Tags="|linear-regression|pca|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14916" PostTypeId="2" ParentId="14914" CreationDate="2016-11-04T07:03:18.200" Score="2" Body="&lt;p&gt;The basic principle that helps us implement PCA is it's ability to explain the variation, so it depends how much PC you are taking and how much variation they are explaining. They are definitely not explaining 100% of variation so there are good chance that Validation Error is higher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before applying PCA you need to understand why we apply it? We apply it to reduce computational effort and practical implementation of problem. The idea of PCA is not to give better result than actual variables in problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Better results are generally available with complete list of variables.&lt;/p&gt;&#xA;" OwnerUserId="20517" LastActivityDate="2016-11-04T07:03:18.200" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14917" PostTypeId="2" ParentId="14648" CreationDate="2016-11-04T07:06:34.290" Score="9" Body="&lt;p&gt;Using &lt;code&gt;lit&lt;/code&gt; would convert all values of the column to the given value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. &lt;code&gt;when&lt;/code&gt; can help you achieve this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql.functions import when   &#xA;&#xA;df.withColumn('c1', when(df.c1.isNotNull(), 1))&#xA;  .withColumn('c2', when(df.c2.isNotNull(), 1))&#xA;  .withColumn('c3', when(df.c3.isNotNull(), 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would result in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{c|lcr}&#xA;&amp;amp; \text{c1} &amp;amp; \text{c2} &amp;amp; \text{c3} \\&#xA;\hline&#xA;1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 \\&#xA;2 &amp;amp; 1 &amp;amp; null &amp;amp; 1 \\&#xA;3 &amp;amp; null &amp;amp; 1 &amp;amp; null&#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you want to replace those null values with some other value too, you can use &lt;code&gt;otherwise&lt;/code&gt; in combination with &lt;code&gt;when&lt;/code&gt;. Let's say you want to impute &lt;code&gt;0&lt;/code&gt; there:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql.functions import when   &#xA;&#xA;df.withColumn('c1', when(df.c1.isNotNull(), 1).otherwise(0))&#xA;  .withColumn('c2', when(df.c2.isNotNull(), 1).otherwise(0))&#xA;  .withColumn('c3', when(df.c3.isNotNull(), 1).otherwise(0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would result in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{c|lcr}&#xA;&amp;amp; \text{c1} &amp;amp; \text{c2} &amp;amp; \text{c3} \\&#xA;\hline&#xA;1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 \\&#xA;2 &amp;amp; 1 &amp;amp; 0 &amp;amp; 1 \\&#xA;3 &amp;amp; 0 &amp;amp; 1 &amp;amp; 0&#xA;\end{array}&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2016-11-04T07:06:34.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14920" PostTypeId="2" ParentId="14913" CreationDate="2016-11-04T13:24:52.577" Score="2" Body="&lt;p&gt;You should look at &lt;strong&gt;t-SNE&lt;/strong&gt; for visualising high dimensional datasets such as words : &lt;a href=&quot;http://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://lvdmaaten.github.io/tsne/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-11-04T13:24:52.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14921" PostTypeId="2" ParentId="14883" CreationDate="2016-11-04T17:07:28.140" Score="0" Body="&lt;p&gt;In order to perform such tasks with high accuracy, I suggest you to build a LSTM model with word embeddings with the help of word2vec. LSTMs can help to retrieve information from sentence as well as predict the next character or word given a set of words is already present in the sentence. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-04T17:07:28.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14923" PostTypeId="2" ParentId="14874" CreationDate="2016-11-04T20:39:22.380" Score="1" Body="&lt;p&gt;A few assumptions I'm making:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I believe you're mapping your input set of integers to another integer, which means your output is discrete, so I think you're using [multinomial logistic regression] (&lt;a href=&quot;https://en.wikipedia.org/wiki/Multinomial_logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Multinomial_logistic_regression&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You're treating your discrete inputs (integers) as continuous--i.e., you're not using a one-hot-encoding for each integer. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your underlying function is something to the like of if $\forall j=2,..,10$ if $x \in (x_{j-1}, x_j)$ then $y = c_j$, correct?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the above three are true, then it's clear why Logistic won't work and why the Decision Tree will. The Logistic will approximate the relationship between your input and output with a $continuous$ linear relationship, which is not exactly what you have here because you have a discontinuity. It's worth mentioning that Decision Trees will perform better at these points, which means the more discontinuities you have the worse your performance will be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But after a quick simulation, the magnitudes you quoted are too large to be coming from this. So I'm guessing you're using $binary$ logistic regression, which is just the wrong choice of model (since it's mapping everything to a single class instead of many classes). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So, it sounds like you're just using the wrong model.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a quick demo in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Loading packages&#xA;require(nnet)&#xA;require(rpart)&#xA;set.seed(024)&#xA;buildData &amp;lt;- function(n){&#xA;  x &amp;lt;- 1:n&#xA;  x_j &amp;lt;- data.frame(vals=quantile(1:n, probs=1:10/10))$vals&#xA;  c_j &amp;lt;- 1:10&#xA;  y &amp;lt;- rep(0, n)&#xA;  for(i in 1:length(c_j)){&#xA;    if(i==1){&#xA;      y &amp;lt;- ifelse(x &amp;lt;= x_j[i], c_j[i], y)&#xA;    } else {&#xA;      y &amp;lt;- ifelse( x &amp;gt;= x_j[i-1] &amp;amp; x &amp;lt;= x_j[i], c_j[i], y)    &#xA;    }&#xA;  }&#xA;  print(table(y))&#xA;  xdf &amp;lt;- data.frame(x, y=factor(y))&#xA;  return( xdf)&#xA;}&#xA;# Building the data&#xA;n &amp;lt;- 1e4&#xA;xdf &amp;lt;- buildData(n)&#xA;# Random 90-10 split&#xA;trnflt &amp;lt;- runif(n) &amp;lt;= 0.5&#xA;&#xA;# Multinomial Logistic Model&#xA;multinommod &amp;lt;- multinom(y~x, data=xdf)&#xA;xdf$preds_mult &amp;lt;- predict(multinommod, xdf, type='class')&#xA;&#xA;# Binary Logistic Model&#xA;logisticmod &amp;lt;- glm(y~x, data=xdf, family='binomial')&#xA;xdf$preds_log &amp;lt;- ifelse(predict(logisticmod, xdf, type='response') &amp;lt;= 0.5, 0, 1)&#xA;&#xA;# Decision Tree Model&#xA;treemod &amp;lt;- rpart(y~x, data=xdf[trnflt,], method='class')&#xA;xdf$preds_tree &amp;lt;- predict(treemod, xdf, type='class')&#xA;&#xA;# Multinomial Logistic confusion Matrix&#xA;print(with(xdf[trnflt,], table(y, preds_mult )))&#xA;print(with(xdf[trnflt==F,], table(y, preds_mult )))&#xA;&#xA;# Binary Logistic confusion Matrix&#xA;print(with(xdf[trnflt,], table(y, preds_log )))&#xA;print(with(xdf[trnflt==F,], table(y, preds_log )))&#xA;&#xA;# Decision Tree confusion Matrix&#xA;print(with(xdf[trnflt,], table(y, preds_tree)))&#xA;print(with(xdf[trnflt==F,], table(y, preds_tree)))&#xA;&#xA;# Here's the performance of all 3&#xA;print(c(paste('Multinomial Logistic Regression Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_mult )))) / sum(trnflt==F)),&#xA;        paste('Logistic Regression Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_log )))) / sum(trnflt==F)), &#xA;        paste('Decision Tree Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_tree )))) / sum(trnflt==F))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2016-11-06T16:07:34.500" LastActivityDate="2016-11-06T16:07:34.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14924" PostTypeId="1" CreationDate="2016-11-04T22:00:57.200" Score="2" ViewCount="118" Body="&lt;p&gt;Let's consider one has built a fully-supervised neural network for some task, e.g. localizing an object in various scenes. As you can imagine, it is quite time-consuming to label data: one has to manually localize the object in an image and draw a bounding-box around it - one at a time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say we have a normal convolutional neural network (CNN) for the fully-supervised localization, thus something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          2D input image&#xA;                |&#xA;       convolutional-layer 1&#xA;                |&#xA;               ...&#xA;                |&#xA;       convolutional-layer N&#xA;                |&#xA;      ---flattened output---&#xA;                |                    &#xA;    fully-connected classifier&#xA;                |&#xA;           output layer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can we adapt this architecture to utilize weakly-labelled data, too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No matter if we have a sliding-window approach, something like OverFeat or probability distributions for actually localizing objects: we always need fully-labelled training data. This is a problem, because labelling data fully-supervised is really time consuming. Hence, fully-labelled data sets are pretty rare.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast to this, typically large amounts of weakly-labelled or unlabeled data exists. In my opinion there is a large potential in weakly-labelled / unlabelled data, due to its vast availability. The problem is using this potential without having to manually label every sample by hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, my question is: how can weakly-labelled data sets (i.e. &quot;object&quot; vs. &quot;no object&quot;) or completely unlabelled data sets be utilized to improve the robustness of fully-supervised architectures, like the mentioned CNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it generally possible to mix a supervised approach with some unsupervised approach? Like enabling a fully-supervised architecture to somehow utilize weakly-labelled data for training?&lt;/p&gt;&#xA;" OwnerUserId="13809" LastEditorUserId="29169" LastEditDate="2021-02-16T15:34:17.307" LastActivityDate="2022-02-07T01:01:30.140" Title="Neural Network: how to utilize weakly-/unsupervised data to improve supervised network?" Tags="|deep-learning|neural-network|convolutional-neural-network|unsupervised-learning|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14926" PostTypeId="1" CreationDate="2016-11-05T10:06:40.810" Score="1" ViewCount="94" Body="&lt;p&gt;Is there a machine learning framework that supports &lt;a href=&quot;https://en.wikipedia.org/wiki/Partial_evaluation&quot; rel=&quot;nofollow noreferrer&quot;&gt;partial evaluation&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We train on &lt;code&gt;[model, year, km, ..., colour, price]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Today we call&lt;br&gt;&#xA;&lt;code&gt;get_prediction(model, year, km, ..., colour)&lt;/code&gt;&lt;br&gt;&#xA;It returns the value of &lt;code&gt;price&lt;/code&gt;, ie a number (or a probability for a label).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I would like to call&lt;br&gt;&#xA;&lt;code&gt;get_prediction_function(model, year, ..., colour)&lt;/code&gt;&lt;br&gt;&#xA;It should return a function &lt;code&gt;price(km)&lt;/code&gt;.  (That function returns a number.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially we are pulling out a dimension or two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory of course it gives us nothing, it just postpones some calculation.  (In practice this small specialised object is very useful in some production applications.)&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2016-11-05T10:06:40.810" Title="Is there a machine learning framework that supports partial evaluation ie can return a function?" Tags="|machine-learning|optimization|performance|tools|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="14927" PostTypeId="2" ParentId="14908" CreationDate="2016-11-05T12:25:37.743" Score="1" Body="&lt;p&gt;If I understand correctly you want to cluster urls by using the keywords extracted as features. As these features are really sparse, you can try to use dimensionality reductions methods to help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way is to treat each URL keywords as a document. Then you can use document embeddings algorithms such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA&lt;/a&gt; or &lt;a href=&quot;https://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt; that learn denser representations of your documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to group keywords, you can try to use word embeddings methods that learn representations of words. Using this you can then measure the similarity between words and groups of words.&#xA;An example is the well-known &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt;. Recent methods like &lt;a href=&quot;https://github.com/facebookresearch/fastText&quot; rel=&quot;nofollow noreferrer&quot;&gt;FastText&lt;/a&gt; can be alternative that take the morphology into account.&lt;/p&gt;&#xA;" OwnerUserId="12558" LastActivityDate="2016-11-05T12:25:37.743" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14928" PostTypeId="1" AcceptedAnswerId="14930" CreationDate="2016-11-05T13:48:56.980" Score="-1" ViewCount="9150" Body="&lt;p&gt;I understand calculus and maths but when i apply statistics and add up numbers they both look kinda same&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody explain the difference in a little detail and simple manner please&lt;/p&gt;&#xA;" OwnerUserId="25876" LastActivityDate="2016-11-05T14:33:43.640" Title="Difference between summation and integration" Tags="|statistics|descriptive-statistics|" AnswerCount="1" CommentCount="2" ClosedDate="2016-11-05T14:56:28.050" ContentLicense="CC BY-SA 3.0" />
  <row Id="14930" PostTypeId="2" ParentId="14928" CreationDate="2016-11-05T14:33:43.640" Score="1" Body="&lt;p&gt;In most simple words-&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Summation&lt;/strong&gt;- Sum of a &lt;em&gt;small numbers&lt;/em&gt; of &lt;em&gt;large quantities&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Integration&lt;/strong&gt;- Sum of a &lt;em&gt;large numbers&lt;/em&gt; of &lt;em&gt;small quantities&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Other Simple Difference can be-&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;Summation&lt;/code&gt; is a &lt;code&gt;discrete sum&lt;/code&gt; whereas &lt;code&gt;Integration&lt;/code&gt; is a &lt;code&gt;continuous sum&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/X0qo3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0qo3.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Here dx is an infinitesimal so that the integral summation is&#xA;continuous.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Hope it helps, cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-05T14:33:43.640" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="14932" PostTypeId="1" CreationDate="2016-11-05T17:04:07.897" Score="1" ViewCount="117" Body="&lt;p&gt;I am new to machine learning and have quite good understanding of basic concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was implementing a 3 Layer neural network on MNIST dataset with 784, 100, 10 neurons in input, hidden, output layer respectively. I did not use any regularization here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly I trained the network on Intel i5 4th generation quad-core CPU with 4GB of ram which gave me 64% of accuracy.&#xA;Then I trained the exact same network, with exact same code on Intel i5 7th generation quad core CPU with 8gb of ram which gave accuracy of about 89%&#xA;&lt;a href=&quot;https://github.com/1byxero/MNIST/blob/master/Without%20Regularization/test.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;This is the link to the implementation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is in Torch, does the compute power effect the accuracy of the network? or is it something else that I am missing which has resulted in this huge change. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not use any different weight initialization method than the default provided in the torch libraries, so that is ruled out.&#xA;I also did not use anything else which may effect the network to change its accuracy to this extent.&lt;/p&gt;&#xA;" OwnerUserId="25881" LastActivityDate="2016-12-06T13:24:18.510" Title="Neural network accuracy in Torch depends on compute power?" Tags="|neural-network|accuracy|jupyter|torch|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14933" PostTypeId="2" ParentId="14883" CreationDate="2016-11-05T17:45:28.193" Score="7" Body="&lt;p&gt;You can possibly use a combination of Named Entity Recognition and Syntactical Analysis - &#xA;while the word &lt;strong&gt;Edwin&lt;/strong&gt; is certainly propping up, imagine a situation where the name is &lt;strong&gt;Edward Philip Martel&lt;/strong&gt;. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. &lt;em&gt;John &amp;amp; Ramsey dined at Winterfell&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP  can help upto a certain extent only). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of leveraging on syntax analysis / parsing and NER is in the following examples -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 1. User: Edwin is my name.&#xA; 2. User: I am Edwin.&#xA; 3. User: My name is Edwin.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem.&#xA;This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.&lt;/p&gt;&#xA;" OwnerUserId="25742" LastActivityDate="2016-11-05T17:45:28.193" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14934" PostTypeId="1" AcceptedAnswerId="15002" CreationDate="2016-11-05T19:19:13.410" Score="1" ViewCount="105" Body="&lt;p&gt;I've been reading through NEAT (NeuroEvolution of Augmenting Topologies)'s whitepaper, and one thing is particularly confusing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Innovation numbers are supposedly used to track the origin of a mutation, so that crossing over is easily done. From what I understand, two identical connections should always have the same innovation number, because their numbers are indexed. However, according to this illustration:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cdn5p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cdn5p.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when the two genomes being crossed over share connections with the same innovation number, the resulting innovation is taken from the more fit parent (or random if they are equal fitness).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the two connections are identical (they have the same innovation number) then why does it matter which parent it comes from, since the connection is the same either way? Or is there something about them that is not the same and I am understanding this wrong?&lt;/p&gt;&#xA;" OwnerUserId="25884" LastActivityDate="2016-11-09T19:36:40.077" Title="Why does the parent connection matter during a NEAT crossover when both connections are identical?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14938" PostTypeId="2" ParentId="14932" CreationDate="2016-11-05T21:36:10.783" Score="1" Body="&lt;p&gt;Available compute power does not directly affect the accuracy of a neural network. If your different runs of the network have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;identical architecture and meta-params &lt;/li&gt;&#xA;&lt;li&gt;identical code (including library code)&lt;/li&gt;&#xA;&lt;li&gt;all training data is identical &lt;/li&gt;&#xA;&lt;li&gt;all stochastic parts of training use the same random seed and generator&lt;/li&gt;&#xA;&lt;li&gt;all data types are identical precision (e.g. all vectors and matrices are 32-bit or 64-bit floats)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;then the behaviour of neural network training in each run is fully deterministic and repeatable. Having a faster processor will just get you to the result faster*.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most likely difference between your tests is due to not seeding the random number generators used in the training the process. For you this includes weight initialisation, possibly train/test split and possibly shuffling training data in each epoch. As you did not use any regularisation, then accuracy of the trained network can vary quite a bit due to over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To verify this, you can train a second or third time on each CPU. I expect you will see a lot of variation in final accuracy, regardless of which machine you run it on.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* This does mean that having a faster machine can result in you having a more accurate final network in practice when you are tuning the parameters, because you can try more variations of meta-params with multiple training sessions.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-06T12:45:22.327" LastActivityDate="2016-11-06T12:45:22.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14940" PostTypeId="1" AcceptedAnswerId="14955" CreationDate="2016-11-05T23:42:26.043" Score="1" ViewCount="742" Body="&lt;p&gt;Having this R statement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;itemsets = apriori(data, parameter=list(support=0.05, confidence=0.5))&#xA;plot(itemsets, method=&quot;graph&quot;, &#xA;     control=list(type=&quot;items&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I change the colour of my plot? I try to insert &quot;col&quot; but don't allows me to do that because I'm getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in i.parse.plot.params(graph, list(...)) : &#xA;  Unknown plot parameters: col&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="24719" LastEditorUserId="24719" LastEditDate="2016-11-06T09:28:17.223" LastActivityDate="2016-11-06T12:07:55.150" Title="Network Plot - R - Modify Col" Tags="|r|visualization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14941" PostTypeId="1" AcceptedAnswerId="14943" CreationDate="2016-11-06T00:34:34.637" Score="9" ViewCount="2880" Body="&lt;p&gt;My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested to me that GPUs are only needed for the training phase. Once trained, it's actually more efficient to run them on CPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this true?&lt;/p&gt;&#xA;" OwnerUserId="25891" LastEditorUserId="29169" LastEditDate="2019-06-09T15:44:21.933" LastActivityDate="2019-06-09T15:44:21.933" Title="After the training phase, is it better to run neural networks on a GPU or CPU?" Tags="|neural-network|deep-learning|gpu|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14942" PostTypeId="1" AcceptedAnswerId="14951" CreationDate="2016-11-06T01:20:12.483" Score="1" ViewCount="466" Body="&lt;p&gt;I would like to know the use of correaltion map in machine leraning. For example, if there are 2 features with high correaltion, should either of the features be removed before appying the algorithm or it depends on every data set. Any explanation would be highly helpful. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="25892" LastActivityDate="2016-11-06T10:28:19.430" Title="Use of Correlation Map in Machine Learning" Tags="|machine-learning|data-mining|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14943" PostTypeId="2" ParentId="14941" CreationDate="2016-11-06T03:02:44.330" Score="9" Body="&lt;p&gt;This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, here is a &lt;a href=&quot;https://github.com/jcjohnson/cnn-benchmarks&quot; rel=&quot;noreferrer&quot;&gt;benchmark comparing CPU with GPU on different CNN-based architectures&lt;/a&gt;. The forward pass is much slower on a CPU in that case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aLNlM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aLNlM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYI: &lt;a href=&quot;https://datascience.stackexchange.com/q/14368/843&quot;&gt;Benchmarks based on neural networks libraries to compare the performance between different GPUs&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-11-06T05:08:45.030" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14944" PostTypeId="1" AcceptedAnswerId="96663" CreationDate="2016-11-06T07:19:07.687" Score="3" ViewCount="125" Body="&lt;p&gt;Suppose I have 2 datawarehouses of two different companies that are being merged. Some of the tables are common but includes different records, while others are different. If I use big data environments, can I put all the tables together as data swamp and to create new data lake with archive of the previous datawarehouses data or should I build a new datawarehouse based on the common tables and adding the different ones? In addition, is there a good source for best practice steps for merging datawarehouses?&lt;/p&gt;&#xA;" OwnerUserId="16251" LastEditorUserId="471" LastEditDate="2016-11-06T12:30:42.787" LastActivityDate="2021-06-15T18:45:34.523" Title="Merging 2 datawarehouses" Tags="|dataset|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14945" PostTypeId="2" ParentId="13923" CreationDate="2016-11-06T07:58:55.727" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;According to distributional hypothesis, individual dimension in the vector of the word does not signify much about the word in real world. You need to worry about the individual dimensions. If your question is so how should I select the number of dimensions, it is purely based on experiment for your data and it can go from 100 to 1000. For many experiments where the training is done on wiki text, the 300 dimension mostly give the best result.&lt;/li&gt;&#xA;&lt;li&gt;Sample param is the parameter used to prune the words having high frequency. E.g., &amp;quot;the&amp;quot;, &amp;quot;is&amp;quot;, &amp;quot;was&amp;quot;, these stop words are not considered in the window while predicting the inside word and the default value works well to identify these stop words whose frequency is higher.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="25174" LastEditorUserId="85045" LastEditDate="2020-12-25T15:14:06.083" LastActivityDate="2020-12-25T15:14:06.083" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="14948" PostTypeId="1" AcceptedAnswerId="14970" CreationDate="2016-11-06T09:34:43.783" Score="1" ViewCount="74" Body="&lt;p&gt;What is the sizing limitation for Oracle 11g? Can I use it as big data platform? From the following &lt;a href=&quot;https://docs.oracle.com/cd/B28359_01/server.111/b28320/limits003.htm#i288032&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; I understand that there is no limitation for the records, only the columns are bounded to 1000. So if my big data has less than 1000 per table but contains many records can I use Oracle 11g as big data platform?&lt;/p&gt;&#xA;" OwnerUserId="16251" LastEditorUserId="471" LastEditDate="2016-11-06T12:30:12.427" LastActivityDate="2016-11-06T22:37:38.910" Title="Is Oracle 11g is able to ingest big data?" Tags="|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14949" PostTypeId="1" AcceptedAnswerId="14952" CreationDate="2016-11-06T09:46:17.067" Score="0" ViewCount="1362" Body="&lt;p&gt;Can anyone please tell me whats the difference between&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Business analyst&#xA;Financial analyst&#xA;Data analyst&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what should one need to learn in order to achieve either of these profile.&lt;/p&gt;&#xA;" OwnerUserId="25876" LastActivityDate="2016-11-06T10:41:36.683" Title="Difference between business and financial and data analyst" Tags="|data-mining|finance|market-basket-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14950" PostTypeId="2" ParentId="14940" CreationDate="2016-11-06T09:51:17.800" Score="0" Body="&lt;p&gt;Okay, try this--&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;itemsets = apriori(data, parameter=list(support=0.05, confidence=0.5))&#xA;plot(itemsets, method=&quot;graph&quot;, &#xA;     control=list(type=&quot;items&quot;, col=&quot;red&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastActivityDate="2016-11-06T09:51:17.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14951" PostTypeId="2" ParentId="14942" CreationDate="2016-11-06T10:28:19.430" Score="1" Body="&lt;p&gt;It depends. A high correlation between two features suggests that they represents almost the same information. For some problems like clustering, it is always useful to remove redundant features while some algorithm like Gradient Boosting in xgboost is not affected at all by such features. So, it all depends on what you want to do with your data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per my opinion, if your dataset has too many features, then I would suggest to check the correlation between those features and apply PCA to reduce the dimensionality of your dataset especially if you are doing tasks like clustering or regression.   &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-06T10:28:19.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14952" PostTypeId="2" ParentId="14949" CreationDate="2016-11-06T10:41:36.683" Score="1" Body="&lt;h2&gt;Business Analyst&lt;/h2&gt;&#xA;&lt;p&gt;A business analyst is one who understands the specific domain of the project &lt;em&gt;(ex. retail, merchandising to be specific, supply chain etc.)&lt;/em&gt;. His/her role is to understand the business problem, analyze the current state and capture requirements using various tools like &lt;code&gt;surveys, interviews, group discussions&lt;/code&gt; and then provide recommendations and create a Requirements document for sign off.&lt;/p&gt;&#xA;&lt;h3&gt;Business Analyst Skillset-&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Communication,  Analytical thinking, Domain Knowledge, Generic&#xA;technical Knowledge, Problem solving skills,Decision solving skills,&#xA;Managerial Skills,Negotiation and Persuasion Skills.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Financial Analyst&lt;/h2&gt;&#xA;&lt;p&gt;Financial analysts use financial data to spot trends and extrapolate into the future, helping their employers and clients make the best investing decisions. Businesses rely on financial analysts to determine when it is an auspicious time to buy or sell specific securities and, in some cases, companies use reports put together by financial analysts to determine if the entire business should be sold.&lt;/p&gt;&#xA;&lt;h3&gt;Financial Analyst Skillset-&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Financial analysts should be great problem-solvers, excel at the use&#xA;of logic and possess strong skills in quantitative analysis. In&#xA;addition, successful financial analysts have an in-depth understanding&#xA;of various financial markets and investment products.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Data Analyst&lt;/h2&gt;&#xA;&lt;p&gt;A data analyst's role is one that works with lots of data to derive meaningful insights to either address business problems or discover hidden trends and patterns that can be leveraged to meet the business objectives.&lt;/p&gt;&#xA;&lt;h3&gt;Data Analyst Skillset-&lt;/h3&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Data analyst also needs similar skills with some additional skills&#xA;like to analyze data like SQL, DATA mining, OLAP, Reports etc.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Strong people skills, leadership ability, and teamwork are beneficial for each type career.&lt;/p&gt;&#xA;&lt;p&gt;For more info, you can visit these resources-&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.investopedia.com/articles/professionals/120315/career-advice-financial-analyst-vsdata-analyst.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;Career Advice: Financial Analyst Vs. Data Analyst&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-data-analyst-and-business-analyst&quot; rel=&quot;nofollow noreferrer&quot;&gt;What is the difference between data analyst and business analyst?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://work.chron.com/differences-between-business-analyst-data-analyst-17090.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Differences Between a Business Analyst &amp;amp; a Data Analyst&#xA;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope it helps, cheers! :)&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-06T10:41:36.683" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14953" PostTypeId="1" AcceptedAnswerId="15054" CreationDate="2016-11-06T11:14:06.773" Score="2" ViewCount="488" Body="&lt;p&gt;I read chapter 5.1.3 of Joshua Bengio's &lt;a href=&quot;http://www.deeplearningbook.org/contents/ml.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;deeplearning book&lt;/a&gt;, which says:&#xA;supervised learning involve observing examples of random vectors $\textbf{x}$ and associated value or vector $\textbf{y}$ and learning to predict $\textbf{y}$ from $\textbf{x}$ by estimating $p(\textbf{y} \vert \textbf{x})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does $p(\textbf{y} \vert \textbf{x})$ mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From basic statistics, I know that $p(\textbf{y} \vert \textbf{x})=\frac{p(\textbf{y},\textbf{x})}{p(\textbf{x})}$.&#xA;How do we find $p(\textbf{y},\textbf{x})$ and  $p(\textbf{x})$?  &lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="1177" LastEditDate="2017-02-28T09:52:54.607" LastActivityDate="2017-02-28T09:52:54.607" Title="In supervised learning, what does &quot;Estimating $p(y \vert x)$&quot; mean?" Tags="|machine-learning|supervised-learning|probability|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14954" PostTypeId="2" ParentId="5703" CreationDate="2016-11-06T11:31:54.937" Score="2" Body="&lt;p&gt;There is also the academic &lt;a href=&quot;http://en.wikipedia.org/wiki/DBpedia&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBPedia project&lt;/a&gt;, an RDF version of a subset of Wikipedia. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the Alexa search engine, which ranks web sites according to their popularity on the internet, it is among top 100.000 of websites - not bad, not completely irrelevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;General interest for the &lt;a href=&quot;http://www.alexa.com/siteinfo/dbpedia.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBpedia.org website&lt;/a&gt;  in the period 2015-2016 seems to have been fairly constant. See screen grab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AWG7x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AWG7x.png&quot; alt=&quot;dbpedia timeseries 2015&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other &lt;a href=&quot;http://www.alexa.com/find-similar-sites#site=dbpedia.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;similar sites&lt;/a&gt; (according to Alexa) are much less popular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'd say Semantic Web Acceptance is not dead, it just smells funny.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2016-11-06T11:31:54.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14955" PostTypeId="2" ParentId="14940" CreationDate="2016-11-06T12:07:55.150" Score="1" Body="&lt;p&gt;Use &lt;code&gt;edgeCol&lt;/code&gt; and &lt;code&gt;nodeCol&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;arulesViz&quot;)&#xA;data(Groceries)&#xA;rules &amp;lt;- apriori(Groceries, parameter=list(support=0.005, confidence=0.5))&#xA;plot(rules, method=&quot;graph&quot;,&#xA;     control=list(nodeCol=&quot;yellow&quot;, edgeCol=&quot;blue&quot;, type=&quot;items&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZHwJo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZHwJo.png&quot; alt=&quot;plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2961" LastActivityDate="2016-11-06T12:07:55.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14957" PostTypeId="1" AcceptedAnswerId="14994" CreationDate="2016-11-06T15:18:32.883" Score="4" ViewCount="6689" Body="&lt;p&gt;I am working on a ranking question, recommending k out of m items to the users. The evaluation metric is &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_retrieval#Precision_at_K&quot; rel=&quot;nofollow noreferrer&quot;&gt;average precision at K&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both R and Python have &lt;a href=&quot;https://en.wikipedia.org/wiki/Xgboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost&lt;/a&gt; can be used for &lt;a href=&quot;https://github.com/dmlc/xgboost/tree/master/demo/rank&quot; rel=&quot;nofollow noreferrer&quot;&gt;pairwise comparison&lt;/a&gt; and can be adapted for ranking problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there other algorithms or approaches that can be applied to ranking problems? &lt;/p&gt;&#xA;" OwnerUserId="25867" LastEditorUserId="1330" LastEditDate="2017-11-30T16:02:21.050" LastActivityDate="2017-11-30T16:02:21.050" Title="Machine learning algorithm for ranking" Tags="|machine-learning|recommender-system|xgboost|ranking|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14958" PostTypeId="1" CreationDate="2016-11-06T16:53:50.050" Score="0" ViewCount="66" Body="&lt;p&gt;With a modern IMU with 9 angles of freedom collecting accelerometer, magnetometer and gyroscope data on 3 axis, what would be the best approach on filtering the data and handling it to accurately estimate the forward velocity of the swimmer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach was to:&#xA;1. Use a 3-point moving average to get rid of any vibrations caused by unneeded movements&#xA;2. Use a median average to get rid of repetitve movements such as shakes or water  resistance&#xA;3. Perform integration of accelerometer data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, I'm unsure of my data filtering approach. Is there a better approach? What methods should I use for acc. integration?&lt;/p&gt;&#xA;" OwnerUserId="18692" LastActivityDate="2017-02-05T21:19:20.377" Title="Estimating forward velocity for a swimmer" Tags="|predictive-modeling|data-cleaning|data|estimators|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14959" PostTypeId="1" CreationDate="2016-11-06T01:19:29.090" Score="3" ViewCount="502" Body="&lt;p&gt;My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested GPUs are only needed for the training phase and that once trained, it's actually more efficient to run them on CPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this true?&lt;/p&gt;&#xA;" OwnerUserId="25891" OwnerDisplayName="Crashalot" LastActivityDate="2016-11-06T18:30:17.030" Title="After the training phase, is it better to run neural networks on a GPU or CPU?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2016-11-06T19:26:15.520" ContentLicense="CC BY-SA 3.0" />
  <row Id="14960" PostTypeId="2" ParentId="14959" CreationDate="2016-11-06T01:41:55.270" Score="2" Body="&lt;p&gt;That would depend entirely on what software platform you're running on, not on any particular characteristic of neural nets or their constituent objects. For example, Theano leverages GPUs because they're optimized for matrix math, which neural net weights and activations are often expressed as. On the other hand, they can also be expressed as sets, which makes them ideal for set-based languages like SQL; I prefer this approach, since sets are more flexible (you can, for example, operate easily on ragged dimensional sets, unlike with matrices) and SQL is a somewhat portable standard that makes it easier to think about neural net layers conceptually. As my screen name suggests, I use SQL Server, which is one of many database server platforms that can do some serious number-crunching on very large neural nets, all without the use of any GPU power. Long ago I also programmed neural nets in VB.Net, VB 6 and C# that efficiently made use of arrays, without leveraging GPUs at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the choice may come down to a combination of your existing skill set, what architectures you have access to, your portability requirements, code maintenance issues and all that, not to any explicit linkage between specific hardware components and the inner workings of neural nets. There may be slight advantages to modeling ragged matrices in SQL or to the matrix-crunching power of GPUs, but I don't think there's much practical difference in most scenarios, nor can I think of a way of testing such a comparison without getting into an apples vs. oranges conundrum. That being said, &lt;strong&gt;within a specific platform&lt;/strong&gt;, it may be true that &quot;GPUs are only needed for the training phase&quot; and &quot;once trained, it's actually more efficient to run them on CPUs.&quot; That I can't comment on, since I still with SQL and the .Net stack and avoid Theano etc. altogether (though I respect their capabilities). I hope that helps. &lt;/p&gt;&#xA;" OwnerUserId="18544" OwnerDisplayName="SQLServerSteve" LastActivityDate="2016-11-06T01:41:55.270" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14961" PostTypeId="1" CreationDate="2016-11-06T19:09:09.900" Score="1" ViewCount="31" Body="&lt;p&gt;For my program on BCH decoding I had to polynomially divide the corrected BCH code with the generator matrix for the final decoded input. I did the same as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for j=1:num_blocks,&#xA;        op4(:,j)=mod(deconv(op3(:,j)',gen),2)';&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where op3 is my corrected BCH code matrix (each block is a column) and gen is my generator polynomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tends to be slow though as my input bit stream is pretty large (~10^6).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read that deconvolution can also be performed through the ifft of the element wise division of the ffts of op3 (its columns) and gen. However for a 15 point fft of both op3 and gen (gen is padded), the deconvolution is also 15 points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need is for a (15,11) code to recover my block size 11 input on deconvolving the 15 point BCH code with the 5 point generator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="21749" LastActivityDate="2016-11-06T19:09:09.900" Title="Applying deconv for a large dataset" Tags="|matlab|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14962" PostTypeId="2" ParentId="14953" CreationDate="2016-11-06T19:21:26.510" Score="1" Body="&lt;p&gt;$P(y|x)$ stands for the &lt;em&gt;probability of y occurring, given x has occurred&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using bayes theorem, this can also be written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(y|x) = \frac {{P(x|y) \times P(y)}}{ P(x)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is these values that you generally know and can use.&lt;/p&gt;&#xA;" OwnerUserId="21749" LastEditorUserId="21749" LastEditDate="2017-02-27T12:38:07.127" LastActivityDate="2017-02-27T12:38:07.127" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14964" PostTypeId="1" AcceptedAnswerId="17410" CreationDate="2016-11-06T19:54:30.057" Score="4" ViewCount="2054" Body="&lt;p&gt;I have 2 digits numbers and 9 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I must pick 2 features, so decided to plot the features against each other to see whether I can get any insight on the best features to train my algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plot colours indicates the two digits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithms I considered to use were, K-Nearest Neighbour and Decision Tree.&#xA;I am very new to machine learning, I chose these two algorithms simply because I have come across them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature matrix of f1 to f9 against f1 to f9&#xA;&lt;a href=&quot;https://i.stack.imgur.com/zc61l.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zc61l.jpg&quot; alt=&quot;Feature matrix&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision Tree decision boundary&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4clAV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4clAV.png&quot; alt=&quot;Decision Tree&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will choosing the feature x against feature y with the least amount of overlap will help achieve the optimal decision boundary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I look at features should I initially consider linear data separation.&#xA;Then work my way up to use an algorithm that can deal with non-linear separated feature points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What important visual properties should I look out for when choosing optimal features for training?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I visualise the tree in sklearn python? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="7830" LastActivityDate="2017-03-07T11:02:50.433" Title="Feature Selection for K Nearest Neighbour and Decision Trees" Tags="|machine-learning|classification|scikit-learn|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14965" PostTypeId="2" ParentId="14964" CreationDate="2016-11-06T20:06:30.600" Score="0" Body="&lt;p&gt;Visual properties to look out for when choosing optimal features for training: choose the two features that show the separate groups the best&lt;/p&gt;&#xA;" OwnerUserId="25910" LastActivityDate="2016-11-06T20:06:30.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14966" PostTypeId="2" ParentId="14880" CreationDate="2016-11-06T20:27:35.467" Score="5" Body="&lt;p&gt;Considering ocam's razor I would recommend to use the simplest model first and increase the complexity if the simple models fail:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Exponentially Weighted Moving Average, allows only for auto correlation with lag one&lt;/li&gt;&#xA;&lt;li&gt;ARIMAX, allows for several lagged autocorrelation, seasonal adjustments and external regressor.&lt;/li&gt;&#xA;&lt;li&gt;Fourier transformation, allows to fit more shapes time series but they are often more complex to explain to users&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All these models can predict one step ahead and you can repeat this up to 60. Of course the confidence intervals become much wider these many steps ahead. Base R has many time series available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two models work well for many time series data. I would only start to use neural nets when these are not working or if you are looking for say edge detection. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Neural nets&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If neural nets are the only option to go you can check &lt;a href=&quot;http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt; or if you have matlab available &lt;a href=&quot;https://www.mathworks.com/help/nnet/gs/neural-network-time-series-prediction-and-modeling.html?s_tid=gn_loc_drop&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently found this very &lt;a href=&quot;https://medium.com/@alexrachnog/neural-networks-for-algorithmic-trading-part-one-simple-time-series-forecasting-f992daa1045a#.i96eokz28&quot; rel=&quot;nofollow noreferrer&quot;&gt;interesting article on medium&lt;/a&gt; that explains how to fit a Neural Net to time series data. A more detailed implementation guide can be found &lt;a href=&quot;http://machinelearningmastery.com/time-series-prediction-with-deep-learning-in-python-with-keras/?utm_content=buffer14c8e&amp;amp;utm_medium=social&amp;amp;utm_source=twitter.com&amp;amp;utm_campaign=buffer&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. I have not tried either approaches but I thought that the may be useful for future reference.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Time series modus operandi&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Note that in any case, before staring off modeling your time series it is highly recommended to take some preparatory steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Make your series stationary as explained &lt;a href=&quot;https://www.otexts.org/fpp/8/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Understand what elements you can decompose in your series as explained &lt;a href=&quot;https://www.otexts.org/fpp/6/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2016-12-28T17:00:49.127" LastActivityDate="2016-12-28T17:00:49.127" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14967" PostTypeId="2" ParentId="14953" CreationDate="2016-11-06T21:10:51.990" Score="0" Body="&lt;p&gt;Say $x$ is &lt;em&gt;sex&lt;/em&gt; and $y$ is &lt;em&gt;employed&lt;/em&gt;. You can restate your $p(y|x)$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ p(employed=1~|~sex=male) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, it's a simple matter of counting the relevant examples or looking up in the appropriate &lt;a href=&quot;https://en.wikipedia.org/wiki/Contingency_table&quot; rel=&quot;nofollow noreferrer&quot;&gt;contingency table&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-11-06T21:10:51.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14968" PostTypeId="1" CreationDate="2016-11-06T21:16:28.357" Score="1" ViewCount="128" Body="&lt;p&gt;I have hand writed classifiers (there are a lot of them). It's implemented as collection of rule sets &lt;code&gt;IIF - THEN&lt;/code&gt;.&#xA;I want to optimize the % of errors. There some classifiers witch have vey big % of &lt;code&gt;False Positive&lt;/code&gt; and &lt;code&gt;False Negative&lt;/code&gt; results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During my reserch about this problem i've found &lt;code&gt;RIPPER&lt;/code&gt; alghorytm witch, seems like, was designed to solve this kind of problems. &#xA;Also there are &lt;code&gt;Multi Naive Bias&lt;/code&gt; alghorythm that can be helpfull.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as i understand, usualy in &lt;code&gt;EA&lt;/code&gt; there is &lt;code&gt;Global Optimization&lt;/code&gt; step, withc usually/sometimes implemented via &lt;code&gt;RIPPER&lt;/code&gt;.&#xA;So, basicly. i have manually generated &lt;code&gt;rule-set&lt;/code&gt; witch i have optimize now, with &lt;code&gt;RIPPER&lt;/code&gt; for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it true? Can You recomend some literature?&lt;/p&gt;&#xA;" OwnerUserId="25880" LastActivityDate="2023-12-27T19:01:00.157" Title="Algorithm for rule set optimization" Tags="|classification|optimization|genetic-algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14969" PostTypeId="1" AcceptedAnswerId="14971" CreationDate="2016-11-06T21:40:40.120" Score="2" ViewCount="1348" Body="&lt;p&gt;I am working with the titanic survivors data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the data as a DataFrame and I can create 1D visualizations such as histograms, and also see the correlations by calling data.corr().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to create a scatter plot to represent the correlation between '&lt;strong&gt;age&lt;/strong&gt;' and '&lt;strong&gt;survived&lt;/strong&gt;'.  I can't figure out how to plot this data because '&lt;strong&gt;survived&lt;/strong&gt;' is effectively an integer of 0 or 1 (died or lived, respectively)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;titanic_data.plot(x='Age', y='Survived', style='o')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get a plot that looks like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/B02iq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B02iq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like is a plot that somehow takes the average survival rate by age and created something more like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IZTBR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IZTBR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25913" LastActivityDate="2016-11-06T23:41:23.397" Title="Plotting relationship between 2 data points where one data point is a boolean" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14970" PostTypeId="2" ParentId="14948" CreationDate="2016-11-06T22:37:38.910" Score="2" Body="&lt;p&gt;There is no reason why you cannot use Oracle 11g to create large databases. The 1000-column limit is a theoretical one, not a practical one imo. The theoretical database size limit is in petabytes. You should question why you would want to put that much data in a single database and be prepared to pay for the infrastructure to support it. This is a good summary link - &lt;a href=&quot;http://awads.net/wp/2010/02/15/oracle-database-limits-you-may-not-know-about/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://awads.net/wp/2010/02/15/oracle-database-limits-you-may-not-know-about/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having administered and tuned dbs of 50TB and more, the limits you should be more concerned with are for query performance. Ask Oracle about licensing as well before you proceed.&lt;/p&gt;&#xA;" OwnerUserId="8762" LastActivityDate="2016-11-06T22:37:38.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14971" PostTypeId="2" ParentId="14969" CreationDate="2016-11-06T23:41:23.397" Score="1" Body="&lt;p&gt;You can precalculate the survival rate (probability) and plot a bar plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;&#xA;x = sns.load_dataset('titanic')&#xA;&#xA;bins = np.linspace(0, 100, 11)&#xA;labels = bins[1:]&#xA;&#xA;# let's group all ages by bins (10, 20, 30, ..., 100)    &#xA;rpt = (x.groupby(pd.cut(x.age, bins, labels=labels))&#xA;        .survived.mean()*100&#xA;      ).dropna().to_frame('survival_rate')&#xA;&#xA;&#xA;rpt.plot.bar(rot=0, width=0.8, alpha=0.5, figsize=(12, 10))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8LOY1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8LOY1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;calculated data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [84]: bins&#xA;Out[84]: array([   0.,   10.,   20.,   30.,   40.,   50.,   60.,   70.,   80.,   90.,  100.])&#xA;&#xA;In [85]: labels&#xA;Out[85]: array([  10.,   20.,   30.,   40.,   50.,   60.,   70.,   80.,   90.,  100.])&#xA;&#xA;In [86]: rpt&#xA;Out[86]:&#xA;      survival_rate&#xA;age&#xA;10.0      59.375000&#xA;20.0      38.260870&#xA;30.0      36.521739&#xA;40.0      44.516129&#xA;50.0      38.372093&#xA;60.0      40.476190&#xA;70.0      23.529412&#xA;80.0      20.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2016-11-06T23:41:23.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14972" PostTypeId="1" AcceptedAnswerId="14974" CreationDate="2016-11-06T23:59:06.437" Score="2" ViewCount="2963" Body="&lt;p&gt;So, let's start out by saying I am NOT asking which is better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like both of these languages for data science and I think this is a matter of and rather than or since there is no need to choose one of them. My general opinion is that R has more data science depth while Python has more breadth in terms of diversity of application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will start this out with a few of my own scattered opinions.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Python&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I tend to use Python as the overarching language (sometimes using it to execute scripts in R) since:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Installed standard on Macs and many *nix setups&#xA;Easy to install/set up in environments with security restrictions&#xA;Has a great suite of tools whether I want to:&#xA;Set up a website&#xA;Interact with the Apache suite of tools&#xA;Build out pipelines&#xA;Build and score predictive models&#xA;Develop an application / front-end&#xA;Has a larger audience if I get stuck and need to ask questions on here&#xA;Debug / Unit test&#xA;Auto Document A good example of this was in building a Robot some facial recognition stuff over the summer. Raspbery Pi comes with Python by default, the sensors had Python API's, and a few of the web API's I bumped up against had Python API's available. This would have been a difficult application to take on using R.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;R&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I like to use R for early data explanation since data table tends to load larger sets into memory more quickly than Pandas and the general environment seems more mature for exploratory stuff. If I need mixed models or something similar that isn't Machine Learning &quot;bread and butter&quot; R is more likely to have what I need.  sklearn is pretty good but doesn't feel quite as polished as what I have in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the data viz side, ggplot2 is obviously better than matplotlib (the html based stuff is tough to share in an environment with heavy IP and security restrictions so I tend to lean towards the simple stuff.)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;How do you use them?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;How do you interact with these languages? Do you mix them together? Specialize in one? What's your rationale? Strengths/weaknesses?&lt;/p&gt;&#xA;" OwnerUserId="25915" LastActivityDate="2016-11-07T07:02:57.113" Title="Pros and Cons of Python and R for Data Science" Tags="|python|r|" AnswerCount="1" CommentCount="0" ClosedDate="2016-11-07T09:33:22.800" ContentLicense="CC BY-SA 3.0" />
  <row Id="14974" PostTypeId="2" ParentId="14972" CreationDate="2016-11-07T07:02:57.113" Score="7" Body="&lt;h2&gt;Interaction - Random Facts&lt;/h2&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Both are good stable languages with interesting complementary qualities. You can get much better packages in one and then stitch them with some data from the other. &lt;strong&gt;An example is using time series forecasting and decision trees in R and doing data munging in Python.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Both languages borrow from each other. Even seasoned package developers like &lt;code&gt;Hadley Wickham (Rstudio)&lt;/code&gt; borrows from &lt;code&gt;Beautiful Soup (python)&lt;/code&gt; to make &lt;code&gt;rvest&lt;/code&gt; for web scraping. In addition to that, &lt;code&gt;Yhat&lt;/code&gt; borrows from &lt;code&gt;sqldf&lt;/code&gt; to make &lt;code&gt;pandasql&lt;/code&gt; and many other.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Rather than  reinvent the wheel in the other language developers can focus on innovation because, in the end, the customer does not care which language the code was written, &lt;strong&gt;the customer cares for insights&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2&gt;Mixing Them Up&lt;/h2&gt;&#xA;&lt;p&gt;AM mentioning few approaches to mix them together-&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a Python package &lt;code&gt;rpy2&lt;/code&gt; to use R within Python . [&lt;a href=&quot;https://sites.google.com/site/aslugsguidetopython/data-analysis/pandas/calling-r-from-python&quot; rel=&quot;noreferrer&quot;&gt;Demo&lt;/a&gt;]&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use Python from within R using the &lt;code&gt;rPython package&lt;/code&gt;. [&lt;a href=&quot;http://www.programmingr.com/content/calling-python-r-rpython/&quot; rel=&quot;noreferrer&quot;&gt;Demo&lt;/a&gt;]&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;code&gt;Jupyter&lt;/code&gt; with the &lt;code&gt;IR Kernel&lt;/code&gt;. Python and R and makes the interactivity of iPython available to other languages.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;a href=&quot;https://decisionstats.com/2015/12/07/decisionstats-interview-scott-draves-beaker-notebook/&quot; rel=&quot;noreferrer&quot;&gt;Beaker Notebook&lt;/a&gt;. It allows you to switch from one language in one code block to another language in another code block in a streamlined way to pass shared objects.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2&gt;Python vs R&lt;/h2&gt;&#xA;&lt;p&gt;Python vs R - This section will answer:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Which will be better?&lt;/li&gt;&#xA;&lt;li&gt;How to choose one over other?&lt;/li&gt;&#xA;&lt;li&gt;Specialization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;See as I said earlier both are stable and you can choose any or work with both. But when it comes to master one I'll suggest keep these 3-4 guidelines in mind-&lt;/p&gt;&#xA;&lt;h3&gt;Personal Preference&lt;/h3&gt;&#xA;&lt;p&gt;Choose the language to begin with based on your personal preference, on which comes more naturally to you, which is easier to grasp from the get-go. To give you a sense of what to expect, mathematicians and statisticians tend to prefer R, whereas computer scientists and software engineers tend to favor Python.&lt;/p&gt;&#xA;&lt;h3&gt;Project selection&lt;/h3&gt;&#xA;&lt;p&gt;You can also make the Python vs. R call based on a project you know you’ll be working on in your data studies. If you’re working with data that’s been gathered and cleaned for you, and your main focus is the analysis of that data, go with R. If you have to work with dirty or jumbled data, or to scrape data from websites, files, or other data sources, you should start learning, or advancing your studies in, Python.&lt;/p&gt;&#xA;&lt;h3&gt;Collaboration&lt;/h3&gt;&#xA;&lt;p&gt;Once you have the basics of data analysis under your belt, another criterion for evaluating which language to further your skills in is what language your teammates are using. If you’re all literally speaking the same language, it’ll make collaboration—as well as learning from each other—much easier.&lt;/p&gt;&#xA;&lt;h3&gt;Job market&lt;/h3&gt;&#xA;&lt;p&gt;Jobs calling for skill in Python compared to R have increased similarly over the last few years.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Have a look at this &lt;a href=&quot;https://www.datacamp.com/community/tutorials/r-or-python-for-data-analysis#gs.k=J5=oY&quot; rel=&quot;noreferrer&quot;&gt;infographic&lt;/a&gt; by DataCamp. For a better view on it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;My Rationale&lt;/h2&gt;&#xA;&lt;p&gt;In my case am doing both and using them interactively and Customizing them as per my use. You can get something really interesting in one &lt;em&gt;(as I mentioned above)&lt;/em&gt; which will be hardly available in other, so it's better to use both together. This is the best way to bridge the gap between these two.&lt;/p&gt;&#xA;&lt;p&gt;But in the last, it's your call keep the guidelines, your interest, and scenarios in mind and make a clear view on that.&lt;/p&gt;&#xA;&lt;h2&gt;Strength &amp;amp; Weaknesses&lt;/h2&gt;&#xA;&lt;h3&gt;R&lt;/h3&gt;&#xA;&lt;p&gt;&lt;strong&gt;Strength&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;R is great for prototyping and for statistical analysis.&lt;/li&gt;&#xA;&lt;li&gt;It has a huge set of libraries available for different statistical type analysis. Check The Comprehensive &lt;a href=&quot;http://Networkr-project.org&quot; rel=&quot;noreferrer&quot;&gt;R Archive&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;RStudio IDE is a definitely a big plus. It eases most of the tedious tasks and fastens your workflow.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Weaknesses&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The syntax could be obscure sometimes.&lt;/li&gt;&#xA;&lt;li&gt;It is harder to integrate to a production workflow.&lt;/li&gt;&#xA;&lt;li&gt;In my opinion, it is better suited for &lt;code&gt;consultancy-type&lt;/code&gt; tasks.&lt;/li&gt;&#xA;&lt;li&gt;The libraries documentation isn't always user friendly.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Python&lt;/h3&gt;&#xA;&lt;p&gt;&lt;strong&gt;Strength&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python is great for scripting and automating your different data mining pipelines. It is the de facto scripting language nowadays.&lt;/li&gt;&#xA;&lt;li&gt;It also integrates easily in a production workflow. Besides, it can be used across different parts of your software engineering team&#xA;&lt;code&gt;(back-end, cloud architecture etc.)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The scikit-learn library is awesome for machine-learning tasks.&lt;/li&gt;&#xA;&lt;li&gt;Ipython (and its notebook) is also a powerful tool for exploratory analysis and presentations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Weaknesses&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It isn't as thorough for statistical analysis as R, but it has come a long way these recent years&lt;/li&gt;&#xA;&lt;li&gt;In my opinion, the learning curve is steeper than R, since you can do much more with Python.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;The Conclusion&lt;/h2&gt;&#xA;&lt;p&gt;Use R and Python. Learn how they inter-operate together. Start with one and then add the other to your workflow. As I like to remind myself- &lt;em&gt;&lt;strong&gt;&amp;quot;choosing the tools should never be the primary problem&amp;quot;&lt;/strong&gt;&lt;/em&gt;. When in doubt, use the one that is available and that gets the work done quickly.&lt;/p&gt;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;&lt;p&gt;Ref- Udacity, Quora, Letustweak, kD, DataCamp&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-07T07:02:57.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14975" PostTypeId="1" CreationDate="2016-11-07T07:34:29.327" Score="6" ViewCount="4335" Body="&lt;p&gt;I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I find a context in the data set and figure out contextual anomalies?&lt;/p&gt;&#xA;" OwnerUserId="25922" LastActivityDate="2017-02-05T19:19:19.920" Title="What methods can be used to detect anomalies in temporal texual data?" Tags="|text-mining|data-cleaning|unsupervised-learning|anomaly-detection|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14976" PostTypeId="1" CreationDate="2016-11-07T07:48:19.913" Score="1" ViewCount="80" Body="&lt;p&gt;i have a multilabel data set which is imbalenced and noisy.i used BR approach to convert single lablel and perform balanceing.data is fragmented into one class now i want to merge these fragmented class to create multilabel dataset is it possible?? &lt;/p&gt;&#xA;" OwnerUserId="25923" LastActivityDate="2016-11-07T07:48:19.913" Title="What is the difference between multilabel dataset and special dataset with respect to imbalance problem in datasets?" Tags="|classification|data-mining|multilabel-classification|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14977" PostTypeId="2" ParentId="14880" CreationDate="2016-11-07T12:06:14.437" Score="3" Body="&lt;p&gt;I would do it like this with a neural net. I would take a fixed interval (depending on your data). Then i feed it into the network and output the next timestep. Then you can remove the last datapoint in your initial input data and add the previous output to the input to predict the next step and so on. For training you can just stop at every point and take the last interval for x and the current point as y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you could also take a look at recurrent networks: &lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/layers/recurrent/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly I do not recommend using neural nets to solve this, because it is overkill in my opinion.&lt;/p&gt;&#xA;" OwnerUserId="25934" LastActivityDate="2016-11-07T12:06:14.437" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14978" PostTypeId="1" AcceptedAnswerId="14980" CreationDate="2016-11-07T12:56:04.243" Score="2" ViewCount="238" Body="&lt;p&gt;I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I followed &lt;a href=&quot;https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; tutorial  in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sigmoid(x):&#xA;    return 1 / (1 + np.math.e ** -x)&#xA;&#xA;&#xA;def sigmoid_deriv(x):&#xA;    return sigmoid(x) * (1 - sigmoid(x))&#xA;&#xA;&#xA;x_data = np.random.rand(500) * 15.0&#xA;y_data = [sin(x) for x in x_data]&#xA;&#xA;ETA = .01&#xA;&#xA;layer1 = 0&#xA;layer1_weights = np.random.rand(10) * 2. - 1.&#xA;layer2 = np.zeros(10)&#xA;layer2_weights = np.random.rand(10) * 2. - 1.&#xA;layer3 = 0&#xA;&#xA;for loop_iter in range(500000):&#xA;    # data init&#xA;    index = np.random.randint(0, 500)&#xA;    x = x_data[index]&#xA;    y = y_data[index]&#xA;&#xA;    # forward propagation&#xA;    # layer 1&#xA;    layer1 = x&#xA;&#xA;    # layer 2&#xA;    layer2 = layer1_weights * layer1&#xA;&#xA;    # layer 3&#xA;    layer3 = sum(sigmoid(layer2) * layer2_weights)&#xA;&#xA;    # error&#xA;    error = .5 * (layer3 - y) ** 2  # L2 loss&#xA;&#xA;    # backpropagation&#xA;    # error_wrt_layer3 * layer3_wrt_weights_layer2&#xA;    error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)&#xA;    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1&#xA;    error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1&#xA;&#xA;    # update the weights&#xA;    layer2_weights -= ETA * error_wrt_layer2_weights&#xA;    layer1_weights -= ETA * error_wrt_layer1_weights&#xA;&#xA;    if loop_iter % 10000 == 0:&#xA;        print(error)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the Tensorflow code it works flawlessly with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_data = np.array(np.random.rand(500)).reshape(500, 1)&#xA;y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)&#xA;&#xA;x = tf.placeholder(tf.float32, shape=[None, 1])&#xA;y_true = tf.placeholder(tf.float32, shape=[None, 1])&#xA;W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))&#xA;hidden1 = tf.nn.sigmoid(tf.matmul(x, W))&#xA;W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))&#xA;output = tf.matmul(hidden1, W_hidden)&#xA;&#xA;loss = tf.square(output - y_true) / 2.&#xA;optimizer = tf.train.GradientDescentOptimizer(.01)&#xA;train = optimizer.minimize(loss)&#xA;&#xA;init = tf.initialize_all_variables()&#xA;sess = tf.Session()&#xA;sess.run(init)&#xA;&#xA;for i in range(500000):&#xA;    rand_index = np.random.randint(0, 500)&#xA;    _, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],&#xA;                                              y_true: [y_data[rand_index]]})&#xA;    if i % 10000 == 0:&#xA;        print(error)&#xA;&#xA;sess.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user25935" LastEditorDisplayName="user25935" LastEditDate="2016-11-07T15:27:55.857" LastActivityDate="2016-12-05T19:58:14.203" Title="Backpropagation derivation problem" Tags="|deep-learning|backpropagation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14979" PostTypeId="2" ParentId="14975" CreationDate="2016-11-07T13:37:34.917" Score="1" Body="&lt;p&gt;This is the first hit on google for &lt;a href=&quot;http://nlp.shef.ac.uk/Completed_PhD_Projects/guthrie.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;anomaly detection text&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a little old but it's likely a good starting point. It seems the author uses heuristic algorithms (several measures of distance) to define an anomaly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option may be to look at the moving average of a given token (word) in your corpora over time and see if its t-stat is greater than some threshold. There are probably many, many better ways to do it but that might work depending upon your goal.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-11-07T13:37:34.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14980" PostTypeId="2" ParentId="14978" CreationDate="2016-11-07T14:37:35.283" Score="2" Body="&lt;p&gt;I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is important:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Added biases.&lt;/li&gt;&#xA;&lt;li&gt;Altered your error_w such that they are correct.&lt;/li&gt;&#xA;&lt;li&gt;Made some good random starting points for biases (&lt;code&gt;np.random.rand(width) * 15. - 7.5&lt;/code&gt;) such that all biases are random points on the desired x-scale.&lt;/li&gt;&#xA;&lt;li&gt;Made a plot that shows the initial guess and final.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let me know if some parts are not clear:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def sigmoid(x):&#xA;    return 1 / (1 + np.math.e ** -x)&#xA;&#xA;&#xA;def sigmoid_deriv(x):&#xA;    return sigmoid(x) * (1 - sigmoid(x))&#xA;&#xA;def guess(x):&#xA;    layer1 = x&#xA;    z_2 = layer1_weights * layer1 + layer1_biases&#xA;    a_2 =sigmoid(z_2)&#xA;    z_3 = np.dot(a_2, layer2_weights) + layer2_biases&#xA;    # a_3 = sigmoid(z_3)&#xA;    a_3 = z_3&#xA;    return a_3&#xA;&#xA;&#xA;x_data = np.random.rand(500) * 15.0 - 7.5&#xA;y_data = [np.sin(x) for x in x_data]&#xA;&#xA;ETA = 0.05&#xA;width = 10&#xA;&#xA;layer1_weights = np.random.rand(width) * 2. - 1.&#xA;layer1_biases = np.random.rand(width) * 15. - 7.5&#xA;layer2_weights = np.random.rand(width) * 2. - 1.&#xA;layer2_biases = np.random.rand(1)* 2. - 1.&#xA;&#xA;error_all = []&#xA;&#xA;x_all = x_data&#xA;y_all = [guess(x_i) for x_i in x_all]&#xA;&#xA;plt.plot(x_all,y_all, '.')&#xA;plt.plot(x_data, y_data, '.')&#xA;plt.show()&#xA;&#xA;epochs = 500000&#xA;&#xA;&#xA;&#xA;for loop_iter in range(epochs):&#xA;    # data init&#xA;    index = np.random.randint(0, 500)&#xA;    x = x_data[index]&#xA;    y = y_data[index]&#xA;&#xA;    # forward propagation&#xA;    # layer 1&#xA;    layer1 = x&#xA;&#xA;    # layer 2&#xA;    #TODO add the sigmoid function here&#xA;&#xA;    z_2 = layer1_weights * layer1 + layer1_biases&#xA;    a_2 =sigmoid(z_2)&#xA;&#xA;    # layer 3&#xA;    #TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid&#xA;    z_3 = np.dot(a_2, layer2_weights) + layer2_biases&#xA;    # a_3 = sigmoid(z_3)&#xA;    a_3 = z_3&#xA;&#xA;&#xA;    # error&#xA;    error = .5 * (a_3 - y) ** 2  # L2 loss&#xA;&#xA;    # backpropagation&#xA;    # error_wrt_layer3 * layer3_wrt_weights_layer2&#xA;&#xA;    # error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)&#xA;&#xA;    delta = (a_3 - y)&#xA;&#xA;    error_wrt_layer2_weights = delta * a_2&#xA;    error_wrt_layer2_biases = delta&#xA;&#xA;    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1&#xA;&#xA;    # error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1&#xA;    error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1&#xA;    # error_wrt_layer1_weights = 0&#xA;&#xA;    error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)&#xA;&#xA;&#xA;    # a = 0&#xA;    # while a ==0:&#xA;    #   a*0&#xA;&#xA;    # update the weights&#xA;    layer2_weights -= ETA * error_wrt_layer2_weights&#xA;    layer1_weights -= ETA * error_wrt_layer1_weights&#xA;    layer2_biases -= ETA * error_wrt_layer2_biases&#xA;    layer1_biases -= ETA * error_wrt_layer1_biases&#xA;&#xA;    error_all.append(error)&#xA;&#xA;    if loop_iter % 10000 == 0:&#xA;        print(error)&#xA;&#xA;&#xA;&#xA;# plt.plot(error_all)&#xA;# plt.show()&#xA;&#xA;x_all = x_data&#xA;y_all = [guess(x_i) for x_i in x_all]&#xA;&#xA;plt.plot(x_all,y_all, '.')&#xA;plt.plot(x_data, y_data, '.')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25104" LastEditorUserId="26596" LastEditDate="2016-12-05T19:58:14.203" LastActivityDate="2016-12-05T19:58:14.203" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14982" PostTypeId="1" AcceptedAnswerId="15003" CreationDate="2016-11-07T15:53:13.907" Score="5" ViewCount="1002" Body="&lt;p&gt;I have a logistic regression model where I care about predictive power solely over comprehensibility. I'm interested in predicting win rates in a video game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 133 characters. Each team picks 5 of them (no repeats). Each of these characters is assigned to one of five positions (again no repeats).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I have each of these characters as a dummy variable. In addition I have an interaction variable between each of these variables. The position of a character is not included in the model at present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I can trim down the size of the model by excluding low-playrate characters, however my concern is that the required sample size is still far too small for the complexity of the model. Any advice would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sample Size: Aprox. Two million&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25939" LastActivityDate="2016-11-14T14:45:39.357" Title="Logistic regression with high cardinality categorical variable" Tags="|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14983" PostTypeId="1" AcceptedAnswerId="15139" CreationDate="2016-11-07T16:10:09.143" Score="2" ViewCount="7632" Body="&lt;p&gt;I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand classification on a 70k database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it exists the Enron email dataset but do you know if it exists a version of this dataset with classified emails ? Or any other already classified email dataset ?&lt;/p&gt;&#xA;" OwnerUserId="25936" LastEditorUserId="25218" LastEditDate="2016-11-11T21:08:41.063" LastActivityDate="2016-11-16T10:26:51.333" Title="Classified Enron Email Dataset" Tags="|machine-learning|classification|dataset|databases|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="14984" PostTypeId="1" CreationDate="2016-11-07T16:13:02.830" Score="4" ViewCount="9535" Body="&lt;p&gt;I'am reading an article called &quot;FaceNet: A Unified Embedding for Face Recognition and Clustering&quot;. And there, they use something called &quot;inception&quot;. I guess it's something about layers, but I can't find &lt;strong&gt;any&lt;/strong&gt; information about it. Just found &lt;a href=&quot;http://pastebin.com/s9ACEvT8&quot; rel=&quot;nofollow noreferrer&quot;&gt;some code&lt;/a&gt;, which doesn't explain much. Can somebody give a clue, how can I search for this correctly, or maybe, kindly and briefly tell the concept idea?&lt;/p&gt;&#xA;" OwnerUserId="21369" LastEditorDisplayName="user380" LastEditDate="2017-06-02T15:43:34.633" LastActivityDate="2017-11-04T17:37:37.753" Title="What is an inception layer?" Tags="|neural-network|inception|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="14985" PostTypeId="2" ParentId="14975" CreationDate="2016-11-07T17:23:23.000" Score="3" Body="&lt;p&gt;This assumes you’re happy with neural networks. If you’re not, this answer probably isn’t of much use to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, a little bit about anomaly detection via autoencoders. Apologies if you’re already familiar with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An autoencoder is a neural network which learns to reproduce its own input when compressed through a “bottleneck” layer. For example, you may want to find a lower-dimensional feature representation of a set of 100 x 100 images. Your neural network architecture has an input layer of 10,000 elements, an output layer of 10,000 elements, and one of its hidden layers will be relatively narrow compared to the input space: say 100 nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to train the network to produce an output as close to the input as possible, while throwing away all but 100 nodes’ worth of activation. You are trying to produce as lossless a compression of the input as possible, so those 100 nodes should be a very information-rich representation of the kind of data you trained it on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“How does this have any bearing on anomaly detection?” I hear you ask. Well, if you train your autoencoder on non-anomalous data, it will learn a non-anomalous lower-dimensional feature representation. This will mean the reconstruction error from pushing something through the autoencoder will be lower for data similar to what it was trained on than it will for other arbitrary data. If it receives input that is substantively different from what it was trained on, the reconstruction error will be higher. So, given a set of novel inputs, those inputs with the highest reconstruction error are the most anomalous, as they are poorly reconstructed from the non-anomalous feature representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data has a temporal structure, and you have plenty of training samples, you might want to consider constructing an autoencoding LSTM. An LSTM is a neural network architecture for encoding and decoding sequentially dependent data, and a full description of how this works is beyond both the scope of this post and my own abilities. There are many magnificent resources available online for getting to grips with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-56.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a relevant paper on using LSTMs for anomaly detection in time series in general. It may be that LSTMs are unnecessary for your purposes if the data isn’t strongly sequentially dependent.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-11-07T17:23:23.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="14986" PostTypeId="1" CreationDate="2016-11-07T19:40:21.787" Score="1" ViewCount="144" Body="&lt;p&gt;There are some &lt;strong&gt;storage space&lt;/strong&gt; type like &lt;code&gt;GigaSpaces&lt;/code&gt;. that inspired from &lt;strong&gt;tuple space&lt;/strong&gt;.&lt;br&gt;&#xA;Would you say &lt;code&gt;mongodb&lt;/code&gt; can act like a &lt;strong&gt;tuple space&lt;/strong&gt; or not?&lt;br&gt;&#xA;What is the difference between &lt;code&gt;mongo&lt;/code&gt; and &lt;code&gt;tuple space&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="25947" LastActivityDate="2021-07-08T19:10:01.947" Title="What is the difference between mongodb and tuple space storages" Tags="|mongodb|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14987" PostTypeId="2" ParentId="14958" CreationDate="2016-11-07T20:20:50.603" Score="1" Body="&lt;p&gt;Your most important question is not the algorithm. You must check the precision of the sensor you can buy. I've tried several sensors, after 50 seconds the cumulated error can be a magnitude higher then the velocity itself. If you can buy a 1000 USD sensor, then it's precision would be sufficient, but you still have the problem that there will be no 0 position and 0 velocity, ie. the sensor will be moving even if the swimmer has not yet started to swim. This movement of the sensor will be greater then the difference between the velocities of two swimmers.&lt;/p&gt;&#xA;" OwnerUserId="25945" LastActivityDate="2016-11-07T20:20:50.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14989" PostTypeId="1" CreationDate="2016-11-07T23:14:08.840" Score="3" ViewCount="438" Body="&lt;p&gt;I have a huge data of nearly 200 features and several of them might be correlated? In order to select best features which is preferred Lasso regression or sequential forward selection considering the correlation between features.(I can not run pca on this because feature transformation would transform features which cannot be interpreted).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Few things I understood were: Lasso is really quick and sequential&#xA;forward selection is very slow. But I am not sure how would lasso&#xA;perform under correlated features i.e. I feel that the weights may be&#xA;distributed among features(assume necessary features) which are&#xA;correlated and because of low weights due to sharing Lasso might&#xA;suggest they are irrelevant features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25953" LastActivityDate="2016-11-08T21:45:25.310" Title="Which is a better method Lasso regression or sequential Foward selection?" Tags="|machine-learning|regression|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14990" PostTypeId="2" ParentId="14982" CreationDate="2016-11-08T00:40:35.060" Score="2" Body="&lt;p&gt;Look at techniques for dimensionality reduction, such as PCA. You can find several different methods in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/unsupervised_reduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn documentation.&lt;/a&gt;This will shrink the feature space based on tranformations from the original input features.&lt;/p&gt;&#xA;" OwnerUserId="426" LastActivityDate="2016-11-08T00:40:35.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14991" PostTypeId="1" CreationDate="2016-11-08T03:15:40.050" Score="1" ViewCount="66" Body="&lt;p&gt;I am studying the performance, over 10 years, of high school students that enrolled in a school district. My Objective is to make inferences about factors leading to poor performance in school exams. I have data and stats on all those who were dropouts. Drop outs are a big group from among the poor performers. For example if there are 1000 students, 250 are poor performers. Of these 250, 200 are drop outs and there is no data on those. I plan to run a logistic regression model and draw inferences, but how do i remedy the imbalance that this missing data is creating?&lt;/p&gt;&#xA;" OwnerUserId="18200" LastActivityDate="2017-05-27T21:23:11.127" Title="Imbalance in observable data" Tags="|predictive-modeling|logistic-regression|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14992" PostTypeId="2" ParentId="14957" CreationDate="2016-11-08T03:38:57.650" Score="0" Body="&lt;p&gt;Are you trying to recommend a list of ranked items to a user? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at collaborative filtering which is one of the most commonly used technique for recommender systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For python, &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pyspark&lt;/a&gt; is something you can look at. &lt;a href=&quot;https://www.codementor.io/spark/tutorial/building-a-recommender-with-apache-spark-python-example-app-part1&quot; rel=&quot;nofollow noreferrer&quot;&gt;This link&lt;/a&gt; contains some example how to implement it.&lt;/p&gt;&#xA;" OwnerUserId="20974" LastEditorUserId="20974" LastEditDate="2016-11-08T09:40:31.760" LastActivityDate="2016-11-08T09:40:31.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14994" PostTypeId="2" ParentId="14957" CreationDate="2016-11-08T04:14:42.693" Score="1" Body="&lt;p&gt;Your problem sounds like the classical top-N personal recommendation to me. There're lots of possibilities in the literature, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;User-based collaborative filtering&lt;/li&gt;&#xA;&lt;li&gt;Content-based collaborative filtering&lt;/li&gt;&#xA;&lt;li&gt;Matrix factorisation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You'll need to read the literature and figure out which one is better for you.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2016-12-09T03:29:35.453" LastActivityDate="2016-12-09T03:29:35.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="14997" PostTypeId="1" AcceptedAnswerId="15047" CreationDate="2016-11-08T10:30:33.353" Score="-1" ViewCount="1143" Body="&lt;p&gt;I know about the &lt;code&gt;fitdist()&lt;/code&gt; function from the &lt;code&gt;fitdistrplus&lt;/code&gt; package in R, however, I am not able to use it to predict a gaussian distribution. I can predict normal, logistic, weibull etc. &#xA;How can I use it for gaussian? are there any other ways to predict this? &lt;/p&gt;&#xA;" OwnerUserId="24742" LastEditorUserId="471" LastEditDate="2016-11-10T17:40:30.637" LastActivityDate="2016-11-11T03:51:55.457" Title="How to check if a data is in gaussian distribution in R or excel?" Tags="|r|visualization|gaussian|distribution|" AnswerCount="3" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="14998" PostTypeId="1" CreationDate="2016-11-08T11:00:02.070" Score="26" ViewCount="10370" Body="&lt;p&gt;I would like to set up a server which could support a data science team in the following way: be a central point for storing, versioning, sharing and possible also executing Jupyter notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some desired properties:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Different users can access the server and open and execute notebooks that were stored by them or by other team members. The interesting question here is what would be the behavior if user X executes cells in a notebook authored by user Y. I guess the notebook should &lt;em&gt;NOT&lt;/em&gt; be changed:&lt;/li&gt;&#xA;&lt;li&gt;Solution should be self-hosted.&lt;/li&gt;&#xA;&lt;li&gt;Notebooks should be stored either on the server or on Google drive or on self-hosted instance of owncloud.&lt;/li&gt;&#xA;&lt;li&gt;(Bonus) Notebooks will be under git versioning control (git may be self-hosted. Cannot be bounded to GitHub or something of that sort).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I looked into &lt;a href=&quot;https://github.com/jupyterhub&quot; rel=&quot;noreferrer&quot;&gt;JupyterHub&lt;/a&gt; and &lt;a href=&quot;https://github.com/binder-project/binder&quot; rel=&quot;noreferrer&quot;&gt;Binder&lt;/a&gt;. With the former, I didn't understand how to allow cross users access. The latter seems to only support GitHub as the storage of the notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have experience with either of the solutions? &lt;/p&gt;&#xA;" OwnerUserId="3591" LastEditorUserId="26596" LastEditDate="2016-12-05T06:28:50.133" LastActivityDate="2022-03-17T11:09:47.040" Title="Sharing Jupyter notebooks within a team" Tags="|software-recommendation|" AnswerCount="7" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="14999" PostTypeId="1" CreationDate="2016-11-08T12:33:03.713" Score="22" ViewCount="24737" Body="&lt;p&gt;Has anyone used (and liked) any good &quot;frequent sequence mining&quot; packages in Python other than the FPM in MLLib? I am looking for a stable package, preferable stilled maintained by people. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="25826" LastEditDate="2016-11-08T22:25:42.747" LastActivityDate="2021-04-08T16:36:12.637" Title="Good &quot;frequent sequence mining&quot; packages in Python?" Tags="|python|sequential-pattern-mining|" AnswerCount="7" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15000" PostTypeId="2" ParentId="14817" CreationDate="2016-11-08T13:05:57.193" Score="1" Body="&lt;p&gt;I would prefer more database-like operations such as joining:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;intersection = pd.merge(ratings_588_5, ratings_3578_3, on=['user_id'], how='inner')&#xA; print intersection.user_id.tolist()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[758, 1024]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17962" LastActivityDate="2016-11-08T13:05:57.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15002" PostTypeId="2" ParentId="14934" CreationDate="2016-11-08T13:11:40.410" Score="1" Body="&lt;p&gt;The genes shown in the diagrams have been simplified so that they only show topology and innovation index number. In addition, the full genome contains weights for each connection and bias for each node. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The innovation numbers are not tracking changes with connection weights or biases, just the rough time of inserting a specific connection between two nodes. So each parent can have matching connections/innovations, but with different weights associated. The full genome is still subject to cross-over and mutation, the excerpt you quote helps explain how the genome is aligned between parents with different architectures, so that GA crossover and mutation can occur in each connection and node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which parent is used to create each offspring gene matters because of the additional connection weight data associated with each parent's gene.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-09T19:36:40.077" LastActivityDate="2016-11-09T19:36:40.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15003" PostTypeId="2" ParentId="14982" CreationDate="2016-11-08T13:47:40.163" Score="5" Body="&lt;p&gt;So I believe you're building a model on the binary outcome {lose, win}:= {0, 1}, correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend just using a one-hot-encoding or a sparse matrix to store these inputs, then the model should run just fine. This is very straightforward in R (sparse.model.matrix) or Python (pd.get_dummies(sparse=True)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a quick demo of how to build a sparse matrix in R out of sampled categories and select a subset of them with at least 5 observations.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(MASS)&#xA;require(glmnet)&#xA;n &amp;lt;- 1000&#xA;x1 &amp;lt;- sample(paste(letters,1), n, replace=T)&#xA;x2 &amp;lt;- sample(paste(letters,2), n, replace=T)&#xA;x3 &amp;lt;- paste(x1,x2,sep='-')&#xA;xdf &amp;lt;- data.frame(x1,x2,x3)&#xA;xs &amp;lt;- sparse.model.matrix(~.-1, data=xdf)&#xA;vars &amp;lt;- colnames(xs)&#xA;colsmry &amp;lt;- colSums(xs)&#xA;colsubset &amp;lt;- colsmry &amp;gt; 4&#xA;xs_ss &amp;lt;- xs[,vars[colsubset]]&#xA;dim(xs)&#xA;dim(xs_ss)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2016-11-14T14:45:39.357" LastActivityDate="2016-11-14T14:45:39.357" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15004" PostTypeId="1" CreationDate="2016-11-08T16:25:57.343" Score="3" ViewCount="406" Body="&lt;p&gt;In many ML problems we collect data and train models using the collected data. Using recommendation as an example, data collected could be biased for various reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;presentation bias. For example, on web one may have pagination; for mobile one may need to scroll down to see items beyond a few.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;bias from existing recommendation system: usually data is collected on a running production recommendation system, this introduces bias. For example, due to personalization, a male consumer may have little chance to see female products, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What are approaches to handle this when training a model?&lt;/p&gt;&#xA;" OwnerUserId="1376" LastActivityDate="2016-11-09T12:58:01.800" Title="How to handle data collecting bias in machine model training" Tags="|machine-learning|predictive-modeling|data-cleaning|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15005" PostTypeId="1" AcceptedAnswerId="15059" CreationDate="2016-11-08T17:28:15.877" Score="3" ViewCount="2314" Body="&lt;p&gt;I am working with a dataframe in R that is formatted like this sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Countries &amp;lt;- c('USA','USA','Australia','Australia')&#xA;Type &amp;lt;- c('a','b','a','b')&#xA;X2014 &amp;lt;- c(10, -20, 30, -40)&#xA;X2015 &amp;lt;- c(20, -40, 50, -10)&#xA;X2016 &amp;lt;- c(15, -10, 10, -100)&#xA;X2017 &amp;lt;- c(5, -5, 5, -10)&#xA;&#xA;df_sample &amp;lt;- data.frame(Countries, Type, X2014, X2015, X2016, X2017)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dataframe looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Countries Type X2014 X2015 X2016 X2017&#xA;1       USA    a    10    20    15     5&#xA;2       USA    b   -20   -40   -10    -5&#xA;3 Australia    a    30    50    10     5&#xA;4 Australia    b   -40   -10  -100   -10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to be able to create columns of year values for each type by each country, yielding something that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        Countries   Year     a     b &#xA;1       USA         X2014    10   -20   &#xA;2       USA         X2015    20   -40   &#xA;3       USA         X2016    15   -10  &#xA;4       USA         X2017     5    -5  &#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;recast&lt;/code&gt; I get this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;recast(df_sample, Countries ~ Type)&#xA;&#xA;  Countries a b&#xA;1 Australia 4 4&#xA;2       USA 4 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;dcast&lt;/code&gt; I get this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dcast(df_sample, Countries ~ Type)&#xA;&#xA;  Countries a   b&#xA;1 Australia 5 -10&#xA;2       USA 5  -5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dataset I'm working with has 44 years of data, so I'd like to be able to indicate all columns of yearly data without having to enter each column id manually into a cast formula. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between &lt;code&gt;dcast&lt;/code&gt; and &lt;code&gt;recast&lt;/code&gt; (i.e. what situations might they be best suited to), and is it possible to shape my data with them?&lt;/p&gt;&#xA;" OwnerUserId="23701" LastActivityDate="2016-11-11T14:43:16.223" Title="What is the difference between dcast and recast in R?" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15006" PostTypeId="1" AcceptedAnswerId="15007" CreationDate="2016-11-08T19:00:48.267" Score="11" ViewCount="22619" Body="&lt;p&gt;I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately what I'm trying to do here is to count unique values on a certain column and then determine which of those unique values have more than one unique value in a matching column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for this data, what I am trying to determine is &quot;who&quot; has &quot;more than one receipt&quot; for all purchases, then determine the same information based on each product category.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;My approach so far:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;We have a dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;receipt,name,etc,category&#xA;1,george,xxx,fish&#xA;1,george,xxx,cat&#xA;2,george,xxx,fish&#xA;3,bill,xxx,fish&#xA;3,bill,xxx,dog&#xA;4,jill,xxx,cat&#xA;5,bill,xxx,cat&#xA;5,bill,xxx,cat&#xA;5,bill,xxx,dog&#xA;6,george,xxx,fish&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So then I can do this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.set_index(['name','receipt'])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And get the more interesting&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                etc category&#xA;name   receipt&#xA;george 1        xxx     fish&#xA;       1        xxx      cat&#xA;       2        xxx     fish&#xA;bill   3        xxx     fish&#xA;       3        xxx      dog&#xA;jill   4        xxx      cat&#xA;bill   5        xxx      cat&#xA;       5        xxx      cat&#xA;       5        xxx      dog&#xA;george 6        xxx     fish&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this point it feels to me like the data is easy to work with, but I haven't figured it out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that is interesting to me is that if I sort the data by name before indexing it, the data displays grouped by name. In both cases the index is the same, so I don't know how to play with the representation of the data after indexing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy to find the data by category using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; orders.loc[orders['category'] == 'fish']&#xA;                etc category&#xA;name   receipt&#xA;george 1        xxx     fish&#xA;       2        xxx     fish&#xA;bill   3        xxx     fish&#xA;george 6        xxx     fish&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But what I can't figure out is how to tell pandas &quot;Find me the list of names that have more than one receipt&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smaller questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the &quot;pandas way&quot; to get the length of the names part of the index? I'm supposing I could just turn the &lt;code&gt;name&lt;/code&gt; column into a set and get the length of that. But I'm curious about indexes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Edit / Update&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Thanks for those answers! Here is clarification on what I am looking for:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to find &quot;repeat customers&quot;: people with more than one receipt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my set of all customers would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names: ['george','bill','jill'], ratio: 1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My repeat customers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names: ['george','bill'], ratio 0.66&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All 'fish' customers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names: ['george','bill'], ratio: 0.666&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My repeat 'fish' customers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names: ['george'], ratio: 0.333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think the examples given look helpful, but feel free to add anything.&lt;/p&gt;&#xA;" OwnerUserId="25970" LastEditorUserId="25970" LastEditDate="2016-11-09T00:23:14.450" LastActivityDate="2016-11-09T00:23:14.450" Title="Counting indexes in pandas" Tags="|python|pandas|indexing|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15007" PostTypeId="2" ParentId="15006" CreationDate="2016-11-08T20:33:57.837" Score="7" Body="&lt;p&gt;I think maybe you are looking for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;receipts_by_name_x_cat = df.groupby(['name','category']).count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or, if you just want the total across all categories:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;receipts_by_name = df.groupby(['name']).count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, you can search those who have more than one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;receipts_by_name[receipts_by_name['receipt']&amp;gt;1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And, you can find the length of an index by typing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;len(df.index.get_level_values(0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Assuming the name was the first index column (otherwise substitute 1, 2, etc.)&lt;/p&gt;&#xA;" OwnerUserId="23262" LastActivityDate="2016-11-08T20:33:57.837" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15008" PostTypeId="2" ParentId="14989" CreationDate="2016-11-08T21:45:25.310" Score="2" Body="&lt;p&gt;I think there is no obvious advantage of one algorithm over another: I'd suggest running cross-validation to investigate which works best for the data at hand. You've suggested forward stepwise and Lasso, which both performs feature selection, you could also investigate backward stepwise and best-subset selection. Linear regression models are fast to train, so maybe you could investigate the cross-validation error for all 4 algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-08T21:45:25.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15009" PostTypeId="2" ParentId="10785" CreationDate="2016-11-08T22:00:02.850" Score="3" Body="&lt;p&gt;I was thinking if Apriori might be more suited for your purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your consideration:&#xA;1) Tokenize the training sentences into bag of words:&#xA;Review Sentence | Upside | Calm | Swimming | Cat&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Tag the correct consequents for bag of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Apriori should produce 1 rule for ambiance and 1 rule for Entertainment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-08T22:00:02.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15010" PostTypeId="1" CreationDate="2016-11-08T22:47:43.667" Score="4" ViewCount="5479" Body="&lt;p&gt;When does it makes senses to use Dot-Product as similarity measure instead of Cosine?. I have seen there is already question asked about this, However that merely explains the difference between calculation of dot-product &amp;amp; cosine and it does not focus on when should we use one vs another with real world example.&lt;/p&gt;&#xA;" OwnerDisplayName="user25976" LastActivityDate="2016-11-12T16:53:49.757" Title="When does it makes senses to use Dot-Product as similarity measure instead of Cosine?" Tags="|machine-learning|similarity|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15012" PostTypeId="2" ParentId="15006" CreationDate="2016-11-08T23:29:27.393" Score="3" Body="&lt;p&gt;It's not quite clear what exactly are you trying to achieve (it would be helpful to understand your goals if you would post desired / expected data sets)...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But i'll try to guess ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [100]: df&#xA;Out[100]:&#xA;   receipt    name  etc category&#xA;0        1  george  xxx     fish&#xA;1        1  george  xxx      cat&#xA;2        2  george  xxx     fish&#xA;3        3    bill  xxx     fish&#xA;4        3    bill  xxx      dog&#xA;5        4    jill  xxx      cat&#xA;6        5    bill  xxx      cat&#xA;7        5    bill  xxx      cat&#xA;8        5    bill  xxx      dog&#xA;9        6  george  xxx     fish&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Setting virtual column &lt;code&gt;count&lt;/code&gt;, showing # of rows grouped by &lt;code&gt;name&lt;/code&gt; and filtering (querying) it using &lt;code&gt;.query()&lt;/code&gt; method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [101]: (df.assign(count=df.groupby('name').receipt.transform('size'))&#xA;     ...:    .query(&quot;category in ['dog','cat'] and count &amp;gt; 1&quot;))&#xA;     ...:&#xA;Out[101]:&#xA;   receipt    name  etc category  count&#xA;1        1  george  xxx      cat      4&#xA;4        3    bill  xxx      dog      5&#xA;6        5    bill  xxx      cat      5&#xA;7        5    bill  xxx      cat      5&#xA;8        5    bill  xxx      dog      5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or you can group by several columns and filter resulting groups:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [102]: df.groupby(['name','category']).filter(lambda x: len(x) &amp;gt; 2)&#xA;Out[102]:&#xA;   receipt    name  etc category&#xA;0        1  george  xxx     fish&#xA;2        2  george  xxx     fish&#xA;9        6  george  xxx     fish&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2016-11-08T23:29:27.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15013" PostTypeId="2" ParentId="15010" CreationDate="2016-11-09T00:08:29.513" Score="2" Body="&lt;p&gt;Borrowing from &lt;a href=&quot;https://www.quora.com/Is-cosine-similarity-effective&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Quora answer&lt;/a&gt; -- which includes some concrete examples of when you might prefer one measure over the other -- it comes down to whether you care about taking the magnitude of the vectors into account. This is highly domain-specific, but the example used there is in information retrieval, where magnitude represents the length of the documents in question. The dot product will take the document length into account, whereas the cosine similarity will not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, you will want to have good reasons specific to your problem space to use one over the other.&lt;/p&gt;&#xA;" OwnerUserId="23581" LastActivityDate="2016-11-09T00:08:29.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15015" PostTypeId="1" CreationDate="2016-11-09T04:38:22.307" Score="2" ViewCount="333" Body="&lt;p&gt;Can someone explain the data visualization of Google's live election results to me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that the states they've &quot;called&quot; have a check mark that is blue if a Democrat (Clinton) victory or red if a Republican (Trump) victory. The results additionally show the number of votes report thusfar as a count and as a percent. The states they consider too early to call based on the % of precincts reporting do not have a check mark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the case of Washington it has been &quot;called&quot; as a Democrat (Clinton) victory, but shows a higher count and percent for Trump (?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;View 1:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aYzWO.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aYzWO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;View 2:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WLIGj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WLIGj.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: There were some smaller third party candidates cut off in the screenshot above, like 2 Socialist candidates.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2016-11-09T04:52:32.453" LastActivityDate="2019-04-30T23:01:20.230" Title="Google's election results data visualization" Tags="|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15016" PostTypeId="1" CreationDate="2016-11-09T07:21:55.643" Score="2" ViewCount="1038" Body="&lt;p&gt;I'm working through &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap3.html&lt;/a&gt;,&#xA;textbook by Michael Nielsen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While most problems are rather straightforward, I'm very stuck on these two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $ \exp(−\eta\lambda/m)$ per epoch;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a self-sufficient context, the weight decay function is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}&#xA;  \sum_x \frac{\partial C_x}{\partial w}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and cost function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln&#xA;(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Classic logloss)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weights in the network are initiated as $N(0,1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$. &#xA;Given that $\lambda$ is arbitrary, we can be sure that &#xA;$|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m}&#xA;  \sum_x \frac{\partial C_x}{\partial w}|$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite questions being rather informal, I don't think this solution is a good one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope someone can give me a hand in this.&#xA;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. There is a chance I'm going to the wrong overflow, but I used&#xA;&lt;a href=&quot;https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms&quot;&gt;https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms&lt;/a&gt;&#xA;question as a guidance for overflow choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A note: I had asked similiar question &lt;a href=&quot;https://cs.stackexchange.com/questions/65679/speed-decay-proof-for-l2-regularization-and-non-normalizied-weight-initiation&quot;&gt;here&lt;/a&gt;, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)&lt;/p&gt;&#xA;" OwnerUserId="3108" LastEditorUserId="-1" LastEditDate="2017-03-20T10:32:30.217" LastActivityDate="2017-03-10T19:17:37.630" Title="Speed decay proof for L2 regularization and non-normalizied weight initiation" Tags="|neural-network|regularization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15017" PostTypeId="1" CreationDate="2016-11-09T08:07:05.340" Score="1" ViewCount="427" Body="&lt;p&gt;I have studied the paper &lt;em&gt;&lt;a href=&quot;https://arxiv.org/abs/1605.06211&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fully Convolutional Networks for Semantic Segmentation&lt;/a&gt;&lt;/em&gt; (Shelhamer, Long and Darrell) and understand the process. They provide &lt;a href=&quot;https://github.com/shelhamer/fcn.berkeleyvision.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;extensive code in Caffe&lt;/a&gt;. I have been analyzing the code for some time, but I however don't understand how to use this code for my own data. I have everything installed correctly (Caffe, Python, etc). I would, for example, like to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Test the code on an arbitrary image and obtain a segmented image as an output.&lt;/li&gt;&#xA;&lt;li&gt;Train the network on own image pairs {image, segmented image}.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there a tutorial I could follow on how to use this code. Or could someone give me a some guidelines. I would appreciate it very much.&lt;/p&gt;&#xA;" OwnerUserId="25983" LastEditorUserId="29169" LastEditDate="2021-02-16T15:34:04.490" LastActivityDate="2021-02-16T15:34:04.490" Title="Fully Convolutional Network: How To" Tags="|neural-network|convolutional-neural-network|" AnswerCount="0" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15020" PostTypeId="2" ParentId="15015" CreationDate="2016-11-09T08:42:24.110" Score="1" Body="&lt;p&gt;Possibly because google was predicting based on changes from previous elections, possibly combined with opinion polls, on a precinct-basis. If all the pro-Trump precincts report in first, you'd still see a blue box because of the very low probability of historically Democrat precincts voting Trump. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Calling&quot; something in a US election is not an official result, its a confident prediction by an agency. Sometimes these are wrong. Search for &quot;Dewey Defeats Truman&quot; for a classic example.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-11-09T08:42:24.110" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15022" PostTypeId="2" ParentId="13284" CreationDate="2016-11-09T09:35:06.100" Score="2" Body="&lt;p&gt;There is nothing specifically called &lt;code&gt;Twitter sentiment analysis&lt;/code&gt;, rather its colloquial. &lt;code&gt;Sentiment analysis&lt;/code&gt; is a common term and could be applied on any dataset depending upon the use-case considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not came across any facebook api for scraping posts (maybe there are privacy concerns, as sometimes you cannot see fb pages when you are not an user), nevertheless there are several apis to scrap tweets from twitter, even specifying person, geographic location, language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/sivu1/537cefa03a83ee4a1beb7cd581a71f40&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is a github gist for reference, it uses python tweepy module to grab current tweets.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-09T09:35:06.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15023" PostTypeId="1" AcceptedAnswerId="15025" CreationDate="2016-11-09T09:44:31.527" Score="2" ViewCount="2985" Body="&lt;p&gt;I'm trying to teach myself data science, with my particular interest being decision trees. A few steps in, I've come across a term, 'parameter convergence' that I can't find a definition for (because, after all, I'm learning on my own and have no access to teachers or peers):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However, even in studies with much lower numbers of predictor&#xA;  variables, the combination of all main and interaction effects of&#xA;  interest – especially in the case of categorical predictor variables –&#xA;  may well lead to cell counts too sparse for &lt;strong&gt;parameter convergence&lt;/strong&gt;. (from &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/19968396&quot; rel=&quot;nofollow noreferrer&quot;&gt;Strobl et al., 2009&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A web search isn't overly helpful because &lt;em&gt;convergence&lt;/em&gt; is such a common term, and I'm not sure which results specifically apply in the context of decision trees. And also, the results don't provide an entry-level definition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, while a definition or explanation of parameter convergence (in the context of recursive partitioning) would be great, it would also be handy to be directed to a resource (academic or otherwise) that might have a 'glossary' of this and similar terms...&lt;/p&gt;&#xA;" OwnerUserId="24871" LastEditorUserId="21024" LastEditDate="2016-11-10T15:53:51.583" LastActivityDate="2016-11-10T15:53:51.583" Title="What is 'parameter convergence'?" Tags="|machine-learning|decision-trees|parameter-estimation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15024" PostTypeId="1" CreationDate="2016-11-09T11:26:47.967" Score="0" ViewCount="41" Body="&lt;p&gt;Lets assume that I have 5000 articles and I create the TF-IDF of these articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I ask som people to answer 30 questions and I create the TF-IDF of these answers from the IDF of the articles and use the cosinus similarity to find the 20 closest articles per person with a weight of 1 for each answer. People can mark these 20 articles as relevant to her/him or not relevant(0 or 1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, which approach would you follow to see which questions are the most important in their choices? In my mind it seems like feature selection problem. Or not? Any recommendation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, what I want to do is reduce the number of questions from 30 -&gt; 5 and I have to find which one are the most important questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @null_9 mentioned in a comment, I am not sure which is the best way of providing the most relevant articles: 1) Summing up the similarity scores from all the answers and give the highest, or 2) find &lt;code&gt;N&lt;/code&gt; closest articles per each question and sample 20 of them and provide to the user.&lt;/p&gt;&#xA;" OwnerUserId="14736" LastEditorUserId="14736" LastEditDate="2016-11-09T13:54:23.330" LastActivityDate="2016-11-09T14:13:28.297" Title="Finding the most important questions from a questionary-&gt; results" Tags="|feature-selection|bayesian|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15025" PostTypeId="2" ParentId="15023" CreationDate="2016-11-09T11:27:23.193" Score="1" Body="&lt;p&gt;A naive definition of Parameter convergence is when the weights or the values of the parameters reach a point asymptotically. What I mean is that, when your model training is not altering the parameter values(maybe less than epsilon-small values) it might be a good fit. For decision trees, I found this &lt;a href=&quot;http://papers.nips.cc/paper/2625-on-the-adaptive-properties-of-decision-trees.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; which explains rate of convergence and more. It might be a good read if you want to get more details.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-11-09T11:27:23.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15026" PostTypeId="2" ParentId="15004" CreationDate="2016-11-09T12:58:01.800" Score="1" Body="&lt;p&gt;In the world of Big data, the data is already there and there are more than enough ML models currently but constantly there are improvements and applications, the main reason for that is how the features are defined from the raw data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, introduce a new feature as penalty for the bias present in the data. For ex. in the presentation bias, high penalty is given to the top items and low penalty is given to the items farther down. similarly, introduce weight feature with less weight for men products when most of the products are male products and vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML still needs human intuition in the form of features ;-) &lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-09T12:58:01.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15027" PostTypeId="2" ParentId="15024" CreationDate="2016-11-09T14:13:28.297" Score="1" Body="&lt;p&gt;If selecting the most important and relative questions is the question and you choose the option:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;option 1.) find the closest document to each answer(30 docs) and after the documents are validated relative or not. You will have the results of each person's evaluation (&lt;code&gt;X&lt;/code&gt;persons x 30 scores). Now order the questions in descending order from most scored question to less scored one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;option 2.) Summing up all the answers and finding 20 docs. After the validation of relativeness by the persons, it is not possible to find the important features(questions) because they are summed up and the information is already lost. From this result you could only find which person validates closer to your model among all, which is not the aim of the work.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-09T14:13:28.297" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15028" PostTypeId="2" ParentId="14997" CreationDate="2016-11-09T14:21:38.417" Score="2" Body="&lt;p&gt;The Normal distribution is the same as the Gaussian distribution. Its just two names for the same thing. Whatever you do - fit parameters, compute goodness-of-fit, etc - if the documentation says its for a Normal distribution then you can say &quot;Gaussian&quot; instead. Completely and totally identical.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-11-09T14:21:38.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15029" PostTypeId="1" CreationDate="2016-11-09T14:36:40.183" Score="1" ViewCount="811" Body="&lt;p&gt;I've run &lt;code&gt;GridSearchCV&lt;/code&gt; to determine 'best parameters' for a linear SVM, and then passed these [in a dictionary along with non-tuned parameters] into a new SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've printed out the parameter dictionaries and confirmed that the output of the grid search feeds the parameters correctly into the new SVM (&lt;code&gt;sklearn.svm.SVC()&lt;/code&gt;), and yet where the grid search achieved AUC of 0.89, on the same dataset I achieve just 0.5, and &lt;em&gt;&quot;no predicted samples&quot;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To confirm, I set the &lt;code&gt;param_grid&lt;/code&gt; to a 1x1 grid (i.e. no searching, just running 5-fold cross-val using SVM estimator) and it still &quot;does well&quot; only to drop to 0.5 AUC score afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please help diagnose this disparity?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;(Note that this code has a reduced &lt;code&gt;logspace&lt;/code&gt; parameter space, to reduce running time after I found the most performant values)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.model_selection import StratifiedShuffleSplit, GridSearchCV&#xA;&#xA;def SVMParamTune(model_class, X, y, params):&#xA;    from sklearn.svm import SVC&#xA;    C_range = np.logspace(-2, -2, 1)&#xA;    gamma_range = np.logspace(3, 3, 1)&#xA;    param_grid = dict(gamma=gamma_range, C=C_range)&#xA;    # If the configuration contains other parameters, use these&#xA;    # These are the 'constants' to parameterising the SVM&#xA;    for param_name in params:&#xA;        if param_name not in param_grid.keys():&#xA;            pass&#xA;            # param_grid[param_name] = params[param_name]&#xA;    cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)&#xA;    # NB the SVC estimator provides a `score` method,&#xA;    # meaning GridSearchCV uses mean accuracy (from this estimator's score)&#xA;    grid = GridSearchCV(SVC(), param_grid=param_grid, scoring='roc_auc', cv=cv, n_jobs=10, verbose=10)&#xA;    grid.fit(X, y)&#xA;    print(&quot;The best parameters are %s with an AUC score of %0.2f&quot;&#xA;                  % (grid.best_params_, grid.best_score_))&#xA;    print(grid.best_params_)&#xA;    for param in params:&#xA;        if param in grid.best_params_.keys():&#xA;            params[param] = grid.best_params_[param]&#xA;    print(params)&#xA;    return params&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is run, I see a good result, yet the result doesn't make it out of the tuning process for reasons unknown.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m_params = SVMParamTune(m_class, X=data[0], y=data[1], params=m_params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The class (&lt;code&gt;sklearn.svm.SVC&lt;/code&gt;, stored in &lt;code&gt;m_class&lt;/code&gt;) is then run with these updated parameters, and yet gives different results&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_run = m_class(**m_params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Switching &lt;code&gt;StratifiedShuffleSplit&lt;/code&gt; to &lt;code&gt;StratifiedKFold&lt;/code&gt; to be consistent with the latter SVM run made negligible difference so cannot be the cause.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt; returning the model itself also gives the same changed AUC score - the model itself cannot be reproduced, on the same data...&lt;/p&gt;&#xA;" OwnerUserId="25236" LastEditorUserId="25236" LastEditDate="2016-11-10T10:05:38.253" LastActivityDate="2016-11-10T10:05:38.253" Title="GridSearchCV with SVM estimator AUC score not reproduced on SVM run" Tags="|scikit-learn|svm|cross-validation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15030" PostTypeId="2" ParentId="14997" CreationDate="2016-11-09T14:45:07.293" Score="0" Body="&lt;p&gt;It depends on what you mean by predicting Gaussian/normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to check the fit, then you can estimate &lt;strong&gt;mean&lt;/strong&gt; and standard &lt;strong&gt;deviation&lt;/strong&gt; of your sample and plot two &lt;strong&gt;histograms&lt;/strong&gt;: one for your sample and second after generating a sample Gaussian distribution with the mean and standard deviation. By comparing the two you can roughly &quot;see&quot; the fit.&lt;/p&gt;&#xA;" OwnerUserId="17962" LastEditorUserId="17962" LastEditDate="2016-11-10T14:36:46.057" LastActivityDate="2016-11-10T14:36:46.057" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15031" PostTypeId="2" ParentId="15023" CreationDate="2016-11-09T23:50:48.730" Score="1" Body="&lt;p&gt;Many ML and minimization tasks make use of an objective function. At each iteration, a parameter set to try is defined, and the objective function returns some scoring value that reflects how good, or bad, that parameter set is. Then the parameter set is altered, and the process repeats. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, when do you stop this process?  You want to stop when the changes in fit (getting ever closer to a local or global minima) become acceptably small. What is acceptable?  It's up to you, but partially influenced by the form of the gradient, or error space. There are guidelines, and many algorithms build those in by default. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a set, these stopping rules comprise convergence criteria. When an algorithm converges, it has found a parameter set meeting your requirements. There are many ways for it to fail, particularly if a suitable parameter set can't be found within some maximum number of iterations. Conversely, you can set up unreasonable stopping criteria that will lead to convergence with very poor parameters. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2016-11-09T23:50:48.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15032" PostTypeId="1" AcceptedAnswerId="15038" CreationDate="2016-11-10T06:38:44.397" Score="13" ViewCount="23873" Body="&lt;p&gt;I have two tensor &lt;code&gt;a:[batch_size, dim]&lt;/code&gt;  &lt;code&gt;b:[batch_size, dim]&lt;/code&gt;.&#xA;I want to do inner product for every pair in the batch, generating &lt;code&gt;c:[batch_size, 1]&lt;/code&gt;, where &lt;code&gt;c[i,0]=a[i,:].T*b[i,:]&lt;/code&gt;. How?&lt;/p&gt;&#xA;" OwnerUserId="25772" LastEditorUserId="34306" LastEditDate="2017-07-07T14:43:01.253" LastActivityDate="2017-11-11T13:31:47.033" Title="How to do batch inner product in Tensorflow?" Tags="|tensorflow|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15036" PostTypeId="1" AcceptedAnswerId="15037" CreationDate="2016-11-10T07:15:27.903" Score="-2" ViewCount="110" Body="&lt;p&gt;I tried &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mtcars &amp;lt;- data.table( mtcars, keep.rownames=T)&#xA;&#xA;mtcars[median(qsec),]&#xA;&#xA;mtcars[order(qsec),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me 15th value though it should have given me the 16 value as the dataset has 32 values in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me if I am doing something wrong... I know it's quite the basics&lt;/p&gt;&#xA;" OwnerUserId="25876" LastEditorUserId="2961" LastEditDate="2016-11-10T15:53:44.613" LastActivityDate="2016-11-10T15:53:44.613" Title="Median function in R" Tags="|r|statistics|" AnswerCount="1" CommentCount="7" ClosedDate="2016-11-10T15:54:32.330" ContentLicense="CC BY-SA 3.0" />
  <row Id="15037" PostTypeId="2" ParentId="15036" CreationDate="2016-11-10T07:54:28.730" Score="2" Body="&lt;p&gt;I don't know what your &lt;code&gt;qsec&lt;/code&gt; is, so I'll assume you've attached &lt;code&gt;mtcars&lt;/code&gt; and its the column from there. PLEASE edit your question and show &lt;em&gt;ALL&lt;/em&gt; your working so anyone else can get your results. Let's proceed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; attach(mtcars) # WARNING: Attaching data frames is a BAD IDEA&#xA;&amp;gt; median(qsec)&#xA;[1] 17.71&#xA;&amp;gt; mtcars[median(qsec),]&#xA;                   mpg cyl disp  hp drat    wt  qsec vs am gear carb&#xA;Chrysler Imperial 14.7   8  440 230 3.23 5.345 17.42  0  0    3    4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because the median of &lt;code&gt;qsec&lt;/code&gt; is 17.7, &lt;code&gt;mtcars[median(qsec),]&lt;/code&gt; will get row 17.7, which is rounded down to 17:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mtcars[17,]&#xA;                   mpg cyl disp  hp drat    wt  qsec vs am gear carb&#xA;Chrysler Imperial 14.7   8  440 230 3.23 5.345 17.42  0  0    3    4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[I don't know why you say 16, maybe you've filtered &lt;code&gt;mtcars&lt;/code&gt; or something, I can't reproduce your work because you've not shown your working.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what were you trying to do? Get the row where &lt;code&gt;qsec&lt;/code&gt; is the median value of its values? Well you can do that, but its pretty pointless because there's no guarantee that the median of a set of values will be one of the values. So if you try this you get nothing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; mtcars[qsec==median(qsec),]&#xA; [1] mpg  cyl  disp hp   drat wt   qsec vs   am   gear carb&#xA;&amp;lt;0 rows&amp;gt; (or 0-length row.names)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e. there are no rows where &lt;code&gt;qsec&lt;/code&gt; equals its median value.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-11-10T07:54:28.730" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15038" PostTypeId="2" ParentId="15032" CreationDate="2016-11-10T08:00:52.620" Score="12" Body="&lt;p&gt;There is no native &lt;code&gt;.dot_product&lt;/code&gt; method. However, a dot product between two vectors is just element-wise multiply summed, so the following example works:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;# Arbitrarity, we'll use placeholders and allow batch size to vary,&#xA;# but fix vector dimensions.&#xA;# You can change this as you see fit&#xA;a = tf.placeholder(tf.float32, shape=(None, 3))&#xA;b = tf.placeholder(tf.float32, shape=(None, 3))&#xA;&#xA;c = tf.reduce_sum( tf.multiply( a, b ), 1, keep_dims=True )&#xA;&#xA;with tf.Session() as session:&#xA;    print( c.eval(&#xA;        feed_dict={ a: [[1,2,3],[4,5,6]], b: [[2,3,4],[5,6,7]] }&#xA;    ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 20.]&#xA; [ 92.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-14T07:14:56.033" LastActivityDate="2017-09-14T07:14:56.033" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15039" PostTypeId="2" ParentId="12720" CreationDate="2016-11-10T08:22:41.263" Score="3" Body="&lt;p&gt;Have you tried &lt;a href=&quot;https://www.tensorflow.org/versions/r0.11/tutorials/word2vec/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; models ?. One of the main application is computing similar words, as they are very closer in the feature vector space. Thus apart from &lt;code&gt;High_school&lt;/code&gt; and &lt;code&gt;H_school&lt;/code&gt;, also &lt;code&gt;secondary_school&lt;/code&gt;, &lt;code&gt;public_school&lt;/code&gt; may also be obtained as closer words. The important thing is that a huge amount of sentences are needed for this unsupervised learning model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref: &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt;, &lt;a href=&quot;https://deeplearning4j.org/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec in java&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-10T08:22:41.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15040" PostTypeId="2" ParentId="14795" CreationDate="2016-11-10T08:34:23.683" Score="0" Body="&lt;p&gt;Both the data could be used for a better model. As your aim is to classify whether the employee will quit or continue, use each person's monthly entries and construct them(marital status, experience, daily working hours, age, salary) as features with the output quit/continue. Train a classifier with the available data and use it to predict new data.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-10T08:34:23.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15041" PostTypeId="1" CreationDate="2016-11-10T08:38:07.877" Score="2" ViewCount="1780" Body="&lt;p&gt;I'm trying to recognize strings coming from a regular language, using positive and negative samples by Recurrent Neural Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I tried to use the &lt;a href=&quot;https://sourceforge.net/p/rnnl/wiki/Home/&quot; rel=&quot;nofollow noreferrer&quot;&gt;rnnlib&lt;/a&gt; by Alex Graves, but I had some troubles in changing the architecture of the network to adapt the network to my needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an example of my training set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;str1: w1 w2 ... wn --&gt; label 1 (it is a valid string for the language)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;str2: w1 w2 ... wi --&gt; label 0 (it does not belong to the language)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where wi are drawn from the alphabet of the unknown language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, I'm trying to use RNN as a binary classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you suggest a good tutorial or reference for this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="25992" LastEditorUserId="21024" LastEditDate="2016-11-10T15:53:49.003" LastActivityDate="2017-02-08T22:01:36.227" Title="Train a RNN with strings using positive and negative samples" Tags="|machine-learning|classification|neural-network|deep-learning|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15042" PostTypeId="2" ParentId="15041" CreationDate="2016-11-10T09:04:26.640" Score="1" Body="&lt;p&gt;I think &lt;a href=&quot;http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras&lt;/a&gt; is one of the best tutorial available for this task. It does sentiment analysis on IMDB Dataset and is the simplest to understand. Also, it uses Keras framework which I use right now because of it's simplicity and it's online support and documentation available. The RNN of keras has multiple options which you can explore based on your requirement. Also, keras make it real easy to use an optimizer from a wide variety of options.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-11-10T09:04:26.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15043" PostTypeId="1" AcceptedAnswerId="15185" CreationDate="2016-11-10T12:09:42.713" Score="7" ViewCount="13977" Body="&lt;p&gt;I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used &lt;code&gt;pandas.DataFrame.hist()&lt;/code&gt; method which gave me a plot as shown below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/73mY7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/73mY7.png&quot; alt=&quot;Figure&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I want to represent the distribution for each value in a column with different color. For example, in column 1, all the values corresponding to '0' should be in red color while the values corresponding to '1' in green color and so on. How can I do this? Please help!&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="1330" LastEditDate="2022-09-07T12:46:12.683" LastActivityDate="2022-09-07T12:46:12.683" Title="Plotting different values in pandas histogram with different colors" Tags="|python|pandas|visualization|distribution|histogram|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="15045" PostTypeId="1" CreationDate="2016-11-10T14:59:31.553" Score="3" ViewCount="1830" Body="&lt;p&gt;I used XGBoost to predict company's bankruptcy, which is an extremely unbalanced dataset. Although I tried weighting method as well as parameter tuning, the best result which I could obtain is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Best Parameters: {'clf__gamma': 0.1, 'clf__scale_pos_weight': 30.736842105263158, 'clf__min_child_weight': 1, 'clf__max_depth': 9}&#xA;Best Score: 0.219278428798&#xA;Accuracy: 0.966850828729&#xA;AUC: 0.850038850039&#xA;F1 Measure: 0.4&#xA;Cohen Kappa: 0.383129792673&#xA;Precision: 0.444444444444&#xA;recall: 0.363636363636&#xA;&#xA;Confusion Matrix:&#xA;[[346   5]&#xA; [  7   4]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the confusion matrix shows my model can not identify bankrupted companies very well, which results in poor performance measures such as precision, recall, Cohen's kappa, F measure. Also, I tried  BlaggingClassifier, which is presented &lt;a href=&quot;https://github.com/silicon-valley-data-science/learning-from-imbalanced-classes/blob/master/blagging.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Really, it gives the following result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Best Parameters: {'clf__n_estimators': 64}&#xA;Best Score: 0.133676613659&#xA;Accuracy: 0.809392265193&#xA;AUC: 0.819606319606&#xA;F1 Measure: 0.188235294118&#xA;Cohen Kappa: 0.142886555487&#xA;Precision: 0.108108108108&#xA;recall: 0.727272727273&#xA;Confusion Matrix: [[285  66]&#xA; [  3   8]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As it is shown, it predicts positive classes well but it does poor on negative class ( too many false positive). &#xA;Could you please let me know how it is possible to combine these two classifier results in order to obtain a better result? A simple way to combine two classifier is to use a convex linear combination for predicted probabilities: t * p1 + (1 - t) * p2, where 0 &amp;lt;= t &amp;lt;= 1 and p1, p2 are predictions of the two classifiers. Then , I should search for an optimal value of t over a grid but I don't know how to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read that anomaly detection for example,one class svm and Isolation Forest could be used for extremely unbalanced data set, so could you please let me know how to do it i.e., by an example code? &#xA;In general, I would appreciate if you could let me know how to deal with this issue.&lt;/p&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2021-01-11T09:40:55.400" LastActivityDate="2021-01-11T09:40:55.400" Title="unbalanced data classification" Tags="|classification|class-imbalance|python-3.x|imbalance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15046" PostTypeId="2" ParentId="2564" CreationDate="2016-11-10T15:20:22.553" Score="1" Body="&lt;p&gt;You can make use of the text classification class for your task.first make sure how you are going to classify the websites.(i.e as sports site,health and wealth site etc).Get some training data and train them &amp;amp; Done &lt;/p&gt;&#xA;" OwnerUserId="20585" LastActivityDate="2016-11-10T15:20:22.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15047" PostTypeId="2" ParentId="14997" CreationDate="2016-11-10T15:28:20.503" Score="0" Body="&lt;p&gt;You can try the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The minimum syntax you can use is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;fit.norm &amp;lt;- fitdist(x, &quot;norm&quot;) &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;to fit the normal density function to the data &lt;strong&gt;x&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the parameters &quot;gamma&quot;, &quot;weibull&quot;, &quot;lnorm&quot; for fitting gamma, weibull and lognormal  distributions respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After doing that, you can use &lt;code&gt;plot()&lt;/code&gt; function on your object &lt;em&gt;fit.norm&lt;/em&gt; to visualize the fitted distribution ,q-q plot, p-p plot and empirical and theoretical CDF's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normal distribution and Gaussian distribution are one and the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is the output due to &lt;code&gt;plot()&lt;/code&gt; function on the object generated by &lt;code&gt;fitdist()&lt;/code&gt; function for 1000 standard normal variates.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/1AON9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1AON9.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25218" LastEditorUserId="25218" LastEditDate="2016-11-11T03:51:55.457" LastActivityDate="2016-11-11T03:51:55.457" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="15048" PostTypeId="2" ParentId="15016" CreationDate="2016-11-10T16:01:07.143" Score="1" Body="&lt;p&gt;Ok, I have found the source of confusion, thanks to @Emre for the insight.&#xA;The task talks about &lt;strong&gt;epochs&lt;/strong&gt; of training, while I was trying to solve &lt;strong&gt;one batch update&lt;/strong&gt;. I think this confusion is statistically possible enough to validate existence of this question :) &lt;/p&gt;&#xA;" OwnerUserId="3108" LastActivityDate="2016-11-10T16:01:07.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15049" PostTypeId="2" ParentId="15010" CreationDate="2016-11-10T16:41:06.347" Score="1" Body="&lt;p&gt;When we want cluster &lt;code&gt;items&lt;/code&gt; use distances as similarity measure. For example, we use Euclidean distances(square root of inner product) in k-means clustering as a similarity measure. Squared Euclidean distances are used as a similarity measure in Ward's method of clustering. However, when we want to cluster &lt;code&gt;variables&lt;/code&gt;, we use correlation(cosine) as a measure of similarity. &lt;/p&gt;&#xA;" OwnerUserId="25218" LastActivityDate="2016-11-10T16:41:06.347" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15051" PostTypeId="1" CreationDate="2016-11-10T20:33:55.523" Score="0" ViewCount="4437" Body="&lt;p&gt;I am using (and loving) Orange 3 for some projects at my school and have a question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using Python and e.g. doing a RandomForest Classification, I can easily access the feature importances by feature_importances_ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Orange 3 there seems to be no feature to access that in the visual programming interface (would be a great add-on), so I tried to write my own python script in orange...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I linked the RandomForest Classifier to the Python script widget's in_classifier and have the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(in_classifier)&#xA;importances = in_classifier.feature_importances_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is however: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;Random Forest Classification {'random_state': 0, 'n_estimators': 500, 'n_jobs': 1, 'min_samples_leaf': 1, 'max_leaf_nodes': None, 'max_features': 'auto', 'bootstrap': True, 'verbose': 0, 'max_depth': None, 'oob_score': False, 'min_samples_split': 2, 'class_weight': None, 'min_weight_fraction_leaf': 0.0, 'criterion': 'gini'}&#xA;    Traceback (most recent call last):&#xA;      File &quot;&quot;, line 1, in &#xA;      File &quot;&quot;, line 2, in &#xA;    AttributeError: 'RandomForestClassifier' object has no attribute 'feature_importances_'&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;=&gt; I don't understand whey the RandomForestClassifier Object doesn't provide this array?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;many thanks!!&lt;/p&gt;&#xA;" OwnerUserId="26032" LastActivityDate="2017-03-11T23:52:40.083" Title="Orange 3 - Feature selection / importance" Tags="|python|scikit-learn|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15052" PostTypeId="2" ParentId="12745" CreationDate="2016-11-10T20:38:29.727" Score="1" Body="&lt;p&gt;My advice to you is to find out exactly what Data Scientist do day to day and take courses on the fundamentals of those specified activities.  Try and find articles and blog post like this one: &lt;a href=&quot;https://www.oreilly.com/ideas/what-is-hardcore-data-science-in-practice?twitter=%40bigdata&quot; rel=&quot;nofollow noreferrer&quot;&gt;What is hardcore data science—in practice?&lt;/a&gt;.  Additionally, you should strive to become an expert in a specific domain.  In other words, focus on Data Science as applied to ???, some field of study for which you have a true passion for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the article referenced above and others like it will point you in the right direction for making solid choices for CS focus.  Your minor should be something you are deeply interested in. It should be something you can apply your data science focus on. It should be something that produces data products.  Products which you can use to form a portfolio.  A portfolio which you can use to demonstrate to employers that you know what data science is and how it is practiced.  This last point is paramount, as you will not have a vetted program, form your university, that you can point to, which indicates you have picked up the essentials.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-11-10T20:38:29.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15053" PostTypeId="1" CreationDate="2016-11-10T23:50:52.223" Score="3" ViewCount="289" Body="&lt;p&gt;I have a bunch of compressed text files each line of which containing a JSON object. Simplified my workflow looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;string_json = sc.textFile('/folder/with/gzip/textfiles/')&#xA;json_objects = string_json.map(make_a_json)&#xA;DataRDD = json_objects.map(extract_data_from_json)&#xA;DataDF = sqlContext.createDataFrame(DataRDD,schema).collect()&#xA;'''followed by some transformations to the dataframe'''&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, the code works fine. The problem arises as soon as the number files can not be evenly divided between executors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is as far as I understand it, because spark is not extracting the files and then distributing the rows to the executors, but rather each executioner gets one file to work with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g If i have 5 files and 4 executors, the first 4 files are processed in parallel and then the 5th file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the 5th is not being processed in parallel with the other 4 and cannot be divided between the 4 executors, it takes the same amount of time as the first 4 together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This happens at every stages of the program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to transform this kind compartmentalized RDD either into a RDD or Dataframe that is not ?&lt;/p&gt;&#xA;" OwnerUserId="26034" LastEditorUserId="5177" LastEditDate="2016-11-11T10:05:14.767" LastActivityDate="2016-11-11T10:05:14.767" Title="RDD of gziped files to &quot;uncompressed&quot; Dataframe" Tags="|apache-spark|pyspark|" AnswerCount="0" CommentCount="2" ClosedDate="2016-12-17T18:55:10.970" ContentLicense="CC BY-SA 3.0" />
  <row Id="15054" PostTypeId="2" ParentId="14953" CreationDate="2016-11-11T02:38:29.593" Score="3" Body="&lt;p&gt;You are correct that &#xA;\begin{equation*}&#xA;p(y|\mathbf{x})=\dfrac{p(\mathbf{x},y)}{p(\mathbf{x})}.&#xA;\end{equation*}&#xA;Similarly, we can write the joint probability $p(\mathbf{x},y)$ as follows:&#xA;\begin{equation*}&#xA;p(\mathbf{x},y)=p(\mathbf{x}|y)\cdot p(y)&#xA;\end{equation*}&#xA;From the above two equations, we obtain&#xA;\begin{equation*}&#xA;p(y|\mathbf{x})=\dfrac{p(\mathbf{x}|y)\cdot p(y)}{p(\mathbf{x})}&#xA;\end{equation*}&#xA;In the context of supervised learning, the variable $y$ is used to denote the class labels, and  the vector $\mathbf{x}$ for measurement vector or feature vector. For the purpose of discussion, let us assume that the class label $y$ takes values in the set $\{1,2\}$, where $1$ denotes  &lt;code&gt;male&lt;/code&gt; and $2$ denotes &lt;code&gt;female&lt;/code&gt;. Similarly, $\mathbf{x}$ is a measurement vector on two variables say, $(x_{1},x_{2})$, where $x_{1}$ stands for &lt;code&gt;height&lt;/code&gt; and $x_{2}$ stands for &lt;code&gt;weight&lt;/code&gt; of individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p(y|\mathbf{x})$ denote the posterior density for $y$ given the observation $\mathbf{x}$. For example, $p(1|\mathbf{x})$ means that given the observation  $\mathbf{x}$, what is the probability that sample belongs to the class &lt;code&gt;males&lt;/code&gt;. Similarly, we can interpret $p(2|\mathbf{x})$.&#xA;$p(\mathbf{x}|y)$ stands for the class conditional probability density. For example, $p(\mathbf{x}|1)$ denote the probability density for males and $p(\mathbf{x}|2)$ denote the probability  density for females respectively. In supervised learning, these class conditional densities are usually known in advance. Finally $p(y)$ denote the prior probability for the class label $y$. For example, $p(1)$ denotes the probability that an individual/example selected randomly from the population is a &lt;code&gt;male&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that, the prior probabilities for the classes and class conditional densities are known, Bayes rule says, assign an observation $x_{0}$ whose class label is not known, to the class, say, &lt;code&gt;male&lt;/code&gt;, if&#xA;\begin{equation*}&#xA;p(\mathbf{x_{0}}|1)p(1)&amp;gt;p(\mathbf{x_{0}}|2)p(2).&#xA;\end{equation*}&#xA;Note that, $p(\mathbf{x})$ is not used in the classifier. Classification rules usually require class conditional densities.&lt;/p&gt;&#xA;" OwnerUserId="25218" LastEditorUserId="25218" LastEditDate="2016-11-11T03:02:19.390" LastActivityDate="2016-11-11T03:02:19.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15055" PostTypeId="1" CreationDate="2016-11-11T06:06:35.760" Score="9" ViewCount="16775" Body="&lt;p&gt;I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like:&#xA;Sample pdf file is &lt;a href=&quot;http://www.heromotocorp.com/en-in/uploads/Annual_Reports/pdf/20160826110519-pdf-303.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;FORWARD-LOOKING STATEMENTS&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In this Annual Report, we have disclosed forward-looking information to enable investors to&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;comprehend our prospects and take investment decisions. This report and other statements written and oral – that we periodically share contain&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;forward-looking statements that set out ...//Content Continued like in this format until the paragraph ends//&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;HERO&#xA;O&#xA;FOREVER&lt;/p&gt;&#xA;&lt;p&gt;Now if check out the actual report on above link the text under Notes to Account and Independent Auditor's Report follows the almost similar structure in all companies reports in India. Only the Chairman's message or Boards reports vary but usually talks about growth, performance, future, investments etc.&lt;/p&gt;&#xA;&lt;p&gt;So is there any way to extract only the paragraphs/multiple paragraphs combines into single(if continuation of same information) which contains useful information. I have searched but i find most of work on paragraph/document summarization but donot find something like extraction of actual continuous blocks of text data from documents.&#xA;Note : There is lot of noisy data(actual data in pdf but they are like page number, if company added some text in header/footer apart from title of page etc) between paragraphs which is generated while converting from pdf to text like:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Multi-line paragraph&#xA;R&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;N&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;O&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;WE W DS REMA INDS&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;BRANVANT IN M&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;RELE MARKETSDES&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;AND SS DECA&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;ACRO&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;6&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Next Multiline paragraph&lt;/p&gt;&#xA;" OwnerUserId="10992" LastEditorUserId="86339" LastEditDate="2020-08-17T15:47:08.687" LastActivityDate="2020-08-17T15:47:08.687" Title="How to extract paragraphs from text document?" Tags="|data-mining|text-mining|data-cleaning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="15056" PostTypeId="1" AcceptedAnswerId="15105" CreationDate="2016-11-11T08:28:32.077" Score="10" ViewCount="38066" Body="&lt;p&gt;I have a number of lists, such as [1,2,3,4], [2,3,4], [1,2], [2,3,4,6,8,10], whose lengths are obviously unsame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use this as input of placeholder in Tensorflow? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have tried, the following setting will raise error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;tf.constant([[1,2],[1,2,3]...],dtype=tf.int32)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess &lt;strong&gt;placeholder&lt;/strong&gt; cannot be set by the upper input of lists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any solution? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is my example. How to make it run without errors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hFiTj.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hFiTj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12936" LastEditorUserId="12936" LastEditDate="2016-11-14T07:50:45.333" LastActivityDate="2021-12-19T01:09:21.960" Title="How to use lists in Tensorflow?" Tags="|tensorflow|" AnswerCount="4" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15057" PostTypeId="2" ParentId="15055" CreationDate="2016-11-11T11:38:40.850" Score="2" Body="&lt;p&gt;It's not always possible to extract paragraphs from a pdf since sometime paragraph are split into multiple pdf frames so &lt;code&gt;pdftotext&lt;/code&gt; split them into different paragraph even if there are actually linked. Similarly some frames ends collocated even they represent different information like the menu in the example pdf.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple approach to split a text file into multiple paragraph using empty lines:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def txt2paragraph(filepath):&#xA;    with open(filepath) as f:&#xA;        lines = f.readlines()&#xA;&#xA;    paragraph = ''&#xA;    for line in lines:&#xA;        if line.isspace():  # is it an empty line?&#xA;            if paragraph:&#xA;                yield paragraph&#xA;                paragraph = ''&#xA;            else:&#xA;                continue&#xA;        else:&#xA;            paragraph += ' ' + line.strip()&#xA;    yield paragraph&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23614" LastActivityDate="2016-11-11T11:38:40.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15058" PostTypeId="1" AcceptedAnswerId="15097" CreationDate="2016-11-11T14:42:33.790" Score="2" ViewCount="1754" Body="&lt;p&gt;I have a binary classification task with imbalance between the two classes.&#xA;I want to compare SMOTE vs down sizing the majority class to the size of the minority class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained the classifier with 3-fold validation using the two methodologies:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SMOTE to increase the size of the minority class to the majority class size&lt;/li&gt;&#xA;&lt;li&gt;Downsizing the majority class to the minority class size with a random subsampling&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To test which methodology works better I trained my classifier (Random Forests) with a 3-fold Cross-Validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The confusion matrices I get from 3-fold CV seems to promote the use of SMOTE (better classification performance for the two classes). I assume that this CV can be used to choose the best methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I test the classifier on a real testing set (which was kept out and not used for training or validation) I don't see a real superiority of the SMOTE algorithm w.r.t. random subsample of the majority class. The minority class is better classified, but at the expense of the majority class performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a limitation of SMOTE algorithm or my model selection methodology (using 3-fold CV) has some flaws?&lt;/p&gt;&#xA;" OwnerUserId="133" LastEditorUserId="25849" LastEditDate="2020-10-03T08:17:13.083" LastActivityDate="2020-10-03T08:17:13.083" Title="Comparing SMOTE to down sampling the majority class in imbalanced binary classification" Tags="|classification|cross-validation|model-selection|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15059" PostTypeId="2" ParentId="15005" CreationDate="2016-11-11T14:43:16.223" Score="2" Body="&lt;p&gt;See &lt;code&gt;?reshape2::recast&lt;/code&gt;: The function &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;conveniently wraps melting and (d)casting a data frame into a single&#xA;  step.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(reshape2)&#xA;recast(df_sample, Countries+variable~Type, id.var=1:2)&#xA;#   Countries variable  a    b&#xA;# 1 Australia    X2014 30  -40&#xA;# 2 Australia    X2015 50  -10&#xA;# 3 Australia    X2016 10 -100&#xA;# 4 Australia    X2017  5  -10&#xA;# 5       USA    X2014 10  -20&#xA;# 6       USA    X2015 20  -40&#xA;# 7       USA    X2016 15  -10&#xA;# 8       USA    X2017  5   -5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, it's just a shortcut for these two steps:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(tmp &amp;lt;- melt(df_sample, id.vars=1:2))&#xA;#    Countries Type variable value&#xA;# 1        USA    a    X2014    10&#xA;# 2        USA    b    X2014   -20&#xA;# 3  Australia    a    X2014    30&#xA;# 4  Australia    b    X2014   -40&#xA;# 5        USA    a    X2015    20&#xA;# ...&#xA;dcast(tmp, Countries+variable~Type)&#xA;#   Countries variable  a    b&#xA;# 1 Australia    X2014 30  -40&#xA;# 2 Australia    X2015 50  -10&#xA;# 3 Australia    X2016 10 -100&#xA;# 4 Australia    X2017  5  -10&#xA;# 5       USA    X2014 10  -20&#xA;# 6       USA    X2015 20  -40&#xA;# 7       USA    X2016 15  -10&#xA;# 8       USA    X2017  5   -5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15202" LastActivityDate="2016-11-11T14:43:16.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15062" PostTypeId="1" AcceptedAnswerId="64709" CreationDate="2016-11-11T15:52:27.770" Score="7" ViewCount="744" Body="&lt;p&gt;We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gxEz9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gxEz9.png&quot; alt=&quot;Plot a histogram of the values of the first, last, 100th and 200th feature/attribute&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Our assumption&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Experiments to test our assumption&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;First experiment:&#xA;Linear Regression with only original data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second test:&#xA;Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third test:&#xA;Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Results&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The results was that the first experiment performed worse than the rest which was expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!&lt;br&gt;&#xA;Why is that?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2019-12-12T15:12:59.203" Title="Why augmenting the training data with binary attributes works better for our dataset?" Tags="|linear-regression|data-augmentation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15063" PostTypeId="2" ParentId="14421" CreationDate="2016-11-11T15:55:45.577" Score="0" Body="&lt;p&gt;Add a fully connected layer at the end of the network and train it. After training, using a simple forward pass, you can classify your image.&#xA;image--&gt;conv layer--&gt;pool layer --&gt; fully connected layer&lt;/p&gt;&#xA;" OwnerUserId="26050" LastActivityDate="2016-11-11T15:55:45.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15066" PostTypeId="2" ParentId="15051" CreationDate="2016-11-11T21:00:03.460" Score="2" Body="&lt;p&gt;You can connect &lt;em&gt;Random Forest&lt;/em&gt; widget into &lt;em&gt;Rank&lt;/em&gt; widget, which I guess is meant for subset feature selection. See &lt;a href=&quot;http://orange.biolab.si/screenshots/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange screenshot #19&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/utS12.png&quot; alt=&quot;Model-based feature selection with Orange Rank widget&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-11-11T21:00:03.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15068" PostTypeId="1" CreationDate="2016-11-11T23:10:06.453" Score="2" ViewCount="91" Body="&lt;p&gt;There are business articles from 2014 and 2015 that the time series analysis of the big data is the next big thing. However Google returns almost no scientific articles under the query &quot;big data time series&quot;. I am sure the the research happens in this direction but &lt;strong&gt;what terms should I use to find this research?&lt;/strong&gt; Big Dependent data? Large time series? Temporal big data? Streaming big data?&lt;/p&gt;&#xA;" OwnerUserId="26058" LastEditorUserId="31513" LastEditDate="2017-05-15T19:32:40.213" LastActivityDate="2017-05-15T19:32:40.213" Title="What is the scientific term/keyword for &quot;big data time series&quot;?" Tags="|bigdata|time-series|terminology|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="15069" PostTypeId="1" CreationDate="2016-11-12T00:12:14.610" Score="0" ViewCount="2634" Body="&lt;p&gt;I got an example from this book on page 318: &lt;code&gt;Chemometrics for Pattern Recognition By Richard G. Brereton.&lt;/code&gt; The example explained that; &lt;code&gt;classes A and B are more often mistaken for each other than they are for classes C and D,..., as they may be more closely related or even can be better viewed as one single group.&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a confusion matrix and assign some values to each class. How do you know which classes get confused with each other the most? Also, what it means by closely related or eve can be better viewed as one single group? How are A and B closely related and viewed as one single group in this example? Based on the value in the confusion matrix? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/SKVLi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SKVLi.png&quot; alt=&quot;Confusion-matrix from the book&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://books.google.com/books?id=Lp5ImWw7bCAC&amp;amp;pg=PA318&amp;amp;lpg=PA318&amp;amp;dq=confusion%20matrix%20classes%20get%20confused%20with%20each%20other&amp;amp;source=bl&amp;amp;ots=zxWO4Vrqv7&amp;amp;sig=k8U8m7gJA4dmPffWIzooZnFXCrY&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwiOuLj-2p3QAhWDRCYKHehACzcQ6AEIODAE#v=onepage&amp;amp;q=confusion%20matrix%20classes%20get%20confused%20with%20each%20other&amp;amp;f=false&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chemometrics for Pattern Recognition&#xA;By Richard G. Brereton - Google Book&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26061" LastActivityDate="2017-01-30T11:38:36.263" Title="Confusion matrix - classes get confused with each other" Tags="|confusion-matrix|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15070" PostTypeId="1" AcceptedAnswerId="15083" CreationDate="2016-11-12T03:17:23.857" Score="1" ViewCount="809" Body="&lt;p&gt;I was looking through a &lt;a href=&quot;http://nbviewer.jupyter.org/github/dnouri/nolearn/blob/master/docs/notebooks/CNN_tutorial.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN tutorial&lt;/a&gt; and towards the end they refer to learning capacity and image coverage during network learning diagnostics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do those 2 terms mean in the context of a convolutional neural network?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T15:33:55.650" LastActivityDate="2021-02-16T15:33:55.650" Title="Convolutional Neural Network: learning capacity and image coverage" Tags="|machine-learning|neural-network|convolutional-neural-network|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15074" PostTypeId="2" ParentId="14983" CreationDate="2016-11-12T14:22:00.187" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://nl.ijs.si/janes/wp-content/uploads/2014/09/klimtyang04b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Enron Corpus:&#xA;A New Dataset for Email Classification Research&lt;/a&gt; paper describes the kind of data set you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper mentions the following link to download the data set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.cmu.edu/~./enron/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.cmu.edu/~./enron/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, the paper also mentions various other papers which have used smaller data sets related to email classification which may not be of much use given this larger dataset.&lt;/p&gt;&#xA;" OwnerUserId="75" LastEditorUserId="75" LastEditDate="2016-11-15T16:34:42.053" LastActivityDate="2016-11-15T16:34:42.053" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15075" PostTypeId="1" AcceptedAnswerId="22893" CreationDate="2016-11-12T14:35:25.700" Score="4" ViewCount="6186" Body="&lt;p&gt;Have some issue with understanding how to use TSFERSH-library (version 0.4.0) to forecast next N-values of particular series. Below my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    # load data train/test datasets        &#xA;    train, Y, test, YY = prepare_train_test()&#xA;    # add series ID         &#xA;    train['TS_ID'] = pd.Categorical(train['QTR_HR_START']).codes&#xA;    test['TS_ID'] = pd.Categorical(test['QTR_HR_START']).codes&#xA;    # add ordered id for concrete event of series&#xA;    for id in sorted(train['TS_ID'].unique()):&#xA;        train.ix[train.TS_ID == id, 'TIME_ORDER_ID'] =  pd.Categorical(train[train.TS_ID == id]['DATETIME']).codes&#xA;    for id in sorted(test['TS_ID'].unique()):&#xA;        test.ix[test.TS_ID == id, 'TIME_ORDER_ID'] = pd.Categorical(test[test.TS_ID == id]['DATETIME']).codes&#xA;    # perform feature extraction for my signal&#xA;    extraction_settings = FeatureExtractionSettings()&#xA;    extraction_settings.IMPUTE = impute  # Fill in Infs and NaNs&#xA;    X = extract_features(train, column_id='TS_ID', feature_extraction_settings=extraction_settings).values&#xA;    XT = extract_features(test, column_id='TS_ID', feature_extraction_settings=extraction_settings).values&#xA;&#xA;    # there should be as example &#xA;    # model = xgb.DMatrix(X, label=Y, missing=np.nan)&#xA;    # model.fit()&#xA;    # model.predict(XT)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, after line &lt;code&gt;X = extract_features(...)&lt;/code&gt; I see at debugger following results&#xA; &lt;a href=&quot;https://i.stack.imgur.com/tHkeu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tHkeu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's mean that initial &lt;code&gt;X-dataset/features&lt;/code&gt; (shape=(722,10) were transformed to shape (80, 1899). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where does '80' come from? I guess from &lt;code&gt;train.TS_ID&lt;/code&gt; comes. But my &lt;code&gt;XT&lt;/code&gt;-dataset still contains 722-rows (9 days * 80 different series per day). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can I predict for 9 days in advance? or is there only forecast for next period?&lt;/p&gt;&#xA;" OwnerUserId="14684" LastEditorUserId="14684" LastEditDate="2016-12-27T11:11:00.680" LastActivityDate="2020-08-04T10:54:19.500" Title="Use TSFRESH-library to forecast values" Tags="|python|predictive-modeling|time-series|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15076" PostTypeId="2" ParentId="15069" CreationDate="2016-11-12T16:39:20.457" Score="1" Body="&lt;p&gt;The confusion matrix as the name suggest helps to identify how many of the predicted classes are being confused with true classes. For instance, in this case, the rows indicate the predicted classes and the columns indicates the correct classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we consider the diagonal values, the predicted and the true class is the same. Hence, the diagonal values do not indicate any confusion. When we check the upper triangular part of this matrix,  the value 11 in your example indicates that 11 examples have been predicted as A when the correct values are actually B. Also, looking at (2,1)(the first value of second row) the value is 10. This implies that 10 values predicted as B actually are A. This shows that A and B are confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When two classes are closely related, it becomes harder to predict. For instance, an example I come across in my work is this. When I want to classify customer reviews, &lt;code&gt;Speed of App&lt;/code&gt; and &lt;code&gt;Speed of delivery&lt;/code&gt; classes seem to get confused because they have almost similar keywords in many of their examples. Since they both refer to speed, they are closely related and this leads to the confusion&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-11-12T16:39:20.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15077" PostTypeId="2" ParentId="15010" CreationDate="2016-11-12T16:53:49.757" Score="1" Body="&lt;p&gt;The relation between dot product and cosine is similar to the relation between covariance and correlation: one is normalized and bounded version of another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience usual dot product is better when you also care about the number of dimensions two vectors have in common (i.e. non zero values in these dimensions with the same sign). For example, it can be matchings tags or attributes. For usual texts cosine typically works better.  &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-11-12T16:53:49.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15081" PostTypeId="1" CreationDate="2016-11-12T20:44:18.730" Score="4" ViewCount="316" Body="&lt;p&gt;Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = max(0, x)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the derivative of this function is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f'(x) = \begin{cases} 0 &amp;amp;\text{if } x \leq 0\\ 1&amp;amp;\text{otherwise}\end{cases}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(ignoring that is not differentiable at $0$, as I think it is done in practice). For inputs &gt; 0 this is fine, but &lt;strong&gt;why doesn't it matter that the gradient is 0 at every point &amp;lt; 0?&lt;/strong&gt; Or does it matter? (Are there publications about this problem?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:33:46.210" LastActivityDate="2021-02-16T15:33:46.210" Title="Why do CNNs with ReLU learn that well?" Tags="|neural-network|convolutional-neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15082" PostTypeId="2" ParentId="15081" CreationDate="2016-11-12T22:03:01.133" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;ignoring that is not differentiable at 00, as I think it is done in practice&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;yes see &lt;a href=&quot;https://stats.stackexchange.com/a/222541/12359&quot;&gt; ReLUs are not differentiable at zero&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;yes see &lt;a href=&quot;https://datascience.stackexchange.com/q/5706/843&quot;&gt;What is the &amp;quot;dying ReLU&amp;quot; problem in neural networks?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-11-12T22:03:01.133" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15083" PostTypeId="2" ParentId="15070" CreationDate="2016-11-12T22:39:29.860" Score="1" Body="&lt;p&gt;You can look at &lt;a href=&quot;https://github.com/dnouri/nolearn/blob/master/nolearn/lasagne/util.py#L153-L157&quot; rel=&quot;nofollow noreferrer&quot;&gt;nolearn/lasagne/util.py&lt;/a&gt; to see how learning capacity and image coverage are computed for each layer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;real_filters = get_real_filter(layers, img_size)&#xA;receptive_fields = get_receptive_field(layers, img_size)&#xA;capacity = 100. * real_filters / receptive_fields&#xA;capacity[np.logical_not(np.isfinite(capacity))] = 1&#xA;img_coverage = 100. * receptive_fields / img_size&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Xudong Cao. &quot;A practical theory for designing very deep convolutional neural networks&quot;. 2015. &lt;a href=&quot;https://www.kaggle.com/c/datasciencebowl/forums/t/13166/happy-lantern-festival-report-and-code&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/datasciencebowl/forums/t/13166/happy-lantern-festival-report-and-code&lt;/a&gt; explains how to compute the capacity of a layer:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To quantitatively measure the learning capacity of a convolutional&#xA;  layer we define the c-value of a convolutional layer as follows.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;c-value = Real Filter Size / Receptive Field Size&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where the real filter size of a k-by-k convolutional layer is k if&#xA;  there is no down sampling, it doubles after each down sampling i.e. 2k&#xA;  after one down sampling and 4k after two down sampling etc. The&#xA;  receptive field size is defined as the maximum size of a neuron can&#xA;  see on the raw image. It grows proportionally as the convolutional&#xA;  neural network goes deep. Figure 3 shows how the receptive fields&#xA;  grows in an exemplar convolutional neural network.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hFTah.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hFTah.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is why the coverage increases as you go deeper in the network, &lt;a href=&quot;http://nbviewer.jupyter.org/github/dnouri/nolearn/blob/master/docs/notebooks/CNN_tutorial.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;e.g.&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Neural Network with 122154 learnable parameters&#xA;&#xA;## Layer information&#xA;&#xA;name        size        total    cap.Y    cap.X    cov.Y    cov.X&#xA;----------  --------  -------  -------  -------  -------  -------&#xA;input0      1x28x28       784   100.00   100.00   100.00   100.00&#xA;conv2d1     32x26x26    21632   100.00   100.00    10.71    10.71&#xA;maxpool2d2  32x13x13     5408   100.00   100.00    10.71    10.71&#xA;conv2d3     64x11x11     7744    85.71    85.71    25.00    25.00&#xA;conv2d4     64x9x9       5184    54.55    54.55    39.29    39.29&#xA;maxpool2d5  64x4x4       1024    54.55    54.55    39.29    39.29&#xA;conv2d6     96x2x2        384    63.16    63.16    67.86    67.86&#xA;maxpool2d7  96x1x1         96    63.16    63.16    67.86    67.86&#xA;dense8      64             64   100.00   100.00   100.00   100.00&#xA;dropout9    64             64   100.00   100.00   100.00   100.00&#xA;dense10     64             64   100.00   100.00   100.00   100.00&#xA;dense11     10             10   100.00   100.00   100.00   100.00&#xA;&#xA;Explanation&#xA;    X, Y:    image dimensions&#xA;    cap.:    learning capacity&#xA;    cov.:    coverage of image&#xA;    magenta: capacity too low (&amp;lt;1/6)&#xA;    cyan:    image coverage too high (&amp;gt;100%)&#xA;    red:     capacity too low and coverage too high&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="843" LastActivityDate="2016-11-12T22:39:29.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15085" PostTypeId="1" CreationDate="2016-11-12T23:53:47.347" Score="0" ViewCount="148" Body="&lt;p&gt;Most problems have a curve whereby the results improve as data are added but level off at some point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there research papers or industry results that discuss the correlation between data set size and prediction accuracy for natural &lt;a href=&quot;https://en.wikipedia.org/wiki/Language_identification&quot; rel=&quot;nofollow noreferrer&quot;&gt;language identification&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2017-08-15T18:52:36.000" Title="Diminishing returns in language identification data set size?" Tags="|dataset|nlp|language-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15086" PostTypeId="1" CreationDate="2016-11-13T05:49:37.260" Score="0" ViewCount="771" Body="&lt;p&gt;I am trying to learn relevant features in a &lt;code&gt;300*299&lt;/code&gt; training matrix by taking a random row from it as my test data and applying &lt;code&gt;sequentialfs&lt;/code&gt; on it. I have used the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; Md1=fitcdiscr(xtrain,ytrain);&#xA;&amp;gt;&amp;gt; func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));&#xA;&amp;gt;&amp;gt; learnt = sequentialfs(func,xtrain,ytrain)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;xtrain&lt;/code&gt; and &lt;code&gt;ytrain&lt;/code&gt; are &lt;code&gt;299*299&lt;/code&gt; and &lt;code&gt;299*1&lt;/code&gt; respectively. Predict will give me the predicted label for &lt;code&gt;xtest&lt;/code&gt;(which is some random row from original xtrain).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I run my code, I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error using crossval&amp;gt;evalFun (line 480)&#xA;The function '@(xtrain,ytrain,xtest,ytest)sum(ytest~=predict(Md1,xtest))' generated the following error:&#xA;X must have 299 columns.&#xA;&#xA;Error in crossval&amp;gt;getFuncVal (line 497)&#xA;funResult = evalFun(funorStr,arg(:));&#xA;&#xA;Error in crossval (line 343)&#xA;    funResult = getFuncVal(1, nData, cvp, data, funorStr, []);&#xA;&#xA;Error in sequentialfs&amp;gt;callfun (line 485)&#xA;    funResult = crossval(fun,x,other_data{:},...&#xA;&#xA;Error in sequentialfs (line 353)&#xA;                crit(k) = callfun(fun,x,other_data,cv,mcreps,ParOptions);&#xA;&#xA;Error in new (line 13)&#xA;    learnt = sequentialfs(func,xtrain,ytrain)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where did I go wrong?&lt;/p&gt;&#xA;" OwnerUserId="26083" LastActivityDate="2017-03-15T05:48:24.567" Title="Matlab feature selection" Tags="|machine-learning|feature-selection|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15087" PostTypeId="1" AcceptedAnswerId="15101" CreationDate="2016-11-13T12:38:12.973" Score="9" ViewCount="19193" Body="&lt;p&gt;I am currently struggling to understand how i should train my regression network using keras. I am not sure how I should pass my input data to the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both the input data and the output data is stored as a list of numpy arrays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each input numpy array is a matrix which has (400 rows, x columns) Each output numpy array is a matrix which has (x number of rows, 13 columns)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So input dimension is 400 and output is 13. But how do I pass each of these sets within the list to the training?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Multilayer Perceptron&#xA;model = Sequential()    # Feedforward&#xA;model.add(Dense(3, input_dim=400, output_dim=13))&#xA;model.add(Activation('tanh'))&#xA;model.add(Dense(1))&#xA;model.compile('sgd', 'mse')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that model.fit trains the model given the parameter, how it actually takes in the data seems a bit like magic to me, and how it knows that the columns of matrix A should be mapped the rows of matrix B this has to be done for the whole matrix, appended to the list.&lt;/p&gt;&#xA;" OwnerUserId="26085" LastActivityDate="2016-11-14T08:55:48.213" Title="How do i pass data into keras?" Tags="|python|regression|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15089" PostTypeId="1" CreationDate="2016-11-13T13:29:06.767" Score="1" ViewCount="2746" Body="&lt;p&gt;I use &lt;a href=&quot;https://cmusatyalab.github.io/openface/&quot; rel=&quot;nofollow noreferrer&quot;&gt;openface&lt;/a&gt; to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted features and those stored in the database).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of what I'm trying to do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use OpenFace to extract the face representation, this is a 128D feature vector.&#xA;I then calculate the euclidean distance between this vector and those stored in the database, returning the vector that has the lowest distance (less than 0.9) as a match. &#xA;If no vector stored in the database matches this criteria, I store the newly extracted features as a new entry in my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a good database engine to achieve this?&lt;/p&gt;&#xA;" OwnerUserId="25225" LastEditorUserId="25225" LastEditDate="2016-11-15T06:31:00.847" LastActivityDate="2016-11-17T19:59:22.650" Title="Database/warehouse for storing 128D feature vectors" Tags="|dataset|bigdata|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15090" PostTypeId="1" CreationDate="2016-11-13T13:46:52.710" Score="0" ViewCount="1693" Body="&lt;p&gt;XGBoost's &lt;code&gt;xgb.train()&lt;/code&gt; method takes a &lt;code&gt;learning_rates&lt;/code&gt; parameter, which can take a custom function to apply a dynamic learning rate, depending on the current training round.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently posted a &lt;a href=&quot;https://hackerz.io/JivanRoquet/VMODe7Lh1&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper explaining how I'm using it&lt;/a&gt; to both speed up training in the beginning, and making more precise towards the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there's a problem with this method: it tends to overfit on the &lt;code&gt;eval&lt;/code&gt; set because there's no cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I noticed that &lt;code&gt;xgb.cv()&lt;/code&gt; method has no &lt;code&gt;learning_rates&lt;/code&gt; parameter and therefore appears to not allow for a dynamic learning rate while doing CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a possibility to use dynamic learning rates with &lt;code&gt;xgb.cv()&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;If not, is there a rationale behind this or is this just not implemented yet?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25640" LastActivityDate="2016-11-18T21:48:06.140" Title="Dynamic learning rates in XGBoost cross-validation" Tags="|machine-learning|python|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15091" PostTypeId="1" AcceptedAnswerId="15095" CreationDate="2016-11-13T16:30:07.010" Score="3" ViewCount="815" Body="&lt;p&gt;I found a couple of sources which mention mean pooling for convolutional neural networks (CNNs) - including all lectures I had about CNNs so far - but I could not find any paper with at least 10 citations which uses mean pooling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of a paper which uses mean pooling?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:33:31.477" LastActivityDate="2021-02-16T15:33:31.477" Title="Did anybody ever use mean pooling and publish it?" Tags="|convolutional-neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15093" PostTypeId="2" ParentId="15091" CreationDate="2016-11-13T18:10:42.603" Score="3" Body="&lt;p&gt;I just found an answer myself:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Boureau, Y-Lan, Jean Ponce, and Yann LeCun. &quot;&lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/boureau-icml-10.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A theoretical analysis of feature pooling in visual recognition&lt;/a&gt;.&quot; In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 111-118. 2010.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;See also: &lt;a href=&quot;http://www.shortscience.org/paper?bibtexKey=conf/icml/BoureauPL10#martinthoma&quot; rel=&quot;nofollow noreferrer&quot;&gt;shortscience summary&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-11-13T18:50:22.420" LastActivityDate="2016-11-13T18:50:22.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15094" PostTypeId="1" CreationDate="2016-11-13T20:25:56.503" Score="2" ViewCount="140" Body="&lt;p&gt;I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a class of 12 to 24 students. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I am wondering if anyone could recommend whether the specifications each for the nodes on a high-end educational-level cluster should be the same as for any other production-level cluster ( ie. 64G-128G RAM, 24TB hard drive space, 8 cores, etc.). I believe that the dataset sizes that students would be using will range between 20MB minimum to about 0.500TB maximum, I imagine that we will ultimately be working on real problems and datasets, even if they are not exactly considered to be big data.&lt;/p&gt;&#xA;" OwnerUserId="26090" LastActivityDate="2017-05-16T11:43:50.403" Title="How many people can use a single Hadoop cluster at one time?" Tags="|data|apache-hadoop|education|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15095" PostTypeId="2" ParentId="15091" CreationDate="2016-11-13T22:06:12.910" Score="3" Body="&lt;p&gt;Sum-pooling, which is of course just a scaled version of mean pooling, has been proposed for the task of content-based image retrieval (CBIR). To my best knowledge, the first paper for this was the following, and has (according to Google Scholar) gathered 35 citations over its first year of being published:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;sup&gt;1&lt;/sup&gt;A. Babenko and V. Lempitsky: &amp;quot;Aggregating Deep Convolutional Features for Image Retrieval&amp;quot;. In &lt;em&gt;IEEE International Conference on Computer Vision (ICCV)&lt;/em&gt;, Dec. 2015, pp. 1269-1277. DOI: &lt;a href=&quot;http://dx.doi.org/10.1109/ICCV.2015.150&quot; rel=&quot;nofollow noreferrer&quot;&gt;10.1109/ICCV.2015.150&lt;/a&gt;. arXiv: &lt;a href=&quot;https://arxiv.org/abs/1510.07493&quot; rel=&quot;nofollow noreferrer&quot;&gt;1510.07493&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A very short explanation of image retrieval: For large-scale image retrieval, a new query image has to be compared to a database of thousands of images to find the most similar image. While previous work has achieved this by matching SIFT descriptors, recently CNN-based descriptors have become the state-of-the-art&lt;sup&gt;2&lt;/sup&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The basic idea when using CNNs for image retrieval is, to use a network which is pre-trained (usually on a classification task, such as ILSVRC) and use the output of a layer as the descriptor vector&lt;sup&gt;3,4&lt;/sup&gt;. Different papers propose using either one of the FC layers or one of the Conv layers as the descriptor.&lt;/p&gt;&#xA;&lt;p&gt;As the outputs of these layers are very large - too large to use as a &amp;quot;compact&amp;quot; descriptor of an image - the usual approach is to reduce the dimensionality to something in the range of 32 to 1024. This is usually done by L2-normalization → PCA Whitening → L2-normalization &lt;sup&gt;1,3,4&lt;/sup&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Now finally, I get to the sum-pooling part: Babenko and Lempitsky&lt;sup&gt;1&lt;/sup&gt; show that sum-pooling leads to a better retrieval performance than max-pooling, when the dimensionality of the resulting descriptor is reduced using PCA and Whitening. Their SPoC (Sum-Pooling of Convolutions) method outperforms other pooling/embedding methods, such as max-pooling, Fisher vectors and triangular embedding.&lt;/p&gt;&#xA;&lt;p&gt;Final words: more recent work (ICLR 2016) proposes the so-called R-MAC descriptor&lt;sup&gt;5&lt;/sup&gt;, which is the regional maximum activations of convolutions, i.e. they use a set of regions at different scales, calculate the feature vector for each region using max-pooling, and finally sum up all these regional feature vectors. This again improves over the sum-pooling proposed in &lt;sup&gt;1&lt;/sup&gt;. Still it has a kind of sum-pooling over the different regions in it.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Footnotes &amp;amp; References&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;sup&gt;2&lt;/sup&gt;If required, I can add a few citations to show this, just comment below.&lt;br/&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;sup&gt;3&lt;/sup&gt;A.S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, &amp;quot;CNN Features Off-the-Shelf: An Astounding Baseline for Recognition&amp;quot;, in &lt;em&gt;IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops&lt;/em&gt;, Jun. 2014, pp. 512–519. DOI: &lt;a href=&quot;http://dx.doi.org/10.1109/CVPRW.2014.131&quot; rel=&quot;nofollow noreferrer&quot;&gt;10.1109/CVPRW.2014.131&lt;/a&gt;. arXiv: &lt;a href=&quot;https://arxiv.org/abs/1403.6382&quot; rel=&quot;nofollow noreferrer&quot;&gt;1403.6382&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;sup&gt;4&lt;/sup&gt;A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky, &amp;quot;Neural Codes for Image Retrieval&amp;quot;, in &lt;em&gt;European Conference on Computer Vision (ECCV)&lt;/em&gt;, Sep. 2014, pp. 584–599. DOI: &lt;a href=&quot;http://dx.doi.org/10.1007/978-3-319-10590-1_38&quot; rel=&quot;nofollow noreferrer&quot;&gt;10.1007/978-3-319-10590-1_38&lt;/a&gt;. arXiv: &lt;a href=&quot;https://arxiv.org/abs/1404.1777&quot; rel=&quot;nofollow noreferrer&quot;&gt;1404.1777&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;sup&gt;5&lt;/sup&gt;G. Tolias, R. Sicre, and H. Jégou, &amp;quot;Particular object retrieval with integral max-pooling of CNN activations&amp;quot;, in &lt;em&gt;International Conference on Learning Representations (ICLR)&lt;/em&gt;, Oct. 2016, pp. 1–11. arXiv: &lt;a href=&quot;http://arxiv.org/abs/1511.05879&quot; rel=&quot;nofollow noreferrer&quot;&gt;1511.05879&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-13T22:06:12.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15096" PostTypeId="2" ParentId="13990" CreationDate="2016-11-13T22:09:24.897" Score="2" Body="&lt;p&gt;I find the proposed answer very good, using a cost sensitive approach would be the first step. Another solution to your issue would be to use some sort of Sampling to balance the two classes, like SMOTE. Have a look at &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn&quot; rel=&quot;nofollow&quot;&gt;imbalance-learn&lt;/a&gt; module. Moreover, you may try to use other Classifiers, depends to what extend you want to go. I would give a try with Naive Bayes via Kernel estimation, which is simple and quick. You will have to check. In the end if no approach seems to predict the minority class you could degrade the majority class  via under sampling.&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-11-13T22:09:24.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15097" PostTypeId="2" ParentId="15058" CreationDate="2016-11-13T22:40:59.667" Score="2" Body="&lt;p&gt;It's difficult to say without the actual data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I can tell you that SMOTE creates artificial instances, hence, when used in much expanse can &#xA;&quot;deviate&quot; from the actual minority class data. It's difficult to determine the expanse. Many factors take place, firstly the Data, then the neighbouring coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. You could try boosting using many random under Samples. Hence, instead of Random Forest you could try first Adaboost for instance were each classifier is trained on a different sub sample. &lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-11-13T22:40:59.667" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15098" PostTypeId="1" AcceptedAnswerId="15145" CreationDate="2016-11-13T22:55:30.010" Score="8" ViewCount="2449" Body="&lt;p&gt;I have trained a simple CNN (using Python + Lasagne) for a 2-class EEG classification problem, however, the network doesn't seem to learn. loss does not drop over epochs and classification accuracy doesn't drop from random guessing (50%):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EIoa4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EIoa4.png&quot; alt=&quot;training curve&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there anything wrong with the code that is causing this?&lt;/li&gt;&#xA;&lt;li&gt;Is there a better (more correct?) way to handle EEG data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EEG setup&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data is collected from participants completing a total of 1044 EEG trials. Each trial lasts 2 seconds (512 time samples), has 64 channels of EEG data, and labelled 0/1. All trials have been shuffled so as to not learn on one set of participants and test on another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to predict the label of a trial after being given the 64x512 matrix of raw EEG data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The raw input data (which I can't show here as its part of a research project) has a shape of &lt;code&gt;(1044, 1, 64, 512)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;train/validation/test splits are then created at 60/20/20%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With such a small dataset I would have thought overfitting would be a problem, but training loss doesn't seem to reflect that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Network architecture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def build_cnn(input_var=None):&#xA;    l_in = InputLayer(shape=(None, 1, 64, 512), input_var=input_var)&#xA;&#xA;    l_conv1 = Conv2DLayer(incoming = l_in, num_filters = 32, filter_size = (1, 3),&#xA;                        stride = 1, pad = 'same', W = lasagne.init.Normal(std = 0.02),&#xA;                        nonlinearity = lasagne.nonlinearities.rectify)&#xA;&#xA;    l_pool1 = Pool2DLayer(incoming = l_conv1, pool_size = (1, 2), stride = (2, 2))&#xA;&#xA;    l_fc = lasagne.layers.DenseLayer(&#xA;            lasagne.layers.dropout(l_pool1, p=.5),&#xA;            num_units=256,&#xA;            nonlinearity=lasagne.nonlinearities.rectify)&#xA;&#xA;    l_out = lasagne.layers.DenseLayer(&#xA;            lasagne.layers.dropout(l_fc, p=.5),&#xA;            num_units=2,&#xA;            nonlinearity=lasagne.nonlinearities.softmax)&#xA;&#xA;    return l_out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: I have tried adding more conv/pool layers as I thought the network wasnt deep enough to learn the categories but 1) this doesn't change the outcome I mentioned above and 2) I've seen other EEG classification code where a simple 1 conv layer network can get above random chance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Helper for creating mini batches:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def iterate_minibatches(inputs, targets, batchsize, shuffle=False):&#xA;    assert len(inputs) == len(targets)&#xA;    if shuffle:&#xA;        indices = np.arange(len(inputs))&#xA;        np.random.shuffle(indices)&#xA;    for start_idx in range(0, len(inputs) - batchsize + 1, batchsize):&#xA;        if shuffle:&#xA;            excerpt = indices[start_idx:start_idx + batchsize]&#xA;        else:&#xA;            excerpt = slice(start_idx, start_idx + batchsize)&#xA;        yield inputs[excerpt], targets[excerpt]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def main(model='cnn', batch_size=500, num_epochs=500):&#xA;    input_var = T.tensor4('inputs')&#xA;    target_var = T.ivector('targets')&#xA;&#xA;    network = build_cnn(input_var)&#xA;&#xA;    prediction = lasagne.layers.get_output(network)&#xA;    loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)&#xA;    loss = loss.mean()&#xA;&#xA;    train_acc = T.mean(T.eq(T.argmax(prediction, axis=1), target_var),&#xA;                      dtype=theano.config.floatX)&#xA;&#xA;    params = lasagne.layers.get_all_params(network, trainable=True)&#xA;&#xA;    updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate=0.01)&#xA;&#xA;    test_prediction = lasagne.layers.get_output(network, deterministic=True)&#xA;    test_loss = lasagne.objectives.categorical_crossentropy(test_prediction,&#xA;                                                            target_var)&#xA;    test_loss = test_loss.mean()&#xA;&#xA;    test_acc = T.mean(T.eq(T.argmax(test_prediction, axis=1), target_var),&#xA;                      dtype=theano.config.floatX)&#xA;&#xA;    train_fn = theano.function([input_var, target_var], [loss, train_acc], updates=updates)&#xA;&#xA;    val_fn = theano.function([input_var, target_var], [test_loss, test_acc])&#xA;&#xA;    print(&quot;Starting training...&quot;)&#xA;&#xA;    for epoch in range(num_epochs):&#xA;        # full pass over the training data:&#xA;        train_err = 0&#xA;        train_acc = 0&#xA;        train_batches = 0&#xA;        start_time = time.time()&#xA;        for batch in iterate_minibatches(train_data, train_labels, batch_size, shuffle=True):&#xA;            inputs, targets = batch&#xA;            err, acc = train_fn(inputs, targets)&#xA;            train_err += err&#xA;            train_acc += acc&#xA;            train_batches += 1&#xA;&#xA;        # full pass over the validation data:&#xA;        val_err = 0&#xA;        val_acc = 0&#xA;        val_batches = 0&#xA;        for batch in iterate_minibatches(val_data, val_labels, batch_size, shuffle=False):&#xA;            inputs, targets = batch&#xA;            err, acc = val_fn(inputs, targets)&#xA;            val_err += err&#xA;            val_acc += acc&#xA;            val_batches += 1&#xA;&#xA;    # After training, compute the test predictions/error:&#xA;    test_err = 0&#xA;    test_acc = 0&#xA;    test_batches = 0&#xA;    for batch in iterate_minibatches(test_data, test_labels, batch_size, shuffle=False):&#xA;        inputs, targets = batch&#xA;        err, acc = val_fn(inputs, targets)&#xA;        test_err += err&#xA;        test_acc += acc&#xA;        test_batches += 1&#xA;&#xA;# Run the model&#xA;main(batch_size=5, num_epochs=30)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T15:33:21.167" LastActivityDate="2021-02-16T15:33:21.167" Title="Convolutional Neural Network not learning EEG data" Tags="|machine-learning|python|neural-network|convolutional-neural-network|theano|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15099" PostTypeId="2" ParentId="15045" CreationDate="2016-11-13T23:03:54.563" Score="1" Body="&lt;p&gt;For starters, I am not confident that combining the results will give you what you expect. Have you checked if the true negative remain the same in both occurrences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, have you tried to alternate the hyper parameters on the XGBoost model responsible for class balancing like max delta step or scale pos weight. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try to use Sampling techniques for Over Sampling and Under Sampling. Or try possibly using many trees on  XGBoost on a different random Under Sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I am interested on the anomaly detection paper you mentioned, could you provide a link, thnx.&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2016-11-13T23:03:54.563" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15100" PostTypeId="1" CreationDate="2016-11-14T00:57:11.477" Score="2" ViewCount="645" Body="&lt;p&gt;I just looked at the PCA in scikit-learn, but I didn't find a way to evaluate the quality of representation for variables and individuals like I usually do using the squared cosine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The squared cosine between the vector and the projection of the individual or the variable on the axis is used to determine the quality of the representation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance with the FactoMineR package in R to get those results for the variables you can simply do : &#xA;&lt;code&gt;res.pca$var$cos2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;res.pca&lt;/code&gt; are the results of your PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how would I do something similar with scikit-learn ? &lt;/p&gt;&#xA;" OwnerUserId="26096" LastEditorUserId="26096" LastEditDate="2016-11-15T01:49:38.450" LastActivityDate="2016-11-15T18:43:11.627" Title="How to evaluate the quality of representation for variables and individuals of a PCA in scikit-learn?" Tags="|scikit-learn|pca|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15101" PostTypeId="2" ParentId="15087" CreationDate="2016-11-14T08:55:48.213" Score="10" Body="&lt;p&gt;If you look at the Keras documentation, you will observe that for Sequential model's first layers takes the required input. So for example, your first layer is Dense layer with input dimension as 400. Hence each input should be a numpy array of size 400. You can pass a 2D numpy array with size (x,400). (I assume that x is the number of input examples). Your output has 13 dimensions for each output document and hence your final Dense layer has to be &lt;code&gt;model.add(Dense(13))&lt;/code&gt; instead of model.add(Dense(1)). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to how it knows, it takes the first dimension of X as number of training examples and second dimension as the size of each example. Similarly for output layer.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2016-11-14T08:55:48.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15105" PostTypeId="2" ParentId="15056" CreationDate="2016-11-14T13:09:32.440" Score="7" Body="&lt;p&gt;When you create a Numpy array like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The internal Numpy dtype is &quot;object&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[1, 2], [4, 5, 6], [1, 2, 3, 4, 5, 6]], dtype=object)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this cannot be used as a Tensor in TensorFlow. In any case, Tensors must have same size in each dimension, they cannot be &quot;ragged&quot; and must have a shape defined by a single number in each dimension. TensorFlow basically assumes this about all its data types. Although the designers of TensorFlow could write it in theory make it accept ragged arrays and include a conversion function, that kind of auto-casting is not always a good idea, because it might hide a problem in the input code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you need to pad the input data to make it a usable shape. On a quick search, I found &lt;a href=&quot;https://stackoverflow.com/questions/32037893/numpy-fix-array-with-rows-of-different-lengths-by-filling-the-empty-elements-wi&quot;&gt;this approach in Stack Overflow&lt;/a&gt;, replicated as a change to your code:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;&#xA;x = tf.placeholder( tf.int32, [3,None] )&#xA;y = x * 2&#xA;&#xA;with tf.Session() as session:&#xA;    x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]] )&#xA;&#xA;    # Get lengths of each row of data&#xA;    lens = np.array([len(x_data[i]) for i in range(len(x_data))])&#xA;&#xA;    # Mask of valid places in each row&#xA;    mask = np.arange(lens.max()) &amp;lt; lens[:,None]&#xA;&#xA;    # Setup output array and put elements from data into masked positions&#xA;    padded = np.zeros(mask.shape)&#xA;    padded[mask] = np.hstack((x_data[:]))&#xA;&#xA;    # Call TensorFlow&#xA;    result = session.run(y, feed_dict={x:padded})&#xA;&#xA;    # Remove the padding - the list function ensures we &#xA;    # create same datatype as input. It is not necessary in the case&#xA;    # where you are happy with a list of Numpy arrays instead&#xA;    result_without_padding = np.array(&#xA;       [list(result[i,0:lens[i]]) for i in range(lens.size)]&#xA;    )&#xA;    print( result_without_padding )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[2, 4] [8, 10, 12] [2, 4, 6, 8, 10, 12]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You don't have to remove the padding at the end - only do this if you require to show your output in the same ragged array format. Also note that when you feed the resulting &lt;code&gt;padded&lt;/code&gt; data to more complex routines, the zeros - or other padding data if you change it - may get used by whatever algorithm you have implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have many short arrays and just one or two very long ones, then you might want to consider using a &lt;a href=&quot;https://www.tensorflow.org/versions/r0.11/api_docs/python/sparse_ops.html&quot; rel=&quot;noreferrer&quot;&gt;sparse tensor representation&lt;/a&gt; to save memory and speed up calculations.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-11-14T13:09:32.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15106" PostTypeId="2" ParentId="15098" CreationDate="2016-11-14T14:15:12.480" Score="1" Body="&lt;p&gt;1.Your input layer seems off, the first dimension is for channels, please try, with the data formatted correctly : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l_in = InputLayer(shape=(None, 64, 512, 1 ), input_var=input_var)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A more clean way would be to replace the conv2dlayer by a conv1dLayer, which is what you are replicating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.There is no correct way to handle eeg. But people often also use spectrograms and feature extraction&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-11-14T14:15:12.480" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15108" PostTypeId="2" ParentId="9532" CreationDate="2016-11-14T16:06:06.620" Score="0" Body="&lt;p&gt;When I run my implementation of MNIST digit recognition I even get those spikes for a sigmoid transfer function. You have to question yourself: how bad are those spikes? I presume you use stochastic gradient descent right. For every iteration you consider a batch (a subspace of the samples) and train on that. Some of those batches contain very difficult pictures and your network will fail at them, resulting in a high cost. This in turn will lead to a temporary high adaptations of your weights and biases. You see that it drops again quite quickly because the other samples will again send it in the right direction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you have more spikes because your ReLU is just better at generalization and thus after a while you will get more peaks (but lower) because the network gets better and better. And in the end the most important criteria is your accuracy and not the cost ;). This has less problems with the spikes because it takes into account all the samples instead of a batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concluding: stochastic gradient descent will go to a low point, but because of it random walk it might bump into some peaks, but in the end it will find a nice low cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still have some questions, feel free to ask, I am still quite new here and still have to learn how to give good answers. &lt;/p&gt;&#xA;" OwnerUserId="25104" LastActivityDate="2016-11-14T16:06:06.620" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15110" PostTypeId="1" AcceptedAnswerId="17010" CreationDate="2016-11-14T17:35:28.093" Score="8" ViewCount="9307" Body="&lt;p&gt;I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast normalization we use this equation:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\mathsf{X}^{\prime}_{i,j,k}=s\frac{\mathsf{X}_{i,j,k}-\overline{\mathsf{X}}}{max\left\lbrace \epsilon, \sqrt{\lambda+\frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3}(\mathsf{X}_{i,j,k}-\overline{\mathsf{X}})^2}\right\rbrace }$$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$\mathsf{X}_{i,j,k}$&lt;/span&gt; is a tensor of the image and &lt;span class=&quot;math-container&quot;&gt;$\mathsf{X}^{\prime}_{i,j,k}$&lt;/span&gt; is a tensor of the normalized image, and &lt;span class=&quot;math-container&quot;&gt;$\overline{\mathsf{X}} = \frac{1}{3rc}\sum_{i=1}^{r}\sum_{j=1}^{c}\sum_{k=1}^{3} \mathsf{X}_{i,j,k}$&lt;/span&gt; is the average value of the pixels of the original image. &lt;span class=&quot;math-container&quot;&gt;$\epsilon$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt; are some constants, usually with &lt;span class=&quot;math-container&quot;&gt;$\lambda=10$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\epsilon$&lt;/span&gt; set to be a very small number, and here is my implementation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import Image&#xA;import numpy as np&#xA;import math&#xA;def global_contrast_normalization(filename, s, lmda, epsilon):&#xA;    X = np.array(Image.open(filename))&#xA;&#xA;    X_prime=X&#xA;    r,c,u=X.shape&#xA;    contrast =0&#xA;    su=0&#xA;    sum_x=0&#xA;&#xA;    for i in range(r):&#xA;        for j in range(c):&#xA;            for k in range(u):&#xA;&#xA;                sum_x=sum_x+X[i][j][k]&#xA;    X_average=float(sum_x)/(r*c*u)&#xA;&#xA;    for i in range(r):&#xA;        for j in range(c):&#xA;            for k in range(u):&#xA;&#xA;                su=su+((X[i][j][k])-X_average)**2&#xA;    contrast=np.sqrt(lmda+(float(su)/(r*c*u)))&#xA;&#xA;&#xA;    for i in range(r):&#xA;        for j in range(c):&#xA;            for k in range(u):&#xA;&#xA;                X_prime[i][j][k] = s * (X[i][j][k] - X_average) / max(epsilon, contrast)&#xA;    Image.fromarray(X_prime).save(&quot;result.jpg&quot;)&#xA;global_contrast_normalization(&quot;cat.jpg&quot;, 1, 10, 0.000000001)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;original image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DPlkE.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DPlkE.jpg&quot; alt=&quot;original image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;result image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RqgqB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RqgqB.jpg&quot; alt=&quot;result&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got an unexpected result. What is wrong with my implementation?&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="94792" LastEditDate="2020-04-13T23:35:46.900" LastActivityDate="2020-04-13T23:35:46.900" Title="How to implement global contrast normalization in python?" Tags="|python|image-classification|computer-vision|preprocessing|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="15111" PostTypeId="2" ParentId="6910" CreationDate="2016-11-14T17:51:33.400" Score="0" Body="&lt;p&gt;Whether or not all tractable problems can be parallelized efficiently (efficiently = achieve a gain in speed) is an open problem in computer science. Assuming that AC != P then not all algorithms can be Map Reduced in a way that results in a speed gain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/NC_(complexity)&quot; rel=&quot;nofollow noreferrer&quot;&gt;NC complexity Wikipedia page&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26121" LastActivityDate="2016-11-14T17:51:33.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15112" PostTypeId="1" AcceptedAnswerId="15117" CreationDate="2016-11-14T19:16:26.197" Score="2" ViewCount="483" Body="&lt;p&gt;I'm having some thoughts on whether I should remove the outliers. I'm trying to find the tags that are commonly used together. Imagine that I have the following dataset. The first column is the Tag_ID and the second column is the Number of People that used that Tag. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1   3472034  &#xA;2   1277918  &#xA;3   1249839  &#xA;4   1010770  &#xA;5   915099  &#xA;6   898292  &#xA;7   636792  &#xA;8   604352  &#xA;9   555673  &#xA;10  298495  &#xA;11  291511  &#xA;12  211074  &#xA;13  200868  &#xA;... &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(This was copied from my actual dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Should I remove a Tag instance when it is much more frequent than the other? Is that regarded as a good practice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="24719" LastEditorUserId="11097" LastEditDate="2016-11-15T07:32:11.293" LastActivityDate="2016-11-15T07:32:11.293" Title="Remove Outliers - Market Basket Analysis" Tags="|data-cleaning|association-rules|outlier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15113" PostTypeId="1" CreationDate="2016-11-14T20:54:07.970" Score="4" ViewCount="299" Body="&lt;p&gt;I have an event &quot;whether an item sold will be returned or not&quot; which I can predict with a certain probability based on information gathered at the time that the purchase occurs (product features, customer information, time and place, etc). So: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(Return | transaction information) = x% for a specific unit sold&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also have a historical time series of total units sold for that item, and a future forecast of sales of that item over the next few weeks. Assuming I gave the relevant transaction data for each historical sale that occurred, is there away to generate a future forecast from the return probability, so that I can state with some confidence that I will get 15% total returns on the item next week, 10% the week after etc? &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2024-03-23T01:05:15.620" Title="How can one generate future forecasts from probabilistic events?" Tags="|time-series|forecast|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15114" PostTypeId="2" ParentId="13357" CreationDate="2016-11-14T21:12:36.380" Score="2" Body="&lt;p&gt;Some flavor of evolutionary algorithm may suit your problem nicely, since:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The gradient of the objective function is unavailable or cannot be computed.&lt;/li&gt;&#xA;&lt;li&gt;The objective function itself is somewhat expensive to compute.&lt;/li&gt;&#xA;&lt;li&gt;The objective function is may have a large number of local maxima.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In short, evolutionary algorithms do optimization by generating candidate solutions and iteratively:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Checking their &quot;fitness&quot; (the value of the objective function)&lt;/li&gt;&#xA;&lt;li&gt;Selecting the best candidate solutions&lt;/li&gt;&#xA;&lt;li&gt;Generating new candidate solutions, somehow based on the best candidates from the previous iteration &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Naturally, the devil is in the details of how the best candidate solutions are selected and combined to generate new candidate solutions. This gives rise to a large number of evolutionary algorithm variants. Two popular variants are genetic algorithms and evolution strategies.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Genetic algorithms&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Standard genetic algorithm implementations require you to encode your input variables in a binary fashion. This may be fine for categorical inputs, but your input variables take real values in the interval $[-1,1]$. It would be possible to implement a genetic algorithm for your problem, but it wouldn't be ideal. For your problem, I'd take a look at... &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Evolution strategies&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;a href=&quot;http://www.scholarpedia.org/article/Evolution_strategies&quot; rel=&quot;nofollow noreferrer&quot;&gt;evolution strategies&lt;/a&gt; handle real-valued inputs by design and, after getting used to the jargon, are fairly easy to implement. You mentioned that you plan to use MATLAB for optimization. I found this &lt;a href=&quot;https://www.mathworks.com/matlabcentral/fileexchange/35801-evolution-strategies--es-&quot; rel=&quot;nofollow noreferrer&quot;&gt;toolbox&lt;/a&gt; on the FEX which implements evolution strategies, but I haven't taken it for a test drive myself. Whatever the case, I think an evolution strategy would be a good option to try, given the dynamic nature of your problem.&lt;/p&gt;&#xA;" OwnerUserId="26024" LastActivityDate="2016-11-14T21:12:36.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15115" PostTypeId="1" CreationDate="2016-11-15T00:07:05.567" Score="1" ViewCount="2108" Body="&lt;p&gt;Currently, we are working on a school project which is trying to predict the number of crimes in some area/neighbourhood. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 8 different categories for crimes and we've tried to find the correlation among those categories and now we only have 4 left. &#xA;Instead of building a model for each category, we want to predict these 4 categories simultaneously by some multi-output algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our sample size is around 27,000 for 6 years (from 2011 to 2016, 4000+ for each year). We are going to use (maybe) cross-validation to build/test our model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you please list 2-3 algorithms which already have fully or partially implemented library in Python (preferred) or R you would recommend to use with our dataset scale?&#xA;I only found scikit-learn with &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this algorithm&lt;/a&gt;. But it's for classification rather then prediction numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a intro-level ML course project, the group is not very experienced in the field and the time is limited so we don't want to implement an algorithm from scratch.&lt;/p&gt;&#xA;" OwnerUserId="26125" LastEditorUserId="26125" LastEditDate="2016-11-15T04:34:31.460" LastActivityDate="2021-03-23T07:41:31.690" Title="What are recommended methods for multi-task prediction?" Tags="|machine-learning|python|neural-network|predictive-modeling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15116" PostTypeId="1" CreationDate="2016-11-15T00:14:16.020" Score="0" ViewCount="70" Body="&lt;p&gt;Does anyone could explain in details how heap structure works in Cluster Algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am planning to code CURE in Matlab (Clustering using representatives) but, at first, the paper mentions that all points of datasets are considerered as clusters and heap helps to merge new clusters and I do not know how exactly this data structure works in this algo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your support.&lt;/p&gt;&#xA;" OwnerUserId="26126" LastActivityDate="2016-11-15T07:30:32.223" Title="Built a Heap in Cluster algorithm" Tags="|machine-learning|data-mining|data|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15117" PostTypeId="2" ParentId="15112" CreationDate="2016-11-15T01:13:30.770" Score="2" Body="&lt;p&gt;Since I cannot comment to ask for clarification, I am asking it here. What is your reason to think about removing the most frequent value in your dataset? If the second column actually represent frequency of usage, you probably should not remove it and I think it would be illogical to throw away that piece of information. Having said that, you may consider removing that tag if it is a &quot;less meaningful&quot; word (e.g. a, an etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you give a little bit more context on what you are trying to achieve?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, one way to find outlier is to look at points that lie beyond 1.5 times of inter-quartile range of the distribution i.e. for the frequency count in your data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just a quick thought, did you try clustering for finding similar tags? What are the ways you are considering to find similar tags?&lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-15T01:13:30.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15118" PostTypeId="2" ParentId="15113" CreationDate="2016-11-15T01:44:07.497" Score="1" Body="&lt;p&gt;If you have historical data on percentage of return for few weeks after the sales, I believe you can do it. It will be a multiple-output regression problem e.g. you will have to formulate the target values (i.e. output matrix) in a way so that %-return in first week will be in column one, %-return in second week will be in columns two and so on. After this, plug in all your input variables and run a multivariate multiple output regression algorithm. Such an algorithm on scikit-learn can be found (sklearn.multioutput.MultiOutputRegressor: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputRegressor.html#sklearn.multioutput.MultiOutputRegressor&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-15T01:44:07.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15119" PostTypeId="1" AcceptedAnswerId="15127" CreationDate="2016-11-15T02:14:15.533" Score="4" ViewCount="457" Body="&lt;p&gt;R supports a wide range of OOP designs like s3,s4,RC and others via packages,and it's a bit overwhelming to decide on which to use and a more basic question that I have is when and where do you use OOP  while doing  machine learning or data analytics ,can someone answer this from a data scientist/ML practitioner point of view .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aware of how OOP works in R at a superficial level ,but &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;should I invest time in learning OOP in R ?&#xA;  how practical is it from a data science point of view  ? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="12771" LastActivityDate="2016-11-15T16:54:13.437" Title="Practical use of oop in R" Tags="|machine-learning|r|data-mining|programming|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15121" PostTypeId="2" ParentId="15115" CreationDate="2016-11-15T03:45:42.843" Score="1" Body="&lt;p&gt;For the project, do you need to explain exactly how the input variables gave you the predictions? If so, trees based methods (R library 'tree', 'rpart') and Logistic Regression would be easy to explain to your stakeholder / lecturer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this explanation is not that important, or you could get by just showing the relative predictor importance, and you'd like to focus on accuracy. XGBoost is my default recommendation for multi-class predictions that I use at work on a daily basis. If XGBoost has too many parameters for your liking, you could try randomForest which only has 2 main parameters to tune. Both XGBoost and randomForest have functions for variable importance measures. (R Library XGBoost and randomForest)&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-15T03:45:42.843" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15122" PostTypeId="2" ParentId="15086" CreationDate="2016-11-15T04:13:36.447" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The function &quot;func&quot; is called by sequentialfs, which does not pass &quot;Md1&quot; to the &quot;predict&quot; function. Therefore, Md1 is unknown to that &quot;cost&quot; function. You should include &quot;Md1=fitcdiscr(xtrain,ytrain);&quot; within your &quot;func&quot; function.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beside, &quot;sequentialfs&quot; does a 10-fold CV. Why are you trying to pass only 299 samples instead of all 300? &lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-15T04:13:36.447" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="15123" PostTypeId="2" ParentId="12109" CreationDate="2016-11-15T05:22:13.097" Score="1" Body="&lt;p&gt;You may be able to overcome the built-in memory limit with the tds library, which lets you build more-or-less equivalent structures that aren't limited by Lua's memory cap. This won't help with the limits of your hardware, but you will be able to have things like tables that are bigger than 2 GB. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/torch/tds&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/torch/tds&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if all else fails, you could consider partitioning your vocabulary into smaller tables. Then, when you need to access it, you'd have some sort of master table in which you'd look up the correct vocab table for the element you're looking for. This would require sorting your vocabulary, so you'd still have to have it all in memory at once (or implement your own crafty sorting algorithm, I guess), but you'd only need to do that once for as long as your vocabulary stays constant. Then you serialize all the vocab tables and load them off disk as-needed, which will be slow, but still faster than filling up your physical memory and eating up swap space. Probably.&lt;/p&gt;&#xA;" OwnerUserId="26129" LastActivityDate="2016-11-15T05:22:13.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15125" PostTypeId="1" CreationDate="2016-11-15T06:00:51.170" Score="3" ViewCount="2067" Body="&lt;p&gt;I have &lt;code&gt;1100&lt;/code&gt; sequences for 2 classes. Of them &lt;code&gt;400&lt;/code&gt; are from one &lt;code&gt;class 1&lt;/code&gt; and &lt;code&gt;700&lt;/code&gt; are from &lt;code&gt;class 2&lt;/code&gt;. I have used an one hidden layer auto-encoder of &lt;code&gt;2&lt;/code&gt; neurons to capture my features. My initial features are tri-grams for each sequences. So, for each sequence I have &lt;code&gt;6860&lt;/code&gt; tri-grams. As a result most of my input vectors are sparse vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if I calculate the parameters for this network, I have&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;6860 * 2 = 13720 paramters (1st layer)&#xA;2 * 6860 = 13720 parameters (2nd layer)&#xA;-----------------------------------------&#xA;           27440 parameters (in total)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, that is way too many parameters in comparison to my number of data points. Hence, I have used a dropout value of &lt;code&gt;0.98&lt;/code&gt;, on &lt;code&gt;layer 1-&amp;gt;hidden layer&lt;/code&gt; as well as &lt;code&gt;hidden layer-&amp;gt;output layer&lt;/code&gt; which makes the number of parameters &lt;code&gt;13720 * 0.02 =  274&lt;/code&gt; on each layer and in total &lt;code&gt;548&lt;/code&gt; parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, after training, I tried the encoder on my test data of &lt;code&gt;500&lt;/code&gt; sequences, and extracted the hidden layer of 2 dimensional data. Then I use that data on another 5 neuron single hidden layer neural network to classify. My results are really good in that I am getting around &lt;code&gt;90%&lt;/code&gt; accuracy on my test data which is around the same ball park of the training data accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is am I overfitting in my autoencoder? Am I overfitting by using another neural network? I am worried about my low number of data points. Does my use of dropout seem sensible? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am worried that I am calculating the parameter numbers wrong, and my network has more parameters than number of data samples even after using Dropout. Does my calculation seem correct?&lt;/p&gt;&#xA;" OwnerUserId="13391" LastActivityDate="2017-03-15T19:03:09.757" Title="Autoencoder and Neural Network Overfitting in terms of parameter number?" Tags="|machine-learning|neural-network|data-mining|deep-learning|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15126" PostTypeId="2" ParentId="15098" CreationDate="2016-11-15T06:04:26.057" Score="2" Body="&lt;p&gt;There are a lot of possible reasons why your setup might not work. However, one very good start is to &lt;strong&gt;try to overfit your model on a very small subsample&lt;/strong&gt; of your entire dataset just to see if the problem is in the code. &lt;/p&gt;&#xA;" OwnerUserId="14732" LastEditorUserId="-1" LastEditDate="2016-11-15T18:42:40.070" LastActivityDate="2016-11-15T18:42:40.070" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15127" PostTypeId="2" ParentId="15119" CreationDate="2016-11-15T07:00:19.430" Score="3" Body="&lt;p&gt;I have used &lt;code&gt;S3&lt;/code&gt;, &lt;code&gt;S4&lt;/code&gt; and &lt;code&gt;R6&lt;/code&gt; (you forgot that one in your overview ;)). I would agree with Hadley Wickham that &lt;code&gt;S3&lt;/code&gt; is &lt;a href=&quot;http://adv-r.had.co.nz/OO-essentials.html#picking-a-system&quot; rel=&quot;nofollow noreferrer&quot;&gt;sufficient for most tasks&lt;/a&gt;. However, this is only required if you start building advanced functions that operate on objects. Say for example you build a model with one function and you want to create a &lt;code&gt;summary&lt;/code&gt; and &lt;code&gt;print&lt;/code&gt; function for the object returned by your model building function. For general Data Science purposes I would say that it helps to know the systems but none of them are very good examples of real OOP. For that I would recommend to work in Python, Ruby or Java. All have been build with OOP in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regards to knowing OOP, I think it is vital for someone involved in ML. Not when you are prototyping in R or Python but definitely when you start working on production code. I think &lt;a href=&quot;https://www.quora.com/Do-researchers-in-computer-vision-and-machine-learning-write-object-oriented-code-when-experimenting-with-ideas&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Quora thread&lt;/a&gt; gives a good run down on when OOP becomes important in ML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your focus is more on statistics in R it may be of less importance.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2016-11-15T16:54:13.437" LastActivityDate="2016-11-15T16:54:13.437" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15129" PostTypeId="2" ParentId="15116" CreationDate="2016-11-15T07:30:32.223" Score="1" Body="&lt;p&gt;A heap works in clustering the same way it works outside of clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's purpose is to efficiently find the minimum or maximum of a set, remove it, then find the next.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Efficient implementations of heaps in scripting languages may be impossible. For good performance, you usually need low-level memory access to avoid copying.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-11-15T07:30:32.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15130" PostTypeId="1" AcceptedAnswerId="15161" CreationDate="2016-11-15T11:58:00.953" Score="0" ViewCount="315" Body="&lt;p&gt;I am using a Dice Coefficient based function to calculate the similarity of two strings:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def dice_coefficient(a,b):&#xA;    try:&#xA;        if not len(a) or not len(b): return 0.0&#xA;    except:&#xA;        return 0.0&#xA;    if a == b: return 1.0&#xA;    if len(a) == 1 or len(b) == 1: return 0.0&#xA;    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]&#xA;    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]&#xA;    a_bigram_list.sort()&#xA;    b_bigram_list.sort()&#xA;    lena = len(a_bigram_list)&#xA;    lenb = len(b_bigram_list)&#xA;    matches = i = j = 0&#xA;    while (i &amp;lt; lena and j &amp;lt; lenb):&#xA;        if a_bigram_list[i] == b_bigram_list[j]:&#xA;            matches += 2&#xA;            i += 1&#xA;            j += 1&#xA;        elif a_bigram_list[i] &amp;lt; b_bigram_list[j]:&#xA;            i += 1&#xA;        else:&#xA;            j += 1&#xA;    score = float(matches)/float(lena + lenb)&#xA;    return score&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I am trying to evaluate the best match out of a large possible list, and i want to use list comprehension/map/vectorize the function calls for a whole series of strings to be matched to make this computationally efficient. However, I am having difficult getting the run time into a reasonable ballpark for even medium sized series (10K-100K elements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to send two input series into/through the function, and then get the best possible match from all candidates on dflist1 against a second series: dflist2 . Ideally, but not necessarily, the return would be another series in the dflist1 dataframe return the best possible score also. I have an implementation of this working (below), but it's incredibly slow. Is it also possible to parrelelize this? I think this would be a hugely valueable problem to solve as it would perform the same function that reconcile csv currently does.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dflist1 = pd.read_csv('\\list1.csv', header = 0,encoding = &quot;ISO-8859-1&quot;)&#xA;dflist2 = pd.read_csv('\\list2.csv', header = 0,error_bad_lines=False)&#xA;dflist1['Best Match'] = 'NA'&#xA;dflist1['Best Score'] = '0'&#xA;d = []&#xA;start = time.time()&#xA;for index, row in dflist1.iterrows():&#xA;    d=[dice_coefficient(dflist1['MasterList'][index],dflist2['TargetList'][indexx]) for indexx,rows in dflist2.itertuples()]&#xA;    dflist1['Best Match'][index]=dflist2['TargetList'][d.index(max(d))]&#xA;    dflist1['Best Score'][index]=max(d)&#xA;    print('Finished '+str(index)+' out of '+str(len(dflist1.index))+' matches after '+str(round(time.time() - start))+' seconds.')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated very much!&lt;/p&gt;&#xA;" OwnerUserId="26138" LastActivityDate="2020-08-04T10:54:24.997" Title="Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)" Tags="|python|pandas|efficiency|parallel|jaccard-coefficient|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15134" PostTypeId="1" CreationDate="2016-11-15T14:48:38.350" Score="0" ViewCount="240" Body="&lt;p&gt;Word2Vec generates a feature vector for each word. I was wondering if exist something2vec that generates one feature vector for one document. In this way I could compute the distance between two documents by only comparing two feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="26145" LastActivityDate="2016-11-25T13:28:40.880" Title="is there any whatever2vec to generate one vector for one document?" Tags="|machine-learning|nlp|word2vec|word-embeddings|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15135" PostTypeId="1" AcceptedAnswerId="15136" CreationDate="2016-11-15T14:55:04.130" Score="198" ViewCount="412255" Body="&lt;p&gt;How could I randomly split a data matrix and the corresponding label vector into a &lt;code&gt;X_train&lt;/code&gt;, &lt;code&gt;X_test&lt;/code&gt;, &lt;code&gt;X_val&lt;/code&gt;, &lt;code&gt;y_train&lt;/code&gt;, &lt;code&gt;y_test&lt;/code&gt;, &lt;code&gt;y_val&lt;/code&gt; with scikit-learn?&lt;/p&gt;&#xA;&lt;p&gt;As far as I know, &lt;code&gt;sklearn.model_selection.train_test_split&lt;/code&gt; is only capable of splitting into two not into three...&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="130343" LastEditDate="2022-12-05T16:13:04.480" LastActivityDate="2023-12-07T19:03:51.640" Title="Train/Test/Validation Set Splitting in Sklearn" Tags="|machine-learning|scikit-learn|cross-validation|" AnswerCount="16" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15136" PostTypeId="2" ParentId="15135" CreationDate="2016-11-15T15:06:00.470" Score="238" Body="&lt;p&gt;You could just use &lt;code&gt;sklearn.model_selection.train_test_split&lt;/code&gt; twice. First to split to train, test and then split train again into validation and train. Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; X_train, X_test, y_train, y_test &#xA;    = train_test_split(X, y, test_size=0.2, random_state=1)&#xA;&#xA; X_train, X_val, y_train, y_val &#xA;    = train_test_split(X_train, y_train, test_size=0.25, random_state=1) # 0.25 x 0.8 = 0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26146" LastEditorUserId="-1" LastEditDate="2020-03-31T16:25:21.620" LastActivityDate="2020-03-31T16:25:21.620" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="15137" PostTypeId="1" AcceptedAnswerId="15144" CreationDate="2016-11-15T15:49:28.457" Score="3" ViewCount="5860" Body="&lt;p&gt;I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values &lt;code&gt;0.0005, 0.005, 0.05, 0.5, 5&lt;/code&gt;. Then if &lt;code&gt;0.005&lt;/code&gt; brigs the best validation result, I continue using a grid like this: &lt;code&gt;0.0015, 0.0025, 0.0035, 0.0045, 0.0055, 0.0065, 0.0075, 0.0085, 0.0095&lt;/code&gt;. And it continues..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if there's a more intelligent way to tune this parameter.&lt;/p&gt;&#xA;" OwnerUserId="4978" LastActivityDate="2019-09-16T14:57:56.757" Title="What's the best way to tune the regularization parameter in neural nets" Tags="|deep-learning|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15138" PostTypeId="1" AcceptedAnswerId="15141" CreationDate="2016-11-15T16:06:22.500" Score="0" ViewCount="523" Body="&lt;p&gt;I'm currently struggling with different Model like AR or MA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If AR(1) is expressed as: $y_t = \beta + \beta_t \times y_{t-1} + \epsilon_t $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I know what the $\beta$ 's would be? What are the dependencies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a simple example would help a lot.&lt;/p&gt;&#xA;" OwnerUserId="26149" LastEditorUserId="26149" LastEditDate="2016-11-15T16:55:49.220" LastActivityDate="2016-11-15T16:55:49.220" Title="Autoregressive (AR) models constants - Time Series Analysis" Tags="|bigdata|time-series|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15139" PostTypeId="2" ParentId="14983" CreationDate="2016-11-15T16:29:39.157" Score="2" Body="&lt;p&gt;You can download the corpus from this &lt;a href=&quot;https://www.cs.cmu.edu/~./enron/&quot; rel=&quot;nofollow noreferrer&quot;&gt;site&lt;/a&gt;. To the best of my knowledge this is the most complete email corpus available. A project to label a subset of this email corpus can be found on this &lt;a href=&quot;http://bailando.sims.berkeley.edu/enron_email.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;UC Berkley site&lt;/a&gt;. I am not sure though whether these emails have the right training labels for you.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="12384" LastEditDate="2016-11-16T10:26:51.333" LastActivityDate="2016-11-16T10:26:51.333" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15140" PostTypeId="1" CreationDate="2016-11-15T16:31:44.717" Score="0" ViewCount="295" Body="&lt;p&gt;I'm looking for articles and other reading material on handling and modeling graph-shaped data for classification systems. As I've never working with graphed base machine learning anything from novice to advanced would be good. I'm after solving graph problems, but representing data that natively fits into a graph diagram in a way that would benefit machine learning on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to experiment with predicting based on a graph of directed interactions between nodes in a system as well as incidental events occurring on one or more of the nodes (In hopes of deriving information from correlations between # of nodes similar incidents happened at, relations between nodes with similar incidents, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anything from published articles to experience tips and best practices would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="13714" LastEditorUserId="8878" LastEditDate="2017-01-12T09:18:00.293" LastActivityDate="2017-02-11T10:10:36.670" Title="How to approach graphed data for a binary classification system?" Tags="|graphs|graphical-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15141" PostTypeId="2" ParentId="15138" CreationDate="2016-11-15T16:49:21.333" Score="1" Body="&lt;p&gt;You have a slight typo in the notation of a AR(1) model. The correct signature is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_t = \beta_0 + \beta_1 \times y_{t-1}+\epsilon_t$ or $y(t) = \beta_0 + \beta_1 \times y(t-1) + \epsilon(t)$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y(t)$ and $\epsilon(t)$ are  random variables. If $y(t)$ is standard Gaussian you can estimate $\beta_{0,1}$ with a maximum likelihood estimator (MLE). If not you will need a more complex method. You can read more about it in this &lt;a href=&quot;http://www-stat.wharton.upenn.edu/~stine/stat910/lectures/12_est_arma.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article on estimating at ARMA Process&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-11-15T16:49:21.333" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15142" PostTypeId="2" ParentId="15125" CreationDate="2016-11-15T17:34:04.687" Score="1" Body="&lt;p&gt;Additional drop out layers do not reduce parameters. Drop out is applied after each sample independently. This is done to Limit overfitting. So you still have the same parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From over 6000 dimensions to only 2 seems a bit too much. But it is hard to see. You could run a PCA Analysis and use that as a starting point for the number of hidden neurons, e.g. get the dimensionality that Accounts for 75% of the Variation in your data and go from there. Also try a second layer and add some form of regularization.&lt;/p&gt;&#xA;" OwnerUserId="26146" LastActivityDate="2016-11-15T17:34:04.687" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15143" PostTypeId="2" ParentId="15089" CreationDate="2016-11-15T17:46:17.443" Score="2" Body="&lt;p&gt;If you really need to do that (I will argue it is not a good idea), with &lt;a href=&quot;https://www.postgresql.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Postgres&lt;/a&gt; you can store an &lt;a href=&quot;https://www.postgresql.org/docs/9.6/static/arrays.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;array type&lt;/a&gt; and write a stored procedure for new item insertion.  This stored procedure can do whatever distance checking you wish, such as checking the distance of the new vector against all others in the database, before storage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would argue against this design because I suspect the criteria for uniqueness* could easily change over time.  I think it'd be a better idea to store all of the vectors except exact matches.  Then, create another table which uses the definition of uniqueness.  Creating this table would be handled on the application side.  If your definition of uniqueness changes, no problem, just make a new table.  You could even compare several different definitions to see how your results are sensitive to it.  If you do it this way, &lt;a href=&quot;http://cassandra.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cassandra&lt;/a&gt; is a great database choice.  It's specifically designed for denormalized data storage (you have the same data stored in different forms or variations, so that your application gets exactly what it needs without further computation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;*In your post you stated that a similarity of &amp;lt; 0.9 would result in storing a new vector.  That's what I mean by &lt;em&gt;criteria for uniqueness&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2016-11-17T19:59:22.650" LastActivityDate="2016-11-17T19:59:22.650" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15144" PostTypeId="2" ParentId="15137" CreationDate="2016-11-15T17:49:36.377" Score="6" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperparameter_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia lists some well-known approaches&lt;/a&gt; to hyper-parameter searches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The brute-force scan/search, or a grid search across multiple parameters, is still a very common and workable approach. As is random search, just trying some variations of parameters automatically and picking the best result from cross validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A guided search by intuitive feel of what might best work is also still quite common approach, despite being characterised as &quot;optimisation by graduate descent&quot; (a pun based on the fact that a senior researcher will hand off the tuning work to their smart junior researchers who make educated guesses on hyperparameter values) - for regularisation parameters you can look at training curves and difference between training loss vs cv loss to help decide where to look. You generally want to increase regularisation if there is a large difference and decrease it if the values become close but not very good. What counts and as &quot;large difference&quot; and &quot;not very good&quot; is subjective though, and typically you only get a feel for what works after a few random attempts and immersing yourself in the problem so that you recognise good vs bad behaviour of a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some claimed &quot;smart&quot; schemes that apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian optimisation&lt;/a&gt; or similar, and even some automated services or paid tools that attempt to make these available for tuning models automatically. However, I don't see these used much in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you are trying to win a Kaggle competition, you quickly hit diminishing returns attempting to fine-tune your parameter. I've rarely found tuning the second significant digit of a meta-param like L2 loss to be worth much attention. Maybe with the exception of testing between e.g. 1.0 and 1.5 - often useful to think in terms of geometric progression, not linear, even when fine-tuning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also have to be careful that you are not simply tuning to your cross-validation set, with no meaningful or justifiable improvement for test cases or in production. You can mitigate this in part by using k-fold cross validation (which can reduce error on metrics estimates effectively by using more data to estimate them), although of course that increases time/CPU costs.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2019-09-16T14:57:56.757" LastActivityDate="2019-09-16T14:57:56.757" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15145" PostTypeId="2" ParentId="15098" CreationDate="2016-11-15T18:31:52.503" Score="2" Body="&lt;p&gt;I had the same problem when I used TensorFlow to build a self driving car. The training error for my neural nets bounced around forever and never converged on a minimum. As a sanity check I couldn't even intentionally get my models to overfit, so I knew something was definitely wrong. What worked for me was scaling my inputs. My inputs were pixel color channels between 0 and 255, so I divided all values by 255. From that point onward, my model training (and validation) error hit a minimum as expected and stopped bouncing around. I was surprised how big of a difference it made. I can't guarantee it will work for your case, but it's definitely worth trying, since it's easy to implement.   &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2016-11-15T18:31:52.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15146" PostTypeId="2" ParentId="13266" CreationDate="2016-11-15T19:28:33.867" Score="6" Body="&lt;p&gt;In my personal opinion the lecturers's attempt to &quot;make things look easy&quot; by showing &quot;magic tricks&quot; (&quot;let us now divide by the norm as if we were dealing with one-dimensional vectors AND VOILA&quot;) is rather unhelpful as it does not really explain neither the methodology nor the math nor the intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some basic, but important notions hidden behind that &quot;voila&quot; step, let me help you by writing them out explicitly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to find the distance between two parallel lines is to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pick any point $x_1$ on the first line.&lt;/li&gt;&#xA;&lt;li&gt;Pick any point $x_2$ on the second line.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Project the vector $x_1 - x_2$ on the direction, perpendicular to two lines and measure the length of the projection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5ZbEy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5ZbEy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here's an illustration, just in case: you have two lines, one point taken on each line, the vector $x_2-x_1$ connects the two points. The dotted line is a perpendicular to the two lines. The vector $v$ points in the direction of the perpendicular, and the red segment shows the distance you want to measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This red distance is exactly the length of the &lt;em&gt;projection&lt;/em&gt; of $x_2 - x_1$ onto the dotted line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point we shall need three facts: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(a) For any &lt;strong&gt;unit&lt;/strong&gt; vector $v$ the value $x^T v$ is equal to $|x|\cos \alpha$, where $\alpha$ is the angle between the vectors. Coincidentally, this is exactly the (signed) length of the projection of $x$ on the line with direction defined by $v$.&lt;/li&gt;&#xA;&lt;li&gt;(b) Any vector $v$ can be made a unit vector (i.e. vector of length 1) by dividing it with its actual length: $\frac{v}{|v|}$.&lt;/li&gt;&#xA;&lt;li&gt;(c) The equation $w^T x + b = 0$ defines a line with the &lt;em&gt;normal&lt;/em&gt; $w$. I.e. $w$ is a vector, perpendicular to the line.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We know from (c) that $w$ is perpendicular to the line, we know from (b) that $\frac{w}{|w|}$ is a unit vector, so we plug it into (a) to get that the (signed) distance between the lines must be equal to $$\frac{w^T(x_2-x_1)}{|w|}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we know that $w^T x_2 + b = -1$ and $w^Tx_1 + b = 1$ we can play a bit with the equations to get $w^T(x_2 - x_1) = -2$ and hence the (signed) distance must be $\frac{-2}{|w|}$. We can drop the sign easily. ... and voila!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is certainly not the only way to solve the problem, so for completeness sake let me show you a similar, yet still useful alternative. Here, you need to know just one fact (which I suggest you memorize just like the facts above):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(d) (Signed) distance from point $p$ to a line defined implicitly by the equation $w^T x + b = 0$ (note the zero on the right) is equal to $\frac{w^T p + b}{|w|}$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, we can compute the distance between two lines by taking a point on one of them and substituting it in (d). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $x_1$ be a point on the line $w^T x+ b = 1$. The second line has the form $w^T x + b = -1$ which is the same as $w^T x + b + 1 = 0$. The distance between the lines is therefore:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{w^T x_1 + b + 1}{|w|} = \frac{1 + 1}{|w|} = \frac{2}{|w|}.$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Yet another way to derive the necessary distance would be to note that the line $x = sw$ is perpendicular to the two lines of interest. If $x_1$ and $x_2$ are the intersection points of this perpendicular with the first and the second parallel line correspondingly, the necessary distance is just the Euclidean distance between the two points:&#xA;$$ d = |x_1 - x_2| = \sqrt{(x_1 - x_2)^T(x_1 - x_2)}. $$&#xA;I'll leave solving this to you as an exercise.&lt;/p&gt;&#xA;" OwnerUserId="14519" LastActivityDate="2016-11-15T19:28:33.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15147" PostTypeId="1" CreationDate="2016-11-15T20:50:09.410" Score="1" ViewCount="1144" Body="&lt;p&gt;I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use Markov chain model to train the data and estimate the transition matrix as shown below. Then we can predict the future movement of any single loan in probability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meanwhile, the dataset contained substantial features for a single loan, like loan amount, borrower age, income, dwelling region, bank account profile, last 90 days bank statement data, some other credit bureau data etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just get this project from the very beginning and all thoughts are rough but not accurate. I need brain storm. Thanks heaps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use the data properly to predict the next state given the current state? Any assumption should be seriously considered in this project and any advice or experiance for me? I will do it in Python and/or R.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6slj6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6slj6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2016-11-15T20:50:09.410" Title="Markov chain modelling?" Tags="|machine-learning|python|r|data-mining|markov-process|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15148" PostTypeId="1" CreationDate="2016-11-15T21:07:21.877" Score="2" ViewCount="1023" Body="&lt;p&gt;Suppose, we use the following code to generate scatter plots,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function res = plot2features(tset, f1, f2)&#xA;% Plots tset samples on a 2-dimensional diagram&#xA;%   using features f1 and f2&#xA;% tset - training set; the first column contains class label&#xA;% f1 - index of the first feature (mapped to horizontal axis)&#xA;% f2 - index of the second feature (mapped to vertical axis)&#xA;% &#xA;% res - matrix containing values of f1 and f2 features&#xA;&#xA;    % plotting parameters for different classes&#xA;    %   restriction to 8 classes seems reasonable&#xA;    pattern(1,:) = 'ks';&#xA;    pattern(2,:) = 'rd';&#xA;    pattern(3,:) = 'mv';&#xA;    pattern(4,:) = 'b^';&#xA;    pattern(5,:) = 'gs';&#xA;    pattern(6,:) = 'md';&#xA;    pattern(7,:) = 'mv';&#xA;    pattern(8,:) = 'g^';&#xA;&#xA;    res = tset(:, [f1, f2]);&#xA;&#xA;    % extraction of all unique labels used in tset&#xA;    labels = unique(tset(:,1));&#xA;&#xA;    % create diagram and switch to content preserving mode&#xA;    figure;&#xA;    hold on;&#xA;    for i=1:size(labels,1)&#xA;        idx = tset(:,1) == labels(i);&#xA;        plot(res(idx,1), res(idx,2), pattern(i,:));&#xA;    end&#xA;    hold off;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following is its usage,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; plot2features(train, 3,4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code generates the following image before removing outliers,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LuXdw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LuXdw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and following image after removing outliers,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kRkqV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kRkqV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I have the following questions,&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) &lt;strong&gt;What do the 1st image tell us about the existence of outliers?&lt;/strong&gt; I can guess that the plot at a distant position is an outlier. But, how can I find which row or column is generating outliers? according to the 1st picture, the outlier is situated at (27,375) coordinates. But, in the actual data it is situated on the &lt;code&gt;train(184:188,:)&lt;/code&gt; rows. So, why is that difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) &lt;strong&gt;What do the color codes in the second picture represent?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &lt;strong&gt;Why has the two images that much different?&lt;/strong&gt; Why only removing 4 rows bring so radical differnce?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) &lt;strong&gt;How can we analyze the existence of outliers using histograms?&lt;/strong&gt; Please, supply me any study material about outlier detection using histograms.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Suppose we have the following training and test data in our hands to be used in testing Bayes Classifier algorithm,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Training data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/open?id=0BxVqTx5Vp2gjSkRTdUdFVk55TzQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;train.txt&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Test data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/open?id=0BxVqTx5Vp2gjNU1vLXFodU9Sakk&quot; rel=&quot;nofollow noreferrer&quot;&gt;test.txt&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First column represents class. Rest of the columns represent features.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerDisplayName="user9232" LastEditorUserId="25218" LastEditDate="2016-12-18T09:38:41.533" LastActivityDate="2017-09-14T12:22:50.380" Title="Outlier detection and removal using scatter-plots and histograms" Tags="|machine-learning|matlab|naive-bayes-classifier|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15149" PostTypeId="1" AcceptedAnswerId="29637" CreationDate="2016-11-15T22:40:58.460" Score="4" ViewCount="5673" Body="&lt;p&gt;I'm trying to play around with a toy implementation of translation or text-summarization. I understand now that most people use an embedding layer before whatever model they use, which produces something like 300-dimensional vector. But what does the model output? Like for a encoder-decoder model, it's inputs are sequences of those vectors. Then what does the final layer of decoder come out with? Not like event extraction or something like that, which we classify to a small number of classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my core question is: Is the output also a 300-dimensional vector, which then I have to generate the word based on the most similar word vector? Or is it the index of a word from the original word space? In the second case, the network is classifying to more than 100000 classes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And also, is there any existing package that support this &quot;reverse embedding to word&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not seen any question alike on the site. Please mark it as duplicate if it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="26166" LastActivityDate="2022-07-31T15:03:37.313" Title="Output a word instead of a vector after word embedding?" Tags="|python|nlp|rnn|word-embeddings|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15150" PostTypeId="2" ParentId="15148" CreationDate="2016-11-15T23:49:52.183" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes. This code does nothing but to plot the points for two chosen columns and assign different color and markers to them. It does not do any outlier selection/removal. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Column 3 and 4 (which are features in your dataset) are reasonably predictive of different classes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Plot a histogram. See if there is any clear disjoint in among the bars. Investigate a little time to find a suitable bin size.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In addition, the title of your question does not match the content. Can you please update the question?&lt;/p&gt;&#xA;" OwnerUserId="26127" LastEditorUserId="26127" LastEditDate="2016-11-16T21:26:59.657" LastActivityDate="2016-11-16T21:26:59.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15151" PostTypeId="2" ParentId="15149" CreationDate="2016-11-16T02:57:57.213" Score="2" Body="&lt;p&gt;It depends on the architecture of the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of work uses the text data to classify some specific category (e.g., the IMDB movie reviews or sentiment analysis on Tweets). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other work can actually take the words of a sentence and predict the last word. Or break it into each word predicting the subsequent word, which is really what the RNN/embedding dimension is doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other's have made these even at the character level. Ultimately, it depends on how you process the data and specify your outcome. Below are some useful tutorials that should add a little more clarity.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/tutorial/lstm.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-11-16T02:57:57.213" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15152" PostTypeId="1" AcceptedAnswerId="15216" CreationDate="2016-11-16T03:25:47.093" Score="2" ViewCount="803" Body="&lt;p&gt;Say you have an organization that requires employees to participate in a Q&amp;amp;A site similar to StackOverflow - questions and answers are voted upon, selected answers get extra points, certain behaviors boost your score etc.  What we need to do is assign a rating from 1-100 to these users with even distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The behaviors that add points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ask a question [fixed]&lt;/li&gt;&#xA;&lt;li&gt;Answer a question [fixed]&lt;/li&gt;&#xA;&lt;li&gt;Receive an upvote on a question [determined by relative ranking]&lt;/li&gt;&#xA;&lt;li&gt;Receive an upvote on an answer [determined by relative ranking]&lt;/li&gt;&#xA;&lt;li&gt;Have your answer selected [determined by relative ranking]&lt;/li&gt;&#xA;&lt;li&gt;Responding to a comment, etc [fixed]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Likewise, there are behaviors that subtract points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a user with a high ranking upvotes a question asked by a lower-ranking user, more points should be awarded than the inverse situation.  Likewise if a lower-ranking user downvotes a higher-ranking user's question, the impact should be minimal compared to the inverse.  There should be a limit to this impact though so that a high-ranking user doesn't unintentionally destroy any momentum of a low-ranking user by issuing a powerful downvote.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a few challenges here: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How do we determine how many points to assign to each type of behavior, with actor/recipient relative rank taken into account?&lt;/strong&gt;&lt;br&gt;&#xA;I'm thinking we just assign a flat number to each behavior, that number decided relative to the importance of the other behaviors, and then have a variable score that can alter the score if there is a wide variance between the users.  The mechanics of this - does the score double at most? - are unclear.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How to we assign this rank?&lt;/strong&gt; This one is a little easier - I'm thinking we just order the users according to score and then split the dataset into 100 sections, assigning each &quot;chunk&quot; a number 1-100.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Should we be worried about these numbers getting &quot;very big&quot;?&lt;/strong&gt;  The scenario described above has been trivialized; actions taken by these users may happen hundreds of times per day so the scores can become very high, very quickly. Is there a way we can keep this under control while avoiding a large number of duplicate scores?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How do we define the &quot;fixed&quot; scores&lt;/strong&gt; as the total scores become very big?  Over time we may have users with hundreds of thousands of points - but the fixed-score behaviors should still reward them.  They should reward lower-ranking users more than higher-ranking users.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don't know if there are some standard practices, algorithms, or terminologies that I should be aware of when facing a problem like this - any input would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="26175" LastEditorUserId="26175" LastEditDate="2016-11-18T03:49:21.217" LastActivityDate="2016-11-18T14:08:19.683" Title="Methods / Algorithms for rank scales based on cumulative scoring" Tags="|data-mining|statistics|algorithms|distribution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15154" PostTypeId="1" AcceptedAnswerId="15156" CreationDate="2016-11-16T04:54:44.413" Score="0" ViewCount="687" Body="&lt;p&gt;My desired output is not 1-hot encoding, but&#xA;like a 10 D vector: &lt;code&gt;[1, 0, 1, 0, 1, 0, 0, 1, 1, 1]&lt;/code&gt;&#xA;and the input is like the normal MNIST data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use TensorFlow to build a model to learn this,&#xA;then which loss function should I choose?&lt;/p&gt;&#xA;" OwnerUserId="26181" LastEditorUserId="836" LastEditDate="2016-11-16T08:22:16.187" LastActivityDate="2016-11-16T08:22:16.187" Title="Loss function for classifying when more than one output can be 1 at a time" Tags="|neural-network|tensorflow|loss-function|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15156" PostTypeId="2" ParentId="15154" CreationDate="2016-11-16T07:58:45.873" Score="1" Body="&lt;p&gt;If your classes arre not mutually exlcusive, then you just have multiple sigmoid outputs (instead of softmax function as seen in example MNIST classifiers). Each output will be a separate probability that the network assigns to membership in that class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a matching loss function, in TensorFlow, you could use the built-in &lt;a href=&quot;https://www.tensorflow.org/versions/r0.11/api_docs/python/nn.html#sigmoid_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf.nn.sigmoid_cross_entropy_with_logits&lt;/a&gt; - note that it works on the logits - the inputs to the sigmoid function - for efficiency. The link explains the maths involved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will still want a sigmoid function on the output layer too, for when you read off the predictions, but you apply the loss function above to the &lt;em&gt;input&lt;/em&gt; of the sigmoid function. Note this is not a requirement of your problem, you can easily write a loss function that works from the sigmoid outputs, just the TensorFlow built-in has been written differently to get a small speed boost. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-16T08:04:43.863" LastActivityDate="2016-11-16T08:04:43.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15157" PostTypeId="1" AcceptedAnswerId="15167" CreationDate="2016-11-16T08:01:31.283" Score="0" ViewCount="395" Body="&lt;p&gt;For example,if I have a corpus with two paragraphs, does paragraph vector generate two vectors?Additionally, on Distributed Representations of Sentences and Documents (Q. Le, T. Mikolov) paper I do not understand why paragraph vectors are unique among paragraphs but the word vectors are shared. Why?&lt;/p&gt;&#xA;" OwnerUserId="26145" LastActivityDate="2016-11-16T19:50:34.037" Title="How many vectors does paragraph vector generate for each paragraph?" Tags="|neural-network|nlp|deep-learning|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15161" PostTypeId="2" ParentId="15130" CreationDate="2016-11-16T13:56:24.717" Score="0" Body="&lt;p&gt;Your function does a lot of pythonic data crunching. In these cases &lt;a href=&quot;http://numba.pydata.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;numba&lt;/a&gt; can be useful.&lt;/p&gt;&#xA;&lt;p&gt;In the below code I split your function into two: sorting and scoring. I then converted your bigrams from strings to integers (to comply with numba datatypes) and decorated the scoring subfunction with numba's @autojit.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from numba import autojit&#xA;import numpy as np&#xA;&#xA;def dice_coefficient(a,b):&#xA;    try:&#xA;        if not len(a) or not len(b): return 0.0&#xA;    except:&#xA;        return 0.0&#xA;    if a == b: return 1.0&#xA;    if len(a) == 1 or len(b) == 1: return 0.0&#xA;    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]&#xA;    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]&#xA;&#xA;    a_bigram_list.sort()&#xA;    b_bigram_list.sort()&#xA;&#xA;    lena = len(a_bigram_list)&#xA;    lenb = len(b_bigram_list)&#xA;    matches = i = j = 0&#xA;    while (i &amp;lt; lena and j &amp;lt; lenb):&#xA;        if a_bigram_list[i] == b_bigram_list[j]:&#xA;            matches += 2&#xA;            i += 1&#xA;            j += 1&#xA;        elif a_bigram_list[i] &amp;lt; b_bigram_list[j]:&#xA;            i += 1&#xA;        else:&#xA;            j += 1&#xA;    score = float(matches)/float(lena + lenb)&#xA;    return score&#xA;&#xA;&#xA;def dice_coefficient_new(a,b):&#xA;    try:&#xA;        if not len(a) or not len(b): return 0.0&#xA;    except:&#xA;        return 0.0&#xA;    if a == b: return 1.0&#xA;    if len(a) == 1 or len(b) == 1: return 0.0&#xA;&#xA;    a_bigram_list, b_bigram_list = dice_coefficient_sorting(a,b)&#xA;    score = dice_coefficient_scoring(a_bigram_list,b_bigram_list)&#xA;&#xA;    return score&#xA;&#xA;&#xA;def dice_coefficient_sorting(a,b):&#xA;&#xA;    a = np.array([ord(i) for i in a])&#xA;    b = np.array([ord(i) for i in b])&#xA;&#xA;    a_bigram_list = 256*a[:-1]+a[1:]&#xA;    b_bigram_list = 256*b[:-1]+b[1:]&#xA;&#xA;    a_bigram_list.sort()&#xA;    b_bigram_list.sort()&#xA;&#xA;    return a_bigram_list,b_bigram_list&#xA;&#xA;@autojit(nopython=True)&#xA;def dice_coefficient_scoring(a_bigram_list,b_bigram_list):&#xA;&#xA;    lena = len(a_bigram_list)&#xA;    lenb = len(b_bigram_list)&#xA;    matches = i = j = 0&#xA;    while (i &amp;lt; lena and j &amp;lt; lenb):&#xA;        if a_bigram_list[i] == b_bigram_list[j]:&#xA;            matches += 2&#xA;            i += 1&#xA;            j += 1&#xA;        elif a_bigram_list[i] &amp;lt; b_bigram_list[j]:&#xA;            i += 1&#xA;        else:&#xA;            j += 1&#xA;    score = float(matches)/float(lena + lenb)&#xA;&#xA;    return score&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Let's time it:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;N = np.power(10,5)&#xA;&#xA;a = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])&#xA;b = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])&#xA;&#xA;%timeit dice_coefficient(a,b)&#xA;%timeit dice_coefficient_new(a,b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;1 loop, best of 3: 204 ms per loop&#xA;10 loops, best of 3: 52.9 ms per loop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So for 100K elements you get a speedup of 4x!&lt;/p&gt;&#xA;&lt;p&gt;For further optimisation you could parallelise your global for loop (for example also using numba or multiprocessing).&lt;/p&gt;&#xA;&lt;p&gt;(Note: I edited my rushed first answer which didn't work)&lt;/p&gt;&#xA;" OwnerUserId="10463" LastEditorUserId="98307" LastEditDate="2020-08-04T10:54:24.997" LastActivityDate="2020-08-04T10:54:24.997" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15162" PostTypeId="2" ParentId="14568" CreationDate="2016-11-16T16:01:51.717" Score="16" Body="&lt;p&gt;&lt;strong&gt;Review of Restricted Boltzmann Machines&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A restricted Boltzmann machine (RBM) is a &lt;em&gt;generative&lt;/em&gt; model, which learns a probability distribution over the input. That means, after being trained, the RBM can generate new samples from the learned probability distribution. The probability distribution over the visible units $\mathbf{v}$ is given by&#xA;$$p(\mathbf{v} \mid \mathbf{h}) = \prod_{i=0}^V p(v_i \mid \mathbf{h}),$$&#xA;where&#xA;$$p(v_i \mid \mathbf{h}) = \sigma\left( a_i + \sum_{j=0}^H w_{ji} h_j \right)$$&#xA;and $\sigma$ is the sigmoid function, $a_i$ is the bias for the visible node $i$, and $w_{ji}$ is the weight from $h_j$ to $v_i$.&#xA;From these two equations, it follows that $p(\mathbf{v} \mid \mathbf{h})$ only depends on the hidden states $\mathbf{h}$. &#xA;That means that the information on how a visible sample $\mathbf{v}$ is generated, has to be stored in the hidden units, the weights and the biases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Using RBMs for classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using RBMs for classification tasks, you use the following idea: as the information on how your training or test data was generated is saved in the hidden units $\mathbf{h}$, you can extract these underlying factors by feeding a training sample into the visible units of the RBM, propagate it forward to the hidden units, and use this vector of hidden units as a feature vector. You don't do any backwards pass to the visible units anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This hidden vector is just a transformed version of the input data - this can not classify anything by itself. To do a classification, you would train any classifier (linear classifier, SVM, a feedforward neural network, or anything else) with the hidden vector instead of the &quot;raw&quot; training data as inputs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are building a deep belief network (DBN) - which was used to pre-train deep feed-forward neural networks in an unsupervised fashion - you would take this hidden vector and use it as the input to a new RBM, which you stack on top of it. That way, you can train the network layer-by-layer until reaching the desired size, without needing any labeled data. Finally, you'd add e.g. a softmax layer to the top, and train the whole network with backpropagation on your classification task.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="26686" LastEditDate="2016-12-01T13:31:37.047" LastActivityDate="2016-12-01T13:31:37.047" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15163" PostTypeId="1" AcceptedAnswerId="15164" CreationDate="2016-11-16T16:53:01.463" Score="3" ViewCount="1751" Body="&lt;p&gt;In neural networks and old classification methods, we usually construct an objective function to achieve dimensionality reduction. But Deep Belief Networks (DBN) with Restricted Boltzmann Machines (RBM) learn the data structure through unsupervised learning. How does it achieve dimensionality reduction without knowing the ground truth and constructing an objective function?&lt;/p&gt;&#xA;" OwnerUserId="26193" LastEditorUserId="14906" LastEditDate="2016-11-17T11:18:01.897" LastActivityDate="2018-06-11T09:23:31.220" Title="How is dimensionality reduction achieved in Deep Belief Networks with Restricted Boltzmann Machines?" Tags="|deep-learning|dimensionality-reduction|rbm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15164" PostTypeId="2" ParentId="15163" CreationDate="2016-11-16T17:25:43.013" Score="2" Body="&lt;p&gt;As you know, a deep belief network (DBN) is a stack of restricted Boltzmann machines (RBM), so let's look at the RBM: a restricted Boltzmann machines is a generative model, which means it is able to generate samples from the learned probability distribution at the visible units (the input). While training the RBM, you teach it how your input samples are distributed, and the RBM learns how it could generate such samples. It can do so by adjusting the visible and hidden biases, and the weights in between.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of the number of hidden units is completely up to you: if you choose to give it less hidden than visible units, the RBM will try to recreate the probability distribution at the input with only the number of hidden units it has. An that is already the objective: $p(\mathbf{v})$, the probability distribution at the visible units, should be as close as possible to the probability distribution of your data $p(\text{data})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that, we assign an energy function (both equations taken from &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Practical Guide to Training RBMs&lt;/a&gt; by G. Hinton)&#xA;$$E(\mathbf{v},\mathbf{h}) = -\sum_{i \in \text{visible}} a_i v_i - \sum_{j \in \text{hidden}} b_j h_j - \sum_{i,j} v_i h_j w_{ij}$$&#xA;to each configuration of visible units $\mathbf{v}$ and hidden units $\mathbf{h}$. Here, $a_i$ and $b_j$ are the biases, and $w_{ij}$ are the weights. Given this energy function, the probability of a visible vector $\mathbf{v}$ is &#xA;$$p(\mathbf{v}) = \frac 1Z \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$&#xA;With that, we know that to increase the probability of the RBM generating a training sample $\mathbf{v}^{(k)}$ (denotes the $k$-th training sample), we need to change $a_i$, $b_j$ and $w_{ij}$ so that the energy $E$ for our given $\mathbf{v}^{(k)}$ and the corresponding $\mathbf{h}$ gets lower. &lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-11-16T17:25:43.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15166" PostTypeId="1" CreationDate="2016-11-16T18:29:50.027" Score="0" ViewCount="21" Body="&lt;p&gt;I am in search of some data science conferences (for 2017) that have a mix of big name speakers and poster sessions.  Ideally, I would like to be able to present a poster that doesn't necessarily have to be a research paper, but simply the development of a tool and its use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide the name and link.  Thank you!&lt;/p&gt;&#xA;" OwnerUserId="26212" LastActivityDate="2016-11-16T23:14:40.150" Title="Data Science Conferences w/ Poster Sessions?" Tags="|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15167" PostTypeId="2" ParentId="15157" CreationDate="2016-11-16T19:50:34.037" Score="1" Body="&lt;p&gt;It would make no sense of the word embeddings to change within a document. It would be as if the spellings of the words changed; how would that help? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use a document embedding, you find its numerical representation in a way that loosely captures its meaning. If you want to capture the meaning of each paragraph separately, find their embeddings separately. If you want to capture meaning of the entire document, feed the entire document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if your document covers a range of topics and you want to allow users to pinpoint where a particular topic is covered, you can find embeddings of each section (paragraph, page, etc.), then find the section that's nearest to your query's embedding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One use case for dynamic word embeddings is to identify temporal dynamics; how the meaning of a word changes over time, as in this paper: &lt;a href=&quot;http://homepages.inf.ed.ac.uk/scohen/arxiv14dynamic.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Visualization of Change in Word Meaning over Time using Temporal Word Embeddings&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-11-16T19:50:34.037" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15172" PostTypeId="2" ParentId="14590" CreationDate="2016-11-16T22:57:55.827" Score="1" Body="&lt;p&gt;Let's look at the example of linear regression. Instead of deriving it from solving the normal equations, we can motivate it by thinking about it as finding the conditional distribution $P(y|x)$. Let's assume that this distribution follows a Gaussian distribution with fixed variance $\sigma^2$ and mean $\hat{y}(x,w)$ and weights $w$. Assuming that samples are i.i.d. we can easily show that applying maximum-likelihood gives the same values of $w$ than minimizing the mean squared error. See &lt;a href=&quot;http://www.deeplearningbook.org/contents/ml.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;example 5.5.1&lt;/a&gt; for a mathematical explanation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we additionally assume a prior Gaussian distribution for $P(w)$ (w.l.o.g. we can assume unit variance $I$), we can now show that the posterior $P(w | x,y)$  of $w$ is also Gaussian distributed (that is because the prior is &quot;conjugate&quot;) with variance $\frac{1}{\alpha} I$ and we recover linear regression with weight decay $\alpha w^T w=\alpha \lVert w \rVert^2_2$  (i.e. &lt;a href=&quot;https://en.wikipedia.org/wiki/Tikhonov_regularization&quot; rel=&quot;nofollow noreferrer&quot;&gt;$L^2$ or Tikhonov regularization&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, a Gaussian distribution's probability mass is centered around the mean (assumed here to be zero). If we assume such a distribution for the coefficients in the linear regression (as our prior knowledge), linear regression will favor coefficients close to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of this is nicely explained &lt;a href=&quot;http://www.deeplearningbook.org/contents/ml.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here (chapter 5)&lt;/a&gt; or &lt;a href=&quot;http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/181115.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here (chapter 18)&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-08-10T15:29:19.513" LastActivityDate="2017-08-10T15:29:19.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15174" PostTypeId="2" ParentId="15166" CreationDate="2016-11-16T23:14:40.150" Score="1" Body="&lt;p&gt;KDNuggets has a really nice list of conferences on their page:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.kdnuggets.com/meetings/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/meetings/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26211" LastActivityDate="2016-11-16T23:14:40.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15176" PostTypeId="2" ParentId="10643" CreationDate="2016-11-16T23:46:35.400" Score="2" Body="&lt;p&gt;If you just want to do descriptive analysis it might be a good idea to do this without imputation at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more complicated method is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You take the data without missing values from your whole dataset.&lt;/li&gt;&#xA;&lt;li&gt;Then in this dataset you manually delete data (trying to reproduce the overall missing data pattern)&lt;/li&gt;&#xA;&lt;li&gt;For this new dataset with missing values you have the real values behind the missing values (since you manually deleted the values)&lt;/li&gt;&#xA;&lt;li&gt;Then you do imputation on this dataset (with the method you would also use for the overall dataset)&lt;/li&gt;&#xA;&lt;li&gt;Then you compare your imputed values to the real values&lt;/li&gt;&#xA;&lt;li&gt;This gives a good estimate how far you could be off for the overall dataset&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26211" LastActivityDate="2016-11-16T23:46:35.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15177" PostTypeId="1" CreationDate="2016-11-16T23:48:55.903" Score="3" ViewCount="6759" Body="&lt;p&gt;I had a simple neural network that was &lt;strong&gt;outputting the same value regardless of the input&lt;/strong&gt;. During training, it was behaving normally, with training and validation loss diminishing to a floor value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data range was in [-1000, +1000].&#xA;The model is a 1D convolutional network. If it is useful : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    train_loss = lcategorical_crossentropy(...).mean()&#xA;&#xA;    net = InputLayer((None, net_z, net_x), input_var=input_var)&#xA;&#xA;    net = ConvLayer(net,  16, 9, pad='same' )#, flip_filters=False)&#xA;    net = ConvLayer(net,  16, 9, pad='same')#, flip_filters=False)&#xA;&#xA;    net = DenseLayer(net,   num_units=32)&#xA;    net = DropoutLayer(net, p=0.5)&#xA;&#xA;    net = DenseLayer(net, num_units=2, nonlinearity=None)&#xA;    net = NonlinearityLayer(net, softmax)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This was &lt;strong&gt;corrected by normalizing the data&lt;/strong&gt; in [-1, 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the reason behind that ? Backpropagation being stuck I guess, but where can I learn about that in depth ?&lt;/p&gt;&#xA;" OwnerUserId="25311" LastEditorUserId="25311" LastEditDate="2016-11-17T10:16:04.887" LastActivityDate="2016-11-17T10:16:04.887" Title="Neural Network outputing the same value / normalization" Tags="|machine-learning|neural-network|backpropagation|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15178" PostTypeId="1" CreationDate="2016-11-16T23:51:10.447" Score="2" ViewCount="170" Body="&lt;p&gt;I am searching for a software / tool / R package (any of these) that enables easy visualization of a time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have a multiple time series with 7 attributes&lt;/li&gt;&#xA;&lt;li&gt;I want to have all 7 attributes plotted beneath each other with a own y-axis&lt;/li&gt;&#xA;&lt;li&gt;I want to be able to set y-axis resolutions for the plot&lt;/li&gt;&#xA;&lt;li&gt;The time series is very long (1 year) but I want to look at each day individually and then basically scroll right to see the next day&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="26211" LastEditorUserId="31513" LastEditDate="2017-08-06T10:59:29.083" LastActivityDate="2017-08-06T10:59:29.083" Title="Tool for visualizing long time series data" Tags="|visualization|tools|software-recommendation|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15180" PostTypeId="2" ParentId="15177" CreationDate="2016-11-17T01:53:40.847" Score="1" Body="&lt;p&gt;If it works for your data, I would suggest normalization by:&#xA;input = (input-median(input)) / std(input)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also try to use a smaller batch size for initial training. And triple check how your data actually looks like. The same goes for the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the framework, you should also check the data type of the labels. Some unfortunate computation might cast your float probabilities to int, thereby killing any meaningful output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can recommend Andrew Ng's famous Coursera lectures. They give good insight into what happens within the neurons.&lt;/p&gt;&#xA;" OwnerUserId="26223" LastActivityDate="2016-11-17T01:53:40.847" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15181" PostTypeId="1" AcceptedAnswerId="36324" CreationDate="2016-11-17T02:12:24.960" Score="5" ViewCount="2701" Body="&lt;p&gt;I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I attempted:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.&lt;/li&gt;&#xA;&lt;li&gt;Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="24792" LastEditDate="2016-11-17T20:57:21.787" LastActivityDate="2018-08-01T17:31:42.080" Title="Multi-task learning, finding a loss function that &quot;ignores&quot; certain samples" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15182" PostTypeId="2" ParentId="15181" CreationDate="2016-11-17T02:55:00.570" Score="2" Body="&lt;p&gt;I seem to have a workable solution using this minor tweak to square loss:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L=(y-\hat{y})^2 \cdot \left(\frac{y}{y + \epsilon}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $y$ is the label and $\hat{y}$ is the prediction, and $\epsilon=1e-6$ is to avoid a divide by zero error (it's ~ the smallest number within 32 bit precision)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In tensorflow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epsilon = 1e-6&#xA;loss = tf.reduce_sum(&#xA;          tf.mul(tf.square(prediction - label),&#xA;                 tf.div(label, label + epsilon))&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second term is 0 when the label is 0 (e.g. loss is 0 and no gradient is added for that sample), and it'll round to 1, or near enough, when the label is non zero. As long as the range of labels does not cross 0 this should work, and that's a simple matter of scaling the labels if it's not the case by default.&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="24792" LastEditDate="2016-11-18T19:44:33.553" LastActivityDate="2016-11-18T19:44:33.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15183" PostTypeId="2" ParentId="14568" CreationDate="2016-11-17T03:23:41.550" Score="5" Body="&lt;p&gt;@hbaderts described the whole workflow perfectly. However, it may not make any sense in case you are completely new to this idea. Therefore, I am going to explain it in layman's way (therefore, I will be omitting details):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of the deep networks as a function to transform your data. Example of transformations include normalization, taking log of data etc. The deep networks you are training has multiple layers. Each of these layers are trained using some kind of learning algorithm. For the first layer, you pass the original data as the input and try to get a function that will give you back those &quot;same original data&quot; as the output. However, you don't get the perfect output. Therefore, you are getting a transformed version of your input as the output of the first layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for the second layer, you take those &quot;transformed data&quot; and pass them as input and repeat the whole learning process. You keep doing that for all the layers in your deep network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the last layer, what you get is a &quot;transformed version&quot; of your original input data. This can be thought of higher level abstraction of your original input data. Note that, you have not used the labels/output in your deep network yet. Therefore, everything till this point is unsupervised learning. This is called layer-wise pre-training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, you want to train a classifier/regression model and this is a supervised learning problem. The way you achieve that goal is by taking the &quot;final transformed version&quot; of your original input from the last layer in your deep network and use them as the input to any classifier (e.g. knn classifier / softmax classifier / logistic regression etc). This is called stacking. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are training this last-step classifier/learner, you propagate all your learning in the complete network. This ensures that your are able to learn from the labels/outputs and modify the learned layer-wise parameters accordingly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, once you have your generative model trained, take the output of your generative model and use that as input to a classifier/learner. Let the error flow through the whole network as the learning continues so that you can modify the layer-wise parameter learned in earlier steps. &lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-17T03:23:41.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15185" PostTypeId="2" ParentId="15043" CreationDate="2016-11-17T08:17:24.257" Score="9" Body="&lt;p&gt;There isn't any built-in function to do this directly in pandas, but by getting the array collection of &lt;code&gt;AxesSubplot&lt;/code&gt;, iterating on them to retrieve the matplotlib patches you can achieve the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's some dummy data to play with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;df = pd.DataFrame(np.random.randint(low=0, high=3, size=(1000,16)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, here's the magic:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;&#xA;# Plot and retrieve the axes&#xA;axes = df.hist(figsize=(12,6), sharex=True, sharey=True)&#xA;&#xA;# Define a different color for the first three bars&#xA;colors = [&quot;#e74c3c&quot;, &quot;#2ecc71&quot;, &quot;#3498db&quot;]&#xA;&#xA;for i, ax in enumerate(axes.reshape(-1)):&#xA;    # Define a counter to ensure that if we have more than three bars with a value,&#xA;    # we don't try to access out-of-range element in colors&#xA;    k = 0&#xA;&#xA;    # Optional: remove grid, and top and right spines&#xA;    ax.grid(False)&#xA;    ax.spines['top'].set_visible(False)&#xA;    ax.spines['right'].set_visible(False)&#xA;&#xA;    for rect in ax.patches:&#xA;        # If there's a value in the rect and we have defined a color&#xA;        if rect.get_height() &amp;gt; 0 and k &amp;lt; len(colors):&#xA;            # Set the color&#xA;            rect.set_color(colors[k])&#xA;            # Increment the counter&#xA;            k += 1&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4w9Ac.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4w9Ac.png&quot; alt=&quot;Resulting hist subplot with colors&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26194" LastActivityDate="2016-11-17T08:17:24.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15186" PostTypeId="2" ParentId="14069" CreationDate="2016-11-17T08:51:48.723" Score="6" Body="&lt;p&gt;First, let's create a &lt;a href=&quot;https://stackoverflow.com/help/mcve&quot;&gt;mcve&lt;/a&gt; to play with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;In [1]: categorical_array = np.random.choice(['Var1','Var2','Var3'],&#xA;                                             size=(5,3), p=[0.25,0.5,0.25])&#xA;        df = pd.DataFrame(categorical_array,&#xA;               columns=map(lambda x:chr(97+x), range(categorical_array.shape[1])))&#xA;        # Add another column that isn't categorical but float&#xA;        df['d'] = np.random.rand(len(df))&#xA;        print(df)&#xA;&#xA;Out[1]:&#xA;      a     b     c         d&#xA;0  Var3  Var3  Var3  0.953153&#xA;1  Var1  Var2  Var1  0.924896&#xA;2  Var2  Var2  Var2  0.273205&#xA;3  Var2  Var1  Var3  0.459676&#xA;4  Var2  Var1  Var1  0.114358&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we can use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;noreferrer&quot;&gt;pd.get_dummies&lt;/a&gt; to encode the first three columns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note that I'm using the &lt;code&gt;drop_first&lt;/code&gt;parameter because &lt;code&gt;N-1&lt;/code&gt; dummies are sufficient to fully describe &lt;code&gt;N&lt;/code&gt; possibilities&lt;/em&gt; (eg: if &lt;code&gt;a_Var2&lt;/code&gt; and &lt;code&gt;a_Var3&lt;/code&gt; are 0, then it's &lt;code&gt;a_Var1&lt;/code&gt;).&#xA;Also, I'm specifically specifying the columns but I don't have to as it will be columns with dtype either &lt;code&gt;object&lt;/code&gt; or &lt;code&gt;categorical&lt;/code&gt; (more below).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [2]: df_encoded = pd.get_dummies(df, columns=['a','b', 'c'], drop_first=True)&#xA;        print(df_encoded]&#xA;Out[2]:&#xA;          d  a_Var2  a_Var3  b_Var2  b_Var3  c_Var2  c_Var3&#xA;0  0.953153       0       1       0       1       0       1&#xA;1  0.924896       0       0       1       0       0       0&#xA;2  0.273205       1       0       1       0       1       0&#xA;3  0.459676       1       0       0       0       0       1&#xA;4  0.114358       1       0       0       0       0       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In your specific application, you'll have to provide a list of column that are Categorical, or you'll have to infer which columns are Categorical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best case scenario your dataframe already has these columns with a &lt;code&gt;dtype=category&lt;/code&gt; and you can pass &lt;code&gt;columns=df.columns[df.dtypes == 'category']&lt;/code&gt; to &lt;code&gt;get_dummies&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise I suggest setting the &lt;code&gt;dtype&lt;/code&gt; of all other columns as appropriate (hint: pd.to_numeric, pd.to_datetime, etc) and you'll be left with columns that have an &lt;code&gt;object&lt;/code&gt; dtype and these should be your categorical columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pd.get_dummies parameter columns defaults as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;columns : list-like, default None&#xA;    Column names in the DataFrame to be encoded.&#xA;    If `columns` is None then all the columns with&#xA;    `object` or `category` dtype will be converted.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26194" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-11-17T09:01:11.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15187" PostTypeId="1" CreationDate="2016-11-17T10:58:19.760" Score="7" ViewCount="4801" Body="&lt;p&gt;I have learnt from some examples the existence of regularization option at ANNs (concretely, at Keras implementation). As far as I know, regularization in general is a kind of &quot;penalty&quot; on parameters to prevent model complexity and overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accordingly, &lt;code&gt;W_regularizer&lt;/code&gt; and &lt;code&gt;b_regularizer&lt;/code&gt; options in Keras are for weight and bias parameter regularization, unless I am mistaken. But what is &lt;code&gt;activity_regularizer&lt;/code&gt; for? How is it related to the weight/bias regularization? And more generally: what is a good practice to using all these regularization possibilites (apart from the blind brute force tuning)? Because of ANNs/CNNs are produce very low overfitting measured on the validation set, it seems me that regularization is not a really useful tool with neural nets.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="11097" LastEditDate="2016-11-17T18:06:52.217" LastActivityDate="2016-11-17T18:06:52.217" Title="Regularization practice with ANNs" Tags="|machine-learning|neural-network|keras|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15188" PostTypeId="1" CreationDate="2016-11-17T11:18:32.383" Score="3" ViewCount="947" Body="&lt;p&gt;I currently tried to figure out which paddings are directly supported by the frameworks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow (&lt;a href=&quot;https://www.tensorflow.org/api_docs/python/nn.html#conv2d&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.nn.conv2d&lt;/code&gt;&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;padding='VALID'&lt;/code&gt;: No padding&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;padding='SAME'&lt;/code&gt;: 0-padding such that the output has the same size as the input&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By applying &lt;a href=&quot;https://www.tensorflow.org/versions/r0.9/api_docs/python/array_ops.html#pad&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.pad&lt;/code&gt;&lt;/a&gt; and then &lt;code&gt;padding='VALID'&lt;/code&gt; one can get reflect and symmetric padding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lasagne (&lt;a href=&quot;http://lasagne.readthedocs.io/en/latest/modules/layers/conv.html#lasagne.layers.Conv2DLayer&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;lasagne.layers.Conv2DLayer&lt;/code&gt;&lt;/a&gt;) (and probably also Theano):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;pad='full'&lt;/code&gt;: A variant of 0-padding which results in a bigger image&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pad='same'&lt;/code&gt;: Like &lt;code&gt;SAME&lt;/code&gt; for TF&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pad='valid'&lt;/code&gt;: Like &lt;code&gt;SAME&lt;/code&gt; for TF&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Caffe (&lt;a href=&quot;http://caffe.berkeleyvision.org/tutorial/layers.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;): I have no idea which padding they support. As &lt;code&gt;pad_h&lt;/code&gt; and &lt;code&gt;pad_w&lt;/code&gt; seem to be in &lt;code&gt;convolution_param&lt;/code&gt;, I guess padding is supported. Probably (only?) zero-padding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there publications where CNNs with padding, which was neither &lt;code&gt;VALID&lt;/code&gt; nor &lt;code&gt;SAME&lt;/code&gt;, or not 0-padding, was used?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I know of the following other options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;reflect&lt;/li&gt;&#xA;&lt;li&gt;nearest&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;both of them might have the advantage, that the filter does not detect a border, where no border is.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:33:05.200" LastActivityDate="2021-02-16T15:33:05.200" Title="Are non-zero paddings used?" Tags="|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15189" PostTypeId="1" AcceptedAnswerId="15198" CreationDate="2016-11-17T12:31:58.313" Score="4" ViewCount="209" Body="&lt;p&gt;I'm trying to build a recommendation engine for an e-commerce site. By using the common recommendation approach, I'm assuming that each product I recommend has the same value, so all I need to do is optimize the conversion rate probably using a common recommendation algorithm, but when the product's price varies a lot, what I really need to optimize is the following formula for each user:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Value of recomendation = (probability to convert) * (product price)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bigger problem than choosing the right algorithm and approach is choosing the right metric, so I could compare the different algorithms. For example, if you would like to only optimize the conversion rate, I would use the precision and recall or false/positive metrics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What metrics and approaches/algorithms are recommended in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="26235" LastEditorUserId="25157" LastEditDate="2016-11-17T17:10:36.897" LastActivityDate="2016-11-18T00:59:57.090" Title="Recommendation/personalization algorithm conflict" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15190" PostTypeId="1" AcceptedAnswerId="15191" CreationDate="2016-11-17T13:10:22.037" Score="11" ViewCount="1894" Body="&lt;p&gt;In &lt;a href=&quot;https://www.tensorflow.org/versions/r0.11/get_started/index.html&quot; rel=&quot;noreferrer&quot;&gt;Introduction&lt;/a&gt; I have just changed&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss = tf.reduce_mean(tf.square(y - y_data))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss = tf.reduce_mean(tf.abs(y - y_data)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and model is unable to learn the loss just became bigger with time. Why?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastActivityDate="2016-11-17T22:56:16.433" Title="Why TensorFlow can't fit simple linear model if I am minimizing absolute mean error instead of the mean squared error?" Tags="|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15191" PostTypeId="2" ParentId="15190" CreationDate="2016-11-17T13:31:34.550" Score="17" Body="&lt;p&gt;I tried this and got same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is because the gradient of &lt;code&gt;.abs&lt;/code&gt; is harder for a simple optimiser to follow to the minima, unlike squared difference where gradient approaches zero slowly, the gradient of the absolute difference has a fixed magnitude which abruptly reverses, which tends to make the optimiser oscillate around the minimum point. Basic gradient descent is very sensitive to magnitude of the gradient, and to the learning rate, which is essentially just a multiplier of the gradient for step sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest fix is to reduce the learning rate e.g. change line&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optimizer = tf.train.GradientDescentOptimizer(0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optimizer = tf.train.GradientDescentOptimizer(0.05)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, have a play with different optimisers. Some will be able to cope with &lt;code&gt;.abs&lt;/code&gt;-based loss better.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-17T22:56:16.433" LastActivityDate="2016-11-17T22:56:16.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15194" PostTypeId="2" ParentId="12886" CreationDate="2016-11-17T16:56:42.063" Score="4" Body="&lt;p&gt;This seems like a good solution for the loss function. I've had success with a similar approach recently, but I think you'd want to reorder where you multiply in the &lt;code&gt;class_weight&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thinking about it logically, the &lt;code&gt;class_weight&lt;/code&gt; will be a constant w.r.t. the output, so it will be carried along and applied to the gradient in the same way it's being applied to the cost function. There is one problem though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way you have it, the &lt;code&gt;class_weight&lt;/code&gt; would affect the prediction value. But you want it to affect the scale of the gradient. If I'm not wrong I think you'd want to reverse the order of operations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Take the cost like normal&#xA;error = tf.nn.softmax_cross_entropy_with_logits(pred, y)&#xA;&#xA;# Scale the cost by the class weights&#xA;scaled_error = tf.mul(error, class_weight)&#xA;&#xA;# Reduce&#xA;cost = tf.reduce_mean(scaled_error)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd be very interested to know how this performs in comparison to simply oversampling the underrepresented class, which is more typical. So if you gain some insight there post about it! :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly I successfully used a very similar technique in a different problem domain just recently (which brought me to this post):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/15181/multi-task-learning-finding-a-loss-function-that-ignores-certain-samples/15182#15182&quot;&gt;Multi-task learning, finding a loss function that &amp;quot;ignores&amp;quot; certain samples&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-11-17T17:02:46.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15195" PostTypeId="2" ParentId="15187" CreationDate="2016-11-17T17:43:45.383" Score="8" Body="&lt;p&gt;&lt;code&gt;activity_regularizer&lt;/code&gt; are used to control the output of a neural network. They tend to make the output smaller. Suppose the loss function is give as :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss function = DataLoss + regularizationLoss&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then for weight_regularizer, &lt;code&gt;regularizationLoss = f(Weights in a network)&lt;/code&gt;.&#xA;But for activity_regularizer, &lt;code&gt;regularizationLoss = f(Predicted outputs from a network)&lt;/code&gt;. Activity regularizers are generally used when you are quite aware of the distribution of the test dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your second argument, I would say you are pretty wrong. ANN as well as CNN both can suffer from overfitting. In order to prevent the model to overfit, we generally use a lot of regularization techniques among which Dropout is quite popular.  &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-17T17:43:45.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15196" PostTypeId="1" CreationDate="2016-11-17T20:01:40.993" Score="2" ViewCount="220" Body="&lt;p&gt;I have a list of objects. Each objects contains longitude, latitude and a list of words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do is to predict the location based on the text contained in the object (similar texts should have similar location). Right now I’m using cosine similarity to calculate the similarity between the objects text but I’m stuck how I can use that information to train my neural network. I have a matrix containing each object and how many time each word appeared in that object. F.x. if I had these two objects&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Obj C:   54.123, 10.123,   [This is a text for object C]&#xA;Obj B:   57.321, 11.113,   [This is a another text for object B]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I have something like the following matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       This is a text for object C another B&#xA;ObjC:   1   1  1   1   1    1    1   0     0&#xA;ObjB:   1   1  1   1   1    1    0   1     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would also have something like, for the distance between the two objects (note, that the numbers are not real)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        ObjC    ObjB&#xA;ObjC    1       0.25&#xA;ObjB    0.25    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have looked at how I use neural network to either classify things into groups (like A,B,C) or predict something like a housing price, but nothing that I find helpful for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would consider the prediction right if it is within some distance X, since I’m dealing with location. &#xA;This might be a stupid question, but someone point me to the right direction.&lt;/p&gt;&#xA;" OwnerUserId="26249" LastEditorUserId="8432" LastEditDate="2017-08-15T09:59:08.180" LastActivityDate="2017-09-14T10:29:08.943" Title="ANN and prediction" Tags="|classification|matlab|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15197" PostTypeId="2" ParentId="15196" CreationDate="2016-11-17T23:00:53.317" Score="1" Body="&lt;p&gt;Looks like you are trying to predict two different things (lat and longitude) with same input. Therefore, this is a problem of predicting multiple output. However, it is easy to understand the basic with only one output. I will describe it that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, your target is just one variable &lt;code&gt;Y&lt;/code&gt;, which is your location. After transformation (possibly TF-IDF), you have a feature matrix &lt;code&gt;X&lt;/code&gt; where one row represents one sample and one column represents one feature. What you need to do now is train an ANN with the input &lt;code&gt;X&lt;/code&gt; and target &lt;code&gt;Y&lt;/code&gt;. This is the way an ANN will train:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It will randomly initiate weight matrices for input and outside side of your network. If you have just one hidden layer, the number of weight matrix is 2.&lt;/li&gt;&#xA;&lt;li&gt;ANN will pass your input through some kind of activation function, get some output from them and then forward those outputs to output layer for further processing. Look up some tutorial for the actual maths. But this is the intuition.&lt;/li&gt;&#xA;&lt;li&gt;Finally, based on the weights and further processing, ANN will generate some output. They are not within acceptable margin in the first iteration in most cases. Therefore, ANN needs to refine those randomly selected weights. This is where the backpropagation algorithm kicks in. It allows ANN to learn the most suitable weights for the best prediction.&lt;/li&gt;&#xA;&lt;li&gt;This backpropagation algorithm requires a cost function which it tries to minimize. This cost function is usually RMS error function. But this cost function can be anything according to your choice. Cost functions take (mostly) two inputs. One of them is your true location (from training/validation data) and the other is predicted location that ANN got after the processing. As ANN tries to minimize the difference between predicted output and true output, it finds suitable weights for the network.&lt;/li&gt;&#xA;&lt;li&gt;When the error from your cost function reduces to a desired value (or a predefined number of iteration has been passed, or a tolerance level has been reached), ANN stops training and returns you those learned weight matrices.&lt;/li&gt;&#xA;&lt;li&gt;Once you have these final weight matrices, you can apply them on any new data to get the prediction. These weight matrices are your ANN.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As you have a multiple output problem, you need an implementation of ANN which support minimizing two errors at the same time. This is similar to multi-objective optimization. Python can help you here. See this &lt;a href=&quot;http://scikit-neuralnetwork.readthedocs.io/en/latest/module_mlp.html#sknn.mlp.Regressor.fit&quot; rel=&quot;nofollow noreferrer&quot;&gt;sknn.mlp documentation&lt;/a&gt; for details on use. It will take multiple output, each on its own column in the target matrix. Therefore, your &lt;code&gt;Y&lt;/code&gt; will be of shape &lt;code&gt;n * 2&lt;/code&gt;, where &lt;code&gt;n&lt;/code&gt; is the number of samples in your dataset. Just call &lt;code&gt;fit&lt;/code&gt; method with your existing &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; of the above format. Once the training is done, you get your minimum viable ANN. How to optimize the performance of ANN is a different story though :)&lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-17T23:00:53.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15198" PostTypeId="2" ParentId="15189" CreationDate="2016-11-18T00:41:12.657" Score="2" Body="&lt;p&gt;This is actually slightly similar to the problem that insurance companies face except that it seems like your loss costs are known.  Insurers have some probability of loss and then, given loss, the magnitude of the loss follows some distribution.  The cost to the insurer is dependent on both and they tend to be inversely related (lower losses are more likely than higher ones.)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, the value is known so you don't need to predict it the way insurers need to predict losses so you could simply:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Model the probability (&lt;strong&gt;phat&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Multiply the predicted probability by the known value (&lt;strong&gt;score = phat * value&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Recommend based on the resulting score&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Insurance companies typically do the same thing in calculating premiums except that they also need to model &lt;strong&gt;value&lt;/strong&gt;.  They sometimes model the two components jointly but typically they have separate models for frequency and severity and then just multiply them together to determine how much premium they should charge somebody.  &lt;/p&gt;&#xA;" OwnerUserId="25915" LastEditorUserId="25915" LastEditDate="2016-11-18T00:59:57.090" LastActivityDate="2016-11-18T00:59:57.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15199" PostTypeId="1" AcceptedAnswerId="15200" CreationDate="2016-11-18T00:48:14.363" Score="0" ViewCount="169" Body="&lt;p&gt;Whenever we talk about the probability distribution of data having more than one feature, we have only one option i-e multivariate normal distribution. Is there any other probability distribution model exist for multivariate data? If yes, then how can we find its parameters using MLE in MATLAB&lt;/p&gt;&#xA;" OwnerUserId="26260" LastActivityDate="2016-11-18T02:31:31.103" Title="Is there any other probability distribution model than gaussian for multivariate data" Tags="|machine-learning|statistics|matlab|probability|descriptive-statistics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15200" PostTypeId="2" ParentId="15199" CreationDate="2016-11-18T02:31:31.103" Score="1" Body="&lt;p&gt;There are quite a number of multivariate distributions, in theory. At the outset, we categorize them as &lt;code&gt;multivariate discrete distributions&lt;/code&gt; and &lt;code&gt;multivariate continuous distributions&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start with, you may explore the following link for :&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Category:Multivariate_discrete_distributions&quot; rel=&quot;nofollow noreferrer&quot;&gt;multivariate discrete distributions&lt;/a&gt; and the following for &lt;a href=&quot;https://en.wikipedia.org/wiki/Category:Multivariate_continuous_distributions&quot; rel=&quot;nofollow noreferrer&quot;&gt;multivariate continuous distributions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are entire volumes printed on the subject, for example, books by:  N. Balakrishnan, Norman L Johnson, and Samuel Kotz and these books are published by John Wiley and Sons, Inc.&lt;/p&gt;&#xA;" OwnerUserId="25218" LastActivityDate="2016-11-18T02:31:31.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15201" PostTypeId="2" ParentId="10158" CreationDate="2016-11-18T03:04:06.497" Score="1" Body="&lt;p&gt;As we can see, FTRL is nothing to do with our model, and it only serves as a way of approximating the optimal solution , just as SGD does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logloss is just a kind of loss pattern in classification problem,and there are other alternatives for it , such as ExpLoss. Least-squres loss is one of all loss patterns in regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why FTRL was mentioned everywhere ,which mainly treated Logloss as a example, is that it was often used in classification problem , such as CTR model in online advertisement scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopes this can helps you ,and good luck !&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-11-18T03:04:06.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15203" PostTypeId="1" CreationDate="2016-11-18T04:42:43.270" Score="1" ViewCount="784" Body="&lt;p&gt;I am trying to make a model of &lt;a href=&quot;http://tinyclouds.org/colorize/residual_encoder.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;this image&lt;/a&gt;. Here is the relevant code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;base_model = VGG16(weights='imagenet')&#xA;conv4_3, conv3_3, conv2_2, conv1_2 = base_model.get_layer('block4_conv3').output,&#xA;    base_model.get_layer('block3_conv3').output,&#xA;    base_model.get_layer('block2_conv2').output,&#xA;    base_model.get_layer('block1_conv2').output&#xA;&#xA;# Use the output of the layers of VGG16 on x in the model&#xA;conv1 = Convolution2D(256, 1, 1, border_mode='same')(BatchNormalization()(conv4_3))&#xA;conv1_scaled = resize(conv1, 56)&#xA;.&#xA;.&#xA;.&#xA;conv5 = Convolution2D(3, 3, 3, border_mode='same')(merge([ip_img, conv4], mode='sum'))&#xA;op = Convolution2D(2, 3, 3, border_mode='same')(conv5)&#xA;&#xA;for layer in base_model.layers:&#xA;    layer.trainable = False&#xA;&#xA;model = Model(input=base_model.input, output=op)&#xA;model.compile(optimizer='sgd', loss=custom_loss_fn)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a bunch of colored images in a directory. The input image should be the grayscale of an image stacked thrice(224x224x3) and the &lt;code&gt;op&lt;/code&gt; should be the UV planes of the image(224x224x2) which I can add to the grayscale(224x224x1) to get the YUV image. The custom loss function works on the UV of the original image and UV of the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I train it?&lt;/p&gt;&#xA;" OwnerUserId="26263" LastEditorUserId="29169" LastEditDate="2021-02-16T15:32:54.363" LastActivityDate="2021-02-16T15:32:54.363" Title="Image as input and output in keras" Tags="|python|deep-learning|keras|convolutional-neural-network|theano|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15206" PostTypeId="1" CreationDate="2016-11-18T06:16:50.233" Score="2" ViewCount="555" Body="&lt;p&gt;I am trying to implement Gaussian Mixture model with stochastic variational inference, following this &lt;a href=&quot;http://jmlr.org/papers/volume14/hoffman13a/hoffman13a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JQnlB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JQnlB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the pgm of Gaussian Mixture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the paper, the full algorithm of stochastic variational inference is:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xMDw2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xMDw2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I am still very confused of the method to scale it to GMM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I thought the local variational parameter is just $q_z$ and others are all global parameters. Please correct me if I was wrong. What does the step 6 mean by &lt;code&gt;as though Xi is replicated by N times&lt;/code&gt;? What am I supposed to do to achieve this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please help me with this? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="26265" LastActivityDate="2016-11-20T14:05:14.527" Title="Implement gaussian mixture model with stochastic variational inference" Tags="|machine-learning|clustering|probability|bayesian|graphical-model|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15207" PostTypeId="2" ParentId="13124" CreationDate="2016-11-18T07:03:12.800" Score="1" Body="&lt;p&gt;The global test shows that the model is significant as the p value is very low which is p-value: 8.916e-13 in alpha level of 0.05. 98.64% of the variability of Minutes  is described by Units. The effect of Units is significant in predicting Minutes. For each one unit value increase in Units, Minutes increases by 15.509.    &lt;/p&gt;&#xA;" OwnerUserId="14416" LastActivityDate="2016-11-18T07:03:12.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15208" PostTypeId="1" AcceptedAnswerId="15217" CreationDate="2016-11-18T07:03:48.593" Score="-1" ViewCount="50629" Body="&lt;p&gt;for example consider this dataframe :    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dam &amp;lt;- data.frame(name = letters[1:5],&#xA;                      re1 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)),&#xA;                      re2 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)),&#xA;                      re3 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)))&#xA;&#xA;#&amp;gt;the dataframe looks like this &#xA;     name re1 re2 re3&#xA;      a   yes yes yes&#xA;      b   no  no  no&#xA;      c   yes yes yes&#xA;      d   no  no  yes&#xA;      e   yes no  yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to replace the yes &amp;amp; no's in re1,re3 with 1 &amp;amp; 2.&#xA;to get something like this - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; #&amp;gt;the output should look like this &#xA;     name re1 re2 re3&#xA;      a   1   1   yes&#xA;      b   2   2   no&#xA;      c   1   1   yes&#xA;      d   2   2   no&#xA;      e   1   1   yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know i could do something like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;replace_re &amp;lt;- c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)&#xA;with_this &amp;lt;- rep(1:2,2,5)&#xA;dam$re1 %&amp;gt;% mapvalues(replace_re,with_this)&#xA;dam$re2 %&amp;gt;% mapvalues(replace_re,with_this)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I know I could use a for loop for many columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but what I want is how do you accomplish this in a functional way ,say with pacakage &quot;purrr&quot; (map,invoke_map functions)and does the job,in an elegant way for n such columns&#xA;i.e replacing my categorical variables with some other values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also, how would I do that with base R apply functions ? &#xA;this might be a very trivial question but I'm unable to come up with a more tidy way of doing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recoding the factors seems to be a bane for me,any help would be much appreciated .&lt;/p&gt;&#xA;" OwnerUserId="12771" LastEditorUserId="12771" LastEditDate="2016-11-18T10:11:44.453" LastActivityDate="2016-11-18T14:24:39.890" Title="Replacing values in multiple columns of a data frame in R" Tags="|r|dataset|data-cleaning|categorical-data|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15209" PostTypeId="2" ParentId="6103" CreationDate="2016-11-18T08:18:19.857" Score="1" Body="&lt;p&gt;I ran into the same problem today, tried googling for people having the same issue and found your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I solved my problem by creating the data source first and then running the prediction from there. So, instead of selecting the following option,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Batch Predictions &amp;gt; Create new batch prediction &amp;gt; ML model for batch prediction &amp;gt; My data is in S3, and I need to create a datasource&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which fails, I first did:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Datasources &amp;gt; Create a new datasource...&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, I ran the batch prediction from an existing datasource successfully. &lt;/p&gt;&#xA;" OwnerUserId="26272" LastEditorUserId="12384" LastEditDate="2016-11-18T09:37:06.993" LastActivityDate="2016-11-18T09:37:06.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15211" PostTypeId="1" AcceptedAnswerId="15212" CreationDate="2016-11-18T11:31:57.270" Score="1" ViewCount="2804" Body="&lt;p&gt;How does Keras' fit function calculate when validation set/validation split are NOT defined (I understand that the default values are None/0.0 respectively, so being not defined is practically default)? One always need a reference set to evaluate model performance...&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-11-18T12:24:58.013" Title="Validation during fitting in Keras" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15212" PostTypeId="2" ParentId="15211" CreationDate="2016-11-18T12:24:58.013" Score="2" Body="&lt;p&gt;If you don't define the validation set/ validation split for your model, then it has no way to check for it's performance because you have not provided anything to the model on which it can validate its performance. In this case, the model will run through training examples, will learn a way to minimize the cost function as per your code and that's it. You get a model which has learned a hypothesis from your data but how good  the model is, it can only  be checked by making predictions for the test set.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-18T12:24:58.013" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15213" PostTypeId="2" ParentId="15208" CreationDate="2016-11-18T13:16:19.790" Score="0" Body="&lt;p&gt;It's not clear exactly what you want. Do you want an &lt;em&gt;elegant&lt;/em&gt; solution using available packages and no for loops? Or, do you just want something that works?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is a solution I wrote before I knew about plyr / dplyr and used the car package's recode function. It works, but it's not what I would call elegant. You'll probably want to read the documentation on car::recode, as its syntax is a bit odd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that I've migrated over to the Hadleyverse / tidyverse, though, I'll probably figure out a solution that uses dplyr's recode function instead. But until then, this works for me.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(car)&#xA;&#xA;recode2 &amp;lt;- function ( data, fields, recodes, as.factor.result = FALSE ) {&#xA;  for ( i in which(names(data) %in% fields) ) { # iterate over column indexes that are present in the passed dataframe that are also included in the fields list&#xA;    data[,i] &amp;lt;- car::recode( data[,i], recodes, as.factor.result = as.factor.result )&#xA;  }&#xA;  data&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dam &amp;lt;- data.frame(name = letters[1:5],&#xA;  re1 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)),&#xA;  re2 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)),&#xA;  re3 = factor(c(&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;,&quot;no&quot;,&quot;yes&quot;)))&#xA;&#xA;recode2(dam, fields = c('re1', 're2'), recodes = &quot;'yes' = 1; 'no' = 2&quot;, as.factor.result = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  name re1 re2 re3&#xA;1    a   1   1 yes&#xA;2    b   2   2  no&#xA;3    c   1   1 yes&#xA;4    d   2   2  no&#xA;5    e   1   1 yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26281" LastEditorUserId="26281" LastEditDate="2016-11-18T13:22:45.013" LastActivityDate="2016-11-18T13:22:45.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15214" PostTypeId="1" AcceptedAnswerId="15232" CreationDate="2016-11-18T13:20:21.787" Score="1" ViewCount="2019" Body="&lt;p&gt;In the &lt;a href=&quot;https://arxiv.org/pdf/1503.03832v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FaceNet paper&lt;/a&gt;, they describe the &lt;a href=&quot;https://arxiv.org/abs/1311.2901&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zeiler&amp;amp;Fergus model&lt;/a&gt; like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] the&#xA;  Zeiler&amp;amp;Fergus model which consists of multiple interleaved&#xA;  layers of convolutions, [...]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What do they mean by &lt;strong&gt;interleaved&lt;/strong&gt;? How does that compare to the &lt;a href=&quot;https://arxiv.org/abs/1409.4842&quot; rel=&quot;nofollow noreferrer&quot;&gt;inception layers&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially, as the Zeiler&amp;amp;Fergus paper states&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We use standard fully supervised convnet models&#xA;  throughout the paper, as defined by (LeCun et al.,&#xA;  1989) and (Krizhevsky et al., 2012).&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The top few layers of the network are conventional&#xA;  fully-connected networks and the final layer is&#xA;  a softmax classifier.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:32:43.580" LastActivityDate="2021-02-16T15:32:43.580" Title="What are interleaved layers of convolutions?" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15215" PostTypeId="1" AcceptedAnswerId="15233" CreationDate="2016-11-18T13:30:56.937" Score="3" ViewCount="970" Body="&lt;p&gt;The &lt;a href=&quot;http://www.cs.tau.ac.il/~wolf/ytfaces/&quot; rel=&quot;nofollow noreferrer&quot;&gt;YouTube Faces database&lt;/a&gt; (YTF) consists of 3,425 videos of 1,595 different people. Given two videos, the task for YTF is to decide if they contain the same person or not. Having $n$ comparisons, the classifier might get $c \leq n$  right. Then the accuracy would be $\frac{c}{n}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1503.03832v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FaceNet&lt;/a&gt; is a CNN which maps an image of a face on a unit sphere of $\mathbb{R}^{128}$. It was evaluated on YTF. How did they decide which person is in the video?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I can imagine several procedures how this could be done, but I couldn't find it in the paper. One example, how it could be done, is by evaluating all images $x_i^{(k)}$ with $i = 1, \dots, \text{length of video }k$ and averaging the results - but I would like to know what they did / how this is usually done.) &lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:32:34.737" LastActivityDate="2021-02-16T15:32:34.737" Title="How is the evaluation setup for YouTube faces of FaceNet?" Tags="|dataset|computer-vision|convolutional-neural-network|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15216" PostTypeId="2" ParentId="15152" CreationDate="2016-11-18T14:08:19.683" Score="1" Body="&lt;p&gt;To solve challenges #3 and #4, let's limit the overall available rank volume. For example, sum of this rank for all the users will be 1 (100%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From challenge #2 I understood, that you accept 2 different ranks: (1) place from 1 to 100, and (2) simple sum of all earned points (fixed and relative). Did I got it right? I so, there is no need to worry about unlimited growth, or fixed scores inflation. Let's just use percentages, not 1-100 ranks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These percentage ranks could be calculated based on interaction behaviors (vote/selecting answer/etc), using &lt;a href=&quot;http://www.eecs.harvard.edu/cs286r/courses/fall11/papers/AT%2705.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PageRank&lt;/a&gt;-like algorithm. Such algorithm will consider all previous reactions (and ranks of acted users), obtained by an exact user. Unfortunately, you cannot use PageRank algorithm &quot;as is&quot;, because it supports only &quot;positive&quot; links, but you can look for it's extensions. For example, look at &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.147.4570&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; with PageRank extension  for both positive and negative links (as users can down vote). You can iteratively estimate percentage rank (TrustRank, TR) using this algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second task is to calculate reward/penalty rate in points for each single action. Let's determine (predefine) maximal reward/penalty rate (&lt;em&gt;X&lt;/em&gt;) for each type of action. And will use coefficient to discount it, based on TrustRanks of acting users (e.g., author and voter). Slightly modified &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sigmoid&lt;/a&gt; will map this ratio from &lt;code&gt;[-Inf,+Inf]&lt;/code&gt; range to &lt;code&gt;[0,1]&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a6vkO.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a6vkO.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here for peer users you will have &lt;code&gt;~0.5&lt;/code&gt; of predefined maximal rate. If &quot;voter&quot; has TR twice more than &quot;author&quot;, &quot;author&quot; will recieve &lt;code&gt;~0.75&lt;/code&gt; of predefined value, and so on. You can tune steepness with additional parameter, or try to find any other mapping transformation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, now simply multiply maximal penalty/reward by this coefficient, and you'll get the number of point, you need to deduct or add. The only issue, I see, is a user with zero TR - such user as a voter will &quot;give&quot; nothing, and as an object of voting, will recieve the maximal amount of points regardless voter's rank. To avoid this, you can predefine minimal TR (like &lt;code&gt;1e-10&lt;/code&gt;), and don't let user's TR to fall beyond this value.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-11-18T14:08:19.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15217" PostTypeId="2" ParentId="15208" CreationDate="2016-11-18T14:24:39.890" Score="2" Body="&lt;p&gt;Given:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dam&#xA;  name re1 re2 re3&#xA;1    a yes yes yes&#xA;2    b  no  no  no&#xA;3    c yes yes yes&#xA;4    d  no  no  no&#xA;5    e yes yes yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dam2 = reshape2::dcast(&#xA;   dplyr::mutate(&#xA;      reshape2::melt(dam,id.var=&quot;name&quot;),&#xA;      value=plyr::mapvalues(&#xA;           value, c(&quot;yes&quot;,&quot;no&quot;),c(&quot;OK&quot;,&quot;notOK&quot;))&#xA;   ),name~variable)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;get that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dam2&#xA;  name   re1   re2   re3&#xA;1    a    OK    OK    OK&#xA;2    b notOK notOK notOK&#xA;3    c    OK    OK    OK&#xA;4    d notOK notOK notOK&#xA;5    e    OK    OK    OK&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've recoded it to &quot;OK&quot; and &quot;notOK&quot; because your remapping doesn't make sense. The &quot;from values&quot; should be unique, not have repeated &quot;yes&quot; and &quot;no&quot; in them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note how this is done. Make a tidy data set by melting. Mutate it. Cast it back into untidy format. Yes you could use pipes.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-11-18T14:24:39.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15220" PostTypeId="1" AcceptedAnswerId="15378" CreationDate="2016-11-18T17:35:24.240" Score="3" ViewCount="666" Body="&lt;p&gt;XGBoost v0.6 Binary Classification&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$ cat predin.txt &#xA;0 1:10 2:10 3:10&#xA;1 1:10 2:10 3:100&#xA;1 1:10 2:100 3:10&#xA;1 1:100 2:10 3:10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In other words, the negative instance label 0 requires low values for all three features.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;./xgboost/xgboost m.conf&#xA;./xgboost/xgboost m.conf task=dump model_in=0002.model name_dump=dump.raw.txt&#xA;$ cat dump.raw.txt &#xA;booster[0]:&#xA;0:[f1&amp;lt;55] yes=1,no=2,missing=1&#xA;    1:[f2&amp;lt;55] yes=3,no=4,missing=3&#xA;        3:[f3&amp;lt;55] yes=5,no=6,missing=5&#xA;            5:leaf=-1.5&#xA;            6:leaf=1.5&#xA;        4:leaf=1.5&#xA;    2:leaf=1.5&#xA;booster[1]:&#xA;0:[f1&amp;lt;55] yes=1,no=2,missing=1&#xA;    1:[f2&amp;lt;55] yes=3,no=4,missing=3&#xA;        3:[f3&amp;lt;55] yes=5,no=6,missing=5&#xA;            5:leaf=-0.784694&#xA;            6:leaf=0.784694&#xA;        4:leaf=0.784694&#xA;    2:leaf=0.784694&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does &quot;yes=3,no=4&quot; mean?&lt;/p&gt;&#xA;" OwnerUserId="25359" LastActivityDate="2016-11-27T16:02:29.860" Title="XGBoost in final model what does &quot;yes=3,no=4&quot; mean?" Tags="|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15221" PostTypeId="1" CreationDate="2016-11-18T18:40:14.540" Score="0" ViewCount="334" Body="&lt;p&gt;I have the following dataset. This dataset contains &lt;code&gt;ATM banking transactions&lt;/code&gt; in different places over 30 days. These ATMs are located in different cities and provinces. The problem that I need to solve is to develop a model that can be able to detect which ATMs are in the same city or  in the same province. The only thing I know about this problem is that is related to &lt;strong&gt;Spatial Data Mining&lt;/strong&gt; but I have no idea which techniques, algorithms, and tools I should use to solve this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this dataset &lt;code&gt;PS ID column&lt;/code&gt; corresponds to the &lt;code&gt;ATM IDs&lt;/code&gt;, &lt;code&gt;AMOUNT column&lt;/code&gt; corresponds to the &lt;code&gt;amount of money transferred&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is which techniques, algorithms, and tools I should use to solve this problem???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I use clustering or I should use other techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2Rumx.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2Rumx.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26284" LastEditorUserId="10902" LastEditDate="2016-11-19T08:40:13.887" LastActivityDate="2016-12-19T09:17:06.580" Title="Spatial Data Mining and ATM banking transactions" Tags="|data-mining|geospatial|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15222" PostTypeId="2" ParentId="13553" CreationDate="2016-11-18T19:27:47.697" Score="1" Body="&lt;p&gt;You should use dummy variables and then you can toss it directly into K-means. If you have a lot of categories the efficient way to do this is through a one-hot-encoding (sparse-encoding).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a little demo using clustering and then using the clusters in a regression model. In general, you should avoid doing that but it's illuminating in this case.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(glmnet)&#xA;library(Matrix)&#xA;n &amp;lt;- 1e5&#xA;nclusters &amp;lt;- 5&#xA;set.seed(420)&#xA;ls &amp;lt;- data.frame(sample(letters, n, replace=TRUE))&#xA;xs &amp;lt;- sparse.model.matrix(~.,data=ls)&#xA;print(head(xs))  &#xA;# Now let's run k-means&#xA;out &amp;lt;- kmeans(xs, centers=nclusters)&#xA;bs &amp;lt;- rep(1, dim(xs)[2])&#xA;# Let's run k-means on the different categories&#xA;clusterpred &amp;lt;- data.frame(out[[1]])&#xA;ys &amp;lt;- xs %*% bs + rnorm(n)&#xA;print(table(clusterpred))&#xA;# Now let's use a clustered data set to predict some outcome&#xA;cxs &amp;lt;- sparse.model.matrix(~.,data=clusterpred)&#xA;model &amp;lt;- glmnet(y=ys, x=xs, alpha=0)&#xA;cmodel &amp;lt;- glmnet(y=ys, x=cxs, alpha=0)&#xA;&#xA;# Predictions&#xA;yhat &amp;lt;- predict(model, xs)&#xA;yhatc &amp;lt;- predict(cmodel, cxs)&#xA;# Looking at the difference RMSEs &#xA;print(sqrt( sum( (ys-yhat)**2 )))&#xA;print(sqrt( sum( (ys-yhatc)**2 )))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-11-18T19:27:47.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15223" PostTypeId="2" ParentId="14069" CreationDate="2016-11-18T19:33:12.040" Score="12" Body="&lt;p&gt;If your categorical columns are currently character/object you can use something like this to do each one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index&#xA;&#xA;for c in char_cols:&#xA;    df[c] = pd.factorize(df[c])[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you need to be able to get back to the categories I'd create a dictionary to save the encoding; something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index&#xA;label_mapping = {}&#xA;&#xA;for c in char_cols:&#xA;    df[c], label_mapping[c] = pd.factorize(df[c])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using Julien's mcve will output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [3]: print(df)&#xA;Out[3]: &#xA;    a   b   c   d&#xA;0   0   0   0   0.155463&#xA;1   1   1   1   0.496427&#xA;2   0   0   2   0.168625&#xA;3   2   0   1   0.209681&#xA;4   0   2   1   0.661857&#xA;&#xA;In [4]: print(label_mapping)&#xA;Out[4]:&#xA;{'a': Index(['Var2', 'Var3', 'Var1'], dtype='object'),&#xA; 'b': Index(['Var2', 'Var1', 'Var3'], dtype='object'),&#xA; 'c': Index(['Var3', 'Var2', 'Var1'], dtype='object')}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26288" LastActivityDate="2016-11-18T19:33:12.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15225" PostTypeId="2" ParentId="15090" CreationDate="2016-11-18T21:48:06.140" Score="1" Body="&lt;p&gt;There isn't as far as I can tell by looking at the source code of the version I'm working with (April 2016). But it's definitely worth adding and it isn't impossible. xgboost.train loops over the iterations extracts corresponding learning rate, sets it on booster via eta parameter and calls booster update. xgboost.cv calls fold update (ie CVPack.update) which calls booster update. Just need to add a learning rate (aka eta) parameter to CVPack.update&lt;/p&gt;&#xA;" OwnerUserId="26297" LastActivityDate="2016-11-18T21:48:06.140" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15226" PostTypeId="1" CreationDate="2016-11-19T00:59:36.460" Score="0" ViewCount="502" Body="&lt;p&gt;(I am newbieeee in machine learning)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of every automated test (Python), I collect a lot of information from our product. The information includes CPU, Memory Usage, Disk Activity, Postfix Queue statistics, and various other stats. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to come up with a script/tool(Python) that leverages machine learning and automatically analyze the results for me, instead of me looking at the results manually every day. I want to train the tool on what acceptable levels are for various components like CPU, Memory, depending on product's specifications, and various inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I read online, I think I need to implement supervised classification in my tool, build a model, train it for a few days. But what I don’t know is where to start. When I looked at &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Supervised learning&lt;/a&gt;, I see a bunch of stuff and don’t really know what is really relevant to me. I appreciate if I can get some suggestions on what I need to focus on and how to implement this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the data can be in memory before my original script completes, or I can get the data from a database to which my original script writes to.&lt;/p&gt;&#xA;" OwnerUserId="26302" LastEditorUserId="8501" LastEditDate="2016-11-20T08:47:08.387" LastActivityDate="2016-11-20T14:07:26.827" Title="How do I use Machine Learning to analyze process statistics of automated test results?" Tags="|python|classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15227" PostTypeId="2" ParentId="15221" CreationDate="2016-11-19T02:09:16.867" Score="1" Body="&lt;p&gt;This sounds like an unsupervised learning problem since you are trying to group observations according to some common association rather than trying to predict a target. My first impression is that you are facing an uphill battle here as your data set doesn't look comprehensive enough. As a starting point, this is what I would try:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could assume that ATMs in the same time zone would exhibit similar seasonality patterns on both daily and weekly scales. For example, on a typical weekday, there would probably be a consistent pattern of an up-tick in usage around say 8am, probably peaking during the lunch break with another peak following the end of the workday (these are just assumptions, I haven't done any analysis on this). So you could convert your TIME column into a factor with 24 levels (one for each hour of the day), but you would need to ensure that the times reported are in some consistent time zone, as opposed to the local time of the ATM from which they come which would of course not make much sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That approach may allow you to cluster on the basis of time zone, but what about city? Well, if you assume that different cities have different levels of affluence, the AMOUNT column might yield some useful associations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in summary, I'd try a clustering algorithm based on seasonal usage patterns and the value of the transactions. I'd be interested to hear if you can get any useful results from this as my feeling is that your data set probably doesn't have enough useful features to be effective. &lt;/p&gt;&#xA;" OwnerUserId="17368" LastActivityDate="2016-11-19T02:09:16.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15231" PostTypeId="1" CreationDate="2016-11-19T17:39:41.377" Score="1" ViewCount="412" Body="&lt;p&gt;I currently trying to train a neural network capable of  mapping a input (400 input features) to a output vector of 13 elements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input features is an fixed length audio sample, and the output is a feature vector extracted from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made a simple network consisting of one layer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(output_dim=13, input_dim=400, init=&quot;normal&quot;,activation=&quot;relu&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4Hea7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Hea7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trained it for 10 epochs gives me these results: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4lC3W.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4lC3W.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training shows that the it does become better, but how do i improve it ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updated model:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o7XsO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o7XsO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training and testing: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/A269d.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A269d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plotting (predicted_output - actual_output)[0]&lt;/strong&gt;&#xA;Plot  showing the different between the predicted_output and the actual_output for only the first feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BGaBO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BGaBO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;overlap plot:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/F874F.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F874F.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="25429" LastEditDate="2016-11-20T20:41:19.923" LastActivityDate="2016-11-20T20:41:19.923" Title="Network design for extracting MFCC features from audio samples" Tags="|neural-network|regression|keras|" AnswerCount="0" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="15232" PostTypeId="2" ParentId="15214" CreationDate="2016-11-19T18:42:52.757" Score="2" Body="&lt;p&gt;The Oxford dictionary explains &lt;a href=&quot;https://en.oxforddictionaries.com/definition/interleave&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;interleave&quot;&lt;/a&gt; as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Place something between the layers of (something)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or in the context of telecommunication, as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mix (digital signals) by alternating between them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://arxiv.org/pdf/1503.03832v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FaceNet paper&lt;/a&gt;, Schroff &lt;em&gt;et. al&lt;/em&gt; describe their first architecture as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;multiple interleaved layers of convolutions, non-linear activations, local response normalizations, and max pooling layers&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which means they create a CNN by &quot;altering between&quot; these layer types (e.g. &lt;code&gt;Conv -&amp;gt; Pool -&amp;gt; Conv -&amp;gt; Pool -&amp;gt; ..&lt;/code&gt;). Or with the wording of the first definition, you &quot;place one layer&quot; of each type &quot;between&quot; the other existing layers (i.e. start with a couple of &lt;code&gt;Conv&lt;/code&gt; layers, then place &lt;code&gt;Pool&lt;/code&gt; layer between two &lt;code&gt;Conv&lt;/code&gt; layers, and so on).&#xA;The final architecture would look somewhat like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Conv -&amp;gt; ReLu -&amp;gt; LRN -&amp;gt; Pool -&amp;gt; Conv -&amp;gt; ReLu -&amp;gt; LRN -&amp;gt; Pool -&amp;gt; Conv -&amp;gt; ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, this &quot;interleaved&quot; CNN is a network where the different layer types are applied in &lt;em&gt;series&lt;/em&gt;, layer-by-layer.&#xA;As a comparison, in &lt;a href=&quot;https://arxiv.org/abs/1409.4842&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;inception&quot; type CNN&lt;/a&gt;, you apply the different layer types in &lt;em&gt;parallel&lt;/em&gt; (the graphic below is 1 layer):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              +---&amp;gt; Conv 3x3 ---+&#xA;              |                 |&#xA;prev layer ---+---&amp;gt; Conv 5x5 ---+---&amp;gt; next layer&#xA;              |                 |&#xA;              +---&amp;gt; Pool 3x3 ---+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-11-19T18:42:52.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15233" PostTypeId="2" ParentId="15215" CreationDate="2016-11-19T19:05:43.247" Score="5" Body="&lt;p&gt;The objective function they use to train the CNN minimizes the squared L2 distance (i.e. the squared Euclidean distance) between two &lt;em&gt;similar&lt;/em&gt; (positive) images and simultaneously maximizes the distance between two &lt;em&gt;different&lt;/em&gt; (negative) images. &#xA;That means, the (squared) Euclidean distance between two representations is a measure of their similarity. &#xA;Then, recognizing a face in a new image is as simple as 1) running it through the CNN and 2) finding its nearest neighbors with a KNN algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last paragraph was only about images - in the Youtube Faces DB, we are handling videos of different persons.&#xA;In section 5.7 of the paper, they describe how they evaluate performance:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We use the average similarity of all pairs of the first one hundred frames that our face detector detects in each video.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, you were partially right: they just average the independent results over video frames. Probably for performance reasons, they chose to average the first 100 frames. They do describe that increasing this to the first 1000 frames increases performance from 95.12% to 95.18%, which is not significantly more.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-11-19T19:05:43.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15234" PostTypeId="2" ParentId="15226" CreationDate="2016-11-19T19:30:59.257" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to come up with a script/tool(Python) that &lt;strong&gt;leverages machine learning&lt;/strong&gt; and automatically analyze the results for me, instead of me looking at the results manually every day.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is the wrong approach. Choose your tools to solve tasks, not because they sound cool. Imagine a craftsman saying &quot;I want to use a hammer&quot;. (However, machine learning might still suit to this task.)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;From what I read online, I think I need to [...] train it for a few days.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Depends on how much data, how much features, how much computing power and which algorithm you use. In many cases you can train a model within seconds on a normal computer.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The real question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The problem: You have some characteristics (features) like CPU usage, memory usage, ... for every automated test. For some instances (meaning a test and the associated features) you have the binary information &quot;normal&quot;, &quot;abnormal&quot;. Now, when you get the characteristics of a new test, you want to automatically decide if it was normal or abnormal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a binary classification problem. If you have the label (&quot;normal&quot;, &quot;abnormal&quot;) for all instances, you can apply supervised learning methods. If you have the label only for some instances, it is semi-supervised learning. If you don't have any labels it is unsupervised.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The Answer&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I assume you are in a supervised learning case. Then, I would suggest to use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision tree&lt;/a&gt; (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;sklearn.tree.DecisionTreeClassifier&lt;/code&gt;&lt;/a&gt; - sklearn is pretty easy to use and has many standard algorithms implemented) for this decision making. It basically automatically generates many nested if/else statements to get to the decision. Hence it is interpretable and you could manually adjust or improve it.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-11-20T14:07:26.827" LastActivityDate="2016-11-20T14:07:26.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15235" PostTypeId="1" CreationDate="2016-11-19T22:56:50.177" Score="4" ViewCount="704" Body="&lt;p&gt;I'm training a Neural Network for pattern recognition. I have a matrix of examples of size ($N$x$4$) with $N$ examples and $4$ variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that the examples are highly correlated between them and adding new examples is not giving new information to the network, but I don't figure how to measure that correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row of the matrix can be labeled $\mathbf{X}_{i}$. Suppose zero mean, the covariance matrix between examples would be $\mathbf{R}_{ij} = E[\mathbf{X}_{i} \mathbf{X}_{j}^T]$.&#xA;I can also calculate the covariance matrix of $\mathbf{X}$, whose elements are $\mathbf{R}_{ij}$ but in that case I would have a matrix of $\sim 300000$x$300000$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of calculate some average correlation or an other measure of correlation between examples?.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I don't want to know the correlation matrix of variables, I want to know the correlation between examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2: This is the learning curve:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EJ0xB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EJ0xB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26316" LastEditorUserId="26316" LastEditDate="2016-11-22T04:26:30.200" LastActivityDate="2017-08-19T09:19:30.153" Title="Correlation between examples" Tags="|machine-learning|classification|neural-network|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15236" PostTypeId="1" AcceptedAnswerId="15238" CreationDate="2016-11-20T00:03:37.957" Score="4" ViewCount="173" Body="&lt;p&gt;Background: I'm studying CNN's outside of my undergraduate CS course on ML. I have a few questions related to CNNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) When training a CNN, we desire tightly bounded/cropped images of the desired classes, correct? I.e. if we were trying to recognize dogs, we would use thousands of images of tightly cropped dogs. We would also feed images of non-dogs, correct? These images are scaled to a specific size, i.e. 255x255.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Let's say training is complete. Our model's accuracy seems sufficient, with no problems. From here, let's have a large, HD image of a non-occluded dog running through a field with various obstacles. With a typical NN and some data, we just take the model, cross it with some input, and bam it's going to output some class. How will the CNN view this large image, and then 'find' the dog? Do we run some type of preprocessing on the image to partition it, and feed the partitions?&lt;/p&gt;&#xA;" OwnerUserId="26317" LastEditorUserId="29169" LastEditDate="2021-02-16T15:29:54.467" LastActivityDate="2022-01-31T18:34:07.873" Title="How do CNNs use a model and find the object(s) desired?" Tags="|neural-network|computer-vision|convolutional-neural-network|beginner|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15238" PostTypeId="2" ParentId="15236" CreationDate="2016-11-20T03:18:09.780" Score="5" Body="&lt;p&gt;Though there can be a very detailed explanation for this question but I will try to make you understand much minimal words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Cropping the images to a particular size isn't a necessary condition and neither is scaling. But put this way, it doesn't matter whether a dog is represented in a B&amp;amp;W image or  RGB image because a convolution network learns features in the images which are independent of colors. Scaling and resizing help to limit the value of pixels between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Once you have trained your CNN model, it has learned all the features like edges,etc. to recognize a dog in the image. Because the model has learned the features, it acquires certain properties like translation invariance which means that no matter where you position a dog in the image, it's still a dog and have the same features. How the model recognize it? It checks for the features of a dog, learned during training, no matter what the size of the new image is or where the dog is in the image or what the dog is doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For getting a in-depth understanding you can refer to the following resources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap6.html&quot; rel=&quot;noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap6.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;noreferrer&quot;&gt;http://cs231n.github.io/convolutional-networks/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-20T03:18:09.780" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15239" PostTypeId="1" AcceptedAnswerId="15247" CreationDate="2016-11-20T07:23:42.357" Score="1" ViewCount="2383" Body="&lt;p&gt;I am training a neural network for classification using Matlab, and I don't understand if I can use the &lt;code&gt;trainbr&lt;/code&gt; training function (Bayesian Regularization Backpropagation). It uses the MSE performance measure, but I want to use the crossentropy. If I set crossentropy as the performance function, the algorithm sets it back to MSE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other way, I can't use a validation set with this training and I don't find how to change it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = A';&#xA;t = y';&#xA;&#xA;% Choose a Training Function&#xA;% For a list of all training functions type: help nntrain&#xA;% 'trainlm' is usually fastest.&#xA;% 'trainbr' takes longer but may be better for challenging problems.&#xA;% 'trainscg' uses less memory. Suitable in low memory situations.&#xA;trainFcn = 'trainbr';  % Scaled conjugate gradient backpropagation.&#xA;&#xA;% Create a Pattern Recognition Network&#xA;net = patternnet(hiddenLayerSize,trainFcn);&#xA;&#xA;% Choose Input and Output Pre/Post-Processing Functions&#xA;% For a list of all processing functions type: help nnprocess&#xA;net.input.processFcns = {'removeconstantrows','mapminmax'};&#xA;net.output.processFcns = {'removeconstantrows','mapminmax'};&#xA;&#xA;% Setup Division of Data for Training, Validation, Testing&#xA;% For a list of all data division functions type: help nndivide&#xA;net.divideFcn = 'dividerand';  % Divide data randomly&#xA;net.divideMode = 'sample';  % Divide up every sample&#xA;net.divideParam.trainRatio = 60/100;&#xA;net.divideParam.valRatio = 20/100;&#xA;net.divideParam.testRatio = 20/100;&#xA;&#xA;% Choose a Performance Function&#xA;% For a list of all performance functions type: help nnperformance&#xA;net.performFcn = 'crossentropy';  % Cross-Entropy&#xA;net.trainParam.epochs = 5000;&#xA;&#xA;% Choose Plot Functions&#xA;% For a list of all plot functions type: help nnplot&#xA;net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...&#xA;    'plotconfusion', 'plotroc'};&#xA;&#xA;% Train the Network&#xA;[net,tr] = train(net,x,t);&#xA;&#xA;% Test the Network&#xA;y = net(x);&#xA;e = gsubtract(t,y);&#xA;net.performParam.regularization = 0; &#xA;performance = perform(net,t,y); &#xA;&#xA;tind = vec2ind(t);&#xA;yind = vec2ind(y);&#xA;percentErrors = sum(tind ~= yind)/numel(tind);&#xA;&#xA;% Recalculate Training, Validation and Test Performance&#xA;trainTargets = t .* tr.trainMask{1};&#xA;valTargets = t .* tr.valMask{1};&#xA;testTargets = t .* tr.testMask{1};&#xA;trainPerformance = perform(net,trainTargets,y); &#xA;valPerformance = perform(net,valTargets,y);&#xA;testPerformance = perform(net,testTargets,y);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="26316" LastEditorUserId="104237" LastEditDate="2020-09-15T13:08:29.377" LastActivityDate="2020-09-15T13:08:29.377" Title="Using the trainbr function for classification in Matlab" Tags="|neural-network|classification|machine-learning-model|matlab|performance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15242" PostTypeId="1" AcceptedAnswerId="15258" CreationDate="2016-11-20T13:43:19.647" Score="15" ViewCount="11736" Body="&lt;p&gt;I have a convolutional + LSTM model in Keras, similar to this (ref 1), that I am using for a Kaggle contest. Architecture is shown below. I have trained it on my labeled set of 11000 samples (two classes, initial prevalence is ~9:1, so I upsampled the 1's to about a 1/1 ratio) for 50 epochs with 20% validation split.I was getting blatant overfitting for a while but I thought it got it under control with noise and dropout layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model looked like it was training wonderfully, at the end scored 91% on the entirety of the training set, but upon testing on the test data set, absolute garbage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UBNm6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UBNm6.png&quot; alt=&quot;Loss by epoch&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/j5GxE.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/j5GxE.png&quot; alt=&quot;Validation accuracy by epoch&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notice: the validation accuracy is higher than the training accuracy. This is the opposite of &quot;typical&quot; overfitting.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is, given the small-ish validation split, the model is still managing to fit too strongly to the input set and losing generalization. The other clue is that val_acc is greater than acc, that seems fishy. Is that the most likely scenario here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is overfitting, would increasing the validation split mitigate this at all, or am I going to run into the same issue, since on average, each sample will see half the total epochs still?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Layer (type)                     Output Shape          Param #     Connected to                     &#xA;====================================================================================================&#xA;convolution1d_19 (Convolution1D) (None, None, 64)      8256        convolution1d_input_16[0][0]     &#xA;____________________________________________________________________________________________________&#xA;maxpooling1d_18 (MaxPooling1D)   (None, None, 64)      0           convolution1d_19[0][0]           &#xA;____________________________________________________________________________________________________&#xA;batchnormalization_8 (BatchNormal(None, None, 64)      128         maxpooling1d_18[0][0]            &#xA;____________________________________________________________________________________________________&#xA;gaussiannoise_5 (GaussianNoise)  (None, None, 64)      0           batchnormalization_8[0][0]       &#xA;____________________________________________________________________________________________________&#xA;lstm_16 (LSTM)                   (None, 64)            33024       gaussiannoise_5[0][0]            &#xA;____________________________________________________________________________________________________&#xA;dropout_9 (Dropout)              (None, 64)            0           lstm_16[0][0]                    &#xA;____________________________________________________________________________________________________&#xA;batchnormalization_9 (BatchNormal(None, 64)            128         dropout_9[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;dense_23 (Dense)                 (None, 64)            4160        batchnormalization_9[0][0]       &#xA;____________________________________________________________________________________________________&#xA;dropout_10 (Dropout)             (None, 64)            0           dense_23[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;dense_24 (Dense)                 (None, 2)             130         dropout_10[0][0]                 &#xA;====================================================================================================&#xA;Total params: 45826&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the call to fit the model (class weight is typically around 1:1 since I upsampled the input): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class_weight= {0:1./(1-ones_rate), 1:1./ones_rate} # automatically balance based on class occurence&#xA;m2.fit(X_train, y_train, nb_epoch=50, batch_size=64, shuffle=True, class_weight=class_weight, validation_split=0.2 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;SE has some silly rule that I can post no more than 2 links until my score is higher, so here is the example in case you are interested:&#xA;Ref 1: machinelearningmastery DOT com SLASH sequence-classification-lstm-recurrent-neural-networks-python-keras&lt;/p&gt;&#xA;" OwnerUserId="23718" LastEditorUserId="23718" LastEditDate="2016-11-22T22:23:03.210" LastActivityDate="2020-08-09T08:47:27.953" Title="Can overfitting occur even with validation loss still dropping?" Tags="|keras|cross-validation|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15244" PostTypeId="2" ParentId="15239" CreationDate="2016-11-20T16:39:57.257" Score="0" Body="&lt;p&gt;Setting a big number of validation checks (Higher than the maximum number of iterations), Matlab computes the validation error, but it doesn't affects the convergence. I still don't know how to compute the crossentropy instead of the MSE.&lt;/p&gt;&#xA;" OwnerUserId="26316" LastActivityDate="2016-11-20T16:39:57.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15247" PostTypeId="2" ParentId="15239" CreationDate="2016-11-20T18:47:45.187" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;https://mathworks.com/help/nnet/ref/trainbr.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;trainbr&lt;/code&gt;&lt;/a&gt; mode uses the Bayesian regularization backpropagation. This method was presented in &lt;sup&gt;1&lt;/sup&gt;, which presents a regression problem with the loss function&#xA;$$ E_D = \sum_{i=1}^n (t_i - a_i)^2 $$&#xA;where $t_i$ is the target and $a_i$ is the network's response. The paper proposes to add a regularization term, leading to a loss function $F$ of the form&#xA;$$ F = \beta E_D + \alpha E_W $$&#xA;where $E_W$ is the square of the sums of all network weights, i.e. $E_W = \sum_{i,j} \| w_{ij} \|^2$.&#xA;The two parameters $\alpha$ and $\beta$ control the weighing of the two parts $E_D$ and $E_W$: For $\alpha \ll \beta$, the network will minimize the loss, without really trying to keep weights low. For $\alpha \gg \beta$, the network will minimize the weights, allowing for some more error. In reality, this means a large $\alpha$ will stop the network from overfitting, which leads to a better generalization at the cost of a larger training error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to find a train a model which generalizes well, but still has a low error rate, is the right setting of $\alpha$ and $\beta$. This is achieved by treating them as random variables and finding an optimal setting, using the Bayesian methods presented in &lt;sup&gt;2&lt;/sup&gt;. (I won't talk about the details on that here, you can find that in the two linked papers.) Finally, the paper presents an algorithm, which calculates the optimal $\alpha$ and $\beta$ in each training iteration. This makes this algorithm generalize really well, especially in the presence of noisy input signals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as described, the loss function is a weighted sum between the MSE ($E_D$) and the regularization term ($E_W$). So, in short: you can only use it with the MSE, and not with cross-entropy. You'll need a different training algorithm, you can find a list in the MATLAB documentation, &lt;a href=&quot;https://ch.mathworks.com/help/nnet/ug/choose-a-multilayer-neural-network-training-function.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;sup&gt;1&lt;/sup&gt; F. D. Foresee and M. T. Hagan: &quot;Gauss-Newton Approximation to Bayesian Learning&quot;, in &lt;em&gt;Proceedings of the 1997 International Joint Conference on Neural Networks&lt;/em&gt;, June 1997. DOI: &lt;a href=&quot;http://dx.doi.org/10.1109/ICNN.1997.614194&quot; rel=&quot;nofollow noreferrer&quot;&gt;10.1109/ICNN.1997.614194&lt;/a&gt;. &lt;a href=&quot;http://hagan.okstate.edu/icnn97a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[Link to PDF]&lt;/a&gt;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;sup&gt;2&lt;/sup&gt; D. J. C. McKay: &quot;Bayesian Interpolation&quot;, Neural Computation, May 1992, Vol. 4, No. 3, pp. 415-447. DOI: &lt;a href=&quot;http://dx.doi.org/10.1162/neco.1992.4.3.415&quot; rel=&quot;nofollow noreferrer&quot;&gt;10.1162/neco.1992.4.3.415&lt;/a&gt;. &lt;a href=&quot;https://pdfs.semanticscholar.org/8e68/c54f39e87daf3a8bdc0ee005aece3c652d11.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[Link to PDF]&lt;/a&gt;.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-11-20T18:47:45.187" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15249" PostTypeId="1" AcceptedAnswerId="15293" CreationDate="2016-11-20T20:27:12.630" Score="1" ViewCount="2419" Body="&lt;p&gt;I am trying to extract features from images using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def process_image(image_fp):&#xA;    image_ = imread(image_fp)&#xA;    image_ = resize(image_, (300, 200,3))&#xA;    image=equalize_hist(rgb2gray(image_))&#xA;    edges = skimage.feature.blob_dog(image)&#xA;    return edges.reshape(edges.size).tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where image_fp is an image path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having a problem due to the different sizes of the return. In general, the reshape guarantees the same size in the others algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to get always the same size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I only see a way: to truncate the lists (the stupid way).&lt;/p&gt;&#xA;" OwnerUserId="26334" LastEditorUserId="27003" LastEditDate="2016-12-27T21:08:15.720" LastActivityDate="2017-10-24T05:32:42.403" Title="Image feature extraction Python skimage blob_dog" Tags="|python|scikit-learn|feature-extraction|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15250" PostTypeId="2" ParentId="15235" CreationDate="2016-11-20T21:44:26.457" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are assuming improving new samples will always improve the performance. This is not true in many cases. Did you plot a learning curve to see how the performance is improving with respect to increasing number of samples? You probably should try to do this before delving deep into sample correlation.&lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2016-11-20T21:44:26.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15251" PostTypeId="1" AcceptedAnswerId="15255" CreationDate="2016-11-21T01:04:15.547" Score="2" ViewCount="6597" Body="&lt;p&gt;I have read &lt;a href=&quot;https://en.wikipedia.org/wiki/Nearest_centroid_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Wikipedia article&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZFmsQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZFmsQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But, the idea is still very fuzzy to me.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lvoFg.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lvoFg.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, k=5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, we have, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_5 = \{A, B, C, D, E\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_2 = \{Triangle, Square\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R_5 = \{9, 8, 5, 1, 4 \}$ (just assumed)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, $\mu_{Triangle} = \frac{5}{2} = 2.50$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and, $\mu_{Square} = \frac{22}{3} = 7.33$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, $\mu_{Triangle} &amp;lt; \mu_{Square}$, $class(?) == Triangle$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Am I correct?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user9232" LastActivityDate="2019-09-05T05:37:05.277" Title="How does the Nearest Centroid method work?" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15252" PostTypeId="2" ParentId="15251" CreationDate="2016-11-21T01:42:40.263" Score="0" Body="&lt;p&gt;What the prediction function does is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Computes the centroid for each class&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Measures the distance ( generally Euclidean ) of the data point &lt;strong&gt;X&lt;/strong&gt; to the centroid of each class. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the distance is of &lt;strong&gt;X&lt;/strong&gt; and the centroid of a particular class is &lt;strong&gt;minimum&lt;/strong&gt; then it assigns that class to the data point &lt;strong&gt;X&lt;/strong&gt; ( The argmin statement in the picture you have provided ) i.e Y predicts the centroid closest to the point X&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is the technique used in K means clusterings well ( though not exactly the same). I suggest you read &lt;a href=&quot;http://dni-institute.in/blogs/k-means-clustering-algorithm-explained/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article for a more intuitive understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="24232" LastEditorUserId="24232" LastEditDate="2016-11-21T01:56:36.233" LastActivityDate="2016-11-21T01:56:36.233" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15253" PostTypeId="1" CreationDate="2016-11-21T01:52:30.907" Score="1" ViewCount="483" Body="&lt;p&gt;For example, if I run a single round (&lt;code&gt;nrounds=1&lt;/code&gt;), how does XGBoost go about making predictions?  I thought it would simply return a linear regression model, but I quickly shot that theory down by working out an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that I've already seen &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/332&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md#general-parameters&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and most of Google's top search results. &lt;/p&gt;&#xA;" OwnerUserId="18246" LastActivityDate="2022-04-16T22:04:55.457" Title="What does &quot;gblinear&quot; do in XGBoost?" Tags="|machine-learning|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15255" PostTypeId="2" ParentId="15251" CreationDate="2016-11-21T02:09:05.050" Score="5" Body="&lt;p&gt;You are making a mistake regarding what is given: during training, you don't have a radius &lt;span class=&quot;math-container&quot;&gt;$R$&lt;/span&gt;. You have the coordinates &lt;span class=&quot;math-container&quot;&gt;$\vec{x}$&lt;/span&gt; and the label &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; for each point:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \vec{x}_1 = [-2;1] \quad \vec{x}_2 = [1;2] \quad \vec{x}_3 = [0;-1] \quad \vec{x}_4 = [1;0] \quad \vec{x}_5 = [1;1] $$&lt;/span&gt;&#xA;and&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ y_1 = T \quad y_2=T \quad y_3=T \quad y_4=S \quad y_5=S $$&lt;/span&gt;&#xA;with that, your &quot;trained&quot; centroids are&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \mu_T = \frac 13 [-2 + 1 + 0; 1 + 2 - 1] = \left[-\frac 13, \frac 23\right] $$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \mu_S = \frac 12 [1 + 1; 0 + 1] =\left[1, \frac 12\right]$$&lt;/span&gt;&#xA;You calculate these centroids before you get any test values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, for testing, for your observed point &lt;span class=&quot;math-container&quot;&gt;$\vec{x} = [1,1]$&lt;/span&gt;, you calculate the Euclidean distance between the point &lt;span class=&quot;math-container&quot;&gt;$\vec{x}$&lt;/span&gt; and the centroids &lt;span class=&quot;math-container&quot;&gt;$\mu_T$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\mu_S$&lt;/span&gt;:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \| \vec{x} - \mu_T \| = \left\| [1;1] - \left[-\frac 13; \frac 23\right] \right\| = \left\| \left[\frac 43; \frac 13\right] \right\| = 1.37 $$&lt;/span&gt; &#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \| \vec{x} - \mu_S \| = \left\| [1;1] - \left[1; \frac 12\right] \right\| = \left\| \left[0; \frac 12 \right] \right\| = 0.5$$&lt;/span&gt;&#xA;Finally, the term &lt;span class=&quot;math-container&quot;&gt;$\hat{y} = \arg\min_{l \in \mathbf{Y}} \|\vec{x}-\mu_l\|$&lt;/span&gt; is used to find the estimated class &lt;span class=&quot;math-container&quot;&gt;$\hat{y}$&lt;/span&gt; (the hat symbol is to denote that this is an &lt;em&gt;estimated&lt;/em&gt; &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt;, not one we knew before.). &lt;span class=&quot;math-container&quot;&gt;$\arg\min$&lt;/span&gt;, means that you find the minimum value - which is 0.5 in our case, and chose which &quot;argument&quot;, i.e. which class, leads to that minimum value. In our case, the class which leads to the minimal distance is &lt;span class=&quot;math-container&quot;&gt;$S$&lt;/span&gt;, so the result is &lt;span class=&quot;math-container&quot;&gt;$\hat{y} = S$&lt;/span&gt;, and our test point is a square.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="14906" LastEditDate="2019-09-05T05:37:05.277" LastActivityDate="2019-09-05T05:37:05.277" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15257" PostTypeId="1" CreationDate="2016-11-21T05:11:39.657" Score="1" ViewCount="166" Body="&lt;p&gt;If the images to test against are only part of the object, does it help to have training set data of the whole object, or is it better for all the training data to be parts of the object? For instance, if test pictures are the legs of a chair or the top of a chair, does it help to have training set image data of the entire chair? Or should I focus on only getting training set data of parts of the chair? I'm planning on using TensorFlow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be classifying the object as a chair, not part of a chair. But if it would give me a higher rate of recognizing chairs by having it recognize parts, it could recognize parts, and I could combine the information at the end. I only need to recognize at least one chair (or much more likely part of a chair) in the image.&lt;/p&gt;&#xA;" OwnerUserId="26341" LastEditorUserId="26342" LastEditDate="2016-11-22T09:49:05.180" LastActivityDate="2016-11-22T12:14:12.823" Title="Machine learning image training set data as whole object but finding part of object" Tags="|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15258" PostTypeId="2" ParentId="15242" CreationDate="2016-11-21T05:37:12.350" Score="10" Body="&lt;p&gt;I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of what you sampled is used to validate your network , this means that you train and validate using balanced data set. In the testing you used imbalanced database. This means that your validation and testing sets are not equivalent.&#xA;In such case you may have high validation accuracy and low testing accuracy.&#xA;Please find &lt;a href=&quot;https://pdfs.semanticscholar.org/69a6/8f9cf874c69e2232f47808016c2736b90c35.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this reference&lt;/a&gt; that talks mainly about data imbalance problem for DNN, you can check how they sample to do the training, validation and testing&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="98307" LastEditDate="2020-08-09T08:47:27.953" LastActivityDate="2020-08-09T08:47:27.953" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="15259" PostTypeId="2" ParentId="685" CreationDate="2016-11-21T05:42:32.930" Score="0" Body="&lt;p&gt;SGD has nothing to do with regularization, and so does FTRL. They are learning methods approximating the optimal solution in classification or regression problem.&lt;/p&gt;&#xA;&lt;p&gt;If you want to see how FTRL works, you can check &lt;a href=&quot;https://github.com/PayneJoe/algo-sensetime/blob/master/src/main/scala/FMWithFTRL.scala&quot; rel=&quot;nofollow noreferrer&quot;&gt;my code&lt;/a&gt; which was applied in my industrial project.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.cmu.edu/%7Eepxing/papers/2016/HuaWei_KDD16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is another learning method called TDAP based on FTRL, you can check the code.&lt;/p&gt;&#xA;&lt;p&gt;Hope this will help you, and good luck!&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="25180" LastEditDate="2021-05-15T21:43:02.720" LastActivityDate="2021-05-15T21:43:02.720" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15260" PostTypeId="1" AcceptedAnswerId="15261" CreationDate="2016-11-21T10:53:05.220" Score="1" ViewCount="551" Body="&lt;p&gt;I'm looking for a good introductory book or course to big data analytics.&#xA;For the practical part, I'm particularly interested into using big data tools in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would prefer a book, but other references are welcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="21686" LastEditorUserId="21686" LastEditDate="2016-11-21T11:41:14.223" LastActivityDate="2016-11-21T11:55:11.767" Title="Big data analytics references" Tags="|bigdata|" AnswerCount="1" CommentCount="3" ClosedDate="2016-11-21T13:28:21.480" ContentLicense="CC BY-SA 3.0" />
  <row Id="15261" PostTypeId="2" ParentId="15260" CreationDate="2016-11-21T11:29:31.087" Score="3" Body="&lt;p&gt;I tried to explore some of the best available resources, which includes online courses (Free/Paid), Books etc.&lt;/p&gt;&#xA;&lt;h2&gt;Books&lt;/h2&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amazon.in/Data-Analytics-WIND-Seema-Acharya/dp/8126554789?tag=googinhydr18418-21&amp;amp;tag=googinkenshoo-21&amp;amp;ascsubtag=30e388b6-c9fe-455b-83e2-c5020caeb097&quot; rel=&quot;nofollow noreferrer&quot;&gt;Big Data and Analytics (WIND)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amazon.in/Hadoop-Dummies-Dirk-Deroos/dp/8126550511?ie=UTF8&amp;amp;redirect=true&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hadoop for Dummies&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.amazon.in/Big-Data-Dummies-Judith-Hurwitz/dp/8126543280?tag=googinhydr18418-21&amp;amp;tag=googinkenshoo-21&amp;amp;ascsubtag=30e388b6-c9fe-455b-83e2-c5020caeb097&quot; rel=&quot;nofollow noreferrer&quot;&gt;Big Data for Dummies&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1449311520&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hadoop: The Definitive Guide&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.amazon.in/Learning-Spark-Lightning-Fast-Data-Analysis-ebook/dp/B00SW0TY8O?ie=UTF8&amp;amp;*Version*=1&amp;amp;*entries*=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Spark: Lightning-Fast Big Data Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/com/1449327176&quot; rel=&quot;nofollow noreferrer&quot; rel=&quot;nofollow noreferrer&quot;&gt;MapReduce Design Patterns&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Online Courses&lt;/h2&gt;&#xA;&lt;p&gt;These are some best platforms that provide lots of courses with rich content and hands-on labs. You can go from beginner to expert level, followed by intermediate. (courses will be - Free/Paid  )&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udemy.com/courses/search/?ref=home&amp;amp;src=ukw&amp;amp;q=Big%20data%20analytics%20&amp;amp;lang=en&amp;amp;instructionalLevel=beginner&quot; rel=&quot;nofollow noreferrer&quot;&gt;Udemy - Big Data Analytics Courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.coursera.org/courses?_facet_changed_=true&amp;amp;languages=en&amp;amp;primaryLanguages=en&amp;amp;query=introduction%20to%20big%20data%20analytics&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera - Big Data Analytics Courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.edx.org/course/?search_query=Big%20data%20analytics&amp;amp;level=Introductory&quot; rel=&quot;nofollow noreferrer&quot;&gt;edX - Big Data Analytics Courses&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/courses/data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;Udacity - Data Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Other References&lt;/h2&gt;&#xA;&lt;p&gt;For more thoughts on that you might like to explore these websites-&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-best-book-to-learn-Hadoop-for-beginners&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://bigdata-madesimple.com/10-books-to-get-started-with-hadoop/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Big Data - Made Simple&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.fromdev.com/2014/07/Best-Hadoop-Books.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;From Dev&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kdnuggets.com/2014/04/9-free-books-learning-data-mining-data-analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;KDnuggets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h2&gt;Update [BigData in R]&lt;/h2&gt;&#xA;&lt;p&gt;You can go through these references-&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Programming_with_Big_Data_in_R&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia - Programming with Big Data in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.rstudio.com/resources/webinars/working-with-big-data-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;RStudio - Working with BigData in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.infoworld.com/article/2880360/big-data/learn-to-crunch-big-data-with-r.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;InfoWorld - Learn To crunch BigData with R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;That's all from my side. Hope it helps. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="10902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-21T11:55:11.767" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15262" PostTypeId="2" ParentId="14688" CreationDate="2016-11-21T11:53:43.523" Score="0" Body="&lt;p&gt;What is your output? Is it a 0-1? If so, you shouldn't be using RMSE and should be using cross-entropy as your loss function.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-11-21T11:53:43.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15263" PostTypeId="2" ParentId="15242" CreationDate="2016-11-21T11:57:44.723" Score="5" Body="&lt;p&gt;If your &lt;strong&gt;training loss goes under you validation loss, you are overfitting&lt;/strong&gt;, even if validation is still dropping. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is the sign that your network is learning patterns in the train set that are not applicable in the validation one&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-11-21T11:57:44.723" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15264" PostTypeId="1" CreationDate="2016-11-21T12:05:21.003" Score="1" ViewCount="1155" Body="&lt;p&gt;When I use a linear or random forest regression to train from data, I can make predictions with test data. I have a problem when I introduce the &quot;impute&quot; data widget in my pipeline to fill in the missing values from the training data. My predictions then are all the same for all rows, even though the data table is correct. I tried to swap the impute widget for the pre-processor and I have the same behaviour. Below is my graph:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/z0YCW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/z0YCW.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&#xA;Adding an impute widget and all my predictions become the same values:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CP1iF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CP1iF.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26350" LastEditorUserId="26350" LastEditDate="2016-11-21T13:31:19.053" LastActivityDate="2016-12-22T15:28:24.383" Title="Orange 3 - Same prediction for all of my data when using impute widget" Tags="|regression|random-forest|linear-regression|orange|prediction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15266" PostTypeId="1" AcceptedAnswerId="15298" CreationDate="2016-11-21T16:07:34.130" Score="2" ViewCount="611" Body="&lt;p&gt;I wanted to enhance a recommendation engine with information relying not only on past purchases or ratings but also on behavioral and demographical variables like &lt;strong&gt;sex, age, location, service usage frequency or hours&lt;/strong&gt;. This information may be sparse (&lt;strong&gt;e.g. the user may not have provided me with his age&lt;/strong&gt;).&lt;/p&gt;&#xA;&lt;p&gt;Can you suggest an approach that would allow for it?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;My first approach:&lt;/h3&gt;&#xA;&lt;p&gt;After screening the general approaches to recommender systems, I think the one that is able to implement my idea is custom(?) &lt;strong&gt;item-based collaborative filtering&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;To be more precise: I would include user profile information in the same way item ratings are introduced, probably with two alterations to raw data:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normalize scale to be compliant with the rating scale.&lt;/li&gt;&#xA;&lt;li&gt;Add a parameter in my algorithm that would put a weight (e.g. 20% or 50%) on user profile rows, as there may be 10 user-profile&#xA;variables and a million product items.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="15430" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-11-23T02:57:09.147" Title="Which recommender system approach allows for inclusion of user profile?" Tags="|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15268" PostTypeId="1" CreationDate="2016-11-21T18:46:06.700" Score="2" ViewCount="629" Body="&lt;p&gt;There existis any way to find the number of cluster needed to use in K-Means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having some difficulties to find a method to determine k...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="23811" LastActivityDate="2016-11-21T19:01:04.337" Title="Spark 1.6.1 - Determining the number of clusters in a data set" Tags="|apache-spark|k-means|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15269" PostTypeId="2" ParentId="15268" CreationDate="2016-11-21T19:01:04.337" Score="1" Body="&lt;p&gt;You can use the Elbow method. &lt;a href=&quot;https://en.wikipedia.org/wiki/Elbow_method_(clustering)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Elbow_method_(clustering)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26316" LastActivityDate="2016-11-21T19:01:04.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15270" PostTypeId="2" ParentId="15266" CreationDate="2016-11-21T23:34:38.533" Score="0" Body="&lt;p&gt;For the problem like this, you should try item-based collaborative filtering. There is a book named &quot;The Ancient Art of Numerati (A guide to data mining)&quot;, a whole chapter on this book is based on item based collaborative filtering with a very good examples and also some coding samples of python for implementation. I suggest you do read this.&lt;/p&gt;&#xA;" OwnerUserId="26260" LastActivityDate="2016-11-21T23:34:38.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15271" PostTypeId="1" AcceptedAnswerId="15286" CreationDate="2016-11-22T00:35:38.340" Score="0" ViewCount="804" Body="&lt;p&gt;A classification system I built is going to go into production soon (it'll be part of a larger dashboard), and I'm looking for ways to better visualize and convey to business folks the results of a classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, given &quot;old&quot; data on which the model was trained, I predict the classes of &quot;new&quot; data, with the goal being to show whether the class distributions for the new data are statistically indifferent from the class distributions observed in the old data. So, if there are three classes &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;, and &lt;code&gt;C&lt;/code&gt; in the old data, with proportions of 50%, 30%, and 20%, respectively, I compare the classification distribution for the new data with those original, observed proportions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outside of a confusion matrix (which I think is probably inappropriate for most dashboard users), how else can I effectively present these results? I was thinking of a bar chart like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZCSOs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZCSOs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13087" LastActivityDate="2016-11-22T16:49:19.073" Title="Visualizing results of a classification problem, excluding confusion matrices?" Tags="|classification|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15274" PostTypeId="1" CreationDate="2016-11-22T05:55:44.447" Score="1" ViewCount="301" Body="&lt;p&gt;If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my data set is sufficient?&lt;/p&gt;&#xA;" OwnerDisplayName="user9232" LastEditorUserId="26146" LastEditDate="2016-11-22T14:50:33.080" LastActivityDate="2016-11-23T02:41:54.423" Title="How do I know if my training data set is sufficient?" Tags="|classification|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15275" PostTypeId="2" ParentId="15274" CreationDate="2016-11-22T06:21:34.960" Score="3" Body="&lt;p&gt;We can not judge whether our training data set is sufficient or not directly from the shape description for your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the most of the time, we sample some instances, hopefully they are distributed as real-world data, so that the final model constructed through these data can reveal the truth of the real-world. Theoretically, the size of training data is not the problem, but we shall collect training samples as many as possible to ensure our model will have good performance on predicted instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, we can get the answer by testing whether the distribution of our predicted value is consistent with that of truth.&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="26146" LastEditDate="2016-11-23T02:41:54.423" LastActivityDate="2016-11-23T02:41:54.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15277" PostTypeId="2" ParentId="15274" CreationDate="2016-11-22T10:35:41.647" Score="1" Body="&lt;p&gt;&lt;strong&gt;It depends of the model&lt;/strong&gt; your training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data needs to be big enough so that you can subsample (with replacement) the data &lt;strong&gt;without loss of F1 score&lt;/strong&gt;. That is the theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Neural network, the rule of thumb is P^2, P being the number of parameters. This is often hard to achieve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, if you have the opportunity to create more data, make some, check the improvement, and if there is, you need more data (for your model, that is)&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-11-22T10:35:41.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15278" PostTypeId="2" ParentId="15257" CreationDate="2016-11-22T12:14:12.823" Score="1" Body="&lt;p&gt;Train with the same kinds of inputs as you wish the network to recognise. There is no easy way to add logical combinations that you think should help the network, because deep neural networks, despite the hype as the best thing to hit AI, are essentially just really dumb complex statistical analysers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided you supply training images, the CNN will learn how to statistically combine components to decide if overall an image is a chair or not. For instance if the image is of a person wearing a padded jacket, perhaps that will look a little like the back of a chair. The NN might output a 0.75 confidence that a back of a chair is in the image. If you manually combine decisions from components, you will need to figure out from that whether to say there is a chair in the image. If you leave it to the network, the CNN itself will combine this data with the additional detection there is a 0.2 confidence that the person's legs are chair legs (and they are in the wrong relative place) and decide - hopefully - that there is no chair.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't need to explicitly label components of objects for the NN to learn that there are components and what they look like. The outputs of Deep Dream and other CNN visualisations demonstrate that internally the CNN builds up its own structures that represent parts of objects. They are not necessarily the same parts as a person would focus on, but by allowing the training process to choose, you should be able to  get a good recognition rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you give the network a lot of chair parts to recognise separately, it will get better at labelling those chair parts accurately. But that will not really help the network recognise a chair. In fact it could be worse at the main task if the training images are too different from images it is expected to recognise. Plus, it will be your task to then decide how to re-combine the classes to detect the chair later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There probably &lt;em&gt;are&lt;/em&gt; architectures and image-processing pipelines that can be used to combine logical knowledge of parts and their poses into whole objects. It is something I very much expect to be an active area of research in computer vision systems. However, these approaches will not use a simple CNN and altering training data as the solution.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-11-22T12:14:12.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15279" PostTypeId="2" ParentId="14710" CreationDate="2016-11-22T13:31:26.580" Score="2" Body="&lt;h2&gt;&lt;strong&gt;Step 1: Changing dates to years&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;First you can change your column of dates (Order_Date) to simply years by using this formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; salesdata$Order_Date = format(salesdata$Order_Date, &quot;%Y&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give your Order Dates just the year the sales was done in.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;Step 2: Summarizing your sales data&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;After this you can create a vector with the summarizations of your sales data per year. This can be done using tapply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tapply(salesdata$Sales, salesdata$Order_Date, FUN=sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give you an overview of the total sales per year.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;Step 3: Plotting your graph&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You can use barplot() to plot your bar chart using the values you got in your tapply function.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;One-liner&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The function as a oneliner:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; barplot(tapply(salesdata$Sales, format(salesdata$Order_Date, &quot;%Y&quot;), FUN=sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course you can edit your graph to your liking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2P04O.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2P04O.png&quot; alt=&quot;Example of the bar chart&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26376" LastActivityDate="2016-11-22T13:31:26.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15280" PostTypeId="2" ParentId="12679" CreationDate="2016-11-22T13:52:23.927" Score="1" Body="&lt;p&gt;I've created some GIST file with steps how to &quot;Install TensorFlow v 0.11 and CUDA 8.1 | CUDNN 5.1 on Ubuntu 16.04&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/denti/41860cb6b55e0847b4f2685016c7f14e&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/denti/41860cb6b55e0847b4f2685016c7f14e&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works perfect for me for fresh 16.04.&#xA;You can skip last part with install bazel and tensorflow from scratch.&lt;/p&gt;&#xA;" OwnerUserId="26377" LastActivityDate="2016-11-22T13:52:23.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15282" PostTypeId="2" ParentId="15264" CreationDate="2016-11-22T14:56:47.107" Score="1" Body="&lt;p&gt;When imputing, certain characteristics of the data have to be recorded. E.g. when you impute missing data by mean value parameter $\mu$, you have to record that $\mu$ from training data in order to be able to reapply the exact same transformation on the test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus you only need to connect your training data to the &lt;em&gt;Impute&lt;/em&gt; widget, while the testing data will be imputed with the &lt;em&gt;same parameters&lt;/em&gt; when the two domains are &lt;em&gt;matched&lt;/em&gt; (in &lt;em&gt;Predictions&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you connect testing data to its own &lt;em&gt;Impute&lt;/em&gt; widget, it gets imputed with its own, testing-data-specific parameters, which is, as you have discovered, incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also: &lt;a href=&quot;https://datascience.stackexchange.com/questions/12321/difference-between-fit-and-fit-transform-in-scikit-learn-models&quot;&gt;Difference between fit and fit_transform in scikit_learn models?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-11-22T14:56:47.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15283" PostTypeId="1" AcceptedAnswerId="15285" CreationDate="2016-11-22T15:49:30.237" Score="2" ViewCount="682" Body="&lt;p&gt;I usually try to form my ANNs with classic fine-tuning approach but I recently learned that there are different &quot;predefined&quot; networks specially for certain tasks. Is there a good summary about these? Are they really perform better than home-made ones?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2016-11-22T17:10:36.297" Title="Predefined Neural Networks instead of fine tuning?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15284" PostTypeId="1" AcceptedAnswerId="15303" CreationDate="2016-11-22T16:09:21.413" Score="3" ViewCount="333" Body="&lt;p&gt;I'm trying to implement the t-SNE algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hI9LZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hI9LZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that to compute the pairwise affinities, I have to follow this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/l08En.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/l08En.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is computing $\sigma_i$. In the Wikipedia I found:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The bandwidth of the Gaussian kernels $\sigma_{i}$,&#xA;  is set in such a way that the perplexity of the&#xA;  conditional distribution equals a predefined perplexity using a binary&#xA;  search. As a result, the bandwidth is adapted to the density of the&#xA;  data: smaller values of $\sigma_{i}$ are&#xA;  used in denser parts of the data space.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't understand what this really means. How can I calculate $\sigma_i$?&lt;/p&gt;&#xA;" OwnerUserId="4978" LastEditorUserId="381" LastEditDate="2016-11-23T05:19:12.073" LastActivityDate="2016-11-23T11:29:15.543" Title="How to equalize the pairwise affinity perplexities when implementing t-SNE?" Tags="|dimensionality-reduction|tsne|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15285" PostTypeId="2" ParentId="15283" CreationDate="2016-11-22T16:22:22.913" Score="1" Body="&lt;p&gt;It depends. If you are doing a task for which the weights of a very large network, trained on the same type of problem, are present, then it's better to use the weights of that pre-trained network. You can also fine tune the layers later on. One such example is the VGG16 Net. This NN was used in the ImageNet challenge in 2014, hence if you are trying to do image classification tasks where your images are the subset of ImageNet, then you should use the pre-trained VGG16 weights. &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is a great tutorial if you want to go in details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the above situation might not be true in every case. For example, the above case is not true if you reverse the situation.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="15412" LastEditDate="2016-11-22T17:10:36.297" LastActivityDate="2016-11-22T17:10:36.297" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15286" PostTypeId="2" ParentId="15271" CreationDate="2016-11-22T16:49:19.073" Score="1" Body="&lt;p&gt;Here are a couple of options. I prefer the stacked column chart, which could also be annotated with the actual proportions if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EFj2G.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EFj2G.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26024" LastActivityDate="2016-11-22T16:49:19.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15288" PostTypeId="2" ParentId="12871" CreationDate="2016-11-22T17:14:05.017" Score="0" Body="&lt;p&gt;I know why your LinearSVC is giving 0 or 1 class results only:&#xA;when you are using one vs rest, you have unbalanced dataset. You must add class_weight=balanced to your estimator, because rest set is much bigger.&lt;/p&gt;&#xA;" OwnerUserId="26382" LastActivityDate="2016-11-22T17:14:05.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15289" PostTypeId="2" ParentId="14998" CreationDate="2016-11-22T17:26:22.157" Score="0" Body="&lt;p&gt;The only &lt;em&gt;self-hosted&lt;/em&gt; solution I know is the paid Anaconda Enterprise cloud setup, &lt;a href=&quot;https://anaconda.org/about&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://anaconda.org/about&lt;/a&gt;. The other solutions I am aware of are not self-hostable!&lt;/p&gt;&#xA;" OwnerUserId="13977" LastActivityDate="2016-11-22T17:26:22.157" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15292" PostTypeId="1" AcceptedAnswerId="15297" CreationDate="2016-11-22T20:07:43.093" Score="1" ViewCount="1567" Body="&lt;p&gt;&lt;strong&gt;I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has missing values so I want to replace them.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made some general calculation to get some understand of the disruption of the data and to help me choose the value which would replace the missing values, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume that I have attribute A with the following: &lt;bR&gt;&#xA;&lt;strong&gt;mean =&lt;/strong&gt; 121.68676278 &lt;br&gt; &#xA;&lt;strong&gt;std =&lt;/strong&gt; 30.51562426 &lt;br&gt;&#xA;&lt;strong&gt;median =&lt;/strong&gt; 117 &lt;br&gt; &#xA;&lt;strong&gt;mode =&lt;/strong&gt; &lt;bR&gt; &#xA;&lt;strong&gt;min =&lt;/strong&gt; 44 &lt;br&gt; &#xA;&lt;strong&gt;max =&lt;/strong&gt; 199 &lt;br&gt; &#xA;[in all the calculations, I ignored the missing values] &lt;br&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to choose between mean, median, or mode which one would be most suitable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and there is something else which was very confusing for me, the std is very large and when I asked about it I was told that this could be normal based on the range of my data but I did not understand what that means? &lt;/p&gt;&#xA;" OwnerUserId="26392" LastActivityDate="2016-11-23T02:36:35.300" Title="Choice of replacing missing values based on the data distribution" Tags="|dataset|statistics|preprocessing|missing-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15293" PostTypeId="2" ParentId="15249" CreationDate="2016-11-22T20:46:24.753" Score="0" Body="&lt;p&gt;Please read through the scikit documentation that is found &lt;a href=&quot;http://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.blob_dog&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and I am assuming that you have gone through the method through which it calculates blobs in the images. If not the link is &lt;a href=&quot;https://en.wikipedia.org/wiki/Blob_detection#The_difference_of_Gaussians_approach&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It return the 2D array of arrays with 3 values in each array, giving coordinates and std.deviation of Gaussian of the blob found. The length of the array returned is the number of blobs it has found in the image. So eliminating elements from that array by resizing it, is a wrong way of approach and may eliminate the features which you might find important.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-11-22T20:46:24.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15295" PostTypeId="1" AcceptedAnswerId="15593" CreationDate="2016-11-23T00:11:33.097" Score="0" ViewCount="158" Body="&lt;p&gt;I have dataset of gold prices and after modifying and some preprocessing i ended up with dataframe below: &lt;a href=&quot;https://i.stack.imgur.com/VkirQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VkirQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is 50,000 record in dataset and all columns expect &lt;code&gt;date&lt;/code&gt; are &lt;code&gt;int&lt;/code&gt; type and &lt;code&gt;date&lt;/code&gt; is &lt;code&gt;datetime&lt;/code&gt; object.&#xA;i need to predict price per unit in some specific dates. but somehow i baffled with so many methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is &lt;strong&gt;what data mining algorithm/method is results good prediction for this kind of data ?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="17384" LastActivityDate="2016-12-07T15:46:37.270" Title="gold price detection using data mining" Tags="|classification|data-mining|predictive-modeling|algorithms|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15296" PostTypeId="1" CreationDate="2016-11-23T00:31:46.837" Score="2" ViewCount="2582" Body="&lt;p&gt;I have about 500k rows of data. I'm new to data science and I'm trying to train a model utilizing support vector machines as part of my analysis. On my little macbook pro, it seems endless. Right now I'm using an RBF kernel and am not seeing an end to this computation.&lt;/p&gt;&#xA;&lt;p&gt;I do however have access to 100+ compute nodes with 16 cores a piece. My problem is, I don't know how to utilize this to my extent as I am lacking knowledge on how I should approach this SVM. Right now I am using scipy.&lt;/p&gt;&#xA;&lt;h1&gt;SciPy Code&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def makeModelandPrediction(trainData, trainLabel, testData):&#xA;    model = svm.SVC(&#xA;        C=1.0,&#xA;        cache_size=200,&#xA;        class_weight=None,&#xA;        coef0=0.0,&#xA;        decision_function_shape=None,&#xA;        degree=3,&#xA;        gamma='auto',&#xA;        kernel='rbf',&#xA;        max_iter=-1,&#xA;        probability=False,&#xA;        random_state=None,&#xA;        shrinking=True,&#xA;        tol=0.001,&#xA;        verbose=False,&#xA;        )&#xA;    model.fit(trainData, trainLabel)&#xA;    prediction = model.predict(testData)&#xA;    return prediction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I've already preprocessed the data and done a 70/30 train/test split on the data. Can someone point a beginner in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="26402" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-26T07:06:07.287" Title="How can I run SVM on 500k rows with 81 columns?" Tags="|machine-learning|python|classification|scikit-learn|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15297" PostTypeId="2" ParentId="15292" CreationDate="2016-11-23T02:36:35.300" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I would &lt;strong&gt;not&lt;/strong&gt; definitely recommend substituting missing values by mean or by median or mode. If you want to go through some techniques and get a glance at them, I would recommend going through this &lt;a href=&quot;http://www.ats.ucla.edu/stat/sas/seminars/missing_data/mi_new_1.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; and for imputation techniques &lt;a href=&quot;https://en.wikipedia.org/wiki/Imputation_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; wiki page gives you a brief .&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you think that there is a way to predict missing values from the other cells. If yes, apply a regression model on those variables and estimate the missing value. But remember this lacks variability as values fall on the regression line itself. There are methods like regression imputation which can add this variability component to the estimated value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you are unable to go anywhere from the previous step, then see how the values are distributed for the missing variable, substitute them according to that distribution by using a random function.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And if your unable to perform any of the above mentioned ones and want to go by mean, median. I cant really give my opinion as in this case they are nearer to each other. See what gives you the best predictable and decide between them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Coming to your final question, Std. deviation just shows how far your values are falling away from mean. If your data has large range with good enough number of  points distributed at extremes, you would be expected to have high std. deviation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-11-23T02:36:35.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15298" PostTypeId="2" ParentId="15266" CreationDate="2016-11-23T02:57:09.147" Score="1" Body="&lt;p&gt;Try &lt;a href=&quot;https://www.google.si/search?q=matrix%20factorization%20recommender%20systems&quot; rel=&quot;nofollow noreferrer&quot;&gt;matrix/tensor factorization&lt;/a&gt; methods and &lt;a href=&quot;http://helikoid.si/bc215/bc2-handouts.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;data fusion&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2016-11-23T02:57:09.147" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15299" PostTypeId="2" ParentId="15284" CreationDate="2016-11-23T03:39:29.067" Score="1" Body="&lt;p&gt;You can find various implementations at Laurens van der Maaten's page here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE by Laurens van der Maaten&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17501" LastActivityDate="2016-11-23T03:39:29.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15301" PostTypeId="1" CreationDate="2016-11-23T03:51:08.543" Score="2" ViewCount="3547" Body="&lt;p&gt;According to the documentation - CRAN&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Boruta is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); by default, Boruta uses Random&#xA;  Forest.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Boruta documentation&quot;&gt;Boruta documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The package has one parameter &lt;strong&gt;getImp&lt;/strong&gt; that defines the importance to be used, which by default is random forests from the Ranger package. So theoretically one could use xgboost's &lt;strong&gt;xgb.importance()&lt;/strong&gt; function as the source of feature importance but I cannot find an example of how to use such parameter in practice because so far I've been unlucky. Is it possible at all to do this with the &lt;strong&gt;Boruta package&lt;/strong&gt; or do I have implement from scratch the Boruta algorithm again using xgboost? And if it's possible how?&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2016-11-23T12:46:36.237" Title="Boruta feature selection in R with custom importance (xgboost feature importance)" Tags="|r|feature-selection|feature-extraction|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15302" PostTypeId="1" CreationDate="2016-11-23T06:39:55.703" Score="0" ViewCount="184" Body="&lt;p&gt;I have a basic multilabel topic classifier(Tfidf vectorizer with OneVsRest Classifier) built on some customer reviews. I observed that there are some classes with right features but it still predicts with very low confidence.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the reason for such a classification in general?&lt;/li&gt;&#xA;&lt;li&gt;How do I boost the confidence?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2019-12-25T00:56:15.540" Title="Multilabel Classification - increasing the confidence" Tags="|machine-learning|multilabel-classification|probability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15303" PostTypeId="2" ParentId="15284" CreationDate="2016-11-23T07:02:26.663" Score="2" Body="&lt;p&gt;It simply means that you should set the bandwidths through &lt;a href=&quot;https://en.wikipedia.org/wiki/Binary_search_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary search&lt;/a&gt;. The way it works is that you start with a preset target perplexity (&lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mark's link&lt;/a&gt; suggests values from 5 to 50 as reasonable values), and bounds for the bandwidth. If the target perplexity is inside the interval defined by the boundary perplexities, you iteratively halve the search space until you converge to the target:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2^{H(p; \sigma_L)} &amp;lt; PP_\mathrm{target} &amp;lt; 2^{H(p; \sigma_U)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the target was not in the initial interval, you expand the interval and try again.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-11-23T11:29:15.543" LastActivityDate="2016-11-23T11:29:15.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15304" PostTypeId="2" ParentId="15302" CreationDate="2016-11-23T07:36:39.497" Score="2" Body="&lt;p&gt;It's hard to say without knowing the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often times the classes are imbalanced. This could cause your problems.&#xA;So let's say that the &quot;features are right&quot; for 5% of your data and they belong to class $A$. You would expect the 5% to be classified correctly as $A$ almost all the time. But what about the remaining 95% in class $B$? The classifier learns that it would be often correct if the sample is classified as $B$. This leads to a low classification rate for class $A$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can tackle this by setting different miss-classification costs for the classes. Often you multiply the error rate by these costs to scale them accordingly. Some classifiers let you set this during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The standard case would be as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$C = \left( \begin{matrix}&#xA;0 &amp;amp; 1  \\&#xA;1 &amp;amp; 0  \\&#xA;\end{matrix} \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meaning that correct classification (diagonal) is not punished and miss-classifying $A$ as $B$ and vice versa is punished in the same way. However, this assumes uniformly distributed classes. To account for the case explained above, you may set it the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$C = \left( \begin{matrix}&#xA;0 &amp;amp; 0.05  \\&#xA;0.95 &amp;amp; 0  \\&#xA;\end{matrix} \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here i set it as the inverse of the class distribution. However, you may try different settings.&lt;/p&gt;&#xA;" OwnerUserId="26146" LastActivityDate="2016-11-23T07:36:39.497" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15305" PostTypeId="1" AcceptedAnswerId="15306" CreationDate="2016-11-23T09:46:50.240" Score="16" ViewCount="24883" Body="&lt;p&gt;So from Algorithm 3 of &lt;a href=&quot;https://arxiv.org/pdf/1603.02754v3.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1603.02754v3.pdf&lt;/a&gt;, it says that an optimum default direction is determined and the missing values will go in that direction. However, or perhaps I have misunderstood/missed the explanation from the article, it doesn't say what exactly is the input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have (parent) node A (with 50 inputs) splitting into node B and node C. Now, say of the 50 inputs there are 7 missing values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other 43 inputs are split into B and C accordingly. What I seem to be understanding is that, it will allocate the remaining 7 into B and C and determine which one gives a higher gain score; that will be the optimal direction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, given the 7 values are missing (Which means I don't know what are these 7 values), how does allocating missing values into any of the child nodes change the gain score, or rather minimize the loss function? This seems to suggest that Xgboost is inputting something for the missing values. I can't seem to find out what is Xgboost inputting for these missing values. I hope this question isn't too vague/general and easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I think &quot;Missing values&quot; may be a vague term. What I meant here is (From wiki) &quot;In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the author himself (&lt;a href=&quot;https://github.com/dmlc/xgboost/issues/21&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/dmlc/xgboost/issues/21&lt;/a&gt;), he said&#xA;&quot; tqchen commented on Aug 13, 2014&#xA;xgboost naturally accepts sparse feature format, you can directly feed data in as sparse matrix, and only contains non-missing value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. features that are not presented in the sparse feature matrix are treated as 'missing'. XGBoost will handle it internally and you do not need to do anything on it.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot; tqchen commented on Aug 13, 2014&#xA;Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically &quot;learn&quot; what is the best imputation value for missing values based on reduction on training loss.&quot;&lt;/p&gt;&#xA;" OwnerUserId="26409" LastEditorUserId="26409" LastEditDate="2016-11-23T11:01:42.683" LastActivityDate="2016-11-23T11:31:52.263" Title="How does Xgboost learn what are the inputs for missing values?" Tags="|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15306" PostTypeId="2" ParentId="15305" CreationDate="2016-11-23T10:20:59.150" Score="21" Body="&lt;p&gt;The procedure is described in &lt;a href=&quot;https://arxiv.org/pdf/1603.02754v3.pdf&quot; rel=&quot;noreferrer&quot;&gt;their paper, section 3.4: Sparsity aware split-finding&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split point possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you have only one binary feature $x$, and your data can be split in three groups:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Group $B$: 20 observations such that $x=B$,&lt;/li&gt;&#xA;&lt;li&gt;Group $C$: 20 observations such that $x=C$,&lt;/li&gt;&#xA;&lt;li&gt;Group $M$: 10 observations such that $x=?$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The algorithm will split based on $x$, but does not know where to send the group $M$. It will try both assignments,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(B,  M), C \qquad \text{and} \qquad B, (C, M),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;compute the value to be assigned to the prediction at each node using all the data, and chose the assignment that minimizes the loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the split $(B,  M), C$ is chosen, the value of the left node will have been computed from all the $B$ and $M$ samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what is meant by&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Automatically &quot;learn&quot; what is the best imputation value for missing values based on reduction on training loss.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Comment question&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;how can group M be calculated if I don't know what are the values for them? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;value&lt;/code&gt; we compute is not based on the feature $x$, which we don't know, but based on the known label/target of the sample. Let say we are doing a regression and trying to minimize the mean square error, and let say we have the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For group $B$, the mean of the target is $5$.&lt;/li&gt;&#xA;&lt;li&gt;For group $C$, the mean of the target is $10$.&lt;/li&gt;&#xA;&lt;li&gt;For the missing value group $M$, the mean of the target is $0$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that even if the value of $x$ is missing, we know the value of target of the sample, otherwise we could not use it to train our model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, the split would be made on $(B,M), C$, the value assigned to the right node containing samples $C$ would be $10$, and the value assigned to the left node containing samples in $(B, M)$ would be the mean of the target for the whole group. In our example, the mean would be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{|M|}{|B| + |M|}\text{mean}(M) + \frac{|B|}{|M|+|B|}\text{mean}(B) = \frac{10}{30}0 + \frac{20}{30}5 = 3.\overline{3}$$&lt;/p&gt;&#xA;" OwnerUserId="15501" LastEditorUserId="15501" LastEditDate="2016-11-23T11:31:52.263" LastActivityDate="2016-11-23T11:31:52.263" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="15307" PostTypeId="1" AcceptedAnswerId="15315" CreationDate="2016-11-23T10:22:55.987" Score="7" ViewCount="3742" Body="&lt;p&gt;I'm trying to implement convolutional autoencoders in tensorflow, on the mnist dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the autoencoder does not seem to learn properly: it will always learn to reproduce the 0 shape, but no other shapes, in fact I usually get an average loss of about 0.09, which is 1/10 of the classes that it should learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using 2x2 kernels with stride 2 for the input and output convolutions, but the filters seems to be learned properly. When I visualize the data, input image is passed thru 16 (1st conv) and 32 filters (2nd conv), and by image inspection it seems running fine (i.e. apparently features like curves, crosses, etc are detected).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem seems to arise in the fully connected part of the network: no matter what is the input image, its encoding will be &lt;em&gt;always&lt;/em&gt; the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought is &quot;I'm probably just feeding it with zeroes while training&quot;, but I don't think I made this mistake (see code below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; I realized the dataset was not shuffled, which introduced a bias and could be the cause of the problem. After introducing it, the average loss is lower (0.06 instead of 0.09), and in fact the output image looks like a blurry 8, but conclusions are the same: the encoded input will be the same no matter what is the input image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here a sample input with the relative output&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pfLTM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pfLTM.png&quot; alt=&quot;A sample input with the relative output&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the activation for the image above, with the two fully connected layers at the bottom (encoding is the bottommost).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/T2l41.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/T2l41.png&quot; alt=&quot;activations&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, here there are the activation for the fully connected layers for different inputs. Each input image corresponds to a line in the activation images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, they always yield the same output. If I use transposed weights instead of initializing different ones, the first FC layer (image in the middle) looks a bit more randomized, but the underlying pattern is still evident. In the encoding layer (image at the bottom), the output will be always the same no matter what is the input (of course, the pattern varies from one training and the next).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/msKZU.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/msKZU.png&quot; alt=&quot;FC layers activation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the relevant code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# A placeholder for the input data&#xA;x = tf.placeholder('float', shape=(None, mnist.data.shape[1]))&#xA;&#xA;# conv2d_transpose cannot use -1 in output size so we read the value&#xA;# directly in the graph&#xA;batch_size = tf.shape(x)[0]&#xA;&#xA;# Variables for weights and biases&#xA;with tf.variable_scope('encoding'):&#xA;    # After converting the input to a square image, we apply the first convolution, using 2x2 kernels&#xA;    with tf.variable_scope('conv1'):&#xA;        wec1 = tf.get_variable('w', shape=(2, 2, 1, m_c1), initializer=tf.truncated_normal_initializer())&#xA;        bec1 = tf.get_variable('b', shape=(m_c1,), initializer=tf.constant_initializer(0))&#xA;    # Second convolution&#xA;    with tf.variable_scope('conv2'):&#xA;        wec2 = tf.get_variable('w', shape=(2, 2, m_c1, m_c2), initializer=tf.truncated_normal_initializer())&#xA;        bec2 = tf.get_variable('b', shape=(m_c2,), initializer=tf.constant_initializer(0))&#xA;    # First fully connected layer&#xA;    with tf.variable_scope('fc1'):&#xA;        wef1 = tf.get_variable('w', shape=(7*7*m_c2, n_h1), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bef1 = tf.get_variable('b', shape=(n_h1,), initializer=tf.constant_initializer(0))&#xA;    # Second fully connected layer&#xA;    with tf.variable_scope('fc2'):&#xA;        wef2 = tf.get_variable('w', shape=(n_h1, n_h2), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bef2 = tf.get_variable('b', shape=(n_h2,), initializer=tf.constant_initializer(0))&#xA;&#xA;reshaped_x = tf.reshape(x, (-1, 28, 28, 1))&#xA;y1 = tf.nn.conv2d(reshaped_x, wec1, strides=(1, 2, 2, 1), padding='VALID')&#xA;y2 = tf.nn.sigmoid(y1 + bec1)&#xA;y3 = tf.nn.conv2d(y2, wec2, strides=(1, 2, 2, 1), padding='VALID')&#xA;y4 = tf.nn.sigmoid(y3 + bec2)&#xA;y5 = tf.reshape(y4, (-1, 7*7*m_c2))&#xA;y6 = tf.nn.sigmoid(tf.matmul(y5, wef1) + bef1)&#xA;encode = tf.nn.sigmoid(tf.matmul(y6, wef2) + bef2)&#xA;&#xA;with tf.variable_scope('decoding'):&#xA;    # for the transposed convolutions, we use the same weights defined above&#xA;    with tf.variable_scope('fc1'):&#xA;        #wdf1 = tf.transpose(wef2)&#xA;        wdf1 = tf.get_variable('w', shape=(n_h2, n_h1), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bdf1 = tf.get_variable('b', shape=(n_h1,), initializer=tf.constant_initializer(0))&#xA;    with tf.variable_scope('fc2'):&#xA;        #wdf2 = tf.transpose(wef1)&#xA;        wdf2 = tf.get_variable('w', shape=(n_h1, 7*7*m_c2), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bdf2 = tf.get_variable('b', shape=(7*7*m_c2,), initializer=tf.constant_initializer(0))&#xA;    with tf.variable_scope('deconv1'):&#xA;        wdd1 = tf.get_variable('w', shape=(2, 2, m_c1, m_c2), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bdd1 = tf.get_variable('b', shape=(m_c1,), initializer=tf.constant_initializer(0))&#xA;    with tf.variable_scope('deconv2'):&#xA;        wdd2 = tf.get_variable('w', shape=(2, 2, 1, m_c1), initializer=tf.contrib.layers.xavier_initializer())&#xA;        bdd2 = tf.get_variable('b', shape=(1,), initializer=tf.constant_initializer(0))&#xA;&#xA;u1 = tf.nn.sigmoid(tf.matmul(encode, wdf1) + bdf1)&#xA;u2 = tf.nn.sigmoid(tf.matmul(u1, wdf2) + bdf2)&#xA;u3 = tf.reshape(u2, (-1, 7, 7, m_c2))&#xA;u4 = tf.nn.conv2d_transpose(u3, wdd1, output_shape=(batch_size, 14, 14, m_c1), strides=(1, 2, 2, 1), padding='VALID')&#xA;u5 = tf.nn.sigmoid(u4 + bdd1)&#xA;u6 = tf.nn.conv2d_transpose(u5, wdd2, output_shape=(batch_size, 28, 28, 1), strides=(1, 2, 2, 1), padding='VALID')&#xA;u7 = tf.nn.sigmoid(u6 + bdd2)&#xA;decode = tf.reshape(u7, (-1, 784))&#xA;&#xA;loss = tf.reduce_mean(tf.square(x - decode))&#xA;opt = tf.train.AdamOptimizer(0.0001).minimize(loss)&#xA;&#xA;try:&#xA;    tf.global_variables_initializer().run()&#xA;except AttributeError:&#xA;    tf.initialize_all_variables().run() # Deprecated after r0.11&#xA;&#xA;print('Starting training...')&#xA;bs = 1000 # Batch size&#xA;for i in range(501): # Reasonable results around this epoch &#xA;    # Apply permutation of data at each epoch, should improve convergence time&#xA;    train_data = np.random.permutation(mnist.data)&#xA;    if i % 100 == 0:&#xA;        print('Iteration:', i, 'Loss:', loss.eval(feed_dict={x: train_data}))&#xA;    for j in range(0, train_data.shape[0], bs):&#xA;        batch = train_data[j*bs:(j+1)*bs]&#xA;        sess.run(opt, feed_dict={x: batch})&#xA;        # TODO introduce noise&#xA;print('Training done')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26257" LastEditorUserId="29169" LastEditDate="2021-02-16T15:29:42.300" LastActivityDate="2021-02-16T15:29:42.300" Title="Convolutional autoencoders not learning" Tags="|tensorflow|convolutional-neural-network|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15308" PostTypeId="2" ParentId="15301" CreationDate="2016-11-23T12:20:20.013" Score="6" Body="&lt;p&gt;Boruta is by universal reputation dog-slow and not very good. Boruta runs take many hours or days. VIF feature-selection algorithm is not objective, anyway.&#xA;You can program your own feature-selection that runs faster. I ran Boruta a few times on various datasets and it wasted 4 days of my time, and the result was inconclusive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the fast-and-dirty not-100%-scientific approach, culled from consensus in a lot of Kaggle competitions and suchlike. It's 10-1000x faster than Boruta and probably more accurate.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Run some fast exploratory trees (RF/XGB), i.e. not very deep, not too many trees. Extract feature importances. Repeat for several random seeds and look at average or absmax feature importances.&lt;/li&gt;&#xA;&lt;li&gt;For now, throw out very-high-cardinality features (e.g. user-ID, zipcode etc.), since trees won't tend to split on them until very deep and small nodes, so initially exclude them. Let's define &quot;Very-high-cardinality&quot; as &gt;= ~0.3 distinct values per record, or less in a larger dataset.&lt;/li&gt;&#xA;&lt;li&gt;Throw out all near-zero features, zero- and very-low-importance features (&amp;lt; 0.02-0.05 f.i. or lower is a rule of thumb).&lt;/li&gt;&#xA;&lt;li&gt;Measure the matrix of which features are highly correlated (&amp;lt; 0.4) to each other. Corrplot is a nice way to visualize these, and order correlated features near each other). Pick a sensible color-scale. Identify subgroups of correlated features. Arbitrarily select one feature from those groups (you can revisit this selection later). Or you can decide if this warrants using PCA.&lt;/li&gt;&#xA;&lt;li&gt;Rerun, your CV accuracy should have gone way up, and your training time will be &lt;em&gt;way&lt;/em&gt; faster (since tree training is quadratic to no. of features). Revisit choices of borderline-low-important or correlated feature, and again use multiple random seeds.&lt;/li&gt;&#xA;&lt;li&gt;For exploratory, 5-fold CV is fine, but if training time is an issue, 3-fold works. 10-fold is unnecessary.&lt;/li&gt;&#xA;&lt;li&gt;For the subset of remaining features, use a 'respectable' method. See the excellent posts on &lt;a href=&quot;http://stats.stackexchange.com&quot;&gt;CrossValidated&lt;/a&gt; in feature-selection.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Somebody with a PhD in Stats will probably wag their finger at me for this :)&lt;/p&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="8501" LastEditDate="2016-11-23T12:46:36.237" LastActivityDate="2016-11-23T12:46:36.237" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15309" PostTypeId="1" AcceptedAnswerId="15653" CreationDate="2016-11-23T13:25:52.420" Score="0" ViewCount="2700" Body="&lt;p&gt;I was wondering mathematically how Precision Recall curve is plotted. How is this curve useful?&lt;/p&gt;&#xA;" OwnerUserId="16971" LastActivityDate="2018-09-17T16:16:08.447" Title="Plotting Precision Recall Curve" Tags="|information-retrieval|model-evaluations|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15310" PostTypeId="2" ParentId="12979" CreationDate="2016-11-23T13:55:37.067" Score="0" Body="&lt;p&gt;Adding to the answer given by @Jan van der Vegt, I would recommend you to go through the basics of &lt;a href=&quot;https://en.wikipedia.org/wiki/Grayscale&quot; rel=&quot;nofollow noreferrer&quot;&gt;gray scale imaging&lt;/a&gt; and of color palettes and how those are represented in matrices in python or in general. Simultaneously work on python scipy to see how images are being represented. Going through some basics like resizing, histograms etc. in theory will be very helpful for a better understanding and representation of imaging before you put yourself into coding something.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-11-23T13:55:37.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15311" PostTypeId="1" AcceptedAnswerId="15916" CreationDate="2016-11-23T15:02:08.460" Score="2" ViewCount="6128" Body="&lt;p&gt;I have a set of 270 documents. I have a list of keywords that are spread across these 270 documents. I would like to rank these keywords based on the &lt;strong&gt;TR-IDF method&lt;/strong&gt;. From what I read, this method helps to determine the importance of a word in a particular document as the TR of a word is of one particular document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone tell me how can I extend this approach to get the importance of a word across all 270 documents?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Eg:&lt;/strong&gt; The word &quot;abc&quot; and &quot;xyz&quot; and spread across all the documents. So&#xA; I can have a document matrix(word and the tr-idf) for &quot;abc&quot; and &quot;xyz&quot;&#xA; for all the documents. Now how do I decide which word is important&#xA; overall, not just in one document?&lt;/p&gt;&#xA;" OwnerUserId="26410" LastEditorUserId="11097" LastEditDate="2016-11-24T07:33:51.170" LastActivityDate="2021-04-28T03:42:06.690" Title="Weights for keywords in a set of documents using Term Frequency and Inverse Document Frequency" Tags="|machine-learning|data-mining|clustering|statistics|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15313" PostTypeId="1" CreationDate="2016-11-23T16:23:03.733" Score="12" ViewCount="5878" Body="&lt;p&gt;I'm currently trying to train a model on a large csv file (&gt;70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples. &#xA;I'm struggling in understanding how this function actually reads the data. If I'm using a batch size of e.g. 50.000, does it read the first 50.000 lines of the file? If I want to loop over the whole file (1 epoch) do I have to use num_rows/batch_size = 1.200 number of steps for the estimator.fit method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the input function im currently using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def input_fn(file_names, batch_size):&#xA;    # Read csv files and create examples dict&#xA;    examples_dict = read_csv_examples(file_names, batch_size)&#xA;&#xA;    # Continuous features&#xA;    feature_cols = {k: tf.string_to_number(examples_dict[k],&#xA;                                           out_type=tf.float32) for k in CONTINUOUS_COLUMNS}&#xA;&#xA;    # Categorical features&#xA;    feature_cols.update({&#xA;                            k: tf.SparseTensor(&#xA;                                indices=[[i, 0] for i in range(examples_dict[k].get_shape()[0])],&#xA;                                values=examples_dict[k],&#xA;                                shape=[int(examples_dict[k].get_shape()[0]), 1])&#xA;                            for k in CATEGORICAL_COLUMNS})&#xA;&#xA;    label = tf.string_to_number(examples_dict[LABEL_COLUMN], out_type=tf.int32)&#xA;&#xA;    return feature_cols, label&#xA;&#xA;&#xA;def read_csv_examples(file_names, batch_size):&#xA;    def parse_fn(record):&#xA;        record_defaults = [tf.constant([''], dtype=tf.string)] * len(COLUMNS)&#xA;&#xA;        return tf.decode_csv(record, record_defaults)&#xA;&#xA;    examples_op = tf.contrib.learn.read_batch_examples(&#xA;        file_names,&#xA;        batch_size=batch_size,&#xA;        queue_capacity=batch_size*2.5,&#xA;        reader=tf.TextLineReader,&#xA;        parse_fn=parse_fn,&#xA;        #read_batch_size= batch_size,&#xA;        #randomize_input=True,&#xA;        num_threads=8&#xA;    )&#xA;&#xA;    # Important: convert examples to dict for ease of use in `input_fn`&#xA;    # Map each header to its respective column (COLUMNS order&#xA;    # matters!&#xA;    examples_dict_op = {}&#xA;    for i, header in enumerate(COLUMNS):&#xA;        examples_dict_op[header] = examples_op[:, i]&#xA;&#xA;    return examples_dict_op&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the code im using to train the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train_and_eval():&#xA;&quot;&quot;&quot;Train and evaluate the model.&quot;&quot;&quot;&#xA;&#xA;m = build_estimator(model_dir)&#xA;m.fit(input_fn=lambda: input_fn(train_file_name, batch_size), steps=steps)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would happen if I would call the fit function again with the same input_fn. Does it start at the beginning of the file again, or will it remember the line where it has stopped last time?&lt;/p&gt;&#xA;" OwnerUserId="26425" LastEditorUserId="-1" LastEditDate="2016-11-25T08:12:20.370" LastActivityDate="2018-03-21T09:41:53.743" Title="Train on batches in Tensorflow" Tags="|python|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15314" PostTypeId="1" CreationDate="2016-11-23T16:33:35.720" Score="4" ViewCount="5892" Body="&lt;p&gt;I aint no data scientist/machine learner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What Im Lookin for&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text = &quot;Donald Trump became the president of America&quot;&#xA;#some data science works&#xA;print(topics) &#xA;#prints [&quot;politics&quot;]&#xA;&#xA;text = &quot;Rihanna is starring in the new movie The Inception 2&quot;&#xA;#some data science works&#xA;print(topics) &#xA;#prints [&quot;movie&quot;,&quot;music&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;What I can do&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I can extract words like &lt;code&gt;Donald Trump&lt;/code&gt;,&lt;code&gt;America&lt;/code&gt;,&lt;code&gt;Rihanna&lt;/code&gt; using POS&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can get huge &lt;strong&gt;paragraphs/lists&lt;/strong&gt; of &lt;em&gt;articles/words&lt;/em&gt; on &lt;code&gt;politics&lt;/code&gt;,&lt;code&gt;movies&lt;/code&gt; etc and save them in a text file.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What i can't do&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Make meaningful topics out of  those words like &lt;code&gt;sports&lt;/code&gt;,&lt;code&gt;politics&lt;/code&gt;,&lt;code&gt;movies&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What i want you do to&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Point me in the right specific direction towards solving this problem aka enlighten me&lt;/p&gt;&#xA;" OwnerUserId="26426" LastActivityDate="2017-08-26T00:32:20.760" Title="NLTK Sklearn Genism Text to Topic" Tags="|python|text-mining|scikit-learn|topic-model|nltk|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15315" PostTypeId="2" ParentId="15307" CreationDate="2016-11-23T17:21:03.100" Score="2" Body="&lt;p&gt;Well, the problem was mainly related to the kernel size. Using 2x2 convolution with stride of (2,2) turned to be a bad idea. Using 5x5 and 3x3 sizes yielded decent results.&lt;/p&gt;&#xA;" OwnerUserId="26257" LastActivityDate="2016-11-23T17:21:03.100" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15316" PostTypeId="1" CreationDate="2016-11-23T17:53:45.863" Score="1" ViewCount="1787" Body="&lt;p&gt;Is there a closed form solution for a ReLU that is bounded with a maximum value at 1?&#xA;I am trying to produce output values for pixel intensities 0 &amp;lt;= x &amp;lt;= 1, but my outputs are producing values greater than 1. How can I counteract this?&lt;/p&gt;&#xA;" OwnerUserId="26427" LastActivityDate="2016-11-23T20:01:49.007" Title="Bounded ReLU at 1 if x&gt;1?" Tags="|machine-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15317" PostTypeId="2" ParentId="15316" CreationDate="2016-11-23T18:23:55.483" Score="4" Body="&lt;p&gt;Being itself a non-linearity, the very advantage of ReLUs is their linear regime, that prevents it from saturating too early.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, a couple of alternatives for your case are to either clip the output of the ReLUs or to place sigmoid (or tanh, adjusting the output range) after them (or instead of them).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-11-23T18:23:55.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15318" PostTypeId="2" ParentId="15316" CreationDate="2016-11-23T20:01:49.007" Score="0" Body="&lt;p&gt;Use a sigmoid or tanh. I think it's pointless to use a clipped ReLU in your case also because, in addition to what @ncasas said, for values less than zero or greater than your clipping threshold (i.e. 1) you would not get a gradient, possibly making learning harder.&lt;/p&gt;&#xA;" OwnerUserId="26257" LastActivityDate="2016-11-23T20:01:49.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15319" PostTypeId="1" AcceptedAnswerId="15320" CreationDate="2016-11-23T21:01:01.640" Score="1" ViewCount="11572" Body="&lt;p&gt;I'm a beginner in Spark and I want to calculate the average of number per name. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a JSON file with this information&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = spark.read.json(&quot;myjson.json&quot;)&#xA;df.select(avg(df[&quot;number&quot;]), df[&quot;name&quot;]).groupBy(&quot;name&quot;).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I'm doing it wrong..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I solve my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot&lt;/p&gt;&#xA;" OwnerUserId="26432" LastActivityDate="2016-11-23T21:45:43.113" Title="Dataframe request with groupBy" Tags="|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15320" PostTypeId="2" ParentId="15319" CreationDate="2016-11-23T21:45:43.113" Score="0" Body="&lt;p&gt;You are probably thinking in terms of regular SQL but spark sql is a bit different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll need to group by field before performing your aggregation. Thus the following, you can write your query as followed :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupBy('name').agg({'number': 'mean'}).show&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also write it using a SQL dialect by registering the DataFrame as a temp table and then query on it use the SQLContext or HiveContext : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.registerTempTable(&quot;people&quot;)&#xA;df2 = sqlContext.sql(&quot;select name, avg(number) as average from people group by name&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope that this answers your question. &lt;/p&gt;&#xA;" OwnerUserId="5177" LastActivityDate="2016-11-23T21:45:43.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15322" PostTypeId="1" CreationDate="2016-11-23T22:33:24.623" Score="2" ViewCount="239" Body="&lt;p&gt;I use random forest to train on my data (My data had imbalance in the target class, i.e. rare 1 and abundant 0). I face 3 issues about the stability of estimator and its prediction power. I think these problems could be common on many machine learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I found the &lt;code&gt;ROC_AUC_Score&lt;/code&gt; highly changeable when I resample the training set (the rest was test set). It can vary from $0.85$ to $0.45$ while changing the training set. &lt;/li&gt;&#xA;&lt;li&gt;The parameter tuning can also cause the move of the estimator and &lt;code&gt;ROC_AUC_Score&lt;/code&gt; but the effect was weaker than the first case above. &lt;/li&gt;&#xA;&lt;li&gt;When running some iteration of model fitting, the results were also different from each other but the effect was the weakest. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For 2, 3, I think we can get the best parameter setting and fitting results by recording each iteration and parameter tuning. (Maybe can do it more efficiently, please advice)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please also advice how to deal with the first problem to make the fitting convinced and reliable? Cross Validated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26146" LastEditDate="2016-11-25T08:12:01.317" LastActivityDate="2017-01-25T19:49:26.350" Title="Random Forest Modelling?" Tags="|machine-learning|python|r|data-mining|random-forest|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15323" PostTypeId="2" ParentId="15322" CreationDate="2016-11-23T23:53:09.343" Score="0" Body="&lt;p&gt;I think, it's a classic class imbalance problem. If it's possible, collect more data of the class with sparse amount of data or if you have sufficient amount of data, omit some of the data of the class with large amount of data. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2016-11-23T23:53:09.343" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15325" PostTypeId="1" CreationDate="2016-11-24T01:17:49.310" Score="1" ViewCount="632" Body="&lt;p&gt;This stack exchange post - &lt;a href=&quot;https://stats.stackexchange.com/questions/80507/what-is-a-gaussian-discriminant-analysis-gda&quot;&gt;https://stats.stackexchange.com/questions/80507/what-is-a-gaussian-discriminant-analysis-gda&lt;/a&gt; - discusses GDA, a machine learning method for classification. I would like to implement something like this analysis in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen posts for discriminant analysis in R using linear and quadratic discriminant analysis (&lt;a href=&quot;http://rstudio-pubs-static.s3.amazonaws.com/35817_2552e05f1d4e4db8ba87b334101a43da.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://rstudio-pubs-static.s3.amazonaws.com/35817_2552e05f1d4e4db8ba87b334101a43da.html&lt;/a&gt;) but nothing for GDA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess my questions are, because I'm not 100% familiar with these methods, (1) whether GDA is different from or a subset of LDA and QDA, and if there is an R package / functions for GDA in R like there are for LDA and QDA (the lda() and qda() functions in the MASS package). Maybe I should be asking if I actually want to be performing GDA with my data (I'm not actually sure if the joint distribution of the features follow a gaussian dist), or if lda or qda may be better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="26445" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-11-24T05:00:42.577" Title="Gaussian Discriminant Analysis (GDA) package in R" Tags="|machine-learning|r|discriminant-analysis|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15326" PostTypeId="2" ParentId="15322" CreationDate="2016-11-24T02:35:50.473" Score="0" Body="&lt;p&gt;I am not sure what the ratio of imbalance is, and with which package of RF or the one you wrote by yourself you have experienced , but I think you can try it again in two ways to solve the problem of stability:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It's obviously that your problem of &lt;strong&gt;imbalance is severe&lt;/strong&gt; , you should keep it balanced as much as you can do.&lt;/li&gt;&#xA;&lt;li&gt;If you have not &lt;strong&gt;sampled features&lt;/strong&gt; for different trees in RF, namely different features for different trees , you can try it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hopes this will help you, good luck !&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-11-24T02:35:50.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15327" PostTypeId="1" AcceptedAnswerId="15409" CreationDate="2016-11-24T09:44:47.397" Score="0" ViewCount="378" Body="&lt;p&gt;I have two RDDs named &quot;&lt;strong&gt;data&lt;/strong&gt;&quot; and &quot;&lt;strong&gt;model&lt;/strong&gt;&quot;, they are repartitioned by key described as below :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/je8Up.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/je8Up.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does the tuple records with the same key reside in the same node in my cluster ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Should it save IO cost in shuffle operation, such as &quot;&lt;em&gt;data.cogroup(model)&lt;/em&gt;&quot; , if it comes true ?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-11-29T21:15:56.850" Title="Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?" Tags="|machine-learning|bigdata|apache-spark|parallel|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15328" PostTypeId="1" CreationDate="2016-11-24T11:10:08.060" Score="23" ViewCount="31930" Body="&lt;p&gt;The paper &lt;a href=&quot;https://arxiv.org/pdf/1409.4842v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;Going deeper with convolutions&lt;/a&gt; describes GoogleNet which contains the original inception modules:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zTinD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zTinD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The change to inception v2 was that they replaced the 5x5 convolutions by two successive 3x3 convolutions and applied pooling:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eindA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eindA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between Inception v2 and Inception v3?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:29:28.697" LastActivityDate="2021-02-16T15:29:28.697" Title="What is the difference between Inception v2 and Inception v3?" Tags="|image-classification|computer-vision|convolutional-neural-network|inception|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15330" PostTypeId="1" AcceptedAnswerId="15334" CreationDate="2016-11-24T13:36:30.580" Score="1" ViewCount="78" Body="&lt;p&gt;First of all I don't know if this is the appropriate place to post this question. If it's not, I apologize in advance. It seems like the most relevant Stack Exchange sub.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little about myself:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a software developer and have been working in the field for about 5 years. I have a two year degree from a community college that didn't involve much math above pre-calc high school level stuff. I've never taken a statistics course. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the software I write is business software. It's typically back end server or database stuff, never user facing. I do allot of data aggregation and normalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lately I've been thinking about expanding my skillset. Data and statistics have always interested and I'd like to learn ways to measure and interpret data. Ideally this would lead to thinking of useful metrics and uses for data that would help the company I work at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is I'm not sure what I should be looking into. There are so many data science topics and I don't know what would be useful for me to learn. I prefer structured learning environments such as courses or video series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this isn't a great question... I'm just not sure where to ask and I'm not sure &lt;strong&gt;how&lt;/strong&gt; to ask for what I'm looking for... or even sure &lt;strong&gt;what&lt;/strong&gt; I am looking for. Any guidance or suggestions would be very much appreciated. Even if the suggestion is a better place to ask the question or a better question for me to ask. Thanks much. &lt;/p&gt;&#xA;" OwnerUserId="26464" LastActivityDate="2016-11-24T15:04:34.760" Title="Topics to cover for software developer interested in data analytics" Tags="|self-study|learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15332" PostTypeId="1" CreationDate="2016-11-24T14:37:28.680" Score="1" ViewCount="636" Body="&lt;p&gt;I have a set of climate data (temperature, pressure and moisture for example), $X$, $Y$, $Z$ which are matricies with dimensions $n \times p$ where $n$ is the number of observations and $p$ is the number of spatial points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Previously, to investigate modes of variability in dataset $X$, I simply performed a empirical orthogonal function (EOF) analysis OR Principle component Analysis (PCA) on $X$. This involved decomposing (via SVD), the matrix $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To investigate the coupling of the modes of variability of $X$ and $Y$, I used maximum covariance analysis (MCA) which involved decomposing a covariance matrix proportional to $XY^{T}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if I wish to looked at all three datasets, how do I go about doing this? One idea I had was to form a fourth matrix, $\Sigma$, which will be the 'feature' concatenation of the three datasets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Sigma = [X, Y, Z]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so that my matrix $\Sigma$ will have dimensions $n \times 3p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would then use standard PCA/EOF analysis and use SVD to decompose this matrix $\Sigma$ and then I would obtain modes of variabiilty with size $3p \times 1$ and thus subsequently the mode associated with $X$ is the first $p$ values, the mode associated with $Y$ is the second set of $p$ values and the mode associated with $Z$ is the last $p$ values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? Or can anyone suggest a better way of looking at the coupling of all three (or more) datasets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much!&lt;/p&gt;&#xA;" OwnerUserId="26466" LastActivityDate="2016-11-24T14:37:28.680" Title="EOF/PCA/MCA Analysis for a set of data" Tags="|machine-learning|data-mining|correlation|pca|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15333" PostTypeId="2" ParentId="9179" CreationDate="2016-11-24T14:41:32.617" Score="1" Body="&lt;p&gt;To see what you can do considering the size(area) of objects, the android app &lt;a href=&quot;https://play.google.com/store/apps/details?id=com.exa.pgomes.photocount&amp;amp;hl=pt_BR&quot; rel=&quot;nofollow noreferrer&quot;&gt;PHOTO COUNT&lt;/a&gt; do the job using opencv 3.1 library.&lt;/p&gt;&#xA;" OwnerUserId="26468" LastEditorUserId="98307" LastEditDate="2020-08-03T13:41:35.077" LastActivityDate="2020-08-03T13:41:35.077" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15334" PostTypeId="2" ParentId="15330" CreationDate="2016-11-24T15:04:34.760" Score="1" Body="&lt;p&gt;Welcome to the community!! There can be a lot of answers to this question but I would suggest you the approach I took when I shifted from software development to the data science field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1)&lt;/strong&gt; Refresh your statistics and probability concepts. You should not go into too much details but you must understand basic things like Gaussian Distribution, Mean, Variance, Probability,etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2)&lt;/strong&gt; Go through the basics of Machine Learning concepts. I prefer Andrew Ng's machine learning course on Coursera. That will help you build a strong foundation and will give you a great start in the field&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3)&lt;/strong&gt; Choose a particular language Python/R for building models. Though it's totally upto you but I prefer python as it has great libraries for machine learning as well as Deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4)&lt;/strong&gt; Take part in competitions. We learn by doing not by taking only lectures. I suggest you should join Kaggle and the slack community out there namely, 'KaggleNoobs'. It's a great community. I learn everyday a new thing from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S: Data Science is a vast field. It demands from you various skill set like Data Analysis, Data Visualisation, Machine Learning,etc. So sometimes it can become frustrating too. But once you start enjoying it, you will become a master eventually.  &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-24T15:04:34.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15335" PostTypeId="2" ParentId="14228" CreationDate="2016-11-24T15:36:43.317" Score="2" Body="&lt;p&gt;You’ve asked your question a while ago, but maybe it helps someone anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the mllib guide they say to refer to the PrefixSpan paper by Jian Pei and his team, there’s another version fully available &lt;a href=&quot;http://idb.csie.ncku.edu.tw/tsengsm/COURSE/DM/Paper/PrefixSpan.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There, they explain that a sequence is composed of elements which are composed of items:&#xA;For example “&lt;em&gt;a sequence (a (abc)(ac)d(cf)) has five elements: a , (abc), (ac) , d and (cf)  where items a and c appear more&#xA;than once respectively in different elements.&lt;/em&gt;” (taken from the paper)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why you need a list of lists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And note that items within an element shouldn’t follow any particular order (in the paper they are assumed to be sorted alphabetically).  So if you are dealing with time series I suppose that the events arriving at the same time (so no particular order) would form the items of an element.&lt;/p&gt;&#xA;" OwnerUserId="26470" LastActivityDate="2016-11-24T15:36:43.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15336" PostTypeId="1" CreationDate="2016-11-24T17:29:10.990" Score="3" ViewCount="920" Body="&lt;p&gt;Imagine I have the following layers in a CNN:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Conv-layer 1 (without reLU) [3 filters @ 1x3x3]&lt;/strong&gt; =&gt;&#xA;&lt;strong&gt;ReLU-layer 1&lt;/strong&gt; =&gt;&#xA;&lt;strong&gt;Maxpooling-layer 1 [2x2]&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Conv-layer 2 (without reLU) [10 filters @ 3x4x4]&lt;/strong&gt; =&gt;&#xA;&lt;strong&gt;ReLU-layer 2&lt;/strong&gt; =&gt;&#xA;&lt;strong&gt;Maxpooling-layer 2 [2x2]&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I know is that, when &quot;deriving&quot; the &lt;strong&gt;Maxpooling-layer&lt;/strong&gt;, the maximum value of that pooling has the value of 1 (because that value is the only one that is affecting the result of the CNN), and the other values take the value of 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this step, I have the reLU-layer, where the derivative of the reLU-layer is 0 when x &amp;lt;= 0, and 1 when x &gt; 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in the last step, I have the Conv-layer, but I don't know how to calculate the derivative. &lt;em&gt;How do I calculate the whole delta for that specific weight?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a way to calculate the gradients of those weights using matrix notation, please let me know.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Notation&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is &quot;equivalent&quot; to &quot;channels&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Steps in Conv layer, using my notation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I have an input &lt;strong&gt;X&lt;/strong&gt; with shape &lt;strong&gt;[depth:1, rows: 28, columns: 28]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I have the conv-layer, where I have &lt;strong&gt;3 filters (you can call the filter as W)&lt;/strong&gt; with shape &lt;strong&gt;[depth:1, rows:3, columns:3]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape =&gt; &lt;strong&gt;[depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter)&lt;/strong&gt;. So, when I do the convolution using the 3 filters, the resulting matrix is shaped [&lt;strong&gt;depth:3&lt;/strong&gt;, rows: 26, columns: 26].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window of 2 rows x 2 columns.&lt;/p&gt;&#xA;" OwnerUserId="26474" LastEditorUserId="29169" LastEditDate="2021-02-16T15:29:12.543" LastActivityDate="2021-02-16T15:29:12.543" Title="Steps for back propagation of convolutional layer in CNN" Tags="|machine-learning|convolutional-neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15337" PostTypeId="1" CreationDate="2016-11-24T19:06:31.867" Score="0" ViewCount="439" Body="&lt;p&gt;I wanted to recreate the model mentioned in this paper:&lt;a href=&quot;https://arxiv.org/pdf/1610.09204v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1610.09204v1.pdf&lt;/a&gt; . I am using keras with tensorflow backend, and a gtx 1050ti.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am an ML beginner, and thought this would be a good way to get a hands on feel for things. However, My model is not converging(loss is same as first epoch).&#xA;This is what I read from that paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The first convolutional layer re- ceives an input of 56px by 56px&#xA;  images with RGB channels. It uses 32 filters of size 5×5×3, stride 1&#xA;  and then sampled with max pooling of size 2 × 2, stride 1. The second&#xA;  convolutional layer has 64 filters of size 5×5×32, stride 1 and a max&#xA;  pooling of size 2 × 2, stride 1. The results of the second max pooling&#xA;  provide the first fully-connected layer with a vector of length 12,544&#xA;  (14 × 14 × 64) which are used by 512 neurons. The final&#xA;  fully-connected output layer uses a 20-wide softmax [21] which&#xA;  represents the probability of each respective 20 class labels. This&#xA;  architecture is similar to the LeNet model [3], but with using&#xA;  rectified linear unit (ReLU) [22] activation functions instead of&#xA;  sigmoid activation functions. We also use dropout [23], a technique to&#xA;  prevent overfitting, with a keep probability of 0.5 for the&#xA;  fully-connected layers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and my code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=(70,52, 3))) model.add(Activation(&quot;relu&quot;)) model.add(MaxPooling2D(pool_size=(2,2)))&#xA;model.add(Convolution2D(64, 5, 5, border_mode='same')) model.add(Activation(&quot;relu&quot;)) model.add(MaxPooling2D(pool_size=(2,2)))&#xA;model.add(Flatten()) model.add(Dense(output_dim=512)) model.add(Activation(&quot;relu&quot;))&#xA;model.add(Dense(output_dim=2))&#xA;model.add(Activation(&quot;softmax&quot;)) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;model.fit(x_train, y_train, nb_epoch=70, batch_size=500,verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the full code can be found here : &lt;a href=&quot;https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d&lt;/a&gt;&#xA;I have scrapped the data from goodreads&#xA;Help appreciated !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I forgot to actually ask what i wanted. Since its my first experiment, i would like to ask what are some things that I should do to make my model converge.&lt;/p&gt;&#xA;" OwnerUserId="26476" LastEditorUserId="29169" LastEditDate="2021-02-16T15:28:55.500" LastActivityDate="2021-02-16T15:28:55.500" Title="My first machine learning experiment , model not converging , tips?" Tags="|machine-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ClosedDate="2016-11-25T08:12:52.360" ContentLicense="CC BY-SA 3.0" />
  <row Id="15338" PostTypeId="1" AcceptedAnswerId="15424" CreationDate="2016-11-24T19:19:00.523" Score="1" ViewCount="766" Body="&lt;p&gt;I'm training my neural network implementation on the &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/MONK&amp;#39;s+Problems&quot; rel=&quot;nofollow noreferrer&quot;&gt;MONK's problems data set&lt;/a&gt;. I read that it can be possible to achieve almost a 0 loss with the hyper-parameters I'm using &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=AF30F83FB7897C5C48A35CE2C3B015CD?doi=10.1.1.45.2363&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(chapter 9)&lt;/a&gt;. These are my results on training set after 90 epochs, they aren't good as you can see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training error after 0 iterations: 0.26427015361&#xA;Training error after 20 iterations: 0.231331617693&#xA;Training error after 40 iterations: 0.216396390383&#xA;Training error after 60 iterations: 0.198107908553&#xA;Training error after 80 iterations: 0.166181613583&#xA;TrErr: 0.155389641876&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is the code of my neural network. I think the code is quite simple and readable for a expert, tell me if you need any clarification.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# paramaters&#xA;etas = [0.1]&#xA;hidden_dim = 2&#xA;alpha = 0.5&#xA;reg_lambda = 0.0&#xA;&#xA;input_dim = 17&#xA;output_dim = 1&#xA;&#xA;# read and parse MONK data&#xA;def parse_data(text):&#xA;    with open(text) as f:&#xA;        content = f.readlines()    &#xA;    t = np.zeros((len(content), output_dim))&#xA;    X = np.zeros((len(content), input_dim))&#xA;    for i,val in enumerate(content):&#xA;        s = val.split(&quot; &quot;)&#xA;        t[i] = float(s[1])&#xA;        for idx in xrange(17):&#xA;            X[i][idx] = 0.0&#xA;        j = -1&#xA;        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16&#xA;        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6&#xA;        X[i][int(s[2])+j] = 1.0 # a1&#xA;        j = 2&#xA;        X[i][int(s[3])+j] = 1.0 # a2&#xA;        j = 5&#xA;        X[i][int(s[4])+j] = 1.0 # a3&#xA;        j = 7&#xA;        X[i][int(s[5])+j] = 1.0 # a4&#xA;        j = 10&#xA;        X[i][int(s[6])+j] = 1.0 # a5&#xA;        j = 14&#xA;        X[idx][int(s[7])+j] = 1.0 # a6&#xA;&#xA;    return (X, t)&#xA;&#xA;def sigmoid(x):&#xA;    output = 1/(1+np.exp(-x))&#xA;    return output&#xA;&#xA;def sigmoid_output_to_derivative(output):&#xA;    return output*(1-output)&#xA;&#xA;# initialize weights&#xA;W0 = np.random.uniform(-0.7,0.7,size=(input_dim,hidden_dim))&#xA;W1 = np.random.uniform(-0.7,0.7,size=(hidden_dim,output_dim))&#xA;b0 = np.zeros((1, hidden_dim))&#xA;b1 = np.zeros((1, output_dim))&#xA;dW0 = 0&#xA;dW1 = 0&#xA;&#xA;for j in xrange(90):&#xA;    # forward propagation&#xA;    l0 = X&#xA;    l1 = sigmoid(np.dot(l0,W0) + b0)&#xA;    l2 = sigmoid(np.dot(l1,W1) + b1)&#xA;&#xA;    # back propagation&#xA;    l2_error = t - l2&#xA;    l2_delta = l2_error*sigmoid_output_to_derivative(l2) &#xA;&#xA;    l1_error = l2_delta.dot(W1.T)&#xA;    l1_delta = l1_error*sigmoid_output_to_derivative(l1)&#xA;&#xA;    if (j% 20) == 0:&#xA;        print &quot;Training error after&quot;, j, &quot;iterations:&quot;,np.mean(np.square(l2_error))&#xA;&#xA;    dW1 = eta*(l1.T.dot(l2_delta) + reg_lambda*W1) + alpha*dW1&#xA;    dW0 = eta*(l0.T.dot(l1_delta) + reg_lambda*W0) + alpha*dW0&#xA;    db1 = eta*(np.sum(l2_delta, axis=0))&#xA;    db0 = eta*(np.sum(l1_delta, axis=0))&#xA;&#xA;    W1 += dW1&#xA;    W0 += dW0&#xA;    b1 += db1&#xA;    b0 += db0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26475" LastEditorUserId="26475" LastEditDate="2016-11-29T10:28:35.803" LastActivityDate="2016-11-29T17:55:17.680" Title="Implementing a neural network: low accuracy" Tags="|neural-network|" AnswerCount="2" CommentCount="6" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15339" PostTypeId="1" CreationDate="2016-11-24T19:39:47.173" Score="1" ViewCount="1454" Body="&lt;p&gt;I've created a neural network in tensorflow. This network is multilabel. Ergo: it tries to predict multiple output labels for one input set, in this case three. Currently I use this code to test how accurate my network is at predicting the three labels:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_, indices_1 = tf.nn.top_k(prediction, 3)&#xA;_, indices_2 = tf.nn.top_k(item_data, 3)&#xA;correct = tf.equal(indices_1, indices_2)&#xA;accuracy = tf.reduce_mean(tf.cast(correct, 'float'))&#xA;percentage = accuracy.eval({champion_data:input_data,  item_data:output_data})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That code works fine. The problem is now that I'm trying to create code that tests if the top 3 items it finds in indices_1 are amongst the top 5 images in indices_2. I know tensorflow has an in_top_k() method, but as far as I know that doesn't accept multilabel. Currently I've been trying to compare them using a for loop:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_, indices_1 = tf.nn.top_k(prediction, 5)&#xA;_, indices_2 = tf.nn.top_k(item_data, 3)&#xA;indices_1 = tf.unpack(tf.transpose(indices_1, (1, 0)))&#xA;indices_2 = tf.unpack(tf.transpose(indices_2, (1, 0)))&#xA;correct = []&#xA;for element in indices_1:&#xA;    for element_2 in indices_2:&#xA;        if element == element_2:&#xA;            correct.append(True)&#xA;        else:&#xA;            correct.append(False)&#xA;accuracy = tf.reduce_mean(tf.cast(correct, 'float'))&#xA;percentage = accuracy.eval({champion_data:input_data, item_data:output_data})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, that doesn't work. Mainly because I don't have a clue what I'm doing when it comes to tf.unpack and tf.transpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I leave the transpose out it gives the following error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Cannot infer num from shape (?, 5)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If the transpose is in there the code runs but my accuracy is always 0.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So I have one of two questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Is there an easy replacement for in_top_k() that accepts multilabel classification that I can use instead of custom writing code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If not 1: what am I doing wrong that results in me getting an accuracy of 0.0?&lt;/p&gt;&#xA;" OwnerUserId="19067" LastActivityDate="2016-11-24T19:39:47.173" Title="Testing a tensorflow network: in_top_k() replacement for multilabel classification" Tags="|neural-network|tensorflow|multilabel-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15340" PostTypeId="2" ParentId="15336" CreationDate="2016-11-24T19:49:22.197" Score="1" Body="&lt;p&gt;Extra Note: The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is &quot;equivalent&quot; to &quot;channels&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Steps in Conv layer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I have an input &lt;strong&gt;X&lt;/strong&gt; with shape &lt;strong&gt;[depth:1, rows: 28, columns: 28]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I have the conv-layer, where I have &lt;strong&gt;3 filters (you can call the filter as W)&lt;/strong&gt; with shape &lt;strong&gt;[depth:1, rows:3, columns:3]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape =&gt; &lt;strong&gt;[depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter)&lt;/strong&gt;. So, when I do the convolution using the 3 filters, the resulting matrix is shaped [&lt;strong&gt;depth:3&lt;/strong&gt;, rows: 26, columns: 26].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window o 2 rows x 2 columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is any doubt of what I explained, please comment me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I'm editing my own answer beacuse I don't have 50 reputation for doing comments.&lt;/p&gt;&#xA;" OwnerUserId="26479" LastEditorUserId="26479" LastEditDate="2016-11-24T21:00:10.830" LastActivityDate="2016-11-24T21:00:10.830" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15341" PostTypeId="2" ParentId="15337" CreationDate="2016-11-25T04:04:49.763" Score="0" Body="&lt;p&gt;There are many things that can go wrong. But the first thing I would check is the learning rate. Adam can be initialized with  learning rate which isn't shown in your code. You could probably try setting a lower learning rate, say, 10e-4, first.&lt;/p&gt;&#xA;" OwnerUserId="26486" LastActivityDate="2016-11-25T04:04:49.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15342" PostTypeId="2" ParentId="15311" CreationDate="2016-11-25T06:00:56.907" Score="0" Body="&lt;p&gt;TF-IDF denotes relative importance: importance of a term in a specific text, compared to the whole corpus. So, comparative corpus required to use TF-IDF to estimate word importance in all your 270 documents. You can use one of &quot;common&quot; corpora, listed &lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_text_corpora&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, or try to find domain-specific corpus. Then treat your 270 docs as a single text, and calculate TF-IDF scores for words, you are interested in, based on comparative corpus.&lt;/p&gt;&#xA;" OwnerUserId="941" LastActivityDate="2016-11-25T06:00:56.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15343" PostTypeId="2" ParentId="15337" CreationDate="2016-11-25T06:51:18.493" Score="1" Body="&lt;p&gt;You may try &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stochastic Gradient Descent &lt;/a&gt;optimizer with a learning rate decay and &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v28/sutskever13.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;nesterov momentum&lt;/a&gt;. You can also try a different &lt;code&gt;batch_size&lt;/code&gt;. Also you are missing drop out layers between the fully connected layers which the authors used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;# flatten the conv layers&#xA;model.add(Flatten())&#xA;model.add(Dropout(0.5))&#xA;&#xA;# fully connected 512&#xA;model.add(Dense(output_dim=512)) &#xA;model.add(Activation(&quot;relu&quot;))&#xA;model.add(Dropout(0.5))&#xA;&#xA;# fully connected output layer&#xA;model.add(Dense(output_dim=2))&#xA;model.add(Activation(&quot;softmax&quot;)) &#xA;&#xA;# compile&#xA;model.compile(loss='categorical_crossentropy', &#xA;              optimizer=SGD(lr=10e-4, &#xA;                            decay=10e-6, &#xA;                            momentum=0.99, &#xA;                            nesterov=True), &#xA;              metrics=['accuracy'])&#xA;&#xA;# train&#xA;model.fit(x_train, y_train, &#xA;          nb_epoch=70, &#xA;          batch_size=64,&#xA;          verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my experience this usually helps a lot. If you get this setting to converge, you may try &lt;code&gt;rmsprop&lt;/code&gt; and &lt;code&gt;adam&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26146" LastEditorUserId="26146" LastEditDate="2016-11-25T10:15:33.933" LastActivityDate="2016-11-25T10:15:33.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15344" PostTypeId="2" ParentId="15134" CreationDate="2016-11-25T06:57:46.347" Score="1" Body="&lt;p&gt;Yes, there are such things, but they aren't really related to word2vec. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, Bag of Words is the old &quot;something2vec&quot; approach of representing documents in a vector space. Of course, the resulting vectors are very sparse, and distance does not always behave well in such cases, and it is better to use dot-product based similarities like cosine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you can reduce the dimensionality of this vector space with SVD or NMF (Non-Negative Matrix Factorization), and this will give you what you want: dense vectors of small dimensionality for which you can compute the distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can go further and use Topic Modelling, which also will give you a vectorial representation of documents, and you can use it for comparing them. &lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-11-25T06:57:46.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15347" PostTypeId="1" CreationDate="2016-11-25T13:12:29.493" Score="1" ViewCount="64" Body="&lt;p&gt;I've been tasked with clustering searches from our website into the different searches for the same product. e.g. such different searches for the same product may be &quot;product name&quot; / &quot;product name 1st release&quot; / &quot;product name 1995&quot; / &quot;product name 2004&quot; / &quot;prodct name&quot; (mis-spelling intended) / &quot;author 1995 part-of-product-name&quot; and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I seem to have a million or so searches like this for various products with various search strings. I don't have category or any other grouping data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I cluster these search strings into their respective products / items with as minimum input from me as possible?&lt;/p&gt;&#xA;" OwnerUserId="2861" LastActivityDate="2016-11-25T13:12:29.493" Title="Clustering product searches into products" Tags="|clustering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15348" PostTypeId="2" ParentId="15134" CreationDate="2016-11-25T13:28:40.880" Score="0" Body="&lt;p&gt;how about &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2vec&lt;/a&gt;. Mikolov released the &lt;a href=&quot;https://arxiv.org/pdf/1405.4053v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;journal&lt;/a&gt; after the successful attempt of word2vec. A single feature vector for a whole paragraph or sentence or document (depending upon your use case) could be generated. Personally, I didn't get quite good results with it for sentence classification problems. Rather I went with averaged word vectors for sentences.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-11-25T13:28:40.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15349" PostTypeId="2" ParentId="15338" CreationDate="2016-11-25T14:41:22.920" Score="0" Body="&lt;p&gt;If I were you, I'd import the sigmoid function from a python lib, e.g. &lt;code&gt;sklearn&lt;/code&gt; as they compute it probably numerically more stable than we would.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And another thing: backprop calculates the error as $t_i - o_i$, where $t_i$ is the correct label of sample $i$ and $o_i$ is the predicted output  by your net. You compute it as $o_i - t_i$, so change&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l2_error = l2 - t&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l2_error = t - l2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26146" LastEditorUserId="26146" LastEditDate="2016-11-29T16:32:27.700" LastActivityDate="2016-11-29T16:32:27.700" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="15350" PostTypeId="1" CreationDate="2016-11-25T19:32:28.437" Score="2" ViewCount="905" Body="&lt;p&gt;For text processing there are plenty of tools out there like CoreNLP, SpaCy, NLTK, textblob etc. each offering different suites of pre-processing functions and people recommend different tool for different tasks like NLTK for tokenization etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which tool would you recommend which performs all these tasks?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2016-11-26T05:34:37.233" Title="Best tool for text pre-processing, involving tokenization, lemmatization, stop-word removal, feature vector extraction?" Tags="|nlp|tools|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15352" PostTypeId="1" CreationDate="2016-11-25T21:47:26.150" Score="0" ViewCount="62" Body="&lt;p&gt;Assume that we have set of Hidden Markov Models (Bayesian networks) Set{(n, m, P, A, B)} (n - number of hidden states, m - number of observable states, P - initial probabilities, A - transition matrix, B - emission matrix). Are the peculiar algorithms or approaches how to classify those HMMs into different subsets of HMMs? If n and m are equal for all the HMMs in the set then classification can be build from the ranges of elements in A and B. If n and m are different for different HMMs then one classify hidden and observable states first...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there some research or trends for this? Google is not helping here because it returns results for classification methods using HMMs, but we want to classify (or find clusters) of HMMs themselves.&lt;/p&gt;&#xA;" OwnerUserId="26058" LastActivityDate="2016-12-15T22:46:15.080" Title="Pecularities of classification of Hidden Markov Models?" Tags="|machine-learning|classification|bayesian-networks|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15353" PostTypeId="2" ParentId="19" CreationDate="2016-11-25T22:34:51.253" Score="4" Body="&lt;p&gt;Data is &quot;Big Data&quot; if it is of such volume that it is less expensive to analyze it on two or more commodity computers, than on one high-end computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essentially how Google's &quot;BigFiles&quot; file system originated. Page and Brin could not afford a fancy Sun server to store and search their web index, so hooked up several commodity computers &lt;/p&gt;&#xA;" OwnerUserId="484" LastActivityDate="2016-11-25T22:34:51.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15354" PostTypeId="2" ParentId="15350" CreationDate="2016-11-26T05:34:37.233" Score="3" Body="&lt;p&gt;Comparing two libraries or tools in terms of these things is somewhat that is opinion dependent. Some people prefer &lt;code&gt;NLTK&lt;/code&gt; for doing almost all the tasks. &lt;code&gt;spacy&lt;/code&gt; has also gained quite reputation. But what is better for you depends on what you want do. In my personal experience, I have found that &lt;code&gt;NLTK&lt;/code&gt; along with &lt;code&gt;gensim&lt;/code&gt; libraries is all that I need to do all the Natural Language Processing tasks. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-11-26T05:34:37.233" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15355" PostTypeId="1" CreationDate="2016-11-26T05:50:30.513" Score="0" ViewCount="108" Body="&lt;p&gt;How can we detect the existence of outliers using mean and median?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it really possible to detect the existence of outliers in a set of data from their feature-wise mean and median?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose, I have a data set with eight features in my hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have juxtaposed their means and medians row by row.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            f1       f2        f3        f4        f5        f6        f7        f8&#xA;mean      2.5000    0.1868    0.0148    0.2105    0.2088   79.6583    1.0604    0.0091&#xA;median    2.5000    0.1826    0.0001    0.0002    0.0000   -0.0000    0.0000   -0.0000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can we tell anything about the existence of outliers from these information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How are the data distributed?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Data set is considered to be normally distributed. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do they all have the same mean and variance?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't know. &lt;/p&gt;&#xA;" OwnerDisplayName="user9232" LastEditorDisplayName="user9232" LastEditDate="2016-11-26T09:44:54.687" LastActivityDate="2016-11-26T09:44:54.687" Title="How can we detect the existence of outliers using mean and median?" Tags="|feature-selection|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15358" PostTypeId="1" CreationDate="2016-11-26T08:54:05.330" Score="2" ViewCount="1019" Body="&lt;p&gt;Suppose, I have a data set with eight features in my hand. I want to find features to predict the diamonds, hearts, clubs, spades.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;------------------------------------------------------------------------------&#xA;|   f1   |   f2   |   f3    |   f4    |    f5    |   f6    |   f7    |  f8&#xA;+--------+--------+---------+---------+---------------------------------------&#xA;|        |        |         |         |          |         |         |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;f1&lt;/code&gt; column is for class-labels and the rest of them are features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I have rendered scatter plots by taking the features &lt;code&gt;f2&lt;/code&gt; and &lt;code&gt;f3&lt;/code&gt; and it looks like the following,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8n8RT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8n8RT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later, I have rendered scatter plots by taking the features &lt;code&gt;f3&lt;/code&gt; and &lt;code&gt;f4&lt;/code&gt; and it looks like the following,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4XTVD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4XTVD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $Red$ = Diamonds, $Blue$ = Spades, $Magenta$ = Hearts, and $Black$ = Clubs,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I have the following questions,&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Why are some plots cohesive and others are separated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) What do these two plots tell about those four types of cards?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Which two features, among &lt;code&gt;f2&lt;/code&gt;, &lt;code&gt;f3&lt;/code&gt;, and &lt;code&gt;f4&lt;/code&gt;, would you choose for further experiments and why?&lt;/p&gt;&#xA;" OwnerDisplayName="user9232" LastEditorDisplayName="user9232" LastEditDate="2016-11-26T11:56:36.360" LastActivityDate="2016-12-28T02:33:09.223" Title="Can we select features by examining scatterplots?" Tags="|feature-selection|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15361" PostTypeId="2" ParentId="15355" CreationDate="2016-11-26T09:38:12.690" Score="1" Body="&lt;p&gt;No. Unless you have some other idea of the scale and spread and distribution of the feature values. You can construct data sets that have any given mean and median with no outliers or with one massive outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, f2 looks like a well-balanced set of numbers with a very close mean and median, maybe all those values are from a N(0.184,3) distribution. Scale that up by a linear transformation to an N(X,Y) and you'll get a mean and median much like f6. Exercise: find X and Y.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-11-26T09:38:12.690" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15363" PostTypeId="1" CreationDate="2016-11-26T11:21:33.540" Score="1" ViewCount="400" Body="&lt;p&gt;I have researched machine learning for quite a while and would like to test out my knowledge. So I am trying to use it for lottery number prediction. The goal is not to have 100% correct prediction (which is of course impossible), but to perform better than pure random prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I have the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA; [3,10,16,19,34,45],&#xA; [7,14,15,20,28,41],&#xA; [2,6,18,24,30,37],&#xA;...&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As it can be seen each round six numbers will be drawn and I have made an assumption that these rounds are not independent and that each round is related to its previous round. So I am looking for a prediction algorithm that will try learn about this and output one or more predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be a supervised learning for the algorithm. Using the above example data, it will be pre-processed so that it becomes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA; [[3,10,16,19,34,45], [7,14,15,20,28,41]],&#xA; [[7,14,15,20,28,41], [2,6,18,24,30,37]],&#xA; ...&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously, for each tuple, input is on the left while output is on the right. Also, I have another assumption that the numbers printed on the balls are really just labels, so the algorithm should consider them as categorical rather than numerical attribute. And there is no ordering for these labels, though the data appears sorted numerically in ascending order. So when the algorithm tries to learn from the training data, it should not be affected by the ball ordering in any way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, after it has been trained, if I input, for example [3,5,6,11,34,42], it should output a list of the algorithm's best guess predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is the best prediction algorithm and the recommended programming tool for such task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: In theory, current round should be unrelated to previous round. But I still think they do have some correlation (though very little), which is not obvious and hard to explain.&lt;/p&gt;&#xA;" OwnerUserId="26515" LastEditorUserId="26515" LastEditDate="2016-11-27T06:41:06.870" LastActivityDate="2016-11-27T06:41:06.870" Title="Best MIMO prediction algorithm for categorical variables" Tags="|machine-learning|predictive-modeling|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15364" PostTypeId="2" ParentId="15358" CreationDate="2016-11-26T11:53:17.057" Score="0" Body="&lt;p&gt;I guess that depends mainly on what you want to do with that data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have labels, I suppose that you want to perform some kind of supervised learning. In that case What you want to avoid are for example features which are correlated. That is something you could identify by scatterplots. To actually select a subset of features, I would say that for example principle component analysis is a better option.&lt;/p&gt;&#xA;" OwnerUserId="26519" LastActivityDate="2016-11-26T11:53:17.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15365" PostTypeId="1" CreationDate="2016-11-26T15:22:28.303" Score="0" ViewCount="473" Body="&lt;p&gt;In machine learning, an overfitted model fits training set very well but cannot generalize to new instances. I evaluated my model using cross-validation and my accuracy drops when setting the maximum number of splits of my decision tree beyond a certain number. Can this be associated with overfitting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&#xA;Here is the result. I am doing Human Activity Recognition in MATLAB with 16 features.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pxEDi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pxEDi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26522" LastEditorUserId="26522" LastEditDate="2016-11-26T15:57:28.777" LastActivityDate="2016-11-27T03:41:24.560" Title="Overfitting and cross-validation" Tags="|machine-learning|overfitting|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15366" PostTypeId="2" ParentId="14237" CreationDate="2016-11-26T15:24:17.650" Score="3" Body="&lt;p&gt;As @Spacedman said, &lt;a href=&quot;http://papers.nips.cc/paper/2909-laplacian-score-for-feature-selection.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the paper&lt;/a&gt; provides a very clear explanation of the algorithm on page 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're a little less comfortable with the math of the notation, here's the intuition/explanation in words.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Make a k-nearest neighbor's graph. That is, for each observation, define an edge in the Graph for that observation if another observation is one of its k-nearest neighbor's. If you're using a supervised algorithm you can define an edge if they share the same label.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If any two nodes (observations) are connected, define a weight matrix S measuring the similarity between those two nodes (using some distance measure). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For each feature define the Laplacian graph.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the Laplacian score based on their equation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Intuitively, you're using KNNs to define a network graph and assessing how similar the features are according to your distance metric. This is just as good of a measure of feature importance as any other but will also has its pitfalls, just like all of the others.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="29575" LastEditDate="2019-11-24T15:22:34.080" LastActivityDate="2019-11-24T15:22:34.080" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15367" PostTypeId="1" CreationDate="2016-11-26T19:59:42.697" Score="1" ViewCount="441" Body="&lt;p&gt;I'm using a Lasagne network, but this is a pretty generalized question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;say I ingest N number of tweets using the same LSTM, grab the final output of each to make a NxOutputSize tensor (&lt;code&gt;Merged_output&lt;/code&gt;) to be condensed into a small 10-vector output through a shallow neural net. What would be the best way to train this LSTM?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="26528" LastActivityDate="2016-11-26T19:59:42.697" Title="LSTM concatenation training" Tags="|machine-learning|neural-network|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15368" PostTypeId="2" ParentId="12883" CreationDate="2016-11-26T20:02:31.257" Score="-1" Body="&lt;p&gt;You first need to convert this data into some numerical representation, and then you can use clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of such ways is applying TF-IDF weighting to tags, and then calculate the cosine similarity between them and, finally, apply some hierarchical clustering to the results. Or you can encode the entire record (including category, sector, etc) in a similar way, and do the same thing afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can apply K-Means to these TF-IDF weights - but if dimensionality of this matrix is large, prior to that you may need to reduce it with SVD or something similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't use Mahout for this and start with scikit-learn. You may want to have a look at these classes and methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine_similarity&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AgglomerativeClustering&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TruncatedSVD&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;KMeans&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-11-26T20:02:31.257" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="15369" PostTypeId="2" ParentId="15365" CreationDate="2016-11-26T23:25:51.417" Score="1" Body="&lt;p&gt;It definitely can be associated with over-fitting. I would suggest plotting the training and validation accuracy in a single graph. So you see how both depend on the number of splits. If it is over-fitting one would expect that the training accuracy continues rising, while the validation accuracy gets to a maximum and then drops. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you have to make sure that your training and validation sets are OK. I assume you do something like k-fold cross validation. Where every sample is k-times in a training set and one time in a validation set. And you have enough samples to train your model etc. &lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2016-11-26T23:25:51.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15370" PostTypeId="2" ParentId="15363" CreationDate="2016-11-26T23:33:41.897" Score="1" Body="&lt;p&gt;You are trying to predict random numbers here. By design the lottery numbers are unrelated and one round has no impact on the next round. For machine learning you should use input that has some kind of &quot;predictive power&quot;. I mean data that as some information or correlation with the target variable. You can of course train a prediction algorithm on that data like that, but it does not make sense to me. In case you find something that is a solid prediction, that might mean the lottery does not draw random numbers.&#xA;...this actually should be a comment, but StackExchange does not let me write a comment.&lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2016-11-26T23:33:41.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15371" PostTypeId="1" CreationDate="2016-11-27T00:49:00.633" Score="3" ViewCount="1911" Body="&lt;p&gt;At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn't it be better to always start with a pretrained model, as the lower layers of weights has already probably learnt general patterns of images that are transferable across all data sets? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My worry is what if the dataset I want to use for finetuning is highly specialized, highly unnatural and very different from the dataset the pretrained model is trained on. Would this still mean finetuning from a pretrained model wouldn't be the best idea? (e.g. training on X-Ray images instead of natural images for a cifar-1000 pretrained model.)&lt;/p&gt;&#xA;" OwnerUserId="13527" LastActivityDate="2024-02-18T23:09:11.063" Title="Is finetuning from a pretrained model always better than training from scratch?" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15373" PostTypeId="2" ParentId="15365" CreationDate="2016-11-27T03:41:24.560" Score="0" Body="&lt;p&gt;In decision trees, one impact of increasing the maximum number of splits is that it can lead to overfitting on the training data set because effectively we are increasing the flexibility of the algorithm to memorize the train set. A decrease in the CV accuracy is understandable in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since it is a common issue with any ml parameter, the go-to option is to search over a set of options and pick the one that gives the best CV score. On the other hand for Decision trees there are more sophisticated methods that fall largely either in the &quot;top-down&quot; vs &quot;bottom-up&quot; category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A top down approach would be to keep increasing the max num of splits (holding everything else constant) until you see a drop in CV score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Bottom up approach is to grow a full tree first, and then prune/coalesce/average leaves to reduce the variance of the predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;one reference for these two options(page 9): [&lt;a href=&quot;http://www.ccs.neu.edu/home/vip/teach/MLcourse/1_intro_DT_RULES_REG/lecture_notes/lectureNotes_DecisionTree_Spring15.pdf][1]&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.ccs.neu.edu/home/vip/teach/MLcourse/1_intro_DT_RULES_REG/lecture_notes/lectureNotes_DecisionTree_Spring15.pdf][1]&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15445" LastActivityDate="2016-11-27T03:41:24.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15374" PostTypeId="1" CreationDate="2016-11-27T04:45:02.170" Score="-2" ViewCount="572" Body="&lt;p&gt;I have a dataset of latitude and longitude coordinates with 140000 values in each and when I apply k means clustering using r I am getting this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Error in do_one(nmeth) : NA/NaN/Inf in foreign function call (arg 1)&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the dataset doesn't contain any missing values I don't think. How should I solve this issue and get clusters out of these locations? Is there any other algorithm which is suitable for this purpose? The intention behind the task is to get areas/regions with more accidents.&lt;/p&gt;&#xA;" OwnerUserId="26535" LastEditorUserId="29169" LastEditDate="2019-06-11T14:54:43.293" LastActivityDate="2019-06-11T14:55:38.183" Title="k means clustering for large dataset" Tags="|r|data-mining|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15376" PostTypeId="1" CreationDate="2016-11-27T12:10:39.420" Score="1" ViewCount="6428" Body="&lt;p&gt;I am struggling to understand why I am getting such a high loss/val_loss rate on my training. I am training a regression network. I've normalized the input data to range between -1 to 1, and left the output data unaltered, its range is approximately between -100 and 100.&lt;/p&gt;&#xA;&lt;p&gt;I chose to normalize the input so that I could use tanh as the activation function since it outputs within this range.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7ikCX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7ikCX.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XkH7J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XkH7J.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The neural network consists of 3 layers.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;  print &amp;quot;Model definition!&amp;quot;&#xA;    model = Sequential()&#xA;&#xA;    #act = PReLU(init='normal', weights=None)&#xA;    model.add(Dense(output_dim=400,input_dim=400, init=&amp;quot;normal&amp;quot;,activation=K.tanh))&#xA;&#xA;    #act1 = PReLU(init='normal', weights=None)&#xA;    model.add(Dense(output_dim=400,input_dim=400, init=&amp;quot;normal&amp;quot;,activation=K.tanh))&#xA;&#xA;    #act2 = PReLU(init='normal', weights=None)&#xA;    #model.add(Dense(output_dim=400, input_dim=400, init=&amp;quot;normal&amp;quot;,activation=K.tanh))&#xA;&#xA;    act4=ELU(100)&#xA;    model.add(Dense(output_dim=13, input_dim=400, init=&amp;quot;normal&amp;quot;,activation=act4))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The mapping between the input and output consists of mapping audio samples into MFCC features.  The samples are the ones i've normalized to the aforementioned range.&lt;/p&gt;&#xA;&lt;p&gt;Why am I getting these results?&lt;/p&gt;&#xA;&lt;p&gt;Am I doing anything that is unclear?&lt;/p&gt;&#xA;&lt;p&gt;Normalizing the output_range +-1:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4q3Me.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4q3Me.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&#xA;Why is the val_loss loss lower than the loss?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-07-10T16:56:33.527" LastActivityDate="2024-01-06T16:06:45.760" Title="Why is my loss so high?" Tags="|neural-network|hyperparameter|loss-function|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="15377" PostTypeId="1" AcceptedAnswerId="15382" CreationDate="2016-11-27T12:19:00.893" Score="0" ViewCount="456" Body="&lt;p&gt;Is the FPG-Growth an Machine Learning algorithm? Because I'm looking at this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.fpm.FPGrowth&#xA;import org.apache.spark.rdd.RDD&#xA;&#xA;val data = sc.textFile(&quot;data/mllib/sample_fpgrowth.txt&quot;)&#xA;&#xA;val transactions: RDD[Array[String]] = data.map(s =&amp;gt; s.trim.split(' '))&#xA;&#xA;val fpg = new FPGrowth()&#xA;  .setMinSupport(0.2)&#xA;  .setNumPartitions(10)&#xA;val model = fpg.run(transactions)&#xA;&#xA;model.freqItemsets.collect().foreach { itemset =&amp;gt;&#xA;  println(itemset.items.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;) + &quot;, &quot; + itemset.freq)&#xA;}&#xA;&#xA;val minConfidence = 0.8&#xA;model.generateAssociationRules(minConfidence).collect().foreach { rule =&amp;gt;&#xA;  println(&#xA;    rule.antecedent.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;)&#xA;      + &quot; =&amp;gt; &quot; + rule.consequent .mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;)&#xA;      + &quot;, &quot; + rule.confidence)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I'm not seeing how the algoritm can learn because it don't have an train, test or validation set...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="23714" LastActivityDate="2016-11-28T09:21:13.693" Title="Spark Mllib - FPG-Growth - Machine Learning" Tags="|machine-learning|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15378" PostTypeId="2" ParentId="15220" CreationDate="2016-11-27T16:02:29.860" Score="2" Body="&lt;p&gt;I assume it should read as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if condition is true, go to branch 3&lt;/li&gt;&#xA;&lt;li&gt;otherwise go to branch 4&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am only guessing, but seems plausible&lt;/p&gt;&#xA;" OwnerUserId="816" LastActivityDate="2016-11-27T16:02:29.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15379" PostTypeId="2" ParentId="15374" CreationDate="2016-11-27T16:21:24.093" Score="0" Body="&lt;p&gt;First of all,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;the dataset doesn't have any missing values I don't think&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are not sure that the data is clean, you should probably check it first. You should check for both columns of your input data frame that there are no NAs and all values are numerical. Then check the datatype within R and the datatypes accepted by kmeans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that does not lead anywhere, a small sample of your data and your code would be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning an alternative approach, I can think of agglomerative clustering and the dbscan algorithm, but I think that is not your issue.&lt;/p&gt;&#xA;" OwnerUserId="26519" LastEditorUserId="29169" LastEditDate="2019-06-11T14:55:38.183" LastActivityDate="2019-06-11T14:55:38.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15380" PostTypeId="1" AcceptedAnswerId="15386" CreationDate="2016-11-27T20:01:21.317" Score="2" ViewCount="115" Body="&lt;p&gt;Suppose that I am interested in predicting an outcome (say, the arrival delay [in seconds] of a flight) based upon a set of features. &#xA;One of these features is a nominal variable - &lt;code&gt;carrier&lt;/code&gt; - that specifies the airline carrier of the flight. This feature has 16 different values. After investigation of how arrival delay is distributed across each carrier, it appears that some carriers could be collapsed into one value (e.g., &quot;AS&quot; and &quot;HA&quot; or &quot;WN&quot; and &quot;B6&quot;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;nycflights13&quot;)&#xA;library(nycflights)&#xA;&#xA;boxplot(&#xA;    formula = arr_delay ~ with(flights, reorder(carrier, -arr_delay, median, na.rm = TRUE)),&#xA;    data = flights,&#xA;    horizontal = TRUE, &#xA;    las = 2,&#xA;    plot = TRUE&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Cgt5o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Cgt5o.png&quot; alt=&quot;Boxplots of Arrival Delay (in seconds) by Carrier.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, are there well-known methods for reducing the dimensionality &lt;strong&gt;within a feature&lt;/strong&gt;? &lt;/p&gt;&#xA;" OwnerUserId="8232" LastActivityDate="2016-11-28T22:45:18.320" Title="Methods to reduce dimensionality within a feature?" Tags="|machine-learning|classification|predictive-modeling|regression|feature-engineering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15381" PostTypeId="2" ParentId="15358" CreationDate="2016-11-27T20:31:01.280" Score="1" Body="&lt;p&gt;To add to molig answer one way is to plot the relationships between all the pairs of attributes by looking at the distribution of their interactions. That way you can at least remove highly correlated features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To at least scale the plotting of feature-feature plotting you would benefit from the automatic plotting in pandas if you are in python-land replacing &quot;data&quot; with your data and tweaking the plotting components:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas.tools.plotting import scatter_matrix&#xA;scatter_matrix(data, alpha=0.3, diagonal='kde')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2uAqC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2uAqC.png&quot; alt=&quot;scatter_matrix_from_pandas_docs&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other directions is to look at the feature-class relationships with a histogram per feature-class pair.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data.groupby('class').hist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Check this post for other viz options for your initial data exploration:&#xA;&lt;a href=&quot;http://machinelearningmastery.com/quick-and-dirty-data-analysis-with-pandas/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/quick-and-dirty-data-analysis-with-pandas/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="15445" LastActivityDate="2016-11-27T20:31:01.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15382" PostTypeId="2" ParentId="15377" CreationDate="2016-11-27T22:34:28.997" Score="0" Body="&lt;p&gt;FP-growth is a frequent pattern association rule learning algorithm. Thus it's a rule based machine learning algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you call the following :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val model = fpg.run(transactions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You are actually creating and Frequent Pattern model without generating candidates. So actually you'll need to generate afterwards if you need to use them with :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.generateAssociationRules(minConfidence)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, concerning the usual flow of building and validating such models, your code doesn't deal with that. It's usually done through quality measures variations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can also be done through &lt;em&gt;Feature extraction techniques&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With such techniques, you should consider the following. For each association rule, you'll need to measure the improvements in accuracy that a commonly used predictor can obtain from an additional feature, constructed according to the exceptions to the rule. In other terms, you'll have to select a reference set of rules that should help your model perform better. I strongly advice you to read &lt;a href=&quot;http://www.cs.upc.edu/~balqui/BalcazarDogbeyIDA2013.pdf&quot; rel=&quot;nofollow&quot;&gt;this paper&lt;/a&gt; about the topic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now what does that mean ? This means that you'll need to implement that pipeline yourself because it's not implemented in Spark yet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that this answers your question. &lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="5177" LastEditDate="2016-11-28T09:21:13.693" LastActivityDate="2016-11-28T09:21:13.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15384" PostTypeId="2" ParentId="9232" CreationDate="2016-11-28T02:31:00.853" Score="1" Body="&lt;p&gt;Assuming that the &quot;best&quot; model is the model that makes the most accurate out-of-training sample predictions, there are two types of measures that you can use. R squared will only increase as you add more variables hence it is not a suitable measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two common approaches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Indirectly estimate test error by making an adjustment to the training error (e.g. Adjusted Rsquared / BIC/ AIC). In your case, the simple linear regression performed slightly better in terms of Adj Rsquared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Directly estimate the test error using either a validation set or cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-28T02:31:00.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15386" PostTypeId="2" ParentId="15380" CreationDate="2016-11-28T04:13:38.600" Score="1" Body="&lt;p&gt;I think you are looking for a method that groups up the 16 nominal categorical values. If you conduct the regression problem on a tree-based algorithm, say, rpart, it would give you the various splits that you could consider aggregating them up to reduce the number of categorical values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the tree based algorithm may suggest a split of carrier IN (AS, HA, VC) vs. NOT IN (AS, HA, VC). This effectively would reduce the number of distinct values to 2. You might want to consider more than 1 split to take into account interactions. Overall, this approach would reduce the number of distinct values in a categorical variable.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-28T04:13:38.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15387" PostTypeId="1" CreationDate="2016-11-28T05:47:53.123" Score="8" ViewCount="10200" Body="&lt;p&gt;Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB.&#xA;Now I have more image files, so the training data set will become even larger, maybe up to 40 GB. And I am afraid that the *.npy files can not save this big of a data file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there another possible way to store such a large file?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I use Keras to build a neural network model, is it acceptable to split the training data into small parts so as to train the model without having to use all the training data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this sound like a reasonable approach?&lt;/p&gt;&#xA;" OwnerUserId="26181" LastEditorUserId="26596" LastEditDate="2016-11-29T10:53:00.257" LastActivityDate="2017-03-09T12:04:42.760" Title="How to deal with large training data?" Tags="|bigdata|tensorflow|keras|computer-vision|training|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15388" PostTypeId="2" ParentId="15387" CreationDate="2016-11-28T07:35:19.900" Score="8" Body="&lt;p&gt;I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;numpy&lt;/a&gt; files into smaller ones, let's say 64 samples per file and then load each file and call &lt;code&gt;train_on_batch&lt;/code&gt; on those images. You can use keras' &lt;code&gt;train_on_batch&lt;/code&gt; function to achieve this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;train_on_batch&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_on_batch(self, x, y, class_weight=None, sample_weight=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's designed to perform a single gradient update over one batch of samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Custom Generator&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another idea is to use generators which provide you with data given a directory. They can also be used for data augmentation, i.e. randomly generating new training data from your data. Here is an example from the keras documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_datagen = ImageDataGenerator(&#xA;        rescale=1./255,&#xA;        shear_range=0.2,&#xA;        zoom_range=0.2,&#xA;        horizontal_flip=True)&#xA;&#xA;# just rescale test data&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;&#xA;# this generator loads data from the given directory and 32 images &#xA;# chunks called batches. you can set this as you like&#xA;train_generator = train_datagen.flow_from_directory(&#xA;        'data/train',&#xA;        target_size=(150, 150),&#xA;        batch_size=32,&#xA;        class_mode='binary')&#xA;&#xA;# same es the train_generator    &#xA;validation_generator = test_datagen.flow_from_directory(&#xA;        'data/validation',&#xA;        target_size=(150, 150),&#xA;        batch_size=32,&#xA;        class_mode='binary')&#xA;&#xA;# loads sequentially images and feeds them to the model. &#xA;# the batch size is set in the constructor &#xA;model.fit_generator(&#xA;        train_generator,&#xA;        samples_per_epoch=2000,&#xA;        nb_epoch=50,&#xA;        validation_data=validation_generator,&#xA;        nb_val_samples=800)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26146" LastEditorUserId="12384" LastEditDate="2017-03-09T08:08:03.590" LastActivityDate="2017-03-09T08:08:03.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15391" PostTypeId="2" ParentId="15314" CreationDate="2016-11-28T11:38:47.823" Score="4" Body="&lt;p&gt;I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have basically two possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Classification of the text (Supervised learning). &lt;br&gt;&#xA;Supervised means&#xA;that you need first to externally apply labels (for example manually by humans) to examples of texts&#xA;(labels could be &quot;politics&quot; or &quot;show&quot;) and then use one of the&#xA;classification algorithms. &lt;br&gt; You have extracted words from the text,&#xA;so you could use a &quot;bag of words&quot; approach for classifying. &lt;br&gt; There exist adaptations&#xA;of classification algorithms (multi-label classification) in order to&#xA;provide multiple labels (such as one text is labelled both with&#xA;&quot;music&quot; and &quot;movie&quot;). &lt;br&gt;&#xA;You can find text corpora already pre-labelled, to train the algorithm and partly avoid the manual effort.    &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clustering the text, topic modelling (unsupervised learning).&lt;br&gt;&#xA;In this case you do not need to provide examples with labels but the algorithm will cluster the texts according to parameters like the similarity of two texts or on keywords/topics extracted from the texts. &lt;br&gt;&#xA;Although this method do not require labeled examples, the clusters you get as output will require fine tuning, e.g. number of clusters to produce, names of the clusters, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Since you mentioned SKlearn, you can find some directions on their web site: &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_extraction.html#applications-and-examples&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text Feature Extraction&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a comprehensive (bit older) summary: &lt;a href=&quot;https://arxiv.org/pdf/cs/0110053.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning in Automated Text Categorization by Fabrizio Sebastiani&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7952" LastActivityDate="2016-11-28T11:38:47.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15392" PostTypeId="1" CreationDate="2016-11-28T12:03:57.997" Score="2" ViewCount="1487" Body="&lt;p&gt;I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have  a collection of names which does not have any specific pattern and I need to separate individual names from that list. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;John Batista Barack Obama George W. Bush Serafim Batzoglou Vijay Pande Fei Fei Li Oussama Khatib Kwame Nkrumah John Oliver&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see there are two things peculiar with the paragraph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) space &quot; &quot; is the only delimiter, however, it is unusable as there are names which have middle names. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I have french, German, Spanish, Chinese and other ethnic sounding names written in English.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the challenge I am facing is to find names from this paragraph. I am very new to NLP and use Python and R for modeling work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="26596" LastEditDate="2016-11-29T08:11:26.403" LastActivityDate="2016-11-29T18:18:01.917" Title="extract names in a list of names" Tags="|nlp|information-retrieval|nltk|named-entity-recognition|stanford-nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15393" PostTypeId="1" CreationDate="2016-11-28T13:28:25.440" Score="6" ViewCount="2476" Body="&lt;p&gt;I am unable to find details for pretrained word2vec models. If someone is about to use a pre-trained model should it be clear what kind of pre-processing was done before the model was trained? Such as if lower cased was applied or removal of specific stopwords?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;noreferrer&quot;&gt;Glove&lt;/a&gt;: Is not referring to any kind of that information except of recommending stanford tokenizer (splitting words).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://code.google.com/archive/p/word2vec/&quot; rel=&quot;noreferrer&quot;&gt;Google pre trained model&lt;/a&gt;: Doesn't give any info for pre-processing of the pretrained model. Refers in a sentence to use a script for the wikipedia training data, but do not mention how the google news were preprocessed before trained.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And in other cases of pretrained models I cant find this kind of information. Someone could assume that there is a specific way of pre-processing for word2vec but I remember that the instructor of stanford NLP course, Richard Socher, that some people are removing stopwords and some others do not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also noticed that people are using different pre-processing methodology and same pre-trained model for the kaggle competition &lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial&quot; rel=&quot;noreferrer&quot;&gt;Bag of Words Meets Bags of Popcorn&lt;/a&gt; which doesnt make sense for me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any one can give some insights?&lt;/p&gt;&#xA;" OwnerUserId="14736" LastEditorUserId="14736" LastEditDate="2016-11-28T13:39:12.897" LastActivityDate="2018-05-02T19:30:48.870" Title="Word2Vec: Using pre-trained models" Tags="|nlp|word2vec|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15394" PostTypeId="1" AcceptedAnswerId="15401" CreationDate="2016-11-28T14:19:30.313" Score="1" ViewCount="1225" Body="&lt;p&gt;Between cross-validation runs of a xgboost classification model, I gather &lt;strong&gt;different validation scores&lt;/strong&gt;. This is normal, the Train/validation split and model state are different each time.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;flds = self.gsk.Splits(X, cv_folds=cv_folds)&#xA;cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=xgb_param['n_estimators'], nfold=cv_folds, flds, metrics='auc', early_stopping_rounds=50, verbose_eval=True)&#xA;&#xA;self.model.set_params(n_estimators=cvresult.shape[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To make the parameters selection, I run &lt;strong&gt;multiple times this CV&lt;/strong&gt; and average the results in order to attenuate those differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once my model parameters have been &quot;found&quot;, what is the correct way to train the model, which seems to &lt;strong&gt;have some inner random states&lt;/strong&gt; ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;train on the full train set and hope for the best? &lt;/li&gt;&#xA;&lt;li&gt;keep the model with the best validation score in my CV loop (I am concerned this will overfit)? &lt;/li&gt;&#xA;&lt;li&gt;bag all of them? &lt;/li&gt;&#xA;&lt;li&gt;bag only the good ones?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25311" LastEditorUserId="26596" LastEditDate="2016-11-29T10:17:27.610" LastActivityDate="2016-11-29T10:17:27.610" Title="Variance in cross validation score / model selection" Tags="|xgboost|cross-validation|model-selection|parameter-estimation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15396" PostTypeId="1" CreationDate="2016-11-28T15:29:51.543" Score="7" ViewCount="12368" Body="&lt;p&gt;I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to generate the training data by convolve  the image of license plat letter with kernels (somethings like gaussian blur,box blur) using python, but it does not resemble real data, here is the font look like:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/baIpq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/baIpq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I want to generate individual letter, that look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/K8FGu.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K8FGu.jpg&quot; alt=&quot;plat kendaraan&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any idea how to generate training data like above picture, ?&#xA;thank you&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="24696" LastEditDate="2017-12-16T15:43:03.583" LastActivityDate="2020-08-20T17:05:59.077" Title="How to generate training data for OCR" Tags="|machine-learning|python|classification|data|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15397" PostTypeId="1" AcceptedAnswerId="15414" CreationDate="2016-11-28T16:12:14.203" Score="1" ViewCount="327" Body="&lt;p&gt;Based on &lt;a href=&quot;https://datascience.stackexchange.com/a/1169&quot;&gt;this answer&lt;/a&gt;, we know that we can perform build covariance matrix incrementally when there are too many observations, whereas we can perform randomised SVD when there are too many variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer provide are clear and helpful. However, what if we have a large amount of observations &lt;strong&gt;AND&lt;/strong&gt; variables? e.g. 500,000 samples with 600,000 observations. In this case, the covariance matrix will be huge (e.g. 2,000 GB, assuming 8byte float, and if my calculation is correct) and will be impossible for us to fit it into memory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such scenario, is there anything that we can do to calculate the PCA, assuming we only want the top PCs (e.g. 15 PCs)?&lt;/p&gt;&#xA;" OwnerUserId="26575" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-11-29T06:29:13.833" Title="PCA on matrix with large M and N" Tags="|bigdata|dimensionality-reduction|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15398" PostTypeId="1" CreationDate="2016-11-28T17:10:45.847" Score="30" ViewCount="135980" Body="&lt;p&gt;I am building a multinomial logistic regression with sklearn (LogisticRegression). But after it finishes, how can I get a p-value and confident interval of my model? It only appears that sklearn only provides coefficient and intercept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you a lot.&lt;/p&gt;&#xA;" OwnerUserId="26579" LastEditorUserId="26596" LastEditDate="2016-11-29T10:51:45.420" LastActivityDate="2022-06-30T17:58:57.593" Title="How to get p-value and confident interval in LogisticRegression with sklearn?" Tags="|scikit-learn|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15400" PostTypeId="2" ParentId="15398" CreationDate="2016-11-28T17:23:15.203" Score="15" Body="&lt;p&gt;The short answer is that sklearn LogisticRegression does not have a built in method to calculate p-values.  Here are a few other posts that discuss solutions to this, however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/27928275/find-p-value-significance-in-scikit-learn-linearregression&quot;&gt;https://stackoverflow.com/questions/27928275/find-p-value-significance-in-scikit-learn-linearregression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/22306341/python-sklearn-how-to-calculate-p-values&quot;&gt;https://stackoverflow.com/questions/22306341/python-sklearn-how-to-calculate-p-values&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-11-28T17:23:15.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15401" PostTypeId="2" ParentId="15394" CreationDate="2016-11-28T17:51:00.360" Score="1" Body="&lt;p&gt;Since you want your model to be a general solution, you want to include all your data when building the final model.  You are correct in saying that keeping the model with the best validation score in the CV is overfitting.  Including these inner random states help generalize your model, and since you have already tuned your model parameters using CV, you can apply these parameters to the final model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for feature selection, you want to separate the data used to perform feature selection and the data used in cross-validation, so feature selection is performed on independent data in the cross-validation fold.  This prevents biasing the model.  If you were to select your features on the same data that you then use to cross-validate, you will likely overestimate your accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some other great posts that help:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation&quot;&gt;https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation&quot;&gt;https://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out Dikran Marsupial's answers to both, they are really good.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-11-28T17:51:00.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15404" PostTypeId="2" ParentId="15380" CreationDate="2016-11-28T18:16:51.393" Score="0" Body="&lt;p&gt;You could reduce the dimensionality by finding correlations within the known set of possibilities.  Perhaps &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Principal component analysis&lt;/a&gt; could be of use.  As a programmer I found the article's math and formal analysis quite academic: Take a good idea, create a formal description, and analize it (stet) to death.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the article has plenty of good references.  If you focus on the intention, there are many good ideas to be found.&lt;/p&gt;&#xA;" OwnerUserId="26582" LastActivityDate="2016-11-28T18:16:51.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15405" PostTypeId="2" ParentId="15398" CreationDate="2016-11-28T19:00:05.737" Score="14" Body="&lt;p&gt;One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for the parameters you are estimating, from which you can find the confidence intervals.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastActivityDate="2016-11-28T19:00:05.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15406" PostTypeId="2" ParentId="15396" CreationDate="2016-11-28T20:53:17.827" Score="8" Body="&lt;p&gt;That's a very good question. I am facing a similar issue with different data, but my research led me to &lt;a href=&quot;https://matthewearl.github.io/2016/05/06/cnn-anpr/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; which should be pretty much what you are looking for.&lt;/p&gt;&#xA;" OwnerUserId="26519" LastEditorUserId="98307" LastEditDate="2020-08-20T17:05:59.077" LastActivityDate="2020-08-20T17:05:59.077" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15408" PostTypeId="2" ParentId="15380" CreationDate="2016-11-28T22:45:18.320" Score="2" Body="&lt;p&gt;You could try clustering the data with the &lt;code&gt;carrier&lt;/code&gt; feature removed, add it back in, and look at the distributions of the &lt;code&gt;carrier&lt;/code&gt; feature per cluster.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-11-28T22:45:18.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15409" PostTypeId="2" ParentId="15327" CreationDate="2016-11-28T23:10:52.347" Score="1" Body="&lt;p&gt;The tuple of one partition is always on the same node because a partition itself is impartible. So if you do a &lt;code&gt;groupBy&lt;/code&gt; or write your own partitioner which partitions by key, all records with the same key/partition number will be shuffled to the same node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, transformations like &lt;a href=&quot;https://stackoverflow.com/questions/26741714/how-does-the-pyspark-mappartitions-function-work&quot;&gt;mapPartition&lt;/a&gt; which pass an iterator to a user defined function wouldn't work.&lt;/p&gt;&#xA;" OwnerUserId="26591" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-11-29T21:15:56.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15414" PostTypeId="2" ParentId="15397" CreationDate="2016-11-29T06:29:13.833" Score="1" Body="&lt;p&gt;There are a couple of things you can do.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sample a representative, but small set of your data, which will allow you to compute PCA in memory. But seeing as you have 600,00 observations this will most likely not result in any meaningful results.&lt;/li&gt;&#xA;&lt;li&gt;Use incremental PCA, here is a link: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html#sklearn.decomposition.IncrementalPCA&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html#sklearn.decomposition.IncrementalPCA&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But the main problem you have is that a number of samples are less than the amount of observations you have. I would recommend a different approach to dimensionality reduction. Autoencoders would be my recommendation to you. Autoencoders can be trained in an iterative fashion, circumventing your memory issue, and can learn more complicated projections than PCA (which is a linear transform). In case you want a linear projection you can have an autoencoder with one hidden layer, and the solution found by the neural network will be equal to the solution found by PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a couple of links you will find helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ai.stanford.edu/~quocle/tutorial2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ai.stanford.edu/~quocle/tutorial2.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/How-is-autoencoder-compared-with-other-dimensionality-reduction-algorithms&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-is-autoencoder-compared-with-other-dimensionality-reduction-algorithms&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~hinton/science.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.toronto.edu/~hinton/science.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2016-11-29T06:29:13.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15415" PostTypeId="1" AcceptedAnswerId="15421" CreationDate="2016-11-29T07:58:34.117" Score="3" ViewCount="1593" Body="&lt;p&gt;The &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton &amp;amp; Barto book on reinforcement learning&lt;/a&gt; mentions the 10 armed test bed in chapter 2, Bandit Problems:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To roughly assess the relative effectiveness of the greedy and&#xA;  ε-greedy methods, we compared them numerically on a suite of test&#xA;  problems. This was a set of 2000 randomly generated n-armed bandit&#xA;  tasks with n = 10. For each bandit, the action values, $q_∗(a), a = 1, . . . , 10,$ were selected according to a normal (Gaussian) distribution with mean 0 and variance 1. On t&lt;sup&gt;th&lt;/sup&gt; time step with a given bandit, the actual reward $R_t$ was the $q_∗(A_t)$ for the bandit (where $A_t$ was the action selected) plus a normally distributed noise term that was mean 0 and variance 1 [. . . .] We call this suite of test tasks the 10-armed testbed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is the reward function in the 10 armed test bed ?&#xA;I interpreted it as something like q*(a) + some normally random value.&#xA;where q*(a) is the true value of action a.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the reward function chosen this way in the test bed and how does the reward function affect the value estimations and the graphs ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the reason i'm asking this question is because i'm not completely clear how a reward looks like in the real world where i don't know anything about q*(a).&lt;/p&gt;&#xA;" OwnerUserId="26606" LastEditorUserId="836" LastEditDate="2017-12-07T16:59:53.497" LastActivityDate="2017-12-07T16:59:53.497" Title="What is the reward function in the 10 armed test bed?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15416" PostTypeId="2" ParentId="11868" CreationDate="2016-11-29T08:08:21.303" Score="2" Body="&lt;p&gt;In this example the learning rate of Stochastic Gradient Descent mentioned is too large and reducing it to 0.001 will solve the issue (but the learning will be slow). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between sigmoid and relu arises because relu has a slope of 1 (constant) while sigmoid gets saturated and therefore can have a much smaller gradient. Therefore even with the high learning rate sigmoid survives but relu keeps fluctuating around 10%.&lt;/p&gt;&#xA;" OwnerUserId="26600" LastActivityDate="2016-11-29T08:08:21.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15419" PostTypeId="1" CreationDate="2016-11-29T10:09:55.733" Score="3" ViewCount="1247" Body="&lt;p&gt;I have a data set of with a distribution that looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZOPbL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZOPbL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in the green area):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vUuMA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vUuMA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to do this by taking the data, putting into separate &quot;buckets&quot; &#xA; (distribute the data into X buckets, taking up to Y samples from each bucket), but I was wondering if there is there an easier way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. the result doesn't have to be 100% accurate - a good approximation is enough.&lt;/p&gt;&#xA;" OwnerUserId="26608" LastEditorUserId="31513" LastEditDate="2017-05-13T11:31:20.700" LastActivityDate="2020-03-31T16:49:51.663" Title="How to sample a statistically uniform dataset" Tags="|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15421" PostTypeId="2" ParentId="15415" CreationDate="2016-11-29T10:15:08.373" Score="2" Body="&lt;p&gt;The reward function in the Chapter 2 test bed is simply the &quot;true&quot; mean value for the chosen action, plus a &quot;noise term&quot; which is normal distribution with mean 0, standard deviation 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The noise has the same distribution as the initial setting of &quot;true&quot; values. The difference is you set the true values at the start and do not change them, then add noise on evaluation of each reward. The goal for the learner is then to find the best &quot;true&quot; value whilst only seeing the reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This matches your understanding as I read it from the question. You could write it like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initialisation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\forall a \in A: q_*(a) \leftarrow N(0,1)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Evaluation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$R_t = r(A_t) = q_*(A_t) + N(0,1)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Where $N(\mu,\sigma)$ is a sample from the normal distribution, mean $\mu$, standard deviation $\sigma$&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why is the reward function chosen this way in the test bed and how does the reward function affect the value estimations and the graphs ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For a bandit problem to be non-trivial, the reward function needs to be stochastic, such that it is not possible to immediately discover the best action, there should be some uncertainty on what the best action to take is, even after taking many samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the noise is there to provide at least some difficulty - without it finding the best action would be a trivial $argmax$ over the 10 possible actions. The noise does &lt;em&gt;not&lt;/em&gt; represent uncertainty in the sensing (although that could also be a real world issue), but variability of the environment in response to an action. The test examples could have almost &lt;em&gt;any&lt;/em&gt; distribution (e.g. $p(-1.0|a=1) = 0.9, p(9.0|a=1) = 0.1$ for $q_*(a=1) = 0.0$), the authors made a choice that was concise to describe and useful for exploring the different techniques in the chapter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The specific reward function will affect the learning graphs. The test bed has been chosen so that ratio of noise to magnitude of &quot;true&quot; values is high. In turn, this means that value estimates will converge relatively slowly (as a ratio to the true values), and this exposes differences between different sampling and estimation techniques when they are plotted by time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your concern:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I guess the reason i'm asking this question is because i'm not &#xA;  completely clear how a reward looks like in the real world where i&#xA;  don't know anything about q*(a).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the real world you may need to &lt;em&gt;sense&lt;/em&gt; or receive the reward from the environment. Obviously that complicates the test scenario, and doesn't add anything to the understanding of the maths, so the test bed just generates some imaginary distribution for the environment inside the problem. The &quot;sensing&quot; in the test is just assumed with a reward amount defined by the test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To qualify as a simple (and static) bandit problem, the reward has to be immediately apparent on taking the action, and have no dependency on current state or history. That constrains the problem somewhat - it is not the &lt;em&gt;full&lt;/em&gt; reinforcement problem. So real-world examples tend to be about gambling with limited choice on independent repeatable events.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-29T12:46:12.600" LastActivityDate="2016-11-29T12:46:12.600" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15423" PostTypeId="1" CreationDate="2016-11-29T12:08:58.323" Score="8" ViewCount="8164" Body="&lt;p&gt;The paper explaining &lt;a href=&quot;http://ieeexplore.ieee.org/document/374604/&quot; rel=&quot;noreferrer&quot;&gt;'Advantage Updating'&lt;/a&gt; as a method to improve Q-learning uses the following as its motivation.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Q-learning requires relatively little computation per update, but it is useful to consider how the number of updates required scales with noise or with the duration of a time step, At. An important consideration is the&#xA;  relationship between Q values for the same state, and between Q values for the same action. The Q values Q(x,u1) and Q(x,u2) represent the long-term&#xA;  reinforcement received when starting in state x and performing action u1 or u2 respectively, followed by optimal actions thereafter. In a typical reinforcement learning problem with continuous states and actions, it&#xA;  is frequently the case that performing one wrong action in a long sequence of optimal actions will have little effect on the total reinforcement. In such a case, Q(x,u1) and Q(x,u2) will have relatively close values. On the other hand, the values of widely separated states will typically not be close to each other. Therefore Q(x1,u) and Q(x2.u) may differ greatly for some choices of x1 and x2. Therefore, if the network representing the Q function makes even small errors, the policy derived from it will have large errors. As the time step duration dt approaches zero, the penalty for one wrong action in a sequence decreases, the Q values for different actions in a given state become closer, and the implied policy becomes even more sensitive to noise or function approximation error. In the limit, for continuous time, the Q function contains no information about the policy.&#xA;  Therefore, Q-learning would be expected to learn slowly when the time steps are of short duration, due to the sensitivity to errors, and it is incapable of learning in continuous time. This problem is not a property of&#xA;  any particular function approximation system; rather, it is inherent in the definition of Q values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How do I mathematically prove that this effect occurs?&lt;/p&gt;&#xA;" OwnerUserId="26612" LastEditorUserId="26612" LastEditDate="2016-11-29T12:15:23.763" LastActivityDate="2016-12-05T17:49:50.710" Title="Understanding advantage functions" Tags="|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15424" PostTypeId="2" ParentId="15338" CreationDate="2016-11-29T13:23:46.237" Score="1" Body="&lt;p&gt;It seems like you just got a bug @ your parse_data. (see my comment: replace &lt;code&gt;X[idx][int(s[7])+j] = 1.0 # a6&lt;/code&gt; to &lt;code&gt;X[i][int(s[7])+j] = 1.0 # a6&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I altered the parameters. You will see that they have a big influence. I guess this problem is easily stuck on a local minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you run my code please (Only difference is that it is python 3, but should run on python 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the link of the data saved in data1.txt:&#xA;&lt;a href=&quot;http://mlearn.ics.uci.edu/databases/monks-problems/monks-2.test&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mlearn.ics.uci.edu/databases/monks-problems/monks-2.test&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;import numpy as np&#xA;    import os&#xA;    import matplotlib.pyplot as plt&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;abspath = os.path.abspath(__file__)&#xA;dname = os.path.dirname(abspath)&#xA;os.chdir(dname)&#xA;&#xA;# paramaters&#xA;eta = 0.05&#xA;hidden_dim = 2&#xA;alpha = 0.8&#xA;reg_lambda = 0.0&#xA;epochs = 100&#xA;&#xA;file = 'data1.txt'&#xA;&#xA;input_dim = 17&#xA;output_dim = 1&#xA;&#xA;# read and parse MONK data&#xA;def parse_data(text):&#xA;    with open(text) as f:&#xA;        content = f.readlines()&#xA;    t = np.zeros((len(content), output_dim))&#xA;    X = np.zeros((len(content), input_dim))&#xA;    for i,val in enumerate(content):&#xA;        s = val.split(&quot; &quot;)&#xA;        t[i] = float(s[1])&#xA;        # for idx in range(17):&#xA;        #   X[i][idx] = 0.0&#xA;        j = -1&#xA;        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16&#xA;        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6&#xA;        X[i][int(s[2])+j] = 1.0 # a1&#xA;        j = 2&#xA;        X[i][int(s[3])+j] = 1.0 # a2&#xA;        j = 5&#xA;        X[i][int(s[4])+j] = 1.0 # a3&#xA;        j = 7&#xA;        X[i][int(s[5])+j] = 1.0 # a4&#xA;        j = 10&#xA;        X[i][int(s[6])+j] = 1.0 # a5&#xA;        j = 14&#xA;        X[i][int(s[7])+j] = 1.0 # a6&#xA;&#xA;    return (X, t)&#xA;&#xA;X, t = parse_data(file)&#xA;&#xA;print(np.sum(t)/len(t))&#xA;&#xA;def sigmoid(x):&#xA;    output = 1/(1+np.exp(-x))&#xA;    return output&#xA;&#xA;def sigmoid_output_to_derivative(output):&#xA;    return output*(1-output)&#xA;&#xA;# initialize weights&#xA;seed = 1&#xA;np.random.seed(seed)&#xA;W0 = np.random.normal(0, 1/input_dim, size = (input_dim,hidden_dim))&#xA;W1 = np.random.normal(0, 1/hidden_dim, size = (hidden_dim,output_dim))&#xA;b0 = np.zeros((1, hidden_dim))&#xA;b1 = np.zeros((1, output_dim))&#xA;dW0 = 0&#xA;dW1 = 0&#xA;&#xA;&#xA;error_list = []&#xA;for j in range(epochs):&#xA;    # forward propagation&#xA;    l0 = X&#xA;    l1 = sigmoid(np.dot(l0,W0) + b0)&#xA;    l2 = sigmoid(np.dot(l1,W1) + b1)&#xA;&#xA;    # back propagation&#xA;    l2_error = l2 - t&#xA;    l2_delta = l2_error*sigmoid_output_to_derivative(l2)&#xA;&#xA;    l1_error = l2_delta.dot(W1.T)&#xA;    l1_delta = l1_error*sigmoid_output_to_derivative(l1)&#xA;&#xA;    error = np.mean(np.square(l2_error))&#xA;    error_list.append(error)&#xA;    if (j% 20) == 0:&#xA;        print(&quot;Training error after&quot;, j, &quot;iterations:&quot;,error)&#xA;&#xA;    dW1 = -eta * (l1.T.dot(l2_delta) + reg_lambda * W1) + alpha * dW1&#xA;    dW0 = -eta * (l0.T.dot(l1_delta) + reg_lambda * W0) + alpha * dW0&#xA;    db1 = -eta*(np.sum(l2_delta, axis=0))&#xA;    db0 = -eta*(np.sum(l1_delta, axis=0))&#xA;&#xA;    W1 += dW1&#xA;    W0 += dW0&#xA;    b1 += db1&#xA;    b0 += db0&#xA;&#xA;plt.plot(error_list)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25104" LastEditorUserId="26475" LastEditDate="2016-11-29T17:55:17.680" LastActivityDate="2016-11-29T17:55:17.680" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15429" PostTypeId="2" ParentId="15392" CreationDate="2016-11-29T18:18:01.917" Score="1" Body="&lt;p&gt;I'm not an expert here, so here's my (brute force?) method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SeatGeek has open-sourced a python library called &lt;a href=&quot;http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;fuzzywuzzy&quot;&gt;fuzzywuzzy&lt;/a&gt; which is great at text matching.  It has a function called &lt;code&gt;token_set_ratio&lt;/code&gt; that compares two multi-word strings and scores their distance.  It can consider just the intersection of individual words and score only that intersection. eg &quot;Barack Obama Bill Clinton Madeleine Albright&quot; will get a score of 100 with &quot;Bill Clinton&quot;, but also with &quot;Bill Barack&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll need to build a list of known names, judging by the list they're famous people, which a quick google search shows hundreds of sites that have that ready for you.  With that list, you can just loop your way through, computing a score with &lt;code&gt;token_set_ratio&lt;/code&gt;.  If the famous name beats some threshold score against your string, you append it to a list of captured names.&lt;/p&gt;&#xA;" OwnerUserId="25611" LastActivityDate="2016-11-29T18:18:01.917" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15430" PostTypeId="2" ParentId="4957" CreationDate="2016-11-29T20:33:32.030" Score="4" Body="&lt;p&gt;(Most) Machine Learning algorithms are essentially optimization problems where you minimize/maximize an objective function subject to certain constraints. Excel comes with the Solver add-in which is pretty handy for lightweight problems, so it is entirely possible for you to build a Machine Learning model within Excel! (I've done it myself)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data size is reasonably small (say &amp;lt;10k rows and not too many columns), it is in fact pretty quick and easy to build certain ML models within Excel. All you need is to learn how to use the Excel Solver, and the built-in matrix functions for vectorized computations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, Neural Networks and Logistic Regressions are particularly easy to build due to the simplicity of their objective function. If you are really inclined to try you could refer to the following link:&#xA;&lt;a href=&quot;http://www.xlpert.com/ebook/Build_Neural_Network_With_MS_Excel_sample.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.xlpert.com/ebook/Build_Neural_Network_With_MS_Excel_sample.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, Excel does has major limitations. There are numerical stability issues, it is tricky to write loops (without VBA), it is a pain to write lengthy functions in Excel and there's no concept of structure so I wouldn't try to build things like Random Forest. That being said, there is still a good amount of algorithms you can implement with Excel for fun. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, for serious analysis, I would still recommend you to use proper tools for that.&lt;/p&gt;&#xA;" OwnerUserId="26629" LastActivityDate="2016-11-29T20:33:32.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15431" PostTypeId="1" AcceptedAnswerId="15454" CreationDate="2016-11-29T21:27:32.290" Score="0" ViewCount="199" Body="&lt;p&gt;I am planning to use heap structure to find the minimum distance between a set of 2D points and form a cluster.. and after to spend a couple of hours surfing on the internet, I have not still gotten a clearly example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have a set of 2d points (x1,y1; x2,y2;....;xn,yn) and then you compute all possible distances between these pairs of points to finally get a matrix with all distances values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, I want to use a min heap structure to sort data and match the pairs with the mininum distance... &lt;em&gt;and here is my question&lt;/em&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which value/data is really stored at every root or leaves in heap structure?&lt;/strong&gt; I know that the root of min Heap is the shortest distance but, the data stored in every globe at Heap graph represent the index of every distances of the pairs of points or the pairs of points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first case, &lt;em&gt;if you store the index value located at the top of heap&lt;/em&gt;, &lt;strong&gt;Do you need to perform after a new search in the distance matrix to obtain which pairs of points form these shortest distance?&lt;/strong&gt; Is it optimal to do that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope my question was clear and thanks for your support.&lt;/p&gt;&#xA;" OwnerUserId="26126" LastActivityDate="2019-11-16T17:07:58.510" Title="What is stored in heap structure in the following example?" Tags="|machine-learning|data-mining|algorithms|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15432" PostTypeId="2" ParentId="5600" CreationDate="2016-11-29T22:24:53.703" Score="0" Body="&lt;p&gt;I think that by doing so, you have expanded the feature space, and I don't think that this additional variable is redundant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cluster categorical variable is a linear combination of the other variables. When you apply Lasso on the dataset, the categorical variable undergoes one-hot encoding and Lasso would pick a subset of values given a level of regularisation. Suppose in the best performing regularisation parameter, Lasso picked clusters 1,3,7 out of clusters 1 - 10; These pockets of clusters might not be picked up using just Lasso on the other variables without the cluster variable.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-29T22:24:53.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15434" PostTypeId="1" AcceptedAnswerId="15435" CreationDate="2016-11-29T23:46:11.157" Score="2" ViewCount="291" Body="&lt;p&gt;What are some good error metrics for multi-label (not mutli-class) problem in industry?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/dev/modules/multiclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/dev/modules/multiclass.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6567" LastActivityDate="2016-11-30T00:19:25.750" Title="What are some good error metrics for multi-label (not mutli-class) problem in industry?" Tags="|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15435" PostTypeId="2" ParentId="15434" CreationDate="2016-11-30T00:19:25.750" Score="1" Body="&lt;p&gt;A common example is the Jaccard similarity coefficient:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$J(Y, P) = \frac{|Y~\cap~P|}{|Y~\cup~P|}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P$ is the set of predicted labels for an instance and $Y$ is the true set of labels. This gives a value between $0$ and $1$ for each instance, which you can average over the whole test set to give a score. If $P = Y$, then $J(Y, P) = 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is implemented in &lt;code&gt;scikit-learn&lt;/code&gt; as &lt;code&gt;sklearn.metrics.jaccard_similarity_score&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-11-30T00:19:25.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15436" PostTypeId="1" AcceptedAnswerId="15437" CreationDate="2016-11-30T01:50:16.640" Score="4" ViewCount="260" Body="&lt;p&gt;I was a given a data set with different predictors about a store and the idea is to forecast the number of daily shoppers. The predictors are the weekday, time of the day (morning, afternoon, evening), week number, month, weather (humidity, dew point, temperature), holidays. The outcome variable is the number of visitors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build a regression model to predict the number of visitors using traditional machine learning algorithms such as random forests, SVM, and the like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main concern is how to validate this model using CV since some of the predictors are time-related. Plain vanilla CV cannot be performed here. &lt;a href=&quot;https://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection&quot;&gt;In this question&lt;/a&gt;, they suggest a way to perform this but my problem is that I only have data from June 2015 to present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial idea was the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;train with data from June 2015-December 2015. Test with January&lt;/li&gt;&#xA;&lt;li&gt;train with June 2015-January 2016. Test with February 2016.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Each time one month of data is added to the training data after having asses the error for that month. Then compute average performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this approach reasonable or not?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If so, should I get rid of the month variable? Note that in a., for instance, I am testing with some data that belongs to different months that the one used for training. I mean, for the training I used data from June to December 2015, but I am testing for January 2016. Seasonality can be something I am missing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to validate such models in general?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10155" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-12-06T20:50:31.323" Title="What is a better approach for cross-validation with time-related predictors" Tags="|machine-learning|predictive-modeling|time-series|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15437" PostTypeId="2" ParentId="15436" CreationDate="2016-11-30T02:47:07.050" Score="1" Body="&lt;p&gt;One such way to handle a time series cross-validation is to take a look at the below Python code from &lt;a href=&quot;http://francescopochetti.com/pythonic-cross-validation-time-series-pandas-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def performTimeSeriesCV(X_train, y_train, number_folds, algorithm, parameters):&#xA;&quot;&quot;&quot;&#xA;Given X_train and y_train (the test set is excluded from the Cross Validation),&#xA;number of folds, the ML algorithm to implement and the parameters to test,&#xA;the function acts based on the following logic: it splits X_train and y_train in a&#xA;number of folds equal to number_folds. Then train on one fold and tests accuracy&#xA;on the consecutive as follows:&#xA;- Train on fold 1, test on 2&#xA;- Train on fold 1-2, test on 3&#xA;- Train on fold 1-2-3, test on 4&#xA;....&#xA;Returns mean of test accuracies.&#xA;&quot;&quot;&quot;&#xA;&#xA;print 'Parameters --------------------------------&amp;gt; ', parameters&#xA;print 'Size train set: ', X_train.shape&#xA;&#xA;# k is the size of each fold. It is computed dividing the number of &#xA;# rows in X_train by number_folds. This number is floored and coerced to int&#xA;k = int(np.floor(float(X_train.shape[0]) / number_folds))&#xA;print 'Size of each fold: ', k&#xA;&#xA;# initialize to zero the accuracies array. It is important to stress that&#xA;# in the CV of Time Series if I have n folds I test n-1 folds as the first&#xA;# one is always needed to train&#xA;accuracies = np.zeros(folds-1)&#xA;&#xA;# loop from the first 2 folds to the total number of folds    &#xA;for i in range(2, number_folds + 1):&#xA;    print ''&#xA;&#xA;    # the split is the percentage at which to split the folds into train&#xA;    # and test. For example when i = 2 we are taking the first 2 folds out &#xA;    # of the total available. In this specific case, we have to split the&#xA;    # two of them in half (train on the first, test on the second), &#xA;    # so split = 1/2 = 0.5 = 50%. When i = 3 we are taking the first 3 folds &#xA;    # out of the total available, meaning that we have to split the three of them&#xA;    # in two at split = 2/3 = 0.66 = 66% (train on the first 2 and test on the&#xA;    # following)&#xA;    split = float(i-1)/i&#xA;&#xA;    # example with i = 4 (first 4 folds):&#xA;    #      Splitting the first       4        chunks at          3      /        4&#xA;    print 'Splitting the first ' + str(i) + ' chunks at ' + str(i-1) + '/' + str(i) &#xA;&#xA;    # as we loop over the folds X and y are updated and increase in size.&#xA;    # This is the data that is going to be split and it increases in size &#xA;    # in the loop as we account for more folds. If k = 300, with i starting from 2&#xA;    # the result is the following in the loop&#xA;    # i = 2&#xA;    # X = X_train[:(600)]&#xA;    # y = y_train[:(600)]&#xA;    #&#xA;    # i = 3&#xA;    # X = X_train[:(900)]&#xA;    # y = y_train[:(900)]&#xA;    # .... &#xA;    X = X_train[:(k*i)]&#xA;    y = y_train[:(k*i)]&#xA;    print 'Size of train + test: ', X.shape # the size of the dataframe is going to be k*i&#xA;&#xA;    # X and y contain both the folds to train and the fold to test.&#xA;    # index is the integer telling us where to split, according to the&#xA;    # split percentage we have set above&#xA;    index = int(np.floor(X.shape[0] * split))&#xA;&#xA;    # folds used to train the model        &#xA;    X_trainFolds = X[:index]        &#xA;    y_trainFolds = y[:index]&#xA;&#xA;    # fold used to test the model&#xA;    X_testFold = X[(index + 1):]&#xA;    y_testFold = y[(index + 1):]&#xA;&#xA;    # i starts from 2 so the zeroth element in accuracies array is i-2. performClassification() is a function which takes care of a classification problem. This is only an example and you can replace this function with whatever ML approach you need.&#xA;    accuracies[i-2] = performClassification(X_trainFolds, y_trainFolds, X_testFolds, y_testFolds, algorithm, parameters)&#xA;&#xA;    # example with i = 4:&#xA;    #      Accuracy on fold         4     :    0.85423&#xA;    print 'Accuracy on fold ' + str(i) + ': ', acc[i-2]&#xA;&#xA;# the function returns the mean of the accuracy on the n-1 folds    &#xA;return accuracies.mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If on the other hand, you prefer R you can explore the &lt;a href=&quot;https://topepo.github.io/caret/model-training-and-tuning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;timeslice method&lt;/a&gt; in the caret package and make use of the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(caret) &#xA;library(ggplot2) &#xA;data(economics) &#xA;myTimeControl &amp;lt;- trainControl(method = &quot;timeslice&quot;,&#xA;                              initialWindow = 36,&#xA;                              horizon = 12,&#xA;                              fixedWindow = TRUE)&#xA;&#xA;plsFitTime &amp;lt;- train(unemploy ~ pce + pop + psavert,&#xA;                    data = economics,&#xA;                    method = &quot;pls&quot;,&#xA;                    preProc = c(&quot;center&quot;, &quot;scale&quot;),&#xA;                    trControl = myTimeControl)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-11-30T02:47:07.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15438" PostTypeId="2" ParentId="10299" CreationDate="2016-11-30T04:54:57.667" Score="5" Body="&lt;p&gt;Non-Negative Matrix Factorization (NMF) is described well in the paper by &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjQvKC6zc_QAhXLMSYKHY9pDVwQFggcMAA&amp;amp;url=http%3A%2F%2Fwww.columbia.edu%2F~jwp2128%2FTeaching%2FW4721%2Fpapers%2Fnmf_nature.pdf&amp;amp;usg=AFQjCNHOf7BKOMfBKKs1wJ2SxSwfj7bgaA&quot; rel=&quot;noreferrer&quot;&gt;Lee and Seung, 1999&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Simply Put&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NMF takes as an input a &lt;a href=&quot;https://en.wikipedia.org/wiki/Document-term_matrix&quot; rel=&quot;noreferrer&quot;&gt;term-document matrix&lt;/a&gt; and generates a set of topics that represent weighted sets of co-occurring terms. The discovered topics form a basis that provides an efficient representation of the original documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;About NMF&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NMF is used for &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_extraction&quot; rel=&quot;noreferrer&quot;&gt;feature extraction&lt;/a&gt; and is generally seen to be useful when there are many attributes, particularly when the attributes are ambiguous or are not strong predictors. By combining attributes NMF can display patterns, topics, or themes which have importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, one encounters NMF typically where text is involved. Consider an example, where the same word (love) in a document could different meanings:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I &lt;em&gt;love&lt;/em&gt; lettuce wraps.&lt;/li&gt;&#xA;&lt;li&gt;I &lt;em&gt;love&lt;/em&gt; the way I feel when I'm on vacation in Mexico.&lt;/li&gt;&#xA;&lt;li&gt;I &lt;em&gt;love&lt;/em&gt; my dog, Euclid.&lt;/li&gt;&#xA;&lt;li&gt;I &lt;em&gt;love&lt;/em&gt; being a Data Scientist.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In all 4 cases, the word 'love' is used, but it has a different meaning to the reader. By combining attributes, NMF introduces context which creates additional predictive power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&quot;love&quot; + &quot;lettuce \ wraps&quot; \ \Rightarrow \ &quot;pleasure \ by \ food&quot;$&#xA;$&quot;love&quot; + &quot;vacation \ in \ Mexico&quot; \ \Rightarrow \ &quot;pleasure \ by \ relaxation&quot;$&#xA;$&quot;love&quot; + &quot;dog&quot; \ \Rightarrow \ &quot;pleasure \ by \ companionship&quot;$&#xA;$&quot;love&quot; + &quot;Data \ Scientist&quot; \ \Rightarrow  \ &quot;pleasure \ by  \ occupation&quot;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How Does It Happen&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NMF breaks down the multivariate data by creating a user-defined number of features. Each one of these features is a combination of the original attribute set. It is also key to remember these coefficients of these linear combinations are non-negative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to think about it is that NMF breaks your original data features (let's call it V) into the product of two lower ranked matrices (let's call it W and H). NMF uses an iterative approach to modify the initial values of W and H so that the product approaches V. When the approximation error converges or the user-defined number of iterations is reached, NMF terminates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NMF data preparation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Numeric attributes are normalized. &lt;/li&gt;&#xA;&lt;li&gt;Missing numerical values are replaced with the mean. &lt;/li&gt;&#xA;&lt;li&gt;Missing categorical values are replaced with the mode.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is important to note that outliers can impact NMF significantly. In practice, most Data Scientist use a clipping transformation before binning or normalizing. In addition, NMF in many cases will benefit from normalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in many other algorithmic cases, to improve matrix factorization, one needs to decrease the error tolerance (which will increase compute time). &lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-11-30T04:54:57.667" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15439" PostTypeId="1" AcceptedAnswerId="15452" CreationDate="2016-11-30T07:00:23.107" Score="3" ViewCount="502" Body="&lt;p&gt;My company has recently engaged a consultant firm to develop a predictive model to detect defective works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that there are many ways to validate the model, for example, using k-fold cross-validation and I believe that the consultant firm will carry out the validation before submitting the model to us.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, at the employer's side, how can I check the accuracy of the model developed by the consultant firm ?? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone suggested that I can give 2000-2015 data to the consultant firm and keep 2016 data for our own checking. However, a model with good accuracy on 2016 data does not imply that it will have good predictive power in the future. In my view, keeping 2016 data for checking seems like adding one more test set for validation, which in my view, is unnecessary since I already hv &quot;k-fold&quot; cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone advise what the employer can do to check the consultant's model? &lt;/p&gt;&#xA;" OwnerUserId="26635" LastEditorUserId="836" LastEditDate="2016-11-30T20:34:18.933" LastActivityDate="2016-11-30T22:15:28.907" Title="Check Accuracy of Model Provided by Consultant" Tags="|machine-learning|predictive-modeling|cross-validation|accuracy|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15440" PostTypeId="1" CreationDate="2016-11-30T07:24:29.827" Score="0" ViewCount="63" Body="&lt;p&gt;I wish to construct feature vectors of words in a document and then calculate their linkage distance to detect anomalies. My question is how can I model these features ? If possible please give an example so I can understand better. Further how distance calculated will vary from the text that is anomalous and the one that is normal ? (Please refer to D. Guthrie, PhD. thesis for more clarification). &lt;/p&gt;&#xA;" OwnerUserId="25922" LastActivityDate="2017-03-02T08:09:41.367" Title="What could be possible features of a textual word?" Tags="|machine-learning|classification|data-cleaning|svm|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15441" PostTypeId="2" ParentId="15439" CreationDate="2016-11-30T07:30:09.040" Score="2" Body="&lt;p&gt;I would agree with the suggestion with holding out the 2016 data to check the external agency's work. Without inspecting the code, you just can't be sure that the k-fold cross-validation process has been performed properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another benefit of using the 2016 held out set is that you can see if the model trained on past years' data work well in future years data. There might be a &lt;a href=&quot;https://en.wikipedia.org/wiki/Concept_drift&quot; rel=&quot;nofollow noreferrer&quot;&gt;concept drift&lt;/a&gt; in the new year as the true relationship between Y and Xs changes. In cross validation, each fold belongs to the same time period as the training data set.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-11-30T07:30:09.040" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15442" PostTypeId="2" ParentId="15436" CreationDate="2016-11-30T08:48:22.853" Score="3" Body="&lt;p&gt;&lt;strong&gt;Your approach 1)&lt;/strong&gt; is correct, use n ordered data to predict n+1.&#xA;You will have to identify the correct window for prediction and a model not too flexible if you feel the amount of data you have is little.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not forget about &lt;strong&gt;feature engineering and data preparation&lt;/strong&gt;. The seasonality you mention can be removed if you identity it correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Validation of the model is done as usual, with a square loss function&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-11-30T08:48:22.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15443" PostTypeId="2" ParentId="15440" CreationDate="2016-11-30T09:20:46.647" Score="1" Body="&lt;p&gt;That is a rather broad question. If you are on a document level, you could (after removing stop words) determine the most important words in your corpus by calculating the tf/idf measure &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Tf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Tf&lt;/a&gt;–idf &#xA;Then you could define a vector over the top n words and represent every document as a vector of the relative frequencies of the top n words in that document and work on these vectors. &#xA;On a sentence level that could work as well, on a word level obviously not. Anyway, information retrieval and computational linguistics are a rather large research area. Maybe the best way would be to start with a book that provides a good overview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you asked for some more specific recommendations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can get access to (or actually want to buy it), definitely this one: &lt;a href=&quot;https://www.amazon.de/o/ASIN/0131873210/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also take a shot this: &lt;a href=&quot;https://www.cs.uic.edu/~liub/FBS/SentimentAnalysis-and-OpinionMining.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sentiment Analysis and Opinion Mining&lt;/a&gt; this is probably not your focus, but it covers feature representation to some extent and is for free. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More like a handbook with practical approaches is: &lt;a href=&quot;http://shop.oreilly.com/product/9780596529321.do&quot; rel=&quot;nofollow noreferrer&quot;&gt;Programming Collective Intelligence&lt;/a&gt;, look for it at Google, it is also available for free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise if you want to look for yourself for books at a library or so, your keywords are &quot;Computational Linguistics&quot; and &quot;Natural Language Processing&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="26519" LastEditorUserId="26519" LastEditDate="2016-12-02T07:22:20.197" LastActivityDate="2016-12-02T07:22:20.197" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15444" PostTypeId="1" CreationDate="2016-11-30T10:32:42.087" Score="4" ViewCount="6540" Body="&lt;p&gt;I am currently trying to understand the BPTT for Long Short Term Memory (LSTM) in TensorFlow. I get that the parameter &lt;code&gt;num_steps&lt;/code&gt; is used for the range that the RNN is rolled out and the Error backpropagated. I got a general question of how this works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference a repitition of the formulas. I'm referring to:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/3zKXt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Formulas LSTM&lt;/a&gt; (&lt;a href=&quot;https://arxiv.org/abs/1506.00019&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1506.00019&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&#xA;What paths are backpropagated that many steps? The constant error carousel is created by the formula 5, and the derivate for backpropagation ($s(t)\rightarrow s(t-1)$) is $1$ for all timesteps. This is why LSTMs capture long range dependencies. I get confused with the dependencies of $g(t)$, $i(t)$, $f(t)$ and $o(t)$ of $h(t-1)$. In words: The current gates do not just depend on the input, but also on the last hidden state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Doesn't this dependency lead to the exploding/vanishing gradients problem again?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I backpropagate along these connections I get gradients that are not one. Peephole connections essentially lead to the same problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="26643" LastEditorUserId="26146" LastEditDate="2016-12-04T15:04:43.500" LastActivityDate="2017-10-15T21:22:53.100" Title="Backpropagation Through Time (BPTT) of LSTM" Tags="|tensorflow|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15445" PostTypeId="1" CreationDate="2016-11-30T11:26:35.583" Score="2" ViewCount="195" Body="&lt;p&gt;I'm trying to create a network visualization to study the flights to and from a certain airport.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data consists of 123000 rows in this format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Origin_ID    Destination_ID    Frequency&#xA;  1726            3504           40000&#xA;  3504            4517              40&#xA;  5616            7205              38&#xA;               ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to create a image similar to this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TVi2r.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TVi2r.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I want to be able to see the strength of the relationship (based on the frequency) and identify which combinations are most common in that airport.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can it be created in R using this format? Or do you suggest any other open source platform to create this visualization?&lt;/p&gt;&#xA;" OwnerUserId="24719" LastEditorUserId="31513" LastEditDate="2017-04-24T09:27:02.657" LastActivityDate="2017-04-24T09:27:02.657" Title="Network Visualization - R code - Which Package is needed?" Tags="|r|visualization|social-network-analysis|sequential-pattern-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15451" PostTypeId="1" CreationDate="2016-11-30T17:32:14.710" Score="3" ViewCount="870" Body="&lt;p&gt;Recently I discussed the following topic with a friend. The setting is that we have a one-dimensional set of data. (In the example it was points of students, we would be grading.) The goal was to make a density estimation, but not to use anything &quot;fancy&quot; like kernel density estimation, but just use a Gaussian as the estimation. (Sure that makes the big assumption that the data is Gaussian, but that is not the point here.) We discussed two ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Make a density estimation using an unsupervised learning method, e.g. using EM-algorithm. In this case the claim is that simply calculating the mean of the data and the standard derivation is already giving one the parameters to get the Gaussian parametrized the right way.&lt;/li&gt;&#xA;&lt;li&gt;Add up the number of occurrences for each value and then use a supervised learning regression with the Gaussian as function, powered by an optimisation algorithm.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Through discussion we found out that the two clearly have different outcomes - though probably similar. For case 2 we optimise the parameters of the Gaussian such, that the sum of the distances from the Gaussian to the occurrences is minimized (along the y-axis if you will). For case 1 we optimise the parameters along the x-axis if you will.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Questions&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;Preliminary question:&lt;/em&gt; Is it correct that the EM-algorithm will have the same result as just calculating mean and standard deviation from the data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Assuming the answer to the preliminary question is &quot;yes, the results are the same&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;What is the intuition interpretation of those two approaches?&lt;/li&gt;&#xA;&lt;li&gt;None of them can be wrong in itself, but one of them could be wrong in the sense of wrong-usage. Meaning: I want to do something and I use the wrong method for it, because of misunderstood interpretation of what happened. So in that sense: Is one of them wrong in a way?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Example Code&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I managed to express myself in R code. As one can see from the plots, the result is definitely not the same for any dataset. Only if the data is Gaussian and large the results get to be similar. But that means little to me ($2^x$ and $exp(x)$ converge both to $infinity$ for $x -&amp;gt; infinity$ and they have little in common otherwise).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(dplyr)&#xA;library(tidyr)&#xA;&#xA;std &amp;lt;- 10&#xA;&#xA;datasets &amp;lt;- list(&#xA;  data.frame( x = round(rnorm(1000, sd = std))),&#xA;  data.frame( x = round(rnorm(1000, sd = std))) %&amp;gt;% filter(x &amp;gt; 0)&#xA;)&#xA;&#xA;for(data1 in datasets){&#xA;&#xA;  data1$densest &amp;lt;- dnorm(data1$x, mean = mean(data1$x), sd = std, log = FALSE)&#xA;  data2 &amp;lt;- data1 %&amp;gt;%&#xA;    group_by(x) %&amp;gt;%&#xA;    summarise(count = n())&#xA;&#xA;  f &amp;lt;- function(x, m, sd, k) {&#xA;    k * exp(-0.5 * ((x - m)/sd)^2) # 1/sqrt(2*pi*sd^2) *&#xA;  }&#xA;&#xA;  cost &amp;lt;- function(par) {&#xA;    rhat &amp;lt;- f(data2$x, par[1], par[2], par[3])&#xA;    sum((data2$count - rhat)^2)&#xA;  }&#xA;&#xA;  o &amp;lt;- optim(c(0, std, 10), cost, method=&quot;BFGS&quot;, control=list(reltol=1e-9))&#xA;  data1$regr &amp;lt;- f(data1$x, o$par[1], o$par[2], o$par[3])&#xA;  data1$regrNormalized &amp;lt;- dnorm(data1$x, mean = o$par[1], sd = abs(o$par[2]), log = FALSE)&#xA;&#xA;  g1 &amp;lt;- ggplot(data=data1, aes(x=x)) +&#xA;    geom_point(data=data2, aes(x=x, y=count), alpha=0.2)+&#xA;    geom_line(aes(y=densest), color=&quot;green&quot;) +&#xA;    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color=&quot;green&quot;) +&#xA;    geom_line(aes(y=regr), color=&quot;blue&quot;) +&#xA;    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color=&quot;blue&quot;)&#xA;&#xA;  g2 &amp;lt;- ggplot(data=data1, aes(x=x)) +&#xA;    geom_line(aes(y=densest), color=&quot;green&quot;) +&#xA;    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color=&quot;green&quot;) +&#xA;    geom_line(aes(y=regrNormalized), color=&quot;blue&quot;) +&#xA;    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color=&quot;blue&quot;)&#xA;&#xA;  plot(g1)&#xA;  plot(g2)&#xA;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Plots&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/P3Lro.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/P3Lro.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LjPwY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LjPwY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/AX7QB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AX7QB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/VHsfq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VHsfq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14909" LastEditorUserId="14909" LastEditDate="2016-12-06T10:22:27.233" LastActivityDate="2016-12-06T16:31:20.313" Title="Fitting Gaussian to data: Density-Estimation vs Regression" Tags="|machine-learning|" AnswerCount="2" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="15452" PostTypeId="2" ParentId="15439" CreationDate="2016-11-30T17:34:48.593" Score="4" Body="&lt;p&gt;Cross validation can be used in parameter tuning or model selection, but it does not evaluate the performance of a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When developing a model, you divide your data between train, validation and testing. In the best case scenario, &lt;strong&gt;testing is only used once&lt;/strong&gt; at the end to score the model. You &lt;strong&gt;should definitely keep the 2016 data&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you give all your data, it is easy to have a model learning &quot;by heart&quot; your expected data but it will not generalize well to future years. This is overfiting. The only way to know is by testing it on unknown data, here, 2016&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Employed for measuring model performance, cross validation can measure more than just the average accuracy and you can &lt;strong&gt;select your features&lt;/strong&gt; to answer the best accuracy score&lt;/p&gt;&#xA;" OwnerUserId="25311" LastEditorUserId="25311" LastEditDate="2016-11-30T17:54:59.003" LastActivityDate="2016-11-30T17:54:59.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15453" PostTypeId="2" ParentId="15439" CreationDate="2016-11-30T19:36:53.267" Score="1" Body="&lt;p&gt;A lot of people are suggesting holding 2016 data, but what you should hold out as a test set depends on what is being predicted. If defective works do not depend on date/time, and the date/time is not being used in the model, it may make sense to hold a random sample (at least w.r.t. date/time) for your tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have features that you want to avoid extrapolating from, then split by those features e.g. if there is some common &quot;location&quot; property for your items, you may want to split by identity of location, because you want to use the model to make predictions in new locations (and the location as a one-hot feature would not have any predictive value for a new location, even if your target class was correlated with existing locations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally the consultancy firm will be helping you here to identify correlations in the data set that you don't want to use in predictions, because they could affect the generalisation you want when using the model in production. If any such thing is identified, then it clearly points to using that feature to split test data by. If there is no such issue, then you may as well split randomly.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-11-30T22:15:28.907" LastActivityDate="2016-11-30T22:15:28.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15454" PostTypeId="2" ParentId="15431" CreationDate="2016-11-30T21:22:11.670" Score="0" Body="&lt;p&gt;You would put triplets into your heap, storing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(distance, i, j)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where distance is the distance, i and j are the indices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you &quot;pop&quot; the topmost (smallest distance) element from the heap using the appropriate algorithm, the second smallest will now become the top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.python.org/2/library/heapq.html&quot; rel=&quot;nofollow&quot;&gt;Python has a module &lt;code&gt;heapq&lt;/code&gt;&lt;/a&gt; for building min-heaps. Simply add tuples with &lt;code&gt;heapq.heappush&lt;/code&gt;, and remove the smallest with &lt;code&gt;heapq.heappop&lt;/code&gt;. To convert an array into a heap, use &lt;code&gt;heapq.heapify&lt;/code&gt; (faster than calling &lt;code&gt;heappush&lt;/code&gt; for every element).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that putting n^2 values into a heap yields runtime O(n^2 log n^2) which is fairly expensive.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2016-12-03T21:58:08.960" LastActivityDate="2016-12-03T21:58:08.960" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="15455" PostTypeId="1" CreationDate="2016-11-30T22:57:09.543" Score="0" ViewCount="716" Body="&lt;p&gt;I’m taking a machine learning course and I’m stuck. I've already tried to ask around in the forums of the class but I don't understand how to implement the formulas they give me. Can you please &lt;strong&gt;help me understand&lt;/strong&gt; how to solve this problem &lt;strong&gt;from the ground up&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;“Calculate the likelihood of this data.”&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;X  | Y&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2.5 | +1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;0.3 | -1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2.8 | +1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;0.5 | +1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;the current estimates of the weights are $w_0 = 0$ and $w_1 = 1$. ($w_0$: the intercept, $w_1$: the weight for $x$).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now Here's what I've already done in case that matters:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I went on to the forums to try to understand I was told this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You can use the sigmoid function to find the likelihood of any given&#xA;  one, and then once you have the probabilities of any given one&#xA;  occurring you should be able to find the probability of the whole&#xA;  thing&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Which is nice; now I know it has something to do with the sigmoid function: $$\frac{1} {1 + e^{-x}}$$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sigmoid(2.5) = 0.9241418&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope I'm on the right track, I really have no way of knowing. So then I calculated the sigmoid for all of them and multiplied them together hoping that would give me the probability of the whole thing because that's how you combine probabilities, right? (The probability of a dice role hitting 2 three times is $1/6 \times 1/6 \times 1/6$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that apparently isn't the right answer, and I didn't take into account the $y$ column or the weights because I don't know how. How sad for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please explain it to me, I feel like I'm a blind man looking for the right colored paper.&lt;/p&gt;&#xA;" OwnerUserId="26671" LastEditorUserId="10156" LastEditDate="2016-12-01T14:42:54.800" LastActivityDate="2016-12-01T14:42:54.800" Title="How do I calculate the maximum likelihood (machine learning statistics) of this table of data?" Tags="|linear-regression|probability|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15457" PostTypeId="1" AcceptedAnswerId="15460" CreationDate="2016-12-01T02:25:07.367" Score="5" ViewCount="4162" Body="&lt;p&gt;I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual decomposition of the matrix A (a user &lt;em&gt;x&lt;/em&gt; movie matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to know more about what goes on after that. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from numpy.linalg import svd&#xA;import numpy as np&#xA;&#xA;A = np.matrix([ &#xA;  [0, 0, 0, 4, 5],&#xA;  [0, 4, 3, 0, 0],&#xA;  ...&#xA;])&#xA;U, S, V = svd(A)&#xA;&#xA;k = 5 #dimension reduction&#xA;A_k = U[:, :k] * np.diag(S[:k]) * V[:k, :]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Three Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do the values of matrix &lt;code&gt;A_k&lt;/code&gt; represent the the predicted/approximate ratings?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What role/ what steps does cosine similarity play in the recommendation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And finally I'm using Mean Absolute Error (MAE) to calculate my error. But what I'm values am I comparing? Something like &lt;code&gt;MAE(A, A_k)&lt;/code&gt; or something else? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26675" LastActivityDate="2017-04-14T12:38:29.067" Title="SVD for recommendation engine" Tags="|python|recommender-system|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15458" PostTypeId="2" ParentId="9302" CreationDate="2016-12-01T02:36:27.567" Score="14" Body="&lt;p&gt;Given &lt;span class=&quot;math-container&quot;&gt;$y_{true}$&lt;/span&gt;, you want to optimize your machine learning method to get the &lt;span class=&quot;math-container&quot;&gt;$y_{predict}$&lt;/span&gt; as close as possible to &lt;span class=&quot;math-container&quot;&gt;$y_{true}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h2&gt;First question:&lt;/h2&gt;&#xA;&lt;p&gt;Above answer has explained the background of your first formula, the cross entropy defined in information theory.&lt;/p&gt;&#xA;&lt;h3&gt;From a opinion other than information theory:&lt;/h3&gt;&#xA;&lt;p&gt;you can examine yourself that first formula does not have penalty on false-positiveness(truth is false but your model predict that it is right), while the second one has penalty on false-positiveness. Therefore, the choice of first formula or second, will affect your metrics(aka what statistic quantity you would like to use to evaluate your model).&lt;/p&gt;&#xA;&lt;h3&gt;In layman word:&lt;/h3&gt;&#xA;&lt;p&gt;If you want to accept almost all good people to be your friend but willing to accept some bad people become your friend, then use first formula for criterion.&lt;/p&gt;&#xA;&lt;p&gt;If you want to punish yourself accepting some bad people to be your friend,but at the same time your good-people accepting rate might be lower than the first condition, then use second formula.&lt;/p&gt;&#xA;&lt;p&gt;While, I guess most of us are critical and would like to choose the second one(so as many ML package assume what is cross entropy).&lt;/p&gt;&#xA;&lt;h2&gt;Second question:&lt;/h2&gt;&#xA;&lt;p&gt;Cross entropy per sample per class: &lt;span class=&quot;math-container&quot;&gt;$$-y_{true}\log{(y_{predict})}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Cross entropy for whole datasets whole classes: &lt;span class=&quot;math-container&quot;&gt;$$\sum_i^n \sum_k^K -y_{true}^{(k)}\log{(y_{predict}^{(k)})}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h3&gt;Thus, when there are only two classes (K = 2), you will have the second formula.&lt;/h3&gt;&#xA;" OwnerUserId="25837" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-12-01T02:53:21.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15459" PostTypeId="1" CreationDate="2016-12-01T02:44:46.880" Score="0" ViewCount="1077" Body="&lt;p&gt;I'm new to sklearn and having trouble formatting the data to predict and evaluate a confusion matrix.  I'm using this &lt;a href=&quot;http://blog.yhat.com/posts/random-forests-in-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn.ensemble import RandomForestClassifier&#xA; import numpy as np&#xA; import pandas as pd&#xA; dataframe = pd.read_csv('output.txt', sep='\t')&#xA; df = pd.DataFrame(dataframe)&#xA; df['is_train'] = np.random.uniform(0, 1, len(df)) &amp;lt;= .75&#xA;&#xA; train, test = df[df['is_train']==True], df[df['is_train']==False]&#xA; features = df.columns[1:5]&#xA; clf = RandomForestClassifier(n_jobs=2)&#xA; y, _ = pd.factorize(train['event_count'])&#xA; clf.fit(train[features], y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It's this line for my predictions which is giving me the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; preds = df[:6][clf.predict(test[features])]&#xA; IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26676" LastEditorUserId="8501" LastEditDate="2016-12-03T17:06:15.950" LastActivityDate="2016-12-03T17:06:15.950" Title="Syntax error in pandas dataframe slice in my sklearn data prep?" Tags="|scikit-learn|random-forest|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15460" PostTypeId="2" ParentId="15457" CreationDate="2016-12-01T02:54:25.823" Score="3" Body="&lt;p&gt;You can use SVD to build a recommendation engine, but I don't think it's the best way to get intuition around what's going on under the hood. &lt;a href=&quot;http://www.columbia.edu/~jwp2128/Teaching/W4721/Spring2017/slides/lecture_3-30-17.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Regardless, here's a presentation with more details, I'd recommend reviewing slide 9&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And to answer your questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;A_k&lt;/code&gt; represents an embedding dimension (i.e. the low-rank approximation) that is used to predict the user-rating matrix.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The cosine similarity is just the dot product for user $i$ and item $j$, which maps to the predicted rating for user $i$ and item $j$. The dot product is what defines the users and items as being similar. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, you should use the MAE on &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;A_k&lt;/code&gt;. You may prefer to use MSE instead. This measures the quality of your predictions for user $i$ and item $j$. Note, this is obviously the MSE of a &lt;strong&gt;&lt;em&gt;matrix&lt;/em&gt;&lt;/strong&gt;, which is the Frobenius Norm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think an easier way to understand SVD is to see it applied to image compression for different components. &lt;a href=&quot;http://math.arizona.edu/~brio/VIGRE/ThursdayTalk.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;See this presentation here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="31095" LastEditDate="2017-04-14T12:38:29.067" LastActivityDate="2017-04-14T12:38:29.067" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15461" PostTypeId="2" ParentId="15455" CreationDate="2016-12-01T02:54:32.710" Score="2" Body="&lt;p&gt;You might have mistaken the equation for maximum likelihood , and it should be like this :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$p^y*(1-p)^{1-y}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;while y is your label , it should be 0 or 1 , 0 for negative and 1 for positive , and p is your sigmoid function value.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so , in your case it should be :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;p(2.5)*(1 - p(0.3))*p(2.8)*p(0.5)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hopes this will help you , good luck !&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-12-01T02:54:32.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15462" PostTypeId="2" ParentId="15455" CreationDate="2016-12-01T06:49:01.667" Score="1" Body="&lt;p&gt;Okay, so here's how it works,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$y$, which is the response variable consist of two binary classes, $0$ and $1$. i.e. $y \in \{0, 1\}$&lt;/li&gt;&#xA;&lt;li&gt;Given a threshold value, (we'll use $0.5$ for illustrative purposes), use it to determine the class of $y$ from the logistic or sigmoid classifier output (i.e. $h_\theta(x)$ ). $$\text{If }\text{ } h_\theta(x) \geq 0.5 \text{ &amp;lt;}threshold\text{&amp;gt;}, \text{ predict } &quot;y = 1&quot;$$&#xA;$$\text{If }\text{ } h_\theta(x) &amp;lt; 0.5 \text{ &amp;lt;}threshold\text{&amp;gt;}, \text{ predict } &quot;y = 0&quot;$$&lt;/li&gt;&#xA;&lt;li&gt;You will compute a logistic function (also called a sigmoid function) to get the estimated probability that $y = 1$ on input $x$ given by this formula,&#xA;$$h_\theta(x) =\frac{1}{1 + e^{-\theta^{T}x}}$$&#xA;where, $\theta$ are the weights, also known as the parameters of the model. In this case, $w_0$ and $w_1$&lt;/li&gt;&#xA;&lt;li&gt;The weight $w_0$ is for the included bias term of $1$ for all row entries in the dataset.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For each row in the dataset, repeat #5 to #7&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;Compute the value of $\theta^Tx$ as follows,&#xA;$$\theta^Tx = w_0x_0 + w_1x_1$$&#xA;For example, $\theta^Tx$ for the first row will be,&#xA;$$\theta^Tx = (0 \times 1) + (1 \times 2.5)$$&lt;/li&gt;&#xA;&lt;li&gt;Substitute that value into the sigmoid function $h_\theta(x)$, i.e. &lt;code&gt;sigmoid(x)&lt;/code&gt; already defined in #3 above. (P.s. remember the minus in that computation)&lt;/li&gt;&#xA;&lt;li&gt;Test your output, which is a probabilistic value between $0$ and $1$ (i.e. $0 \leq h_\theta(x) \leq 1$) against the threshold value given in the problem. This assigns the output of &lt;code&gt;sigmoid(x)&lt;/code&gt; to classes as shown in #2 above.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I hope this helps :)&lt;/p&gt;&#xA;" OwnerUserId="10156" LastEditorUserId="10156" LastEditDate="2016-12-01T06:56:02.373" LastActivityDate="2016-12-01T06:56:02.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15463" PostTypeId="1" CreationDate="2016-12-01T09:04:02.687" Score="2" ViewCount="2266" Body="&lt;p&gt;I could find max-pooling in most -- actually, exactly all -- concrete CNN implementations I've seen. However, I see that there are several other kinds (min, average, global average pooling, etc.) but I cannot see what they differ on and what would be the best criteria to select them at application. Has anyone a clue?&lt;/p&gt;&#xA;&lt;p&gt;I have learned that pooling is a sort of downsampling where we can preserve the locations of the original image that showed the strongest correlation to the specific features we searching for. So far, so reasonable. But what is the point of finding the weakest correlations with min pooling, for example?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="113681" LastEditDate="2021-03-22T19:35:29.343" LastActivityDate="2021-03-22T19:35:29.343" Title="What are the practical differences between pooling types at convolution?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15464" PostTypeId="2" ParentId="15459" CreationDate="2016-12-01T09:11:41.283" Score="1" Body="&lt;p&gt;It is not possible to write numpy style slices in dataframe, like : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;observations = data[:6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Try DataFrame.ix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.ix[:,6:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Cf panda indexing : &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-advanced&quot; rel=&quot;nofollow noreferrer&quot;&gt;Panda indexing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-12-01T09:11:41.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15465" PostTypeId="2" ParentId="14998" CreationDate="2016-12-01T09:50:23.340" Score="2" Body="&lt;p&gt;Airbnb recently open sourced their internal data science knowledge repository: &lt;a href=&quot;https://github.com/airbnb/knowledge-repo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/airbnb/knowledge-repo&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From its readme, it seems it could loosely fit your use case:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The Knowledge Repository project is focused on facilitating the&#xA;  sharing of knowledge between data scientists and other technical roles&#xA;  using data formats and tools that make sense in these professions. It&#xA;  provides various data stores (and utilities to manage them) for&#xA;  &quot;knowledge posts&quot;, with a particular focus on notebooks (R Markdown&#xA;  and Jupyter / iPython Notebook) to better promote reproducible&#xA;  research.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There's also a &lt;a href=&quot;https://medium.com/airbnb-engineering/scaling-knowledge-at-airbnb-875d73eff091#.x9argsxas&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; commenting on its motivation.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-12-01T09:50:23.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15466" PostTypeId="2" ParentId="15069" CreationDate="2016-12-01T10:23:51.250" Score="0" Body="&lt;p&gt;10/75 class A is mispredicted as B, 11/60 class B is mispredicted as A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contrast this to The misclassification seen in C &amp;amp; D:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;9 / 70 class C is mispredicted as D&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And only 3 / 90 class D is mispredicted as C. There is actually more cases of class D that is mispredicted as A than C. Hence we cannot argue that D is confused with class C.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-01T10:23:51.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15470" PostTypeId="2" ParentId="15463" CreationDate="2016-12-01T13:52:45.733" Score="4" Body="&lt;p&gt;See &lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf#page=21&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analysis and Optimization of Convolutional Neural Network Architectures&lt;/a&gt; for the pooling types in general.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the difference between max pooling, average pooling and min pooling?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A pooling function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ is a function which is very symmetric:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x_1, x_2, \dots, x_n) = f(x_2, x_1, \dots, x_n) = f(x_n, x_1, \dots, x_2) = \dots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the function does not care about the order of the arguments. This is the case for $\min$, $\max$ and average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of CNNs, you apply the pooling function to each feature map independantly. The output of a pooling layer has exactly as many feature maps as the input. For each feature map, you take a $p \times p$ section and apply the pooling function $f$ to it. Usually, you apply it with a stride $s &amp;gt; 1$ so that the output is much smaller than the input (only $\frac{1}{s^2}$).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have learnt that pooling is a sort of downsampling where we can preserve the locations of the original image that showed the strongest correlation to the specific features we searching for. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is only true if you talk about max-pooling with $s &amp;gt; 1$. Which is usually the case. Although I would not say &quot;correlation&quot; but &quot;activation&quot;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But what is the point of find weakest correlations with min pooling, for example?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't know. I have never seen min pooling even be mentioned. Where have you seen it?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-08-01T05:47:08.643" LastActivityDate="2017-08-01T05:47:08.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15472" PostTypeId="1" CreationDate="2016-12-01T15:42:09.270" Score="4" ViewCount="5316" Body="&lt;p&gt;I have a data set of 36K processes each with 7 features. Each set represents a task and I know how long it took to complete each of the tasks. &lt;strong&gt;I want to build a model which will be able to predict the duration of the future tasks.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've only learned about Decision Tree (DT) and tried to apply it to my problem. The resulting accuracy score is 0.03. I believe DT is not suitable because the time is continuous and DT is meant for categorization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which algorithm is suitable for duration prediction?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My environment: Python with sklearn, if that matters.&lt;/p&gt;&#xA;" OwnerUserId="26659" LastActivityDate="2017-03-05T12:32:59.077" Title="Which algorithm to use to predict the duration of some task" Tags="|machine-learning|algorithms|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15473" PostTypeId="2" ParentId="15472" CreationDate="2016-12-01T16:03:40.897" Score="-1" Body="&lt;p&gt;You might want to look into scikit's &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html#tree-regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;DecisionTreeRegressor() class&lt;/a&gt;. A decision tree regressor will predict a real number. The decision tree classifier, by contrast, will predict a discrete class for an observation. A more advanced version would be the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomForestRegressor() class&lt;/a&gt;, which builds a &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest&lt;/a&gt; of regression trees. A third option to consider might be a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor&quot; rel=&quot;nofollow noreferrer&quot;&gt;GradientBoostedRegressor()&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also recommend &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this flow chart&lt;/a&gt; from Scikit that can help you choose an estimator.&lt;/p&gt;&#xA;" OwnerUserId="16548" LastActivityDate="2016-12-01T16:03:40.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15475" PostTypeId="1" AcceptedAnswerId="15476" CreationDate="2016-12-01T19:41:30.380" Score="3" ViewCount="2780" Body="&lt;p&gt;I am experimenting with xgboost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator on the same data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;roc_auc_score(Y, clf_best_xgb.predict(X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it gives me score ~0.878&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you tell me how the score is evaluated in both cases? Thanks&lt;/p&gt;&#xA;" OwnerUserId="26701" LastActivityDate="2016-12-01T20:55:41.517" Title="roc_auc score GridSearch" Tags="|scikit-learn|xgboost|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15476" PostTypeId="2" ParentId="15475" CreationDate="2016-12-01T20:34:45.433" Score="2" Body="&lt;p&gt;Try using &lt;strong&gt;predict_proba&lt;/strong&gt; instead of &lt;strong&gt;predict&lt;/strong&gt; as below. It should give you the same number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;roc_auc_score(Y, clf_best_xgb.&lt;strong&gt;predict_proba&lt;/strong&gt;(X)[:,1]) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we compute AUC, most of time people will use the probability instead of the actual classs.&lt;/p&gt;&#xA;" OwnerUserId="6567" LastEditorUserId="6567" LastEditDate="2016-12-01T20:55:41.517" LastActivityDate="2016-12-01T20:55:41.517" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15477" PostTypeId="1" CreationDate="2016-12-02T06:14:31.097" Score="0" ViewCount="767" Body="&lt;p&gt;I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have worked on vector space models (tfidf-cosine similarirty ) and LSA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried Wordnet, NER, fuzzy matching etc for improving the accuracy.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to know how to improve accuracy of IR tasks by applying nueral networks, word embeddingstopic models etc by capturing more context/sematics information&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2018-03-01T12:54:33.993" Title="State of the art approaches for Information retrieval tasks based on deep learning" Tags="|python|neural-network|nlp|information-retrieval|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15479" PostTypeId="2" ParentId="15472" CreationDate="2016-12-02T08:56:02.037" Score="-1" Body="&lt;p&gt;You are facing a &lt;strong&gt;regression&lt;/strong&gt; problem: your aim is to predict the value of a continuous variable given the value of a set of input variables (these input variables can be of any type: numbers, categories, etc.) A decision tree is usually applied to classification problems, in which you are aiming at predicting a discrete value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several regression methods in sklearn that you could use. The simplest ones, and maybe the ones you should start from, are linear models: &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/linear_model.html&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that you may need to transform your input data. For instance, if one of the features is categorical, you may need to transform it into a set of &lt;a href=&quot;https://en.wikipedia.org/wiki/Dummy_variable_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary variables&lt;/a&gt;. Other processes, like data normalisation, may also be advisable in order to get better results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: as stated in the comments, Decision Trees can also be applied to regression problems. However, and in my own experience, the output curve that you obtain by means of this algorithm usually has a step-wise shape that may affect the final bias (see, for instance &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the example in the scikit-learn docs.&lt;/a&gt;). I would suggest not to constraint yourself and try different types of algorithms. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="2576" LastEditDate="2016-12-05T11:12:59.600" LastActivityDate="2016-12-05T11:12:59.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15482" PostTypeId="2" ParentId="15451" CreationDate="2016-12-02T16:00:46.687" Score="0" Body="&lt;p&gt;Assuming that your data is really Gaussian, the best way to fit to a Gaussian is simply to calculate the mean and variance of your data (sample mean and sample variance) and to use those parameters Gaussian fit. I.e. fit the moments. No need for any complex E-M estimation or supervised learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only possible weakness of fitting using estimated moments is that if your sample is small, you might be overly sensitive to outlier points, so you have to be careful about that. &lt;/p&gt;&#xA;" OwnerUserId="23416" LastActivityDate="2016-12-02T16:00:46.687" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15483" PostTypeId="2" ParentId="15444" CreationDate="2016-12-02T16:47:47.510" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Doesn't this dependency lead to the exploding/vanishing gradients problem again?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Absolutely&lt;/em&gt;&lt;/strong&gt;, and you better have vanishing gradients otherwise you have a training problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vanishing gradient in this case is not a bad thing, it is a good thing (unlike in feedforward). Let $C(t)$ be the cost function evaluated at time $t$ and $ W(t)$ be some weight of the network at time $t$. What vanishing gradient in this case means is that $ dC(t)/dW(t-u)$ becomes smaller and smaller as u becomes bigger and bigger. That is good because $ dC(t)/dW =  \sum_{u=0}^{num\_steps} dC(t)/dW(t-u) $, so if the gradients didn't vanish in time, then the gradients would explode. So $W$ gets a proper non-vanishing gradient even if the gradients in time vanish because the gradient for $W$ is the sum at all times of the gradient for $W$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In LSTMs the gradients are sure to vanish in time because the activation functions are sigmoids and tanh's so their derivatives are less than or equal to one, so as they get multiplied they slowly become smaller. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This compares to what is normally called the vanishing gradient problem which occurs when gradients vanish while passing from top layers to bottom layers, because that means that $ dC/dW $ for $W $ of the lower layer is vanishing and so the lower layers don't get trained, only the upper layers get trained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as mentioned in the comments, the above applies to any RNN, not only LSTMs. What sets LSTMs appart from vanilla RNNs in with regards to this question is the gating functions which allows the LSTM to control what it remembers and what it forgets and how much of the new input it takes in. While the above is true in practice for LSTMs (and is true on average also in theory), in theory, one could have a time step $t$ where the output has ignored the last 10 inputs and only depends on the input 11 timestep back ($t-11$), in which case the gradient for the weights 11 timesteps ago will not have decayed. Of course that means that at the next time step ($t+1$) the gradients for 11 steps ago ($t+1 -11 = t-10$) will be zero because the input was totally disregarded at $t-10$. So on average it averages out and you still have the same situation for LSTMs.&lt;/p&gt;&#xA;" OwnerUserId="23416" LastEditorUserId="23416" LastEditDate="2016-12-19T14:41:19.007" LastActivityDate="2016-12-19T14:41:19.007" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15484" PostTypeId="1" AcceptedAnswerId="15488" CreationDate="2016-12-02T18:33:45.830" Score="8" ViewCount="5758" Body="&lt;p&gt;The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.&lt;/p&gt;&#xA;" OwnerUserId="26477" LastEditorUserId="29169" LastEditDate="2021-02-16T15:28:43.967" LastActivityDate="2021-02-16T15:28:43.967" Title="Sigmoid vs Relu function in Convnets" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15485" PostTypeId="1" AcceptedAnswerId="16679" CreationDate="2016-12-02T19:41:52.823" Score="2" ViewCount="837" Body="&lt;p&gt;My girlfriend has recently been struggling with finding a new job, so I thought I'd make a website to help her out. The basic idea is that she'll be shown a list of jobs, rate her interest, and then a hopefully interesting job will be emailed to her every day. So far I have the frontend worked out, but I'm struggling with the engine. I've looked into different kinds of recommendation engines, and it seems like a content based engine is the way to go since she will probably be the only user.&#xA;With that said, feature selection is problematic because job categorizations kind of unrealistic. For instance, I could have job categories like &quot;administrative&quot; or &quot;analytical&quot;, but those would be hard to define, especially if I was analyzing the job description for certain keywords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry about the wall of text. My basic question is how should I go about selecting features for this job recommendation engine?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help.&lt;/p&gt;&#xA;" OwnerUserId="26719" LastActivityDate="2017-04-02T15:54:54.303" Title="Job Recommendation Engine" Tags="|machine-learning|feature-selection|recommender-system|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15486" PostTypeId="2" ParentId="14998" CreationDate="2016-12-02T20:05:34.507" Score="3" Body="&lt;p&gt;JupyterHub does not provide version control system nor facilitates sharing of Notebooks. You mentioned yourself limitation of Binder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try &lt;a href=&quot;https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zeppelin&lt;/a&gt;. Version 0.7 should be released within a few next days. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;As you can see from the &lt;a href=&quot;https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap&quot; rel=&quot;nofollow noreferrer&quot;&gt;roadmap&lt;/a&gt;, this version delivers &quot;enterprise&quot; features which are exactly about collaboration.&lt;/li&gt;&#xA;&lt;li&gt;Version control system (git) is integrated.&lt;/li&gt;&#xA;&lt;li&gt;It's self-hosted.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In essence, I think it meets all requirements you posted. On top of that it delivers richer visualisation capabilities and plethora of other features (works with Shiro, Knox, Kerberos - secure Spark anyone?). &lt;/p&gt;&#xA;" OwnerUserId="26523" LastActivityDate="2016-12-02T20:05:34.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15488" PostTypeId="2" ParentId="15484" CreationDate="2016-12-02T23:07:01.690" Score="14" Body="&lt;p&gt;The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between 0 and 1. The maximum of the $\sigma'$ derivative function is equal to $\frac{1}{4}$. Therefore when we have multiple stacked sigmoid layers, by the backprop derivative rules we get multiple multiplications of $\sigma'$. And as we stack more and more layers the maximum gradient decreases exponentially. This is commonly known as the vanishing gradient problem. The opposite problem is when the gradient is greater than 1, in which case the gradients explode toward infinity (exploding gradient problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's check out the ReLU activation function which is defined as: $$R(x) = max(0,x)$$ The graph of which looks like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WSX7a.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WSX7a.jpg&quot; alt=&quot;RELU&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the derivatives of the function (slopes on the graph), the gradient is either 1 or 0. In this case we do not have the vanishing gradient problem or the exploding problem. And since the general trend in neural networks has been deeper and deeper architectures ReLU became the choice of activation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2016-12-02T23:07:01.690" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="15489" PostTypeId="2" ParentId="15485" CreationDate="2016-12-02T23:25:25.163" Score="1" Body="&lt;p&gt;The basic ways to make a recommendation engine are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Link a profile to similar ones&lt;/li&gt;&#xA;&lt;li&gt;Extract characteristics of a profile, and match them with characteristics of, here, jobs&lt;/li&gt;&#xA;&lt;li&gt;A mix of the two solutions above&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Either way, you need data, labels, characteristics extractors...&lt;/p&gt;&#xA;" OwnerUserId="25311" LastActivityDate="2016-12-02T23:25:25.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15491" PostTypeId="2" ParentId="14342" CreationDate="2016-12-03T09:35:02.247" Score="3" Body="&lt;p&gt;NB This advice assumes your goal is to recognise expression in pictures of any person, and not just people from your training data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should the validation set consists of unseen subjects as well? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. This will give you the most accurate measure of performance in the task you want to use the network in, in order to choose the best generalisation and take it forward to testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would only use a simple random split if the end goal of your trained network is to recognise expressions from images of the people in the training set.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No. If you take random samples where the same face appears in training and cv, you will get an over-estimate of generalisation. I have seen this effect first hand in the Kaggle &lt;a href=&quot;https://www.kaggle.com/c/state-farm-distracted-driver-detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;State Farm Distracted Driver&lt;/a&gt; contest, and you should see it discussed in the forums there. It might be helpful for ideas to improve performance too.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a &lt;em&gt;data leak&lt;/em&gt; between train and cross validation - the network has learnt to correctly classify expressions in images of people it has already seen, and that is what you are measuring by taking a split in this way. It is not surprising that the test result does not match the promising values from cross validation.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are over-fitting regardless of how you split train,CV. When you split the CV incorrectly, then in addition to over-fitting, the data leak gives you bad guidance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like that you have very little diversity in the training data, whilst training an image classifier from scratch requires a &lt;em&gt;lot&lt;/em&gt; of data. Consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Collecting more labelled data, perhaps some other dataset you can download.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Adding a lot of regularisation to your model (e.g. multiple dropout layers)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using a pre-trained image classifier network (e.g. VGG-19 or Inception) as a starting point and fine-tuning it for your classification task.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use full k-fold cross-validation regardless of the computational cost, to mitigate problems of using small training set. This won't help solve your training problems directly, but will give you a better shot at tuning your network hyper-parameters once you solve that issue.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-12-03T09:46:00.690" LastActivityDate="2016-12-03T09:46:00.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15495" PostTypeId="2" ParentId="11928" CreationDate="2016-12-04T02:46:18.560" Score="10" Body="&lt;p&gt;For anybody happening across this, to actually modify the original:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test.fillna(X_train.mean(), inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To overwrite the original:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test = X_test.fillna(X_train.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To check if you're in a copy vs a view:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test._is_view&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26751" LastEditorUserId="26751" LastEditDate="2018-06-18T20:08:30.013" LastActivityDate="2018-06-18T20:08:30.013" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15496" PostTypeId="2" ParentId="13782" CreationDate="2016-12-04T03:02:33.747" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/users/9123/student-t&quot;&gt;@Student T&lt;/a&gt;: Pandas can have memory issues with only a few thousand rows, depends on width and operation (&lt;code&gt;merge()&lt;/code&gt; can be bad).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do have issues I've found moving to a database the least painful, just dump your dataframes to tables with &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;DataFrame.to_sql()&lt;/code&gt;&lt;/a&gt; and pull it back into your scikit-learn pipeline with &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pandas.read_sql_query()&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26751" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-04T03:02:33.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15497" PostTypeId="2" ParentId="10077" CreationDate="2016-12-04T05:02:36.913" Score="16" Body="&lt;p&gt;The &lt;a href=&quot;https://catalog.ldc.upenn.edu/LDC2012T21&quot; rel=&quot;noreferrer&quot;&gt;Google Research Blog&lt;/a&gt; should be helpful in the context of &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above article, there is a reference to the &lt;a href=&quot;https://catalog.ldc.upenn.edu/LDC2012T21&quot; rel=&quot;noreferrer&quot;&gt;Annotated English Gigaword dataset&lt;/a&gt; which is routinely used for text summarization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 2014 paper by &lt;a href=&quot;https://arxiv.org/abs/1409.3215&quot; rel=&quot;noreferrer&quot;&gt;Sutskever et al&lt;/a&gt; titled &lt;em&gt;Sequence to Sequence Learning with Neural Networks&lt;/em&gt; could be a meaningful start on your journey as it turns out that for shorter texts, summarization can be learned end-to-end with a deep learning technique. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/research/textsum&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; is a great Github repository demonstrating text summarization while making use of TensorFlow.&lt;/p&gt;&#xA;" OwnerUserId="26596" LastEditorUserId="42774" LastEditDate="2017-12-03T14:04:29.543" LastActivityDate="2017-12-03T14:04:29.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15498" PostTypeId="1" AcceptedAnswerId="15501" CreationDate="2016-12-04T05:30:10.880" Score="0" ViewCount="2306" Body="&lt;p&gt;Suppose I've this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Employee_ID  Store_ID  Company_ID  Stock_ID  Product_Value&#xA;    1            1          1          2           3.7&#xA;    4            1          4          2            8&#xA;                           ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Employee_ID: Is the unique number for a employee that sold the product&#xA;Store_ID: Unique number for Store Chain&#xA;Company_ID: Unique number for product supplier&#xA;Stock_ID: Unique number for the product purchased&#xA;Product_Value: Product Value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to make a segmentation over my dataset using K-Means... Its make sense use all the variables for my dataset? &lt;/p&gt;&#xA;" OwnerUserId="23811" LastEditorUserId="23811" LastEditDate="2016-12-04T10:04:58.377" LastActivityDate="2016-12-05T07:15:02.577" Title="K-Means Algorithm - Feature Selection" Tags="|feature-selection|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15499" PostTypeId="1" AcceptedAnswerId="15504" CreationDate="2016-12-04T06:31:41.067" Score="0" ViewCount="2994" Body="&lt;p&gt;I need to build a recommendation system that takes certain parameters as input, computes a score and order suggestions to users based on this score. Well this is what I need to do loosely speaking. I am new to the scene of data science and haven't come across anything that could help me out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This guy asked a similar question 5 years ago. I basically need something like this:&#xA;&lt;a href=&quot;https://stackoverflow.com/questions/6828013/recommendation-systems-for-a-mobile-market-and-algorithm-suggestion&quot;&gt;https://stackoverflow.com/questions/6828013/recommendation-systems-for-a-mobile-market-and-algorithm-suggestion&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies for the open-ended and vague question.&#xA;Just need to be pointed out in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I do not intend to place the system on the android device. It will be running back-end&lt;/p&gt;&#xA;" OwnerUserId="25004" LastEditorUserId="25004" LastEditDate="2016-12-05T03:44:32.387" LastActivityDate="2016-12-05T12:51:44.577" Title="Recommendation System to integrate with an android app" Tags="|machine-learning|neural-network|recommender-system|" AnswerCount="1" CommentCount="0" ClosedDate="2016-12-05T19:58:29.647" ContentLicense="CC BY-SA 3.0" />
  <row Id="15501" PostTypeId="2" ParentId="15498" CreationDate="2016-12-04T11:02:29.767" Score="2" Body="&lt;p&gt;So long as you want to use those variables to define closeness then yes, you just have to encode them differently. You should use a one-hot-encoding for the discrete variables (e.g., Employee_ID, Store_ID, etc.) and just the Product_Value as is. I answered a similar question about K-means for just categorical variables (like Employee_ID) so I've copied the code below that gives a quick demo for using the Clusters as a feature for predicting something after you use K-means. As I said in my old answer, in general, this framework isn't optimal but it's okay for a simulation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(glmnet)&#xA;library(Matrix)&#xA;n &amp;lt;- 1e5&#xA;nclusters &amp;lt;- 5&#xA;set.seed(420)&#xA;ls &amp;lt;- data.frame(sample(letters, n, replace=TRUE))&#xA;xs &amp;lt;- sparse.model.matrix(~.,data=ls)&#xA;print(head(xs))  &#xA;# Now let's run k-means&#xA;out &amp;lt;- kmeans(xs, centers=nclusters)&#xA;bs &amp;lt;- rep(1, dim(xs)[2])&#xA;# Let's run k-means on the different categories&#xA;clusterpred &amp;lt;- data.frame(out[[1]])&#xA;ys &amp;lt;- xs %*% bs + rnorm(n)&#xA;print(table(clusterpred))&#xA;# Now let's use a clustered data set to predict some outcome&#xA;cxs &amp;lt;- sparse.model.matrix(~.,data=clusterpred)&#xA;model &amp;lt;- glmnet(y=ys, x=xs, alpha=0)&#xA;cmodel &amp;lt;- glmnet(y=ys, x=cxs, alpha=0)&#xA;&#xA;# Predictions&#xA;yhat &amp;lt;- predict(model, xs)&#xA;yhatc &amp;lt;- predict(cmodel, cxs)&#xA;# Looking at the difference RMSEs &#xA;print(sqrt( sum( (ys-yhat)**2 )))&#xA;print(sqrt( sum( (ys-yhatc)**2 )))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-12-04T11:02:29.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15502" PostTypeId="2" ParentId="15472" CreationDate="2016-12-04T11:12:42.513" Score="4" Body="&lt;p&gt;What others have said is accurate, that you need to build a regression model of some sort. Depending on the scale of the duration of your task, you will have to model it slightly differently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, there's an entire class of models that try to predict duration. These are called survival models. &lt;a href=&quot;https://github.com/sashaostr/sklearn-lifelines&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a Python library on survival analysis.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But these models are fairly academic. A typical hack is to use Gamma, Poison, or Log-Normal regression which works out nicely because these models predict non-negative values. &lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-12-04T11:12:42.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15503" PostTypeId="1" AcceptedAnswerId="15508" CreationDate="2016-12-04T12:06:23.483" Score="0" ViewCount="455" Body="&lt;p&gt;I was working on Binning by Mean, Median and Boundary in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# R CODE&#xA;a=c(20.5, 52.5, 62.6, 72.4, 104.8, 63.9, 35.3, 83.9, 37.4, 71.6, 74.6, 44.5, 66.6, 56.1, 45.3, 37.2)&#xA;a=sort(a)&#xA;binsize=4&#xA;median(a[1:4])&#xA;# BINS ARE&#xA;for(i in 1:length(a))&#xA;{&#xA;  if(i%%binsize==0)&#xA;  {&#xA;    print(&quot;HI&quot;)&#xA;    print(a[i-3])&#xA;    print(a[i-2])&#xA;    print(a[i-1])&#xA;    print(a[i])&#xA;  }&#xA;}&#xA;&#xA;# BINNING BY MEAN&#xA;sum=0&#xA;for(i in 1:length(a))&#xA;{&#xA;  sum=sum+a[i]&#xA;    if(i%%binsize==0){&#xA;      avg=sum/binsize&#xA;      sum=0&#xA;      print(rep.int(avg,binsize))&#xA;    }&#xA;}&#xA;&#xA;# BINNING BY MEDIAN&#xA;i=1&#xA;for(i in 1:length(a)){&#xA;  if(i%%binsize==0)&#xA;  {&#xA;    print(rep.int(median(a[i-3:i]),binsize))&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me why binning by median is giving me wrong output.&#xA;The &lt;code&gt;median(a[i-3:i])&lt;/code&gt; for 1st bin returns a value which is not same as &lt;code&gt;median(a[1:4])&lt;/code&gt; for 1st bin. Why?&lt;/p&gt;&#xA;" OwnerUserId="26535" LastEditorUserId="8501" LastEditDate="2016-12-05T19:56:52.127" LastActivityDate="2016-12-05T19:56:52.127" Title="Why is this Binning by Median code wrong?" Tags="|r|data-mining|dataset|statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15504" PostTypeId="2" ParentId="15499" CreationDate="2016-12-04T13:38:02.940" Score="1" Body="&lt;p&gt;I don't want to sound harsh but building a recommender system on a android isn't really &lt;em&gt;Data science&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any recommender system context, the science on data is usually done in ad-hoc manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the algorithm implementation, validation and scaling, data science can play a part in that. But this is still not related to the fact of serving the data in a mobile application nor a website. &lt;strong&gt;&lt;em&gt;That's mainly computer engineering.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recommender systems algorithms a quite expensive to compute on any mobile nevertheless. It's best fit to be done by a dedicated server which can also play a role in serving the information when needed. I can't get into the details of this thought as it's quite broad to fit in a single answer on the site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I advice you to read my answer on the &lt;a href=&quot;https://datascience.stackexchange.com/questions/13279/solution-for-in-time-space-complexity-challenge-in-recommendation-system/13488#13488&quot;&gt;time/space complexity challenge in building this types of application&lt;/a&gt;. That may help you getting some insights about the architectural design of a recommender engine in the real world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that this answers your question.&lt;/p&gt;&#xA;" OwnerUserId="5177" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-05T12:51:44.577" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15508" PostTypeId="2" ParentId="15503" CreationDate="2016-12-04T18:20:41.263" Score="1" Body="&lt;p&gt;The error is due to the &lt;strong&gt;well-known R gotcha that the &lt;code&gt;:&lt;/code&gt; (colon operator, which calls &lt;code&gt;seq()&lt;/code&gt;) takes higher precedence than arithmetic&lt;/strong&gt;. Always parenthesize arguments to &lt;code&gt;:&lt;/code&gt; if they involve arithmetic or are expressions: &lt;code&gt;a[(i-3):i]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your code &lt;code&gt;a[i-3:i]&lt;/code&gt; doesn't do what you want it to do &lt;code&gt;a[(i-3):i]&lt;/code&gt;, it does &lt;code&gt;a[i - (3:i)])&lt;/code&gt;. So the medians you are printing are for these slices:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4-3:1 # i.e. 1:3&#xA;8-3:1 # i.e. 5:7&#xA;12-3:1 # i.e. 9:11&#xA;16-4:1 # i.e. 13:15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PS some coding-style tips&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You don't need to iterate over all possible values of i and check them modulo binsize, just do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;for(i in seq(binsize, length(a), by=binsize)) { ... }&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;so in your median case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in seq(binsize, length(a), by=binsize)) {&#xA;  print(rep.int(median(a[(i-3):i]),binsize))&#xA;}&#xA;[1] 36.25 36.25 36.25 36.25&#xA;[1] 48.9 48.9 48.9 48.9&#xA;[1] 65.25 65.25 65.25 65.25&#xA;[1] 79.25 79.25 79.25 79.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;But in fact you can replace even that with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;split(a, ceiling(seq_along(a)/binsize))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as per the &lt;a href=&quot;https://stackoverflow.com/questions/3318333/split-a-vector-into-chunks-in-r&quot;&gt;&quot;Split a vector into chunks in R&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it even clearer, you could define a helper function &lt;code&gt;chunk &amp;lt;- function(x, binsize) { split(x, ceiling(seq_along(x)/binsize)) }&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then you can replace the for-loop with sapply:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sapply(split(a, ceiling(seq_along(a)/binsize)), mean)&#xA;sapply(chunk(a,binsize), mean)&#xA;     1      2      3      4 &#xA;32.600 49.600 66.175 83.925&#xA;&#xA;sapply(split(a, ceiling(seq_along(a)/binsize)), median)&#xA;sapply(chunk(a,binsize), median)&#xA;    1     2     3     4 &#xA;36.25 48.90 65.25 79.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Much cleaner, easier to read, and prevents errors, right?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The generalization of the colon operator is the &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/base/html/seq.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;seq()&lt;/code&gt; function&lt;/a&gt;, give it a read, it's pretty useful.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-05T19:22:36.860" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15509" PostTypeId="1" AcceptedAnswerId="48689" CreationDate="2016-12-04T18:24:42.620" Score="6" ViewCount="6395" Body="&lt;p&gt;I try to write a program to crop background from an image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a sample of my training data.&#xA;I have images with and without a background. (manually cropped)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The background is always similar (changes of light and so on. Sometimes people on random places.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The object is always on the same place but not always the same (there are different models).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can someone give me an advice how I can solve this Problem?&#xA;What is the right way to do something like this???&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CoiS1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CoiS1.png&quot; alt=&quot;lern data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NszYZ.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NszYZ.jpg&quot; alt=&quot;Image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26408" LastEditorUserId="29575" LastEditDate="2017-08-30T23:02:07.777" LastActivityDate="2019-04-05T10:45:24.910" Title="Crop background from Image" Tags="|deep-learning|image-classification|image-recognition|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15512" PostTypeId="1" CreationDate="2016-12-05T00:32:02.713" Score="2" ViewCount="11426" Body="&lt;p&gt;I have my confusion matrix as C.mat&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8263    20    39&#xA;2       3826  14&#xA;43       7    4431 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My predicted class labels are Ypred and actual labels are Ytest. Ypred size is 16000*1 and Ytest 16000*1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to calculate the &lt;strong&gt;R-squared&lt;/strong&gt; and &lt;strong&gt;RMSE&lt;/strong&gt;. Is there a way to directly calculate &lt;strong&gt;RMSE&lt;/strong&gt; and &lt;strong&gt;R-squared&lt;/strong&gt; from the confusion matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RMSE = sqrt(immse(Ypred, Ytest))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it didn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can use  either R or Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice will be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="26775" LastEditorUserId="26596" LastEditDate="2016-12-05T19:55:00.093" LastActivityDate="2019-07-01T08:49:58.593" Title="Calculating RMSE AND R-squared from the confusion matrix" Tags="|multiclass-classification|confusion-matrix|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15514" PostTypeId="2" ParentId="15512" CreationDate="2016-12-05T01:29:56.857" Score="2" Body="&lt;p&gt;The confusion matrix suggests that you are performing classification rather than regression. RMSE and R-square are measures associated with continuous variables; For categorical variables, I'd suggest using Accuracy / Recall / Precision / F1 score to measure the performance of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="25800" LastEditDate="2016-12-05T01:54:27.940" LastActivityDate="2016-12-05T01:54:27.940" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15515" PostTypeId="2" ParentId="15512" CreationDate="2016-12-05T02:18:07.767" Score="1" Body="&lt;p&gt;If you use RMSE for classification, then effectively every squared error will be a &lt;code&gt;1&lt;/code&gt;. The mean squared error will be your misclassification rate and the RMSE the square root of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Helpful Information&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The square root of the mean/average of the square of all of the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of RMSE is very common and it makes an excellent general purpose error metric for numerical predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compared to the similar Mean Absolute Error, RMSE amplifies and severely punishes large errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}((y_{i} -  \hat{y}_{i})^2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MATLAB Code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RMSE = sqrt(mean((y-y_pred).^2));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R Code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RMSE &amp;lt;- sqrt(mean((y-y_pred)^2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python (Scikit Learn)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import mean_squared_error&#xA;RMSE = mean_squared_error(y, y_pred)**0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-12-05T02:18:07.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15520" PostTypeId="2" ParentId="15498" CreationDate="2016-12-05T07:15:02.577" Score="2" Body="&lt;h2&gt;Never apply k-means to &lt;code&gt;id&lt;/code&gt; columns&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Even when they are numerical: the &lt;em&gt;mean&lt;/em&gt; does not make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, you need to &lt;strong&gt;aggregate&lt;/strong&gt; and pivot your data appropriately. One row per entity that you want to cluster (e.g. stores, or employees?). Multiple &lt;code&gt;value&lt;/code&gt; columns. No &lt;code&gt;id&lt;/code&gt; except the row number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget to normalize / preprocess your data. E.g. one store may have a much better location. If you cluster your employees, expect to find a cluster that is simply a store. (Which is a correct, albeit useless, clustering).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-12-05T07:15:02.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15522" PostTypeId="2" ParentId="13823" CreationDate="2016-12-05T09:50:45.357" Score="1" Body="&lt;p&gt;There are many factors which influence choice of classifier algorithm. Number of target classes does not generally have an influence, compared to the nature of input features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As one example, if your input data is natural audio or image, then regardless of the number of classes, a deep convolutional neural network is very likely going to have the best performance.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What algorithms generally make the best predictions with the smallest amount of data at each of these tiers?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is no a priori best approach based on number of output classes. The &quot;best predictions&quot; vs &quot;smallest amount of data&quot; is also a trade-off, where simpler models will perform better than complex ones on small amounts of data, but the more complex models will cope better with larger amounts of data, and will then give you better predictions. At some point you might have enough data so that sampling more will not improve your trained models, but you need to establish that empirically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most algorithms allow you to explore the trade-off within them by varying hyper-parameters to make the model simpler for smaller data sets and more complex when there is more training data. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is not strictly true. In general, the distinction between classifying and regression is a hard line. If you are classifying hand-written symbols into an alphabet for instance, it doesn't really matter if you are doing this for 10, 100 or 1000 classes, there is not a &lt;em&gt;practical&lt;/em&gt; point at which the symbols turn from being a set of objects into a continuous space to perform regression over.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could be true if your target class represents a range within some continuous variable (e.g. classifying an event by some of its properties into which year it occurred). But in that case the problem is inherently a regression problem to start with. In fact you may be better off training a regression algorithm in this case even for small number of target classes, and simply binning the predictions into relevant classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could also be true that your target class represents a rank or sequence within an ordered set. In which case this does look more like a regression problem when you have a longer sequence. In general, if you can &lt;em&gt;arrange&lt;/em&gt; your target classes into a meaningful sequence, then you might be able to perform some kind of &lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;ordinal regression&lt;/a&gt; which could be a better choice than using a classifier. However, classifying symbols/alphabets does not work this way, because the sequence in those is arbitrary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you might be facing such a large number of classes that a single classifier model is overwhelmed and you need to approach the problem differently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an example of this last case, consider a classifier for images of pets. If it had three classes (cats, dogs, rabbits), then you'd clearly use standard classification approach. Even when classifying by breed - 100s of classes - then this approach still works well enough, as seen in &lt;a href=&quot;http://www.image-net.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageNet&lt;/a&gt; competitions. However, once you decide to try and detect the &lt;em&gt;identity&lt;/em&gt; of each pet (still technically a class) you hit a problem using simple classifier techniques - in that case the structure of the solution needs more thought. One possible solution is a regression algorithm trained to extract biometric data from the image (nose length, distance between eyes, angle subtended between centre of jaw and ears) and move the classification stage into KNN based on a database of biometric data for observed individuals. This is how some face identifying algorithms work, by mapping images of faces into an easy-to-classify continuous space first (typically using a deep CNN), then using a simpler classifier that scales well across that space.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-12-05T15:04:43.857" LastActivityDate="2016-12-05T15:04:43.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15523" PostTypeId="2" ParentId="488" CreationDate="2016-12-05T11:08:21.250" Score="2" Body="&lt;p&gt;GLM is absolutely a statistical model , while more and more statistical methods have being applied in industrial production as machine learning &lt;strong&gt;tricks&lt;/strong&gt; . Meta-analysis which I read the most during these days is a good example in statistical field . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A perfect industrial application with GLM can explain why your friend told you that GLM was regarded as a machine learning &lt;strong&gt;technique&lt;/strong&gt; . You can refer the source paper &lt;a href=&quot;http://www.kdd.org/kdd2016/papers/files/adf0562-zhangA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdd.org/kdd2016/papers/files/adf0562-zhangA.pdf&lt;/a&gt; about that .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I implemented a simplified one which was treated as the main framework for my recommendation system in production scenario few weeks ago . Much appreciated if you give me some tips , and you can check the source code : &lt;a href=&quot;https://github.com/PayneJoe/algo-sensetime/blob/master/src/main/scala/GLMM.scala&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/PayneJoe/algo-sensetime/blob/master/src/main/scala/GLMM.scala&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopes this will helps you , good day !&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="9728" LastEditDate="2016-12-05T11:43:00.267" LastActivityDate="2016-12-05T11:43:00.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15524" PostTypeId="1" CreationDate="2016-12-05T12:56:09.450" Score="9" ViewCount="3845" Body="&lt;p&gt;I'm using the keras package in order to train an LSTM for a univariate time series of type numeric (float). Performing a 1-step ahead forecast is trivial, but I'm not sure how to perform a, let's say, 10-step ahead forecast. Two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I read about sequence to sequence NNs, but can barely find anything of it in the context of time series forecasting. Am I right with the assumption that the forecasting of more than 1 time step in advance is a seq2seq problem? That makes sense to me because each forecast depends on its predecessor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) An intuitive solution without seq2seq would be: Perform 1-step ahead forecast, then append this forecast to the series and use it to obtain the next forecast, and so on. How would this differ from a seq2seq approach?&lt;/p&gt;&#xA;" OwnerUserId="26797" LastActivityDate="2019-10-08T17:46:07.433" Title="Is time series multi-step ahead forecasting a sequence to sequence problem?" Tags="|time-series|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15526" PostTypeId="1" AcceptedAnswerId="15536" CreationDate="2016-12-05T14:01:22.910" Score="18" ViewCount="22355" Body="&lt;p&gt;If I have a training data set and I train a Naive Bayes Classifier on it and I have an attribute value which has probability zero. How do I handle this if I later want to predict the classification on new data? The problem is, if there is a zero in the calculation the whole product becomes zero, no matter how many other values I got which maybe would find another solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|spam=yes) = P(TimeZone = US | spam=yes) \cdot P(GeoLocation  = EU | spam = yes)  \cdot  ~ ... ~  = 0.004 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x|spam=no) = P(TimeZone = US | spam=no) \cdot P(GeoLocation  = EU | spam = no)  \cdot  ~ ... ~  = 0 $ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole product becomes $0$ because in the training data the attribute TimeZone US is always Yes in our small training data set. How can I handle this? Should I use a bigger set of training data or is there another possibility to overcome this problem?&lt;/p&gt;&#xA;" OwnerUserId="26799" LastEditorUserId="26562" LastEditDate="2018-05-03T21:11:06.160" LastActivityDate="2018-05-03T21:11:06.160" Title="How to handle a zero factor in Naive Bayes Classifier calculation?" Tags="|classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="15529" PostTypeId="1" CreationDate="2016-12-05T16:25:07.073" Score="0" ViewCount="46" Body="&lt;p&gt;I have a very large dataset consisting of approximately 15,000 rows of survey data. The surveys are quantitative summaries of interviews with investors regarding their rankings of financial asset managers. They rate a manager on a scale from 1-5 and we have approximately 5 years of data for each investor on asset manager.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is indexed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Investor name | Manager name | Year | Rating parameter 1 | R. parameter 2 | ... | R. parameter 10 |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an idea, I made an example below. Currently the data is saved in an Access-database with a ton of SQL JOINT's - but I can convert the data pretty much how I want to. So compatibility is not an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is:&#xA;I have a wild idea that somehow the answers are correlated between years. So maybe an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for ways to test my hypothesis without having to do it manually (this would be quite tedious). Maybe through machine-learning or subgroup discovery (I have not idea if this is correct). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are any of you familiar with tools or algorithms that can help me test my hypotheses? Naturally I'll have to make sure my analyses are statistically significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking very much forward hearing what you think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Example of data&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Buffet Holdings | Goldman Sachs AM | 2014 |1|5|5|1|4|5|1|1|1|5| &lt;br&gt;&#xA;Buffet Holdings | Goldman Sachs AM | 2015 | 4 | ........| 1 | &lt;br&gt;&#xA;Gates Ltd. | JP Morgan AM | 2014  | 4 | ........| 2 | &lt;br&gt;&#xA;Gates Ltd. | JP Morgan AM | 2015 | 4 | ........| 3 |&lt;br&gt;&#xA;Buffet Holdings | JP Morgan AM | 2014 | 3 | ........| 5 | &lt;br&gt;&#xA;Buffet Holdings | JP Morgan AM | 2015 | 2 | ........| 1 | &lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="26795" LastActivityDate="2016-12-05T21:52:07.103" Title="Recommendations for algorithm" Tags="|machine-learning|data-mining|finance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15532" PostTypeId="2" ParentId="15423" CreationDate="2016-12-05T17:49:50.710" Score="11" Body="&lt;p&gt;&lt;strong&gt;1. Why are the q-values of different actions very close to each other for a given state ?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm going to explain this with a small example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the game of &quot;Catch&quot;. Fruits(circular) keep falling from the top of the screen (vertically) and the agent(square) needs to just align itself to the fruit to get the reward. There are three actions that it can take : move left , stay , move right. Let's say that a2 refers to not moving the paddle, a3 refers to moving right and a1 refers to moving left. The image on the screen is to be used as the state in the MDP formalism.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CnyU2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CnyU2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we take a sub-optimal action a3(move right) and move to the next state. Then the best action in that state would be to move left (a1) and then execute the optimal action. So, the only cost difference between the actions a2 and a3 will be the two steps wasted to go and come back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is no negative reward for taking the sub-optimal action, then the agent has no incentive to choose the optimal action. So, the negative rewards for taking the sub-optimal action should be high enough that the agent is discouraged from doing it. I've tried to put this intuition mathematically here. This could explain why q-values are so close to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the optimal Q* function satisfies the following :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8RFqP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8RFqP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Is there a bound on this gap ?&lt;/strong&gt;&#xA;The bound that I've shown is approximate. The technical term for this difference is &quot;Action Gap&quot;. A much better analysis is available here : &lt;a href=&quot;https://arxiv.org/abs/1512.04860/&quot; rel=&quot;noreferrer&quot;&gt;Increasing the Action Gap: New Operators for Reinforcement Learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Why are the q-values of different states very close to each other for a given action ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should be fairly obvious. Consider two different cases. In the first one, the fruit falls from the top left corner and in the second one, the fruit falls from the top right corner. The agent in both cases is at the center. Then, consider the action of moving left. This action will give a high q-value in the first case and a low q-value in the second case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. How to solve the action gap problem ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's define the average reward in a state to be $V$. $V$ is approximately the average of Q-values of all the actions in a given state. If we now subtract this $V$ from each of the $Q$ values, we get a quantity which describes how good the action is compared to the average of all the action. This is called as the advantage of the action. In some sense, this is approximately the action gap. So, instead of predicting the q-values, we are directly predicting the action gap itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link : &lt;a href=&quot;https://arxiv.org/pdf/1602.01783.pdf&quot; rel=&quot;noreferrer&quot;&gt;A3C paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9577" LastActivityDate="2016-12-05T17:49:50.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15534" PostTypeId="2" ParentId="15529" CreationDate="2016-12-05T21:52:07.103" Score="1" Body="&lt;p&gt;A correlation matrix would help discover any pair-wise correlation (between any 2 variables). I think that would be a great place to start, as it is just one line of code for initial analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, if you want to investigate &quot;an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10&quot;, you could write code to form interaction variables like sum / product / divide and build a correlation matrix between the interaction variables vs. the original variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I found the corrplot R package useful in visualizing the correlation matrix. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-05T21:52:07.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15535" PostTypeId="1" CreationDate="2016-12-05T22:08:51.903" Score="7" ViewCount="4611" Body="&lt;p&gt;I built a CNN to learn to classify EEG data (only about 4000 training examples, 2 classes, 50-50 class balance). Each training example is 64x512, with 5 channels each&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ive tried to keep the network as simple/small as possible for testing:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ConvLayer (4 filters)&lt;/li&gt;&#xA;&lt;li&gt;MaxPool&lt;/li&gt;&#xA;&lt;li&gt;Dropout 50%&lt;/li&gt;&#xA;&lt;li&gt;Fully connected (50 neurons)&lt;/li&gt;&#xA;&lt;li&gt;Dropout 50%&lt;/li&gt;&#xA;&lt;li&gt;Softmax&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Im also using weight decay (L2 reg, lambda = 0.001)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is no matter how I play with the filter parameters (size, stride, number) my network keeps overfitting. It fits the training data 100%, but no matter what I do I can't get the test accuracy over 65%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is such a small network overfitting? I thought it was a sample size issue, but I've read a number of research papers on EEG and BCI and they occasionally have even smaller sample sizes than I do&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What else can be done to regularize a CNN?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T15:28:28.963" LastActivityDate="2022-01-24T14:58:55.350" Title="Convolutional Neural Network overfitting" Tags="|machine-learning|neural-network|classification|convolutional-neural-network|regularization|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15536" PostTypeId="2" ParentId="15526" CreationDate="2016-12-05T22:28:47.640" Score="14" Body="&lt;p&gt;An approach to overcome this 'zero frequency problem' in a Bayesian setting is to add one to the count for every attribute value-class combination when an attribute value doesn’t occur with every class value. So, for example, say your training data looked like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c|c|} &#xA; &amp;amp; \text{Spam} = yes &amp;amp; \text{Spam} = no \\ \hline&#xA;\text{TimeZone} = US &amp;amp; 10 &amp;amp; 5 \\ \hline&#xA;\text{TimeZone} = EU &amp;amp; 0 &amp;amp; 0 \\ \hline&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{10}{10} = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{0}{10} = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you should add one to every value in this table when you're using it to calculate probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c|c|} &#xA; &amp;amp; \text{Spam} = yes &amp;amp; \text{Spam} = no \\ \hline&#xA;\text{TimeZone} = US &amp;amp; 11 &amp;amp; 6 \\ \hline&#xA;\text{TimeZone} = EU &amp;amp; 1 &amp;amp; 1 \\ \hline&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{11}{12}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{1}{12}$&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-05T22:28:47.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15537" PostTypeId="1" AcceptedAnswerId="15539" CreationDate="2016-12-05T23:07:13.300" Score="1" ViewCount="1138" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NVaHC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NVaHC.png&quot; alt=&quot;sample data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to run a logistic regression on a data set where my dependent variable is a proportion of a binary variable, rather than the binary variable itself.  I have seen a ton of documentation that says this is possible, but I am having trouble finding an example of how to actually do it.  I am open to using scikit learn, statsmodels, or any other library that will do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have added a photo showing a simplified version of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;successes&lt;/code&gt; here is just a count of a binary (1/0) outcome.  instead of having the individual observations, I only have them rolled up, but my understanding is that it is still a logistic regression problem.  I want to predict the dependent variable 'proportion' based on the features.  I understand this conceptually, but am just trying to find an example of this in python.  all of the examples I have seen assume a binary dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;your help is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="26810" LastActivityDate="2016-12-05T23:48:23.247" Title="Example of Logistic Regression using a proportion as the dependent variable?" Tags="|python|logistic-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15539" PostTypeId="2" ParentId="15537" CreationDate="2016-12-05T23:48:23.247" Score="0" Body="&lt;p&gt;As the link by @Spacedman shows, binomial regression works well. If you don't have the attempts and successes but, instead, have just the proportion, then you'll want to use beta-regression. After a little digging, it doesn't seem like this is available in Python. &lt;a href=&quot;http://www.win-vector.com/blog/2014/01/generalized-linear-models-for-predicting-rates/?utm_source=rss&amp;amp;utm_medium=rss&amp;amp;utm_campaign=generalized-linear-models-for-predicting-rates&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a blog post demoing it in R&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-12-05T23:48:23.247" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15540" PostTypeId="1" AcceptedAnswerId="15545" CreationDate="2016-12-06T00:00:03.877" Score="9" ViewCount="5319" Body="&lt;p&gt;so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But because people change, data and many other factors change as well. And performance of our model eventually will decrease. So then it needs to be replaced with the new, better model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the common techniques, model stability tests, model performance tests, metrics after deployment? How to decide when to replace current model with newer one?&lt;/p&gt;&#xA;" OwnerUserId="26701" LastActivityDate="2020-07-09T07:56:14.490" Title="How do you evaluate ML model already deployed in production?" Tags="|machine-learning|model-evaluations|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15541" PostTypeId="2" ParentId="11921" CreationDate="2016-12-06T00:17:38.320" Score="0" Body="&lt;p&gt;The MXNetR package is an interface of the MXNet library written in C++. It contains feed-forward neural networks and convolutional neural networks (CNN) (MXNetR 2016a).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.is.uni-freiburg.de/resources/r-oeffentlicher-zugriff/deep-learning-in-r/deep-learning-in-r-en?set_language=en&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.is.uni-freiburg.de/resources/r-oeffentlicher-zugriff/deep-learning-in-r/deep-learning-in-r-en?set_language=en&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10815" LastActivityDate="2016-12-06T00:17:38.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15543" PostTypeId="2" ParentId="15540" CreationDate="2016-12-06T01:00:49.857" Score="5" Body="&lt;p&gt;After the models are deployed in production, I'd monitor the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) The same metric you used to evaluate the performance of your model, in some cases it is accuracy, or it could be precision, recall, RMSE. I'd plot a daily time series charting the metric and see that it is still performing above a satisfactory threshold. There might be seasonality within the calender, the model performs well around certain months and not so well in other months. I'd compare the performance against the test/validation sets of the same months to account for seasonality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Apart from looking at the performance of the model, especially if one is working with shared computing resources, I'd also recommend keeping a close tab on the data aggregation runtime, model runtime, success rates of models runs over the past period.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-06T01:00:49.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15545" PostTypeId="2" ParentId="15540" CreationDate="2016-12-06T02:40:51.703" Score="3" Body="&lt;p&gt;What you should consider more often in production scenario is &lt;strong&gt;revenue&lt;/strong&gt; for your model , and A/B test is a must . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in your case , you can exactly measure how much money can your model for loan default prediction bring to you , or how much loss can your model can save for you .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides , you can check if the distribution of your prediction is consistent with that of ground truth concerning accuracy and stability for your model .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopes this will help you , good luck -)&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-12-06T02:40:51.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15547" PostTypeId="2" ParentId="15540" CreationDate="2016-12-06T06:51:48.513" Score="4" Body="&lt;p&gt;This question is very common in the automation when machine learning used to perform specific tasks. Guaranteeing the quality is always a must.&#xA;&lt;br&gt;&lt;br&gt;Evaluating the model while it is in production is not an easy task. the reason, why?&#xA;&lt;br&gt;In order to evaluate the model in production you need to have the ground truth. This ground truth is not available (if it is available no need to have the model).&#xA;&lt;br&gt;Getting the ground truth (by using human for example) is not a good solution: &lt;br&gt;1- It is very expensive,&#xA;&lt;br&gt;2- Again, if you will generate the ground truth for data in production then no need to have a model. BUT how to deal with this problem in reality?&#xA;&lt;br&gt;&lt;br&gt;I have worked recently on a prediction model that is used to predict the vehicles (Makes, Models), since every year we may have new models, makes, it is a good question to ask how often do I have to repeat the training process?&#xA;&lt;br&gt;ImpThree different ways I used to answer this question:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I analyzed the changes in the training data I have year by year. based on this variation I can estimate the number of new makes and models appear every year and the number of makes and models disappear every year, and thus I can estimate the expected degradation in performance.&lt;/li&gt;&#xA;&lt;li&gt;I did several experiments using the data from 1990-2014 to predict 2015. Using Data from 1991-2015 to predict 2016. This helps me a lot in understanding how much my model is invariant from year to year.&lt;/li&gt;&#xA;&lt;li&gt;Instead of scanning all the data from production, you can randomly sample. The used distribution can be adaptive, such that the number of sampled records increases gradually with time. The reason why the distribution is adaptive because we are expecting the model performance to deviate from the expected performance with time.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="58341" LastEditDate="2020-07-08T23:57:55.053" LastActivityDate="2020-07-08T23:57:55.053" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15549" PostTypeId="2" ParentId="15535" CreationDate="2016-12-06T09:44:01.643" Score="3" Body="&lt;p&gt;This is because you have very little amount of data. If not enough data is provided to CNNs, they are very likely to overfit. You can do the following things in order to overcome this problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Augmentation&lt;/strong&gt; : It is a technique to create new examples from the training examples by doing some preprocessing on them e.g. Rotation, Scaling, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rigorous Dropout&lt;/strong&gt; : Dropout is a very powerful technique to control overfittting. You can try using more rigorous dropout in your architecture&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-12-06T09:44:01.643" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15550" PostTypeId="1" CreationDate="2016-12-06T12:21:00.633" Score="1" ViewCount="356" Body="&lt;p&gt;In the &lt;a href=&quot;https://www.amazon.com/b?ie=UTF8&amp;amp;node=16008589011&quot; rel=&quot;nofollow noreferrer&quot;&gt;Amazon Go page&lt;/a&gt; they claim to use &quot;deep learning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is deep learning used in the &quot;Go&quot; service? Is it computer vision or inventory management?&lt;/p&gt;&#xA;" OwnerUserId="10972" LastActivityDate="2016-12-06T12:38:43.470" Title="How is Amazon using Deep Learning in the new Amazon Go?" Tags="|deep-learning|amazon-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15551" PostTypeId="2" ParentId="15550" CreationDate="2016-12-06T12:38:43.470" Score="3" Body="&lt;p&gt;&quot; Our Just Walk Out technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. &quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out of this sentence I understand that camera's will record everything you do. Based on these videos they will record the things you bought etc. To get the information out the video deep learning will be used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick search on google about &quot;video analysis deep neural network&quot; will give you results like: &lt;a href=&quot;https://arxiv.org/abs/1512.03131&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1512.03131&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25104" LastActivityDate="2016-12-06T12:38:43.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15552" PostTypeId="2" ParentId="410" CreationDate="2016-12-06T13:26:33.873" Score="6" Body="&lt;p&gt;Learning rate , transformed as &quot;step size&quot; during our iteration process , has been a hot issue for years , and it will go on .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three options for step size in my concerning :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One is related to &quot;&lt;strong&gt;time&lt;/strong&gt;&quot; , and each dimension shall share the&#xA;same step size . You might have noticed something like &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\it\huge\bf\frac{\alpha}{\sqrt{t}}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;while t demonstrates the current iteration number , alpha is hyper parameter&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the next one is connected with &lt;strong&gt;gradient&lt;/strong&gt; , and each dimension have their&#xA;own step size . You might have noticed something like&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\it\huge\frac{1}{\frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t - 1}{g_{s}^2}}} -&#xA; \frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t}{g_{s}^2}}}}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;while alpha and beta are hyper parameter , g demonstrates gradient&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the last one is the &lt;strong&gt;combination of time and gradient&lt;/strong&gt; , and it should be&#xA;like&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\it\huge\frac{1}{\frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t - 1}{g_{s}^2}}} -\frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t}{g_{s}^2}}}} + \frac{\gamma}{\sqrt{t}}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;or&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\it\huge\frac{1}{\frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t - 1}{g_{s}^2}}} -  \frac{\alpha}{\beta + \sqrt{\sum_{s = 1}^{t}{g_{s}^2}}}} * \frac{\gamma}{\sqrt{t}}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hopes this will help you , good luck -)&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="9728" LastEditDate="2016-12-08T02:22:50.643" LastActivityDate="2016-12-08T02:22:50.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15553" PostTypeId="1" CreationDate="2016-12-06T13:31:05.340" Score="15" ViewCount="3438" Body="&lt;p&gt;There are several &lt;strong&gt;feature selection&lt;/strong&gt; / variable selection approaches (see for example &lt;a href=&quot;http://www.jmlr.org/papers/v3/guyon03a.html&quot; rel=&quot;noreferrer&quot;&gt;Guyon &amp;amp; Elisseeff, 2003&lt;/a&gt;; &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v10/liu10b/liu10b.pdf&quot; rel=&quot;noreferrer&quot;&gt;Liu et al., 2010&lt;/a&gt;): &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;filter methods (e.g., correlation-based, entropy-based, random forest importance based),&lt;/li&gt;&#xA;&lt;li&gt;wrapper methods (e.g., forward-search, hill-climbing search), and &lt;/li&gt;&#xA;&lt;li&gt;embedded methods where the feature selection is part of the model learning. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Many published algorithms are also implemented in the machine learning tools like R, Python, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be an appropriate method to compare different feature selection algorithms and to select the best method for a given problem / dataset? &#xA;A further question would be, whether there are any metrics known that measure the performance of feature selection algorithms?&lt;/p&gt;&#xA;" OwnerUserId="25217" LastActivityDate="2017-09-27T11:09:46.483" Title="How to compare the performance of feature selection methods?" Tags="|feature-selection|performance|model-selection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15554" PostTypeId="2" ParentId="14676" CreationDate="2016-12-06T13:37:47.953" Score="-1" Body="&lt;p&gt;It is important to know that you can get the values of for the parameters you are seeking by using &lt;em&gt;grid search&lt;/em&gt; and &lt;em&gt;cross-validation&lt;/em&gt;. Take a look at the documentation on &lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit Learn&lt;/a&gt; for how to find the best parameters for fitting models and determine if it is helpful. &lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-12-06T13:37:47.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15555" PostTypeId="1" CreationDate="2016-12-06T14:38:19.687" Score="0" ViewCount="180" Body="&lt;p&gt;I'm trying to build a bayesian network for satisfcation survey data. &#xA;My data is made of 13 questions about services, products etc... each customer can answer from 1 (Very unsatisfied) to 4 (very satisfied) with no &quot;neutral feeling&quot;). There is an other question about the overall satisfaction where the customer can answer 1 (very unsatisfied), 2 (unsatisfied), 3 (satisfied), 4(very satisfied) and 5 (totally satisfied). &#xA;I plan to use bnlearn to build the bayesian network. The goal is to identify important features who lead customers to give 5 on overall satisfaction item. &#xA;In your opinion, what is the best way to recode the overall satisfaction : 0 for 1, 2, 3, 4 and 1 for 5 (totally satisfied)? I tried but results let me skeptical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best regards,&lt;/p&gt;&#xA;" OwnerUserId="26828" LastActivityDate="2016-12-07T01:30:51.150" Title="R - Bayesian network for satisfaction survey data" Tags="|r|bayesian-networks|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15556" PostTypeId="1" AcceptedAnswerId="15608" CreationDate="2016-12-06T15:17:54.990" Score="1" ViewCount="290" Body="&lt;p&gt;Here's the scenario, There's a database with thousands of single-option questions for testing a specific skills, and a large number of users (either professional or amateur in this skill), each of which answer 10 random questions from the database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing that I can think of is to differentiate question difficulty level according to the correct rate of each question. But how could I take fully use of other informations like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the correct rate from the user's perspective and feedback to its own influence to the difficulty level of questions (if user A answers 9 out of 10 questions correct, then the incorrect one (question_10) is more likely to be harder, than an user B answers 1 out of 10 questions correct and question_10 is in the incorrect set)&lt;/li&gt;&#xA;&lt;li&gt;the answer time for each question by each user&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Could anyone give me some ideas on this model, like where should I delve more into to make the difficulty level of question more accurate? Great thanks!&lt;/p&gt;&#xA;" OwnerUserId="21525" LastActivityDate="2022-11-09T20:03:40.400" Title="Which machine learning algorithm should I apply for differentiate question difficulty level with users' result" Tags="|machine-learning|classification|predictive-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15557" PostTypeId="2" ParentId="15451" CreationDate="2016-12-06T16:31:20.313" Score="1" Body="&lt;p&gt;The difference comes about because the sample mean and sd estimators are assuming the data is a random sample from the distribution on the full range of the distribution, in this case -Inf to +Inf. In your second case above there's no data below zero. The sample mean estimator can't see that you've only got half a Gaussian there, it will always be a positive number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the curve-fitting approach of the least-squares fitting procedure (which is presumably possible via EM, or any of a dozen other algorithms) &lt;em&gt;can&lt;/em&gt; see the data as &lt;em&gt;part of&lt;/em&gt; a curve that has a Gaussian density shape. If the data is just curving upwards, it will fit to the left side of a Gaussian. If its curving down it will fit to the right side. If its flat it will either fit it to the peak with a very large variance, or a tail and might even go degenerate and divide by zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you are doing parametric estimation of a distribution, or curve-fitting, which are categorically different things.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-12-06T16:31:20.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15558" PostTypeId="2" ParentId="9440" CreationDate="2016-12-06T16:43:49.917" Score="3" Body="&lt;p&gt;There are several ways in which you can achieve your desired result:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Implement the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;make_scorer function&lt;/a&gt; from Scikit learn&lt;/li&gt;&#xA;&lt;li&gt;Make modification to the class_weight argument&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In regards to your SVM question take a look at the below code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; class sklearn.svm.SVC(C=1.0, kernel='rbf', degree=4, gamma=0.0,&#xA; coef0=0.0, shrinking=True, probability=False, tol=0.003,&#xA; cache_size=300, class_weight=None, verbose=False, max_iter=-1,&#xA; random_state=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the above example, the class_weight function can be changed to 'auto' or you can pass dictionary values which have the user-defined class weights. &lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-12-06T16:43:49.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15560" PostTypeId="2" ParentId="15445" CreationDate="2016-12-06T18:45:43.367" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;https://cran.r-project.org/web/packages/visNetwork/vignettes/Introduction-to-visNetwork.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;visNetwork&lt;/a&gt; R package is the best I've worked with.  It renders with the vis.js Javascript package right in your RStudio window.  Nodes and edge visualization is fully customizable and may be data-driven.  They implement click-functionality on your graphs, so you can rearrange them or highlight a node and its neighbors.  I've found people really enjoy this in presentations, and the highlight functionality is great for making static PowerPoint presentations or figures.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-12-06T18:45:43.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15562" PostTypeId="2" ParentId="15553" CreationDate="2016-12-06T19:53:40.190" Score="7" Body="&lt;p&gt;This is a hard problem and researchers are making a lot of progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for &lt;strong&gt;&lt;em&gt;supervised&lt;/em&gt;&lt;/strong&gt; feature selection, I'd recommend &lt;a href=&quot;http://statweb.stanford.edu/~tibs/lasso.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LASSO&lt;/a&gt; and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two major caveats of LASSO are that (1) the selected features will not automatically detect an interaction, so you have to craft all of your features a priori (i.e., before running them through the model) and (2) LASSO will not identify non-linear relationships (e.g., &lt;a href=&quot;https://stats.stackexchange.com/questions/211534/how-to-interpret-quadratic-terms&quot;&gt;a quadratic relationship&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A way to try and get past these two caveats is to use &lt;a href=&quot;http://www-stat.stanford.edu/~jhf/ftp/trebst.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient Boosted Machines&lt;/a&gt; which does feature selection automatically. It's worth noting the statistical properties of GBM are a little more ambiguous than that of the LASSO. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for unsupervised feature selection, it seems there's a similar regularization approach used by &lt;a href=&quot;http://www.ijcai.org/Proceedings/13/Papers/241.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;these&lt;/a&gt; researchers, but evaluation in this particular case becomes less obvious. People try a lot of different things like PCA/SVD or K-Means which ultimately will try to find a linear approximation to the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, the typical measures of performance are the reconstruction error or the RMSE of the clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of software, R and Python both have GBM, LASSO, K-Means, SVD, and PCA. GLMNET and XGBoost in R and Sklearn for Python are the relevant libraries.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2017-09-27T11:09:46.483" LastActivityDate="2017-09-27T11:09:46.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15563" PostTypeId="2" ParentId="9375" CreationDate="2016-12-06T20:10:47.167" Score="1" Body="&lt;p&gt;If the problem is that you don't know all the categories in advance and different records have different categories, then the &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_hashing&quot; rel=&quot;nofollow noreferrer&quot;&gt;hashing trick&lt;/a&gt; might help.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2016-12-06T20:10:47.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15564" PostTypeId="2" ParentId="10429" CreationDate="2016-12-06T20:51:47.023" Score="2" Body="&lt;p&gt;I am hesitant to think of features as bad features. However, there are features more or less useful relative to other features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kUWT8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kUWT8.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general pruning features are seen as a good best practice. &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Overfitting&lt;/a&gt; certainly is a real problem within machine learning. Usually a combination of creating too powerful of a model, not having enough data, and/or having too many features can create an undesirable outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Proper partitioning of a dataset can help to adjust the model parameters where warranted helping in a more generalizable model.&lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2016-12-06T20:51:47.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15565" PostTypeId="1" CreationDate="2016-12-06T21:19:22.113" Score="2" ViewCount="2632" Body="&lt;p&gt;The dataset that I am categorizing with TensorFlow ML library contains multiple labels per image. The contents are real estate images photographed from outside that are analyzed for various image features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to assign the labels, the most straightforward way is to use either many long labels, or few short labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Categories:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Building type (&quot;house&quot;, &quot;apartment building&quot;, &quot;condominium&quot;)&#xA;Build year (&quot;old&quot;, &quot;new&quot;)&#xA;Garage (boolean) - only for house&#xA;Floors (1, 2) - only for house&#xA;Construction (&quot;standalone&quot;, &quot;row house&quot;) - only for house&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Many long labels (&lt;code&gt;softmax_classifier&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;house_old_nogarage_onefloor_standalone&#xA;house_old_nogarage_onefloor_row&#xA;house_old_nogarage_twofloor_standalone&#xA;...43 more&#xA;apartment_old&#xA;apartment_new&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Few short labels (something like a &lt;code&gt;Multiclass Support Vector Machine&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;house&#xA;apartment&#xA;condominium&#xA;new&#xA;old&#xA;...7 more&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Alternative to this would be to use a multi-label classifier by replacing the default &lt;code&gt;softmax_classifier&lt;/code&gt; of the &lt;code&gt;Inception Model v3&lt;/code&gt; by something like the &lt;code&gt;sequence_classifier&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to represent the data accurately, but also I don't mind doing simple categorization if the accuracy is acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which of the proposed 3 labeling solutions suits the problem?&lt;/p&gt;&#xA;" OwnerUserId="13235" LastEditorUserId="13235" LastEditDate="2016-12-07T00:07:00.700" LastActivityDate="2016-12-08T16:57:00.173" Title="Classifying multilabel images with TensorFlow" Tags="|tensorflow|image-classification|multilabel-classification|image-recognition|labels|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15568" PostTypeId="1" AcceptedAnswerId="15570" CreationDate="2016-12-06T22:43:19.713" Score="5" ViewCount="6407" Body="&lt;p&gt;I've never understood how to calculate an autoencoder loss function because the prediction has many dimensions, and I always thought that a loss function had to output a single number / scalar estimate for a given record. However, on GitHub I recently came across a &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples&quot; rel=&quot;noreferrer&quot; title=&quot;abc&quot;&gt;repository&lt;/a&gt; that has implemented an example autoencoder in TensorFlow, and the &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/autoencoder.py#L82&quot; rel=&quot;noreferrer&quot; title=&quot;asb&quot;&gt;squared error implementation&lt;/a&gt; is a lot simpler than I thought: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The TensorFlow &lt;a href=&quot;https://www.tensorflow.org/versions/r0.12/api_docs/python/math_ops.html#reduce_mean&quot; rel=&quot;noreferrer&quot; title=&quot;asf&quot;&gt;documentation&lt;/a&gt; on &lt;code&gt;reduce_mean&lt;/code&gt; says, among other things:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If axis has no entries, all dimensions are reduced, and a tensor with&#xA;  a single element is returned.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can I conclude from all of this that the squared error of an autoencoder prediction is just the average across all of the record's dimensions?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2016-12-06T22:48:49.940" LastActivityDate="2016-12-06T22:50:56.193" Title="Understanding autoencoder loss function" Tags="|autoencoder|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15570" PostTypeId="2" ParentId="15568" CreationDate="2016-12-06T22:50:56.193" Score="2" Body="&lt;p&gt;Yes, you are correct in thinking that the squared error of an autoencoder prediction for a single example is the average of the squared error of the prediction for all dimensions. Similarly, the squared error for a whole batch of examples will be the average of the error of each example in the batch. &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-06T22:50:56.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15573" PostTypeId="1" CreationDate="2016-12-07T00:08:48.120" Score="8" ViewCount="6352" Body="&lt;p&gt;In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2019-08-30T14:40:56.947" Title="Unbalanced class: class_weight for ML algorithms in Spark MLLib" Tags="|machine-learning|apache-spark|class-imbalance|weighted-data|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15575" PostTypeId="2" ParentId="15555" CreationDate="2016-12-07T01:30:51.150" Score="2" Body="&lt;p&gt;I see the task is to understand the pattern rather than developing a prediction model.&#xA;I would use regression trees rather than Bayesian network.&lt;/p&gt;&#xA;" OwnerUserId="14707" LastActivityDate="2016-12-07T01:30:51.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15576" PostTypeId="2" ParentId="15573" CreationDate="2016-12-07T03:39:24.093" Score="1" Body="&lt;p&gt;Algorithms in MLLib are always used as &lt;strong&gt;baseline&lt;/strong&gt; in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides , mechanism of &lt;strong&gt;BSP&lt;/strong&gt; in Spark , you can simply see as &lt;strong&gt;data parallel&lt;/strong&gt; ,  might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the &lt;strong&gt;same label distribution&lt;/strong&gt; as the whole . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At last , you only have to &lt;strong&gt;weight the loss value&lt;/strong&gt; for every minor labeled instance during your iteration process if you want to implement it . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopes this will help you , good luck -)&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-12-07T03:39:24.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15577" PostTypeId="1" AcceptedAnswerId="15605" CreationDate="2016-12-07T06:20:35.227" Score="1" ViewCount="1485" Body="&lt;p&gt;I'm processing geospatial data using Spark 2.0 Dataframes with the following schema:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;root&#xA; |-- date: timestamp (nullable = true)&#xA; |-- lat: double (nullable = true)&#xA; |-- lon: double (nullable = true)&#xA; |-- accuracy: double (nullable = true)&#xA; |-- track_id: long (nullable = true)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have seen that there are jumps of the location signal to a complete different place. The strange thing is, that the signal then remains for a certain time, say aound 25 seconds or 5 samples at the remote location and then jumps back to where I stand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to remove these outliers by calculating the speed between the current and the &quot;last valid record&quot; by calculating the speed between the points. &#xA;If the speed is above a given threshold, the current record should be dropped and the &quot;last valid record&quot; remains the same.&#xA;If the speed is below the threshold the current record is added to the result data frame and becomes the new &quot;last valid record&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Spark 2.0 with Dataframes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions of how to implement this strategy or any better strategy are highly appreciated. Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I asked the same questions in stackoverflow, with a concrete implementation. But, since I'm not sure if this is the right approach, and do not want bias the answers to a certain Spark method, I ask here for any suggestions. &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/41002844/how-to-filter-outlier-rows-from-spark-dataframe-based-on-distance-to-previous-va&quot;&gt;https://stackoverflow.com/questions/41002844/how-to-filter-outlier-rows-from-spark-dataframe-based-on-distance-to-previous-va&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16201" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-08T06:33:30.110" Title="Filtering outliers in Apache Spark based on calculations of previous values" Tags="|apache-spark|geospatial|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15579" PostTypeId="1" CreationDate="2016-12-07T06:24:15.550" Score="0" ViewCount="531" Body="&lt;p&gt;I am working on a personal project sponsored by a data scientists, he offered some data sets to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I have built a neural network system based on my own theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I tried to train it, I always can't get a ideal result. I think carefully chose the training data set samples may help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would like to know if there are some principles or theories things which can guide researchers choosing samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any books, papers or blogs would help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;by the way, I am working on a fraud detection project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, I think there are some misunderstandings in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is &lt;em&gt;a systematical theory which explains how to organize the training set&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a naive example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have example data set from US. There are tons of data, so it is not possible to train the machine with all the example set. Here comes the question, how should I pick carefully the training data set from the example data set? how to do that is a reasonable way? and a logical way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a further example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose the training set according to states evenly? or choose training set according to their population as weight?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, and I will provide more information if it is needed.&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="26856" LastEditDate="2016-12-09T00:48:01.263" LastActivityDate="2016-12-09T00:48:01.263" Title="How to chose training sample for neural network in fraud detection?" Tags="|machine-learning|neural-network|data-cleaning|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15580" PostTypeId="2" ParentId="15553" CreationDate="2016-12-07T06:28:40.620" Score="1" Body="&lt;p&gt;I always consider features selection as a step to a final result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hereunder, I somehow mix features selection and dimensionality reduction, which might have some goals and can be confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some typical uses:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;reduction of computations in machine learning: the quality of the selection is a factor of the final learning result and also, obviously, the speed to get that learning done&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;visualization/understanding of the data, where you combine eventually multiple dimensions. It is good when it doesn't hides interesting stuffs, and when that's understandable&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;simplification of the learning results, still to make them understandable (eg root cause analysis). Good if simple but still sufficient in terms of quality&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;controlling over fitting, as the previous reply suggests&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, I don't think there's general rule (as always in ML), but this is a case by case problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just a personal belief...&lt;/p&gt;&#xA;" OwnerUserId="3024" LastActivityDate="2016-12-07T06:28:40.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15581" PostTypeId="1" CreationDate="2016-12-07T07:03:20.337" Score="-1" ViewCount="64" Body="&lt;p&gt;what should be the min or max size of trained datasets should be used to feed a classifier ? can we use 1 GB or more data as trained data to feed a classifier for jvm related ML frameworks? &lt;/p&gt;&#xA;" OwnerUserId="20585" LastActivityDate="2016-12-07T07:31:24.313" Title="What is the minimum or maximum amount of trained data is used to train a classifier?" Tags="|machine-learning|classification|naive-bayes-classifier|training|" AnswerCount="1" CommentCount="0" ClosedDate="2016-12-07T18:58:03.860" ContentLicense="CC BY-SA 3.0" />
  <row Id="15582" PostTypeId="2" ParentId="15581" CreationDate="2016-12-07T07:31:24.313" Score="1" Body="&lt;p&gt;Instead of thinking it in terms of GBs, unless you're asking about the physical limits of JVM, my suggested approach towards this question would be to plot the cross-validated accuracy vs. the number of rows used in the modelling dataset through stratified sampling. You might observe a point of diminishing return where additional rows do not contribute towards higher accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, breadth-wise in terms of the number of variables used, you could plot a variable importance plot and plot the cross-validated accuracy vs. the top X important variables. In this case, you might even observe that accuracy increases when you use fewer variables as the noisy variables are discarded. You could then select the top X variables that give you the highest cross-validated accuracy.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-07T07:31:24.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15583" PostTypeId="1" CreationDate="2016-12-07T07:59:01.690" Score="2" ViewCount="3991" Body="&lt;p&gt;I have a bag-of-words consisting of about 60000 features. Each feature represents a diminsion. I want to represent this bag-of-words in a reduced 2D space. How do I do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have see an example &lt;a href=&quot;https://stackoverflow.com/questions/28160335/plot-a-document-tfidf-2d-graph&quot;&gt;here&lt;/a&gt;, which looks more like what I want but not really the same. In the example they have 2 transform and I only have one. Therefore as suggested I do not want to use pipeline. Below is my code which taking for ever and does not show any error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#myList contents about 800000 words&#xA;bag_of_words = vec.fit_transform(myList)&#xA;X = bag_of_words.todense()&#xA;pca = PCA(n_components=2).fit(X)&#xA;data2D = pca.transform(X)&#xA;plt.scatter(data2D[:,0], data2D[:,1])&#xA;plt.show() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have not found any better option and right now it looks like I am doing something wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to visualize a bag-of-words in a scatterplot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bag_of_words looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0, 548)    3&#xA;(0, 4000)   6&#xA;(0, 15346)  1&#xA;(0, 23299)  1&#xA;(0, 22931)  2&#xA;(0, 32817)  1&#xA;(0, 51733)  1&#xA;(0, 38308)  6&#xA;(0, 14784)  1&#xA;(0, 146873) 1&#xA; ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26858" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2020-08-11T16:13:00.043" Title="how can I visualize the bag-of-words in a scatterplot" Tags="|machine-learning|python|data-mining|bigdata|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15585" PostTypeId="2" ParentId="15444" CreationDate="2016-12-07T10:59:00.530" Score="0" Body="&lt;p&gt;I found a &lt;strong&gt;very good explanation for this in the paper &quot;LSTM: A Search Space Odyssey&quot;&lt;/strong&gt; (&lt;a href=&quot;https://arxiv.org/abs/1503.04069&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1503.04069&lt;/a&gt;). Read chapter 3 if you want to understand the history of the trainig algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a version of &lt;strong&gt;Truncated BPTT for LSTM which was used first&lt;/strong&gt;, where the cell state is propagated back many steps, but the gradients along other parts of the LSTM are truncated. In &lt;strong&gt;later papers also full-gradient-BPTT&lt;/strong&gt; ist used, where the gradients along gates and so on are also backpropagated in time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps you guys!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers, Torben&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2016-12-07T10:59:00.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15586" PostTypeId="1" AcceptedAnswerId="15674" CreationDate="2016-12-07T11:33:18.407" Score="-1" ViewCount="63" Body="&lt;p&gt;I have a set of predefined categories(like: cars, girls, etc.) and a set short texts and I need to assign one or several categories to each text. What is the best way to do this? I don't have trainings texts to make supervised learning classification.&#xA;Is Word2Vec the best choice or there is better approach?&lt;/p&gt;&#xA;" OwnerUserId="26867" LastActivityDate="2016-12-10T23:22:53.387" Title="Texts classifications approaches" Tags="|machine-learning|classification|neural-network|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15587" PostTypeId="1" CreationDate="2016-12-07T13:06:55.723" Score="3" ViewCount="1273" Body="&lt;p&gt;I am using python SKlearn DecisionTreeRegressor for my data. As DecisionTreeRegressor defines constant data at leaf node, my prediction is looking like a step, instead I wanted to force polyfit Equation of my interest. I was looking some thing like to use Decision tree to some depth, then apply normal regression at leaf node to predict continuous data.&#xA;For my data set normal regression doesn't work as it looks for global error minimization, so i am looking for this.&#xA;I am a beginner, any response would be appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="26869" LastActivityDate="2017-12-04T23:36:54.753" Title="How to force DecisionTreeRegressor to use polyfit equation instead of mse at leaf level in python SKlearn" Tags="|machine-learning|python|regression|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15589" PostTypeId="1" CreationDate="2016-12-07T13:10:52.220" Score="4" ViewCount="144651" Body="&lt;p&gt;I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part?&#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    INTERACTOR_A INTERACTOR_B&#xA;1          ce7380       ce6058&#xA;2          ce7380      ce13812&#xA;3          ce7382       ce7382&#xA;4          ce7382       ce5255&#xA;5          ce7382       ce1103&#xA;6          ce7388        ce523&#xA;7          ce7388       ce8534&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="26870" LastEditorUserId="118656" LastEditDate="2021-06-02T18:20:01.570" LastActivityDate="2021-07-01T20:53:43.867" Title="Remove part of string in R" Tags="|r|dataset|bioinformatics|" AnswerCount="7" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15590" PostTypeId="1" AcceptedAnswerId="15591" CreationDate="2016-12-07T13:49:58.090" Score="4" ViewCount="1366" Body="&lt;p&gt;So I have a background in computer programming and a little in machine learning in general.  What I would like to do is create a fun project in A.I. with deep learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset that has a whole bunch of stock prices at a certain date, with a bunch of features for each entry to go with it.  I also have some &quot;experts&quot; who made predictions on whether the stock will go up or down. As my dataset grows I can evolve the game to make selections from multiple stocks...etc &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially what I would love to do is create an A.I. app that will be fed the same data that the &quot;experts&quot; had and see if I can create something more accurate and beat them at it.  Is this a viable approach?  &lt;/p&gt;&#xA;" OwnerUserId="26800" LastEditorUserId="836" LastEditDate="2016-12-07T14:46:30.280" LastActivityDate="2020-02-21T10:27:34.240" Title="Deep Learning Project to Predict Stock Prices" Tags="|deep-learning|finance|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15591" PostTypeId="2" ParentId="15590" CreationDate="2016-12-07T14:44:06.983" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Essentially what I would love to do is create an A.I. app that will be fed the same data that the &quot;experts&quot; had and see if I can create something more accurate and beat them at it. Is this a viable approach?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sure, you can use one or more supervised learning techniques to train a model here. You have features, a target variable and ground truth for that variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to applying ML you have learned, all you need to do to test your application fairly is reserve some of the data you have with expert predictions for comparison as test data (i.e. do not train using it). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would caveat that with some additional thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You haven't really outlined an &quot;approach&quot; here, other than mentioning use of ML.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Be careful not to leak future data back into the predictive model when building a test version.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Predicting stock and markets is hard, because they react to their own predictability and many professional organisations trade on the slightest advantage they can calculate, with experienced and highly competent staff both gathering and analysing data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Not directly part of the answer, but to anyone just starting out and discovering machine learning, and finding this Q&amp;amp;A:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please don't imagine rich rewards from predicting markets using stats at home, it doesn't happen. If you think that this is a route to &quot;beating the market&quot; be aware that you are far from the first to think of doing this, and such a plan can be summarised like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Market Data + ML&lt;/li&gt;&#xA;&lt;li&gt;???&lt;/li&gt;&#xA;&lt;li&gt;Profit&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can fill in the ??? by learning &lt;em&gt;loads&lt;/em&gt; about financial markets - i.e. essentially by becoming one of the experts. ML is not a short-cut, but it might be a useful tool if you are, or plan to be, a market analyst.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-12-07T15:01:33.177" LastActivityDate="2016-12-07T15:01:33.177" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15592" PostTypeId="2" ParentId="15589" CreationDate="2016-12-07T15:42:04.007" Score="7" Body="&lt;p&gt;You may use &lt;code&gt;gsub&lt;/code&gt; function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; c &amp;lt;-  &quot;ce7382&quot;&#xA;&amp;gt; gsub(&quot;[a-zA-Z ]&quot;, &quot;&quot;, c)&#xA;[1] &quot;7382&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Feel free to add other characters you need to remove to the regexp and / or to cast the result to number with &lt;code&gt;as.numeric&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2016-12-07T15:42:04.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15593" PostTypeId="2" ParentId="15295" CreationDate="2016-12-07T15:46:37.270" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Recurrent Neural Network (RNN)&lt;/a&gt; are quite efficient at dealing with sequential information such as time-series.&lt;br&gt;&#xA;&lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM (Long Short Term Memory)&lt;/a&gt; is a type of RNN that can learn long-term dependencies and overcome some problems that vanilla RNNs have.&lt;br&gt;&#xA;If you want to try it on your problem, I'd suggest &lt;a href=&quot;http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial on LSTM for time-series prediction&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12558" LastActivityDate="2016-12-07T15:46:37.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15594" PostTypeId="2" ParentId="15589" CreationDate="2016-12-07T15:59:04.783" Score="0" Body="&lt;p&gt;Try this if you are not sure about characters in string:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;{as.numeric(gsub(&amp;quot;,&amp;quot;,&amp;quot;&amp;quot;, x)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="29169" LastEditDate="2021-06-01T21:36:58.760" LastActivityDate="2021-06-01T21:36:58.760" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15595" PostTypeId="1" AcceptedAnswerId="15609" CreationDate="2016-12-07T16:06:33.377" Score="16" ViewCount="2272" Body="&lt;p&gt;I went through Geoff Hinton's Neural Networks course on Coursera and also through &lt;a href=&quot;http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/&quot; rel=&quot;noreferrer&quot;&gt;introduction to restricted boltzmann machines&lt;/a&gt;, still I didn't understand the intuition behind RBMs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do we need to compute energy in this machine? And what is the use of the probability in this machine? I also saw this &lt;a href=&quot;https://www.youtube.com/watch?v=-l1QTbgLTyQ&quot; rel=&quot;noreferrer&quot;&gt;video&lt;/a&gt;. In the video, he just wrote the probability and energy equations before the computation steps and didn't appear to use it anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding to the above, I am not sure what the likelihood function is for?&lt;/p&gt;&#xA;" OwnerUserId="26872" LastEditorUserId="836" LastEditDate="2016-12-07T16:29:05.010" LastActivityDate="2017-01-06T17:56:55.787" Title="Intuition Behind Restricted Boltzmann Machine (RBM)" Tags="|unsupervised-learning|rbm|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15596" PostTypeId="2" ParentId="15590" CreationDate="2016-12-07T16:46:33.343" Score="2" Body="&lt;p&gt;&lt;em&gt;typically I would add this as a comment but, since my score threshold &amp;lt; 50, I am unable to do so - hence the &quot;Answer&quot; response&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interested in running ML algorithms against historic and future index prices, you might be interested in &lt;a href=&quot;https://www.quantopian.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quantopian - Kaggle for Finance Quants&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At Quantopian, you can upload, test, and compare your results with other ML Finance Quants. Additionally, you'll learn about certain financial metrics/ratios that are native to the financial sector. &lt;/p&gt;&#xA;" OwnerUserId="24945" LastActivityDate="2016-12-07T16:46:33.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15597" PostTypeId="1" CreationDate="2016-12-07T16:50:08.260" Score="2" ViewCount="889" Body="&lt;p&gt;I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database, it will ask the DBA to tune the database. My tracing data looks like this.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N4fAg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N4fAg.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xf35C.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xf35C.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The images show the query execution count from 2 different months. According to the images there is a considerable pattern chnage in query execution. Query2 is executed most of the times in the first dataset and Query3 is executed most of the time in second dataset. I want to mathamatically calcuate this pattern difference. One solution I can think of is calculating the percentage differene of execution count of each query and sum it all. If the value is greater than the predefined thresold I can send a notification to the DBA. Is there a better way to do that? Any data analysis technique that can be applied? Any libraries to do that?&lt;/p&gt;&#xA;" OwnerUserId="26881" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-01-07T19:36:09.933" Title="The best way to calculate variations between 2 datasets?" Tags="|data-mining|dataset|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15598" PostTypeId="1" AcceptedAnswerId="15621" CreationDate="2016-12-07T17:55:33.757" Score="2" ViewCount="1738" Body="&lt;p&gt;My question is really simple, how to find the filename associated with a prediction in Keras? That is, if I have a set of 100 test samples named  and I get a numpy array which contains the estimated class probabilities, how do I map the filenames to the probabilities?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;import os&#xA;import glob &#xA;&#xA;def load_test():&#xA;    X_test = []&#xA;    y_test = []&#xA;    os.chdir(testing_path)&#xA;    file_list = glob.glob('*.png')&#xA;    for test_image in file_list:&#xA;        img = cv2.imread(test_image,1)&#xA;        X_test.append(img)&#xA;        y_test.append(1)&#xA;   return X_test,y_test&#xA;&#xA;if __name__ == '__main__':&#xA;   X_test = np.array(X_test, dtype = np.uint8)&#xA;   X_test = X_test.reshape(X_test.shape[0],3,100,100)&#xA;   X_test = X_test.astype('float32')&#xA;   X_test /= 255&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="14360" LastEditDate="2016-12-08T10:26:16.703" LastActivityDate="2016-12-08T18:54:07.067" Title="How to find the filename associated with a prediction in Keras?" Tags="|python|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15599" PostTypeId="2" ParentId="15579" CreationDate="2016-12-07T22:06:33.180" Score="1" Body="&lt;p&gt;I think you are facing an unbalanced dataset problem (fraud), and here are some Q&amp;amp;A associated with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/6042/poor-performance-shown-on-rare-event-modeling/6062#6062&quot;&gt;Poor performance shown on Rare event modeling&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/134599/how-to-classify-a-unbalanced-dataset-by-convolutional-neural-networks-cnn&quot;&gt;https://stats.stackexchange.com/questions/134599/how-to-classify-a-unbalanced-dataset-by-convolutional-neural-networks-cnn&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-07T22:06:33.180" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15601" PostTypeId="1" AcceptedAnswerId="15606" CreationDate="2016-12-07T22:36:56.490" Score="5" ViewCount="4005" Body="&lt;p&gt;I am trying to build a binary classification system using different classification algorithms like random forests, support vector machines, AdaBoost. I want to use the output of these classifiers to visualize a score. For example, when using random forests, I would like to use the probability of a sample belonging to class A to build a score from 0 to 100. Given that random forests output a probability (from 0 to 1) using it as the score is intuitive (I would just multiply it by 100). However, given that SVMs output a classification but not a probabilistic output (i.e. distance to the hyperplane, but not probability), would it be legitimate to use the distance to the hyperplane as some sort of &quot;pseudo probability&quot;?  I would, for example, do max-min scale on the distance to the hyperplane for all samples so all distances are scaled from 0 to 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be sure that I can use the distance to the hyperplane as a pseudo-probability and that this pseudo-probability is comparable to the probability to belong to a given class outputted by the random forest. For example, that a sample with a probability of .80 of belonging to class A is the same than another sample of a (min-max transformed) probability of belonging to class A according to the SVM.&lt;/p&gt;&#xA;" OwnerUserId="24245" LastActivityDate="2016-12-10T23:35:47.383" Title="Can you interpret probabilistically the output of a Support Vector Machine?" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15602" PostTypeId="1" CreationDate="2016-12-08T02:26:22.877" Score="5" ViewCount="12035" Body="&lt;p&gt;I want to train my neural net to overfit the training data. Should I just keep fitting my model with the same training data by using K-fold validation and setting epochs to infinity? Then, after it reaches nearly 100 percent prediction, I can just stop the training. Is it the right way to do it?&lt;/p&gt;&#xA;" OwnerUserId="26649" LastEditorUserId="26562" LastEditDate="2016-12-08T11:45:28.897" LastActivityDate="2018-04-20T22:11:35.177" Title="Training my neural network to overfit my training dataset" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15603" PostTypeId="2" ParentId="15602" CreationDate="2016-12-08T03:32:41.283" Score="7" Body="&lt;p&gt;If you &lt;em&gt;want&lt;/em&gt; to overfit, then yes you just need to keep fitting the training data through your network until you reach as close to zero training loss as possible (note that zero loss is stronger than 100% prediction, and will result in a greater amount of overfitting). There is no need to use cross validation. If your network is sufficiently complex, then it won't take very long to reach very high accuracy on the training data.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2016-12-08T21:25:37.497" LastActivityDate="2016-12-08T21:25:37.497" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15604" PostTypeId="2" ParentId="15602" CreationDate="2016-12-08T05:06:03.927" Score="1" Body="&lt;p&gt;It is very important to understand the difference between overfitting / underfitting and bias error / variance error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Three things to have high probability to overfitting:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Complex model with very large number of parameters , there is a relation between the number of samples used for training and the number of parameters. Read about optimism &lt;/li&gt;&#xA;&lt;li&gt;Few number of training samples &lt;/li&gt;&#xA;&lt;li&gt;Training for large number of iterations, until you have training error very close to zero&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It is very important to know that these things increase the probability to have overfitting, but another important factor is the problem itself. If you have two classes that are separable using your feature space, even with all these conditions you may not overfit, even if the training error is zero. &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="26562" LastEditDate="2018-04-20T22:11:35.177" LastActivityDate="2018-04-20T22:11:35.177" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15605" PostTypeId="2" ParentId="15577" CreationDate="2016-12-08T06:33:30.110" Score="1" Body="&lt;p&gt;This is actually a general problem with time-series data: you have some logic to implement based on one or more values in the series.  You always have two choices: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Feed the time series through some module that calculates as each data point arrives&lt;/li&gt;&#xA;&lt;li&gt;Use the &quot;spreadsheet method&quot; to calculate a series of columns eventually arriving at the goal&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The advantage of the first approach is you can use the same module to process your real-time data.  The advantage of the second approach is that it's very fast and usually easier to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you're already in a Spark Dataset, here's the strategy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate a speed column: $p_t - p_{t-1}$ where $p$ is the position&lt;/li&gt;&#xA;&lt;li&gt;Calculate a &quot;jump&quot; column: 1 if the speed is over a certain threshold, -1 if under, 0 otherwise&lt;/li&gt;&#xA;&lt;li&gt;Calculate a &quot;jumpsum&quot; column: the cumulative sum of the jump column&lt;/li&gt;&#xA;&lt;li&gt;Bad data will have a jumpsum of 1; filter them out&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here's how you do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.sql.expressions.Window&#xA;import org.apache.spark.sql.functions._&#xA;import org.apache.spark.sql.SparkSession&#xA;&#xA;val ss: SparkSession = SparkSession.builder.getOrCreate()&#xA;&#xA;// note the file must be on each executor in the same directory&#xA;val ds = ss.read&#xA;   .option(&quot;header&quot;, &quot;true&quot;)&#xA;   .option(&quot;inferSchema&quot;, &quot;true&quot;)&#xA;   .csv(&quot;file:///home/peter/data.csv&quot;)&#xA;&#xA;val w = Window.partitionBy().orderBy(&quot;datetime&quot;)&#xA;val threshold = 10&#xA;def jump(v: Double): Int = if (v &amp;gt; threshold) 1 else if (v &amp;lt; -threshold) -1 else 0&#xA;val sqlJump = udf(jump _)&#xA;&#xA;val cleanDS = ds&#xA;    .withColumn(&quot;speed&quot;, $&quot;position&quot; - lag($&quot;position&quot;, 1).over(w.rowsBetween(-1, -1)))&#xA;    .withColumn(&quot;jump&quot;, sqlJump($&quot;speed&quot;))&#xA;    .withColumn(&quot;jumpsum&quot;, sum($&quot;jump&quot;).over(w.rowsBetween(Long.MinValue, 0)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's what the output Dataset looks like (I didn't remove the bad rows so you can see the calculation):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+--------+-----+----+-------+&#xA;|datetime|position|speed|jump|jumpsum|&#xA;+--------+--------+-----+----+-------+&#xA;|       1|       1| null|null|   null|&#xA;|       2|       1|    0|   0|      0|&#xA;|       3|       1|    0|   0|      0|&#xA;|       4|       1|    0|   0|      0|&#xA;|       5|       1|    0|   0|      0|&#xA;|       6|       2|    1|   0|      0|&#xA;|       7|       1|   -1|   0|      0|&#xA;|       8|       1|    0|   0|      0|&#xA;|       9|      46|   45|   1|      1|&#xA;|      10|      45|   -1|   0|      1|&#xA;|      11|      48|    3|   0|      1|&#xA;|      12|      45|   -3|   0|      1|&#xA;|      13|       1|  -44|  -1|      0|&#xA;|      14|       2|    1|   0|      0|&#xA;|      15|       1|   -1|   0|      0|&#xA;+--------+--------+-----+----+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &quot;data.csv&quot; is just the first two columns of that Dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;datetime,position&#xA;1,1&#xA;2,1&#xA;3,1&#xA;4,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that's left to do is filter out &lt;code&gt;jumpsum === 1&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2016-12-08T06:33:30.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15606" PostTypeId="2" ParentId="15601" CreationDate="2016-12-08T06:43:40.957" Score="4" Body="&lt;p&gt;One standard way to obtain a &quot;probability&quot; out of a SVM is to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Platt_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Platt scaling&lt;/a&gt;.  See, e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Platt_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Wikipedia page&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/q/55072/2921&quot;&gt;this question on Stats.SE&lt;/a&gt;.  Platt scaling involves fitting a logistic regression model to predict the &quot;probability&quot;, based on the distance to the hyperplane.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-12-08T06:43:40.957" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15607" PostTypeId="2" ParentId="15565" CreationDate="2016-12-08T06:51:42.800" Score="2" Body="&lt;p&gt;I suggest you try building multiple networks: one network for building type (outputs &quot;house&quot;, &quot;apartment&quot;, or &quot;condominum&quot;), another network for build year, another for garage (yes vs no), and so on.  This keeps the number of classes small for each network, and allows each network to tune itself for the specific task it is focusing on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to avoid multiple networks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conceptually, share the first n-1 layers, and have a separate nth layer for each of these tasks.  In other words, the last layer is made wider, with elements for building type, elements for building year, and so on.  You end with multiple softmax elements.  A simple example would be one softmax for building type outputs (&quot;house&quot;, &quot;apartment&quot;, or &quot;condominum&quot;) and one softmax for build year (&quot;new&quot; or &quot;old&quot;).  Thus, you'd have 5 output wires.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can of course experiment with other splits, say, where the first n-3 layers are shared and then for the last 3 layers you have multiple different networks in parallel.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2016-12-08T16:57:00.173" LastActivityDate="2016-12-08T16:57:00.173" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15608" PostTypeId="2" ParentId="15556" CreationDate="2016-12-08T07:16:08.613" Score="0" Body="&lt;p&gt;I suggest you look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Item_response_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;item response theory&lt;/a&gt; and the &lt;a href=&quot;https://en.wikipedia.org/wiki/Rasch_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rasch model&lt;/a&gt;.  They construct a model that attempts to identify both (a) the proficiency/knowledge/ability of the user and (b) the level of difficulty of the question.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2016-12-08T07:16:08.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15609" PostTypeId="2" ParentId="15595" CreationDate="2016-12-08T08:15:31.137" Score="5" Body="&lt;p&gt;RBM's are an interesting beast. To answer your question, and to jog my memory on them, I'll derive RBMs and talk through the derivation. You mentioned that you're confused on the likelihood, so my derivation will be from the perspective of trying to maximize the likelihood. So let's begin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RBMs contain two different sets of neurons, visible and hidden, I'll denote them $v$ and $h$ respectively. Given a specific configuration of $v$ and $h$, we map it the probability space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(v,h) = \frac{e^{-E(v,h)}}{Z}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple things more to define. The surrogate function we use to map from a specific configuration to the probability space is called the energy function $E(v,h)$. The $Z$ constant is a normalization factor to ensure that we actually map to the probability space. Now let's get to what we're really looking for; the probability of a set of visible neurons, in other words, the probability of our data. &#xA;$$Z = \sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}$$&#xA;$$p(v)=\sum_{h \in H}p(v,h)=\frac{\sum_{h \in H}e^{-E(v,h)}}{\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although there are a lot of terms in this equation, it simply comes down to writing the correct probability equations. Hopefully, so far, this has helped you realize why we need energy function to calculate the probability, or what is done more usually the unnormalized probability $p(v)*Z$. The unnormalized probability is used because the partition function $Z$ is very expensive to compute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's get to the actual learning phase of RBMs. To maximize likelihood, for every data point, we have to take a gradient step to make $p(v)=1$. To get the gradient expressions it takes some mathematical acrobatics. The first thing we do is take the log of $p(v)$. We will be operating in the log probability space from now on in order to make the math feasible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\log(p(v))=\log[\sum_{h \in H}e^{-E(v,h)}]-\log[\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}]$$&#xA;Let's take the gradient with respect to the paremeters in $p(v)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{\partial \log(p(v))}{\partial \theta}=&amp;amp; &#xA;-\frac{1}{\sum_{h' \in H}e^{-E(v,h')}}\sum_{h' \in H}e^{-E(v,h')}\frac{\partial E(v,h')}{\partial \theta}\\ &amp;amp; + \frac{1}{\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}}\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}\frac{\partial E(v,h)}{\partial \theta}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I did this on paper and wrote the semi-final equation down as to not waste a lot of space on this site. I recommend you derive these equations yourself. Now I'll write some equations down that will help out in continuing our derivation. Note that: $Zp(v,h)=e^{-E(v,h')}$, $p(v)=\sum_{h \in H}p(v,h)$ and that $p(h|v) = \frac{p(v,h)}{p(h)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\frac{\partial log(p(v))}{\partial \theta}&amp;amp;=&#xA;-\frac{1}{p(v)}\sum_{h' \in H}p(v,h')\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}\\&#xA;\frac{\partial log(p(v))}{\partial \theta}&amp;amp;=&#xA;-\sum_{h' \in H}p(h'|v)\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there we go, we derived maximum likelihood estimation for RBM's, if you want you can write the last two terms via expectation of their respective terms (conditional, and joint probability).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes on energy function and stochasticity of neurons.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see above in my derivation, I left the definition of the energy function rather vague. And the reason for doing that is that many different versions of RBM implement various energy functions. The one that Hinton describes in the lecture linked above, and shown by @Laurens-Meeus is:&#xA;$$E(v,h)=−a^Tv−b^Th−v^TWh.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be easier to reason about the gradient terms above via the expectation form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial \log(p(v))}{\partial \theta}=&#xA;-\mathop{\mathbb{E}}_{p(h'|v)}\frac{\partial E(v,h')}{\partial \theta}+\mathop{\mathbb{E}}_{p(v',h')}\frac{\partial E(v',h')}{\partial \theta}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expectation of the first term is actually really easy to calculate, and that was the genius behind RBMs. By restricting the connection the conditional expectation simply becomes a forward propagation of the RBM with the visible units clamped. This is the so called wake phase in Boltzmann machines. Now calculating the second term is much harder and usually Monte Carlo methods are utilized to do so. Writing the gradient via average of Monte Carlo runs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial \log(p(v))}{\partial \theta}\approx&#xA;-\langle \frac{\partial E(v,h')}{\partial \theta}\rangle_{p(h'|v)}+\langle\frac{\partial E(v',h')}{\partial \theta}\rangle_{p(v',h')}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculating the first term is not hard, as stated above, therefore Monte-Carlo is done over the second term. Monte Carlo methods use random successive sampling of the distribution, to calculate the expectation (sum or integral). Now this random sampling in classical RBM's is defined as setting a unit to be either 0 or 1 based on its probability stochasticly, in other words, get a random uniform number, if it is less than the neurons probability set it to 1, if it is greater than set it to 0.  &lt;/p&gt;&#xA;" OwnerUserId="14779" LastEditorUserId="8560" LastEditDate="2017-01-06T17:56:55.787" LastActivityDate="2017-01-06T17:56:55.787" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="15610" PostTypeId="2" ParentId="15602" CreationDate="2016-12-08T10:00:12.383" Score="1" Body="&lt;p&gt;To deliberately over fit neural network , give stopping threshold and error metric to zero and let the neural network run for huge number of iteration until zero error is reached .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And do not perform any regularization , input or hidden layer drop out , cross validation &lt;/p&gt;&#xA;" OwnerUserId="26896" LastEditorUserId="26896" LastEditDate="2016-12-08T19:41:05.637" LastActivityDate="2016-12-08T19:41:05.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15611" PostTypeId="2" ParentId="15595" CreationDate="2016-12-08T10:32:25.587" Score="0" Body="&lt;p&gt;The answer of @Armen has gave myself a lot of insights. One question hasn't been answered however. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to maximize the probability (or likelihood) of the $v$. This is correlated to minimizing the energy function related to $v$ and $h$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E(v,h) = -a^{\mathrm{T}} v - b^{\mathrm{T}} h -v^{\mathrm{T}} W h$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our variables are $a$, $b$ and $W$, which have to be trained. I'm quite sure this training will be the ultimate goal of the RBM.&lt;/p&gt;&#xA;" OwnerUserId="25104" LastActivityDate="2016-12-08T10:32:25.587" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="15612" PostTypeId="1" CreationDate="2016-12-08T10:32:32.207" Score="1" ViewCount="2005" Body="&lt;p&gt;In ubuntu I can find &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;~/.keras/keras.json &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and change the backend&#xA;but in windows, I can not find this file. How can I  change keras backend in windows.&lt;/p&gt;&#xA;" OwnerUserId="26181" LastActivityDate="2016-12-08T13:05:43.743" Title="how to change keras backend in windows?" Tags="|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15614" PostTypeId="1" CreationDate="2016-12-08T12:51:27.467" Score="-4" ViewCount="690" Body="&lt;p&gt;I am coding an algorithm for clustering that is explained in &lt;a href=&quot;https://drive.google.com/open?id=0B7f2CpH4zpJIbjlGbmFyMDNNUWM&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper. According to the paper, the algorithm clusters by structural similarity and attribute similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In clustering with attribute similarity, I have problem. I believe that it says, first, to cluster with structural similarity and then partition the cluster with attribute similarity between the vertex. However, I can't understand how to find the attribute similarity between the vertex in a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is described &lt;a href=&quot;https://drive.google.com/open?id=0B7f2CpH4zpJIbjlGbmFyMDNNUWM&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; on page 4.&lt;/p&gt;&#xA;" OwnerUserId="26898" LastEditorUserId="29169" LastEditDate="2019-06-11T14:46:16.577" LastActivityDate="2019-06-11T14:46:16.577" Title="Similarity matrix in clustering" Tags="|data-mining|clustering|similarity|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-29T08:10:51.480" ContentLicense="CC BY-SA 4.0" />
  <row Id="15615" PostTypeId="2" ParentId="15612" CreationDate="2016-12-08T13:05:43.743" Score="2" Body="&lt;p&gt;You can create keras.json in the same way as on linux. The '&lt;code&gt;backend&lt;/code&gt;' key will have either '&lt;code&gt;tensorflow&lt;/code&gt;' or '&lt;code&gt;theano&lt;/code&gt;'. Place the file in &lt;code&gt;%USERPROFILE%/.keras&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-12-08T13:05:43.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15616" PostTypeId="2" ParentId="15614" CreationDate="2016-12-08T13:14:11.953" Score="2" Body="&lt;p&gt;A &lt;em&gt;vertex&lt;/em&gt; (the same as: &lt;em&gt;node&lt;/em&gt;) is a word used in &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph theory&lt;/a&gt; to denote an object. Also: an &lt;em&gt;edge&lt;/em&gt; joining two &lt;em&gt;vertices&lt;/em&gt;, denotes a relation between objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in this case, a similarity between objects A and B is the weight of an edge joining vertices corresponding to those objects.&lt;/p&gt;&#xA;" OwnerUserId="26141" LastActivityDate="2016-12-08T13:14:11.953" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15618" PostTypeId="2" ParentId="14957" CreationDate="2016-12-08T14:54:32.313" Score="0" Body="&lt;p&gt;I think you are looking for &lt;a href=&quot;http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-444.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Item Rank&lt;/a&gt;. It's a very simple algorithm which is used for ranking and cover your needs.&lt;/p&gt;&#xA;" OwnerUserId="26863" LastActivityDate="2016-12-08T14:54:32.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15619" PostTypeId="1" CreationDate="2016-12-08T15:13:26.143" Score="2" ViewCount="586" Body="&lt;p&gt;We have a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data AND active inventory, which can be discerned with the column titled &quot;File Type&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We suspect that data science applied to the set--such as a decision tree analysis or logistic regression, or some other machine learning model---can help us generate a value (i.e., probability score) for each product, that can be used as the main determinant evaluating the inventory. Each row in the file represents one product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is important to note that we have MANY products in our inventory, and very few of them tend to sell (only about 10% sell each year) and many of the products only have a single sale in the course of a year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The file contains historical sales data (identified with the column titled File_Type) along with current active inventory that is in need of evaluation (i.e., File Type = &quot;Active&quot;). The historical data shows sales for the past 6 months. The binary target (1 = sale, 0 = no sale in past six months) is likely the primary target that should drive the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other columns contain numeric and categorical attributes that we deem relevant to sales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that some of the historical sales SKUs are ALSO included in the active inventory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few comments about the attributes included, as we realize we may have some attributes that are unnecessary or may need to be explained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SKU_number: This is the unique identifier for each product. Order: Just a sequential counter. Can be ignored. SoldFlag: 1 = sold in past 6 mos. 0 = Not sold MarketingType = Two categories of how we market the product. This should probably be ignored, or better yet, each type should be considered independently. New_Release_Flag = Any product that has had a future release (i.e., Release Number &gt; 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data will be in front of the world's largest data science community. What questions do you want to see answered?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) What is the best model to use that will provide us with a probability estimate of a sale for each SKU? We are mainly interested in a relative unit that we can continuously update based on these attributes (and others that we add, as we are able).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Is it possible to provide a scored file (i.e., a probability score for each SKU in the file), and to provide an evaluation of the accuracy of the selected model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) What are the next steps we should take?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is possible for me to send you the file, please let me know. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks very much for any suggestions you may provide. &lt;/p&gt;&#xA;" OwnerUserId="26902" LastActivityDate="2016-12-08T15:13:26.143" Title="Sales Dataset to determine best model for predicting future sales" Tags="|machine-learning|logistic-regression|decision-trees|xgboost|azure-ml|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15620" PostTypeId="2" ParentId="8698" CreationDate="2016-12-08T16:26:27.860" Score="1" Body="&lt;p&gt;Use the option &lt;code&gt;--silent=True&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16211" LastActivityDate="2016-12-08T16:26:27.860" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15621" PostTypeId="2" ParentId="15598" CreationDate="2016-12-08T18:54:07.067" Score="3" Body="&lt;p&gt;The order of the files that populate file_list, is the same order X_test appears in, by row. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So just match the indices to correlate filename with prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X_test[0] ~ prediction[0] ~ file_list[0]&lt;/p&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2016-12-08T18:54:07.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15622" PostTypeId="2" ParentId="15597" CreationDate="2016-12-08T19:07:27.507" Score="4" Body="&lt;p&gt;What you are attempting to do is calculate the distance between two discrete probability density functions. The standard way to calculate this distance is via the Kullback-Leibler Divergence. It is defined as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$D(P||Q) = \sum_iP(i)*log(\frac{P(i)}{Q(i)})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be technically correct, this is not a metric in a mathematical sense because it is not symmetric and does not abide by the triangle inequality. But never the less it is a &quot;distance&quot; that is used a lot to compare distributions. From an information theoretical perspective, the KL divergence (given that you're using base 2 for the log) represents the number of bits it would take to encode distribution P as Q. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you would have to normalize each distribution to ensure that the sum is 1, and then calculate distance via the KL divergence. If you want a real metric you can use the Jensen-Shannon Divergence which is defined as:&#xA;$${{\rm {JSD}}}(P\parallel Q)={\frac  {1}{2}}D(P\parallel M)+{\frac  {1}{2}}D(Q\parallel M)$$$$&#xA;{\displaystyle M={\frac {1}{2}}(P+Q)} M={\frac  {1}{2}}(P+Q)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2016-12-08T19:07:27.507" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15626" PostTypeId="1" AcceptedAnswerId="15639" CreationDate="2016-12-08T21:41:52.877" Score="0" ViewCount="189" Body="&lt;p&gt;There are a lot of ml-algorithms which cannot directly deal with categorical variables. &#xA;A very common solution is to apply binary (dummy-) coding to still properly handle the categorical nature of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very often e.g. in &lt;code&gt;sk-learn&lt;/code&gt; or &lt;code&gt;apache-spark&lt;/code&gt; the actual dummy-coder can only handle numeric values. So label-encoding needs to be performed beforehand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a real live ml-scenario, the fitted model will encounter new and formerly not known data. Usually, such a label-encoder (string-indexer) for spark has the option to either skip (ignore) a row of data which contains any unknown value or to throw an error. If multiple values require coding this can lead to a big loss of &quot;new&quot; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any approaches which &quot;tolerate&quot; up to &lt;code&gt;x&lt;/code&gt; new values per row and still properly evaluate the fitted pipeline?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example for spark string-indexing + dummy-coding is shown below.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;val df = spark.createDataFrame(Seq(&#xA;  (0, &quot;a&quot;),&#xA;  (1, &quot;b&quot;),&#xA;  (2, &quot;c&quot;),&#xA;  (3, &quot;a&quot;),&#xA;  (4, &quot;a&quot;),&#xA;  (5, &quot;c&quot;)&#xA;)).toDF(&quot;id&quot;, &quot;category&quot;)&#xA;&#xA;val indexer = new StringIndexer()&#xA;  .setInputCol(&quot;category&quot;)&#xA;  .setOutputCol(&quot;categoryIndex&quot;)&#xA;  .fit(df)&#xA;val indexed = indexer.transform(df)&#xA;&#xA;val encoder = new OneHotEncoder()&#xA;  .setInputCol(&quot;categoryIndex&quot;)&#xA;  .setOutputCol(&quot;categoryVec&quot;)&#xA;val encoded = encoder.transform(indexed)&#xA;encoded.select(&quot;id&quot;, &quot;categoryVec&quot;).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;http://spark.apache.org/docs/latest/ml-features.html#onehotencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://spark.apache.org/docs/latest/ml-features.html#onehotencoder&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9053" LastActivityDate="2016-12-09T09:49:21.313" Title="Failure tolerant factor coding" Tags="|scikit-learn|apache-spark|binary|labels|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15627" PostTypeId="1" CreationDate="2016-12-08T21:43:41.223" Score="0" ViewCount="104" Body="&lt;p&gt;I want to learn machine learning methods, preferably high-performance methods, by R in the &lt;strong&gt;quickest&lt;/strong&gt; way possible. I am totally familiar with R but very less experience with ML packages. I know some resources like books but what I intend by opening this topic is to find the &lt;strong&gt;quickest&lt;/strong&gt; way possible. I also need to see some &lt;strong&gt;examples&lt;/strong&gt; in the tutorial and I &lt;strong&gt;DO NOT&lt;/strong&gt; need any &lt;strong&gt;theory&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26903" LastActivityDate="2016-12-09T02:02:19.383" Title="What is the best tutorial to quickly learning machine learning in R" Tags="|self-study|" AnswerCount="1" CommentCount="3" ClosedDate="2016-12-09T07:14:32.740" ContentLicense="CC BY-SA 3.0" />
  <row Id="15628" PostTypeId="2" ParentId="15595" CreationDate="2016-12-08T21:48:49.420" Score="5" Body="&lt;p&gt;In addition to the existing answers, I would like to talk about this energy function, and the intuition behind that a bit. Sorry if this is a bit long and physical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The energy function describes a so-called &lt;a href=&quot;https://en.wikipedia.org/wiki/Ising_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ising model&lt;/a&gt;, which is a model of ferromagnetism in terms of statistical mechanics / quantum mechanics. In statistical mechanics, we use a so-called Hamiltonian operator to describe the energy of a quantum-mechanical system. And a system always tries to be in the state with the lowest energy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the Ising model basically describes the interaction between electrons with a spin $\sigma_k$ of either +1 or -1, in presence of an external magnetic field $h$. The interaction between two electrons $i$ and $j$ is described by a coefficient $J_{ij}$. This Hamiltonian (or energy function) is&#xA;$$\hat{H} = \sum_{i,j} J_{ij} \sigma_i \sigma_j - \mu \sum_j h_j \sigma_j$$&#xA;where $\hat{H}$ denotes the Hamiltonian. &#xA;A standard procedure to get from an energy function to the probability, that a system is in a given state (i.e. here: a configuration of spins, e.g. $\sigma_1 = {+1}, \sigma_2 = {-1}, ...$) is to use the Boltzmann distribution, which says that at a temperature $T$, the probability $p_i$ of the system to be in a state $i$ with energy $E_i$ is given by&#xA;$$p_i = \frac{\exp(-E_i/kT)}{\sum_{i}\exp(-E_i/kt)}$$&#xA;At this point, you should recognize that these two equations are the exact same equations as in the videos by Hinton and the &lt;a href=&quot;https://datascience.stackexchange.com/a/15609/14906&quot;&gt;answer by Armen Aghajanyan&lt;/a&gt;. This leads us to the question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What does the RBM have to do with this quantum-mechanical model of ferromagnetism?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We need to use a final physical quantity: the entropy. As we know from thermodynamics, a system will settle in the state with the minimal energy, which also corresponds to the state with the maximal entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As introduced by Shanon in 1946, in information theory, the entropy $H$ can also be seen as a measure of the information content in $X$, given by the following sum over all possible states of $X$:&#xA;$$H(X) = -\sum_i P(x_i) \log P(x_i)$$&#xA;Now, the most efficient way to encode the information content in $X$, is to use a way that maximizes the entropy $H$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finally&lt;/strong&gt;, this is where we get back to RBMs: Basically, we want this RBM to encode as &lt;em&gt;much&lt;/em&gt; information as possible. So, as we have to &lt;em&gt;maximize&lt;/em&gt; the (information-theoretical) entropy in our RBM-System. As proposed by Hopfield in 1982, we can maximize the information-theoretical entropy exactly like the physical entropy: by modelling the RBM like the Ising model above, and use the same methods to minimize the energy. And that is why we need this strange energy function for in an RBM!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice mathematical derivation in &lt;a href=&quot;https://datascience.stackexchange.com/a/15609/14906&quot;&gt;Armen Aghajanyan's answer&lt;/a&gt; shows everything we need to do, to minimize the energy, thus maximizing entropy and storing / saving as much information as possible in our RBM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;PS: Please, dear physicists, forgive any inaccuracies in this engineer's derivation. Feel free to comment on or fix inaccuracies (or even mistakes). &lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-08T21:48:49.420" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15630" PostTypeId="1" AcceptedAnswerId="15633" CreationDate="2016-12-09T00:19:45.343" Score="21" ViewCount="30086" Body="&lt;p&gt;I am dealing with a highly unbalanced dataset so I used SMOTE  to resample it.&lt;/p&gt;&#xA;&lt;p&gt;After SMOTE resampling, I split the resampled dataset into training/test sets using the training set to build a model and the test set to evaluate it.&lt;/p&gt;&#xA;&lt;p&gt;However, I am worried that some data points in the test set might actually be jittered from data points in the training set (i.e. the information is leaking from the training set into the test set) so the test set is not really a clean set for testing.&lt;/p&gt;&#xA;&lt;p&gt;Does anyone have any similar experience? Does the information really leak from the training set into the test set? Or does SMOTE actually take care of this and we do not need to worry about it?&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="29169" LastEditDate="2021-07-09T21:23:34.987" LastActivityDate="2021-12-16T11:54:52.270" Title="Train/Test Split after performing SMOTE" Tags="|machine-learning|model-evaluations|class-imbalance|smote|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15631" PostTypeId="2" ParentId="15627" CreationDate="2016-12-09T02:02:19.383" Score="1" Body="&lt;p&gt;There is the Wikibook &lt;a href=&quot;https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Mining Algorithms In R&lt;/a&gt;  and some useful examples at &lt;a href=&quot;http://www.rdatamining.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;rdatamining.com&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26917" LastActivityDate="2016-12-09T02:02:19.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15632" PostTypeId="2" ParentId="14957" CreationDate="2016-12-09T03:12:27.030" Score="2" Body="&lt;p&gt;I think you should get started with &quot;&lt;strong&gt;learning to rank&lt;/strong&gt;&quot; , there are three solutions to deal with ranking problem .&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;point-wise&lt;/strong&gt; , learning the score for &lt;strong&gt;relevance&lt;/strong&gt; between each item&#xA;within list and specific user is your target . This can be accomplished as recommendation do . &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;pair-wise&lt;/strong&gt; , learning the &quot;&lt;strong&gt;relations&lt;/strong&gt;&quot;  between items within list ,&#xA;which respectively are beat loss or even , is your goal .&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;list-wise&lt;/strong&gt; , learning the &quot;&lt;strong&gt;effectiveness&lt;/strong&gt;&quot; of specific ranking list&#xA;for items is your object .&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2016-12-09T03:12:27.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15633" PostTypeId="2" ParentId="15630" CreationDate="2016-12-09T03:37:10.527" Score="28" Body="&lt;p&gt;When you use any sampling technique (specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you do the training, you use the test set (which contains only original samples) to evaluate.&lt;/p&gt;&#xA;&lt;p&gt;The risk if you use your strategy is having the original sample in training (testing) and the synthetic sample (that was created based on this original sample) in the test (training) set.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="29169" LastEditDate="2021-07-09T21:25:16.557" LastActivityDate="2021-07-09T21:25:16.557" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15635" PostTypeId="1" CreationDate="2016-12-09T07:10:44.187" Score="1" ViewCount="88" Body="&lt;p&gt;I get some metrics on validation data while training a model , and in my case the they are :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(0.25, 0.31, 0.46, 0.57, 0.65, 0.75, 0.77, 0.78, 0.84, 0.84, 0.85,&#xA;  0.84, 0.84, 0.84, 0.82, 0.8, 0.8, 0.79, 0.78, 0.77, 0.77, 0.77, 0.75, 0.74, 0.73, 0.73, 0.73, 0.73, 0.73, 0.73)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They can described like this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ExXnW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ExXnW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my view , the &lt;strong&gt;ideal result&lt;/strong&gt; should be like :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LN7ou.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LN7ou.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it a matter of overfitting&lt;/strong&gt; ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately , I tried few times to change the regular coefficients to avoid overfitting , and adjust learning rate coefficients to slow down , but it was still &quot;convex&quot; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I achieve the ideal result showed above&lt;/strong&gt; ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much appreciated if anyone would give me some constructive tips ?&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="9728" LastEditDate="2016-12-09T09:15:07.743" LastActivityDate="2022-11-19T05:45:03.117" Title="How can I fix this &quot;convex&quot; problem ? Is it just a matter of overfitting?" Tags="|machine-learning|supervised-learning|model-evaluations|regularization|overfitting|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15637" PostTypeId="2" ParentId="15635" CreationDate="2016-12-09T08:58:55.233" Score="1" Body="&lt;p&gt;Yes, what you are seeing is a classic case of overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You stated that you use a linear model such as logistic regression. To regularize these types of models, usually L1 and/or L2 regularization is applied. L1 regularization is simply $||W||_1$ and L2 is $||W||_2^2$ usually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another method is to alter the labels of the model in a specific way, which is a method of regularization I created (shameless plug). Here is the link to the paper: &lt;a href=&quot;https://arxiv.org/abs/1609.06693&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1609.06693&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="14779" LastActivityDate="2016-12-09T08:58:55.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15638" PostTypeId="1" AcceptedAnswerId="15641" CreationDate="2016-12-09T09:00:57.517" Score="1" ViewCount="437" Body="&lt;p&gt;I have a training set, I provide it (consider a data from training set) to the visible layer. Then the normal process is followed, i.e. Positive Phase-&gt; Negative Phase-&gt; Reconstruction of weights, bias units take place. Does it end here or should I have to iterate with the present visible unit value again (the process follows for few fixed iterations) ? Because it is not properly stated anywhere whether with each data of training set, it has to iterated again and again or with each data of training set.&lt;/p&gt;&#xA;" OwnerUserId="26872" LastActivityDate="2016-12-09T13:53:34.900" Title="# of iterations in Restricted Boltzmann Machine (RBM)" Tags="|machine-learning|neural-network|deep-learning|unsupervised-learning|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15639" PostTypeId="2" ParentId="15626" CreationDate="2016-12-09T09:49:21.313" Score="1" Body="&lt;p&gt;For a categorical variable, if the fitted model encounters a previously &quot;unseen&quot; category, i.e. it did not exist in the training set when the model was trained; then, you should skip that record. You could also opt to throw an error so that you're notified of the existence of new categories and can re-train the model based on that trigger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you skip the records with new categories, you would still be able to evaluate the pipeline successfully. That may be the preferred option for a fully automated production setup.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The one-hot encoding creates a new &quot;column&quot; of data for a new category and if this hasn't been used for training, the ML algorithm has no way of knowing how to use the new dummy variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your sample code, each of the categories are encoded into new variables e.g. &lt;code&gt;is_category_a&lt;/code&gt; = 0/1, &lt;code&gt;is_category_b&lt;/code&gt; = 0/1 , &lt;code&gt;is_category_c&lt;/code&gt; = 0/1, etc. If such a model is sent data with a new category &quot;d&quot;, then it would be encoded in another column called &lt;code&gt;is_category_d&lt;/code&gt; = 0/1 , but the model would ignore this column (or throw an error) since it doesn't expect its input matrix to contain &lt;code&gt;is_category_d&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume you've fit a linear regression model with the coefficients as:&#xA;$$ y = 1 + 2.is\_category\_a + 3.is\_category\_b + 4.is\_category\_c  $$&#xA;Now, when you try to evaluate a record with new category = &quot;d&quot;, then, the model is not able to use the new dummy variable since it doesn't have a coefficient for &lt;code&gt;is_category_d&lt;/code&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, so you should not &quot;tolerate&quot; new categories and process such records.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2016-12-09T09:49:21.313" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15641" PostTypeId="2" ParentId="15638" CreationDate="2016-12-09T13:53:34.900" Score="2" Body="&lt;p&gt;When you are unsure about how something (e.g. RBMs) should really be implemented, it is often useful to look at the code of others. G. Hinton himself has published a MATLAB-script (&lt;a href=&quot;http://www.cs.toronto.edu/~hinton/MatlabForSciencePaper.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) which demonstrates the training of an RBM. There, you can see that for each mini-batch, he does the positive phase, then the negative phase, and finally updates the weights - and that's it. So he &lt;em&gt;doesn't iterate&lt;/em&gt; between the visible and hidden states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this is not the full truth: for the weight updates we need to know the probability $p(v,h)$. This is very complicated to calculate, as it would contain a sum over all possible states of the RBM. There is a &quot;mathematical trick&quot; called Gibbs sampling: it allows us to iterate back and forth between visible and hidden units to calculate this probability $p(v,h)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But: for the result to be correct, we have to iterate &lt;strong&gt;forever&lt;/strong&gt;, which is not really practical. So what Hinton proposed is to iterate for only 1 step instead (this is $CD_1$), so he only goes back-and-forth once. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, you can also iterate any number of times $k$, which is denoted by $CD_k$. &#xA;While for Hinton's $CD_1$, you would do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;visible --&amp;gt; hidden --&amp;gt; visible&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a $CD_3$, you would iterate from visible to hidden and back, three times:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;visible --&amp;gt; hidden --&amp;gt; visible --&amp;gt; hidden --&amp;gt; visible --&amp;gt; hidden --&amp;gt; visible&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And just to be clear: you run this iteration for &lt;em&gt;every&lt;/em&gt; data point in the training set. Actually, you usually make so-called mini-batches of maybe 10 data points, which you run at the same time. Then you calculate the average weight update from this batch. But then, you do this iteration again, and again, and again, until you have finished your training.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-12-09T13:53:34.900" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15642" PostTypeId="1" AcceptedAnswerId="15648" CreationDate="2016-12-09T13:53:54.977" Score="0" ViewCount="75" Body="&lt;p&gt;Having a schema which the majority of the values are IDs. Like this example (this isn't my real data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID   SCHOOL_ID   CLASSE_ID   STUDENT_ID   GRADE&#xA;1       1            1           1          17&#xA;2       1            1           2          10&#xA;3       1            1           3          4&#xA;4       1            2           19          11&#xA;5       1            2           21         8&#xA;...    ...          ...         ...         ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which one of this can be a better approach to detect outliers using SQL:&#xA;- Standard Deviation + Average&#xA;- Try to implement an clustering algorithm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a little bit confusing about this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="25824" LastActivityDate="2016-12-09T21:02:38.847" Title="Outliers Approach" Tags="|preprocessing|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15644" PostTypeId="2" ParentId="15587" CreationDate="2016-12-09T14:39:57.737" Score="2" Body="&lt;p&gt;This is an expanded answer based on my comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, there are many interesting &lt;strong&gt;highly non-linear classifiers&lt;/strong&gt; besides decision trees. Most widely used are &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural networks&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVMs&lt;/a&gt;. (Nitpick but SVMs can be seen as neural networks.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within the &lt;strong&gt;world of decision trees,&lt;/strong&gt; there are things like decision trees that have linear regressions &lt;a href=&quot;https://stats.stackexchange.com/questions/78563/regression-tree-algorithm-with-linear-regression-models-in-each-leaf&quot;&gt;in the nodes.&lt;/a&gt; However, these are very slow to train. The problem is that you can no longer take advantage of the fact that decision trees entropy or gini score can be calculated incrementally as the threshold $t$ is varied (where $t$ is the right-hand part of the node rule, $x_i &amp;lt; t$). This trick can not be used for these trees. There are of course many other rule induction algorithms besides decision trees, which are not as step-like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, &lt;strong&gt;usually ensembles are used.&lt;/strong&gt; Ensembles are models made of several models, which then &quot;vote&quot; on the result. You can use them with any class of models, but they are particularly excellent when combined with decision trees because:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Decision trees can have very different results if the sample is slightly different. Ensembles fix that;&lt;/li&gt;&#xA;&lt;li&gt;Decision trees decision boundaries are orthogonal to the features, which have &quot;step&quot; effects;&lt;/li&gt;&#xA;&lt;li&gt;Decision trees have to be pruned, while some ensembles can be used without pruning;&lt;/li&gt;&#xA;&lt;li&gt;Most winners of Kaggle and other data mining competitions have used decision trees ensembles;&lt;/li&gt;&#xA;&lt;li&gt;Ensembles give you probabilities if you need them; probabilities from decision trees are highly dependent on how you prune them (and unpruned decision trees cannot give you probabilities at all).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_iris.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;See here the results&lt;/a&gt; of some decision trees ensembles:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9s0OV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9s0OV.png&quot; alt=&quot;decision trees ensembles&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two types of ensembles widely used:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest:&lt;/a&gt; where each decision tree is trained independently.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient Boosting Trees:&lt;/a&gt; these are based on AdaBoost. The idea is that the next tree is trained by weighting more strongly the observations that the previous trees have not been able to qualify correctly.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Usually, Kaggle winners use gradient boosting trees, especially using the &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost package.&lt;/a&gt; But they have more hyperparameters to tweak, so if you don't have much time to tweak them, just go with a random forest. Gradient boosting trees are more ticky to get right, and the results can be different if you change parameters like &lt;code&gt;n_estimators&lt;/code&gt;. Random forests, on the other hand, tend to the expected behavior when &lt;code&gt;n_estimators&lt;/code&gt;$\rightarrow\infty$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, gradient boosting trees can overfit, albeit I think it's rare. (Random forests can also overfit, but it is because the underlying estimator is overfitting. They tend to the average behavior as you increase &lt;code&gt;n_estimators&lt;/code&gt;. Tell me if this isn't clear.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; &lt;code&gt;n_estimators&lt;/code&gt; in sklearn is sometimes very low, and you should increment it. In real problems, people use thousands of trees.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-12-09T15:13:29.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15648" PostTypeId="2" ParentId="15642" CreationDate="2016-12-09T21:02:38.847" Score="0" Body="&lt;p&gt;Student ID (and ID) doesn't make sense as a column to cluster on because it's not continuous, and is unique and high cardinality too so isn't even usable as a categorical value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering school and class ID could make a little sense if converted to a one-hot encoded value, but it's also probably high cardinality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you may need to question whether those are even meaningful dimensions to cluster on. You might just drop them.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-12-09T21:02:38.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15652" PostTypeId="2" ParentId="15309" CreationDate="2016-12-09T23:15:04.987" Score="2" Body="&lt;p&gt;What is the x and y axis of this scatter plot? If precision and recall are on these axes, then the range of both axes would be 0 - 1. I'm assuming one point would then represent one model, in this case you'd want to select the model on the top right, I.e. where precision and recall are both high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the concepts behind precision and recall, wiki has a good write up &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Precision_and_recall&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-09T23:15:04.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15653" PostTypeId="2" ParentId="15309" CreationDate="2016-12-09T23:42:52.390" Score="3" Body="&lt;p&gt;In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a great explanation &lt;a href=&quot;https://sanchom.wordpress.com/2011/09/01/precision-recall/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, using the classification of images of airplanes and geese as an example.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A good way to characterize the performance of a classifier is to look at how precision and recall change as you change the threshold. A good classifier will be good at ranking actual airplane images near the top of the list, and be able to retrieve a lot of airplane images before retrieving any geese: its precision will stay high as recall increases. A poor classifier will have to take a large hit in precision to get higher recall. Usually, a publication will present a precision-recall curve to show how this tradeoff looks for their classifier.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="24968" LastActivityDate="2016-12-09T23:42:52.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15655" PostTypeId="1" CreationDate="2016-12-10T01:02:58.117" Score="1" ViewCount="826" Body="&lt;p&gt;I am a newbie to datascience. I installed the Jupyter notebook and was trying to create model for the kaggle titanic dataset. Below is the code I wrote - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn.metrics import roc_auc_score&#xA;import pandas as pd&#xA;X = pd.read_csv(&quot;train.csv&quot;)&#xA;y = X[&quot;Survived&quot;]&#xA;X[&quot;Age&quot;].fillna(X.Age.mean(),inplace = True)&#xA;numeric_variables = list(X.dtypes[X.dtypes != &quot;object&quot;].index)&#xA;model = RandomForestRegressor(n_estimators=100, oob_score = True, random_state = 42)&#xA;model.fit(X[numeric_variables],y)&#xA;model.oob_score_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the &lt;code&gt;oob_score_&lt;/code&gt; output as 1.0 while the actual output as per tutorial should be 0.13420....&#xA;After that I did &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_oob = model.oob_prediction_&#xA;print &quot;c,stat:&quot;, roc_auc_score(y,y_oob)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of this as well I am getting 1.0. Is there something wrong in my code or setup of Jupyter?&lt;/p&gt;&#xA;" OwnerUserId="26953" LastEditorUserId="8501" LastEditDate="2016-12-13T09:18:54.093" LastActivityDate="2016-12-13T09:18:54.093" Title="sklearn RandomForestRegressor oob_score_ looks wrong?" Tags="|python|scikit-learn|random-forest|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15656" PostTypeId="1" AcceptedAnswerId="15667" CreationDate="2016-12-10T01:43:09.343" Score="24" ViewCount="15295" Body="&lt;p&gt;Say I have done transfer learning on a pre-trained network to recognize 10 objects. How can I add a &lt;span class=&quot;math-container&quot;&gt;$11^{th}$&lt;/span&gt; item that the network can classify without losing all the 10 categories I already trained and the information from the original pre-trained model? A friend told me that active research is going on in this field, but I cannot find any relevant papers or a names  to search for?&lt;/p&gt;&#xA;" OwnerUserId="26955" LastEditorUserId="29169" LastEditDate="2021-03-11T19:11:23.307" LastActivityDate="2021-03-11T19:11:23.307" Title="How to add a new category to a deep learning model?" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="15657" PostTypeId="1" AcceptedAnswerId="15669" CreationDate="2016-12-10T02:56:59.787" Score="2" ViewCount="1448" Body="&lt;p&gt;In RBM, in the positive phase for updating the hidden layer(which should also be binary), [Acually consider a node of h1 ∈ H(hidden layer vector)] to make h1 a binary number we compute the probability of turning on a hidden unit by operating activation function over total input (after the activation function operation, we would be getting values in the range between 0 and 1, since activation function I am using - sigmoid). &#xA;My doubt is that how do we make it binary by leveraging the probability computed. I don't think if P&gt;=0.5, make it 1 else 0 is a proper method to work on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By few literature reviews,&#xA;I found this &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;document&lt;/a&gt; (by Hinton), in section 3.1: he has stated &quot;the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1&quot;. What does this actually mean? And also in this &lt;a href=&quot;http://fisheyefocus.com/fisheyeview/?p=443&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;, they say &quot;Then the jth unit is on if upon choosing s uniformly distributed random number between 0 and 1 we find that its value is less than sig[j]. Otherwise it is off.&quot; I actually didn't get this. Whether the random number generated is same for all h ∈ H ? Another query is, what about the random in next sampling iteration?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw this &lt;a href=&quot;https://youtu.be/FJ0z3Ubagt4?list=PLehuLRPyt1Hyi78UOkMPWCGRxGcA9NVOE&amp;amp;t=1772&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;. Just watch the video from that point as per the link. How do you get that sampled number? Whether we have to just run rand() in Matlab and obtain it? Should it would be different for each h(i) (oh nooo! I don't think the machine will learn properly)? Whether the random number should be different for each iteration or the same random number can be used for all iteration to compare?&lt;/p&gt;&#xA;" OwnerUserId="26872" LastActivityDate="2016-12-10T18:25:40.710" Title="How hidden layer is made binary in Restricted Boltzmann Machine (RBM)?" Tags="|machine-learning|neural-network|deep-learning|unsupervised-learning|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15660" PostTypeId="1" CreationDate="2016-12-10T06:38:20.213" Score="2" ViewCount="106" Body="&lt;p&gt;I want to make the following Python data-processing code more efficient by replacing for loops. Is there any way to vectorize code like this?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have a DataFrame object &lt;code&gt;df&lt;/code&gt; that looks somewhat like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names number&#xA;bob 5&#xA;sara 10&#xA;bob 8&#xA;foo 12&#xA;moo 16&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I want to subset the DataFrame to find out all the rows associated with each name, and then perform an operation on &lt;code&gt;number&lt;/code&gt;. This is what I am doing now:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for myName in set(df['names']):    &#xA;    nameSubset = df.loc[df['names']==myName]    &#xA;    operation(nameSubset['number'], **args) &#xA;&#xA;'''Basically,perform an operation on the `number` column of nameSubset.'''&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there any way to make this code run faster? Theoretically, this could be made faster if, instead of running through each myName at a time, the computer could process several myNames at any given moment. I'm not sure how to vectorize/parallelize my code to make this happen though.&lt;/p&gt;&#xA;" OwnerUserId="26956" LastEditorUserId="8501" LastEditDate="2016-12-13T09:18:39.137" LastActivityDate="2017-07-11T15:10:55.290" Title="Vectorizing/Parallelizing DataFrame indexing" Tags="|python|pandas|performance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15661" PostTypeId="2" ParentId="15656" CreationDate="2016-12-10T09:19:23.503" Score="2" Body="&lt;p&gt;Your network topology might look different, but in the very end, your pre-trained network has a layer, which handles the recognition of 10 original classes. The easiest (and working) trick to introduce the 11th, 12th.. nth class, is to use all the layers &lt;em&gt;before&lt;/em&gt; the last as granted and add an additional layer (in a new model, or as a parallel one) that will also sit on top of all but last layers, will be looking alike to the 10class layer (which is most probably matmul of dense layer and a matrix of shape &lt;code&gt;[len(dense layer), 10]&lt;/code&gt; with optional bias).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your new layer would be a matmul layer with shape &lt;code&gt;[len(dense layer), len(new classes)]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without access to original training data, you would have two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Freeze all the weights in original layers by allowing &quot;new&quot; model to optimize only new weights. That will give you exactly same predictive power for original 10 classes and might give OK performance for new ones.&lt;/li&gt;&#xA;&lt;li&gt;Train whole network at once (by propagating error of new classes), which might be working for new class(es), but you will end up with ineffective original solution for 10 classes (since weights will be changed for the lower classes and final layer won't be updated to match those changes).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Although, given you have access to original training data, you can easily add new class to the original network and re-train it to support 11 classes out of the box.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2016-12-10T09:19:23.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15662" PostTypeId="1" CreationDate="2016-12-10T10:05:38.083" Score="1" ViewCount="4761" Body="&lt;p&gt;I try to train convolutional neural net for a classification problem. However, my neural net is not learning anything. It guesses only two labels and ignores the rest. Even though I try to train to overfit my neural net, the loss function is not decreasing at all. My I try to make my network go as deep as 12 layers of the convolutional neural net in order to overfit the subsampling data. However, it is not working. Do you have any suggestion what to look?&lt;/p&gt;&#xA;" OwnerUserId="26649" LastActivityDate="2020-09-27T10:05:52.887" Title="My Neural Network is not learning anything" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15667" PostTypeId="2" ParentId="15656" CreationDate="2016-12-10T16:44:52.797" Score="19" Body="&lt;p&gt;If this is just a one-time case, you can simply re-train the neural network. If you frequently have to add new classes, then this is a bad idea. What you want to do in such cases is called content-based image retrieval (CBIR), or simply image retrieval or visual search. I will explain both cases in my answer below.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;One-time case&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If this just happens once - you forgot the 11th class, or your customer changed his/her mind - but it &lt;em&gt;won't happen again&lt;/em&gt;, then then you can simply an 11th output node to the last layer. Initialize the weights to this node randomly, but use the weights you already have for the other outputs. Then, just train it as usual. It might be helpful to fix some weights, i.e. don't train these. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An extreme case would be to only train the new weights, and leave all others fixed. But I am not sure whether this will work that well - might be worth a try.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Content-based image retrieval&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Consider the following example: you are working for a CD store, who wants their customers to be able to take a picture of an album cover, and the application shows them the CD they scanned in their online store. &#xA;In that case, you would have to re-train the network for every new CD they have in the store. That might be 5 new CDs each day, so re-training the network that way is not suitable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is to train a network, which maps the image into a feature space. Each image will be represented by a descriptor, which is e.g. a 256-dimensional vector. You can &quot;classify&quot; an image by calculating this descriptor, and comparing it to your database of descriptors (i.e. the descriptors of all CDs you have in your store). The closest descriptor in the database wins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you train a neural network to learn such a descriptor vector? That is an active field of research. You can find recent work by searching for keywords like &quot;image retrieval&quot; or &quot;metric learning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, people usually take a pre-trained network, e.g. VGG-16, cut off the FC layers, and use the final convolutional as your descriptor vector. You can further train this network e.g. by using a siamese network with triplet loss.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-12-10T16:44:52.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15668" PostTypeId="2" ParentId="15660" CreationDate="2016-12-10T17:02:17.277" Score="1" Body="&lt;p&gt;Is that what you want?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [261]: df&#xA;Out[261]:&#xA;  names  number&#xA;0   bob       5&#xA;1  sara      10&#xA;2   bob       8&#xA;3   foo      12&#xA;4   moo      16&#xA;&#xA;In [262]: def my_op(ser):&#xA;     ...:     return ser.sum()&#xA;     ...:&#xA;&#xA;In [263]: df.groupby('names').agg({'number':my_op})&#xA;Out[263]:&#xA;       number&#xA;names&#xA;bob        13&#xA;foo        12&#xA;moo        16&#xA;sara       10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2016-12-10T17:02:17.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15669" PostTypeId="2" ParentId="15657" CreationDate="2016-12-10T18:25:40.710" Score="1" Body="&lt;p&gt;As you correctly say, we calculate the probability of a hidden unit $h_j$&#xA;being one and then make it binary. That probability is given by&#xA;$$p(h_j=1) = \sigma\left(b_j + \sum_{i=1}^V w_{ij}v_i \right)$$&#xA;where $\sigma$ is the sigmoid function, $b_j$ is the bias of hidden unit $h_j$, $V$ is the number of visible units, $v_i$ is the (binary!) state of visible unit $i$, and $w_{ij}$ are the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, your MATLAB code for obtaining the probabilities &lt;code&gt;hidden_probs&lt;/code&gt; is something like this (we write the sum implicitly by making a matrix multiplication):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hidden_probs = sigmoid(hidden_bias + data * weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, we have the probability $p(h_j=1)$ for each hidden unit $j \in [1,H]$. Now, this is only a probability. And we need a binary number, either &lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;. So the only thing we can do is &lt;em&gt;pick a random sample&lt;/em&gt; from the probability distribution of $h_j$, which is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bernoulli_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bernoulli distribution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As all hidden units are independent, we need to get one sample for &lt;em&gt;each&lt;/em&gt; hidden unit independently. And also, in each training step, we need to draw new samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To draw these samples from the Bernoulli distribution, you can use the built-in functions of e.g. MATLAB (&lt;a href=&quot;https://mathworks.com/help/stats/binornd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;binornd&lt;/code&gt;&lt;/a&gt;) or Python (&lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.binomial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;numpy.random.binomial&lt;/code&gt;&lt;/a&gt;). Note that these functions are to sample from a binomial distribution, but the Bernoulli distribution is just a special case of the binomial distribution with &lt;code&gt;N=1&lt;/code&gt;. In MATLAB, that would be something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hidden_states = binornd(1, hidden_probs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which would create vector &lt;code&gt;hidden_states&lt;/code&gt; which contains either &lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;, drawn randomly for each probability in &lt;code&gt;hidden_probs&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you probably have noticed, &lt;em&gt;nobody does that!&lt;/em&gt; E.g. describes it in his &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical Guide to Training RBMs&lt;/a&gt;, as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is exactly what Hinton does in his &lt;a href=&quot;http://www.cs.toronto.edu/~hinton/MatlabForSciencePaper.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RBM code&lt;/a&gt;: he gets a random number for &lt;em&gt;each&lt;/em&gt; hidden unit using &lt;code&gt;rand&lt;/code&gt;, i.e. randomly sampled from the uniform distribution between &lt;code&gt;[0,1]&lt;/code&gt;. He then does the comparison:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hidden_states = hidden_probs &amp;gt; rand(1, H)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is equivalent to using &lt;code&gt;binornd&lt;/code&gt;, but is probably faster. For example to generate a random number that is &lt;code&gt;1&lt;/code&gt; with &lt;code&gt;p=0.9&lt;/code&gt;, you get a random number from &lt;code&gt;[0,1]&lt;/code&gt;. Now, in 90% of the cases, this random number is smaller than 0.9, and in 10% of the cases it is larger than 0.9. So to get a random number that is &lt;code&gt;1&lt;/code&gt; with &lt;code&gt;p=0.9&lt;/code&gt;, you can call &lt;code&gt;0.9 &amp;gt; rand(1)&lt;/code&gt; - which is exactly what they do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;tl;dr:&lt;/strong&gt; Pick a new random number from the range &lt;code&gt;[0,1]&lt;/code&gt; for each hidden unit in each iteration. Compare it to your probability with &lt;code&gt;hidden_probs &amp;gt; rand(1,H)&lt;/code&gt; to make it binary.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2016-12-10T18:25:40.710" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15670" PostTypeId="1" CreationDate="2016-12-10T19:51:29.767" Score="3" ViewCount="2192" Body="&lt;p&gt;I have training data that is classified into 2 categories: &lt;strong&gt;X&lt;/strong&gt; and &lt;strong&gt;Not X&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each piece of training or experimental data has a variable number of boolean features. Each piece of data may have ~100 features, but the total number of possible boolean features may be in the tens or hundreds of thousands. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train a classifier so that given an observation (which has a set of boolean features), it will output the probability that it is &lt;strong&gt;X&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you recommend which ML algorithms could be used for a large number of sparse boolean features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Follow-up question:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have a large amount of training data that would be best described as &lt;strong&gt;Maybe X&lt;/strong&gt; (I haven't verified is &lt;strong&gt;X&lt;/strong&gt;, but there's a good chance it is &lt;strong&gt;X&lt;/strong&gt;). I could further segment those &lt;strong&gt;Maybe X&lt;/strong&gt;'s into &lt;em&gt;Likely X&lt;/em&gt;, &lt;em&gt;Maybe X&lt;/em&gt;, &lt;em&gt;Probably Not X&lt;/em&gt;. Is there a way I can train with &lt;strong&gt;Maybe X&lt;/strong&gt; or it's subclassifications? In the end, what I still only care for is the probability that an observation is &lt;strong&gt;X&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26967" LastActivityDate="2018-02-11T15:19:44.720" Title="Which classifier should I use for sparse boolean features?" Tags="|machine-learning|classification|data-mining|data-cleaning|supervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15671" PostTypeId="1" AcceptedAnswerId="15688" CreationDate="2016-12-10T20:51:52.123" Score="5" ViewCount="580" Body="&lt;p&gt;In the case my question is not clear, I am talking about the patterns that are detected in each of the layers of an image-trained Convolutional Neural Network (CNN). Take the following image as an example (by Honglak Lee). I think I understand the concept: different layers start coding for different features with increasing complexity. The first layers code for edges, the middle layers code for simple features (e.g. nose, eyes), and the later layers code for whole faces.&#xA;However, I do not see the equivalence betweence each of the patterns in the picture and the network elements. Do each pattern inside a pattern-class correspond to one neuron of that layer? How are these cool pattern detection pictures even plotted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KJ3UF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KJ3UF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="29169" LastEditDate="2021-02-16T15:28:18.247" LastActivityDate="2021-02-16T15:28:18.247" Title="How are per-layer-detected-patterns in a trained CNN plotted?" Tags="|deep-learning|visualization|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15673" PostTypeId="2" ParentId="10868" CreationDate="2016-12-10T22:42:55.340" Score="0" Body="&lt;p&gt;Which version are you running? Does the same happen on the 0.2.2 version released a few days ago (note that for 0.2.2 you need also the newest version of Orange; i.e 3.3.9)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like you're not doing anything wrong but rather that this is a bug. If you could paste the whole error log, perhaps also a schema and a sample of the data if possible, along with the version of Orange and Orange3-Text that you are running I promise to take a look and fix the bug.&lt;/p&gt;&#xA;" OwnerUserId="15685" LastActivityDate="2016-12-10T22:42:55.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15674" PostTypeId="2" ParentId="15586" CreationDate="2016-12-10T23:22:53.387" Score="0" Body="&lt;p&gt;If you don't have labeled data, Word2vec can be one solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps have a look at &lt;a href=&quot;https://github.com/facebookresearch/fastText&quot; rel=&quot;nofollow noreferrer&quot;&gt;FastText unsupervised&lt;/a&gt; (recently released by Facebook) which is similar but uses character n-grams. Character n-grams helps to take into account the morphology, is more robust to typos and make it possible to compute vectors for unknown words.&lt;/p&gt;&#xA;" OwnerUserId="12558" LastActivityDate="2016-12-10T23:22:53.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15675" PostTypeId="2" ParentId="15601" CreationDate="2016-12-10T23:35:47.383" Score="2" Body="&lt;p&gt;As @D.W. mentioned, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Platt_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Platt scaling&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, keep in mind that the probability estimates may be inconsistent with the prediction.&lt;br&gt;&#xA;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/40169b94eb0637b8334bea38072294bc32b17a2d/doc/modules/svm.rst#scores-and-probabilities&quot; rel=&quot;nofollow noreferrer&quot;&gt;From sklearn documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Needless to say, the cross-validation involved in Platt scaling is an&#xA;  expensive operation for large datasets. In addition, the probability&#xA;  estimates may be inconsistent with the scores, in the sense that the&#xA;  &quot;argmax&quot; of the scores may not be the argmax of the probabilities.&#xA;  (E.g., in binary classification, a sample may be labeled by predict as&#xA;  belonging to a class that has probability &amp;lt;½ according to&#xA;  predict_proba.) Platt's method is also known to have theoretical&#xA;  issues. If confidence scores are required, but these do not have to be&#xA;  probabilities, then it is advisable to set probability=False and use&#xA;  decision_function instead of predict_proba.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="12558" LastActivityDate="2016-12-10T23:35:47.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15676" PostTypeId="2" ParentId="8840" CreationDate="2016-12-11T04:52:04.210" Score="0" Body="&lt;p&gt;The number of features should take into account the &lt;a href=&quot;http://fastml.com/how-much-data-is-enough/&quot; rel=&quot;nofollow noreferrer&quot;&gt;number of examples&lt;/a&gt; you have in the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model training takes too long, have you considered principal components analysis or &lt;a href=&quot;http://machinelearningmastery.com/an-introduction-to-feature-selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature selection&lt;/a&gt;?  Both techniques can cut down the number of features into a more manageable size without throwing away information which describes the target variable.&lt;/p&gt;&#xA;" OwnerUserId="24968" LastActivityDate="2016-12-11T04:52:04.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15678" PostTypeId="1" AcceptedAnswerId="15680" CreationDate="2016-12-11T11:43:55.263" Score="-2" ViewCount="1117" Body="&lt;p&gt;I am participating in a kaggle competition. I am planning to use the XGBoost package (in R). I read the XGBoost documentation and understood the basics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain how is feature engineering done using XGBoost?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example for explanation would be of great help.&lt;/p&gt;&#xA;" OwnerUserId="26976" LastActivityDate="2016-12-11T14:27:34.503" Title="Feature engineering using XGBoost" Tags="|xgboost|feature-engineering|kaggle|" AnswerCount="1" CommentCount="4" ClosedDate="2020-07-29T08:03:20.020" ContentLicense="CC BY-SA 3.0" />
  <row Id="15680" PostTypeId="2" ParentId="15678" CreationDate="2016-12-11T14:27:34.503" Score="1" Body="&lt;p&gt;It turns out that the question I asked is incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially feature engineering is done, then xgboost is used to build a model out of it. If we not satisfied with the model's performance we can go back to feature engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks Dan Levin for the explanation.&lt;/p&gt;&#xA;" OwnerUserId="26976" LastActivityDate="2016-12-11T14:27:34.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15682" PostTypeId="2" ParentId="694" CreationDate="2016-12-11T21:03:29.573" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://neupy.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;NeuPy&lt;/a&gt; is a Python library for Artificial Neural Networks. NeuPy supports many different types of Neural Networks from a simple perceptron to deep learning models.&lt;/p&gt;&#xA;" OwnerUserId="9634" LastActivityDate="2016-12-11T21:03:29.573" CommentCount="0" CommunityOwnedDate="2016-12-12T15:58:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="15683" PostTypeId="1" CreationDate="2016-12-11T21:12:28.930" Score="3" ViewCount="5487" Body="&lt;p&gt;I am starting with the generic TensorFlow example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To classify my data I need to use multiple labels (ideally multiple &lt;code&gt;softmax&lt;/code&gt; classifiers) on the final layer, because my data carries multiple independent labels (sum of probabilities is not 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically in the &lt;code&gt;retrain.py&lt;/code&gt; these lines in &lt;code&gt;add_final_training_ops()&lt;/code&gt; add the final tensor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;final_tensor = tf.nn.softmax(logits, name=final_tensor_name)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = tf.nn.softmax_cross_entropy_with_logits(&#xA;      logits, ground_truth_input)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there already a generic classifier in TensorFlow? If not, how to achieve multilevel classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;add_final_training_ops()&lt;/code&gt; from &lt;code&gt;tensorflow/examples/image_retraining/retrain.py&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def add_final_training_ops(class_count, final_tensor_name, bottleneck_tensor):&#xA;&#xA;  with tf.name_scope('input'):&#xA;    bottleneck_input = tf.placeholder_with_default(&#xA;        bottleneck_tensor, shape=[None, BOTTLENECK_TENSOR_SIZE],&#xA;        name='BottleneckInputPlaceholder')&#xA;&#xA;    ground_truth_input = tf.placeholder(tf.float32,&#xA;                                        [None, class_count],&#xA;                                        name='GroundTruthInput')&#xA;&#xA;  layer_name = 'final_training_ops'&#xA;  with tf.name_scope(layer_name):&#xA;    with tf.name_scope('weights'):&#xA;      layer_weights = tf.Variable(tf.truncated_normal([BOTTLENECK_TENSOR_SIZE, class_count], stddev=0.001), name='final_weights')&#xA;      variable_summaries(layer_weights)&#xA;    with tf.name_scope('biases'):&#xA;      layer_biases = tf.Variable(tf.zeros([class_count]), name='final_biases')&#xA;      variable_summaries(layer_biases)&#xA;    with tf.name_scope('Wx_plus_b'):&#xA;      logits = tf.matmul(bottleneck_input, layer_weights) + layer_biases&#xA;      tf.summary.histogram('pre_activations', logits)&#xA;&#xA;  final_tensor = tf.nn.softmax(logits, name=final_tensor_name)&#xA;  tf.summary.histogram('activations', final_tensor)&#xA;&#xA;  with tf.name_scope('cross_entropy'):&#xA;    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(&#xA;      logits, ground_truth_input)&#xA;    with tf.name_scope('total'):&#xA;      cross_entropy_mean = tf.reduce_mean(cross_entropy)&#xA;  tf.summary.scalar('cross_entropy', cross_entropy_mean)&#xA;&#xA;  with tf.name_scope('train'):&#xA;    train_step = tf.train.GradientDescentOptimizer(FLAGS.learning_rate).minimize(&#xA;        cross_entropy_mean)&#xA;&#xA;  return (train_step, cross_entropy_mean, bottleneck_input, ground_truth_input,&#xA;          final_tensor)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even after adding the &lt;code&gt;sigmoid&lt;/code&gt; classifier and retraining, Tensorboard still shows &lt;code&gt;softmax&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vAsNo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vAsNo.png&quot; alt=&quot;Tensorboard with softmax&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13235" LastEditorUserId="13235" LastEditDate="2016-12-12T16:20:56.627" LastActivityDate="2016-12-12T16:20:56.627" Title="Adding multilabel classifier to TensorFlow example" Tags="|classification|tensorflow|image-classification|image-recognition|labels|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="15684" PostTypeId="1" AcceptedAnswerId="15685" CreationDate="2016-12-11T23:59:42.707" Score="1" ViewCount="240" Body="&lt;p&gt;I am doing a project on finding famous people who are similar to each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to test this out quantitatively? Take the example of actors for easy reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, one model predicts Robert Downey Jr. to be most similar to Christian Bale. While after applying more weights to the 'heights' feature, George Clooney is found to be most similar to Christian Bale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how do I quantitatively assert that one is better than the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One might say that Robert Downey is a better result because both Christian Bale and Robert Downey have starred in cult superhero movies (albeit different ones), while someone else might say that since George Clooney and Christian Bale were both batmans at one point, George Clooney is the better result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to test this other than intuition?&lt;/p&gt;&#xA;" OwnerUserId="12595" LastActivityDate="2016-12-12T01:27:05.407" Title="How to test People similarity measure?" Tags="|dataset|similarity|model-evaluations|distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15685" PostTypeId="2" ParentId="15684" CreationDate="2016-12-12T01:27:05.407" Score="2" Body="&lt;p&gt;Not unless you have an idea of what makes two people similar to each other. By choosing a distance function you are defining what it means for two people to be similar to one another. The question of whether one metric is quantitatively better than another may not be answerable without a defined objective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;a href=&quot;https://en.wikipedia.org/wiki/Taxicab_geometry&quot; rel=&quot;nofollow noreferrer&quot;&gt;compare the Manhattan distance (L1 norm) to the euclidean distance (L2 norm)&lt;/a&gt;. The former may be more effective for a pedestrian trying to navigate a city on foot, whereas the latter may be more effective for an athlete running across a sports field. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now how do I quantitatively assert that one is better than the other?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You would need some examples which embody what it means for two people to be similar (or not) to one another. This would have to be generated by a person or group of people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than generate a &quot;similarity value&quot; directly between two people, one could instead generate a ranked list of people which are most similar to a given person of interest:&#xA;For person A, the most similar other person is B, the second most similar person is C, etc. &#xA;Better yet, presented with a person of interest and a list of other people someone would just have to rank each person in the list by their similarity to the person of interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If more than one person generate ranks of a given list of people for a given person of interest, you could agglomerate the results by summing the ranks for each person and reordering the list by summed rank. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have some data in this form, then you may evaluate your similarity model by how well it recreates the ranked lists. Some possible metrics for evaluation are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Recall (at rank n): What fraction of the top n people are returned in the model's top n results?&lt;/li&gt;&#xA;&lt;li&gt;Precision (at rank n): What fraction of the model's top n results are in the true top n results?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Look &lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank#Evaluation_measures&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for other methods of evaluating ranking methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps. Let me know if I'm not understanding your question.&lt;/p&gt;&#xA;" OwnerUserId="26724" LastActivityDate="2016-12-12T01:27:05.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15686" PostTypeId="2" ParentId="15655" CreationDate="2016-12-12T02:48:10.667" Score="1" Body="&lt;p&gt;You are using Random Forest Regressor, but since you have a binary response where a passenger survives or not, the Random Forest Classifier should be used.  If this doesn't fix the issue, please send the link to the tutorial.&lt;/p&gt;&#xA;" OwnerUserId="26999" LastActivityDate="2016-12-12T02:48:10.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15688" PostTypeId="2" ParentId="15671" CreationDate="2016-12-12T04:02:57.060" Score="4" Body="&lt;p&gt;The image you posted is derived from figures which depict the basis functions of a Convolutional &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;Restricted Boltzmann Machine&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found them in this &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1553453&quot; rel=&quot;nofollow noreferrer&quot;&gt;ICML paper&lt;/a&gt; (behind a paywall, sorry) and this &lt;a href=&quot;https://web.eecs.umich.edu/~honglak/thesis_final.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PhD thesis&lt;/a&gt;, both are by Honglak Lee, who was advised by Andrew Ng.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strictly speaking they are not visualizations of a feed-forward Convolutional Neural Network, which might be why you are having a hard time interpreting what they depict. I don't really know much about RBM's myself so I'm afraid I can't help with their interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as visualizing CNN's, there are several common methods. You can visualize filter weights directly, find input images which maximally activate a filter, plot activation as a function of image occlusion, etc.&#xA;&lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This page&lt;/a&gt; has a good summary of techniques, also &lt;a href=&quot;http://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; may be useful.&lt;/p&gt;&#xA;" OwnerUserId="26724" LastEditorUserId="26724" LastEditDate="2016-12-12T09:04:19.067" LastActivityDate="2016-12-12T09:04:19.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15689" PostTypeId="1" AcceptedAnswerId="15690" CreationDate="2016-12-12T06:39:25.770" Score="3" ViewCount="1723" Body="&lt;p&gt;I am reading about backpropagation, one of many processes involved in gradient descent, and I couldn't quite grasp the intuition in one of the equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html#MathJax-Element-185-Frame&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap2.html#MathJax-Element-185-Frame&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An equation for the error $\delta^l$ in terms of the error in the next layer, $\delta^{l+1}$: In particular&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$$&#xA;\delta^l=((w^{l+1})^T\delta^{l+1})\circ\sigma'(z^l),&#xA;$$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where $(w^{l+1})^T$ is the transpose of the weight matrix $w^{l+1}$ for the $(l+1)^{th}$ layer.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;When we apply the transpose weight matrix, $(w^{l+1})^T$, we can think intuitively of this as moving the error &lt;em&gt;backward&lt;/em&gt; through the network, giving us some sort of measure of the error at the output of the $l^{th}$ layer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know how each terms are derived algebraically (from further down the text), but I can't grasp the intuitive part. In particular, I don't understand how applying the transpose of the weight matrix is like moving the error backwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mathematics is quite straightforward. The equation is the simplification of the error in layer $l$ expressed in terms of error in layer $l+1$. But, the whole intuition bit doesn't come so naturally. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please explain, in what sense applying the transpose is like &quot;moving the error backwards&quot;?&lt;/p&gt;&#xA;" OwnerUserId="27000" LastEditorUserId="27000" LastEditDate="2016-12-12T06:48:47.430" LastActivityDate="2019-07-18T15:09:41.220" Title="Intuition in Backpropagation (gradient descent)" Tags="|machine-learning|gradient-descent|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15690" PostTypeId="2" ParentId="15689" CreationDate="2016-12-12T08:11:00.847" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;in what sense applying the transpose is like &quot;moving the error backwards&quot;?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It isn't. Or at least you shouldn't think too hard about the analogy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are actually doing is calculating the gradient - or partial derivative - of the error/cost term with respect to the activations and weights in the network. This is done by repeatedly applying the &lt;a href=&quot;https://en.wikipedia.org/wiki/Chain_rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;chain rule&lt;/a&gt; to terms that depend on each other in the network, until you have the gradient for all the variables that you control in the network (usually you are interested in changing the weights between neurons, but you can alter other things if they are under your control - this is exactly how deep dreaming works too, instead of treating the weights as a variable, you treat the input image as one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analogy is useful in that it explains the progression and goal of the repeated calculations across layers. It is also correct in that values of the gradient in one layer depend critically on values in &quot;higher&quot; layers. This can lead to intuitions about managing problems with gradient values in deep networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no relation between taking the matrix transpose and &quot;moving the error backwards&quot;. Actually you are taking sums over terms connected by the weights in the NN model, and that happens to be the same &lt;em&gt;operation&lt;/em&gt; as the matrix multiplication given. The matrix multiplication is a concise way of writing the relationship (and is useful when looking at optimisations), but is not fundamental to understanding neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The statement &quot;moving the error backwards&quot; is in my opinion a common cause of problems when developers who have not studied the theory carefully try to implement it. I have seen quite a few attempts at implementing back propagation that cargo cult in error terms and multiplications without comprehension. It does not help here that some combinations of cost functions and activation functions in the output layer have been deliberately chosen to have very simple derivatives, so that it &lt;em&gt;looks like&lt;/em&gt; the neural network is copying error values around with almost magical multiplications by activation value etc. Actually deriving those simple terms takes a reasonably competent knowledge of basic calculus.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-12-12T08:11:00.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15692" PostTypeId="2" ParentId="13229" CreationDate="2016-12-12T10:04:06.530" Score="1" Body="&lt;p&gt;In my opinion, the methodological correct way to do it would be to &lt;strong&gt;first randomly select the observations&lt;/strong&gt; (e.g., with stratified random sampling to sustain the class balances as in the original data) and &lt;strong&gt;then do any model building&lt;/strong&gt; (feature selection is a part of model building) based on the sected examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should asses the quality of your model with remaining observations that have not been used for either selecting features or training a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="25217" LastActivityDate="2016-12-12T10:04:06.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15693" PostTypeId="2" ParentId="13753" CreationDate="2016-12-12T10:18:58.213" Score="1" Body="&lt;p&gt;Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Possible strategies to get a better estimation of the model performance would be:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)&lt;/li&gt;&#xA;&lt;li&gt;considering simpler models (e.g., less variables, more general parameters)&lt;/li&gt;&#xA;&lt;li&gt;considering other machine learning algorithms&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25217" LastActivityDate="2016-12-12T10:18:58.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15694" PostTypeId="1" CreationDate="2016-12-12T10:38:34.430" Score="4" ViewCount="2184" Body="&lt;p&gt;I'm using keras for multiple-step ahead time series forecasting of a univariate time series of type float. Judging by the results I got, the approach works&#xA;works perfectly well. There is, however, an important detail in the training process that baffles me: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;keras requires the &lt;strong&gt;sequence length of the input sequences (X matrix) to be equal to the forecasting horizon (y matrix)&lt;/strong&gt;. That means, for example,&#xA;that keras needs input sequences of length 20 in order to forecast the next 20 time steps. My goal is to be able to &lt;strong&gt;forecast as many time steps as I specify, given the last 20 time steps&lt;/strong&gt;. With the below code, this is not possible:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import numpy as np&#xA;    from keras.layers.core import Dense, Activation, Dropout, TimeDistributedDense&#xA;    from keras.layers.recurrent import LSTM&#xA;    from keras.models import Sequential&#xA;    from keras.optimizers import RMSprop&#xA;    np.random.seed(1234)&#xA;&#xA;&#xA;    model = Sequential()&#xA;    layers = [1, 20, 40, 1]&#xA;&#xA;    model.add(LSTM(&#xA;        input_dim=layers[0],&#xA;        output_dim=layers[1],&#xA;        return_sequences=True))&#xA;    model.add(Dropout(0.3))&#xA;&#xA;    model.add(LSTM(&#xA;        layers[2],&#xA;        return_sequences=True))&#xA;    model.add(Dropout(0.3))&#xA;&#xA;    model.add(TimeDistributedDense(&#xA;        output_dim=layers[3]))&#xA;    model.add(Activation(&quot;linear&quot;))&#xA;&#xA;    rms = RMSprop(lr=0.001)&#xA;    model.compile(loss=&quot;mse&quot;, optimizer=rms)&#xA;    model.fit(&#xA;        X_train, y_train,&#xA;        batch_size=512, nb_epoch=100, validation_split=0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get a dimension error whenever the sequence lengths of the sequences in X_train and in y_train differ from one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong and how can I fix it? Why are my results still pretty good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is related to &lt;a href=&quot;https://datascience.stackexchange.com/questions/15524/is-time-series-multi-step-ahead-forecasting-a-sequence-to-sequence-problem&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27002" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-11-07T03:45:07.263" Title="Sequence forecasting in keras not possible for variable-length sequence forecasts" Tags="|time-series|keras|rnn|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15695" PostTypeId="1" AcceptedAnswerId="15714" CreationDate="2016-12-12T11:28:06.323" Score="0" ViewCount="778" Body="&lt;p&gt;I'm trying to use binary relevance for multi-label text classification.&#xA;Here is the data I have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a training set with 6000 short texts (around 500-800 words each) and&#xA;some labels attached to them (around 4-6 for each text). There are&#xA;almost 500 different labels in the entire set.&lt;/li&gt;&#xA;&lt;li&gt;a test set with 6000 shorter texts (around 100-200 words each).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The difference of size between my two sets exists because of the source is different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I want to use binary relevance to find the labels of the texts in the test set. To do it, I created a dictionary with all the different words in the entire training set and removed stop words, words who appear only once and words who appear in more that 10% of the texts. I got 14714 different words in my dictionary.&lt;br&gt;&#xA;My idea was to create a matrix where each row represents a document and each column a words and each value was the number of occurrence of a word in a document. But with 14714 words and 6000 documents, I will get a matrix of 88 millions of integers! I tried, just to see, to create it and my laptop didn't support it. :)&lt;br&gt;&#xA;I even didn't have the time to create my Y matrix and generate a model (I wanted to use a logistic regression) for only one label...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Was it a good way to make multi-label classification or is there a better method?&lt;/li&gt;&#xA;&lt;li&gt;Is it a problem to have a training from one source and to use it to make a model to predict data from another source? Is the different size of the documents a problem?&lt;/li&gt;&#xA;&lt;li&gt;Did you use logistic regression for this kind of problem?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I also want to add the most frequent words in my dictionary (after the cleaning part) are common words and totally useless in the field of my research (biology): used, much, two, use, possible, example, ... How can I pass through it?&lt;/p&gt;&#xA;" OwnerUserId="27003" LastEditorUserId="27003" LastEditDate="2016-12-12T11:38:04.320" LastActivityDate="2016-12-13T13:37:26.687" Title="How to use binary relevance for multi-label text classification?" Tags="|machine-learning|text-mining|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15697" PostTypeId="1" CreationDate="2016-12-12T14:09:00.097" Score="0" ViewCount="191" Body="&lt;p&gt;So I have finally gotten a hold of ggplot2 and ggvis packages and tried out shiny too to have interactive graphics and visualizations. but i have been using Power BI before and came to R as a an good alternative for creating effective visualizations. But there is one thing that has been missing in R which is to be able to filter out all visualizations on a page on the basis of a mouse click in one of the visualizations on the page. I understand how we can create slicers and other ways to filter graphics using shiny or better with ggvis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can anyone suggest a package or something i have missed in the above packages to help me achieve this.&lt;/p&gt;&#xA;" OwnerUserId="27007" LastActivityDate="2017-02-12T08:20:26.323" Title="Interactive Data Visualisations" Tags="|r|visualization|ggplot2|powerbi|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15699" PostTypeId="1" CreationDate="2016-12-12T15:10:35.677" Score="2" ViewCount="1108" Body="&lt;p&gt;I have certain videos for which the frames are labeled either as dirty (meaning the camera lens is occluded by soil or rain) or as clean.&#xA;The goal is to test a convolutional neural net on this data to evaluate how well it can classify if a frame is dirty or clean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea is to use the first layers of an existing network and train last (fully connected) layers using the available data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But one issue might be that basically all available networks are trained for object classification and might not be very suitable for tasks like soil and rain detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any recommendations about networks or models which might be suitable for this task?&lt;/p&gt;&#xA;" OwnerUserId="27010" LastActivityDate="2018-05-22T14:08:15.243" Title="How do I use Convolutional Neural Nets to classify if there are rain drops or soil on the camera lens?" Tags="|deep-learning|computer-vision|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15700" PostTypeId="1" CreationDate="2016-12-12T15:34:51.823" Score="-1" ViewCount="51" Body="&lt;p&gt;The scenario is purchasing of a product or raw material from multiple suppliers on a regular basis, and the problem is how to best allocate order quantities among the suppliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example need 100 units of the product in a certain time period.  Based on historical averages for example the allocation percents can be 50,25,15,10 among 4 suppliers.  But sometimes suppliers are impacted by circumstance and resulting split is like 60,30,5,5.  There is a re-allocation if one producer cannot meet the demand so requests get re-routed to the most productive suppliers in the time period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other variables in play such as how busy the suppliers are with other clients, or even seasonal adjustments.  Some variables can be known, others are unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to optimize this problem so that the initial allocation is most accurate and re-allocation is minimized?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note: trying to understand if there is any tool or approach in Python Scikit-learn that generally can be used for this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any ideas!&lt;/p&gt;&#xA;" OwnerUserId="27011" LastActivityDate="2017-04-12T10:24:50.557" Title="How to optimize the allocation of product aquisition" Tags="|predictive-modeling|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15703" PostTypeId="1" CreationDate="2016-12-12T17:05:46.307" Score="1" ViewCount="72" Body="&lt;p&gt;I will soon deal with multiple projects in python. Some of them have to run regularly (many times a day), they can take some time (many days) and they use/produce some data coming from and going to other servers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know what tools you use to handle this kind of processes, knowing that they run on python. These tools could focus on:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;workflow management (what task should be executed when ?)&lt;/li&gt;&#xA;&lt;li&gt;data synchronization (database updates/synchronization) &lt;/li&gt;&#xA;&lt;li&gt;resources monitoring (space on disk, RAM and CPU ?)&lt;/li&gt;&#xA;&lt;li&gt;versioning (when these input/output arrived, should I treat them, or is there a new version ?)  &lt;/li&gt;&#xA;&lt;li&gt;visualization/monitoring (easy estimate the state of the processes, be alerted if an error occurred)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As an example, &lt;a href=&quot;https://github.com/spotify/luigi&quot; rel=&quot;nofollow noreferrer&quot;&gt;luigi&lt;/a&gt; tackle some of these problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Share links to articles or to tutorials if you have some in mind. I think this kind of challenges are faced by many people starting machine learning projects for production purposes.  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2016-12-12T17:05:46.307" Title="What tools to manage multiple machine learning projects in python do you use?" Tags="|machine-learning|bigdata|management|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15704" PostTypeId="2" ParentId="326" CreationDate="2016-12-12T17:58:21.800" Score="8" Body="&lt;p&gt;An issue all other answers fail to address is &lt;strong&gt;licensing&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Most of the aforementioned wonderful R libraries are GPL (e.g. &lt;a href=&quot;https://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ggplot2&lt;/a&gt;, &lt;a href=&quot;https://cran.r-project.org/web/packages/data.table/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;data.table&lt;/a&gt;). This &lt;a href=&quot;https://softwareengineering.stackexchange.com/questions/47032/can-i-use-gpl-software-in-a-commercial-application&quot;&gt;prevents you&lt;/a&gt; from distributing your software in a proprietary form. As of July 11 2022, ggplot2 uses the MIT + file &lt;a href=&quot;https://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;license&lt;/a&gt; while data.table moved on to the &lt;a href=&quot;https://www.mozilla.org/en-US/MPL/2.0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MPL2 licence&lt;/a&gt; with more R packages, including caret using more flexible licences.&lt;/p&gt;&#xA;&lt;p&gt;Although many usages of those libraries do not imply distribution of the software (e.g. to train models offline), the GPL may by itself lure away companies from using them. At least in my experience.&lt;/p&gt;&#xA;&lt;p&gt;In the python realm, on the other hand, &lt;em&gt;most&lt;/em&gt; libraries have business-friendly distribution licenses, such as BSD or MIT.&lt;/p&gt;&#xA;&lt;p&gt;In academia, licensing issues normally are non-issues.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="62993" LastEditDate="2022-07-11T21:50:49.357" LastActivityDate="2022-07-11T21:50:49.357" CommentCount="0" CommunityOwnedDate="2016-12-12T17:58:21.800" ContentLicense="CC BY-SA 4.0" />
  <row Id="15705" PostTypeId="1" AcceptedAnswerId="15708" CreationDate="2016-12-12T18:44:09.567" Score="2" ViewCount="339" Body="&lt;p&gt;I have two classifiers which I am implementing, and they are both non-deterministic in the sense that they can each give different results (FPR and TPR) when you run them multiple times. I would like to compare these two algorithms to evaluate their performance. How do I go about this? Usually what people do most times is to run the classifier till they get the best FPR and TPR values, then they publish the results. But the problem with this approach is that it might not be a good representation of the performance of such a classifier. This is what I planned on doing so far, but don't know if that is correct:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split my evaluation data into train and test, and after training, predict using the test data to get the FPR and TPR, then repeat this prediction for 99 more times to form 100 FPR and TPR readings, then take an average of this. To get an ROC, use the mean FPR and TPR.  OR&lt;/li&gt;&#xA;&lt;li&gt;Use k fold cross validation of say k=3 or 10 on the data, and this will return 3 or 10 different values for TPR and FPR, then I will take the mean to get the mean FPR and TPR, and also use this mean for plotting the mean ROC.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which of the two methods I stated above is ok? And if they are both wrong, what do you suggest I do. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="27019" LastActivityDate="2016-12-13T00:16:10.050" Title="Comparing Non-deterministic Binary Classifiers" Tags="|machine-learning|classification|cross-validation|performance|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15707" PostTypeId="1" CreationDate="2016-12-12T22:00:08.537" Score="0" ViewCount="969" Body="&lt;p&gt;I have a dataset that contains 20 predictor variables (both categorical and numeric) and one target variable (2 classes - Good and Bad). But, there are only 23 observations in the dataset. While I wait to receive significantly more observations, what tests / models can I perform on the available dataset to understand the &lt;i&gt;variance between the good and bad cases&lt;/i&gt;, and to understand the &lt;i&gt;variance within the cases classified as 'good'&lt;/i&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, for the data to make sense, I would want the variance within the good cases to be low, and the variance between the good and bad cases to be high. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would multivariate analysis of variance (MANOVA) work in this case? &lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2017-05-21T21:17:10.513" Title="How to measure variance in a classification dataset?" Tags="|classification|variance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15708" PostTypeId="2" ParentId="15705" CreationDate="2016-12-13T00:16:10.050" Score="0" Body="&lt;p&gt;A good strategy is to do &lt;em&gt;n times k-fold cross validation&lt;/em&gt;, which should give a pretty good estimation of the average performance of both algorithms. What this means is you perform your &lt;em&gt;k&lt;/em&gt;-fold cross validation &lt;em&gt;n&lt;/em&gt; times, with different random folds each time, and average the results. People commonly use &lt;em&gt;k = n&lt;/em&gt; = 10, but the higher the values of &lt;em&gt;k&lt;/em&gt; and &lt;em&gt;n&lt;/em&gt; the better.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-13T00:16:10.050" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15709" PostTypeId="2" ParentId="12648" CreationDate="2016-12-13T00:49:46.517" Score="1" Body="&lt;p&gt;PCA is a &lt;strong&gt;transform:&lt;/strong&gt; it creates new (transformed) features from the original data. In general if you choose fewer dimensions (e.g. you chose to reduce m=12 -&gt; n=2 dimensions), it's lossy and will throw away some of in the information content of the original data. &lt;strong&gt;The higher n is, the less you lose, and for m=n, you preserve all the original information&lt;/strong&gt; (although you still do a vector transform on the data, so the extracted features are != the original data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was your (arbitrary) decision to choose the parameter n=2 (number of Principal Components), you could try other values or explore a range. You could have chosen n=5, n=9, or even the maximum possible: n=12.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;standard rules-of-thumb on how to choose n, see e.g.&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/44060/choosing-number-of-principal-components-to-retain&quot;&gt;Choosing number of principal components to retain&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(Scree plot, Proportion of total variance explained, Average eigenvalue rule, Log-eigenvalue diagram, etc.) &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where a Scree Plot is a simple line-segment plot that shows the fraction of total variance in the data as explained or represented by each PC. Usually the scree plot will have a knee where the number of PCs explains most of the variance, and if so that might suggest you an upper bound on n.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other rules-of-thumb discussed there too.&#xA;You can find tons of articles on this subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also e.g. &lt;a href=&quot;https://stackoverflow.com/questions/12067446/how-many-principal-components-to-take&quot;&gt;How many principal components to take?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-13T00:49:46.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15710" PostTypeId="1" AcceptedAnswerId="15712" CreationDate="2016-12-13T06:32:17.460" Score="4" ViewCount="9751" Body="&lt;p&gt;I am looking a python lib named &lt;code&gt;deap&lt;/code&gt;, but stuck at beginning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://deap.readthedocs.io/en/master/examples/ga_onemax.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;first paragraph&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The problem is very simple, we search for a 1 filled list individual.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does &lt;code&gt;1 filled list&lt;/code&gt; means? Search a 1 filled list, from where? individual list or individual 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google &lt;code&gt;One Max Problem&lt;/code&gt; only gives some information which seems to be useful:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There is a &lt;a href=&quot;https://github.com/Oddsor/EvolAlgo/wiki/Max-One-Problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Max One Problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can understand this, but is it same as &lt;code&gt;One Max Problem&lt;/code&gt;? If so, I have a question that why need &lt;code&gt;evolutionary algorithm to &quot;evolve&quot; our population until eventually the target emerges.&lt;/code&gt; ? If I am a Medical Researcher, I should already have the entire DNA(gene list) , what I need to do is just search in that list, not evolve a random list.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Maximum_satisfiability_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Maximum_satisfiability_problem&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is understandable too, seems relate to &lt;code&gt;One Max Problem&lt;/code&gt; , another saying?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="29575" LastEditDate="2017-12-14T12:37:12.463" LastActivityDate="2017-12-14T12:37:12.463" Title="What is the One Max Problem in detail?" Tags="|genetic-algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15711" PostTypeId="2" ParentId="15700" CreationDate="2016-12-13T08:07:04.560" Score="0" Body="&lt;p&gt;To break the problem down, I think you are seeking to predict the capacity of the 5 suppliers ahead of time. So that you could allocate the order sizes for each supplier that is smaller than the capacity so that there is no need for reallocation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I doubt that you would be able to know what is the full capacity of each supplier, as you have mentioned, each supplier also have other clients so you don't know how much more could each supplier handle, on top of your previous order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a (big) compromise, you could model as though each supplier is operating at 100% capacity when they serviced your past orders. With that, you have the series of Y to feed into your regression model. As for the inputs, you could add seasonality e.g. Time of year. I'm not sure as a outsider, you would have enough meaningful input variables to train the model. Maybe a naive approach of Mean(OrderSize) + 2StdDev(OrderSize) might work better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2016-12-13T08:07:04.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15712" PostTypeId="2" ParentId="15710" CreationDate="2016-12-13T08:19:50.257" Score="6" Body="&lt;p&gt;&lt;code&gt;deap&lt;/code&gt; is an evolutionary algorithm library. In an evolutionary algorithm you usually want to optimize a function. For this, you define individuals as a collection of genes (e.g. a string of numbers) that condense a possible solution, you create a population of such individuals, and define a fitness function to evaluate how good they are; then you apply evolutionary operators (e.g. mutation, pairing) to evolve the population, effectively searching the solution space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem referred to as &quot;max one problem&quot; in the linked page can be reworded as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;let's create a toy evolutionary algorithm where we want to evolve a population of individuals (where each individual is a list of N integer numbers) until one of them is exactly comprised of N ones (i.e. 1,1,....,1).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &quot;maximum satifiability problem&quot; is not related to this.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-12-13T08:19:50.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15713" PostTypeId="2" ParentId="10011" CreationDate="2016-12-13T08:22:22.083" Score="1" Body="&lt;p&gt;You may use bootstrapping to estimate a confidence interval for the prediction error. Further help can be found in some &lt;a href=&quot;https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/cv_boot.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford online course slides&lt;/a&gt;, but I haven't done this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides that, it should be no problem to compare the estimated performance obtained with cross-validation (mean and standard deviation) with a reference point (e.g., AUC = 0.5) or with the results of another benchmark model such as logistic regression or nearest-neighbor classifier using a simple statistical test using a given level of confidence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar question is discussed &lt;a href=&quot;https://stats.stackexchange.com/questions/69831/confidence-intervals-for-cross-validated-statistics&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="25217" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-12-13T08:22:22.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15714" PostTypeId="2" ParentId="15695" CreationDate="2016-12-13T13:37:26.687" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Just some views/suggestions. After removing the stop words did you stem/lemmatize the text you got?  That would probably reduce the number of unique words in your corpus and brings some forms of words to the same level. But caution while using stemming as it sometimes create noise. &lt;/li&gt;&#xA;&lt;li&gt;Try postagging and see what can be the important tags you want to keep and eliminate the ones that you feel are giving less relevance to the text.&lt;/li&gt;&#xA;&lt;li&gt;Did you try to find the some important terms from each document by using &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf&lt;/a&gt; or &lt;a href=&quot;http://glowingpython.blogspot.in/2014/02/terms-selection-with-chi-square.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;chi-square method&lt;/a&gt;. They might be helpful to see the relevance of terms for each class/document.&lt;/li&gt;&#xA;&lt;li&gt;See how far can you reduce the dimensions of the matrix after using the above, if not you have already used them and then apply logistic regression or whatever classifier you wanted to.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On your second question.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I don't think having data from different sources should be a problem since you are trying to create model based on keywords, Hopefully it should be able to see. I am not confident on this part.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I had worked on text classification before with bigger documents and larger corpus with not so fruitful results with various models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-12-13T13:37:26.687" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15715" PostTypeId="1" AcceptedAnswerId="15723" CreationDate="2016-12-13T14:27:08.040" Score="0" ViewCount="174" Body="&lt;p&gt;I'm trying to look for an idea to create a predictive model having the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Customer_ID - Integer&#xA;Catalog_ID - Integer&#xA;Country_Code - Integet&#xA;Year - Integet&#xA;Month - Integer&#xA;Day - Integer&#xA;Quantity_Purchased - Integer&#xA;Product_Purchased - Double&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to look for a use case that creates a predictive model than can give me the ability to propose a product for a customer the next time that he comes to my website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a collaborative filtering use case? If yes, I only can use the last two fields of my dataset, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="25824" LastEditorUserId="25157" LastEditDate="2016-12-13T17:12:50.540" LastActivityDate="2016-12-13T21:01:16.253" Title="POC - Get an idea to create a Predictive Model" Tags="|predictive-modeling|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15716" PostTypeId="1" AcceptedAnswerId="15758" CreationDate="2016-12-13T14:47:53.763" Score="0" ViewCount="212" Body="&lt;p&gt;Is there a way to differentiate between scanned images(with only text) that are well-lit and with good contrast, from the ones that are not(that have poor quality), making use of Content-based image retrieval? More exactly, is there a feature extraction method that can help to query a database to retrieve the well-lit, good contrast, scanned images?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="27043" LastActivityDate="2016-12-15T17:53:57.903" Title="How to apply Content-based image retrieval for scanned images?" Tags="|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15717" PostTypeId="1" AcceptedAnswerId="15720" CreationDate="2016-12-13T15:43:28.023" Score="0" ViewCount="774" Body="&lt;p&gt;Trying to do a binary classification and have the class names as string but when it gives me the error mentioned in the title. I tried to give integer labels as 0 and 1, the code works perfectly fine. I tried all the suggestions from stackoverflow regarding the same error but nothing is working.Here is the code snippet: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labels = [&quot;No&quot;, &quot;priorlocaltx&quot;]&#xA;#labels = [0, 1]&#xA;keyword_identifiers_no = ['surgery', 'radiotherapy', 'brachytherapy', 'Newadjuvant', 'neo', 'Adjuvant', 'Mets at first diagnosis', 'M1HSPC']&#xA;keyword_identifiers_yes = ['no prior tx', 'BCR', 'Neoadjuvant', 'neo', 'Adjuvant', 'Hormone refractory', 'Mets', 'No mets']&#xA;&#xA;cancer_df.insert(2, 'Result', 0, allow_duplicates = False)&#xA;#print(cancer_df.head())&#xA;for i in range(1, len(cancer_df)):&#xA;&#xA;    for word in keyword_identifiers_no:&#xA;        if word in cancer_df.iloc[i]['Patient Segment(s)']:&#xA;&#xA;            cancer_df.at[i,'Result'] = labels[0]&#xA;            #print(cancer_df['Patient Segment(s)'].dtype)&#xA;&#xA;    for word in keyword_identifiers_yes:&#xA;        if word in cancer_df.iloc[i]['Patient Segment(s)']:&#xA;            cancer_df.at[i,'Result']= labels[0]&#xA;&#xA;    cancer_df.to_csv('Result1.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24011" LastEditorUserId="381" LastEditDate="2016-12-13T16:52:09.850" LastActivityDate="2016-12-13T17:06:17.877" Title="ValueError:invalid literal for int() with base 10: 'No''" Tags="|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15718" PostTypeId="1" CreationDate="2016-12-13T16:03:43.037" Score="4" ViewCount="2351" Body="&lt;p&gt;I am using word2vec to train a big corpus of textual information. While analyzing the scatter plot of words I found that it has not only grouped numbers together but also ordered them based on their value. I am able to understand that it groups together words sharing similar context but not able to understand how it got the values also correctly. Can someone please give an intuitive explanation of how this happens? I am attaching the image where I have zoomed in on the numbers.&lt;a href=&quot;https://i.stack.imgur.com/rkvoZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rkvoZ.png&quot; alt=&quot;word2vec zoomed-in output&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25766" LastActivityDate="2016-12-15T20:12:16.407" Title="How word2vec understands the relationship between numbers?" Tags="|machine-learning|neural-network|nlp|text-mining|word2vec|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15719" PostTypeId="2" ParentId="14795" CreationDate="2016-12-13T16:33:50.743" Score="1" Body="&lt;p&gt;Creating model on second scenario is difficult but more information driven and give you more prediction capibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is use second scenario and create some featured variable. These variable will give information like after marrying in how many months attrition happen and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So think what other info you can take out of this trend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First scenario is not of much use, because it's just giving you last information of employee which hides many important relevant information which are actually driving attrition.&lt;/p&gt;&#xA;" OwnerUserId="20517" LastActivityDate="2016-12-13T16:33:50.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15720" PostTypeId="2" ParentId="15717" CreationDate="2016-12-13T17:06:17.877" Score="1" Body="&lt;p&gt;It seems that your 'Result' column is of type &lt;code&gt;int&lt;/code&gt; and you are trying to set some rows to &lt;code&gt;str&lt;/code&gt;. Stick to &lt;code&gt;int&lt;/code&gt; or &lt;code&gt;category&lt;/code&gt; (&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/categorical.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-12-13T17:06:17.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15722" PostTypeId="1" CreationDate="2016-12-13T18:18:46.497" Score="-1" ViewCount="3880" Body="&lt;p&gt;I want to classify images in a few different groups with a Neural Network algorithm in R. I have tried different things but nothing works. So my question is: what is the best way to read in images so that I can put the data in a Neural Network algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out my other question as well for things that I have tried already: &lt;a href=&quot;https://stackoverflow.com/questions/40660306/picture-classification-with-neural-network-in-r&quot;&gt;https://stackoverflow.com/questions/40660306/picture-classification-with-neural-network-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25779" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2017-05-04T09:29:10.437" Title="Image classification with Neural Network in R" Tags="|machine-learning|neural-network|r|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15723" PostTypeId="2" ParentId="15715" CreationDate="2016-12-13T19:41:01.627" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;Recommender systems&lt;/a&gt; are usually either collaborative (&quot;other people who bought x also bought y&quot;) or content-based (&quot;item y has similar properties as item x&quot;). Given your very few data features, collaborative filtering seems appropriate. In the basic setting it could use only the purchases of your users. Ideally, you would also have some sort of rating of their purchases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, nothing stops you from using a hybrid approach with additional data if available. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2016-12-13T21:01:16.253" LastActivityDate="2016-12-13T21:01:16.253" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15724" PostTypeId="2" ParentId="12790" CreationDate="2016-12-14T00:58:23.737" Score="1" Body="&lt;p&gt;Just about any ETL tool can manage fixed width, CSV, TSV, or PSV input, and just about any tool should be able to manage 100B records.  The limiting part of the question really has to do with what your destination format is, and what disk throughput you need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expected throughput on an i2.4xLarge is 250mb/s.  If an 8xLarge is double that, times 32 machines, you are looking at the ability to write a petabyte in ~138 hours.  Not to mention the time and bandwidth of bringing in the source data in the first place.  Unless my math is completely off, that means 30 Petabytes can get written to disk in about 6 months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems odd that you are looking to either normalize or turn into human readable format that much data (it's only going to get bigger), and even odder that you'd want to leverage machine learning as part of a transformation/load of that size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your solution will need to be on local hardware in order to keep costs reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't recommend a system (commercial or open source) that would scale to the degree necessary to perform this kind of ETL on 30 Petabytes in a matter of days.  At that scale, I'd be looking into lots of memory, ram backed/fronted SSDs, and custom development on FPGAs for the actual transformations.  Of course, if my math on the write timing is wrong this whole answer is invalid.&lt;/p&gt;&#xA;" OwnerUserId="434" LastActivityDate="2016-12-14T00:58:23.737" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15725" PostTypeId="1" CreationDate="2016-12-14T01:59:18.823" Score="1" ViewCount="415" Body="&lt;p&gt;I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Xe3HY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xe3HY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ROC Curve as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dtCQq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dtCQq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YfPEe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YfPEe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26163" LastEditDate="2016-12-14T03:07:13.047" LastActivityDate="2016-12-14T03:38:23.063" Title="How to interpret the AUC score in this case?" Tags="|machine-learning|python|random-forest|confusion-matrix|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15726" PostTypeId="2" ParentId="15725" CreationDate="2016-12-14T02:58:58.940" Score="0" Body="&lt;p&gt;AUC is based on rank order of your predictions, not the actual class to which it's assigned. It's very likely that the scale of the output is misbehaving. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the values of your predictions, I suspect that the predictions of your model are within a tight range. If that's the case, the argmax will yield the same class for all of your observations (which is what's happening).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may wish to tinker with some of the hyperparameters to see which one is causing this exactly (might start with the learning rate). It's probably worth testing if a logistic regression gives you the same problem, which will help identify whether or not it's a problem with your inputs/features.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-12-14T02:58:58.940" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15728" PostTypeId="2" ParentId="15725" CreationDate="2016-12-14T03:38:23.063" Score="2" Body="&lt;p&gt;If you want to figure it out how this ROC happens , you would better &lt;strong&gt;LIST&lt;/strong&gt; the tuples including your &amp;quot;predicted&amp;quot; values and the &amp;quot;truth&amp;quot; values, and &lt;strong&gt;SORT&lt;/strong&gt; with the &amp;quot;predicted&amp;quot; value , then &lt;strong&gt;PLOT&lt;/strong&gt; the ROC .&lt;/p&gt;&#xA;&lt;p&gt;In your case , the &lt;strong&gt;tuples&lt;/strong&gt; and &lt;strong&gt;points&lt;/strong&gt; should be like this :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;  predicted truth (x,y)&#xA;    0.53 0  (6/6,14/14)&#xA;    0.55 0  (5/6,14/14)&#xA;    0.57 1  (4/6,14/14)&#xA;    0.59 0  (4/6,13/14)&#xA;    0.60 1  (3/6,13/14)&#xA;    0.62 1  (3/6,12/14)&#xA;    0.63 0  (3/6,11/14)&#xA;    0.64 0  (2/6,11/14)&#xA;    0.66 1  (1/6,11/14)&#xA;    0.68 1  (1/6,10/14)&#xA;    0.71 1  (1/6,9/14)&#xA;    0.73 1  (1/6,8/14)&#xA;    0.77 1  (1/6,7/14)&#xA;    0.78 0  (1/6,6/14)&#xA;    0.82 1  (0/6,6/14)&#xA;    0.86 1  (0/6,5/14)&#xA;    0.89 1  (0/6,4/14)&#xA;    0.92 1  (0/6,3/14)&#xA;    0.94 1  (0/6,2/14)&#xA;    0.96 1  (0/6,1/14)&#xA;            (0/6,0/14)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Put these points (x,y) on ROC picture , it should be like this :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/cfaKV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cfaKV.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Just much like that of yours !&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double&#xA;    = {&#xA;    &#xA;        // retrieve number of positive and negative samples in ground truth&#xA;        val nPos = groundTruth.toArray.filter(_&amp;gt;0).length&#xA;        val nNeg = groundTruth.toArray.filter(_&amp;lt;=0).length&#xA;    &#xA;        // tuple predict with ground truth , and sort with predict&#xA;        val pair = predict.toArray.zip(groundTruth.toArray)&#xA;        val sortedPair = pair.sortBy(_._1)&#xA;        var auc = 0.0.toDouble&#xA;        val x = BDV.zeros[Double](predict.length + 1)&#xA;        val y = BDV.zeros[Double](predict.length + 1)&#xA;        x(0) = 1.0&#xA;        y(0) = 1.0&#xA;    &#xA;        // calculate auc incrementally&#xA;        var i = 1.toInt&#xA;        while(i &amp;lt; sortedPair.length) {&#xA;          y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 &amp;gt; 0).length) / nPos&#xA;          x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 &amp;lt;= 0).length) / nNeg&#xA;          auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)&#xA;          i += 1&#xA;        }&#xA;        auc = auc + ((y(i - 1) * x(i - 1))/2.0)&#xA;    &#xA;        auc&#xA;   }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .&lt;/p&gt;&#xA;&lt;p&gt;Hopes this contributes you , good luck -)&lt;/p&gt;&#xA;" OwnerUserId="9728" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-12-14T03:38:23.063" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15729" PostTypeId="1" CreationDate="2016-12-14T04:17:54.097" Score="0" ViewCount="900" Body="&lt;p&gt;The &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD&quot; rel=&quot;nofollow noreferrer&quot;&gt;YearPredictionMSD&lt;/a&gt; dataset, is a machine learning practice dataset extracted from the &lt;a href=&quot;http://labrosa.ee.columbia.edu/millionsong/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Million Song Dataset&lt;/a&gt; and found in the UCI Machine Learning Repository. I couldn't find the field list that describes the features. Does anyone know where I can find them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I'm confused why there are only 54 features in the original data from LabRosa but 90 in the subset of the data on UCI.&lt;/p&gt;&#xA;" OwnerUserId="27065" LastEditorUserId="836" LastEditDate="2016-12-14T14:22:45.390" LastActivityDate="2016-12-14T21:31:53.980" Title="Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)" Tags="|dataset|supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15730" PostTypeId="2" ParentId="5345" CreationDate="2016-12-14T04:52:54.337" Score="5" Body="&lt;p&gt;You can try R-Brain platform (r-brain.io). R-Brain provides an integrated cloud/on-premises data science platform for developing models with popular open source languages. Powered by Jupyter, our IDE, console, notebook and markdown are all integrated into one environment with full language support for R and Python. R-Brain editor is built with Monaco, the heart of VS code. With Docker technology and prebuilt images, R-Brain empowers data scientists with quick setup, instant collaboration and version control at workspace level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am founder of R-Brain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shadi&lt;/p&gt;&#xA;" OwnerUserId="27066" LastActivityDate="2016-12-14T04:52:54.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15731" PostTypeId="2" ParentId="10228" CreationDate="2016-12-14T05:35:49.870" Score="21" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Gini is intended for continuous attributes and Entropy is for attributes that occur in classes  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gini&lt;/strong&gt; is to minimize misclassification&lt;br&gt;&#xA;&lt;strong&gt;Entropy&lt;/strong&gt; is for exploratory analysis  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Entropy is a little slower to compute&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14738" LastEditorUserId="50954" LastEditDate="2018-06-29T21:44:04.857" LastActivityDate="2018-06-29T21:44:04.857" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15732" PostTypeId="1" CreationDate="2016-12-14T08:12:16.487" Score="3" ViewCount="698" Body="&lt;p&gt;Is there any good library in stanfordNLP which is meant for name parsing? Basically i want to separate first name, last name, nickname, surname, givenname from the full name. Something similar to HumanName and parserator.tag, parserator.parse? Lets say i have a name &lt;code&gt;Mr George &quot;Gob&quot; Bluth II&lt;/code&gt; and if i use parserator.parse, it gives me below output. Is there something like this in stanford NLP? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; probablepeople.parse('Mr George &quot;Gob&quot; Bluth II')  &#xA;[('Mr', 'PrefixMarital'), ('George', 'GivenName'), ('&quot;Gob&quot;', 'Nickname'), ('Bluth', 'Surname'), ('II', 'SuffixGenerational')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2018-01-11T20:36:08.683" Title="Name Tagger in Stanford NLP" Tags="|nlp|stanford-nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15733" PostTypeId="1" CreationDate="2016-12-14T08:16:39.453" Score="0" ViewCount="424" Body="&lt;p&gt;I am very new to Java and primarily a Python and R user. I want to use &lt;a href=&quot;https://opennlp.apache.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenNLP&lt;/a&gt; for certain NLP tasks that I have been entrusted with. Is there any good tutorial or book on OpenNLP that anyone can recommend?&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="31513" LastEditDate="2017-05-25T10:20:55.193" LastActivityDate="2019-07-11T03:00:58.877" Title="OpenNLP tutorial or book" Tags="|nlp|reference-request|self-study|books|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15736" PostTypeId="2" ParentId="15699" CreationDate="2016-12-14T17:02:51.880" Score="0" Body="&lt;p&gt;This is a great problem. Rain affects the image through a localized lensing effect, so I'd partition the image into patches big enough to capture a raindrop, then run a classifier on each patch. Training data is not an issue since you say you have video. I'm not sure how to capture this lensing effect, but my intuition would be to use a 2D FFT or wavelet decomposition to extract the raw features your CNN will run on top off. If this does not work well enough, I suggest asking about what kind of features to use to detect lensing on physics.SE.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-12-14T17:07:52.017" LastActivityDate="2016-12-14T17:07:52.017" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15737" PostTypeId="1" AcceptedAnswerId="15738" CreationDate="2016-12-14T17:15:48.210" Score="1" ViewCount="1209" Body="&lt;p&gt;I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a corresponding row, but if no measurements were taken, the values are the same as the day before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of the trend I am trying to manage:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date, Weight, BMI&#xA;1/1/2016, 155.1, 21.9&#xA;1/2/2016, 155.1, 21.9&#xA;1/3/2016, 155.1, 21.9&#xA;--continued for several weeks--&#xA;3/1/2016, 170.2, 25.0&#xA;3/2/2016, 170.1, 25.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;edit: I should clarify that these repeated values are how the data is put together for missing values. The days that are repeated are days when no measurement was taken&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the value stays the same for an extended period, should there be a gap in any graphical representation? Should I keep the numbers as they were (155.1 &amp;amp; 21.9) until the next measurement was taken, or should the numbers increase over that time to &quot;bridge the gap&quot; - meaning they would increase by the difference in measurements divided by the number of days?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It feels like I should be increasing the value over time to account for what would happen in reality, but I don't know if this would have negative implications on the data.&lt;/p&gt;&#xA;" OwnerUserId="15056" LastEditorUserId="15056" LastEditDate="2016-12-14T18:32:55.337" LastActivityDate="2016-12-14T18:32:55.337" Title="Handling time series data with gaps" Tags="|r|dataset|visualization|data-cleaning|graphical-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15738" PostTypeId="2" ParentId="15737" CreationDate="2016-12-14T17:32:55.473" Score="1" Body="&lt;p&gt;I'd represent actual repeated values differently from missing data. The latter is straightforward; the former I'd interpolate with &lt;a href=&quot;https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_process_prediction.2C_or_kriging&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian process regression&lt;/a&gt;. This way you'll be able to get error bars like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Uj3oM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Uj3oM.png&quot; alt=&quot;Gaussian process regression&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note how the sample functions (and thus error bars) expand and contract as you depart and approach the measurements, as you would intuitively expect.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2016-12-14T17:32:55.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15741" PostTypeId="2" ParentId="15729" CreationDate="2016-12-14T21:31:53.980" Score="1" Body="&lt;p&gt;You don't get a full data dictionary with this dataset. Instead the explanation (from your link) is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;90 attributes, 12 = timbre average, 78 = timbre covariance &#xA;  The first value is the year (target), ranging from 1922 to 2011. &#xA;  Features extracted from the 'timbre' features from The Echo Nest API. &#xA;  We take the average and covariance over all 'segments', each segment &#xA;  being described by a 12-dimensional timbre vector.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is basically features derived direct from the audio. This is typical for problems in signal analysis, and the UCI dataset has been cut down so that it is purely about matching audio summaries to year of production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://labrosa.ee.columbia.edu/millionsong/pages/example-track-description&quot; rel=&quot;nofollow noreferrer&quot;&gt;The data dictionary from the original MSD dataset&lt;/a&gt; shows much more metadata. There are 54 feature &lt;em&gt;types&lt;/em&gt;, but actually thousands of features - each type can be repeated many times. The UCI subset is not only subset by row, but by columns too, and only uses some of the MSD data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-12-14T21:31:53.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15742" PostTypeId="1" CreationDate="2016-12-14T22:34:04.497" Score="2" ViewCount="1100" Body="&lt;p&gt;Is it possible to write a program using deep learning that categorizes an article (news, entertainment,..., etc. ) into a positive category or negative category based on a specific theme?&lt;/p&gt;&#xA;" OwnerUserId="27094" LastEditorUserId="25755" LastEditDate="2016-12-15T20:21:16.137" LastActivityDate="2016-12-16T19:09:29.567" Title="categorizing articles using deep learning" Tags="|machine-learning|python|data-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15743" PostTypeId="1" CreationDate="2016-12-15T00:27:26.477" Score="1" ViewCount="1156" Body="&lt;p&gt;I have a field in a Redshift table that has user-generated text.  The field is where users can say how much they think something costs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally it'd just be a decimal, but it's varchar.  So users can type&#xA;&quot;I think this is worth \$25&quot;, or &quot;I'd pay 55&quot; or &quot;\$117&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm trying to use &lt;code&gt;regexp_substr&lt;/code&gt; to pull this out.  Specifically &lt;code&gt;regexp_substr(f.comment_text, '\\$?[0-9]*')&lt;/code&gt;.  But this doesn't work on a subset of entries for some reasons (eg &lt;code&gt;Could do for $115&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I remove the &lt;code&gt;?&lt;/code&gt; it works on that, but no longer on entries that don't use &lt;code&gt;$&lt;/code&gt;.  Why?  And what should I use instead?&lt;/p&gt;&#xA;" OwnerUserId="25611" LastActivityDate="2017-03-15T07:49:28.743" Title="Using regex in redshift to find dollar values" Tags="|redshift|regex|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15744" PostTypeId="2" ParentId="15742" CreationDate="2016-12-15T02:04:11.373" Score="6" Body="&lt;p&gt;Yes, this is classic sentiment analysis. You'll need training data in order to write this program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, people will build a language modeling using the text data in the article, tweets, posts, etc. Most folks specify the language model using a Recurrent Neural Network with a Long Short Term Memory. &lt;a href=&quot;http://k8si.github.io/2016/01/28/lstm-networks-for-sentiment-analysis-on-tweets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a blog post that has code linked to their github.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before diving into a more advanced deep neural network like an LSTM, it may be easier to start with a &lt;a href=&quot;http://deeplearning.net/tutorial/mlp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;multilayer perceptron&lt;/a&gt; using &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow noreferrer&quot;&gt;n-grams&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like you may be asking for a couple of different things. So, let me be very explicit. The models I listed above take as input the articles and a label that evaluated the articles as either positive or negative sentiment, the model then returns the likelihood of a given article being positive or negative sentiment. It in no way returns a topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To return the &quot;topic&quot; your best bet would be to run &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Latent Dirichlet Allocation&lt;/a&gt;, which would then return the likelihood of a given article being generated from a given topic that was previously learned. It's important to note that these will only be from topics learned on previous training data and not on &lt;strong&gt;&lt;em&gt;new&lt;/em&gt;&lt;/strong&gt; topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are standard algorithms available to visualize topic models (e.g., &lt;a href=&quot;https://github.com/bmabey/pyLDAvis&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDAVis&lt;/a&gt;) and the embeddings of language models (e.g., &lt;a href=&quot;https://www.quora.com/How-do-I-visualise-word2vec-word-vectors&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec visualizations&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth noting that there is research being done in learning these items jointly (i.e., the latent topic and the classification of good/bad). &lt;a href=&quot;https://www.cs.nyu.edu/~wanli/wan-zhu-fergus12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;I'd recommend reviewing this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be worth reviewing these items more thoroughly before actually diving into applying them. It sounds like you may not have a complete picture of what your goal is and how deep learning can help you. &lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2016-12-16T19:09:29.567" LastActivityDate="2016-12-16T19:09:29.567" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15747" PostTypeId="2" ParentId="15743" CreationDate="2016-12-15T03:37:07.563" Score="1" Body="&lt;p&gt;I am getting that you want extract numbers with either '$' or even not without a dollar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are not using just [0-9]+ for that field? The results will be like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    some $1 something &amp;gt;&amp;gt; ['1']&#xA;    some other 115 &amp;gt;&amp;gt; ['115']&#xA;    $115 some thing &amp;gt;&amp;gt; ['115']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to include '\$' sign, then '\$' can be repeated zero or once or by typo more, then regex would be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    \$*[0-9]+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which outputs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    some $1 something &amp;gt;&amp;gt; ['$1']&#xA;    some other 115 &amp;gt;&amp;gt; ['115']&#xA;    $115 some thing &amp;gt;&amp;gt; ['$115']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this is not what you want, comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend using online regex matchers like &lt;a href=&quot;http://rubular.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. They have short and sweet references which really help us get to the context.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2016-12-15T07:12:51.043" LastActivityDate="2016-12-15T07:12:51.043" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="15749" PostTypeId="1" CreationDate="2016-12-15T08:03:51.997" Score="1" ViewCount="83" Body="&lt;p&gt;I am working on a problem where I need to identify the most likely used routes for a number of trips. The trips have their own starting and ending locations, along with distances covered and durations of these trips. I am not sure how to use the underlying road maps. Is indirectly modeling the route segments from these attributes a good idea? Any suggestion on how could I implement and test it? Some alternatives?&lt;/p&gt;&#xA;" OwnerUserId="10163" LastActivityDate="2017-03-15T12:57:32.267" Title="Modeling maps using trip data" Tags="|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15750" PostTypeId="1" CreationDate="2016-12-15T08:04:42.003" Score="1" ViewCount="119" Body="&lt;p&gt;I have a group of images. All contain some common object (let's assume all at the same size and no other alterations). I want to train a CNN that will learn the filter for the common object and gives me back a boolean flag of whether it exists in a given image or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(for simplicity, I assume the &quot;object&quot; is smaller than 5x5 pixels)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of something like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A Conv layer with 1 filter of 5x5&lt;/li&gt;&#xA;&lt;li&gt;A MaxPooling layer with pool_size = input size (meaning - returns a single value: the maximal activation of the filter)&lt;/li&gt;&#xA;&lt;li&gt;Output max activation (or a variation of it)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Would you output the raw max activation and use a threshold to decide? sigmoid of it? or force the network to output &quot;object&quot; and &quot;no_object&quot; classes and do softmax?&lt;/li&gt;&#xA;&lt;li&gt;Quite related - what loss function? &lt;/li&gt;&#xA;&lt;li&gt;Is it a problem to have a large MaxPooling layer (one that reduces all activation to a single one)? What happens to the gradient on backprop?&lt;br&gt;&#xA;It seems like it would just go randomly to one of the activations and I'm not sure that it can &quot;learn&quot; the object that way..&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how to look at this problem as it's not really a 2-class classification problem. Even if I can come up with images that doesn't contain the object, there is nothing to &quot;learn&quot; from this class except of the fact that it &lt;em&gt;doesn't&lt;/em&gt; contain something (which is a valuable information here, I must agree). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried playing with it on simulated small black-white images and a cross &quot;object&quot; that I randomly added. Couldn't get the filter to learn the cross pattern although on some tries it got the horizontal line..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea would help! Thanks!&lt;/p&gt;&#xA;" OwnerUserId="27093" LastEditorUserId="29169" LastEditDate="2021-02-16T15:28:08.197" LastActivityDate="2021-02-16T15:28:08.197" Title="How to build a network that detects common object in images?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15751" PostTypeId="2" ParentId="15750" CreationDate="2016-12-15T08:14:47.163" Score="3" Body="&lt;p&gt;You probably just need a deeper network to learn the cross test example. Convolutional layers need to have multiple feature channels and be stacked in order to learn anything more complex that edge/corner counting. Look at example networks for e.g. MNIST and you will usually see typically 2 or 3 CNN layers, with multiple channels (e.g. 32 or 64 channels, each of which is associated with previous layer channels amount of NxN kernels) plus 1 or 2 fully connected layers before classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a binary choice, yes train on object present/not present classes. There is no major difference between a single sigmoid output or softmax with two outputs. Use cross-entropy loss function, multiclass version if you use softmax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest just adapt an MNIST example network to your problem by changing input shape and number of outputs. So that would use softmax and multiclass cross-entropy loss.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it a problem to have a large MaxPooling layer&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It could work in some situations, but by having a single layer, single channel CNN, you have made the network too simple to classify even moderately complex shapes. Reducing the activations from this layer to just the max is going to limit the network to essentially classifying by strongest matching 5x5 template to the filter in the image. With enough training examples it should be able to create such a filter, but it would not cope very well with noise, rotation or any partial matches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might get something more effective with a large pooling layer over say a 2-deep multi-channel CNN, and then a small fully-connected layer over those max channel outputs. It would still be an unusual choice of architecture, but I suspect it could learn problems like your cross test.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2016-12-15T17:23:17.253" LastActivityDate="2016-12-15T17:23:17.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15752" PostTypeId="2" ParentId="15749" CreationDate="2016-12-15T11:34:44.920" Score="2" Body="&lt;p&gt;It depends on the application and the Data.Could you provide more details about the data of the trips? Do you have GPS-positions for the trips?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are Libraries such as &lt;a href=&quot;https://github.com/bmwcarit/barefoot&quot; rel=&quot;nofollow noreferrer&quot;&gt;Barefoot&lt;/a&gt; which support you in map-matching, meaning that it maps your positions to the underlying map, in most cases OpenStreetMap. This library uses Hidden Markov Models for online and offline Map Matching. Maybe this could help you as well, when implementing your own model.&lt;/p&gt;&#xA;" OwnerUserId="16201" LastActivityDate="2016-12-15T11:34:44.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15753" PostTypeId="2" ParentId="15750" CreationDate="2016-12-15T12:25:05.013" Score="0" Body="&lt;p&gt;Also, you could take some ideas of an architecture from&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/soumith/convnet-benchmarks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/jcjohnson/cnn-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jcjohnson/cnn-benchmarks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There, are multiple state-of-the-art cnn architectures&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2016-12-15T12:25:05.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15758" PostTypeId="2" ParentId="15716" CreationDate="2016-12-15T17:44:35.397" Score="1" Body="&lt;p&gt;Your problem is studied under the rubric of &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Image_quality&quot; rel=&quot;nofollow noreferrer&quot;&gt;image (Quality) assessment&lt;/a&gt;&quot; and, more specifically, &quot;document image assessment&quot;. Here are links to and abstracts from some relevant surveys:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pdfs.semanticscholar.org/0970/c10b5af6c5bbaeedf5d696c3c355aaa97959.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Document Image Quality Assessment: A Brief Survey&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To maintain, control and enhance the quality of document images and minimize the negative impact of degrada- tions on various analysis and processing systems, it is critical to understand the types and sources of degradations and develop reliable methods for estimating the levels of degradations. This paper provides a brief survey of research on the topic of document image quality assessment. We first present a detailed analysis of the types and sources of document degradations. We then review techniques for document image degradation modeling. Finally, we discuss objective measures and subjective experiments that are used to characterize document image quality.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/profile/Abbas_Ebrahimi-Moghadam/publication/263544805_Subjective_and_Objective_Quality_Assessment_of_Image_A_Survey/links/567fd91608ae1e63f1e9358e.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Subjective and Objective Quality Assessment of Image: A Survey&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With the increasing demand for image-based applications, the efficient and reliable evaluation of image quality has increased in importance. Measuring the image quality is of fundamental importance for numerous image processing applications, where the goal of image quality assessment (IQA) methods is to automatically evaluate the quality of images in agreement with human quality judgments. Numerous IQA methods have been proposed over the past years to fulfill this goal. In this paper, a survey of the quality assessment methods for conventional image signals, as well as the newly emerged ones, which includes the high dynamic range (HDR) and 3-D images, is presented. A comprehensive explanation of the subjective and objective IQA and their classification is provided. Six widely used subjective quality datasets, and performance measures are reviewed. Emphasis is given to the full-reference image quality assessment (FR-IQA) methods, and 9 often-used quality measures (including mean squared error (MSE), structural similarity index (SSIM), multi-scale structural similarity index (MS-SSIM), visual information fidelity (VIF), most apparent distortion (MAD), feature similarity measure (FSIM), feature similarity measure for color images (FSIMC), dynamic range independent measure (DRIM), and tone-mapped images quality index (TMQI)) are carefully described, and their performance and computation time on four subjective quality datasets are evaluated. Furthermore, a brief introduction to 3-D IQA is provided and the issues related to this area of research are reviewed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/profile/Vipin_Kamble3/publication/276271541_No-reference_image_quality_assessment_algorithms_A_survey/links/5726d3d208ae262228b21594.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;No-reference image quality assessment algorithms: A survey&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;valuation of noise content or distortions present in an image is same as assessing the quality of an image. Measurement of such quality index is challenging in the absence of reference image. In this paper, a survey of existing algorithms for no-reference image quality assessment is presented. This survey includes type of noise and distortions covered, techniques and parameters used by these algorithms, databases on which the algorithms are validated and benchmarking of their performance with each other and also with human visual system.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.tandfonline.com/doi/abs/10.1080/00207160.2013.816415&quot; rel=&quot;nofollow noreferrer&quot;&gt;Objective image quality assessment: a survey&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Image quality assessment (IQA) is critically important for the image-processing field. IQA aims to build a computational model to predict human perceived image quality, accurately and automatically. Until now, great efforts have been employed to design IQA metrics. In this paper, we systematically and comprehensively review the fundamental, brief history, and state-of-the-art developments of IQA, with emphasis on natural image quality assessment (NIQA). First, the definition of image quality is discussed, which contains three aspects and lead to different philosophies of designing IQA metrics. Afterwards, classic NIQA metrics are presented with some further discussions. Widely used databases and the performances of classic NIQA metrics on them are also listed. We highlight the most significant works and some open issues about the developments of IQA, and provide the benchmarks for the researchers and scholars who work on IQA.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Research papers specifically on document image assessment&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/profile/Xujun_Peng/publication/286929531_Document_Image_Quality_Assessment_Using_Discriminative_Sparse_Representation/links/5714643d08aeebe07c064296.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Document Image Quality Assessment Using Discriminative Sparse Representation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The goal of document image quality assessment (DIQA) is to build a computational model which can predict the degree of degradation for document images. Based on the estimated quality scores, the immediate feedback can be provided by document processing and analysis systems, which helps to maintain, organize, recognize and retrieve the information from document images. Recently, the bag-of-visual-words (BoV) based approaches have gained increasing attention from researchers to fulfill the task of quality assessment, but how to use BoV to represent images more accurately is still a challenging problem. In this paper, we propose to utilize a sparse representation based method to estimate document image’s quality with respect to the OCR capability. Unlike the conventional sparse representation approaches, we introduce the target quality scores into the training phase of sparse representation. The proposed method improves the discriminability of the system and ensures the obtained codebook is more suitable for our assessment task. The experimental results on a public dataset show that the proposed method outperforms other hand-crafted and BoV based DIQA approaches.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://link.springer.com/chapter/10.1007/978-3-319-30348-2_18&quot; rel=&quot;nofollow noreferrer&quot;&gt;Discrete Orthogonal Moments Based Framework for Assessing Blurriness of Camera Captured Document Images&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One of the most widely used tasks in the area of image processing is automated processing of documents, which is done using Optical Character Readers (OCR) from document images. The most common form of distortion in document images is blur which can be caused by defocus, motion, camera shake etc. In this paper we propose a no reference image sharpness measure framework using discrete orthogonal moments and image gradients for assessing quality of document images and validated the results against state of the art image sharpness measures and accuracy of three well known Optical Character Readers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/profile/Alireza_Alaei/publication/303950933_Document_Image_Quality_Assessment_Based_on_Texture_Similarity_Index/links/5762779808ae5c6f86db9657.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Document Image Quality Assessment Based on Texture Similarity Index&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this paper, a full reference document image quality assessment (FR DIQA) method using texture features is proposed. Local binary patterns (LBP) as texture features are extracted at the local and global levels for each image. For each extracted LBP feature set, a similarity measure called the LBP similarity index (LBPSI) is computed. A weighting strategy is further proposed to improve the LBPSI obtained based on local LBP features. The LBPSIs computed for both local and global features are then combined to get the final LBPSI, which also provides the best performance for DIQA. To evaluate the proposed method, two different datasets were used. The first dataset is composed of document images, whereas the second one includes natural scene images. The mean human opinion scores (MHOS) were considered as ground truth for performance evaluation. The results obtained from the proposed LBPSI method indicate a significant improvement in automatically/accurately predicting image quality, especially on the document image-based dataset.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/abstract/document/7532968/&quot; rel=&quot;nofollow noreferrer&quot;&gt;No-reference document image quality assessment based on high order image statistics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Document image quality assessment (DIQA) aims to predict the visual quality of degraded document images. Although the definition of “visual quality” can change based on the specific applications, in this paper, we use OCR accuracy as a metric for quality and develop a novel no-reference DIQA method based on high order image statistics for OCR accuracy prediction. The proposed method consists of three steps. First, normalized local image patches are extracted with regular grid and a comprehensive document image codebook is constructed by K-means clustering. Second, local features are softly assigned to several nearest codewords, and the direct differences between high order statistics of local features and codewords are calculated as global quality aware features. Finally, support vector regression (SVR) is utilized to learn the mapping between extracted image features and OCR accuracies. Experimental results on two document image databases show that the proposed method can accurately predict OCR accuracy and outperforms previous algorithms.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pdfs.semanticscholar.org/8407/184b67114cbdddfec1bded6bf5a35eb8cb82.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A deep learning approach to document image quality assessment&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This paper proposes a deep learning approach for document image quality assessment. Given a noise corrupted document image, we estimate its quality score as a prediction of OCR accuracy. First the document image is divided into patches and non-informative patches are sifted out using Otsu’s bina- rization technique. Second, quality scores are obtained for all selected patches using a Convolutional Neural Network (CNN), and the patch scores are averaged over the image to obtain the document score. The proposed CNN contains two layers of convolution, location blind max-min pooling, and Rectified Linear Units in the fully connected layers. Exper- iments on two document quality datasets show our method achieved the state of the art performance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=2110719&quot; rel=&quot;nofollow noreferrer&quot;&gt;Metric-based no-reference quality assessment of heterogeneous document images&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;No-reference image quality assessment (NR-IQA) aims at computing an image quality score that best correlates with either human perceived image quality or an objective quality measure, without any prior knowledge of reference images. Although learning-based NR-IQA methods have achieved the best state-of-the-art results so far, those methods perform well only on the datasets on which they were trained. The datasets usually contain homogeneous documents, whereas in reality, document images come from different sources. It is unrealistic to collect training samples of images from every possible capturing device and every document type. Hence, we argue that a metric-based IQA method is more suitable for heterogeneous documents. We propose a NR-IQA method with the objective quality measure of OCR accuracy. The method combines distortion-specific quality metrics. The final quality score is calculated taking into account the proportions of, and the dependency among different distortions. Experimental results show that the method achieves competitive results with learning-based NR-IQA methods on standard datasets, and performs better on heterogeneous documents.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2016-12-15T17:53:57.903" LastActivityDate="2016-12-15T17:53:57.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15759" PostTypeId="2" ParentId="9667" CreationDate="2016-12-15T19:01:20.367" Score="2" Body="&lt;p&gt;I know it has been a while since the question has been posted but for future readers I propose here another solution for the problem of classifying time-series of proportional data i.e., vectors of proportions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hidden Markov Models (HMMs) are broadly used for time-series classification. The original implementations were developed for discrete and Gaussian data but since then, they have been extended to more probability distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among them, the Dirichlet, the generalized Dirichlet, and the Beta-Liouville distributions which are perfectly suited to work with proportional data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, a model is trained for each category (&quot;Tag&quot; in this question) from time-series for which the category is known. When a new time-series has to be classified, its likelihood with respect to each model is computed. The new time-series is usually assigned to the category leading to the highest likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the reference papers for these models presenting them more deeply and providing all the equations needed for implementation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dynamical Dirichlet Mixture Model, Research Report from IDIAP, L. Chen, D. Barber, J.-M. Odobez, April 2007&lt;/li&gt;&#xA;&lt;li&gt;Proportional data modeling with hidden Markov models based on generalized Dirichlet and Beta-Liouville mixtures applied to anomaly detection in public areas, E. Epaillard, N. Bouguila, Pattern Recognition 55, pp 125-136, 2016&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the additional two features mentioned in the question, HMMs for mixed data have also been developed in the following research paper:&#xA;Hybrid hidden Markov model for mixed continuous/continuous and discrete/continuous data modeling, E. Epaillard, N. Bouguila, MMSP, pp. 1-6, 2015.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These models are heavier to deploys than regression models but have the advantage of being discriminative and generative models and to be able to generalize well when some variability is present in the processed data.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastEditorUserId="25157" LastEditDate="2016-12-15T19:07:36.587" LastActivityDate="2016-12-15T19:07:36.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15761" PostTypeId="2" ParentId="15718" CreationDate="2016-12-15T20:12:16.407" Score="1" Body="&lt;p&gt;Attempting an answer from the understanding I have of word2vec and deep learning through my personal experience and some live presentations of Yoshua Bengio and Yann Le Cun that I had the chance to attend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned in the question, I think too that numbers are grouped together as they are have similar functions in the sentences where they are found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the reason of how they get ordered, as said in my comment:&#xA;Intuitively I would say that it will be true for small numbers and less true for big numbers (the way 7-9-8 are ordered seems to go that way). When we speak we often say things like: &quot;I ate 1 or 2 apples&quot; or &quot;I will buy 2 or 3 things&quot; but rarely &quot;I will buy 3 or 2 things&quot;. I think this is how it ends up ordering the numbers. Bigger numbers will appear less in a corpus so they will most probably not be ordered as precisely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When looking at tutorials about word2vec, an example that is often given is that, if we see the relationship between 2 words as a vector, the final representation of the words by the model will give that the vector linking &quot;man&quot; to &quot;woman&quot; is approximately the same as the one linking &quot;king&quot; to &quot;queen&quot;. Or that the vector linking &quot;Paris&quot; to &quot;France&quot; is approximately equal to the one linking &quot;Rome&quot; to &quot;Italy&quot;. This even allows to do operations such as: France + Italy - Paris = Rome! The clear mechanics of how these relationships are so accurate (compared to other approaches) are, as of today, still obscur to the scientific community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these kind of operations works for countries and main cities, it seems logical that the vectors &quot;3-2&quot; and &quot;2-1&quot; point in the same direction, i.e. that 1-2-3 are ordered (given that, as said previously, consecutive numbers are more likely to be used together in a sentence). Of course, to achieve this, the numbers have to frequently appear in the training corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am very happy that someone showed this result with numbers because the way word2vec has been trained in most of the models that are available online did not includ digits. They were replaced by some symbols i.e., 12.34 becoming ##.##, and such observations cannot be made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion to improve this answer is welcome!&lt;/p&gt;&#xA;" OwnerUserId="25157" LastActivityDate="2016-12-15T20:12:16.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15764" PostTypeId="2" ParentId="15352" CreationDate="2016-12-15T22:18:13.520" Score="3" Body="&lt;p&gt;What you are looking for is called a metric (or distance, or similarity measure) for HMMs (some people would say SVM kernel for HMMs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somehow, if you have a distance, you can cluster the HMMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most classical distance, defined by Rabiner and Juang is a Monte-Carlo approximation of a Kullback-Leibler divergence for HMM. It requires a long observation sequence $O$ generated from the first HMM $\lambda_1$ and is then computed as: &#xA;$$ D_{KL}(\lambda_1||\lambda_2) = 1/T * (log(P(O|\lambda_1)-log(P(O|\lambda_2))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of this is that it does not imply any constraint on the number of hidden state. However, $O$ needs to be a long sequence if you want accurate results.&#xA;There is some research on the topic but not too much, and even less for HMMs with different topology (which is another reason why Google does not give much results...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: This paper proposes a distance measure that does not require the number of hidden states to be the same but is only valid for uni-dimensional observations.&#xA;A new distance measure for hidden Markov models by Zeng, Duan, and Wu in Experts Systems with Applications 37, p. 1550-1555, 2010. It is rather easy to implement.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastEditorUserId="25157" LastEditDate="2016-12-15T22:46:15.080" LastActivityDate="2016-12-15T22:46:15.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15765" PostTypeId="1" AcceptedAnswerId="15766" CreationDate="2016-12-15T22:20:16.537" Score="29" ViewCount="9928" Body="&lt;p&gt;I am trying to remove stop words before performing topic modeling. I noticed that some negation words (not, nor, never, none etc..) are usually considered to be stop words. For example, NLTK, spacy and sklearn include &quot;not&quot; on their stop word lists. However, if we remove &quot;not&quot; from these sentences below they lose the significant meaning and that would not be accurate for topic modeling or sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1). StackOverflow is helpful      =&amp;gt; StackOverflow helpful&#xA;2). StackOverflow is not helpful  =&amp;gt; StackOverflow helpful&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone please explain why these negation words are typically considered to be stop words? &lt;/p&gt;&#xA;" OwnerUserId="27129" LastEditorUserId="27129" LastEditDate="2016-12-21T22:48:25.443" LastActivityDate="2016-12-21T22:48:25.443" Title="NLP - why is &quot;not&quot; a stop word?" Tags="|nlp|topic-model|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15766" PostTypeId="2" ParentId="15765" CreationDate="2016-12-16T00:30:35.750" Score="29" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;noreferrer&quot;&gt;Stop words&lt;/a&gt; are usually thought of as &quot;the most common words in a language&quot;. However, other definitions based on different tasks are possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It clearly makes sense to consider 'not' as a stop word if your task is based on word &lt;strong&gt;frequencies&lt;/strong&gt; (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;tf–idf&lt;/a&gt; analysis for document classification). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're concerned with the context (e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Sentiment_analysis&quot; rel=&quot;noreferrer&quot;&gt;sentiment analysis&lt;/a&gt;) of the text it might make sense to treat negation words differently. Negation changes the so-called &lt;em&gt;valence&lt;/em&gt; of a text. This needs to be treated carefully and is usually not trivial. One example would be the Twitter negation corpus. An explanation of the approach is given in &lt;a href=&quot;https://www.cs.cmu.edu/~ark/EMNLP-2015/proceedings/WASSA/pdf/WASSA14.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="27129" LastEditDate="2016-12-16T00:59:48.647" LastActivityDate="2016-12-16T00:59:48.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15767" PostTypeId="1" AcceptedAnswerId="15811" CreationDate="2016-12-16T01:43:54.107" Score="9" ViewCount="1087" Body="&lt;p&gt;I'm a student of machine learning, and these days I was trying to learn how to use the TensorFlow library. I've gone through various tutorials and trial&amp;amp;errors with tensorflow, and I thought the best way to learn it for real would be to make use of it in a little project of my own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've decided that I should attempt to make a program that writes poems. I'm not aiming for top-end quality program; for my first model, I'd be happy with just a string of non-sense words groups together in poem format. The problem is that I'm having problem looking up books or videos about machine learning programs that deal with writing sentence structures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you make any suggestions on what I could look for (even google keywords are fine) to get the sample programs and basic knowledge that I need?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="27135" LastActivityDate="2020-08-02T12:23:25.053" Title="Machine Learning: Writing Poems" Tags="|machine-learning|neural-network|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15768" PostTypeId="2" ParentId="15722" CreationDate="2016-12-16T11:18:05.953" Score="1" Body="&lt;p&gt;If you want to use R, loading images can be done with imager. General information, tutorials and case studies can be found &lt;a href=&quot;http://dahtah.github.io/imager/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for neural networks in R, the basic ones which are available like nnet, RSNNS and deepnet are to probably too slow. I recommend looking into H2O for deeplearning. If you want to go with GPU support you need to look at other options, like tensorflow or mxnet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow in R: You can use tensorflow from R with the tensorflow package. Information on how to install and use it from R can be found &lt;a href=&quot;https://rstudio.github.io/tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MXNET in R: follow the instructions from the &lt;a href=&quot;https://github.com/dmlc/mxnet/tree/master/R-package&quot; rel=&quot;nofollow noreferrer&quot;&gt;mxnet github page&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2016-12-16T11:18:05.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15770" PostTypeId="1" AcceptedAnswerId="15771" CreationDate="2016-12-16T14:46:38.257" Score="4" ViewCount="386" Body="&lt;p&gt;I posted this question on Cross Validated before I realized that this existed. I think it is better suited here and got no answers over there so I have deleted other post. I have reproduced the question below: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been playing around with the neural network toolbox in MATLAB to develop an intuition for how the architectural requirements scale with feature dimension. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I put together a simple example, and the results have surprised me. I am hoping someone can point to either (a) an unrealistic expectation of mine, or (b) a mistake/misuse of the neural network toolbox. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example is as follows: I have a simple un-normalized one-dimensional Gaussian that I am trying to learn. I do the following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = -5:0.2:5;&#xA;y = exp(-x.^2/2);&#xA;&#xA;net = feedforwardnet(2);&#xA;net = configure(net, x, y); &#xA;net = train(net, x, y); &#xA;y2 = net(x); &#xA;&#xA;plot(x, y, 'o', x, y2);&#xA;legend('Data', 'NN');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me good results. I get the plot below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EtgeD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EtgeD.png&quot; alt=&quot;ANN 1D&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I try to extend this to 2 dimensions and this is where I run into trouble. I don't think I'm asking too much. My data is not noisy, or is it sparse. I figure if I double the number of neurons that should be sufficient for an increase in dimensionality. Here's my code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 = -5:0.2:5;&#xA;x2 = -5:0.2:5;&#xA;[x1g, x2g] = meshgrid(x1, x2);&#xA;xv = [x1g(:)'; x2g(:)'];&#xA;yv = exp(-dot(xv,xv)/2);&#xA;&#xA;net = feedforwardnet(4); &#xA;net = configure(net, xv, yv);&#xA;net = train(net, xv, yv);&#xA;y2v = net(xv);  &#xA;plot3(xv(1,:), xv(2,:), yv, xv(1,:), xv(2,:), y2v, 'o');&#xA;legend('Data', 'NN');&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The plot I get is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PComY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PComY.png&quot; alt=&quot;NNet 2D&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is pretty poor. Perhaps I need more neurons? Maybe if I double the number of dimensions, I need to quadruple the number of neurons. I get this for 8 neurons:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VHut2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VHut2.png&quot; alt=&quot;8 neurons&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe with 8 neurons I have a lot of weights to fit, so let me try training with regularization. I get the plot below with &lt;code&gt;trainbr&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XMwyc.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XMwyc.png&quot; alt=&quot;8 neurons with regularization&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's only at around 16 neurons that I start getting something I would consider reasonable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yZ24G.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yZ24G.png&quot; alt=&quot;16 neurons&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are still oscillations which I don't like. Now I know I'm using it out of the box in a naïve manner, or perhaps I'm expecting too much. But this simple example resembles the real problem I want to tackle. I have the following questions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why is it that an increase from 1 to 2 dimensions increases the number of neurons required to get a decent fit considerably? &lt;/li&gt;&#xA;&lt;li&gt;Even when I go to a larger number of neurons, I get oscillations that are going to be a problem in my real world application. How can I get rid of that? &lt;/li&gt;&#xA;&lt;li&gt;Most resources on NN that I've read indicate a substantially lower number of neurons. They usually state something like &quot;equal to or less than the number of input variables&quot;. Why is that? Is a multidimensional Gaussian a pathological case? &lt;/li&gt;&#xA;&lt;li&gt;If I need to be more intelligent with how I treat my network for a given number of neurons, what do I need to do? I tried retraining the network to see if it was a local minima issue, but I generally get a similar fit. &lt;/li&gt;&#xA;&lt;li&gt;Anything else that may be remotely useful to this issue is appreciated!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27121" LastActivityDate="2018-01-29T07:00:33.747" Title="Performance and architecture of neural network for increased dimensions" Tags="|machine-learning|neural-network|regression|matlab|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15771" PostTypeId="2" ParentId="15770" CreationDate="2016-12-16T15:23:59.803" Score="5" Body="&lt;p&gt;Some things to take into account:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try to apply appropriate input space transformations, e.g. convert to polar coordinates.&lt;/li&gt;&#xA;&lt;li&gt;Despite the fact that a single hidden layer feedforward network can be a &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;noreferrer&quot;&gt;universal approximator&lt;/a&gt;, there is no guarantees about the number of neurons needed to approximate an arbitrarily complex function. Instead of having a single hidden layer and making it increasingly wide, try stacking more layers. This enables the network to model non linearities in an easier way.&lt;/li&gt;&#xA;&lt;li&gt;Do not verify the performance of your model based on visual inspection (at least not only), but based on error measurements (e.g. MSE).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2016-12-16T23:03:18.453" LastActivityDate="2016-12-16T23:03:18.453" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15774" PostTypeId="2" ParentId="10077" CreationDate="2016-12-16T19:54:58.313" Score="16" Body="&lt;p&gt;This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were talking about single-document summarization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.cmu.edu/~afm/Home_files/Das_Martins_survey_summarization.pdf&quot; rel=&quot;noreferrer&quot;&gt;It's worth briefly reviewing the literature.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link provided by &lt;a href=&quot;https://datascience.stackexchange.com/users/26596/society-of-data-scientists&quot;&gt;u/Society Of Data Scientists&lt;/a&gt; is great and it's useful for the &lt;strong&gt;&lt;em&gt;abstractive&lt;/em&gt;&lt;/strong&gt; summarization task across a single document. There's also work done on &lt;strong&gt;&lt;em&gt;extractive&lt;/em&gt;&lt;/strong&gt; summarizations, which identifies important sentences to extract. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1509.00685v2.pdf&quot; rel=&quot;noreferrer&quot;&gt;Rush et. al has a nice paper on the abstractive summarization with Attention&lt;/a&gt;, which is based on deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an extractive summarization, you could use an LSTM to build your classifier and use standard TensorFlow/Torch libraries but there doesn't seem to be any current publications on using deep learning for this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some additional GitHub repos:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/facebook/NAMAS&quot; rel=&quot;noreferrer&quot;&gt;The Original Rush et. al repo.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/carpedm20/neural-summary-tensorflow&quot; rel=&quot;noreferrer&quot;&gt;A TensorFlow implementation of the same paper&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-16T19:54:58.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15780" PostTypeId="1" CreationDate="2016-12-17T09:12:46.500" Score="1" ViewCount="7370" Body="&lt;p&gt;The only way I came out is using tf.argmax with tf.reduce_max. But I'm afraid of that it may be inefficient to find the max value twice.&lt;/p&gt;&#xA;" OwnerUserId="25772" LastActivityDate="2022-10-02T00:24:49.883" Title="How can I get the max value and corresponding index of a tensor simultaneously in Tensorflow?" Tags="|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15781" PostTypeId="2" ParentId="15780" CreationDate="2016-12-17T09:36:58.260" Score="1" Body="&lt;p&gt;Before you worry about performance, I'd recommend profiling it. Finding the maximum is an inexpensive operation. Also, can you index into the element using the return value of argmax? That way you'd only take the max once.&lt;/p&gt;&#xA;" OwnerUserId="27171" LastActivityDate="2016-12-17T09:36:58.260" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15782" PostTypeId="1" CreationDate="2016-12-17T10:10:38.287" Score="1" ViewCount="495" Body="&lt;p&gt;I want to train the network based on two sets of data. For example, I want the network to predict the humidity based on past humidity trends AND past temperature trends. In this case, how should I organize the input layer? I would think that for just one time-series, I would use a regular sliding window on the series. With two series, do I just present both windows (from the humidity and temperature series) to the input layer that is twice the size of the window? if not, how else can I configure the input layer so that it doesn't confuse between the two sets of training data?﻿&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I just let the network sort this out by itself or is there a preferred method of presenting two (or more) sets of training data to the network? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="27172" LastActivityDate="2016-12-17T23:53:11.847" Title="How to feed two (or more) time-series data sets to a neural network?" Tags="|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15784" PostTypeId="1" CreationDate="2016-12-17T11:36:12.127" Score="7" ViewCount="4338" Body="&lt;p&gt;I am training a big corpus using word2vec and averaging the word vectors to get sentence vectors. What is the best way to address negative words so that negative and positive sentences are away from each other? For e.g.: &quot;After the fix code worked&quot; and &quot;After the fix code did not work&quot; should ideally give sentence vectors which are far from each other. I heard one approach is to look for negative words like &quot;not&quot; and negate the next word vector. Can someone please clarify if that's a good approach or can suggest a better approach? &lt;/p&gt;&#xA;" OwnerUserId="25766" LastActivityDate="2019-04-23T06:02:47.727" Title="How to handle negative words in word2vec?" Tags="|machine-learning|neural-network|nlp|word2vec|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15787" PostTypeId="2" ParentId="15782" CreationDate="2016-12-17T23:53:11.847" Score="1" Body="&lt;p&gt;The whole idea of NN is that it learns by itself to give the appropriate weights and biases to the input neurons, so yes - just give your network the 2 series and it should return the appropriate weight and bias for each neuron in the input. &lt;/p&gt;&#xA;" OwnerUserId="25905" LastActivityDate="2016-12-17T23:53:11.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15788" PostTypeId="2" ParentId="5684" CreationDate="2016-12-18T00:20:48.310" Score="1" Body="&lt;p&gt;You could always do a Lasso regression by setting the elastic net parameter to 1:&#xA;&lt;code&gt;&#xA;val reg = new LogisticRegression()&#xA;  .setElasticNetParam(1)&#xA;&lt;/code&gt;&#xA;The Lasso regression penalizes the number of coefficients, so it is indirectly doing variable selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-linear-methods.html#linear-least-squares-lasso-and-ridge-regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;the help of Spark's Mllib&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27186" LastActivityDate="2016-12-18T00:20:48.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15789" PostTypeId="1" CreationDate="2016-12-18T02:00:28.013" Score="0" ViewCount="148" Body="&lt;p&gt;I am working on a project building a system to automate much of the tasks in my home garden.  For a few weeks now I have had the system setup to control a humidifier as well as have a sensor to read the current humidity.  I have been reading the humidity every minute and my program has a min and max humidity set, if the humidity goes below the min it kicks on my humidifier and if it goes above the max it kicks on my exhaust fans.  I have also set the system up to log the temp and humidity reading every time it updates to a database.  What I am looking to do is write an algorithm that will take this data and calculate when it needs to turn the humidifier on/off to keep the humidity in the tent as stable as possible.  Right now if you look at the humidity data it is consistently being brought up to just under the max when the humidifier is running (I have it setup to shut off when it reaches 3/4 of the max humidity) and then dropping back down to the min humidity.  The humidifier is a warm vapour humidifier so it takes a few minutes to heat up every time it kicks on.  I want to write something that would look at past data, look at all these factors and calculate when it should turn the humidifier on or off to try and keep a steady base humidity but, I am not really sure where to start or what to google to work toward an answer.  Any info anyone could provide would be greatly appreciated.  I apologise if I am not giving a great description and feel free to ask me for more details.&lt;/p&gt;&#xA;" OwnerUserId="27189" LastActivityDate="2016-12-18T02:15:31.003" Title="Machine Learning For Keeping Stable Humidity" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15790" PostTypeId="2" ParentId="15789" CreationDate="2016-12-18T02:15:31.003" Score="2" Body="&lt;p&gt;This sounds like something you might want to do with a PID or Kalman Filter. It'd probably work better than a complex machine learning model. The humidity is probably cyclical in nature and if you can just tighten the loop a bit, you should be able to achieve your result with a simple PID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this might be what you're looking for:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.eurotherm.com/pid-control-made-easy&quot; rel=&quot;nofollow noreferrer&quot;&gt;PID example&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.allaboutcircuits.com/projects/embedded-pid-temperature-control-part-6-zieglernichols-tuning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Application&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27172" LastActivityDate="2016-12-18T02:15:31.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15791" PostTypeId="1" CreationDate="2016-12-18T04:20:25.047" Score="1" ViewCount="50" Body="&lt;p&gt;Can somebody explain me in a simple language how multiple instance ranking algorithm works? What is ranking function? how mathematically it is expressed?&lt;/p&gt;&#xA;" OwnerUserId="18139" LastActivityDate="2016-12-18T04:20:25.047" Title="Multiple Instance Ranking Algorithm" Tags="|image-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15792" PostTypeId="1" CreationDate="2016-12-18T09:48:32.947" Score="1" ViewCount="49" Body="&lt;p&gt;I run a random forest model on a dataset to assess the credit risk. I found the periodic income and periodic liability were important features in my estimator. Then I try to set a ratio metric which was periodic income/periodic liability which measured the relevant scale on these two variables. Then I found this ratio was also important but the model performance like f1 score, confusion matrix were not improved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case here, does it mean the ratio was redundant and took no extra effect in enhancing the prediction? In math, periodic income , periodic liability and ratio represented different quantitative meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I say in machine learning scope, if Variable A and B were all important from the fitting result, then the ratio of them or even any other algebraic operation on them can be expected as important as well but no extra contribution on the model fitting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone give me advice and thanks heaps.&lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2016-12-18T09:48:32.947" Title="What is effect when I set up some self defined predisctor variables?" Tags="|machine-learning|predictive-modeling|feature-selection|random-forest|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15793" PostTypeId="2" ParentId="15784" CreationDate="2016-12-18T15:21:52.560" Score="3" Body="&lt;p&gt;When you look at the vectors that word2vec generates - negative words may have unique features but can be treated just like positive words. That is to say, as far as the NN is concerned - these are just similar words. You may have to construct &quot;concept vectors&quot; on top of the word vectors to do what you would like to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your parts of speech tagging should automatically mark negating words as ADV. You can then train on these adverbs in conjunction to your verbs as a positive or negative output. Here's an example using spacy:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import spacy&#xA;&#xA;nlp = spacy.load('en')        # this can take a while&#xA;sample_text = u'Do not go.'&#xA;parsed_text = nlp(sample_text)&#xA;token_text = [token.orth_ for token in parsed_text]&#xA;token_pos = [token.pos_ for token in parsed_text]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this point token_text will be a list of your words and token_pos will be the POS tagging:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Do - VERB&#xA;not - ADV&#xA;go - VERB&#xA;. - PUNCT&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, &quot;not&quot; is tagged as ADV here. You can now feed this tagged output (or a better parse tree) into a second network to train for a negative or positive output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="27172" LastActivityDate="2016-12-18T15:21:52.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15794" PostTypeId="1" CreationDate="2016-12-18T15:42:43.527" Score="1" ViewCount="121" Body="&lt;p&gt;I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_data (labeled phrase segments): &#xA;&quot;has watched movie&quot; =&amp;gt; phraselabel1:&quot;movie watch activity&quot;&#xA;&quot;watched with friends X,Y,Z movie&quot; =&amp;gt; phraselabel1:&quot;movie watch activity&quot;&#xA;&quot;watched show&quot; =&amp;gt; phraselabel2: &quot;show watch activity&quot;&#xA;&quot;is jobless&quot; =&amp;gt; phraselabel3: &quot;lack of job&quot;&#xA;&#xA;input phrase segment: &#xA;&quot;watched movie in the past&quot; =&amp;gt; ?? &#xA;&#xA;output:&#xA;recommend ordered set of phrase labels: phraselabel1 then phraselabel3 based on similarity criteria&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to build a (scalable) experimental recommendation system that finds the most similar phrase segments that already have phrase labels to the input unlabeled one and return the best labels based on the similarity. Labels will be added by users to subset of the data initially and the system will help future users with existing labels.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of phrase-labels will be way less than then total number of phrase segments but more granular than topics/tags&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I was thinking of the following approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do TF-IDF (&lt;a href=&quot;http://spark.apache.org/docs/latest/mllib-feature-extraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://spark.apache.org/docs/latest/mllib-feature-extraction.html&lt;/a&gt;) between input phrase segment and all the labeled phrase segments. By finding the most similar I can derive the labels that are best to be suggested&lt;/li&gt;&#xA;&lt;li&gt;Do TF-IDF between input phrase segment and all the phrase-labels. &lt;/li&gt;&#xA;&lt;li&gt;Word2Vec?? I am not sure if it can be used for phrase segments as input and not just words.. &lt;/li&gt;&#xA;&lt;li&gt;Labeled LDA (Latent Dirichlet Allocation (&lt;a href=&quot;https://github.com/myleott/JGibbLabeledLDA&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/myleott/JGibbLabeledLDA&lt;/a&gt;) although I don't think this is readily available in Spark&lt;/li&gt;&#xA;&lt;li&gt;Neural Network (&lt;a href=&quot;https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier&lt;/a&gt;) trained on the words of all input phrase segments as inputs and phraselabels as outputs using the user click feedback on the selected phraselabel for back-propagation training. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would appreciate if you could shed some light on which approach (from above or a new one) is better based on the use case..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="26413" LastEditorUserId="26413" LastEditDate="2016-12-19T09:01:51.300" LastActivityDate="2016-12-19T18:52:16.447" Title="Supervised Recommendation System trained on labeled phrase segments" Tags="|neural-network|apache-spark|recommender-system|supervised-learning|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15795" PostTypeId="1" CreationDate="2016-12-18T17:11:02.450" Score="5" ViewCount="3912" Body="&lt;p&gt;I'm dealing with an imbalanced dataset for binary classification (about 70% to 30%). I was wondering what is the best way to optimize the F1 score for such a task when using a convolutional neural net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As of now, I'm sampling the dataset in order to create a balanced training set and am using the mean of softmax_cross_entropy_with_logits (from tf) with a regularization term as my loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I optimize for the F1 score? As it isn't convex I can't just plug it in as my loss, right? Most papers I found mentioned finding the best threshold. However, these were relatively old, is there something better which can be done for convolutional neural nets ?&lt;/p&gt;&#xA;" OwnerUserId="27196" LastActivityDate="2016-12-19T15:01:36.303" Title="F1 maximization with convolutional neural net. for an imbalanced dataset" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15798" PostTypeId="1" AcceptedAnswerId="15821" CreationDate="2016-12-18T20:12:28.193" Score="2" ViewCount="80" Body="&lt;p&gt;I have done several machine learning projects but all of them have been connected to the traditional machine learning (predictions, classifications, etc.). I have currently been offered a project to finish in less than 6 months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to develop/improve a pre-existing software. The software takes the image of a molecule from an advanced molecule and then tries to highlight the cell line with red, some times the software takes the background or the lines of other cells also as the highlighted part, and thus the user has to manually edit and trim such mistakes. The idea is to make the software learn from user's edits and behavior over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I want to know is whether such a 6-month project is realistic for someone with no background in image processing and pattern recognition? Or is it going to be terribly difficult because I only have had experience with &quot;data-oriented&quot;/statistical machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My other question is; what type of concepts/topics should I dig deep into to learn the fundamentals of carrying out this project?&lt;/p&gt;&#xA;" OwnerUserId="27202" LastActivityDate="2016-12-19T18:43:23.457" Title="What knowledge should I gain for developing a supervised image processing software that learns how to edit photos based on past behavior?" Tags="|machine-learning|deep-learning|image-recognition|" AnswerCount="1" CommentCount="1" ClosedDate="2016-12-20T13:13:01.197" ContentLicense="CC BY-SA 3.0" />
  <row Id="15801" PostTypeId="2" ParentId="15795" CreationDate="2016-12-19T03:20:01.383" Score="2" Body="&lt;p&gt;The problem with directly optimising the F1 score is not that it is non-convex, rather that it is non-differentiable. The surface for any loss function for typical neural networks is highly non-convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do instead, is optimise a surrogate function that is close to the F1 score, or when minimised produces a good F1 score. One way to do this is to simply optimise the cross entropy as usual with weights on the classes - see &lt;a href=&quot;https://stackoverflow.com/questions/35155655/loss-function-for-class-imbalanced-binary-classifier-in-tensor-flow&quot;&gt;this answer on stackoverflow.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-19T03:20:01.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15804" PostTypeId="1" CreationDate="2016-12-19T08:18:22.517" Score="1" ViewCount="20800" Body="&lt;p&gt;I have 10,000 customer data of a supermarket. And I want to split the data into training set and testing set. So, which train test split gives me a better accuracy: 50:50 or 60:40?&lt;/p&gt;&#xA;" OwnerUserId="27211" LastEditorUserId="85045" LastEditDate="2021-01-29T19:16:48.970" LastActivityDate="2021-01-29T19:16:48.970" Title="Which train test split performs better: 50:50 or 60:40?" Tags="|machine-learning|training|performance|" AnswerCount="6" CommentCount="0" ClosedDate="2021-01-29T19:27:03.993" ContentLicense="CC BY-SA 4.0" />
  <row Id="15805" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T08:45:23.013" Score="0" Body="&lt;p&gt;I will assume that the dataset here is being split into training and validation sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you split up your dataset into training and validation sets, you need to take care of that you don't throw away too much data for validation because it is generally seen that with more training data, we get better results in most of the cases. So 50:50 is too bad, 60:40 is fine but not that good. You can make it 80:20 or 75:25 for getting better results.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2016-12-19T08:45:23.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15806" PostTypeId="2" ParentId="15795" CreationDate="2016-12-19T09:38:43.733" Score="4" Body="&lt;p&gt;I have &lt;a href=&quot;http://pong.inescporto.pt/~rpcruz/ijcnn2016/imbalance.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a paper&lt;/a&gt; where we try to maximize F1 score by using different techniques. We hoped that a ranking algorithm like RankNet would be able to get better F1 score than the others. But as you can see looking at our table, regular neural networks, without even using a cost matrix, were good enough. Creating synthetic samples until classes were balanced and even using MetaCost was also largely irrelevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UGp03.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UGp03.png&quot; alt=&quot;F1 maximization table&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We did have a F1 gain when using RankNet, but that was likely due to our post-processing where we convert the ranking score into classes. But that is a pain. I think you can get similar results by, instead of using 0.5 as your threshold for choosing the positive class, choosing a threshold that maximizes the F1 score by just using the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After working on class imbalance for awhile, I think this topic is a great way to get many publications easily, but, in the real world, a post-processing step and/or using a cost matrix to balance priors is more than enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps: Please do not take this results as holy grail. Neural networks are notoriously difficult to optimize, especially across a range of very different datasets. Some cross validation was performed as mentioned in the paper, but, due to lack of time, not as much as we should have. And we did not allow that many iterations. I would try in your own code to introduce weights and some simple things to see if it makes a difference. Do let me know how it works please.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2016-12-19T15:01:36.303" LastActivityDate="2016-12-19T15:01:36.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15807" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T09:47:56.103" Score="2" Body="&lt;p&gt;It sounds like you have a lot of data, so probably a simple train-test split is enough. No need for cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would just use something like 75-25. That is in fact the default value in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use less data in the training only if your algorithm is too slow and cannot cope with the extra data. In that case, instead of throwing away the data, you might as well use it for validation or for testing hyperparameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this being said, more important than all is how you split the data. You should make sure that customers in the testing data are not in the training data to make sure your algorithm is generalizing, and not merely memorizing customers. This is standard procedure in medical data mining, and it is very important. Make sure you do not have customer overlap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to make sure that the distribution of the variable you want to predict is similar in the training and the testing data.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-12-19T09:47:56.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15808" PostTypeId="1" AcceptedAnswerId="15810" CreationDate="2016-12-19T10:53:01.247" Score="0" ViewCount="946" Body="&lt;p&gt;I am writing a musical transcription system with a RNN (LSTM).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input: 1 vector of features per timestep (about 40 timesteps in a second)&lt;/li&gt;&#xA;&lt;li&gt;Output: 1 binary vector of notes per timestep (dimension=36) (1 is on, 0 if off).&lt;/li&gt;&#xA;&lt;li&gt;Model : LSTM(512) + LSTM(256) + Dense(36, activation='sigmoid')&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am currently training on monophonic data (i.e. my output has at most one 1 per timestep).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that because of the sparsity the best strategy to use is to always return 0.&#xA;I tried the loss functions 'mean_squared_error', 'mean_absolute_error', 'binary_crossentropy', 'cosine_proximity' and one custom I wrote with keras :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;K.sum(K.abs(y_pred - y_true), axis=-1) * K.mean(y_pred, axis=-1) / K.maximum(K.mean(y_pred * y_true, axis=-1), K.epsilon())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All those function led with sufficient training to the always zero output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can either change my loss function or my encoding, but the problem is that I need to support polyphonic data, i.e. when there is more than one class to select.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or can I train as many LSTM as there are notes (88 for a piano keyboard), and each one detects if its note is played but I think this is pretty much equivalent, isn't it ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think my data is pretty much equivalent to what keras calls categorical data (binary matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I have only 36 classes, and 24 are always zero.&lt;/p&gt;&#xA;" OwnerUserId="27213" LastEditorUserId="27213" LastEditDate="2016-12-19T11:04:24.213" LastActivityDate="2020-06-03T17:56:49.153" Title="Loss function for sparse tagging" Tags="|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15809" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T10:54:13.590" Score="2" Body="&lt;p&gt;In a draft copy currently being written by Andrew Ng, he discusses about the amount of data in train-test dataset. My understanding from the book, The traditional and most common value is 70-30 or 75-25. If you have 10k or 30k samples, it is fine to go with 70-30 split. But when dealing with Big-data, for example if you have 1 million samples, it is not recommended to have 30k samples as test data, so in that case, 90-10 is actually okay. Because 10k test samples can pretty much provide an intuition about the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in brief: for less samples, go with recommended 70-30 split, for much higher samples, go with number of samples&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Draft copy link : &lt;a href=&quot;http://www.mlyearning.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ML Yearning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2016-12-19T10:54:13.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15810" PostTypeId="2" ParentId="15808" CreationDate="2016-12-19T11:03:26.717" Score="1" Body="&lt;p&gt;In the case of 1 note at a time maximum you could use categorical_crossentropy as loss and add a class for when no note is played. This is a log loss on a last softmax layer. This would turn it into a 37 one-hot-encoding representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of having the possibility of multiple notes I would keep the 36 (or 88 for the pian) dimensional representation and have a sigmoid activation on each of the notes at the end. Then sum the binary log loss of each of the nodes and use that as the loss. Then you can treshold the nodes individually at prediction time to see if a note is on or off.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2016-12-19T11:03:26.717" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15811" PostTypeId="2" ParentId="15767" CreationDate="2016-12-19T11:06:01.773" Score="5" Body="&lt;p&gt;This is just the comment from Emre expanded, but yes you should look into recurrent neural networks for generating text in the style of a given corpus. RNNs and LSTM work really quite well for this.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This writeup&lt;/a&gt; is widely cited, and to your question, shows how it's pretty easy to generate something like this, given the text of Shakespeare's plays:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;PANDARUS: Alas, I think he shall be come approached and the day When&#xA;little srain would be attain'd into being never fed, And who is but a&#xA;chain and subjects of his death, I should not sleep.&lt;/p&gt;&#xA;&lt;p&gt;Second Senator: They are away this miseries, produced upon my soul,&#xA;Breaking and strongly should be buried, when I perish The earth and&#xA;thoughts of many states.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;If you follow &lt;a href=&quot;https://github.com/jcjohnson/torch-rnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; you can easily run this on your local GPU too to generate text from whatever input poetry you like. I have had pretty good results with 0.1-0.5 dropout, 2 layers, layers of size 512-1024.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="98307" LastEditDate="2020-08-02T12:23:25.053" LastActivityDate="2020-08-02T12:23:25.053" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15812" PostTypeId="1" AcceptedAnswerId="15814" CreationDate="2016-12-19T11:22:03.587" Score="12" ViewCount="32104" Body="&lt;p&gt;I have time series of parameters &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;, &lt;code&gt;C&lt;/code&gt; and &lt;code&gt;D&lt;/code&gt;. All of them are under influence of the same major conditions, but each one has minor differences. They are placed in different locations, &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;, &lt;code&gt;C&lt;/code&gt; are in &lt;code&gt;local1&lt;/code&gt; and &lt;code&gt;D&lt;/code&gt; is in &lt;code&gt;local2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know which one (&lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;, &lt;code&gt;C&lt;/code&gt;) has the major similarity to &lt;code&gt;D&lt;/code&gt;. How should I approach this issue? &lt;/p&gt;&#xA;" OwnerUserId="27194" LastActivityDate="2020-12-04T21:54:36.503" Title="Check similarity between time series" Tags="|python|classification|data-mining|time-series|pandas|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15813" PostTypeId="2" ParentId="15812" CreationDate="2016-12-19T11:38:19.507" Score="0" Body="&lt;p&gt;You could just check the &lt;code&gt;RMSE&lt;/code&gt; between &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;,&lt;code&gt;C&lt;/code&gt; to &lt;code&gt;D&lt;/code&gt; and take the minimal one. &lt;/p&gt;&#xA;" OwnerUserId="25905" LastActivityDate="2016-12-19T11:38:19.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15814" PostTypeId="2" ParentId="15812" CreationDate="2016-12-19T11:47:37.357" Score="9" Body="&lt;p&gt;Sounds like a job for &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Time Warping&lt;/a&gt;, there are implementations in Python and R.&lt;/p&gt;&#xA;" OwnerUserId="11007" LastEditorUserId="98307" LastEditDate="2020-08-01T15:43:41.400" LastActivityDate="2020-08-01T15:43:41.400" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15816" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T17:26:28.540" Score="0" Body="&lt;p&gt;Any train-test split which has more data in the training set will most likely give you better accuracy as calculated on that test set. So the direct answer to your question is 60:40. And 99:1 would even give better accuracy...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, at the end of the day, you are not interested in the accuracy on your test set. You are interested in the &quot;real&quot; accuracy, which gets estimated by your test set. And you better want to make sure that the test set can predict that accuracy well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, which split should you pick?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Make sure &quot;enough&quot; data is in the test set. What &quot;enough&quot; means depends on your dataset (the number of classes, number of features)&lt;/li&gt;&#xA;&lt;li&gt;If you rather want a good estimate of your real error, make the test set bigger&lt;/li&gt;&#xA;&lt;li&gt;If you doubt that you can get more training data and you think the more data in your training set will improve your model (in real, not on the test set) a lot, then &quot;sacrifice&quot; a bit for the training set.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Whatever you do, make sure you define your training set before you start your experiments.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-12-19T17:26:28.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15817" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T18:14:32.737" Score="0" Body="&lt;p&gt;I read all answers, I think the simplest answer to this question is based on the understanding of the train - test strategy. There is NO exact or correct answer to this question. Any split that can guarantee I am not under fitting or over fitting the data is a good a split. under fitting and over fitting are two different problems and are directly connected to the bias error and the variance error.&#xA;You are highly recommended to read these two tutorials:&#xA;&lt;a href=&quot;http://scott.fortmann-roe.com/docs/BiasVariance.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding the Bias-Variance Tradeoff&lt;/a&gt; &amp;amp; &lt;a href=&quot;http://scott.fortmann-roe.com/docs/MeasuringError.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Accurately Measuring Model Prediction Error&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="85045" LastEditDate="2021-01-29T15:11:35.367" LastActivityDate="2021-01-29T15:11:35.367" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15818" PostTypeId="2" ParentId="15812" CreationDate="2016-12-19T18:15:00.477" Score="3" Body="&lt;p&gt;First, you should define what you mean with similarity and a corresponding metric to measure it. The second step is to apply the metric to (A, D), (B, D), etc. Then, the set that gives the smallest value is the one with the highest similarity. If you find that your metric does not what you want, simply change it until it meets your requirements. You need to be clear about what you mean with &quot;similarity&quot; and how to measure it though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility is to interpret the sets of parameters as a point in a N-dimensional space and calculate the average euclidean distance. &lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2016-12-19T18:15:00.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15819" PostTypeId="2" ParentId="15804" CreationDate="2016-12-19T18:21:44.963" Score="0" Body="&lt;p&gt;If you have enough data, then you can actually go for a 50-50 split but there is no such thing as what would be better, depends completely on the amount of data you have and the complexity of the task you are trying to perform.If you train it on enough data, the size of the test set is of no concern. The whole reason for splits comes from the fact that we often have limited and finite data and we want to make the best use of it and train on as much data as we can. So go by the heuristic, and do a 75-25 split. But don't forget to cross-validate on the training set, I would recommend a stratified-K-fold. If your performance metric is suffering, the split would be the last reason for it. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-19T18:21:44.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15820" PostTypeId="1" CreationDate="2016-12-19T18:43:18.290" Score="0" ViewCount="2136" Body="&lt;p&gt;Does anyone have a good idea for how to compare topic modeling done by NMF and LDA? Let's say I fit LDA to a dataset and generate topic-word and document-topic distributions--I can use perplexity, for instance, to measure the goodness of fit of the model. If I fit NMF to the same dataset and generate topic-word and document-topic distributions, I can take the resultant fitted data as $X^{Est} = HW^T$ and use $\|X^{Original} - X^{Est}\|_{Fro}$ as a goodness of fit of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is an objective metric that I can use to compare the goodness of fits to each other? Can either of these metrics be used for the other model?&lt;/p&gt;&#xA;" OwnerUserId="24379" LastActivityDate="2016-12-22T00:48:10.143" Title="Goodness of fit metric to compare topic models NMF vs LDA" Tags="|topic-model|lda|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15821" PostTypeId="2" ParentId="15798" CreationDate="2016-12-19T18:43:23.457" Score="-1" Body="&lt;p&gt;From the description of your problem, you need both Computer Vision and Deep Learning for a task like that. It is going to be extremely difficult, but you are more in advantage than anyone else, given you have a strong statistical and machine learning background. You don't have to worry a lot about the image processing part as there are libraries that will do that for you. You can look into &lt;a href=&quot;https://pillow.readthedocs.io/en/3.4.x/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PIL&lt;/a&gt; for that.The hard part is the learning from the edits part.A simpler way to solve this problem would be to focus on image processing and pick out the cell line clearly. The other way would be to train a Convnet on a large collection of 'labelled-images' of the cell-line, so that it is able to identify it in any picture. I do think it is quite a hard problem to solve but do give it a try. Cheers. All the best.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-19T18:43:23.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15822" PostTypeId="2" ParentId="15794" CreationDate="2016-12-19T18:52:16.447" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN-Text-classification&lt;/a&gt;. Try this. Keep in mind that a deep learning algorithm will require a large data-set if you want a good accuracy. Since you want to give ranked recommendations to the users, a soft-max layer at the end will give you the probability of the phrase belonging to all the classes, so you can rank the labels in the order of the probability and suggest them to the user. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first approach is the simplest and best given that you have finite data. I don't think it will be very scalable as your data grows. Also this approach will work best only when your phrases are so limited in their context and vocabulary. Using a convnet you can train on bigger and more ambiguous phrases and get good predictions.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-19T18:52:16.447" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15825" PostTypeId="1" AcceptedAnswerId="15826" CreationDate="2016-12-20T02:25:04.843" Score="1" ViewCount="260" Body="&lt;p&gt;Recently I am working on some predictive analytic which based on neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I tried some tests on MLP with one hidden layer or multiple hidden layers, the results showed that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;one hidden layer performance on prediction is always better than multiple hidden layers&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the tests were executed both on Knime and R models, they gave me the same trends&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Based on my knowledge, more hidden layers with more neurons should perform better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a principle on this about for which kind of dataset turns out this kind of result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or I may need some book / article / paper to read?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any abstract idea (not mathematical algorithms) for me, thanks!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;UPDATE&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am working on a fraud detection dataset, which includes 1000 observations. and all the dimensions are numericed and normalized...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am actually asking for a general idea, for the general idea to choose algorithm for different dataset&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="26856" LastEditDate="2016-12-20T02:46:18.530" LastActivityDate="2016-12-20T03:16:54.297" Title="An abstract idea for the performance diffs between SLP and MLP" Tags="|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15826" PostTypeId="2" ParentId="15825" CreationDate="2016-12-20T02:52:14.757" Score="0" Body="&lt;p&gt;More layers/more neurons does not necessarily mean you will get better performance. If your data is too simple or the number of observations is not that high, then adding more parameters (more layers/neurons) may result in overfitting the data. During training, the network will try to represent the training data as closely as possible. When there are a good number of units in the network, then the representation that is learned by the network will mainly model the general trend of the data faithfully. But, if there are too many neurons, then it is possible that some of the neurons will simply model noise in the training data which does not generalise to unseen data, resulting in worse performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you are using a saturating nonlinearity as an activation function (i.e. sigmoids or tanh) then adding too many layers may result in vanishing gradients which will cause your network to train very slowly, or perhaps not at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Designing neural networks that work well is not very easy, and the best way to get an intuition for what will work is to experiment. One thing to try is to evaluate a range of sizes for your hidden layer(s). If you are using sigmoid or tanh as your activation function, I suggest you try using rectified linear units as well.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2016-12-20T03:16:54.297" LastActivityDate="2016-12-20T03:16:54.297" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15830" PostTypeId="1" CreationDate="2016-12-20T11:35:03.197" Score="1" ViewCount="520" Body="&lt;p&gt;when I look into the &lt;em&gt;R package&lt;/em&gt; &lt;code&gt;pmml&lt;/code&gt;, I found that, it is possible to directly export &lt;code&gt;nnet&lt;/code&gt; model into &lt;code&gt;PMML&lt;/code&gt; format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But actually I am using &lt;code&gt;neuralnet&lt;/code&gt; package since it has more features and flexibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I export &lt;code&gt;neuralnet&lt;/code&gt; model to &lt;code&gt;PMML&lt;/code&gt; format, I checked &lt;code&gt;pmml&lt;/code&gt; package, it doesn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other options like transform &lt;code&gt;neuralnet&lt;/code&gt; model to &lt;code&gt;nnet&lt;/code&gt; model, or some other &lt;code&gt;PMML&lt;/code&gt; packages to use? thanks&lt;/p&gt;&#xA;" OwnerUserId="26856" LastActivityDate="2016-12-23T10:44:23.013" Title="export R neuralnet package model to PMML" Tags="|r|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15831" PostTypeId="1" AcceptedAnswerId="15875" CreationDate="2016-12-20T13:31:31.580" Score="3" ViewCount="507" Body="&lt;p&gt;Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;feat. 1&lt;/li&gt;&#xA;&lt;li&gt;feat. 2&lt;/li&gt;&#xA;&lt;li&gt;feat. 3&lt;/li&gt;&#xA;&lt;li&gt;feat. 4. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, whereas for dataset 2 I have for example the following ranking: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;feat. 3&lt;/li&gt;&#xA;&lt;li&gt;feat. 1&lt;/li&gt;&#xA;&lt;li&gt;feat. 2&lt;/li&gt;&#xA;&lt;li&gt;feat. 4. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to filter out those features which end up in the top of the ranking the most (incorporating that finishing in the top is better than finishing in the 3rd place). Which kind of ranking metric can I use for this problem? &lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="14372" LastEditDate="2016-12-22T09:57:19.083" LastActivityDate="2016-12-23T15:06:24.727" Title="Rank feature selection over multiple datasets" Tags="|machine-learning|classification|feature-selection|ranking|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15832" PostTypeId="1" CreationDate="2016-12-20T13:41:54.690" Score="4" ViewCount="1964" Body="&lt;p&gt;I want to compare two addresses for their similarity (purely similarity in terms of textual occurrences rather than semantic similarity). I have a labelled dataset having &quot;Address1&quot;, &quot;Address2&quot; and &quot;Similar&quot;, where &quot;Similar&quot; takes a value of 0 (not similar) or 1 (similar). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, how would I convert the text in &quot;Address1&quot; and &quot;Address2&quot; into numerical vectors, so as to treat this as a binary classification problem?&lt;/p&gt;&#xA;" OwnerUserId="22059" LastActivityDate="2017-01-05T00:21:40.840" Title="How to create vectors from text for address matching using binary classification?" Tags="|classification|text-mining|similarity|feature-engineering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15833" PostTypeId="1" CreationDate="2016-12-20T15:02:36.753" Score="0" ViewCount="3240" Body="&lt;p&gt;Im trying to learn and get the hang of tflearn.  One thing that I have tried to do is save my model, and then reload it.  I get an error that says:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for quicktest.model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Below I have my model as well as that checkpoint file that gets created with the saving of a model.  I have googled this error and tried many different file changes but nothing has fixed it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import tflearn&#xA;    data, labels = load_csv('my_csv', target_column=0, categorical_labels=True, n_classes=2)&#xA;    net = tflearn.input_data(shape=[None, 32])&#xA;    net = tflearn.fully_connected(net, 32)&#xA;    net = tflearn.fully_connected(net, 2, activation='softmax')&#xA;    net = tflearn.regression(net)&#xA;    model = tflearn.DNN(net)&#xA;    model.fit(data, labels, n_epoch=1, batch_size=10, show_metric=True)&#xA;    model.save('quicktest.model')&#xA;    model.load('quicktest.model')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;checkpoint&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_checkpoint_path: &quot;quicktest.model&quot;&#xA;all_model_checkpoint_paths: &quot;quicktest.model&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26800" LastActivityDate="2017-05-19T17:13:15.480" Title="TFLearn Does Not Load Model" Tags="|python|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15834" PostTypeId="2" ParentId="15833" CreationDate="2016-12-20T15:10:44.170" Score="1" Body="&lt;p&gt;Found out what the issue was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.load('filename')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should actually be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.load('./filename')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hopefully this helps out someone else.&lt;/p&gt;&#xA;" OwnerUserId="26800" LastActivityDate="2016-12-20T15:10:44.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15836" PostTypeId="2" ParentId="15832" CreationDate="2016-12-20T15:27:56.473" Score="2" Body="&lt;p&gt;I would rather recommend to consider each address as a separate instance. (Separate row in feature matrix)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each address can be further broken into attributes like: House no., Street, City, State, Country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Above attributes can be vectorised based on distance between them. Like vectorise each country from west to east(incremental numeric value). Then vectorise each state in country from west to east. Similarly for street. In case two entities fall on same longitude, assign consecutive numerical values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once each attribute is vectorised, similarity between two addresses can be Euclidean distance between vectors of two addresses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways leaving aside data science approach, this problem can be solved further simply by getting distance between two addresses using Google Maps API.&lt;/p&gt;&#xA;" OwnerUserId="26415" LastActivityDate="2016-12-20T15:27:56.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15839" PostTypeId="1" CreationDate="2016-12-20T19:22:03.547" Score="2" ViewCount="1825" Body="&lt;p&gt;I am using &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt; library to compute similarity between documents but it only uses cosine similarity. I was wondering if there was a way to use jaccard similarity or any other similarity measure for that matter instead &lt;/p&gt;&#xA;" OwnerUserId="2504" LastActivityDate="2016-12-21T14:02:33.893" Title="Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?" Tags="|machine-learning|data-mining|similarity|gensim|cosine-distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15840" PostTypeId="2" ParentId="14432" CreationDate="2016-12-20T20:24:51.810" Score="5" Body="&lt;p&gt;It is very unlikely for such huge dataset(450k ) to overfit after just one epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to run the code for 20-30 epochs and see if you see any decrease in the validation set error (increase in validation accuracy.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to change your learning rate if (unless you are using adaptive learning parameter)&lt;/p&gt;&#xA;" OwnerUserId="27251" LastActivityDate="2016-12-20T20:24:51.810" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15841" PostTypeId="1" AcceptedAnswerId="15843" CreationDate="2016-12-20T20:32:03.843" Score="1" ViewCount="1776" Body="&lt;p&gt;Being new to theano, pls bear with me. I thought the shape of the tensor variable is already well defined out of the Conv2D layer since the input is specified, as follow,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Input, Convolution2D&#xA;import theano&#xA;input_img = Input(shape=(1, 28, 28))&#xA;x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')   (input_img)&#xA;print type(x)&#xA;print theano.tensor.shape(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the output is,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;class 'theano.tensor.var.TensorVariable'&amp;gt;&#xA;Shape.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since I'm taking the default stride of 1, and &lt;code&gt;same&lt;/code&gt; border mode here means that padding is added so the output is the same as input. Using this information I could calculate by hand what the output shape should be.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did I miss something here? The question is how to get the shape of the output of a convolution layer?&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="29169" LastEditDate="2021-02-16T15:27:57.217" LastActivityDate="2021-02-16T15:27:57.217" Title="shape of theano tensor variable out of keras Conv2D" Tags="|keras|convolutional-neural-network|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15842" PostTypeId="1" CreationDate="2016-12-20T21:27:06.570" Score="1" ViewCount="715" Body="&lt;p&gt;Training a model in keras can go very fast. I trained the following model. To train the model with 1 epoch finishes in a few second. However, using 20 epochs it lasts half an hour or so? An epoch means how often each sample in the training set used to train the model. Therefore, I would naively expect a linear dependency. Which factors do lead to a non-linear behavior?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def keras_model(X_train, y_train):&#xA;    from keras.models import Sequential&#xA;    from keras.layers import Dense, Dropout, Activation&#xA;    from keras.optimizers import SGD&#xA;&#xA;    model = Sequential()&#xA;    # Dense(64) is a fully-connected layer with 64 hidden units.&#xA;    # in the first layer, you must specify the expected input data shape:&#xA;    # here, 20-dimensional vectors.&#xA;    model.add(Dense(63, input_dim=63, init='uniform'))&#xA;    model.add(Activation('tanh'))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(63, init='uniform'))&#xA;    model.add(Activation('tanh'))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(22, init='uniform'))&#xA;    model.add(Activation('softmax'))&#xA;&#xA;    sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)&#xA;&#xA;    model.compile(loss='categorical_crossentropy',&#xA;                  optimizer=sgd,&#xA;                  metrics=['accuracy'])&#xA;&#xA;    model.fit(X_train, convert_labels_to_matrix(y_train),&#xA;              nb_epoch=20, batch_size=15)&#xA;&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2016-12-20T21:27:06.570" Title="Does the time to train a model using keras increase linear with epoches?" Tags="|python|deep-learning|tensorflow|keras|theano|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15843" PostTypeId="2" ParentId="15841" CreationDate="2016-12-20T23:31:48.390" Score="1" Body="&lt;p&gt;You can't get the shape of a theano tensor, because it is not fixed. The output of the convolutional layer is just a symbolic variable and its shape depends on whatever you put into the layer as input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get the shape of the output for a specific input by making a theano function for the output of the layer, and feeding a numpy array through the function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;input = np.ones(28*28).reshape(1, 1, 28, 28).astype('float32')&#xA;fn = theano.function([input_img], x)&#xA;&#xA;print fn(input).shape&#xA;&amp;gt;&amp;gt;&amp;gt; (1, 16, 28, 28)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-20T23:31:48.390" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15845" PostTypeId="2" ParentId="15832" CreationDate="2016-12-21T04:29:46.393" Score="0" Body="&lt;p&gt;Mainly to add on to the previous answer, firstly what do you mean by similarity in textual occurences, do you mean what other words they frequently occur or you want to recognize how closely a piece of text looks like an address. If you are just looking for a metric to compare addresses, then the best thing would be to use the Geo Location API mentioned in the previous answer. If its just about converting them to vectors, you can take the entire vocabulary and assign an index to each token, if in the future you need semantic similarity you can apply a word2vec to the indexes. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-21T04:29:46.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15847" PostTypeId="1" CreationDate="2016-12-21T08:26:43.100" Score="1" ViewCount="3340" Body="&lt;p&gt;Once we know that the problem needs to be solved using supervised learning, how do we know if we have to use regression analysis or classification algorithm to solve it? Is there some thumb rules that can be used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the resources online give the standard example of prices of house for regression model and malignant/benign cancer plot for classification model. This isn't helping me much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there come conceptual method to analyze the given problem?&lt;/p&gt;&#xA;" OwnerUserId="27261" LastEditorUserId="81999" LastEditDate="2020-03-11T14:40:28.890" LastActivityDate="2020-03-11T14:40:28.890" Title="How to tell if a problem should use regression or classification model?" Tags="|machine-learning|classification|regression|supervised-learning|" AnswerCount="4" CommentCount="2" ClosedDate="2020-03-11T16:02:52.510" ContentLicense="CC BY-SA 3.0" />
  <row Id="15849" PostTypeId="2" ParentId="15847" CreationDate="2016-12-21T09:10:40.920" Score="0" Body="&lt;p&gt;Normally the cases in which you use a &lt;strong&gt;Regression model&lt;/strong&gt; is when you want to predict a &lt;strong&gt;continuous&lt;/strong&gt; value from a set of given independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;E.g :&lt;/strong&gt;  Let the following values be of the type [independent_variable, dependent_variable] or simply $[marks,height]$ and the values be $[2,0],[3,2],[4,5],[1,1]$. You fit a line or curve through these values ($[2,0],[3,2]$ etc.) and then see the case when a value of $[10,y]$ is given or marks of $10$ are obtained, what can be the $y$ (height) value from the fitted line or curve you had modeled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Linear Regression&lt;/a&gt; for the above type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Classification model&lt;/strong&gt; is used in the case when you got a set of independent variables as in the previous case but the dependent value used in training is not continuous value but tells what class the value belong to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;E.g :&lt;/strong&gt; $([2,1],fail),([3,2],fail),([4,5],pass),([1,2],fail)$. Here [2,1] belongs to class $fail$ etc. So later time when a point say [7,8] is given, you will be finding which class (pass or fail) it might probably belong to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM's&lt;/a&gt; for this case create a hyperplane(a multidimensional plane) and based on where the points falls in the space, it is going to find the class with some probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply, choose Regression if the dependent value is continuous else choose Classification if the dependent value is a class.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-12-21T09:10:40.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15850" PostTypeId="2" ParentId="15847" CreationDate="2016-12-21T09:14:16.513" Score="7" Body="&lt;p&gt;A good rule of thumb is to look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;noreferrer&quot;&gt;level of measurement&lt;/a&gt; of the target/response variable. If the response is measured on a nominal scale, the problem is a classification problem. Values on a nominal scale are for example labels of a categories where the categories have no natural order, like political parties in political science, species in biology, or parts-of-speech in grammar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the response is measured on a ratio or interval scale, you have a regression problem. Values on an interval scale are values where you can compare the degree of difference between values, but not the ratio between them, for instance temperature (on Farenheit or Celsius scales, but not Kelvin), or date values in a calendar. Values on ratio scales can be compared both with regards to degree of difference and ratio, like most physical quantities like mass, velocity or temperature on the Kelvin scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinal scales are more difficult to place in either corner. I would generally say that you have a ranking problem with an ordinal response. However, the ranking problem can be approached using both classification, for instance using comparators, and and regression, like ordinal regression. Values on ordinal scales are ordered, or ranked, but you can't say anything meaningful about the degree of difference between any two values, for instance the ranking of racing drivers in a race.&lt;/p&gt;&#xA;" OwnerUserId="27229" LastEditorUserId="27229" LastEditDate="2016-12-21T10:15:28.980" LastActivityDate="2016-12-21T10:15:28.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15852" PostTypeId="1" CreationDate="2016-12-21T12:17:53.973" Score="2" ViewCount="85" Body="&lt;p&gt;Imagine I'm conducting an ongoing poll asking people's favourite animal out of a list of animals, &lt;code&gt;[cat, dog, penguin, chimpanzee, ...]&lt;/code&gt; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to provide an interface that lets people query this poll data to see the relative popularity of each animal by different demographics. For example, querying the general population might reveal the plurality of respondents (36%) prefer penguins, but querying the 18-25 age-bracket might the plurality of respondents in that cohort (41%) prefer cats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's desirable to preserve the privacy of my respondents' animal preferences as much as possible. However, an attacker may be able to use prior knowledge of a given respondent to deduce their response by asking a specific enough series of queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to limit an attacker's ability to do this by noisifying the data presented to those querying the data. As such, I want a procedure that pseudorandomly adds or removes a fraction of a percentage point from each category, but preserves their relative ordering. I also wish this procedure to be deterministic over the same set of data (though this can easily be achieved by using a fixed seed in the pseudorandom procedure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, I want&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f : \mathbb R_{&amp;gt;0}^n \rightarrow \mathbb R_{&amp;gt;0}^n;\;\; f(\mathbf{x}) = \mathbf{y}; \;\; |\mathbf{x}| = |\mathbf{y}| = 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbf{x}$ is the vector of proportions of each category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One naive way of going about this would be to simply add a pseudorandom Gaussian noise vector to the original vector and then renormalise. This poses at least two problems:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) the &quot;zero problem&quot;: if a cohort has zero people who like cats, how should the noisification procedure treat this? I'm inclined to say it should maintain the value at zero, but I can't think of a principled way of achieving this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) the variance of the noise should ideally be the same for all elements in the vector, but any obvious procedure for forcing positivity would all typically result in smaller variance of noise for smaller values, so the noisification would end up making large values larger and small values smaller after renormalisation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like this should be a problem people have encountered before, but I can't find it in the literature.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2016-12-21T12:17:53.973" Title="Noisification of categorical data proportions for privacy-preservation" Tags="|noisification|privacy|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15854" PostTypeId="2" ParentId="15839" CreationDate="2016-12-21T14:02:33.893" Score="2" Body="&lt;p&gt;If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://radimrehurek.com/gensim/models/word2vec.html&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$ model['computer']  # raw numpy vector of a word&#xA;array([-0.00449447, -0.00310097,  0.02421786, ...], dtype=float32)&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is possible to manually implement any vector comparison that you choose.  Cosign similarity is typically chosen because it performs relatively well when compared to other methods of grouping high dimensional projections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I could envision implementing Jaccard Similarity would be to identify a list of key words on a per document basis, and when comparing document, include words that are synonyms as intersections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on reviewing the gensim document comparison text (&lt;a href=&quot;https://radimrehurek.com/gensim/similarities/docsim.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://radimrehurek.com/gensim/similarities/docsim.html&lt;/a&gt;) I don't believe there is a native implementation.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2016-12-21T14:02:33.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15855" PostTypeId="2" ParentId="11308" CreationDate="2016-12-21T14:49:32.940" Score="1" Body="&lt;p&gt;You might have a &lt;a href=&quot;https://en.wikipedia.org/wiki/Domain_adaptation&quot; rel=&quot;nofollow noreferrer&quot;&gt;domain adaptation&lt;/a&gt; problem.&#xA;The samples are actually taken from two different sources and have different rules governing their behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you'll try the following method.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Split the data into two data set using the target concept.&lt;/li&gt;&#xA;&lt;li&gt;Build a model for each data set. In case your performance doesn't improve it is likely that the domain adaptation assumption is wrong. &lt;/li&gt;&#xA;&lt;li&gt;Otherwise, we gain an improvement. Now we should find out which model to apply. Use the full dataset and use the target in order to create a new concept indicating to which dataset the sample belongs. Since you said that given the target you can deduce that I assume the two are different.&lt;/li&gt;&#xA;&lt;li&gt;Build a model for the new dataset. If it performs well, the problem is solved. Run this model first and apply the proper model for each sample. Otherwise, it means that the two data set are indistinguishable (at least with respect to the methods you applied), weakening the assumption the the cause of the problem is domain adaptation.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-12-21T14:49:32.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15856" PostTypeId="2" ParentId="15847" CreationDate="2016-12-21T15:12:37.143" Score="0" Body="&lt;p&gt;Are the target values ordered? Then it is likely to be regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://en.wikipedia.org/wiki/Level_of_measurement#Interval_scale&quot; rel=&quot;nofollow noreferrer&quot;&gt;Level of measurement&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Examples&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Regression&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Price prediction&lt;/li&gt;&#xA;&lt;li&gt;Rating prediction&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Classification&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Predict the gender of a user&lt;/li&gt;&#xA;&lt;li&gt;Predict the class (dog, cat, house, airplane, ...) of an object in an image&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I'd say for two classes it doesn't really make a big difference. In the case of predicting the gender of a user, you could also predict the probability of the user being female. It's just a minor variation of the same problem (ignoring &lt;a href=&quot;https://medium.com/gender-2-0/falsehoods-programmers-believe-about-gender-f9a3512b4c9c#.77b5ni8g0&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2016-12-21T15:12:37.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15857" PostTypeId="2" ParentId="15847" CreationDate="2016-12-21T16:29:51.823" Score="1" Body="&lt;p&gt;Learning from a set of examples &lt;code&gt;x&lt;/code&gt; mapping to &lt;code&gt;y&lt;/code&gt; can be conceptualised as finding function &lt;code&gt;f&lt;/code&gt; such that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = f(x)&#xA;x is vector of features, for e.g., car_model, car_version, city as vector of features for price prediction of used car.&#xA;y is output variable, for e.g., price of car x sold at.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if &lt;code&gt;y&lt;/code&gt; is continuous, the problem is a regression problem&#xA;else if &lt;code&gt;y&lt;/code&gt; is discrete, the problem is a classification problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Continuous implies &lt;code&gt;y&lt;/code&gt; can take any value [i, j] on Real scale and discrete implies &lt;code&gt;y&lt;/code&gt; can take a value from set of {a, b, ..., d}&lt;/p&gt;&#xA;" OwnerUserId="26415" LastActivityDate="2016-12-21T16:29:51.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15860" PostTypeId="1" AcceptedAnswerId="15877" CreationDate="2016-12-21T19:43:06.103" Score="3" ViewCount="503" Body="&lt;p&gt;I think I get the main idea, and I almost understand the derivation except for this one line, see picture below: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IzNHf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IzNHf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.dropbox.com/sh/3xowt7qvyadvejn/AABpWQMKWX3KVbeqVlBcxNYra/slides%20(pdf%20and%20keynote)?dl=0&amp;amp;preview=13-multistep.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.dropbox.com/sh/3xowt7qvyadvejn/AABpWQMKWX3KVbeqVlBcxNYra/slides%20(pdf%20and%20keynote)?dl=0&amp;amp;preview=13-multistep.pdf&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insight. I could be missing something simple/obvious as I dive into these details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: &lt;a href=&quot;http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27279" LastEditorUserId="836" LastEditDate="2017-04-04T20:27:21.247" LastActivityDate="2017-04-04T20:27:21.247" Title="Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm" Tags="|machine-learning|statistics|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15862" PostTypeId="1" AcceptedAnswerId="15866" CreationDate="2016-12-21T23:50:51.620" Score="6" ViewCount="24509" Body="&lt;p&gt;I am trying to understand the difference between Jaccard and Cosine. However, there seem to be a disagreement in the answers provided in &lt;a href=&quot;https://datascience.stackexchange.com/questions/5121/applications-and-differences-for-jaccard-similarity-and-cosine-similarity&quot;&gt;Applications and differences for Jaccard similarity and Cosine Similarity&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am seeking if anyone could step me through the calculations of the Jaccard Similarity in this Cosine Similarity example from &lt;a href=&quot;https://bioinformatics.oxfordjournals.org/content/suppl/2009/10/24/btp613.DC1/bioinf-2008-1835-File004.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://bioinformatics.oxfordjournals.org/content/suppl/2009/10/24/btp613.DC1/bioinf-2008-1835-File004.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/G6zAc.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/G6zAc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: &lt;strong&gt;&lt;em&gt;How do we compute the Jaccard Similarity index between t1 and t2?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2016-12-22T01:58:26.330" Title="How to compute the Jaccard Similarity in this example? (Jaccard vs. Cosine)" Tags="|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15864" PostTypeId="1" AcceptedAnswerId="16059" CreationDate="2016-12-22T00:44:50.120" Score="6" ViewCount="7494" Body="&lt;p&gt;The simplest example is to have faster/slower learning rates in the upper/lower layers of a network. I found &lt;a href=&quot;https://stackoverflow.com/questions/34945554/how-to-set-layer-wise-learning-rate-in-tensorflow&quot;&gt;this post&lt;/a&gt; on tensorflow. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there a similar trick in Keras?&lt;/li&gt;&#xA;&lt;li&gt;Going one step further, can we set different learning rates for specific range/set of neurons/weights in a particular layer?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="29575" LastEditDate="2019-06-09T13:39:30.080" LastActivityDate="2019-06-09T13:39:30.080" Title="Possible to use different learning rate for different neuron in Keras/Tensorflow?" Tags="|deep-learning|tensorflow|keras|training|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="15865" PostTypeId="2" ParentId="15820" CreationDate="2016-12-22T00:48:10.143" Score="3" Body="&lt;p&gt;NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document clustering/classification (i.e. evaluating document-topic distributions) or information retrieval (i.e. evaluating topic-word and document-topic distributions together). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See some examples in &quot;&lt;a href=&quot;https://transacl.org/ojs/index.php/tacl/article/view/582/158&quot; rel=&quot;nofollow noreferrer&quot;&gt;Improving topic models with latent feature word representations&lt;/a&gt;&quot;, &quot;&lt;a href=&quot;https://ai2-s2-pdfs.s3.amazonaws.com/0ef3/11acf523d4d0e2cc5f747a6508af2c89c5f7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA-Based Document Models for Ad-hoc Retrieval&lt;/a&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="18164" LastActivityDate="2016-12-22T00:48:10.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15866" PostTypeId="2" ParentId="15862" CreationDate="2016-12-22T00:56:54.587" Score="5" Body="&lt;p&gt;Cosine similarity is for comparing two &lt;em&gt;real-valued vectors&lt;/em&gt;, but Jaccard similarity is for comparing two &lt;em&gt;binary vectors (sets)&lt;/em&gt;. So you cannot compute the standard Jaccard similarity index between your two vectors, but there is a generalized version of the Jaccard index for real valued vectors which you can use in this case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$J_g(\Bbb{a}, \Bbb{b}) =\frac{\sum_i min(\Bbb{a}_i, \Bbb{b}_i)}{\sum_i &#xA;max(\Bbb{a}_i, \Bbb{b}_i)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for your examples of $t_1 = (1, 1, 0, 1), t_2 = (2, 0, 1, 1)$, the generalized Jaccard similarity index can be computed as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$J(t_1, t_2) = \frac{1+0+0+1}{2+1+1+1} = 0.4$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you can treat your bag-of-words vector as a binary vector, where a value $1$ indicates a words presence and $0$ indicates a words absence &lt;em&gt;i.e.&lt;/em&gt; $t_1 = (1, 1, 0, 1), t_2 = (1, 0, 1, 1)$. From there, you can compute the original Jaccard similarity index:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$J(t_1, t_2) = \frac{2}{2+1+1} = 0.5$&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2016-12-22T01:58:26.330" LastActivityDate="2016-12-22T01:58:26.330" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15867" PostTypeId="1" AcceptedAnswerId="15889" CreationDate="2016-12-22T03:11:47.320" Score="0" ViewCount="216" Body="&lt;p&gt;Im looking for an algorithm that can deduct a set of rules based on a dataset of &quot;training documents&quot; that can be applied to classify a new unseen document. The problem is that I need these rules to be viewable by the user in the form of some string representation. For example, the algorithm found that documents have a minimum word count of 1000 and that there are 4 citations in each document. The key is that these rules must be deducted by a algorithm. An example of this in practice would be:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Document 1 contains 890 words and only 2 citations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need it to return something like:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;- You should add more words to make it better&lt;br&gt;&#xA;- Add more citations to prove your point&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="27286" LastActivityDate="2016-12-22T22:01:19.990" Title="Algorithm for generating rules for classifying documents" Tags="|machine-learning|classification|data-mining|nlp|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15868" PostTypeId="2" ParentId="15867" CreationDate="2016-12-22T04:14:23.107" Score="0" Body="&lt;p&gt;I think a simplest way is like this: convert the document to a vector like &quot;bag of words&quot; or &quot;n-grams&quot;, then apply algorithm which can be used to derive rules Decision tree or LEM2. &lt;/p&gt;&#xA;" OwnerUserId="27287" LastActivityDate="2016-12-22T04:14:23.107" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15869" PostTypeId="1" AcceptedAnswerId="15873" CreationDate="2016-12-22T05:52:30.113" Score="2" ViewCount="962" Body="&lt;p&gt;Is there a website where people store their pickled models for others to try? E.g. different people might try different ML approaches on the iris dataset, is there a place where I can find/download models others have constructed?&lt;/p&gt;&#xA;" OwnerUserId="12985" LastActivityDate="2021-02-28T00:53:38.747" Title="Pickled machine learning models" Tags="|machine-learning|python|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15871" PostTypeId="2" ParentId="11308" CreationDate="2016-12-22T08:21:32.603" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensembles&lt;/a&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This consists of building several classifiers, each of them acting particularly well in a subset of the data, and the combine their predictions into the final prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your description, &lt;a href=&quot;https://en.wikipedia.org/wiki/Boosting_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;boosting algorithms&lt;/a&gt; seem to be good candidates, e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow noreferrer&quot;&gt;AdaBoost&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-12-22T08:21:32.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15873" PostTypeId="2" ParentId="15869" CreationDate="2016-12-22T09:31:49.590" Score="3" Body="&lt;p&gt;On Kaggle.com, there is a &lt;a href=&quot;https://www.kaggle.com/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;datasets section&lt;/a&gt;. In this section, you can post or find datasets. For each dataset, you have the data, possibility to discuss about it and, of course, a list of kernels posts by the user. For instance, for the iris dataset, you have &lt;a href=&quot;https://www.kaggle.com/uciml/iris/kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;592 kernels&lt;/a&gt; available.&lt;/p&gt;&#xA;" OwnerUserId="27003" LastActivityDate="2016-12-22T09:31:49.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15874" PostTypeId="1" CreationDate="2016-12-22T11:02:52.763" Score="0" ViewCount="785" Body="&lt;p&gt;Here is a DF which I take as a representation of my future vertices:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Id   Name&#xA;01   Alpha&#xA;02   Beta&#xA;03   Delta&#xA;04   Epsilon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to get an igraph representation of a full graph with these nodes in R. As far as I know igraph's &lt;code&gt;graph.full&lt;/code&gt; function cannot take DF as input and I cannot really want to create an intermediary DF with all possible conection combination. Any clever idea?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-02-20T16:14:06.467" Title="How to create full graph from vertex data frame with igraph?" Tags="|r|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15875" PostTypeId="2" ParentId="15831" CreationDate="2016-12-22T11:14:13.517" Score="2" Body="&lt;p&gt;An easy one to try would be average ranks, where you take the mean of the ranks for each feature. For your example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;\begin{array}{cc}&#xA;  \textbf{Feature} &amp;amp; \textbf{Avg. Rank} \\&#xA;  1 &amp;amp; 1.5 \\&#xA;  3 &amp;amp; 2 \\&#xA;  2 &amp;amp; 2.5 \\&#xA;  4 &amp;amp; 4 \\&#xA;\end{array}&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also weight the ranks by the size of the dataset, if the datasets that you are testing on are not the same size.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-22T11:14:13.517" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15876" PostTypeId="2" ParentId="15874" CreationDate="2016-12-22T14:11:55.797" Score="1" Body="&lt;p&gt;All full graphs of the same order and direction are the same, so just make a full graph of the right size and then put your vertex data on the vertices:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d = data.frame(Id=c(&quot;01&quot;,&quot;02&quot;,&quot;03&quot;,&quot;04&quot;),Name=c(&quot;Alpha&quot;,&quot;Beta&quot;,&quot;Delta&quot;,&quot;Epsilon&quot;),stringsAsFactors=FALSE)&#xA;&amp;gt; d&#xA;  Id    Name&#xA;1 01   Alpha&#xA;2 02    Beta&#xA;3 03   Delta&#xA;4 04 Epsilon&#xA;&amp;gt; g = graph.full(nrow(d))&#xA;&amp;gt; V(g)$Name = d$Name&#xA;&amp;gt; V(g)$Id = d$Id&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="471" LastActivityDate="2016-12-22T14:11:55.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15877" PostTypeId="2" ParentId="15860" CreationDate="2016-12-22T14:37:55.627" Score="3" Body="&lt;p&gt;The slides and the book are consistent. Notice how in the slides there is a restriction in the summatory: i.e.  $a \neq A_{t+1}$. For $G^{(2)}$, you need to &quot;remove&quot; from $V_{t+1}$ the term that should not be there, i.e. $A_{t+1}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, why this term is removed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you keep this term you will be adding $A_{t+1}$ twice. In 1-step backup, it is part of the expectation of step $S_{t+1}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you calculate 2-step backup you want to replace $(S_{t+1}, A_{t+1})$ in the 1-step expectation with the discounted expected value of $S_{t+2}$.&#xA;So you substract the term and add the discounted expectation for $S_{t+2}$&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2016-12-22T18:41:32.967" LastActivityDate="2016-12-22T18:41:32.967" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15878" PostTypeId="1" AcceptedAnswerId="16314" CreationDate="2016-12-22T15:06:47.067" Score="2" ViewCount="1712" Body="&lt;p&gt;When explaining advantage function, it is usually claimed that using a baseline reduces the variance. I have not found any specific reference to justify this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this an application of control variates or something similar?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone provide some reference or formal justification for the variance reduction?&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2017-01-14T10:53:21.350" LastActivityDate="2022-09-25T12:32:58.767" Title="Advantage Function - Variance Reduction" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15879" PostTypeId="2" ParentId="15869" CreationDate="2016-12-22T15:59:25.270" Score="3" Body="&lt;p&gt;Pickled models probably will not work on a different computer. It is also a very insecure format &lt;a href=&quot;https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;(see here)&lt;/a&gt;. Rather go to kaggle.com or github.com and look for kernels/scripts that allow you to train a model on your own. Some frameworks, such as XGBoost, have a build in saving functionality with is more reliable than pickle.&lt;/p&gt;&#xA;" OwnerUserId="26202" LastEditorUserId="26202" LastEditDate="2017-01-10T09:58:50.140" LastActivityDate="2017-01-10T09:58:50.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15880" PostTypeId="2" ParentId="13029" CreationDate="2016-12-22T16:06:17.933" Score="4" Body="&lt;p&gt;This was answered in &lt;a href=&quot;https://stats.stackexchange.com/questions/184657/difference-between-off-policy-and-on-policy-learning&quot;&gt;cross-validated&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/q/6848828/674487&quot;&gt;stackoverflow&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state &lt;span class=&quot;math-container&quot;&gt;$s′$&lt;/span&gt; and the greedy action &lt;span class=&quot;math-container&quot;&gt;$a′$&lt;/span&gt;. In other words, it estimates the return (total discounted future reward) for state-action pairs assuming a greedy policy were followed despite the fact that it's not following a greedy policy.&lt;/p&gt;&#xA;&lt;p&gt;The reason that SARSA is on-policy is that it updates its Q-values using the Q-value of the next state &lt;span class=&quot;math-container&quot;&gt;$s′$&lt;/span&gt; and the current policy's action &lt;span class=&quot;math-container&quot;&gt;$a′′$&lt;/span&gt;. It estimates the return for state-action pairs assuming the current policy continues to be followed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.inf.ed.ac.uk/teaching/courses/rl/slides15/rl05.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;These slides&lt;/a&gt; offer some insight on pros and cons of each one:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;On-policy methods:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;attempt to evaluate or improve the policy that is used to make decisions,&lt;/li&gt;&#xA;&lt;li&gt;often use soft action choice, i.e. &lt;span class=&quot;math-container&quot;&gt;$\pi(s,a) &amp;gt;0,  \forall a$&lt;/span&gt;,&lt;/li&gt;&#xA;&lt;li&gt;commit to always exploring and try to find the best policy that still explores,&lt;/li&gt;&#xA;&lt;li&gt;may become trapped in local minima.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Off-policy methods:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;evaluate one policy while following another, e.g. tries to evaluate the greedy policy while following a more exploratory scheme,&lt;/li&gt;&#xA;&lt;li&gt;the policy used for behaviour should be soft,&lt;/li&gt;&#xA;&lt;li&gt;policies may not be sufficiently similar,&lt;/li&gt;&#xA;&lt;li&gt;may be slower (only the part after the last exploration is reliable), but remains more flexible if alternative routes appear.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For reference, these are the formulations of Q-learning and SARSA from &lt;a href=&quot;https://webdocs.cs.ualberta.ca/%7Esutton/book/ebook/the-book.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton and Barto seminal book&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://webdocs.cs.ualberta.ca/%7Esutton/book/ebook/node65.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Q-learning&lt;/a&gt;:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rsdlP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rsdlP.png&quot; alt=&quot;Q-learning algorithm&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://webdocs.cs.ualberta.ca/%7Esutton/book/ebook/node64.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SARSA&lt;/a&gt;:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R7xR1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R7xR1.png&quot; alt=&quot;SARSA algorithm&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;P.S.: I referenced and quoted the original answer from a different stackexchange site, as indicated in &lt;a href=&quot;https://meta.stackexchange.com/a/172309/174973&quot;&gt;this meta question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2016-12-22T16:12:05.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15881" PostTypeId="1" AcceptedAnswerId="15900" CreationDate="2016-12-22T16:25:54.273" Score="3" ViewCount="4495" Body="&lt;p&gt;I just started using Orange, and am having trouble finding how to get basic summary statistics, like the n (count), average, and standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a widget that does this and I'm simply overlooking it?&lt;/p&gt;&#xA;" OwnerUserId="27310" LastActivityDate="2021-05-23T20:49:54.063" Title="How to get summary statistics in Orange?" Tags="|orange|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15882" PostTypeId="1" CreationDate="2016-12-22T17:06:56.187" Score="6" ViewCount="8136" Body="&lt;p&gt;I am new to GBM and xgboost, and am currently using &lt;code&gt;xgboost_0.6-2&lt;/code&gt; in R. The  modeling runs well with the standard objective function &lt;code&gt;&amp;quot;objective&amp;quot; = &amp;quot;reg:linear&amp;quot;&lt;/code&gt; and after reading &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this NIH paper&lt;/a&gt; I wanted to run a quantile regression using a custom objective function, but it iterates exactly 11 times and the metric does not change.&lt;/p&gt;&#xA;&lt;p&gt;I just simply switched out the 'pred' statement following the &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/R-package/demo/custom_objective.R&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub xgboost demo&lt;/a&gt;, but am afraid it is more complicated than that and I cannot find any other examples on using the custom objective function. Do I need to take it a step further and take derivatives for the 'grad' and 'hess' part?&lt;/p&gt;&#xA;&lt;p&gt;Or could it be a problem with xgboost (doubtful)?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;qntregobj &amp;lt;- function(preds, dtrain) {&#xA;  qr_alpha = .5&#xA;  labels &amp;lt;- getinfo(dtrain, &amp;quot;label&amp;quot;)&#xA;  preds &amp;lt;- ifelse( preds - labels &amp;gt;= 0&#xA;                 , (1-qr_alpha)*abs(preds - labels)&#xA;                 , qr_alpha*abs(preds - labels)&#xA;                 )&#xA;  grad &amp;lt;- preds - labels&#xA;  hess &amp;lt;- preds * (1 - preds)&#xA;  return(list(grad = grad, hess = hess))&#xA;}&#xA;&#xA;step1.param &amp;lt;- list( &amp;quot;objective&amp;quot; = qntregobj&#xA;                   , &amp;quot;booster&amp;quot; = &amp;quot;gbtree&amp;quot;&#xA;                   , &amp;quot;eval.metric&amp;quot; = &amp;quot;rmse&amp;quot;&#xA;                   , 'nthread' = 16&#xA;                   )&#xA;set.seed(123)&#xA;step1.xgbTreeCV &amp;lt;- xgb.cv(param = step1.param&#xA;              , data = xgb.train&#xA;              , nrounds  = nrounds&#xA;              , nfold = 10&#xA;              , scale_pos_weight = 1&#xA;              &#xA;              , stratified = T&#xA;              , watchlist = watchlist&#xA;              &#xA;              , verbose = F&#xA;              , early_stopping_rounds = 10&#xA;              , maximize = FALSE&#xA;              &#xA;              ## set default parameters here - baseline&#xA;              , max_depth = 6&#xA;              , min_child_weight = 1&#xA;              , gamma = 0&#xA;              , subsample = 1&#xA;              , colsample_bytree = 1&#xA;              , lambda = 1&#xA;              , alpha = 0&#xA;              , eta = 0.3&#xA;  )&#xA;  print(Sys.time() - start.time)&#xA;&#xA;  step1.dat &amp;lt;- step1.xgbTreeCV$evaluation_log&#xA;  step1.dat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Which produces:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;iter train_rmse_mean train_rmse_std test_rmse_mean test_rmse_std nround&#xA; 1:    1        122.6362     0.04268346       122.6354     0.3849658      1&#xA; 2:    2        122.6362     0.04268346       122.6354     0.3849658      2&#xA; 3:    3        122.6362     0.04268346       122.6354     0.3849658      3&#xA; 4:    4        122.6362     0.04268346       122.6354     0.3849658      4&#xA; 5:    5        122.6362     0.04268346       122.6354     0.3849658      5&#xA; 6:    6        122.6362     0.04268346       122.6354     0.3849658      6&#xA; 7:    7        122.6362     0.04268346       122.6354     0.3849658      7&#xA; 8:    8        122.6362     0.04268346       122.6354     0.3849658      8&#xA; 9:    9        122.6362     0.04268346       122.6354     0.3849658      9&#xA;10:   10        122.6362     0.04268346       122.6354     0.3849658     10&#xA;11:   11        122.6362     0.04268346       122.6354     0.3849658     11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27309" LastEditorUserId="29169" LastEditDate="2021-09-23T16:11:54.867" LastActivityDate="2021-09-23T16:11:54.867" Title="Xgboost quantile regression via custom objective" Tags="|machine-learning|predictive-modeling|xgboost|gradient-descent|gbm|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15883" PostTypeId="2" ParentId="13760" CreationDate="2016-12-22T17:18:40.250" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Automated_readability_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;ARI&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Coleman%E2%80%93Liau_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coleman-Liau&lt;/a&gt; are basically the same thing with different numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Gunning_fog_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gunning Fog&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Dale%E2%80%93Chall_readability_formula&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dale-Chall&lt;/a&gt; are similar in that they're simple functions of how &quot;complicated&quot; the words and sentences are, though at least they appeal to a notion of &quot;complicated words&quot;, though that just pushes the problem somewhere else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are more and more like this, and I know you're not looking for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be stating the obvious, but if I were building such a measure, I'd look at things like misspellings and grammar errors. These are fairly possible to check automatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the success of &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;RNNs&lt;/a&gt; in reproducing text in the style of given input, I suspect you could quite successfully apply deep learning to learn what about a text makes it readable, or at least find whether models trained on high or low readability texts seem to find a given new input more probable.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2016-12-22T17:18:40.250" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15885" PostTypeId="1" AcceptedAnswerId="17029" CreationDate="2016-12-22T17:31:20.917" Score="6" ViewCount="2671" Body="&lt;p&gt;I'm trying to implement a convolutional neural network at the moment. A simple feedforward network is not the problem but I'm having some trouble with the weight adjustment in the conv layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume I have four layers. Input, convolution, hidden and output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/85bTT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/85bTT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;src:&lt;a href=&quot;http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the picture above we just see the input and the convolution layer.&#xA;The deltas of the convolution layer are calculated as in a normal feedforward network.&#xA;But how do I update the weights/filtermatrix between input and convolutionlayer? &lt;/p&gt;&#xA;" OwnerUserId="27303" LastEditorUserId="29169" LastEditDate="2021-02-16T15:25:51.327" LastActivityDate="2021-02-16T15:25:51.327" Title="Adjusting weights in an convolutional neural network" Tags="|machine-learning|neural-network|convolutional-neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15886" PostTypeId="1" AcceptedAnswerId="15888" CreationDate="2016-12-22T17:42:55.723" Score="2" ViewCount="235" Body="&lt;p&gt;I have been reading several papers and articles related to Principal Component Analysis (PCA) and in some of them, there is one step which is quite unclear to me (in particular (3) in [&lt;a href=&quot;http://www.face-rec.org/algorithms/Kernel/kernelPCA_scholkopf.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Schölkopf 1996&lt;/a&gt;]). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me reproduce their reasoning below.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Consider the centered data set $D = \{\textbf{x}_k\}_{k=1}^M$ with $\textbf{x}_k \in \textbf{R}^N$ and $ \sum_{k=1}^M \textbf{x}_k = 0$. PCA diagonalizes the (sample) covariance matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;C = \frac{1}{M} \sum_{j=1}^M \textbf{x}_j \textbf{x}_j^T. \tag{1}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this we find the solution to the eigenvector equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\lambda \textbf{v} = C \textbf{v} \tag{2}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for eigenvalues $\lambda \geq 0$ and eigenvectors $\textbf{v} \in \textbf{R}^N\backslash \{{0}\}$. As &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\lambda \textbf{v} = C \textbf{v} = \frac{1}{M} \sum_{j=1}^M (\textbf{x}_j^T \textbf{v}) \textbf{x}_j, \tag{3}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;all solutions $\textbf{v}$ with $\lambda \neq 0$ must lie in the span of $\textbf{x}_1, \dots, \textbf{x}_M$, hence (2) is equivalent to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\lambda(\textbf{x}_k^T \textbf{v}) = \textbf{x}_k^T C \textbf{v}, \qquad \text{for } k = 1, \dots, M \tag{4}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In (4), doesn't $\lambda(\textbf{x}^T \textbf{v}) = \textbf{x}^T C \textbf{v}$ hold for $\textbf{any}$ value of $\textbf{x}$? Why does (4) only hold when $\textbf{x} \in D$? I do not understand how their end up with (4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="17978" LastActivityDate="2016-12-23T15:57:45.470" Title="Principal Component Analysis, Eigenvectors lying in the span of the observed data points?" Tags="|dimensionality-reduction|pca|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15887" PostTypeId="1" CreationDate="2016-12-22T18:58:18.543" Score="5" ViewCount="5015" Body="&lt;p&gt;Does any one has a recommendation for what libraries to use for deep learning?&lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2016-12-23T05:24:39.197" Title="What is the best deep learning library for scala?" Tags="|deep-learning|scala|" AnswerCount="2" CommentCount="2" ClosedDate="2016-12-23T20:52:01.410" ContentLicense="CC BY-SA 3.0" />
  <row Id="15888" PostTypeId="2" ParentId="15886" CreationDate="2016-12-22T19:30:52.963" Score="2" Body="&lt;p&gt;The statement says, that (2) and (4) are equal. That means (2)$\Rightarrow$(4) and (4)$\Rightarrow$(2). The first implication is trivial, as you correctly pointed out.&#xA;$$\lambda v=Cv$$&#xA;implies&#xA;$$\lambda x^Tv=x^TCv$$&#xA;for all $v$, not just those from $D$. But the second implication is a bit more tricky, but that is what the proof is about. It tells you, that if you want to check, whether a vector is an eigenvector of $C$, you don't have to check if (2) is satisfied. It tells you, that when (4) is satisfied, (2) is satisfies as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have 2 points in 3D space. Those 2 points are&#xA;$$x_1=(-1,-1,0)$$&#xA;$$x_2=(1,1,0)$$&#xA;(please excuse me for not making this &quot;dataset&quot; centered) Note, that both points lie in the $xy$ plane. Now the correlation matrix is&#xA;$$C=\begin{bmatrix}&#xA;       1/2 &amp;amp; -1/2 &amp;amp; 0           \\[0.3em]&#xA;       -1/2 &amp;amp; 1/2           &amp;amp; 0 \\[0.3em]&#xA;       0           &amp;amp; 0 &amp;amp; 0&#xA;     \end{bmatrix}$$&#xA;Now you want to know, whether $v=[1\ -1\ 0]^T$ is an eigenvector with eigenvalue 1. The statement tells you, that you can either just check, if (2) is satisfied (3 equations) or if&#xA;$$\frac{1}{2}x_1^Tv=x_1^TCv=0$$&#xA;and&#xA;$$\frac{1}{2}x_2^Tv=x_2^TCv=0$$&#xA;which are only two equations.&lt;/p&gt;&#xA;" OwnerUserId="27313" LastEditorUserId="27313" LastEditDate="2016-12-23T15:57:45.470" LastActivityDate="2016-12-23T15:57:45.470" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15889" PostTypeId="2" ParentId="15867" CreationDate="2016-12-22T22:01:19.990" Score="0" Body="&lt;p&gt;It sounds like you have two issues. The first one is preprocessing and feature extraction. The second one is how to learn classification rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second issue is the easier one to approach. There are a number of algorithms for learning classification rules. You could use a decision tree algorithm such as CART or C.4.5 but there are also rule induction algorithms like the CN2 algorithm. Both these types of algorithms can learn the types of rules you mention, however, rule induction based systems can usually be supplemented with hand crafted rules in a more straight forward way than decision tree based systems, while, unless my memory fails me, decision tree algorithms generally perform better on classification tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first issue is bit hairier. To recommend the types of changes you suggest you first need to extract the relevant features. There are pre-processors which perform part-of-speech tagging, syntactic parsing, named entity recognition etc. and if the citations follow a strict format, I guess a regular expression could perhaps solve the problem, but otherwise you have to first train a system to recognize and count the number of citations in a text (and the same for any other non-trivial feature). Then you can pass the output of this feature extraction system into the classification system. However, on reading your question again I'm unsure whether this problem might already be solved in your case?&lt;/p&gt;&#xA;" OwnerUserId="27229" LastActivityDate="2016-12-22T22:01:19.990" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15890" PostTypeId="2" ParentId="15882" CreationDate="2016-12-22T22:11:28.827" Score="0" Body="&lt;p&gt;Yes,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grad &amp;lt;- preds - labels&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is specific to the logistic loss. See &lt;a href=&quot;https://stats.stackexchange.com/questions/219241/gradient-for-logistic-loss-function&quot;&gt;this question&lt;/a&gt; for a derivation.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2016-12-22T22:11:28.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15891" PostTypeId="1" AcceptedAnswerId="16015" CreationDate="2016-12-22T22:40:06.607" Score="3" ViewCount="632" Body="&lt;p&gt;I some code that identifies ngrams of interest in a small bit of text, then searches for the ngram in a larger bit of text to provide a snippet of the term used in context.  One ngram might be &lt;code&gt;unordered numerical set&lt;/code&gt;.  Right now, if I can't find &lt;code&gt;unordered numerical set&lt;/code&gt;, I start trimming the string from the left to see if &lt;code&gt;numerical set&lt;/code&gt; is available in the larger text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do before I trim from the left, is to see if the any of the words have an antonym, like &lt;code&gt;ordered&lt;/code&gt; for &lt;code&gt;unordered&lt;/code&gt;.  This is because my ngram might be defined in one way, but used in the opposite.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a known list or other way of finding out if there is a highly correlated antonym I might try for a set of words?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, I think I could write some regexps that look for prefixes, like &quot;un&quot; or &quot;dis&quot; and see if the word that results from removing these prefixes is a valid english word.  This seems likely to be already solved, so before I try to create anything, I want to find out what might already exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using Python using Gensim, NLTK, and Word2Vec for the rest of my processing if it matters.&lt;/p&gt;&#xA;" OwnerUserId="26939" LastEditorUserId="26939" LastEditDate="2016-12-23T21:50:26.850" LastActivityDate="2016-12-31T16:51:33.817" Title="Antonym search for expanding search terms" Tags="|python|nlp|gensim|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15892" PostTypeId="1" CreationDate="2016-12-22T22:58:16.880" Score="3" ViewCount="1253" Body="&lt;p&gt;I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We randomize the parameters with uniform distribution U(-0.003,&#xA;  0.003), &lt;strong&gt;set the clipping threshold of softmax layer as 200&lt;/strong&gt; and set learning rate as 0.01.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of &quot;set the clipping threshold of softmax layer as 200&quot;. 200???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(int k = 0; k &amp;lt; softmax.outputG.length; k++)&#xA;    softmax.outputG[k] = 0.0;&#xA;softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];&#xA;&#xA;// if ||g|| &amp;gt;= threshold, then g &amp;lt;- g * threshold / ||g|| &#xA;if(Math.abs(softmax.outputG[goldPol]) &amp;gt; clippingThreshold)&#xA;{&#xA;    if(softmax.outputG[goldPol] &amp;gt; 0)&#xA;        softmax.outputG[goldPol] =  clippingThreshold;&#xA;    else&#xA;        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I still can't say I fully understand the use of this threshold..&lt;/p&gt;&#xA;" OwnerUserId="2454" LastEditorUserId="29575" LastEditDate="2017-08-31T21:09:49.023" LastActivityDate="2017-08-31T21:09:49.023" Title="Clipping threshold of softmax layer" Tags="|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15894" PostTypeId="2" ParentId="13649" CreationDate="2016-12-23T03:02:01.653" Score="2" Body="&lt;p&gt;If the dim of your data is more than 2, you may want to transform your data with PCA, t-SNE or Custom linear projections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t-SNE is recommended for clustering result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.manifold import TSNE&#xA;tsne = TSNE(n_components=2, random_state=0)&#xA;data = tsne.fit_transform(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This may lose some information, but useful for visualization.&lt;/p&gt;&#xA;" OwnerUserId="27287" LastActivityDate="2016-12-23T03:02:01.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15895" PostTypeId="2" ParentId="15887" CreationDate="2016-12-23T03:12:14.970" Score="5" Body="&lt;p&gt;I think currently there is no pure scala deep learning library which can be compared with mxnet, caffe, keras and etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For lang on JVM, u can try this one:&#xA;&lt;a href=&quot;https://deeplearning4j.org/&quot; rel=&quot;noreferrer&quot;&gt;https://deeplearning4j.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27287" LastActivityDate="2016-12-23T03:12:14.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15896" PostTypeId="1" CreationDate="2016-12-23T04:03:38.190" Score="3" ViewCount="309" Body="&lt;p&gt;I have used TensorFlow before for training a model to recognize images (something similar to this example: &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/inception&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/models/tree/master/inception&lt;/a&gt;). I was trying to do something similar with inception-v4 from scratch (using the model from &lt;a href=&quot;https://github.com/tensorflow/models&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/models&lt;/a&gt;). But it was not working, it wasn't learning, so I decided to reduce my code and simplify everything. The model is a hidden layer with the same neurons as inputs and what the model has to learn is that input=output. Seems pretty simple and easy, but the results of my code are far from good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried different things: sigmoid cross entropy instead of softmax cross entropy, other optimizers, different learning rates, models more comples with more neuros and more layers, reduce_sum instead of reduce mean for the loss ... But the only times I got &quot;good results&quot; (&gt;90%) it seems because the model falls into a local minimum and I am not able to replicate them. Usually I see 70% precision with a high loss (&gt;=0.7).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My conclusion is that there is something wrong in my code that I am not able to spot and that is the reason the learning process is slow and get stuck at high losses. Otherwise the code with inception-v4 for images will at least learn something and don't get the loss stuck at the same value all the time.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;import random&#xA;import shutil&#xA;&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;import tensorflow.contrib.layers as layers&#xA;&#xA;# parameters&#xA;max_steps = 10000&#xA;batch_size = 64&#xA;output_size = 10&#xA;summary_path = './summary'&#xA;&#xA;with tf.Graph().as_default() as graph:&#xA;    shape = [batch_size, output_size]&#xA;    inputs = tf.placeholder(dtype=tf.float32, shape=shape, name='inputs')&#xA;    outputs = tf.placeholder(dtype=tf.float32, shape=shape, name='outputs')&#xA;    global_step = tf.Variable(0, name='global_step', trainable=False)&#xA;    logits = layers.fully_connected(inputs, output_size)&#xA;    learning_rate = tf.train.exponential_decay(0.1, global_step, max_steps / 3, 0.1,&#xA;                                               name='learning_rate')&#xA;    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, outputs, name='loss'))&#xA;    optimizer = tf.train.GradientDescentOptimizer(learning_rate)&#xA;    train_op = optimizer.minimize(loss, global_step=global_step)&#xA;    init_op = tf.global_variables_initializer()&#xA;    input_label = tf.argmax(inputs, axis=1) # vector with the input labels&#xA;    output_label = tf.argmax(logits, axis=1) # vector with the predicted labels&#xA;    tf.summary.scalar('loss', loss)&#xA;    tf.summary.scalar('learning_rate', learning_rate)&#xA;    summary_op = tf.summary.merge_all()&#xA;&#xA;# prepare summary&#xA;if os.path.exists(summary_path):&#xA;    shutil.rmtree(summary_path)&#xA;os.makedirs(summary_path)&#xA;summary_writer = tf.train.SummaryWriter(summary_path, graph=graph)&#xA;&#xA;with tf.Session(graph=graph).as_default() as session:&#xA;    session.run(init_op)&#xA;    to_run = {'step': global_step, 'loss': loss, 'logits': logits, 'train_op': train_op,&#xA;              'learning_rate': learning_rate, 'summary': summary_op,&#xA;              'in_label': input_label, 'out_label': output_label}&#xA;    step = 0&#xA;    while step &amp;lt; max_steps:&#xA;        # fake input&#xA;        data = np.zeros(shape, dtype=float)&#xA;        for i in range(batch_size):&#xA;            data[i, random.randint(0, output_size - 1)] = 1.0&#xA;        # run&#xA;        results = session.run(to_run, feed_dict={inputs: data, outputs: data})&#xA;        # print data&#xA;        step, in_label, out_label = results['step'],  results['in_label'], results['out_label']&#xA;        precision = (in_label == out_label).sum() * 100.0 / batch_size&#xA;        print('step: {0}/{1}  learning rate: {2:.4f}  loss: {3:.4f}  precision: {4:.1f}%'&#xA;              .format(step, max_steps, results['learning_rate'], results['loss'], precision))&#xA;        summary_writer.add_summary(results['summary'], step)&#xA;    # print last batch&#xA;    print(data)&#xA;    print(results['logits'])&#xA;    print(in_label)&#xA;    print(out_label)&#xA;    print('precision {}%'.format(precision))&#xA;&#xA;summary_writer.flush()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/p11tY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/p11tY.png&quot; alt=&quot;learning rate and loss graphs&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20933" LastActivityDate="2016-12-23T04:03:38.190" Title="TensorFlow doesn't learn when input=output (or probably I am missing something)" Tags="|python|neural-network|deep-learning|tensorflow|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15897" PostTypeId="2" ParentId="15887" CreationDate="2016-12-23T05:24:39.197" Score="6" Body="&lt;p&gt;Hi (I created deeplearning4j: warning biased source),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's fairly new but we are open to feedback. We ported keras to scala:&#xA;&lt;a href=&quot;http://github.com/deeplearning4j/ScalNet/&quot; rel=&quot;noreferrer&quot;&gt;http://github.com/deeplearning4j/ScalNet/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This uses deeplearning4j underneath. We would love contributions or issues.&lt;/p&gt;&#xA;" OwnerUserId="27321" LastActivityDate="2016-12-23T05:24:39.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15898" PostTypeId="1" AcceptedAnswerId="15910" CreationDate="2016-12-23T08:04:11.383" Score="0" ViewCount="2339" Body="&lt;p&gt;I am using the MinMaxScaler() of sklearn to scale my features before using kmeans.I needed to find the scale used.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import preprocessing&#xA;scaler = preprocessing.MinMaxScaler()&#xA;scaler= scaler.fit_transform(finance_features)&#xA;print scaler.scale_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However on using the line&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print scaler.scale_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It shows an error saying no such attribute&lt;/p&gt;&#xA;" OwnerUserId="27296" LastActivityDate="2016-12-23T21:09:24.897" Title="Finding out the scale when using MinMaxScaler()" Tags="|feature-scaling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15899" PostTypeId="2" ParentId="15812" CreationDate="2016-12-23T09:13:40.017" Score="11" Body="&lt;p&gt;There are many ways in which you can compute a distance between time series, and the method to use will depend on your data.&lt;/p&gt;&#xA;&lt;p&gt;As stated by other answers, &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;noreferrer&quot;&gt;Dynamic Time Warping&lt;/a&gt; may be the way to go. However, this method assumes that there may be a non-linear warp between different parts of the time series.&lt;/p&gt;&#xA;&lt;p&gt;If you are not expecting warping or delays in the signal, something as simple as Euclidean distance may be a better way to go. Of course, you should apply Euclidean distance only after you applied some preprocessing (for instance, amplitude scaling).&lt;/p&gt;&#xA;&lt;p&gt;Take a look to &lt;a href=&quot;http://didawiki.cli.di.unipi.it/lib/exe/fetch.php/dm/time_series_from_keogh_tutorial.pdf&quot; rel=&quot;noreferrer&quot;&gt;this presentation&lt;/a&gt;, that introduces the pros and cons of these methods and discuss preprocessing in more detail:&lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="98307" LastEditDate="2020-08-01T15:43:30.640" LastActivityDate="2020-08-01T15:43:30.640" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="15900" PostTypeId="2" ParentId="15881" CreationDate="2016-12-23T09:43:57.543" Score="3" Body="&lt;p&gt;I found 3 ways to do it :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you have to use the Data Info widget who gives the number of rows of your dataset and the Box Plot widget who prints the average and the standard deviation of each feature in your dataset. Pro: easy to use. Con: you are unable to use this data.&lt;/li&gt;&#xA;&lt;li&gt;you can create a Python script who will compute the average and the standard deviation using the statistics package. Pro: you can reuse your data. Con: you have to write a script and format the output data correctly.&lt;/li&gt;&#xA;&lt;li&gt;it exists an add-on call &quot;Orange3-Timeseries&quot; with a Moving Transform widget who can compute the average and the standard deviation of your series. Pro: easy to use. Cons: the package is still a beta-version and have some bugs, the widget only works with timeseries.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27003" LastActivityDate="2016-12-23T09:43:57.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15902" PostTypeId="2" ParentId="15830" CreationDate="2016-12-23T10:44:23.013" Score="1" Body="&lt;p&gt;You already mentioned the alternatives...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;convert the &lt;code&gt;neuralnet&lt;/code&gt; object to a &lt;code&gt;nnet&lt;/code&gt; class object&lt;/li&gt;&#xA;&lt;li&gt;implement &lt;code&gt;pmml.neuralnet()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both &lt;code&gt;pmml&lt;/code&gt;, &lt;code&gt;nnet&lt;/code&gt; and &lt;code&gt;neuralnet&lt;/code&gt; packages are easy to read. You can find the source code &lt;a href=&quot;https://cran.r-project.org/web/packages/available_packages_by_name.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;over cran.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think implementing &lt;code&gt;pmml.neuralnet()&lt;/code&gt; might be the cleanest. You can just copy-paste the current &lt;a href=&quot;https://cran.r-project.org/web/packages/pmml/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pmml.nnet()&lt;/code&gt; &lt;/a&gt; method, call it &lt;code&gt;pmml.neturalnet()&lt;/code&gt; and make the necessary changes. Should not take more than a day's work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps: It would be cool if you'd sent your function to &lt;code&gt;neuralnet&lt;/code&gt; authors afterwards of course, and link it from here. :)&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2016-12-23T10:44:23.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15903" PostTypeId="1" CreationDate="2016-12-23T12:43:47.203" Score="36" ViewCount="16170" Body="&lt;p&gt;I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated features as we go up the layers?&#xA;What caused them create such a stack of features and would this also be true for other types of deep neural networks?&lt;/p&gt;&#xA;" OwnerUserId="27330" LastEditorUserId="29169" LastEditDate="2021-02-16T15:27:41.027" LastActivityDate="2021-02-16T15:27:41.027" Title="Why do convolutional neural networks work?" Tags="|machine-learning|deep-learning|neural-network|cnn|convolutional-neural-network|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15905" PostTypeId="2" ParentId="15831" CreationDate="2016-12-23T15:06:24.727" Score="2" Body="&lt;p&gt;A plan should be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;consider a dataset as a 'match' between features&lt;/li&gt;&#xA;&lt;li&gt;randomize matches' order (of insert them in time order, if any) and insert results in a rating system engine. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you're interested in use more than in development, &lt;a href=&quot;https://rankade.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;rankade&lt;/a&gt;, our free ranking system for sports, games, and more, allows matches with both 2 and 3+ factions, while &lt;em&gt;Elo&lt;/em&gt; and &lt;em&gt;Glicko&lt;/em&gt; works just for one-on-one (here's a &lt;a href=&quot;https://rankade.com/ree#ranking-system-comparison&quot; rel=&quot;nofollow noreferrer&quot;&gt;comparison&lt;/a&gt;).&#xA;In addition, rankade has a weight feature (all datasets have same impact?) that might refine your work.&lt;/p&gt;&#xA;" OwnerUserId="27334" LastActivityDate="2016-12-23T15:06:24.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15907" PostTypeId="2" ParentId="15903" CreationDate="2016-12-23T15:34:51.563" Score="18" Body="&lt;p&gt;ConvNets work because they exploit feature locality. They do it at different granularities, therefore being able to model &lt;em&gt;hierarchically&lt;/em&gt; higher level features. They are translation invariant thanks to pooling units. They are not rotation-invariant &lt;em&gt;per se&lt;/em&gt;, but they usually converge to filters that are &lt;a href=&quot;https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html&quot; rel=&quot;noreferrer&quot;&gt;rotated versions of the same filters&lt;/a&gt;, hence supporting rotated inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of no other neural architecture that profits from feature locality in the same sense as ConvNets do.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2016-12-23T15:34:51.563" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15908" PostTypeId="1" CreationDate="2016-12-23T16:31:20.490" Score="0" ViewCount="1800" Body="&lt;p&gt;I am trying to do clustering in my dataset which has 4 numerical fields. Please find the file attached : &lt;a href=&quot;http://www.filedropper.com/example_3&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.filedropper.com/example_3&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried with this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import KMeans&#xA;kmeans = KMeans(n_clusters=2, random_state=0, max_iter = 300).fit(dffinal)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that there are 2 classes in this example, that was the reason i tried with 2 clusters. Out of 4200 rows, first 3196 rows belong to a class and the remaining rows belong to an another class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when i do clustering, cluster labels are randomly assigned, and accuracy is less than 10%. Just wondering whether my features are not good enough for clustering or should I try with some other clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="27335" LastEditorUserId="25218" LastEditDate="2016-12-27T21:08:18.387" LastActivityDate="2020-02-20T18:29:00.317" Title="Clustering of numerical data" Tags="|python|clustering|scikit-learn|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15910" PostTypeId="2" ParentId="15898" CreationDate="2016-12-23T21:09:24.897" Score="0" Body="&lt;p&gt;This is because you were importing a class as name 'scalar' and modifying it by assigning value obtained after transforming. Just change &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import preprocessing&#xA;scaler = preprocessing.MinMaxScaler()&#xA;scaled = scaler.fit_transform(finance_features)&#xA;print scaler.scale_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should work.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-12-23T21:09:24.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15911" PostTypeId="2" ParentId="15908" CreationDate="2016-12-23T23:21:31.487" Score="2" Body="&lt;p&gt;You forgot to preprocess your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-means is really sensitive to scale and outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also: &lt;strong&gt;clustering is not classification&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may well be that e.g. one of your classes has a dense &lt;em&gt;subtype&lt;/em&gt;. Just because the clustering found something &lt;em&gt;else&lt;/em&gt; than your labels (and it's not supposed to find the same, is it?) does not mean it failed. It just didn't classify the data the same way as your labels (but if that is what you wsnt, then you should have been using a classification method instead.)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-12-23T23:21:31.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15912" PostTypeId="2" ParentId="15908" CreationDate="2016-12-24T11:43:02.153" Score="1" Body="&lt;p&gt;I suspect that clustering this data will not be very productive. Just make a simple plot showing two variables at a time.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ZPF2j.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZPF2j.png&quot; alt=&quot;Pairs of variables&quot;&gt;&lt;/a&gt;&#xA;None of these seems to give evidence of natural clustering. One of these plots may give you a glimmer of hope: Count vs Duration3&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WxfMF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WxfMF.png&quot; alt=&quot;Count vs Duration3&quot;&gt;&lt;/a&gt;&#xA;There is something going on here that some points have Count bigger than 10, others have Duration3 bigger than 10^6 and those two things never happen together. This might hint at a mixture of two groups,  but I doubt that clustering is the way to get at that.  Most of the points are in the puddle near the origin. &lt;/p&gt;&#xA;" OwnerUserId="26917" LastActivityDate="2016-12-24T11:43:02.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15913" PostTypeId="1" CreationDate="2016-12-24T18:24:34.547" Score="0" ViewCount="162" Body="&lt;p&gt;My goal is to come up with a set of research questions for which to drive my self study into my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have Smart Meter data that is taken every minute, on my home's power usage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I possibly use the time series data on my household energy consumption that would help me extract practical insights from it?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="29169" LastEditDate="2021-01-16T00:13:05.280" LastActivityDate="2021-01-16T00:13:05.280" Title="How to develop a good model for predicting electricity usage?" Tags="|data-mining|predictive-modeling|data-cleaning|bigdata|" AnswerCount="1" CommentCount="1" ClosedDate="2017-01-27T10:13:32.233" ContentLicense="CC BY-SA 3.0" />
  <row Id="15914" PostTypeId="2" ParentId="15903" CreationDate="2016-12-25T06:03:46.643" Score="3" Body="&lt;p&gt;One should never forget the other components in a typical ConvNet. The convolution filters pick out the spatial invariant features, like edges and circles. These features are quantified in a pooling layer which follows the C layer. Finally, they are fed into (usually) multiple fully connected layers (fc). Credit must be given to these fully connected layers which are nothing more than what you find in any ordinary MLP.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastActivityDate="2016-12-25T06:03:46.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15915" PostTypeId="1" AcceptedAnswerId="15934" CreationDate="2016-12-25T06:28:38.973" Score="2" ViewCount="8550" Body="&lt;p&gt;I am reading pattern Recognition and machine learning by Bishop and in the chapter about probability, &quot;noise in the observed data&quot; is mentioned many times. I have read on the internet that noise refers to the inaccuracy while reading data but I am not sure whether it is correct. SO, what actually is noise in observed data? And what is additive noise and Gaussian noise ? &lt;/p&gt;&#xA;" OwnerUserId="24077" LastActivityDate="2021-03-09T06:16:23.703" Title="What is &quot;noise&quot; in observed data?" Tags="|machine-learning|data|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15916" PostTypeId="2" ParentId="15311" CreationDate="2016-12-25T07:53:59.873" Score="3" Body="&lt;p&gt;TF - IDF stands for  &lt;code&gt;term frequency–inverse document frequency&lt;/code&gt;&lt;br&gt;&#xA;&lt;strong&gt;TF&lt;/strong&gt; counts the &lt;code&gt;frequency of a term / total #terms&lt;/code&gt; in a given document. For each term in a document, this value changes.&lt;br&gt;&#xA;&lt;strong&gt;IDF&lt;/strong&gt; counts the log of ratio of &lt;code&gt;total document / term appearing in #documents&lt;/code&gt; . This value is constant for a given unique term. Greater the idf value for a term, higher its significance.&lt;br&gt;&#xA;&lt;strong&gt;Example:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Document 1: This is a sample example.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Document 2: This is another example.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Lets calculate for term = &quot;is&quot;:&lt;br&gt;&#xA;TF(is, Document 1) = 1/5&lt;br&gt;&#xA;TF(is, Document 2) = 1/4&lt;br&gt;&#xA;IDF(is) = log(2/2) = 0&lt;br&gt;&#xA;TFIDF = TF*IDF&lt;br&gt;&#xA;TFIDF(is, document 1) = (1/5)*0 = 0&lt;br&gt;&#xA;TFIDF(is, document 2) = (1/4)*0 = 0&lt;br&gt;&#xA;It means that the term &quot;is&quot; is not a significant term in the list of documents(corpus).&lt;br&gt;&#xA;Lets consider term = &quot;another&quot;&lt;br&gt;&#xA;TF(another, document 1) = 0/5 = 0&lt;br&gt;&#xA;TF(another, document 2) = 1/4&lt;br&gt;&#xA;IDF(another) = log(2/1) = 0.301&lt;br&gt;&#xA;TFIDF(another, document 1) = 0*0.301 = 0&lt;br&gt;&#xA;TFIDF(another, document 2) = (1/4)*0.301&lt;br&gt;&#xA;You can observe from both examples that TF varies per document while IDF is constant.&lt;br&gt;&#xA;You can convert your entire 270 documents into term document matrix.&lt;br&gt;&#xA;Demo in python:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;&#xA;d = pd.Series(['this is a sample example','this is another example'])&#xA;df = pd.DataFrame(d)&#xA;tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0)&#xA;# if you want, say only top 2 features(terms)&#xA;# tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0, max_features=2, max_df = 3)&#xA;# Terms with given below:&#xA;# occurred in too many documents (max_df, tfidf score = 3)  &#xA;# occurred in too few documents (min_df, tfidf score = 0)&#xA;# cut off by feature selection (max_features, tfidf score = 2).&#xA;tfidf = tfidf_vectorizer.fit_transform(df[0])&#xA;print tfidf_vectorizer.vocabulary_&#xA;# output: {u'this': 4, u'sample': 3, u'is': 2, u'example': 1, u'another': 0}&#xA;print tfidf_vectorizer.idf_&#xA;# output(constant): [ 1.40546511  1.          1.          1.40546511  1.        ]&#xA;print tfidf&#xA;# output: &#xA;#(0, 1)        0.448320873199    Document 1, term = example&#xA;#(0, 3)        0.630099344518    Document 1, term = sample&#xA;#(0, 2)        0.448320873199    Document 1, term = is&#xA;#(0, 4)        0.448320873199    Document 1, term = this&#xA;#(1, 0)        0.630099344518    Document 2, term = another&#xA;#(1, 1)        0.448320873199    Document 2, term = example&#xA;#(1, 2)        0.448320873199    Document 2, term = is&#xA;#(1, 4)        0.448320873199    Document 2, term = this&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Source:&lt;br&gt;&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27356" LastActivityDate="2016-12-25T07:53:59.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15917" PostTypeId="2" ParentId="15913" CreationDate="2016-12-25T08:20:39.863" Score="2" Body="&lt;p&gt;Adding as a answer instead of comment due to insufficient reps. &#xA;Consider developing a model that can identify the pattern of electricity and water usage per house throughout the year. And predict the upcoming requirements. It will be a time series problem. This will help in maximising resource utilisation and better availability of resources among everyone.&lt;br&gt;&#xA;Later, this can help in smart power and water distribution which is not yet applied.&lt;br&gt;&#xA;If you want a bigger community working into this data set, you can share it online or provide it to &lt;a href=&quot;http://www.kaggle.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;. Many people are out there ready to help in it. &lt;/p&gt;&#xA;" OwnerUserId="27356" LastActivityDate="2016-12-25T08:20:39.863" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="15918" PostTypeId="2" ParentId="10211" CreationDate="2016-12-25T08:33:04.560" Score="10" Body="&lt;p&gt;Take a look at the &lt;a href=&quot;http://polyglot.readthedocs.io/en/latest/Sentiment.html&quot; rel=&quot;noreferrer&quot;&gt;polyglot&lt;/a&gt; library. It has polarity lexicons for 136 languages, including Russian.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The scale of the words’ polarity consisted of three degrees: +1 for&#xA;  positive words, and -1 for negatives words. Neutral words will have a&#xA;  score of 0.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can use it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from polyglot.text import Text as T&#xA;&amp;gt;&amp;gt;&amp;gt; text = T(&quot;это очень плохо. А это намного лучше, даже хорошо!&quot;)&#xA;&amp;gt;&amp;gt;&amp;gt; text.polarity&#xA;0.33333333333333331&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; for s in text.sentences:&#xA;...  s, s.polarity&#xA;... &#xA;(Sentence(&quot;это очень плохо.&quot;), -1.0)&#xA;(Sentence(&quot;А это намного лучше, даже хорошо!&quot;), 1.0)&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; for w in text.words:&#xA;...   print(&quot;{:&amp;lt;16}{:&amp;gt;2}&quot;.format(w, w.polarity))&#xA;... &#xA;это              0&#xA;очень            0&#xA;плохо           -1&#xA;.                0&#xA;А                0&#xA;это              0&#xA;намного          0&#xA;лучше            1&#xA;,                0&#xA;даже             0&#xA;хорошо           1&#xA;!                0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27358" LastActivityDate="2016-12-25T08:33:04.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15919" PostTypeId="1" AcceptedAnswerId="15986" CreationDate="2016-12-25T12:01:22.920" Score="1" ViewCount="1297" Body="&lt;p&gt;I installed XGBoost for anaconda on windows 10 based on the instructions provided &lt;a href=&quot;https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_For_Anaconda_on_Windows?lang=en&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It seems that xgboost 0.6 is already installed. It performs well using &quot;&lt;code&gt;reg:linear&lt;/code&gt;&quot;. However, if I use &quot;&lt;code&gt;reg:tweedie&lt;/code&gt;&quot;, this error is reported:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgboost.core.XGBoostError: b'[23:08:27] src/objective/objective.cc:21: Unknown objective function reg:tweedie'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2020-09-11T07:24:34.947" LastActivityDate="2020-09-11T07:24:34.947" Title="How to run XGBoostregressor using reg: tweedie as objective?" Tags="|python|regression|xgboost|python-3.x|anaconda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15920" PostTypeId="2" ParentId="15903" CreationDate="2016-12-25T12:48:31.907" Score="8" Body="&lt;p&gt;Convolutional neural networks work because it's a good extension from the standard deep-learning algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given unlimited resources and money, there is no need for convolutional because the standard algorithm will also work. However, convolutional is more efficient because it &lt;strong&gt;reduces the number of parameters&lt;/strong&gt;. The reduction is possible because it takes advantage of feature locality, exactly what @ncasas writes.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2016-12-25T12:48:31.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15921" PostTypeId="1" AcceptedAnswerId="26796" CreationDate="2016-12-25T15:55:26.067" Score="10" ViewCount="600" Body="&lt;p&gt;I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been looking at various posts now related to recurrent neural network, and I understand the concept behind it, but I fail to understand certain parts of the logic behind the RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are four main questions I have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How does back-propagation work in recurrent neural network?&lt;/li&gt;&#xA;&lt;li&gt;Are the weights that lead from the input to the hidden nodes the same for every other step? What about the weights from the hidden nodes to the outputs?&lt;/li&gt;&#xA;&lt;li&gt;How do biases exactly work in a recurrent neural network?&lt;/li&gt;&#xA;&lt;li&gt;Why is tanh function usually used instead of sigmoid function as the activtion function?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I realize some of these questions are very basic, but I guess the basics are exactly what I need right now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even links to related videos or posts will be extremely helpful, and so will google keywords that show the right results. These four questions are hindering me from understanding sample Python codes, so I really need some help.&lt;/p&gt;&#xA;" OwnerUserId="27135" LastEditorUserId="26562" LastEditDate="2017-11-06T10:30:24.530" LastActivityDate="2018-02-05T11:54:19.163" Title="Questions When Advancing from Vanilla Neural Network to Recurrent Neural Network" Tags="|machine-learning|python|neural-network|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15922" PostTypeId="1" CreationDate="2016-12-25T19:11:40.650" Score="1" ViewCount="82" Body="&lt;p&gt;I have dataset of posts from blog and for each post I have the number of views.  I want to extract the topics (or phrases) that made the posts with more views. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am planning divide all posts in two sets based on number of views (one set with low number of views and the other with higher numbers), then extract topics using LDA from each set and compare how they differ.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if this is right approach and if there are other approaches that can be better or similar? &lt;/p&gt;&#xA;" OwnerUserId="27371" LastActivityDate="2017-07-24T17:12:52.677" Title="Extracting popular keyword terms or topics from blog posts based on post usage" Tags="|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15923" PostTypeId="1" AcceptedAnswerId="16037" CreationDate="2016-12-25T19:35:16.417" Score="4" ViewCount="709" Body="&lt;p&gt;In the attached image, I am plotting actual vs predicted along with confidence intervals for each of the predicted points. The black star represents the average of all the predicted points. What formula can I use to combine the confidence intervals of all the predicted data points to get a confidence interval for the average? Also, please assume that the average could be a weighted average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using python, numpy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8KzMz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8KzMz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12985" LastEditorUserId="8560" LastEditDate="2017-06-04T01:20:47.477" LastActivityDate="2017-08-03T02:10:55.650" Title="Computing confidence interval for average from individual predictions" Tags="|python|statistics|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15924" PostTypeId="1" AcceptedAnswerId="15926" CreationDate="2016-12-25T22:29:59.157" Score="6" ViewCount="76410" Body="&lt;p&gt;I am trying to learn data analysis and machine learning by trying out some problems.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a &lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques&quot; rel=&quot;nofollow noreferrer&quot;&gt;competition&lt;/a&gt; &quot;House prices&quot; which is actually a playground competition. Since I am very new to this field, I got confused after exploring the data. The data has 81 columns out of which 1 is the target column which is the house value. This data contains multiple columns where majority of values are &quot;NaN&quot;. When I ran: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nulls = data.isnull().sum()&#xA;nulls[nulls &amp;gt; 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This shows the columns with missing values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LotFrontage     259 &#xA;Alley           1369&#xA;MasVnrType      8   &#xA;MasVnrArea      8   &#xA;BsmtQual        37  &#xA;BsmtCond        37  &#xA;BsmtExposure    38  &#xA;BsmtFinType1    37  &#xA;BsmtFinType2    38  &#xA;Electrical      1   &#xA;FireplaceQu     690 &#xA;GarageType      81  &#xA;GarageYrBlt     81  &#xA;GarageFinish    81  &#xA;GarageQual      81  &#xA;GarageCond      81  &#xA;PoolQC          1453&#xA;Fence           1179&#xA;MiscFeature     1406&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this point I am totally lost and I don't know how to get rid of these &quot;NaN&quot; values.&lt;br&gt;&#xA;Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="27197" LastEditorUserId="94450" LastEditDate="2020-04-18T10:49:44.220" LastActivityDate="2021-09-20T23:16:05.243" Title="How can I fill NaN values in a Pandas DataFrame in Python?" Tags="|python|data-cleaning|kaggle|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15926" PostTypeId="2" ParentId="15924" CreationDate="2016-12-26T00:06:23.560" Score="7" Body="&lt;p&gt;You can use the &lt;code&gt;DataFrame.fillna&lt;/code&gt; function to fill the &lt;code&gt;NaN&lt;/code&gt; values in your data. For example, assuming your data is in a DataFrame called &lt;code&gt;df&lt;/code&gt;, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.fillna(0, inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will replace the missing values with the constant value &lt;code&gt;0&lt;/code&gt;. You can also do more clever things, such as replacing the missing values with the mean of that column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.fillna(df.mean(), inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or take the last value seen for a column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.fillna(method='ffill', inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Filling the &lt;code&gt;NaN&lt;/code&gt; values is called &lt;em&gt;imputation&lt;/em&gt;. Try a range of different imputation methods and see which ones work best for your data.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-26T00:06:23.560" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15928" PostTypeId="1" AcceptedAnswerId="15929" CreationDate="2016-12-26T01:05:23.850" Score="0" ViewCount="48" Body="&lt;p&gt;I've trained a neural network network that given a minmax normalized input, provides a minmax normalized output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;this might be late, but is it possible from a minmax normalized output to create the actual output, given you know the actual min and max value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so an unormalized output?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastActivityDate="2016-12-26T01:21:26.987" Title="adjust output from normalization?" Tags="|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15929" PostTypeId="2" ParentId="15928" CreationDate="2016-12-26T01:21:26.987" Score="2" Body="&lt;p&gt;Yes, just rearrange the formula for finding the normalised value. Where $x_i$ is the original attribute, and $z_i$ is the normalised value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xA;\begin{align}&#xA;  z_i &amp;amp;= \frac{x_i - min(x)}{max(x) - min(x)} \\&#xA;  x_i &amp;amp;= z_i(max(x) - min(x)) + min(x) &#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2016-12-26T01:21:26.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15930" PostTypeId="1" CreationDate="2016-12-26T05:12:30.053" Score="1" ViewCount="2467" Body="&lt;p&gt;I have trained a xgboost model for a classification problem. I'm able to get the feature importance for the model as below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/feature-importance-and-feature-selection-with-xgboost-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/feature-importance-and-feature-selection-with-xgboost-in-python/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I would like to get feature importance per input. Basically, one input may give me a output probability 0.9, another one input may give me a output probability 0.1. I want to know because which features (with values) give me a probability 0.9? which features (with values) give me a probability 0.1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I approach that? Is there a package for this?&lt;/p&gt;&#xA;" OwnerUserId="6567" LastActivityDate="2017-01-05T06:37:43.227" Title="Xgboost (classification problem) feature importance per input not for the model" Tags="|machine-learning|predictive-modeling|xgboost|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15931" PostTypeId="2" ParentId="15930" CreationDate="2016-12-26T08:15:08.130" Score="1" Body="&lt;p&gt;A partial dependence plot might be what you are after: these plots show the relationship between the probability and the input variable. The mlr package in R takes care of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Discussion of Partial Dependence on XGBoost Git: &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/486&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/xgboost/issues/486&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;General Tutorial using mlr: &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/devel/html/partial_dependence/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://mlr-org.github.io/mlr-tutorial/devel/html/partial_dependence/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="25800" LastEditDate="2017-01-05T06:37:43.227" LastActivityDate="2017-01-05T06:37:43.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15932" PostTypeId="2" ParentId="15922" CreationDate="2016-12-26T08:37:21.637" Score="1" Body="&lt;p&gt;Seems right. However, establishing causality will not be as simple as extracting keywords and noticing the differences. And I would suggest not to divide the posts, instead club them together run LDA, extract the keywords, then analyse the differences. By separating you are introducing quite a huge bias into your model.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-26T08:37:21.637" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15933" PostTypeId="2" ParentId="15922" CreationDate="2016-12-26T09:51:56.613" Score="0" Body="&lt;p&gt;Instead of diving into LDA directly, I would be rather start with simpler ones like &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF&lt;/a&gt; and see whether it can extract keywords from each class/blog. Recently I got into this kind of problem where I need to extract topics out of tweets and I got fruitful results with TF-IDF being a part of my method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would treat each blog as individual data point rather than merging them, so that documents can be clubbed based on similarity of words obtained and then extracting the topic out of them. At the end you can use views to see the average views each topic has got.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well you got tools like &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_semantic_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSA&lt;/a&gt; which constructs a matrix based on word counts. This matrix is reduced by SVD which can be computationally time taking on matrices of huge sizes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So before trying any of the bigger methods, do try simpler ones and if the results are unsatisfactory, approach other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-12-26T09:51:56.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15934" PostTypeId="2" ParentId="15915" CreationDate="2016-12-26T10:25:04.550" Score="6" Body="&lt;p&gt;When you have sensors, the values you receive change even if the signal that was recorded didn't change. This is one example of noise.&lt;/p&gt;&#xA;&lt;p&gt;When you have a model of the world, it abstracts from the real relationships by simplifying things which are not too important. To take into account for the simplification, you model the error as noise (e.g. in a &lt;a href=&quot;https://martin-thoma.com/kalman-filter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kalman filter&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;But noise sources can be anything. For example, in an image classification problem, data compression can distort an image. Images can have different resolution; low resolution signals are harder to classify than high resolution figures. Aliasing effects can also distort images.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;And what is additive noise?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Suppose your system equation is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$z = H \cdot x$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$z \in \mathbb{R}^{n_m}$&lt;/span&gt; is the observation, &lt;span class=&quot;math-container&quot;&gt;$x \in \mathbb{R}^{n_x}$&lt;/span&gt; is the state you're interested in and &lt;span class=&quot;math-container&quot;&gt;$H \in \mathbb{R}^{n_m \cdot n_x}$&lt;/span&gt; is a transformation matrix. Then the noise could interact with your system in any way. But most of the time it is logical and practical that the noise is additive, meaning your model is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$z = H \cdot x + r$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; is sampled from a random variable of any distribution.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What is (additive) Gaussian noise?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$r \sim \mathcal{N}(\mu, \sigma^2)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;See &lt;a href=&quot;https://en.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;normal distribution&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="-1" LastEditDate="2021-03-09T06:16:23.703" LastActivityDate="2021-03-09T06:16:23.703" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="15935" PostTypeId="1" CreationDate="2016-12-26T11:59:16.473" Score="1" ViewCount="71" Body="&lt;p&gt;I am interested in joining some Data Science Groups on Linked In.  The relevant list is long.  I don't think it is to my advantage to join all, but I am not sure how to evaluate the individual group's potential to elevate my career.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some clear criteria for which I can evaluate potential groups? &lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2016-12-26T21:35:31.033" Title="Data Science Groups on Linked In" Tags="|social-network-analysis|career|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15936" PostTypeId="2" ParentId="15891" CreationDate="2016-12-26T14:33:39.207" Score="2" Body="&lt;p&gt;You can start with small lists of antonyms like this &lt;a href=&quot;http://www.englishleap.com/vocabulary/antonym&quot; rel=&quot;nofollow noreferrer&quot;&gt;one&lt;/a&gt;.&#xA;Maybe you can get a comprehensive list that will take care of most of the cases you are interested in. For now, let assume that you don't have such a list an discuss algorithmic ways to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you wrote regarding prefixes like &quot;un&quot; and &quot;dis&quot;, you can use rules based of morphology too. Such rules are likely to have high precision (pairs that obey the morphological rule will be antonyms) but low recall (you will miss many pairs).&#xA;We should use these rules in order to increase our dataset of antonyms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we should take a dataset of texts as as Wikipeida or the Wall Street Journal even Google ngrams data set. Antonyms will tend to appear in the same context but not together.&#xA;For example, people will write about &quot;ordered list&quot;, &quot;unordered list&quot; but not about &quot;ordered unordered list&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A proper association for this purpose is words that appear in distance smaller than X more than Y times and the joint probability is higher than the expected by Z.&#xA;You can use the dataset of antonyms we have in order to find a proper values of the parameter above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you calculate the association level among words, the antonyms will be associated to the same words but not associated to each other. Note that here you are expected to have high recall but lower precision since synonyms also tend to have such relations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While that option of lists is of the lowest effort and the lowest benefit, and the association method takes a considerable effort, there is another middle way.&#xA;Many dictionaries, like &lt;a href=&quot;https://en.wiktionary.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;wiktionary&lt;/a&gt;, have a section of antonyms. You can scrape them and build such a list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such a list is of great use and I was quite surprised that there wasn't such a common resource. If you will build one and will be willing to share it, you'll be very helpful. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2016-12-26T14:33:39.207" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15937" PostTypeId="1" CreationDate="2016-12-26T14:48:19.327" Score="0" ViewCount="1003" Body="&lt;p&gt;I am new to the Machine Learning area and I have a question to ask. But let me first post the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&#xA;The problem is very simple. I want to classify images as Category-1(&quot;Images containing garbage&quot;) or Category-2(&quot;Images not containing garbage&quot;). Garbage is used in every literal sense of the word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Solution I opted for:&lt;/strong&gt;&#xA;The solution also happens to be pretty straightforward for the most part. Extract points of interest using an Algorithm like SIFT,SURF etc etc. Then obtain the descriptors of these key points and cluster them using the K-means algorithm. Then using this clustered data generate the bag of words and proceed from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing which I am unable to comprehend is the number of clusters which I might need. Any help with the above example would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="27389" LastActivityDate="2016-12-30T15:03:37.367" Title="Deciding the number of clusters in K-means clustering of descriptors" Tags="|machine-learning|clustering|k-means|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15938" PostTypeId="2" ParentId="15935" CreationDate="2016-12-26T14:52:36.870" Score="0" Body="&lt;p&gt;I found &lt;a href=&quot;http://www.forbes.com/sites/williamarruda/2015/09/30/the-best-way-to-get-started-with-linkedin-groups/#61bf120356a4&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; at Forbes.com.  It does answer my question directly. In summary it says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define what it is that you are looking for from the group: connections, information, exposure.  To do this you need to know who is your authentic self.  Put another way, what is your &lt;strong&gt;Brand&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be discriminant about the groups you join.  Only join those that further the goals you have defined.  Research the potential groups to find out if they are in line with your goals.  Join and try the groups found with your research efforts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KDNuggets used number of members, growth rate, number of conversations and member engagement in their &lt;a href=&quot;http://www.kdnuggets.com/2016/10/top-linkedin-groups-analytics-big-data-mining-data-science.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;analysis of data science groups&lt;/a&gt;.  This analysis can be used to answer if the specified groups are inline with your goals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article goes on to give advice about etiquette once joined, but that is beyond the scope of my question.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastEditorUserId="24409" LastEditDate="2016-12-26T21:35:31.033" LastActivityDate="2016-12-26T21:35:31.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15939" PostTypeId="1" CreationDate="2016-12-26T15:13:17.570" Score="1" ViewCount="98" Body="&lt;p&gt;For the XOR problem, 2 decision boundaries are needed to solve it using 2 inputs neurons, 2 hidden neurons, 1 output neuron. From the book &quot;Neural Network Design&quot; on page 84, the author says that &quot;each neuron in the network divides the input space into two regions.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the xor network, there are 3 neurons excluding the inputs. Therefore 3 decision boundaries should be drawn. But most books show only 2 decision boundaries for the xor problem. Can someone explain why is it so?&lt;/p&gt;&#xA;" OwnerUserId="27200" LastActivityDate="2016-12-26T15:13:17.570" Title="neural network decision boundary" Tags="|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15940" PostTypeId="1" AcceptedAnswerId="15941" CreationDate="2016-12-26T15:30:33.053" Score="3" ViewCount="75" Body="&lt;p&gt;I remember years ago, Yahoo detailed how they were able to reduce a webpage down to a short paragrah of text succently summarising the content in sentences, as opposed to a list of keywords. What is this called? Are there any open / free code to do this?&lt;/p&gt;&#xA;" OwnerUserId="2861" LastActivityDate="2016-12-26T15:33:51.060" Title="Reducing a page of content to a short paragraph" Tags="|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15941" PostTypeId="2" ParentId="15940" CreationDate="2016-12-26T15:33:51.060" Score="2" Body="&lt;p&gt;Look into TextRank algorithm, here is the paper &lt;a href=&quot;https://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. You can find a neat python implementation here &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://github.com/summanlp/textrank&amp;amp;ved=0ahUKEwiSpsaXl5LRAhWIn5QKHW0CBlIQFgglMAE&amp;amp;usg=AFQjCNGjD49GTQDcAMWDEDAuhV-KiZEI4Q&amp;amp;sig2=MqCu22mjcHy89ob9sZNIYg&quot; rel=&quot;nofollow noreferrer&quot;&gt;textrank implementation&lt;/a&gt; If I am not wrong gensim also offers an implementation. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-26T15:33:51.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15942" PostTypeId="2" ParentId="15937" CreationDate="2016-12-26T15:39:32.370" Score="0" Body="&lt;p&gt;Why go for something so complicated? You can do it quite simply with a big enough dataset using convolutional neural network. There are tons of material available discussing the use of CNNs to image classification, &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/&amp;amp;ved=0ahUKEwiB7beUmJLRAhXEo5QKHUBNC9wQFgg0MAI&amp;amp;usg=AFQjCNGXKwI8lACXDndxiXyRlr158yfJjg&amp;amp;sig2=Ya3A7rjJEJJ86IwmOeTAUQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;MNIST digit recognition&lt;/a&gt;. Hope this helps. You can use PIL to extract pixel values. Though you will need labelled data. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-26T15:39:32.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15943" PostTypeId="2" ParentId="15935" CreationDate="2016-12-26T16:24:41.980" Score="1" Body="&lt;p&gt;You are limited to&#xA;&lt;a href=&quot;https://www.quora.com/What-is-the-max-number-of-groups-on-LinkedIn-allowed&quot; rel=&quot;nofollow noreferrer&quot;&gt;50 groups&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KDNuggets used number of members, growth rate, number of conversations and member engagment in their &#xA;&lt;a href=&quot;http://www.kdnuggets.com/2016/10/top-linkedin-groups-analytics-big-data-mining-data-science.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;analysis of data science groups&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27360" LastActivityDate="2016-12-26T16:24:41.980" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15945" PostTypeId="1" AcceptedAnswerId="15949" CreationDate="2016-12-26T21:13:03.400" Score="0" ViewCount="72" Body="&lt;p&gt;I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets? I was considering a Pearson correlation, but after thinking about things more I don't believe that it is the most appropriate correlation metric. &#xA;My data looks similar to below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&#xA;Before&amp;nbsp;&amp;nbsp;&amp;nbsp;After&#xA;1&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;2&#xA;1&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp&amp;nbsp;&amp;nbsp;&amp;nbsp;2&#xA;2&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp&amp;nbsp;&amp;nbsp;&amp;nbsp;2&#xA;3&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp&amp;nbsp;&amp;nbsp;&amp;nbsp;3&#xA;3&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp&amp;nbsp;&amp;nbsp;&amp;nbsp;4&#xA;4&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp&amp;nbsp;&amp;nbsp;&amp;nbsp;2&#xA;&#xA;&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt; XXX?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks all&lt;/p&gt;&#xA;" OwnerUserId="27396" LastActivityDate="2016-12-27T04:03:36.240" Title="Measuring Difference Between Two Sets of Likert Values" Tags="|dataset|data|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15949" PostTypeId="2" ParentId="15945" CreationDate="2016-12-27T04:03:36.240" Score="2" Body="&lt;p&gt;If you are trying to see whether the survey had made any positive effect on the people, modeling a relationship between before and after with correlation is a fair approach while avoiding some edge cases. Correlation looks whether $X's$ and $Y's$ are linearly correlated or how one variable is influenced by other. But I guess what you might be looking for is just something simpler like average increase in rating after the activity. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from scipy import stats&#xA;ratings_before = np.array([1,2,1,3,4,5])&#xA;ratings_after = np.array([5,5,5,5,5,5])&#xA;print &quot;P.Corr: &quot;,stats.pearsonr(ratings_before, ratings_after)[0]&#xA;print &quot;Avg.Increase: &quot;,np.mean(ratings_after-ratings_before)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Nan&#xA;2.333&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And assume that your activity went so good that everyone rated you a 5-star. Then correlation turns as $NaN$ as denominator goes to zero as no $Y$ can be explained by $X$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2016-12-27T04:03:36.240" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15952" PostTypeId="1" CreationDate="2016-12-27T05:08:42.457" Score="1" ViewCount="199" Body="&lt;p&gt;I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet for research. However, I am just wondering, what are the data sources from which I can get Social Network Data apart from the Facebook, Linkedin, Twitter connections.&#xA;Can I get (buy) phone call records, or any other social network dataset?&#xA;Also, do FB, linkedin sell their connections data?&lt;/p&gt;&#xA;" OwnerUserId="24742" LastActivityDate="2016-12-27T13:56:57.803" Title="How to get Real life Social Network Data?" Tags="|dataset|social-network-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15956" PostTypeId="1" CreationDate="2016-12-27T13:42:28.397" Score="1" ViewCount="811" Body="&lt;p&gt;I am using xgboost for a recommender system. There are 3-4 recommended content on each page. My data consists of columns like page_id and advertisement_id. Currently for every page_id, there are 3-4 rows of advertisement_id in data. Xgboost then gives me the probability of likelihood of an ad being clicked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is known that one content(advertisement_id) from each page(page_id) is definitely clicked. I want to leverage this fact while training xgboost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I tell xgboost that one ad_id from each page_id is definitely clicked?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this be done somehow by using a vector of advertisement_id for each row of page_id? &lt;/p&gt;&#xA;" OwnerUserId="26976" LastActivityDate="2016-12-27T13:42:28.397" Title="Feeding data to Xgboost for recomender system" Tags="|xgboost|recommender-system|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15957" PostTypeId="2" ParentId="15952" CreationDate="2016-12-27T13:56:57.803" Score="1" Body="&lt;p&gt;Take a look at these links:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://opendata.stackexchange.com/questions/1670/any-cdr-call-data-record-dataset&quot;&gt;https://opendata.stackexchange.com/questions/1670/any-cdr-call-data-record-dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/26229570/dataset-needed-for-cdr-analysis&quot;&gt;https://stackoverflow.com/questions/26229570/dataset-needed-for-cdr-analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://vlado.fmf.uni-lj.si/pub/networks/doc/seminar/nicta02.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://vlado.fmf.uni-lj.si/pub/networks/doc/seminar/nicta02.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="27360" LastActivityDate="2016-12-27T13:56:57.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15960" PostTypeId="1" AcceptedAnswerId="15961" CreationDate="2016-12-27T19:28:17.920" Score="5" ViewCount="1281" Body="&lt;p&gt;I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I have removed duplicates and near-duplicates. What are the well-known techniques for reducing sample size without losing too much of the information possibly encoded in the initial dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about using some clustering algorithm (which scales well with respect to number of clusters, possibly &lt;em&gt;BIRCH&lt;/em&gt;) and use the resulting clusters to find $ N $ nearest points to cluster centroid. However this feels somehow wrong.&lt;/p&gt;&#xA;" OwnerUserId="10751" LastActivityDate="2016-12-28T10:12:55.523" Title="Reducing sample size" Tags="|dataset|optimization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15961" PostTypeId="2" ParentId="15960" CreationDate="2016-12-27T19:47:57.473" Score="6" Body="&lt;p&gt;One method would be to take many subsets of your dataset, i.e.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bootstrapping_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;bootstrapping&lt;/a&gt;, build your models, perform cross-validation and calculate the average performance.  This is a good explanation of how the amount of data affects the model outcomes: &lt;a href=&quot;https://stackoverflow.com/questions/25665017/does-the-dataset-size-influence-a-machine-learning-algorithm&quot;&gt;https://stackoverflow.com/questions/25665017/does-the-dataset-size-influence-a-machine-learning-algorithm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Play around with the size of your subsets until you start getting stable results.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2016-12-28T09:40:07.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15962" PostTypeId="1" AcceptedAnswerId="15976" CreationDate="2016-12-27T22:50:17.103" Score="11" ViewCount="9048" Body="&lt;p&gt;I am using tensorflow to write simple neural networks for a bit of research and I have had many problems with 'nan' weights while training. I tried many different solutions like changing the optimizer, changing the loss, the data size, etc. but with no avail. Finally, I noticed that a change in the learning rate made an unbelievable difference in my weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a learning rate of .001 (which I thought was pretty conservative), the minimize function would actually exponentially raise the loss. After one epoch the loss could jump from a number in the thousands to a trillion and then to infinity ('nan'). When I lowered the learning rate to .0001, everything worked fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Why does a single order of magnitude have such an effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Why does the minimize function literally perform the opposite of its function and maximize the loss? It seems to me that that shouldn't occur, no matter the learning rate.&lt;/p&gt;&#xA;" OwnerUserId="27420" LastEditorUserId="27420" LastEditDate="2019-04-17T19:21:46.580" LastActivityDate="2019-04-17T19:21:46.580" Title="Why is learning rate causing my neural network's weights to skyrocket?" Tags="|machine-learning|python|tensorflow|optimization|gradient-descent|" AnswerCount="2" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="15964" PostTypeId="1" AcceptedAnswerId="15978" CreationDate="2016-12-28T01:03:56.870" Score="2" ViewCount="90" Body="&lt;p&gt;I am trying to get idea how variables of my data correspond to target variable (binary class).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regression, Pearson r statistic is quite good to get sense of variable relationship. Also I can use it for classification, treating classes 0 and 1 as real values but it's a risky trick.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: is there any equivalent statistic saying which variable is good for classification task ? Thanks&lt;/p&gt;&#xA;" OwnerUserId="26701" LastActivityDate="2016-12-29T00:22:30.033" Title="analogy for pearson r statistics for binary classification task" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15965" PostTypeId="1" AcceptedAnswerId="15967" CreationDate="2016-12-28T01:36:16.590" Score="8" ViewCount="12220" Body="&lt;p&gt;When would a neural network be defined as a Deep Neural Network (DNN) and not a NN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A DNN as I understand them are neural networks with many layers, and simple neural networks usually have fewer layer... but what a many and a few in numbers? or is there some other definition?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are networks trained used Tensorflow, Caffee as such? I haven't (as far I know) seen anybody manually design a network with many many layers. &#xA;They seem to promote their tools for creating DNN, but is it actually DNN if you only make a network with two layers? &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="25157" LastEditDate="2016-12-31T11:46:37.043" LastActivityDate="2017-02-25T19:30:25.110" Title="When is something a Deep Neural Network (DNN) and not NN?" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15966" PostTypeId="2" ParentId="15358" CreationDate="2016-12-28T02:33:09.223" Score="2" Body="&lt;p&gt;Like one of the commenters, I also question the utility of these scatterplots in this situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you've described is a standard &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-class classification problem&lt;/a&gt;. Your outcomes are labeled (clubs, spades, etc.), and you have seven features (f2-f8) with which to predict the outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, simply try putting &lt;em&gt;all&lt;/em&gt; of the features into a standard classification model (e.g., logistic regression), and see what comes out. If you believe that some features are unimportant, you can eliminate those features from the model, and examine the cross-validation score (accuracy, f1-score, etc.) to see if the feature was truly predictive or not. For a linear model, looking at the coefficients and their standard errors can also be useful in this regard. Another approach for feature selection for generalized linear models is to examine the &lt;a href=&quot;https://en.wikipedia.org/wiki/Deviance_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;deviance&lt;/a&gt;. Hill/Gelman write in their book (Data Analysis using Regression and Multilevel/Hierarchical Models) that &quot;when an informative predictor is added to a model, we expect the deviance to decrease by more than 1. When $k$ predictions are added to a model, we expect the deviance to decrease by more than $k$&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you should &lt;em&gt;not&lt;/em&gt; do is to examine scatter plots to determine which features are important. The fundamental problem is that even though the data might be well-separated in seven dimensions (i.e., using all seven features), this separation might not show up well in a scatter plot since this plot is only two-dimensional.&lt;/p&gt;&#xA;" OwnerUserId="27421" LastActivityDate="2016-12-28T02:33:09.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15967" PostTypeId="2" ParentId="15965" CreationDate="2016-12-28T04:35:52.147" Score="10" Body="&lt;p&gt;You are right. Mainly any network with more than two layers between the input and output is considered a deep neural network. Libraries like tensorflow provide efficient architecture for deep learning applications such as image recognition, or language modelling using Convolutional neural networks and Recurrent neural networks. Another thing to keep in mind, is the depth of the network also has to do with the number of units being used in the layer. Mainly, as your non-linear hypotheses get complex you will need deep neural networks. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2016-12-28T04:35:52.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15969" PostTypeId="2" ParentId="15960" CreationDate="2016-12-28T10:12:55.523" Score="4" Body="&lt;p&gt;It depends: Some algorithms would benefit from near duplicates (such as kNN), while some of them use the outliers of clusters to build their rules on (such as SVM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience it is very important that you understand how your data is structured and what concept you are actually trying to learn, before you settle on a downsampling method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently ran into a problem in which I wanted to predict a certain behavior in time, but local dependecies (spacially as in x,y) kept throwing the algorithm in local optima. Downsampling maximizing distibution over x,y actually increased performance of almost all trained models.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2016-12-28T10:12:55.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15970" PostTypeId="2" ParentId="15891" CreationDate="2016-12-28T10:31:01.453" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word2Vec&lt;/a&gt; can be used to find a word that relates to another word in the same way an example pair does. (fi: &lt;strong&gt;x&lt;/strong&gt; is to &lt;em&gt;happy&lt;/em&gt;, as &lt;em&gt;bad&lt;/em&gt; is to &lt;em&gt;good&lt;/em&gt;). You could use that to generate candidates of antonyms on the fly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might not be as accurate as precompiled list, but it will cover almost every candidate. In fact Word2Vec could also help you find other (other than antonyms) likely candidates for likely words given a context.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2016-12-28T10:31:01.453" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="15973" PostTypeId="1" AcceptedAnswerId="15987" CreationDate="2016-12-28T15:25:35.603" Score="0" ViewCount="1902" Body="&lt;p&gt;I have the following code in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from scipy.sparse import csr_matrix&#xA;M = csr_matrix(np.ones([2, 2],dtype=np.int32))&#xA;print(M)&#xA;print(M.data.shape)&#xA;for i in range(np.shape(M)[0]):&#xA;    for j in range(np.shape(M)[1]):&#xA;        if i==j:&#xA;            M[i,j] = 0&#xA;print(M)&#xA;print(M.data.shape)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of the first 2 prints is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  (0, 0)    1&#xA;  (0, 1)    1&#xA;  (1, 0)    1&#xA;  (1, 1)    1&#xA;(4,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code is changing the value of the same index (i==j) and setting the value to zero.&#xA;After executing the loops then the output of the last 2 prints is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  (0, 0)    0&#xA;  (0, 1)    1&#xA;  (1, 0)    1&#xA;  (1, 1)    0&#xA;(4,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I understand the concept of sparse matrices correctly, it should not be the case. It should not show me the zero values and the output of last 2 prints should be like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  (0, 1)    1&#xA;  (1, 0)    1&#xA;(2,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone have explanation for this? Am I doing something wrong?&lt;/p&gt;&#xA;" OwnerUserId="27435" LastEditorUserId="12384" LastEditDate="2016-12-31T11:46:25.810" LastActivityDate="2016-12-31T11:46:25.810" Title="CSR scipy matrix does not update after updating its values" Tags="|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15974" PostTypeId="1" CreationDate="2016-12-28T16:07:24.727" Score="5" ViewCount="714" Body="&lt;p&gt;I am looking for some general advice on where to start with this problem.  There are 350 sparse (low positive integer) features.  I have 2000 positives, 1000 negatives, and infinite unlabeled data, where the estimated true positive rate in the unlabeled data is ~1% (but this isn't known for certain).  The goal is to predict the probability of an example being positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you approach the problem if you had roughly 40 hours to attain the best possible model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about if you had more time?&lt;/p&gt;&#xA;" OwnerUserId="27436" LastActivityDate="2017-01-05T00:34:49.347" Title="General strategy for imbalanced, semi-supervised, sparse problem" Tags="|machine-learning|predictive-modeling|semi-supervised-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15975" PostTypeId="2" ParentId="15973" CreationDate="2016-12-28T16:15:29.833" Score="0" Body="&lt;p&gt;&lt;strong&gt;Sparse matrix&lt;/strong&gt; is a matrix that has most of its elements as zero. Matrix on the other side of it, is &lt;strong&gt;Dense&lt;/strong&gt;. &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.sparse.csr_matrix.html#scipy.sparse.csr_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;csr_matrix&lt;/a&gt; creates an all zero element matrix and if you are updating the elements as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m[i,j] = 0. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Well, you're not really doing anything as elements are already zero. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from scipy.sparse import csr_matrix&#xA;mat = csr_matrix((2,2), dtype = np.int16).toarray()&#xA;print mat,&quot;\n&quot;&#xA;for i in range(np.shape(mat)[0]):&#xA;    for j in range(np.shape(mat)[1]):&#xA;        mat[i,j] = i+j&#xA;print mat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[0 0]&#xA; [0 0]]&#xA;&#xA;[[0 1]&#xA; [1 2]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is true, the first matrix,an all zero matrix created and the second, is the updated one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After substituting when $i==j$, the resultant matrix still has four elements but this time with $(0,0)$ and $(1,1)$ indices being $0$. So the result is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0, 0)    0&#xA;(0, 1)    1&#xA;(1, 0)    1&#xA;(1, 1)    0&#xA;(4,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But explaining your expected result&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0, 1)    1&#xA;(1, 0)    1&#xA;(2,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result has basically two elements with $(0,1)$ and $(1,0)$ indices being $1$ and $1$ and shape is just $(2,)$ when flattened which does not make sense as matrix is still of size $(4,)$ . So python output is basically giving all the values in the matrix with index. If I am rightly understanding what you are assuming, since the matrix is sparse, you are expecting it to give only the values where the elements are non-zero, well this does not do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2016-12-28T19:50:49.220" LastActivityDate="2016-12-28T19:50:49.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15976" PostTypeId="2" ParentId="15962" CreationDate="2016-12-28T16:56:21.230" Score="9" Body="&lt;p&gt;You might find Chapter 8 of &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;noreferrer&quot;&gt;Deep Learning&lt;/a&gt; helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility (besides user error) is that your problem is highly ill-conditioned. Gradient descent methods use only the first derivative (gradient) information when computing an update. This can cause problems when the second derivative (the Hessian) is ill-conditioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quoting from the authors:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Some challenges arise even when optimizing convex functions. Of these, the most prominent is ill-conditioning of the Hessian matrix $H$. This is a very general problem in most numerical optimization, convex or otherwise, and is described in more detail in section 4.3.1.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The ill-conditioning problem is generally believed to be present in neural network training problems. Ill-conditioning can manifest by causing SGD to get “stuck” in the sense that &lt;em&gt;even very small steps increase the cost function.&lt;/em&gt; [my emphasis added]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The authors provide a simple derivation to show that this can be the case. Using gradient descent, the cost function should change (to second order) by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\frac{\varepsilon^2}{2} g^{T} H g - \varepsilon g^{T} g&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $g$ is the gradient, $H$ is the Hessian, and $\varepsilon$ is the learning rate. Clearly, if the second derivatives are large, then the first term can swamp the second, and the cost function will increase, not decrease. Since the first and second terms scale differently with $\varepsilon$, one way to alleviate this problem is to reduce $\varepsilon$ (although, of course, this can result in learning &lt;em&gt;too&lt;/em&gt; slowly). &lt;/p&gt;&#xA;" OwnerUserId="27421" LastActivityDate="2016-12-28T16:56:21.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15977" PostTypeId="2" ParentId="15314" CreationDate="2016-12-28T17:35:42.410" Score="4" Body="&lt;p&gt;To build off Mashimo's answer, one straightforward approach for topic modeling is &quot;Latent Dirichlet Allocation&quot; (LDA). The basic idea behind LDA is explained in &lt;a href=&quot;http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this really good tutorial&lt;/a&gt;. Essentially, documents are assumed to be composed of mixtures of topics, which are in turn composed of mixtures of words. If we knew the topic and document distributions, we could generate new documents using a probabilistic model. In LDA, we run this process in reverse to infer the topic and document distributions given the documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the tutorial, the author uses LDA to find topics within Sarah Palin's emails, which is not too dissimilar to what you're trying to do, if I understand correctly. For example, one topic is composed of the words &quot;gas, oil, pipeline, agia, project, natural, north&quot;, which corresponds roughly to the topic &quot;energy&quot; or &quot;gas&quot;. Note that LDA doesn't name the topics for you; you'll have to apply your own judgment to construct a sensible name for the group of words comprising a topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LDA has been implemented in packages like &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gensim&lt;/a&gt;. To see how to use LDA in Python, you might find &lt;a href=&quot;http://nbviewer.jupyter.org/github/skipgram/modern-nlp-in-python/blob/master/executable/Modern_NLP_in_Python.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this SpaCy tutorial&lt;/a&gt; (which covers a lot of stuff in addition to LDA) useful.&lt;/p&gt;&#xA;" OwnerUserId="27421" LastActivityDate="2016-12-28T17:35:42.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15978" PostTypeId="2" ParentId="15964" CreationDate="2016-12-28T19:57:26.683" Score="5" Body="&lt;p&gt;The Pearson Chi-Squared test may give you what you need:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a good resource for analyzing categorical data:  &lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, confusion matrices can be very helpful:  &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Confusion_matrix&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21951" LastActivityDate="2016-12-28T19:57:26.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15980" PostTypeId="2" ParentId="15937" CreationDate="2016-12-28T21:50:30.443" Score="1" Body="&lt;h2&gt;Measure, don't guess&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It appears to be common to try k=1024, 2048, 4096, ... and use what works best for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is possivle because the clustering is actually a quantization task, and you do classification afterwards.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2016-12-28T21:50:30.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15982" PostTypeId="1" CreationDate="2016-12-29T02:30:57.607" Score="1" ViewCount="3338" Body="&lt;p&gt;&lt;strong&gt;Short Version:&lt;/strong&gt; How do I get renderUI to react to a drop-down list that is in a wellPanel that contains a submitButton &lt;em&gt;without having to click the submitButton&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Long Version:&lt;/strong&gt; I am building a Shiny app to perform many different analyses on a large dataset. The user will select their analysis from a drop-down list at the top of a wellPanel. Depending on the analysis selected, I'd like to provide a different set of additional inputs/filters in the wellPanel with a submitButton at the bottom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like the user to have to change the dynamic inputs, then click the submitButton for their inputs to take effect. However, I want the renderUI function to react to the drop-down list at the top &lt;em&gt;without&lt;/em&gt; the submitButton being clicked. From what I've read, I think I need to use observeEvent or something similar, but I can't figure it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simplified version of what I'm building. Changing the drop-down list selection and clicking submit will change the inputs. Submit will also change the text rendered on the right. However, I need to uncouple the dynamic UI from the submit button. Any help is much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ui.r:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(shiny)    &#xA;&#xA;ui &amp;lt;- fluidPage(&#xA;    fluidRow(&#xA;      column(4,wellPanel(&#xA;          selectInput(&quot;analysis&quot;, label = &quot;Type of Analysis:&quot;, &#xA;                    c(&quot;Award Total&quot; = &quot;total&quot;, &#xA;                      &quot;Award Average&quot; = &quot;average&quot;),&#xA;                    width = validateCssUnit(&quot;70%&quot;)),&#xA;          uiOutput(&quot;filter_box&quot;),&#xA;          submitButton()&#xA;               )),&#xA;      column(8, textOutput(&quot;sample_text&quot;))&#xA;      )&#xA;  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;server.r:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shinyServer(function(input, output) {&#xA;&#xA;  output$filter_box &amp;lt;- renderUI({&#xA;if(input$analysis == &quot;total&quot;){&#xA;  tagList(radioButtons(inputId = &quot;input1&quot;, label = &quot;Select One:&quot;,c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;), selected = &quot;A&quot;))&#xA;} else {&#xA;  tagList(checkboxGroupInput(inputId = &quot;input2&quot;, label = &quot;Select all that apply:&quot;,c(&quot;1&quot;,&quot;2&quot;,&quot;3&quot;,&quot;4&quot;,&quot;5&quot;)),&#xA;          dateRangeInput(inputId = &quot;input3&quot;, label = &quot;Enter Date Range&quot;))&#xA;}&#xA;  })&#xA;&#xA;  output$sample_text &amp;lt;- renderText({&#xA;if(input$analysis == &quot;total&quot;){&#xA;  input$input1&#xA;} else if(input$analysis == &quot;average&quot;) {&#xA;  c(input$input2, input$input3)&#xA;}&#xA;  })&#xA;&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18429" LastActivityDate="2016-12-29T02:30:57.607" Title="Shiny: renderUI with Submit Button - How to make renderUI react to selectInput without Submit Button being pressed?" Tags="|r|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="15983" PostTypeId="1" CreationDate="2016-12-29T03:02:14.170" Score="2" ViewCount="103" Body="&lt;p&gt;I have a dataset of approximately 48,000 rows each one a click of a an article, some of these clicks were also comments. For each article I have the country and subject of the article and name of person who clicked on it these are all categorical variables with different levels from each other. I want to determine which of these categorical variables has the highest association or correlation with comments i.e. based the dataset is the best variable country, subject, or person to predict whether or not there will be a comment on an article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which statistical method would you use to determine correlation and how would you implement it in R?&lt;/p&gt;&#xA;" OwnerUserId="17267" LastActivityDate="2016-12-29T03:02:14.170" Title="Type of Test to Determine Correlation in R" Tags="|r|logistic-regression|correlation|categorical-data|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="15984" PostTypeId="1" AcceptedAnswerId="16188" CreationDate="2016-12-29T03:35:35.837" Score="6" ViewCount="6276" Body="&lt;p&gt;I am new to machine learning, working on object detection, but not interested in the location of the object in the image, so I just want to know is it possible to train such a neural network, if yes, how? (I just want a list of objects present in the image). I am not sure what kind of dataset I would require to build such a network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: I will be more clear on my question. First I know about CNN and the models like VGG16, Inception, Resnet etc. My doubt is that these networks can be trained to identify an object i.e. it will predict which object is present in the image and give the probability. What if I want to train a network to tell whether particular objects are &lt;strong&gt;present in the image or not&lt;/strong&gt; i.e. The output should be yes if any of the objects is present no if not&lt;/p&gt;&#xA;" OwnerUserId="27450" LastEditorUserId="27450" LastEditDate="2016-12-31T15:44:21.880" LastActivityDate="2019-07-31T23:38:50.007" Title="Can we train a neural network to tell if an object is present or not in an Image?" Tags="|deep-learning|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15985" PostTypeId="2" ParentId="15984" CreationDate="2016-12-29T05:59:33.513" Score="0" Body="&lt;p&gt;yes you can train but before that, you have to train network to identify the object you want. It means first you need to input dataset to the network with samples.  Let's take an example. If you want to identify chair object from the image then make the dataset of chairs having different designs. So based on that if you give any new chair image as an input to check it will identify that object as a chair from the trained dataset. &lt;/p&gt;&#xA;" OwnerUserId="27453" LastActivityDate="2016-12-29T05:59:33.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15986" PostTypeId="2" ParentId="15919" CreationDate="2016-12-29T10:34:58.277" Score="1" Body="&lt;p&gt;As I understood, it is a versioning issue. However, it should be noted that before installing the latest version, the old version should be manually removed. Then, try to install the new version. &lt;/p&gt;&#xA;" OwnerUserId="26019" LastActivityDate="2016-12-29T10:34:58.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15987" PostTypeId="2" ParentId="15973" CreationDate="2016-12-29T10:46:29.687" Score="0" Body="&lt;p&gt;It appears that CSR does not remove the zeros by default. You will first have to call &lt;code&gt;eliminate_zeros()&lt;/code&gt; on your object. Once you do this you will see that the data contains only the non zero elements of your matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;after running the loop in your code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(M)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  (0, 0)    0&#xA;  (0, 1)    1&#xA;  (1, 0)    1&#xA;  (1, 1)    0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;let us inspect the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(M.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;we see that the zero elements are stored,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0 1 1 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;let us remove the zero elements to make our matrix sparse&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;M.eliminate_zeros()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and inspect whether the matrix is sparse,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(M.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this confirms that the zero elements are removed and hence the matrix is sparse,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-12-29T10:46:29.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15988" PostTypeId="2" ParentId="15984" CreationDate="2016-12-29T10:52:29.197" Score="1" Body="&lt;p&gt;This problem has been researched in several projects. An interesting overview of articles on the topic can be found &lt;a href=&quot;https://github.com/kjw0612/awesome-deep-vision#object-detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. For a crash course in image recognition and neural nets see &lt;a href=&quot;http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; interesting post. I hope this helps you to get started.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2016-12-29T10:52:29.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15989" PostTypeId="1" CreationDate="2016-12-29T17:39:07.967" Score="289" ViewCount="373176" Body="&lt;p&gt;I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training a multiclass classifier on this dataset and I am getting the following performance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                    Precision           Recall           F1-Score&#xA;Micro Average       0.731               0.731            0.731&#xA;Macro Average       0.679               0.529            0.565&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure why all Micro average performances are equal and also Macro average performances are low compared to Micro average.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="-1" LastEditDate="2020-01-19T20:02:56.793" LastActivityDate="2022-04-02T09:28:24.697" Title="Micro Average vs Macro average Performance in a Multiclass classification setting" Tags="|multiclass-classification|model-evaluations|" AnswerCount="8" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="15990" PostTypeId="2" ParentId="12883" CreationDate="2016-12-29T18:13:10.873" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;K-means is a reasonable approach and a sensible way to understand the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've never used mahout, but I would use R or Python for this sort of analysis because of the nice libraries available to quickly implement K-means.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The clustering approach with the tags is fairly straightforward. You can essentially encode this using an indicator variable (also known as a binary encoding). You can set this variable/feature to 1 if the tag appeared in the list of tags and 0 otherwise. Then you only need to allocate space for the total number of tags that exist. If you have a large set of tags, you can limit them by taking tags with at least some frequency or some other &quot;sensible&quot; way.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can choose $k$ in a number of ways. Typically people choose K arbitrarily because they want, e.g., 10 groups to segment their customers or data into. In the simulation I've provided, it'll give you a lame way to optimize for K using an incremental improvement. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/franciscojavierarceo/Python/blob/master/Clustering%20with%20Unstructured%20Data.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;I've made a notebook with a simulation walking you through how to &quot;tokenize&quot; your tags and represent them with a binary/one-hot encoding.&lt;/a&gt; It's worth noting that this tokenization ignores the order of the tags, which may be okay for your use case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also worth noting that K-means certainly isn't the only way to measure the similarity of your data but I think it's a nice intuitive start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, for choosing $k$ the approach outlined in the notebook is exhaustive, since it starts from 1 and goes until each observation is a cluster. This means you'll have to run K-means $n$ times, which is silly in practice to do but useful from a learning perspective here. In general, this isn't ideal because it's expensive and typically you don't want to set $k = n$ but this simulation gives nice intuition about what's happening. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, you can just do it in gaps for a large number of clusters (e.g., 5, 10, 15, 20, .., 100) or something like that and choose the one that has the biggest drop-off by eye-balling it. This is a very arbitrary and unsatisfying way to choose $k$, but it seems to work okay for many people.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2017-01-01T14:38:44.453" LastActivityDate="2017-01-01T14:38:44.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="15991" PostTypeId="2" ParentId="13767" CreationDate="2016-12-29T18:49:35.863" Score="0" Body="&lt;p&gt;We need more details besides what you've provided here.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What does your data look like? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are you sure you don't have a bug in your data?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What does your LSTM architecture look like?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are you sure you're outputting the softmax for your output layer?&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are your hyperparameters? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Did you set the learning rate too high? If everything else is correct, it's probably this.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also, are the notes you have for your prediction mapped to Keras as categories/indices? Need to make sure it's not treating them as real-valued numbers.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastActivityDate="2016-12-29T18:49:35.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15992" PostTypeId="2" ParentId="12883" CreationDate="2016-12-29T21:27:51.230" Score="0" Body="&lt;p&gt;If you are looking to cluster a very large amount of data located in a relational SQL or Hadoop type database you may want to use some of the algorithms built for parallel processing on Spark. The ML Lib main package for Spark can do this but I would recommend the H2O package as the documentation seems top notch, gives you a choice of many languages and can consume Terabytes of data quickly. If you already have a good development background this may play more to your engineering skill set and with H2O's great documentation you can focus more on your data pipeline, and feature selection than the theoretical underpinnings of your clustering model. You can implement H2O on Spark in Java or Scala as well staying with more of your engineering prospective, rather than working in a scripting language like R or Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link to their K-means algorithm documentation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/k-means.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/k-means.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12861" LastActivityDate="2016-12-29T21:27:51.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15993" PostTypeId="1" CreationDate="2016-12-29T23:05:30.263" Score="1" ViewCount="711" Body="&lt;p&gt;I am currently working with a dataset with complex numbers, and wanted to know how neural network handles situation as such. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of activation should one use, what form of normalisation would sense and so on.  &lt;/p&gt;&#xA;" OwnerUserId="26085" LastActivityDate="2016-12-29T23:52:22.847" Title="How is complex number handled in NN?" Tags="|neural-network|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15994" PostTypeId="2" ParentId="15993" CreationDate="2016-12-29T23:52:22.847" Score="3" Body="&lt;p&gt;The theory of backpropagation &lt;em&gt;should&lt;/em&gt; work with complex numbers, it doesn't make any assumptions about all numbers being real. However, most NN libraries would not be able to calculate with them. Your weights and biases would also need to be complex. I have no idea if you could reasonably expect convergence using gradient descent - many activation functions have periodic output and numerically unstable regions when calculating with complex numbers, which makes me think it would be risky.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest thing to do would be to split each complex number up into its components at the input, representing as a 2-dimensional vector of real numbers - either using cartesian or polar approach. Neural networks should cope just fine with splitting features up like this.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2016-12-29T23:52:22.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15995" PostTypeId="2" ParentId="5600" CreationDate="2016-12-30T01:25:23.137" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Since the information on all variables is already captured by the cluster variable, using it again in the Lasso model with some other variables, does it become redundant/biased?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It does not become biased, at least not in the statistical sense. Depending on the clustering mechanism, the information may be redundant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using regression with the RMSE as the loss function and the squared-error as your loss metric for your clustering algorithm, then the information will work out to be redundant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, it's harmless to throw into the model and test this. The LASSO should learn that this information is redundant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a simulation in R.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(glmnet)&#xA;library(Matrix)&#xA;&#xA;n &amp;lt;- 1e5&#xA;nclusters &amp;lt;- 5&#xA;set.seed(420)&#xA;rmse &amp;lt;- function(y, yhat){&#xA;  return (sqrt( sum( (y-yhat)**2 )))&#xA;}&#xA;ls &amp;lt;- data.frame(sample(letters, n, replace=TRUE))&#xA;xs &amp;lt;- sparse.model.matrix(~.-1,data=ls)&#xA;&#xA;print(head(xs))  &#xA;&#xA;# Now let's run k-means&#xA;out &amp;lt;- kmeans(xs, centers=nclusters)&#xA;bs &amp;lt;- runif(dim(xs)[2])&#xA;&#xA;# Let's run k-means on the different categories&#xA;clusterpred &amp;lt;- data.frame(out[[1]])&#xA;ys &amp;lt;- xs %*% bs + rnorm(n)&#xA;print(table(clusterpred))&#xA;&#xA;# Now let's use a clustered data set to predict some outcome&#xA;cxs &amp;lt;- sparse.model.matrix(~.-1, data = data.frame(cluster = factor(clusterpred[,1])))&#xA;# Concatenating the original features and the assigned clusters&#xA;totalxs &amp;lt;- cbind2(xs, cxs)&#xA;&#xA;head(cxs)&#xA;&#xA;# Setting alpha = 1 implies LASSO for the GLMNET Package&#xA;model &amp;lt;- cv.glmnet(y=ys, x=xs, alpha=1)&#xA;cmodel &amp;lt;- cv.glmnet(y=ys, x=cxs, alpha=1)&#xA;&#xA;# Running on the clusters and the original features&#xA;totalmodel &amp;lt;- cv.glmnet(y=ys, x=totalxs, alpha=1)&#xA;&#xA;# Predictions&#xA;yhat &amp;lt;- predict(model, xs)&#xA;yhatc &amp;lt;- predict(cmodel, cxs)&#xA;yhatt &amp;lt;- predict(totalmodel, totalxs)&#xA;&#xA;# Looking at the difference RMSEs &#xA;print(rmse(ys, yhat))&#xA;print(rmse(ys, yhatc))&#xA;print(rmse(ys, yhatt))&#xA;&#xA;# It seems to select most of the features and *one* cluster&#xA;print(coef(totalmodel, s='lambda.min'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this simulation, we see that the performance of the model slightly worse, but fairly similar, from including the clusters. &lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2016-12-30T20:27:17.357" LastActivityDate="2016-12-30T20:27:17.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="15996" PostTypeId="1" CreationDate="2016-12-30T07:19:58.860" Score="0" ViewCount="457" Body="&lt;p&gt;While dealing with stochastic gradient descent, I got confused by several sources which explained it quite differently.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One source explained stochastic gradient descent as standard gradient descent approach but with substantial smaller training set.&lt;/li&gt;&#xA;&lt;li&gt;The other approach first sums the gradients of all training set and then multiplies with learning rate and updates it. $Q_{t+1}&amp;lt;-Q_t - \tau * \frac{n}{|S|}*\sum_{j\in S}\nabla L_j(Q_t)$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What is the difference? and How does it work with matrices since I take just couple of training examples? (see example below)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example:&#xA;I have sparse matrix M with user x movies where the value in the matrix represents how much the user rates the movie. I decomposed it into matrices P and Q and using stochastic gradient descent I want to optimize P and Q by minimizing the loss function.&#xA;Each value of M can be reconstructed by multiplying vector of Q and P.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I take batches of 10 so it means I take just 10 values from matrix M and respective rows and columns of P and Q which I will optimize.&#xA;If I take the second approach there is sum of gradients of all point and then I do the update but how can I sum those vectors since they do not correspond to the same place in matrix Q or P?&#xA;In this case, should I then just make new P and Q just with vectors that corresponds to taken values of M? but then it does not make sense because I should optimize the whole Q and P matrices right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I made it clear enough.&lt;/p&gt;&#xA;" OwnerUserId="27435" LastActivityDate="2016-12-30T14:54:59.023" Title="Stochastic gradient descent and different approaches" Tags="|optimization|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16001" PostTypeId="2" ParentId="15989" CreationDate="2016-12-30T09:53:59.037" Score="46" Body="&lt;p&gt;This is the &lt;a href=&quot;http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html&quot; rel=&quot;noreferrer&quot;&gt;Original Post&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics.&lt;/p&gt;&#xA;&lt;p&gt;Tricky, but I found this very interesting. There are two methods by which you can get such average statistic of information retrieval and classification.&lt;/p&gt;&#xA;&lt;h2&gt;1. Micro-average Method&lt;/h2&gt;&#xA;&lt;p&gt;In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics. For example, for a set of data, the system's&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;True positive (TP1)  = 12&#xA;False positive (FP1) = 9&#xA;False negative (FN1) = 3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Then precision (P1) and recall (R1) will be &lt;span class=&quot;math-container&quot;&gt;$57.14 \%=\frac {TP1}{TP1+FP1}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$80\%=\frac {TP1}{TP1+FN1}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;and for a different set of data, the system's&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;True positive (TP2)  = 50&#xA;False positive (FP2) = 23&#xA;False negative (FN2) = 9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Then precision (P2) and recall (R2) will be 68.49 and 84.75&lt;/p&gt;&#xA;&lt;p&gt;Now, the average precision and recall of the system using the Micro-average method is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\text{Micro-average of precision} = \frac{TP1+TP2}{TP1+TP2+FP1+FP2} = \frac{12+50}{12+50+9+23} = 65.96$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\text{Micro-average of recall} = \frac{TP1+TP2}{TP1+TP2+FN1+FN2} = \frac{12+50}{12+50+3+9} = 83.78$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The Micro-average F-Score will be simply the harmonic mean of these two figures.&lt;/p&gt;&#xA;&lt;h2&gt;2. Macro-average Method&lt;/h2&gt;&#xA;&lt;p&gt;The method is straight forward. Just take the average of the precision and recall of the system on different sets. For example, the macro-average precision and recall of the system for the given example is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\text{Macro-average precision} = \frac{P1+P2}{2} = \frac{57.14+68.49}{2} = 62.82$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$\text{Macro-average recall} = \frac{R1+R2}{2} = \frac{80+84.75}{2} = 82.25$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The Macro-average F-Score will be simply the harmonic mean of these two figures.&lt;/p&gt;&#xA;&lt;p&gt;Suitability&#xA;Macro-average method can be used when you want to know how the system performs overall across the sets of data. You should not come up with any specific decision with this average.&lt;/p&gt;&#xA;&lt;p&gt;On the other hand, micro-average can be a useful measure when your dataset varies in size.&lt;/p&gt;&#xA;" OwnerUserId="27426" LastEditorUserId="85045" LastEditDate="2021-01-06T21:25:51.170" LastActivityDate="2021-01-06T21:25:51.170" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="16002" PostTypeId="2" ParentId="9232" CreationDate="2016-12-30T11:18:07.327" Score="0" Body="&lt;p&gt;Most software packages allow you to perform a joint wald test of whether the additional parameters are significant. In other words, you estimate the more complex model and test whether you can reduce it to the simpler model by setting some parameters to zero. &lt;/p&gt;&#xA;" OwnerUserId="27397" LastActivityDate="2016-12-30T11:18:07.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16004" PostTypeId="2" ParentId="15984" CreationDate="2016-12-30T14:44:03.677" Score="7" Body="&lt;p&gt;A standard computer vision and deep learning dataset for this problem was developed by the Canadian Institute for Advanced Research (CIFAR). The CIFAR-10 dataset consists of 60,000 photos divided into 10 classes (hence the name CIFAR-10). Classes include common objects such as airplanes, automobiles, birds, cats and so on. The dataset is split in a standard way, where 50,000 images are used for training a model and the remaining 10,000 for evaluating its performance. Other datasets are MNIST, CIFAR-100 STL-10, SVHN, ILSVRC2012 etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;State of the art results are achieved using very large &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot; rel=&quot;noreferrer&quot;&gt;Convolutional Neural networks&lt;/a&gt;. You can learn about state of the are results on CIFAR-10 on &lt;a href=&quot;http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#43494641522d3130&quot; rel=&quot;noreferrer&quot;&gt;Rodrigo Benenson’s webpage&lt;/a&gt;. Model performance is reported in classification accuracy, with very good performance above 90%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a sample &lt;a href=&quot;http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/&quot; rel=&quot;noreferrer&quot;&gt;tutorial&lt;/a&gt; on convolutional neural network with caffe and python  &lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2016-12-30T14:44:03.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16005" PostTypeId="2" ParentId="15996" CreationDate="2016-12-30T14:54:59.023" Score="3" Body="&lt;p&gt;I think you're actually asking two separate questions: 1) What is stochastic gradient descent (SGD)? and 2) how do I find the matrices $P$ and $Q$ in the matrix factorization of $M$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For question 1, I don't think either of your explanations is correct. (Explanation 2 sounds to me like normal (batch) gradient descent, although I found your description a little confusing.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless, both the &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia article&lt;/a&gt; and Chapter 5 of the &lt;a href=&quot;http://www.deeplearningbook.org/contents/ml.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt; book provides good explanations of SGD. In many machine learning problems, the loss function, $L$ can be decomposed as the sum of losses for individual training examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;L(x, y; \theta) = \sum\limits_{i=1}^{m} L(x^{(i)}, y^{(i)}; \theta)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, in normal gradient descent, the number of terms in the gradient scales with the number of training examples, $m$. Computing this term can be very costly as $m$ grows large. In stochastic gradient descent, we instead &lt;em&gt;approximate&lt;/em&gt; the gradient by using a &quot;mini-batch&quot; of samples to compute it. So, instead of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\nabla_{\theta} L(x, y; \theta) = \sum\limits_{i=1}^{m} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;\nabla_{\theta} L(x, y; \theta) \approx \sum\limits_{i=1}^{m_{mini}} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $m_{mini} \ll m$. The key point is that, when running SGD, we randomly shuffle the training examples and iterate through &lt;em&gt;different&lt;/em&gt; minibatches during training. So, SGD is &lt;em&gt;not&lt;/em&gt; equivalent to normal (batch) gradient descent for a smaller training set. In SGD, we train on &lt;em&gt;all&lt;/em&gt; of the training examples, but it just takes us $m/m_{mini}$ iterations to get there. For instance, if $m_{mini} = 1$, and we took $m$ SGD steps, each step would correspond to using a single training example to approximate the gradient, and all $m$ steps would involve using a different training example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For question 2, solving for the parameters in matrix factorization is usually done via alternating least squares (ALS) or SGD. &lt;a href=&quot;http://blog.ethanrosenthal.com/2016/01/09/explicit-matrix-factorization-sgd-als/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's&lt;/a&gt; an excellent blog post explaining how both techniques work in the context of matrix factorization. The post is too long to summarize, but the author carefully and slowly derives all of the relevant equations from scratch, so I recommend you read it! &lt;/p&gt;&#xA;" OwnerUserId="27421" LastActivityDate="2016-12-30T14:54:59.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16006" PostTypeId="2" ParentId="15937" CreationDate="2016-12-30T15:03:37.367" Score="0" Body="&lt;p&gt;Build classifier using a deep learning algorithm called convolutional neural network (CNN) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classification using a machine learning algorithm has 2 phases:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training phase: In this phase, we train a machine learning algorithm using a dataset comprised of the images and their corresponding labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction phase: In this phase, we utilize the trained model to predict labels of unseen images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training phase for an image classification problem has 2 main steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature Extraction: In this phase, we utilize domain knowledge to extract new features that will be used by the machine learning algorithm. HoG and SIFT are examples of features used in image classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model Training: In this phase, we utilize a clean dataset composed of the images' features and the corresponding labels to train the machine learning model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the predicition phase, we apply the same feature extraction process to the new images and we pass the features to the trained machine learning algorithm to predict the label.&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2016-12-30T15:03:37.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16007" PostTypeId="2" ParentId="6355" CreationDate="2016-12-30T15:52:00.867" Score="3" Body="&lt;p&gt;I realize this question was asked more than a year ago, but I think one possibility is to use the bias-variance decomposition to calculate a lower bound on the error rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, the error rate is &lt;a href=&quot;https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff&quot; rel=&quot;nofollow noreferrer&quot;&gt;written as&lt;/a&gt; the sum of three terms, the bias, the variance, and the irreducible error. One good source for learning about these terms is &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow noreferrer&quot;&gt;An Introduction to Statistical Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume that the true function ($f(x)$) lies within the family of functions that our machine learning model is capable of fitting, and take the limit as the amount of training data we have goes to infinity. Then, if our machine learning model has a finite number of parameters, both the bias and the variance will be zero. So, the actual error will simply be equal to the irreducible error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, suppose our &lt;em&gt;true&lt;/em&gt; data is linear with Gaussian noise: $y \sim N(a + bx, \sigma^2)$. One of the the optimal estimators is obviously linear regression, $\hat{y} = \hat{a} + \hat{b}x$, and, as we add more training examples, the estimated coefficients $\hat{a}$ and $\hat{b}$ will approach $a$ and $b$, respectively. So, the best error (assuming squared loss) we could hope to achieve would be equal $\sigma^2$, the inherent error/irreducible noise associated with the data generation itself&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, computing the irreducible error is difficult (impossible?), since it requires knowledge of the true process for generating the data. But, this critique is also applicable to the Bayes error, since that requires knowledge of the true class probabilities.&lt;/p&gt;&#xA;" OwnerUserId="27421" LastActivityDate="2016-12-30T15:52:00.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16008" PostTypeId="1" AcceptedAnswerId="16033" CreationDate="2016-12-30T17:29:52.647" Score="7" ViewCount="18356" Body="&lt;p&gt;How to determine feature importance while using xgboost (&lt;code&gt;XGBclassifier&lt;/code&gt; or &lt;code&gt;XGBregressor&lt;/code&gt;) in pipeline? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AttributeError: 'Pipeline' object has no attribute 'get_fscore'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The answer provided &lt;a href=&quot;https://datascience.stackexchange.com/questions/6683/feature-selection-using-feature-importances-in-random-forests-with-scikit-learn&quot;&gt;here&lt;/a&gt; is similar but I couldn't get the idea.&lt;/p&gt;&#xA;" OwnerUserId="26019" LastEditorDisplayName="user380" LastEditDate="2017-05-30T14:50:26.120" LastActivityDate="2017-12-14T16:33:13.247" Title="How to determine feature importance while using xgboost in pipeline?" Tags="|python|scikit-learn|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16010" PostTypeId="2" ParentId="15974" CreationDate="2016-12-30T23:00:17.177" Score="3" Body="&lt;p&gt;It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Supervised&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;From your labeled data: create a training, validation and test set. Don't touch the test set until the very end.&lt;/li&gt;&#xA;&lt;li&gt;Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 output node (giving the probability of &quot;true&quot;).&lt;/li&gt;&#xA;&lt;li&gt;Try more stuff (e.g. &lt;a href=&quot;https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Try to combine classifiers in ensembles (see &lt;a href=&quot;https://martin-thoma.com/images/2015/12/ml-ensemble-learning.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;examples&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;A (naive) bayes classifier might be worth being investigated.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Semi-supervised&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You could train an auto-encoder with a bottleneck on the unlabeled data. Then remove everything after the bottleneck. Use this network as a preprocessing step. The idea is that this network finds a more meaningful abstraction of the relevant data. However, similar as PCA can have a very low projection error and still destroy the possibility to distinguish the classes.&lt;/li&gt;&#xA;&lt;li&gt;SVMs can also be used in a semi-supervised setting.&lt;/li&gt;&#xA;&lt;li&gt;You could try to find clusters in the unlabeled data + labeled data. Then you can get an a priori probability of &quot;true&quot; for each cluster (ignoring the unlabeled data). You could train very small models (overfitting!) on the data in the clusters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Usually, you can interpret features. This might help to develop new features.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2016-12-30T23:07:29.693" LastActivityDate="2016-12-30T23:07:29.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16011" PostTypeId="1" AcceptedAnswerId="16014" CreationDate="2016-12-31T01:29:25.177" Score="1" ViewCount="1189" Body="&lt;p&gt;A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time data to the millisecond. The processing also has to be efficient in view of the large data volume. What is the recommended methodology and tool to handle this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen others convert excel to CSV, and then confining their analysis to the CSV itself. Is this method advantageous in the sense that CSV is faster to read in, and can be processed by a larger toolset? Is there any powerful tool\library that can do batch conversion, and even extract custom formatting data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and Regards&lt;/p&gt;&#xA;" OwnerUserId="27506" LastActivityDate="2017-01-01T20:14:34.927" Title="What are recommended ways\tools for processing large data from Excel Files?" Tags="|data|preprocessing|excel|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16012" PostTypeId="1" CreationDate="2016-12-31T02:22:25.740" Score="2" ViewCount="1360" Body="&lt;p&gt;I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AZii0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AZii0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="29169" LastEditDate="2021-02-16T15:25:42.243" LastActivityDate="2021-02-16T15:25:42.243" Title="Test accuracy of neural net is going up and down" Tags="|neural-network|keras|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16013" PostTypeId="2" ParentId="16012" CreationDate="2016-12-31T15:16:30.637" Score="2" Body="&lt;p&gt;Based on the image you are sharing, the training accuracy continues to increase, the validation accuracy is changing around the 50%. I think either you do not have enough data to use neural network or the network is small to capture all the information, in both cases I feel there either under fitting or over fitting problem. Can you give us some information about the database so we can help more&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2016-12-31T15:16:30.637" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="16014" PostTypeId="2" ParentId="16011" CreationDate="2016-12-31T16:26:58.657" Score="2" Body="&lt;p&gt;pandas loads a csv upto 10x faster than excel. So, if you can, please convert these files to csv, which ever ones that are being loaded several times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been using cython, which seems to speedup read functions and processing. Please follow this &lt;a href=&quot;http://wesmckinney.com/blog/speeding-up-pandass-file-parsers-with-cython/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; for more details. It says, that using cython speeds up the processing 10x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestion: If you can, I would recommend to store data in database and use an orm like django/peewee for the data processing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please lemme know if you have any other queries.&lt;/p&gt;&#xA;" OwnerUserId="27426" LastActivityDate="2016-12-31T16:26:58.657" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16015" PostTypeId="2" ParentId="15891" CreationDate="2016-12-31T16:51:33.817" Score="5" Body="&lt;p&gt;That's an interesting problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This according to me is the most comprehensive way(if speed is not a problem. Or you can just pull all these words and create a dictionary/database of your own).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can this , &#xA;&lt;a href=&quot;https://wordsapiv1.p.mashape.com/words/love/antonyms&quot; rel=&quot;noreferrer&quot;&gt;https://wordsapiv1.p.mashape.com/words/love/antonyms&lt;/a&gt; (More info about this API at this &lt;a href=&quot;https://www.wordsapi.com/&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you can restrict results to antonyms with this api. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use requests to make API calls. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import requests&#xA;import simplejson&#xA;response = requests.get(url)&#xA;result = simplejson.loads(response)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then search for the antonyms from the result. If you are getting a big list of antonyms, use only the top n results for your searching. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although, W2V gives most commonly used words in the context of the keyword, it's hard to guess which is the antonym of the keyword.&lt;/p&gt;&#xA;" OwnerUserId="27426" LastActivityDate="2016-12-31T16:51:33.817" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16019" PostTypeId="2" ParentId="14793" CreationDate="2017-01-01T12:51:40.267" Score="2" Body="&lt;p&gt;The only way to obtain a high-quality dataset in your specific domain is to do it manually. There exists no other method that can give you the sentiment labels for texts in arbitrary domains. If there would exists such a method, why would you even bother to create your own model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should probably find/hire people that can do this work for you. Adding the sentiment meta-data to the texts could be as easy as creating an additional column in excel. But you could also smoothen the process by creating a little app that shows the sentence and let the user decide what sentiment it is (e.g. by swiping to left or right). Find a bunch of people that are willing to classify 10 sentences a day and your dataset will grow steadily. Even better, apply a little gamification and give people points or a rank and give an award to the best performing user to boost their performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that if these texts are really domain-specific you probably need domain-experts to do the classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When having enough training data you could learn a (simple) that predicts the output. Then you could let the human classifiers focus on the sentences with low confidence, assuming that high confidence automatic classifications are correct. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2017-01-01T12:51:40.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16020" PostTypeId="2" ParentId="16008" CreationDate="2017-01-01T13:14:20.327" Score="2" Body="&lt;h1&gt;Getting a reference to the xgboost object&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You should first get the &lt;code&gt;XGBClassifier&lt;/code&gt; or &lt;code&gt;XGBRegressor&lt;/code&gt; element from the pipeline. You could do this either by getting the n-th element or by specifying the name. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = XGBClassifier()&#xA;pipe = Pipeline([('other', other_element), ('xgboost', clf)])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get the &lt;code&gt;XGBClassifier&lt;/code&gt; you could either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use &lt;code&gt;clf&lt;/code&gt; if you still have a reference to it&lt;/li&gt;&#xA;&lt;li&gt;index the pipeline by name:  &lt;code&gt;pipe.named_steps['xgboost']&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;index the pipeline by location: &lt;code&gt;pipe.steps[1]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Getting the importance&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Secondly, it seems that importance is not implemented for the sklearn implementation of xgboost. See &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/757&quot; rel=&quot;nofollow noreferrer&quot;&gt;this github issue&lt;/a&gt;. A solution to add this to your &lt;code&gt;XGBClassifier&lt;/code&gt; or &lt;code&gt;XGBRegressor&lt;/code&gt; is also offered over their. It boils down to adding the methods to the class yourself.  &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2017-01-01T13:14:20.327" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16021" PostTypeId="1" AcceptedAnswerId="16106" CreationDate="2017-01-01T14:39:13.250" Score="0" ViewCount="83" Body="&lt;p&gt;I am using this MATLAB tutorial for Frequency-Domain Linear Regression. There is one part of code they provide, where it is necessary to determine the &quot;frequency bin of the positive and negative frequency&quot;, for fft and ifft:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;freqbin = 72/12;&#xA;freqbins = [freqbin 72-freqbin]+1;&#xA;tsfit = zeros(72,1);&#xA;tsfit(freqbins) = tsdft(freqbins);&#xA;tsfit = ifft(tsfit);&#xA;mu = mean(ts);&#xA;tsfit = mu+tsfit;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Length of time series is 72, and 12 months is considered as one cycle. How there can be only one frequency bin for positive frequency? How do we know that it is exactly 72/12 + 1 (+1 is because first bin is for zero frequency)? Is this some formula or what?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand what is exacty frequency bin, and why there is only one bin for positive frequency? I thought that we have many bins in time series.&lt;/p&gt;&#xA;" OwnerUserId="27460" LastActivityDate="2017-01-05T09:53:59.403" Title="Frequency bin of the positive and negative frequency" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16023" PostTypeId="1" CreationDate="2017-01-01T16:22:06.997" Score="-1" ViewCount="92" Body="&lt;p&gt;This is actually about scientific writing (and I'm not sure which Exchange site I should ask), a very simple one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some group-wise comparisons in data analysis (in general), and in some cases, it is impossible to change group assignment. For example, you can put someone in a real drug group or a placebo group. But you cannot put someone in a male group or a female group. Then in this case, if you find a significant group difference, does it sound okay for you to say &quot;increase&quot; or &quot;decrease&quot; in that statistic? I find it very strange and think it's wrong because the condition (or explanatory variable) cannot be changed, so I have been avoiding using 'increase/decrease' for group-wise statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But (obviously) I'm not a native English speaker nor a highly experienced reader. So I wonder how the native writers think, and why they keep using 'increase' for group differences.&lt;/p&gt;&#xA;" OwnerUserId="2956" LastActivityDate="2017-04-05T10:26:07.410" Title="&quot;Increase&quot; or &quot;decrease&quot; in a between-subject design" Tags="|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16024" PostTypeId="1" CreationDate="2017-01-01T16:34:15.503" Score="1" ViewCount="223" Body="&lt;p&gt;In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x)  \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\right] + \sigma_\epsilon^2 \\ = Bias(x)^2 + Variance(x) + Irreducible, $$&#xA;the first two terms depend on the argument $x$. So, in principle, if we choose a set of hyperparameters that yields the best tradeoff between the global bias and variance (somehow approximated, depending on our cross-validation strategy), these hyperparameters may lead to regions of $x$-es that have sub-optimal tradeoffs between their local biases and variances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In which scenarios may it be an actual problem? I mean: when are we better off training two classifiers on two separate sets of observations, with hyperparameters tweaked separately?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How could we identify such regions? I was expecting that a Boltzmann machine might be adapted for discovering highly volatile regions of $x$-es, but I didn't find anything of that sorts.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26141" LastEditorUserId="26141" LastEditDate="2017-01-02T07:17:03.240" LastActivityDate="2017-01-02T07:17:03.240" Title="Global vs. local bias-variance tradeoff" Tags="|unsupervised-learning|hyperparameter|theory|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16025" PostTypeId="2" ParentId="14314" CreationDate="2017-01-01T18:18:47.337" Score="4" Body="&lt;p&gt;For various metrics feel free to look at various benchmarking libraries including &lt;a href=&quot;http://www.mymedialite.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MyMediaLite&lt;/a&gt; and &lt;a href=&quot;http://www.librec.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LibRec&lt;/a&gt;. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is simple convert the ratings into binary likes and dislikes based on some threshold. Essentially you would take the &quot;likes&quot; of a user. Find the user is the testing set. Find the number of items that the user likes and see if the recommendation guesses the missing data correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to thinking about these datasets is that the fact that they chose to rate a movie at all means the user actually went out an saw the movie. The fact that they saw the movie indicates an implicit approval of the basic premise. As such, a good recommender should be able to predict which items that user rates in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to evaluate the recommendation system though is just to plug into an evaluation framework such as LibRec, MyMediaLite, or various other ones and compare the returned metrics to your algorithm. These libraries typically have code that will generate the testing, training, and evaluation subsets for you and all you have to do is hook up your code as a module.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a bonus, feel free to add your recommendation algorithm as a pull request to the library so other researchers can try it too and you'll have an excellent open source implementation of your algorithm that future researchers can reference in their own work.&lt;/p&gt;&#xA;" OwnerUserId="27533" LastActivityDate="2017-01-01T18:18:47.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16026" PostTypeId="2" ParentId="16011" CreationDate="2017-01-01T20:14:34.927" Score="1" Body="&lt;p&gt;If you want to stay in Excel format without converting to CSV first, have a look at the &lt;a href=&quot;https://github.com/hadley/readxl&quot; rel=&quot;nofollow noreferrer&quot;&gt;readxl&lt;/a&gt; package in the R programming environment. Since you have multiple files to read in and assuming they are in the same folder, I would use a for loop and iterate over all the file names in the folder. Then if all the files have the same variables you can bind them together into one data frame. Available memory may limit your ability to bind everything together in one frame though. You may also need to modify the read_excel statement depending on how your spreadsheets are structured. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;setwd(&quot;directory_path&quot;) # Set the working directory&#xA;&#xA;# List all the excel (xlsx) files in the directory&#xA;&#xA;files &amp;lt;- list.files(&quot;directory_path&quot;, full.names = TRUE, pattern=&quot;*.xlsx&quot;)&#xA;&#xA;# Create empty &quot;collector&quot; data frame&#xA;&#xA;df &amp;lt;- data.frame()&#xA;&#xA;# loop through files in subset list and read the xlsx file and bind to &quot;collector&quot; data frame&#xA;&#xA;for (i in 1:length(files)) {&#xA;  tempdf &amp;lt;- readxl::read_excel(files[i], sheet = 1, col_names = TRUE, col_types = NULL, na = &quot;&quot;, skip = 0)&#xA;  df &amp;lt;- rbind(df,tempdf)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20768" LastActivityDate="2017-01-01T20:14:34.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16027" PostTypeId="1" CreationDate="2017-01-02T09:15:31.383" Score="2" ViewCount="717" Body="&lt;p&gt;We have a function $f:R^N \rightarrow R$ and a set of points $D=\{ x\in R^N\}$. How is it possible to linearly lower the dimension of points to $M \ll N$ such that the fraction of explained variance* will be preserved as much as possible? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;we refer to the fraction of explained variance in prediction of output of $f$ using dataset $D$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27540" LastActivityDate="2017-08-31T18:21:14.583" Title="Preserving explained variance while reducing dimensionality" Tags="|dimensionality-reduction|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16028" PostTypeId="2" ParentId="9216" CreationDate="2017-01-02T10:40:13.563" Score="1" Body="&lt;p&gt;Yes, you can do this without reworking your, and as always, there are few ways to do it. Without seeing exact data, I would try setting table calculation to percent of total and then alter compute with to Table Down / Table Across depending on the data layout&lt;/p&gt;&#xA;" OwnerUserId="27544" LastActivityDate="2017-01-02T10:40:13.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16030" PostTypeId="2" ParentId="13107" CreationDate="2017-01-02T12:44:23.630" Score="0" Body="&lt;p&gt;You are right in last paragraph. You can use tensorflow for this purpose. Tensorflow supports having output as vector of classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The introductory tutorial of tensorflow for MNIST dataset uses output as vector of dimension 10 (one for each class 0..9)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/tutorials/mnist/beginners/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/mnist/beginners/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26415" LastActivityDate="2017-01-02T12:44:23.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16033" PostTypeId="2" ParentId="16008" CreationDate="2017-01-02T19:12:40.707" Score="6" Body="&lt;p&gt;As I found, there are two ways to determine feature importance:&#xA;First: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(grid_search.best_estimator_.named_steps[&quot;clf&quot;].feature_importances_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.14582562  0.08367272  0.06409663  0.07631433  0.08705109  0.03827286&#xA;  0.0592836   0.05025916  0.07076083  0.0699278   0.04993521  0.07756387&#xA;  0.05095335  0.07608293]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Second:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(grid_search.best_estimator_.named_steps[&quot;clf&quot;].booster().get_fscore())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'f2': 1385, 'f11': 1676, 'f12': 1101, 'f6': 1281, 'f9': 1511, 'f7': 1086, 'f5': 827, 'f0': 3151, 'f10': 1079, 'f1': 1808, 'f3': 1649, 'f13': 1644, 'f8': 1529, 'f4': 1881}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Third:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(grid_search.best_estimator_.named_steps[&quot;clf&quot;].get_booster().get_fscore())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2017-12-14T16:33:13.247" LastActivityDate="2017-12-14T16:33:13.247" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16034" PostTypeId="1" AcceptedAnswerId="16035" CreationDate="2017-01-02T19:45:40.040" Score="8" ViewCount="8804" Body="&lt;p&gt;I'm trying &lt;a href=&quot;http://mlpy.sourceforge.net/docs/3.4/dtw.html&quot; rel=&quot;noreferrer&quot;&gt;DTW from mlpy&lt;/a&gt;, to check similarity between time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I normalize the series before processing them with DTW? Or is it somewhat tolerant and I can use the series as they are?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All time series stored in a Pandas Dataframe, each in one column. Size is less than 10k points.&lt;/p&gt;&#xA;" OwnerUserId="27194" LastEditorUserId="27194" LastEditDate="2017-01-03T10:04:56.393" LastActivityDate="2017-01-03T10:04:56.393" Title="DTW (Dynamic Time Warping) requires prior normalization?" Tags="|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16035" PostTypeId="2" ParentId="16034" CreationDate="2017-01-02T20:07:30.330" Score="5" Body="&lt;p&gt;DTW often uses a distance between symbols, e.g. a Manhattan distance $(d(x, y) = {\displaystyle |x-y|} $). Whether symbols are samples or features, they might require amplitude (or at least) normalization. Should they? I wish I could answer such a question in all cases. However, you can find some hints in:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/55083/dynamic-time-warping-and-normalization&quot;&gt;Dynamic Time Warping and normalization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Section 1.2.1 of &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/SIGKDD_trillion.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://luscinia.sourceforge.net/page26/page14/page14.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Normalization for Dynamic Time Warping&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-02T20:07:30.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16036" PostTypeId="1" CreationDate="2017-01-02T20:41:13.493" Score="15" ViewCount="13816" Body="&lt;p&gt;I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At the moment, the code works as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Process the text of each job listing to extract skills that are mentioned in the listing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) For each career (e.g. &quot;Data Analyst&quot;), combine the processed text of the job listings for that career into one document&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Calculate the TF-IDF of each skill within the career documents&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this, I'm not sure which method I should use to rank careers based on a list of a user's skills. The most popular method that I've seen would be to treat the user's skills as a document as well, then to calculate the TF-IDF for the skill document, and use something like cosine similarity to calculate the similarity between the skill document and each career document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't seem like the ideal solution to me, since cosine similarity is best used when comparing two documents of the same format. For that matter, TF-IDF doesn't seem like the appropriate metric to apply to the user's skill list at all. For instance, if a user adds additional skills to their list, the TF for each skill will drop. In reality, I don't care what the frequency of the skills are in the user's skills list -- I just care that they have those skills (and maybe how well they know those skills).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like a better metric would be to do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) For each skill that the user has, calculate the TF-IDF of that skill in the career documents&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) For each career, sum the TF-IDF results for all of the user's skill&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Rank career based on the above sum&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I thinking along the right lines here? If so, are there any algorithms that work along these lines, but are more sophisticated than a simple sum? Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="27559" LastActivityDate="2019-02-28T06:23:59.853" Title="Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats" Tags="|nlp|text-mining|similarity|cosine-distance|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16037" PostTypeId="2" ParentId="15923" CreationDate="2017-01-02T22:41:58.653" Score="1" Body="&lt;p&gt;OK, we have two dozen predictions-with-uncertainty, and actual point values. I'm still not sure we know enough to answer, so I'm going to just outline some points and hope someone can improve upon this. A quick search found three similar StackExchange questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/159790/combining-confidence-intervals-from-several-regression-point-estimates&quot; title=&quot;13 independent regressions&quot;&gt;Combining 13 independent regressions&lt;/a&gt; seems the most similar -- Note comment warning this might be meaningless. The best-answer suggests a sampling approach because the result has no closed form. Using a Normal approximation would have a closed form, but I think your plot suggests the errors are not Normal, or your 95% CI is much too narrow.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/3652/combining-two-confidence-intervals-point-estimates&quot; title=&quot;Combining 2&quot;&gt;Combining 2&lt;/a&gt; -- See esp. the reply discussing &lt;strong&gt;meta-analysis&lt;/strong&gt; and whether you have a &lt;em&gt;fixed effects&lt;/em&gt; or a &lt;em&gt;random effects&lt;/em&gt; model. This may be what you are doing, in which case re-formulating the question for that community may be best.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/95331/combining-confidence-intervals-for-a-particular-equation&quot; title=&quot;Delta Method&quot;&gt;Delta Method&lt;/a&gt; -- Seems to me the same question, but takes a direct maximization approach, assuming approximations hold.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A couple of observations suggesting any simple average is not going to give you what you want, and you need to be very clear what is being combined, and what question the result is going to answer:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If these are 95% Confidence Intervals, then your error model is much too conservative. Nearly all of them should include the true value (gray dashed diagonal) but only about 5 do. Therefore any parametric combination will be at least as unreliable. But suppose you have a black box with uncalibrated estimates, and you just want to know the &quot;average&quot; from that black box, however uncalibrated.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If these are two-dozen unrelated &lt;em&gt;estimates&lt;/em&gt; from a crowdsourcing platform, the &quot;average&quot; is not meaningful. On the other hand, if they are performance measures of the crowd, then the average is something like the average performance of the crowd across all question types. In that case:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;1st approximation would just average the two-dozen points and use the standard deviation of that. (Each point can be weighted.) &lt;/li&gt;&#xA;&lt;li&gt;2nd approximation would treat each estimate and CI as Normal, and use one of the formulas for combining Normal CIs. Note however that it seems error bars are wider towards the edges. That may require special handling.&lt;/li&gt;&#xA;&lt;li&gt;3rd approximation: sample say 100 forecasts from the two-dozen &lt;em&gt;distributions&lt;/em&gt; and then combine those ~2400 samples. (Weighting can be done by varying sample size for each forecast.) Works even if the distributions are not assumed to be Normal.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If instead you have two-dozen forecasts from a single statistical model on different input data, and you want to know the average forecast across all input combinations, don't combine them. Just run the model with no factors specified, and use that result. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hopefully a &lt;em&gt;bona fide&lt;/em&gt; statistician can supply a more definitive answer.&lt;/p&gt;&#xA;" OwnerUserId="12417" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-02T22:41:58.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16039" PostTypeId="2" ParentId="16034" CreationDate="2017-01-03T05:32:45.893" Score="13" Body="&lt;p&gt;I am glad you asked ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In 99% of cases, you must z-normalize.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Want to know why? I wrote a tutorial on this, page 46&#xA;&lt;a href=&quot;http://www.cs.unm.edu/~mueen/DTW.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.unm.edu/~mueen/DTW.pdf&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="27567" LastActivityDate="2017-01-03T05:32:45.893" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16040" PostTypeId="1" AcceptedAnswerId="16057" CreationDate="2017-01-03T06:54:01.460" Score="1" ViewCount="1768" Body="&lt;p&gt;I am trying to use an LSTM to predict &lt;strong&gt;daily usage for users&lt;/strong&gt;. I have data for (say) 90 days of usage for a large number of users. Based on business knowledge (and initial analysis) we know users fall roughly into different categories. E.g. daily users would have a non-zero usage almost every day, weekly users would have one or two days of non-zero usage every 7 days and monthly users would have a couple of days with non-zero usage per 30 days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample data where each column is one day starting from October 1st and each row is data for one user. (The usage 'cycle' of each user might start on any day).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;User 1: 10,  8, 10,  9, 0, 0, 11, ...&#xA;User 2:  0,  0,  0, 20, 0, 0,  0,  0,  0, 18,  0,  0,  0, ...&#xA;User 3: 40,  0,  0,  0, .....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where User 1 might be a &quot;daily&quot; user, User 2 is a &quot;weekly&quot; user and User 3 is a monthly user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first question is that can a single LSTM/deep learning model capture these different types of patterns?&#xA;The goal is to predict the daily usage (next couple of days based on past 90 days) for individual users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am using a really simple LSTM (in Keras):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(LSTM(1, input_shape=(90, 1), unroll=True))&#xA;model.compile(optimizer='rmsprop', loss='mse')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To help the model 'capture' the fact that different users might have different levels of usage I added the average usage (of non-zero values) for each user as the first data point for each row. The remaining 90 data points for each user remain as shown in the table above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second question is if I really need to add the average of the values to 'help' the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that even after 100 epochs the error remains unchanged. &#xA;And finally what can I change to make this to work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="27566" LastEditorUserId="27566" LastEditDate="2017-01-03T23:56:09.297" LastActivityDate="2017-01-04T18:30:34.633" Title="LSTM for capturing multiple patterns" Tags="|time-series|keras|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16041" PostTypeId="2" ParentId="16027" CreationDate="2017-01-03T07:51:32.850" Score="1" Body="&lt;p&gt;You could try to apply Principal Component Analysis, a form or ordination in which you linearly transform the vector space to maximize variance over the new dimensions. Consequently you'll end up with a new ordering in which more variance is contained in fewer dimensions, enabling you to consider less dimensions in your analyses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most, if not all, statistical software packages offer PCA as a component. &lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-03T07:51:32.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16042" PostTypeId="1" CreationDate="2017-01-03T08:17:43.560" Score="1" ViewCount="35" Body="&lt;p&gt;I recently ran into a problem that seems to have considerate local effects, in space and time in fact: Two objects that are close are likely yo be in a similar state (if they are in the same year). I'm interested in modelling the behavior over time, but many classifiers find rules that are based on locality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have taken some action to prevent that of course, such as downsampling the set to achieve maximum variance over x and y and testing on a set that has been selected on a year that is not in the train set. I do however suspect that some techniques (tree learners) could benefit from  a prune set that has been separated on the same grounds as the test set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now most algorithms separate prune sets on initialisation themselves, but I want to experiment with a user defined set. In the current setup, rules that have been generated on local effects, would contribute less to performance in a test run, but the prune routine should be less equipped to predict that behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody have experience with this? Do you know how to practically do that? Or do you know of a good theoretical resource on the subject?&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-03T08:17:43.560" Title="How to use a predfined separate prune set in Weka (alternatives in R or Python also acceptable)" Tags="|python|r|decision-trees|weka|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16043" PostTypeId="1" CreationDate="2017-01-03T09:07:19.657" Score="1" ViewCount="268" Body="&lt;p&gt;I am using the rpart function (in the &quot;rpart&quot; package) in R to fit a cost-sensitive classification tree. I have read the corresponding vignette, which says that misclassication costs are incorporated via altered priors (section 3.2.2 on page 8). However, I am not sure that I understand exactly how these altered probabilities affect the splitting criteria/impurity index. Does anyone know? &lt;/p&gt;&#xA;" OwnerUserId="23195" LastActivityDate="2017-01-05T23:48:30.020" Title="Altered priors for classification trees" Tags="|classification|decision-trees|cost-function|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16044" PostTypeId="2" ParentId="16036" CreationDate="2017-01-03T10:05:45.263" Score="0" Body="&lt;p&gt;Use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jaccard Index&lt;/a&gt;. This will very much serve your purpose. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="31095" LastEditDate="2018-07-04T15:42:59.993" LastActivityDate="2018-07-04T15:42:59.993" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16045" PostTypeId="1" CreationDate="2017-01-03T11:50:09.250" Score="12" ViewCount="10721" Body="&lt;p&gt;Dropout (&lt;a href=&quot;https://arxiv.org/abs/1207.0580&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;, &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot; rel=&quot;noreferrer&quot;&gt;explanation&lt;/a&gt;) sets the output of some neurons to zero. So for a MLP, you could have the following architecture for the &lt;a href=&quot;https://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;noreferrer&quot;&gt;Iris flower dataset&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;4 : 50 (tanh) : dropout (0.5) : 20 (tanh) : 3 (softmax)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would work like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$softmax(W_3 \cdot \tanh(W_2 \cdot \text{mask}(D, \tanh(W_1 \cdot input\_vector)))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $input\_vector \in \mathbb{R}^{4 \times 1}$, $W_1 \in \mathbb{R}^{50 \times 4}$, $D \in \{0, 1\}^{50 \times 1}$, $W_2 \in \mathbb{R}^{20 \times 50}$, $W_3 \in \mathbb{R}^{20 \times 3}$ (ignoring biases for the sake of simplictiy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $D = (d)_{ij}$ and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d_{ij} \sim B(1, p=0.5)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the $\text{mask}(D, M)$ operation multiplies $D$ point-wise with $M$ (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Hadamard_product_(matrices)&quot; rel=&quot;noreferrer&quot;&gt;Hadamard product&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence we just sample the matrix $D$ each time and thus the dropout becomes a multiplication of a node with 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for CNNs, it is not clear to me what exactly is dropped out. I can see three possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dropping complete feature maps (hence a kernel)&lt;/li&gt;&#xA;&lt;li&gt;Dropping one element of a kernel (replacing an element of a kernel by 0)&lt;/li&gt;&#xA;&lt;li&gt;Dropping one element of a feature map&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please add a reference / quote to your answer.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I think Lasagne does (3) (see &lt;a href=&quot;https://github.com/Lasagne/Lasagne/blob/master/lasagne/layers/noise.py#L20-L106&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;). This might be the simplest to implement. However, closer to the original idea might be (1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems to be similar for Caffe (see &lt;a href=&quot;https://github.com/BVLC/caffe/blob/master/src/caffe/layers/dropout_layer.cu&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;). For tensorflow, the user has to decide (&lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/layers/core.py#L215-L293&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt; - I'm not sure what happens when &lt;code&gt;noise_shape=None&lt;/code&gt; is passed).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;How it should be&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;(2) and (3) don't make much sense as it would cause the network to add invariance to spacial positions, which is probably not desired. Hence (1) is the only variant which makes sense. But I'm not sure what happens if you use the default implementation.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-01-14T12:47:49.450" LastActivityDate="2020-11-24T18:55:47.140" Title="How exactly does DropOut work with convolutional layers?" Tags="|dropout|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16048" PostTypeId="1" AcceptedAnswerId="16052" CreationDate="2017-01-03T13:23:35.277" Score="2" ViewCount="752" Body="&lt;p&gt;I'm working on a project and we are using XGBoost to make predictions. My colleague sent me the model file but when I load on my computer it don't run as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I changed one variable from the model from 0 to 1 it didn't changed the result (in 200 different lines), so I started to investigate. We compared a lot of different results and it was all different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run the &lt;code&gt;xgb_tree&lt;/code&gt; it showed the &lt;code&gt;max_depth&lt;/code&gt; is 0, but it is supposed to be 4. When I run &lt;code&gt;xgb_tree$results&lt;/code&gt; it says the &lt;code&gt;max_depth = 4&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also tried a lot of different save methods (.rda, .rds, .model) but none of them worked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion would be welcome, thanks.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT: Posting the sessionInfo()&lt;/p&gt;&#xA;&#xA;&lt;p&gt;His:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R version 3.2.5 (2016-04-14)&#xA;Platform: x86_64-w64-mingw32/x64 (64-bit)&#xA;Running under: Windows &amp;gt;= 8 x64 (build 9200)&#xA;&#xA;locale:&#xA;[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   &#xA;[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          &#xA;[5] LC_TIME=English_United States.1252    &#xA;&#xA;attached base packages:&#xA;[1] stats     graphics  grDevices utils     datasets  methods   base     &#xA;&#xA;other attached packages:&#xA; [1] plyr_1.8.4            car_2.1-4             acepack_1.4.1         Ckmeans.1d.dp_3.4.6-4&#xA; [5] Hmisc_4.0-0           Formula_1.2-1         survival_2.40-1       memisc_0.99.7-1      &#xA; [9] MASS_7.3-45           Information_0.0.9     minerva_1.4.5         randomForest_4.6-12  &#xA;[13] pROC_1.8              xgboost_0.4-4         caret_6.0-73          lattice_0.20-33      &#xA;[17] ggplot2_2.2.0         miscTools_0.6-22      reshape_0.8.6         data.table_1.9.8     &#xA;[21] dplyr_0.5.0           e1071_1.6-7           lubridate_1.6.0       psych_1.6.9          &#xA;[25] readr_1.0.0           stringr_1.1.0         stringi_1.1.2        &#xA;&#xA;loaded via a namespace (and not attached):&#xA; [1] Rcpp_0.12.8         class_7.3-14        assertthat_0.1      digest_0.6.10      &#xA; [5] foreach_1.4.3       R6_2.2.0            MatrixModels_0.4-1  stats4_3.2.5       &#xA; [9] lazyeval_0.2.0      minqa_1.2.4         SparseM_1.74        nloptr_1.0.4       &#xA;[13] rpart_4.1-10        Matrix_1.2-4        labeling_0.3        splines_3.2.5      &#xA;[17] lme4_1.1-12         foreign_0.8-66      munsell_0.4.3       compiler_3.2.5     &#xA;[21] mnormt_1.5-5        mgcv_1.8-12         htmltools_0.3.5     nnet_7.3-12        &#xA;[25] tibble_1.2          gridExtra_2.2.1     htmlTable_1.7       codetools_0.2-14   &#xA;[29] ModelMetrics_1.1.0  grid_3.2.5          nlme_3.1-125        gtable_0.2.0       &#xA;[33] DBI_0.5-1           magrittr_1.5        scales_0.4.1        reshape2_1.4.2     &#xA;[37] doParallel_1.0.10   latticeExtra_0.6-28 RColorBrewer_1.1-2  iterators_1.0.8    &#xA;[41] tools_3.2.5         parallel_3.2.5      pbkrtest_0.4-6      colorspace_1.3-1   &#xA;[45] cluster_2.0.3       knitr_1.15.1        quantreg_5.29&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Mine:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R version 3.3.1 (2016-06-21)&#xA;Platform: x86_64-w64-mingw32/x64 (64-bit)&#xA;Running under: Windows &amp;gt;= 8 x64 (build 9200)&#xA;&#xA;locale:&#xA;[1] LC_COLLATE=Portuguese_Brazil.1252  LC_CTYPE=Portuguese_Brazil.1252    LC_MONETARY=Portuguese_Brazil.1252&#xA;[4] LC_NUMERIC=C                       LC_TIME=Portuguese_Brazil.1252    &#xA;&#xA;attached base packages:&#xA;[1] stats     graphics  grDevices utils     datasets  methods   base     &#xA;&#xA;other attached packages:&#xA;[1] caret_6.0-73         ggplot2_2.1.0        lattice_0.20-33      plyr_1.8.4           xgboost_0.4-4       &#xA;[6] shinydashboard_0.5.3 shiny_0.14.2        &#xA;&#xA;loaded via a namespace (and not attached):&#xA; [1] Rcpp_0.12.8        nloptr_1.0.4       iterators_1.0.8    tools_3.3.1        digest_0.6.10      lme4_1.1-12       &#xA; [7] jsonlite_1.1       nlme_3.1-128       gtable_0.2.0       mgcv_1.8-12        Matrix_1.2-6       foreach_1.4.3     &#xA;[13] parallel_3.3.1     SparseM_1.72       stringr_1.1.0      MatrixModels_0.4-1 stats4_3.3.1       grid_3.3.1        &#xA;[19] nnet_7.3-12        data.table_1.9.6   R6_2.1.3           minqa_1.2.4        reshape2_1.4.2     car_2.1-3         &#xA;[25] magrittr_1.5       scales_0.4.0       codetools_0.2-14   ModelMetrics_1.1.0 htmltools_0.3.5    MASS_7.3-45       &#xA;[31] splines_3.3.1      rsconnect_0.4.3    pbkrtest_0.4-6     mime_0.5           xtable_1.8-2       colorspace_1.2-6  &#xA;[37] httpuv_1.3.3       quantreg_5.29      stringi_1.1.1      munsell_0.4.3      chron_2.3-47      &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21044" LastEditorUserId="21044" LastEditDate="2017-01-03T16:32:29.150" LastActivityDate="2017-01-03T16:39:25.683" Title="Problem when loading a XGBoost model in a different computer" Tags="|machine-learning|r|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16049" PostTypeId="1" CreationDate="2017-01-03T14:28:56.730" Score="0" ViewCount="1010" Body="&lt;p&gt;I have a time versus current data for a days work which is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;11/15/2016  5:12:27 10:42:27    26.61&#xA;11/15/2016  5:12:28 10:42:28    42.27&#xA;11/15/2016  5:12:29 10:42:29    25.48&#xA;11/15/2016  5:12:30 10:42:30    24.24&#xA;11/15/2016  5:12:31 10:42:31    25.91&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first column being the date&#xA;The second column being the time in GMT&#xA;The third column being time in IST&#xA;The fourth column being the value of the current used by the machine&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest me an algorithm for finding the anomalies in the pattern plotted by current versus time in IST?&#xA;Any help regarding my approach will be appreciated too.&lt;/p&gt;&#xA;" OwnerUserId="27577" LastEditorUserId="471" LastEditDate="2017-01-03T14:40:12.017" LastActivityDate="2017-01-03T15:04:25.103" Title="Anomaly detection on a time-series data in a CSV format using python" Tags="|machine-learning|python|anomaly-detection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16051" PostTypeId="2" ParentId="16049" CreationDate="2017-01-03T15:04:25.103" Score="4" Body="&lt;p&gt;With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. Visually inspect the data for outliers. People are pretty good at this. I would then try to describe a mathematical model to fit the data with the outliers removed. Finally you can identify the outliers as those points greater than some statistical measure away from the difference of the model and the data. &lt;/p&gt;&#xA;" OwnerUserId="27581" LastActivityDate="2017-01-03T15:04:25.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16052" PostTypeId="2" ParentId="16048" CreationDate="2017-01-03T16:39:25.683" Score="0" Body="&lt;p&gt;The error was due to the R version. My colleague was running a 3.2.5 and I was running the 3.3.1, and thanks to &lt;a href=&quot;https://datascience.stackexchange.com/users/12384/stereo&quot;&gt;Stereo&lt;/a&gt; I could notice this and test on the same versions.&lt;/p&gt;&#xA;" OwnerUserId="21044" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-01-03T16:39:25.683" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16055" PostTypeId="1" CreationDate="2017-01-03T23:18:05.600" Score="3" ViewCount="6709" Body="&lt;p&gt;While thinking about similarity between two time series, one can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;DTW&lt;/a&gt; to approach the issue. There is a Python package for that &lt;a href=&quot;http://mlpy.sourceforge.net/docs/3.4/dtw.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;mlpy&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also &lt;a href=&quot;https://datascience.stackexchange.com/questions/15812/check-similarity-between-time-series&quot;&gt;said&lt;/a&gt; to compare time series via simple euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a Python package to this?&lt;/p&gt;&#xA;" OwnerUserId="27194" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-03-05T00:23:03.607" Title="Euclidean distance between time series in Python" Tags="|time-series|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16056" PostTypeId="2" ParentId="16055" CreationDate="2017-01-03T23:48:02.987" Score="6" Body="&lt;p&gt;This only makes sense if you look at the time series as vectors with same length. In that case, there is no need for a package. Just use, for example, &#xA;&lt;code&gt;numpy.linalg.norm(a-b)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-01-03T23:48:02.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16057" PostTypeId="2" ParentId="16040" CreationDate="2017-01-04T01:38:15.390" Score="1" Body="&lt;p&gt;Modeling is a process where you invent hypotheses and then try to reject them. Without seeing your data, there is no guarantee any of these would go far. But here are some thoughts.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;s&gt;90 days of daily observations seem too few, which won't lead to much confidence no matter what model you use.&lt;/s&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;1.1 If you treat each user as one sequential observation, yes your dataset is sufficient for models like RNN. But ask yourself the question, are there users out there who are unique and different and you don't know about how they differ from the rest.  &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;You seem to start with a belief that there are three types of users. I would suggest don't. Let the data tells you how many types are there. And for the purpose of predicting future usage, you don't care about this type information. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recurrent neural net is a good choice if you believe the user behaviors are temporally dependent individually and have cross-sectional dependence. Otherwise, a time series model like ARIMA may do a better job and require less resources to fit.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Lastly, &lt;em&gt;if&lt;/em&gt; you want to normalize the data, you should do so by subtracting each row by the row mean and divide by the row standard deviation. By adding the mean as the first observation, you're going to confuse the neural net and it's not smart enough to treat this data point differently. Unless there is a huge variance across users, I would start without normalization, fit a model, figure out some sort of out-of-sample MSE, then repeat on normalized data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;EDIT : There is no reason to train multiple RNNs on this set of data if you do not have a good segmentation argument.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="27250" LastEditDate="2017-01-04T18:30:34.633" LastActivityDate="2017-01-04T18:30:34.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16059" PostTypeId="2" ParentId="15864" CreationDate="2017-01-04T08:34:41.397" Score="3" Body="&lt;p&gt;You can follow this thread here: &lt;a href=&quot;https://github.com/fchollet/keras/pull/3004&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/pull/3004&lt;/a&gt; Seems at least that the layer-wise tuning should come.&lt;/p&gt;&#xA;" OwnerUserId="27551" LastActivityDate="2017-01-04T08:34:41.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16060" PostTypeId="1" AcceptedAnswerId="16084" CreationDate="2017-01-04T08:41:15.700" Score="83" ViewCount="48140" Body="&lt;p&gt;I'm having trouble understanding the difference between &lt;em&gt;equivariant to translation&lt;/em&gt; and &lt;em&gt;invariant to translation&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the book &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;noreferrer&quot;&gt;Deep Learning&lt;/a&gt;. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio), one can find on the convolutional networks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[...] the particular form of parameter sharing causes the layer to have a property called &lt;em&gt;equivariance&lt;/em&gt; to translation &lt;/li&gt;&#xA;&lt;li&gt;[...] pooling helps to make the representation become approximately &lt;em&gt;invariant&lt;/em&gt; to small translations of the input&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there any difference between them or are the terms interchangeably used?&lt;/p&gt;&#xA;" OwnerUserId="27607" LastEditorDisplayName="user10640" LastEditDate="2019-03-04T05:28:32.837" LastActivityDate="2022-11-17T17:04:30.790" Title="What is the difference between &quot;equivariant to translation&quot; and &quot;invariant to translation&quot;" Tags="|neural-network|deep-learning|convolution|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16061" PostTypeId="2" ParentId="11747" CreationDate="2017-01-04T08:43:01.157" Score="19" Body="&lt;p&gt;From the Keras documentation, you can load the data into Train and Test sets like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As for cross validation, you could follow this example from &lt;a href=&quot;https://github.com/fchollet/keras/issues/1711&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import StratifiedKFold&#xA;&#xA;def load_data():&#xA;    # load your data using this function&#xA;&#xA;def create model():&#xA;    # create your model using this function&#xA;&#xA;def train_and_evaluate__model(model, data_train, labels_train, data_test, labels_test):&#xA;    model.fit...&#xA;    # fit and evaluate here.&#xA;&#xA;if __name__ == &amp;quot;__main__&amp;quot;:&#xA;    n_folds = 10&#xA;    data, labels, header_info = load_data()&#xA;    skf = StratifiedKFold(labels, n_folds=n_folds, shuffle=True)&#xA;&#xA;    for i, (train, test) in enumerate(skf):&#xA;        print &amp;quot;Running Fold&amp;quot;, i+1, &amp;quot;/&amp;quot;, n_folds&#xA;        model = None # Clearing the NN.&#xA;        model = create_model()&#xA;        train_and_evaluate_model(model, data[train], labels[train], data[test], labels[test])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27551" LastEditorUserId="85045" LastEditDate="2020-12-28T08:00:19.283" LastActivityDate="2020-12-28T08:00:19.283" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16062" PostTypeId="1" AcceptedAnswerId="16064" CreationDate="2017-01-04T08:46:42.270" Score="15" ViewCount="16142" Body="&lt;p&gt;I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a binary variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to run feature selection before the model fitting? Does it affect the model performance significantly or is there not much difference if I directly fit the model using all predictor variables?  &lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="29169" LastEditDate="2019-06-09T16:37:56.507" LastActivityDate="2019-06-09T16:37:56.507" Title="Is feature selection necessary?" Tags="|machine-learning|predictive-modeling|feature-selection|random-forest|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="16064" PostTypeId="2" ParentId="16062" CreationDate="2017-01-04T12:03:04.640" Score="12" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;noreferrer&quot;&gt;Feature selection&lt;/a&gt; might be consider a stage to avoid.&#xA;You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are not optimal since the problem is &lt;a href=&quot;https://en.wikipedia.org/wiki/NP-completeness&quot; rel=&quot;noreferrer&quot;&gt;NP-Complete&lt;/a&gt;.&#xA;Using it doesn't sound like an offer that you cannot refuse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what are the benefits of using it?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Many features and low samples/features ratio will introduce noise into your dataset. In such a case your classification algorithm are likely to overfit, and give you a false feeling of good performance.&lt;/li&gt;&#xA;&lt;li&gt;Reducing the number of features will reduce the running time in the later stages. That in turn will enable you using algorithms of higher complexity, search for more hyper parameters or do more evaluations.&lt;/li&gt;&#xA;&lt;li&gt;A smaller set of feature is more comprehendible to humans. That will enable you to focus on the main sources of predictability and do more exact feature engineering. If you will have to explain your model to a client, you are better presenting a model with 5 features than a model with 200 features. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now for your specific case:&#xA;I recommend that you'll begin in computing the correlations among the features and the concept. Computing correlations among all features is also informative.&#xA;Note that there are many types of useful correlations (e.g., &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient&quot; rel=&quot;noreferrer&quot;&gt;Pearson&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;noreferrer&quot;&gt;Mutual information&lt;/a&gt;) and many attributes that might effect them (e.g., sparseness, concept imbalance). Examining them instead of blindly go with a feature selection algorithm might save you plenty of time in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think that you will have a lot of running time problems with your dataset. However, your samples/features ratio isn't too high so you might benefit from feature selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose a classifier of low complexity(e.g., linear regression, a small decision tree) and use it as a benchmark. Try it on the full data set and on some dataset with a subset of the features. Such a benchmark will guid you in the use of feature selection. You will need such guidance since there are many options (e.g., the number of features to select, the feature selection algorithm) an since the goal is usually the predication and not the feature selection so feedback is at least one step away.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-01-04T12:03:04.640" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16065" PostTypeId="2" ParentId="989" CreationDate="2017-01-04T12:17:19.627" Score="0" Body="&lt;p&gt;You need to scale your data. Scaling will normalize your data points to -1 to 1 range, which will help in faster convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try using following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# X is your numpy data array.&#xA;&#xA;from sklearn import preprocessing&#xA;&#xA;X = preprocessing.scale(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27613" LastEditorUserId="27613" LastEditDate="2018-09-06T12:19:47.263" LastActivityDate="2018-09-06T12:19:47.263" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16066" PostTypeId="1" AcceptedAnswerId="16067" CreationDate="2017-01-04T13:15:42.680" Score="9" ViewCount="752" Body="&lt;p&gt;I'm a B.Sc graduate. One of my courses was 'Introduction to Machine Learning', and I always wanted to do a personal project in this subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recently heard about different AI training to play games such as Mario, Go, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What knowledge do I need to acquire in order to train a simple AI program to play a game? And what game do you recommend for a beginner?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I know in Machine Learning so far -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Introduction to the course and to machine learning. K-Nearest Neighbor algorithm, and K-means algorithm&lt;/li&gt;&#xA;&lt;li&gt;Statistical Inference &lt;/li&gt;&#xA;&lt;li&gt;Gaussian Mixture Model (GMM) and Expectation Maximization (EM)    &lt;/li&gt;&#xA;&lt;li&gt;Probably Approximately Correct (PAC) model, including generalization bounds and model selection   &lt;/li&gt;&#xA;&lt;li&gt;Basic hyperplane algorithms: Perceptron and Winnow.   &lt;/li&gt;&#xA;&lt;li&gt;Support Vector Machines (SVM) &lt;/li&gt;&#xA;&lt;li&gt;Kernels&lt;/li&gt;&#xA;&lt;li&gt;Boosting weak learners to strong learners: AdaBoost   &lt;/li&gt;&#xA;&lt;li&gt;Margin-Perceptron &lt;/li&gt;&#xA;&lt;li&gt;Regression&lt;/li&gt;&#xA;&lt;li&gt;PCA&lt;/li&gt;&#xA;&lt;li&gt;Decision Trees    &lt;/li&gt;&#xA;&lt;li&gt;Decision Trees pruning and random forests     &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27614" LastEditorUserId="836" LastEditDate="2017-01-04T13:48:31.867" LastActivityDate="2018-03-19T07:59:29.710" Title="What knowledge do I need in order to write a simple AI program to play a game?" Tags="|machine-learning|algorithms|beginner|reinforcement-learning|training|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16067" PostTypeId="2" ParentId="16066" CreationDate="2017-01-04T14:17:56.523" Score="8" Body="&lt;p&gt;There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. For instance, &lt;a href=&quot;https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)&quot; rel=&quot;nofollow noreferrer&quot;&gt;IBM's Deep Blue&lt;/a&gt; was essentially a fast heuristic-driven search for optimal moves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, probably the most generic machine learning algorithm for training an agent to perform a task optimally is &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;reinforcement learning&lt;/a&gt;. Technically it is not one algorithm, but an extended family of related algorithms that all solve a specific formalisation of the learning problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Informally, Reinforcement Learning (RL) is about finding optimal solutions to problems defined in terms of an &lt;em&gt;agent&lt;/em&gt; that can observe the &lt;em&gt;state&lt;/em&gt; of an &lt;em&gt;environment&lt;/em&gt;, take &lt;em&gt;actions&lt;/em&gt; in that environment and experience &lt;em&gt;rewards&lt;/em&gt; which are somehow related to the state and action. RL solvers need to be designed to cope with situations where rewards are received later than when important actions were taken, and this is usually achieved by the algorithm learning an internal expectation of later rewards associated with state and/or state-action pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some resources for studying Reinforcement Learning:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning: An Introduction (Second Edition)&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Algorithms for Reinforcement Learning (PDF)&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://eu.udacity.com/course/reinforcement-learning--ud600&quot; rel=&quot;nofollow noreferrer&quot;&gt;Udacity Reinforcement Learning course&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;David Silver UCL lectures on Reinforcement Learning&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You will find the subject itself is quite large as more and more sophisticated variations of the algorithms are necessary as the problem to solve becomes harder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Starting games for studying reinforcement learning might include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Tik-tac-toe (aka Noughts and crosses) - this can be solved easily using search, but it makes for a simple toy problem to solve using basic RL techniques.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mazes - in the reinforcement learning literature, there are many examples of &quot;grid world&quot; games where an agent moves in single N,E,S,W steps on a small board that can be populated with hazards and goals.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Blackjack (aka 21)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you want to work with agents for playing video games, you will also want to learn about &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural networks&lt;/a&gt; and probably in some detail - you will need deep, convolutional neural networks to process screen graphics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A relatively new resource for RL is &lt;a href=&quot;https://universe.openai.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI Universe&lt;/a&gt;. They have done a lot of work to package up environments ready to train agents against, meaning you can concentrate on studying the learning algorithms, as opposed to the effort of setting up the environment.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Regarding your list of current skills: None of them are directly relevant to reinforcement learning. However:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you can understand the maths and theory from your previous course, then you should also be able to understand reinforcement learning theory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you have studied any online or batch supervised learning techniques, then these can be used as components inside a RL framework. Typically they can be used to approximate a &lt;em&gt;value function&lt;/em&gt; of the game state, based on feedback from successes and failures so far.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-19T07:59:29.710" LastActivityDate="2018-03-19T07:59:29.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16068" PostTypeId="2" ParentId="16066" CreationDate="2017-01-04T14:19:07.273" Score="7" Body="&lt;p&gt;It highly depends on the type of game and the information about the state of the game that is available to your AI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the most famous game playing AIs from last few years are based on deep reinforcement learning (e.g. &lt;a href=&quot;https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf&quot; rel=&quot;noreferrer&quot;&gt;Playing Atari with Deep Reinforcement Learning&lt;/a&gt;), which is normal reinforcement learning (e.g. Q-learning) with a deep neural network as reward value function approximation. These approaches receive the raw pixels of the game plus the points of the player, and output the actions of a game pad, much like a human. In order to do something like that, you need to master reinforcement learning (see &lt;a href=&quot;https://webdocs.cs.ualberta.ca/~sutton/book/bookdraft2016sep.pdf&quot; rel=&quot;noreferrer&quot;&gt;Sutton and Barto's seminal book&lt;/a&gt;) and deep learning (see &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;noreferrer&quot;&gt;Ian Goodfellow et al. book&lt;/a&gt;), and then how to fuse them into deep reinforcement learning (search for &quot;reinforcement learning&quot; in any curated list of deep learning papers like &lt;a href=&quot;https://github.com/terryum/awesome-deep-learning-papers&quot; rel=&quot;noreferrer&quot;&gt;this one&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if the information about the game that is available to your AI is more structured than that (e.g. position of the player, description of the environment), you can do well with more classical approaches where you decompose your game into tractable problems and solve each one algorithmically e.g. by searching with &lt;a href=&quot;https://en.wikipedia.org/wiki/A*_search_algorithm&quot; rel=&quot;noreferrer&quot;&gt;A*&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-01-04T14:19:07.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16069" PostTypeId="1" AcceptedAnswerId="16095" CreationDate="2017-01-04T15:14:30.100" Score="0" ViewCount="169" Body="&lt;p&gt;I'm trying to build a system which tackles the following problem. Given a list of research papers &lt;em&gt;L&lt;/em&gt;, I would like to extract the following information from each paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The problem addressed by the paper&lt;/li&gt;&#xA;&lt;li&gt;Solution to the problem proposed by the paper&lt;/li&gt;&#xA;&lt;li&gt;(Optional) Results, i.e. how good is the solution&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have dealt with the easy part, i.e. setting up the blocks that interact with web search engines (e.g. Google scholar) and gather the data, given a set of search keys. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't have the knowledge to set up the 'intelligent' part, i.e. the one which extracts the aforementioned information from the text data. I would prefer to extract information based on the Abstract, but analysis the complete text wouldn't be a dealbreaker for me. I'm mostly interested in Computer Networking papers, e.g. published on SIGCOMM, CoNEXT, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there tools - preferably open source - which can extract this information from research papers?&lt;/li&gt;&#xA;&lt;li&gt;If not, what techniques do you recommend me to start looking at (e.g. research papers, tutorials, etc.) to accomplish this?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27621" LastActivityDate="2017-01-05T05:29:35.933" Title="Text Mining of Research Paper Abstracts" Tags="|text-mining|information-retrieval|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16070" PostTypeId="1" AcceptedAnswerId="16074" CreationDate="2017-01-04T16:03:51.747" Score="0" ViewCount="144" Body="&lt;p&gt;I have a reasonably large set of images that I want to classify using a neural network. I can't fil them all into memory at once, so I decided to process them in batches of 200. I'm using an cross-entropy cost function with a minimization algorithm from &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.optimize.fmin_cg.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;numpy&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is it correct to pass learned weights between batches and use them as a starting point of the minimization? Would this eventually cause my hypothesis to fit all the data, or will each iteration simply re-fit the weights for itself? What is the general approach to such a problem?&lt;/p&gt;&#xA;" OwnerUserId="27624" LastActivityDate="2017-01-04T18:17:52.647" Title="Computing weights in batch gradient descent" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16071" PostTypeId="1" CreationDate="2017-01-04T17:14:55.407" Score="-1" ViewCount="98" Body="&lt;p&gt;I want to know how to display those statement ( what the correct statement for them&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Display the name of the school that contain the maximum number of students&#xA;For each school, display the number of students in each class&#xA;I am building school management system project and the data base made in sql server&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanx a lot &lt;/p&gt;&#xA;" OwnerUserId="27625" LastActivityDate="2017-01-04T19:34:52.360" Title="Sql satament sql server" Tags="|sql|" AnswerCount="1" CommentCount="0" ClosedDate="2017-01-04T20:13:34.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="16072" PostTypeId="2" ParentId="16066" CreationDate="2017-01-04T17:36:53.577" Score="4" Body="&lt;p&gt;What you are looking for is called &lt;em&gt;Reinforcement Learning&lt;/em&gt;. At my university, there is a complete course ($15 \cdot 3h = 45h$) only to introduce students to this topic. Here are my (mostly german) &lt;a href=&quot;https://martin-thoma.com/probabilistische-planung/&quot; rel=&quot;nofollow noreferrer&quot;&gt;lecture notes to probabilistic planning&lt;/a&gt;. I would say this is definitively an advanced topic for machine learning.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Topcis to learn about&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Markov Decision Processes (MDPs)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Policy and Value iteration&lt;/li&gt;&#xA;&lt;li&gt;Project: Rock-Paper-Scissors / &lt;a href=&quot;https://en.wikipedia.org/wiki/Tic-tac-toe&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tic-Tac-Toe&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Partially Obversable Markov Decision Processes&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Project: Black Jack&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Reinforcment learning&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Q-Learning&lt;/li&gt;&#xA;&lt;li&gt;SARSA&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Other simple games&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pong&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pong&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Inverted_pendulum&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inverted Pendulum&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Backgammon&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Other resources&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://gym.openai.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI Gym&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Book by Sutton&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="836" LastEditDate="2017-12-07T17:04:30.710" LastActivityDate="2017-12-07T17:04:30.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16073" PostTypeId="1" AcceptedAnswerId="16097" CreationDate="2017-01-04T17:48:59.110" Score="1" ViewCount="138" Body="&lt;p&gt;Let's say I have a supervised learning problem with a sequence of features and labels. First, I learn on the training data and then I decide to stream in data, point by point and do online learning. Is it possible to update the weights or figure out the feature importances as each data point comes in? Also, what online learning algorithms would allow me to do this and can this be done in Python?&lt;/p&gt;&#xA;" OwnerUserId="27626" LastActivityDate="2017-01-05T05:54:35.033" Title="Online learning w/ feature weighting/adjusting" Tags="|feature-selection|online-learning|data-stream-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16074" PostTypeId="2" ParentId="16070" CreationDate="2017-01-04T18:17:52.647" Score="3" Body="&lt;p&gt;What you need to use is mini-batch gradient descent or stochastic gradient descent. You will need to shuffle your samples and make draws from it of the batch size you are aiming for, also you will have to make sure all of the samples in your data are included which will constitute of one epoch. Train for a few epochs depending on the data. Here is a good blog post on mini-batch &lt;a href=&quot;http://sebastianruder.com/optimizing-gradient-descent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-04T18:17:52.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16075" PostTypeId="2" ParentId="16071" CreationDate="2017-01-04T19:34:52.360" Score="2" Body="&lt;p&gt;Without understanding the table structure, it is hard to provide you much help. If you could update us with the table structure, I can absolutely help with this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you want this as one query, but, I feel this as two queries would be the most logical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Please see the schema below:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LLqFy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LLqFy.png&quot; alt=&quot;table_structure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Max Enrolled Students, by School:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT school.id                   AS `School ID`, &#xA;       COUNT(DISTINCT students.id) AS `Enrollment Count` &#xA;FROM   courses &#xA;       LEFT JOIN course_enrollment ce &#xA;              ON ce.course_id = courses.id &#xA;       LEFT JOIN school school &#xA;              ON school.id = courses.school_id &#xA;       LEFT JOIN students students &#xA;              ON students.id = ce.student_id &#xA;GROUP  BY school.id &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Expected Result:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;School ID   Enrollment Count&#xA;1   7&#xA;2   4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Enrollment, per class, by school:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT school.id           AS `School ID`, &#xA;       courses.course_name AS `Course Name`, &#xA;       COUNT(students.id)  AS `Enrollment Count` &#xA;FROM   courses &#xA;       LEFT JOIN course_enrollment ce &#xA;              ON ce.course_id = courses.id &#xA;       LEFT JOIN school school &#xA;              ON school.id = courses.school_id &#xA;       LEFT JOIN students students &#xA;              ON students.id = ce.student_id &#xA;GROUP  BY school.id, &#xA;          courses.course_name &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Expected Result:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;School ID   Course Name Enrollment Count&#xA;1   CHEM1301    7&#xA;1   ENGL1301    3&#xA;2   ACCT1301    4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;All schools, courses, professors and the respective enrolled students:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SELECT school.id                AS `School ID`, &#xA;       school.name              AS `School Name`, &#xA;       courses.course_name      AS `Course Name`, &#xA;       courses.course_professor AS `Professor`, &#xA;       students.first_name      `Student First Name`, &#xA;       students.last_name       AS `Student Last Name` &#xA;FROM   school school &#xA;       LEFT JOIN courses courses &#xA;              ON courses.school_id = school.id &#xA;       LEFT JOIN course_enrollment ce &#xA;              ON ce.course_id = courses.id &#xA;       LEFT JOIN students students &#xA;              ON students.id = ce.student_id &#xA;GROUP  BY school.id, &#xA;          courses.id, &#xA;          students.id &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Expected Output:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;School ID   School Name Course Name Professor   Student First Name  Student Last Name&#xA;1   School A    CHEM1301    John Doe    Aden    Baxter&#xA;1   School A    CHEM1301    John Doe    Anthony Leslie&#xA;1   School A    CHEM1301    John Doe    Laird   Jewel&#xA;1   School A    CHEM1301    John Doe    Osmond  Mikey&#xA;1   School A    CHEM1301    John Doe    Isidore Josiah&#xA;1   School A    CHEM1301    John Doe    Rafferty    Tate&#xA;1   School A    CHEM1301    John Doe    Dave    Lenox&#xA;1   School A    ENGL1301    Jacob Prask Aden    Baxter&#xA;1   School A    ENGL1301    Jacob Prask Rafferty    Tate&#xA;1   School A    ENGL1301    Jacob Prask Dave    Lenox&#xA;2   School B    ACCT1301    Glenn Ogg   Sachie  Baldwin&#xA;2   School B    ACCT1301    Glenn Ogg   Cletis  Nicky&#xA;2   School B    ACCT1301    Glenn Ogg   Garey   Simon&#xA;2   School B    ACCT1301    Glenn Ogg   Bennie  Howard&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can create the same tables I made by executing this SQL script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# ************************************************************&#xA;# Sequel Pro SQL dump&#xA;# Version 4541&#xA;#&#xA;# http://www.sequelpro.com/&#xA;# https://github.com/sequelpro/sequelpro&#xA;#&#xA;# Host: REDACTED (MySQL 5.7.14-google-log)&#xA;# Database: school&#xA;# Generation Time: 2017-01-04 19:30:50 +0000&#xA;# ************************************************************&#xA;&#xA;&#xA;/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;&#xA;/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;&#xA;/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;&#xA;/*!40101 SET NAMES utf8 */;&#xA;/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;&#xA;/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;&#xA;/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;&#xA;&#xA;&#xA;# Dump of table course_enrollment&#xA;# ------------------------------------------------------------&#xA;&#xA;DROP TABLE IF EXISTS `course_enrollment`;&#xA;&#xA;CREATE TABLE `course_enrollment` (&#xA;  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,&#xA;  `student_id` int(11) unsigned DEFAULT NULL,&#xA;  `course_id` int(11) unsigned NOT NULL,&#xA;  PRIMARY KEY (`id`),&#xA;  KEY `course_id` (`course_id`),&#xA;  KEY `student_id` (`student_id`),&#xA;  CONSTRAINT `course_enrollment_ibfk_1` FOREIGN KEY (`course_id`) REFERENCES `course_enrollment` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,&#xA;  CONSTRAINT `course_enrollment_ibfk_2` FOREIGN KEY (`student_id`) REFERENCES `students` (`id`) ON DELETE CASCADE ON UPDATE CASCADE&#xA;) ENGINE=InnoDB DEFAULT CHARSET=utf8;&#xA;&#xA;LOCK TABLES `course_enrollment` WRITE;&#xA;/*!40000 ALTER TABLE `course_enrollment` DISABLE KEYS */;&#xA;&#xA;INSERT INTO `course_enrollment` (`id`, `student_id`, `course_id`)&#xA;VALUES&#xA;    (1,1,1),&#xA;    (2,1,2),&#xA;    (3,2,1),&#xA;    (4,3,1),&#xA;    (5,4,1),&#xA;    (6,5,1),&#xA;    (7,6,1),&#xA;    (8,6,2),&#xA;    (9,7,3),&#xA;    (10,8,3),&#xA;    (11,9,3),&#xA;    (12,10,3),&#xA;    (13,11,1),&#xA;    (14,11,2);&#xA;&#xA;/*!40000 ALTER TABLE `course_enrollment` ENABLE KEYS */;&#xA;UNLOCK TABLES;&#xA;&#xA;&#xA;# Dump of table courses&#xA;# ------------------------------------------------------------&#xA;&#xA;DROP TABLE IF EXISTS `courses`;&#xA;&#xA;CREATE TABLE `courses` (&#xA;  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,&#xA;  `course_name` text,&#xA;  `course_professor` text,&#xA;  `school_id` int(11) unsigned DEFAULT NULL,&#xA;  PRIMARY KEY (`id`),&#xA;  KEY `school_id` (`school_id`),&#xA;  CONSTRAINT `courses_ibfk_1` FOREIGN KEY (`school_id`) REFERENCES `school` (`id`) ON DELETE CASCADE ON UPDATE CASCADE&#xA;) ENGINE=InnoDB DEFAULT CHARSET=utf8;&#xA;&#xA;LOCK TABLES `courses` WRITE;&#xA;/*!40000 ALTER TABLE `courses` DISABLE KEYS */;&#xA;&#xA;INSERT INTO `courses` (`id`, `course_name`, `course_professor`, `school_id`)&#xA;VALUES&#xA;    (1,'CHEM1301','John Doe',1),&#xA;    (2,'ENGL1301','Jacob Prask',1),&#xA;    (3,'ACCT1301','Glenn Ogg',2);&#xA;&#xA;/*!40000 ALTER TABLE `courses` ENABLE KEYS */;&#xA;UNLOCK TABLES;&#xA;&#xA;&#xA;# Dump of table school&#xA;# ------------------------------------------------------------&#xA;&#xA;DROP TABLE IF EXISTS `school`;&#xA;&#xA;CREATE TABLE `school` (&#xA;  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,&#xA;  `name` text,&#xA;  PRIMARY KEY (`id`)&#xA;) ENGINE=InnoDB DEFAULT CHARSET=utf8;&#xA;&#xA;LOCK TABLES `school` WRITE;&#xA;/*!40000 ALTER TABLE `school` DISABLE KEYS */;&#xA;&#xA;INSERT INTO `school` (`id`, `name`)&#xA;VALUES&#xA;    (1,'School A'),&#xA;    (2,'School B\n');&#xA;&#xA;/*!40000 ALTER TABLE `school` ENABLE KEYS */;&#xA;UNLOCK TABLES;&#xA;&#xA;&#xA;# Dump of table students&#xA;# ------------------------------------------------------------&#xA;&#xA;DROP TABLE IF EXISTS `students`;&#xA;&#xA;CREATE TABLE `students` (&#xA;  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,&#xA;  `first_name` text,&#xA;  `last_name` text,&#xA;  PRIMARY KEY (`id`)&#xA;) ENGINE=InnoDB DEFAULT CHARSET=utf8;&#xA;&#xA;LOCK TABLES `students` WRITE;&#xA;/*!40000 ALTER TABLE `students` DISABLE KEYS */;&#xA;&#xA;INSERT INTO `students` (`id`, `first_name`, `last_name`)&#xA;VALUES&#xA;    (1,'Aden','Baxter'),&#xA;    (2,'Anthony','Leslie'),&#xA;    (3,'Laird','Jewel'),&#xA;    (4,'Osmond','Mikey'),&#xA;    (5,'Isidore','Josiah'),&#xA;    (6,'Rafferty','Tate'),&#xA;    (7,'Sachie','Baldwin'),&#xA;    (8,'Cletis','Nicky'),&#xA;    (9,'Garey','Simon'),&#xA;    (10,'Bennie','Howard'),&#xA;    (11,'Dave','Lenox');&#xA;&#xA;/*!40000 ALTER TABLE `students` ENABLE KEYS */;&#xA;UNLOCK TABLES;&#xA;&#xA;&#xA;&#xA;/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;&#xA;/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;&#xA;/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;&#xA;/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;&#xA;/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;&#xA;/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27627" LastActivityDate="2017-01-04T19:34:52.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16076" PostTypeId="1" AcceptedAnswerId="16085" CreationDate="2017-01-04T20:31:22.607" Score="1" ViewCount="41" Body="&lt;p&gt;I am looking for a relatively large amount of jazz (and other) solos to use for data science purposes - probably somewhere up to a 100 transctipts. Although that is not that much and many pdf transcripts are available, I want the data to be in a format where I can easily access all the information in Python (which note was played and at what beat). Manually writing all that data would, however, be very tedious and time-consuming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus my question: does anyone know of any software that automatically reads pdf transcripts and puts it in a different file format, or alternatively of a website with transcripts already in a format in which I can access all the data easily from Python?&lt;/p&gt;&#xA;" OwnerUserId="23511" LastActivityDate="2017-01-04T22:57:00.263" Title="Best way to gather musical data" Tags="|data-mining|dataset|scraping|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16077" PostTypeId="2" ParentId="16060" CreationDate="2017-01-04T20:39:22.017" Score="60" Body="&lt;p&gt;The terms are different:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Equivariant to translation&lt;/em&gt; means that a translation of input features results in an equivalent translation of outputs. So if your pattern 0,3,2,0,0 on the input results in 0,1,0,0 in the output, then the pattern 0,0,3,2,0 might lead to 0,0,1,0&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Invariant to translation&lt;/em&gt; means that a translation of input features doe not change the outputs at all. So if your pattern 0,3,2,0,0 on the input results in 0,1,0 in the output, then the pattern 0,0,3,2,0 would also lead to 0,1,0&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For feature maps in convolutional networks to be useful, they typically need both properties in some balance. The equivariance allows the network to generalise edge, texture, shape detection in different locations. The invariance allows precise location of the detected features to matter less. These are two complementary types of generalisation for many image processing tasks.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-01-04T20:45:52.867" LastActivityDate="2017-01-04T20:45:52.867" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16078" PostTypeId="1" CreationDate="2017-01-04T21:32:00.173" Score="1" ViewCount="374" Body="&lt;p&gt;When processing texts written in a human-readable language, typically a first step is &lt;em&gt;tokenization&lt;/em&gt;, where we have a dictionary of tokens (words) and we collapse each substring in that dictionary into a single token.  For a language like English, the dictionary is known in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I want to parse files with a format not based on any human language.  I don't have a dictionary of known tokens; instead, I want to learn this dictionary.  I do have lots of sample data I can analyze.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there techniques for generating a dictionary, given many example files?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect that any common substring that appears in many files is a good candidate for a token, so an obvious approach is to look for all common substrings and add them to the dictionary.  However, a complication is that there is no equivalent of whitespace that separates tokens, so it seems like some filtering might be needed to avoid treating phrases (sequences of multiple tokens) as a new token.  For example, imagine if we had a large corpus of English writing, except that all spaces and punctuation are omitted, and we tried to infer a dictionary of English words from that.  We might naturally infer that &quot;ARE&quot;, &quot;THE&quot;, and &quot;HER&quot; are reasonable tokens, as they occur frequently as substrings.  All well and good so far.  However, we wouldn't want the algorithm to add &quot;ARETHE&quot; to the dictionary, even though &quot;ARETHE&quot; appears fairly frequently, as &quot;ARETHE&quot; is just the concatenation of the two tokens &quot;ARE&quot; and &quot;THE&quot;.  I suspect it might be possible to detect this, because the frequency of &quot;ARETHE&quot; is not much more than the product of the frequencies of &quot;ARE&quot; and &quot;THE&quot;, but I'm not sure.  Is there a clean way to handle this wrinkle?&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2017-01-04T21:59:19.950" LastActivityDate="2017-01-05T08:28:00.523" Title="Learning token dictionary" Tags="|machine-learning|nlp|text-mining|parsing|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16079" PostTypeId="2" ParentId="16062" CreationDate="2017-01-04T21:34:48.797" Score="6" Body="&lt;p&gt;I've posted a very similar question on Cross Validated few months ago and got a very large number of responses. Read the responses &lt;em&gt;and&lt;/em&gt; the comments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/215154/variable-selection-for-predictive-modeling-really-needed-in-2016&quot;&gt;https://stats.stackexchange.com/questions/215154/variable-selection-for-predictive-modeling-really-needed-in-2016&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-04T21:34:48.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16080" PostTypeId="2" ParentId="16078" CreationDate="2017-01-04T21:55:27.700" Score="2" Body="&lt;p&gt;Actually you can often just split on the white space, remove punctuation, and lowercase. This will provide your tokens. For example, if I had a string &quot;jdf asdsa sdfr&quot; (no English), then I could derive the tokens jdf, asdsa, and sdfr. The only thing that is known before hand would be the stopwords (and, be it the), which do come from the English dictionary. However, in this case, it does not sound like you would need stopwords. I would suggest looking at the following libraries available in python:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SpaCy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NLTK&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scikit-learn&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-04T21:55:27.700" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16081" PostTypeId="2" ParentId="16078" CreationDate="2017-01-04T22:01:16.713" Score="1" Body="&lt;p&gt;Also, for words that appear together as one such as &quot;wordsthatappeartogetherasone&quot;, I suggest using Microsoft Cognitive Services Bing Spell Check API. It has word break functionality. You can test it out here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.microsoft.com/cognitive-services/en-us/bing-spell-check-api&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.microsoft.com/cognitive-services/en-us/bing-spell-check-api&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-04T22:01:16.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16082" PostTypeId="2" ParentId="16073" CreationDate="2017-01-04T22:05:04.510" Score="0" Body="&lt;p&gt;Yes, this can be done in Python. Scikit-Learn has a few online learning algorithms available, of which you can derive the feature importances. Look at the following webpage under 6.1.3. Incremental learning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/scaling_strategies.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-04T22:05:04.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16083" PostTypeId="1" CreationDate="2017-01-04T22:09:12.513" Score="1" ViewCount="142" Body="&lt;p&gt;I have been implementing a convolutional neural net for Character recognition (MNIST) and I'm having troubles when trying to train it (basically the CNN doesn't learn). So I was wondering if the topology that I used is wrong...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;CNN topology:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input -&gt; Conv layer 3*3 (10 filters)  -&gt; Maxpool 2*2 -&gt; Conv layer 4*4 (30 filters) -&gt; Maxpool 2*2 -&gt; Fully Connected layer -&gt; Fully Connected layer -&gt; Fully Connected layer (the result of this layer is the output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see, I'm not using softmax, but I think that not using it should not be the problem.&#xA;I also use &lt;strong&gt;root-mean-square error&lt;/strong&gt; as &lt;strong&gt;loss function&lt;/strong&gt;.&#xA;Another thing I want to add is that, when I say that it doesn't learn, I mean that 10% of the time the output is correct, but I noticed that the output that is correct is when the image is a 9 (and not any other number).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone could give me some advice about some CNN structure or about an existing CNN, I would thank you a lot.&lt;/p&gt;&#xA;" OwnerUserId="26477" LastEditorUserId="29169" LastEditDate="2021-02-16T15:25:30.447" LastActivityDate="2021-02-16T15:25:30.447" Title="Convolutional net topology" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16084" PostTypeId="2" ParentId="16060" CreationDate="2017-01-04T22:53:51.223" Score="84" Body="&lt;p&gt;Equivariance and invariance  are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by &lt;a href=&quot;https://datascience.stackexchange.com/users/6622/xian&quot;&gt;@Xi'an&lt;/a&gt;, you can find previous uses (anterior to Convolutional Neural Networks) in the statistical literature, for instance on the notions of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Invariant_estimator&quot; rel=&quot;noreferrer&quot;&gt;invariant estimator&lt;/a&gt; and especially the &lt;a href=&quot;https://en.wikipedia.org/wiki/Invariant_estimator#Pitman_estimator&quot; rel=&quot;noreferrer&quot;&gt;Pitman estimator&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would like to mention that &lt;strong&gt;it would be better if both terms keep separate meaning&lt;/strong&gt;, as the prefix &lt;a href=&quot;https://en.wiktionary.org/wiki/in-#Latin&quot; rel=&quot;noreferrer&quot;&gt;&quot;&lt;em&gt;in-&lt;/em&gt;&quot;&lt;/a&gt; in &lt;em&gt;invariant&lt;/em&gt; is privative (meaning &quot;no variance&quot; at all), while &lt;a href=&quot;https://en.wiktionary.org/wiki/equi-&quot; rel=&quot;noreferrer&quot;&gt;&quot;&lt;em&gt;equi-&lt;/em&gt;&quot;&lt;/a&gt; in &lt;em&gt;equivariant&lt;/em&gt; refers to &quot;varying in a similar or equivalent proportion&quot;. &lt;strong&gt;In other words, one &lt;em&gt;in-&lt;/em&gt; does not vary, the other &lt;em&gt;equi-&lt;/em&gt; does&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us start from simple image features, and suppose that image &lt;span class=&quot;math-container&quot;&gt;$I$&lt;/span&gt; has a unique maximum &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; at spatial pixel location &lt;span class=&quot;math-container&quot;&gt;$(x_m,y_m)$&lt;/span&gt;, which is here the main classification feature. &lt;strong&gt;In other words: an image and all its translations are &quot;the same&quot;&lt;/strong&gt;.&#xA;An interesting property of classifiers is their ability to classify in the same manner some distorted versions &lt;span class=&quot;math-container&quot;&gt;$I'$&lt;/span&gt; of &lt;span class=&quot;math-container&quot;&gt;$I$&lt;/span&gt;, for instance translations by all vectors  &lt;span class=&quot;math-container&quot;&gt;$(u,v)$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maximum value  &lt;span class=&quot;math-container&quot;&gt;$m'$&lt;/span&gt; of &lt;span class=&quot;math-container&quot;&gt;$I'$&lt;/span&gt; is &lt;em&gt;invariant&lt;/em&gt;: &lt;span class=&quot;math-container&quot;&gt;$m'=m$&lt;/span&gt;: the value is the same. While its location will be at &lt;span class=&quot;math-container&quot;&gt;$(x'_m,y'_m)=(x_m-u,y_m-v)$&lt;/span&gt;, and is &lt;strong&gt;equivariant&lt;/strong&gt;, meaning that &lt;em&gt;is varies &quot;equally&quot; with the distortion&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The precise formulations given (in mathematical terms) for equivariance depend on the class of objects and transformations one considers: translation, rotation, scale, shear, shift, etc. So I prefer here to focus on the notion that is most often used in practice (I accept the blame from a theoretical stand-point).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, translations by vectors  &lt;span class=&quot;math-container&quot;&gt;$(u,v)$&lt;/span&gt; of the image (or some more generic actions) can be equipped with a structure of composition, like that  of a group &lt;span class=&quot;math-container&quot;&gt;$G$&lt;/span&gt; (here the group of translations). One specific &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt; denotes a specific element of the translation group (&lt;a href=&quot;https://en.wikipedia.org/wiki/Translational_symmetry&quot; rel=&quot;noreferrer&quot;&gt;translational symmetry&lt;/a&gt;). A function or feature &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; is invariant under the group of actions &lt;span class=&quot;math-container&quot;&gt;$G$&lt;/span&gt; if for all images in a class, and for any &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt;, &#xA;&lt;span class=&quot;math-container&quot;&gt;$$f(g(I)) = f(I)\,.$$&lt;/span&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words: if you change the image by action &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt;, the values for feature or function &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; are the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It becomes equivariant if there exists another mathematical &lt;em&gt;structure or action&lt;/em&gt;  (often a group again) &lt;span class=&quot;math-container&quot;&gt;$G'$&lt;/span&gt; that reflects &#xA; the&#xA;transformations  (from &lt;span class=&quot;math-container&quot;&gt;$G$&lt;/span&gt;) in &lt;span class=&quot;math-container&quot;&gt;$I$&lt;/span&gt;  &lt;em&gt;in a meaningful way&lt;/em&gt;. In other words, such that for each &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt;, you have some (unique?) &lt;span class=&quot;math-container&quot;&gt;$g' \in G'$&lt;/span&gt; such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$f(g(I)) = g'(f(I))\,.$$&lt;/span&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example on the group of translations, &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$g'$&lt;/span&gt; are the same (and hence &lt;span class=&quot;math-container&quot;&gt;$G'=G$&lt;/span&gt;): an integer translation of the image reflects as the exact same translation of the maximum location. This is sometimes refered to as &quot;same-equivariance&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another common definition is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$f(g(I)) = g(f(I))\,.$$&lt;/span&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I however used potentially different &lt;span class=&quot;math-container&quot;&gt;$G$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$G'$&lt;/span&gt; because sometimes &lt;span class=&quot;math-container&quot;&gt;$f(.)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$g(.)$&lt;/span&gt; do not lie in the same domain. This happens for instance in multivariate statistics (see e.g. &lt;a href=&quot;http://dx.doi.org/10.1080/10485250903431710&quot; rel=&quot;noreferrer&quot;&gt;Equivariance and invariance properties of multivariate quantile and related functions, and the role of standardisation&lt;/a&gt;).&#xA;But here, the uniqueness of the mapping between &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$g'$&lt;/span&gt; allows to get back to the original transformation &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often, people use the term invariance because the equivariance concept is unknown, or everybody else uses invariance, and equivariance would seem more pedantic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the record, other related notions (esp. in maths and physics) are termed &lt;em&gt;covariance&lt;/em&gt;, &lt;em&gt;contravariance&lt;/em&gt;, differential &lt;em&gt;invariance&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, translation-invariance, as least approximate, or in envelope, has been a quest for several signal and image processing tools. Notably, multi-rate (filter-banks) and multi-scale (wavelets or pyramids) transformations have  been design in the past 25 years, for instance under the hood of shift-invariant, cycle-spinning, stationary, complex, dual-tree wavelet transforms (for a review on 2D wavelets, &lt;a href=&quot;https://arxiv.org/abs/1101.5320&quot; rel=&quot;noreferrer&quot;&gt;A panorama on multiscale geometric representations&lt;/a&gt;). The wavelets can absorb a few discrete scale variations. All theses (approximate) invariances  often come with the price of redundancy in the number of transformed coefficients.&#xA;But they are more likely to yield shift-invariant, or shift-equivariant features.&lt;/p&gt;&#xA;" OwnerUserId="12527" LastEditorUserId="12527" LastEditDate="2020-01-11T21:11:17.923" LastActivityDate="2020-01-11T21:11:17.923" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="16085" PostTypeId="2" ParentId="16076" CreationDate="2017-01-04T22:57:00.263" Score="1" Body="&lt;p&gt;I think you are looking for musical optical character recognition software. A google search pointed me to &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Music_OCR&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt; where links to two open source applications resides: &lt;a href=&quot;https://audiveris.kenai.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Audiveris&lt;/a&gt; and &lt;a href=&quot;http://sourceforge.net/projects/openomr/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenOMR&lt;/a&gt;. I can not vouch for either program. &lt;/p&gt;&#xA;" OwnerUserId="27581" LastActivityDate="2017-01-04T22:57:00.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16086" PostTypeId="2" ParentId="8054" CreationDate="2017-01-04T23:03:28.410" Score="2" Body="&lt;p&gt;Use log-probabilities, then use k-shortest paths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability of a path is the product of the probabilities on its edges.  If you log-transform everything (so that each edge is annotated with the log of the probability rather than the probability itself), the log-probability of a path becomes the &lt;em&gt;sum&lt;/em&gt; of the log-probabilities on the edges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you want to rank the paths by decreasing log-probability, and find the $k$ paths with the highest log-probability.  Note that the log-probability of a path is just the length of the path (the sum of the lengths of the edges).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the problem becomes: given a graph with a length on each edge, find the $k$ shortest paths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are efficient algorithms for finding the $k$ shortest paths.  See, e.g., &lt;a href=&quot;https://cstheory.stackexchange.com/a/32501/5038&quot;&gt;https://cstheory.stackexchange.com/a/32501/5038&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:32:33.517" LastActivityDate="2017-01-04T23:03:28.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16087" PostTypeId="1" AcceptedAnswerId="16089" CreationDate="2017-01-04T23:12:54.290" Score="2" ViewCount="1698" Body="&lt;p&gt;Rather than creating 15 additional columns full of sparse binary data, could I: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) use the first 15 prime numbers as indexes for the 15 categories&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) store data by multiplying the prime numbers of the categories that otherwise would have a value of 1 in one-hot encoding&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) retrieve data by factorizing the value generated by multiplying unique prime numbers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex: 1914 would yield the list [2, 3, 11, 29] which would let you know that the user with the 1914 value has property 2, 3, 11, and 29 but nothing else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand this is limited because BIGINTs can only hold the product of the first 15 prime numbers, but would it not still be useful in some situations and save time when searching the database? The entire table would be 14 columns smaller. I guess this is less about machine learning algorithms and more about storing and retrieving data.&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2017-01-06T21:11:31.427" Title="Instead of one-hot encoding, can I store the same information in one column using a single value?" Tags="|bigdata|data|databases|sql|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16088" PostTypeId="1" AcceptedAnswerId="16094" CreationDate="2017-01-04T23:48:16.433" Score="9" ViewCount="8038" Body="&lt;p&gt;I have heard of genetic algorithms, but I have never seen practical examples and I've never got a systematic introduction to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now looking for a textbook which introduces genetic algorithms in detail and gives practical examples how they are used, what their strengths are compared to other solution methods and what their weaknesses are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard textbook for this?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2021-08-21T01:57:59.877" Title="Which book is a standard for introduction to genetic algorithms?" Tags="|reference-request|books|genetic-algorithms|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16089" PostTypeId="2" ParentId="16087" CreationDate="2017-01-05T00:03:03.740" Score="6" Body="&lt;p&gt;I suppose you &lt;em&gt;could&lt;/em&gt; do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the prime factors of the stored value, why don't you just store the flags as a bit string? Your example of 15 different possible values could be stored in a single SMALLINT (2-byte) SQL column. After retrieving the value, you would just need to extract the bits of interest for your record with some basic bitwise arithmetic.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2017-01-05T00:37:58.147" LastActivityDate="2017-01-05T00:37:58.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16090" PostTypeId="2" ParentId="15832" CreationDate="2017-01-05T00:21:40.840" Score="1" Body="&lt;p&gt;This falls into the category of &quot;distance metric learning&quot; or &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Similarity_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;similarity learning&lt;/a&gt;&quot;; specifically, it is what Wikipedia calls a &quot;classification similarity learning problem&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your specific case, I suggest you use logistic regression to classify pairs of addresses as either &quot;Similar&quot; or &quot;Not Similar&quot;.  In particular, map a pair of addresses (Address1,Address2) to a $d$-dimensional feature vector $x \in \mathbb{R}^d$.  Then, train a logistic regression model to predict &quot;Similar&quot; vs &quot;Not Similar&quot; from the feature vector $x$.  This turns your problem into a boolean classification task, and solves it by applying logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you form a feature vector?  That's where you'll use your domain knowledge to identify features, where each feature represents a factor that might be helpful in determining whether the two addresses match.  You could parse the address into elements (house number, street name, city, state, zip code), and have one feature (one dimension) per element of the address:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;One feature is 0 or 1 according to whether the house number matches in both addresses.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another feature has the Levenshtein edit distance between the street name in the first address and the street name in the second address (after normalizing both, e.g., by lowercasing them).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another feature has the Levenshtein edit distance between the two cities in the two addresses.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another has 0 or 1 according to whether the state matches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another has 0 or 1 according to whether the first five digits of ZIP code match.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another has 0 or 1 according to whether both have the same ZIP+4.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then, train a classifier using logistic regression to predict &quot;Similar&quot; vs &quot;Not Similar&quot; from these features, using your labelled training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One nice thing about this approach is that it is easy to add other features, if you discover that the classifier makes mistakes.  For instance, it is easy to add the edit distance between both addresses as another feature, or to replace the edit distance with a thresholded version of the edit distance, or feed each address into a geolocation service to get back (latitude,longitude) and then compute the distance between those two locations, or any other scheme of your choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, of course, you can replace logistic regression with any other boolean classification method, such as random forests.  However, logistic regression is probably a nice simple starting point for this problem.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-01-05T00:21:40.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16091" PostTypeId="2" ParentId="15974" CreationDate="2017-01-05T00:34:49.347" Score="2" Body="&lt;p&gt;Given your time budget and the potential challenges associated with class imbalance, I'd throw away the unlabelled data and use supervised learning on the labelled data.  Try a simple classifier, e.g., logistic regression or random forests or xgBoost, and use cross-validation to see how well they perform.  In advance put aside a held-out test data and don't touch it until the end.  Use cross-validation on the rest of the data to try different classifiers and different approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to deal with a severe class imbalance problem.  My suspicion is that you might need to budget a large fraction of your time dealing with class imbalance.  A true positive rate of 1% corresponds to a 100x class imbalance, which causes severe problems for many classifiers and might turn out to be a real headache.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the true underlying distribution doesn't match the distribution in your training set: the true positive rate in the wild is 1%, but 67% of your training set is positives.  You'll need to adjust for this as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways to deal with class imbalance and to adjust for differences between the imbalance in your training set vs in the wild.  I would suggest you start by setting class weights: you'll need to upweight the negatives by a factor of 198 (so that an error on a negative in your training set costs 198 times as much as an error on a positive in your positive set), since in the wild the distribution you expect to see is 2000 positives and 198000 negatives rather than 2000 positives and 1000 negatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge is that some classifiers struggle to handle this level of class imbalance well.  For instance, SVMs are known for sometimes failing badly in the presence of this kind of class imbalance; they can end up defaulting to always predict &quot;negative&quot;, since this achieves an error rate of only 1%.  In contrast, logistic regression tends to be better behaved, and provides an easy way to adjust for class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do some searching on class imbalance; you'll find lots written on multiple strategies you can use for handling it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If class imbalance turns out not to be a problem and you end up with a decent classifier on your first try, and you have more time to try to improve the results, then the next thing I'd spend time on is seeing if you design better/more features, given domain knowledge about where these instances come from.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-01-05T00:34:49.347" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16092" PostTypeId="2" ParentId="16088" CreationDate="2017-01-05T00:36:13.527" Score="2" Body="&lt;p&gt;I can recommend &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0201157675&quot; rel=&quot;nofollow noreferrer&quot;&gt;Genetic Algorithms in Search, Optimization, and Machine Learning by Goldberg&lt;/a&gt;. In particular, chapter 1 gives a great &quot;introduction to genetic algorithms with examples.&quot; The code examples are unfortunately in Pascal but readable even if not familiar with the language. &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/B005LQBV4C&quot; rel=&quot;nofollow noreferrer&quot;&gt;The book by Thomas Back&lt;/a&gt; is a little more advanced but also more complete (more &quot;evolutionary programming&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0262111705&quot; rel=&quot;nofollow noreferrer&quot;&gt;Genetic Programming by Koza&lt;/a&gt; is a good choice with many examples but less tutorial. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-01-05T00:36:13.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16093" PostTypeId="2" ParentId="16087" CreationDate="2017-01-05T00:38:12.113" Score="4" Body="&lt;p&gt;&quot;Premature optimization is the root of all evil&quot;.  --Knuth&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;could&lt;/em&gt; do this, but why?  You definitely don't want to do this, if you're going to feed the result into a classifier: most classifiers will perform worse after this transformation.  There's no point to do this, if you're trying to save space in a database: hard disks can store hundreds of gigabytes, so a measly 15 columns probably won't make any noticeable difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;15 additional columns is &lt;em&gt;tiny&lt;/em&gt;.  Just save the additional 15 columns and spare yourself headaches.  The cost of your time to program this up and troubleshoot all of the problems it causes downstream will almost surely exceed the miniscule cost of computation or storage to store the data in the natural format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you seriously have big data where you truly need to do some optimization, the very first step is to &lt;em&gt;measure&lt;/em&gt;: measure how much space/time you're actually consuming, and what the dominant contributors to that total cost is, and then focus on optimizing those dominant factors.  Optimizations should be guided by &lt;em&gt;data&lt;/em&gt;; otherwise you risk implementing complicated optimizations that make little difference overall but add needless complexity and epicycles to your data processing workflow.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-01-05T00:38:12.113" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16094" PostTypeId="2" ParentId="16088" CreationDate="2017-01-05T02:39:13.873" Score="11" Body="&lt;p&gt;Most of the &quot;standard textbooks&quot; (e.g., Goldberg, Mitchell, etc.) are pretty dated now. If you just want to have some confidence that you understand how the basic algorithms work, they're fine, but they tend to emphasize material that's doesn't necessarily match the more modern way of understanding and talking about things like theoretical issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used the Eiben and Smith (&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3642072852&quot; rel=&quot;noreferrer&quot;&gt;https://www.amazon.com/Introduction-Evolutionary-Computing-Natural/dp/3642072852&lt;/a&gt;) book to teach evolutionary algorithms to my students over the past few years. It's a bit more concise than some of the others, but if you're OK with that, I think it does a better job of covering the stuff that's relevant about evolutionary computation today.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastEditorUserId="2975" LastEditDate="2017-01-05T18:40:16.423" LastActivityDate="2017-01-05T18:40:16.423" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16095" PostTypeId="2" ParentId="16069" CreationDate="2017-01-05T05:29:35.933" Score="2" Body="&lt;p&gt;In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet.  So you would need pieces of article text that you know are 'Problems' and 'Solutions'.  Then the trained algorithm could identify new articles based on the training set.  But I imagine this is a lot of work to generate if you don't have this training data to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try some unsupervised method, for example you may split each article into multiple overlapping parts, and compute TFIDF vectors of each part.  Do this for many articles, then cluster the TFIDF vectors.  You can then look at your clusters and hope one of them is generally capturing the 'Problem', and another the 'Solution'.  It will probably be very messy, and you'll have to clean it up with some post-hoc heuristic, depending on what the result looks like.  I think this approach would be hard to actually get to work well, but it would avoid having to obtain (or create) a labeled data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you look at articles you may be able to devise generally good heuristics (e.g. the solution might be the first paragraph in a section titled 'Discussion' 70% of the time) to bolster both of these approaches, and if the article patterns are repeatable enough you may be able to just use heuristics to do a generally decent job.  Since articles are often very similarly structured, this is where I would start to see how far I can get, and how much I can simplify the problem using heuristics before passing the simplified problem to the other methods.&lt;/p&gt;&#xA;" OwnerUserId="13002" LastActivityDate="2017-01-05T05:29:35.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16097" PostTypeId="2" ParentId="16073" CreationDate="2017-01-05T05:54:35.033" Score="0" Body="&lt;p&gt;Online learning actually is an &lt;strong&gt;optimization method&lt;/strong&gt; , dealing with large scale data and huge feature space .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;FTRL&lt;/strong&gt; is a typical one , derived from &lt;strong&gt;stochastic gradient descent&lt;/strong&gt; . You can refer paper &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v15/mcmahan11b/mcmahan11b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.jmlr.org/proceedings/papers/v15/mcmahan11b/mcmahan11b.pdf&lt;/a&gt; if you want to know more about that . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other specific online methods developed based on it , such as &lt;strong&gt;TDAP&lt;/strong&gt; , you can check paper &lt;a href=&quot;http://www.cs.cmu.edu/~epxing/papers/2016/HuaWei_KDD16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.cmu.edu/~epxing/papers/2016/HuaWei_KDD16.pdf&lt;/a&gt; to know more .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you said , you wanted to know 『feature importances』while training . Model changes while iteration goes on or data points comes in , so the model will tell you the exact 『feature importances』. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At such circumstances , most of them are developed with scala or java based on Spark , others may be developed with c++ based on OMP , you can develop your own online learning method with python . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopes this contributes you -)&lt;/p&gt;&#xA;" OwnerUserId="9728" LastActivityDate="2017-01-05T05:54:35.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16098" PostTypeId="1" AcceptedAnswerId="16214" CreationDate="2017-01-05T06:16:26.033" Score="0" ViewCount="226" Body="&lt;p&gt;I know I can retrain Inception to label images, but can I just provide an image to an untrained Inception and get a label back? Is there an API for this? If there is, then why does Google open source this if they're trying to make money off of &lt;a href=&quot;https://cloud.google.com/vision/&quot; rel=&quot;nofollow noreferrer&quot;&gt;google cloud vision&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="27641" LastActivityDate="2017-01-09T19:44:09.723" Title="How to use InceptionV3 without retraining" Tags="|machine-learning|tensorflow|inception|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16099" PostTypeId="1" CreationDate="2017-01-05T06:29:57.993" Score="0" ViewCount="1150" Body="&lt;p&gt;I fit a dataset with a binary target class by the random forest. In python, I can do it either by randomforestclassifier or randomforestregressor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can get the classification directly from randomforestclassifier or I could run randomforestregressor first and get back a set of estimated probabilities. Then I can find a cutoff value to derive the predicted classes out of the set of probabilities. Both methods can achieve the same goal (i.e. predict the classes for the test data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I can observe that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;randomforestclassifier.predict_proba(X_test)[:,1]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is different from&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;randomforestregressor.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I just wanna confirm that both methods are valid and then which one is better in random forest application?&lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2020-06-20T07:32:56.767" Title="Which one I should choose for random forest?" Tags="|machine-learning|python|random-forest|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16100" PostTypeId="2" ParentId="16099" CreationDate="2017-01-05T07:54:32.877" Score="2" Body="&lt;p&gt;If you are doing a classification task, use random forest classifier. If you want probabilities use predict_proba or you can use predict directly to get classes. You might be getting correct results but understand that the random forest regressor works on a different cost function, and is not constrained to give outputs between 0 and 1, so what you are getting out of the regressor are not really probabilities. If you run it on enough datasets it might start giving you outputs greater than 1. Hence, bottom line stick to classifier. Also I will recommend reading up a bit on the differences of regression and classification tasks. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-05T07:54:32.877" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16102" PostTypeId="2" ParentId="16078" CreationDate="2017-01-05T08:28:00.523" Score="2" Body="&lt;p&gt;Do you have a way to evaluate the results?&#xA;How can you know if a string is indeed a token in this language?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to get possible tokens, I recommend that you'll try &lt;a href=&quot;https://en.wikipedia.org/wiki/Huffman_coding&quot; rel=&quot;nofollow noreferrer&quot;&gt;Huffman coding&lt;/a&gt;, the compression algorithm.&#xA;The tree that it builds will contain the tokens.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It will identify that &quot;the&quot; is more frequent that expected and worth compression.&lt;/li&gt;&#xA;&lt;li&gt;It will identify the &quot;ARETHE&quot; is not significantly more frequent than &quot;Are&quot; &quot;The&quot; and won't waste representation space in order to compress it.&lt;/li&gt;&#xA;&lt;li&gt;It is a classical algorithm so you'll probably find many suitable implementation and will be able to analyze your data quickly.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-01-05T08:28:00.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16103" PostTypeId="2" ParentId="16023" CreationDate="2017-01-05T08:48:04.627" Score="0" Body="&lt;p&gt;In the specific case where the grouping variable is categorical, it is only very slightly clunky but it makes sense and is just fine.  It sounds a bit smoother to use something like 'greater' in that case.  For example you might want to say:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The average effect for the female group was X.  When looking at the&#xA;  male group, the average effect increased to Y.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But you probably would be slightly better off to say:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The average salary for the female group was X.  When looking at the&#xA;  male group, the average effect, Y, was greater.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If the grouping variable in question is numeric and you use 'increase' or 'decrease' to describe it, then describing the metric with 'increase' or 'decrease' also sounds just fine.  E.g.:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As height increased, X decreased.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="13002" LastActivityDate="2017-01-05T08:48:04.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16104" PostTypeId="1" CreationDate="2017-01-05T08:53:11.223" Score="0" ViewCount="346" Body="&lt;p&gt;I am using Rotation Forest in r. &#xA;I have a very big list (which I can't give a sample) that has 50 attributes and 4000 rows.&#xA;for every row there is a few missing values in the attributes, it supposed to be like that and I cant fill the missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y &amp;lt;- list[,1]&#xA;x &amp;lt;- list[,2:ncol(list)]&#xA;rF &amp;lt;- rotationForest(x,y,K = round(ncol(x)/3),L=15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this is the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in cov.wt(z) : 'x' must contain finite values only&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;from what I gathered the problem is with NA in relevance to PCA in the rotation forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;any suggestion on what I can do?&lt;/p&gt;&#xA;" OwnerUserId="27645" LastActivityDate="2018-06-25T23:06:34.587" Title="Dealing with NA values while using Rotation Forest in R" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16105" PostTypeId="1" CreationDate="2017-01-05T09:14:43.460" Score="1" ViewCount="1151" Body="&lt;p&gt;Hello I am new to Applied ML and trying to solve a problem where I have given several images with few patches in it and masked images where these patches are classified/extracted as training data.Given that i need to use ML to train network such that It can generate those masked images for test data set.&#xA;I would like to know which technique will be best suitable in Both the cases where I want to use Python extended libraries like Scikit or Using just Numpy and Scipy.&#xA;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="17435" LastActivityDate="2018-02-27T11:27:41.783" Title="Supervised learning for image segmentation" Tags="|machine-learning|python|image-classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16106" PostTypeId="2" ParentId="16021" CreationDate="2017-01-05T09:44:17.657" Score="0" Body="&lt;p&gt;The tutorial you are looking at is attempting to model only the &lt;code&gt;seasonality&lt;/code&gt; of a time-series.  They first look at power of different frequencies, in the second figure in the tutorial.  They find a lot of power for the 1/12 frequency, as well as a few other harmonics, and also the 1/72 frequency.  But they decide that since the 1/12 frequency has the most power, and most of the others are harmonics of this, they will only model the 1/12 frequency.  You can see this from this quote:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Based on the spectral analysis of the data, fit a simple linear&#xA;  regression model using a cosine and sine term with a frequency of the&#xA;  most signficant component: 1 cycle/year (1 cycle/12 months).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So you therefore only need one frequency bin, and its negative, since you are only modeling one frequency.  In other words, in this part, they are just using essentially the equivalent of a single sine wave (at the 1/12 frequency) as their model to fit the data.  All the frequency bin variable does is to act as an index and select the component of the fft() that contains the 1/12 frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They then analyze their result (the autocorrelation plots) and find that they have indeed missed some of the cyclical patterns at other frequencies they did not model.  So then they add three more frequencies to the model.  Rather than using frequency bins this time to do the indexing, they simply ask for the 6 highest in power (6 because each of the 3 needs both positive and negative frequencies) components of the fft() with these lines: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Y,I] = sort(abs(tsdft),'descend');&#xA;indices = I(1:6);&#xA;tsfit2dft(indices) = tsdft(indices);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see the resulting plot that follows now better models the time series since it includes more of the frequencies that showed increased power.&lt;/p&gt;&#xA;" OwnerUserId="13002" LastEditorUserId="13002" LastEditDate="2017-01-05T09:53:59.403" LastActivityDate="2017-01-05T09:53:59.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16107" PostTypeId="1" AcceptedAnswerId="16109" CreationDate="2017-01-05T09:44:22.833" Score="2" ViewCount="3100" Body="&lt;p&gt;I fit the random forest to my dataset with a binary target class. I reset the probabilistic cutoff to a much lower value rather than the default 0.5 according to the ROC curve. Then I can improve the sensitivity (recall) but meanwhile sacrificed the precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just wanna confirm that the default 0.5 is not much meaningful and a practical probabilistic cutoff was often derived from ROC curve in practice. Am I on the right track on the application of random forest and other tree based models. &lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2017-01-06T08:48:46.090" Title="Is the probabilistic cutoff in random forest flexible?" Tags="|machine-learning|random-forest|decision-trees|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16109" PostTypeId="2" ParentId="16107" CreationDate="2017-01-05T10:36:47.203" Score="2" Body="&lt;p&gt;Yes, you are exactly right. 0.5 is just a heuristic, ROC curve and precision-recall curve give a much better idea of what the cut off should be. You can then use predict_proba, extract the probabilities and do the classification based on the cut-off you have inferred from ROC curve and.  precision-recall curve&lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-05T10:36:47.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16110" PostTypeId="1" CreationDate="2017-01-05T11:55:44.857" Score="-1" ViewCount="141" Body="&lt;p&gt;I'm trying to develop a deep learning implementation in Python to learn how the math behind it works. I've seen a lot of tutorials and they define weight matrices with different sizes(2x2, 3x3, 4x3 ....). How can I define a matrix size for a certain type of model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model I'm replicating is a linear model. So it's basics are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w1*x1 + w2*x2 + w3*x3 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27649" LastEditorUserId="27649" LastEditDate="2017-01-05T12:07:55.810" LastActivityDate="2017-04-05T13:30:32.343" Title="How I can define a weight matrix for a certain model?" Tags="|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16111" PostTypeId="2" ParentId="14222" CreationDate="2017-01-05T12:20:06.890" Score="0" Body="&lt;p&gt;I would solve this the following way - &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split all the words in the string&lt;/li&gt;&#xA;&lt;li&gt;Use the regex &lt;code&gt;^[a-zA-Z0-9]*$&lt;/code&gt; to get alphanumeric characters only. This is the model.&lt;/li&gt;&#xA;&lt;li&gt;Words before the regex are part of the brand's name.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explanation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;^&lt;/code&gt; asserts position at start of a line&#xA;&lt;code&gt;*&lt;/code&gt; Quantifier — Matches between zero and unlimited times, as many times as possible, giving back as needed (greedy)&#xA;&lt;code&gt;$&lt;/code&gt; asserts position at the end of a line&lt;/p&gt;&#xA;" OwnerUserId="20354" LastActivityDate="2017-01-05T12:20:06.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16112" PostTypeId="2" ParentId="16110" CreationDate="2017-01-05T12:35:35.157" Score="0" Body="&lt;p&gt;Depends completely on the kind of network you are building, what kind and how much data you have. For the sake of simplicity, lets talk about a network with just one hidden layer and binary output. Then the weight matrix for your hidden layer will be input dimension x hidden dimension where dimension is the number of units i.e.features. Then to convert your hidden features into the two outputs, you need a set of weights of the size hidden dimension x 2. If you want to know how many layers and how many units you should use, thats quite a vast topic. Here is a SO question that might help with that &lt;a href=&quot;https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw&quot;&gt;question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-05T12:35:35.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16113" PostTypeId="2" ParentId="11940" CreationDate="2017-01-05T17:25:57.907" Score="0" Body="&lt;p&gt;You might want to try a Bayesian classifier instead of SVM or Decision Trees. Your problem is a generalization of the spam filter problem, i.e. classifying emails into 'Spam' (class 1) or 'Not Spam' (class 2). Most Spam filters use Bayesian classification instead of the more common supervised learning method. &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2017-01-05T17:25:57.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16114" PostTypeId="1" AcceptedAnswerId="16201" CreationDate="2017-01-05T17:50:51.690" Score="0" ViewCount="58" Body="&lt;p&gt;I know I can retrain Inception to label images, but can I just provide an image to a non retrained Inception and get a label back?&lt;/p&gt;&#xA;" OwnerUserId="27641" LastActivityDate="2017-02-08T11:00:02.163" Title="Using TensorFlow's preexistent Inception labeler" Tags="|machine-learning|tensorflow|inception|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16115" PostTypeId="1" CreationDate="2016-12-07T00:45:33.510" Score="3" ViewCount="2276" Body="&lt;p&gt;I have a supervised learning (boolean classification) problem that involves strings.  Are there any resources where I can learn about the state of the art in techniques for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm familiar with many supervised learning techniques where we assume that each instance has a fixed number of attributes: e.g., the instance is represented by a feature vector.  But now I have something different; I have a string, i.e., a sequence of characters.  I don't know a priori what suitable features might be.  Are there any general techniques for doing machine learning in this context?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm especially interested in techniques that work given an existing training set (I don't have the ability to label additional data; I don't want an active learning algorithm), and that support some kind of regularization or can handle noisy labels.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Research I've done and approaches that don't meet the above requirements:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm familiar with regexps, finite automata, &lt;a href=&quot;https://en.wikipedia.org/wiki/Induction_of_regular_languages&quot; rel=&quot;nofollow noreferrer&quot;&gt;grammar induction&lt;/a&gt;, &lt;a href=&quot;http://learnlib.de/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LearnLib&lt;/a&gt;, &lt;a href=&quot;https://cs.stackexchange.com/q/118/755&quot;&gt;Angluin's algorithm&lt;/a&gt;, and similar concepts.  However, most of that work is in the active learning context, where the algorithm generates a new instance (a query) and asks the human to label that instance.  In my setting, I must work with passive learning: I have a training set $(x_i,y_i)$ where each $x_i$ is a string and $y_i$ is its class (positive or negative), but I can't obtain any more examples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm also familiar with automata minimization, so I know how to find the minimal DFA that can generate all positive instances in the training set, but that's not really a good solution.  It doesn't have any notion of regularization or Occam's razor and is not robust to a small amount of error/noise in the labels.  There might be a very small DFA that predicts the proper label with 99% accuracy, but automata minimization won't find it; it is overly focused on finding an automaton that generates all positive instance, even if this yields a much larger automaton.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm also familiar with the notion of &lt;a href=&quot;https://cs.stackexchange.com/q/19687/755&quot;&gt;separating automata&lt;/a&gt;, i.e., the smallest DFA that has 100% accuracy on the training set (accepts all positive instances and rejects all negative instances).  However, this has the same problems as automata minimization.  Also, this seems likely to overfit and essentially end up memorizing the training set.  Finally, the research literature I've seen stops at noting that finding the smallest separating automaton is NP-hard, and doesn't concern itself with practical algorithms for finding a separating automaton that is as small as possible.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm familiar with recurrent neural networks, but my impression is that they tend to require very large training sets, so probably won't be useful in my setting.  Are there variants of this technique that work with training sets that aren't enormous (say, hundreds of examples in the training set)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm familiar with some techniques from the natural language processing (NLP) literature, but they seem very specialized to parsing human languages.  Are there any general techniques that apply to other structured strings?  In my situation, the strings are not human-readable text written in some human language, but rather have some other content.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8560" OwnerDisplayName="D.W." LastEditorUserId="-1" LastEditDate="2017-04-13T12:48:38.763" LastActivityDate="2017-02-05T20:19:17.970" Title="Boolean classification on strings" Tags="|machine-learning|algorithms|supervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16116" PostTypeId="2" ParentId="15535" CreationDate="2017-01-05T18:54:07.050" Score="1" Body="&lt;p&gt;Have you tried early stopping?  Hold out 20% of the data and stop training when the error on the hold-out set starts to increase.  If you can't get high enough accuracy, increase the size of your model until you do.&lt;/p&gt;&#xA;" OwnerUserId="18416" LastActivityDate="2017-01-05T18:54:07.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16117" PostTypeId="1" CreationDate="2017-01-05T19:15:16.787" Score="0" ViewCount="319" Body="&lt;p&gt;I have been working on the Kaggle tutorial on the Titanic Disaster.&#xA;Although I get a result which seems good to me (on the training set) the trained model performs bad on the test set. &#xA;It would be great if you wanted to help me to understand what I am doing wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my notebook&#xA;&lt;a href=&quot;https://www.kaggle.com/asindico/titanic/my-titanic-attempt&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/asindico/titanic/my-titanic-attempt&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-01-05T19:52:36.797" Title="Titanic Disaster" Tags="|machine-learning|python|jupyter|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16118" PostTypeId="2" ParentId="16117" CreationDate="2017-01-05T19:37:06.697" Score="1" Body="&lt;p&gt;Here is a pretty solid notebook for you to compare with (their random forest yielded 0.98):&#xA;&lt;a href=&quot;https://www.kaggle.com/startupsci/titanic/titanic-data-science-solutions/notebook&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/startupsci/titanic/titanic-data-science-solutions/notebook&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you tried adjusting the number of estimators in your random forest model?  It looks like you are using 10, but more should help improve the model.  Try 100.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I get a result which seems good to me (on the training set) the trained model performs bad on the test set&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Don't base your results on the training set. Your test set is how you assess your model.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="19161" LastEditDate="2017-01-05T19:52:36.797" LastActivityDate="2017-01-05T19:52:36.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16119" PostTypeId="1" AcceptedAnswerId="16142" CreationDate="2017-01-05T20:17:52.940" Score="2" ViewCount="9692" Body="&lt;p&gt;I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data. &lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26163" LastEditDate="2017-01-05T20:38:25.413" LastActivityDate="2023-10-16T20:28:25.683" Title="The model performance vary between different train-test split?" Tags="|machine-learning|random-forest|training|" AnswerCount="5" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16120" PostTypeId="2" ParentId="11940" CreationDate="2017-01-05T20:42:53.923" Score="0" Body="&lt;p&gt;You could study the features: I noticed you removed the numbers. It could be that you did that because they are not likely to collide, but you could also solve that by introducing word classes (@number@ or @big_number@, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try to use the word classes on other word groups too. If that works it might indicate that your set suffers from sparsity (which wouldnt surprise me). You could use a feature selector for that purpose, for instance by ranking your dimensions on information gain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More advanced ways to battle sparsity include: Rocchio's algorithm or word2vec.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-05T20:42:53.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16121" PostTypeId="2" ParentId="14964" CreationDate="2017-01-05T21:16:11.427" Score="0" Body="&lt;p&gt;In the basics a good visual split is a good starting point. And yes, it is smart to keep in mind how the algorithms divide the space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good strategy, I personally like to apply is to start of with simple learners to learn how your data is structered. Hoe well does NN work, are there hints of local behavior? How well does Naive Bayes work? Is the concept complex or do individual features hold information? Etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for selecting the features: You could try ranking your features on methods that compare their use (such as information gain), or simply write a scheme that tries all combinations of two on your two methods (it's only 9 * 8 runs). If the space was a little bigger I would suggest a combination of the two. You might also want to try combining features (fi: PCA).&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-05T21:16:11.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16122" PostTypeId="1" AcceptedAnswerId="16125" CreationDate="2017-01-05T23:14:01.860" Score="0" ViewCount="2073" Body="&lt;p&gt;I am trying to make a chatbot using a deep neural network in python using keras. The problem I am having is that for the deep neural network to work, the input dimension has to be fixed. So my question is, how can I map a sequence of words (sentence of different lengths) to a fixed vector so I am able to feed this through a deep neural network?&lt;/p&gt;&#xA;" OwnerUserId="27668" LastEditorUserId="27668" LastEditDate="2017-01-05T23:24:35.903" LastActivityDate="2020-08-08T08:58:16.477" Title="Mapping sequences of different lengths to fixed vector - Python" Tags="|machine-learning|python|neural-network|nlp|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16124" PostTypeId="1" CreationDate="2017-01-06T00:45:08.940" Score="1" ViewCount="54" Body="&lt;p&gt;Is it possible for an algorithm to predict a new class that has never been before in training? For example, in my training data I have: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;I need a flight to Boston --&amp;gt; flight query&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;What is two plus two --&amp;gt; math query&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Who is the president of the United States --&amp;gt; Donald Trump&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I make a model that can predict a new class for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;How are you today --&amp;gt; ??? (Should be something like conversation query)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am almost looking for a way to combine supervised learning and unsupervised learning. Using supervised learning for a model to predict new classes (unsupervised learning)&lt;/p&gt;&#xA;" OwnerUserId="27579" LastActivityDate="2017-02-06T15:43:46.987" Title="Creating new classifications" Tags="|machine-learning|python|neural-network|classification|nlp|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16125" PostTypeId="2" ParentId="16122" CreationDate="2017-01-06T01:26:46.687" Score="1" Body="&lt;p&gt;A recurrent neural network (RNN) is probably what you want to look into. It has exactly that capability of dealing with variable length input sequence. In fact, currently some of the most successful applications of RNN are in the NLP area, where the inputs are sentences of varying size. Just google recurrent neural network LSTM and you will find lots of tutorials.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastActivityDate="2017-01-06T01:26:46.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16127" PostTypeId="1" AcceptedAnswerId="16775" CreationDate="2017-01-06T03:34:07.237" Score="3" ViewCount="553" Body="&lt;p&gt;All supervised learning techniques have some kind of an inductive bias. What is the inductive bias in Gaussian process models ?&lt;/p&gt;&#xA;" OwnerUserId="17169" LastEditorUserId="471" LastEditDate="2017-01-06T18:52:08.320" LastActivityDate="2017-02-06T07:33:05.777" Title="Inductive Bias in Gaussian process" Tags="|machine-learning|gaussian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16128" PostTypeId="1" AcceptedAnswerId="16241" CreationDate="2017-01-06T06:05:56.000" Score="9" ViewCount="18231" Body="&lt;p&gt;Can anyone suggest any algorithm and technique for spell checking?&#xA;After some googling, I found some interesting ones such as this one from Peter Norvig, &lt;a href=&quot;http://norvig.com/spell-correct.html&quot; rel=&quot;noreferrer&quot;&gt;http://norvig.com/spell-correct.html&lt;/a&gt; and few others. However, most of them were written many years ago. Therefore, I am trying to find out whether is there any newer/improved approach to tackle such problem.&lt;/p&gt;&#xA;" OwnerUserId="20974" LastActivityDate="2023-08-31T17:06:41.193" Title="Algorithms and techniques for spell checking" Tags="|machine-learning|python|nlp|algorithms|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16130" PostTypeId="2" ParentId="16128" CreationDate="2017-01-06T06:53:34.887" Score="4" Body="&lt;p&gt;Character level Recurrent Neural Network to the rescue here. Mainly you will build a character level language model using a recurrent neural network. Look into this github project. &lt;a href=&quot;https://github.com/joiharalds/rnnicespellchecker&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github&lt;/a&gt; and also this medium &lt;a href=&quot;https://medium.com/@majortal/deep-spelling-9ffef96a24f6#.bqk3hb576&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; to get a good understanding of the process. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-06T06:53:34.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16131" PostTypeId="1" AcceptedAnswerId="16139" CreationDate="2017-01-06T06:58:10.847" Score="0" ViewCount="127" Body="&lt;p&gt;I want to understand the intent of the customer using his search queries, let's say if a customer is interested in yoga pants, he can either search for yoga pants or exercise pants or workout tights etc. Is there a model that I can use to find out all the search keywords that can be related to yoga pants? &lt;/p&gt;&#xA;" OwnerUserId="27680" LastActivityDate="2017-01-06T10:17:38.540" Title="which machine learning technique can be used?" Tags="|machine-learning|nlp|stanford-nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16132" PostTypeId="2" ParentId="16124" CreationDate="2017-01-06T07:01:26.420" Score="1" Body="&lt;p&gt;Do you mean a class label that the algorithm has never seen before? Then no, it is not possible. If you dont have labelled data for all your samples you can run a LDA, to get some topics and then assign labels based on the topics obtained. Even this approach wouldnt be really great. Since your problem is classification, I would recommend restricting your domain and using only samples for which the label is known. For better classification accuracy &lt;a href=&quot;http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;you can look into CNN&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="836" LastEditDate="2017-02-06T15:43:46.987" LastActivityDate="2017-02-06T15:43:46.987" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16134" PostTypeId="2" ParentId="16131" CreationDate="2017-01-06T07:39:21.140" Score="0" Body="&lt;p&gt;What you are looking for is named entity recommendation. I must tell you this is an extremely tough problem. Stanford has open sourced a NER &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://nlp.stanford.edu/software/CRF-NER.shtml&amp;amp;ved=0ahUKEwiLzc_ygK3RAhXBNY8KHV-RB3QQFggaMAA&amp;amp;usg=AFQjCNENbNmwtXhxEAdBwHrWsA_NeD9hWg&amp;amp;sig2=c1wwmcqWyG8EG7-Hgf3n2g&quot; rel=&quot;nofollow noreferrer&quot;&gt;tagger&lt;/a&gt; but you need to train it on a large amount of data, and you will have to create a tagged dataset. Look into this medium &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://medium.com/%40klintcho/training-a-swedish-ner-model-for-stanford-corenlp-part-1-3e3f281a753a&amp;amp;ved=0ahUKEwjHjouiga3RAhXCRo8KHW3_B2EQFggaMAA&amp;amp;usg=AFQjCNFOzjSqMiOUgXFQlgYS_5Z7K29Lrg&amp;amp;sig2=OhW0VwSAqvTQrN4rE-h3vg&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; for a good tutorial. I would suggest that you look into your requirements, if they are too limited then you dont need something like this. You can work through a simple vocabulary. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-06T07:39:21.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16135" PostTypeId="2" ParentId="16107" CreationDate="2017-01-06T08:48:46.090" Score="0" Body="&lt;p&gt;Yes, 0.5 is the standard cut-off value. It always depends on the business problem what threshold probability you should use to classify the values as 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. - If you are building a fraud model, a person with fraudulent probability of 0.3 and above may make sense to be marked as fraud. Or if you are building some similarity matrix, then a value less than 0.7 could be taken at 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, ROCR curve is helpful to identify the exact cut-off percentage for the specific business problem.   &lt;/p&gt;&#xA;" OwnerUserId="24742" LastActivityDate="2017-01-06T08:48:46.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16136" PostTypeId="2" ParentId="11043" CreationDate="2017-01-06T09:20:27.940" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://conversations.golastmile.com/do-it-yourself-nlp-for-bot-developers-2e2da2817f3d#.15kl6ya20&quot; rel=&quot;nofollow noreferrer&quot;&gt;This post&lt;/a&gt; has an approach. Basically they use bag of words - they convert the words to sparse vectors and then add them up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears to work fairly well but one major flaw is the answer is independent of word order, so you can't do queries like &quot;How many kilos in a pound&quot; unless you special case them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I did text with Alexa and it is fairly insensitive to word order changes so maybe they do use something similar.&lt;/p&gt;&#xA;" OwnerUserId="17618" LastActivityDate="2017-01-06T09:20:27.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16138" PostTypeId="2" ParentId="16131" CreationDate="2017-01-06T10:06:47.810" Score="0" Body="&lt;p&gt;You could look into Rocchio's algorithm, word2vec and other methods that use co-occurence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple starting point would be to query a large docment collection (your collection, the internet or a combination of both) with the query at hand, take the most prominent words in the result (td-idf result/collection) and add that to the original query. The general idea here is that the significant words in the result are words that are closely related to the query. You cn experiment in how far you need the general context (the Internet) or the domain specific context (your collection) and how you weigh the context(how much of the significant words you'll add and what their weight will be), but we can discuss that later.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-06T10:06:47.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16139" PostTypeId="2" ParentId="16131" CreationDate="2017-01-06T10:17:38.540" Score="1" Body="&lt;p&gt;I think these are the methods that you can try out (Please feel free to add more to this list):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Highly precise with a little low recall is to use a dictionary with almost all possibilities (manual effort, but must be worth it.).&lt;/li&gt;&#xA;&lt;li&gt;Using Word2Vec. Mikolov has already trained text data and created word vectors. Using this vector space, you can figure out which words are similar. You can try out and find a threshold above which you can say which words are similar (for example, yoga and exercise would have decent similarity.)&lt;/li&gt;&#xA;&lt;li&gt;Train custom W2V, if you have enough data(This is an unsupervised model, so you don't need to worry about tagging the data but finding huge amounts of data relevant to the working domain.)&lt;/li&gt;&#xA;&lt;li&gt;You can use an RNN to find the most similar words in a corpus and use it for queries. This gives a bit more flexibility than W2V.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="27426" LastActivityDate="2017-01-06T10:17:38.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16140" PostTypeId="1" AcceptedAnswerId="16146" CreationDate="2017-01-06T10:23:05.917" Score="16" ViewCount="35769" Body="&lt;p&gt;Im studying machine learning, and I would like to know how to calculate VC-dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$h(x)=\begin{cases} 1 &amp;amp;\mbox{if } a\leq x \leq b \\ &#xA; 0 &amp;amp; \mbox{else } \end{cases} $&lt;/span&gt;, with parameters &lt;span class=&quot;math-container&quot;&gt;$(a,b) ∈ R^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the VC-dimension of it?&lt;/p&gt;&#xA;" OwnerUserId="27682" LastEditorUserId="29169" LastEditDate="2019-06-08T03:20:06.850" LastActivityDate="2023-07-25T22:12:17.023" Title="How to calculate VC-dimension?" Tags="|machine-learning|classification|vc-theory|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16141" PostTypeId="2" ParentId="16119" CreationDate="2017-01-06T10:36:56.807" Score="2" Body="&lt;p&gt;What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation. &lt;/p&gt;&#xA;" OwnerUserId="24996" LastActivityDate="2017-01-06T10:36:56.807" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16142" PostTypeId="2" ParentId="16119" CreationDate="2017-01-06T12:19:33.963" Score="2" Body="&lt;p&gt;While training, your model will not have the same output when you train with different parts of the dataset.  Cross validation is used to help negate this, by rotating the training and validation sets and training more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your dataset most likely has high variance, given the large jump in accuracy based on different validation sets.  This means that the data is spread out, and can result in overfitting the model.  You can imagine an overfitted model like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ADmhS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ADmhS.png&quot; alt=&quot;Green line is the overfitted model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The green line represents the overfitted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Common techniques to reduce overfitting in random forests is k-fold Cross Validation, with k being between 5 and 10, and growing a larger forest.&lt;/p&gt;&#xA;" OwnerUserId="27551" LastActivityDate="2017-01-06T12:19:33.963" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16143" PostTypeId="1" AcceptedAnswerId="16173" CreationDate="2017-01-06T15:04:25.043" Score="1" ViewCount="806" Body="&lt;p&gt;I have more than 1500 black and white classified images in a training set and I want to create a probabilistic model to classified new images. To be more explicit, given a new black and white image, my model has to predict:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;animal: 84%&#xA;vegetal: 12%&#xA;mineral: 4%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I watched &lt;a href=&quot;https://www.youtube.com/watch?v=iGZpJZhqEME&quot; rel=&quot;nofollow noreferrer&quot;&gt;this lecture&lt;/a&gt; and I still have questions about the procedure to create this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Extraction of the keypoints of each image&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I correctly understood the video, the first step is to extract all the keypoints with SIFT from all the images to create a kind of dictionary of visual words. Each word is described by a multidimensional vector.&lt;br&gt;&#xA;Next, I have to use the $k$-means method to create groups of visual words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1: how many groups should I create?&lt;/strong&gt; Does it exist a rule-of-thumb to determine the $k$ number I have to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Creation of histograms&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for each image, I have to create an histogram/a vector where each features corresponds to one group defined in part 1. The value associated to each feature corresponds to the frequency of the word in the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2: how can I create this vector?&lt;/strong&gt; Indeed, each image is unique and will never match perfectly with the words in my dictionary (words who is, by the way, mean of different words). Who can I bypass this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Creation of the model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I have to create my model. In the video, the lecturer used SVM and has to create one classifier by category (binary classifier). In my case, I have 100 different categories (my introduction was a simplification) and I prefer to have only one classifier. Also, I want to get the probabilities, given an image, to be part of a category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3: is it possible to create only one classifier who gives probabilistic data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To finish, don't hesitate to make suggestions or corrections about the procedure I described if you know a better way to classified my data.&lt;/p&gt;&#xA;" OwnerUserId="27003" LastEditorUserId="27003" LastEditDate="2017-01-14T10:02:54.807" LastActivityDate="2017-01-14T10:02:54.807" Title="What is the procedure to create a bag of visual words model with SIFT?" Tags="|predictive-modeling|image-classification|probability|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16144" PostTypeId="2" ParentId="16140" CreationDate="2017-01-06T15:42:07.827" Score="5" Body="&lt;p&gt;The VC dimension of a classifier is determined the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;VC = 1&#xA;found = False&#xA;while True:&#xA;    for point_distribution in all possible point distributions of VC+1 points:&#xA;        allcorrect = True&#xA;        for classdist in every way the classes could be assigned to the classes:&#xA;            adjust classifier&#xA;            if classifier can't classify everything correct:&#xA;                allcorrect = False&#xA;                break&#xA;        if allcorrect:&#xA;            VC += 1&#xA;            continue&#xA;    break&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So there has only to be one way to place three points such that all possible class distributions among this point-placement can be classified the correct way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't place the three points on a line, the perception gets it right. But there is no way to get the perception classify all possible class distributions of 4 points, no matter how you place the points&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Your example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Your features are in $\mathbb{R}$. Every classifier has at least dimension 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;VC-Dimension 2: It can classify all four situations correctly.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Points: 0 and 42&lt;/li&gt;&#xA;&lt;li&gt;Distributions:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;class(0) = False, class(42) = False =&gt; $a = 1337, b=3141$ classifies this correctly&lt;/li&gt;&#xA;&lt;li&gt;class(0) = False, class(42) = True =&gt; $a = 40, b = 1337$ classifies this correctly&lt;/li&gt;&#xA;&lt;li&gt;class(0) = True, class(42) = False =&gt; $a = -1, b = 1$ classifies this correctly&lt;/li&gt;&#xA;&lt;li&gt;class(0) = True, class(42) = True =&gt; $a = -1, b = 1337$ classifies this correctly.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;VC-Dimension 3: No, that doesn't work. Imagine the classes &lt;code&gt;true&lt;/code&gt; and &lt;code&gt;false&lt;/code&gt; being ordered like &lt;code&gt;True False True&lt;/code&gt;. Your classifier can't deal with that. Hence it has a VC-Dimension of 2.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Proof&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Obviously, the points $x_1, x_2, x_3 \in \mathbb{R}$ can only be distinguished if they have different values. Without loss of generality, we can assume that $x_1 &amp;lt; x_2 &amp;lt; x_3$. Hence the classifier has to be able to classify&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;class($x_1$) = True, class($x_2$) = False, class($x_3$) = True&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;correctly to have VC dimension 3. For $x_1$ to be classified as True,&#xA;$$a \leq x_1 \leq b$$&#xA;is required. For $x_2$ to be False,&#xA;$$x_2 &amp;lt; a \qquad\text{ or }\qquad b &amp;lt; x_2$$&#xA;is required. As $a \leq x_1$ and $x_1 &amp;lt; x_2$, it has to be $b &amp;lt; x_2$. So the situation is currently:&#xA;$$a \leq x_1 \leq b &amp;lt; x_2 &amp;lt; x_3$$&#xA;For $x_3$ to be classified as True, &#xA;$$a \leq x_3 \leq b$$&#xA;is required. But the other constraints already required $b &amp;lt; x_3$. Hence it is not possible to classify all class distributions of any 3 points correctly with this classifier. Hence it does not have VC dimension 3.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-01-07T07:29:27.077" LastActivityDate="2017-01-07T07:29:27.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16146" PostTypeId="2" ParentId="16140" CreationDate="2017-01-06T17:43:05.910" Score="19" Body="&lt;p&gt;The VC dimension is an estimate for the capability of a binary classifier. If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify &lt;em&gt;all&lt;/em&gt; possible $2^n$ labelings correctly) and you &lt;strong&gt;cannot&lt;/strong&gt; find &lt;strong&gt;any&lt;/strong&gt; set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not seperate all points correctly), then the VC dimension is $n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, first consider two points $x_1$ and $x_2$, such that $x_1 &amp;lt; x_2$. Then there are are $2^2=4$ possible labelings &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$x_1:1$, $x_2:1$&lt;/li&gt;&#xA;&lt;li&gt;$x_1:0$, $x_2:0$&lt;/li&gt;&#xA;&lt;li&gt;$x_1:1$, $x_2:0$&lt;/li&gt;&#xA;&lt;li&gt;$x_1:0$, $x_2:1$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All labelings can be achieved through the classifier $h$ by setting the parameters $a&amp;lt;b \in R$ such that&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$a&amp;lt;x_1&amp;lt;x_2&amp;lt;b$&lt;/li&gt;&#xA;&lt;li&gt;$x_1&amp;lt;x_2&amp;lt;a&amp;lt;b$&lt;/li&gt;&#xA;&lt;li&gt;$a&amp;lt;x_1&amp;lt;b&amp;lt;x_2$&lt;/li&gt;&#xA;&lt;li&gt;$x_1&amp;lt;a&amp;lt;x_2&amp;lt;b$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;respectively. (Actually, $x_1 &amp;lt; x_2$ can be assumed w.l.o.g. but it is enough to find one set that can be shattered.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, consider three arbitrary(!) points $x_1$, $x_2$, $x_3$ and w.l.o.g. assume $x_1&amp;lt;x_2&amp;lt;x_3$, then you can't achieve the labeling (1,0,1). As in case 3 above, the labels $x_1$:1 and $x_2$:0 imply $a&amp;lt;x_1&amp;lt;b&amp;lt;x_2$. Which implies $x_3$ &gt; b and therefore the label of $x_3$ has to be 0. Thus, the classifier cannot shatter any set of three points and therefore the VC dimension is 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe it becomes clearer with a more useful classifier. Let's consider hyperplanes (i.e. lines in 2D). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy to find a set of three points that can be classified correctly no matter how they are labeled:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nN0qv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nN0qv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For all $2^3=8$ possible labelings we can find a hyperplane that separates them perfectly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we cannot find any set of 4 points so that we could classify all $2^4=16$ possible labelings correctly. Instead of a formal proof, I try to present a visual argument:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume for now, that the 4 points form a figure with 4 sides. Then it is impossible to find a hyperplane that can separate the points correctly if we label the opposite corners with the same label:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/fCil8.png&quot; width=&quot;400&quot; height=&quot;200&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they don't form a figure with 4 sides, there are two &quot;boundary cases&quot;: The &quot;outer&quot; points must either form a triangle or all form a straight line. In the case of the triangle, it is easy to see that the labeling where the &quot;inner&quot; point (or the point between two corners) is labeled different from the others can't be achieved:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CfDbI.png&quot; width=&quot;200&quot; height=&quot;100&quot;&gt; &#xA;&lt;img src=&quot;https://i.stack.imgur.com/u64PV.png&quot; width=&quot;200&quot; height=&quot;100&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of a line segment, the same idea applies. If the end points are labeled differently than one of the other points, they cannot be separated by a hyperplane. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we covered all possible formations of 4 points in 2D, we can conclude that there are no 4 points that can be shattered. Hence, the VC dimension must be 3.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-01-09T18:54:21.093" LastActivityDate="2017-01-09T18:54:21.093" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16147" PostTypeId="2" ParentId="16128" CreationDate="2017-01-06T17:51:24.710" Score="1" Body="&lt;p&gt;I would assume you want to do this for the cleaning phase of your project. Microsoft Cognitive Services provides a spell check API &lt;a href=&quot;https://www.microsoft.com/cognitive-services/en-us/bing-spell-check-api&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.microsoft.com/cognitive-services/en-us/bing-spell-check-api&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to group words into embedding's, then I might suggest Word2Vec or Glove. Glove is pre-trained and has a different specified number of dimensions for each vector. However, both can potentially account for misspellings as well as similarities.   &lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-06T17:51:24.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16148" PostTypeId="2" ParentId="16115" CreationDate="2017-01-06T18:01:40.803" Score="2" Body="&lt;p&gt;There are many approaches to this type of problem, which would be defined as text classification. You could create a Term Frequency or Term Frequency Inverse Document Frequency matrix based on the tokens (words) in the strings, then use standard machine learning algorithms such as SVM, Random Forest, Gradient Boosted Trees, Logistic Regression, Naive Bayes. You could also potentially use word embedding's and neural networks. In this case, each word would be represented as a vector of similar words. Finally, I have recently been experimenting with character level convolutional neural networks and this seems to work well for user generated text classification. There is link below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-06T18:01:40.803" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16150" PostTypeId="1" CreationDate="2017-01-06T18:42:58.267" Score="1" ViewCount="51" Body="&lt;p&gt;I have a number of domain names that may or may not be related to a particular brand.  For instance, if the brand is UPS, www.upssucks.com, www.upspackagesupplier.com, and www.ihateups.com might all be labeled as &quot;related&quot; because the website content is talking about UPS.  www.ilovepups.com and www.pushupssuck.com aren't related to the website UPS.  I want to use my trained dataset to create a prediction of whether a given domain is related to a brand using only the registered domain name as the input.  It seems like some off-the-shelf classifiers should work, but I am very new to this type of ML project.  What would be the first approach one would take to start making predictions?  I am planning on doing this in Python with scikit learn if that makes any difference.&lt;/p&gt;&#xA;" OwnerUserId="26939" LastActivityDate="2017-01-07T05:21:02.133" Title="What is the general approach I can use to predict whether a domain is related to a brand using a supervised learning algorithm?" Tags="|predictive-modeling|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16152" PostTypeId="2" ParentId="16087" CreationDate="2017-01-06T21:11:31.427" Score="1" Body="&lt;p&gt;What is wrong with simply enumerating them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. 0= apple, 1 = banana, 2 = orange?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By definition of &lt;strong&gt;one hot encoding&lt;/strong&gt;, only one of them &lt;em&gt;may&lt;/em&gt; be set at a time. No need to jugle with bit encoding (i.e. power set) or prime encoding (bag-of-x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But thrmain question is &lt;strong&gt;how do you (plan to) &lt;em&gt;use&lt;/em&gt; the data&lt;/strong&gt;, not so much how you encode it. Sparse data can be stored compactly by compression and sparse formats (which pretty much reduces to above approach). But &lt;strong&gt;the storage must fit you access pattern&lt;/strong&gt; or performance will be bad.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-01-06T21:11:31.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16153" PostTypeId="2" ParentId="352" CreationDate="2017-01-06T21:45:45.060" Score="1" Body="&lt;p&gt;If you are interested in an empirical solution, you might consider the Venneuler package in R. It provides residuals (percentage difference between input intersection area and fitted intersection area). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.rforge.net/doc/packages/venneuler/venneuler.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.rforge.net/doc/packages/venneuler/venneuler.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27702" LastActivityDate="2017-01-06T21:45:45.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16154" PostTypeId="1" AcceptedAnswerId="16200" CreationDate="2017-01-06T22:57:36.107" Score="1" ViewCount="374" Body="&lt;p&gt;When building deep learning models for image analytics-related applications, we sometimes apply various types of operations to enhance the image, such as an image denoising operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my study, we have images generated by physical simulations. In other words, the physical simulations generate matrices of dimensions such as 256*256, which can be visualized as an image as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to apply a deep learning model to perform some analysis over these physics-based images. In the pre-processing steps, I can always apply those image analytics related techniques to pre-process my images, but I am not sure whether it makes sense. For example, denoising or some other contrast enhancement operations can be used to improve the quality of images, like photos. But would it make sense to use them process the images generated by physics simulation?&lt;/p&gt;&#xA;" OwnerUserId="20978" LastEditorUserId="26562" LastEditDate="2017-01-07T14:57:47.257" LastActivityDate="2017-02-08T09:59:34.123" Title="the feasibility of image processing techniques for physics based images" Tags="|machine-learning|deep-learning|image-recognition|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16155" PostTypeId="1" CreationDate="2017-01-07T02:44:48.257" Score="2" ViewCount="3660" Body="&lt;p&gt;Here's what my data frame look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Number  Age      Famous_for                            &#xA;1       35       &quot;businessman chairman of IBM (1973–1981)&quot;&#xA;2       42       &quot;musician (House of Freaks Gutterball)&quot;&#xA;3       87       &quot;baseball player (Oakland Athletics)&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to create additional column that will contain a dummy variable whether or not person was involved with entertaining business. Something like that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Number..Age..................Famous_for.....................Entertaining                            &#xA;1.......35.......businessman chairman of IBM (1973–1981)........0              &#xA;2.......42.......musician (House of Freaks Gutterball)..........1                &#xA;3.......87.......baseball player (Oakland Athletics)............0       &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I create a column based on the certain words in the famous for column (such as &quot;musician&quot;, &quot;club&quot;, &quot;actor&quot; and etc)?&#xA;I tried the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['entertaining'] = np.where(df['famous_for']&amp;gt;=&quot;musician&quot;, 1, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but it wouldn't work.&#xA;How can I do this in Python?&lt;/p&gt;&#xA;" OwnerUserId="27706" LastEditorUserId="20492" LastEditDate="2017-01-08T05:50:18.933" LastActivityDate="2017-01-08T05:50:18.933" Title="How do I add a column to dataframe on condition that certain observations in my dataframe contain a target word in Python?" Tags="|python|data-mining|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16156" PostTypeId="1" CreationDate="2017-01-07T03:52:18.910" Score="1" ViewCount="561" Body="&lt;p&gt;Consider two binary classifiers A and B. Suppose that both A and B are predicting the same target, but that A is trained on a subset of the data for which a different set of features is available than for B. Suppose further that A and B output probability esitmates and that for a set of mixed input types (some with features that A can accept, others for B) I need to rank them by probability of being positive.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps (and this is completely fabricated) I'm trying to predict early mortality of lions and tigers in a zoo. Different features are available for early lion mortality than for early tiger mortality. However, I ultimately will need to order all of the big cats, (among both lions AND tigers) by likelihood of early mortality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My ideas so far:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Train a meta-estimator that accepts the input type (belonging to A or B) and the scores from A and B. I'm thinking xgboost or another tree-based regressor.&lt;/li&gt;&#xA;&lt;li&gt;Attempt to directly and monotonically transform the probability estimates from A and B so as to 'bring them into alignment' in such a way as to maximize the cross-validated AUC. This seems to be similar to the first option.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other information:&#xA; In my actual application there is class imbalance, and the class imbalance varies between A type datapoints and B type datapoints. Precision TPR/(TPR+FPR) is most important in a business context. &lt;/p&gt;&#xA;" OwnerUserId="23633" LastActivityDate="2017-01-07T03:52:18.910" Title="Compare probability estimates of two classifiers" Tags="|classification|ensemble-modeling|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16158" PostTypeId="2" ParentId="16150" CreationDate="2017-01-07T05:21:02.133" Score="1" Body="&lt;p&gt;It will be almost impossible to do this your way as you arw trying to derive context based solely on the domain names, and domain names more than often don't carry a specific context. Since the word ups occurs in each domain and the words coming before or after hardly carry much specific information. If you could extract some information about the website by hitting the url, you will not need any machine learning at all. I am not saying this can't be done through ML, all I am saying it won't be that effective. If you use scikit learn you will need to find a way to tokenize the domain, in your case a character level tokenization will be useful, throw in a BOW (Bag of words, characters in your case) create one-hot vectors out of it and use them as features to predict the label. If you have a lot and I mean lots you can train a character level convolutional neural network and it might perform better. Cheers. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-07T05:21:02.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16159" PostTypeId="1" CreationDate="2017-01-07T06:55:08.723" Score="0" ViewCount="5240" Body="&lt;p&gt;I want to test a few devices as pure black box tests. That would mean some kind of robotics to press buttons and the touch screen and detecting screen elements like buttons in varying light conditions with a video camera. I can handle the mechanics but I am not sure how to got about screen detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would machine learning be suitable for such a task? I imagine training the system under different lighting conditions.  &lt;/p&gt;&#xA;" OwnerUserId="27710" LastActivityDate="2020-05-05T04:47:04.810" Title="Is machine learning suitable for detecting screen elements?" Tags="|image-recognition|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16160" PostTypeId="2" ParentId="16122" CreationDate="2017-01-07T07:12:59.113" Score="1" Body="&lt;p&gt;There is a lot of research going on in this field of generating fixed length sentence vectors. Here are some of the ways.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://code.google.com/archive/p/word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Train/Download vectors&lt;/a&gt;.&#xA;Add the word_vectors in the sentence to get the fixed length sentence vector. Surprisingly it gives decent results.&lt;/li&gt;&#xA;&lt;li&gt;If you have a huge corpus. One popular algorithm and used for various baselines is &amp;quot;Quoc Le &amp;amp; Tomáš Mikolov: “Distributed Representations of Sentences and Documents”. The gensim implementation can be found &lt;a href=&quot;https://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;You may also download pretrained sentence vectors. Here are some of the &lt;a href=&quot;https://github.com/jhlau/doc2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;sources&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="25174" LastEditorUserId="98307" LastEditDate="2020-08-08T08:58:16.477" LastActivityDate="2020-08-08T08:58:16.477" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16161" PostTypeId="2" ParentId="16155" CreationDate="2017-01-07T08:51:37.303" Score="1" Body="&lt;p&gt;You can use str.contains on the DataFrame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame({'Age' : pd.Series([35,42,87], index=[1,2,3]),&#xA;        'Famous_for': pd.Series(['businessman chairman of IBM (1973–1981)',&#xA;                                 'musician (House of Freaks Gutterball)',&#xA;                                 'baseball player (Oakland Athletics)'], &#xA;        index=[1,2,3])&#xA;        })&#xA;&#xA;df['entertaining'] = df['Famous_for'].str.contains('musician')&#xA;print (df)&#xA;&#xA;   Age                               Famous_for entertaining&#xA;1   35  businessman chairman of IBM (1973–1981)        False&#xA;2   42    musician (House of Freaks Gutterball)         True&#xA;3   87      baseball player (Oakland Athletics)        False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that str.contains also accepts a regular expression, so if you wish to search for various words you might use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['entertaining'] = df['Famous_for'].str.contains('musician|businessman')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27712" LastActivityDate="2017-01-07T08:51:37.303" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16163" PostTypeId="1" CreationDate="2017-01-07T10:34:43.093" Score="1" ViewCount="5931" Body="&lt;p&gt;I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally I tune the n_rounds parameters by cross-validation, but what if you have too less observations to do proper CV? For example if I have 30 variables and 4000 observations in my training data, how can I find a nice value for n_round which is not over/underfitting the training data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any &quot;best practices&quot; for parameter tuning (also max_depth etc.) having small datasets?&lt;/p&gt;&#xA;" OwnerUserId="27714" LastActivityDate="2017-02-21T21:51:16.657" Title="XGboost classification with very small data set" Tags="|r|xgboost|hyperparameter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16164" PostTypeId="1" CreationDate="2017-01-07T11:46:41.237" Score="1" ViewCount="91" Body="&lt;p&gt;I just wonder since the manifold learning under scikit-learn has component of graph-based transformation (e.g. Shortest-path graph search under Isomap) I can then transform the feature data set (i.e. measurements of a chemical physical attributes to decide its toxicity) to something that will measure the graph-based measurements.&lt;/p&gt;&#xA;" OwnerUserId="10915" LastActivityDate="2017-01-07T11:46:41.237" Title="Can I use manifold learning to transform the feature set as a substitute of graph kernel of SVC" Tags="|python|scikit-learn|graphical-model|manifold|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16165" PostTypeId="1" AcceptedAnswerId="16171" CreationDate="2017-01-07T16:17:32.300" Score="2" ViewCount="603" Body="&lt;p&gt;I am currently wondering if the following graphic of &lt;a href=&quot;https://arxiv.org/abs/1512.03385&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep residual networks&lt;/a&gt; is wrong:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/msvse.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/msvse.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say the graphic describes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the text, they define&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathcal{F}(x) := \mathcal{H}(x) - x$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathcal{H}(x)$ is &quot;the desired underlying mapping&quot; (whatever that exactly means).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, equation (1) seems strange to me:&#xA;$$y = \mathcal{F}(x, \{W_i\}) + x$$&#xA;In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I think the authors could mean&#xA;$$\mathcal{F}_i = \varphi(W_i x)$$&#xA;In that case, in the image where $\mathcal{F}(x)$ is it should be&#xA;$$\mathcal{F}_1(x) = \varphi(W_1 x)$$&#xA;and where $\mathcal{F}(x) + x$ is should be&#xA;$$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-01-07T21:31:01.413" Title="Is the graphic of deep residual networks wrong?" Tags="|neural-network|deep-learning|computer-vision|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16168" PostTypeId="2" ParentId="16165" CreationDate="2017-01-07T19:19:56.747" Score="0" Body="&lt;p&gt;(edited) answer of &lt;a href=&quot;https://www.reddit.com/user/mostly_reasonable&quot; rel=&quot;nofollow noreferrer&quot;&gt;/u/mostly_reasonable&lt;/a&gt; on &lt;a href=&quot;https://www.reddit.com/r/MLQuestions/comments/5ml72j/is_the_graphic_of_deep_residual_networks_wrong/dc4frg3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;reddit&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing to note here is that $F(x)$ can refer to the functioning of more than one layer.&#xA;The paper's authors use '$H(x)$' to mean something like 'the function we want to learn in some (possibly more than one) consecutive layers of a neural network', see their statement&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] hoping each few stacked layers directly [...]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then '$F(x)$' is then that same possibly multi layer function, minus the residuals. The author of course hypothesizes that $F(x)$ is easier to learn than $H(x)$.&#xA;So I think that in the figure the $F(x)$ is supposed to refer to everything in the Figure besides the ('$+ x$') part. Note how the F(x) symbol is centered with respect the the network, not attached to either layer. Then $F(x) + x$ references the entire $F(x)$ two layer network above combined with the skip connections.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-01-07T19:19:56.747" CommentCount="0" CommunityOwnedDate="2017-01-07T19:19:56.747" ContentLicense="CC BY-SA 3.0" />
  <row Id="16169" PostTypeId="2" ParentId="15662" CreationDate="2017-01-07T20:34:00.593" Score="2" Body="&lt;p&gt;Make sure that while training, you are passing the data in a random order. It looks like you are passing all images of one class at a time then the next class etc. Shuffle the images and pass it in a random manner.&lt;/p&gt;&#xA;" OwnerUserId="26050" LastActivityDate="2017-01-07T20:34:00.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16171" PostTypeId="2" ParentId="16165" CreationDate="2017-01-07T21:31:01.413" Score="3" Body="&lt;p&gt;(Per the diagram), $F(x)$ here is simply the &lt;em&gt;entire&lt;/em&gt; two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an &lt;em&gt;entire&lt;/em&gt; deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation. &lt;/p&gt;&#xA;" OwnerUserId="599" LastActivityDate="2017-01-07T21:31:01.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16172" PostTypeId="2" ParentId="15662" CreationDate="2017-01-07T22:01:07.930" Score="1" Body="&lt;p&gt;There are a bazillion things that can go wrong when training a DNN, and it's often helpful to heavily simplify your starter net/data before you move onto your actual problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without any more information on the specifics of your setup, etc, the first thing I would recommend for you to try is to overfit your net to random noise. Don't make it deep just yet! You want to make sure that everything else if your setup is kosher first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: Create a random-noise data set with labels, OR extract a small (&amp;lt; 50) or so images for you to use as your dev set, and try to overfit on those in the very beginning, with a vastly simplified net. This should always be your first step. &lt;/p&gt;&#xA;" OwnerUserId="599" LastActivityDate="2017-01-07T22:01:07.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16173" PostTypeId="2" ParentId="16143" CreationDate="2017-01-07T23:44:40.007" Score="1" Body="&lt;p&gt;&lt;strong&gt;Question 1: how many groups should I create?&lt;/strong&gt;&#xA;There is no rule of thumb but typically if you have n categories of data you may want to set k as 10*n. This has known to work well, however you can always change this number by looking at the accuracy on your validation data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2: Creation of histograms&lt;/strong&gt;&#xA;Let us say you extract m features from each image and your dictionary contains k words, for each of these m features you would find the closest match to the k words and assign this word to that particular feature. This way you will have m number of visual words taking one of the k values. Now you count the occurrence of these visual words and creating a normalized histogram out of it. So each image will have a normalized histogram of the k visual words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3: Classification&lt;/strong&gt;&#xA;Now that we know how to extract a histogram for a given image, we would compute the histograms on the training data and given a test image we extract its histogram. Now, how we classify this is entirely up to you. You could just find the closest match in the histograms of training data using euclidian distance or do a weighted sum etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple way of obtaining probabilistic data is to train a neural network with softmax at the output layer and given a test image pass its histogram through the network and obtain the probabilities. &lt;/p&gt;&#xA;" OwnerUserId="26050" LastEditorUserId="27003" LastEditDate="2017-01-08T22:16:11.840" LastActivityDate="2017-01-08T22:16:11.840" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16174" PostTypeId="1" AcceptedAnswerId="25012" CreationDate="2017-01-08T04:04:39.287" Score="6" ViewCount="3961" Body="&lt;p&gt;There are 2 parts to this question. Suppose we are looking at sales $S$ of a product across $&amp;gt; 1000$ stores where a it sells. For each of these $1000$ stores we have 24 months recorded data. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We want to be able to predict $S_t \leftarrow f(S_{t-1})$. We could build a RNN for each of the store time series, calculate test RMSE and then take an average after taking care of normalizing values etc. But the problem is there are very few samples per time series. If we were to segment stores into groups (by say Dynamic Time Warping) then could we create a monologue of text sentiment mining where like in text two sentences are separated by a dot here we would have two time series separated by a special symbol (let's say). In that case, we would generate a RNN model on &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$$Train_1 | Train_2 |\ldots|Train_t$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;data and predict on &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Test_1 | Test_2 |\ldots|Test_t$$&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;After this, we would like to set it up as a panel data problem where $S_t\leftarrow f(x_{t1},x_{t2},\ldots,x_{tn})$. In that case should I build a separate neural network for each $t$ and then connect the hidden layers from $t \rightarrow t+1 \rightarrow t+2 \ldots$. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How should I implement these through packages like Keras/Theano/Mxnet etc.? Any help would be great! &lt;/p&gt;&#xA;" OwnerUserId="27735" LastActivityDate="2017-11-22T14:09:06.920" Title="Recurrent Neural Network on Panel Data" Tags="|neural-network|time-series|keras|rnn|theano|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16175" PostTypeId="1" CreationDate="2017-01-08T06:59:37.690" Score="2" ViewCount="158" Body="&lt;p&gt;As the title indicates, over multiple runs of my program, I get different values of trans and est. Are these local minimum? If so how do I get the optimal one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE - They all have the same starting value of trans and est.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I realized why I was getting different values of trans and est over different runs but it raised another question in my mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My objective is to train an HMM on classical song data (MFCC coefficients). To discretize it, I assigned the MFCC coefficients to 7 clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a sequence of 10 observations stored in matrix M that I am providing to the hmmtrain function as input. Each row represents a song, and each column are the different clusters the MFCC coefficients (for different frames of the song) were assigned to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M =&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 1     1     1     3     3     3     3     2     7     7     7&#xA; 1     1     1     3     1     1     1     1     3     1     1&#xA; 3     3     3     2     2     3     3     2     2     3     3&#xA; 6     6     6     2     2     5     5     2     2     5     5&#xA; 3     3     3     3     1     1     1     1     1     1     1&#xA; 7     7     7     7     5     5     5     7     2     5     5&#xA; 3     3     3     2     2     3     3     2     2     6     6&#xA; 7     7     5     7     2     3     6     2     2     2     2&#xA; 7     7     7     7     7     7     7     7     7     7     2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, since I was recomputing the clustering every time, my input data would change. By saving the matrix to a file and using that every time my results were consistent. What I do not understand is why for certain formations of clusters (i.e. for certain input matrices) I would get better performance? Shouldn't cluster formation be the same by and large (even if the number ex. 1,2.. might represent a different cluster)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry about earlier. I'll repost the question if necessary. Thanks again!&lt;/p&gt;&#xA;" OwnerUserId="21749" LastEditorUserId="21749" LastEditDate="2017-01-08T17:37:30.257" LastActivityDate="2017-01-08T17:37:30.257" Title="hmmtrain in Matlab converging to different values of trans and est over different runs" Tags="|matlab|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16176" PostTypeId="1" CreationDate="2017-01-08T09:41:43.147" Score="1" ViewCount="761" Body="&lt;p&gt;Does logistic regression only solve binary classification problems?&#xA;How can logistic regression solve multiple-class problems?&lt;/p&gt;&#xA;" OwnerUserId="27682" LastEditorUserId="27682" LastEditDate="2017-01-09T00:55:49.880" LastActivityDate="2017-01-09T00:55:49.880" Title="Does logistic regression can only solve binary classification problem?" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16177" PostTypeId="1" CreationDate="2017-01-08T09:45:10.323" Score="4" ViewCount="1725" Body="&lt;p&gt;Given the experience on MIST, I try this problem as a character level.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a handwritten text and I want to &quot;OCR&quot; it. Even though I made progresses with openCV (on the image pre-processing, before a DNN classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that the best approach would be to move to word level (into RNN). I am writing this in Python. But I don't know how to change the last layer of the DNN into an input layer for RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any code for dummies to see how is this done (for images as input of the first DNN layer)? Besides I see that you could put a language model on top (this would be a dictionary right?). Advice?&lt;/p&gt;&#xA;" OwnerUserId="27740" LastEditorUserId="28175" LastEditDate="2018-03-18T15:30:53.577" LastActivityDate="2020-08-13T13:56:58.130" Title="Handwritting Recognition moving from character level to word level" Tags="|machine-learning|deep-learning|rnn|lstm|cnn|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16178" PostTypeId="2" ParentId="16176" CreationDate="2017-01-08T09:59:27.397" Score="5" Body="&lt;p&gt;No, multiclass classification is also possible. Try reading up on 'One vs All' multiclass classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Multiclass_classification&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiclass, One vs All classification in the case of logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/68Pol/multiclass-classification-one-vs-all&quot; rel=&quot;noreferrer&quot;&gt;https://www.coursera.org/learn/machine-learning/lecture/68Pol/multiclass-classification-one-vs-all&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21749" LastActivityDate="2017-01-08T09:59:27.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16179" PostTypeId="1" AcceptedAnswerId="16443" CreationDate="2017-01-08T09:59:50.133" Score="9" ViewCount="13714" Body="&lt;p&gt;I have a set of 10 experiments that compute precision, recall and f1-score for each experiment.&#xA;Now, average precision &amp;amp; average recall is easy to compute. I have some confusion regarding average f1-score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 ways on how i can compute mean f1-score:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take f1 scores for each of the 10 experiments and compute their average. &lt;/li&gt;&#xA;&lt;li&gt;Take average precision &amp;amp; average recall and then compute f1-score using the formula &lt;code&gt;f1 = 2*p*r/(p+r)&lt;/code&gt;  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I could not find any strong reference to support any of the arguments. The closest document i could find is this: &lt;a href=&quot;https://www.kaggle.com/wiki/MeanFScore&quot; rel=&quot;noreferrer&quot;&gt;https://www.kaggle.com/wiki/MeanFScore&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain with some reference (if possible) which of the methods is correct and why? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; One of the members suggested &lt;a href=&quot;http://metaoptimize.com/qa/questions/8284/does-precision-equal-to-recall-for-micro-averaging&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; source. Though, i still suspect the reliability of the source. I have seen people not using the method explained above in their research publications. (Even i would be using it in one of my publications) I would expect some more opinions from the community to verify this idea.&lt;/p&gt;&#xA;" OwnerUserId="27741" LastEditorUserId="27741" LastEditDate="2017-01-12T18:12:34.180" LastActivityDate="2018-01-01T12:32:16.917" Title="What is the correct way to compute Mean F1 score?" Tags="|parameter|" AnswerCount="3" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="16182" PostTypeId="1" CreationDate="2017-01-08T11:14:38.313" Score="8" ViewCount="13370" Body="&lt;p&gt;I'm trying to predict movie genres using a neural network. I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, how would I have to format my data to make it work with Keras?&lt;/p&gt;&#xA;" OwnerUserId="26214" LastEditorUserId="28175" LastEditDate="2018-01-17T14:27:45.693" LastActivityDate="2018-01-17T14:46:25.190" Title="Multiple output classes in keras" Tags="|machine-learning|neural-network|deep-learning|keras|multiclass-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16183" PostTypeId="1" CreationDate="2017-01-08T12:59:05.660" Score="1" ViewCount="355" Body="&lt;p&gt;In the context of collecting disparate data sets holding similar information, are their examples of algorithms being able to resolve attributes of records being similar (while their values are different) based on their relation to other attributes? (So fuzzy matching would not necessarily apply)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplistic example: 2 Datasets regarding vehicle test data (perhaps with headings in different languages)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Source A: Metric information: Car, Distance tested (km), Fuel Used (l), Consumption (km/l)&lt;/li&gt;&#xA;&lt;li&gt;Source B: Imperial information: Car, Distance tested (miles), Fuel Used (gallons), Consumption (mpg)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Goal:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mapping between A &amp;amp; B, which recognises that the consumption data in different units, is actually the same because they are simply &quot;column 2 / column 3&quot; ?&lt;/li&gt;&#xA;&lt;li&gt;Could it identify that the distance columns were holding similar information with a 1.6x scaling factor applied etc?&lt;/li&gt;&#xA;&lt;li&gt;In this hypothetical example, matching could apply on column 1 to identify cars for record linkage, however I'm curious to see if there are methods which can identify the relationships between attributes, and recognise where attributes are similarly linked in other sets.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The closest material I have found seems to be &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67.9222&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;test&quot;&gt;this paper&lt;/a&gt; however I'm under the impression that this is probably a well investigated problem where I am not aware of the correct avenue to search!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your input!&lt;/p&gt;&#xA;" OwnerUserId="27743" LastActivityDate="2017-01-08T12:59:05.660" Title="Is there a parallel to record linkage/entity resolution where ML can be applied to the data for schema matching?" Tags="|machine-learning|data-mining|dataset|similarity|fuzzy-logic|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16184" PostTypeId="1" CreationDate="2017-01-08T14:36:34.430" Score="3" ViewCount="123" Body="&lt;p&gt;Reading the &lt;a href=&quot;https://arxiv.org/pdf/1311.2901v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zeiler&amp;amp;Fergus paper&lt;/a&gt; (&lt;a href=&quot;http://www.shortscience.org/paper?bibtexKey=journals/corr/ZeilerF13#martinthoma&quot; rel=&quot;nofollow noreferrer&quot;&gt;my summary&lt;/a&gt;), I wonder how exactly they trained the deconv net. What was their data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think for one CNN which they want to analyze, they train exactly one deconv net (in contrast to training one deconv net per layer). The featuers (inputs) of the deconv net are the activations of the layer they want to analyze. The output they train them on are the activations that actually was the input of the layer they want to analyze. So although they have one deconv-net in total, they train it layer-wise. So for each training run, the weights of only one deconv layer are adjusted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I wonder why the images look that unrealistic:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hCqz2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hCqz2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it gray because MSE is the training objective? Why aren't the first layer filter outputs gray then, too?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:25:19.027" LastActivityDate="2021-02-16T15:25:19.027" Title="Why do the deconv outputs of layer &gt;= 2 of Zeiler&amp;Fergus look so unrealistic?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16186" PostTypeId="1" AcceptedAnswerId="16260" CreationDate="2017-01-08T17:22:36.960" Score="2" ViewCount="1954" Body="&lt;p&gt;I have a model that predicts the outcome of ATP tennis matches. The quality of predictions varies, and I want to develop a second binary classification model that optimises the decision to bet (or not) based on a number of features about the match. Amongst the second model's features are the &lt;strong&gt;probabilities&lt;/strong&gt; from the first, and archived bookmaker's &lt;strong&gt;odds&lt;/strong&gt; for each match. The size of the bet is determined by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kelly_criterion&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kelly Criterion&lt;/a&gt;. The training data has been classified such that all bets that would have won = 1, all losing bets = 0. I'm using &lt;strong&gt;xgboost&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm attempting to incorporate the Kelly Criterion into my xgb loss function but without success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've had a look at the custom objective example in the xgb demo. From my understanding, for xgb to &lt;strong&gt;maximise the expected value of the logarithmic bankroll&lt;/strong&gt;, my objective function needs to return the first:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial}{\partial x}(p\: log(1 + bx)+(1-p)\:log(1-x))=\frac{-(b+1)\:p+b\:x+1}{(x-1)(b\:x+1)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and second order derivatives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial ^2}{\partial x^2}(p\: log(1 + bx)+(1-p)\:log(1-x))=-\frac{b^{2}p}{(bx+1)^{2}}-\frac{1-p}{(1-x)^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;b is the net odds received on the wager (&quot;b to 1&quot;); that is, you could win \$b (on top of getting back your \$1 wagered) for a $1 bet&lt;/li&gt;&#xA;&lt;li&gt;p is the probability of winning;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've also implemented my own cost function that calculates the profit and loss for each bet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code I have so far is below.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import xgboost as xgb&#xA;import numpy as np&#xA;import StringIO #  ('import io' in python 3.x)&#xA;import requests&#xA;&#xA;&#xA;url_train = 'https://gist.githubusercontent.com/martinstaniforth/162b9691132f7099b4da08fd14defc39/raw/9372c5cac42b545ecde4200503b97f895e24cbfe/train.csv'&#xA;url_test = 'https://gist.githubusercontent.com/martinstaniforth/4445b884abea22d4ae238cda869b5e0e/raw/84d3dead9122a644f8d273e04b898920a8e5a811/test.csv'&#xA;&#xA;train_content = requests.get(url_train).content&#xA;test_content = requests.get(url_test).content&#xA;&#xA;train_df = pd.read_csv(StringIO.StringIO(train_content.decode('utf-8')), index_col='match_id')&#xA;test_df = pd.read_csv(StringIO.StringIO(test_content.decode('utf-8')), index_col='match_id')&#xA;&#xA;train_target_df = train_df.reset_index(drop=True)[['bet_wins']]&#xA;train_df = train_df.reset_index(drop=True).drop(['bet_wins'], axis=1)&#xA;&#xA;test_target_df = test_df.reset_index(drop=True)[['bet_wins']]&#xA;test_df = test_df.reset_index(drop=True).drop(['bet_wins'], axis=1)&#xA;&#xA;odds_train = train_df['player_odds'].values - 1&#xA;probs_train = train_df['win_prob'].values&#xA;&#xA;odds_test = test_df['player_odds'].values - 1&#xA;probs_test = test_df['win_prob'].values&#xA;&#xA;dtrain = xgb.DMatrix(train_df.values, train_target_df.values)&#xA;dtest = xgb.DMatrix(test_df.values, test_target_df.values)&#xA;&#xA;param = {&#xA;    'max_depth': 3,&#xA;    'eta': 0.05,&#xA;    'silent': 1,&#xA;    'n_estimators': 50,&#xA;    'seed': 366}&#xA;watchlist = [(dtest, 'eval'), (dtrain, 'train')]&#xA;num_round = 200&#xA;&#xA;&#xA;def kelly_loss(odds_train, probs_train, odds_test, probs_test):&#xA;    def logregobj(x, dmatrix):&#xA;        bet_outcome = dmatrix.get_label()&#xA;&#xA;        odds = odds_train if len(bet_outcome) == len(odds_train) else odds_test&#xA;        probs = probs_train if len(bet_outcome) == len(probs_train) else probs_test&#xA;&#xA;        y = -((odds + 1) * probs + odds * x + 1) / ((x - 1) * (odds * x + 1))&#xA;        grad = y - bet_outcome&#xA;        hess = -(np.power(odds, 2) * probs) / np.power(odds * x + 1, 2) - \&#xA;                (1 - probs) / np.power(1 - x, 2)&#xA;&#xA;        return grad, hess&#xA;&#xA;    return logregobj&#xA;&#xA;&#xA;def kelly_error(odds_train, probs_train, odds_test, probs_test):&#xA;    def evalerror(preds, dmatrix):&#xA;        bet_outcome = dmatrix.get_label()&#xA;&#xA;        odds = odds_train if len(bet_outcome) == len(odds_train) else odds_test&#xA;        probs = probs_train if len(bet_outcome) == len(probs_train) else probs_test&#xA;&#xA;        kelly_fraction = (probs * (odds + 1) - 1) / odds&#xA;&#xA;        def value_bets(f):  # ignore any bets with a negative kelly fraction&#xA;            return 0 if f &amp;lt; 0 else f&#xA;&#xA;        kelly_fraction = np.array([value_bets(x) for x in kelly_fraction])&#xA;&#xA;        profit = preds * kelly_fraction * odds&#xA;        loss = (1 - preds) * kelly_fraction&#xA;&#xA;        total_profit = float(sum(profit - loss))&#xA;&#xA;        return 'error', total_profit&#xA;&#xA;    return evalerror&#xA;&#xA;bst = xgb.train(&#xA;    param,&#xA;    dtrain,&#xA;    num_round,&#xA;    watchlist,&#xA;    kelly_loss(odds_train, probs_train, odds_test, probs_test),&#xA;    kelly_error(odds_train, probs_train, odds_test, probs_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I execute the code, xgb does not update predictions.  I suspect the &lt;strong&gt;logregobj&lt;/strong&gt; function is incorrect as I haven't fully understood its purpose. Can someone please assist in correctly implementing the &lt;strong&gt;Kelly Criterion&lt;/strong&gt; in a binary classification model?  Training data is available in this &lt;a href=&quot;https://gist.github.com/martinstaniforth/162b9691132f7099b4da08fd14defc39&quot; rel=&quot;nofollow noreferrer&quot;&gt;gist&lt;/a&gt;, as referenced in the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="27724" LastActivityDate="2019-06-27T21:29:34.400" Title="Kelly Criterion in xgboost loss function" Tags="|python|optimization|xgboost|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16187" PostTypeId="1" CreationDate="2017-01-08T18:15:42.087" Score="2" ViewCount="163" Body="&lt;p&gt;I've just been reading&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Zeiler, M.D. and Fergus, R., 2014, September. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (pp. 818-833). Springer International Publishing. (&lt;a href=&quot;https://arxiv.org/abs/1311.2901&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;, &lt;a href=&quot;http://www.shortscience.org/paper?bibtexKey=journals/corr/1512.02017#martinthoma&quot; rel=&quot;nofollow noreferrer&quot;&gt;my summary&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and (partially)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mahendran, A. and Vedaldi, A., 2016. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, pp.1-23. (&lt;a href=&quot;https://arxiv.org/abs/1512.02017v3&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;, &lt;a href=&quot;http://www.shortscience.org/paper?bibtexKey=journals/corr/1512.02017#martinthoma&quot; rel=&quot;nofollow noreferrer&quot;&gt;my summary&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They are both about visualizing features being learned in CNNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although both papers have an introduction where you can read things like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;our understanding of [CNN features] remains limited&lt;/li&gt;&#xA;&lt;li&gt;While the performance of representations has been improving significantly in the past few years, their design remains eminently empirical&lt;/li&gt;&#xA;&lt;li&gt;In this paper, with the aim of obtaining a better understanding of representations, we develop a family of methods to investigate CNNs and other image features by means of visualizations&lt;/li&gt;&#xA;&lt;li&gt;there is no clear understanding of why they perform so well, or how they might be improved&lt;/li&gt;&#xA;&lt;li&gt;there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Although I like the images, I don't see how these methods are better than simply pushing all images through the network and showing the top-$n$ images which activate the neuron of interest most. Was this evaluated? Did the authors have any insights into the features which other authors didn't have before / without those techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The Zeiler&amp;amp;Fergus paper at least added the occlusion sensitivity analysis which does help. However, a big part of the paper is this filter visualization by deconv-nets. And I don't see how this helps to address any of the points mentioned above)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:25:07.583" LastActivityDate="2021-02-16T15:25:07.583" Title="What are natural (computed) pre-images useful for?" Tags="|image-classification|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16188" PostTypeId="2" ParentId="15984" CreationDate="2017-01-08T19:17:17.057" Score="0" Body="&lt;p&gt;I feel awkward for asking this question because all I had to do was add a Threshold to the prediction i.e. after training the model to recognise the objects in the picture ( I used faster-rcnn to find multiple objects). I just had to make a threshold such that, only if the model is at least 50% sure that the object is a car then a car exists in the picture. &lt;/p&gt;&#xA;" OwnerUserId="27450" LastActivityDate="2017-01-08T19:17:17.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16190" PostTypeId="1" CreationDate="2017-01-08T22:36:52.447" Score="2" ViewCount="219" Body="&lt;p&gt;It really intrigues me what is the time complexity of learning phase of Hopfield neural network, how it depends of the number of training examples and number of attributes?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://code.activestate.com/recipes/577461-hopfield-artificial-neural-network/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source code of Hopfield neural network&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Can we say that time complexity of Hopfield neural network quadratically depend on the number of training examples or quadratically depend on the number of attributes?&lt;/p&gt;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="27754" LastEditorUserId="1330" LastEditDate="2022-03-28T18:41:46.890" LastActivityDate="2022-03-28T18:41:46.890" Title="What is the time complexity of learninig phase of Hopfield neural network?" Tags="|machine-learning|neural-network|data-mining|time-complexity|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16191" PostTypeId="1" CreationDate="2017-01-09T01:02:22.160" Score="1" ViewCount="448" Body="&lt;p&gt;Edit: I found a solution to my problem; there's a scipy installer that doesn't require openBLAS, and that seems to have worked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to find a neural network library for Python, but nothing I've tried has been what I've wanted; the libraries a lot of people seem to be recommending (e.g Keras, scikit-learn) require scipy, which won't actually install on my computer.  (I think it requires openBLAS, which doesn't seem to have a tutorial for installation.)  I did find a program that looks like it will work, but it has a maximum of one hidden layer.  Can someone point me to a library that actually works, or show me how to install scipy?&lt;/p&gt;&#xA;" OwnerUserId="27756" LastEditorUserId="27756" LastEditDate="2017-01-09T18:34:16.073" LastActivityDate="2019-12-25T09:01:50.247" Title="Python Neural Network library/program with simple installation? (Solved?)" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="2" ClosedDate="2019-12-27T02:58:36.273" ContentLicense="CC BY-SA 3.0" />
  <row Id="16192" PostTypeId="2" ParentId="11159" CreationDate="2017-01-09T02:56:35.973" Score="1" Body="&lt;p&gt;Sharing what sort of outcome you are most interested will be helpful in directing you towards a proper answer. However, consider this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested, in exploring the likelihood of will win or lose think &lt;strong&gt;classification&lt;/strong&gt; algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Linear&lt;/em&gt; Machine Learning Algorithms&#xA; - Logistic regression&#xA; - Linear Discriminant Analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Non-linear&lt;/em&gt; Machine Learning Algorithms&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;K-nearest neighbors&lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes&lt;/li&gt;&#xA;&lt;li&gt;Support Vector Machines&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are interested in exploring the scores of each team in a given game think &lt;strong&gt;regression&lt;/strong&gt; algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Linear regression&lt;/li&gt;&#xA;&lt;li&gt;Logistic regression&lt;/li&gt;&#xA;&lt;li&gt;Stepwise regression&lt;/li&gt;&#xA;&lt;li&gt;Multivariate Adaptive Regression Splines (MARS)&lt;/li&gt;&#xA;&lt;li&gt;Locally estimated Scatterplot Smoothing (LOESS)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In my judgement, giving us more direction on your desired outcome we can give you more direction with a solid answer.&lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2017-01-09T02:56:35.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16193" PostTypeId="2" ParentId="11721" CreationDate="2017-01-09T03:03:48.817" Score="2" Body="&lt;p&gt;Take a look at the &lt;a href=&quot;https://www.bjs.gov/rawdata.cfm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bureau of Justice Statistics&lt;/a&gt; site. Other sites that you might find interesting would be &lt;a href=&quot;http://us-city.census.okfn.org/dataset/crime-stats&quot; rel=&quot;nofollow noreferrer&quot;&gt;U.S. City Census Crimes Data&lt;/a&gt;. Of course, &lt;a href=&quot;https://knoema.com/atlas/topics/crime/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Knoema&lt;/a&gt;, will provide more of an international flare that you might find useful.&lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2017-01-09T03:03:48.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16194" PostTypeId="2" ParentId="11750" CreationDate="2017-01-09T03:16:53.110" Score="3" Body="&lt;p&gt;Here we are creating data to for a given model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main advantage with this code snippet is you are able to see how the model performers over various sizes of the training set. The &lt;a href=&quot;https://www.rdocumentation.org/packages/caret/versions/6.0-71/topics/learing_curve_dat&quot; rel=&quot;nofollow noreferrer&quot;&gt;learing_curve_data&lt;/a&gt; function can be found in the &lt;code&gt;[6.0-7.1 version of caret][1]&lt;/code&gt;. Of course, &lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_curve#In_machine_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;learning curves are useful in machine learning&lt;/a&gt; for several reason which include comparison of algorithms, choosing model parameters, optimization, right data size to use for training.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(1412)&#xA;class_dat &amp;lt;- twoClassSim(1000)&#xA;&#xA;set.seed(29510)&#xA;lda_data &amp;lt;- learing_curve_dat(dat = class_dat, &#xA;                               outcome = &quot;Class&quot;,&#xA;                               test_prop = 1/4, &#xA;                               ## `train` arguments:&#xA;                               method = &quot;lda&quot;, &#xA;                               metric = &quot;ROC&quot;,&#xA;                               trControl = trainControl(classProbs = TRUE, &#xA;                                                        summaryFunction = twoClassSummary))&#xA;&#xA;&#xA;ggplot(lda_data, aes(x = Training_Size, y = ROC, color = Data)) + &#xA;geom_smooth(method = loess, span = .8) + &#xA;theme_bw()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2017-01-09T03:16:53.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16195" PostTypeId="2" ParentId="14527" CreationDate="2017-01-09T03:32:37.297" Score="3" Body="&lt;p&gt;Curious Georg if you ran across &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; in your pursuit of trying to generate probabilities. It is worth noting that &lt;code&gt;binary:logistic&lt;/code&gt; and &lt;code&gt;multi:softprob&lt;/code&gt; return predicted probability of each data point belonging to each class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look &lt;a href=&quot;https://gist.github.com/slaypni/b95cb69fd1c82ca4c2ff&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; to see how the following code is used:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yJ6S4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yJ6S4.png&quot; alt=&quot;XGBoost Predict_Proba Code&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26596" LastActivityDate="2017-01-09T03:32:37.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16196" PostTypeId="2" ParentId="16191" CreationDate="2017-01-09T04:50:24.080" Score="0" Body="&lt;p&gt;If you are trying out neural networks, I would recommend &lt;a href=&quot;http://keras.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;, it is a wrapper around TensorFlow. With just a few lines of code, you can build a neural net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scikit learn is really good for ml algorithms like logistic or linear regression, naive​ Bayes, svm and others. I am not sure if it is good enough for neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would still think, you would require some blas library.&lt;/p&gt;&#xA;" OwnerUserId="27426" LastEditorUserId="63031" LastEditDate="2018-11-23T12:25:47.440" LastActivityDate="2018-11-23T12:25:47.440" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="16197" PostTypeId="2" ParentId="11721" CreationDate="2017-01-09T06:15:07.000" Score="1" Body="&lt;p&gt;Take a look at :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has the data of crimes from 2001 to present. This data is extracted from the Chicago Police Department's CLEAR (Citizen Law Enforcement Analysis and Reporting) system&lt;/p&gt;&#xA;" OwnerUserId="27760" LastActivityDate="2017-01-09T06:15:07.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16198" PostTypeId="2" ParentId="11729" CreationDate="2017-01-09T07:54:05.113" Score="1" Body="&lt;p&gt;Have a look on &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow noreferrer&quot;&gt;conditional random field&lt;/a&gt;. Conditional random fields are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data.&#xA;&lt;a href=&quot;https://taku910.github.io/crfpp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRF++&lt;/a&gt; is a simple, customizable, and open source implementation of Conditional Random Fields (CRFs) &lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-01-09T07:54:05.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16200" PostTypeId="2" ParentId="16154" CreationDate="2017-01-09T09:42:45.563" Score="0" Body="&lt;p&gt;I think your intuition is correct, that it doesn't make sense to &quot;clean&quot; the matrices, if they are generated perfectly.  I agree with @SyedAliHamza that noise should be eliminated as much as possible in the simulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, depending on what analysis you are trying to do, you may want your model to generalize better. In that case, it would be good to &lt;em&gt;apply&lt;/em&gt; some data augmentation, meaning adding some noise or other types of augmentation to your matrices.&lt;/p&gt;&#xA;" OwnerUserId="27551" LastActivityDate="2017-01-09T09:42:45.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16201" PostTypeId="2" ParentId="16114" CreationDate="2017-01-09T09:57:41.760" Score="1" Body="&lt;p&gt;Yes, but the labels are limited to the ones from ImageNet.&lt;/p&gt;&#xA;" OwnerUserId="27551" LastActivityDate="2017-01-09T09:57:41.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16202" PostTypeId="2" ParentId="11721" CreationDate="2017-01-09T10:15:16.607" Score="0" Body="&lt;p&gt;There was a Kaggle competition last year about &lt;a href=&quot;https://www.kaggle.com/c/sf-crime&quot; rel=&quot;nofollow noreferrer&quot;&gt;San Francisco Crime Classification&lt;/a&gt;.&lt;br&gt;&#xA;You can &lt;a href=&quot;https://www.kaggle.com/c/sf-crime/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;download&lt;/a&gt; the training set and the test set which contains criminal records from 2003 to 2015 in SF. It just need a registration on the website.&lt;br&gt;&#xA;You also can access to different interesting &lt;a href=&quot;https://www.kaggle.com/c/sf-crime/kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernels&lt;/a&gt; about this subject.&lt;/p&gt;&#xA;" OwnerUserId="27003" LastActivityDate="2017-01-09T10:15:16.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16203" PostTypeId="2" ParentId="16182" CreationDate="2017-01-09T10:44:20.490" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;how would I have to format my data to make it work with Keras?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your training labels in the output layer should be a binary vector that is 1 for class which is present and 0 for class which is not. For example, let us assume you have 3 classes of genres - comedy, romantic and horror. There are many ways to make it and Scikit-learn has a method which makes it very easy which I show below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.preprocessing import MultiLabelBinarizer&#xA;&amp;gt;&amp;gt;&amp;gt; mlb = MultiLabelBinarizer()&#xA;&amp;gt;&amp;gt;&amp;gt; y = mlb.fit_transform([[0,2],[1]])&#xA;array([[1, 0, 1],&#xA;   [0, 1, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a simple Keras example I suggest.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from keras.models import Sequential&#xA;&amp;gt;&amp;gt;&amp;gt; from keras.layers import Dense, Activation&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; model = Sequential([&#xA;    Dense(32, input_dim=784),&#xA;    Activation('relu'),&#xA;    Dense(10),&#xA;    Activation('sigmoid'),&#xA;    ])&#xA;&amp;gt;&amp;gt;&amp;gt; model.compile(optimizer='rmsprop', loss='binary_crossentropy')&#xA;&amp;gt;&amp;gt;&amp;gt; model.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Refer &lt;a href=&quot;https://datascience.stackexchange.com/questions/13949/activation-method-and-loss-function-for-multilabel-multiclass-classification&quot;&gt;this&lt;/a&gt; for more info. I used sigmoid because it is better for multilabel classification.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-01-10T10:19:37.080" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16204" PostTypeId="2" ParentId="16182" CreationDate="2017-01-09T11:00:03.310" Score="1" Body="&lt;p&gt;In the case where you can have multiple labels individually from each other you can use a sigmoid activation for every class at the output layer and use the sum of normal binary crossentropy as the loss function. You would just use a vector with binary numbers as the target, for each label a 1 if it includes the label and a 0 if not.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-01-09T11:00:03.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16205" PostTypeId="1" CreationDate="2017-01-09T11:22:46.860" Score="2" ViewCount="512" Body="&lt;p&gt;I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on &lt;a href=&quot;http://yaroslavvb.blogspot.in/2011/09/notmnist-dataset.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;notMNIST dataset&lt;/a&gt;. My code is as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))&#xA;bias = tf.Variable(tf.zeros(shape = [10]))&#xA;logits = tf.matmul(features, weights) + bias&#xA;prediction = tf.nn.softmax(logits)&#xA;cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)&#xA;loss = tf.reduce_mean(cross_entropy)&#xA;&#xA;train_feed_dict = {features: train_features, labels: train_labels}&#xA;valid_feed_dict = {features: valid_features, labels: valid_labels}&#xA;test_feed_dict = {features: test_features, labels: test_labels}&#xA;&#xA;is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))&#xA;accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))&#xA;&#xA;epochs = 5&#xA;batch_size = 50&#xA;learning_rate = 0.1&#xA;&#xA;optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)&#xA;&#xA;validation_accuracy = 0.0&#xA;&#xA;with tf.Session() as session:&#xA;&#xA;    session.run(tf.global_variables_initializer())&#xA;    batch_count = int(math.ceil(len(train_features)/batch_size))&#xA;&#xA;    for epoch_i in range(epochs):&#xA;&#xA;        for i in range(batch_count):&#xA;&#xA;            session.run(optimizer, feed_dict = train_feed_dict)&#xA;            print(session.run(accuracy, feed_dict = train_feed_dict))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="27616" LastEditorUserId="27616" LastEditDate="2017-01-09T18:13:24.133" LastActivityDate="2017-03-22T07:56:40.270" Title="Accuracy stagnated while training notMNIST data" Tags="|python|logistic-regression|tensorflow|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16206" PostTypeId="1" CreationDate="2017-01-09T13:37:51.417" Score="1" ViewCount="228" Body="&lt;p&gt;Do people in Machine Learning (or, more generally, in Data Mining) realise that no causal link can be inferred from a correlation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My impression of the ML/AI community is that most people involved have a Computer Science background. I think it uncontroversial to point out that this is not a Science in the academic sense (think Physics, Chemistry, Psychology). Therefore, most of these people have no formal training in the Scientific Method, in Statistics or even in Mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;em&gt;Correlation does not equal causation&lt;/em&gt;&quot; is lesson nr. 1 in Statistics. In order to establish causal links, experiments are required (under some very special circumstances, other techniques are available). To me, this seems a fundamental and disastrous limitation to Machine Learning. ML will do fine with describing data (and classifying it), but it will never substitute an experiment. The application of ML is, therefore, limited in scope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is about the atmosphere within the ML community.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do people within the field recognise this limitation? Or do people think the causal-inference problem is not fundamental and will be solved in the future by better CPU's and better algorithms?&lt;/li&gt;&#xA;&lt;li&gt;Do they restrict the use of ML to descriptive analyses? Or do they, erroneously, assume ML can help us understand and influence the world?&lt;/li&gt;&#xA;&lt;li&gt;Is there a lively debate on this topic within the community? And are there articles/blogs related to these issues?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27776" LastEditorUserId="27776" LastEditDate="2017-01-09T14:44:56.703" LastActivityDate="2017-01-09T15:42:13.433" Title="Do Data Miners realise Machine Learning cannot establish Causality?" Tags="|machine-learning|statistics|experiments|" AnswerCount="1" CommentCount="6" ClosedDate="2017-01-10T14:11:20.377" ContentLicense="CC BY-SA 3.0" />
  <row Id="16207" PostTypeId="1" CreationDate="2017-01-09T14:07:54.290" Score="1" ViewCount="466" Body="&lt;p&gt;I'm trying to extract percentages using Stanford NER. But it is not extracting percentage properly.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;from nltk.tag import StanfordNERTagger&#xA;os.environ['CLASSPATH'] = 'stanford-ner-2015-12-09/stanford-ner.jar'&#xA;os.environ['STANFORD_MODELS'] = 'stanford-ner-2015-12-09/classifiers'&#xA;&#xA;inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'&#xA;split_inp_str = inp_str.split()&#xA;st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')&#xA;print(st.tag(split_inp_str))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives following output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'O'), ('hundred', 'O'), ('and', 'O'), ('twenty', 'O'), ('five', 'PERCENT'), ('percent', 'PERCENT'), ('125%', 'O'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Expected ouput:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'PERCENT'), ('hundred', 'PERCENT'), ('and', 'PERCENT'), ('twenty', 'PERCENT'), ('five', 'PERCENT'), ('percent', 'PERCENT'),('125%', 'PERCENT'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why is it not extracting &lt;strong&gt;125%&lt;/strong&gt; or &lt;strong&gt;one hundred and twenty five percent&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="25100" LastEditorUserId="25100" LastEditDate="2017-01-10T06:33:33.460" LastActivityDate="2018-06-26T11:07:10.140" Title="Stanford NER is not properly extracting percentages" Tags="|named-entity-recognition|stanford-nlp|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16209" PostTypeId="2" ParentId="16206" CreationDate="2017-01-09T15:42:13.433" Score="6" Body="&lt;p&gt;I think you're a bit confused about the role machine learning plays in the order of things, and I find your impression of the ML community to be a bit bizarre.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To begin, ML experts are often not the people who use ML algorithms to solve science problems - those people are usually called &quot;data scientists&quot; these days, and in principle they are supposed to have some sort of background in science, statistics, and/or mathematics.  And yes, (good) data scientists spend a lot of time worrying about causal inference - they have to because there are big companies with lots of money riding on their predictions.  Contrary to your impression ML experts are usually aware of these issues too, but their job is to build and analyze modeling tools rather than actually apply them to real data.  Of course there are lots of ML and data science people which straddle the line between the two fields, but my point is that your question is a bit like asking why mathematicians don't worry about experimental error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, many (most?) ML algorithms are more oriented towards classification problems which are not quite as vulnerable to your concerns.  You still have to worry about sticky statistical problems like overfitting and building good training / validation datasets, but the focus is more on understanding the structure of the data that you have than predicting the future.&lt;/p&gt;&#xA;" OwnerUserId="27780" LastActivityDate="2017-01-09T15:42:13.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16211" PostTypeId="2" ParentId="10695" CreationDate="2017-01-09T16:56:48.833" Score="2" Body="&lt;p&gt;If you are looking for a pre-trained net for word-embeddings, I would suggest GloVe. The following blog from Keras is very informative of how to implement this. It also has a link to the pre-trained GloVe embeddings. There are pre-trained word vectors ranging from a 50 dimensional vector to 300 dimensional vectors. They were built on either Wikipedia, Common Crawl Data, or Twitter data. You can download them &lt;a href=&quot;http://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Additionally, you should examine the &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras blog&lt;/a&gt; on how to implement them.&lt;/p&gt;&#xA;" OwnerUserId="27632" LastEditorUserId="98307" LastEditDate="2020-08-06T15:06:33.957" LastActivityDate="2020-08-06T15:06:33.957" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16212" PostTypeId="2" ParentId="16177" CreationDate="2017-01-09T17:08:25.193" Score="1" Body="&lt;p&gt;I do not think that you would need a recurrent neural net here. This will be much slower to train than a ConvNet. Additionally, your data is images, correct? I think you could use a deep convolutional net to a fully-connected net. This should be fast and perform well. Please look at the following archived blogs from keras.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Building powerful image classification models using very little data&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;How convolutional neural networks see the world&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27632" LastEditorUserId="98307" LastEditDate="2020-08-13T13:56:58.130" LastActivityDate="2020-08-13T13:56:58.130" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="16213" PostTypeId="2" ParentId="16104" CreationDate="2017-01-09T17:57:50.023" Score="1" Body="&lt;p&gt;Are your variables categorical? If they are, then you do not need to remove them. You could just make new columns like NA_Variable1, which will indicate whether a NA value exists. &lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-09T17:57:50.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16214" PostTypeId="2" ParentId="16098" CreationDate="2017-01-09T19:44:09.723" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;can I just provide an image to an untrained Inception and get a label back?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You will get a random output from the network because it hasn't been trained and has been initialized with random values for the weights.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there an API for this?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Cloud Vision is an API with already trained models. If you want to train your own model you can use tensorflow, theano, caffe... there are several frameworks for it. None of them is easy.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;why does Google open source this if they're trying to make money off of google cloud vision?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Open sourcing tools and models gives to the community a lot of knowledge, in exchange the researchers community use the Google tools to improve the models, more people use the models, so it is easier for Google to hire Machine-Learning developers and, most important, because the real value of machine learning today is in the data and the infrastructure you have. The more data you have the better your models. The better the infrastructure the faster and cheaper you can train and deploy your models. And people will pay you to use your infrastructure.&lt;/p&gt;&#xA;" OwnerUserId="20933" LastActivityDate="2017-01-09T19:44:09.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16215" PostTypeId="2" ParentId="15962" CreationDate="2017-01-09T20:00:39.350" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Why does a single order of magnitude have such an effect?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) Why does the minimize function literally perform the opposite of its function and maximize the loss? It seems to me that that shouldn't occur, no matter the learning rate.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are two main reasons. First one you are not using the same data in the first step than in the second. If in the first step the model learns those values and falls into a local minimum then it is very likely to give a bigger loss for new values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second reason is the shape of the cost function. You try to minimize the value by small steps, the length of those steps is given by two factors: the gradient and the learning rate. Image your function is like x^2. If your values are close to 0, the gradient is going to be small than if it further, but if your learning rate is big then, instead of getting closer to 0 you actually increase the error because your new point based on the grading and the learning rate is further to 0 than your previous step. And this can happen several times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look to this link: &lt;a href=&quot;http://www.statisticsviews.com/details/feature/5722691/Getting-to-the-Bottom-of-Regression-with-Gradient-Descent.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.statisticsviews.com/details/feature/5722691/Getting-to-the-Bottom-of-Regression-with-Gradient-Descent.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you see the figures with alpha 0.01 and alpha 0.12, you will see how in the first figure the learning rate is small and so the gradient is getting closer to the minimum but in the second case the learning rate is so big that the gradient moves further in every step.&lt;/p&gt;&#xA;" OwnerUserId="20933" LastActivityDate="2017-01-09T20:00:39.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16218" PostTypeId="1" CreationDate="2017-01-10T02:43:10.493" Score="1" ViewCount="1386" Body="&lt;p&gt;I have 2 classes of data A and C.   I want to create a NN with 3 output nodes, classifying data as either A, B, or C.   Classifying an A or C correctly should have zero cost (very good).  Classifying an A as a C or vice versa should have high cost (very bad).  However, for samples that are &quot;blurry&quot;, I'd like to classify them as B, which would have relatively low cost (better than a misclassification, but still not ideal).   I am new to tensorflow, and am not comfortable with the different ways to define cost functions.   Can anyone point me in the right direction on this one?  Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="27796" LastActivityDate="2017-02-09T04:09:47.747" Title="Cost functions penalizing certain types of misclassification more heavily in tensorflow" Tags="|machine-learning|neural-network|classification|tensorflow|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16219" PostTypeId="1" CreationDate="2017-01-10T02:45:38.720" Score="8" ViewCount="11787" Body="&lt;p&gt;I found packages being used to calculating &quot;Information Gain&quot; for selecting main attributes in C4.5 Decision Tree and I tried using them to calculating &quot;Information Gain&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the results of calculation of each packages are different like the code below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; IG.CORElearn &amp;lt;- attrEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1, estimator = &quot;InfGain&quot;)&#xA;&amp;gt; IG.RWeka     &amp;lt;- InfoGainAttributeEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1)&#xA;&amp;gt; IG.FSelector &amp;lt;- information.gain(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi,dataUSE1)&#xA;&#xA;&amp;gt; IG.CORElearn&#xA;   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi &#xA;0.04472928 0.02705100 0.09305418 0.35064927 0.44299167 0.01832216 0.05551973 &#xA;&amp;gt; IG.RWeka&#xA;   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi &#xA;0.11964771 0.04340197 0.12266724 0.38963327 0.44299167 0.03831816 0.07705798 &#xA;&amp;gt; IG.FSelector&#xA;         attr_importance&#xA;In_Temp       0.08293347&#xA;In_Humi       0.02919697&#xA;In_CO2        0.08411316&#xA;In_Illu       0.27007321&#xA;In_LP         0.30705843&#xA;Out_Temp      0.02656012&#xA;Out_Humi      0.05341252&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why do the results of calculation of each packages be different?&#xA;And Which one is right?&lt;/p&gt;&#xA;" OwnerUserId="27797" LastActivityDate="2017-09-28T05:56:00.870" Title="Information Gain in R" Tags="|r|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16220" PostTypeId="2" ParentId="16218" CreationDate="2017-01-10T03:47:21.293" Score="2" Body="&lt;p&gt;You are talking about having different missclassification costs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classification consists of outputing a continuous &lt;em&gt;score&lt;/em&gt; which is then &lt;em&gt;thresholded&lt;/em&gt; to decide the discrete classification. The threshold (ex. probability&gt;0.5 leads to C) is chosen to satisfy your apetite on sensitivty/specificity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training algorithm only cares about your continuous score. This means that you can make it easy for yourself by doing simple binary classification (typically with &lt;code&gt;softmax_cross_entropy_with_logits&lt;/code&gt;-loss) and create your vague class B &lt;em&gt;after&lt;/em&gt; training is done by thresholding say [0,0.4) is A, [0.4,0.6] is B and (0.6,1] is C. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To achieve exactly what you're saying, penalizing certain errors more, then you want to use the &lt;a href=&quot;https://www.tensorflow.org/versions/master/api_docs/python/nn/classification#weighted_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;weighted_cross_entropy_with_logits&lt;/a&gt;. This throws out the interpretation of your outputted score as a probability, and you will need to work more to get a reasonable threshold afterwards so this is not recommended as a first step as it will probably add more complexity than results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also recommend reading the wikipedia article on &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow noreferrer&quot;&gt;roc analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25295" LastActivityDate="2017-01-10T03:47:21.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16221" PostTypeId="2" ParentId="16207" CreationDate="2017-01-10T07:23:52.590" Score="1" Body="&lt;p&gt;Looking at your code, I dont believe there is any mistake in it. The Stanford NER tagger is modelled by a Conditional Random Field Classifier and is hence bound to make mistakes. If you are able to check for a large number of samples and calculate an accuracy metric on the basis of that you might get a good idea if the pre-trained model is useful for you. Else, I suggest that you train your own model using. The stanford NLP group provide an implementation for training, you just have to give it the data in the required format. Look at this &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://medium.com/%40klintcho/training-a-swedish-ner-model-for-stanford-corenlp-part-1-3e3f281a753a&amp;amp;ved=0ahUKEwjUsO3bhbfRAhXCv48KHaa_A7gQFggaMAA&amp;amp;usg=AFQjCNFOzjSqMiOUgXFQlgYS_5Z7K29Lrg&amp;amp;sig2=pN31J9ECKKB7-qH75mUfWw&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; to get a good idea on how to do it. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-10T07:23:52.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16222" PostTypeId="1" CreationDate="2017-01-10T07:37:05.683" Score="1" ViewCount="64" Body="&lt;p&gt;I am training a model in Spark using Dl4J library in Yarn-Cluster mode. When I train the model on 2 lakh data (approx 200MB) then the job succeeds but when I go to train the model with 3 lakh data (approx 300MB) then the job finishes with a fail status and the model doesn't get saved in HDFS. So somehow I am reaching to a bottleneck over there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using 9 containers with 1 core and 1.5GB memory to run the training job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no error in yarn or spark logs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I identify where the problem has occurred? How do I identify if it is the insufficient resources that is causing the problem or is the code that is the problem?&lt;/p&gt;&#xA;" OwnerUserId="24157" LastActivityDate="2017-01-10T07:37:05.683" Title="Spark Deeplearning4j Training Problem" Tags="|neural-network|deep-learning|apache-spark|training|clustering|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16225" PostTypeId="1" CreationDate="2017-01-10T09:49:00.067" Score="12" ViewCount="27071" Body="&lt;p&gt;For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?&lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2023-11-02T01:23:23.737" Title="Would you recommend feature normalization when using boosting trees?" Tags="|random-forest|xgboost|normalization|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16226" PostTypeId="2" ParentId="16225" CreationDate="2017-01-10T10:02:00.933" Score="11" Body="&lt;p&gt;Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Decision_tree_advantages&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia for decision tree&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Requires little data preparation. Other techniques often require data normalization....&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it's always a good idea to normalize your features because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It's easier to visualize and interpret your model&lt;/li&gt;&#xA;&lt;li&gt;It's easier to compare another model (e.g. SVM) with the same data set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-01-10T10:02:00.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16229" PostTypeId="1" CreationDate="2017-01-10T12:38:56.927" Score="2" ViewCount="753" Body="&lt;p&gt;I implemented a standard RNN in Tensorflow (0.12) using the classes&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;tf.python.ops.rnn_cell.GRUCell&lt;/li&gt;&#xA;&lt;li&gt;tf.nn.dynamic_rnn&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am interested in the gates and which values they take during. Unfortunately the function dynamic_rnn doesn't support this and just gives the 'output' and 'state' (The final hidden state). In the code of the GRUCell the gates are called 'r' and 'u' for reset and update.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a clever way to save the values of the gates over time or do I have to write my own GRU cell?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2024-01-30T14:04:50.613" Title="Get Gate Activation of GRU-Cells in TensorFlow" Tags="|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16231" PostTypeId="1" AcceptedAnswerId="16238" CreationDate="2017-01-10T15:16:44.587" Score="1" ViewCount="401" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q6pN3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q6pN3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently this is what I get. &quot;community_area_clean&quot; is the new variable that I added by extracting a variable that lists all the communities from a data frame. &quot;CommArea_Name&quot; is the original &quot;unclean&quot; variable. But I find that the names in &quot;community_area_clean&quot; do not correspond to the names in &quot;CommArea_Name&quot;. How can I fix it? Thanks. &lt;/p&gt;&#xA;" OwnerDisplayName="user27822" LastActivityDate="2017-03-11T17:35:51.317" Title="R data frame create a new variable which corresponds to one of the existing one" Tags="|r|data-cleaning|data-formats|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16232" PostTypeId="1" AcceptedAnswerId="16263" CreationDate="2017-01-10T16:29:11.570" Score="15" ViewCount="27647" Body="&lt;p&gt;I am using XGBoost for payment fraud detection.  The objective is binary classification, and the data is very unbalanced.  One out of every 3-4k transactions is fraud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect the best way to evaluate the results is a Precision-Recall (PR) curve, not a ROC curve, since the data is so unbalanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However in the eval_metric options I see only area under the ROC curve (AUC), and there is no PR option.  &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the documentation recommends AUC  &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html&quot; rel=&quot;noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to not use a Precision-Recall (PR) curve?&lt;/p&gt;&#xA;" OwnerUserId="25359" LastActivityDate="2018-10-26T22:12:57.297" Title="In XGBoost would we evaluate results with a Precision Recall curve vs ROC?" Tags="|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16233" PostTypeId="1" CreationDate="2017-01-10T16:33:08.953" Score="1" ViewCount="359" Body="&lt;p&gt;I have a lasgane code. I want to create the same network using caffe. I could conver the network.But i need help with the hyperparameters in lasagne. The hyperparameters  look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lr = 1e-2&#xA;weight_decay = 1e-5&#xA;&#xA;prediction = lasagne.layers.get_output(net['out'])&#xA;loss = T.mean(lasagne.objectives.squared_error(prediction, target_var))&#xA;&#xA;weightsl2 = lasagne.regularization.regularize_network_params(net['out'], lasagne.regularization.l2)&#xA;loss += weight_decay * weightsl2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do i perform the L2 regularization part in caffe? Relevant parts from my solver.prototxt is as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;base_lr: 0.01&#xA;lr_policy: &quot;fixed&quot;&#xA;weight_decay: 0.00001&#xA;regularization_type: &quot;L2&quot;&#xA;stepsize: 300&#xA;gamma: 0.1  &#xA;max_iter: 2000&#xA;momentum: 0.9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27810" LastActivityDate="2017-01-10T16:33:08.953" Title="L2 regularization in caffe" Tags="|theano|regularization|caffe|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16234" PostTypeId="1" CreationDate="2017-01-10T16:41:24.980" Score="1" ViewCount="31" Body="&lt;p&gt;Suppose a person (John Doe) uses different user names (handles) in different circumstances (for simplicity, let us limit the conversation to a single social network, e.g., &lt;code&gt;twitter&lt;/code&gt;). E.g., he &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;posts pictures of kittens as &lt;code&gt;john_doe&lt;/code&gt;, &lt;/li&gt;&#xA;&lt;li&gt;posts political rants as &lt;code&gt;deplorable_john&lt;/code&gt;,&lt;/li&gt;&#xA;&lt;li&gt;shares technology news as &lt;code&gt;super_doe&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&amp;amp;c (and each &lt;code&gt;screen_name&lt;/code&gt; has a &lt;em&gt;different&lt;/em&gt; &lt;code&gt;id&lt;/code&gt;, so no short cut).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How would you go about detecting that all these accounts are used by the same individual?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;SNA: look at who they follow (and maybe who follows them): if these sets are sufficiently similar, it's a good signal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;NLP: compare the posted texts and try to detect that they have been written by the same author.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Meta data: see if the accounts are usually active from the same geo location at the same time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What other approaches have been / could be tried?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any published work?&lt;/p&gt;&#xA;" OwnerUserId="9867" LastActivityDate="2017-01-10T16:41:24.980" Title="How to detect multiple handles used by the same individual?" Tags="|social-network-analysis|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16235" PostTypeId="1" AcceptedAnswerId="16239" CreationDate="2017-01-10T16:48:25.010" Score="4" ViewCount="3499" Body="&lt;p&gt;I have read some papers on text classification but they are pretty abstract. &lt;strong&gt;I fail at understanding how to train&lt;/strong&gt; a multi layer perceptron with data made of sentences -&gt; label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A perceptron takes a vector input of size m and outputs a vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a first step I reduced the dimensions and I map each word to a vector of n features, where n is a constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how do I mix the words in a sentence to make an input of size m ? Each word maps to a vector of size n but then what ? Knowing that every sentence has a different number of words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="27824" LastActivityDate="2017-10-05T07:56:25.987" Title="In text classification, how can I use a neural network on word embeddings?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16236" PostTypeId="2" ParentId="14065" CreationDate="2017-01-10T16:55:42.600" Score="1" Body="&lt;p&gt;With the current version of &lt;code&gt;simputation&lt;/code&gt; you can impute group means with the following trick:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;impute_lm(df, rating ~ 1 | id)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is linear regression imputation without predictors (hence: mean). The grouping makes sure group means are imputed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;simputation&lt;/code&gt; (&gt;=0.2.1) [not on cran yet] you can do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;impute_proxy(df, rating ~ mean(rating,na.rm=TRUE) | id)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At the moment this is not on CRAN but you can install it from the &lt;a href=&quot;https://CRAN.R-project.org/package=drat&quot; rel=&quot;nofollow noreferrer&quot;&gt;drat&lt;/a&gt; repo as described &lt;a href=&quot;https://github.com/markvanderloo/simputation#materials&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27827" LastActivityDate="2017-01-10T16:55:42.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16237" PostTypeId="1" CreationDate="2017-01-10T16:59:56.543" Score="3" ViewCount="833" Body="&lt;h2&gt;The Problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a representation of a document's text layout as in the image below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kl91P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kl91P.png&quot; alt=&quot;Document layout&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, each rectangle represents a chunk of text (one word, an expression or even part of a sentence). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each rectangle has a set of features and connections to adjacent rectangles (the image doesn't show all connections, but they are present):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;top/bot/lft/rgh sibling&lt;/li&gt;&#xA;&lt;li&gt;top/bot/lft/rgh distance to sibling (in case of sibling presence)&lt;/li&gt;&#xA;&lt;li&gt;margins alignment (for rectangles aligned in columns)&lt;/li&gt;&#xA;&lt;li&gt;font size (continuous variable type)&lt;/li&gt;&#xA;&lt;li&gt;font type (categorical)&lt;/li&gt;&#xA;&lt;li&gt;all_caps (binary)&lt;/li&gt;&#xA;&lt;li&gt;and some other&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I would like to achieve is to train an algorithm (ML? DNN? Graphs? something other?) that would structure these rectangles, would create a tree structure out of these individual nodes. For instance the above example would be structured as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          ROOT&#xA; __________|___&#xA;|          |   |&#xA;2          10  12&#xA;|_         |   |_________&#xA;| |        |   |         |&#xA;3 9        11  14        20...&#xA;|________      |__       |__&#xA;| | | | |      |  |      |  |&#xA;4 5 6 7 8      13 15...  19 21...&#xA;                            |  \&#xA;                            22 23...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;My results so far&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I spend some time (~8h already) searching for a solution, but couldn't find anything applicable to this case. I'm relatively new to ML and don't have much experience, however I think it can be solved using a supervised learning algorithm by feeding a training set consisting of documents with all these nodes with their features and annotated trees that should be generated as a result out of these nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some ideas I found while searching that might help (probably one of these &lt;em&gt;is&lt;/em&gt; the solution, but I need some guidance on how to adjust in properly):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Using some sort of DNN. It would figure out the relationship between distances and other features of the nodes from the training set so that it would be able to predict a structure given only the nodes and their features. I didn't work with DNN and I don't know how to make it &lt;em&gt;generate&lt;/em&gt; the structure instead of &lt;em&gt;categorize&lt;/em&gt; which digit is shown in the image (I did some basic tutorials)&lt;/li&gt;&#xA;&lt;li&gt;Creating/Training a PCFG (Probabilistic Context Free Grammar). This approach seems to be valid, as it used in generating parsed trees in NLP (those that structure a sentence in form of a tree) and I found some resources on it, however I don't understand how to use all the features that are available on my nodes in learning such a grammar? Also, I don't see how this approach could make use of the positioning data in 2D space and the distances between siblings, as it is one of the key features in structuring the nodes.&lt;/li&gt;&#xA;&lt;li&gt;Probably some Graph or Tree related algorithms would solve my problem, but I don't know how to formulate the query that would help me find the solution.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A very similar question was asked on &lt;a href=&quot;https://cs.stackexchange.com/questions/55185/coercing-a-list-of-nodes-into-the-most-probable-tree?newreg=ef5b085f837340d79ca2d2e6dcc995e3&quot;&gt;Computer Science SE site&lt;/a&gt;, but there are no answers and the OP doesn't respond.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I would really appreciate if you would provide a solution, or even a hint in what direction to move in order to solve this problem.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="27788" LastActivityDate="2017-01-11T07:46:26.930" Title="Predict a tree structure out of nodes with different features" Tags="|machine-learning|neural-network|clustering|deep-learning|graphs|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16238" PostTypeId="2" ParentId="16231" CreationDate="2017-01-10T17:00:02.033" Score="0" Body="&lt;p&gt;If my assumptions about &lt;code&gt;CommArea_Num&lt;/code&gt; are correct, then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data$community_area_clean = data$community_area_clean[data$CommAreaNum]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should remap that column. Note that the &lt;code&gt;AREA_KM&lt;/code&gt; fields will remain in the order currently shown.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-01-10T17:00:02.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16239" PostTypeId="2" ParentId="16235" CreationDate="2017-01-10T17:02:01.860" Score="2" Body="&lt;p&gt;Okay. These are the steps to follow. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Pad your sentences to a fixed length. Use the maximum length. If you are comfortable with creating buckets, you can pad the sentences to maximum length for individual bucket. To pad, add  PAD tokens to each sentence till they become the size of the fixed sequence length you want to use. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Extract the unique words out of the padded corpus and assign a numeric id to every word. You can use there index in the unique list as IDs. Filtering out rare words and replacing them with unknown token will prove useful. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now if you had a sentence like &quot;I am John&quot;, and your maximum sequence length was 5 the sentence will become, &quot;I am John PAD PAD.&quot; Once you extract the unique words, you can represent the sentence as IDs like [1,2,3,4,4] Now if you had 100 sentences you will have a matrix of size 100 X 5. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Now you can go in two ways, represent each word as a one hot vector i.e.a vectors of zeros with a one at the index it represents or a simpler way would be to train a word embedding. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For details look into &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/&amp;amp;ved=0ahUKEwib7PPGhrjRAhXIMY8KHQhrAuEQFggiMAE&amp;amp;usg=AFQjCNF3HZnrONtqhVKMnjlyiBYcPg654w&amp;amp;sig2=y6xkPfvyxPrnQ41HAGH1sw&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; RNN tutorial by Denny Britz, this is from scratch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are of the deep learning persuasaion then check out Denny Britz's CNN text classification &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&amp;amp;ved=0ahUKEwjngfbrhrjRAhUSUI8KHf2nCaUQFggfMAA&amp;amp;usg=AFQjCNFfCQwHiPUd3lNXTvUXgh0g7xYhCw&amp;amp;sig2=HwdGfSDELpDjPOKz9P4EpA&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; where he uses tensorflow and trains his own word embedding. The first blog will give you all the information you need on how to prepare your text dataset. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-10T17:02:01.860" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16241" PostTypeId="2" ParentId="16128" CreationDate="2017-01-10T18:10:18.903" Score="10" Body="&lt;p&gt;Here is what I built...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Step 1: Store all the words in a Trie data structure. &lt;a href=&quot;https://en.wikipedia.org/wiki/Trie&quot; rel=&quot;noreferrer&quot;&gt;Wiki&lt;/a&gt; about&#xA;trie.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Step 2: Train an RNN or RNTN to get seq2seq mapping for words and&#xA;store the model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Step 3: Retrieve top n words with  &lt;strong&gt;levenshtein&#xA;distance&lt;/strong&gt; with the.  &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;noreferrer&quot;&gt;Wiki&lt;/a&gt; about LD with the word that you are trying to&#xA;correct.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/edit-distance-1.html&quot; rel=&quot;noreferrer&quot;&gt;Naive Approach&lt;/a&gt;: Calculating the edit distance between the query term and every dictionary term. Very expensive. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://norvig.com/spell-correct.html&quot; rel=&quot;noreferrer&quot;&gt;Peter Norvig's Approach&lt;/a&gt;: Deriving all possible terms with an edit distance&amp;lt;=2 from the query term, and looking them up in the dictionary. Better, but still expensive (114,324 terms for word length=9 and edit distance=2) &lt;a href=&quot;http://blog.faroo.com/2012/06/07/improved-edit-distance-based-spelling-correction/&quot; rel=&quot;noreferrer&quot;&gt;Check this.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Faroo's Approach: Deriving deletes only with an edit distance&amp;lt;=2 both from the query term and each dictionary term. Three orders of magnitudes faster. &lt;a href=&quot;http://blog.faroo.com/2012/06/07&quot; rel=&quot;noreferrer&quot;&gt;http://blog.faroo.com/2012/06/07&lt;/a&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Step 4: Based on the previous 2 or 3 or 4 words, predict the words&#xA;that are retrieved from Step 3 above. Select the number of words&#xA;depending on how much accuracy you want (Of course we want 100%&#xA;accuracy), and the processing power of the machine running the code,&#xA;you can select the number of words you wanna consider.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Check out &lt;a href=&quot;http://www.aclweb.org/anthology/D/D09/D09-1093.pdf&quot; rel=&quot;noreferrer&quot;&gt;this approach&lt;/a&gt; too. It's an acl paper from 2009. They call it - language independent auto correction.&lt;/p&gt;&#xA;" OwnerUserId="27426" LastEditorUserId="21024" LastEditDate="2017-01-11T03:29:41.803" LastActivityDate="2017-01-11T03:29:41.803" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16242" PostTypeId="1" CreationDate="2017-01-10T18:31:34.957" Score="1" ViewCount="341" Body="&lt;p&gt;I am working on a feature extraction problem (ECG signal). Within my literature review I stumbled across the following text:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The wavelet transform is used to extract the coefficients of the transform as&#xA;the features of each ECG segment. Simultaneously, autoregressive&#xA;modelling(AR) is also applied to obtain the temporal structures&#xA;of ECG waveforms.&quot; &lt;a href=&quot;https://www.researchgate.net/profile/Liqing_Zhang3/publication/4234035_ECG_Feature_Extraction_and_Classification_Using_Wavelet_Transform_and_Support_Vector_Machines/links/5557571b08aeaaff3bf76f0c.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question now is, how do the wavelet transformation and the AR differ, i.e. how does the AR 'obtain the temporal structure'?&lt;/p&gt;&#xA;" OwnerUserId="17333" LastActivityDate="2024-03-04T06:08:41.107" Title="Feature extraction - wavelet transformation + autoregression" Tags="|machine-learning|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16243" PostTypeId="1" CreationDate="2017-01-10T20:35:37.160" Score="6" ViewCount="5180" Body="&lt;p&gt;I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/z3p9p.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/z3p9p.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Univariate Selection&#xA;import pandas&#xA;import numpy&#xA;from sklearn.feature_selection import SelectKBest&#xA;from sklearn.feature_selection import chi2&#xA;&#xA;# feature extraction&#xA;test = SelectKBest(score_func=chi2, k=4)&#xA;fit = test.fit(X, y)&#xA;# summarize scores&#xA;numpy.set_printoptions(precision=3)&#xA;print(fit.scores_)&#xA;&#xA;&#xA;# Feature Extraction with RFE&#xA;from sklearn.feature_selection import RFE&#xA;&#xA;# feature extraction&#xA;model = LogisticRegression()&#xA;rfe = RFE(model, 15)&#xA;fit = rfe.fit(X, y)&#xA;print(&quot;Num Features: %d&quot; % fit.n_features_)&#xA;print(&quot;Selected Features: %s&quot; % fit.support_)&#xA;print(&quot;Feature Ranking: %s&quot; % fit.ranking_)&#xA;&#xA;&#xA;&#xA;# VarianceThreshold&#xA;from sklearn.feature_selection import VarianceThreshold&#xA;sel = VarianceThreshold(threshold=(.8 * (1 - .8)))&#xA;sel.fit_transform(X)&#xA;&#xA;idxs = sel.get_support(indices=True)&#xA;np.array(X)[:, idxs]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2017-01-10T22:09:56.473" Title="Which feature selection I should trust?" Tags="|machine-learning|python|feature-selection|weka|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16244" PostTypeId="2" ParentId="16243" CreationDate="2017-01-10T21:33:31.630" Score="3" Body="&lt;p&gt;It is generally not best practice to trust ML techniques to do the work for you. A data scientist knowns on what principles the methods that he or she applies work, and then stills tries multiple in the same setting, to see which methods best apply to the domain at hand. How well certain measures work can be a clue to the structure of your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right off the bat it is important to understand that Weka offers 3 kinds of feature selection methods: Filters, a Wrapper and PCA. Filters apply criteria (mostly based on a measure of how it divides a set) for feature suitedness (variations on how well they divide the search space) , wrappers use a learning algorithm for their evaluation and PCA doesn't so much select features as that it recombines features to maximize variance in the dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said: I suggest you start looking at InformationGainEval a method that uses information gain to select attributes. It is quite popular and easy to understand. You can find a good explantation here: &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection&quot;&gt;https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection&lt;/a&gt;. Than work your way through the others (the documentation is accessible from the GUI).&lt;/p&gt;&#xA;" OwnerUserId="27432" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2017-01-10T21:33:31.630" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16245" PostTypeId="2" ParentId="16243" CreationDate="2017-01-10T22:09:56.473" Score="2" Body="&lt;p&gt;I would look at &lt;a href=&quot;https://sebastianraschka.com/faq/docs/feature_sele_categories.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sebastian Raschka's discussion on different types of feature selection methods.&lt;/a&gt;. In short, there appear to be three categories (each with advantages and disadvantages):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Filters&lt;/li&gt;&#xA;&lt;li&gt;Wrappers&lt;/li&gt;&#xA;&lt;li&gt;Embedded Methods&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sebastian goes on to discuss &lt;a href=&quot;https://sebastianraschka.com/Articles/2015_pca_in_3_steps.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;specific feature selection techniques (i.e PCA) and describes the process in 3 simple steps&lt;/a&gt; - might be worth looking into&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;NOTE: I would typically make this a 'comment', but an unable to do so due to my low reputation within this forum&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="24945" LastActivityDate="2017-01-10T22:09:56.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16246" PostTypeId="1" CreationDate="2017-01-11T01:11:40.887" Score="5" ViewCount="587" Body="&lt;p&gt;I am a layperson who has searched for years in vain for a human-sounding TTS program, but it seems that over the years, hardly any progress has been made (a purely unscientific evaluation using just my ear and also the ears of many others). It still sounds like the same robotic word-by-word reading-out that I believe was already accomplished decades ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is puzzling (for a layperson like me who doesn't know anything about data science, linguistics, machine learning, etc.). Especially when contrasted to say speech-to-text or driverless cars, which have noticeably improved over the years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you explain to a layperson (like myself) why, in early 2017, do most text-to-speech (TTS) programs still sound so robotic and non-human-like? And why so little progress has been made (especially in comparison to some other achievements)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Might it just be that there simply isn't much demand for TTS (as compared to speech-to-text or driverless cars) and so few resources have been devoted to this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If this isn't the right StackExchange for the question, I apologize in advance. Please migrate this to a more suitable StackExchange site instead, thanks!)&lt;/p&gt;&#xA;" OwnerDisplayName="user27838" LastActivityDate="2017-10-04T21:06:35.387" Title="Explain to a layperson why text-to-speech (TTS) has made little progress?" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16247" PostTypeId="1" CreationDate="2017-01-11T03:10:45.100" Score="4" ViewCount="9855" Body="&lt;p&gt;I am quite new to machine learning and python as well. I faced an imbalanced dataset and wanna use cross validation and oversamopling like the figure shown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L0p6c.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L0p6c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realised the Python function below cannot be directly used for this purpose and please advice some codes for this task.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_val_score(model, X_train,np.ravel(y_train), cv=n_folds, n_jobs=1, scoring='roc_auc')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2019-10-26T04:49:51.613" Title="Cross validation plus oversampling?" Tags="|machine-learning|python|cross-validation|sampling|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16248" PostTypeId="1" CreationDate="2017-01-11T03:43:15.140" Score="1" ViewCount="1591" Body="&lt;p&gt;I have setup a four layer CNN designed to predict two classes. The two classes are more or less in the same ratio. The negative class is 55% of the data and the positive class makes up the remaining. When I train the network on the data and look at the final results, I see two problems: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The validation accuracy fluctuates. In fact, it can be thought of as oscillating. The validation loss increases and the accuracy however keeps hovering.&lt;/li&gt;&#xA;&lt;li&gt;The final results indicate a possible reason for this behaviour. The negative classes are very well classified. However, the postive classes are very poorly classified. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is why is the network exhbiting this behaviour. I've enough training data, ~10k images in training, and the classes are more or less balanced (5400 in the negative and 4500 in the positive). The issue raised in point 2 increases with an increase in the number of epochs. I've tried decreasing the learning rate, different optimizers, increased the number of layers, decreased the number of layers, used transfer learning (VGG-16). I've also tried reducing the number of epochs, but this leads to poor classification in the negative class as well. &lt;/p&gt;&#xA;" OwnerUserId="14360" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:55.983" LastActivityDate="2021-02-16T15:24:55.983" Title="Neural net learning only one class?" Tags="|deep-learning|keras|convolutional-neural-network|" AnswerCount="0" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="16249" PostTypeId="2" ParentId="16219" CreationDate="2017-01-11T03:43:36.570" Score="2" Body="&lt;p&gt;This is not a complete answer to your question, but I can explain at least part of the problem. Since you do not provide your data, I cannot reproduce your results. However, it is easy to demonstrate the same problem with other data.  I will use the well-known iris data set that is provided with R and Weka and is easy to access. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same problem is apparent with the iris data. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(CORElearn)&#xA;library(RWeka)&#xA;library(FSelector)&#xA;IG.CORElearn &amp;lt;- attrEval(Species ~ ., data=iris,  estimator = &quot;InfGain&quot;)&#xA;IG.RWeka     &amp;lt;- InfoGainAttributeEval(Species ~ ., data=iris,)&#xA;IG.FSelector &amp;lt;- information.gain(Species ~ ., data=iris,)&#xA;&#xA;IG.CORElearn&#xA;Sepal.Length  Sepal.Width Petal.Length  Petal.Width &#xA;   0.5572327    0.2831260    0.9182958    0.9182958 &#xA;&#xA;IG.RWeka &#xA;Sepal.Length  Sepal.Width Petal.Length  Petal.Width &#xA;   0.6982615    0.3855963    1.4180030    1.3784027 &#xA;&#xA;IG.FSelector&#xA;             attr_importance&#xA;Sepal.Length       0.4521286&#xA;Sepal.Width        0.2672750&#xA;Petal.Length       0.9402853&#xA;Petal.Width        0.9554360&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As with your example,  all three packages give completely different results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;H3&gt;One Issue: Units (Base of Logarithm)&lt;/H3&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the documentation for &lt;code&gt;information.gain&lt;/code&gt; in FSelector,  you will see this parameter description:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;unit&lt;br&gt;&#xA;  Unit for computing entropy (passed to entropy). Default is &quot;log&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Following that trail,  we look at the description of the entropy function and see:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;unit&lt;br&gt;&#xA;  the unit in which entropy is measured. The default is &quot;nats&quot; &#xA;  (natural units). For computing entropy in &quot;bits&quot; set unit=&quot;log2&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If we override the default and calculate IG using unit=&quot;log2&quot; we get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IG.FSelector2 &amp;lt;- information.gain(Species ~ ., data=iris, unit=&quot;log2&quot;)&#xA;IG.FSelector2 &#xA;             attr_importance&#xA;Sepal.Length       0.6522837&#xA;Sepal.Width        0.3855963&#xA;Petal.Length       1.3565450&#xA;Petal.Width        1.3784027&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that now the values for Information Gain agree with RWeka for Sepal.Width and Petal.Width. Part of the difference was simply using a different base for the logarithm. RWeka uses base 2 (entropy measured in bits). By default, FSelector uses base &lt;i&gt;e&lt;/i&gt;,  but allows you to change the base and get some of the same results. It appears from the documentation that neither RWeka nor CORElearn let you select the base for the logarithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it is almost stranger that once we get RWeka and FSelector in the same units, they agree on two variables, but not the other two. There is still something else going on. &lt;/p&gt;&#xA;" OwnerUserId="26917" LastEditorUserId="26917" LastEditDate="2017-01-20T15:25:53.157" LastActivityDate="2017-01-20T15:25:53.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16250" PostTypeId="2" ParentId="16247" CreationDate="2017-01-11T05:28:32.180" Score="-1" Body="&lt;p&gt;Use stratified K-Fold cross validation, it tries to balance the number of positive and negative classses for each fold. Kindly look &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html&amp;amp;ved=0ahUKEwiMi4KrrbnRAhUIsI8KHZFjBcgQFghGMAM&amp;amp;usg=AFQjCNEEkNeFy4wQQU7u3xbaXoU59fh1tg&amp;amp;sig2=IgBIB-TKrSAXCWkUfCVsEw&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for the documentation and examples. If it still doesnt solve your problem of imbalance please look into SMOTE algorithm, &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://github.com/scikit-learn-contrib/imbalanced-learn&amp;amp;ved=0ahUKEwiNoabcrbnRAhXEN48KHb77DxoQFggaMAA&amp;amp;usg=AFQjCNHN3piThYQKFjMCCV5zQXllUSOK3g&amp;amp;sig2=JPkUj_M0C4ikt1dXm0TK6g&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is a scikit learn implementation of it.  &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-11T05:28:32.180" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16251" PostTypeId="1" CreationDate="2017-01-11T05:55:13.463" Score="6" ViewCount="968" Body="&lt;p&gt;I am teaching myself Information Retrieval from Christopher Manning's book (PDF link: &lt;a href=&quot;http://nlp.stanford.edu/IR-book/pdf/01bool.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://nlp.stanford.edu/IR-book/pdf/01bool.pdf&lt;/a&gt;). I tried Exercise 1.13:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Try using the Boolean search features on a couple of major web search engines. For&#xA;instance, choose a word, such as burglar, and submit the queries (i) burglar, (ii) burglar&#xA;AND burglar, and (iii) burglar OR burglar. Look at the estimated number of results and&#xA;top hits. Do they make sense in terms of Boolean logic? Often they haven’t for major&#xA;search engines. Can you make sense of what is going on?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By my knowledge of Boolean logic, the number of results should be like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;burglar AND burglar &amp;lt;= burglar OR burglar = burglar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this isn't so. In fact, on Google, it is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;burglar &gt; burglar OR burglar &gt; burglar AND burglar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what exactly is happening behind the scenes? Any pointers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: This is NOT a homework problem, even though it is from the exercise of a textbook.&lt;/p&gt;&#xA;" OwnerDisplayName="user21595" LastEditorUserId="13727" LastEditDate="2017-01-11T10:24:10.210" LastActivityDate="2017-01-12T09:09:18.613" Title="Why do popular search engines not follow the usual AND, OR logic for queries?" Tags="|information-retrieval|search|search-engine|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16252" PostTypeId="1" CreationDate="2017-01-11T06:08:45.480" Score="0" ViewCount="1979" Body="&lt;p&gt;I am trying to work with the &lt;a href=&quot;http://ufldl.stanford.edu/housenumbers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Street View House Numbers Dataset&lt;/a&gt; in Python. To load the .mat file digitStruct.mat in python I am using H5py&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import h5py&#xA;labels_file = './sv/train/digitStruct.mat'&#xA;f = h5py.File(labels_file)&#xA;struct= f.values()&#xA;names = struct[1].values()&#xA;print(names[1][1].value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get &lt;code&gt;[&amp;lt;HDF5 object reference&amp;gt;]&lt;/code&gt; but I need to know the actual string&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-03-12T18:15:15.563" Title="Loading Street View House Numbers Dataset in Python" Tags="|python|matlab|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16253" PostTypeId="2" ParentId="16251" CreationDate="2017-01-11T07:28:17.787" Score="5" Body="&lt;p&gt;Nice question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An exact answer should be given by looking in the search engine source code but here is a possible explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run the queries at Google&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;burglar 33,800,000&lt;/li&gt;&#xA;&lt;li&gt;burglar AND burglar  29,200,000&lt;/li&gt;&#xA;&lt;li&gt;burglar OR burglar 26,500,000&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The results indeed do not respect the expected Boolean relation burglar AND burglar &amp;lt;= burglar OR burglar = burglar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, that is since the search engine doesn't process the &quot;and&quot; and &quot;or&quot; as binary operator but just as search tokens.&#xA;Looking for them we get&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;And 25,270,000,000&lt;/li&gt;&#xA;&lt;li&gt;Or  16,320,000,000&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A term alone appears most times. &quot;and&quot; is more common than &quot;or&quot; so a term with &quot;and&quot; is more common than the term with &quot;or&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;burglar burglar 29,000,000  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Apparently looking for documents in which the term appears twice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, &lt;a href=&quot;https://support.google.com/websearch/answer/2466433?hl=en&quot; rel=&quot;noreferrer&quot;&gt;Google's Search operators documentation&lt;/a&gt; claim that &quot;OR&quot; should indeed act as a binary operator. &#xA;You found a case in which they fail to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this behaviour is very specific to search engine.&#xA;In Bing you get the following results:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;burglar 4,400,000 &lt;/li&gt;&#xA;&lt;li&gt;burglar AND burglar  1,610,000&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;burglar OR burglar 1,610,000&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And 10,400,000,000&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or  3,750,000,000&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;burglar burglar 1,610,000  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The number of results is similar for  &quot;burglar AND burglar&quot;, &quot;burglar OR burglar&quot; and &quot;burglar burglar&quot; though we see that &quot;And&quot; is more popular than &quot;OR&quot;. It seems that Bing treatments is the removal of &quot;And&quot; and &quot;OR&quot;, possibly as &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;noreferrer&quot;&gt;stop words&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://msdn.microsoft.com/en-us/library/ff795628.aspx&quot; rel=&quot;noreferrer&quot;&gt;Bing documentation&lt;/a&gt; suggest the operators &quot;&amp;amp;&amp;amp;&quot; for &quot;and&quot; and &quot;||&quot; for &quot;or&quot;.&#xA; - burglar || burglar  4,400,000 = burglar&#xA; - burglar &amp;amp;&amp;amp; burglar  1,610,000 = burglar burglar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These results fit the claim that when a term appears twice in the search query it should appear at least twice in the document too.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2017-01-12T09:09:18.613" LastActivityDate="2017-01-12T09:09:18.613" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16254" PostTypeId="2" ParentId="16237" CreationDate="2017-01-11T07:46:26.930" Score="1" Body="&lt;p&gt;Don't use machine-learning for this - you probably don't have millions of training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, what you are looking for is a simple &lt;strong&gt;depth-first search&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Begin at the root, then define a priority list, e.g. below-left, below, below-right, in which order to proceed. Don't visit nodes twice, but when first visiting them mark them as seen, and add them to your tree instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, define a priority pattern of how to define the parent node.  Construct a tree based on this single &quot;best parent&quot; relationship,, but make sure you don't create cycles.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-01-11T07:46:26.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16255" PostTypeId="2" ParentId="6318" CreationDate="2017-01-11T08:15:55.613" Score="0" Body="&lt;p&gt;Have a look at : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-learn-data-science-python-scratch-2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-learn-data-science-python-scratch-2/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has a step by step tutorial which will give you an idea of the entire process of Data Exploration, Data Analysis and Building  a predictive model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Explanation regarding Data Exploration and Feature Engineering ( how to choose relevant features ) is here : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the first 5 Datasets here which have tutorials and work on them to get practical experience:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/10/17-ultimate-data-science-projects-to-boost-your-knowledge-and-skills/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com/blog/2016/10/17-ultimate-data-science-projects-to-boost-your-knowledge-and-skills/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also have a look at : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/machine-learning-in-python-step-by-step/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/machine-learning-in-python-step-by-step/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where he uses multiple models on a single dataset which will give you a basic level understanding of different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For understanding more about the model selection have a look at this : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Data-Science-How-do-Data-Scientists-perform-model-selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/Data-Science-How-do-Data-Scientists-perform-model-selection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above link has answers given by people working in the field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get insights on different datasets you can always log into kaggle and go through competitions and have a look at the wide range of datasets, where you have access to people's code in kernels. The forums in Kaggle are helpful as people discuss about using different models for a problems and their approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="27760" LastActivityDate="2017-01-11T08:15:55.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16256" PostTypeId="2" ParentId="16251" CreationDate="2017-01-11T08:30:40.540" Score="4" Body="&lt;p&gt;Google &lt;em&gt;used&lt;/em&gt; to do, to some extend. For a long time, using +word could be used to require the presence of a word. So &quot;a AND b&quot; would be &quot;+a +b&quot; whereas &quot;a OR b&quot; would be &quot;a b&quot; (with a preference to both occurring).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But people did not use it much, so they eventually removed it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google thinks it is more important to be able to process &lt;em&gt;natural language queries&lt;/em&gt; rather than some mathematical formalism less than 0.1% of the users understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although there are also some other hypotheses why it was removed: &lt;a href=&quot;https://productforums.google.com/forum/#!topic/websearch/H4XbbwWmtAY&quot; rel=&quot;noreferrer&quot;&gt;Why was the Plus Sign (+) removed as a Search Operator?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-01-11T10:07:39.640" LastActivityDate="2017-01-11T10:07:39.640" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16257" PostTypeId="2" ParentId="16252" CreationDate="2017-01-11T09:28:54.290" Score="0" Body="&lt;p&gt;I have not checked the dataset but this is what I suggest. &#xA;Have a look at &lt;a href=&quot;http://docs.h5py.org/en/latest/high/group.html#Group.get&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; method - &lt;em&gt;get&lt;/em&gt; to extract. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from this, I also suggest checking &lt;a href=&quot;http://www.mathworks.com/help/matlab/matlab-engine-for-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matlab engine&lt;/a&gt; for python.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eng = matlab.engine.start_matlab()&#xA;data = eng.load(labels_file,nargout=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-01-11T09:28:54.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16258" PostTypeId="2" ParentId="16246" CreationDate="2017-01-11T12:06:18.777" Score="6" Body="&lt;p&gt;Text-to-speech suffers from problems similar to the drivers of &quot;&lt;a href=&quot;http://www.tsdconference.org/tsd2014/download/preprints/673.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;uncanny valley&lt;/a&gt;&quot; effects when viewing faces of humanoid robots and computer renderings of faces. Our ability to distinguish depth of meaning, emotional content and other subtle cues makes humans very sensitive to small details in audio containing spoken language. Note this is not quite the same as claiming text-to-speech actually &lt;em&gt;has&lt;/em&gt; an &quot;uncanny valley&quot;; there doesn't appear to be much analysis of that claim either way around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Existing text-to-speech systems have two basic generative processes: &lt;em&gt;Concatenative&lt;/em&gt; models are essentially databases of phoneme samples that the system strings together. These suffer from lack of flexibility. &lt;em&gt;Parametric&lt;/em&gt; models attempt to represent sound generation at a lower level to improve on this, but it is quite a challenge to create a model that includes all the fine detail that we notice. For instance, we notice many non-verbal cues including imperfections such as mouth noise, breathing etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There has been recent progress in this area by researchers running a very detailed generative model - one that literally generates sound, sample by sample, after consuming a large amount of training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://deepmind.com/blog/wavenet-generative-model-raw-audio/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This report on WaveNet by DeepMind team&lt;/a&gt; explains the different synthesis techniques used to date and shows off the capabilities of the new approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process creates state-of-the-art results, and it is easy to hear the improvement from examples on the site. However, it is too computationally intensive to be used in real-time systems yet. Give it a few years to be refined, and it could be the basis of far better text-to-speech systems.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The above written in January 2017. In the intervening time (it is now October 2017) the DeepMind team have been working on the efficiency of their model, and &lt;a href=&quot;https://deepmind.com/blog/wavenet-launches-google-assistant/&quot; rel=&quot;nofollow noreferrer&quot;&gt;it is now much faster and sounds even better&lt;/a&gt;. This is close to becoming a solved problem, albeit with some proprietary ownership. Give it a little while longer though and this breakthrough will allow real-time and natural-sounding parametric voice models in many applications.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-04T21:06:35.387" LastActivityDate="2017-10-04T21:06:35.387" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16259" PostTypeId="2" ParentId="16159" CreationDate="2017-01-11T12:36:28.527" Score="1" Body="&lt;p&gt;&lt;strong&gt;Object recognition&lt;/strong&gt; is a subfield of Computer Vision which consists of methods/algorithms aimed at determining whether a given object is present on a 2D image. Many object recognition systems just return a 'yes/no' answer, but there are also many methods that return the location of the object on the image by means of a bounding box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This subfield comprises a vast range of methods, from low-complexity ones (like methods based solely on applying simple filters on the image, like colour/edges/etc.) to more complex ones in which machine learning methods (like deep learning) play a very important role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the complexity of the objects you are planning to detect, and on the overall complexity of the images, machine learning may be completely unnecessary.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Outline_of_object_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;This Wikipedia page&lt;/a&gt; summarises many of these methods. You can start exploring from there, or you can start by using any introductory computer vision text. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-01-11T12:36:28.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16260" PostTypeId="2" ParentId="16186" CreationDate="2017-01-11T20:41:55.073" Score="1" Body="&lt;p&gt;I have solved this to a certain extent.  xgb is able to optimise the problem without altering the objective function.  I still don't understand when you should pass in your own function as a parameter but it looks like it isn't necessary.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There were some bugs in my kelly_error function which I have fixed below, including changing the code to optimise for ROI.  You will also need to tell xgb to maximise the output (default is to minimise)&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def kelly_error(odds_train, probs_train, odds_test, probs_test):&#xA;    def evalerror(preds, dmatrix):&#xA;        bet_outcome = dmatrix.get_label()&#xA;&#xA;        odds = odds_train if len(bet_outcome) == len(odds_train) else odds_test&#xA;        probs = probs_train if len(bet_outcome) == len(probs_train) else probs_test&#xA;&#xA;        kelly_fraction = (probs * (odds + 1) - 1) / odds&#xA;&#xA;        def value_bets(f):  # ignore any bets with a negative kelly fraction&#xA;            return 0 if f &amp;lt; 0 else f&#xA;&#xA;        kelly_fraction = np.array([value_bets(x) for x in kelly_fraction])&#xA;&#xA;        profit = preds * bet_outcome * kelly_fraction * odds&#xA;        loss = preds * (1 - bet_outcome) * kelly_fraction&#xA;&#xA;        stake = float(sum(preds * kelly_fraction))&#xA;        total_profit = float(sum(profit - loss))&#xA;&#xA;        roi = 100 * total_profit / stake&#xA;        return 'roi', roi&#xA;&#xA;    return evalerror&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27724" LastActivityDate="2017-01-11T20:41:55.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16261" PostTypeId="1" AcceptedAnswerId="16271" CreationDate="2017-01-11T20:43:36.357" Score="1" ViewCount="263" Body="&lt;p&gt;I have a seq2seq conversational model (based on &lt;a href=&quot;https://github.com/suriyadeepan/easy_seq2seq&quot; rel=&quot;nofollow noreferrer&quot;&gt;this implementation&lt;/a&gt;) trained on the Cornell movie dialogs.&#xA;Now I want to fine-tune it on a much smaller dataset. The new data comes with the new words, and I want UNKs for as few new words as possible. So I'm going to create a new network with respect to the new input/output sizes, and I'm going to initialize its submatrices with learned weights I have at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you say if this method can cause problems with the resulting model's performance? E.g. are the softmaxes likely to be affected significantly with these new initially untrained weights?&#xA;And if it's OK, do you have some examples on how to do it with the least pain in tensorflow's seq2seq setup?&lt;/p&gt;&#xA;" OwnerUserId="27864" LastActivityDate="2017-01-12T14:50:09.510" Title="Extending a trained neural network for a larger input" Tags="|python|neural-network|nlp|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16262" PostTypeId="1" CreationDate="2017-01-11T22:41:30.513" Score="1" ViewCount="54" Body="&lt;p&gt;Looking for books (chapters?)/reviews/papers on the task of recommending a few (possibly non-independent) items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;: office supply shop recommending a person a &lt;strong&gt;pre-compiled package of items&lt;/strong&gt; of different kind (pens/pencils/erasers/paper/...) based on one's and other users purchases and views. The total cost of the package may be limited; other constraints could be introduced as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hints?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="14048" LastActivityDate="2017-01-11T22:41:30.513" Title="Books/reviews/papers on recommending groups of items?" Tags="|recommender-system|books|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16263" PostTypeId="2" ParentId="16232" CreationDate="2017-01-11T23:29:16.020" Score="9" Body="&lt;p&gt;David, you can use mean average precision ('map') or even better logloss ('logloss'). Yes, for unbalanced data precision and recall are very important. I would suggest individually examining these metrics after optimizing with whatever eval_metric you choose.Additionally, there is a parameter called scale_pos_weight, which will help tell the model the distribution of you data. I have found this to greatly improve the performance of &quot;rare event&quot; cases. The following markdown doc has a list of all the parameters and their options. &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-11T23:29:16.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16264" PostTypeId="2" ParentId="16229" CreationDate="2017-01-12T00:31:21.613" Score="0" Body="&lt;p&gt;I am trying to figure out the same problem as well. I don't have the full solution but maybe we can help each other out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do: &lt;code&gt;names =[n.name for n in tf.get_default_graph().as_graph_def().node]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give you a list of all the names of the nodes in the graph. For Rnn the out put for &lt;code&gt;names&lt;/code&gt; should look some what like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;.&#xA;'RNN/Shape',&#xA;'RNN/strided_slice/stack',&#xA;'RNN/strided_slice/stack_1',&#xA;.&#xA;. &#xA;.&#xA;'RNN/transpose'&#xA;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can use &lt;code&gt;get_operation_by_name&lt;/code&gt; to get the operation you want to look at. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rnn_transpose=tf.get_default_graph().get_operation_by_name('RNN/transpose').outputs[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From that you can run the &lt;code&gt;eval()&lt;/code&gt; with your regular &lt;code&gt;sess&lt;/code&gt; and &lt;code&gt;feed_dict&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; rnn_transpose.eval(session=sess,feed_dict =feed_dict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem here is now we have to find which operations in the graph correspond to the proper gate operations. &lt;/p&gt;&#xA;" OwnerUserId="19263" LastActivityDate="2017-01-12T00:31:21.613" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16265" PostTypeId="1" CreationDate="2017-01-12T00:58:27.320" Score="15" ViewCount="15218" Body="&lt;p&gt;I use Python to run a random forest model on my imbalanced dataset (the target variable was a binary class). When splitting the training and testing dataset, I struggled whether to used stratified sampling (like the code shown) or not. So far, I observed in my project that the stratified case would lead to a higher model performance. But I think if I will use my model to predict the new cases which would highly probably differ in the distribution of target class with my current dataset. So I inclined to loosen this constrain and use the unstratified split. &#xA;Could anyone can advice to clarify this point?   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train,test=train_test_split(myDataset, test_size=0.25, stratify=y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26163" LastEditDate="2017-01-12T08:00:40.707" LastActivityDate="2017-01-12T08:00:40.707" Title="Is stratified sampling necessary (random forest, Python)?" Tags="|machine-learning|python|random-forest|sampling|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16266" PostTypeId="1" CreationDate="2017-01-12T01:02:43.547" Score="1" ViewCount="5473" Body="&lt;p&gt;I have a dataset consisting of 181 samples(&lt;strong&gt;classes are not balanced there are 41 data points with 1 label and rest 140 are with label 0)&lt;/strong&gt; and 10 features and one target variable. The 10 features are numeric and continuous in nature. I have to perform binary classification. I have done the following work:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I have performed 3 Fold cross validation and got following accuracy results using various models:-&#xA;&#xA;LinearSVC:&#xA;0.873&#xA;DecisionTreeClassifier:&#xA;0.840&#xA;Gaussian Naive Bayes:&#xA;0.845&#xA;Logistic Regression:&#xA;0.867&#xA;Gradient Boosting Classifier&#xA;0.867&#xA;Support vector classifier rbf:&#xA;0.818&#xA;Random forest:&#xA;0.867&#xA;K-nearest-neighbors:&#xA;0.823&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please guide me how could I choose the best model for this size of dataset and make sure my model is not overfitting ? I am thinking of applying random under sampling to handle the unbalanced data.&lt;/p&gt;&#xA;" OwnerUserId="27868" LastEditorUserId="27868" LastEditDate="2017-01-13T02:43:33.767" LastActivityDate="2019-03-20T01:41:56.327" Title="Binary Classification on small dataset &lt; 200 samples" Tags="|machine-learning|python|classification|predictive-modeling|scikit-learn|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16268" PostTypeId="1" AcceptedAnswerId="16270" CreationDate="2017-01-12T03:40:27.480" Score="1" ViewCount="148" Body="&lt;p&gt;Im new to ML, I have a data set for Music sales info for Vinyls, the data set contains:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Author &lt;/li&gt;&#xA;&lt;li&gt;Album Title &lt;/li&gt;&#xA;&lt;li&gt;Genre&lt;/li&gt;&#xA;&lt;li&gt;Country&lt;/li&gt;&#xA;&lt;li&gt;RevenueGenerated&lt;/li&gt;&#xA;&lt;li&gt;AverageRevenueGenerated&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My goal is to create a Model which I can help me understand which Music may generate a lot of revenue (Boolean). I created a field AverageRevenueGenerated which is the average of all Revenue generated for all artists.&#xA;Im looking for a tool that can help me associate or generate insights based on input signals above. This cold be automatically or a specific guide that allows me to say for example if:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;UK + Industrial &lt;/li&gt;&#xA;&lt;li&gt;IT + Opera&lt;/li&gt;&#xA;&lt;li&gt;Daft Punk + Electro&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Will be potential high revenues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found house prices example: &lt;a href=&quot;https://yalantis.com/blog/predictive-algorithm-for-house-price/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://yalantis.com/blog/predictive-algorithm-for-house-price/&lt;/a&gt; is it the same type of problem?  I'm looking which input signals may be the highest revenue. Any insights or pointers will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2017-01-12T06:38:09.893" Title="Binary classification problem" Tags="|predictive-modeling|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16269" PostTypeId="2" ParentId="16265" CreationDate="2017-01-12T06:13:20.880" Score="14" Body="&lt;p&gt;If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class proportion is same as of the whole dataset, which is the &lt;strong&gt;right thing&lt;/strong&gt; to do. If your classes are balanced then a shuffle (no stratification needed here) can basically guarantee a fair test and train split. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now your model will be capable or at least enough equipped to predict the outnumbered class (class with lesser points in number). That is why instead of just calculating Accuracy, you have been given other metrics like &lt;a href=&quot;https://en.wikipedia.org/wiki/Sensitivity_and_specificity#Sensitivity&quot; rel=&quot;noreferrer&quot;&gt;Sensitivity and Specificity&lt;/a&gt;. Keep a watch on these, these are the guardians.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2017-01-12T07:17:15.550" LastActivityDate="2017-01-12T07:17:15.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16270" PostTypeId="2" ParentId="16268" CreationDate="2017-01-12T06:32:58.503" Score="1" Body="&lt;p&gt;Yes, you can run a regression using the features you have, and predict the revenue but you will need more features to run an effective analysis. Maybe adding features for the genre and wether the music is trending or not. To turn it into a binary classification problem you will have to decide on a cut-off and label everything above that as 1 and below it as 0, the median might be a good cut-off but solely depends on your data. This entire problem will boil down to feature engineering, which in my view is one of the hardest thing to do. Also you will need a lot of data for this. You can create features wether the song was featuted on the billboard. See, try to think of it like a human and add such features to your model. Like as a person I think the music that will generate more revenue will be :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Popularity and Reputation of the artist. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Wether the genre is trending these days. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Has the song won any awards. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Stuff like this. So you wilk have to find features that quantify these things. Try to find some open source datatset for this kind of problem and work from there. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="21608" LastEditDate="2017-01-12T06:38:09.893" LastActivityDate="2017-01-12T06:38:09.893" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16271" PostTypeId="2" ParentId="16261" CreationDate="2017-01-12T06:52:05.480" Score="1" Body="&lt;p&gt;Its okay as long as the nerwork you are planning to create has the same number of layers and units i.e the dimensions of your network must be compatible with the weights that you are borrowing from the trained model. Also it would be better if you follow the second blog post of suriyadeepan practical seq2seq where he trains a conversation model on twitter chat. The code is much simpler and easier to understand, also it is on a smaller dataset, also he mentioned that the bot trained on cornell movie dialog corpus wasnt performing so well. Mainly to use the pre-trained weights all you have to do is load the model, create placeholders for the weights, assign thr weights from the loaded model to the placeholders and run a forward pass. This &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=/amp/s/nathanbrixius.wordpress.com/2016/05/24/checkpointing-and-reusing-tensorflow-models/amp/&amp;amp;ved=0ahUKEwjxlaWWgrzRAhXHQ48KHTF3C10QFggsMAE&amp;amp;usg=AFQjCNEhPsE6cCxQqVAK9dhod_kykgwWwg&amp;amp;sig2=1lBUIU2MeZsCpm4U9XAXlg&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; and this &lt;a href=&quot;https://stackoverflow.com/questions/36193553/get-the-value-of-some-weights-in-a-model-trained-by-tensorflow&quot;&gt;question&lt;/a&gt; might help you with this task &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-01-12T06:52:05.480" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16272" PostTypeId="1" CreationDate="2017-01-12T07:53:26.733" Score="6" ViewCount="1864" Body="&lt;p&gt;Does RELU means to change pixel value to 0 if it is negative anywhere , and later if we apply maximum pooling then what is the use of RELU because in this step we choose maximum value so no matter it is negative or zero if there is any positive value must be greater then 0 or negative number. &lt;/p&gt;&#xA;" OwnerUserId="27607" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:43.327" LastActivityDate="2021-02-16T15:24:43.327" Title="RELU vs Pooling" Tags="|deep-learning|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16273" PostTypeId="2" ParentId="16272" CreationDate="2017-01-12T08:26:34.697" Score="4" Body="&lt;p&gt;Yes RELU means changing the pixel value to zero if it is negative. It matters if we apply RELU or not before max pooling. Let us say your max pooling window is of size 2x2. If an image has only negative values in 2x2 square at particular location then RELU plus max pooling gives a zero as output where as only max pooling would give the maximum negative number.&lt;/p&gt;&#xA;" OwnerUserId="26050" LastActivityDate="2017-01-12T08:26:34.697" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16274" PostTypeId="2" ParentId="11131" CreationDate="2017-01-12T09:14:38.520" Score="1" Body="&lt;p&gt;Let me start with an important point; &lt;strong&gt;what other criteria than classification success can U use to identify my cluster separation performance?&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Classification as an indicator for clustering performance has an internal paradox. If you have the classification the clustering question does not apply anymore. These two concept are coming form two totally different philosophies so I would say be careful if you have already understood the concepts (what you say may make sense in semi-supervised learning which is not in your tags so I assume there is a misunderstanding).&lt;/li&gt;&#xA;&lt;li&gt;Clustering has no performance evaluation! this is the problem I see most of data scientists struggling with. In practice you may define a &lt;em&gt;good&lt;/em&gt; criterion (but honestly, what is &lt;em&gt;good&lt;/em&gt;?!!) and deliver your solution, but in research schema there is no evaluation for clustering as the question itself is not well-defined i.e. you never have label to be sure who is who so you need to define &lt;em&gt;closeness&lt;/em&gt; of points from which the problem starts; &lt;em&gt;how close is called closeness?!!&lt;/em&gt;  &lt;/li&gt;&#xA;&lt;li&gt;Be careful about &lt;em&gt;Curse of Dimensionality&lt;/em&gt; while using k-means for time-series clustering (I'm not sure how you do it).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After these points let's have a look at your question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are time-series? If time-series are pretty non-stationary or simply speaking is the dynamic behind variation is complicated enough, then there is not a one-to-one map for time-series characteristics and time points (imagine a simple &lt;a href=&quot;https://en.wikipedia.org/wiki/Electrocardiography&quot; rel=&quot;nofollow noreferrer&quot;&gt;ECG signal&lt;/a&gt;. It's pretty simple but if you explore research community you will find super sophisticated methods for feature extraction on ECGs. I'm pretty confident finding a time point at which ECGs differ is almost impossible.). You may extract features from your time series or embed it into some n-dimensional manifolds and look at it. In best case you may find some time-related features which describe your time-series and you may find some time-related criteria at which time-series differ (however I'd say it's not likely to find them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming time-series are pretty well-behaved (!!) with a simple dynamic (should be super simple). Then a solution might be to define a distance function of time-series which outputs the pair-wise distances of all time-series as a single score. Then the maximum of this function returns the time-point at which these time-series are pretty distinguishable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you provide more information on your data I might be able to give a more detailed precise answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-01-12T09:14:38.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16275" PostTypeId="2" ParentId="15140" CreationDate="2017-01-12T09:23:28.397" Score="2" Body="&lt;p&gt;Super comprehensive question. You are basically asking for tones of directions! I'd suggest to start with link prediction problem. So assume you have a directed/signed/weighted graph and you want to find the most probable next interaction (link/edge). I'd suggest to have a look at &lt;a href=&quot;https://scholar.google.com/citations?hl=en&amp;amp;user=tyyAqKsAAAAJ&amp;amp;view_op=list_works&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jerome Kunegis&lt;/a&gt; papers and you may see it as a starting point and go on from there. &lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-01-12T09:23:28.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16276" PostTypeId="1" CreationDate="2017-01-12T11:28:45.477" Score="1" ViewCount="73" Body="&lt;p&gt;I am working on to implement the approach in the paper &lt;a href=&quot;https://arxiv.org/pdf/1603.03101v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1603.03101v1.pdf&lt;/a&gt; on detecting text in wild using recursive convolution neural net and attention modelling. Being a bit new to deep nets and have developed the following interpretation about the recursive CNN given in the paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is given in the paper that while training could be done by &quot; reusing the same convolutional weight matrix multiple times at each layer.&quot; Does this means the following thing in the pseudo code below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;layer1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;iterating multiple times over { train convolution_layer(weights) with labels&#xA;}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;layer2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;iterating multiple times over { train convolution_layer(weights) with labels }&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here in the consecutive iteration in a layer we use weights of previous iteration. We use different weight initialisation for different layers here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While using the network for test set, we will input the weights we got from the last iterations.&#xA;Please comment on this interpretation and I would be grateful if anyone could suggest some links and literature for the same. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2017-09-03T08:58:32.680" Title="Implementation and interpretation of recursive convolution neural net(RCNN)" Tags="|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16280" PostTypeId="1" CreationDate="2017-01-12T19:35:44.177" Score="5" ViewCount="4517" Body="&lt;p&gt;I am trying to create an autoencoder where a few inputs can affect the other inputs, but not the other way around. An example of this is given by the following picture. I looked into defining my own Keras layers, but don't know how to proceed. &lt;a href=&quot;https://i.stack.imgur.com/rA9R1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rA9R1.png&quot; alt=&quot;Custom selectively connected neural network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21262" LastActivityDate="2017-01-23T04:13:21.380" Title="Keras create custom network with certain unconnected neurons" Tags="|neural-network|keras|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16281" PostTypeId="2" ParentId="16266" CreationDate="2017-01-12T21:17:08.113" Score="2" Body="&lt;p&gt;This &lt;a href=&quot;https://stats.stackexchange.com/questions/81576/how-to-judge-if-a-supervised-machine-learning-model-is-overfitting-or-not&quot;&gt;post&lt;/a&gt; might be of interest. Basically by selecting the model with the best crossvalidation score, you already account for overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you should separate your dataset into two parts. For the first one (validation) you run the crossvalidation on to select a model, in your case LinearSVC. For the second one (testing) you run crossvalidation again, but this time only with LinearSVC to get unbiased estimates of the accuracy.  &lt;/p&gt;&#xA;" OwnerUserId="21262" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-01-12T21:17:08.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16283" PostTypeId="1" CreationDate="2017-01-12T22:14:32.763" Score="3" ViewCount="214" Body="&lt;p&gt;I am using the Darknet Convolutional Neural Networks to detect people (as in, humans) and furniture in a single image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I train the model twice, one for people, one for furniture. I seem to get better results by lowering the threshold opposed to training the model once on both people and furniture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is this: would this be considered an anti-pattern? Or is it common practice to solve my problem like this?&lt;/p&gt;&#xA;" OwnerUserId="27896" LastEditorUserId="25157" LastEditDate="2017-01-13T12:29:32.477" LastActivityDate="2017-01-13T12:29:32.477" Title="Multiple models vs. Single model for prediction" Tags="|predictive-modeling|training|model-selection|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16284" PostTypeId="2" ParentId="8941" CreationDate="2017-01-13T00:09:12.180" Score="1" Body="&lt;p&gt;&lt;em&gt;Disclosure&lt;/em&gt;: This message is posted by The Data Science Conference® team.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the idea of a no-sponsor, no-vendor, no-recruiter data science conference is appealing to you, The Data Science Conference® might be a good option. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link: &lt;a href=&quot;http://www.thedatascienceconference.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.thedatascienceconference.com/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="27900" LastActivityDate="2017-01-13T00:09:12.180" CommentCount="0" CommunityOwnedDate="2017-01-13T00:09:12.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="16285" PostTypeId="1" AcceptedAnswerId="16294" CreationDate="2017-01-13T03:20:03.703" Score="1" ViewCount="906" Body="&lt;p&gt;What is the difference between subgradient svm and kernel svm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my understanding subgradient svm is a linear classifier that uses hinge loss and kernel svm  uses some kernel function for non linear classification. I was wondering, if subgradient is only a linear classifier, could I use a tree of non linear svms to separate non linear data? I would essentially do binary classification separate 1 class vs the rest then the child node of the tree would work on the rest and separate the next class from the rest and so on.  Any general idea or feedback would be great. &lt;/p&gt;&#xA;" OwnerUserId="27902" LastActivityDate="2017-01-13T11:08:42.250" Title="Difference between subgradient SVM and kernel SVM?" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16286" PostTypeId="2" ParentId="16266" CreationDate="2017-01-13T04:17:13.903" Score="1" Body="&lt;p&gt;Firstly your data's amount is very small for any kind of analysis, so if it was posssible to get more data then that would be better. Secondly as you mentioned that your data was imbalanced then the accuracy metrics you have posted lose all meaning, since 140 samples are of the same class, the algorithm is predicting that class for every sample. So for better evaluation calculate precision, recall and f-score. Thirdly, since your data is already less than needed don't undersample, instead oversample using the SMOTE (Synthetic Minority Over Sampling Technique) &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://contrib.scikit-learn.org/imbalanced-learn/generated/imblearn.over_sampling.SMOTE.html&amp;amp;ved=0ahUKEwi38p2rob7RAhWHt48KHak3DnwQFggrMAE&amp;amp;usg=AFQjCNFGr3LlWKUCp-P8k-f2Ttq_Hk4k2w&amp;amp;sig2=fi3X9j6EgVBHaWlwCeeI6Q&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation&lt;/a&gt;. Using a stratified KFold, and a Random Forest will mostly be your best bet here. But remember with this less than needed data, it would be impossible to achieve a model without underfitting or overfitting. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="69822" LastEditDate="2019-03-20T01:41:56.327" LastActivityDate="2019-03-20T01:41:56.327" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16287" PostTypeId="2" ParentId="16280" CreationDate="2017-01-13T05:03:50.103" Score="0" Body="&lt;p&gt;Keras does give a chance to add custom layers. I do not know about the previous versions but Keras 1.1.3 is flexible to do this. Have a look at this &lt;a href=&quot;https://keras.io/layers/writing-your-own-keras-layers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;. This also contains a good example for a custom layer. Here, you can try to change the weight matrix according to your need. I'd also suggest you to try reading source code of some basic layers to understand it better.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-01-13T05:03:50.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16288" PostTypeId="1" CreationDate="2017-01-13T05:21:48.103" Score="1" ViewCount="391" Body="&lt;p&gt;I'm trying to map tags/attributes from xml files from different sources. I've found a paper called &lt;a href=&quot;https://www.researchgate.net/publication/2520270_Semantic_Mapping_of_XML_Tags_using_Inductive_Machine_Learning?enrichId=rgreq-5b2cac5993ee9c786bd3203650c98216-XXX&amp;amp;enrichSource=Y292ZXJQYWdlOzI1MjAyNzA7QVM6MTAyNzcyNTAyNDMzODA5QDE0MDE1MTQyODI4MzM%3D&amp;amp;el=1_x_3&amp;amp;_esc=publicationCoverPdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Semantic Mapping of XML Tags using Inductive Machine Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among various things, it also talks about a Rule Generation Algorithm called &lt;a href=&quot;https://books.google.co.in/books?id=YvTxwaLJJ2kC&amp;amp;pg=PA394&amp;amp;lpg=PA394&amp;amp;dq=data%20reduction%20via%20prototypical%20concept%20learning&amp;amp;source=bl&amp;amp;ots=lYaHb2xgMG&amp;amp;sig=VsIwE5-T4H1gMfJ0RV9sDThbtlw&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwiJ0LDu9LvRAhWMtY8KHRyXDxQQ6AEIJjAC#v=onepage&amp;amp;q=data%20reduction%20via%20prototypical%20concept%20learning&amp;amp;f=false&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Squeezer&lt;/a&gt;.&#xA;However, I'm wondering what other (better) algorithms can be used for deep learning xml tags.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also do we have any such algorithm implemented in &lt;a href=&quot;http://tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow&lt;/a&gt; or any other open source library ?&lt;/p&gt;&#xA;" OwnerUserId="27872" LastActivityDate="2017-01-13T05:21:48.103" Title="Mapping xml tags by Rule Learning/Generation Algorithms" Tags="|machine-learning|tensorflow|association-rules|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16289" PostTypeId="2" ParentId="8941" CreationDate="2017-01-13T06:46:55.633" Score="1" Body="&lt;p&gt;Have a look at : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.kdnuggets.com/meetings/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/meetings/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has a list of conferences and meetings of Data Science split by each month.&lt;/p&gt;&#xA;" OwnerUserId="27760" LastActivityDate="2017-01-13T06:46:55.633" CommentCount="0" CommunityOwnedDate="2017-01-13T06:46:55.633" ContentLicense="CC BY-SA 3.0" />
  <row Id="16291" PostTypeId="2" ParentId="15697" CreationDate="2017-01-13T07:49:13.187" Score="1" Body="&lt;p&gt;You could give the &lt;code&gt;plotly&lt;/code&gt;package a try. You basically code up the plot you want first with &lt;code&gt;ggplot2&lt;/code&gt; and then call &lt;code&gt;ggplotly()&lt;/code&gt; at the end, which will render an interactive version of it. You get zooming in on mouse selection for example and labels of points on hover. It has a pretty extensive documentation too. I can try and help you out with a specific example if you post some code. Another cool package you can check out is &lt;code&gt;trelliscopejs&lt;/code&gt;. hth&lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2017-01-13T07:49:13.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16292" PostTypeId="1" CreationDate="2017-01-13T07:57:43.317" Score="9" ViewCount="3159" Body="&lt;p&gt;I was recently thinking about the memory cost of (a) training a CNN and (b) inference with a CNN. Please note, that I am not talking about the storage (which is simply the number of parameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How much memory does a given CNN (e.g. VGG-16 D) need for &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(a) Training (with ADAM)&lt;/li&gt;&#xA;&lt;li&gt;(b) Inference on a single image?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;My thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Basically, I want to make sure that I didn't forget anything with this question. If you have other sources which explain this kind of thought, please share them with me.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;(a) Training&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;For training with ADAM, I will now assume that I have a Mini-batch size of &lt;span class=&quot;math-container&quot;&gt;$B \in \mathbb{N}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$w \in \mathbb{N}$&lt;/span&gt; is the number of parameters of the CNN. Then the memory footprint (the maximum amount of memory I need at any point while training) for a single training pass is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$2w$&lt;/span&gt;: Keep the weights and the weight updates in memory&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$B \cdot $&lt;/span&gt; Size of all generated feature maps (forward pass)&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt;: Gradients for each weight (backpropagation)&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt;: Learning rates for each weight (ADAM)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;(b) Inference&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In inference, it is not necessary to store a feature map of layer &lt;span class=&quot;math-container&quot;&gt;$i-1$&lt;/span&gt; if the feature maps of layer &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; are already calculated. So the memory footprint while inference is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt;: The model&lt;/li&gt;&#xA;&lt;li&gt;The two most expensive successive layers (one which is already calculated, the net one which gets calculated)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:33.570" LastActivityDate="2021-02-16T15:24:33.570" Title="What is the memory cost of a CNN?" Tags="|cnn|convolutional-neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="16294" PostTypeId="2" ParentId="16285" CreationDate="2017-01-13T11:08:42.250" Score="2" Body="&lt;p&gt;You are gravely misunderstanding SVM. Sub-gradient descent algorithm for SVM is a method to solve the underlying optimization problem of SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVM is always a linear classifier, which can yet by using kernels operate in a higher dimensional space. Therefore in input space, the separating hyperplane (linear!) computed in feature space (kernel!), seems non-linear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Effectively you are thereby solving a non-linear classification task, but you are projecting into a higher dimensional feature space where the classification task is solved by a linear classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please read: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia on svm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please watch:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=_PwhiWxHK8o&quot; rel=&quot;nofollow noreferrer&quot;&gt;great lecture about SVM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24996" LastActivityDate="2017-01-13T11:08:42.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16295" PostTypeId="1" AcceptedAnswerId="16304" CreationDate="2017-01-13T12:03:41.483" Score="4" ViewCount="1405" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#Derivation_directly_in_terms_of_matrices&quot; rel=&quot;nofollow noreferrer&quot;&gt;derivation of the normal equation&lt;/a&gt; can be noted&#xA;$\theta  = (X^TX)^{-1}(X^T)y$, where $X^{-1}$ is the inverse of $X$ and can also be written $inv(X)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But why can't we write $inv(X^T X)$ as $inv(X)inv(X^T)$, and also use the definition of inverse $inv(X^T)(X^T) = I$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then $\theta  = inv(X^T X)(X^T)y = inv(X)inv(X^T)(X^T)y = inv(X)Iy = inv(X)y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why isn't the final result of normal equation in regression in machine learning written as  $\theta = inv(X) y$ after simplifying?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="836" LastEditDate="2017-01-13T19:35:35.700" LastActivityDate="2017-01-13T19:48:19.360" Title="Normal equation result simplification" Tags="|machine-learning|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16297" PostTypeId="1" AcceptedAnswerId="16300" CreationDate="2017-01-13T13:01:01.133" Score="1" ViewCount="909" Body="&lt;p&gt;I am using Factorization Machines ( libfm) and also the Field Aware Factorization Machines (libffm) for a kaggle competition.  I am currently using the single models of each respectively for prediction. &#xA;I came to know that we can use ensemble of models for both FM and FFM.&#xA;Can someone explain how  this is done ?&lt;/p&gt;&#xA;" OwnerUserId="26976" LastActivityDate="2017-01-13T14:15:58.033" Title="How to use ensemble of models in FM or FFM?" Tags="|ensemble-modeling|kaggle|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16298" PostTypeId="1" AcceptedAnswerId="16311" CreationDate="2017-01-13T13:49:02.133" Score="5" ViewCount="1532" Body="&lt;p&gt;This question is part of a sample exam that I'm working on to prepare for the real one. I've been stuck on this one for quite a while and can't really motivate my answers since it keeps referring to guarantees, can someone explain what guarantees we can have in a neural network such as the one described?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Network description:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume a deep neural network (DNN) that takes as input images images of handwritten digits (e.g., MNIST). The input network consists of 28x28 = 784 input units, a hidden layer of logistic units, and a softmax group of 10 units as the output layer. The loss function is the cross-entropy. We have many training cases and we always compute our weight update based on the entire training set, using the error backpropagation algorithm. We use a learning rate thats small enough for all practical purposes, but not so small that the network doesnt learn. We stop when the weight update becomes zero. For each of the following questions, answer yes or no, and explain very briefly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Will this DNN configuration enable the the weights to minimize the loss value (there may be multiple global optima)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this DNN configuration guarantee that the loss reduces on every step? We use our network to classify images by simply seeing which of the 10 output units gets the largest probability when the network is presented with the image, and declaring the number of that output unit to be the networks guessed label.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this DNN configuration guarantee that the number of mistakes that the network makes on training data (by following this classification strategy) never increases?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="27921" LastActivityDate="2017-01-15T23:36:41.027" Title="Neural Network accuracy and loss guarantees?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16300" PostTypeId="2" ParentId="16297" CreationDate="2017-01-13T14:15:58.033" Score="1" Body="&lt;p&gt;Most ensembling methods are independent from the internals of individual models, e.g. something like &lt;a href=&quot;http://www.scholarpedia.org/article/Ensemble_learning#Voting_based_methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;majority voting&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;nofollow noreferrer&quot;&gt;bagging&lt;/a&gt; or &lt;a href=&quot;http://mlwave.com/kaggle-ensembling-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;model stacking&lt;/a&gt; can be directly applicable to any model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: this was originally a comment to the OP.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-01-13T14:15:58.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16302" PostTypeId="2" ParentId="16179" CreationDate="2017-01-13T16:19:44.663" Score="0" Body="&lt;p&gt;As you observed, one can argue for any of your definitions. It is most important that you document what you mean with &quot;mean F1-score&quot;. You should also consider which of the 2 options provides a more meaningful evaluation. This depends on your specific application or task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, &quot;mean F1-score&quot; clearly means that you calculate the mean of individual F1-scores. In some situations, option 2 can be described as the overall F1-score. This depends on what you are aggregating. &quot;F1-score of the mean precision and recall&quot; might be a good general description of option 2.&lt;/p&gt;&#xA;" OwnerUserId="19255" LastActivityDate="2017-01-13T16:19:44.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16303" PostTypeId="1" CreationDate="2017-01-13T17:27:33.767" Score="1" ViewCount="19" Body="&lt;p&gt;I'm looking for real datasets which are multi-dimension time-series and there exist partial similarities between different categories based on some of the features/dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider two persons waving, but one is doing that while walking and the other while driving in a car. So their generally they are related to different motion categories,  are different but both are similar according to their arm movements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24837" LastActivityDate="2017-01-13T17:27:33.767" Title="Is there a real multi-dimension timeseies dataset in which data samples have feature based similarities?" Tags="|dataset|time-series|multilabel-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16304" PostTypeId="2" ParentId="16295" CreationDate="2017-01-13T17:39:56.933" Score="8" Body="&lt;p&gt;Only square matrices have an inverse, and neither $X$ nor $X^T$ here are (necessarily) square. I guess that's the most direct answer. $X$ may have a left-inverse $A$, such that $AX = I$. In that case, if $X\theta = y$ then $AX\theta = Ay$ so $\theta = Ay$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The direct way to construct that $A$ is just where you started though; it's $(X^TX)^{-1}X^T$. The Gramian matrix $X^TX$ is (square, and) invertible if the features (columns) of $X$ are linearly independent, and, $X^TX$ is more importantly relatively quite small and so it's feasible to invert.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You normally wouldn't compute the inverse anyway, but try to solve the system $X^TX\theta = X^Ty$ with QR decomposition or something, but that's not what you're asking.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-01-13T17:39:56.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16305" PostTypeId="2" ParentId="16298" CreationDate="2017-01-13T18:26:41.947" Score="1" Body="&lt;p&gt;A guarantee is a proof that some algorithm will &lt;em&gt;always&lt;/em&gt; have a certain performance (or the performance will never be worse than the guarantee).  In general there are not a whole lot of theoretical guarantees for NNs.  But this question is sufficiently limited in scope that some guarantees hold. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;This question is maybe a little ambiguous, but I would lean towards answering &quot;no&quot;.  The question is asking whether the training will lead to a minimum, but they don't specify whether the minimum needs to be a global or a local minimum.  If you simply descend down the gradient of the loss function (as we are doing here), you are extremely likely to end up at a local minimum, though unlikely to end up at a global minimum.  However, in practice local minima are not much worse than global minima.  I say that this is extremely likely to happen because there is a very small chance that you could accidentally land right on a saddle point.  This would be a point with a vanishing gradient that is not a local minimum, and training would stop when you land there.  (Or, even more unlikely, you could be very unlucky and start on a local maximum!)  If they're looking for guarantees, you can't guarantee that that won't happen.  But if the DNN is fairly large and the weight initialization is random, you can guarantee that it almost surely won't happen, so there is an argument to be made in favor of &quot;yes&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes.  Since we are descending the gradient of the loss function, the loss function can never increase on any step.  (This is assuming the loss function is reasonably smooth.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;EDIT: After some discussion in another answer I have decided that my original answer was incorrect.  My updated answer is below:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;No.  Although training will decrease the overall cross-entropy of the entire training set, it is possible that the cross-entropy on a single case will increase.  It is therefore possible to construct a training set such that the classifications remain the same from one step to the next for all cases except this one particular example, which moves from a correct classification to an incorrect classification.  The overall cross-entropy on the training set will decrease, but the accuracy will increase.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18416" LastEditorUserId="18416" LastEditDate="2017-01-15T23:36:41.027" LastActivityDate="2017-01-15T23:36:41.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16306" PostTypeId="1" CreationDate="2017-01-13T19:13:19.177" Score="4" ViewCount="7698" Body="&lt;p&gt;I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't the whole purpose of running scripts on PySpark to overcome the limitations of packages like Pandas? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does Pandas performance improve if it run on Spark? Is it compatible with Spark's RDD ?&lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2020-10-07T16:40:23.910" Title="Is there any use to running Pandas on Spark?" Tags="|pandas|pyspark|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16307" PostTypeId="2" ParentId="16295" CreationDate="2017-01-13T19:48:19.360" Score="4" Body="&lt;p&gt;The value of $inv(X)$ or $inv(X^T)$ cannot be calculated for non-square matrices, and usually your number of samples and features is different, so your simplification is not generic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you can simplify the functional notation in a very similar way to your suggestion using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse#Linear_least-squares&quot; rel=&quot;noreferrer&quot;&gt;pseudo-inverse&lt;/a&gt; - note that the link here is direct to the part of the article that describes this inverse as a direct solution to the linear least squares problem. Noting the pseudo inverse as $pinv(X)$ then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta = pinv(X)y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this is simply hiding the more basic derivation behind a complex function. The computation required is essentially the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice if you are using matrix library that supports inverses and pseudo-inverses, you can choose to use a longer formulation using inverses or the pseudo-inverse version and it will give near identical results.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-01-13T19:48:19.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16308" PostTypeId="2" ParentId="15699" CreationDate="2017-01-13T19:51:28.337" Score="0" Body="&lt;p&gt;There is this prevailing perception that convolution neural net is the panacea to solve all image recognition problems, which understandably comes from the many success stories in the computer vision literature. However, keep in mind what a ConvNet does for you. It finds the translational invariant features, like edges. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what you described here, a ConvNet is unlikely to do much for you. A raindrop or a dirt spot is equally likely to fall on any location of the video frames. The problem you have is one of texture classification /discrimination, and there is a huge literature on that subject. A google scholar search finds this[1].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Smith, J. R., &amp;amp; Chang, S. F. (1994, November). Transform features for texture classification and discrimination in large image databases. In Image Processing, 1994. Proceedings. ICIP-94., IEEE International Conference (Vol. 3, pp. 407-411). IEEE.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastActivityDate="2017-01-13T19:51:28.337" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16309" PostTypeId="1" CreationDate="2017-01-13T20:07:57.757" Score="2" ViewCount="418" Body="&lt;p&gt;So, I created several pie charts showing voting percentages for republicans and democrats, from 2008 to 2016 with plotly package in Rstudio. What I would like to do, to make it sleek, is create a menu for a user to click on(2008, 2012, 2016), and the pie chart would motion into the correct vote totals. Would I need to create a shiny app to do this?&lt;/p&gt;&#xA;" OwnerUserId="27929" LastEditorUserId="31513" LastEditDate="2017-05-19T23:16:12.480" LastActivityDate="2017-06-18T23:45:47.010" Title="Creating a pie chart with historical motion in Rstudio" Tags="|r|bigdata|visualization|rstudio|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16311" PostTypeId="2" ParentId="16298" CreationDate="2017-01-13T20:56:08.557" Score="3" Body="&lt;p&gt;For my answers, I assume you are talking about batch (not mini-batch or stochastic) gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No. Assume you initialize all weights with the same value. Then all gradients (in the same layer) will be the same. Always. Hence the network effectively only learns one parameter per layer. It is possible (and likely) that this is neither a gloabl nor a local minimum of the network (which has more parameters).&lt;/li&gt;&#xA;&lt;li&gt;Yes, as the learning rate is &quot;small enough for all practical purposes&quot;. (No, if you use SGD or mini-batch gradient descent)&lt;/li&gt;&#xA;&lt;li&gt;Usure. I think the correct answer is &quot;No, the network can make more mistakes in between with cross entropy.&quot;. It is certainly sure to improve CE loss while at the same time getting worse at accuracy (see proof below). However, I'm not sure if the gradient would ever lead to such a result.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Example for 3&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;from math import log&#xA;&#xA;def ce(vec):&#xA;    &quot;&quot;&quot;index 0 is the true class.&quot;&quot;&quot;&#xA;    return -(log(vec[0]) + sum([log(1-el) for el in vec[1:]]))&#xA;&#xA;a = [0.1001, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0999]&#xA;print(ce(a))&#xA;b = [0.49, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]&#xA;print(ce(b))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ce(a) = 3.24971912864&#xA;ce(b) = 1.42669977575&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hence the cross entropy loss dropped (as expected), the probability for the correct class increased (as expecte) but it makes a mistake (if you simply take the argmax).&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-01-14T12:20:33.160" LastActivityDate="2017-01-14T12:20:33.160" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16312" PostTypeId="2" ParentId="14346" CreationDate="2017-01-14T00:40:38.083" Score="1" Body="&lt;p&gt;Although the previous answers cover a lot to get you started in Reinforcement Learning (RL) field, I give you here an illustrative simple example to understand the concept and also what is the relationship between Supervised Learning (SL) and Unsupervised Learning (UL).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that you have a robot and you want to teach it to drive a car. Every let's say image of the road that the robot receives is going to be an input. One option that you have, in order to teach the robot, is that you can instruct it EVERY time that it receives the image of the road how much to steer the wheel. This is SL as you will have for every input state of the road a mapping to the proper angle of rotating the wheel. The main point here is that you know what is the optimal thing for your robot to do and you teach it by examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a RL setting, you just let the robot try whatever it wants and you give it a reward/punishment regarding the action(s) it takes. The magnitude of the reward/punishment might be dependent on e.g. damage to the car, staying long time on the same lane etc. the reward/punishment might be given delayed and not at every single action that the robot takes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first example (SL) the robot tries to minimize the error between your recommendation and its choices. In the second example the robot tries to maximize its reward by finding on its own what is the best to do. The SL approach at its best will lead you to a robot that &quot;mimimcs&quot; what you taught it. In the RL approach at its best, the robot will have a behavior that will be optimal in terms of driving the car and also might be better than yours. In other words it will create its own strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up,in SL you have a teacher that tells you at every single timestep exactly whats the correct response. In RL you try and find it on your own and the teacher gives you a reward/punishment. In UL you dont have any external feedback. So RL falls between SL and RL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I simplified lots of the terms just to conceptualize the learning techniques with the example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps! &lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-01-14T00:40:38.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16314" PostTypeId="2" ParentId="15878" CreationDate="2017-01-14T02:35:30.770" Score="2" Body="&lt;p&gt;I assume that you are referring to policy gradient estimates. Adding any kind of function to your policy estimation,  which is dependent on the state of the environment, first of all, does not bias your gradient estimator (&lt;a href=&quot;https://arxiv.org/abs/1301.2315&quot; rel=&quot;nofollow noreferrer&quot;&gt;Proof Here&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;The basic idea of subtracting a baseline from your action value function (and thus forming the advantage function) is that an unbiased estimator of your policy gradient is still unbiased if a constant is subtracted from that estimator. Then, that constant can be chosen appropriately in order to reduce the variance of the new estimator by optimization. If you have access, you can find a very good explanation in the book Statistical Reinforcement Learning: Modern Machine Learning Approaches in section 7.2.2. Also [&lt;a href=&quot;https://papers.nips.cc/paper/2011/hash/85d8ce590ad8981ca2c8286f79f59954-Abstract.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;] and section 3 in [&lt;a href=&quot;http://people.eecs.berkeley.edu/%7Erussell/papers/uai03-motor.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;3&lt;/a&gt;].&lt;/p&gt;&#xA;&lt;p&gt;As you mention it can be viewed as a control covariate addition [&lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume5/greensmith04a/greensmith04a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;4&lt;/a&gt;] which is used to reduce variance in Monte Carlo estimations. A good choice for that function is to use the usual value function (&lt;span class=&quot;math-container&quot;&gt;$V(s)$&lt;/span&gt;) which reduces the variance of your estimate.&lt;/p&gt;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="140783" LastEditDate="2022-09-25T12:32:58.767" LastActivityDate="2022-09-25T12:32:58.767" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16315" PostTypeId="1" CreationDate="2017-01-14T04:36:59.700" Score="-1" ViewCount="2131" Body="&lt;p&gt;Now, There is a dataset name called 'iris' which needed to load in my program. I use the following code to read. And the last column is label column.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = np.genfromtxt(ChooseDataset('iris'), delimiter=',', dtype=None, names=    ('sepal length', 'sepal width', 'petal length', 'petal width', 'label'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, it displays tuple style in the data. (And I want to ask how to change this tuple style into list style.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[(5.1, 3.5, 1.4, 0.2, b'Iris-setosa') (4.9, 3.0, 1.4, 0.2, b'Iris-setosa')&#xA;(4.7, 3.2, 1.3, 0.2, b'Iris-setosa') ...,&#xA;(6.5, 3.0, 5.2, 2.0, b'Iris-virginica')&#xA;(6.2, 3.4, 5.4, 2.3, b'Iris-virginica')&#xA;(5.9, 3.0, 5.1, 1.8, b'Iris-virginica')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I want to manipulate these attribute without class.&#xA;So I type this code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(data[0][:1])&#xA;IndexError: invalid index&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which methods do I need to change? Does someone can give some suggestions to me?&#xA;Thank you. (I know pandas is convenient, but I only can use sci-py and numpy to achive this code)&lt;/p&gt;&#xA;" OwnerUserId="27934" LastEditorUserId="27935" LastEditDate="2017-01-14T10:53:15.153" LastActivityDate="2022-01-21T16:02:35.170" Title="Use genfromtxt function and can't slice the data in python 3" Tags="|python|" AnswerCount="3" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16316" PostTypeId="2" ParentId="16315" CreationDate="2017-01-14T04:42:17.857" Score="0" Body="&lt;p&gt;It's straightforward with pandas:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame as DF&#xA;data = [(5.1, 3.5, 1.4, 0.2, b'Iris-setosa'),&#xA;(4.9, 3.0, 1.4, 0.2, b'Iris-setosa'),&#xA;(4.7, 3.2, 1.3, 0.2, b'Iris-setosa'),&#xA;(6.5, 3.0, 5.2, 2.0, b'Iris-virginica'),&#xA;(6.2, 3.4, 5.4, 2.3, b'Iris-virginica'),&#xA;(5.9, 3.0, 5.1, 1.8, b'Iris-virginica')]&#xA;DF(data, columns=('sepal length', 'sepal width', 'petal length', 'petal width', 'label')).drop('label', 1)&#xA;&#xA;=&amp;gt;    sepal length  sepal width  petal length  petal width&#xA;0           5.1          3.5           1.4          0.2&#xA;1           4.9          3.0           1.4          0.2&#xA;2           4.7          3.2           1.3          0.2&#xA;3           6.5          3.0           5.2          2.0&#xA;4           6.2          3.4           5.4          2.3&#xA;5           5.9          3.0           5.1          1.8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With just numpy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from numpy import array&#xA;array([row[:4] for row in data])&#xA;&#xA;=&amp;gt; array([[ 5.1,  3.5,  1.4,  0.2],&#xA;       [ 4.9,  3. ,  1.4,  0.2],&#xA;       [ 4.7,  3.2,  1.3,  0.2],&#xA;       [ 6.5,  3. ,  5.2,  2. ],&#xA;       [ 6.2,  3.4,  5.4,  2.3],&#xA;       [ 5.9,  3. ,  5.1,  1.8]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-01-14T05:43:04.560" LastActivityDate="2017-01-14T05:43:04.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16317" PostTypeId="1" CreationDate="2017-01-14T08:14:50.093" Score="1" ViewCount="2265" Body="&lt;p&gt;I am new to deep learning and I am trying to train a NN to recognize house numbers gathered from street view. I have already managed to recognized MNIST sequence of hand written digits by means of a CNN. In that case I have trained the CNN with a thousands of examples consisting of 1 from up to 5 digits sequences created by me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thus thought to do the same thing for SVHN, I have cropped the numbers and I now have a large dataset of single digits from street view. However the MNIST data set is very homogenous (i.e. images have same size, same angle, same color, etc.) while SVHN is obviously not. Trying to create sequences of digits from different SV photos led to weird examples that are actually not representative of the real word objects I want to model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering if it was better to train a NN on the single digits and then use it to recognize sequences. Would it be possible? How?&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-01-14T15:07:48.527" Title="Recognize Street View House Numbers" Tags="|machine-learning|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16318" PostTypeId="1" CreationDate="2017-01-14T08:59:36.870" Score="28" ViewCount="18852" Body="&lt;p&gt;I'm working on speech recognition with Tensorflow and plan to train LSTM NN with massive waves dataset. Because of the performance gains, I plan to use tfrecords. There are several examples on internet (Inception for ex.) where tfrecords files are split into shards.&#xA;My question is: what is the benefit of having tfrecords file into shards? Is there any additional performance gain of this split?&lt;/p&gt;&#xA;" OwnerUserId="27938" LastActivityDate="2020-10-21T11:32:35.493" Title="What is the benefit of splitting tfrecord file into shards?" Tags="|python|tensorflow|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16320" PostTypeId="2" ParentId="16315" CreationDate="2017-01-14T11:34:55.480" Score="1" Body="&lt;p&gt;In numpy, you can use double indexing to slice a column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = data[:, :-1]&#xA;y = data[:, -1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;X&lt;/code&gt; are all columns except for the last (2D), &lt;code&gt;y&lt;/code&gt; is only the last column (1D).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is the equivalent to matlab's:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = data(:, 1:end-1)&#xA;y = data(:, end-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or in R, this would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = data[, 1:ncol(data)-1]&#xA;y = data[, ncol(data)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure I understood your question because this is very trivial stuff. You really need to know these language features if you want to do anything interesting.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If you want to convert your labels to integers, just use list comprehension:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labels = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica']&#xA;y = [labels.index(i) for i in y]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; @Emre said I might have misunderstood the question. Indeed, I thought the problem was slicing an already imported CSV, not importing the CSV file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;numpy.genfromtxt&lt;/code&gt; will replace your strings by &lt;code&gt;nans&lt;/code&gt;. If all you want is to remove that columns, then that is good enough. But if you want to keep them, you will have to convert them to integers or some such. I suggest using &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;numpy.loadtxt&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# get the iris dataset from UCI (change to urllib.urlretrieve if using python2)&#xA;import urllib&#xA;urllib.request.urlretrieve('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', 'iris.data')&#xA;&#xA;labels = [b'Iris-setosa', b'Iris-versicolor', b'Iris-virginica']&#xA;conv = lambda x: labels.index(x)&#xA;np.loadtxt('iris.data', delimiter=',', converters={4: conv})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to import every column except the last, just use &lt;code&gt;np.loadtxt('iris.data', delimiter=',', usecols=range(4))&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2017-01-14T21:27:31.807" LastActivityDate="2017-01-14T21:27:31.807" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16321" PostTypeId="2" ParentId="16317" CreationDate="2017-01-14T15:07:48.527" Score="1" Body="&lt;p&gt;Have you seen this &lt;a href=&quot;http://matthewearl.github.io/2016/05/06/cnn-anpr/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; post? It is doing exactly what you are trying to do.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Convolutional neural network&lt;/li&gt;&#xA;&lt;li&gt;Use a fixed sliding window to crop plate number&lt;/li&gt;&#xA;&lt;li&gt;Train on a fixed-length number plate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you want to train on the digit-level, you should define a fixed rectangle size, large enough to fit a single digit in your data set. If your convolutional neural network is good, it should be able to detect your digit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't you should take a look at the Google paper: &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-01-14T15:07:48.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16322" PostTypeId="1" CreationDate="2017-01-14T16:07:25.703" Score="6" ViewCount="3214" Body="&lt;p&gt;I created a multi-scale CNN in python keras. The network architecture is similar to the diagram. Here, same image is fed to 3 CNN's with different architectures. The weights are NOT shared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2H4xD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2H4xD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I coded the following multiscale CNN in keras which loosely resembles the architecture in the diagram. But I keep getting &lt;strong&gt;&quot;Out of memory ERROR&quot;&lt;/strong&gt; even when the train_dir has 2 images. Would appreciate help... &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#main CNN model - CNN1&#xA;main_model = Sequential()&#xA;main_model.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))&#xA;main_model.add(Activation('relu'))&#xA;main_model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&#xA;main_model.add(Convolution2D(32, 3, 3))&#xA;main_model.add(Activation('relu'))&#xA;main_model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&#xA;main_model.add(Convolution2D(64, 3, 3))&#xA;main_model.add(Activation('relu'))&#xA;main_model.add(MaxPooling2D(pool_size=(2, 2))) # the main_model so far outputs 3D feature maps (height, width, features)&#xA;&#xA;main_model.add(Flatten())&#xA;&#xA;#lower features model - CNN2&#xA;lower_model1 = Sequential()&#xA;lower_model1.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))&#xA;lower_model1.add(Activation('relu'))&#xA;lower_model1.add(MaxPooling2D(pool_size=(2, 2)))&#xA;lower_model1.add(Flatten())&#xA;&#xA;#lower features model - CNN3&#xA;lower_model2 = Sequential()&#xA;lower_model2.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))&#xA;lower_model2.add(Activation('relu'))&#xA;lower_model2.add(MaxPooling2D(pool_size=(2, 2)))&#xA;lower_model2.add(Flatten())&#xA;&#xA;#merged model&#xA;merged_model = Merge([main_model, lower_model1, lower_model2], mode='concat')&#xA;&#xA;final_model = Sequential()                     &#xA;final_model.add(merged_model)                  &#xA;final_model.add(Dense(64))&#xA;final_model.add(Activation('relu'))&#xA;final_model.add(Dropout(0.5))      &#xA;final_model.add(Dense(1))&#xA;final_model.add(Activation('sigmoid'))&#xA;final_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])&#xA;&#xA;print 'About to start training merged CNN'&#xA;train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)&#xA;train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(224, 224), batch_size=32, class_mode='binary')&#xA;&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;test_generator = test_datagen.flow_from_directory(args.test_images, target_size=(224, 224), batch_size=32, class_mode='binary')&#xA;&#xA;final_train_generator = zip(train_generator, train_generator, train_generator)&#xA;final_test_generator  = zip(test_generator, test_generator, test_generator)&#xA;final_model.fit_generator(final_train_generator, samples_per_epoch=nb_train_samples, nb_epoch=nb_epoch, validation_data=final_test_generator, nb_val_samples=nb_validation_samples)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 1:&lt;/strong&gt; Providing more system info &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Theano-0.9.0.dev5 | Keras-1.2.1 | Python 2.7.12 | OSX Sierra 10.12.3 (16D32) | Macbook Pro 16GB RAM | CPU mode &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;~/.keras/keras.json&lt;/code&gt; contents&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;    &quot;image_dim_ordering&quot;: &quot;th&quot;,&#xA;    &quot;epsilon&quot;: 1e-07,&#xA;    &quot;floatx&quot;: &quot;float64&quot;,&#xA;    &quot;backend&quot;: &quot;theano&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Dont have &lt;code&gt;.theanorc&lt;/code&gt; file&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note individual CNN models are training fine. Only the merged code above causes issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 2:&lt;/strong&gt; on 27th January, 2017. Tried the following - &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Reduced the no. of parameters of the CNN from 53 million to 100K. But still no use. The network eventually fails to train due to memory issues. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reduced the batch size of images training to 8. The network training fails with the same reason. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;No working solution at the time of writing this update...&lt;/p&gt;&#xA;" OwnerUserId="77" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:24.053" LastActivityDate="2021-02-16T15:24:24.053" Title="Multi scale CNN Network Python" Tags="|python|deep-learning|neural-network|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="16323" PostTypeId="1" CreationDate="2017-01-15T02:04:12.993" Score="4" ViewCount="373" Body="&lt;p&gt;my research question is the examine the effect of &quot;receiving attention&quot; from other members in an online community on &quot;sustained participation&quot; on the website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to measure &quot;sustained participation&quot; of each user by  calculating average time difference between the submissions of the user. I calculated it in the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/icZvv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/icZvv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I measured &quot;attention&quot; by calculating total number of the comments each user received for all the submissions he/she has posted.I also want to consider total number of votes and total number of views as well. I am not sure if it is a good idea to add those as the independent variables into model too or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is about Dependent variable:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;since some people just participated two times in two successive days and the average between submission days is 1. and some people participated 100 times and their average between submission days of them is also 1. but it is obvious that the second group that have participated 100 times had sustained participation not the first group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I need to consider the number of the submissions in the model too! I do not know if there is a way to consider it? how I can handle this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;should I group the users and do data analysis separately for them. for example users who have participated less than 10 times in one group! users with 10-20 participation in another group and etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate if anyone can help me! my paper's due is so close and I need some preliminary results.&lt;/p&gt;&#xA;" OwnerUserId="27954" LastEditorUserId="843" LastEditDate="2018-02-11T07:26:48.000" LastActivityDate="2024-01-03T14:08:16.017" Title="What kind of regression model should I do?" Tags="|regression|research|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16324" PostTypeId="1" AcceptedAnswerId="16329" CreationDate="2017-01-15T08:02:40.347" Score="3" ViewCount="997" Body="&lt;p&gt;I have used the &lt;strong&gt;Iris Dataset's&lt;/strong&gt; &lt;em&gt;1st&lt;/em&gt; and &lt;em&gt;3rd&lt;/em&gt; Column for the features. and the labels of &lt;em&gt;Iris Setosa (-1)&lt;/em&gt; and &lt;em&gt;Iris Versicolor (1)&lt;/em&gt;. I am using &lt;strong&gt;ADALINE&lt;/strong&gt; as a simple classification model for my dataset. I am using gradient descent as the cost minimizing function. But on every iteration the error increases. What am I doing wrong in the python code? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;class AdalineGD(object):&#xA;&#xA;    def __init__(self, eta = 0.01, n_iter = 50):&#xA;        self.eta = eta&#xA;        self.n_iter = n_iter&#xA;&#xA;    def fit (self, X, y):&#xA;        &quot;&quot;&quot;Fit training data.&quot;&quot;&quot;&#xA;&#xA;        self.w_ = np.random.random(X.shape[1])&#xA;        self.cost_ = []&#xA;        print ('Initial weights are: %r' %self.w_)&#xA;        for i in range(self.n_iter):&#xA;            output = self.net_input(X)&#xA;            print (&quot;On iteration %d, output is: %r&quot; %(i, output))&#xA;            errors = output - y&#xA;            print(&quot;On iteration %d, Error is: %r&quot; %(i, errors))&#xA;            self.w_ += self.eta * X.T.dot(errors)&#xA;            print ('Weights on iteration %d: %r' %(i, self.w_))&#xA;            cost = (errors**2).sum() / 2.0&#xA;            self.cost_.append(cost)&#xA;            print (&quot;On iteration %d, Cost is: %r&quot; %(i, cost))&#xA;            prediction = self.predict(X)&#xA;            print (&quot;Prediction after iteration %d is: %r&quot; %(i, prediction))&#xA;            input()&#xA;        return self&#xA;&#xA;    def net_input(self, X):&#xA;        &quot;&quot;&quot;Calculate net input&quot;&quot;&quot;&#xA;        return X.dot(self.w_)&#xA;&#xA;    def activation(self, X):&#xA;        &quot;&quot;&quot;Computer Linear Activation&quot;&quot;&quot;&#xA;        return self.net_input(X)&#xA;&#xA;    def predict(self, X):&#xA;        &quot;&quot;&quot;Return class label after unit step&quot;&quot;&quot;&#xA;        return np.where(self.activation(X) &amp;gt;= 0.0, 1, -1)&#xA;&#xA;####### END OF THE CLASS ########&#xA;#importing the Iris Dataset &#xA;df = pd.read_csv(&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data&quot;, header = None)&#xA;y = df.iloc[0:100, 4].values&#xA;y = np.where(y == 'Iris-setosa', -1, 1)&#xA;X = df.iloc[0:100, [0, 2]].values&#xA;#Adding the ones column to the X matrix&#xA;X = np.insert(X, 0,  np.ones(X.shape[0]), axis = 1)&#xA;ada = AdalineGD(n_iter = 20, eta = 0.001).fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27958" LastActivityDate="2019-02-19T16:49:38.620" Title="My ADALINE model using Gradient Descent is increasing error on each iteration" Tags="|machine-learning|python|classification|gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16325" PostTypeId="1" AcceptedAnswerId="16381" CreationDate="2017-01-15T10:04:31.793" Score="0" ViewCount="596" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Pa1qM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Pa1qM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;This is the distribution of my data.&#xA;I want to use SVM with only one 'circle' to cluster most of the 0.&#xA;I tried to run it with the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = svm.SVC(max_iter=1, kernel='rbf')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it gives a strange result&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HLvE0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HLvE0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;How should I do it correctly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X&#xA;array([[  5.46787217e+00,   2.09073426e-02],&#xA;       [ -2.57653443e+00,   9.77145456e+00],&#xA;       [  1.09476747e+02,  -1.71182599e+00],&#xA;       [  4.94810319e+01,  -2.77826146e+00],&#xA;       [ -1.15407498e+01,   1.94848276e+00],&#xA;       [  7.47153419e+00,  -7.67879236e-01],&#xA;       [ -3.45243619e+01,  -1.75370697e+00],&#xA;       [  2.46902913e+00,  -1.87289298e-01],&#xA;       [  3.04749853e+01,  -1.46262345e+00],&#xA;       [  1.24751661e+01,  -1.54323119e+00],&#xA;       [ -1.85219673e+01,  -2.28503662e+00],&#xA;       [ -6.53200731e+00,  -3.28311541e-02],&#xA;       [  3.44650256e+01,   7.53756355e-01],&#xA;       [  2.44812829e+01,  -2.89021667e+00],&#xA;       [  5.24685269e+01,   3.66211052e-02],&#xA;       [  3.94686575e+01,  -2.15955996e-02],&#xA;       [ -2.65369170e+01,   1.08356664e+00],&#xA;       [ -3.15280741e+01,  -9.42529350e-01],&#xA;       [ -2.35494219e+01,   3.89844921e+00],&#xA;       [ -2.85181847e+01,  -3.12756169e+00],&#xA;       [ -3.95243116e+01,  -1.77609801e+00],&#xA;       [ -8.52462300e+00,  -1.63727356e+00],&#xA;       [ -1.15245929e+01,  -1.65070818e+00],&#xA;       [ -4.25492311e+01,   3.81336326e+00],&#xA;       [ -2.25182450e+01,  -3.10069245e+00],&#xA;       [ -2.05293114e+01,  -6.98507045e-01],&#xA;       [  1.14627013e+01,   1.25373855e+00],&#xA;       [ -4.65242413e+01,  -1.80744547e+00],&#xA;       [  4.14811122e+01,  -2.81408713e+00],&#xA;       [ -5.15256196e+01,  -1.42161754e+00],&#xA;       [ -2.45182249e+01,  -3.10964886e+00],&#xA;       [  4.48148382e+00,  -2.97978083e+00],&#xA;       [ -2.75218769e+01,  -2.32534049e+00],&#xA;       [  4.94623197e+01,   1.42391045e+00],&#xA;       [ -2.95181747e+01,  -3.13203990e+00],&#xA;       [  2.04251866e+01,   9.69838626e+00],&#xA;       [  5.45907950e+00,   2.02461229e+00],&#xA;       [ -1.75219773e+01,  -2.28055841e+00],&#xA;       [ -4.05430138e+01,   2.42159570e+00],&#xA;       [ -4.52466317e+00,  -1.61936073e+00],&#xA;       [  3.64525308e+01,   3.56416072e+00],&#xA;       [ -1.45367359e+01,   9.23848192e-01],&#xA;       [ -4.95216559e+01,  -2.42386107e+00],&#xA;       [ -5.45377626e+01,   1.15293883e+00],&#xA;       [ -2.55743514e+01,   9.49238869e+00],&#xA;       [  9.24618878e+01,   1.61647340e+00],&#xA;       [  8.54806703e+01,  -2.61704597e+00]])&#xA;Y&#xA;[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, &#xA; 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA; 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, &#xA; 0, 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20907" LastEditorUserId="20907" LastEditDate="2017-01-16T02:29:30.947" LastActivityDate="2017-01-18T04:15:39.123" Title="Python SVM rgb cluster" Tags="|scikit-learn|svm|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16326" PostTypeId="1" AcceptedAnswerId="16327" CreationDate="2017-01-15T13:34:00.177" Score="2" ViewCount="512" Body="&lt;p&gt;I am a Python beginner, just getting into machine learning and need advice on the approach i should use for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of my data-set.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/STKE0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/STKE0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the RESULT is a corresponding INDEX in each VALUES array and every row is a separate array, i need to find the probability of each index of an array being the RESULT based on its configuration, where the overall configuration and distribution is most important rather than any individual value.&lt;/p&gt;&#xA;" OwnerUserId="27962" LastEditorUserId="27962" LastEditDate="2017-01-20T05:39:10.783" LastActivityDate="2017-01-20T05:39:10.783" Title="Classification problem approach with Python" Tags="|machine-learning|python|predictive-modeling|probability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16327" PostTypeId="2" ParentId="16326" CreationDate="2017-01-15T14:22:07.850" Score="4" Body="&lt;p&gt;You are looking at a &lt;strong&gt;Classification problem&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logistic regression&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision trees&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM&lt;/a&gt;. Any of the above can solve the job for you. But selecting the best model depends upon how good it is able to predict the test set. Use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross-validations&lt;/a&gt; and see which model can give you better accuracy. So split you test and train set accordingly. You don't expect the model to predict, if you haven't trained it. See &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;stratified sampling&lt;/a&gt; for starters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though your predictors might be &lt;strong&gt;categorical&lt;/strong&gt; or &lt;strong&gt;ordinals&lt;/strong&gt;, they can be treated as &lt;strong&gt;numericals&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have built in functions like &lt;em&gt;predict_proba&lt;/em&gt; to find class probability. The references for this can be found in above links. But go through how they work and what is the policy for selecting best plane in any classification, so that you don't feel like you are using a black-box. Since you are saying that you are a beginner, &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas&lt;/a&gt; will be a very useful module for reading data into dataframes and mending it as you like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this clears something.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-15T14:22:07.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16329" PostTypeId="2" ParentId="16324" CreationDate="2017-01-15T15:07:28.363" Score="1" Body="&lt;p&gt;I think something is wrong here.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.w_ += self.eta * X.T.dot(errors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You are going to the positive to the gradient while you should be doing is going to the negative direction of it&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.w_ -= self.eta * X.T.dot(errors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.w_ += -self.eta * X.T.dot(errors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;see &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_descent#Python&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for more clarification.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-15T15:07:28.363" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16332" PostTypeId="2" ParentId="16323" CreationDate="2017-01-15T18:56:06.450" Score="0" Body="&lt;p&gt;One thing you can do about your 'participation' variable is including the beginning and end of your data window. Let's say your data ranges from $start=$ 1/1/2016 to $end=$ 1/1/2017. Instead of only calculating the difference between the second and the first post, you'd calculate the difference between the first post and 1/1/2016 and then the difference between the second and the first post. (If some people join the platform after 1/1/2016 then you'd take the minimum of the join date and 1/1/2016). And you'd also calculate the difference between 1/1/2017 and last post. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if someone only had two posts $p_1$ and $p_2$, you would get the difference $(p_1-start, p_2-p_1,end-p_2)$. The differences $p_1-start$ and  $end-p_2$ would then be larger than the differences $p_1-start$ and  $end-p_n$ for some with $n&amp;gt;&amp;gt;2$ posts.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-01-15T18:56:06.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16333" PostTypeId="1" CreationDate="2017-01-15T19:40:06.347" Score="1" ViewCount="393" Body="&lt;p&gt;I am trying to learn a bit of deep learning playing with the Street View House Numbers data set. I have managed to recognize sequences of digits and I'd like now to train a CNN to localize digits and provide boxes coordinates. The problem is that I have the boxes coordinates related to the original image sizes which are always different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to resize the images in order to have homogenous input for the NN but I don't know hot to transform the boxes coordinates coherently.&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-04-16T03:32:56.950" Title="Digits Localization on Streets View House Numbersm" Tags="|machine-learning|python|deep-learning|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16335" PostTypeId="2" ParentId="6107" CreationDate="2017-01-15T21:58:56.943" Score="16" Body="&lt;p&gt;Just found a great article from the theaon website on this topic [1]:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The need for transposed convolutions generally arises from the desire to use a transformation going in the opposite direction of a normal convolution, [...] to project feature maps to a higher-dimensional space.&#xA;[...]&#xA;i.e., map from a 4-dimensional space to a 16-dimensional space, while keeping the connectivity pattern of the convolution.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Transposed convolutions&lt;/strong&gt;&lt;/em&gt; – also called fractionally strided convolutions – work by swapping the forward and backward passes of a convolution. One way to put it is to note that the kernel defines a convolution, but whether it’s a direct convolution or a transposed convolution is determined by how the forward and backward passes are computed.&lt;/p&gt;&#xA;&lt;p&gt;The transposed convolution operation can be thought of as the gradient of some convolution with respect to its input, which is usually how transposed convolutions are implemented in practice.&lt;/p&gt;&#xA;&lt;p&gt;Finally note that it is always possible to implement a transposed convolution with a direct convolution. The disadvantage is that it usually involves adding many columns and rows of zeros to the input, resulting in a much less efficient implementation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So in simplespeak, a &amp;quot;transposed convolution&amp;quot; is mathematical operation using matrices (just like convolution) but is more efficient than the normal convolution operation in the case when you want to go back from the convolved values to the original (opposite direction). This is why it is preferred in implementations to convolution when computing the opposite direction (i.e. to avoid many unnecessary 0 multiplications caused by the sparse matrix that results from padding the input).&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Image ---&amp;gt; convolution ---&amp;gt; Result&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Result ---&amp;gt; transposed convolution ---&amp;gt; &amp;quot;originalish Image&amp;quot;&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;Sometimes you save some values along the convolution path and reuse that information when &amp;quot;going back&amp;quot;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Result ---&amp;gt; transposed convolution ---&amp;gt; Image&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;That's probably the reason why it's wrongly called a &amp;quot;deconvolution&amp;quot;. However, it does have something to do with the matrix transpose of the convolution (C^T), hence the more appropriate name &amp;quot;transposed convolution&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;So it makes a lot of sense when considering computing cost. You'd pay a lot more for amazon gpus if you wouldn't use the transposed convolution.&lt;/p&gt;&#xA;&lt;p&gt;Read and watch the animations here carefully:&#xA;&lt;a href=&quot;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html#no-zero-padding-unit-strides-transposed&quot; rel=&quot;noreferrer&quot;&gt;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html#no-zero-padding-unit-strides-transposed&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Some other relevant reading:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The transpose (or more generally, the Hermitian or conjugate transpose) of a filter is simply the matched filter[3]. This is found by time reversing the kernel and taking the conjugate of all the values[2].&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I am also new to this and would be grateful for any feedback or corrections.&lt;/p&gt;&#xA;&lt;p&gt;[1] &lt;a href=&quot;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html&quot; rel=&quot;noreferrer&quot;&gt;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[2] &lt;a href=&quot;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html#transposed-convolution-arithmetic&quot; rel=&quot;noreferrer&quot;&gt;http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html#transposed-convolution-arithmetic&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[3] &lt;a href=&quot;https://en.wikipedia.org/wiki/Matched_filter&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Matched_filter&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27973" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-01-25T17:05:34.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16336" PostTypeId="2" ParentId="16325" CreationDate="2017-01-15T22:11:47.453" Score="0" Body="&lt;p&gt;The result is expected because the data is not linearly separable. Try using kernel SVM's instead of SVM.&lt;/p&gt;&#xA;" OwnerUserId="26050" LastActivityDate="2017-01-15T22:11:47.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16337" PostTypeId="1" CreationDate="2017-01-15T22:47:24.237" Score="2" ViewCount="686" Body="&lt;p&gt;I'm working on a project that seeks to identify clusters in urban development based on location (in lat/lon) and a categorical variable (what the particular site is zoned for). Ideally, the analysis would identify clusters of sites that are 1) near each other and 2) zoned the same. Below is a sample of what my data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; lat       lon      zone&#xA; 33.22320 -112.6741 R-43      &#xA; 33.45324 -113.0888 R-43      &#xA; 33.71800 -112.3885 R-43      &#xA; 33.45626 -111.9408 AG        &#xA; 33.45746 -111.9313 R-6       &#xA; 33.45747 -111.9309 R-6 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've seen methods that define distance on just lat/lons using great circle distances, but I haven't seen any mixed clustering of the sort I'm trying to implement. I'm fairly new to cluster analysis so any guidance on implementing something like this in R or Python would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="27974" LastEditorUserId="201" LastEditDate="2019-04-07T12:43:23.483" LastActivityDate="2019-04-07T12:43:23.483" Title="Mixed geospatial and categorical clustering" Tags="|python|r|clustering|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16338" PostTypeId="2" ParentId="16333" CreationDate="2017-01-16T02:26:39.353" Score="2" Body="&lt;p&gt;Since you are saying that you got box coordinates in the original image, why not reduce coordinates by exactly same scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example a coordinate say $(100,100)$ in the image of say $(1024,1024)$ size when resized to say $(256,256)$ will be $(25,25)$ (assuming you didn't crop anything in the mean while), which is in general &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(x_{new},y_{new}) = (\frac{x_{old}*l_{new}}{l_{old}},\frac{y_{old}*b_{new}}{b_{old}})$$ where $l$ is the length of the image, $b$ is the width.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In times of fractional answers after reduction make sure your bounding box covers a bigger area. For example, Y-coordinate for the left side of bounding box should be pushed towards the Y-axis while on the right should be pushed towards the ceil value like say you got $25.5$ by reducing, go to 25 when on the left side while $26$ when on right side of bounding box. Similarly up and down for $x's$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-16T02:26:39.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16339" PostTypeId="1" AcceptedAnswerId="16344" CreationDate="2017-01-16T09:12:17.357" Score="2" ViewCount="155" Body="&lt;p&gt;I have sales data in weekly buckets like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;weekID product SoldQty&#xA;1      1       10&#xA;2      1       20&#xA;3      1       30&#xA;4      1       40&#xA;5      1       50&#xA;6      1       60&#xA;7      1       70&#xA;&#xA;1      2       10&#xA;2      2       20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Calculating the standard deviation of weekly sales per product is pretty straightforward.&lt;/p&gt;&#xA;&lt;p&gt;Now, I am asking the question: how do you calculate the standard deviation for the same data, but bi-weekly bucketed? x-weekly bucketed?&lt;/p&gt;&#xA;&lt;p&gt;Question 2: is there an efficient algorithm for calculating it on weekly data instead of materializing the x-weekly combinations?&lt;/p&gt;&#xA;&lt;p&gt;From the business side it means that I have various forecast horizons (1wk, 4 wks, 6wks...) per product. And I would like to build the confidence intervals for predictions of SoldQty within the forecast horizon.&lt;/p&gt;&#xA;&lt;p&gt;It all seems very similar to the Safety Stock calculations from logistics, but I would like to be sure.&lt;/p&gt;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&lt;p&gt;Please consider there are multiple ways to re-bucket (re-combine) this data. I think that if the new bucket contains W weeks, there are W ways to re-bucket data. For an example of 2-weekly buckets.&lt;/p&gt;&#xA;&lt;p&gt;Way 1:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;bucket wkid prod sold&#xA;1      1    1    10&#xA;1      2    1    20&#xA;2      3    1    30&#xA;2      4    1    40&#xA;3      5    1    50&#xA;3      6    1    60&#xA;4      7    1    70&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Way 2:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;bucket wkid prod sold&#xA;1      1    1    10&#xA;2      2    1    20&#xA;2      3    1    30&#xA;3      4    1    40&#xA;3      5    1    50&#xA;4      6    1    60&#xA;4      7    1    70&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Both options make sense business-wise. Essentially you need to calculate standard deviations between sales during fortnights. And, standard deviation in way 2 should be different from way 1.&lt;/p&gt;&#xA;&lt;p&gt;Thanks in advance for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="27979" LastEditorUserId="29169" LastEditDate="2021-06-02T18:19:43.043" LastActivityDate="2021-06-02T18:19:43.043" Title="Re-bucket weekly sales data and calculate descriptive statistics" Tags="|time-series|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16340" PostTypeId="2" ParentId="14999" CreationDate="2017-01-16T11:20:21.163" Score="5" Body="&lt;p&gt;The only Python package I've found is on &lt;a href=&quot;https://github.com/bartdag/pymining&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;They have an implementation of &lt;code&gt;BIDE&lt;/code&gt; there, but it's not maintained code.&lt;/p&gt;&#xA;" OwnerUserId="27984" LastEditorUserId="29575" LastEditDate="2020-08-04T13:23:04.527" LastActivityDate="2020-08-04T13:23:04.527" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16341" PostTypeId="2" ParentId="12645" CreationDate="2017-01-16T11:26:42.087" Score="-1" Body="&lt;p&gt;If you want count of missing values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.logical_not(df.isnull()).sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24818" LastActivityDate="2017-01-16T11:26:42.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16342" PostTypeId="1" AcceptedAnswerId="18823" CreationDate="2017-01-16T12:53:12.653" Score="41" ViewCount="52745" Body="&lt;p&gt;I have 3 classes with this distribution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Class 0: 0.1169&#xA;Class 1: 0.7668&#xA;Class 2: 0.1163&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I am using &lt;code&gt;xgboost&lt;/code&gt; for classification. I know that there is a parameter called &lt;code&gt;scale_pos_weight&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how is it handled for 'multiclass' case, and how can I properly set it?&lt;/p&gt;&#xA;" OwnerUserId="14434" LastEditorUserId="134" LastEditDate="2018-12-30T14:29:15.347" LastActivityDate="2022-08-05T18:23:02.977" Title="Unbalanced multiclass data with XGBoost" Tags="|classification|xgboost|multiclass-classification|class-imbalance|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16343" PostTypeId="2" ParentId="16323" CreationDate="2017-01-16T12:59:23.143" Score="0" Body="&lt;p&gt;I would segment the users on the site by their overall activity measure until the point of test, and train a model using these segments as a categorical variable(or train a different model for each).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts are that in the case of two users:&lt;br&gt;&#xA;a) A very active user who was on a long vacation.&lt;br&gt;&#xA;b) A new user -  who had one action(only on sign up day)&lt;br&gt;&#xA;Might have the same sustained-participation metric - if measured as a function of time passed since last action.&lt;br&gt;&#xA;But we expect the community to react differently to their actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A model might look like:&lt;br&gt;&#xA;attention = M(segment_type, time_since_last_activity).&lt;br&gt;&#xA;segment_type = G(activity_signals_until_now)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where activity_signal_until_now may consist:&lt;br&gt;&#xA;- total action&lt;br&gt;&#xA;- time since first action&lt;br&gt;&#xA;- average time between actions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;M can be a simple Regressor.&lt;br&gt;&#xA;G could be supervised(if you have prior on what segments you have) or unsupervised using some kind of clustering algorithm.   &lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-01-16T12:59:23.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16344" PostTypeId="2" ParentId="16339" CreationDate="2017-01-16T14:03:04.910" Score="1" Body="&lt;p&gt;If the week ID is given as you state, calculate a bucket variable $w_x=int(weekID/x)$.  Then use a SQL statement to summarize the volume to levels of $w_x$.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-01-16T14:03:04.910" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16346" PostTypeId="1" AcceptedAnswerId="16505" CreationDate="2017-01-16T15:48:04.103" Score="3" ViewCount="1211" Body="&lt;p&gt;I want to build a system that recognises (with a given uncertainty) the make and model of a car from an image. I have decided to use Convolution Neural Networks, specifically the &lt;a href=&quot;http://caffe.berkeleyvision.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe&lt;/a&gt; framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My next decision is how best to build my data set. According to &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; book, I need around 5000 data points for each class (so lets say ~500k images).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done a little bit of reading on here and other resources, and it seems that the &lt;a href=&quot;https://developers.google.com/custom-search/json-api/v1/overview&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Custom Search API&lt;/a&gt; is a potential option; but that limits me to at most 100 searches per day (for free). I thought about building a script to scrape sites like &lt;a href=&quot;http://www.autotrader.co.uk/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autotrader&lt;/a&gt;; but my experience with scraping the web is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have experience generating image data sets of this size? Any pearls of wisdom that you could share with me? I am happy to invest time and effort learning for example &lt;a href=&quot;https://www.crummy.com/software/BeautifulSoup/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Beautiful Soup&lt;/a&gt; or this Google API, but I don't want to waste time going down the wrong rabbit hole.&lt;/p&gt;&#xA;" OwnerUserId="27991" LastActivityDate="2017-01-31T18:06:32.573" Title="Generating Image data sets for training CNNs" Tags="|neural-network|dataset|caffe|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16347" PostTypeId="1" AcceptedAnswerId="16383" CreationDate="2017-01-16T16:16:12.190" Score="8" ViewCount="16951" Body="&lt;p&gt;Which machine learning or deep learning model(&lt;em&gt;has to be supervised learning&lt;/em&gt;) will be best suited for recognizing patterns in financial markets ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I mean by pattern recognition in financial market :&#xA;Following Image shows how a sample pattern (i.e. Head and shoulder) looks like:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Image 1:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EfGSA.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EfGSA.gif&quot; alt=&quot;Prototype of Head and Shoulder Pattern&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And Following Image shows how it actually forms in real chart events: &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Image 2:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XDa6K.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XDa6K.png&quot; alt=&quot;Head and Shoulder in real chart events&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm trying to do is:&#xA;Any pattern similar to Image 1 can be defined as &lt;strong&gt;&lt;em&gt;Head and Shoulder Pattern&lt;/em&gt;&lt;/strong&gt; but in a Chart (Price Chart) it will not form as clearly as Image 1. Image 2 is the sample of &lt;strong&gt;&lt;em&gt;Head and Shoulder Pattern&lt;/em&gt;&lt;/strong&gt; form in Chart (Price Chart).&#xA;As it seems in Image 2, it can not be identified as &lt;strong&gt;&lt;em&gt;Head and Shoulder Pattern&lt;/em&gt;&lt;/strong&gt; by normal algorithms or analysis (Because there are a lot of highers and low forming a lot of structure, which can easily mislead into a lot of shoulders or head or any other structures).&#xA;I'm expecting to train machine to recognize the &lt;strong&gt;&lt;em&gt;Head and Shoulder Pattern&lt;/em&gt;&lt;/strong&gt; when similar (as Image 2) pattern is formed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Let me know if I'm taking it to wrong way. &#xA;I only have beginners knowledge on Machine Learning.&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="27992" LastEditorUserId="27992" LastEditDate="2017-01-17T07:26:45.823" LastActivityDate="2019-01-08T06:53:17.503" Title="Pattern Recognition on Financial Market" Tags="|machine-learning|deep-learning|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16348" PostTypeId="1" AcceptedAnswerId="16356" CreationDate="2017-01-16T17:04:05.850" Score="5" ViewCount="3801" Body="&lt;p&gt;I'm training an LSTM for sentiment analysis on a review dataset downloaded from &lt;a href=&quot;http://www.cs.jhu.edu/~mdredze/datasets/sentiment/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After creating a dictionary, I'm running a script in Python to replace strings (words) with numbers that keras/theano will embed later. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that such a large dataset requires a lot of time for lookup. I would appreciate if anyone had suggestion on a tool for faster lookup or similar. Currently I just loop through every word in the corpus and replace it with the corresponding number from the dictionary (1-hot encoding essentially)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm doing roughly the following: each Python list is a sentence (before tokenization here):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;['noble', 'interesting_superlatives',...,'the_idea']&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which I want to conver to a list of integers, like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[143599, 12387,...,7582]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I referred to it (probably incorrectly) as one-hot encoding because for each word there is exactly one number in the dictionary. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="1426" LastEditDate="2017-01-16T17:35:15.527" LastActivityDate="2017-01-16T23:10:48.247" Title="Words to numbers faster lookup" Tags="|python|language-model|sentiment-analysis|encoding|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16349" PostTypeId="1" CreationDate="2017-01-16T17:36:42.360" Score="1" ViewCount="518" Body="&lt;p&gt;I am doing &lt;a href=&quot;https://i.stack.imgur.com/Nlsw4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Forest-Cover-Type-prediction&lt;/a&gt; on Kaggle &lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is the train and test data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dim(train)&#xA;[1] 15120    56&#xA;&amp;gt; dim(test)&#xA;[1] 565892     56&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far I have done this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;combined this data &lt;code&gt;comb &amp;lt;- rbind(train,test)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then mapped 40 mutually exclusive Soil_Type columns into one and 4 mutually exclusive Wilderness_Area column into one.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;comb$Soil &amp;lt;- apply(comb[grep(&quot;Soil_Ty+&quot;,colnames(comb))], 1 , function(x){which(x == 1)})&#xA;comb$Wilderness &amp;lt;- apply(comb[grep(&quot;Wilderness_Are+&quot;,colnames(comb))], 1 , function(x){which(x == 1)})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;after deleting these 44 columns I got it down to 14 columns.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dim(comb)&#xA;[1] 581012     14&#xA;&amp;gt; colnames(comb)&#xA; [1] &quot;Id&quot;                                 &quot;Elevation&quot;                          &quot;Aspect&quot;                            &#xA; [4] &quot;Slope&quot;                              &quot;Horizontal_Distance_To_Hydrology&quot;   &quot;Vertical_Distance_To_Hydrology&quot;    &#xA; [7] &quot;Horizontal_Distance_To_Roadways&quot;    &quot;Hillshade_9am&quot;                      &quot;Hillshade_Noon&quot;                    &#xA;[10] &quot;Hillshade_3pm&quot;                      &quot;Horizontal_Distance_To_Fire_Points&quot; &quot;Cover_Type&quot;                        &#xA;[13] &quot;Soil&quot;                               &quot;Wilderness&quot;       &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I separated the data    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train &amp;lt;- comb[1:15120,]&#xA;test &amp;lt;- comb[15121:581012,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and ran randomForest on this data &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(415)&#xA;fit &amp;lt;- randomForest(Cover_Type ~ .,data=train[-1], importance=TRUE, ntree=2000, na.action = na.omit)&#xA;varImpPlot(fit)&#xA;predi &amp;lt;- predict(fit,test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and got an accuracy of 0.70372 on Kaggle. Now I have hit the wall. I tried plotting various variables against the Cover_Type(variable to be predicted) but couldn't do figure out what to do with those.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Nlsw4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Nlsw4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I improve accuracy from here? what is the general approach in such cases?&lt;/p&gt;&#xA;" OwnerUserId="27989" LastActivityDate="2017-01-31T21:11:05.567" Title="How to improve accuracy further for forest cover prediction" Tags="|machine-learning|r|predictive-modeling|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16350" PostTypeId="1" CreationDate="2017-01-16T17:53:04.680" Score="15" ViewCount="26471" Body="&lt;p&gt;Are there any rules of thumb (or actual rules) pertaining to the minimum, maximum and &quot;reasonable&quot; amount of LSTM cells I should use? Specifically I am relating to &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/rnn_cell/rnn_cells_for_use_with_tensorflow_s_core_rnn_methods#BasicLSTMCell.__init__&quot; rel=&quot;noreferrer&quot;&gt;BasicLSTMCell&lt;/a&gt; from TensorFlow and &lt;code&gt;num_units&lt;/code&gt; property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please assume that I have a classification problem defined by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t - number of time steps&#xA;n - length of input vector in each time step&#xA;m - length of output vector (number of classes)&#xA;i - number of training examples&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it true, for example, that the number of training examples should be larger than:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;4*((n+1)*m + m*m)*c&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;c&lt;/code&gt; is number of cells? I based this on this: &lt;a href=&quot;https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network&quot;&gt;How to calculate the number of parameters of an LSTM network?&lt;/a&gt; As I understand, this should give the total number of parameters, which should be less than number of training examples.&lt;/p&gt;&#xA;" OwnerDisplayName="user27994" LastEditorUserId="-1" LastEditorDisplayName="user27994" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-12-11T15:47:32.693" Title="How many LSTM cells should I use?" Tags="|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16352" PostTypeId="2" ParentId="16348" CreationDate="2017-01-16T21:40:15.567" Score="2" Body="&lt;p&gt;You're doing something wrong. I can query a 100K word dict in nanoseconds&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word_list = open('/usr/share/dict/words').read().split()&#xA;len(word_list)&#xA;&#xA;&amp;gt; 99171&#xA;&#xA;word_dict = {word: hash(word) for word in word_list}&#xA;%timeit word_dict['blazing']&#xA;&#xA;&amp;gt; 10000000 loops, best of 3: 33.8 ns per loop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-01-16T21:40:15.567" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16353" PostTypeId="2" ParentId="16348" CreationDate="2017-01-16T21:51:00.147" Score="1" Body="&lt;p&gt;You could use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Trie&quot; rel=&quot;nofollow noreferrer&quot;&gt;trie&lt;/a&gt; from the Wikipedia definition: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;is a kind of search tree—an ordered tree data structure that is used to  store a dynamic set or associative array where the keys are usually &#xA;  strings.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/google/pygtrie&quot; rel=&quot;nofollow noreferrer&quot;&gt;pygtrie&lt;/a&gt; offers an implementation of tries with a dict interface. Here goes an example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pygtrie as trie&#xA;&#xA;words = ['cat', 'caterpillar', 'dog', 'mouse']&#xA;&#xA;structure = trie.Trie()&#xA;&#xA;for i, word in enumerate(words):&#xA;   structure[word] = i&#xA;&#xA;print structure['caterpillar']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-01-16T21:51:00.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16356" PostTypeId="2" ParentId="16348" CreationDate="2017-01-16T23:10:48.247" Score="3" Body="&lt;p&gt;I'd like to extend &lt;a href=&quot;https://datascience.stackexchange.com/a/16352/20492&quot;&gt;the great @Emre's answer&lt;/a&gt; with another example - we are going to replace all tokenized words from the &quot;1984&quot; (c) George Orwell (120K words):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [163]: %paste&#xA;import requests&#xA;import nltk&#xA;import pandas as pd&#xA;&#xA;# source: https://github.com/dwyl/english-words&#xA;fn = r'D:\temp\.data\words.txt'&#xA;url = 'http://gutenberg.net.au/ebooks01/0100021.txt'&#xA;&#xA;r = requests.get(url)&#xA;&#xA;# read words into Pandas DataFrame&#xA;df = pd.read_csv(fn, header=None, names=['word'])&#xA;# shuffle DF, so we will have random indexes&#xA;df = df.sample(frac=1)&#xA;# convert Pandas DF into dictionary: {'word1': unique_number1, 'word2': unique_number2, ...}&#xA;lkp = df.reset_index().set_index('word')['index'].to_dict()&#xA;&#xA;# tokenize &quot;1984&quot; (c) George Orwell&#xA;words = nltk.tokenize.word_tokenize(r.text)&#xA;&#xA;print('Word Dictionary size: {}'.format(len(lkp)))&#xA;print('We have tokenized {} words...'.format(len(words)))&#xA;## -- End pasted text --&#xA;Word Dictionary size: 354983&#xA;We have tokenized 120251 words...&#xA;&#xA;In [164]: %timeit [lkp.get(w, 0) for w in words]&#xA;10 loops, best of 3: 66.3 ms per loop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion:&lt;/strong&gt; it took 66 ms to build a list of numbers for the list with 120K words from the dictionary containing 354.983 entries.&lt;/p&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-01-16T23:10:48.247" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16357" PostTypeId="1" CreationDate="2017-01-17T01:34:14.597" Score="1" ViewCount="191" Body="&lt;p&gt;My university has a building on campus where electrical usage and water usage is monitored down to the second.  Many electrical loads and water usageis tracked and stored in a database.  In the water dataset, the column headers are the location, and the rows contain the gallons used (per second).  In the electrical dataset, the power consumption (in Watts) is tracked in the rows and the load type is the column header.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My general problem is as follows: - &lt;strong&gt;how could I use the data from this highly monitored building, to benefit the university and give them insights into buildings that aren't monitored like this one?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is of the time-series, so I am trying to think of certain clustering or classification algorithms that could help me extract actionable data from it, and perhaps even association rule learning to find relations between the water and electricity data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insights!&lt;/p&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="12202" LastEditDate="2017-01-26T14:08:59.223" LastActivityDate="2022-12-24T17:23:23.360" Title="Association Rule Learning for Home Electricity or Water Data?" Tags="|machine-learning|classification|data-mining|feature-selection|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16361" PostTypeId="1" AcceptedAnswerId="16362" CreationDate="2017-01-17T09:25:54.793" Score="12" ViewCount="5646" Body="&lt;p&gt;In many tensorflow tutorials (&lt;a href=&quot;https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py&quot; rel=&quot;noreferrer&quot;&gt;example&lt;/a&gt;) &quot;towers&quot; are mentioned without a definition. What is meant by that?&lt;/p&gt;&#xA;" OwnerUserId="28011" LastEditorUserId="31513" LastEditDate="2017-05-18T09:25:32.997" LastActivityDate="2021-11-03T00:12:08.727" Title="What is a tower?" Tags="|deep-learning|tensorflow|terminology|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16362" PostTypeId="2" ParentId="16361" CreationDate="2017-01-17T09:30:14.260" Score="8" Body="&lt;p&gt;According to tensorflow documentation about CNN,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The first abstraction we require is a function for computing inference and gradients for a single model replica. In the code we term this abstraction a &quot;tower&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To get the relevant context and more, check &lt;a href=&quot;https://www.tensorflow.org/tutorials/deep_cnn/&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-01-17T09:30:14.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16363" PostTypeId="1" AcceptedAnswerId="16493" CreationDate="2017-01-17T09:50:27.347" Score="4" ViewCount="508" Body="&lt;p&gt;I am looking for a challenge that is suitable for a group of novices who want to learn the basics of data science and machine learning. The challenge should match the following criteria:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is based on a real application or is at least realistic&lt;/li&gt;&#xA;&lt;li&gt;has a clearly defined goal and partial progress is measurable&lt;/li&gt;&#xA;&lt;li&gt;includes a machine learning component, but also other aspects of data science&lt;/li&gt;&#xA;&lt;li&gt;should be doable within 3 to 6 weeks&lt;/li&gt;&#xA;&lt;li&gt;is suitable for novices&lt;/li&gt;&#xA;&lt;li&gt;it should be an actual challenge in the sense that you cannot just look up near-optimal solutions from the internet&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28012" LastEditorUserId="28012" LastEditDate="2017-01-17T15:03:21.937" LastActivityDate="2017-01-24T11:58:02.120" Title="Can you recommend a machine learning challenge that is suitable for novices?" Tags="|machine-learning|education|kaggle|" AnswerCount="3" CommentCount="2" CommunityOwnedDate="2017-01-24T12:14:21.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="16366" PostTypeId="2" ParentId="16347" CreationDate="2017-01-17T12:34:34.527" Score="5" Body="&lt;p&gt;You should look at a classifier based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;noreferrer&quot;&gt;Dynamic Time Warping (DTW) distance&lt;/a&gt;. DTW is a method that calculates an optimal match between two given sequences (e.g. time series). It has been used in a supervised learning setup, in particular it has been &lt;a href=&quot;http://dl.acm.org/citation.cfm?doid=1454159.1454226&quot; rel=&quot;noreferrer&quot;&gt;reported&lt;/a&gt; to achieved state of the art results, when used in a nearest neighbour classifier. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-01-17T12:34:34.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16367" PostTypeId="2" ParentId="16363" CreationDate="2017-01-17T12:53:21.653" Score="2" Body="&lt;p&gt;You've already answered yourself by tagging &lt;code&gt;kaggle&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me share the two competitions that I have every new hire in my team go through- and then have them keep improving their solurion for the next six months. These are a very well curated set of problems for a budding data scientist!&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Titanic: Machine Learning from Disaster&#xA;&lt;a href=&quot;https://www.kaggle.com/c/titanic&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/titanic&lt;/a&gt;&#xA;This one helps bridge the gap between an analyst and a data scientist, as well as eases you into the world of code from Excel.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Digit Recognizer&#xA;&lt;a href=&quot;https://www.kaggle.com/c/digit-recognizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/digit-recognizer&lt;/a&gt;&#xA;The MNIST dataset has prepared almost every data scientist for the real world of dirty data, in a sweet, cushy manner. It makes very real the fact that not always will we have structured and organized data, but it should not deter us from gaining insight from it!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Happy Coding!&lt;/p&gt;&#xA;" OwnerUserId="8830" LastActivityDate="2017-01-17T12:53:21.653" CommentCount="1" CommunityOwnedDate="2017-01-24T12:14:21.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="16370" PostTypeId="1" CreationDate="2017-01-17T16:45:11.940" Score="0" ViewCount="368" Body="&lt;p&gt;I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the idea how to create the synthetic example to bring noticeable accuracy on a given dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to go into deep and understand how a classifier, especially SVM handle the data with the synthetic example to classify more accurately. It would also helpful to know for other techniques like boosting algorithms, random forest etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any kind of guidance on above question will be very much helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already asked this question on StackExchange &lt;a href=&quot;https://stackoverflow.com/q/40154819/5520970&quot;&gt;here&lt;/a&gt; but didn't get any answer.&lt;/p&gt;&#xA;" OwnerUserId="27925" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-01-21T02:38:10.240" Title="How synthetic sampling of data helps classifier to predict better result?" Tags="|data-mining|dataset|algorithms|data|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16371" PostTypeId="1" AcceptedAnswerId="16375" CreationDate="2017-01-17T18:36:03.783" Score="2" ViewCount="191" Body="&lt;p&gt;I have a dataset which contains time-series data of water flow over time.  I have a flow meter connected to a kitchen faucet, and I am trying to cluster or classify specific water usage events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is collected every second, and in each row I am given a value for the amount of gallons which are flowing through my flow meter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I am trying to classify someone washing their hands, filling a teapot, cleaning dishes, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something that I can use a k-NN Classification Approach to cluster these events?  If a clustering based approach isn't good, what other method of classified would be good for this type of data?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I run some experiments, I can classify each event and turn it into a supervised learning problem.  But at the moment, none of the water events are classified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very abridged version of my dataset looks like the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0fvXQ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0fvXQ.jpg&quot; alt=&quot;Time Series Water Usage&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;water = pd.DataFrame(shower1)&#xA;rng = pd.date_range('2016-09-01 00:00:00', '2016-09-30 23:59:58', freq='S')&#xA;water = water.reindex(rng,fill_value=0.0)&#xA;water = water['shower1']&#xA;df = pd.DataFrame({'time_stamp':rng,'water_amount':water})&#xA;&#xA;starts = (df['water_amount']&amp;gt;0)&amp;amp;(df['water_amount'].shift(1)==0) #find all starts of events&#xA;n_events = sum(starts) #total number of events&#xA;df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n&#xA;df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values&#xA;df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0&#xA;&#xA;df.groupby('event_number').agg({'time_stamp':'first',&#xA;                                    'water_amount':'sum'}) #feature matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DrGs3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DrGs3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="12202" LastEditDate="2017-01-19T19:08:45.670" LastActivityDate="2017-01-19T19:08:45.670" Title="Using a K-NN Classification Approach for Time Series Data?" Tags="|neural-network|clustering|unsupervised-learning|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16373" PostTypeId="2" ParentId="16363" CreationDate="2017-01-17T22:13:00.077" Score="1" Body="&lt;p&gt;I would highly recommend the platform - &lt;a href=&quot;https://datahack.analyticsvidhya.com/contest/all/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://datahack.analyticsvidhya.com/contest/all/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AnalyticsVidhya is an amazing community for data science. Not only contests, but they also have technical articles on their blog (&lt;a href=&quot;https://www.analyticsvidhya.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com&lt;/a&gt;). Though I used to blog for them, but my opinion is not biased by that fact. I used to follow them for a couple of years before that and I felt lucky that I got a chance to contribute to the community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of interesting problems, I would recommend the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The Smart Recruits&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The Creative Analyst&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Date Your Data&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sorry I'm new here and not allowed to post more than 2 links. You'll find these competitions on their website. The datasets are available and you can make submissions to benchmark yourself. Also, I would suggest sign up for their emails. They conduct short 2-3 day hackathons which offer great learning experiences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this help!&lt;/p&gt;&#xA;" OwnerUserId="28031" LastActivityDate="2017-01-17T22:13:00.077" CommentCount="2" CommunityOwnedDate="2017-01-24T12:14:21.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="16374" PostTypeId="1" AcceptedAnswerId="16501" CreationDate="2017-01-17T23:08:13.620" Score="10" ViewCount="2401" Body="&lt;p&gt;Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be looking for something, which is conceptually similar to MATLAB's persistent variables. There it is possible to call a custom function / library from multiple individual editors (notebooks), and have that external function cache some result (or large table).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mostly I would like to avoid reloading a heavily used table (which is loaded through a custom library that is is called from the notebooks ), since reading it takes around 2-3 minute whenever I start a new analysis.&lt;/p&gt;&#xA;" OwnerUserId="23437" LastActivityDate="2017-01-26T20:01:49.067" Title="Avoid reloading DataFrame between different python kernels" Tags="|python|bigdata|matlab|jupyter|ipython|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16375" PostTypeId="2" ParentId="16371" CreationDate="2017-01-17T23:41:01.520" Score="3" Body="&lt;p&gt;It seems pretty clear from looking at the data &lt;em&gt;when&lt;/em&gt; an event starts and ends(basically whenever there is a sequence of positive values). So, instead of starting with some complicated models, I'd suggest calculating a few simple features (like length of the event, total amount of water, amount/seconds, time to previous event, time of day in seconds from start of recording) for every event and then try some clustering algorithm on that new data. k-NN might even produce something meaningful. But a statistical summary of the features can probably already give you a better idea of how to further approach this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT1&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')&#xA;water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]&#xA;df = pd.DataFrame({'time_stamp':rng,'water_amount':water,'event_number':np.zeros(len(water))})&#xA;&#xA;j = 1&#xA;for k in range(len(df)):&#xA;    if df.ix[k,'water_amount']== 0:&#xA;        df.ix[k,'event_number'] = -1&#xA;    else:&#xA;        if df.ix[k-1,'water_amount'] &amp;gt; 0:&#xA;            df.ix[k,'event_number'] = df.loc[k-1,'event_number']&#xA;        else:&#xA;            df.ix[k,'event_number'] = j&#xA;            j = j+1&#xA;&#xA;&#xA;df.groupby('event_number').agg({'time_stamp':'first',&#xA;                                'water_amount':'sum'}) #feature matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')&#xA;water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]&#xA;df = pd.DataFrame({'time_stamp':rng,'water_amount':water})&#xA;&#xA;starts = (df['water_amount']&amp;gt;0)&amp;amp;(df['water_amount'].shift(1)==0) #find all starts of events&#xA;n_events = sum(starts) #total number of events&#xA;df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n&#xA;df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values&#xA;df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0&#xA;&#xA;df.groupby('event_number').agg({'time_stamp':'first',&#xA;                                    'water_amount':'sum'}) #feature matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-01-19T16:44:19.697" LastActivityDate="2017-01-19T16:44:19.697" CommentCount="14" ContentLicense="CC BY-SA 3.0" />
  <row Id="16376" PostTypeId="2" ParentId="16371" CreationDate="2017-01-18T00:27:57.763" Score="1" Body="&lt;p&gt;Don't forget preprocessing your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, do &lt;strong&gt;feature extraction&lt;/strong&gt;n&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;total amount of water&lt;/li&gt;&#xA;&lt;li&gt;duration&lt;/li&gt;&#xA;&lt;li&gt;variance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-01-18T00:27:57.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16377" PostTypeId="2" ParentId="6232" CreationDate="2017-01-18T00:44:44.920" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;You might want to normalize all the continuous variables into one range (0-1)&lt;/li&gt;&#xA;&lt;li&gt;Normalize the categorical variables as a  &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;One Hot Enconder&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Apply Similarity algorithms like Pearson Correlation / Distance algorithms like (Euclidean, Cosine similarity)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28034" LastActivityDate="2017-01-18T00:44:44.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16378" PostTypeId="2" ParentId="527" CreationDate="2017-01-18T00:53:31.453" Score="1" Body="&lt;p&gt;I think your goal is to find how similar two documents are, if that's the case I suggest apply following algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach gives how much similar Doc1 is wrt Doc2. (Similarity values will be different for Doc2 wrt Doc1 if its not a square matrix)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;In your matrix between Doc1 and Doc2, Get the max similarity value row by row.&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Take the sum and divide by number of rows&lt;/li&gt;&#xA;&lt;li&gt;This will give you the similarity index.&#xA;For eg. &#xA;In your matrix image, I see maximum similarity row-by-row is:&#xA;0.88 , 1, 0.6&#xA;So (0.88 + 1 + 0.6)/3 = 82.67%&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This means &lt;strong&gt;Doc2 is 82.67% similar to Doc1&lt;/strong&gt;. The similarity cannot go beyond this value as we selected max similar items in each row.&lt;/p&gt;&#xA;" OwnerUserId="28034" LastActivityDate="2017-01-18T00:53:31.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16379" PostTypeId="1" AcceptedAnswerId="16638" CreationDate="2017-01-18T02:02:18.210" Score="7" ViewCount="10363" Body="&lt;p&gt;I am building a entity detection and relation classification method using deep learning approach which requires vector representation of POS tags and entity label. I am familiar with word-embedding method but I don't know the answer of following questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to convert pos tags into vector representation(say 20 dim)&lt;/li&gt;&#xA;&lt;li&gt;How to combine both word embeddings and pos embedding together to build the classifier.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Similar question was asked in Cross validate community but I couldn't find the answer. Here is the link to question:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech&quot;&gt;https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Research paper link:&#xA;&lt;a href=&quot;https://arxiv.org/abs/1601.00770&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1601.00770&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27906" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2020-02-04T05:02:05.363" Title="Word embedding/Word2vec for POS tagging" Tags="|neural-network|nlp|deep-learning|rnn|word2vec|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16380" PostTypeId="1" CreationDate="2017-01-18T02:47:10.753" Score="1" ViewCount="227" Body="&lt;p&gt;I've run random forest on my dataset (imbalanced, binary target class) and used cross validation to tune the parameter and use recursive feature elimination with cross-validation to get the subset of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I can present the optimal parameter setting as well as the set of features for model fitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how about the probabilistic threshold? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I conclude a threshold as the important part of the final model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also using CV? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this threshold is not stable among folds of CV, what to do next? &lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="8285" LastEditDate="2017-01-18T16:14:48.107" LastActivityDate="2017-01-18T22:38:34.623" Title="How to present the final model (e.g. random forest)?" Tags="|machine-learning|python|feature-selection|random-forest|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16381" PostTypeId="2" ParentId="16325" CreationDate="2017-01-18T04:02:29.163" Score="1" Body="&lt;p&gt;After some researches and trial, the dimensions need to be normalised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, I consider 0 as normal and 1 be outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I normalise the dimensions by take the 10% and 90% of the samples.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = svm.SVC(max_iter=1, kernel='rbf')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ozGjA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ozGjA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20907" LastEditorUserId="20907" LastEditDate="2017-01-18T04:15:39.123" LastActivityDate="2017-01-18T04:15:39.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16382" PostTypeId="2" ParentId="15328" CreationDate="2017-01-18T04:05:03.930" Score="1" Body="&lt;p&gt;The answer can be found in the  Going deeper with convolutions paper: &lt;a href=&quot;https://arxiv.org/pdf/1512.00567v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1512.00567v3.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check Table 3. Inception v2 is the architecture described in the Going deeper with convolutions paper. Inception v3 is the same architecture (minor changes) with different training algorithm (RMSprop, label smoothing regularizer, adding an auxiliary head with batch norm to improve training etc).&lt;/p&gt;&#xA;" OwnerUserId="28040" LastEditorUserId="28040" LastEditDate="2017-01-18T04:50:54.220" LastActivityDate="2017-01-18T04:50:54.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16383" PostTypeId="2" ParentId="16347" CreationDate="2017-01-18T07:32:52.263" Score="1" Body="&lt;p&gt;These are some suggestions that might be useful.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The data on the curve are bumpier than the roads in my country. So I think you should start by smoothing the curve. There are many smoothing filters like from the simplest &lt;a href=&quot;https://en.wikipedia.org/wiki/Median_filter&quot; rel=&quot;nofollow noreferrer&quot;&gt;median smoothing&lt;/a&gt; to Local Regression models like &lt;a href=&quot;https://en.wikipedia.org/wiki/Local_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;LOESS&lt;/a&gt;. There are some parameters to tweak. Take a look at the example.&lt;/li&gt;&#xA;&lt;li&gt;Finding the local maxima. &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.argrelextrema.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python's numpy&lt;/a&gt; has an implementation for this and this should help.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My idea is to basically smooth till you get your head and shoulders i.e., three maxima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;: Smoothing though reduces the amount of noise (not in literal noise sense) on the curve, it tends to shift the curve from its original position to represent it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample Python implementation will be like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from statsmodels.nonparametric.smoothers_lowess import lowess&#xA;import numpy as np&#xA;from scipy.signal import argrelextrema&#xA;import matplotlib.pyplot as plt&#xA;sample_points =  np.array([1,2.3,3.5,3,4.5,5,2.25,33.3,5,6.7,7.3,56.0,70.1,4.2,5.4,6.2,4.4,100,2.9,45,10,3.4,4.8,50,2.3,3.45,5.5,6.7,7.9,8.7,6.1])&#xA;for i in np.arange(0,0.5,0.05):&#xA;    # i in the loop is the percentage of data points we are inputing for the loess regression. Wiki atricle explains it, I guess&#xA;    filtered = lowess(sample_points,range(len(sample_points)), is_sorted=True, frac=i, it=0)&#xA;    maxima = argrelextrema(filtered[:,1], np.greater)&#xA;    if len(maxima[0]) == 3:&#xA;        plt.plot(filtered[:,1])&#xA;        plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this kind of gives the direction of where you might need some checking.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-18T07:32:52.263" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16384" PostTypeId="1" CreationDate="2017-01-18T10:38:58.987" Score="2" ViewCount="1242" Body="&lt;p&gt;I have a rather general question.&#xA;I am trying to create a classification model and my training data consists of various variables. I have figured out that it would be most efficient if I convert all my data into numeric types. However, as I am using 2D numpy array and I have to use float for values that are originally stored as integers. I wanted to ask if having such conversion would effect the efficiency of my prediction,&#xA;or simply:&#xA;Is there a difference between having 1.00-0.00's and 1-0's in a training set.&lt;/p&gt;&#xA;" OwnerUserId="28049" LastEditorUserId="21024" LastEditDate="2017-01-18T11:53:48.383" LastActivityDate="2017-01-18T11:53:48.383" Title="data type (int vs float) with sklearn models" Tags="|classification|scikit-learn|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16385" PostTypeId="2" ParentId="16370" CreationDate="2017-01-18T12:32:59.943" Score="0" Body="&lt;p&gt;How about using a GAN ( Generative Adversarial Network) to generate undisdinguishable data for your imbalanced dataset. An example for this can be seen here: (&lt;a href=&quot;https://github.com/osh/KerasGAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/osh/KerasGAN&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="25934" LastActivityDate="2017-01-18T12:32:59.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16388" PostTypeId="1" CreationDate="2017-01-18T15:36:08.237" Score="3" ViewCount="59" Body="&lt;p&gt;I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by the explanation. Here is what I did using Stata:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Dependent variable is team win's and independent variables is different types of opponent shooting data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Paneled data using xtset to account for the different NBA seasons&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Ran robust xtreg for both fixed and random effects&lt;/li&gt;&#xA;&lt;li&gt;Hausman test had a high chi^2, so I stayed with the GLS random effects regression.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If this is incorrect, please let me know so I can fixed this. I always thought that the mixed effects model was for logistic regressions when my dependent variable was categorical. I was told that I did not take into account this team grouping structure within your analysis, but the Group Variable was the teams.&lt;/p&gt;&#xA;" OwnerUserId="28062" LastActivityDate="2023-03-27T20:05:25.800" Title="Am I supposed to be using Mixed Effects?" Tags="|dataset|regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16389" PostTypeId="1" AcceptedAnswerId="16403" CreationDate="2017-01-18T19:54:04.680" Score="-1" ViewCount="1508" Body="&lt;p&gt;&lt;strong&gt;EDITED:&lt;/strong&gt;&#xA;Is it accepted practice to be able to use a KMeans clustering algorithm to help label data fed into a supervised model? (Unsupervised --feeds-&gt; supervised)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason being, relabeling millions of records is not possible and it is a class imbalance problem, where that historical minority class is very useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like this is a reinforcement learning problem, but do not know enough about it to say so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If none of the above, what is a good approach for an imbalanced fraud detection model? Where Precision and recall are better measures than Accuracy.&lt;/p&gt;&#xA;" OwnerUserId="23800" LastEditorUserId="23800" LastEditDate="2017-01-19T23:34:58.910" LastActivityDate="2017-01-19T23:34:58.910" Title="KMeans clustering to help label Multi-class Supervised model" Tags="|clustering|scikit-learn|k-means|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16390" PostTypeId="1" CreationDate="2017-01-18T20:00:55.200" Score="1" ViewCount="123" Body="&lt;p&gt;Has anyone created statistics on how fast and accurate Inception V3 can classify an image based on criteria such as: different models of GPUs/CPUs, input image size, input image ratio, file format, how it runs on iOS and Android, etc?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example chart I'd hope to see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;GPU | CPU | Ratio |   Size   |  Speed  | Accuracy    &#xA; x           1:1     100x100    1000ms     93%&#xA;       x     3:4     3k x 4k   40000ms     95%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;X as in type of GPU/ CPU&lt;/p&gt;&#xA;" OwnerUserId="27641" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:12.207" LastActivityDate="2021-02-16T15:24:12.207" Title="Inception V3 Running Stats?" Tags="|tensorflow|convolutional-neural-network|inception|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16392" PostTypeId="2" ParentId="16380" CreationDate="2017-01-18T21:49:42.750" Score="1" Body="&lt;p&gt;The probability threshold can be changed and you would obtain different results for accuracy / recall / precision / F1 score (the objective). You could see this as an aspect of the parameters you have tuned to obtain the best objective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly for CV, the out-of-training fold objective would be reported together with the other parameters, such as number of trees and predictor subset size. One reason I could think of regarding the instability of threshold across CV folds is that the CV folds were not split up using stratified sampling, resulting in different proportions of positives observed across the folds.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2017-01-18T21:49:42.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16393" PostTypeId="2" ParentId="16380" CreationDate="2017-01-18T22:38:34.623" Score="0" Body="&lt;p&gt;As a follow up to jkyh, I also would like to note that if the variance of your model is high that might also contribute to the changing metrics bewteen cv folds. You should examine bias/variance tradeoff.&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-01-18T22:38:34.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16395" PostTypeId="1" CreationDate="2017-01-19T02:42:31.800" Score="0" ViewCount="81" Body="&lt;p&gt;Given a list of software installed by users as features, e.g.,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Microsoft_VC80_DebugCRT_x86_x64 1.0.0; Microsoft_VC80_DebugCRT_x86 1.0.0; ;Windows UPnP Browser 0.1.01;Adobe Acrobat Professional 10;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict whether the student will buy certain product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the question is: what are the ways to turn the list of software into something learnable by a machine learning algorithm?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2017-01-19T14:04:54.107" Title="Feature usage for machine learning algorithm" Tags="|machine-learning|feature-extraction|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16396" PostTypeId="2" ParentId="16395" CreationDate="2017-01-19T03:17:11.280" Score="0" Body="&lt;p&gt;Treat the installed software as categorical variables, and train a binary classifier such as logistic regression using training data, if you have it. If you don't, there is nothing you can do. You could create derived categorical variables from the company and type of software product, etc.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-01-19T03:17:11.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16397" PostTypeId="2" ParentId="16395" CreationDate="2017-01-19T03:17:19.480" Score="1" Body="&lt;p&gt;I would build a logistic regression with multiple independent variables. I don't think this is the only possibility, but logistic regression makes sense you are trying to model for the probability of purchase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, your dependent variable will be whether a product is purchased, so it should be binary. Your independent variables will also be binary. You can also merge the binary variables into smaller number of categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should do do at least pairwise correlation (e.g. Phi coefficient). You can use it to merge highly correlated variables, such as, &lt;code&gt;Microsoft Windows Word&lt;/code&gt; and &lt;code&gt;Microsoft Windows Excel&lt;/code&gt; into &lt;code&gt;Microsoft Office&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-01-19T03:17:19.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16399" PostTypeId="1" CreationDate="2017-01-19T06:58:18.047" Score="2" ViewCount="658" Body="&lt;p&gt;I just stumbled across &lt;a href=&quot;http://kur.deepgram.com/tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kur&lt;/a&gt;. At first look, it seems to be making deep learning easy in almost similar lines like Keras on a high level. What are the pros and cons of each and what is suggested to be used for beginners?&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-02-16T20:54:45.667" Title="Kur vs Keras - pros and cons" Tags="|deep-learning|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16401" PostTypeId="2" ParentId="16399" CreationDate="2017-01-19T07:15:02.123" Score="2" Body="&lt;p&gt;I have been using Keras for quite some time.The following are some pros and cons of keras:-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Easier to use and build model(pro)&lt;/li&gt;&#xA;&lt;li&gt;Can use both Tensorflow and Theano backend(pro)&lt;/li&gt;&#xA;&lt;li&gt;Less time consuming and reusability is excellent.(pro)&lt;/li&gt;&#xA;&lt;li&gt;A lot of the folks have complained about performance issues with Tensorflow backend(con).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Kur recently got open sourced by Deepgram.I find the way of defining models unique and simpler, but I am not sure of the robustness.&#xA; Click &lt;a href=&quot;https://techcrunch.com/2017/01/18/deepgram-open-sources-kur-to-make-diy-deep-learning-less-painful/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; to know about it.&lt;/p&gt;&#xA;" OwnerUserId="28086" LastActivityDate="2017-01-19T07:15:02.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16403" PostTypeId="2" ParentId="16389" CreationDate="2017-01-19T08:15:06.897" Score="3" Body="&lt;p&gt;k-means will not 'label' points for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is not classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a much harder problem. Most of the time, you get bad results!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So rather than trying to automate this, use clustering to understand your data. Try to derive some rules to identify e.g. different kind of fraud. But &lt;strong&gt;never assume the clusters are all good&lt;/strong&gt; (because they never are &lt;em&gt;all&lt;/em&gt; good).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-01-19T08:15:06.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16404" PostTypeId="1" CreationDate="2017-01-19T08:21:21.470" Score="2" ViewCount="157" Body="&lt;p&gt;I'm very new to machine learning (and statistics) and I'm struggling with some basics (I'm using R as my primary environment). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lexicon with words and all of their forms (in different genders, persons - for a Slavic language). So each word with its word forms is one training sample. My training attributes are last character, last two characters, last three characters of each word form.  So each training sample contains from around 100 categorical attributes. I've got 10000 samples. One attribute has around 1700 levels, one around 1200, 5 around 500, others have less than 100 levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used DBSCAN (and gower distance) to cluster similars words to the same cluster. This step was pretty successful. I get (depending whether is a noun, verb or adjective) from 40-100 clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now here comes the part where I don't know how to continue. If I get a new (unknown) word I want to predict to which cluster it belongs. Sometimes for this new word I only know its base form, sometimes I also part of its word forms and sometimes I know all of its word forms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this should be some kind of classification with multiple classes. Question: which algorithm should I use? I've tried to use &lt;code&gt;multinom&lt;/code&gt; (in R) but I get an error of too many weights. After I increased &lt;code&gt;MaxNWts&lt;/code&gt; I get an error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weights:  641676 (633954 variable)&#xA;Error: cannot allocate vector of size 396 Kb&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where should I continue? Clean the data? Use a different algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, around 15 attributes have only 1 level or NA. How does this impact to clustering (and classification)? Is it safe to remove those attributes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="28085" LastEditorUserId="25157" LastEditDate="2017-01-20T05:35:55.690" LastActivityDate="2017-02-23T19:58:16.047" Title="What kind of classification should I use?" Tags="|r|classification|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16405" PostTypeId="2" ParentId="10541" CreationDate="2017-01-19T09:43:24.863" Score="-1" Body="&lt;p&gt;Another approach would be to add a feature that captures which stores the customer was eligible to avail discounts at. A classification model given a large enough dataset would probably figure out that connection:)&lt;/p&gt;&#xA;" OwnerUserId="28092" LastActivityDate="2017-01-19T09:43:24.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16407" PostTypeId="2" ParentId="6107" CreationDate="2017-01-19T11:18:55.407" Score="-1" Body="&lt;p&gt;The following paper discusses deconvolutional layers.Both from the architectural and training point of view.&lt;a href=&quot;http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deconvolutional networks&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="28086" LastActivityDate="2017-01-19T11:18:55.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16409" PostTypeId="2" ParentId="16395" CreationDate="2017-01-19T14:04:54.107" Score="-1" Body="&lt;p&gt;Your problem fits the domain of recommendation engines.&lt;br&gt;&#xA;Based on user's used software, you wish to evaluate his chance of using another one(and maybe buying it).   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;document&lt;/a&gt; describes the logic pretty well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The item profiling phase can be done using a pairwise correlation, and other heuristics.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be aware that usually these method need more data than standard classifiers(logistic regression/decision trees/~SVM)&lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-01-19T14:04:54.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16410" PostTypeId="2" ParentId="9915" CreationDate="2017-01-19T14:05:09.513" Score="1" Body="&lt;p&gt;To put it simply, it will give you a equation (less error relative to data-set presumed to be correct equation). Depending on data-set, equation has constants value for each variables which will provides the ratio of one variable with other variables.&#xA;You might find equation on similar format as this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = (b0 + b1.x1 + b2.x2 + b3.x3) + c&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;strong&gt;&lt;em&gt;x1&lt;/em&gt;&lt;/strong&gt;  , &lt;strong&gt;&lt;em&gt;x2&lt;/em&gt;&lt;/strong&gt; , &lt;strong&gt;&lt;em&gt;x3&lt;/em&gt;&lt;/strong&gt; can be different characteristics (as variables which can be &lt;em&gt;IQ, age, working momory, etc.&lt;/em&gt;) and &lt;strong&gt;&lt;em&gt;b0&lt;/em&gt;&lt;/strong&gt; , &lt;strong&gt;&lt;em&gt;b1&lt;/em&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;em&gt;b2&lt;/em&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;em&gt;b3&lt;/em&gt;&lt;/strong&gt; are the constant calculated from given data-set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find exact value of &lt;strong&gt;&lt;em&gt;y&lt;/em&gt;&lt;/strong&gt; or &lt;strong&gt;&lt;em&gt;x1&lt;/em&gt;&lt;/strong&gt; or &lt;strong&gt;&lt;em&gt;x2&lt;/em&gt;&lt;/strong&gt; or &lt;strong&gt;&lt;em&gt;x3&lt;/em&gt;&lt;/strong&gt; if you have exact value of other &lt;strong&gt;&lt;em&gt;3&lt;/em&gt;&lt;/strong&gt; variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say, it was the data-set provided to an android/robot as in sci-fi movies; that android/robot will make it's decision based on this equation as &lt;strong&gt;&lt;em&gt;law/fact/pattern&lt;/em&gt;&lt;/strong&gt; and judge a person response time (if every other variables are known) or IQ (if every other variables are known) etc. by this equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also a lot of details in &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat501/node/284&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: This logic will only practical with Multiple Linear Regression Method. &lt;/p&gt;&#xA;" OwnerUserId="27992" LastEditorUserId="27992" LastEditDate="2017-01-19T14:22:23.590" LastActivityDate="2017-01-19T14:22:23.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16411" PostTypeId="1" AcceptedAnswerId="16421" CreationDate="2017-01-19T16:28:21.157" Score="5" ViewCount="548" Body="&lt;p&gt;This question is only about the vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do / can you say&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;data item&lt;/li&gt;&#xA;&lt;li&gt;data sample&lt;/li&gt;&#xA;&lt;li&gt;recording&lt;/li&gt;&#xA;&lt;li&gt;sample&lt;/li&gt;&#xA;&lt;li&gt;data point&lt;/li&gt;&#xA;&lt;li&gt;something else&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;when you talk about elements of the training / test set? For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The figure shows 100 data items of the training set.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Database A contains the same data items as database B, but in another format.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The remaining data items were removed from the dataset.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Those 10 classes have 123456 data items.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please provide papers with examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;https://books.google.com/ngrams/graph?content=data+item%2Cdata+point%2Cdata+sample&amp;amp;case_insensitive=on&amp;amp;year_start=1920&amp;amp;year_end=2020&amp;amp;corpus=15&amp;amp;smoothing=3&amp;amp;share=&amp;amp;direct_url=t4%3B%2Cdata%20item%3B%2Cc0%3B%2Cs0%3B%3Bdata%20item%3B%2Cc0%3B%3BData%20Item%3B%2Cc0%3B%3BData%20item%3B%2Cc0%3B%3BDATA%20ITEM%3B%2Cc0%3B.t4%3B%2Cdata%20point%3B%2Cc0%3B%2Cs0%3B%3Bdata%20point%3B%2Cc0%3B%3BData%20Point%3B%2Cc0%3B%3BData%20point%3B%2Cc0%3B%3BDATA%20POINT%3B%2Cc0%3B.t4%3B%2Cdata%20sample%3B%2Cc0%3B%2Cs0%3B%3Bdata%20sample%3B%2Cc0%3B%3BData%20Sample%3B%2Cc0%3B%3Bdata%20Sample%3B%2Cc0%3B%3BDATA%20SAMPLE%3B%2Cc0%3B%3BData%20sample%3B%2Cc0%3B%3BDATA%20Sample%3B%2Cc0&quot; rel=&quot;noreferrer&quot;&gt;Google n-grams&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1tzNa.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1tzNa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-01-20T09:52:20.163" Title="How is a single element of the training set called?" Tags="|data|terminology|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16413" PostTypeId="1" CreationDate="2017-01-19T23:12:22.880" Score="3" ViewCount="7585" Body="&lt;p&gt;I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import nltk&#xA;from nltk.corpus import inaugural&#xA;cfd = nltk.ConditionalFreqDist(&#xA;            (target, fileid[:4]) # &amp;quot;[:4]&amp;quot; slices only the years of the speeches&#xA;            for fileid in inaugural.fileids()&#xA;            for word in inaugural.words(fileid)&#xA;            for target in [&amp;quot;liberty&amp;quot;, &amp;quot;equality&amp;quot;, &amp;quot;brotherhood&amp;quot;]&#xA;            if word.lower().startswith(target))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Because it gets &lt;strong&gt;very crowded&lt;/strong&gt; otherwise:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;cfd.plot(title=&amp;quot;French ideals in US-American speeches through time&amp;quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JUiKB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JUiKB.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;the &lt;code&gt;__doc__&lt;/code&gt; doesn't seem to mention it:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;print(cfd.plot.__doc__)&#xA;&#xA;        Plot the given samples from the conditional frequency distribution.&#xA;        For a cumulative plot, specify cumulative=True.&#xA;        (Requires Matplotlib to be installed.)&#xA;&#xA;        :param samples: The samples to plot&#xA;        :type samples: list&#xA;        :param title: The title for the graph&#xA;        :type title: str&#xA;        :param conditions: The conditions to plot (default is all)&#xA;        :type conditions: list&#xA;        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And I think there is nothing on it in the &lt;a href=&quot;http://www.nltk.org/py-modindex.html#cap-p&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK documentation&lt;/a&gt;,nor the &lt;a href=&quot;https://matplotlib.org/2.0.2/api/pyplot_api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;matplotlib documentation&lt;/a&gt; (where I figured the plot functionality comes from)&lt;/p&gt;&#xA;" OwnerUserId="28111" LastEditorUserId="24968" LastEditDate="2022-06-17T06:01:29.640" LastActivityDate="2022-06-17T06:01:29.640" Title="How to change plot size in nltk.plot()" Tags="|python|nlp|nltk|jupyter|ipython|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16416" PostTypeId="1" AcceptedAnswerId="16419" CreationDate="2017-01-20T05:29:48.710" Score="5" ViewCount="2185" Body="&lt;p&gt;In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what are the theoretical reasons(prerequisite) for this choice?&lt;/p&gt;&#xA;" OwnerUserId="28081" LastEditorUserId="28081" LastEditDate="2017-01-20T05:36:16.513" LastActivityDate="2021-11-29T19:15:39.240" Title="Why the number of neurons or convolutions chosen equal powers of two?" Tags="|machine-learning|neural-network|information-theory|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16419" PostTypeId="2" ParentId="16416" CreationDate="2017-01-20T09:44:42.377" Score="4" Body="&lt;p&gt;Deep Neural Networks are usually trained on GPUs to speed up training time. Using power of two for the network topology follows the same logic as using power of two for image textures in computer games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The GPU can take advantage of optimizations related to efficiencies in working with powers of two. (see &lt;a href=&quot;https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent&quot;&gt;https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="28129" LastEditorUserId="-1" LastEditDate="2017-04-13T12:18:55.920" LastActivityDate="2017-01-20T09:44:42.377" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16420" PostTypeId="2" ParentId="6084" CreationDate="2017-01-20T09:46:06.670" Score="6" Body="&lt;p&gt;Here is a small modification of &lt;a href=&quot;https://datascience.stackexchange.com/a/6593&quot;&gt;Kyler Brown's solution for Python&lt;/a&gt; that also allows negative values on the polar axes (&lt;a href=&quot;https://github.com/matplotlib/matplotlib/issues/2203&quot; rel=&quot;noreferrer&quot;&gt;which are currently not officially supported by matplotlib&lt;/a&gt;), basically by simply removing the check for negative values from &lt;code&gt;set_rgrids&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/pLeoo.png&quot; alt=&quot;plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns # improves plot aesthetics&#xA;&#xA;&#xA;def _invert(x, limits):&#xA;    &quot;&quot;&quot;inverts a value x on a scale from&#xA;    limits[0] to limits[1]&quot;&quot;&quot;&#xA;    return limits[1] - (x - limits[0])&#xA;&#xA;def _scale_data(data, ranges):&#xA;    &quot;&quot;&quot;scales data[1:] to ranges[0],&#xA;    inverts if the scale is reversed&quot;&quot;&quot;&#xA;    # for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;    for d, (y1, y2) in zip(data, ranges):&#xA;        assert (y1 &amp;lt;= d &amp;lt;= y2) or (y2 &amp;lt;= d &amp;lt;= y1)&#xA;&#xA;    x1, x2 = ranges[0]&#xA;    d = data[0]&#xA;&#xA;    if x1 &amp;gt; x2:&#xA;        d = _invert(d, (x1, x2))&#xA;        x1, x2 = x2, x1&#xA;&#xA;    sdata = [d]&#xA;&#xA;    for d, (y1, y2) in zip(data[1:], ranges[1:]):&#xA;        if y1 &amp;gt; y2:&#xA;            d = _invert(d, (y1, y2))&#xA;            y1, y2 = y2, y1&#xA;&#xA;        sdata.append((d-y1) / (y2-y1) * (x2 - x1) + x1)&#xA;&#xA;    return sdata&#xA;&#xA;def set_rgrids(self, radii, labels=None, angle=None, fmt=None,&#xA;               **kwargs):&#xA;    &quot;&quot;&quot;&#xA;    Set the radial locations and labels of the *r* grids.&#xA;    The labels will appear at radial distances *radii* at the&#xA;    given *angle* in degrees.&#xA;    *labels*, if not None, is a ``len(radii)`` list of strings of the&#xA;    labels to use at each radius.&#xA;    If *labels* is None, the built-in formatter will be used.&#xA;    Return value is a list of tuples (*line*, *label*), where&#xA;    *line* is :class:`~matplotlib.lines.Line2D` instances and the&#xA;    *label* is :class:`~matplotlib.text.Text` instances.&#xA;    kwargs are optional text properties for the labels:&#xA;    %(Text)s&#xA;    ACCEPTS: sequence of floats&#xA;    &quot;&quot;&quot;&#xA;    # Make sure we take into account unitized data&#xA;    radii = self.convert_xunits(radii)&#xA;    radii = np.asarray(radii)&#xA;    rmin = radii.min()&#xA;    # if rmin &amp;lt;= 0:&#xA;    #     raise ValueError('radial grids must be strictly positive')&#xA;&#xA;    self.set_yticks(radii)&#xA;    if labels is not None:&#xA;        self.set_yticklabels(labels)&#xA;    elif fmt is not None:&#xA;        self.yaxis.set_major_formatter(FormatStrFormatter(fmt))&#xA;    if angle is None:&#xA;        angle = self.get_rlabel_position()&#xA;    self.set_rlabel_position(angle)&#xA;    for t in self.yaxis.get_ticklabels():&#xA;        t.update(kwargs)&#xA;    return self.yaxis.get_gridlines(), self.yaxis.get_ticklabels()&#xA;&#xA;class ComplexRadar():&#xA;    def __init__(self, fig, variables, ranges,&#xA;                 n_ordinate_levels=6):&#xA;        angles = np.arange(0, 360, 360./len(variables))&#xA;&#xA;        axes = [fig.add_axes([0.1,0.1,0.9,0.9],polar=True,&#xA;                label = &quot;axes{}&quot;.format(i)) &#xA;                for i in range(len(variables))]&#xA;        l, text = axes[0].set_thetagrids(angles, &#xA;                                         labels=variables)&#xA;        [txt.set_rotation(angle-90) for txt, angle &#xA;             in zip(text, angles)]&#xA;        for ax in axes[1:]:&#xA;            ax.patch.set_visible(False)&#xA;            ax.grid(&quot;off&quot;)&#xA;            ax.xaxis.set_visible(False)&#xA;        for i, ax in enumerate(axes):&#xA;            grid = np.linspace(*ranges[i], &#xA;                               num=n_ordinate_levels)&#xA;            gridlabel = [&quot;{}&quot;.format(round(x,2)) &#xA;                         for x in grid]&#xA;            if ranges[i][0] &amp;gt; ranges[i][1]:&#xA;                grid = grid[::-1] # hack to invert grid&#xA;                          # gridlabels aren't reversed&#xA;            gridlabel[0] = &quot;&quot; # clean up origin&#xA;            # ax.set_rgrids(grid, labels=gridlabel, angle=angles[i])&#xA;            set_rgrids(ax, grid, labels=gridlabel, angle=angles[i])&#xA;            #ax.spines[&quot;polar&quot;].set_visible(False)&#xA;            ax.set_ylim(*ranges[i])&#xA;        # variables for plotting&#xA;        self.angle = np.deg2rad(np.r_[angles, angles[0]])&#xA;        self.ranges = ranges&#xA;        self.ax = axes[0]&#xA;    def plot(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;    def fill(self, data, *args, **kw):&#xA;        sdata = _scale_data(data, self.ranges)&#xA;        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)&#xA;&#xA;# example data&#xA;variables = (&quot;Normal Scale&quot;, &quot;Inverted Scale&quot;, &quot;Inverted 2&quot;, &#xA;            &quot;Normal Scale 2&quot;, &quot;Normal 3&quot;, &quot;Normal 4 %&quot;, &quot;Inverted 3 %&quot;)&#xA;data = (-1.76, 1.1, 1.2, &#xA;        4.4, 3.4, 86.8, 20)&#xA;ranges = [(-5, 3), (1.5, 0.3), (1.3, 0.5),&#xA;         (1.7, 4.5), (1.5, 3.7), (70, 87), (100, -50)]            &#xA;# plotting&#xA;fig1 = plt.figure(figsize=(6, 6))&#xA;radar = ComplexRadar(fig1, variables, ranges)&#xA;radar.plot(data)&#xA;radar.fill(data, alpha=0.2)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28127" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-01-20T09:46:06.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16421" PostTypeId="2" ParentId="16411" CreationDate="2017-01-20T09:52:20.163" Score="4" Body="&lt;p&gt;The term you are looking for is &quot;Example&quot;.&#xA;Source: Martin Zinkevich, Research Scientist at Google (&lt;a href=&quot;http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&quot;&gt;http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Instance&lt;/strong&gt;: The thing about which you want to make a prediction. For example, the instance might be a web page that you want to classify as either &quot;about cats&quot; or &quot;not about cats&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Label&lt;/strong&gt;: An answer for a prediction task ­­ either the answer produced by a machine learning system, or the right answer supplied in training data. For example, the label for a web page might be &quot;about cats&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature&lt;/strong&gt;: A property of an instance used in a prediction task. For example, a web page might have a feature &quot;contains the word 'cat'&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;: An instance (with its features) and a label.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-01-20T09:52:20.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16422" PostTypeId="1" AcceptedAnswerId="16427" CreationDate="2017-01-20T10:45:27.660" Score="12" ViewCount="5800" Body="&lt;p&gt;I am a bit confused by the difference between the terms &quot;Machine Learning&quot; and &quot;Deep Learning&quot;. I have Googled it and read many articles, but it is still not very clear to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A known definition of Machine Learning by Tom Mitchell is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A computer program is said to learn from experience &lt;strong&gt;E&lt;/strong&gt; with respect to&#xA;  some class of tasks &lt;strong&gt;T&lt;/strong&gt; and performance measure &lt;strong&gt;P&lt;/strong&gt;, if its performance at&#xA;  tasks in &lt;strong&gt;T&lt;/strong&gt;, as measured by &lt;strong&gt;P&lt;/strong&gt;, improves with experience &lt;strong&gt;E&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I take an image classification problem of classifying dogs and cats as my taks &lt;strong&gt;T&lt;/strong&gt;, from this definition I understand that if I would give a ML algorithm a bunch of images of dogs and cats (experience &lt;strong&gt;E&lt;/strong&gt;), the ML algorithm could learn how to distinguish a new image as being either a dog or cat (provided the performance measure &lt;strong&gt;P&lt;/strong&gt; is well defined).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then comes Deep Learning. I understand that Deep Learning is part of Machine Learning, and that the above definition holds. The performance at task &lt;strong&gt;T&lt;/strong&gt; improves with experience &lt;strong&gt;E&lt;/strong&gt;. All fine till now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/&quot; rel=&quot;noreferrer&quot;&gt;This blog &lt;/a&gt; states that there is a difference between Machine Learning and Deep Learning. The difference according to Adil is that in (Traditional) Machine Learning the features have to be hand-crafted, whereas in Deep Learning the features are learned. The following figures clarify his statement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S1hsn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S1hsn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am confused by the fact that in (Traditional) Machine Learning the features have to be hand-crafted. From the above definition by Tom Mitchell, I would think that these features would be learned from experience &lt;strong&gt;E&lt;/strong&gt; and performance &lt;strong&gt;P&lt;/strong&gt;. What could otherwise be learned in Machine Learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Deep Learning I understand that from experience you learn the features and how they relate to each other to improve the performance. Could I conclude that in Machine Learning features have to be hand-crafted and what is learned is the combination of features? Or am I missing something else?&lt;/p&gt;&#xA;" OwnerUserId="25983" LastEditorUserId="25983" LastEditDate="2017-01-23T14:34:36.923" LastActivityDate="2023-06-12T08:04:48.107" Title="Machine Learning vs Deep Learning" Tags="|machine-learning|deep-learning|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16423" PostTypeId="2" ParentId="16422" CreationDate="2017-01-20T11:15:46.677" Score="-2" Body="&lt;p&gt;Okay, think of it like this. In machine learning algirithms, such as linear regression or random forest you give the algorithms a set of features and the target and then it tries to minimize the cost function, so no it doesnt learn any new features, it just learns the weights. Now when you come to deep learning, you have atleast one, (almost always more) hidden layer with a set number of units, these are the features that are being talked about. So a deep learning algorithm doesnt just learn the sets of weights, in that process it also learns the values for hidden units which are complex high level features of the trivial data that you have given. Hence while practicing vanilla machine learning a lot of expertise lies in your ability to engineer features because the algorithm isnt learning any by itself. I hope I answered your question. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-20T11:15:46.677" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16424" PostTypeId="1" AcceptedAnswerId="16425" CreationDate="2017-01-20T12:24:41.057" Score="5" ViewCount="7562" Body="&lt;p&gt;I am trying to implement a recurrent neural network machine translation system, and I am just learning the things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am creating a word embedding matrix. In order to do that, I should know my vocabulary size, dimension of the embedding space, and context window size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is context window?&lt;/p&gt;&#xA;" OwnerUserId="28138" LastActivityDate="2017-01-20T12:36:00.163" Title="What is context window size?" Tags="|rnn|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16425" PostTypeId="2" ParentId="16424" CreationDate="2017-01-20T12:36:00.163" Score="6" Body="&lt;p&gt;A context window applies to the number of words you will use to determine the context of each word. Like of your statement is &quot;the quick brown fox&quot;  a context window of two would mean your samples are like (the,quick) and ( the, brown). Then you slide one word and your samples become  (quick, the ), (quick, brown) and (quick fox) and so on. I would suggest reading up this &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&amp;amp;ved=0ahUKEwieoaeF3tDRAhVKQo8KHRf-CHkQFggyMAI&amp;amp;usg=AFQjCNFRBSxWBWA8Qw1C5rn0pvyHEwjNqw&amp;amp;sig2=FuVfcDwAJBnmD_UiUHwGmg&quot; rel=&quot;noreferrer&quot;&gt;word 2vec tutorial&lt;/a&gt; to understand the training method and terminology. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-20T12:36:00.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16427" PostTypeId="2" ParentId="16422" CreationDate="2017-01-20T13:59:43.840" Score="5" Body="&lt;p&gt;In addition to what Himanshu Rai said, Deep learning is a subfield which involves the use of neural networks.These neural networks try to learn the underlying distribution by modifying the weights between the layers.&#xA;Now, consider the case of image recognition using deep learning:&#xA;a neural network model is divided among layers, these layers are connected by links called weights, as the training process begins, these layers adjust the weights such that each layer tries to detect some feature and help the next layer for its processing.The key point to note is we don't explicitly tell the layer to learn to detect edges, or eyes, nose or faces.The model learns to do that itself.Unlike classical machine learning models.&lt;/p&gt;&#xA;" OwnerUserId="28086" LastActivityDate="2017-01-20T13:59:43.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16428" PostTypeId="1" AcceptedAnswerId="16430" CreationDate="2017-01-20T14:40:32.013" Score="0" ViewCount="751" Body="&lt;p&gt;I'm kinda new to tensorflow and just wanted to know if it already performs vectorization in its operators like multiplying matrices and so on.&#xA;And if not if its doable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example let's say we have 2 matrices and we want to multiply them. Instead of for loops to do the operation I'm looking for a vectorized way to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="20410" LastEditorUserId="20410" LastEditDate="2017-01-20T15:42:56.853" LastActivityDate="2017-01-20T16:24:34.647" Title="Does Tensorflow uses vectorization in its operators" Tags="|tensorflow|" AnswerCount="1" CommentCount="5" ClosedDate="2017-01-20T18:47:16.123" ContentLicense="CC BY-SA 3.0" />
  <row Id="16430" PostTypeId="2" ParentId="16428" CreationDate="2017-01-20T16:24:34.647" Score="0" Body="&lt;p&gt;Yes, tensorflow works with tensors, which are like vectors but more generic, so dont worry about loops. Just make sure the matrices you are multiplying have compatible dimension. Its completely tensor arithmetic, which for your case will be vector algebra. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-20T16:24:34.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16433" PostTypeId="1" AcceptedAnswerId="16460" CreationDate="2017-01-20T18:21:48.480" Score="2" ViewCount="4814" Body="&lt;p&gt;I have a keras model that takes in an image with (up to) 5 MNIST digits and outputs a length and then (up to) 5 digits. I see that &lt;code&gt;model.evaluate()&lt;/code&gt; reports accuracies for each of the outputs but how do I determine how good the model is at predicting the numbers? Do I need to write that myself?&lt;/p&gt;&#xA;" OwnerUserId="28152" LastEditorUserId="378" LastEditDate="2017-10-19T04:18:12.580" LastActivityDate="2017-10-19T04:18:12.580" Title="How to calculate accuracy on keras model with multiple outputs?" Tags="|keras|accuracy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16434" PostTypeId="1" CreationDate="2017-01-21T01:35:45.910" Score="1" ViewCount="2084" Body="&lt;p&gt;When running the object detection tutorial, you can use train.py which is supplied. On the console, the following is printed : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INFO:tensorflow:global step 1595: loss = 2.5473 (46.045 sec/step)&#xA;INFO:tensorflow:global step 1596: loss = 2.9730 (37.213 sec/step)&#xA;INFO:tensorflow:global step 1597: loss = 2.4274 (26.814 sec/step)&#xA;INFO:tensorflow:Recording summary at step 1597.&#xA;INFO:tensorflow:global step 1598: loss = 2.2615 (38.410 sec/step)&#xA;INFO:tensorflow:global step 1599: loss = 2.5232 (26.713 sec/step)&#xA;INFO:tensorflow:global step 1600: loss = 2.3982 (28.633 sec/step)&#xA;INFO:tensorflow:global step 1601: loss = 2.4807 (29.261 sec/step)&#xA;INFO:tensorflow:Recording summary at step 1601.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This information is also displayed on Tensorboard while monitoring the performance of training : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vAbak.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vAbak.png&quot; alt=&quot;steps in tensorboard&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first glance, I believed that one step is one epoch, but apparently this isn't true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What exactly is a &lt;em&gt;step&lt;/em&gt; in Tensorflow / Tensorboard?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example came from &lt;a href=&quot;https://medium.com/@aescart1/la-computer-vision-d%C3%A9tection-dobjet-avec-tensorflow-5ee819d74f61&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="45729" LastEditDate="2018-02-05T01:11:49.950" LastActivityDate="2018-02-05T01:11:49.950" Title="What exactly is a step in Tensorflow prebuilt architectures?" Tags="|deep-learning|tensorflow|" AnswerCount="0" CommentCount="6" ClosedDate="2017-10-23T07:59:19.523" ContentLicense="CC BY-SA 3.0" />
  <row Id="16435" PostTypeId="2" ParentId="997" CreationDate="2017-01-21T02:28:02.380" Score="0" Body="&lt;p&gt;You can get yellow and green taxi trip records from NYC taxi dataset. The website collects data from 2009 until now. You can download the data from the following link: &lt;a href=&quot;http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28161" LastActivityDate="2017-01-21T02:28:02.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16436" PostTypeId="2" ParentId="16370" CreationDate="2017-01-21T02:38:10.240" Score="1" Body="&lt;p&gt;I think that in order to understand how the SVM handles the new synthetic&#xA;data, you should look at the loss function SVM uses, i.e. hinge loss and the behavior on an imbalanced dataset. Intuitively this function will try to fit the hyperplane that best separates the data. For example imagine you have a dataset that is not linearly separable and you have to classes A (with a 1000 samples) and B (with 50 samples). So the SVM will &quot;move&quot; the hyperplane to classify right all samples of A even if this means failing to classify correctly some samples of class B.Then when you create new samples of the classes B the classifier will try to find a balance between the error of the both classes. A more detailed explanation can be found in the 6.3 section fo this &lt;a href=&quot;http://www.cs.ox.ac.uk/people/vasile.palade/papers/Class-Imbalance-SVM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general all classifiers tend to classify correctly the most dominating class because of the loss function they use. One way of dealing with it is generating synthetic data, other way is using a cost sensitive classifier, this is the cost of misclassification of the minority class is higher than the majority class. More details can be found in the following links:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://svds.com/learning-imbalanced-classes/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://svds.com/learning-imbalanced-classes/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-01-21T02:38:10.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16441" PostTypeId="1" CreationDate="2017-01-21T15:34:07.977" Score="1" ViewCount="478" Body="&lt;p&gt;Since Orange is able to read datetime variables, i tried to build a simple workflow to predict datetime variables via regression. Loading the data etc. is no problem, but when it comes to predict the data, the output of Predictions Widget (in this case with AdaBoost Regression) is something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1450396800.000;&#xA;1438905600.000;&#xA;1438732800.000;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whereas the original datetime is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2015-12-22;&#xA;2015-08-07;&#xA;2015-08-05;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does that mean? Did i make any mistakes? Is there any other way to predict datetime variables in orange?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The workflow does its job for any &quot;common&quot; continuous variable.&lt;/p&gt;&#xA;" OwnerUserId="28172" LastActivityDate="2017-04-23T15:33:40.203" Title="Predict datetime variable with orange canvas" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16442" PostTypeId="2" ParentId="16349" CreationDate="2017-01-21T22:01:40.147" Score="1" Body="&lt;p&gt;Couple of suggestions:&#xA;Your training set is smaller than your test set? It should be the other way around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also tune the RF hypermeters using the held out set or Cross Validation. The two parameter people tune on RF are ntrees and mtry.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2017-01-21T22:01:40.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16443" PostTypeId="2" ParentId="16179" CreationDate="2017-01-21T23:47:01.680" Score="2" Body="&lt;p&gt;As mentioned by other users, the solution is not very clear. The general approach is to follow what is mentioned &lt;a href=&quot;http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, as suggested by one of the senior R&amp;amp;D employee and my mentor, the method in practice is to calculate average f1-score as the HM of average precision and average recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This surely depends on your usecase as well as how you are calculating the metric (micro/macro).&lt;/p&gt;&#xA;" OwnerUserId="27741" LastEditorUserId="27741" LastEditDate="2018-01-01T12:32:16.917" LastActivityDate="2018-01-01T12:32:16.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16444" PostTypeId="1" CreationDate="2017-01-22T09:43:56.553" Score="9" ViewCount="8834" Body="&lt;p&gt;Keras has a way to extract the features of a pretrained model, described here &lt;a href=&quot;https://keras.io/applications/&quot; rel=&quot;noreferrer&quot;&gt;https://keras.io/applications/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.applications.vgg16 import VGG16&#xA;from keras.preprocessing import image&#xA;from keras.applications.vgg16 import preprocess_input&#xA;import numpy as np&#xA;&#xA;model = VGG16(weights='imagenet', include_top=False)&#xA;&#xA;img_path = 'elephant.jpg'&#xA;img = image.load_img(img_path, target_size=(224, 224))&#xA;x = image.img_to_array(img)&#xA;x = np.expand_dims(x, axis=0)&#xA;x = preprocess_input(x)&#xA;&#xA;features = model.predict(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried this with some sample images. My vector x is of shape (100, 3, 224, 224) for 100 observations, 3 for RGB and 224x224 pixel size.&#xA;the &lt;code&gt;preprocess_input&lt;/code&gt; reshapes this for the VGG model (it expects a different order).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the output shape of &lt;code&gt;features&lt;/code&gt; is (100, 512, 7, 7). What is this shape?&#xA;I want to use the features as input for a logistic regression. So I need a shape like (100, n): one row for each observation and the features in the columns.&#xA;How do I reshape the output to this dimension?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I now want to build my own simple Convnet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Convolution2D, MaxPooling2D&#xA;from keras.layers import Activation, Dropout, Flatten, Dense&#xA;&#xA;model = Sequential()&#xA;model.add(Convolution2D(32, 3, 3, input_shape=(1, 299, 299)))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&#xA;model.add(Convolution2D(32, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&#xA;model.add(Convolution2D(64, 3, 3))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This model expects grayscale images as input, hence the shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of layer do I have to add to get features of this model (something I can input in a logistic regression or random forest).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="15664" LastActivityDate="2017-01-22T10:50:41.303" Title="feature extraction for a pretrained model in keras" Tags="|deep-learning|keras|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16445" PostTypeId="2" ParentId="16444" CreationDate="2017-01-22T10:50:41.303" Score="9" Body="&lt;p&gt;The &lt;code&gt;features&lt;/code&gt; variable contains the outputs of the final convolutional layers of your network. The final convolutional layer of VGG16 outputs 512 7x7 feature maps. All you need to do in order to use these features in a logistic regression model (or any other model) is reshape it to a 2D tensor, as you say. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reshaped_features = features.reshape(100, 512*7*7)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will flatten out the feature maps to a long one-dimensional vector for each instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tip: If you can't be bothered working out the actual dimensions for your reshape, you can replace &lt;code&gt;512*7*7&lt;/code&gt; with &lt;code&gt;-1&lt;/code&gt; and &lt;code&gt;numpy&lt;/code&gt; will figure out how big the final dimension should be.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-01-22T10:50:41.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16446" PostTypeId="1" AcceptedAnswerId="16739" CreationDate="2017-01-22T11:49:11.153" Score="0" ViewCount="1308" Body="&lt;p&gt;I would like to train convolutional neural networks using a gtx580 gpu. I tried setting up TensorFlow but it did not work (wrong cuda compute compability).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which deep learning framework can best support a gtx580 gpu in terms of functionality and speed?&lt;/p&gt;&#xA;" OwnerUserId="28182" LastActivityDate="2017-02-26T19:46:29.973" Title="Which deep learning framework have support for gtx580 GPU?" Tags="|machine-learning|deep-learning|gpu|" AnswerCount="2" CommentCount="0" ClosedDate="2017-02-04T13:55:43.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="16448" PostTypeId="1" CreationDate="2017-01-22T12:08:30.773" Score="3" ViewCount="493" Body="&lt;p&gt;I work at a University and have a project to be able to score applicants based on their likelihood to enroll (convert), using their answers to application form questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The applications contain name, DOB, date &amp;amp; time of application, country, gender, course selected, English proficiency, funding availability and some other similar fields. There are also various free-text fields, but I think these will complicate things too much to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought is to use a regression model to do this, using R. But I am a complete noob - I studied regression at uni 10 years ago.... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have had a search around and I think once I know I'm on the right path I will be able to figure the process out but I am unsure where to begin, and do not want to start by going down the wrong avenue. My main concerns are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is a regression model the correct approach? If not what is?&lt;/li&gt;&#xA;&lt;li&gt;Are categorical fields a problem - as opposed to continuous fields?&lt;/li&gt;&#xA;&lt;li&gt;There is additional information which is only available for some applicants - can this be included, or do we need to use the same information for all applicants?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27683" LastEditorUserId="29575" LastEditDate="2017-12-14T12:37:06.897" LastActivityDate="2017-12-14T12:37:06.897" Title="Data Science Noob - Customer Scoring based on conversion probability" Tags="|r|predictive-modeling|regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16449" PostTypeId="2" ParentId="16448" CreationDate="2017-01-22T12:35:45.780" Score="2" Body="&lt;p&gt;Your question could be closed for too broad, but let's give a try. You want the enrolment probability, this sounds like a logistic regression for me. Neither categorical nor continuous data type should present a problem. You may model the additional information, applications who don't have one simply be assigned a &lt;code&gt;NA&lt;/code&gt; category. You can include the &lt;code&gt;NA&lt;/code&gt; category in your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend you read the book &lt;code&gt;Applied Predictive Modelling&lt;/code&gt;, I think it has a section on credit-card applications, which is close to what you are doing. You should try to learn from it.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-01-22T12:35:45.780" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16450" PostTypeId="2" ParentId="16179" CreationDate="2017-01-22T13:11:32.377" Score="9" Body="&lt;p&gt;This paper &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1882479&quot; rel=&quot;noreferrer&quot;&gt;Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement&lt;/a&gt; by Forman and Scholz discuss the different methods for computing the average F-score in cross validation. The paper shows that under very high class imbalance some of the computation methods (average of individual folds F-scores or F-score based on average of individual folds precision and recall) can lead to biased results. The paper recommends computing the F-score by adding the TP, FP, FN from each fold, computing the precision and recall and finally the F-score. &lt;/p&gt;&#xA;" OwnerUserId="8792" LastActivityDate="2017-01-22T13:11:32.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16451" PostTypeId="2" ParentId="16448" CreationDate="2017-01-22T14:00:21.293" Score="0" Body="&lt;p&gt;Let's see (broad strokes ahead)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two ways of doing it supervised learning and unsupervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For supervised learning we either need previous years data (with the final enrollments) or you need to calculate the probability of some of the applications(Enough so that the model can predict accurately for new data). Then it becomes a problem of linear regression(as probability will be a continuous variable). If you decide a certain cutoff(threshold) that probability(y =0 below and y =1 above ) then it becomes a problem of logistic regression.&#xA;Once the data is ready machine learning can be done in half an hour. &#xA;Now to answer your questions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Yes, Regression model is a correct approach but it's a broad term.&lt;/li&gt;&#xA;&lt;li&gt;Not a problem, you need to make the categorical fields factor using &lt;code&gt;as.factor()&lt;/code&gt; in R.&lt;/li&gt;&#xA;&lt;li&gt;you should use same information for all the applicants but that doesn't mean you can't use that info. If the fact that information is available has a bearing on the outcome then make a variable and put it 1 or 0 accordingly. If what's available in that information has a bearing then it gets a little tricky, may be count the number of common relevant words and put that number in a variable.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27989" LastActivityDate="2017-01-22T14:00:21.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16452" PostTypeId="2" ParentId="14181" CreationDate="2017-01-22T14:11:29.493" Score="2" Body="&lt;p&gt;Just some quick fixes.&#xA;Normalization takes care of scale of data across the columns. e.g.&#xA;If one data column is in the ranges of 1000s and other in 10s then normalization will work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you want to take care of the skew-ness one way is to simply take &lt;code&gt;log&lt;/code&gt; of the data. It pulls in the outliers. as log increases very slowly for positive values. Do it before the normalization (Z score in your case)&#xA;&lt;a href=&quot;https://i.stack.imgur.com/h67LV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h67LV.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other ways are taking nth root of the data.&lt;/p&gt;&#xA;" OwnerUserId="27989" LastEditorUserId="27989" LastEditDate="2017-01-22T14:22:37.173" LastActivityDate="2017-01-22T14:22:37.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16454" PostTypeId="1" CreationDate="2017-01-22T17:50:57.620" Score="0" ViewCount="291" Body="&lt;p&gt;there is the PMML standard defined as XML Schema for Data Mining Model portability between different DM software. My question is, does this xml file serve as a sort of &quot;config file&quot; with the specifics of the underlying domain-model relation (with the specific model implementation logic for the data mining problem concerned), for the other software that you want to import your model? &#xA;Or is the PMML model the implementation itself, I mean is there any logic implemented to execute the model with the PMML xml file itself (e.g if you add some test data and let it run through your xml) or must there exist an implementation of the learning algorithm in the software and the PMML is just the configuration of the specific model?&lt;/p&gt;&#xA;" OwnerUserId="26761" LastActivityDate="2017-03-22T02:44:32.673" Title="PMML XML Model == same as config file or implementation?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16455" PostTypeId="1" CreationDate="2017-01-22T18:05:44.503" Score="0" ViewCount="3502" Body="&lt;p&gt;I am new to deep learning and tensor flow and I am trying to train a CNN at localizing digits in the Street View House Numbers data set. To this end I have an input set of 32x32 images and, since I want to recognize up to 5 digits, I am using as labels vectors of 20 elements like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[top_x_digit1,top_y_digit1,widht_digit1,height_digit1,top_x_digit2, etc..]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0,0,0,0 when there is no digit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understand, after (let me say) 3 layers of convolution and pooling I can add 5 (parallel) fully connected layers aimed at extracting each the box features of a different digits (when present, 0 0 0 0 otherwise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my approach correct?&lt;/p&gt;&#xA;" OwnerUserId="27347" LastEditorUserId="29169" LastEditDate="2021-02-16T15:24:02.933" LastActivityDate="2021-02-16T15:24:02.933" Title="Objects Localization Through CNN" Tags="|deep-learning|tensorflow|convolutional-neural-network|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16456" PostTypeId="2" ParentId="16446" CreationDate="2017-01-22T21:11:06.213" Score="1" Body="&lt;p&gt;I'm using Tensorflow with GeForce GT640 on Windows 7 x64. Also your GPU, GTX 540 is on the CUDA GPUs &lt;a href=&quot;https://developer.nvidia.com/cuda-gpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;list&lt;/a&gt;. So try to reinstall CUDA. It'll work!   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: cuDNN needs GPU of compute capability 3.0 or higher.&lt;/p&gt;&#xA;" OwnerUserId="17020" LastEditorUserId="17020" LastEditDate="2017-02-26T19:46:29.973" LastActivityDate="2017-02-26T19:46:29.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16457" PostTypeId="2" ParentId="16163" CreationDate="2017-01-22T21:26:12.497" Score="2" Body="&lt;p&gt;You can use grid search or xgb.cv for find the best iteration. Run xgb.cv for example 500 trees and add early stopping criteria. Then you can use the best iteration for xgb.train. Have a look at these links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/9364/hypertuning-xgboost-parameters/9368#9368&quot;&gt;Hypertuning XGBoost parameters&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees&quot;&gt;https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17020" LastActivityDate="2017-01-22T21:26:12.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16458" PostTypeId="1" CreationDate="2017-01-23T02:43:00.160" Score="1" ViewCount="359" Body="&lt;p&gt;For the image dataset I am working with, I need to use B&amp;amp;W version of images (otherwise, I would need to build a network to give false colors to a set of my images, since they have an overpowering color tint). I am attempting to retrain VGG's bottom layers (including the last convolution block) in order to perform this task, but am running into the problem that by rephrasing the space in which the input operates in (and thus also completely changing the loss surface) it is a difficult task to retrain these layers. I've experimented with learning rates from everywhere from 0.01 to 100 (using different decay schedules for each) with no success. Does anyone have any tips with retraining networks for significantly different loss surfaces? Should I ditch my efforts and attempt to train a new network end-to-end (my dataset isn't too large, so this is quite prone to overfitting if I am to use deep learning)? Are there networks built for B&amp;amp;W images that I could instead retrain? &lt;/p&gt;&#xA;" OwnerUserId="28209" LastEditorUserId="29169" LastEditDate="2021-02-16T15:23:50.143" LastActivityDate="2021-02-16T15:23:50.143" Title="Tips for retraining convolutional neural networks given a drastically different loss surface" Tags="|deep-learning|training|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16460" PostTypeId="2" ParentId="16433" CreationDate="2017-01-23T03:46:06.507" Score="0" Body="&lt;p&gt;It's going to take a bit of engineering - since you have a variable size output, you need to encode the length into the output in order to evaluate the accuracy of the model overall. If instead of outputting &quot;up to 5 digits&quot;, you output an array of 5 predictions, where some non-digit (such as -1) operates as indicating that there is no digit present, you can better evaluate your network. If you retrain your network as such (where $X$ is the array of images and $Y$ is an array containing arrays of form $[1,4,3,-1,-1]$, for example), then model.evaluate($X_{test}$,$Y_{test}$) will work as expected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to re-train your network, you can write a simple function to take the output from model.predict($X_{test}$) and encode it into the corresponding format. This encode function will simply go from $[1,4,3]$ to $[1,4,3,-1,-1]$. You can then calculate the  accuracy by sklearn.metrics.accuracy_score($encode$(model.predict($X_{test}$)),$Y_{test}$), where $encode$ is the aforementioned function. &lt;/p&gt;&#xA;" OwnerUserId="28209" LastActivityDate="2017-01-23T03:46:06.507" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16462" PostTypeId="2" ParentId="16280" CreationDate="2017-01-23T04:13:21.380" Score="3" Body="&lt;p&gt;I &lt;em&gt;believe&lt;/em&gt; you can do this using the functional API and playing around with your input. Suppose you have the sets of input features (can be singleton sets) $a$,$b$,and $c$ for some $x$ in $X$. If you split your input into $X_a$, $X_b$, and $X_c$, where all elements in $x$ is transformed to include only features in $a$, you can theoretically build this model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a code snippet demonstrating this idea, given $X_a$, $X_b$, and $X_c$, and their corresponding shapes. You could widen this model by expanding the fully-connected layers (by increasing the variable set to 1 in the Dense calls). No promises on the code working off the bat: I've brushed over some details but this should give you a start on your problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_A = Input(shape=shape_A)&#xA;input_B = Input(shape=shape_B)&#xA;input_C = Input(shape=shape_C)&#xA;A = Dense(1)(input_A)&#xA;A = Dense(1)(A) &#xA;B = Merge([input_A, input_B, input_C], mode='concat')&#xA;B = Dense(1)(B) &#xA;C = Merge([A,B],mode='concat)&#xA;C = Dense(1)(C)&#xA;B = Merge([A,B],mode='concat)&#xA;B = Dense(1)(B)&#xA;A = Dense(1)(A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From here, if you want one output, you can call merge on $A$, $B$, and $C$ and then tack on a Dense (Fully-Connected) layer with, for example, a Softmax layer for multi-label classification.&lt;/p&gt;&#xA;" OwnerUserId="28209" LastActivityDate="2017-01-23T04:13:21.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16463" PostTypeId="1" AcceptedAnswerId="16466" CreationDate="2017-01-23T08:07:38.737" Score="35" ViewCount="21830" Body="&lt;p&gt;I am pretty new to neural networks, but I understand linear algebra and the mathematics of convolution pretty decently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to understand the example code I find in various places on the net for training a Keras convolutional NN with MNIST data to recognize digits. My expectation would be that when I create a convolutional layer, I would have to specify a filter or set of filters to apply to the input. But the three samples I have found all create a convolutional layer like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Convolution2D(nb_filter = 32, nb_row = 3, nb_col = 3,&#xA;                        border_mode='valid',&#xA;                        input_shape=input_shape))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seems to be applying a total of 32 3x3 filters to the images processed by the CNN. But what are those filters? How would I describe them mathematically? The keras documentation is no help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&lt;/p&gt;&#xA;" OwnerUserId="28213" LastEditorUserId="29169" LastEditDate="2021-02-16T15:23:41.117" LastActivityDate="2021-02-16T15:23:41.117" Title="What is/are the default filters used by Keras Convolution2d()?" Tags="|keras|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16464" PostTypeId="1" AcceptedAnswerId="16576" CreationDate="2017-01-22T08:03:18.690" Score="3" ViewCount="1695" Body="&lt;p&gt;I'm working with a high-dimensional dataset, and have found that my attempts at dimensionality-reduction hurt the accuracy of downstream classifiers, suggesting that I am effectively losing information in the course of feature-compression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, when I sort the features by importance (per estimation via random forest) and then train neural networks on different numbers of the most important features, I find that the resulting classifier accuracy stops increasing before I have exhausted all of the useful-seeming features.  I suspect that simultaneous training on many thousands of features is the problem, and I'd like to simplify the problem by learning from the features incrementally, that is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to explore training a neural network adding a few features at a time: e.g. train using the top 100 features, then add in the next 100, train again (using coefs from previous round plus random coefs for the new part of the input layer), rinse, repeat.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE that this &quot;incrementalness&quot; is across features, not samples; it doesn't affect whether training is incremental (&quot;online&quot;) with respect to samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I gather this sort of thing has been tried before with some success but I haven't been able to find implementations in the Python ML libraries I've checked, nor can I find good explanations of the details of successful algorithms.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about 70K examples, divided over 6 balanced classes. For most of my work I care only about distinctions between a total of 4 classes, so I merge two pairs of the initial classes and end up with a total of 4 (I end up with unbalanced training sets as a result, but since the two larger classes are the ones whose classification is most important, I'm fine with that for now). I have a total of about 28K features, but am finding that I get best classifier performance when using no more than the top 4K features (pre-sorted by importance via Random Forest).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does my approach seem reasonable?&lt;/li&gt;&#xA;&lt;li&gt;Are there accepted published detailed algorithms (e.g., is it reasonable to add the features as extensions of the original input layer, or is another approach preferred)?&lt;/li&gt;&#xA;&lt;li&gt;Do any Python libs, particularly any compatible with scikit-learn, implement such an algorithm?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28208" OwnerDisplayName="J. Lerman" LastEditorUserId="836" LastEditDate="2017-01-23T20:45:37.150" LastActivityDate="2017-02-26T11:43:45.247" Title="Are there established good algorithms for incremental feature learning for a neural network? Do any python ML libraries implement such algorithms?" Tags="|python|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16465" PostTypeId="2" ParentId="16463" CreationDate="2017-01-23T11:07:14.063" Score="9" Body="&lt;p&gt;They are convolution kernels. For instance your image $A$ is 5x5, you have 32 3x3 convolution kernels $F_k$. The border_mode is 'valid' that means there is no padding around input, so the pixel (i,0),(0,j),(i,4),(4,j) are lost. Thus your results are 32 3x3 images $B_k(i,j)$, (i=1,2,3,j=1,2,3), each result image is defined by the convolution:&#xA;$$&#xA;B_k(i,j) = (F_k * A) (i,j) = \sum_{l=0,1,2}\sum_{m=0,1,2}F_k(l,m)A(i-l,j-m)&#xA;$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XvatD.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XvatD.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The traned model will train the kernels according to your cost function, and in the end these kernels are the filters of your model. &lt;/p&gt;&#xA;" OwnerUserId="19046" LastEditorUserId="19046" LastEditDate="2017-06-21T07:50:37.480" LastActivityDate="2017-06-21T07:50:37.480" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16466" PostTypeId="2" ParentId="16463" CreationDate="2017-01-23T12:08:08.623" Score="28" Body="&lt;p&gt;By default, the filters &lt;span class=&quot;math-container&quot;&gt;$W$&lt;/span&gt; are initialised randomly using the &lt;code&gt;glorot_uniform&lt;/code&gt; method, which draws values from a uniform distribution with positive and negative bounds described as so: &#xA;&lt;span class=&quot;math-container&quot;&gt;$$W \sim \mathcal{U}\left(\frac{6}{n_{in} + n_{out}}, \frac{-6}{n_{in} + n_{out}}\right),$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$n_{in}$&lt;/span&gt; is the number of units that feed into this unit, and &lt;span class=&quot;math-container&quot;&gt;$n_{out}$&lt;/span&gt; is the number of units this result is fed to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are using the network to make a prediction, these filters are applied at each layer of the network. That is, a discrete convolution is performed for each filter on each input image, and the results of these convolutions are fed to the next layer of convolutions (or fully connected layer, or whatever else you might have).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During training, the values in the filters are optimised with backpropogation with respect to a loss function. For classification tasks such as recognising digits, usually the cross entropy loss is used. &#xA;Here's a visualisation of some filters learned in the first layer (top) and the filters learned in the second layer (bottom) of a convolutional network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D6pXk.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D6pXk.jpg&quot; alt=&quot;conv net filters visualisation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the first layer filters basically all act as simple edge detectors, while the second layer filters are more complex. As you go deeper into a network, the filters are able to detect more complex shapes. It gets a little tricky to visualise though, as these filters act on images that have been convolved many times already, and probably don't look much like the original natural image.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-11-30T01:04:57.287" LastActivityDate="2018-11-30T01:04:57.287" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="16467" PostTypeId="2" ParentId="13490" CreationDate="2017-01-23T12:22:33.680" Score="39" Body="&lt;p&gt;I use this kind of rule for &lt;code&gt;class_weight&lt;/code&gt; :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import math&#xA;&#xA;# labels_dict : {ind_label: count_label}&#xA;# mu : parameter to tune &#xA;&#xA;def create_class_weight(labels_dict,mu=0.15):&#xA;    total = np.sum(list(labels_dict.values()))&#xA;    keys = labels_dict.keys()&#xA;    class_weight = dict()&#xA;    &#xA;    for key in keys:&#xA;        score = math.log(mu*total/float(labels_dict[key]))&#xA;        class_weight[key] = score if score &amp;gt; 1.0 else 1.0&#xA;    &#xA;    return class_weight&#xA;&#xA;# random labels_dict&#xA;labels_dict = {0: 2813, 1: 78, 2: 2814, 3: 78, 4: 7914, 5: 248, 6: 7914, 7: 248}&#xA;&#xA;create_class_weight(labels_dict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;code&gt;math.log&lt;/code&gt; smooths the weights for very imbalanced classes !&#xA;This returns :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;{0: 1.0,&#xA; 1: 3.749820767859636,&#xA; 2: 1.0,&#xA; 3: 3.749820767859636,&#xA; 4: 1.0,&#xA; 5: 2.5931008483842453,&#xA; 6: 1.0,&#xA; 7: 2.5931008483842453}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28222" LastEditorUserId="56913" LastEditDate="2021-02-23T11:58:51.177" LastActivityDate="2021-02-23T11:58:51.177" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="16468" PostTypeId="2" ParentId="16422" CreationDate="2017-01-23T13:05:29.930" Score="9" Body="&lt;p&gt;As a research area, Deep Learning is really just a sub-field of Machine Learning as Machine Learning is a sub-field of Artificial Intelligence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) Unsupervised Feature Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conceptually, the first main difference between &quot;&lt;em&gt;traditional&lt;/em&gt;&quot; (or &quot;&lt;em&gt;shallow&lt;/em&gt;&quot;) Machine Learning and Deep Learning is Unsupervised Feature Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you already know, successfully training a &quot;&lt;em&gt;traditional&lt;/em&gt;&quot; Machine Learning model &lt;em&gt;(ex: SVM, XGBoost...)&lt;/em&gt; is only possible after suitable pre-processing and judicious feature extraction to select meaningful information from the data.&#xA;That is, good feature vectors contain features distinctive between data points with different labels and consistent among data points with the same label.&#xA;Feature Engineering is thus the process of manual feature selection from experts. This is a very important but tedious taks to perform! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unsupervised Feature Learning is a process where the model itself selects features automatically through training.&#xA;The topology of a Neural Network organized in layers connected to each other have the nice property of mapping a low-level representation of the data to a higher-level representation. Through training, the network can thus &quot;&lt;em&gt;decide&lt;/em&gt;&quot; what part of the data matters and what part of the data doesn't. This is particularly interesting in Computer Vision or Natural Language Processing where it is quite hard to manually select or engineer robust features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PTUE8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PTUE8.png&quot; alt=&quot;Unsupervised Feature Learning, credits: Tony Beltramelli&quot;&gt;&lt;/a&gt;&#xA;&lt;em&gt;(picture credits: Tony Beltramelli)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, let's assume we want to classify cat pictures. Using a Deep Neural Net, we can feed in the raw pixel values that will be mapped to a set of weights by the first layer, then these weights will be mapped to other weights by the second layer, until the last layer allows some weights to be mapped to numbers representing your problem. (ex: in this case the probability of the picture containing a cat)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though Deep Neural Networks can perform Unsupervised Feature Learning, it doesn't prevent you from doing Feature Engineering yourself to better represent your problem. &lt;strong&gt;Unsupervised Feature Learning, Feature Extraction, and Feature Engineering are not mutually exclusive!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning.stanford.edu/tutorial/&quot; rel=&quot;noreferrer&quot;&gt;http://deeplearning.stanford.edu/tutorial/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1404.7828&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1404.7828&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1512.05616&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1512.05616&lt;/a&gt; (Chapter 2, Section 2)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) Linear Separability&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep Neural Networks can solve some non-linearly separable problems by bending the feature space such that features become linearly separable.&#xA;Once again, this is possible thanks to the network topology organized in layers mapping inputs to new representations of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CsdXg.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CsdXg.png&quot; alt=&quot;The hidden layer learns a representation so that the data is linearly separable, credits: Christopher Olah&quot;&gt;&lt;/a&gt;&#xA;&lt;em&gt;(picture credits: Christopher Olah)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources:&#xA;&lt;a href=&quot;http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/&quot; rel=&quot;noreferrer&quot;&gt;http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3) Statistical Invariance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, Deep Neural Networks are surpassing traditional Machine Learning algorithms in some domains because some architectures are showcasing Statistical Invariance (ex: Spacial Statistical Invariance with Convolutional Neural Networks and Temporal Statistical Invariance with Recurrent Neural Networks)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this Udacity video for more details:&#xA;&lt;a href=&quot;https://www.youtube.com/watch?v=5PH2Vot-tD4&quot; rel=&quot;noreferrer&quot;&gt;https://www.youtube.com/watch?v=5PH2Vot-tD4&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-01-23T13:05:29.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16469" PostTypeId="2" ParentId="16441" CreationDate="2017-01-23T14:17:59.053" Score="2" Body="&lt;p&gt;It appears those numbers are &lt;a href=&quot;https://en.wikipedia.org/wiki/Unix_time&quot; rel=&quot;nofollow noreferrer&quot;&gt;Unix timestamps&lt;/a&gt;. The numbers you quote correspond to the following human-readable dates in &lt;a href=&quot;https://en.wikipedia.org/wiki/ISO_8601&quot; rel=&quot;nofollow noreferrer&quot;&gt;ISO format&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch date  Human readable date (GMT) &#xA;1450396800  2015-12-18 00:00:00&#xA;1438905600  2015-08-07 00:00:00&#xA;1438732800  2015-08-05 00:00:00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can use an &lt;a href=&quot;http://www.epochconverter.com/batch&quot; rel=&quot;nofollow noreferrer&quot;&gt;online epoch converter&lt;/a&gt; to confirm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently, &lt;em&gt;Predictions&lt;/em&gt; widget treats datetime variables as continuous variables (which is ok) but fails to present them nicely. You should probably &lt;a href=&quot;https://github.com/biolab/orange3/issues&quot; rel=&quot;nofollow noreferrer&quot;&gt;submit a bug report&lt;/a&gt; about it.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-01-23T14:17:59.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16470" PostTypeId="2" ParentId="16357" CreationDate="2017-01-23T15:41:55.397" Score="2" Body="&lt;p&gt;In order to draw conclusions from this one building's data on other buildings on the campus, you would need a &lt;em&gt;set of parameters shared&lt;/em&gt; between the building you know about and the buildings you want to predict for. For example, if the monitored building is of a male dormitory &lt;strong&gt;type&lt;/strong&gt;, perhaps you can infer something for male dormitories; if the building is at &lt;strong&gt;location&lt;/strong&gt; X, perhaps you can infer something for buildings at or close to location X. You'd need &lt;em&gt;as many as possible&lt;/em&gt; of such additional columns to extrapolate knowledge to other buildings.&lt;/p&gt;&#xA;&lt;p&gt;As for visualizing/predicting resource usage of that single building, I suggest you try data mining tool &lt;a href=&quot;https://orangedatamining.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt;, which I find phenomenal for prototyping machine learning workflows. It also has a &lt;a href=&quot;https://orange3-timeseries.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;time series add-on&lt;/a&gt;, which I can't vouch for, but it does seem to have some &lt;a href=&quot;https://orange3-timeseries.readthedocs.io/en/latest/widgets/var.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;forecast capabilities&lt;/a&gt;.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6g0aF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6g0aF.png&quot; alt=&quot;Forecasting using VAR model in Orange&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://orange3-timeseries.readthedocs.io/en/latest/_images/line-chart-ex1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;readthedocs.io&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="25180" LastEditDate="2022-12-24T17:23:23.360" LastActivityDate="2022-12-24T17:23:23.360" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16472" PostTypeId="1" CreationDate="2017-01-23T19:51:34.230" Score="2" ViewCount="643" Body="&lt;p&gt;I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have just a few basic questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If this is inappropriate for this board, apologies in advance. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Edit: Clarifying my question:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x = $ layer and $y =$ node. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} &#xA;\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial E}{\partial Out} = (Out - Target)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting it all together: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O} &#xA;\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = $  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?&lt;/p&gt;&#xA;" OwnerUserId="28231" LastEditorUserId="28231" LastEditDate="2017-01-24T05:23:42.237" LastActivityDate="2017-01-24T10:09:30.670" Title="Basic backpropagation question" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16474" PostTypeId="1" CreationDate="2017-01-23T20:49:00.927" Score="2" ViewCount="117" Body="&lt;p&gt;I'm working on a regression problem with a continuous dependent variable (sale price of a home). Amongst my features are several categorical features, which I'm transforming to &quot;one hot encoded&quot; dummy variables via &lt;code&gt;Pandas&lt;/code&gt; &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;get_dummies&lt;/code&gt;&lt;/a&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, in order to avoid the trap of multi-colinearity, one level from each categorical variable must be excluded from the model.&lt;code&gt;Pandas&lt;/code&gt; provides this with the &lt;code&gt;drop_first&lt;/code&gt; argument within the &lt;code&gt;get_dummies&lt;/code&gt; function in which I can optionally exclude the first level from being included in the one hot encoding. However, this strikes me as somewhat arbitrary. Is there a strategy to ensure that the excluded level doesn't affect the accuracy of my regression model? How does one decide which level to exclude?&lt;/p&gt;&#xA;" OwnerUserId="426" LastActivityDate="2017-01-23T20:49:00.927" Title="What's the strategy for deciding which feature level is excluded from one hot encoding of a categorical variable?" Tags="|predictive-modeling|regression|feature-selection|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16475" PostTypeId="1" AcceptedAnswerId="16498" CreationDate="2017-01-23T21:49:00.643" Score="3" ViewCount="880" Body="&lt;p&gt;I have some data reduced by TSNE into a 2D representation, which shows clear spatial features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/M5rLY.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/M5rLY.jpg&quot; alt=&quot;2D data organized by TSNE&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'd like to format this into a grid – not just snapping data to the nearest grid square but spreading everything out to fill up a grid, preserving (as much as possible) the existing spatial relationships.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I've only found &lt;a href=&quot;https://www.computer.org/csdl/trans/tg/2012/05/ttg2012050797-abs.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; which might close to what I need? This process might already have a name and I'm just one step from an easy Google solution, but at the moment I'm stuck!&lt;/p&gt;&#xA;" OwnerUserId="28234" LastActivityDate="2017-01-24T13:34:46.303" Title="Organize TSNE data into grid" Tags="|graphs|tsne|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16480" PostTypeId="1" CreationDate="2017-01-24T04:15:38.543" Score="5" ViewCount="1071" Body="&lt;p&gt;I have a dataset {$X_i$, $Y_i$}. $Y_i$ - real value targets, they can be negative. The task is to teach classifier $f: X_i \rightarrow \{0, 1\}$ to maximize the sum below&#xA;$$&#xA;L(f) = \sum_i f(X_i) * Y_i&#xA;$$&#xA;So $f(X_i) = I[Y_i &amp;gt; 0]$ would be the ideal classifier. &#xA;I'm looking for appropriate smooth loss function to teach NN. NN output is probability $p(X_i)$, so $f(X_i) = I[p(X_i) &amp;gt; 0.5]$. I have tried the next loss function:&#xA;$$&#xA;BadLoss(p) = \sum_i p(X_i) * Y_i&#xA;$$&#xA;but because $Y_i$ can be negative it is bad-defined loss and NN converges to nothing good. The best results I got were with usual logloss:&#xA;$$&#xA;Loss(p) = \sum_i logloss(p(X_i), \; I[Y_i &amp;gt; 0])&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can be there better loss functions to maximize the sum $L(f)$?&lt;/p&gt;&#xA;" OwnerUserId="28241" LastActivityDate="2017-12-20T23:46:08.740" Title="Loss function to maximize sum of targets" Tags="|neural-network|classification|regression|loss-function|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16482" PostTypeId="2" ParentId="16480" CreationDate="2017-01-24T06:38:10.397" Score="2" Body="&lt;p&gt;Define $Z_i = 1[Y_i&amp;gt;0]$, i.e., $Z_i = 1$ if $Y_i &amp;gt; 0$, else $Z_i = -1$.  Now  the data set $(X_i,Z_i)$ defines a two-class boolean classification problem.  The solution to that boolean classification problem with highest accuracy solves your original problem (minimizes your loss function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you are basically asking how to train a neural network to solve a two-class boolean classification problem.  A reasonable approach is to put a softmax layer at the output of the neural network and train it using the cross-entropy loss, as usual.  During test time, to pick a class, you pick whichever of the two classes has the higher likelihood output from the softmax stage.  In other words, at training time, you look at the actual outputs (continuous-valued) from the softmax layer and use the cross-entropy; at test time, you compare the two outputs and pick whichever is larger to choose the classification (and apart from that ignore their exact values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The shortcoming of this approach is that it doesn't take into account the value of the $Y_i$'s, only their sign.  To improve this, use &lt;em&gt;weights&lt;/em&gt; on the samples.  We're again going to train a boolean classifier on the training set $(X_i,Z_i)$, but this time we'll weight each sample in the training set differently.  Put the weight $Y_i$ on the sample $(X_i,Z_i)$ in your training set.  In other words, when training the boolean classifier, the loss function will be a weighted sum of the loss for each sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{Loss}(\theta) = \sum_i Y_i \cdot \ell(f_\theta(X_i),Z_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta$ are the model parameters, $f_\theta(X_i)$ is the output of the classifier on input $X_i$, and $\ell(\cdot,\cdot)$ is the cross-entropy loss.  Notice that errors in samples where $Y_i$ is large are penalized more, and errors in samples where $Y_i$ is small are penalized less; this is appropriate, as that's exactly what will happen when you use the classifier $f_\theta$ to compute $L(f_\theta)$.  This should improve on the approach you're taking.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2017-01-24T18:15:12.303" LastActivityDate="2017-01-24T18:15:12.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16485" PostTypeId="1" AcceptedAnswerId="16487" CreationDate="2017-01-24T09:47:04.627" Score="1" ViewCount="171" Body="&lt;p&gt;I'm trying to build recommender based on user history from e-commerce. There are two(potentially more) types of events: purchase and view.&#xA;Is it okay to sum up number of purchases and views for a given item(with purchase and view having different weights)? Or I`ll just mix up user intent this way? &lt;/p&gt;&#xA;" OwnerUserId="27145" LastActivityDate="2017-01-24T10:20:07.717" Title="Recommender System: how to treat different events" Tags="|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16486" PostTypeId="2" ParentId="16472" CreationDate="2017-01-24T10:09:30.670" Score="2" Body="&lt;p&gt;To make our life easier, let's omit optimization methods here &lt;em&gt;(e.g. Learning Rate, Momentum)&lt;/em&gt; and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the chain rule, the gradient can be expressed as:&#xA;\begin{equation}&#xA;    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $x_{i}$ the input value to neuron $i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the partial derivative with respect to $W_{ij}$ can be computed as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial x_{i}}{\partial W_{ij}} = y_{j}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is simply equal to the output value $y_{j}$!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, the partial derivative with respect to $x_{i}$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial y_{i}}{\partial x_{i}} = \frac{\partial \phi(x_i)}{\partial x_{i}}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is simply $\frac{\partial \phi(x_i)}{\partial x_{i}}$, namely the derivative of the activation function &lt;em&gt;(i.e. Sigmoid, Tanh, ReLU...)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thirdly, the partial derivative with respect to $y_{i}$ can be computed as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial E}{\partial y_{i}} =&#xA;    \begin{cases}&#xA;        \begin{aligned}&#xA;            \frac{\partial}{\partial y_{i}} (T_i - y_i)&#xA;        \end{aligned} &amp;amp;&#xA;        \text{if $i \in$ output layer,}&#xA;        \\&#xA;        \begin{aligned}&#xA;            \frac{\partial}{\partial y_{i}} \left(\sum\limits^{n}_{j=1}W_{ij}\frac{\partial E}{\partial y_{j}}\right)&#xA;        \end{aligned} &amp;amp;&#xA;        \text{otherwise;}&#xA;    \end{cases}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With $T$ the target expected output &lt;em&gt;(i.e. the label)&lt;/em&gt;. The local error $e$ is computed differently depending on the layer position. For the output layer, the error is proportional to the difference between the predicted value (i.e. $y_{i}$) and the expected output (i.e. $T_{i}$). Otherwise, the error in hidden layers is proportional to the weighted sum of errors from connected layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourthly, combining the partial derivatives allow the computation of the error at a given neuron $i$ such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} = e_{i} =&#xA;    \begin{cases}&#xA;        \begin{aligned}&#xA;            \frac{\partial \phi(x_i)}{\partial x_{i}} (T_i - y_i)&#xA;        \end{aligned} &amp;amp;&#xA;        \text{if $i \in$ output layer,}&#xA;        \\&#xA;        \begin{aligned}&#xA;            \frac{\partial \phi(x_i)}{\partial x_{i}} \left(\sum\limits^{n}_{j=1}W_{ij} e_{j}\right)&#xA;        \end{aligned} &amp;amp;&#xA;        \text{otherwise;}&#xA;    \end{cases}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see here, you need to recursively compute the local error $e$ all the way to the input layer, that's why it's called backprop! ;) &lt;strong&gt;I think that what you missed!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we now simplify:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} = e_{i}y_{j}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We already know $y_{j}$, and we now know how to compute $e_{i}$ all the way to the input layer. So the weight can finally be updated from the gradient as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;    W_{ij} = W_{ij} - \,e_{i}y_{j}&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=i94OvYb6noo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=i94OvYb6noo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Backpropagation, Intuitions: &lt;a href=&quot;http://cs231n.github.io/optimization-2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/optimization-2/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Chapter 9.2 Classification by Backpropagation in Data mining: concepts and techniques: concepts and techniques. Jiawei Han, Micheline Kamber, and Jian Pei. 2011.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-01-24T10:09:30.670" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16487" PostTypeId="2" ParentId="16485" CreationDate="2017-01-24T10:20:07.717" Score="1" Body="&lt;p&gt;I assume you're using an implicit feedback recommender approach like ALS. Otherwise, summing data points generally won't make sense, such as if you're feeding it to a recommender that expects ratings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input to implicit ALS is, conceptually, weighted user-item pairs. Therefore it makes sense to perhaps use a sum of user-item clicks as the weight. Summing makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, does a purchase and view seem to carry the same weight? obviously not. A purchase is a much stronger association and should be weighted accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to how much, I'd suggest weighting purchases simply by price, to start. Then weight clicks by price times purchase-to-click ratio. A $10 item purchase is weight 10; if 1 in 200 clicks results in a purchase, then weight a click 0.2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is crude but probably about as close as anything for capturing this info in the context of ALS.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-01-24T10:20:07.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16488" PostTypeId="1" CreationDate="2017-01-24T11:13:22.463" Score="5" ViewCount="8413" Body="&lt;p&gt;Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the similarity APIs provided by the package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a smaller subset of this data which is already tagged as duplicated. Currently, I have not factored in this as part of model training. Also, using the text2vec package, the results are not great on this test set. So now I am considering RNNs which are known to perform well in text similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I need help in feature engineering and preparing my input layer. Sentences S1 and S2 that need to be compared differ in length (the word representations differ in their dimensions). How to normalize this difference? Should I consider bag-of-words or glove word vectors (word vectors being much superior representations)? Any inputs in this regard will help.&lt;/p&gt;&#xA;" OwnerUserId="28251" LastActivityDate="2017-02-25T19:11:28.223" Title="Text similarity using RNN" Tags="|machine-learning|nlp|similarity|rnn|text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16490" PostTypeId="1" AcceptedAnswerId="16497" CreationDate="2017-01-24T11:35:04.153" Score="0" ViewCount="755" Body="&lt;p&gt;I am new to machine learning and seek your help in clarifying my elementary doubts. I did a fair amount of googling, but find most literature jumping directly into math. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I know is that given a labelled training data, a ML algorithm chooses from a hypothesis space H a hypothesis function h.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, assume that a feature vector in training data contains 3 features (x1 through x3) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the data from the training set is taken and plugged into a formula (function type). If x is the feature vector and w represents the coefficients of the formula, then output y = A pre-determined f(w,x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Who decides the range of each of the coefficients?   &#xA;2. Who decides the formula? Is the formula fixed for a ML algorithm?&#xA;3. What exactly is the hypothesis space? Is it range of w, or is it different formulas or both?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I acknowledge I have asked more than one question and against the rules, but it was convenient to logically group them in a single post.  &lt;/p&gt;&#xA;" OwnerUserId="28211" LastActivityDate="2017-01-24T13:02:29.540" Title="Machine Learning - Range of Hypothesis space and choiceof Hypothesis function type" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16491" PostTypeId="1" CreationDate="2017-01-24T11:38:07.230" Score="1" ViewCount="78" Body="&lt;p&gt;After helpful advice from here, i have started my ML journey with SVM. Initially i started with 15 features and after cross validation i got about 40% accuracy. Problem is when i decided to just load all the features of the data available (205 in total) there appears to be no change at all. Am i doing something wrong? I tried removing all but 3 (many combinations) of the features and got the same results. Here is my code. Thank you for any help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import svm, preprocessing&#xA;from sklearn.model_selection import StratifiedShuffleSplit&#xA;import pandas as pd&#xA;&#xA;FEATURES =  ['b1', 'b2', 'b3', 'b4', 'b5', 'b6', 'b7', 'b8', 'b9', 'b10', 'b11', 'b12', 'b13', 'b14', 'b15',&#xA;             'r1', 'r2', 'r3', 'r4', 'r5', 'r6', 'r7', 'r8', 'r9', 'r10', 'r11', 'r12', 'r13', 'r14', 'r15',&#xA;             'd1', 'd2', 'd3', 'd4', 'd5', 'd6', 'd7', 'd8', 'd9', 'd10', 'd11', 'd12', 'd13', 'd14', 'd15',&#xA;             'j1', 'j2', 'j3', 'j4', 'j5', 'j6', 'j7', 'j8', 'j9', 'j10', 'j11', 'j12', 'j13', 'j14', 'j15',&#xA;             't1', 't2', 't3', 't4', 't5', 't6', 't7', 't8', 't9', 't10', 't11', 't12', 't13', 't14', 't15',&#xA;             'c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8', 'c9', 'c10', 'c11', 'c12', 'c13', 'c14', 'c15',&#xA;             's1', 's2', 's3', 's4', 's5', 's6', 's7', 's8', 's9', 's10', 's11', 's12', 's13', 's14', 's15',&#xA;             'a1', 'a2', 'a3', 'a4', 'a5', 'a6', 'a7', 'a8', 'a9', 'a10', 'a11', 'a12', 'a13', 'a14', 'a15',&#xA;             'pm1', 'pm2', 'pm3', 'pm4', 'pm5', 'pm6', 'pm7', 'pm8', 'pm9', 'pm10', 'pm11', 'pm12', 'pm13', 'pm14', 'pm15',&#xA;             'w1', 'w2', 'w3', 'w4', 'w5', 'w6', 'w7', 'w8', 'w9', 'w10', 'w11', 'w12', 'w13', 'w14', 'w15',&#xA;             '1l1', '1l2', '1l3', '1l4', '1l5', '1l6', '1l7', '1l8', '1l9', '1l10', '1l11', '1l12', '1l13', '1l14', '1l15',&#xA;             'pp1', 'pp2', 'pp3', 'pp4', 'pp5', 'pp6', 'pp7', 'pp8', 'pp9', 'pp10', 'pp11', 'pp12', 'pp13', 'pp14', 'pp15',&#xA;             'rs1', 'rs2', 'rs3', 'rs4', 'rs5', 'rs6', 'rs7', 'rs8', 'rs9', 'rs10', 'rs11', 'rs12', 'rs13', 'rs14', 'rs15',&#xA;             'dis', 'ral', 'str', 'tot', 'std', 'rstd1']&#xA;&#xA;def Predict():&#xA;    data_df = pd.DataFrame.from_csv(&quot;DATA.csv&quot;)&#xA;    data_df = data_df.reindex(np.random.permutation(data_df.index))&#xA;&#xA;    X = np.array(data_df[FEATURES].values)&#xA;&#xA;    y = (data_df[&quot;aares&quot;].values)&#xA;&#xA;    X = preprocessing.scale(X)&#xA;&#xA;    sss = StratifiedShuffleSplit(n_splits=5, test_size=0.5, random_state=0)&#xA;    sss.get_n_splits(X, y)&#xA;    for train_index, test_index in sss.split(X, y):&#xA;        #print(&quot;TRAIN:&quot;, train_index, &quot;TEST:&quot;, test_index)&#xA;        X_train, X_test = X[train_index], X[test_index]&#xA;        y_train, y_test = y[train_index], y[test_index]&#xA;&#xA;&#xA;    clf = svm.SVC(kernel='rbf', C=1).fit(X_train, y_train)&#xA;    print clf.score(X_test, y_test)&#xA;&#xA;    return X,y&#xA;&#xA;Predict()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: I am not necessarily expecting a better result, it seems counter-intuitive that such a large addition of features wouldn't have more of an impact.&lt;/p&gt;&#xA;" OwnerUserId="27962" LastEditorUserId="27962" LastEditDate="2017-01-24T11:53:55.527" LastActivityDate="2017-01-24T11:53:55.527" Title="Python svm classification, result vs amount of features not as expected" Tags="|machine-learning|python|classification|svm|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16493" PostTypeId="2" ParentId="16363" CreationDate="2017-01-24T11:58:02.120" Score="1" Body="&lt;p&gt;Let me share my list of ML training resources:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.crowdanalytix.com/listContests&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.crowdanalytix.com/listContests&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://datahack.analyticsvidhya.com/contest/all/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://datahack.analyticsvidhya.com/contest/all/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.chalearn.org/challenges.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.chalearn.org/challenges.html&lt;/a&gt; (some links may be dead)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And I am sure you read this Reddit: &lt;a href=&quot;http://www.reddit.com/r/MachineLearning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.reddit.com/r/MachineLearning/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12903" LastActivityDate="2017-01-24T11:58:02.120" CommentCount="0" CommunityOwnedDate="2017-01-24T12:14:21.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="16494" PostTypeId="1" AcceptedAnswerId="16502" CreationDate="2017-01-24T12:13:13.107" Score="0" ViewCount="526" Body="&lt;p&gt;I have a dataset of web browsing histories for users visiting a particular website over a period of time (say the last 90 days). Each user has a unique ID and several records showing when he/she visited a particular page on the website.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UserID,Timestamp,Path&#xA;U_1,2017-01-24 12:05:43,/sport/rugby/article_title&#xA;U_1,2017-01-24 12:06:56,/sport/football/article_title&#xA;U_1,2017-01-24 15:26:12,/finance/local/article_title&#xA;......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I do not have access to the content of the articles, I just know the path to the article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to build a classifier to predict if a user will take an action or not. So I need to extract features from each user data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I have ground-truth information associated with each user, indicating when a user did the action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first guess is to aggregate all the records of each user and extract frequency features (hashing TF) from each level of the path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for demonstration, a particular user might visit the &lt;code&gt;/sport&lt;/code&gt; category 5 times (first level category), and the &lt;code&gt;/sport/football&lt;/code&gt; category 3 times (second level category), and the &lt;code&gt;/sport/rugby&lt;/code&gt; 2 times (second level category).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for each user I will have a feature vector representing the frequency of the first level categories, and another one for the second level categories and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can now train a classifier for each feature vector and do a late fusion of the results, or I can concatenate (early fusion) the different features and train a single classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can also extract the terms from the article titles and build a TFIDF feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying now is to extract the features from the &lt;code&gt;N&lt;/code&gt; days proceeding taking the actions for the positive samples, and randomly select N consequent days from negative users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the possible other features that I can extract, and is there any better ML techniques to use in order to model and learn the user web browsing behaviour?&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="13891" LastEditDate="2017-01-24T12:41:48.230" LastActivityDate="2018-07-03T16:23:29.753" Title="Feature extraction from web browsing history of one website" Tags="|classification|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16496" PostTypeId="2" ParentId="16488" CreationDate="2017-01-24T12:33:50.683" Score="3" Body="&lt;p&gt;Doc2Vec, Mikolov's paper will solve your problem. Here is the &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://cs.stanford.edu/~quocle/paragraph_vector.pdf&amp;amp;ved=0ahUKEwi3oeH-49rRAhWMK48KHThDDtQQFgg1MAI&amp;amp;usg=AFQjCNE-2SsR07iQm78dQqVV-6_Y3ERurA&amp;amp;sig2=w78U9ToUkS-bQ3XJHJTRHg&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. You can find a gensim implementation&lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://rare-technologies.com/doc2vec-tutorial/&amp;amp;ved=0ahUKEwi3oeH-49rRAhWMK48KHThDDtQQFggpMAE&amp;amp;usg=AFQjCNEHwo92394MuL8R32lEWY-werXY0w&amp;amp;sig2=ni-EDOJ_zsQ-A8iuLeEH4g&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. While using RNN, using GLOVE or Googl Word2Vec will be always useful even if your sequences are of the same length. To solve the variable sequence problem you can pad the sentences and use bucketing or just pad to one uniform length, truncating longer sentences and padding shorter sentences. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-24T12:33:50.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16497" PostTypeId="2" ParentId="16490" CreationDate="2017-01-24T13:02:29.540" Score="1" Body="&lt;p&gt;I will answer your question as I understand them, the clarity you are looking for comes from the Maths behind ML.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There is no fix range of values of the coefficients or weights. Finding the value of these weights is the work of ML algorithms. although using regularization you can reduce the value of these weights but range depends on algorithm. no. of weights you can decide&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, formula is loosely fixed for a ML algorithm. &#xA;e.g. for linear regression in your case&lt;/p&gt;&#xA;&#xA;&lt;p&gt;h(w,x) = w0*x0 + w1*x1 + w2*x2 +w3*x3 (where x0 =1)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;the weights will change after each iteration though. for neural network it is multiplication of a weight matrix with an input matrix for each layer. &#xA;The output of h(w,x) is the prediction of ML which you can compare against y.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;All the ML algorithms available are your hypotheses space(H)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;e.g., quadratic, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;linear equation(the one above), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;high degree polynomial, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;complex neural network.&lt;/p&gt;&#xA;" OwnerUserId="27989" LastActivityDate="2017-01-24T13:02:29.540" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16498" PostTypeId="2" ParentId="16475" CreationDate="2017-01-24T13:34:46.303" Score="4" Body="&lt;p&gt;There seem to be a few options, but I found &lt;a href=&quot;https://github.com/Quasimondo/RasterFairy&quot; rel=&quot;nofollow noreferrer&quot;&gt;rasterfairy&lt;/a&gt; which is very easy to install and use. Has the added bonus of being able to fit to a rectangular grid, but also circular and other arbitrary shapes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very nice IronPython notebook example: &lt;a href=&quot;https://github.com/Quasimondo/RasterFairy/blob/master/examples/Raster%20Fairy%20Demo%201.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Quasimondo/RasterFairy/blob/master/examples/Raster%20Fairy%20Demo%201.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And some example results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3wbeI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3wbeI.png&quot; alt=&quot;tsne before gridding&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H2iU8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H2iU8.png&quot; alt=&quot;gridded tsne&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28234" LastActivityDate="2017-01-24T13:34:46.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16501" PostTypeId="2" ParentId="16374" CreationDate="2017-01-24T18:16:08.793" Score="4" Body="&lt;p&gt;If it's important for your use cases, you could try switching to Apache Zeppelin.&#xA;As all Spark notebooks there share the same Spark context, same Python running environment.&#xA;&lt;a href=&quot;https://zeppelin.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://zeppelin.apache.org/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between all Spark notebooks (they're called 'notes' in Zeppelin):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/w57wT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w57wT.png&quot; alt=&quot;Spark Interpreter Sharing options in Zeppelin&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can choose to share context Globally (default Zeppelin's behavior), Per Note (the only possible Jupyter's behavior), or Per User.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can't / don't want to switch to Zeppelin, look at other options of sharing common dataframes between your notebooks using:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arrow.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Arrow&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/wesm/feather&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feather&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;ps. You can't import ipynb files to Zeppelin currently as of now (it has its own notebook format stored as a json file), until &lt;a href=&quot;https://issues.apache.org/jira/browse/ZEPPELIN-1793&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://issues.apache.org/jira/browse/ZEPPELIN-1793&lt;/a&gt; is implemented; although it's not that hard to convert them manually in most cases.&lt;/p&gt;&#xA;" OwnerUserId="5380" LastEditorUserId="5380" LastEditDate="2017-01-26T20:01:49.067" LastActivityDate="2017-01-26T20:01:49.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16502" PostTypeId="2" ParentId="16494" CreationDate="2017-01-24T18:32:38.120" Score="1" Body="&lt;p&gt;What you describe uses the data that the paths can offer. You can easily generate features from the data and time. For instance, given the date, you can generate a categorical variable denoting the weekday (Monday, Tuesday, etc..). Given the timestamp, you can generate binary variables to partition the day in four or more partitions: is_morning, is_afternoon etc.. Somebody may only read in the morning or at night, and the aim of these features is to capture this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, you can get the interactions between weekdays and day partitions. Such features may help to distinguish users that in the Sunday mornings read about sports while Monday mornings they are at work and read financial news. Be careful of the overfitting though. Note that trees have been shown to capture such complex interactions; given them explicitly is beneficial though.   &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2017-01-24T18:32:38.120" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16503" PostTypeId="2" ParentId="16404" CreationDate="2017-01-24T18:33:37.813" Score="2" Body="&lt;p&gt;Using your input data as-is (many categorical attributes), you should start with random forests.  The levels of your features probably aren't ordinal in any sense, which would explain why multinom (a linear model) didn't work.  A decision tree (component of a random forest) won't have trouble with categorical features that aren't ordinal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your features are correlated (I'm guessing they are highly correlated), you might also try first reducing the dimensionality of your input dataset.  A nice way to do this is to view your categorical features as &quot;words&quot; and map each input sample (a word in all its forms) to a sparse vector (training a TF-IDF model).  Then, reduce the dimensionality of your sparse vectors using LSI or LDA (dense vectors).  Train a classifier (again, random forest is a good starting point) using these dense vectors as input data.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2017-01-24T18:33:37.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16505" PostTypeId="2" ParentId="16346" CreationDate="2017-01-24T20:26:40.590" Score="2" Body="&lt;p&gt;I decided turning my comment into an answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to go pro, use a framework such as &lt;a href=&quot;https://scrapy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scrapy.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I find them overly-cumbersome and I have been successful using the following approach. I think your use case is simple enough for it to be of use to you as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you are using Python3 as well, you can grab a webpage easily, and then get what you want using XPath notation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from lxml import html&#xA;import urllib.request&#xA;&#xA;# keep running until there are no &quot;next&quot; pages&#xA;for page in range(999):  &#xA;    url = 'http://blablabla.com/?page=%d' % page&#xA;    text = urllib.request.urlopen(url).read()&#xA;    tree = html.fromstring(text)&#xA;    images = tree.xpath('//img[@class=&quot;car&quot;]/href()')&#xA;    types = tree.xpath('//div[@class=&quot;type&quot;]/text()')&#xA;    if not images:&#xA;        break&#xA;    for i, (cartype, image) in enumerate(zip(types, images)):&#xA;        urllib.request.urlretrieve(image, '%s-page%d-img%d.png' % (cartype, page, i))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(Purely illustrative example.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now adjust as you may. XPath is an incredibly powerful notation of accessing XML nodes. Much more is possible than I am writing here. Take &lt;a href=&quot;http://www.w3schools.com/xml/xpath_intro.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt; to learn the full XPath syntax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some web designers make it much harder to access whatever you want because they do not properly &lt;code&gt;class&lt;/code&gt;-ify their HTML objects. In those cases, you may have to access a parent node and ask for their child. Or access a sibling and then get the siblings. Anyway, XPath and Python's &lt;code&gt;lxml&lt;/code&gt; package make all this incredibly easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any modern browser like Chrome and Firefox also lets you easily explore the DOM of any webpage. Just right-click and press Inspect or go to Developer Tools in the Tools menu or some such.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; some website like &lt;a href=&quot;https://scholar.google.pt/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scholar.google.com&lt;/a&gt; disallow scrapers and are very good at detecting if that's what you're doing. You can specify an user-agent for urllib, but it might be futile. Even advanced frameworks may not be able to help you there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I have made a &lt;a href=&quot;https://rpmcruz.github.io/machine%20learning/2017/01/25/python-scrapper.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; where I elaborate a little more.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2017-01-31T14:37:41.850" LastActivityDate="2017-01-31T14:37:41.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16506" PostTypeId="2" ParentId="16346" CreationDate="2017-01-24T21:03:52.347" Score="1" Body="&lt;p&gt;Have you looked at the &lt;a href=&quot;http://ai.stanford.edu/~jkrause/cars/car_dataset.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Car Dataset&lt;/a&gt;? It has about 16k images of 200 cars. While it does not have the number of images you are looking for, it does seem sufficient for building a classifier (see references below)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.justindchien.com/blog/motordex&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; by Justin Chien provides a good overview of the approach for building the classifier on this data using CNN. And &lt;a href=&quot;http://cs231n.stanford.edu/reports/lediurfinal.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; also provides an overview of a few different approaches.&lt;/p&gt;&#xA;" OwnerUserId="28228" LastActivityDate="2017-01-24T21:03:52.347" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16507" PostTypeId="2" ParentId="12974" CreationDate="2017-01-24T21:44:14.103" Score="0" Body="&lt;p&gt;The correct answer is: &quot;An external process is not differentiable (unless you know each detail, what is impossible in this case), so this problem should be faced as a Reinforcement Learning problem&quot;&lt;/p&gt;&#xA;" OwnerUserId="21775" LastActivityDate="2017-01-24T21:44:14.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16509" PostTypeId="1" AcceptedAnswerId="40575" CreationDate="2017-01-25T01:14:59.223" Score="4" ViewCount="1077" Body="&lt;p&gt;I have a set of features, one of which is a string. I convert the string to an integer by treating the string as a base 36 number (I only use the first 13 characters). Then I can use DecisionTrees since in the sklearn implementation you need to convert it to a number. When I tried a different model, say Logistic Regression, performance drops drastically, say from 80% to 30% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might have accepted this result if I had been able to use the strings as such in the DecisionTrees model, but since I used the same string to integer conversion for both models, why such great difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot go into the details, but let me provide you with an analogy. Let's say you are classifying millions of objects by their usefulness. So you say hammers are 4, screwdrivers 6, washers 10, etc. Of course you have more than one screwdriver, and sometime you forget and give it a value of 5, or something else. The model goes through millions of example, and then makes a prediction about the number for each object. I converted the names into integers, as I explained, and decision trees gives me an 80% accuracy, linear regression 30%. I assume that the problem is that linear regression tries to figure out some mathematical rule that does not exist. But why is decision trees immune from this problem?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2017-01-25T12:57:21.617" LastActivityDate="2018-11-01T16:43:50.630" Title="Performance difference between decision trees and logistic regression when one of the features is a string" Tags="|python|scikit-learn|decision-trees|linear-regression|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16510" PostTypeId="1" CreationDate="2017-01-25T02:50:08.390" Score="1" ViewCount="2865" Body="&lt;p&gt;I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ordering in my labels which I would like to avoid. If I use one hot encoding I would end up with really large vectors with 100 million dimensions. What are my options?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2020-01-03T19:11:34.673" LastActivityDate="2020-01-03T19:11:34.673" Title="What should I use if I have millions of possible values for a feature in a sklearn predictive model?" Tags="|python|scikit-learn|categorical-data|" AnswerCount="3" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="16511" PostTypeId="1" CreationDate="2017-01-25T03:55:49.053" Score="3" ViewCount="584" Body="&lt;p&gt;I started with a dataset that contained many dimensions for individuals (each id is a separate individual), and extracted three &lt;code&gt;Features/Attributes&lt;/code&gt; columns for each &lt;code&gt;individual Id&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to bucket these individuals into two or three distinct groups, based on these features, to see if I can identify distinct separations between certain groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if anyone has any recommendations, or algorithms (preferably in Python) that would cluster these individuals into distinct groups?  I do not have these &lt;code&gt;Individuals&lt;/code&gt; classified, so it's an unsupervised clustering problem.  I was thinking that &lt;code&gt;K-Means&lt;/code&gt; might be a good option, or something similar to PCA which could reduce my dimensionality and provide insight into the Features that seem to separate the groups of individuals into distinct groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for looking!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The data shown below was artificially generated to illustrate my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MaU9y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MaU9y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appendix: Reference Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pattern,Feature A,Feature B,Feature C&#xA;&#xA;6,2.18,0.13,8.00&#xA;9,9.31,3.67,6.58&#xA;11,0.89,1.83,4.33&#xA;13,9.73,9.50,1.59&#xA;23,0.51,6.49,0.09&#xA;26,9.04,3.42,3.90&#xA;27,8.35,9.18,3.40&#xA;28,3.04,5.63,6.88&#xA;32,9.78,6.52,8.50&#xA;43,4.11,3.36,1.83&#xA;49,9.57,1.52,7.09&#xA;51,1.13,9.98,9.42&#xA;53,6.22,1.37,7.07&#xA;62,8.79,3.03,7.52&#xA;63,6.27,7.29,0.98&#xA;71,4.64,0.06,6.55&#xA;73,1.34,9.32,5.15&#xA;83,4.53,3.85,2.04&#xA;84,9.48,9.71,3.23&#xA;86,3.80,3.00,0.76&#xA;88,1.73,0.64,9.96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="381" LastEditDate="2017-01-25T05:55:46.000" LastActivityDate="2017-01-27T09:04:42.507" Title="What are some methods for clustering individuals into distinct groups based on Features?" Tags="|data-mining|k-means|pca|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16512" PostTypeId="1" CreationDate="2017-01-25T04:29:01.873" Score="1" ViewCount="188" Body="&lt;p&gt;Using basic Backpropagation and a Sigmoid activation function, solving for $\cfrac{\partial E}{\partial W_i}$ for all $W_i$ is rather straightforward. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when using time series data, how do you approach backpropagation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition tells me that you just do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(1) \sum_{t=1}^n \cfrac{\partial  E_t}{\partial W_i} = \sum_{t=1}^n \cfrac{\partial  E_t}{\partial o_{i_t}} \cfrac{\partial o_{i_t}}{\partial W_i}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $o_{i_t}$ is the output node at time $t$, and $W_i$ is the weight feeding into that node from the previous node, $x_i$, across all times $t = 1,..., n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That would mean, if $\delta_{o_i} = \cfrac{\partial E}{\partial \sigma(o_i)} \cfrac{\partial \sigma (o_i)}{\partial o_i}$, where $\sigma (x) = \cfrac{1}{1+e^{-x}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the appropriate representation of $\delta_{o_i}$ within the time series should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{t=1}^n \delta_{o_i} = \sum_{t=1}^n \cfrac{\partial E_t}{\partial \sigma(o_{i,t})} \frac{\partial \sigma (o_{i,t})}{\partial o_{i,t}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or rewritten, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{t=1}^n \delta_{o_i,t} \frac{\partial o_{i,t}}{\partial W_i} = \sum_{t=1}^n (Out_t-Target_t)(Out_t)(1-Out_t)(x_{i,t})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This requires me to calculate $n$ backpropagation problems per iteration. Is this the proper way to do it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that if I use this method and a particular node adds random error across the time series &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(say, the mean of $\cfrac{\partial  E_t}{\partial W_i}$ for all $t$ is roughly $0$), then the backpropagation algorithm will not adjust that weight since $$\sum_{t=1}^n \cfrac{\partial  E_t}{\partial W_i}$$ will be close to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any brief summaries of how to approach backpropagation across a time series? Or should I just train the network using each observation in the time series as an independent training point?&lt;/p&gt;&#xA;" OwnerUserId="28231" LastActivityDate="2017-01-25T04:29:01.873" Title="Backpropagation in a neural network with time series data?" Tags="|neural-network|backpropagation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16513" PostTypeId="1" CreationDate="2017-01-25T04:51:50.617" Score="1" ViewCount="2392" Body="&lt;p&gt;I recently came across Matt Zeiler's deconvolution (reversing convolution) &lt;a href=&quot;http://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;abc&quot;&gt;paper &lt;/a&gt;. How is deconvolution able to reverse the rectified scalar output? From what I understand it sounds analogous to reversing the output of a linear / logistic regression model. How is it possible to go from a scalar to a vector?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29169" LastEditDate="2021-02-16T02:53:26.483" LastActivityDate="2021-02-16T02:53:26.483" Title="How to reverse ReLU activation in deconvolution" Tags="|machine-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16514" PostTypeId="1" CreationDate="2017-01-25T05:12:23.750" Score="1" ViewCount="145" Body="&lt;p&gt;I have a tremendous amount of experience training supervised machine learning models. However, I recently became a data scientist at a small financial services company, and I've been asked to build some sort of model that assesses portfolio-level risk. I'm not sure how to go about this task. Is this a supervised learning problem? It doesn't sound like one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used to work as a software/data engineer at a different financial services company, and while there I was briefly exposed to the concept of Basel credit risk management. Is Basel used for assessing portfolio-level risk? As far as I could tell (I could be wrong), Basel seemed to aggregate charge-off predictions by running every single client through each of the company's relevant credit risk models (which were trained in a supervised fashion). If my memory is right, how does this aggregation work? Is it effective?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-04-26T02:18:15.923" Title="Understanding portfolio-level risk models" Tags="|machine-learning|supervised-learning|finance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16516" PostTypeId="2" ParentId="16514" CreationDate="2017-01-25T05:43:42.037" Score="2" Body="&lt;p&gt;I think this is more of a finance question. Assuming portfolio here refers to an investment of two assets, then &lt;a href=&quot;https://i.stack.imgur.com/qwNWh.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qwNWh.gif&quot; alt=&quot;Formula&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To determine the portfolio risk therefore, you'd need to either predict, or use historicals for $\sigma_1$, $\sigma_2$, $Cov_{1,2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formula from: &lt;a href=&quot;http://www.investopedia.com/exam-guide/cfa-level-1/portfolio-management/portfolio-calculations.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.investopedia.com/exam-guide/cfa-level-1/portfolio-management/portfolio-calculations.asp&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="25800" LastEditDate="2017-01-25T23:30:41.960" LastActivityDate="2017-01-25T23:30:41.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16517" PostTypeId="1" CreationDate="2017-01-25T08:14:08.530" Score="2" ViewCount="265" Body="&lt;p&gt;I am focusing on implement an automated solution that will take as input many time series data (one at a time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After padding any missing values in order to have a time series without gaps, I am trying to check whether each one is stationary or not and then proceed into using either hybridModel() or auto.arima() in order to get a proper forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the questions that have arisen is that I haven't found a complete solution/algorithm describing the steps one has to take to do so. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically it is not clear to me what to do if p-value is not &amp;lt; 0.05 and is for instance between 95% and 90% , i.e. &lt;code&gt;0.05 &amp;lt; p-value &amp;lt; 0.1&lt;/code&gt;. Is it stationary or not?&#xA;For instance after decomposing I get the screenshot below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/34Ph4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/34Ph4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I wonder whether I have to remove the trend in the series, and if that is the case how I could do so. It is not clear to me after the decomposition whether I have to get for instance the log() of my time series or find a way to remove the trend. Can this task be automated or it needs to be &quot;supervised&quot; in terms of having to beforehand examine the series decomposition etc?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why the seasonal seems so repeating even thought my time series shows one distinct spike during&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, I found information about the procedure online but I haven't found a tutorial of explanatory step-by-step guide in order not to replicate existing solutions but to be able to produce tailored solutions as there are huge differences between the time series I examine.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Sorry for the long post, I tried to gather some time series questions in one place that I think will benefit others as well.&lt;/p&gt;&#xA;" OwnerUserId="5029" LastActivityDate="2017-01-25T08:14:08.530" Title="Time Series Analysis in R" Tags="|r|time-series|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16518" PostTypeId="2" ParentId="12366" CreationDate="2017-01-25T08:32:46.303" Score="1" Body="&lt;p&gt;The Box Plot gives a T-Test for two-group comparisons and Anova for &gt; 2 group comparisons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Sieve Diagram gives a Chi2 test.&lt;/p&gt;&#xA;" OwnerUserId="28285" LastActivityDate="2017-01-25T08:32:46.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16519" PostTypeId="1" CreationDate="2017-01-25T09:04:55.027" Score="1" ViewCount="219" Body="&lt;p&gt;I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (&lt;a href=&quot;https://wikileaks.org/plusd/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://wikileaks.org/plusd/&lt;/a&gt;), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option for download these in any raw text format, so I'm afraid I'm forced to apply some kind of scraping I guess, but I'd be thankful if anyone would give a clue for some simpler solution, if exists any.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-01-25T09:04:55.027" Title="How to dowload Wikileaks Cable Leaks documents as text corpus?" Tags="|python|nlp|nltk|mongodb|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16521" PostTypeId="2" ParentId="10612" CreationDate="2017-01-25T09:36:12.433" Score="0" Body="&lt;p&gt;I was trying to solve this problem today, and coudnt find any module given by gensim.Doc2Vec that calculate similarity between inferred document vector not in trained model to the ones in trained model. So basically I did this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy import spatial&#xA;inferred_vector=model.infer_vector(sentence.split())&#xA;for labels in labelled_documents:&#xA;    print (1-  spatial.distance.cosine(inferred_vector,model.docvecs[labels]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-25T09:36:12.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16522" PostTypeId="2" ParentId="16510" CreationDate="2017-01-25T10:10:50.507" Score="4" Body="&lt;p&gt;Unfortunately, AFAIK no sklearn model supports categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, sklearn &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision trees&lt;/a&gt; only support rules such as &lt;code&gt;X&amp;lt;n&lt;/code&gt;, not &lt;code&gt;X==n&lt;/code&gt; which would be desirable here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the decision tree algorithm they implement only produces local one-look-ahead optimization. What this means is that, it may not produce rules such as &lt;code&gt;X&amp;lt;n&lt;/code&gt; followed by &lt;code&gt;X&amp;gt;n-1&lt;/code&gt;, even if such rule would be highly desirable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end you will end up with non-sense things like: &lt;code&gt;Car &amp;gt; 1&lt;/code&gt; and then &lt;code&gt;Car &amp;lt; 6&lt;/code&gt; where 1 is Volkswagen and 6 is Ferrari.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The typical workaround is to use one-hot encoding, which might be a pain in the ass in your case. And then, it might not: sklearn decision tree supports sparse matrices, so the memory penalty would be low. You can use &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.18.1/reference/sparse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy&lt;/a&gt; for this. (Sparse matrices stores data differently than regular matrices. Instead of requiring $n\times m$ size, the memory requirement is proportional to the number of non-zeros in your matrix.) In terms of speed, it should not be any different than if the algorithm natively supported categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This being said, your data may not allow for use of sparse, if the rest of the features are non-sparse. I don't think scipy has support for a sparse-dense hybrid matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another workaround I can think of would be to produce an Euclidean distance matrix between observations of the various categories (you may want to normalize first). Then group categories that are close together. Then build a hierarchical model, where you predict the final category for each category. In python, this is easier than it seems. You can create a class for your model using sklearn base classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I love Python and sklearn. But I believe in using the right tool for each job. I would use R which has native support for categorial variables (they call them &lt;code&gt;factors&lt;/code&gt;) and has a plethora of &lt;a href=&quot;http://www.statmethods.net/advstats/cart.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision tree packages.&lt;/a&gt; (Note: xgboost for R does not support categorial variables, it ignores the factor class-type.) Weka could also be a good tool, which also has very powerful decision tree algorithms.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2017-01-25T10:18:00.327" LastActivityDate="2017-01-25T10:18:00.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16523" PostTypeId="2" ParentId="15457" CreationDate="2017-01-25T10:27:13.380" Score="2" Body="&lt;p&gt;Let me try to answer your questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The $a^{(k)}_{ij}$ element of the matrix $A_k$ is an approximation of the $a_{ij}$ element of the original matrix $A$. In particular, if you were to use the $U$, $\Sigma$, and $V$ matrices of expansion as they are (without the dimension reduction) you would have $a^{(k)}_{ij} = a^{ij}$. As a corollary, it's impossible to use the $A_k$ as a prediction matrix, because you &lt;em&gt;already&lt;/em&gt; have this information in matrix $A$. For example, if the $p^{th}$ user did not rate the $q^{th}$ product, $a_{pq}$ will be $0$. At the same time $a^{(k)}_{pq}$ will be $0$ too. Not a very insightful prediction, is it?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Cosine similarity allows you to judge if two vectors have the same orientation, in other words to what extent are they similar. Now let's look at the components of the SVD decomposition once again. $V$ defines the hidden product's features and the $i^{th}$ column of the matrix $V$ addresses the $i^{th}$ product. $U$ defines how a user reacts to the presence or absence of a product's particular feature. Knowing the features of the products you can find similar products using the cosine similarity. For example, the similarity of the 1st and the 2nd products is simply the cosine between the $V$'s first and second columns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You should use the mean absolute error or the mean squared error multiple times while implementing the recommendation engine. First, if you want to perform dimension reduction you will need to find $k$ for which the value of $\mbox{MEAN}(A, A_k)$ is reasonable. The second case for MEAN will be described below.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Interim summary: you shouldn't use $A_k$ for predictions, instead you should use the components of expansion themselves to find similar products. Now let's formulate said above as an algorithm that implements a naive recommendation engine using SVD.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Decompose the original matrix $A$ into three matrices $U$, $\Sigma$, and $V$.&lt;/li&gt;&#xA;&lt;li&gt;Knowing the hidden features $V$, for each product find its similarity with all other products.&lt;/li&gt;&#xA;&lt;li&gt;For each user find products he or she has already rated high. Pick the necessary quantity of products similar to those the user likes and return them as the recommendations.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you split your data into train and test sets, you can examine your predictions at the moment by calculating the MEAN between what your engine predictions are for the test set and how users actually rated products from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This schema can be inverted in the sense of you can find similar users instead of similar products and recommend products similar users like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this will help.&lt;/p&gt;&#xA;" OwnerUserId="27943" LastEditorUserId="27943" LastEditDate="2017-01-27T14:29:13.883" LastActivityDate="2017-01-27T14:29:13.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16525" PostTypeId="1" CreationDate="2017-01-25T12:57:36.497" Score="2" ViewCount="109" Body="&lt;p&gt;In order to ensemble a decision tree, let me explain the specific situation.&#xA;I have split the dataset in 5 sections, per protocol e.g. TCP, HTTP etc.&#xA;Now I've trained a decision tree for each one and run this against my test dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I go about combining the 5 prediction models. &#xA;Do I....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) combine the predicted output e.g. 1,0;1,1 for each section done separate and then run it against the test dataset to identify confusion matrix. This is actual class, predicted class.&#xA;or&lt;br&gt;&#xA;b) do I take the tree build and then add the additional ones to the tree model, in effect combining them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which method should be suitable, would option a even be a good solution?&lt;/p&gt;&#xA;" OwnerUserId="28293" LastActivityDate="2017-05-15T09:20:01.897" Title="Decision Tree Ensembling" Tags="|decision-trees|ensemble-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16526" PostTypeId="2" ParentId="16525" CreationDate="2017-01-25T13:09:51.033" Score="1" Body="&lt;p&gt;Popular ways to combine trained models are schema's based on voting: predictions are treated as weighted votes, the class with the majority of votes is selected as ultimate prediction, and/or stacking: predictions are treated as features in a newly trained model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the domain at hand one of this approaches might help you. Stacking (depending on the metalearner of choice) introduces more freedom and might introduce a complexity that is not warranted by the concept you are searching for, while voting won't help much when identifying subdomain fitted for perticular problems in your search space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also when combining multiple classifiers it might be worth your while to try out some fundamentally different approaches. &lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-01-25T13:09:51.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16528" PostTypeId="2" ParentId="16511" CreationDate="2017-01-25T16:09:27.233" Score="3" Body="&lt;p&gt;Sounds like you are on the right track.  There are many ways to attack this problem.  If you want to visualize the clustering, it would help to reduce the data down to two components.  This could be done via PCA or manifold learning if you want to go non-linear (&lt;a href=&quot;http://scikit-learn.org/stable/modules/manifold.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/manifold.html&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of clustering, there are many different methods to do this.  Here are some comparisons of different methods: &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/clustering.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-01-25T16:09:27.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16529" PostTypeId="1" AcceptedAnswerId="16531" CreationDate="2017-01-25T16:24:44.737" Score="4" ViewCount="422" Body="&lt;p&gt;I know that Keras is developed for quick deployment. Is it just for beginners or also useful in industry for professionals?&lt;/p&gt;&#xA;" OwnerUserId="27616" LastActivityDate="2017-01-27T09:14:54.593" Title="Is Keras useful for professionals?" Tags="|keras|" AnswerCount="1" CommentCount="0" ClosedDate="2017-01-26T10:17:10.577" ContentLicense="CC BY-SA 3.0" />
  <row Id="16531" PostTypeId="2" ParentId="16529" CreationDate="2017-01-25T17:15:38.683" Score="5" Body="&lt;p&gt;Keras is used in academia (see &lt;a href=&quot;https://scholar.google.es/scholar?cites=17868569268188187229&amp;amp;as_sdt=2005&amp;amp;sciodt=0,5&amp;amp;hl=en&quot; rel=&quot;nofollow noreferrer&quot;&gt;google scholar citations&lt;/a&gt; of Keras as a proxy for academy adoption) and hobbyists (see &lt;a href=&quot;https://github.com/fchollet/keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;github stars&lt;/a&gt; or &lt;a href=&quot;https://www.google.es/webhp?#q=site:www.kaggle.com+keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;google results for &lt;em&gt;keras&lt;/em&gt; in www.kaggle.com&lt;/a&gt; as proxies for hobbyist adoption).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It was &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/5jg7b8/p_deep_learning_for_coders18_hours_of_lessons_for/dbhaizx/&quot; rel=&quot;nofollow noreferrer&quot;&gt;recently&lt;/a&gt; bundled together with Google's tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also used in industry, at least for prototyping (I know because I use it!), but I have found no source to back this statement.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-01-27T09:14:54.593" LastActivityDate="2017-01-27T09:14:54.593" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16532" PostTypeId="1" AcceptedAnswerId="16544" CreationDate="2017-01-25T17:55:49.477" Score="12" ViewCount="10317" Body="&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Given a big image dataset (around 36 GiB of raw pixels) of unlabeled data, how can I cluster the images (based on the pixel values) without knowing the number of clusters &lt;code&gt;K&lt;/code&gt; to begin with?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working on an unsupervised learning project to cluster images; think of it as clustering MNIST with 16x16x3 RGB pixel values, only that I have about 48 million examples that I need to cluster.&#xA;Without knowing their identities, I do know that &lt;em&gt;some&lt;/em&gt; of the images are &lt;em&gt;definitely&lt;/em&gt; related because they come from the same source, but - say - I also don't know an appropriate &lt;code&gt;K&lt;/code&gt; in order to &quot;just&quot; run K-means on the set yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of doing some manual 2D embedding using t-SNE and then clustering manually in the embedded space (a simpler task than doing it manually in &lt;code&gt;16x16x3&lt;/code&gt;-d), but all t-SNE implementations I could find required loading the data into memory. I also thought about first running t-SNE, then K-means on the t-SNE embedded data, but if you look at the results of t-SNE from MNIST, it's very obvious that these clusters might and probably will be distorted and skewed in nonlinear ways. So even if I were to know a &lt;code&gt;K&lt;/code&gt;, the clusters would probably be split up. Using Mahalanobis distances for K-means might be an interesting thing, but since I don't know covariances to begin with, that appears to be a dead end as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm trying if I can run PCA compression on the examples to at least gain some memory back for t-SNE, but that might or might not work ... can't say for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody give me a pointer in the right direction to do this (&lt;em&gt;ideally&lt;/em&gt;, but definitely not necessary in a Python, TensorFlow or Apache Beam/Dataflow context)? I was working on porting a Streaming/Ball K-means a while ago which does have the nice property of creating new clusters &quot;on demand&quot;, but before I start implementing that again in Python/TensorFlow/Dataflow, I was hoping that somebody could give me some ideas where to start or what to avoid.&lt;/p&gt;&#xA;" OwnerUserId="3255" LastEditorUserId="3255" LastEditDate="2017-01-25T21:05:30.467" LastActivityDate="2019-08-07T13:12:51.963" Title="Clustering high dimensional data" Tags="|clustering|tensorflow|k-means|unsupervised-learning|tsne|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16533" PostTypeId="2" ParentId="16510" CreationDate="2017-01-25T19:05:28.900" Score="0" Body="&lt;p&gt;An elastic net (also known as L1 and L2 regression) model can be an effective technique for dealing with very high dimensional problems. It's often used for dealing with genomic data. Elastic net documentation for sklearn can be found &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;abc&quot;&gt;here&lt;/a&gt;. Elastic net differentiates between variables that have predictive value and those that don't, and it often sets useless variables' coefficients to zero. This can dramatically reduce the number of categories in your problem.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take advantage of sklearn by creating &quot;dummy&quot; variables out of your categories. If one of your levels/columns were &quot;color&quot;, and your categories were red, green and blue, then you would make two columns, e.g. for one for red and another for green (blue is not necessary by process of elimination). &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-01-25T19:05:28.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16534" PostTypeId="2" ParentId="16510" CreationDate="2017-01-25T19:24:50.437" Score="3" Body="&lt;p&gt;You can use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;sklearn.preprocessing.OneHotEncoder&lt;/code&gt;&lt;/a&gt; with &lt;code&gt;sparse=True&lt;/code&gt;. It will return a a &lt;code&gt;scipy.sparse&lt;/code&gt; matrix &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression.fit&quot; rel=&quot;nofollow noreferrer&quot;&gt;some models&lt;/a&gt; can work with. Your matrix will indeed be 100e6 columns wide, but not densely populated won't take a lot of RAM.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-01-25T19:24:50.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16535" PostTypeId="1" CreationDate="2017-01-25T19:42:33.667" Score="0" ViewCount="1379" Body="&lt;p&gt;I want to create a Google Data studio dashboard from Infusion soft data.&#xA;The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like Klipfolio, Clicdata, Grow etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a direct connection is not possible, I want to use some combination of Google sheets and Zapier or other free tools to create a data flow that can be constantly refreshed for data coming in from &quot;infusionsoft&quot; to &quot;Google data studio&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions summary:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there any direct way of connecting Infusionsoft data or API with Google data studio (maybe with some Database like MySQL/postgresql/cloud sql etc?)&lt;/li&gt;&#xA;&lt;li&gt;What are some free mid-tier tools I can use to create this data connection?(Zapier, google sheets, excel etc)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;All your suggestions are highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="28311" LastEditorUserId="103697" LastEditDate="2020-08-25T18:10:51.260" LastActivityDate="2020-08-25T18:10:51.260" Title="Connecting Infusionsoft data to Google data studio" Tags="|data|google|data-formats|google-data-studio|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16536" PostTypeId="2" ParentId="15322" CreationDate="2017-01-25T19:49:26.350" Score="1" Body="&lt;p&gt;You should definitely use nested cross-validation for model selection and performance estimation. I have also found that the AUC of the precision-recall (PR) curve (compared to that of the ROC curve) to be a better, that is, more stable, estimator of the performance of my Random Forest classifiers when I have a highly unbalanced dataset; there is research on this topic, showing that the AUC of the PR curve is more informative than that of the ROC curve. You can use &lt;code&gt;average_precision_score()&lt;/code&gt; in scikit-learn to use the PR AUC score. Along the lines of resampling the data, you could try approaches like EasyEnsenble and BalanceCascade; search for the papers titled &quot;Exploratory Undersampling for&#xA;Class-Imbalance Learning&quot; and &quot;Learning from Imbalanced Data&quot; for more information.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastActivityDate="2017-01-25T19:49:26.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16539" PostTypeId="2" ParentId="9131" CreationDate="2017-01-25T23:26:23.553" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Therefore, the year of the observation was considered as a predictor variable which had a positive impact on the dependent variable. However, the year itself doesn’t have any effect on the dependent variable per se; rather, it is the other factors occurring in the same time period which result in improvements.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This sounds like a challenge for trees with sufficient interaction depth, as you've found that the year interacts with other factors that results in improvements. Ordinary least squares regression here do not capture that type of interaction well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest setting up the following regression model:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Transform the date column into (CurrentYear - YearOfDateStamp) AS NumOfYearsAway. I'd recommend leaving this as a numerical feature rather than a categorical one. This would allow the tree based model to select cuts like NumOfYearsAway &gt;= 5.5 instead of NumOfYearsAway in (6,7,10). This could also be more helpful when NumOfYearsAway = 0 occurs in your scoring dataset, where you might not have training data for current year data set.&lt;/li&gt;&#xA;&lt;li&gt;Fit a tree based model, I'd pick XGBoost, with the usual CV to tune the hyperparameters such as interaction depth.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The drawback of picking XGBoost in your application however, is that the interpretation of the impact of a particular variable on the target variable is not obvious. You'd need partial dependence plot to observe how the target variable vary with the bespoke input variable. If interpretability is very important, one could pick a single tree regression model like rpart.&lt;/p&gt;&#xA;" OwnerUserId="25800" LastEditorUserId="25800" LastEditDate="2017-01-25T23:35:25.877" LastActivityDate="2017-01-25T23:35:25.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16540" PostTypeId="1" CreationDate="2017-01-25T23:29:28.693" Score="1" ViewCount="780" Body="&lt;p&gt;Lets say we have a training data and we have estimated a fit for a model of square ft of living area vs price of houses. Suppose we know the probability distribution of sq ft and for a fixed sq ft we know the probability distribution of its price. Now, we know the joint probability for each sq ft and price pair. How do I calculate the generalization error(both total and for a particular sq ft)?? &lt;/p&gt;&#xA;" OwnerUserId="28317" LastActivityDate="2017-01-25T23:29:28.693" Title="Generalization error for simple linear regression" Tags="|machine-learning|linear-regression|probability|distribution|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16542" PostTypeId="1" CreationDate="2017-01-26T06:52:48.163" Score="0" ViewCount="1273" Body="&lt;p&gt;I am a newbie to machine learning and have the following elementary questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Given a labeled dataset with multiple features, is it for a ML algorithm determine on its own what features are to be considered for determining the hypothesis function? Or is it for a user who is training the algorithm to explicitly indicate what features are to be considered?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a mapping between problem space versus the hypothesis function type? As an example based on past experience, can we confidently state that for a housing price prediction problem, linear equation is appropriate. While for determining diabetes based on a given dataset, a quadratic hypothesis equation has proven reasonable. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Or is it usual for a designer to specify a custom function and allow the ML algo to determine the coefficients of this custom function (e.g. square root of sum of squares of 9 of 15 features divided by e^n) by reducing the error using gradient descent?   &lt;/p&gt;&#xA;" OwnerUserId="28211" LastActivityDate="2017-01-27T08:49:50.620" Title="Machine Learning - Choice of features for determining hypothesis" Tags="|machine-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16544" PostTypeId="2" ParentId="16532" CreationDate="2017-01-26T08:17:44.337" Score="13" Body="&lt;p&gt;I don't think &lt;em&gt;any&lt;/em&gt; of the clustering techniques &quot;just&quot; work at such scale. The most scalable supposedly is k-means (just do not use Spark/Mahout, they are really bad) and DBSCAN (there are some good distributed versions available).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you will be &lt;strong&gt;facing many other challenges&lt;/strong&gt; besides scale because &lt;strong&gt;clustering is difficult&lt;/strong&gt;. It's not as if it's just enough to run the algorithm and then you have clusters. Clustering is an explorative technique. There is no &quot;correct&quot; clustering. But rather &lt;strong&gt;you will need to run clustering again and again, and look at every cluster&lt;/strong&gt;. Because there &lt;em&gt;will&lt;/em&gt; not be a single parameter setting that gets everything right. Instead, different clusters may appear only at different parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the main challenge in your case will likely be the distance function. Except for idealized settings like MNIST, Euclidean distance will not work at all. Nor will anything working on the raw pixels. So first you need to do feature extraction, then define a similarity function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to clustering, &lt;strong&gt;work with a sample&lt;/strong&gt;. Cluster the sample, identify interesting clusters, &lt;em&gt;then&lt;/em&gt; think of a way to generalize the label to your entire data set. For example by classification (your labeled data points are your training set, predict the labels of unlabeled points).&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="29575" LastEditDate="2019-08-07T13:12:51.963" LastActivityDate="2019-08-07T13:12:51.963" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16545" PostTypeId="1" CreationDate="2017-01-26T08:23:04.067" Score="4" ViewCount="1296" Body="&lt;p&gt;Let's say I begin with an exceptionally large dataframe (e.g. imported/munged from tsv files). Several of these columns are categorical labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(As a more concrete example, let's imagine a group of students in a school district, pre-school to high-school). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I begin using sklearn and instantiate a t-SNE model, similar to the example here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; import numpy as np&#xA; from sklearn.manifold import TSNE&#xA; X  # my data&#xA; model = TSNE(n_components=2, random_state=0)&#xA; np.set_printoptions(suppress=True)&#xA; model.fit_transform(X) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then we plot this. The plot might look something like this: &lt;a href=&quot;https://i.stack.imgur.com/Dgeu8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://i.stack.imgur.com/Dgeu8.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my problem: with real datasets, after using t-sne to learn/cluster, you will have a number of &quot;clusters&quot;. Then, using the categorical labels, I try to go through each of these, and try to figure out what structure the t-SNE plot is giving me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For our school example, I'd get the t-SNE output, then I would label the datapoints. (Let's assume that the clusters are actually representative of age/classroom, e.g. the first-graders group together, the second-graders are a group, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try to color this plot with &quot;grades&quot;, I'll see that the grades does not really explain the structure of this plot. (Why? Because every class-level has students with As, Bs, Cs, etc.) Then I might try height...that does pretty good (because there's a correlation between short students--&gt; pre-school, tall students --&gt; high school seniors). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does one use a t-SNE plot to infer the &quot;most correct&quot; labels of the data? How does one use t-SNE plots to explain (and further explore) the plot structure? &lt;/p&gt;&#xA;" OwnerUserId="23120" LastActivityDate="2017-01-26T15:57:24.603" Title="Given a t-SNE plot, how can I infer the &quot;most correct&quot; labels? How does one understand its structure?" Tags="|clustering|labels|tsne|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="16546" PostTypeId="1" CreationDate="2017-01-26T08:49:30.760" Score="0" ViewCount="112" Body="&lt;p&gt;Is there a way to weight instances based upon the values of a feature? &#xA;For instance for making a sample better resemble a population on certain parameters.&lt;/p&gt;&#xA;" OwnerUserId="28285" LastActivityDate="2017-04-26T23:37:38.793" Title="Is there a way to weight instances based upon the values of a feature?" Tags="|orange|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16547" PostTypeId="2" ParentId="16546" CreationDate="2017-01-26T09:10:40.950" Score="1" Body="&lt;p&gt;Try reading up on Loss/ Cost functions. Samples that are closer to your model will be assigned a smaller cost while others a higher one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Loss_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Loss_function&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21749" LastActivityDate="2017-01-26T09:10:40.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16548" PostTypeId="1" AcceptedAnswerId="16741" CreationDate="2017-01-26T09:34:44.130" Score="3" ViewCount="16182" Body="&lt;p&gt;I use a neural network with 3 inputs and 1 output with Keras. I'm using MinMaxScaler from sklearn to normalize my inputs in the range [0,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my input shape is (XX,3)&#xA;my output shape is (XX,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have any input while scaling the input and output arrays&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    self.scaler = MinMaxScaler(feature_range=(0,1))&#xA;&#xA;    dataX = self.scaler.fit_transform(dataX)&#xA;    dataY = self.scaler.fit_transform(dataY)&#xA;&#xA;    self.model.fit(dataX,dataY,nb_epoch=1000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After the training I have able to make prediction like this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    data    = numpy.array([[ val1,val2,val3]])&#xA;    data = self.scaler.fit_transform(data)&#xA;    prediction = self.model.predict(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but my output is not scaled correctly, it make sense, as I didn't call inverse transform to apply the inverse scaling &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but when I call it, I get a ValueError , I tried to apply the transformation on a single prediction or on an array of prediction but the problem is the same&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.scaler.inverse_transform(predictions) # prediction shape is (18,) :&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ValueError: operands could not be broadcast together with shapes (18,) (3,) (18,)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ValueError: non-broadcastable output operand with shape (18,1) doesn't match the broadcast shape (18,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I undestand it is a shape issue, but the error message does not help so much&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(12,1) was accepted by fit_transform (18,1) seems pretty similar.. so I dont get the error.&lt;/p&gt;&#xA;" OwnerUserId="27107" LastActivityDate="2017-02-04T10:29:16.867" Title="MinMaxScaler broadcast shapes" Tags="|scikit-learn|keras|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16550" PostTypeId="2" ParentId="16542" CreationDate="2017-01-26T11:53:47.950" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;There are algorithms that use Feature Selection to utilize only the &quot;best&quot; features for the given dataset (variants of adaboost for example). There are also algorithms randomly use subsets of the features (such as random forest). Finally there are algorithms that learn weigh your inputs and use them accordingly (Neural networks and the like)&lt;/li&gt;&#xA;&lt;li&gt;There is &quot;common knowledge&quot;, but it is more related to the type of features and classification problem that you are facing (i.e. regression vs. classification, continuous vs discrete, binary vs. multi-class, etc.).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you have a specific problem that you can model using such a function then it is a completely reasonable approach to design such a function.&#xA;Once you have modeled your problem with such a function and defined your loss function - you could use any optimization method that seems reasonable.&#xA;However it is beneficial to try and find similar problems that others have solved before and understand how their solution relates to your problem.&lt;/p&gt;&#xA;" OwnerUserId="28328" LastActivityDate="2017-01-26T11:53:47.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16552" PostTypeId="2" ParentId="16542" CreationDate="2017-01-26T12:04:29.747" Score="0" Body="&lt;p&gt;Firstly, I would like to tell you that domain knowledge plays a crucial role applying machine learning models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to your question 1: ML algorithm will not strictly determine weather to use a feature or not. ML algorithm will only tell you about the relevance of a particular feature present in your feature set. For example, while doing regression, variables associated with higher coefficients are of greater importance than the ones having lower weights,it might be very close to zero as well, which gives you hint to not consider the feature. Similar is the case with the fbeta score of a variable while we deal with trees. User can specify what features are to be considered. User could also make some derivative features which helps a lot while dealing with ML problems Generally this is an iterative process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second question : There is no specific mapping between the problem statement and hypothesis. You may develop some of them with experience. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should not(usually) choose such custom functions as they might be fitting your training data perfectly, but could give poor results on test data as there is loss of generality by using a specific function. You could try different algorithms, with making new features, using ensemble methods and other machine learning techniques to improve your score    &lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2017-01-26T12:04:29.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16553" PostTypeId="1" AcceptedAnswerId="16554" CreationDate="2017-01-26T13:06:08.353" Score="2" ViewCount="9282" Body="&lt;p&gt;I have the last couple of month worked with an regression problem, of turning a framed audio file into a set of mfcc features, for a  speech recognition application&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried a lot different network structures, Cnn, different normalisation techniques, different optimizer, adding more layers and so on.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but finally i've got some decent result, but i don't understand why.. &#xA;What i did was i added a linear layer as output, and somehow that minimised the error tremendeosly, and bit puzzled why a linear layer would have that much effect?... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean am still tried to fit the actual output to the desired output?..&#xA;Why would the activation function matter here?... I mean the weight are being adjusted based on the error, so why is the neural network better at adjusting for the error when the output is linear rather than non-linear (such as: tanh, Relu).. ?&lt;/p&gt;&#xA;" OwnerUserId="26085" LastActivityDate="2017-01-30T06:54:47.837" Title="The effect of an linear layer?" Tags="|neural-network|regression|audio-recognition|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16554" PostTypeId="2" ParentId="16553" CreationDate="2017-01-26T13:48:05.607" Score="2" Body="&lt;p&gt;If you are performing regression, you would usually have a final layer as linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely in your case - although you do not say - your target variable has a range outside of (-1.0, +1.0). Many standard activation functions have restricted output values. For example a sigmoid activation can only output values in range (0.0, 1.0) and a ReLU activation can only output positive values. If the target value is outside of their range, they will never be able to match it and loss values will be high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lesson to learn here is that it &lt;em&gt;is&lt;/em&gt; important which activation function you use in the output layer. The function must be able to output the full range of values of the target variable.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-01-26T13:48:05.607" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16556" PostTypeId="2" ParentId="16513" CreationDate="2017-01-26T14:54:14.573" Score="2" Body="&lt;p&gt;It's not reversing the ReLU, but applying normal ReLU activation again after max pooling and max unpooling.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is used again &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;to obtain valid feature reconstructions at each layer (which also should be positive)&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="27551" LastActivityDate="2017-01-26T14:54:14.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16557" PostTypeId="2" ParentId="16545" CreationDate="2017-01-26T15:57:24.603" Score="4" Body="&lt;p&gt;With t-SNE none of the input parameters are weighted more than any other parameter so the differences you want to see like students forming islands by grade level will not happen because there is so much other data present to pull those students/data points in different directions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly encourage you to have a specific question in mind and tailor your input categories so that your question can be answered by the t-sne map.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might try asking a specific question and changing which input categories you ask t-SNE to look at. For example, do taller students get better grades? Feed in height and grades  categories while leaving out grade level and age. This is a silly example but I hope it gives an idea of how you can use t-sne to help you learn about your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also find that there are categories that mask meaningful findings. Height might not be very useful for pulling out meaningful information and since the height range is going to be much larger than the A-F grade range, it will likely influence the t-sne map more. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the data as you described on a color scale for each category is a good place to start with each new t-sne run. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is okay to run multiple t-snes with different parameters. To be sure you are getting a meaningful answer, I recommend running t-sne multiple times with the same parameters as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have enough students, it would be great to included half in your training set where you explore and figure out what questions to ask and then when you think you've found something meaningful, apply those conditions to the other students in the testing set to see if it holds true. &lt;/p&gt;&#xA;" OwnerUserId="16590" LastActivityDate="2017-01-26T15:57:24.603" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16558" PostTypeId="1" CreationDate="2017-01-26T17:01:11.907" Score="0" ViewCount="338" Body="&lt;p&gt;When is a neural network considers to be overfitted to a specific task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use CV for training.&#xA;first i split the data into train/test =&gt; 90/10.&#xA;and within the training set ,i do another 90/10 split, which are those data I use for the training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then use the model for processing the complete dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;CV would state how much i am overfitting (I am getting minor error, deviating with 2%).. But since i am using most of the data for training? am I then creating an overfitted neural network for this dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently having 230043 training samples,  107145 total params, and validates on 25561 samples.  am I overfitting?&lt;/p&gt;&#xA;" OwnerUserId="26085" LastActivityDate="2017-02-25T19:41:53.710" Title="When is a Neural networked considered to be overfitted?" Tags="|neural-network|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16560" PostTypeId="1" CreationDate="2017-01-26T17:39:09.650" Score="3" ViewCount="462" Body="&lt;p&gt;I've done a simple naive bayes classification task with a very small data set. As the training set size increases from 100 data points to 300 data points, the F1 score on the test set decreases. But why? Is it likely overfitting? What are the first places I should examine to explain this behavior?&lt;/p&gt;&#xA;" OwnerUserId="14858" LastEditorUserId="14858" LastEditDate="2017-01-26T17:44:45.953" LastActivityDate="2017-10-24T04:32:42.380" Title="F1 on test set using Naive Bayes classifier decreases as training set size increases. Why?" Tags="|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16561" PostTypeId="2" ParentId="16558" CreationDate="2017-01-26T17:57:09.927" Score="1" Body="&lt;p&gt;A model is said to have overfitted when it performs awesomely on the training set (error's low), but when tested on test set, it fails badly. This might be due to bad sampling distribution over our training and tes divisions, so that model might not have seen this kind of data or due to using many irrelevant features in the samples than we were supposed to etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;You cannot really say that the cat is dead until you open the box. So until you test, you would never know.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This statement can be explained in general, not only for neural nets. Assume you know the relationship is linear for your data, but you got some error and you are not happy with this, then say you add an extra term say $ax^2$ to this equation and the model now has an extra parameter to tune and this makes our line into curve. For a neural net we basically start by random weights from a distribution and then we keep reducing them until we feel like our training error is cool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you are using a multi-layer perceptron (fully connected), pruning is one of the techniques to remove weights from the model which might not be so contributing still making sure your model still behaves well with lesser number of parameters. &lt;a href=&quot;https://en.wikipedia.org/wiki/Singular_value_decomposition&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVD&lt;/a&gt; on this weight matrix can be one of the useful techniques to do this. But how many weights to keep depends on how many &lt;a href=&quot;https://en.wikipedia.org/wiki/Singular_value&quot; rel=&quot;nofollow noreferrer&quot;&gt;singular values&lt;/a&gt; you keep for reconstructing the matrix. I heard that early stopping can reduce overfitting, but I had never really done this, so can't really give my opinions about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-26T17:57:09.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16563" PostTypeId="1" AcceptedAnswerId="16565" CreationDate="2017-01-26T18:29:03.823" Score="1" ViewCount="1421" Body="&lt;p&gt;I have to impute missing values in the column named &lt;code&gt;&amp;lt;Age&amp;gt;&lt;/code&gt; with the mean of the nearest available values above and below in the column &lt;code&gt;&amp;lt;Age&amp;gt;&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the &lt;code&gt;Age&lt;/code&gt; column had values in order&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;NA,7,6,NA,7,8,NA,NA,NA,10,5,NA,NA,5,9,9,12,8,6,NA,NA&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After imputation, column should look like&#xA; &lt;code&gt;7,7,6,6.5,7,8,9,9,9,10,5,5,5,5,9,9,12,8,6,6,6&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know about filling value with mean, mode, or median forward filling, backward filling but not this one. I am new to this field and a student. Any help would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="28344" LastEditorUserId="12384" LastEditDate="2017-01-27T10:13:45.773" LastActivityDate="2017-01-27T10:13:45.773" Title="handling missing data in pandas python" Tags="|python|data-cleaning|pandas|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16565" PostTypeId="2" ParentId="16563" CreationDate="2017-01-26T19:37:43.050" Score="5" Body="&lt;p&gt;IIUC you can simply use Pandas &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.interpolate.html&quot; rel=&quot;noreferrer&quot;&gt;Series.interpolate()&lt;/a&gt; method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [8]: NA = np.nan&#xA;&#xA;In [9]: s = pd.Series([NA,7,6,NA,7,8,NA,NA,NA,10,5,NA,NA,5,9,9,12,8,6,NA,NA])&#xA;&#xA;In [10]: s&#xA;Out[10]:&#xA;0      NaN&#xA;1      7.0&#xA;2      6.0&#xA;3      NaN&#xA;4      7.0&#xA;5      8.0&#xA;6      NaN&#xA;7      NaN&#xA;8      NaN&#xA;9     10.0&#xA;10     5.0&#xA;11     NaN&#xA;12     NaN&#xA;13     5.0&#xA;14     9.0&#xA;15     9.0&#xA;16    12.0&#xA;17     8.0&#xA;18     6.0&#xA;19     NaN&#xA;20     NaN&#xA;dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [11]: s.interpolate().bfill()&#xA;Out[11]:&#xA;0      7.0&#xA;1      7.0&#xA;2      6.0&#xA;3      6.5&#xA;4      7.0&#xA;5      8.0&#xA;6      8.5&#xA;7      9.0&#xA;8      9.5&#xA;9     10.0&#xA;10     5.0&#xA;11     5.0&#xA;12     5.0&#xA;13     5.0&#xA;14     9.0&#xA;15     9.0&#xA;16    12.0&#xA;17     8.0&#xA;18     6.0&#xA;19     6.0&#xA;20     6.0&#xA;dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if you need rounded integers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [13]: s.interpolate().round().bfill().astype(int)&#xA;Out[13]:&#xA;0      7&#xA;1      7&#xA;2      6&#xA;3      6&#xA;4      7&#xA;5      8&#xA;6      8&#xA;7      9&#xA;8     10&#xA;9     10&#xA;10     5&#xA;11     5&#xA;12     5&#xA;13     5&#xA;14     9&#xA;15     9&#xA;16    12&#xA;17     8&#xA;18     6&#xA;19     6&#xA;20     6&#xA;dtype: int32&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-01-26T19:37:43.050" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16566" PostTypeId="2" ParentId="16546" CreationDate="2017-01-26T19:49:46.427" Score="1" Body="&lt;p&gt;In &lt;a href=&quot;https://orange-visual-programming.readthedocs.io/loading-your-data/index.html#header-with-attribute-type-information&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange documentation - Loading your Data&lt;/a&gt;, it says to prefix your column with &lt;code&gt;w#&lt;/code&gt; to be marked as instance weights column.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-01-26T19:49:46.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16567" PostTypeId="2" ParentId="16560" CreationDate="2017-01-26T19:50:27.910" Score="2" Body="&lt;p&gt;Overfitted model is something that shows very less error on your training set and then when you test it on a test set, it fails badly. This might be due to class imbalance and you might be giving something that it has not seen or due to using more number of features than you were supposed to, making the model to fit in every datum. &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wiki&lt;/a&gt; page is nice overview read for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answering your next question, F1 score as might already be knowing has two components in it, &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow noreferrer&quot;&gt;Precision and Recall&lt;/a&gt;. Check whether there are reasonable number of &lt;em&gt;TP,TN&lt;/em&gt; when compared to &lt;em&gt;FN, FP&lt;/em&gt; and if you find those numbers to be fair enough, then F1-score is just a metric for you. There is no benchmark score for an awesome F1 score. The decrease might be just because your model has just more data to predict now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;So overfitting can lead to less F1-score but reduced F1-score does not mean overfitting always.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-01-26T19:50:27.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16569" PostTypeId="1" AcceptedAnswerId="16651" CreationDate="2017-01-27T03:46:45.190" Score="0" ViewCount="1092" Body="&lt;p&gt;I was reading a machine learning book with applications in computer vision. In it, it mentioned that &quot;in vision, it is common to ignore the fact that the intensity of a pixel is quantized and model it with continuous normal distribution.&quot; So my question is why do we choose to do that? &lt;/p&gt;&#xA;" OwnerUserId="28350" LastActivityDate="2017-01-31T17:05:26.823" Title="Modeling pixel intensity with the normal distribution" Tags="|machine-learning|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16570" PostTypeId="1" AcceptedAnswerId="16575" CreationDate="2017-01-27T07:11:48.217" Score="1" ViewCount="417" Body="&lt;p&gt;I am working through an FNN tutorial, right now it outputs a sigmoid probability from 0-1 (0.8956 for example). My own data has 3+ possible targets so i need the output to be a list of the targets and the associated probability for testing on new samples. (In this instance it would be: 0 (probability), 1 (probability)). How would i go about this? Here is the code. Thanks.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;# sigmoid function&#xA;def nonlin(x,deriv=False):&#xA;    if(deriv==True):&#xA;    return x*(1-x)&#xA;    return 1/(1+np.exp(-x))&#xA;&#xA;# input dataset&#xA;X = np.array([  [0,0,1],&#xA;                [0,1,1],&#xA;                [1,0,1],&#xA;                [1,1,1] ])&#xA;&#xA;# output dataset            &#xA;y = np.array([[0,0,1,1]]).T&#xA;&#xA;# seed random numbers to make calculation&#xA;# deterministic (just a good practice)&#xA;np.random.seed(1)&#xA;&#xA;# initialize weights randomly with mean 0&#xA;syn0 = 2*np.random.random((3,1)) - 1&#xA;&#xA;for iter in xrange(10000):&#xA;&#xA;# forward propagation&#xA;l0 = X&#xA;l1 = nonlin(np.dot(l0,syn0))&#xA;&#xA;# how much did we miss?&#xA;l1_error = y - l1&#xA;&#xA;# multiply how much we missed by the &#xA;# slope of the sigmoid at the values in l1&#xA;l1_delta = l1_error * nonlin(l1,True)&#xA;&#xA;# update weights&#xA;syn0 += np.dot(l0.T,l1_delta)&#xA;&#xA;print &quot;Output After Training:&quot;&#xA;print l1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27962" LastActivityDate="2017-01-27T10:12:49.210" Title="Feed forward neural network, output as list of targets and associated probabilities" Tags="|python|neural-network|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16571" PostTypeId="2" ParentId="16542" CreationDate="2017-01-27T08:49:50.620" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Choosing the best set of features is the objective of &lt;strong&gt;feature engineering&lt;/strong&gt;. Feature engineering combines domain expertise and algorithms in order to extract the best set of predictors for a given problem. The process follows these steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Domain experts determine which factors may be useful towards predicting the target variable.&lt;/li&gt;&#xA;&lt;li&gt;These factors are transformed into input features&lt;/li&gt;&#xA;&lt;li&gt;Data is collected&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature selection&lt;/strong&gt; is applied in order to measure the relative importance of each feature to predict the target variable. We can both measure such importance for each feature individually or for combinations of features. &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multiple feature selection methods&lt;/a&gt; exist, so you would have to choose which one is the best fit for your project requirements.&lt;/li&gt;&#xA;&lt;li&gt;Additionally, and as part of the feature selection process, one could create new features by combining sets of raw input features.&lt;/li&gt;&#xA;&lt;li&gt;We check how the features work with the model, and repeat the process from step 1 if necessary by contacting the domain experts again.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As you can see, creating the best set of features is not an automatic process, and domain expertise is very important. This is a very common thing in many data science related tasks, as you will see in the next point.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;There is not such a thing that a perfect mapping between type of problems and algorithms. The model you may end up using will strongly depend on the nature of your data. You would need to explore the data and ask the domain experts in order to make an informed decision on what method to use. Experience will make the process easier with time, but it is likely that you will have to apply different types of models to your problem in order to look for the best one. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-01-27T08:49:50.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16573" PostTypeId="2" ParentId="16511" CreationDate="2017-01-27T09:04:42.507" Score="2" Body="&lt;p&gt;Clustering is the process by which you create groups of similar items so that the difference between the items within groups is minimised and the difference between the items in different groups is maximised. Therefore, you are on the right track, as the previous answer already stated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could start by something simple, like k-means, and then keep improving your model depending on the results (another simple option is hierarchical clustering). However, there are several things that you should take into account. For instance, it may be advisable to normalise/standardise the data before clustering, so that the relative importance of all the features is the same. Additionally, some methods (like for instance k-means) require a priori knowledge of the number of groups in your data. Therefore, if you don't have such knowledge, you will have to decide how to test your clustering results for several numbers of clusters and how will you choose the best model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is not a clustering algorithm, but a dimensionality reduction algorithm. If your data has only three features, I don't really think that you would need PCA beforehand. Clustering methods based on distance metrics do not perform well in case of higher dimensional data (however, some tricks can still be used), but three features is not bad in that regard. However, applying PCA may be useful if you want to visualise the groups and your data. Once again, since you only have three features, just making a 3D plot may be enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just start applying some methods, explore your data, and let the results help you decide what the next step is. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-01-27T09:04:42.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16575" PostTypeId="2" ParentId="16570" CreationDate="2017-01-27T10:12:49.210" Score="2" Body="&lt;p&gt;If you have more than two targets, then you should use &lt;code&gt;softmax&lt;/code&gt; activation instead of sigmoid. Softmax activation gives you the probability associated with each class in the output. The only thing you need to do before applying softmax is to convert your targets into one-hot encoded vectors. If you want to know more about softmax in detail, you can read about it &lt;a href=&quot;http://cs231n.github.io/linear-classify/#softmax&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-01-27T10:12:49.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16576" PostTypeId="2" ParentId="16464" CreationDate="2017-01-27T11:19:43.770" Score="2" Body="&lt;p&gt;The opposite of what you describe is used more often. You start with many features and then prune the neural network at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't believe any python neural network package does what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it's very simple to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, using sklearn, you can just pass &lt;code&gt;warm_start=True&lt;/code&gt;, so that it doesn't reinitialize weights every time you call &lt;code&gt;fit()&lt;/code&gt;, and then you can increase the number of nodes in your feature layer, or any layer you wish.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;from sklearn.neural_network import MLPClassifier&#xA;import numpy as np&#xA;&#xA;X, y = datasets.load_iris(True)&#xA;X = (X-np.mean(X, 0))/np.std(X, 0)  # normalize&#xA;&#xA;nhidden = 10&#xA;m = MLPClassifier(nhidden, max_iter=1000, warm_start=True).fit(X[:, :1], y)&#xA;&#xA;# every 1,000 iterations, throw in a new feature&#xA;for nfeatures in range(2, X.shape[1]+1):&#xA;    m.coefs_[0] = np.r_[m.coefs_[0], np.random.randn(1, nhidden)]&#xA;    m.fit(X[:, :nfeatures], y)&#xA;&#xA;# in the end, we have a neural network trained with all features&#xA;print(m.score(X, y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2017-01-27T11:19:43.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16579" PostTypeId="2" ParentId="14264" CreationDate="2017-01-27T13:44:06.940" Score="0" Body="&lt;p&gt;This is quite old, but I'll try to explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need labelled data if you wish to attempt supervised learning. Assemble a list row with key-value pairs that denote 'events' relevant to your problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;row {&#xA;        company:&amp;lt;some name&amp;gt;, &#xA;        company_attribute_A:&amp;lt;some value&amp;gt;, &#xA;        company_attribute_B:&amp;lt;some value&amp;gt;, &#xA;        company_attribute_C:&amp;lt;some value&amp;gt;, ...,&#xA;        tender_attribute_A:&amp;lt;some value&amp;gt;, &#xA;        tender_attribute_B:&amp;lt;some value&amp;gt;, &#xA;        tender_attribute_C:&amp;lt;some value&amp;gt;, ...,&#xA;        company_participation:&amp;lt;0|1&amp;gt;&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You wish to find out whether company X with attributes A,B,C will participate in a tender with its own attributes, so the training data is used to model the tender behaviour of a certain company by statistically analysing its history.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Historical might only include 'positive' values: events that &lt;strong&gt;did&lt;/strong&gt; happen. An idea might be to artificially add some events that do exist but in which the company did not participate, so as to include 'negative' values as well. For a company Y, you could do it like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take a historical event that DID happen for company X&lt;/li&gt;&#xA;&lt;li&gt;Check if company Y also participated in this event&lt;/li&gt;&#xA;&lt;li&gt;If so: ignore this case (it's also positive for company Y). If not: Add this same row to the list of data but add company Y's attributes to it and change the class value (the binary participation value) to 0.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Try to mimic the actual distribution when adding negative data. Historically speaking, how likely are companies to participate? 50/50? 10/90?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have this data set, you can divide it into training data (80%?), development data (10%?) and test data (10%?).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train your model on the training data. &lt;/li&gt;&#xA;&lt;li&gt;Move your test data somewhere where you can't touch it.&lt;/li&gt;&#xA;&lt;li&gt;Experiment and improve your model by testing it on the development data.&lt;/li&gt;&#xA;&lt;li&gt;Finally, when you're happy with the results or when you have to publish the findings of your model, test it on the withheld set of test data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Honestly, the selection of models and algorithms is secondary. The most important aspect of your problem is the set of attributes (&lt;em&gt;features&lt;/em&gt;) you choose to collect. This is the most human part of data modelling / machine learning, because this is where &lt;strong&gt;you&lt;/strong&gt; decide &lt;strong&gt;what&lt;/strong&gt; you want to teach the machine. How you want to teach it (what &lt;em&gt;algorithm&lt;/em&gt; or &lt;em&gt;model&lt;/em&gt; you'll use) is also important, but not as important as the content. Try using simple models such as kNN, Bayesian flavours or Logistic Regression. This gives you a good overview of how well chosen your attributes are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I would investigate models that output probabilities. You mentioned SciKit. Check out &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this overview&lt;/a&gt; to see which classifiers support probability scores.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2017-01-27T13:44:06.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16580" PostTypeId="1" AcceptedAnswerId="16596" CreationDate="2017-01-27T17:45:17.133" Score="1" ViewCount="9360" Body="&lt;p&gt;I was wondering (about a more semantic question), is there a difference between &lt;em&gt;data-driven&lt;/em&gt; methods and &lt;em&gt;machine learning&lt;/em&gt;? Or is it more correct to state that machine learning is a category of data-driven methods (and what then are other categories)? &lt;/p&gt;&#xA;" OwnerUserId="14372" LastActivityDate="2017-12-16T08:40:04.887" Title="What is the difference between data-driven methods and machine learning?" Tags="|machine-learning|data-mining|data|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16581" PostTypeId="2" ParentId="694" CreationDate="2017-01-27T18:46:50.827" Score="3" Body="&lt;p&gt;DyNet: The Dynamic Neural Network Toolkit. From {1}:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at &lt;a href=&quot;https://github.com/clab/dynet&quot; rel=&quot;nofollow noreferrer&quot;&gt;this http URL&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It was previously known as &lt;a href=&quot;https://github.com/clab/cnn-v1&quot; rel=&quot;nofollow noreferrer&quot;&gt;cnn&lt;/a&gt; (whose python binding was named pycnn).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;{1} Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, Pengcheng Yin. DyNet: The Dynamic Neural Network Toolkit. &lt;a href=&quot;https://arxiv.org/abs/1701.03980&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1701.03980&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-01-27T18:46:50.827" CommentCount="0" CommunityOwnedDate="2017-01-27T18:46:50.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="16584" PostTypeId="1" CreationDate="2017-01-27T22:47:31.503" Score="0" ViewCount="1195" Body="&lt;p&gt;I built a recommendation model on a user-item transactional dataset where each transaction is represented by 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = LightFM(learning_rate=0.05, loss='warp')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here are the results of this recommendation model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train precision at k=3:  0.115301&#xA;Test precision at k=3:  0.0209936&#xA;&#xA;Train auc score:  0.978294&#xA;Test auc score : 0.810757&#xA;&#xA;Train recall at k=3:  0.238312330233&#xA;Test recall at k=3:  0.0621618086561&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me interpret this result? How is it that I am getting such good AUC score and such bad precision/recall? The precision/recall gets even worse for 'bpr' Bayesian personalized ranking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction task:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;users = [0]&#xA;items = np.array([13433, 13434, 13435, 13436, 13437, 13438, 13439, 13440])&#xA;model.predict(users, item)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result from the prediction task:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([-1.45337546, -1.39952552, -1.44265926, -0.83335167, -0.52803332,&#xA;   -1.06252205, -1.45194077, -0.68543684])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I interpret the prediction scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="1284" LastEditorUserId="28194" LastEditDate="2017-01-29T10:40:08.820" LastActivityDate="2017-10-01T08:29:03.423" Title="Interpret results from a lightFM factorization machines" Tags="|machine-learning|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16585" PostTypeId="1" CreationDate="2017-01-28T00:36:50.357" Score="0" ViewCount="378" Body="&lt;p&gt;In theory, Deep Learning NN can predict a class with very few observations.  My problem, I have a class that happens less than 4% of the time.  Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class.  Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive.&#xA;I know that using F Score (with beta &gt;= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow?&#xA;Any suggestions are appreciated.&#xA;By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want.&#xA;My network has 20 inputs 5 hidden layers with 60 unites&#xA;My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is in Mathematica.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; netDrop1 = &#xA;     NetGraph[&amp;lt;| &quot;i1&quot; -&amp;gt; BatchNormalizationLayer[], &#xA;    &quot;l1&quot; -&amp;gt; CatenateLayer[], &quot;l12&quot; -&amp;gt; DropoutLayer[0.2], &quot;l2&quot; -&amp;gt; 60, &#xA;    &quot;l21&quot; -&amp;gt; DropoutLayer[0.5], &quot;l3&quot; -&amp;gt; Tanh, &quot;l22&quot; -&amp;gt; 60, &#xA;    &quot;l221&quot; -&amp;gt; DropoutLayer[0.5], &quot;l33&quot; -&amp;gt; Tanh , &quot;l23&quot; -&amp;gt; 60, &#xA;    &quot;l234&quot; -&amp;gt; BatchNormalizationLayer[0.5] , &quot;l34&quot; -&amp;gt; LogisticSigmoid, &#xA;    &quot;l24&quot; -&amp;gt; 60, &quot;l2435&quot; -&amp;gt; DropoutLayer[0.5], &quot;l35&quot; -&amp;gt; Ramp, &#xA;    &quot;l45&quot; -&amp;gt; 60, &quot;l435&quot; -&amp;gt; DropoutLayer[0.5], &quot;l36&quot; -&amp;gt; Tanh, &#xA;    &quot;l55&quot; -&amp;gt; 60, &quot;l37&quot; -&amp;gt; Tanh, &quot;l4&quot; -&amp;gt; 2, &#xA;    &quot;l5&quot; -&amp;gt;  SoftmaxLayer[]|&amp;gt;, {NetPort[&quot;Input1&quot;] -&amp;gt; &#xA;     &quot;i1&quot; -&amp;gt; &quot;l1&quot; -&amp;gt; &#xA;      &quot;l12&quot; -&amp;gt; &#xA;       &quot;l2&quot; -&amp;gt; &quot;l21&quot; -&amp;gt; &#xA;         &quot;l3&quot; -&amp;gt; &quot;l22&quot; -&amp;gt; &#xA;           &quot;l221&quot; -&amp;gt; &#xA;            &quot;l33&quot; -&amp;gt; &#xA;             &quot;l23&quot; -&amp;gt; &#xA;              &quot;l234&quot; -&amp;gt; &#xA;               &quot;l34&quot; -&amp;gt; &#xA;                &quot;l24&quot; -&amp;gt; &#xA;                 &quot;l2435&quot; -&amp;gt; &#xA;                  &quot;l35&quot; -&amp;gt; &#xA;                   &quot;l45&quot; -&amp;gt; &#xA;                    &quot;l435&quot; -&amp;gt; &#xA;                    &quot;l36&quot; -&amp;gt; &#xA;                    &quot;l55&quot; -&amp;gt;  &quot;l37&quot; -&amp;gt; &quot;l4&quot; -&amp;gt; &quot;l5&quot;, {{NetPort[&#xA;        &quot;MaritalStatus&quot;], NetPort[&quot;Gender&quot;], NetPort[&quot;Input&quot;]} -&amp;gt; &#xA;       &quot;l1&quot;}}, &quot;Input&quot; -&amp;gt; 11, &quot;Input1&quot; -&amp;gt; 2, &#xA;       &quot;MaritalStatus&quot; -&amp;gt; NetEncoder[{&quot;Class&quot;, marital, &quot;UnitVector&quot;}],&#xA;       &quot;Gender&quot; -&amp;gt; NetEncoder[{&quot;Class&quot;, {&quot;M&quot;, &quot;F&quot;}, &quot;UnitVector&quot;}],&#xA;       &quot;Output&quot; -&amp;gt; NetDecoder[{&quot;Class&quot;, {&quot;no&quot;, &quot;yes&quot;}}]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5.&#xA;I have used both Tangent and ReLu (but found little differences in performance between them)&#xA;As you notice, a simple FeedForward Network&#xA;&lt;a href=&quot;https://i.stack.imgur.com/gtj0X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gtj0X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I do under sampling, the under sampling with great success:  the model is able to recall most of the yes, but still predict many negative samples as positive.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  &amp;lt;|{&quot;yes&quot;, &quot;no&quot;} -&amp;gt; 972, {&quot;no&quot;, &quot;no&quot;} -&amp;gt; 1082, {&quot;yes&quot;, &quot;yes&quot;} -&amp;gt; &#xA;  81, {&quot;no&quot;, &quot;yes&quot;} -&amp;gt; 3|&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By the way, I have used these same data,and used RandomForest with a utility function with great results &lt;a href=&quot;http://femvestor.blogspot.com/search?q=geico&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://femvestor.blogspot.com/search?q=geico&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28367" LastEditorUserId="33331" LastEditDate="2018-06-01T16:03:06.593" LastActivityDate="2018-06-01T16:03:06.593" Title="Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow" Tags="|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="16586" PostTypeId="1" AcceptedAnswerId="16593" CreationDate="2017-01-28T06:19:00.657" Score="3" ViewCount="421" Body="&lt;p&gt;I found this review paper by Guyon and Elisseeff in a 2003 JMLR publication but, although not outdated, it is quite old. Is there a more recent review or resource on the topic of feature selection?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;JMLR Review 2003&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28384" LastEditorUserId="14372" LastEditDate="2017-03-31T14:12:32.943" LastActivityDate="2017-03-31T14:12:32.943" Title="Modern Feature Selection Review/Resources" Tags="|machine-learning|feature-selection|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16587" PostTypeId="1" CreationDate="2017-01-28T06:59:18.823" Score="0" ViewCount="121" Body="&lt;p&gt;I started a course in Deep Learning. I'm trying to make an example in order to explain to myself how the weights are found mathematically.&#xA;If what I wrote below is nonsense I'll be glad to hear an explanation. Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for a given image we do &lt;strong&gt;WX+b&lt;/strong&gt;. We get some vector &lt;strong&gt;Y&lt;/strong&gt; and then we compare it to a desired label vector &lt;strong&gt;L&lt;/strong&gt; according to &lt;a href=&quot;https://i.stack.imgur.com/vOsk0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vOsk0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;. I'm assuming that we calculate &lt;strong&gt;D&lt;/strong&gt; with &quot;&lt;strong&gt;Cosine Similarity&lt;/strong&gt;&quot;. For simplicity &lt;strong&gt;S(Y)==Y&lt;/strong&gt;. So what we're trying to do is to calculate &lt;a href=&quot;https://i.stack.imgur.com/cSKDw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cSKDw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; so it will be one.&#xA;Let’s say we have image &lt;strong&gt;X&lt;/strong&gt; of the letter “a” &lt;a href=&quot;https://i.stack.imgur.com/QrMhD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QrMhD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; and two labels (“a”, “b”). Then &lt;a href=&quot;https://i.stack.imgur.com/0dQ0I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0dQ0I.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; . We want to calculate &lt;strong&gt;W&lt;/strong&gt; and &lt;strong&gt;b&lt;/strong&gt; for which we will get such vector &lt;a href=&quot;https://i.stack.imgur.com/iA7Dy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iA7Dy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; that when we’ll insert it into &lt;a href=&quot;https://i.stack.imgur.com/fv9QD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fv9QD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; we’ll get zero. We convert &lt;strong&gt;X&lt;/strong&gt; to a vector&lt;a href=&quot;https://i.stack.imgur.com/uTNmV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uTNmV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;. Since we have 2 labels and size of the &lt;strong&gt;X&lt;/strong&gt; is 9, the &lt;strong&gt;W&lt;/strong&gt; and &lt;strong&gt;b&lt;/strong&gt; are the following: &lt;a href=&quot;https://i.stack.imgur.com/PmVAh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PmVAh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;.  So, we get: &lt;a href=&quot;https://i.stack.imgur.com/mGZor.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mGZor.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;. This gives us the following system of equations: &lt;a href=&quot;https://i.stack.imgur.com/pjEq7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pjEq7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;. So, now we need to solve the following &lt;a href=&quot;https://i.stack.imgur.com/klH97.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/klH97.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If what I wrote above is not nonsense, I don't quite understand where finding minimum is applied?&lt;/p&gt;&#xA;" OwnerUserId="28385" LastEditorUserId="28385" LastEditDate="2017-01-28T19:25:51.363" LastActivityDate="2017-03-01T18:18:51.713" Title="Is my general understanding of finding weights correct?" Tags="|machine-learning|deep-learning|logistic-regression|loss-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16590" PostTypeId="2" ParentId="14289" CreationDate="2017-01-28T15:37:09.093" Score="0" Body="&lt;p&gt;There is a huge amount fo spam tweets in my twitter data collected last year during the election. I have trained a spam classifier using the algorithm, features and training corpus linked below, and included it in my data pipeline. I have tested several clustering algorithms before but my results were not good enough..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure about Chinese or Japanese, but my assumption is only the text preprocessing would be different (e.g. different tokenisation method) and it would not effect the user features which showed the highest performance in the paper (unless Asian spammers spam much differently)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://wrap.warwick.ac.uk/71344/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link to its paper&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.slideshare.net/azubiaga/microposts2015-social-spam-detection-on-twitter&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link to its slides&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2017-01-28T15:37:09.093" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16591" PostTypeId="1" CreationDate="2017-01-28T15:44:43.717" Score="1" ViewCount="70" Body="&lt;p&gt;When using &lt;strong&gt;Batch Normalization&lt;/strong&gt; we can use the &lt;strong&gt;batch mean&lt;/strong&gt; and &lt;strong&gt;batch variance&lt;/strong&gt; during training.&#xA;However this fails to give good enough results in &lt;strong&gt;validation&lt;/strong&gt; case. We get the best results by providing the &lt;strong&gt;population mean&lt;/strong&gt; and &lt;strong&gt;population variance&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this means that the two graphs will be almost identical except the part where we need to subtract the mean and divide the standard deviation (we use the population ones)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our current implementation worked ok but it includes too much &lt;strong&gt;boilerplate code&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note that in other frameworks you can extract and manipulate layers of a model quite easily for such cases, but tensorflow has an only append policy to its graphs&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the optimal way to go in such cases where one graph is slightly different from the other?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-01-28T15:44:43.717" Title="What is the right way to implement the graph of validation process of Batch Normalization in tensorflow?" Tags="|tensorflow|normalization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16593" PostTypeId="2" ParentId="16586" CreationDate="2017-01-28T17:24:44.767" Score="3" Body="&lt;p&gt;I looked into it quite recently and found these papers: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://academic.oup.com/bioinformatics/article/23/19/2507/185254/A-review-of-feature-selection-techniques-in&quot; rel=&quot;nofollow noreferrer&quot;&gt;Saeys et al. (2007) &lt;/a&gt; (also already a bit dated, but a good overview, and I think the 'key' paper in this area). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2766788/pdf/CG-10-365.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dougherty et al. (2009) &lt;/a&gt; (interesting paper discussing various difficulties such as classifier dependence, label distribution and sample size). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0020025514006021&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bolòn-Canedo et al. (2014) &lt;/a&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0957417415000925&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dessì &amp;amp; Pes (2015)&lt;/a&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I do agree with you that there is a need for an updated review regarding this topic. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="14372" LastEditDate="2017-01-30T13:20:27.270" LastActivityDate="2017-01-30T13:20:27.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16596" PostTypeId="2" ParentId="16580" CreationDate="2017-01-28T22:06:26.337" Score="2" Body="&lt;p&gt;Based on the quotation you have added in your comments, data-driven approaches are approaches where you use data that describes past states (&quot;historical data&quot;) to get a (not defined) system to give a desired output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand whether this definition includes machine learning or not we will have to define &quot;machine learning&quot;, and while there could be plenty of ways to define it I expect that it will be quite difficult to come up with a definition that does not include within it &quot;Using a system that, based on given states will give a desired output&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in this last definition I use &quot;given states&quot; and not &quot;past states&quot; as to include approaches such as online learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottom line is that unless you really want to hold to a narrow definition of &quot;past states&quot; it seems that machine learning approaches &lt;strong&gt;are&lt;/strong&gt; a subset of data-driven approaches.&lt;/p&gt;&#xA;" OwnerUserId="28328" LastActivityDate="2017-01-28T22:06:26.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16599" PostTypeId="1" CreationDate="2017-01-29T04:13:14.863" Score="1" ViewCount="1702" Body="&lt;p&gt;I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I own 10 shops and have a certain algorithm that I use to price items, and perhaps I want to test a new algorithm that would result in price changes. There are a few approaches I can see but it isn't clear to me if I'm missing others and would love to hear more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Standard A/B testing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to randomly assign shoppers to test or control buckets then this would obviously be a bad experience for them (given that prices are visible). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Switchback experiment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps instead I decide to run  a switchback experiment, in which the control pricing algorithm runs for 6 hours, then the test pricing algorithm, runs for 6 hours, and so on throughout my test period. This is obviously better than the standard A/B approach but I feel like it still has problems (ie what if the price change results in a change to long term customer behavior but it takes some time for the behavior to change - perhaps the switchback wouldn't capture this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Causal Impact approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach might be to use a &lt;a href=&quot;https://google.github.io/CausalImpact/CausalImpact.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Causal Impact&lt;/a&gt; style analysis, in which I set certain shops as 'control' shops, and other shops as 'test' shops, and then change the algorithm at the test shops and analyze my metric of interest, where we look for a causal effect in the time series at the test shops with respect to the control shops. The drawback here is that you seem highly dependent on how well the controls represent the test shops.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any other suggestions, or even somewhere I can read more about different experimental setups, beyond A/B testing?&lt;/p&gt;&#xA;" OwnerUserId="10462" LastActivityDate="2017-01-29T07:00:03.123" Title="Approaches to A/B testing when you can't randomize on the user level" Tags="|ab-test|experiments|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16600" PostTypeId="2" ParentId="16599" CreationDate="2017-01-29T07:00:03.123" Score="1" Body="&lt;p&gt;I would recommend against switch back as the purchase data is likely to exhibit seasonality (e.g. 2pm - 6pm / tuesdays exhibit lower volume, December exhibit higher volumes, company running advertisements during certain months etc.), which would become an unnecessary exogenous factor that complicates the post implementation analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am inclined towards the causal impact approach. However, there are certain things that should be checked before selecting the stores as control vs. test (it is a kick-start list, please feel free to adapt to your business domain).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Decide on the KPI to be measured as the objective, is that revenue per customer? Profit per customer? Or total profit per store? Is the algorithm designed to increase per customer KPI or more customers?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check that the stores selected have some history to give assurance that they were similar in terms of the selected KPI before the implementation of the tests. Remember to remove outlier customers / transactions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check that enough stores are selected in control vs. test buckets to detect a desired threshold increase that is statistically significant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! :)&lt;/p&gt;&#xA;" OwnerUserId="25800" LastActivityDate="2017-01-29T07:00:03.123" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16601" PostTypeId="1" AcceptedAnswerId="16602" CreationDate="2017-01-29T10:02:56.527" Score="13" ViewCount="8825" Body="&lt;p&gt;Most of the advanced deep learning models like VGG, ResNet, etc. require square images as input, usually with a pixel size of &lt;span class=&quot;math-container&quot;&gt;$224x224$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a reason why the input has to be of equal shape, or can I build a convnet model with say &lt;span class=&quot;math-container&quot;&gt;$100x200$&lt;/span&gt; as well (if I want to do facIAL recognition for example and I have portrait images)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there increased benefit with a larger pixel size, say &lt;span class=&quot;math-container&quot;&gt;$512x512$&lt;/span&gt;?&lt;/p&gt;&#xA;" OwnerUserId="15664" LastEditorUserId="29169" LastEditDate="2019-06-09T16:34:55.397" LastActivityDate="2019-06-09T16:34:55.397" Title="Reason for square images in deep learning" Tags="|deep-learning|image-classification|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16602" PostTypeId="2" ParentId="16601" CreationDate="2017-01-29T10:20:00.380" Score="13" Body="&lt;p&gt;There is no requirement for specific pixel dimensions for convolutional neural networks to function normally. It is likely the values have been chosen for pragmatic reasons - such as a compromise between using image details vs number of parameters and training set size required. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, if source data has a range of different aspect ratios, some portrait, some landscape, with the target object usually in the centre, then taking a square crop from the middle could be a reasonable compromise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you increase the input image size, you will also increase the amount of noise and variance that the network will need to deal with in order to process that input. That could mean more layers - both convolutional and pooling. It could also mean that you need more training examples, and of course each training example will be larger. Together, these increase the computation resources you need to complete training. However, if you can overcome this requirement, it is possible that you will end up with a more accurate model, for any task where the extra pixels could make a difference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible rule of thumb for whether you would want higher resolution is if, for goal of your network, a human expert could make use of the extra resolution and perform better at the task. This might be the case in regression systems, where the network is deriving some numerical quantities from the image - e.g. for face recognition extracting biometrics such as distance between facial features. It might also be desirable for image-processing tasks such as automated masking - state of the art results for these tasks may still be lower resolution than the commercial images where we would like to apply them in practice.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-01-29T10:28:46.870" LastActivityDate="2017-01-29T10:28:46.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16603" PostTypeId="1" CreationDate="2017-01-29T11:34:11.150" Score="3" ViewCount="301" Body="&lt;p&gt;I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and constructing structures out of a document of Python (for example) code, similar in concept to NLP tokenization and higher level language processing, adjusted for programming languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I'm able to find a lot of general NLP related material, it seems to me a better starting point would be previous research specifically focusing on programming languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for tools, resources, academia articles and keywords to search for, basically any help is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="13714" LastActivityDate="2018-02-22T15:34:12.290" Title="What tools are available for programming language parsing for ML?" Tags="|nlp|language-model|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16605" PostTypeId="2" ParentId="15589" CreationDate="2017-01-29T16:38:15.820" Score="0" Body="&lt;p&gt;Similar to one of the earlier, you can also apply the logic of extracting everything starting from (i.e. including) the first numeric digit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;interactor &amp;lt;- c(&quot;ce7380&quot;, &quot;ce7382&quot;, &quot;ce7388&quot;)&#xA;x &amp;lt;- gregexpr(&quot;[0-9]+&quot;, interactor)&#xA;x &amp;lt;- unlist(regmatches(interactor, x))&#xA;x&#xA;## [1] &quot;7380&quot; &quot;7382&quot; &quot;7388&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28194" LastActivityDate="2017-01-29T16:38:15.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16609" PostTypeId="1" AcceptedAnswerId="16617" CreationDate="2017-01-30T04:52:22.863" Score="5" ViewCount="6853" Body="&lt;p&gt;Say I am training a neural network and can fit all my data into memory. Are there any benefits to using mini batches with SGD in this case? Or is batch training with the full gradient always superior when possible? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it seems like many of the more modern optimization algorithms (RMSProp, Adam, etc.) were designed with SGD in mind. Are these methods still superior to standard gradient descent (with momentum) with the full gradient available?&lt;/p&gt;&#xA;" OwnerUserId="28384" LastEditorUserId="28384" LastEditDate="2017-01-30T07:04:36.007" LastActivityDate="2017-01-30T10:12:50.503" Title="Benefits of stochastic gradient descent besides speed/overhead and their optimization" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16611" PostTypeId="2" ParentId="16586" CreationDate="2017-01-30T05:17:25.677" Score="2" Body="&lt;p&gt;I found a 2016 fairly comprehensive review-like paper with an associated python package built on top of scikit-learn. &lt;a href=&quot;https://arxiv.org/pdf/1601.07996v4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Selection: A Data Perspective. Li et al. 2016&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28384" LastEditorUserId="28384" LastEditDate="2017-01-30T17:14:44.333" LastActivityDate="2017-01-30T17:14:44.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16612" PostTypeId="2" ParentId="16553" CreationDate="2017-01-30T06:41:53.123" Score="0" Body="&lt;p&gt;There is a short article here regarding hidden and output activation functions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-10.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Why use activation functions?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of my regression models actually performs best with linear hidden and output activations. Like the article states, a linear function of linear functions is still linear, but in my experience sometimes the data is best modeled as such.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, MIT's &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning Book&lt;/a&gt; has a good breakdown of the various activation functions in &lt;a href=&quot;http://www.deeplearningbook.org/contents/mlp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chapter 6&lt;/a&gt; starting on page 181. &lt;/p&gt;&#xA;" OwnerUserId="28384" LastEditorUserId="28384" LastEditDate="2017-01-30T06:54:47.837" LastActivityDate="2017-01-30T06:54:47.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16613" PostTypeId="1" CreationDate="2017-01-30T07:04:33.703" Score="2" ViewCount="1091" Body="&lt;p&gt;Recently I've started using RMarkdown in RStudio, and I don't know how to adapt the output for a useful presentation. The variables in the data set being: &lt;code&gt;Cia&lt;/code&gt;, &lt;code&gt;Riesgo&lt;/code&gt;, &lt;code&gt;Tipo.Refa&lt;/code&gt;, &lt;code&gt;Estado.Poliza&lt;/code&gt;, &lt;code&gt;Tipo.Endoso&lt;/code&gt;, &lt;code&gt;Asegurado&lt;/code&gt;, &lt;code&gt;Moneda&lt;/code&gt;, &lt;code&gt;Suma.Asegurada&lt;/code&gt;, &lt;code&gt;Prima.Pesos&lt;/code&gt;, &lt;code&gt;Premio.Pesos&lt;/code&gt;, &lt;code&gt;Puntos&lt;/code&gt;, and &lt;code&gt;Pesos&lt;/code&gt;.&#xA;The only thing I want is find a way to present the data in a more understandable format, unlike the images I attached. (dimensions of the R object are 9 rows, 12 variables)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bit of help would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;```{r setup, echo = FALSE}&#xA;knitr::opts_knit$set(root.dir = '/Users/Igna43/Dropbox/Nacho/Nacho 2016/Análisis R')&#xA;```&#xA;&#xA;&#xA;```{r echo = FALSE}&#xA;data1&amp;lt;-read.csv(file = &quot;ListadoEnero2016.csv&quot;, header = TRUE, sep = &quot;,&quot;)&#xA;opneg1&amp;lt;-subset(data1, data1$Pesos. &amp;lt; 0, )&#xA;opneg1&#xA;```&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/P8SbY.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/P8SbY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/grUcF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/grUcF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28435" LastEditorUserId="2961" LastEditDate="2017-01-30T11:26:45.043" LastActivityDate="2017-06-28T18:08:27.457" Title="How to adapt output with multiple columns for better presentation?" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16616" PostTypeId="1" AcceptedAnswerId="16636" CreationDate="2017-01-30T09:24:49.403" Score="3" ViewCount="304" Body="&lt;p&gt;I am working with a real dataset of around 2.5 million short text data (posts, around 10 words in an average data element). I wanted to group similar posts together. So, I applied KMeans clustering on the dataset after doing the standard pre processing. After vectorization (I did TfIdf), the matrix was huge (2.5 million rows and around 60 k columns). I did an SVD + LSA transformation which made it smaller (2.5 million x 150 ). I gave 200 as the number of clusters and applied K Means. I must say that the results are not very bad. There are clusters which have similar posts (not the semantics, but the words similar), but there are those clusters which are randomly placed too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the large size of the data, I am not able to effectively visualize the results or measure the goodness of the clusters. Though I tried implementing silhouette coefficient, elbow methods for finding the optimal number of clusters, I was getting continuous memory errors. I don't know if 150 is the right number to use during dimensionality reduction. Could you please suggest any other methods for solving these issues? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I reached this point, I started doubting the effectiveness of clustering as a method. I am curious to know if anybody is using clustering in large production scale systems, if so, what kind of techniques do people use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please find the code below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;`&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; tfidf_vectorizer = TfidfVectorizer(max_df=0.5, max_features=200000,&#xA;                                 min_df=0.001,decode_error='ignore', stop_words='english',strip_accents='ascii',use_idf=True, ngram_range=(1,2),tokenizer=tokenizeAndFilter,lowercase=True)`&#xA;&#xA;`    tfidf_matrix = tfidf_vectorizer.fit_transform(tokens.values()) #fit the vectorizer to synopses&#xA;&#xA;&#xA;    svd = TruncatedSVD(150)&#xA;    normalizer = Normalizer(copy=False)&#xA;    lsa = make_pipeline(svd, normalizer)&#xA;&#xA;    X = lsa.fit_transform(tfidf_matrix) &#xA;    feature_names = tfidf_vectorizer.get_feature_names()&#xA;&#xA;    with open(path+'terms.csv','w') as f:&#xA;       for col in tfidf_matrix.nonzero()[1]:&#xA;          f.write(feature_names[col]+','+str( tfidf_matrix[0, col]))&#xA;          f.write('\n')&#xA;&#xA;        sil_dict={}&#xA;        for n_clusters in range(50,500,10):&#xA;            km=KMeans(n_clusters=n_clusters,init='k-means++',max_iter=500,n_init=1)&#xA;        km.fit(X)&#xA;            label = km.labels_&#xA;            sil_coeff = silhouette_score(X, label, metric='cosine',n_jobs=10)&#xA;    #        print(&quot;For n_clusters={}, The Silhouette Coefficient is {}&quot;.format(n_cluster, sil_coeff))&#xA;        sil_dict[n_clusters]=sil_coeff&#xA;            with open('kmeans_model'+str(n_clusters)+'.pkl','wb') as f:&#xA;               cPickle.dump(km,f)&#xA;&#xA;&#xA;&#xA;    original_centroids=svd.inverse_transform(km.cluster_centers_)&#xA;    order_centroids=original_centroids.argsort()[:,::-1]&#xA;&#xA;    terms=tfidf_vectorizer.get_feature_names()&#xA;    clusters=collections.defaultdict(list)&#xA;    k=0&#xA;    for i in km.labels_:&#xA;        clusters[i].append(tokens.values()[k])&#xA;        k+=1&#xA;        for i in range(10):&#xA;            print i&#xA;            for j in order_centroids[i, :10]:&#xA;                  print(terms[j] )&#xA;        for key,valueList in clusters.items():&#xA;            with open(path+'\\'+OUTPUTFILE+'_'+str(key)+'.csv','w') as f:&#xA;                 for value in valueList:&#xA;                    f.write(value+'\n')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;`&lt;/p&gt;&#xA;" OwnerUserId="6514" LastActivityDate="2017-02-01T20:03:33.107" Title="Is Clustering used in real world systems/products involving large amounts of data? How are the nuances taken care of?" Tags="|clustering|text-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16617" PostTypeId="2" ParentId="16609" CreationDate="2017-01-30T10:02:14.740" Score="4" Body="&lt;p&gt;On large datasets, SGD can converge faster than batch training because it performs updates more frequently. We can get away with this because the data often contains redundant information, so the gradient can be reasonably approximated without using the full dataset. Minibatch training can be faster than training on single data points because it can take advantage of vectorized operations to process the entire minibatch at once. The stochastic nature of online/minibatch training can also make it possible to hop out of local minima that might otherwise trap batch training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One reason to use batch training is cases where the gradient can't be approximated using individual points/minibatches (e.g. where the loss function can't be decomposed as a sum of errors for each data point). This isn't an issue for standard classification/regression problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't recall seeing RMSprop/Adam/etc. compared to batch gradient descent. But, given their potential advantages over vanilla SGD, and the potential advantages of vanilla SGD over batch gradient descent, I imagine they'd compare favorably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, we have to keep the &lt;a href=&quot;https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization&quot; rel=&quot;noreferrer&quot;&gt;no free lunch theorem&lt;/a&gt; in mind; there must exist objective functions for which each of these optimization algorithms performs better than the others. But, there's no guarantee whether or not these functions pertain to the set of practically useful, real-world learning problems.&lt;/p&gt;&#xA;" OwnerUserId="19014" LastEditorUserId="19014" LastEditDate="2017-01-30T10:12:50.503" LastActivityDate="2017-01-30T10:12:50.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16619" PostTypeId="2" ParentId="16587" CreationDate="2017-01-30T13:32:26.180" Score="0" Body="&lt;p&gt;So I think one important point you need to note, we try to &lt;strong&gt;minimize the error&lt;/strong&gt; and &lt;strong&gt;not the weights&lt;/strong&gt;. I didn't get exactly the number of nodes in each layer you have from your example, but I will give a generic understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So from your example, to obtain cos(θ) you use L which is the result but this value is unknown. You need to first train on the weights and then calculate resultant L. Then check the difference between this resultant L and the actual L. Please separate the knowns and unknowns or input and output for a better understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you should check &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; to actually get a deep understanding. A&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-01-30T13:32:26.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16620" PostTypeId="1" AcceptedAnswerId="16621" CreationDate="2017-01-30T15:26:23.530" Score="8" ViewCount="2308" Body="&lt;p&gt;I'm looking for a good explanation of how convolutions in deep learning work when applied to multi-channel images. For example,  let's say I  have a 100 x 100 pixel image with three channels, RGB. The input tensor would then have dimensions 100 x 100 x 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I apply a convolution with N filters and a stride of one, will the output dimension be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;100 x 100 x 3 x N ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;100 x 100 x N ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, does the convolution that is applied &quot;flatten&quot; the image, or is the convolution applied on a channel by channel basis?&lt;/p&gt;&#xA;" OwnerUserId="28449" LastEditorUserId="29169" LastEditDate="2021-02-16T02:53:15.473" LastActivityDate="2021-02-16T02:53:15.473" Title="Do convolutions &quot;flatten images&quot;?" Tags="|deep-learning|convolutional-neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16621" PostTypeId="2" ParentId="16620" CreationDate="2017-01-30T15:37:03.663" Score="7" Body="&lt;p&gt;In all the implementations for CNNs processing images that I have seen, the output in any layer is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Width x Height x Channels&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or some permutation. This is the same number of dimensions as the input, no additional dimensions are added by the convolutional layers. Each feature map channel in the output of a CNN layer is a &quot;flattened&quot; 2D array created by adding the results of multiple 2D kernels (one for each channel in the input layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually even greyscale input images are expected to be represented as &lt;code&gt;Width x Height x 1&lt;/code&gt; so that they fit the same pattern and the same layer model can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is entirely feasible to build a layer design which converts a standard 2D+channels input layer into a 3D+channels layer. It is not something I have seen done before, but you can never rule out that it could be useful in a specific problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also see 3D+channels convolutions in CNNs applied to video, but in that case, the structure will be some variation of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Width x Height x Frames x Channels&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-01-31T12:46:26.137" LastActivityDate="2017-01-31T12:46:26.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16622" PostTypeId="1" AcceptedAnswerId="16635" CreationDate="2017-01-30T16:07:14.580" Score="1" ViewCount="300" Body="&lt;p&gt;I am analysing the log of a website and I would like to build a classifier to predict the users that are likely to click on an Ad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Ad can be displayed to the visitor several times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To build any classifier I need positive and negative examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The positives are the visitors who clicked on the Ad (easy).&lt;/li&gt;&#xA;&lt;li&gt;The negatives are the visitors who saw the Ad but didn't click (not very obvious).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Is there a convention about how/when to consider a user as a negative example?&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I presume that I should define a threshold of impressions (views) per user, if the user reaches this threshold without clicking on the Ad, I consider him/her as as negative example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any reference or guidance is highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="13891" LastEditorUserId="13891" LastEditDate="2017-01-30T16:52:46.610" LastActivityDate="2017-01-30T23:12:46.753" Title="Ad click prediction: what are the negative examples?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16623" PostTypeId="1" CreationDate="2017-01-30T16:38:18.777" Score="5" ViewCount="5429" Body="&lt;p&gt;In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are learned from scratch. Since users from yesterday don't really differ that much from users today I decided to try to somehow use the old models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Deep Net and SVM models it's easy - I just make the algorithms start with the weights that were obtained last time and make them work through the new dataset using gradient descent.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the XGBoost though? I see that I have two options to use previous data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/tqchen/xgboost/blob/master/demo/guide-python/boost_from_prediction.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Make predictions using the old model and feed them as a parameter to the new model&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/8120/can-i-fine-tune-the-xgboost-model-instead-of-re-training-it&quot;&gt;Use the old booster &lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first one looks like it will make the new model focus too much on the old results, whereas the second one also feels a bit weird: I built 1000 trees and now I boost them with new data - so here it looks like I will end up mostly disregarding the old data and just rebuilding the forest to match the new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought was to simply use half of trees from the old model and half from the new one. This way I would still use old knowledge but balance it out with new data. After some time I would get something like a funny pyramid-shaped forest with very few &quot;old&quot; trees and very many &quot;young&quot; ones (another way to think about it is like a power series model where data from period t-1 is given half the importance that data from period t gets). The idea looks neat, but I haven't seen anyone do that anywhere, so I remain skeptical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any options that I missed? I would be grateful for both comments on the possibilities that I outlined above and new ideas on treating this problem.&lt;/p&gt;&#xA;" OwnerUserId="28450" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2018-11-03T00:01:17.323" Title="What does it mean to &quot;warm-start&quot; XGBoost?" Tags="|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16624" PostTypeId="1" CreationDate="2017-01-30T16:42:51.820" Score="3" ViewCount="6120" Body="&lt;p&gt;There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and uploading pictures of their food, that this should be readily available. Can anyone help? Food101 and UEC are good but limited. Thank you. &lt;/p&gt;&#xA;" OwnerUserId="28446" LastActivityDate="2017-01-30T18:40:17.813" Title="Training data set for food image recognition" Tags="|neural-network|dataset|image-classification|image-recognition|" AnswerCount="2" CommentCount="1" ClosedDate="2017-01-31T10:22:15.833" ContentLicense="CC BY-SA 3.0" />
  <row Id="16625" PostTypeId="1" AcceptedAnswerId="16627" CreationDate="2017-01-30T16:49:56.007" Score="4" ViewCount="1380" Body="&lt;p&gt;I have probably a very simple question. When I convolve an (grayscale) image using some kernel I get some output. The original pixel can be only between 0 and 255. Is possible that the output of a convolution can be higher? Because we are creating feature map, which I understand is another image. Is the output only up to 255, do we scale it down or it does not matter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="28453" LastEditorUserId="28453" LastEditDate="2017-01-30T16:56:00.063" LastActivityDate="2021-05-25T04:56:19.017" Title="Can the output of convolution on image be higher than 255?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16627" PostTypeId="2" ParentId="16625" CreationDate="2017-01-30T17:19:08.830" Score="3" Body="&lt;p&gt;It is possible.  You may have values less than 0 or greater than 255.  It will depend on the values in the kernel.  If you want to display the convolution output properly you will need to scale it first.  This process is referred to as 'normalization&quot;.&lt;/p&gt;&#xA;" OwnerUserId="23144" LastEditorUserId="23144" LastEditDate="2017-01-30T17:49:54.943" LastActivityDate="2017-01-30T17:49:54.943" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16628" PostTypeId="2" ParentId="16624" CreationDate="2017-01-30T18:04:19.200" Score="2" Body="&lt;p&gt;I would suggest Pittsburgh Fast Food Dataset &lt;a href=&quot;http://pfid.rit.albany.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pfid.rit.albany.edu/&lt;/a&gt;&#xA;You might find it limited, but you can join it with the other available datasets.&lt;/p&gt;&#xA;" OwnerUserId="13891" LastActivityDate="2017-01-30T18:04:19.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16629" PostTypeId="2" ParentId="16624" CreationDate="2017-01-30T18:40:17.813" Score="4" Body="&lt;p&gt;You could use the &lt;a href=&quot;https://www.vision.ee.ethz.ch/datasets_extra/food-101/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Food 101 dataset&lt;/a&gt; or &#xA;&lt;a href=&quot;http://foodcam.mobi/dataset256.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;UEC Food 256&lt;/a&gt;, both contains real-world food images.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-01-30T18:40:17.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16631" PostTypeId="2" ParentId="14527" CreationDate="2017-01-30T19:47:47.930" Score="0" Body="&lt;p&gt;LightGBM forum was the answer ;) &lt;a href=&quot;https://github.com/Microsoft/LightGBM/issues/272#issuecomment-276168493&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Microsoft/LightGBM/issues/272#issuecomment-276168493&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;apparently, my model is fitting the result very good, i.e. it is very sure about the class probabilities&lt;/li&gt;&#xA;&lt;li&gt;I did not expect such clear-cut boundaries and thus was confused if this could be correct.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9053" LastEditorUserId="9053" LastEditDate="2017-01-30T23:21:06.327" LastActivityDate="2017-01-30T23:21:06.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16634" PostTypeId="2" ParentId="13175" CreationDate="2017-01-30T23:04:50.497" Score="3" Body="&lt;p&gt;No.  What you want is probably not achievable in practice, because the approach you are considering obfuscates only the weights but &lt;em&gt;does not obfuscate the inputs and outputs to the network&lt;/em&gt;.  In any reasonable ischeme I can imagine, the input $x$ to the obfuscated network will be known and under the attacker's control, and the output $y$ from the obfuscated network will be known to the attacker.  In particular, the attacker can pick any $x$ of his choice, and observe the output $y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is true, your goose is cooked.  The attacker can use the ability to invoke your neural network to learn his own neural network that is just as good as yours.  In particular, the attacker can assemble an arbitrarily large training set by picking many potential inputs $x$ and for each one computing the corresponding $y$ by running your obfuscated network on $x$, then putting the pair $(x,y)$ in the training set.  This doesn't require &lt;em&gt;labelled examples&lt;/em&gt;; it only requires the adversary to be able to put together a large set of &lt;em&gt;unlabelled instances&lt;/em&gt; (and then use your obfuscated network to label those instances).  Typically, it's not too hard for someone to create a large set of unlabelled instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, once the adversary has assembled this training set, the adversary can train their own neural network using standard techniques.  It's likely that the resulting neural network will be approximately as good as yours -- i.e., have approximately the same accuracy.  (This is what seems to happen in practice.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result, no obfuscation scheme for obfuscating the weights is likely to be terribly effective, because that doesn't hide the inputs and outputs to the network.  The most you can hope for is a scheme that acts as a &quot;speed bump&quot; that slightly increases the cost of de-obfuscation or that raises the bar a little bit, but nothing you do will provide strong security against a knowledgeable adversary.  So, don't spend too much time or energy or money on trying to make this work.  Instead, you might be better off looking for other ways to deal with this issue.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;P.S. Even if you could hide the outputs and reveal only the ultimate classification (i.e., hide the continuous probability values from the softmax and just reveal the highest-probability class), that's probably still not enough.  Revealing the class is still enough for an adversary to label a bunch of instances, create a training set, and then train their own network.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-01-30T23:04:50.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16635" PostTypeId="2" ParentId="16622" CreationDate="2017-01-30T23:12:46.753" Score="2" Body="&lt;p&gt;You're overthinking it.  You might not need a threshold.  Start with the simplest approach you possibly can: If you showed the ad to a visitor, then that's a negative example.  Each time you show an ad to a visitor, you end up with an instance, whether positive or negative.  If you've showed the ad three times to the same visitor, you end up with three examples.  (Maybe all negative; or maybe one is positive and two are negative; but that's fine.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonus tip: &lt;em&gt;Do&lt;/em&gt; research methods for handling class imbalance.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-01-30T23:12:46.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16636" PostTypeId="2" ParentId="16616" CreationDate="2017-01-30T23:16:33.417" Score="4" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Clustering (mostly) does not apply to huge data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you cannot automate result evaluation and validation (forget the 'internal' measures, they are useless) nor use the automatic results in any meaningful way (partially because clustering is just way too unreliable for automatic use).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Clustering requires you to study the result, learn, and then decide what to do next. You can't automate this.&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It is &lt;em&gt;explorative&lt;/em&gt;. A tool for &lt;em&gt;you&lt;/em&gt; to study your data; not a tool for machines to automatically predict labels. &lt;strong&gt;Some clusters will be good, some will be bad. This is fine, you got &lt;em&gt;some&lt;/em&gt; good clusters!&lt;/strong&gt; Just don't rely on everything being clustered. &lt;strong&gt;There will always be data that does not belong to any cluster&lt;/strong&gt;. And often, there will be clusters that just don't make any sense to you, or that are too trivial to be of use or interest.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Results just do not improve with data size.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Methods don't scale well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most clustering methods (including k-means) have worst case runtimes of O(n^2) or O(n^3). That is really problematic. But since they are analyzing an NP-hard problem, that is quite decent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess, you &lt;em&gt;want&lt;/em&gt; to work on a sample.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RAM is big these days, and vectors are compact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can usually squeeze a billion vectors into RAM. So anything that - like k-means - works with numerical vectors (and not, say, videos and images) just fits into main memory. No need to go larger than that.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-01-30T23:16:33.417" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16637" PostTypeId="1" CreationDate="2017-01-31T02:08:37.560" Score="2" ViewCount="277" Body="&lt;p&gt;I have started to read more about &lt;a href=&quot;http://people.brandeis.edu/~berkes/software/sfa-tk/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Slow Feature Analysis&lt;/a&gt; and I was wondering how SFA differed from simply taking a moving average?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The linked article suggests, &quot;&lt;em&gt;SFA is an unsupervised algorithm that learns (nonlinear) functions that extract slowly-varying signals from their input data.&lt;/em&gt;&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it is certainly more complicated than just taking a moving average on a signal, but I don't really understand what the benefit or purpose is of the learned non-linear functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be some typical applications of SFA?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2017-07-06T19:48:13.883" Title="What is the difference between Slow Feature Analysis (SFA) and a Moving Average?" Tags="|machine-learning|data-mining|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16638" PostTypeId="2" ParentId="16379" CreationDate="2017-01-31T04:33:02.480" Score="0" Body="&lt;p&gt;Word Embedding for Pos tags can easily be trained using pos tags sequence. There are lot of ways that you can get the trained model. I did it by gensim's word2vec api. Here is the link to it: &#xA;&lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://radimrehurek.com/gensim/models/word2vec.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you want memory efficient solution, radim(creater of gensim) provides a great tutorial: &lt;a href=&quot;https://rare-technologies.com/word2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://rare-technologies.com/word2vec-tutorial/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You just need to pass pos sequence of your training data, resulting vector size, min frequency count etc. You can view api's documentation for more detail.&lt;/p&gt;&#xA;" OwnerUserId="27906" LastActivityDate="2017-01-31T04:33:02.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16639" PostTypeId="1" CreationDate="2017-01-31T05:32:36.167" Score="19" ViewCount="16728" Body="&lt;p&gt;Say you have a dataset with millions of rows and the attributes Plain Text, Key, and Output Ciphertext. Could Deep Learning, theoretically, be used to find patterns in the outputs that help decipher the ciphertext? Are there any other potential approaches?&lt;/p&gt;&#xA;" OwnerUserId="28473" LastActivityDate="2021-06-30T17:34:52.160" Title="Could Deep Learning be used to crack encryption?" Tags="|machine-learning|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16640" PostTypeId="1" AcceptedAnswerId="17913" CreationDate="2017-01-31T06:34:12.980" Score="2" ViewCount="1399" Body="&lt;p&gt;I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data:&#xA;    8000 labelled news articles (Sports:Non-sports::15:85)&#xA;&#xA;Parameters:&#xA;    embedding size = 128&#xA;    vocabulary size = 100000&#xA;    No. of LSTM cell in each layer = 128&#xA;    No. of hidden layers = 2&#xA;    batch size = 16&#xA;    epochs = 10000&#xA;&#xA;Result:&#xA;    AUC on training set = 0.60&#xA;    AUC on testing set = 0.55&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What would be the optimum data size required?&lt;/li&gt;&#xA;&lt;li&gt;Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28475" LastEditorUserId="28475" LastEditDate="2017-01-31T07:21:30.307" LastActivityDate="2018-02-08T20:20:45.763" Title="How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding?" Tags="|classification|rnn|word-embeddings|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16641" PostTypeId="2" ParentId="16639" CreationDate="2017-01-31T07:57:54.937" Score="17" Body="&lt;p&gt;Probably not. Modern encryption systems are designed around cryptographic random number generators, their output is designed to be statistically indistinguishable from true randomness. Machine learning is generally based on discovering statistical patterns in the data, and with truly random data there is none. Even for flawed crypto where there is some small pattern to be found, the large amount of randomness in the input will overwhelm any direct attempt to decrypt the ciphertext.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, there is no heuristic you can use to tell if you are getting close to a correct decryption - a single bit out in a guess at a key for example will completely scramble the output (blame Hollywood for when it shows decryption on screen like some crossword puzzle where the correct letters drop into place). That all-or-nothingness rules out discovering algorithms via a machine learning process, even when you have the encryption key. The best you can do is brute-force all known algorithms. If you don't have the key, then you have to brute-force all possible keys too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could explore how difficult the problem is by attempting to guess the seed value used for a random number generator. Using the Mersenne Twister RNG (the standard one used in e.g. Python), then the input could be the bit pattern for 624 32-bit unsigned integers, and the output could be the 32 bits of the seed used to generate that series. The reason I suggest those specific numbers is because it &lt;em&gt;is&lt;/em&gt; in fact possible to crack Mersenne Twister with that much data. However, I still think that ML approaches would be entirely the wrong tool to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another simple variant would be to see if you can teach a network to either produce or reverse a cryptographic hash. You could start with a known broken one such as MD5. Input and output could be 80 bits, which simplifies the architecture and pipeline enough that you could put together this test in a few hours. Even though MD5 is known to be compromised, I think there is zero chance you could teach a neural network to find any pattern.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;One important detail: If you want to &quot;crack&quot; an encryption, you will not be able to use the key as a known value. Chances are though, that even if you provide the key to an ML process, it will be unable to learn how to decrypt.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-01-31T08:13:29.043" LastActivityDate="2017-01-31T08:13:29.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16642" PostTypeId="2" ParentId="16640" CreationDate="2017-01-31T08:50:52.407" Score="1" Body="&lt;p&gt;There is a major problem of imbalance in your dataset, the classes are in the ratio 15:85. Also the data is lesser, but firstly you will have to address the problem of imbalance. That is probably the root cause of the underfitting of your data. Tweaking some parameters might give you better results, but I dont think you will experience any significant improvement until you address the formerly mentioned isssue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : I mainly use CNNs for text classification, and in experience have always found it performing at par or better than RNNs. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-31T08:50:52.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16643" PostTypeId="1" CreationDate="2017-01-31T09:11:39.080" Score="3" ViewCount="19558" Body="&lt;p&gt;I have achieved 68% accuracy using glm with family = 'binomial' while doing logistic regression in R. I don't have any idea on how to specify the number of iterations through my code. Any suggestions on it? Will application of stochastic gradient descent work for epoch. If so, how do I apply it in my code? I want to increase the accuracy of my model. Following is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rm(list=ls())&#xA;library(dplyr)&#xA;data1 &amp;lt;- read.csv(&quot;~/hj.csv&quot;, head`enter preformatted text here`er=T)&#xA;train&amp;lt;- data1[1:116,]&#xA;VALUE&amp;lt;-as.numeric(rownames(train))&#xA;testset&amp;lt;- data1[1:116,]&#xA;mylogit &amp;lt;- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family =&quot;binomial&quot;)&#xA;testset$predicted.value = predict(mylogit, newdata = testset, type=&quot;response&quot;)&#xA;for (i in 1: nrow(testset)){&#xA;  if(testset$predicted.value[i] &amp;lt;= 0.50)&#xA;    testset$outcome[i] &amp;lt;- 0 &#xA;  else testset$outcome[i] &amp;lt;- 1&#xA;}&#xA;print(testset)&#xA;tab = table(testset$VALUE, testset$outcome) %&amp;gt;% as.matrix.data.frame()&#xA;accuracy = sum(diag(tab))/sum(tab)&#xA;print(accuracy)&#xA;print(tab)&#xA;table(testset$VALUE, testset$outcome)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cfQoh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cfQoh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27231" LastEditorUserId="9123" LastEditDate="2017-02-08T00:44:52.877" LastActivityDate="2017-02-08T02:07:42.933" Title="Increase number of iterations in a logistic regression" Tags="|r|logistic-regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16644" PostTypeId="1" AcceptedAnswerId="16657" CreationDate="2017-01-31T09:28:40.410" Score="7" ViewCount="3829" Body="&lt;p&gt;It looks to me like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Leaky_ReLUs&quot; rel=&quot;noreferrer&quot;&gt;leaky ReLU&lt;/a&gt; should have much better performance since the standard ReLU can’t use half of its space (x &amp;lt; 0 where the gradient is zero). But this doesn't happen and in practice most people use standard ReLU.&lt;/p&gt;&#xA;" OwnerUserId="21319" LastEditorUserId="23305" LastEditDate="2017-02-01T07:15:39.897" LastActivityDate="2018-09-12T18:24:47.720" Title="Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x&lt;0?" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16645" PostTypeId="1" CreationDate="2017-01-31T10:02:58.740" Score="1" ViewCount="37" Body="&lt;p&gt;I'm trying to build a regression model, where I see which attributes are influencing the margin. My data set looks like something below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UserId  |  [products_bought] | revenue |  [places_visited] | discount  --&amp;gt; Margin&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the above mentioned schema a single user might have bought a set of products maximum of 4000 and places visited can be in hundreds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to flatten the data but this is very sparse, are there any approaches to efficiently model this kind of problem &lt;/p&gt;&#xA;" OwnerUserId="18315" LastActivityDate="2017-01-31T10:02:58.740" Title="Modelling query in regression" Tags="|classification|regression|feature-selection|linear-regression|dimensionality-reduction|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16646" PostTypeId="1" CreationDate="2017-01-31T10:03:08.303" Score="1" ViewCount="1669" Body="&lt;p&gt;I'm going through &lt;a href=&quot;http://suriyadeepan.github.io/2016-06-28-easy-seq2seq/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; chatbot example, which uses the Cornell movie dialog corpus. Expanding this example, is it possible to build a &quot;knowledge bot&quot; (ie) a bot that can chat and be knowledgeable in a specific domain? So for example, I use the Bible to train the system. Then it should be able to answer questions on Bible. Or I use the wikia content of Star Wars and be able to answer all the questions related to Star Wars. Is such a system even possible, or is it still in the realm of scifi?&lt;/p&gt;&#xA;" OwnerUserId="13056" LastEditorUserId="29169" LastEditDate="2019-06-07T21:39:56.630" LastActivityDate="2019-06-07T21:39:56.630" Title="Build knowledge bot using deep learning" Tags="|deep-learning|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16647" PostTypeId="2" ParentId="16646" CreationDate="2017-01-31T10:44:55.607" Score="2" Body="&lt;p&gt;It is definitely possible, but obviously to a certain extent. What you need, is a sequence to sequence model trained on questions and answers data of a domain. Denny Britz has &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://www.wildml.com/2016/07/deep-learning-for-chatbots-2-retrieval-based-model-tensorflow/&amp;amp;ved=0ahUKEwiFn_uOmezRAhVKsY8KHYgIDPcQFggiMAA&amp;amp;usg=AFQjCNHHaIIvFcWJw0AKq7fJ_zY3mZXhpQ&amp;amp;sig2=LX5hiHI7zYp69wd2hGtGqA&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; amazing blog post on impelementing a  retreival based chatbot trained on ubuntu dialog corpus using tensorflow. Go through the post to understand the difference between retreival based and generative model chatbot. The efficiency and efficacy of such a bot, is another question all together and rigorous research is going on to improve such bots. IBM watson does it pretty well as mentioned in the comments. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-31T10:44:55.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16648" PostTypeId="2" ParentId="16620" CreationDate="2017-01-31T12:00:54.687" Score="2" Body="&lt;p&gt;It depends on the number of filters you choose. say you have chosen 64 filters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;your weight tensor will be of shape [3, 3, 3, 64] (3rd 3 is number of channels in the input layer and 64 is the number of channels in the output layer ) &#xA;and bias tensor shape [64]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;output will be if Pad = &quot;SAME&quot; and stride 1 for an input image of 224 * 224 * 3 = 224 * 224 * 64 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;output will be if Pad = &quot;Valid&quot; and stride 1 for an input image of 224 * 224 * 3 = 221 * 221 * 64 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now with an input of 221 * 221 * 64 if you want to create 128 filters in the next layer your &#xA;- weight vector shape will be [3, 3, 64, 128] and &#xA;- bias vecotr shape will be [128]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and output shape &#xA;- if Pad = &quot;SAME&quot; [ 221 , 221 , 128 ]&#xA;- if Pad = &quot;VALID&quot; [ 198 , 198 , 128 ] considering stride = [1,1,1,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can check these results while building the graph using layername.get_shape().&lt;/p&gt;&#xA;" OwnerUserId="28482" LastActivityDate="2017-01-31T12:00:54.687" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16649" PostTypeId="1" AcceptedAnswerId="17062" CreationDate="2017-01-31T12:59:33.923" Score="4" ViewCount="1334" Body="&lt;p&gt;I wasn't sure if I should open this question in Cross Validated or here. But since the question belongs to a bigger project related with Data Science, I chose this one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will present a simplified version of my working project, since the original is too complicated and domain specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that we have a timeline of 1 hour (60 minutes). During this period a job starts running and create user notifications in random points. I have written a Monte Carlo simulation to study this process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How well those notifications are spread on the period of 60 minutes?&lt;/li&gt;&#xA;&lt;li&gt;Do we have parts of this timeline that don't contains notifications and are the rest clustered in a specific time?&lt;/li&gt;&#xA;&lt;li&gt;Changing the random functions in the implementation, how the above answers will be affected?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A pseudo code for the Monte Carlo, which mimic the actual code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Repeat one million times&#xA;    number_of_notifications  = get_random_number_of_notifications ()&#xA;    previous_point = 0&#xA;    for i in range(0 to number_of_notifications):&#xA;        interval = get_random_interval()&#xA;        new_point = previous_point + interval&#xA;        previous_point = new_point&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note:&lt;/em&gt; In the current implementation, two or more notifications can be at the same minute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I thought that creating a histogram of the specific minutes during the simulations would help me to answer the first question. But then, I realize that I could have one simulation with all the events in the first half and another one in the second half and the histogram misleading that it is well spread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I thought that it might be nice to plot also the min, max, average and std of the intervals in each simulation. But then, would be enough to answer them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of visualizations should I try to give me insights about the notifications in the Monte Carlo simulation?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-18T01:52:34.537" Title="What visualization I should choose for Monte Carlo simulations in timeline events?" Tags="|visualization|simulation|monte-carlo|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16651" PostTypeId="2" ParentId="16569" CreationDate="2017-01-31T17:05:26.823" Score="2" Body="&lt;p&gt;The first thing to consider is that, even continuous measurements are actually discrete. For instance, temperature is always considered as continuous but every time you process temperature data, these are discrete data to the precision of the sensor used to measure them. Same is true for every physical, measurable quantity (including intensity). That being said, with our modern physics models, it makes no sense to consider that temperatures follow discrete mass probabilities with several thousands possible output and everyone studies them under continuous assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So where do we set the boundary? When do we consider that there are &quot;too many&quot; possible outputs to work with probability mass functions and choose to work with continuous distributions? This is an open question! :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the thing, we do not lose information when processing discrete data in a continuous way so there is no restriction for doing so!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider the pixel intensities of an image, most of the time, we would need to consider discrete mass probabilities with 256 possible outputs, meaning 256 parameters (or weights) to be estimated or tuned. If we model the intensity by a continuous distribution, the number of parameters is drastically reduced! For a Gaussian for instance: 2 parameters. Even using more complex distributions or mixture models, we will not need to tune hundreds of parameters. This is the main advantage! Compact representation!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the choice of the Gaussian specifically, let say that it has been empirically proven to work great. The Central Limit Theorem also supports this common choice. And now, on the top of this, many algorithms are readily available under the assumption that the data follows Gaussian distributions making them even more used by people in the field. Doesn't mean that nothing else could be used but this is the reality of things.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastActivityDate="2017-01-31T17:05:26.823" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16652" PostTypeId="2" ParentId="16616" CreationDate="2017-01-31T17:11:53.180" Score="1" Body="&lt;p&gt;I agree with Anony-Mousse that if you can't easily visualize the resulting clusters and evaluate them manually then this technique is probably not going to be very useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Choosing Number of dimensions to reduce to:&lt;/strong&gt; typically you decide on an acceptable level of information loss in terms of how much total variance of the original data is lost once you execute your dimensionality reduction. Then you make the number of dimensions as small as possible without dropping below this limit.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Choosing Number of clusters:&lt;/strong&gt; You can take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt; for some info on how people assess this. If you take average distance of points from their assigned cluster centers as a metric, then one common approach is to graph this &quot;error&quot; as a function of number of clusters. Obviously it will always get smaller with more clusters but sometimes there is an &quot;elbow&quot; where you stop getting much more &quot;bang for your buck&quot; as you add more clusters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It looks like you're using sklearn, so you could take a closer look at the documentation and User Guide for those tools.&lt;/p&gt;&#xA;" OwnerUserId="28488" LastActivityDate="2017-01-31T17:11:53.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16653" PostTypeId="1" AcceptedAnswerId="16662" CreationDate="2017-01-31T17:15:04.190" Score="1" ViewCount="15656" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nw5lt.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nw5lt.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to convert a json file into a dataframe in pandas (Python). I tried with &lt;code&gt;read_json()&lt;/code&gt; but got the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UnicodeDecodeError:'charmap' codec can't decode byte 0x81 in position 21596351:character maps to &amp;lt;undefined&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think I have some unwanted data in the json file like noise. The data is server generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a collection from the json file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;_id&quot;:{&quot;$oid&quot;:&quot;57a30ce368fd0809ec4d1b41&quot;},&quot;session&quot;:{&quot;start_timestamp&quot;:{&quot;$numberLong&quot;:&quot;1470151881189&quot;},&quot;session_id&quot;:&quot;8356bd90-20160802-153121189&quot;},&quot;metrics&quot;:{},&quot;arrival_timestamp&quot;:{&quot;$numberLong&quot;:&quot;1470152028294&quot;},&quot;event_type&quot;:&quot;OfferViewed&quot;,&quot;event_timestamp&quot;:{&quot;$numberLong&quot;:&quot;1470151943271&quot;},&quot;event_version&quot;:&quot;3.0&quot;,&quot;application&quot;:{&quot;package_name&quot;:&quot;com.think.vito&quot;,&quot;title&quot;:&quot;Vito&quot;,&quot;version_code&quot;:&quot;5&quot;,&quot;app_id&quot;:&quot;7ffa58dab3c646cea642e961ff8a8070&quot;,&quot;cognito_identity_pool_id&quot;:&quot;us-east-1:4d9cf803-0487-44ec-be27-1e160d15df74&quot;,&quot;version_name&quot;:&quot;2.0.0.0&quot;,&quot;sdk&quot;:{&quot;version&quot;:&quot;2.2.2&quot;,&quot;name&quot;:&quot;aws-sdk-android&quot;}},&quot;client&quot;:{&quot;cognito_id&quot;:&quot;us-east-1:1d507b8f-857c-42a4-a705-8db07d46bc8f&quot;,&quot;client_id&quot;:&quot;aa092911-b9a7-498a-82da-76318356bd90&quot;},&quot;device&quot;:{&quot;locale&quot;:{&quot;country&quot;:&quot;US&quot;,&quot;code&quot;:&quot;en_US&quot;,&quot;language&quot;:&quot;en&quot;},&quot;platform&quot;:{&quot;version&quot;:&quot;5.1.1&quot;,&quot;name&quot;:&quot;ANDROID&quot;},&quot;make&quot;:&quot;Xiaomi&quot;,&quot;model&quot;:&quot;Redmi Note 3&quot;},&quot;attributes&quot;:{&quot;Category&quot;:&quot;90000&quot;,&quot;CustomerID&quot;:&quot;4077&quot;,&quot;OfferID&quot;:&quot;11846&quot;}}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28344" LastEditorUserId="25157" LastEditDate="2017-02-01T16:19:02.260" LastActivityDate="2017-02-01T16:45:20.843" Title="Unable to open .json file in pandas" Tags="|python|pandas|json|" AnswerCount="3" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16654" PostTypeId="2" ParentId="16653" CreationDate="2017-01-31T17:28:29.097" Score="0" Body="&lt;pre&gt;&lt;code&gt;import codecs&#xA;import pandas as pd&#xA;pd.read_json(codecs.open('json_file','r','utf-8'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should work. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-01-31T17:28:29.097" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16655" PostTypeId="2" ParentId="16653" CreationDate="2017-01-31T17:53:38.950" Score="-1" Body="&lt;p&gt;You can try this. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;#read line by line&#xA;with open('json_file','rb') as f:&#xA;  entries=f.readlines()&#xA;lines=list(entries)&#xA;Cleaned=[str(line).rstrip() for line in lines]&#xA;#Removes \n&#xA;Json=&quot;[&quot; + ','.join(str(cl) for cl in Cleaned) + &quot;]&quot;&#xA;&#xA;pd.read_json(Json)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="21608" LastEditDate="2017-01-31T19:18:19.430" LastActivityDate="2017-01-31T19:18:19.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16656" PostTypeId="2" ParentId="16346" CreationDate="2017-01-31T18:06:32.573" Score="1" Body="&lt;p&gt;There are many different publicly available datasets out there, and most come with a paper describing how the dataset was acquired. Almost nobody takes a camera and starts taking thousands of pictures themselves. You may find some inspiration by looking at those papers and adapting their methods for finding images.&lt;/p&gt;&#xA;&lt;p&gt;A very popular way is to download the images from &lt;a href=&quot;https://www.flickr.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flickr&lt;/a&gt;: This is a photo platform where users share their photos and add comments or tags, describing the contents of the images.&#xA;Flickr also has an &lt;a href=&quot;https://www.flickr.com/services/api/&quot; rel=&quot;nofollow noreferrer&quot;&gt;API&lt;/a&gt; to find and download images.&lt;/p&gt;&#xA;&lt;p&gt;A couple of test queries show that there are thousands of photos available:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Query            No of Matches&#xA;-------------------------------&#xA;VW Passat             57,702&#xA;Ford Focus           187,344&#xA;Toyota Corolla        81,529&#xA;Mitsubishi Lancer    126,242&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;However, this is not a &lt;em&gt;clean&lt;/em&gt; high-quality dataset: it includes old models, wrong tags, photos from the interiors, and so on. Still, it might be a good starting point to acquire huge numbers of images.&lt;/p&gt;&#xA;&lt;h3&gt;Dataset Cleanup&lt;/h3&gt;&#xA;&lt;p&gt;Maybe you can live with a couple of low-quality images, but I guess the better way is to clean the dataset up. There are a lot of different possible steps - some may not be needed in your case, and you might need other or additional steps:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Remove non-car photos. You probably don't want photos from car interiors, or photos that don't show the car at all. You could e.g. classify all your images with an ImageNet classifier and discard all images that aren't recognized as &amp;quot;car&amp;quot;.&lt;/li&gt;&#xA;&lt;li&gt;Use image retrieval algorithms (e.g. SIFT descriptors and matching) to build a graph containing all images and their similarities, as described in [1]. Discard all images that have very little similarity to the rest of the images (or at least review those images).&lt;/li&gt;&#xA;&lt;li&gt;Manual labelling. This is the best way to ensure that you have a &lt;em&gt;really&lt;/em&gt; high-quality dataset. Have someone go through all images and make sure that they satisfy all conditions you have and are labelled correctly.&#xA;This is &lt;em&gt;very very&lt;/em&gt; expensive, but will definitely give you the best results. If you don't really need to - don't do this.&#xA;If you have to, you could rely on the &lt;a href=&quot;https://www.mturk.com/mturk/welcome&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mechanical Turk&lt;/a&gt; or similar websites.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3&gt;Licensing&lt;/h3&gt;&#xA;&lt;p&gt;Flickr's API description says that:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The Flickr API is available for non-commercial use by outside developers. Commercial use is possible by prior arrangement.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Important note: all photos are property of their respective owners. All images on Flickr have specific license conditions, which you can also query through the API. A list of the available licenses on Flickr is available on &lt;a href=&quot;https://www.flickr.com/creativecommons/&quot; rel=&quot;nofollow noreferrer&quot;&gt;their website&lt;/a&gt;. You &lt;em&gt;have&lt;/em&gt; to make sure you don't infringe the copyrights of the respective owners. Especially if your work is commercial, this complicates things.&lt;/p&gt;&#xA;&lt;h3&gt;References&lt;/h3&gt;&#xA;&lt;p&gt;[1]: Gordo, A., Almazan, J., Revaud, J., &amp;amp; Larlus, D. (2016). Deep Image Retrieval: Learning global representations for image search. arXiv: &lt;a href=&quot;http://arxiv.org/abs/1604.01325&quot; rel=&quot;nofollow noreferrer&quot;&gt;1604.01325&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-01-31T18:06:32.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16657" PostTypeId="2" ParentId="16644" CreationDate="2017-01-31T18:49:27.567" Score="13" Body="&lt;p&gt;One reason that ReL Units have been introduced is to circumvent the problem of vanishing gradients of sigmoidal units at -1 and 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another advantage of ReL Units is that they saturate at exactly 0 allowing for sparse representations, which can be helpful when hidden units are used as input for a classifier. The zero gradient can be problematic in cases where the unit never activates in a gradient based scenario when the unit is initially not activated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem can be alleviated by using leaky ReL Units. On the other hand, leaky ReL Units don't have the ability to create a hard-zero sparse representation which can be useful in certain cases. So, there is a bit of a trade-off and, as in general with NN, it depends on the use cases when which unit performs better. In most cases, if the initial settings can make sure that the ReL Unit is activated (e.g. by setting the biases to small positive values) one would expect ReL and leaky Rel Units to perform very similarly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, leaky RelU (if parametric) introduces another parameter (the slope for $x&amp;lt;0$) that needs to be learned during training and therefore adds more complexity/training time.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-09-12T18:24:47.720" LastActivityDate="2018-09-12T18:24:47.720" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16658" PostTypeId="1" AcceptedAnswerId="16675" CreationDate="2017-01-31T19:59:46.387" Score="4" ViewCount="25026" Body="&lt;p&gt;I would appreciate if you could let me know how to resolve this error:&lt;/p&gt;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)&#xA;y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)&#xA;&#xA;def Ridgecv(alpha):&#xA;    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),&#xA;                           X_train, y_train, 'mae', cv=5).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The error is related to &lt;code&gt;X_train, y_train&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Found input variables with inconsistent numbers of samples: [1052, 1052, 3]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2020-09-24T11:25:43.187" LastActivityDate="2020-09-24T11:25:43.187" Title="Found input variables with inconsistent numbers of samples" Tags="|python|scikit-learn|cross-validation|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="16661" PostTypeId="2" ParentId="16349" CreationDate="2017-01-31T21:11:05.567" Score="0" Body="&lt;p&gt;You can try random search or grid search for better accuracy. Also you can create new models using variable importance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/tune-machine-learning-algorithms-in-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/tune-machine-learning-algorithms-in-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17020" LastActivityDate="2017-01-31T21:11:05.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16662" PostTypeId="2" ParentId="16653" CreationDate="2017-01-31T21:33:49.060" Score="1" Body="&lt;p&gt;You have to read the file line by line, you can find a detailed answer in this &lt;a href=&quot;https://stackoverflow.com/questions/30088006/cant-figure-out-how-to-fix-the-error-in-the-following-code&quot;&gt;question of stackoverflow&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-01-31T21:33:49.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16666" PostTypeId="1" CreationDate="2017-01-31T23:23:30.950" Score="1" ViewCount="132" Body="&lt;p&gt;I have a question about if the modeling of the output detection affects the neural nets capability.&#xA;In my case I want to train a CNN for object recognition and classification.&#xA;As an output I want to get the object classification and a bounding box where it roughly is in the image.&#xA;So my thoughts were to model the output as a sequence of vectors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;c = vector for object classification has the length of o (total count of objects to classyify)&#xA;l = vector for the bounding box with x, y, width, height &#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so the complete sequence would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Output = (c + l) x N&#xA;where N is the maximum count of objects i would like to detect&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does the order of the subsequence &lt;code&gt;(c + l)&lt;/code&gt; have an affect on the effectiveness of the training? Do I have to specify an order, e.g the sequences get ordered by size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly I have another question, with the specific output format I got numbers in range from 0-1 in the vector c, but numbers from range from 0 -the  size of the image in the bounding box vector l. Does this also affect the training if the activation is set to &quot;relu&quot;? Would it be better to normalize the bounding box vector to the image width and height or the input width and height? So all number have an equal range?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These questions came into my mind while labeling the dataset? So I don't want to waste my time labeling the dataset the wrong way :/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your help!&lt;/p&gt;&#xA;" OwnerUserId="28502" LastEditorUserId="29169" LastEditDate="2021-02-16T02:53:06.070" LastActivityDate="2021-02-16T02:53:06.070" Title="Specifying neural network output layout for object detection" Tags="|deep-learning|dataset|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16667" PostTypeId="2" ParentId="16666" CreationDate="2017-02-01T00:09:04.607" Score="1" Body="&lt;p&gt;I'd say how you order your sequences or normalize data will have much less importance than where you get features for those classifier/detection vectors (where you attach those to your base CNN network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your idea falls into approach called &quot;Single Shot Detector&quot; (as opposed to region proposal/sliding windows) and I'd recommend looking at &lt;a href=&quot;https://arxiv.org/abs/1512.02325&quot; rel=&quot;nofollow noreferrer&quot;&gt;SSD&lt;/a&gt; (Liu et al) and &lt;a href=&quot;https://arxiv.org/abs/1612.08242&quot; rel=&quot;nofollow noreferrer&quot;&gt;YOLO&lt;/a&gt; (Redmon, Farhadi) models – there is much more to it to get it performing well.&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2017-02-01T00:09:04.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16673" PostTypeId="2" ParentId="6355" CreationDate="2017-02-01T07:48:03.423" Score="0" Body="&lt;p&gt;Yes, that would  be the sum of the  squares of distances of the response variable from the true or the actual regression line(provided you know it).&lt;/p&gt;&#xA;" OwnerUserId="28456" LastActivityDate="2017-02-01T07:48:03.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16675" PostTypeId="2" ParentId="16658" CreationDate="2017-02-01T11:35:40.060" Score="7" Body="&lt;p&gt;It seems that I missed the word &quot;scoring&quot;. In fact, the extra 3 was related to the number of characters of 'mae'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def Ridgecv(alpha):&#xA;    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),&#xA;                           X_train, y_train, scoring='mae', cv=5).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastActivityDate="2017-02-01T11:35:40.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16676" PostTypeId="1" CreationDate="2017-02-01T12:40:53.313" Score="0" ViewCount="633" Body="&lt;p&gt;I am just trying to assign value to a placeholder but i m am getting error:Cannot feed value of shape () for Tensor 'image_input_22:0', which has shape '(1, 51, 51, 64)'&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;res=np.random.random([1,51,51,64])&#xA;image=tf.placeholder(tf.float32,shape=[1,51,51,64],name=&quot;image_input&quot;)&#xA;reshaped_image=tf.reshape(image,[1,51,51,64])&#xA;init=tf.global_variables_initializer()&#xA;res=0&#xA;with tf.Session() as sess:&#xA; sess.run(init)&#xA; res=sess.run(reshaped_image,feed_dict={image:res})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28086" LastActivityDate="2017-02-01T14:03:03.833" Title="Shape Error in Tensorflow" Tags="|python|neural-network|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16677" PostTypeId="2" ParentId="16676" CreationDate="2017-02-01T14:03:03.833" Score="3" Body="&lt;p&gt;You reinitialized res to 0, which is a scalar, hence the error. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-02-01T14:03:03.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16678" PostTypeId="1" CreationDate="2017-02-01T14:21:13.687" Score="1" ViewCount="55" Body="&lt;p&gt;I'm considering going back to graduate school after a few years working as a market analyst.  I prefer to stay in my current city, where there are two statistics programs I'm interested in.  The first is a more traditional stats program at a private university and the other is a biostats program at a public university.  The biostats program is significantly cheaper, which isn't the sole factor but it is a big one.  I would like to do consulting after grad school (a company like Accenture, for example).  The core curriculum for both is similar - regression analysis, applied modeling, etc - but obviously the Biostats program has classes like epidemiology and survival analysis whereas the traditional program has electives in data mining and visualization.  Would the biostats program be enough to get a solid data science job or should I stick to a traditional program?&lt;/p&gt;&#xA;" OwnerUserId="28521" LastActivityDate="2017-02-01T15:12:03.697" Title="Is training in Biostats sufficient to get into Data Science?" Tags="|statistics|" AnswerCount="1" CommentCount="1" ClosedDate="2017-02-01T16:17:59.700" ContentLicense="CC BY-SA 3.0" />
  <row Id="16679" PostTypeId="2" ParentId="15485" CreationDate="2017-02-01T14:28:11.333" Score="1" Body="&lt;p&gt;For a first pass at this problem, I suggest just use a simple document search/classifier feature set such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag of words&lt;/a&gt;, or maybe &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf&lt;/a&gt; against each full job description, and see what results you can get from a basic classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can train the model based on your girlfriend short-listing or rejecting each item. These features can also be used more or less directly to find and order other documents by degree of similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bag of words model is not very sophisticated. However, it is simple to implement, and has reasonable chance of being trained with limited amount of data. With this project you are more interested in optimising a human's search time than it getting the best possible accuracy. Any accuracy better that random chance should be helpful.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-02-01T14:28:11.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16680" PostTypeId="2" ParentId="16678" CreationDate="2017-02-01T15:12:03.697" Score="0" Body="&lt;p&gt;a question, and then some comments.  First, what kind of market analyst do you want to be?  This field is rapidly becoming specialized, and the answer will affect the decision.  If you want to specialize in the pharma or especially the medical field, the biostat background could be an advantage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking, you need to think outside the box of what you will learn purely as defined by the program, either stats or biostats.  You will need to branch out into areas not defined by either program.  This could include (1) software language, (2) data science problems (Kaggle or external projects run through a campus department), or (3) courses not expressly given in either department, whether it's in &quot;hard&quot; skills like data mining or &quot;soft&quot; skills like marketing.  Ask for exceptions to do things that don't fit the program, but fit your career goals, because being a market analyst doesn't fit squarely into either field.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-02-01T15:12:03.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16681" PostTypeId="2" ParentId="15135" CreationDate="2017-02-01T16:04:55.370" Score="7" Body="&lt;p&gt;Most often you will find yourself not splitting it once but in a first step you will split your data in a training and test set. Subsequently you will perform a parameter search incorporating more complex splittings like cross-validation with a 'split k-fold' or 'leave-one-out(LOO)' algorithm. &lt;/p&gt;&#xA;" OwnerUserId="28528" LastActivityDate="2017-02-01T16:04:55.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16684" PostTypeId="1" AcceptedAnswerId="16688" CreationDate="2017-02-01T18:41:56.600" Score="-1" ViewCount="281" Body="&lt;p&gt;A colleague defines a random variable $X = \frac{Z}{Y^2}$, where $Z$ is a known normal random variable, $Y$ is a known gamma random variable, and $Z$ and $Y$ are independent of each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are not able to get an analytical form for the cumulative distribution function for $X$, but you need to find the approximate 90th percentile of $X$. How do you go about doing this?&lt;/p&gt;&#xA;" OwnerUserId="28437" LastEditorUserId="28437" LastEditDate="2017-02-01T20:51:13.187" LastActivityDate="2017-02-01T22:27:35.880" Title="Gamma random variable , need to find the approximate 90th percentile of X?" Tags="|distribution|" AnswerCount="1" CommentCount="4" ClosedDate="2017-02-04T10:04:31.730" ContentLicense="CC BY-SA 3.0" />
  <row Id="16685" PostTypeId="2" ParentId="155" CreationDate="2017-02-01T19:31:04.257" Score="3" Body="&lt;p&gt;Obviously, there exists a large set of public databases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One not yet mentioned, is from the &lt;strong&gt;FAO&lt;/strong&gt; (Food and Agriculture Organization of the United Nations), accessible at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.fao.org/faostat/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.fao.org/faostat/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It contains data about food production for worldwide countries.&lt;/p&gt;&#xA;" OwnerUserId="25571" LastActivityDate="2017-02-01T19:31:04.257" CommentCount="0" CommunityOwnedDate="2017-02-01T19:31:04.257" ContentLicense="CC BY-SA 3.0" />
  <row Id="16686" PostTypeId="2" ParentId="16616" CreationDate="2017-02-01T20:03:33.107" Score="1" Body="&lt;p&gt;Already great answers have been provided, but I will add my two cents. What worked for me is to take a random sample and make hierarchical clustering. Since it requires a distance matrix to be created, the sample size must be small enough to fit the memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the results, you can choose the optimal number of clusters. Note that you should play a bit with the random sample, distance measure and the way you &lt;strong&gt;standardize&lt;/strong&gt; data (z-score worked best for me, I would recommend to read &lt;a href=&quot;https://pdfs.semanticscholar.org/1d35/2dd5f030589ecfe8910ab1cc0dd320bf600d.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To put it simply, the tinkering with hierarchical clustering could look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;from matplotlib import pyplot as plt&#xA;from scipy.cluster.hierarchy import dendrogram, linkage&#xA;from scipy import stats&#xA;&#xA;# import some data to play with&#xA;iris = datasets.load_iris()&#xA;X = iris.data&#xA;Y = iris.target&#xA;&#xA;X_std = stats.zscore(X)&#xA;&#xA;Z = linkage(X_std,'ward')&#xA;&#xA;# calculate full dendrogram&#xA;plt.figure(figsize=(5, 5))&#xA;plt.title('Hierarchical Clustering Dendrogram')&#xA;plt.xlabel('sample index')&#xA;plt.ylabel('distance')&#xA;dendrogram(&#xA;    Z,&#xA;    leaf_rotation=90.,  # rotates the x axis labels&#xA;    leaf_font_size=8.,  # font size for the x axis labels&#xA;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a great tutorial &lt;a href=&quot;https://joernhees.de/blog/2015/08/26/scipy-hierarchical-clustering-and-dendrogram-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mentioned the silhouette plots - I would do the same as with the hierarchical clustering. Draw random sample large enough to fit the memory and have a look at the plot. Repeat this to see whether you get similar results with different samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, do you have anyone to discuss the amount of clusters with? Since this is not exact method, but merely exploratory as mentioned by others, have business knowledge can be of great value.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-02-01T20:03:33.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16688" PostTypeId="2" ParentId="16684" CreationDate="2017-02-01T22:27:35.880" Score="0" Body="&lt;p&gt;In  R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Z = function(n){rnorm(n)} # normal(0,1) here&#xA;&amp;gt; Y = function(n){rgamma(n,10,10)} # some gamma variable&#xA;&amp;gt; quantile(Z(100000)/Y(100000)^2,.9) # 100000 from your target dist&#xA;     90% &#xA;1.635033 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So 90% of the values are less than 1.635&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-02-01T22:27:35.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16689" PostTypeId="1" CreationDate="2017-02-01T22:34:49.137" Score="3" ViewCount="2186" Body="&lt;p&gt;I am trying to understand what is going on so I built a simpler version of my project. I set the X and the Y to be identical and I'm trying to predict Y using X, this should be very simple, but my setup isn't working. Here is my code : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import keras&#xA;import pandas&#xA;&#xA;&#xA;# I want to evaluate the model when X and Y are the same&#xA;# This should be very easy for the model to evaluate&#xA;X = numpy.random.randint(2, size=100000)&#xA;Y = X&#xA;&#xA;# Setup the model&#xA;model = keras.models.Sequential()&#xA;&#xA;model.add(keras.layers.Dense(1, input_dim=1, init='uniform', activation='relu'   ))&#xA;model.add(keras.layers.Dense(1,              init='uniform', activation='sigmoid'))&#xA;&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;hist = model.fit(X, Y, nb_epoch=10, validation_split=.3, batch_size=10, verbose=1)&#xA;&#xA;df             = pandas.DataFrame()&#xA;df['loss']     = hist.history['loss']&#xA;df['acc']      = hist.history['acc']&#xA;df['val_loss'] = hist.history['val_loss']&#xA;df['val_acc']  = hist.history['val_acc']&#xA;df.index       = df.index + 1&#xA;#&#xA;print(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is my output: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        loss       acc  val_loss   val_acc&#xA;1   0.693162  0.504357  0.693300  0.496233&#xA;2   0.693150  0.503100  0.693250  0.496233&#xA;3   0.693157  0.502357  0.693132  0.503767&#xA;4   0.693171  0.502214  0.693119  0.503767&#xA;5   0.693167  0.502043  0.693121  0.503767&#xA;6   0.693129  0.504014  0.693133  0.503767&#xA;7   0.693167  0.503243  0.693129  0.503767&#xA;8   0.693157  0.502357  0.693181  0.496233&#xA;9   0.693180  0.502614  0.693141  0.503767&#xA;10  0.693170  0.502300  0.693119  0.503767&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I expected that accuracy to go to 100%, but that is not the case. What am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is the example that I was following. &lt;/p&gt;&#xA;" OwnerUserId="20800" LastActivityDate="2017-09-03T08:24:08.933" Title="Keras: X and Y are the same, yet validation accuracy is 50%, what is wrong?" Tags="|machine-learning|neural-network|deep-learning|tensorflow|keras|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16690" PostTypeId="2" ParentId="16689" CreationDate="2017-02-01T23:02:21.527" Score="0" Body="&lt;p&gt;The answer is that the code above works as I thought it should, most of the time. Each run of the program is slightly different due to some randomness, and sometimes this randomness means that the program will not find a link between the X and the Y. The way to fix this is to run the program several times over. After running it 10 times, I got a successful result 8/10 times. &lt;/p&gt;&#xA;" OwnerUserId="20800" LastActivityDate="2017-02-01T23:02:21.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16691" PostTypeId="2" ParentId="16689" CreationDate="2017-02-01T23:06:30.770" Score="3" Body="&lt;p&gt;You should think about how the initial values impact the ReL Units. If, for example, you use &lt;code&gt;init='one'&lt;/code&gt; for the &lt;code&gt;activation='relu'&lt;/code&gt; layer you'll get the desired result (in this simple setup).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-02-01T23:06:30.770" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16692" PostTypeId="1" CreationDate="2017-02-01T23:49:46.850" Score="6" ViewCount="6661" Body="&lt;p&gt;I have a dataframe with about 50 columns. The columns are either categorical or continuous data. The continuous data can be between 0.000001-1.00000 or they can be between 500,000-5,000,000. The categorical data is usually a name, for example a store name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I normalize this data so that I can feed it into a dense layer of a Sequential model?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Y values are either 0 or 1, so it is a binary classification problem. I am currently normalizing all of the continuous data to be 0-1 and one-hot encoding all of the categorical data, so that if I have a column with 5 names it in, I will get a matrix with 5 columns filled with 0's and 1's. Then I join all of the continuous and categorical data and feed it into a Dense layer with &lt;code&gt;init='uniform'&lt;/code&gt; and &lt;code&gt;activation='relu'&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the standard way of doing things? &lt;/p&gt;&#xA;" OwnerUserId="20800" LastEditorUserId="29169" LastEditDate="2019-06-09T15:45:39.730" LastActivityDate="2019-06-09T15:45:39.730" Title="Keras: How to normalize dataframe with continuous and categorical data?" Tags="|neural-network|deep-learning|tensorflow|keras|theano|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16693" PostTypeId="1" AcceptedAnswerId="37571" CreationDate="2017-02-02T00:29:32.877" Score="13" ViewCount="19658" Body="&lt;p&gt;I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the tree and the list of feature importances. What I don't understand is how the feature importance is determined in the context of the tree. For example, here is my list of feature importances:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature ranking:&#xA;1. FeatureA (0.300237)&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureB (0.166800)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureC (0.092472)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureD (0.075009)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureE (0.068310)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureF (0.067118)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureG (0.066510)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureH (0.043502)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureI (0.040281)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureJ (0.039006)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureK (0.032618)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureL (0.008136)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;FeatureM (0.000000)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, when I look at the top of the tree, it looks like this:&lt;a href=&quot;https://i.stack.imgur.com/yJ2xy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yJ2xy.png&quot; alt=&quot;Tree Snippet&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, some of the features that are ranked &quot;most important&quot; don't appear until much further down the tree, and the top of the tree is FeatureJ which is one of the lowest ranked features. My naive assumption would be that the most important features would be ranked near the top of the tree to have the greatest impact. If that's incorrect, then what is it that makes a feature &quot;important&quot;?&lt;/p&gt;&#xA;" OwnerUserId="28544" LastActivityDate="2018-08-29T16:48:39.087" Title="Interpreting Decision Tree in context of feature importances" Tags="|machine-learning|visualization|scikit-learn|data|decision-trees|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16694" PostTypeId="2" ParentId="16693" CreationDate="2017-02-02T01:42:41.827" Score="-2" Body="&lt;p&gt;Variable importance is measured by decrease in model accuracy when the variable is removed. The new decision tree created with the new model without the variable could look very different to the original tree. Splitting decision in your diagram is done while considering all variables in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What variable to split at the root (and other nodes) is measured by impurity. Good purity (e.g: everything in the left branch has the same target value) is not a guarantee for good accuracy. You data might be skewed, your right branch have more responses than your left branch. Therefore, it's no good just correctly classify the left branch, we also need to consider the right branch as well. Therefore, the splitting variable might or might not be an important variable for overall model accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variable importance is a better measure for variable selection.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-02-02T02:55:15.190" LastActivityDate="2017-02-02T02:55:15.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16695" PostTypeId="2" ParentId="16693" CreationDate="2017-02-02T02:19:29.607" Score="2" Body="&lt;p&gt;Just because a node is lower on the tree does not necessarily mean that it is less important. The feature importance in sci-kitlearn is calculated by how purely a node separates the classes (Gini index). You will notice in even in your cropped tree that A is splits three times compared to J's one time and the entropy scores (a similar measure of purity as Gini) are somewhat higher in A nodes than J. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you could only choose one node you would choose J because that would result in the best predictions. But if you were to have the option to have many nodes making several different decisions A would be the best choice.&lt;/p&gt;&#xA;" OwnerUserId="17640" LastActivityDate="2017-02-02T02:19:29.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16697" PostTypeId="2" ParentId="13295" CreationDate="2017-02-02T10:02:26.723" Score="1" Body="&lt;p&gt;You may want to take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hidden Markov Models&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_context-free_grammar&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stochastic Context-Free Grammars&lt;/a&gt;. They both can be used to model grammars and infer them from data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, both have been used in the past to model genetic sequences, e.g.&lt;a href=&quot;https://pdfs.semanticscholar.org/b496/1c20df8e0f04c80fc5c0955cdd0323586803.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://www.biostat.wisc.edu/~craven/776/lecture18.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-02-02T10:02:26.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16699" PostTypeId="1" AcceptedAnswerId="16701" CreationDate="2017-02-02T14:14:16.143" Score="0" ViewCount="3738" Body="&lt;p&gt;I am using the SVM function provided by &lt;code&gt;scikit-learn&lt;/code&gt;. I would like to know whether I need to perform standardization before fitting the model. As I know, LibSVM tends to require pre-processing the data. I am not sure whether scikit-learn automatically normalizes the data instead of expecting us to handle it ourselves.&lt;/p&gt;&#xA;" OwnerUserId="21767" LastEditorUserId="8501" LastEditDate="2017-03-05T15:15:17.700" LastActivityDate="2017-05-10T13:56:58.040" Title="Is standardization needed before using scikit-learn SVM?" Tags="|machine-learning|scikit-learn|svm|libsvm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16700" PostTypeId="1" AcceptedAnswerId="16703" CreationDate="2017-02-02T14:27:35.667" Score="10" ViewCount="64338" Body="&lt;p&gt;I am trying to apply a basic use of the scikitlearn KMeans Clustering package, to create different clusters that I could use to identify a certain activity.  For example, in my dataset below, I have different usage events (0,...,11), and each event has the wattage used and the duration.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the &lt;code&gt;Wattage&lt;/code&gt;, &lt;code&gt;Duration&lt;/code&gt;, and &lt;code&gt;timeOfDay&lt;/code&gt;, I would like to cluster these into different groups to see if I can create clusters and hand-classify the individual activities of each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was having trouble with the KMeans package because I think my values needed to be in integer form. And then, how would I plot the clusters on a scatter plot?  I know I need to put the original datapoints onto the plot, and then maybe I can separate them by color from the cluster?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;km = KMeans(n_clusters = 5)&#xA;myFit = km.fit(activity_dataset)&#xA;&#xA;       Wattage        time_stamp       timeOfDay   Duration (s)&#xA;    0    100      2015-02-24 10:00:00    Morning      30&#xA;    1    120      2015-02-24 11:00:00    Morning      27&#xA;    2    104      2015-02-24 12:00:00    Morning      25&#xA;    3    105      2015-02-24 13:00:00  Afternoon      15&#xA;    4    109      2015-02-24 14:00:00  Afternoon      35&#xA;    5    120      2015-02-24 15:00:00  Afternoon      49&#xA;    6    450      2015-02-24 16:00:00  Afternoon      120&#xA;    7    200      2015-02-24 17:00:00    Evening      145&#xA;    8    300      2015-02-24 18:00:00    Evening      65&#xA;    9    190      2015-02-24 19:00:00    Evening      35&#xA;    10   100      2015-02-24 20:00:00    Evening      45&#xA;    11   110      2015-02-24 21:00:00    Evening      100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;Here is the output from one of my runs of K-Means Clustering.  How do I interpret the means that are zero?  What does this mean in terms of the cluster and the math? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print (waterUsage[clmns].groupby(['clusters']).mean())&#xA;          water_volume   duration  timeOfDay_Afternoon  timeOfDay_Evening  \&#xA;clusters                                                                    &#xA;0             0.119370   8.689516             0.000000           0.000000   &#xA;1             0.164174  11.114241             0.474178           0.525822   &#xA;&#xA;          timeOfDay_Morning  outdoorTemp  &#xA;clusters                                 &#xA;0                       1.0   20.821613  &#xA;1                       0.0   25.636901  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="12202" LastEditDate="2017-02-10T04:25:23.267" LastActivityDate="2017-09-19T01:50:08.917" Title="Confused about how to apply KMeans on my a dataset with features extracted" Tags="|python|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16701" PostTypeId="2" ParentId="16699" CreationDate="2017-02-02T15:39:33.453" Score="5" Body="&lt;p&gt;scikit learn does not standardize data, but it does offer utilities for you to standardize your input data yourself:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/preprocessing.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the rule of thumb is to standardize if your data aren't related.  That is, if channel X is not a function of channel Y, you should standardize&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Qualitatively, think about it this way, SVM 'creates a hyperplane' to separate data into categories; if the data are skewed too far in one axis, that will make it harder to draw a plane to separate them&lt;/p&gt;&#xA;" OwnerUserId="21232" LastActivityDate="2017-02-02T15:39:33.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16703" PostTypeId="2" ParentId="16700" CreationDate="2017-02-02T18:06:21.737" Score="26" Body="&lt;p&gt;For clustering, your data must be indeed integers. Moreover, since k-means is using euclidean distance, having categorical column is not a good idea. Therefore you should also encode the column &lt;code&gt;timeOfDay&lt;/code&gt; into three dummy variables. Lastly, don't forget to standardize your data. This might be not important in your case, but in general, you risk that the algorithm will be pulled into direction with largest values, which is not what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I downloaded your data, put into .csv and made a very simple example. You can see that I am using different dataframe for the clustering itself and then once I retrieve the cluster labels, I add them to the previous one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I omit the variable timestamp - since the value is unique for every record, it will only confuse the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from scipy import stats&#xA;from sklearn.cluster import KMeans&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns&#xA;&#xA;df = pd.read_csv('C:/.../Dataset.csv',sep=';')&#xA;&#xA;#Make a copy of DF&#xA;df_tr = df&#xA;&#xA;#Transsform the timeOfDay to dummies&#xA;df_tr = pd.get_dummies(df_tr, columns=['timeOfDay'])&#xA;&#xA;#Standardize&#xA;clmns = ['Wattage', 'Duration','timeOfDay_Afternoon', 'timeOfDay_Evening',&#xA;         'timeOfDay_Morning']&#xA;df_tr_std = stats.zscore(df_tr[clmns])&#xA;&#xA;#Cluster the data&#xA;kmeans = KMeans(n_clusters=2, random_state=0).fit(df_tr_std)&#xA;labels = kmeans.labels_&#xA;&#xA;#Glue back to originaal data&#xA;df_tr['clusters'] = labels&#xA;&#xA;#Add the column into our list&#xA;clmns.extend(['clusters'])&#xA;&#xA;#Lets analyze the clusters&#xA;print df_tr[clmns].groupby(['clusters']).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This can tell us what are the differences between the clusters. It shows mean values of the attribute per each cluster. Looks like cluster 0 are evening people with high consumption, whilst 1 are morning people with small consumption.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clusters  Wattage     Duration   timeOfDay_Afternoon  timeOfDay_Evening timeOfDay_Morning   &#xA;0         225.000000  85.000000             0.166667           0.833333  0.0 &#xA;1         109.666667  30.166667             0.500000           0.000000  0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You asked for visualization as well. This is tricky, because everything above two dimensions is difficult to read. So i put on scatter plot &lt;code&gt;Duration&lt;/code&gt; against &lt;code&gt;Wattage&lt;/code&gt; and colored the dots based on cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that it looks quite reasonable, except the one blue dot there.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Scatter plot of Wattage and Duration&#xA;sns.lmplot('Wattage', 'Duration', &#xA;           data=df_tr, &#xA;           fit_reg=False, &#xA;           hue=&quot;clusters&quot;,  &#xA;           scatter_kws={&quot;marker&quot;: &quot;D&quot;, &#xA;                        &quot;s&quot;: 100})&#xA;plt.title('Clusters Wattage vs Duration')&#xA;plt.xlabel('Wattage')&#xA;plt.ylabel('Duration')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jpFkN.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jpFkN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2017-02-02T18:15:31.617" LastActivityDate="2017-02-02T18:15:31.617" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="16704" PostTypeId="1" AcceptedAnswerId="24185" CreationDate="2017-02-02T18:48:59.820" Score="7" ViewCount="2544" Body="&lt;p&gt;I'm new to Named Entity Recognition and I'm having some trouble understanding what/how features are used for this task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some papers I've read so far mention features used, but don't really explain them, for example in &#xA;&lt;a href=&quot;https://arxiv.org/pdf/cs/0306050.pdf&quot; rel=&quot;noreferrer&quot;&gt;Introduction to the CoNLL-2003 Shared Task:Language-Independent Named Entity Recognition&lt;/a&gt;, the following features are mentioned: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Main features used by the the sixteen systems that participated in the&#xA;  CoNLL-2003 shared task sorted by performance on the English test data.&#xA;  Aff: affix information (n-grams); bag: bag of words; cas: global case&#xA;  information; chu: chunk tags; doc: global document information; gaz:&#xA;  gazetteers; lex: lexical features; ort: orthographic information; pat:&#xA;  orthographic patterns (like Aa0); pos: part-of-speech tags; pre:&#xA;  previously predicted NE tags; quo: flag signing that the word is&#xA;  between quotes; tri: trigger words.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm a bit confused by some of these, however. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;isn't bag of words supposed to be a method to generate features (one for each word)? How can BOW itself be &lt;em&gt;a feature&lt;/em&gt;? Or does this simply mean we have a feature for each word as in BOW, besides all the other features mentioned?&lt;/li&gt;&#xA;&lt;li&gt;how can a gazetteer be a feature?&lt;/li&gt;&#xA;&lt;li&gt;how can POS tags exactly be used as features ? Don't we have a POS tag for each word? Isn't each object/instance a &quot;text&quot;?&lt;/li&gt;&#xA;&lt;li&gt;what is global document information?&lt;/li&gt;&#xA;&lt;li&gt;what is the feature trigger words?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think all I need here is to just to look at an example table with each of these features as columns and see their values to understand how they really work, but so far I've failed to find an easy to read dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please clarify or point me to some explanation or example of these features being used?&lt;/p&gt;&#xA;" OwnerUserId="28562" LastActivityDate="2017-10-30T02:48:39.900" Title="Named entity recognition (NER) features" Tags="|machine-learning|classification|nlp|feature-selection|named-entity-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16705" PostTypeId="1" CreationDate="2017-02-02T19:25:54.537" Score="2" ViewCount="105" Body="&lt;p&gt;I read &lt;a href=&quot;https://www.kaggle.com/gzuidhof/data-science-bowl-2017/full-preprocessing-tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;this interesting kernel&lt;/a&gt; on kaggle. I learned that you can make a 3D plot starting from CT sections. Is there a way to calculate the volume of the lungs or other structures you plot using 3d_plot? I couldn't find any reference so I need help.&lt;/p&gt;&#xA;" OwnerUserId="15464" LastEditorUserId="31513" LastEditDate="2017-08-09T07:08:36.940" LastActivityDate="2017-08-09T07:08:36.940" Title="Calculate volume from 3d_plot" Tags="|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16706" PostTypeId="1" AcceptedAnswerId="16722" CreationDate="2017-02-02T20:20:49.033" Score="0" ViewCount="4216" Body="&lt;p&gt;I am reading tensorflow documentation now and I can not understand what does &amp;quot;batch&amp;quot; and &amp;quot;batch_size&amp;quot; mean in explanation skip-gram model. Please, can someone explain me?&lt;/p&gt;&#xA;&lt;p&gt;Here this paragraph:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Recall that skip-gram inverts contexts and targets, and tries to predict each context word from its target word, so the task becomes to predict 'the' and 'brown' from 'quick', 'quick' and 'fox' from 'brown', etc. Therefore our dataset becomes&lt;/p&gt;&#xA;&lt;p&gt;(quick, the), (quick, brown), (brown, quick), (brown, fox), ...&lt;/p&gt;&#xA;&lt;p&gt;of (input, output) pairs. The objective function is defined over the entire dataset, but we typically optimize this with stochastic gradient descent (SGD) using one example at a time (or a &lt;strong&gt;'minibatch' of batch_size&lt;/strong&gt; examples, where typically 16 &amp;lt;= &lt;strong&gt;batch_size&lt;/strong&gt; &amp;lt;= 512).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28576" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-03T14:10:35.033" Title="What does &quot;batch&quot; and &quot;batch_size&quot; mean in word2vec skip-gram model?" Tags="|nlp|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16707" PostTypeId="2" ParentId="16693" CreationDate="2017-02-02T21:28:47.393" Score="3" Body="&lt;p&gt;In scikit-learn the feature importance is the decrease in node impurity. The key is that it measures the importance only at a node level. Then, all the nodes are weighted by how many samples reach that node. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if only a few samples end up in the left node after the first split, this might not mean that J is the most important feature because the gain on the left node might only affect very few samples. If you additionally print out the number of samples in each node you might get a better picture of what is going on.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-02-02T21:28:47.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16708" PostTypeId="1" AcceptedAnswerId="16710" CreationDate="2017-02-03T00:11:57.100" Score="2" ViewCount="1805" Body="&lt;p&gt;I am facing an issue that Azure Machine Learning Studio fails to find the &lt;code&gt;to_numeric&lt;/code&gt; method in pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading a .csv in AMLS I try to process it in a python script. The line that is throwing me an error is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataframe1['Monthly Debt'] = pd.to_numeric(dataframe1['Monthly Debt'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;pd of course is pandas, dataframe1 is my working dataframe. The error thrown is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AttributeError: 'module' object has no attribute 'to_numeric'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Of course everything works on my local python. Do you have any idea what AMLS might be talking about?&lt;/p&gt;&#xA;" OwnerUserId="28581" LastEditorUserId="12384" LastEditDate="2017-02-04T10:05:37.387" LastActivityDate="2017-02-04T10:05:37.387" Title="Using Pandas to_numeric() in Azure Machine Learning Studio" Tags="|python|pandas|azure-ml|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16709" PostTypeId="2" ParentId="16708" CreationDate="2017-02-03T00:36:33.810" Score="0" Body="&lt;p&gt;OK this is an issue with Azure Machine Learning Studio. I just confirmed this with one of their data scientists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was using the &lt;strong&gt;Anaconda 2.0/Python 2.7.7&lt;/strong&gt; python version. Here, for some unknown reason the error will appear. If you just use &lt;strong&gt;Anaconda 4.0/Python 2.7.11&lt;/strong&gt; it will work as intended.&lt;/p&gt;&#xA;" OwnerUserId="28581" LastActivityDate="2017-02-03T00:36:33.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16710" PostTypeId="2" ParentId="16708" CreationDate="2017-02-03T00:57:12.790" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.17.0/whatsnew.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;What’s new in Pandas v0.17.0&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;DataFrame.convert_objects&lt;/code&gt; has been deprecated in favor of&#xA;  type-specific functions &lt;code&gt;pd.to_datetime&lt;/code&gt;, &lt;code&gt;pd.to_timestamp&lt;/code&gt; and&#xA;  &lt;code&gt;pd.to_numeric&lt;/code&gt; (&lt;strong&gt;new in 0.17.0&lt;/strong&gt;) (&lt;a href=&quot;https://github.com/pydata/pandas/issues/11133&quot; rel=&quot;nofollow noreferrer&quot;&gt;GH11133&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So for Pandas versions &amp;lt; 0.17.0 you can and should use: &lt;code&gt;df.convert_objects(convert_numeric=True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [213]: x = pd.DataFrame({'a':['11', 'aaa', '0', np.nan, '123']})&#xA;&#xA;In [214]: x&#xA;Out[214]:&#xA;     a&#xA;0   11&#xA;1  aaa&#xA;2    0&#xA;3  NaN&#xA;4  123&#xA;&#xA;In [215]: x.dtypes&#xA;Out[215]:&#xA;a    object&#xA;dtype: object&#xA;&#xA;In [216]: x = x.convert_objects(convert_numeric=True)&#xA;&#xA;In [217]: x&#xA;Out[217]:&#xA;       a&#xA;0   11.0&#xA;1    NaN&#xA;2    0.0&#xA;3    NaN&#xA;4  123.0&#xA;&#xA;In [218]: x.dtypes&#xA;Out[218]:&#xA;a    float64&#xA;dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2017-02-03T10:09:10.307" LastActivityDate="2017-02-03T10:09:10.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16712" PostTypeId="2" ParentId="13990" CreationDate="2017-02-03T06:46:22.567" Score="1" Body="&lt;p&gt;The question mentions three training data characteristics.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Very large&lt;/li&gt;&#xA;&lt;li&gt;Skewed&lt;/li&gt;&#xA;&lt;li&gt;Similarity in values for all but the last dimension&lt;/li&gt;&#xA;&lt;li&gt;Binary final dimension (~95% 1)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The large N is actually a good thing in terms of classification accuracy, but the volume of data may necessitate distributed computing and/or large memory, disk, network, and CPU provisions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The narrow range of the real dimensions and the skew of the binary dimensions do not provide any information necessary to decide what classifier from scikit-learn is best.  If you plot any two dimensions and look at the pictures from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the visual comparison page&lt;/a&gt; you may be able to make a few best guesses yourself and try those first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to getting the best performance from these classifiers is to and normalize what you can in three main ways.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Skew&lt;/li&gt;&#xA;&lt;li&gt;Range&lt;/li&gt;&#xA;&lt;li&gt;Offset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One cannot de-skew the binary dimension since there are no intermediate values and changing a 1 to a 0 makes no sense statistically.  However, if the desired range to use in the classifier is -1 to 1, the data in that 7&lt;sup&gt;th&lt;/sup&gt; dimension can be translated using x&lt;sub&gt;7&lt;/sub&gt;' = 2 x&lt;sub&gt;7&lt;/sub&gt; - 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one of the other dimensions has a skewed distribution and the classifier is known to work best with a normal distribution, it is possible that the data is exponentially distributed, in which case, if it is the 2&lt;sup&gt;nd&lt;/sup&gt; dimension, the general translation might be x&lt;sub&gt;2&lt;/sub&gt;' = ln (x&lt;sub&gt;2&lt;/sub&gt;) / k&lt;sub&gt;1&lt;/sub&gt; - k&lt;sub&gt;0&lt;/sub&gt; where the constants are chosen to translate the distribution data to within the desired ideal range of the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Various non-linear functions that compensate for skew can be used to improve classification significantly and the resulting learned behavior (model) can then be re-expressed using the inverse of these non-linear functions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scatter plots will become clearer indicators of which classifiers are likely to work best after distributions are normalized.&lt;/p&gt;&#xA;" OwnerUserId="28587" LastActivityDate="2017-02-03T06:46:22.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16713" PostTypeId="1" AcceptedAnswerId="16715" CreationDate="2017-02-03T07:58:43.357" Score="1" ViewCount="809" Body="&lt;p&gt;I am completely new to analyze cluster texts, I'm using Goodreads API to get Books synopsis.&#xA;My goal is to group similar books, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Politics&lt;/li&gt;&#xA;&lt;li&gt;Music&lt;/li&gt;&#xA;&lt;li&gt;Biographies&#xA;etc...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;While Goodreads provide genre, I would like to use synopsis and use the text for this.&#xA;Lets say I will get N books synopsis like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;description&amp;gt;&#xA;&amp;lt;![CDATA[&#xA;&amp;lt;b&amp;gt;Alternate cover edition can be found &amp;lt;a href=&quot;https://www.goodreads.com/book/show/10249685-dune&quot; rel=&quot;nofollow&quot;&amp;gt;here&amp;lt;/a&amp;gt;. &amp;lt;/b&amp;gt; and &amp;lt;a href=&quot;https://www.goodreads.com/book/show/11273438-dune&quot; rel=&quot;nofollow&quot;&amp;gt;here&amp;lt;/a&amp;gt;&amp;lt;br /&amp;gt;&amp;lt;br /&amp;gt;Here is the novel that will be forever considered a triumph of the imagination. Set on the desert planet Arrakis, &amp;lt;b&amp;gt;Dune&amp;lt;/b&amp;gt; is the story of the boy Paul Atreides, who would become the mysterious man known as Muad'Dib. He would avenge the traitorous plot against his noble family--and would bring to fruition humankind's most ancient and unattainable dream.&amp;lt;br /&amp;gt;A stunning blend of adventure and mysticism, environmentalism and politics, Dune won the first Nebula Award, shared the Hugo Award, and formed the basis of what it undoubtedly the grandest epic in science fiction.&#xA;]]&amp;gt;&#xA;&amp;lt;/description&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have read about cosine similarity and new google &lt;a href=&quot;https://cloud.google.com/natural-language/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLP&lt;/a&gt;. But I want to start with this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Represent books description (features, usually a bag of words with TF-IDF)&lt;/li&gt;&#xA;&lt;li&gt;Calculate similarity between two books (cosine similarity)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What's the most efficient algorithm to create a matrix of cosine similarity between all books (N)&lt;/li&gt;&#xA;&lt;li&gt;How to &lt;a href=&quot;https://www.quora.com/How-does-Google-News-cluster-stories&quot; rel=&quot;nofollow noreferrer&quot;&gt;cluster&lt;/a&gt; books together based on the above? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any other ideas will be great.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2017-02-03T09:26:50.360" Title="Text processing" Tags="|nlp|text|google|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16714" PostTypeId="1" CreationDate="2017-02-03T08:35:56.120" Score="1" ViewCount="110" Body="&lt;p&gt;In the kaggle forums I found an example model where someone was using XGBRegressor for a binary (0/1) classification problem (sorry, cannot find the link any more).&#xA;This was for a competition where output is measured by AUC, so only ranks mattered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then tried this for a different problem with my data and also found a slight improvement using the Regressor instead of the Classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of my predictions where negative, but it would be no problem to shift them to the [0,1] interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is using regression for binary classification a common approach, and what are the pros and cons?&lt;/p&gt;&#xA;" OwnerUserId="15664" LastEditorUserId="83275" LastEditDate="2019-11-28T20:21:06.200" LastActivityDate="2019-11-28T20:21:06.200" Title="Regression for binary classification and AUC metric" Tags="|machine-learning|classification|regression|auc|" AnswerCount="0" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16715" PostTypeId="2" ParentId="16713" CreationDate="2017-02-03T09:26:50.360" Score="2" Body="&lt;p&gt;Since you are going to use TF-IDF representations, you already have a feature matrix. To calculate cosine similairty between all vectors, you can use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import cosine_similarity&#xA;similarity = cosine_similarity(tfidfmat)&#xA;#tfidfmat is your TF-IDF matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;#Use numpy arrays&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To begin clustering, you can use K-means algorithm to begin with, and use cosine similairty as the distance metric. &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://scikit-learn.org/stable/auto_examples/text/document_clustering.html&amp;amp;ved=0ahUKEwiL546uzfPRAhUFS48KHaPUD_YQFggiMAA&amp;amp;usg=AFQjCNGEotteEEXQ0LYCgdYBkfueBqYdiw&amp;amp;sig2=V5p4Eo89BPexxI8oNVCfGA&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's&lt;/a&gt; an example from scikit-learn itself on clustering documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further things to try:&#xA;If you find the above methods not working to your expectations, look into word2vec and doc2vec, and instead of using tfidf, which isa Bag of Words approach, use word vector representations. &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&amp;amp;ved=0ahUKEwjMrvT1zfPRAhXLu48KHaICBR4QFghuMAk&amp;amp;usg=AFQjCNFRBSxWBWA8Qw1C5rn0pvyHEwjNqw&amp;amp;sig2=-8VAcUKRz8yOOxWpwRwhAg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a good blog explaining the concept. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-02-03T09:26:50.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16717" PostTypeId="1" CreationDate="2017-02-03T11:06:01.103" Score="4" ViewCount="1102" Body="&lt;p&gt;I made a neural network with word embedding, RNN and some dense layers to predict the score of computer game reviews based on the title of the game. I used Keras in Python. I was wondering if it would be somehow possible to transform my model into a generative one that would produce titles with high (or low) predicted score. It could be possible to do brute forcing and run all combinations through my network but maybe there is a smarter solution.&lt;/p&gt;&#xA;" OwnerUserId="5211" LastActivityDate="2017-02-03T12:44:06.583" Title="From regression neural network to generative one" Tags="|neural-network|regression|keras|rnn|text-generation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16718" PostTypeId="1" CreationDate="2017-02-03T12:25:10.417" Score="0" ViewCount="539" Body="&lt;p&gt;I have a dataset of oil temperatures. The time series consist of 100 hours of measurement at every second. There is an anomaly in the data that I would like to detect using Hidden Markov Models (HMM). In Matlab I used &lt;code&gt;hmmgenerate&lt;/code&gt;, however I'm not able to link my data to the assumed probabilities. Instead random states and sequences are generated. Could anyone help me?&lt;/p&gt;&#xA;" OwnerUserId="28598" LastEditorUserId="8878" LastEditDate="2017-02-03T23:55:50.960" LastActivityDate="2017-02-03T23:55:50.960" Title="HMM - Matlab for data set to detect anomaly" Tags="|machine-learning|unsupervised-learning|anomaly-detection|markov-process|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16719" PostTypeId="2" ParentId="16717" CreationDate="2017-02-03T12:29:24.587" Score="4" Body="&lt;p&gt;There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is expected to produce a given output. So I am suggesting a couple of generative approaches that I have seen working, but that will require that you construct and train a new network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can bring in some knowledge about the typical size of network that learns the regression model, but you cannot AFAIK directly re-use the regression model and somehow reverse it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A caveat: Although I have played briefly with both types of generative network, I have never constructed one conditioned on desired goal like the one you want to work with.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;With a purely RNN-based approach, you might do OK by making your network predict the next letter/word in the title - a classifier - whilst taking the title so far (or X characters/words of it) and the rating (normalised) as inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, once trained, you can sample from the RNN randomly to generate new strings. This is the technique used by e.g. Karpathy in his now famous blog &quot;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;noreferrer&quot;&gt;The Unreasonable Effectiveness of Recurrent Neural Networks&lt;/a&gt;&quot;. There are many examples of such sequence sampling generators available to study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also take the output of such a model and see if it matches your regression model from earlier. But I don't think there is much you can do if it does not, except perhaps filter out generated titles if they don't meet expectations - e.g. generate many titles with intended rating of 10, and only display one when your initial model also agrees with a close to 10 rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most relevant Keras example for this I could find is &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py&quot; rel=&quot;noreferrer&quot;&gt;lstm_text_generation.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A RNN-based &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_adversarial_networks&quot; rel=&quot;noreferrer&quot;&gt;generative adversarial network (GAN)&lt;/a&gt; might also be able to achieve what you want. However, please note that GANs are notoriously fiddly to train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A GAN is actually 2 networks. You create a discriminator and a generator and train them in parallel. The generator takes a small completely random vector (e.g. 10 numbers sampled from N(0,1)), plus the rating you want to achieve. Then it generates a text sequence output. The discriminator would take a text input, and a rating, and outputs 1 if it is real, or 0 if it is fake. You present either real training data or output form the generator to the discriminator, and use these to train it. You train the generator based on whether it fools the discriminator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tricky part is to maintain balance between the two components - if either becomes too good relative to the other, training will stall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you can get it to work, you will have a true generative model, which samples from a population space (the noise vector that you input) plus is conditioned on the rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most relevant Keras example for this I could find is &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/mnist_acgan.py&quot; rel=&quot;noreferrer&quot;&gt;mnist_acgan.py&lt;/a&gt; which generates an image, not text sequence, but hopefully should give a start.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-03T12:44:06.583" LastActivityDate="2017-02-03T12:44:06.583" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16720" PostTypeId="2" ParentId="14998" CreationDate="2017-02-03T13:58:21.797" Score="0" Body="&lt;p&gt;Isn't &lt;a href=&quot;https://ipython.org/ipython-doc/3/notebook/public_server.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; solution good enough ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can protect the access with ssh, and the hosted files could be the git repository you want, with different linux (or whatever) user access. You'll need your own server. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-02-03T13:58:21.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16722" PostTypeId="2" ParentId="16706" CreationDate="2017-02-03T14:10:35.033" Score="1" Body="&lt;p&gt;A minibatch, is a group of (input,output) pairs that you present to your neural net in one pass (or epoch), without computing the stochastic gradient descent between two pairs (you only do it at the end of the minibatch, summing the errors over the pairs). This improve the speed and prevent over learning on single elements, thus improving learning. It also doesn't need to much memory since your batches are &quot;mini&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people call stochastic training when your mini-batch has a size of 1, and batch training when it has the size of your training dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get some nice explanations &lt;a href=&quot;https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-03T14:10:35.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16724" PostTypeId="2" ParentId="9832" CreationDate="2017-02-03T20:48:50.143" Score="19" Body="&lt;p&gt;You have it right, the &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; function gives you the value of a state, and &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt; gives you the value of an action in a state (following a given policy &lt;span class=&quot;math-container&quot;&gt;$\pi$&lt;/span&gt;). I found the clearest explanation of Q-learning and how it works in Tom Mitchell's book &quot;Machine Learning&quot; (1997), ch. 13, which is downloadable. &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; is defined as the sum of an infinite series but its not important here. What matters is the &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt; function is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;Q(s,a ) = r(s,a ) + \gamma V^{*}(\delta(s,a))&#xA;$$&lt;/span&gt;&#xA;where V* is the best value of a state if you could follow an optimum policy which you don't know. However it has a nice characterization in terms of &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;     V^{*}(s)= \max_{a'} Q(s,a')&#xA;$$&lt;/span&gt;&#xA;Computing &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt; is done by replacing the &lt;span class=&quot;math-container&quot;&gt;$V^*$&lt;/span&gt; in the first equation to give&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;Q(s, a) = r(s, a) + \gamma \max_{a'} Q(\delta(s, a), a')&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may seem an odd recursion at first because its expressing the Q value of an action in the current state in terms of the best Q value of a &lt;em&gt;successor&lt;/em&gt; state, but it makes sense when you look at how the backup process uses it: The exploration process stops when it reaches a goal state and collects the reward, which becomes that final transition's Q value. Now in a subsequent training episode, when the exploration process reaches that predecessor state, the backup process uses the above equality to update the current Q value of the predecessor state. Next time &lt;em&gt;its&lt;/em&gt; predecessor is visited that state's Q value gets updated, and so on back down the line (Mitchell's book describes a more efficient way of doing this by storing all the computations and replaying them later). Provided every state is visited infinitely often this process eventually computes the optimal Q&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes you will see a learning rate &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; applied to control how much Q actually gets updated: &#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;Q(s, a) = (1-\alpha)Q(s, a) + \alpha(r(s, a) + \gamma \max_{a'} Q(s',a'))&#xA;$$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;= Q(s, a) + \alpha(r(s, a) + \gamma \max_{a'} Q(s',a') - Q(s,a))&#xA;$$&lt;/span&gt;&#xA;Notice now that the update to the Q value &lt;em&gt;does&lt;/em&gt; depend on the current Q value. Mitchell's book also explains why that is and why you need &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt;: its for stochastic MDPs. Without &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt;, every time a state,action pair was attempted there would be a different reward so the Q^ function would bounce all over the place and not converge. &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is there so that as the new knowledge is only accepted in part. Initially &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is set high so that the current (mostly random values) of Q are less influential. &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is decreased as training progresses, so that new updates have less and less influence, and now Q learning converges&lt;/p&gt;&#xA;" OwnerUserId="28611" LastEditorUserId="28611" LastEditDate="2019-02-23T23:07:39.477" LastActivityDate="2019-02-23T23:07:39.477" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16728" PostTypeId="1" AcceptedAnswerId="16749" CreationDate="2017-02-04T03:17:18.067" Score="3" ViewCount="44408" Body="&lt;p&gt;I am trying to perform a comparison between 5 algorithms against the KDD Cup 99 dataset and the NSL-KDD datasets using Python and I am having an issue when trying to build and evaluate the models against the KDDCup99 dataset and the NSL-KDD dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whenever I try to run the algorithms on the datasets I get the following error &lt;strong&gt;'could not convert string to float: S0'&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This error is produced during the during the evaluation of the 5 models; Logistic Regression, Linear Discriminant Analysis, K-Nearest Neighbors, Classification and Regression Trees, Gaussian Naive Bayes and Support Vector Machines. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code that I am using to evaluate the datasets: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Load KDD dataset&#xA;&#xA;dataset = pandas.read_csv('Datasets/KDDCUP 99/kddcup.csv', names = ['duration','protocol_type','service','src_bytes','dst_bytes','flag','land','wrong_fragment','urgent',&#xA;'hot','num_failed_logins','logged_in','num_compromised','root_shell','su_attempted','num_root','num_file_creations',&#xA;'num_shells','num_access_files','num_outbound_cmds','is_host_login','is_guest_login','count','serror_rate',&#xA;'rerror_rate','same_srv_rate','diff_srv_rate','srv_count','srv_serror_rate','srv_rerror_rate','srv_diff_host_rate',&#xA;'dst_host_count','dst_host_srv_count','dst_host_same_srv_rate','dst_host_diff_srv_rate','dst_host_same_src_port_rate',&#xA;'dst_host_srv_diff_host_rate','dst_host_serror_rate','dst_host_srv_serror_rate','dst_host_rerror_rate','dst_host_srv_rerror_rate','class'])&#xA;&#xA;&#xA;# split data into X and y&#xA;array = dataset.values&#xA;X = array[:,0:41]&#xA;Y = array[:,41]&#xA;&#xA;# Split-out validation dataset&#xA;validation_size = 0.20&#xA;seed = 7&#xA;X_train, X_validation, Y_train, Y_validation = cross_validation.train_test_split(X, Y, test_size=validation_size, random_state=seed)&#xA;&#xA;# Test options and evaluation metric&#xA;num_folds = 7&#xA;num_instances = len(X_train)&#xA;seed = 7&#xA;scoring = 'accuracy'&#xA;&#xA;#  Algorithms&#xA;models = []&#xA;models.append(('LR', LogisticRegression()))&#xA;models.append(('LDA', LinearDiscriminantAnalysis()))&#xA;models.append(('KNN', KNeighborsClassifier()))&#xA;models.append(('CART', DecisionTreeClassifier()))&#xA;models.append(('NB', GaussianNB()))&#xA;models.append(('SVM', SVC()))&#xA;&#xA;# evaluate each model in turn&#xA;results = []&#xA;names = []&#xA;for name, model in models:&#xA;    kfold = cross_validation.KFold(n=num_instances, n_folds=num_folds, &#xA;&#xA;random_state=seed)&#xA;&#xA;    #Here is where the error is spit out&#xA;{&#xA;            cv_results = cross_validation.cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring) # Could not convert string to float happens here. Scoring uses string. &#xA;            results.append(cv_results)&#xA;            names.append(name)&#xA;            msg = &quot;%s: %f (%f)&quot; % (name, cv_results.mean()*100, cv_results.std()*100)#multiplying by 100 to show percentage&#xA;            print(msg)&#xA;}&#xA;&#xA;# Compare Algorithms&#xA;fig = plt.figure()&#xA;fig.suptitle('Algorithm Comparison')&#xA;ax = fig.add_subplot(111)&#xA;plt.boxplot(results)&#xA;ax.set_xticklabels(Y)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a 3 line sample from the KDDcup99 datatset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0   tcp http    SF  215 45076   0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   0   0   normal.&#xA;0   tcp http    SF  162 4528    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   2   2   0   0   0   0   1   0   0   1   1   1   0   1   0   0   0   0   0   normal.&#xA;0   tcp http    SF  236 1228    0   0   0   0   0   1   0   0   0   0   0   0   0   0   0   0   1   1   0   0   0   0   1   0   0   2   2   1   0   0.5 0   0   0   0   0   normal.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried using label encoding and it still spits out the same error and when I was looking through the sklearn websites, I noticed that the scoring value was for the string type, is this the cause of the issue? and if not, is there a problem with the way I have loaded the dataset? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; I tried removing scoring value from the code and still got the same error. &lt;/p&gt;&#xA;" OwnerUserId="18129" LastEditorUserId="18129" LastEditDate="2017-02-05T00:26:16.177" LastActivityDate="2019-11-19T12:19:17.337" Title="Could not convert string to float error on KDDCup99 dataset" Tags="|machine-learning|python|scikit-learn|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16729" PostTypeId="1" CreationDate="2017-02-04T03:46:45.427" Score="7" ViewCount="9860" Body="&lt;p&gt;In a neural network, each neuron value is multiplied by the weight of the connection. Then, each neuron's input is the sum of all those values, on which we apply the activation function (sigmoid, relu, tanh, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why can't the input be a vector of all those values, so that I can decide to multiply them if I want, or apply another function from the vector space to the reals, and then apply the activation function on that real value? Any implementation I have seen does not allow this, and if I wanted to do it, it seem I would have to implement my whole neural network architecture by myself without the possibility of using Theano, Keras or other libraries. Is there a reason that escapes me of why this would not work? And if there isn't, is there any library out there I could use that already does this without having to create it from scratch?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2019-05-22T07:50:35.153" Title="Why neural networks models do not allow for multiplication of inputs?" Tags="|neural-network|deep-learning|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16731" PostTypeId="2" ParentId="16729" CreationDate="2017-02-04T05:56:57.370" Score="1" Body="&lt;p&gt;This is the standard definition of matrix multiplication. Summing those values I mean. You don't have to make it work like that, you can in fact multiply them if you like. Simply do a series of scalar multiplications instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Input vector&#xA;X = tf.placeholder(&quot;float32&quot;, (-1, 30)) # so a 1x30 input vector&#xA;&#xA;# &quot;Layer 1&quot;&#xA;w_l1 = []&#xA;for n in range(50):&#xA;  w_l1.append(tf.Variable(tf.random_normal((1,))))&#xA;&#xA;l1 = []&#xA;for n in range(50):&#xA;  l1.append(X * w_l1[n])&#xA;&#xA;l1_out = l1[0]&#xA;for n in range(1, 50):&#xA;  l1_out *= l1[n]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this close to what you mean ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, I think most of what you want to achieve can be done with the reduce_prod method in tensorflow, and theano has an analogous function. I think it isn't done in practice because it doesn't form a ring over Tensors.&lt;/p&gt;&#xA;" OwnerUserId="28586" LastActivityDate="2017-02-04T05:56:57.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16733" PostTypeId="2" ParentId="14251" CreationDate="2017-02-04T06:24:21.187" Score="1" Body="&lt;p&gt;An interesting one is to slice the image using energy-measurements. The idea is to separate letters. You subdivide the image by having lines &quot;move&quot; across it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every line is like a &quot;lightning strike&quot; in the sense that it follows the path of least resistance. There's a general direction, for instance top to bottom. You draw it by choosing moving a point on the top of the image, and you go down. Every step downward it either goes straight down, or one pixel down and one to the left, or one pixel down and one to the right. It picks the step with the least energy, which means minimum color change. You can use straight euclidean distance, and you'll be amazed how well this deals with things like gradient backgrounds, but it works even better to use information metrics. Another slight improvement can be had by adding a small cost to horizontal offsets, so the lines go more-or-less, but not quite, straight down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You keep subdividing the image, and then subdividing the subdivisions until you have to accept quite big jumps in energy, while enforcing a minimum subdivision size. This should &quot;mostly&quot; give you individual letters and those are easy to OCR.&lt;/p&gt;&#xA;" OwnerUserId="28586" LastActivityDate="2017-02-04T06:24:21.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16735" PostTypeId="2" ParentId="16099" CreationDate="2017-02-04T09:02:10.773" Score="0" Body="&lt;p&gt;On the one hand, as &lt;code&gt;Himanshu Rai&lt;/code&gt; has mentioned, if you are using &lt;code&gt;RandomForestRegressor&lt;/code&gt; for classification tasks, the &quot;probability&quot; you get might greater than 1, or less than 0. That's not what we expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;code&gt;RandomForestClassifier&lt;/code&gt; uses &lt;code&gt;accuracy_score&lt;/code&gt; as loss function, while &lt;code&gt;RandomForestRegressor&lt;/code&gt; uses &lt;code&gt;r2_score&lt;/code&gt;. That's a big difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And some advice&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.score&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.score&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor.score&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor.score&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T09:02:10.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16736" PostTypeId="1" AcceptedAnswerId="16737" CreationDate="2017-02-04T09:21:39.950" Score="0" ViewCount="396" Body="&lt;p&gt;I'm working in the medical field and I'd like to learn applications of CNN for image recognition and classification. All the (few) things I learned come from self-learning on the web or sparse books. I'm studying now Tensorflow for CNN implementation but I'm having trouble finding clear documentation for my actual level, so I think I'm missing the basic knowledge in order to understand this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm at a basic level of python programming, I have better understanding of classical machine learning algorithms, which resources should I learn in order to get a good grasp of the argument? Is there such an ideal pathway to this?&lt;/p&gt;&#xA;" OwnerUserId="15464" LastActivityDate="2017-02-04T09:41:23.620" Title="Roadmap to learn CNN in tensorflow from scratch" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16737" PostTypeId="2" ParentId="16736" CreationDate="2017-02-04T09:41:23.620" Score="2" Body="&lt;p&gt;I'm assuming you're talking about medical images classification, rather than localization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I recommend &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;, it has an awesome forum and people share their codes and opinions there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can start at &lt;a href=&quot;https://www.kaggle.com/c/digit-recognizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Digit Recognizer&lt;/a&gt;, it's actually the well-known MNIST dataset(hand-written numbers). There's no relationship between MNIST and medical field. However there are some common techniques and tricks as they are both image recognition/classification problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you encounter problems, read the other ones' codes at the &lt;code&gt;kernel&lt;/code&gt; section, especially those with upvotes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meanwhile, comment section is a good place to learn. I learned a lot there when I started to learn CNN.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T09:41:23.620" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16738" PostTypeId="2" ParentId="16692" CreationDate="2017-02-04T09:50:42.733" Score="6" Body="&lt;p&gt;Yes it does, you're doing well!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases, categorical features(columns) should be one-hot encoded. However, continuous features might be a little complicated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two common ways to preprocess continuous feature:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;scaling features to range [0, 1] (as you have done)&lt;/li&gt;&#xA;&lt;li&gt;removing the mean and scaling to unit variance(make the feature has zero mean and 1 standard variance)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my practice, I take these two ways depending on my dataset.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T09:50:42.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16739" PostTypeId="2" ParentId="16446" CreationDate="2017-02-04T10:10:59.600" Score="3" Body="&lt;p&gt;Unfortunately you can't use GTX 580+Tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow GPU support requires having a GPU card with NVidia Compute Capability &gt;= 3.0, while GTX 580 has only 2.0 compute capability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;a href=&quot;https://github.com/BVLC/caffe&quot; rel=&quot;nofollow noreferrer&quot;&gt;caffe&lt;/a&gt; supports CUDA compute capability 2.0. You can have a try.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/versions/r0.10/get_started/os_setup#prepare_environment_for_linux&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/versions/r0.10/get_started/os_setup#prepare_environment_for_linux&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://developer.nvidia.com/cuda-gpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://developer.nvidia.com/cuda-gpus&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T10:10:59.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16740" PostTypeId="2" ParentId="16729" CreationDate="2017-02-04T10:24:30.267" Score="6" Body="&lt;p&gt;Using a low-level library, such as Theano or TensorFlow, it is likely that you can construct new schemes for reducing tensors (maybe via some learnable weight vector etc). In TensorFlow, you also get automated gradient calculations, so you should be able to still define a cost function and use existing optimisers, without needing to analyse the new design yourself or re-write back propagation formulae.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;noreferrer&quot;&gt;The universal approximation theorem&lt;/a&gt; essentially states that in order to have a network that &lt;em&gt;could&lt;/em&gt; learn a specific function, you don't need anything more than one hidden layer using the standard matrix multiplying &#xA;plus non-linear activation function. So, invention and use of new architectures needs some reason. The reason is usually that these additions improve the speed or scope of what can be learned, they generalise better from training data, or they model something from a problem domain really well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No doubt there have been explorations of all sorts of variations to the standard NN model over time. The ones that have become popular have all proven themselves on some task or other, and often have associated papers demonstrating their usefulness. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example Convolutional Neural Networks have proven themselves good at image-based tasks. The design of a 2-dimensional CNN layer has a logical match to how pixels in an image relate to each other locally - defining edges, textures etc, so the architecture in the model nicely matches to some problem domains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can find a good match from your vector multiplication model to a specific problem domain, then that's an indication that it may be worth implementing in order to test that theory. If you just want to explore other NN structures to see whether they work, you can still do so, but without a specific target you will be searching for a problem to solve (unless by chance you stumble upon something generally useful that has been overlooked before).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address the question in the title:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why neural networks models do not allow for multiplication of inputs?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Some low-level ones do (e.g. TensorFlow). However, this is not an architecture that has proven itself useful, so authors of higher-level libraries (e.g. Keras) have no reason to implement or support it. There is the possibility that this situation is an oversight, and the idea is generally useful. In which case, once someone can show this you will find it would get support across actively-developed libraries pretty quickly since it seems straightforward to implement.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-02-04T10:24:30.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16741" PostTypeId="2" ParentId="16548" CreationDate="2017-02-04T10:29:16.867" Score="4" Body="&lt;p&gt;You made some mistakes on &lt;code&gt;MinMaxScaler&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;MinMaxScaler&lt;/code&gt; shouldn't be fitted twice(as internal parameters inside &lt;code&gt;MinMaxScaler&lt;/code&gt; will be changed), and &lt;code&gt;dataX&lt;/code&gt; &amp;amp; &lt;code&gt;dataY&lt;/code&gt; should have their own scaler(as they have different minimum and maximum values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.x_scaler = MinMaxScaler()&#xA;self.y_scaler = MinMaxScaler()&#xA;&#xA;dataX = self.x_scaler.fit_transform(dataX)&#xA;dataY = self.y_scaler.fit_transform(dataY)&#xA;&#xA;self.model.fit(dataX, dataY, nb_epoch=1000)&#xA;&#xA;data = numpy.array([[val1, val2, val3]])&#xA;data = self.x_scaler.transform(data)&#xA;prediction = self.model.predict(data)&#xA;real_prediction = self.y_scaler.inverse_transform(prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T10:29:16.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16742" PostTypeId="2" ParentId="11332" CreationDate="2017-02-04T10:57:09.160" Score="1" Body="&lt;p&gt;Theano has a &lt;code&gt;check_blas.py&lt;/code&gt; which can benchmark your BLAS computing capability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For CNN, try &lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;convnet-benchmarks&lt;/a&gt;. I used to benchmark tensorflow with it, you can have a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For RNN, take a look at &lt;a href=&quot;https://github.com/glample/rnn-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;rnn-benchmarks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-04T10:57:09.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16743" PostTypeId="2" ParentId="16585" CreationDate="2017-02-04T16:12:13.170" Score="1" Body="&lt;p&gt;To Improve the performance of the network, I added a Batch Normalization layer, and other variables that I had to remove when using Random Forest.  One of these variables is a categorical variable with 51 levels.  I made sure to add an embedding layer that will represent the values by a vector of size 10.&#xA;As the original dataset had 15 numerical variables and 9 categorical variables, I have decided to also perform dimension reduction using auto-encoding.&lt;/p&gt;&#xA;" OwnerUserId="28367" LastActivityDate="2017-02-04T16:12:13.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16746" PostTypeId="1" AcceptedAnswerId="16748" CreationDate="2017-02-04T22:07:32.580" Score="0" ViewCount="929" Body="&lt;p&gt;Suppose I want to predict the probability of a person to buy something. I want to analyze the person image and I can use a convolutional neural network, but I also want to input in my predictive model how many times he bought something, where he goes more frequently and so on. What is a correct way to input in a deep learning model informations from different deep learning models?&lt;/p&gt;&#xA;" OwnerUserId="15464" LastActivityDate="2017-02-05T01:15:05.253" Title="What's a good method for combining data of different deep learning models?" Tags="|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16748" PostTypeId="2" ParentId="16746" CreationDate="2017-02-05T01:15:05.253" Score="1" Body="&lt;p&gt;Actually you can merge two different DL model into a big one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Keras+Tensorflow for deep learning, and there's an &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; about models with multiple inputs(outputs). In this example, tweets are fed into a RNN, while the extra data are fed into a fully-connected layer along with the output of this RNN.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-05T01:15:05.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16749" PostTypeId="2" ParentId="16728" CreationDate="2017-02-05T01:38:20.500" Score="6" Body="&lt;p&gt;I notice you mentioned that you used Label encoding but I did it myself and the code runs just fine. I used the 10 percent version of the &lt;a href=&quot;http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz&quot; rel=&quot;noreferrer&quot;&gt;dataset &lt;/a&gt;. Just put this piece of code after you load the dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for column in dataset.columns:&#xA;    if dataset[column].dtype == type(object):&#xA;        le = LabelEncoder()&#xA;        dataset[column] = le.fit_transform(dataset[column])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After label encoding you should use a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;noreferrer&quot;&gt;One Hot Encoder&lt;/a&gt; to improve the performance of some algorithms. You should also avoid using cross_validation module as it is deprecated, it will be removed in version 0.20.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-02-05T01:38:20.500" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16750" PostTypeId="2" ParentId="14812" CreationDate="2017-02-05T01:38:30.003" Score="4" Body="&lt;p&gt;As far as I know, we can't control the random seed by adding &lt;code&gt;np.random.seed&lt;/code&gt; when it comes to GPU. In fact, the randomness(non-determinstic) is a behavior of GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason behind is that cuDNN(and othere CUDA stuffs) uses a non-deterministic algorithm to compute gradients, thus we can't determine anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For theano backend, you can add &lt;code&gt;deterministic&lt;/code&gt; flag when using GPU, which leads a determine way, and a slower way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For tensorflow backend, checkout &lt;a href=&quot;https://stackoverflow.com/questions/39938307/determinism-in-tensorflow-gradient-updates&quot;&gt;this solution&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/issues/850&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/issues/850&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/2732&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/issues/2732&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/2652&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/issues/2652&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-05T01:38:30.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16751" PostTypeId="1" CreationDate="2017-02-05T04:09:52.570" Score="5" ViewCount="18203" Body="&lt;p&gt;In sklearn, for logistic regression, you can define the penalty, the regularization rate and other variables. Is there a way to set the learning rate?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="50727" LastEditDate="2018-05-17T10:29:35.697" LastActivityDate="2018-05-17T10:29:35.697" Title="Learning rate in logistic regression with sklearn" Tags="|machine-learning|scikit-learn|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16752" PostTypeId="2" ParentId="16751" CreationDate="2017-02-05T08:40:56.190" Score="6" Body="&lt;p&gt;&lt;code&gt;sklearn.linear_model.LogisticRegression&lt;/code&gt; doesn't use SGD, so there's no learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think &lt;code&gt;sklearn.linear_model.SGDClassifier&lt;/code&gt; is what you need, which is a linear classifier with SGD training.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-05T08:40:56.190" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16754" PostTypeId="1" CreationDate="2017-02-05T09:50:09.423" Score="0" ViewCount="83" Body="&lt;p&gt;Is there a way to get an explanation of the model prediction for a specific example?&lt;/p&gt;&#xA;" OwnerUserId="4770" LastActivityDate="2017-02-05T09:50:53.613" Title="Explaining machine learning models" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ClosedDate="2017-02-05T16:19:34.850" ContentLicense="CC BY-SA 3.0" />
  <row Id="16755" PostTypeId="2" ParentId="16754" CreationDate="2017-02-05T09:50:53.613" Score="2" Body="&lt;p&gt;You can use &lt;a href=&quot;https://github.com/marcotcr/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;LIME&lt;/a&gt;. It gives you the most relevant features to a certain prediction the model gave by approximating the model locally as a simpler model. Here is a good lecture on it: &lt;a href=&quot;https://www.youtube.com/watch?v=HcaAKI1tVGM&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=HcaAKI1tVGM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4770" LastActivityDate="2017-02-05T09:50:53.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16757" PostTypeId="1" CreationDate="2017-02-05T10:39:46.953" Score="1" ViewCount="547" Body="&lt;p&gt;I am trying to run some experiments on internal word structure of morphologically rich language (Russian).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each valid character is 1-hot encoded and then fixed-length vectors are concatenated to produce word vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; c             a             t&#xA;&#xA;(0... 1... 0) (0... 1... 0) (0... 1... 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This approach produces vectors of different length for words of different length. To unify them I have applied zero-padding:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c a t 0 0 0 0 0 0 0 0&#xA;&#xA;k i t t e n 0 0 0 0 0&#xA;&#xA;t r a n q u i l i t y&#xA;&#xA;h a p p i n e s s 0 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this approach sounds valid? What are best practices on the described issue?&lt;/p&gt;&#xA;" OwnerUserId="16162" LastActivityDate="2017-02-05T17:15:14.800" Title="How to deal with word length variability while using char-level one-hot encoding?" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16762" PostTypeId="1" AcceptedAnswerId="19550" CreationDate="2017-02-05T16:16:10.063" Score="7" ViewCount="406" Body="&lt;p&gt;I am working on a regression problem, where the goal is to estimate historic traffic volumes throughout a transportation network. I have traffic counters at 100 locations, so a model can learn the relation between traffic volumes and a number of explanatory variables (e.g., speeds, road characteristics, weather). Afterwards, I can apply the model to estimate historic traffic volumes in places where I don't have traffic counters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My neural network works reasonably well, but I am wondering if there are machine learning models that could explicitly account for the topology of my road network and the fact that traffic on neighboring road links is highly correlated. I could add &quot;traffic volume at the closest traffic counter&quot; as an input variable to my ANN, but I am wondering if there is a more intelligent approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this regard, I came across Bayesian networks, which can account for the network topology and correlation. However, they seem applicable to cases when we have sensors at 100 locations and we want to predict the traffic state (at these 100 locations) at a future time point. On the other hand, I have measurements at 100 locations and are looking to estimate traffic at a different location for the same time point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion is much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="1021" LastActivityDate="2017-06-08T21:23:29.977" Title="Are there Machine Learning Models for Networks?" Tags="|machine-learning|neural-network|bayesian-networks|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16763" PostTypeId="2" ParentId="16757" CreationDate="2017-02-05T17:08:10.450" Score="1" Body="&lt;p&gt;Your padded vector should be OK as a starting model. You could take it a step further and have a specific &quot;no character&quot; encoding, as that will be a clearer learning signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You say that you are using a feed-forward neural network. In that case, there is a possible change and improvement to your model, which is to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Recurrent Neural Network (RNN)&lt;/a&gt;, probably a &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network#Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;long short-term memory (LSTM)&lt;/a&gt; variation, because that has had some success in language models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a RNN, instead of having the character encoding repeated N times, you have it represented just once, and run the network N times for each sample, feeding in each character in turn. Instead of a &quot;no character&quot; symbol, you would have an &quot;end of sequence&quot; symbol. For an open-ended model with no specific target, it is common to have the output of the network be a prediction of the next character. This allows you to explore what the model has learnt by sampling from its predictions - a lot like using n-grams to generate realistic-looking words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of a RNN for character-level language modelling is that the model is built around the assumption that each input is somehow equivalent - the &lt;em&gt;same&lt;/em&gt; encoding is used at each time step. This is clearly true for character strings, so it can lead to better models from less training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main disadvantage of using a RNN is that it is more complex to understand, build and train such a model.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-05T17:15:14.800" LastActivityDate="2017-02-05T17:15:14.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16767" PostTypeId="1" CreationDate="2017-02-05T20:03:21.447" Score="3" ViewCount="1438" Body="&lt;p&gt;I have a large amount of CSV files, an example of which (for job titles) is listed below. The data is noisy (there are misspellings, difference in capitalisation, missing values, and they are not well-formed (some files have headers, some have not, and if headers are present, they don't always agree on name). I have gazetteers available.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...,IT Manager,...&#xA;...,Senior IT Manager,...&#xA;...,it manager,...&#xA;...,IT managre,...&#xA;...,junior IT managre,...&#xA;...,NULL,...&#xA;...,Business Consultant,...&#xA;...,Business consultent,...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a finite set of entity types (First Name, Last Name, Location etc.), and the task is: &lt;em&gt;Given the content of this column, which entity type, if any, does it represent?&lt;/em&gt; (It is the first step in a processing pipeline.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a form of named entity recognition [supervised learning classification] task, but all the papers I have read about named entity recognition uses conditional random fields or maximum entropy for natural language tasks, and I don't think this count as natural language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the appropriate approach/ML algorithm for such a task? How should my training data be formatted?&lt;/p&gt;&#xA;" OwnerUserId="28522" LastActivityDate="2017-04-07T09:45:58.593" Title="ML algorithm for determining CSV file header names based on content" Tags="|machine-learning|named-entity-recognition|csv|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16769" PostTypeId="1" CreationDate="2017-02-05T22:12:49.700" Score="13" ViewCount="29450" Body="&lt;p&gt;Could anyone please recommend a good frequent itemset package in python? I only need to find frequent itemset, no need of finding the association rules.&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="83275" LastEditDate="2022-07-22T16:34:16.070" LastActivityDate="2022-07-22T16:34:16.070" Title="Best frequent itemset package in python" Tags="|python|association-rules|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16770" PostTypeId="1" AcceptedAnswerId="16788" CreationDate="2017-02-06T01:57:55.603" Score="2" ViewCount="143" Body="&lt;p&gt;Say I have a data set such as the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;person, Time, Value, Event&#xA;person1, 2010-07-02 00:00:00, 5.4, 0&#xA;person2, 2010-07-02 10:00:00, 12.7, 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We have a current model in place at work that doesn't take into account the temporal aspect of our data. In that implementation, the model was trained with only unique values for 'person', and it throws away the time variable. However, it has come to our attention that we can look at our data as a sequence instead. This starting time is unique for each person, and clearly associated with only that person, so merely pretending each person is independent and just treating each row as an individual data point wouldnt make any sense. The following is what I've restructured the data as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;person, Time, Value, Event&#xA;person1, 2010-07-02 00:00:00, 5.4, 0&#xA;person1, 2010-07-02 00:00:15, 3.6, 0&#xA;person1, 2010-07-02 00:00:30, 2.4, 0&#xA;person2, 2010-07-02 10:00:00, 12.7, 0&#xA;person2, 2010-07-02 10:01:15, 12.8, 0&#xA;person2, 2010-07-02 10:01:30, 13.1, 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This sequence for each person would continue until and 'event' or 'non-event'. I'm totally unfamiliar with machine learning on time series data. All of the examples I've read with different models treat the data as one big sequence corresponding to one entity, while our data clearly doesn't work like that. Is the way I've structured the data the right way to approach a time series model? And if so, what would be an appropriate model to consider?&lt;/p&gt;&#xA;" OwnerUserId="28665" LastActivityDate="2017-02-06T16:18:03.317" Title="Learning with groups of sequential data" Tags="|machine-learning|time-series|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16771" PostTypeId="1" CreationDate="2017-02-06T06:14:49.303" Score="2" ViewCount="68" Body="&lt;p&gt;Is there an alternative to JupyterHub that allows sharing of notebooks across team members. We need the following requirements:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Be able to run the server on a HPC. &#xA;Multiple user login.&#xA;Version control with Git or similar service.&#xA;Sharing of notebooks between team members.&lt;/p&gt;&#xA;" OwnerUserId="28669" LastEditorUserId="28628" LastEditDate="2017-02-07T06:29:18.890" LastActivityDate="2017-02-07T06:29:18.890" Title="Team Collaboration using Jupyter Notebooks?" Tags="|jupyter|" AnswerCount="0" CommentCount="1" ClosedDate="2017-02-07T13:07:07.863" ContentLicense="CC BY-SA 3.0" />
  <row Id="16773" PostTypeId="1" CreationDate="2017-02-06T06:59:26.183" Score="0" ViewCount="133" Body="&lt;p&gt;In&#xA;&lt;a href=&quot;https://www.kaggle.com/c/santander-product-recommendation/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/santander-product-recommendation/data&lt;/a&gt;&#xA;it mentions that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Please note: This sample does not include any real Santander Spain&#xA;  customers, and thus it is not representative of Spain's customer base.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What are the ways where the Santander can anonymize their customers yet the solutions by Kaggle can be useful for them?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2017-02-06T07:52:13.787" Title="Anonymizing data" Tags="|anonymization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16774" PostTypeId="2" ParentId="13750" CreationDate="2017-02-06T07:11:18.957" Score="6" Body="&lt;p&gt;I suggest reinstalling the GPU version of Tensorflow, although you can install both version of Tensorflow via &lt;a href=&quot;https://virtualenv.pypa.io/en/stable/&quot; rel=&quot;noreferrer&quot;&gt;virtualenv&lt;/a&gt;. GPU version of Tensorflow supports CPU computation, you can switch to CPU easily:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with device('/cpu:0'):&#xA;    # your code here&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have been using GPU version of Tensorflow on my Tesla K80 for a few months, it works like a charm. Feel free to have a try!&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-06T07:11:18.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16775" PostTypeId="2" ParentId="16127" CreationDate="2017-02-06T07:33:05.777" Score="1" Body="&lt;p&gt;The inductive bias of a Gaussian process (GP) is encoded in the covariance kernel. A GP is a distribution over functions — when we choose a kernel, we are specifying characteristics that we expect the solution function to have, e.g., smooth, linear, periodic, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a common covariance kernel is the squared exponential:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$k_{SE}(x, x') = \exp \left( - \frac{||x - x'||_2^2}{2l^2} \right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $l$ is the characteristic length-scale. This kernel specifies that an unobserved point will be similar to each observed point as an exponentially decaying function of the Euclidean distance between the two points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More detailed explanations can be found in section 2.4 of Andrew Gordon Wilson's &lt;a href=&quot;https://www.cs.cmu.edu/~andrewgw/andrewgwthesis.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;thesis&lt;/a&gt; or in the book &lt;a href=&quot;http://www.gaussianprocess.org/gpml/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian Processes for Machine Learning&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9616" LastActivityDate="2017-02-06T07:33:05.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16776" PostTypeId="2" ParentId="16773" CreationDate="2017-02-06T07:43:50.300" Score="1" Body="&lt;p&gt;If a model predicts useful information for a class of customers, maybe customers over 50, or those with more than 1000EUR, then that's useful even without knowing who the individuals in the model are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual data doesn't seem to be anonymous data though, it is implied to be &lt;em&gt;synthetic&lt;/em&gt; data or possibly from another bank altogether (&quot;does not include any real Santander Spain customers&quot;).&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-02-06T07:43:50.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16777" PostTypeId="2" ParentId="16773" CreationDate="2017-02-06T07:52:13.787" Score="1" Body="&lt;p&gt;I think data masking technique is what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The main reason for applying masking to a data field is to protect data that is classified as personal identifiable data, personal sensitive data or commercially sensitive data, however the data must remain usable for the purposes of undertaking valid test cycles.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Take your santander problem for example, there is an &lt;code&gt;age&lt;/code&gt; feature in the original dataset. As we all know, $1\leqslant \text{age}\leqslant 200$ (nobody survives 201 years right?). If we do something like $\text{new_age}:=(\ln{(\pi* \text{age}+\sqrt{2})})^2$, &lt;code&gt;new_age&lt;/code&gt; is kind of &quot;encrypted&quot; and no one knows the actual age of John Doe.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Data_masking&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Data_masking&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-06T07:52:13.787" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16778" PostTypeId="2" ParentId="16767" CreationDate="2017-02-06T08:04:14.823" Score="2" Body="&lt;p&gt;Yes, it might not be exactly natural Language Understanding but CRF is an excellent algorithm to train Named Entity Recognition tasks and is the stamdard model used by Stanford NLP group. You can try out their NER tagger &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://nlp.stanford.edu:8080/ner/&amp;amp;ved=0ahUKEwjTnK7w__rRAhXFtI8KHXZ_AOQQFggrMAE&amp;amp;usg=AFQjCNEKaDnE73ltvYDdrxetD1m9dlkVzw&amp;amp;sig2=saNZg1PMiDhhQxu7umD3qA&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If you want something that accounts for language understanding then there are certain papers who have trained recurrent neural network architectures such as LSTM and Bi-directional RNNs. Look at this &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://arxiv.org/abs/1511.08308&amp;amp;ved=0ahUKEwiCgsKsgPvRAhVKNY8KHaAgCzsQFgglMAE&amp;amp;usg=AFQjCNEKC-6oepdOFqKNFkeiCdN29AVAlQ&amp;amp;sig2=Gq4FteG3yJFdROM9v32mPQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. I must tell you that Named Entity Recognition is an incredibly hard problem and if you want to use Deep Learning architectures, it will require tremendous amount of data. I will suggest try out the Stanford NER tagger, since your data doesn't have much of a sequential nature, I am certain it will perform well. In the end &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://medium.com/%40klintcho/training-a-swedish-ner-model-for-stanford-corenlp-part-1-3e3f281a753a&amp;amp;ved=0ahUKEwjhoMH5gPvRAhWIt48KHQSBAgMQFggaMAA&amp;amp;usg=AFQjCNFOzjSqMiOUgXFQlgYS_5Z7K29Lrg&amp;amp;sig2=fVSkqYWp-mTJydN5uRgjYA&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is part 1 of an excellent blog post that goes in detail about training your own NER model using the Stanford NER tool. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-02-06T08:04:14.823" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16779" PostTypeId="2" ParentId="16705" CreationDate="2017-02-06T08:04:34.063" Score="3" Body="&lt;p&gt;Interesting question! I'm assuming you have a bunch of points $(X, Y, Z)$ plotting the lung you want to measure. &#xA;So your question is actually the measure of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Convex_hull&quot; rel=&quot;noreferrer&quot;&gt;convex hull&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Checkout &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.ConvexHull.html&quot; rel=&quot;noreferrer&quot;&gt;scipy.spatial.ConvexHull&lt;/a&gt;. The &lt;code&gt;ConvexHull&lt;/code&gt; object has a &lt;code&gt;volumn&lt;/code&gt; attribute which indicates the column of the convex hull you want. It supports N dimensions, which your question is about three dimensions.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-06T08:04:34.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16780" PostTypeId="2" ParentId="11843" CreationDate="2017-02-06T10:14:04.257" Score="1" Body="&lt;p&gt;Generally memory cost increases linearly while mini-batch size increases. If batch size 32 costs you 2GB memory, then batch size 64 will cost you 4GB memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, deep learning framework(for instance, &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;) will load the &lt;strong&gt;entire&lt;/strong&gt; full-batch into your memory, thus you cannot observe apparent memory increment or decrement during mini-batch tuning.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/issues/146&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/issues/146&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-06T10:14:04.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16781" PostTypeId="2" ParentId="10278" CreationDate="2017-02-06T11:04:44.550" Score="1" Body="&lt;p&gt;The code and data combination is too small for predicting a solution. But note that you will not get any answer if there are no recommendations made. In this case there will be no exception either. Looks like your code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;List&amp;lt;RecommendedItem&amp;gt; recommendations =&#xA;     recommender.recommend(userid, noOfRecommendations);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returned no recommendations. You should try with a variety of data.&lt;/p&gt;&#xA;" OwnerUserId="28677" LastActivityDate="2017-02-06T11:04:44.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16782" PostTypeId="2" ParentId="11745" CreationDate="2017-02-06T11:09:10.147" Score="2" Body="&lt;p&gt;I am assuming that when you say log transforming the data, it is log transformation of outcome variable. In such case, you may want to check &lt;a href=&quot;https://www.kaggle.com/wiki/RootMeanSquaredLogarithmicError&quot; rel=&quot;nofollow noreferrer&quot;&gt;root means squared logarithmic error&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2017-02-06T11:09:10.147" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16783" PostTypeId="1" AcceptedAnswerId="16787" CreationDate="2017-02-06T13:02:14.960" Score="1" ViewCount="3469" Body="&lt;p&gt;I'm looking for a data set that shows hospital patients' vital signs (body temperature and/or heart rate, etc.) and their illnesses, over time. Is there such a data set?&lt;br&gt;&#xA;&lt;br&gt;&#xA;For example:&lt;br&gt;&#xA;Patient #1, Male, age:35, ..., Diagnosed w/: Heart failure&lt;br&gt;&#xA;22:32:10: Temperature:32.1&lt;br&gt;&#xA;22:32:20: Temperature:34.2&lt;br&gt;&#xA;22:32:30: Temperature:33&lt;br&gt;&#xA;22:32:40: Temperature:35&lt;br&gt;&#xA;&lt;br&gt;&#xA;Or:&lt;br&gt;&#xA;Patient #2, Female, age:35, ..., Diagnosed w/: Migraine Attack&lt;br&gt;&#xA;22:32: Heart Rate:68&lt;br&gt;&#xA;22:33: Heart Rate:75&lt;br&gt;&#xA;22:34: Heart Rate:69&lt;br&gt;&#xA;22:35: Heart Rate:70&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have any insistence for the data to come from a hospital. I already have searched some sources but with no luck.&lt;/p&gt;&#xA;" OwnerDisplayName="user28683" LastEditorDisplayName="user28683" LastEditDate="2017-02-06T13:25:07.300" LastActivityDate="2019-02-12T10:46:10.160" Title="Looking for a data set that shows hospital patients' illnesses and their vital signs, during their stay" Tags="|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16784" PostTypeId="2" ParentId="16124" CreationDate="2017-02-06T13:48:58.840" Score="0" Body="&lt;p&gt;Identifying &lt;em&gt;new&lt;/em&gt; classes sounds like unsupervised learning to me, not supervised learning. Supervised learning implies a fixed set of labels (and appropriate training data) that you as a human provide, which is how you teach the machine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Teaching the machine to identify new classes is a different matter altogether, you're essentially asking the machine to intelligently identify something it has no knowledge of, because no designated training data (=knowledge) exists in the model. This is, as far as I know, not possible.&lt;/p&gt;&#xA;" OwnerUserId="8152" LastActivityDate="2017-02-06T13:48:58.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16785" PostTypeId="2" ParentId="16783" CreationDate="2017-02-06T14:13:29.897" Score="0" Body="&lt;p&gt;I'm assuming you are looking for medical datasets, not only what you have stated in the question, but also some image recognition datasets which need something like CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Heart+Disease&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;. This dataset contains several medical features including blood sugar, serum cholesterol etc, and wants you to find out the presence of heart disease.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meanwhile, there are some medical &lt;a href=&quot;https://www.kaggle.com/competitions&quot; rel=&quot;nofollow noreferrer&quot;&gt;competitions&lt;/a&gt; and &lt;a href=&quot;https://www.kaggle.com/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;datasets&lt;/a&gt; on &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;, including the famous &lt;a href=&quot;https://www.kaggle.com/c/data-science-bowl-2017&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science Bowl&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-06T14:13:29.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16786" PostTypeId="1" AcceptedAnswerId="28665" CreationDate="2017-02-06T14:53:02.613" Score="4" ViewCount="1633" Body="&lt;p&gt;I am trying to draw a process flow (like a template) to be followed while on text analysis projects. So far, I've come up with this.&lt;br&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;b&gt;Text Analytics Steps&lt;/b&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Data Collection&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Acquire data&lt;/li&gt;&#xA;&lt;li&gt;Convert data into plain text&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;Remove Duplicate Entries&lt;/li&gt;&#xA;&lt;li&gt;Text Parsing and Extracting Features&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tokenization&lt;/li&gt;&#xA;&lt;li&gt;Parsing&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove HTML characters&lt;/li&gt;&#xA;&lt;li&gt;Decode complex symbols to UTF-8&lt;/li&gt;&#xA;&lt;li&gt;Spell check&lt;/li&gt;&#xA;&lt;li&gt;Apostrophe look-up&lt;/li&gt;&#xA;&lt;li&gt;Remove punctuation marks&lt;/li&gt;&#xA;&lt;li&gt;Remove expressions / emojis&lt;/li&gt;&#xA;&lt;li&gt;Split attached words&lt;/li&gt;&#xA;&lt;li&gt;Slangs look-up&lt;/li&gt;&#xA;&lt;li&gt;Remove URLs&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;Lemmatization / Stemming (Normalization of Tokens)&lt;/li&gt;&#xA;&lt;li&gt;Parts-of-Speech Tagging&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;Text Filtering&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Remove start-words&lt;/li&gt;&#xA;&lt;li&gt;Remove stop-words&lt;/li&gt;&#xA;&lt;li&gt;Remove irrelevant words based on frequency&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;Text Transformation&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bag of Words Representation&lt;/li&gt;&#xA;&lt;li&gt;TF-IDF&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;Text Mining / Analysis (whichever analysis needed)&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Text Categorization&lt;/li&gt;&#xA;&lt;li&gt;Text Classification (supervised)&lt;/li&gt;&#xA;&lt;li&gt;Topic Modeling (unsupervised)&lt;/li&gt;&#xA;&lt;li&gt;Text Clustering&lt;/li&gt;&#xA;&lt;li&gt;Similarity Analysis&lt;/li&gt;&#xA;&lt;li&gt;Sentiment Analysis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;br&gt;&#xA;Is this flow in the right order of steps?&lt;br&gt;&#xA;What are the steps/sub-steps that I am missing?&lt;br&gt;&#xA;Does the process flow look like a template or go-to flow chart when undertaking any text analytics project?&#xA;&lt;p&gt;&lt;b&gt;Edit: Updated process flow &lt;/b&gt;&lt;/p&gt;&#xA;" OwnerUserId="2647" LastEditorUserId="98307" LastEditDate="2020-08-07T07:24:21.213" LastActivityDate="2020-08-07T07:24:21.213" Title="Is there a process flow to follow for text analytics?" Tags="|nlp|text-mining|text-filter|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16787" PostTypeId="2" ParentId="16783" CreationDate="2017-02-06T15:53:26.150" Score="0" Body="&lt;p&gt;This is not exactly what you are looking for, but here is a collection of clinical data sets that might provide a good alternative: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dgroppe.com/2016/11/24/public-clinical-data-science-data-sets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dgroppe.com/2016/11/24/public-clinical-data-science-data-sets/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, you might want to look into the Physionet archive or the Sepsis Patient Event Log.&lt;/p&gt;&#xA;" OwnerUserId="26907" LastActivityDate="2017-02-06T15:53:26.150" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16788" PostTypeId="2" ParentId="16770" CreationDate="2017-02-06T16:18:03.317" Score="1" Body="&lt;p&gt;The data that you are showing is typical survival data. If you want to model the event depending on time and value you should look into survival models with time dependent covariates. If you want to do this without any assumptions on the distribution you can start with a Kaplan Meier estimate as explained &lt;a href=&quot;https://stats.stackexchange.com/questions/46754/visualize-survival-analysis-with-time-dependent-covariates&quot;&gt;here&lt;/a&gt;. If you want to use parametric models you can look at Weibull or Gamma regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are new to this topic I can highly recommend to browse the examples of the packages in the &lt;a href=&quot;https://cran.r-project.org/web/views/Survival.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRAN Survival Task View&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-06T16:18:03.317" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16789" PostTypeId="1" CreationDate="2017-02-06T20:09:57.593" Score="3" ViewCount="692" Body="&lt;p&gt;I'm working on a Veolia challenge on Ens Data Challenge &lt;a href=&quot;https://challengedata.ens.fr/en/home&quot; rel=&quot;nofollow noreferrer&quot;&gt;ens-data&lt;/a&gt; (equivalent to Kaggle) the goal is to classify very rare binary events (the failure of a pipeline ) for 2014 and 2015 (y={2014,2015}). In input we have 5 features 3 categorical features (which I turned into dummy variable) and two continuous. The score is average AUC, $0.6*AUC_1 + 0.4*AUC_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is the following, when I compute each AUC (for 2014 and 2015) with a stratified kfold cross validation and I compute the average AUC I get roughly 0.88 and when I submit on the website I end up with 0.67, I guess there is a problem in my code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code for choosing the best model for 2014:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rk: to predict on the test set (2014,2015 unknown), I first predict with all 5 features, 2014. Then I add the prediction of 2014 to my feature to predict 2015&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Spot Check Algorithms&#xA;models = []&#xA;models.append(('LG', LogisticRegression()))&#xA;models.append(('LDA', LinearDiscriminantAnalysis()))&#xA;models.append(('RF', RandomForestClassifier()))&#xA;models.append(('GBC', GradientBoostingClassifier()))&#xA;models.append(('SVM', SVC()))&#xA;# evaluate each model in turn&#xA;results = []&#xA;names = []&#xA;&#xA;# stratifiedkfold is defined by default when there is an integer&#xA;scoring = 'roc_auc'&#xA;num_folds = 10&#xA;&#xA;for name, model in models:&#xA;    cv_results = cross_validation.cross_val_score(model, X, Y, cv=num_folds, scoring=scoring)&#xA;    results.append(cv_results)&#xA;    names.append(name)&#xA;    msg = &quot;%s: %f (%f)&quot; % (name, cv_results.mean(), cv_results.std())&#xA;    print(msg)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19065" LastEditorUserId="19065" LastEditDate="2017-02-07T09:17:49.400" LastActivityDate="2018-02-03T07:31:53.523" Title="Gap leaderboard score and model scoring on a Competition" Tags="|machine-learning|python|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16790" PostTypeId="1" CreationDate="2017-02-06T22:21:10.997" Score="2" ViewCount="1917" Body="&lt;p&gt;Hdbscan is an excellent technique to find the &quot;optimal&quot; number of clusters within your data when you have little a priori idea how many clusters should exist. This makes the method great for exploratory analysis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/comparing_clustering_algorithms.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://hdbscan.readthedocs.io/en/latest/comparing_clustering_algorithms.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my problem: All results using hdbscan with the python implement in the link above rely on the crucial min_cluster_size &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/parameter_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://hdbscan.readthedocs.io/en/latest/parameter_selection.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If users have a priori little idea how many clusters best fit the data, what is the correct approach above? Isn't there a metric one uses to decide what the optimal number of clusters is?&lt;/p&gt;&#xA;" OwnerUserId="23120" LastActivityDate="2017-02-07T07:11:28.540" Title="HDBSCAN cluster: still unclear to me how to chose 'min_cluster_size`" Tags="|machine-learning|python|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16791" PostTypeId="1" AcceptedAnswerId="16792" CreationDate="2017-02-06T23:24:56.353" Score="1" ViewCount="2669" Body="&lt;p&gt;Im new to ML. I'm trying to predict if a new Music Album will exceed X amount of dollars in Sales.&#xA;I'm looking to build a model to go only after potential best sellers. I do have historic data for Music Sales from 2010 till 2016. I have many signals:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Music Genre&lt;/li&gt;&#xA;&lt;li&gt;Music Band/Artist name&lt;/li&gt;&#xA;&lt;li&gt;Label&lt;/li&gt;&#xA;&lt;li&gt;Year released&lt;/li&gt;&#xA;&lt;li&gt;Country of origin&lt;/li&gt;&#xA;&lt;li&gt;Part of a Series/Volume... etc.&lt;/li&gt;&#xA;&lt;li&gt;Sales per month&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What type of ML problem is this one?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2017-03-08T22:32:38.337" LastActivityDate="2022-08-11T17:14:55.683" Title="Price prediction based on historic data" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16792" PostTypeId="2" ParentId="16791" CreationDate="2017-02-07T00:37:27.917" Score="8" Body="&lt;p&gt;There are two broad classes of problems in machine learning, classification and regression. As in this &lt;a href=&quot;https://math.stackexchange.com/q/183001&quot;&gt;answer&lt;/a&gt;, Regression involves estimating or predicting a response (the dependent variable is continuous). Classification is identifying group membership (the dependent variable is discrete).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem is a regression problem, you must try to estimate the real number of sales. You can look &lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; for a similar problem and techniques to solve it. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2017-02-07T00:47:17.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16793" PostTypeId="2" ParentId="16769" CreationDate="2017-02-07T00:41:13.090" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://pypi.python.org/pypi/Orange3-Associate&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange3-Associate&lt;/a&gt; package provides &lt;a href=&quot;https://orange3-associate.readthedocs.io/en/latest/scripting.html#fpgrowth.frequent_itemsets&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;frequent_itemsets()&lt;/code&gt;&lt;/a&gt; function based on &lt;a href=&quot;https://www.cs.sfu.ca/~jpei/publications/dami03_fpgrowth.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FP-growth&lt;/a&gt; algorithm.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-02-07T00:41:13.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16795" PostTypeId="2" ParentId="16790" CreationDate="2017-02-07T07:11:28.540" Score="2" Body="&lt;p&gt;Optimal in which sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crucial thing with clustering is that there is no optimal solution. Different solutions tell you a different part of the story. And to be able to get different views, you will &lt;strong&gt;need parameters&lt;/strong&gt;. It is a exploratory technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Various attempts at defining &quot;optimal&quot; solutions have failed for practical use, just think of k-means.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-02-07T07:11:28.540" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16797" PostTypeId="1" AcceptedAnswerId="16813" CreationDate="2017-02-07T09:09:15.993" Score="48" ViewCount="83179" Body="&lt;p&gt;For detection, a common way to determine if one object proposal was right is &lt;em&gt;Intersection over Union&lt;/em&gt; (IoU, IU). This takes the set $A$ of proposed object pixels and the set of true object pixels $B$ and calculates:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$IoU(A, B) = \frac{A \cap B}{A \cup B}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Commonly, IoU &gt; 0.5 means that it was a hit, otherwise it was a fail. For each class, one can calculate the&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;True Positive ($TP(c)$): a proposal was made for class $c$ and there actually was an object of class $c$&lt;/li&gt;&#xA;&lt;li&gt;False Positive ($FP(c)$): a proposal was made for class $c$, but there is no object of class $c$&lt;/li&gt;&#xA;&lt;li&gt;Average Precision for class $c$: $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The mAP (mean average precision) = $\frac{1}{|classes|}\sum_{c \in classes} \frac{\#TP(c)}{\#TP(c) + \#FP(c)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one wants better proposals, one does increase the IoU from 0.5 to a higher value (up to 1.0 which would be perfect). One can denote this with mAP@p, where $p \in (0, 1)$ is the IoU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what does &lt;code&gt;mAP@[.5:.95]&lt;/code&gt; (as found in &lt;a href=&quot;https://arxiv.org/abs/1512.04412&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;) mean?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2020-08-20T18:45:48.813" Title="What does the notation mAP@[.5:.95] mean?" Tags="|computer-vision|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16798" PostTypeId="2" ParentId="13754" CreationDate="2017-02-07T09:26:38.447" Score="1" Body="&lt;p&gt;A common reason for this is that the parameters you supplied (or defaulted) to &lt;code&gt;RandomForestClassifier&lt;/code&gt; are not suited for your dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common way to address this problem is to search the hyperparameter space using e.g. &lt;code&gt;GridSearchCV&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import GridSearchCV&#xA;from sklearn.metrics import accuracy_score, make_scorer&#xA;&#xA;param_grid = {'n_estimators': [10, 100, 1000], 'max_features': [5, 10, 20, 30]}&#xA;clf = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring=make_scorer(accuracy_score))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;param_grid&lt;/code&gt; here is the permutations of parameters that you want to search in, and the &lt;code&gt;make_scorer(accuracy_score)&lt;/code&gt; is the measure you want to optimize.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;code&gt;accuracy_score&lt;/code&gt; is suitable for balanced sets, but not for unbalanced sets. Choose a suitable metric to on your particular objective.&lt;/p&gt;&#xA;" OwnerUserId="28714" LastActivityDate="2017-02-07T09:26:38.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16799" PostTypeId="2" ParentId="13754" CreationDate="2017-02-07T09:52:35.657" Score="2" Body="&lt;p&gt;Your result is not that weird. As &lt;a href=&quot;https://datascience.stackexchange.com/users/9085/lanenok&quot;&gt;lanenok&lt;/a&gt; states, you should in a first step increase the number of trees in order to make sure that you get a 'statistical' result concerning the feature importances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as this &lt;a href=&quot;http://ac.els-cdn.com/S0167865510000954/1-s2.0-S0167865510000954-main.pdf?_tid=76378d52-ed1a-11e6-98c1-00000aab0f27&amp;amp;acdnat=1486461039_56773ffa03e91d2b763381344e93d09f&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Genuer et al. (2010) shows, you can actually use the standard deviations in order to eliminate features. To quote: &quot;&lt;em&gt;We can see that true variables standard deviation is large compared to the noisy variables one, which is close to zero.&lt;/em&gt; &quot;&lt;/p&gt;&#xA;" OwnerUserId="14372" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:40.600" LastActivityDate="2017-02-07T09:52:35.657" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16800" PostTypeId="1" AcceptedAnswerId="16804" CreationDate="2017-02-07T10:12:01.437" Score="10" ViewCount="9890" Body="&lt;p&gt;Consider the following simple classification problem (Python, scikit-learn)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;from sklearn.model_selection import train_test_split&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.metrics import accuracy_score&#xA;&#xA;def get_product_data(size):&#xA;    '''&#xA;    Given a size(int), sets `log10(size)` features to be uniform &#xA;    random variables `Xi` in [-1,1] and an target `y` given by 1 if &#xA;    their product `P` is larger than 0.0 and zero otherwise. &#xA;    Returns a pandas DataFrame.&#xA;    '''&#xA;    n_features = int(max(2, np.log10(size)))&#xA;    features = dict(('x%d' % i, 2*np.random.rand(size) - 1) for i in range(n_features))&#xA;    y = np.prod(list(features.values()), axis=0)&#xA;    y = y &amp;gt; 0.0&#xA;    features.update({'y': y.astype(int)})&#xA;    return pd.DataFrame(features)&#xA;&#xA;# create random data&#xA;df = get_product_data(1000)&#xA;&#xA;X = np.array(df.drop(df.columns[-1], axis=1))&#xA;y = df['y']&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, &#xA;                                                        random_state=1)    &#xA;&#xA;def predict(clf):&#xA;    '''&#xA;    Splits train/test with a fixed seed, fits, and returns the accuracy&#xA;    '''&#xA;    clf.fit(X_train, y_train)&#xA;    return accuracy_score(y_test, clf.predict(X_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the following classifiers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foo10 = RandomForestClassifier(10, max_features=None, bootstrap=False)&#xA;foo100 = RandomForestClassifier(100, max_features=None, bootstrap=False)&#xA;foo200 = RandomForestClassifier(200, max_features=None, bootstrap=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why does &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict(foo10)  # 0.906060606061&#xA;predict(foo100)  # 0.933333333333&#xA;predict(foo200)  # 0.915151515152&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;give different scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, with&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;max_features=None&lt;/code&gt;, all features are selected for each tree&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;bootstrap=False&lt;/code&gt;, there is no bootstrap of samples&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;max_depth=None&lt;/code&gt; (default), all trees reach the maximum depth&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would expect each tree to be exactly the same. Thus, regardless of how many trees the forest has, the predictions should be equal. Where is the tree's variability coming from in this example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What further parameters would I have to introduce in the &lt;code&gt;RandomForestClassifier.__init__&lt;/code&gt; in such a way that &lt;code&gt;foo*&lt;/code&gt; have all the same score?&lt;/p&gt;&#xA;" OwnerUserId="28714" LastActivityDate="2018-07-06T23:29:46.807" Title="Why `max_features=n_features` does not make the Random Forest independent of number of trees?" Tags="|python|random-forest|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16801" PostTypeId="2" ParentId="12645" CreationDate="2017-02-07T10:30:46.063" Score="50" Body="&lt;p&gt;When using pandas, try to avoid performing operations in a loop, including &lt;code&gt;apply&lt;/code&gt;, &lt;code&gt;map&lt;/code&gt;, &lt;code&gt;applymap&lt;/code&gt; etc. That's slow!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A DataFrame object has two axes: “axis 0” and “axis 1”. “axis 0” represents rows and “axis 1” represents columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to count the missing values in each column, try:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.isnull().sum()&lt;/code&gt; as default or &lt;code&gt;df.isnull().sum(axis=0)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, you can count in each row (which is your question) by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.isnull().sum(axis=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's roughly 10 times faster than Jan van der Vegt's solution(BTW he counts valid values, rather than missing values):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [18]: %timeit -n 1000 df.apply(lambda x: x.count(), axis=1)&#xA;1000 loops, best of 3: 3.31 ms per loop&#xA;&#xA;In [19]: %timeit -n 1000 df.isnull().sum(axis=1)&#xA;1000 loops, best of 3: 329 µs per loop&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="99060" LastEditDate="2020-06-14T13:15:03.997" LastActivityDate="2020-06-14T13:15:03.997" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16804" PostTypeId="2" ParentId="16800" CreationDate="2017-02-07T11:39:33.980" Score="8" Body="&lt;p&gt;Interesting puzzle indeed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First things first. The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;DecisionTreeClassifier&lt;/code&gt;&lt;/a&gt; has some stochastic behavior. For instance, the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/14031f65d144e3966113d3daec836e443c6d7a5b/sklearn/tree/_splitter.pyx&quot; rel=&quot;nofollow noreferrer&quot;&gt;splitter code&lt;/a&gt; iterates through the features at random:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        f_j = rand_int(n_drawn_constants, f_i - n_found_constants,&#xA;                       random_state)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your data is small and comes from the same distribution. What this means is that you'll have a lot of identical purity scores depending on how iteration is done. If you (a) increase your data, or (b) make it more separable, you'll see the problem should ameliorate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify: if the algorithm computes the score for feature A and then computes the score for feature B and it gets score N. Or if it computes first the score for feature B and then for feature A and it gets the same score N, you can see how each decision tree will be different, and have different scores during test, even if the train test is the same (100% if max_depth=None of course). (You can confirm this.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During my exploration of your question, I have produced the following code with my own implementation of a random forest. Since it took me some time, I figured I might as well paste it here. :) Seriously, it can be useful. You can try to disable &lt;code&gt;random_state&lt;/code&gt; from my implementation to see what I mean.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;from sklearn.metrics import accuracy_score&#xA;import numpy as np&#xA;&#xA;&#xA;class MyRandomForestClassifier:&#xA;    def __init__(self, n_estimators):&#xA;        self.n_estimators = n_estimators&#xA;&#xA;    def fit(self, X, y):&#xA;        self.trees = [DecisionTreeClassifier(random_state=1).fit(X, y)&#xA;                      for _ in range(self.n_estimators)]&#xA;        return self&#xA;&#xA;    def predict(self, X):&#xA;        yp = [tree.predict(X) for tree in self.trees]&#xA;        return ((np.sum(yp, 0) / len(self.trees)) &amp;gt; 0.5).astype(int)&#xA;&#xA;    def score(self, X, y):&#xA;        return accuracy_score(y, self.predict(X))&#xA;&#xA;&#xA;for alpha in (1, 0.1, 0.01):&#xA;    np.random.seed(1)&#xA;    print('# alpha: %s' % str(alpha))&#xA;    N = 1000&#xA;    X = np.random.random((N, 10))&#xA;    y = np.r_[np.zeros(N//2, int), np.ones(N//2, int)]&#xA;    X[y == 1] = X[y == 1]*alpha&#xA;    Xtr, Xts, ytr, yts = train_test_split(X, y)&#xA;&#xA;    print('## sklearn forest')&#xA;    for n_estimators in (1, 10, 100, 200, 500):&#xA;        m = RandomForestClassifier(&#xA;            n_estimators, max_features=None, bootstrap=False)&#xA;        m.fit(Xtr, ytr)&#xA;        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))&#xA;&#xA;    print('## my forest')&#xA;    for n_estimators in (1, 10, 100, 200, 500):&#xA;        m = MyRandomForestClassifier(n_estimators)&#xA;        m.fit(Xtr, ytr)&#xA;        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))&#xA;    print()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary:&lt;/strong&gt; Each &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; is stochastic, data such as yours, which is small and comes from the same distribution, are bound to produce slightly different trees, even if the random forest itself is deterministic. You can fix this by passing the same seed to each &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; which you can do using &lt;code&gt;random_state=something&lt;/code&gt;. &lt;code&gt;RandomForestClassifier&lt;/code&gt; also has a &lt;code&gt;random_state&lt;/code&gt; parameter which it passes along each &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;. (This is slightly incorrect, see the edit.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT2:&lt;/strong&gt; While this removes the stochasticity component of the training, the decision trees would still be different. The thing is that sklearn ensembles generate a new random seed for each child based on the random state they are given. They do not pass along the same &lt;code&gt;random_state&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see this is the case by checking the &lt;code&gt;_set_random_states&lt;/code&gt; method from the ensemble base module, in particular &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b464d094d2c468a16ea9f8bf8d42d949f84/sklearn/ensemble/base.py#L549&quot; rel=&quot;nofollow noreferrer&quot;&gt;this line&lt;/a&gt;, which propagates the &lt;code&gt;random_state&lt;/code&gt; across the ensembles' children.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2018-07-06T23:29:46.807" LastActivityDate="2018-07-06T23:29:46.807" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="16805" PostTypeId="1" AcceptedAnswerId="16868" CreationDate="2017-02-07T11:55:53.757" Score="1" ViewCount="252" Body="&lt;p&gt;I have a complex problem that I have simplified coming to a simple &lt;em&gt;integer linear programming&lt;/em&gt; formulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the scalar $K &amp;gt; 0$, the vectors $v(t) \in R^n$ and $b_i \in R^n, \forall i=1,\ldots,K$ are known. In particular $v(t)$  is the measured value (ex: every 10 minutes) while the $b_i$ are fixed (computed ones, previously).&#xA;Consider that I have: $$K &amp;gt;&amp;gt; n,$$ &#xA;i.e., I have to identify the presence of $K$ (ex: $K=20$) elements, each one of dimension $n$ (ex: $n=2$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for every fixed $t$, the problem that I have to solve has the following formulation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \newcommand\norm[1]{\left\lVert#1\right\rVert}&#xA;\begin{equation*}\begin{aligned}&#xA;&amp;amp; \underset{x_i}{\text{minimize}}&#xA;&amp;amp; &amp;amp; \norm{v - \sum_{i=1}^{K} b_i x_i}_{2} \\&#xA;&amp;amp; \text{subject to}&#xA;&amp;amp; &amp;amp; x_i \in \{0,1\},\ \forall i = 1, \ldots, K.&#xA;\end{aligned} \end{equation*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know which algorithms solve that formulation? In particular I am looking for an algorithm implemented in Python.&lt;/p&gt;&#xA;" OwnerUserId="10024" LastEditorUserId="10024" LastEditDate="2017-02-09T09:57:20.193" LastActivityDate="2017-02-10T05:40:32.373" Title="Integer programming formulation: which algorithms" Tags="|python|algorithms|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16806" PostTypeId="2" ParentId="12645" CreationDate="2017-02-07T12:37:44.440" Score="6" Body="&lt;p&gt;The simplist way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.isnull().sum(axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21195" LastActivityDate="2017-02-07T12:37:44.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16807" PostTypeId="1" AcceptedAnswerId="16818" CreationDate="2017-02-07T12:40:25.200" Score="67" ViewCount="76669" Body="&lt;p&gt;I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration; and epoch is a full turn when each of the training instances have been seen by the model. If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a &quot;max batch&quot; by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-04-27T16:48:40.700" Title="Why mini batch size is better than one single &quot;batch&quot; with all training data?" Tags="|machine-learning|deep-learning|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16808" PostTypeId="2" ParentId="16789" CreationDate="2017-02-07T12:48:01.243" Score="1" Body="&lt;p&gt;As you have commented, you are concerning about over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, cross validation will help to weaken over-fitting, but it can't eliminate over-fitting. Model scoring on TRAIN dataset sometimes exceeds scoring on TEST dataset. Here are some examples I can find:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1512.03385v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Residual Learning for Image Recognition&lt;/a&gt; (the famous ResNet paper). Checkout figure.1&lt;/li&gt;&#xA;&lt;li&gt;This &lt;a href=&quot;https://www.kaggle.com/sudalairajkumar/two-sigma-financial-modeling/univariate-analysis-regression-lb-0-006&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernel&lt;/a&gt; of &lt;a href=&quot;https://www.kaggle.com/c/two-sigma-financial-modeling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Two Sigma Financial Modeling Challenge&lt;/a&gt; on &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt; says:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;we are getting a public score of 0.0169 which is slightly better than the previous one.&#xA;  Submitting this model to the LB gave me a score of 0.006&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In my practice, I am more willing to concern the scoring gap between TRAIN, VALIDATION(DEVELOP) and TEST dataset. Personally speaking, TRAIN&gt;VALIDATION=TEST is better than TRAIN=VALIDATION&gt;TEST.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: For class imbalance problem, there are some resources:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset&lt;/a&gt;. This blog shows a common workflow dealing with imbalanced class issue.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Class Imbalance Problem in Data Mining: Review&lt;/a&gt;. This paper compares several algorithms created for solving the class imbalance problem.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="28628" LastEditDate="2017-02-08T01:58:33.483" LastActivityDate="2017-02-08T01:58:33.483" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16809" PostTypeId="2" ParentId="16800" CreationDate="2017-02-07T12:54:00.147" Score="4" Body="&lt;p&gt;Picking up &lt;a href=&quot;https://datascience.stackexchange.com/a/16804/28714&quot;&gt;answer from Ricardo Cruz&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is that &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; is not a deterministic classifier. Specifically, when splitting between two features that lead to the same decrease of the metric, &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; picks one at random. The fluctuations observed in the question are caused by this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The corollary of this observation is that scikit-learn's Random Forest uses bootstrap for 3 different things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;items (samples different items)&lt;/li&gt;&#xA;&lt;li&gt;features (samples different features)&lt;/li&gt;&#xA;&lt;li&gt;splitting (samples different splittings)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Freezing &lt;code&gt;items&lt;/code&gt; (using &lt;code&gt;bootstrap=False&lt;/code&gt;) and &lt;code&gt;features&lt;/code&gt; (using &lt;code&gt;max_depth=None&lt;/code&gt;) is not sufficient to freeze &lt;code&gt;splitting&lt;/code&gt;. The only way to freeze &lt;code&gt;splitting&lt;/code&gt; is by using a fixed &lt;code&gt;random_state&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that using the same &lt;code&gt;random_state&lt;/code&gt; alone is not enough to freeze everything, i.e. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foo10 = RandomForestClassifier(10, random_state=1)&#xA;foo100 = RandomForestClassifier(100, random_state=1)&#xA;foo200 = RandomForestClassifier(200, random_state=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;do not give the same score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the only way to guarantee that the classifier gives the same results for any &lt;code&gt;n_estimators&lt;/code&gt; is to initialize it as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RandomForestClassifier(n_estimators, max_features=None, bootstrap=False, random_state=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is not particularly helpful for classification per se, but allows to better understand what is being done under the hood.&lt;/p&gt;&#xA;" OwnerUserId="28714" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-02-07T12:54:00.147" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16810" PostTypeId="2" ParentId="16805" CreationDate="2017-02-07T14:17:54.750" Score="1" Body="&lt;p&gt;You may reparameterize $x_i$ as $x_i = sigmoid(\alpha \cdot z_i), z_i \in \mathbb{R}$, with $\alpha$ being a constant such that $\alpha \gg 1$, and then proceed with your favorite optimization method for finding $z_i$, and from there $x_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This, of course, may present problems, as $x_i$ are no longer bounded in $\{0, 1\}$, but may suffice your use case.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-02-07T15:13:24.363" LastActivityDate="2017-02-07T15:13:24.363" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16811" PostTypeId="2" ParentId="16643" CreationDate="2017-02-07T15:01:50.757" Score="5" Body="&lt;p&gt;Logistic regression in R uses the iterative re-weighted least squares algorithm. You can specify the maximum iterations and accuracy with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m &amp;lt;- glm(..., family = &quot;binomial&quot;, control = list(maxit = 2, epsilon=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please read the documentation for glm.control &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.control.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check the actual number of iterations used in fitting:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m$iter&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BUT...&lt;/strong&gt; The default settings should be enough.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; glm.control()&#xA;&#xA;$epsilon&#xA;[1] 1e-08&#xA;&#xA;$maxit&#xA;[1] 25&#xA;&#xA;$trace&#xA;[1] FALSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The default maximum number of iterations is 25, and I &lt;strong&gt;**doubt**&lt;/strong&gt; you will get anything by changing it to anything larger. The accuracy is 1e-08, which is already &lt;strong&gt;very small&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;strong&gt;shouldn't&lt;/strong&gt; blindly adjust the iteration number, most likely it won't help. Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; glm(admit ~ gre + gpa + rank, data = mydata, family = &quot;binomial&quot;, control = list(trace=TRUE))&#xA;&#xA;Deviance = 458.875865177 Iterations - 1&#xA;Deviance = 458.517518881 Iterations - 2&#xA;Deviance = 458.517492476 Iterations - 3&#xA;Deviance = 458.517492476 Iterations - 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got the dataset from &lt;a href=&quot;http://www.ats.ucla.edu/stat/r/dae/logit.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.ats.ucla.edu/stat/r/dae/logit.htm&lt;/a&gt; (because I don't have your data file).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you see I have &lt;code&gt;trace=TRUE&lt;/code&gt;? I print off the deviance for each iteration. R stops because the deviance difference between iteration 3 and 4 is too small. My model only requires 4 iterations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Print your deviance for each iteration and convince yourself you don't need more than 25 iterations&lt;/li&gt;&#xA;&lt;li&gt;Check your model &lt;strong&gt;properly&lt;/strong&gt;. Do you need more independent variables? Do you need to transform your variables to improve prediction accuracy?&lt;/li&gt;&#xA;&lt;li&gt;Ask yourself is 60% accuracy enough? In many fields 60% is a huge success. &lt;/li&gt;&#xA;&lt;li&gt;Does your model &lt;strong&gt;require interaction&lt;/strong&gt;? Note a saturated model will give you prefect accuracy, but it has all the interaction terms. Your model assumes the effects of your independent variables are additive in predicting the log-odd. Is this a fair assumption?&lt;/li&gt;&#xA;&lt;li&gt;Do you want to run automatic stepwise model selection algorithm in R? You might get a better model than manually done it yourself.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some useful links you might want to consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/5354/logistic-regression-model-does-not-converge&quot;&gt;https://stats.stackexchange.com/questions/5354/logistic-regression-model-does-not-converge&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/60958/logistic-regression-fishers-scoring-iterations-do-not-match-the-selected-itera&quot;&gt;https://stats.stackexchange.com/questions/60958/logistic-regression-fishers-scoring-iterations-do-not-match-the-selected-itera&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-08T02:07:42.933" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16813" PostTypeId="2" ParentId="16797" CreationDate="2017-02-07T16:19:09.817" Score="41" Body="&lt;p&gt;&lt;code&gt;mAP@[.5:.95]&lt;/code&gt;(someone denoted &lt;code&gt;mAP@[.5,.95]&lt;/code&gt;) means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;There is&#xA;an associated MS COCO challenge with a new evaluation&#xA;metric, that averages mAP over different IoU thresholds,&#xA;from 0.5 to 0.95 (written as “0.5:0.95”). [&lt;a href=&quot;https://www.cs.cornell.edu/%7Esbell/pdf/cvpr2016-ion-bell.pdf&quot; rel=&quot;noreferrer&quot;&gt;Ref&lt;/a&gt;]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We evaluate the&#xA;mAP averaged for IoU ∈ [0.5 : 0.05 : 0.95] (COCO’s&#xA;standard metric, simply denoted as mAP@[.5, .95])&#xA;and mAP@0.5 (PASCAL VOC’s metric). [&lt;a href=&quot;https://arxiv.org/pdf/1506.01497.pdf&quot; rel=&quot;noreferrer&quot;&gt;Ref&lt;/a&gt;]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;To evaluate our final detections, we use the official&#xA;COCO API [20], which measures mAP averaged over IOU&#xA;thresholds in [0.5 : 0.05 : 0.95], amongst other metrics. [&lt;a href=&quot;https://arxiv.org/pdf/1611.10012.pdf&quot; rel=&quot;noreferrer&quot;&gt;Ref&lt;/a&gt;]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;BTW, the &lt;a href=&quot;https://github.com/pdollar/coco/blob/master/PythonAPI/pycocotools/cocoeval.py#L501&quot; rel=&quot;noreferrer&quot;&gt;source code&lt;/a&gt; of &lt;a href=&quot;https://github.com/pdollar/coco&quot; rel=&quot;noreferrer&quot;&gt;coco&lt;/a&gt; shows exactly what &lt;code&gt;mAP@[.5:.95]&lt;/code&gt; is doing:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/pdollar/coco&quot; rel=&quot;noreferrer&quot;&gt;cocoapi&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.cs.cornell.edu/%7Esbell/pdf/cvpr2016-ion-bell.pdf&quot; rel=&quot;noreferrer&quot;&gt;Inside-Outside Net: Detecting Objects in Context with Skip Pooling and&#xA;Recurrent Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1506.01497.pdf&quot; rel=&quot;noreferrer&quot;&gt;Faster R-CNN: Towards Real-Time Object&#xA;Detection with Region Proposal Networks&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1611.10012.pdf&quot; rel=&quot;noreferrer&quot;&gt;Speed/accuracy trade-offs for modern convolutional object detectors&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="98307" LastEditDate="2020-08-20T18:45:48.813" LastActivityDate="2020-08-20T18:45:48.813" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16814" PostTypeId="2" ParentId="16807" CreationDate="2017-02-07T17:34:17.467" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Correct (although I would call it &quot;weight update step&quot;)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;and epoch is a full turn when each of the training instances have been seen by the model&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Correct&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a &quot;max batch&quot; by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Well, pretty much that. You usually don't have enough memory. Lets say we are talking about image classification. ImageNet is a wildly popular dataset. For quite a while, VGG-16D was one of the most popular mod.els. It needs calculcate 15 245 800 floats (in the feature maps) for one 224x224 image. This means about 61MB per image. This is just a rough lower bound on how much memory you need during training for each image.  ImageNet contains several thousand (I think about 1.2 million?) images. While you might have that much main memory, you certainly do not have that much GPU memory. I've seen GPU speeding up things to about 21x. So you definitely want to use the GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also: The time for one mini-batch is much lower. So the question is: Would you rather do n update steps with mini-batch per hour on a GPU or m update steps with batch without GPU, where n &gt;&gt; m.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-02-07T17:34:17.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16815" PostTypeId="1" CreationDate="2017-02-07T17:55:26.707" Score="1" ViewCount="1758" Body="&lt;p&gt;Is there a special header structure required for text classification problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to do Tweet classification using the text mining plugin. I have the data in a CSV file, with one column labeled category and containing my two categories, and the other labeled text containing the tweets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I load the file with the corpus widget and follow the workflow given at &lt;a href=&quot;http://orange3-text.readthedocs.io/en/latest/widgets/bagofwords.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://orange3-text.readthedocs.io/en/latest/widgets/bagofwords.html&lt;/a&gt; the test widget gives the error &quot;Train data input requires a class variable.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I opened the sample bookexcerpts.tab data and discovered it has a different header structure than just category and text.  If I modify my tweet CSV to match it seems to work. Is this a bug, or am I missing a step?&lt;/p&gt;&#xA;" OwnerUserId="28726" LastActivityDate="2017-03-09T22:47:56.733" Title="Orange Text Mining Data Format" Tags="|text-mining|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16817" PostTypeId="1" CreationDate="2017-02-07T20:07:00.923" Score="8" ViewCount="10161" Body="&lt;p&gt;My question is what are potential reasons for Naive Bayes to perform well on a train set but poorly on a test set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working with a variation of the &lt;a href=&quot;http://qwone.com/~jason/20Newsgroups/&quot; rel=&quot;nofollow noreferrer&quot;&gt;20news dataset&lt;/a&gt;. The dataset has documents, which are represented as &quot;bag of words&quot; with no metadata. My goal is to classify each document into 1 of 20 labels. My error rate on training data is 20%, but my error rate on test data is 90% (for comparison, guessing yields 95% error rate). For some reason, my classifier is predicting class 16 for almost all documents in test set. In train set this problem does not occur. Furthermore, this issue persists with different train/test splits. I'm trying to figure out what I'm doing wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some things I've considered:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).&lt;/li&gt;&#xA;&lt;li&gt;Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.&lt;/li&gt;&#xA;&lt;li&gt;Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.&lt;/li&gt;&#xA;&lt;li&gt;Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.&lt;/li&gt;&#xA;&lt;li&gt;Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.&lt;/li&gt;&#xA;&lt;li&gt;Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Edit: Turns out I had an implementation bug. I won't detail it here as it will be unlikely to help anyone with similar issues.&lt;/p&gt;&#xA;" OwnerUserId="23480" LastEditorUserId="23480" LastEditDate="2017-03-18T20:38:44.463" LastActivityDate="2018-01-30T21:58:12.273" Title="Overfitting Naive Bayes" Tags="|classification|naive-bayes-classifier|overfitting|variance|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16818" PostTypeId="2" ParentId="16807" CreationDate="2017-02-07T20:29:06.160" Score="73" Body="&lt;p&gt;The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent&lt;a href=&quot;https://arxiv.org/abs/1804.07612&quot; rel=&quot;noreferrer&quot;&gt;1&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point).&lt;/p&gt;&#xA;&lt;p&gt;In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a &lt;em&gt;single&lt;/em&gt; instance of the dataset. Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in [2]). The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution.&lt;/p&gt;&#xA;&lt;p&gt;The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1804.07612&quot; rel=&quot;noreferrer&quot;&gt;1&lt;/a&gt; Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 (pp. 177-186). Physica-Verlag HD.&lt;/p&gt;&#xA;&lt;p&gt;[2] Ge, R., Huang, F., Jin, C., &amp;amp; Yuan, Y. (2015, June). Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. In COLT (pp. 797-842).&lt;/p&gt;&#xA;&lt;h2&gt;EDIT :&lt;/h2&gt;&#xA;&lt;p&gt;I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.)&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Training with large minibatches is bad for your health.&#xA;More importantly, it's bad for your test error.&#xA;Friends dont let friends use minibatches larger than 32.&#xA;Let's face it: the &lt;em&gt;only&lt;/em&gt; people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,&lt;/p&gt;&#xA;&lt;p&gt;Dominic Masters, Carlo Luschi, &lt;a href=&quot;https://arxiv.org/abs/1804.07612&quot; rel=&quot;noreferrer&quot;&gt;Revisiting Small Batch Training for Deep Neural Networks&lt;/a&gt;, arXiv:1804.07612v1&lt;/p&gt;&#xA;&lt;p&gt;From the abstract,&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance ...&lt;/p&gt;&#xA;&lt;p&gt;The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-27T16:48:40.700" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16819" PostTypeId="1" AcceptedAnswerId="16824" CreationDate="2017-02-07T20:43:14.833" Score="0" ViewCount="816" Body="&lt;p&gt;I am trying to learn Neural Networks using &lt;a href=&quot;http://scikit-neuralnetwork.readthedocs.io/en/latest/guide_model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-neuralnetwork&lt;/a&gt; framework and I know basics about Neural Networks and now trying to implement it with scikit-learn. but I am confused on 2 points.&lt;/p&gt;&#xA;&lt;p&gt;1- what is the structure of this NN given below? Somehow, in some examples felt to me, some people don't put input layer as a layer. Otherwise, I am thinking this as a 2 layer NN has input layer with 100 nodes and 1 node at the ouput layer.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sknn.mlp import Classifier, Layer&#xA;&#xA;nn = Classifier(&#xA;   layers=[&#xA;    Layer(&amp;quot;Maxout&amp;quot;, units=100, pieces=2),&#xA;    Layer(&amp;quot;Softmax&amp;quot;)],&#xA;learning_rate=0.001,&#xA;n_iter=25)&#xA;&#xA;nn.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;2- Does scikit-neuralnetwork do back propagation within the code that I put above?&lt;/p&gt;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="28732" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-07T23:35:37.910" Title="Back propagation and Structure of a Neural Network in scikit-neuralnetwork" Tags="|machine-learning|python|neural-network|deep-learning|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16820" PostTypeId="2" ParentId="16805" CreationDate="2017-02-07T22:36:22.523" Score="0" Body="&lt;p&gt;Another option could be to model your problem as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_linear_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian linear regression&lt;/a&gt; where $x_i$ are random variables that follow a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bernoulli_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bernoulli distribution&lt;/a&gt; with probability $p_i$, that is, $x_i \sim Bernoulli (p_i)$, and $v \sim \mathcal{N}(x^T b, \sigma^2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The posterior distribution of $p_i$ would enable you to tell the values of $x_i$ that most likely fit with the observed data. These posterior distributions may be inferred via &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo&quot; rel=&quot;nofollow noreferrer&quot;&gt;MCMC&lt;/a&gt; or &lt;a href=&quot;https://arxiv.org/abs/1603.00788&quot; rel=&quot;nofollow noreferrer&quot;&gt;variational inference&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The three main frameworks to implement bayesian models in Python are &lt;a href=&quot;https://github.com/pymc-devs/pymc3&quot; rel=&quot;nofollow noreferrer&quot;&gt;pymc3&lt;/a&gt; (&lt;a href=&quot;http://blog.applied.ai/bayesian-inference-with-pymc3-part-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;), &lt;a href=&quot;http://edwardlib.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;edward&lt;/a&gt; (&lt;a href=&quot;https://github.com/blei-lab/edward/blob/master/examples/bayesian_linear_regression.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;) and &lt;a href=&quot;https://pystan.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pystan&lt;/a&gt; (&lt;a href=&quot;https://climateecology.wordpress.com/2014/01/27/pystan-a-basic-tutorial-of-bayesian-data-analysis-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;). The three of them allow straighforward implementation of bayesian linear regression as formulated above.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-02-07T22:36:22.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16822" PostTypeId="2" ParentId="16815" CreationDate="2017-02-07T22:40:38.377" Score="3" Body="&lt;p&gt;No, no special structure is required for text classification problems in comparison with core Orange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However — and this holds for core Orange as well as for text addon — if you would like to perform classification one feature must be set as class variable. Otherwise Orange cannot know which feature you would like to predict. When you are reading the data from CSV by default no feature is selected as class variable and hence the message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two possible approaches how one can mark a feature as class variable. The more permanent one, as you already figured, is to modify the data file itself. Second, a bit more user friendly one, is to do this in Orange. To do so you just pass the data to Select Columns widget, drag the desired feature to the Target Variable field and click Send. The data on the output of the widget will have a class variable and can be passed to Test &amp;amp; Score. The following scheme should work without modifying the header of a file:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lQOK2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lQOK2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to know more about how the Select Columns widget works you can also check &lt;a href=&quot;http://docs.orange.biolab.si/3/visual-programming/widgets/data/selectcolumns.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;its documentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in a nutshell you were missing a step and what happened is expected. If the header does not state that some feature is a class variable — as was the case for your CSV — Orange simply won't mark any feature as class variable. We don't use any heuristics to guess what might be a class variable but solely rely on what is written in the file.&lt;/p&gt;&#xA;" OwnerUserId="15685" LastActivityDate="2017-02-07T22:40:38.377" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16824" PostTypeId="2" ParentId="16819" CreationDate="2017-02-07T23:35:37.910" Score="1" Body="&lt;p&gt;1) &#xA;From what I understand, scikit-neuralnetwork tries to automatically determine the correct input and output sizes by the X and y data you give it when  calling &lt;code&gt;nn.fit&lt;/code&gt;. Therefore structure should be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Input layer with shape determined by X_train &lt;/li&gt;&#xA;&lt;li&gt;Dense layer with 100 units and maxout activation with 2 linear pieces&lt;/li&gt;&#xA;&lt;li&gt;Softmax classification layer with as many units as needed for y_train &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Seems to use input shape from data here:&#xA;&lt;a href=&quot;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L183&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L183&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And output shape from data here:&#xA;&lt;a href=&quot;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/mlp.py#L62&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/mlp.py#L62&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;However, note that maxout seems no longer supported&lt;/strong&gt;: &lt;a href=&quot;https://github.com/aigamedev/scikit-neuralnetwork/issues/142&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aigamedev/scikit-neuralnetwork/issues/142&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Yes it uses backpropagation by calling appropriate lasagne/theano functions to create/compile the backpropagation training function:&#xA;&lt;a href=&quot;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L50-L103&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L50-L103&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Actual training seems to happen here: &lt;a href=&quot;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L316-L335&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L316-L335&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="27704" LastActivityDate="2017-02-07T23:35:37.910" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16825" PostTypeId="1" AcceptedAnswerId="16827" CreationDate="2017-02-08T02:46:51.497" Score="0" ViewCount="162" Body="&lt;p&gt;Is there a way to reload all attributes after having removed someones without reopening the data file ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2JIIM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2JIIM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help please ?&lt;/p&gt;&#xA;" OwnerUserId="9165" LastActivityDate="2017-02-08T07:56:03.530" Title="How to reload all attributes in WEKA" Tags="|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16826" PostTypeId="1" AcceptedAnswerId="16830" CreationDate="2017-02-08T06:16:55.980" Score="9" ViewCount="37394" Body="&lt;p&gt;Are there any machine learning libraries for C.&#xA;Specifically interested in unsupervised learning.&lt;/p&gt;&#xA;" OwnerUserId="28746" LastActivityDate="2017-02-08T08:52:37.197" Title="Is there a C library for machine learning algorithms?" Tags="|machine-learning|beginner|c|" AnswerCount="2" CommentCount="4" ClosedDate="2017-02-08T20:36:01.867" ContentLicense="CC BY-SA 3.0" />
  <row Id="16827" PostTypeId="2" ParentId="16825" CreationDate="2017-02-08T07:56:03.530" Score="0" Body="&lt;p&gt;Judging from the screenshot, you are currently looking at the data in the preproces tab from the explorer module. In the menu above the top menu in your screenshot should be an undo option (5th option from the left).&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2017-02-08T07:56:03.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16828" PostTypeId="2" ParentId="16826" CreationDate="2017-02-08T08:25:57.553" Score="4" Body="&lt;p&gt;If you want to use SVMs: &lt;a href=&quot;https://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;libsvm&lt;/code&gt;&lt;/a&gt; is written in C.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-02-08T08:25:57.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16829" PostTypeId="2" ParentId="12865" CreationDate="2017-02-08T08:26:20.983" Score="0" Body="&lt;p&gt;What you are looking for might be a corpus. In Python, there is the Natural Language Toolkit (NLTK). This also has a corpus containing treebanks such as Brown etc. In the corpus folder, there is also textblob and other words/phrases which can be used. You would have to import it though...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for nltk.corpus import TextBlob&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might want to read on Corpora&lt;/p&gt;&#xA;" OwnerUserId="28750" LastActivityDate="2017-02-08T08:26:20.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16830" PostTypeId="2" ParentId="16826" CreationDate="2017-02-08T08:52:37.197" Score="5" Body="&lt;p&gt;Here's a detailed table about different machine learning libraries on different languages:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/josephmisiti/awesome-machine-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Checkout C version &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning#c&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and C++ version &lt;a href=&quot;https://github.com/josephmisiti/awesome-machine-learning#cpp&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally speaking, try &lt;a href=&quot;http://docs.opencv.org/2.4/index.html&quot; rel=&quot;noreferrer&quot;&gt;OpenCV&lt;/a&gt; ! OpenCV provides multiple machine learning implementations including &lt;a href=&quot;http://docs.opencv.org/2.4/modules/core/doc/clustering.html#kmeans&quot; rel=&quot;noreferrer&quot;&gt;KMeans&lt;/a&gt;, &lt;a href=&quot;http://docs.opencv.org/2.4/modules/ml/doc/k_nearest_neighbors.html&quot; rel=&quot;noreferrer&quot;&gt;kNN&lt;/a&gt;, &lt;a href=&quot;http://docs.opencv.org/2.4.13/modules/ml/doc/support_vector_machines.html&quot; rel=&quot;noreferrer&quot;&gt;SVM&lt;/a&gt;, etc.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-08T08:52:37.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16831" PostTypeId="1" AcceptedAnswerId="16833" CreationDate="2017-02-08T09:27:50.197" Score="0" ViewCount="61" Body="&lt;p&gt;I had no idea how to explain this in the title but anyway... Let's say I have a data points like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;John, Happy | Greedy | Smart | Funny, 0.8&#xA;Ann, Smart | Sad | Funny, 0.6&#xA;Joel, Greedy | Prideful | Stupid, 0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where the first part is the name the second is there characteristics and the third is their overall character score (how nice they are to be around or something). Is there a good way to work with this data so that I can work out what the best possible combinations are? Assume I have a large enough data set. There may be say 30 of those characteristics and any person can have any number of them and every characteristic is equally valid to every person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully that explains it. Essentially I am want a way to organise the traits so that I can say &quot;smart and happy&quot; make a better combination than &quot;sad and greedy&quot;. I also need to be able to asses the ultimate combination and be able to compare any two possible combinations.&lt;/p&gt;&#xA;" OwnerUserId="28753" LastActivityDate="2017-02-08T09:45:21.483" Title="ideas for variable with branching data?" Tags="|machine-learning|dataset|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16832" PostTypeId="2" ParentId="16831" CreationDate="2017-02-08T09:38:07.847" Score="1" Body="&lt;p&gt;The typical way to address this is to transform those &lt;em&gt;characteristics&lt;/em&gt; into a representation where you compose a vector with one component for each possible characteristic, with a $1$ in the position of each characteristic owned by the individual, or zero otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, you would have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;name | greedy | smart | funny | sad | prideful | stupid | ... | score&#xA;----------------------------------------------------------------------&#xA;John |   1    |   1   |   0   |  0  |    0     |   0    | ... | 0.8&#xA;Ann  |   0    |   1   |   1   |  1  |    0     |   0    | ... | 0.6&#xA;Joel |   1    |   0   |   0   |  0  |    1     |   1    | ... | 0.2&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-02-08T09:38:07.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16833" PostTypeId="2" ParentId="16831" CreationDate="2017-02-08T09:45:21.483" Score="0" Body="&lt;p&gt;It's a regression machine learning problem. Assuming you have 30 characteristics, one-hot encoded into 30 columns. And your target is the character score, min-max scaled into $[0, 1]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we have &lt;code&gt;X.shape=(None, 30)&lt;/code&gt;, &lt;code&gt;Y.shape=(None,)&lt;/code&gt; (just like what ncasas has stated), thus we can train a regression model using your favorite algorithm (linear regression, random forest, even neural network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After we have this model working, as each person has and only has 3 characters, we can predict the character score for each character combination one by one. The time complexity is roughly $O(n^3)$. However, $n$ is small in your case, so maybe we can just brute-force every score on every combination. That's what you want.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-08T09:45:21.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16835" PostTypeId="1" AcceptedAnswerId="18073" CreationDate="2017-02-08T10:26:16.887" Score="28" ViewCount="9853" Body="&lt;p&gt;Both &lt;a href=&quot;https://github.com/hughperkins/pytorch&quot; rel=&quot;noreferrer&quot;&gt;PyTorch&lt;/a&gt; and &lt;a href=&quot;https://github.com/tensorflow/fold&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow Fold&lt;/a&gt; are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are useful or needed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know how they compare, in the sense of paradigms they rely on (e.g. dynamic batching) and their implications, things that can/cannot be implemented in each one, weaknesses/strengths, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I intend to use this info to choose one of them to start exploring dynamic computation graphs, but I have no specific task in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 1: other dynamic computation graph frameworks like &lt;a href=&quot;https://github.com/clab/dynet&quot; rel=&quot;noreferrer&quot;&gt;DyNet&lt;/a&gt; or &lt;a href=&quot;https://github.com/pfnet/chainer&quot; rel=&quot;noreferrer&quot;&gt;Chainer&lt;/a&gt; are also welcome in the comparison, but I'd like to focus on PyTorch and Tensorflow Fold because I think they are/will be the most used ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 2: I have found &lt;a href=&quot;https://news.ycombinator.com/item?id=13428098&quot; rel=&quot;noreferrer&quot;&gt;this hackernews thread on PyTorch&lt;/a&gt; with some sparse info, but  not much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 3: Another relevant &lt;a href=&quot;https://news.ycombinator.com/item?id=13591578&quot; rel=&quot;noreferrer&quot;&gt;hackernews thread&lt;/a&gt;, about Tensorflow Fold, that contains some info about how they compare.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 4: relevant &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/5sn987/n_announcing_tensorflow_fold_deep_learning_with/&quot; rel=&quot;noreferrer&quot;&gt;Reddit thread&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 5: &lt;a href=&quot;https://github.com/tensorflow/fold/issues/8&quot; rel=&quot;noreferrer&quot;&gt;relevant bug in Tensorflow Fold's github&lt;/a&gt; that identifies an important limitation: impossibility to do conditional branching during evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note 6: &lt;a href=&quot;https://discuss.pytorch.org/t/about-the-variable-length-input-in-rnn-scenario/345&quot; rel=&quot;noreferrer&quot;&gt;discussion on pytorch forum&lt;/a&gt; about variable length inputs in relation to the algorithms used (e.g. dynamic batching).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-02-23T11:31:56.133" LastActivityDate="2017-05-03T12:43:48.440" Title="PyTorch vs. Tensorflow Fold" Tags="|python|deep-learning|tensorflow|pytorch|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16839" PostTypeId="1" CreationDate="2017-02-08T18:16:17.203" Score="4" ViewCount="263" Body="&lt;p&gt;I have the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A company, let's call it X, has a job-adds website. It works as a marketplace where job-seekers and companies that have job vacancies can meet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their business model is such that job-seekers pay a subscription to get access to the website after a trial period. Companies, on the other hand, don't pay anything to post job-adds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to attract companies to post jobs on the website they call them and ask if they're interested. They have a database with &lt;em&gt;N&lt;/em&gt; companies that did business with them in the past. Out of those &quot;N&quot; they can only call 20% due to budget and personnel constraints. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They select those 20% out of simple heuristic: companies that posted job adds in the last six months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What they want is to call companies that have high probability of posting a job add that will generate a match. They consider that a match has happened based on a proxy: the company contacted n different candidates for a particular job-add. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find a data-driven approach to select the best companies for them to call. In other words, provide them with a list of companies that have a high probability of generating adds that match with their current database of users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I'm having is how to transform this business problem into a machine learning solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have thought so far is to build a classification model. Each row is a job-add. The features are variables related to the company (size, location), to the user (age, CV, last time logged in), and the job-add itself (salary,education required, position, industry). The target variable is a binary of &quot;the job had a mach or not.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with my approach is that the model is not actually learning anything since my target variable is already known for each job-post (I know which job-adds had a match and which didn't). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for some ideas on how I can structure this problem in a data-driven framework. Should I continue with my classification model? Maybe some type of collaborative filtering? I'm sure machine learning can provide better results than their simple heuristic. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;User data, including CV;&lt;/li&gt;&#xA;&lt;li&gt;Who applied to what job and when;&lt;/li&gt;&#xA;&lt;li&gt;Data from the companies that posted job-adds;&lt;/li&gt;&#xA;&lt;li&gt;Contacts made by companies to job-seekers (This just tells me if the company clicked a button to look at the user's contact info);&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28599" LastEditorUserId="28599" LastEditDate="2017-02-09T07:10:52.950" LastActivityDate="2017-02-09T13:13:17.490" Title="Translating a business problem into a machine learning solution: job-adds website" Tags="|machine-learning|data-mining|recommender-system|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16840" PostTypeId="1" CreationDate="2017-02-08T19:35:54.180" Score="5" ViewCount="25089" Body="&lt;p&gt;During training, the neural net settles into a place where it always predicts 1 of the 5 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My train and test sets are distributed as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train Set&#xA;Samples: 269,501. Features: 157&#xA;Data distribution&#xA;16.24% 'a'&#xA;39.93% 'b'&#xA;9.31%  'c'&#xA;20.86% 'd'&#xA;13.67% 'e'&#xA;&#xA;Test Set&#xA;Samples: 33,967. Features: 157&#xA;Data distribution&#xA;10.83% 'a'&#xA;35.39% 'b'&#xA;19.86% 'c'&#xA;16.25% 'd'&#xA;17.66% 'e'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the percentages of class &lt;code&gt;b&lt;/code&gt;! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training an mlp with dropout, and training &lt;em&gt;and&lt;/em&gt; test (aka validation) accuracies both plateau, perfectly matching the train and test distributions of 1 of my 5 classes, i.e. it is learning to &lt;em&gt;always predict 1 class out of 5 classes&lt;/em&gt;! I've verified the classifier is always predicting &lt;code&gt;b&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’ve tried &lt;code&gt;batch_size&lt;/code&gt; of 0.25 and 1.0 and made double-y sure the data was shuffled the data. I tried both &lt;code&gt;SGD&lt;/code&gt; and &lt;code&gt;Adam&lt;/code&gt; optimizers with and without decay and different learning rates and still the same result. Tried dropout of 0.2 and 0.5. &lt;code&gt;EarlyStopping&lt;/code&gt; of 300 epochs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every so often I'll get a situation where during training it'll pop out of where it has settled for training accuracy and validation accuracy but then validation always goes down and training goes up -- or in other words, overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output, cut off after 6 epochs. It doesn't always converge this fast, just with this particular SGD optimizer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/2000&#xA;Epoch 00000: val_acc improved from -inf to 0.35387, saving model to /home/user/src/thing/models/weights.hdf&#xA;269501/269501 [==============================] - 0s - loss: 1.6094 - acc: 0.1792 - val_loss: 1.6073 - val_acc: 0.3539&#xA;Epoch 2/2000&#xA;Epoch 00001: val_acc did not improve&#xA;269501/269501 [==============================] - 0s - loss: 1.6060 - acc: 0.3993 - val_loss: 1.6042 - val_acc: 0.3539&#xA;Epoch 3/2000&#xA;Epoch 00002: val_acc did not improve&#xA;269501/269501 [==============================] - 0s - loss: 1.6002 - acc: 0.3993 - val_loss: 1.6005 - val_acc: 0.3539&#xA;Epoch 4/2000&#xA;Epoch 00003: val_acc did not improve&#xA;269501/269501 [==============================] - 0s - loss: 1.5930 - acc: 0.3993 - val_loss: 1.5967 - val_acc: 0.3539&#xA;Epoch 5/2000&#xA;Epoch 00004: val_acc did not improve&#xA;269501/269501 [==============================] - 0s - loss: 1.5851 - acc: 0.3993 - val_loss: 1.5930 - val_acc: 0.3539&#xA;Epoch 6/2000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code:&#xA;Model creation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_mlp(input_dim, output_dim, dropout=0.5, arch=None):&#xA;    &quot;&quot;&quot;Setup neural network model (keras.models.Sequential)&quot;&quot;&quot;&#xA;    # default mlp architecture&#xA;    arch = arch if arch else [64,32,32,16]&#xA;&#xA;    # setup densely connected NN architecture (MLP)&#xA;    model = Sequential()&#xA;    model.add(Dropout(dropout, input_shape=(input_dim,)))&#xA;    for output in arch:&#xA;        model.add(Dense(output, activation='relu', W_constraint=maxnorm(3)))&#xA;        model.add(Dropout(dropout))&#xA;    model.add(Dense(output_dim, activation='sigmoid'))&#xA;&#xA;    # compile model and save architecture to disk&#xA;    sgd = SGD(lr=0.01, momentum=0.9, decay=0.0001, nesterov=True)&#xA;    # adam = Adam(lr=0.001, decay=0.0001)&#xA;    model.compile(loss='categorical_crossentropy',&#xA;                  optimizer=sgd,&#xA;                  metrics=['accuracy'])&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And inside main after some preprocessing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    # labels must be one-hot encoded for loss='categorical_crossentropy'&#xA;    # meaning, of possible labels 0,1,2: 0-&amp;gt;[1,0,0]; 1-&amp;gt;[0,1,0]; 2-&amp;gt;[0,0,1]&#xA;    y_train_onehot = to_categorical(y_train, n_classes)&#xA;    y_test_onehot = to_categorical(y_test, n_classes)&#xA;&#xA;    # get neural network architecture and save to disk&#xA;    model = create_mlp(input_dim=train_dim, output_dim=n_classes)&#xA;    with open(clf_file(typ='arch'), 'w') as f:&#xA;        f.write(model.to_yaml())&#xA;&#xA;    # output logs to tensorflow TensorBoard&#xA;    # NOTE: don't use param histogram_freqs until keras issue fixed&#xA;    #       https://github.com/fchollet/keras/pull/5175&#xA;    tensorboard = TensorBoard(log_dir=opts.tf_dir)&#xA;&#xA;    # only save model weights for best performing model&#xA;    checkpoint = ModelCheckpoint(clf_file(typ='weights'),&#xA;                                 monitor='val_acc',&#xA;                                 verbose=1,&#xA;                                 save_best_only=True)&#xA;&#xA;    # stop training early if validation accuracy doesn't improve for long enough&#xA;    early_stopping = EarlyStopping(monitor='val_acc', patience=300)&#xA;&#xA;    # shuffle data for good measure before fitting&#xA;    x_train, y_train_onehot = shuffle(x_train, y_train_onehot)&#xA;&#xA;    np.random.seed(seed)&#xA;    model.fit(x_train, y_train_onehot,&#xA;              nb_epoch=opts.epochs,&#xA;              batch_size=train_batch_size,&#xA;              shuffle=True,&#xA;              callbacks=[tensorboard, checkpoint, early_stopping],&#xA;              validation_data=(x_test,y_test_onehot))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25115" LastActivityDate="2020-07-23T14:00:46.547" Title="Multi-class neural net always predicting 1 class after optimization" Tags="|python|neural-network|keras|multiclass-classification|multilabel-classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16842" PostTypeId="2" ParentId="15670" CreationDate="2017-02-09T00:27:43.593" Score="2" Body="&lt;p&gt;Mapping your inputs into a sparse matrix and using logistic regression with an $\ell_1$ penalty should work well. In R this is very straight forward with the glmnet and matrix libraries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, if you have boolean features with a low frequency, you can filter some of them by taking a subset with at least some number of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a quick simulation in R of a lasso logistic regression with sparse features. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(glmnet)&#xA;library(Matrix)&#xA;&#xA;n &amp;lt;- 1e5&#xA;set.seed(420)&#xA;&#xA;# These are all converted to boolean variables since they're arbitrary categories&#xA;ls &amp;lt;- data.frame(cat1=sample(letters, n, replace=TRUE), &#xA;                 cat2=sample(letters, n, replace=TRUE), &#xA;                 cat3=sample(letters, n, replace=TRUE))&#xA;ls$cat1cat2 &amp;lt;- factor(paste(ls$cat1, ls$cat2, sep=&quot;-&quot;))&#xA;xs &amp;lt;- sparse.model.matrix(~.-1,data=ls)&#xA;betas&amp;lt;- rnorm(dim(xs)[2])&#xA;error &amp;lt;- runif(n)&#xA;&#xA;ytrue &amp;lt;- xs %*%betas&#xA;ylogit &amp;lt;- 1/(1 + exp(- (ytrue + error))&#xA;ylabel &amp;lt;- ifelse(ylogit&amp;gt;0.5,1,0)&#xA;&#xA;trainfilter &amp;lt;- runif(n)&amp;lt;=0.3&#xA;sparsemodel &amp;lt;- cv.glmnet(x=xs[trainfilter,],&#xA;                         y=ylabel[trainfilter],&#xA;                         alpha=1,&#xA;                         nfolds=3,&#xA;                         family='binomial')&#xA;&#xA;preds &amp;lt;- predict(sparsemodel, xs, s='lambda.min', type='class')&#xA;print(table(ylabel, preds))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The classifier doesn't matter as much in this case as the underlying algorithm's ability to handle sparse matrices. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also worth noting that you can handle sparse boolean features by mapping them to arbitrary numeric indices and running them through a tree based algorithm (e.g., Random Forest or Gradient Boosting). In practice, this seems to work well.&lt;/p&gt;&#xA;" OwnerUserId="25755" LastEditorUserId="25755" LastEditDate="2018-02-11T15:19:44.720" LastActivityDate="2018-02-11T15:19:44.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16843" PostTypeId="1" CreationDate="2017-02-09T03:41:47.597" Score="9" ViewCount="16179" Body="&lt;p&gt;Today I've seen many Perceptron implementations with learning rates. According to &lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;there is no need for a learning rate in the perceptron algorithm.&#xA;  This is because multiplying the update by any constant simply rescales&#xA;  the weights but never changes the sign of the prediction&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is there some benefit to implementing a learning rate with Perceptron? If there is not, why do so many implementations have it?&lt;/p&gt;&#xA;" OwnerUserId="23480" LastEditorUserId="56981" LastEditDate="2018-08-14T11:43:12.650" LastActivityDate="2019-11-12T04:10:28.130" Title="Perceptron learning rate" Tags="|neural-network|perceptron|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16844" PostTypeId="2" ParentId="16843" CreationDate="2017-02-09T03:52:26.330" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The choice of learning rate m does not matter because it just changes&#xA;  the scaling of w.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I agree that it is just the scaling of &lt;code&gt;w&lt;/code&gt; which is done by the learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, as I have explained in &lt;a href=&quot;https://math.stackexchange.com/a/1548252/264808&quot;&gt;this answer&lt;/a&gt;, the magnitude of learning rate does play a part in the accuracy of the perceptron.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2017-02-09T03:52:26.330" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16846" PostTypeId="2" ParentId="16843" CreationDate="2017-02-09T04:27:16.413" Score="6" Body="&lt;p&gt;I agree with &lt;a href=&quot;https://datascience.stackexchange.com/a/16844/29575&quot;&gt;Dawny33&lt;/a&gt;, choosing learning rate only scales w.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While training of Perceptron we are trying to determine minima and choosing of learning rate helps us determine how fast we can reach that minima. If we choose larger value of learning rate then we might overshoot that minima and smaller values of learning rate might take long time for convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is okay in case of Perceptron to neglect learning rate because Perceptron algorithm guarantees to find a solution (if one exists) in an upperbound number of steps, in other implementations it is not the case so learning rate becomes a necessity in them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be useful in Perceptron algorithm to have learning rate but it's not a necessity.                  &lt;/p&gt;&#xA;" OwnerUserId="17003" LastEditorUserId="29575" LastEditDate="2018-08-14T10:11:03.423" LastActivityDate="2018-08-14T10:11:03.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16847" PostTypeId="2" ParentId="16840" CreationDate="2017-02-09T07:16:15.117" Score="7" Body="&lt;p&gt;It could be a bug in your code, problems with your training set (maybe you don't have the file format quite right), or some other implementation issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you sure you want to use a sigmoid activation function in your last layer?  I would have expected that the normal approach would be to use a softmax as the last layer (so that you can treat the outputs as the probability of each class, i.e., so that they're normalized to sum to 1).  You might try that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, this might be a 'class imbalance' problem.  Do some searching on that term and you'll find a bunch of standard methods for dealing with it.  You can balance the training set, or use 'weights' on the instances, or adjust the threshold based on the priors.  However, as others have pointed out, the imbalance is not severe enough that I would have expected it to cause this strong of an bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible that your features are useless and don't help predict the output (e.g., they are not related to or correlated to the output).  That would also be consistent with what you are seeing. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Side note: My understanding is that the Adam optimizer generally is more effective than plain SGD, though I don't see any reason to expect that to be the issue here. &lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2017-02-10T05:06:38.847" LastActivityDate="2017-02-10T05:06:38.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16848" PostTypeId="1" CreationDate="2017-02-09T10:51:22.247" Score="1" ViewCount="310" Body="&lt;p&gt;Most of the &lt;code&gt;scikit-learn&lt;/code&gt; metric functions have an option to take into account sample weights. What is the purpose or use case to use this options? Unbalanced classes?&lt;/p&gt;&#xA;" OwnerUserId="25774" LastActivityDate="2017-02-09T12:54:54.663" Title="Weights in scikit-learn metric functions" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16849" PostTypeId="1" CreationDate="2017-02-09T11:22:54.873" Score="0" ViewCount="209" Body="&lt;p&gt;I want to implement a system that takes as input scribbles (i.e. drawings with writings on paper, input to the algorithm is a photo of this) and outputs the forms / writings e.g. in Powerpoint. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What it the best algorithm to detect forms (e.g. this is a circle, this is a box, this is handwritten text and it writes xyz)? There will be multiple forms and writings within a single input scribble / photo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any hints!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Here is an example of a typical scribble. In this case conversion target would be powerpoint&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HkHs9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HkHs9.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28802" LastEditorUserId="28802" LastEditDate="2017-02-09T14:58:47.333" LastActivityDate="2017-02-09T14:58:47.333" Title="Algorithm to create graphics from scribble" Tags="|machine-learning|object-recognition|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16850" PostTypeId="1" AcceptedAnswerId="16855" CreationDate="2017-02-09T11:51:03.067" Score="7" ViewCount="14066" Body="&lt;p&gt;I'm having problem understanding how we choose the weight function. In Andrew Ng's notes, a method for calculating a local weight, a standard choice of weights is given by:&lt;a href=&quot;https://i.stack.imgur.com/kbB5e.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kbB5e.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;What I don't understand is, what exactly is the &lt;em&gt;x&lt;/em&gt; here? Apparently &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that the weights depend on the particular point x at which we’re trying&#xA;  to evaluate x.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I don't get it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the example of housing prices predicted by number of rooms and size in square feet. So each &lt;code&gt;x^(i)&lt;/code&gt; is a &lt;code&gt;[roomnum, size]&lt;/code&gt; array. So what's in &lt;code&gt;x&lt;/code&gt;? I guess that also should be a &lt;code&gt;[roomnum, size]&lt;/code&gt; array, but what's in it? Is it even a vector? Or is it the target variable? If so, why isn't it marked with &lt;em&gt;y&lt;/em&gt;? I don't get it, please help! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;Ok so what I want is to make create a regression line like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/efEaJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/efEaJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I choose the &lt;em&gt;x&lt;/em&gt;-es? What would they be in an algorithm? Do I have to make guesses for each &lt;em&gt;x&lt;/em&gt;? How can I produce a line like this? &lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="21254" LastEditDate="2017-02-09T13:08:16.763" LastActivityDate="2018-05-29T22:40:11.893" Title="Understanding Locally Weighted Linear Regression" Tags="|linear-regression|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16851" PostTypeId="2" ParentId="16848" CreationDate="2017-02-09T11:52:27.860" Score="1" Body="&lt;p&gt;Unbalanced classes is definitely one use case, another purpose is &lt;a href=&quot;http://cse-wiki.unl.edu/wiki/index.php/Cost_Sensitive_Learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;cost sensitive learning&lt;/a&gt;, where the cost of misclassifying classes is different for each class. In particular scikit-learn give weights for each sample so it allows a greater flexibility. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-02-09T11:52:27.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16852" PostTypeId="2" ParentId="16848" CreationDate="2017-02-09T12:54:54.663" Score="0" Body="&lt;p&gt;Sample weights change the probability of each sample to be fed into model training. For example, we have 4 samples &lt;code&gt;A, B, C, D&lt;/code&gt; with weights $1, 2, 3, 4$, it means our model has two times the probability to fit on sample &lt;code&gt;B&lt;/code&gt; than sample &lt;code&gt;A&lt;/code&gt;, and 3 times the probability to fit on sample &lt;code&gt;C&lt;/code&gt; than sample &lt;code&gt;A&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you wonder what's the purpose of doing that, let's imagine what our model will become when it's fed with different weights of samples.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example: Durian World&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If you're born in a country full of durian, when you grow up, the word &quot;fruit&quot; means something smells awful, but we all know that most fruit smells tasty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is machine learning models, when our model doesn't recognize durian, just feed more durian(increase weight as there are few durian in our life compared to apple or banana). That's the situation called &quot;unbalanced class&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When(and if) you hate durian very very much (assume you would die when you smelt durian), but you can't recognize durian well, how to keep you survived? Just feed more durian and non-durian samples, you might loss the capability to distinguish apple from banana, but you shall survive. That's the situation called &quot;cost sensitive&quot;.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-09T12:54:54.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16853" PostTypeId="2" ParentId="16850" CreationDate="2017-02-09T13:05:35.573" Score="2" Body="&lt;p&gt;As the &lt;a href=&quot;https://datajobs.com/data-science-repo/Generalized-Linear-Models-[Andrew-Ng].pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; has mentioned:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the original linear regression algorithm, to &lt;strong&gt;make a prediction at a query&#xA;  point&lt;/strong&gt; $x$ (i.e., to evaluate $h(x)$)...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$x$ you want is the exact data point you want to predict, in your case, $x$ is a &lt;code&gt;[roomnum, size]&lt;/code&gt; tuple.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you want to  find a $f(r, s)$, where $f(roomnum_i, size_i) = price_i, \forall i$. You have a dataset, and an algorithm to fit $f(r,s)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most interesting part of locally weighted linear regression is that, t&lt;strong&gt;he model changes when $x$ changes&lt;/strong&gt; (keep in mind $x$ is the data point you want to query).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume $x=(R,S)=(3,30)$, the algorithm becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find $\theta$ to minimize&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{i} exp(-\frac{|x^{(i)}-x|^2}{2\tau^2})(y^{(i)}-\theta^T x^{(i)})$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x^{(i)}$ and $y^{(i)}$ is your dataset, $x=(3, 30)$ is the point you want to query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://datajobs.com/data-science-repo/Generalized-Linear-Models-[Andrew-Ng].pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; shows that clearly:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The (unweighted) linear regression algorithm&#xA;  that we saw earlier is known as a parametric learning algorithm, because&#xA;  it has a &lt;strong&gt;fixed&lt;/strong&gt;, finite number of parameters (the θi&#xA;  ’s), which are fit to the&#xA;  data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;“non-parametric” (roughly) refers&#xA;  to the fact that the amount of stuff we need to keep in order to represent the&#xA;  hypothesis $h$ grows linearly with the size of the training set.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="28628" LastEditDate="2017-02-09T13:26:30.620" LastActivityDate="2017-02-09T13:26:30.620" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16854" PostTypeId="2" ParentId="16839" CreationDate="2017-02-09T13:13:17.490" Score="1" Body="&lt;p&gt;With this kind of general problem there are many possible approaches, and I can't list them all.  In general, you want to go with the simplest model that creates value over a more naive approach.  So we can look at the different approaches, ranked by complexity, and see what you need for each.  This is the order I would try for this analysis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) The simplest approach is just contact a flat 20%, drawn randomly.  This is very easy to do, but hopefully it can be improved upon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Next, you can try an approach based on the basic statistics of the problem at hand.  For example, you can calculate for those employers which previously placed ads, how many applications* did they get.  And then start with the ones that got the most, and work your way down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Now that you have baselines sets from doing a blind and then a basic descriptive analytics approach, you can try something a bit more complicated.  Assuming you have the text from the past ads, you can mine that text for features which might be descriptive of what they are looking for.  If the employers placed multiple ads, you can combine those and look for features in the aggregate.  Once you do the same with the resume data, you can create a recommendation engine to match potential employers with employees.  You can then use that data to determine which possible employer is most likely to have potential matches in your client pool.  As a side effect, this engine can also guide your clients to which jobs might be the best matches.  There are many variations which you can use here.  For example, for the recommendation engine, you may want to weight your client sample towards clients which have logged in the most recently, or most frequently, so or you can just filter out the &quot;stale&quot; clients who haven't really been looking recently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*An alarm bell went off when you said you know who applied.  Are you sure?  Or do you just have clickthrough statistics to know whether they started an application?  How do you know that they applied?  The first rule of real world data analytics is that you need to be skeptical of your data and especially of what you think your data represents.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-02-09T13:13:17.490" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16855" PostTypeId="2" ParentId="16850" CreationDate="2017-02-09T13:15:04.963" Score="11" Body="&lt;p&gt;Locally weighted linear regression is a non-parametric method for fitting data points. What does that mean?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Instead of fitting a single regression line, you fit &lt;strong&gt;many&lt;/strong&gt; linear regression models. The final resulting smooth curve is the product of all those regression models.&lt;/li&gt;&#xA;&lt;li&gt;Obviously, we can't fit the same linear model again and again. Instead, for each linear model we want to fit, we find a point x and use that for fitting a &lt;strong&gt;local&lt;/strong&gt; regression model.&lt;/li&gt;&#xA;&lt;li&gt;We find points that closest to x to fit each of our local regression model. That's why you'll see the algorithm is also known as nearest neighbours algorithm in the literature.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, if your data points have the x-values from 1 to 100: [1,2,3 ... 98, 99, 100]. The algorithm would fit a linear model for 1,2,3...,98,99,100. That means, you'll have &lt;strong&gt;100&lt;/strong&gt; regression models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, when we fit each of the model, we can't just use all the data points in the sample. For each of the model, we find the closest points and use that for fitting. For example, if the algorithm wants to fit for &lt;code&gt;x=50&lt;/code&gt;, it will put higher weight on [48,49,50,51,52] and less weight on [45,46,47,53,54,55]. When it tries to fit for &lt;code&gt;x=95&lt;/code&gt;, the points [92,93,95,96,97] will have higher weight than any other data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you see the pattern? The points closer the where you want to fit have higher weight and the points further have lower weight (zero if too far). That's what the weight function is for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question directly:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;x are the data points for each local regression model. They are usually (but not always) the data points in your sample.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-02-09T20:19:55.107" LastActivityDate="2017-02-09T20:19:55.107" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="16856" PostTypeId="2" ParentId="13185" CreationDate="2017-02-09T15:25:20.050" Score="7" Body="&lt;p&gt;Nested cross validation estimates the generalization error of a model, so it is a good way to choose the best model from a list of candidate models and their associated parameter grids. The original post is close to doing nested CV: rather than doing a single train–test split, one should instead use a second cross-validation splitter. That is, one &quot;nests&quot; an &quot;inner&quot; cross-validation splitter inside an &quot;outer&quot; cross validation splitter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The inner cross-validation splitter is used to choose hyperparameters. The outer cross-validation splitter averages the test error over multiple train–test splits. Averaging the generalization error over multiple train–test splits provides a more reliable estimate of the accuracy of the model on unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I modified the original post's code to update it to the latest version of &lt;code&gt;sklearn&lt;/code&gt; (with &lt;code&gt;sklearn.cross_validation&lt;/code&gt; superseded by &lt;code&gt;sklearn.model_selection&lt;/code&gt; and with &lt;code&gt;'mean_squared_error'&lt;/code&gt; replaced by &lt;code&gt;'neg_mean_squared_error'&lt;/code&gt;), and I used two &lt;code&gt;KFold&lt;/code&gt; cross-validation splitters to select the best model. To learn more about nested cross validation, see the &lt;code&gt;sklearn&lt;/code&gt;'s &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;example on nested cross-validation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import KFold, cross_val_score, GridSearchCV&#xA;from sklearn.datasets import make_regression&#xA;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn.svm import SVR&#xA;import numpy as np&#xA;&#xA;# `outer_cv` creates 3 folds for estimating generalization error&#xA;outer_cv = KFold(3)&#xA;&#xA;# when we train on a certain fold, we use a second cross-validation&#xA;# split in order to choose hyperparameters&#xA;inner_cv = KFold(3)&#xA;&#xA;# create some regression data&#xA;X, y = make_regression(n_samples=1000, n_features=10)&#xA;&#xA;# give shorthand names to models and use those as dictionary keys mapping&#xA;# to models and parameter grids for that model&#xA;models_and_parameters = {&#xA;    'svr': (SVR(),&#xA;            {'C': [0.01, 0.05, 0.1, 1]}),&#xA;    'rf': (RandomForestRegressor(),&#xA;           {'max_depth': [5, 10, 50, 100, 200, 500]})}&#xA;&#xA;# we will collect the average of the scores on the 3 outer folds in this dictionary&#xA;# with keys given by the names of the models in `models_and_parameters`&#xA;average_scores_across_outer_folds_for_each_model = dict()&#xA;&#xA;# find the model with the best generalization error&#xA;for name, (model, params) in models_and_parameters.items():&#xA;    # this object is a regressor that also happens to choose&#xA;    # its hyperparameters automatically using `inner_cv`&#xA;    regressor_that_optimizes_its_hyperparams = GridSearchCV(&#xA;        estimator=model, param_grid=params,&#xA;        cv=inner_cv, scoring='neg_mean_squared_error')&#xA;&#xA;    # estimate generalization error on the 3-fold splits of the data&#xA;    scores_across_outer_folds = cross_val_score(&#xA;        regressor_that_optimizes_its_hyperparams,&#xA;        X, y, cv=outer_cv, scoring='neg_mean_squared_error')&#xA;&#xA;    # get the mean MSE across each of outer_cv's 3 folds&#xA;    average_scores_across_outer_folds_for_each_model[name] = np.mean(scores_across_outer_folds)&#xA;    error_summary = 'Model: {name}\nMSE in the 3 outer folds: {scores}.\nAverage error: {avg}'&#xA;    print(error_summary.format(&#xA;        name=name, scores=scores_across_outer_folds,&#xA;        avg=np.mean(scores_across_outer_folds)))&#xA;    print()&#xA;&#xA;print('Average score across the outer folds: ',&#xA;      average_scores_across_outer_folds_for_each_model)&#xA;&#xA;many_stars = '\n' + '*' * 100 + '\n'&#xA;print(many_stars + 'Now we choose the best model and refit on the whole dataset' + many_stars)&#xA;&#xA;best_model_name, best_model_avg_score = max(&#xA;    average_scores_across_outer_folds_for_each_model.items(),&#xA;    key=(lambda name_averagescore: name_averagescore[1]))&#xA;&#xA;# get the best model and its associated parameter grid&#xA;best_model, best_model_params = models_and_parameters[best_model_name]&#xA;&#xA;# now we refit this best model on the whole dataset so that we can start&#xA;# making predictions on other data, and now we have a reliable estimate of&#xA;# this model's generalization error and we are confident this is the best model&#xA;# among the ones we have tried&#xA;final_regressor = GridSearchCV(best_model, best_model_params, cv=inner_cv)&#xA;final_regressor.fit(X, y)&#xA;&#xA;print('Best model: \n\t{}'.format(best_model), end='\n\n')&#xA;print('Estimation of its generalization error (negative mean squared error):\n\t{}'.format(&#xA;    best_model_avg_score), end='\n\n')&#xA;print('Best parameter choice for this model: \n\t{params}'&#xA;      '\n(according to cross-validation `{cv}` on the whole dataset).'.format(&#xA;      params=final_regressor.best_params_, cv=inner_cv))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28807" LastEditorUserId="28807" LastEditDate="2017-07-24T13:46:18.487" LastActivityDate="2017-07-24T13:46:18.487" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16857" PostTypeId="1" AcceptedAnswerId="16859" CreationDate="2017-02-09T15:49:58.403" Score="2" ViewCount="140" Body="&lt;p&gt;If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to generate new data from my initial set that will respect the implicit relations that could exist between the features without having to find explicitly these relations ?&lt;/p&gt;&#xA;" OwnerUserId="28812" LastActivityDate="2017-02-09T16:16:49.823" Title="Generating data that look alike my original data" Tags="|dataset|data|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16859" PostTypeId="2" ParentId="16857" CreationDate="2017-02-09T16:16:49.823" Score="4" Body="&lt;p&gt;What you want is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;generative model&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple generative model from the deep learning family are &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;autoencoders&lt;/strong&gt;&lt;/a&gt;: neural networks that receive as input your data and are trained to output that very same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different types of autoencoders. One of the most simple are &lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Rifai_455.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;contractive autoencoders&lt;/a&gt;, that have a bottle neck layer, that is, a layer with very few units. For your case, with only 4 input features, you may have a 2 unit (or even 1 unit, you may try to tune this hyperparameter) hidden layer as bottleneck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once fully trained, you only take the part of the autoencoder from the bottleneck to the output, and feed it with random numbers as input, and expect to get as output data that follows the same distribution as the original inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that the training has allowed the net to learn &lt;em&gt;representations&lt;/em&gt; of the input data distributions in the form of latent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the distribution of your input data, a simple contractive autoencoder may not be able to properly learn good representations. More advanced variants include &lt;a href=&quot;http://deeplearning.net/tutorial/dA.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;denoising autoencoders&lt;/a&gt;, &lt;a href=&quot;https://stats.stackexchange.com/questions/118199/what-are-the-differences-between-sparse-coding-and-autoencoder&quot;&gt;sparse autoencoders&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/abs/1606.05908&quot; rel=&quot;nofollow noreferrer&quot;&gt;variational autoencoders&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other generative models that are currently very trendy are &lt;a href=&quot;https://arxiv.org/abs/1406.2661&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-09T16:16:49.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16860" PostTypeId="1" AcceptedAnswerId="16867" CreationDate="2017-02-09T16:01:38.533" Score="10" ViewCount="3361" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I randomly generate millions groups of triplet $\lbrace x_0, x_1, x_2 \rbrace$ within range $(0,1)$, then calculate the corresponding coefficients of the polynomial $(x-x_0)(x-x_1)(x-x_2)$, which result in triplet groups normalized in a form of $\lbrace { {x_0+x_1+x_2 \over 3} , {\sqrt{x_0x_1+x_1x_2+x_0x_2 \over 3}} , {\sqrt[3]{x_0x_1x_2}}} \rbrace$; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;After that, I feed the coefficient triplets in to a 5-layered neural network $\lbrace 3,4,5,4,3 \rbrace$, in which all the activation function is set to sigmoid and the learning rate is set to 0.1;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;However, I only get a very poor cross validation, around 20%. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How can I fix this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BackGround&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original problem is a &lt;a href=&quot;http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.117.015501&quot; rel=&quot;noreferrer&quot;&gt;dynamic inverse problem&lt;/a&gt;. &#xA;In that problem, I have hundreds of thousands of observations $O$, from these observations, I need to recover several hundred parameters $P$.&#xA;The simulation process from $P$ to $O$ is very easy and cheap to calculate, but the inversion from $O$ to $P$ is highly nonlinear and nearly impossible. My idea is to train a neural network getting $O$ as inputs and $P$ as outputs. To check the feasibility of this idea, I employ a 3-ordered polynomial equation to do the validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;update half a year later&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With more nodes per layer, I have successfully trained a neural network. The topology is set to $\lbrace 3, 64, 64, 64 \rbrace$. And the most important trick is, sorting the generated triplet $\lbrace x_0, x_1, x_2 \rbrace$, ensuring $x_0 &amp;lt;= x_1 &amp;lt;= x_2$ always holds. &lt;/p&gt;&#xA;" OwnerUserId="28815" OwnerDisplayName="Feng Wang" LastEditorUserId="28815" LastEditDate="2017-09-16T10:12:30.780" LastActivityDate="2017-09-16T10:12:30.780" Title="Is it possible to train a neural network to solve polynomial equations?" Tags="|machine-learning|neural-network|data-mining|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16861" PostTypeId="2" ParentId="16840" CreationDate="2017-02-09T17:16:31.697" Score="5" Body="&lt;p&gt;You learn a lot by comparing to a naive model.  A naive model is one without any features.  As a default, it will always predict the most likely Target.  Note that this is exactly what your model is doing.  This indicates that the features are not helping with making a prediction.  Have you done a basic distribution analysis to see what are the features impact the distribution of the target?  This is where I'd start.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-02-09T17:16:31.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16862" PostTypeId="2" ParentId="12124" CreationDate="2017-02-09T19:39:08.863" Score="1" Body="&lt;p&gt;The R package &lt;a href=&quot;https://cran.r-project.org/web/packages/conclust/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;conclust&lt;/a&gt; seems to be what you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the package description:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are 4 main functions in this package: ckmeans(), lcvqe(), mpckm() and ccls(). They take an unlabeled dataset and two lists of must-link and cannot-link constraints as input and produce a clustering as output.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There exist also a &lt;a href=&quot;https://github.com/Behrouz-Babaki/copkmeans&quot; rel=&quot;nofollow noreferrer&quot;&gt;python&lt;/a&gt; implementations (Disclaimer: I have developed it). &lt;/p&gt;&#xA;" OwnerUserId="28820" LastActivityDate="2017-02-09T19:39:08.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16864" PostTypeId="1" AcceptedAnswerId="16865" CreationDate="2017-02-09T22:02:53.597" Score="0" ViewCount="53" Body="&lt;p&gt;Is there a way I can &quot;look inside&quot; a model once it's trained? For example, if I train a spam filter with a &lt;code&gt;multinomialNB&lt;/code&gt;, is there a way I can extract which words are most likely to make an email classify as spam? I'd like to see how the models determine the outcome once fitted.&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2017-02-09T23:16:49.563" Title="Scikit-Learn - Learned model description?" Tags="|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16865" PostTypeId="2" ParentId="16864" CreationDate="2017-02-09T23:16:49.563" Score="0" Body="&lt;p&gt;For the particular case of the MultinomialNB you can look &lt;a href=&quot;https://stackoverflow.com/questions/11116697/how-to-get-most-informative-features-for-scikit-learn-classifiers&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if want you want is to determine which features are the most important, the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SelectFromModel&lt;/a&gt; for selecting the most important features for the model.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-09T23:16:49.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16867" PostTypeId="2" ParentId="16860" CreationDate="2017-02-10T05:23:55.497" Score="6" Body="&lt;p&gt;You're trying to fit a very complicated function.  There is no reason to expect that neural networks will be very good at this.  Neural networks aren't magic pixie dust.  They can do some things well, but don't expect a silver bullet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're trying to get the neural network to learn to compute some complicated function.  We &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;do know&lt;/a&gt; that given sufficiently nodes and sufficiently many layers, you can get an &lt;a href=&quot;https://cstheory.stackexchange.com/q/17545/5038&quot;&gt;arbitrarily good&lt;/a&gt; approximation to this function, but there's no a priori reason to expect it should be possible with the particular (small) number of nodes and layers you chose.  In fact, two layers suffices, if you have sufficiently many nodes -- but we have no way to compute how many nodes are needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, the function you are trying to compute amounts to computing $x_0,x_1,x_2$ from $a,b,c,d$, where&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{align*}&#xA;x_k &amp;amp;= - {1 \over 3a} \left(b + \eta^k C + {\Delta_0 \over \eta^k C}\right)\\&#xA;\Delta_0 &amp;amp;= b^2 - 3ac\\&#xA;C &amp;amp;= \sqrt[3]{\Delta_1 \pm \sqrt{\Delta_1^2 - 4\Delta_0^3} \over 2}\\&#xA;\Delta_1 &amp;amp;= 2b^3 - 9abc + 27a^2d\\&#xA;\eta &amp;amp;= -{1 \over 2} + {1 \over 2} \sqrt{3} i&#xA;\end{align*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This comes from &lt;a href=&quot;https://en.wikipedia.org/wiki/Cubic_function#General_formula&quot; rel=&quot;nofollow noreferrer&quot;&gt;the general formula&lt;/a&gt; for solving a cubic equation.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's an extremely messy function, and thus it might be challenging to approximate using a neural network of the architecture you present.  You can always try increasing the number of nodes and/or number of layers, but there's no a priori theory to tell you what the right number is.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:32:33.517" LastActivityDate="2017-02-10T05:23:55.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16868" PostTypeId="2" ParentId="16805" CreationDate="2017-02-10T05:40:32.373" Score="1" Body="&lt;p&gt;Contrary to what you wrote in the first sentence of the question, your problem is not an instance of integer linear programming (ILP) and cannot be formulated as an ILP problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you used the $L_1$ norm (instead of the $L_2$ norm), it &lt;em&gt;could&lt;/em&gt; be formulated as an ILP problem.  You'd introduce additional variables $t_1,\dots,t_n$, add the linear inequalities&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$-t_j \le \left(v - \sum_{i=1}^K b_ix_i \right)_j \le t_j$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $(\cdots)_j$ denotes the $j$th coefficient of the vector, and then minimize the objective function $t_1+\dots+t_n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the $L_2$ norm, this is no longer an ILP problem.  It &lt;em&gt;is&lt;/em&gt; an instance of integer quadratic programming.  You could try applying an off-the-shelf solver for mixed-integer quadratic programming (MIQP).  However, MIQP is pretty tough in general, so I don't know whether this will be effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As another alternative, you could relax your MIQP instance to an instance of quadratic programming or semidefinitive programming, solve for a solution in the reals, and then round each real number to the nearest integer (or use randomized rounding), and hope that the resulting solution is &quot;pretty good&quot;.  This might be more computationally feasible (as quadratic programming / semidefinitive programming over the reals is easier than MIQP) but there are no guarantees on the quality of the resulting solution; it might be arbitrarily bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem seems related to the Closest Vector Problem (CVP) in lattices, which is &lt;a href=&quot;https://en.wikipedia.org/wiki/Lattice_problem#Closest_vector_problem_.28CVP.29&quot; rel=&quot;nofollow noreferrer&quot;&gt;believed to be hard&lt;/a&gt;. Here you have the additional constraint that the coefficients be 0 or 1 (instead of them being arbitrary integers). If $n$ is not too large, you might be able to use existing algorithms, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Lattice_reduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;LLL basis reduction&lt;/a&gt;.  I don't know whether this will work or not.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-02-10T05:40:32.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16869" PostTypeId="1" AcceptedAnswerId="17018" CreationDate="2017-02-10T06:02:39.710" Score="2" ViewCount="245" Body="&lt;p&gt;I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS. &#xA;Does the training take place on the subset of data on each node for a fixed value of K( say 3)? But then we will have 3*number of nodes centroids? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would it be different if we applied linear regression? Will it only train on data stored on each node locally?&lt;/p&gt;&#xA;" OwnerUserId="28834" LastEditorUserId="28628" LastEditDate="2017-02-10T14:51:30.330" LastActivityDate="2017-02-16T14:57:58.557" Title="K-means clustering on big data stored on multiple nodes on HDFS" Tags="|machine-learning|bigdata|k-means|apache-hadoop|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16871" PostTypeId="1" AcceptedAnswerId="16880" CreationDate="2017-02-10T09:30:43.550" Score="0" ViewCount="264" Body="&lt;p&gt;We are working on a project for creating music based on crowd sourcing. People vote for every note until the vote is closed, and then move on to the next vote until the canvas for the music is filled. A similar project is &lt;a href=&quot;https://crowdsound.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;crowdsound&lt;/a&gt;, if you want to get an idea of what it looks like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the fun part is, based on all the votes we get from various people, we would like to be able to build a Neural Network that can build an entire song on its own. The idea is for it to take in account every preceding vote and predict the one that will follow. That way, when trained, we could give it one note and let it predict the rest of the votes on its own and thus create a song on its own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I've read a few things here and there about neural networks, but there are two things I don't understand:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How to build one that takes into account a dynamic number of inputs (all preceding votes).&lt;/li&gt;&#xA;&lt;li&gt;How exactly should I decide the number of hidden layers (I still only vaguely understand what those hidden layers represent) I need for it to work well.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We are using Java for the project and we were planning on using Neuroph for the neural network.&lt;/p&gt;&#xA;" OwnerUserId="28840" LastActivityDate="2017-02-10T13:53:20.497" Title="Creating a neural network for predicting next vote in a series of votes" Tags="|neural-network|java|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16872" PostTypeId="1" CreationDate="2017-02-10T09:56:29.783" Score="4" ViewCount="94" Body="&lt;p&gt;I have users reports about an accident, i want to know how to make sure that the number of reports is enough to take that accident as a true accident not a spam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is to consider a minimum number of reports in a specific time interval, for example 4 reports in 20 minutes are good enough to believe the existence of that accident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I choose the number of minimum reports and that time interval? Is there another logic to take that decision?&#xA;I will appreciate your answers .&lt;/p&gt;&#xA;" OwnerUserId="28842" LastEditorUserId="21" LastEditDate="2017-05-19T16:09:16.393" LastActivityDate="2021-09-09T09:07:10.657" Title="how to make decision based on users reports" Tags="|optimization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16873" PostTypeId="1" CreationDate="2017-02-10T11:10:26.250" Score="0" ViewCount="93" Body="&lt;p&gt;How do I find the hyperplane (line) that touches a cloud of 2D points from one side? I used $w^Tx+b=0$ based line definition and implemented a gradient-descent based routine that would start the support line from a certain side of points, away from them. The loss function is $w^Tx_i+b$ for all points, then I take log sum. The use of hyperplanes is similar as in SVM but in this case there are no -1 or +1 labels, there are only points I am trying to approach as close as possible. I am hoping to do this from 4 sides of a point cloud to get 4 approximate / defining lines of this point cloud. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My solution was fine for some carefully selected step sizes, and iteration numbers, but the procedure can overshoot, trying to find another minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am curious if there is a better way of defining this loss function. My implementation based on automatic differentiation package autograd, seen below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;def plot_sep(w):&#xA;    Q = np.array([[0, -1],[1, 0]])&#xA;    x = np.linspace(-20,20,3000)&#xA;    w2 = np.dot(Q,w[:2])&#xA;    m = w2[1]/w2[0]&#xA;    y = m*x + (-w[2]/w[1])&#xA;    plt.plot(x,y,'.')&#xA;&#xA;df = pd.read_csv('in.csv')&#xA;df['1'] = 1.&#xA;df2 = np.array(df)&#xA;&#xA;#w1_init = np.array([0.,1.,-6.]);eta  = 1e-3;iters = 30 # good&#xA;w1_init = np.array([0.,1.,-30.]);eta  = 1e-2;iters = 40 # bad&#xA;&#xA;plt.plot(df2[:,0],df2[:,1],'.')&#xA;plot_sep(w1_init)&#xA;plt.xlim(0,15);plt.ylim(-10,40)&#xA;plt.savefig('out1.png')&#xA;&#xA;import autograd.numpy as np&#xA;from autograd import elementwise_grad&#xA;from autograd import grad&#xA;&#xA;def compute_loss(w1):&#xA;    tmp = np.dot(df2,w1)&#xA;    tmp2 = np.log(np.dot(tmp,tmp))&#xA;    return tmp2&#xA;&#xA;gradient = grad(compute_loss)&#xA;&#xA;w1 = np.copy(w1_init)&#xA;&#xA;for i in range(iters):  &#xA;    loss = compute_loss(w1)&#xA;    print &quot;iteration %d: loss %f&quot; % (i, loss) &#xA;    dw1 = gradient(w1)&#xA;    w1 += -eta*dw1&#xA;print &quot;iteration %d: loss %f&quot; % (i, loss)&#xA;print w1&#xA;&#xA;plt.figure()&#xA;plt.plot(df2[:,0],df2[:,1],'.')&#xA;plot_sep(w1)&#xA;plt.xlim(0,15);plt.ylim(-10,40)&#xA;plt.savefig('out2.png')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x,y&#xA;6.85483870968,11.875&#xA;8.06451612903,12.3958333333&#xA;7.37903225806,12.34375&#xA;8.18548387097,12.34375&#xA;8.83064516129,12.6041666667&#xA;9.43548387097,12.96875&#xA;10.0,13.0729166667&#xA;10.5241935484,13.1770833333&#xA;11.0483870968,13.2291666667&#xA;6.97580645161,10.9895833333&#xA;6.97580645161,10.4166666667&#xA;8.46774193548,10.15625&#xA;7.98387096774,10.15625&#xA;9.1935483871,10.15625&#xA;9.79838709677,10.15625&#xA;10.6048387097,10.0&#xA;11.1290322581,10.1041666667&#xA;11.1290322581,10.5208333333&#xA;10.9274193548,11.0416666667&#xA;10.9274193548,11.40625&#xA;10.9274193548,11.7708333333&#xA;10.8870967742,12.4479166667&#xA;10.0,12.7083333333&#xA;9.07258064516,11.9270833333&#xA;8.75,11.9270833333&#xA;7.86290322581,11.8229166667&#xA;7.33870967742,11.09375&#xA;7.9435483871,11.3541666667&#xA;9.15322580645,11.5104166667&#xA;9.39516129032,11.5104166667&#xA;8.50806451613,10.8854166667&#xA;9.47580645161,10.78125&#xA;9.91935483871,10.78125&#xA;10.1612903226,10.8333333333&#xA;10.1612903226,11.9270833333&#xA;9.91935483871,12.03125&#xA;9.83870967742,12.03125&#xA;9.63709677419,11.9270833333&#xA;10.564516129,11.3020833333&#xA;10.564516129,10.6770833333&#xA;9.11290322581,10.5208333333&#xA;8.02419354839,10.625&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28014" LastEditorUserId="28014" LastEditDate="2017-02-10T11:23:47.857" LastActivityDate="2019-08-24T15:02:30.400" Title="Closest Supporting Hyperplane to Points" Tags="|optimization|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16876" PostTypeId="2" ParentId="13818" CreationDate="2017-02-10T12:10:40.620" Score="3" Body="&lt;p&gt;You're close! You're just missing a &lt;code&gt;.rdd&lt;/code&gt;! Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupBy(&quot;Product_ID&quot;).agg(collect_list($&quot;Stock&quot;)).rdd.saveAsTextFile(&quot;PATH/results.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28849" LastEditorUserId="13413" LastEditDate="2017-03-03T21:08:11.417" LastActivityDate="2017-03-03T21:08:11.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16878" PostTypeId="2" ParentId="16873" CreationDate="2017-02-10T12:35:34.687" Score="1" Body="&lt;p&gt;I pondered this question a little more, there might not be an optimal loss function.. But here is an idea which could utilize SVMs hyperplane seperation ability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I take the point cloud, if I need an approximating line from the bottom, copy points below with user defined gap, and I flip the points, as in mirror image along the x-axis. I give both point groups opposing labels, then feed the whole thing to SVM. Ideally I should get something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9jeYOm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9jeYOm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The top line is the line I am looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;points1 = np.array(pd.read_csv('quadri.csv'))&#xA;plt.plot(points1[:,0], points1[:,1], 'o')&#xA;points2 = np.copy(points1)&#xA;points2[:,1] -= 7.&#xA;xmin = np.min(points2[:,0]); xmax = np.max(points2[:,0])&#xA;points2[:,0] = (xmax-points2[:,0])+xmin&#xA;points = np.vstack((points1,points2))&#xA;plt.plot(points[:,0], points[:,1], 'o')&#xA;plt.xlim(4,14); plt.ylim(0,15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28014" LastActivityDate="2017-02-10T12:35:34.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16880" PostTypeId="2" ParentId="16871" CreationDate="2017-02-10T13:53:20.497" Score="1" Body="&lt;p&gt;Recurrent neural network may fit your needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read about LSTM &amp;amp; GRU , which has been implemented into various NN libraires.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the link to the keras documentayion of RNN :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/layers/recurrent/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some interesting music project that takes advantage of RNN :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tensorflow/magenta&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/magenta&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/jisungk/deepjazz&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jisungk/deepjazz&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27107" LastActivityDate="2017-02-10T13:53:20.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16881" PostTypeId="1" AcceptedAnswerId="16883" CreationDate="2017-02-10T14:28:13.587" Score="4" ViewCount="1907" Body="&lt;p&gt;Here the code from Tensorflow tutorial:&#xA;&lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/multilayer_perceptron.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Multilayer Perceptron implementation example&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With batch size = 100 we quickly got Accuracy: 94.59%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I set the batch size to be one, the training takes ten times more time but the accuracy is only nine percent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tested different learning rates with no luck. SGD performance is terrible for small batch sizes. We can expect that SGD performance will be lower, but not ten times less!  What is the reason for this performance loss?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastEditorUserId="12384" LastEditDate="2017-02-16T07:48:51.710" LastActivityDate="2017-02-16T07:48:51.710" Title="Why is stochastic gradient descent so much worse than batch GD for MNIST task?" Tags="|neural-network|tensorflow|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16883" PostTypeId="2" ParentId="16881" CreationDate="2017-02-10T15:24:25.253" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why is stochastic gradient descent so much worse then batch GD for MNIST task?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It isn't inherently worse. Instead, by changing just one parameter on its own you have adjusted the example outside of where it has been &quot;tuned&quot; to work, because it is a simplified example for learning purposes, and it is missing some features that most users of NNs would consider standard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The batch size of 1 is performing just fine. In fact, although it takes longer to process the same number of epochs, each epoch actually has more weight updates. You get 100 times as many weight updates, although each one has far more noise in it than a batch size of 100. It is these extra weight updates, plus extra time spent running interpreted Python code for 100 times as many batches, which adds the large amount of time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with accuracy is that the example network has no protection from &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot;&gt;overfitting&lt;/a&gt;. By running so many more weight updates, the training starts to learn the &lt;em&gt;precise&lt;/em&gt; image data to match each digit, in order to get the best score. And by doing that exclusively it learns rules that work really well on the training data but generalise really badly to new data in the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try batch size of 1 and number of epochs = 3 (I tried this and got accuracy of 94.32%). Basically that is using early stopping as a form of regularisation. It's not the best form of regularisation, but it is quick to try and often effective - the problem is how to tell when to stop, so you need to measure a test set (often separate to final test set, called cross-validation set) at any potential stopping point, and save the best model so far. That will obviously involve adjusting the example code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably the 15 epochs in the original example has been chosen carefully so as to not make overfitting a problem with batch size of 100, but as soon as you change batch size, without any other form of regularisation, the network is very likely to over-fit. In general neural networks strongly tend to over-fit, and you have to spend time and effort to understand and defend against this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at &lt;a href=&quot;http://www.ritchieng.com/machine-learning/deep-learning/tensorflow/regularization/&quot;&gt;regularisation in TensorFlow&lt;/a&gt; for other options. For this kind of problem, &lt;a href=&quot;https://www.tensorflow.org/tutorials/mnist/pros/&quot;&gt;dropout&lt;/a&gt; (explained lower down page in the link) is highly recommended, which is not purely regularisation, but works to improve generalisation for many neural network problems.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-10T15:50:11.760" LastActivityDate="2017-02-10T15:50:11.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16885" PostTypeId="2" ParentId="16252" CreationDate="2017-02-10T17:51:56.670" Score="1" Body="&lt;p&gt;If you already have a &lt;code&gt;&amp;lt;HDF5 object reference&amp;gt;&lt;/code&gt;, and you want to know the real data, here is one approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import h5py&#xA;labels_file = './sv/train/digitStruct.mat'&#xA;f = h5py.File(labels_file)&#xA;ref = f.get('/digitStruct/name').value[0][0]  # &amp;lt;HDF5 object reference&amp;gt;&#xA;print(f[ref].value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my environment, &lt;code&gt;f[ref].value&lt;/code&gt; shows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [84]: f[ref].value&#xA;Out[84]: &#xA;array([[ 49],&#xA;       [ 46],&#xA;       [112],&#xA;       [110],&#xA;       [103]], dtype=uint16)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more information on object references, you can refer to &lt;a href=&quot;http://docs.h5py.org/en/latest/refs.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://docs.h5py.org/en/latest/refs.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-10T17:51:56.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16887" PostTypeId="1" CreationDate="2017-02-10T19:20:05.713" Score="1" ViewCount="70" Body="&lt;p&gt;I have time series gravity variation data with inherent uncertainty in the data itself e.g. ($x+\sigma_x$) where $\sigma_x$ represent the error in data. Now I am doing time series modeling (ARIMA) for the same data. Whatever theory I have read, they consider that original data has no error but this is not true in my data. When I do modelling, definitely the error in data itself is going to affect the model parameter. I want to know how to incorporate inherent error $\sigma_x$ in my modeling and check its impact on model.&lt;/p&gt;&#xA;" OwnerUserId="28864" LastEditorUserId="28864" LastEditDate="2017-02-15T09:00:38.830" LastActivityDate="2017-02-16T12:16:25.480" Title="Time series with erroneous data" Tags="|time-series|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16888" PostTypeId="1" CreationDate="2017-02-10T22:58:06.777" Score="3" ViewCount="2312" Body="&lt;p&gt;Is there any good libraries that allow me to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Construct a Bayesian network manually&lt;/li&gt;&#xA;&lt;li&gt;Specify the conditional probabilities with any continuous PDF, not just Guassian&lt;/li&gt;&#xA;&lt;li&gt;Perform inference, either exact or approximate&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I looked at the following libraries so far, none of them meet the 3 requirements:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;pgmpy: only work on discrete distribution or linear Guassian distribution&lt;/li&gt;&#xA;&lt;li&gt;bnlearn: same as pgmpy&lt;/li&gt;&#xA;&lt;li&gt;gRain: only discrete distribution&lt;/li&gt;&#xA;&lt;li&gt;Huggin: only discrete distribution and Guassian&lt;/li&gt;&#xA;&lt;li&gt;deal: no support for inference&lt;/li&gt;&#xA;&lt;li&gt;abn: same as deal&lt;/li&gt;&#xA;&lt;li&gt;libpgm: only discrete distribution and Guassian&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17727" LastActivityDate="2019-03-05T19:05:09.680" Title="Libraries for Bayesian network inference with continuous data" Tags="|bayesian-networks|graphical-model|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16889" PostTypeId="1" CreationDate="2017-02-10T23:17:25.817" Score="2" ViewCount="92" Body="&lt;p&gt;I have a 50 000 documents of 1000 words or more ranked between 0 and 2000. They all deal with a similar topic. I'd like to create an algorithm that can learn to score new documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What approach do you think I should take?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a newbie in the field of machine learninig so if you could point me to introduction material about the particular solution you think of. I will be glad.&lt;/p&gt;&#xA;" OwnerUserId="23614" LastEditorUserId="23614" LastEditDate="2017-02-11T00:47:33.257" LastActivityDate="2017-02-11T20:26:21.893" Title="How to learn to score new documents based on a existing set of scored documents?" Tags="|machine-learning|predictive-modeling|text-mining|scoring|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16890" PostTypeId="1" CreationDate="2017-02-10T23:17:41.920" Score="31" ViewCount="53558" Body="&lt;p&gt;I have a dataset containing 34 input columns and 8 output columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to solve the problem is to take the 34 inputs and build individual regression model for each output column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if this problem can be solved using just one model particularly using Neural Network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used Multilayer Perceptron but that needs multiple models just like linear regression. Can Sequence to Sequence be a viable option? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using TensorFlow. I have code but I think it is more important to understand what I am missing out in terms of the multilayer perceptron theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that in MLP if you have one output node it will provide one output. If you have 10 output nodes then it is a multi class problem. You pick the class with the highest probability out of the 10 outputs. But in my case it is certain there will be 8 outputs for same input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say, for a set of inputs you will get the 3D coordinate of something (X,Y,Z). Like, Inputs = {1,10,5,7} Output = {1,2,1}. So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model. So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though.&lt;/p&gt;&#xA;" OwnerUserId="28868" LastEditorUserId="836" LastEditDate="2017-02-11T20:44:38.790" LastActivityDate="2021-02-09T04:25:18.410" Title="Neural Network for Multiple Output Regression" Tags="|neural-network|regression|tensorflow|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16892" PostTypeId="2" ParentId="16889" CreationDate="2017-02-11T09:59:35.080" Score="3" Body="&lt;p&gt;First come up with some features for the document. Stuff like frequency of some popular words associated with that topic might work. In this way get the features for all the documents and then apply some algorithm. Some ways you could apply them are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) k means - cluster the documents on the basis of the features. Each cluster should be predominantly associated with a particular score value. Then see which cluster a new document will be assigned to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Supervised learning - use neural networks, multiclass SVMs etc to classify the new document to a particular class (score) using the model you would have generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these are examples of classification to a discrete score value. However, since you are dealing with a large score range (0-2000), you could also try something like regression which will give you a continuous value, but could be rounded off to the nearest discrete one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out the Coursera course on Machine Learning for a great introduction!&lt;/p&gt;&#xA;" OwnerUserId="21749" LastActivityDate="2017-02-11T09:59:35.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16893" PostTypeId="1" AcceptedAnswerId="16896" CreationDate="2017-02-11T10:05:52.190" Score="0" ViewCount="151" Body="&lt;p&gt;I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   F1   F2&#xA;0. 8673 7490&#xA;1. 5602 5602&#xA;2. 4352 2365&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When i go to process this, say MinMaxScaler 0-1, even though '1.' in both features should be the same, because both columns have different min-max values won't the normalized value be different and therefor cause problems in pattern finding? If so, what method can i use to make sure that they are of the same relative scale? Thanks for any help.&lt;/p&gt;&#xA;" OwnerUserId="27962" LastActivityDate="2017-02-11T10:45:47.880" Title="Data preprocessing, relative scale problems in features of same type" Tags="|data|preprocessing|normalization|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16895" PostTypeId="1" AcceptedAnswerId="17749" CreationDate="2017-02-11T10:40:21.707" Score="4" ViewCount="5363" Body="&lt;p&gt;My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns (say A and B). Now, in the calculation, for each row in the test dataset, I have to get the result of the following query. This time, the program takes forever. I can optimize this on my own writing some extra code (sorting will help in my case and in fact I can skip pandas altogether if I want to), but I wanted to check if there is a better built-in solution should I ever require that (also, I don't want to if I don't have to). The query is like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def f(A, B, x, y):&#xA;    data = df.loc[df[A].isin([x]) &amp;amp; df.B.isin([y])]&#xA;    return len(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, df is the DataFrame (training dataset). For each row in test dataset, data find all rows where column A has value x and B has value y. As far as I know, isin is slightly faster, so I used it. However, I think this is why it does not run within a second like it did before because each time I run this query, it needs a lot of time. What I have to do next is something like this this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(0,l):&#xA;    ret = f(A, B, test[A].values[i],test[B].values[i])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Clearly, in every loop, it will calculate the result of the query. Other than this, rest of the functions in my code are mathematical that do not need much calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if the problem is not clear since I did not use direct codes (replaced it with dummy variables and other lines do not have anything that has high complexity).&lt;/p&gt;&#xA;" OwnerUserId="28881" LastActivityDate="2017-03-21T20:29:21.047" Title="Pandas Query Optimization On Multiple Columns" Tags="|python|dataset|pandas|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16896" PostTypeId="2" ParentId="16893" CreationDate="2017-02-11T10:45:47.880" Score="1" Body="&lt;p&gt;If want you want is to scale both columns using the maximum of both and the minimum of both. You can do it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame(data=data, columns=['F1', 'F2'])&#xA;columns = ['F1', 'F2']&#xA;&#xA;ma = float(np.amax(df[columns].values))&#xA;mi = float(np.amin(df[columns].values))&#xA;&#xA;for column in columns:&#xA;    df[column] = df[column].apply(lambda e: (e - mi) / (ma - mi))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to the documentation of sklearn that is what MaxMinScaler does, if you want to use predefined values for max and min, you can just change ma and mi. If you have more columns you just have to change the list of columns. For rescaling to an arbitrary range (max, min) read the sklearn documentation on &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MaxMinScaler&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-02-11T10:45:47.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16898" PostTypeId="2" ParentId="16889" CreationDate="2017-02-11T11:39:57.840" Score="1" Body="&lt;p&gt;Basically this is a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would want to model y(rank) ~ x(Document). &#xA;And when you get a new document you need an estimated rank.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things you may want to consider.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do in need 2000 class labels, or is there any way to reduce the class labels to 3 classes etc. (class labels == score)&lt;/li&gt;&#xA;&lt;li&gt;How do I represent my documents. In short they must be numerically represented. (Approaches - One Hot Encoding, TFIDF, Embedding etc)&lt;/li&gt;&#xA;&lt;li&gt;Finally the last question which classification model should I use?.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Linear Algebra followed by Andrew Ngs ML course is a good starting point. &lt;/p&gt;&#xA;" OwnerUserId="16971" LastEditorUserId="16971" LastEditDate="2017-02-11T11:48:58.903" LastActivityDate="2017-02-11T11:48:58.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16900" PostTypeId="1" AcceptedAnswerId="16913" CreationDate="2017-02-11T12:37:50.200" Score="4" ViewCount="435" Body="&lt;p&gt;We've got a list of approximately 18,000 product names (they're from 80-90 sources, so quite a few that are similar but not duplicates - these were picked as DISTINCT from a table) unfortunately there are different ways of expressing these names. We have to try and normalize the dataset so we present our users with more meaningful names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a list like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Canon EOS 5D Mark III&#xA;Canon EOS 5D mk III&#xA;Canon EOS 5DMK3&#xA;Canon EF 70-200mm f/2.8L IS II USM Lens&#xA;Canon EF 70-200mm f/2.8L IS II USM Telephoto Zoom Lens&#xA;Canon EF 70-200mm f/2.8L IS USM Lens&#xA;Canon EF 70-200mm f/4L USM Lens&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to be able to assess those strings and collapse them into something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Canon EOS 5D Mark III&#xA;Canon EF 70-200mm f/2.8L IS II USM &#xA;Canon EF 70-200mm f/2.8L IS USM Lens&#xA;Canon EF 70-200mm f/4L USM Lens&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I'd like to know how similar two strings are to be able to determine this. I do realise that the F2.8 IS II and IS USM may be a bit hard, but thought I'd throw it in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real product names are far less exciting (they're parts for the farm equipment we stock).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also store these names in a Postgres (9.5) database table. Examples i've seen compare two lists, but we don't have a master product list to do that unfortunately.&lt;/p&gt;&#xA;" OwnerUserId="28883" LastEditorUserId="28883" LastEditDate="2017-02-11T21:41:16.393" LastActivityDate="2017-02-12T11:27:42.730" Title="Check similarity of table/csv of Product Names" Tags="|similarity|fuzzy-logic|text-filter|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="16902" PostTypeId="1" CreationDate="2017-02-11T15:22:27.670" Score="0" ViewCount="1868" Body="&lt;p&gt;I work for a business-to-business company that has a large database of existing clients (small businesses) with various columns of data (e.g., industry, credit-worthiness, financial data, etc) describing each client. There are several marketing data vendors I know of that maintain small-business databases with columns of data similar to what my company has. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; Using the data my company already has, how I can I identify which customers in the vendors' databases have the highest potential of becoming a customer if marketed to? Should I build a predictive model? Should I do some sort of clustering?    &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-07-21T21:11:57.633" Title="Develop a marketing look-alike profile for prospective customers" Tags="|machine-learning|predictive-modeling|marketing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16903" PostTypeId="2" ParentId="16159" CreationDate="2017-02-11T19:44:09.903" Score="0" Body="&lt;p&gt;Yes.  There has been work on using computer vision techniques to detect screen elements, to aid testing of GUIs and for automation.  See, e.g., the following papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Tsung-Hsiang Chang, Tom Yeh, Robert C. Miller.  &lt;a href=&quot;http://up.csail.mit.edu/projects/sikuli/sikuli-chi2010.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;GUI Testing Using Computer Vision&lt;/a&gt;.  CHI 2010.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tom Yeh, Tsung-Hsiang Chang, Robert C. Miller.  &lt;a href=&quot;http://up.csail.mit.edu/projects/sikuli/sikuli-uist2009.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sikuli: Using GUI Screenshots for Search and Automation&lt;/a&gt;.  UIST 2009.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This led to &lt;a href=&quot;http://www.sikuli.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Siku software package&lt;/a&gt;.  There's probably lots of other work out there; those are just two papers I happen to be familiar with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual machine learning needed is not too sophisticated.  This is a basic object detection task.  If you put together a good training set and train a convolutional neural network, I would imagine that the result should be pretty effective.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-02-11T19:44:09.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16904" PostTypeId="1" AcceptedAnswerId="16908" CreationDate="2017-02-11T20:03:23.843" Score="83" ViewCount="124852" Body="&lt;p&gt;I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost almost always performs better than GBM. What makes XGBOOST so fast?&lt;/p&gt;&#xA;" OwnerUserId="28834" LastEditorUserId="29169" LastEditDate="2019-06-05T06:53:31.967" LastActivityDate="2020-04-07T13:52:49.293" Title="GBM vs XGBOOST? Key differences?" Tags="|machine-learning|algorithms|xgboost|ensemble-modeling|gbm|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16905" PostTypeId="2" ParentId="16889" CreationDate="2017-02-11T20:26:21.893" Score="1" Body="&lt;p&gt;This requires that the dataset can stay in RAM but it does what I want using sklearn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.feature_extraction.text import CountVectorizer&#xA;from sklearn.linear_model import Ridge&#xA;&#xA;&#xA;corpus = [&#xA;    'postgresql is great software for database engineering',&#xA;    'postgresql is spam',&#xA;    'spam &amp;amp; egg',&#xA;    'postgresql is the same as pg',&#xA;    'more database is great stuff',&#xA;]&#xA;&#xA;test = [&#xA;    'postgresql is spam &amp;amp; egg',&#xA;    'pg is a great database software',&#xA;]&#xA;&#xA;vectorizer = CountVectorizer(min_df=1)&#xA;X = vectorizer.fit_transform(corpus)&#xA;y = np.array([[10, 1, 0, 7, 6]]).T  # documents score&#xA;clf = Ridge(alpha=1.0)&#xA;clf.fit(X, y)&#xA;&#xA;Z = vectorizer.transform(test)&#xA;&#xA;print(clf.predict(Z))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23614" LastActivityDate="2017-02-11T20:26:21.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16906" PostTypeId="2" ParentId="16890" CreationDate="2017-02-11T21:04:34.923" Score="17" Body="&lt;p&gt;What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the nature of the problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any neural network framework is able to do something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to do that is to remember that the last layer should have linear activations (i.e. no activation at all).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per your requirements, the shape of the input layer would be a vector (34,) and the output (8,).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: the usual loss function used for regression problems is mean squared error (MSE). &lt;a href=&quot;https://github.com/fchollet/keras/issues/1874#issuecomment-193083902&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s an example of multidimensional regression using &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt;; the network is not an MLP but it should be Ok to illustrate the idea.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="78365" LastEditDate="2019-12-18T07:15:04.250" LastActivityDate="2019-12-18T07:15:04.250" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16907" PostTypeId="2" ParentId="16904" CreationDate="2017-02-11T23:52:07.320" Score="15" Body="&lt;p&gt;One very important difference is &lt;code&gt;xgboost&lt;/code&gt; has implemented &lt;a href=&quot;https://arxiv.org/abs/1505.01866&quot; rel=&quot;noreferrer&quot;&gt;DART, the dropout regularization for regression trees&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Rashmi, K. V., &amp;amp; Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="28628" LastEditDate="2017-02-13T11:01:00.263" LastActivityDate="2017-02-13T11:01:00.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16908" PostTypeId="2" ParentId="16904" CreationDate="2017-02-12T02:22:53.637" Score="67" Body="&lt;p&gt;Quote from the author of &lt;code&gt;xgboost&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Both xgboost and gbm follows the principle of gradient boosting.  There are however, the difference in modeling details. Specifically,  xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.&lt;/p&gt;&#xA;&lt;p&gt;We have updated a comprehensive tutorial on introduction to the model, which you might want to take a look at. &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/tutorials/model.html&quot; rel=&quot;noreferrer&quot;&gt;Introduction to Boosted Trees&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost. For model, it might be more suitable to be called as regularized gradient boosting.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Edit: There's a detailed &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/&quot; rel=&quot;noreferrer&quot;&gt;guide&lt;/a&gt; of xgboost which shows more differences.&lt;/p&gt;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting&quot; rel=&quot;noreferrer&quot;&gt;https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/tutorials/model.html&quot; rel=&quot;noreferrer&quot;&gt;https://xgboost.readthedocs.io/en/latest/tutorials/model.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-10-17T17:43:00.413" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16909" PostTypeId="2" ParentId="16902" CreationDate="2017-02-12T03:47:40.393" Score="-1" Body="&lt;p&gt;I think that your problem would be well suited for &lt;a href=&quot;http://www.cs.cmu.edu/afs/cs/Web/People/awm/tutorials/dtree18.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision Tree Classification&lt;/a&gt;.  The presentation that I linked to should give you a good overview to learn about decision tree models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://www.patricklamle.com/Tutorials/Decision%20tree%20python/tuto_decision%20tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python tutorial&lt;/a&gt; will teach you how to build your own decision tree model from scratch.  With this knowledge, and through searching many more decision tree examples online, I'm sure you'll find yourself in a good spot!&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2017-02-12T03:47:40.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16910" PostTypeId="1" CreationDate="2017-02-12T04:03:56.963" Score="1" ViewCount="402" Body="&lt;p&gt;I have referred and used materials from these - &lt;a href=&quot;http://web.stanford.edu/class/cs224n/lecture_notes/cs224n-2017-notes1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;cs224n stanford&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is the algorithm provided in the stanford course notes for training a skip gram model. Here |V| is the vocab size, matrix V is hidden layer matrix that would select a particular input word embedding for center word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that hidden layer matrix V is of dimensions - |V| x d. (d - feature lengths), and input one hot vector of length |V| x 1. The result of hidden layer then would be of (1 x d) dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt; - In step 4, it is said to generate 2m score vectors. To me it seems the dimension of the score vectors would be &lt;strong&gt;1 x 1&lt;/strong&gt; each, as they would be a product of result from the hidden layer of dimension (1 x d), and output word vector matrix U, of dimension |V| x d. The overall result should be of |V| x 1, And each word score will be of 1 x 1 dimension (column vector).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then how can we compare each of them to one hot vectors as asked in step 6 (in doing so, won't it make probability of a particular word ideally 1, and hence all other word probabilities should be zero)? Also, what are the actual output one hot vectors? And is the probability of each word in the 2m window equal to all other words there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3YZKK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3YZKK.png&quot; alt=&quot;Skip gram algorithm in cs224n lecture notes 1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28904" LastActivityDate="2017-02-12T04:03:56.963" Title="Skip gram Word2Vec model, neural network implementation" Tags="|neural-network|deep-learning|nlp|word2vec|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16911" PostTypeId="1" AcceptedAnswerId="16912" CreationDate="2017-02-12T10:37:25.913" Score="11" ViewCount="12070" Body="&lt;p&gt;In cs231n course , it is mentioned that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If the initial weights are too large then most neurons would become&#xA;  saturated and the network will barely learn.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How do the neurons get saturated? Large weights may lead to a z (output of saturation) which is not cery close 0 or 1 and so doesn't let &lt;code&gt;z*(1-z)&lt;/code&gt; to saturate&lt;/p&gt;&#xA;" OwnerUserId="27854" LastActivityDate="2017-02-12T10:53:29.873" Title="How does sigmoid saturate with large weights?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16912" PostTypeId="2" ParentId="16911" CreationDate="2017-02-12T10:53:29.873" Score="12" Body="&lt;p&gt;The sigmoid function $$ \theta(z) = \frac{1}{1+e^{-z}}$$ looks like this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/voC4J.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/voC4J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $$z=w_i a_i + bias$$ &#xA;for activations $a_i$ from the previous layer, and weights $w_i$ of the current neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the weights $w_i$ are too large (positive or negative), $z$ tends to be large as well, driving the output of the sigmoid to the far left (value 0) or far right (value 1). These are saturation regions where the gradient/derivative is too small, slowing down learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning slows down when the gradient is small, because the weight upgrade of the network at each iteration is directly proportional to the gradient magnitude.&lt;/p&gt;&#xA;" OwnerUserId="860" LastActivityDate="2017-02-12T10:53:29.873" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16913" PostTypeId="2" ParentId="16900" CreationDate="2017-02-12T11:27:42.730" Score="2" Body="&lt;p&gt;Your problem is known as detection of near-duplicate documents, i.e. you have strings that are similar but not exact duplicate. The most common approaches are using &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine similarity&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jaccard similarity&lt;/a&gt;. You could check this &lt;a href=&quot;http://www.cs.utexas.edu/~ml/publications/area/81/record_linkage_and_duplicate_detection/abstracts/&quot; rel=&quot;nofollow noreferrer&quot;&gt;page&lt;/a&gt; for more information &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First you have to convert your strings to a vector of features, this features could be &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf vectors&lt;/a&gt; of all the tokens (words) that appear in the database, could also be a vector of &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow noreferrer&quot;&gt;n-grams&lt;/a&gt;. For a discussion of semantic and syntactic features you can look &lt;a href=&quot;http://www2013.org/proceedings/p1273.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally you would have to check the similarity of every pair of documents, in your case 18000. The brute force approach is O(n^2), so this could be unfeasible. A common technique to deal with this issue is to use fingerprinting (hashing) together with Locality Sensitive Hashing (LSH). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find an introduction and a general discussion of the whole topic in Chapter 3 of &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mining Massive Datasets&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-02-12T11:27:42.730" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16916" PostTypeId="1" CreationDate="2017-02-12T21:30:42.330" Score="0" ViewCount="59" Body="&lt;p&gt;I am trying to clean a set of e-commerce data.&#xA;Data consists of products which has multiple images, a title and a description.&#xA;Some of these products has duplicates and I need to de-duplicate the set.&#xA;I have worked on NN's and Genetic Algorithms back in college but that's it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need is a starting point to dive into. What methods/techniques should I use, what should I learn first?&lt;/p&gt;&#xA;" OwnerUserId="28917" LastActivityDate="2017-03-14T22:34:09.733" Title="Deduplicating e-commerce products based on title/description/images" Tags="|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16917" PostTypeId="2" ParentId="16916" CreationDate="2017-02-12T21:46:55.983" Score="1" Body="&lt;p&gt;As a starting point &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;chapter 3&lt;/a&gt; of Mining Massive Datasets (MMDS) offers a very good introduction on document similarity. You can also look &lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. For near-duplicate image detection look at this &lt;a href=&quot;https://stackoverflow.com/questions/1034900/near-duplicate-image-detection&quot;&gt;question&lt;/a&gt; on StackOverflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the text data, first you have to convert your strings to a vector of features, this features could be &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf vectors&lt;/a&gt; of all the tokens (words) that appear in the database, could also be a vector of &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram&quot; rel=&quot;nofollow noreferrer&quot;&gt;n-grams&lt;/a&gt;. For a discussion of semantic and syntactic features you can look &lt;a href=&quot;http://www2013.org/proceedings/p1273.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the vectors for each product on your database you would have to compute the similarity among all pair of products. Then the most common similarities measures are &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine similarity&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jaccard similarity&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The brute force approach is O(n^2), where is the number of products in the database so this could be unfeasible. A common technique to deal with this issue is to use fingerprinting (hashing) together with Locality Sensitive Hashing (LSH), both techniques are discussed in chapter 3 of MMDS.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-12T21:46:55.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16918" PostTypeId="1" CreationDate="2017-02-13T01:57:18.537" Score="1" ViewCount="496" Body="&lt;p&gt;I have a Convolutional Neural Network, that is structured as binary classifier. I have two relatively standard convolutional/relu/pooling layers followed by 2 layer fully connected network outputting to a softmax with loss layer for the binary classification. However I observed something unusual:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Version 1 of the network I had both convolutional layers derive 10 features, upon initialisation I had a Cross Entropy Error of around 28. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Version 2 I upped the number of features of the convolutional layers to 64 features. Despite still having the same fully connected layers and the same softmax, my Cross Entropy Error jumped to 340.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, why would this happen. Surely the randomness is the same and the softmax with loss function should normalise so that both outputs add up to 1. So why would Cross Entropy suddenly jump so high&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of the cross entropy effects for outputting large numbers was helped by &lt;a href=&quot;https://datascience.stackexchange.com/questions/9302/the-cross-entropy-error-function-in-neural-networks&quot;&gt;this answer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28922" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-09-14T20:29:10.200" Title="Improving initial Cross Entropy in CNN" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16919" PostTypeId="2" ParentId="15699" CreationDate="2017-02-13T02:16:55.493" Score="1" Body="&lt;p&gt;So neural nets can be quite powerful for end-to-end training. So if you have a dataset that contains both clean and dirty images, and also the classes you are trying to detect in the first place, it should be fairly straightforward to try to train you net. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd proceed as follows. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;adjust the network to produce a logits vector that outputs not only the object classification but also a binary classifier for whether your image is clean or dirty. &lt;/li&gt;&#xA;&lt;li&gt;then adjust the labels for each image to indicate the classification label and the cleanliness of your image.&lt;/li&gt;&#xA;&lt;li&gt;I'd then train and test on the dataset to see if a model emerges. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The convolutional parts of your network should converge to features that are good at&lt;br&gt;&#xA;A. classifying the subject of the image, and &#xA;B. detecting whether the camera is dirty or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its a good idea and I might use it myself!&lt;/p&gt;&#xA;" OwnerUserId="28922" LastActivityDate="2017-02-13T02:16:55.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16920" PostTypeId="2" ParentId="13188" CreationDate="2017-02-13T04:57:51.063" Score="1" Body="&lt;p&gt;In this context I think it means you take the output representation and learn an additional softmax layer that corresponds to the tokens in your language model (in this case letters).&lt;/p&gt;&#xA;" OwnerUserId="28924" LastActivityDate="2017-02-13T04:57:51.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16921" PostTypeId="1" CreationDate="2017-02-13T05:12:42.300" Score="4" ViewCount="9556" Body="&lt;p&gt;I would like to ask if Savitzky-Golay can be implemented on real-time data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone refer me to appropriate implementation or hint online implementation. Thanks. &lt;/p&gt;&#xA;" OwnerUserId="26709" LastEditorUserId="26709" LastEditDate="2017-02-15T15:47:33.240" LastActivityDate="2017-10-14T03:38:48.650" Title="Real time noise removal using Savitzky-Golay Method" Tags="|time-series|preprocessing|online-learning|data-stream-mining|noisification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16922" PostTypeId="1" CreationDate="2017-02-13T06:13:10.090" Score="5" ViewCount="30162" Body="&lt;p&gt;I am new to tensorflow. I have manged to train and validate a CNN, saved the session through the Saver object into a CPKT file and loaded it back. Now I'd like to use the trained model in order to check how it performs against a photo I took myself. I can't figure out how to do it.&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2018-03-06T10:41:18.257" Title="Using Tensorflow model for prediction" Tags="|python|deep-learning|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16923" PostTypeId="2" ParentId="16904" CreationDate="2017-02-13T06:48:52.443" Score="20" Body="&lt;p&gt;In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use of sparse matrices with sparsity aware algorithms&lt;/li&gt;&#xA;&lt;li&gt;Improved data structures for better processor cache utilization which makes it faster.&lt;/li&gt;&#xA;&lt;li&gt;Better support for multicore processing which reduces overall training time.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my experience when using GBM and xgboost while training large datasets (5 million+ records), I've experienced significantly reduced memory utilization (in R) for the same dataset and found it easier to use multiple cores to reduce training time.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastActivityDate="2017-02-13T06:48:52.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16924" PostTypeId="2" ParentId="16922" CreationDate="2017-02-13T08:04:08.753" Score="7" Body="&lt;p&gt;Look at &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=/amp/s/nathanbrixius.wordpress.com/2016/05/24/checkpointing-and-reusing-tensorflow-models/amp/&amp;amp;ved=0ahUKEwi20sqJyIzSAhVIPo8KHTK5ARgQFgg3MAI&amp;amp;usg=AFQjCNEhPsE6cCxQqVAK9dhod_kykgwWwg&amp;amp;sig2=DTb8Bs_jGP47bH1c7T1Rsw&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; blog. Mainly you have saved operations as a part of your computational graph. So after you load your model, you can restore the session and call the predict operation that you created for training and validating your data, and run it on the new data hy feeding into the feed_dict. Make sure it is in the same format and same shape as your training data. I am adding a small snippet of code, maybe that will help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;checkpoint_file=tf.train.latest_checkpoint(checkpoint_directory)&#xA;graph=tf.Graph()&#xA;&#xA; with graph.as_default():&#xA;     session_conf = tf.ConfigProto(allow_safe_placement=True, log_device_placement =False)&#xA;     sess = tf.Session(config = session_conf)&#xA;     with sess.as_default():&#xA;          saver =     tf.train.import_meta_graph(&quot;{}.meta&quot;.format(checkpoint_file))&#xA;          saver.restore(sess,checkpoint_file)&#xA;          input = graph.get_operation_by_name(&quot;input&quot;).outputs[0]&#xA;          prediction=graph.get_operation_by_name(&quot;prediction&quot;).outputs[0]&#xA;          #newdata=put your data here&#xA;          print sess.run(prediction,feed_dict={input:newdata})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To use operation_by_name, you will have to name the operations and variables in the original model. Secondly, in the feed dict you will have to give all the parameters needed to perform the operation(prediction here), so if you are using dropout you will have to give that too. There might be more simpler ways of doing this. This is the way I did it. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-02-13T08:04:08.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16928" PostTypeId="1" AcceptedAnswerId="17007" CreationDate="2017-02-13T11:33:35.880" Score="0" ViewCount="84" Body="&lt;p&gt;I'm looking at datasets where the the attributes and the target class have a logical relationsships. All attributes and the target class are binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example: &lt;em&gt;Neither &lt;code&gt;feature#1&lt;/code&gt; nor &lt;code&gt;feature#2&lt;/code&gt; have a significant correlation with the target class. But the conjunction &lt;code&gt;feature#1 AND feature#2&lt;/code&gt; is correlated with the target class.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any Feature Selection Algorithms able to cope with situations like that? I'm thinking that frequent itemsets could be useful. &#xA;It would be tremendously helpful, if anyone could point me to a related paper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="28937" LastEditorUserId="28937" LastEditDate="2017-02-13T16:05:39.133" LastActivityDate="2017-02-15T19:50:00.540" Title="Feature Selection Algorithm for Attributes with Logical Relationships (like &quot;AND&quot;)" Tags="|feature-selection|feature-extraction|correlation|association-rules|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="16929" PostTypeId="1" CreationDate="2017-02-13T11:42:02.953" Score="1" ViewCount="203" Body="&lt;p&gt;I have physical measurements with length 2*n, where the first vector represents a charge or a capacity (in Coulomb) $C$ and the second one is a voltage $V$. Let's call this measurement &quot;forming&quot;.&#xA;A measurement consist in increasing the charge until a voltage threshold is reached. I want to predict the rank of a sample depending on the these physical measurements. A rank is a letter: A, B, C...&#xA;However the physical measurements have different lengths ($n\in [1000,3000]$). Besides this, the voltage thresholds are the same for all samples: $V_a[i] = V_b[i]$ with $a,b$ being any sample number and $i\in {0,-1}$ (first and last value in python notation). So we can say that the $V$ vectors are already normalized (same mean and std) and can be used as indexes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Determining the rank can be done with a similar measurement but is too long, that is why I want to establish a predictive model. Let's call this other long measurement &quot;determining&quot;.&#xA;I have used &quot;PCA&quot; so far, which requires same-length vectors to make a matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my plan:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;make the &quot;forming&quot; measurement for 100 samples (done)&lt;/li&gt;&#xA;&lt;li&gt;establish the rank for 100 samples with the &quot;determining&quot; measurement (done)&lt;/li&gt;&#xA;&lt;li&gt;develop a predictive model&lt;/li&gt;&#xA;&lt;li&gt;use the model on the next 100 000 samples&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I would like to know:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;your opinion on whether my handling of heterogeneous length is correct&lt;/li&gt;&#xA;&lt;li&gt;Is PCA the right tool for the model?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I have done so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;imports&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import pyplot as plt&#xA;import numpy as np&#xA;from numba import jit&#xA;import pandas as pd&#xA;from sklearn.decomposition import PCA as sklearnPCA&#xA;from scipy.signal import savgol_filter&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;get data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;logs = ... #list of 100 pandas.DataFrame&#xA;Ranks = ... #list of corresponding 100 ranks (e.g. characters)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the $C$ vs $V$ function is not linear but has plateaus and walls, I cannot simply use a fixed sampling rate for each sample to resize to a fixed length. Otherwise, I will have missing data on the &quot;walls&quot;. Hence I'm using on the first sample the &lt;a href=&quot;https://stackoverflow.com/questions/36284100/how-we-can-down-sample-a-1d-array-values-by-averaging-method-using-float-and-int/36295839#36295839&quot;&gt;'resized' function of B.M.&lt;/a&gt; which both smooths and resizes by averaging neighboring data points, then I use a Savitzky-Golay filter for smoothing and a linear interpolation for resizing, using as target for the interpolation the $V$ points found by the &lt;code&gt;resized&lt;/code&gt; function on the first sample:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#%% resizing functions&#xA;def resized(data,N):&#xA;    M=data.size&#xA;    res=np.empty(N,data.dtype)&#xA;    carry=0&#xA;    m=0&#xA;    for n in range(N):&#xA;        summ = carry&#xA;        while m*N - n*M &amp;lt; M :&#xA;            summ += data[m]&#xA;            m += 1&#xA;        carry = (m-(n+1)*M/N)*data[m-1]&#xA;        summ -= carry&#xA;        res[n] = summ*N/M&#xA;    return res&#xA;resized = jit(resized)&#xA;&#xA;def res_interp(xnew,xp,yp):&#xA;    yp = savgol_filter(yp,7,2)#smooth with a 7-point window&#xA;    return np.interp(xnew,xp,yp)&#xA;&#xA;#%% make the same length&#xA;charge1=list()&#xA;for i,l in enumerate(logs):&#xA;    ch1=l[l.Step==2]#get only correct measurements&#xA;&#xA;    if i==0:#base sampling on first data&#xA;        y_ch1=resized(ch1.Capacity.values,1000)&#xA;        x_ch1=resized(ch1.Volt.values,1000)&#xA;    else:&#xA;        y_ch1=res_interp(x_ch1,ch1.Volt.values,ch1.Capacity.values)&#xA;&#xA;&#xA;    charge1.append(pd.Series(data=y_ch1,&#xA;                 index=x_ch1,&#xA;                 name='Capacity'))&#xA;&#xA;charge1=pd.concat(charge1,axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;charge1 is a $1000\times100$ matrix where rows correspond to 1000 voltages between $V[0]$ and $V[-1]$ (in charge1.index) and columns represent the charges of the 100 samples.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sklearn_pca = sklearnPCA(n_components=4)#I picked 4 for readability&#xA;sklearn_transf = sklearn_pca.fit_transform(charge1.Capacity.values)&#xA;ranks='ABCD'&#xA;Rank_int = [ranks.find(x) for x in Rank]#string to int&#xA;compo = sklearn_pca.components_&#xA;# plots&#xA;fig, ax = plt.subplots(nrows=1)&#xA;ax.plot(charge1.index,(sklearn_transf[:,0]-sklearn_transf[:,0].mean())/sklearn_transf[:,0].std())&#xA;ax.plot(charge1.index,(sklearn_transf[:,1]-sklearn_transf[:,1].mean())/sklearn_transf[:,1].std())&#xA;ax.plot(charge1.index,(sklearn_transf[:,2]-sklearn_transf[:,2].mean())/sklearn_transf[:,2].std())&#xA;ax.plot(charge1.index,(sklearn_transf[:,3]-sklearn_transf[:,3].mean())/sklearn_transf[:,3].std())&#xA;fig, ax = plt.subplots(nrows=2,ncols=2)&#xA;ax = ax.flatten()&#xA;for i in range(compo.shape[0]):&#xA;    ax[i].plot(Rank_int,compo[i,:],'o',label=str(i))&#xA;    ax[i].set_xticks([1,2])&#xA;    ax[i].set_xticklabels(['B','C'])&#xA;    ax[i].set_title('PC'+str(i+1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The plots of charge1.index versus PC1 is not so interesting: all the curves $C$C versus $V$ have a similar shape, so PC1 is simply an average of all of them. I believe more interesting are the &quot;scores&quot; given to the following PCs which are uncorrelated to PC1.  I see still no correlation which means I might not be using the right tool or my resizing made my data wrong:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Lj8bq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Lj8bq.png&quot; alt=&quot;Eigenvalues&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fell on a sample of samples with no Rank A so the experiment was certainly not optimal, but I think my method is more problematic than my samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;side note: I have seen so far little examples of &quot;real life&quot; measurements with noise and heterogeneous lengths. Most of them are either number recognition or face recognition of fixed-length images, without counting stock market analysis which is a very different experimental set-up (there are no &quot;sample&quot;, only continuous measurements).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt;&#xA;I believe I need to use from sklearn.ensemble.GradientBoostingClassifier and not PCA. I saw using PCA on physical measurements in a lecture given by ORNL (Manhattan project place) in order to extract &quot;hidden&quot; physical behaviors (PCi, i&gt;2) behind the main physical feature (PC1) that could correspond to different materials properties. But it seems it's not the right tool as it's used by real data scientists to simplify a multi dimensional problem.&#xA;Any input would still be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="28887" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-13T15:55:18.583" Title="predict rank from physical measurements with various lengths" Tags="|python|scikit-learn|pandas|pca|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16930" PostTypeId="1" CreationDate="2017-02-13T12:17:42.503" Score="5" ViewCount="3647" Body="&lt;p&gt;I have transaction details for credit data (bank transfers, peer to peer transfers, etc). Currently, I have one year worth of data which I cannot properly classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for input and suggestions about these two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How do I detect anomalies in the last hour with previous credit transaction data that I have?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I detect anomalies in real-time for the current transaction?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28941" LastEditorUserId="29169" LastEditDate="2019-06-09T15:48:17.107" LastActivityDate="2019-06-09T15:48:17.107" Title="Anomaly detection for transaction data" Tags="|machine-learning|time-series|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16935" PostTypeId="1" AcceptedAnswerId="17438" CreationDate="2017-02-13T14:36:51.723" Score="2" ViewCount="279" Body="&lt;p&gt;I have the following problem:&#xA;Suppose we are interested in estimating the distribution over the English letters. We assume an alphabet that consists of 26 letters and the space symbol, and we ignore all other punctuation and the upper/lower case disctinction. We model the distribution over the 27 symbols as a multinomial parametrized by $\theta =(\theta_1,...,\theta_{27})$ where $\sum_i \theta_i = 1$ and all $\theta_i \geq 0$/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we go to Stanford's Green Library and repeat the following experiment: randomly pick up a book, open a page, pick a spot on the page, and write down the nearest symbol that is in our alphabet. We use $X[m]$ to denote the letter we obtain in the $mth$ experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end we have collected a databse $D=\{x[1],...,x[2000] \}$ consisting of 2000 symbols, among which &quot;a&quot; appears 100times and &quot;p&quot; 87 times. We use a Dirichlet prior over $\theta$, i.e. $P(\theta)=Dirichlet(\alpha_1,...,\alpha_{27})$, where each $\alpha_i=10$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we draw two more samples, X[2001] and X[2002]. If we use $\alpha_i=10$ for all $i$, what is the probability of $P(X[2001]=&quot;p&quot;,X[2002]=&quot;a&quot;|D)$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought we could compute: $P(x[2001]=p|D) \times P(x[2002]=a|D)=\frac{10+87}{270+2001} \times \frac{10+100}{270+2002}$ but it's wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall formula : $P(x|u,D)=\frac{\alpha_{x,u}+M[x,u]}{\alpha_u+M[u]}$&lt;/p&gt;&#xA;" OwnerUserId="19065" LastActivityDate="2017-03-08T09:16:15.207" Title="Learning with dirichlet prior - probabilistic graphical models exercise" Tags="|machine-learning|graphical-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16936" PostTypeId="2" ParentId="16930" CreationDate="2017-02-13T15:32:35.137" Score="3" Body="&lt;p&gt;There are a few factors to consider in anomaly detection.  A simple method would be to plot a boxplot of the data and calculate outliers this way (&lt;a href=&quot;http://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;boxplot description&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the time series is seasonal, you could take an approach similar to Twitter's anomaly detection algorithm (&lt;a href=&quot;https://github.com/twitter/AnomalyDetection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Twitter Anomaly Detection&lt;/a&gt;).  This uses an outlier detection method (ESD) applied to the seasonal decomposition of the time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use clustering and SVMs as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also change-point detection which I understand less but can be very effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really, your answer depends on how complex your data is.  Sometimes, a simple heuristic algorithm works the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some similar questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/11446/anomaly-detection-in-time-series-data-help-required?rq=1&quot;&gt;Anomaly detection in Time Series Data - Help Required&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/1174/how-to-classify-and-cluster-this-time-series-data?rq=1&quot;&gt;How to classify and cluster this time series data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-02-13T15:32:35.137" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16938" PostTypeId="1" CreationDate="2017-02-13T21:22:20.730" Score="1" ViewCount="91" Body="&lt;p&gt;I am looking for a machine-learning (classification) model that can adapt to new data. By that, I don’t mean completely new samples in the online-learning sense, but rather additional details about an existing sample. Over time, for an existing sample, I learn an additional feature value that was missing before. (Once it is learned it doesn't change.) When I run an updated sample through the model, I would like to get a &quot;more accurate&quot; prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The time and order at which new data becomes available are not consistent and features can still be missing at the end. In my training data, I only know the data that is available at the end and not when which feature value became available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the features are categorical and I am wondering if it would be enough to train a model by creating many synthetic samples where I randomly replace some of the categories with a “missing category” value. It seems to me though that this approach would not make sure that additional data would lead to a more confident prediction. The model would just see it as a new and different sample. How can I make sure that an update leads to a better prediction? Any hints are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-02-16T22:18:04.253" LastActivityDate="2017-02-16T22:18:04.253" Title="Model that adapts to sample updates" Tags="|machine-learning|predictive-modeling|supervised-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16939" PostTypeId="1" CreationDate="2017-02-13T22:08:33.167" Score="0" ViewCount="427" Body="&lt;p&gt;I was wondering, with the advent of deep learning, many tasks related to images have been solved to near human accuracy such as classification, object detection etc., however in videos, traditional methods still compare to deep learning methods and both are not as good. I wanted to understand what and how is it that with videos that understanding them makes it much harder.&lt;/p&gt;&#xA;" OwnerUserId="26050" LastEditorUserId="85045" LastEditDate="2019-12-12T16:02:28.267" LastActivityDate="2019-12-12T16:02:28.267" Title="Why is video classification still not that accurate?" Tags="|machine-learning|classification|deep-learning|computer-vision|object-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16940" PostTypeId="1" AcceptedAnswerId="16944" CreationDate="2017-02-13T22:24:00.833" Score="1" ViewCount="1361" Body="&lt;p&gt;I have a bunch of unlabeled text data that I would like to hand label. Are there any tools out there that you can use to produce your own labels? Ideally it would show a sentence at a time and give me the option of choosing a label for each word.&lt;/p&gt;&#xA;" OwnerUserId="5344" LastEditorUserId="843" LastEditDate="2017-03-04T22:13:13.453" LastActivityDate="2022-12-10T07:40:20.527" Title="How to create a training set for sequence labelling" Tags="|nlp|software-recommendation|labels|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16944" PostTypeId="2" ParentId="16940" CreationDate="2017-02-14T01:27:19.207" Score="2" Body="&lt;p&gt;There are several tools for this you could check the &lt;a href=&quot;http://nlp.stanford.edu/software/stanford-manual-annotation-tool-2004-05-16.tar.gz&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Simple Annotation Tool&lt;/a&gt;, also &lt;a href=&quot;http://brat.nlplab.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Brat&lt;/a&gt; has online demos. For a comprehensive list you could check this &lt;a href=&quot;https://www.quora.com/Natural-Language-Processing-What-are-the-best-tools-for-manually-annotating-a-text-corpus-with-entities-and-relationships&quot; rel=&quot;nofollow noreferrer&quot;&gt;question&lt;/a&gt; on Quora&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-02-14T01:27:19.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16945" PostTypeId="2" ParentId="10459" CreationDate="2017-02-14T01:58:09.423" Score="34" Body="&lt;p&gt;Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the Auto data set from the ISLR package in R (the same as in the example you showed).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas.rpy.common as com&#xA;import seaborn as sns&#xA;%matplotlib inline&#xA;&#xA;# load the R package ISLR&#xA;infert = com.importr(&quot;ISLR&quot;)&#xA;&#xA;# load the Auto dataset&#xA;auto_df = com.load_data('Auto')&#xA;&#xA;# calculate the correlation matrix&#xA;corr = auto_df.corr()&#xA;&#xA;# plot the heatmap&#xA;sns.heatmap(corr, &#xA;        xticklabels=corr.columns,&#xA;        yticklabels=corr.columns)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AGfW6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AGfW6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to be even more fancy, you can use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/style.html&quot; rel=&quot;noreferrer&quot;&gt;Pandas Style&lt;/a&gt;, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cmap = cmap=sns.diverging_palette(5, 250, as_cmap=True)&#xA;&#xA;def magnify():&#xA;    return [dict(selector=&quot;th&quot;,&#xA;                 props=[(&quot;font-size&quot;, &quot;7pt&quot;)]),&#xA;            dict(selector=&quot;td&quot;,&#xA;                 props=[('padding', &quot;0em 0em&quot;)]),&#xA;            dict(selector=&quot;th:hover&quot;,&#xA;                 props=[(&quot;font-size&quot;, &quot;12pt&quot;)]),&#xA;            dict(selector=&quot;tr:hover td:hover&quot;,&#xA;                 props=[('max-width', '200px'),&#xA;                        ('font-size', '12pt')])&#xA;]&#xA;&#xA;corr.style.background_gradient(cmap, axis=1)\&#xA;    .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\&#xA;    .set_caption(&quot;Hover to magify&quot;)\&#xA;    .set_precision(2)\&#xA;    .set_table_styles(magnify())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZmIpq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZmIpq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28965" LastEditorUserId="28965" LastEditDate="2017-02-19T03:09:41.420" LastActivityDate="2017-02-19T03:09:41.420" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16946" PostTypeId="2" ParentId="16817" CreationDate="2017-02-14T02:15:46.520" Score="0" Body="&lt;p&gt;Unless you're doing it on some other related file, you're not removing stopwords. If label 16 makes a great use of such, that's a plausible explanation for such result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand (unless again, you're doing it on some other file) you're not reducing words to their morphemes. Not doing so might cause these kind of anomalies. Check the Nltk documentation to learn how to do such thing. &lt;/p&gt;&#xA;" OwnerUserId="28966" LastActivityDate="2017-02-14T02:15:46.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16947" PostTypeId="2" ParentId="16939" CreationDate="2017-02-14T02:52:39.003" Score="1" Body="&lt;p&gt;I've used py-faster-rcnn on video feeds and found it to be pretty usable in a sunny environment. Although on a GTX660, I could only process one image every 600ms, which may highlight a limitation. It still takes a lot of time to compute an inference on a deep net, so for a minimally fast framerate, of 10fps, you're still limited to a speed of 100ms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There might also be other artifacts due to camera dynamics such as image blurring, sensor saturation, or low light levels that might be more present in video feeds of natural scenes as opposed to the fairly well exposed images in static image datasets.&lt;/p&gt;&#xA;" OwnerUserId="28922" LastEditorUserId="28922" LastEditDate="2017-02-14T03:08:01.873" LastActivityDate="2017-02-14T03:08:01.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16948" PostTypeId="1" CreationDate="2017-02-14T03:24:29.140" Score="3" ViewCount="6303" Body="&lt;p&gt;I'm quite new to Deep Learning and trying to solve the problem of Multi-Class, multi-label text classification using Deep Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/imdb_cnn_lstm.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/blob/master/examples/imdb_cnn_lstm.py&lt;/a&gt; .&#xA;I've another dataset. int form of a csv file (&quot;text&quot;,&quot;classifier&quot;), on which i want to perform text classification task. I've tried a few ways to pass my training text to keras but couldn't so I'm stuck at this point. Can anyone suggest me how should I pass my &quot;train.csv&quot; and &quot;test.csv&quot; file to the X_train, y_train and X_test, y_test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically stuck in this line.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;'train.csv' has this format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Job Description:An ideal fitment would apply his/ her advanced analytics expertise at a cutting edge Industrial Analytics specialized Data Science organization; primarily, in any of the following areas- automotive/ energy/ oil &amp;amp; gas/ aerospace/ marine/ chemical.  Experience in statistical modeling, predictive modeling, Random forests, Decision tree, Linear Regression, Correlation, Time- series.  BE / MS/ PhD in Mechanical/ OR/ IE/ computer science/ EE/ chemical.    Mentor/ Lead a small team of data scientist&quot;,Business Analytics &lt;/p&gt;&#xA;&#xA;&lt;p&gt;'test.csv' has same format that is &quot;job_description&quot;,&quot;category&quot;&lt;/p&gt;&#xA;" OwnerUserId="20308" LastEditorUserId="20308" LastEditDate="2017-02-14T05:25:27.983" LastActivityDate="2017-12-17T11:16:41.767" Title="Multi-class text classification with LSTM in Keras" Tags="|keras|dimensionality-reduction|multiclass-classification|multilabel-classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16951" PostTypeId="2" ParentId="16948" CreationDate="2017-02-14T06:11:05.190" Score="3" Body="&lt;p&gt;After I read the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;, I find out that &lt;code&gt;keras.datasets.imdb.load_data&lt;/code&gt; doesn't actually load the plain text data and convert them into vector, it just loads the vector which has been converted before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your problem, I assume you want to convert your &lt;code&gt;job_description&lt;/code&gt; into vector. Maybe you can try &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.feature_extraction.text.CountVectorizer&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no expert on NLP, but I've encountered problems (e.g. &lt;a href=&quot;https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries&quot; rel=&quot;nofollow noreferrer&quot;&gt;Two Sigma Connect: Rental Listing Inquiries&lt;/a&gt;) which need such technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meanwhile, there are other word2vec/embedding techniques you may try. Here is a &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; from keras which shows a detailed example on word embedding.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/sudalairajkumar/two-sigma-connect-rental-listing-inquiries/xgb-starter-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/sudalairajkumar/two-sigma-connect-rental-listing-inquiries/xgb-starter-in-python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-14T06:11:05.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16953" PostTypeId="1" CreationDate="2017-02-14T07:46:30.607" Score="1" ViewCount="140" Body="&lt;p&gt;I have trained my classifier on pictures with a mixture of several classes&#xA;on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes&#xA;on the images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I got more data with pictures showing class G. To minimize my work, I only labeled&#xA;class G on the images and left the rest out (Invalid). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two questions for my training arise:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If there are no examples of class G in my first dataset (because it could have been a subclass&#xA;of A-F), how do I train it correctly? &#xA;Suggested Solution: I can add my new data to the old one, but I have to tell the CNN when there is an image with class G, because I have to change the error measurement of my CNN somehow. But how?  &lt;/li&gt;&#xA;&lt;li&gt;If G was included in my first dataset (assume G is 'police car' and B is 'car', but&#xA;some images showed police car), how do I train in this case correctly?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28977" LastActivityDate="2024-02-12T14:01:10.200" Title="How to train on extended data set correctly" Tags="|neural-network|classification|dataset|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16957" PostTypeId="2" ParentId="16953" CreationDate="2017-02-14T09:06:45.423" Score="0" Body="&lt;p&gt;Transfer learning is what you want to need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Checkout this &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;note&lt;/a&gt; from CS231n, it provides some general advices on transfer learning and model fine-tuning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the meanwhile, this &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; from keras shows how to use pre-trained VGG16 network to classify dogs&amp;amp;cats.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-14T09:06:45.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16958" PostTypeId="1" CreationDate="2017-02-14T09:15:09.943" Score="0" ViewCount="393" Body="&lt;p&gt;I am using Orange 3.3.11 and I cannot find &quot;Geo Map&quot; widget on OS X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the &lt;a href=&quot;http://docs.orange.biolab.si/3/visual-programming/widgets/visualize/geomap.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; says there is such a widget. How can I find this widget?&lt;/p&gt;&#xA;" OwnerUserId="28980" LastEditorUserId="29169" LastEditDate="2022-09-25T14:59:08.633" LastActivityDate="2024-03-06T14:03:50.440" Title="Orange does not show Geo Map tool on OS X" Tags="|orange|geospatial|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="16959" PostTypeId="1" AcceptedAnswerId="17210" CreationDate="2017-02-14T10:00:10.560" Score="2" ViewCount="188" Body="&lt;p&gt;I am preparing for clustering the data which can be only represent as a extremely sparse binary vectors. Each of the objects is represented by a large set the binary features (&lt;span class=&quot;math-container&quot;&gt;$10^3$&lt;/span&gt; ~ &lt;span class=&quot;math-container&quot;&gt;$10^6$&lt;/span&gt;), each identified by a 32-bit hash code. We can assume that in order to represent the data object without loss of information we need to see it as a vector of &lt;span class=&quot;math-container&quot;&gt;$2^{32}$&lt;/span&gt; bits - in which only the bits at positions identified by hash are set to one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, having two data objects, each represented by a sorted set of hashes, how can I calculate the distance between them without storing two &lt;span class=&quot;math-container&quot;&gt;$2^{32}$&lt;/span&gt; bits long vectors in memory?&lt;/p&gt;&#xA;" OwnerUserId="28938" LastEditorUserId="29169" LastEditDate="2019-06-07T06:11:40.373" LastActivityDate="2019-06-07T06:11:40.373" Title="Memory-efficient metric calculation for ultra high dimensional data" Tags="|clustering|distance|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16961" PostTypeId="1" AcceptedAnswerId="16980" CreationDate="2017-02-14T12:35:59.897" Score="1" ViewCount="744" Body="&lt;p&gt;I'm having trouble finding a good reward function for the pendulum problem, the function I'm using: $-x^2 - 0.25*(\text{xdot}^2)$&#xA;which is the quadratic error from the top. with $x$ representing the current location of the pendulum and $\text{xdot}$ the angular velocity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It takes a lot of time with this function and sometimes doesn't work.&#xA;Any one have some other suggestions?&#xA;I've been looking in google but didn't find anything i could use&lt;/p&gt;&#xA;" OwnerUserId="28986" LastEditorUserId="28628" LastEditDate="2017-02-14T16:32:13.553" LastActivityDate="2017-02-15T09:43:36.167" Title="Reinforcement learning, pendulum python" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16963" PostTypeId="2" ParentId="16961" CreationDate="2017-02-14T12:52:30.680" Score="0" Body="&lt;p&gt;In reinforcement learning, you should avoid scoring interim results based on heuristics. Unlike supervised learning, or a search algorithm, you are not trying to guide the behaviour, just reward good results. For an inverted pendulum a good result might simply be &quot;has not fallen over so far&quot;, although there is nothing &lt;em&gt;inherently&lt;/em&gt; wrong with a cost function which expresses cost in terms of minimising differences from an ideal, you do have to take more care with the values used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you are using discounting, and continuous (not episodic) approach, then reward can be 0 for &quot;not falling over&quot; and -1 for &quot;it fell over&quot;, followed by a re-set/continue. You can check for falling by measuring whether the pendulum has reached some large angle to the vertical (e.g. 45 degrees or more).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an episodic approach, it is more natural to have +1 &quot;ok&quot; and 0 for the end state &quot;fell over&quot;, although the 0/-1 scheme would also work. However, you want to avoid having negative values for any state which is &quot;ok&quot;, because that is basically telling the agent to hurry up and end the episode. In your case, ending the episode is bad, so you don't want that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want to reward &quot;perfection&quot; in your episodic approach, then your formula might work better if you added a positive offset, so that the agent has an incentive to continue the episode if possible. You should choose a value such that recoverable states are positive.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Note that the above analysis applies only to certain episode-based approaches. It depends critically on what you count as an episode, and whether the agent is able to take an action which ends the episode.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-15T09:43:36.167" LastActivityDate="2017-02-15T09:43:36.167" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16965" PostTypeId="1" AcceptedAnswerId="16969" CreationDate="2017-02-14T14:07:41.557" Score="0" ViewCount="3156" Body="&lt;p&gt;Hi I am experimenting with stanford parser and NER with python&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Input = &quot;Rami Eid is studying at Stony Brook University in NY&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Parser Output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vc1t6.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vc1t6.jpg&quot; alt=&quot;stanford parser&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NER Output :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[(u'Rami', u'PERSON'), (u'Eid', u'PERSON'), (u'is', u'O'), (u'studying', u'O'), (u'at', u'O'), (u'Stony', u'ORGANIZATION'), (u'Brook', u'ORGANIZATION'), (u'University', u'ORGANIZATION'), (u'in', u'O'), (u'NY', u'O')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now can I combine NER results with Parser result ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(u'Rami', u'NNP'), (u'Eid', u'NNP') ==&amp;gt; u('Rami EID', u'PERSON')&#xA;&#xA;(u'Stony', u'NNP'), (u'Brook', u'NNP'), (u'University', u'NNP') ==&amp;gt; (u'Stony Brook University',u'ORGANIZATION')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will get replaced in the graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How this can be done ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-02-14T15:38:16.313" Title="Stanford parser Python : Combine NER and POS tags" Tags="|python|nlp|named-entity-recognition|stanford-nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16966" PostTypeId="2" ParentId="9325" CreationDate="2017-02-14T14:40:53.063" Score="1" Body="&lt;p&gt;Try &lt;a href=&quot;http://rasbt.github.io/mlxtend/&quot; rel=&quot;nofollow noreferrer&quot;&gt;mlxtend&lt;/a&gt;. &lt;a href=&quot;http://rasbt.github.io/mlxtend/user_guide/evaluate/confusion_matrix/#example-2-multi-class-classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s an example of multi-class case.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ATQ26.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ATQ26.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="98307" LastEditDate="2020-08-03T12:56:58.390" LastActivityDate="2020-08-03T12:56:58.390" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="16968" PostTypeId="2" ParentId="6435" CreationDate="2017-02-14T15:25:09.120" Score="1" Body="&lt;p&gt;You can look these paper. First one is quite related to your task.&lt;br&gt;&#xA;&lt;a href=&quot;http://icamt2016.org/papers/SS-LTMLBDA-06-05.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://icamt2016.org/papers/SS-LTMLBDA-06-05.pdf&lt;/a&gt;&#xA;&lt;a href=&quot;http://researcher.watson.ibm.com/researcher/files/us-kephart/icml00_qrt.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://researcher.watson.ibm.com/researcher/files/us-kephart/icml00_qrt.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2017-02-14T15:25:09.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16969" PostTypeId="2" ParentId="16965" CreationDate="2017-02-14T15:38:16.313" Score="0" Body="&lt;p&gt;This question has already been answered. See Alexis' answer at &lt;a href=&quot;https://stackoverflow.com/questions/30664677/extract-list-of-persons-and-organizations-using-stanford-ner-tagger-in-nltk&quot;&gt;https://stackoverflow.com/questions/30664677/extract-list-of-persons-and-organizations-using-stanford-ner-tagger-in-nltk&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use chunking NER without the Stanford library see alvas' answer at &lt;a href=&quot;https://stackoverflow.com/questions/31836058/nltk-named-entity-recognition-to-a-python-list&quot;&gt;https://stackoverflow.com/questions/31836058/nltk-named-entity-recognition-to-a-python-list&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23144" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-14T15:38:16.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16970" PostTypeId="2" ParentId="9456" CreationDate="2017-02-14T15:49:46.903" Score="1" Body="&lt;p&gt;At my work we've had some success just using averages of pre-trained embeddings (e.g. the GloVe vectors) for classifying short texts. Have you tried that?&lt;/p&gt;&#xA;" OwnerUserId="13232" LastActivityDate="2017-02-14T15:49:46.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16974" PostTypeId="2" ParentId="16959" CreationDate="2017-02-14T17:26:13.393" Score="0" Body="&lt;p&gt;Euclidean distance can be decomposed into dot products&#xA;$$\|a-b\|^2=\|a\|^2-2(a\cdot{b})+\|b\|^2$$&#xA;which can be efficiently computed for sparse vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For binary vectors, $\|a\|^2$ is the size of $a$'s non-zero index set, while $a\cdot{b}$ is the size of the intersection of the index-sets of $a$ and $b$. (For &lt;a href=&quot;https://en.wikipedia.org/wiki/Bit_array&quot; rel=&quot;nofollow noreferrer&quot;&gt;bit arrays&lt;/a&gt; intersection is just a bitwise &quot;and&quot;.)&lt;/p&gt;&#xA;" OwnerUserId="24240" LastEditorUserId="24240" LastEditDate="2017-02-14T19:24:12.740" LastActivityDate="2017-02-14T19:24:12.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16976" PostTypeId="1" CreationDate="2017-02-14T21:04:46.843" Score="1" ViewCount="5717" Body="&lt;p&gt;I have a huge string in R where I want to remove single characters from. So for example, the string might be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;blabla a test this that b á&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the result should be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;blabla test this that&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do this in R?&lt;/p&gt;&#xA;" OwnerUserId="29007" LastActivityDate="2017-02-14T21:34:13.033" Title="R: remove single characters from string" Tags="|r|regex|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16977" PostTypeId="2" ParentId="16959" CreationDate="2017-02-14T21:11:54.940" Score="0" Body="&lt;p&gt;Simply &lt;em&gt;join&lt;/em&gt; the two sorted lists. You only need one pass over each list if you do them in parallel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you want this to be fast, you may want to use roaring bitmaps, to count the number of shared and different bits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you have not noticed: on binary data, pretty much any distance function reduces to counting intersection sizes and number of bits set.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-02-14T21:11:54.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16978" PostTypeId="2" ParentId="16976" CreationDate="2017-02-14T21:34:13.033" Score="2" Body="&lt;p&gt;This splits the string and filters the substrings by length, no regex:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;string &amp;lt;- &quot;blabla a test this that b á&quot;&#xA;paste(Filter(function(x) nchar(x) &amp;gt; 1,&#xA;             unlist(strsplit(string, &quot; &quot;))), &#xA;      collapse=&quot; &quot;)&#xA;# [1] &quot;blabla test this that&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(does not work with multiple space characters)&lt;/p&gt;&#xA;" OwnerUserId="2961" LastActivityDate="2017-02-14T21:34:13.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="16980" PostTypeId="2" ParentId="16961" CreationDate="2017-02-14T23:50:35.533" Score="1" Body="&lt;p&gt;You could use the same reward function that &lt;a href=&quot;https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Openai&lt;/a&gt;'s Inverted Pendulum is using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$costs=-(\Delta_{2\pi}\theta)^2 - 0.1(\dot{\theta})^2 - 0.001u^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $(\Delta_{2\pi}\theta)$ is the difference between current and desired angular position performed using modulo $2\pi$. The variable $u$ denotes the torque (the action of your RL agent). The optimal is to be as close to zero costs as it gets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is that you have a control problem in which you can come up with a quadratic 'energy' or cost function that tells you the cost of performing an action at EVERY single time step. In this paper (&lt;a href=&quot;https://webdocs.cs.ualberta.ca/~sutton/papers/SSR-98.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;p.33 section 5.2&lt;/a&gt;) you can find a detailed description.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tested RL algorithms in this objective function and I did not encounter any problems for convergence in both MATLAB and Python. If you still have problems let us know what kind of RL approach you implemented and how you encoded the location of the pendulum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-02-14T23:50:35.533" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="16982" PostTypeId="2" ParentId="10338" CreationDate="2017-02-14T23:58:21.597" Score="3" Body="&lt;p&gt;It's hard to say what is state-of-the-art in general without breaking aspect-level sentiment analysis down into its subtasks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Aspect extraction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Sentiment classification&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you've probably read in Liu's book, aspect extraction can be done relatively well by extracting the most common noun phrases and adding some heuristics. This works particularly well when you are dealing with texts that revolve around a few topics. Topic-model based techniques (LDA etc) are better, but more complicated to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for classification, all current state-of-the-art approaches use neural networks (Recurrent NNs or Convolutional NNs). At sentence-level &lt;a href=&quot;http://www.people.fas.harvard.edu/~yoonkim/data/emnlp_2014.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kim&lt;/a&gt; (2014) is still soa on several datasets. There was a &lt;a href=&quot;https://www.aclweb.org/anthology/D/D16/D16-1058.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Wang et al. about attention-based LSTMs for aspect-level sentiment analysis in EMNLP last year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd suggest looking at the recent SemEval tasks (2014 task 4, 2015 task 12, 2016 task 5) on aspect-based sentiment analysis. There's a lot of good ideas that you could pick up on there.&lt;/p&gt;&#xA;" OwnerUserId="29010" LastActivityDate="2017-02-14T23:58:21.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16984" PostTypeId="1" AcceptedAnswerId="17039" CreationDate="2017-02-15T03:49:26.383" Score="2" ViewCount="175" Body="&lt;p&gt;Currently I need one data mining platform (or solution) meets such requirements:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Data Source&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I can import data from: CSV, MySQL, PostgreSQL, &lt;strong&gt;HDFS&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optional: NoSQL, MongoDB.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Interface&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I can assemble models and datas with drag &amp;amp; drops (like Weka).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Algorithms&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I can model with common machine learning algorithms: Logistic Regression, Random Forest, SVM, word2vec, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be awesome if it had deep learning support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optional: PMML&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Customization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I can implement my own algorithms (under certain standard).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optional: I can implement my own APIs (under certain standard).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Programming language is not limited.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Visualization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Simple visualization is okay.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;License&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Commercial solution is acceptable, open-source solution would be better (I can do some tweaks in order to meet my requirement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two software I've investigated: Weka and SPSS Modeler. Weka doesn't support HDFS, while SPSS Modeler doesn't support customized algorithms.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="28628" LastEditDate="2017-02-17T18:10:01.620" LastActivityDate="2017-03-12T10:31:18.933" Title="Need some help on data mining platform recommendation" Tags="|data-mining|software-recommendation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="16985" PostTypeId="1" AcceptedAnswerId="17000" CreationDate="2017-02-15T04:18:02.997" Score="3" ViewCount="172" Body="&lt;p&gt;From &lt;a href=&quot;https://github.com/google/deepdream/blob/master/dream.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/google/deepdream/blob/master/dream.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def objective_L2(dst):          # Our training objective. Google has since release a way to load&#xA;    dst.diff[:] = dst.data      # arbitrary objectives from other images. We'll go into this later.&#xA;&#xA;def make_step(net, step_size=1.5, end='inception_4c/output', &#xA;              jitter=32, clip=True, objective=objective_L2):&#xA;    '''Basic gradient ascent step.'''&#xA;&#xA;    src = net.blobs['data'] # input image is stored in Net's 'data' blob&#xA;    dst = net.blobs[end]&#xA;&#xA;    ox, oy = np.random.randint(-jitter, jitter+1, 2)&#xA;    src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift&#xA;&#xA;    net.forward(end=end)&#xA;    objective(dst)  # specify the optimization objective&#xA;    net.backward(start=end)&#xA;    g = src.diff[0]&#xA;    # apply normalized ascent step to the input image&#xA;    src.data[:] += step_size/np.abs(g).mean() * g&#xA;&#xA;    src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image&#xA;&#xA;    if clip:&#xA;        bias = net.transformer.mean['data']&#xA;        src.data[:] = np.clip(src.data, -bias, 255-bias)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I understand what is going on correctly, the input image in &lt;code&gt;net.blobs['data']&lt;/code&gt; is inserted into the NN until the layer &lt;code&gt;end&lt;/code&gt;. Once, the forward pass is complete until &lt;code&gt;end&lt;/code&gt;, it calculates how &quot;off&quot; the blob at the &lt;code&gt;end&lt;/code&gt; is from &quot;something&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What is this &quot;something&quot;? Is it &lt;code&gt;dst.data&lt;/code&gt;? I stepped through a debugger and found that &lt;code&gt;dst.data&lt;/code&gt; was just a matrix of zeros right after the assignment and then filled with values after the backward pass.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Anyways, assuming it finds how &quot;off&quot; the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but &quot;morph&quot; the input image into whatever the original model's layer represents.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What exactly does &lt;code&gt;src.data[:] += step_size/np.abs(g).mean() * g&lt;/code&gt; do? It seems like applying whatever calculation was done above to the original image. Is this line what actually &quot;morphs&quot; the image?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Links that I have already read through&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/a/31028871/2750819&quot;&gt;https://stackoverflow.com/a/31028871/2750819&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would be interested in what the author of the accepted answer meant by &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;we take the original layer blob and &quot;enchance&quot; signals in it. What&#xA;  does it mean, I don't know. Maybe they just multiply the values by&#xA;  coefficient, maybe something else.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.kpkaiser.com/machine-learning/diving-deeper-into-deep-dreams/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kpkaiser.com/machine-learning/diving-deeper-into-deep-dreams/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this blog post the author comments next to &lt;code&gt;src.data[:] += step_size/np.abs(g).mean() * g&lt;/code&gt;: &quot;get closer to our target data.&quot; I'm not too clear what &quot;target data&quot; means here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note I'm cross posting this from &lt;a href=&quot;https://stackoverflow.com/q/40690099/2750819&quot;&gt;https://stackoverflow.com/q/40690099/2750819&lt;/a&gt; as I was recommended to in a comment.&lt;/p&gt;&#xA;" OwnerUserId="29015" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-05-01T07:48:40.503" Title="Clarification wanted for make_step function of Google's deep dream script" Tags="|machine-learning|python|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16987" PostTypeId="5" CreationDate="2017-02-15T05:49:11.733" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-02-15T05:49:11.733" LastActivityDate="2017-02-15T05:49:11.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16988" PostTypeId="4" CreationDate="2017-02-15T05:49:11.733" Score="0" Body="A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-03T08:26:31.300" LastActivityDate="2019-04-03T08:26:31.300" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16989" PostTypeId="5" CreationDate="2017-02-15T05:51:58.990" Score="0" Body="&lt;p&gt;PyTorch is a python package that provides two high-level features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Tensor computation (like numpy) with strong GPU acceleration&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Deep Neural Networks built on a tape-based autodiff system&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pytorch is an alternative to other deep learning libraries like Tensorflow and Theano with an emphasis on 'Python First' Imperative Programming, designed to be familiar to users for faster development iterations.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-05-07T02:42:02.160" LastActivityDate="2018-05-07T02:42:02.160" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16990" PostTypeId="4" CreationDate="2017-02-15T05:51:58.990" Score="0" Body="Pytorch is an open source library for Tensors and Dynamic neural networks in Python&#xD;&#xA;with strong GPU acceleration. For details, see https://pytorch.org. " OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-05-07T02:41:46.043" LastActivityDate="2018-05-07T02:41:46.043" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="16991" PostTypeId="1" AcceptedAnswerId="16996" CreationDate="2017-02-15T07:08:31.970" Score="2" ViewCount="592" Body="&lt;p&gt;I have achieved 68% accuracy with my logistic regression model. I want to increase the accuracy of the model. How can I apply stepwise regression in this code and how beneficial it would be for my model? What changes shall I make in my code to get more accuracy with my data set. I have attached my dataset below. Following is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;data1 &amp;lt;- read.csv(&quot;~/hj.csv&quot;, header=T)&#xA;train&amp;lt;- data1[1:116,]&#xA;VALUE&amp;lt;-as.numeric(rownames(train))&#xA;testset&amp;lt;- data1[1:116,]&#xA;mylogit &amp;lt;- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family =&quot;binomial&quot;)&#xA;testset$predicted.value = predict(mylogit, newdata = testset, type=&quot;response&quot;)&#xA;for (i in 1: nrow(testset)){&#xA;  if(testset$predicted.value[i] &amp;lt;= 0.50)&#xA;    testset$outcome[i] &amp;lt;- 0 &#xA;  else testset$outcome[i] &amp;lt;- 1&#xA;}&#xA;print(testset)&#xA;tab = table(testset$VALUE, testset$outcome) %&amp;gt;% as.matrix.data.frame()&#xA;accuracy = sum(diag(tab))/sum(tab)&#xA;print(accuracy)&#xA;print(tab)&#xA;table(testset$VALUE, testset$outcome)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cfQoh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cfQoh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is my dataset:&#xA;Link 1: &lt;a href=&quot;http://www.filedropper.com/hj_2&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.filedropper.com/hj_2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27231" LastEditorUserId="12384" LastEditDate="2017-02-15T17:10:17.000" LastActivityDate="2017-02-26T19:35:37.757" Title="Accuracy improvement for logistic regression model" Tags="|logistic-regression|accuracy|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16992" PostTypeId="1" CreationDate="2017-02-15T07:24:16.077" Score="0" ViewCount="57" Body="&lt;p&gt;I am reading this paper &lt;a href=&quot;https://arxiv.org/pdf/1606.07792.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;wide and deep learning recommedation system&lt;/a&gt; paper and haven't understood one thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To serve the latency, they actually first get the 100 best apps according to user query and rank them according to the probability score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that while buidling the model we have to build individual model for each app?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other Deep learning frameworks similar to this ?&lt;/p&gt;&#xA;" OwnerUserId="28482" LastActivityDate="2018-03-10T10:49:13.710" Title="In Recommendation systems, Does we need to build each model for each product if we are using Logistic regression?" Tags="|deep-learning|logistic-regression|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16993" PostTypeId="1" CreationDate="2017-02-15T07:40:44.403" Score="3" ViewCount="312" Body="&lt;p&gt;I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What strategies are there to deal with this problem (besides a hold out group). &#xA;Recently I read &lt;a href=&quot;http://ieeexplore.ieee.org/document/7280527/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/7280527/&lt;/a&gt; wich additionally incorporates early feedback.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up  to 30 days will in reality be up to 6 classes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 prediction of label 0&lt;/li&gt;&#xA;&lt;li&gt;1 prediction of label 1&lt;/li&gt;&#xA;&lt;li&gt;feedback that label was actually&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 for a prediction of 0 (regular outcome)&lt;/li&gt;&#xA;&lt;li&gt;1 for a prediction of 1 (regular /desired outcome)&lt;/li&gt;&#xA;&lt;li&gt;0 for a prediction of 1 (model did falsely classify, groundtruth affected)&lt;/li&gt;&#xA;&lt;li&gt;1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?&lt;/p&gt;&#xA;" OwnerUserId="9053" LastEditorUserId="9053" LastEditDate="2017-02-15T08:15:15.667" LastActivityDate="2023-09-06T15:17:00.917" Title="How to deal with a machine learning model which affects future ground truth data?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="16994" PostTypeId="1" AcceptedAnswerId="18724" CreationDate="2017-02-15T08:15:32.047" Score="5" ViewCount="1136" Body="&lt;p&gt;One need to provide LDA with a predefined number of latent topics. Let say I have a text corpus in which I hypothesize there are 10 major topics, all composed of 10 minor subtopics. My objective is to be able to define proximity between documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How do you estimate the number of topics in practice ? Empirically ? With another method like Hierarchical Dirichlet Process (HDP) ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Do you build several models ? For major and minor topics ? Is there a way to capture the hierarchical structure of the topics ?&lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-02-15T10:03:06.153" LastActivityDate="2017-05-02T20:44:29.283" Title="How do you work with Latent Dirichlet Allocation in practice" Tags="|dirichlet|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="16995" PostTypeId="1" AcceptedAnswerId="17003" CreationDate="2017-02-15T08:19:33.420" Score="0" ViewCount="445" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dJbqN.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dJbqN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The graph above shows how accuracy stops increasing after reaching a certain number of features. There are also sudden drops in accuracy at some points. Can this be attrrubuted to overfitting? I am training a decision tree by the way.&lt;/p&gt;&#xA;" OwnerUserId="26522" LastActivityDate="2017-02-15T14:50:47.533" Title="Overfitting in machine learning" Tags="|machine-learning|classification|decision-trees|overfitting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="16996" PostTypeId="2" ParentId="16991" CreationDate="2017-02-15T08:50:07.347" Score="1" Body="&lt;p&gt;Try:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;mylogit &amp;lt;- glm(VALUE ~ POINT1 * POINT2 * POINT3 * POINT4, data = data1, family =&quot;binomial&quot;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with about 72% accuracy.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-02-15T08:50:07.347" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="16999" PostTypeId="1" CreationDate="2017-02-15T09:35:14.283" Score="1" ViewCount="471" Body="&lt;p&gt;I am debugging a code which I haven't written. I want to print out &lt;code&gt;state_below_&lt;/code&gt; variable which is calculated as following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;state_below_ = tensor.dot(state_below*emb_dropout[1], tparams[pp(prefix, 'W')]) +\&#xA;        tparams[pp(prefix, 'b')]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I use &lt;code&gt;state_below_.eval()&lt;/code&gt;, I get &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MissingInputError: (&quot;An input of the graph, used to compute&#xA;  Reshape{1}(y_sampler, TensorConstant{(1,) of -1}), was not provided&#xA;  and not given a value.Use the Theano flag&#xA;  exception_verbosity='high',for more information on this error.&quot;,&#xA;  y_sampler)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;error. How can I print this damn &quot;state_below_&quot; value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="28138" LastActivityDate="2017-02-15T17:32:52.177" Title="printing theano.tensor without having any idea about the shape and type of the tensor" Tags="|python|tensorflow|theano|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17000" PostTypeId="2" ParentId="16985" CreationDate="2017-02-15T10:03:47.787" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What is this &quot;something&quot;? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. &lt;code&gt;dst.data&lt;/code&gt; is the working contents of the layer inside the CNN that you are trying to maximise. The idea is that you want to generate an image that has a high neuron activation in this layer by making changes to the input. If I understand this correctly though, it should be populated immediately after the forward pass here: &lt;code&gt;net.forward(end=end)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Anyways, assuming it finds how &quot;off&quot; the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but &quot;morph&quot; the input image into whatever the original model's layer represents.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is not training. However, like with training we cannot directly measure how &quot;off&quot; the source that we want to change is from an ideal value, so instead we calculate how to move toward a better value by taking gradients. Back propagation is the usual method for figuring out gradients to parameters in the CNN. There are some main differences with training:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of trying to minimise a cost, we want to increase a metric which summarises how excited the target layer is - we are not trying to find any stationary point (e.g. the maximum possible value), and instead Deep Dream usually just stops arbitrarily after a fixed number of iterations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We back propagate further than usual, all the way to the input layer. That's not something you do normally during training.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We don't use any of the gradients to the weights. The weights in the neural network are never changed. We are only interested in the gradients at the input - but to get them we need to calculate all the others first.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually &quot;morphs&quot; the image?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It takes a single step in the image data along the gradients that we have calculated will trigger more activity in the target layer. Yes this alters the input image. It should be repeated a few times, according to how extreme you want the Deep Dream effect to be.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-15T13:43:28.580" LastActivityDate="2017-02-15T13:43:28.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17001" PostTypeId="2" ParentId="14568" CreationDate="2017-02-15T10:52:59.567" Score="0" Body="&lt;p&gt;You can train stacked RBMs on your images, and then train the final RBM on a concatenation of the output from the RBM stack and the labels. Then you can actually use the RBM for classification. This article by Hinton ++ explains this approach &lt;a href=&quot;http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Fast Learning Algorithm for Deep Belief Nets&#xA;&lt;/a&gt;, also you can have a look at this &lt;a href=&quot;http://www.cs.toronto.edu/~hinton/adi/index.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;demo&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29025" LastActivityDate="2017-02-15T10:52:59.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17002" PostTypeId="2" ParentId="9325" CreationDate="2017-02-15T11:32:11.393" Score="0" Body="&lt;p&gt;Scikit-learn does support multi-label confusion matrix. See the links below for documentation and user guide:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29025" LastActivityDate="2017-02-15T11:32:11.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17003" PostTypeId="2" ParentId="16995" CreationDate="2017-02-15T12:05:34.263" Score="0" Body="&lt;p&gt;I can tell from your screenshot that you are plotting the validation accuracy. When you overfit your &lt;strong&gt;training accuracy&lt;/strong&gt; should be very high, but your &lt;strong&gt;validation accuracy&lt;/strong&gt; should get lower and lower. Or if you think in terms of error rather than accuracy you should see the following plot in case of overfitting. In the figure below the x-axis contains the training progress, i.e. the number of training iterations. The training error (blue) keeps decreasing, while the validation error (red) starts increasing at the point where you start overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TVkSt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TVkSt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This picture is from the wikipedia article on overfitting by the way: &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Overfitting&lt;/a&gt; Have a look. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your question: No, I don't think you are overfitting. If increasing the number of features would make the overfitting more and more significant the validation accuracy should be falling, not stay constant. In your case it seems that more features are simply no longer adding additional benefit for the classification. &lt;/p&gt;&#xA;" OwnerUserId="29026" LastEditorUserId="29026" LastEditDate="2017-02-15T14:50:47.533" LastActivityDate="2017-02-15T14:50:47.533" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="17005" PostTypeId="2" ParentId="16999" CreationDate="2017-02-15T17:32:52.177" Score="1" Body="&lt;p&gt;For evaluating and tensor, you have to pass the value of the input tensors as shown in the below code snippet.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import theano.tensor as T&#xA;x = T.dscalar('x')&#xA;y = T.dscalar('y')&#xA;z = x + y&#xA;print(z.eval({x : 16.3, y : 12.1}))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2017-02-15T17:32:52.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17006" PostTypeId="1" CreationDate="2017-02-15T18:35:38.193" Score="3" ViewCount="573" Body="&lt;p&gt;I am doing a study on unsupervised data with various categorical variables. So I have found the FactoMineR package to be really handy for this, particularly with the FAMD functions. I can get to a base FAMD object just fine (and the corresponding charts) but I can't seem to get the predict.FAMD side working. Are there any working examples out there? Here's what I've done:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have a mix of continuous and categorical data&lt;/li&gt;&#xA;&lt;li&gt;I have split the data into a test and training set&lt;/li&gt;&#xA;&lt;li&gt;I have created a FAMD object with my training set&lt;/li&gt;&#xA;&lt;li&gt;I try to run predict.FAMD against the test set data but I keep on getting an an error of &lt;code&gt;Error in [.data.frame(newdata, , i) : undefined columns selected&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Generally, my code looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(FactoMineR)&#xA;res = FAMD(myTrainingData)&#xA;res.predict = predict(object=res,newdata=myTestData)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've also tried variations on explicit columns and even no rows, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res.predict = predict(object=res,newdata=myTestData[,1:7])&#xA;res.predict = predict(object=res,newdata=myTestData[,c(1,2,3,4,5,6,7)])&#xA;res.predict = predict(object=res,newdata=myTestData[0,1:7])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All of these result in the same error. Help?&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-02-15T18:35:38.193" Title="Examples for predict.FAMD?" Tags="|r|unsupervised-learning|pca|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17007" PostTypeId="2" ParentId="16928" CreationDate="2017-02-15T19:50:00.540" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.467.2430&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Emergent pattern mining&lt;/a&gt; could be useful for you. In classification, emergent patterns are subsets of features which differentiate one class from the others. Look at Example 1.1 in the paper. Two feature subests, denoted $X$ and $Y$, are EPs because $X$ has large support in one class and small support in the other, and vice versa for $Y$. The nifty thing about EP mining is that it considers &lt;em&gt;subsets&lt;/em&gt; of features instead of individual features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mining patterns in this way could address your problem of &lt;code&gt;feature1&lt;/code&gt; and &lt;code&gt;feature2&lt;/code&gt; having no correlation with the target class alone, but the combination &lt;code&gt;feature1 AND feature2&lt;/code&gt; having high correlation with the target class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I don't see any open-source software implementations on the web. However, there are a number of papers out there which implement some sort of EP mining.&lt;/p&gt;&#xA;" OwnerUserId="26024" LastActivityDate="2017-02-15T19:50:00.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17010" PostTypeId="2" ParentId="15110" CreationDate="2017-02-16T05:43:34.930" Score="8" Body="&lt;p&gt;there are multiple issues with the code:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1.&#xA;This will later be interpreted as image in black and the darkest form of gray (1 out of 255).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once you have proper floats as values PIL or pillow can't handle the array (they only do images with values in [0, 255])&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The first problem happened because you/numpy wants the array to be a uint8.&#xA;The normalize version will have floats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should have used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_prime = X.astype(float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a working version of the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import scipy&#xA;import scipy.misc&#xA;from PIL import Image&#xA;&#xA;&#xA;def global_contrast_normalization(filename, s, lmda, epsilon):&#xA;    X = numpy.array(Image.open(filename))&#xA;&#xA;    # replacement for the loop&#xA;    X_average = numpy.mean(X)&#xA;    print('Mean: ', X_average)&#xA;    X = X - X_average&#xA;&#xA;    # `su` is here the mean, instead of the sum&#xA;    contrast = numpy.sqrt(lmda + numpy.mean(X**2))&#xA;&#xA;    X = s * X / max(contrast, epsilon)&#xA;&#xA;    # scipy can handle it&#xA;    scipy.misc.imsave('result.jpg', X)&#xA;&#xA;&#xA;global_contrast_normalization(&quot;cat.jpg&quot;, 1, 10, 0.000000001)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;PS: &lt;code&gt;X_prime = X&lt;/code&gt; will make &lt;code&gt;X_prime&lt;/code&gt; reference &lt;code&gt;X&lt;/code&gt;. So changing &lt;code&gt;X_prime&lt;/code&gt; will also change &lt;code&gt;X&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29054" LastEditorUserId="67546" LastEditDate="2019-02-13T03:14:35.253" LastActivityDate="2019-02-13T03:14:35.253" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17012" PostTypeId="1" CreationDate="2017-02-16T07:32:17.380" Score="6" ViewCount="598" Body="&lt;p&gt;I have a list of about 20 tasks that a machine need to be performed by a machine. Each tasks consists of a sequence of 3 to 5 actions that must be executed sequentially to complete a task. There are a total of 50 different actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want an algorithm to learn which actions to execute to complete the tasks. To learn this I have a dataset of approximately 1000 samples. Each sample consists of a task with corresponding actions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a task $T_n$, the machine needs to come up with the sequence of actions to complete the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample dataset ($T$ for tasks out of 20, $A$ for actions out of 50):&lt;br&gt;&#xA;$T_1$ - $(A_5 \rightarrow A_1 \rightarrow A_3)$&lt;br&gt;&#xA;$T_2$ - $(A_7\rightarrow A_12\rightarrow A_4\rightarrow A_1)$&lt;br&gt;&#xA;$T_1$ - $(A_5\rightarrow A_3\rightarrow A_{13})$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to solve this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my knowledge of linear regression and classification, I believe that both are not the right approach to solve the task at hand.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;EDIT 1:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;One way I can think to solve this problem.&#xA;The machine doesn't have to come up with the whole sequence instantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a new task has to be done, the machine has an option to choose the next action (first action) from, say, 5 available actions it can perform. After performing this action, it would have, say, 5 more actions and it has to choose it's next action to complete. And go on doing these actions to complete the given task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this kind of a finite state machine?&lt;/p&gt;&#xA;" OwnerUserId="3299" LastEditorUserId="836" LastEditDate="2017-02-16T08:13:01.680" LastActivityDate="2020-10-08T11:39:38.167" Title="Identifying sequences of actions required to complete tasks, based on data of completed tasks" Tags="|machine-learning|supervised-learning|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="17013" PostTypeId="1" CreationDate="2017-02-16T09:43:52.383" Score="4" ViewCount="9060" Body="&lt;p&gt;For machine learning, we need to normalized the inputs (features) for good results.&#xA;Do we need to normalize the output as well during training?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2017-02-16T11:39:31.457" Title="Normalized output of machine learning" Tags="|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17014" PostTypeId="1" CreationDate="2017-02-16T09:46:34.167" Score="3" ViewCount="105" Body="&lt;p&gt;Does there exist a definition for weighted degrees of multidimensional networks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the basic definition would be: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let $v\in V$ be a node of a network $G=(N,V,L)$. The function Weighted Degree: $$V \times P(L) \longrightarrow \mathbb{N}$$ is defined as&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$${weighted\_degree}(v,D)=\sum_{(u,v,d)\in G, d\in D}\ weight(u,v)$$&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where $D\subseteq L$ and $weight(u,v)$ is the weight of the edge $(u,v)$. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I cannot find this definition anywhere on the Internet. But would that be right? Is a multidimensional version of any interest?&lt;/p&gt;&#xA;" OwnerUserId="28934" LastEditorUserId="17538" LastEditDate="2017-08-29T13:04:11.420" LastActivityDate="2018-12-23T04:03:30.830" Title="Weighted degree in Multidimensional networks" Tags="|predictive-modeling|clustering|graphs|social-network-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17015" PostTypeId="2" ParentId="17013" CreationDate="2017-02-16T11:26:45.903" Score="6" Body="&lt;p&gt;First you do not always need to normalize (standardize) the input vectors (feature vectors), sometimes is good, sometimes is bad. In general you scale your feature vector when the magnitude of a feature dominates the others, so the model cannot pick up the contribution of the smaller magnitude features. Read &lt;a href=&quot;https://stats.stackexchange.com/questions/189652/is-it-a-good-practice-to-always-scale-normalize-data-for-machine-learning&quot;&gt;here&lt;/a&gt; for a detailed explanation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second there are two general classes of machine learning problems: classification and regression. In a classification type problem the output (dependent variable) is discrete, so you do not need to normalize it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a regression type problem scaling the output do not affect the shape of your function see &lt;a href=&quot;https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re&quot;&gt;here&lt;/a&gt;. Moreover it does not affect error functions like Mean Squared Error, i.e. the error is also scaled. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-16T11:39:31.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17016" PostTypeId="1" CreationDate="2017-02-16T12:18:27.260" Score="6" ViewCount="111" Body="&lt;p&gt;I'm trying to find an indexing data structure most suitable for my &lt;strong&gt;metric space&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;set of IP network related data (IP addresses, ports, TCP flags, ...),&lt;/li&gt;&#xA;&lt;li&gt;distance function is continuous, non-Euclidean but satisfies non-negativity, symmetry and triangle inequality,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;with respect to the &lt;strong&gt;fixed-radius range query&lt;/strong&gt; performance. I want to use it for a clustering algorithm (DBSCAN or similar) on a data sets with millions of elements. So far I have studied:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ball trees/VPT,&lt;/li&gt;&#xA;&lt;li&gt;MVPT,&lt;/li&gt;&#xA;&lt;li&gt;GHT, GNAT,&lt;/li&gt;&#xA;&lt;li&gt;AESA/LAESA,&lt;/li&gt;&#xA;&lt;li&gt;cover trees,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and basically all other methods from &lt;a href=&quot;http://www.academia.edu/download/44050685/p273-chavez.pdf&quot; rel=&quot;noreferrer&quot; title=&quot;Searching in metric spaces&quot;&gt;Searching in metric spaces&lt;/a&gt; survey, but all mentioned are suited for varying-radius queries. Many related SO answers advise &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;noreferrer&quot;&gt;Locality-sensitive hashing (LSH)&lt;/a&gt; as a current trend, but I didn't find any clear information whether it exploits the advantage of fixed-range and whether it is usable for non-Euclidean metrics. Any advices/references?&lt;/p&gt;&#xA;" OwnerUserId="28809" LastActivityDate="2017-02-16T12:18:27.260" Title="Fixed-radius range search in non-Euclidean space" Tags="|data-mining|clustering|bigdata|algorithms|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17017" PostTypeId="1" AcceptedAnswerId="17019" CreationDate="2017-02-16T14:49:37.093" Score="2" ViewCount="197" Body="&lt;p&gt;k&lt;a href=&quot;https://i.stack.imgur.com/ejuQZ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ejuQZ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training a kNN classifier with 144 features and graphed the accuracy vs number of features used and got this. What might be the reason for the drops in the accuracy at some points of the graph? I am using accelerometer-gyroscope-magnetometer fusion to recognize human activities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The one presented is validation accuracy. Should I use training accuracy instead? And why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ranked the features using ReliefF feature selection algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used time domain features such as mean, standard deviation, rms, median, variance, iqr, mad, zcr and mcr, and frequency domain features such as skewness, kurtosis and pca&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the top 8 features chosen. Peak accuracy occurs at 8 features.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/3TzaO.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3TzaO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26522" LastEditorUserId="26522" LastEditDate="2017-02-16T16:08:50.293" LastActivityDate="2017-02-16T19:51:29.280" Title="Feature selection where adding features are deteriorating model" Tags="|machine-learning|feature-selection|matlab|overfitting|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17018" PostTypeId="2" ParentId="16869" CreationDate="2017-02-16T14:57:58.557" Score="0" Body="&lt;p&gt;Posting details from the link in comments above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The data can be partitioned among multiple processors (or streams or threads). Each processor can read the previous iteration's cluster centers and assign the rows on the processor to clusters. Each processor then calculates new centers for its of data. Each actual cluster center (for the data across all processors) is then the weighted average of the centers on each processor.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In other words, the rows of data do not need to be combined globally.&#xA;  They can be combined locally, with the reduced set of results combined&#xA;  across all processors. In fact, MapReduce even contains a &quot;combine&quot;&#xA;  method for just this type of algorithm.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;You postulate a shared file that has the centroids as calculated for&#xA;  each processor.  This file contains:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The iteration number.  &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The cluster id.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The cluster coordinates.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The    number of rows assigned to the cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This is the centroid file.&lt;/strong&gt;  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;An iteration through the algorithm is going to add another set of&lt;br&gt;&#xA;   rows to this file. This information is the only information that&lt;br&gt;&#xA;   needs to be communicated globally.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There are two ways to do this in the MapReduce framework. The first&#xA;  uses map, combine, and reduce. The second only uses map and reduce.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.data-miners.com/2008/02/mapreduce-and-k-means-clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link with all details&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2017-02-16T14:57:58.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17019" PostTypeId="2" ParentId="17017" CreationDate="2017-02-16T16:32:51.053" Score="0" Body="&lt;p&gt;I guess the measurements from accelerometer-gyroscope-magnetometer are noisy and redundant in some sense. This means you can find some sort of correlation among the values of the measurements, for instance a correlation between the values from the accelerometer and the gyroscope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA captures the principal directions of variation on your data, removing the correlation between the measurements and also reducing the noise, therefore increasing the accuracy. From the graph it can be seen that the accuracy just diminishes a little when using all the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other factor I will consider is the magnitude of the features, a feature with a very large magnitude affects the behavior of K-NN.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-02-16T19:51:29.280" LastActivityDate="2017-02-16T19:51:29.280" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="17020" PostTypeId="2" ParentId="16918" CreationDate="2017-02-16T17:19:37.393" Score="1" Body="&lt;p&gt;By increasing number of features in convolutional layers, you are actually increasing the number of learnable parameters which were initialized to random values. So, randomness increases and that's why the cross entropy loss increases.Hope this is satisfiable justification to your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex, Suppose convolution layer with : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;filter_size = (3, 3) and number_feature = 10 =&amp;gt; number_parameters = 3*3*10 = 90&#xA;filter_size = (3, 3) and number_feature = 64 =&amp;gt; number_parameters = 3*3*64 = 576&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2017-02-16T17:19:37.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17021" PostTypeId="1" CreationDate="2017-02-16T18:16:56.650" Score="5" ViewCount="2755" Body="&lt;p&gt;I'm working predicting a value $y \in \mathbb{R}$ from the value of $x_{n+1}$, where $n$ is the number of samples ($x_{i \in [1,n]}$) used for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each training sample $x_{i}$ is a time series of variable lengths. How can I engineer features to predict $y$ while making the most of the samples available?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nota bene:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'm working with Python &amp;amp; Scikit-learn&lt;/li&gt;&#xA;&lt;li&gt;the length of the time series may be considered as an explanatory variable&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help :)&lt;/p&gt;&#xA;" OwnerUserId="29073" LastEditorUserId="29073" LastEditDate="2017-02-16T18:28:52.803" LastActivityDate="2017-04-06T05:30:48.430" Title="How can I use variable length inputs to train a regression model?" Tags="|python|time-series|regression|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17024" PostTypeId="1" AcceptedAnswerId="17034" CreationDate="2017-02-16T19:35:30.860" Score="37" ViewCount="36743" Body="&lt;p&gt;I have a bit of self taught knowledge working with Machine Learning algorithms (the basic Random Forest and Linear Regression type stuff). I decided to branch out and begin learning RNN's with Keras. When looking at most of the examples, which usually involve stock predictions, I haven't been able to find any basic examples of multiple features being implemented other than 1 column being the feature date and the other being the output. Is there a key fundamental thing I'm missing or something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone has an example I would greatly appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="26800" LastEditorUserId="29169" LastEditDate="2019-06-09T17:18:53.340" LastActivityDate="2019-06-09T17:18:53.340" Title="RNN's with multiple features" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17026" PostTypeId="2" ParentId="16399" CreationDate="2017-02-16T20:54:45.667" Score="8" Body="&lt;p&gt;(&lt;em&gt;Disclaimer: I am the core maintainer of Kur.&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kur and Keras are both deep learning frameworks, but they serve different purposes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Keras builds on Theano and TensorFlow to provide a high-level API to the underlying tensor libraries. This means that you use Keras as part of your Python program, and you programmatically create your model like this (this is Keras' &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;noreferrer&quot;&gt;functional API&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras.layers as L&#xA;import keras.models as M&#xA;&#xA;my_input = L.Input(shape=(100,))&#xA;intermediate = L.Dense(10, activation='relu')(my_input)&#xA;my_output = L.Dense(1, activation='softmax')(intermediate)&#xA;&#xA;model = M.Model(input=my_input, output=my_output)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is great when you are developing deep learning networks that you want to &quot;embed&quot; in some larger Python program, or when you are developing a single model that you intend to use. Keras is also quite mature and has a great community supporting it. Keras currently offers the superior API for models that you will create and use entirely within Python.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Kur builds &lt;em&gt;on top of Keras&lt;/em&gt; to provide a descriptive framework for designing and running deep learning models; you don't write programs when using Kur. Instead of programming, you write &quot;Kurfiles&quot; which describe your model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model:&#xA;  - input: my_input&#xA;  - dense: 10&#xA;  - activation: relu&#xA;  - dense: 1&#xA;  - activation: softmax&#xA;  - output: my_output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's the same model as I just wrote in Keras. Kur's descriptive language makes quickly prototyping and testing your models. The advantage of the Kurfile is that you can tweak large or small pieces of your model without having to touch your codebase. And when &lt;a href=&quot;http://kurhub.com&quot; rel=&quot;noreferrer&quot;&gt;Kurhub&lt;/a&gt; is released, you'll have a Docker-like repository for sharing models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kur also takes care of a lot of details that you need to consider when writing deep learning systems. For example, Kur knows about several different datatypes (CSV, Python pickle, etc.), and it can load the data and infer shapes for you. Without Kur, you'd have to write your own data processing pipeline (which may be easy or hard, depending on your data). Once you write the Kurfile (in YAML or JSON), you can train with &lt;code&gt;kur train Kurfile&lt;/code&gt; or use it for inference with &lt;code&gt;kur evaluate Kurfile&lt;/code&gt;. This makes it very easy to use &lt;em&gt;as a tool&lt;/em&gt;, rather than as a Python API.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both Kur and Keras can use Theano or TensorFlow, both offer Python APIs, and both are easy to write models in. It really depends on how to plan to develop and deploy your system.&lt;/p&gt;&#xA;" OwnerUserId="29083" LastActivityDate="2017-02-16T20:54:45.667" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17027" PostTypeId="1" AcceptedAnswerId="17035" CreationDate="2017-02-16T23:43:37.707" Score="1" ViewCount="200" Body="&lt;p&gt;I am analyzing a population movement pattern and I would like to design a visualization like this one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f2Uxp.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f2Uxp.jpg&quot; alt=&quot;Desired sample data visualization.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any &lt;em&gt;tool&lt;/em&gt; or &lt;em&gt;language&lt;/em&gt; (preferably R) I can use for that purpose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.dropbox.com/s/gidutndki3oepl1/Graphs_SK.xlsx?dl=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a sample of my dataset.&lt;/p&gt;&#xA;" OwnerUserId="29090" LastEditorUserId="8878" LastEditDate="2017-02-17T09:03:17.330" LastActivityDate="2017-02-18T01:05:31.360" Title="Visualize changes other time" Tags="|visualization|tools|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17028" PostTypeId="2" ParentId="17027" CreationDate="2017-02-17T01:01:21.010" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Gantt_chart&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gantt chart&lt;/a&gt; is what you need. There're numbers of visualization libraries which supports gantt chart. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Checkout this post: &lt;a href=&quot;https://stackoverflow.com/questions/3550341/gantt-charts-with-r&quot;&gt;https://stackoverflow.com/questions/3550341/gantt-charts-with-r&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-17T01:01:21.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17029" PostTypeId="2" ParentId="15885" CreationDate="2017-02-17T05:16:41.810" Score="1" Body="&lt;p&gt;For learning kernel/filter matrix in convolution layer, we find partial derivative of loss w.r.t. filter matrix and use gradient descent method to update filters.&#xA;$$ W = W - \alpha\frac{\partial L}{\partial W} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convolutional Neural Networks also use back-propagation algorithm to find partial derivatives of loss w.r.t. filter matrix.&lt;/p&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2017-02-17T05:16:41.810" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17031" PostTypeId="1" AcceptedAnswerId="17078" CreationDate="2017-02-17T12:30:03.880" Score="9" ViewCount="357" Body="&lt;p&gt;I am currently working on recreating the results of this &lt;a href=&quot;https://www.microsoft.com/en-us/research/publication/convolutional-neural-networks-for-speech-recognition-2/&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;. &#xA;In the paper they  describe a method for using CNN for features extraction, and have a acoustic model that is Dnn-hmm and pretrained using RBM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Section III subsection A states different ways the input data can be represented. I decided to vertically stack the spectrum plots of the static, delta and delta deltas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as such: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/mjFp8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mjFp8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper then describes how the network should be.  They state that they use a convolutional network, but nothing about the structure of network?. &#xA;Further more is the network always refered to as a convolutional ply? which i am sure i see any difference in  compared to an ordinary network convolutional neural network (cnn).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper states this regarding the difference: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(from section III subsection B)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A convolution ply differs from a standard, fully connected hidden&#xA;  layer in two important aspects, however. First, each convolutional&#xA;  unit receives input only from a local area of the input. This means&#xA;  that each unit represents some features of a local region of the&#xA;  input. Second, the units of the convolution ply can themselves be&#xA;  organized into a number of feature maps, where all units in the same&#xA;  feature map share the same weights but receive input from different&#xA;  locations of the lower layer&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Another thing i was wondering is whether the paper actually states how many output parameter is needed to feed the dnn-hmm acoustic model. I can't seem to decode the number of filters, filters sizes.. in general details of the network? &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:56.277" LastActivityDate="2021-02-16T02:52:56.277" Title="How does a convolutional ply differ from an ordinary convolutional network?" Tags="|neural-network|convolutional-neural-network|feature-extraction|audio-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17032" PostTypeId="1" CreationDate="2017-02-17T13:12:03.300" Score="1" ViewCount="120" Body="&lt;p&gt;I am building a model that is trying to predict the 5 year sales production of a salesperson in every zip code in the United States. I have only 7 years of data. This task is one piece of a larger model whose purpose is to determine the most optimal location (as determined by the zip code) to place a new salesperson.  My firm has approximately 20,000  sales people located in 10,000 zip codes. These 20,000 salespeople have customers in 35,000 different zip codes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have historic sales information and zip code demographics. My problem is I am trying to determine the “home zip code” to place a salesperson. Only 10,000 of the 43,000 zip codes are the “home zip code” to a salesperson. My company faces different constraints in every state, so our representation is not uniform, but we are in most states. I can build a machine learning model to predict the production of the salesperson in the 10,000 “home zips”, but how should I approach the other 33,000 zip codes. I’ve thought about building a similarity index of the zip codes based on demographics, sales history and regulatory constraints and building a unique model for each these groupings, but this seems like I’m adding a lot of uncertainty with the clustering/similarity grouping not to mention smaller data sets for learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is based on an annual contracts, i.e., the customer signs a one-year agreement on an annual basis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to best approach this problem would be appreciated&lt;/p&gt;&#xA;" OwnerUserId="29112" LastActivityDate="2017-02-17T13:12:03.300" Title="Predicting sales in unobserved locations" Tags="|machine-learning|python|pyspark|geospatial|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17033" PostTypeId="1" AcceptedAnswerId="17051" CreationDate="2017-02-17T13:14:50.697" Score="2" ViewCount="184" Body="&lt;p&gt;From Wang et al (2015) &quot;Visual Tracking with Fully Convolutional Networks&quot;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A top layer encodes more semantic features and serves as a category&#xA;  detector, while a lower layer carries more discriminative information&#xA;  and can better separate the target from distracters with similar&#xA;  features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't quite understand this statement: does this mean that the algorithm first identifies the semantic group to which the object belongs and then tries to distinguish among them? Also, are there any other publications where the difference in the roles of layers been demonstrated? &lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:44.850" LastActivityDate="2021-02-16T02:52:44.850" Title="Layers' function in a convolutional neural networks" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17034" PostTypeId="2" ParentId="17024" CreationDate="2017-02-17T14:22:52.930" Score="32" Body="&lt;p&gt;Recurrent neural networks (RNNs) are designed to learn sequence data. As you guess, they can definitely take multiple features as input! Keras' RNNs take 2D inputs (&lt;em&gt;T&lt;/em&gt;, &lt;em&gt;F&lt;/em&gt;) of timesteps &lt;em&gt;T&lt;/em&gt; and features &lt;em&gt;F&lt;/em&gt; (I'm ignoring the batch dimension here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you don't always need or want the intermediate timesteps, &lt;em&gt;t&lt;/em&gt; = 1, 2 ... (&lt;em&gt;T&lt;/em&gt; - 1). Therefore, Keras flexibly supports both modes. To have it output all &lt;em&gt;T&lt;/em&gt; timesteps, pass &lt;code&gt;return_sequences=True&lt;/code&gt; to your RNN (e.g., &lt;code&gt;LSTM&lt;/code&gt; or &lt;code&gt;GRU&lt;/code&gt;) at construction. If you only want the last timestep &lt;em&gt;t&lt;/em&gt; = &lt;em&gt;T&lt;/em&gt;, then use &lt;code&gt;return_sequences=False&lt;/code&gt; (this is the default if you don't pass &lt;code&gt;return_sequences&lt;/code&gt; to the constructor).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below are examples of both of these modes.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example 1: Learning the sequence&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Here's a quick example of training a LSTM (type of RNN) which keeps the entire sequence around. In this example, each input data point has 2 timesteps, each with 3 features; the output data has 2 timesteps (because &lt;code&gt;return_sequences=True&lt;/code&gt;), each with 4 data points (because that is the size I pass to &lt;code&gt;LSTM&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras.layers as L&#xA;import keras.models as M&#xA;&#xA;import numpy&#xA;&#xA;# The inputs to the model.&#xA;# We will create two data points, just for the example.&#xA;data_x = numpy.array([&#xA;    # Datapoint 1&#xA;    [&#xA;        # Input features at timestep 1&#xA;        [1, 2, 3],&#xA;        # Input features at timestep 2&#xA;        [4, 5, 6]&#xA;    ],&#xA;    # Datapoint 2&#xA;    [&#xA;        # Features at timestep 1&#xA;        [7, 8, 9],&#xA;        # Features at timestep 2&#xA;        [10, 11, 12]&#xA;    ]&#xA;])&#xA;&#xA;# The desired model outputs.&#xA;# We will create two data points, just for the example.&#xA;data_y = numpy.array([&#xA;    # Datapoint 1&#xA;    [&#xA;        # Target features at timestep 1&#xA;        [101, 102, 103, 104],&#xA;        # Target features at timestep 2&#xA;        [105, 106, 107, 108]&#xA;    ],&#xA;    # Datapoint 2&#xA;    [&#xA;        # Target features at timestep 1&#xA;        [201, 202, 203, 204],&#xA;        # Target features at timestep 2&#xA;        [205, 206, 207, 208]&#xA;    ]&#xA;])&#xA;&#xA;# Each input data point has 2 timesteps, each with 3 features.&#xA;# So the input shape (excluding batch_size) is (2, 3), which&#xA;# matches the shape of each data point in data_x above.&#xA;model_input = L.Input(shape=(2, 3))&#xA;&#xA;# This RNN will return timesteps with 4 features each.&#xA;# Because return_sequences=True, it will output 2 timesteps, each&#xA;# with 4 features. So the output shape (excluding batch size) is&#xA;# (2, 4), which matches the shape of each data point in data_y above.&#xA;model_output = L.LSTM(4, return_sequences=True)(model_input)&#xA;&#xA;# Create the model.&#xA;model = M.Model(input=model_input, output=model_output)&#xA;&#xA;# You need to pick appropriate loss/optimizers for your problem.&#xA;# I'm just using these to make the example compile.&#xA;model.compile('sgd', 'mean_squared_error')&#xA;&#xA;# Train&#xA;model.fit(data_x, data_y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Example 2: Learning the last timestep&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If, on the other hand, you want to train an LSTM which only outputs the last timestep in the sequence, then you need to set &lt;code&gt;return_sequences=False&lt;/code&gt; (or just remove it from the constructor entirely, since &lt;code&gt;False&lt;/code&gt; is the default). And then your output data (&lt;code&gt;data_y&lt;/code&gt; in the example above) needs to be rearranged, since you only need to supply the last timestep. So in this second example, each input data point still has 2 timesteps, each with 3 features. The output data, however, is just a single vector for each data point, because we have flattened everything down to a single timestep. Each of these output vectors still has 4 features, though (because that is the size I pass to &lt;code&gt;LSTM&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras.layers as L&#xA;import keras.models as M&#xA;&#xA;import numpy&#xA;&#xA;# The inputs to the model.&#xA;# We will create two data points, just for the example.&#xA;data_x = numpy.array([&#xA;    # Datapoint 1&#xA;    [&#xA;        # Input features at timestep 1&#xA;        [1, 2, 3],&#xA;        # Input features at timestep 2&#xA;        [4, 5, 6]&#xA;    ],&#xA;    # Datapoint 2&#xA;    [&#xA;        # Features at timestep 1&#xA;        [7, 8, 9],&#xA;        # Features at timestep 2&#xA;        [10, 11, 12]&#xA;    ]&#xA;])&#xA;&#xA;# The desired model outputs.&#xA;# We will create two data points, just for the example.&#xA;data_y = numpy.array([&#xA;    # Datapoint 1&#xA;    # Target features at timestep 2&#xA;    [105, 106, 107, 108],&#xA;    # Datapoint 2&#xA;    # Target features at timestep 2&#xA;    [205, 206, 207, 208]&#xA;])&#xA;&#xA;# Each input data point has 2 timesteps, each with 3 features.&#xA;# So the input shape (excluding batch_size) is (2, 3), which&#xA;# matches the shape of each data point in data_x above.&#xA;model_input = L.Input(shape=(2, 3))&#xA;&#xA;# This RNN will return timesteps with 4 features each.&#xA;# Because return_sequences=False, it will output 2 timesteps, each&#xA;# with 4 features. So the output shape (excluding batch size) is&#xA;# (2, 4), which matches the shape of each data point in data_y above.&#xA;model_output = L.LSTM(4, return_sequences=False)(model_input)&#xA;&#xA;# Create the model.&#xA;model = M.Model(input=model_input, output=model_output)&#xA;&#xA;# You need to pick appropriate loss/optimizers for your problem.&#xA;# I'm just using these to make the example compile.&#xA;model.compile('sgd', 'mean_squared_error')&#xA;&#xA;# Train&#xA;model.fit(data_x, data_y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29083" LastEditorUserId="58237" LastEditDate="2018-08-27T21:08:33.050" LastActivityDate="2018-08-27T21:08:33.050" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="17035" PostTypeId="2" ParentId="17027" CreationDate="2017-02-17T16:03:50.607" Score="1" Body="&lt;p&gt;Of course you can reproduce that graph with some &lt;code&gt;ggplot&lt;/code&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e0a4d.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e0a4d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code to create it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;library(scales)&#xA;&#xA;df &amp;lt;- read.csv2('Graphs_SK.csv')    &#xA;&#xA;p &amp;lt;- ggplot(data = df, aes(x = Governorate)) +&#xA;  geom_segment(aes(y = Mid.May.2015, yend = Dec.16, xend = Governorate),&#xA;               color = 'grey', size = 1) +&#xA;  geom_point(aes(y = Mid.May.2015, color = 'start'), size = 2) +&#xA;  geom_point(aes(y = Dec.16, color = 'end'), size = 2) +&#xA;  scale_colour_manual(name = '(Population in millions)', &#xA;                      labels = c('May 2015', 'Dec 2016'), &#xA;                      values = c('start' = 'steelblue4', 'end' = 'steelblue')) +&#xA;  scale_y_continuous(labels = comma, name = '', position = 'top') +&#xA;  coord_flip() +&#xA;  ggtitle('1 Year of something of Governorate') +&#xA;  guides(colour = guide_legend(title.position = 'right')) +&#xA;  theme(panel.background = element_rect(fill = 'white'),&#xA;        legend.position = c(-.15, 1.03),&#xA;        legend.direction = 'horizontal',&#xA;        legend.justification = c(0, 0),&#xA;        legend.key = element_rect(fill = NA),&#xA;        legend.title = element_text(size = 10),&#xA;        axis.text.y = element_text(hjust = 0),&#xA;        axis.ticks.y = element_line(size = NA), &#xA;        plot.title = element_text(hjust = .5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29086" LastEditorUserId="29086" LastEditDate="2017-02-18T01:05:31.360" LastActivityDate="2017-02-18T01:05:31.360" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17036" PostTypeId="1" CreationDate="2017-02-17T16:15:37.950" Score="3" ViewCount="358" Body="&lt;p&gt;I have a data set consisting of 100 features each of which are ternary: values of -1 if it exists in one category, 0 if it doesn't exist, and 1 if it exists  in the second category. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F1 F2 F3 ... F90 F91 F92 ... F99 F100&#xA;0  0  0  ... 1   -1  0   ... 0   -1&#xA;0  -1 0  ... -1   0  1   ... 0   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data is very sparse, ~20 of the 100 features have values of -1 or 1 for each row of data. I want to find similar rows of data through a heatmap visualization and dendrogram but I was confused on whether to use Euclidean distance or Cityblock distance. I'm quite new to data mining and while reading the &lt;code&gt;scipy&lt;/code&gt; pages, I found many distance measures which I have no idea means what. Is there a good distance measure for my type of data set?&lt;/p&gt;&#xA;" OwnerUserId="29116" LastActivityDate="2018-08-13T14:00:37.907" Title="Distance measure for ternary feature" Tags="|clustering|distance|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17037" PostTypeId="2" ParentId="17036" CreationDate="2017-02-17T17:37:05.733" Score="0" Body="&lt;p&gt;If you think of each row of your data as a vector a reasonable method for &quot;distance&quot; (similarity) would be &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine similarity&lt;/a&gt;. This is something that is commonly used to find similarity between user-user or item-item vectors in &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;collaborative filtering&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="25759" LastActivityDate="2017-02-17T17:37:05.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17038" PostTypeId="2" ParentId="17036" CreationDate="2017-02-17T17:48:18.687" Score="0" Body="&lt;p&gt;Frankly, I don't think it matters, whether you use city-block or any generalization of &lt;a href=&quot;https://en.wikipedia.org/wiki/Minkowski_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Minkowski distance&lt;/a&gt; in this case, as long as the metric don't give different expected values when you are calculating string/vector distances. You can use city-block distance as it is computationally faster than Euclidean if you had got many combinations to calculate.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-02-17T17:48:18.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17039" PostTypeId="2" ParentId="16984" CreationDate="2017-02-17T18:19:04.053" Score="2" Body="&lt;p&gt;As @oW_ mentioned, &lt;a href=&quot;http://orange.biolab.si&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange&lt;/a&gt; seems to be a good choice.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Data Source&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Orange3-spark provides Hive/Spark support, cool!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Interface&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A little laggy, but acceptable.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Algorithms&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Full algorithm supports (based on scikit-learn). I can implement deep learning stuff myself.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Customization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Highly supported. Widgets &amp;amp; addons can be &lt;a href=&quot;http://orange-development.readthedocs.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;developed&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Visualization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Not bad.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="28628" LastEditDate="2017-03-12T10:31:18.933" LastActivityDate="2017-03-12T10:31:18.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17040" PostTypeId="1" CreationDate="2017-02-17T21:10:32.637" Score="2" ViewCount="72" Body="&lt;p&gt;Consider the following problem, simplified:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that you have some data where you have only 5 columns/attributes.&lt;br&gt;&#xA;One of them can be seen as a &lt;strong&gt;reward&lt;/strong&gt;. How well we performed at a given run.&lt;br&gt;&#xA;The other two attributes, let's say x and y are our input to the system. A human hand picked those &lt;strong&gt;x&lt;/strong&gt;, &lt;strong&gt;y&lt;/strong&gt; values and they were recorded.&lt;br&gt;&#xA;Another two attributes are observations from sensors. These are not inputs and we have no control over them. Let's call them &lt;strong&gt;ob1&lt;/strong&gt; and &lt;strong&gt;ob2&lt;/strong&gt;.&lt;br&gt;&#xA;All are real values (nothing is discrete).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea is to look at this dataset as an ordered dataset.&lt;br&gt;&#xA;We are asked to &lt;strong&gt;choose input values x and y&lt;/strong&gt; that &lt;strong&gt;maximize our reward&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have available ~70.000 instances of this five-tupled dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;This is one approach that comes into mind but not sure if it is correct in principle or the simpler one.&lt;br&gt;&#xA;We could build a predictor which takes as input the four attributes (x, y, ob1, ob2) and has as target the reward.&lt;br&gt;&#xA;Then try a repetitive process for the inputs by using reinfocement learning (?) to get the inputs that maximize the reward?..&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-08-17T05:28:39.687" Title="How to choose inputs to maximize reward for an ordered dataset?" Tags="|predictive-modeling|model-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17041" PostTypeId="1" AcceptedAnswerId="17048" CreationDate="2017-02-17T22:22:49.100" Score="2" ViewCount="964" Body="&lt;p&gt;I recently had a phone interview with a consumer tech company for a quant position. The question was basically, &quot;imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded &quot;you decide&quot; - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?&lt;/p&gt;&#xA;" OwnerUserId="426" LastActivityDate="2017-02-18T20:43:57.537" Title="How to approach a Data Science case study question?" Tags="|statistics|beginner|ab-test|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17042" PostTypeId="2" ParentId="17040" CreationDate="2017-02-18T01:57:25.197" Score="1" Body="&lt;p&gt;It seems like a &lt;strong&gt;reinforcement learning&lt;/strong&gt; question with &lt;strong&gt;continuous action space&lt;/strong&gt;. &lt;code&gt;(ob1, ob2)&lt;/code&gt; is the observation, &lt;code&gt;(x, y)&lt;/code&gt; the action, and &lt;code&gt;reward&lt;/code&gt; the reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may refer to this &lt;a href=&quot;https://arxiv.org/abs/1509.02971&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; from &lt;a href=&quot;https://deepmind.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;DeepMind&lt;/a&gt;, which provides a general deep Q-learning way solving this type of question, including some physics tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as I don't have your data, I'm not sure if you have sufficient instances. ~70,000 instances with 4 features is (roughly) sufficient for general ML problems, but in reinforcement learning world, people use a simulation environment which is able to generate infinite samples, thus ~70,000 instances may be the bottleneck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you encounter such problem, you may try classic reinforcement learning (without &quot;deep&quot;) techniques.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-18T01:57:25.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17043" PostTypeId="1" CreationDate="2017-02-18T11:28:58.593" Score="3" ViewCount="1420" Body="&lt;pre&gt;&lt;code&gt;Error in XGBoost: Error in contrasts&amp;lt;-(tmp, value = contr.funs[1 + isOF[nn]]): &#xA;  contrasts can be applied only to factors with 2 or more levels&#xA;&#xA;param &amp;lt;- list(  objective           = &quot;binary:logistic&quot;, &#xA;                booster             = &quot;gbtree&quot;,&#xA;                eval_metric         = &quot;auc&quot;,&#xA;                eta                 = 0.01,&#xA;                max_depth           = 5,&#xA;                subsample           = 0.6,&#xA;                colsample_bytree    = 0.6&#xA;)&#xA;&#xA;&#xA;t1.y &amp;lt;- t1$target   &#xA;&#xA;t1 &amp;lt;- sparse.model.matrix(target ~ ., data = t1)&#xA;#Error occurs after the above command. If i change the above statement to (replacing . by -1 to drop collinear columns)&#xA;t1 &amp;lt;- sparse.model.matrix(target ~ -1, data = t1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No error occurs after changing to -1 but the dgcMatrix that is generated looks suspicious as it contains just one column and lot of rows. I was expecting the columns of the original matrix to be one-hot encoded rather than getting one columns as an output. If i proceed further ignoring the inspection of generated sparse matrix then I get an error after i run the xgboost below: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) : &#xA;  [05:14:55] amalgamation/../src/objective/regression_obj.cc:108: label must be in [0,1] for logistic regression&#xA;&#xA;&#xA;dtrain &amp;lt;- xgb.DMatrix(data=t1, label=t1.y)&#xA;watchlist &amp;lt;- list(train=dtrain)&#xA;bst1 &amp;lt;- xgboost(   params              = param, &#xA;                   data                = dtrain, &#xA;                   nrounds             = 1750,&#xA;                   #watchlist           = watchlist,&#xA;                   verbose             = 1,&#xA;                   maximize            = FALSE&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to convert the target column to both factor and numeric (converted it to 0 &amp;amp; 1, since as.numeric converts it to 1 &amp;amp; 2). But it did not help :'(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I convert the target to numeric then after running the Xgboost, I get the following error : &lt;code&gt;Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) : &#xA;  [05:20:12] amalgamation/../src/tree/updater_colmaker.cc:162: Check failed: (n) &amp;gt; (0) colsample_bytree=0.6 is too small that no feature can be included&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*t1 is a data.frame initially containing factor,numeric and integers which i am converting to sparse matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="28834" LastEditorUserId="28628" LastEditDate="2017-02-18T19:32:59.980" LastActivityDate="2017-02-18T19:32:59.980" Title="XgBoost error: contrasts can be applied only to factors with 2 or more levels" Tags="|r|xgboost|error-handling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17045" PostTypeId="1" CreationDate="2017-02-18T14:14:11.953" Score="1" ViewCount="24" Body="&lt;p&gt;The formula for Information given by a data of occurring with probability p is:&#xA;I=-log2 p &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This formula gives the bits if information needed to know the outcome of the event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This formula captures the intuition that the information needed to know the outcome of an event with probability 1 is 0 as we already know that outcome of the event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So shouldn't the formula give the information as 0 for the event with probability 0 as we know the outcome of the event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So shouldn't the graph of the I vs p be a symmetric as the values of p close to 1 will give similar information as the events with p close to 0?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastActivityDate="2017-02-18T14:14:11.953" Title="Antisymmetry of graph of Information vs probability" Tags="|categorical-data|probability|metadata|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17046" PostTypeId="2" ParentId="17041" CreationDate="2017-02-18T14:15:01.367" Score="0" Body="&lt;p&gt;Personally,  I don't think this question is reasonable. The first thing you need to do is determine from the stakeholders what &quot;success&quot; is. This could be increase traffic, increase revenue, etc. Without knowing how the stakeholders view success you can find all kind of interesting things in whatever data you have and never satisfy your clients. It is very common for data scientist to look for a needle in the haystack only to find the wrong needle if they can even find one.&lt;/p&gt;&#xA;" OwnerUserId="17959" LastActivityDate="2017-02-18T14:15:01.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17047" PostTypeId="1" AcceptedAnswerId="17053" CreationDate="2017-02-18T14:30:31.000" Score="7" ViewCount="374" Body="&lt;p&gt;In the section about batch normalization of Deep Learning book by Ian Goodfellow (&lt;a href=&quot;http://www.deeplearningbook.org/contents/optimization.html&quot; rel=&quot;noreferrer&quot;&gt;chapter link&lt;/a&gt;) there is the follwing text:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As  example, suppose we have a deep neural network that has only one&#xA;  unit per layerand does not use an activation function at each hidden&#xA;  layer: $y=x w_1 w_2 w_3 \ldots w_l$. Here, $w_i$ provides the weight used by&#xA;  layer $i$. The output of layer $i$ is  $h_i=h_{i−1} wi$. The output $y$ is a&#xA;  linear function of the input $x$, but a nonlinear function of the weights&#xA;  $w_i$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why y is nonlinear with respect to w_i?&lt;/p&gt;&#xA;" OwnerUserId="29137" LastEditorUserId="27250" LastEditDate="2017-02-19T07:37:32.390" LastActivityDate="2017-12-24T19:20:11.937" Title="Question about the simple example for batch normalization given in &quot;deep learning&quot; book" Tags="|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17048" PostTypeId="2" ParentId="17041" CreationDate="2017-02-18T15:17:22.043" Score="2" Body="&lt;p&gt;This question (something I've asked variants of several times in interviews) has absolutely nothing to do with statistical or other quantitative procedures. What is being asked here is for an understanding of the overall data mining process itself.  The first thing to determine is what the definition of success. So you have to ask. The stakeholder usually will not volunteer this unless asked anyway. Then, depending on the answer describe the overall process for data mining based on this end goal.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-02-18T15:17:22.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17049" PostTypeId="1" AcceptedAnswerId="17063" CreationDate="2017-02-18T15:52:10.020" Score="1" ViewCount="95" Body="&lt;p&gt;I have few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a website to upload huge research dataset (over 100GB) for free?&lt;/li&gt;&#xA;&lt;li&gt;Which type of compression (rar, zip ... etc) is good for jpeg images?&lt;/li&gt;&#xA;&lt;li&gt;In case of dataset of 120GB. what is the best split for this big files (eg: 20 GB each)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29138" LastActivityDate="2017-02-19T21:23:50.170" Title="Uploading huge dataset" Tags="|dataset|bigdata|data|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17051" PostTypeId="2" ParentId="17033" CreationDate="2017-02-18T17:12:50.903" Score="1" Body="&lt;p&gt;This is a basic property of convolutional networks. First layers identify simple features and as you go deeper in the CNN each layer uses the features of the previous layer to build more complex ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say the state-of-the-art article about this is &lt;strong&gt;Visualizing and understanding Convolutional Networks&lt;/strong&gt; by &lt;em&gt;Matthew D. Zeiler and Rob Fergus&lt;/em&gt;.&#xA;You can download it &lt;a href=&quot;http://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-02-18T17:12:50.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17052" PostTypeId="1" CreationDate="2017-02-18T17:42:50.980" Score="2" ViewCount="6225" Body="&lt;p&gt;I'm trying to predict what service a customer wants when he comes to our office from his previous transactions history. I have 7 years transactions data(3 crore txns) and good amount of customers are frequent ones.Each service is personal to each customers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;    {&#xA;        &quot;customerId&quot;:&quot;1xxxx&quot;,&#xA;        &quot;txns&quot;:[&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;12ds23&quot;,&#xA;                &quot;date&quot;:&quot;2016-08-03T08:43:33Z&quot;&#xA;            },&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;1dsd89&quot;,&#xA;                &quot;date&quot;:&quot;2016-09-03T08:43:33Z&quot;&#xA;            },&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;1dbbb89&quot;,&#xA;                &quot;date&quot;:&quot;2016-10-03T09:43:33Z&quot;&#xA;            }&#xA;        ]&#xA;&#xA;    },{&#xA;        &quot;customerId&quot;:&quot;2xxxx&quot;,&#xA;        &quot;txns&quot;:[&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;dds2dfsd&quot;,&#xA;                &quot;date&quot;:&quot;2016-08-03T08:43:33Z&quot;&#xA;            },&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;dsdsdsdf&quot;,&#xA;                &quot;date&quot;:&quot;2016-09-03T08:43:33Z&quot;&#xA;            },&#xA;            {&#xA;                &quot;serviceId&quot;:&quot;sdfbb9&quot;,&#xA;                &quot;date&quot;:&quot;2017-10-03T09:43:33Z&quot;&#xA;            },&#xA;            ...&#xA;            ...&#xA;        ]&#xA;&#xA;    },&#xA;    ...&#xA;    ..&#xA;    ..&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone please advice which Machine learning technique or statistical approach would be best in this case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a Decision tree classification/Logistic regression model taking date(month,day,day of week) as features for predicting the service he wants as Class labels&lt;/p&gt;&#xA;" OwnerUserId="18665" LastEditorUserId="13727" LastEditDate="2017-02-19T12:45:20.103" LastActivityDate="2017-02-19T12:45:20.103" Title="Predict customer action from previous buying history" Tags="|machine-learning|recommender-system|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17053" PostTypeId="2" ParentId="17047" CreationDate="2017-02-18T18:26:54.413" Score="2" Body="&lt;p&gt;I think what the statement meant was when given weights $w_1,...,w_n$ are fixed, output is linear proportional to $x$, but as it mentions&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;nonlinear function of the weights w_i&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Given a set of weights (more than one being varied), they do not linearly add to produce an output like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=w_1x_1+...+w_nx_n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but rather non-linearly like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=w_1w_2..w_n*x$ (each $w_i$ is a dimension in the hyperspace)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I think it would become more clearer from the statement &quot;&lt;em&gt;Output is linear to any weight $w_i$ but non-linear to weights $w_i$&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-02-18T18:26:54.413" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17054" PostTypeId="2" ParentId="17041" CreationDate="2017-02-18T20:43:57.537" Score="0" Body="&lt;p&gt;I believe open ended questions like this one have the goal to see what is your thought process. The interviewer wants to know how would you tackle the problem, what do you do first, what hypothesis do you consider, and, most important, how do you defend your decisions. Asking questions and, sometimes, even thinking out loud can be helpful as you show the interviewer your thought process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would probably employ a similar approach and assume that this feature was rolled out for a subset of users. Then I would check metrics such as user engagement (here you can define that as number of comments, number of liked comments, number of comment replies, or a function of all of those) and perform an A/B test. Based on the results a conclusion can be reached if this new feature is indeed a success (based on the aforementioned metrics) or not. &lt;/p&gt;&#xA;" OwnerUserId="8792" LastActivityDate="2017-02-18T20:43:57.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17055" PostTypeId="1" AcceptedAnswerId="17058" CreationDate="2017-02-19T05:52:07.960" Score="1" ViewCount="6701" Body="&lt;p&gt;I have a data frame that contains random values between 0-60 (inclusive 0 &amp;amp; 60). These values denote months. I want to bin the data into three categories&#xA;(x&amp;lt;=6, 6&amp;lt; x &amp;lt;=12, x&gt;12) and generate a new single columns which will be a factor containing 3 values (0,1,2) denoting the respective bins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to generate 3 columns one-hot encoded style but I am unable to think of a way to generate single column having 3 factors.&lt;/p&gt;&#xA;" OwnerUserId="28834" LastActivityDate="2017-02-19T12:15:33.110" Title="Binning data in one of the columns of a dataframe(Using R)" Tags="|r|dataset|data-cleaning|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17056" PostTypeId="2" ParentId="17052" CreationDate="2017-02-19T09:14:42.347" Score="2" Body="&lt;p&gt;You can have a variety of solutions, starting from very simple to a more complex and beneficial ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you'll start with the simple solutions, jain much of the benefit and continue by need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that though the problem can be treated as a supervised learning problem, it is a multi label problem (a customer can be interested in many services). Therefore, decision tree or logistic regression is not suitable (unless you have a handful of services and you try to predict if the user would like a service, for each service).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classical method to provide such recommendations are &quot;People that like X, also liked Y&quot;. Though very simple, at time 50% of Amazon's revenues where due to that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should make the recommandation more sepcific by considering the lift - Harri Potter is very popular and many people like it. However, it isn't the most sutabile recommandation for someone reading browsing books.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take such a item-to-item recommendation and use all user history by aggregating using &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; or similar methods.&#xA;You can take care of temporality and dates by giving more eight to recent data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next step is moving to &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;recommendation system&lt;/a&gt;. You can find on the web some implementation. &#xA;The winner in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Netflix_Prize&quot; rel=&quot;nofollow noreferrer&quot;&gt;Netflix challenge&lt;/a&gt; used &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_decomposition&quot; rel=&quot;nofollow noreferrer&quot;&gt;matrix factorization&lt;/a&gt; algorithm, so this direction is very popular for recommendation systems. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-02-19T09:14:42.347" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="17057" PostTypeId="1" AcceptedAnswerId="17059" CreationDate="2017-02-19T10:30:31.107" Score="2" ViewCount="2887" Body="&lt;p&gt;I am using Keras with Theanos backend in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2117 samples and each sample has an individual target (on purpose) ie. 2117 outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As opposed to categories, the targets are ratings eg. (16.4714494876, 17.4129353234, 17.4476570289) the entirety of the number is important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having problems/don't know where to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) When i run the NN it only outputs the targets as whole integers as opposed to the format of the actual values. eg. 16 instead of 16.xxxxxx&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Presumably i will only be able to gauge the accuracy of predictions based on how close the output is to the target since there are so many targets, does this type of classification problem have a name that i can research?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) In 3 research papers i have read that apply NN to my specific classification problem they list the output layer as only having 1 neuron but provide no further explanation, how could this be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# fix random seed for reproducibility&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;&#xA;X = np.array(df[FEATURES].values)&#xA;Y = np.array(df[&quot;MTPS&quot;].values)&#xA;&#xA;# define baseline model&#xA;def baseline_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))&#xA;    model.add(Dense(2117, init='normal', activation='softmax'))&#xA;    # Compile model&#xA;    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;    return model&#xA;#build model&#xA;estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)&#xA;&#xA;#cross validation&#xA;X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)&#xA;estimator.fit(X_train, Y_train)&#xA;&#xA;#print class predictions&#xA;print estimator.predict(X_test)&#xA;print Y_test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help.&lt;/p&gt;&#xA;" OwnerUserId="27962" LastActivityDate="2020-08-11T16:13:07.613" Title="Python Keras NN, handling n_samples of float outputs" Tags="|python|neural-network|classification|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17058" PostTypeId="2" ParentId="17055" CreationDate="2017-02-19T12:15:33.110" Score="3" Body="&lt;p&gt;Use &lt;code&gt;cut&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df = data.frame(v=sample(1:60,1000,TRUE))&#xA;&amp;gt; df$cat = cut(df$v,c(-Inf,6,12,Inf))&#xA;&amp;gt; table(df$cat)&#xA;&#xA; (-Inf,6]    (6,12] (12, Inf] &#xA;       97        92       811 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, simple R questions are better asked on StackOverflow.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-02-19T12:15:33.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17059" PostTypeId="2" ParentId="17057" CreationDate="2017-02-19T12:39:30.070" Score="2" Body="&lt;p&gt;I'm confused. As you have said, the targets are ratings. It's definitely a regression problem to me, instead of a classification problem.&lt;/p&gt;&#xA;&lt;p&gt;There's several problems in your code:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For regression problem, we usually use &lt;code&gt;linear&lt;/code&gt; as activation function of the last layer (sometimes &lt;code&gt;relu&lt;/code&gt;, even &lt;code&gt;sigmoid&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;And also we use &lt;code&gt;mse&lt;/code&gt; as metric(sometimes &lt;code&gt;mae&lt;/code&gt;, &lt;code&gt;msle&lt;/code&gt;, etc). &lt;code&gt;categorical_crossentropy&lt;/code&gt; is used for classification problem, and &lt;code&gt;sparse_categorical_crossentropy&lt;/code&gt; is used for sparse input classification problem. &lt;a href=&quot;https://keras.io/objectives/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ref&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;KerasClassifier&lt;/code&gt; is used for classification problem, use &lt;code&gt;KerasRegressor&lt;/code&gt; instead.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;metrics=['accuracy']&lt;/code&gt; is used for classification problem, and it's meaningless in regression problem. &lt;a href=&quot;https://github.com/fchollet/keras/issues/4308&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ref&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Assuming &lt;code&gt;df&lt;/code&gt; is a Pandas DataFrame, then &lt;code&gt;df.values&lt;/code&gt; is naturally a &lt;code&gt;ndarrary&lt;/code&gt;, there's no need to cast &lt;code&gt;np.array&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Now answering your question:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As I've never used &lt;a href=&quot;https://keras.io/scikit-learn-api/&quot; rel=&quot;nofollow noreferrer&quot;&gt;wrappers for scikit-learn AP&lt;/a&gt;I, I'm not very sure why you have integer output, my best guess is because of &lt;code&gt;KerasClassifier&lt;/code&gt; and &lt;code&gt;.predict&lt;/code&gt;(in scikit-learn APIs, &lt;code&gt;.predict&lt;/code&gt; returns integer, basically the predicted class and &lt;code&gt;.predict_proba&lt;/code&gt; returns float, indicating the probability of each class). Try to use &lt;code&gt;.predict_proba&lt;/code&gt;, it would help. BTW, you should really use &lt;code&gt;KerasRegressor&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;As you have mentioned, there's so many classes to predict. In fact, it's a regression problem.&lt;/li&gt;&#xA;&lt;li&gt;Could you please add the links of these 3 papers? Neural networks having only 1 neuron in output layer seem to be a regression NN to me. Regression NN usually uses &lt;code&gt;Dense(1, activation='linear')&lt;/code&gt; as the output layer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Here's my version of your code, it might work:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# fix random seed for reproducibility&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;&#xA;X = df[FEATURES].values  # no need to cast np.array&#xA;Y = df[&amp;quot;MTPS&amp;quot;].values&#xA;&#xA;# define baseline model&#xA;def baseline_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))&#xA;    model.add(Dense(1, init='normal', activation='linear'))  # one neuron, linear activation&#xA;    # Compile model&#xA;    model.compile(loss='mse', optimizer='adam')  # mse loss&#xA;    return model&#xA;#build model&#xA;estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)  # KerasRegressor for regression problem&#xA;&#xA;#cross validation&#xA;X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)&#xA;estimator.fit(X_train, Y_train)&#xA;&#xA;#print class predictions&#xA;print estimator.predict(X_test)&#xA;print Y_test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h2&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/wrappers/scikit_learn.py#L174&quot; rel=&quot;nofollow noreferrer&quot;&gt;References&lt;/a&gt;&lt;/h2&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="98307" LastEditDate="2020-08-11T16:13:07.613" LastActivityDate="2020-08-11T16:13:07.613" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17060" PostTypeId="1" CreationDate="2017-02-19T18:25:26.177" Score="1" ViewCount="520" Body="&lt;p&gt;I am newbie to whole tensorflow and keras. I am trying to implement icGAN in which I feed a conditional vector of length 'ny' after first convolution(I have added network architecture for better insight) . I am having trouble merging the output of first convolution with the condition vector. I tried using reshape but it gave me weird errors. Below is my code snippet. The below code does not give error but I want to know is how should I change the y which will be of shape (batch_dim, 100) to (batchdim, 32,32,100).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/L2L5a.jpg&quot; alt=&quot;Network Architecture&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    x = Input(shape=img_dim, name=&quot;discriminator_input&quot;)&#xA;    y = Input(shape=cat_dim, name=&quot;category_input&quot;)&#xA;    list_f = [128, 256,512]&#xA;    # First conv&#xA;    x = Convolution2D(64, 3, 3, subsample=(2, 2),     name=&quot;disc_convolution2d_1&quot;, border_mode=&quot;same&quot;)(disc_input)&#xA;    x = LeakyReLU(0.2)(x)&#xA;&#xA;    #How to reshape the cat_input. &#xA;&#xA;    # Next convs&#xA;    for i, f in enumerate(list_f):&#xA;        name = &quot;disc_convolution2d_%s&quot; % (i + 2)&#xA;        x = Convolution2D(f, 3, 3, subsample=(2, 2), name=name, border_mode=&quot;same&quot;)(x)&#xA;        x = BatchNormalization(mode=bn_mode, axis=bn_axis)(x)&#xA;        x = LeakyReLU(0.2)(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="29082" LastEditorUserId="29082" LastEditDate="2017-02-19T22:03:10.967" LastActivityDate="2017-02-20T13:48:07.427" Title="Unable to perform Keras Reshape to an input to match convolution output" Tags="|python|tensorflow|keras|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17061" PostTypeId="1" CreationDate="2017-02-19T19:30:56.810" Score="5" ViewCount="1255" Body="&lt;p&gt;I am using tensorflow to train two instances of the same neural network with two different datasets. the network itself is quite simple with an input and output layer and 6 hidden layers (each layer is a 20 meurons followed by a non-linear activation function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can train the network with two different datasets and that is fine. Now, what i want to do is basically create a new network which is a combination of these two trained networks. In particular, I want the input and the first 3 layers to be from one of the trained network and the last 3 layers and the output layer to be from the other network. I am very new to tensorflow and have not found a way to do this. Can someone point me to the API or some way to do this sort of hybrid networks?&lt;/p&gt;&#xA;" OwnerUserId="8337" LastActivityDate="2020-08-13T16:26:25.367" Title="combining trained neural nets in tensorflow" Tags="|deep-learning|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17062" PostTypeId="2" ParentId="16649" CreationDate="2017-02-19T20:40:46.827" Score="2" Body="&lt;p&gt;Instead of  (or additionally to) histogram you may check the density plot of the distribution of the intervals between the &#xA;events. In the &lt;code&gt;r&lt;/code&gt; snipped below the process id identified with  &lt;code&gt;ID&lt;/code&gt; and the interval length is &lt;code&gt;INT_LEN&lt;/code&gt; .&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;densityplot(~INT_LEN,  groups = ID,&#xA;data =  df,    &#xA;scales=list(x=list(rot=90, cex= .9),y=list(cex=.9)),par.strip.text=list(cex=.8), plot.points = F,  &#xA;ylab=&quot;density&quot;, xlab=&quot;Interval between Events&quot;, main=paste(&quot;Interval Length Density per Process&quot;  ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f89S4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f89S4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This show the distribution (in my simple case there are two different distribution). One group of processes has a normal distribution of event with a mean of 1,2. So you may argue that there are 50 events per minute on average.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this may not be true as the time sequence is lost in this view.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the time series information I select a basic interval length, where the distribution of the event doesn't play a role any more.&#xA;I use a minute, but if this is to coarse go down. The point is there is no difference to get all events at the beginning of the interval (minute)&#xA;or uniformly distribute within the interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This selection allows to aggregate the data on the interval level and present it as a time series. The graph below shows the count of event per minute.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xy &amp;lt;- xyplot(CNT ~ MINUTE_NO , groups = ID,   &#xA;data =  df ,    type=c(&quot;o&quot;,&quot;g&quot;),&#xA;scales=list(rot=90, x=list( cex= .9 ),y=list(rot=0, cex=.8, alternating = c(1,1), tck=c(1,0), relation = &quot;free&quot;)),par.strip.text=list(cex=.8), &#xA;ylab=&quot;event count per minute&quot;, xlab=&quot;time&quot;, main=paste(&quot;Time Series - Event Count per Minute&quot;  ) )&#xA;print (xy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7D0Ht.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7D0Ht.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can spot a possible trend in the distribution of the events.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2017-02-20T20:13:08.140" LastActivityDate="2017-02-20T20:13:08.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17063" PostTypeId="2" ParentId="17049" CreationDate="2017-02-19T21:23:50.170" Score="0" Body="&lt;p&gt;Don't compress files that are already compressed (like how JPEG/PNG images or video files are). Their inherent compression is usually good enough, and you only trade some 5% in lower compressed size for a much lengthier and often non-seekable decompression that results in using twice the disk space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need to batch the files together, just use &lt;a href=&quot;https://en.wikipedia.org/wiki/Tar_%28computing%29&quot; rel=&quot;nofollow noreferrer&quot;&gt;tar&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-02-19T21:23:50.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17064" PostTypeId="1" AcceptedAnswerId="17065" CreationDate="2017-02-20T00:23:15.170" Score="10" ViewCount="15701" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/pdf/1409.1556.pdf&quot; rel=&quot;noreferrer&quot;&gt;this highly cited paper&lt;/a&gt;, authors give the following discussion on the number of weight parameters. I am not very clear why it has $49C^2$ parameters. I think it should be $49C$ since each of $C$ input channels shares the same filter, which has $49$ parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EZ3m0.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EZ3m0.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21767" LastEditorUserId="28628" LastEditDate="2017-02-20T22:59:32.180" LastActivityDate="2017-02-21T10:11:54.997" Title="number of parameters for convolution layers" Tags="|machine-learning|deep-learning|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17065" PostTypeId="2" ParentId="17064" CreationDate="2017-02-20T03:01:16.110" Score="18" Body="&lt;p&gt;Actually it's $49C*C$, the first $C$ is the number of input channels, and the second $C$ is the number of filters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quote from &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/#conv&quot; rel=&quot;noreferrer&quot;&gt;CS231n&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To summarize, the Conv Layer:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Accepts a volume of size $W_1 \times H_1 \times D_1$&lt;/li&gt;&#xA;  &lt;li&gt;Requires four hyperparameters:&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Number of filters $K$,&lt;/li&gt;&#xA;  &lt;li&gt;their spatial extent $F$,&lt;/li&gt;&#xA;  &lt;li&gt;the stride $S$,&lt;/li&gt;&#xA;  &lt;li&gt;the amount of zero padding $P$.&lt;/li&gt;&#xA;  &lt;/ul&gt;&lt;/li&gt;&#xA;  &lt;li&gt;Produces a volume of size $W_2 \times H_2 \times D_2$ where:&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;$W_2 = (W_1 - F + 2P)/S + 1$&lt;/li&gt;&#xA;  &lt;li&gt;$H_2 = (H_1 - F + 2P)/S + 1$ (i.e. width and height are computed equally by symmetry)&lt;/li&gt;&#xA;  &lt;li&gt;$D_2 = K$&lt;/li&gt;&#xA;  &lt;/ul&gt;&lt;/li&gt;&#xA;  &lt;li&gt;With parameter sharing, it introduces $F \cdot F \cdot D_1$ weights per filter, for a total of $(F \cdot F \cdot D_1) \cdot K$ weights and $K$ biases.&lt;/li&gt;&#xA;  &lt;li&gt;In the output volume, the $d$-th depth slice (of size $W_2 \times H_2$) is the result of performing a valid convolution of the $d$-th filter over the input volume with a stride of $S$, and then offset by $d$-th bias.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;A common setting of the hyperparameters is $F = 3, S = 1, P = 1$. However, there are common conventions and rules of thumb that motivate these hyperparameters. See the ConvNet architectures section below.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="28628" LastEditDate="2017-02-21T10:11:54.997" LastActivityDate="2017-02-21T10:11:54.997" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17066" PostTypeId="1" AcceptedAnswerId="17068" CreationDate="2017-02-20T04:12:46.797" Score="0" ViewCount="204" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LYCbl.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LYCbl.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am attempting to load in the Iris.tab data set into the Orange load data widget. However, when I do this, I get the following error message (shown above).&lt;/p&gt;&#xA;&lt;p&gt;This keeps popping up when I start up Orange Data Mining and try to load the iris.tab data file. It used to work before, but now it is showing this error.&lt;/p&gt;&#xA;&lt;p&gt;Does anyone have any idea why this might be happening and what might be causing this error?&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2022-09-25T14:58:48.563" LastActivityDate="2022-09-25T14:58:48.563" Title="Error opening Iris.tab file in Orange load data widget" Tags="|orange|" AnswerCount="1" CommentCount="1" ClosedDate="2017-02-20T13:47:27.950" ContentLicense="CC BY-SA 4.0" />
  <row Id="17068" PostTypeId="2" ParentId="17066" CreationDate="2017-02-20T08:49:20.633" Score="5" Body="&lt;p&gt;This was an &lt;a href=&quot;https://github.com/biolab/orange3/issues/1997&quot; rel=&quot;noreferrer&quot;&gt;error&lt;/a&gt; in Orange 3.3.11, and fixed in release 3.3.12. Update to the newest release to get rid of the error.&lt;/p&gt;&#xA;" OwnerUserId="15711" LastActivityDate="2017-02-20T08:49:20.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17070" PostTypeId="1" CreationDate="2017-02-20T13:58:03.280" Score="2" ViewCount="216" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5Sz8t.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Sz8t.png&quot; alt=&quot;Error surface&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, can someone please explain why the descent happens in a direction perpendicular to the contour lines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2sj1g.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2sj1g.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29182" LastActivityDate="2017-10-17T20:57:36.620" Title="Why does a horizontal cross section of a squared error surface yield ellipsis?" Tags="|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17072" PostTypeId="1" AcceptedAnswerId="17082" CreationDate="2017-02-20T21:05:45.800" Score="3" ViewCount="1014" Body="&lt;p&gt;For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a person asks why did you choose this combination and algorithms!? You can reach different combinations and use other algorithms and they may yield better results. What should I answer? I cannot just say based on tests as there are many other algorithms that I haven't tested (cannot test all algorithms). It is not a logical response, I think. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for tests to justify my selected methods and combination and say why I have selected these methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know your suggestions.&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="29169" LastEditDate="2021-01-14T18:55:19.877" LastActivityDate="2021-01-14T18:55:19.877" Title="Which Outlier Detection Method? Why?" Tags="|statistics|anomaly-detection|outlier|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17074" PostTypeId="1" CreationDate="2017-02-20T22:48:48.653" Score="8" ViewCount="7180" Body="&lt;p&gt;In the tutorial &lt;a href=&quot;https://github.com/Microsoft/LightGBM/blob/master/R-package/demo/boost_from_prediction.R&quot; rel=&quot;nofollow noreferrer&quot;&gt;boosting from existing prediction&lt;/a&gt; in lightGBM R, there is a &lt;code&gt;init_score&lt;/code&gt; parameter in function &lt;code&gt;setinfo&lt;/code&gt;. I am wondering what  &lt;code&gt;init_score&lt;/code&gt; means? In the help page, it says:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;init_score&lt;/code&gt;: initial score is the base prediction lightgbm will boost&#xA;from&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Another thing is what does &lt;em&gt;&amp;quot;boost&amp;quot;&lt;/em&gt; mean in lightGBM?&lt;/p&gt;&#xA;" OwnerUserId="29204" LastEditorUserId="29169" LastEditDate="2021-09-17T19:45:51.350" LastActivityDate="2021-09-17T19:45:51.350" Title="What is init_score in lightGBM?" Tags="|gbm|self-study|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17076" PostTypeId="2" ParentId="17074" CreationDate="2017-02-21T01:54:40.040" Score="7" Body="&lt;p&gt;Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;init_score&lt;/code&gt; value represents the prediction that you are trying to correct with the first boosting iteration. By default this probably predicts the majority class for all examples or a randomly selected class, but you can input the prediction that was outputted by any other model here if you like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first tree that is learned by LightGBM will try to correct the errors of this initial prediction. In the example that you linked, they train a LightGBM model for one iteration, and use the score to continue training another LightGBM model. So in practice, what they have done in this example is kind of pointless because they could have just trained a single LightGBM from the start, but I suppose it is just to illustrate the concept and how it works.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-02-21T01:54:40.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17078" PostTypeId="2" ParentId="17031" CreationDate="2017-02-21T03:54:04.910" Score="4" Body="&lt;p&gt;It seems that a convolutional ply is exactly the same as an ordinary convolutional layer. From their paper, they argue that the term &quot;CNN layer&quot; usually refers to a convolutional layer followed by a pooling layer. In an attempt to reduce confusion, they name the convolutional part a &quot;convolution ply&quot; and a the pooling part a &quot;pooling ply&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In CNN terminology, a pair of convolution and pooling layers in Fig. 2 in succession is usually referred to as one CNN “layer.” A deep CNN thus consists of two or more of these pairs in succession. To avoid confusion, we will refer to convolution and pooling layers as convolution and pooling plies, respectively.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ironically, this has &lt;em&gt;increased&lt;/em&gt; the confusion, leading to this post. At the time I suppose it was not common to have several convolutional layers in a row before a pooling layer, but we see this often in modern architectures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your other question about the network structure; they state the structure of the network that they use in the Experiments section (Section V-B). To &lt;em&gt;hopefully&lt;/em&gt; reduce confusion, I've replaced the word &quot;ply&quot; with &quot;layer&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In these experiments we used one convolution [layer], one pooling [layer] and two fully connected hidden layers on the top. The fully connected layers had 1000 units in each. The convolution and pooling parameters were: pooling size of 6, shift size of 2, filter size of 8, 150 feature maps for FWS, and 80 feature maps per frequency band for LWS.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2017-02-21T04:08:47.213" LastActivityDate="2017-02-21T04:08:47.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17079" PostTypeId="1" AcceptedAnswerId="25008" CreationDate="2017-02-21T05:21:58.127" Score="14" ViewCount="12981" Body="&lt;p&gt;I have recently published a dataset (&lt;a href=&quot;https://arxiv.org/abs/1701.08380&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;) with 369 classes. I ran a couple of experiments on them to get a feeling for how difficult the classification task is. Usually, I like it if there are confusion matrices to see the type of error being made. However, a $369 \times 369$ matrix is not practical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to give the important information of big confusion matrices? For example, usually there are a lot of 0s which are not so interesting. Is it possible to sort the classes so that most non-zero entries are around the diagonal in order to allow showing multiple matrices which are part of the complete confusion matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is &lt;a href=&quot;https://github.com/MartinThoma/algorithms/tree/master/ML/confusion-matrix&quot; rel=&quot;noreferrer&quot;&gt;an example for a big confusion matrix&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Examples in the Wild&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Figure 6 of &lt;a href=&quot;https://arxiv.org/pdf/1702.05373.pdf&quot; rel=&quot;noreferrer&quot;&gt;EMNIST&lt;/a&gt; looks nice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NKEgU.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NKEgU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easy to see where many cases are. However, those are only $26$ classes. If the whole page was used instead of only one column this could probably be 3x as many, but that would still only be $3 \cdot 26 = 78$ classes. Not even close to 369 classes of HASY or 1000 of ImageNet.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;My similar question on &lt;a href=&quot;https://cs.stackexchange.com/q/70627/2914&quot;&gt;CS.stackexchange&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="-1" LastEditDate="2017-04-13T12:48:38.763" LastActivityDate="2020-07-14T08:36:34.707" Title="How can I make big confusion matrices easier to read?" Tags="|visualization|confusion-matrix|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17080" PostTypeId="1" CreationDate="2017-02-21T06:25:41.010" Score="6" ViewCount="3659" Body="&lt;p&gt;I am dealing with the Street View House Number recognition problem. I am trying to train a CNN with Keras. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how I prepared the input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import Image&#xA;from PIL import ImageFilter&#xA;train_folders = 'sv_train/train'&#xA;test_folders = 'test'&#xA;extra_folders = 'extra'&#xA;SV_IMG_SIZE = 28&#xA;SV_CHANNELS = 1&#xA;train_imsize = np.ndarray([len(train_data),2])&#xA;k = 500&#xA;sv_images = []&#xA;max_images = 20000#len(train_data)&#xA;max_digits = 5&#xA;sv_labels = np.ones([max_images, max_digits], dtype=int) * 10 # init to 10 cause it would be no digit&#xA;nboxes = [[] for i in range(max_images)]&#xA;print (&quot;%d to load&quot; % len(train_data))&#xA;def getBBox(i,perc):&#xA;    '''&#xA;    Given i, the desired i.png, returns&#xA;    x_min, y_min, x_max, y_max,&#xA;    the four numbers which define the small rectangular bounding&#xA;    box that contains all individual character bounding boxes&#xA;    '''&#xA;    boxes = train_data[i]['boxes'] &#xA;    x_min=9990&#xA;    y_min=9990&#xA;    x_max=0&#xA;    y_max=0&#xA;    for bid,b in enumerate(boxes):&#xA;        x_min = b['left'] if b['left'] &amp;lt;= x_min else x_min&#xA;        y_min = b['top'] if b['top'] &amp;lt;= y_min else y_min&#xA;        x_max = b['left']+b['width'] if  b['left']+b['width'] &amp;gt;= x_max else x_max&#xA;        y_max = b['top']+b['height'] if b['top']+b['height'] &amp;gt;= y_max else y_max&#xA;&#xA;    dy = y_max-y_min&#xA;    dx = x_max-x_min&#xA;    dpy = dy*perc&#xA;    dpx = dx*perc&#xA;    nboxes[i]=[dpx,dpy,dx,dy]&#xA;    return x_min-dpx, y_min-dpy, x_max+dpx, y_max+dpy&#xA;&#xA;for i in range(max_images):&#xA;    print (&quot; \r%d&quot; % i ,end=&quot;&quot;)&#xA;    filename = train_data[i]['filename']&#xA;    fullname = os.path.join(train_folders, filename)&#xA;    boxes = train_data[i]['boxes']&#xA;    label = [10,10,10,10,10]&#xA;    lb = len(boxes)&#xA;    if lb &amp;lt;= max_digits:&#xA;        im = Image.open(fullname)&#xA;        x_min, y_min, x_max, y_max = getBBox(i,0.3)&#xA;        im = im.crop([x_min,y_min,x_max,y_max])&#xA;        owidth, oheight = im.size&#xA;        wr = SV_IMG_SIZE/float(owidth)&#xA;        hr = SV_IMG_SIZE/float(oheight)&#xA;        for bid,box in  enumerate(boxes):&#xA;            sv_labels[i][max_digits-lb+bid] = int(box['label'])&#xA;&#xA;        box = nboxes[i]&#xA;        box[0]*=wr&#xA;        box[1]*=wr&#xA;        box[2]*=hr&#xA;        box[3]*=hr&#xA;        im = im.resize((SV_IMG_SIZE,SV_IMG_SIZE),Image.ANTIALIAS)&#xA;        img = img - np.mean(img)&#xA;        im = im.filter(ImageFilter.EDGE_ENHANCE)&#xA;        img = img - np.mean(img)&#xA;&#xA;        array = np.asarray(im)&#xA;        array =  array.reshape((SV_IMG_SIZE,SV_IMG_SIZE,3)).astype(np.float32)&#xA;        na = np.zeros([SV_IMG_SIZE,SV_IMG_SIZE],dtype=int)&#xA;        for x in range (array.shape[0]):&#xA;            for y in range (array.shape[1]):&#xA;                na[x][y]=np.average(array[x][y][:])&#xA;        na = na.reshape(SV_IMG_SIZE,SV_IMG_SIZE,1)&#xA;        #print(array.shape)&#xA;        sv_images.append(na.astype(np.float32))&#xA;&#xA;sv_train, sv_validation, svt_labels, svv_labels = train_test_split(sv_images, sv_labels, test_size=0.33, random_state=42)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is how I created and trained the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;x = Input((28, 28,1))&#xA;&#xA;y = Convolution2D(16, 3, 3, border_mode=&quot;same&quot;)(x)&#xA;#y = MaxPooling2D(pool_size = (2, 2), strides = (2, 2)) (y)&#xA;#y = Dropout(0.25)(y)&#xA;&#xA;y = Convolution2D(32, 4, 4, border_mode=&quot;same&quot;)(y)&#xA;y = MaxPooling2D(pool_size = (3, 3)) (y)&#xA;#y = Dropout(0.25)(y)&#xA;&#xA;y = Convolution2D(64, 5, 5, border_mode=&quot;same&quot;, activation=&quot;relu&quot;)(y)&#xA;y = MaxPooling2D((2, 2))(y)&#xA;#y = Dropout(0.25)(y)&#xA;&#xA;y = Convolution2D(128, 5, 5, border_mode=&quot;same&quot;, activation=&quot;relu&quot;)(y)&#xA;y = MaxPooling2D((2, 2))(y)&#xA;#y = Dropout(0.25)(y)&#xA;&#xA;&#xA;y = Flatten()(y)&#xA;y = Dense(1024, activation=&quot;relu&quot;)(y)&#xA;&#xA;digit1 = Dense(11, activation=&quot;softmax&quot;)(y)&#xA;digit2 = Dense(11, activation=&quot;softmax&quot;)(y)&#xA;digit3 = Dense(11, activation=&quot;softmax&quot;)(y)&#xA;digit4 = Dense(11, activation=&quot;softmax&quot;)(y)&#xA;digit5 = Dense(11, activation=&quot;softmax&quot;)(y)&#xA;model = Model(input=x, output=[digit1, digit2, digit3,digit4,digit5])&#xA;&#xA;&#xA;model.compile(optimizer='adam',&#xA;          loss='sparse_categorical_crossentropy',&#xA;          metrics=['accuracy'])&#xA;print(model.layers[0].output_shape)&#xA;print(model.layers[2].output_shape)&#xA;print(model.layers[4].output_shape)&#xA;print(model.layers[6].output_shape)&#xA;print(model.layers[8].output_shape)&#xA;&#xA;sv_train_labels = [svt_labels[:,0],svt_labels[:,1],svt_labels[:,2],svt_labels[:,3],svt_labels[:,4]]&#xA;sv_validation_labels = [svv_labels[:,0],svv_labels[:,1],svv_labels[:,2],svv_labels[:,3],svv_labels[:,4]]&#xA;&#xA;model.fit(sv_train, sv_train_labels, nb_epoch=10, batch_size=64,validation_data=(sv_validation, sv_validation_labels))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that I get very low accuracies that remain with the same value at each epoch:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train on 13400 samples, validate on 6600 samples&#xA;Epoch 1/10&#xA;13400/13400 [==============================] - 78s - loss: 34.7407 - dense_740_loss: 0.1161 - dense_741_loss: 0.6879 - dense_742_loss: 4.7988 - dense_743_loss: 14.7893 - dense_744_loss: 14.3486 - dense_740_acc: 0.9902 - dense_741_acc: 0.9542 - dense_742_acc: 0.7001 - dense_743_acc: 0.0810 - dense_744_acc: 0.1055 - val_loss: 34.7760 - val_dense_740_loss: 0.0049 - val_dense_741_loss: 0.7131 - val_dense_742_loss: 4.8721 - val_dense_743_loss: 14.8091 - val_dense_744_loss: 14.3769 - val_dense_740_acc: 0.9997 - val_dense_741_acc: 0.9558 - val_dense_742_acc: 0.6977 - val_dense_743_acc: 0.0812 - val_dense_744_acc: 0.1080&#xA;Epoch 2/10&#xA;13400/13400 [==============================] - 70s - loss: 34.7032 - dense_740_loss: 0.0036 - dense_741_loss: 0.6760 - dense_742_loss: 4.7861 - dense_743_loss: 14.8118 - dense_744_loss: 14.4257 - dense_740_acc: 0.9998 - dense_741_acc: 0.9581 - dense_742_acc: 0.7031 - dense_743_acc: 0.0810 - dense_744_acc: 0.1050 - val_loss: 34.7760 - val_dense_740_loss: 0.0049 - val_dense_741_loss: 0.7131 - val_dense_742_loss: 4.8721 - val_dense_743_loss: 14.8091 - val_dense_744_loss: 14.3769 - val_dense_740_acc: 0.9997 - val_dense_741_acc: 0.9558 - val_dense_742_acc: 0.6977 - val_dense_743_acc: 0.0812 - val_dense_744_acc: 0.1080&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27347" LastEditorUserId="29169" LastEditDate="2019-06-09T15:26:56.867" LastActivityDate="2019-06-09T15:26:56.867" Title="Keras CNN with low/constant accuracies" Tags="|machine-learning|python|deep-learning|tensorflow|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17082" PostTypeId="2" ParentId="17072" CreationDate="2017-02-21T10:08:25.503" Score="4" Body="&lt;p&gt;You can justify your choices by using data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treat the anomaly detection like a supervised learning problem where the concept is being anomaly.&#xA;Then you'll be able to present - for each method - its confusion matrix.&#xA;Not only it will be a good justification, it will enable to understand the expected results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many times, we have models and we wonder which confidence threshold we should use for alerting. In the supervised learning framework you'll be able to do trade-off like &quot;increasing the confidence to X will lead to a better precision Y yet a decrease of the recall to Z&quot;.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="26194" LastEditDate="2017-02-22T05:29:07.397" LastActivityDate="2017-02-22T05:29:07.397" CommentCount="14" ContentLicense="CC BY-SA 3.0" />
  <row Id="17083" PostTypeId="1" AcceptedAnswerId="17086" CreationDate="2017-02-21T10:14:08.863" Score="1" ViewCount="1005" Body="&lt;p&gt;I want to try out Keras (Theano backend) for regressions after already using sklearn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this I uses this nice tutorial &lt;a href=&quot;http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/&lt;/a&gt;&#xA; and tried to replace the training data there with my own. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;import pandas&#xA;import pickle&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.wrappers.scikit_learn import KerasRegressor&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.model_selection import KFold&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.pipeline import Pipeline&#xA;&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;&#xA;[X, Y] = pickle.load(open(&quot;training_data_1_week_imp_lt_15.pkl&quot;, &quot;rb&quot;))&#xA;&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split( X, Y, test_size=0.5, random_state=42)&#xA;scaler = StandardScaler()&#xA;scaler.fit(X_train)  # Don't cheat - fit only on training data&#xA;X_train = scaler.transform(X_train)&#xA;&#xA;X_test = scaler.transform(X_test)&#xA;&#xA;print (X_train.shape)&#xA;&#xA;&#xA;# define base mode&#xA;def baseline_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Dense(8, input_dim=8, init='normal', activation='relu'))&#xA;    model.add(Dense(1, init='normal'))&#xA;    # Compile model&#xA;    model.compile(loss='mean_squared_error', optimizer='adam')&#xA;    return model&#xA;&#xA;&#xA;# fix random seed for reproducibility&#xA;seed = 7&#xA;numpy.random.seed(seed)&#xA;# evaluate model with standardized dataset&#xA;estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=0)    &#xA;&#xA;estimator.fit(numpy.array(X_train),y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, i get the following error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Exception: Error when checking model target: the list of Numpy arrays that you are &#xA;passing to your model is not the size the model expected. Expected to see 1 &#xA;arrays but instead got the following list of 6252 arrays: ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The format of X is the usual sklearn format: print (X_train.shape) = (6252, 8)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I format my input X correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I tried was transposing but this did not work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also already searched the web but could not find a solution/explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; here is a small sample file &lt;a href=&quot;https://ufile.io/8a428&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ufile.io/8a428&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[X, Y] = pickle.load(open(&quot;test.pkl&quot;, &quot;rb&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-02-21T10:37:04.763" LastActivityDate="2017-02-21T13:49:52.720" Title="Format for X_train in keras using theano" Tags="|regression|scikit-learn|keras|data-formats|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17085" PostTypeId="2" ParentId="15328" CreationDate="2017-02-21T12:11:02.470" Score="24" Body="&lt;p&gt;In the paper &lt;a href=&quot;https://arxiv.org/pdf/1502.03167.pdf&quot; rel=&quot;noreferrer&quot;&gt;Batch Normalization&lt;/a&gt;,Sergey et al,2015. proposed &lt;strong&gt;Inception-v1&lt;/strong&gt; architecture which is a variant of the &lt;strong&gt;GoogleNet&lt;/strong&gt; in the paper &lt;strong&gt;&lt;em&gt;Going deeper with convolutions&lt;/em&gt;&lt;/strong&gt;, and in the meanwhile they introduced Batch Normalization to Inception(BN-Inception).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The main difference to the network described in (Szegedy et al.,2014)&#xA;  is that the 5x5 convolutional layers are replaced by two consecutive&#xA;  layer of 3x3 convolutions with up to 128 filters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And in the paper &lt;a href=&quot;https://www.arxiv.org/pdf/1512.00567v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;Rethinking the Inception Architecture for Computer Vision&lt;/a&gt;, the authors proposed Inception-v2 and Inception-v3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;strong&gt;Inception-v2&lt;/strong&gt;, they introduced Factorization(factorize convolutions into smaller convolutions) and some minor change into Inception-v1.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that we have factorized the traditional 7x7 convolution into&#xA;  three 3x3 convolutions&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As for &lt;strong&gt;Inception-v3&lt;/strong&gt;, it is a variant of Inception-v2 which adds BN-auxiliary.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;BN auxiliary refers to the version in which the fully connected layer of the auxiliary classifier is also-normalized, not just convolutions. We are refering to the model [Inception-v2 + BN auxiliary] as Inception-v3.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29223" LastEditorUserId="33689" LastEditDate="2017-06-28T17:07:10.523" LastActivityDate="2017-06-28T17:07:10.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17086" PostTypeId="2" ParentId="17083" CreationDate="2017-02-21T13:49:52.720" Score="1" Body="&lt;p&gt;I solved this (still banging my head against the wall): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimator.fit(numpy.array(X_train),numpy.array(y_train))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this works. I am not sure why. The error message is very misleading IMHO.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-02-21T13:49:52.720" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17087" PostTypeId="1" CreationDate="2017-02-21T14:09:28.900" Score="2" ViewCount="406" Body="&lt;p&gt;There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman Filtering- though I could not come up with suitable parameters to successfully use it.)&lt;/p&gt;&#xA;" OwnerUserId="26709" LastEditorUserId="83275" LastEditDate="2022-06-19T06:23:11.993" LastActivityDate="2022-06-19T06:23:11.993" Title="Online noise removal techniques" Tags="|dataset|data-mining|data-cleaning|kalman-filter|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17088" PostTypeId="1" CreationDate="2017-02-21T15:36:55.623" Score="1" ViewCount="87" Body="&lt;p&gt;Let assume that I own a model agency (that is a far-fetched example), and I would like to create a tool for my HR department, that gives an estimate salary for any given profile. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a huge database of profiles, where each name has a salary, sex,age, height,eye color, country of origin, weight, experience...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example : $($Mister $1234$ ,Male, $\$200$k/year,$6'3$ or $190$cm , $200$ pounds or $90$kg, Blue eyes, Brown hair, $22$ years old,...$)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that every time there is a new applicant, I input his/her characteristics into the tool, and it returns a salary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first idea that came in my mind is to assume that the salary $S_{\delta}$ is written as $$S_{\delta}=S(\delta_1,...,\delta_n)=\prod_{k=1}^n S_k^{\delta_k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\delta=(\delta_1,...,\delta_n)$, $S_k$ is a coefficient  related to the $k^{th}$ criteria, and $\delta_k$ equals one(resp. zero) when the $k^{th}$  criteria is matched(resp. not matched). $\delta$ entirely defines a model profile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can rewrite $S_{\delta}$ as $$log(S_{\delta})=\sum_{k=1}^{n}{\delta_klog(S_k)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y_{\delta}=\sum_{k=1}^{n}{\delta_kY_k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $Y_k=log(S_k)$ and $Y_{\delta}=log(S_{\delta})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess you know what I want to do next : I look at my database and I create a binary table for all profiles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example , &#xA;$$&#xA;\begin{array}{c|lcr}&#xA;Names &amp;amp;  \text{Salaries}&amp;amp; \text{Male}&amp;amp; \text{Female}&amp;amp; \text{under 185 cm}&amp;amp; \text{above 185cm} &amp;amp; \text{17-20yo} &amp;amp; \text{21-24yo} \\&#xA;\hline&#xA;C.C &amp;amp; 200 &amp;amp; 1 &amp;amp; 0 &amp;amp;0 &amp;amp;1&amp;amp;0 &amp;amp;1  \\&#xA;A.A &amp;amp; 300 &amp;amp; 0 &amp;amp; 1 &amp;amp;1 &amp;amp;0&amp;amp;1 &amp;amp;0  \\&#xA;B.B &amp;amp; 400 &amp;amp; 0 &amp;amp; 1 &amp;amp;1 &amp;amp;0&amp;amp;0 &amp;amp;1 &#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let assume I have a sample of size $N_{sample}$, I will solve &#xA;$$\min_{Y_1,...,Y_n}\sum_{l=1}^{N_{sample}}{\left(Y_{\delta_l}-Salary(l)\right)^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\delta_l$ and $Salary(l)$ are respectively the criteria , and the salary   of the $l^{th}$model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Excel$ can solve such a problem quickly by using the function $LINETEST$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My issue is not about solving the equation , but &lt;strong&gt;how to pre-process the data when the input is a binary table.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, in another context , I would use the classical tool such as PCA on correlation matrix. However, here, I have a binary table , and in that case, we can clearly distinguish causality and correlation .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides using common sense to get rid of irrelevant factors( ex : male and above 185 cm always go along, one factor is redundant) , &lt;strong&gt;are there any other techniques to reduce the dimension of the problem in that context ?&lt;/strong&gt;  &lt;/p&gt;&#xA;" OwnerUserId="29234" LastActivityDate="2017-02-21T15:36:55.623" Title="Dimension reduction : binary description and regression" Tags="|regression|data-cleaning|svm|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17089" PostTypeId="1" CreationDate="2017-02-21T17:12:58.333" Score="1" ViewCount="52" Body="&lt;p&gt;I created an algorithm which works on categorical attributes. The input data comes with categorical attributes, but numerical ones as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I apply a pre-processing which transforms the numerical attributes into categorical ones, in clever ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the number of categories can adapt to the distribution of numerical values&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;-&gt; for example if there are 100 values, 99 of them grouped in a small range, and one clear outlier, then 2 categories should be created.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;the cutoff between categories should do its best to avoid that two close values are in different categories.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;-&gt; for example, it is annoying that for 2 numerical values ranging from 0 to 100, a cutoff at 50 leaves values 49 and 51 in 2 different categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a field in computer science which deals with these issues? If there is an algorithm, is it pre-implemented in any language?&lt;/p&gt;&#xA;" OwnerUserId="29236" LastEditorUserId="12384" LastEditDate="2017-02-21T22:03:36.440" LastActivityDate="2017-02-21T22:03:36.440" Title="Looking for an algo transforming numerical attributes into categorical attributes -cleverly" Tags="|clustering|categorical-data|unsupervised-learning|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="17090" PostTypeId="1" CreationDate="2017-02-21T18:29:26.647" Score="0" ViewCount="279" Body="&lt;p&gt;I am new to machine learning and I was wondering what algorithm to use to predict waveforms from an already existing waveform data samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the x coordinates and Y coordinates or (waveforms) for a few set of features for the different set of values. I want to be able to predict further waveforms for more values using this as training data. Which type of Machine Learning Algorithm should be used in such cases ?&lt;/p&gt;&#xA;" OwnerUserId="29240" LastActivityDate="2021-11-28T15:13:41.250" Title="Which Machine Learning Algorithm should be used to predict Waveforms from an already existing collection of waveforms ?" Tags="|machine-learning|predictive-modeling|dataset|regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17095" PostTypeId="1" AcceptedAnswerId="17108" CreationDate="2017-02-21T20:24:10.763" Score="2" ViewCount="386" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e7l93.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e7l93.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am currently trying to make a facial recognition workflow with Orange but since the widgets are still prototypes they are not all available in Orange through the add-on menu. I found the Github files for the widgets but don't know how to install this so that I can see and use the widgets in Orange.&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2022-09-25T14:58:38.213" LastActivityDate="2022-09-25T14:58:38.213" Title="How do I integrate Github files to Orange for ML?" Tags="|orange|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17096" PostTypeId="1" CreationDate="2017-02-21T20:26:03.690" Score="6" ViewCount="1102" Body="&lt;p&gt;I want to compare two corpora (two different collections of texts) using Topic Modeling. I trained the model separately on the two collections and manually matched similar topics based on their frequent words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a systematic way of comparing the topics across two corpora and measuring their similarity. &lt;/p&gt;&#xA;" OwnerUserId="17120" LastActivityDate="2017-02-23T14:51:08.273" Title="Comparing two Corpora using Topic Model" Tags="|text-mining|topic-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17097" PostTypeId="2" ParentId="17095" CreationDate="2017-02-21T21:06:51.080" Score="0" Body="&lt;p&gt;You may be able to install it through the &lt;em&gt;Options &gt; Add-ons&lt;/em&gt; menu in the program.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-02-21T21:06:51.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17098" PostTypeId="2" ParentId="17079" CreationDate="2017-02-21T22:00:31.530" Score="2" Body="&lt;p&gt;Instead of trying to re-order the columns and rows, I would suggest trying to find some other way to visualize the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one possible alternative suggestion.   You could cluster the classes, say into ~ 20 clusters, where each cluster has ~ 20 classes in it, using some kind of clustering algorithm that puts similar classes together into the same cluster (e.g., if two classes are frequently confused with each other, they should be more likely to be in the same cluster).  Then you can show a coarse-grained confusion matrix, with one row/column per cluster; the cell at $(i,j)$ shows how often an instance of some class in cluster $i$ is predicted to have some class in cluster $j$.  Also, you can have ~ 20 fine-grained confusion matrices: for each cluster, you can show the confusion matrix of classes, for the ~ 20 classes in each cluster.  Of course, you could also extend this by using hierarchical clustering and have confusion matrices at multiple granularities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may be other possible visualization strategies as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general philosophical point: it might also help to clarify your goals (what you want to get out of the visualization).  You can distinguish two kinds of uses of visualization:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Exploratory analysis: You're not sure what you're looking for; you just want a visualization that might help you look for interesting patterns or artifacts in the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Figures with a message: You have a particular message you want the reader to take away, and you want to devise a visualization that helps support that message or provide evidence for the message.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It might help you to know which you're trying to aim for, and then devise a visualization aimed at that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you're doing exploratory analysis, rather than trying to pick one perfect visualization, it's often helpful to try creating as many visualizations as you can think of.  Don't worry about whether any of them are perfect; it's OK if each one is flawed, as each might give you a potentially different perspective on the data (it will probably be good in some ways and bad in others).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you have a particular message you're trying to convey or a theme you're trying to develop, look for a visualization that supports that theme.  It's hard to make a specific suggestion without knowing what that theme/message might be.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-02-21T22:00:31.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17099" PostTypeId="1" CreationDate="2017-02-21T22:17:40.000" Score="69" ViewCount="50721" Body="&lt;p&gt;have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to adding additional features to what is already a list of time series features. Assuming you have your dataset up like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;t-3,t-2,t-1,Output&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now lets say you know you have a feature that does affect the output but its not necessarily a time series feature, lets say its the weather outside. Is this something you can just add and the LSTM will be able to distinguish what is the time series aspect and what isnt?&lt;/p&gt;&#xA;" OwnerUserId="26800" LastEditorUserId="1330" LastEditDate="2019-05-16T15:59:54.660" LastActivityDate="2020-08-05T10:35:35.827" Title="Adding Features To Time Series Model LSTM" Tags="|machine-learning|neural-network|deep-learning|time-series|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17100" PostTypeId="2" ParentId="334" CreationDate="2017-02-21T22:47:29.407" Score="2" Body="&lt;p&gt;It really depends on the credibility of the institution granting the certificate. For example, &lt;a href=&quot;https://www.experfy.com/training/certifications/data-science-certification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science Certification&lt;/a&gt; from a Harvard-based company is recognized by many industry partners and may make a good choice. You did not say what kind of certificate you are looking for?&lt;/p&gt;&#xA;" OwnerUserId="29249" LastActivityDate="2017-02-21T22:47:29.407" CommentCount="0" CommunityOwnedDate="2017-02-21T22:47:29.407" ContentLicense="CC BY-SA 3.0" />
  <row Id="17104" PostTypeId="2" ParentId="17079" CreationDate="2017-02-22T01:43:35.767" Score="0" Body="&lt;p&gt;Its important to know why the EMNIST confusion matrix looks good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I find it weird that they haven't maintained the colouring with higher numbers being darkest, for example most of the empty miss-classifications containing zeros are of a darker grey than those that contain an integer. Doesn't seem consistent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd try using the EMINST style except keep it consistent where colour indicates the number of entries in a cell. White for zero entires, and black for the most entries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A perfect classification would be a black diagonal with completely white upper and lower triangles. Where there was any grey patches in the triangles would indicate problems. Even on a 1000 class set this would be helpful.&#xA;For ImageNet where the classes are hierarchical, perhaps sorting the columns so that subclasses  are grouped the right of the parent class would lead to squareish dark patches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you are getting the top 5 responses for an image, classes may not be mutually exclusive, such that dog classification for an image of a lap_dog should still be true, hence in such a confusion matrix, the more general classes should be much darker than the precise classifications (if the colours are normalised.) Hence the top left square would be darkest.&lt;/p&gt;&#xA;" OwnerUserId="28922" LastActivityDate="2017-02-22T01:43:35.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17105" PostTypeId="2" ParentId="17080" CreationDate="2017-02-22T02:14:45.250" Score="2" Body="&lt;p&gt;As I don't have your data, I can only give you some suggestions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;10 epoches may be too few. In my practice, 100+ epoches will be applied. I think 100 epoches will be a good start for you.&lt;/li&gt;&#xA;&lt;li&gt;Assuming your data is fully-prepared, some suggestions of the CNN:&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first and the second convolution layers should have ReLU activations, at least not linear activations.&lt;/li&gt;&#xA;&lt;li&gt;Why do your output layer have 11 neurons? There are only 10 digits, right?&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sparse_categorical_crossentropy&lt;/code&gt; is used for sparse input. Your &lt;code&gt;sv_train_labels&lt;/code&gt; seems to be a non-sparse array, thus &lt;code&gt;categorical_crossentropy&lt;/code&gt; is better.&lt;/li&gt;&#xA;&lt;li&gt;Your 3x3, 4x4, 5x5 convolution layer seems weird, but I can't give you a reason (maybe someone else?). If I were you, I would use 3x3 layer.&lt;/li&gt;&#xA;&lt;li&gt;If you want to print your output shape of your model, try &lt;code&gt;model.summary()&lt;/code&gt;. It's awesome!&lt;/li&gt;&#xA;&lt;li&gt;If I were you, I'll try &lt;code&gt;RMSprop&lt;/code&gt; rather than &lt;code&gt;Adam&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My model if you wanna try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = Input((28, 28, 1))&#xA;&#xA;y = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(x)&#xA;y = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(y)&#xA;y = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(y)&#xA;y = MaxPooling2D((2, 2))(y)&#xA;y = Convolution2D(128, 3, 3, activation='relu', border_mode='same')(y)&#xA;y = MaxPooling2D((2, 2))(y)&#xA;y = Flatten()(y)&#xA;y = Dense(1024, activation='relu')(y)&#xA;&#xA;digit1 = Dense(10, activation=&quot;softmax&quot;)(y)&#xA;digit2 = Dense(10, activation=&quot;softmax&quot;)(y)&#xA;digit3 = Dense(10, activation=&quot;softmax&quot;)(y)&#xA;digit4 = Dense(10, activation=&quot;softmax&quot;)(y)&#xA;digit5 = Dense(10, activation=&quot;softmax&quot;)(y)&#xA;model = Model(input=x, output=[digit1, digit2, digit3, digit4, digit5])&#xA;&#xA;model.compile('RMSprop', 'categorical_crossentropy', ['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-22T02:14:45.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17107" PostTypeId="1" CreationDate="2017-02-22T02:54:39.870" Score="2" ViewCount="99" Body="&lt;p&gt;I'm looking at trying to build a model to score an archery end from a photo but i'm not sure about the optimal ConvNet architecture.  What output neurons would I use given I can have an unknown number of arrows in a photo?  I could do a regression type solution of 'total score' I guess but i'd like to get a per arrow score (or perhaps the x,y coordinates of each arrow on a projected grid).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts, wisdom, etc would be appreciated.  I'm likely not quite thinking about this the right way.&lt;/p&gt;&#xA;" OwnerUserId="29256" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:35.663" LastActivityDate="2021-02-16T02:52:35.663" Title="ConvNet for Archery App" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17108" PostTypeId="2" ParentId="17095" CreationDate="2017-02-22T02:55:05.020" Score="4" Body="&lt;p&gt;There are three ways to install Orange add-ons:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;code&gt;Options -&amp;gt; Add-ons&lt;/code&gt; menu of Orange.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use pip: &lt;code&gt;pip3 install orange3-prototypes&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Install from the source:&lt;/p&gt;&#xA;&lt;p&gt;Terminal commands:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;git clone https://github.com/biolab/orange3-prototypes.git&#xA;cd orange3-prototypes&#xA;python3 setup.py install&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-28T22:53:36.477" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17109" PostTypeId="1" CreationDate="2017-02-22T02:56:25.207" Score="1" ViewCount="277" Body="&lt;p&gt;I'm analyzing the GDELT dataset and I want to determine thematic clusters. Simplifying considerably, GDELT parses news articles and extracts events. As part of that, it recognizes, let's say, 250 &quot;themes&quot; and tags each &quot;event&quot; it records in a column a semi-colon separated list of all themes identified in the article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that preamble, I've extracted, for 2016, a list of approximately 350,000 semi-colon separated theme lists, such as these two:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;TAX_FNCACT;TAX_FNCACT_QUEEN;CRISISLEX_T11_UPDATESSYMPATHY;CRISISLEX_CRISISLEXREC;MILITARY;TAX_MILITARY_TITLE;TAX_MILITARY_TITLE_SOLDIER;TAX_FNCACT_SOLDIER;USPEC_POLITICS_GENERAL1;WB_1458_HEALTH_PROMOTION_AND_DISEASE_PREVENTION;WB_1462_WATER_SANITATION_AND_HYGIENE;WB_635_PUBLIC_HEALTH;WB_621_HEALTH_NUTRITION_AND_POPULATION;MARITIME_INCIDENT;MARITIME;MANMADE_DISASTER_IMPLIED;&lt;/li&gt;&#xA;&lt;li&gt;CRISISLEX_CRISISLEXREC;EDUCATION;SOC_POINTSOFINTEREST;SOC_POINTSOFINTEREST_COLLEGE;TAX_FNCACT;TAX_FNCACT_MAN;TAX_ECON_PRICE;SOC_POINTSOFINTEREST_UNIVERSITY;TAX_FNCACT_JUDGES;TAX_FNCACT_CHILD;LEGISLATION;EPU_POLICY;EPU_POLICY_LAW;TAX_FNCACT_CHILDREN;WB_470_EDUCATION;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you can see, both of these lists both contain &quot;TAX_FNACT&quot; and &quot;CRISISLEX_CRISISLEXREC&quot;. Thus, &quot;TAX_FNACT;CRISISLEX_CRISISLEXREC&quot; is a 2-item cluster. A better understanding of GDELT informs us that it isn't a particularly useful cluster, but it is one nevertheless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to do, ideally, is compose a dictionary of lists. The key for the dictionary is the number of items in the cluster and value is a list of tuples of all theme clusters with that &quot;key&quot; number of elements paired with the number of times that cluster appeared. This ideal algorithm would run until it identified the largest cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does an algorithm already exist that I can use for this purpose and if so, what is it named? If I had to guess, I would imagine we've created something to extract x-item clusters and then I would just loop from 2-&gt;? until I don't get any results.&lt;/p&gt;&#xA;" OwnerUserId="29257" LastActivityDate="2017-02-22T02:56:25.207" Title="Seeking Appropriate Clustering Algorithm" Tags="|clustering|algorithms|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17110" PostTypeId="1" CreationDate="2017-02-22T03:42:03.373" Score="2" ViewCount="81" Body="&lt;p&gt;I am  having  some images on which  I   trained some neural   network  models in order to  do classification( binary  task).&#xA;I am interested  of a  method    to identify  some patterns  or regions  in those images  that are  influencial  the classification   part in  other words  what  regions or  what patterns are formed  for a particular  class.&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2017-02-22T04:44:47.097" Title="Regions /patterns in images that creates classification" Tags="|predictive-modeling|deep-learning|image-classification|image-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17111" PostTypeId="2" ParentId="17110" CreationDate="2017-02-22T04:44:47.097" Score="1" Body="&lt;p&gt;I think visualizing convolutional neural networks is what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs231n.github.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n&lt;/a&gt; is a good place to start, here is a step-by-step &lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;guide&lt;/a&gt; about how to visualize the activations, first-layer weights, and shows how to retrieve images which maximally activate a neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also there's a &lt;a href=&quot;https://transcranial.github.io/keras-js/&quot; rel=&quot;nofollow noreferrer&quot;&gt;demo&lt;/a&gt; which visualized multiple convnets (including ResNet50, Inception v3, etc). You may learn a lot from this demo (at least I do).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're interesting in details, this well-known &lt;a href=&quot;https://arxiv.org/abs/1311.2901&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; may help you.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-22T04:44:47.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17112" PostTypeId="1" CreationDate="2017-02-22T07:14:45.970" Score="1" ViewCount="36" Body="&lt;p&gt;I have a silly question. Below is the output of a logistic regression analysis I did. I notice that when I switch the order of the arguments I put in the table function in R that it also switch the false positives and the false negatives values but it did not switch the the location of the Female and Male rows and columns. This to me seems like it could really affect the interpretation if the false positives/negatives can change just from the order the arguments are put into the table. I need to know which order output is correct and the proper order for placing arguments in the table function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Vtf5M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Vtf5M.png&quot; alt=&quot;Results&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17959" LastEditorUserId="29169" LastEditDate="2021-02-01T20:12:11.513" LastActivityDate="2021-02-01T20:12:11.513" Title="Table function output and order of arguments" Tags="|logistic-regression|data-table|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17113" PostTypeId="1" AcceptedAnswerId="17117" CreationDate="2017-02-22T07:28:34.113" Score="1" ViewCount="979" Body="&lt;p&gt;To my knowledge, recommender systems are broadly classified into collaborative and content. Collaborative in turn is divided into 1) Memory (uses similarity metrics) and 2) Model (well known Matrix/Tensor factorization). Content based involves constructing a user profile and then an algorithm like SVM to classify and recommend items. Now here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What other algorithms can I use for recommendation and why?&lt;/li&gt;&#xA;&lt;li&gt;Can I use neural networks? (understanding them has been a bit difficult for me)&lt;/li&gt;&#xA;&lt;li&gt;Is it true that neural networks (NN) are only suited for text and image processing and numerical data doesn't need complex algorithms like NN?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29128" LastEditorUserId="8878" LastEditDate="2017-05-24T14:28:18.283" LastActivityDate="2017-05-24T14:28:18.283" Title="What algorithms should I choose for a recommender system and why?" Tags="|machine-learning|neural-network|algorithms|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17114" PostTypeId="1" CreationDate="2017-02-22T08:00:45.617" Score="2" ViewCount="147" Body="&lt;p&gt;In the following hand made charts I show some value for years. In the first chart I've evenly spaced each year. On the second chart I've spaced them relativelly to their actual year value within time (i.e 2016 is closer to 2017 than 2010).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a terminology for the spacing of the second chart? Imagine building a software which would have a toggle control to switch the view from A to B. How would you call it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OMFR4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OMFR4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29265" LastEditorUserId="8878" LastEditDate="2017-02-23T07:52:56.120" LastActivityDate="2023-09-25T05:01:25.130" Title="chart x-axis spacing terminology question" Tags="|visualization|terminology|plotting|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17115" PostTypeId="2" ParentId="15313" CreationDate="2017-02-22T11:15:39.423" Score="1" Body="&lt;p&gt;As there is no answer yet I want to try to give an at least somehow useful answer.&#xA;Including the constants definitions would help a bit to understand the provided code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally speaking a batch uses n times a record or item. How you define an item depends on your problem. In tensorflow the batch is encoded in the first dimension of a tensor. In your case with the csv file it might be line by line (&lt;code&gt;reader=tf.TextLineReader&lt;/code&gt;). It could learn by column but I don't think that this is happening in your code. If you want train with your whole dataset (=&lt;a href=&quot;http://www.cse.unsw.edu.au/~billw/mldict.html#epoch&quot; rel=&quot;nofollow noreferrer&quot;&gt;one epoch&lt;/a&gt;) you can do so by using &lt;code&gt;numBatches=numItems/batchSize&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28011" LastEditorUserId="28011" LastEditDate="2017-11-21T07:24:43.353" LastActivityDate="2017-11-21T07:24:43.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17116" PostTypeId="1" CreationDate="2017-02-22T11:56:01.313" Score="2" ViewCount="92" Body="&lt;p&gt;I have a collection of data points. Each point has 6 dimensions (&lt;code&gt;x1, x2,...x6&lt;/code&gt;). I want to find a relation between two dimension (e.g. &lt;code&gt;x1 vs x2&lt;/code&gt;). What I have been doing so far is look for points where the other dimensions (&lt;code&gt;x3 to x6&lt;/code&gt;) are relatively constant, by defining a band. This way I would get several groups of data points where only the two dimensions of interest would change. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a better way of analyzing the relationship between these two dimensions. I looked at PCA, but I have a feeling that it does not help me much. If I reduce the problem to two dimensions the axes are basically meaningless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you guys give me some directions to look at?&lt;/p&gt;&#xA;" OwnerUserId="29277" LastEditorUserId="31513" LastEditDate="2017-05-25T10:35:27.967" LastActivityDate="2017-05-26T16:53:14.350" Title="Finding the relation between two dimensions in a multi-dimensional problem" Tags="|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17117" PostTypeId="2" ParentId="17113" CreationDate="2017-02-22T16:19:42.547" Score="0" Body="&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on your data. What do you want to do ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer answer:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use content based approaches if you have data on your items. Use collaborative approaches is you have data on your users. Use both if you have both. I would say content based approaches are general machine learning problems (how do I extract meaningful information from data) whereas collaborative filtering is really a recommender system specific work (how user's behavior can suggest users/items similarity/connections). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Well you can. Neural nets are just a kind of algorithm, you surely can use them for content based analysis, and it might be possible to use them to enhance your collaborative algorithm. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;NN use texts and images as numerical data, so I don't understand your question. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you want a good insight in today's recommender systems, take a look at this &lt;a href=&quot;https://datajobs.com/data-science-repo/Collaborative-Filtering-[Koren-and-Bell].pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-02-22T16:19:42.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17118" PostTypeId="1" CreationDate="2017-02-22T16:31:23.933" Score="3" ViewCount="114" Body="&lt;p&gt;Excuse the potentially dumb question, I've only just started learning about data science.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do we find out how much data we should collect before using it to start making decisions?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of knowing if we should wait to collect more data? Or if the quantity of data can lead to meaningful results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; we have a mobile app and some stats about it, shown below. It's early days and the data set is small. We want to start using this data to make better decisions about improving the product. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/E2DoY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/E2DoY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29286" LastActivityDate="2017-06-23T14:12:13.283" Title="How much app analytics data to collect?" Tags="|data-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17119" PostTypeId="1" AcceptedAnswerId="17123" CreationDate="2017-02-22T18:47:11.223" Score="2" ViewCount="114" Body="&lt;p&gt;I am writing a decision tree trained with the ID3 algorithm from scratch. I wanted to be able to train on and classify continuous data, so I implemented k-means clustering and reduced the range of values of any input training or predicting data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I ran into the problem where an attribute value that was not encountered during training, in a deep node somewhere in the tree, but that exists further up was encountered. All data points with this specific attribute value probably ended up on a different branch of the tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to 'solve' this, whenever an unknown attribute value is encountered for a node, I sent it down a random existing branch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get 95.45% accuracy using randomly split 85%-15% training-test data with iris.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this an acceptable approach to take or have I gotten something wrong here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code: &lt;a href=&quot;https://github.com/jamalmoir/ml_components/blob/master/ml_components/models/decision_tree.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jamalmoir/ml_components/blob/master/ml_components/models/decision_tree.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="19241" LastActivityDate="2017-02-23T03:05:48.593" Title="Is it acceptable to select a random child node when using a Decision Tree (trained via ID3) to predict if an unknown attribute value is encountered" Tags="|algorithms|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17121" PostTypeId="1" CreationDate="2017-02-23T02:47:26.133" Score="3" ViewCount="2348" Body="&lt;p&gt;I have heard the term &lt;em&gt;'feature coverage'&lt;/em&gt; in machine learning. However I found no relative infomation after I googled this term. Could some one give me a definition of &lt;em&gt;'feature coverage'&lt;/em&gt; and some references?&lt;/p&gt;&#xA;" OwnerUserId="29298" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:50.220" LastActivityDate="2018-04-24T12:56:09.617" Title="Definition of 'feature coverage'" Tags="|feature-selection|terminology|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17122" PostTypeId="1" CreationDate="2017-02-23T03:00:57.410" Score="6" ViewCount="1212" Body="&lt;p&gt;Is there any way to implement a loss function that is shared between outputs? I have a 2D image output and scalar classification that are both used by a single loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attempted writing a function that returns a function, as in &lt;a href=&quot;https://github.com/fchollet/keras/issues/2121#issuecomment-214551349&quot; rel=&quot;noreferrer&quot;&gt;this comment&lt;/a&gt;, but I would need the input to the function to be the current training example. I also thought of using a merge layer, but that wouldn't work due to an incompatibility of the layer dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of a way to write such a loss function in keras?&lt;/p&gt;&#xA;" OwnerUserId="29297" LastActivityDate="2017-02-23T03:00:57.410" Title="Keras objective function shared between outputs" Tags="|keras|loss-function|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17123" PostTypeId="2" ParentId="17119" CreationDate="2017-02-23T03:05:48.593" Score="1" Body="&lt;p&gt;Do your leaf nodes return a probability distribution rather than a single class value &lt;em&gt;e.g.&lt;/em&gt;, the majority class of training instances that arrived at the node? If so, a better approach would be to send the test instance down every child path and average the probability distributions that are returned by the leaves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your good score of 95.45% on iris data suggests to me that it's not a very important detail in this example. Try some more challenging datasets and see if it makes a difference.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-02-23T03:05:48.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17124" PostTypeId="1" CreationDate="2017-02-23T03:51:58.727" Score="3" ViewCount="722" Body="&lt;p&gt;I have a feature for machine learning (using methods like SVM, naive bayes, neural network and random forest) called member duration as follows:&#xA;Should I make it as numerical or categorical data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fyMsI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fyMsI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="7812" LastEditDate="2017-02-23T06:32:46.777" LastActivityDate="2017-12-20T14:40:27.307" Title="numerical or categorical data" Tags="|feature-engineering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17125" PostTypeId="1" CreationDate="2017-02-23T04:01:44.687" Score="3" ViewCount="1207" Body="&lt;p&gt;I have a feature for machine learning as follow that skew to the left, and only have number in certain number range (here 0-2000). Will skewness and range of number affect the learning? If yes what should I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/M00zz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/M00zz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user29151" LastActivityDate="2018-03-27T16:53:32.057" Title="Effect of Skewness and data range in machine learning" Tags="|feature-engineering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17126" PostTypeId="1" AcceptedAnswerId="17131" CreationDate="2017-02-23T04:29:01.447" Score="1" ViewCount="1018" Body="&lt;p&gt;I'm building a report that has month over month data, but also &quot;this month last year&quot;. Is there a better/standard way of describing this?&lt;/p&gt;&#xA;" OwnerUserId="29302" LastActivityDate="2017-02-23T05:44:34.900" Title="Is there a term for &quot;this month last year&quot; in a report?" Tags="|visualization|databases|terminology|" AnswerCount="1" CommentCount="1" ClosedDate="2017-02-23T21:10:52.937" ContentLicense="CC BY-SA 3.0" />
  <row Id="17127" PostTypeId="2" ParentId="17124" CreationDate="2017-02-23T04:54:13.820" Score="0" Body="&lt;p&gt;It looks like counting data to me. Without further information in the question, I'd keep it as a categorical data and model it with discrete techniques (e.g. Poisson GLM)&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-02-23T04:54:13.820" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17128" PostTypeId="1" AcceptedAnswerId="17132" CreationDate="2017-02-23T05:06:06.003" Score="1" ViewCount="1196" Body="&lt;p&gt;I showed some results of one implemented NN MLP model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the result, for classification of two categories, if I sum up the probabilities of both cats, them some sum would be greater than 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I was asked why the sum is greater than 1, I gave the guess that the probability stands for &lt;em&gt;confidence&lt;/em&gt; concept in mathematics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone tell me am I right?&lt;/p&gt;&#xA;" OwnerUserId="26856" LastActivityDate="2017-02-23T07:35:29.610" Title="Explanation for MLP classification probability" Tags="|machine-learning|neural-network|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17129" PostTypeId="2" ParentId="17118" CreationDate="2017-02-23T05:11:12.277" Score="1" Body="&lt;p&gt;Short answer is: &lt;strong&gt;as much as possible&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more data you have, the more likely you can make correct inference. In the worse case, you could simply discard anything that you don't like. It's like money, the more the better - you always have an option to spend or use less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Statistics could be doggy without sufficient sample size, your statistical power would be affected and any model you have could be biased.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, you &lt;strong&gt;don't want everything&lt;/strong&gt; from your user. Use your common sense, and ask yourself what exactly you want to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based &lt;strong&gt;only&lt;/strong&gt; on the data in your question, I'd say: (I could be wrong because I don't understand where you got the data)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How many total users you have that &lt;strong&gt;didn't drop out&lt;/strong&gt;? If your total user base is like 10K, 28 users dropping out is not a problem. But if you only have 30 users...&lt;/li&gt;&#xA;&lt;li&gt;Your sample size is small. Is that one-day installation? In any case, 28 users for a mobile app is a tiny data set. You may want to &lt;strong&gt;focus more on marketing&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many respondent you need is hard to tell. Apple probably needs better data quality than you. If you'd like to do it statistically, you may calculate the minimum sample size and it's related to &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_power&quot; rel=&quot;nofollow noreferrer&quot;&gt;statistical power&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-02-23T10:36:01.850" LastActivityDate="2017-02-23T10:36:01.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17130" PostTypeId="2" ParentId="17125" CreationDate="2017-02-23T05:13:32.500" Score="2" Body="&lt;p&gt;Typically, folks would transform the variable. When it is strictly greater than zero, a log transform is usually sufficient. If zero is included, as in your case, one popular alternative is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Power_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;box-cox transformation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="27250" LastEditDate="2017-02-23T05:18:42.047" LastActivityDate="2017-02-23T05:18:42.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17131" PostTypeId="2" ParentId="17126" CreationDate="2017-02-23T05:44:34.900" Score="0" Body="&lt;p&gt;&lt;strong&gt;Same period last year&lt;/strong&gt; is what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, your question does not appear to be about data science.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://msdn.microsoft.com/en-us/library/ee634972.aspx&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://msdn.microsoft.com/en-us/library/ee634972.aspx&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wiseowl.co.uk/blog/s2477/same-period-previous-year.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wiseowl.co.uk/blog/s2477/same-period-previous-year.htm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-23T05:44:34.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17132" PostTypeId="2" ParentId="17128" CreationDate="2017-02-23T07:35:29.610" Score="3" Body="&lt;p&gt;It depends a little on the specific model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model is using a softmax output layer, then the values are usually interpreted as mutually-exclusive class probabilities and should sum to 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model is using a sigmoid output layer, then the values for a classification problem can be interpreted as the individual, &lt;em&gt;non-exclusive&lt;/em&gt; probabilities. In that case, it is possible for an example to not be in either of your classes (an output close to [0,0]), or to definitely be in both classes (an output close to [1,1]) - if that is not the case in the problem that you have trained the NN to model, then you might want to consider altering the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if your classification &lt;em&gt;should&lt;/em&gt; be mutually-exclusive, then the person asking you the question has a valid point. The probabilities should sum to 1, and you may need to fix the model. In that case, you should use a softmax output layer, and would not have this problem because the probabilities would always sum to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether it is worth fixing the model, depends on your understanding of mutually-exclusive vs non-exclusive classes in the problem, whether the current model meets its goals (it is not a big issue in practice, if the model is not a strict theoretical match to the problem, but has a perfectly good accuracy) and how much effort it would take to alter your code and re-train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the probability stands for &lt;em&gt;confidence&lt;/em&gt; concept in mathematics.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is technically correct, but not useful as an answer to your colleague's query. For classification, the terms &lt;em&gt;confidence&lt;/em&gt; and &lt;em&gt;probability&lt;/em&gt; are almost interchangeable, so you cannot really use one word to explain the other. If someone asked why one of the numbers was larger than another in your model, you would not answer &quot;that is because x is greater than y&quot; - essentially this is what you have done here. So your statement is not wrong as such, but neither does it explain the result.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-02-23T07:35:29.610" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17133" PostTypeId="2" ParentId="17114" CreationDate="2017-02-23T07:52:04.183" Score="0" Body="&lt;p&gt;It's usually called &quot;Tick&quot; and as it's for x axis, let's say &quot;xtick&quot;. You can label it as xticks and give it two values &quot;evenly spacing&quot; and &quot;actual spacing&quot; (This is just an idea as there is no standard terminology as far as I know).&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-02-23T07:52:04.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17134" PostTypeId="1" CreationDate="2017-02-23T11:15:41.380" Score="11" ViewCount="1457" Body="&lt;p&gt;I'm wondering how to interpret a recurrent architecture in an EEG context. Specifically I'm thinking of this as a Recurrent CNN (as opposed to architectures like LSTM), but maybe it applies to other types of recurrent networks as well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I read about R-CNNs, they're usually explained in image classification contexts. Theyre typically described as &quot;learning over time&quot; or &quot;including the effect of time-1 on the current input&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This interpretation/explanation gets really confusing when working with EEG data. An example of an R-CNN being used on EEG data can be found &lt;a href=&quot;https://github.com/stupiding/kaggle_EEG&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I have training examples each consisting of a 1x512 array. This array captures a voltage reading for 1 electrode at 512 consecutive time points. If I use this as input to a Recurrent CNN (using 1D convolutions), the recurrent part of the model isn't actually capturing &quot;time&quot;, right? (as would be implied by the descriptions/explanations discussed earlier) Because in this context time is already captured by the second dimension of the array&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So with a setup like this, what does the recurrent part of the network actually allow us to model that a regular CNN can't (if not time)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:26.147" LastActivityDate="2021-02-16T02:52:26.147" Title="Recurrent (CNN) model on EEG data" Tags="|machine-learning|neural-network|rnn|convolutional-neural-network|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17135" PostTypeId="1" AcceptedAnswerId="17145" CreationDate="2017-02-23T11:22:11.837" Score="4" ViewCount="168" Body="&lt;p&gt;I'm currently training CNNs using Tensorflow (Python) on my GTX 970 (specs &lt;a href=&quot;http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-970/specifications&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;). I recently took a look at the new pascal based &lt;a href=&quot;https://www.nvidia.com/en-us/geforce/products/10series/titan-x-pascal/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Titan Xs&lt;/a&gt; and I'm wondering what an estimated performance/speed gain would be if I upgraded?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The memory increase is an obvious benefit for larger models, but I'm mostly wondering about speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the doubling of core count and bump in memory speed offer that much of a performance gain? Anyone have any experience with the new (or old) Titan X cards?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:14.503" LastActivityDate="2021-02-16T02:52:14.503" Title="Estimating Titan X graphics card impact on performance" Tags="|python|tensorflow|convolutional-neural-network|performance|gpu|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17136" PostTypeId="1" CreationDate="2017-02-23T12:43:28.863" Score="1" ViewCount="316" Body="&lt;p&gt;I'm searching for a paper or book that list several possible measures for deciding which variable to split a Decision Tree at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic ones are well known and very well documented in several sources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Entropy (or Information Gain)&lt;/li&gt;&#xA;&lt;li&gt;Gini impurity&lt;/li&gt;&#xA;&lt;li&gt;Classification Error&lt;/li&gt;&#xA;&lt;li&gt;Variance reduction&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I suppose people come up with all kinds of other split criteria all the time, don't they? &#xA;Has someone ever collected a list? &#xA;Preferably with some explanations about the use case and particular advantages (and disadvantages) of the new split criterions.&lt;/p&gt;&#xA;" OwnerUserId="28937" LastEditorUserId="28937" LastEditDate="2017-02-23T13:18:13.883" LastActivityDate="2017-02-23T13:18:13.883" Title="Measures for choosing the best Decision Tree split?" Tags="|random-forest|decision-trees|research|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17137" PostTypeId="2" ParentId="17072" CreationDate="2017-02-23T12:43:33.220" Score="3" Body="&lt;p&gt;I would add to Dan Levin's answer that when you want to justify a method, the &quot;scientific/engineering way&quot; is to first produce a &lt;strong&gt;bibliographical study&lt;/strong&gt;, where you basically prove that your approach covers an important part of what is commonly known as state-of-the-art methods. I would resume this as follow:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Look for commonly used methods that are known to be efficient for outlier detection.&lt;/li&gt;&#xA;&lt;li&gt;Summarize their applicability domains (medicine, biology, network security..) and try to link their strong points to your application in order to select some promising methods.&lt;/li&gt;&#xA;&lt;li&gt;Try the selected methods with usual validation processes inherent to machine learning problems.    &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Defining what the state-of-the-art&lt;/strong&gt; consists of a &lt;strong&gt;lot of work&lt;/strong&gt;, is absolutely &lt;strong&gt;necessary&lt;/strong&gt; and is very &lt;strong&gt;specific to your application&lt;/strong&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-02-23T12:43:33.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17138" PostTypeId="2" ParentId="17116" CreationDate="2017-02-23T12:59:10.853" Score="1" Body="&lt;p&gt;You might take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;canonical correlation analysis&lt;/a&gt;. It tries to find correlation between two sets of data. I guess you could adapt it to explore your data. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="31513" LastEditDate="2017-05-26T16:53:14.350" LastActivityDate="2017-05-26T16:53:14.350" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17139" PostTypeId="2" ParentId="17099" CreationDate="2017-02-23T13:06:56.750" Score="59" Body="&lt;p&gt;For RNNs (e.g., LSTMs and GRUs), the layer input &lt;em&gt;is&lt;/em&gt; a list of timesteps, and each timestep &lt;em&gt;is&lt;/em&gt; a feature tensor. That means that you could have a input tensor like this (in Pythonic notation):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Input tensor to RNN&#xA;[&#xA;    # Timestep 1&#xA;    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],&#xA;    # Timestep 2&#xA;    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],&#xA;    # Timestep 3&#xA;    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],&#xA;    ...&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So absolutely, you can have multiple features at each timestep. In my mind, weather is a time series feature: where I live, it happens to be a function of time. So it would be quite reasonable to encode weather information as one of your features in each timestep (with an appropriate encoding, like cloudy=0, sunny=1, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have non-time-series data, then it doesn't really make sense to pass it through the LSTM, though. Maybe the LSTM will work anyway, but even if it does, it will probably come at the cost of higher loss / lower accuracy per training time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can introduce this sort of &quot;extra&quot; information into your model outside of the LSTM by means of additional layers. You might have a data flow like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TIME_SERIES_INPUT ------&amp;gt; LSTM -------\&#xA;                                       *---&amp;gt; MERGE ---&amp;gt; [more processing]&#xA;AUXILIARY_INPUTS --&amp;gt; [do something] --/&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So you would merge your auxiliary inputs into the LSTM outputs, and continue your network from there. Now your model is simply multi-input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say that in your particular application, you only keep the last output of the LSTM output sequence. Let's say that it is a vector of length 10. You auxiliary input might be your encoded weather (so a scalar). Your merge layer could simply append the auxiliary weather information onto the end of the LSTM output vector to produce a single vector of length 11. But you don't &lt;em&gt;need&lt;/em&gt; to just keep the last LSTM output timestep: if the LSTM outputted 100 timesteps, each with a 10-vector of features, you could still tack on your auxiliary weather information, resulting in 100 timesteps, each consisting of a vector of 11 datapoints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Keras documentation on its &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;noreferrer&quot;&gt;functional API&lt;/a&gt; has a good overview of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other cases, as @horaceT points out, you may want to condition the LSTM on non-temporal data. For example, predict the weather tomorrow, given location. In this case, here are three suggestions, each with positive/negatives:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Have the first timestep contain your conditioning data, since it will effectively &quot;set&quot; the internal/hidden state of your RNN. Frankly, I would &lt;strong&gt;not&lt;/strong&gt; do this, for a bunch of reasons: your conditioning data needs to be the same shape as the rest of your features, makes it harder to create stateful RNNs (in terms of being really careful to track how you feed data into the network), the network may &quot;forget&quot; the conditioning data with enough time (e.g., long training sequences, or long prediction sequences), etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Include the data as part of the temporal data itself. So each feature vector at a particular timestep includes &quot;mostly&quot; time-series data, but then has the conditioning data appended to the end of each feature vector. Will the network learn to recognize this? Probably, but even then, you are creating a harder learning task by polluting the sequence data with non-sequential information. So I would also &lt;strong&gt;discourage&lt;/strong&gt; this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Probably the &lt;strong&gt;best&lt;/strong&gt; approach would be to directly affect the hidden state of the RNN at time zero. This is the approach taken by &lt;a href=&quot;https://arxiv.org/abs/1412.2306&quot; rel=&quot;noreferrer&quot;&gt;Karpathy and Fei-Fei&lt;/a&gt; and by &lt;a href=&quot;https://arxiv.org/abs/1411.4555&quot; rel=&quot;noreferrer&quot;&gt;Vinyals et al&lt;/a&gt;. This is how it works:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For each training sample, take your condition variables $\vec{x}$.&lt;/li&gt;&#xA;&lt;li&gt;Transform/reshape your condition variables with an affine transformation to get it into the right shape as the internal state of the RNN: $\vec{v} = \mathbf{W} \vec{x} + \vec{b}$ (these $\mathbf{W}$ and $\vec{b}$ are trainable weights). You can obtain it with a Dense layer in keras.&lt;/li&gt;&#xA;&lt;li&gt;For the very first timestep, add $\vec{v}$ to the hidden state of the RNN when calculating its value.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This approach is the most &quot;theoretically&quot; correct, since it properly conditions the RNN on your non-temporal inputs, naturally solves the shape problem, and also avoids polluting your inputs timesteps with additional, non-temporal information. The downside is that this approach often requires graph-level control of your architecture, so if you are using a higher-level abstraction like Keras, you will find it hard to implement unless you add your own layer type.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29083" LastEditorUserId="41096" LastEditDate="2017-11-05T09:29:34.667" LastActivityDate="2017-11-05T09:29:34.667" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="17140" PostTypeId="1" AcceptedAnswerId="17149" CreationDate="2017-02-23T13:28:07.500" Score="5" ViewCount="6784" Body="&lt;p&gt;Is &lt;code&gt;paragraph2vec&lt;/code&gt; the same as &lt;code&gt;Doc2vec&lt;/code&gt; or is every approach different?&lt;/p&gt;&#xA;" OwnerUserId="29314" LastEditorUserId="29575" LastEditDate="2018-04-04T14:00:57.817" LastActivityDate="2018-04-04T14:00:57.817" Title="Difference between paragraph2vec and doc2vec" Tags="|machine-learning|deep-learning|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17141" PostTypeId="2" ParentId="17096" CreationDate="2017-02-23T14:51:08.273" Score="4" Body="&lt;p&gt;In my eyes, this is not a valid approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that there is not &lt;strong&gt;one&lt;/strong&gt; unique topic model (given some parameters like the number of topics and the algorithm for topic modelling) for a corpus. Different runs with different random seeds will give you different topic models for the &lt;strong&gt;same&lt;/strong&gt; corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, any comparison comes down to a comparison of specific topic models, but not to a comparison of the corpora.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An approach with some better validity is to combine both corpora into one super-corpus, create a topic model of it, and than investigate the distribution of the topics with respect to the sub-corpora formed by the original corpora 1 and 2.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2017-02-23T14:51:08.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17142" PostTypeId="2" ParentId="14883" CreationDate="2017-02-23T15:03:39.253" Score="4" Body="&lt;p&gt;This can be easily done with &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRF&lt;/a&gt;s. You can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Inside_Outside_Beginning&quot; rel=&quot;nofollow noreferrer&quot;&gt;BIO&lt;/a&gt; encoding to tag your sentence. Then pass it to CRFs. You just have to create a few tagged sentences for training purpose as follows,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; I am Edwin.&#xA; O O  B-NAME&#xA;&#xA; You can call me Alfred&#xA; O   O    O    O B-NAME&#xA;&#xA; My name is  Edwin   thomas&#xA; O  O     O  B-NAME  I-NAME&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.chokkan.org/software/crfsuite/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRFsuite&lt;/a&gt; and &lt;a href=&quot;https://taku910.github.io/crfpp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRF++&lt;/a&gt; are some of the good implementations. CRFsuite has a python wrapper called &lt;a href=&quot;https://python-crfsuite.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pycrfsuite&lt;/a&gt;, which is quite easy to implement.&#xA;Check &lt;a href=&quot;https://github.com/scrapinghub/python-crfsuite/blob/master/examples/CoNLL%202002.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this  ipython notebook&lt;/a&gt; or &lt;a href=&quot;https://github.com/alfredfrancis/ai-chatbot-framework/blob/master/app/core/sequenceLabeler.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;this code snippet on github&lt;/a&gt; for end-to-end implementation of NER.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;check this &lt;a href=&quot;https://github.com/alfredfrancis/ai-chatbot-framework/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open source Chat bot project&lt;/a&gt; on github with NER and Intent Classification written in python. They have an easy to use training UI where you can train your bot to extract information from sentences.&lt;/p&gt;&#xA;" OwnerUserId="18665" LastActivityDate="2017-02-23T15:03:39.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17143" PostTypeId="1" AcceptedAnswerId="17144" CreationDate="2017-02-23T15:27:56.217" Score="1" ViewCount="99" Body="&lt;p&gt;I would like some pointers about the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to detect anomalies in a pretty huge collection of railway data. Or create a baseline model for detecting future anomalies. The data I have at my disposal exists out of coordinates and speed at the given coordinate (also the time of measurement). Could this perhaps be approached as a regression problem where there's a clear(?) connection between location and speed of a train. For instance a train suddenly moving at snail's pace on a track that from historical data is considered high velocity could be a potential anomaly. If this could indeed be approached in such a way would something like SVM be an option or should I look into other algorithms?&lt;/p&gt;&#xA;" OwnerUserId="29315" LastActivityDate="2017-02-23T15:39:57.913" Title="Analysis of railway data - Detecting outliers" Tags="|data-mining|regression|self-study|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17144" PostTypeId="2" ParentId="17143" CreationDate="2017-02-23T15:39:57.913" Score="0" Body="&lt;p&gt;There are a number of directions you can take, and there are a number of questions related to this topic (&lt;a href=&quot;https://datascience.stackexchange.com/questions/16930/anamoly-detection-for-transaction-data&quot;&gt;anamoly-detection-for-transaction-data&lt;/a&gt;).  One question to answer first is whether the data is seasonal or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You already suggested a valid approach.  Determine which areas are correlated to low and high speeds and than determine when a speed differs.  I think it's best to start with a heuristic approach and than try more complicated methods to try to improve.  If you want to use an SVM, a one-class SVM may work well (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn.svm.OneClassSVM&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-class SVM&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-02-23T15:39:57.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17145" PostTypeId="2" ParentId="17135" CreationDate="2017-02-23T17:33:43.077" Score="8" Body="&lt;p&gt;There are a lot of parameters which matter when using GPU's for machine learning, some of them are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;CUDA core count&lt;/li&gt;&#xA;&lt;li&gt;Memory bandwidth (GB/s)&lt;/li&gt;&#xA;&lt;li&gt;Memory per core (MB)&lt;/li&gt;&#xA;&lt;li&gt;Raw Speed (MHz)&lt;/li&gt;&#xA;&lt;li&gt;Total Memory available (GB)&lt;/li&gt;&#xA;&lt;li&gt;Performance on 16-bit, 32-bit floating ops/sec&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://timdettmers.com/2014/08/14/which-gpu-for-deep-learning/&quot; rel=&quot;noreferrer&quot; title=&quot;Tim Dettmers&quot;&gt;Tim Dettmers&lt;/a&gt; has an excellent (frequently updated) blog where he's compared different cards, near the end he's also given a simple speed-up comparison. Using that as a guide, it can be estimated that Titan X pascal would be upto 5 times faster than a GTX 970 for Deep Learning.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2017-02-24T04:24:06.857" LastActivityDate="2017-02-24T04:24:06.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17146" PostTypeId="1" CreationDate="2017-02-23T17:45:29.633" Score="7" ViewCount="2569" Body="&lt;p&gt;Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is that does the ensembling always &lt;em&gt;at least&lt;/em&gt; increase the performance &lt;strong&gt;in practice&lt;/strong&gt;? I guess mathematically, it is not true. I am jsut asking a real life situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I could train 10 base learner, and then stack them with another learner, which is at 2nd level. Does this 2nd-level learner always outperform the best of the base learners in practice? &lt;/p&gt;&#xA;" OwnerUserId="29204" LastEditorUserId="29204" LastEditDate="2017-02-23T17:50:34.337" LastActivityDate="2021-03-04T20:29:51.190" Title="Does ensemble (bagging, boosting, stacking, etc) always at least increase performance?" Tags="|ensemble-modeling|self-study|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17147" PostTypeId="1" CreationDate="2017-02-23T18:29:53.903" Score="2" ViewCount="95" Body="&lt;p&gt;Boosting is a sequential technique in which, the first algorithm is trained on the entire dataset and the subsequent algorithms are built by fitting the residuals of the first algorithm, thus giving higher weight to those observations that were poorly predicted by the previous model. Examples are adaboost and GBM, etc. My question is that how to perform boosting when ensembling the base learner? Especially how to get the residual if it is a classification problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to bagging the base learners and stacking the base learners. I just have no idea how to boosting the base learners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2018-09-21T12:45:53.223" Title="How to do boosting in model-ensembling?" Tags="|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17149" PostTypeId="2" ParentId="17140" CreationDate="2017-02-23T19:24:24.657" Score="7" Body="&lt;p&gt;There may be differing implementations, but these two terms refer to the same thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph vectors don't need to refer to paragraphs as they are traditionally laid out in text. They can theoretically be applied to phrases, sentences, paragraphs, or even larger blocks of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one definition of a paragraph vector:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An unsupervised algorithm that learns fixed-length feature&#xA;  representations from variable-length pieces of texts, such as&#xA;  sentences, paragraphs, and documents.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the full paper if you are interested: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15620" LastEditorUserId="15620" LastEditDate="2017-02-23T20:09:20.090" LastActivityDate="2017-02-23T20:09:20.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17151" PostTypeId="1" CreationDate="2017-02-24T05:08:31.977" Score="5" ViewCount="372" Body="&lt;p&gt;In GATE, default values for ANNIE are set during initialization, but sometimes based on requirements they have to be changed.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;My Requirement&lt;/strong&gt; : I want to extract English sentences without considering the &amp;quot;nextline character&amp;quot; but considering &amp;quot;full stop&amp;quot; which gives correct sentences. For that, I need to change the default value of &lt;em&gt;transducerURL&lt;/em&gt; in SentenceSplitter in ANNIE.&#xA;This can be done in two ways:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using &lt;em&gt;ANNIE_with_defaults.gapp&lt;/em&gt; - changing initparams value in Sentencesplitter and accessing from java:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; Gate.setGateHome(new File(Configuration.GATE_HOME));                        &#xA; Gate.init();&#xA; // Load ANNIE with defaults from the plug-ins folder&#xA; File pluginsHome = Gate.getPluginsHome();&#xA; File anniePlugin = new File(pluginsHome, ANNIEConstants.PLUGIN_DIR);&#xA; File annieGapp = new File(anniePlugin, ANNIEConstants.DEFAULT_FILE);&#xA; annieController = (CorpusController) PersistenceManager.loadObjectFromFile(annieGapp);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;I am not being able to find where to change its value in gapp file.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;After initialising GATE, accessing ResourceData using CreoleRegister and changing the default value of parameter &lt;em&gt;transducerURL&lt;/em&gt; as shown below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; String resourceClassName = &amp;quot;gate.creole.splitter.SentenceSplitter&amp;quot;;&#xA; ResourceData resData = Gate.getCreoleRegister().get(resourceClassName);&#xA; //System.out.println(resData.getParameterList());&#xA; ParameterList params = resData.getParameterList();&#xA; List&amp;lt;List&amp;lt;Parameter&amp;gt;&amp;gt; param =  params.getInitimeParameters();&#xA; System.out.println(param);&#xA; //System.out.println(param.get(0));&#xA; for(List&amp;lt;Parameter&amp;gt; plist : param)&#xA; {&#xA;     for(Parameter pm : plist)&#xA;     {&#xA;         System.out.println(&amp;quot;Name : &amp;quot;+pm.getName()+&amp;quot;, &amp;quot;+pm.getDefaultValue());&#xA;     }&#xA; }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;But Parameter does not have &lt;code&gt;setDefaultvalue&lt;/code&gt; method.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;How can I fix this?&lt;/p&gt;&#xA;" OwnerUserId="29335" LastEditorUserId="85045" LastEditDate="2020-11-13T13:39:01.703" LastActivityDate="2023-12-14T15:06:10.733" Title="Changing default values of ANNIE resources in GATE from Java code" Tags="|nlp|java|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17153" PostTypeId="2" ParentId="17151" CreationDate="2017-02-24T08:41:24.623" Score="0" Body="&lt;p&gt;For any of these situations the best approach is to construct the application as you need it using GATE Developer, &quot;save application state&quot; to create your own custom gapp file, then pass your new gapp to the &lt;code&gt;PersistenceManager&lt;/code&gt; instead of loading the standard &lt;code&gt;ANNIE_with_defaults&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29340" LastActivityDate="2017-02-24T08:41:24.623" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17155" PostTypeId="1" CreationDate="2017-02-24T11:13:33.760" Score="2" ViewCount="33" Body="&lt;p&gt;Your help would be appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to calculate the net returns of a series loans, at different stages of completion. I've categorised the loans into open loans (repaying &amp;amp; late loans) and closed loans (repaid &amp;amp; default).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Closed Loans, I have put the loans into a cashflow and used IRR methodology to approximate the net returns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Open Loans, I have put these loans into a similar cashflow, assuming they complete as scheduled and don't default.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Default Assumption&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to come up with a sensible default assumption to discount the open loans, based on the default performance of the previous years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tried Methodology:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Proposed Methodology enter image description here This methodology attempts to calculate the impact on net returns of defaulted loans across all years (2010-2016). We then take the average and compare against the actual default and discount the difference from the overall net returns. As the majority of loans are open on in the later years, particularly 2016, the impact of this discounting is great.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the % of closed (repaid+ defaulted) loans. In 2010 all&#xA;loans are closed so 100%. In 2016 only 7% of loans are closed. Use&#xA;this to calculate a weighting for averaging. Hint: the loans in 2010&#xA;are more influencial in the average that 2016 because all loans are&#xA;closed.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The overall net returns is calculated by a summation of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. % of repaid loans x repaid loans net returns &#xA;b. % of defaulted loans x  defaulted loans net returns (negative number)&#xA;c. % of late loans x late loans net returns &#xA;d. % of repaying loans x repaying loans net return &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Take the weighted average, using the figures calculated in 1&#xA;of the (% of defaulted loans x defaulted loans net returns. This&#xA;single figure gives an effective default rate on the overall net&#xA;returns of the total portfolio. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use this new estimate to discount the open loans by difference between the new calculated average and &lt;a href=&quot;https://i.stack.imgur.com/JOHF1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JOHF1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;the actual default.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't get my head around this calculation and whether I'm way off. Be good to get your thoughts. I've attached some data to play around with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers, Iain&lt;/p&gt;&#xA;" OwnerUserId="29346" LastActivityDate="2017-02-24T11:13:33.760" Title="Find a Default Assumption for a series of Forecasted Loans" Tags="|finance|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17156" PostTypeId="1" CreationDate="2017-02-24T18:29:38.860" Score="4" ViewCount="2714" Body="&lt;p&gt;Let's assume we have a huge database gathered which contains winning sequences, which each player played and won. (Both the players are programmed, to obtain random numbers and attempt playing the game).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What would be the best way to use these winning sequences each player&#xA;  has gathered for future plays? Are we going to program one occurrence&#xA;  to play without using historical data and few other occurrences using&#xA;  historical data?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;What would be the best way to save the winning sequences, the entire&#xA;  path which both the player One and Player Two has taken or only the&#xA;  sequence used by the winning player?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is my first attempt to machine learning paradigm and hope some support could be taken!&lt;/p&gt;&#xA;" OwnerUserId="29350" LastActivityDate="2017-03-03T01:19:26.253" Title="Building a Tic Tac Toe game which learns by itself" Tags="|machine-learning|data-mining|java|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17157" PostTypeId="1" AcceptedAnswerId="17212" CreationDate="2017-02-24T19:33:34.840" Score="20" ViewCount="16828" Body="&lt;p&gt;I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to methodically process incoming data so that it can be standardized and put to work in reports, visualizations, and analytics projects. Prior to starting here, I had very little experience with SQL, aside from writing the most basic of queries. The vast majority of my analysis prep work was all done in R. My boss insists that I improve my SQL skills, even though there seem to be very few assignments that can't be done more efficiently and with far fewer lines of code using R packages like dplyr, data.table, and tidyr (to name a few). My question is--does this make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple weeks ago, I found myself faced with the task of getting a list of column names for each row in a table that met certain criteria and concatenating them into a vector of strings. There was a tight deadline and at the time, I was experiencing some blockage and couldn't quite wrap my head around the problem. I asked my boss, who in turn asked my colleague to write a script TSQL to solve the problem. While he was working on it, I figured out a way to do it in R writing a fairly simple function and applying it over the data frame. My colleague came back with his script about two hours later. It was at least 75 lines comprising two nested for loops. I asked him to tell notify when it was finished running and he said it would take several hours. Meanwhile my R script was able to loop over the ~45,000 records in about 30 seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right to assume that R is a much better choice for cleaning and munging data?  Maybe the SQL developer in my office is just inept?  I'm curious if anyone who has worked with both R and SQL (or Python and SQL for that matter) have any thoughts on this.&lt;/p&gt;&#xA;" OwnerUserId="29353" LastActivityDate="2021-10-12T11:08:25.230" Title="Do modern R and/or Python libraries make SQL obsolete?" Tags="|python|r|data-cleaning|data|sql|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17158" PostTypeId="1" AcceptedAnswerId="17162" CreationDate="2017-02-24T20:05:21.440" Score="1" ViewCount="133" Body="&lt;p&gt;I am pretty new for time series analysis and I would like to share one of my research questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the graph:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/uqYnd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uqYnd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently, both 1 and 2 are time series data. 1 represents monthly humidity and 2 represents a morbidity rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to analyze if the morbidity rate (2) is associated with seasonal humidity (1), what method is the best in this scenario? Also, from the visualization, it seems like the morbidity peak is about 3 months after the peak of humidity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me some suggestions of this question?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any replies! &lt;/p&gt;&#xA;" OwnerUserId="28860" LastActivityDate="2017-02-25T01:10:12.607" Title="Analyzing time series association" Tags="|data-mining|statistics|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17159" PostTypeId="2" ParentId="17157" CreationDate="2017-02-24T23:18:16.020" Score="3" Body="&lt;p&gt;These aren't even comparable, really.  SQL is a language meant for accessing data, R is a language meant for working with data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SQL isn't an effective tool for munging because it it's difficult to see intermediate steps and when it throws errors, it isn't likely to address the form/quality/structure of your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My workflow is typically:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Get raw data from SQL query (in R)&lt;/li&gt;&#xA;&lt;li&gt;Build munging routine&lt;/li&gt;&#xA;&lt;li&gt;If possible, re-write SQL query to accomplish munging I accomplished in R&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also realize that not all consumers of data use R, but many still interface their platform of choice with data using SQL. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-02-24T23:18:16.020" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17160" PostTypeId="1" CreationDate="2017-02-24T23:55:57.767" Score="1" ViewCount="391" Body="&lt;p&gt;There is a &lt;code&gt;subsample&lt;/code&gt; parameter for the XGBoost (&lt;code&gt;xgb.train()&lt;/code&gt; function in R). What is the corresponding subsample parameter for lightgbm in R? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python, the &lt;a href=&quot;https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.md#io-parameters&quot; rel=&quot;nofollow noreferrer&quot;&gt;parameter&lt;/a&gt; is &lt;code&gt;bagging_fraction&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2017-02-25T01:27:47.950" Title="How to set subsample in lightgbm in R?" Tags="|r|xgboost|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17161" PostTypeId="2" ParentId="17121" CreationDate="2017-02-25T01:07:59.740" Score="3" Body="&lt;p&gt;Depends on the context it may mean differently. One common usage of the term is when not all the data points have valid/reliable values for a specific feature. For example for a feature set of $x=\{x_1, x_2, \dots, x_p, \dots, x_n\}$ not all data points may have specific feature $p$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible that some data points lack a value for feature $p$ either because of data corruption, noise or etc. In this case the percentage of data point which actually have a reliable value for this feature is called the feature coverage of feature $p$ for this specific data set.&lt;/p&gt;&#xA;" OwnerUserId="29362" LastActivityDate="2017-02-25T01:07:59.740" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17162" PostTypeId="2" ParentId="17158" CreationDate="2017-02-25T01:10:12.607" Score="1" Body="&lt;p&gt;Here's some ways what I would try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Shift humidity for &lt;code&gt;n&lt;/code&gt; days, which makes &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pearson correlation coefficient&lt;/a&gt; minimize.&lt;/li&gt;&#xA;&lt;li&gt;Extract frequencies using &lt;a href=&quot;https://en.wikipedia.org/wiki/Discrete_Fourier_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fourier Transform&lt;/a&gt; etc.&lt;/li&gt;&#xA;&lt;li&gt;Do power spectral density estimation using &lt;a href=&quot;https://en.wikipedia.org/wiki/Welch&amp;#39;s_method&quot; rel=&quot;nofollow noreferrer&quot;&gt;Welch's method&lt;/a&gt; etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-25T01:10:12.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17164" PostTypeId="1" CreationDate="2017-02-25T03:35:50.090" Score="1" ViewCount="61" Body="&lt;p&gt;I have the following code for Grid search, but it only return the accuracy result using 5 folds cross-validation. Is it possible to obtain standard deviation from the 5 folds CV. How would you do that? Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i=1:numLog2c&#xA;    log2c = log2c_list(i);&#xA;    for j=1:numLog2g&#xA;        log2g = log2g_list(j);&#xA;        cmd = ['-q -v ', int2str(nFold), ' -c ', num2str(2^log2c), ' -g ', num2str(2^log2g),' ', svmCmd];&#xA;        cv = svmtrain(trainLabel, trainData, cmd);&#xA;        cvMatrix(i,j) = cv;&#xA;&#xA;        if(cv &amp;gt;= bestcv)&#xA;            bestcv= cv; bestLog2c = log2c; bestLog2g = log2g;&#xA;        end&#xA;        fprintf('(%g, %g) %g (best c=%g, g=%g, Rate=%g)\n', log2c, log2g, cv, bestLog2c, bestLog2g, bestcv);&#xA;    end&#xA;end&#xA;&#xA;disp(['CV scale1: best log2c:',num2str(bestLog2c),' best log2g:',num2str(bestLog2g),' cv-accuracy:',num2str(bestcv),'%']);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25543" LastEditorUserId="25543" LastEditDate="2017-02-25T13:05:25.037" LastActivityDate="2017-02-25T13:05:25.037" Title="Obtain standard deviation for libsvm" Tags="|r|cross-validation|libsvm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17165" PostTypeId="1" CreationDate="2017-02-25T04:24:51.060" Score="2" ViewCount="317" Body="&lt;p&gt;I'm trying to learn machine learning with tensorflow and wrote a program that uses CNNs to determine game results for a given tic-tac-toe board. Its inputs and outputs are -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input  - An array of 9 elements that represents the board (0=empty, 1='X', 2='O')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output - An array of 3 elements ([1,0,0]=draw, [0,1,0]=player1 won, [0,0,1]=player2 won).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the tensorflow graph part of the program (it is a modified version of tensorflow tutorial at - &lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/pros&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/get_started/mnist/pros&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;baseFeatureSize = 64&#xA;x = tf.placeholder(tf.float32, shape=[None, 9])&#xA;x_image = tf.reshape(x, [-1, 3, 3, 1])&#xA;W_conv1 = nn_utils.weight_variable([2, 2, 1, baseFeatureSize])&#xA;b_conv1 = nn_utils.bias_variable([baseFeatureSize])&#xA;h_conv1 = tf.nn.relu(nn_utils.conv2d(x_image, W_conv1) + b_conv1)&#xA;W_conv2 = nn_utils.weight_variable([2, 2, baseFeatureSize, baseFeatureSize * 2])&#xA;b_conv2 = nn_utils.bias_variable([baseFeatureSize * 2])&#xA;h_conv2 = tf.nn.relu(nn_utils.conv2d(h_conv1, W_conv2) + b_conv2)&#xA;W_fc1 = nn_utils.weight_variable([3 * 3 * baseFeatureSize * 2, baseFeatureSize * 4])&#xA;b_fc1 = nn_utils.bias_variable([baseFeatureSize * 4])&#xA;h_pool2_flat = tf.reshape(h_conv2, [-1, 3 * 3 * baseFeatureSize * 2])&#xA;h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)&#xA;keep_prob = tf.placeholder(tf.float32)&#xA;h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)&#xA;W_fc2 = nn_utils.weight_variable([baseFeatureSize * 4, 3])&#xA;b_fc2 = nn_utils.bias_variable([3])&#xA;y_ = tf.placeholder(tf.float32, shape=[None, 3])&#xA;y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2&#xA;&#xA;cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))&#xA;train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)&#xA;correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&#xA;def conv2d(x, W): return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')&#xA;def max_pool_2x2(x): return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')&#xA;def weight_variable(shape):&#xA;    initial = tf.truncated_normal(shape, stddev=0.1)&#xA;    return tf.Variable(initial)&#xA;def bias_variable(shape):&#xA;    initial = tf.constant(0.1, shape=shape)&#xA;    return tf.Variable(initial)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt; - The above model works well but requires to train with over 700k inputs to get output prediction accuracy up to 80%. But the total possible permutations of the game board are less than &lt;a href=&quot;https://math.stackexchange.com/questions/269066/game-combinations-of-tic-tac-toe&quot;&gt;300K&lt;/a&gt;. If I feed just all of the unique permutations, then the accuracy worsens.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it normal for CNNs to require training data size to be much larger than all possible permutations of the inputs? I'm assuming this is not true or else they'd not be able to play chess and go etc, so what am I doing wrong? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Based on your experience, how big a training sample size would typically make sense for this setup? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29365" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2017-02-26T05:41:23.103" Title="CNN training data size for determing the winner of tic-tac-toe" Tags="|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="17166" PostTypeId="1" CreationDate="2017-02-25T06:56:39.867" Score="1" ViewCount="63" Body="&lt;p&gt;I'm using Google Cloud NLP Entity Recognition to recognize Persons/People, when I paste a chunk of text, it correctly provides PERSON &lt;a href=&quot;https://cloud.google.com/natural-language/&quot; rel=&quot;nofollow noreferrer&quot;&gt;information&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Google, headquartered in Mountain View, unveiled the new Android phone at the Consumer Electronic Show.  Sundar Pichai said in his keynote that users love their new Android phones.&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sundar Pichai&lt;/strong&gt; is correctly identified as a person, but also the word &lt;strong&gt;users&lt;/strong&gt;.&#xA;How can I differentiate real names vs words which refer to persons?&#xA;I have seen that for popular people there is metadata like Knowledge Graph mid or Wikipedia articles, but for others there is no reference, (Example: from &quot;Susan Fowler&quot; recent Uber scandal)&#xA;Any ideas/pointers will be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2017-02-25T06:56:39.867" Title="Google Cloud NLP PERSON entities" Tags="|nlp|google|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17167" PostTypeId="1" CreationDate="2017-02-25T12:51:01.773" Score="1" ViewCount="167" Body="&lt;p&gt;I am trying to train a keras CNN against the Street View House Numbers Dataset. You can find the project &lt;a href=&quot;https://github.com/asindico/digitRecognition/tree/9faa600f956e264ca61afe48d64c833f8b58711c&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. The problem is that during training neither loss nor accuracy change over time. I have tried with 1 Channel (Gray Scale) images, with RGB (3 channels) images, with wider (50,50) and smaller (28,28) images, with more or less filters in the convolutional layers, with wider and smaller patches in the pooling layers, with and without dropout, with bigger and smaller batches, with smaller and bigger learning step for the optimizers, with different optimizers, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still the training gets stuck to constant loss and accuracy&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-02-25T12:51:01.773" Title="Keras del stuck with constant loss and accuracy" Tags="|machine-learning|python|deep-learning|keras|" AnswerCount="0" CommentCount="2" ClosedDate="2017-03-02T19:05:10.263" ContentLicense="CC BY-SA 3.0" />
  <row Id="17168" PostTypeId="1" AcceptedAnswerId="17209" CreationDate="2017-02-25T13:40:25.847" Score="4" ViewCount="116" Body="&lt;p&gt;I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I automatically generate one graphic for each achievement as in the file attached &lt;a href=&quot;https://www.dropbox.com/s/z3xyxu3cnf46dzp/KPI%20R.xlsx?dl=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have many KPIs I would like to generate my graphics in a batch process&lt;/p&gt;&#xA;" OwnerUserId="29090" LastEditorUserId="29090" LastEditDate="2017-02-27T05:49:24.590" LastActivityDate="2017-02-27T07:57:43.513" Title="How to generate bulk graphics using R" Tags="|r|dataset|visualization|data|excel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17169" PostTypeId="1" AcceptedAnswerId="17170" CreationDate="2017-02-25T16:52:36.497" Score="1" ViewCount="1567" Body="&lt;p&gt;The problem I face is of a language barrier. I have a set of samples consisting of the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Sample number    Organism Count&#xA;1                5&#xA;2                8&#xA;3                5&#xA;4                7&#xA;5                13&#xA;...              ...&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to find the number of samples required to make an accurate estimate of the organism within a square meter of ground. This is for a school assignment. I don't need an answer to that. What I need to do is find the standard deviations between the set like follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;(sample 1) = Standard deviation&#xA;(sample 1, sample 2) = standard deviation&#xA;(sample 1, sample 2, sample 3) = standard deviation&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a rolling way like that. The only information I can find about rolling standard deviation is with a specific window of samples. What I need is to grow from the first one. What is this called? I'm writing a program in python to calculate and display it in a graph. I want to know the name so I can prevent reinventing the wheel in a likely less performant way (if there is a name to this specific question). It is likely that the libraries I'm working with already provide a method to calculate what I'm asking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the problem is: &#xA;I need to find the standard deviation of a 'growing set' and I want to know if that has a specific name, so I can look for it in the documentation of the python library I'm using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feel free to change the name of the question to make it better fit, since I didn't know the name I couldn't state it in the title.&lt;/p&gt;&#xA;" OwnerUserId="29376" LastEditorUserId="29376" LastEditDate="2017-02-25T17:06:30.020" LastActivityDate="2017-02-25T23:10:00.073" Title="Expanding Standard deviation" Tags="|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17170" PostTypeId="2" ParentId="17169" CreationDate="2017-02-25T18:47:20.960" Score="2" Body="&lt;p&gt;If you are using Python, you can use pandas. It comes with an expanding standard deviation function. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;# Generate some random data&#xA;df = pd.DataFrame(np.random.randn(100))&#xA;&#xA;# Calculate expanding standard deviation&#xA;exp_std = pd.expanding_std(df, min_periods=2)&#xA;&#xA;# Print results&#xA;print exp_std&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2017-02-25T18:47:20.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17171" PostTypeId="2" ParentId="16488" CreationDate="2017-02-25T19:11:28.223" Score="2" Body="&lt;p&gt;Unless you have a lot of data, I have my doubts whether training RNNs for similarity will give you significant improvements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a baseline, I would go the traditional way first and engineer some  features for each pair of text, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Number of common words in both texts (words should be stemmed)&lt;/li&gt;&#xA;&lt;li&gt;Number of words in each sentence&lt;/li&gt;&#xA;&lt;li&gt;Cosine similarity on TF-IDF vectors (unigrams, word n-grams, character n-grams&lt;/li&gt;&#xA;&lt;li&gt;Glove similarity (you can try the pretrained vectors from wikipedia)&lt;/li&gt;&#xA;&lt;li&gt;Longest common subsequence (LCS)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now you can train a binary classifier on your labeled pairs dataset. For example, start with Logistic Regression or Random Forests. Maybe you can already reach the performance you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further reference, search relevance classification is a very similar problem. You can check out the solutions from the Kaggle competitions Homedepot and Crowdflower to get more inspiration for feature engineering.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2017-02-25T19:11:28.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17172" PostTypeId="2" ParentId="16640" CreationDate="2017-02-25T19:23:47.773" Score="1" Body="&lt;p&gt;To mitigate the problem of class imbalance, you can do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Apply weighted cross entropy.&lt;/li&gt;&#xA;&lt;li&gt;When you are feeding each batch to the model, try to make sure there is no class imbalance in the batch data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Training word embeddings on such small data doesn't make sense. Try to use pre-trained word vectors like the Google news word vectors.&lt;/p&gt;&#xA;" OwnerUserId="16915" LastActivityDate="2017-02-25T19:23:47.773" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17173" PostTypeId="2" ParentId="17147" CreationDate="2017-02-25T19:29:01.920" Score="1" Body="&lt;p&gt;You can use any base learner for boosting (Adaboost requires sample weighting though). Keep in mind however that the original idea is to use weak learners with strong bias and reducing that bias through boosting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is a classification problem, usually logarithmic loss is used to calculating the residual/gradient for boosting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Python, there is a nice AdaBoost wrapper in scikit-learn (AdaBoostClassifier) which can take for example a Random Forest as base learner.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2017-02-25T19:29:01.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17174" PostTypeId="2" ParentId="15965" CreationDate="2017-02-25T19:30:25.110" Score="0" Body="&lt;p&gt;The depth and width of network are independent of each other. Depth offers generalization while width offers memorization. Also, one crucial thing is that just increasing no. of layers doesn't make it &quot;deep learning&quot; ! If you are efficiently learning feature representations (like in autoencoders) and not manually handcrafting features (like in old POS tagging techniques), you are still performing deep learning.&lt;/p&gt;&#xA;" OwnerUserId="16915" LastActivityDate="2017-02-25T19:30:25.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17175" PostTypeId="2" ParentId="16558" CreationDate="2017-02-25T19:41:53.710" Score="0" Body="&lt;p&gt;The risk of overfitting increases with the number of parameters, but there are techniques to limit overall model complexity so that the number of parameters can still be higher than the number of samples. In general they are called regularization. Implementation depends on the type of model - for neural nets, commonly used methods are weight decay, L1/L2 regularization and dropout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact most non-linear models are slightly overfitted in practice. As long as you cross-validate properly and choose the model only based on out-of-sample performance, this is not a major concern.  If your model overfits badly (train error is significantly lower than test error), increase regularization to get a potential performance gain.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2017-02-25T19:41:53.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17176" PostTypeId="2" ParentId="919" CreationDate="2017-02-25T19:47:29.907" Score="0" Body="&lt;p&gt;Unless the label distribution is balanced, stratified sampling of folds will give you a better estimate of performance than random sampling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, try to avoid that correlated samples end up in different folds. Otherwise your models are likely overfitted and the error is underestimated. For example, if your data contains temporal correlation, always split by time.&lt;/p&gt;&#xA;" OwnerUserId="12280" LastActivityDate="2017-02-25T19:47:29.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17178" PostTypeId="1" AcceptedAnswerId="17366" CreationDate="2017-02-25T21:37:01.093" Score="1" ViewCount="759" Body="&lt;p&gt;I ran the Neat Python XOR example and a neural network it found has the following properties:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Best individual in generation 22 meets fitness threshold - complexity:&#xA;  (2, 8) Number of evaluations: 3450&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Best genome: Nodes:&#xA;          NodeGene(id=0, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)&#xA;          NodeGene(id=1, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)&#xA;          NodeGene(id=2, type=OUTPUT, bias=0.0, response=4.921960812206318, activation=sigmoid)&#xA;          NodeGene(id=3, type=HIDDEN, bias=-2.340619398413843, response=4.930091408143276, activation=sigmoid)&#xA;          NodeGene(id=4, type=HIDDEN, bias=0.4674809078722945, response=4.918937894580367, activation=sigmoid) Connections:&#xA;          ConnectionGene(in=0, out=2, weight=1.0534262789427578, enabled=True, innov=0)&#xA;          ConnectionGene(in=1, out=2, weight=-1.6479324848766828, enabled=True, innov=1)&#xA;          ConnectionGene(in=0, out=3, weight=-3.5479647367966765, enabled=True, innov=2)&#xA;          ConnectionGene(in=3, out=2, weight=5.970619032226201, enabled=True, innov=3)&#xA;          ConnectionGene(in=1, out=3, weight=4.634338591600734, enabled=True, innov=4)&#xA;          ConnectionGene(in=0, out=4, weight=0.19203601809123594, enabled=True, innov=5)&#xA;          ConnectionGene(in=4, out=2, weight=-0.6588709650159562, enabled=True, innov=6)&#xA;          ConnectionGene(in=4, out=3, weight=-3.0010419817357072, enabled=True, innov=8) Node order: [4, 3]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Output: expected 0.00000 got 0.11990 &#xA;  expected 1.00000 got 1.00000&#xA;  expected 1.00000 got 0.92939 &#xA;  expected 0.00000 got 0.00395 {2} {2} {0,&#xA;  1, 3, 4} {0, 1, 2, 3, 4}&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Auxb9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Auxb9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when trying to implement the neural network in Python, I don't get the same results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import math&#xA;&#xA;def sigmoid(x):&#xA;    f = 1 / (1 + math.exp(-x))&#xA;    return f&#xA;&#xA;&#xA;node0 = 1&#xA;node1 = 1&#xA;node2 = 0&#xA;&#xA;node3_bias = -2.340619398413843&#xA;node4_bias = 0.4674809078722945&#xA;&#xA;connection_0_2 = 1.0534262789427578&#xA;connection_1_2 = -1.6479324848766828&#xA;connection_0_3 = -3.5479647367966765&#xA;connection_3_2 = 5.970619032226201&#xA;connection_1_3 = 4.634338591600734&#xA;connection_0_4 = 0.19203601809123594&#xA;connection_4_2 = -0.6588709650159562&#xA;connection_4_3 = -3.0010419817357072&#xA;&#xA;&#xA;def main():&#xA;    activated_node4 = sigmoid(node4_bias * 1 +&#xA;                              node0 * connection_0_4)&#xA;    activated_node3 = sigmoid(node3_bias * 1 +&#xA;                              activated_node4 * connection_4_3 +&#xA;                              node0 * connection_0_3 +&#xA;                              node1 * connection_1_3)&#xA;    activated_node2 = sigmoid(activated_node4 * connection_4_2 +&#xA;                              activated_node3 * connection_3_2 +&#xA;                              node1 * connection_1_2 +&#xA;                              node0 * connection_0_2)&#xA;&#xA;    print(&quot;The value is: &quot;, activated_node2)&#xA;&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;&#xA;    main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For nodes 1 and 1 I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value is:  0.30957048732254944&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which does not match the expected output above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please help me with some pointers as to what I am getting wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="19214" LastActivityDate="2017-03-05T10:05:57.120" Title="XOR Neat Python Result is incorrect?" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17179" PostTypeId="2" ParentId="17169" CreationDate="2017-02-25T23:10:00.073" Score="1" Body="&lt;p&gt;The word you might be looking for is &quot;rolling standard deviation&quot; or &quot;running standard deviation&quot;. At small scale it might be simplest to just re-compute the standard deviation of each subset. But you can compute the standard deviation of a bunch of numbers without storing all of the numbers, and there are versions that are highly numerically stable too. See for instance &lt;a href=&quot;https://www.johndcook.com/blog/standard_deviation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.johndcook.com/blog/standard_deviation/&lt;/a&gt; if you're interested in the details or implemetning it yourself, but, you should probably use a library method if possible. Java's Commons Math certainly does this.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-02-25T23:10:00.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17181" PostTypeId="1" AcceptedAnswerId="17185" CreationDate="2017-02-26T05:30:22.173" Score="2" ViewCount="716" Body="&lt;p&gt;This is inspired from my &lt;a href=&quot;https://datascience.stackexchange.com/questions/17165/cnn-training-data-size-for-determing-the-winner-of-tic-tac-toe&quot;&gt;previous question&lt;/a&gt;, comments to which made me realize that a CNN was unsuitable for the problem The CNN required over 700k training datasets while a MLP did it in in less than 50k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm trying to solve the next problem and need to figure out if a CNN makes sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN detail -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input - The board as an array of 9 elements that represents the board (0=empty, 1='X', 2='O')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Output - Recommended move as a one-hot encoded array of 9 elements. The index of 1 is the recommended move (for example, in [0,0,1,0,0,0,0,0,0] the recommended move is 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, basically the CNN will be trained with a dataset that consists of boards and the move that the winner of the game has made for each of the boards. Then during evaluation, it'll try to predict the best move for a given board.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does a convolution neural network make sense for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The convnet that I was going to use for this problem is the same as in my &lt;a href=&quot;https://datascience.stackexchange.com/questions/17165/cnn-training-data-size-for-determing-the-winner-of-tic-tac-toe&quot;&gt;previous question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29365" LastEditorUserId="29169" LastEditDate="2021-02-16T02:52:05.420" LastActivityDate="2021-02-16T02:52:05.420" Title="Would a convolution network make sense for policy based tic-tac-toe approach?" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17182" PostTypeId="2" ParentId="17165" CreationDate="2017-02-26T05:41:23.103" Score="2" Body="&lt;p&gt;I experimented with some other approaches as well and discovered that MLPs produce equivalent or better results for this problem. Thanks to the commenters Neil Slater and Student T for pointing me in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For anyone interested, this is the model that eventually worked for me &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boardSize = 3*3&#xA;featureSize = #I experimented with 500-1500 with varying results here&#xA;output = 3&#xA;x = tf.placeholder(tf.float32, [None, boardSize])&#xA;y_ = tf.placeholder(tf.float32, [None, featureSize])&#xA;learning_rate = 0.01&#xA;seed = 128&#xA;weights = {&#xA;    'hidden': tf.Variable(tf.random_normal([boardSize, featureSize], seed=seed)),&#xA;    'output': tf.Variable(tf.random_normal([featureSize, output], seed=seed))&#xA;}&#xA;biases = {&#xA;    'hidden': tf.Variable(tf.random_normal([featureSize], seed=seed)),&#xA;    'output': tf.Variable(tf.random_normal([output], seed=seed))&#xA;}&#xA;keep_prob = tf.placeholder(tf.float32)&#xA;hidden_layer = tf.add(tf.matmul(x, weights['hidden']), biases['hidden'])&#xA;hidden_layer = tf.nn.relu(hidden_layer)&#xA;hidden_layer2 = tf.add(tf.matmul(hidden_layer, tf.Variable(tf.random_normal([featureSize, featureSize], seed=seed))), &#xA;                tf.Variable(tf.random_normal([hidden_num_units], seed=seed)))&#xA;hidden_layer2 = tf.nn.relu(hidden_layer2)&#xA;&#xA;y_conv = tf.matmul(hidden_layer2, weights['output']) + biases['output']&#xA;cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_))&#xA;train_step = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)&#xA;correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29365" LastActivityDate="2017-02-26T05:41:23.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17183" PostTypeId="1" CreationDate="2017-02-26T07:03:27.583" Score="1" ViewCount="151" Body="&lt;p&gt;I am a newbie to machine learning fields. One question I always have when dealing with data problem is that whether feature selection helps the performance of prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that if you have small number of observations but relatively large number of features, it helps to select important feature before prediction. (Right?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if the sample size is really big (n&gt;&gt;p), does feature selection before prediction help? Especially in random forest, XGBoost, etc, they can &quot;ignore&quot; the features that contributes very less to the model (based on my experience, even I put small number of noise variables, the performance does not change much). Do we need to do a feature selection before these algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! Enlighten me!&lt;/p&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2017-02-26T07:03:27.583" Title="Does variable (feature) selection help machine learning performance?" Tags="|feature-selection|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17184" PostTypeId="2" ParentId="15296" CreationDate="2017-02-26T07:06:07.287" Score="1" Body="&lt;p&gt;If you want to stick to SVC(sklearn) , you can use Ensembling&#xA;we have many ways of ensembling &#xA;good guide is here : &lt;a href=&quot;http://mlwave.com/kaggle-ensembling-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mlwave.com/kaggle-ensembling-guide/&lt;/a&gt;&#xA;for your problem you can train multiple SVC models on different subset of your data,and as quick solution you can combine them whit &lt;code&gt;BaggingClassifier&lt;/code&gt; like this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = BaggingClassifier(SVC(C=1.0,&#xA;        cache_size=200,&#xA;        class_weight=None,&#xA;        coef0=0.0,&#xA;        decision_function_shape=None,&#xA;        degree=3,&#xA;        gamma='auto',&#xA;        kernel='rbf',&#xA;        max_iter=-1,&#xA;        probability=False,&#xA;        random_state=None,&#xA;        shrinking=True,&#xA;        tol=0.001,&#xA;        verbose=False,&#xA;        ))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to use each record is used only once for training in the &lt;code&gt;BaggingClassifier&lt;/code&gt;, set the &lt;code&gt;bootstrap&lt;/code&gt; parameter to &lt;code&gt;False&lt;/code&gt;.&#xA;now you can parallel each of this models in on core of your cluster.&lt;/p&gt;&#xA;" OwnerUserId="29392" LastActivityDate="2017-02-26T07:06:07.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17185" PostTypeId="2" ParentId="17181" CreationDate="2017-02-26T08:59:02.953" Score="1" Body="&lt;p&gt;This problem is not complex enough to justify a large convolutional network. However, if you are determined to use a CNN, then you could, just try to keep the architecture very simple. Just one convolutional layer (probably SAME/padded), no pooling or dropout, only a few feature maps (e.g. no more than 4, maybe just 2 will do) - and a softmax fully-connected layer for the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bear in mind that it can take more epochs, and perhaps more tuning of hyper-params, in order to get a more complex model to fit a simple problem. If you follow the plan in your earlier problem, and train against the whole population of valid states and moves, then you don't need to worry about over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should bear in mind that tic-tac-toe is simple enough that you can use tabular methods (i.e. approaches that simply enumerate and score all possible game states) to find an optimal policy. The network is being used in your case as a policy function approximation, and is a bit like using a sledgehammer to crack a nut. This makes sense if you are learning a technique used on more sophisticated grid-based games, using a toy problem. However, if your goal was more directly to learn a policy for a tic-tac-toe playing bot, then you would be better off not using any supervised learning model for the policy.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-02-26T08:59:02.953" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17186" PostTypeId="1" AcceptedAnswerId="17190" CreationDate="2017-02-26T09:52:09.630" Score="2" ViewCount="1367" Body="&lt;p&gt;I'm new to data mining and have been going through constraint-based query mining lately. I came across the concept of succinctness which basically details a constraint as succinct, if we can generate all the candidate item-sets precisely, based on an itemset satisfying the constraint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more formal definition is :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Example, &lt;code&gt;min(S.Price) &amp;lt;= v&lt;/code&gt;  is succinct &lt;br/&gt;&#xA;But, &lt;code&gt;sum(S.Price) &amp;gt;= v&lt;/code&gt;  is not succinct&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand why the former is a succinct constraint =&gt; as all the candidates can be generated by ensuring that one of the subsets satisfies that constraint.&#xA;But I fail to understand why the latter is not a succinct constraint. Any pointers on this would be helpful !&lt;/p&gt;&#xA;" OwnerUserId="29397" LastEditorUserId="18637" LastEditDate="2017-02-26T14:48:10.630" LastActivityDate="2017-02-26T14:48:10.630" Title="Why is sum a succinct constraint?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17187" PostTypeId="1" CreationDate="2017-02-26T11:00:19.497" Score="3" ViewCount="6441" Body="&lt;p&gt;I have following attributes in my data-set.&#xA;Name, Date of Birth, Annual Income, Tax Paid, Gender.&#xA;What is the type of each variable i.e. Categorical or Continuous? and why ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. I believe categorical attributes are the ones having fixed set of Values. Please let me know if I am wrong with this assumption. Also the &quot;Name&quot; attribute is just a label. but what category does it fall in ?&lt;/p&gt;&#xA;" OwnerUserId="29399" LastActivityDate="2017-02-27T11:12:39.090" Title="What is the type of each variable i.e. Categorical or Continuous?" Tags="|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17189" PostTypeId="1" CreationDate="2017-02-26T13:26:45.413" Score="1" ViewCount="729" Body="&lt;p&gt;I am trying to make a neural network to predict some values but I know my training data contains lots of &quot;bad&quot; expected outputs. That is I know some of the data would train it in the wrong direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it make sense to train the network once, then remove data that the network produces a big error for (supposing that that's the &quot;bad&quot; data) and then retrain it with the rest? Is there some other better way to approach this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sorry I'm only starting out dealing with NNs and I don't know much of the related jargon to make the question clearer.&lt;/p&gt;&#xA;" OwnerUserId="29405" LastActivityDate="2017-02-26T19:29:06.370" Title="Neural network training with known erroneous data" Tags="|neural-network|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17190" PostTypeId="2" ParentId="17186" CreationDate="2017-02-26T14:11:49.173" Score="2" Body="&lt;p&gt;We can show that the &quot;sum above threshold&quot; is not succinct by providing a counter example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you wrote the definition is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hence, we can provide a counter example by providing a set A1 satisfying the constraint while none of its subsets satisfying it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider three items, a,b and c such that each one on them costs 1.&#xA;Let the constraint C be &lt;code&gt;sum(S.Price) &amp;gt;= 3&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the set {a, b, c} the sum of prices is 3 and therefore the constraint C is satisfied.&#xA;For each of the subsets of {a, b, c} the sum of prices is lower than 3 and therefore C is not satisfied.&#xA;We found a counter example in which a set satisfies a &quot;sum above threshold&quot; while none of its subsets satisfies it. Hence, &quot;sum above threshold&quot; is not succinct.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-02-26T14:11:49.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17191" PostTypeId="1" AcceptedAnswerId="24951" CreationDate="2017-02-26T14:29:15.667" Score="4" ViewCount="1688" Body="&lt;p&gt;I would greatly appreciate if you could guide me. In fact, I used &quot;Bayesian Optimization &quot; to tune hyper-parameters  of Lasso but the estimated Lasso coefficients of almost all variables are equal to zero.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)&#xA; y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())&#xA;&#xA; ln=X.shape  &#xA;&#xA;names = [&quot;x%s&quot; % i for i in range(1,ln[1]+1)]&#xA;&#xA;# split data into train and test sets&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)&#xA;&#xA;&#xA;def Lassocv(alpha):&#xA;    return cross_val_score(Lasso(alpha=float(alpha), random_state=42),&#xA;                           X_train, y_train, scoring='neg_mean_absolute_error', cv=5).mean()&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;&#xA;    LassoBO = BayesianOptimization(Lassocv, {'alpha': (0, 8)})&#xA;    LassoBO.maximize(init_points=2, n_iter = 30)&#xA;    print('Final Results')&#xA;    print('Lasso: %f' % LassoBO.res['max']['max_val'])&#xA;    alpha=LassoBO.res['max']['max_params']['alpha']&#xA;&#xA;lasso = Lasso(alpha=alpha)&#xA;lasso.fit(X_train, y_train)&#xA;print('train Score:', lasso.score(X_train, y_train))&#xA;print('test Score:', lasso.score(X_test, y_test))&#xA;print('train MSE:', mean_squared_error(y_train, lasso.predict(X_train)))&#xA;print('test MSE:', mean_squared_error(y_test, lasso.predict(X_test)))&#xA;print('train MAE:', mean_absolute_error(y_train, lasso.predict(X_train)))&#xA;print('test MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))&#xA;&#xA;print(&quot;Features sorted by their score:&quot;)&#xA;&#xA;# A helper method for pretty-printing linear models&#xA;def pretty_print_linear(coefs, names=None, sort=False):&#xA;    if names == None:&#xA;        names = [&quot;X%s&quot; % x for x in range(len(coefs))]&#xA;    lst = zip(coefs, names)&#xA;    if sort:&#xA;        lst = sorted(lst, key=lambda x: -np.abs(x[0]))&#xA;    return &quot; + &quot;.join(&quot;%s * %s&quot; % (round(coef, 3), name)&#xA;                      for coef, name in lst)&#xA;&#xA;&#xA;print(&quot;Lasso model:&quot;, pretty_print_linear(lasso.coef_,names=names,sort=True))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Final Results&#xA;Lasso: -0.786422&#xA;train Score: 0.476819501615&#xA;test Score: 0.459836314561&#xA;train MSE: 1.10511096023&#xA;test MSE: 1.02388727356&#xA;train MAE: 0.681164633717&#xA;test MAE: 0.663119930613&#xA;Features sorted by their score:&#xA;Lasso model: 0.001 * x25 + 0.0 * x54 + 0.0 * x48 + -0.0 * x17 + 0.0 * x31 + 0.0 * x12 + -0.0 * x15 + -0.0 * x34 + 0.0 * x1 + 0.0 * x2 + -0.0 * x3 + -0.0 * x4 + 0.0 * x5 + 0.0 * x6 + 0.0 * x7 + 0.0 * x8 + 0.0 * x9 + 0.0 * x10 + 0.0 * x11 + 0.0 * x13 + -0.0 * x14 + 0.0 * x16 + 0.0 * x18 + -0.0 * x19 + 0.0 * x20 + -0.0 * x21 + 0.0 * x22 + 0.0 * x23 + -0.0 * x24 + -0.0 * x26 + 0.0 * x27 + -0.0 * x28 + -0.0 * x29 + 0.0 * x30 + 0.0 * x32 + 0.0 * x33 + 0.0 * x35 + 0.0 * x36 + -0.0 * x37 + 0.0 * x38 + 0.0 * x39 + -0.0 * x40 + -0.0 * x41 + -0.0 * x42 + 0.0 * x43 + 0.0 * x44 + 0.0 * x45 + 0.0 * x46 + -0.0 * x47 + -0.0 * x49 + -0.0 * x50 + -0.0 * x51 + 0.0 * x52 + 0.0 * x53 + 0.0 * x55 + 0.0 * x56 + 0.0 * x57 + 0.0 * x58 + 0.0 * x59 + 0.0 * x60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2017-02-26T19:37:09.857" LastActivityDate="2017-11-20T14:50:04.617" Title="Why the estimated Lasso coefficients of almost all variables are equal to zero?" Tags="|machine-learning|python|regression|scikit-learn|bayesian|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17193" PostTypeId="1" CreationDate="2017-02-26T15:05:30.823" Score="4" ViewCount="2382" Body="&lt;p&gt;I've build a CNN in Tensorflow with 2 conv layers, 1 pool layer and 2 FC layers. When I don't use dropout I get 98% accuracy on training dataset and 90% on test dataset. But, when I do use dropout, I get 62% accuracy on training dataset and 83% on test dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use 25 labels when each label has between 500-1200 samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BUILD NETWORK&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = 50&#xA;conv1_kernel_size = 3&#xA;conv1_num_kernels = 16&#xA;conv2_kernel_size = 3&#xA;conv2_num_kernels = 16&#xA;num_hidden = 64&#xA;num_channels = 1&#xA;image_size = 32&#xA;&#xA;with tf.Graph().as_default() as graph:&#xA;&#xA;  # input data&#xA;  tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size, image_size, num_channels))&#xA;  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels)) &#xA;  tf_test_dataset = tf.constant(test_dataset)&#xA;  tf_test_single_data = tf.placeholder(tf.float32, shape=(1, image_size, image_size, num_channels))&#xA;&#xA;  conv1_weights = tf.Variable(tf.truncated_normal([conv1_kernel_size, conv1_kernel_size, num_channels, conv1_num_kernels]), name='conv1_weights')  &#xA;  conv1_biases = tf.Variable(tf.zeros([conv1_num_kernels]), name='conv1_biases')&#xA;  conv2_weights = tf.Variable(tf.truncated_normal([conv2_kernel_size, conv2_kernel_size, conv1_num_kernels, conv2_num_kernels]), name='conv2_weights')  &#xA;  conv2_biases = tf.Variable(tf.constant(1.0, shape=[conv2_num_kernels]), name='conv2_biases')&#xA;  fc1_weights = tf.Variable(tf.truncated_normal([image_size // 2 * image_size // 2 * conv2_num_kernels, num_hidden], stddev=0.1), name='fc1_weights')&#xA;  fc1_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]), name='fc1_biases')&#xA;  fc2_weights = tf.Variable(tf.truncated_normal([num_hidden, num_labels], stddev=0.1), 'fc2_weights')&#xA;  fc2_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]), 'fc2_biases')     &#xA;&#xA;  keep_prob = tf.placeholder(tf.float32)&#xA;&#xA;  # model&#xA;  def model(data):&#xA;    conv1 = tf.nn.conv2d(data, conv1_weights, strides=[1, 1, 1, 1], padding='SAME')   &#xA;    conv1_hidden = tf.nn.relu(conv1 + conv1_biases)    &#xA;    conv2 = tf.nn.conv2d(conv1_hidden, conv2_weights, strides=[1, 1, 1, 1], padding='SAME')&#xA;    conv2_hidden = tf.nn.relu(conv2 + conv2_biases)    &#xA;    pool_conv2_hidden = tf.nn.max_pool(conv2_hidden, ksize=[1,2,2,1], strides=[1, 2, 2, 1], padding='SAME')        &#xA;    pool_conv2_hidden_shape = pool_conv2_hidden.get_shape().as_list()   &#xA;    fc1 = tf.reshape(pool_conv2_hidden, [pool_conv2_hidden_shape[0], pool_conv2_hidden_shape[1] * pool_conv2_hidden_shape[2] * pool_conv2_hidden_shape[3]])    &#xA;    fc1_hidden = tf.nn.relu(tf.matmul(fc1, fc1_weights) + fc1_biases)&#xA;    fc1_drop_hidden = tf.nn.dropout(fc1_hidden, keep_prob)&#xA;    fc2 = tf.matmul(fc1_drop_hidden, fc2_weights) + fc2_biases&#xA;    return fc2&#xA;&#xA;  # training computation&#xA;  logits = model(tf_train_dataset)&#xA;  loss_cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=tf_train_labels))&#xA;&#xA;  optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss_cross_entropy)&#xA;&#xA;  # predictions &#xA;  train_prediction = tf.nn.softmax(logits)&#xA;  test_prediction = tf.nn.softmax(model(tf_test_dataset))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;RUN NETWORK&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_steps = 20000&#xA;with tf.Session(graph=graph) as session:&#xA;      tf.global_variables_initializer().run()&#xA;      print('Initialized')&#xA;&#xA;      for step in range(num_steps):      &#xA;        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)    &#xA;        batch_data = train_dataset[offset:(offset + batch_size), :, :, :]&#xA;        batch_labels = train_labels[offset:(offset + batch_size), :]         &#xA;        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, keep_prob : 1.0}&#xA;        _, loss, predictions = session.run([optimizer, loss_cross_entropy, train_prediction], feed_dict=feed_dict)&#xA;&#xA;        train_acc = accuracy(predictions, batch_labels)&#xA;&#xA;        if (step % 50 == 0):&#xA;          epoch = (step * batch_size) // (train_labels.shape[0] - batch_size)      &#xA;          print('Epoch-%d - Minibatch loss at step %d: %f' % (epoch, step, loss))          &#xA;          print('Epoch-%d - Minibatch train accuracy: %.1f%%' % (epoch, train_acc))          &#xA;&#xA;      print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(feed_dict={keep_prob : 1.0}), test_labels))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28385" LastEditorUserId="28385" LastEditDate="2017-02-27T09:57:34.250" LastActivityDate="2017-02-27T09:57:34.250" Title="Why does dropout ruin my accuracy in CNN?" Tags="|neural-network|tensorflow|regularization|dropout|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17194" PostTypeId="1" CreationDate="2017-02-26T15:54:07.707" Score="2" ViewCount="326" Body="&lt;p&gt;For both item-item and user-user collaborative filtering the recommendation matrix $Γ_{m x n}$, which is an (m x n) matrix, can be defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Γ(i,j)=r_{ij}$$&#xA;For user-user collaborative recommendation r_ij is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r_{ij}= ∑_{(x∈ users)}[cosSim(x,i)*R_{xj} ]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For item-item collaborative recommendation $r_{ij}$ is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r_{ij}= ∑_{(x ∈ items)}[R_{ix}*cosSim(x,j) ]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The matrix Γ can thus be defined as: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;for user-user filtering: $Γ= S_U * R $&lt;/li&gt;&#xA;&lt;li&gt;for user-user filtering $Γ= R * S_I$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In which scenarios does the equation Γ holds true. Assuming that we need to pick the k items with highest $r_{ij}$ in order to recommend the top-k items for a user &lt;em&gt;i&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="29401" LastEditorUserId="29401" LastEditDate="2017-02-26T17:20:25.557" LastActivityDate="2017-02-26T17:20:25.557" Title="Recommendation matrix as a product of User Similarity and Ratings" Tags="|similarity|recommender-system|cosine-distance|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17196" PostTypeId="2" ParentId="17187" CreationDate="2017-02-26T16:14:30.647" Score="0" Body="&lt;p&gt;Continuous and discrete data are types of numerical variables, in the sense that one can perform mathematical operations on them ( for example things like height, weight, income, etc.). On the other hand categorical data types are such that fall under a certain category but are not of a mathematical nature (i.e. mathematical operations on such vars make little sense). &#xA;In your example, income and tax paid are numbers, they are continuous, but name, gender and DOB would be categorical. &lt;/p&gt;&#xA;" OwnerUserId="29408" LastActivityDate="2017-02-26T16:14:30.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17198" PostTypeId="2" ParentId="17189" CreationDate="2017-02-26T19:15:14.553" Score="1" Body="&lt;p&gt;If you want to identify and remove problematic data, then this is much better done &lt;em&gt;before&lt;/em&gt; training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might have some luck with your approach, but you have no guarantees that the neural network will help you isolate the problem entries based on error values, if it has been trained on the &quot;good&quot; and &quot;bad&quot; entries at the same time. It depends on the nature of the problem data. You take a significant risk that the model will fit to the problem data well enough to cause you to reject good data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should instead try to think of a way to identify the bad data more directly, and prior to training for the predictive task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's one idea: Are you able to identify enough good and bad data manually - enough to train and test a classifier? Then label some of the data &quot;good&quot; or &quot;bad&quot;, train a classifier and test it - the test should be on some held out labeled values, to help you assess accuracy. If your classifier has a good accuracy, you can have some confidence to use it to filter the remainder of your data, and only use the &quot;good&quot; classified data for your original training goal.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-26T19:29:06.370" LastActivityDate="2017-02-26T19:29:06.370" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17199" PostTypeId="2" ParentId="14991" CreationDate="2017-02-26T19:30:13.107" Score="1" Body="&lt;p&gt;So You have 750 : 50 ratio of positive and negative class. Try running LR on this data. Class imbalance would not be a problem. Even if so, you under sample the positive class and then re-calibrate the probabilities after model execution.&lt;br&gt;&#xA;I would also suggest you to be active on cross-validation stack-exchange community. There are similar questions answered. &lt;/p&gt;&#xA;" OwnerUserId="29412" LastActivityDate="2017-02-26T19:30:13.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17200" PostTypeId="2" ParentId="16991" CreationDate="2017-02-26T19:35:37.757" Score="0" Body="&lt;p&gt;Try all possible combinations of interaction terms. Have you checked co-linearity of variables? Have you checked all variables interaction? Why do you stick to LR? Try Random Forest also. See what gives u best accuracy on k-fold validation.&lt;/p&gt;&#xA;" OwnerUserId="29412" LastActivityDate="2017-02-26T19:35:37.757" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17202" PostTypeId="1" AcceptedAnswerId="27678" CreationDate="2017-02-26T19:57:58.250" Score="2" ViewCount="223" Body="&lt;p&gt;I trained an XGBoost regressor, and I'm looking for a way to interpret new instances that the model will receive in production.&#xA;There's a nice solution for classification models (&lt;a href=&quot;https://github.com/marcotcr/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lime&lt;/a&gt;), but it doesn't offer explanations to regression problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of an existing &quot;explainer&quot; for regressions?&lt;/p&gt;&#xA;" OwnerUserId="29415" LastActivityDate="2018-05-11T16:54:38.130" Title="Explaining instances of a machine learning regressor" Tags="|machine-learning|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17203" PostTypeId="2" ParentId="16535" CreationDate="2017-02-26T21:53:19.160" Score="1" Body="&lt;p&gt;I'm starting to look at this now, to start with simply on batched/CSV export/import process pushing data into Google BigQuery first and using it as a source in DataStudio. I've thought about how to automate and I'm not sure Zapier will cut it - it doesn't seem to have an action for Contact 'updates' for one, which is a showstopper for us (though you don't mention what objects you want to query).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be to use the &lt;a href=&quot;https://pages.novaksolutions.com/data-warehouse/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Novak Solutions Data Warehouse&lt;/a&gt; service to get everything into SQL and either query directly from DataStudio or replicate into BigQuery using &lt;a href=&quot;https://www.stitchdata.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;StitchData&lt;/a&gt; or similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a cheaper option still, it could be done via the Infusionsoft API, but you'd have the time and cost of creating the synchronisation tasks and managing changes. If you have a limited subset of objects (eg. Contacts + Tags) this might be cost effective.&lt;/p&gt;&#xA;" OwnerUserId="29419" LastActivityDate="2017-02-26T21:53:19.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17204" PostTypeId="1" CreationDate="2017-02-26T23:33:42.580" Score="2" ViewCount="48" Body="&lt;p&gt;Here is a simple example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a 3D space, if point A is the geocenter of a planet, point B is its north pole, and point C has a fixed latitude/longitude on the planet surface. Then the position of point C can be inferred from the position of A and B (using geodetic projection).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assuming that I have thousands of points in the space, their incomplete relations are in a big multipgraph. I would like to deduce the position of a point that is initially unknown. What is the best algorithm to search search through all its relations and try to deduce it?&lt;/p&gt;&#xA;" OwnerUserId="29421" LastActivityDate="2017-02-26T23:33:42.580" Title="What is the best algorithm for deterministic belief propagation?" Tags="|graphs|graphical-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17205" PostTypeId="1" CreationDate="2017-02-27T00:42:42.743" Score="1" ViewCount="507" Body="&lt;p&gt;This is a bit of a confusion I have and I never saw any explanation of it, so I have to ask the question. Whenever I see the discussion on how a ConvNNs extracts features, I see something like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ydp7Y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ydp7Y.png&quot; alt=&quot;Layers&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the discussion that follows is that features are increasingly (hierarchically) more complicated and meaningful, as becomes obvious from the example here. Then, I extracted some layers from a ConvNN (CRF as RNN).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 64 feature maps in conv1_1, 256 in conv3_1 and 512 in conv5_1. &#xA;So my first problem is that I can't relate the results I have to the example above. I fail to see how the features become more complicated. All I see is that the image becomes increasingly less distinct. The final output (pixelwise prediction) is very good, so it must work. But I totally don't understand how: all I see is that visible features are increasingly harder to distinguish as feature map size reduces from 1280x768 to 75x50.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the original image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A02vn.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A02vn.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conv1_1&#xA;&lt;a href=&quot;https://i.stack.imgur.com/k7Ufq.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k7Ufq.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conv3_1&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HCEOm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HCEOm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conv5_1&#xA;&lt;a href=&quot;https://i.stack.imgur.com/EprS7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EprS7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="29169" LastEditDate="2021-02-16T02:51:55.470" LastActivityDate="2021-02-16T02:51:55.470" Title="Feature extraction in ConvNNs - layers progression" Tags="|convolutional-neural-network|feature-extraction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17206" PostTypeId="2" ParentId="17193" CreationDate="2017-02-27T03:03:01.283" Score="7" Body="&lt;p&gt;As your network is working without dropout, I think your problem is about how many epoches you run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your code, it seems that only one epoch will be run. With dropout enabled, each neuron has 50% percent (for example) chance to be activated. Maybe there are some un-trained neurons in your network, which ruin your accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is worth trying more epoches. In my experience, 100 epoches is always a good start.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-27T03:03:01.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17207" PostTypeId="1" AcceptedAnswerId="17208" CreationDate="2017-02-27T07:00:12.147" Score="3" ViewCount="2336" Body="&lt;p&gt;I was training a binary classifier using XGBClassifier (basically boosted decision trees if I understand it correctly). I have 10K training examples. I have two distinct set of features (but they could be dependent), one contains 26 features (call the set A) and the other contains 96 features (call the set B). I tried to train 3 classifiers, each w/ a different combination of the feature sets, namely A, B and A+B. The result is that using only A is clearly better than using both A and B. At this point I thought it might be overfitting, so that using less number of features actually avoids the overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The # of trees used in the above trainings was 100. So I used 10-fold cross validation to find the optimal # of trees for each feature set combination, and they are all beyond 100 (like 300-500). So it seems to me the models were learning w/ less-than-desired degree of freedom and they were underfitting if you will.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this confuses me: why providing additional features make things worse (i.e. using only A is better than using both A and B) when the models are on the under-fitting side (as opposed to overfitting)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe a more general question: how do I find out the real problem in this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I have read &lt;a href=&quot;https://datascience.stackexchange.com/questions/361/when-is-a-model-underfitted&quot;&gt;this question about overfitting/underfitting&lt;/a&gt; and I still think my models should be underfitting, because apparently increasing model complexity helps (e.g. from 100 trees to 300-500 trees).&lt;/p&gt;&#xA;" OwnerUserId="29428" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-02-27T07:31:02.183" Title="More features hurts when underfitting?" Tags="|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17208" PostTypeId="2" ParentId="17207" CreationDate="2017-02-27T07:31:02.183" Score="3" Body="&lt;p&gt;Theoretically speaking, more data leads to better model. However, in practice, more features often leads to the difficulty of model training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming there are 30 &quot;main&quot; features of your dataset. Feature set A contains 20 &quot;main&quot; features, so it might be easy (20 out of 26) for one &quot;main&quot; feature to be &quot;chosen&quot; and &quot;trained&quot;, under certain hyperparameters (in your case, 100 trees). When it comes to feature set B, which contains all &quot;main&quot; features, it's hard (30 out of 96) for one &quot;main&quot; feature to be chosen, and it's harder when there's only 100 trees (cause there are 66 &quot;minor&quot; features which should not be trained, relatively). That's what we called &quot;under-fitting&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to your question, when the model is under-fitting, if we're lucky that model trained on feature set B (namely model B) contains all these 30 &quot;main&quot; features, the model will be good, and we might not able to find out it's under-fitting. But in most cases, we're not that lucky, model can be ruined with all 66 &quot;minor&quot; features, relatively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my ML practice, I'll try more training iterations when more samples come, and try a more complex model when more features come, with the control of over-fitting/under-fitting.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-27T07:31:02.183" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17209" PostTypeId="2" ParentId="17168" CreationDate="2017-02-27T07:57:43.513" Score="3" Body="&lt;p&gt;There are multiple ways you can achieve this. I guess the easiest would be to create a function that saves a bar chart for your KPI to a file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;save_KPI_plot &amp;lt;- function(fn, kpi_data) {&#xA;    png(paste0(fn, &quot;.png&quot;))&#xA;    # Plot code&#xA;    dev.off()&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can call the function either as follows,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;save_KPI_plot(&quot;kpi_1&quot;, kpi_1_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or store your file names in one list and your data in another and loop over both:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kpi_fns &amp;lt;- list(&quot;filename_1&quot;, &quot;filename_2&quot;)&#xA;kpi_data &amp;lt;- list(kpi_1_data, kpi_2_data)&#xA;for (i in seq_along(kpi_fns)) {&#xA;    save_KPI_plot(kpi_fns[i], kpi_data[i])&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you prefer to have a different image format you can change &lt;code&gt;png()&lt;/code&gt; to &lt;code&gt;bmp()&lt;/code&gt; or &lt;code&gt;jpeg()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2017-02-27T07:57:43.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17210" PostTypeId="2" ParentId="16959" CreationDate="2017-02-27T08:32:25.150" Score="1" Body="&lt;p&gt;The way to go was to use relative similarity metric:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation*}&#xA;d(x,y)=\left(1-\frac{\left\vert{x \cap y}\right\vert }{\left\vert{x \cup y}\right\vert }\right)^k&#xA;\end{equation*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; are sets containing hashes identifying features and &lt;code&gt;k&lt;/code&gt; is experimentally selected factor that allows &quot;spreading&quot; the range of distances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that data point, being really a binary vector of size &lt;code&gt;2^32&lt;/code&gt; have bits set only on the positions denoted by the hashes, we can calculate the distance as size of intersection of hash sets divided by size of sum of hash sets.&lt;/p&gt;&#xA;" OwnerUserId="28938" LastActivityDate="2017-02-27T08:32:25.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17212" PostTypeId="2" ParentId="17157" CreationDate="2017-02-27T08:48:56.083" Score="18" Body="&lt;p&gt;R and SQL are two completely different beasts. SQL is a language that you can use to query data that is stored in databases as you already experienced. The benefits of SQL versus R lays mostly in the fact of the database server (MS SQL, Oracle, PostgreSQL, MySQL, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most, if not all, modern database servers permit multiple users to query data from the same data source and insert, update and delete data in the same tables all while ensuring that the data remains consistent. This is essential for say recording a bank transaction. Can you imagine running a bank on R? That's where database servers come in. They ensure ACID properties of procedures run on the database. ACID stands for Atomicity, concurrency, isolation and durability (see &lt;a href=&quot;https://en.wikipedia.org/wiki/ACID&quot; rel=&quot;noreferrer&quot;&gt;ACID description on wikipedia&lt;/a&gt;). R is a single user platform where everything happens in memory. So, if your computer stops working halfway in a big operation, your data will not be stored. You are also the only person who can access the data. To be clear, R is not considered an alternative for database servers and/or SQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another main advantage of database servers is that a good database design will ensure that you can query your database fast by performing query optimization. To achieve this database servers keep track of the design of a table. See for a full discussion of this topic the &lt;a href=&quot;https://en.wikipedia.org/wiki/Query_optimization&quot; rel=&quot;noreferrer&quot;&gt;wiki page&lt;/a&gt;. R cannot perform query optimization. Poor database design,  can lead to slow execution of your queries. Database servers can also perform optimization over queries that query multiple tables if foreign keys are properly used in the database design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SQL language has a very different syntax and I share your experience that it is shorter to write data munging steps using the data table or dplyr syntax. However, sometimes your data is too big for R or you need to store the results in the database as part of a periodic batch job, which will require to code your logic in SQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience there are particular use cases for SQL and R/Python. SQL is great for storing business critical data and for allowing multiple people to access, modify, insert and delete data in a centralized environment. For any one-off data munging R and Python are great. If your data munging needs to be periodically executed, you will need to port your R/Python script to SQL.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="44990" LastEditDate="2018-11-07T14:07:55.357" LastActivityDate="2018-11-07T14:07:55.357" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17213" PostTypeId="1" AcceptedAnswerId="17214" CreationDate="2017-02-27T09:08:46.150" Score="0" ViewCount="2575" Body="&lt;p&gt;how i use Orange3 package for association rule mining ,in Orange 2.7 version Orange.associate.AssociationRulesSparseInducer() method is present but its not available in Orange3 ..&lt;/p&gt;&#xA;" OwnerUserId="29431" LastActivityDate="2017-02-27T09:23:22.090" Title="Association rule mining and Orange3 package Python" Tags="|python|" AnswerCount="1" CommentCount="0" ClosedDate="2017-02-27T19:59:41.710" ContentLicense="CC BY-SA 3.0" />
  <row Id="17214" PostTypeId="2" ParentId="17213" CreationDate="2017-02-27T09:23:22.090" Score="0" Body="&lt;p&gt;In Orange 3, &lt;code&gt;AssociationRulesSparseInducer&lt;/code&gt; is removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://orange3-associate.readthedocs.io/en/latest/scripting.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; of Orange-Associate may help you on your task.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/biolab/orange3-associate/issues/4&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/biolab/orange3-associate/issues/4&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/39667272/orange-data-mining-version-3-3-python-association-rules&quot;&gt;https://stackoverflow.com/questions/39667272/orange-data-mining-version-3-3-python-association-rules&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-27T09:23:22.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17215" PostTypeId="2" ParentId="11868" CreationDate="2017-02-27T10:06:49.840" Score="1" Body="&lt;p&gt;There are several changes between current version of &lt;a href=&quot;https://github.com/fchollet/keras/blob/309f586424ea229fa9f6fa963e9345b4dbd22849/examples/cifar10_cnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;cifar10_cnn.py&lt;/a&gt; and the code in your question. However, both version is not that bad (10% accuracy) in my environment.&lt;/p&gt;&#xA;&lt;p&gt;Result of your code:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;X_train shape: (50000, 32, 32, 3)&lt;/p&gt;&#xA;&lt;p&gt;50000 train samples&lt;/p&gt;&#xA;&lt;p&gt;10000 test samples&lt;/p&gt;&#xA;&lt;p&gt;Not using data augmentation.&lt;/p&gt;&#xA;&lt;p&gt;Train on 50000 samples, validate on 10000 samples&lt;/p&gt;&#xA;&lt;p&gt;Epoch 1/200&lt;/p&gt;&#xA;&lt;p&gt;50000/50000 [==============================] - 21s - loss: 1.7470 - acc: 0.3528 - val_loss: 1.3405 - val_acc: 0.5222&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;Epoch 20/200&lt;/p&gt;&#xA;&lt;p&gt;50000/50000 [==============================] - 20s - loss: 0.6308 - acc: 0.7800 - val_loss: 0.7183 - val_acc: 0.7588&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Result of &lt;a href=&quot;https://github.com/fchollet/keras/blob/309f586424ea229fa9f6fa963e9345b4dbd22849/examples/cifar10_cnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;cifar10_cnn.py&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;X_train shape: (50000, 32, 32, 3)&lt;/p&gt;&#xA;&lt;p&gt;50000 train samples&lt;/p&gt;&#xA;&lt;p&gt;10000 test samples&lt;/p&gt;&#xA;&lt;p&gt;Using real-time data augmentation.&lt;/p&gt;&#xA;&lt;p&gt;Epoch 1/200&lt;/p&gt;&#xA;&lt;p&gt;50000/50000 [==============================] - 26s - loss: 1.5978 - acc: 0.4226 - val_loss: 1.1876 - val_acc: 0.5712&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;Epoch 20/200&lt;/p&gt;&#xA;&lt;p&gt;50000/50000 [==============================] - 24s - loss: 1.7479 - acc: 0.4229 - val_loss: 1.6603 - val_acc: 0.4292&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-02-27T10:06:49.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17216" PostTypeId="1" CreationDate="2017-02-27T10:45:11.683" Score="4" ViewCount="12065" Body="&lt;p&gt;If I applied PCA on feature vectors and then I do clustering, such like following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reduced_data = PCA(n_components=2).fit_transform(data)&#xA;kmeans = KMeans(init='k-means++', n_clusters=n_digits, n_init=10)&#xA;kmeans.fit(reduced_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The reduced data will be the in terms of PCA components, so after&#xA;clustering in kmean, you can get a label for each point&#xA;(reduced_data), how to know which one from the origin data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;how to play with a number of PCA components regarding the number of&#xA;clusters? Thanks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29434" LastActivityDate="2017-04-08T04:00:01.630" Title="PCA before K-mean clustering" Tags="|clustering|feature-selection|scikit-learn|k-means|pca|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17217" PostTypeId="2" ParentId="17187" CreationDate="2017-02-27T11:12:39.090" Score="1" Body="&lt;p&gt;Name attribute cant be continuous, if you really want to use it then take it as a categorical. It depends on what is your use case.&#xA;Gender is categorical. &#xA;Date of Birth/Annual Income/Tax Paid are in general continuous variable. But you may change them in categorical depending on your use case.  &lt;/p&gt;&#xA;" OwnerUserId="29435" LastActivityDate="2017-02-27T11:12:39.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17218" PostTypeId="1" CreationDate="2017-02-27T11:20:26.800" Score="2" ViewCount="510" Body="&lt;p&gt;&lt;strong&gt;Data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training set consists of ~450k obs and 26 variables, out of which 1 is an ordinal factor (order_month, 12 levels) and the rest is numerical. Moreover, some of my predictors are highly correlated (Pearson's &gt; 0.5), which is expected and sometimes intentional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained a good RF model (ROC 0.96, ~90% specificity &amp;amp; ~90% sensitivity when tested on unseen data) and I derived Variable importance for it using &lt;code&gt;varImp()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I know that having factorial and / or correlated predictors could result in biased variable selection in RF models, to which conditional inference trees (CIT) should be a remedy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to compare results of &lt;code&gt;randomForest::varImp()&lt;/code&gt; with &lt;code&gt;party::varimp(myforest, conditional=TRUE)&lt;/code&gt; however I get an error in the latter:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Error in model.matrix.default(as.formula(f), data = blocks) : &#xA;  term 1 would require 9e+12 columns&lt;/code&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suggesting that I have either too many variables or variables with too many levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On top of that it takes over 24hrs to run &lt;code&gt;party::cforest()&lt;/code&gt; and running &lt;code&gt;predict()&lt;/code&gt; on CIT model crashes my computer altogether. Given the above, I have a couple of&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) what are alternative ways of obtaining conditional variable importance if not through &lt;code&gt;party::varimp()&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Big thanks for your suggestions! Kasia&lt;/p&gt;&#xA;" OwnerUserId="28291" LastActivityDate="2017-02-27T11:20:26.800" Title="randomForest::varImp VS conditional variable importance" Tags="|r|predictive-modeling|feature-selection|random-forest|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17219" PostTypeId="1" AcceptedAnswerId="17221" CreationDate="2017-02-27T11:21:46.193" Score="5" ViewCount="25581" Body="&lt;p&gt;I am trying to do binary classification of News Articles (Sports/Non-Sports) using recurrent neural net in tensorflow. The training data is highly skewed [Sports:Non-Sports::1:9].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using cross-entropy as my cost function, which treats both classes equally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the ways by which user can penalise one class? Or is there any other cost function suitable for this purpose?&lt;/p&gt;&#xA;" OwnerUserId="28475" LastActivityDate="2017-02-27T12:01:36.977" Title="How to do imbalanced classification in deep learning (tensorflow, RNN)?" Tags="|classification|deep-learning|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17220" PostTypeId="1" CreationDate="2017-02-27T11:53:32.373" Score="1" ViewCount="880" Body="&lt;p&gt;I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. E.g. would this guy, non-customer, of this age and salary, be interested in product X?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train the model, I have a basin of 100K company customers, of which only 5K bought product X - the remaining 95K bought other products. Any prediction model guesses 'nobody will buy product X' accepting those ~5% false negatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I compensate for this skewness of the data? i.e. 95% vs. 5%? Thanks&lt;/p&gt;&#xA;" OwnerUserId="29436" LastEditorUserId="8501" LastEditDate="2017-02-28T11:44:22.783" LastActivityDate="2017-03-30T12:06:36.333" Title="How to compensate for class imbalance in prediction model?" Tags="|predictive-modeling|dataset|class-imbalance|prediction|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17221" PostTypeId="2" ParentId="17219" CreationDate="2017-02-27T12:01:36.977" Score="4" Body="&lt;p&gt;This has already been answered both in &lt;a href=&quot;https://stackoverflow.com/questions/35155655/loss-function-for-class-imbalanced-binary-classifier-in-tensor-flow&quot;&gt;stackoverflow&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/questions/197273/class-balancing-in-deep-neural-network&quot;&gt;crossvalidated&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The suggestion in both cases was to add class weights to the loss function, by multiplying logits:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;loss(x, class) = weights[class] * (-x[class] + log(\sum_j exp(x[j])))&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example, in tensorflow you could do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;ratio = 31.0 / (500.0 + 31.0)&#xA;class_weight = tf.constant([ratio, 1.0 - ratio])&#xA;logits = ... # shape [batch_size, 2]&#xA;weighted_logits = tf.mul(logits, class_weight) # shape [batch_size, 2]&#xA;xent = tf.nn.softmax_cross_entropy_with_logits(&#xA;  weighted_logits, labels, name=&quot;xent_raw&quot;)&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-02-27T12:01:36.977" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17222" PostTypeId="2" ParentId="17216" CreationDate="2017-02-27T12:23:19.863" Score="1" Body="&lt;p&gt;I haven't used sklearn but I'll try to answer the question to the best of my ability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you have an $n \times f$ matrix (data), now reduced to $n\times d$ (reduced data). Say the 1st reduced data point was mapped to some cluster, wouldn't that correspond to the first data point in the reduced matrix being classified to that same cluster? (Each row denotes a data point) Basically, the ordering will not change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you want to look at it more mathematically, assume $W=(w1,w2...wd)$ form the basis vectors of the subspace you want to map your feature vectors to (the first d eigenvectors of your data covariance matrix). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now every data point can be represented as a linear combination of these basis vectors:&#xA;$x = \alpha1w1 + \alpha2w2 ... + \alpha dwd$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the vector $\alpha = (\alpha1, \alpha2..., \alpha d)$ is the representation of your data point in d dimensions, i.e. in the subspace, and your original data point is:&#xA;$x = W^T\alpha$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not so clear about what you are asking in part 2). You choose the number of components you want to keep in PCA depending on the amount of error you are willing to tolerate.&#xA;Suppose you want the reduced data points to correspond to 95% of the variance, you select the first d eigenvectors such that their eigenvalues correspond to the above percentage of variance.&#xA;$\frac {\sum\limits_{i=1}^d \lambda_i}{\sum\limits_{i=1}^n \lambda_i} \times 100 = 95\% $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helped!&lt;/p&gt;&#xA;" OwnerUserId="21749" LastEditorUserId="21749" LastEditDate="2017-03-11T07:59:04.073" LastActivityDate="2017-03-11T07:59:04.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17223" PostTypeId="1" AcceptedAnswerId="17228" CreationDate="2017-02-27T12:27:16.310" Score="1" ViewCount="73" Body="&lt;p&gt;Hi guys I'm very new to data science,&#xA;I have intermediate background on programming and have used Pentaho Data Integration tool once for DB migration &amp;amp; data cleansing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have this kind of data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;item_details, timestamp&#xA;Wooden chairs, 01-07-2017&#xA;Plastic chairs, 02-07-2017&#xA;Stainless table, 11-07-2017&#xA;Decorated window, 12-07-2017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know based on monthly time frame  what are the top trending items in that month. &#xA;Let's say in January the top 3 item is:&#xA;1. Table&#xA;2. Chairs&#xA;3. Window&#xA;In February :&#xA;1. Door&#xA;2. Chair&#xA;3. Cupboard&#xA;.. And so on&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achive this and using what kind of tool? (preferably free or open source tools, can be GUI based or script library, having a visualization or  dashboard is a plus) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help. Sorry for noob questions&lt;/p&gt;&#xA;" OwnerUserId="29438" LastEditorUserId="28628" LastEditDate="2017-02-27T14:35:47.050" LastActivityDate="2017-02-27T14:39:42.737" Title="How to extract most occuring words based on month &amp; what tool to use?" Tags="|tools|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17224" PostTypeId="1" CreationDate="2017-02-27T13:04:18.407" Score="1" ViewCount="24" Body="&lt;p&gt;Consider a matrix, $\mathbf{V}$, where each row corresponds to one of $m$ electoral district and each column corresponds to one of $n$ candidates or political parties. Each element, $V_{ij}$, is the proportion of votes cast in district $i$ for party $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now imagine taking leave of your senses and deciding to carry out a low-rank matrix factorisation of $\mathbf{V}$. It turns out that regular non-negative matrix factorisation minimising generalised KL-Divergence will preserve the row and column sums in the approximation&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0024379507001115&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, so where the rank of your factorisation, $k$, is less than $n$, it's straightforward to find $k$ latent variables for each electoral district that describe contributions to the share of each party's vote in that district.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Are there any obvious reasons I shouldn't be doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is there a sensible interpretation of these $k$ variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Is there a more canonical method for factorising matrices like this? Or a more canonical method for finding latent variables for data like this?&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-02-27T13:04:18.407" Title="Matrix factorisation of vote share data" Tags="|matrix-factorisation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17225" PostTypeId="1" CreationDate="2017-02-27T13:04:20.537" Score="1" ViewCount="65" Body="&lt;p&gt;I am already aware of usage of information retrieval for search engine. But still trying to understand what can be possible life-cycle phases for a search engine. I mean for a search engine is it only about build a search engine, deploy and occasional maintenance? I assume so. Perhaps at best augment it with some deep learning or NLP algorithm. Is my assumption accurate or there can be more to it for the search engine life cycle, purely from information retrieval perspective?&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2017-02-27T13:04:20.537" Title="What is search engine life cycle from information retrieval perspective" Tags="|machine-learning|deep-learning|nlp|search-engine|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17227" PostTypeId="1" CreationDate="2017-02-27T14:31:24.897" Score="3" ViewCount="2808" Body="&lt;p&gt;I'm trying to perform clustering of the data to improve the efficiency of brute-force kNN . The dataset consists of objects described by a large set of binary features, each identified by a 32-bit hash code. Data point can be understood as a &lt;code&gt;2^32&lt;/code&gt; elements long very sparse binary vector, with bits set to 1 on the positions denoted by hash code of feature. To simplify, each of the data points is being represented as an array of hashes - if we know which of the bits are set to 1, we know which of them are set to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a working distance function (mentioned in &lt;a href=&quot;https://datascience.stackexchange.com/questions/16959/memory-efficient-metric-calculation-for-ultra-high-dimensional-data&quot;&gt;here&lt;/a&gt;) but have difficulties to cluster the dataset in reasonable time.  Because of the binary nature of the data it is impossible to create any kind of mean value based on a collection of datapoints, so k-Centroids is not an option. I tried k-Clustroids but it doesn't converge, hierarchical approaches are too inefficient. Do you happen to know any clustering algorithm that would efficiently handle fixed size dataset, with custom metric calculation method without the need to create any temporary, centroid datapoints?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks.&lt;/p&gt;&#xA;" OwnerUserId="28938" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-07-29T18:23:40.467" Title="Efficient clustering of sparse binary vectors" Tags="|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17228" PostTypeId="2" ParentId="17223" CreationDate="2017-02-27T14:39:42.737" Score="0" Body="&lt;p&gt;If you are new to data science and data munging, this could be kind of a tricky task, but a good one to get your feet wet.  Many programming languages have the capability to do this (R, Python, Matlab, etc.).  I use R primarily, so I'll give you a brief heuristic for how I'd approach the task in R.  Perhaps looking into these steps will get you started.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Install R&lt;/li&gt;&#xA;&lt;li&gt;Install some packages that will help you along ('tm' for text mining,'dplyr' for cleaning/organizing your data, and perhaps also 'lubridate' for working with dates/times)&lt;/li&gt;&#xA;&lt;li&gt;Read in your data from your source, be it a text file, spreadsheet, or some database (if a database, you'll have to conquer connecting R to said database too)&lt;/li&gt;&#xA;&lt;li&gt;You want to do a word frequency analysis for each month. How you accomplish this will depend on how the data is organized, which I do not know, but it would involve first rounding all your dates to month (using lubridate's 'floor_date()' function is one way), then parsing the text for each month into a corpus that can be analyzed (using package tm).&lt;/li&gt;&#xA;&lt;li&gt;Finally, for each month I would make a table counting words, sorting by frequency.  That would give you, for each month, the top 'trending' words. To discount words like 'the' and 'a', I might also use some of the tools in the 'tm' package to clean things up.&lt;/li&gt;&#xA;&lt;li&gt;Note that in #5 I said 'words', not 'terms'.  If you want to account for terms consisting of &gt; 1 words, you'll have to 'tokenize' them, but that's beyond the scope of this very brief intro.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As with many data science tasks, there are many ways to attack this; the above is but one of many possibilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-02-27T14:39:42.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17229" PostTypeId="2" ParentId="17220" CreationDate="2017-02-27T14:49:57.703" Score="0" Body="&lt;p&gt;You can try to set different weights on each samples. For example, you can set 0.05 weight on those who bought other products, and 0.95 weight on those who bought product X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://scikit-learn.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;, there's a &lt;code&gt;sample_weight&lt;/code&gt; parameter which provides this method. In R, checkout this &lt;a href=&quot;http://dpmartin42.github.io/blogposts/r/imbalanced-classes-part-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;guide&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost&lt;/a&gt;, here is a &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html#handle-imbalanced-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;guide&lt;/a&gt; which shows how to deal with imbalanced class in XGBoost.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-02-27T14:49:57.703" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17230" PostTypeId="1" CreationDate="2017-02-27T14:54:52.940" Score="3" ViewCount="248" Body="&lt;p&gt;I am currently studying Deep Learning based Machine Translation systems but I'm not sure in my understanding the logic of the process. I understand that the source and destination language translation sentence pairs must be represented as word2vec vectors, but why is it necessary to apply two (encoder-decoder) Recurrent Neural Networks? My first idea would be applying only one RNN, where the input is the source language examples (in the form of word2vec vectors) and the output is simply a word2vec sequence of the destination language. Why is it necessary to use another RNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My additinal question is if this system is flexible enough to cope with synonimes, word order variations and other disambiguities? Is it capable of approximate the correct meaning of a new to-be-translated source language sentence?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And last but not least: how could one evaluate such a model where many translations can be correct at the same time?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-05-29T05:22:49.057" Title="High Level Understanding of Neural Machine Translation" Tags="|machine-learning|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17231" PostTypeId="2" ParentId="17220" CreationDate="2017-02-27T17:06:51.290" Score="0" Body="&lt;p&gt;You've described a null model for a logistic regression, which in this case would predict P(will buy X)=0.05. Any other model that incorporates your covariates of age and salary, will do better. The fact that there are 19 zeroes for every 1 in your response shouldn't stop you trying to make a model that does &lt;strong&gt;significantly&lt;/strong&gt; better than a null model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So just put age and salary in as linear terms in the regression and go from there.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-02-27T17:06:51.290" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17233" PostTypeId="1" AcceptedAnswerId="17236" CreationDate="2017-02-27T21:48:19.527" Score="2" ViewCount="498" Body="&lt;p&gt;I am trying to install an Add-on package from GitHub that contains prototype widgets for Orange Data Mining. I am trying to install it from the GitHub page found &lt;a href=&quot;https://github.com/biolab/orange3-prototypes.git&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am using the following Terminal code to install this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;git clone http://github.com/biolab/orange3-prototypes.git&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Everything then appears to install correctly and the download shows 100%. Then, however, it throws an error and says:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Orange requires Python &amp;gt;= 3.4&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I am using Mac OS. Clearly, it is suggesting that I need to use a different version of python to install, however, I already updated my pip install. Any insight into how I can fix this?&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2022-09-25T14:58:25.083" LastActivityDate="2022-09-25T14:58:25.083" Title="Installing Orange Package with older Python Version?" Tags="|python|orange|pip|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17235" PostTypeId="1" CreationDate="2017-02-27T23:46:29.300" Score="3" ViewCount="117" Body="&lt;p&gt;I am reading about the EM (Expectation-Maximization) algorithm in a machine learning book. At the end remark of the chapter, the authors mentioned that we cannot decide the &quot;optimality&quot; of the number of components (# of mixtures of Gaussians distributions) based on each model's log likelihood at the end--since models with more parameters will inevitably describe the data better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, my questions are &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How do we compare the performance of each model using a different number of components?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What are the important factors that help us to decide that an EM model is sufficient for modeling the observed data? &lt;/p&gt;&#xA;" OwnerUserId="28350" LastActivityDate="2017-04-29T09:44:25.270" Title="How to compare the performance of different number of mixing components for EM algorithm?" Tags="|machine-learning|data-mining|clustering|expectation-maximization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17236" PostTypeId="2" ParentId="17233" CreationDate="2017-02-28T02:04:05.917" Score="3" Body="&lt;p&gt;The error message shows exactly what you need to do:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Orange requires Python&amp;gt;=3.4&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;You have to specify a Python&amp;gt;=3.4 version (with orange3 itself installed) while installing &lt;code&gt;orange3-prototypes&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I'm not a Mac user, however, in Windows, Orange3 installer will automatically install a Python 3.4 if there's no compatible Python available.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="29169" LastEditDate="2020-12-11T03:15:24.497" LastActivityDate="2020-12-11T03:15:24.497" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17237" PostTypeId="1" CreationDate="2017-02-28T02:35:55.417" Score="1" ViewCount="63" Body="&lt;p&gt;Pardon my question if it seems too general... I'm new to networks in general, but recently I came across a type of network called citations network. However, I couldn't figure out why we do citation analysis or what kind of special information we can get out of it.&lt;/p&gt;&#xA;" OwnerUserId="29457" LastActivityDate="2017-02-28T15:03:03.970" Title="Why do we do citation analysis?" Tags="|graphs|social-network-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17238" PostTypeId="2" ParentId="17235" CreationDate="2017-02-28T03:22:40.267" Score="1" Body="&lt;p&gt;1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest and most common way is to use &lt;strong&gt;AIC&lt;/strong&gt; or &lt;strong&gt;BIC&lt;/strong&gt;. You would pick a model with the minimum AIC/BIC value. AIC/BIC work well here because you have a likelihood function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bayesian model selection&lt;/strong&gt; is another possibility. It's more advances than AIC or BIC, but you get chance to add your own prior distribution. Section 5.3 in &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Machine Learning - A Probabilistic perspective (Kevin P.Murphy)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;has the details. There're also lots of papers on Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use the &lt;strong&gt;cross-validated likelihood&lt;/strong&gt; as a performance measure, although this can be slow, since it requires fitting each model N times, where N is the number of CV holds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slide 17 in&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://www.doc.ic.ac.uk/~dfg/ProbabilisticInference/IDAPISlides13.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.doc.ic.ac.uk/~dfg/ProbabilisticInference/IDAPISlides13.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is a good reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EM algorithm is a technique for estimating the most likely parameters and is not limited to mixed Gaussian. The algorithm is useful when there's no closed form formula for the solving maximum likelihood problem with one or more &lt;strong&gt;latent variables&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-02-28T08:05:53.753" LastActivityDate="2017-02-28T08:05:53.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17239" PostTypeId="2" ParentId="17237" CreationDate="2017-02-28T03:44:22.620" Score="2" Body="&lt;p&gt;One simple example of an application of citation analysis is ranking. Imagine you do a keyword search over a collection of scientific articles, How do you rank the result set from most relevant to least? A measure one could use is the amount of citations an article has or the quality of the citations it has and also the quality of the articles the paper refers to. This problem is closely related to the problem of ranking web pages according to its hyperlinks. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-02-28T03:51:23.463" LastActivityDate="2017-02-28T03:51:23.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17240" PostTypeId="2" ParentId="13058" CreationDate="2017-02-28T05:34:19.493" Score="0" Body="&lt;p&gt;Please use the following dependency:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;dependency&amp;gt; &#xA;&amp;lt;groupId&amp;gt;com.orientechnologies&amp;lt;/groupId&amp;gt; &#xA;&amp;lt;artifactId&amp;gt;orientdb-jdbc&#xA;&amp;lt;/artifactId&amp;gt;&#xA; &amp;lt;version&amp;gt;2.2.10&amp;lt;/version&amp;gt;&#xA;&amp;lt;/dependency&amp;gt; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following code works fine for me: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SparkConf conf = new SparkConf();&#xA;conf.setMaster(&quot;local&quot;);&#xA;SparkSession spark = SparkSession.builder().appName(&quot;Orient_Spark_Session&quot;).config(conf).getOrCreate();&#xA;Map&amp;lt;String, String&amp;gt; orientDBProps = new HashMap&amp;lt;&amp;gt;(); &#xA;orientDBProps.put(&quot;url&quot;,&quot;jdbc:orient:REMOTE:localhost/TestDB&quot;);&#xA;orientDBProps.put(&quot;user&quot;, &quot;root&quot;);&#xA;orientDBProps.put(&quot;password&quot;, &quot;root&quot;);&#xA;orientDBProps.put(&quot;spark&quot;, &quot;true&quot;);&#xA;orientDBProps.put(&quot;dbtable&quot;, &quot;test&quot;);&#xA;Dataset&amp;lt;Row&amp;gt; tableDataset = spark.read().format(&quot;jdbc&quot;).options(dbProps).load();&#xA;tableDataset.show();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29461" LastEditorUserId="381" LastEditDate="2017-02-28T06:40:18.920" LastActivityDate="2017-02-28T06:40:18.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17241" PostTypeId="1" CreationDate="2017-02-28T08:12:08.210" Score="15" ViewCount="19234" Body="&lt;p&gt;I have a good general understanding of the role and mechanism of convolutional layers in Deep Learning for image processing in case of 2D or 3D implementations - they &quot;simply&quot; try to catch 2D patterns in images (in 3 channels in case of 3D).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But recently I bumped into 1D convolutional layers in the context of Natural Language Processing, which is a kind of surprise for me, because in my understanding the 2D convolution is especially used to catch 2D patterns that are impossible to reveal in 1D (vector) form of image pixels. What is the logic behind 1D convolution?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="29575" LastEditDate="2018-08-27T12:57:26.027" LastActivityDate="2018-08-27T12:57:26.027" Title="What is a 1D Convolutional Layer in Deep Learning?" Tags="|deep-learning|nlp|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17243" PostTypeId="2" ParentId="17216" CreationDate="2017-02-28T08:21:38.347" Score="4" Body="&lt;p&gt;PCA will not change the order of your points. The first point will still be the first point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second, this is too unclear to answer. There is no obvious relationship between the number of clusters and the number of PCs.&#xA;If you use too few PCs, your data approximation is too crude. If you use too many, then you will be working with random deviations too much, and the results will usually be worse, independent of the number of clusters.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-02-28T08:21:38.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17244" PostTypeId="1" AcceptedAnswerId="17252" CreationDate="2017-02-28T08:49:23.827" Score="1" ViewCount="284" Body="&lt;p&gt;In the field of machine learning, I'm wondering about the interest of applying feature selection techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, I often read articles or lectures speaking about how to reduce the number of feature (dimensionality reduction, PCA), how to select the best features (feature selection etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure of the main purpose of this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does feature reduction techniques always improve accuracy of the learned model?&lt;/li&gt;&#xA;&lt;li&gt;Or is it just a computational cost purpose?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to understand when it is necessary to reduce the number of features and when it is not, in order to improve interpretability or accuracy.&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="29190" LastActivityDate="2017-02-28T12:53:30.387" Title="Feature reduction convenience" Tags="|machine-learning|feature-selection|feature-engineering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17245" PostTypeId="2" ParentId="17244" CreationDate="2017-02-28T08:59:29.533" Score="-1" Body="&lt;p&gt;Feature selection could be done to prevent overfitting. The more features you have, the more likely you'll overfit. And yes, I'd say one of the purposes could be to reduce computational costs.&lt;/p&gt;&#xA;" OwnerUserId="29457" LastActivityDate="2017-02-28T08:59:29.533" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17246" PostTypeId="2" ParentId="14788" CreationDate="2017-02-28T09:21:59.767" Score="1" Body="&lt;p&gt;Without seeing your data it is hard to tell whether there is seasonality or not. The &lt;code&gt;decompose()&lt;/code&gt; function will try to find seasonality using a different approach than &lt;code&gt;tbats()&lt;/code&gt; as discussed in &lt;a href=&quot;https://stats.stackexchange.com/questions/55716/interpreting-time-series-decomposition-using-tbats-from-r-forecast-package&quot;&gt;this post&lt;/a&gt; and the user comments of &lt;a href=&quot;http://robjhyndman.com/hyndsight/dailydata/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post by the author of tbats&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend to post your data and discuss what you are trying to achieve to get more guidance.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-28T09:21:59.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17247" PostTypeId="2" ParentId="16613" CreationDate="2017-02-28T09:29:38.200" Score="1" Body="&lt;p&gt;You can use &lt;code&gt;kable()&lt;/code&gt; (see &lt;a href=&quot;https://www.rdocumentation.org/packages/knitr/versions/1.15.1/topics/kable&quot; rel=&quot;nofollow noreferrer&quot;&gt;manual&lt;/a&gt; to create tables that are directly in markdown format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your code would become:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;```{r,results='asis'}&#xA; knitr::kable(opneg1)&#xA;```&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See also &lt;a href=&quot;http://rmarkdown.rstudio.com/authoring_rcodechunks.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this great manual on rmarkdown&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2017-02-28T09:29:38.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17248" PostTypeId="2" ParentId="17220" CreationDate="2017-02-28T09:40:50.480" Score="1" Body="&lt;p&gt;There is a lot of information and techniques for rare event or imbalanced classes. Sorry to post links but (&lt;a href=&quot;https://stats.stackexchange.com/search?q=rare+event&quot;&gt;https://stats.stackexchange.com/search?q=rare+event&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/search?q=imbalanced&quot;&gt;https://stats.stackexchange.com/search?q=imbalanced&lt;/a&gt;) but I do not want to duplicate all of the other work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experiences, often I left the data set alone. I usually got good results. If that is the true ratio, then my model should know that. Sometimes I downsampled the majority class. I played with SMOTE (&lt;a href=&quot;https://arxiv.org/pdf/1106.1813.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1106.1813.pdf&lt;/a&gt;) before as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your questions above, yes - changing the data with weights or over/unsampling may bias the results.  Need to check. If you downsample, make sure you are not tossing out signal from the majority class.&lt;/p&gt;&#xA;" OwnerUserId="28780" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-02-28T09:40:50.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17249" PostTypeId="2" ParentId="17241" CreationDate="2017-02-28T10:03:35.733" Score="16" Body="&lt;p&gt;In short, there is nothing special about number of dimensions for convolution. Any dimensionality of convolution could be considered, if it fit a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of dimensions is a property of the problem being solved. For example, 1D for audio signals, 2D for images, 3D for movies . . .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ignoring number of dimensions briefly, the following can be considered &lt;em&gt;strengths&lt;/em&gt; of a convolutional neural network (CNN), compared to fully-connected models, when dealing with certain types of data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The use of shared weights for each location that the convolution processes significantly reduces the number of parameters that need to be learned, compared to the same data processed through fully-connected network.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Shared weights is a form of regularisation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The structure of a convolutional model makes strong assumptions about local relationships in the data, which when true make it a good fit to the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.1 Local patterns provide good predictive data (and/or can be usefully combined into more complex predictive patterns in higher layers)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.2 The types of pattern found in the data can be found in multiple places. Finding the same pattern in a different set of data points is meaningful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These properties of CNNs are independent of the number of dimensions. One-dimensional CNNs work with patterns in one dimension, and tend to be useful in signal analysis over fixed length signals. They work well for analysis of audio signals, for instance. Also for some natural language processing - although recurrent neural networks, which allow for different sequence lengths, may be a better fit there, especially ones with memory gate arrangements such as LSTM or GRU. Still a CNN can be easier to manage, and you could simply pad the input to be fixed length.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-02-28T10:10:41.023" LastActivityDate="2017-02-28T10:10:41.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17251" PostTypeId="2" ParentId="5292" CreationDate="2017-02-28T12:46:24.240" Score="2" Body="&lt;p&gt;The framework you cope with is &lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;semi supervised&lt;/a&gt;. You have mostly unlabelled data and you can have some labeled data by manual labelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Active_learning_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Active learning&lt;/a&gt; is one method to cope with the situation, by focusing your labelling efforts in the most beneficial areas. You can read a survey on these techniques at &lt;a href=&quot;http://burrsettles.com/pub/settles.activelearning.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Settles, Burr (2010), &quot;Active Learning Literature Survey&quot; (PDF), Computer Sciences Technical Report 1648. University of Wisconsin–Madison, retrieved 2014-11-18&#xA;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that even if you are focusing your labelling efforts using active learning, it is still a significant constraint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other methods of copying with semi-supervised framework, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Co-training&quot; rel=&quot;nofollow noreferrer&quot;&gt;co training&lt;/a&gt;.The classical reference on co-training is &quot;&lt;a href=&quot;http://repository.cmu.edu/cgi/viewcontent.cgi?article=1181&amp;amp;context=compsci&amp;amp;sei-redir=1&amp;amp;referer=http%3A%2F%2Fscholar.google.com%2Fscholar%3Fq%3DCombining%2Blabeled%2Band%2Bunlabeled%2Bdata%2Bwith%2Bco-training%26hl%3Den%26as_sdt%3D0%26as_vis%3D1%26oi%3Dscholart%26sa%3DX%26ei%3D_NlxT7nlHNSA8gPKgqVK%26ved%3D0CBoQgQMwAA#search=%22Combining%20labeled%20unlabeled%20data%20co-training%22&quot; rel=&quot;nofollow noreferrer&quot;&gt;Blum, A., Mitchell, T. Combining labeled and unlabeled data with co-training. COLT: Proceedings of the Workshop on Computational Learning Theory, Morgan Kaufmann, 1998, p. 92-100&lt;/a&gt;&quot;&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-02-28T12:46:24.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17252" PostTypeId="2" ParentId="17244" CreationDate="2017-02-28T12:53:30.387" Score="3" Body="&lt;p&gt;Feature Selection (FS) methods are focused on specializing the data as much as possible to find accurate models for your problem. Some of the main issues that drive the need for FS are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Curse of dimensionality: Most algorithms suffer to grasp relevant chacteristics of data for a specific prediction task, when the number of dimensions (features) of the data is high, and  the number of examples is not sufficiently big. &lt;a href=&quot;https://en.wikipedia.org/wiki/Curse_of_dimensionality&quot; rel=&quot;nofollow noreferrer&quot;&gt;Check here some more detailed explanation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Correlation between variables: Typically, the presence of highly correlated pair of variables can cause ML algorithms to pay to much attention to a particular effect that is &quot;over-represented&quot;. For this reason many FS methods address the reduction of this correlation. Reducing the number of correlated variables oftenly increases the model predictive power.&lt;/li&gt;&#xA;&lt;li&gt;Latent features: Although specific variables might be highly expressive for your problem, a lot of power can be achieved when finding &quot;latent features&quot;, such as linear and non-linear combinations of the original variables. Here there are hundreds of approaches, from PCA to Neural Networks. Independently from the approach (and statistical assumptions) the idea is to create new features that condense the information of a bigger set of features into a smaller one. Hopefuly the new set of features is more representative, and being smaller could be more easily learnt.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Feature selection does not necessarily improve the predictive quality of the model. Reducing or transforming the features might lead you to loss of information and then a less accurate model. Is an open and complex field of research. However in many cases it becomes quite useful. It will depend on how good and different are your original features for describing the target variable.&#xA;If you look into bionformatics, you'll see people dealing with thousands or even millions of features, while having only hundred of examples. Here feature selection becomes increasingly relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Most commonly I've seen the term &quot;feature selection&quot; used for the creation of compound features, as most examples I've mentioned, while Feature Extraction term is used for actually removing specific features from the dataset without taking into account is relation with the target variable.&lt;/p&gt;&#xA;" OwnerUserId="5143" LastActivityDate="2017-02-28T12:53:30.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17253" PostTypeId="2" ParentId="17151" CreationDate="2017-02-28T13:18:36.057" Score="0" Body="&lt;p&gt;I solved the issue by using &lt;em&gt;creole.xml&lt;/em&gt;. I have tried with SentenceSplitter but it is giving me incorrect sentences. So, I am using RegexSentenceSplitter instead.&#xA;Added few lines in creole.xml-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;RESOURCE&amp;gt;&#xA;    &amp;lt;CLASS&amp;gt;gate.creole.splitter.RegexSentenceSplitter&amp;lt;/CLASS&amp;gt;&#xA;    &amp;lt;PARAMETER NAME=&quot;externalSplitListURL&quot; DEFAULT=&quot;file:/C:/Program%20Files/GATE_Developer_8.1/plugins/ANNIE/resources/regex-splitter/external-split-patterns.txt&quot; COMMENT=&quot;The URL to the custom external-split-patterns.txt&quot; SUFFIXES=&quot;txt;text&quot;&amp;gt;java.net.URL&amp;lt;/PARAMETER&amp;gt;&#xA;    &amp;lt;PARAMETER NAME=&quot;internalSplitListURL&quot; DEFAULT=&quot;file:/C:/Program%20Files/GATE_Developer_8.1/plugins/ANNIE/resources/regex-splitter/internal-split-patterns.txt&quot; COMMENT=&quot;The URL to the custom internal-split-patterns.txt&quot; SUFFIXES=&quot;txt;text&quot;&amp;gt;java.net.URL&amp;lt;/PARAMETER&amp;gt;&#xA;    &amp;lt;PARAMETER NAME=&quot;nonSplitListURL&quot; DEFAULT=&quot;file:/C:/Program%20Files/GATE_Developer_8.1/plugins/ANNIE/resources/regex-splitter/non-split-patterns.txt&quot; COMMENT=&quot;The URL to the custom non-split-patterns.txt&quot; SUFFIXES=&quot;txt;text&quot;&amp;gt;java.net.URL&amp;lt;/PARAMETER&amp;gt;&#xA;&amp;lt;/RESOURCE&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then loading &lt;em&gt;ANNIE_with_defaults.gapp&lt;/em&gt; from backend code as shown in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hope it may be helpful.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="29335" LastActivityDate="2017-02-28T13:18:36.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17254" PostTypeId="1" CreationDate="2017-02-28T13:34:54.610" Score="6" ViewCount="3543" Body="&lt;p&gt;Suppose that at a layer $N$ within a CNN, my &quot;image&quot; is a 200$\times$200$\times$10 array. Thus, if I convolve such an array with, for example, 15 filters of size 3$\times$3$\times$10, I will end up with a new &quot;image&quot; whose shape is $A\times{B}\times{15}$ where $A$ and $B$ depend on the stride of the convolution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thus wondering how such 3D (3$\times$3$\times$10 in my toy example) convolution filters are usually displayed as 2D images (i.e. sum or mean, ...)?&lt;/p&gt;&#xA;" OwnerUserId="29473" LastEditorUserId="31513" LastEditDate="2017-04-24T10:38:16.077" LastActivityDate="2018-03-13T16:13:37.917" Title="What is the classical way to visualize 3D filters in convolutional neural networks?" Tags="|deep-learning|visualization|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="17255" PostTypeId="1" CreationDate="2017-02-28T13:36:59.987" Score="1" ViewCount="55" Body="&lt;p&gt;I have two dataset A and B. We have 93 persons in A and 88 in B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset A contains number of person who had made a payment. So &lt;code&gt;A&lt;/code&gt; is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;number of payments      number of persons made that payment&#xA;1                       93  &#xA;2                       41&#xA;3                       35&#xA;4                       28&#xA;5                       20&#xA;6                       14&#xA;7                       7&#xA;8                       4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similar way with dataset B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this: How can I calculate if A has a higher performance than B, ie if we have significant more person in A who make a payment compared to B.&lt;/p&gt;&#xA;" OwnerUserId="18787" LastActivityDate="2017-02-28T15:48:16.870" Title="How can one compare these groups?" Tags="|ab-test|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17256" PostTypeId="1" CreationDate="2017-02-28T14:03:50.473" Score="1" ViewCount="255" Body="&lt;p&gt;For feature selection I use reliefF provided by matlab.&#xA;The reliefF function offers a parameter k to influence its ouput, additionaly for my specific task I can also vary a window length l on which the features are computed. ( I am classifying sensor signals )&#xA;My question is how do I choose the optimal parameters &lt;strong&gt;k&lt;/strong&gt; and &lt;strong&gt;l&lt;/strong&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially how should I score the different parameter configurations ?&#xA;Can I use the weights outputed by reliefF ( average weights of all features ) ? I dont't want to use classification error based on the selected features, since then an other question arises : Which classifier should be used, and how are its hyperparameters selected ?&lt;/p&gt;&#xA;" OwnerUserId="29471" LastActivityDate="2017-02-28T14:03:50.473" Title="How do I choose the optimal parameters for reliefF" Tags="|feature-selection|hyperparameter|parameter-estimation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17257" PostTypeId="2" ParentId="17237" CreationDate="2017-02-28T15:03:03.970" Score="0" Body="&lt;p&gt;Not sure if this is relevant to what you're investigating, but I believe reading up on &lt;strong&gt;&lt;em&gt;Bibliometrics&lt;/em&gt;&lt;/strong&gt; might give you some insight into citation analysis. This type of analysis is used within research/academia to identify trends in topics of interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interestingly, I was reading &lt;a href=&quot;http://journal.frontiersin.org/article/10.3389/fnins.2017.00120/abstract&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; yesterday which analyzed and mapped the shift in research priority within the neuroscientific community. Hopefully this helps! &lt;/p&gt;&#xA;" OwnerUserId="29479" LastActivityDate="2017-02-28T15:03:03.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17258" PostTypeId="1" AcceptedAnswerId="17269" CreationDate="2017-02-28T15:25:38.020" Score="7" ViewCount="10876" Body="&lt;p&gt;Let's say that I have two similar datasets with the same size of elements, for example 3D points : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dataset &lt;strong&gt;A&lt;/strong&gt; : { (1,2,3), (2,3,4), (4,2,1) }&lt;/li&gt;&#xA;&lt;li&gt;Dataset &lt;strong&gt;B&lt;/strong&gt; : { (2,1,3), (2,4,6), (8,2,3) }&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And the question is that is there a way to measure the correlation/similarity／Distance between these two datasets ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="24005" LastActivityDate="2018-11-09T06:39:44.347" Title="Is there a way to measure correlation between two similar datasets?" Tags="|correlation|similarity|distance|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17259" PostTypeId="2" ParentId="17258" CreationDate="2017-02-28T15:43:53.447" Score="1" Body="&lt;p&gt;Well, if your samples are collections of points, I would separate this in two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculate distances between inner points: choose how to calculate the distance between (1,2,3) and (2,1,3), for instance. Here, depending on the nature of your problem, you could go for something akin to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Euclidean_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;euclidean distance&lt;/a&gt; or if you only care about the orientation of the points, something like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine similarity&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Summarize all the distances as a single number: depending on your problem, you could get its average, its median or some other quantity. The main idea is to reduce all the numbers to a single one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4855" LastActivityDate="2017-02-28T15:43:53.447" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17260" PostTypeId="2" ParentId="17258" CreationDate="2017-02-28T15:47:05.053" Score="4" Body="&lt;p&gt;I would take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Canonical_correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Canonical correlation Analysis&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-02-28T15:47:05.053" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17262" PostTypeId="2" ParentId="17156" CreationDate="2017-02-28T15:54:11.480" Score="1" Body="&lt;p&gt;I took an introductory computer science course on Coursera a few years back where we had to use a Monte Carlo algorithm to build a tic-tac-toe AI.  Instead of using historical data, we generated data using a function that would play through a series of games with random moves, but I think the principle is basically the same.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each square on the board has a separate score associated with it. For each game completed, the scores change based on which player won.  If the player designated as the machine player won, the machine player's score for each square on the board where it placed a mark would increase and the &quot;other&quot; player's score would decrease for every square where it placed a mark.  Conversely, if the other player won, its score would increase for each square on the board where it placed a mark and the machine player's score would decrease.  If the game is a tie, the number of points for each square would not change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to answer your question, I think it is important to track both the winning and losing player's moves because we need to know not only what works but also what doesn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  I just remembered that my first implementation did not deduct points on the grid for games lost and because of this, my AI had an all-out &quot;offensive&quot; strategy.  In other words, it focused on getting three-in-a-row without making any effort to block the human player.  This is why it's important to track moves that result in a loss.&lt;/p&gt;&#xA;" OwnerUserId="29353" LastEditorUserId="29353" LastEditDate="2017-03-03T01:19:26.253" LastActivityDate="2017-03-03T01:19:26.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17263" PostTypeId="1" CreationDate="2017-02-28T16:09:56.263" Score="7" ViewCount="1471" Body="&lt;p&gt;First of all we have a classification task.&lt;br&gt;&#xA;So we use the typical softmax cross entropy to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Current implementation of curriculum learning is as follows.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First we train our best version of the neural net&lt;/li&gt;&#xA;&lt;li&gt;At the last epoch we get all of the cross entropies for all the instances. So smaller cross entropies are better classified and larger ones not so well.&lt;/li&gt;&#xA;&lt;li&gt;Then we sort the instances based on the cross entropies.&lt;/li&gt;&#xA;&lt;li&gt;Then we start training the instances from easier to harder as the curriculum learning theory suggests&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that we have already experimented with various steps and repetitions. So in one example we took the first 200 batches and trained them two times before going to the next batch and so on until an epoch is completed.&lt;br&gt;&#xA;In another example we took the first 10 batches and trained them only once  and then the next 10 and the next 10 and so on until the end of the epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the experiments so far have concluded that the neural network is has a relatively ok accuracy at the beginning and this gets worsen as the more difficult instances come along.&#xA;The final accuracy is much worse than expected and in addition the maximum accuracy is still quite bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this curriculum learning not working? Is anything missing?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="24885" LastEditDate="2017-02-28T18:51:00.837" LastActivityDate="2017-03-02T17:27:27.153" Title="What is missing from the following Curriculum Learning implementation in a Deep Neural Net?" Tags="|deep-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17266" PostTypeId="1" CreationDate="2017-02-28T19:36:11.590" Score="6" ViewCount="160" Body="&lt;p&gt;I guess LSTM is good for sequence modeling but how would you model &quot;clustering&quot; with it? Meaning, the input is a sequence and the output is labels with similar properties (I have labeled data). For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input:  1 2 1 1 2 1 2 5 6 5 4 5 1 1 2 1 1 2 1 &#xA;output: 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is how to tell the model to learn differences and not the particular example, as a new example could have totally different values. I could have taught it to predict the N*(N-1)/2 pairwise differences (similar non similar) but it will not learn the dynamics.&lt;/p&gt;&#xA;" OwnerUserId="21596" LastEditorUserId="381" LastEditDate="2017-02-28T23:00:44.673" LastActivityDate="2022-02-11T00:05:30.907" Title="How to model segmentation of a sequence to similar parts?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17269" PostTypeId="2" ParentId="17258" CreationDate="2017-03-01T04:27:25.347" Score="4" Body="&lt;p&gt;I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as means, deviations, min/max, but there's no magical formula to say that data set A looks like B, especially if they are varying data sets by rows and columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work at one of the largest credit score/fraud analytics companies in the US. Our models utilize large number of variables. When my team gets a request for a report, we have to look at each individual variable to inspect that the variables are populated as they should be with respect to the context of the client. This is very time consuming, but necessary. Some tasks do not have magical formulas to get around inspecting and digging deep into the data. However, any good data analyst should understand this already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your situation, I believe you should identify key statistics of interest to your data/problems. You may also want to look at what distributions look like graphically, as well as how variables relate to others. If for data set A, &lt;code&gt;Temp&lt;/code&gt; and &lt;code&gt;Ozone&lt;/code&gt; are positively correlated, and if B is generated through the same source (or similar stochastic process), then B's &lt;code&gt;Temp&lt;/code&gt; and &lt;code&gt;Ozone&lt;/code&gt; should also exhibit a similar relationship.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My I will illustrate my point via this example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data(&quot;airquality&quot;)&#xA;head(airquality)&#xA;dim(airquality)&#xA;&#xA;set.seed(123)&#xA;indices &amp;lt;- sample(x = 1:153, size = 70, replace = FALSE) ## randomly select 70 obs&#xA;&#xA;A = airquality[indices,]&#xA;B = airquality[-indices,]&#xA;&#xA;&#xA;summary(A$Temp) ## compare quantiles&#xA;&#xA;summary(B$Temp)&#xA;&#xA;plot(A)&#xA;plot(B)&#xA;&#xA;plot(density(A$Temp), main = &quot;Density of Temperature&quot;)&#xA;plot(density(B$Temp), main = &quot;Density of Temperature&quot;)&#xA;&#xA;&#xA;plot(x = A$Temp, y = A$Ozone, type = &quot;p&quot;, main = &quot;Ozone ~ Temp&quot;,&#xA;     xlim = c(50, 100), ylim = c(0, 180))&#xA;lines(lowess(x = A$Temp, y = A$Ozone), col = &quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/i8ZHZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/i8ZHZ.png&quot; alt=&quot;Scatter plot: Ozone ~ Temp for set A&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(x = B$Temp, y = B$Ozone, type = &quot;p&quot;, main = &quot;Ozone ~ Temp&quot;,&#xA;     xlim = c(50, 100), ylim = c(0, 180))&#xA;lines(lowess(x = B$Temp, y = B$Ozone), col = &quot;blue&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oiLIf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oiLIf.png&quot; alt=&quot;Scatterplot: Ozone ~ Temp for set B&quot;&gt;&lt;/a&gt;    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cor(x = A$Temp, y = A$Ozone, method = &quot;spearman&quot;, use = &quot;complete.obs&quot;) ## [1] 0.8285805&#xA;&#xA;cor(x = B$Temp, y = B$Ozone, method = &quot;spearman&quot;, use = &quot;complete.obs&quot;) ## [1] 0.6924934&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2017-03-01T04:27:25.347" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17270" PostTypeId="2" ParentId="17227" CreationDate="2017-03-01T07:51:41.950" Score="2" Body="&lt;p&gt;You can efficiently implement e.g. Euclidean distance or Cosine on sparse data (iterate over nonzero values only!). Then you can use e.g. Hierarchical clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also consider frequent itemset mining. Usually, on sparse binary data, frequent itemsets are better than clusters.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-03-01T07:51:41.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17271" PostTypeId="2" ParentId="14179" CreationDate="2017-03-01T08:23:50.743" Score="0" Body="&lt;p&gt;Is this what you want?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_csv('Fielding.csv')&#xA;df['FP'] = df['E'] / (df['PO'] + df['A'] + df['E'])&#xA;df.groupby(['yearID', 'POS']).sum().reset_index().pivot_table(index='yearID', columns='POS', values='FP')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;POS      1B    2B    3B     C   CF  DH   LF    OF     P   RF    SS&#xA;yearID&#xA;1871   1.84  5.98 10.05  6.26  nan nan  nan 12.50  2.10  nan  8.24&#xA;1872   2.67  9.71  9.46  6.29  nan nan  nan 23.92  5.84  nan  9.48&#xA;1873   3.02  9.03  7.72  5.36  nan nan  nan 15.26  6.74  nan  8.48&#xA;1874   3.74 10.29  8.32 10.04  nan nan  nan 15.72  3.20  nan 12.85&#xA;1875   5.81 12.65 15.62 13.35  nan nan  nan 30.67  9.10  nan 14.42&#xA;1876   2.08  7.31  3.98  8.34  nan nan  nan 14.48  5.14  nan  3.55&#xA;......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-01T08:23:50.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17272" PostTypeId="2" ParentId="14905" CreationDate="2017-03-01T09:08:30.757" Score="0" Body="&lt;p&gt;Disclaimer: I am not 100% sure if my solution will be good for all cases but this solved my problem considerably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me changing to XGBoost based imputation worked wonderfully. In the earlier case I was doing multiple different type of imputation like mean, mode etc. From there I retained only those imputations on which I was very much confident. Imputations on which I didn't have much confidence I did let XGBoost do the job. After this change most of the overestimation cases vanished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another benefit I got from this change was: model fit parameters were direct reflective of the overestimation. So models in overestimation was still happening; all of those have had bad model fit Vs models with no overestimation always have good good fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe this is happening because with XGBoost also doing the imputation it uncovers and leverages deeply hidden patterns in data. With this knowledge it is able to do better imputation of the scenarios where my ways of ordinary imputation were not much useful.&lt;/p&gt;&#xA;" OwnerUserId="12792" LastActivityDate="2017-03-01T09:08:30.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17273" PostTypeId="2" ParentId="17258" CreationDate="2017-03-01T09:22:01.443" Score="1" Body="&lt;p&gt;If you are interested in the 1-Dimensional distributions you could use a test (like a Kolmogorov-Smirnov test). I would naively expect that while this cant tell you if data is similar it can tell you if it is not. Or you create multidimensional histograms and calculate a Chi2 similar quantity. Obviously this can run into some problems if the parameter space is rather sparsely filled. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-03-01T09:22:01.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17274" PostTypeId="1" CreationDate="2017-03-01T09:56:14.587" Score="4" ViewCount="369" Body="&lt;p&gt;I need to calculate statistical significance of difference between two time series, each with 4500 terms. My null hypothesis is that &lt;span class=&quot;math-container&quot;&gt;$H_0: \mu=\mu_0$&lt;/span&gt;. How can I calculate p-value? Is Z-statistic useful for p value calculation? How to get p-value after computing Z-statistic? I have &lt;span class=&quot;math-container&quot;&gt;$\alpha = 0.05$&lt;/span&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27460" LastEditorUserId="91724" LastEditDate="2022-02-17T12:34:00.087" LastActivityDate="2022-03-20T03:05:25.820" Title="how do I calculate p-value?" Tags="|statistics|inference|pvalue|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17276" PostTypeId="1" CreationDate="2017-03-01T12:58:02.843" Score="3" ViewCount="145" Body="&lt;p&gt;I apologize for a vague title, but I know only the basic machine learning jargon. Let's assume that we have domain set $X$ and label set $L = \left\{0, 1\right\}$, but the labelling function $f$ is only partially defined by function $g:P→L$, where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1)  $P$ is a partition of $X$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2)  $\it g(p) = 0 ⟹ \forall x \in p: f(x) = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3)  $\it g(p) = 1 ⟹ \exists x \in p: f(x) = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The meaning of function $g$ is that for some elements of $P$ all members have label $0$ and for remaining elements of $P$ at least one member has label $1$, but we don't know which is $1$ and which is $0$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: we have boxes of apples with possibly different number of apples in each box. We know which boxes have all good apples (apple with label $0$) and remaining boxes have at least one bad apple (apple with label $1$), but we don't know which ones are bad or good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a machine learning technique to deal with such partially defined labelling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume that in boxes of apples example above, a classifier we looking for is: good apples are green and hard, bad apples are brown and soft. As a model of an apple, my lucky guess is a 4-d feature vector with color, hardness, weight and size components. All of the apples are observable, so we can calculate the feature vector for each of them. We know labels for all apples in &quot;good&quot; boxes, but have only an existential statement about labels of apples in &quot;bad&quot; boxes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After measuring all apples, I look for clusters of feature vectors or run some unsupervised learning algorithm. I will discover 2 distinct classes: (green, hard, _, _) and (brown, soft, _, _). These classes will also satisfy conditions (2) and (3) above, so I will accept (green, hard) vs. (brown, soft) as a correct classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this procedure in mind, let me rephrase my question. Assuming my guess of feature vector is correct, so in about 10-d vector there are enough significant features for correct classification, the partition set $P$ has 1K elements and the domain set $X$ has 100K elements, what is the correct machine learning technique to use?&lt;/p&gt;&#xA;" OwnerUserId="29498" LastEditorUserId="29498" LastEditDate="2017-03-03T12:18:30.967" LastActivityDate="2017-03-03T21:23:56.663" Title="Machine learning with partially defined labelling using existential quantifiers" Tags="|machine-learning|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="17278" PostTypeId="2" ParentId="16587" CreationDate="2017-03-01T15:56:56.197" Score="1" Body="&lt;p&gt;This is interesting !?!&#xA;The way you think about it is as solving a linear system equation, which is not exactly the case.&#xA;Imagine if you have thousands of data points , and you try to find the set of weights that solves ( satisfies) these points, what if an exact solution is not exist. What if there is no set of weights that exactly solves all these points. It becomes an optimization problem. What is the set of weight that minimizes the error. In this case the error is the difference between the actual output and the output you get by using the current weights. &#xA;In fact, in current machine learning technology, it is difficult even to construct such a linear system.&#xA;We use optimization techniques to solve this and find the optimal set of weights &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2017-03-01T16:06:32.120" LastActivityDate="2017-03-01T16:06:32.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17279" PostTypeId="1" AcceptedAnswerId="18272" CreationDate="2017-03-01T16:30:20.760" Score="4" ViewCount="772" Body="&lt;p&gt;I am currently working with speech recognition, in which i would like to try to use CNN instead of the normal feature extraction step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I been reading this &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; which proposes method using cnn. The input is a  visual representation of mel-log filter bank energies of audio files. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/J24cx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J24cx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the output is phoneme recognised for each third frame section a portion (a frames, b frequency_bands) of the image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network is a CNN, and they propose a different weight sharing - limited weight sharing, since the patterns being seeked for doesn't occur  equally anywhere on the image, but localised to certain frequency areas. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Using separate sets of weights for different frequency bands may be more suitable since it allows for detection of distinct feature patterns in different filter bands along the frequency axis. Fig. 5 shows an example of the limited weight sharing (LWS) scheme for CNNs, where only the convolution units that are attached to the same pooling unit share the same convolution weights. These convolution units need to share their weights so that they compute comparable features, which may then be pooled together.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not sure I understand the concept of this weight sharing.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should the weights be shared for each frame but limited in frequency range?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or should it both be limited for each frame and frequency range?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They made a illustration of this weight sharing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MaZ2h.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MaZ2h.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what i can decipher from the image - is Limited weight sharing option 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each frame do not have the same weight, multiple convolution is applied on to the same frame, and the convolution for the next frame starts at a lower frequency than than the previous frame and has stride = 2. So somehow is the convolution only performed in the diagonal of the image... sounds weird?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sounds like i've misinterpreted something incorrectly here?&#xA;Any ideas on how to implement?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:51:41.013" LastActivityDate="2021-02-16T02:51:41.013" Title="Training a CNN with limited weight sharing" Tags="|python|keras|convolutional-neural-network|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17281" PostTypeId="1" AcceptedAnswerId="17292" CreationDate="2017-03-01T18:51:33.493" Score="1" ViewCount="615" Body="&lt;p&gt;I have a general question of unbalanced data. I'm performing a T test on Group A and Group B. Group A has 20 data while Group B has 500. I set unequal variance (Welch) for the adjustment and the P-value is 0.01. Can I conclude that there is significance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have another idea that I'm not sure about. I randomly select about 20 samples from Group B multiple times and test against the Group A. Should I do some adjustment for the result? I am not sure if this is works or not. If it works, what kind of adjustment I should use for multi-testing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate any replies!&lt;/p&gt;&#xA;" OwnerUserId="28860" LastEditorUserId="15620" LastEditDate="2017-03-02T03:41:09.827" LastActivityDate="2017-03-02T03:41:09.827" Title="Multiple testing of unbalanced data by R" Tags="|statistics|class-imbalance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17282" PostTypeId="1" AcceptedAnswerId="17293" CreationDate="2017-03-01T19:15:54.660" Score="22" ViewCount="33048" Body="&lt;p&gt;If I train my model using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xg&#xA;params = {'max_depth':3,&#xA;'min_child_weight':10,&#xA;'learning_rate':0.3,&#xA;'subsample':0.5,&#xA;'colsample_bytree':0.6,&#xA;'obj':'reg:linear',&#xA;'n_estimators':1000,&#xA;'eta':0.3}&#xA;&#xA;features = df[feature_columns]&#xA;target = df[target_columns]&#xA;dmatrix = xg.DMatrix(features.values,&#xA;                     target.values,&#xA;                     feature_names=features.columns.values)&#xA;clf = xg.train(params, dmatrix)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it finishes in about 1 minute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I train my model using the Sci-Kit learn method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xg&#xA;max_depth = 3&#xA;min_child_weight = 10&#xA;subsample = 0.5&#xA;colsample_bytree = 0.6&#xA;objective = 'reg:linear'&#xA;num_estimators = 1000&#xA;learning_rate = 0.3&#xA;&#xA;features = df[feature_columns]&#xA;target = df[target_columns]&#xA;clf = xg.XGBRegressor(max_depth=max_depth,&#xA;                min_child_weight=min_child_weight,&#xA;                subsample=subsample,&#xA;                colsample_bytree=colsample_bytree,&#xA;                objective=objective,&#xA;                n_estimators=num_estimators,&#xA;                learning_rate=learning_rate)&#xA;clf.fit(features, target)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it takes over 30 minutes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think the underlying code is nearly exactly the same (i.e. &lt;code&gt;XGBRegressor&lt;/code&gt; calls &lt;code&gt;xg.train&lt;/code&gt;) - what's going on here?&lt;/p&gt;&#xA;" OwnerUserId="21690" LastActivityDate="2017-03-02T02:00:56.863" Title="XGBRegressor vs. xgboost.train huge speed difference?" Tags="|machine-learning|python|decision-trees|xgboost|efficiency|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17283" PostTypeId="1" CreationDate="2017-03-01T19:31:22.463" Score="3" ViewCount="2109" Body="&lt;p&gt;My dream occupation is to work as a Data Scientist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm soon going to be completing my first semester as a physics student, as part of an incredibly prestigious physics program. We get to run an experiment at CERN in our third year! I'm currently ranked highly in this program (top 25%), and I could push myself to top 10% if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thing is, I'm not really interested in working as a physicist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I switch to Math &amp;amp; CS? (I'd be in a mediocre program.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note - I'm currently in both physics and CS, not just physics. I'm asking if I should switch the physics major with a math one.&lt;/p&gt;&#xA;" OwnerUserId="29511" LastEditorUserId="29512" LastEditDate="2017-03-01T21:51:49.563" LastActivityDate="2017-03-01T21:51:49.563" Title="Which is better for Data Science, a double major in Math &amp; CS or Physics &amp; CS?" Tags="|data|education|career|learning|" AnswerCount="2" CommentCount="1" ClosedDate="2017-03-02T07:31:05.677" ContentLicense="CC BY-SA 3.0" />
  <row Id="17284" PostTypeId="2" ParentId="17283" CreationDate="2017-03-01T20:07:10.520" Score="0" Body="&lt;p&gt;Depends what field of data science you want to be in, and what you find most enjoyable. Its entirely possible to be a data scientist with just a physics degree. I did IT because I was interested network analytics, I know people who did psych because they wanted to do people analytics. Business degree to do marketing analytics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I was to do it again I would major stats, minor comp sci. Because really understand statistical concepts would be a huge plus, and having some programming skills is a huge plus to put you ahead of the pack. &lt;/p&gt;&#xA;" OwnerUserId="29513" LastActivityDate="2017-03-01T20:07:10.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17285" PostTypeId="2" ParentId="17283" CreationDate="2017-03-01T20:07:41.660" Score="2" Body="&lt;p&gt;My background is Physics undergrad, Physics grad, now in industry.  Some advice off the top of my head: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you can push top 10%, do it (always do your best).  &lt;/li&gt;&#xA;&lt;li&gt;Don't be surprised when jobs in industry don't care about you ranking/gpa (some do, some don't.). Rank/gpa is  important for grad school however, and grad school is important for careers in data science &lt;/li&gt;&#xA;&lt;li&gt;Major in what makes you happy.  &lt;/li&gt;&#xA;&lt;li&gt;If you stay in physics, learn to code and take some stats.  &lt;/li&gt;&#xA;&lt;li&gt;Physics will always impress people and there are many machine learning/ data science opportunities in physics.  &lt;/li&gt;&#xA;&lt;li&gt;Physics will teach you how and where to apply machine learning rather than being given perfect datasets to learn with.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-03-01T20:07:41.660" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17286" PostTypeId="1" CreationDate="2017-03-01T21:09:48.080" Score="13" ViewCount="23073" Body="&lt;p&gt;I'd like to be able to estimate whether a proposed model is small enough to be trained on a GPU with a given amount of memory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have a simple CNN architecture like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Input&lt;/code&gt;: 50x50x3&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;C1&lt;/code&gt;: 32 3x3 kernels, with padding (I guess in reality theyre actually 3x3x3 given the input depth?)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;P1&lt;/code&gt;: 2x2 with stride 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;C2&lt;/code&gt;: 64 3x3 kernels, with padding&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;P2&lt;/code&gt;: 2x2 with stride 2&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;FC&lt;/code&gt;: 500 neurons&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Output&lt;/code&gt;: softmax 10 classes&lt;/li&gt;&#xA;&lt;li&gt;Mini batch size of 64&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming 32bit floating point values, how do you calculate the memory cost of each layer of the network during training? and then the total memory required to train such a model?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T02:50:40.717" LastActivityDate="2021-02-16T02:50:40.717" Title="CNN memory consumption" Tags="|machine-learning|neural-network|convolutional-neural-network|gpu|parameter-estimation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17287" PostTypeId="1" AcceptedAnswerId="17290" CreationDate="2017-03-01T21:16:32.973" Score="5" ViewCount="5499" Body="&lt;p&gt;I've read that some convolution implementations use FFT to calculate the output feature/activation maps and I'm wondering how they're related. I'm familiar with applying CNNs, and (mildly) familiar with the use of FFT in signal processing, but I'm not sure how the 2 work together&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I think of convolutions, I imagine taking a kernel, flipping it, multiplying (and adding) the elements of the kernel with the overlapping input, shifting the kernel and repeating the process. How does a FFT fit into this process?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T02:50:30.590" LastActivityDate="2021-02-16T02:50:30.590" Title="Convolutional neural network fast fourier transform" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17288" PostTypeId="1" CreationDate="2017-03-01T22:10:48.180" Score="9" ViewCount="3687" Body="&lt;p&gt;Recently, I was working on a project and found my cross-validation error rate very low, but the testing set error rate very high. This might indicate that my model is overfitting. Why does my cross-validation not overfit while my test set overfits?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, I have about 2 million observations with 100 variables (n&gt;&gt;p). I randomly split the data set into 80/20 train and test. Then, I fit a model (i.e. XGboost) using a 5-fold cross validation on the training set and the estimated error rate is pretty low. Then, I used the same parameter setting and used the entire training set to fit the model. Surprisingly, when I used the test set to evaluate the performance of the model, the error rate is significantly higher than the CV error rate. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(About the error rate)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;error rate&lt;/em&gt; is actually multinomial logloss. I achieved a CV error rate of 1.320044 (+/- 0.002126) and a testing error rate of 1.437881. They might seem close by staring at these two numbers, but actually, they are not. I don't know how to justify this, but I am sure that they are different within the scale of performance of this project, which is from ~1.55 to ~1.30.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way of 5-fold cross validation is like following,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;divide the train set into 5 sets.&lt;/li&gt;&#xA;&lt;li&gt;iteratively fit a model on 4 sets and test the performance on the rest set.&lt;/li&gt;&#xA;&lt;li&gt;average the performance of all the five iterations.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I mean, if my parameter settings make model overfit, then I should see it at this cross-validation procedure, right? But I don't see it until I use the test set. Under what circumstances on the earth this could happen?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Added:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only reason I could think of that why CV error rate diffs from test set error rate is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict!&lt;/em&gt; &lt;em&gt;-- &lt;a href=&quot;https://stats.stackexchange.com/questions/193661/is-cross-validation-enough-to-prevent-overfitting?rq=1&quot;&gt;here&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I randomly 8/2 split the 2-million-sample data set and I believe that the train set and test set should have a same distribution of variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Pardon me that I post the same question &lt;a href=&quot;https://stats.stackexchange.com/questions/264855/why-k-fold-cross-validation-cv-overfits-or-why-discrepancy-occurs-between-cv?noredirect=1#comment506773_264855&quot;&gt;here&lt;/a&gt; also.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(About the data leakage)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got one interesting commend from @darXider in &lt;em&gt;cross validated&lt;/em&gt;. He says,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Sometimes when feature engineering, you have to be careful to avoid any data leak between training and test sets. For example, if you do a&#xA;  PCA on your original, untouched data, use PC1 and PC2 as &quot;new&quot;&#xA;  features, and then split your dataset into train and test, you are&#xA;  leaking information from training set into test set. That will boost&#xA;  your score up. You mentioned that after some feature engineering, your&#xA;  CV score and test score started to disagree. That could suggest some&#xA;  sort of information leak between training set and test set.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I wonder what is exactly the &quot;data leakage&quot; and why feature engineering &lt;strong&gt;BEFORE&lt;/strong&gt; splitting can still cause &quot;data leakage&quot;??&lt;/p&gt;&#xA;" OwnerUserId="25867" LastEditorUserId="29169" LastEditDate="2019-06-09T15:41:06.247" LastActivityDate="2019-06-09T15:41:06.247" Title="Why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?" Tags="|cross-validation|overfitting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17290" PostTypeId="2" ParentId="17287" CreationDate="2017-03-01T22:19:19.593" Score="6" Body="&lt;p&gt;By transforming both your signal and kernel tensors into frequency space, &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolution_theorem&quot; rel=&quot;noreferrer&quot;&gt;a convolution becomes a single element-wise multiplication&lt;/a&gt;, with no shifting or repeating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can convert your data and kernel into frequencies using FFT, multiply them &lt;em&gt;once&lt;/em&gt; then convert back with an inverse FFT. There are some fiddly details about aligning your data first, and correcting for gain caused by the conversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a good FFT library, this can be very efficient, but there is overhead cost for running the Fourier transform and its inverse, so your convolution needs to be relatively large before it is worth looking at FFT.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have explored this a while ago in a &lt;a href=&quot;https://github.com/neilslater/convolver&quot; rel=&quot;noreferrer&quot;&gt;Ruby gem called convolver.&lt;/a&gt; You can see some of the code for &lt;a href=&quot;https://github.com/neilslater/convolver/blob/master/lib/convolver.rb#L44-L59&quot; rel=&quot;noreferrer&quot;&gt;an FFT-based convolution here&lt;/a&gt; and the project includes unit tests that prove that direct convolution gets same numerical results as FFT-based convolution. There is also code that attempts to estimate when it would be more efficient to calculate convolutions directly by repeated multiplications or use FFT-based solution (that is rough and ready guesswork though, and implementation-dependent). &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-01T22:29:01.930" LastActivityDate="2017-03-01T22:29:01.930" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17291" PostTypeId="2" ParentId="17281" CreationDate="2017-03-01T22:28:29.430" Score="2" Body="&lt;p&gt;It sounds like you are using Welch's t-test to compare groups of different sizes. That should work in your situation. It's more reliable that the standard Student's t-test when you have unequal variances and unequal sample sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that sample sizes do not have to be identical for a Student's t-test. The important thing is that the variances are the same. If the variances are not equal, this is where the extra robustness of Welch's becomes helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether or not you can conclude significance depends on what P-value you are you are looking for. It sounds like you are targeting p&amp;lt;.01, so if your result is less than .01, you can report that you found a significant difference.&lt;/p&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-03-01T22:28:29.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17292" PostTypeId="2" ParentId="17281" CreationDate="2017-03-01T22:31:06.273" Score="2" Body="&lt;p&gt;it does not have anything to do with whether t test or welch t test. As long as your group A samples estimate the mean and standard deviation of population A very well, and group B samples estimate the mean and standard deviation of population B very well, then there shouldn't be any problem. If the SD of population A is &lt;strong&gt;assumed&lt;/strong&gt; different from that of population B, then you better go welch t test. Your question may be better asked at cross-validated, who cares more statistics, I think.&lt;/p&gt;&#xA;" OwnerUserId="25867" LastEditorUserId="25867" LastEditDate="2017-03-01T22:48:48.277" LastActivityDate="2017-03-01T22:48:48.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17293" PostTypeId="2" ParentId="17282" CreationDate="2017-03-02T02:00:56.863" Score="33" Body="&lt;p&gt;&lt;code&gt;xgboost.train&lt;/code&gt; will ignore parameter &lt;code&gt;n_estimators&lt;/code&gt;, while &lt;code&gt;xgboost.XGBRegressor&lt;/code&gt; accepts. In &lt;code&gt;xgboost.train&lt;/code&gt;, boosting iterations (i.e. &lt;code&gt;n_estimators&lt;/code&gt;) is controlled by &lt;code&gt;num_boost_round&lt;/code&gt;(default: 10)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, the first code will do 10 iterations (by default), but the second one will do 1000 iterations. There won't be any big difference if you try to change &lt;code&gt;clf = xg.train(params, dmatrix)&lt;/code&gt; into &lt;code&gt;clf = xg.train(params, dmatrix, 1000)&lt;/code&gt;, &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train&quot; rel=&quot;noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor&quot; rel=&quot;noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-02T02:00:56.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17294" PostTypeId="1" CreationDate="2017-03-02T07:04:41.123" Score="9" ViewCount="12867" Body="&lt;p&gt;I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are splitted by whitespace, which may well cause accuracy problems in subsequent processing. So I want to get all the most interesting/informative collocations in these texts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any good packages for doing multi-word tokenization, regardless of specific programming language?&lt;/strong&gt; For example, &quot;He studies Information Technology&quot; ===&gt; &quot;He&quot; &quot;studies&quot; &quot;Information Technology&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've noticed NLTK (Python) has some related functionalities. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;collocations Module:&#xA;&lt;a href=&quot;http://www.nltk.org/api/nltk.html#module-nltk.collocations&quot; rel=&quot;noreferrer&quot;&gt;http://www.nltk.org/api/nltk.html#module-nltk.collocations&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;nltk.tokenize.mwe module:&#xA;&lt;a href=&quot;http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.mwe&quot; rel=&quot;noreferrer&quot;&gt;http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.mwe&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What the difference between these two?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The MWETokenizer class in the nltk.tokenize.mwe module seems working towards my objective. However MWETokenizer seems to require me to use its construction method and .add_mwe method to add multi-word expressions. &lt;strong&gt;Is there a way to use external multi-word expression lexicon to achieve this? If so, are there any multi-word lexicon?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="3443" LastEditorUserId="13727" LastEditDate="2017-03-02T11:07:40.813" LastActivityDate="2019-02-08T15:52:38.203" Title="NLP: What are some popular packages for multi-word tokenization?" Tags="|nlp|nltk|tokenization|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17296" PostTypeId="2" ParentId="17274" CreationDate="2017-03-02T07:57:51.863" Score="1" Body="&lt;p&gt;I will rephrase you setup to make sure I understood it correctly.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You have two time series $x = (x_1, x_2, .., x_n)$ and $y = (y_1, y_2,..,y_n)$, where $n=4500$. You want to understand if $x_t = y_{t-1}$, where $t \in [1,n]$ and by $=$ means &lt;em&gt;approximately equal in statistical sense&lt;/em&gt;. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is one way but you have to make some assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general a time series can be thought as a signal plus additional noise. The signal is an explicit function which models some phenomenon and the noise is some alteration of the output which is random by nature. Using this model you split $x_t$ and $y_{t-1}$ into something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x_t = s^{(x)}_t + w^{(x)}_t$$&#xA;$$y_{t-1} = s^{(y)}_{t-1} + w^{(y)}_{t-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $s^{(x)}_t$ is the signal of $x$, $w^{(x)}_t$ is the noise of $x$, respectively $s^{(y)}_{t-1}$ is the signal of $y$, and $w^{(y)}_{t-1}$ is the noise in $y$. Notice that $s^{(x)}_t$ and $s^{(y)}_{t-1}$ are well determined and noises are random. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this notation the idea of &lt;em&gt;approximately equal in statistical sense&lt;/em&gt; means that the signal is the same and if there are difference in samples, than the differences are caused by random components (aka. noises). This means that if you assume that signals are the same and remains only the noise. So what your null hypothesis is $H_0 : x_t-y_{t-1}=0$ can be written as:&#xA;$$H_0 : s^{(x)}_t + w^{(x)}_t - s^{(y)}_{t-1} - w^{(y)}_{t-1} = 0$$ &#xA;$$H_0 : w^{(x)}_t - w^{(y)}_{t-1} = 0$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now here comes the assumptions. You have two sorts of random variables (noises) which pose some difficulties when you are trying to apply tools based on central limit theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One problem is that the random noises depends on time. You have to assume that the noise does not change with time, in both series. This is translated as independent and identically distributed or ($w^{(x)}_t = w^{(x)}$ and $w^{(y)}_{t-1} = w^{(y)}$). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;strong&gt;if you assume that noises are independent and identically distributed&lt;/strong&gt; your question can be worked out with a paired difference test. This means you can test if the differences depart significantly from $0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a t-test &lt;a href=&quot;https://en.wikipedia.org/wiki/Student%27s_t-test#Dependent_t-test_for_paired_samples&quot; rel=&quot;nofollow noreferrer&quot;&gt;dependent t-test for paired samples&lt;/a&gt; if you additionally assume that the difference is normally distributed. You cannot use z-test on paired differences because you do not know the variance of the difference's distribution (when you use sample variance for variance in a z-test, you fall into a t-test). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do not want to assume that the differences are normally distributed (that can be checked graphically at least with histograms or qq plots for skewness) you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wilcoxon signed-rank test&lt;/a&gt;, which should be fine for the number of samples you have. I would pick this test, for the same reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Attention&lt;/strong&gt;: I restate again that those tests are valid if an only if the noises in both series are independent of time. Also you can test only if they different. You cannot prove that they are equal, you can only state that you do not have evidence that they are different, which for most cases should be enough.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2017-03-02T07:57:51.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17297" PostTypeId="1" CreationDate="2017-03-02T07:58:02.700" Score="1" ViewCount="237" Body="&lt;p&gt;I am working on a problem to identify anomaly in Network. I am stuck at how to handle the following issues &#xA;1. VPN Land Based Violation (Login from Multiple locations within unrealistic situation&#xA;2. VPN Timebased violation- Multiple logins at the same time&#xA;I know we can figure this out using a simple sql query.. but How do we deploy the model in production to find these? Like how do we deploy a model to find out if the person has logged in Asia and in the next hour login from Europe. &#xA;Aprreciate any help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will collect data from Spark using Kafka stream and then will run a machine learning model to detect Anomalies. I will be building a behaviour profile for each user. I need help with How do i actually set it up so that the data can be collected from spark continuously and exported to cloud or should I use Spark MLlib to run the model and then export the results(or threats) to a database which can be viewed by Security? or Just use Spark MLlib and setup such that alerts can be sent to the network team. New to Spark and cloud sorry if I am not clear&#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="29527" LastEditorUserId="29527" LastEditDate="2017-03-06T08:54:51.850" LastActivityDate="2017-03-06T08:54:51.850" Title="Network Anomaly detection" Tags="|machine-learning|data-mining|anomaly-detection|prediction|" AnswerCount="0" CommentCount="2" ClosedDate="2017-03-03T14:34:14.967" ContentLicense="CC BY-SA 3.0" />
  <row Id="17299" PostTypeId="1" CreationDate="2017-03-02T10:07:54.570" Score="1" ViewCount="266" Body="&lt;p&gt;I must say Orange has really done great job in the world of machine learning. I have 3 questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) I have found &lt;strong&gt;&quot;Orange Infrared&quot;&lt;/strong&gt; has been added recently. But I haven't found any kind of data regarding infrared spectra. I would like to know how to get that Infrared data? Whether the IR spectra data from &lt;strong&gt;Gaussian&lt;/strong&gt; software can be used to be trained in Orange?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) I have seen a video orange has Neural Network classifier, But I downloaded and installed orange 3.3.12, No neural network model has been found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) What is CN2 rule Induction, by any means is it related to Convolution Neural Network. &lt;/p&gt;&#xA;" OwnerUserId="29533" LastEditorUserId="29169" LastEditDate="2021-02-16T02:48:11.953" LastActivityDate="2021-02-16T02:48:11.953" Title="Orange Infrared" Tags="|machine-learning|python|neural-network|convolutional-neural-network|orange|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17300" PostTypeId="2" ParentId="17294" CreationDate="2017-03-02T11:06:55.463" Score="1" Body="&lt;p&gt;The tokenization process shouldn't be changed even when you are interested in multi words. After all, the words are still the basic tokens. What you should do it to find a way to combine the proper words into term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple way to do so is to look for term in which the probability of the term is higher than that of the independent tokens. &#xA;For example P(&quot;White house&quot;) &gt; P(&quot;White&quot;)*P(&quot;House&quot;)&#xA;Choosing the proper values of need lift, number of occurrences and term classification can be deduce if you have a dataset of terms form the domain.&#xA;If you don't have such a domain then requirement at least 10 occurrences and and a lift of at least 2 (usually it is much higher since each token probability is low) will work quite well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case can can also extract terms by combining contexts relevant to your domain (e.g., &quot;studied X&quot;, &quot;practiced Y&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, you can build complex and elegant models for that but usually, looking for the few next words after the context indicators will be very beneficial.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-02T11:06:55.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17303" PostTypeId="1" CreationDate="2017-03-02T11:42:44.543" Score="6" ViewCount="18326" Body="&lt;p&gt;I trained a LSTM network on a time series dataset. Predictions seem to follow the dataset. In fact, they are nearly a right shifted form of real values. Thus, in my opinion, it doesn't provide any predictive capability since only shifting a signal in time results in low RMSE but is not useful.&lt;/p&gt;&#xA;&lt;p&gt;How to properly evaluate a time series model? &lt;a href=&quot;https://i.stack.imgur.com/k5whR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k5whR.png&quot; alt=&quot;&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://image.ibb.co/bN34aa/flip.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;image.ibb.co&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="10469" LastEditorUserId="25180" LastEditDate="2023-01-03T13:44:43.703" LastActivityDate="2023-01-03T13:44:43.703" Title="How to evaluate performance of a time series model?" Tags="|predictive-modeling|time-series|rnn|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="17304" PostTypeId="2" ParentId="17181" CreationDate="2017-03-02T12:13:02.600" Score="2" Body="&lt;p&gt;For anyone interested, I'm posting my results of trying out both CNNs and MLPs -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 3x3 kernel 4 feature maps CNN consistently performed better than a 9 neuron MLP. I trained both on completely random tic-tac-toe training data size of 10kx20 and the CNN learned to to achieve ~80% win/draw against a random opponent while the MLP achieved around ~60% for the same data. A 2x2 CNN performed much worse than 3x3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training data - Inputs and outputs are same as mentioned in the question with the exception that the AI is trained only if it loses or draws the game and the dataset is chosen from randomized player for drawn games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: It is possible to achieve much better accuracy with less data but in this case my goal was to just use this game to learn machine learning programming so the code was not particularly optimized for anything. The comparison of MLPs and CNNs is also incidental and needs to be done with more rigor for any serious purposes.&lt;/p&gt;&#xA;" OwnerUserId="29365" LastActivityDate="2017-03-02T12:13:02.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17305" PostTypeId="1" CreationDate="2017-03-02T13:08:05.060" Score="2" ViewCount="430" Body="&lt;p&gt;In a regression model is there a need to scale/normalize a dependent/response variable when doing the same for independent variables? If not then why?&#xA;An example will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="29542" LastActivityDate="2017-03-02T13:08:05.060" Title="Scaling dependent variables" Tags="|feature-scaling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17306" PostTypeId="2" ParentId="17303" CreationDate="2017-03-02T13:08:23.893" Score="3" Body="&lt;p&gt;The best summary on evaluating time series forecast is probably explained in detail on &lt;a href=&quot;https://www.otexts.org/fpp/2/5&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rob Hyndman's site&lt;/a&gt;. I typically use the mean absolute percentage error which is baked in Keras. However, what I found in a different setting is that the MAPE prevents the neural network from converging if combined with the &lt;a href=&quot;https://arxiv.org/abs/1412.6980&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adam optmization&lt;/a&gt;. I had much better success with the rooted mean square error (RMSE). Since you have poor experience with that maybe you could use the symmetric MAPE.&lt;/p&gt;&#xA;" OwnerUserId="12384" LastActivityDate="2017-03-02T13:08:23.893" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17308" PostTypeId="1" AcceptedAnswerId="17390" CreationDate="2017-03-02T14:42:27.017" Score="3" ViewCount="1238" Body="&lt;p&gt;I have installed TensorFlow on Linux (Anaconda) by following the documentation which states that one should create and activate a virtual environment &lt;code&gt;tensorflow&lt;/code&gt;. So far, so good (albeit it is not entirely clear why is this virtual environment is necessary when I want to incorporate TF to my existing env).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I activate the tensorflow environment I observe that several packages are unavailable in the new environment while available outside the environment at the same time:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$source activate tensorflow&#xA;$python&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt;import h5py&#xA;&#xA;... No module named h5py&#xA;&#xA;$source deactivate tensorflow&#xA;$python&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt;import h5py&#xA;&amp;gt;&amp;gt;&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No problem in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guessed that I should install the missing package in the tensorflow environment as well, but when I try to, I get informed that the package in question has already been installed and nothing happens:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$source activate tensorflow&#xA;$pip install h5py&#xA;&#xA;Requirement already statisfied ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The same inconsistency occurs with several other packages. What is the problem here?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-03-06T19:44:25.223" Title="Different available packages in TensorFlow virtualenv?" Tags="|python|tensorflow|linux|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17309" PostTypeId="1" CreationDate="2017-03-02T15:05:55.370" Score="-1" ViewCount="370" Body="&lt;p&gt;I have just recently started creating crawlers to scrape data. As I was creating a script yesterday, a question popped in my head that I have still not been able to find online : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a difference in using different web browsers(Chrome vs Firefox vs Opera, etc.) when scraping data, or automating a task (for instance, automating clicking through a set of links that ultimately lead to downloading a file) ?&lt;/strong&gt;      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, I'm fairly new to scraping and automation so I just want to get a sense of the best way to go about doing such things. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure if this matters, but I'm utilizing python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any resources/reading material would also be greatly appreciated! &lt;/p&gt;&#xA;" OwnerUserId="29479" LastActivityDate="2017-03-02T16:22:28.063" Title="Is there a difference in scraping across different web browsers?" Tags="|scraping|" AnswerCount="1" CommentCount="2" ClosedDate="2017-03-02T19:05:24.893" ContentLicense="CC BY-SA 3.0" />
  <row Id="17310" PostTypeId="2" ParentId="17286" CreationDate="2017-03-02T15:11:52.067" Score="3" Body="&lt;p&gt;Maybe this link will give you an explanation on how to compute the memory usage of an arbitrary neural network. Bellow in the link is explained the memory usage of the VGGNet model. Click &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/#case&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and scroll down a bit))&lt;/p&gt;&#xA;" OwnerUserId="28272" LastActivityDate="2017-03-02T15:11:52.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17313" PostTypeId="2" ParentId="17309" CreationDate="2017-03-02T16:15:40.603" Score="3" Body="&lt;p&gt;If you are using a scraping tool that manages a web session for you, you should be able to get most things done using a scraper. You should be able to log in to sites and services and download the content that you are interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The biggest difference between scraping directly in Python using BeautifulSoup or something similar is that the basic Python web scrapers will not execute active content on a page. That is, if a page uses Javascript to load some dynamic content and then display it, this content will not be available to you from your scraper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may not matter for the sites that you are working with, but more and more sites are relying on Javascript to function, so you may find that you eventually need a scraper that can run Javascript.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do need to execute Javascript from your scraper, you'll want use full headless browser to do your scraping. Some examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://phantomjs.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PhantomJS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://slimerjs.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SlimerJS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;Headless Chromium&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In terms of actual differences between browsers, there should be minimal differences at this point unless you are dealing with fairly cutting edge features (real time communication, live streaming, etc).&lt;/p&gt;&#xA;" OwnerUserId="15620" LastEditorUserId="15620" LastEditDate="2017-03-02T16:22:28.063" LastActivityDate="2017-03-02T16:22:28.063" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17314" PostTypeId="1" AcceptedAnswerId="17971" CreationDate="2017-03-02T16:46:05.247" Score="20" ViewCount="1972" Body="&lt;p&gt;Are there any meanings for the dimensions of a t-sne embedding? Like with PCA we have this sense of linearly transformed variance maximizations but for t-sne is there intuition besides just the space we define for mapping and minimization of the KL-distance?&lt;/p&gt;&#xA;" OwnerUserId="28384" LastActivityDate="2017-04-01T01:01:10.493" Title="Are t-sne dimensions meaningful?" Tags="|dimensionality-reduction|tsne|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17315" PostTypeId="2" ParentId="17263" CreationDate="2017-03-02T17:27:27.153" Score="5" Body="&lt;p&gt;You need a lot more information to figure out what's going on here. Some possibilities include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your &quot;hard&quot; examples are actually unlearnable. You can check this by seeing if you can at minimum overfit the hard examples, and if you as a human can label the hard examples correctly.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your network is not large enough to learn the hard examples. That is, you are sure that the hard examples can be learned, it's just that the network isn't sufficiently complex to do it. Again you can check this by seeing if you as a human are able to figure out the correct labels for the hard instances.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your curriculum learning may actually be hurting be the training. At the end of training, your network is only seeing examples that it has the most trouble with. These examples will cause the largest changes in the network, and you might have short-term gradient explosion. You can check if this is happening by seeing if you are getting significantly worse performance on &quot;easy&quot; examples at the end of training on the hard examples. One solution here might be something like slowly expanding the training set after you sort by difficulty. So if you have 10k easy examples and 10k hard examples, train on the 10k easy + 1k hard until you're satisified with the network performance, then grow the training set by adding the next 1k hardest, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="6403" LastActivityDate="2017-03-02T17:27:27.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17316" PostTypeId="2" ParentId="2368" CreationDate="2017-03-02T18:36:19.047" Score="1" Body="&lt;p&gt;I don't know if this is a common/best practice, but it's another point of view of the matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have, let's say, a date, you can treat each field as a &quot;category variable&quot; instead a &quot;continuous variable&quot;. The day would have a value in the set {1, 2... ,31}, the month would have a value in {1,...,12} and, for the year, you choose a minimum and a maximum value and build a set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, as the specific numeric values of days, months and years might not be useful for finding trends in the data, use a binary representation to encode the numeric values, being each bit a feature. For example, month 5 would be &lt;code&gt;0 0 0 0 1 0 0 0 0 0 0 0&lt;/code&gt; (11 0's an a 1 in 5th position, each bit being a feature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, having, for example, 10 years in the &quot;year's set&quot;, a date would be transformed into a vector of 43 features (= 31 + 12 + 10). Using &quot;sparse vectors&quot;, the amount of features shouldn't be a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something similar could be done for time data, day of the week, day of the month...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It all depends of the question you want your machine learning model to answer.&lt;/p&gt;&#xA;" OwnerUserId="29556" LastActivityDate="2017-03-02T18:36:19.047" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17317" PostTypeId="1" AcceptedAnswerId="17387" CreationDate="2017-03-02T18:42:44.680" Score="4" ViewCount="417" Body="&lt;p&gt;let me begin by saying that I understand how to build a stacked ensemble by using cross-validation to generate out-of-fold predictions for the base learners to generate meta-features. My question is about the methodology when cross-validating the entire stacked ensemble to check generalization error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To eliminate any confusion, I'm going to call the cross-validation to generate out of fold predictions for the base learner CV A, while I'll call the cross-validation of the entire stacking ensemble CV B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process? Or do I have to keep doing CV A and generate new out of fold predictions during each fold of CV B?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of. The main reason I'm asking this is because doing the second method would surely remove any data leakage but there would be an order of magnitude of additional computational complexity involved.&lt;/p&gt;&#xA;" OwnerUserId="24412" LastActivityDate="2017-03-06T13:42:23.717" Title="Cross-validation of a cross-validated stacking ensemble?" Tags="|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17320" PostTypeId="1" CreationDate="2017-03-02T23:43:21.820" Score="1" ViewCount="712" Body="&lt;p&gt;I have a question about cross-validation using sklearn in Python (2.7).&lt;br&gt;&#xA;I have updated this to include the code I use prior to cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I import a csv into a dataframe.  Some of these columns contain categories.  So I use get_dummies to convert them to binary columns (i.e., One-Hot Encoding). This retains the non-categorical, removes the categorical data, and substitutes the categorical data with binary columns. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;feature_df = pd.get_dummies(df, columns=feature_cat)&lt;br&gt;&#xA;  X = feature_df.drop(target_cat, axis=1)&lt;br&gt;&#xA;  print X   # visually confirm it is formatted correctly&lt;br&gt;&#xA;  y = df[target_cat] &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;# target_cat is a string containing the name of the column containing my target data.  I have printed this out to confirm it is correctly extracting this from the df.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;X = scale(X)  # Scaling is important for preventing misleading results&lt;br&gt;&#xA;  X = pd.DataFrame(X)   &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(I also printed this out to confirm it contains and is formatted as it is supposed to be.)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;# Note: sklearn LeaveOneOut() cross-validation needs X to be a data frame, although right now I am only using k-fold. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then I create my pipeline:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;model = DecisionTreeClassifier()&lt;br&gt;&#xA;  pipe = Pipeline([('pca', PCA()), ('clf', model)])&lt;br&gt;&#xA;  pipe.fit(X,y)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;After this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;# Make predictions on training set:&lt;br&gt;&#xA;    predictions = pipe.predict(X)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;# Print initial accuracy&lt;br&gt;&#xA;  accuracy = metrics.accuracy_score(y, predictions)&lt;br&gt;&#xA;  print &quot;Accuracy : %s&quot; % &quot;{0:.3%}&quot;.format(accuracy)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;kfold=KFold(n_splits=100)&lt;br&gt;&#xA;  acc_score = np.mean(cross_val_score(pipe, X, y, scoring='accuracy', cv=kfold))&lt;br&gt;&#xA;  print acc_score&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When I measure my classifier accuracy using accuracy_score I get: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accuracy : &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;88.416%&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And my normalized confusion matrix is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[ 0.97  0.    0.02  0.    0.01  0.01]&lt;br&gt;&#xA;  [ 0.08  0.89  0.01  0.    0.01  0.01]&lt;br&gt;&#xA;  [ 0.09  0.01  0.88  0.    0.01  0.01]&lt;br&gt;&#xA;  [ 0.06  0.02  0.04  0.84  0.01  0.02]&lt;br&gt;&#xA;  [ 0.06  0.02  0.04  0.02  0.83  0.02]&lt;br&gt;&#xA;  [ 0.06  0.01  0.04  0.02  0.04  0.83]]  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When I run cross validation using:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;kfold=KFold(n_splits=10)&lt;br&gt;&#xA;  cross_val_score(pipe, X, y, scoring='accuracy', cv=kfold) &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(pipe is PCA and DecisionTreeClassification pipeline.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It gives me a mean accuracy of:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.231595981914&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;With the accuracies for each of the 10 folds:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 0.26901521  0.20176141  0.22177742  0.29967949  0.22435897  0.24919872&#xA;    0.27163462  0.21153846  0.21794872  0.20192308]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While I would expect variation in the accuracy of the classifier, I wouldn't expect it all to be this completely different from the original accuracy, and I certainly wouldn't expect it to be consistently this different.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further test the feasibility of the these results, I shuffled the rows of my original data frame containing both the features and targets. I ran it again, and now average cross-validation accuracy is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.553633419556&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;with individual fold accuracies of:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 0.55644516  0.56044836  0.53242594  0.55849359  0.53926282  0.54326923&#xA;    0.55128205  0.55528846  0.57532051  0.54887821]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I try this a few more times by changing the seed for random and/or shuffling the shuffled dataset. My cross-validation accuracy remained pretty much the same.  For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.553075665661&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Fold accuracies:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 0.5236189   0.54923939  0.54043235  0.55769231  0.55048077  0.55208333&#xA;    0.54567308  0.55769231  0.56570513  0.58092949]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So then I increased the folds to 100, and the mean accuracy was still 0.55, and the 100 resulting accuracies from each test run were all between .49 (min) and .64(max).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I run it with 100 folds on my original, non-shuffled dataset, I get a mean cross-validation accuracy score of 0.26, with individual accuracies ranging between .09 and .52, with most being .2 or .3.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it me or is this weird?  Shouldn't these results be more consistent?  Is there something wrong with metrics.accuracy_score (which I use to compute my original classification accuracy, prior to cross-validation)? Presumably this is also what cross_val_score uses when I set scorer to accuracy...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice and/or feedback would be greatly appreciated!!!  For what it's worth, I am using PyCharm on a Mac (Sierra).  Thank you!&lt;/p&gt;&#xA;" OwnerUserId="29456" LastEditorUserId="29456" LastEditDate="2017-03-03T17:34:18.487" LastActivityDate="2017-03-03T17:34:18.487" Title="Consistently inconsistent cross-validation results that are wildly different from original model accuracy" Tags="|machine-learning|python|classification|scikit-learn|cross-validation|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17321" PostTypeId="1" CreationDate="2017-03-02T23:50:13.623" Score="1" ViewCount="1183" Body="&lt;p&gt;I would like to take a set of documents where each document already has an assigned popularity change value (&quot;trending&quot;, i.e. -10%, +25%, etc.), and create a machine learning model that would be able to predict a &quot;trending&quot; value for a new document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After creating TF-IDFs for all documents, dropping some of the rarest words that only appear in 1 document, possibly removing stop words, etc., I'm a little lost. Is it the right approach to use some kind of sentiment analysis model that would train the TF-IDF vectors on the popularity change values? I am also thinking about how to train a regular regression model for this task, but it doesn't seem like an easy choice to integrate with TD-IDF data.&lt;/p&gt;&#xA;" OwnerUserId="29562" LastActivityDate="2017-04-02T13:54:52.560" Title="TF-IDF Regression &amp; Machine Learning" Tags="|machine-learning|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17322" PostTypeId="1" AcceptedAnswerId="17331" CreationDate="2017-03-03T00:58:55.973" Score="5" ViewCount="3182" Body="&lt;p&gt;Self-Organizing Maps is a pretty smart yet fast &amp;amp; simple method to cluster data. But Self-Organizing maps were developed in 1990 and a lot of robust and powerful clustering method using dimensionality reduction methods have been developed since then. What are some of its applications in today's world of science and engineering?&lt;/p&gt;&#xA;" OwnerUserId="29564" LastEditorUserId="29564" LastEditDate="2017-03-03T17:54:12.607" LastActivityDate="2017-03-03T17:54:12.607" Title="How relevant is Self Organizing Maps in today's science?" Tags="|machine-learning|neural-network|clustering|" AnswerCount="2" CommentCount="2" ClosedDate="2017-03-03T14:33:54.190" ContentLicense="CC BY-SA 3.0" />
  <row Id="17324" PostTypeId="1" AcceptedAnswerId="17338" CreationDate="2017-03-03T06:07:26.880" Score="9" ViewCount="4201" Body="&lt;p&gt;I had an interesting discussion come up based on a project we were working on: why use a CNN visual inspection system over a template matching algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Background: I had shown a demo of a simple CNN vision system (webcam + laptop) that detected if a particular type of object was &quot;broken&quot;/defective or not - in this case, a PCB circuit board. My CNN model was shown examples of the proper and broken circuit boards (about 100 images of each) on a static background. Our model used the first few conv/maxpool layers of pre-trained VGG16 (on imagenet), and then we added a few more trainable convs/pools, with a few denses, leading to a dim-3 one hot encoded vectored output for classification: (is_empty, has_good_product, has_defective_product).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model trained pretty easily and reached 99% validation acc no problems; we also trained with various data augmentation since we know our dataset was small. In practice, it worked about 9 times out of 10, but a few random translations/rotations of the same circuit board would occasionally put it in the opposite class. Perhaps more aggressive data augmentation would have helped. Anyways, for a prototype concept project we were happy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we were presenting to another engineer and his colleague, and he brought up the argument that NNs are overkill for this, should just use template matching, why would one want to do CNNs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We didn't have a great answer for why our approach could be better in certain applications (e.g. other parts to inspect). Some points we brought up:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) More robust to invariances (through e.g. data augmentation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Can do online learning to improve the system (e.g. human can tell the software which examples it got wrong)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) No need to set thresholds like in classical computer vision algorithms&#xA;What do you guys think, are there more advantages for a CNN system for this type of inspection task? In what cases would it be better than template matching?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few more random ideas for when deep NNs could be the tech for the job: for systems that require 3D depth sensing as part of the input, or any type of object that can be deformed/stretched/squished but still be &quot;good&quot; and not defective (e.g. a stuffed animal, wires, etc). Curious to hear your thoughts :)&lt;/p&gt;&#xA;" OwnerUserId="27279" LastEditorUserId="29169" LastEditDate="2021-02-16T02:47:59.470" LastActivityDate="2021-02-16T02:47:59.470" Title="Why use convolutional NNs for a visual inspection task over classic CV template matching?" Tags="|machine-learning|image-classification|convolutional-neural-network|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17325" PostTypeId="2" ParentId="17324" CreationDate="2017-03-03T06:18:01.683" Score="5" Body="&lt;p&gt;The answer depends on the task.  Template matching can work for some tasks but not for all.  CNNs potentially have the ability to generalize to unseen inputs that don't match any of your templates, so can potentially generalize better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But whether CNNs will beat template matching will depend on the specific task and what specifically you're trying to achieve.  This is an empirical science; ultimately, the way you find out which works better is to try them both -- or learn from others who have tried them (e.g., by reading the literature).  I don't think you're going to find some theory or taxonomy that is going to substitute for empirical evaluation on real-world data.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-03T06:18:01.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17326" PostTypeId="2" ParentId="155" CreationDate="2017-03-03T06:25:45.277" Score="6" Body="&lt;p&gt;Besides all these datasets, if you are interested in data related to India.&#xA;The publicly official site of Indian Government is&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://data.gov.in/&quot; rel=&quot;noreferrer&quot;&gt;https://data.gov.in/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It provides datasets from different departments of Indian government which can be well used for Big Data Analysis &amp;amp; Machine Learning.&lt;/p&gt;&#xA;" OwnerUserId="29576" LastActivityDate="2017-03-03T06:25:45.277" CommentCount="0" CommunityOwnedDate="2017-03-03T06:25:45.277" ContentLicense="CC BY-SA 3.0" />
  <row Id="17327" PostTypeId="2" ParentId="17324" CreationDate="2017-03-03T07:35:31.420" Score="5" Body="&lt;p&gt;One problem you might run into with a NN (and other classification methods) is that since you've only shown it certain defects, it might not know how to react to completely new / yet unseen defects that might pop up in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want the NN to learn &quot;anything that doesn't look like a non-defective PCB is a defective PCB&quot;. But what if it has learned &quot;anything that doesn't look like a defective PCB is a non-defective PCB&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try to modify some images of non-defective PCBs by adding a small white spot (or another small perturbation) to them at random locations and have the neural network classify these modified images. It should definitely classify them as defective, right? But it'll probably miss some (or quite many) because it has never seen such defects before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To detect completely new defects, anomaly detection methods / one class classifiers might be more.. trustworty, because they should pick up anything that's never been seen before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As D.W. said, you're just going to have to try both methods and find out which one works better. Just make sure to have a really good test set that also contains completely new defects!&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="676" LastEditDate="2017-03-03T12:05:39.920" LastActivityDate="2017-03-03T12:05:39.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17328" PostTypeId="1" CreationDate="2017-03-03T09:02:37.583" Score="4" ViewCount="11469" Body="&lt;p&gt;How much data should we use during training, and how much in testing? Can anyone explain why does it always seem to be 70:30 or 80:20 ratios?&lt;/p&gt;&#xA;" OwnerUserId="29580" LastEditorUserId="13413" LastEditDate="2017-03-03T21:09:43.547" LastActivityDate="2017-03-08T16:55:38.703" Title="Statistics - Train and test data split" Tags="|statistics|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17329" PostTypeId="2" ParentId="17328" CreationDate="2017-03-03T09:43:07.833" Score="0" Body="&lt;p&gt;As far as reason behind train/test classification is concerned..you can have some help here &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://info.salford-systems.com/blog/bid/337783/Why-Data-Scientists-Split-Data-into-Train-and-Test&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1]: http://info.salford-systems.com/blog/bid/337783/Why-Data-Scientists-Split-Data-into-Train-and-Test&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29435" LastActivityDate="2017-03-03T09:43:07.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17331" PostTypeId="2" ParentId="17322" CreationDate="2017-03-03T10:17:22.223" Score="3" Body="&lt;p&gt;One important and well known (in their respective research domains) usage of SOMs today is the usage of SOMs for data visualization and clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The U-Matrix uses a SOM with a lot of neurons in order to achieve emergent behavior. Based upon the SOM, a so called U-height is calculated for each neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a neuron $n$, a weight $w_n$ of the neuron $n$, as well as a set $NN(n)$ of immediate neighbours of $n$ the U-height is calculated as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{m \in NN(n)} d(w_n - w_m)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$d$ is the same distance as used in the SOM training. This U-height is used to visualize the SOM and can be capable of catching quite interesting &#xA;inherent structure. It is visualized by using a color scheme similar to topographical maps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zF5va.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zF5va.png&quot; alt=&quot;U-matrix&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The U-Matrix uses distances for visualization. It can be combined with the so called P-Matrix, which uses densities, to use the density of data points as well, and allows the combination of densities and distances. Based on the U-Matrix one can do classification or clustering as well. An R package is distributed by the research group of Prof. Ultsch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't say that it's important in Machine Learning per se, but in the fields of Data Mining and KDD it is, at least academically. You can clearly see this if you look on the reception of this or similar work on semanticscholar or researchgate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: I studied with Prof. Ultsch, the head behind U-Matrix and U*-Matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: Sorry for the missing links to the papers, I haven't got enough reputation yet to post more than two links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Papers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;U*-matrix: a Tool to Visualize Clusters in High Dimensional Data, Ultsch 2004&lt;/li&gt;&#xA;&lt;li&gt;Self-Organizing Neural Networks for Visualisation and Classification, Ultsch 1993&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I even spotted the U-Matrix in the wild &lt;a href=&quot;https://github.com/jimfleming/numerai/blob/master/notebooks/Visualization.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Also see the sompy package.&lt;/p&gt;&#xA;" OwnerUserId="29582" LastActivityDate="2017-03-03T10:17:22.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17332" PostTypeId="2" ParentId="17328" CreationDate="2017-03-03T10:18:37.250" Score="6" Body="&lt;p&gt;Those rule of thumbs for ratio make no sense. The use case for a test set is to measure your performance in a real scenario on unseen data. How much you need for an accurate measure depends on the level of accuracy you need, the amount of variance you expect, not so much on the size of the original dataset. Something to be aware of is that you do decrease the size of your training set, which is a cost of testing. But when you have 10 million rows of data it makes no sense to validate on 3 million, when 10,000 could be more than enough. Outside of the cost of losing training data you should look at absolute numbers in your test set.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-03-03T10:18:37.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17333" PostTypeId="1" CreationDate="2017-03-03T10:25:17.950" Score="7" ViewCount="6780" Body="&lt;p&gt;I have a dataset with around 900.000 records, around 1000 of which are marked as positive (the studied event occurred).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability of the event occurring is always low (i.e. &amp;lt; 0.1), and I would like to create a regression model to predict the probability of the event occurring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first thought was to use logistic regression, but I am not sure if I could directly interpret the output as the probability of the event happening. The same doubt arises when using other models, such as SVM or RF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another doubt would be whether usual evaluation metrics (e.g. RMSE) would work well on such a model, since even a predictor that always outputs 0 would have a very good score.&lt;/p&gt;&#xA;" OwnerUserId="29579" LastActivityDate="2017-03-04T09:13:29.367" Title="Regression model to predict probability of rare event" Tags="|machine-learning|regression|logistic-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17334" PostTypeId="2" ParentId="17322" CreationDate="2017-03-03T11:00:06.243" Score="2" Body="&lt;p&gt;To give a practical example, still quite relevant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the field of &lt;a href=&quot;https://en.wikipedia.org/wiki/Flow_cytometry&quot; rel=&quot;nofollow noreferrer&quot;&gt;flow cytometry&lt;/a&gt;, a recent (2015) clustering algorithm, &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/cyto.a.22625/abstract;jsessionid=C5DD514B2E7C87492864945CD9F5547B.f04t01&quot; rel=&quot;nofollow noreferrer&quot;&gt;FlowSom&lt;/a&gt;,  makes use of self-organizing maps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even more recently (2016, see &lt;a href=&quot;http://onlinelibrary.wiley.com/doi/10.1002/cyto.a.23030/abstract&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), it has been shown to be amongst the top performing clustering algorithms in  the field of flow and mass cytometry. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastActivityDate="2017-03-03T11:00:06.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17335" PostTypeId="1" AcceptedAnswerId="17345" CreationDate="2017-03-03T12:45:02.607" Score="2" ViewCount="210" Body="&lt;p&gt;I have a Spark dataset containing a column of SparseVector types. Additionally, I have another SparseVector $X$ which is not a part of the dataset. I want to order my dataset according to the closest distance (or similarity) relative to $X$.&#xA;Can anyone help me with how to implement this?&lt;/p&gt;&#xA;" OwnerUserId="29534" LastEditorUserId="13413" LastEditDate="2017-03-03T21:13:11.990" LastActivityDate="2017-03-03T21:13:11.990" Title="Order SparseVectors by the closest distance to given SparseVector" Tags="|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17337" PostTypeId="2" ParentId="17321" CreationDate="2017-03-03T13:45:34.393" Score="1" Body="&lt;p&gt;I'm not sure what you already know, so I'll answer how I'd do it. If you do not understand things, or already know some things I say, please leave a comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would tackle it as a multi-class problem if your output is categorical, hence indeed only $10\%$, $25\%$ and so on. If you want to treat it as a regression problem, I'd say you need continuous deltas of popularity, so you need all kinds of values as the delta, and not only a fixed number of them. The training data should dictate how you approach the problem. Note that you can always reduce the regression to a multi-class problem (which asks for an instance which of the deltas it should assign) by bucketing to intervals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For argument's sake let's assume you have multi-class data. One document has one of the classes, i.e. $10\%$. You then go ahead and encode these classes into a binary vector, which probably even is a part of the library you're using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd then go ahead and split your data with the hold-out method to have some vlalidation examples set aside to see how your model actually works. You then go ahead and train some model, say a random forest. With the validation data you calculate some measure of error and see if it is enough. If not you go ahead and choose other models to train, other features to add and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd suggest to try use another approach than bag-of-words such as word2vec or GloVe. You can also add the sentiment classifiers values as feature columns. You could then compare the different features and see how it performs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if this helps you or if I misunderstood your question.&lt;/p&gt;&#xA;" OwnerUserId="29582" LastActivityDate="2017-03-03T13:45:34.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17338" PostTypeId="2" ParentId="17324" CreationDate="2017-03-03T13:59:36.383" Score="3" Body="&lt;p&gt;The engineer in question that proposed traditional CV methods for your application simply did so out of habit. Using template matching is extremely outdated and has been shown to perform very poorly. However, I do think a CNN is overkill depending on your dataset's size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does template matching work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Template matching&lt;/strong&gt; slides a window across your image that will provide a percent match with the template. If the percent match is above a certain predefined threshold then it is assumed to be a match. For example if you have an image of a dog and you want to determine if there is a dog in the image, you would slide a dog template around the entire image area and see if there is a sufficiently large percent match. This will likely result in very poor performance because it requires the template to &lt;strong&gt;overlap the image identically&lt;/strong&gt;. What is the likelihood of that in practice? Not very high. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only time template matching is a sufficient technique is if you know exactly what you are looking for and you are confident that it will appear almost identically in every example of a given class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why use machine learning instead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning techniques are not rigid. Unlike what stmax said, &lt;strong&gt;CNNs are able to generalize a dataset very well&lt;/strong&gt;. That is why they are so powerful. Using the dog example, the CNN does not need to see a picture of every dog in existence to understand what constitutes as a dog. You can show it maybe 1000 images from a Google search, and then the algorithm will be able to detect that your dog, is in fact a dog. The fact that machine learning algorithms generalize very well is the reason that they replaced all the ancient CV techniques. Now the problem is the amount of data that you need to train a CNN. &lt;strong&gt;They are extremely data intensive&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not think that 100 data points is sufficient to train a robust CNN. Due to the deep complexity of the model in order to limit the bias you need to increase your number of examples. I usually suggest 100 examples for every feature for deep models and 10 examples for every feature for shallow models. It really all depends on your feature-space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I suggest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are truly doing is &lt;strong&gt;anomaly detection&lt;/strong&gt;. You have a lot of examples that will be presented of PCBs that are otherwise in good shape. You want to detect those which are broken. Thus I would attempt some anomaly detection methods instead. They are much simpler to implement and you can get good results using shallow models especially in skewed datasets (1 class is over represented). &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-03T13:59:36.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17340" PostTypeId="2" ParentId="2368" CreationDate="2017-03-03T15:06:13.233" Score="3" Body="&lt;p&gt;Depending on what you are interested in with the date/time info, you might just want to bin it.  For e.g., if you are interested in distance from a starting point (e.g., Jan 1, 2015), and you want to measure it in months, I would just code it as month 1 (for Jan 1-31, 2015), 2 (Feb 1-28, 2015), 3, 4, 5, 6, etc. Since the distance between the start dates are approximately the same, this represents time distance in a straightforward continuous format.  And I say continuous because you can say month 6.5 and know that it is half-way through June, 2015. Then you don't have to worry about actual date coding and you can use all your typical classification methods.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to measure in days, I know MySql has a 'to_days' function, if you happen to use that to pull data prior to classification.  Python probably has something similar, or use the unix-time format suggested by mork.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="29456" LastActivityDate="2017-03-03T15:06:13.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17341" PostTypeId="1" AcceptedAnswerId="17343" CreationDate="2017-03-03T15:23:20.343" Score="10" ViewCount="6125" Body="&lt;p&gt;I have a time-series dataset (daily frequency) representing the sales of a product to a customer over time. The sales is represented as the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$[0, 0, 0, 0, 24, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 17, 0, 0, 0, 0, 9, 0, ...]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in which each number represents the sales of the product in a day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that time-series forecast methods (ARMA, HoltWinters) work well for &quot;continuous&quot; and &quot;smooth&quot; data, but is not producing good results in this case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to make a forecast of that series, with attention to 2 points: (1) assuring non-negative values and (2) sparse/ non-continuous data. Anyone knows how to approach this problem? What methods/ technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="15215" LastActivityDate="2021-11-19T04:15:15.007" Title="Forecasting non-negative sparse time-series data" Tags="|time-series|forecast|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17343" PostTypeId="2" ParentId="17341" CreationDate="2017-03-03T17:37:50.040" Score="4" Body="&lt;p&gt;I have two ideas here, maybe they will be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Idea 1: Model time between events&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might think of your data as being generated by two processes: the first is a distribution over time intervals, and the second is a distribution over purchase amounts. So to model your data you could create one distribution (gaussian?) over the nonzero values in your dataset, and another over the lengths of sequences of zeros (poisson?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Idea 2: Model customer inventory&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though the sales events in your dataset are sparse, you could spend a little time to come up with a model of why the customer is making purchases when they do. In one possible model, the customer has an inventory that shrinks over time, and they make purchases when their inventory crosses some minimum threshold. You could use your sales data to fit the slope (for linear shrinkage) or rate (for exponential shrinkage) as well as the threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could get arbitrarily complex, since the customer under this model might have different thresholds or shrink rates at different times ... but for starters it could be a useful approach to get a sense of things.&lt;/p&gt;&#xA;" OwnerUserId="4647" LastActivityDate="2017-03-03T17:37:50.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17344" PostTypeId="2" ParentId="17276" CreationDate="2017-03-03T18:58:03.917" Score="3" Body="&lt;p&gt;It's probably possible.  I'll suggest a few plausibly-practical methods, starting from very crude (but probably not super-effective) to more complex (might be more effective but might be computationally expensive).  I expect that many of these approaches might be relatively slow to train the classifier but doable and worth a try.  But first, let me try to formalize the  problem more precisely.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Theory&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In machine learning it's often helpful to formulate your problem by &lt;em&gt;identifying a loss function&lt;/em&gt;.  Then you can formulate your problem as finding $f$ that minimizes the loss function.  In your case, I think a natural loss function would be that $L(f,g)$ counts the number of partitions $p$ such that conditions (2) or (3) are violated.  You could of course add a regularization term as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've defined a loss function, your problem becomes: given $g$, find $f \in \mathcal{H}$ that minimizes $L(f,g)$, where $\mathcal{H}$ is the hypothesis space (the space of valid/legal models).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's start with a theoretical thought experiment.  Suppose you didn't care about the computational time to train a classifier, and just want to find a valid classifier $f$.  Let $\mathcal{H}$ be the hypothesis space, i.e., the set of allowable functions.  (For instance, for a linear SVM, $\mathcal{H}$ would be the set of SVM weights -- i.e., the set of linear decision boundaries.)  If you didn't care about computational time, you could just enumerate all $f$ to find one that minimizes $L(f,g)$ -- so this proves that the problem is well-defined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, suppose we have a space of classifiers $f$ that output probabilities instead of hard labels (think: logistic regression).  Then we can define a loss function based on the cross-entropy loss:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(f,g) = \sum_{p : g(p)=0} \max \{-\log(1-f(x)) : x \in p\} + \sum_{p : g(p)=1} \min \{-\log(f(x)) : x \in p\}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could add a regularization term here, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, other loss functions are possible as well; but the point is that once you have a loss function, then classification becomes well-defined as an optimization problem.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Plausible method: one-class classifier&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;One crude approach is to use a one-class classifier.  For each partition $p$ such that $g(p)=0$, you obtain many instances $x$ where you want to have $f(x)=0$.  So, you could add each one of those to a training set, and then train a one-class classifier on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is simple and easy to implement.  However, I suspect its accuracy might be low.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Plausible method: active learning&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a way to obtain labelled instances if you need, through manual training.  In particular: let's assume that given a partition $p$, if you really needed to, you could construct hard labels for instances $x \in p$, and specifically, if $g(p)=1$, you could find an instance $x \in p$ that should receive the label $1$.  This manual-labelling process might be difficult and tedious and time-consuming/expensive, so not feasible to do on a large scale, but let's suppose you could do it on a small scale if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you could use active learning:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Let $f$ be an arbitrary classifier (maybe trained using a one-class classifier).  Let $T$ be an initial training set, initially empty.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat until convergence:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. Pick a partition $p$ such that $f(x)$ is wrong for some $x \in p$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. If $g(p)=0$, add find $x \in p$ such that $f(x)=1$, and add $(x,0)$ to $T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. If $g(p)=1$, use the manual-labelling process to find some $x \in p$ that should be labelled $1$ and add $(x,1)$ to $T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d. Train a new boolean classifier $f$ using standard supervised learning methods on the training set $T$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are various optimizations you could use to break ties in step 2c (in hopes of reducing the amount of manual labelling); e.g., if you're using the cross-entropy loss defined above, you could choose the $p$ that contributes the most to the total loss for $f$.  Or, you could choose the $p$ that is most &quot;different&quot; from all previously considered $p$ with $g(p)=1$.  Or, you might be able to adapt other active learning methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might work.  However, it does require the ability to do some manual labelling of individual instances, which is more than you promised us in the original problem statement.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Plausible approach: direct optimization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Another possible approach would be to choose a differentiable classifier, and the optimize the (cross-entropy) loss $L(f,g)$ defined above directly using gradient descent.  For instance, you could choose $f$ to be a logistic regression or neural network classifier, as these are differentiable.  Now, you can compute the gradient of $L(f_\theta,g)$ with respect to the parameters $\theta$ of the model (e.g., for logistic regression, $\theta$ are the weights) and then apply gradient descent to find the model parameters $\theta$ that minimize $L(f_\theta,g)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may get a little messy, as loss function $L$ contains $\min$ and $\max$ functions, which have points where they are non-differentiable.  If you don't do anything special, this can cause oscillation around the non-differentiable boundary.  One possible trick is to replace each $\min$ by a &lt;a href=&quot;https://math.stackexchange.com/q/2103911/14578&quot;&gt;softened version of the min&lt;/a&gt;, and replace each $\max$ by a softened version of the max, so that the loss function becomes differentiable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could adjust ordinary gradient descent to behave well around them.  One standard technique is to reduce the step size when you get near the boundary to avoid stepping across the boundary, or to use projected gradient descent.  Thus, if we have a term&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\ell(\theta) = \min(\ell_1(\theta),\ell_2(\theta))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we compute the gradient $\nabla \ell(\theta)$, the gradient will be based on whichever of $\ell_1(\theta),\ell_2(\theta)$ is smaller for this particular value of $\theta$.  Suppose the gradient is $g = \nabla \ell(\theta)$.  Ordinary gradient descent would take a step in the direction $g$, say adjusting $\theta$ to $\theta - \lambda \cdot g$ for some small constant $\lambda$.  You could additionally require that the step avoid crossing the boundary line $\ell_1(\theta)=\ell_2(\theta)$.  For instance, if $\ell_1(\theta)&amp;lt;\ell_2(\theta)$ for this particular value of $\theta$, we'll have $g = \nabla \ell(\theta) = \nabla \ell_1(\theta)$.  So, take a step $- \lambda' \cdot g$, but constrain $\lambda'$ to be in the range $0 \le \lambda' \le \lambda$ and to satisfy $\ell_1(\theta - \lambda' \cdot g) \le \ell_2(\theta - \lambda' \cdot g)$ -- i.e., don't step across the other side of the discontinuity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that using ordinary gradient descent will probably be very slow, as each evaluation of the loss function requires evaluating $f$ on every point in $X$: $O(|X|)$ time.  You'll probably do better by using stochastic gradient descent: in each iteration, pick a single partition $p$, compute the gradient of the term in $L(f,g)$ for $p$, and move in the direction of that gradient.  Or, pick a minibatch of 20 partitions $p$ and compute the gradient of the sum of those 20 terms.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Plausible method: boosting&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Another reasonable method would be to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Boosting_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;boosting&lt;/a&gt; with a very simple classifier, such as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_stump&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision stump&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A decision stump is a decision tree with a single level: you pick one feature, compare it to some threshold, and make a binary decision based on whether the value of that feature is above or below the threshold.  Decision stumps are convenient because you can enumerate all candidate stumps -- all possible combinations of a feature and a threshold -- and evaluate the loss function for each.  (In practice, as an optimization, we don't enumerate all possible stumps.  Rather, for a particular feature, we randomly pick 100 candidate thresholds by sampling 100 times from the training set, and we compute the loss for each candidate threshold; we do this for each feature, and take the best overall combination.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow noreferrer&quot;&gt;AdaBoost&lt;/a&gt; directly with decision stumps and minimize the loss function defined above, and you might get a reasonable classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For even better results, you could use &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow noreferrer&quot;&gt;gradient boosting&lt;/a&gt;.  This is a slight generalization where each leaf of each stump outputs a continuous value rather than a boolean; the classifier's output is obtained by summing the value from each stump, and then checking whether the sum is positive or negative.  The general methodology can be applied with any differentiable loss function, so you could apply that methodology to the cross-entropy loss function defined above.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Plausible method: incremental optimization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Here's another approach that can be applied to classifiers that aren't differentiable.&#xA;In my algorithm, I will maintain a tentative training set $T \subseteq X \times \{0,1\}$ that assigns a tentative label to some of the elements of $X$.  The training set $T$ will evolve throughout the algorithm, but it will always remain &lt;em&gt;consistent&lt;/em&gt; with $g$, defined as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We say that $T$ is consistent with $g$ if, (a) for every $p$ such that $g(p)=0$, we have $(x,0) \in T$ for all $x \in p$, and (b) for every $p$ such that $g(p)=1$, there exists at least one $x \in p$ such that $(x,1) \in T$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training set $T$ specifies one possible labelling that would be consistent with $g$.  It's not necessarily the right one; it's just one possibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each step, the algorithm uses $T$ as labels to train a classifier $f$ using standard supervised learning methods and computes $f$'s loss on $T$, then adjusts $T$ to reduce the loss of $f$ on $T$.  Define $\text{train}(T)$ to be the classifier $f$ obtained by training on the training set $T$, using some boolean classifier of your choice.  Here is the algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Set $T$ to be any training set that is consistent with $g$.  Compute $f := \text{train}(T)$ and $\ell = L(f,g)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat until convergence:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. Enumerate all training sets $T'$ that differ from $T$ in only one element yet remains consistent with $g$, and for each train a classifier and compute the loss.  Pick the $T'$ that minimizes $L(\text{train}(T'),g)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. Set $T := T'$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Step 2a can be done by enumerating all partitions $p$ such that $g(p)=1$. Suppose $T$ includes $(x,1)$ where $x \in p$.  Then enumerate all $x' \in p$ and consider $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  There are at most $|X|$ such possibilities, and you have to train the classifier on each, so each iteration of the loop might be pretty slow... but one could hope that it will eventually converge on a reasonable solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, this tried to maintain some tentative labels that are consistent with your $g$, and that also are consistent with the regularity conditions imposed by the classifier $f$ (e.g., linear separability, or whatever), incrementally adjusting it at each step of the iteration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your particular parameter values ($|X|=10^5$, $|P|=10^3$), I suspect this will be too slow.  But you could try experimenting with the approach for scaled-down versions of your problem, with a smaller feature space ($|X|$ is smaller) and see if it seems to lead to useful results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this algorithm gives useful results, there many heuristics and optimizations you could use to speed up convergence:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You could probably make a smarter choice for the initial value of $T$ and speed up convergence, by applying a one-class classifier to the set of 0-labelled instances (since the instances of the form $(x,0) \in T$ never change throughout the algorithm) and using that to choose tentative initial label.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You could use something akin to stochastic gradient descent to speed convergence: in step 2a, pick a single $p$ (or a small minibatch of $p$'s) and  consider just training sets $T'$ that differ from $T$ only in $p$ (but not in other partitions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You could try using an online/incremental training method for training $f$.  Since $T'$ is so similar to $T$, rather than training a new classifier from scratch, you might be able to reuse much of the effort from training on $T$ to more quickly compute $\text{train}(T')$ given $\text{train}(T)$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Depending on the nature of the feature space and the partition function, you might be able to solve the following problem more efficiently than brute force: given a training set $T$ and a partition $p$ such that $g(p)=1$ and an $x \in p$ such that $(x,1) \in T$, find some $x' \in p$ such that minimizes the loss when training on $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  For instance, with logistic regression, you might be able to train a classifier $f_0$ on $T \setminus \{(x,1)\}$, then classify each $x' \in p$ using $f_0$ and rank them by the probability score that logistic regression outputs.  This is a heuristic and not guaranteed to find the optimal $x'$, but it's also a lot more efficient than enumerating all candidates for $T'$ and training a classifier on each.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;A useful subroutine&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Many of the above schemes require solving the following subproblem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a partition $p$ and a classifier $f$, find $x \in p$ such that $f(x)=1$, or report that none exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the classifier outputs confidence/probability scores, then the subproblem is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a partition $p$ and a classifier $f$, find $x \in p$ that maximizes $f(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The naive way to solve this subproblem is to enumerate all $x \in p$.  Depending on the particular classifier you choose and the structure of the partitions, there may be more efficient algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if we have a linear classifier, maximizing $f(x)$ becomes equivalent to maximizing $x \cdot w$ for some known vector $w$.  If $p$ has a simple structure, there might be faster ways to solve this optimization problem.  For instance, if $p$ is a convex region defined by linear inequalities, you can maximize $x \cdot w$ subject to $x \in p$ using linear programming.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can use a fast algorithm for this subproblem to speed up the training process, for many of the methods sketched above.  Whether that is possible will depend on the specific type of classifier you choose and the structure of the partitions, but if you're trying to do this in practice, I recommend you ask a separate question about how to solve this subproblem for the particular type of classifier and partition structure that arises in your application -- that might enable significant computational optimizations.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:19:36.717" LastActivityDate="2017-03-03T21:23:56.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17345" PostTypeId="2" ParentId="17335" CreationDate="2017-03-03T19:06:00.853" Score="1" Body="&lt;p&gt;it appears that problem was solved with &lt;a href=&quot;https://spark.apache.org/docs/2.1.0/api/java/index.html?org/apache/spark/ml/feature/BucketedRandomProjectionLSH.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;BucketedRandomProjectionLSH&lt;/a&gt;. After &lt;strong&gt;fit&lt;/strong&gt; &amp;amp;&amp;amp; &lt;strong&gt;transformation&lt;/strong&gt;, and &lt;strong&gt;approxNearestNeighbors&lt;/strong&gt; resulted dataset contains distCol, which (as per &lt;a href=&quot;https://fossies.org/linux/spark/mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSH.scala&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;@param distCol Output column for storing the distance between each result row and the key.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29534" LastActivityDate="2017-03-03T19:06:00.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17346" PostTypeId="2" ParentId="11454" CreationDate="2017-03-03T19:22:43.437" Score="1" Body="&lt;p&gt;Hi you need to implement &lt;code&gt;get_params&lt;/code&gt; method to export all your hyper-parameters or inherit your classifier from &lt;code&gt;sklearn.base.BaseEstimator&lt;/code&gt; since under hood of &lt;code&gt;OneVsRestClassifier&lt;/code&gt;, every classifier is being distributed. so your classifier is being cloned. &lt;/p&gt;&#xA;" OwnerUserId="27562" LastActivityDate="2017-03-03T19:22:43.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17347" PostTypeId="2" ParentId="9699" CreationDate="2017-03-03T21:06:01.783" Score="1" Body="&lt;p&gt;If you want to compute (approximate) nearest neighbors in 10-dimensional data, I would recommend using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kd-tree&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-d tree&lt;/a&gt;.  A k-d tree can handle skewed/non-uniform data sets and adjusts naturally to that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, you could look at data structures for &lt;a href=&quot;https://en.wikipedia.org/wiki/Nearest_neighbor_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;nearest neighbor search&lt;/a&gt;, including &lt;a href=&quot;https://en.wikipedia.org/wiki/Binary_space_partitioning&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary space partitioning trees&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_tree&quot; rel=&quot;nofollow noreferrer&quot;&gt;metric trees&lt;/a&gt;.  But k-d trees are a good starting point, and they support both exact nearest neighbor queries as well as approximate nearest neighbor queries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect this to work better than locality-sensitive hashing (LSH).&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-03T21:06:01.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17348" PostTypeId="2" ParentId="13120" CreationDate="2017-03-03T21:28:24.850" Score="7" Body="&lt;p&gt;Let me suggest three simple options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them.  (as suggested by others).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;take the (component-wise) maximum of the vectors.  (max, instead of average)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;take the (component-wise) minimum of the vectors.  (min, instead of average)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Each of these yields a feature vector that is independent of the length of the text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is some research suggesting that concatenating the max and the min yields a pretty effective feature space: it's not the absolute optimal, but it's close to optimal, and is simple and easy to implement.  See &lt;a href=&quot;https://stats.stackexchange.com/q/221715/2921&quot;&gt;this question on Statistics.SE&lt;/a&gt; for details.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is an alternative idea, inspired by &lt;a href=&quot;https://datascience.stackexchange.com/a/13144/8560&quot;&gt;cubone's answer&lt;/a&gt;, that as far as I know hasn't been tested before.  The idea is to tag the text using a part-of-speech tagger and then use those tags to inform the featurization process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, write down a list of all possible POS tags that could be emitted by the POS tagger.  Suppose there are 20 possible tags (CC, DT, JJS, MD, NNP, ...).  Then the feature vector will be 20*300 = 6000 elements long: it will have one 300-vector per POS tag, concatenated in some canonical order.  The 300-vector for each tag could be computed by averaging the word embedding vectors of all words that are tagged by the POS-tagger with that tag.  Or, you could get one 600-vector per POS tag, obtained by computing the min and max over all vectors of words with that tag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might yield a richer feature space, I don't know if it would yield any improvement, but it's something you could try if you wanted to experiment with different ideas.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-03-03T21:36:19.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17349" PostTypeId="2" ParentId="17328" CreationDate="2017-03-03T23:27:57.757" Score="0" Body="&lt;p&gt;The proportion of your data split is arbitrary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Things to remember&lt;/strong&gt; however: the &lt;strong&gt;more training data&lt;/strong&gt; you have, the &lt;strong&gt;better your model&lt;/strong&gt; will be. The &lt;strong&gt;more testing data&lt;/strong&gt; you have, the &lt;strong&gt;less variance you can expect in your results&lt;/strong&gt; (ie. accuracy, false positive rate, etc.). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally if you had infinite training data, you would be able to maximize the model's performance. The model would be the best it would ever be. And if you had infinite testing data you will be 100% confident in the resulting accuracy of your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you do not have sufficient testing data, for example you only have 1 example. Then it is possible that for the one example your algorithm correctly identifies its class and thus you assume that you have a 100% accuracy. But that would be wrong. You need multiple testing examples in order to correctly assess the performance of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 80:20 split is popular. As well as the 90:10. It's arbitrary. It all depends on how much data you have at hand. It also depends on how much data you expect to be sufficient to accurately train your model. If you only have 100 examples and you are training a data intensive model such as an NN then a 90:10 split is probably better. Although you will have high variance in your accuracy but your model will generalize better due to it having more data to train with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I usually stick to 80:20 unless the dataset suggests otherwise. Also, it is good practice to use a validation set. If you have sufficient data, it is better to do 60:20:20. Train on 60% of the data, validate your model and tweek it on 20% of the data and when you are ready to submit your model test it on the final 20% of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cross-validation is also a good technique&lt;/strong&gt;. You split your data into n bins. You then do leave-one-out training. You train on all the data bins except for 1, use the remaining bin to test. Repeat this procedure n times and take the average of your accuracies. Be wary however that you are spoiling some of your data by doing this since you are making use of all the data. It's always better to keep a chunk of the data for the final product, untouched until the last minute. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-03-08T16:55:38.703" LastActivityDate="2017-03-08T16:55:38.703" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17350" PostTypeId="1" AcceptedAnswerId="17795" CreationDate="2017-03-03T23:31:52.123" Score="2" ViewCount="445" Body="&lt;p&gt;I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does such a dataset exist? Or is there an easy way to hack one together?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2017-03-23T06:36:24.510" Title="Dataset: language audio clips and country labels" Tags="|machine-learning|dataset|bigdata|data|audio-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17351" PostTypeId="2" ParentId="17333" CreationDate="2017-03-03T23:40:32.300" Score="0" Body="&lt;p&gt;This type of dataset is commonly called a skewed dataset. This is when a certain class is over-represented in contrast to the other labels in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many algorithms that are well suited to these types of problems. Specifically, anomaly detection algorithms are capable of learning the distribution a single set of labels (event not occurring) and then it will be able to flag when an anomaly occurs (event occurs). This is when an instance is sufficiently beyond the learned distribution. You can use this to get the probability of an event occurring based on a p-statistic test using the feature-space you have set up in contrast with those from your learned distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest method would be doing a generalized likelihood ratio test (GLRT). But, I think you will most likely find more luck using a K-NN based method for skewed datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have a lot of data you can use ranking algorithms to generate your p-statistics. Look into rankSVM. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-03T23:40:32.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17352" PostTypeId="1" CreationDate="2017-03-04T00:59:41.293" Score="9" ViewCount="17224" Body="&lt;p&gt;In graph-based clustering, why is it preferred to use the Gaussian kernel rather than the distance between two points as the similarity metric?&lt;/p&gt;&#xA;" OwnerUserId="29606" LastEditorDisplayName="user10640" LastEditDate="2019-04-18T16:26:56.837" LastActivityDate="2019-04-18T17:46:00.207" Title="Why do we use a Gaussian kernel as a similarity metric?" Tags="|machine-learning|data-mining|clustering|metric|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17353" PostTypeId="2" ParentId="17352" CreationDate="2017-03-04T03:21:57.600" Score="10" Body="&lt;p&gt;Let's be precise. &quot;Distance&quot; has lots of meanings in data science, I think you're talking about &lt;strong&gt;Euclidean distance&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Gaussian kernel is a &lt;strong&gt;non-linear function&lt;/strong&gt; of Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bfQ3r.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bfQ3r.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The kernel function &lt;strong&gt;decreases&lt;/strong&gt; with distance and ranges between zero and one. In euclidean distance, the value &lt;strong&gt;increases&lt;/strong&gt; with distance. Thus, the kernel function is a more useful metrics for &lt;strong&gt;weighting&lt;/strong&gt; observations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The fact that it's bounded between zero and one is a nice property, whereas the absolute distance (it can be anything) in Euclidean distance can cause instability and difficulty in modelling.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Euclidean distance (without the negative sign) is not a similarity measure, it's a distance function. The gaussian kernel is a similarity measure.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can think the Gaussian kernel like a normalization function for Euclidean distance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-03-04T03:21:57.600" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17355" PostTypeId="2" ParentId="17288" CreationDate="2017-03-04T08:37:48.447" Score="4" Body="&lt;p&gt;I agree with the comment you received from Cross Validated – &lt;em&gt;data leakage&lt;/em&gt; is something that fits this problem setting as it's known to cause too optimistic CV score when compared to test score. We could confirm that it's actually a data leakage problem if you provided information about the data pre-processing steps that you've taken.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data leakage means that information is leaking from the test data to the training data, causing the trained model to over-estimate expected performance and to generalize poorly to unseen data, because the leaked information allows the model to learn something that it otherwise wouldn't learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, let's say that you had only a small amount of training examples and you end up synthesizing more training examples that are somehow similar to the other training examples. If you implement the synthesis before CV loop and not inside it (within each fold), your CV accuracy will skyrocket but test accuracy will be low. In addition to data synthesis, data leakage can also happen with other pre-processing steps, such as scaling the data, inputting missing values or performing feature selection.&lt;/p&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2017-03-04T08:37:48.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17356" PostTypeId="2" ParentId="17333" CreationDate="2017-03-04T09:13:29.367" Score="5" Body="&lt;p&gt;The output of logistic regression is &lt;em&gt;exactly&lt;/em&gt; that - the probability of an event happening. If your covariates are informative then your model will do better than just saying &quot;P=1000/900000&quot; everytime, because it might say &quot;P=10000/900000&quot; for a positive event, or even &quot;P=0.9&quot; of a positive event given certain covariates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there's no predictive power in the covariates (ie they don't correlate with the positive events) then yes, the model will say P=1000/900000 but that's the way it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a binary outcome then you have to decide what threshold of P you choose, based on your fear of false positives/negatives or value of true positives/true negatives. For example, if one false positive means people die, you set your threshold such that you don't get false positives. These thresholds are generally not statistical decisions but based on application logic. ROC curves can help here.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-03-04T09:13:29.367" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17358" PostTypeId="1" CreationDate="2017-03-04T14:50:46.737" Score="1" ViewCount="267" Body="&lt;p&gt;I wanted to ensure that I understood the random forest algorithm right. I think it's fundamental to understand how something work before using it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right with this:&lt;a href=&quot;https://i.stack.imgur.com/3VwbV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3VwbV.jpg&quot; alt=&quot;Random Forest Algorithm&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not: what did I understand wrong?&lt;/p&gt;&#xA;" OwnerUserId="29616" LastActivityDate="2017-03-05T13:46:37.490" Title="Primer on Random Forest Algorithm" Tags="|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17360" PostTypeId="1" CreationDate="2017-03-04T15:23:08.763" Score="1" ViewCount="45" Body="&lt;p&gt;Let say for one job application we received 10,000 applications. But after a long exercise we found only 10 resumes are relevant fit. Which ML model would be the right choice to determine the most relevant candidates?&lt;/p&gt;&#xA;" OwnerUserId="29618" LastEditorUserId="29618" LastEditDate="2017-03-04T15:43:44.727" LastActivityDate="2017-03-04T15:43:44.727" Title="Which learning model should I use to find out the most relevant applicants?" Tags="|machine-learning|data-mining|" AnswerCount="0" CommentCount="5" ClosedDate="2017-03-05T15:14:55.607" ContentLicense="CC BY-SA 3.0" />
  <row Id="17361" PostTypeId="2" ParentId="16584" CreationDate="2017-03-04T16:43:13.937" Score="1" Body="&lt;p&gt;The prediction scores are only used for ranking. The scores themselves do not provide more insight than that.&lt;/p&gt;&#xA;" OwnerUserId="8262" LastActivityDate="2017-03-04T16:43:13.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17362" PostTypeId="1" CreationDate="2017-03-04T22:22:14.247" Score="3" ViewCount="4390" Body="&lt;p&gt;I would like to create some annotations on some texts using &lt;a href=&quot;http://brat.nlplab.org/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;BRAT (brat rapid annotation tool)&lt;/a&gt;. Can BRAT be installed on Microsoft Windows? If so, how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://web.archive.org/web/20170304222627/http://brat.nlplab.org/installation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;installation instructions&lt;/a&gt; do not rule out Microsoft Windows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The brat server is a Python (version 2.5+) program that runs by default as a CGI application, and the installation script assumes a UNIX-like environment. If you are setting up brat server in a compatible environment with an existing web server that supports CGI, the quick start instructions for using CGI should work. If you don't have a web server installed and want to try brat out locally, you may want try out the quick start instructions for the standalone server. (For security reasons, we strongly recommend serving brat via a full web server such as Apache in production environments.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However they don't seem to give any instructions for Microsoft Windows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't want to use some in emulation of Unix such as virtual machines or &lt;a href=&quot;https://www.cygwin.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cygwin&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If interested, to use BRAT with &lt;a href=&quot;https://www.cygwin.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cygwin&lt;/a&gt;: &lt;a href=&quot;https://github.com/Franck-Dernoncourt/NeuroNER/blob/master/install_windows.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Franck-Dernoncourt/NeuroNER/blob/master/install_windows.md&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;After installing Cygwin, run &lt;code&gt;Cygwin.bat&lt;/code&gt; (by default located on&#xA;  &lt;code&gt;C:\cygwin64\&lt;/code&gt; if you used the 64-bit installation for Cygwin). Then,&#xA;  in the terminal that just opened, run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mkdir brat&#xA;cd brat&#xA;wget http://weaver.nlplab.org/~brat/releases/brat-v1.3_Crunchy_Frog.tar.gz&#xA;tar xzf brat-v1.3_Crunchy_Frog.tar.gz&#xA;cd brat-v1.3_Crunchy_Frog&#xA;./install.sh -u&#xA;python standalone.py&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;BRAT should now be accessible through the web browser at&#xA;  &lt;a href=&quot;http://127.0.0.1:8001&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://127.0.0.1:8001&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastEditorUserId="1330" LastEditDate="2020-06-12T19:42:03.263" LastActivityDate="2020-06-12T19:42:03.263" Title="Can BRAT be installed on Microsoft Windows?" Tags="|nlp|windows|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17363" PostTypeId="2" ParentId="17362" CreationDate="2017-03-05T03:04:01.187" Score="3" Body="&lt;p&gt;The &lt;a href=&quot;http://brat.nlplab.org/installation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;installation documentation&lt;/a&gt; shows that the easiest way to run a brat server on Windows is in a virtual machine running a UNIX-like operating system such as Ubuntu. Also, there's an &lt;a href=&quot;https://github.com/nlplab/brat/issues/833&quot; rel=&quot;nofollow noreferrer&quot;&gt;issue&lt;/a&gt; (still opening) about running brat server on windows. I think you're on your own right now.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-05T03:04:01.187" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17364" PostTypeId="1" AcceptedAnswerId="17365" CreationDate="2017-03-05T06:50:20.507" Score="12" ViewCount="11749" Body="&lt;p&gt;From the &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/model.html&quot; rel=&quot;noreferrer&quot;&gt;tutorial&lt;/a&gt; of the XGBoost, I think when each tree grows, all the variables are scanned to be selected to split nodes, and the one with the maximum gain split will be chosen. So my question is that what if I add some noise variables into the data set, would these noise variables influence the selection of variables (for each tree growing)? My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the answer is yes, then is it true that &quot;the more variables the better for XGBoost&quot;? Let's not consider the training time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if the answer is yes, then is it true that &quot;we do not need to filter out non-important variables from the model&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2017-03-05T09:57:28.073" Title="Gradient Boosting Tree: &quot;the more variable the better&quot;?" Tags="|xgboost|self-study|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17365" PostTypeId="2" ParentId="17364" CreationDate="2017-03-05T09:19:27.757" Score="12" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is only perfectly correct for very large, near infinite data sets, where the number of samples in your training set gives good coverage of all variations. In practice, with enough dimensions you end up with a lot of sampling noise, because your coverage of possible examples is weaker the more dimensions your data has.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Noise on weak variables that ends up correlating &lt;em&gt;by chance&lt;/em&gt; with the target variable can limit the effectiveness of boosting algorithms, and this can more easily happen on deeper splits in the decision tree, where the data being assessed has already been grouped into a small subset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more variables you add, the more likely it is that you will get weakly correlated variables that just happen to look good to the split selection algorithm for some specific combination, which then creates trees that learn this noise instead of the intended signal, and ultimately generalise badly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practise, I have found XGBoost quite robust to noise on a small scale. However, I have also found that it will sometimes select poor quality engineered variables, in preference to better-correlated data, for similar reasons. So it is not an algorithm where &quot;the more variables the better for XGBoost&quot; and you do need to care about possible low-quality features.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-05T09:57:28.073" LastActivityDate="2017-03-05T09:57:28.073" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17366" PostTypeId="2" ParentId="17178" CreationDate="2017-03-05T10:05:57.120" Score="-1" Body="&lt;p&gt;In case anyone has the same issue, please follow the github thread at:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/CodeReclaimers/neat-python/issues/79&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/CodeReclaimers/neat-python/issues/79&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19214" LastActivityDate="2017-03-05T10:05:57.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17367" PostTypeId="1" CreationDate="2017-03-05T10:35:45.193" Score="4" ViewCount="1391" Body="&lt;p&gt;I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn implementation, or maybe another library? &lt;/p&gt;&#xA;" OwnerUserId="29635" LastEditorUserId="29575" LastEditDate="2019-04-04T03:49:08.740" LastActivityDate="2023-10-21T06:03:00.373" Title="Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis" Tags="|scikit-learn|dimensionality-reduction|lda|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17368" PostTypeId="1" CreationDate="2017-03-05T12:17:50.687" Score="3" ViewCount="4319" Body="&lt;p&gt;I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created around 5 features (All continuous data, with 1 column as transaction id)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to return list of all transaction id,which looks suspicious&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What have I tried&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried using K means algorithm, but it does does not fit my laptop's memory (8 GB) and it crashes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to try 1 class SVM, but I do not see any good tutorial to get me started. I tried scikitlearn official tutorial, but it already has outliers added to it and they are just plotting it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I automatically detect the outliers and return those observations ?&lt;/p&gt;&#xA;" OwnerUserId="21707" LastActivityDate="2018-09-22T19:06:59.603" Title="Outlier detection by unsupervised algorithm: Fraud Detection" Tags="|python|clustering|scikit-learn|svm|outlier|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17369" PostTypeId="2" ParentId="17358" CreationDate="2017-03-05T13:46:37.490" Score="-1" Body="&lt;p&gt;Ok. I think I got it now. It should be like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UztoC.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UztoC.jpg&quot; alt=&quot;RandomForest2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right? If not: whats wrong?&lt;/p&gt;&#xA;" OwnerUserId="29616" LastActivityDate="2017-03-05T13:46:37.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17370" PostTypeId="1" AcceptedAnswerId="17478" CreationDate="2017-03-05T14:18:29.963" Score="7" ViewCount="2527" Body="&lt;p&gt;I have a data frame looking like the following:&#xA;p1-p5 are different products&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A-C: price bands&#xA;1-3: size bands&#xA;e.g. A1: small store selling the expensive that product&#xA;     D3: big store selling cheap that product&#xA;values are the numbers of stores for each category A1-C2 per product&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;type    p1  p2  p3  p4  p5&#xA;  A1    158 48  18  6   62&#xA;  A2    343 151 58  49  55&#xA;  B1    400 181 200 29  863&#xA;  B2    398 093 448 215 795&#xA;  C1    809 217 154 15  422&#xA;  C2    877 549 340 46  576&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would it be the best way to visualize that information and which packages are available for that. I have found gridplot but unfortunately I havent managed to try it yet as I have issues with the Rstudio version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples are more than welcome!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got that so far from a function found online&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Mw4ov.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Mw4ov.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14981" LastEditorUserId="14981" LastEditDate="2017-03-05T15:34:13.967" LastActivityDate="2017-10-31T15:51:44.347" Title="options available for visualizing a matrix type data frame in R" Tags="|r|visualization|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17371" PostTypeId="1" CreationDate="2017-03-05T14:55:26.210" Score="2" ViewCount="206" Body="&lt;p&gt;I am not sure i understand the concept of CNN and why it is so good at image processing. I always be told that CNN is good at finding unique features, thereby good at classifying stuff.. But does that make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess i understand how the conv. layer extract features and pooling layer maximized an window in from the convolution layer, but how are the these features used for classification in the fully connected layer. As far I know is the fully connected layer similar to a neural network, so what basically it does is classification using the features extracted from the prior steps, in which a training would make sense here, but then again, one wouldn't extract unique features but extract already known features, and train a neural network based on that?... If that is the case?... What is the benefit of using CNN?  If it is basically a NN with a preprocessing/feature extraction step? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How come is CNN so good image processing, if it require one to know what features are being extracted, and train a neural based on that... It basically boils downs to a regression problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To state the question in a way so that it makes sense with the title - How come is CNN a thing, when the thing it does combine feature extraction (known) + Neural network (training)?&lt;/p&gt;&#xA;" OwnerUserId="29639" LastEditorUserId="29169" LastEditDate="2021-02-16T02:47:20.967" LastActivityDate="2021-02-16T02:47:20.967" Title="Why is CNN a thing?" Tags="|convolutional-neural-network|" AnswerCount="0" CommentCount="4" ClosedDate="2017-03-05T15:15:08.513" ContentLicense="CC BY-SA 3.0" />
  <row Id="17372" PostTypeId="1" CreationDate="2017-03-05T16:35:44.127" Score="1" ViewCount="254" Body="&lt;p&gt;&lt;a href=&quot;http://stats.idre.ucla.edu/stata/webbooks/logistic/chapter2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://stats.idre.ucla.edu/stata/webbooks/logistic/chapter2/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This above website gives an example about using interaction effect between variables. Interacting variables are multiplied and used in the model.However I have read on Kaggle on using 3 or 4-way interaction effect. I use Xgbfi to find interactions but how do I use it in model ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read on kaggle about hashing , bayesian coding and other few techniques. Kindly elaborate on using such 3 or 4-way interaction among &lt;strong&gt;Categorical&lt;/strong&gt; , &lt;strong&gt;Numerical variables&lt;/strong&gt; and between &lt;strong&gt;Cat - Num Vars&lt;/strong&gt; in my model. Example codes using Python or R is also highly welcomed. Thank You.&lt;/p&gt;&#xA;" OwnerUserId="29644" LastEditorUserId="29644" LastEditDate="2017-03-05T17:46:42.800" LastActivityDate="2017-03-05T17:46:42.800" Title="HIgher Order Interaction Variables. How to use them in model?" Tags="|statistics|feature-selection|feature-extraction|feature-engineering|kaggle|" AnswerCount="0" CommentCount="0" ClosedDate="2017-03-05T16:55:05.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="17373" PostTypeId="2" ParentId="17368" CreationDate="2017-03-05T16:46:15.913" Score="8" Body="&lt;p&gt;First of all, start with a &lt;em&gt;subset&lt;/em&gt; until you know what you are doing.&#xA;There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my experience, one-class SVM does not work well. It assumes &lt;em&gt;all&lt;/em&gt; your training data is 'normal' class (no outliers), and this a representative sample of all normal data. Secondly,.you need to tune kernel parameters, but how.can you do so without labeled anomalies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I'd try knn outlier detection, LOF and LoOP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for these you need to make sure your distance is a &lt;em&gt;very good&lt;/em&gt; measure of similarity. If you don't preprocess well, distance does not work, and then nearest-neighbor methods don't work either.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2018-09-22T19:06:59.603" LastActivityDate="2018-09-22T19:06:59.603" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="17374" PostTypeId="1" CreationDate="2017-03-05T18:56:46.910" Score="4" ViewCount="110" Body="&lt;p&gt;As part of my DS work I spend some fraction of my time helping the team make growth projections, either for setting growth targets or when forecasting actual data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is obviously a range of ways to go about doing this but the one thing I don't have a good solution for at the moment is being able to fold in or at least quantify how much harder it is to grow in a market this year as opposed to historical years. It is easy to anchor growth on historical rates but in markets at which you've got a solid penetration already and are perhaps heading toward saturation it is likely a harder task to maintain a historical level of growth than it was to achieve that level the first time around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've thought about a few approaches that basically involve taking the derivative of a growth curve to look at the rate of change of growth, but it is hard to know if that is really probing the truth of how much harder it is to grow, or if its just that we're doing a worse job and growing more slowly!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have a good approach to either measuring some sort of 'growth friction' for a current year vs historical years, or perhaps some other forecasting technique that allows information about this to be folded in somehow?&lt;/p&gt;&#xA;" OwnerUserId="10462" LastActivityDate="2024-03-10T07:07:05.850" Title="Quantifying 'growth friction' when projecting target goals" Tags="|forecast|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17375" PostTypeId="1" AcceptedAnswerId="17399" CreationDate="2017-03-05T22:49:32.790" Score="1" ViewCount="442" Body="&lt;p&gt;I am having trouble downloading some add-ons for the Orange data mining software that are not available through the normal add-ons menu. To do this, I am attempting to install a particular set of add-ons from the GitHub page &lt;a href=&quot;http://git@github.com:biolab/orange3-prototypes.git&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am very novice when it comes to using Terminal and terminal commands. Could anyone help provide some insight on how to install this package?&lt;/p&gt;&#xA;&lt;p&gt;Some people have suggested that they should appear in the list of the prototypes, but they do not. Does anyone have Orange running on Mac so that they can try this to see if it works?&lt;/p&gt;&#xA;&lt;p&gt;Specifics for the exact code that I need to implement in the terminal to get this to work would also be helpful.&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2022-09-25T02:58:05.350" LastActivityDate="2022-09-25T02:58:05.350" Title="Installing Prototype Widgets in Orange running on Mac" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17376" PostTypeId="1" CreationDate="2017-03-05T23:45:40.880" Score="3" ViewCount="398" Body="&lt;p&gt;My question is regarding the paper &lt;a href=&quot;https://arxiv.org/abs/1605.06676&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning to Communicate with Deep Multi-Agent Reinforcement Learning&lt;/a&gt;. Can anyone explain what is the significance of Colour-digit MNIST game in the paper? I understand from &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/4kzmis/learning_to_communicate_with_deep_multiagent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; that the agent has to choose an action that represents&lt;/p&gt;&#xA;&lt;p&gt;if (case 1) color of agent 1 == parity of agent 2&lt;br /&gt;&#xA;or (case 2) color of agent 2 == parity of agent 1.&lt;/p&gt;&#xA;&lt;p&gt;The reward for case 1 is twice that of case 2. Hence, it optimal case, both agents should learn (case 1) to communicate their parity to other agent via the message and learn to check the received message against its own color. But if this seems to mean that case 2 is just acting like a noise or additional challenge in attaining optimal solution. Even if the case 2 terms did not exist, still the agents had a learn non-trivial communication protocol. Having said that, does case 2 term has any fundamental implications/need?&lt;/p&gt;&#xA;" OwnerUserId="29656" LastEditorUserId="98307" LastEditDate="2020-08-11T07:00:05.300" LastActivityDate="2020-08-11T07:00:05.300" Title="What is significance of Colour-digit MNIST game in paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning?" Tags="|neural-network|deep-learning|reinforcement-learning|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17379" PostTypeId="1" CreationDate="2017-03-06T01:53:24.720" Score="3" ViewCount="93" Body="&lt;p&gt;After a half year of working as a data scientist, I found myself tend to clone my previous work (in machine learning, like predictive modeling, EDA, data preprocessing etc.) in the new project and the work becomes a repeated style. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need new directions to lift my work with some innovative pieces. That is a key incentive to my work and also make me feel fresh and confident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone put forth advice for me? Cheers&lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26163" LastEditDate="2017-12-02T10:54:12.877" LastActivityDate="2017-12-02T10:54:12.877" Title="How can I promote my work?" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="0" CommentCount="2" ClosedDate="2017-03-06T22:54:05.247" ContentLicense="CC BY-SA 3.0" />
  <row Id="17380" PostTypeId="1" CreationDate="2017-03-06T05:10:10.167" Score="0" ViewCount="76" Body="&lt;p&gt;I've created a Decision Tree Model for expected candidates that buyed Villa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though the Drilling down Green highlighted box, show data has Age in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;- But why my model is not bifurcating it further into Age?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Deposit &amp;amp; Age&lt;/strong&gt; both are having &lt;strong&gt;content type&lt;/strong&gt; as &lt;strong&gt;Discretized&lt;/strong&gt; as &lt;strong&gt;DicretizationBucketCount&lt;/strong&gt; as &lt;strong&gt;5 &amp;amp; 10 respectively&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://1drv.ms/u/s!AtFu80OEpmZKgRRXvMdn4L79AXHU&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source Data Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://1drv.ms/u/s!AtFu80OEpmZKgRRXvMdn4L79AXHU&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QU4DU.png&quot; alt=&quot;Model Data Viewer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29664" LastEditorUserId="29664" LastEditDate="2017-03-07T10:57:20.037" LastActivityDate="2017-03-07T10:57:20.037" Title="Why my Decision Tree Model is not further bifurcated in to Age" Tags="|predictive-modeling|decision-trees|beginner|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17382" PostTypeId="1" AcceptedAnswerId="23060" CreationDate="2017-03-06T05:53:12.197" Score="3" ViewCount="2850" Body="&lt;p&gt;&lt;a href=&quot;http://brat.nlplab.org/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;BRAT (brat rapid annotation tool)&lt;/a&gt; can be used for named-entity annotation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/crLFh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/crLFh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can BRAT be used for text classification annotation? I.e., given the text, annotate whether it belongs to some classes?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2018-01-05T15:11:57.793" Title="Can BRAT be used for text classification annotation?" Tags="|nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17387" PostTypeId="2" ParentId="17317" CreationDate="2017-03-06T13:42:23.717" Score="2" Body="&lt;p&gt;I posted this same question on Reddit and someone was kind enough to answer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By &lt;a href=&quot;https://www.reddit.com/user/patrickSwayzeNU&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.reddit.com/user/patrickSwayzeNU&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This&lt;/p&gt;&#xA;  &#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process?&lt;/p&gt;&#xA;    &#xA;    &lt;p&gt;Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of.&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;  &#xA;  &lt;p&gt;Yes, your data set created by CV A is now &quot;good as new&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="24412" LastActivityDate="2017-03-06T13:42:23.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17388" PostTypeId="1" AcceptedAnswerId="37216" CreationDate="2017-03-06T14:38:06.250" Score="3" ViewCount="851" Body="&lt;p&gt;&lt;strong&gt;NOTE:&lt;/strong&gt; This question was first posted in Cross Validated Stack Exchange but I was instructed to move it off that website as it was not a good fit.&lt;/p&gt;&#xA;&lt;p&gt;I am new to the implementation of machine learning, neural networks in python. I am trying to implement &lt;code&gt;patternet&lt;/code&gt; in python as it is in MATLAB; more specifically to keep the network configuration settings. Below is what I get when I use paternet(10):&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    Neural Network&#xA; &#xA;              name: 'Pattern Recognition Neural Network'&#xA;          userdata: (your custom info)&#xA; &#xA;    dimensions:&#xA; &#xA;         numInputs: 1&#xA;         numLayers: 2&#xA;        numOutputs: 1&#xA;    numInputDelays: 0&#xA;    numLayerDelays: 0&#xA; numFeedbackDelays: 0&#xA; numWeightElements: 230&#xA;        sampleTime: 1&#xA; &#xA;    connections:&#xA; &#xA;       biasConnect: [1; 1]&#xA;      inputConnect: [1; 0]&#xA;      layerConnect: [0 0; 1 0]&#xA;     outputConnect: [0 1]&#xA; &#xA;    subobjects:&#xA; &#xA;             input: Equivalent to inputs{1}&#xA;            output: Equivalent to outputs{2}&#xA; &#xA;            inputs: {1x1 cell array of 1 input}&#xA;            layers: {2x1 cell array of 2 layers}&#xA;           outputs: {1x2 cell array of 1 output}&#xA;            biases: {2x1 cell array of 2 biases}&#xA;      inputWeights: {2x1 cell array of 1 weight}&#xA;      layerWeights: {2x2 cell array of 1 weight}&#xA; &#xA;    functions:&#xA; &#xA;          adaptFcn: 'adaptwb'&#xA;        adaptParam: (none)&#xA;          derivFcn: 'defaultderiv'&#xA;         divideFcn: 'divideind'&#xA;       divideParam: .trainInd, .valInd, .testInd&#xA;        divideMode: 'sample'&#xA;           initFcn: 'initlay'&#xA;        performFcn: 'crossentropy'&#xA;      performParam: .regularization, .normalization&#xA;          plotFcns: {'plotperform', plottrainstate, ploterrhist,&#xA;                    plotconfusion, plotroc}&#xA;        plotParams: {1x5 cell array of 5 params}&#xA;          trainFcn: 'trainscg'&#xA;        trainParam: .showWindow, .showCommandLine, .show, .epochs,&#xA;                    .time, .goal, .min_grad, .max_fail, .sigma,&#xA;                    .lambda&#xA; &#xA;    weight and bias values:&#xA; &#xA;                IW: {2x1 cell} containing 1 input weight matrix&#xA;                LW: {2x2 cell} containing 1 layer weight matrix&#xA;                 b: {2x1 cell} containing 2 bias vectors&#xA; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;My intention is not to create the exact net data structure, I would like to mainly implement &lt;strong&gt;the activation functions&lt;/strong&gt; to different layers of my network as is listed above. Let's assume I can initialize the weights (network parameters).&lt;/p&gt;&#xA;&lt;p&gt;I followed this post &lt;a href=&quot;http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Getting Started with Deep Learning and Python&lt;/a&gt; as well as &lt;a href=&quot;http://scikit-learn.org/stable/modules/neural_networks_supervised.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-layer Perceptron&lt;/a&gt; to start. In the latter link, I learned that I can implement MLP by utilizing &lt;code&gt;MLPclassifier&lt;/code&gt; from &lt;code&gt;sklearn.neural_network&lt;/code&gt; library:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;MLPClassifier(activation='relu', alpha=1e-05, batch_size='auto',&#xA;       beta_1=0.9, beta_2=0.999, early_stopping=False,&#xA;       epsilon=1e-08, hidden_layer_sizes=(5, 2), learning_rate='constant',&#xA;       learning_rate_init=0.001, max_iter=200, momentum=0.9,&#xA;       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,&#xA;       solver='lbfgs', tol=0.0001, validation_fraction=0.1, verbose=False,&#xA;       warm_start=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;However, its setting is not exactly the one that I am trying to have. Reading MLPclassifier document page did not help me further.&lt;/p&gt;&#xA;&lt;p&gt;I would appreciate it if I get to know how to implement it as such? I would be happy to be directed to any post or link which has already discussed this matter.&lt;/p&gt;&#xA;" OwnerUserId="29000" LastEditorUserId="85045" LastEditDate="2021-09-19T05:15:52.970" LastActivityDate="2021-09-19T05:15:52.970" Title="Implementing PatterNet in Python as it is in MATLAB" Tags="|machine-learning|python|neural-network|scikit-learn|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17389" PostTypeId="2" ParentId="14222" CreationDate="2017-03-06T16:14:10.657" Score="0" Body="&lt;p&gt;I like to do things simply first, and then add more complexity if it is needed. I'd start with simply removing the TV features that we don't care about and then returning what's left assuming that the brand always precedes the model. In Python for example:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_brand_model(input):&#xA;  &quot;&quot;&quot;&#xA;  Returns the brand and model number from a TV description&#xA;&#xA;  &amp;gt;&amp;gt;&amp;gt; get_brand_model('50&quot; Sony KDL 50W756CSAEP Smart LED Full HD')&#xA;  ('SONY', 'KDL 50W756CSAEP')&#xA;&#xA;  &amp;gt;&amp;gt;&amp;gt; get_brand_model('55&quot; Samsung UE55JU6400 Smart LED HD')&#xA;  ('SAMSUNG', 'UE55JU6400')&#xA;&#xA;  &amp;gt;&amp;gt;&amp;gt; get_brand_model('LG 55LF652V 55&quot; SMART 3D FULL HD')&#xA;  ('LG', '55LF652V')&#xA;&#xA;  &amp;gt;&amp;gt;&amp;gt; get_brand_model(&quot;HITACHI 55HGW69 55'' LED ULTRA SMART WIFI&quot;)&#xA;  ('HITACHI', '55HGW69')&#xA;&#xA;  &amp;gt;&amp;gt;&amp;gt; get_brand_model('TV 65&quot; SAMSUNG UE65KS7500 4K LED Smart')&#xA;  ('SAMSUNG', 'UE65KS7500')&#xA;  &quot;&quot;&quot;&#xA;&#xA;  def filter(word):&#xA;    # Basic filter to remove TV features from the input string&#xA;    skip_words = ['3d', '720p', '1080p', 'hd', '4k', 'smart', 'wifi',&#xA;                  'led', 'full', 'tv', 'ultra', 'inch']&#xA;&#xA;    is_measurement = '&quot;' in word or &quot;'&quot; in word&#xA;&#xA;    return not word.lower() in skip_words and not is_measurement &#xA;&#xA;  words = [w.upper() for w in input.split(' ') if filter(w)]&#xA;&#xA;  # Return a tuple of (brand, model number)&#xA;  return (words[0], ' '.join(words[1:]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will likely need some tweaking, but the 5 examples from the question all pass when running the included doctests.&lt;/p&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-03-06T16:14:10.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17390" PostTypeId="2" ParentId="17308" CreationDate="2017-03-06T17:03:58.787" Score="2" Body="&lt;p&gt;I can't offer any suggestions as to why your virtual environment can't find the packages (I'm not familiar with anaconda). The only thing that comes to mind is a PATH problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps try installing the package with the following command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install -n yourenvname [package]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or, in your case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install -n tensorflow h5py&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To install additional packages only to your virtual environment, enter&#xA;  the following command where yourenvname is the name of your&#xA;  environemnt, and [package] is the name of the package you wish to&#xA;  install. Failure to specify “-n yourenvname” will install the package&#xA;  to the root Python installation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;source: &lt;a href=&quot;https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29685" LastActivityDate="2017-03-06T17:03:58.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17391" PostTypeId="1" AcceptedAnswerId="17417" CreationDate="2017-03-06T17:24:18.957" Score="3" ViewCount="1845" Body="&lt;p&gt;I want to train speech-to-text model on tensorflow.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code at:&#xA;&lt;a href=&quot;https://github.com/bikramjitroy/speech-to-text/blob/master/TrainingUnitTest-CTCTrain.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/bikramjitroy/speech-to-text/blob/master/TrainingUnitTest-CTCTrain.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample &quot;audio.wav&quot; and &quot;label.txt&quot; file also present in repo.&#xA;&lt;a href=&quot;https://github.com/bikramjitroy/speech-to-text&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/bikramjitroy/speech-to-text&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am loading a &lt;code&gt;wav&lt;/code&gt; file. Getting 20 mfcc feature per frame. Passing input to one layer RNN (256 hidden unit) and then added a fully connected layer and then using &lt;code&gt;ctc_loss&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The accuracy is not increasing even with 500 steps with one training example&lt;/strong&gt; : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error = 70.932137   # Expecting it should go near zero&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accuracy = 0.62068963     # expecting this will go to zero&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried bi-directional rnn as well but that is diverging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong in code?&lt;/p&gt;&#xA;" OwnerUserId="24158" LastEditorUserId="24158" LastEditDate="2017-03-07T15:27:52.640" LastActivityDate="2017-03-07T15:55:42.157" Title="Tensorflow speech-to-text training on single file with tf.nn.ctc_loss not converging" Tags="|machine-learning|python|tensorflow|rnn|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17394" PostTypeId="2" ParentId="14676" CreationDate="2017-03-06T18:13:31.287" Score="0" Body="&lt;p&gt;For topic modeling I have measured the within topic cosine distance and used that to optimize the number of topics derived. For each topic measure the pairwise cosine distance --&gt; take the mean. Then for all topics, take the mean of the corresponding mean of the pairwise cosine distances between all vectors (within  a topic). This will be your metric. Measure this value while iterating over varying numbers of topics and you should notice an elbow curve, telling you precisely where to cut the number of topics.&lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-03-06T18:13:31.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17395" PostTypeId="2" ParentId="17021" CreationDate="2017-03-06T18:32:08.097" Score="3" Body="&lt;p&gt;If I got your question right, the $x_i$ has different length $l$ over $i\in[1,n]$ as your training data. A very common method is to padding each training sample and testing sample to the same length, or use a fixed length time window for sampling your time series data. As for here, you may pad all the $x_i$ to the length of $L$ with $L = max(l_i), i \in [1,2,...,n,n+1...]$. The values for padding depends on your data, use some values that are not supposed to occur in your data to represent the &quot;padding&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make the prediction on $x_{n+1}$, you also pad or use time window to sample the $x_{n+1}$ first, and then make the prediction using the model you trained.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what kind of model you are using and why your time series vary in length, I assume you have used subsampling on your original data. For deep learning models like RNN, LSTM, you can use fixed length time series data since the model extract the features over time and subsampling is unnecessary.&lt;/p&gt;&#xA;" OwnerUserId="29690" LastActivityDate="2017-03-06T18:32:08.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17396" PostTypeId="1" CreationDate="2017-03-06T19:16:35.167" Score="1" ViewCount="110" Body="&lt;p&gt;In the context of anomaly detection, which is a better language to use, python or R ?&lt;/p&gt;&#xA;" OwnerUserId="29695" LastActivityDate="2017-03-06T19:16:35.167" Title="In the context of anomaly detection, which is a better language to use, python or R?" Tags="|machine-learning|python|r|anomaly-detection|" AnswerCount="0" CommentCount="0" ClosedDate="2017-03-06T22:53:45.160" ContentLicense="CC BY-SA 3.0" />
  <row Id="17397" PostTypeId="2" ParentId="17380" CreationDate="2017-03-06T19:23:08.413" Score="2" Body="&lt;p&gt;You have your Decision Tree set to a default expansion of 3 levels. The Decison Tree works with the most significant features first, and it has not deemed age as important. Increase the default expansion to more levels to see the bifurcation of age. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A word of caution: deep decision trees are prone to overfitting.&lt;/p&gt;&#xA;" OwnerUserId="29692" LastActivityDate="2017-03-06T19:23:08.413" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17398" PostTypeId="2" ParentId="17308" CreationDate="2017-03-06T19:44:25.223" Score="0" Body="&lt;p&gt;I believe that there might be a conflicting version of python, try to see which python you're using when you're in the virtual environment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;which python&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the python version you're using is not the one in the conda environment, try setting it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;export PATH=path/to/python&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can try two more things, first try the conda pip install to see if this works:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install pip&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It might be possible that you have two conflicting versions of pip, to see if this is the case try this and set it to the right version with export:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;which -a pip&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29692" LastActivityDate="2017-03-06T19:44:25.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17399" PostTypeId="2" ParentId="17375" CreationDate="2017-03-06T19:54:06.377" Score="2" Body="&lt;p&gt;The easiest way to install Orange's addons is through the application itself. Open Orange, in the menu click &lt;code&gt;Options -&amp;gt; Addons&lt;/code&gt;. In the popup window mark &lt;code&gt;Orange3-Prototypes&lt;/code&gt; and click OK. Note, that by doing so you will get the latest version that is published on PyPI.&lt;/p&gt;&#xA;&lt;p&gt;If you would want to install the bleeding edge version directly from GitHub — assuming that you installed Orange by downloading a bundle from its website and not for example by using Anaconda — do the following. Open a terminal and paste this command:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Doing so you will trigger the &lt;code&gt;pip&lt;/code&gt; that came with the bundle and instruct it to install the addon directly from Github.&lt;/p&gt;&#xA;&lt;p&gt;Further, to install &lt;code&gt;opencv&lt;/code&gt;, which is required by some widgets (e.g. webcamecapture) run&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Note, however, that this is an unofficial version of OpenCV but it seems easier to install.&lt;/p&gt;&#xA;" OwnerUserId="15685" LastEditorUserId="85045" LastEditDate="2021-09-20T00:09:48.337" LastActivityDate="2021-09-20T00:09:48.337" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="17401" PostTypeId="1" CreationDate="2017-03-06T21:41:59.770" Score="1" ViewCount="339" Body="&lt;p&gt;I have sparsely populated matrix of users as rows  with columns being categorical answers to various questions ( question are of various domain about  preferences / behaviors of the users ) . answers may be be either numerical ( for example an answer to the question &quot;what's the number of children in your household ? &quot; ), or categorical   ( &quot;specify your education level  ? BS / PHd  / etc.&quot;). As mentioned the matrix is sparse , and the aim is to infer the missing entries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;do you think matrix factorization techniques ( for example  ALS ) could be suitable for solving this ? ( with proper normalization of the response ) and do you suggest another learning algorithm ?&lt;/p&gt;&#xA;" OwnerUserId="29701" LastActivityDate="2017-08-04T01:24:25.157" Title="matrix factorization?" Tags="|machine-learning|deep-learning|matrix-factorisation|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17402" PostTypeId="1" CreationDate="2017-03-06T22:14:11.877" Score="2" ViewCount="114" Body="&lt;p&gt;I have a system which sends invitations to users to participate in online questionnaires and want to use machine learning in order to predict the likelihood of fulfilling the questionnaires in a predefined time ( i.e. within 1 day , 2 day , 3 days ,a week, 2 weeks, etc) based on various feature related to the users to whom the invitations are sent to , details of the questionnaires ( i.e. how long are they , their topic ,etc.) , other contextual data ( time of day , day of year , in which media the invitation are sent - i.e. sms / email etc). &#xA;I can train with positive examples ( invitations that were responded to by users ) and negative examples ( invitations that WERE not responded to ) , however , I'm not sure how to take into consideration the &quot;predefined time&quot; into the feature vector. for example, should I simply include a feature of &quot;days since invitation was sent&quot; and in the positive examples include the time , and in negative examples replicate each example X each of the predefined times to indicate that the users didn't respond at all ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advise would be welcome  !&lt;/p&gt;&#xA;" OwnerUserId="29701" LastEditorUserId="13727" LastEditDate="2017-03-07T09:18:29.923" LastActivityDate="2017-03-07T09:18:29.923" Title="classification feature selection" Tags="|machine-learning|classification|data-leakage|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17403" PostTypeId="2" ParentId="17401" CreationDate="2017-03-06T22:24:23.500" Score="1" Body="&lt;p&gt;I think ALS is very applicable here. I would imagine it would need to be fine tuned quite a bit though. Unless the questions are correlated, it might yield some weird results. Just because two people with PHD's have one child, does not mean the third would only have one child. Basically, the more the data the better. Recommending movies works so well because all the columns are related to movies, but this should work with enough data. &lt;/p&gt;&#xA;" OwnerUserId="27632" LastActivityDate="2017-03-06T22:24:23.500" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17405" PostTypeId="1" CreationDate="2017-03-07T03:16:55.153" Score="1" ViewCount="50" Body="&lt;p&gt;Specifically, I am wondering if you can determine this when each factor explains only one of the variables well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an illustration of a factor pattern to clarify:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s0IWt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s0IWt.png&quot; alt=&quot;Factor Pattern&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to say that variable 1 and variable 3 are positively correlated? If not, what other information would be needed?&lt;/p&gt;&#xA;" OwnerUserId="28694" LastActivityDate="2017-03-07T03:16:55.153" Title="Is it possible to determine the correlation direction of two variables using factor loadings?" Tags="|correlation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17407" PostTypeId="1" AcceptedAnswerId="17856" CreationDate="2017-03-07T05:57:41.937" Score="4" ViewCount="239" Body="&lt;p&gt;i'm trying to implement my own LSTM network. I implemented back propagation algorithm, but it doesn't pass gradient check. Can't realize where is the mistake. Please help&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code problem piece of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def backward_propagation(self, x, y, cache):&#xA;    # T - the length of the sequence&#xA;    T = len(y)&#xA;    # perform forward propagation&#xA;    cache = self.forward_propagation(x)&#xA;&#xA;    #...&#xA;&#xA;&#xA;    # delta for output layer&#xA;    dy = cache['y'].copy()&#xA;    dy[np.arange(len(y)), y] -= 1. # softmax loss gradient&#xA;&#xA;    for t in reversed(range(T)):&#xA;        dV += np.outer(dy[t], h[t].T)&#xA;        dh[t] += self.V.T.dot(dy[t])&#xA;        dhtmp = self.V.T.dot(dy[t])&#xA;        dctmp = dct[0] * (1.0 - ct[0]**2)&#xA;&#xA;        for bptt_step in np.arange(t)[::-1]:&#xA;            # add to gradients at each previous step&#xA;            do[bptt_step] = dhtmp * ct[bptt_step]&#xA;            dct[bptt_step] = dhtmp * o[bptt_step]&#xA;&#xA;            dctmp += dct[bptt_step] * (1.0 - ct[bptt_step]**2)&#xA;&#xA;            di[bptt_step] = dctmp * g[bptt_step]&#xA;            df[bptt_step] = dctmp * c[bptt_step-1]&#xA;            dg[bptt_step] = dctmp * i[bptt_step]&#xA;&#xA;            # backprop activation functions&#xA;            diga[bptt_step] = di[bptt_step] * i[bptt_step] * (1.0 - i[bptt_step])&#xA;            dfga[bptt_step] = df[bptt_step] * f[bptt_step] * (1.0 - f[bptt_step])&#xA;            doga[bptt_step] = do[bptt_step] * o[bptt_step] * (1.0 - o[bptt_step])&#xA;            dgga[bptt_step] = dg[bptt_step] * (1.0 - g[bptt_step] ** 2)&#xA;&#xA;            # backprop matrix multiply&#xA;            dWi += np.dot(diga[bptt_step].T, h[bptt_step-1])&#xA;            dWf += np.dot(dfga[bptt_step].T, h[bptt_step-1])&#xA;            dWo += np.dot(doga[bptt_step].T, h[bptt_step-1])&#xA;            dWg += np.dot(dgga[bptt_step].T, h[bptt_step-1])&#xA;&#xA;            dUi[:, x[bptt_step]] += diga[bptt_step]&#xA;            dUf[:, x[bptt_step]] += dfga[bptt_step]&#xA;            dUo[:, x[bptt_step]] += doga[bptt_step]&#xA;            dUg[:, x[bptt_step]] += dgga[bptt_step]&#xA;&#xA;            # update deltas for next step&#xA;            dhtmp = np.dot(self.Wi.T, diga[bptt_step])&#xA;            dhtmp += np.dot(self.Wf.T, dfga[bptt_step])&#xA;            dhtmp += np.dot(self.Wo.T, doga[bptt_step])&#xA;            dhtmp += np.dot(self.Wg.T, dgga[bptt_step])&#xA;            dctmp += dctmp * f[bptt_step]&#xA;&#xA;    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here 'ga' suffix is for 'gate activation' - designation of gate input before non-linear activation function is applied.&lt;/p&gt;&#xA;" OwnerUserId="29714" LastEditorUserId="29714" LastEditDate="2017-03-07T12:10:31.540" LastActivityDate="2017-03-24T21:25:25.370" Title="My LSTM doesn't pass gradient check" Tags="|python|deep-learning|rnn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17409" PostTypeId="2" ParentId="17402" CreationDate="2017-03-07T09:16:04.793" Score="2" Body="&lt;p&gt;You have a problem of &lt;a href=&quot;http://machinelearningmastery.com/data-leakage-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;data leakage&lt;/a&gt;.&#xA;The &quot;days since invitation was sent&quot; feature contains all the information on the concept. Therefore, adding it as a feature will prevent most common classifiers from using rules based on other features and will lead to misleading results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Duplicating the negative row for each predefined time value will lead to a different distribution than the one you should be tested one, which need needles problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that even that way, the feature might leak information (e.g., a small ratio of the positives will replay in two weeks while its distribution in the negatives will be higher, making it a good predicator for negative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case that you will have more complex representation of time (e.g., was the email sent on weekend? was the reply sent on weekend), the duplication of rows will be more complex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would try a different direction. It is likely that the negative rows has no influence on the time to reply in the positive ones. &#xA;Therefore I would do a first research aimed to differ between replies and no replies without using the predefined time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, do a second research only on the positives where the concept will be the predefined time, aiming to find what is influencing out given that the user had replied.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-07T09:16:04.793" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="17410" PostTypeId="2" ParentId="14964" CreationDate="2017-03-07T11:02:50.433" Score="2" Body="&lt;ul&gt;&#xA;&lt;li&gt;Like others said Good visual split is a good starting point. (To me it seems the f8-f1 is a good starting point.&lt;/li&gt;&#xA;&lt;li&gt;However you could get better results by transforming the feature set via PCA and using top eigen factors(new combination features) to train.&lt;/li&gt;&#xA;&lt;li&gt;You've not clarified whether it is a supervised(you know the no. of classes and actual class for some data set), assuming you don't and it is unsupervised, I'd also try an algorithm like &lt;a href=&quot;https://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;dbscan&lt;/a&gt;. It is generally fast. &lt;/li&gt;&#xA;&lt;li&gt;If you want to try the neural network approach try either &lt;a href=&quot;https://github.com/sevamoo/SOMPY&quot; rel=&quot;nofollow noreferrer&quot;&gt;sompy&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Neural_gas&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural gas&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13382" LastActivityDate="2017-03-07T11:02:50.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17411" PostTypeId="2" ParentId="10545" CreationDate="2017-03-07T12:26:26.833" Score="1" Body="&lt;p&gt;as i understand, you have the wrong backprop gradient implementation. Here you should take into account, that rnn's hidden state h has its previous state in the equation: h[time-1]. This is also must be extracted via chain rule. For more information suggest to refer &lt;a href=&quot;http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;.&lt;br /&gt;&#xA;It also contains Python rnn implementation.&lt;/p&gt;&#xA;" OwnerUserId="29714" LastEditorUserId="98307" LastEditDate="2020-08-07T07:24:09.640" LastActivityDate="2020-08-07T07:24:09.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17412" PostTypeId="2" ParentId="10615" CreationDate="2017-03-07T13:17:06.903" Score="15" Body="&lt;p&gt;According to &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LSTM cell structure&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/r8O3S.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/r8O3S.png&quot; alt=&quot;LSTM cell structure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LSTM equations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sFvTE.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sFvTE.png&quot; alt=&quot;LSTM equations&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ingoring non-linearities&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DHi92.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DHi92.png&quot; alt=&quot;Ingoring non-linearities&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the input &lt;strong&gt;x_t&lt;/strong&gt; is of size n×1, and there are &lt;strong&gt;d&lt;/strong&gt; memory cells, then the size of each of &lt;strong&gt;W∗&lt;/strong&gt; and &lt;strong&gt;U∗&lt;/strong&gt; is &lt;strong&gt;d×n&lt;/strong&gt;, and &lt;strong&gt;d×d&lt;/strong&gt; resp. The size of &lt;strong&gt;W&lt;/strong&gt; will then be &lt;strong&gt;4d×(n+d)&lt;/strong&gt;. Note that each one of the dd memory cells has its own weights &lt;strong&gt;W∗&lt;/strong&gt; and &lt;strong&gt;U∗&lt;/strong&gt;, and that the only time memory cell values are shared with other LSTM units is during the product with &lt;strong&gt;U∗&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to Arun Mallya for great presentation.&lt;/p&gt;&#xA;" OwnerUserId="29714" LastEditorUserId="29714" LastEditDate="2017-03-08T19:37:33.507" LastActivityDate="2017-03-08T19:37:33.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17414" PostTypeId="1" CreationDate="2017-03-07T14:06:42.100" Score="2" ViewCount="1207" Body="&lt;p&gt;We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already done Twitter ingestion using Flume, but I don't think using Flume would suit my current use-case because I am not using a continuous data firehose like this one in Twitter, but rather discrete regular time-bound invocations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please I would like to hear suggestions / alternatives (if there's easier than what I'm thinking right now) about design and which Hadoop-based component(s) to use for this use-case. If you feel I can stick to Flume, then kindly give me also an idea how to do this.&lt;/p&gt;&#xA;" OwnerUserId="27647" LastEditorUserId="27647" LastEditDate="2017-03-08T07:29:26.797" LastActivityDate="2017-03-08T08:41:41.227" Title="Ingestion of periodic REST API Calls into Hadoop" Tags="|bigdata|apache-hadoop|json|" AnswerCount="1" CommentCount="1" ClosedDate="2017-03-11T05:03:46.353" ContentLicense="CC BY-SA 3.0" />
  <row Id="17415" PostTypeId="1" AcceptedAnswerId="17425" CreationDate="2017-03-07T14:57:14.243" Score="2" ViewCount="958" Body="&lt;p&gt;Has anybody seen any application that would use GAN that would take input image and would output image of the same size, that could be used as a layer for the first image. The layer would contain in eg. point of interest in the input image. Would that even be a good practice of the GAN usage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for articles, examples of application that used something similar.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastEditorUserId="2750" LastEditDate="2017-03-07T15:34:43.333" LastActivityDate="2017-03-07T18:47:42.210" Title="Using Generative Adversarial Networks for a generation of image layer" Tags="|neural-network|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17416" PostTypeId="1" CreationDate="2017-03-07T14:59:00.657" Score="0" ViewCount="301" Body="&lt;p&gt;If this is a problem of text classification, is a similar dataset available or i have to make one on my own??&#xA;I have a dataset of emails for classification into spam/ham.&lt;/p&gt;&#xA;" OwnerUserId="29729" LastActivityDate="2017-03-07T17:31:00.127" Title="i have to classify an sms into categories like educational, bank related,etc. Is this a problem of topic modelling or text classification?" Tags="|machine-learning|text-mining|topic-model|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-11T05:04:39.327" ContentLicense="CC BY-SA 3.0" />
  <row Id="17417" PostTypeId="2" ParentId="17391" CreationDate="2017-03-07T15:55:42.157" Score="1" Body="&lt;p&gt;I have found the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Earlier I have a &lt;strong&gt;Dropout layer&lt;/strong&gt; after single layer of RNN. I have changed the rnn_layer function as below. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Removed the DropoutWrapper from the RNN layer - This helped me to overfit&lt;/li&gt;&#xA;&lt;li&gt;Added 2 layers of LSTM cell. Single layer LSTM cell taking time to&#xA;converge but rather than making LSTM layer wide changed it to deep.&#xA;This helped me achieve faster convergence than making network wider.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Below is the code of rnn_layer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;def rnn_layer(input_tensor, n_cell_units, dropout, seq_length, batch_size):&#xA;    lstm_fw_cell = tf.contrib.rnn.BasicLSTMCell(2*n_cell_units, forget_bias=1.0, state_is_tuple=True)&#xA;# Dropout layer dropped&#xA;#    lstm_fw_cell = tf.contrib.rnn.DropoutWrapper(lstm_fw_cell, input_keep_prob=dropout, output_keep_prob=dropout)&#xA;# Multi layer RNN&#xA;    lstm_fw_cell = tf.contrib.rnn.MultiRNNCell([lstm_fw_cell] * 2, state_is_tuple=True)&#xA;&#xA;    outputs, output_states = tf.nn.dynamic_rnn(cell=lstm_fw_cell,&#xA;                                               inputs=input_tensor,&#xA;                                               dtype=tf.float32,&#xA;                                               time_major=True,&#xA;                                               sequence_length=seq_length)&#xA;&#xA;    return outputs&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="24158" LastActivityDate="2017-03-07T15:55:42.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17419" PostTypeId="1" CreationDate="2017-03-07T16:08:36.270" Score="3" ViewCount="847" Body="&lt;p&gt;I have a pipeline of models:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training pipeline (on training data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Tokenizer -&amp;gt; HashingTF -&amp;gt; IDF -&amp;gt; KMeans&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Test pipeline (prediction on query data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trained pipeline -&amp;gt; selecting features from training data which are in the same cluster for query data -&amp;gt; BucketedRandomProjectionLSH&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What it does explained a bit &lt;a href=&quot;https://stackoverflow.com/questions/42557854/suggestion-on-algorithm-selection-and-implementation/42586806#42586806&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was curious if that is possible to select params of &lt;em&gt;BucketedRandomProjectionLSH&lt;/em&gt;, &lt;em&gt;KMeans&lt;/em&gt; and &lt;em&gt;HashingTF&lt;/em&gt; wisely enough, so they have rationale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disclaimer: I'm new to all of that, so below are my assumptions/thoughts only.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, next params:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;&lt;a href=&quot;http://spark.apache.org/docs/2.1.0/ml-features.html#tf-idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;hashingTF&lt;/a&gt;.numFeatures&lt;/em&gt; - &quot;number of features&quot;. Underneath, it uses hashing in order not to face collision and false matches of different words during frequency calculations, so rationale beneath of that - #of words I likely will face. In my case I selected ~16k as it seems far beyond the dictionary of paper language + some performance considerations taken into account (e.g. default 2^18 led to long calculation and as the results - timeout has to be tuned due to connection timeout exception)&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;&lt;a href=&quot;http://spark.apache.org/docs/2.1.0/ml-clustering.html#k-means&quot; rel=&quot;nofollow noreferrer&quot;&gt;kmeans&lt;/a&gt;.k&lt;/em&gt; - &quot;the number of clusters to create&quot;. In unsupervised learning I have no insight on how to select this one param - just to guess. In future, it might be the #of topics my words belong to. Probably &lt;a href=&quot;http://spark.apache.org/docs/2.1.0/ml-clustering.html#latent-dirichlet-allocation-lda&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDH&lt;/a&gt; + some manual consideration may help to set it correctly.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;&lt;a href=&quot;http://spark.apache.org/docs/2.1.0/ml-features.html#bucketed-random-projection-for-euclidean-distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;bucketedRandomProjectionLSH&lt;/a&gt;.bucketLength&lt;/em&gt; - &quot;the length of each hash bucket&quot;. As I understand it - it represents the param of reduction of initial word vector (represented by frequency of words occurrences), squeezing it to &quot;length&quot;, with further distance calculating and assigning the vector (based on the &quot;distance&quot; accordingly) to the next param - &lt;em&gt;bucketedRandomProjectionLSH.numHashTables&lt;/em&gt;. So the rationale on its value I see as the amount of word I more likely want to take into account to make a prediction on distance between vectors. E.g. if I have an article which is decomposed further into word tokens - how many words I want to take into account? e.g. half of the average #.&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;bucketedRandomProjectionLSH.numHashTables&lt;/em&gt; - &quot;number of hash tables&quot; - as I see it - how many &quot;distances&quot; I want to consider. In other words - I have #of clusters (i.e. 5): it answers how further I want to split the cluster to get narrowed results when comparing what it related to my &lt;em&gt;query&lt;/em&gt;. Will definately depends on the amount of features in the cluster and #of results I expect.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So if to think about that backwards - if given &lt;strong&gt;X&lt;/strong&gt; is: #results I expect out of prediction:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[&lt;em&gt;bucketedRandomProjectionLSH.numHashTables&lt;/em&gt;] = [Average #of articles / X]&lt;/li&gt;&#xA;&lt;li&gt;[&lt;em&gt;bucketedRandomProjectionLSH.bucketLength&lt;/em&gt;] = [Average #of words per article / 2]&lt;/li&gt;&#xA;&lt;li&gt;[&lt;em&gt;kmeans.k&lt;/em&gt;] = [X * some magic number &gt;= 2]&lt;/li&gt;&#xA;&lt;li&gt;[&lt;em&gt;hashingTF.numFeatures&lt;/em&gt;] = [number of words in dictionary]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Kindly ask you to evaluate my thoughts process and correct.&lt;/p&gt;&#xA;" OwnerUserId="29534" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-03-07T20:36:10.073" Title="Model params tuning" Tags="|machine-learning|apache-spark|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17420" PostTypeId="1" CreationDate="2017-03-07T16:50:06.637" Score="1" ViewCount="268" Body="&lt;p&gt;I'm interested in masking all non-zero values in a plotly time series plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to simply not include the zeroes when rendering the plot, and connect all data points greater than zero?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried limiting the range in the layout, but this did not change what was plotted:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import plotly.graph_objs as go&#xA;import plotly.plotly as py&#xA;&#xA;data = [go.Scatter(&#xA;          x=data['timestamp'],&#xA;          y=data['units_sold'])]&#xA;&#xA;layout = go.Layout(&#xA;    yaxis=dict(&#xA;        range=[2, 15]))&#xA;&#xA;fig = go.Figure(data=data, layout=layout)&#xA;&#xA;py.iplot(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the general structure of my code as a Pandas dataframe:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd    &#xA;&#xA;d = {'timeseries' : [801, 802, 803, 804, 805],&#xA;         'units_sold' : [0, 0, 1, 3, 0]}&#xA;&#xA;pd.DataFrame(d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23701" LastEditorUserId="23701" LastEditDate="2017-03-08T05:40:45.787" LastActivityDate="2017-03-08T05:40:45.787" Title="Masking numbers in plotly time series plot" Tags="|visualization|plotting|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17421" PostTypeId="1" AcceptedAnswerId="17422" CreationDate="2017-03-07T17:02:15.513" Score="4" ViewCount="2267" Body="&lt;p&gt;The classes of &lt;a href=&quot;http://ieeexplore.ieee.org/abstract/document/5206848/&quot; rel=&quot;noreferrer&quot;&gt;ImagNet&lt;/a&gt; have a hierarchy. Did anybody try to build a hierarchy of classifiers to use this fact?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Searching for &quot;multistage classification&quot; leads to different results.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-03-07T17:07:29.870" LastActivityDate="2017-03-07T17:28:21.100" Title="Has anyone tried to use the hierarchy of ImageNet?" Tags="|image-classification|computer-vision|reference-request|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17422" PostTypeId="2" ParentId="17421" CreationDate="2017-03-07T17:28:21.100" Score="5" Body="&lt;p&gt;Yes, multiple papers have used this. I've heard of multiple ways to exploit this hierarchial structure. This paper &lt;a href=&quot;https://arxiv.org/pdf/1410.0736.pdf&quot; rel=&quot;noreferrer&quot;&gt;Hierarchical Deep Convolutional Neural Network for Large Scale&#xA;Visual Recognition&lt;/a&gt; uses multiple levels by predicting the more coarse distribution and I think it then passes this as features to the more low level classification. &lt;a href=&quot;https://arxiv.org/abs/1612.08242&quot; rel=&quot;noreferrer&quot;&gt;YOLO9000&lt;/a&gt; actually uses the ImageNet hierarchy for the problem of object detection. Hierarchial classification is the name of the problem you are describing.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-03-07T17:28:21.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17423" PostTypeId="2" ParentId="17416" CreationDate="2017-03-07T17:31:00.127" Score="1" Body="&lt;p&gt;Topic modeling is usually used in the context of unsupervised learning, while classification is used for supervised learning. So it more depends on your dataset, does it have labels that you can learn from (classification) or does it need to figure out topics itself (topic modeling)?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-03-07T17:31:00.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17424" PostTypeId="2" ParentId="17328" CreationDate="2017-03-07T17:54:40.693" Score="0" Body="&lt;p&gt;You and others are correct to question these &quot;rules&quot;.  Truth is, choosing a test/training split is a simple question with a complicated answer.  See &lt;a href=&quot;http://projecteuclid.org/download/pdfview_1/euclid.ssu/1268143839&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; which discusses different cross-validation methodologies, where&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As a conclusion, guidelines are provided for choosing the best&#xA;  cross-validation procedure according to the particular features of the&#xA;  problem in hand.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In particular, they found the signal-to-noise ratio of the data had a lot to do with how the model should be trained and tested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also find empirical papers (here's my dataset, here's my goal, here's what worked best) such as &lt;a href=&quot;https://pdfs.semanticscholar.org/0be0/d781305750b37acb35fa187febd8db67bfcc.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt; which found 10-fold cross-validation was best for &lt;em&gt;model selection with that particular data set&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I don't claim to have read everything, I haven't yet stumbled upon some who found a simple 80/20 or 70/30 split was the best way to do &lt;em&gt;anything&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2017-03-07T17:54:40.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17425" PostTypeId="2" ParentId="17415" CreationDate="2017-03-07T17:56:27.810" Score="3" Body="&lt;p&gt;In terms of generating an image &quot;layer&quot;, that is just the same as generating an output image that can be overlaid on the input using standard graphics software. If you want pixel-level accuracy in the output then the output will need to be the same size as the input, otherwise it could be smaller, provided it is the same aspect ratio, and in which case it would need to be scaled up in order to be used as an overlay. In any case, as the output of a GAN can be an image (and often is), then this part is easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;G&quot; in GAN stands for Generative. The purpose of a generative network is to create samples from a population, where there are typically many possibilities. Those samples can be conditioned on some additional data, and that additional data &lt;em&gt;could&lt;/em&gt; be an image, although many examples will simpler conditioning, such as a category that the training output is representative of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility for using a GAN, is where your population contains a range of traits, and you can calculate vectors that control that trait. So you can take an input image, reconstruct it in the GAN then modify it by adding/subtracting the trait-related vector. An interesting example of this is &lt;a href=&quot;https://arxiv.org/pdf/1702.01983.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Face Aging With Conditional Generative Adversarial Networks&lt;/a&gt;, and similar examples are around of adding/removing glasses etc. For this to work for you, you would literally need images that had your points-of interest in them and ones without them, and then you would be able to control addition/removal of points-of-interest. The network would not detect these points in the input, instead it would &lt;em&gt;add&lt;/em&gt; them into the output. From reading your question this does not seem to be what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://arxiv.org/abs/1701.05957&quot; rel=&quot;nofollow noreferrer&quot;&gt;similar paper uses a GAN to remove rain from photos, based on training many images with and without rain&lt;/a&gt; then learning the &quot;rain vector&quot;, encoding new images with rain in them into the GAN's internal representation and subtracting this &quot;rain vector&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GANs conditioned on input images (as opposed to categories or internal embeddings) are also possible - this &lt;a href=&quot;http://cs231n.stanford.edu/reports2016/209_Report.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;example of image completion&lt;/a&gt; might be closer to your goal. If your points of interest are variable with many options feasible, then it could work for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if your points of interest are supposed to always be the same pixels in each image, then your goal might be better defined by strict ground truth and become more like semantic segmentation, which can be attempted with variations on CNNs, such as &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Dai_Convolutional_Feature_Masking_2015_CVPR_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;described in this paper by Microsoft&lt;/a&gt;. These are much easier to set up and train than GANs, so if you can reasonably frame your problem as pixel classification from the original image, this is probably the way to go.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-07T18:47:42.210" LastActivityDate="2017-03-07T18:47:42.210" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17426" PostTypeId="1" CreationDate="2017-03-07T17:59:21.197" Score="1" ViewCount="397" Body="&lt;p&gt;I have a linear model in R, named Q1fit. I am interested in the plot of the residuals vs leverage, with Cook's distance. When I run&lt;/p&gt;&#xA;&#xA;&lt;p&gt;plot(Q1fit, which=(5))  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/K9ESb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K9ESb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no Cook's distance line visible here. Does that mean the Cook's distance goes beyond the axes, and I need to change the limits? If so, how can I do that without giving an error (as my attempts keep doing).&lt;/p&gt;&#xA;" OwnerUserId="29731" LastActivityDate="2017-03-07T17:59:21.197" Title="Cook's distance, altering diagnostic plot in R,?" Tags="|r|predictive-modeling|statistics|data|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17427" PostTypeId="1" CreationDate="2017-03-07T18:09:48.740" Score="1" ViewCount="96" Body="&lt;p&gt;I need to find a good method to extract key points from drawing symbols like elements in CAD drawings (eg in 2D objects for the bathroom that can be placed on a floor plan).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already tried all type available in openCV/accord (FAST, SURF, Harris, ..) but all of them failed to give me any point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The images are blank and white (with some sort of grey scale) and I'd like to find the position of a limited set of symbols inside bigger images containing other unknown objects/shapes (that I'm not interested in anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic idea is to get the descriptors for my symbols and check if the target image contains similar descriptors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have a good method for descriptors then I can train a classifier to check all the descriptors from the input images and so I can also know the position related to the recognized object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Framework can be any with preference on OPENCV, Accord.net, tensorflow&lt;/p&gt;&#xA;" OwnerUserId="29732" LastActivityDate="2017-04-13T20:56:59.827" Title="what could it be a good descriptor for simple greyscale symbols?" Tags="|classification|object-recognition|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17428" PostTypeId="1" CreationDate="2017-03-07T19:28:43.240" Score="6" ViewCount="742" Body="&lt;p&gt;Vanishing gtadient causes too small values which are come from non-linear functions like sigmoid, tanh. LSTM architecture was developed to fight these issues.&#xA;But how can it avoid a problem if there are 3 sigmoid, 2 tanh functions and, moreover, they are combinated via multiplication?&lt;/p&gt;&#xA;" OwnerUserId="29714" LastActivityDate="2018-04-29T11:08:49.497" Title="How does LSTM fights vanishing gradient?" Tags="|neural-network|deep-learning|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17429" PostTypeId="2" ParentId="8909" CreationDate="2017-03-07T20:15:55.927" Score="1" Body="&lt;p&gt;One newer library to look at as well is &lt;a href=&quot;https://github.com/denizyuret/Knet.jl&quot; rel=&quot;nofollow noreferrer&quot;&gt;Knet.jl&lt;/a&gt;. It will do things like use GPUs under the hood.&lt;/p&gt;&#xA;" OwnerUserId="25507" LastEditorUserId="98307" LastEditDate="2020-08-05T13:58:08.623" LastActivityDate="2020-08-05T13:58:08.623" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17430" PostTypeId="1" CreationDate="2017-03-07T21:24:26.520" Score="4" ViewCount="3272" Body="&lt;p&gt;I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A vector embedding (word2vec, GloVe or fasttext), averaging over word vectors in a document, and using cosine similarity.&lt;/li&gt;&#xA;&lt;li&gt;tf-idf or its variations such as BM25.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Will one of these yield a significantly better result? Has someone done a quantitative comparison of tf-idf versus averaging word2vec for document similarity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there another approach, that allows to dynamically refine the document's vectors as more text is added?&lt;/p&gt;&#xA;" OwnerUserId="29735" LastActivityDate="2017-06-17T19:06:43.897" Title="Document similarity: Vector embedding versus BoW performance?" Tags="|machine-learning|similarity|word2vec|information-retrieval|word-embeddings|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17431" PostTypeId="1" CreationDate="2017-03-07T21:42:42.570" Score="2" ViewCount="369" Body="&lt;p&gt;I am experiencing a weird offset in my Support Vector Regression prediction (code below).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A brief overview: I have a set of 10 .xls sheets as input data (each with 30 rows and 27 columns), and a 11th .xls sheet that I am using as my testing dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working with an electricity dataset, and I am trying to predict the 11th month of electricity usage using the SVR.  I am using the code written and released by Luke Benning on Github: &lt;a href=&quot;https://github.com/lbenning/Load-Forecasting&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/lbenning/Load-Forecasting&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of my datasets are structured the same, but when I run the prediction model, I am receiving an offset (as shown in the photo below).  The actual data versus the predicted data seem to follow a similar pattern, but the two plots are offset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of what I may want to explore in order to try and understand the source of this offset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/lbenning/Load-Forecasting&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eBYCh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#! /usr/bin/python&#xA;&#xA;import math&#xA;import statistics&#xA;import visualizer&#xA;import numpy as np&#xA;from datagen import constructData&#xA;from sklearn import svm&#xA;&#xA;# Applies Support Vector Regression to the electricity dataset,&#xA;# prints out the accuracy rate to the terminal and plots&#xA;# predictions against actual values&#xA;def suppVectorRegress():&#xA;&#xA;    kernelList = [&quot;linear&quot;,&quot;rbf&quot;,polyKernel]&#xA;    names = [&quot;linear&quot;,&quot;radial basis&quot;,&quot;poly&quot;]&#xA;    preds = []&#xA;&#xA;    # Retrieve time series data &amp;amp; apply preprocessing&#xA;    data = constructData()&#xA;&#xA;    cutoff = len(data)-30&#xA;    xTrain = data[0][0:cutoff]&#xA;    yTrain = data[1][0:cutoff]&#xA;    xTest = data[0][cutoff:]&#xA;    yTest = data[1][cutoff:]&#xA;&#xA;    # Fill in missing values denoted by zeroes as an average of&#xA;    # both neighbors&#xA;    statistics.estimateMissing(xTrain,0.0)&#xA;    statistics.estimateMissing(xTest,0.0)&#xA;&#xA;    # Logarithmically scale the data&#xA;    xTrain = [[math.log(y) for y in x] for x in xTrain]&#xA;    xTest = [[math.log(y) for y in x] for x in xTest]&#xA;    yTrain = [math.log(x) for x in yTrain]&#xA;&#xA;    # Detrend the time series&#xA;    indices = np.arange(len(data[1]))&#xA;    trainIndices = indices[0:cutoff]&#xA;    testIndices = indices[cutoff:]&#xA;    detrended,slope,intercept = statistics.detrend(trainIndices,yTrain)&#xA;    yTrain = detrended&#xA;&#xA;    for gen in range(len(kernelList)):&#xA;&#xA;        # Use SVR to predict test observations based upon training observations&#xA;        pred = svrPredictions(xTrain,yTrain,xTest,kernelList[gen])&#xA;        # Add the trend back into the predictions&#xA;        trendedPred = statistics.reapplyTrend(testIndices,pred,slope,intercept)&#xA;        # Reverse the normalization&#xA;        trendedPred = [math.exp(x) for x in trendedPred]&#xA;        # Compute the NRMSE&#xA;        err = statistics.normRmse(yTest,trendedPred)&#xA;&#xA;        print (&quot;The Normalized Root-Mean Square Error is &quot; + str(err) + &quot; using kernel &quot; + names[gen] + &quot;...&quot;)&#xA;&#xA;        preds.append(trendedPred)&#xA;&#xA;    names.append(&quot;actual&quot;)&#xA;    preds.append(yTest)&#xA;&#xA;    # Change the parameters 2017,2,1 based on the month you want to predict.&#xA;    visualizer.comparisonPlot(2017,2,1,preds,names,plotName=&quot;Support Vector Regression Load Predictions vs. Actual&quot;, &#xA;        yAxisName=&quot;Predicted Kilowatts&quot;)&#xA;&#xA;# Construct a support vector machine and get predictions&#xA;# for the test set&#xA;# Returns a 1-d vector of predictions&#xA;def svrPredictions(xTrain,yTrain,xTest,k):&#xA;    clf = svm.SVR(C=2.0,kernel=k)&#xA;    clf.fit(xTrain,yTrain)&#xA;    return clf.predict(xTest)&#xA;&#xA;# A scale invariant kernel (note only conditionally semi-definite)&#xA;def polyKernel(x,y):&#xA;    return (np.dot(x,y.T)+1.0)**0.95&#xA;&#xA;if __name__==&quot;__main__&quot;:&#xA;    suppVectorRegress()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am generating the data as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'''&#xA;Functions for retrieving Elia dataset &#xA;&amp;amp; forming training/testing datasets&#xA;'''&#xA;&#xA;# constructs dataset for simulations&#xA;# the last dataset in this file list, is the one used as the training set.&#xA;def constructData():&#xA;  files = [&quot;data/jan_16_elec_scaled.xls&quot;, &quot;data/feb_16_elec_scaled.xls&quot;, &#xA;           &quot;data/mar_16_elec_scaled.xls&quot;, &quot;data/apr_16_elec_scaled.xls&quot;,&#xA;           &quot;data/may_16_elec_scaled.xls&quot;,&quot;data/jun_16_elec_scaled.xls&quot;,&#xA;           &quot;data/jul_16_elec_scaled.xls&quot;, &quot;data/aug_16_elec_scaled.xls&quot;,&#xA;           &quot;data/sep_16_elec_scaled.xls&quot;, &quot;data/oct_16_elec_scaled.xls&quot;,&#xA;           &quot;data/nov_16_elec_scaled.xls&quot;]&#xA;&#xA;#  files = [&quot;data/jan_16_elec_NOscaled.xls&quot;, &quot;data/feb_16_elec_NOscaled.xls&quot;, &#xA;#           &quot;data/mar_16_elec_NOscaled.xls&quot;, &quot;data/apr_16_elec_NOscaled.xls&quot;,&#xA;#           &quot;data/may_16_elec_NOscaled.xls&quot;,&quot;data/jun_16_elec_NOscaled.xls&quot;,&#xA;#           &quot;data/jul_16_elec_NOscaled.xls&quot;, &quot;data/aug_16_elec_NOscaled.xls&quot;,&#xA;#           &quot;data/sep_16_elec_NOscaled.xls&quot;, &quot;data/oct_16_elec_NOscaled.xls&quot;,&#xA;#           &quot;data/nov_16_elec_NOscaled.xls&quot;]         &#xA;  return labelSeries(loadSeries(files))&#xA;&#xA;# constructs labelled data from a&#xA;# univariate time series&#xA;sdef labelSeries(series):&#xA;  xData = []&#xA;  yData = []&#xA;  for x in range(len(series)-1):&#xA;    xData.append(series[x]) # xData contains all of the items up until the last item&#xA;    yData.append(np.mean(series[x+1])) # yData is the last item in the list&#xA;  return (xData,yData)&#xA;&#xA;# arg1 : list of excel spreadsheets filenames&#xA;# returns : load univariate time series&#xA;def loadSeries(fileList):&#xA;  # Retrieve time series examples&#xA;  xData = []&#xA;  for fileName in fileList:&#xA;    book = xlrd.open_workbook(fileName)&#xA;    sheet = book.sheet_by_index(0)&#xA;    for rx in range(2,sheet.nrows):&#xA;      row = sheet.row(rx)[3:]&#xA;      row = [row[x].value for x in range(0,len(row)-4)]&#xA;      xData.append(row)&#xA;  return xData&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2017-03-15T05:55:29.633" Title="Confusing Offset in my Support Vector Regression and other Models" Tags="|regression|svm|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17433" PostTypeId="1" CreationDate="2017-03-07T22:59:23.357" Score="1" ViewCount="376" Body="&lt;p&gt;While I understand that if I train a model using the same data that I test on, then I'll certainly be overfitting, but will the accuracy of that model always be 100%?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, is one method of testing whether I've created my model correctly, to test the model by training it on the testing data?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="12202" LastEditDate="2017-03-07T23:59:27.477" LastActivityDate="2017-03-08T02:18:15.627" Title="If I train a model using only my testing data, will the accuracy be 100%?" Tags="|predictive-modeling|regression|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17434" PostTypeId="2" ParentId="17433" CreationDate="2017-03-07T23:13:09.863" Score="2" Body="&lt;p&gt;It will likely be not 100% even though it is possible. But, for example, a linear model might not be able to represent a non-linear relationship in your data. Or your features don't contain sufficient information to separate all classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your other question doesn't seem to make sense. If you train your model on the test data, the test data wouldn't really be test data anymore?! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are tons of resources on bias-variance trade-off and overfitting.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-03-07T23:13:09.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17436" PostTypeId="1" AcceptedAnswerId="17441" CreationDate="2017-03-08T01:23:09.807" Score="3" ViewCount="98" Body="&lt;p&gt;I seem to have a problem modelling my CNN network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to extract from features vector from different sized images. &#xA;Whats consistent with the images is the y-axis, and the color dimension, but the x-axis is not constant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the length of the x-axis will the length of the feature vector also be altered. I already know how long they should be, &#xA;but not the ratio between the length of the x-axis and length of the feature vector, I guess one does exist. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to train a CNN network such that it can alter the feature vector length depending on the length of the input of the x-axis of the image?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:47:03.857" LastActivityDate="2021-02-16T02:47:03.857" Title="Problem designing CNN network" Tags="|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17437" PostTypeId="2" ParentId="17433" CreationDate="2017-03-08T02:18:15.627" Score="2" Body="&lt;ul&gt;&#xA;&lt;li&gt;Just because you're training on your training set, that doesn't &lt;strong&gt;mean&lt;/strong&gt; you overfit your model. In fact, you &lt;strong&gt;must&lt;/strong&gt; train on your training set, this is definition. If you use the test set for training, then it's &lt;strong&gt;not&lt;/strong&gt; a test set.&lt;/li&gt;&#xA;&lt;li&gt;A common way to avoid overfitting is by regularization. Cross-validation is another possibility. There are many many other ways.&lt;/li&gt;&#xA;&lt;li&gt;It's possible to achieve 100% accuracy in both training, validation and testing set. That depends on your complexity of the problem and model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-03-08T02:18:15.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17438" PostTypeId="2" ParentId="16935" CreationDate="2017-03-08T03:14:52.187" Score="1" Body="&lt;p&gt;Your formula is correct, but the final computing is wrong.&#xA;It should be: $\frac{10+87}{270+2000} \times \frac{10+100}{270+2001}$&lt;/p&gt;&#xA;" OwnerUserId="29744" LastEditorUserId="29744" LastEditDate="2017-03-08T09:16:15.207" LastActivityDate="2017-03-08T09:16:15.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17439" PostTypeId="2" ParentId="9767" CreationDate="2017-03-08T05:20:38.237" Score="1" Body="&lt;p&gt;clustering algo would take any data type as long as it is measurable, ideally shouldn't be string (which can be translated into vectors using topics).&#xA;If you tell me what application you are using i can guide you on how to go about it step by step&lt;/p&gt;&#xA;" OwnerUserId="29746" LastActivityDate="2017-03-08T05:20:38.237" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17441" PostTypeId="2" ParentId="17436" CreationDate="2017-03-08T07:58:56.163" Score="2" Body="&lt;p&gt;It is not really possible to alter input feature array size per example on normal CNNs. Instead this is fixed when the model is built for the first time, before you start training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your goal, it might be possible to work around that using some kind of pipeline that worked with image patches (taking multiple slightly randomised patches to augment the training data can improve results and doing the same with prediction inputs can also drive up classifier accuracy). Or &lt;a href=&quot;https://arxiv.org/abs/1406.6247&quot; rel=&quot;nofollow noreferrer&quot;&gt;a more complex variant using RNN/CNN hybrid&lt;/a&gt; to consume an image as a sequence of parts, which might also be used for multi-object recognition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, these solutions are complex, and state-of-the-art results in image classification can be achieved by simpler techniques such as taking a centre crop and/or padding. Provided your training data is also treated the same way, and aspect ratios are not extreme this can work adequately.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-08T08:07:46.147" LastActivityDate="2017-03-08T08:07:46.147" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17442" PostTypeId="1" CreationDate="2017-03-08T08:34:57.180" Score="1" ViewCount="273" Body="&lt;p&gt;1) After using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA of scikit learn&lt;/a&gt; I realized it overloads the swap, but never reach more than 20% of the RAM. Do you have any idea why ? I can't really see if this is a problem, but still I would like to understand what happens in memory !   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The parallelization of this algorithm (using parameter n_jobs=-1) leads to memory errors on my server (192 GB of RAM, 32 CPU cores). I think this is due to the fact that parallelization uses forks to create threads, leading to copying 32 time the current memory (which can be quite big if data are loaded in memory). How could I improve this ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fq4EW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fq4EW.png&quot; alt=&quot;A screen shot of my server&amp;#39;s state when i launched LDA (n_jobs=1) around 16:00&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-03-08T14:37:55.960" LastActivityDate="2017-03-08T14:37:55.960" Title="Scikit Learn Latent Dirichlet Allocation overload my SWAP and/or RAM" Tags="|python|scikit-learn|lda|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17443" PostTypeId="2" ParentId="17414" CreationDate="2017-03-08T08:41:41.227" Score="2" Body="&lt;p&gt;You can use Kafka to ingest data into HDFS or any other cloud storage like S3 or Google storage.&#xA; And you can use Gobblin to schedule your kafka consumer to write into HDFS.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Kafka Producers =&amp;gt; Kafka Consumer =&amp;gt; Gobblin (Monthly/weekly/Daily/hourly/minutes) =&amp;gt; HDFS&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this will work for you &lt;/p&gt;&#xA;" OwnerUserId="29749" LastActivityDate="2017-03-08T08:41:41.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17444" PostTypeId="1" CreationDate="2017-03-08T11:01:26.670" Score="13" ViewCount="7672" Body="&lt;p&gt;So far there are many interesting applications for deep learning in computer vision or natural language processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is it in other more traditional fields?&#xA;For example, I have traditional socio-demographic variables plus maybe a lot of lab measurements and want to predict a certain disease. Would this be a deep learning application if I have lots of observations?&#xA;How would I construct a network here, I think all the fancy layers (convolutional etc.) are not really necessary?! Just make it deep?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On my specific data set, I tried some common machine learning algorithms like random forests, gbm etc with mixed results regarding accuracy.&#xA;I have limited deep learning experience with image recognition.&lt;/p&gt;&#xA;" OwnerUserId="15664" LastEditorUserId="15664" LastEditDate="2017-03-08T12:59:37.727" LastActivityDate="2018-09-25T18:33:01.117" Title="deep learning for non-image non-NLP tasks?" Tags="|deep-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17445" PostTypeId="2" ParentId="15135" CreationDate="2017-03-08T13:03:35.687" Score="81" Body="&lt;p&gt;There is a great answer to this question over on &lt;a href=&quot;https://stackoverflow.com/a/38251213/1185293&quot;&gt;SO&lt;/a&gt; that uses numpy and pandas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The command (see the answer for the discussion):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;produces a 60%, 20%, 20% split for training, validation and test sets.&lt;/p&gt;&#xA;" OwnerUserId="13821" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-03-08T13:03:35.687" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17446" PostTypeId="2" ParentId="17444" CreationDate="2017-03-08T13:20:23.043" Score="12" Body="&lt;p&gt;Yes you can use deep learning techniques to process non-image data. However, other model classes are still very competitive with neural networks outside of signal-processing and related tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use deep learning approaches on non-signal/non-sequence data, typically you use a simple feed-forward multi-layer network. No need for convolutional layers or pooling layers. The best architecture other than that needs to be explored with cross-validation, and can be time-consuming to discover as deep NNs take a lot of computation to train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience attempting to use deep(-ish, typically ~ 5 layers) neural networks in Kaggle competitions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Dropout is still highly effective for regularisation and improving accuracy&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Input normalisation - usually to mean 0, standard deviaton 1, is important&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hidden layer activation functions can make a difference. Although ReLU reduces some problems with vanishing gradients, in my experience it is less robust with non-signal data and you will want some other form. If you have only a few layers, then sigmoid or tanh still work OK. Otherwise, look into leaky ReLU, PReLU, ELU and other ReLU variants that attempt to patch its problems with &quot;dead&quot; neurons.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Make use of optimisers designed for deep learning, such as Adam, Adagrad or RMSProp&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a weight initialisation approach that works with deep learning, such as Glorot.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider using Batch Normalisation layers. Not something I have much experience with, but I have seen other people do well with this approach.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Despite all this, XGBoost can routinely and easily beat deep NNs with minimal tuning and training effort in comparison (depending of course on the problem and the data you have). If accuracy is everything to you though, it is possible - although not guaranteed - that an ensemble of deep NNs and other model such as XGBoost will perform better than either singly.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-08T13:39:05.683" LastActivityDate="2017-03-08T13:39:05.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17447" PostTypeId="1" CreationDate="2017-03-08T13:53:21.987" Score="1" ViewCount="98" Body="&lt;p&gt;Is the longer encoding array (word2vec or any other kind) always more precise than the shorter (not regarding the computational cost, only the performance)?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21560" LastEditDate="2017-03-09T09:25:18.103" LastActivityDate="2017-03-10T14:34:24.593" Title="The larger an encoding dimension in NLP the better?" Tags="|nlp|encoding|" AnswerCount="2" CommentCount="3" ClosedDate="2017-03-11T05:04:51.463" ContentLicense="CC BY-SA 3.0" />
  <row Id="17448" PostTypeId="2" ParentId="17444" CreationDate="2017-03-08T15:27:35.917" Score="1" Body="&lt;p&gt;A network can be appropriate for classification purposes. For this, you need to be able to define a training-set and a test-set of your data that represents the data the network will be asked to classify in production. This determines if you can get a bad, a reasonable or a good working network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I consider terms as &quot;deep-learning&quot; as misleading: a network does not learn, you can only train it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you can create a training and test set, on high level you can use a&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Multi-layer: if your data has no order and structures have a fixed position.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recursive networks: if the order of the data is of importance for the classification&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Convolution: if your data has structures like in images but there is no fixed position. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Getting a good setup, like the number of layers, requires trial and error; it is a kind of black magic. &lt;/p&gt;&#xA;" OwnerUserId="15480" LastActivityDate="2017-03-08T15:27:35.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17449" PostTypeId="1" CreationDate="2017-03-08T16:05:08.237" Score="1" ViewCount="173" Body="&lt;p&gt;I am trying to create a sentiment analysis system using deep learning (with R).&#xA;I have a review base of customer reviews which for each review the customer make a mark (from 0 to 10). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this, using marks in the range 0..10?  Is it possible?  In some tutorials, I see that authors use 0, 1 or -1 (0 for neutral, 1, positive and -1 for negative), and I'm not sure how to adjust to marks in the range 0..10.&lt;/p&gt;&#xA;" OwnerUserId="29768" LastEditorUserId="8560" LastEditDate="2017-03-09T04:30:27.260" LastActivityDate="2017-03-09T04:30:27.260" Title="Sentiment analysis with continuous scale for responses" Tags="|predictive-modeling|deep-learning|regression|sentiment-analysis|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17450" PostTypeId="5" CreationDate="2017-03-08T16:31:14.750" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-03-08T16:31:14.750" LastActivityDate="2017-03-08T16:31:14.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17451" PostTypeId="4" CreationDate="2017-03-08T16:31:14.750" Score="0" Body="Rattle is a graphical user interface for data mining using R. http://rattle.togaware.com" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-03-11T05:06:36.563" LastActivityDate="2017-03-11T05:06:36.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17452" PostTypeId="1" AcceptedAnswerId="17456" CreationDate="2017-03-08T17:33:35.040" Score="1" ViewCount="326" Body="&lt;p&gt;I'm trying to retrain Inception model final layer for a binary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training image set contain 2000 images in class 1 and more than 6000 images in class 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will this huge difference in number of images of each class in training set affect my classification?&lt;/p&gt;&#xA;" OwnerUserId="29771" LastActivityDate="2017-03-08T18:59:22.883" Title="Retrain final layer of Inception model" Tags="|deep-learning|tensorflow|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17453" PostTypeId="1" CreationDate="2017-03-08T17:44:34.347" Score="1" ViewCount="388" Body="&lt;p&gt;Consider the following code in Keras for building a LSTM model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    model = Sequential()&#xA;    model.add(LSTM(30, input_dim=22,return_sequences=True, init = 'glorot_uniform'))&#xA;&#xA;    model.add(TimeDistributed(Dense(1)))&#xA;    model.compile(loss='mean_squared_error', optimizer='Nadam')&#xA;&#xA;    model.fit(train3d, trainY3d, nb_epoch=100, batch_size=8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As one can see there is only 1 LSTM layer. I see there are 14 matrices as output. If you check this &lt;a href=&quot;http://euler.stat.yale.edu/~tba3/stat665/lectures/lec21/notebook21.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; you would notice that there are 4 triples (Input gate, Forget gate, Cell State, Output gate) of W, U (parameter matrices) and b (bias vector) each for the LSTM layer. And then there are 2 matrices W and b for the dense layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is let's say for this case of 1 layer LSTM, is there a way to attribute 100% of Y to the impact for each input feature $X_i \forall i$. &lt;/p&gt;&#xA;" OwnerUserId="27735" LastActivityDate="2017-03-08T17:44:34.347" Title="LSTM Weight Matrix Interpretation" Tags="|neural-network|keras|rnn|theano|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17454" PostTypeId="2" ParentId="13304" CreationDate="2017-03-08T18:29:45.283" Score="2" Body="&lt;p&gt;Using frequentist hypothesis testing &lt;em&gt;in this manner&lt;/em&gt; (using the p-value to determine pass/fail) has serious drawbacks even in bug-free software.  My suggestion is to use some form of Bayesian A-B testing.  There is much nerd debate about this (frequentist vs. Bayesian A/B testing).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, some references.  Then we'll have a look at your AA test with the online calculator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;On using p-values as pass/fail criteria:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomerror/EP713_RandomError7.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Limitations of p-values&lt;/a&gt; (blog regarding epidemiological studies)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/isre.2013.0480&quot; rel=&quot;nofollow noreferrer&quot;&gt;Too Big to Fail: Large Samples and the p-Value Problem&lt;/a&gt; (research article)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.statisticsdonewrong.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Statistics Done Wrong&lt;/a&gt; (very accessible book)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;On Bayesian A/B testing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2788602&quot; rel=&quot;nofollow noreferrer&quot;&gt;LinkedIn's use of Bayesian A/B testing&lt;/a&gt; (conference proceedings)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://conversionxl.com/bayesian-frequentist-ab-testing/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian/Frequentist nerd debate example&lt;/a&gt; (blog)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Methods for Hackers&lt;/a&gt; (book, online notebook)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Peak Conversion Online Calculator&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7x34c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7x34c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F5H4E.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case the frequentist and Bayesian approaches agree-- your &quot;A&quot; is not equal to your other &quot;A&quot;.  It appears that there are one or more currently unseen variable(s) you need to take into account. If it was a question of insufficient data, the two peaks would not be so nicely separated.  I'd be interested in hearing other possible explanations. &lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2017-03-08T18:29:45.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17455" PostTypeId="1" AcceptedAnswerId="17457" CreationDate="2017-03-08T18:53:45.300" Score="1" ViewCount="79" Body="&lt;p&gt;I have some terminal charging values for US and CHINA &#xA;comes in a pandas DataFrame like the following, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       value country  &#xA;0      550.0     USA        &#xA;1      820.0   CHINA        &#xA;2      835.0   CHINA        &#xA;3      600.0   USA        &#xA;4      775.0   CHINA        &#xA;5      785.0   USA        &#xA;6      790.0   USA   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the sample data and I have in total 5K+ entries. The data is cleared for the outliers and needs to be visualized. What kind of visualization can I use to plot my data meaningfully?     &lt;/p&gt;&#xA;" OwnerUserId="29772" LastEditorUserId="8878" LastEditDate="2017-03-09T08:34:50.433" LastActivityDate="2017-03-09T09:29:02.753" Title="Which graph will be appropriate for the visualization task?" Tags="|visualization|data|pandas|plotting|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17456" PostTypeId="2" ParentId="17452" CreationDate="2017-03-08T18:59:22.883" Score="0" Body="&lt;p&gt;The short answer is &lt;strong&gt;yes&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While your classes are imbalanced, model will be more likely to &quot;learn&quot; class 2 during the training, especially when it comes to mini-batch updates. (even though each mini-batch is unlikely to have only class 2 samples)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The common solution is to use weighted loss function or feed weighted samples to mini-batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my practice, I always use weighted samples if I have sufficient data, even when my classes are not that skewed. Weights always help, at least it never ruins the model.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-08T18:59:22.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17457" PostTypeId="2" ParentId="17455" CreationDate="2017-03-08T19:11:57.663" Score="1" Body="&lt;p&gt;Maybe you can try something like this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.hist(by='country', bins=50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to plot these histograms in the same figure, checkout &lt;a href=&quot;https://stackoverflow.com/questions/25539195/multiple-histograms-in-pandas&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-03-08T19:11:57.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17458" PostTypeId="1" AcceptedAnswerId="17481" CreationDate="2017-03-08T20:52:22.510" Score="2" ViewCount="426" Body="&lt;p&gt;Does anyone have any books or blogs that specifically sheds light on questions to ask your organization (from a consulting POV) as a data scientist? I am a new data scientist, which I have a background in consulting and predictive analytics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several good reads such as &quot;The Data Scientist's Field Guide&quot; by Booz Allen, &quot;The McKinsey Mind&quot; from McKinsey, or maybe &quot;Business Analytics for Managers&quot; from SAS. I have come across interview guides to understand and execute strategy, but none that specifically structure questions for a preliminary assessment of analytic capability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any recommendations would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="27519" LastActivityDate="2017-03-09T23:06:34.927" Title="Data Scientist Consulting Interview Guide" Tags="|reference-request|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-11T05:05:13.487" ContentLicense="CC BY-SA 3.0" />
  <row Id="17459" PostTypeId="2" ParentId="17455" CreationDate="2017-03-08T20:57:00.913" Score="1" Body="&lt;p&gt;If you need to be able to identify good and bad charge times then the points themselves are important. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have my x-axis be my country names and then simply plot the charge times on the y-axis as scattered points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give you scattered data in the y-direction digitized for each country. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you will be able to extract a lot of meaningful information from this plot for comparing different countries such as: max, min, approx. mean, etc..&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-03-08T21:19:54.087" LastActivityDate="2017-03-08T21:19:54.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17460" PostTypeId="1" AcceptedAnswerId="17553" CreationDate="2017-03-08T22:22:07.980" Score="0" ViewCount="33295" Body="&lt;p&gt;I use the &lt;code&gt;randomForest&lt;/code&gt; package in R but I'm getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in eval(expr, envir, enclos) : object 'a' not found&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The line that causes the problem is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model = randomForest(class ~ a + b + c, data=train, ntree=100&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked up the error and apparently it has to do with the fact that &lt;code&gt;train&lt;/code&gt; does not contain &lt;code&gt;a&lt;/code&gt;, but I manually checked &lt;code&gt;train&lt;/code&gt; and it does contain &lt;code&gt;a&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a matter of fact, when I try to build a decision tree instead, it works just fine! (with the same formula and dataset)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = rpart(class ~ a + b + c, data=train, method=&quot;class&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I don't understand where this error is coming from. Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: There seems to be a problem with the fact that I use bigrams for my text. If I just use unigrams, it works fine. If I use this code for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = randomForest(class ~ `stack overflow`, data=train, ntree=100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It returns the error named above: object 'stack overflow' cannot be found.&lt;/p&gt;&#xA;" OwnerUserId="29008" LastEditorUserId="29008" LastEditDate="2017-03-08T22:34:03.423" LastActivityDate="2017-03-13T13:26:18.283" Title="Random Forest in R: Error in eval(expr, envir, enclos): object not found" Tags="|r|random-forest|decision-trees|" AnswerCount="1" CommentCount="2" ClosedDate="2017-03-16T22:32:46.343" ContentLicense="CC BY-SA 3.0" />
  <row Id="17461" PostTypeId="1" AcceptedAnswerId="17464" CreationDate="2017-03-09T00:08:12.270" Score="7" ViewCount="57367" Body="&lt;p&gt;I am trying to test how well my unsupervised K-Means clustering properly clusters my data.  I have an unsupervised K-Means clustering model output (as shown in the first photo below) and then I clustered my data using the actual classifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1y0zS.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1y0zS.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The photo below are the actual classifications.  I am trying to test, in Python, how well my K-Means classification (above) did against the actual classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AeHnt.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AeHnt.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my K-Means code, I am using a simple model, as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kmeans = KMeans(n_clusters=4, random_state=0).fit(myData)&#xA;labels = kmeans.labels_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What would be the best way for me to compare how well my unsupervised KMeans clustering model did against the actual classifications?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2019-12-04T14:13:13.087" Title="How to test accuracy of an unsupervised clustering model output?" Tags="|clustering|k-means|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17462" PostTypeId="1" CreationDate="2017-03-09T01:28:16.007" Score="2" ViewCount="108" Body="&lt;p&gt;I am not sure if the title accurately reflects my problem but I essentially would like to aggregate a set of metrics of similar nature that comes from different data sources into a single metric. Say we are measuring MetricA from SourceA, MetricB from SourceB,...and MetricN from SourceN sampled over the same general population. There is no guarantee that the union of samples will cover all in the population at least once. With varying sampling rates and frequencies across sources, I would like to come to some level of normalization to sum it up at the top and track that over time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what the best way is to come up with weights though. Has any of you come across with a similar problem or perhaps some papers that found reasonable approaches to avoid/minimize bias?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="15930" LastEditorUserId="15930" LastEditDate="2017-03-09T01:54:18.103" LastActivityDate="2017-03-09T01:54:18.103" Title="Aggregating over sparse data" Tags="|aggregation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17463" PostTypeId="1" AcceptedAnswerId="17474" CreationDate="2017-03-09T02:45:50.277" Score="1" ViewCount="1694" Body="&lt;p&gt;I am currently working on designing a certain number of CNN for extracting features from images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The images are spectogram, and each have a shape being (276,x,3).&#xA;X is here the number of column, which incidently also the length of the feature vector that should  be created. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution. The most important factor is the 1d convolution and the shape of the 2d kernel, as it is being used to alter the importance of some of the entries in the spectogram. &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:46:43.417" LastActivityDate="2021-02-16T02:46:43.417" Title="Designing CNN that does one column convolution across the x-axis" Tags="|python|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17464" PostTypeId="2" ParentId="17461" CreationDate="2017-03-09T06:45:01.350" Score="6" Body="&lt;p&gt;Since you have the actual labels, you can compare them with the obtained labels and evaluate performance. Typically purity and nmi (normalized mutual information) are used. Read this (&lt;a href=&quot;http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html&quot; rel=&quot;noreferrer&quot;&gt;Evaluation of Clustering&lt;/a&gt;) document for detailed explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have actual labels then you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Modularity_(networks)&quot; rel=&quot;noreferrer&quot;&gt;modularity&lt;/a&gt; or [Silhouette] for measuring clustering performance.&lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-03-09T06:45:01.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17465" PostTypeId="1" CreationDate="2017-03-09T07:15:26.977" Score="2" ViewCount="754" Body="&lt;p&gt;I have a function $f(x_1, x_2) = Y$ and the goal would be a user gives me a variable list of Y's where each Y was generated from the same constant $x_2$ parameter but $x_1$ varies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want my program to give my best guess of $x_2$. I'd like to try and train a deep neural network to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few problems is that the order of Y that the user can give me will be random and the number of Y's will vary as well. I'd hope the more Y's they gave me the more accurate my prediction of what $x_2$ would be is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not quite sure how to setup a model where it's unordered with variable length inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x_1, x_2) = x_1 \cdot x_2 + noise$ where $x_1$ and $x_2$ are $\in \mathbb{R}^3$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&#xA;&lt;li&gt; $x_2$ is unknown and I'm giving lots of examples where $x_2$ is fixed but $x_1$ varies&#xA;&lt;li&gt; I don't know what $x_1$ is but I observe the output $f(x_1, x_2)$&#xA;&lt;li&gt; I can train the neural network with synthesized data&#xA;&lt;li&gt; I'd imagine if the user gave more observations I'd get a more accurate response&lt;/p&gt;&#xA;" OwnerUserId="29790" LastEditorUserId="29790" LastEditDate="2017-03-11T19:06:14.870" LastActivityDate="2017-03-11T19:06:14.870" Title="Variable length unordered inputs for a deep neural network" Tags="|deep-learning|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="17467" PostTypeId="2" ParentId="17455" CreationDate="2017-03-09T09:29:02.753" Score="0" Body="&lt;p&gt;It depends on what is your objective.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have so many values, you may want to plot some summary data, so you can use either two &lt;a href=&quot;http://seaborn.pydata.org/generated/seaborn.distplot.html#seaborn.distplot&quot; rel=&quot;nofollow noreferrer&quot;&gt;Distplots&lt;/a&gt; in a FacetGrid to have the two countries side by side or even better a &lt;a href=&quot;http://seaborn.pydata.org/generated/seaborn.boxplot.html#seaborn.boxplot&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boxplot&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the order of the appearance of the values matters you can try plot them as two different lines with the values on the y-axis by using FactorPlot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(All the mentioned plots above are part of the Seaborn library)&lt;/p&gt;&#xA;" OwnerUserId="29799" LastActivityDate="2017-03-09T09:29:02.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17468" PostTypeId="2" ParentId="15387" CreationDate="2017-03-09T10:10:36.540" Score="1" Body="&lt;p&gt;Concerning the size of the data - have you tried compressing the files using the Python tarfile library? You could compress it in chunks and after each successful compression operation, and keep a sql db with the metadata in it. That way when you do your sample selection you can perform the selection on the db and pull only the files you need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would go for it and split it up. As far as a selection method I would kick it old school and do a random selection with replacement. &lt;/p&gt;&#xA;" OwnerUserId="29801" LastEditorUserId="29801" LastEditDate="2017-03-09T12:04:42.760" LastActivityDate="2017-03-09T12:04:42.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17469" PostTypeId="1" CreationDate="2017-03-09T12:32:53.170" Score="3" ViewCount="262" Body="&lt;p&gt;Where exactly bigdata platforms fit in to a data science/ machine learning projects ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a large dataset for a binary classification problem - cats and dogs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to create a model for real time classification&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Since dataset is huge I can make use any distributed platform for faster computation and model creation right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Once the model is ready, then there is no need of these distributed platform right ? or are they needed for feature extractions ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastEditorUserId="28628" LastEditDate="2017-03-22T08:46:07.137" LastActivityDate="2018-04-30T18:33:11.157" Title="Deploying models on bigdata platforms like Hadoop and Spark" Tags="|machine-learning|data-mining|bigdata|apache-spark|apache-hadoop|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17470" PostTypeId="1" AcceptedAnswerId="17472" CreationDate="2017-03-09T12:33:59.773" Score="6" ViewCount="1312" Body="&lt;p&gt;I'm taking a machine learning course that introduces the related assignments as such that a student may use either Pandas or SFrame in solving them. As a beginner, it's hard for me to assess which approach would be more beneficial for me in the long run. Hence the question; should I prefer Pandas or SFrame for learning purposes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I might've preferred Pandas to make sure I learn the &quot;proper foundations&quot; first, so to speak. However, after reading about this online, SFrame appears to be very popular after having been made open source. Can it be considered a Pandas replacement from a (future) data scientist's point of view? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a bunch! Protips always welcome in comments ;)&lt;/p&gt;&#xA;" OwnerUserId="29806" LastActivityDate="2017-03-09T16:33:51.563" Title="Pandas v. SFrame in learning data science" Tags="|machine-learning|python|data-mining|pandas|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17471" PostTypeId="1" CreationDate="2017-03-09T13:17:33.503" Score="13" ViewCount="6592" Body="&lt;p&gt;I have understood how GAN works while two networks (generative and discriminative) compete with each other. I have built a DCGAN (GAN with convolutional discriminator and de-convolutional generator) which now successfully generates handwritten digits similar to those in MNIST dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read a lot about GAN's applications for extracting features from images. How can use my trained GAN model (on MNIST dataset) to extract feature from MNIST handwritten digist images?&lt;/p&gt;&#xA;" OwnerUserId="17947" LastActivityDate="2019-12-04T04:10:02.527" Title="How to use GAN for unsupervised feature extraction from images?" Tags="|feature-extraction|unsupervised-learning|gan|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17472" PostTypeId="2" ParentId="17470" CreationDate="2017-03-09T16:33:51.563" Score="3" Body="&lt;p&gt;SFrame is not used much in industry, so I'd stick to pandas or &lt;strong&gt;Spark&lt;/strong&gt; DataFrames. But they're all rather similar, and you should not spend much time thinking about it: it is easy to pick these things up, and employers understand this. Concentrate on the algorithms; that's the real &quot;foundation&quot;, not the tools.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-03-09T16:33:51.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17474" PostTypeId="2" ParentId="17463" CreationDate="2017-03-09T17:23:12.417" Score="1" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;Designing CNN that does &lt;strong&gt;one column convolution across the x-axis&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I'm not sure I understand your question correctly, but if you have input images that are 276 high, X=500 wide and have 3 color channels, then the following performs &lt;strong&gt;column-wise convolutions across the X-axis&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;rows, cols, chans = 276, 500, 3&#xA;&#xA;...&#xA;model.add(Convolution2D(64, rows, 1, input_shape=(rows, cols, chans)))&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The output shape of this layer will be (1, 500, 64).&lt;/p&gt;&#xA;&lt;p&gt;I inserted &amp;quot;64&amp;quot; as the number of filters.. you can set that to whatever you need / whatever works best.&lt;/p&gt;&#xA;&lt;p&gt;Note: shapes are specified in tensorflow mode (i.e. the order is (rows, cols, channels)). Depending on your configuration you might have to change those shapes to theano mode: (channels, rows, cols).&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;So the cnn somehow has to use a kernel that has to be &lt;strong&gt;276 rows and 1 column wide&lt;/strong&gt;,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;It actually has to use a kernel that is 276 rows, 1 column &lt;strong&gt;and 3 channels&lt;/strong&gt; wide since your input images have 3 channels.. it would be 276 rows and 1 column wide if you used grayscale images.. but that shouldn't matter.&lt;/p&gt;&#xA;" OwnerUserId="676" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-03-09T17:23:12.417" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17476" PostTypeId="2" ParentId="17370" CreationDate="2017-03-09T18:58:41.887" Score="3" Body="&lt;p&gt;You can simply use the &lt;code&gt;heatmap&lt;/code&gt; &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/heatmap.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;function&lt;/a&gt; in the basic {stats} package. If desired, this can also display a dendrogram to show clusters. For your sample data (with &lt;code&gt;heatmap(df)&lt;/code&gt;) this would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ge5ui.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ge5ui.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On &lt;a href=&quot;https://www.r-bloggers.com/drawing-heatmaps-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;r-bloggers&lt;/a&gt; you can find some additional comments about the functionalities of &lt;code&gt;heatmap&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-03-09T18:58:41.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17478" PostTypeId="2" ParentId="17370" CreationDate="2017-03-09T19:36:12.957" Score="4" Body="&lt;p&gt;I'm expecting your data in this form (use &lt;code&gt;melt&lt;/code&gt; to transform the data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   type product value&#xA;1   A1      p1   158&#xA;2   A2      p1   343&#xA;3   A3      p1   247&#xA;4   B1      p1   400&#xA;5   B2      p1   398&#xA;6   B3      p1   408&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You may easily (using &lt;code&gt;lattice&lt;/code&gt; package) draw a &lt;code&gt;barchart&lt;/code&gt; diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Personally, I'd switch to heatmap only after this diagram is no more feasible due to large number of levels in the product or channel factors.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lattice)&#xA;barchart(product~ value | type,&#xA;data =  dfp,&#xA;scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,  &#xA;ylab=&quot;Product&quot;, xlab=&quot;Volume&quot;, &#xA;main= &quot;Matrix Visualization&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZnqYm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZnqYm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course you may also swap the dimensions by changing the formula. Use the parameter &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/lattice/html/xyplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;layout&lt;/a&gt; to align the barchart based on the number of levels.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;barchart(type~ value | product,&#xA;data =  dfp,&#xA;scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,  layout= c(5,1),&#xA;ylab=&quot;Channel&quot;, xlab=&quot;Volume&quot;, &#xA;main= &quot;Matrix Visualization&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZnqYm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VeMp4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="10620" LastEditDate="2017-03-14T18:03:21.180" LastActivityDate="2017-03-14T18:03:21.180" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17479" PostTypeId="1" CreationDate="2017-03-09T21:13:23.373" Score="-1" ViewCount="310" Body="&lt;p&gt;I am using Orange data mining in order to create a facial recognition workflow with some prototype widgets. To do this, I am trying to import some images (test data) using the import widget.&lt;/p&gt;&#xA;&lt;p&gt;My question is: which file format do I need to use in order for the image data to be compatible with the import widget? In other words, which file format should I use to upload the image?&lt;/p&gt;&#xA;&lt;p&gt;I have tried using JPEG but when I click &amp;quot;browse for images&amp;quot; to search for images on my desktop, it doesn't allow me to select anything to import.&lt;/p&gt;&#xA;&lt;p&gt;Does this imply that it is not compatible? Is there a specific file format that I should be using in order to do the image detection/embedding process?&lt;/p&gt;&#xA;" OwnerUserId="29169" LastEditorUserId="29169" LastEditDate="2022-09-24T20:36:28.047" LastActivityDate="2022-09-24T20:36:28.047" Title="What is the correct file extension for Orange image embedding?" Tags="|orange|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17480" PostTypeId="1" CreationDate="2017-03-09T22:30:51.587" Score="1" ViewCount="182" Body="&lt;p&gt;A question on the paper &lt;a href=&quot;https://arxiv.org/abs/1503.03832&quot; rel=&quot;nofollow noreferrer&quot;&gt;Schroff, Kalenichenko and Philbin &quot;FaceNet: A Unified Embedding for Face Recognition and Clustering&quot;, 2015.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is my understanding that the input to the CNN is a triplet (hence the name) of three pictures or thumbnails - one picture is the anchor, another one is a picture of the same person as the anchor picture, and the third one is a picture of a different person as the anchor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output is one 128 to 256 embedding for the anchor picture, so that embeddings of pictures of the same person are closer to such output as are embeddings of pictures of different people.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the input for training, as I understood, is three pictures and careful selection of the embeddings is crucial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;But what about testing? I thought the goal is to get the embedding of any one picture - or are three pictures (anchor, same person, different person) also necessary for testing?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="9197" LastActivityDate="2017-03-09T22:30:51.587" Title="conceptual question about triplet loss embeddings" Tags="|embeddings|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17481" PostTypeId="2" ParentId="17458" CreationDate="2017-03-09T23:06:34.927" Score="0" Body="&lt;p&gt;&lt;a href=&quot;http://datadrivenculture.org/CCI_Analytics_Capability_Assessment%202017.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; might be a resource that you would be interested in. It's a preliminary assessment for health centers, but a lot of the structure can be used in any industry. &lt;/p&gt;&#xA;" OwnerUserId="28694" LastActivityDate="2017-03-09T23:06:34.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17483" PostTypeId="1" CreationDate="2017-03-09T23:21:26.503" Score="1" ViewCount="18" Body="&lt;p&gt;I'm trying to find a tool or workflow that would allow a team to analysis data in multiple steps, where intermediate results are cached. (I'll assume that the raw data is in a relational database and the end result shall be shown in a web UI.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiple steps: This requirement can be describe in SQL as VIEWS:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CREATE VIEW b AS SELECT FROM a WHERE ... GROUP BY ... HAVING ...&#xA;CREATE VIEW c AS SELECT FROM b WHERE ... GROUP BY ... HAVING ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Caching: Additionally to the process described above, we want to temporarily store the 'views' b and c. (Either until the underlying data changes or for a set amount of time. The latter is acceptable for the use case.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saving the view as a (temporary) tables, updated periodically, comes to mind, but we would have to build a workflow around this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Team: This project requires some collaboration. A user should be able share queries with a team and have others build on top of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to avoid reinventing the wheel. Is there a technology stack that does this already?&lt;/p&gt;&#xA;" OwnerUserId="29822" LastActivityDate="2017-03-09T23:21:26.503" Title="Run queries against results of previously cached queries" Tags="|visualization|tools|sql|" AnswerCount="0" CommentCount="1" ClosedDate="2017-03-11T05:05:29.580" ContentLicense="CC BY-SA 3.0" />
  <row Id="17484" PostTypeId="1" AcceptedAnswerId="32003" CreationDate="2017-03-09T23:38:07.887" Score="5" ViewCount="659" Body="&lt;p&gt;I am using tensorflow-serving to write a server to consume models in production.&#xA;I have a question about consuming the service by clients: does tensorflow-serving support a REST API? Is there is anyway to modify it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have checked several github projects: &lt;a href=&quot;https://github.com/sugyan/tensorflow-mnist/tree/master/mnist&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, &lt;a href=&quot;https://github.com/Irtza/tensorflow_Server&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, and &lt;a href=&quot;https://github.com/benman1/tensorflow_flask&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="29575" LastEditDate="2017-12-29T16:18:27.673" LastActivityDate="2018-05-23T00:03:56.547" Title="tensorflow in production" Tags="|machine-learning|python|deep-learning|data-mining|tensorflow|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17485" PostTypeId="1" AcceptedAnswerId="17488" CreationDate="2017-03-10T00:39:05.120" Score="1" ViewCount="383" Body="&lt;p&gt;I have two years historical health claims data of one thousand members. Based on this two years data, I have to predict length of stay in hospital in 3rd year for all members. here is the data sample.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Year MembID x1 x2 x3 x4 x5 x6 x7 LengthOfStay&#xA;&#xA;2010 1 6 35 0 3 0 0 4 1&#xA;&#xA;2010 1 8 35 0 5 0 0 3 0&#xA;&#xA;2009 1 5 35 0 5 0 0 3 3&#xA;&#xA;2009 1 3 35 0 8 2 0 8 0&#xA;&#xA;2010 2 6 30 0 3 3 2 4 0&#xA;&#xA;2010 2 8 30 0 5 0 0 3 0&#xA;&#xA;2009 2 5 30 0 5 0 0 3 0&#xA;&#xA;2010 2 5 30 1 5 0 2 2 0&#xA;&#xA;2009 3 5 55 1 5 1 2 2 0&#xA;&#xA;2010 3 10 55 1 5 0 2 2 0&#xA;&#xA;2010 3 5 55 1 5 1 2 2 0&#xA;&#xA;2009 3 10 55 1 5 0 0 2 0&#xA;&#xA;2010 4 5 24 1 5 0 0 2 0&#xA;&#xA;2009 4 3 24 1 8 0 0 2 0&#xA;&#xA;2009 5 10 65 1 5 1 2 4 5&#xA;&#xA;2009 5 5 65 1 5 0 2 3 0&#xA;&#xA;2010 5 6 65 1 3 0 0 4 1&#xA;&#xA;2010 5 4 65 1 5 0 0 4 0&#xA;2010 6 10 44 1 5 1 2 4 5 &#xA;2011--- i expect------ 1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I did the classification with randomforest. How can I proceed further for prediction on 2011?&lt;/p&gt;&#xA;" OwnerUserId="29824" LastEditorUserId="29824" LastEditDate="2017-03-10T01:20:23.273" LastActivityDate="2017-03-10T17:14:14.970" Title="Which algorithm can i use for predicting length of stay in coming year based on historical claims data?" Tags="|machine-learning|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17488" PostTypeId="2" ParentId="17485" CreationDate="2017-03-10T06:09:35.267" Score="1" Body="&lt;p&gt;If you want to do prediction using 2011 features, the answer is &lt;strong&gt;yes&lt;/strong&gt;, you can do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as you don't want to use these features, the answer might be &lt;strong&gt;no&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without using 2011 features, your dataset will have only 2 samples(2009 and 2010) under the assumption that every &lt;code&gt;memberID&lt;/code&gt; is different. Prediction from two samples is neither reliable nor feasible.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-10T06:09:35.267" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17489" PostTypeId="2" ParentId="16413" CreationDate="2017-03-10T06:36:18.990" Score="5" Body="&lt;p&gt;Try something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;plt.figure(figsize=(30, 20))  # the size you want&#xA;&#xA;# your code goes here&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-10T06:36:18.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17491" PostTypeId="1" CreationDate="2017-03-10T08:03:58.447" Score="3" ViewCount="477" Body="&lt;p&gt;I really confused about how GRU computation really works. I am not really good at math btw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read the original source of GRU equations from Cho, et al. (2014), and also a Colah's blog post on the same topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And somehow, GRU equations at Colah's blog post &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding LSTM Networks&lt;/a&gt; are different than equations in the Cho's original paper &lt;a href=&quot;https://arxiv.org/pdf/1406.1078v3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the equations from Colah's blog:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z_t = \sigma(W_z \cdot [h_{t-1},x_t])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r_t = \sigma(W_r \cdot [h_{t-1},x_t])$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h_t = (1-z_t) \ast h_{t-1} + z_t \ast \tilde{h}_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\tilde h_t = tanh(W \cdot [r_t \ast h_{t-1}, x_t]_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the equations from the original paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z_j = \sigma([\mathbf W_z \mathbf x]_j + [\mathbf U_z \mathbf h_{(t-1)}]_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r_j = \sigma([\mathbf W_r \mathbf x]_j + [\mathbf U_r \mathbf h_{(t-1)}]_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h_j^{(t)} = z_jh_j^{(t-1)} + (1-z_j) \tilde h_j^{(t)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\tilde h_j^{(t)} = \phi([\mathbf W \mathbf x]_j + [\mathbf U(\mathbf r \odot h_{(t-1)})]_j)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain if Colah's equations are the same as the equations in the paper version?&lt;/p&gt;&#xA;" OwnerUserId="26108" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:42.270" LastActivityDate="2017-05-29T04:59:42.270" Title="Explaining GRU equations from two sources" Tags="|neural-network|deep-learning|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17492" PostTypeId="2" ParentId="17491" CreationDate="2017-03-10T10:10:55.360" Score="2" Body="&lt;p&gt;The equations are &lt;strong&gt;&lt;em&gt;almost&lt;/em&gt;&lt;/strong&gt; the same. First of all, they are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referred to as $W$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example you could write these two equivalent forms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j = \sigma(W_z[x,h_{t-1}]_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and if we use the same order as Colah's equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation.&#xA;If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)&lt;/p&gt;&#xA;" OwnerUserId="28129" LastEditorUserId="28129" LastEditDate="2017-03-10T11:30:53.677" LastActivityDate="2017-03-10T11:30:53.677" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17494" PostTypeId="1" AcceptedAnswerId="17495" CreationDate="2017-03-10T11:28:08.157" Score="1" ViewCount="333" Body="&lt;p&gt;I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, since &quot;less data&quot; is very subjective, is there a statistical test you can perform or even a rule of thumb on which split to follow?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current problem is a classification problem with 145 observations, 22 features, 2 labels (18-True, 127-False), but I'm interested on the general approach.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a8P9X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a8P9X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29799" LastActivityDate="2017-03-10T11:51:36.710" Title="Selecting dataset splitting strategy" Tags="|dataset|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17495" PostTypeId="2" ParentId="17494" CreationDate="2017-03-10T11:51:36.710" Score="0" Body="&lt;p&gt;As usual in these cases, there is no magic wand to determine which splitting method to use. It all depends on your specific data. Is your collected data redundant enough so that k-fold cross validation is not necessary? Does your data capture most of the input space?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, taking a look at your numbers, I'd say that the number of observations you have (145) is not likely to be large enough to capture all the potential variability in the input space, giving the fact that you have a high number of features (22). This conclusion, of course, depends on the type of the features (are they binary? categorical? numerical?), and whether all these features are actually necessary to make a prediction (are there redundant/correlated features? features that do not give any information about the output variables?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, and not knowing more about the data, I'd go for a cross-validation splitting scheme. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-03-10T11:51:36.710" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17496" PostTypeId="2" ParentId="17485" CreationDate="2017-03-10T12:06:59.907" Score="0" Body="&lt;p&gt;If I'm getting it right,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, since the value you want to predict (days of staying) is a continuous variable, this is a typical &lt;strong&gt;regression&lt;/strong&gt; problem, &lt;strong&gt;not classification&lt;/strong&gt; and if you follow &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn's flowchart&lt;/a&gt;, you will find the right estimator.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But most important, you have to follow the basic steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Understanding the Dataset&lt;/strong&gt;  &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data Exploration  &lt;/li&gt;&#xA;&lt;li&gt;Outlier Investigation  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimize Feature Selection/Engineering&lt;/strong&gt;  &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create new features (if needed - depends on the existing features)  &lt;/li&gt;&#xA;&lt;li&gt;Select features  &lt;/li&gt;&#xA;&lt;li&gt;Properly scale features  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Pick and Tune an Algorithm&lt;/strong&gt;  &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pick an algorithm  &lt;/li&gt;&#xA;&lt;li&gt;Tune the algorithm  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Validate and Evaluate&lt;/strong&gt;  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After you have completed the above steps you will have a model able to give it an array of features (e.g. [4, 5, 24, 1, 5, 0, 0, 2]) and it will predict the days of staying. (I skipped the year cause I don't think it will help your prediction. You will find it out on step 2).&lt;/p&gt;&#xA;" OwnerUserId="29799" LastActivityDate="2017-03-10T12:06:59.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17497" PostTypeId="2" ParentId="17447" CreationDate="2017-03-10T12:33:13.147" Score="0" Body="&lt;p&gt;No, it isn't that way. From my personal experience, I have built custom word2vec's to some datasets and observed that at times, shorter vectors did better than longer ones. I think you can try shorters length vectors for smaller datasets. I cannot tell any metric in specific for comparison of dataset size and word2vec length though.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-03-10T12:33:13.147" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17498" PostTypeId="2" ParentId="17447" CreationDate="2017-03-10T14:34:24.593" Score="0" Body="&lt;p&gt;50-300 is the range of dimensions I would use. If you go through the word representation papers, most uses 200 or 300 dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Skip thought uses 500 I think but it is a representation for sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh and to answer your question, not always. It obviously depends on the size of your training data but also your task at hand.&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2017-03-10T14:34:24.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17499" PostTypeId="2" ParentId="17485" CreationDate="2017-03-10T17:14:14.970" Score="0" Body="&lt;p&gt;If you want to do this as a truly predictive model, you are going to need &lt;em&gt;a lot&lt;/em&gt; more information. You are going to need various factors that could have predicted the hospital stay in previous years (gender, age, weight, etc) and then create a cross-validated model for predicting the nights in a hospital for each person. From there you will be able to predict future years. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you don't have any more information, you might be able to use the Poisson distribution. However, I don't know if the clustered hospital nights (technically they are not singular) would fit the criteria for that distribution.&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-03-10T17:14:14.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17500" PostTypeId="1" CreationDate="2017-03-10T17:16:15.423" Score="1" ViewCount="2140" Body="&lt;p&gt;I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson natively within SQL? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-06-10T12:07:59.310" Title="Does SQL Server support the Poisson distribution?" Tags="|predictive-modeling|databases|sql|programming|distribution|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17501" PostTypeId="1" AcceptedAnswerId="17502" CreationDate="2017-03-10T18:14:17.707" Score="1" ViewCount="4740" Body="&lt;p&gt;Karpathy's' LSTM batch network &lt;a href=&quot;https://gist.github.com/karpathy/587454dc0146a6ae21fc&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM batch network&lt;/a&gt; operates with batches&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def checkSequentialMatchesBatch():&#xA;    &quot;&quot;&quot; check LSTM I/O forward/backward interactions &quot;&quot;&quot;&#xA;    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size&#xA;    input_size = 10&#xA;    WLSTM = LSTM.init(input_size, d) # input size, hidden size&#xA;    X = np.random.randn(n,b,input_size)&#xA;    #...&#xA;&#xA;def checkBatchGradient():&#xA;    &quot;&quot;&quot; check that the batch gradient is correct &quot;&quot;&quot;&#xA;    # lets gradient check this beast&#xA;    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size&#xA;    input_size = 10&#xA;    WLSTM = LSTM.init(input_size, d) # input size, hidden size&#xA;    X = np.random.randn(n,b,input_size)&#xA;    #...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does batch applied for? I'm only familiar with feeding one-hot word representation vector and can't understand LSTM learning process with batches. Please explain in terms of text processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="29714" LastActivityDate="2017-04-21T21:13:46.893" Title="What is a batch in machine learning?" Tags="|python|neural-network|deep-learning|rnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17502" PostTypeId="2" ParentId="17501" CreationDate="2017-03-10T18:21:14.920" Score="0" Body="&lt;p&gt;A batch is a grouping of instances from your dataset. For example a batch of 100 text samples that will be fed to train your model together. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-10T18:21:14.920" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17503" PostTypeId="1" CreationDate="2017-03-10T19:34:55.453" Score="1" ViewCount="60" Body="&lt;p&gt;I have an evenly-spaced timeseries $a_1, a_2, a_3,..., a_n$ and a function $f_k=\mathbf 1(a_k-a_{k-5})$, where $\mathbf 1$ is the unit step function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict $f_k$ using $a_1, a_2, a_3,..., a_{k-5}$, that is, I want to predict if $a_k&amp;gt;a_{k-5}$, given the values $a_1..., a_{k-5}$ .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it better to construct and train an ANN to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;directly predict values of $f_k$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;OR&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;predict values of $a_k$, and then apply $f_k$ to the result?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29844" LastActivityDate="2017-03-10T19:34:55.453" Title="Predict arguments of indicator function or the value the indicator function itself" Tags="|neural-network|training|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17504" PostTypeId="1" CreationDate="2017-03-10T20:20:39.107" Score="3" ViewCount="18121" Body="&lt;p&gt;I built a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10,000 observations of 39 variables takes almost 5 minutes for one iteration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do neural networks typically take many hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed. &lt;/p&gt;&#xA;" OwnerUserId="28231" LastEditorUserId="54855" LastEditDate="2018-09-20T14:56:47.667" LastActivityDate="2018-09-20T14:56:47.667" Title="Neural Network training time?" Tags="|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17505" PostTypeId="2" ParentId="17504" CreationDate="2017-03-10T20:31:53.643" Score="2" Body="&lt;p&gt;This is quite standard for the training time. It depends on how much optimization you did on your code. The speed of your processing unit, it's often better to use a GPU as opposed to a CPU. GPUs do mathematical operations much faster. Also, you should use parallel computing when you can, in the case of NN you definitely can.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training a machine learning algorithm only needs to be done once. Let it run all night and then you will be ready to do some pretty good predictions.  &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-10T20:31:53.643" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17506" PostTypeId="1" CreationDate="2017-03-10T22:16:02.370" Score="3" ViewCount="1349" Body="&lt;p&gt;I have two probability distribution curves, a Gamma and a standarized Normal, that I need to compare:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(ggplot2)&#xA;&#xA;pgammaX &amp;lt;- function(x) pgamma(x, shape = 64.57849, scale = 0.08854802)&#xA;&#xA;f &amp;lt;- ggplot(data.frame(x=c(-4, 9)), aes(x)) + stat_function(fun=pgammaX)&#xA;f + stat_function(fun = pnorm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is like this&#xA;&lt;a href=&quot;https://i.stack.imgur.com/hCJbg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hCJbg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;However I need to have the two curves separated by means of the faceting mechanism provided by ggplot2, sharing the Y axis, in a way like shown below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/skQyL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/skQyL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to do the faceting if the depicted graphics come from data (i.e., from a data.frame), but I don't understand how to do it in a case like this, when the graphics are generated &lt;em&gt;on line&lt;/em&gt; by functions.&#xA;Do you have any idea on this?&lt;/p&gt;&#xA;" OwnerUserId="29841" LastActivityDate="2017-07-30T06:08:09.753" Title="How to plot two distribution curves in a faceted way in R/ggplot2?" Tags="|r|ggplot2|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17507" PostTypeId="1" AcceptedAnswerId="17520" CreationDate="2017-03-10T22:28:54.713" Score="1" ViewCount="561" Body="&lt;p&gt;I have the following labeled cluster, which is what an ideal clustering algorithm would generate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S4jOR.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S4jOR.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have applied a basic K-Means clustering algorithm to the data, and the outcome is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S4jOR.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/52DvK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recognize that this is a tough problem to properly cluster because some of the classes are so similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I was wondering if there are any alternative algorithms that may help me improve the separability of the clusters, and improve how well my unsupervised clustering algorithm would work on new data?  &lt;/p&gt;&#xA;" OwnerUserId="12202" LastEditorUserId="12202" LastEditDate="2017-03-10T22:34:32.627" LastActivityDate="2017-03-22T02:18:52.117" Title="Alternative methods for improved clustering separation?" Tags="|clustering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17508" PostTypeId="1" AcceptedAnswerId="30753" CreationDate="2017-03-10T22:47:04.963" Score="3" ViewCount="164" Body="&lt;p&gt;I have been looking into interpreting feature coefficient in a bounded capacity (0,1) as probability scores. So for example, if one has the following linear model for classification(&lt;em&gt;Logistic regression&lt;/em&gt;),&#xA;then inverse logit should return one linear probability scores right as it is the inverse logs of odd ratio?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y&amp;lt;- c(1:3, 7, 6)&#xA;x1&amp;lt;- 1:5&#xA;x2&amp;lt;- c(20:25)&#xA;temp&amp;lt;-coef(lm(y ~ x1 + x2))&#xA;library('arm')&#xA;invlogit(temp[1]+temp[2]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in-case of linear regression for continuous value prediction, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coeff1/sum(coeff)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should return one probability scores right ? &#xA;Did I miss something ?&lt;/p&gt;&#xA;" OwnerUserId="5179" LastActivityDate="2018-04-24T12:41:18.677" Title="Converting feature coefficients to probabilities scores for linear models" Tags="|python|r|linear-regression|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17510" PostTypeId="1" AcceptedAnswerId="17512" CreationDate="2017-03-10T23:25:09.697" Score="7" ViewCount="4042" Body="&lt;p&gt;I have been playing around with a lot of different machine learning models (clustering, neural nets, etc...), but I am sort of stuck on understanding what happens after you finish building the model in Python or Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say that I trained a basic neural network model for a binary classification problem.  How does one deploy that to colleagues, for example, so that they can load in the dataset to spit out a prediction?  I have trained a model, but what happens to that model, now?  How do I &quot;save&quot; that model that I just trained in Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see a lot of tutorials on how to pre-process data, train the model, spit out the statistics and predictions; but, &lt;strong&gt;what comes next&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously Facebook, Google, and anyone else heavily involved in machine learning / AI applications are creating a framework to use their models.  But, is there software that allows you to pull in data and then apply it to your Python code?  Is this what Weka, TensorFlow, and these other packages do?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2018-06-26T18:03:15.077" Title="How does one deploy a model, after building it in Python or Matlab?" Tags="|machine-learning|predictive-modeling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17512" PostTypeId="2" ParentId="17510" CreationDate="2017-03-11T02:37:28.923" Score="3" Body="&lt;p&gt;what comes after training a model is persisting (in layman terms, saving) a model, if you want to save the parameters for later deployment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All companies that do serious machine learning spend many hours (sometimes even days or weeks) training the models. Obviously, when you need rapid predictions in a deployment mode, you use a pre-trained model to generate said prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In python, for example, there are a number of way to persist models. The most common libraries used are pickle and cpickle (i/o much faster I believe as the core is in C).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link to model persistence in sklearn:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/model_persistence.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/model_persistence.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9197" LastActivityDate="2017-03-11T02:37:28.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17513" PostTypeId="1" AcceptedAnswerId="17530" CreationDate="2017-03-11T11:51:50.120" Score="8" ViewCount="38694" Body="&lt;p&gt;I'm trying to extract features of set of images. I'm using CNN from this &lt;a href=&quot;https://pythonprogramming.net/cnn-tensorflow-convolutional-nerual-network-machine-learning-tutorial/&quot; rel=&quot;noreferrer&quot;&gt;site&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please tell me how to do feature extraction of images using CNN? I looked for various places. But nowhere it's clearly mention the feature extraction part.&lt;/p&gt;&#xA;" OwnerUserId="10810" LastActivityDate="2018-10-10T05:41:11.677" Title="Extracting Features Using TensorFlow CNN" Tags="|python|neural-network|feature-extraction|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17514" PostTypeId="1" AcceptedAnswerId="19411" CreationDate="2017-03-11T12:43:56.790" Score="5" ViewCount="1376" Body="&lt;p&gt;I'm training a character based RNN model for text prediction and want to compare it to similar models. Since most literature uses word based perplexity as a performance metric, what would be the &quot;proper&quot; way to calculate word based perplexity from a character based model?&lt;/p&gt;&#xA;" OwnerUserId="29864" LastEditorUserId="29864" LastEditDate="2017-03-20T19:47:11.523" LastActivityDate="2017-08-06T18:51:39.147" Title="Word based perplexity from char-rnn model" Tags="|nlp|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17515" PostTypeId="1" CreationDate="2017-03-11T13:08:57.893" Score="2" ViewCount="1416" Body="&lt;p&gt;I need a method / algorithm for identifying which adjective describes which noun in a sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;The product itself is good however this company has a terrible service&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As an output i'd like to get something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[product, good]&#xA;[service, terrible]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could you please point me to some resources / names / algorithms or libraries (no matter which programming language) that might help with solving this?&lt;/p&gt;&#xA;" OwnerUserId="29862" LastEditorUserId="29862" LastEditDate="2017-03-11T23:04:13.033" LastActivityDate="2017-03-11T23:04:13.033" Title="NLP - identifying which adjective describes which noun in a sentence" Tags="|nlp|text-mining|" AnswerCount="0" CommentCount="3" ClosedDate="2017-03-15T10:30:11.630" ContentLicense="CC BY-SA 3.0" />
  <row Id="17516" PostTypeId="1" AcceptedAnswerId="17517" CreationDate="2017-03-11T13:42:10.793" Score="27" ViewCount="38365" Body="&lt;p&gt;I am newbie on machine learning and keras and now working a multi-class image classification problem using keras. The input is tagged image. After some pre-processing, the training data is represented in Python list as:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[[&amp;quot;dog&amp;quot;, &amp;quot;path/to/dog/imageX.jpg&amp;quot;],[&amp;quot;cat&amp;quot;, &amp;quot;path/to/cat/imageX.jpg&amp;quot;], &#xA; [&amp;quot;bird&amp;quot;, &amp;quot;path/to/cat/imageX.jpg&amp;quot;]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;the &amp;quot;dog&amp;quot;, &amp;quot;cat&amp;quot;, and &amp;quot;bird&amp;quot; are the class labels. I think one-hot encoding should be used for this problem but I am not very clear on how to deal it with these string labels. I've tried sklearn's &lt;code&gt;LabelEncoder()&lt;/code&gt; in this way:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;encoder = LabelEncoder()&#xA;trafomed_label = encoder.fit_transform([&amp;quot;dog&amp;quot;, &amp;quot;cat&amp;quot;, &amp;quot;bird&amp;quot;])&#xA;print(trafomed_label)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;And the output is [2 1 0], which is different that my expectation output of somthing like [[1,0,0],[0,1,0],[0,0,1]]. It can be done with some coding, but I'd like to know if there is some &amp;quot;standard&amp;quot; or &amp;quot;traditional&amp;quot; way to deal with it?&lt;/p&gt;&#xA;" OwnerUserId="29865" LastEditorUserId="29169" LastEditDate="2021-02-09T04:40:06.540" LastActivityDate="2021-02-09T15:58:10.563" Title="How to deal with string labels in multi-class classification with keras?" Tags="|machine-learning|scikit-learn|tensorflow|keras|encoding|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17517" PostTypeId="2" ParentId="17516" CreationDate="2017-03-11T14:23:41.533" Score="19" Body="&lt;p&gt;Sklearn's &lt;code&gt;LabelEncoder&lt;/code&gt; module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly generate one-hot-encodings in other code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want one-hot encoding, you can use &lt;code&gt;LabelBinarizer&lt;/code&gt; instead. This works very similarly:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn.preprocessing import LabelBinarizer&#xA; encoder = LabelBinarizer()&#xA; transfomed_label = encoder.fit_transform([&quot;dog&quot;, &quot;cat&quot;, &quot;bird&quot;])&#xA; print(transfomed_label)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[0 0 1]&#xA; [0 1 0]&#xA; [1 0 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-03-11T14:23:41.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17518" PostTypeId="1" CreationDate="2017-03-11T16:53:39.577" Score="1" ViewCount="196" Body="&lt;p&gt;The forest coverType Dataset Contains the following attributes distributed in many boolean features.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Wilderness_Area (4 binary columns) / qualitative / 0 (absence) or 1 (presence) / Wilderness area designation&#xA;Soil_Type (40 binary columns) / qualitative / 0 (absence) or 1 (presence) / Soil Type designation &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Training the SVC classifier with RBF kernel without attributes re-engineering takes too much time. Can we merge all 40 binary columns for form a Soil_Type Attribute representing those columns. What are the pros and cons of this approach ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, do we have any direct approach to achieve this in weka or sklearn ?&lt;/p&gt;&#xA;" OwnerUserId="29399" LastActivityDate="2017-03-14T07:32:52.520" Title="How To Merge Features in the Dataset Forest Cover Type Classification Problem?" Tags="|classification|feature-selection|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17520" PostTypeId="2" ParentId="17507" CreationDate="2017-03-11T18:56:53.663" Score="3" Body="&lt;p&gt;Your data doesn't appear to be easily separable. In general, one could apply some kind of transformation that pulls apart the distributions for each class. Having labels available makes it possible, in principle, to learn such a transformation (as @Emre metnioned in the comments). But, there are a couple issues with your particular data set. 1) You don't appear to have many data points (unless you've only plotted a small subset). This would limit you to very simple transformations (otherwise you'd probably get severe overfitting). 2) The points are simply overlapping. A transformation can only work based on its inputs and, if the coordinates are indistinguishable, there's nothing that can be done. In the best case, you might be able to pull the lower left turquoise cluster and the yellow points further from the main mass, but the rest of the points are pretty much intermingled. Any transformation that could manage to separate them in the training data would be very complicated, and probably just reflect sample noise (i.e. it would probably be completely overfit, and not generalize to new data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ideal thing would be to find/measure additional (relevant) variables. In this case, the classes may become separable in the higher dimensional space. For example, imagine additing a third axis, where the red points become 'lifted' above the blue points.&lt;/p&gt;&#xA;" OwnerUserId="19014" LastActivityDate="2017-03-11T18:56:53.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17522" PostTypeId="1" AcceptedAnswerId="17529" CreationDate="2017-03-11T20:43:33.303" Score="2" ViewCount="9480" Body="&lt;p&gt;I'm trying to retrain the final layer of a pretrained model with a new image dataset using &lt;strong&gt;TensorFlow-Slim&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I want to fine-tuning inception-v3 on flowers dataset. Inception_v3 was trained on ImageNet with 1000 class labels, but the flowers dataset only have 5 classes. Since the dataset is quite small we will only train the new layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example at &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/slim#fine-tuning-a-model-from-an-existing-checkpoint&quot; rel=&quot;nofollow noreferrer&quot;&gt;official tf github page&lt;/a&gt; shows how to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$ DATASET_DIR=/tmp/flowers&#xA;$ TRAIN_DIR=/tmp/flowers-models/inception_v3&#xA;$ CHECKPOINT_PATH=/tmp/my_checkpoints/inception_v3.ckpt&#xA;$ python train_image_classifier.py \&#xA;    --train_dir=${TRAIN_DIR} \&#xA;    --dataset_dir=${DATASET_DIR} \&#xA;    --dataset_name=flowers \&#xA;    --dataset_split_name=train \&#xA;    --model_name=inception_v3 \&#xA;    --checkpoint_path=${CHECKPOINT_PATH} \&#xA;    --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits \&#xA;    --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I couldn't fully understand all the parameters in the above code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_dir&lt;/code&gt; = &lt;strong&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dataset_dir&lt;/code&gt; = new dataset directory location&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dataset_name&lt;/code&gt; = name of the dataset (but why &lt;strong&gt;?&lt;/strong&gt; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dataset_split_name&lt;/code&gt; = &lt;strong&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model_name&lt;/code&gt; = name of the model on which we want to train&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;checkpoint_path&lt;/code&gt; = path to the model checkpoint&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;checkpoint_exclude_scopes&lt;/code&gt; = &lt;strong&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;trainable_scopes&lt;/code&gt; = &lt;strong&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Help me figure out what these parameter means and correct me if I'm wrong about any of the parameter?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I'm aware that we can retrain inception_v3 with a method mentioned at &lt;a href=&quot;https://www.tensorflow.org/tutorials/image_retraining&quot; rel=&quot;nofollow noreferrer&quot;&gt;official tensorflow website&lt;/a&gt; but I want to do the same with tensorflow-slim.&lt;/p&gt;&#xA;" OwnerUserId="29771" LastEditorUserId="29771" LastEditDate="2017-03-12T07:50:16.153" LastActivityDate="2017-04-02T21:38:58.977" Title="Fine-tuning a model from an existing checkpoint with TensorFlow-Slim" Tags="|deep-learning|tensorflow|inception|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17523" PostTypeId="2" ParentId="17507" CreationDate="2017-03-11T21:12:41.747" Score="2" Body="&lt;p&gt;You forgot the most important step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preprocessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the axes. Scale them the &lt;em&gt;same&lt;/em&gt; way, and you will realize that your y axis has zero effect. Your data really looks like this squeezed slice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/S4jOR.jpg&quot; width=&quot;999&quot; height=&quot;10&quot; /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't blindly run clustering. Your work is likely 70% figuring out how to adequately preprocess your data, 10% clustering, and 20% making sense of the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't underestimate how difficult preprocessing is. The way you used k-means, &lt;strong&gt;you assumed 1 gallon of water = 1 second&lt;/strong&gt;. That assumption is probably wrong...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least: since you have labels, why use clustering at all?&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-03-11T21:26:19.293" LastActivityDate="2017-03-11T21:26:19.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17525" PostTypeId="1" CreationDate="2017-03-12T02:16:29.307" Score="3" ViewCount="152" Body="&lt;p&gt;Here is one of Owen Zhang's &lt;a href=&quot;https://www.slideshare.net/OwenZhang2/tips-for-data-science-competitions&quot; rel=&quot;nofollow noreferrer&quot;&gt;slideshares&lt;/a&gt;. At page 12, what does the &lt;strong&gt;time&lt;/strong&gt; mean?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pbSyd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pbSyd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2017-03-12T02:44:06.927" Title="Owen Zhang's slides: what does the &quot;time&quot; mean?" Tags="|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17526" PostTypeId="2" ParentId="17525" CreationDate="2017-03-12T02:44:06.927" Score="2" Body="&lt;p&gt;Here's my understanding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's the explanation of page 10, and &lt;strong&gt;time&lt;/strong&gt; is the same as the &quot;time&quot; in page 10, which means the time axis in real life.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In page 10, it shows that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;hold out data should be out-of-time, we can not do a random shuffle because it will not represent the real situation our model will be faced.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;unless data is extremely small&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As the slide has said before:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When you looked at your validation result for the Nth time, you&#xA;  are training models on it&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case: validation data = hold out data = Public LB (if my understanding is correct)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first figure of page 12 is the normal condition, we shouldn't check our Public LB to tune our model(or we're training model on it). So the answer is no.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second figure is the small data condition, we have to train model using the feedback from Public LB, because of the data limitaion. So the answer is yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third figure shows that Public Data is the same as Private Data, they both represent something under same time period, sharing the same &quot;reality&quot;, thus we can use the feedback from Public LB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The forth figure shows that Public Data is reality, and Private Data is more &quot;realistic&quot; than Public Data. It's the &quot;hold out data should be out-of-time&quot; case, the answer is yes.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-12T02:44:06.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17527" PostTypeId="1" CreationDate="2017-03-12T05:27:00.493" Score="2" ViewCount="40" Body="&lt;p&gt;I have some data for medical diagnosis, consisting of some rules about relationship of diseases and their symptoms, for example disease D1 frequently has symptom S1 or disease D2 rarely has symptom S1. Given some symptoms, I want to somehow calculate most probable diseases based on this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about using a naive bayes model, but using that needs some assumptions about disease prevalence or probability of some symptoms given a disease that are not present in my rules set. Is there an standard way of doing this kind of analysis? For example are fuzzy systems used in this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: I think that in the cases where relation between a disease and a symptom is not explicitly stated as a rule it is an indication of either independence or a very low probability of the symptom given that disease.&lt;/p&gt;&#xA;" OwnerUserId="29878" LastActivityDate="2017-03-12T05:27:00.493" Title="Can inferencing come from incomplete rule sets?" Tags="|naive-bayes-classifier|association-rules|fuzzy-logic|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17529" PostTypeId="2" ParentId="17522" CreationDate="2017-03-12T09:49:46.993" Score="2" Body="&lt;p&gt;You can get all information you need from the &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/slim/train_image_classifier.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;train_dir: Directory where checkpoints and event logs are written to. (default: /tmp/tfmodel/)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;dataset_dir: The directory where the dataset files are stored.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;dataset_name: The name of the dataset to load. (default: imagenet)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;dataset_split_name: The name of the train/test split. (default: train)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;model_name: The name of the architecture to train. (default: inception_v3)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;checkpoint_path: The path to a checkpoint from which to fine-tune.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;checkpoint_exclude_scopes: Comma-separated list of scopes of variables to exclude when restoring from a checkpoint.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;trainable_scopes: Comma-separated list of scopes to filter the set of variables to train. By default, None would train all the variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-12T09:49:46.993" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="17530" PostTypeId="2" ParentId="17513" CreationDate="2017-03-12T10:02:25.087" Score="3" Body="&lt;p&gt;Actually, after you've completed your training, the &lt;strong&gt;weights&lt;/strong&gt; of all these convolution layers are the feature maps you've extracted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may try to visualize these weights as well as activations to get the full feature maps. Here's a &lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;guide&lt;/a&gt; you may refer to.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/issues/12&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/issues/12&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/understanding-cnn/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-12T10:02:25.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17531" PostTypeId="2" ParentId="17500" CreationDate="2017-03-12T10:27:44.220" Score="0" Body="&lt;p&gt;It seems that someone has managed to implement a Poisson sampling in SQL Server. Checkout this link: &lt;a href=&quot;http://bilquist.com/poisson-sampling-in-sql/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://bilquist.com/poisson-sampling-in-sql/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-12T10:27:44.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17532" PostTypeId="1" CreationDate="2017-03-12T10:41:07.603" Score="1" ViewCount="585" Body="&lt;p&gt;Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that I have to compute the similarity of the new document with each document in the index. If the similarity is below a certain threshold, one can consider this document as novel. One common approach - which I want to do - is to use a Vector Space Model and compute the cosine similarity (e.g. by using Apache Lucene).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this approach has two shortcomings: &lt;strong&gt;1)&lt;/strong&gt; it is computationally expensive and &lt;strong&gt;2)&lt;/strong&gt; it does not incorporate the semantics of documents and words respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to overcome these shortcomings, my idea was to either use an LDA topic distribution or named entities to augment the Lucene index and the query (i.e. the document collection and each new document) with semantics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I am completely lost regarding the concrete implementation. I have already trained an LDA topic model using Mallet and I am also able to do Named Entity Recognition on the corpus. But I do not know &lt;strong&gt;how&lt;/strong&gt; to use these topics and named entities in order to realise novelty detection. More specifically, I do not know how to use these features for index and query creation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, is it already sufficient to store all named entities of one document as a separate field in the index, add certain weights (i.e. boost them) and use a MultiFieldQuery? I do not think that this already adds some kind of semantics to the similarity detection. The same applies to LDA topics: is it sufficient to add the topic probability of each term as a Payload and implement a new similarity score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be very happy if you could provide some hints or even code snippets on how to incorporate LDA topics or named entities in Lucene for some kind of novelty detection or semantic similarity measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="29886" LastActivityDate="2017-04-23T14:33:51.513" Title="Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities" Tags="|machine-learning|nlp|topic-model|lda|named-entity-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17533" PostTypeId="1" AcceptedAnswerId="17546" CreationDate="2017-03-12T11:38:30.190" Score="1" ViewCount="1192" Body="&lt;p&gt;I recently moved to python for data analysis and apparently I am stuck on the basics. I am trying to regress the parameters of the following expression: z=20+x+3*y+noise, and I get the right intercept but the x and y parameters are clearly wrong. What am I doing missing? Code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;import statsmodels.formula.api as smf &#xA;&#xA;# generate true values, and noise around them&#xA;np.random.seed(5)&#xA;x = np.arange(1, 101)&#xA;y = np.arange(1, 101)&#xA;z = 20 + x + 3* y + np.random.normal(0, 20, 100)&#xA;&#xA;data = pd.DataFrame({'x':x, 'y':y, 'z': z})&#xA;&#xA;lm = smf.ols(formula='z ~ x + y', data=data).fit()&#xA;&#xA;# print the coefficients&#xA;lm.summary()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returns &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kMaqG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kMaqG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the x and y parameters are both 1.5, instead of being 1 and 3. What's wrong?&lt;/p&gt;&#xA;" OwnerUserId="27768" LastActivityDate="2017-03-13T03:02:32.520" Title="Wrong output multiple linear regression statsmodels" Tags="|python|linear-regression|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17536" PostTypeId="1" AcceptedAnswerId="17537" CreationDate="2017-03-12T15:21:15.230" Score="10" ViewCount="5288" Body="&lt;p&gt;In support vector machines, I understand it would be computationally prohibitive to calculate a basis function at every point in the data set. However, it is possible to find this optimal solution due to the so-called &lt;em&gt;kernel trick&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-kernel-trick&quot; rel=&quot;noreferrer&quot;&gt;Other answers&lt;/a&gt; to this question use advanced math and statistics jargon to answer the question  (I assume) properly, causing it to be inaccessible to general a data science audience.  Could someone post a &quot;big-picture&quot; description  (i.e., not necessarily thorough or technically complete) illustrating what the kernel trick is and how it works? &lt;/p&gt;&#xA;" OwnerUserId="21198" LastActivityDate="2017-03-14T20:55:25.077" Title="Kernel trick explanation" Tags="|svm|efficiency|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17537" PostTypeId="2" ParentId="17536" CreationDate="2017-03-12T16:08:30.597" Score="14" Body="&lt;p&gt;The kernel trick is based on some concepts: you have a dataset, e.g. two classes of 2D data, represented on a cartesian plane. It is not linearly separable, so for example a SVM could not find a line that separates the two classes. Now, what you can do it project this data into an higher dimension space, for example 3D, where it could be divided linearly by a plane. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/UvH8A.png&quot; alt=&quot;&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, a basic concept in ML is the dot product. You often do dot products of the features of a data sample with some weights w, the parameters of your model. Instead of doing explicitly this projection of the data in 3D and then evaluating the dot product, you can find a &lt;strong&gt;kernel function&lt;/strong&gt; that simplifies this job by simply doing the dot product in the projected space for you, without the need to actually compute projections and then the dot product. &#xA;This allows you to find a complex non linear boundary that is able to separate the classes in the dataset. This is a very intuitive explaination. &lt;/p&gt;&#xA;" OwnerUserId="29896" LastActivityDate="2017-03-12T16:08:30.597" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17539" PostTypeId="1" CreationDate="2017-03-12T18:03:54.467" Score="-1" ViewCount="1589" Body="&lt;p&gt;There is a categorical dataset consisting of n instances, m attributes. We are performing categorical clustering into K clusters. What is the &lt;a href=&quot;https://www.techopedia.com/definition/22229/space-complexity&quot; rel=&quot;nofollow noreferrer&quot;&gt;space complexity&lt;/a&gt; for the following classifiers:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Decision Tree classifier&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Support Vector Machine classifier&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Artificial neural network with one hidden layer consisting of 2/3rd neurons of input data&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;KNN classifier?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29900" LastEditorUserId="31513" LastEditDate="2017-08-12T10:14:47.897" LastActivityDate="2017-08-12T10:14:47.897" Title="Space complexity of classification algorithms" Tags="|machine-learning|neural-network|svm|decision-trees|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17540" PostTypeId="1" AcceptedAnswerId="17558" CreationDate="2017-03-12T18:32:25.667" Score="26" ViewCount="90783" Body="&lt;p&gt;I create a &lt;code&gt;corr()&lt;/code&gt; df out of an original df. The &lt;code&gt;corr()&lt;/code&gt; df came out 70 X 70 and it is impossible to visualize the heatmap... &lt;code&gt;sns.heatmap(df)&lt;/code&gt;. If I try to display the &lt;code&gt;corr = df.corr()&lt;/code&gt;, the table doesn't fit the screen and I can see all the correlations. Is it a way to either print the entire &lt;code&gt;df&lt;/code&gt; regardless of its size or to control the size of the heatmap?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/21NBn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/21NBn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29897" LastEditorUserId="31513" LastEditDate="2017-05-18T23:58:00.890" LastActivityDate="2022-07-14T15:23:32.393" Title="make seaborn heatmap bigger" Tags="|visualization|pandas|plotting|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17542" PostTypeId="1" CreationDate="2017-03-12T21:29:31.383" Score="1" ViewCount="196" Body="&lt;p&gt;I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick &lt;strong&gt;log(feature+SHIFT)&lt;/strong&gt; to correct skewness. The procedure I follow to get prediction is just take &lt;strong&gt;exp(predictions)-SHIFT&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now how do I get my predictions for the following cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I have &lt;strong&gt;Normal - dependent variable&lt;/strong&gt; and the &lt;strong&gt;independent variables&lt;/strong&gt; are &lt;strong&gt;skewed&lt;/strong&gt; , I transform only a &lt;strong&gt;few of them&lt;/strong&gt; using same &lt;strong&gt;log(feature+SHIFT)&lt;/strong&gt; where the &lt;strong&gt;shifts&lt;/strong&gt; are &lt;strong&gt;unique&lt;/strong&gt; for every feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Transformations on &lt;strong&gt;both&lt;/strong&gt; dependent and independent variables &lt;strong&gt;with different SHIFT values.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suitable code examples will be highly appreciated. Thanks&lt;/p&gt;&#xA;" OwnerUserId="29644" LastActivityDate="2017-03-12T21:29:31.383" Title="Transformation of Dependent and Independent Variables" Tags="|feature-engineering|preprocessing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17546" PostTypeId="2" ParentId="17533" CreationDate="2017-03-13T03:02:32.520" Score="1" Body="&lt;p&gt;I think that you are seeing what you are seeing because the model sees the relationship of each set of points in your data frame, which are governed by the equation:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; z = 20 + x +3*y + noise &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But all the model sees is the resulting &lt;code&gt;Z&lt;/code&gt;, NOT that equation which you know created &lt;code&gt;z&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it attempts to build a model which considers how Z was accomplished without knowing there was noise, while KNOWING that both x and y were in this equation because you explicitly told it the are in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this data. (at least this is what I got without a seed, when I ran your data..so it is close with probably different Z due to different noise) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x   y   z&#xA;1   1   32.824550&#xA;2   2   21.382597&#xA;3   3   80.615424&#xA;4   4   30.958157&#xA;5   5   42.192197&#xA;6   6   75.649622&#xA;7   7   29.815352&#xA;8   8   40.167267&#xA;9   9   59.752065&#xA;10  10  53.402601&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; are the same for each point  and because your formula has &lt;code&gt;x + 3*y + noise&lt;/code&gt; ,  &lt;code&gt;z&lt;/code&gt; is also equal  to &lt;code&gt;4*x+ noise&lt;/code&gt; or &lt;code&gt;4*y+ noise&lt;/code&gt; or &lt;code&gt;2*x +2*y+ noise&lt;/code&gt; for each row. There are many ways to get the same change to Z with x &amp;amp; y in exact proportion plus some noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the regressors are being assigned equal influence plus an equal share of the noise. It is the most parsimonious evaluation of &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not what you expect knowing the equation, but it is the answer you should get. If you use a linear model which reduces variables of zero influent, you might even get a 0 or NA value for y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test that &lt;code&gt;lm&lt;/code&gt; is working, simply reverse the order of &lt;code&gt;y&lt;/code&gt; changing the relationship between x &amp;amp; y within that equations and you have a completely different result. I think the one you were looking to find.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = np.arange(100,0,-1)&#xA;&#xA;&#xA;            coef    std err    t    P&amp;gt;|t|   [0.025  0.975]&#xA;Intercept   0.0439  0.000   119.009 0.000   0.043   0.045&#xA;x           1.2184  0.038   32.471  0.000   1.144   1.293&#xA;y           3.2130  0.038   85.629  0.000   3.139   3.288&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You built your model right, but in this case the data set was not created in a way to test for what you hoped to see...but it was not wrong.&lt;/p&gt;&#xA;" OwnerUserId="29905" LastActivityDate="2017-03-13T03:02:32.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17547" PostTypeId="2" ParentId="14530" CreationDate="2017-03-13T08:54:03.533" Score="1" Body="&lt;p&gt;The short answer is: nothing. The &lt;code&gt;numDecimalPlaces&lt;/code&gt; option, like &lt;code&gt;debug&lt;/code&gt; and &lt;code&gt;doNotCheckCapabilities&lt;/code&gt;, is part of the base class that all classifiers in WEKA inherit from. However, it is up to the implementation of the actual classifier to use this value to change the way the model is printed. There are no mentions of &lt;code&gt;numDecimalPlaces&lt;/code&gt; anywhere in the source code for J48, which suggests to me that it does nothing in this case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like the only classifier in WEKA that does anything with the value entered into &lt;code&gt;numDecimalPlaces&lt;/code&gt; is &lt;code&gt;LinearRegression&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-03-13T08:54:03.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17548" PostTypeId="1" AcceptedAnswerId="17549" CreationDate="2017-03-13T10:02:26.910" Score="6" ViewCount="2063" Body="&lt;p&gt;I have discovered that Amazon has a dedicated Deep Learning AMI with TensorFlow, Keras etc. preinstalled (not to mention other prebuilt custom AMIs). I tried out this with a typical job on several GPU-based instances to see the performances. There are five such in the Ireland region (maybe in other regions exist even more, I don't know, this variance is a bit confusing): &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;g2.2xlarge&lt;/li&gt;&#xA;&lt;li&gt;g2.8xlarge&lt;/li&gt;&#xA;&lt;li&gt;p2.xlarge&lt;/li&gt;&#xA;&lt;li&gt;p2.8xlarge&lt;/li&gt;&#xA;&lt;li&gt;p2.16xlarge&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My first question is, what is the difference between the two groups (g-something and p-something)? Both group mentions &quot;GPUs along with CPU&quot;, but no further clue for Deep Learning usability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second problem is that I have been running my job on g2.2 and g2.8 as well, and while the task processing toke quite long time to run, the workload of the GPUs was relatively low (20-40%). Why didn't the framework increase the workload if there is spare processor capacity? Is is necessary/possible to paramter/set anything to optimize the work?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-03-13T11:34:50.870" Title="Which Amazon EC2 instance for Deep Learning tasks?" Tags="|deep-learning|amazon-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17549" PostTypeId="2" ParentId="17548" CreationDate="2017-03-13T11:34:50.870" Score="6" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Pe9JX.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Pe9JX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/DAuW6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DAuW6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the differences and use cases are well pointed here. As far the workload, there are features which help you optimise it. According to the official &lt;a href=&quot;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;, you can try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For persistency, &#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sudo nvidia-smi -pm 1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;Disabling the autoboost feature&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sudo nvidia-smi --auto-boost-default=0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;Set all GPU clock speeds to their maximum frequency.&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sudo nvidia-smi -ac 2505,875&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-03-13T11:34:50.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17550" PostTypeId="1" AcceptedAnswerId="17563" CreationDate="2017-03-13T12:22:43.720" Score="3" ViewCount="1960" Body="&lt;p&gt;I'm having issues with my CNN, using Keras with Theano backend.&#xA;Basically, I need to classify 340x340 grayscale images into 6 categories. The problem is my CNN gives too &quot;hard&quot; probabilities, for instance it will rarely give predictions with some uncertainty, and always tries to push for a 90%+ for one class. The problem is that for my coursework, the penalty used is very harsh for complete miss classification, and uncertainty is much preferred. ( so having a prediction like [0.6, 0.3, 0.2, ...] is much better than having [0.9,0.03,0.02,..].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure why this is happening. My dataset consists of 2400 images,&#xA;which are from different CCTV, and task is about recognising possible objects. Only 800 of the samples are actually from the data, the other 1600 have been generated through data augmentation. &#xA;Note that it is therefore extremely likely a that some pictures are either identical, or extremely similar (e.g. the same scene, one second later) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;#1 &#xA;# Few filter to take big stuff out&#xA;# Also, first layer is not conv so that I can reuse that layer separately&#xA;model.add(Dropout(0.1, input_shape=(1,340,340)))&#xA;model.add(Convolution2D(64, 4, 4, border_mode='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;))   &#xA;&#xA;#2&#xA;model.add(Dropout(0.1))&#xA;model.add(Convolution2D(128, 4, 4, border_mode='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;))   &#xA;&#xA;#3&#xA;model.add(Dropout(0.1))&#xA;model.add(Convolution2D(256, 4, 4, border_mode='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;))   &#xA;&#xA;#4&#xA;model.add(Flatten())&#xA;model.add(Dense(1024))&#xA;model.add(Activation('relu'))&#xA;#5&#xA;model.add(Dense(512))&#xA;model.add(Activation('relu'))&#xA;#6&#xA;model.add(Dense(6))&#xA;model.add(Activation('softmax'))&#xA;&#xA;opt = SGD(lr=0.001)&#xA;model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])&#xA;&#xA;print &quot;Training..&quot;&#xA;&#xA;&#xA;filepath = &quot;log/weights-improvement-{epoch:02d}---{val_acc:.2f}.hdf5&quot;&#xA;checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')&#xA;callbacks_list = [checkpoint] &#xA;model.fit(X_t, y_t, validation_split=0.1, nb_epoch=500, batch_size=32, callbacks=callbacks_list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do you suggest I fix this? &#xA;Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="29914" LastEditorUserId="29914" LastEditDate="2017-03-13T16:30:13.537" LastActivityDate="2017-03-13T20:27:29.780" Title="CNN for classification giving extreme result probabilities" Tags="|python|neural-network|keras|theano|convolution|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17551" PostTypeId="2" ParentId="5706" CreationDate="2017-03-13T13:12:59.290" Score="9" Body="&lt;p&gt;To be more specific in language, while the local gradient of ReLU (which is $1$) multiply the gradient that flow-back because of back-propagation, the result of the updated gradient could be a large negative number (if the gradient that flow-back is a large negative number). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such large negative updated gradient produce a large negative $w_i$ when learning rate is relatively big, hence will repress updates that going to happen in this neuron, since is almost impossible to put up a large positive number to offset the large negative number brought by that &quot;broken&quot; $w_i$.&lt;/p&gt;&#xA;" OwnerUserId="29916" LastActivityDate="2017-03-13T13:12:59.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17552" PostTypeId="2" ParentId="406" CreationDate="2017-03-13T13:13:47.390" Score="0" Body="&lt;p&gt;&lt;em&gt;Bringing this thread back to life, as this could be useful to others landing here with similar questions.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Facebook&lt;/strong&gt; recently released and open-sourced one of their internal forecasting tool called &lt;strong&gt;&lt;a href=&quot;https://facebookincubator.github.io/prophet/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prophet&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;It is available as both R &amp;amp; Python packages, and proves to be an interesting solution for someone with little Machine Learning background. However, some additional ML knowledge allows to tune and optimize the produced models.&lt;/p&gt;&#xA;&lt;p&gt;I recommend giving Prophet a try as a first step. The quick win on this solution is the ease and speed of the model building and testing: you can literally get a descent projection in a matter of minutes.&#xA;It behaves very well on time series, catching the seasonality of the data at hand &amp;quot;naturally&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;Under the hood, it's similar to a generalized additive model (GAM) - more details on the dedicated paper: &lt;a href=&quot;https://facebookincubator.github.io/prophet/static/prophet_paper_20170113.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://facebookincubator.github.io/prophet/static/prophet_paper_20170113.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29915" LastEditorUserId="29169" LastEditDate="2021-03-11T19:03:30.347" LastActivityDate="2021-03-11T19:03:30.347" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17553" PostTypeId="2" ParentId="17460" CreationDate="2017-03-13T13:26:18.283" Score="0" Body="&lt;p&gt;In general column names in R need to be formatted without spaces for R to recognize them properly. Importing files with spaces, R will drop in a period and remove the space&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This limit is is true for variables as well, but it will not auto-format them for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most accepted methods for naming are:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;stackOverflow&lt;/code&gt; camel case  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;stack_overflow&lt;/code&gt; with the underscore and a combination of lower and upper as you see fit  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;stack.overflow&lt;/code&gt; with period (which many dislike because it is often easy to not see the period)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use these conventions you will avoid the name issue and be able to focus on the underlying models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A caveat, which may or may not matter to you. If you are going to pick a naming style and you work with other languages and pass data around, consider picking a style which works in both or all of them. It is easier for you to have one way, and it makes moving files easier if the names are compatible.&lt;/p&gt;&#xA;" OwnerUserId="29905" LastActivityDate="2017-03-13T13:26:18.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17554" PostTypeId="2" ParentId="17427" CreationDate="2017-03-13T13:49:56.150" Score="1" Body="&lt;p&gt;If you can take a quick look with photoshop and get information on the shade of gray and zoom into the objects and count a pixel pattern. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would be able to build an algorithm around the color 0-255 in 8-bit grayscale and create a simple matrix for each symbol based on the number of pixels you counted.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 110, 0,   0,   0,   0&#xA; 110, 0,   0,   0,   0&#xA; 0,  110, 110, 110,  0&#xA; 0,   0,   110, 110, 0&#xA; 0,   0,   110, 110, 0  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see a shape emerge from the matrix...that is what you train on, but you do so by having the script read pixel by pixel and create a matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be a gray (110) on black (0)  shape. If you only have a few, you could go into photoshop and cut the symbols out at the exact shape and feed the shape into your model and train it to look for that specific color and shape. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might need to build an equation to scale it up or down so that it could account for using the identifier at different sizes or on files with different pixel depth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you could also do this in R or python by reading a file in and localizing that data, but it would be easier to bring in exactly the shapes as separate files. &#xA;You might also need to give the model some leeway as to the darkness like from 100-120 depending on how the images enter the system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they are digital and unmodified straight out of CAD this is likely unnecessary, but if they are scanned or grabbed online, you will not have consistent exact numerical color-depth and need to account for the variation in your algorithm.&lt;/p&gt;&#xA;" OwnerUserId="29905" LastActivityDate="2017-03-13T13:49:56.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17555" PostTypeId="1" CreationDate="2017-03-13T14:03:10.473" Score="2" ViewCount="98" Body="&lt;p&gt;Would a simply Python Script to extract Information from a Webpage (e.g with use of beautiful soup for example) be called a Wrapper (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Wrapper_(data_mining)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Wrapper_(data_mining)&lt;/a&gt;)?&lt;/p&gt;&#xA;" OwnerUserId="29921" LastActivityDate="2017-03-13T17:36:31.170" Title="Is a Python Webscraper a Wrapper?" Tags="|python|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-15T10:31:19.897" ContentLicense="CC BY-SA 3.0" />
  <row Id="17556" PostTypeId="2" ParentId="13249" CreationDate="2017-03-13T14:04:07.647" Score="-1" Body="&lt;p&gt;The GLRT is normally applied to a 1) composite hypothesis &#xA;                                  2) two or more parameter density functions&lt;/p&gt;&#xA;" OwnerUserId="29922" LastActivityDate="2017-03-13T14:04:07.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17557" PostTypeId="1" CreationDate="2017-03-13T14:07:24.480" Score="7" ViewCount="8899" Body="&lt;p&gt;I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the &lt;a href=&quot;https://www.tensorflow.org/tutorials/seq2seq&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow se2seq machine translation tutorial&lt;/a&gt; in Keras). My questions are the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that Embedding layers turn word values in a sentence into fixed-dimension-long representation. But I observe two distinct usage of Embedding layers: one on one hand (like &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;noreferrer&quot;&gt;this tutorial on Keras Blog&lt;/a&gt;) utilizes external pre-trained word2vec vectors via the &lt;code&gt;weights&lt;/code&gt; parameter:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Embedding&#xA;&#xA;embedding_layer = Embedding(len(word_index) + 1,&#xA;  EMBEDDING_DIM,&#xA;  weights=[embedding_matrix],&#xA;  input_length=MAX_SEQUENCE_LENGTH,&#xA;  trainable=False)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while in other cases there is no such an external output but users just leave to the Embedding layer to decide the representation vectors. I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, what is the purpose/effect of the &lt;code&gt;trainable&lt;/code&gt; parameter of the Embedding layer? Am I correct guessing that this set to True let the Embedding layer fine-tune the imported word2vec weights to take the actual training examples into consideration?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, how to instruct Embedding layer to properly encode &quot;metacharacters&quot;? Setting the &lt;code&gt;mask_zero&lt;/code&gt; parameter True it can incorporate padding zeroes but what about UNK (unknown), EOS (End of Sentence)? (By the way, I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally: how could a model predict the translation of a word which is not represented in the training set? Is it tries to approximate it with the &quot;closest&quot; one in the vocabulary?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-02-28T17:51:41.787" Title="How to user Keras's Embedding Layer properly?" Tags="|deep-learning|tensorflow|keras|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17558" PostTypeId="2" ParentId="17540" CreationDate="2017-03-13T14:56:49.033" Score="27" Body="&lt;p&gt;I found out how to increase the size of my plot with the following code...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.subplots(figsize=(20,15))&#xA;sns.heatmap(corr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U5vKH.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U5vKH.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29897" LastActivityDate="2017-03-13T14:56:49.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17559" PostTypeId="1" AcceptedAnswerId="17657" CreationDate="2017-03-13T17:27:14.670" Score="4" ViewCount="1981" Body="&lt;p&gt;The &lt;strong&gt;tensorflow example CIFAR10&lt;/strong&gt; uses &lt;strong&gt;input pipelines&lt;/strong&gt; to load data from the disk to a queue. I would like to implement this for my own models, but I run into an error that I can't fix somehow. My minimal example looks like the following (taken from the tutorials: &lt;a href=&quot;https://www.tensorflow.org/programmers_guide/reading_data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/programmers_guide/reading_data&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def read_sample_of_csv(filename_queue):&#xA;&#xA;   # Define Reader&#xA;   reader = tf.TextLineReader()&#xA;   key,value = reader.read(filename_queue)&#xA;&#xA;   # Decode CSV&#xA;   record_defaults = [[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]]&#xA;   col0, *col = tf.decode_csv(value, record_defaults=record_defaults)&#xA;   features = tf.stack([*col])&#xA;   reference = col0&#xA;&#xA;   # Return&#xA;   return features, reference&#xA;&#xA;&#xA;def input_pipeline(filenames, batch_size, ...):&#xA;   # Reader for samples&#xA;   slipangle_filename_queue = tf.train.string_input_producer(filenames, shuffle=shuffle)&#xA;   features, reference = read_sample_of_csv(slipangle_filename_queue)&#xA;&#xA;   # Batch samples&#xA;   capacity = min_after_dequeue + (num_threads+3) * batch_size&#xA;   feature_batch, reference_batch = tf.train.shuffle_batch([features,reference], batch_size=batch_size)&#xA;&#xA;   # Return&#xA;   return feature_batch, reference_batch&#xA;&#xA;&#xA;with tf.device(MY_DEVICE):&#xA;   feature_batch, ref_batch = input_pipeline(filenames, batch_size, ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is basically what is done in CIFAR10, just with a csv. The script runs with MY_DEVICE = '/cpu:0', but &lt;strong&gt;when I try to run it on the GPU with MY_DEVICE = '/gpu:0' I get the following error&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;InvalidArgumentError (see above for traceback): Cannot assign a device to node 'shuffle_batch/random_shuffle_queue': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.&#xA;Colocation Debug Info:&#xA;Colocation group had the following types and devices: &#xA;QueueEnqueue: CPU &#xA;QueueSize: CPU &#xA;QueueClose: CPU &#xA;QueueDequeueMany: CPU &#xA;RandomShuffleQueue: CPU &#xA;[[Node: shuffle_batch/random_shuffle_queue = RandomShuffleQueue[capacity=1500, component_types=[DT_FLOAT, DT_FLOAT], container=&quot;&quot;, min_after_dequeue=1000, seed=0, seed2=0, shapes=[[9], []], shared_name=&quot;&quot;, _device=&quot;/device:GPU:0&quot;]()]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can I not use input pipelines on GPUs? Isn't this what is done in CIFAR10?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2017-03-16T16:21:42.797" Title="Input Pipeline for Tensorflow on GPU" Tags="|tensorflow|gpu|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17560" PostTypeId="2" ParentId="17555" CreationDate="2017-03-13T17:36:31.170" Score="1" Body="&lt;p&gt;It might be, or it might not. It depends on how it performs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to a wrapper in the more general sense is that it provides for the interoperability of two distinct classes of object which are not inherently able to work together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the specific case you are talking about, simply creating a script to scrape the web is not necessarily a wrapper. Once you format the xml into a recognized format such as JSON, pandas dataframe or  in a SQL schema that your program can operate over, it becomes a wrapper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The wrapper 'repackages' the data to work in your current system's class transparently so that you do not see the underlying nature of the incompatible structure.&lt;/p&gt;&#xA;" OwnerUserId="29905" LastActivityDate="2017-03-13T17:36:31.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17561" PostTypeId="1" CreationDate="2017-03-13T18:38:15.223" Score="2" ViewCount="215" Body="&lt;p&gt;I was wondering if I could get recommendations for Motif-based classification packages for time series data in Python? Currently I  modified a an old python sax package to work with Python 3.5 and feed into scikit learn's tfidf vectorizor to make a 'bag of patterns'. Which seems to work but I don't have results yet. There is also this package in R &lt;a href=&quot;https://cran.r-project.org/web/packages/TSMining/TSMining.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/TSMining/TSMining.pdf&lt;/a&gt;. I appreciate any thoughts or insight. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="29930" LastActivityDate="2017-03-13T18:38:15.223" Title="Recommendations for Motif Based Classification of Time Series with Python" Tags="|python|classification|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17562" PostTypeId="2" ParentId="17539" CreationDate="2017-03-13T19:21:11.663" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;Decistion Tree: $\mathcal{O}(1)$ for constant depth (you might have bad accuracy, though)&lt;/li&gt;&#xA;&lt;li&gt;SVM: $\mathcal{O}(K)$ if you use a linear kernel and one-vs-all&lt;/li&gt;&#xA;&lt;li&gt;ANN: I don't know what &quot;2/3rd neurons of input data&quot; means&lt;/li&gt;&#xA;&lt;li&gt;KNN: $\mathcal{O}(n)$ - you have to store all $n$ samples.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-03-13T19:21:11.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17563" PostTypeId="2" ParentId="17550" CreationDate="2017-03-13T19:50:04.323" Score="2" Body="&lt;p&gt;There are a few different factors involved here. It is difficult to tell, without getting heavily involved, which could be the most important. I will put them in the order I think worth looking at first.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your data is images from CCTV, so you likely have more than one image from each camera. From your results (reasonable training and CV scores, but bad training), it looks like you are over-fitting. But your CV approach is not spotting this. So I think that the test set is likely to be from a different set of cameras to the training set. In order to properly measure CV therefore, you have to split train/cv &lt;em&gt;by camera&lt;/em&gt; - you cannot just use the 0.1 split, because a random split will include images which are correlated with training data and will give you too high an estimate, allowing you to overfit without noticing.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It occurs to me it &lt;em&gt;might&lt;/em&gt; just be your data augmentation causing a problem for you here. If you augment &lt;em&gt;first&lt;/em&gt; then randomly split to train/CV, then your CV set will contain images very similar to training set, and will get too high a score. You can more easily check and fix this than split by camera, so give it a try.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;800 original images is not much to work with. You need to do something about that. Here are some ideas: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Scale down the images. You probably don't need 340x340. Depending on the target object, maybe just 78x78 will do. You can assess this easily enough - scale down and check if you can still differentiate the classes easily by eye.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You don't have enough data to get best quality filters in the lower layers, which will limit the capabilities of the CNN. You might bootstrap from a pre-trained image model. Take a publicly available pre-trained CNN, such as VGG-19, use its weights in convolutional layers as a starting point, put different classifier layers on top and fine-tune your classifier starting from this. This might change the ideal image scale too - you want something that fits the pre-trained CNN.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Augmenting the data, which you have started. You could go a lot further. Take random patches from training examples, possibly flipped horizontally (if this maintains the object class). However, &lt;em&gt;don't&lt;/em&gt; augment data used for cross validation, unless your model used for testing also includes augmentation - e.g. if it takes 8 random augmented variants of the test image and returns an average of predictions, then you can do similar for cross-validation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A 0.1 train/CV split run once on this much data is not going to give you an accurate assessment of the model. You need to run k-fold cross-validation. This is annoying because NNs take a long time to train, but if you want some confidence that you have really found some good parameters, you will need to do this. Remember to split by camera if you can.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The scoring appears to be strongly related to categorical cross-entropy, so you have the right loss function. You should optimise, using cross-validation, to find the model with the lowest loss, not the best accuracy.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-13T20:27:29.780" LastActivityDate="2017-03-13T20:27:29.780" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17565" PostTypeId="1" AcceptedAnswerId="17660" CreationDate="2017-03-13T20:46:57.017" Score="3" ViewCount="873" Body="&lt;p&gt;I am implementing logistic regression in Python with the regularized loss function like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MqJsv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MqJsv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; But the gradient algorithm works bad. Read the bold text first, please! Just paste the code cell by cell&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np, scipy as sp, sklearn as sl&#xA;from scipy import special as ss&#xA;from sklearn.base import ClassifierMixin, BaseEstimator&#xA;from sklearn.datasets import make_classification&#xA;import theano.tensor as T&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is the loss function: (scipy is to &quot;clip&quot; the logarithm's arg near &lt;code&gt;1&lt;/code&gt;&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def lossf(w, X, y, l1, l2):&#xA;     w.resize((w.shape[0],1))&#xA;     y.resize((y.shape[0],1))&#xA;&#xA;     lossf1 = np.sum(ss.log1p(1 + ss.expm1(np.multiply(-y, np.dot(X, w)))))&#xA;     lossf2 = l2 * (np.dot(np.transpose(w), w))&#xA;     lossf3 = l1 * sum(abs(w))&#xA;     lossf = np.float(lossf1 + lossf2 + lossf3)&#xA;     return lossf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is the gradient function:(??PROBLEM HERE?? -see the end)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gradf(w, X, y, l1, l2):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    gradw1 = l2 * 2 * w &#xA;    gradw2 = l1 * np.sign(w)&#xA;    gradw3 = np.multiply(-y,(2 + ss.expm1(np.multiply(-y, np.dot(X, w)))))&#xA;    gradw3 = gradw3 / (2 + (ss.expm1((np.multiply(-y, np.dot(X, w))))))&#xA;    gradw3 = np.sum(np.multiply(gradw3, X), axis=0)&#xA;    gradw3.resize(gradw3.shape[0],1)&#xA;    gradw = gradw1 + gradw2 + gradw3&#xA;    gradw.resize(gradw.shape[0],)&#xA;    return np.transpose(gradw)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here is my LR class:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class LR(ClassifierMixin, BaseEstimator):&#xA;    def __init__(self, lr=0.0001, l1=0.1, l2=0.1, num_iter=100, verbose=0):&#xA;        self.l1 = l1&#xA;        self.l2 = l2&#xA;        self.w = None&#xA;        self.lr = lr&#xA;        self.verbose = verbose&#xA;        self.num_iter = num_iter&#xA;&#xA;def fit(self, X, y):        &#xA;    n, d = X.shape &#xA;    self.w = np.zeros(shape=(d,))&#xA;    for i in range(self.num_iter):&#xA;        g = gradf(self.w, X, y, self.l1, self.l2)&#xA;        g.resize((g.shape[0],1))&#xA;        self.w = self.w - g&#xA;        print &quot;Loss: &quot;, lossf(self.w, X, y, self.l1, self.l2)&#xA;    return self&#xA;&#xA;def predict_proba(self, X):&#xA;    probs = 1/(2 + ss.expm1(np.dot(-X, self.w)))&#xA;    return probs &#xA;&#xA;def predict(self, X):&#xA;    probs = self.predict_proba(X)&#xA;    probs = np.sign(2 * probs - 1)&#xA;    probs.resize((probs.shape[0],))&#xA;    return probs &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here are the tests:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X, y = make_classification(n_features=100, n_samples=100)&#xA;y = 2 * (y - 0.5)&#xA;clf = LR(lr=0.000001, l1=0.1, l2=0.1, num_iter=10, verbose=0)&#xA;clf = clf.fit(X, y)&#xA;yp = clf.predict(X)&#xA;yp.resize((100,1))&#xA;accuracy = int(sum(y == yp))/len(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This doesn't converge. But if I replace my &lt;code&gt;gradw3&lt;/code&gt; with theano:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gradw3 = get_gradw3(w,X,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;where:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w,X,y = T.matrices(&quot;wXy&quot;) &#xA;logloss = T.sum(T.log1p(1 + T.expm1(-y* T.dot(X, w)))) &#xA;get_gradw3 = theano.function([w,X,y],T.grad(logloss,w).reshape(w.shape))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;it converges to 100% accuracy. That means, my gradw3 is implemented wrong, but I can't find a mistake.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="29931" LastEditorDisplayName="user380" LastEditDate="2017-04-23T19:44:58.687" LastActivityDate="2017-04-23T19:44:58.687" Title="Logistic regression GD implementation in Python" Tags="|python|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17567" PostTypeId="1" AcceptedAnswerId="17610" CreationDate="2017-03-14T02:26:04.137" Score="2" ViewCount="1231" Body="&lt;p&gt;How should one structure an input data matrix (containing EEG data) for an RNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, RNNs are presented as language models where you have a one hot vector indicating the presence of a word. So if you input was the sentence &quot;hello how are you&quot;, you would have 4 one hot vectors (I think):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1, 0, 0, 0]&#xA;[0, 1, 0, 0]&#xA;[0, 0, 1, 0]&#xA;[0, 0, 0, 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do these individual vectors get condensed into a single data matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of single channel EEG (with only 1 electrode), 256 samples per second, 1 second long samples, how should this data be structured? Would it be 256 vectors? If so, what does each vector represent/contain? Or should it be 1 vector that is 256 elements long?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, how does this extend to multi channel EEG with, say, 64 electrodes over 256 time samples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would prefer to use the raw EEG data, rather than trying some dimensionality reduction (calculating means, spectrograms etc)&lt;/p&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2017-03-15T12:45:19.570" Title="EEG data layout for RNN" Tags="|machine-learning|neural-network|time-series|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17569" PostTypeId="2" ParentId="17518" CreationDate="2017-03-14T07:32:52.520" Score="1" Body="&lt;p&gt;On Some Analysis I found out the columns were mutually Exclusive and therefore created one column named Soil_Type(0..40 possible values) replacing 40 binary columns and Wilderness_Area(0..4 possible values) replacing 4 binary columns. This did improve performance of the classifiers (Random Forest, Decision Tree ) with respect to time and prediction accuracy as well.&lt;/p&gt;&#xA;" OwnerUserId="29399" LastActivityDate="2017-03-14T07:32:52.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17570" PostTypeId="2" ParentId="761" CreationDate="2017-03-14T08:01:27.870" Score="-1" Body="&lt;p&gt;The k-means algorithm to cluster the locations is a bad idea. Your locations can be spread across the world and the number of clusters cant be predicted by you, not only that if you put the cluster as 1 then the locations will be grouped to 1 single cluster. I am using OPTICS clustering for the same. It worked like a Charm.&lt;/p&gt;&#xA;" OwnerUserId="29949" LastEditorUserId="29949" LastEditDate="2020-05-22T14:08:19.840" LastActivityDate="2020-05-22T14:08:19.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17571" PostTypeId="2" ParentId="17461" CreationDate="2017-03-14T08:22:40.430" Score="0" Body="&lt;p&gt;Another possibility to check the performance is to evaluate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;confusion matrix&lt;/a&gt;. Therein the predicted and actual labels are compared and the classification result for each class is measured&lt;/p&gt;&#xA;" OwnerUserId="29953" LastActivityDate="2017-03-14T08:22:40.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17572" PostTypeId="1" AcceptedAnswerId="17573" CreationDate="2017-03-14T08:31:49.310" Score="1" ViewCount="9750" Body="&lt;p&gt;I've been trying to come up with an intelligent solution to build a Time table scheduling application with the use of Machine learning or Neural networks. What would be the algorithm or approach to build such application. I'm planing to take data from google calendar API and through the system. The system should purpose best time slots to conduct lectures considering the data taken from lecturers and students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found some research documents where they have used some genetic algorithms but can this be done with the help of neural networks?&lt;/p&gt;&#xA;" OwnerUserId="29952" LastEditorUserId="13727" LastEditDate="2017-03-14T10:26:23.113" LastActivityDate="2017-03-14T10:26:23.113" Title="Application of Machine learning or Neural Networks for automatic Time table scheduling" Tags="|machine-learning|neural-network|optimization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17573" PostTypeId="2" ParentId="17572" CreationDate="2017-03-14T10:25:40.703" Score="4" Body="&lt;p&gt;Scheduling problems might be &lt;a href=&quot;https://en.wikipedia.org/wiki/NP-completeness&quot; rel=&quot;nofollow noreferrer&quot;&gt;NP-Complete&lt;/a&gt; problems.&#xA;It is not clear what are the specific details.&#xA;You might get lucky and have specific constraints that are leading to an easy sub problem or just an easy instance.&#xA;However, since many variations like &lt;a href=&quot;http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1255&amp;amp;context=theses&quot; rel=&quot;nofollow noreferrer&quot;&gt;On the Complexity of Scheduling University Courses&lt;/a&gt; (which might be your case), &lt;a href=&quot;https://en.wikipedia.org/wiki/Job_shop_scheduling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Job shop scheduling&lt;/a&gt;,&lt;a href=&quot;https://en.wikipedia.org/wiki/Multiprocessor_scheduling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multiprocessor scheduling&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Open-shop_scheduling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open-shop scheduling&lt;/a&gt; are NP-complete, you are probably in the same situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, it is better to treat such problems like &lt;a href=&quot;https://en.wikipedia.org/wiki/Optimization_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;optimization problems&lt;/a&gt; and not like classification problems. If you'll try to treat the problem like a classification problem you might have severe problems in building a classified dataset that will represent well the time scheduling problem that you would like to solve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are general technique in order to cope with optimization problem. I also found some work related to your case. I'm not familiar with this specific case but it seems that &lt;a href=&quot;http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1255&amp;amp;context=theses&quot; rel=&quot;nofollow noreferrer&quot;&gt;Solving the Course Scheduling Problem Using&#xA;Simulated Annealing&lt;/a&gt; and the work done &lt;a href=&quot;http://www.nottingham.ac.uk/research/groups/asap/?q=error&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; might help you.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-14T10:25:40.703" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17574" PostTypeId="1" AcceptedAnswerId="17576" CreationDate="2017-03-14T11:50:29.153" Score="3" ViewCount="445" Body="&lt;p&gt;The idea is to motivate the SVD for use in a recommender system.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Consider a matrix &lt;span class=&quot;math-container&quot;&gt;$A\in \mathbb{R}^{f\times u}$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$A_{ij}$&lt;/span&gt; caputures how user &lt;span class=&quot;math-container&quot;&gt;$j$&lt;/span&gt; rates film &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; (on a scale from 1-10, some entries may be missing).&lt;/p&gt;&#xA;&lt;p&gt;Considering matrices &lt;span class=&quot;math-container&quot;&gt;$K=AA^T$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$L = A^TA$&lt;/span&gt; what do &lt;span class=&quot;math-container&quot;&gt;$K_{ij}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$L_{ij}$&lt;/span&gt; tell us?&lt;/p&gt;&#xA;&lt;p&gt;What interpretations can you give for matrices &lt;span class=&quot;math-container&quot;&gt;$U$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; in the SVD &lt;span class=&quot;math-container&quot;&gt;$A=U\Sigma V^T$&lt;/span&gt;?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Riiight ... so&lt;/p&gt;&#xA;&lt;p&gt;K = &lt;span class=&quot;math-container&quot;&gt;$AA^T$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$U\Sigma^2U^T$&lt;/span&gt;, L = &lt;span class=&quot;math-container&quot;&gt;$A^TA = V\Sigma^2V^T$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;With &lt;span class=&quot;math-container&quot;&gt;$K_{ij} = \langle A[i,:], A[j,:] \rangle$&lt;/span&gt; being the dot product of all ratings for film &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; with all ratings for film &lt;span class=&quot;math-container&quot;&gt;$j$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Similarly, &lt;span class=&quot;math-container&quot;&gt;$L_{ij} = \langle A[:, i], A[:,j] \rangle$&lt;/span&gt; being the dot product of all ratings from user &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; with all ratings from user &lt;span class=&quot;math-container&quot;&gt;$j$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;That tells us ... what exactly? I expect it to amount to some kind of similarity measure, but beyond that, I have no idea.&lt;/p&gt;&#xA;&lt;p&gt;As for &lt;span class=&quot;math-container&quot;&gt;$U$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; ...  I know the first &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; of them to be Eigenvectors of &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$L$&lt;/span&gt; for some &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt;. I also know&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$u_1,..,u_r$&lt;/span&gt; is an orthonormal basis for the column space&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$v_1,..v_r$&lt;/span&gt; is an orthonormal basis for the row space&lt;/p&gt;&#xA;&lt;p&gt;but that doesn't really tell me anything.&lt;/p&gt;&#xA;&lt;p&gt;Yet have no idea if the notion of &amp;quot;row space&amp;quot; and &amp;quot;column space&amp;quot; even makes sense. What would the &amp;quot;span&amp;quot; of &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt;'s columns / rows even mean?&lt;/p&gt;&#xA;&lt;p&gt;I also have no idea what the nullspace of &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$A^T$&lt;/span&gt; would represent - &lt;em&gt;if&lt;/em&gt; it represents anything at all - nor what&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$u_{r+1},..,u_{m}$&lt;/span&gt; being an orthonormal basis for &lt;span class=&quot;math-container&quot;&gt;$ker(A)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;or &lt;span class=&quot;math-container&quot;&gt;$v_{r+1},..,v_{n}$&lt;/span&gt; being an orthonormal basis for &lt;span class=&quot;math-container&quot;&gt;$ker(A^T)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;implies - again, if anything at all.&lt;/p&gt;&#xA;" OwnerUserId="29957" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-03-14T14:26:32.617" Title="Interpretation of an SVD for recommender systems" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17576" PostTypeId="2" ParentId="17574" CreationDate="2017-03-14T14:26:32.617" Score="4" Body="&lt;p&gt;First of all, note that the dot product between two movies/users is by definition the correlation between them. So your intuition for treating it as a similarity measure is not wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, applying SVD to $A$ is simply a re-representation of it in some other basis. Let's call it &lt;em&gt;feature space&lt;/em&gt;. If we'll take it one step further and keep only $k$ leading vectors of $U$ and $V$, we'll get two matrices: $U^{(k)} \in \mathbb{R}^{f,k}$ and $V^{(k)} \in \mathbb{R}^{k,u}$ (where $f$ and $u$ are the dimensions you supplied in your question). where $U^{(k)}\cdot V^{(k)} \approx A^{(k)}$ is a $k$-rank approximation for $A$ ** (note it has the same dimensions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can of $U^{(k)}$ as a matrix which rows movies, and each movie has some $k$ latent features. Similarly, $V^{(k)}$ will be a &quot;list&quot; (columns) of users and each user is described by some $k$ values (latent features). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you can &quot;split&quot; the scaling matrix $\Sigma^{(k)}$ between $U^{(k)}$ and $V^{(k)}$: $U'^{(k)} = U^{(k)}\Sigma^{(k)0.5}$ and $V'^{(k)} = V^{(k)}\Sigma^{(k)0.5}$, to make up for the lack of scale of $U$ and $V$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This factorization can be elaborated into more interpretable methods such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;** The &lt;em&gt;variational approach to singular values&lt;/em&gt; tell us that this approximation minimizes the difference between $A$ and any other $k$ rank matrix in terms of Frobenius-norm and operator-norm (and possibly other norms I'm not aware of).&lt;/p&gt;&#xA;" OwnerUserId="29960" LastActivityDate="2017-03-14T14:26:32.617" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17578" PostTypeId="1" AcceptedAnswerId="17591" CreationDate="2017-03-14T17:42:18.763" Score="40" ViewCount="75529" Body="&lt;p&gt;Is there any way now to use TensorFlow with Intel GPUs? If yes, please point me in the right direction.&lt;/p&gt;&#xA;&lt;p&gt;If not, please let me know which framework, if any, (Keras, Theano, etc) can I use for my Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller.&lt;/p&gt;&#xA;" OwnerUserId="27616" LastEditorUserId="29169" LastEditDate="2021-02-09T04:23:22.740" LastActivityDate="2023-09-13T00:59:20.543" Title="Using TensorFlow with Intel GPU" Tags="|tensorflow|keras|theano|gpu|" AnswerCount="8" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17580" PostTypeId="1" CreationDate="2017-03-14T18:11:38.933" Score="2" ViewCount="112" Body="&lt;p&gt;I would like to share a question about my research. I have data about networking of a contagious disease. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 60 pigs, and I have the contact time with others pigs of each individual in 28 study day. So the structure of data would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Day         ID     Network_friend (s)     Network_time (s)&#xA;1            1      200                       20000&#xA;1            2      300                       3000&#xA;.&#xA;.&#xA;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Those 60 pigs are in similar age and body condition so there is not too much variety among the sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if there is a way to predict the individuals who will get sick in 2 days to prevent the spread of the disease. Should I use any network analysis for it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate any replies!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="28860" LastEditorUserId="28860" LastEditDate="2017-08-09T13:59:06.017" LastActivityDate="2017-08-09T13:59:06.017" Title="how to use machine learning or network analysis to prevent contagious disease" Tags="|machine-learning|predictive-modeling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17582" PostTypeId="1" AcceptedAnswerId="17586" CreationDate="2017-03-14T18:48:50.970" Score="3" ViewCount="699" Body="&lt;p&gt;I have a relatively small dataset of 30 samples with binary labels (16 positive and 14 negative).  I also have five continuous features for each these samples.  I'm trying to use the support-vector classifier (SVC) for this task.  I tested the performance of different feature combinations and regularization strengths in the classification task using leave-one-out cross-validation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One odd thing that I found is that if I took feature A and used it alone for classification, I might get, say 87% classification accuracy.  If I use feature B in isolation, I might get 60% classification accuracy (i.e., same as majority classifier baseline).  But then combining all the features, I would get only 63% classification accuracy.  This is despite performing a search across a large range of regularization strengths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it matters, I'm using the sklearn SVC implementation, and varying the regularization parameter C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this sort of behavior typical with an SVC classifier?  I'm not too familiar with this support vector algorithms in general.&lt;/p&gt;&#xA;" OwnerUserId="29968" LastEditorUserId="29968" LastEditDate="2017-03-14T19:33:13.967" LastActivityDate="2017-03-14T20:37:31.260" Title="Adding more features in SVC leading to worse performance, even w/ regularization" Tags="|machine-learning|classification|scikit-learn|svm|overfitting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17583" PostTypeId="1" AcceptedAnswerId="17605" CreationDate="2017-03-14T19:00:23.950" Score="1" ViewCount="195" Body="&lt;p&gt;Does anyone know if there is functionality similar to &lt;em&gt;StopWordsRemover&lt;/em&gt; but intended to clean out HTML syntax? e.g. get the text without any html tags after transformation.&lt;/p&gt;&#xA;" OwnerUserId="29534" LastActivityDate="2017-03-15T09:27:15.973" Title="HTML Words Remover?" Tags="|apache-spark|" AnswerCount="1" CommentCount="1" ClosedDate="2017-03-15T10:30:45.320" ContentLicense="CC BY-SA 3.0" />
  <row Id="17584" PostTypeId="2" ParentId="8457" CreationDate="2017-03-14T19:54:20.420" Score="10" Body="&lt;p&gt;&lt;code&gt;numpy.piecewise&lt;/code&gt; can do this.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;piecewise(x, condlist, funclist, *args, **kw)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Evaluate a piecewise-defined function.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Given a set of conditions and corresponding functions, evaluate each&#xA;      function on the input data wherever its condition is true.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;An example is given on SO &lt;a href=&quot;https://stackoverflow.com/questions/29382903/how-to-apply-piecewise-linear-fit-in-python&quot;&gt;here&lt;/a&gt;. For completeness, here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy import optimize&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;&#xA;x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ,11, 12, 13, 14, 15], dtype=float)&#xA;y = np.array([5, 7, 9, 11, 13, 15, 28.92, 42.81, 56.7, 70.59, 84.47, 98.36, 112.25, 126.14, 140.03])&#xA;&#xA;def piecewise_linear(x, x0, y0, k1, k2):&#xA;    return np.piecewise(x, [x &amp;lt; x0, x &amp;gt;= x0], [lambda x:k1*x + y0-k1*x0, lambda x:k2*x + y0-k2*x0])&#xA;&#xA;p , e = optimize.curve_fit(piecewise_linear, x, y)&#xA;xd = np.linspace(0, 15, 100)&#xA;plt.plot(x, y, &quot;o&quot;)&#xA;plt.plot(xd, piecewise_linear(xd, *p))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="378" LastEditorUserId="63886" LastEditDate="2018-12-07T09:53:32.943" LastActivityDate="2018-12-07T09:53:32.943" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17585" PostTypeId="2" ParentId="17578" CreationDate="2017-03-14T20:34:41.250" Score="5" Body="&lt;p&gt;Keras is an abstraction layer for tensorflow/ theano. You need a nvidia card but tensorflow as well as theano can be used with CPU support only. Instructions can be found on their websites.&lt;/p&gt;&#xA;" OwnerUserId="29723" LastEditorUserId="29723" LastEditDate="2017-03-15T07:03:54.957" LastActivityDate="2017-03-15T07:03:54.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17586" PostTypeId="2" ParentId="17582" CreationDate="2017-03-14T20:37:31.260" Score="1" Body="&lt;p&gt;30 samples probably just isn't enough. The more features you want to use, the more samples you'll need - otherwise you'll get huge model instability and overfitting (especially with bad/useless features). With only 30 samples you'll probably get the &quot;best&quot; results with 1 or 2 carefully selected features. Get 100 or 200 samples, then try again with 5 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also make sure you're standardizing your features - for example by removing the mean and scaling to unit variance. SVMs don't like features that are a lot larger than other features.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2017-03-14T20:37:31.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17587" PostTypeId="2" ParentId="17427" CreationDate="2017-03-14T20:46:39.817" Score="0" Body="&lt;p&gt;For that scenario i think training a classifier like viola-jones/adaboost for segmentation would lead to a region of interest you can use.&lt;/p&gt;&#xA;" OwnerUserId="29723" LastActivityDate="2017-03-14T20:46:39.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17588" PostTypeId="2" ParentId="17536" CreationDate="2017-03-14T20:55:25.077" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;suppose you have a 5 classes of data ordered like a 5 on a dice.&lt;/li&gt;&#xA;&lt;li&gt;to separate the middle cluster from the Rest,&#xA;you do a nonlinear transformation on all data points.&lt;/li&gt;&#xA;&lt;li&gt;since the middle cluster is in the middle of our &quot;kernel&quot;  the other clusters will move to another direction, so that a linear separation will be possible.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29723" LastActivityDate="2017-03-14T20:55:25.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17589" PostTypeId="1" AcceptedAnswerId="17590" CreationDate="2017-03-14T21:25:00.500" Score="1" ViewCount="1356" Body="&lt;p&gt;I have a DataFrame in python by using pandas which has 3 columns and 80.000.000 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Columns are: {event_id,device_id,category}.[&lt;img src=&quot;https://i.stack.imgur.com/YcRCT.png&quot; alt=&quot; here is the first 5 rows of my df&quot;&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;each device has many events and each event can have more than one category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to run Apriori algorithm to find out which categories seem together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is to create a list of lists[[]]: to save the categories which are in the same event for each device. like: [('a'),('a','b')('d'),('s','a','b')] then giving the list of lists as transactions to the algorithm. I need help to create the list of lists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have better idea please tell me because I am new in Python and this was the only way I found out.&lt;/p&gt;&#xA;" OwnerUserId="26589" LastActivityDate="2017-03-14T22:35:14.030" Title="Run Apriori algorithm in python 2.7" Tags="|python|dataset|pandas|association-rules|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17590" PostTypeId="2" ParentId="17589" CreationDate="2017-03-14T22:35:14.030" Score="1" Body="&lt;pre&gt;&lt;code&gt;df.groupby('device_id')['category'].apply(list).tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There's your transactions LOL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you aren't limited to Python 2.7, I'd suggest &lt;a href=&quot;https://pypi.org/project/Orange3-Associate/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange3-Associate&lt;/a&gt; which contains a &lt;a href=&quot;https://orange3-associate.readthedocs.io/en/latest/scripting.html#fpgrowth.frequent_itemsets&quot; rel=&quot;nofollow noreferrer&quot;&gt;frequent_itemsets() function&lt;/a&gt; based on &lt;a href=&quot;http://hanj.cs.illinois.edu/pdf/dami04_fptree.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FP-growth algorithm&lt;/a&gt;, which is &lt;em&gt;orders of magnitude&lt;/em&gt; faster than Apriori.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-03-14T22:35:14.030" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17591" PostTypeId="2" ParentId="17578" CreationDate="2017-03-15T01:39:00.460" Score="21" Body="&lt;p&gt;At this moment, the answer is &lt;strong&gt;no&lt;/strong&gt;. Tensorflow uses CUDA which means only NVIDIA GPUs are supported.&lt;/p&gt;&#xA;&lt;p&gt;For OpenCL support, you can track the progress &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/22&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;BTW, Intel/AMD CPUs are supported.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;The default version of Tensorflow doesn't work with Intel and AMD GPUs, but there are ways to get Tensorflow to work with Intel/AMD GPUs:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For Intel GPUs, follow this &lt;a href=&quot;https://docs.microsoft.com/en-us/windows/ai/directml/gpu-tensorflow-windows&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial from Microsoft&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;For AMD GPUs, use this &lt;a href=&quot;https://izziswift.com/using-keras-tensorflow-with-amd-gpu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28628" LastEditorUserId="127869" LastEditDate="2021-11-17T23:26:07.227" LastActivityDate="2021-11-17T23:26:07.227" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17592" PostTypeId="1" AcceptedAnswerId="17607" CreationDate="2017-03-15T03:56:41.023" Score="0" ViewCount="499" Body="&lt;p&gt;I'm new into the ML Scene and I want to create a phonegap app involving Tensorflow but I'm unsure where to start or if this is even possible. Can anyone give me a hand (Probably by linking me to some resources)? My app will just use tensor flow image recognition (probably pre-trained).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, Felix.&lt;/p&gt;&#xA;" OwnerUserId="29981" LastActivityDate="2017-03-15T11:37:19.150" Title="TensorFlow with Phonegap" Tags="|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17594" PostTypeId="1" CreationDate="2017-03-15T05:37:14.407" Score="1" ViewCount="574" Body="&lt;p&gt;I have the following dataset with the dimensions: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rows: 41174&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Columns: 439316&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I get an error saying that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Exception in thread &quot;main&quot; java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I ran the same computeSVD on the following dataset and it ran perfectly fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rows: 3502&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Columns: 103301&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both the cases, I am passing the value of &quot;k&quot; to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?&lt;/p&gt;&#xA;" OwnerUserId="10717" LastActivityDate="2017-03-16T14:59:04.807" Title="Issue with Spark SVD" Tags="|apache-spark|dimensionality-reduction|scala|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17595" PostTypeId="2" ParentId="17431" CreationDate="2017-03-15T05:55:29.633" Score="1" Body="&lt;p&gt;See this &lt;a href=&quot;http://www.alivelearn.net/?p=1666&quot; rel=&quot;nofollow noreferrer&quot;&gt;answer&lt;/a&gt; by Xu Cui. The reason, as he stated is simple: the model itself is very simple SVM. He suggests lagging 2 points behind to &quot;improve&quot; and get a more powerful model. Let me know what you think of it. We might be overthinking and he might be on point.&lt;/p&gt;&#xA;" OwnerUserId="29987" LastActivityDate="2017-03-15T05:55:29.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17597" PostTypeId="1" CreationDate="2017-03-15T07:21:17.103" Score="6" ViewCount="714" Body="&lt;p&gt;I have some labeled sensor data.&lt;/p&gt;&#xA;&lt;p&gt;Now, I would like to know how to extract features from time series using DFT, DWT, and HAAR transforms.&lt;/p&gt;&#xA;&lt;p&gt;I know that the transformations above transform a signal to frequency or time frequency domain, but how can I get average frequency for this non stationary signal?&lt;/p&gt;&#xA;" OwnerUserId="29989" LastEditorUserId="29169" LastEditDate="2021-10-19T16:57:27.433" LastActivityDate="2022-08-04T17:03:20.307" Title="Time Series pattern recognition and classification problem" Tags="|machine-learning|data-mining|time-series|feature-extraction|anomaly-detection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="17598" PostTypeId="1" AcceptedAnswerId="17600" CreationDate="2017-03-15T07:36:40.437" Score="17" ViewCount="9057" Body="&lt;p&gt;I hope this question is the most suitable in this site...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python, usually the class name is defined using the capital letter as its first character, for example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Vehicle:&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in machine learning field, often times train and test data are defined as &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; - not &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;. For example, I'm now reading &lt;a href=&quot;https://www.packtpub.com/books/content/keras&quot; rel=&quot;noreferrer&quot;&gt;this tutorial on Keras&lt;/a&gt;, but it uses the &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; as its variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;&#xA;mnist = datasets.load_digits()&#xA;X = mnist.data&#xA;Y = mnist.target&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why are these defined as capital letters? Is there any convention (at least in Python) among machine learning field that it is better to use the capital letter to define these variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe do people distinguish the upper vs lower case variables in machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact the same tutorial later distinguish these variables like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import train_test_split&#xA;&#xA;train_X, test_X, train_y, test_y = train_test_split(X, Y, train_size=0.7, random_state=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8432" LastEditorUserId="8432" LastEditDate="2017-03-15T07:43:26.723" LastActivityDate="2022-09-06T12:47:15.213" Title="Why are variables of train and test data defined using the capital letter (in Python)?" Tags="|python|dataset|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17599" PostTypeId="2" ParentId="17598" CreationDate="2017-03-15T07:49:47.237" Score="9" Body="&lt;p&gt;I think this has nothing to do with Python but with mathematics. X is a matrix and y is a vector (most of the time). Usually upper case letters are used for matrices and lower case letters are used for vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why you often see something like this (&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_digits_linkage.html#sphx-glr-auto-examples-cluster-plot-digits-linkage-py&quot; rel=&quot;noreferrer&quot;&gt;from sklearn examples&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;digits = datasets.load_digits(n_class=10)&#xA;X = digits.data&#xA;y = digits.target&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or that (from the same example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_min, x_max = np.min(X_red, axis=0), np.max(X_red, axis=0)&#xA;X_red = (X_red - x_min) / (x_max - x_min)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here X_red is an m x n matrix (upper case) and x_min is a vector (lower case) of length n.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2017-03-15T07:49:47.237" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17600" PostTypeId="2" ParentId="17598" CreationDate="2017-03-15T07:49:58.523" Score="25" Body="&lt;p&gt;The X (and sometimes Y) variables are matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some math notation, it is common practice to write vector variable names as lower case and matrix variable names as upper case. Often these are in bold or have other annotation, but that does not translate well to code. Either way, I believe that the practice has transferred from this notation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also notice in code, when the target variable is a single column of values, it is written &lt;code&gt;y&lt;/code&gt;, so you have &lt;code&gt;X, y&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this has no special semantic meaning in Python and you are free to ignore the convention. However, because it has become a convention, it may be worth maintaining if you share your code.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-03-15T07:49:58.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17601" PostTypeId="5" CreationDate="2017-03-15T08:49:06.893" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-03-15T08:49:06.893" LastActivityDate="2017-03-15T08:49:06.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17602" PostTypeId="4" CreationDate="2017-03-15T08:49:06.893" Score="0" Body="Amazon Redshift is a petabyte-scale data warehousing service using existing business intelligence tools to analyze the data. Redshift is a column-oriented database based on ParAccel (MPP over PostgreSQL 8.0)." OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-03-15T10:32:00.487" LastActivityDate="2017-03-15T10:32:00.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17604" PostTypeId="1" AcceptedAnswerId="17606" CreationDate="2017-03-15T09:21:54.283" Score="1" ViewCount="576" Body="&lt;p&gt;I have been reading about Generative Adversarial Networks (GANs) and was wondering if it would make sense to train a generator function only to use it for creating more training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a scenario where I don't have enough training data to build a robust classifier, can I use this limited data to train a generator that'll produce samples good enough to improve the accuracy of my discriminator (classifier)?&lt;/p&gt;&#xA;" OwnerUserId="29995" LastActivityDate="2017-03-15T11:21:25.163" Title="GANs to augment training data" Tags="|neural-network|dataset|accuracy|training|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17605" PostTypeId="2" ParentId="17583" CreationDate="2017-03-15T09:27:15.973" Score="1" Body="&lt;p&gt;Wrote simple class - if someone will be interested:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.ml.Transformer;&#xA;import org.apache.spark.ml.param.Param;&#xA;import org.apache.spark.ml.param.ParamMap;&#xA;import org.apache.spark.sql.Column;&#xA;import org.apache.spark.sql.Dataset;&#xA;import org.apache.spark.sql.Row;&#xA;import org.apache.spark.sql.functions;&#xA;import org.apache.spark.sql.types.DataTypes;&#xA;import org.apache.spark.sql.types.StructType;&#xA;&#xA;import java.util.UUID;&#xA;&#xA;public class HTMLStripper extends Transformer {&#xA;private static final String HTMLStripper = &quot;HTMLStripper&quot;;&#xA;private String inputColumn;&#xA;private String outputColumn;&#xA;&#xA;public HTMLStripper (String inputColumn, String outputColumn) {&#xA;    this.inputColumn = inputColumn;&#xA;    this.outputColumn = outputColumn;&#xA;}&#xA;&#xA;&#xA;@Override&#xA;public String uid() {&#xA;    return UUID.fromString(&quot;HTMLStripper&quot;).toString();&#xA;}&#xA;&#xA;@Override&#xA;public StructType transformSchema(StructType schema) {&#xA;    return schema.add(outputColumn, DataTypes.StringType, true);&#xA;}&#xA;&#xA;@Override&#xA;public Dataset&amp;lt;Row&amp;gt; transform(Dataset&amp;lt;?&amp;gt; dataset) {&#xA;    dataset.sqlContext().udf().register(HTMLStripper, (String str) -&amp;gt; str.replaceAll(&quot;&amp;lt;[^&amp;gt;]*&amp;gt;&quot;, &quot;&quot;),&#xA;            DataTypes.StringType);&#xA;    Column col = dataset.col(inputColumn);&#xA;    col = functions.callUDF(HTMLStripper, col);&#xA;    return dataset.withColumn(outputColumn, col);&#xA;}&#xA;&#xA;@Override&#xA;public Transformer copy(ParamMap extra) {&#xA;    return new HTMLStripper(inputColumn, outputColumn);&#xA;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29534" LastActivityDate="2017-03-15T09:27:15.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17606" PostTypeId="2" ParentId="17604" CreationDate="2017-03-15T11:21:25.163" Score="1" Body="&lt;p&gt;Yes and no depending on how you define &quot;good enough samples&quot;.&#xA;You will likely end up with a chicken and egg problem: you want to use the GAN to generate training data, but the GAN doesn't have enough training data itself to generate convincing enough samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other techniques exist for data synthesis of training images. For example: adding noise, flipping axis, change luminosity, change color, random cropping, random distorsion.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-03-15T11:21:25.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17607" PostTypeId="2" ParentId="17592" CreationDate="2017-03-15T11:37:19.150" Score="1" Body="&lt;p&gt;TensorFlow is a tool to write computation using data flow graphs; this being said, if you want your app to use a pre-trained model only, there is no requirements to use TensorFlow specifically. You could even use one of the ML library written in Javascript to import and run the pre-trained model.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Javascript DL libs: &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs.stanford.edu/people/karpathy/convnetjs/&lt;/a&gt;, &lt;a href=&quot;https://github.com/dmlc/mxnet.js/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/mxnet.js/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;TensorFlow for mobile: &lt;a href=&quot;https://www.tensorflow.org/mobile/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/mobile/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Caffe Android: &lt;a href=&quot;https://github.com/sh1r0/caffe-android-lib&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/sh1r0/caffe-android-lib&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;DL4J Android: &lt;a href=&quot;https://deeplearning4j.org/android&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://deeplearning4j.org/android&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-03-15T11:37:19.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17609" PostTypeId="1" CreationDate="2017-03-15T11:45:17.660" Score="3" ViewCount="66" Body="&lt;p&gt;I have a CNN outputting probabilities using a logistic output. The performances are good on the test set. Yet the probabilities it outputs are very clustered, they are either 0 or 1! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FlcUg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FlcUg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like some more granularity in the probabilities outputed by the network. Some features are more obvious than other and the network should be able to learn this. Any idea on how I could solve this problem? &lt;/p&gt;&#xA;" OwnerUserId="25418" LastActivityDate="2019-01-06T14:36:21.633" Title="How to prevent a neural network from outputing too clustered probabilities?" Tags="|neural-network|logistic-regression|prediction|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17610" PostTypeId="2" ParentId="17567" CreationDate="2017-03-15T12:10:18.203" Score="1" Body="&lt;p&gt;RNNs are not designed to do language modeling exclusively, they are designed to process time series data, and language happened to be representable as time series.&#xA;There is plenty of papers demonstrating how to use RNNs to do classification and regression on time series (&lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;q=time%20series%20recurrent%20neural%20network&amp;amp;btnG=&amp;amp;as_sdt=1%2C5&amp;amp;as_sdtp=&amp;amp;oq=time%20series%20recurr&quot; rel=&quot;nofollow noreferrer&quot;&gt;awesome list of papers&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One-hot encoding is often used in cases where the input is discrete and not a number that can be directly fed into a model. However, one-hot encoding is actually not always the norm for language-modeling, some research actually map each character (or each word depending on how one wants to model the problem) to a unique numerical identifier (e.g. with $id\in \left\{ 0\ldots n \right\}$; with $n$ the size of the vocabulary) and the model map that identifier to a vector representation. This is particularly useful when the size of the vocabulary is huge and you want to avoid having to deal with big one-encoded vectors. Take a look at &lt;a href=&quot;https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; and &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec Tensorflow&lt;/a&gt; for more details about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you want to process sensor data. There is no need for one-hot encoded because your data are continuous and numerical. In other words, you can input the recorded EEG data directly; although it's usually better to clean them up and normalize them beforehand. There is actually plenty of papers about &lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;q=eeg%20recurrent%20neural%20network&amp;amp;btnG=&amp;amp;as_sdt=1%2C5&amp;amp;as_sdtp=&amp;amp;oq=EEG%20recurrent&quot; rel=&quot;nofollow noreferrer&quot;&gt;EEG data processing with RNNs&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of the type of data, the idea to fed them into a RNN remains the same: &lt;strong&gt;provide a data sample $x$ for each timestep $t$&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For EEG data recorded for 5 timesteps with 1 electrode $a$, you would have a 1-dimensional vector with one sample per timestep:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a = [0.12, 0.44, 0.134, 0.39, 0.23]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$inputvector = [0.12, 0.44, 0.134, 0.39, 0.23]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For EEG data recorded for 5 timesteps with 3 electrodes $a, b, c$, you would have a 3-dimensional vector with one sample per timestep:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a = [0.12, 0.44, 0.134, 0.39, 0.23]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b = [0.43, 0.92, 0.3, 0.37, 0.4]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c = [0.13, 0.1, 0.4, 0.21, 0.14]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$inputvector = [[0.12, 0.43, 0.13], [0.44, 0.92, 0.1], ...]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend you to read some of the literature I linked above.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastEditorUserId="28129" LastEditDate="2017-03-15T12:45:19.570" LastActivityDate="2017-03-15T12:45:19.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17611" PostTypeId="2" ParentId="15553" CreationDate="2017-03-15T13:19:31.967" Score="0" Body="&lt;p&gt;It's very dependent on the specific situation and the problem you want to solve. There exist some general rules, for example &lt;em&gt;wrapper methods&lt;/em&gt; are more flexible and also more prone to overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature selection performance can be evaluated by the overall performance of learning task for example one can select features with different methods and then use these different feature sets for classification and compare the precision of obtained classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important factor in some scenarios like some biological applications is the interpretability of selected features and the results, for example  in a clustering problem, meaning of selected features and resulted clusters is a very important measure of performance. &lt;/p&gt;&#xA;" OwnerUserId="29878" LastActivityDate="2017-03-15T13:19:31.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17612" PostTypeId="1" CreationDate="2017-03-15T14:51:06.853" Score="2" ViewCount="1323" Body="&lt;p&gt;The goal is to run Poisson regression for neural networks (multi-layer perceptron) in R.&lt;/p&gt;&#xA;&lt;p&gt;I am currently using the &lt;code&gt;neuralnet&lt;/code&gt; package in R.&lt;/p&gt;&#xA;&lt;p&gt;I have read &lt;a href=&quot;https://datascience.stackexchange.com/q/12738/98307&quot;&gt;Is there an R package which uses neural networks to explicitly model count data?&lt;/a&gt; and &lt;a href=&quot;http://www.mathstat.dal.ca/%7Ehgu/Neural%20Comput%20&amp;amp;%20Applic.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nonlinear Poisson regression using neural networks: a simulation study&lt;/a&gt;. I have figured out that I should change the error function to &lt;code&gt;err.fct = function(x, y) { -(-x+y*log(x))}&lt;/code&gt; (for Poisson).&lt;/p&gt;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;fit &amp;lt;- neuralnet(nclaims ~ age_ph + gender + tar_region + weight + age_car + ptw,  data = df,  hidden = 1,  err.fct = function(x, y) { -(-x+y*log(x))},  act.fct = &amp;quot;logistic&amp;quot;)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I know that &lt;code&gt;act.fct = &amp;quot;logistic&amp;quot;&lt;/code&gt; specifies the sigmoid function in the hidden layer, but how can I change to the exponential function in the output layer?&lt;/p&gt;&#xA;" OwnerUserId="30012" LastEditorUserId="98307" LastEditDate="2020-08-12T15:32:35.357" LastActivityDate="2020-08-12T15:32:35.357" Title="Neural network (multi-layer perceptron) in R with count data as response" Tags="|neural-network|r|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17615" PostTypeId="1" CreationDate="2017-03-15T16:08:25.220" Score="1" ViewCount="16" Body="&lt;p&gt;I have built a logistic regression algorithm in R and have trained it and tested it and now I want to apply it on new incoming data. My only question is that whether I always have to keep my training set data in the csv I want to get output of. If I am removing training set data, I am getting all values as NA&lt;/p&gt;&#xA;" OwnerUserId="27231" LastActivityDate="2017-03-15T16:08:25.220" Title="Application of validated logistic regression model on new data" Tags="|logistic-regression|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17616" PostTypeId="1" AcceptedAnswerId="17617" CreationDate="2017-03-14T23:21:31.650" Score="4" ViewCount="155" Body="&lt;p&gt;I am learning machine learning and I'm trying to implement a solution for a real problem: predict from a human sentence what programming function he/she is trying to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a series of programming functions related with a series of descriptions (there can be n &gt; 0 descriptions for each unique function).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created a neural network and a bag of words model trying to convert a human sentence &quot;we get the data from the database&quot; to a programming function. So far it works with &lt;em&gt;very&lt;/em&gt; easy examples but not with my real data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this works:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;description&quot;     | programming function&#xA;lala lolo lulu ka | function1&#xA;lala lolo lulu ke | function1&#xA;lala lolo lulu ko | function1&#xA;lala lele lili ka | function2&#xA;lala lele lili ki | function2&#xA;lala lele lili ko | function2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Every word in the description is converted to an neuron-input (with value of 1 if present and 0 if not present) and every possible function is converted to a neuron-output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using pyBrain with back propagation and an error threshold of 0.005. The neural network has three layers, and the middle one has lenght: number of possible words + number of possible programming functions (this is kind of arbitrary).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know full text search or auto-complete is probably a better alternative for this task, but I'm just experimenting with machine learning and I'd like this to work if possible. In my real data I have 1000 descriptions related with ~500 functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is bag of words + neural networks a good approach for solving this?&lt;/li&gt;&#xA;&lt;li&gt;Maybe Word2vec is a better option? &lt;/li&gt;&#xA;&lt;li&gt;If neither is good, is there any known machine learning approach that could work with something like that?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30009" OwnerDisplayName="hhaamm" LastActivityDate="2017-03-15T16:36:22.677" Title="Is ML a good solution for identifying what the user wants to do from a sentence?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17617" PostTypeId="2" ParentId="17616" CreationDate="2017-03-15T00:38:53.440" Score="4" Body="&lt;p&gt;Yes this problem is extremely well suited for machine learning. However, I think you should be careful as to which algorithms you tend to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Feature Extraction&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The Model&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, you have 1000 instances!!! Yes. But, you also have 500 classes. So imagine you have a very young child and you want him to be able to correctly classify 500 different types of images. Then you can't just show the kid 2 different examples of each class for him to truly understand what each class really means. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a very general rule of thumb, the number of instances you need to train a model increases exponentially with number of classes. So you will need a MASSIVE amount of data to properly train a neural network model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest a less intensive model. Moreover, looking at your example, it seems that the classes should be linearly separable. So you can use something really simple, linear regression, logistic regression, naive bayes or knn. These methods would do MUCH MUCH better than a neural network. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My Suggestion&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I would start with bag of words and then use knn. This should be a good starting point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A neural network is 0% recommended for the amount of data you have. &lt;/p&gt;&#xA;" OwnerUserId="29587" OwnerDisplayName="JahKnows" LastActivityDate="2017-03-15T00:38:53.440" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17618" PostTypeId="1" CreationDate="2017-03-15T17:18:13.047" Score="4" ViewCount="660" Body="&lt;p&gt;Take $ H $ to be a class of binary classifiers ever a domain $X$. $D$ is a unknown distrbution over $X$, and $f$ is the target hypothesis in $H$. If we fix some $h$ $\in H$ we have the expected value of the training error equal to generalization error, that is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbb{E}[L_S(h)]=L_{(D,f)}(h)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please come up with the proof of this statement?&#xA;Would you recommend another page inside stackexchange for these theoretical questions about machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="29999" LastActivityDate="2018-10-14T22:43:01.727" Title="Prove that the expected value of the training error is equal to generalization error for a class of binary clasiffiers" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17619" PostTypeId="1" CreationDate="2017-03-15T17:20:28.607" Score="4" ViewCount="129" Body="&lt;p&gt;I'm fairly new to machine learning and for that matter, neural networks, but for the past couple of days I decided to take a stab at a fairly classical and practical problem of neural networks/machine learning which is recommendation systems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies if this is an unnecessarily broad question, but I found it hard to read up on resources answering this particular question. My main question is, how do you even model the problem (or what directions/advice is there on how to model it)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $M$ denote the set of all movies you can recommend, each of which have an associated $id$ to them. What exactly would the input to the model be (and the output) and how would you segregate the training data and the observed results? For example, if I have a row of training data (row of size $|M|$ and each entry is a number between $1$ and $10$ that denotes the user's rating, or $0$ if the user has not seen it yet), would I just remove, say half of the user's ratings that are between $1$ and $10$ (so are meaningful) and replace them with $0$ in the training row and then in my observed row (for testing) it would be the entire row with no missing ratings like illustrated below?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Information} \rightarrow \underbrace{[0 \ 0 \ 0 \ 0 \ 0 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Training} \rightarrow \underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Observed} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few naive ways that are obvious are:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Input size = $|M|$, output size = $|M|$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Each input neuron is given the associated user rating for the $i^{th}$ movie. Assume some number of intermediate layers and then we get to the output size, which is $|M|$ output neurons each of which attempt to predict the predicted rating of the user for that movie. This seems weird, because basically almost all of the input neurons will be given $0$ and almost all of the output neurons will try to output $0$ (since it's likely a user has only seen an insignificant portion of all $|M|$ movies). This doesn't seem like the neural network will learn anything useful. And what do I do for movies that the user hasn't seen? Do I just consider this a $0$ loss? So the loss function for neuron $i$ is just:&#xA;$$\text{Loss}(y^{(i)}) = 0 \ \text{if the user hasn't seen ith movie, otherwise: } \ \phi{(y^{(i)}, t^{(i)})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\phi$ is some loss function of the neuron predicted output $y^{(i)}$ and the observed value (the actual user rating) $t^{(i)}$. I find it difficult to believe that this will yield any useful ratings though..since the loss has no effect on the nonsensical outputs of my neural network (it doesn't affect the $i^th$ neurons' values if the user hasn't seen it yet). This also poses the problem of how a prediction will be made then. Do I just take all the highest predicted values from the neural network and return the respective movies?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the above problems, I tried to express the model differently, but I don't think there is anyway to overcome the sparsity of the data and also the fact that the output size must be $|M|$ as I can't have it try to predict discrete integer movie ids. Could someone please enlighten me on this topic or provide some possible insight to obvious mistakes I am making?&lt;/p&gt;&#xA;" OwnerUserId="30017" LastActivityDate="2024-01-02T00:03:50.527" Title="Neural Network - Sparsity of collaborative based filtering and modelling the prediction problem" Tags="|neural-network|recommender-system|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17620" PostTypeId="1" AcceptedAnswerId="17656" CreationDate="2017-03-15T17:23:16.390" Score="1" ViewCount="1638" Body="&lt;p&gt;I was looking at the arguments in the linear regularization methods with cross validation within scikit-learn. RidgeCV has an argument scoring which is None by default but one can use a custom scorer (e.g. RMSE). I noticed that no such option is given for the rest of the classes (LassoCV, LassoLarsCV, ElasticNetCV). Why is this? Is MSE used by default in all of them?&lt;/p&gt;&#xA;" OwnerUserId="14487" LastActivityDate="2017-03-16T16:09:47.363" Title="scoring argument in scikit-learn LassoCV, LassoLarsCV, ElasticNetCV" Tags="|regression|scikit-learn|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17621" PostTypeId="1" CreationDate="2017-03-15T18:35:38.523" Score="2" ViewCount="210" Body="&lt;p&gt;I work for a fire and EMS dispatch agency in Florida. All of our apparatus have gps to record the longitude, latitude, timestamp, apparatus, speed, unit status and the incident number. &#xA;The klaxon in the fire house sounds and dispatches the ambulance to an emergency. The crew has 60 seconds to respond and this is called turnout time. The crew gets in the apparatus and starts driving to the scene. Wheels starting to roll is called en-route time. When the ambulance arrives at the emergency it is coded as Arrive on Scene. The gps is recording all of this and transmits it into a SQL Server 2012 database.&#xA;My problem is this: Occasionally a crew will call on the radio and put themselves into en-route status while they are loitering at the fire house (using the bathroom, finishing their sandwich, waiting for a commercial break from their favorite show). At the dispatch center we think the ambulance is en-route - but the ambulance is actually standing still.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to achieve:&#xA;I want to skip the first n rows where the ambulance is loitering at the fire house. I have provided some data in this post. Skipping the first n rows where the ambulance is loitering allows me to calculate a true en-route time. It also allows me to monitor my system to make sure emergencies are being responded to quickly - or correct it before it becomes a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: This query is not as simple as (WHERE Speed &amp;lt;&gt; 0) and here is why; An ambulance goes en-route and arrives at the emergency and forgets to call in to Dispatch to tell them they've arrived on scene. The speed is registering as 0 but they have actually arrived on the scene, just forgot to call in to the dispatch center. All of these type of errors are on the back end of the incident. That is ok - at least the ambulance is at the scene of the emergency. I am looking to skip the first n observations at the front end of the incident where the speed is 0 but the crew claims to be en-route.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Geo-fencing is not an option. I am looking for pure Transact SQL 2012 method.&#xA;I have tried to use OFFSET and FETCH but I can't figure it out.&#xA;I have also tried to do some fancy stuff with the latitude and longitude to measure movement, but that doesn't solve my problem of the back-end when the ambulance is arrived on scene but is not moving.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the sample data below my goal would be to skip the first 17 rows up until 06:51:04 which is when the ambulance starts to move at 6 miles per hour. I want that to be my starting row. &#xA;Also note towards the end of the incident at 06:55:02 the ambulance appears to be on scene. I want to keep these rows because they do not pertain to loitering at the station. It simply appears that the crew forgot to call in &quot;arrive on scene&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day I want to skip first n rows because the speed is 0, I suspect the crew is likely to be goofing off at the fire house. But leave the last n rows where the speed is 0, I suspect the crew is arrived on scene and merely forgot to register Arrive on Scene.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This database is massive. I have more than 200 fire engines and 40 ambulances running 24 hours a day 7 days a week. My sample below is for just one emergency with a single ambulance. Just to get a start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;/&lt;em&gt;There is not a primary key because this is just a flat table.&lt;/em&gt;/&#xA;/&lt;em&gt;My date format is yyyy-mm-dd hh:mm:ss.xxx&lt;/em&gt;/&#xA;CREATE TABLE AmbulanceGPS&#xA;(&#xA;  Incident VARCHAR(20) NULL&#xA;, Ambulance VARCHAR(5) NULL&#xA;, DateTimeStamp DATETIME NOT NULL&#xA;, UnitStatus VARCHAR(10) NULL&#xA;, Latitude FLOAT NULL&#xA;, Longitude FLOAT NULL &#xA;, Speed TINYINT NULL&#xA;)&#xA;;&#xA; INSERT INTO AmbulanceGPS&#xA; VALUES&#xA;  (MLC170314022391,LCM04,2017-03-14 06:49:29.410,EnRoute,26.56674,-81.95833,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:29.670,EnRoute,26.56674,-81.95833,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:29.680,EnRoute,26.56674,-81.95833,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:34.633,EnRoute,26.56674,-81.95833,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:39.520,EnRoute,26.56674,-81.95833,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:44.477,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:49.520,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:54.697,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:49:59.470,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:04.497,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:09.470,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:14.510,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:19.577,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:24.537,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:29.580,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:34.453,EnRoute,26.56674,-81.95832,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:39.603,EnRoute,26.56674,-81.95832,1)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:44.420,EnRoute,26.56685,-81.95834,6)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:49.437,EnRoute,26.56696,-81.95828,6)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:54.427,EnRoute,26.56694,-81.95827,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:50:59.450,EnRoute,26.56694,-81.95826,1)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:04.563,EnRoute,26.56694,-81.95811,6)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:09.510,EnRoute,26.56705,-81.95783,17)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:14.153,EnRoute,26.56755,-81.9576,33)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:17.183,EnRoute,26.56794,-81.95731,39)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:20.130,EnRoute,26.56829,-81.95692,42)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:23.230,EnRoute,26.56859,-81.95642,44)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:26.200,EnRoute,26.56877,-81.95584,46)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:29.167,EnRoute,26.56892,-81.95522,48)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:32.117,EnRoute,26.56908,-81.95457,50)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:35.097,EnRoute,26.56924,-81.95391,51)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:38.160,EnRoute,26.5694,-81.95324,51)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:41.270,EnRoute,26.56957,-81.95257,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:44.170,EnRoute,26.56976,-81.95191,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:47.203,EnRoute,26.57001,-81.95126,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:50.197,EnRoute,26.57029,-81.95063,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:53.300,EnRoute,26.57062,-81.95003,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:56.210,EnRoute,26.57098,-81.94946,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:51:59.153,EnRoute,26.5714,-81.9489,53)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:02.183,EnRoute,26.57185,-81.94837,54)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:05.220,EnRoute,26.57233,-81.94788,54)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:08.173,EnRoute,26.57283,-81.94743,53)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:11.180,EnRoute,26.57336,-81.94703,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:14.210,EnRoute,26.5739,-81.9467,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:17.133,EnRoute,26.57447,-81.94639,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:20.197,EnRoute,26.57505,-81.94614,50)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:23.133,EnRoute,26.57558,-81.94589,46)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:26.223,EnRoute,26.57603,-81.94552,45)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:29.110,EnRoute,26.57634,-81.94502,45)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:32.207,EnRoute,26.57646,-81.94443,44)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:35.150,EnRoute,26.57646,-81.94391,33)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:40.250,EnRoute,26.57645,-81.94359,7)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:45.087,EnRoute,26.57648,-81.94343,13)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:49.190,EnRoute,26.57685,-81.94319,28)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:53.117,EnRoute,26.57739,-81.94308,37)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:56.083,EnRoute,26.57788,-81.94296,42)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:52:58.950,EnRoute,26.57823,-81.94287,44)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:02.067,EnRoute,26.57899,-81.94271,48)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:05.140,EnRoute,26.57958,-81.94259,49)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:08.067,EnRoute,26.58017,-81.94248,50)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:11.083,EnRoute,26.58078,-81.94237,50)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:14.137,EnRoute,26.58139,-81.94228,51)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:17.140,EnRoute,26.58201,-81.94219,51)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:20.040,EnRoute,26.58263,-81.94212,52)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:23.090,EnRoute,26.58327,-81.94206,53)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:26.083,EnRoute,26.58392,-81.942,54)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:29.203,EnRoute,26.58458,-81.94197,55)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:31.100,EnRoute,26.58503,-81.94195,55)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:33.007,EnRoute,26.58548,-81.94195,56)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:35.110,EnRoute,26.58593,-81.94193,56)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:37.067,EnRoute,26.58639,-81.94191,56)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:39.090,EnRoute,26.58684,-81.94191,56)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:41.123,EnRoute,26.58728,-81.94191,55)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:44.073,EnRoute,26.5879,-81.94191,48)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:47.060,EnRoute,26.5884,-81.9419,36)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:52.130,EnRoute,26.58877,-81.94186,16)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:53:57.073,EnRoute,26.58888,-81.94146,23)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:01.183,EnRoute,26.58888,-81.94081,31)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:05.123,EnRoute,26.58888,-81.9402,35)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:09.100,EnRoute,26.58887,-81.93953,37)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:12.123,EnRoute,26.58887,-81.93901,38)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:15.103,EnRoute,26.58887,-81.93848,38)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:18.160,EnRoute,26.58887,-81.93797,40)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:22.160,EnRoute,26.58887,-81.93733,28)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:27.083,EnRoute,26.58904,-81.93703,15)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:32.120,EnRoute,26.58936,-81.93704,15)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:37.147,EnRoute,26.58965,-81.93704,13)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:42.133,EnRoute,26.58992,-81.93703,13)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:47.153,EnRoute,26.59019,-81.93704,13)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:52.107,EnRoute,26.59044,-81.93702,10)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:54:57.220,EnRoute,26.59062,-81.93701,7)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:02.180,EnRoute,26.59071,-81.93701,2)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:10.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:15.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:20.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:25.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:30.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:35.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:40.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:45.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:50.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:55:55.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:56:00.180,EnRoute,26.59071,-81.93701,0)&#xA;, (MLC170314022391,LCM04,2017-03-14 06:56:10.180,EnRoute,26.59071,-81.93701,0)&#xA;;&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="3504" LastActivityDate="2017-07-13T22:29:33.707" Title="SQL Server and skip first n rows" Tags="|sql|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17623" PostTypeId="2" ParentId="17621" CreationDate="2017-03-15T19:44:48.360" Score="2" Body="&lt;p&gt;In your problem description you want to extract all the rows after the truck starts moving. You state that you want all the rows after 06:51:04 where the speed has reached 6. However, the speed reached 6, twice before that. Without stating a more robust decision boundary this would not be an easy to solve problem. However, if I assume you want to detect the first presence of a speed greater than or equal to 5 then it is simple. You can play with this threshold speed value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would first find the DateTimeStamp for the first non-zero element in the recordlist and then get all rows that are greater than that. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Select * from AmbulanceGPS&#xA;where AmbulanceGPS.DateTimeStamp &amp;gt;= &#xA;   (Select DateTimeStamp from AmbulanceGPS&#xA;    WHERE Speed &amp;gt;= 5 &#xA;    order by DateTimeStamp &#xA;    LIMIT 1);&#xA;order by DateTimeStamp &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Furthermore, if you also want to reject all excess points in your calculation that are found once they have arrived at their location you can add this. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Select * from AmbulanceGPS&#xA;where AmbulanceGPS.DateTimeStamp &amp;gt;= &#xA;   (Select DateTimeStamp from AmbulanceGPS&#xA;    WHERE Speed &amp;gt;= 5 &#xA;    order by DateTimeStamp &#xA;    LIMIT 1)&#xA;and&#xA;AmbulanceGPS.DateTimeStamp &amp;lt;= &#xA;   (Select DateTimeStamp from AmbulanceGPS&#xA;    WHERE Speed &amp;gt;= 5 &#xA;    order by DateTimeStamp DESC &#xA;    LIMIT 1);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From here you can do last record of datetime minus the first record. And get the time it took for it to get to the emergency. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-03-15T19:53:35.863" LastActivityDate="2017-03-15T19:53:35.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17624" PostTypeId="1" CreationDate="2017-03-15T20:12:00.330" Score="1" ViewCount="274" Body="&lt;p&gt;I am using SVM algorithm for text classification. I need to know where can I find twitter dataset and how can I use it in weka tool or any other tool? &lt;/p&gt;&#xA;" OwnerUserId="30024" LastEditorUserId="29575" LastEditDate="2017-12-01T16:08:27.053" LastActivityDate="2020-12-13T20:32:35.950" Title="Text classification" Tags="|classification|text-mining|svm|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17625" PostTypeId="2" ParentId="17624" CreationDate="2017-03-15T20:24:02.637" Score="0" Body="&lt;p&gt;These are a few sites I found for that. I am not an R programmer so I don't know any Weka tools and how to use them but hope this helps. You can find them &lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/sanders-twitter-0.2.zip&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Twitter+Data+set+for+Arabic+Sentiment+Analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28295" LastEditorUserId="85045" LastEditDate="2020-12-13T20:32:35.950" LastActivityDate="2020-12-13T20:32:35.950" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17626" PostTypeId="1" CreationDate="2017-03-15T21:00:08.093" Score="4" ViewCount="27845" Body="&lt;p&gt;after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse.  I get the following message, seemingly because the package DBI is missing:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;install.packages(&quot;tidyverse&quot;)&#xA;  Installing package into ‘C:/Users/jshea/Documents/R/win-library/3.3’&#xA;  (as ‘lib’ is unspecified)&#xA;  trying URL '&lt;a href=&quot;https://cran.rstudio.com/bin/windows/contrib/3.3/tidyverse_1.1.1.zip&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.rstudio.com/bin/windows/contrib/3.3/tidyverse_1.1.1.zip&lt;/a&gt;'&#xA;  Content type 'application/zip' length 41994 bytes (41 KB)&#xA;  downloaded 41 KB&#xA;  package ‘tidyverse’ successfully unpacked and MD5 sums checked&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The downloaded binary packages are in&#xA;    C:\Users\jshea\AppData\Local\Temp\1\RtmpmoPd5V\downloaded_packages&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;library(tidyverse)&#xA;  Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : &#xA;    there is no package called ‘DBI’&#xA;  Error: package or namespace load failed for ‘tidyverse’&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Earlier this morning, when I did this the first time, the system seemed to download all the packages except there was some kind of error associated with DBI- I think a database package.  Even uninstalling and deleting obvious remnants of Rstudio does not seem to give me a clean base to fix this.  What can I do? &lt;/p&gt;&#xA;" OwnerUserId="30025" LastActivityDate="2017-03-15T21:17:13.480" Title="Having trouble installing and loading tidyverse- No DIB package" Tags="|dataset|" AnswerCount="1" CommentCount="1" ClosedDate="2017-03-16T22:32:15.953" ContentLicense="CC BY-SA 3.0" />
  <row Id="17627" PostTypeId="2" ParentId="17626" CreationDate="2017-03-15T21:17:13.480" Score="8" Body="&lt;p&gt;It sounds like you do not have a dependency installed which &lt;code&gt;tidyverse&lt;/code&gt; needs to run correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually dependencies install with the package, but at times you need to coerce them to do so, so I usually install with the coercion for all packages.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;tidyverse&quot;, dependencies=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, if that was the only error, or you installed in this manner and did not get &lt;code&gt;DBI&lt;/code&gt;, then I would just install it directly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages(&quot;DBI&quot;, dependencies=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once you have done that, if you are still having problems, explicitly load DBI:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;require(DBI)&lt;/code&gt; or &lt;code&gt;library(DBI)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and you should be good to go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a problem still, comment and I will try to help you chase it down.&lt;/p&gt;&#xA;" OwnerUserId="29905" LastActivityDate="2017-03-15T21:17:13.480" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17628" PostTypeId="1" CreationDate="2017-03-16T02:51:14.553" Score="-3" ViewCount="52" Body="&lt;p&gt;I would like to examine an existing Git repository and extract all defined releases into a subfolder. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if application A had 26 releases, my bash script would extract all 26 versions into subfolders such as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A/(folder) for each of the defined releases&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The preferred language is bash&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks all&lt;/p&gt;&#xA;" OwnerUserId="27396" LastEditorUserId="27396" LastEditDate="2017-03-16T03:03:56.607" LastActivityDate="2017-03-16T04:55:42.220" Title="Extract all releases from GIT repository" Tags="|version-control|linux|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-16T12:06:10.423" ContentLicense="CC BY-SA 3.0" />
  <row Id="17629" PostTypeId="1" CreationDate="2017-03-16T02:52:28.617" Score="6" ViewCount="1622" Body="&lt;p&gt;I am currently trying to recreate the result of this &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;, in which they do feature extraction from a &quot;spectogram&quot; of log-melfilter energies.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xbGLI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xbGLI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the paper doesn't state what kind of feature I am seeking, i am currently trying to extract features, and match them to MFCC features. The paper states a technique called LWS (Limited weight sharing) in which the spectogram frequency axis will be divided into section, and each section don't share their weight with others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i've divided the my input image into 13 section to receive 1 output features  from a (6,3,3) input image. 6 for the number of rows, 3 as each column represent [static delta delta_delta] data of the given log melfilter energi, and the last 3 is the color channels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If i'd used 13 filterbanks, and made the plot, will the result of this be that each (1,3,3) matrix would result in one feature, but that seemed a bit too good to be true, so i decided to use 78 filterbanks and divide it into 13 section which should result in one feature can be extracted from a matrix of size (6,3,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training the network with this model structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_model(init_mode='normal',activation_mode='softsign',optimizer_mode=&quot;Adamax&quot;, activation_mode_conv = 'softsign'):&#xA;    model = Sequential()&#xA;&#xA;&#xA;    model.add(ZeroPadding2D((6,4),input_shape=(6,3,3)))&#xA;    model.add(Convolution2D(32,3,3 , activation=activation_mode_conv))&#xA;    print model.output_shape&#xA;    model.add(Convolution2D(32, 3,3, activation=activation_mode_conv))&#xA;    print model.output_shape&#xA;    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))&#xA;    print model.output_shape&#xA;    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))&#xA;    print model.output_shape&#xA;    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))&#xA;    print model.output_shape&#xA;    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))&#xA;    model.add(Flatten())&#xA;    print model.output_shape&#xA;    model.add(Dense(output_dim=32, input_dim=64, init=init_mode,activation=activation_mode))&#xA;    model.add(Dense(output_dim=13, input_dim=50, init=init_mode,activation=activation_mode))&#xA;    model.add(Dense(output_dim=1, input_dim=13, init=init_mode,activation=activation_mode))&#xA;    model.add(Dense(output_dim=1,  init=init_mode, activation=activation_mode))&#xA;    #print model.summary()&#xA;    model.compile(loss='mean_squared_error',optimizer=optimizer_mode)&#xA;&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This model keeps for some reason providing me very bad results.. &#xA;I seem to keep getting an loss of 216, which is nearly 3 times the data range... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did a grid seach to find out which parameter (activation function, init_mode, epochs and batch_size would be best, which are those chosen in the function above (eventhough there wasn't much change in the outcome..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can i do to get better results?&#xA;Is the CNN network poorly designed?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:46:32.483" LastActivityDate="2021-02-16T02:46:32.483" Title="Optimizing CNN network" Tags="|python|keras|regression|convolutional-neural-network|audio-recognition|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17630" PostTypeId="1" CreationDate="2017-03-16T03:13:47.567" Score="2" ViewCount="1116" Body="&lt;p&gt;Say you have a binary classification problem, and a dataset with 20,000 observations and 20 columns. The target variable is very imbalanced, there are are missing values, skewed distributions, outliers, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, in a general sense, what order should these data preprocessing steps be performed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fill in missing values,&#xA;Normalize/standardize data,&#xA;Deal with skewness,&#xA;Deal with outliers,&#xA;Balance target variable classes&lt;/p&gt;&#xA;" OwnerUserId="30035" LastActivityDate="2017-04-15T05:22:29.143" Title="Correct sequence of data prep steps?" Tags="|machine-learning|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17631" PostTypeId="1" CreationDate="2017-03-16T03:52:59.093" Score="2" ViewCount="317" Body="&lt;p&gt;Association rule learning has a fair bit of material based around the correlation of products purchased on the same order/at the same time. However I'd like to discover if there is a method for identifying such a relationship between products that are ordered near each other, but not together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say for example a customer purchases a pencil in week one, but later purchases an eraser in week two. Then a year later they do the same thing. But then also apply this across all customers who show similar behaviour. I would then aim to combine this with the correlation coefficient of products purchased in an order together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I considered giving each of the dates before or after a purchase a diminishing weight (1/n days) and then comparing the aggregate date value of the customer purchase date history with a product A against product B. But this seems like a fairly drawn out method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just looking for a process to achieve this, but python hints would also help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="30038" LastActivityDate="2017-05-15T16:27:40.140" Title="Correlation between products based on purchases placed around the same date" Tags="|correlation|association-rules|market-basket-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17632" PostTypeId="2" ParentId="17630" CreationDate="2017-03-16T04:31:43.240" Score="1" Body="&lt;p&gt;You're asking a complex question that is dependant on what you aim to find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are missing values, are they in columns relevant to the result you're looking for? If not then they needn't be filled at all, so the order does not matter. That being said, if a column with missing values is pertinent to your desired result, then it's often prudent to fill the missing dataset before analysing it further.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, outliers need to be addressed in relation to what you're trying to ascertain. In many cases, presence of the outliers can highlight a result in itself. Which may not answer what you're asking, but instead make you question the accuracy of the data itself, or make you determine to include them or exclude them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to be more specific in what it is you're trying to ascertain. Then approach each of the preprocessing steps in the order that is relevant to you. This changes in many different situations, so it's difficult to give a definitive answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tutorial at kaggle to resolve the Titanic Data Science Solutions lays out a good methodology for approaching such problem, and reinforces the concept of sometimes reorganising workflow tasks.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Workflow stages&lt;/strong&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The competition solution workflow goes through seven stages described in the Data Science Solutions book.&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Question or problem definition.&lt;/li&gt;&#xA;  &lt;li&gt;Acquire training and testing data.&lt;/li&gt;&#xA;  &lt;li&gt;Wrangle, prepare, cleanse the data.&lt;/li&gt;&#xA;  &lt;li&gt;Analyze, identify patterns, and explore the data.&lt;/li&gt;&#xA;  &lt;li&gt;Model, predict and solve the problem.&lt;/li&gt;&#xA;  &lt;li&gt;Visualize, report, and present the problem solving steps and final solution.&lt;/li&gt;&#xA;  &lt;li&gt;Supply or submit the results.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;The workflow indicates general sequence of how each stage may follow the other. However there are use cases with exceptions.&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;We may combine mulitple workflow stages. We may analyze by visualizing data.&lt;/li&gt;&#xA;  &lt;li&gt;&lt;strong&gt;Perform a stage earlier than indicated.&lt;/strong&gt; We may analyze data before and after wrangling.&lt;/li&gt;&#xA;  &lt;li&gt;Perform a stage multiple times in our workflow. Visualize stage may be used multiple times.&lt;/li&gt;&#xA;  &lt;li&gt;Drop a stage altogether. We may not need supply stage to productize or service enable our dataset for a competition.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/startupsci/titanic/titanic-data-science-solutions&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle - Titanic Data Science Solutions&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="30038" LastActivityDate="2017-03-16T04:31:43.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17633" PostTypeId="1" CreationDate="2017-03-16T04:35:28.260" Score="2" ViewCount="423" Body="&lt;p&gt;I'm working on a consulting project for a tech client, and caught myself scratching my head about what the best way to present advanced analytics workflow is. What will be shown to the panel will focus on results, but in this particular case it is warranted to show a visual for what the process behind the scenes is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I need to show the following: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Some raw data file is used as input to a cleaning script which performs imputation and adds/drops variables based on some criteria. Some are added with joins on external files.&lt;br&gt;&#xA;2) The cleaned data is passed to a script which creates variable subsets dependent on remaining granularity and subsets of interest. This outputs many cleaned data sets.&lt;br&gt;&#xA;3) The new data sets are passed to an external computing cluster, which builds a model on each data set, and outputs a table with performance metrics.&lt;br&gt;&#xA;4) Top performing models are re-run locally and score observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am leaning toward UML, as the panel is likely to be familiar with it and appreciate its use. Still, I would like to know if there is any convenient workflow diagramming standard which exists in data science.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Some perspective:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Information systems have historically been well established in diagramming both on a conceptual and hardware level with ERD's and data flow diagrams of all kinds. Software development has also had a heavy emphasis put on this through flowcharts and UML diagrams. Business analytics relies on BPMN to show business processes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when trying to find a diagramming standard in the realm of data science, all I could find were references to the many beautiful and clear visualizations of the findings we make ourselves. We certainly don't shy away from standard methods in visualization (and often for good reason). So, is there such a method for documenting our own workflow, as there is in other fields?&lt;/p&gt;&#xA;" OwnerUserId="30036" LastActivityDate="2017-03-16T04:35:28.260" Title="Diagramming data science workflow?" Tags="|visualization|software-development|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17634" PostTypeId="2" ParentId="17628" CreationDate="2017-03-16T04:55:42.220" Score="1" Body="&lt;p&gt;Looks like I figured out my own question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    git checkout $tag&#xA;		git --work-tree=$outputLoc checkout HEAD -- .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27396" LastActivityDate="2017-03-16T04:55:42.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17635" PostTypeId="1" CreationDate="2017-03-16T05:00:30.263" Score="5" ViewCount="19439" Body="&lt;p&gt;I am new to Machine learning in that Artificial Neural Network. I am using nnet package in R for training of Neural Network. What is the difference between model parameters and hyperparameters? I have heard that the hyper parameters are set before we set the model parameters. What are the hyper parameters and model parameters of a Artificial Neural Network and when they are actually tuned? i.e. Are hyper parameters tuned in Training stage of Neural Network and model parameters in validation stage? &lt;/p&gt;&#xA;" OwnerUserId="30006" LastActivityDate="2017-03-16T12:48:56.530" Title="Model parameters &amp; Hyper parameters of Neural Network &amp; their tuning in training &amp; validation stage" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17636" PostTypeId="1" CreationDate="2017-03-16T06:13:34.790" Score="0" ViewCount="936" Body="&lt;p&gt;I am using nnet package in R for training Artificial Neural&#xA;Network. Some tell that weights are assigned to each case in training set,&#xA;and some say that no. of weights are equal to the no. of connections in &#xA;structure of neural network. I didn't understand this. Can anybody help me &#xA;out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked at the R code of nnet i.e. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between weights &amp;amp; Wts in the nnet code below?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;nnet(x, y, weights, size, Wts, mask, linout = FALSE, entropy = FALSE, softmax = FALSE, censored = FALSE, skip = FALSE, rang = 0.7, decay = 0, maxit = 100, Hess = FALSE, trace = TRUE, MaxNWts = 1000, abstol = 1.0e-4, reltol = 1.0e-8, ...)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="28628" LastEditDate="2017-03-16T14:15:01.500" LastActivityDate="2017-05-15T14:27:39.730" Title="Assigning vector of weights to Neural Network" Tags="|neural-network|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17637" PostTypeId="2" ParentId="11930" CreationDate="2017-03-16T06:44:20.243" Score="0" Body="&lt;p&gt;Hoping is only about issues with portion of image or pixels moving within dimension (mostly) and sometime into other dim (different receptive field) but output remains same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This issue is dealt with invariance or equivariance and looks like weight and age example is easy way to state. Suppose if we are aware of this weight and age hopping we would easily make changes to the algo and get right result. But like data/information hopping, image hopping also happens, if we considered a '4' and a '4' shifted several pixels to the left to be different classes which has different target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Translation Invariance or better equivariance throguh filter this movement or hopping is not much issue though it increases complexity and at the cost of throwing away information, such as location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pls let me know if you need more clarity I will try to.&lt;/p&gt;&#xA;" OwnerUserId="29746" LastEditorUserId="29746" LastEditDate="2017-03-16T08:20:27.197" LastActivityDate="2017-03-16T08:20:27.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17638" PostTypeId="1" AcceptedAnswerId="17653" CreationDate="2017-03-16T07:22:54.090" Score="1" ViewCount="107" Body="&lt;p&gt;I am new to this data mining. Can anyone please help me with an example for Sliding Window Filtering algorithm(SWF) for incremental mining? &lt;/p&gt;&#xA;" OwnerUserId="30039" LastActivityDate="2017-03-16T15:55:42.927" Title="SWF - Incremental mining" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17639" PostTypeId="2" ParentId="16817" CreationDate="2017-03-16T08:08:59.593" Score="1" Body="&lt;p&gt;Text mining is rather a tricky field of machine learning application, since all you've got is &quot;unstructured and semi structured data&quot; and the preprocessing and feature extraction step matters a lot. &lt;a href=&quot;http://www.goodreads.com/book/show/893219.The_Text_Mining_Handbook&quot; rel=&quot;nofollow noreferrer&quot;&gt;The text mining handbook&lt;/a&gt; is a priceless reference in this area of research. but to be specific about your case, I can suggest two answers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;as noted, preprocessing step plays a very important role here. in text mining it is likely to get trapped inside the curse of dimensionality, since you probably have say around 1000 documents but more than 15000 unique words in a dataset. techniques such as stemming and lemmatizing, static and dynamic stopword and punctuation erasing all aim solving this issue. So preprocessing and feature extraction is not an option. it is a MUST&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Naive Bayes model is a linear classifier. Even though it is a very popular algorithm in text classification, there are still risks of rising such problems as yours. the main reason could be that your word-space matrix is highly sparse. you must have paid attention to the fact that in calculating the posterior probability of belonging to a class, Naive Bayes, naively multiplies all single probabilities of &lt;code&gt;P(y|x_i)&lt;/code&gt;. and if there is at least one zero probability, your final answer would be zero, no matter what other inverse observation probabilities are. If you have implemented the algorithm yourself, try already-constructed tools in MATLAB, Python sci-kit learn library, or data mining softwares like KNIME and RapidMiner. they have delicately handled such practical issues in implementing Naive Bayes algorithm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30043" LastActivityDate="2017-03-16T08:08:59.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17640" PostTypeId="1" CreationDate="2017-03-16T08:42:28.730" Score="1" ViewCount="408" Body="&lt;p&gt;Say for example, you have an artificial neural network like the one given below- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m0bb9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m0bb9.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections i.e. See the first connection between first input(top) node and first hidden(top) node. So is it that the connection is assigned first value from the vector of weights,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the second the connection between first(top) input and second(second from top) hidden node is assigned second value from the vector of weights .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, what am I asking is that - Is there any correspondence between the position of the weight in the vector of weights and the connections in the structure. Is there any matrix that is assigned?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30006" LastEditDate="2017-03-21T04:34:42.973" LastActivityDate="2017-03-21T05:53:35.167" Title="How the vector of weights is assigned to a Neural Network" Tags="|neural-network|r|" AnswerCount="1" CommentCount="2" ClosedDate="2017-03-22T16:48:04.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="17641" PostTypeId="1" CreationDate="2017-03-16T09:31:07.317" Score="7" ViewCount="2091" Body="&lt;p&gt;I am trying to port this little piece of R code to python:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;rf &amp;lt;- randomForest(features, proximity = T, oob.prox = T, ntree = 2000)&#xA;dists &amp;lt;- as.dist(1 - rf$proximity)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;with parameters&lt;br /&gt;&#xA;&lt;code&gt;oob.prox&lt;/code&gt;: Should proximity be calculated only on “out-of-bag” data?&lt;br /&gt;&#xA;&lt;code&gt;proximity&lt;/code&gt;: if &lt;code&gt;proximity=TRUE&lt;/code&gt; when &lt;code&gt;randomForest&lt;/code&gt; is called, a matrix of proximity measures among the input (based on the frequency that pairs of data points are in the same terminal nodes).&lt;/p&gt;&#xA;&lt;p&gt;I am currently trying using &lt;code&gt;sklearn.ensemble.RandomTreesEmbedding&lt;/code&gt; for this task, however there is no functionality for the proximity matrix. I found the following developer comment though:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We don't implement proximity matrix in Scikit-Learn (yet).&#xA;However, this could be done by relying on the apply function provided&#xA;in our implementation of decision trees. That is, for all pairs of&#xA;samples in your dataset, iterate over the decision trees in the forest&#xA;(through forest.estimators_) and count the number of times they fall&#xA;in the same leaf, i.e., the number of times apply give the same node&#xA;id for both samples in the pair.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;And so I tried, utilizing numpy's &lt;code&gt;pdist()&lt;/code&gt; function along with my custom distance (or in this case, proximity) measure. I still have several problems:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The proximity function is outstandingly slow&lt;/li&gt;&#xA;&lt;li&gt;How to handle the out-of-bag behaviour&lt;/li&gt;&#xA;&lt;li&gt;How to recreate the exact behaviour of &lt;code&gt;as.dist(1- rf$proximity)&lt;/code&gt;: I think I need to normalize my count matrix, then subtract it from 1 and then afterwards compute the euclidean distances between its rows!?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;My code as of now looks like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# grow a random forest from points&#xA;rf = ensemble.RandomTreesEmbedding(n_estimators=200, &#xA;    random_state=0,&#xA;    max_depth=5&#xA;)&#xA;rfdata = rf.fit_transform(xdata);&#xA;&#xA;&#xA;# define an affinity measure function to use with numpy's pdist&#xA;def treeprox(u, v):&#xA;    leafcount = 0&#xA;    # needs reshaping for single samples&#xA;    u = u.reshape(1,-1)&#xA;    v = v.reshape(1,-1)&#xA;    a = rf.apply(u)&#xA;    b = rf.apply(v)&#xA;    # count number of times they fall in the same leaf &#xA;    # (use of np forces element-wise)&#xA;    c = np.sum(np.array(a)==np.array(b))&#xA;    return c&#xA; &#xA;distm = pdist(xdata, proxfun)&#xA;distm = squareform(distm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;There must be a better way I guess, since this functionality is readily provided by the R package &lt;code&gt;randomForest&lt;/code&gt;.&lt;br /&gt;&#xA;Any suggestions?&lt;br /&gt;&#xA;tia&lt;/p&gt;&#xA;" OwnerUserId="16579" LastEditorUserId="98307" LastEditDate="2020-08-11T12:07:28.443" LastActivityDate="2020-08-11T12:07:28.443" Title="Reproducing randomForest Proximity Matrix from R package in Python" Tags="|python|r|scikit-learn|random-forest|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="17642" PostTypeId="2" ParentId="17635" CreationDate="2017-03-16T11:06:19.720" Score="2" Body="&lt;p&gt;I'd characterize model parameters as the architectural choices of the neural net, i.e. how many layers, the number of nodes per layer, the type of unit (sigmoid, tanh etc.), whereas hyperparameters are things such as the learning rate, momentum, regularization coefficient and such like. The hyperparameters need to be tuned during training for any given neural net architecture. &lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2017-03-16T11:06:19.720" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17643" PostTypeId="2" ParentId="17635" CreationDate="2017-03-16T11:12:19.540" Score="4" Body="&lt;p&gt;The parameters of a neural network are typically the weights of the connections. In this case, these parameters are learned during the training stage. So, the algorithm itself (and the input data) tunes these parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hyper parameters are typically the learning rate, the batch size or the number of epochs. The are so called &quot;hyper&quot; because they influence how your parameters will be learned. You optimize these hyper parameters as you want (depends on your possibilities): grid search, random search, by hand, using visualisations... The validation stage help you to both know if your parameters have been learned enough and know if your hyper parameters are good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know more about hyper parameters and parameters in general in machine learning, look for &quot;deep learning versus shallow learning&quot;.     &lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-03-16T12:48:56.530" LastActivityDate="2017-03-16T12:48:56.530" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17644" PostTypeId="2" ParentId="17636" CreationDate="2017-03-16T14:14:22.420" Score="2" Body="&lt;p&gt;From the &lt;a href=&quot;https://cran.r-project.org/web/packages/nnet/nnet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; we can find out that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;weights&lt;/code&gt;: (case) weights for each example – if missing defaults to 1.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Wts&lt;/code&gt;: initial parameter vector. If missing chosen at random.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Though I'm no R expert, my understanding is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;weights&lt;/code&gt; is something like &lt;code&gt;sample_weight&lt;/code&gt; in python, which controls the weight of each sample. It's useful when dealing with imbalanced class issue.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Wts&lt;/code&gt; is the initial weights of neural network. Random weights will be initiated if this parameter is not passed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-16T14:14:22.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17645" PostTypeId="1" AcceptedAnswerId="17651" CreationDate="2017-03-16T14:44:12.933" Score="5" ViewCount="6952" Body="&lt;p&gt;I come to this question as I read the use of PCA to reduce overfitting is a bad practice. That is because PCA does not consider labels/output classes and so Regularization is always preferred. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That seems purely valid in Supervised Learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the case for &lt;strong&gt;Unsupervised Learning&lt;/strong&gt;?&#xA;We don't have any labels whatsoever. &#xA;So 2 questions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is overfitting a problem in Unsupervised learning?&lt;/li&gt;&#xA;&lt;li&gt;If yes, Can we use PCA to prevent overfitting?Is that a good practice?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23773" LastEditorUserId="23773" LastEditDate="2017-03-16T15:14:47.240" LastActivityDate="2017-03-21T17:36:14.723" Title="Is Overfitting a problem in Unsupervised learning?" Tags="|machine-learning|unsupervised-learning|pca|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17648" PostTypeId="1" AcceptedAnswerId="18741" CreationDate="2017-03-16T14:56:23.027" Score="1" ViewCount="148" Body="&lt;p&gt;Crossposted from &lt;a href=&quot;https://stackoverflow.com/q/42809054/9214357&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;This question is to ask for directions/suggestions/help on the use of deepmind opensource libraries: &lt;a href=&quot;https://github.com/deepmind/lab&quot; rel=&quot;nofollow noreferrer&quot;&gt;DeepMind Lab&lt;/a&gt; or &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow&lt;/a&gt; in Python.&lt;/p&gt;&#xA;&lt;p&gt;Consider that I'm new to concepts like deep learning and AI.&lt;/p&gt;&#xA;&lt;p&gt;Questions are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there examples on using Deepmind or Tensorflow for math problems where I need to observe values and take actions?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Using an approach similar to to the one described in &lt;a href=&quot;https://deepmind.com/blog/open-sourcing-deepmind-lab/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page&lt;/a&gt; based on observations, actions, rewards, etc., I'd like to call a learing agent to choose among some values. I was thinking to something like this:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input: a list of list of tuple (the list will change at each step)&lt;/li&gt;&#xA;&lt;li&gt;Action: pick up a value from the input (based on experience)&lt;/li&gt;&#xA;&lt;li&gt;Reward: If the value it returned was good or bad for the rest of the algorithm I'm implementing, I'll reward the deep learning agent.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Additional notes:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I cannot train the algorith in advance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The Input is something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;edge: (1, 2), face_down: 4, face_up: 5, face_left: 4, face_right: 5&#xA;edge: (0, 1), face_down: 4, face_up: 4, face_left: 4, face_right: 5&#xA;edge: (5, 4), face_down: 4, face_up: 5, face_left: 4, face_right: 5&#xA;edge: (6, 7), face_down: 3, face_up: 5, face_left: 5, face_right: 5&#xA;edge: (3, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4&#xA;edge: (4, 1), face_down: 4, face_up: 5, face_left: 5, face_right: 4&#xA;edge: (8, 5), face_down: 4, face_up: 5, face_left: 5, face_right: 4&#xA;edge: (3, 8), face_down: 4, face_up: 5, face_left: 4, face_right: 5&#xA;edge: (2, 3), face_down: 4, face_up: 5, face_left: 5, face_right: 4&#xA;edge: (5, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4&#xA;edge: (0, 5), face_down: 4, face_up: 4, face_left: 4, face_right: 5&#xA;edge: (1, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4&#xA;edge: (9, 6), face_down: 3, face_up: 5, face_left: 5, face_right: 5&#xA;edge: (0, 3), face_down: 4, face_up: 4, face_left: 4, face_right: 5&#xA;edge: (7, 9), face_down: 3, face_up: 5, face_left: 5, face_right: 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The idea is to use the same aproach deepmind uses to play games, but instead of analyzing pixels and use the pad (up, dowm, left, right, fire, jump), is to let the learning agent to analyze some math values and, as the only action, to select one of them.&lt;/p&gt;&#xA;&lt;p&gt;Are there other approaches or libraries/framework to address such a problem?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wygvK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wygvK.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18305" LastEditorUserId="98307" LastEditDate="2020-08-11T14:52:42.340" LastActivityDate="2020-08-11T16:13:15.740" Title="Tensorflow / Deepmind: how do I take actions from observations for math algorithms related to proofs?" Tags="|python|deep-learning|tensorflow|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17649" PostTypeId="2" ParentId="17594" CreationDate="2017-03-16T14:59:04.807" Score="1" Body="&lt;p&gt;After reading the &lt;a href=&quot;https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala#L79&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;, I find out that the problem is nothing related to your memory, but the limit of integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than &lt;code&gt;Integer.MAX_VALUE&lt;/code&gt;, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968&amp;gt;2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888&amp;gt;2147483647$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big! &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-16T14:59:04.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17650" PostTypeId="1" CreationDate="2017-03-16T15:06:55.983" Score="4" ViewCount="2737" Body="&lt;p&gt;I am trying to train a computer to play with LEGO bricks (simulated) using DQN.&#xA;My input is an image with 4 color channels (RGB and depth) and the output of the neural network is the coordinates of the best position where the next brick should be put. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of having $m_x * m_y * m_z$ (where $m_k$ is the size of positions along the k axis) different outputs, I want to have 3 output layers, each of which returns the x, y or z coordinates of the best position, so I have only $m_x + m_y + m_z$ outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is replacing the fully-connected output layer by three fully-connected output layers, but only have one reward value (labeled value) for three layers. Can I do this? Will the three layers returns different outputs if the reward is same?&lt;/p&gt;&#xA;" OwnerUserId="30054" LastEditorUserId="31513" LastEditDate="2017-05-10T08:05:28.360" LastActivityDate="2018-02-04T17:40:03.323" Title="Multiple Output Layers in Neural Networks in Deep Q Learning" Tags="|machine-learning|neural-network|deep-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17651" PostTypeId="2" ParentId="17645" CreationDate="2017-03-16T15:11:17.427" Score="8" Body="&lt;p&gt;Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the &lt;em&gt;specific&lt;/em&gt; sample rather than producing a more &lt;em&gt;general&lt;/em&gt; model of the phenomena or underlying process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be presented using &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;noreferrer&quot;&gt;Bayesian methods&lt;/a&gt;. If I use &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier&quot; rel=&quot;noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; then I have a simple model that might not fit either the dataset or the distribution too well but of low complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that we use a very large Bayesian network. It might end up not being able to gain more insight and use the complexity to model the dataset (or even just trash).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, overfitting is possible in unsupervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In PCA we start with a model in the size of the dataset. We have assumptions about the way the data behaves and use them to reduce the model size by removing parts which don't explain the main factors of variation. Since we reduce the model size one could expect to benefit always.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we face problems.&#xA;First, a model in the size of the dataset is extremely large (given such a large size you can model any dataset). Compressing it a bit is not enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem is that it is possible that our assumptions are not correct. Then we will have a smaller model be it won't be aligned with the distribution. Again, it this case it might overfit or just not fit the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though that, PCA is aimed to reduce the dimensionality, what lead to a smaller model and possibly reduce the chance of overfitting. So, in case that the distribution fits the PCA assumptions, it should help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, overfitting is possible in unsupervised learning too. PCA might help with it, on a suitable data.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="6478" LastEditDate="2017-03-17T13:47:42.163" LastActivityDate="2017-03-17T13:47:42.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17652" PostTypeId="2" ParentId="17631" CreationDate="2017-03-16T15:42:18.480" Score="1" Body="&lt;p&gt;Association rule mining is nearly always done with the apriori or eclat algorithms. There are a few papers which have utilized these in the time series context. &lt;a href=&quot;https://archive.org/details/RBB005&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; paper details a thorough implementation, though this is done with relation to identifying gene patterns, rather than customer purchases. The idea is still the same though, only there is slightly less complexity in your case. Unfortunately, this does not give a detailed Python implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your situation (this is unlikely to be applicable from your description, but I list this just in case...), it may be useful to break from association rules and cluster your data instead, provided you have enough of it to treat it as continuous. A Python implementation of this is featured in &lt;a href=&quot;http://alexminnaar.com/time-series-classification-and-clustering-with-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="30036" LastActivityDate="2017-03-16T15:42:18.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17653" PostTypeId="2" ParentId="17638" CreationDate="2017-03-16T15:55:42.927" Score="1" Body="&lt;p&gt;There are pseudo-codes in this &lt;a href=&quot;http://arbor.ee.ntu.edu.tw/~mschen/paperps/cikm414.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;, Section 3.1 shows an example of incremental mining by this algorithm. However I can't find any projects or working source codes with this algorithm implemented.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-16T15:55:42.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17654" PostTypeId="1" CreationDate="2017-03-16T16:02:18.020" Score="6" ViewCount="3231" Body="&lt;p&gt;I have a dataset of microscope images and I want to train a ML/DL algorithm to perform binary classification. The &lt;strong&gt;positive&lt;/strong&gt; class is when there is &lt;em&gt;only one&lt;/em&gt; cell in the image, and the &lt;strong&gt;negative&lt;/strong&gt; class is &lt;em&gt;everything else&lt;/em&gt; (i.e. when there are either more than 1 cells, or no cells at all).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is one of the original images. (there is a cell in the curved site on the center of the image)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mCzz0m.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mCzz0m.jpg&quot; alt=&quot;original image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the big size of the images (2048x2048) and the excess of information (the cells can only be in the tube-system), I decided to preprocess them. So, I set everything outside the tube-system to 0 (black) and I crop all the images to the boundaries I got by averaging the images of the whole dataset. Below you can see the end result. (there are 2 cells in the tube, one in the center and one at the upper left part)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rLrzEm.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rLrzEm.jpg&quot; alt=&quot;preprocessed image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I tried to train a CNN (using python and TensorFlow). I played a few times by changing its hyperparameters, but I had no luck. I think the problem is that the cells (region of interest) are occupying a very small portion of the image, which makes it hard for the algorithm to focus on. To make things worse, there are random dust particles around which make the image noisy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any ideas of how I could perform a robust binary classification in such a dataset?&lt;/p&gt;&#xA;" OwnerUserId="30051" LastEditorUserId="29169" LastEditDate="2021-02-16T02:46:20.260" LastActivityDate="2021-02-16T02:46:20.260" Title="Binary classification of similar images with small region of interest" Tags="|machine-learning|python|tensorflow|image-classification|convolutional-neural-network|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17655" PostTypeId="1" AcceptedAnswerId="17658" CreationDate="2017-03-16T16:08:37.383" Score="2" ViewCount="107" Body="&lt;p&gt;I am looking for an algorithm that allows me the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a webpage and I want to randomly show one &quot;content&quot; from a list of contents on that webpage depending on who (visitor) sees the webpage. I know my visitors' demographic features like age, gender, locale. I assume that visitors with similar demographics have a similar taste regarding content on my website. I also know that they liked my content when they share it in the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A list of available contents, let's say: red, green, blue, purple,&#xA;violet&lt;/li&gt;&#xA;&lt;li&gt;A constant stream of events of visitors with specific&#xA;demographic data that share a content&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I want:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First of all, all contents should be displayed completely randomly.&#xA;Each user should randomly get one content without any preference&lt;/li&gt;&#xA;&lt;li&gt;As soon as the first user with a specific demographic shares the content&#xA;they got I want other visitors with similar demographic features that&#xA;end up on my webpage to see this specific content with a higher&#xA;probability.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So basically I want a self optimizing system that learns in realtime.&lt;/p&gt;&#xA;" OwnerUserId="30057" LastEditorDisplayName="user380" LastEditDate="2017-07-13T15:48:49.323" LastActivityDate="2017-07-13T15:48:49.323" Title="Finding the perfect algorithm for realtime optimizing of content" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17656" PostTypeId="2" ParentId="17620" CreationDate="2017-03-16T16:09:47.363" Score="4" Body="&lt;p&gt;&lt;code&gt;RidgeCV&lt;/code&gt; implements CV by &lt;code&gt;GridSearchCV&lt;/code&gt;, which supports custom scorer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, &lt;code&gt;LassoCV&lt;/code&gt; and &lt;code&gt;ElasticNetCV&lt;/code&gt; implement CV by &lt;code&gt;LinearModelCV&lt;/code&gt;, with MSE as scoring hard-coded. &lt;code&gt;LassoLarsCV&lt;/code&gt; impletments CV by &lt;code&gt;LarsCV&lt;/code&gt;, with MSE as scoring hard-coded too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use other scorers, maybe you can use &lt;code&gt;GridSearchCV&lt;/code&gt; directly.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/ridge.py#L1092&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/ridge.py#L1092&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/coordinate_descent.py#L1181&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/coordinate_descent.py#L1181&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/least_angle.py#L1143&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/least_angle.py#L1143&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-16T16:09:47.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17657" PostTypeId="2" ParentId="17559" CreationDate="2017-03-16T16:21:42.797" Score="4" Body="&lt;p&gt;Solution to my question: If you run a tensorflow session you can choose if the placement of nodes on the different devices is hard or soft coded. In the specific case of CIFAR10 this means:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The Queues be placed on the GPU and that's why the error popped up, but:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you turn on soft placement it works. This means, that even though the placement on the GPU of several parts of the model can't be done, Tensorflow solves this problem automaticly by placing them somewhere else&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I don't actually know where the nodes get placed. I guess on the CPU, but I'm not sure. In tensorboard the color of the input pipeline nodes claims a placement on the GPU. Very confusing!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The soft placement can be done by starting the session like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I found this on the following tutorial page: &lt;a href=&quot;https://www.tensorflow.org/tutorials/using_gpu&quot; rel=&quot;noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/using_gpu&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2017-03-16T16:21:42.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17658" PostTypeId="2" ParentId="17655" CreationDate="2017-03-16T19:55:31.293" Score="2" Body="&lt;p&gt;This sounds like a classic use for a &lt;a href=&quot;http://hunch.net/?p=298&quot; rel=&quot;nofollow noreferrer&quot;&gt;contextual bandit&lt;/a&gt; solver.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence you can run a simple online model (pretty much any regression model, or even a simple classifier like logistic regression if your reward signal is binary success/fail such as in your case) that learns to associate your demographic data with &lt;em&gt;expected reward&lt;/em&gt; from each possible action - for you the reward can simply be 1 for a share link created or 0 for no share link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst the model is learning, you select the next action according to predicted reward from the model. There are choices between different workable strategies. For instance you could use an $\epsilon$-greedy approach: Pick the action with maximum predicted expected reward (or randomly choose between shared maximum values), but sometimes - with probability $\epsilon$ - you choose random content. There are other approaches and options that you can discover by researching contextual bandits and the simpler multi-armed bandit problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, you could use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;logistic regression&lt;/a&gt; model to predict expected reward from user demographics, with one such model per possible action. For a version that picks evenly to start, but prefers items that have been shared more over time, you can use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Boltzmann_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boltzmann distribution (also called Gibbs distribution)&lt;/a&gt; using the predicted rewards as the inverse &quot;energies&quot; for the actions, and lowering the temperature as you collect more data. You can also initialise the weights of your model to predict a small but optimistic positive reward to start with to encourage early exploration. Whenever a user views your page, you pick the action to take based on the predicted rewards, and afterwards take the user response (share or not share) as feedback to update the one model associated with that action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example, the logistic regression learning rate, temperature scheme and starting reward are hyper-parameters of your model, and you use them to trade off responsiveness to individual events versus long-term accuracy for selecting the best action.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-16T21:18:26.477" LastActivityDate="2017-03-16T21:18:26.477" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17660" PostTypeId="2" ParentId="17565" CreationDate="2017-03-16T21:13:22.113" Score="0" Body="&lt;p&gt;Actually, i have finally made it work. I dont know, what exactly was the crucial change, but here's the extract of my changes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;replaced all &lt;code&gt;np.multiply&lt;/code&gt; with &lt;code&gt;*&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decreased learning rate and regulizers&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Applied &lt;code&gt;np.nan_to_num&lt;/code&gt; to exponents&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So here is the final code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def lossf(w, X, y, l1, l2):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    lossf1 = np.sum(ss.log1p(1 + np.nan_to_num(ss.expm1(-y * np.dot(X, w)))))&#xA;    lossf2 = l2 * (np.dot(np.transpose(w), w))&#xA;    lossf3 = l1 * sum(abs(w))&#xA;    lossf = np.float(lossf1 + lossf2 + lossf3)&#xA;    return lossf&#xA;&#xA;def gradf(w, X, y, l1, l2):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    gradw1 = l2 * 2 * w &#xA;    gradw2 = l1 * np.sign(w)&#xA;    gradw3 = -y * (1 + np.nan_to_num(ss.expm1(-y * np.dot(X, w))))&#xA;    gradw3 = gradw3 / (2 + np.nan_to_num(ss.expm1(-y * np.dot(X, w))))&#xA;    gradw3 = np.sum(gradw3 * X, axis=0)&#xA;    gradw3.resize(gradw3.shape[0],1)&#xA;    gradw = gradw1 + gradw2 + gradw3&#xA;    gradw.resize(gradw.shape[0],)&#xA;    return np.transpose(gradw)&#xA;class LR(ClassifierMixin, BaseEstimator):&#xA;    def __init__(self, lr=0.000001, l1=0.1, l2=0.1, num_iter=100, verbose=0):&#xA;        self.l1 = l1&#xA;        self.l2 = l2&#xA;        self.w = None&#xA;        self.lr = lr&#xA;        self.verbose = verbose&#xA;        self.num_iter = num_iter&#xA;&#xA;    def fit(self, X, y):       &#xA;        n, d = X.shape &#xA;        self.w = np.zeros(shape=(d,))&#xA;        for i in range(self.num_iter):&#xA;            print &quot;\n&quot;, &quot;Iteration &quot;, i&#xA;            g = gradf(self.w, X, y, self.l1, self.l2)&#xA;            g.resize((g.shape[0],1))&#xA;            self.w = self.w - g&#xA;            print &quot;Loss: &quot;, lossf(self.w, X, y, self.l1, self.l2)&#xA;        return self&#xA;&#xA;    def predict_proba(self, X):&#xA;        probs = 1/(2 + ss.expm1(np.dot(-X, self.w)))&#xA;        return probs &#xA;&#xA;    def predict(self, X):&#xA;        probs = self.predict_proba(X)&#xA;        probs = np.sign(2 * probs - 1)&#xA;        probs.resize((probs.shape[0],))&#xA;        return probs &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29931" LastActivityDate="2017-03-16T21:13:22.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17661" PostTypeId="1" CreationDate="2017-03-16T22:37:36.680" Score="1" ViewCount="818" Body="&lt;p&gt;When training a CNN one option is either to zero pad an image to make it bigger or upsample it. When should I choose each one?&lt;/p&gt;&#xA;&lt;p&gt;What criteria is leveraged for choosing a method?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:46:07.733" LastActivityDate="2024-04-03T03:05:50.657" Title="What is preferred upsampling or zero padding?" Tags="|machine-learning|deep-learning|neural-network|cnn|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17662" PostTypeId="1" CreationDate="2017-03-16T22:44:05.003" Score="2" ViewCount="611" Body="&lt;p&gt;I've been attempting to do multi-step ahead prediction with the NARX (Non-Linear with Exogenous Inputs) Neural Network. As I understand it, this network can be defined mathematically as follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y(t+1) = F(y(n), ..., y(n-q+1), u(n), ..., u(n-q+1))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where the size of the tapped delay line memory is represented by $q$. So in order to do a 1-step ahead prediction, the network simply takes as input the current and past values of the exogenous function $u(n)$ as well as the current and past values of the time-series I am interested on $y(t)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that I want to predict all the values from $y(t+1)$ to $y(t+10)$, how would I go about doing this? It seems that I need the &lt;strong&gt;current&lt;/strong&gt; value of the exogenous time-series $u(n)$ at each time-step however this is not available to me. Am I missing something here? I would greatly appreciate if anyone can point me in the right direction. &lt;/p&gt;&#xA;" OwnerUserId="30068" LastActivityDate="2017-03-16T22:44:05.003" Title="NARX Network Multi-Step Forecasting Question" Tags="|machine-learning|neural-network|prediction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17663" PostTypeId="2" ParentId="15094" CreationDate="2017-03-17T00:22:25.083" Score="1" Body="&lt;p&gt;For initial learning you could very easily do proof of concept work against individual VMs (a 4GB VM with a pseudo cluster is enough to do basic mapreduce examples in). If you're going to use Spark I would lean towards higher memory counts if they're within budget. I'd also keep an eye towards more, lower cost nodes. A stack of Intel i7 NUCs (or similar) with 2TB consumer SSDs and 32GB of RAM will cost less than $1k per node and 10-15 of them could easily handle a class of 24 students.&lt;/p&gt;&#xA;" OwnerUserId="30071" LastActivityDate="2017-03-17T00:22:25.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17664" PostTypeId="1" AcceptedAnswerId="17667" CreationDate="2017-03-17T03:56:38.540" Score="1" ViewCount="1097" Body="&lt;p&gt;My task involves a POS Tagging using HMM. I am given a training data set (word/tag). I have to write a file with transition probabilities and emission probabilities. I am currently using a nested dictionary of the form &lt;code&gt;{State1: {State2: count, State3 :count}}&lt;/code&gt;. However, while calculating the probabilities now via the counts in nested dict, my program is running very slow for mid size files (e.g. 2000 sentences and tags) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to store a HMM in python? For my project, I cannot use any external library that already does this, I must use standard python libraries. &lt;/p&gt;&#xA;" OwnerUserId="30075" LastEditorUserId="836" LastEditDate="2017-03-17T10:31:40.820" LastActivityDate="2017-03-17T10:36:07.680" Title="How do I represent a hidden markov model in data structure?" Tags="|python|dataset|nlp|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17665" PostTypeId="2" ParentId="17654" CreationDate="2017-03-17T05:12:52.453" Score="2" Body="&lt;p&gt;I would use a CLAHE preprocessing, SIFT Image features and mask the wrong detected keypoints out as geometric constrains. then i would count sift keypoints in a image without and keypoints to get a thredhold boundary and classify by amount of robust keypoints or logistic regression &lt;/p&gt;&#xA;" OwnerUserId="29723" LastEditorUserId="29723" LastEditDate="2017-03-17T06:16:20.767" LastActivityDate="2017-03-17T06:16:20.767" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17667" PostTypeId="2" ParentId="17664" CreationDate="2017-03-17T08:06:03.160" Score="0" Body="&lt;p&gt;With 29 states and 841 possible transitions to track whilst reading a file with 2000 entries (word, tag), then you should not be experiencing a speed problem when using a dictionary of dictionaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming your data structure as described called &lt;code&gt;transition_counts&lt;/code&gt;, and receiving data in pairs, &lt;code&gt;(this_pos, next_pos)&lt;/code&gt; then running 2000 times:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transition_counts[this_pos][next_pos] += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;takes only a fraction of a second. This is similar for code that calculates $p(POS_{t+1}|POS_t)$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;total_from_pos_t =  sum(transition_counts[pos_t].values())&#xA;prob_pos_tplus_one = transition_counts[pos_t][pos_tplus_one] / total_from_pos_t  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is very fast. Your problem is not with the representation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-17T10:36:07.680" LastActivityDate="2017-03-17T10:36:07.680" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17668" PostTypeId="2" ParentId="17654" CreationDate="2017-03-17T09:22:57.610" Score="2" Body="&lt;p&gt;I would still stick with using a CNN for that specific application. Think about CNNs being used to detect various types of cancer in noisy images with an insane precision (&lt;a href=&quot;http://www.nature.com/articles/nature21056.epdf?referrer_access_token=RcIKxkNHJfvxwTkuTW4lPNRgN0jAjWel9jnR3ZoTv0NXpMHRAJy8Qn10ys2O4tuPxB4B9GCLPvFTfGPu3BrO0euO-oKmEt01tc_3Bad0Edx-p21z_GXJAeVxTVS6U_o7mmt1TxNO3XcR6ZK9ofxEEeFaRl3oGpNMQBIFWEz9lVYs1gWvSGHUzq1_WTPh2nfp3Rx8zKAPHxpbvedE5qWe3w7F8nyvpmzbjlR_EbOOvaY%3D&amp;amp;tracking_referrer=www.wired.co.uk&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford&lt;/a&gt;, &lt;a href=&quot;https://edgylabs.com/2017/03/10/google-ai-cancer-diagnosis/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google&lt;/a&gt;). This type of input is actually very similar to yours with cancer cells hiding in a cluster of healthy ones. And yet the models are performing as good as cancer experts in some cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN have shown to work best when trained with a HUGE amount of data. If possible try to provide more training data with a decent class distribution (roughly the same number of positive and negative examples).&#xA;Moreover, apart from tuning hyperparameters you could also experiment with different CNN architectures. You will fin plenty of inspiration in the &lt;a href=&quot;https://scholar.google.com/scholar?q=cancer%20detection%20convolutional%20neural%20network&amp;amp;hl=en&amp;amp;as_sdt=0&amp;amp;as_vis=1&amp;amp;oi=scholart&amp;amp;sa=X&amp;amp;ved=0ahUKEwidrq33ld3SAhXJAxoKHY6DDR0QgQMIGDAA&quot; rel=&quot;nofollow noreferrer&quot;&gt;litterature&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-03-17T09:22:57.610" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17670" PostTypeId="1" CreationDate="2017-03-17T14:00:19.510" Score="1" ViewCount="441" Body="&lt;p&gt;I am trying to estimate parameters from a three-parameter Log-logistic distribution in R. I have a code as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nloglike &amp;lt;- function(p,y) {&#xA; n &amp;lt;- length(y)&#xA; alpha &amp;lt;- p[1];   beta &amp;lt;-p[2];gamma &amp;lt;-p[3];&#xA;if (alpha &amp;gt; 0 &amp;amp;&amp;amp; y &amp;gt; gamma &amp;amp;&amp;amp; beta &amp;gt;= 1)&#xA;{logl &amp;lt;- sum(log((beta/alpha)*((y-gamma)/alpha)^(beta - 1))-log(1+((y-&#xA; gamma)/alpha)^beta))^2}  &#xA; else { logl &amp;lt;- 10000 } &#xA;return((logl))}&#xA;&#xA;yy&amp;lt;-exp(data[,2])&#xA;sol&amp;lt;-optim(p=c(1,1,0),nloglike,method =&quot;BFGS&quot;,y=yy,control=list(trace=0))$par&#xA;print(sol)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code does not throw any errors, but the output is far different from what's expected. I would like to know what to consider when choosing the initial / starting values from the optim command. i.e. p=c(1,1,0)&lt;/p&gt;&#xA;" OwnerUserId="30089" LastActivityDate="2017-03-17T14:00:19.510" Title="Maximum likelihood Estimation of three-parameter log-logistic distribution in R" Tags="|r|distribution|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17671" PostTypeId="1" AcceptedAnswerId="17708" CreationDate="2017-03-17T14:06:04.600" Score="6" ViewCount="7226" Body="&lt;p&gt;What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.&lt;/p&gt;&#xA;&lt;p&gt;Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in &lt;a href=&quot;https://datascience.stackexchange.com/questions/11853&quot;&gt;Question about bias in Convolutional Networks&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Total number of weights in the layer kernel trivially equals to &lt;span class=&quot;math-container&quot;&gt;$3 \times 5 \times 5 \times 32$&lt;/span&gt;.&#xA;Now let us count biases. The link above states that total count of biases is &lt;span class=&quot;math-container&quot;&gt;$1 \times 32$&lt;/span&gt;, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.&lt;/p&gt;&#xA;&lt;p&gt;But from the other side: we apply activation function to each cell of output feature map separately, so if we will have &lt;strong&gt;different&lt;/strong&gt; bias for each cell, they do not sum together, so the number &lt;span class=&quot;math-container&quot;&gt;$0 \times 0 \times 32$&lt;/span&gt; instead of &lt;span class=&quot;math-container&quot;&gt;$1 \times 32$&lt;/span&gt; makes sense too (here &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; is the output feature map  height or width).&lt;/p&gt;&#xA;&lt;p&gt;As I can see, first approach is widely used, but I also saw the second approach in some papers.&lt;/p&gt;&#xA;&lt;p&gt;So, (&lt;span class=&quot;math-container&quot;&gt;$3 \times 5 \times 5 + 1) \times 32$&lt;/span&gt; or  &lt;span class=&quot;math-container&quot;&gt;$(3 \times 5 \times 5 + 0 \times 0) \times 32$&lt;/span&gt;?&lt;/p&gt;&#xA;" OwnerUserId="29891" LastEditorUserId="126607" LastEditDate="2021-10-18T19:55:20.417" LastActivityDate="2021-10-18T19:55:20.417" Title="Correct number of biases in CNN" Tags="|deep-learning|convolutional-neural-network|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17672" PostTypeId="2" ParentId="17671" CreationDate="2017-03-17T15:32:52.553" Score="1" Body="&lt;p&gt;When i tried to output my CNN weights from theano's grapgh, i got one bias vector for each layer. &lt;/p&gt;&#xA;" OwnerUserId="29931" LastActivityDate="2017-03-17T15:32:52.553" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17673" PostTypeId="1" CreationDate="2017-03-17T16:44:11.403" Score="2" ViewCount="1442" Body="&lt;p&gt;I am facing some issues with a text classification problem and need your help to understand the best way to approach it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem statement is as follows: &#xA;Assume I have a set of sentences describing a certain product (assume they are furniture items). Each of the sentences needs to be tagged with a class coming from a roster of 10 possible classes. The 10 classes refer to possible attributes of the item, such as softness, brightness etc. &#xA;On top of this, sentences contain sentiment information on each item (good, bad, neutral). Example: &quot;I am satisfied with the performance of this couch, it is really soft&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question is: &lt;strong&gt;how would you approach the problem of labeling each sentence with both classes AND sentiment?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, to make things even more complex, assume each sentence can describe up to 3 items, with relative sentiment. &#xA;&lt;strong&gt;How would you approach the multi-ouput sentiment+class problem in this case?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="30094" LastActivityDate="2018-04-18T19:02:56.247" Title="Multi-ouput Classification + Sentiment Analysis" Tags="|machine-learning|classification|multiclass-classification|multilabel-classification|sentiment-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17674" PostTypeId="2" ParentId="17673" CreationDate="2017-03-17T17:28:09.907" Score="2" Body="&lt;h1&gt;The &quot;low hanging fruit&quot; approach is &quot;bag of words&quot;.&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stanfordnlp.github.io/CoreNLP/lemma.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lemmatize the sentence&lt;/a&gt; and use the lemma frequencies as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Build &lt;strong&gt;two&lt;/strong&gt; models:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sentiment (or reuse &lt;a href=&quot;http://stanfordnlp.github.io/CoreNLP/sentiment.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;StanfordCoreNLP&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Topic (for this you need a training sample - a set of sentences labeled with the topic).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now you can map every sentence to topic and sentiment and this will tell you whether the customer &lt;em&gt;liked or disliked&lt;/em&gt; (sentiment) the &lt;em&gt;softness&lt;/em&gt; (topic).&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Multi-topic sentences&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Usually a sentence has a single topic, but, if you do require multiple topics per sentence, you can use &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html#module-sklearn.multiclass&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;predict_proba&lt;/code&gt;&lt;/a&gt; or similar and use all classes with probability over, say, 0.3.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Class proliferation&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The above approach fails on sentences like&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I liked the softness but hated the design.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;because the total (average) sentiment of the sentence is probably &lt;code&gt;neutral&lt;/code&gt; and we miss out &lt;code&gt;softness-good&lt;/code&gt; and &lt;code&gt;design-bad&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is thus tempting to &lt;strong&gt;&lt;em&gt;triple&lt;/em&gt;&lt;/strong&gt; the number of classes (&lt;code&gt;topic-sentiment&lt;/code&gt;) to accommodate reviews like&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The table combines pleasant design, flimsy build and neutral color.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The problem is that building models for very many classes requires a lot of training data. IOW, you will need to manually annotate a &lt;em&gt;lot&lt;/em&gt; of reviews - and even then the model quality if unlikely to meet your expectations.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Real NLP&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If you really need to handle multi-topic sentences, you might want to consider a &quot;deeper&quot; approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, take a look at the &lt;a href=&quot;http://stanfordnlp.github.io/CoreNLP/parse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;parse tree&lt;/a&gt; of the sentence.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;split&lt;/em&gt; it into constituent phrases&lt;/li&gt;&#xA;&lt;li&gt;use the &lt;em&gt;topic&lt;/em&gt; model to identify the &lt;em&gt;unique&lt;/em&gt; topic of the each phrase &lt;/li&gt;&#xA;&lt;li&gt;use the &lt;em&gt;sentiment&lt;/em&gt; model on the phrase&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9867" LastEditorUserId="9867" LastEditDate="2018-04-18T19:02:56.247" LastActivityDate="2018-04-18T19:02:56.247" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17675" PostTypeId="1" CreationDate="2017-03-17T18:45:16.867" Score="10" ViewCount="1673" Body="&lt;p&gt;As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central server or location where all of us can access (which means unix permissions etc). For models I also tend to send a weights file over to my colleague and share my github. Both I've found pretty cumbersome. What have some of you done? &lt;/p&gt;&#xA;" OwnerUserId="30099" LastActivityDate="2020-11-29T21:46:14.057" Title="What are some of the best practices for sharing data and models with colleagues?" Tags="|machine-learning|predictive-modeling|dataset|data|model-selection|" AnswerCount="6" CommentCount="3" CommunityOwnedDate="2017-03-23T18:33:17.413" ContentLicense="CC BY-SA 3.0" />
  <row Id="17676" PostTypeId="1" CreationDate="2017-03-17T19:36:58.347" Score="2" ViewCount="323" Body="&lt;p&gt;I'm trying to understand the DHP(Direct Hashing and Pruning) algorithm and I got stuck at explaining the selection of modulo number.&#xA;The &lt;a href=&quot;http://user.it.uu.se/~kostis/Teaching/DM-01/Handouts/PCY.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; shows an example of using the hash function at page 7: &lt;code&gt;h{{x y}) = ((order of x)*10 + (order of y)) mod 7&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are the basis to define the function this way? &lt;/li&gt;&#xA;&lt;li&gt;How is the modulo number selected(7, in this example)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kLjpI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kLjpI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="3179" LastEditorUserId="3179" LastEditDate="2017-03-18T05:11:38.183" LastActivityDate="2024-03-02T06:07:00.770" Title="How is the modulo number selected to build the hash table in DHP algorithm?" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17677" PostTypeId="1" CreationDate="2017-03-17T20:17:19.553" Score="1" ViewCount="746" Body="&lt;p&gt;I am constructing a neural network using Sigmoid neurons. My current network is  3 layers of 39 nodes each, using an input of 39 input variables across 10,000 observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am calculating loss by summing the loss in each observation. That being said, I have to numerically calculate $39^2*3*10,000$ weight deltas in the network for every iteration of the code using backpropagation, starting from the output neuron and working backwards. This takes a long time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a more efficient way to approach backpropagation?&lt;/p&gt;&#xA;" OwnerUserId="28231" LastActivityDate="2017-03-18T14:20:31.763" Title="Neural Network backpropagation taking forever" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17679" PostTypeId="1" CreationDate="2017-03-17T23:42:23.603" Score="1" ViewCount="542" Body="&lt;p&gt;I created a two class data set, &lt;code&gt;dat&lt;/code&gt;. This data set only contains 25 variables. 24 are numeric and 1 is categorical (with 26 levels). The result of the auto-tuned model gives a &lt;code&gt;mtry&lt;/code&gt; of 2, 14, 26, 38 and 50. So is this true that the caret created the dummy variable for the categorical variable? What other models does caret create dummy variable automatically?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see following example code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pacman::p_load(R.utils, caret, doParallel)&#xA;cl &amp;lt;- makeCluster(8, outfile=&quot;&quot;)&#xA;registerDoParallel(cl)&#xA;set.seed(2)&#xA;dat &amp;lt;- twoClassSim(48842, noiseVars = 10)# 25 variables.&#xA;&#xA;dat = cbind(dat,data.frame(categorical = sample(LETTERS,nrow(dat),replace = T)))&#xA;&#xA;fit_on &amp;lt;-  list(rs1 = 1:1300)&#xA;pred_on &amp;lt;- list(rs1 = 1330:10000)&#xA;indexFinal = 1:1000&#xA;&#xA;fitControl &amp;lt;- trainControl(method = &quot;cv&quot;,number=1,repeats=1,&#xA;                           classProbs = TRUE,summaryFunction = twoClassSummary,&#xA;                           index= fit_on, indexOut = pred_on,&#xA;                           indexFinal = indexFinal,&#xA;                           verboseIter = TRUE,&#xA;                           savePredictions = TRUE&#xA;)&#xA;&#xA;model = train(Class ~ ., data = dat,  &#xA;              method = &quot;rf&quot;,&#xA;              preProcess=c(&quot;center&quot;,&quot;scale&quot;),&#xA;              trControl = fitControl,&#xA;              tuneLength = 5)&#xA;&#xA;model&#xA;&#xA;# mtry  ROC        Sens       Spec     &#xA;# 2    0.9166711  0.9298806  0.6778085&#xA;# 14    0.9237881  0.8851236  0.7645137&#xA;# 26    0.9177710  0.8746803  0.7770797&#xA;# 38    0.9132661  0.8689258  0.7753204&#xA;# 50    0.9093577  0.8687127  0.7728072&#xA;# there are only 25 variables.&#xA;&#xA;stopCluster(cl)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29204" LastActivityDate="2019-03-26T04:06:08.453" Title="R, caret: does caret automatically create dummy variable for random Forest?" Tags="|machine-learning|r|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17681" PostTypeId="1" AcceptedAnswerId="17691" CreationDate="2017-03-18T04:36:01.347" Score="1" ViewCount="533" Body="&lt;p&gt;I have a data set which deals with response variable in the order of 10&lt;sup&gt;-20&lt;/sup&gt;. The scatter plot for such a regression appears linear, but the problem being when I predict for test cases using values very small compared to the trained sample the predicted response variables appear in negative values. &lt;strong&gt;Please note that the values of the data set predicted should not be negative.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the 3d scatter plot of my data&#xA;&lt;a href=&quot;https://i.stack.imgur.com/sj0nn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sj0nn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a type of regression which has better predictive power which can perform such operations without such an error?&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-03-19T08:06:51.130" Title="Which type of regression has the best predictive power for extrapolating for smaller values?" Tags="|machine-learning|predictive-modeling|regression|scikit-learn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17685" PostTypeId="2" ParentId="17677" CreationDate="2017-03-18T10:18:05.370" Score="1" Body="&lt;p&gt;To make this question answerable, I have to assume that you have implemented back propagation correctly. That is, in terms of the underlying maths, the steps are in the right order, there is not any non-necessary repetition, and the values are calculated using correct formulae. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check whether the gradient values are correct by implementing &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;gradient checking for weights&lt;/a&gt;, which in general you do by taking a random test case (any network architecture, random weights, inputs and target values), run your back propagation code to get your gradients, then for each weight in turn run the network forward, measure error, then adjust the weight by a very small delta, measure error again and use the difference in error to get an estimate for the gradient. If the estimate and your back propagation agree - approximately - for all weights, then your back propagation code is likely correct. If you are going to implement just one unit test for a NN implementation, this is the one to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address the question directly: &quot;Is there a more efficient way to approach backpropagation?&quot;, the answer is complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The basic answer is &quot;no&quot;&lt;/em&gt;. You absolutely have to do all the calculations involved in back propagation, there are no shortcuts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, most neural network frameworks will perform much better than your implementation, where you give an example of a network that has layers &lt;code&gt;[39,39,39,1]&lt;/code&gt; training on 10,000 examples and taking 28 minutes to process a single epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you don't link your implementation, it is not possible to figure out precisely what the difference is. However, ignoring the possibility of bugs, the following things will probably apply:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Avoid calculating results using loops in an interpreted language. If you are looping through neuron or weight indices to multiple some values together and store back in an array in a high-level language, then this can be very inefficient. Most frameworks take advantage of fast loops and parallelisation that is possible if you frame the neural network processes (both forward and back propagation) in terms of matrix and vector manipulations. This is sometimes called &lt;a href=&quot;https://en.wikipedia.org/wiki/Array_programming&quot; rel=&quot;nofollow noreferrer&quot;&gt;vectorizaton&lt;/a&gt;, and most frameworks split this up so that the vector and matrix operations are black box routines implemented with strong attention to optimisation in a low-level language (like C), whilst passing the results around and deciding what to do with them can be done in higher-level language like Python or Matlab.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mini-batches. You have to process forward and back propagation for each example seen by the network. But you don't have to apply weight updates for each example. It turns out that applying an update only once every so many examples is not only more efficient in terms of calculations, but also has benefits for convergence. Frameworks that support large amounts of parallel processing via GPU can also improve performance greatly from working in batches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is unlikely you will find the time to explore optimisation to the extent that the better resourced public frameworks already have. However, you can get quite far by taking advantage of vectorisation options built into your language - for instance using matrix operations in Matlab, or Numpy features in Python. This might be worthwhile so you can continue to work on and understand low-level detail in neural network models. However, at some point, if you want to work on practical problems instead of your own NN library, you will find yourself naturally switching to a third-party library because the optimisation problems and many useful additions have been solved for you already.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-18T14:20:31.763" LastActivityDate="2017-03-18T14:20:31.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17689" PostTypeId="1" CreationDate="2017-03-18T19:03:29.133" Score="2" ViewCount="2726" Body="&lt;p&gt;I would appreciate if you could let me know what machine learning techniques are suitable for a small longitudinal data set with time varying covariates. Some part of data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  TIME        Y         x1      x2      x3      x4      x5&#xA;1   1        0 .060     1.28    0.02    0.87    1.22    0.06&#xA;1   2        0.110      1.27    0.01    0.82    1.00    -0.01&#xA;1   3        -0.456     1.05    -0.06   0.92    0.73    0.02&#xA;1   4        -0.325     1.11    -0.02   0.86    0.81    0.08&#xA;1   5        -0.567     1.22    -0.06   0.89    0.48    0.01&#xA;2   1        1.251      1.06    0.11    0.81    0.84    0.20&#xA;2   2        1.357      1.06    0.08    0.88    0.69    0.14&#xA;2   3        0.007      0.97    0.08    0.91    0.81    0.17&#xA;2   4        1.20       1.06    0.13    0.82    0.88    0.23&#xA;2   5        2.235      1.12    0.15    0.76    1.08    0.28&#xA;2   6        1.340      1.60    0.26    0.55    1.31    0.37&#xA;2   7        2.470      1.58    0.26    0.56    1.16    0.35&#xA;2   8        2.346      1.54    0.24    0.59    1.08    0.33&#xA;2   9        2.250      1.72    0.22    0.55    0.84    0.29&#xA;2   10       2.283      1.72    0.21    0.53    0.79    0.29&#xA;2   11       1.798      1.63    0.19    0.55    0.73    0.27&#xA;2   12       2.828      2.17    0.32    0.44    0.95    0.43&#xA;3   1        -0.303     0.87    -0.03   0.79    0.61    0.00&#xA;3   2        -0.888     0.83    -0.14   0.95    0.57    -0.02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems that &quot;MixRF&quot; and &quot;mboost&quot; are among those R packages that could deal with panel data sets but unfortunately I couldn't understand how to use them on my data set.&#xA;Best regards,&lt;/p&gt;&#xA;" OwnerUserId="26019" LastActivityDate="2017-03-18T19:03:29.133" Title="machine learning for a small longitudinal or panel data set" Tags="|machine-learning|r|regression|" AnswerCount="0" CommentCount="20" ContentLicense="CC BY-SA 3.0" />
  <row Id="17691" PostTypeId="2" ParentId="17681" CreationDate="2017-03-19T08:06:51.130" Score="2" Body="&lt;p&gt;OK, so you apply linear regression to create a model for your data, and when you use that model to predict new values, the output values don't satisfy a constraint (namely, being positive). I can think of only a few different things that might be going on here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The new input values you are giving to the model are not within the allowable range for the problem. This is unlikely given your setup - presumably you know your situation well enough to determine what input values are allowed - but if this does happen to be the case, you need to change the way in which you obtain inputs.&lt;/li&gt;&#xA;&lt;li&gt;A linear model is not accurate over the full range between your original input data and new input data. Without more domain knowledge, it's impossible to tell what sort of model would be a better substitute. If this is a situation where you can come up with some sort of theoretical prediction about what general form of model should describe the data, you should use that form. If not... try an exponential I guess?&lt;/li&gt;&#xA;&lt;li&gt;A linear model is accurate, it's just not this particular one. You might have to do something like forcing the regression to include a particular point (e.g. the origin), for example. That would mean that you're not actually doing a full linear regression, but rather fitting a model with one fewer free parameter - e.g. in slope-intercept form, you're basically forcing the intercept to zero and only fitting the slope. Or the equivalent if using another point. This should be doable with some regression library that you can use. But if you use a linear model, it &lt;em&gt;will&lt;/em&gt; predict negative values somewhere. You'll have to live with declaring those regions of parameter space to be outside the domain of the model (or possibly finding a way to make sense of negative values).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Based on the little information you've provided here, there's really not much more I can say. Maybe if you described your situation in more detail, I could offer more specific advice.&lt;/p&gt;&#xA;" OwnerUserId="695" LastActivityDate="2017-03-19T08:06:51.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17692" PostTypeId="1" CreationDate="2017-03-19T13:48:16.153" Score="1" ViewCount="380" Body="&lt;p&gt;GROBID (&lt;a href=&quot;https://github.com/kermitt2/grobid&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/kermitt2/grobid&lt;/a&gt;) is terrific for fine-grained (section, chapter) segmentation of PDF files, but I need to do the equivalent for HTML and plaintext files. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried TextTiling but it seems to work only for coarse-grained topic change detection (like distinguishing between newspaper articles regarding very different subjects).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are methods and/or SW toolkits that I could use to segment a document into sections regarding particular fine-grained topics?&lt;/p&gt;&#xA;" OwnerUserId="30140" LastActivityDate="2017-03-19T13:48:16.153" Title="Section/Topic segmentation in HTML and plaintext documents" Tags="|nlp|text-mining|text|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17693" PostTypeId="1" CreationDate="2017-03-19T15:34:46.507" Score="6" ViewCount="3603" Body="&lt;p&gt;I'm using the sklearn wrapper for XGBoost.&#xA;I didn't manage to find a clear explanation for the way the probabilities given as output by &lt;code&gt;predict_proba()&lt;/code&gt; are computed.&lt;/p&gt;&#xA;&lt;p&gt;In random forest, for example, I understand it reflects the mean of proportions of the samples belonging to the class among the relevant leaves of all the trees.&lt;/p&gt;&#xA;&lt;p&gt;However, in XGBoost I couldn't understand the computation from the documentation or the code. Shouldn't it give different weights for each tree?&lt;/p&gt;&#xA;" OwnerUserId="30144" LastEditorUserId="83275" LastEditDate="2023-03-14T05:49:22.047" LastActivityDate="2023-04-16T14:06:50.773" Title="How does XGBoost compute the probabilities in predict_proba()?" Tags="|scikit-learn|random-forest|decision-trees|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17695" PostTypeId="1" AcceptedAnswerId="17742" CreationDate="2017-03-19T17:17:02.720" Score="3" ViewCount="5234" Body="&lt;p&gt;I've read the following article about how to treat outliers in a dataset: &lt;a href=&quot;http://napitupulu-jon.appspot.com/posts/outliers-ud120.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://napitupulu-jon.appspot.com/posts/outliers-ud120.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, he removes all the y which has a huge difference with the majority:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def outlierCleaner(predictions, ages, net_worths):&#xA;    &quot;&quot;&quot;&#xA;        clean away the 10% of points that have the largest&#xA;        residual errors (different between the prediction&#xA;        and the actual net worth)&#xA;&#xA;        return a list of tuples named cleaned_data where &#xA;        each tuple is of the form (age, net_worth, error)&#xA;    &quot;&quot;&quot;&#xA;&#xA;    #calculate the error,make it descend sort, and fetch 90% of the data&#xA;&#xA;    errors = (net_worths-predictions)**2&#xA;    cleaned_data =zip(ages,net_worths,errors)&#xA;    cleaned_data = sorted(cleaned_data,key=lambda x:x[2][0], reverse=True)&#xA;    limit = int(len(net_worths)*0.1)&#xA;&#xA;&#xA;    return cleaned_data[limit:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But how may I apply this technique to a time series dataset if its rows are correlative?&lt;/p&gt;&#xA;" OwnerUserId="30149" LastEditorUserId="30149" LastEditDate="2017-03-19T20:56:27.267" LastActivityDate="2018-04-16T03:36:03.650" Title="How to treat outliers in a time series dataset?" Tags="|python|dataset|regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17696" PostTypeId="2" ParentId="13504" CreationDate="2017-03-19T19:01:18.807" Score="0" Body="&lt;p&gt;AFTER graduation, you will [continue to] learn how to learn, how to accelerate your learning process with smart people who get smarter because they accelerate how they learn ... away from academia, without institutional support, in difficult environments ... each of us must transcend our own algorithmic or predictable approach to learning or how our machine has been programmed in the past to recognize patterns and learn.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How you collaborate, communicate, work with teams of solid, qualified, antifragile LEARNERS will shape your career, success, satisfaction, friendships, life ... and our species.&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smart people are often suckers for the propaganda and comforts of respectable human systems, such as academia or bureaucratic machines built on piles of smart people who settle for calcifying into smart, predictable cogs ... but the world is chaotic and throws sand into the working of inflexible systems and smart cogs -- so how you learn will determine whether chaos serves you or defeats you.  Learn to think in a generally &lt;a href=&quot;https://en.wikipedia.org/wiki/Antifragility&quot; rel=&quot;nofollow noreferrer&quot;&gt;antifragile manner&lt;/a&gt; rather than simply being resilient or defensively robust ... as you wrangle data, remember that the most interesting and useful things will tend to happen in &lt;em&gt;Extremeistan&lt;/em&gt;.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuring quality of the data you are wrangling [or that some intern has wrangled] for the analysis is always going to be essential; the GIGO rule will always take precedence over the sophisticated wonkery. That is why it is so necessary for us to step back, see our roles in a larger picture. Being proficient in skill like Python or experimental design is awesome and necessary -- but do not settle for being pigeon-holed as just an expert in _____ or a phenomenal code jockey. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data, like humans, have more interesting stories to tell than just the narrow questions we want to ask AND just because we have a bias, it does not mean that it's necessary to wrangle up compliant data to confirm it. Outliers need to be listened to; there's a reason why they show up, why they persist. Listen to extreme, challenging, heretical points of view ... especially when the local inquisition is angry about the uncomfortable &lt;a href=&quot;https://en.wikipedia.org/wiki/Galileo_affair&quot; rel=&quot;nofollow noreferrer&quot;&gt;someone can prove that the planets do not orbit around them&lt;/a&gt;. Not all uncomfortable, unpopular opinions should taken seriously, but some are absolutely essential and necessary truth. Beware of the zealots, &lt;a href=&quot;https://xkcd.com/1732/&quot; rel=&quot;nofollow noreferrer&quot;&gt;even brilliant cartoonists&lt;/a&gt;, who claim to think independently when 97% of them conform to a generally-accepted truth and adamantly demand that others fall into line ... sometimes, cartoons are just metaphors for how seriously we should take cartoons. &lt;/p&gt;&#xA;" OwnerUserId="30147" LastEditorUserId="30147" LastEditDate="2017-03-20T01:31:13.360" LastActivityDate="2017-03-20T01:31:13.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17698" PostTypeId="1" AcceptedAnswerId="17717" CreationDate="2017-03-19T21:10:44.770" Score="2" ViewCount="113307" Body="&lt;p&gt;I am trying to retrieve the column names of the data set model$data using the following formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample(colnames(model$data),1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run it I receive the following error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in sample.int(length(x), size, replace, prob) : &#xA;  invalid first argument&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Appreciate any help!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;str(model) looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(model)&#xA;    List of 13&#xA;     $ data            :List of 1&#xA;      ..$ : num [1:1000, 1:56] 1 1 1 1 0 1 1 0 1 1 ...&#xA;      .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;      .. .. ..$ : chr [1:1000] &quot;7530&quot; &quot;5975&quot; &quot;552&quot; &quot;815&quot; ...&#xA;      .. .. ..$ : chr [1:56] &quot;Agriculture_and_Hunting&quot; &quot;Baking&quot; &quot;Biochemistry&quot; &quot;Braiding&quot; ...&#xA;     $ unit.classif    : num [1:1000] 3 5 5 5 16 3 5 1 3 3 ...&#xA;     $ distances       : num [1:1000] 0.000806 0.000239 0.000239 0.000239 0.001953 ...&#xA;     $ grid            :List of 6&#xA;      ..$ pts              : num [1:25, 1:2] 1.5 2.5 3.5 4.5 5.5 1 2 3 4 5 ...&#xA;      .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;      .. .. ..$ : NULL&#xA;      .. .. ..$ : chr [1:2] &quot;x&quot; &quot;y&quot;&#xA;      ..$ xdim             : num 5&#xA;      ..$ ydim             : num 5&#xA;      ..$ topo             : chr &quot;hexagonal&quot;&#xA;      ..$ neighbourhood.fct: Factor w/ 2 levels &quot;bubble&quot;,&quot;gaussian&quot;: 1&#xA;      ..$ toroidal         : logi FALSE&#xA;      ..- attr(*, &quot;class&quot;)= chr &quot;somgrid&quot;&#xA;     $ codes           :List of 1&#xA;      ..$ : num [1:25, 1:56] 0.000388 0.99996 1 1 1 ...&#xA;      .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;      .. .. ..$ : chr [1:25] &quot;V1&quot; &quot;V2&quot; &quot;V3&quot; &quot;V4&quot; ...&#xA;      .. .. ..$ : chr [1:56] &quot;Agriculture_and_Hunting&quot; &quot;Baking&quot; &quot;Biochemistry&quot; &quot;Braiding&quot; ...&#xA;     $ changes         : num [1:100, 1] 0.00261 0.00263 0.00262 0.00254 0.00254 ...&#xA;     $ alpha           : num [1:2] 0.05 0.01&#xA;     $ radius          : Named num [1:2] 3 0&#xA;      ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;67%&quot; &quot;&quot;&#xA;     $ user.weights    : num 1&#xA;     $ distance.weights: num 1&#xA;     $ whatmap         : int 1&#xA;     $ maxNA.fraction  : int 0&#xA;     $ dist.fcts       : chr &quot;sumofsquares&quot;&#xA;     - attr(*, &quot;class&quot;)= chr &quot;kohonen&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30136" LastEditorUserId="30136" LastEditDate="2017-03-20T15:59:35.390" LastActivityDate="2017-03-20T17:55:36.390" Title="Retrieving column names in R" Tags="|r|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17699" PostTypeId="2" ParentId="17698" CreationDate="2017-03-19T21:21:54.247" Score="0" Body="&lt;p&gt;$ is not valid as part of the data set (or frame) name since R uses it to denote column name (:)) so R actually tried to get the column name 'data' from the data frame named 'model'&#xA;Try:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names(model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And rename model$data to model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;str(model)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(model)&#xA;List of 13&#xA; $ data            :List of 1&#xA;  ..$ : num [1:1000, 1:56] 1 1 1 1 0 1 1 0 1 1 ...&#xA;  .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;  .. .. ..$ : chr [1:1000] &quot;7530&quot; &quot;5975&quot; &quot;552&quot; &quot;815&quot; ...&#xA;  .. .. ..$ : chr [1:56] &quot;Agriculture_and_Hunting&quot; &quot;Baking&quot; &quot;Biochemistry&quot; &quot;Braiding&quot; ...&#xA; $ unit.classif    : num [1:1000] 3 5 5 5 16 3 5 1 3 3 ...&#xA; $ distances       : num [1:1000] 0.000806 0.000239 0.000239 0.000239 0.001953 ...&#xA; $ grid            :List of 6&#xA;  ..$ pts              : num [1:25, 1:2] 1.5 2.5 3.5 4.5 5.5 1 2 3 4 5 ...&#xA;  .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;  .. .. ..$ : NULL&#xA;  .. .. ..$ : chr [1:2] &quot;x&quot; &quot;y&quot;&#xA;  ..$ xdim             : num 5&#xA;  ..$ ydim             : num 5&#xA;  ..$ topo             : chr &quot;hexagonal&quot;&#xA;  ..$ neighbourhood.fct: Factor w/ 2 levels &quot;bubble&quot;,&quot;gaussian&quot;: 1&#xA;  ..$ toroidal         : logi FALSE&#xA;  ..- attr(*, &quot;class&quot;)= chr &quot;somgrid&quot;&#xA; $ codes           :List of 1&#xA;  ..$ : num [1:25, 1:56] 0.000388 0.99996 1 1 1 ...&#xA;  .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;  .. .. ..$ : chr [1:25] &quot;V1&quot; &quot;V2&quot; &quot;V3&quot; &quot;V4&quot; ...&#xA;  .. .. ..$ : chr [1:56] &quot;Agriculture_and_Hunting&quot; &quot;Baking&quot; &quot;Biochemistry&quot; &quot;Braiding&quot; ...&#xA; $ changes         : num [1:100, 1] 0.00261 0.00263 0.00262 0.00254 0.00254 ...&#xA; $ alpha           : num [1:2] 0.05 0.01&#xA; $ radius          : Named num [1:2] 3 0&#xA;  ..- attr(*, &quot;names&quot;)= chr [1:2] &quot;67%&quot; &quot;&quot;&#xA; $ user.weights    : num 1&#xA; $ distance.weights: num 1&#xA; $ whatmap         : int 1&#xA; $ maxNA.fraction  : int 0&#xA; $ dist.fcts       : chr &quot;sumofsquares&quot;&#xA; - attr(*, &quot;class&quot;)= chr &quot;kohonen&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10824" LastEditorUserId="30136" LastEditDate="2017-03-20T10:30:04.927" LastActivityDate="2017-03-20T10:30:04.927" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17701" PostTypeId="1" CreationDate="2017-03-20T06:59:06.727" Score="5" ViewCount="1794" Body="&lt;p&gt;I am trying to train a convolutional neural network with Keras at recognizing tags for Stack Exchange questions about cooking. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The i-th question element of my data-set is like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id                                                         2&#xA;title                    How should I cook bacon in an oven?&#xA;content    &amp;lt;p&amp;gt;I've heard of people cooking bacon in an ov...&#xA;tags                                 oven cooking-time bacon&#xA;Name: 1, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have removed tags with BeautifulSoup and removed punctuation too.&#xA;Since questions' content are very big I have decided to focus on titles. &#xA;I have used sklearn CountVectorizer to vectorize words in titles. However they were more than 8000 words (excluding stop words). So I decided apply a part of speech tagging and retrieve only Nouns and Gerunds.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;vectorizer = CountVectorizer(stop_words='english')&#xA;titles = dataframes['cooking']['title']&#xA;pos_titles = []&#xA;for i,title in enumerate(titles):&#xA;    pos = []&#xA;    pt_titl = nltk.pos_tag(word_tokenize(title))&#xA;    for pt in pt_titl:&#xA;        if pt[1]=='NN' or pt[1]=='NNS' or pt[1]=='VBG':# or pt[1]=='VBP' or pt[1]=='VBS':&#xA;            pos.append(pt[0])&#xA;    pos_titles.append(&quot; &quot;.join(pos))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This represents my input vector. I have vectorized tags too and extract dense matrixes for both input and tags. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tags = [&quot; &quot;.join(x) for x in dataframes['cooking']['tags']]&#xA;Xd = X.todense()&#xA;&#xA;Y = vectorizer.fit_transform(tags)&#xA;Yd = Y.todense()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Split data into train and validation set &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(Xd, Yd, test_size=0.33, random_state=42)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I am trying to train a Conv1D network&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense, Activation,Flatten&#xA;from keras.layers import Conv2D, MaxPooling2D,Conv1D, Embedding,GlobalMaxPooling1D,Dropout,MaxPooling1D&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(Embedding(Xd.shape[1],&#xA;                    128,&#xA;                    input_length=Xd.shape[1]))&#xA;model.add(Conv1D(32,5,activation='relu'))&#xA;model.add(MaxPooling1D(100,stride=50))&#xA;model.add(Conv1D(32,5,activation='relu'))&#xA;model.add(GlobalMaxPooling1D())&#xA;model.add(Dense(Yd.shape[1], activation ='softmax'))&#xA;&#xA;&#xA;model.compile(optimizer='rmsprop',&#xA;          loss='categorical_crossentropy',&#xA;          metrics=['accuracy'])&#xA;model.fit(X_train, y_train, batch_size=32,verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it gets stucked on a very low accuracy and it shows a barely increasing loss along the epochs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/10&#xA;10320/10320 [==============================] - 401s - loss: 15.8098 - acc: 0.0604   &#xA;Epoch 2/10&#xA;10320/10320 [==============================] - 339s - loss: 15.5671 - acc: 0.0577   &#xA;Epoch 3/10&#xA;10320/10320 [==============================] - 314s - loss: 15.5509 - acc: 0.0578   &#xA;Epoch 4/10&#xA;10320/10320 [==============================] - 34953s - loss: 15.5493 - acc: 0.0578  &#xA;Epoch 5/10&#xA;10320/10320 [==============================] - 323s - loss: 15.5587 - acc: 0.0578   &#xA;Epoch 6/10&#xA; 6272/10320 [=================&amp;gt;............] - ETA: 133s - loss: 15.6005 - acc: 0.0550&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2020-06-05T19:07:30.523" Title="Convolutional Network for Text Classification" Tags="|machine-learning|deep-learning|nlp|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17704" PostTypeId="1" CreationDate="2017-03-20T09:49:41.587" Score="-2" ViewCount="91" Body="&lt;p&gt;I have a set of news paper report.&#xA;I need to classify them as Sports, Politics related at first.&#xA;Once the given document is in Sports category, then I need to furthur classify them as Football, Cricket, Tennis etc. For Politics it's International, National, Local etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again I would like assign a positive, negative class on this  sub category result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for a given document ouput will be something like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document- &amp;gt;[Main_Category, Sub_Category, Sentiment] &#xA;Document_101 -&amp;gt;[Sports, Football, Positive] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the best approcah to create a model for this case ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shall I create different models for each of this class (sports, politics, cricket, international, positive, negative etc) ?&#xA;If yes then is it possible to create a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pipeline&lt;/a&gt; of each class prediction ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it advisable to trian a single model per each output like sports_cricket_positive,s ports_cricket_negative etc&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2018-04-30T18:12:09.283" Title="How to create a model that can have multiple labels associated to it?" Tags="|machine-learning|python|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17705" PostTypeId="2" ParentId="17578" CreationDate="2017-03-20T09:52:25.237" Score="17" Body="&lt;p&gt;You might want to check out &lt;a href=&quot;https://github.com/benoitsteiner/tensorflow-opencl/&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/benoitsteiner/tensorflow-opencl/&lt;/a&gt; which is a fork of Tensorflow with OpenCL support.&#xA;If your OS is supported by the fork and you are able to properly install it in your system then you can run Keras on top of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note however that integrated GPUs in general do not offer a lot of calculating power, roughly your GPU will be around 125 GFLOPS (&lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_Intel_graphics_processing_units#Notes&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/List_of_Intel_graphics_processing_units#Notes&lt;/a&gt;), that is your CPU is most likely more powerful. For example, comparing your GPU with a Radeon RX 480 or a GeForce GTX 1080 Ti, they are respectively almost 50 and 100 times more powerful.&lt;/p&gt;&#xA;" OwnerUserId="30166" LastEditorUserId="30166" LastEditDate="2017-03-20T15:15:37.973" LastActivityDate="2017-03-20T15:15:37.973" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17708" PostTypeId="2" ParentId="17671" CreationDate="2017-03-20T11:47:45.517" Score="7" Body="&lt;p&gt;As you say, both approaches are used.  It's called &lt;strong&gt;tied biases&lt;/strong&gt;  if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and &lt;strong&gt;untied biases&lt;/strong&gt; if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Untied biases increase the capacity of your model, so they can be a good idea if you are underfitting. But in this case using tied biases and more filters and/or layers might also help, see &lt;a href=&quot;https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/&quot; rel=&quot;noreferrer&quot;&gt;https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27704" LastActivityDate="2017-03-20T11:47:45.517" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17709" PostTypeId="1" AcceptedAnswerId="17715" CreationDate="2017-03-20T13:29:59.400" Score="0" ViewCount="116" Body="&lt;p&gt;I am working on a project that is about Natural Language Processing. However I am stuck at the point which is I have a ANN that has fixed size of input neurons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to do sentiment analysis with using Imdb movie review set. To able to do that, firstly, I calculated the word embeddings for each word with creating a word-context matrix then applied SVD. So I have the word embedding matrix. But I do not know the best way to compress sentence's vector (which contains embeddings for each word in the sentence) into a fixed size to be able to feed the neural net. I tried PCA but result was not satisfying. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help?&lt;/p&gt;&#xA;" OwnerUserId="28916" LastActivityDate="2017-03-20T16:43:24.413" Title="Best way to fix the size of a sentence [Sentiment Analysis]" Tags="|word-embeddings|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17710" PostTypeId="1" CreationDate="2017-03-20T13:58:09.653" Score="25" ViewCount="39331" Body="&lt;p&gt;I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, interactions. And it can handle both numerical and categorical variables and it also seems that redundant variables does not affect this method too much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, in predictive modeling, you may do some selection among all the features you have and you may also create some new features from the set of features you have. So select a subset of features means you think there are some redundancy in your set of features; create some new features from the current feature set means you do some functional transformations on your current features. Then, both of these two points should be covered in XGBoost. Then, does it mean that to use XGBoost, you only need to choose those tunning parameters wisely? What is the value of doing feature engineering using XGBoost?&lt;/p&gt;&#xA;" OwnerUserId="30170" LastActivityDate="2021-01-02T02:51:19.867" Title="Is feature engineering still useful when using XGBoost?" Tags="|xgboost|feature-engineering|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17711" PostTypeId="1" CreationDate="2017-03-20T13:58:30.447" Score="1" ViewCount="725" Body="&lt;p&gt;I have two corpora (in the form of list of sentences); one is the &quot;original&quot; one, the other is a version of it with some minor changes (with contractions expaned, for instance):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s1 = ['He's a real stupid', 'There's no need to worry', 'All right, my friends']&#xA;s2 = ['He is a real stupid', 'There is no need to worry', 'All right, my friends']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to get some rendering of the corresponding sentence pairs where IS some difference between the sentences. Is there any elegant solution to this?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-10-24T14:01:07.187" Title="Comparing text corpora sentence-wise in Python" Tags="|python|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17712" PostTypeId="2" ParentId="17710" CreationDate="2017-03-20T14:50:48.693" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the value of doing feature engineering using XGBoost?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Performance maybe?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note we don't use XGBoost, but another gradient boosting library - though XGBoost's performance probably also depends on the dimensionality of the data in some way.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have a dataset where each item consists of 3 signals, each 6000 samples long - that's 18k features. Using these features directly takes ages (days), so we did some manual feature engineering to reduce the number of features to about 200. Now training (including parameter tuning) is a matter of a few hours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For comparison: a short time ago we also started training ConvNets with the same data and the whole 18k features (no feature engineering). They reach the same accuracy as the gradient boosting models after only about 2 hours of training.&lt;/p&gt;&#xA;" OwnerUserId="676" LastActivityDate="2017-03-20T14:50:48.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17713" PostTypeId="1" AcceptedAnswerId="17969" CreationDate="2017-03-20T15:41:40.600" Score="2" ViewCount="6767" Body="&lt;p&gt;I'm new in tensorflow and machine learning. Could you explain me how implement deconvulation on this CNN model(&lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&lt;/a&gt;) for text classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class TextCNN(object):&#xA;&quot;&quot;&quot;&#xA;A CNN for text classification.&#xA;Uses an embedding layer, followed by a convolutional, max-pooling and softmax layer.&#xA;&quot;&quot;&quot;&#xA;def __init__(&#xA;  self, sequence_length, num_classes, vocab_size,&#xA;  embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0):&#xA;&#xA;    # Placeholders for input, output and dropout&#xA;    self.input_x = tf.placeholder(tf.int32, [None, sequence_length], name=&quot;input_x&quot;)&#xA;    self.input_y = tf.placeholder(tf.float32, [None, num_classes], name=&quot;input_y&quot;)&#xA;    self.dropout_keep_prob = tf.placeholder(tf.float32, name=&quot;dropout_keep_prob&quot;)&#xA;&#xA;    # Keeping track of l2 regularization loss (optional)&#xA;    l2_loss = tf.constant(0.0)&#xA;&#xA;    # Embedding layer&#xA;    with tf.device('/cpu:0'), tf.name_scope(&quot;embedding&quot;):&#xA;        self.W = tf.Variable(&#xA;            tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),&#xA;            name=&quot;W&quot;)&#xA;        self.embedded_chars = tf.nn.embedding_lookup(self.W, self.input_x)&#xA;        self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)&#xA;&#xA;    # Create a convolution + maxpool layer for each filter size&#xA;    pooled_outputs = []&#xA;    for i, filter_size in enumerate(filter_sizes):&#xA;        with tf.name_scope(&quot;conv-maxpool-%s&quot; % filter_size):&#xA;            # Convolution Layer&#xA;            filter_shape = [filter_size, embedding_size, 1, num_filters]&#xA;            W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name=&quot;W&quot;)&#xA;            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name=&quot;b&quot;)&#xA;            conv = tf.nn.conv2d(&#xA;                self.embedded_chars_expanded,&#xA;                W,&#xA;                strides=[1, 1, 1, 1],&#xA;                padding=&quot;VALID&quot;,&#xA;                name=&quot;conv&quot;)&#xA;            # Apply nonlinearity&#xA;            h = tf.nn.relu(tf.nn.bias_add(conv, b), name=&quot;relu&quot;)&#xA;            # Maxpooling over the outputs&#xA;            pooled = tf.nn.max_pool(&#xA;                h,&#xA;                ksize=[1, sequence_length - filter_size + 1, 1, 1],&#xA;                strides=[1, 1, 1, 1],&#xA;                padding='VALID',&#xA;                name=&quot;pool&quot;)&#xA;            pooled_outputs.append(pooled)&#xA;&#xA;    # Combine all the pooled features&#xA;    num_filters_total = num_filters * len(filter_sizes)&#xA;    self.h_pool = tf.concat(pooled_outputs, 3)&#xA;    self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total])&#xA;&#xA;    # Add dropout&#xA;    with tf.name_scope(&quot;dropout&quot;):&#xA;        self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)&#xA;&#xA;    # Final (unnormalized) scores and predictions&#xA;    with tf.name_scope(&quot;output&quot;):&#xA;        W = tf.get_variable(&#xA;            &quot;W&quot;,&#xA;            shape=[num_filters_total, num_classes],&#xA;            initializer=tf.contrib.layers.xavier_initializer())&#xA;        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name=&quot;b&quot;)&#xA;        l2_loss += tf.nn.l2_loss(W)&#xA;        l2_loss += tf.nn.l2_loss(b)&#xA;        self.scores = tf.nn.xw_plus_b(self.h_drop, W, b, name=&quot;scores&quot;)&#xA;        self.predictions = tf.argmax(self.scores, 1, name=&quot;predictions&quot;)&#xA;&#xA;    # CalculateMean cross-entropy loss&#xA;    with tf.name_scope(&quot;loss&quot;):&#xA;        losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y)&#xA;        self.loss = tf.reduce_mean(losses) + l2_reg_lambda * l2_loss&#xA;&#xA;    # Accuracy&#xA;    with tf.name_scope(&quot;accuracy&quot;):&#xA;        correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))&#xA;        self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, &quot;float&quot;), name=&quot;accuracy&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know about tf.nn.conv2d_transpose, but I don't understand how can I get original data (not embedded_chars)&lt;/p&gt;&#xA;" OwnerUserId="30173" LastEditorUserId="30173" LastEditDate="2017-03-23T17:59:16.787" LastActivityDate="2018-03-12T11:16:10.847" Title="How can I implement deconvolution on CNN (TensorFlow)?" Tags="|python|deep-learning|tensorflow|convolution|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17714" PostTypeId="2" ParentId="17711" CreationDate="2017-03-20T16:43:00.650" Score="1" Body="&lt;p&gt;There are several ways to do this, but, assuming the lists are the same length and the sentences you wish to compare are in the same indices int their respective lists, you can handle this with a simple list comprehension:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diffs = [(s1[i], s2[i]) for i in range(len(s1)) if s1[i] != s2[i]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2017-03-20T16:43:00.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17715" PostTypeId="2" ParentId="17709" CreationDate="2017-03-20T16:43:24.413" Score="3" Body="&lt;p&gt;The easiest way is to average the word- embeddings. This works quite well. &#xA;Another thing you can try is to represent each document as a bag of words - i.e. - to have a vector in the size of your vocabulary, where each element in the vector represents the number of times a certain word had been mentioned in your document (for example, the first element in the vector will represent how many times the word &lt;code&gt;a&lt;/code&gt; was mentioned, and so on). &#xA;Afterwords, to reduce the size of the vector you can use techniques like &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA&lt;/a&gt;, SVD, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;autoencoders&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="25905" LastActivityDate="2017-03-20T16:43:24.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17717" PostTypeId="2" ParentId="17698" CreationDate="2017-03-20T17:55:36.390" Score="6" Body="&lt;p&gt;Your data boils down to something like this structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(model)&#xA;List of 2&#xA; $ data:List of 1&#xA;  ..$ : int [1:3, 1:4] 1 2 3 4 5 6 7 8 9 10 ...&#xA;  .. ..- attr(*, &quot;dimnames&quot;)=List of 2&#xA;  .. .. ..$ : NULL&#xA;  .. .. ..$ : chr [1:4] &quot;a&quot; &quot;b&quot; &quot;c&quot; &quot;d&quot;&#xA; $ foo : num 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but you have some other components that we don't need to bother with, and my data doesn't have row names and is a lot smaller.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model&lt;/code&gt; is a list of 2 (for me) and 13 (for you) parts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;$data&lt;/code&gt; component is also a &quot;List of 1&quot; component. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;code&gt;colnames(model$data)&lt;/code&gt; is trying to get the &lt;code&gt;colnames&lt;/code&gt; of a list, and failing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; colnames(model$data)&#xA;NULL&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which you would have spotted if you'd tried running &lt;code&gt;colnames(model$data)&lt;/code&gt; yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want the &lt;code&gt;colnames&lt;/code&gt; of the first element of the list &lt;code&gt;model$data&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; colnames(model$data[[1]])&#xA;[1] &quot;a&quot; &quot;b&quot; &quot;c&quot; &quot;d&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and hence:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; sample(colnames(model$data[[1]]),1)&#xA;[1] &quot;b&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Its possible that because this is a &quot;kohonen&quot; class object that there are functions that get these data matrices for you. You'll need to read the documentation to figure this out. What I've shown above is digging in the structure to find the data you want.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-03-20T17:55:36.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17718" PostTypeId="1" AcceptedAnswerId="17720" CreationDate="2017-03-20T19:26:11.217" Score="3" ViewCount="3557" Body="&lt;p&gt;Pandas has a method called get_dummies() that creates a dummy encoding of a categorical variable. Scikit-learn also has a OneHotEncoder that needs to be used along with a LabelEncoder. What are the pros/cons of using each of them? Also both yield dummy encoding (k dummy variables for k levels of a categorical variable) and not one-hot encoding (k-1 dummy variables), how can one get rid of the extra category? How much of a problem does this dummy encoding create in regression models (collinearity issues - a.k.a. dummy variable trap)?&lt;/p&gt;&#xA;" OwnerUserId="14487" LastActivityDate="2017-03-20T20:44:27.033" Title="Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)" Tags="|regression|scikit-learn|pandas|categorical-data|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17719" PostTypeId="1" CreationDate="2017-03-20T19:58:03.860" Score="2" ViewCount="2213" Body="&lt;p&gt;I'm using Fine Tuning with caffenet and it works really well but then I read this in Keras &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog entry on Fine Tuning&lt;/a&gt; (They use a trained VGG16 model):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;in order to perform fine-tuning, all layers should start with properly trained weights:&#xA; for instance you should not slap a randomly initialized fully-connected network on top of a pre-trained convolutional base. &#xA;This is because the large gradient updates triggered by the randomly initialized weights would wreck the learned weights in the convolutional base.&#xA; In our case this is why we first train the top-level classifier, and only then start fine-tuning convolutional weights alongside it.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as a separate step in Fine tuning they save the output of the last layer before the fully connected layer (the &quot;bottleneck features&quot;) and then they train a &quot;small fully-connected model&quot; on those features and only then they put the newly trained fully connected layer on top of the whole net and train the &quot;last convolutional block&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this always the right way to do Fine Tuning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks&lt;/p&gt;&#xA;" OwnerUserId="30182" LastActivityDate="2018-02-10T21:35:54.240" Title="Right way to Fine Tune - Train a fully connected layer as a separate step" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17720" PostTypeId="2" ParentId="17718" CreationDate="2017-03-20T20:34:46.660" Score="7" Body="&lt;p&gt;One advantage of &lt;code&gt;get_dummies&lt;/code&gt; is that it can operate on values other than integers (so you don't need the &lt;code&gt;LabelEncoder&lt;/code&gt;) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using &lt;code&gt;drop_first=True&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One advantage of scikit-learn's &lt;code&gt;OneHoteEncoder&lt;/code&gt; lies in the scikit-learn API. OHE gives you a transformer which you can apply to your training and test set separately if you specify the total number of categories. This doesn't work with &lt;code&gt;get_dummies&lt;/code&gt; ,for example, if the training set misses categories present in the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can still delete categories by simply deleting columns from the resulting numpy array (e.g. using &lt;code&gt;n_values_&lt;/code&gt; or &lt;code&gt;feature_indices_&lt;/code&gt; to see which columns correspond to the same feature). Some models work regardless, for example tree-based models. Also, L1 regularization can often set redundant features to zero (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Lasso_(statistics)&quot; rel=&quot;noreferrer&quot;&gt;Lasso regression&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-03-20T20:44:27.033" LastActivityDate="2017-03-20T20:44:27.033" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17721" PostTypeId="1" CreationDate="2017-03-20T22:13:14.097" Score="3" ViewCount="99" Body="&lt;p&gt;I am working with a client on a special project that I am going to obfuscate in this question. Basically, I'm trying to overcome some short-term, results-oriented thinking from my client. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have a model to forecast the performance of a racehorse. Your model tells your client to sell racehorse X because the probability of its performance is low (&amp;lt;10%). The horse is sold and said racehorse goes on to win 3 races. Your client says, &quot;See, we should have never let go of that racehorse! The model is wrong!&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As data scientist we can understand that anomalies happen and that this horse may have also lost some races - we were just on the wrong side of probability this time. But how do you overcome those objections with the client? How do you turn short-term thinking into a long-term outlook of predictive modeling? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-03-21T13:51:11.143" Title="How do you overcome results oriented thinking from a client?" Tags="|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17723" PostTypeId="2" ParentId="11430" CreationDate="2017-03-20T22:56:36.443" Score="2" Body="&lt;p&gt;In the following posts &lt;a href=&quot;https://stackoverflow.com/a/11156353/7742341&quot;&gt;[1]&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/a/22867877/7742341&quot;&gt;[2]&lt;/a&gt; the plotting of 3D arrows in matplotlib is discussed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly Annotation3D class (inherited from Annotation) can be created:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from mpl_toolkits.mplot3d.proj3d import proj_transform&#xA;from matplotlib.text import Annotation&#xA;&#xA;class Annotation3D(Annotation):&#xA;    '''Annotate the point xyz with text s'''&#xA;&#xA;    def __init__(self, s, xyz, *args, **kwargs):&#xA;        Annotation.__init__(self,s, xy=(0,0), *args, **kwargs)&#xA;        self._verts3d = xyz        &#xA;&#xA;    def draw(self, renderer):&#xA;        xs3d, ys3d, zs3d = self._verts3d&#xA;        xs, ys, zs = proj_transform(xs3d, ys3d, zs3d, renderer.M)&#xA;        self.xy=(xs,ys)&#xA;        Annotation.draw(self, renderer)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Further, we can define the annotate3D() function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def annotate3D(ax, s, *args, **kwargs):&#xA;    '''add anotation text s to to Axes3d ax'''&#xA;&#xA;    tag = Annotation3D(s, *args, **kwargs)&#xA;    ax.add_artist(tag)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this function annotation tags can be added to Axes3d as in example bellow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Sim6H.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Sim6H.gif&quot; alt=&quot;3D graph example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt    &#xA;from mpl_toolkits.mplot3d import axes3d&#xA;from mpl_toolkits.mplot3d.art3d import Line3DCollection&#xA;&#xA;# data: coordinates of nodes and links&#xA;xn = [1.1, 1.9, 0.1, 0.3, 1.6, 0.8, 2.3, 1.2, 1.7, 1.0, -0.7, 0.1, 0.1, -0.9, 0.1, -0.1, 2.1, 2.7, 2.6, 2.0]&#xA;yn = [-1.2, -2.0, -1.2, -0.7, -0.4, -2.2, -1.0, -1.3, -1.5, -2.1, -0.7, -0.3, 0.7, -0.0, -0.3, 0.7, 0.7, 0.3, 0.8, 1.2]&#xA;zn = [-1.6, -1.5, -1.3, -2.0, -2.4, -2.1, -1.8, -2.8, -0.5, -0.8, -0.4, -1.1, -1.8, -1.5, 0.1, -0.6, 0.2, -0.1, -0.8, -0.4]&#xA;group = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 2, 2, 2, 3, 3, 3, 3]&#xA;edges = [(1, 0), (2, 0), (3, 0), (3, 2), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0), (9, 0), (11, 10), (11, 3), (11, 2), (11, 0), (12, 11), (13, 11), (14, 11), (15, 11), (17, 16), (18, 16), (18, 17), (19, 16), (19, 17), (19, 18)]&#xA;xyzn = zip(xn, yn, zn)&#xA;segments = [(xyzn[s], xyzn[t]) for s, t in edges]                &#xA;&#xA;# create figure        &#xA;fig = plt.figure(dpi=60)&#xA;ax = fig.gca(projection='3d')&#xA;ax.set_axis_off()&#xA;&#xA;# plot vertices&#xA;ax.scatter(xn,yn,zn, marker='o', c = group, s = 64)    &#xA;# plot edges&#xA;edge_col = Line3DCollection(segments, lw=0.2)&#xA;ax.add_collection3d(edge_col)&#xA;# add vertices annotation.&#xA;for j, xyz_ in enumerate(xyzn): &#xA;    annotate3D(ax, s=str(j), xyz=xyz_, fontsize=10, xytext=(-3,3),&#xA;               textcoords='offset points', ha='right',va='bottom')    &#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30189" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.150" LastActivityDate="2017-03-20T23:43:39.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17724" PostTypeId="1" AcceptedAnswerId="17730" CreationDate="2017-03-20T23:27:38.020" Score="0" ViewCount="91" Body="&lt;p&gt;I'm the product manager for an online app.  I'm currently researching a new feature where our users will be able to access &quot;all their raw application data&quot;.  This data is likely to be used by data scientists, it's also likely to be loaded into BI tools.  The datasets will contain up to a few million rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I actually expose the data in a practical sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Online datasource like Amazon Redshift&lt;/li&gt;&#xA;&lt;li&gt;Some other RDBMS available online (e.g. a dedicated postgres installation)&lt;/li&gt;&#xA;&lt;li&gt;CSV files available on S3&lt;/li&gt;&#xA;&lt;li&gt;CSV files available for download in a web interface&lt;/li&gt;&#xA;&lt;li&gt;Dumps into Google sheets&lt;/li&gt;&#xA;&lt;li&gt;Doesn't matter as decent data scientists can easily handle and automate anything&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="25798" LastEditorUserId="21" LastEditDate="2017-05-19T16:08:51.487" LastActivityDate="2017-05-19T16:08:51.487" Title="How should I expose data from my app for data scientists?" Tags="|dataset|data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17725" PostTypeId="2" ParentId="17719" CreationDate="2017-03-21T01:50:36.360" Score="1" Body="&lt;p&gt;First of all, you can do this in automatic way by setting the layers to be trainable or not. By selecting this hyper parameter to be false you freez the layer. we know that the first few layers are features extraction and the assumption when you perform fine tuning that the original problem and your problem are correlated, thus the features should be also the same. &#xA;What is the right thing to do? &#xA;It is very difficult to find a direct and absolute answer, it depends mainly on the original problem and the new one. In fact, I prefer to fine tune the whole network using different learning rates. &#xA;You can use smaller learning rate for the convolution layers and larger rate with the fully connected layers. &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-03-21T01:50:36.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17726" PostTypeId="2" ParentId="17640" CreationDate="2017-03-21T05:53:35.167" Score="0" Body="&lt;p&gt;Yes, I am asking about Backpropagation algorithm. I want to understand how the weights get assigned to the neural network. If we want to assign a weight matrix to the neural network, how will we assign it?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastActivityDate="2017-03-21T05:53:35.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17727" PostTypeId="1" CreationDate="2017-03-21T06:34:13.777" Score="1" ViewCount="1136" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The dataset which was extracted from the database consists of more than 50         columns, I call these columns &lt;em&gt;dimension&lt;/em&gt;s, can I call them &lt;em&gt;dimension&lt;/em&gt;s?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Obviously, I have to do &lt;em&gt;dimension reduction&lt;/em&gt; on them. But since PCA like algorithms often do axis rotating to generate some new axises. I don't think I will PCA algorithm in &lt;em&gt;dimensions reduction&lt;/em&gt;. So I calculated the correlations between these columns(parameters), and filtered these who has a high value and some other rules. So can I still call it &lt;em&gt;dimensions reduction&lt;/em&gt;? Since I only did some &lt;em&gt;parameters filtering&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The reason I don't use PCA like algorithms is because I want to implement &lt;strong&gt;Neural Network&lt;/strong&gt; classification, and I need the real parameters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please comment on these, anything even criticizing is welcomed.&lt;/p&gt;&#xA;" OwnerUserId="26856" LastActivityDate="2020-11-14T01:08:11.050" Title="PCA on Neural Networks dimensions reduction?" Tags="|machine-learning|neural-network|dimensionality-reduction|pca|" AnswerCount="3" CommentCount="0" ClosedDate="2017-03-22T16:47:34.887" ContentLicense="CC BY-SA 3.0" />
  <row Id="17728" PostTypeId="1" CreationDate="2017-03-21T07:29:58.680" Score="2" ViewCount="1000" Body="&lt;p&gt;I am trying to start training Imagenet classification training using Tensorflow's inception model. I am a bit confused as I am not sure how to fully train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wherever I go everyone seems to be using the 1000 classes training list. However, while digging in tensorflows inception model files I found a list of labels containing 21000+ classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am a bit confused as to whether I should train using the 1000 classes that everyone seems to be using or should I train on the 21000 classes. I have the full imagenet data and I checked the images I downloaded and almost all 21000 classes are there (missing only 200 classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really want to train against the full imagenet dataset as this should give more accuracy but I am not sure what the downfalls will be or how to properly prepare the data and how many steps are appropriate for that.&lt;/p&gt;&#xA;" OwnerUserId="25225" LastActivityDate="2017-07-21T07:46:06.480" Title="Understanding Imagenet training" Tags="|tensorflow|image-classification|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17730" PostTypeId="2" ParentId="17724" CreationDate="2017-03-21T08:33:08.423" Score="0" Body="&lt;p&gt;Usually, people are indeed quite indifferent to the format, as long as there is an easy way in which they can transfer it to their favorite format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CSV is a very common format, since most tools can load it.&#xA;Note that there are some dataset whose CSV representation is inconvenient (e.g., the text has many commas, data set in which the type is importnat yet hard to deduce). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the storage of the format most people are even more indifferent to the source too. Web interface is the common option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use a relational database you gain few advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The dataset is structured so you are protected from wrong structuring problems.&lt;/li&gt;&#xA;&lt;li&gt;The dataset can be naturally updated (e.g., you can keep appending records to it on a daily basis).&lt;/li&gt;&#xA;&lt;li&gt;You can allow the users load only part of the dataset, which is convenient with large dataset (e.g., give me just America users from the last month).&lt;/li&gt;&#xA;&lt;li&gt;In case that you are willing to enable that, you can let the users work on your database directly (e.g., as Google Big Query host github data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To summarize, the location of the information is not important.&#xA;Unless you need one of the advantages provided by a database, use a CSV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Need a database:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Online datasource like Amazon Redshift &lt;/li&gt;&#xA;&lt;li&gt;Some other RDBMS available online (e.g. a dedicated postgres installation)&lt;/li&gt;&#xA;&lt;li&gt;Big query&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The raw data is enough&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;CSV files available on S3 &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CSV files available for download in a web interface&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Dumps into Google sheets&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General true&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Doesn't matter as decent data scientists can easily handle and automate anything (anything reasonable but yet, it is nice that you are looking for the most convenient format).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-21T08:33:08.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17731" PostTypeId="2" ParentId="17727" CreationDate="2017-03-21T11:13:55.710" Score="0" Body="&lt;p&gt;The approach you use to do dimensions reduction is agnostic to the method you use for classification. You can use PCA to preprocess your data before to train any type of classifier, including artificial neural networks if that's what you want to use.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-03-21T11:13:55.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17732" PostTypeId="2" ParentId="17721" CreationDate="2017-03-21T11:32:54.617" Score="1" Body="&lt;p&gt;I think that you should manage the customer expectations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, try to define and use a common terminology.&#xA;You spoke about a low probability of performance while your customer spoke about winning races.&#xA;Can you tie the terms together?&#xA;For example, if the probability of performance is the probability of winning a race, once that you have a common terminology you can explain to the customer that the probability of a low performer to win 3 races in not zero and given X horses, you are expecting to see Y such horses, even if the model is perfect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Short term vs. long term issues, should be discussed and incorporated into the concept that you would like to predict. I'm not familiar with horse races but I assume that there are horses that are sprinter and can win a single race while there are horses with more stamina that can win a full match. If there are such differences of interest, they should guide the whole research process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you should explain to the customer that the model being wrong, doesn't means it is not useful. Indeed, what I have in mind in &lt;a href=&quot;https://en.wikipedia.org/wiki/George_E._P._Box&quot; rel=&quot;nofollow noreferrer&quot;&gt;Box&lt;/a&gt;'s &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;essentially, all models are wrong, but some are useful&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can help the customer see the benefit by comparing your model to some benchmark model (e.g., assuming that all horses are fast/slow, a simple approximation, a common domain knowledge estimation, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we understand that perfect prediction is not a requirement to benefit, give your customer additional power. Tell your customer that your model will have errors, but he can choose to which side he prefers that the errors will fall. He can do that by assigning a cost to false positive and false negatives. Given the costs, you can tune the model to better fit his needs (e.g., by choosing a confidence level that optimize the cost).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once that you have a model, compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Confusion_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;confusion matrix&lt;/a&gt; and discuss it with the customer. Explain to him the precision and recall and their meaning (e.g., the model will recommend to sell x% horses that are indeed good ones).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least, incase that it is an ongoing project, you should use the errors as a source of improvement. The details of improvement methods depend on the specific project. However, at least go over these cases and try to understand why did your model fail on them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-21T11:32:54.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17733" PostTypeId="1" CreationDate="2017-03-21T11:38:29.923" Score="0" ViewCount="217" Body="&lt;p&gt;I have three sets of data that were sampled at the same interval (20hz). The data is from a data logger that measured vehicle speed at a specific time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each of the tests, the Car starts at the same starting point A, and is manually drive to the end point B. This was performed 3 times. The point at which the vehicle comes to a stop is different each time since it is manually driven but that is intended.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intention is to gather enough data sets from this test route so if another person drives the test route, we can compare how he performed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would ultimately like to average the data sets from my run and call it the optimal path and add an error band, so if someone else runs the test route, I can compare how they did to it, like distance from end of route, and max speed through a turn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially I thought I could plot the data vs distance and take the average that way but the problem I am running into is, log A is 400 samples, log B is 420 samples, log C is 440 samples. So they arent exactly 1:1. Any idea how to skin this cat?&lt;/p&gt;&#xA;" OwnerUserId="30206" LastActivityDate="2019-04-24T16:02:15.460" Title="How to Standardize/Normalize three sets of test data of different lengths" Tags="|machine-learning|statistics|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17734" PostTypeId="2" ParentId="17727" CreationDate="2017-03-21T12:01:00.510" Score="0" Body="&lt;p&gt;If your data consists of observations in rows and each column is a variable for that observation, then you can call these columns dimensions. These can also be called features of the observation.&lt;/p&gt;&#xA;&lt;p&gt;Dropping columns/features from your dataset is essentially dimensionality reduction. You go from a higher dimensional space to a lower dimensional one  (less columns).&#xA;The fact that you're not combining features, rotating spaces doesn't mean it's not a valid form of dimensionality reduction.&lt;/p&gt;&#xA;&lt;p&gt;Why are you against using PCA for Neural Networks in particular? PCA doesn't care about what kind of classification you're running after it. In fact there is a convenience &lt;a href=&quot;https://artax.karlin.mff.cuni.cz/r-help/library/caret/html/pcaNNet.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;function&lt;/a&gt; in R's caret package implement PCA before Neural nets to ease computation of the NN.&lt;/p&gt;&#xA;" OwnerUserId="22000" LastEditorUserId="98307" LastEditDate="2020-08-11T07:09:06.080" LastActivityDate="2020-08-11T07:09:06.080" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17737" PostTypeId="1" AcceptedAnswerId="17750" CreationDate="2017-03-21T12:53:56.477" Score="12" ViewCount="15686" Body="&lt;p&gt;I work with analyzing EEG data, which will eventually need to be classified. However, obtaining labels for the recordings is somewhat expensive, which has led me to consider unsupervised approaches, to better utilize our quite large amounts of unlabeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This naturally leads to considering stacked autoencoders, which may be a good idea. However, it would also make sense to use convolutional neural networks, since some sort of filtering is generally a very useful approach to EEG, and it is likely that the epochs considered should be analyzed locally, and not as a whole. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way to combine the two approaches? It seems that when people use CNN's they generally use supervised training, or what? The two main benefits of exploring neural networks for my problem seem to be the unsupervised aspect, and the fine-tuning (it would be interesting to create a network on population data, and then fine tune for an individual, for instance). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, does anyone know if I could just pretrain a CNN as if it was a &quot;crippled&quot; autoencoder, or would that be pointless?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I be considering some other architecture, like a deep belief network, for instance?&lt;/p&gt;&#xA;" OwnerUserId="30210" LastEditorUserId="29169" LastEditDate="2021-02-16T02:45:45.790" LastActivityDate="2021-02-16T02:45:45.790" Title="Does it make sense to train a CNN as an autoencoder?" Tags="|deep-learning|convolutional-neural-network|unsupervised-learning|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17738" PostTypeId="2" ParentId="11928" CreationDate="2017-03-21T13:05:43.230" Score="6" Body="&lt;p&gt;I faced similar problem and saw that numpy handles NaN and Inf differently.&lt;br&gt;&#xA;Incase if you data has Inf, try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.where(x.values &amp;gt;= np.finfo(np.float64).max)&#xA;Where x is my pandas Dataframe &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will be giving a tuple of location of places where NA values are present. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incase if your data has Nan, try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.isnan(x.values.any())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28482" LastActivityDate="2017-03-21T13:05:43.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17739" PostTypeId="2" ParentId="17733" CreationDate="2017-03-21T13:25:00.670" Score="1" Body="&lt;p&gt;Here is one idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First plot the time vs distance. These lines would have different slopes between the measurement periods and you would essentially estimate the speed between measurements. Here is a graph with one measurement window:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kNBeO.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kNBeO.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since they're racing on the same track, the total distance driven should be approximately equal. You might need to stretch or shrink the height of lines to make sure they travel the same distance. (This might not be a good assumption for your case.)&#xA;Now, go from the y-axis and for regular distance intervals, look up the slope of the graph.&#xA;Then plot the observed estimated speed for each distance period. You'll end up with speed measurements at regular distances which will now have the same length of samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way you'll have a fairly good approximation of the speed at which racers traveled at specific parts of the track. I.e. you can compare cornering speed etc.&lt;/p&gt;&#xA;" OwnerUserId="22000" LastActivityDate="2017-03-21T13:25:00.670" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17740" PostTypeId="2" ParentId="17721" CreationDate="2017-03-21T13:51:11.143" Score="4" Body="&lt;p&gt;The scenario you described with &lt;strong&gt;one&lt;/strong&gt; horse is basically anecdotal evidence. To objectively evaluate what you're doing, you need to agree with the client to make this an experiment. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I understand you don't trust my horse predictions and that's totally&#xA;  understandable. It is human nature to worry about the ones that got&#xA;  away. Let's handle this objectively and run an experiment. I'll give&#xA;  you buy/sell ratings on 50 horses. We then track their performance for&#xA;  six months. If on average my ratings outperform a coin flip, we'll&#xA;  know we're onto something at a macro level even if a few got away. If&#xA;  my models end up being worse than a coinflip, we'll know the model&#xA;  needs to be improved or you need to fire me.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-21T13:51:11.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17742" PostTypeId="2" ParentId="17695" CreationDate="2017-03-21T14:23:16.050" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Decide how auto-correlative your usual event in the time series is.&#xA;For example, &quot;I'm tracking temperature over time and it rarely changes more&#xA;than 30 degrees F in an hour&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Throw out or smooth any values where the observed value changes&#xA;&lt;em&gt;more&lt;/em&gt; than that. In other words, &quot;If ever I see the temperature&#xA;changing more than 30 degrees in an hour, I'm going to ignore that value and substitute the average of the prior and the next value because that must be a sensor malfunction&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Once you were comfortable with doing that, use something like the standard deviation of the data over a rolling window instead of an absolute, arbitrary value like I did.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="30212" LastEditDate="2017-03-21T18:47:59.187" LastActivityDate="2017-03-21T18:47:59.187" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17744" PostTypeId="2" ParentId="17737" CreationDate="2017-03-21T14:29:05.293" Score="4" Body="&lt;p&gt;Yes, you can use a convolutional network in an autoencoder setup. There is nothing strange with it. People have problems figuring out &lt;a href=&quot;https://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers&quot;&gt;&lt;em&gt;deconvolution&lt;/em&gt; layers&lt;/a&gt;, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can find an example of a convolutional autoencoder using Keras framework: &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.keras.io/building-autoencoders-in-keras.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-03-21T14:29:05.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17745" PostTypeId="2" ParentId="17645" CreationDate="2017-03-21T15:35:05.260" Score="1" Body="&lt;p&gt;This is the summary of what I have researched so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fundamentally:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;: A model is a set of rules that fit/represent the trends/rules in data supplied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Overfitting&lt;/strong&gt;: Overfitting in general sense is modeling of noise/randomness along with the sample where the effect of noise affects model result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the fundamentals at hand, one can have an intuition that, WHEN YOU FIT, THERE IS A CHANCE THAT YOU CAN OVERFIT. i.e. When you can model something that is REQUIRED, there is a good chance you can model something which is NOT REQUIRED.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, YES, OVERFITTING IS POSSIBLE IN UNSUPERVISED LEARNING.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If PCA can be used to remove/reduce overfitting in Unsupervised Learning?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supervised learning uses &lt;strong&gt;labels&lt;/strong&gt; as a comparison measure, i.e. 2 samples are compared in &lt;strong&gt;data&lt;/strong&gt;(feature sets, feature vectors or whatever jargon one throws in here.), &lt;strong&gt;w.r.t. their labels to identify patterns.&lt;/strong&gt;&#xA;So, PCA is a technique which does not consider Labels. So, removal of data with PCA is not preferred for supervised, as it may &lt;em&gt;remove data for which feature may not have enough information but labels do.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So PCA is not recommended for removing Overfitting for Supervised Learning.&#xA;You can use it, if at all, with the risk that you may lose information from your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unsupervised learning does not have labels, instead, it inter-compares 2 samples to identify patterns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, there is &lt;strong&gt;NO data for which feature may not have enough information but labels do&lt;/strong&gt; as labels don't exist. So, PCA will help you reduce dimensionality as it would tend to defer data that doesn't add much information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said above, it is NOT necessary that it would definitely help you reduce overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, it's worth a try. As, if the noise is dominant in the data, there is a definite pattern in the data, and your model is just abstracting it which drills down to modeling your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, PCA can help you reduce overfitting in your data, and to the question that is it a good practice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't come across an article or reasoning that defers it for Unsupervised. Regardless of everything, &lt;em&gt;PCA does seem a practical approach to reducing Overfitting for Unsupervised Learning without loss of information.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="23773" LastEditorUserId="23773" LastEditDate="2017-03-21T17:36:14.723" LastActivityDate="2017-03-21T17:36:14.723" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17746" PostTypeId="2" ParentId="17727" CreationDate="2017-03-21T16:11:00.403" Score="0" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;The dataset which was extracted from the database consists of more than 50 columns&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;You can call them &lt;strong&gt;Features&lt;/strong&gt;. &lt;strong&gt;Feature Set&lt;/strong&gt; is something that you operate on.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Obviously, I have to do dimension reduction on them. But since PCA like algorithms often does axis rotating to generate some new axises. I don't think I will PCA algorithm in dimensions reduction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I am not sure if you understand PCA completely. PCA does not generate new &lt;strong&gt;random&lt;/strong&gt; axis, but an axis, that represents the MOST variance of your data. So, PCA will surely perform dimension reduction given that data has some values which don't add info to your model.&#xA;You can call this &lt;em&gt;Feature Extraction&lt;/em&gt; or &lt;em&gt;Feature Filtering&lt;/em&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The reason I don't use PCA like algorithms is because I want to implement Neural Network classification, and I need the real parameters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;What makes you think, post PCA you don't have '&lt;strong&gt;real&lt;/strong&gt;' parameters. It's just a different view of the same data.(Imagine these axes as eyes, and your data as the object, you just have a different view of the same object.)&lt;/p&gt;&#xA;&lt;p&gt;The only problem I see is the axes on which the components are projected are orthogonal, which if you want to avoid, you can go ahead with ICA (Independent Component Analysis)&lt;/p&gt;&#xA;" OwnerUserId="23773" LastEditorUserId="43000" LastEditDate="2020-11-14T01:08:11.050" LastActivityDate="2020-11-14T01:08:11.050" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17749" PostTypeId="2" ParentId="16895" CreationDate="2017-03-21T20:29:21.047" Score="3" Body="&lt;h3&gt;Short answer&lt;/h3&gt;&#xA;&lt;p&gt;There are &lt;em&gt;far&lt;/em&gt; faster ways vectorized ways to do this than brute-force loops. One particular option while remaining Pandas-level would be&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;(tra_df.groupby(tra_df.columns.tolist())&#xA;       .size()&#xA;       .reindex(tst_df.values.T.tolist(), fill_value=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This should offer you an enormous performance boost, which could be further improved with a NumPy vectorized solution, depending on what you're satisfied with.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;Long answer&lt;/h3&gt;&#xA;&lt;p&gt;Suppose that your training and test data looked something like this.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd; import numpy as np; from numpy.random import RandomState&#xA;&#xA;rnd_st = RandomState(8675309)&#xA;&#xA;tra_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3), &#xA;                           B=rnd_st.randint(0, 10, 10**3)))&#xA;tst_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3), &#xA;                           B=rnd_st.randint(0, 10, 10**3)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;First of all, you aren't really using Pandas correctly with&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def f(A, B, x, y):&#xA;    data = df.loc[df[A].isin([x]) &amp;amp; df.B.isin([y])]&#xA;    return len(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.isin.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;isin&lt;/code&gt;&lt;/a&gt; isn't intended to match against single values like this, but rather a list or set of values when you don't have another option. In your case, you can achieve the same boolean index directly with&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;(df.A == x) &amp;amp; (df.B == y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Furthermore, there's no point indexing your DataFrame again with &lt;code&gt;df.loc&lt;/code&gt; only to take the length of it... you might as well just use &lt;code&gt;sum&lt;/code&gt; directly on the boolean vector to count the matches. Putting this together and using &lt;code&gt;apply&lt;/code&gt; over the test DataFrame, you can achieve your same output with&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;tst_df.apply(lambda x: ((tra_df.A == x.A) &amp;amp; (tra_df.B == x.B)).sum(), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But this is still very slow - wrapping the above approach in a function and comparing it to your original approach on the sample data&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;%timeit your_way(tra_df, test_df)&#xA;1 loops, best of 3: &lt;b&gt;819 ms per loop&lt;/b&gt;&#xA;%timeit direct_er_way(tra_df, test_df)&#xA;1 loops, best of 3: &lt;b&gt;673 ms per loop&lt;/b&gt;&#xA;&lt;/code&gt;&lt;/pre&gt; &#xA;&lt;p&gt;Much faster would be to use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;groupby&lt;/code&gt;&lt;/a&gt; and then &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reindex.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;reindex&lt;/code&gt;&lt;/a&gt;, as instead of brute-force looping this offers a vectorized solution where we are effectively hashing the counts.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;(tra_df.groupby(tra_df.columns.tolist())&#xA;       .size()&#xA;       .reindex(tst_df.values.T.tolist(), fill_value=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Benchmarking,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; %timeit groupby_way()&#xA;100 loops, best of 3: &lt;b&gt;3.04 ms per loop&lt;/b&gt;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I'm not that well-versed in NumPy, but I can safely assume that were this function still not fast enough to meet your needs then a NumPy vectorized solution avoiding some of the overhead would be the next step.&lt;/p&gt;&#xA;" OwnerUserId="30179" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-03-21T20:29:21.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17750" PostTypeId="2" ParentId="17737" CreationDate="2017-03-21T22:00:07.273" Score="10" Body="&lt;p&gt;Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (convolutional and/or stacked) autoencoders.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1511.04306&quot; rel=&quot;noreferrer&quot;&gt;Deep Feature Learning for EEG Recordings&lt;/a&gt; uses convolutional autoencoders with custom constraints to improve generalization across subjects and trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0923596516300832&quot; rel=&quot;noreferrer&quot;&gt;EEG-based prediction of driver's cognitive performance by deep convolutional neural network&lt;/a&gt; uses convolutional deep belief networks on single electrodes and combines them with fully connected layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://iopscience.iop.org/article/10.1088/1741-2560/14/1/016003/meta;jsessionid=A1E21FCD5521812068616ABB45BFF236.c4.iopscience.cld.iop.org&quot; rel=&quot;noreferrer&quot;&gt;A novel deep learning approach for classification of EEG motor imagery signals&lt;/a&gt; uses fully connected stacked autoencoders on the output of a supervisedly trained (fairly shallow) CNN.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also purely supervised CNNs have had success on EEG data, see for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1611.08024&quot; rel=&quot;noreferrer&quot;&gt;EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1703.05051&quot; rel=&quot;noreferrer&quot;&gt;Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG&lt;/a&gt; (disclosure: I am the first author of this work, more related work see p. 44)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the EEGNet paper shows that also with a smaller number of trials, purely supervised training of their CNN can outperform their baselines (see Figure 3). Also in our experience on a dataset with only 288 training trials, purely supervised CNNs work fine, slightly outperforming a traditional filter bank common spatial patterns baseline.&lt;/p&gt;&#xA;" OwnerUserId="27704" LastActivityDate="2017-03-21T22:00:07.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17752" PostTypeId="1" CreationDate="2017-03-22T00:57:47.440" Score="6" ViewCount="1758" Body="&lt;p&gt;Is anyone aware of any open source / python packages for Netflow Anomaly detection  ? I found some on github but anyone who has more experience with it. please advise.&lt;/p&gt;&#xA;" OwnerUserId="30229" LastEditorUserId="924" LastEditDate="2017-03-22T06:53:20.343" LastActivityDate="2020-08-08T21:01:00.070" Title="Netflow anomaly detection python packages" Tags="|machine-learning|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17753" PostTypeId="2" ParentId="17507" CreationDate="2017-03-22T02:18:52.117" Score="1" Body="&lt;p&gt;Clustering cannot inherently extract labelled classes. If you have labels then you should use those with a supervised algorithm. There is no reason that any clustering should agree with pre-provided class labels. Imagine a data set that was perfectly uniformly distributed. One could have class labels for that data that can be quite arbitrary (you could even restrict to arbitrary convex regions). What clustering result should you expect for perfectly uniformly distributed data? Is there any reason that should ever be able to match any particular class labelling other than by chance? Clustering corresponds to the something about the distributional properties of your data set. Unless the class labels &lt;em&gt;happen to align&lt;/em&gt; with those distributional properties (and there is no reason that they should) then clustering cannot recover the labels.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-03-22T02:18:52.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17754" PostTypeId="2" ParentId="16454" CreationDate="2017-03-22T02:44:32.673" Score="1" Body="&lt;p&gt;Essentially, the PMML is just a set of formulas (and the corresponding coefficient values). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The story goes something like this:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You develop a model capable of taking some inputs and outputting certain coefficients that correspond to a certain formula.&lt;/li&gt;&#xA;&lt;li&gt;Then, these ouputs are turned into a PMML file (an XML-based encoding of your model's outputs).&lt;/li&gt;&#xA;&lt;li&gt;We can now say, &quot;please evaluate this formula, using these coefficients. As input for the independent variable(s), please use these attributes from my dataset here.&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Take a very simple linear regression, for example.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The PMML consumer application parses your PMML file, and finds&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;model = &lt;/li&gt;&#xA;&lt;li&gt;coefficient = &amp;lt;5.46&gt;&lt;/li&gt;&#xA;&lt;li&gt;X = &amp;lt; attribute:  colour of the leaf &gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;It now goes ahead and scores the formula for the reggression&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: there’s got a to a file somewhere in your application which contains a column called “colour of the leaf”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, there you have it.&lt;/p&gt;&#xA;" OwnerUserId="17240" LastActivityDate="2017-03-22T02:44:32.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17755" PostTypeId="1" AcceptedAnswerId="17835" CreationDate="2017-03-22T04:04:33.683" Score="11" ViewCount="2181" Body="&lt;p&gt;I feel that this question is related to the theory behind cross-validation. I present my empirical finding here and wrote a question related to the theory of cross-validation at &lt;a href=&quot;https://stats.stackexchange.com/questions/269997/why-models-with-better-cross-validation-performance-are-not-necessarily-do-bette&quot;&gt;there&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two models M1 and M2, I use the same data set to train them and perform cross validation using that same data set to find the optimal parameters for each model. Say eventually I found that M1 under its optimal parameter, performs better than M2 under its optimal parameter in terms of the 10-fold cross validation score. Now if I have another independent test data set with both predictors and labels and this test data set is generated from the same distribution of my training data set, then before I apply these 2 well-tuned model on that new test data set, can I claim or should I expect to see that M1 will still perform better than M2 over that new test data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was playing Kaggle Titanic example. I have 2 xgboost model, M1 is well-tuned and M2 is less well-tuned in the sense that M1 has a better 10 fold cross validation performs on the training data set. But then when I submit both, I found that the less well-tuned model actually has a better scores on the test data set. How could that be? And if it is true, then what should we looking for when we fit the data to different models and tune the model parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my specific submission results: I did a random grid search&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params_fixed = {'silent': 1,'base_score': 0.5,'reg_lambda': 1,&#xA;'max_delta_step': 0,'scale_pos_weight':1,'nthread': 4,&#xA;'objective': 'binary:logistic'}&#xA;params_grid = {'max_depth': list(np.arange(1,10)),&#xA;'gamma': [0,0.05,0.1,0.3, 0.5,0.7,0.9],&#xA;'n_estimators':[1,2,5,7,10,15,19,25,30,50], &#xA;'learning_rate': [0.01,0.03,0.05,0.1,0.3,0.5,0.7,0.9,1],&#xA;'subsample': [0.5,0.7,0.9], 'colsample_bytree': [0.5,0.7,0.9], &#xA;'min_child_weight': [1,2,3,5], 'reg_alpha': [1e-5, 1e-2, 0.1, 0.5,1,10]&#xA;}&#xA;rs_grid = RandomizedSearchCV(&#xA;          estimator=XGBClassifier(**params_fixed, seed=seed),&#xA;          param_distributions=params_grid,&#xA;          n_iter=5000,   &#xA;          cv=10,&#xA;          scoring='accuracy',&#xA;          random_state=seed&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each time I change the variable &lt;code&gt;n_iter&lt;/code&gt;. First, I set &lt;code&gt;n_iter=10&lt;/code&gt;, it gives me a set of values of those hyper parameters, let's call this vector $\alpha_1$ and the cv score (accuracy rate) is &lt;strong&gt;0.83389&lt;/strong&gt;, then I use $\alpha_1$ to train my model and generate prediction on the independent test data set, and when I submit to Kaggle it generates true accuracy on the test data set &lt;strong&gt;0.79426&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, I set &lt;code&gt;n_iter=100&lt;/code&gt;, it gives me $\alpha_2$ and the cv score is &lt;strong&gt;0.83614&lt;/strong&gt;, i.e., higher than the first one, makes sense, but when I submit to Kaggle, &lt;strong&gt;0.78469&lt;/strong&gt;, lower than the first one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third, I set &lt;code&gt;n_iter = 1000&lt;/code&gt;, it gives me $\alpha_3$ and the cv score is &lt;strong&gt;0.83951&lt;/strong&gt;, i.e., higher than the second one, makes sense, but when I submit to Kaggle, &lt;strong&gt;0.77990&lt;/strong&gt;, lower than the second one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourth, I set &lt;code&gt;n_iter = 5000&lt;/code&gt;, it gives me $\alpha_4$ and the cv score is &lt;strong&gt;0.84512&lt;/strong&gt;, i.e., higher than the third one, makes sense, but when I submit to Kaggle, &lt;strong&gt;0.72249&lt;/strong&gt;, lower than the third one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is really frustrated. The model is getting better and better on the cross-validation score but when performed on an actual independent data set, its performance is getting worse and worse. &lt;strong&gt;Did I interpret the CV scores in the exactly opposite way?&lt;/strong&gt; I see some paper mentioned that the CV score can be too optimistic for inferring the true test score. However, even if that is true, then I think the CV scores for all of my 4 models should be all optimistic about their own true test score, i.e., the order should preserve. But when applying on the real test data set, the order reversed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only reason I can imagine would be, that test data set has a different distribution than the training data set. However, if it is indeed the case, then I believe there is no method under then sun that can cure this problem.&lt;/p&gt;&#xA;" OwnerUserId="30170" LastEditorUserId="303" LastEditDate="2020-03-14T08:51:14.300" LastActivityDate="2020-03-14T08:51:14.300" Title="Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?" Tags="|xgboost|cross-validation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17756" PostTypeId="2" ParentId="17755" CreationDate="2017-03-22T06:33:45.243" Score="0" Body="&lt;p&gt;It is possible. Think of a simple scenario where model &lt;code&gt;M1&lt;/code&gt; has learnt the variance of the training dataset &lt;code&gt;D&lt;/code&gt; better than model &lt;code&gt;M2&lt;/code&gt; since it's parameters are better tuned. This means &lt;code&gt;M1&lt;/code&gt; performs better at &lt;code&gt;D&lt;/code&gt; than &lt;code&gt;M2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when we test them at test set &lt;code&gt;T&lt;/code&gt;, it is possible that &lt;code&gt;M2&lt;/code&gt; performs better as &lt;code&gt;M1&lt;/code&gt; might be overfitting &lt;code&gt;D&lt;/code&gt; while &lt;code&gt;M2&lt;/code&gt; was not. Hence &lt;code&gt;M1&lt;/code&gt; performs worse at &lt;code&gt;T&lt;/code&gt; than &lt;code&gt;M2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might be due to the fact that you performed your cross validation on the same dataset instead of a validation set. If you train and validate at the same set, you are likely to miss the fact that it might be overfitting. Thus it is always better to train, validate and test at different sets of data. So flow should be &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train different models at same training set &lt;/li&gt;&#xA;&lt;li&gt;Validated at validation set&lt;/li&gt;&#xA;&lt;li&gt;Choose the best performing model basis performance at validation set&lt;/li&gt;&#xA;&lt;li&gt;Use it to score your test set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2017-03-22T06:33:45.243" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="17757" PostTypeId="2" ParentId="17469" CreationDate="2017-03-22T06:47:23.360" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, that is what distributed platforms are for.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2 Once the model is ready, then there is no need of these disributed&#xA;  platform right ? or are they needed for feature extractions ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes for data extraction, feature generation for scoring and scoring your dataset (the actual reason you trained at model for!), you would still need the platform as that data is also likely to be huge.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastEditorUserId="10904" LastEditDate="2018-04-30T18:33:11.157" LastActivityDate="2018-04-30T18:33:11.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17759" PostTypeId="1" AcceptedAnswerId="24003" CreationDate="2017-03-22T07:43:57.223" Score="44" ViewCount="39331" Body="&lt;p&gt;Is it better to encode features like month and hour as factor or numeric in a machine learning model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the one hand, I feel numeric encoding might be reasonable, because time is a forward progressing process (the fifth month is followed by the sixth month), but on the other hand I think categorial encoding might be more reasonable because of the cyclic nature of years and days ( the 12th month is followed by the first one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general solution or convention for this? &lt;/p&gt;&#xA;" OwnerUserId="30238" LastEditorUserId="30238" LastEditDate="2017-03-23T06:43:28.750" LastActivityDate="2019-05-28T23:15:04.640" Title="Encoding features like month and hour as categorial or numeric?" Tags="|machine-learning|feature-extraction|feature-engineering|encoding|numerical|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17760" PostTypeId="2" ParentId="16205" CreationDate="2017-03-22T07:56:40.270" Score="1" Body="&lt;p&gt;Use &lt;code&gt;tf.softmax_cross_entropy_with_logits&lt;/code&gt; function instead of writing it by yourself because &lt;code&gt;tf.softmax_cross_entropy_with_logits&lt;/code&gt; is more computationally stable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try this,&#xA;&lt;code&gt;loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refrence: &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2017-03-22T07:56:40.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17761" PostTypeId="2" ParentId="17759" CreationDate="2017-03-22T08:10:17.610" Score="1" Body="&lt;p&gt;Because of all the data you have is well defined I would suggest you a categorical encoding, which is also easier to apply.&lt;/p&gt;&#xA;" OwnerUserId="30240" LastActivityDate="2017-03-22T08:10:17.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17762" PostTypeId="2" ParentId="17629" CreationDate="2017-03-22T08:23:05.640" Score="1" Body="&lt;p&gt;There are some suggestions which I think can improve CNN performance:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Input size is (6, 3) so it is not preferable to use &lt;code&gt;MaxPooling&lt;/code&gt; layers.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;padding = 'same'&lt;/code&gt; in all the convolution layers which output convolved image of the same dimension as input.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;Relu&lt;/code&gt; or &lt;code&gt;LeakyRelu&lt;/code&gt; as activation function.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;Adam optimizer&lt;/code&gt; and tune the learning rate.&lt;/li&gt;&#xA;&lt;li&gt;Convert some of the &lt;code&gt;Dense&lt;/code&gt; layers to &lt;code&gt;1x1 Convolution with no. of filters = no. of units in dense layer&lt;/code&gt; as suggested by sh37211&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;no_of_filters = [32, 64, 64, 32]&#xA;kernel_size = [3, 3, 3, 1]&#xA;&#xA;input = Input(shape = (6,3,3), name = &quot;input&quot;)&#xA;layer_output = [input]&#xA;for i in range(4):&#xA;    convolution = Convolution2D(no_of_filters[i], kernel_size[i], kernel_size[i], padding = 'same')(layer_output[-1])&#xA;    activation = LeakyReLU()(convolution)&#xA;    layer_output.append(activation)&#xA;&#xA;flatten = Flatten()(layer_output[-1])&#xA;flatten_dropout = Dropout(0.5)(flatten)&#xA;fc = Dense(output_dim = 13)&#xA;activation = LeakyReLU()(fc)&#xA;layer_output.append(activation)&#xA;fc = Dense(output_dim = 1, activation = 'tanh')&#xA;&#xA;model.compile(loss = 'mean_squared_error',optimizer = 'adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14310" LastEditorUserId="14310" LastEditDate="2017-03-24T12:42:06.303" LastActivityDate="2017-03-24T12:42:06.303" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="17763" PostTypeId="2" ParentId="17759" CreationDate="2017-03-22T08:45:12.337" Score="5" Body="&lt;p&gt;It depends on which algorithm you're using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For other algorithms like neural network, I suggest trying both method(continuous &amp;amp; categorical). The effect differs between different situations.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-22T08:45:12.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17765" PostTypeId="2" ParentId="17629" CreationDate="2017-03-22T10:23:19.993" Score="1" Body="&lt;p&gt;A suggestion I give is to change the layers of CNN. You have 3 CNN layers, all sequentially added acting on the same shape and outputting the same output shape with same filter size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can try is to change the filter size and using multiple filter sizes to catch different features of different sizes. For this, try this model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;main_input = Input(shape=input_shape, name=&quot;main input&quot;)&#xA;flattened_outputs = []&#xA;for i in filter_sizes:&#xA;    conv_filter_i = Convolution1D(no_of_filters, i, border_mode='same', activation='relu', W_constraint=maxnorm(3))(main_input)&#xA;    pooling_i = MaxPooling1D(pool_length=2)(conv_filter_i)&#xA;    flattened_i = Flatten()(pooling_i)&#xA;    flattened_outputs.append(flattened_i)&#xA;merged_conv_outputs = merge(flattened_outputs, mode=&quot;concat&quot;)&#xA;softmax = Dense(output_shape, activation=&quot;softmax&quot;)(merged_conv_outputs)&#xA;model = Model(input=main_input, output=softmax)&#xA;model.compile(loss='mean_squared_error',optimizer=optimizer_mode)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: change the dimensions as necessary.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Apart from this, I suggest you to use &lt;code&gt;Dropout&lt;/code&gt; layer. &lt;/li&gt;&#xA;&lt;li&gt;It helps a lot according to my personal experience. Also, do use adam optimizer. &lt;/li&gt;&#xA;&lt;li&gt;Experiment with various pooling mechanisms. Remember that there is a lot of experimentation that has to go on, so try various other parameters and filter sizees too.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21024" LastEditorUserId="21024" LastEditDate="2017-03-23T07:35:45.907" LastActivityDate="2017-03-23T07:35:45.907" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17767" PostTypeId="1" CreationDate="2017-03-22T11:13:26.903" Score="1" ViewCount="1697" Body="&lt;p&gt;I am using &quot;segment&quot; package in R for fitting piecewise linear segments to a sample parabolic curve in R. I am not able to able to interpret the functions&#xA;&quot;psi&quot; and &quot;segZ&quot; from the R documentation. I have two variables X and Y .&#xA;If we consider X as dependent and Y as independent variable then,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;reg&amp;lt;- lm(Y~X, data = curved_data)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am creating a parabola from the sample data given below-&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;x&amp;lt;- c(-37, -36, -35, -34, -33, -32, -30, -28, -26, -25, -24, -23, -22, -21, -20, -19, -18, -17, -16, -15, -14, -12, -10, -8, -7, -5, -4, -2, -1, 0, 1, 2, 4, 5, 7, 8, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 30, 32, 33, 34, 35, 36, 37)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;y=4x^2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is I am constructing a parabola with the help of the above equation in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this case y is independent and x is the dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to have a piece wise linearity fit for this parabolala.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried using two different codes of R i.e. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;seg_model&amp;lt;- segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30), control = seg.control(display = TRUE))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-35, 4, 30 are the 3 breakpoints that are chosen by me. Are they correctly chosen? How should they be chosen?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get the error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Error in segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30),  : &#xA;      &lt;code&gt;psi' should be a list with more than one covariate in&lt;/code&gt;seg.Z'&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this error means?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I have tried using code by writing &lt;strong&gt;psi = NA&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have got the below error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Error in segmented(reg, seg.Z = ~curved_data$X, psi = NA, control = seg.control(display = TRUE)) : &#xA;      &lt;code&gt;psi' should be a list with more than one covariate in&lt;/code&gt;seg.Z'&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not able to understand &lt;strong&gt;psi&lt;/strong&gt; and &lt;strong&gt;segZ&lt;/strong&gt; from the R documentation. Please can I get an elaborate functioning of these two functions?&#xA;What is the relation between &lt;strong&gt;psi&lt;/strong&gt; and &lt;strong&gt;segZ&lt;/strong&gt;? What should I do to correct this error? Does number of points in vector of psi means number of breakpoints OR what is it?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30006" LastEditDate="2017-03-22T11:19:44.920" LastActivityDate="2017-10-13T08:52:47.053" Title="What is exactly psi and segZ function in &quot;segment&quot; function in R" Tags="|r|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17768" PostTypeId="2" ParentId="10025" CreationDate="2017-03-22T11:57:32.890" Score="4" Body="&lt;p&gt;For automatic speech recognition (ASR), filter bank features perform as good as CNN on spectrograms &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/38131.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Table 1&lt;/a&gt;. You can train a DBN-DNN system on fbank for classifying animals sounds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice longer speech utterances are divided into shorter utterances since Viterbi decoding doesn't work well for longer utterances. You could do the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can divide the longer utterances into smaller utterances of fixed length. &#xA;Dividing the longer utterances into smaller is easy. The problem comes in increasing the length the smaller utterances to reach fixed length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could warp the frequency axis of the spectrogram for augmenting the smaller utterances. This data augmentation has been shown to improve ASR performance &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/perturb.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;data augumentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a longer utterance with multiple sounds in it, you could use music segmentation algorithms to divide it into multiple utterances. These utterances can be made of fixed length either by division or augmentation. &lt;/p&gt;&#xA;" OwnerUserId="29788" LastEditorUserId="29788" LastEditDate="2017-03-23T05:16:13.450" LastActivityDate="2017-03-23T05:16:13.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17769" PostTypeId="1" AcceptedAnswerId="17770" CreationDate="2017-03-22T12:57:52.953" Score="31" ViewCount="177385" Body="&lt;p&gt;Suppose I have a 5*3 data frame in which third column contains missing value&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2 3&#xA;4 5 NaN&#xA;7 8 9&#xA;3 2 NaN&#xA;5 6 NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope to generate value for missing value based rule that first product second column &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2 3&#xA;4 5 20 &amp;lt;--4*5&#xA;7 8 9&#xA;3 2 6 &amp;lt;-- 3*2&#xA;5 6 30 &amp;lt;-- 5*6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I do it use data frame? Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to add condition to calculate missing value like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;if 1st % 2 == 0 then 3rd = 1st * 2nd&#xA;else 3rd = 1st + 2nd&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 2 3&#xA;4 5 20 &amp;lt;-- 4*5 because 4%2==0&#xA;7 8 9&#xA;3 2 5 &amp;lt;-- 3+2 because 3%2==1&#xA;5 6 11 &amp;lt;-- 5+6 because 5%2==1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29911" LastEditorUserId="29911" LastEditDate="2017-03-22T13:10:44.210" LastActivityDate="2021-08-04T20:04:05.780" Title="How to fill missing value based on other columns in Pandas dataframe?" Tags="|pandas|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17770" PostTypeId="2" ParentId="17769" CreationDate="2017-03-22T13:15:45.900" Score="33" Body="&lt;p&gt;Assuming three columns of your dataframe is &lt;code&gt;a&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;. This is what you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['c'] = df.apply(&#xA;    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],&#xA;    axis=1&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Full code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame(&#xA;    np.array([[1, 2, 3], [4, 5, np.nan], [7, 8, 9], [3, 2, np.nan], [5, 6, np.nan]]), &#xA;    columns=['a', 'b', 'c']&#xA;)&#xA;df['c'] = df.apply(&#xA;    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],&#xA;    axis=1&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2017-03-22T13:15:45.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17771" PostTypeId="2" ParentId="17769" CreationDate="2017-03-22T13:45:01.903" Score="5" Body="&lt;p&gt;Assuming that the three columns in your dataframe are &lt;code&gt;a&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt;. Then you can do the required operation like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values = df['a'] * df['b']&#xA;df['c'] = values.where(df['c'] == np.nan, others=df['c'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-03-22T13:45:01.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17772" PostTypeId="2" ParentId="16184" CreationDate="2017-03-22T13:47:34.883" Score="4" Body="&lt;p&gt;The reconstructed images from &lt;code&gt;deconvnet&lt;/code&gt; represent the patterns in the original images which are very likely to strongly activate a particular featuremap at a particular convolution layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gray portion in the reconstructed images shows that the values of those pixels in input images are not likely to strongly activate that featuremap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no such gray portion in the first layer filters because first layer filter are representing the very simple patterns(e.g. edges) and those are very likely to present in input image at everywhere.&lt;/p&gt;&#xA;" OwnerUserId="14310" LastActivityDate="2017-03-22T13:47:34.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17773" PostTypeId="1" AcceptedAnswerId="17788" CreationDate="2017-03-22T16:36:34.200" Score="1" ViewCount="1596" Body="&lt;p&gt;I have a big dataset (1million x 50) to which I want to predict a particular class. I have thought of segregating the dataset in batches of 20k. And then train a classifier (lets say random forest or a basic SVM). How do I then improve that classifier by providing it with additional dataset. In other words, how do I preserve the random forests created in iteration &lt;em&gt;i&lt;/em&gt; and use as starting model in interation &lt;em&gt;i+1&lt;/em&gt; to improve the model in python?&lt;/p&gt;&#xA;" OwnerUserId="30254" LastActivityDate="2018-09-05T17:49:19.263" Title="How to improve an existing machine learning classifier in python?" Tags="|machine-learning|python|bigdata|random-forest|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17774" PostTypeId="2" ParentId="17773" CreationDate="2017-03-22T17:36:52.613" Score="2" Body="&lt;p&gt;The simplest, and likely best, way is to just rebuild the model every time you add more data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to build on existing models, one way is to average multiple random forest models.  So you build a model on 20k, test it, then decide to build on it by building another random forest model on a separate set of 20k and averaging the model parameter results with the previous model.  This is basically a poorly described ensemble method.  Check out this: &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/ensemble.html&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-03-22T17:36:52.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17776" PostTypeId="1" AcceptedAnswerId="18617" CreationDate="2017-03-22T19:21:51.103" Score="5" ViewCount="12164" Body="&lt;p&gt;I was wondering, if there is any way to install pytorch in windows like the way we can install tensorflow. My machine is not supporting docker.&lt;/p&gt;&#xA;" OwnerUserId="25500" LastEditorUserId="23305" LastEditDate="2017-09-21T23:34:34.030" LastActivityDate="2018-02-17T00:22:31.713" Title="How to install pytorch in windows?" Tags="|pytorch|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17777" PostTypeId="1" CreationDate="2017-03-22T20:15:59.880" Score="0" ViewCount="1779" Body="&lt;p&gt;I am new in scala. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a csv file stored in hdfs. I am reading that file in scala using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  val salesdata = sc.textFile(&quot;hdfs://localhost:9000/home/jayshree/sales.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; a    b  c    d&#xA; 5  199 1   500&#xA; 33 235 1   500&#xA; 20 249 3   749&#xA; 35 36  4   757&#xA; 19 201 4   757&#xA; 17 94  5   763&#xA; 39 146 5   763&#xA; 42 162 5   763&#xA; 49 41  6   824&#xA; 3  70  6   824&#xA; 24 161 6   824&#xA; 48 216 6   824&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have to perform the following operation on it.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))&lt;/li&gt;&#xA;&lt;li&gt;Find customer wise minimum and maximum item value after applying 30% discount to each item.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can convert it into  a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.&lt;/p&gt;&#xA;" OwnerUserId="8729" LastEditorUserId="8729" LastEditDate="2017-03-23T19:30:15.583" LastActivityDate="2022-07-29T07:04:22.310" Title="Scala RDD operation" Tags="|apache-spark|scala|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17781" PostTypeId="1" AcceptedAnswerId="17785" CreationDate="2017-03-22T21:41:52.413" Score="2" ViewCount="3619" Body="&lt;p&gt;I have a data set that is highly categorical and has a lot of missing values. For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i | A_foo | A_bar | A_baz | outcome&#xA;--+-------+-------+-------+--------&#xA;0 |  nan  |  nan  |  nan  |   1&#xA;1 |   0   |   1   |   0   |   1&#xA;2 |  nan  |  nan  |  nan  |   0&#xA;3 |   1   |   0   |   0   |   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that 1 and 0 have a different meaning that nan. I don't want to impute the data because assigning values of 0 or 1 will bias my data, but many machine learning algorithms will not work on a dataset with missing values. How can I handle this?&lt;/p&gt;&#xA;" OwnerUserId="26675" LastActivityDate="2017-03-22T22:39:09.423" Title="How can I handle missing categorical data that has significance?" Tags="|classification|categorical-data|missing-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17782" PostTypeId="2" ParentId="17781" CreationDate="2017-03-22T21:41:52.413" Score="1" Body="&lt;p&gt;The easiest way to handle missing categorical data with out imputing is to just treat it as a category itself&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i | A_foo | A_bar | A_baz | A_nan | outcome&#xA;--+-------+-------+-------+-------+---------&#xA;0 |   0   |   0   |   0   |   1   |   1   &#xA;1 |   0   |   1   |   0   |   0   |   1   &#xA;2 |   0   |   0   |   0   |   1   |   0   &#xA;3 |   1   |   0   |   0   |   0   |   0   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Happy Training!&lt;/p&gt;&#xA;" OwnerUserId="26675" LastActivityDate="2017-03-22T21:41:52.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17783" PostTypeId="1" AcceptedAnswerId="17820" CreationDate="2017-03-22T22:22:49.650" Score="3" ViewCount="481" Body="&lt;p&gt;I met a question when I ran the random forest. I used &quot;V1&quot;, &quot;V2&quot;, &quot;V3&quot; to predict a binary outcome (1: sick; 0: no) with random forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got a very high accuracy score (99%) however, when I check the confusion matrix, it shows that none of sick individuals were caught in testing data set (30% of entire data set). Here is the confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[[856   0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[  9   0]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result means that 0 out of 9 people was detected as sick and it causes my attention. Maybe because the data set is imbalanced (very few sick individuals)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to see if there is any other ways to detect sick individuals rather than a high accuracy rate, which means it is OK it has higher false positive rate but I would like to catch all 9 (true positive) individuals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="28860" LastActivityDate="2017-07-07T10:07:41.110" Title="Using random forest to learn Imbalanced Data (rare disease)" Tags="|machine-learning|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17784" PostTypeId="2" ParentId="17777" CreationDate="2017-03-22T22:23:24.117" Score="1" Body="&lt;p&gt;For the first you can do as follow :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val discount = salesdata.map( str =&amp;gt; str.split(&quot;,&quot;))&#xA;                        .map( array =&amp;gt; (array(0), array(1), array(2), array(3).toDouble) )&#xA;                        .map{ case(a, b, c, d) =&amp;gt; (a, b, c, d-0.3*d)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not sure to understand the second, this will gives you the min and max per c-itemID&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val productPrices = discount.map{ case(a, b, c, d) =&amp;gt; (c,(d,d)) }&#xA;&#xA;val minMaxPerItemRDD = productPrices.reduceByKey{ case((min1,max1),(min2,max2)) =&amp;gt; (math.min(min1,min2), math.max(max1, max2))}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope that's what you need.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="9257" LastEditDate="2017-03-24T14:31:39.637" LastActivityDate="2017-03-24T14:31:39.637" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17785" PostTypeId="2" ParentId="17781" CreationDate="2017-03-22T22:39:09.423" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Imputation_%28statistics%29&quot; rel=&quot;nofollow noreferrer&quot;&gt;Imputation&lt;/a&gt; and dealing with missing data a broad subject; you should start by researching standard material on this subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question to figure out is &lt;em&gt;Why is some data missing?&lt;/em&gt;  and &lt;em&gt;What is the process that causes data to be missing?&lt;/em&gt;  It's important to understand how this happens, because this will affect what solution is appropriate.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Randomly missing data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If data is missing totally at random (whether a value is missing does not depend on any of the feature values of that item), then imputation can be appropriate.  It should not create bias, if you do it appropriately.  There are many techniques for imputation.  You don't mention what you tried or why you think it will bias your results, but in general, if you use an appropriate method of imputation, there is no reason why it needs to bias your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use a classifier that can tolerate missing data.  Some classifiers are designed to handle missing data and can tolerate it.  However, I don't know of any reason to use them over imputation.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Non-randomly missing data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In contrast, if the chance for data to go missing for some object depends on the value of the features of that object, then you have a bigger problem.  In that case imputation can create bias -- as can any other method.  Your best hope is to understand in greater depth the random process that causes data to be missing and the probability distribution (probability that data goes missing, as a function of feature values), and try to design a procedure that is appropriate for that process.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Your specific situation: all features missing&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Your specific situation is especially weird: it appears in your case either all features are missing, or none are.  That's a weird one.  For instances where the features are missing, you have absolutely no information about those instances.  So, the best classification decision in that case is probably a very simple rule: take whichever class appears most frequently in your training set (or, most frequently among instances with missing data).  Run the classifier on the remaining instances, i.e., the instances with no missing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in real life this situation is pretty rare.  It's more typical that some features are missing and others are present, and that requires more work to handle.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-22T22:39:09.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17786" PostTypeId="1" CreationDate="2017-03-22T23:00:25.470" Score="1" ViewCount="632" Body="&lt;p&gt;I am building a recommender system. I have a list that shows me what a user has disliked and I use it to create a dataset. The dataset shows me: &lt;code&gt;ProductID, Likes, Disklikes&lt;/code&gt; where the values are counts from other users that have disliked the same products. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the algorithmic approaches I can use from this base? From this data I know I can get a probability. How would I also get a weighted average? Maybe some sort of Bayesian filtering approach is called for in these scenarios? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-03-22T23:00:25.470" Title="Recommender system based on binary likes/disklikes?" Tags="|recommender-system|naive-bayes-classifier|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17787" PostTypeId="1" CreationDate="2017-03-23T00:15:23.413" Score="1" ViewCount="743" Body="&lt;p&gt;I am running an SVR prediction on some time series data, and I am receiving this weird offset between my actual and predicted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this &lt;a href=&quot;http://www.alivelearn.net/?p=1666&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM Regression&lt;/a&gt; lag post, that mentions adding a lag of 2 data points behind, instead of one.  However, I am not sure how to incorporate that into my code (which I've included below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any ideas on why my predicted vs. actual is offset in this manner? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GWIC1.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GWIC1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#! /usr/bin/python&#xA;&#xA;import math&#xA;import statistics&#xA;import visualizer&#xA;import numpy as np&#xA;from datagen import constructData&#xA;from sklearn import svm&#xA;&#xA;# Applies Support Vector Regression to the electricity dataset,&#xA;# prints out the accuracy rate to the terminal and plots&#xA;# predictions against actual values&#xA;def suppVectorRegress():&#xA;&#xA;    kernelList = [&quot;linear&quot;,&quot;rbf&quot;,polyKernel]&#xA;    names = [&quot;linear&quot;,&quot;radial basis&quot;,&quot;poly&quot;]&#xA;    preds = []&#xA;&#xA;    # Retrieve time series data &amp;amp; apply preprocessing&#xA;    data = constructData()&#xA;&#xA;    cutoff = len(data)-30&#xA;    xTrain = data[0][0:cutoff]&#xA;    yTrain = data[1][0:cutoff]&#xA;    xTest = data[0][cutoff:]&#xA;    yTest = data[1][cutoff:]&#xA;&#xA;    # Fill in missing values denoted by zeroes as an average of&#xA;    # both neighbors&#xA;    statistics.estimateMissing(xTrain,0.0)&#xA;    statistics.estimateMissing(xTest,0.0)&#xA;&#xA;    # Logarithmically scale the data&#xA;    xTrain = [[math.log(y) for y in x] for x in xTrain]&#xA;    xTest = [[math.log(y) for y in x] for x in xTest]&#xA;    yTrain = [math.log(x) for x in yTrain]&#xA;&#xA;    # Detrend the time series&#xA;    indices = np.arange(len(data[1]))&#xA;    trainIndices = indices[0:cutoff]&#xA;    testIndices = indices[cutoff:]&#xA;    detrended,slope,intercept = statistics.detrend(trainIndices,yTrain)&#xA;    yTrain = detrended&#xA;&#xA;    for gen in range(len(kernelList)):&#xA;&#xA;        # Use SVR to predict test observations based upon training observations&#xA;        pred = svrPredictions(xTrain,yTrain,xTest,kernelList[gen])&#xA;        # Add the trend back into the predictions&#xA;        trendedPred = statistics.reapplyTrend(testIndices,pred,slope,intercept)&#xA;        # Reverse the normalization&#xA;        trendedPred = [np.exp(x) for x in trendedPred]&#xA;        # Compute the NRMSE&#xA;        err = statistics.normRmse(yTest,trendedPred)&#xA;&#xA;        print (&quot;The Normalized Root-Mean Square Error is &quot; + str(err) + &quot; using kernel &quot; + names[gen] + &quot;...&quot;)&#xA;&#xA;        preds.append(trendedPred)&#xA;&#xA;    names.append(&quot;actual&quot;)&#xA;    preds.append(yTest)&#xA;&#xA;    # Change the parameters 2017,2,1 based on the month you want to predict.&#xA;    visualizer.comparisonPlot(2017,2,1,preds,names,plotName=&quot;Support Vector Regression Load Predictions vs. Actual&quot;, &#xA;        yAxisName=&quot;Predicted Kilowatts&quot;)&#xA;&#xA;# Construct a support vector machine and get predictions&#xA;# for the test set&#xA;# Returns a 1-d vector of predictions&#xA;def svrPredictions(xTrain,yTrain,xTest,k):&#xA;    clf = svm.SVR(C=2.0,kernel=k)&#xA;    clf.fit(xTrain,yTrain)&#xA;    return clf.predict(xTest)&#xA;&#xA;# A scale invariant kernel (note only conditionally semi-definite)&#xA;def polyKernel(x,y):&#xA;    return (np.dot(x,y.T)+1.0)**0.95&#xA;&#xA;if __name__==&quot;__main__&quot;:&#xA;    suppVectorRegress()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2017-03-23T15:23:03.777" Title="Why could my SVR prediction be offset, but still follow the actual pattern?" Tags="|python|regression|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17788" PostTypeId="2" ParentId="17773" CreationDate="2017-03-23T00:19:17.423" Score="1" Body="&lt;p&gt;The answer depends on your motivation for breaking the data into blocks. It could range from 'use online training' to 'use ensemble methods' to 'don't break the data into blocks'. Here are a few options to consider.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could train multiple models on different subsets of the data, then ensemble them, as @Hobbes suggested. This would let you train the models in parallel on different machines. The function learned would be different than training a single model on the full data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If using a linear SVM, you can simply continue training the model. Say you train on the first 20k points, then receive an additional 20k points. Initialize the optimization using the first model, then train using the full set of 40k points. Online methods like stochastic gradient descent are a good way to train SVMs on large data sets. SGD processes a single point at a time, so dividing the data into blocks isn't necessary. If the full dataset is available, you'd probably get faster convergence by sweeping through it all instead of breaking it into blocks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd venture a guess that you're not directly using a kernelized SVM on 1M data points. But, you could approximate one by using the kernel to perform a feature space mapping, then train a linear SVM in the feature space. For example, the Nyström technique approximates the mapping using a random subsample of data points. One option would be to learn the feature space mapping using all data, then incrementally train the linear SVM. An alternative would be to learn the mapping using only the first block of data. This is valid if all blocks are equally representative of the full data set and adding new data wouldn't change the mapping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The case isn't so straightforward for random forests because the individual trees choose hierarchical splits in a greedy manner. The splits chosen for the initial 20k points may not match what would have been chosen had the full 40k points been available. But, it's not easy to go back and revise them. However, a number of papers have proposed online variants of random forests that could be used in this setting. A search for 'incremental' or 'online' random forests should turn up the relevant results.&lt;/p&gt;&#xA;" OwnerUserId="19014" LastActivityDate="2017-03-23T00:19:17.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17789" PostTypeId="2" ParentId="17675" CreationDate="2017-03-23T02:04:34.387" Score="0" Body="&lt;p&gt;Listening to the podcast Partially Derivative Episode &lt;a href=&quot;http://partiallyderivative.com/podcast/2017/03/06/maksim-percherskiy&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Data Science On The Silicon Beach&quot;&lt;/a&gt; the host interviews Maksim Percherskiy, Chief Data Officer for the City of San Diego.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Talking about the stack he uses for the City of San Diego: (08:50) &lt;em&gt;&quot;The way we move data around ... we use &lt;a href=&quot;http://airflow.apache.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;Airflow&lt;/a&gt; [...] and Airflow is just Python.&quot;&lt;/em&gt; Percherskiy continues characterizing the data sharing problem in the context of city government. &lt;/p&gt;&#xA;" OwnerUserId="2742" LastActivityDate="2017-03-23T02:04:34.387" CommentCount="3" CommunityOwnedDate="2017-03-23T18:33:17.413" ContentLicense="CC BY-SA 3.0" />
  <row Id="17790" PostTypeId="2" ParentId="17675" CreationDate="2017-03-23T04:25:29.803" Score="0" Body="&lt;p&gt;For big files, I use cloud storage (Google, Amazon, Microsoft, or whichever ecosystem your company's on), with folders named after the issue/project ticket name/number. These services support file versioning, by the way. Small files I just attach to the ticket. If have to share something small and transient with a handful of people I can use email or our corporate chat application.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-03-23T04:25:29.803" CommentCount="2" CommunityOwnedDate="2017-03-23T18:33:17.413" ContentLicense="CC BY-SA 3.0" />
  <row Id="17791" PostTypeId="1" AcceptedAnswerId="17792" CreationDate="2017-03-23T05:32:20.430" Score="2" ViewCount="602" Body="&lt;p&gt;One of the researchers, Marco Ribeiro, who developed this method of explaining how black box models make their decisions has developed a Python implementation of the algorithm available through Github, but has anyone developed a R package? If so, can you report on using it?&lt;/p&gt;&#xA;" OwnerUserId="18843" LastEditorUserId="1367" LastEditDate="2017-03-23T18:33:27.473" LastActivityDate="2020-08-12T15:32:37.743" Title="Is there an R package for Locally Interpretable Model Agnostic Explanations?" Tags="|neural-network|r|predictive-modeling|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17792" PostTypeId="2" ParentId="17791" CreationDate="2017-03-23T05:41:47.547" Score="4" Body="&lt;p&gt;I think you're talking about the &lt;code&gt;lime&lt;/code&gt; Python package. No, there is &lt;strong&gt;no&lt;/strong&gt; R port for the package. The implementation for the localized model requires enhancements to the existing machine-learning code (explained in the paper), a new implementation for R would be very time consuming.&lt;/p&gt;&#xA;&lt;p&gt;You may want to take a look at &lt;a href=&quot;https://stackoverflow.com/q/11716923/9214357&quot;&gt;this&lt;/a&gt; for interfacing Python in R.&lt;/p&gt;&#xA;&lt;p&gt;My suggestion is stick with Python. The package is only useful for highly complicated non-linear models, which Python offers better support than R.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="98307" LastEditDate="2020-08-12T14:32:38.380" LastActivityDate="2020-08-12T14:32:38.380" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="17793" PostTypeId="1" AcceptedAnswerId="17805" CreationDate="2017-03-23T06:11:55.510" Score="0" ViewCount="416" Body="&lt;p&gt;Im new to forecasting time series and Im looking for some advice on selecting the best method based on the analaysis of the graph.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/uSBhE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uSBhE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I have the following data and based on the little knowledge I have, Im assuming it is a stationary time series because of the shape of the data and because of the result I got when performing a Dicker Fulley Test&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Test Statistic                -6.560544e+00&#xA;p-value                        8.402824e-09&#xA;#Lags Used                     2.100000e+01&#xA;Number of Observations Used    1.164000e+03&#xA;Critical Value (5%)           -2.864026e+00&#xA;Critical Value (1%)           -3.435980e+00&#xA;Critical Value (10%)          -2.568094e+00&#xA;dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That made me choose Arima with a p and q values of 1. However the result Im getting are pretty awful. Can anyone guide on how should I choose the adequeate method for forecasting and  if case ARIMA is a good choice, how should I tweak parameters to improve my results? The end goal is to predict the next month of data, so it might not be necessary to use all of the data.&lt;/p&gt;&#xA;" OwnerUserId="30275" LastActivityDate="2017-03-23T11:39:00.860" Title="Forecasting time series: Method Selection" Tags="|time-series|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17794" PostTypeId="1" AcceptedAnswerId="17796" CreationDate="2017-03-23T06:27:43.773" Score="3" ViewCount="4490" Body="&lt;p&gt;I was reading about &lt;code&gt;PAC framework&lt;/code&gt; and faced the definition of &lt;code&gt;Generalization Error&lt;/code&gt;. The book defined it as: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given a hypothesis h ∈ H, a target concept c ∈ C, and an underlying distribution&#xA;  D, the generalization error or risk of h is defined by &lt;br /&gt;&#xA;  &lt;a href=&quot;https://i.stack.imgur.com/BbKi5.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BbKi5.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;  &lt;br /&gt;&#xA;  The generalization error of a hypothesis is not directly accessible to the learner&#xA;  since both the distribution D and the target concept c are unknown. However, the&#xA;  learner can measure the empirical error of a hypothesis on the labeled sample S.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I can not understand the equation. Can anyone please tell me how it can be interpreted? Also what is x~D?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;How do I formally write this term? Is something like $$\mathbb{E}_{x \sim D} [1_{h(x)\neq c(x)}] = \int_X 1_{h(\cdot) \neq c(\cdot)} (\omega) dD(\omega)$$&#xA;correct or do I need to define some random variable? Also, to show that the empirical error &#xA;$$ \hat{R}(h) = \frac{1}{m} \sum_{i =1}^m 1_{h(x_i)\neq c(x_i)} $$&#xA;is unbiased, we have&#xA;$$\mathbb{E}_{S \sim D^m} [\hat{R}(h)] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x_i)\neq c(x_i)} \right] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]$$,&#xA;but how do we formally get&#xA;$$ \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]= \mathbb{E}_{X \sim D} ~ \left[ 1_{h(x)\neq c(x)} \right] = R(h)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that I understand it intuitionally, but I can't write it down formally. Any help is much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="29575" LastEditDate="2018-07-11T10:40:44.803" LastActivityDate="2018-07-11T10:40:44.803" Title="Generalization Error Definition" Tags="|machine-learning|deep-learning|pac-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17795" PostTypeId="2" ParentId="17350" CreationDate="2017-03-23T06:36:24.510" Score="3" Body="&lt;p&gt;Topcoder hosted competition for identifying spoken languages. They released a database containing 176 languages. The datasets and the related resources can be found &lt;a href=&quot;https://community.topcoder.com/longcontest/?module=ViewProblemStatement&amp;amp;rd=16555&amp;amp;compid=49304&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-03-23T06:36:24.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17796" PostTypeId="2" ParentId="17794" CreationDate="2017-03-23T07:16:20.390" Score="4" Body="&lt;p&gt;There exists somewhere in the world a distribution $D$ from which you can draw some samples $x$. The notation $x \sim D$ simply states that the sample $x$ came from the specific distribution that was noted as $D$ (e.g. Normal or Poisson distributions, but also the possible pixel values of images of beaches). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have some ground truth function, mark it as $c$, that given a sample $x$ gives you its true label (say the value 1). Furthermore, you have some function of your own, $h$ that given some input, it outputs some label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now given that, the risk definition is quite intuitive: it simply &quot;counts&quot; the number of times that $c$ and $h$ didn't agree on the label. In order to do that, you (ideally) will &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;go over every sample $x$ in your distribution (i.e. $x \sim D$).&lt;/li&gt;&#xA;&lt;li&gt;run it through $c$ (i.e. $c(x)$) and obtain some label $y$.&lt;/li&gt;&#xA;&lt;li&gt;run it through $h$ (i.e. $h(x)$) and obtain some label $\hat{y}$.&lt;/li&gt;&#xA;&lt;li&gt;check if $y \neq \hat{y}$. If so, you add 1 to your count (i.e. $1_{h(x) \neq c(x)}$ - that notes the &lt;a href=&quot;https://en.wikipedia.org/wiki/Indicator_function&quot; rel=&quot;noreferrer&quot;&gt;indicator function&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now last thing to note is that I wrote above &quot;count&quot;, but we don't really care if the number is 500 or 100, we care for the &lt;em&gt;relative number of mistakes&lt;/em&gt; (like 40% or 5% of the samples that were checked were classified differently). That is why it is noted as expectancy ($\mathbb{E}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if that was clear enough :-)&lt;/p&gt;&#xA;" OwnerUserId="29960" LastActivityDate="2017-03-23T07:16:20.390" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17797" PostTypeId="2" ParentId="12697" CreationDate="2017-03-23T07:27:40.717" Score="3" Body="&lt;p&gt;This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;choose the k best attributes (fixed number defined)&lt;/li&gt;&#xA;&lt;li&gt;choose the best k * 100% of attributes (relative number of features defined)&lt;/li&gt;&#xA;&lt;li&gt;make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You find a nice implementation doing this in R with the &lt;a href=&quot;https://cran.r-project.org/web/packages/FSelector/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;FSelector&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.&lt;/p&gt;&#xA;" OwnerUserId="25217" LastActivityDate="2017-03-23T07:27:40.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17798" PostTypeId="2" ParentId="17793" CreationDate="2017-03-23T08:49:27.563" Score="0" Body="&lt;p&gt;I'm not very familiar with ARIMA but I've used for a very similar chart a RNN and I got good results. Look at GRU which are  very efficient.&lt;/p&gt;&#xA;" OwnerUserId="30240" LastActivityDate="2017-03-23T08:49:27.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17799" PostTypeId="1" CreationDate="2017-03-23T08:53:00.907" Score="4" ViewCount="1925" Body="&lt;p&gt;I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have played with the following possible approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use character recognition to extract pure text and later try to&#xA;puzzle with the fragments. This method has some considerable&#xA;problems: the quality of the OCR (at least of the implemented one in&#xA;tesseract library) are quite mediocre and the output is hopelessly&#xA;unstructured (practically a big pile of words), it is very difficult&#xA;to come out any regexp or other rule even for regular phrases.&lt;/li&gt;&#xA;&lt;li&gt;My other approach would be to apply some kind of deep learning either to&#xA;the raw image itself or the text pile where we leave the heavy&#xA;lifting to the network, but in this case I'm not sure what is&#xA;supposed to be the output? Is it some kind of a sequence to sequence&#xA;mapping?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Very unusual task, indeed.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2020-08-13T16:26:11.583" Title="What is the best approach for specified optical character recognition?" Tags="|python|deep-learning|nlp|regex|ocr|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17800" PostTypeId="1" CreationDate="2017-03-23T09:06:27.647" Score="1" ViewCount="81" Body="&lt;p&gt;I would like to visualize how a percentage evolves in time over a geographical area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular I have the percentage of products delivered in different regions and I want to show the weekly trend for each of the 21 regions. Currently I show a heatmap with the values of the current week and three line plots showing the trends of the 3 worst regions. I would like to show the full picture. Unluckily I can't use animations as I have to put the visualization on a static powerpoint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working with d3, sql and excel.&#xA;Any idea? &lt;/p&gt;&#xA;" OwnerUserId="30278" LastActivityDate="2017-05-22T13:33:16.993" Title="How to visualize trending over geographical area?" Tags="|visualization|sql|geospatial|plotting|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17801" PostTypeId="2" ParentId="17799" CreationDate="2017-03-23T09:13:36.240" Score="0" Body="&lt;p&gt;One way could be applying a good segmentation technique  and then classifying all the regions with some algorithms  like(ann,svm,knn...), this method doesn't need a lot of training sample at all.&#xA;Another way is applying deep learning (but this would mean a lot of training samples) you could use a CNN which perform a regression and so as output it gives you the text(this would mean you have to set a max sentences length). But to avoid a max sentences' length you could use a CNN with also some RNN layers and then you feed the network with parts of the image(a cascade approach).&lt;/p&gt;&#xA;" OwnerUserId="30240" LastActivityDate="2017-03-23T09:13:36.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17802" PostTypeId="1" AcceptedAnswerId="17813" CreationDate="2017-03-23T10:14:05.380" Score="0" ViewCount="89" Body="&lt;p&gt;I am currently going through Hastie and Tibshirani's 'Introduction to Statistical Learning' textbook and I have come across something I don't understand on page 77. I have two questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author states that if we had 100 variable predictors (with respective coefficients $\beta_i$) and supposed that the null hypothesis, $$H_0 : \beta_1 = ... = \beta_{100} = 0$$ were true, then roughly $5\%$ of the $p$-values would fall below $0.05$ by chance, and therefore we might &lt;em&gt;wrongly&lt;/em&gt; conclude that certain predictors are related to the response. Why would this happen by chance? Is this simply a mathematical truth?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, the author then goes on to state that the $F$-statistic is a better measure because &quot;if $H_0$ were true, then there is only a $5\%$ chance that the $F$-statistic would result in a $p$-value below $0.05$&quot;. I don't understand the difference - could somebody explain a bit more clearly?&lt;/p&gt;&#xA;" OwnerUserId="30281" LastActivityDate="2017-03-23T13:33:44.630" Title="In multiple linear regression why is it best to use an $F$-statistic when evaluating predictors?" Tags="|statistics|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17804" PostTypeId="1" CreationDate="2017-03-23T11:09:52.050" Score="-2" ViewCount="1081" Body="&lt;p&gt;I have trained my word2vec model on a movie dataset with star cast, director name and other similar features/columns in the training data set. The text is not free flowing (it is comma separated). As a result, the SIMILARITY function and SCORE functions don’t produce satisfactory results as embedding generated are not up to the mark&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is word2vec the right approach for such a problem with more large number of proper nouns and no free flowing text? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If yes, which parameters to tune for training with proper nouns?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30286" LastActivityDate="2017-10-13T17:58:52.550" Title="word2vec for propernouns" Tags="|machine-learning|python|nlp|word2vec|word-embeddings|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17805" PostTypeId="2" ParentId="17793" CreationDate="2017-03-23T11:39:00.860" Score="1" Body="&lt;p&gt;You can try two different approaches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Kalman filter&lt;/strong&gt;, the method is battle-tested and has proven useful in many areas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cl.cam.ac.uk/~rmf25/papers/Understanding%20the%20Basis%20of%20the%20Kalman%20Filter.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding the Basis of the Kalman Filter&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/&quot; rel=&quot;nofollow noreferrer&quot;&gt;kalman filter in pictures&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Recurrent Neural Networks&lt;/strong&gt;, the LSTM and GRU architectures are particularly interesting for time series predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;RNN effectiveness&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding LSTMs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To do regression and predict future data points, you would need to build a training dataset consisting of a sequence of events. Let's say a value $x$ for every timestamp $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data seems to have 1 dimension, so both the network input layer and the output layer would consist of 1 unit. You would then train your model to predict $(x_{t+1})$ given $(x_{t})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $M$ be our trained model and let's say you want to forecast a data point at time $k$ and you know the current value at time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M(x_{t}) = (x_{t+1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$t = t+1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M(x_{t}) = (x_{t+1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$...$ increment $t$ and keep predicting until $t+1 = k-1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$M(x_{t+1}) = (x_{k})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Put in pictures this corresponds to:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mPrOc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mPrOc.png&quot; alt=&quot;credits Udacity&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(picture from Udacity lecture about Deep Learning)&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-03-23T11:39:00.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17806" PostTypeId="2" ParentId="17799" CreationDate="2017-03-23T11:48:55.660" Score="1" Body="&lt;p&gt;Optical character recognition is a well-studied problem with many possible solutions (&lt;a href=&quot;https://scholar.google.com/scholar?q=optical%20character%20recognition&amp;amp;btnG=&amp;amp;hl=en&amp;amp;as_sdt=0%2C5&amp;amp;oq=&quot; rel=&quot;nofollow noreferrer&quot;&gt;ressources&lt;/a&gt;). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://yann.lecun.org/exdb/publis/pdf/lecun-89e.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Backpropagation Applied to Handwritten zip code&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://yann.lecun.org/exdb/publis/pdf/bottou-94.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/beginners&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a beginner tutorial to do just that with Tensorflow.&lt;/p&gt;&#xA;&lt;p&gt;If you need extra data to train your model, take a look at the &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MNIST dataset&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastEditorUserId="98307" LastEditDate="2020-08-13T16:26:11.583" LastActivityDate="2020-08-13T16:26:11.583" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17807" PostTypeId="2" ParentId="17800" CreationDate="2017-03-23T12:18:42.087" Score="1" Body="&lt;p&gt;Overly simple answer: &quot;animate&quot; this in PowerPoint by having one slide per time slice then advancing the slides quickly with the keyboard or set them to auto-advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a little more flexibility outside PPT, there are free tools (e.g. Microsoft Power BI) where you could do a shaded map and have a time slider to make it more interactive. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-23T12:18:42.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17809" PostTypeId="1" CreationDate="2017-03-23T12:36:07.850" Score="2" ViewCount="7516" Body="&lt;p&gt;My data file is of 4 GB (json), I need to apply everything on this dataset (from applying clustering/ML algorithm to wrangling it) like the way people do with pandas/scikit. But it is difficult to work with them on my local machine. I tried using &lt;a href=&quot;http://dask.pydata.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;dask&lt;/a&gt; and &lt;a href=&quot;http://blaze.readthedocs.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;blaze&lt;/a&gt; but could not find them efficient for my case because of their limited functionalities. I don't wanna use hadoop because it is not meant for such small datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to know approaches for working with not so large datasets with python and applying ML algos on them. Just like the way you work on small datasets using pandas (if any exists).&lt;/p&gt;&#xA;" OwnerUserId="30290" LastEditorUserId="30290" LastEditDate="2017-03-24T04:15:28.817" LastActivityDate="2017-03-24T04:15:28.817" Title="Working with large datasets python" Tags="|python|bigdata|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17810" PostTypeId="1" AcceptedAnswerId="17846" CreationDate="2017-03-23T13:10:21.960" Score="1" ViewCount="467" Body="&lt;p&gt;I have used the &lt;a href=&quot;http://ai.stanford.edu/~amaas/data/sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;stanford movie review dataset&lt;/a&gt; for creating a experimentation of sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Managed to create a basic application on top of Spark using the Naive bayes classification algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps that I did for pre-processing from the spark &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ML pipeline&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tokenization&lt;/li&gt;&#xA;&lt;li&gt;Bigrams&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The provided dataset above also has a testing dataset with itself which is separate of the training set. After training it I got around 97% accuracy which I believe is pretty good for Naive bayes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now can I use this ML model to predict for other texts such as email/chat etc., My guess is that this dataset has a large enough collection of words to perform good predictions and certain english words regardless of the business context like &quot;I dont like this&quot;,&quot;This does not look good&quot; is the same across different domains such as Movies/Emails/Chats etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not done the experiment since the data that I need to get hold of belongs to the customer and due to privacy restrictions we cannot access the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help/guidance would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="30292" LastActivityDate="2017-03-24T15:07:25.983" Title="Can generic data sets be suitable for specific sentiment analysis" Tags="|machine-learning|apache-spark|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17811" PostTypeId="2" ParentId="17809" CreationDate="2017-03-23T13:21:14.717" Score="2" Body="&lt;p&gt;With today's standard amount of RAM for a personal desktop or laptop computer (usually 8Gb or more), there should be little problem in handling 4Gb in memory. If your hardware setup is short in memory and your ML algorithms make your SO start paging, the best hint is to use algorithms and settings that do not copy the data around, but do everything &lt;em&gt;in place&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas&lt;/a&gt;, many operations provide a boolean parameter precisely called &lt;code&gt;inplace&lt;/code&gt; that controls this behaviour, e.g. &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataFrame.reset_index&lt;/a&gt;. &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Numpy&lt;/a&gt; also has support for in-place operations, e.g. &lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.sort.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sort&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take into account that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;inplace=True&lt;/code&gt; does not guarantee that the operation is actually performed in place, it is just a hint that may or may not be honored.&lt;/li&gt;&#xA;&lt;li&gt;for some operations. &lt;code&gt;inplace=True&lt;/code&gt; can worsen performance, like &lt;a href=&quot;https://stackoverflow.com/questions/22532302/pandas-peculiar-performance-drop-for-inplace-rename-after-dropna&quot;&gt;&lt;code&gt;dropna&lt;/code&gt;&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All that said, in my experience &lt;code&gt;inplace=True&lt;/code&gt; actually makes a difference in memory consumption and can be used to avoid paging and therefore enabling computations to take place completely in RAM for datasets like yours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, be careful with &lt;code&gt;inplace&lt;/code&gt; default values. As noted in this &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;warning in the pandas documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Warning: For backwards compatability, inplace defaults to True if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-03-23T13:21:14.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17813" PostTypeId="2" ParentId="17802" CreationDate="2017-03-23T13:33:44.630" Score="1" Body="&lt;p&gt;The example given in your textbook proposes a multiple linear regression with 100 predictors, all of which have a &quot;true&quot; regression coefficient of 0. In other words, your independent variables have no statistical association with your dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you calculate the $p$-value of an individual coefficient, you're looking at the magnitude of the coefficient, the standard error of the coefficient, making some distributional assumptions about it, and asking the following question: &quot;what is the probability of seeing a coefficient value this extreme if the true value is actually zero?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If our distributional assumptions are correct, any given coefficient with a true value of 0 will report a &amp;lt;0.05 $p$-value approximately 5% of the time. That's not so much of a problem if we only have one predictor, but by the law of large numbers, if we have lots of predictors, we'd expect 5% of them to report a $p$-value this low. This makes the $p$-values in high-dimensional regressions hard to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $F$-test is different. Instead of evaluating every single coefficient for statistical significance, it applies a single test to the &lt;em&gt;entire regression&lt;/em&gt;. So instead of having 100 chances to throw up an erroneous $p$-value, it only has one chance. This makes the $F$-test useful for evaluating whether or not there is a regression effect for high-dimensional regressions.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-03-23T13:33:44.630" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17814" PostTypeId="1" AcceptedAnswerId="17816" CreationDate="2017-03-23T13:35:36.277" Score="0" ViewCount="3736" Body="&lt;p&gt;I am using sklearn's RandomForestClassifier to build a binary prediction model. As expected, I am getting an array of predictions, consisting of 0's and 1's. However I was wondering if it is possible for me to get a value between 0 and 1 along with the prediction array and set a threshold to tune my model. &#xA;Many thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="28049" LastActivityDate="2017-03-23T14:52:27.287" Title="RandomForestClassifier : binary classification scores" Tags="|scikit-learn|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17816" PostTypeId="2" ParentId="17814" CreationDate="2017-03-23T14:52:27.287" Score="0" Body="&lt;p&gt;Referring to &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&lt;/a&gt;, you are using teh predict method to derive the binary response.  If you call predict_proba, you will get an array of probabilities corresponding to the binary response.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-03-23T14:52:27.287" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17817" PostTypeId="2" ParentId="17787" CreationDate="2017-03-23T15:23:03.777" Score="1" Body="&lt;p&gt;Two things that come to my mind: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A) - have you compared the distributions with a Kolgomorov Smirnov Test or similar - (aka: are they really the same. Sure there is the big spike but besides its not that similar (by eye)) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;B) - Are training and test in the same magnitudes for the input data? If they are not this might confuse your SVR. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-03-23T15:23:03.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17819" PostTypeId="1" CreationDate="2017-03-23T17:21:21.237" Score="-1" ViewCount="243" Body="&lt;p&gt;I have a list of time series data, which I want to classified them with HMM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for this I’m using &lt;strong&gt;Jahmm&lt;/strong&gt;.&#xA;the data are binary vectors and they ate transformed to Observation vectors for more simplicity. Now, I want to do a binary classification. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As an example:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;obs1 = [1,0,0,0,1,0,1]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;obs2 = [1,0,1,0,1,0,1]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;obs3 = [0,0,0,0,1,1,1]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;obs4 = [0,1,1,0,1,0,1]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;first I have created K-means learner and optimize the learning process with Baum welch (as its said in jahmm), then when I want to calculate the Hmm probability which the trained observation, its always Infinity,&#xA;I have also tried with log and scaling methods and the LSE. none of them change the Infinity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;here is my code&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;    OpdfMultiGaussianFactory opdfMGF = new OpdfMultiGaussianFactory(hcodeDim);&#xA;    KMeansLearner&amp;lt;ObservationVector&amp;gt; kml = new KMeansLearner&amp;lt;ObservationVector&amp;gt;(hSNr, opdfMGF, trainData);&#xA;    RegularHmmBase&amp;lt;ObservationVector&amp;gt; trainHmm = kml.iterate();     &#xA;    System.out.println(&quot;Result TrainHmm &quot; + trainHmm);&#xA;    /* Baum-Welch learning */&#xA;    RegularBaumWelchScaledLearnerBase bwl = new RegularBaumWelchScaledLearnerBase();&#xA;    bwl.setNbIterations(100);&#xA;    Hmm learntHmmTrain = bwl.learn(trainHmm, trainData);&#xA;    System.out.println(&quot;Resulting HMM:\n&quot; + learntHmmTrain);&#xA;    System.out.println(&quot;probability at learnt hmm &quot; +learntHmmTrain.probability(trainData.firstElement()));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    // This object measures the distance between two HMMs&#xA;    KullbackLeiblerDistanceCalculator klc = new KullbackLeiblerDistanceCalculator();&#xA;&#xA;    // Incrementally improve the solution&#xA;    for (int i = 0; i &amp;lt; 10; i++) {&#xA;            System.out.println(&quot;Distance at iteration &quot; + i + &quot;: &quot;&#xA;                + klc.distance(learntHmmTrain,trainHmm));&#xA;        learntHmmTrain = bwl.iterate(learntHmmTrain, trainData);&#xA;    }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have to mention that the klc.distance in always NaN.&#xA;does anybody have experience with this problem in jahmm? &#xA;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="18516" LastActivityDate="2017-04-04T14:25:01.423" Title="HMM probability and log probability always Infinity!" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17820" PostTypeId="2" ParentId="17783" CreationDate="2017-03-23T17:51:19.710" Score="2" Body="&lt;p&gt;Use class weights, to weight errors, so that &quot;incorrectly labelling a sick person as healthy&quot; is penalized more than &quot;incorrectly labelling a healthy person as sick&quot;.  Or look up any of the other standard techniques for dealing with &lt;a href=&quot;https://stats.stackexchange.com/questions/tagged/unbalanced-classes&quot;&gt;class imbalance&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-03-23T17:51:19.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17821" PostTypeId="2" ParentId="16902" CreationDate="2017-03-23T17:54:11.507" Score="1" Body="&lt;p&gt;The problem is that you only have positive instances (businesses who &lt;em&gt;have&lt;/em&gt; become a customer) but no negative instances (businesses who &lt;em&gt;haven't&lt;/em&gt; become a customer).  This prevents use of supervised learning.  You could try unsupervised learning -- e.g., a one-class classifier -- but I don't expect it to be particularly effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had data on businesses who you tried approaching, but ultimately did not become a customer of yours, then you could try applying supervised learning.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-23T17:54:11.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17822" PostTypeId="1" CreationDate="2017-03-23T18:07:59.900" Score="-2" ViewCount="88" Body="&lt;p&gt;I am new to data science and I am hoping I can start applying it to my job. I have watched some videos on places like Udemy for Machine Learning and Python etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, I have a task but I am not sure what I need to look into to solve it.&#xA;I have a PCB/Circuit board which has 80 components on it. I also have a piece of test equipment that measures each component and logs the results with a reading of what was measured and then a simple 1 or 0 for the result I.E. Pass/Fail&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lately we have notice a component has started to fail a lot on the tester. One of the tasks I have been give is has any of the other components changed in their readings and could this have an effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, what would I use if I wanted to see if another component potentially has influence?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example may help if my explanation is poor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets says I have these components and the general readings I expect&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Component    Reading     Outcome&#xA;C1            1.5 nf     Pass&#xA;C2            1.0 uf     Pass&#xA;C25           10 nf      Pass&#xA;R22           100 kohms  Pass &#xA;R53           150 kohms  Pass&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now lets say C25 is failing because the reading has gone up.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Component    Reading    Outcome&#xA;C1            1.5 nf    Pass&#xA;C2            1.0 uf    Pass&#xA;C25           12 nf     Fail&#xA;R22           80 kohms  Pass&#xA;R53           150 kohms Pass&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you notice, R22 has dropped. &#xA;Now say that this is the pattern. If R22 reads low then C25 can read high and potentially fail. What can I use to find this link/pattern/relationship?&lt;/p&gt;&#xA;" OwnerUserId="30308" LastActivityDate="2021-02-06T18:36:54.863" Title="New to Data Science - What to use when looking for a pattern/relationship between items and an outcome" Tags="|python|recommender-system|prediction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17823" PostTypeId="2" ParentId="17783" CreationDate="2017-03-23T19:17:58.720" Score="1" Body="&lt;p&gt;I would pick a different scoring function than accuracy; the problem with accuracy is that if you classify &lt;em&gt;all&lt;/em&gt; the instances under the majority class, you will automatically end up with a very high accuracy score, which is rather meaningless!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, the area under the curve (AUC) of the precision-recall curve (&lt;code&gt;sklearn.metrics.average_precision_score()&lt;/code&gt; in Python) works well and is representative of the actual performance of the model when one is dealing with unbalanced data. The AUC of the receiver operating characteristic (ROC) curve is also another metric that is most often used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;del&gt;Having said this, it seems to me that you specifically want to maximize the recall score, which is the ratio of the number of true positives to the total number of actual positives.&lt;/del&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: As per @stmax's comment below, you don't want to maximize the recall score either.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastEditorUserId="21433" LastEditDate="2017-03-24T14:21:20.753" LastActivityDate="2017-03-24T14:21:20.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17828" PostTypeId="2" ParentId="17822" CreationDate="2017-03-23T20:09:02.603" Score="0" Body="&lt;p&gt;You could try running a stepwise logistic regression with C25 pass/fail as the response and the readings of the other components as predictors.  &lt;/p&gt;&#xA;" OwnerUserId="29353" LastActivityDate="2017-03-23T20:09:02.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17830" PostTypeId="1" AcceptedAnswerId="17834" CreationDate="2017-03-23T21:17:10.930" Score="3" ViewCount="10028" Body="&lt;p&gt;So, I've been trying to implement my first algorithm to predict the (sales/month) of a single product,  I've been using linear regression since that was what were recommended to me. I'm using data from the past 42 months, being the first 34 months as training set, and the remaining 8 as validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to use 4 features to start:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Month number(1~12)&lt;/li&gt;&#xA;&lt;li&gt;Average price that the product was sold during that month&lt;/li&gt;&#xA;&lt;li&gt;Number of devolutions previous month&lt;/li&gt;&#xA;&lt;li&gt;Number of units sold previous month&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are images with graphs comparing the Real Data x Predicted Data and a Error x number of elements graph:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MrciW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MrciW.png&quot; alt=&quot;Real Data x Trained Data&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/S9RNi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S9RNi.png&quot; alt=&quot;Error x Number of trained elements&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far the results are not good at all (as shown in the images above), the algorithm can't even get the training set right. I tried to use higher degrees polynomials, and the regularization parameter, it seems to make it worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I would like to know if there is a better approach for this problem, or what could I do to improve the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance!&lt;/p&gt;&#xA;" OwnerUserId="30314" LastEditorUserId="30314" LastEditDate="2017-03-23T21:37:09.063" LastActivityDate="2018-02-05T01:24:33.323" Title="Sales prediction of an Item" Tags="|machine-learning|predictive-modeling|regression|linear-regression|prediction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17831" PostTypeId="2" ParentId="17809" CreationDate="2017-03-23T21:33:48.137" Score="0" Body="&lt;p&gt;One thing you can do is explicitly set types when loading the dataset using the dtypes option i.e. (... dtype={'ORDER NO': np.int32, 'CUSTOMER': str} ...) - that could potentially save memory when you load the file.&lt;/p&gt;&#xA;" OwnerUserId="12408" LastActivityDate="2017-03-23T21:33:48.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17833" PostTypeId="2" ParentId="17809" CreationDate="2017-03-23T22:22:03.977" Score="1" Body="&lt;p&gt;There are at least two uncomplicated ways to deal with this issue. If your data has a lot of zeros etc. it might be worth trying to load it as a &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/sparse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sparse dataframe&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that always works is to process your data in an iterative fashion. For most datatypes pandas can let you iterate through your file, e.g. with &lt;code&gt;pandas.read_csv(data, iterator=True)&lt;/code&gt;. This is not available for JSON data, but you can use the &lt;a href=&quot;https://pypi.python.org/pypi/ijson/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;ijson&lt;/code&gt; package&lt;/a&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many scikit-learn algorithms can also be trained iteratively on one or a few samples (mini-batches) at a time. In particular, the flexible linear models with stochastic gradient descent (SGDClassifier and SGDRegressor). For clustering, you can use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html#sklearn.cluster.MiniBatchKMeans&quot; rel=&quot;nofollow noreferrer&quot;&gt;MiniBatchKMeans&lt;/a&gt;. You can read more about this in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data fits &lt;em&gt;almost&lt;/em&gt; into memory, you may just read in as much as possible. If most of it doesn't fit, there is probably no way that lets you use pandas/sklearn without some adjustments. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-03-23T22:22:03.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17834" PostTypeId="2" ParentId="17830" CreationDate="2017-03-24T03:51:18.063" Score="2" Body="&lt;p&gt;Based on the information given by you. I'm assuming you have performed multiple linear regression ie multiple features and one response feature to be predicted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, apply PCA on all of your features except the response variable you want to predict. In your case the four features you mentioned. Then transform it into a 2 component matrix using PCA. Once you are done with that plot the new Matrix you formed with the response features as a scatter plot.So effectively a 3D scatter plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you generate this scatter plot you will be able to visualize much better on which regression you have to use. You can decide for yourself if it is linear or not. Depending on how many outliers you are comfortable with. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastEditorUserId="30107" LastEditDate="2017-03-24T04:04:40.820" LastActivityDate="2017-03-24T04:04:40.820" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17835" PostTypeId="2" ParentId="17755" CreationDate="2017-03-24T05:03:30.000" Score="3" Body="&lt;p&gt;First off, a pragmatic answer: don't discount the possibility that the test set comes from a somewhat different distribution than the data set you're using for training and cross-validation.  You might think that shouldn't happen, but in practice it does seem to occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, let's go with your hypothetical and assume that the test set comes from exactly the same distribution as the rest of your data.  In that case, it is possible for cross-validation to lead you astray about which model is better, if you're using cross-validation to select hyper-parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use cross-validation to either (a) select hyper-parameters, &lt;em&gt;or&lt;/em&gt; (b) estimate the accuracy of your model -- but not both at the same time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears you're using cross-validation to select the optimal hyper-parameters: you try many different choices for the hyper-parameters, for each choice estimate accuracy of that choice using cross-validation, and select the best choice.  When you do that, there's no guarantee that the resulting accuracy (with the best parameter) will be predictive of performance on the test set -- it might be an overestimate (due to overfitting).  If it's more of an overestimate for M1 than M2, then you might see what you saw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to both select hyper-parameters and estimate accuracy, I suggest that you have a separate held-out validation set for estimating accuracy, or use nested cross-validation.  See &lt;a href=&quot;https://stats.stackexchange.com/q/65128/2921&quot;&gt;https://stats.stackexchange.com/q/65128/2921&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-03-24T05:03:30.000" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17836" PostTypeId="2" ParentId="14761" CreationDate="2017-03-24T09:26:56.430" Score="2" Body="&lt;p&gt;It turns out the Adaboost or VotingClassifier doesn't work for multilabel classification data. We get a NotImplementedError. I believe they are working towards it and might be out in next version or so.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-03-24T09:26:56.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17837" PostTypeId="1" CreationDate="2017-03-24T10:01:30.713" Score="0" ViewCount="3861" Body="&lt;p&gt;In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate. &#xA;For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques? &lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-03-25T11:17:22.517" Title="Ensemble modelling using model's probabilities" Tags="|machine-learning|classification|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17838" PostTypeId="2" ParentId="13212" CreationDate="2017-03-24T10:29:05.283" Score="1" Body="&lt;p&gt;I will give you some other idea of it, but without mathematical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like horaceT's mentioned, it's all about feature construction and feature selection, then you have many machine learning method which can do the regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, just look it as regression model, not just the time series model. Then you will get the following model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y(Dependent variable 'Spending') = X1(moving average term)...X2(lastquarter moving average term) + X3(country id) + X4(Fourier term) ... + Error term&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The moving average can catch the dependent variable's trend, Fourier term can catch the dependent variable's cycle. Maybe you can construct other feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can selection the features using tree-based model, such as xgboost. &lt;/p&gt;&#xA;" OwnerUserId="19275" LastActivityDate="2017-03-24T10:29:05.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17839" PostTypeId="1" AcceptedAnswerId="17840" CreationDate="2017-03-24T12:09:14.510" Score="48" ViewCount="103509" Body="&lt;p&gt;In the context of &lt;em&gt;Machine Learning&lt;/em&gt;, I have seen the term &lt;em&gt;Ground Truth&lt;/em&gt; used a lot. I have searched a lot and found the following definition in &lt;a href=&quot;https://en.wikipedia.org/wiki/Ground_truth&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In machine learning, the term &amp;quot;ground truth&amp;quot; refers to the accuracy of the training set's classification for supervised learning techniques. This is used in statistical models to prove or disprove research hypotheses. The term &amp;quot;ground truthing&amp;quot; refers to the process of gathering the proper objective (provable) data for this test. Compare with gold standard.&lt;/p&gt;&#xA;&lt;p&gt;Bayesian spam filtering is a common example of supervised learning. In this system, the algorithm is manually taught the differences between spam and non-spam. This depends on the ground truth of the messages used to train the algorithm – inaccuracies in the ground truth will correlate to inaccuracies in the resulting spam/non-spam verdicts.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The point is that I really can not get what it means. Is that the &lt;em&gt;label&lt;/em&gt; used for each &lt;em&gt;data object&lt;/em&gt; or the &lt;em&gt;target function&lt;/em&gt; which gives a label to each &lt;em&gt;data object&lt;/em&gt;, or maybe something else?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2020-09-15T14:01:41.990" Title="What is Ground Truth" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17840" PostTypeId="2" ParentId="17839" CreationDate="2017-03-24T12:17:44.930" Score="43" Body="&lt;p&gt;The ground truth is what you measured for your target variable for the training and testing examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nearly all the time you can safely treat this the same as the label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases it is not precisely the same as the label. For instance if you augment your data set, there is a subtle difference between the ground truth (your actual measurements) and how the augmented examples relate to the labels you have assigned. However, this distinction is not usually a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ground truth can be wrong. It is a measurement, and there can be errors in it. In some ML scenarios it can also be a subjective measurement where it is difficult define an underlying objective truth - e.g.  expert opinion or analysis, which you are hoping to automate. Any ML model you train will be limited by the quality of the ground truth used to train and test it, and that is part of the explanation on the Wikipedia quote. It is also why published articles about ML should include full descriptions of how the data was collected.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-03-24T13:18:04.840" LastActivityDate="2017-03-24T13:18:04.840" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="17843" PostTypeId="2" ParentId="17532" CreationDate="2017-03-24T14:00:00.670" Score="1" Body="&lt;p&gt;Short answer: Topic Models and LDA aren't suited at all for novelty detection. I did some experiments with data that show a succession of topics in temporal order, and at the boundaries of the time interval one topic gets really high percentages in the document-topic matrix, i.e., the upcoming new trend (or the outrunning old trend) is not seen by LDA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Topic Models are also not really suited for &quot;online&quot; processing—you read the whole corpus of documents once and than infer the topics.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastEditorDisplayName="user10169" LastEditDate="2017-03-24T14:15:00.230" LastActivityDate="2017-03-24T14:15:00.230" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17844" PostTypeId="2" ParentId="17822" CreationDate="2017-03-24T14:55:17.440" Score="0" Body="&lt;p&gt;I would look at &lt;a href=&quot;https://stackoverflow.com/questions/3949226/calculating-pearson-correlation-and-significance-in-python&quot;&gt;this answer&lt;/a&gt; here.&lt;/p&gt;&#xA;&lt;p&gt;Statistically speaking, there may be a correlation if you see variables moving together (either towards or opposite) which can help you understand the relationship. You would want to use the Readings column here.&lt;/p&gt;&#xA;" OwnerUserId="30325" LastEditorUserId="29169" LastEditDate="2021-02-06T18:36:54.863" LastActivityDate="2021-02-06T18:36:54.863" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17846" PostTypeId="2" ParentId="17810" CreationDate="2017-03-24T15:07:25.983" Score="2" Body="&lt;p&gt;It depends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're basically asking if your sample (training data) is representative of the population (all written words). &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are you doing sentiment analysis on movie reviews? It'll work great.&lt;/li&gt;&#xA;&lt;li&gt;Are you doing sentiment analysis on TV reviews? It'll probably work great.&lt;/li&gt;&#xA;&lt;li&gt;Are you doing sentiment analysis on book reviews? I would give better than 50-50 odds it'll work.&lt;/li&gt;&#xA;&lt;li&gt;Are you doing sentiment analysis on Twitter posts? Now we're getting shaky. People tend to write much less, use less formal language, and use more emojis which your movie review model wouldn't have seen.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;That being said, there are definitely &quot;generic&quot; sentiment analysis services like &lt;a href=&quot;https://algorithmia.com/algorithms/nlp/SentimentAnalysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Try out your model against Algorithmia on what you would consider a generic set of data (e.g. a bunch of &lt;a href=&quot;https://github.com/guyz/twitter-sentiment-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;tweets&lt;/a&gt;) and see how it does.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-24T15:07:25.983" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17850" PostTypeId="1" CreationDate="2017-03-24T18:10:22.677" Score="3" ViewCount="155" Body="&lt;p&gt;I'm assuming that each time someone trains a model and wants to tweak it/iterate, that they don't have to wait hours and hours for it to learn and then output. So my question is, how do people manage this workflow?&lt;/p&gt;&#xA;" OwnerUserId="30344" LastActivityDate="2017-07-23T06:59:46.910" Title="What is industry standard for workflow when training models?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="2" ClosedDate="2017-07-24T10:44:11.533" ContentLicense="CC BY-SA 3.0" />
  <row Id="17851" PostTypeId="2" ParentId="17850" CreationDate="2017-03-24T18:35:18.343" Score="0" Body="&lt;p&gt;There are online models and there are offline models. Online models make sense when you need to have your model constantly be updated. Otherwise you can save money by updating your model as frequently as necessary; nightly, weekly, etc. The machines used to train models are resources that cost money, after all, and you do not want to reserve them longer than necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One trick to save time is to use the last run's model parameters as initial values for the next iteration.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-03-24T21:06:11.127" LastActivityDate="2017-03-24T21:06:11.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17854" PostTypeId="1" AcceptedAnswerId="18701" CreationDate="2017-03-24T20:19:21.747" Score="1" ViewCount="392" Body="&lt;p&gt;I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can R + Hadoop overcome R's memory constraints in any case?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer might be clear as the accepted answer to &#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/41/is-the-r-language-suitable-for-big-data/44#44/&quot;&gt;this question&lt;/a&gt; implies but for me it isn't. To be more precise:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?&lt;/p&gt;&#xA;" OwnerUserId="30261" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-05-01T08:23:36.763" Title="Can R + Hadoop overcome R's memory constraints in any case?" Tags="|machine-learning|r|bigdata|apache-hadoop|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17855" PostTypeId="1" CreationDate="2017-03-24T20:44:07.477" Score="2" ViewCount="2126" Body="&lt;p&gt;This question is from this &lt;a href=&quot;https://datascience.stackexchange.com/questions/10839/what-is-difference-between-one-hot-encoding-and-leave-one-out-encoding&quot;&gt;post&lt;/a&gt; asking about what is the difference between one-hot encoding and leave-one-out encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now understand how to calculate the leave-one-out encoding for the training set. But how about the test set, where is Y (label) is blinded? If the encoding for test set is always 0.5, then what is the point of this encoding?&lt;/p&gt;&#xA;" OwnerUserId="25867" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2018-02-17T13:38:40.687" Title="why leave-one-out encoding?" Tags="|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17856" PostTypeId="2" ParentId="17407" CreationDate="2017-03-24T21:25:25.370" Score="1" Body="&lt;p&gt;Hope somebody will safe lots of hours:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def backward_propagation(self, x, y, cache):&#xA;    # T - the length of the sequence&#xA;    T = len(y)&#xA;    # perform forward propagation&#xA;    cache = self.forward_propagation(x)&#xA;&#xA;    #...&#xA;&#xA;    # delta for output layer&#xA;    dy = cache['y'].copy()&#xA;    dy[np.arange(len(y)), y] -= 1.0 # softmax loss gradient&#xA;    # print(&quot;dy: &quot;, dy)&#xA;    dhtmp = np.zeros((1, self.hidden_dim))&#xA;    dh_prev = np.zeros((1, self.hidden_dim))&#xA;    dctmp = np.zeros((1, self.hidden_dim))&#xA;&#xA;    for t in np.arange(T)[::-1]:&#xA;        dV += np.outer(dy[t], h[t].T)&#xA;        dhtmp = self.V.T.dot(dy[t]) + dh_prev&#xA;&#xA;        # add to gradients at each previous step&#xA;        do[t] = dhtmp * ct[t]&#xA;        dct[t] = dhtmp * o[t]&#xA;&#xA;        dctmp += dct[t] * (1.0 - ct[t]**2)&#xA;&#xA;        di[t] = dctmp * g[t]&#xA;        df[t] = dctmp * c[t-1]&#xA;        dg[t] = dctmp * i[t]&#xA;&#xA;        # backprop activation functions&#xA;        diga[t] = di[t] * i[t] * (1.0 - i[t])&#xA;        dfga[t] = df[t] * f[t] * (1.0 - f[t])&#xA;        doga[t] = do[t] * o[t] * (1.0 - o[t])&#xA;        dgga[t] = dg[t] * (1.0 - g[t] ** 2)&#xA;&#xA;        # backprop matrix multiply&#xA;        dWi += np.outer(diga[t], h[t-1])&#xA;        dWf += np.outer(dfga[t], h[t-1])&#xA;        dWo += np.outer(doga[t], h[t-1])&#xA;        dWg += np.outer(dgga[t], h[t-1])&#xA;&#xA;&#xA;        dUi[:, x[t]] += diga[t]&#xA;        dUf[:, x[t]] += dfga[t]&#xA;        dUo[:, x[t]] += doga[t]&#xA;        dUg[:, x[t]] += dgga[t]&#xA;&#xA;        # update deltas for next step&#xA;        # here dh is accumulated as shared variable&#xA;        dh_prev = np.dot(self.Wi.T, diga[t])&#xA;        dh_prev += np.dot(self.Wf.T, dfga[t])&#xA;        dh_prev += np.dot(self.Wo.T, doga[t])&#xA;        dh_prev += np.dot(self.Wg.T, dgga[t])&#xA;        dctmp = dctmp * f[t]&#xA;&#xA;    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29714" LastActivityDate="2017-03-24T21:25:25.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17857" PostTypeId="1" CreationDate="2017-03-24T22:23:56.137" Score="12" ViewCount="31830" Body="&lt;p&gt;I am working on a highly-imbalanced binary-labeled dataset, where number of &lt;em&gt;true&lt;/em&gt; labels is just 7% from the whole dataset. But some combination of features could yield higher than average number of ones in a subset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. we have the following dataset with a single feature (color):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;180 red samples —  0&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;20 red samples — 1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;300 green samples — 0&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;100 green samples — 1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We can build a simple decision tree:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                      (color)&#xA;&#xA;                red /       \ green&#xA;&#xA; P(1 | red) = 0.1              P(1 | green) = 0.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;P(1) = 0.2 for the overall dataset&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I run XGBoost on this dataset it can predict probabilities no larger that 0.25. Which means, that if I make a decision at 0.5 threshold:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0 - P &amp;lt; 0.5&lt;/li&gt;&#xA;&lt;li&gt;1 - P &gt;= 0.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then I will always get all samples labeled as &lt;em&gt;zeroes&lt;/em&gt;. Hope that I clearly described the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, on the initial dataset I am getting the following plot (threshold at x-axis):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XYiDR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XYiDR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having maximum of &lt;em&gt;f1_score&lt;/em&gt; at threshold = 0.1. Now I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;should I even use &lt;em&gt;f1_score&lt;/em&gt; for a dataset of such a structure?&lt;/li&gt;&#xA;&lt;li&gt;is it always reasonable to use 0.5 threshold for mapping probabilities to labels when using XGBoost for binary classification?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update.&lt;/strong&gt; I see that topic draws some interest. Below is the Python code to reproduce red/green experiment using XGBoost. It actually outputs the expected probabilities:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from xgboost import XGBClassifier&#xA;from sklearn.model_selection import train_test_split&#xA;import numpy as np&#xA;&#xA;X0_0 = np.zeros(180) # red - 0&#xA;Y0_0 = np.zeros(180)&#xA;&#xA;X0_1 = np.zeros(20) # red - 1&#xA;Y0_1 = np.ones(20)&#xA;&#xA;X1_0 = np.ones(300) # green - 0&#xA;Y1_0 = np.zeros(300)&#xA;&#xA;X1_1 = np.ones(100) # green  - 1&#xA;Y1_1 = np.ones(100)&#xA;&#xA;X = np.concatenate((X0_0, X0_1, X1_0, Y1_1))&#xA;Y = np.concatenate((Y0_0, Y0_1, Y1_0, Y1_1))&#xA;&#xA;# reshaping into 2-dim array&#xA;X = X.reshape(-1, 1)&#xA;&#xA;import xgboost as xgb&#xA;&#xA;xgb_dmat = xgb.DMatrix(X_train, label=y_train)&#xA;&#xA;param = {'max_depth': 1,&#xA;         'eta': 0.01,&#xA;         'objective': 'binary:logistic',&#xA;         'eval_metric': 'error',&#xA;         'nthread': 4}&#xA;&#xA;model = xgb.train(param, xg_mat, 400)&#xA;&#xA;X0_sample = np.array([[0]])&#xA;X1_sample = np.array([[1]])&#xA;&#xA;print('P(1 | red), predicted: ' + str(model.predict(xgb.DMatrix(X0_sample))))&#xA;print('P(1 | green), predicted: ' + str(model.predict(xgb.DMatrix(X1_sample))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;P(1 | red), predicted: [ 0.1073855]&#xA;P(1 | green), predicted: [ 0.24398108]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16162" LastEditorUserId="16162" LastEditDate="2017-03-25T08:30:22.023" LastActivityDate="2017-03-27T01:15:28.253" Title="XGBoost for binary classification: choosing the right threshold" Tags="|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17859" PostTypeId="1" AcceptedAnswerId="17893" CreationDate="2017-03-25T06:32:06.293" Score="1" ViewCount="2401" Body="&lt;p&gt;I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with GridSearchCVs best results.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;clf = KNeighborsClassifier(n_neighbors=15, weights='uniform')&#xA;&#xA;&#xA;clf.fit(features_train, labels_train)&#xA;&#xA;print('Score using optimized parameters: {}'.format(clf.score(features_test,       labels_test)))&#xA;&#xA;&#xA;params = {'n_neighbors':[1,10,15,20,25,30,35,40,45,50,60,70,80,90,100],    'weights':['uniform', 'distance']}&#xA;grid = GridSearchCV(clf, params, cv=10, )&#xA;grid.fit(features_train, labels_train)&#xA;&#xA;print('Optimized Parameters:{}'.format(grid.best_params_))&#xA;print('Best Score from GridsearchCV parameters{}'.format(grid.best_score_))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Score using optimized parameters: 0.928&lt;/p&gt;&#xA;&lt;p&gt;Optimized Parameters:{'n_neighbors': 15, 'weights': 'uniform'}&lt;/p&gt;&#xA;&lt;p&gt;Best Score from GridsearchCV parameters: 0.962666666667&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="30355" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-03-27T04:09:49.440" Title="Can't reproduce results from GridSearchCV?" Tags="|machine-learning|python|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17860" PostTypeId="1" AcceptedAnswerId="17864" CreationDate="2017-03-25T08:39:13.420" Score="7" ViewCount="1221" Body="&lt;p&gt;I am studying reinforcement learning and I am working methodically through &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton and Barto's book&lt;/a&gt; plus &lt;a href=&quot;https://www.youtube.com/playlist?list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT&quot; rel=&quot;nofollow noreferrer&quot;&gt;David Silver's lectures&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have noticed a minor difference in how the Markov Decision Processes (MDPs) are defined in those two sources, that affects the formulation of the Bellman equations, and I wonder about the reasoning behind the differences and when I might choose one or the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Sutton and Barto, the expected reward function is written $R^a_{ss'}$, whilst in David Silver's lectures it is written $R^a_{s}$. In turn this leads to slightly different formulations of all the Bellman equations. For instance, in Sutton and Barto, the equation for policy evaluation is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s'))&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst David Silver's lectures show:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;v_{\pi}(s) = \sum_a \pi(a|s) \left(R_{s}^a + \gamma \sum_{s'} P_{ss'}^a v_{\pi}(s') \right)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\pi(a|s)$ is policy function - probability of choosing action $a$ given state $s$. &lt;/li&gt;&#xA;&lt;li&gt;$\gamma$ is discount factor.&lt;/li&gt;&#xA;&lt;li&gt;$P_{ss'}^a$ is transition function, probability of state changing to $s'$ given $s, a$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I understand that $R_{ss'}^a$ and $R_{s}^a$ are related (via $P_{ss'}^a$), so that these two sources are explaining the exact same thing. Note that the first equation can also be written as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;v_{\pi}(s) &#xA;&amp;amp;= \sum_a \pi(a|s) \sum_{s'} (P_{ss'}^aR_{ss'}^a + \gamma P_{ss'}^av_{\pi}(s'))\\&#xA;&amp;amp;= \sum_a \pi(a|s) \left( \sum_{s'} P_{ss'}^aR_{ss'}^a + \sum_{s'} \gamma P_{ss'}^av_{\pi}(s') \right) \\&#xA;&amp;amp;= \sum_a \pi(a|s) \left( \sum_{s'} P_{ss'}^aR_{ss'}^a + \gamma \sum_{s'}  P_{ss'}^av_{\pi}(s') \right)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, it must be true that $R_{s}^a = \sum_{s'} P_{ss'}^a R_{ss'}^a$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether there is any reason I should prefer to use one or the other notation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started with Sutton and Barto, and find that notation more intuitive - the reward may depend on the eventual state, and this is explicit in the equations. However, it looks like in practice that the notation used in the video lectures describes more efficient calculations (essentially $R_{s}^a = \sum_{s'} P_{ss'}^a R_{ss'}^a$ is cached, if the formula is translated &lt;em&gt;directly&lt;/em&gt; to code). Is that all there is to it?&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorDisplayName="user10640" LastEditDate="2018-07-27T13:48:33.103" LastActivityDate="2021-08-07T17:44:19.177" Title="Reward dependent on (state, action) versus (state, action, successor state)" Tags="|reinforcement-learning|markov-process|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17861" PostTypeId="2" ParentId="17837" CreationDate="2017-03-25T11:17:22.517" Score="2" Body="&lt;p&gt;Let's say you fit n models on given dataset, to put them together, you have in fact few options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Take the average (I'd play with weighted or geometric)&lt;/li&gt;&#xA;&lt;li&gt;Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; for that.&lt;/li&gt;&#xA;&lt;li&gt;Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.&lt;/li&gt;&#xA;&lt;li&gt;You can go further for blending, boosting or bagging, there is really nice article about these methods &lt;a href=&quot;http://mlwave.com/kaggle-ensembling-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-03-25T11:17:22.517" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17862" PostTypeId="1" CreationDate="2017-03-25T12:17:19.890" Score="1" ViewCount="634" Body="&lt;p&gt;Is there any way to determine upfront how accurate a model should be before it's worth putting into production? Just aiming for the highest accuracy possible doesn't tell us anything about the money a company would save with a 75% versus 80% model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say we needed to predict the number of mechanical failures. If the national average is 100/year but our company is getting 125, how accurate does our model need to be (how many correct predictions does it need to make) to get our company back down the national average? &lt;/p&gt;&#xA;" OwnerUserId="30359" LastActivityDate="2017-03-26T14:58:38.863" Title="How accurate does my machine learning model need to be?" Tags="|machine-learning|predictive-modeling|statistics|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17864" PostTypeId="2" ParentId="17860" CreationDate="2017-03-25T13:40:02.270" Score="3" Body="&lt;p&gt;Your intuition is correct. In the most general case (Sutton's definitions), the model of the environment consists of the state transition distribution and the reward distribution. The latter one is rarely considered, as many times the reward is being assigned by the modeler, dependent only by the selected action from the current state and being deterministic. As you mentioned it simplifies a lot the coding implementation.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-03-25T13:40:02.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17865" PostTypeId="1" CreationDate="2017-03-25T14:16:11.787" Score="0" ViewCount="819" Body="&lt;p&gt;I have a dataset of following format -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Number of Machines&#xA;CustId   Month0   Month-1   Month-2   Month-3   Month-4&#xA;abc      23       26        29        0         0&#xA;def      53       26        22        22        12&#xA;ghi      11       26        150       120       10&#xA;&#xA;Size of data protected&#xA;CustId   Month0   Month-1   Month-2   Month-3   Month-4&#xA;abc      23       26        29        0         0&#xA;def      53       26        22        22        12&#xA;ghi      11       26        150       120       10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data is Month-over-Month data. For simplicity I have used the same data in both tables, but for a given CustId data will be present in both the tables. Similarly there are tables for other parameters as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use machine learning for some classification. What is the best way to serialize this MoM data for different parameters? Is there any standard practice for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure of what are the distinctive patterns in the graphs of the parameters for different classification categories. By graph of the parameter I mean Number of machines plotted against Month.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I can do is to combine all the tables with format something like - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    CustId   ----# machine---       -------data protected-----&#xA;             Month0   Month-1 ...   Month0   Month-1   Month-2 ...&#xA;    abc      23       26      ...   23       26        29      ...&#xA;    def      53       26      ...   53       26        22      ...&#xA;    ghi      11       26      ...   11       26        150     ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I don't want to use this format because in here all the columns (monthly data) would be treated as independent data points. I want the columns under '# machine&quot; category to be treated as group and the columns under &quot;data protected&quot; as another group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across this &lt;a href=&quot;http://tsfresh.readthedocs.io/en/latest/text/introduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Library (tsFresh)&lt;/a&gt; which extracts information from the graphs. I am looking for something similar or any other way to approach this problem.&lt;/p&gt;&#xA;" OwnerUserId="30363" LastEditorUserId="30363" LastEditDate="2017-03-28T06:03:27.653" LastActivityDate="2017-10-01T08:08:51.690" Title="How to represent time based periodic data for use in ML" Tags="|machine-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17866" PostTypeId="1" CreationDate="2017-03-25T16:24:20.010" Score="1" ViewCount="63" Body="&lt;p&gt;For one hidden layer exist Universal approximation theorem by  George Cybenko.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for multi layers net converges result is stohastic or determ and guaranteed as one hidden layer net?&#xA;And is there an appropriate proof?&lt;/p&gt;&#xA;" OwnerUserId="29206" LastActivityDate="2017-03-25T16:24:20.010" Title="Is it true that a neural network with many layers converges to a solution?" Tags="|neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17867" PostTypeId="1" AcceptedAnswerId="17869" CreationDate="2017-03-25T18:31:07.617" Score="0" ViewCount="56" Body="&lt;p&gt;Let's say there is a distribution, call it D, for which I don't know details (i.e. mean and variance) but can assume that it's a normal distribution. I now have N samples from D. I cannot take more samples because it's too heavy. Given the N samples, and the assumption that D is a normal distribution, how can I compute the mean and variance of the mean of D? Naturally it's less likely for the mean of D to be far away from the samples (thought still possible), while more likely to be near the samples.&lt;/p&gt;&#xA;" OwnerUserId="29428" LastActivityDate="2017-03-25T19:10:24.227" Title="Estimate the normal distribution of the mean of a normal distribution given a set of samples?" Tags="|distribution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17868" PostTypeId="1" AcceptedAnswerId="19237" CreationDate="2017-03-25T18:57:42.863" Score="4" ViewCount="1334" Body="&lt;p&gt;I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced  as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the confusion matrix is of no help too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can shed any light on what approach should I take, that would really help me move forward.&lt;/p&gt;&#xA;" OwnerUserId="30370" LastActivityDate="2020-03-29T17:43:06.590" Title="Usage of Precision Recall on an unbalanced dataset" Tags="|machine-learning|r|classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17869" PostTypeId="2" ParentId="17867" CreationDate="2017-03-25T19:10:24.227" Score="1" Body="&lt;p&gt;The same can be done through any of the statistical estimation technique such as maximum likelihood or Minimum variance unbiased estimator(MVUE). Generally maximum likelihood is the easiest way to evaluate the mean and variance of the distribution. If you perform Maximum likelihood method the estimated mean of the normal distribution is the sample mean. Given x1,x2.....xn are the samples &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimated mean u = (x1 + x2 + ... + xn)/n&#xA;Similarly the estimated variance is  Σ(xi - u)^2/n&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both of these are obtained by setting the partial derivative of the likelihood function with respect to the mean and the standard deviation to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="28337" LastActivityDate="2017-03-25T19:10:24.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17870" PostTypeId="2" ParentId="17854" CreationDate="2017-03-25T19:39:24.143" Score="1" Body="&lt;p&gt;To answer your question. I would give an analogous answer to clarify it better. &lt;strong&gt;Apache Spark&lt;/strong&gt; is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to that &lt;strong&gt;R+ Hadoop&lt;/strong&gt; also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing &lt;strong&gt;multithreading&lt;/strong&gt; to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;  &lt;strong&gt;As requested more elaboration on the last point&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply put when looking at just &lt;strong&gt;R&lt;/strong&gt; as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;And to answer your question whether R can perform the same tasks as R+Hadoop provided if there are no time constraints&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is theoretically Yes practically No!. Intuitively we might think R can handle such tasks by consuming more time. But that isn't always the case. It depends more on the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For example&lt;/strong&gt;, if R were doing a simple frequency Count of the number occurrences in an array of a few million records you can obviously say that even with a few threads it might be slower than R+Hadoop and finish the task after a certain amount of time. But if the task were to say something like Eigen Value Decomposition of a Matrix, such a transformation requires a lot of calculation at the element level of a matrix. Assuming that the size of that matrix is a few million x few million, this type of calculation will surely cause a memory error using just any normal application as it will bottleneck all the data it has to hold temporarily into the RAM. This bottlenecking problem might cause the whole system to fail at times but the most simple case is that of a memory error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using R+Hadoop solves such problems as it lets you distribute the same strategy over multiple computers with their corresponding cores. Hence it lets you handle loads much more easily than running it on the same application, this manner you can also prevent the errors which I mentioned above. This is the power of distributed computing.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastEditorUserId="30107" LastEditDate="2017-03-30T18:47:58.547" LastActivityDate="2017-03-30T18:47:58.547" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17871" PostTypeId="1" CreationDate="2017-03-25T20:27:11.967" Score="1" ViewCount="98" Body="&lt;p&gt;I am trying to find a cutoff value, in the feature importance space to eliminate spurious features. So I am injecting a completely random generated feature (as one of the input features) and I cut the features that are scoring below the value of the random feature in the importance ranking. But once I removed the features, the performance drop. I don't understand what is happening. I checked this behavior in a gradient boost and in a random forest. Someone has a clue of what is happening?&lt;/p&gt;&#xA;" OwnerUserId="10938" LastActivityDate="2017-03-25T20:27:11.967" Title="Injecting random values as one input feature for feature selection results in a odd beaviour" Tags="|feature-selection|random-forest|decision-trees|xgboost|gbm|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17872" PostTypeId="1" CreationDate="2017-03-25T23:44:58.557" Score="1" ViewCount="1154" Body="&lt;p&gt;As part of a more elaborate project, I am trying to get a rather simple network to distinguish between 5 very distinct classes of 1D signals (the members of each class is just the same bit of signal copied over 1000 times with a slight bit of noise added). And failing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the network setup, and the output. As far as I can tell (through matplotlib), I haven't done anything really silly with data (such as flip the dimensions, for instance). Does anyone have a suggestion for what could be going wrong? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize for not generating dummy data within the script, I'm still much more comfortable in matlab, so it was easier to just port some dummy data from there. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(Flatten(input_shape=(6000,1),name='S2'))&#xA;model.add(Dense(40,activation='relu',name='F1'))&#xA;model.add(Dense(10,activation='relu',name='F2'))&#xA;model.add(Dense(5,activation='relu',name='output'))&#xA;model.add(Activation(activation='softmax',name='softmax'))&#xA;&#xA;model.summary()&#xA;&#xA;# load fake data&#xA;earEEG=scipy.io.loadmat('forPython_dummy.mat')&#xA;data=earEEG['data'][:,:]&#xA;data=np.transpose(data)&#xA;data=data[:,:,None]&#xA;&#xA;labels=earEEG['labels']&#xA;labels= keras.utils.to_categorical(labels)&#xA;&#xA;&#xA;sgd=keras.optimizers.SGD()&#xA;model.compile(loss='categorical_crossentropy',&#xA;              optimizer=sgd,&#xA;              metrics=['accuracy'])&#xA;&#xA;hist=model.fit(data, labels, epochs=10, batch_size=50,class_weight='auto',shuffle='True')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Layer (type)                 Output Shape              Param #   &#xA;=================================================================&#xA;S2 (Flatten)                 (None, 6000)              0         &#xA;_________________________________________________________________&#xA;F1 (Dense)                   (None, 40)                240040    &#xA;_________________________________________________________________&#xA;F2 (Dense)                   (None, 10)                410       &#xA;_________________________________________________________________&#xA;output (Dense)               (None, 5)                 55        &#xA;_________________________________________________________________&#xA;softmax (Activation)         (None, 5)                 0         &#xA;=================================================================&#xA;Total params: 240,505.0&#xA;Trainable params: 240,505.0&#xA;Non-trainable params: 0.0&#xA;_________________________________________________________________&#xA;Epoch 1/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;Epoch 2/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;Epoch 3/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;Epoch 4/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;Epoch 5/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;Epoch 6/10&#xA;5000/5000 [==============================] - 0s - loss: 12.8945 - acc: 0.2000     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Clearly, the algorithm is just giving the same label to everything (this is also apparent when outputting the actual labels). The input is a 5000 x 6000 matrix, and 5000 labels. As the algorithm does not complain about dimensional mismatch, I feel pretty confident that the data is not being transposed or similar. As an easy test, I tried feeding the same dataset to a pca+decision tree in matlab, and got a perfect score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classes are perfectly balanced, by the way. &lt;/p&gt;&#xA;" OwnerUserId="30210" LastActivityDate="2017-03-27T09:44:34.423" Title="Why does my neural net always converge on a trivial solution?" Tags="|neural-network|classification|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17873" PostTypeId="1" CreationDate="2017-03-26T06:32:14.227" Score="3" ViewCount="9155" Body="&lt;p&gt;I plan to use many methods to solve the imbalanced dataset problem on the training set. But I couldn't find any paper that describes how they dealt with the test dataset? I assume that they just tested on the original dataset without any adjustments? Will I need to adjust the threshold on the test set with original imbalanced ratio?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;update. Thanks for everyone's response!&lt;/p&gt;&#xA;&lt;p&gt;I've found a paper that discusses how we should adjust the posterior probability and threshold.&#xA;&lt;a href=&quot;http://www.ulb.ac.be//di/map/adalpozz/pdf/SSCI_calib_final_noCC.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Calibrating Probability with Undersampling&#xA;for Unbalanced Classification&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;However, I also believe that in practice we can just fit test set directly.&lt;/p&gt;&#xA;" OwnerUserId="30376" LastEditorUserId="85045" LastEditDate="2021-02-09T17:01:12.340" LastActivityDate="2021-02-09T17:01:12.340" Title="Imbalanced dataset: how to deal with test data?" Tags="|class-imbalance|sampling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17876" PostTypeId="1" AcceptedAnswerId="17897" CreationDate="2017-03-26T10:05:56.357" Score="2" ViewCount="2079" Body="&lt;p&gt;-1&#xA;down vote&#xA;favorite&#xA;I have a dataset including missing data for most of the variables. Assume the dataset is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Obs.  var1  var2  var3  var4  var5  var6&#xA;1     x11   x12   x13   x14    Nan  Nan&#xA;2     x21   x22   x23   Nan    x25  x26&#xA;3     x31   x32   x33   x34    x35  x36&#xA;...&#xA;n     xn1   xn2   xn3   xn4    Nan  xn6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have split the dataset to d1 where we have complete data for all variables and d2 where all records have at least one missing variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made different models using KNN: To predict the values of var5 and var6 for the first observation, I used d1 (dataset without missing value) and modeled on var1, var2, var3 and var4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To predict the value of var5 for the last observation, I used d1 and modeled on var1, var2, var3, var4, and var5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does my approach make sense?! Any suggestion are welcome. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="30381" LastActivityDate="2018-02-27T16:59:35.880" Title="Missing data imputation with KNN" Tags="|missing-data|data-imputation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17877" PostTypeId="1" CreationDate="2017-03-26T10:10:38.683" Score="1" ViewCount="213" Body="&lt;p&gt;Are there any consumer level graphics cards (i.e. less than, say, $1500) that have decent half precision (fp16) support?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latest Titan X was rumoured to have native fp16 support but that didn't turn out to be true. Are there any others that support fp16, maybe on older architectures (but are still semi-decent for deep learning)?&lt;/p&gt;&#xA;" OwnerUserId="12206" LastEditorUserId="29169" LastEditDate="2021-02-16T02:44:41.730" LastActivityDate="2021-02-16T02:44:41.730" Title="Machine learning graphics card - half precision performance" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|gpu|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17880" PostTypeId="2" ParentId="17862" CreationDate="2017-03-26T14:58:38.863" Score="4" Body="&lt;p&gt;The accuracy level that should satisfy you depends on the reliability that your customer would have on it and the costs matrix of false predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He should put into consideration the costs of the unpredicted failures (low recall of your model) causing him production delays and the costs of over/false maintenance alarms (low precision of your model). If you know these costs you can minimize the expected cost equation which considers the precision recall and accuracy of your model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expected cost&#xA;  = p(true positive) × &quot;maintenance cost&quot; &#xA;  + p(false positive) × &quot;inspection cost&quot; &#xA;  + p(true negative) × 0 (no action)&#xA;  + p(false negative) × &quot;mechanical failure cost&quot;&lt;/p&gt;&#xA;" OwnerUserId="30385" LastActivityDate="2017-03-26T14:58:38.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17883" PostTypeId="2" ParentId="17873" CreationDate="2017-03-26T15:55:39.997" Score="1" Body="&lt;p&gt;The idea of balancing the training set + validating the balancing method is for being able to generalize your model that is would discriminate (in classification assignment) better a sample from the minority class, in an unseen and imbalanced test set. &#xA;Meaning, that the model already adjusted the classification boundaries for considering the minority class underrepresentation and you shouldn't adjust the test set for it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you should make sure that both sets are chosen randomly, for having approximately the same representation ratios of examples from each feature&lt;/p&gt;&#xA;" OwnerUserId="30385" LastActivityDate="2017-03-26T15:55:39.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17884" PostTypeId="2" ParentId="17873" CreationDate="2017-03-26T17:37:58.233" Score="7" Body="&lt;p&gt;You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy. &#xA;In overall accuracy you find ( number of samples predicted correctly/ total number of samples) in average accuracy, you find the overall accuracy per class and then you find the average of these overall accuracies. &#xA;When you know that you are working with imbalanced database, where all classes are important, you should use the average accuracy &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand what  this means: imagine you have two classes, class A and class B , and the ratio is 90 to 10 .&#xA;If you are sampling randomly for the training and testing, then the ratio is still 90:10 in the testing set. If your model is very biased , that predicts all the samples to be class A , then:&#xA;Overall accuracy = 90% &#xA;Average accuracy = 50 % &#xA;( 100% for class A + 0% for class B) / 2 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The overall accuracy is really high but it does not reflect the actual quality of the model. The average accuracy gives you a better indication of the quality &lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2017-03-26T17:47:09.840" LastActivityDate="2017-03-26T17:47:09.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17885" PostTypeId="1" AcceptedAnswerId="18863" CreationDate="2017-03-26T18:40:50.630" Score="1" ViewCount="1312" Body="&lt;p&gt;&lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Original blog post&lt;/a&gt; claims that it is possible to get 95% accuracy on the validation set (20 Newsgroup dataset) after only 2 epochs using pretrained word embeddings (glove.6B.100d). All code is located &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/pretrained_word_embeddings.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. I did not changed anything in this example but getting only 40% accuracy on the validation set after 2 epochs and 75% after 10 epochs. I can't get to 95% accuracy even after 20 epochs. Switching from tensorflow to theano backend do not make any significant change. I'm using Keras 2.0.2, tensorflow-gpu 1.0.1, theano 0.9.0, python 2.7.12. What I'm doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="30396" LastActivityDate="2017-05-09T23:07:57.877" Title="Getting low accuracy on keras pretrained word embeddings example" Tags="|python|tensorflow|keras|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="17888" PostTypeId="1" AcceptedAnswerId="17889" CreationDate="2017-03-26T22:58:12.367" Score="0" ViewCount="1907" Body="&lt;p&gt;&lt;strong&gt;Input:&lt;/strong&gt; I have csv file like below as input....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID, Year,Specialty,AgeRange,PlaceSvc,Count, Group&#xA;101,2009,Internal,  20-29,  Office,     0,  PRGNCY&#xA;101,2010,Emergency, 20-29,  Urgent Care,0,  GIOBSENT&#xA;101,2011,Internal,  20-29,  Office,     0,  GYNEC1&#xA;102,2010,Other,     30-39,  Office,     1,  PRGNCY&#xA;102,2010,Laboratory,30-39,  Independent,1,  MSC2a3&#xA;103,2009,Laboratory,30-39,  Independent,1,  MSC2a3&#xA;103,2011,Other,     30-39,  Office,     0,  PRGNCY&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output:&lt;/strong&gt; I want output like below...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID,Year,Specialty_Internal,Specialty_Emergency,Specialty_Labrotory,Specialty_Other,Age20_29,Age30_39,PlaceSvc_Urgent,PlaceSvc_Office,PlaceSvc_Independent,Count,GroupPrgncy,GroupGiobsent,GroupGynec1,GroupMsc2a3&#xA;&#xA;101,2009,1,0,0,0,1,0,0,1,0,0,1,0,0,0&#xA;101,2010,0,1,0,0,1,0,1,0,0,0,0,1,0,0&#xA;101,2011,1,0,0,0,1,0,0,1,0,0,0,0,1,0&#xA;102,2010,0,0,1,1,0,1,0,1,1,2,1,0,0,1&#xA;103,2009,0,0,1,0,0,1,0,0,1,1,0,0,0,1&#xA;103,2011,0,0,0,1,,1,0,1,0,0,1,0,0,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can i do this by pandas? or is there any other techinque to do this?&lt;/p&gt;&#xA;" OwnerUserId="29824" LastEditorUserId="381" LastEditDate="2017-03-26T23:12:15.113" LastActivityDate="2017-03-26T23:12:15.113" Title="How to shift rows values as columns in pandas?" Tags="|python|data|pandas|csv|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17889" PostTypeId="2" ParentId="17888" CreationDate="2017-03-26T23:05:02.050" Score="2" Body="&lt;p&gt;You want &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas.get_dummies&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you call &lt;code&gt;get_dummies&lt;/code&gt; on a categorical column, it will output the binary dummy variables you're looking for. You should then be able to merge this with your original DataFrame on the index, or construct a new DataFrame using only the columns you want.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-03-26T23:05:02.050" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17890" PostTypeId="2" ParentId="17857" CreationDate="2017-03-27T01:15:28.253" Score="9" Body="&lt;p&gt;You have to decide what you want to maximize.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classifying by comparing the probability to 0.5 is appropriate if you want to maximize accuracy.  It's not appropriate if you want to maximize the f1 metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to maximize accuracy, always predicting zero &lt;em&gt;is&lt;/em&gt; the optimal classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, given a probability score $p$, another option is to randomly flip a biased coin; with probability $p$, output classification 1, otherwise output classification 0.  This doesn't always predict zero.  However it's probably not actually any better in any useful way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to maximize f1 metric, one approach is to train your classifier to predict a probability, then choose a threshold that maximizes the f1 score.  The threshold probably won't be 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to understand the cost of type I errors vs type II errors, and then assign class weights accordingly.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-27T01:15:28.253" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17891" PostTypeId="1" AcceptedAnswerId="24028" CreationDate="2017-03-27T03:23:51.823" Score="0" ViewCount="961" Body="&lt;p&gt;I am working on a classification problem, and am applying gradient boosted tress on the dataset, to classify the items into two classes (Fraud -&gt; 1 and No Fraud -&gt; 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the below code for my glm methos : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;formula = paste(class.label,paste(vars,collapse = ' + '), sep = ' ~ ')&#xA;&#xA;modelGBM = gbm(as.formula(formula), data =  creditdataset,&#xA;           distribution = 'bernoulli',&#xA;           n.trees = 400,&#xA;           interaction.depth = 3,&#xA;           shrinkage = 0.05,&#xA;           bag.fraction = 0.5,&#xA;           keep.data = FALSE,&#xA;           cv.folds = 5&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where creditdataset is my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I execute the code , I get the following error message  :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Error in serialize(data, node$con) : error writing to connection&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this kind of error has been discussed here : &#xA; [&lt;a href=&quot;https://stackoverflow.com/questions/28503208/doparallel-error-in-r-error-in-serializedata-nodecon-error-writing-to-con][1]&quot;&gt;https://stackoverflow.com/questions/28503208/doparallel-error-in-r-error-in-serializedata-nodecon-error-writing-to-con][1]&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and here : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/37943295/parallel-programming-in-r-error-writing-to-connection&quot;&gt;https://stackoverflow.com/questions/37943295/parallel-programming-in-r-error-writing-to-connection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but these two posts don't really solve the problem. Can anyone point me in the right direction here.&lt;/p&gt;&#xA;" OwnerUserId="30370" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-10-24T14:22:53.240" Title="gbm function not working" Tags="|machine-learning|r|classification|gbm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17892" PostTypeId="2" ParentId="17868" CreationDate="2017-03-27T03:26:24.960" Score="-1" Body="&lt;p&gt;Found a solution, for an unbalanced dataset, first use SMOTE and then apply any model to use check AUC&lt;/p&gt;&#xA;" OwnerUserId="30370" LastActivityDate="2017-03-27T03:26:24.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17893" PostTypeId="2" ParentId="17859" CreationDate="2017-03-27T04:09:49.440" Score="1" Body="&lt;p&gt;The score from your GridsearchCV is biased.  You can use cross-validation either for estimating accuracy, or for choosing hyperparameters; but not both.  If you use cross-validation to pick the best choice of hyperparameters, by measuring the accuracy of each possible option, the accuracy you got for the option you chose will tend to overestimate the accuracy you'll see on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid this bias, select a separate hold-out validation set for estimating the accuracy of your selected parameters, or use nested cross-validation (or a scikit-learn &lt;code&gt;Pipeline&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://datascience.stackexchange.com/a/17835/8560&quot;&gt;https://datascience.stackexchange.com/a/17835/8560&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2017-03-27T04:09:49.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17895" PostTypeId="2" ParentId="17755" CreationDate="2017-03-27T06:39:11.147" Score="1" Body="&lt;p&gt;The theory behind cross validation ( v-fold cross validation) has been addressed in many papers. There is a proof for that in a set papers published from 2003-2007.&#xA;Please refer to :&#xA;- oracle selector. 2006&#xA;- super learner 2007&#xA;- super learner in prediction 2010&#xA;- unified cross validation 2003&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-03-27T06:39:11.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17896" PostTypeId="1" CreationDate="2017-03-27T06:51:03.553" Score="0" ViewCount="32" Body="&lt;p&gt;Is there any way get a constraint on features (e.g., ear longer than 100cm and body weight&gt;20kg and ...) or (...)) where it could say 90% guarantee it is a dog or cat? Say if we want to classify dog/cat.&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="7812" LastEditDate="2017-03-27T07:11:26.390" LastActivityDate="2017-03-27T13:27:59.713" Title="Getting kinds on features that has high confidence on classfication" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17897" PostTypeId="2" ParentId="17876" CreationDate="2017-03-27T07:32:57.197" Score="1" Body="&lt;p&gt;There are various approaches for dealing with missing values. Suppose we've got 4 instances in a dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 = [1 2 3]&#xA;x2 = [1 ? 3]&#xA;x3 = [2 4 2]&#xA;x4 = [1 3 3.5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;one simple approach (&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2833083&quot; rel=&quot;nofollow noreferrer&quot;&gt;specially popular in medical datasets&lt;/a&gt;) is finding values with regard to most similar instance; in the above case, missing value of &lt;code&gt;x2&lt;/code&gt; would be then: 2 (since &lt;code&gt;x1&lt;/code&gt; is the most similar)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;a more sophisticated approach is weighted averaging through k most similar instances (of course only applicable if the missing value is numeric or at least ordinal) in this case, you should calculate: &lt;code&gt;x2(2) = (2*(1-(0/3)) + 3*(1-(0.5/3)) / (2-(0/3)-(0.5/3))&lt;/code&gt; (&lt;code&gt;x4&lt;/code&gt; has also been counted as a similar case&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;another approach, is voting among k most similar instances (applicable both for categorical and ordinal values).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;cases 1 and 3 are what you have implemented (case1 = 1 Nearest Neighbor / case2 = k-nearest neighbors)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;there are other approaches for handling a missing value, but it depends on what you're going to do with your dataset. For example in very large datasets, sometimes it is efficient to simply ignore every instance that contains at least one missing value, or ignore only the missing value (not the entire instance) in further processes (e.g. in &lt;a href=&quot;http://www.cs.bilkent.edu.tr/tech-reports/1997/BU-CEIS-9708.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;VFI&lt;/a&gt; algorithm)&lt;/p&gt;&#xA;" OwnerUserId="30043" LastEditorUserId="30043" LastEditDate="2017-03-27T07:44:39.710" LastActivityDate="2017-03-27T07:44:39.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17898" PostTypeId="2" ParentId="2368" CreationDate="2017-03-27T08:47:14.667" Score="3" Body="&lt;p&gt;Ben is talking about &lt;strong&gt;the static features&lt;/strong&gt;, and make use of the timestamp features.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an extension, i will introduce &lt;strong&gt;the lag features&lt;/strong&gt;, I am not talking the raw time series, but the aggregates on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most mystical part is that the future value is unseen for us, how can we use that aggregate features in the training data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A little example:&lt;/strong&gt; &#xA;There is yearly electric consumption data from 1991 to 2015, I want predict the electric consumption in the future 5 years, 2016 to 2020. I will calculate the last 5 years moving average of electric consumption as the 2020's feature values, but the 2016 to 2020 is unknown for us, so we leading (opposite the lagging) the time series 5 years, lets do the moving average on 2010 to 2015, then use this value as 2020's feature values. So, we can construct the future 5 years' feature data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next step is just using the moving function (count\mean\median\min\max.etc) and try different windows, then you will construct lots of features!&lt;/p&gt;&#xA;" OwnerUserId="19275" LastEditorUserId="19275" LastEditDate="2017-03-27T08:54:42.273" LastActivityDate="2017-03-27T08:54:42.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17899" PostTypeId="1" CreationDate="2017-03-27T09:00:07.037" Score="3" ViewCount="4602" Body="&lt;p&gt;I want to do sequence learning for that I want to find frequent sequential rules ,this rule consider the order of occurrence &lt;/p&gt;&#xA;" OwnerUserId="29431" LastActivityDate="2023-10-21T13:04:25.487" Title="best python library for finding sequential rules mining?" Tags="|python|data-mining|sequential-pattern-mining|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17900" PostTypeId="2" ParentId="17872" CreationDate="2017-03-27T09:44:34.423" Score="1" Body="&lt;p&gt;The problem turned out to be that the input had not been properly scaled - making sure all epochs had standard deviation = 1 solved it. &lt;/p&gt;&#xA;" OwnerUserId="30210" LastActivityDate="2017-03-27T09:44:34.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17902" PostTypeId="2" ParentId="17899" CreationDate="2017-03-27T12:09:44.087" Score="1" Body="&lt;p&gt;Probably you could try Conditional Random Fields (&lt;a href=&quot;https://github.com/scrapinghub/python-crfsuite&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scrapinghub/python-crfsuite&lt;/a&gt;) or a Long Short term Memory RNN (Keras+Tensorflow backend)&lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2017-03-27T12:09:44.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17903" PostTypeId="1" AcceptedAnswerId="17904" CreationDate="2017-03-27T12:12:36.990" Score="0" ViewCount="491" Body="&lt;p&gt;I was asked to verify the feasibility for solving a particular problem: recognizing for a fashion brand the model of its products. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have little experience with image recognition in general, I always used Google Vision Api or some pre trained nets from Google/tensorflow like the VGG-16. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering: &lt;strong&gt;Do you think this level of granularity can be achieved?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I should for example train the last layer of a pre-trained network, but I need many images to do that.&lt;/p&gt;&#xA;" OwnerUserId="27767" LastActivityDate="2017-03-27T12:34:49.667" Title="State of the art for Object detection/image recognition" Tags="|neural-network|deep-learning|tensorflow|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17904" PostTypeId="2" ParentId="17903" CreationDate="2017-03-27T12:34:49.667" Score="1" Body="&lt;p&gt;Maybe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do they have a unique feature on the item  (e.g. a polo player, a red sole) the model can be trained on? I have seen people try identifying clothing model from images via Google's Vision API and it didn't work well. If you are in a fashion category where the business model is to copy designs like H&amp;amp;M or Zara do, you're likely going to have trouble. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-27T12:34:49.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17905" PostTypeId="2" ParentId="13216" CreationDate="2017-03-27T12:57:16.737" Score="42" Body="&lt;p&gt;Taken from this post:&lt;a href=&quot;https://stats.stackexchange.com/a/245452/154812&quot;&gt;https://stats.stackexchange.com/a/245452/154812&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;The issue&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;There are some issues with learning the word vectors using a &amp;quot;standard&amp;quot; neural network. In this way, the word vectors are learned while the network learns to predict the next word given a window of words (the input of the network).&lt;/p&gt;&#xA;&lt;p&gt;Predicting the next word is similar to predicting a class in a classification problem. That is, such a network is just a &amp;quot;standard&amp;quot; multinomial (multi-class) classifier. And this network must have as many output neurons as classes there are. When classes are actual words, the number of neurons is, well, huge.&lt;/p&gt;&#xA;&lt;p&gt;A &amp;quot;standard&amp;quot; neural network is usually trained with a cross-entropy cost function which requires the values of the output neurons to represent probabilities - which means that the output &amp;quot;scores&amp;quot; computed by the network for each class have to be normalized, converted into actual probabilities for each class. This normalization step is achieved by means of the softmax function. Softmax is very costly when applied to a huge output layer.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;The (a) solution&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;In order to deal with this issue, that is, the expensive computation of the softmax, Word2Vec uses a technique called noise-contrastive estimation. This technique was introduced by [A] (reformulated by [B]) then used in [C], [D], [E] to learn word embeddings from unlabelled natural language text.&lt;/p&gt;&#xA;&lt;p&gt;The basic idea is to convert a multinomial classification problem (as it is the problem of predicting the next word) to a binary classification problem. That is, instead of using softmax to estimate a true probability distribution of the output word, a binary logistic regression (binary classification) is used instead.&lt;/p&gt;&#xA;&lt;p&gt;For each training sample, the enhanced (optimized) classifier is fed a true pair (a center word and another word that appears in its context) and a number of k randomly corrupted pairs (consisting of the center word and a randomly chosen word from the vocabulary). By learning to distinguish the true pairs from corrupted ones, the classifier will ultimately learn the word vectors.&lt;/p&gt;&#xA;&lt;p&gt;This is important: instead of predicting the next word (the &amp;quot;standard&amp;quot; training technique), the optimized classifier simply predicts whether a pair of words is good or bad.&lt;/p&gt;&#xA;&lt;p&gt;Word2Vec slightly customizes the process and calls it negative sampling. In Word2Vec, the words for the negative samples (used for the corrupted pairs) are drawn from a specially designed distribution, which favours less frequent words to be drawn more often.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;[A] &lt;a href=&quot;http://www.cs.cmu.edu/%7Enasmith/papers/smith+eisner.acl05.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2005) - Contrastive estimation: Training log-linear models on unlabeled data&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[B] &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v9/gutmann10a/gutmann10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2010) - Noise-contrastive estimation: A new estimation principle for unnormalized statistical models&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[C] &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1390177&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2008) - A unified architecture for natural language processing: Deep neural networks with multitask learning&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;[D] &lt;a href=&quot;https://www.cs.toronto.edu/%7Eamnih/papers/ncelm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2012) - A fast and simple algorithm for training neural probabilistic language models&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;[E] &lt;a href=&quot;https://papers.nips.cc/paper/5165-learning-word-embeddings-efficiently-with-noise-contrastive-estimation.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2013) - Learning word embeddings efficiently with noise-contrastive estimation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30425" LastEditorUserId="63078" LastEditDate="2023-12-10T14:57:24.837" LastActivityDate="2023-12-10T14:57:24.837" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="17906" PostTypeId="2" ParentId="17896" CreationDate="2017-03-27T13:27:59.713" Score="1" Body="&lt;p&gt;Yes you can use a decision tree for example. These trees can have a stop criterion. For example if the leaf is too small in sample size or, and this is interesting for you, is above a pre-defined purity threshold. By discarding all the final leafs that were created due to not purity related conditions you should end up with a set of rules like you described. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-03-27T13:27:59.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17910" PostTypeId="1" AcceptedAnswerId="18111" CreationDate="2017-03-27T18:48:43.817" Score="7" ViewCount="9637" Body="&lt;p&gt;I am trying to fit a logistic regression model to an imbalanced dataset (0.5/99.5) with high dimensionality(about 15k). I used random forest to select top 200 important features. Observations are around 120K.&lt;/p&gt;&#xA;&lt;p&gt;When I fit a logistic regression model on based dataset (using Smote for over sampling) , on training f1, recall and precision are good. But on testing, precision score and f1 are bad. I assume it makes sense because in training there were a lot more of the minority case while in reality/testing there is only very small percentage. So the algorithm is still looking for more minority cases, which caused the high false positive.&lt;/p&gt;&#xA;&lt;p&gt;I was wondering what kind of methods could I try to improve the performance?&lt;/p&gt;&#xA;&lt;p&gt;I am currently trying different sampling method for imbalanced dataset, also plan to try PCA.&lt;/p&gt;&#xA;" OwnerUserId="30376" LastEditorUserId="85045" LastEditDate="2021-02-09T17:00:55.380" LastActivityDate="2021-02-09T17:00:55.380" Title="Bad classification performance of logistic regression on imbalanced data in testing as compared to training" Tags="|classification|logistic-regression|class-imbalance|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="17911" PostTypeId="1" AcceptedAnswerId="17919" CreationDate="2017-03-27T19:53:17.807" Score="4" ViewCount="3614" Body="&lt;p&gt;I am creating a Text to Speech system for a phonetic language called &quot;Kannada&quot; and I plan to train it with a Neural Network. The input is a word/phrase while the output is the corresponding audio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While implementing the Network, I was thinking the input should be the segmented characters of the word/phrase as the output pronunciation only depends on the characters that make up the word, unlike English where we have slient words and Part of Speech to consider. However, I do not know how I should train the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my Dataset is a collection of words/phrases and the corrusponding MP3 files, I thought of converting these files to WAV using pydub for all audio files.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pydub import AudioSegment&#xA;sound = AudioSegment.from_mp3(&quot;audio/file1.mp3&quot;)&#xA;sound.export(&quot;wav/file1.wav&quot;, format=&quot;wav&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, I open the wav file and convert it to a normalized byte array with values between 0 and 1. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import wave&#xA;&#xA;f = wave.open('wav/kn3.wav', 'rb')&#xA;frames = f.readframes(-1)&#xA;&#xA;#Array of integers of range [0,255]&#xA;data = np.fromstring(frames, dtype='uint8')&#xA;&#xA;#Normalized bytes of wav&#xA;arr  = np.array(data)/255&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How Should I train this?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here, I am not sure how to train this with the input text. From this, I would need a variable number of input and output neurons in the First and Last layers as the number of characters (1st layer) and the bytes of the corresponding wave (Last layer) change for every input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since RNNs deal with such variable data, I thought it would come in handy here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correct me if I am wrong, but the output of Neural Networks are actually probability values between 0 and 1. However, we are not dealing with a classification problem. The audio can be anything, right? In my case, the &quot;output&quot; should be a vector of bytes corresponding to the WAV file. So there will be around 40,000 of these with values between 0 and 255 (without the normalization step) for every word. How do I train this speech data? Any suggestions are appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 1 :&lt;/strong&gt; In response to &lt;strong&gt;arduinolover's&lt;/strong&gt; Answer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand, Phonemes are the basic sounds of the language.  So, why do I need a neural network to map phoneme labels with speech? Can't I just say, &quot;whenever you see this alphabet, pronounce it like &lt;em&gt;this&lt;/em&gt;&quot;. After all, this language, Kannada, is phonetic: There are no silent words. All words are pronounced the same way they are spelled. How would a Neural Network help here then? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On input of a new text, I just need to break it down to the corresponding alphabets (which are also the phonemes) and retrieve it's file (converted from WAV to raw byte data).  Now, merge the bytes together and convert it to a wav file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this this too simplistic? Am I missing something here? What would be the point of a Neural Network for this particular language (Kannada) ? &lt;/p&gt;&#xA;" OwnerUserId="30435" LastEditorUserId="29788" LastEditDate="2017-04-29T05:14:47.893" LastActivityDate="2018-01-12T12:26:33.873" Title="python - What is the format of the WAV file for a Text to Speech Neural Network?" Tags="|python|neural-network|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17912" PostTypeId="2" ParentId="17910" CreationDate="2017-03-27T20:28:31.750" Score="2" Body="&lt;p&gt;The dimensionality of your data is an important consideration here. Having 15K features will likely lead to very poor results. The higher dimensionality your features the more training examples you will need. For a shallow method such as logistic regression a general rule of thumb is to use $10\times \#features$. So unless you have over 150K examples, using 15K features is not recommended. Think to yourself what kinds of questions need to be answered in your data and how you can remodel your data to better answer those questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furhtermore, logistic regression is not recommended for skewed datasets. There are many algorithms that are well suited to dealing with skewed dataset types of problems. Specifically, anomaly detection algorithms are capable of learning the distribution of a single set of labels (event not occurring) and then it will be able to flag when an anomaly occurs (event occurs). This is when an instance is sufficiently beyond the learned distribution. You can use this to get the probability of an event occurring based on a p-statistic test using the feature-space you have set up in contrast with those from your learned distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest method would be doing a generalized likelihood ratio test (GLRT). But, I think you will most likely find more luck using a K-NN based method for skewed datasets.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-27T20:28:31.750" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17913" PostTypeId="2" ParentId="16640" CreationDate="2017-03-27T21:47:38.473" Score="4" Body="&lt;p&gt;I think you should be careful as to which algorithms you tend to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Extraction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. If you have a vocabulary size of 100,000 then you really need to use some extra feature extraction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, you have 8000 instances!!! Yes. But, you also have 100,000 features for each instance. This is insufficient even for shallow machine learning models. For a neural network, i usually suggest to follow this very general rule of thumb,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\#examples = 100 * \#features$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you will need a MASSIVE amount of data to properly train your neural network model. Moreover, if you have a skewed dataset then you should expect to be using even more training examples, to show sufficient examples such that the model is capable of distinguishing the two classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest a less intensive model. You should try to use: naive bayes, kernel-SVM or knn for text classification. These methods would do MUCH MUCH better than a neural network. Especially considering you have a skewed dataset!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Suggestion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would start with bag of words and then use kernel-SVM. This should be a good starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recurrent neural network is 0% recommended for the amount of data you have.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-27T21:47:38.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17914" PostTypeId="1" CreationDate="2017-03-27T21:51:44.913" Score="1" ViewCount="189" Body="&lt;p&gt;I am getting into LSTM  and I have one technical question. I have time  series in 1 minute interval and 2 weeks data.  I need predict every next minute in the future. One sliding window has 120 minutes length. My question is, if &quot; the past data outside  of one sliding window&quot; has an influence on training and testing in current window of a LSTM ?  Different way  - could be  the sliding windows from the 2 weeks mixed randomly for training ( out of any order) ? Thank`s.&lt;/p&gt;&#xA;" OwnerUserId="30440" LastActivityDate="2017-03-27T21:51:44.913" Title="Has a past data outside sliding window any Influence on learning of a LSTM?" Tags="|machine-learning|deep-learning|time-series|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17915" PostTypeId="2" ParentId="17865" CreationDate="2017-03-27T23:39:17.413" Score="1" Body="&lt;p&gt;Answering the question JahKnows asked in the comment above will help with finding the suitable solution for your use cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not exactly sure what you are after, here are a few things you can do with a dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;One of the basic things you can do is to find customers that behave similarly. With the data that you have, you can consider the number of machines used by each customer in each month as features. In this case, each customer is described with 5 features. I would run a clustering algorithm against this data (of course leave CustId field out) and see how many types (clusters) of customers you have. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another thing to do is to run statistical analysis over the number of machines your customers use each month and find those that stand out from the rest of the population (customers). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another analysis you could do is to analyze the trend for each customer separately and based on their past behaviour, determine if the number of machines they have used in the past month was in accordance with their historical behaviour. In this case, you can take proper actions based on the shift of behaviour. E.g. a drop in the number of machines could mean the customer is having technical difficulties or churning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8149" LastActivityDate="2017-03-27T23:39:17.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17916" PostTypeId="1" AcceptedAnswerId="17922" CreationDate="2017-03-28T00:07:37.103" Score="3" ViewCount="1024" Body="&lt;p&gt;Here is a newbie question; when one trains an autoencoder or a variational autoencoder, does the order of the objects in the training vector $x$ matter?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I take an MNIST image image $(28\times28)$ and turn it into a feature vector of size $x \in \mathbb{R}^{1\times784}$. Then does it matter if I e.g. flatten the whole image vertically, or horizontally, or some other fancy way? Or if I were to scramble the order of the elements in the feature vector, would that make the VAE or AE mess up?&lt;/p&gt;&#xA;" OwnerUserId="30227" LastEditorUserId="30227" LastEditDate="2017-03-28T11:14:59.787" LastActivityDate="2019-04-25T02:08:44.033" Title="Ordered elements of feature vectors for autoencoders?" Tags="|machine-learning|deep-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17917" PostTypeId="1" AcceptedAnswerId="17918" CreationDate="2017-03-28T03:29:45.020" Score="3" ViewCount="128" Body="&lt;p&gt;Consider a software A/B test with the hypothesis that &quot;the addition of feature &lt;code&gt;F&lt;/code&gt; is predicted to increase metric &lt;code&gt;X&lt;/code&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the test, the data &lt;em&gt;doesn't&lt;/em&gt; show any significant change in &lt;code&gt;X&lt;/code&gt;, but it &lt;em&gt;does&lt;/em&gt; show a significant increase in &lt;code&gt;Y&lt;/code&gt; - something that wasn't expected or even considered at the beginning of the experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, is it scientifically valid to say that &lt;code&gt;F&lt;/code&gt; increases &lt;code&gt;Y&lt;/code&gt;, or should a new A/B test be designed and executed? &lt;/p&gt;&#xA;" OwnerUserId="30450" LastEditorUserId="13727" LastEditDate="2017-03-29T06:18:01.737" LastActivityDate="2017-03-29T06:18:01.737" Title="Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data" Tags="|ab-test|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17918" PostTypeId="2" ParentId="17917" CreationDate="2017-03-28T04:53:56.013" Score="5" Body="&lt;p&gt;It looks analagous to drug testing, where reporting of side effects during drug trials is obviously very important - i.e. the increase in Y seems analagous to a side effect. And some famous drugs have begun their lives as research into a side effect. Viagra is probably the most famous case, being a spinoff from a drug developed as angina medication. So in your write-up on your experiment you should definitely report the apparent effect on Y.&#xA;However, if the effect on Y is commercially important, then you still need to go back and do an experiment around a hypothesis that references the increase in Y to validate the existence of the effect properly.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2017-03-28T04:53:56.013" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17919" PostTypeId="2" ParentId="17911" CreationDate="2017-03-28T05:54:20.580" Score="4" Body="&lt;p&gt;Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a &lt;a href=&quot;http://www.speech.cs.cmu.edu/cgi-bin/cmudict?in=play&quot; rel=&quot;nofollow noreferrer&quot;&gt;phonetic dictionary&lt;/a&gt; for converting your text files into sequence of phonemes. e.g play -&gt; P L EY &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are many things that affect the pronunciation. Some of them are listed below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt; 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Pitch:&lt;/strong&gt; Female speakers usually have higher pitch than male speakers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Speaking rate:&lt;/strong&gt; Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;length_of_output_phoneme:&lt;/strong&gt; The length of wav file to generate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in the end input to your NN will look something like this&#xA;&lt;strong&gt;[left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme]&lt;/strong&gt; and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have time marked data. You can use Hidden Markov model &lt;a href=&quot;https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/TrajectoryModelling/HTS-Introduction.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;HMM&lt;/a&gt; for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some speech synthesis resources are listed below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;a href=&quot;http://www.festvox.org/bsv/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CMU festvox&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;a href=&quot;https://deepmind.com/blog/wavenet-generative-model-raw-audio/&quot; rel=&quot;nofollow noreferrer&quot;&gt;wavenet&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41539.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning in Speech Synthesis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The biggest challenge will be to make it sound like human voice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1: Some of the problems with &quot;whenever you see this alphabet, pronounce it like this&quot; are listed below &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) context: From the above to and go example which pronunciation of 'o' will be used&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) discontinuity: Vocal tract vibrations (lip, tongue motions) produce phoneme. Phonemes diffuse into neighboring phonemes since vocal tract doesn't stop vibrating immediately. If you copy-paste phonemes then there will be an abrupt change. You can copy-paste sounds for words since they are independent (kind of, 'can not' becomes 'can't' while speaking) and insert small pauses in between words. But then you have to store pronunciations for all the words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;strong&gt;&lt;a href=&quot;https://www.teachingenglish.org.uk/article/word-stress&quot; rel=&quot;nofollow noreferrer&quot;&gt;stress phoneme&lt;/a&gt;&lt;/strong&gt;: While speaking one or more phonemes are stressed (more focused, longer), Copy-pasting leaves out the stress information since everything is same&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds.&#xA;With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated. Go through the first and third resource for detailed explanations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these problem arise because we want to make it sound more human. Phoneme transition and diffusion are the major hurdles. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Even if you don't change any condition each time there will be a slight variation in pronunciation of same word. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.  &lt;/p&gt;&#xA;" OwnerUserId="29788" LastEditorUserId="29788" LastEditDate="2017-03-28T14:32:02.357" LastActivityDate="2017-03-28T14:32:02.357" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17920" PostTypeId="1" CreationDate="2017-03-28T07:30:36.537" Score="2" ViewCount="2078" Body="&lt;p&gt;I am using SOM (Self-Organizing Maps) of Kohonen, or more specifically, the MiniSom, &lt;a href=&quot;https://github.com/JustGlowing/minisom&quot; rel=&quot;nofollow noreferrer&quot;&gt;found here&lt;/a&gt; to cluster and visualize my data.&#xA;&lt;br&gt;&#xA;As you can see in the above site, the example given is:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from minisom import MiniSom    &#xA;som = MiniSom(6, 6, 4, sigma=0.3, learning_rate=0.5) # initialization of 6x6 SOM&#xA;print &amp;quot;Training...&amp;quot;&#xA;som.train_random(data, 100) # trains the SOM with 100 iterations&#xA;print &amp;quot;...ready!&amp;quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;the three parameters of &lt;strong&gt;sigma&lt;/strong&gt; and &lt;strong&gt;learning rate&lt;/strong&gt; and &lt;strong&gt;number of iterations&lt;/strong&gt; are hardcoded, and apparently it is working for the given examples. However, my data is bigger and different in size, so when I use these hardcoded parameters, the results is awful. Or even worse, when I change my corpus and implement it on another corpus size, the results get worse... however when I try to find the &amp;quot;best initial values&amp;quot; by trial and error, I get something that is meaningful!&lt;/p&gt;&#xA;&lt;h2&gt;Question:&lt;/h2&gt;&#xA;&lt;p&gt;How can I determine the initial values of these parameters? Is there any other SOM implementation that facilitates this?&lt;/p&gt;&#xA;&lt;p&gt;Your comments and answers are welcome. thanks.&lt;/p&gt;&#xA;&lt;h2&gt;annex:&lt;/h2&gt;&#xA;&lt;p&gt;There is an &lt;a href=&quot;https://www.academia.edu/30688343/Improving_Feature_Map_Quality_of_SOM_Based_on_Adjusting_the_Neighborhood_Function&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; on this, but apparently no right answer...&lt;/p&gt;&#xA;" OwnerUserId="25597" LastEditorUserId="98307" LastEditDate="2020-08-11T07:09:11.130" LastActivityDate="2020-08-11T07:09:11.130" Title="SOM initial values for learning rate and neighborhood sigma" Tags="|machine-learning|python|neural-network|clustering|unsupervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17921" PostTypeId="1" AcceptedAnswerId="20389" CreationDate="2017-03-28T08:12:11.110" Score="4" ViewCount="1749" Body="&lt;p&gt;I have been reading this book and I have no idea how to make an argument for this problem. &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Book&quot;&gt;http://neuralnetworksanddeeplearning.com/chap3.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Connecting regularization and the improved method of weight initialization L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) supposing $λ$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) provided $ηλ≪n$ the weights will decay by a factor of $e^{\frac{−ηλ}{m}}$ per epoch; and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) supposing λ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where n is the total number of weights in the network. Argue that these conditions are all satisfied in the examples graphed in this section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain a little bit more and guide my direction? Thank you so much for reading it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My argument is that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) I am thinking about this formula $w' = w(1-\frac{ηλ}{n}) - \frac{n}{m} \sum \frac{dCx}{dw}$ is related. Since the weight is larger without using new weight initialization (the standard deviation of $w = \frac{1}{\sqrt{total number of weights in the network}}$, $\frac{ηλw}{n}$ will be significantly larger than sum of $\frac{dCx}{dw}$ （the average rate of change of $C$ in respect of weight for a minibatch). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) By substituting $ηλ+b = n$ into $w'$?? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you again.&lt;/p&gt;&#xA;" OwnerUserId="30459" LastEditorUserId="38887" LastEditDate="2018-07-11T15:29:48.413" LastActivityDate="2024-01-25T22:30:51.180" Title="Heuristic argument for Weight decay and regularization" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17922" PostTypeId="2" ParentId="17916" CreationDate="2017-03-28T12:07:17.597" Score="2" Body="&lt;p&gt;For a fully-connected network the precise &lt;strong&gt;order of features does not matter initially&lt;/strong&gt; (i.e. before you start to train), as long as it is consistent for each example. This is independent of whether you have an auto-encoder to train or some other fully-connected network. Processing images with pixels as features does not change this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some caveats:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;To succeed in training, you will &lt;strong&gt;need the pixel order to be the same for each example&lt;/strong&gt;. So it can be randomly shuffled, but only if you keep the same shuffle for each and every example. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;As an aside, you will still get &lt;em&gt;some&lt;/em&gt; training effect from fully random shuffling the variables, because for example writing an &quot;8&quot; has more filled pixels than writing a &quot;1&quot; on average. But the performance will be very bad, accuracy only a little better than guessing, for most interesting problem domains.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To visualise what the auto-encoder has learned, your output needs to be unscrambled. You can actually input a (same shuffle each example) scrambled image and train the autoencoder to unscramble it - this will in theory get the same accuracy as training to match the scrambled input, showing again that pixel order is not important. You could also train autoencoder to match scrambled input to scrambled output and visualise it by reversing the scrambling effect (again this must be a consistent scramble, same for each example).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In a fully-connected neural network, there is nothing in the model that represents the local differences between pixels, or even that they are somehow related. So the network will learn relations (such as edges) irrespective of how the image is presented. But it will also suffer from being unable to generalise. E.g. just because an edge between pixels 3 and 4 is important, the network will not learn that the same edge between pixels 31 and 32 is similar, unless lots of examples of both occur in the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addressing poor generalisation due to loss of knowledge about locality in the model is one of the motivations for convolutional neural networks (CNNs). You can have &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN autoencoders&lt;/a&gt;, and for those, you intentionally preserve the 2D structure and local relationships between pixels - if you did not then the network would function very poorly or not at all.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2019-04-25T02:08:44.033" LastActivityDate="2019-04-25T02:08:44.033" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17923" PostTypeId="1" CreationDate="2017-03-28T12:49:46.673" Score="1" ViewCount="2760" Body="&lt;p&gt;I have been trying to plot my data from a cvs file with &lt;a href=&quot;http://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bokeh&lt;/a&gt; but I couldn't find a smoothing option to smooth my graph curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any in &lt;code&gt;bokeh.plotting&lt;/code&gt; for this task? If not, how else should I proceed? &lt;/p&gt;&#xA;" OwnerUserId="29771" LastEditorUserId="31513" LastEditDate="2017-08-09T07:08:22.487" LastActivityDate="2017-08-09T07:08:22.487" Title="Smooth curve in Bokeh" Tags="|python|visualization|plotting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17925" PostTypeId="1" CreationDate="2017-03-28T13:16:57.263" Score="-1" ViewCount="187" Body="&lt;p&gt;I try to fit a data matrix X to an output vector y with a regression model in sklearn. I have some training data and some test data, where the score is the RMSE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my best score I achieved with SVR, kernel 'poly' and tuning the hyperparameters 'C', 'degree' and 'gamma' with optunity and cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually don't know how to achieve better scores so I ask here in this forum for another approach. I tried already KernelRidge, Linear Regression, SVR with other kernels, Neural Networks but all of them gave worse results. It is actually possible to do better, since other people do better in this task, but I have no more Idea what I can do to imporve the score. Any Ideas?&lt;/p&gt;&#xA;" OwnerUserId="30473" LastEditorUserId="30473" LastEditDate="2017-03-28T15:23:23.653" LastActivityDate="2017-08-25T19:29:45.847" Title="Sklearn regression problem" Tags="|scikit-learn|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="17926" PostTypeId="2" ParentId="11912" CreationDate="2017-03-28T13:18:49.147" Score="3" Body="&lt;p&gt;&lt;strong&gt;Method of regularization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the following 4 techniques, L1 Regularization and L2 Regularization are needless to say that they must be a method of regularization. They shrink the weight. L1 would concentrate on shrinking a smaller amount of weight if the weights have higher importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dropout prevents overfitting by temporarily dropping out neurons. Eventually, it calculates all weights as an average so that the weight won't be too large for a particular neuron and hence it is a method of regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Batch Normalization should not be a method of regularization because the main purpose of it is to speed up the training by selecting a batch and forcing the weight to be distributed near 0, not too large, not too small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choosing it&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, mini-batch is a must because it can speed up the process and improve the performance of network every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L1 and L2 are both similar and I would prefer L1 in small network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, dropout should apply if there is a large variation problem or overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not the least, I agree with Neil Slater that it depends on the situation and there will never be an optimum solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend you to read this for further information. This is a very good material. &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap3.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30459" LastEditorUserId="30459" LastEditDate="2017-10-25T11:09:37.303" LastActivityDate="2017-10-25T11:09:37.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17927" PostTypeId="1" CreationDate="2017-03-28T13:35:46.103" Score="1" ViewCount="15804" Body="&lt;p&gt;I need to perform clustering in a given dataset.&#xA;There are atrributes with numerical as well as categorical values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to convert categorical to numeric value ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as an example one field is colour and the values are red, green, blue&#xA;so can I assign a  mapping like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;red : 1, green : 2, blue : 3 or&#xA;red : 11, green : 25, blue : 30 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and if I provide a mapping like this will this affect the Euclidian distance for clustering ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or is there any other way ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-11-09T13:23:05.367" Title="Categorical data in Kmeans" Tags="|clustering|k-means|" AnswerCount="4" CommentCount="0" ClosedDate="2017-11-10T16:34:07.897" ContentLicense="CC BY-SA 3.0" />
  <row Id="17929" PostTypeId="2" ParentId="17927" CreationDate="2017-03-28T14:07:26.193" Score="-1" Body="&lt;p&gt;Try to categorize them as efficiently based on what you care about. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to stay true to the color wheel than just use the RGB values for the colors you are using. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if not then you can use any mapping, but be careful because it will affect your Euclidean distance even after you normalize the feature (which is highly recommended for K-means).&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-28T14:07:26.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17930" PostTypeId="2" ParentId="17927" CreationDate="2017-03-28T14:10:09.123" Score="0" Body="&lt;p&gt;You can't really use k means with categorical/nominal data because it's not ranked. In other words, you can't say &quot;green is greater than red&quot; or &quot;green is less than blue.&quot; Even if you did want to assign rankings to them (treating them as ordinal), you can't say what the difference between them is. I would recommend using hierarchical clustering or recursive partitioning instead.&lt;/p&gt;&#xA;" OwnerUserId="29353" LastActivityDate="2017-03-28T14:10:09.123" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17931" PostTypeId="1" AcceptedAnswerId="17933" CreationDate="2017-03-28T14:24:29.293" Score="0" ViewCount="4647" Body="&lt;p&gt;I would like to predict the date a item will be sold using features such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Product ID          Price        Days_since_first_post       Last_repost     Type       First_Post_Date         Sold_Date&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would machine learning principles be used in such a way to do this task? If so, what is the proper means by which to design such an algorithm? What would be the different steps which would be involved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, what are the logical steps that should be taken when tackling a machine learning problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="30478" LastEditorUserId="29587" LastEditDate="2017-03-28T16:39:46.197" LastActivityDate="2017-03-28T16:39:46.197" Title="Predict the date an item will be sold using machine learning" Tags="|machine-learning|classification|regression|scikit-learn|supervised-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-28T16:39:14.877" ContentLicense="CC BY-SA 3.0" />
  <row Id="17932" PostTypeId="2" ParentId="17927" CreationDate="2017-03-28T14:50:36.457" Score="5" Body="&lt;p&gt;Converting the categorical data into numerical data isn't really meaningful. Different mappings will give you different solutions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an extension of K-means algorithm for categorical data called k-modes. You can read about K-modes in detail &lt;a href=&quot;http://www.cs.ust.hk/~qyang/Teaching/537/Papers/huang98extensions.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. This &lt;a href=&quot;https://shapeofdata.wordpress.com/2014/03/04/k-modes/&quot; rel=&quot;noreferrer&quot;&gt;article&lt;/a&gt; explains the difference between K-modes and converting data into numeric vectors and then running K-means. &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-03-28T14:50:36.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17933" PostTypeId="2" ParentId="17931" CreationDate="2017-03-28T15:11:22.513" Score="2" Body="&lt;p&gt;A machine learning problem can be separated into a few modular parts. Of course these are all massive in reach and possibility. However, every problem you encounter should be thought of in this way at first. You can then skip things where you feel necessary. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data pre-processing and feature extraction&lt;/li&gt;&#xA;&lt;li&gt;The model&lt;/li&gt;&#xA;&lt;li&gt;Post-processing&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;Data pre-processing and feature extraction&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This and feature extraction are the two most important parts of a machine learning technique. That's right, NOT THE MODEL. If you have good features then even a very simple model will get amazing results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. If you are using text data you should build word vectors. There are countless ways pre-processing and feature extraction can be implemented. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In your case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, let's consider the data pre-processing. You notice that type might not be an integer value. This may cause problems when using most machine learning algorithms. You will want to bin these different types and map them onto numbers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature selection: besides using the techniques I outlined above, you should also notice that productID is a useless feature. It should for sure NOT be included in your model. It will just confuse the model and sway it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule of thumb, the amount of data that is suggested to have for shallow machine learning models is $10 \times \#features$. So you are limited by the size of your dataset. Also, make sure the outputs are quite well distributed. If you have some skew in your dataset, like a lot of examples where the item was sold right away, then the model will learn this tendency. You do not want this. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The Model&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Now that you have your feature-space, which might be entirely different from the original columns you posted, it is time to choose a model. You are trying to estimate the time of sale for an algorithm. Thus, this can be done in two different ways. Either as a classifier problem or as a regression problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classifier problem would separate the different times of sales into distinct bins. For example&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class 1: [0 - 5] days&lt;/li&gt;&#xA;&lt;li&gt;Class 2: [5 - 10] days&lt;/li&gt;&#xA;&lt;li&gt;etc...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course the more classes you will choose to have then the harder it will be to train the model. That means the resolution of your results is limited by the amount of the data you have available to you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other option is to use a regression algorithm. This will learn the tendency of your curve in higher dimensional space and then estimates where along that line a new example would fall. Think of it in 1-dimension. I give you a bunch of heights $x$ and running speeds $y$. The model will learn a function $y(x)$. Then if I give you just a height, you will be able to estimate the running speed of the individual. You will be doing the same thing but with more variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are really a ton of methods that can do this. You can look through some literature reviews on the subject to get a hold of all of them. But, I warn you there is A LOT. I usually start with kernel-Support Vector Regression (k-SVR). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test your model. Separate your dataset into three parts (training, validating, testing) if you have sufficient data. Otherwise two parts (training, testing) is also fine. Only, train your model on the training set and then evaluate it using the example it has not seen yet which are reserved in the testing set. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Post-processing&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the step where you can further model your output $y(x)$. In your case that might not be needed. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-28T15:11:22.513" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17934" PostTypeId="1" AcceptedAnswerId="17948" CreationDate="2017-03-28T15:28:19.837" Score="1" ViewCount="797" Body="&lt;p&gt;i have read a few  blogs and papers on the IMDB exercise w.r.t sentiment classification using LSTM's (and at times in conjunction with CNN) but there the output layer can contain just 1 neuron with a sigmoid since the sentiment can either be good or bad. But if i need to use the same technique to classify, say, 30,000 sentences into 20 different labels then what should my output layer look like ? if i am not mistaken i should have the same number of neurons in my output layer as the number of labels i am training the data for (20 in this case) each of which has the same sigmoid. Can you please let me know if this makes sense ?&lt;/p&gt;&#xA;" OwnerUserId="29157" LastActivityDate="2017-03-29T01:45:25.847" Title="sentence classification with RNN-LSTM - output layer" Tags="|deep-learning|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17938" PostTypeId="2" ParentId="17925" CreationDate="2017-03-28T16:51:35.713" Score="0" Body="&lt;p&gt;Data pre-processing and feature extraction are the two most important parts of a machine learning technique. That's right, NOT THE MODEL. If you have good features then even a very simple model will get amazing results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. If you are using text data you should build word vectors. There are countless ways pre-processing and feature extraction can be implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-28T16:51:35.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17940" PostTypeId="2" ParentId="17923" CreationDate="2017-03-28T17:28:57.173" Score="3" Body="&lt;p&gt;&lt;code&gt;bokeh&lt;/code&gt; library internally uses &lt;code&gt;_glyph_function&lt;/code&gt; function to plot, if you take a look at their &lt;a href=&quot;https://github.com/bokeh/bokeh/blob/master/bokeh/plotting/helpers.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt; and which takes help from basic numpy, scipy library for defining arrays and other stuff and this so goes for curve smoothing too. So better do achieve this via &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/signal.html#filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-03-28T17:28:57.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17941" PostTypeId="1" AcceptedAnswerId="17949" CreationDate="2017-03-28T17:31:48.233" Score="2" ViewCount="2234" Body="&lt;p&gt;I am currently studying this &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; and are trying to understand what exactly the input and output shape is.  The paper describes an acoustic model consisting of using cnn-hmm as the acoustic model.  The input is a image of mel-log filter energies visualised as spectograms. The paper describes a method  for phone recognition in which (as far I understand) applying a CNN on these spectograms with a limited weight sharing scheme should be beneficial for phone recognition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input shape as far i understand, is 9-15 frames, which seem a bit confusing, as they don't consider number of phonemes a utterance may have, or the length of them, but simply just &quot;choose&quot; a number of frames to operate with.. The number doesn't seem to be connected with the output in any way - or am I misinterpreting something? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the output&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We used 183 target class labels, i.e., 3 states for each HMM of 61 phones. After decoding, the original 61 phone classes were mapped to a set of 39 classes as in [47] for final scoring. In our experiments, a bigram language model over phones, estimated from the training set, was used in decoding. To prepare the ANN targets, a mono-phone HMM model was trained on the training data set, and it was used to generate state-level labels based on forced alignment. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So the output is divided in to 183 classes, being mapped into HMM's with 3 states for each 61 phonemes, and the ANN target (As I see it target = posterior probability) by training a monophone hmm with forced alignment. I am not sure I understand this process.  If the ANN targets are those the CNN should aim/regress to and at the end classify the state based on, Why then process the input?.. why not make a simple DNN that does the regression/classification? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like the improvement lies in the use of forced alignment here, and only on monophone? where is the improvement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And again how I am supposed to link the input shape and the output shape based this? This would require the audio files to have a certain length, the length of the audio is never specified, so I am assuming that this is not the case. &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:44:30.600" LastActivityDate="2021-02-16T02:44:30.600" Title="Input and output feature shapes in CNN for speech recognition" Tags="|neural-network|classification|regression|convolutional-neural-network|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17942" PostTypeId="1" CreationDate="2017-03-28T18:09:44.277" Score="0" ViewCount="405" Body="&lt;p&gt;I have dataset of 50000 values (rows) and 1000 variables (columns). Since this is high dimensional, I am unable to work with just DBSCAN.&#xA;So I am trying to use PCA (principle component analysis). Since PCA reduces dimension to few 100's. &#xA;I request you to help you to help me with &#xA;EITHER&#xA; 1. Other high dimensional clustering algorithm &#xA; 2. How to interpret results of PCA&lt;/p&gt;&#xA;" OwnerUserId="27469" LastEditorUserId="924" LastEditDate="2017-03-28T19:37:19.263" LastActivityDate="2017-03-28T20:52:36.567" Title="How to Interpret the output of PCA?" Tags="|clustering|dimensionality-reduction|pca|dbscan|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17944" PostTypeId="2" ParentId="17942" CreationDate="2017-03-28T18:27:18.747" Score="1" Body="&lt;p&gt;In short PCA, returns an orthogonal set of basis features that best represent the variance in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, imagine you want to identify whether we are talking about a dog or a cat. Your features are: size, weight, color, fur type, etc... but you also have features like weather, owner name, etc... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be evident that the first set of features obviously explains the variance between a dog and a cat much better than the second set. Thus, you should only consider those and completely disregard the second set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what PCA does however, it adds the extra restriction that all features must be orthogonal. It then assigns a metric to each component based on the amount of variance that feature explains. You can then sort the components and extract the top $n$ components (feature reduction). &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-03-28T20:52:36.567" LastActivityDate="2017-03-28T20:52:36.567" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17945" PostTypeId="1" CreationDate="2017-03-28T19:20:11.050" Score="3" ViewCount="1881" Body="&lt;p&gt;I have a binary classification task for time series data.&#xA;Every 14 rows in my CSV is relevant to one time slot. How should I prepare this data to be used in LSTM? In other word how to feed the model with this data?&lt;/p&gt;&#xA;" OwnerUserId="15064" LastActivityDate="2023-12-03T10:01:05.187" Title="How to prepare data for LSTM time series prediction" Tags="|python|learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17947" PostTypeId="1" CreationDate="2017-03-28T21:04:07.547" Score="3" ViewCount="82" Body="&lt;p&gt;Ideally, I would like to add demographic features to my data.  I have zip codes for each observation, so I was hoping to add in the demographic data by zip code.  Unfortunately, I cannot find any demographic data by zip code.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any suggestions on using the zip code tabulation area data for zip codes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="30502" LastActivityDate="2017-03-30T01:37:03.683" Title="I want to add demographic data to a data set. Any suggestions on where to find zip code level data?" Tags="|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17948" PostTypeId="2" ParentId="17934" CreationDate="2017-03-29T01:45:25.847" Score="2" Body="&lt;p&gt;When doing a multiclass classification problem, in which the goal is predict exactly one class label for each input, it is standard to use the softmax function (a normalized exponential) as the activation function for the last layer. However, if your problem is a multilabel classification (in which the classes are not mutually exclusive), then using a sigmoid as the activation function would be appropriate. In both situations, you would have as many output units as labels (20 in your case).&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-03-29T01:45:25.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17949" PostTypeId="2" ParentId="17941" CreationDate="2017-03-29T04:04:46.943" Score="2" Body="&lt;p&gt;DNN/CNN prediction(training) is done for 1 frame at a time. The output can be any of the 183 outputs states. Length of the audio files is not a problem since the input to the DNN/CNN is of same dimension only the number of inputs change with audio length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g 1.wav has 500 features and each feature is 39 dimensional and 2.wav has 300 features, &#xA;the trained model will take a 39 dimensional input and output will be 183 dimensional. So depending on length we'll get different number of outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since all the frames in an utterances are tested against all the 183 possibilities so the output always remains 183 dimensional. There is no need to specify number of phonemes an utterance can have as everything is being done at frame level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frame concatenation (9-15 frames) is done to leverage contextual properties of speech data. Phone changes are context dependent. For 15 frame context, we change the input of DNN to [7*39 (left_context) 39 7*39(right_context)], a 585 dimensional vector. So now DNN will take 585 dimensional data as input and will output a 183 dimensional vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN input&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There exist several different alternatives to organizing these&#xA;  MFSC features into maps for the CNN. First, as shown in&#xA;  Fig. 1(b), they can be arranged as three 2-D feature maps,&#xA;  each of which represents MFSC f&#xA;  eatures (static, delta and&#xA;  delta-delta) distributed along both frequency (using the fre-&#xA;  quency band index) and time (using the frame number within&#xA;  each context window). In this case, a two-dimensional con-&#xA;  volution is performed (explained below) to normalize both&#xA;  frequency and temporal variations simultaneously. Alterna-&#xA;  tively, we may only consider normalizing frequency variations.&#xA;  In this case, the same MFSC features are organized as a number&#xA;  of one-dimensional (1-D) feature maps (along the frequency&#xA;  band index), as shown in Fig. 1(c). For example, if the context&#xA;  window contains 15 frames and 40&#xA;  fi&#xA;  lter banks are used for each&#xA;  frame, we will construct 45 (i.e., 15 times 3) 1-D feature maps,&#xA;  with each map having 40 dimensions, as shown in Fig. 1(c).&#xA;  As a result, a one-dimensional convolution will be applied&#xA;  along the frequency axis. In this paper, we will only focus on&#xA;  this latter arrangement found in Fig. 1(c), a one-dimensional&#xA;  convolution along frequency&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So the input to the CNN will be an image patch of size 45 * 40 regardless of the length of the audio file, just the number of such inputs will depend on the length of audio file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why force alignment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we are doing everything at frame level so for each frame we need the state labels. Usually this timing information is not available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transcribed data usually looks like this, &#xA;1.wav -&gt; I am a cat&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I don't know how many frames belong to I or to a.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HMMs are trained on this data and force alignment is done to generate state level labels for each frame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Better modeling of input-output relation gives improvement over other methods. Filter bank features also contribute to improvements. A DNN trained with filter bank features gives better performance as compared a DNN trained with MFCC features. &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-03-29T04:04:46.943" CommentCount="18" ContentLicense="CC BY-SA 3.0" />
  <row Id="17951" PostTypeId="2" ParentId="17917" CreationDate="2017-03-29T06:16:16.740" Score="2" Body="&lt;p&gt;The problem is moving from estimating a single hypothesis into few ones.&#xA;One could claim that X and Y are symmetric, if we were willing to examine X, why shouldn't we examine Y?&#xA;The difference is that since Y wasn't part of the original plan, it is possible that there are many other variables there Y1, Y2, Yn...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider that we have extra n variables, all purely random. If we have a large enough n, one of them will have observations that seem to be correlated to F.&#xA;In case that you consider a pair of variables, the number of options you have becomes O(n^2). &#xA;The more complex hypothesis set you'll have, the more options you will have and more likely you will be to gat a false correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn't mean that you should ignore the result regarding Y. Many discoveries were accidentals. &#xA;As Robert de Graaf suggested, you can do another experiment and check the Y-F relation.&#xA;You can also check &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiple_comparisons_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;multiple hypothesis techniques&lt;/a&gt; in order to evaluate your current results in order to estimate whether the new relation is significant.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-03-29T06:16:16.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17953" PostTypeId="1" CreationDate="2017-03-29T08:34:43.283" Score="2" ViewCount="647" Body="&lt;p&gt;It seems to me that t-SNE and other dimensionality reduction algorithms which reduce the dimensionality to two dimensions are mainly used to get an impression of the dataset. If done well, they look nice (e.g. &lt;a href=&quot;https://blog.sourced.tech/post/lapjv/&quot; rel=&quot;nofollow noreferrer&quot;&gt;like this&lt;/a&gt;), but I wonder if this is better than just showing random images / grouping them by class on a grid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to get an answer to the following aspects:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How is t-SNE* better than just taking a random (probably stratified) sample of the data?&lt;/li&gt;&#xA;&lt;li&gt;How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?&lt;/li&gt;&#xA;&lt;li&gt;Does t-SNE* give any guarantees?&lt;/li&gt;&#xA;&lt;li&gt;Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?&lt;/li&gt;&#xA;&lt;li&gt;There are &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;many dimensionality reduction algorithms&lt;/a&gt;. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;*: You could probably answer this for other dimensionality reduction algorithms as well, but t-SNE seems to be the most popular one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note: I do see the advantage of a reduced dimensionality for compression / easier optimization / faster inference. However, the reduction to 2 dimensions seems to be only for visualization. Hence my question if one can see more in those embeddings than a visually pleasing image of the dataset.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-03-29T08:40:55.667" LastActivityDate="2017-03-29T13:53:15.477" Title="What are 2D dimensionality reduction algorithms good for?" Tags="|dimensionality-reduction|tsne|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17954" PostTypeId="2" ParentId="10228" CreationDate="2017-03-29T10:39:45.540" Score="10" Body="&lt;p&gt;To add upon the fact that there are more or less the same, consider also the fact that:&#xA;$$&#xA;\begin{split}&#xA;\forall \; 0 &amp;lt; u &amp;lt; 1,\; \log (1-u) &amp;amp;= -u - u^2/2 - u^3/3 \, + \, \cdots\\&#xA;\forall \; 0 &amp;lt; p &amp;lt; 1,\; \log (p) &amp;amp;= p-1 - (1-p)^2/2 - (1-p)^3/3 \, + \, \cdots\\&#xA;\end{split}&#xA;$$&#xA;so that:&#xA;$$&#xA;\forall \; 0 &amp;lt; p &amp;lt; 1,\; -p \log (p) = p(1-p) + p(1-p)^2/2 + p(1-p)^3/3 \, + \, \cdots&#xA;$$&#xA;See the following plot of the two functions normalised to get 1 as maximum value: red curve is for Gini while black one is for entropy.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WID8n.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WID8n.jpg&quot; alt=&quot;Normalised Gini and Entropy criteria&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end as explained by @NIMISHAN Gini is more suitable to minimise misclassfication as it is symetric to 0.5, while entropy will more penalised small probabilities.&lt;/p&gt;&#xA;" OwnerUserId="30527" LastActivityDate="2017-03-29T10:39:45.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17955" PostTypeId="2" ParentId="17755" CreationDate="2017-03-29T11:07:15.660" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;can I claim or should I expect to see that &#xA;  M1 will still perform better than M2 over that new test data set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes you should. Of course under the conditions that&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the test data comes from the same generating process as the training and validation data, and&lt;/li&gt;&#xA;&lt;li&gt;you have enough data in each set to make statistical fluctuations unlikely.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The model is getting better and better on the cross-validation score but &#xA;  when performed on an actual independent data set, its performance is &#xA;  getting worse and worse. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I can think of two reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The test data set is indeed not generated in the same way. Therefore, it is better to not rely on the Kaggle test set to which you do not have access. Use the data that you do have.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You are over fitting, which means that you are not executing the cross validation correctly. Make really sure that the training of parameters happens on the training data and, at the same time, that the validation happens on the data that you &lt;em&gt;did not use&lt;/em&gt; for the training. Compare the histograms of the training losses and the validation losses. The training losses should be consistently smaller than the validation losses. Do the same for the losses on the test data to get a consistent picture.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As and end note: It is to be expected, that the performance on the test set is lower than that on the validation set. This is because the model is chosen &lt;em&gt;based on&lt;/em&gt; the validation set. So it is biased to that data set.&lt;/p&gt;&#xA;" OwnerUserId="30520" LastActivityDate="2017-03-29T11:07:15.660" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="17956" PostTypeId="1" AcceptedAnswerId="17959" CreationDate="2017-03-21T04:14:25.857" Score="2" ViewCount="6320" Body="&lt;p&gt;I am using the Keras library (with Python 3.6) to create a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My network maintains a constant overall maximum accuracy of 62.5%, over 16 training samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In what ways can I increase this accuracy? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I increase the number of training samples, or will restricting some of the data in the training samples help? Or something else that I might not know of?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the layers of my neural network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Build neural network&#xA;# Neural net with multiple layers&#xA;model = Sequential()&#xA;&#xA;model.add(Dense(32, input_dim=17, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(32, init='uniform', activation='relu'))&#xA;model.add(Dense(16, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(4, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(1, init='uniform', activation='sigmoid'))&#xA;&#xA;# Compile model&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;# Fit model&#xA;history = model.fit(X, Y, validation_split=0.46, nb_epoch=150, batch_size=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32511" OwnerDisplayName="Larry" LastActivityDate="2019-04-20T14:33:38.303" Title="Why is my neural network not learning?" Tags="|keras|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17957" PostTypeId="2" ParentId="17956" CreationDate="2017-03-21T06:51:45.917" Score="1" Body="&lt;p&gt;First if your goal is to make a classification change the last sigmoid with softmax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are performing regression add some dropout layers, increase number of training examples, change the optimizer, try to change the ReLus with sigmoid or tanh, try a different normalization technique for your input data.&lt;/p&gt;&#xA;" OwnerUserId="30240" OwnerDisplayName="simonepi" LastActivityDate="2017-03-21T06:51:45.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17958" PostTypeId="2" ParentId="17956" CreationDate="2017-03-22T15:37:05.810" Score="0" Body="&lt;p&gt;You can make a table with variables inside which you can call for, due to that you can also remove a variable a replace it with a new one.&#xA;So your program will have an organized table system which the size can be infinite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your library it could be repeatedly calling: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(Dense(32, input_dim=17, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(64, init='uniform', activation='relu'))&#xA;model.add(Dense(32, init='uniform', activation='relu'))&#xA;model.add(Dense(16, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(4, init='uniform', activation='sigmoid'))&#xA;model.add(Dense(1, init='uniform', activation='sigmoid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;have you checked that? or the library might need more variables to run. Have you tried to make a variable per layer for backups?  &lt;/p&gt;&#xA;" OwnerDisplayName="joshua chris" LastEditorUserId="29575" LastEditDate="2019-04-20T14:33:38.303" LastActivityDate="2019-04-20T14:33:38.303" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="17959" PostTypeId="2" ParentId="17956" CreationDate="2017-03-22T15:47:24.213" Score="7" Body="&lt;p&gt;&lt;strong&gt;A neural network is the wrong approach for a problem with a small training set.&lt;/strong&gt; Even if you only have 2 features that are very representative of your function then 16 feature are not sufficient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a very general rule of thumb I use 100 examples for each feature in my dataset. This then increases exponentially with every single different class you expect. 16 instances is not enough to train a neural network. You will always have huge error margins when applying your model on a testing set. Even more problematic is the fact that you are using a very deep neural network. This will require even more training instances to properly learn the function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you use a general machine learning technique such as &lt;strong&gt;SVM&lt;/strong&gt;. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, be warned 16 training instances is still very little.&lt;/p&gt;&#xA;" OwnerUserId="29587" OwnerDisplayName="JahKnows" LastActivityDate="2017-03-22T15:47:24.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="17961" PostTypeId="1" CreationDate="2017-03-29T13:53:07.967" Score="0" ViewCount="85" Body="&lt;p&gt;I want to do my MSc Project on Data Mining. And I want to work on pollution in cities.&#xA;Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in specific city or area?&#xA;If not, do you have any advice on how to get or search for them?&#xA;Thank you..&lt;/p&gt;&#xA;" OwnerUserId="30532" LastActivityDate="2017-03-29T16:19:59.277" Title="City pollution data for data mining project" Tags="|data-mining|dataset|bigdata|" AnswerCount="1" CommentCount="0" ClosedDate="2017-03-30T13:42:51.510" ContentLicense="CC BY-SA 3.0" />
  <row Id="17962" PostTypeId="2" ParentId="17953" CreationDate="2017-03-29T13:53:15.477" Score="2" Body="&lt;p&gt;There are many questions there, I'll try to address each in turn.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How is t-SNE* better than just taking a random (probably stratified) sample of the data?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If your goal is to provide a visual overview of the data then clearly a stratified sample is not going to do that -- the samples each still live in high dimensional space, so are no easier to visualise, and if you are interested in how the data relate to each other then looking through a sample item by item does not necessarily help to build a reasonable mental model of those relationships. So sampling and visualisation with t-SNE (or similar approaches) are both attacking similar &lt;em&gt;underlying&lt;/em&gt; problems: &quot;What does my data look like&quot;; they provide very different views on that, highlighting different aspects however, and I would say are entirely complementary. Why not both!&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is harder, because I can't point you to explicit studies demonstrating otherwise, but t-SNE generally does a better job of providing a visual representation that is meaningful to users. It is worth noting that t-SNE was co-created by Geoff Hinton, who I am sure knows a great deal about neural networks and their potential uses and benefits. If Geoff felt t-SNE was worth using then one can reasonably presume it has some merits over other NN approaches. Now, with that in mind one can probably interpret t-SNE as a NN with 2-neuron bottleneck; I haven't tried to write that out in detail, so don't quote me on it. Finally, NNs aren't necessarily the answer to all problems; t-SNE is an algorithm for manifold learning specifically designed for low dimensional embedding, while an autoencoder with suitable bottleneck may happen to give similar results, but certainly isn't necessarily clearly suited to the task.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Does t-SNE* give any guarantees?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Guarantees of what? Providing a visual representation of the data? Yes it will do that. Guarantee that the Kullback-Liebler divergence of inferred distributions for the high dimensional and low dimensional representations has been minimised? Up to the quality of the optimisation, yes. Guarantee that local structure is preserved? Up to a suitable choice of perplexity for the dataset, yes. Guarantee that the low dimensional representation is a fully faithful representation of the global structure of the data? No, no guarantees there. It really depends on what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;4&quot;&gt;&#xA;&lt;li&gt;Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Not really -- not any more than what you potentially expect from any dimensionality reduction technique, and as you note forcing things to 2 dimensions constrains that a lot. In principle, since t-SNE focuses on preserving local structure and sacrifices global structure to do so, if your classification relies heavily on local structure then t-SNE could perform better than other dimension reduction techniques that seek to preserve global structure and thus have poorer local structure representation. In the case of unsupervised (density) clustering it is certainly true that local structure is more important than global structure so potentially t-SNE can help. However compression to 2 dimensions may be rather extreme.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;There are many dimensionality reduction algorithms. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If your interest is in visualisation (as is the case with t-SNE) then unfortunately comparisons are inevitably fairly subjective. This is much like the case of clustering. Certainly there are many different measures of clustering quality, but often that comes down to &quot;what do you mean by a cluster?&quot;, and for each measure there is a clustering algorithm that optimises for that measure. Likewise there are measures for how well an embedding has succeeded, but ultimately it depends on what you mean by a successful embedding (is preserving large scale relationships critical? is preserving local structure more important? How do you weigh one against the other?), and for any given measure there is an algorithm that optimises that measure. For a subjective evaluation the standard approach seems to be to embed labelled data and then view the result colored by label -- no, that is never going to clearly demonstrate superiority, but it can hint at what seems to be working. In practice t-SNE seems to fair well at this (hence its popularity). Are any of these better than bottlenecked autoencoders? I'm not sure why one assumes that a bottlenecked autoencoder is by default the superior option, but certainly in my experience whenever I've tried several options on data I've found t-SNE to provide better intuition about the dataset. Anecdotes are not data of course, so by all means try the options yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this covers most of it, although perhaps these were not quite the sort of answers you wanted.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-03-29T13:53:15.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17964" PostTypeId="1" CreationDate="2017-03-29T15:11:33.567" Score="1" ViewCount="895" Body="&lt;p&gt;How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-09-25T18:51:11.420" Title="Eigenvectors and eigenvalues for natural language processing" Tags="|nlp|linear-algebra|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17965" PostTypeId="2" ParentId="17964" CreationDate="2017-03-29T15:45:28.587" Score="2" Body="&lt;p&gt;Latent Semantic Analysis (LSA) relies on linear-algebraic decompositions (e.g. SVD), which in turn involve eigenvectors/values (see &lt;a href=&quot;https://www.quora.com/How-is-Principal-Component-Analysis-used-in-Natural-Language-Processing&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure if this is quite what your question is driving at, but in general, eigenvectors are useful in data analysis because they define some &quot;natural&quot; direction in the data. For example, the eigenvectors of the covariance matrix of some data are all perpendicular to one another, and the i'th one points in the direction of i'th greatest variance in the data. In other words, the first eigenvector points in the direction where the data's variance is greatest, the second one in (perpendicular) direction of second-greatest variance, and so on. &#xA;This is why eigenvector-based decompositions can be so useful in feature selection and dimensionality reduction. In informal terms, they transform the data into perpendicular features that are aligned with some &quot;natural&quot; axes in the data.&lt;/p&gt;&#xA;" OwnerUserId="23581" LastActivityDate="2017-03-29T15:45:28.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17967" PostTypeId="2" ParentId="17961" CreationDate="2017-03-29T16:19:59.277" Score="1" Body="&lt;p&gt;The second module of this free machine learning class revolves around working with geospatial pollution data for a river in the Netherlands:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edx.org/course/applied-machine-learning-microsoft-dat203-3x-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.edx.org/course/applied-machine-learning-microsoft-dat203-3x-1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-29T16:19:59.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17968" PostTypeId="1" CreationDate="2017-03-29T16:23:51.577" Score="0" ViewCount="63" Body="&lt;p&gt;This is a major confusion for me - I've always thought filters (those small sliding windows size 3x3 or 5x5) are strictly 2D, this among other is specified in Caffe API: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;kernel_size / kernel_h / kernel_w&lt;/strong&gt;. The filter dimensions, given by&#xA;  kernel_size for square filters or kernel_h and kernel_w for&#xA;  rectangular filters&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, I've never been 100% clear on what these convolutions mean (taken from Matt Zeiler's article (Visualizing and Understanding Convolutional Networks) and how they were obtained:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/awi7f.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/awi7f.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After searching it turned out that filters can be 3D?? How does it work then? I found a bit of information on CS 231n course, but it's not explained in detail anywhere. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2017-05-29T03:59:23.903" Title="Filter dimensionality in ConvNNs" Tags="|neural-network|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17969" PostTypeId="2" ParentId="17713" CreationDate="2017-03-29T16:30:15.713" Score="1" Body="&lt;p&gt;Deconvolution have very simple structure: unpooling &amp;rarr; deconv&#xA;like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Unpooling&#xA;Ps = (tf.gradients(pooled, h))[0]&#xA;unpooled = tf.multiply(Ps, P)&#xA;&#xA;# Deconv&#xA;batch_size = tf.shape(self.input_x)[0]&#xA;ds = [batch_size]&#xA;ds.append(self.embedded_chars_expanded.get_shape()[1])&#xA;ds.append(self.embedded_chars_expanded.get_shape()[2])&#xA;ds.append(self.embedded_chars_expanded.get_shape()[3])&#xA;deconv_shape = tf.stack(ds)&#xA;deconv = tf.nn.conv2d_transpose(&#xA;    unpooled,&#xA;    W,&#xA;    deconv_shape,&#xA;    strides=[1, 1, 1, 1],&#xA;    padding='VALID',&#xA;    name=&quot;Deconv&quot;&#xA;    )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30173" LastActivityDate="2017-03-29T16:30:15.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17970" PostTypeId="1" AcceptedAnswerId="17982" CreationDate="2017-03-29T17:09:02.750" Score="1" ViewCount="9910" Body="&lt;p&gt;I'm following &lt;em&gt;Introduction to Machine Learning with Python: A Guide for Data Scientists&lt;/em&gt; by Andreas C. Müller and Sarah Guido, and in Chapter 2 a demonstration of applying &lt;code&gt;LinearSVC()&lt;/code&gt; is given.  The result of classifying three blobs is shown in this screenshot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U23J4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U23J4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The three blobs are obviously correctly classified, as depicted by the colored output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is &lt;strong&gt;how are we supposed to know how to interpret the model fit output&lt;/strong&gt; in order to draw the three lines?  The output parameters are given by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(LinearSVC().fit(X,y).coef_)&#xA;[[-0.17492286  0.23139933]&#xA; [ 0.47621448 -0.06937432]&#xA; [-0.18914355 -0.20399596]]&#xA;&#xA;print(LinearSVC().fit(X,y).intercept_)&#xA;[-1.07745571  0.13140557 -0.08604799]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the authors walk us through how to draw the lines:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.svm import LinearSVC  &#xA;linear_svm = LinearSVC().fit(X,y)  &#xA;...&#xA;line = np.linspace(-15, 15)  &#xA;for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):  &#xA;    plt.plot(line, -(line * coef[0] + intercept) / coef[1])  #HOW DO WE KNOW&#xA;plt.ylim(-10, 15)  &#xA;plt.xlim(-10, 8)  &#xA;plt.show()  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The line of code with the comment is the one that converts our coefficients into a slope/intercept pair for the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = -(coef_0 / coef_1) x - intercept/coef_1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the term in front of &lt;code&gt;x&lt;/code&gt; is the slope and &lt;code&gt;-intercept/coef_1&lt;/code&gt; is the intercept.  In the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation on LinearSVC&lt;/a&gt;, the &lt;code&gt;coef_&lt;/code&gt; and &lt;code&gt;intercept_&lt;/code&gt; are just called &quot;attributes&quot; but don't point to any indicator that &lt;code&gt;coef_0&lt;/code&gt; is the slope and &lt;code&gt;coef_1&lt;/code&gt; is the negative of some overall scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I look up the interpretation of the output coefficients of this model and others similar to it in Scikit-learn without relying on examples in books and StackOverflow?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="21198" LastActivityDate="2020-04-02T19:51:13.033" Title="How can I know how to interpret the output coefficients (`coefs_`) from the model sklearn.svm.LinearSVC()?" Tags="|python|visualization|scikit-learn|svm|parameter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17971" PostTypeId="2" ParentId="17314" CreationDate="2017-03-29T17:33:29.067" Score="18" Body="&lt;p&gt;The dimensions of the low dimensional space have no meaning. Note that the t-SNE loss function is solely based on the distances between points ($y_i$ and $y_j$) and probability distributions over those distances ($p_{ij}$ and $q_{ij}$): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac{\delta C}{\delta y_i}=4 \sum_j(p_{ij} - q_{ij})(y_i-y_j)(1+||y_i -y_j||^2)^{-1}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus there is no projection from the whole high-dimensional space to the low-dimensional space, t-SNE only finds a mapping from a specific set of high-dimensional points to a specific set of low dimensional points. Because there is no function from one space to the other there is also no inherent meaning of the axes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things you can imagine to illustrate this: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rotating or translating the high-dimensional or low-dimensional space does not influences distances between the points. Hence, t-SNE does not care about rotation or translation in both spaces. Thus there is not absolute interpretation of the axes. &lt;/li&gt;&#xA;&lt;li&gt;The t-Student distribution has fat tails. This causes the low-dimensional representation to be invariant to changes in points that are far away in the high-dimensional space. This also causes that points that are far away in the high-dimensional space can be either reasonably far away, far away or really far away in the low dimensional space. In this sense it stretches certain parts of the low-dimensional axes (in any arbitrary direction). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That being said, t-SNE is primarily a visualization technique and its dimension reduction effectiveness for other purpose is not obvious (probably not suitable for clustering, feature extraction or feature selection). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also: &lt;a href=&quot;https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf&quot; rel=&quot;noreferrer&quot;&gt;the paper&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="14003" LastEditDate="2017-04-01T01:01:10.493" LastActivityDate="2017-04-01T01:01:10.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17972" PostTypeId="1" AcceptedAnswerId="18582" CreationDate="2017-03-29T17:51:32.480" Score="2" ViewCount="6710" Body="&lt;p&gt;Doing a logistic regression in R version 3.1.2 and I want to conduct a Hosmer Lemeshow test&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- glm(y ~ a + b + c + d, data = data, family = binomial)&#xA;&#xA;summary(model)&#xA;&#xA;Call:&#xA;glm(formula = y ~ a + b + c + d, family = binomial, &#xA;    data = data)&#xA;&#xA;Deviance Residuals: &#xA;    Min       1Q   Median       3Q      Max  &#xA;-2.5607  -0.7642  -0.4456   0.8998   2.3812  &#xA;&#xA;Coefficients:&#xA;             Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;(Intercept) -7.827805   1.169216  -6.695 2.16e-11&#xA;&#xA;a            0.582214   0.134207   4.338 1.44e-05&#xA;&#xA;b           -0.697126   0.462983  -1.506  0.13214   &#xA;&#xA;c            0.027485   0.009669   2.843  0.00448 &#xA;&#xA;d            0.996241   0.163006   6.112 9.86e-10&#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 504.53  on 373  degrees of freedom&#xA;Residual deviance: 381.91  on 369  degrees of freedom&#xA;  (3 observations deleted due to missingness)&#xA;AIC: 391.91&#xA;&#xA;Number of Fisher Scoring iterations: 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using the function hoslem.test from package &quot;Resource Selection&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hoslem.test(model$y, fitted(model), g = 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in model.frame.default(formula = cbind(y0 = 1 - y, y1 = y) ~ cutyhat) : &#xA;  variable lengths differ (found for 'cutyhat')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking at the code in the hoslem.test function, cutyhat is derived only from fitted(model) values, which I've verified contain no nas and are all integers, so why would they differ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pls help.&lt;/p&gt;&#xA;" OwnerUserId="30539" LastActivityDate="2021-07-08T18:42:10.233" Title="Hosmer Lemeshow Test, variable lengths differ" Tags="|r|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17973" PostTypeId="2" ParentId="17947" CreationDate="2017-03-29T18:17:05.493" Score="1" Body="&lt;p&gt;The US Census American Community Survey can provide a lot of demographics by different geographies, including zip. Check out:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://factfinder.census.gov/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://factfinder.census.gov/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-29T18:17:05.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17975" PostTypeId="2" ParentId="2466" CreationDate="2017-03-29T18:44:57.727" Score="1" Body="&lt;p&gt;Standard PCA only sees your data as real values and hence doesn't really provide the best low rank approximation. There exist versions of PCA designed for boolean data types. Personally I think the best paper covering the whole broad topic of generalised PCA is &lt;a href=&quot;https://web.stanford.edu/~boyd/papers/pdf/glrm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generalized Low Rank Models&lt;/a&gt;. In section 5.2 they specifically cover Boolean PCA.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-03-29T18:44:57.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17976" PostTypeId="2" ParentId="9731" CreationDate="2017-03-29T19:14:09.653" Score="1" Body="&lt;p&gt;In the simplest view, maybe this will suffice: the &lt;code&gt;backward()&lt;/code&gt; method is used for training a neural network with backpropagation; compute the output &lt;code&gt;y&lt;/code&gt; given input &lt;code&gt;x&lt;/code&gt; using your network's &lt;code&gt;forward()&lt;/code&gt; method, then find out the error of the output from your target using the &lt;code&gt;criterion&lt;/code&gt; you defined (e.g. negative log likelihood etc.). Now, if there were only one layer of network, then you could simply use those errors between the output layer and the target to update the weights in that single layer. When you have more than one layer (or more complex structure), you could update layers one at a time, each time computing the error in &lt;em&gt;that&lt;/em&gt; layer (no longer the output layer), then using that error to update weights for a previous layer. That is backpropogation. For this, you obviously need some way to map the error in the output $\Delta y$ onto $\Delta x$ using the same state (a.k.a. model state and input $x$). Thus, the &lt;code&gt;backward()&lt;/code&gt; method is essentially in the form:&#xA;$$f : (x, \Delta y) \rightarrow \Delta x$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the same of completeness, a &lt;code&gt;forward()&lt;/code&gt; method can be represented as:&#xA;$$f:x \rightarrow y$$&#xA;Alternatively, if the previous state of &lt;code&gt;y&lt;/code&gt; persists, then all you need to calculate is $\Delta y$, thus, equivalently speaking, a &lt;code&gt;forward()&lt;/code&gt; can also be represented as:&#xA;$$ f: (x, y) \rightarrow \Delta y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This form can be easily compared to the &lt;code&gt;backward()&lt;/code&gt; method, and it is easy to see why it is called as such.&lt;/p&gt;&#xA;" OwnerUserId="30542" LastEditorUserId="30542" LastEditDate="2017-03-30T21:07:03.073" LastActivityDate="2017-03-30T21:07:03.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17977" PostTypeId="1" CreationDate="2017-03-29T19:22:04.047" Score="5" ViewCount="1361" Body="&lt;p&gt;I know some techniques for augmenting data when images are used, but I don't know if there are any such techniques specifically catered to videos. Since videos have an additional temporal dimension compared to images I was wondering if there were any techniques that augments the video data using the view points and the motion information etc &lt;/p&gt;&#xA;" OwnerUserId="26050" LastActivityDate="2017-03-30T02:54:39.340" Title="Data Augmentation in videos" Tags="|deep-learning|data-augmentation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17978" PostTypeId="1" CreationDate="2017-03-29T20:26:29.340" Score="3" ViewCount="366" Body="&lt;p&gt;I’m looking for a large dataset of store receipt pictures or scans. The image quality is not of the highest priority. However, some annotations would be nice:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;vendor&lt;/li&gt;&#xA;&lt;li&gt;date of purchase&lt;/li&gt;&#xA;&lt;li&gt;total amount&lt;/li&gt;&#xA;&lt;li&gt;(optional: tax amount)&lt;/li&gt;&#xA;&lt;li&gt;(optional: currency)&lt;/li&gt;&#xA;&lt;li&gt;(optional: list of each individual item with associated amount, price, discounts)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30544" LastActivityDate="2017-03-29T20:26:29.340" Title="Where can I find a &gt;1K dataset of annotated store receipt pictures/scans?" Tags="|dataset|computer-vision|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17980" PostTypeId="1" AcceptedAnswerId="18023" CreationDate="2017-03-30T00:11:36.927" Score="-1" ViewCount="110" Body="&lt;p&gt;I am following a tutorial on R-Bloggers on an introduction to stock market data analysis. I got to this part -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if (!require(&quot;magrittr&quot;)) {&#xA;install.packages(&quot;magrittr&quot;)&#xA;library(magrittr)&#xA;}&#xA;&#xA;## Loading required package: magrittr&#xA;stock_return % t % &amp;gt; % as.xts&#xA;&#xA;head(stock_return)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, I keep getting this error - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error: unexpected '&amp;gt;' in &quot;stock_return % t % &amp;gt;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please how do I resolve this?&lt;/p&gt;&#xA;" OwnerUserId="6448" LastActivityDate="2017-03-31T07:59:41.020" Title="Error While Trying To Transform Data Into Stocks Object" Tags="|r|rstudio|" AnswerCount="1" CommentCount="0" ClosedDate="2017-04-06T07:55:32.847" ContentLicense="CC BY-SA 3.0" />
  <row Id="17981" PostTypeId="2" ParentId="17947" CreationDate="2017-03-30T01:37:03.683" Score="0" Body="&lt;p&gt;I'd second using &lt;a href=&quot;https://factfinder.census.gov/&quot; rel=&quot;nofollow noreferrer&quot;&gt;US Census Data&lt;/a&gt; as was already mentioned in another answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an additional source of government data with a bit more information about wages and income, you could also check out the &lt;a href=&quot;https://www.irs.gov/uac/soi-tax-stats-individual-income-tax-statistics-zip-code-data-soi&quot; rel=&quot;nofollow noreferrer&quot;&gt;aggregate ZIP code data provided by the IRS&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ZIP Code data show selected income and tax items classified by State, ZIP Code, and size of adjusted gross income. Data are based on individual income tax returns filed with the IRS and are available for Tax Years 1998, 2001, and 2004 through 2014. The data include items, such as:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Number of returns, which approximates the number of households&lt;/li&gt;&#xA;  &lt;li&gt;Number of personal exemptions, which approximates the population&lt;/li&gt;&#xA;  &lt;li&gt;Adjusted gross income &lt;/li&gt;&#xA;  &lt;li&gt;Wages and salaries&lt;/li&gt;&#xA;  &lt;li&gt;Dividends before exclusion&lt;/li&gt;&#xA;  &lt;li&gt;Interest received&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've used that to try to get at socioeconomic status in the past. It's helpful because you can, as one example, look at the percentage of income coming from interest and capital gains to get a sense for how wealthy and area is.&lt;/p&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-03-30T01:37:03.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17982" PostTypeId="2" ParentId="17970" CreationDate="2017-03-30T02:45:37.297" Score="2" Body="&lt;p&gt;Here's one (admittedly hard) way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to understand the low-level details, you can always work through the source code. For example, we can see that the &lt;code&gt;LinearSVC&lt;/code&gt; &lt;code&gt;fit&lt;/code&gt; method &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/classes.py#L210&quot; rel=&quot;nofollow noreferrer&quot;&gt;calls &lt;code&gt;_fit_liblinear&lt;/code&gt;&lt;/a&gt;. That &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/668a24c9e6e1a541bede64819612fdb5e169a341/sklearn/svm/base.py#L884&quot; rel=&quot;nofollow noreferrer&quot;&gt;calls &lt;code&gt;train_wrap&lt;/code&gt; in liblinear&lt;/a&gt;, which gets everything ready to &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/4d9a12d175a38f2bcb720389ad2213f71a3d7697/sklearn/svm/liblinear.pyx#L55&quot; rel=&quot;nofollow noreferrer&quot;&gt;call into the C++ function &lt;code&gt;train&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/4d9a12d175a38f2bcb720389ad2213f71a3d7697/sklearn/svm/src/liblinear/linear.cpp#L2364&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;train&lt;/code&gt; in linear.cpp&lt;/a&gt; is where the heavy lifting begins. Note that the &lt;code&gt;w&lt;/code&gt; member of the model struct in the train function gets mapped back to &lt;code&gt;coef_&lt;/code&gt; in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you understand exactly what the underlying &lt;code&gt;train&lt;/code&gt; function does, it should be clear exactly what &lt;code&gt;coef_&lt;/code&gt; means and why we draw the lines that way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this can be a little laborious, once you get used to doing things this way, you will really understand how everything works from top to bottom.&lt;/p&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-03-30T02:45:37.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17984" PostTypeId="2" ParentId="17977" CreationDate="2017-03-30T02:54:39.340" Score="3" Body="&lt;p&gt;You can augment videos in the temporal dimension through clipping, or taking random sequences of consecutive frames. You can also augment in the spatial dimension by cropping frames randomly to simulate camera jitter. Unfortunately, augmenting video data by extrapolating motion information or by producing a video taken from a different viewpoint seems extremely computationally intensive.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-03-30T02:54:39.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17985" PostTypeId="2" ParentId="17968" CreationDate="2017-03-30T03:07:59.500" Score="1" Body="&lt;p&gt;The diagram below Example 2 in this CS231 lecture is a good visualization of how filters work: &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/convolutional-networks/&lt;/a&gt;. Since each layer of a ConvNet has a depth corresponding to the number of filters (in an input RGB image, for example, the depth of the input layer would be 3), each filter of size $k\times k$ applied to a layer of delth $d$ slides over a volume of $k\times k\times d$ (note that the filter stays at a constant depth).&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-03-30T03:07:59.500" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17986" PostTypeId="1" CreationDate="2017-03-30T04:12:35.080" Score="3" ViewCount="634" Body="&lt;p&gt;How did we arrive at the sigmoid function for calculating probabilities?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not use some other function that &quot; squashes &quot; the values to lie between [0, 1]. Maybe even just normalise the values so they all add up to one. &lt;/p&gt;&#xA;" OwnerUserId="30562" LastActivityDate="2017-04-06T07:55:47.153" Title="Sigmoid function" Tags="|logistic-regression|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17987" PostTypeId="1" CreationDate="2017-03-30T04:40:43.763" Score="17" ViewCount="19053" Body="&lt;p&gt;I've read a couple of papers about kernel initialization and many papers mention that they use L2 regularization of the kernel (often with $\lambda = 0.0001$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anybody do something different than initializing the bias with constant zero and not regularizing it?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Kernel initialization papers&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mishkin and Matas: &lt;a href=&quot;https://arxiv.org/abs/1511.06422&quot; rel=&quot;noreferrer&quot;&gt;All you need is a good init&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Xavier Glorot and Yoshua Bengio: &lt;a href=&quot;http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;noreferrer&quot;&gt;Understanding the difficulty of training deep feedforward neural networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;He et al: &lt;a href=&quot;https://arxiv.org/abs/1502.01852&quot; rel=&quot;noreferrer&quot;&gt;Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-03-30T04:45:45.250" LastActivityDate="2020-08-05T13:58:16.093" Title="How should the bias be initialized and regularized?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17988" PostTypeId="1" CreationDate="2017-03-30T05:59:40.427" Score="0" ViewCount="213" Body="&lt;p&gt;I have a lot of .txt files. I want to train the model which remove bad words from the document(bad word - random sequence of characters). &#xA;For example: bad words - hsA , ubj8 ,t+h ,SSSh ,_^[ ,SVWh ,SSSSSS&#xA;normal words : *.exe , copy , write ,write ,ftp.exe&lt;/p&gt;&#xA;" OwnerUserId="30566" LastActivityDate="2017-03-30T21:46:44.583" Title="How to use NLP to determine the normal words in the text" Tags="|machine-learning|nlp|word2vec|" AnswerCount="2" CommentCount="1" ClosedDate="2017-04-01T14:17:56.653" ContentLicense="CC BY-SA 3.0" />
  <row Id="17989" PostTypeId="2" ParentId="17988" CreationDate="2017-03-30T06:15:11.413" Score="1" Body="&lt;p&gt;You can probably train a classification model to classify normal words from bad words, using character subsequences as features in the words. For example, &lt;code&gt;exe&lt;/code&gt; can be a feature for distinguishing a normal word from a bad word in your case.&lt;/p&gt;&#xA;" OwnerUserId="30567" LastActivityDate="2017-03-30T06:15:11.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17991" PostTypeId="1" AcceptedAnswerId="18047" CreationDate="2017-03-30T08:10:04.327" Score="1" ViewCount="393" Body="&lt;p&gt;I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so when I list them, the most similar are listed together in groups. That's how I would avoid having numerous (almost) same documents listed one, after another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking some kind of clustering would come handy. But the problem is, I don't know how many clusters I need. Also the number would have to be dynamic. And still, most of the documents wouldn't belong to a cluster, because they don't have any similar documents. So I would cluster just similar documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me to the direction that would help me solve this problem, or provide some examples of similar problems.&lt;/p&gt;&#xA;" OwnerUserId="2750" LastActivityDate="2017-03-31T20:35:23.603" Title="Dynamic clustering for text documents" Tags="|clustering|text-mining|similarity|similar-documents|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="17992" PostTypeId="2" ParentId="9785" CreationDate="2017-03-30T08:28:54.770" Score="6" Body="&lt;p&gt;Missing word prediction has been added as a functionality in the latest version of Word2Vec. Of course your sentence need to match the Word2Vec model input syntax used for training the model (lower case letters, stop words, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usage for predicting the top 3 words for &quot;When I open ? door&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(model.predict_output_word(['When','I','open','door']), topn = 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30575" LastEditorUserId="30575" LastEditDate="2017-08-07T09:35:34.127" LastActivityDate="2017-08-07T09:35:34.127" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="17993" PostTypeId="2" ParentId="14428" CreationDate="2017-03-30T09:14:29.010" Score="2" Body="&lt;p&gt;Check out &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;silhouette score&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Formula for &lt;em&gt;i&lt;/em&gt; th data point&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(b(i) - a(i)) / max(a(i),b(i))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;strong&gt;b(i)&lt;/strong&gt; -&gt; dissimilarity from nearest neighbouring cluster&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;a(i)&lt;/strong&gt; -&gt; dissimilarity between points within cluster&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives a score between -1 and +1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Interpretation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;+1 means very good fit&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-1 means misclassified [should have belonged to a  different cluster]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After calculating silhouette score for each data point, you can take a call on the choice for the number of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import make_blobs&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.metrics import silhouette_samples, silhouette_score&#xA;&#xA;X, y = make_blobs(n_samples=500,&#xA;                  n_features=2,&#xA;                  centers=4,&#xA;                  cluster_std=1,&#xA;                  center_box=(-10.0, 10.0),&#xA;                  shuffle=True,&#xA;                  random_state=1)  # For reproducibility&#xA;&#xA;range_n_clusters = [2, 3, 4, 5, 6]&#xA;&#xA;for n_clusters in range_n_clusters:&#xA;    # Initialize the clusterer with n_clusters value and a random generator&#xA;    # seed of 10 for reproducibility.&#xA;    clusterer = KMeans(n_clusters=n_clusters, random_state=10)&#xA;    cluster_labels = clusterer.fit_predict(X)&#xA;&#xA;    # The silhouette_score gives the average value for all the samples.&#xA;    # This gives a perspective into the density and separation of the formed&#xA;    # clusters&#xA;    silhouette_avg = silhouette_score(X, cluster_labels)&#xA;    print(&quot;For n_clusters =&quot;, n_clusters,&#xA;          &quot;The average silhouette_score is :&quot;, silhouette_avg)&#xA;&#xA;    # Compute the silhouette scores for each sample&#xA;    sample_silhouette_values = silhouette_samples(X, cluster_labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2017-03-30T09:14:29.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17995" PostTypeId="2" ParentId="17991" CreationDate="2017-03-30T12:04:21.120" Score="2" Body="&lt;p&gt;As you say, thinking ahead about the number of clusters may be limiting. A simple solution would be to use KNN.&lt;br&gt;&#xA;However, KNN can be pretty expensive to run over hundreds of ks of documents. In order to limit your search space, you should first filter out (and do it quickly) documents that there is no chance that would share a cluster with the new document. So, I'm thinking some sort of hash, but a simpler solution can be to take the $l$ leading indexes in your TF-IDF representation or something. So you'll need some dictionary mapping sets of &lt;em&gt;indices&lt;/em&gt; (as in the words in the corpus) to some integer value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a general scheme can follow these lines:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF&lt;/a&gt; representation $d$ to your document.&lt;/li&gt;&#xA;&lt;li&gt;Extract some $l$ words that have highest value in that document, call this set $W$.&lt;/li&gt;&#xA;&lt;li&gt;Extract a set $S$ of all documents that had $W$ (or, say, 90% of $W$) as their leading words.&lt;/li&gt;&#xA;&lt;li&gt;Run KNN for $d$ over set $S$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another, simpler but more mathematically based way to do this, is named &lt;a href=&quot;https://en.wikipedia.org/wiki/MinHash&quot; rel=&quot;nofollow noreferrer&quot;&gt;MinHash&lt;/a&gt; and is thoroughly depicted in Jure Leskovec book `&lt;a href=&quot;http://www.mmds.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mining Massive Data Sets&lt;/a&gt;, chapter 3 section 3.4. The method uses &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag-of-words model&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jaccard similarity&lt;/a&gt; to calculate random permutation signatures and then apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSH&lt;/a&gt; on the signatures to bucket similar documents together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="29960" LastActivityDate="2017-03-30T12:04:21.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17996" PostTypeId="1" AcceptedAnswerId="18015" CreationDate="2017-03-30T12:07:06.003" Score="1" ViewCount="1263" Body="&lt;p&gt;I've improved my text classification to topic module, from simple word2vec to piped tfidf and OneVsRestClassifier (using sklearn). It does improve the classification but with word2vec I was able to calculate the match percentage for each topic and with OneVsRestClassifier i get a match or no match to a specific topic. Is there a way to see with OneVsRestClassifier what was the percentage of the classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.&#xA;I am not talking about evaluating the performance of the training but the actual real time matching percentage.&lt;/p&gt;&#xA;" OwnerUserId="30519" LastEditorUserId="30519" LastEditDate="2017-03-31T06:52:06.460" LastActivityDate="2017-04-02T10:29:33.627" Title="How to detect the match precision of OneVsRestClassifier" Tags="|classification|nlp|scikit-learn|multiclass-classification|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17997" PostTypeId="1" CreationDate="2017-03-30T13:33:47.193" Score="-1" ViewCount="710" Body="&lt;p&gt;Lets give an example&#xA;&lt;code&gt;X: 1 2 3 4 5&#xA;Y: .9 .91 .92 .93 .94&#xA;Z: 20 36 999 211&#xA;M. 4000 3456 1 0&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I have such dataset, Which clustering algorithm to choose ? Also, How to interpret the results after clustering ?&#xA;Meaning: How to feed 4D dataset into cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found DBSCAN available on internet for 2D with which plot is possible. Since my dataset is 4 D and varies ILLOGICALLY...I am afraid to feed this to Algorithm&lt;/p&gt;&#xA;&#xA;&lt;p&gt;`&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pdb&#xA;import matplotlib.pyplot as plt&#xA;from numpy.random import rand&#xA;from numpy import square, sqrt&#xA;def regionQuery(P, eps, D): &#xA;    neighbourPts = []&#xA;    for point in D:&#xA;        #print point        &#xA;        if sqrt(square(P[1] - point[1]) + square(P[2] - point[2]))&amp;lt;eps:&#xA;            neighbourPts.append(point)&#xA;    return neighbourPts&#xA;def DBSCAN(D, eps, MinPts):&#xA;    noise = []&#xA;    visited = []&#xA;    C = []&#xA;    c_n = -1&#xA;    for point in D:&#xA;        visited.append(point) #marking point as visited&#xA;    #   print point     &#xA;        neighbourPts = regionQuery(point, eps, D)&#xA;        if len(neighbourPts) &amp;lt; MinPts:&#xA;            noise.append(point)&#xA;        else:&#xA;            C.append([])            &#xA;            c_n+=1&#xA;            expandCluster(point, neighbourPts, C, c_n,eps, MinPts, D, visited)&#xA;    print(&quot;no. of clusters: &quot; , len(C)  )&#xA;    print(&quot;length of noise:&quot;, len(noise))&#xA;    for cluster in C:&#xA;        col =[rand(1),rand(1),rand(1)]      &#xA;        #print(cluster)     &#xA;        plt.scatter([i[1] for i in cluster],[i[2] for i in cluster],color=col)&#xA;    plt.show()&#xA;def expandCluster(P, neighbourPts, C, c_n,eps, MinPts, D, visited):&#xA;    C[c_n].append(P)&#xA;    for point in neighbourPts:&#xA;        if point not in visited:&#xA;            visited.append(point) &#xA;            neighbourPts_2 = regionQuery(point, eps, D)&#xA;            if len(neighbourPts_2) &amp;gt;= MinPts:&#xA;                neighbourPts += neighbourPts_2&#xA;        if point not in (i for i in C):&#xA;            C[c_n].append(point)&#xA;eps =20#input(&quot;enter eps&quot;)&#xA;x=200*rand(10)&#xA;y=200*rand(10)&#xA;l=[]&#xA;for i in range(10):&#xA;    l.append([i,x[i],y[i]])&#xA;#pdb.set_trace()&#xA;DBSCAN(l,eps,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;` &lt;/p&gt;&#xA;" OwnerUserId="27469" LastEditorUserId="27469" LastEditDate="2017-03-30T18:05:32.170" LastActivityDate="2018-02-19T12:02:42.573" Title="Which Clustering algorithm to use for unique 4Dimension dataset before feeding to correlation?" Tags="|python|clustering|dbscan|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="17999" PostTypeId="1" CreationDate="2017-03-30T16:24:43.480" Score="1" ViewCount="115" Body="&lt;p&gt;First off, I am new to machine learning, so these questions may be trivial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I am trying to tune an object with numeric knobs and numeric outputs. By doing a brute force tuning (permutations), there is a solution to find ideal values of the outputs, but it takes time. I'm trying to take a stab at using ML to at least shorten the process of tuning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JMXBJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JMXBJ.png&quot; alt=&quot;object to tune&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset for a large number of good units that were tuned successfully, at the various number of attempts though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;   passing requirement: X = near 10, Y = near 5, Z = near 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Object 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Try 1 =&amp;gt; A = 1, B = 2, C = 3 ; X = 1, Y = 0, Z = 1  =&amp;gt; not good&#xA;Try 2 =&amp;gt; A = 1, B = 1, C = 1 ; X = 10, Y = 5, Z = 4 =&amp;gt; good enough&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Object 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Try 1 =&amp;gt; A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 =&amp;gt; lucky!!!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Object 3:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;Try 10 =&amp;gt; A = 1.4, B = 2.6, C = 3.8 ; X = 10, Y = 5, Z = 3.9 =&amp;gt; took a while!!!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how do I prepare the data for training and testing with this type of problem, as each object had a variable number of tries before it got successfully tuned. &#xA;Should I just take the last successful combination for each object, and maintain the same columns (A, B, C, X, Y, Z). Or take them all (multiple rows per object)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, for each object record, append another set of columns, such that per object, I have only one row. For example (A1,B1,C1,X1,Y1,Z1, A2,B2,C2,X2,Y2,Z2,... &lt;em&gt;An,Bn,Cn,Xn,Yn,Zn&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the algorithm choice, I can only articulate that this isn't a classification problem (or a binary outcome). Something like regression, or decision tree, or hill climbing if there's such a thing?&lt;/p&gt;&#xA;" OwnerUserId="30597" LastEditorUserId="81999" LastEditDate="2020-02-19T04:49:52.423" LastActivityDate="2020-02-19T04:49:52.423" Title="Preparing data, choosing algorithm" Tags="|algorithms|scikit-learn|data-cleaning|decision-trees|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="18000" PostTypeId="1" CreationDate="2017-03-30T16:56:05.620" Score="2" ViewCount="150" Body="&lt;p&gt;I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I went through &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=10&amp;amp;ved=0ahUKEwj3rPOB0v7SAhVCUbwKHR0-Bx4QFghQMAk&amp;amp;url=https%3A%2F%2Fwww.ntnu.no%2Fwiki%2Fdownload%2Fattachments%2F71733389%2FWEBIST_2014_Ozgobek%2520Final.pdf%3Fversion%3D1%26modificationDate%3D1400066559000%26api%3Dv2&amp;amp;usg=AFQjCNHAk1isl05ldRN8g7Pm0m4Hc8bteA&amp;amp;sig2=tpRXPiT6_aupz1336hHT2A&amp;amp;bvm=bv.151325232,d.dGc&amp;amp;cad=rja&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; which mentions different methods used for news recommendation (at 7th page) and most of them are using some sort of probabilistic methods (&lt;strong&gt;bayesian networks, latent dirichlet allocation, naive bayes model, probabilistic matrix factorization models&lt;/strong&gt;). Also, some news recommenders  are based on &lt;strong&gt;multi-armed-bandit&lt;/strong&gt; problem (e.g. yahoo's front page). I have zero knowledge about these methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am very confused regarding what should be my next step. Right now, i am planning to go ahead with a very basic hybrid approach (with collaborative filtering and content based filtering). But it seems like i will have to explore these fields very soon, so what will be a good strategy (and resource) to explore these fields?&lt;/p&gt;&#xA;" OwnerUserId="30599" LastActivityDate="2017-07-28T23:42:52.627" Title="Proceeding with various methods for news recommendation" Tags="|recommender-system|probability|naive-bayes-classifier|lda|bayesian-networks|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18001" PostTypeId="1" CreationDate="2017-03-30T17:14:41.463" Score="1" ViewCount="134" Body="&lt;p&gt;I have user data login logs from Active directory data that lists when a user was logged in to a server. I am looking to find a deviation in the normal behavior of the user. For example if a user is usually active from say 9am to 5pm, if there is some activity outside this, it should be detected.&#xA;I am unsure if the min(first logon) and the max(last logon) should be considered.&#xA;Any input will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="25407" LastActivityDate="2017-03-30T20:49:40.827" Title="How to determine the most active time of a user from active directory logs" Tags="|time-series|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18006" PostTypeId="2" ParentId="18000" CreationDate="2017-03-30T19:35:19.683" Score="0" Body="&lt;p&gt;The best way to decide a good strategy will be to look at this in the two following ways:-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;From a scoring approach(&lt;strong&gt;Not Recommended&lt;/strong&gt;): You can test all the aforementioned algorithms with your data sets and check which gives the best score. You can use tools like Sci-Kit Learn to implement all those algorithms and which has the best score as corresponding to your data set and use that algorithm. The disadvantage being that sometimes even of the score is very high the data set might not be coherent with that algorithm with that I mean you might not get the results you want despite having a high score. This is why I don't recommend this.&lt;/li&gt;&#xA;&lt;li&gt;From a dataset oriented approach: For this approach, you have to have carefully studied your dataset. Even starting off with content based filtering isn't a bad idea. But depending on your needs and your necessity to improve precision in specific cases will ensure you explore algorithms to solve that particular problem. This way you won't stress yourself with all the unnecessary algorithms. You can start off using Naive Bayes as it is simple to understand and then pick it up from there depending on what you need in your recommendation, what are the features you are using to provide recommendations. All such factors come into play. This way you can explore the required methods you need to make your model more efficient. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-03-30T19:35:19.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18007" PostTypeId="2" ParentId="18001" CreationDate="2017-03-30T19:57:57.823" Score="2" Body="&lt;p&gt;As always, start by &lt;em&gt;looking at the data&lt;/em&gt;.  Pick a user, and look at their typical login times.  Maybe plot a histogram to show the distribution of time of day of their login.  Pick another user, and look at them.  Pick a few more.  Do you see any patterns?  Do they have a common shape?  Do users fall into a few clusters?  Try to understand the data well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, once you have a feeling for what normal behavior tends to look like, and how it varies from user to user, you can start to think about how to build a &lt;em&gt;model&lt;/em&gt; that captures this random behavior and helps you distinguish normal vs abnormal behavior (perhaps by some kind of anomaly detection).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as always: start by visualizing the data.  Asking how to approach the problem or what machine learning method to use is premature, until you have a better understanding of the data.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-03-30T19:57:57.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18008" PostTypeId="2" ParentId="18001" CreationDate="2017-03-30T20:49:40.827" Score="0" Body="&lt;p&gt;Ok, this is a very exciting problem. Quite different than the ones we usually see on this site. And, even more exciting is that I think that one of the oldest methods in the book is what you are truly looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So from what I gather, you have the time that a user is active per day as your data and you want to determine times when they log on that deviates away from their usual activity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex: User 1, active from 9:00am to 15:00pm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is often called &lt;strong&gt;anomaly detection&lt;/strong&gt;. You are trying to detect when a behavior from a specific user is anomalous (different) than their normal. So how is this done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anomaly detection algorithms are capable of learning the distribution a single set of labels (normal activity) and then it will be able to flag when an anomaly occurs (anomalous activity). This is when an instance is sufficiently different than the learned distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the baseline distribution in your case it is very easy which makes it quite exciting! All you need to do is collect information about the user during a burn-in period. Here you need to make a choice. Do you want to track the users activity daily, or throughout the week, the month? etc... If we are talking about activity in a server that is used for work, weekly would likely be better. However, if it's a gaming server then maybe daily is better. You need to think about the regularity of the users schedule in regards to the kind of server you are hosting. Let's assume you are interested in their daily activity. Then you can have a burn-in period of a few weeks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So set up a probability graph which will be your probability distribution function. The $x$-axis is going to be the hours of the day, and the $y$-axis will be the frequency (you will need this to be normalized afterwards for the math to work out). So, each time the user logs on, you will add onto the plot a box unit 1 ($y$-axis) for that time slot. This is essentially a histogram!! At the end you will end up with a curve that shows the likelihood of the user being active for any given hour of the day.     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you see that a user logs on at a time $X$, you can use the probability density function to get the probability of the event having been drawn from that distribution. In other words, what is the likelihood that the user would be online at a time $X$. If this likelihood is below a threshold $\alpha$ then you are safe to assume that it is anomalous. This is more commonly known as the generalized likelihood ratio test (GLRT).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The disadvantage of this technique is that you need to train the hyper-parameter $\alpha$, your threshold. Which is easy to do given you have labeled data. You know what behavior is normal/anomalous. If you don't have this information. You can just set an arbitrary threshold and fix it as you see fit based on the resulting false positive or false negative rates you will encounter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am suggesting this because you can easily get the parametrization of your probability distribution. So if you do not use it you are missing out on a lot of information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another non-parametric way to do anomaly detection would be to use a specialized K-NN method which makes use of the $p$-statistic test in a very clever way. I have found a lot of success on very limited datasets using their methods. The paper you should start with is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning Minimum Volume Sets and Anomaly Detectors from KNN Graphs&#xA;&lt;a href=&quot;https://arxiv.org/pdf/1601.06105.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1601.06105.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then you can go onto their simplest implementation, which you can then expand on by reading their other papers (bagging, ranking, etc...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anomaly detection with score functions based on nearest neighbor graphs&#xA;&lt;a href=&quot;https://papers.nips.cc/paper/3723-anomaly-detection-with-score-functions-based-on-nearest-neighbor-graphs.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://papers.nips.cc/paper/3723-anomaly-detection-with-score-functions-based-on-nearest-neighbor-graphs.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-30T20:49:40.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18009" PostTypeId="2" ParentId="17988" CreationDate="2017-03-30T21:46:44.583" Score="1" Body="&lt;p&gt;One way is to generate a list of acceptable words along with maybe some regular expressions to check for things like vowels and common parts of speech.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way is to count the frequency of each word/token and eliminate the words that only appear once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neither of these methods would work perfectly because the definition of the words you want to eliminate isn't well defined.  I think you just want to eliminate low frequency words but that could include some valid words as well.  &lt;/p&gt;&#xA;" OwnerUserId="25915" LastActivityDate="2017-03-30T21:46:44.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18011" PostTypeId="2" ParentId="17999" CreationDate="2017-03-30T22:24:44.430" Score="1" Body="&lt;p&gt;I don't know that I'd call this a &lt;em&gt;machine learning&lt;/em&gt; problem per se...though others may disagree, this sounds like an optimization problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your description is vague, but let's pretend it's a guitar that you're trying to tune and you have 6 strings. The trick is to define an objective function that quantifies, with a single number, how 'good' the tuning is. (The program can independently manipulate each string's tension). Perhaps it's some deviance in power from an ideal FFT, and you sum the deviance from each string. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optimization routines such as simplex, sub plex, Nelder-Mead, GA, etc., are ways of manipulating the tension in 'smart' ways that obviate the need for a brute force (or 'grid search') solution. So long as you can define an objective function, you can start trying it out, though the local max/min problem will need to be dealt with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd look into some tutorials on minimization algorithms &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-03-30T22:24:44.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18012" PostTypeId="1" CreationDate="2017-03-30T22:46:41.447" Score="0" ViewCount="132" Body="&lt;p&gt;I am studying caret package in R. I know in this package, all the data are assumed to be numeric. But I always have some data that has a lot of strings or factors. For example, a product category could include hundreds of thundred of strings indicating all the different products. A worker ID list may includes a lot of numbers. They look like numeric, but you cannot do any calculation because it does not make any sense. They are actually supposed to be characters or strings. So, we cannot use model.matrix or dummyVars to transfer them to numeric because 1). It should not be transferred like this; or 2). The matrix will be so huge after transferring that we cannot deal with. So my question is:&#xA;How can we deal with an attribute with a lot of characters or a lot of so-called numbers when using caret package in R?&lt;/p&gt;&#xA;" OwnerUserId="16911" LastActivityDate="2017-03-31T12:14:01.340" Title="How to deal with an attribute including a lot of character or factor values when using caret package in R" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18013" PostTypeId="1" AcceptedAnswerId="19270" CreationDate="2017-03-30T22:53:45.133" Score="7" ViewCount="4135" Body="&lt;p&gt;I am new to VAEs but find them quite fascinating. I was wondering if anyone might have any tips or tricks regarding, how one should build the encoder and decoder layers w.r.t. to time-series data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose my inputs are $x \in \mathbb{R}^{1\times1000}$ so they have a thousand dimensions. The data is highly-nonlinear and rather noisy. I have a couple of tens of thousand $x$:s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind, what are some general guidelines (because I cannot find any, owing to the VAE field being somewhat particular and very new):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How deep should my encoder and decoder network be? Are there any good guidelines?&lt;/li&gt;&#xA;&lt;li&gt;Should one use fully connected dense networks, or stacked conv1Dnets?&lt;/li&gt;&#xA;&lt;li&gt;What activation functions are good choices?&lt;/li&gt;&#xA;&lt;li&gt;Can we say anything about the 'best' dimensionality of the latent dimension?&lt;/li&gt;&#xA;&lt;li&gt;Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?&lt;/li&gt;&#xA;&lt;li&gt;Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?&lt;/li&gt;&#xA;&lt;li&gt;In the decoder layer, is it better to up-sample before returning to the original input dimension?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any tips or tricks would be most welcome. I have naturally tried a few of these, but I cannot discern a trend at present.&lt;/p&gt;&#xA;" OwnerUserId="30227" LastActivityDate="2017-05-29T15:55:22.513" Title="Tips and tricks for designing time-series variational autoencoders" Tags="|deep-learning|keras|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18014" PostTypeId="1" AcceptedAnswerId="18019" CreationDate="2017-03-31T02:34:52.697" Score="4" ViewCount="11126" Body="&lt;p&gt;I have install &quot;Orange Data Mining v3.4.1&quot; in Anaconda Python v3 environment using commands : &quot;conda install orange3&quot; sucessfully. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I do not know how to call &quot;Orange&quot; as Application to start using it. There is also NO icon on the desktop as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help. Thank you very much in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards&#xA;Pearapon S.&lt;/p&gt;&#xA;" OwnerUserId="30614" LastActivityDate="2023-05-18T21:38:47.880" Title="Orange (Data Mining) : How to start using &quot;Orange&quot; from Python Anaconda Environment?" Tags="|machine-learning|data-mining|orange|" AnswerCount="1" CommentCount="0" ClosedDate="2017-04-01T18:26:09.073" ContentLicense="CC BY-SA 3.0" />
  <row Id="18015" PostTypeId="2" ParentId="17996" CreationDate="2017-03-31T02:42:04.890" Score="1" Body="&lt;p&gt;Yes, of course. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you have used sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;OneVsRestClassifier&lt;/a&gt; and so you have a decision function for example a Support Vector Classifier with say linear kernel. Use &lt;code&gt;set_params&lt;/code&gt; to change &lt;code&gt;probability&lt;/code&gt; key to &lt;code&gt;True&lt;/code&gt;, default is &lt;code&gt;False&lt;/code&gt;. Use this in the OneVsRestClassifier classifier and then go with the inbuilt function &lt;code&gt;predict_proba&lt;/code&gt; like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.multiclass import OneVsRestClassifier&#xA;from sklearn.svm import SVC&#xA;mod = OneVsRestClassifier(SVC(kernel='linear').set_params(probability=True)).fit(samples,classes)&#xA;print mod.predict_proba(np.array([your_sample_vector]).reshape(1,-1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use your old &lt;code&gt;LinearSVC&lt;/code&gt; with &lt;code&gt;decision_function&lt;/code&gt; to find the distance from the hyperplane and convert them to probabilities like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mod = OneVsRestClassifier(LinearSVC()).fit(sample,clas)&#xA;proba = (1./(1.+np.exp(-mod.decision_function(np.array(your_test_array).reshape(1,-1)))))&#xA;proba /= proba.sum(axis=1).reshape((proba.shape[0], -1))\&#xA;print proba&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you don't need tuning the parameters, I guess. :)&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2017-04-02T10:29:33.627" LastActivityDate="2017-04-02T10:29:33.627" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18017" PostTypeId="1" CreationDate="2017-03-31T05:06:09.040" Score="1" ViewCount="195" Body="&lt;p&gt;I analyse log files, very often there are strings that only differ on few places and are same every where else, I am trying to find the generic string that they most likely belong to e.g. :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UHGUYGUYGUYGUY id = U1234 UYAG*&amp;amp;^T*#@G*(&amp;amp;G@ id2 = 8767 ib97y79yh0978&#xA;UHGUYGUYGUYGUY id = Z1D#34 UYAG*&amp;amp;^T*#@G*(&amp;amp;G@ id2 = 98h ib97y79yh0978&#xA;Sss3ug87g87g78ghs837g8 obj { 876t7g }937hs937hs973h97sh397 jh7897y98h&#xA;Sss3ug87g87g78ghs837g8 obj { 98u2 }937hs937hs973h97sh397 ZN7897y98h&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to me the only difference is between the two the value of ids, so a generic forms/groupings will be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;UHGUYGUYGUYGUY id = * UYAG*&amp;amp;^T*#@G*(&amp;amp;G@ id2 = * ib97y79yh0978&#xA;Sss3ug87g87g78ghs837g8 obj { * }937hs937hs973h97sh397 *7897y98h&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure what in machine learning I should be looking up for this problem, or even if this problem has a name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;of course this is a very simplified example, the number, location of id's can vary for different generic cases, that is why I can not write old fashioned code for this, too many things can change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything in machine learning to help find groups for such strings? if yes, what is it called?&lt;/p&gt;&#xA;" OwnerUserId="3613" LastActivityDate="2017-03-31T13:59:43.157" Title="String classification beginner question" Tags="|beginner|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18018" PostTypeId="1" CreationDate="2017-03-31T06:36:54.547" Score="8" ViewCount="10570" Body="&lt;p&gt;Is there a R implementation of isolation forest for anomaly detection?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to the implementation from &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html&quot; rel=&quot;noreferrer&quot;&gt;sklearn&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="29695" LastActivityDate="2018-12-09T09:23:42.863" Title="Is there a R implementation of isolation forest for anomaly detection?" Tags="|machine-learning|r|scikit-learn|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18019" PostTypeId="2" ParentId="18014" CreationDate="2017-03-31T06:51:35.293" Score="5" Body="&lt;p&gt;Instructions can be found on their Github page: &lt;a href=&quot;https://github.com/biolab/orange3#starting-orange-gui&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/biolab/orange3#starting-orange-gui&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Short Answer:&#xA;In the conda env run orange-canvas&#xA;or&#xA;python3 -m Orange.canvas. Add --help for a list of program options.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastEditorUserId="149532" LastEditDate="2023-05-18T21:38:47.880" LastActivityDate="2023-05-18T21:38:47.880" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18020" PostTypeId="1" CreationDate="2017-03-31T06:55:38.740" Score="1" ViewCount="291" Body="&lt;p&gt;I am confused about the group parameter in caffe convolution layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image I have an input of &lt;code&gt;20x3x227x227&lt;/code&gt; and filter size is &lt;code&gt;96x3x11x11&lt;/code&gt; with &lt;code&gt;stride=4&lt;/code&gt; and &lt;code&gt;padding=0&lt;/code&gt;, if I set the &lt;code&gt;group=2&lt;/code&gt;. Then the setting becomes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;group1 : 20x1x227x227, filter1: 96x1x11x11&#xA;group2 : 20x2x227x227, filter2: 96x2x11x11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is that correct? If that is correct, how can you concatenate them to the output &lt;code&gt;20x96x55x55&lt;/code&gt;? &lt;/p&gt;&#xA;" OwnerUserId="30622" LastActivityDate="2017-03-31T06:55:38.740" Title="Intuition of group parameter in Caffe Layer" Tags="|caffe|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18021" PostTypeId="2" ParentId="18013" CreationDate="2017-03-31T06:59:42.647" Score="2" Body="&lt;p&gt;I am facing a pretty similar set of challenges with the data I am dealing with. So far I have found success with 1-2 hidden layers with a large number of nodes (50 or so).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I try to increase the complexity of the network the reconstruction errors go up. Would love some pointers if you find success with VAEs.&lt;/p&gt;&#xA;" OwnerUserId="30623" LastActivityDate="2017-03-31T06:59:42.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18022" PostTypeId="2" ParentId="18018" CreationDate="2017-03-31T07:28:01.323" Score="10" Body="&lt;p&gt;See the &lt;code&gt;iforest&lt;/code&gt; package on &lt;a href=&quot;https://sourceforge.net/projects/iforest/&quot; rel=&quot;noreferrer&quot;&gt;Sourceforge&lt;/a&gt; or on &lt;a href=&quot;https://r-forge.r-project.org/projects/iforest/&quot; rel=&quot;noreferrer&quot;&gt;R-Forge&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This package implements an anomaly detection method that detects data-anomalies using binary trees. Using the property that anomalies are more susceptible to isolation, anomalies can be detected as data points which have short expected path lengths.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Isolation Forest detects data-anomalies using binary trees. Platform:&#xA;  R Reference: Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou, “Isolation&#xA;  Forest”, IEEE International Conference on Data Mining 2008 (ICDM 08)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or the &lt;a href=&quot;https://github.com/Zelazny7/isofor&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;isofor&lt;/code&gt; package on GitHub&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One efficient way of performing outlier detection in high-dimensional datasets is to use random forests. The isofor ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split&#xA;value between the maximum and minimum values of the selected feature. &lt;/p&gt;&#xA;" OwnerUserId="2961" LastEditorUserId="2961" LastEditDate="2017-07-25T15:37:58.080" LastActivityDate="2017-07-25T15:37:58.080" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18023" PostTypeId="2" ParentId="17980" CreationDate="2017-03-31T07:59:41.020" Score="0" Body="&lt;p&gt;Turns out there were two versions of the tutorial. One version incomplete, the other complete. This is the complete version - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if (!require(&quot;magrittr&quot;)) {&#xA;    install.packages(&quot;magrittr&quot;)&#xA;    library(magrittr)&#xA;}&#xA;&#xA;stock_return = apply(stocks, 1, function(x) {x / stocks[1,]}) %&amp;gt;% &#xA;                                t %&amp;gt;% as.xts&#xA;&#xA;head(stock_return)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="6448" LastActivityDate="2017-03-31T07:59:41.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18025" PostTypeId="1" AcceptedAnswerId="18028" CreationDate="2017-03-31T11:16:22.653" Score="1" ViewCount="2367" Body="&lt;p&gt;I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the occurrence of a word in all other documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have merge test and train data every time I receive a new document for classification? It will change the model as well every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something? I think, although SVM on TF-IDF giving good results it can not be used in production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other way to tackle this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets take an example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training Set:&#xA;Doc_1: Chelsea won the match. {Sports}&#xA;Doc_2: India won the third test match against Austrailia {Sports}&#xA;Doc_3: I want to sleep {Non-Sport}&#xA;Doc_4: 13 palace to see in Auckland {Non-Sport}&#xA;&#xA;New Testing Set:&#xA;Doc_5: Climate change impacts in Austrailia&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now how can I find IDF score of &quot;Austrailia&quot; in Doc_5 without merging this document with training set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Doc_5 contains the word &quot;Austrailia&quot;, it will change the IDF score of &quot;Australia&quot; in Doc_1 will also change, thus model needs retraining&lt;/p&gt;&#xA;" OwnerUserId="28475" LastEditorUserId="28475" LastEditDate="2017-03-31T13:46:12.167" LastActivityDate="2017-03-31T13:46:12.167" Title="How to use binary text classifier(built using SVM with TF-IDF) to classify new text document?" Tags="|classification|svm|text|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18026" PostTypeId="1" AcceptedAnswerId="20419" CreationDate="2017-03-31T11:43:56.663" Score="6" ViewCount="4664" Body="&lt;p&gt;I'm trying to use MITIE to extract named entities from short text. I'm interested in entities such as dates, times, names, and locations. Out of the box, MITIE only recognises names, locations, and organisations. I'd like to train it to recognise dates, times and other categories as well. From looking at the structure of MITIE's directories and from the dlib website, I gather that this is done via an SVM. Is this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to adding new categories to the named entity recogniser, I have several questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can this be done in an augmentative fashion? That is, given an existing NER system, can I just add categories examples and train it to recognise those as well? Or do I need to train models from scratch?&lt;/li&gt;&#xA;&lt;li&gt;If I do need train models from scratch, what dataset can I use to do this?&lt;/li&gt;&#xA;&lt;li&gt;Related to adding new examples, is there an online method that I can use, feeding the system new examples and categories as and when I need to?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="7922" LastActivityDate="2020-11-24T09:24:34.493" Title="How does MITIE perform named entity recognition?" Tags="|nlp|named-entity-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18028" PostTypeId="2" ParentId="18025" CreationDate="2017-03-31T12:04:39.213" Score="1" Body="&lt;p&gt;What is your model built in? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most popular libraries have a score function separate from the training part. You should be able to just pass the new document to the score function of the trained model and get back the predicted class.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-31T12:04:39.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18029" PostTypeId="2" ParentId="18017" CreationDate="2017-03-31T12:07:58.017" Score="2" Body="&lt;p&gt;That sounds like k nearest neighbor clustering because a large portion of the string is similar. This answer to a similar question from 2014 has code you could try in Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/158090&quot;&gt;https://stats.stackexchange.com/a/158090&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-03-31T12:07:58.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18030" PostTypeId="2" ParentId="18025" CreationDate="2017-03-31T12:08:35.993" Score="1" Body="&lt;p&gt;So your TF-IDF is trained only using the training set. It will determine the frequency of the occurrence of words. If you show the TF-IDF a new word it has not yet seen then it will simply ignore it. It will only use words that are in its training set. So, NO you do not retrain your model after you have built it. Once, you go through the training stage for your TF-IDf that is the library of words that your algorithm can detect. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-31T12:08:35.993" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18031" PostTypeId="2" ParentId="18012" CreationDate="2017-03-31T12:14:01.340" Score="1" Body="&lt;p&gt;Yeah, some (not all) libraries have that numeric input limitation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you're looking for is called &quot;one-hot encoding&quot; which can basically convert an N level factor like your product categories to N binary columns. Here is a post describing a few ways of doing that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://discuss.analyticsvidhya.com/t/how-to-do-one-hot-encoding-in-r/6982/5&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://discuss.analyticsvidhya.com/t/how-to-do-one-hot-encoding-in-r/6982/5&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-31T12:14:01.340" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18032" PostTypeId="2" ParentId="18025" CreationDate="2017-03-31T12:34:25.647" Score="0" Body="&lt;p&gt;Through TF-IDF you have got the important words which you have used to train your SVM Model. So when you pass in the test data, only the words(features) that are selected through TF-IDF are important and will be used by the SVM Model to predict the label &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just make sure that the matrix used to train SVM is similar for both training and testing &lt;/p&gt;&#xA;" OwnerUserId="29793" LastActivityDate="2017-03-31T12:34:25.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18033" PostTypeId="2" ParentId="155" CreationDate="2017-03-31T12:41:01.497" Score="3" Body="&lt;p&gt;Quite late to the party, but I believe the following might be helpful:&#xA;&lt;a href=&quot;http://data.worldbank.org/indicator&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://data.worldbank.org/indicator&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://www.quandl.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quandl.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29173" LastActivityDate="2017-03-31T12:41:01.497" CommentCount="0" CommunityOwnedDate="2017-03-31T12:41:01.497" ContentLicense="CC BY-SA 3.0" />
  <row Id="18036" PostTypeId="1" AcceptedAnswerId="18039" CreationDate="2017-03-31T13:37:07.090" Score="0" ViewCount="91" Body="&lt;p&gt;I'm trying to apply machine learning to pharmaceutical manufacturing to predict whether batches of drug products manufactured are good or not. for the sake of relatability, let's use coffee brewing as an analogous process. Let's imagine that I'm trying to predict the acidity of the coffee that I've brewed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset that I have contains features such as temperature of water, stirring speed and pressure that are constantly measured (say..on a per second basis) over a variable amount of time (the first cup may be brewed in 5 minutes, the second in 10 etc). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of preprocessing should I perform on such a multidimensional dataset? One stumbling block is that for each observation, the duration is different, which may complicate dimension reduction? Once preprocessed, is there any specific model that would suit the task at hand? I'm looking at something like a regression but alternatively, classifiers seem to be fine as well if I split the acidity(pH) into &quot;&amp;lt;5.5&quot; or &quot;&gt;5.5&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope to get some general directions and if you can paste a few links to texts or examples that'll be good! Also, I'm more familiar with python and scikit learn, so if you can point me in the right section in the documentation that'll be great too!&lt;/p&gt;&#xA;" OwnerUserId="30624" LastActivityDate="2017-03-31T14:58:52.593" Title="Building a machine learning model based on a set of timestamped features to predict/classify a label/value?" Tags="|machine-learning|classification|predictive-modeling|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18037" PostTypeId="2" ParentId="18036" CreationDate="2017-03-31T13:58:44.203" Score="0" Body="&lt;p&gt;You can use panda date time function to create various features representing time; Total time, day etc.  Then use these features as your independent variable to train your model &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2017-03-31T13:58:44.203" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18038" PostTypeId="2" ParentId="18017" CreationDate="2017-03-31T13:59:43.157" Score="1" Body="&lt;p&gt;If you will have a lot of instances, K-NN will quickly become very computationally expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest using percent match. The computer can learn the distribution of the different types of strings that you have. When a new string comes, it will compare the string with all other strings and can calculate the likelihood that the string came from a specific class. Then you can pick the greatest likelihood which is the most likely class that the new string will belong to.  &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-03-31T13:59:43.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18039" PostTypeId="2" ParentId="18036" CreationDate="2017-03-31T14:58:52.593" Score="0" Body="&lt;p&gt;I don't know much about coffee or pharmaceuticals but I think the widely varying time samples is a problem. If I brewed one batch of coffee for a minute and another for 5 hours, I'm pretty sure the 5 hour batch would come out burnt-tasting in all cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you break the samples up into cohorts by duration and then train on each cohort? You'd end up with a model for the &quot;1 minute batch&quot;, a model for the &quot;1 hour batch&quot;, etc.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-03-31T14:58:52.593" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18042" PostTypeId="1" CreationDate="2017-03-31T16:39:02.923" Score="0" ViewCount="592" Body="&lt;p&gt;My question is: how can I mask or crop the results, using R, of IDW interpolation to only the area containing the original set of data points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example below, 20 random points are used to interpolate a surface using the IDW function in gstat. A convex hull is obtained from the points set and plotted on the interpolated map. But, I would like to crop the map so only areas within the point cloud show the interpolation result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any suggestions!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;###&#xA;#                             IDW in R&#xA;#&#xA;library(gstat)&#xA;#             &#xA;set.seed(1234)&#xA;x &amp;lt;- rnorm(20)+10&#xA;y &amp;lt;- rnorm(20)+10&#xA;z &amp;lt;- rnorm(20)+10&#xA;#&#xA;xyz &amp;lt;- data.frame(x,y,z)&#xA;coordinates(xyz) &amp;lt;- ~x+y&#xA;xr &amp;lt;- range(x)&#xA;yr &amp;lt;- range(y)&#xA;b &amp;lt;- round((xr[2]-xr[1])/100,2)&#xA;#&#xA;grd &amp;lt;- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),&#xA;y=seq(from=yr[1],to=yr[2],by=b))&#xA;#&#xA;coordinates(grd) &amp;lt;- ~ x+y&#xA;gridded(grd) &amp;lt;- TRUE&#xA;#&#xA;idw&amp;lt;-idw(formula=z~1, locations=xyz, newdata=grd)&#xA;#&#xA;image(idw, col=terrain.colors(16))&#xA;contour(idw, add=T)&#xA;points(x,y, col=4, pch=19)&#xA;#&#xA;library(spatstat)&#xA;conv&amp;lt;-convexhull.xy(x,y)&#xA;plot(conv,add=TRUE)&#xA;#&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22065" LastEditorUserId="29169" LastEditDate="2019-09-07T08:02:03.900" LastActivityDate="2024-03-12T22:02:50.360" Title="How to mask or crop IDW results" Tags="|r|interpolation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18045" PostTypeId="1" CreationDate="2017-03-31T19:47:44.603" Score="3" ViewCount="928" Body="&lt;p&gt;There's a not well-known fact in statistics. That as your sample size increases, more p-values become significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working with a massive sample size consisting of 3 million samples (~ 1% of the U.S. population). It's a Logistic Regression, and p-values of several relations are significant at 5%, 1% and ~0%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fielded my paper's abstract to several journals to see if there was interest. Only a German journal I submitted to (those smart Germans, really) caught on that an extremely large sample size like this might more easily produce significant p-values and that I'd need to somehow adjust for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be clear, an excessively large sample size doesn't produce spurious p-values. It's that more effects with tiny effect sizes start showing significant p-values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want your views on strategies for how to deal with such a situation, to produce reliable research. Some strategies I've thought of are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Conduct the research as is, and only consider as relevant results that have effect sizes that are above a certain threshold (&lt;em&gt;What threshold though?&lt;/em&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do what they do in Data Mining. Split up my data into train, validate, and test sets. (&lt;em&gt;Is this done with Logistic Regression though? Can anyone point me to past papers where the technique is illustrated?&lt;/em&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Skip the usual Logistic Regression and instead use a Data Mining technique such as CART (which may also use Logistic Regression internally).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd like your views on the relative worth of these strategies. Other suggestions, too, would be welcome. Especially welcome will be pointers to prior papers where the authors illustrate how to deal with a similar problem.&lt;/p&gt;&#xA;" OwnerUserId="16928" LastActivityDate="2022-11-21T19:04:14.987" Title="Strategy for dealing with giant sample size" Tags="|logistic-regression|sampling|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18046" PostTypeId="2" ParentId="17997" CreationDate="2017-03-31T19:59:36.277" Score="1" Body="&lt;p&gt;DBSCAN is not limited to 2d (apparently you only found very bad implementations) and can be used with correlation distance, too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But mostly, you need to focus on preprocessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you intend to do correlation anyway, you should consider to z normalize your input data, I.e. &lt;code&gt;x'= (x - mean(x))/stddev(x)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-03-31T19:59:36.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18047" PostTypeId="2" ParentId="17991" CreationDate="2017-03-31T20:35:23.603" Score="3" Body="&lt;p&gt;It sounds as if you &lt;strong&gt;don't need clustering&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But rather you are trying to &lt;strong&gt;detect near duplicates&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is that clustering tries to organize &lt;em&gt;everything&lt;/em&gt; with a focus on the larger, overall structure. But much of your data probably isn't duplicate. Clustering is difficult and slow. Near duplicates is much easier, and much faster (e.g., with MinHash or similarity search)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-03-31T20:35:23.603" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18048" PostTypeId="1" AcceptedAnswerId="18146" CreationDate="2017-04-01T00:12:43.787" Score="0" ViewCount="1158" Body="&lt;p&gt;I am trying to create a CNN network for classification purposes, the network with both input and output is illustrated as such: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nkZx4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nkZx4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input the image is separated into sections, each section is given to a certain model in which a convolution and pooling is applied. This is done as such to ensure full weight share does not occur, as I am not interested in the complete image shares the same weight.  After the convolution an pooling, is all the output of models concatenated into one, and fed into a fully connected layer, which then performs classification and outputs a class.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input I have is the image (either the complete or separated into sections) and the output being the class, but I am not sure how I should train the model, given these input and output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I've constructed the model is as such: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Convolution for different section conv2d_[convolution#]_[model#]&#xA;conv2d_1_1 = Conv2D(filters = 32, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' , input_shape = (3,6,3))&#xA;conv2d_2_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_1_1)&#xA;conv2d_3_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_2_1)&#xA;conv2d_4_1 = Conv2D(filters = 32, kernel_size = (1,1) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_3_1)&#xA;conv2d_4_1_flatten = Flatten()(conv2d_4_1)&#xA;&#xA;conv2d_1_2 = Conv2D(filters = 32, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' , input_shape = (3,6,3))&#xA;conv2d_2_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_1_2)&#xA;conv2d_3_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_2_2)&#xA;conv2d_4_2 = Conv2D(filters = 32, kernel_size = (1,1) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_3_2)&#xA;conv2d_4_2_flatten = Flatten()(conv2d_4_2)&#xA;&#xA;conv2d_1_3 = Conv2D(filters = 32, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' , input_shape = (3,6,3))&#xA;conv2d_2_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_1_3)&#xA;conv2d_3_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_2_3)&#xA;conv2d_4_3 = Conv2D(filters = 32, kernel_size = (1,1) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_3_3)&#xA;conv2d_4_3_flatten = Flatten()(conv2d_4_3)&#xA;&#xA;conv2d_1_4 = Conv2D(filters = 32, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' , input_shape = (3,6,3))&#xA;conv2d_2_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_1_4)&#xA;conv2d_3_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_2_4)&#xA;conv2d_4_4 = Conv2D(filters = 32, kernel_size = (1,1) , padding = &quot;same&quot; , activation = 'relu' )(conv2d_3_4)&#xA;conv2d_4_4_flatten = Flatten()(conv2d_4_4)&#xA;&#xA;#Merging the output of the convolution&#xA;merge = Merged([conv2d_4_1_flatten, conv2d_4_2_flatten,&#xA;                conv2d_4_3_flatten, conv2d_4_4_flatten] mode = 'concat')&#xA;&#xA;#Connecting the merged layer to the fully connected layer&#xA;dense1 = Dense(activation = 'relu')(merge)&#xA;dense2 = Dense(activation = 'relu')(dense1)&#xA;dense3 = Dense(output = 1 ,activation = 'softmax')(dense3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This (I guess) CNN model should now be connected, but how do I train it given the input and outputs I have? How do I parse it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My input data is stored as list of numpy.ndarray. Each row of each ndarray, is the input for each input.  How do I parse each row to each input? And the output is a vector of classes. I have in total 145 classes, and output is vector length 3 with the true labels extracted from 4 input channels?  &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditorDisplayName="user380" LastEditDate="2021-02-16T02:44:18.793" LastActivityDate="2021-02-16T02:44:18.793" Title="How do I provide input and output for such a network structure in keras" Tags="|python|classification|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18049" PostTypeId="2" ParentId="16350" CreationDate="2017-04-01T00:36:49.097" Score="4" Body="&lt;p&gt;The minimum number of training examples is what you have up there:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$4(nm + n^2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information refer to this article:&#xA;Refer to this link if you needed some visual help:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/10615/number-of-parameters-in-an-lstm-model&quot;&gt;Number of parameters in an LSTM model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of units in each layer of the stack can vary. For example in translate.py from Tensorflow it can be configured to 1024, 512 or virtually any number. The best range can be found via cross validation. But I have seen both 1000 and 500 number of units in each layer of the stack. I personally have tested with smaller numbers as well. &lt;/p&gt;&#xA;" OwnerUserId="25481" LastEditorUserId="16853" LastEditDate="2017-12-11T15:47:32.693" LastActivityDate="2017-12-11T15:47:32.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18050" PostTypeId="2" ParentId="17014" CreationDate="2017-04-01T08:50:37.783" Score="1" Body="&lt;p&gt;I think your weight function should depend on $d$ as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The contribution from a single dimension is just the sum of incoming weights. Even for simple graphs this does not seem to have a standard name, although it is quite commonly encountered.&#xA;In the context of random walks it is sometimes called simply &lt;em&gt;degree&lt;/em&gt;, in the context of empirical networks, I found &lt;em&gt;node strength&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is a simple and natural statistic to look at if you wish to model something as a multidimensional network. However, being a linear sum of rather trivial functionals over simple graphs, I doubt it is of much fundamental interest.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2017-04-01T08:50:37.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18054" PostTypeId="1" CreationDate="2017-04-01T19:48:42.537" Score="10" ViewCount="702" Body="&lt;p&gt;I came across two interesting papers that describe promising approaches for document classification using word embedding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1.&lt;/strong&gt; &lt;strong&gt;The fasttext algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Described in the paper &lt;em&gt;Bag of Tricks for Efficient Text Classification&lt;/em&gt; &lt;a href=&quot;https://arxiv.org/abs/1607.01759v3&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(With further explanation &lt;a href=&quot;https://web.stanford.edu/class/cs224n/lectures/cs224n-2017-lecture5-highlight.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. DANs&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Described in the paper &lt;em&gt;Deep Unordered Composition Rivals Syntactic Methods for Text Classification&lt;/em&gt; &lt;a href=&quot;https://pdfs.semanticscholar.org/5bd3/71142c9afee9ec926d9f42b807f36294f82b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between both approaches? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are they essentially the same as they both seem to average word embedding and pass it through an MLP or am I missing something crucial? &lt;/p&gt;&#xA;" OwnerUserId="30437" LastEditorUserId="83275" LastEditDate="2019-11-20T12:16:27.767" LastActivityDate="2020-08-05T19:48:30.503" Title="What is the difference between fasttext and DANs in document classification?" Tags="|machine-learning|neural-network|word2vec|similar-documents|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="18056" PostTypeId="1" AcceptedAnswerId="18058" CreationDate="2017-04-02T03:37:47.290" Score="8" ViewCount="21014" Body="&lt;p&gt;I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn't there an inherent ordering involved? There must be something I'm missing here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add to my confusion I took a problem I was working on and tried using one-hot encoding on a categorical feature versus converting to an integer using xgboost in R. The generalization error using one-hot encoding was marginally better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I took another variable and did the same test, and saw the opposite result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help explain this?&lt;/p&gt;&#xA;" OwnerUserId="26847" LastEditorUserId="13727" LastEditDate="2017-04-02T13:31:05.990" LastActivityDate="2017-04-06T17:34:21.793" Title="Why don't tree ensembles require one-hot-encoding?" Tags="|machine-learning|decision-trees|xgboost|categorical-data|representation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18057" PostTypeId="1" CreationDate="2017-04-02T10:29:23.190" Score="1" ViewCount="94" Body="&lt;p&gt;The tl;dr version first:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I pass the model two number &lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt;, it builds a fully connected state machine with &lt;code&gt;C1&lt;/code&gt; states and &lt;code&gt;C2&lt;/code&gt; emissions. I map all tokens from my training data onto the range &lt;code&gt;[0, C2)&lt;/code&gt; and each label the HMM is supposed to assign a token during inference onto &lt;code&gt;[0, C1)&lt;/code&gt;. Then the HMM goes ahead and does Baum Welch Learning. When it is done it has configured its state machine to maximize the likelihood of the training data (locally).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to my problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume I had two isomorphic initial state machines (isomorphic under consideration of all the probabilities, as structurally all the HMMs are isomorphic anyway, because ergodic). They vary only in their state IDs, so the IDs have been scrambled around from one machine to the other.&#xA;Now, after training both HMMs will be isomorphic again when trained with the same data. That means there is absolutely no connection between the IDs I map the labels from my label set to and the IDs of the states of the HMM. So how then can I interpret the HMM after training? How do I know which state corresponds to which POS-tag? Seems impossible, so I guess I am missing some crucial point.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now a little more detail if the above was unclear:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I take my training data (texts, like newspaper etc.) and count the number of different words (types).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I pass &lt;code&gt;count(types&lt;/code&gt; and &lt;code&gt;count(labels)&lt;/code&gt; the labels being a set of POS-tags. It then randomly constructs a probabilistic fully connected state machine with &lt;code&gt;pow(count(labels), order_of_model)&lt;/code&gt; different states. &lt;code&gt;order_of_model&lt;/code&gt; being the number of hidden variables (POS-tag ngrams) that get combined into an individual state. It also assigns each of these states an initial and an emission probability for all of the types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model assumes a mapping from &lt;code&gt;[0, pow(count(labels), order_of_model))&lt;/code&gt; as state IDs onto external tuples of labels. And a mapping &lt;code&gt;[0,count(types)-1)&lt;/code&gt; for the emissions onto words.&lt;/p&gt;&#xA;" OwnerUserId="30668" LastEditorUserId="30668" LastEditDate="2017-04-02T15:13:50.470" LastActivityDate="2017-04-02T15:13:50.470" Title="Hidden Markov Models: Linking states to labels after EM training" Tags="|machine-learning|unsupervised-learning|probability|language-model|expectation-maximization|" AnswerCount="0" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18058" PostTypeId="2" ParentId="18056" CreationDate="2017-04-02T11:20:00.593" Score="8" Body="&lt;p&gt;The encoding leads to a question of representation and the way that the algorithms cope with the representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider 3 methods of representing n categorial values of a feature:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A single feature with n numeric values.&lt;/li&gt;&#xA;&lt;li&gt;one hot encoding (n Boolean features, exactly one of them must be on)&lt;/li&gt;&#xA;&lt;li&gt;Log n Boolean features,representing the n values.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that we can represent the same values in the same methods. The one hot encoding is less efficient, requiring n bits instead of log n bits.&#xA;More than that, if we are not aware that the n features in the on hot encoding are exclusive, our &lt;a href=&quot;https://en.wikipedia.org/wiki/VC_dimension&quot; rel=&quot;noreferrer&quot;&gt;vc dimension&lt;/a&gt; and our hypothesis set are larger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, one might wonder why use one hot encoding in the first place?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that in the single feature representation and the log representation we might use wrong deductions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a single feature representation the algorithm might assume order. Usually the encoding is arbitrary and the value 3 is as far for 3 as from 8. However, the algorithm might treat the feature as a numeric feature and come up with rules like &quot;f &amp;lt; 4&quot;. Here you might claim that if the algorithm found such a rule, it might be beneficial, even if not intended. While that might be true, small data set, noise and other reason to have a data set that mis represent the underlying distribution might lead to false rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same can happen in logarithmic representation (e.g., having rules like &quot;third bit is on). Here we are likely to get more complex rules, all unintended and sometimes misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, we should had identical representations, leading to identical results in ideal world. However, in some cases the less efficient representation can lead to worse results while on other cases the badly deduce rules can lead to worse results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if the values are indeed very distinct in behaviour, the algorithm will probably won't deduce such rule and you will benefit from the more efficient representation. Many times it is hard to analyze it beforehand so what you did, trying both representations, is a good way to choose the proper one. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-04-02T11:20:00.593" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18061" PostTypeId="2" ParentId="17522" CreationDate="2017-04-02T21:38:58.977" Score="1" Body="&lt;p&gt;In addition to what @Icyblade suggested by citing the source code, I'd like to add a few things.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;dataset_name&lt;/code&gt; is an option that tells slim how to read the TFRecords located in the &lt;code&gt;dataset_dir&lt;/code&gt; folder.&lt;br&gt;&#xA;The default value &lt;code&gt;imagenet&lt;/code&gt; means that the TFRecords have to have this format: &lt;code&gt;train-00146-of-00168.tfrecord&lt;/code&gt; or &lt;code&gt;validation-00003-of-00019.tfrecord&lt;/code&gt;.&lt;br&gt;&#xA;The value &lt;code&gt;flowers&lt;/code&gt; means that slim expects the TFRecords to have the form &lt;code&gt;flowers_train_00146-of-00168.tfrecord&lt;/code&gt; or &lt;code&gt;flowers_validation_00003-of-00019.tfrecord&lt;/code&gt;. This peculiar way of formatting the tfrecords is defined &lt;a href=&quot;https://github.com/tensorflow/models/blob/546fd48ecb70635e5b086143c252649b217df59d/slim/datasets/download_and_convert_flowers.py#L102&quot; rel=&quot;nofollow noreferrer&quot;&gt;here in the source code&lt;/a&gt;.&lt;br&gt;&#xA;Different sample datasets have different formatting rules, e.g. &lt;a href=&quot;https://github.com/tensorflow/models/blob/546fd48ecb70635e5b086143c252649b217df59d/slim/datasets/download_and_convert_cifar10.py#L118&quot; rel=&quot;nofollow noreferrer&quot;&gt;here you can see cifar10&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Moreover, &lt;code&gt;dataset_split_name&lt;/code&gt; tells the converter how to name the&#xA;TFRecords files. For the flowers dataset they are &lt;code&gt;train&lt;/code&gt; and&#xA;&lt;code&gt;validation&lt;/code&gt;, for cifar10 they are &lt;code&gt;train&lt;/code&gt; and &lt;code&gt;test&lt;/code&gt;, etc.&lt;/li&gt;&#xA;&lt;li&gt;About &lt;code&gt;checkpoint_exclude_scopes&lt;/code&gt; and &lt;code&gt;trainable_scopes&lt;/code&gt;, to make a&#xA;long story short, these commands tell the training algorithm to&#xA;remove the final two layers of the network from the last checkpoint&#xA;available (either passed by the &lt;code&gt;checkpoint_path&lt;/code&gt; option or read from&#xA;the folder specified in &lt;code&gt;train_dir&lt;/code&gt;), and make it retrain the CNN on&#xA;them, but using the images you provide inside the TFRecords file.&#xA;Thus, using these options means to perform a &lt;em&gt;retrain&lt;/em&gt; -or, as the&#xA;slim devs call it in the README, a &lt;em&gt;fine-tuning&lt;/em&gt; from an existing&#xA;checkpoint (link in the comments)- of a CNN on another set of images.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To give you an hint about how to build your own TFRecord converter/creator, I'd suggest to base it on the download_and_convert_flowers.py script, since the input dataset is regular, with the images divided in categories, where each category is a single folder.&lt;br&gt;&#xA;Moreover, I'd extend it by adding the option to read input png images (the flowers pictures are only jpg, and thus the script doesn't need this conversion, but your future dataset may). Start by de-assembling it piece by piece and remove the hard-coded information related to the flowers dataset, to make it less specific.&lt;/p&gt;&#xA;" OwnerUserId="30252" LastActivityDate="2017-04-02T21:38:58.977" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18062" PostTypeId="1" CreationDate="2017-04-02T23:37:59.717" Score="1" ViewCount="1218" Body="&lt;p&gt;I am trying to reduce the dimensionality of topic vectors (300, 1) to a two dimensional space. This has been done with various methods (e.g. t-SNE and autoencoders). A published example of reducing topic vectors to 2-dim space can be found &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/science.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train a test autoencoder, I took the top 10k words from Google's word vector model and tried to encode/decode them. &#xA;My autoencoder (300, 100, 2) with tanh activation functions (due to the nature of the vector elements of the word2vec model) seems to learn the vectors quickly (the loss function quickly goes negative). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I compare the closest words based on the 2-dim reduction the results from the autoencoder (cosine and euclidean distance of the word presentations) I notice that they don't match the similar words provided from the original w2v model (output of &lt;code&gt;most_similar&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most similar words based on the autoencoded representation (2-dim, calculated via cosine distance). Results look random and it seems that the original distances didn't get preserved. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; extract_from_embed('car')  # extract method for my own embedding&#xA;[(football', 0.91979183998553593), (u'\xa9', 0.9515135906035519), (u'Thank', 0.96150527440098321), (u'innings', 0.96893565858480013), (u'thank', 0.9699300787802696), (u'balls', 0.97004978663463826), (u'Admission', 0.97050270191042776), (u'thanked', 0.97098601610349322), (u'Announces', 0.97186348679591361), (u'drills', 0.97214077185079129)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Word2Vec most similar words based on 300, 1 vectors&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; word_vectors.most_similar(positive=['car'])  # std gensim method&#xA;[(u'vehicle', 0.7821096777915955),&#xA; (u'cars', 0.7423831224441528),&#xA; (u'SUV', 0.7160963416099548),&#xA; (u'truck', 0.6735789775848389),&#xA; (u'Car', 0.667760968208313),&#xA; ...&#xA; (u'automobile', 0.5838367938995361)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Is there a way to &quot;preserve&quot; the relative distances between vectors when I used an autoencoder?&lt;/em&gt;&lt;/strong&gt; I understand that the absolute distances differ between vectors from a 2 and 300 dim space. But can I preserve the relative distances between the vectors? Is this even possible?&lt;/p&gt;&#xA;" OwnerUserId="30683" LastEditorUserId="30683" LastEditDate="2017-04-03T04:23:03.463" LastActivityDate="2017-04-03T07:09:36.690" Title="Retain similarity distances when using an autoencoder for dimensionality reduction" Tags="|word2vec|word-embeddings|autoencoder|tsne|embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18063" PostTypeId="2" ParentId="18045" CreationDate="2017-04-03T01:58:09.023" Score="4" Body="&lt;p&gt;My link in the comment has useful advice. I'd like to emphasis:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This is a very well known fact in statistics&lt;/li&gt;&#xA;&lt;li&gt;Large sample size is &lt;strong&gt;good&lt;/strong&gt;. There's nothing wrong with more and better quality data.&lt;/li&gt;&#xA;&lt;li&gt;It's silly to split up the data in your second point. You are voluntarily giving away information and statistical power. People do that for training an unbiased model, but &lt;strong&gt;not&lt;/strong&gt; to reduce sample size for the p-values.&lt;/li&gt;&#xA;&lt;li&gt;There's no reason to switch to another model. The standard error in your logistic model should be small, which makes it ideal for predictive modelling. Your model should be robust and stable.&lt;/li&gt;&#xA;&lt;li&gt;You should do your analysis on &lt;strong&gt;effects size&lt;/strong&gt; if you have a large sample. Read @Sympa's answer in the link I post. What's your magnitude of the effect size? What's the relative effects?&lt;/li&gt;&#xA;&lt;li&gt;The threshold you have for your first point is domain specific. If you write a paper, you should know it very well. You could compare your data with the literature. Simple descriptive statistics like percentage and how it compare with the standard deviation (which is independent to sample size) would be useful.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please also take a look at &lt;a href=&quot;https://stats.stackexchange.com/questions/125750/sample-size-too-large&quot;&gt;https://stats.stackexchange.com/questions/125750/sample-size-too-large&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a long but excellent article on Bayes factor and p-value:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-with-bayes-factors-a-miracle-cure-for-the-replicability-crisis-in-psychological-science/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-with-bayes-factors-a-miracle-cure-for-the-replicability-crisis-in-psychological-science/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I should draw to your attention the following paragraph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The more interesting argument against p-value is not that significant results in large studies are type-I errors, but that these results are practically meaningless. To make this point, statistics books often distinguish statistical significance and practical significance and warn that statistically significant results in large samples may have little practical significance. This warning was useful in the past when researchers would only report p-values (e.g., women have higher verbal intelligence than men, p &amp;lt; .05). The p-value says nothing about the size of the effect. When only the p-value is available, it makes sense to assume that significant results in smaller samples are larger because only large effects can be significant in these samples. However, large effects can also be significant in large samples and large effects in small studies can be inflated by sampling error. &lt;strong&gt;Thus, the notion of practical significance is outdated and should be replaced by questions about effect sizes.&lt;/strong&gt; Neither p-values nor Bayes-Factors provide information about the size of the effect or the practical implications of a finding.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The reviewer was correct. Neither p-value nor the bayes-factor reveals anything about the effect sizes.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-04-04T07:58:20.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18064" PostTypeId="1" CreationDate="2017-04-03T06:29:59.597" Score="-1" ViewCount="55" Body="&lt;p&gt;I am working on the prediction of the behaviour of a new well in a map.&#xA;The data that I have is a map in a form of a grid and its properties (9 of them), and a set of wells with the position of each well in the map. And I have 4 outputs for each well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say i have 200 000 cells in my grid, it means that i have at least 9*200 000 features in my input, apart from the high dimension, these attributes are the same for all the wells, but at the same time they are important and I need to use them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is how can I use the map and its properties to choose my input features without being redundant for each well?  &lt;/p&gt;&#xA;" OwnerUserId="30688" LastActivityDate="2017-04-04T17:28:42.573" Title="How do I create my features" Tags="|machine-learning|feature-selection|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18065" PostTypeId="2" ParentId="18062" CreationDate="2017-04-03T06:58:40.713" Score="2" Body="&lt;p&gt;No it is not possible to preserve relative distances when reducing dimensions for arbitrary data. This is not due to a property of auto-encoders compared to e.g. PCA or T-SNE. It is due to geometry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see this relatively easily by considering a reduction of dimensions from 3 to 2, and examining a tetrahedron where all four corner points are 1 unit apart. There is no two-dimensional shape that can place four points mutually equidistant (unless perhaps you consider non-Euclidean spaces). It should be clear that relative distances near those corner vertices would also be affected, and thus this special shape demonstrates a general property of dimension reduction.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-03T07:09:36.690" LastActivityDate="2017-04-03T07:09:36.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18066" PostTypeId="1" AcceptedAnswerId="18169" CreationDate="2017-04-03T08:56:19.450" Score="11" ViewCount="4582" Body="&lt;p&gt;By entire data I mean train + test + validation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once I have fixed my hyperparameter using the validation data, and choose the model using the test data, won't it be better to have a model trained on the entire data so that the parameters are better trained rather than having the model trained on just train data &lt;/p&gt;&#xA;" OwnerUserId="30693" LastEditorUserId="11097" LastEditDate="2017-04-12T08:55:09.240" LastActivityDate="2023-11-09T21:55:39.537" Title="Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data?" Tags="|machine-learning|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18067" PostTypeId="1" AcceptedAnswerId="18068" CreationDate="2017-04-03T08:56:28.883" Score="-2" ViewCount="377" Body="&lt;p&gt;&lt;strong&gt;New update:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand PCA components ensure we select variables responsible for high variance, but I would like to know how to extract key variables responsible only for high variance through PCA components.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, a simple example would help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Implementing PCA for visualizing after Kmeans clustering&#xA;&#xA;`# Interpret 3 cluster solution&#xA;model3=KMeans(n_clusters=3)&#xA;model3.fit(clus_train)&#xA;clusassign=model3.predict(clus_train)&#xA;# plot clusters&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;'''The new variables, called canonical variables, are ordered in terms&#xA;  of the proportion of variance and the clustering variables that is&#xA;  accounted for by each of the canonical variables.  So the first&#xA;  canonical variable will count for the largest proportion of the&#xA;  variance. The second canonical variable will account for the next&#xA;  largest proportion of variance, and so on. Usually, the majority of&#xA;  the variance in the clustering variables will be accounted for by the&#xA;  first couple of canonical variables and those are the variables that&#xA;  we can plot. '''&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.decomposition import PCA&#xA;pca_2 = PCA(2) # Selecting 2 components&#xA;plot_columns = pca_2.fit_transform(clus_train)&#xA;plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model3.labels_,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Observations are more spread out indicating less correlation among the&#xA; observations and higher within cluster variance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.xlabel('Canonical variable 1')&#xA;plt.ylabel('Canonical variable 2')&#xA;plt.title('Scatterplot of Canonical Variables for 3 Clusters')&#xA;plt.show()`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27469" LastEditorUserId="29169" LastEditDate="2019-06-10T17:32:54.473" LastActivityDate="2019-06-10T17:32:54.473" Title="How is PCA is different from SubSpace clustering and how do we extract variables responsible for the first PCA component?" Tags="|clustering|visualization|pca|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="18068" PostTypeId="2" ParentId="18067" CreationDate="2017-04-03T09:07:03.357" Score="1" Body="&lt;p&gt;Reducing the dimensionality of a dataset with PCA does not only benefits humans trying to look at the data in a graspable number of dimensions. It is also useful for machine learning algorithms to be trained on a subset of dimensions. Both to reduce the complexity of the data and the computational cost of training such machine learning model.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-04-03T09:07:03.357" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18069" PostTypeId="2" ParentId="18067" CreationDate="2017-04-03T09:31:01.730" Score="0" Body="&lt;p&gt;PCA is a very common technique, so you might want to google around. PCA is awfully common for data visualisation, but it has many other uses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if you want to fit a linear regression on average income. Now, you have collected 500+ predictors, but lots of them are correlated like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How much the person pays tax last year&lt;/li&gt;&#xA;&lt;li&gt;How much the person pays tax the year before&lt;/li&gt;&#xA;&lt;li&gt;How much the person pays tax three years ago&lt;/li&gt;&#xA;&lt;li&gt;....&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those predictors are highly correlated and might present modelling problems in your linear model. A very common technique is to use PCA to reduce into a set of reduced orthogonal principal components. You can then use those components for building your model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/22665/how-to-use-principal-components-as-predictors-in-glm&quot;&gt;https://stats.stackexchange.com/questions/22665/how-to-use-principal-components-as-predictors-in-glm&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2017-04-03T09:31:01.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18071" PostTypeId="1" AcceptedAnswerId="18078" CreationDate="2017-04-03T10:53:31.840" Score="1" ViewCount="1411" Body="&lt;p&gt;I have been working on this code for a while and it gave me a lot of headache before I got it to work. It basically tries to use the mnist dataset to classify handwritten digits. I am not using the prepackaged mnist in TensorFlow because I want to learn preprocessing the data myself and for deeper understanding of TensorFlow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its finally working but I would love it if someone with expertise could take a look at it and tell me what they think and if the results its producing are actually real stats or if its overfitting or not learning at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's giving me accuracy between 83% and 91% from the test dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;the dataset I'm using is from &lt;a href=&quot;https://pjreddie.com/projects/mnist-in-csv/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pjreddie.com/projects/mnist-in-csv/&lt;/a&gt; basically the two links on top of the page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;sess = tf.Session()&#xA;from sklearn import preprocessing&#xA;import matplotlib.pyplot as plt&#xA;with tf.Session() as sess:&#xA;    # lets load the file&#xA;    train_file = 'mnist_train.csv'&#xA;    test_file = 'mnist_test.csv'&#xA;    #train_file = 'mnist_train_small.csv'&#xA;    #test_file = 'mnist_test_small.csv'&#xA;&#xA;    train = np.loadtxt(train_file, delimiter=',')&#xA;    test = np.loadtxt(test_file, delimiter=',')&#xA;&#xA;    x_train = train[:,1:785]&#xA;    y_train = train[:,:1]&#xA;&#xA;    x_test = test[:,1:785]&#xA;    y_test = test[:,:1]&#xA;    print(x_test.shape)&#xA;&#xA;    # lets normalize the data&#xA;    def normalize(input_data):&#xA;        minimum = input_data.min(axis=0)&#xA;        maximum = input_data.max(axis=0)&#xA;        #normalized = (input_data - minimum) / ( maximum - minimum )&#xA;        normalized = preprocessing.normalize(input_data, norm='l2')&#xA;        return normalized&#xA;&#xA;    # convert to a onehot array &#xA;    def one_hot(input_data):&#xA;        one_hot = []&#xA;        for item in input_data:&#xA;            if item == 0.:&#xA;                one_h = [1.,0.,0.,0.,0.,0.,0.,0.,0.,0.]&#xA;            elif item == 1.:&#xA;                one_h = [0.,1.,0.,0.,0.,0.,0.,0.,0.,0.]&#xA;            elif item == 2.:&#xA;                one_h = [0.,0.,1.,0.,0.,0.,0.,0.,0.,0.]&#xA;            elif item == 3.:&#xA;                one_h = [0.,0.,0.,1.,0.,0.,0.,0.,0.,0.]&#xA;            elif item == 4.:&#xA;                one_h = [0.,0.,0.,0.,1.,0.,0.,0.,0.,0.]&#xA;            elif item == 5.:&#xA;                one_h = [0.,0.,0.,0.,0.,1.,0.,0.,0.,0.]&#xA;            elif item == 6.:&#xA;                one_h = [0.,0.,0.,0.,0.,0.,1.,0.,0.,0.]&#xA;            elif item == 7.:&#xA;                one_h = [0.,0.,0.,0.,0.,0.,0.,1.,0.,0.]&#xA;            elif item == 8.:&#xA;                one_h = [0.,0.,0.,0.,0.,0.,0.,0.,1.,0.]&#xA;            elif item == 9.:&#xA;                one_h = [0.,0.,0.,0.,0.,0.,0.,0.,0.,1.]&#xA;&#xA;            one_hot.append(one_h)&#xA;        one_hot = np.array(one_hot)&#xA;        #one_hot = one_hot.reshape(len(one_hot),10,1)&#xA;        #one_hot = one_hot.reshape(len(one_hot), 7,1)&#xA;        #return tf.constant([one_hot])&#xA;        return one_hot&#xA;    def one_hot_tf(val):&#xA;        indices = val&#xA;        depth = 10&#xA;        on_value = 1.0&#xA;        off_value = 0.0&#xA;        axis = -1&#xA;        oh = tf.one_hot(indices, depth,&#xA;                   on_value=on_value, off_value=off_value,&#xA;                   axis=axis, dtype=tf.float32,&#xA;                   name='ONEHOT')&#xA;        return (oh)&#xA;    x_train = normalize(x_train)&#xA;    x_test =  normalize(x_test)&#xA;    #    x_train = sess.run(tf.convert_to_tensor(x_train))&#xA;    #    x_test =  sess.run(tf.convert_to_tensor(x_test))&#xA;&#xA;    '''&#xA;    data_initializer = tf.placeholder(dtype=x_train.dtype,&#xA;                                        shape=x_train.shape)&#xA;    label_initializer = tf.placeholder(dtype=x_test.dtype,&#xA;                                         shape=x_test.shape)&#xA;    x_train= sess.run(tf.Variable(data_initializer, trainable=False, collections=[]))&#xA;    x_test = sess.run(tf.Variable(label_initializer, trainable=False, collections=[]))&#xA;    '''&#xA;&#xA;&#xA;    y_test =  one_hot(y_test)&#xA;    y_train =  one_hot(y_train)&#xA;    print(y_test[:5])&#xA;    #   y_test =  sess.run(one_hot_tf(y_test))&#xA;    #   y_train =  sess.run(one_hot_tf(y_train))&#xA;&#xA;&#xA;    # define the parameters&#xA;    input_nodes = 784&#xA;    output_nodes = 10&#xA;    hl1_nodes = 500&#xA;    hl2_nodes = 500&#xA;    hl3_nodes = 500&#xA;    epochs = 10&#xA;    x = tf.placeholder(tf.float32, [None, input_nodes])&#xA;    y = tf.placeholder(tf.float32)&#xA;&#xA;    # graphing&#xA;    loss_rate = []&#xA;&#xA;&#xA;    def nn(data):&#xA;        layer1 = {'w':tf.Variable(tf.random_normal([input_nodes, hl1_nodes])),&#xA;                  'b':tf.Variable(tf.random_normal([hl1_nodes]))}&#xA;        layer2 = {'w':tf.Variable(tf.random_normal([hl1_nodes, hl2_nodes])),&#xA;                  'b':tf.Variable(tf.random_normal([hl2_nodes]))}&#xA;        layer3 = {'w':tf.Variable(tf.random_normal([hl2_nodes, hl3_nodes])),&#xA;                  'b':tf.Variable(tf.random_normal([hl3_nodes]))}&#xA;        output_layer = {'w':tf.Variable(tf.random_normal([hl3_nodes, output_nodes])),&#xA;                  'b':tf.Variable(tf.random_normal([output_nodes]))}&#xA;&#xA;        l1 = tf.add(tf.matmul(data, layer1['w']), layer1['b'])&#xA;        l1 = tf.nn.relu(l1)&#xA;&#xA;        l2 = tf.add(tf.matmul(l1, layer2['w']), layer2['b'])&#xA;        l2 = tf.nn.relu(l2)&#xA;&#xA;        l3 = tf.add(tf.matmul(l2, layer3['w']), layer3['b'])&#xA;        l3 = tf.nn.relu(l3)&#xA;&#xA;        output = tf.add(tf.matmul(l3, output_layer['w']), output_layer['b'])&#xA;&#xA;        return(output)&#xA;&#xA;&#xA;    def train(x):&#xA;        prediction = nn(x)&#xA;        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=y))&#xA;        optimizer = tf.train.GradientDescentOptimizer(0.001).minimize(loss)&#xA;&#xA;        init = tf.global_variables_initializer()&#xA;        sess.run(init)&#xA;&#xA;        for epoch in range(epochs):&#xA;            epochloss = 0&#xA;            batch_size = 10&#xA;            batches = 0&#xA;            for batch in range(int(len(x_train)/batch_size)):&#xA;                next_batch = batches+batch&#xA;                _, c = sess.run([optimizer, loss], feed_dict={x:x_train[batches:next_batch, :], y:y_train[batches:next_batch, :]})&#xA;                epochloss = epochloss + c&#xA;                batches += batch&#xA;                loss_rate.append(c)&#xA;&#xA;            print(&quot;Epoch &quot;, epoch, &quot; / &quot;, epochs, &quot; - Loss &quot;, epochloss)&#xA;&#xA;        correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))&#xA;        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))&#xA;        print(&quot;Accuracy : &quot;, accuracy.eval({x:x_test, y:y_test}))&#xA;&#xA;&#xA;    train(x)&#xA;&#xA;    plt.plot(loss_rate)&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of 3 different runs are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=========== RESTART: /Users/macbookpro/Desktop/AI/tf/OWN/test3.py ===========&#xA;(10000, 784)&#xA;[[ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]&#xA; [ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]]&#xA;Epoch  0  /  5  - Loss  nan&#xA;Epoch  1  /  5  - Loss  nan&#xA;Epoch  2  /  5  - Loss  nan&#xA;Epoch  3  /  5  - Loss  nan&#xA;Epoch  4  /  5  - Loss  nan&#xA;Accuracy :  0.9053&#xA;&#xA;=========== RESTART: /Users/macbookpro/Desktop/AI/tf/OWN/test3.py ===========&#xA;(10000, 784)&#xA;[[ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]&#xA; [ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]]&#xA;Epoch  0  /  5  - Loss  nan&#xA;Epoch  1  /  5  - Loss  nan&#xA;Epoch  2  /  5  - Loss  nan&#xA;Epoch  3  /  5  - Loss  nan&#xA;Epoch  4  /  5  - Loss  nan&#xA;Accuracy :  0.8342&#xA;&#xA;=========== RESTART: /Users/macbookpro/Desktop/AI/tf/OWN/test3.py ===========&#xA;(10000, 784)&#xA;[[ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]&#xA; [ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]&#xA; [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.]]&#xA;Epoch  0  /  5  - Loss  nan&#xA;Epoch  1  /  5  - Loss  nan&#xA;Epoch  2  /  5  - Loss  nan&#xA;Epoch  3  /  5  - Loss  nan&#xA;Epoch  4  /  5  - Loss  nan&#xA;Accuracy :  0.9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;---Update---&#xA;I found the answer in rewriting the code as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;&#xA;sess = tf.Session()&#xA;&#xA;file = &quot;mnist_train.csv&quot;&#xA;data = np.loadtxt(file, delimiter=',')&#xA;&#xA;&#xA;y_vals = data[:,0:1]&#xA;x_vals = data[:,1:785]&#xA;&#xA;seed = 3&#xA;tf.set_random_seed(seed)&#xA;np.random.seed(seed)&#xA;batch_size = 90&#xA;&#xA;# split into 80/20 datasets, normalize between 0:1 with min max scaling&#xA;train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False)&#xA;# up there we chose randomly 80% of the data&#xA;test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))&#xA;# up we chose the remaining 20% &#xA;print(test_indices)&#xA;&#xA;x_vals_train = x_vals[train_indices]&#xA;x_vals_test = x_vals[test_indices]&#xA;y_vals_train = y_vals[train_indices]&#xA;y_vals_test = y_vals[test_indices]&#xA;&#xA;def normalize_cols(m):&#xA;    col_max = m.max(axis=0)&#xA;    col_min = m.min(axis=0)&#xA;    return (m-col_min)/(col_max - col_min)&#xA;x_vals_train = np.nan_to_num(normalize_cols(x_vals_train))&#xA;x_vals_test = np.nan_to_num(normalize_cols(x_vals_test))&#xA;&#xA;# function that initializes the weights and the biases &#xA;def init_weight(shape, std_dev):&#xA;    weight = tf.Variable(tf.random_normal(shape, stddev=std_dev))&#xA;    return(weight)&#xA;&#xA;def init_bias(shape, std_dev):&#xA;    bias= tf.Variable(tf.random_normal(shape, stddev=std_dev))&#xA;    return(bias)&#xA;&#xA;# initialize placeholders. &#xA;x_data = tf.placeholder(shape=[None, 784], dtype=tf.float32)&#xA;y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)&#xA;&#xA;&#xA;# the fully connected layer will be used three times for all three hidden layers&#xA;def fully_connected(input_layer, weights, biases):&#xA;    layer = tf.add(tf.matmul(input_layer, weights), biases)&#xA;    return (tf.nn.relu(layer))&#xA;&#xA;# Now create the model for each layer and the output layer.&#xA;# we will initialize a weight matrix, bias matrix and the fully connected layer&#xA;# for this, we will use hidden layers of size 500, 500, and 10&#xA;&#xA;'''&#xA;This will mean many variables variables to fit. This is because between the data and the first hidden layer we have &#xA;784*500+500 = 392,500 variables to change.&#xA;continuing this way we will have end up with how many variables we have overall to fit&#xA;'''&#xA;&#xA;# create first layer (500 hidden nodes)&#xA;weight_1 = init_weight(shape=[784,500], std_dev=10.0)&#xA;bias_1 = init_bias(shape=[500], std_dev=10.0)&#xA;layer_1 = fully_connected(x_data, weight_1, bias_1)&#xA;&#xA;# create second layer (5-- hidden nodes)&#xA;weight_2 = init_weight(shape=[500,500], std_dev=10.0)&#xA;bias_2 = init_bias(shape=[500], std_dev=10.0)&#xA;layer_2 = fully_connected(layer_1, weight_2, bias_2)&#xA;&#xA;# create third layer (10 hidden nodes)&#xA;weight_3 = init_weight(shape=[500,10], std_dev=10.0)&#xA;bias_3 = init_bias(shape=[10], std_dev=10.0)&#xA;layer_3 = fully_connected(layer_2, weight_3, bias_3)&#xA;&#xA;# create output layer (1 output value)&#xA;weight_4 = init_weight(shape=[10,1], std_dev=10.0)&#xA;bias_4 = init_bias(shape=[1], std_dev=10.0)&#xA;final_output = fully_connected(layer_3, weight_4, bias_4)&#xA;&#xA;&#xA;# define the loss function and the optimizer and initializing the model&#xA;loss = tf.reduce_mean(tf.abs(y_target - final_output))&#xA;optimizer = tf.train.AdamOptimizer(0.05)&#xA;train_step = optimizer.minimize(loss)&#xA;&#xA;init = tf.global_variables_initializer()&#xA;sess.run(init)&#xA;&#xA;# we will now train our model 10 times, store train and test los, select a random batch size, &#xA;# and print the status every 1 generation&#xA;&#xA;# initalize the loss vectors&#xA;loss_vec = []&#xA;test_loss = []&#xA;for i in range(10):&#xA;    # choose random indices for batch selection&#xA;    rand_index = np.random.choice(len(x_vals_train), size=batch_size)&#xA;    # get random batch&#xA;    rand_x = x_vals_train[rand_index]&#xA;    #rand_y = np.transpose(y_vals_train[rand_index])&#xA;    rand_y = y_vals_train[rand_index] #???????????&#xA;    # run the training step&#xA;    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})&#xA;    # get and store train loss&#xA;    temp_loss = sess.run(loss, feed_dict={x_data:rand_x, y_target:rand_y})&#xA;    loss_vec.append(temp_loss)&#xA;    # get and store test loss &#xA;    #test_temp_loss = sess.run(loss, feed_dict={x_data:x_vals_test, y_target:np.transpose([y_vals_test])})&#xA;    test_temp_loss = sess.run(loss, feed_dict={x_data:x_vals_test, y_target:y_vals_test}) #???????&#xA;    test_loss.append(test_temp_loss)&#xA;    if(i+1) %1==0:&#xA;        print('Generation: '+str(i+1)+&quot;. Loss = &quot;+str(temp_loss))&#xA;&#xA;plt.plot(loss_vec, 'k-', label='Train Loss')&#xA;plt.plot(test_loss, 'r--', label='Test Loss')&#xA;plt.title('Loss Per generation ')&#xA;plt.xlabel('Generation')&#xA;plt.ylabel('Loss')&#xA;plt.legend(loc='upper right')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I commented most of it just so if someone stumbles here and needs some help they can understand whats going on.&lt;/p&gt;&#xA;" OwnerUserId="30639" LastEditorUserId="30639" LastEditDate="2017-04-03T19:21:32.283" LastActivityDate="2017-04-03T19:21:32.283" Title="MNIST Deep Neural Network using TensorFlow" Tags="|python|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18073" PostTypeId="2" ParentId="16835" CreationDate="2017-04-03T12:42:14.497" Score="10" Body="&lt;p&gt;There are a couple of good threads on Reddit right now (&lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/5w3q74/d_so_pytorch_vs_tensorflow_whats_the_verdict_on/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/5otdwl/d_tensorflow_vs_pytorch_why_is_dynamic_better/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic graphs in &lt;em&gt;PyTorch&lt;/em&gt; is a 'top down design principle', whereas &lt;em&gt;TensorFlow Fold&lt;/em&gt; is bolted on to the original Tensorflow framework, so if you're doing anything reasonably complicated with &lt;em&gt;Tensorflow Fold&lt;/em&gt; you're probably going to end up doing a lot more hacking around than if you're using &lt;em&gt;PyTorch&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="378" LastActivityDate="2017-04-03T12:42:14.497" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18074" PostTypeId="2" ParentId="18066" CreationDate="2017-04-03T12:42:18.700" Score="1" Body="&lt;p&gt;Yes you can.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As test data supposed to come from similiar distribution to train data, you won't break your model. If you have trained model properely, then you will notice no significant change(except better accuracy metric on previous test / validation data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it's rareraly true that test data comes from precisely same distribution as train data, so in real application case scenario you may get better generalizability of your model.&lt;/p&gt;&#xA;" OwnerUserId="12919" LastActivityDate="2017-04-03T12:42:18.700" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18075" PostTypeId="1" CreationDate="2017-04-03T13:29:04.643" Score="0" ViewCount="105" Body="&lt;p&gt;I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Iris&quot; rel=&quot;nofollow noreferrer&quot;&gt;Iris Data Set&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using that data just as a test to check if my implimentation is working.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.3563204029871746&#xA;0.34553220619664271&#xA;0.36660774391053058&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I run it for 2000 iterations I get this;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.33748041999368034&#xA;0.33555095097164173&#xA;0.33929502380142113&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it matters I'm using a logistic activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Pseudocode&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok here's my code;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;**Sub DataInput**&#xA;Load in the iris text file&#xA;split file into lines&#xA;redim x(#lines)&#xA;redim y(#lines)&#xA;for each line&#xA;  datain = split line into columns (comma as separator)&#xA;  redim x(line #)(3)&#xA;  x(0) = datain(0)/7.9&#xA;  x(1) = datain(1)/4.4&#xA;  x(2).... etc.&#xA;&#xA;  redim y(line #)(2)&#xA;  y(0) = 0&#xA;  y(1) = 0&#xA;  y(2) = 0&#xA;&#xA;  select case datain(5) 'selects which type of flower&#xA;    case is flower1&#xA;      y(0) = 1&#xA;    case is flower2&#xA;      y(1) = 1&#xA;    case is flower3&#xA;      y(2) = 1&#xA;  end select&#xA;next line&#xA;**end sub**&#xA;&#xA;**sub initialise**&#xA;'sets up array sizes for all required values&#xA;'sets all weights to random&#xA;**end sub**&#xA;&#xA;**sub Train**&#xA;Do until cost function is below threshold&#xA;set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer)&#xA;&#xA;for i = 0 to number of training examples&#xA;  call sub ForwardPropogate(i)&#xA;  call sub BackwardPropogate(i)&#xA;end for&#xA;&#xA;for l = 0 to number of layers -1&#xA;  for i = 0 to number of neuron in next layer&#xA;    for j = 1 to number of neurons in this layer&#xA;       CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j)&#xA;    end for&#xA;    CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0)&#xA;  end for&#xA;end for&#xA;&#xA;for l = 0 to number of layers -1&#xA;  theta(l) = theta(l) - alpha*CapD(l)&#xA;end for&#xA;loop&#xA;&#xA;**end sub**&#xA;&#xA;**Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated&#xA;&#xA;a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n&#xA;a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron&#xA;for i = 1 to number of layers -1&#xA;  z(i) = theta(i-1) * a(i-1)&#xA;  a(i) = g(z(i)) ' g is the activation function&#xA;  a(i) = leadingBias(a(i))&#xA;end for&#xA;z(number of layers) = theta(number of layers -1) * a(number of layers -1)&#xA;a(number of layers) = g(z(number of layers))&#xA;&#xA;**end sub**&#xA;&#xA;**Sub BackwardPropogate**(byval exNum as int)&#xA;delta(number of layers) = a(number of layers) - y(exNum)&#xA;for i = number of layers - 1 to 1 step -1&#xA;  delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i))    g' is the derivative of the activation function&#xA;end for&#xA;for i = 0 to number of layers -1&#xA;  DR(i) = DR(i) +delta(i+1) * transpose(a(i))&#xA;end for&#xA;&#xA;**end sub**&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14586" LastEditorUserId="14586" LastEditDate="2017-04-04T05:00:57.357" LastActivityDate="2017-04-04T16:16:15.493" Title="My neural network 'learns' to estimate each class equally" Tags="|neural-network|" AnswerCount="1" CommentCount="8" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18076" PostTypeId="2" ParentId="18066" CreationDate="2017-04-03T14:24:40.423" Score="1" Body="&lt;p&gt;The answer for this question depends on the training algorithm ( technology) that you use. For example, I have seen some approaches in ensemble classification where training and validation ( but not testing) sets are combined at the end. &#xA;It is very important to know that even validation is used mainly to decide the hyper parameters some of these hyper parameters can be a function of the used data for training. For example, in DNN validation used to know when to stop, because overfitting can happen as a result of keeping tuning the parameters ( weights) of the network, we need a way to know when to stop. Without the validation set you will be walking blindly in the training process. In the other hand, if you use exactly the same number of iterations as specified before, there is a high prob that you will not gain from these additional samples.&#xA;Testing set should not be touched at all, as indicated above without the testing set you will have no method to evaluate your model. This is gambling, you CAN NOT deliver any model or solution without the estimation of its accuracy on the true data distribution ( which represented by the testing data) .   &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-04-03T14:24:40.423" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18077" PostTypeId="2" ParentId="18045" CreationDate="2017-04-03T14:47:03.953" Score="1" Body="&lt;p&gt;How easy is it to formulate a null-hypothesis / random data? If it is possible you can just see how often a certain p-value is calculated. By creating enough pseudo-experiments you can arrive at a probability density function of p-values which you can use to compare your measured ones. You can do this for one variable alone but also for several variables combined. Example: How likely is it to find 10 variables with a p-value &amp;lt; x when I have 100 variables. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-04-03T14:47:03.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18078" PostTypeId="2" ParentId="18071" CreationDate="2017-04-03T14:57:46.453" Score="2" Body="&lt;p&gt;Given that you have such high error on the test set and have so many hidden layers/nodes, it's quite possible that your model is overfitting. Try using dropout or weight decay to regularize the weights of your network.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-03T14:57:46.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18079" PostTypeId="1" CreationDate="2017-04-03T14:57:50.457" Score="1" ViewCount="455" Body="&lt;p&gt;working through Dr. Ng's coursera course&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning/supplement/nhzyF/cost-function&quot; rel=&quot;nofollow noreferrer&quot;&gt;cost function intro&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;p&gt;basically, you want to minimize the difference squared between your guess and the actual value&lt;/p&gt;&#xA;&lt;p&gt;cost function for SVM: &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/supplement/pSe2X/lecture-slides&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; or &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/3eNnh/mathematics-behind-large-margin-classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;he says the cost function is now the norm of the vector representing the hyperparameters: sqrt(theta_1^2 + theta_2^2... )&lt;/p&gt;&#xA;&lt;p&gt;Why is that the case?&lt;/p&gt;&#xA;" OwnerUserId="21232" LastEditorUserId="26019" LastEditDate="2020-08-20T17:07:40.983" LastActivityDate="2020-08-20T17:07:40.983" Title="why is SVM cost function the norm of the hyperplane parameters, and not the mean square error?" Tags="|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18080" PostTypeId="1" AcceptedAnswerId="18092" CreationDate="2017-04-03T16:30:20.660" Score="0" ViewCount="1349" Body="&lt;p&gt;I am trying to start the learning of a cnn network which has 72 input and one output being a vector of length 24 stating the a class for each third input 72/24 = 3. There are 145 classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is how i've designed the network currently passed my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print &quot;After test_output/input&quot;&#xA;&#xA;print &quot;Length:&quot;&#xA;print len(data_train_input)&#xA;print len(data_train_output)&#xA;&#xA;print len(data_test_input)&#xA;print len(data_test_output)&#xA;&#xA;print &quot;Type;&quot;&#xA;print type(data_train_input[0])&#xA;print type(data_train_output[0])&#xA;&#xA;print &quot;Size [0]&quot;&#xA;print data_train_input[0].shape&#xA;print data_train_output[0].shape&#xA;&#xA;list_of_input = [Input(shape = (78,3)) for i in range(72)]&#xA;list_of_conv_output = []&#xA;list_of_max_out = []&#xA;for i in range(72):&#xA;    list_of_conv_output.append(Conv1D(filters = 32 , kernel_size = 6 , padding = &quot;same&quot;, activation = 'relu')(list_of_input[i]))&#xA;    list_of_max_out.append(MaxPooling1D(pool_size=3)(list_of_conv_output[i]))&#xA;&#xA;merge = keras.layers.concatenate(list_of_max_out)&#xA;reshape = Reshape((-1,))(merge)&#xA;&#xA;dense1 = Dense(500, activation = 'relu')(reshape)&#xA;dense2 = Dense(250,activation = 'relu')(dense1)&#xA;dense3 = Dense(1 ,activation = 'softmax')(dense2)&#xA;&#xA;model = Model(inputs = list_of_input , outputs = dense3)&#xA;model.compile(loss=&quot;sparse_categorical_crossentropy&quot;, optimizer=&quot;adam&quot; , metrics = [metrics.sparse_categorical_accuracy])&#xA;&#xA;reduce_lr=ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1, mode='auto', epsilon=0.01, cooldown=0, min_lr=0.000000000000000000001)&#xA;stop  = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')&#xA;&#xA;print &quot;Train!&quot;&#xA;list_train_input = []&#xA;list_test_input = []&#xA;&#xA;for i in range(len(data_train_input)):&#xA;    list_train_input.append(data_train_input[i])&#xA;&#xA;for i in range(len(data_test_input)):&#xA;    list_test_input.append(data_test_input[i])&#xA;&#xA;hist_current = model.fit(x = [np.array(list_train_input[i]) for i in range(72)],&#xA;                    y = np.array(data_train_output),&#xA;                    shuffle=False,&#xA;                    validation_data=([np.array(list_test_input[i]) for i in range(72)], np.array(data_test_output)),&#xA;                    validation_split=0.1,&#xA;                    epochs=150000,&#xA;                    verbose=1,&#xA;                    callbacks=[reduce_lr,stop])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which generates this output: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;After test_output/input&#xA;Length:&#xA;9436&#xA;9417&#xA;1017&#xA;1035&#xA;Type;&#xA;&amp;lt;type 'numpy.ndarray'&amp;gt;&#xA;&amp;lt;type 'numpy.ndarray'&amp;gt;&#xA;Size [0]&#xA;(72, 78, 3)&#xA;(24,)&#xA;Train!&#xA;Traceback (most recent call last):&#xA;  File &quot;keras_cnn_phoneme_classification.py&quot;, line 382, in &amp;lt;module&amp;gt;&#xA;    model(train_input_data_interweawed_normalized, output_train, test_input_data_interweawed_normalized, output_test, test_name_interweawed_normalized)&#xA;  File &quot;keras_cnn_phoneme_classification.py&quot;, line 361, in model&#xA;    callbacks=[reduce_lr,stop])&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 1405, in fit&#xA;    batch_size=batch_size)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 1299, in _standardize_user_data&#xA;    exception_prefix='model target')&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 133, in _standardize_input_data&#xA;    str(array.shape))&#xA;ValueError: Error when checking model target: expected dense_3 to have shape (None, 1) but got array with shape (9417, 24)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why am i getting this error? &#xA;I tried changing the output size of the dense3 to many  different?  but is is it expecting?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:44:07.737" LastActivityDate="2021-02-16T02:44:07.737" Title="Incorrect output dimension?" Tags="|python|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18081" PostTypeId="1" AcceptedAnswerId="18147" CreationDate="2017-04-03T16:49:47.273" Score="2" ViewCount="8863" Body="&lt;p&gt;I have a binary classification task where all of my features are boolean (0 or 1).  I have been considering two possible supervised learning algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Logistic regression&lt;/li&gt;&#xA;&lt;li&gt;Gradient boosting with &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_stump&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision stumps&lt;/a&gt; (e.g., xgboost) and cross-entropy loss&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I understand how they work, it seems like these two might be equivalent.  Are they in fact equivalent?  Are there any reasons to choose one over the other?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In particular, here's why I'm thinking they are equivalent.  A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0.  This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where  $x_i$ is the value of feature $i$.  Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum.  Thus the total sum of the gradient boosted stumps can be expressed in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or equivalently, in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S = c_0 + \sum_{i=1}^n c_i x_i.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's exactly the form of a final logit for a logistic regression model.  That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2017-04-03T17:58:25.780" LastActivityDate="2017-04-05T21:18:02.030" Title="Gradient boosting vs logistic regression, for boolean features" Tags="|classification|logistic-regression|xgboost|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18082" PostTypeId="1" CreationDate="2017-04-03T16:55:19.947" Score="0" ViewCount="3485" Body="&lt;p&gt;I am self learning data analysis using (mostly) R. I'm currently on K-nearest neighbour topic, and I went through the process of normalizing- splitting the dataset to train/test and fitting the model and evaluating it, and it all went well. But, there is one fundamental thing I just don't understand: what is the point in classifying a known data? How do I predict the value of &lt;em&gt;NEW&lt;/em&gt; data? The outcome of knn() is a factor with labels, but after using it for evaluating my model (namely, the value of K I have chosen), what should come next? For example, in regression model I can use predict() to predict new data, but I do I predict (as opposed to just classify) using knn?&#xA;Thank!&lt;/p&gt;&#xA;" OwnerUserId="30709" LastActivityDate="2017-04-04T13:53:32.107" Title="Predicting unknown data using Knn" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18084" PostTypeId="2" ParentId="18082" CreationDate="2017-04-03T17:24:39.163" Score="1" Body="&lt;p&gt;when you &quot;predict&quot; something in KNN-classify problems, you are classifying new information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;yah, KNN can be used for regression, but let's ignore that for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The root of your question is why bother handling known data, and how can we predict new data.  Let's do KNN in R1, with two training examples.  The first one will be 0 and it will be class A, the next one will be 100 and it will be class B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, KNN is what's known as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Lazy_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;lazy classifier&lt;/a&gt;. You actually aren't training, any hyperparameters, just loading the training data.  I've loaded two points, and now I want to classify a new point.  Let's consider the point 0.  Since we already have the point 0 labelled as class A, we know it's going to be class A.  What about the point 4?  It's closer to 0 than to 100, so it's class A.&#xA;Similarly, the point 95 is closer to 100, so it's class B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KNN is really just doing those comparisons with more points and/or more dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: user asked the following: how would I predict if it is yes or no, based on the classification I have done before? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most basic implementations of KNN check the distance from your new case to all of the points in the training data.  If you want to visualize it, it's possible with one, two, and maybe 3 dimensional data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In python, you can do as follow (from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html#sphx-glr-auto-examples-neighbors-plot-classification-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn.org&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X1 = np.random.normal(0,1,100) #values centered around 0&#xA;X2 = np.random.normal(6,.1,100) #values centered around 6&#xA;X = np.concatenate([X1,X2])&#xA;X = np.array(X).reshape(-1,1)&#xA;Y = [0 for y in range(100)]+[1 for y in range(100)] #2 classes&#xA;Y = np.array(Y).reshape(-1,1)&#xA;&#xA;clf = neighbors.KNeighborsClassifier()&#xA;clf.fit(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now, you can predict by calling predict&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.predict(1)&#xA;clf.predict(4)&#xA;clf.predict(.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;KNN visualized in 2D: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D3ppe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D3ppe.png&quot; alt=&quot;KNN visualized&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from: &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21232" LastEditorUserId="21232" LastEditDate="2017-04-03T18:43:01.640" LastActivityDate="2017-04-03T18:43:01.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18085" PostTypeId="2" ParentId="18079" CreationDate="2017-04-03T18:02:50.177" Score="6" Body="&lt;p&gt;Given a linearly separable data set, the intuition behind the SVM-algorithm is to &lt;strong&gt;find a hyperplane that seperates both classes and maximizes the margin&lt;/strong&gt; &lt;em&gt;(which is twice the distance of the hyperplane to the closest data point)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So from all hyperplanes that seperate both classes, we want to find the one with the biggest margin &lt;em&gt;(to achieve the best generalization)&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's imagine we have two parallel seperating hyperplanes defined by the vector $\theta$ and with no data points in between, their distance is then given by: $$m= \frac{2}{\|\theta\|}$$&#xA;To maximize this distance, the two hyperplanes must pass through data points of the opposing classes. &#xA;In this case we can define another hyperplane in the middle &lt;em&gt;(with same distance to both hyperplanes and the same vector $\theta$)&lt;/em&gt; that now has the margin $m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see from the formula: the smaller $\|\theta\|$ the greater the margin $m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;strong&gt;maximizing the margin $m$ is equivalent to minimizing $\|\theta\|$&lt;/strong&gt;. Thus, a separating hyperplane is optimal if it minimizes $\|\theta\|$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just the rough intuition - if you want to go deeper into the maths behind it, I could recommend: &lt;a href=&quot;http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/&quot; rel=&quot;noreferrer&quot;&gt;http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30538" LastActivityDate="2017-04-03T18:02:50.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18086" PostTypeId="1" CreationDate="2017-04-03T18:06:34.540" Score="0" ViewCount="244" Body="&lt;p&gt;I have a classifier (and vectorizer) that I can export as a pickled model. I am using Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is on my local machine. Now, in order to perform the classifications in &quot;production&quot; I need to be able to run this in another machine that does not have Python installed. Any idea how to do this? Is it possible to create a Python executable with the pickled model in it?&lt;/p&gt;&#xA;" OwnerUserId="30712" LastEditorUserId="15620" LastEditDate="2017-04-04T09:01:19.057" LastActivityDate="2017-04-04T09:01:19.057" Title="Running sklearn trained classifier in a windows machine without Python" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18087" PostTypeId="1" AcceptedAnswerId="20242" CreationDate="2017-04-03T19:19:49.480" Score="1" ViewCount="1608" Body="&lt;p&gt;Is there any benefits from using Doc2vec for word embedding ( replacing word2vec ) ? in other hand if I train word2vec and doc2vec with the same dataset will I have the same word vectors ?&lt;/p&gt;&#xA;" OwnerUserId="29314" LastActivityDate="2017-07-07T13:26:50.030" Title="Doc2Vec or Word2vec for word embedding" Tags="|machine-learning|deep-learning|word2vec|word-embeddings|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18088" PostTypeId="1" AcceptedAnswerId="18097" CreationDate="2017-04-03T19:38:20.810" Score="3" ViewCount="1571" Body="&lt;p&gt;According to classical paper&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dropout operation affects not only training step but also test step - we need to multiply all neuron output weights by probability p.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in keras library I found the following implementation of dropout operation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;retain_prob = 1. - level&#xA;&#xA;...&#xA;&#xA;random_tensor = rng.binomial(x.shape, p=retain_prob, dtype=x.dtype)&#xA;&#xA;...&#xA;&#xA;x *= random_tensor&#xA;x /= retain_prob&#xA;return x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(see &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/backend/theano_backend.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/blob/master/keras/backend/theano_backend.py&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why x is divided by retain_prob while it should be multiplied?&#xA;Or I'm just confused, and multiplying weights is equivelent to dividing output value?&lt;/p&gt;&#xA;" OwnerUserId="29891" LastActivityDate="2018-04-29T07:22:36.910" Title="Convolutional layer dropout layer in keras" Tags="|machine-learning|neural-network|keras|theano|dropout|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18089" PostTypeId="1" AcceptedAnswerId="18090" CreationDate="2017-04-03T20:44:58.267" Score="1" ViewCount="75" Body="&lt;p&gt;I am trying to create a comparison group. So far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These remaining 55 need to be a randomized selection supplied from a larger data set. Any recommendations for R code that would create a randomization loop?&lt;/p&gt;&#xA;" OwnerUserId="30714" LastEditorUserId="30635" LastEditDate="2020-05-20T20:17:03.597" LastActivityDate="2020-05-20T20:17:03.597" Title="Randomizing selection in R" Tags="|r|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18090" PostTypeId="2" ParentId="18089" CreationDate="2017-04-03T20:52:27.490" Score="1" Body="&lt;p&gt;This will give you a sample with 55 record from the larger data-set that you have.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; sample &amp;lt;- df[sample(1:nrow(df), 55, replace=FALSE),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to make a reproducible example of the sample, you need to &lt;code&gt;set seed&lt;/code&gt; like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; set.seed(57)&#xA; sample &amp;lt;- df[sample(1:nrow(df), 55, replace=FALSE),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So later you can use the same seed for getting the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. &lt;code&gt;replace=FALSE&lt;/code&gt; means that each record after that has been picked from the &lt;code&gt;df&lt;/code&gt; to be included in the sample, will be excluded from the &lt;code&gt;df&lt;/code&gt; and it will not be chosen multiple times.&lt;/p&gt;&#xA;" OwnerUserId="30635" LastActivityDate="2017-04-03T20:52:27.490" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18091" PostTypeId="2" ParentId="18086" CreationDate="2017-04-03T20:55:49.313" Score="3" Body="&lt;p&gt;The simplest thing to do is to get Python installed properly on your production system. If you aren't able to do this for whatever reason, you can still run Python without installing it, but it's a bit more work to get everything to work properly, particularly if you don't have Administrator privileges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you truly can't install Python, your best bet would probably be to use something like &lt;a href=&quot;http://py2exe.org/index.cgi/Tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;py2exe&lt;/a&gt; to build a self-contained version of your application that includes a Python interpreter and required libraries.&lt;/p&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-04-03T20:55:49.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18092" PostTypeId="2" ParentId="18080" CreationDate="2017-04-03T21:06:12.013" Score="0" Body="&lt;p&gt;The error message is pretty clear... you feed a vector of length 24 to your model, but your model is outputting a vector of length 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dense3 = Dense(1 ,activation = 'softmax')(dense2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dense3 = Dense(24 ,activation = 'softmax')(dense2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11113" LastActivityDate="2017-04-03T21:06:12.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18093" PostTypeId="1" AcceptedAnswerId="18153" CreationDate="2017-04-04T00:07:00.883" Score="6" ViewCount="6715" Body="&lt;p&gt;I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii. Maybe this is an incorrect example or wrong information. Help me to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the sum or multiplication of any arbitrary numbers? Are there any other specific functions neural networks can't approximate or not?&lt;/p&gt;&#xA;" OwnerUserId="29206" LastEditorUserId="29206" LastEditDate="2017-04-04T11:10:42.510" LastActivityDate="2018-01-04T02:49:19.173" Title="Which functions neural net can't approximate" Tags="|neural-network|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18096" PostTypeId="2" ParentId="18093" CreationDate="2017-04-04T04:59:49.840" Score="2" Body="&lt;p&gt;Not sure what you mean by NN not approximating $pi$.&#xA;NN can approximate any continuous function (provide it contains enough nodes, one layer suffices). See &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Universal_approximation_theorem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7763" LastActivityDate="2017-04-04T04:59:49.840" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18097" PostTypeId="2" ParentId="18088" CreationDate="2017-04-04T07:26:03.537" Score="3" Body="&lt;p&gt;You are looking at the Keras code implementing dropout for training step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the Keras implementation, the output values are corrected during training (by dividing, in addition to randomly dropping out the values) instead of during testing (by multiplying). This is called &quot;inverted dropout&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inverted dropout is functionally equivalent to original dropout (as per your link to Srivastava's paper), with a nice feature that the network does not use dropout layers at all during test and prediction. This is explained a little in this &lt;a href=&quot;https://github.com/fchollet/keras/issues/3305&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras issue&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-04T07:33:40.330" LastActivityDate="2017-04-04T07:33:40.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18098" PostTypeId="2" ParentId="18088" CreationDate="2017-04-04T07:35:05.360" Score="4" Body="&lt;p&gt;Let's clarify few things about dropout. And &lt;a href=&quot;https://datascience.stackexchange.com/users/836/neil-slater&quot;&gt;Neil Slater&lt;/a&gt; is to credit for this answer since his comments helped formulate a more clear explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, dropout is a regularization method, it is usually only applied during training (although it can be used in prediction as an approximation to a Bayesian Neural Network as is explained by &lt;a href=&quot;https://arxiv.org/abs/1506.02142&quot; rel=&quot;nofollow noreferrer&quot;&gt;Yarin Gal's paper&lt;/a&gt;). As you might have understood, its goal is to limit overfitting. That is, to train a model that will generalize better to newly unseen data samples. So it has nothing to do with testing but everything to do with training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, the reason why you might have seen the output multiplied by $p$ at prediction time is a trick used with the very basic implementation of dropout referred to as vanilla dropout. At prediction time (or testing time if you prefer this wording) no need to drop anymore, but it is needed to scale the outputs by $p$. The reason is that because at training time, dropout was perform with a probability $p$, then the output need to be scaled to adjust for $p$ at prediction time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third, inverted dropout (which is the dropout implementation used in all serious DL libraries) does not need to scale the output at prediction time because scaling by $p$ is actually already performed at training time (by dividing by $p$). Therefore, no need for any trick at prediction time!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the concept is nicely explained in these videos from the &lt;a href=&quot;https://www.youtube.com/watch?v=NhZVe50QwPM&quot; rel=&quot;nofollow noreferrer&quot;&gt;Udacity Deep Learning course&lt;/a&gt; or this &lt;a href=&quot;http://cs231n.github.io/neural-networks-2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;course&lt;/a&gt; from Stanford.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this is clear enough. :)&lt;/p&gt;&#xA;" OwnerUserId="28129" LastEditorUserId="26562" LastEditDate="2018-04-29T07:22:36.910" LastActivityDate="2018-04-29T07:22:36.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18099" PostTypeId="1" AcceptedAnswerId="30547" CreationDate="2017-04-04T08:09:03.733" Score="8" ViewCount="1908" Body="&lt;p&gt;In the context of performance measures for classification, I have a question about recall and precision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the definition of recall:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$recall = \frac{T_p}{T_p+F_n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I look at this, it sounds like a $conditional$ probability to me -- probability that a test instance will be classified as positive, given that it is indeed positive:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ recall = Pr(X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive | X=positive ) = \frac{T_p}{T_p+F_n}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I am taking liberty to think (as it goes from the definition of $F_n$ ) that $F_n$ in the denominator is actually count of the test instances which are positive but got mis-classified as negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same light, if I think of precision now, this is how I am thinking of it as a probabilitty that a new test instance actually being positive given that it is predicted as a positive:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ precision = \frac{T_p}{T_p+F_p} = Pr(X=positive | X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this interpretation of precision and recall correct?&lt;/p&gt;&#xA;" OwnerUserId="30697" LastActivityDate="2022-09-05T18:42:42.567" Title="Interpretation of 'recall' as a conditional probability P( X=&gt;+ | X=+ )" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18100" PostTypeId="1" AcceptedAnswerId="18116" CreationDate="2017-04-04T09:32:15.630" Score="0" ViewCount="88" Body="&lt;p&gt;Based on what I know, symbolic computing is based on hard rules in the code. Statistical translation is based on statistic and probabilities. If so, could you give an easy to understand example to demonstrate this difference? If it is not correct, please give a correct explanation of the differences.&lt;/p&gt;&#xA;" OwnerUserId="30513" LastActivityDate="2017-04-04T18:12:39.833" Title="What's the difference of Symbolic translation approach and statistical translation?" Tags="|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18101" PostTypeId="1" AcceptedAnswerId="18102" CreationDate="2017-04-04T10:03:04.610" Score="2" ViewCount="564" Body="&lt;p&gt;I'm trying to debug why my neural network isn't working. One of the things I've observed is that the weights between the input layer and the first hidden layer hardly change at all, whereas weights later in the network (eg. the weights between the last hidden layer and the output) change significantly. Is this to be expected or a symptom of an error in my code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm applying backpropagation and gradient descent to alter the weights.&lt;/p&gt;&#xA;" OwnerUserId="14586" LastActivityDate="2017-04-04T11:20:59.417" Title="Should weights on earlier layers change less than weights on later layers in a neural network" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18102" PostTypeId="2" ParentId="18101" CreationDate="2017-04-04T11:20:59.417" Score="3" Body="&lt;p&gt;This is expected and well established - &lt;strong&gt;Vanished Gradient&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;This has the effect of multiplying n of these small numbers to compute gradients of the &amp;quot;front&amp;quot; layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;While I'm not an expert in neural network (experts, please add an answer), I'm sure professional implementation for neural network is tricky and &lt;strong&gt;highly optimized&lt;/strong&gt;. If you simply implement a vanilla text-book-styled neural network, you won't be able to use it to train a large network. Keep it small and you'll be fine.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-04-04T11:20:59.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18103" PostTypeId="1" AcceptedAnswerId="18104" CreationDate="2017-04-04T11:26:17.290" Score="15" ViewCount="7743" Body="&lt;p&gt;I'm dealing with a problem where I couldn't find enough dataset(images) to feed into my deep neural network for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was so inspired by the paper &lt;a href=&quot;https://arxiv.org/abs/1605.05396&quot; rel=&quot;noreferrer&quot;&gt;Generative Adversarial Text to Image Synthesis&lt;/a&gt; published by Scott Reed et al. on Generative Adversarial Networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was curious to know that, can I use available small dataset as an input to a GAN model and generate a much bigger dataset to deal with deeper network models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will it be good enough?&lt;/p&gt;&#xA;" OwnerUserId="29771" LastEditorUserId="29771" LastEditDate="2017-04-04T11:31:27.193" LastActivityDate="2021-03-11T20:16:20.637" Title="Can we generate huge dataset with Generative Adversarial Networks" Tags="|deep-learning|gan|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18104" PostTypeId="2" ParentId="18103" CreationDate="2017-04-04T12:11:31.253" Score="10" Body="&lt;p&gt;This is unlikely to add much beyond your direct data collection efforts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quality of current GAN outputs (as of 2017) will not be high enough. The images produced by a GAN are typically small and can have unusual/ambiguous details and odd distortions. In the paper you linked, the images generated by the system from a sentence have believable blocks of colour given the subject matter, but without the sentence priming you what to expect most of them are not recognisable as &lt;em&gt;any&lt;/em&gt; specific subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GANs with a less ambitious purpose than generating images from sentences (which is despite my criticism above, a truly remarkable feat IMO) should produce closer to photo-realistic images. But their scope will be less and probably not include your desired image type. Also, typically the output size is small e.g. 64x64 or 128x128*, and there are still enough distortions and ambiguities that original ground truth photos would be far preferable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The GAN is itself limited by training library available - it will not do well if you attempt to generate images outside of the scope of its training data. The results shown in the research paper of course focus on the domain supplied by the training data. But you cannot just feed &lt;em&gt;any&lt;/em&gt; sentence into this model and expect a result that would be useful elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find a GAN that has been trained on a suitable data set for your problem, then you are most likely better off trying to source the same data directly for your project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are facing a problem with limited ground truth data, then maybe a better approach to using a GAN would be to use a pre-trained classifier such as VGG-19 or Inception v5, replace the last few fully-connected layers, and fine tune it on your data. &lt;a href=&quot;https://gist.github.com/fchollet/7eb39b44eb9e16e59632d25fb3119975&quot; rel=&quot;noreferrer&quot;&gt;Here is an example of doing that using Keras library in Python&lt;/a&gt; - other examples can be found with searches like &quot;fine tune CNN image classifier&quot;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* State-of-the art GANs have got better since I posted this answer. &lt;a href=&quot;http://research.nvidia.com/publication/2017-10_Progressive-Growing-of&quot; rel=&quot;noreferrer&quot;&gt;A research team at Nvidia has had remarkable success creating 1024x1024 photo-realistic images&lt;/a&gt;. However, this does not change the other points in my answer. GANs are not a reliable source of images for image classification tasks, except maybe for sub-tasks of whatever the GAN has already been trained on and is able to generate conditionally (or maybe more trivially, to provide source data for &quot;other&quot; categories in classifiers).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-24T11:47:46.607" LastActivityDate="2017-11-24T11:47:46.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18107" PostTypeId="2" ParentId="18082" CreationDate="2017-04-04T13:45:26.593" Score="0" Body="&lt;p&gt;You can use KNN for regression in your case. You can either use fuctions for knn regression like knn.reg or you can implement your own solution. Suppose you have decided to use K=1 then find distances of test point from all known points and put the target variable as closest value. average of three values for k=3 and so on. You can use different parametrics for distance like euclidean, hamming and standarised the dataset before applying operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example using knn.reg:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;library(FNN)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;data(iris)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;train_index &amp;lt;- sample(1:150,120)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;train &amp;lt;- iris[train_index,2:4]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_train &amp;lt;- iris[train_index,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;test &amp;lt;- iris[-train_index,2:4]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_predict &amp;lt;- knn.reg(train,test,y_train,k=3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_actual &amp;lt;- iris[-train_index,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;rmse &amp;lt;- sqrt(sum((y_actual - y_predict$pred)^2)/length(y_actual))&lt;/p&gt;&#xA;" OwnerUserId="30637" LastEditorUserId="30637" LastEditDate="2017-04-04T13:53:32.107" LastActivityDate="2017-04-04T13:53:32.107" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18108" PostTypeId="2" ParentId="17819" CreationDate="2017-04-04T14:25:01.423" Score="1" Body="&lt;p&gt;I have found a solution and wanted to post it here, maybe its usefull for others.&#xA;as its seen above my observation vector is always 1 or zero.that make the probabilities too small.Another problem was the lenght of the Observation sequences it was too long.&#xA;so I just change the 1 to 100, and cut the Observation sequences at a point.&#xA;now is every thing fine.&lt;/p&gt;&#xA;" OwnerUserId="18516" LastActivityDate="2017-04-04T14:25:01.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18111" PostTypeId="2" ParentId="17910" CreationDate="2017-04-04T16:10:34.787" Score="7" Body="&lt;p&gt;I suspect the reason is that the class balance in your test set is different from the class balance in your training set.  That will throw everything off.  The fundamental assumption made by statistical machine learning methods (including logistic regression) is that the distribution of data in the test set matches the distribution of data in the training set.  SMOTE can throw that off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you have used SMOTE to augment the training set by adding additional synthetic positive instances (i.e., oversampling the minority class) -- but you haven't added any negative instances.  So, the class balance in the training set might have shifted from 0.5%/99.5% to something like (say) 10%/90%, while the class balance in the test set remains 0.5%/99.5%.  That's bad; it will cause the classifier to over-predict positive instances.  For some classifiers, it's not a major problem, but I expect that logistic regression might be more sensitive to this mismatch between training distribution and test distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are two candidate solutions for the problem that you can try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Stop using SMOTE.  Ensure the training set has the same distribution as the test set.  SMOTE might actually be unnecessary in your situation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Continue to augment the training set using SMOTE as you're currently doing, and compensate for the train/test mismatch by shifting the threshold for classification.  Logistic regression produces an estimated probability that a particular instance is from the positive class.  Typically, you then compare that probability to the threshold 0.5 and use that to classify it as positive or negative.  You can adjust the threshold to correct for that: replace $0.5$ with $0.5/k$, where $k$ is the ratio of positives in your training set after augmentation to positive before (e.g., if augmentation shifted the training set from 0.5%/99.5% to 10%/90%, then $k=10/0.5=20$); or you can use cross-validation to find a suitable threshold that maximizes the F1 score (or some other metric).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Incidentally, I recommend you make sure to use regularization with your logistic regression model, and use cross-validation to select the regularization hyper-parameter.  There's nothing wrong with 15K features if you have 120K instances in your training set, but you might want to regularize it strongly (choose a large regularization parameter) to avoid overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, understand that dealing with severe class imbalance such as you have is just hard.  Fortunately, there are many techniques available.  Do some reading and research (including on Stats.SE) and you should be able to find other methods you could try, if these don't work well enough.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-04T16:10:34.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18112" PostTypeId="2" ParentId="18075" CreationDate="2017-04-04T16:16:15.493" Score="1" Body="&lt;p&gt;Some things you can try, to help debug your network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implemented a gradient check, to make sure you are calculating the gradients correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment with different learning rates, and different initializations of the initial weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure to use randomization / symmetry breaking, or some suitable standard method, for initializing the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try starting with a single layer, and make sure your model can learn (you can solve some simple learning tasks) with a single-layer network.  Then try two layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try a ReLU activation function.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-04T16:16:15.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18115" PostTypeId="2" ParentId="18064" CreationDate="2017-04-04T17:28:42.573" Score="0" Body="&lt;p&gt;Can I say this about your dataset? Say a well is a point on your map, so if each well has an x-axis and y-axis coordinates. So effectively I've converted your dataset into an 11-dimensional problem. Now let's talk prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, perform PCA on your dataset with the number of components = 1.Remember the dataset which you have to perform PCA is on all the data &lt;strong&gt;but&lt;/strong&gt; the X-axis and Y-axis values Then plot your data set in a 3D scatter plot with the other two dimensions being X-axis and Y-axis and the third being the new Dimensionally reduced values done by PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the scatter plot if the following data follows any type of regression trend. Then use that type of regression to predict the values.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-04T17:28:42.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18116" PostTypeId="2" ParentId="18100" CreationDate="2017-04-04T18:12:39.833" Score="0" Body="&lt;p&gt;By symbolic translation, I assume that you meant rule-based translation. As a simple example, consider translating the English sentence &quot;I eat apples&quot; to Spanish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rule-based translation works by first tagging the words with their part of speech (&quot;I&quot; as noun/subject, &quot;eat&quot; as verb, &quot;apples&quot; as noun/object). Then, create an English parse tree and transform that into a Spanish parse tree (this requires knowledge of how English phrases map to Spanish phrases). Lastly, translate each English word into its corresponding Spanish word, and use the Spanish parse tree to determine the structure of the sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest statistical machine translation system are n-gram models based on n-gram frequencies. For an explanation of the mathematics behind statistical translation, see this paper: &lt;a href=&quot;http://www.mitpressjournals.org/doi/pdf/10.1162/coli.2006.32.4.527&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.mitpressjournals.org/doi/pdf/10.1162/coli.2006.32.4.527&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-04T18:12:39.833" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18118" PostTypeId="2" ParentId="18042" CreationDate="2017-04-04T19:32:04.960" Score="0" Body="&lt;p&gt;Here I am answering my own question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To obtain an IDW surface interpolation restricted to the area containing the original set of xyz data points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create a rectangular grid based on the xy point ranges using expand.grid {base} &lt;/li&gt;&#xA;&lt;li&gt;Create a convex hull using the function convexhull.xy{spatstat}&lt;/li&gt;&#xA;&lt;li&gt;Cut an irregular grid using the function inout{splancs} on the&#xA;rectangular grid masked by the convex hull&lt;/li&gt;&#xA;&lt;li&gt;Perform the interpolation using the function idw{gstat} with the new grid.&lt;/li&gt;&#xA;&lt;li&gt;Be sure to detach spatstat, because it has a different function for&#xA;IDW with the same name&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;###&#xA;# IDW in R&#xA;#&#xA;# Load gstat:&#xA;library(gstat)&#xA;# &#xA;# An example XYZ data set:&#xA;set.seed(1234)&#xA;x &amp;lt;- rnorm(20)*1000+3000&#xA;y &amp;lt;- rnorm(20)*1000+3000&#xA;z &amp;lt;- rnorm(20)+10&#xA;xyz &amp;lt;- data.frame(x,y,z)&#xA;# &#xA;# Calculate ranges and spacing:&#xA;xr &amp;lt;- round(range(x),1);xr&#xA;yr &amp;lt;- round(range(y),1);yr&#xA;b &amp;lt;- round((xr[2]-xr[1])/60,2);b&#xA;#&#xA;# Create a grid:&#xA;grd1 &amp;lt;- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),&#xA;    y=seq(from=yr[1],to=yr[2],by=b))&#xA;plot(grd1)&#xA;#&#xA;# Load spatstat and splancs&#xA;library(spatstat)&#xA;library(splancs)&#xA;#&#xA;# Extract convex hull &#xA;w2 &amp;lt;- convexhull.xy(x,y)&#xA;#&#xA;# detach spatstat because it has a different &quot;idw&quot; function:&#xA;detach(&quot;package:spatstat&quot;, unload=TRUE)&#xA;#&#xA;# Create a new cropped grid:&#xA;grd2 &amp;lt;- grd1[inout(grd1,w2$bdry[[1]]),]&#xA;plot(grd2)&#xA;#&#xA;# Convert grids and dataset to sp objects:&#xA;coordinates(grd1) &amp;lt;- ~x+y&#xA;gridded(grd1)   &amp;lt;- TRUE&#xA;coordinates(grd2) &amp;lt;- ~x+y&#xA;gridded(grd2)   &amp;lt;- TRUE&#xA;coordinates(xyz) &amp;lt;- ~x+y&#xA;#&#xA;# Run the interpolations:&#xA;idw1&amp;lt;-idw(formula=z~1, locations=xyz, newdata=grd1)&#xA;idw2&amp;lt;-idw(formula=z~1, locations=xyz, newdata=grd2)&#xA;#&#xA;# Plot the results:&#xA;image(idw1, col=terrain.colors(16))&#xA;contour(idw1, add=T)&#xA;points(x,y, col=4, pch=19)&#xA;image(idw2, col=terrain.colors(16))&#xA;contour(idw2, add=T)&#xA;points(x,y, col=4, pch=19)&#xA;#&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22065" LastEditorUserId="40853" LastEditDate="2018-03-12T02:37:02.150" LastActivityDate="2018-03-12T02:37:02.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18119" PostTypeId="1" CreationDate="2017-04-04T19:35:01.440" Score="1" ViewCount="117" Body="&lt;p&gt;I 'think' &lt;a href=&quot;https://datascience.stackexchange.com/questions/8029/online-incremental-unsupervised-dimensionality-reduction-for-use-with-classifica&quot;&gt;this&lt;/a&gt; is a related question, but not sure how to apply it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to build out a very crude recommendation system using Amazon ML, Facebook likes, and historical actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So lets say we have a number of users within a system that promotes products within several categories. To better predict which categories of items to present to the user, we will consider their past interactions with specific items, and the past interactions of other users who share a similar profile. The profile consisting of basically the users Facebook likes data and some demographic info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure of how to distill the Facebook likes data in a way that lets me make meaningful comparisons between users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure its obvious, but I'm completely new to machine learning, and data science in general. I'm currently limited to the capabilities of Amazon ML. Let me know if the question needs more clarification, constructive criticism is appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;br&gt;&#xA;As @liangjy pointed out, the solution to the recommendation system in general will be to use the collaborative filtering technique. This is most useful when their is sufficient data to link users based on their individual actions. Because we do not have enough/any data on new users, we are trying to use additional data (Facebook likes in this case) to help create that initial link. Where I'm stuck at is the vast number of profile/sites any one user may have liked. We could be comparing the likes of one user against tens of thousands of possibilities the rest of the users have presented. What is the best way to make this disparate data into something manageable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've considered taking a subset of users and pulling the top n sites (1,000?) from each demographic groups (age/gender). Then compare all other users to this base set, and create sub-groups based on their relationships to those sites. However, something about this approach feels like it would be skewed. I'm not sure what, but I'm pretty sure I won't get the results I'm looking for.&lt;/p&gt;&#xA;" OwnerUserId="30759" LastEditorUserId="-1" LastEditDate="2017-04-13T12:50:41.230" LastActivityDate="2024-01-10T10:07:00.967" Title="Compressing profiles with a large number of dimensions" Tags="|machine-learning|amazon-ml|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18120" PostTypeId="2" ParentId="18119" CreationDate="2017-04-04T21:59:40.673" Score="0" Body="&lt;p&gt;This seems similar to the collaborative filtering problem of assigning ratings to movies based on similar users and past reviews: &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Collaborative_filtering&lt;/a&gt;. The essence of this approach is to find a low-rank factorization of the matrix of ratings, where each user has one rating for each movie/product, and some of the ratings are unseen.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-04T21:59:40.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18123" PostTypeId="1" AcceptedAnswerId="18124" CreationDate="2017-04-05T03:14:05.157" Score="3" ViewCount="566" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the meanings of &lt;em&gt;Bias&lt;/em&gt;?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And is &lt;em&gt;Under fitting&lt;/em&gt;, which is used in &lt;em&gt;machine learning&lt;/em&gt; contexts, the same as &quot;Bias&quot;?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have faced &lt;em&gt;biased data&lt;/em&gt; in sampling in statistics but it seems this is a different thing to &lt;em&gt;bias&lt;/em&gt; in learning concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have heard that some data sets are &lt;em&gt;biased&lt;/em&gt;, also have heard the model (for example neural network) has &lt;em&gt;low bias&lt;/em&gt; or e.g. 'high bias' problem. Are these uses of &lt;em&gt;bias&lt;/em&gt; different?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="836" LastEditDate="2017-10-07T07:45:48.153" LastActivityDate="2017-10-08T06:58:15.897" Title="Does bias have multiple meanings in Data Science?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18124" PostTypeId="2" ParentId="18123" CreationDate="2017-04-05T04:15:54.293" Score="4" Body="&lt;p&gt;&lt;code&gt;Bias&lt;/code&gt; can mean different things in statistics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If your model is biased, it's likely your model is under-fitting.&lt;/li&gt;&#xA;&lt;li&gt;Some data set is biased in sample collection. For instance, if you assume your sample responses are independent, but somehow it's not, this is a bias in your data set. If you want to sample everybody in the country, but you skip some cities for no reason, this causes bias in your data set.&lt;/li&gt;&#xA;&lt;li&gt;Your estimators could be biased - the expectation of your estimator is not equal to the true value in the population.&lt;/li&gt;&#xA;&lt;li&gt;&quot;Bias&quot; is also used to describe an learnable offset parameter when using transfer functions, e.g. in neural networks when calculating activation of an artificial neuron.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-10-08T06:58:15.897" LastActivityDate="2017-10-08T06:58:15.897" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18126" PostTypeId="2" ParentId="18123" CreationDate="2017-04-05T07:19:34.803" Score="0" Body="&lt;p&gt;Bias is mainly how much error is there which can be caused due to empty fields or Nan values and under fitting will usually be caused if our training data will not have too much information which can help to train our model properly . A sample is biased if individuals or groups from the data are not represented in the sample as mentioned above. These are all related with statistics.&lt;/p&gt;&#xA;" OwnerUserId="30781" LastActivityDate="2017-04-05T07:19:34.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18131" PostTypeId="2" ParentId="11485" CreationDate="2017-04-05T10:50:35.973" Score="-2" Body="&lt;p&gt;the explanation doesn't appear naywhere in doc. but I think that setting nstart higher let kmeans run nstart(say 50) random initialization of centroids and choose the one that minimize better the cost; therefore you end up with a much more stable clusters becouse of kmeans always choose the better separation for your data&lt;/p&gt;&#xA;" OwnerUserId="30788" LastActivityDate="2017-04-05T10:50:35.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18132" PostTypeId="1" CreationDate="2017-04-05T12:05:10.957" Score="-1" ViewCount="54" Body="&lt;p&gt;I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for particular users(or groups). Example of what I want to find can be formulated like: 30% of users take their loans in 10 days after they pay the previous one OR 40% of users take loan in the same time of the month +/- 1 day.&#xA;My background in software development and usually i get reports by simply making a sql query, also I sometime use SSAS. But in this case I dont know how to simply ask. one way is to make a hypothesis and then trying to prove it, but it needs so much resource to program appropriate reports in that case and also I am limited to my imagination that generates hypothesis. &#xA;So what approach would you recommend?&lt;/p&gt;&#xA;" OwnerUserId="30792" LastActivityDate="2017-04-05T13:18:24.610" Title="what is the proper tool to analyse data and find trends in my case?" Tags="|data-mining|bigdata|data|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ClosedDate="2017-04-05T13:33:11.027" ContentLicense="CC BY-SA 3.0" />
  <row Id="18133" PostTypeId="2" ParentId="18132" CreationDate="2017-04-05T12:21:10.473" Score="0" Body="&lt;p&gt;It seems like you're trying to see what are the different types of users? In that case I would suggest clustering? Clustering is an unsupervised learning algorithm that finds patterns and splits your data into the observed patterns. There are several clustering techniques and which to use depends on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit learn resource for clustering&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're unsure which, just go with k means first. I suggest you read the documentation and examples in the scikit learn page!&lt;/p&gt;&#xA;" OwnerUserId="30624" LastActivityDate="2017-04-05T12:21:10.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18134" PostTypeId="1" CreationDate="2017-04-05T13:01:44.553" Score="2" ViewCount="208" Body="&lt;p&gt;The case is to model if the sequence of events influences the probability of binary target variable. We have for example five different events which occur in time (event: A,B,C,D,E). They can occur in order from 1 to 5. I would like to check if the order of their occurrence influences the target variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first idea was to convert the time of occurrence into numbers from 1 to 5 and then for example use logistic regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do You know any other practices? &#xA;Any whitepapers and ideas will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="30796" LastEditorUserId="30538" LastEditDate="2017-04-06T04:03:42.457" LastActivityDate="2024-02-02T04:07:55.973" Title="Modeling the influence of events order on probability" Tags="|data-mining|probability|sequence|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18135" PostTypeId="2" ParentId="18132" CreationDate="2017-04-05T13:18:24.610" Score="1" Body="&lt;p&gt;To piggy-back off of @&lt;em&gt;Impul3H&lt;/em&gt;, I recommend checking out &lt;a href=&quot;http://orange.biolab.si/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange Data Mining Tool&lt;/a&gt;.  In case you are unfamiliar with Python and think that you'd experience a steep learning curve with scikit learn, then the drag and drop interface from Orange would be a good tool for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outside of clustering, I would think that a &lt;a href=&quot;http://machinelearningmastery.com/naive-bayes-classifier-scratch-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; classifier may be useful for you; if your data is in categorical form.  This would be a supervised learning classification model, and is often one of the first and more easy to implement models on data in this format.&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2017-04-05T13:18:24.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18137" PostTypeId="2" ParentId="18134" CreationDate="2017-04-05T14:30:31.823" Score="0" Body="&lt;p&gt;If the order in which the events appears matters, consider using a recurrent neural network. The setup that you propose is invariant to event ordering, whereas in a RNN the events are fed in in sequential order.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-05T14:30:31.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18139" PostTypeId="2" ParentId="18134" CreationDate="2017-04-05T16:23:20.497" Score="0" Body="&lt;p&gt;If you have a large enough sample size, you can indeed carry this out the way you propose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For five events, you have 120 ($^5P_5$) possible permutations of the order of events. This allows you to run a logistic regression with 120 dummy independent variables, each of which corresponds to a permutation of your order of events. The F-test of this regression will function as a significance test to see if there is any difference in frequency of your outcome between different orderings of events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does require a large sample size, however. A good rule of thumb is at least 20 observations per independent variable in a General Linear Model, so if you have a few thousand samples, we'd expect this model to fit reasonably well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does assume you have a relatively small number of events. Five seems manageable, but as your number of events increases, you quickly run into problems as your number of independent variables grows factorially.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-04-05T16:23:20.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18140" PostTypeId="1" CreationDate="2017-04-05T17:10:51.403" Score="10" ViewCount="5506" Body="&lt;p&gt;Say we have used the TFIDF transform to encode documents into continuous-valued features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would we now use this as input to a Naive Bayes classifier? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bernoulli naive-bayes is out, because our features aren't binary anymore.&lt;br&gt;&#xA;Seems like we can't use Multinomial naive-bayes either, because the values are continuous rather than categorical.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an alternative, would it be appropriate to use gaussian naive bayes instead? Are TFIDF vectors likely to hold up well under the gaussian-distribution assumption?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sci-kit learn documentation for MultionomialNB suggests the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The multinomial Naive Bayes classifier is suitable for classification&#xA;  with discrete features (e.g., word counts for text classification).&#xA;  The multinomial distribution normally requires integer feature counts.&#xA;  However, in practice, fractional counts such as tf-idf may also work.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Isn't it fundamentally impossible to use fractional values for MultinomialNB?&lt;br&gt;&#xA;As I understand it, the likelihood function itself assumes that we are dealing with discrete-counts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(From Wikipedia):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;${\displaystyle p(\mathbf {x} \mid C_{k})={\frac {(\sum _{i}x_{i})!}{\prod _{i}x_{i}!}}\prod _{i}{p_{ki}}^{x_{i}}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would TFIDF values even work with this formula, since the $x_i$ values are all required to be discrete counts?&lt;/p&gt;&#xA;" OwnerUserId="28974" LastActivityDate="2017-06-05T07:40:52.183" Title="How to use TFIDF vectors with multinomial naive bayes?" Tags="|scikit-learn|naive-bayes-classifier|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18141" PostTypeId="1" AcceptedAnswerId="18160" CreationDate="2017-04-05T18:23:12.657" Score="12" ViewCount="4545" Body="&lt;p&gt;I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many levels (on the order of 100-1000). Given that one-hot encoding creates a dummy variable for each level, the feature importances are for each level and not each feature (column). What is a good way to aggregate these feature importances?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about summing or getting the average importance for all levels of a feature (probably the former will be biased towards those features with more levels). Are there any references on this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What else can one do to decrease the number of features? I am aware of group lasso, could not find anything easy to use for scikit-learn.&lt;/p&gt;&#xA;" OwnerUserId="14487" LastEditorUserId="15620" LastEditDate="2017-04-05T18:58:49.443" LastActivityDate="2017-04-06T12:03:39.117" Title="Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)" Tags="|scikit-learn|feature-selection|random-forest|xgboost|categorical-data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18142" PostTypeId="2" ParentId="17021" CreationDate="2017-04-05T19:00:14.757" Score="4" Body="&lt;p&gt;A quick approach can be using a library like &lt;a href=&quot;https://github.com/blue-yonder/tsfresh&quot; rel=&quot;nofollow noreferrer&quot;&gt;tsfresh&lt;/a&gt;, which extracts features from your time series, e.g. max value, number of peaks, median value, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, a thoughtful solution to this kind of problem involves domain knowledge, that is, insights from an expert that can tell you which aspects of your time series are important, e.g. frequent variations, very high peaks, plateaus, local patterns, etc. This knowledge is then used to derive &lt;em&gt;ad hoc&lt;/em&gt; feature extractors.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-04-06T05:30:48.430" LastActivityDate="2017-04-06T05:30:48.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18143" PostTypeId="1" AcceptedAnswerId="18148" CreationDate="2017-04-05T20:15:49.810" Score="0" ViewCount="2486" Body="&lt;p&gt;Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout and CNN mode is &lt;strong&gt;training mode&lt;/strong&gt;? Do I need to compute activation in the first layer and then apply dropout with division by retain probability p, or I need to apply activation to the result of the division?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say we have the following keras code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(128, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I understand correctly, dropout &lt;strong&gt;with division by p&lt;/strong&gt; will be applied to the activated result:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;result = [survive_mask] * relu(output)/p&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? Wouldn't it be more natural to have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;result = [survive_mask] * relu(output/p)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;because otherwise dropout operation breaks activation value normalization (i.e. to [0, 1]) ?&lt;/p&gt;&#xA;" OwnerUserId="29891" LastEditorUserId="29891" LastEditDate="2017-04-05T20:49:01.723" LastActivityDate="2017-04-05T22:04:01.020" Title="Correct order of operations involved into Dropout" Tags="|machine-learning|neural-network|deep-learning|keras|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18144" PostTypeId="2" ParentId="17986" CreationDate="2017-04-05T20:47:20.757" Score="3" Body="&lt;p&gt;I think a really nice explanation for the popularity of the sigmoid function is in these lecture notes (&lt;a href=&quot;http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The most obvious idea is to let $p(x)$ be a linear function of $x$. Every increment&#xA;of a component of $x$ would add or subtract so much to the probability. The&#xA;conceptual problem here is that $p$ must be between $0$ and $1$, and linear functions&#xA;are unbounded. Moreover, in many situations we empirically see “diminishing&#xA;returns” — changing $p$ by the same amount requires a bigger change in&#xA;$x$ when $p$ is already large (or small) than when $p$ is close to $1/2$. Linear models&#xA;can’t do this.&lt;/li&gt;&#xA;&lt;li&gt;The next most obvious idea is to let $\log p(x)$ be a linear function of $x$, so that&#xA;changing an input variable multiplies the probability by a fixed amount. The&#xA;problem is that logarithms are unbounded in only one direction, and linear&#xA;functions are not.&lt;/li&gt;&#xA;&lt;li&gt;Finally, the easiest modification of $\log p$ which has an unbounded range is the&#xA;logistic (or logit) transformation, $\log (p(1−p))$&#xA;. We can make this a linear function&#xA;of $x$ without fear of nonsensical results. (Of course the results could still&#xA;happen to be wrong, but they’re not guaranteed to be wrong.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30810" LastEditorUserId="26394" LastEditDate="2017-04-06T07:55:47.153" LastActivityDate="2017-04-06T07:55:47.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18145" PostTypeId="2" ParentId="17987" CreationDate="2017-04-05T20:51:34.957" Score="22" Body="&lt;p&gt;From the &lt;a href=&quot;http://cs231n.github.io/neural-networks-2/&quot; rel=&quot;noreferrer&quot;&gt;Stanford CS231N Notes&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Initializing the biases. It is possible and common to initialize the&#xA;biases to be zero, since the asymmetry breaking is provided by the&#xA;small random numbers in the weights. For ReLU non-linearities, some&#xA;people like to use small constant value such as 0.01 for all biases&#xA;because this ensures that all ReLU units fire in the beginning and&#xA;therefore obtain and propagate some gradient. However, it is not clear&#xA;if this provides a consistent improvement (in fact some results seem&#xA;to indicate that this performs worse) and it is more common to simply&#xA;use 0 bias initialization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;In LSTMs it's common to initialize the biases to 1 - see for &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf&quot; rel=&quot;noreferrer&quot;&gt;example&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30810" LastEditorUserId="98307" LastEditDate="2020-08-05T13:58:16.093" LastActivityDate="2020-08-05T13:58:16.093" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18146" PostTypeId="2" ParentId="18048" CreationDate="2017-04-05T21:05:08.743" Score="0" Body="&lt;p&gt;You can pass in a list to the input parameter when you create the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Model(input=[conv2d_1_1, conv2d_1_2, conv2d_1_3, conv2d_1_4], output=dense3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Make the input x a list of four numpy arrays of shape (3,6,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make the output y an array of one-hot vectors corresponding to your 145 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then call&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(x, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You probably want to change your last layer to have 143 dimensions in the output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_classes =143&#xA;Dense(num_classes, activation='softmax', activation = 'softmax')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30810" LastActivityDate="2017-04-05T21:05:08.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18147" PostTypeId="2" ParentId="18081" CreationDate="2017-04-05T21:18:02.030" Score="4" Body="&lt;p&gt;You are right that the models are equivalent in terms of the functions they can express, so with infinite training data and a function where the input variables don't interact with each other in any way they will both probably asymptotically approach the underlying joint probability distribution.  This would definitely not be true if your features were not all binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient boosted stumps adds extra machinery that sounds like it is irrelevant to your task.  Logistic regression will efficiently  compute a maximum likelihood estimate assuming that all the inputs are independent.  I would go with logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="30810" LastActivityDate="2017-04-05T21:18:02.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18148" PostTypeId="2" ParentId="18143" CreationDate="2017-04-05T21:21:27.950" Score="3" Body="&lt;p&gt;The usual processing for your suggested layers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(128, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;would be (reading left to right)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dense output -&amp;gt; relu -&amp;gt; apply dropout mask -&amp;gt; apply &quot;inverse dropout&quot; divide by p  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;precise&lt;/em&gt; combination may vary depending upon optimisations, and can in theory be changed a little without affecting the result (it doesn't matter to the end result numerically if we scale then mask or mask then scale for instance). However when dealing with vectorised optimisations (like those found in TensorFlow and Theano), it is normal to accept a percentage of &quot;wasted&quot; processing, and just have that naive left-to-right processing happen. It is often harder to parallelise decision branches than to simply process all items, even repeated multiplying by and adding zeroes for a significant fraction of each array.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no &quot;normalization&quot; implied by activation functions, so this is not a concern. From your comments, it seems you are worried that dividing by &lt;code&gt;p&lt;/code&gt; could mean that the output of a neuron that would be between 0 and 1 (because you were using sigmoid activation for example) would now be between 0 and 1/p - i.e. larger. That is true. Is this a problem? No it is not, and in fact it is required. The impact of the larger values is fully compensated for by the weights learned in the connections between layers. If you used &quot;vanilla&quot; dropout then the weights would be correspondingly larger, but you would need to scale the outputs down (new range would be 0 to p) during testing/prediction.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-05T22:04:01.020" LastActivityDate="2017-04-05T22:04:01.020" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18149" PostTypeId="2" ParentId="14676" CreationDate="2017-04-06T05:35:42.797" Score="1" Body="&lt;p&gt;In the word sampling steps in LDA the word count is used as weights for the multinomial dist. Re-weighting the TF's by its IDF's would dispropotionally increase the chance of rare words being sampled, making them have a stronger influence in topic assignment. But rare words have equally low probabilities of showing up in all documents and topics, hence flattening out the topic distribution of each document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is a similar reason why it is recommended to remove stop-words before running LDA.&lt;/p&gt;&#xA;" OwnerUserId="30818" LastActivityDate="2017-04-06T05:35:42.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18150" PostTypeId="2" ParentId="18140" CreationDate="2017-04-06T06:23:28.383" Score="1" Body="&lt;p&gt;When predicting the most likely class, the factorial terms are present in the probability calculation for every class, so they can essentially be ignored in the calculation. This leaves word probabilities which are raised to fractional powers, which are easily calculated. See this paper for a more detailed explanation: &lt;a href=&quot;http://www.cs.waikato.ac.nz/ml/publications/2004/kibriya_et_al_cr.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.waikato.ac.nz/ml/publications/2004/kibriya_et_al_cr.pdf&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-06T06:23:28.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18152" PostTypeId="1" AcceptedAnswerId="18173" CreationDate="2017-04-06T07:14:14.323" Score="1" ViewCount="2221" Body="&lt;p&gt;I am trying to write an optimal control agent for a simple game that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RhGtU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RhGtU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The agent can only move along the x-axis, and has three actions available to it: left, right, and do nothing. A random number of falling rocks are spawned at arbitrary positions along the top row. The goal is to survive as long as possible by avoiding collision on each time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I've done so far: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I use a feature vector $Φ$ with $Φ_0(s)$ being the current x-coordinate of the agent and $Φ_1(s)...Φ_n(s)$ taking on a 0 or 1 (1 indicating the presence of a rock).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The corresponding weight vector $θ$ is initialized to 0 for all weights. So I have the linear function approximation $$\hat v(s,θ)=\sum_{i=0}^nΦ_i(s)θ_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The reward on each time step is 1, and 0 upon collision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm actually trying to implement the algorithm below from &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton and Barto's 2017 draft&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Semi-gradient TD(0) for estimating $\hat{v} \approx v_{\pi}$&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Input: the policy ${\pi}$ to be evaluated&lt;br/&gt;&#xA;Input: a differentiable function $\hat{v} : \mathbf{S^+} \times \mathbb{R}^n \rightarrow \mathbb{R}$ such that $\hat{v}(terminal,·) = 0$&lt;br/&gt;&lt;br/&gt;&#xA;Initialize value-function weights $\theta$ arbitrarily (e.g., $\theta = 0$)&lt;br/&gt;&#xA;Repeat (for each episode):&lt;br/&gt;&#xA;$\qquad$Initialize $S$&lt;br/&gt;&#xA;$\qquad$Repeat (for each step of episode):&lt;br/&gt;&#xA;$\qquad\qquad$Choose $A \sim \pi(·|S)\qquad$ &lt;em&gt;# Not sure how to choose action here&lt;/em&gt;&lt;br/&gt;&#xA;$\qquad\qquad$Take action $A$, observe $R, S'$&lt;br/&gt;&#xA;$\qquad\qquad\theta \leftarrow \theta + \alpha[R + \gamma \hat{v}(S',\theta) − \hat{v}(S,\theta)]\nabla\hat{v}(S,\theta)$&lt;br/&gt;&#xA;$\qquad\qquad S \leftarrow S'$&lt;br/&gt;&#xA;$\qquad$until $S'$ is terminal&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My problem is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Is this even an appropriate algorithm to apply to this sort of task?&lt;/strong&gt; It feels like a policy gradient approach would be more suitable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;If yes to 1), how do I choose an action?&lt;/strong&gt; In the algorithm above, this appears as &quot;Choose $A \sim \pi(·|S)$&quot; Since the policy is implicit,  I compute the approximate value of the state as shown in 2) above and then greedify over the three actions (no ε-greedy, though), but the only thing that changes is $Φ_0(s)$. I'm definitely missing something.&lt;/p&gt;&#xA;" OwnerUserId="30817" LastEditorUserId="836" LastEditDate="2017-12-07T17:02:34.040" LastActivityDate="2017-12-07T17:02:34.040" Title="Semi-gradient TD(0) Choosing an Action" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18153" PostTypeId="2" ParentId="18093" CreationDate="2017-04-06T07:22:00.433" Score="16" Body="&lt;p&gt;A neural network can approximate any continuous function, provided it has at least one hidden layer and uses non-linear activations there. This has been proven by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;noreferrer&quot;&gt;universal approximation theorem&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, there are no exceptions for specific functions. You ask:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A neural network to approximate $\pi$ is very easy. Possibly what you read is that a neural network cannot generate new digits of $\pi$ that it has not already been shown. More on that later . . .&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What about the sum or multiplication of any arbitrary numbers? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, a neural network can approximate that.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any other specific functions neural networks can't approximate or not?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, there are no specific functions that a neural network cannot approximate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are some important caveats:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Neural networks do not encode the actual functions, only numeric approximations. This means there are practical limits on the ranges of inputs for which you can achieve a good approximation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A neural network being able to approximate a function in theory is not the same thing as you or I being able to construct a neural network that approximates that function. There is no known method to construct a neural network by analysis of a function alone (it can be done for specific simple functions such as xor).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The usual way to achieve approximation is to &lt;em&gt;train&lt;/em&gt; a neural network by giving example data. The network will approximate to data it has been shown. There is no guarantee that this will generalise to new inputs that it has not been trained on and approximate the correct outputs. In fact for certain types of input/output it cannot possibly do so. For instance, it will not learn how to generate the 4th digit of $pi$ if it has been shown digits 1,2,3,5,6,7,8,9. The best generalisation results occur for functions that have smooth transitions between the training examples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Neural networks do not extrapolate well to inputs outside of the data they have used for training. They &quot;fit&quot; to the training data (imagine a rubber sheet draped over all the points in the training set).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Neural networks do not learn to copy algorithms, only functions. So if you take a complex algorithm, such as AES encryption, and attempt to train a neural network to perform this given lots of input examples, has no real chance of working. Now, AES encryption can be considered a function e.g. $output = encrypt( input, key )$. So the NN &lt;em&gt;can&lt;/em&gt; approximate it. But it will only do so for the specific inputs and outputs it has been shown. In addition AES does not respond well to approximation - a single bit wrong will cause it to be a bad encryption. So you won't see NNs used to encrypt or decrypt in cryptography.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The capability of a neural network to approximate is limited by the number of neurons and connections it has. More complex functions require larger networks. In order to train a larger network on a more complex function takes more time and more training data. You could in theory train a neural network to learn a random number generator function. However, that would take an impossible amount of resources - memory to store the network, and time to train it against the whole output of the RNG.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="29575" LastEditDate="2018-01-04T02:49:19.173" LastActivityDate="2018-01-04T02:49:19.173" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18154" PostTypeId="1" AcceptedAnswerId="24501" CreationDate="2017-04-06T08:23:21.717" Score="6" ViewCount="19303" Body="&lt;p&gt;I want to count number of code by month.&#xA;This is my example dataframe.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        id    month  code&#xA;0     sally    0  s_A&#xA;1     sally    0    s_B&#xA;2     sally    0   s_C&#xA;3     sally    0   s_D&#xA;4     sally    0    s_E&#xA;5     sally    0   s_A&#xA;6     sally    0    s_A&#xA;7     sally    0   s_B&#xA;8     sally    0   s_C&#xA;9     sally    0   s_A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I transformed to this Series using count().&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby(['id', 'code', 'month']).m.count()&#xA;&#xA;id      code   month  count&#xA;sally  s_A      0    12&#xA;                1    10&#xA;                2     3&#xA;                7    15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, I want to include zero occurrence, like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id      code   month  count&#xA;sally  s_A      0    12&#xA;                1    10&#xA;                2     3&#xA;                3    0&#xA;                4    0&#xA;                5    0&#xA;                6    0&#xA;                7    15&#xA;                8    0&#xA;                9    0&#xA;                10   0&#xA;                11   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="22054" LastActivityDate="2020-03-24T18:22:48.390" Title="How to count categorical values including zero occurrence?" Tags="|python|pandas|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18156" PostTypeId="1" AcceptedAnswerId="18172" CreationDate="2017-04-06T09:06:06.453" Score="8" ViewCount="2961" Body="&lt;p&gt;Standard decision tree algorithms, such as ID3 and C4.5, have a brute force approach for choosing the cut point in a continuous feature. Every single value is tested as a possible cut point. (By tested I mean that e.g. the Information gain is calculated at every possible value.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With many continuous features and a lot of data (hence many values for each feature) this apporach seems very inefficient!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming finding a better way to do this is a hot topic in Machine Learning. In fact my Google Scholar search revealed some alternative approaches. Such as discretizing with k-means. Then there seem to be a lot of papers that tackle specific problems in specific domains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But is there a recent review paper, blog post or book that gives an overview on common apporaches for discretization?&lt;/strong&gt; I couldn't find one...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or else, maybe one of you is an expert on the topic and willing to write up a small overview. That would be tremendously helpful!&lt;/p&gt;&#xA;" OwnerUserId="28937" LastEditorUserId="28937" LastEditDate="2017-04-06T09:48:14.433" LastActivityDate="2017-04-06T20:35:56.533" Title="How to (better) discretize continuous data in decision trees?" Tags="|decision-trees|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18158" PostTypeId="1" AcceptedAnswerId="18159" CreationDate="2017-04-06T09:47:06.167" Score="4" ViewCount="457" Body="&lt;p&gt;In &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html&quot; rel=&quot;noreferrer&quot;&gt;this nice tutorial&lt;/a&gt; about CNNs, the authors build a single-layer CNN. The initial convolution weights are set randomly, according to a uniform distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the end of &lt;a href=&quot;http://deeplearning.net/tutorial/lenet.html#the-convolution-operator&quot; rel=&quot;noreferrer&quot;&gt;this scetion&lt;/a&gt;, the authors note that the randomly initialised kernel behaves very similar to an edge detector and give the following input and output as example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fTuG4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fTuG4.png&quot; alt=&quot;In/Output&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does the randomly initialised kernel behave like an edge detector? &lt;/p&gt;&#xA;" OwnerUserId="30825" LastActivityDate="2019-02-20T18:21:39.683" Title="Why does a randomly initialised convolution kernel correspond to an edge detector?" Tags="|convolution|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18159" PostTypeId="2" ParentId="18158" CreationDate="2017-04-06T10:14:38.143" Score="3" Body="&lt;p&gt;Take this with a grain of salt, but I think this is simply not true. You can evaluate it with the &lt;a href=&quot;https://github.com/MartinThoma/algorithms/tree/master/ML/filter-kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;code I just wrote&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3OMdh.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3OMdh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only 2, probably 3 of the 25 look like edge filters to me. The result of an edge filter looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bOciP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bOciP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-04-06T10:29:05.637" LastActivityDate="2017-04-06T10:29:05.637" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18160" PostTypeId="2" ParentId="18141" CreationDate="2017-04-06T12:03:39.117" Score="5" Body="&lt;p&gt;It depends on how you're one-hot encoding them. Many automated solutions for that will name all the converted booleans with a pattern so that a categorical variable called &quot;letter&quot; with values A-Z would end up like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;letter_A, letter_B, letter_C, letter_D,....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If after you've figured out feature importance you've got an array of feature and the associated weight/importance, I would analyze the array and perhaps sum up the feature importance weights for anything starting with &quot;letter%&quot;.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-06T12:03:39.117" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18167" PostTypeId="1" AcceptedAnswerId="18178" CreationDate="2017-04-06T17:26:11.667" Score="0" ViewCount="380" Body="&lt;p&gt;I'm looking for methods of Community Detection in networks. For example if I have a network of 100 people (each node is a person), how can I cluster nodes? What would be the best approach for grouping these People? I know this question is rather open, but I'm just looking for a nudge in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="20429" LastEditorUserId="8878" LastEditDate="2017-04-07T08:15:09.550" LastActivityDate="2017-04-07T08:41:09.923" Title="Clustering Multiple Networks" Tags="|clustering|graphs|social-network-analysis|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18168" PostTypeId="2" ParentId="18056" CreationDate="2017-04-06T17:34:21.793" Score="5" Body="&lt;p&gt;&lt;code&gt;xgboost&lt;/code&gt; usually performs better after one-hot encoding. Otherwise, it will treat your categorical variables as numerical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But most other tree packages support categorical variables; in other words, they support rules such as: &lt;code&gt;If(Car = Mercedes).&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, &lt;code&gt;xgboost&lt;/code&gt;, unfortunately, does not. Therefore, you must convert your categorical variables to binary variables so that it can do: &lt;code&gt;If(Car[Mercedes] &amp;gt;= 0.5).&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2017-04-06T17:34:21.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18169" PostTypeId="2" ParentId="18066" CreationDate="2017-04-06T17:41:05.680" Score="8" Body="&lt;p&gt;The question is under a wrong assumption. Many people do what you say they &quot;cannot&quot; do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;grid search&lt;/a&gt; implementation in the widely deployed sklearn package does just that. Unless &lt;code&gt;refit=False&lt;/code&gt;, it will retrain the final model using the entire data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think for some hyperparameters this might not be very desirable, because they are relative to the volume of data. For instance, consider the &lt;code&gt;min_samples_leaf&lt;/code&gt; pre-pruning tactic for a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision tree&lt;/a&gt;. If you have more data, the pre-pruning may not perform as you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But again, most people do in fact retrain using the entire data after cross-validation, so that they end up with the best model possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Addendum:&lt;/strong&gt; @NeilSlater says below that some people perform hold-out on top of CV. In other words, they have a train-test split and then perform model selection on the training. According to him, they re-train using the original training set split, but not the testing set. The testing set is then used to perform a final model estimation. Personally, I see three flaws on this: (a) it does not solve the problem I mentioned with some hyperparameters being dependent on the volume of training since you are re-training anyway, (b) when testing many models, I prefer more sophisticated methods such as nested cross validation so that no data goes to waste, and (c) hold-out is an awful method to infer how a model will generalize when you have little data.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2018-01-05T23:41:07.803" LastActivityDate="2018-01-05T23:41:07.803" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="18172" PostTypeId="2" ParentId="18156" CreationDate="2017-04-06T20:35:56.533" Score="8" Body="&lt;p&gt;No, you probably don't want to try all possible cut points in a serious implementation.  That's how we describe it in simple introductions to ID3, because it's easier to understand, but it's typically not how it is actually implemented, because it is slow.  In particular, if there are $n$ data points, then you'll need to test $n-1$ candidate thresholds; using the naive algorithm to calculate the information gain of each of those candidate thresholds takes $O(n)$ time per candidate, for a total of $O(n^2)$ time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, there are optimizations that speed this up significantly:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Don't try all possible thresholds.  Instead, pick a random sample of 1000 candidate thresholds (chosen uniformly at random out of the set of $n-1$ candidate thresholds), calculate the information gain for each, and choose the best one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use dynamic programming to efficiently compute the information gain of all $n-1$ splits, in total of $O(n)$ time, by reusing computation.  The algorithm is pretty straightforward to derive.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-06T20:35:56.533" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18173" PostTypeId="2" ParentId="18152" CreationDate="2017-04-06T20:47:55.503" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Is this even an appropriate algorithm to apply to this sort of task?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, you have selected an evaluation algorithm from chapter 9 of the book. None of the algorithms in chapter 9 are control algorithms. They are all designed to estimate the value function for a policy supplied as input. The corresponding control algorithms are discussed in chapter 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current draft of the book does not give the corresponding TD(0) control algorithm with linear estimator. However, that algorithm does exist and might be suitable (with caveats). In fact in your case it could even have benefits over action-value based methods, because you reduce the scope of estimates needed by a factor of 3. This is something that you can take advantage of only if you have a full model of the environment, so can look ahead one time step to determine the best action. Without a model of the environment, or if you don't want to use the model in your agent, then you must use an action value based algorithm like Monte Carlo, SARSA or Q Learning.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2) If yes to 1), how do I choose an action?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Well it was a no, but you could use the control version of TD(0). Then you have the problem of using your state value function to figure out the policy. The rule here is that to use state values you need to use a model of the environment. Specifically you need to be able to predict the next state and immediate reward given the current state and action. In the book, this is usually represented by the transition function $p(s′,r|s,a)$ which gives the probability of each possible successor state and reward. In a fully deterministic game, the probability is just 1.0 for one target state and reward caused by each action. In your case you have new rocks appearing randomly at the top. To be complete you would probably have to model this in detail (which would be painfully slow). However, given the very low influence of this top row, and how little planning the agent can do to deal with it, I'd be tempted to just sample it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you want to choose the greedy action, then you can find a policy by taking $argmax_a$ over the next step. When you have a state value function, then you have to run the step forward in simulation to figure out the expectation over each action. This is the same calculation for greedy policy as used in dynamic programming (back in chapter 4 of the book):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi(s) = argmax_a \sum_{s'} p(s',r|s,a) (r + \gamma \hat{v}(s',\theta))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this is a lot simpler if you used action values istead (e.g. in SARSA):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\pi(s) = argmax_a \hat{q}(s,a,\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . . so despite the fact this is less efficient, you might want to use action values for less effort in this part.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;One additional thing you are likely to have problems with: Your choice of state representation does not have good linear relationship with the true value function. The linear estimator is quite limited in what it can do. A better estimator here would be a neural network with at least one hidden layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you can probably tweak the representation slightly to get something that will work a little bit with a linear estimator - the trick is to have the rocks part of the state represented &lt;em&gt;relative to the agent&lt;/em&gt; - i.e. don't have a grid of absolute positions of rocks, but make the grid relative to the agent. That way, a rock directly above the agent will always contribute the same to the current state value, which is important. Without this tweak, and using a linear approximator, your agent will not learn optimal control, it will instead learn a kind of fatalistic &quot;with these many rocks, I have roughly this long to live&quot; value function and probably just take random actions (if the distribution of rocks is not even it might learn to move to a particular column . . .)&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-07T09:01:17.033" LastActivityDate="2017-04-07T09:01:17.033" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18174" PostTypeId="1" CreationDate="2017-04-06T21:21:57.417" Score="1" ViewCount="34" Body="&lt;p&gt;Context: I'm training an RNN with LSTM layers using the keras api. I have sequences of 20 timesteps, but just a binary response. For example: X=[[[1,2,3], [2,3,1]], [[4,1,2], [2,1,3]]], and y = [[1.0], [0.0]]. The binary variable y explains the action of the &lt;strong&gt;next&lt;/strong&gt; timestep, because that is all I'm interested in, even though each vector in the sequence has an associated response to it. When training the model with one timestep ahead, I get specifications that look like the following.&lt;a href=&quot;https://i.stack.imgur.com/ccBCo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ccBCo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything looks normal there. However, since I also want to predict two timesteps ahead, I train the same model, but cutoff the last vector in the sequence, but keeping the same response, thereby forcing to look ahead by two timesteps. After plotting the same curve, something strange happens:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/3hI1G.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3hI1G.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sharp vertical jump occurs around the cutoff of .43, in such a way that I've never seen before. When I test it with 3 timesteps ahead, the exact same thing happens, just at a different cutoff (around .2). What I curious about, is this common, and is it indicative of something greater? Why would such a drastic change in just the sensitivity occur?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure if it's relevant, but this is the layout of the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Masking(mask_value=0, input_shape=(seq_length, 66)))&#xA;model.add(LSTM(64, return_sequences=False, W_regularizer=regularizers.l2(.001)))&#xA;model.add(Dropout(.2))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;rms = RMSprop(lr=.001)&#xA;model.compile(loss='binary_crossentropy', optimizer=rms, metrics=['acc', 'fmeasure'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28665" LastActivityDate="2017-04-06T21:21:57.417" Title="What might explain a dramatic vertical jump in sensitivity" Tags="|cross-validation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18175" PostTypeId="2" ParentId="13026" CreationDate="2017-04-06T22:17:10.847" Score="0" Body="&lt;p&gt;This isn't leakage. However, this is still a problem. You are training your data with testing data; basically your model is cheating by looking at the answers ahead of time&lt;/p&gt;&#xA;" OwnerUserId="30846" LastActivityDate="2017-04-06T22:17:10.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18177" PostTypeId="1" AcceptedAnswerId="18189" CreationDate="2017-04-07T08:14:29.147" Score="3" ViewCount="513" Body="&lt;p&gt;I'm currently learning about Collaborative Learning and Content-based Recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the main things that is discussed in both methods is about calculating similarity between two users or two items. Commonly, similarity between two entity (each represented by a vector) can be calculated using Cosine similarity or other similarity coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One question that I haven't found the answer is about calculating similarity between two entities in which each has a feature that explains each entity's location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, the entities in my data is about restaurants and each restaurant has several attributes such as&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;food price average&lt;/li&gt;&#xA;&lt;li&gt;user rating/review score&lt;/li&gt;&#xA;&lt;li&gt;location in $(\text{lat}, \text{long})$&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For food price average or user rating, maybe it's still relevant to include them in cosine similarity function, but how about location–which is described in a tuple of real numbers? What's the best way to include distance between entities as a component in the similarity function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="30859" LastActivityDate="2017-04-07T22:34:29.880" Title="Item Similarity with Location Feature" Tags="|recommender-system|similarity|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18178" PostTypeId="2" ParentId="18167" CreationDate="2017-04-07T08:41:09.923" Score="4" Body="&lt;p&gt;Your question is not clear in a way there are two different Graph Clustering problems. One is having a dataset of different graphs and you would like to cluster similar graphs (in this case each object is a graph), and the other when you have a graph (e.g. a social network) and you would like to group similar nodes inside that graph (here each object is a node).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The first problem&lt;/strong&gt; needs Graph Embedding which is to transfer graphs into a n-dimensional manifold and from there on, you have a classical clustering problem. For this case have a look at what &lt;a href=&quot;https://www.informatik.uni-ulm.de/ni/ANNPR10/InvitedTalks/BuhnkeANNPR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Horst Bunke&lt;/a&gt; has done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more simplified approach would be &lt;a href=&quot;https://en.wikipedia.org/wiki/Category:Network_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature extraction from networks&lt;/a&gt; and apply classical clustering methods. These features are statistical and topological network measures such as Clustering Coefficient, Assortativity Index, Average Shortest Path Length, Density, Diameter, Average Centrality, etc. This approach is more about the characteristics of real-world networks e.g. biological networks or social networks as it takes into account those characteristics which determine the real-world phenomena. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The second problem&lt;/strong&gt; is usually called &lt;a href=&quot;https://en.wikipedia.org/wiki/Community_structure&quot; rel=&quot;nofollow noreferrer&quot;&gt;Community Detection&lt;/a&gt; in literature. There are several method for doing that based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Louvain_Modularity&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modularity Score&lt;/a&gt;, &lt;a href=&quot;http://www.mapequation.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Information Theory&lt;/a&gt;, &lt;a href=&quot;https://www.researchgate.net/publication/283531730_SGPD_Volume_Maximization_for_Community_Detection/stats&quot; rel=&quot;nofollow noreferrer&quot;&gt;Topological Structure&lt;/a&gt; or &lt;a href=&quot;http://www.cs.cmu.edu/~aarti/Class/10701/readings/Luxburg06_TR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spectral Graph Theory&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-04-07T08:41:09.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18179" PostTypeId="1" CreationDate="2017-04-07T09:01:21.087" Score="2" ViewCount="269" Body="&lt;p&gt;I have used a &quot;adabag&quot;(boosting + bagging) model on imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best result I got where: ROC= 0.711 SENS=0.94 SPEC=0.21.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used classifierplots package in R for a diagnostic plot.&#xA;this is the plot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AdO2c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AdO2c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't really understand how to interrupt the plots of positive instances per decile, prediction density and calibration. I have tried to find a comprehensive explanation with no success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can you help me understand what the graphs mentioned above represent? I really want to improve the results.&lt;/p&gt;&#xA;" OwnerUserId="30860" LastActivityDate="2017-04-07T09:01:21.087" Title="interpret diagnostics plots from classifierplots package in R" Tags="|r|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18180" PostTypeId="1" CreationDate="2017-04-07T10:49:26.163" Score="3" ViewCount="781" Body="&lt;p&gt;I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that maintenance crew would always performance maintenance/servicing just in time before hardware failure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, servicing are carried out at a fixed interval regardless of the condition of the hardware and it has practically never failed. The dataset we have is of this nature:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;equipment: pump1&#xA;location: outdoor (or indoor)&#xA;provider: providerA&#xA;operation: -&#xA;   temperature (as a function of time)&#xA;   moisture (as a function of time)&#xA;   team (as a discreet function of time, ie teamA, teamB, teamC)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, we know some features of the equipment (here pump1), and we know some operating parameters (here temperature, moisture) as a function over time, and we also know which team did the servicing over time. I understand that survival analysis would be most suitable for a predictive maintenance problem but each operating parameters vary over time of study. Nothing is constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While regression does not take into account time. As such how do we actually model it to predict its failure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The above parameters are adapted from &lt;a href=&quot;https://www.kaggle.com/ludobenistant/predictive-maintenance&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/ludobenistant/predictive-maintenance&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30864" LastEditorUserId="30864" LastEditDate="2017-04-09T06:17:54.213" LastActivityDate="2017-04-11T11:36:59.147" Title="Predicting hardware failures with limited data" Tags="|dataset|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18181" PostTypeId="2" ParentId="13750" CreationDate="2017-04-07T11:08:21.207" Score="1" Body="&lt;p&gt;Once you installed the GPU version of Tensorflow, you don't have anything to do in Keras. As written in the &lt;a href=&quot;https://keras.io/getting-started/faq/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras documentation&lt;/a&gt;, &lt;em&gt;&quot;If you are running on the TensorFlow backend, your code will automatically run on GPU if any available GPU is detected.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you want to check that the GPU is correctly detected, start your script with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The standard output should not show any error and print the name of the GPU. If so, you are ready to run Keras and Tensorflow in GPU mode.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-04-07T11:08:21.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18182" PostTypeId="2" ParentId="13706" CreationDate="2017-04-07T11:14:41.780" Score="2" Body="&lt;p&gt;If you are looking for a way to compile code written for the Keras API to code only using the Tensorflow API, there is no ready-to-use solution for that. And I cannot see any good reason why someone would want to do that in the first place...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can simply look at the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/backend/tensorflow_backend.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras backend source code for Tensorflow&lt;/a&gt; and rewrite your entire implementation. But once again I don't see any valid reason that would make this worth it.&lt;/p&gt;&#xA;" OwnerUserId="28129" LastActivityDate="2017-04-07T11:14:41.780" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18186" PostTypeId="1" AcceptedAnswerId="20246" CreationDate="2017-04-07T15:13:02.473" Score="2" ViewCount="878" Body="&lt;p&gt;I'm currently trying to implement the &lt;a href=&quot;https://arxiv.org/abs/1611.02361&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang&lt;/a&gt; in Keras.&#xA;For me this is the first network to implement in Keras, so I came up with some questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DXEqf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DXEqf.png&quot; alt=&quot;Figure 1: DSCNN (document modeling) by Rui Zhang, copied from his paper.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings. &#xA;The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will have a batch of the following type (only symbolic -&gt; according to `fit()´):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions))&#xA;doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)]&#xA;batch = [doc_1, doc_2, doc_3, ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;we_idx_ij&lt;/code&gt; is the word embedding index for the j-th word in the i-th sentence of a certain document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt; &#xA;Does this make sense? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Includes:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Include the needed methods/layers and initialize symbolic constants&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed&#xA;&#xA;# constants (for testing)&#xA;max_sentences_per_document = 10&#xA;max_words_per_sentence = 100&#xA;w2v_dimensions = 300&#xA;vocab_size = 20000&#xA;batch_size = 60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Model:&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# preparing some shared layers&#xA;shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W])  # optional,cause no training&#xA;shared_sentence_lstm = TimeDistributed(&#xA;    LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'),&#xA;    input_shape=(max_words_per_sentence, w2v_dimensions)&#xA;)&#xA;shared_sentence_lstm_2 = LSTM(activation='tanh')&#xA;&#xA;# sentence modeling&#xA;sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)]&#xA;sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)]&#xA;sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)]&#xA;sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)]&#xA;sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)]&#xA;&#xA;# document modeling&#xA;doc_modeling = concatenate(sentence_modeling)&#xA;doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling)&#xA;doc_modeling = MaxPool1D()(doc_modeling)&#xA;doc_modeling = Activation('softmax')&#xA;doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to have the word embedding already in the batch-data or should I provide the &lt;code&gt;s_i&lt;/code&gt; only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 4:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this network work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 5:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this code implement the network as it is proposed in the paper?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 6:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions to improve the performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I'm not too far off the track with this implementation and looking forward to your answers :-)&#xA;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="30835" LastEditorUserId="30835" LastEditDate="2017-04-08T09:55:30.613" LastActivityDate="2020-12-23T23:01:25.210" Title="Implementing the Dependency Sensitive CNN (DSCNN ) in Keras" Tags="|neural-network|nlp|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18187" PostTypeId="1" AcceptedAnswerId="18188" CreationDate="2017-04-07T17:19:00.823" Score="0" ViewCount="474" Body="&lt;p&gt;In general we use word2vec for word embedding in seq2seq model, is it possible to add the document vector from Doc2vec with the input words , I mean using the tag of the document as a word and its vector for emending : the picture will explain mu point a view  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;default seq2seq &#xA;&lt;a href=&quot;https://i.stack.imgur.com/uQfFU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uQfFU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My opinion&#xA;&lt;a href=&quot;https://i.stack.imgur.com/VLVdb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VLVdb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the doc_tag is followed by its original words so will that improve my seq2seq model ( logically ) ?&lt;/p&gt;&#xA;" OwnerUserId="29314" LastActivityDate="2017-05-24T09:20:40.850" Title="Adding document vector doc2ec to seq2seq model" Tags="|machine-learning|deep-learning|nlp|tensorflow|word2vec|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18188" PostTypeId="2" ParentId="18187" CreationDate="2017-04-07T17:40:07.230" Score="0" Body="&lt;p&gt;The purpose of the encoder (green part) is to determine this document vector, so if you want to provide it yourself use the decoder and feed the embedding as the initial state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose you could use your embedding as a prior for the encoder output. One alternative to your suggestion, which might work, is to use the embeddings as an MSE regularizer to the encoder output.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-04-07T17:40:07.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18189" PostTypeId="2" ParentId="18177" CreationDate="2017-04-07T18:28:41.273" Score="2" Body="&lt;p&gt;Perhaps you could transform the latitude and longitude into spherical coordinates. In this coordinate system, the cosine of the angle between the vectors has a natural geometric interpretation.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-07T18:28:41.273" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18191" PostTypeId="1" AcceptedAnswerId="18194" CreationDate="2017-04-07T19:56:02.253" Score="-1" ViewCount="51" Body="&lt;p&gt;can someone please tell me the difference between a BI trendline, and a linear/exponential regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When explaining this to a hardcore BI person, what can be used to mark the difference?   Thanks.&lt;/p&gt;&#xA;" OwnerUserId="10881" LastActivityDate="2017-04-07T20:57:16.243" Title="BI vs Data Science. Looking for a difference in definitions" Tags="|regression|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18194" PostTypeId="2" ParentId="18191" CreationDate="2017-04-07T20:57:16.243" Score="1" Body="&lt;p&gt;Any difference in regression models can be reduced to differences in the latent model (e.g., linear vs. exponential), regularizer (e.g., $L^p$ norm), and loss function. So you can have subtle differences by keeping some of these three parameters fixed while modifying the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of a BI trend line is that it assumes an affine latent model without saying anything about the regularizer or loss function (though I'd assume it's the MSE unless stated otherwise). In the data science world, you should also state what loss function and regularizer you used if you want to be clear.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-04-07T20:57:16.243" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18195" PostTypeId="2" ParentId="18177" CreationDate="2017-04-07T21:26:07.573" Score="1" Body="&lt;p&gt;This answer is from my view point of how I would insert distance component into restaurant recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, I don't feel it is right to insert this distance component into cosine similarity because it is just a innocent dot product which says how similar vectors are and nothing more. But sorting order of recommendations can be changed by using a simple formula like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sort order based on the values $r_i*(1+e^{-b(dist)})$ or any similar formula for that reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $b(.)$ is the bucketed value for the distance between the restaurant and customer which can be found from latitude longitude position of the same by using &lt;a href=&quot;https://en.wikipedia.org/wiki/Haversine_formula&quot; rel=&quot;nofollow noreferrer&quot;&gt;Haversine formula&lt;/a&gt; and $r_i$ the original restaurant rating. This does not change the restaurant rating but merely changes the sorting order based on user location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt; : Let [(A, 3.5, 2 miles), (B, 4, 10 miles)] be a sample list of elements of tuple type (Restaurant, rating, distance)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our buckets are say 0-2 miles is 0 and 2-5 miles as 1 and 5-10 miles as 2 etc. So A and B compound rating comes to $7$, $4.5$. So A will be on top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this kind of helps. Any edits giving some cool ideas and good references are always welcome.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-04-07T21:26:07.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18196" PostTypeId="2" ParentId="14838" CreationDate="2017-04-07T22:15:28.980" Score="1" Body="&lt;p&gt;I didn't find any good explanations or papers on this topic, other than things about category based models and multigrams (parts of words). So, I came up with one myself. I'm using Java, but here's the code for the length normalization translated to Python. Also, this is assuming that you're adding the log probabilities for each word together, like KenLM does for the total sentence (or phrase) score.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Get likelihood normalized by length and number of words.&#xA;# Since the likelihoods of each word are added together, phrases with more words tend to &#xA;# have a lower probability.&#xA;def getNormLikelihood(phrase):&#xA;    score = getLikelihood(phrase) # The normal score&#xA;    numWords = phrase.count(' ')+1 # The number of words in the phrase.&#xA;    # Get the absolute difference between the &#xA;    #   average word length of all words and the average word length of the phrase.&#xA;    # Then, weight that weight by the reciprocal of the normal score, such that the &#xA;    #   lenWeight is less if the overall score is less.&#xA;    # Intuitively, this means that we care less about normalizing by word length the more &#xA;    #   common the phrase is.&#xA;    # This effectively solves the problem of giving a greater likelihood for long words &#xA;    #   that aren't even in the language model (and are given the default &quot;&amp;lt;unk&amp;gt;&quot; score).&#xA;    lenWeight = abs(len(phrase) / numWords - avgLen) * (-1/score)&#xA;    return score / (1/lenWeight)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How you want to use this normalized score, and how effective it is, depends on your application. In my case (splitting words), I used this normalized score to score each splitting option, but then also added (logarithm) the normal score of the option with context, which you could consider the prior for the option (in Naive Bayes terms). I then added an overall prior (manually picked, but could be tuned) to the original, unsplit option, for the probability of needing to split a word. The normalized score greatly helped, and with these probabilities together, I was able to get 100% accuracy for my test cases.&lt;/p&gt;&#xA;" OwnerUserId="30812" LastActivityDate="2017-04-07T22:15:28.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18197" PostTypeId="2" ParentId="18177" CreationDate="2017-04-07T22:24:21.843" Score="1" Body="&lt;p&gt;Let's recall the rating prediction of restaurant $ x $ and user $i$ as a weighted sum according to the similarity:&lt;br&gt;&#xA;$$r_{x,i} = \frac{\sum_{j\in N(i,x)} s_{i,j} \cdot r_{x,j}} {\sum_{j'\in N(i,x)} s_{i,j'}} $$&#xA;were $ s_{i,j}$ is the similarity between user $i$ and user $j$ and $N(i,x)$ is the set of all restaurants that were both similar to restaurant $x$ &lt;strong&gt;and&lt;/strong&gt; were rated by user $x$. This set can be found by applying knn and finding $k$ similar users to user $i$ (i.e. the &lt;em&gt;collaborative&lt;/em&gt; in collaborative filtering).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now given more prior knowledge, such as geographical location, we can tweak our formula to account for those biases. Generally speaking, when you see a user rates a restaurant, you should account the habits rating of that user (how much does he usually rates above or below the mean) and the restaurant affect (how the restaurant is rated in respect to the mean rating).&lt;br&gt;&#xA;Denoting the total rating mean as $\mu $, the user mean rating as $\bar{r_i}$ and the restaurant average rating as $\bar{r_x}$, we can break down the rating to it's biases (deviances from the mean) as such:&#xA;$$b_{x,i} = \mu + \underset{b_i}{\underbrace{(\bar{r_i} - \mu)}}+ \underset{b_x}{\underbrace{(\bar{r_x} - \mu)}}$$&#xA;Now the prediction will look as follows:&#xA;$$r_{xi} = b_{x,i} + \frac{\sum_{j\in N(i,x)} s_{i,j} \cdot (r_{x,j}-b_{x,j})} {\sum_{j'\in N(i,x)} s_{i,j'}} $$&#xA;Now this was quite general, just in order for us to speak in terms of biases. Once at hand we can apply them as we see fit.&lt;br&gt;&#xA;The most simple way I can think of (you're welcome to share more elaborate ones) is to take the restaurant bias $b_x$ and instead of calculating it over the whole data (using the general $\mu$), you can do as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Take all your lats and longs, project them onto some 2D sheet or 3D ellipsoid (Depends on the accuracy you wish to achieve. What projection to use is somewhat an art for it self, there's vast literature, but usually simpler method work good enough).&lt;/li&gt;&#xA;&lt;li&gt;Cluster the restaurants into some $H$ clusters. Those will be geographically related. (Again, Euclidean distance will do, but there are &lt;a href=&quot;https://en.wikipedia.org/wiki/Geographical_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;more sophisticated ones&lt;/a&gt;)  &lt;/li&gt;&#xA;&lt;li&gt;For every cluster $h\in H$ you calculate it's own $\mu^{(h)}$, and now for all the restaurants $x^{(h)}$ related to cluster $h$ you calculate it's own bias term: $$b_{x^{(h)},i} = \mu^{(h)} + \underset{b_i}{\underbrace{(\bar{r_i} - \mu^{(h)})}}+ \underset{b_{x^{(h)}}}{\underbrace{(\bar{r_{x^{(h)}}} - \mu^{(h)})}}$$ &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, basically, you broke down your general rating data into $H$ geographically close datasets that capture some spatial locality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll be happy to elaborate more if needed.&lt;/p&gt;&#xA;" OwnerUserId="29960" LastEditorUserId="29960" LastEditDate="2017-04-07T22:34:29.880" LastActivityDate="2017-04-07T22:34:29.880" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18199" PostTypeId="1" AcceptedAnswerId="18211" CreationDate="2017-04-08T00:19:57.353" Score="4" ViewCount="3215" Body="&lt;p&gt;In Random Forest method, for each tree we randomly select a set of variables (features) of fixed size. But once this set is frozen for that particular tree, does the tree behave like a regular decision tree algorithm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that random forest is nothing but generating a bunch of classical 'decision trees' and taking their votes towards the final classification. But in many places whatever description I have read seems to suggest that; for a given decision tree within the forest even at each node we randomly select variables. Is that the case?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it mean that at each node in the tree, we randomly select m variables from the variable set which is fixed for that tree? Or from the global variable set of the training dataset? And then from the selected set of variables we select 1 variable heuristically (e.g. whichever variable maximises information gain) -- is that a correct statement?&lt;/p&gt;&#xA;" OwnerUserId="30697" LastActivityDate="2017-04-08T22:20:07.387" Title="selecting variable randomly at each node in a tree in Random Forest" Tags="|machine-learning|classification|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18201" PostTypeId="2" ParentId="17216" CreationDate="2017-04-08T04:00:01.630" Score="7" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;PCA reduces dimensionality. It does not change the number of observations you have. Nor does it change the order of the data. The n-th observation in your original dataset will still be the n-th observation post-PCA.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Choosing the number of components in PCA and choosing the number of clusters in K-Means are independent of each other. Both K-Means and PCA seek to &quot;simplify/summarize&quot; the data, but their mechanisms are deeply different. PCA looks to find a low-dimensional representation of the observation that explains a good fraction of the variance. K-Means looks to find homogeneous subgroups among the observations.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For PCA, the optimal number of components is determined visually through the scree plot or mathematically using Kaiser's criterion (drop all components with eigenvalue &amp;lt;1). From my experience the two don't always give the same results, but the difference is negligible.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Screeplot &lt;a href=&quot;http://www.sthda.com/&quot; rel=&quot;noreferrer&quot;&gt;(source)&lt;/a&gt;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/7Dsnv.png&quot; /&gt;&#xA;It does not make much sense to go beyond the 4th component. Law of diminishing marginal returns.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For K-Means: Since increasing the number of clusters will always reduce the distance from centroid to data points, increasing K will always decrease this metric, to the extreme of reaching zero when K is the same as the number of data points. If we plot the average within cluster distance to centroids vs number of clusters, we will find the &quot;elbow&quot; to be a good place to stop.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://www.datascience.com/hs-fs/hubfs/Blog/introduction-to-k-means-clustering-elbow-point-example.png?t=1491601291903&amp;amp;width=760&amp;amp;height=411&amp;amp;name=introduction-to-k-means-clustering-elbow-point-example.png&quot;&gt;&#xA;&lt;a href=&quot;https://www.datascience.com/blog/introduction-to-k-means-clustering-algorithm-learn-data-science-tutorials&quot; rel=&quot;noreferrer&quot;&gt;source&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30890" LastActivityDate="2017-04-08T04:00:01.630" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18202" PostTypeId="1" CreationDate="2017-04-08T10:08:35.800" Score="4" ViewCount="915" Body="&lt;p&gt;I'm currently studying from Andrew Ng's Stanford handouts &lt;a href=&quot;https://see.stanford.edu/materials/aimlcs229/cs229-notes3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; (I'm at part 8). Now from what I gathered from before, all the time our goal was to &lt;em&gt;minimize&lt;/em&gt; ||w||^2 so that we can &lt;em&gt;maximize&lt;/em&gt; the margin. However, now as he's writing about the regularization, he's saying:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The parameter C controls the relative weighting between the twin goals of &lt;strong&gt;making the ||w||^2 large (which we saw earlier makes the margin small)&lt;/strong&gt; and of ensuring that most examples have functional margin at least 1.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why would making the margin &lt;em&gt;small&lt;/em&gt; be currently a &quot;goal&quot;? &lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="21254" LastEditDate="2017-04-08T10:19:34.317" LastActivityDate="2020-10-17T22:36:41.747" Title="SVM regularization - minimizing margin?" Tags="|svm|regularization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18205" PostTypeId="1" CreationDate="2017-04-08T13:34:35.793" Score="1" ViewCount="31" Body="&lt;p&gt;I have a total dataset of, lets say, 1000 Items. Each item will get a response from the crowd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I get all the 1000 responses, I can sample the dataset to calculate accuracy. Sampling would involve checking the responses for items and comparing it with our own knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg: If the task was to tag images as &lt;code&gt;T-Shirts&lt;/code&gt; and &lt;code&gt;Jeans&lt;/code&gt;. Out of 1000 responses, I can sample a few and open the images and see what they are, and then compare them with the response that I got from the crowd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But how can I calculate the accuracy of the total dataset in real-time, as the responses are being submitted?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="30895" LastEditorUserId="11097" LastEditDate="2017-04-08T13:37:39.730" LastActivityDate="2017-04-08T13:37:39.730" Title="Calculate accuracy of crowdsourced responses in realtime" Tags="|sampling|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18206" PostTypeId="1" AcceptedAnswerId="18210" CreationDate="2017-04-08T16:20:09.640" Score="2" ViewCount="3684" Body="&lt;p&gt;I came across this research paper released by YouTube, on how they use deep learning neural networks for recommendations. It's located here: &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an &quot;output embedding&quot; of each of the 1M video classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is this possible to implement in tensorflow, for example? Isn't softmax supposed to be only 1-Dimensional. If the model outputs an &quot;embedding&quot; like this, as they say it does, how would the training data's labels be formatted as 256-dimensional? In other words, how do they compute the 256-dimensional vector for each of the videos in their training dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you so much for your time and help, guys!&lt;/p&gt;&#xA;" OwnerUserId="30900" LastActivityDate="2017-04-09T08:29:59.713" Title="How to create a multi-dimensional softmax output in Tensorflow?" Tags="|machine-learning|python|deep-learning|tensorflow|embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18208" PostTypeId="1" CreationDate="2017-04-08T19:07:54.517" Score="1" ViewCount="78" Body="&lt;p&gt;I have a dataset which belongs to a hospital. It contains data about patients and healthy people. The problem is separating healthy ones from patients. I add some new features to dataset to solve this problem. When I reduce the dimensions of data including the new features and visualize the data, the patient and healthy individuals are distinguishable(visually separable). Now if one asks what is the relation between the used approach (feature extraction, visualization, using the human ability, unsupervised methods) and expert system, what can I say? How can I use this method to build an expert system?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="12345" LastEditDate="2017-04-08T22:30:37.197" LastActivityDate="2017-04-08T23:47:22.610" Title="unsupervised learning in medical systems and intelligent systems?" Tags="|visualization|feature-extraction|unsupervised-learning|knowledge-base|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18210" PostTypeId="2" ParentId="18206" CreationDate="2017-04-08T21:09:02.297" Score="1" Body="&lt;p&gt;You are confusing &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Dimension&quot; rel=&quot;nofollow noreferrer&quot;&gt;dimensions&lt;/a&gt;&quot; with &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Tensor&quot; rel=&quot;nofollow noreferrer&quot;&gt;order of tensor&lt;/a&gt;&quot;. A softmax with 256 different categories is a 256 dimensional vector, but is also a tensor with order 1 (whilst a matrix is a tensor of order 2). The paper is using the technical terms correctly, so the 256 dimensional vector is just a normal vector with 256 scalar entries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore a 256-dimensional softmax in TensorFlow is typically an output layer that looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = tf.nn.softmax(tf.matmul(h, W) + b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;h&lt;/code&gt; is the last hidden layer, &lt;code&gt;W&lt;/code&gt; is the weight matrix n x 256, and &lt;code&gt;b&lt;/code&gt; is the bias 1 x 256 vector.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an &quot;output embedding&quot; of each of the 1M video classes&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is a description of the training process that compresses 1M different inputs to 256-dimensional output for use as an embedding for recommendation matches. The softmax is at the output, and as far as I can see is just a normal softmax classifier output as seen in many other classifier networks (except the result is not technically being used to classify anything). I am not clear on what supervision data was used or on what the input representation was. However, I don't think it likely that 1M &quot;classes&quot; ever appear as e.g. 1-hot encoding, because that would not scale out usefully to the many other millions of videos - the point of the embedding is to turn disparate features of the videos into something that be used as a similarity measure, that can be run on any video stored in YouTube.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-09T08:29:59.713" LastActivityDate="2017-04-09T08:29:59.713" CommentCount="15" ContentLicense="CC BY-SA 3.0" />
  <row Id="18211" PostTypeId="2" ParentId="18199" CreationDate="2017-04-08T22:20:07.387" Score="3" Body="&lt;p&gt;To train a tree in a random forest, at each split m variables are randomly selected from the global variable set, and the variable that maximizes information gain is chosen to be split on. Source: &lt;a href=&quot;https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf&lt;/a&gt; (the original random forest paper).&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-08T22:20:07.387" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18212" PostTypeId="1" CreationDate="2017-04-08T22:43:03.643" Score="1" ViewCount="56" Body="&lt;p&gt;One of our famous mathematicians, James Simons, used an extension of the Baum-Welch algorithm to 'crack' the wall street when he started trading on the stock market. Now, as Google, all informations his team used to trade are hidden to the public. I would like to do such thing, but it'll be very hard. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it should be great to start with using the Baum-Welch Algorithm. Some guys suggest me to look up &lt;a href=&quot;https://arxiv.org/abs/0811.4413&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Spectral Algorithm for Learning Hidden Markov Models&lt;/a&gt; and &lt;a href=&quot;http://www.wisdom.weizmann.ac.il/~nadler/Hmms/learning_pohmm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Parametric Output HMMs&lt;/a&gt;. Another recommend me just starting with deep neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To implement a great model, what are you suggesting to use first? In fact,  I just don't know if I need to learn deep learning first or starting implement my algorithm. What is it advised to learn first?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="30907" LastEditorUserId="30907" LastEditDate="2017-04-09T17:41:00.330" LastActivityDate="2017-04-09T17:41:00.330" Title="Suggestions the way to start" Tags="|machine-learning|deep-learning|markov-process|" AnswerCount="0" CommentCount="3" ClosedDate="2017-04-10T18:19:29.283" ContentLicense="CC BY-SA 3.0" />
  <row Id="18213" PostTypeId="2" ParentId="18208" CreationDate="2017-04-08T23:47:22.610" Score="1" Body="&lt;p&gt;Expert systems rely on knowledge &quot;extracted&quot; from human subject matter experts (e.g. a team of physicians). They then use this knowledge to make prediction about the data (e.g. if blood pressure &gt; x then patient).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your approach used properties of the dataset you worked on. The knowledge was in fact extracted from the data themselves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second approach may or may not be compatible with expert advice. If your algorithm found that blood pressure was a good feature to split healthy people and patient, you will probably find experts agreeing. However, if your main feature comes from a combination of many variables (e.g. PCA) you may have trouble finding an expert validating your approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the way you could use your approach would be to team up with subject matter experts and make sure that you use only features that have a meaning for them (i.e. don't over-engineer data). &lt;/p&gt;&#xA;" OwnerUserId="14003" LastActivityDate="2017-04-08T23:47:22.610" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18214" PostTypeId="2" ParentId="11186" CreationDate="2017-04-09T00:12:50.047" Score="2" Body="&lt;p&gt;Try t-SNE (&lt;a href=&quot;http://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://lvdmaaten.github.io/tsne/&lt;/a&gt;), it can help you group pictures that have the same properties via dimensionality reduction. You can see an example here &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/cnnembed/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs.stanford.edu/people/karpathy/cnnembed/&lt;/a&gt; where t-sne is applied on various imagenet samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will group images with similar content close to each other (dogs near dogs, black dogs near black dogs...). However, if your black dog is yellowish because of the lighting, i suspect it will not be close to the same black dog under normal lighting condition. For that, maybe you can try black and white pictures? or add a black &amp;amp; white layer on top of your RGB input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not experiment it myself, but you may also want to have a look at DeepBit &lt;a href=&quot;http://www.iis.sinica.edu.tw/~kevinlin311.tw/cvpr16-deepbit.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.iis.sinica.edu.tw/~kevinlin311.tw/cvpr16-deepbit.pdf&lt;/a&gt; &quot;an unsupervised deep learning approach to learn compact binary descriptor for efficient visual object matching&quot;.&lt;/p&gt;&#xA;" OwnerUserId="14003" LastEditorUserId="14003" LastEditDate="2017-04-09T17:23:19.497" LastActivityDate="2017-04-09T17:23:19.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18217" PostTypeId="1" CreationDate="2017-04-09T10:28:51.780" Score="1" ViewCount="4487" Body="&lt;p&gt;I know that there are various pre-trained models available for ImageNet (e.g. &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/vgg16.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;VGG 16&lt;/a&gt;, &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inception v3&lt;/a&gt;, &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/resnet50.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Resnet 50&lt;/a&gt;, &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/xception.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Xception&lt;/a&gt;). Is there something similar for the tiny datasets (CIFAR-10, CIFAR-100, SVHN)?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-04-09T18:40:30.640" Title="Are pre-trained models vor CIFAR-10 / CIFAR-100 / SVHN available?" Tags="|machine-learning|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18218" PostTypeId="1" AcceptedAnswerId="18261" CreationDate="2017-04-09T10:58:10.293" Score="1" ViewCount="2400" Body="&lt;p&gt;I'm running a &lt;code&gt;GridSearchCV&lt;/code&gt; with a &lt;code&gt;OneVsRestClasssifer&lt;/code&gt; using &lt;code&gt;SVC&lt;/code&gt; as an estimator. This is the aspect of my &lt;code&gt;Pipeline&lt;/code&gt; and &lt;code&gt;GridSearchCV&lt;/code&gt; parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipeline = Pipeline([&#xA;    ('clf', OneVsRestClassifier(SVC(verbose=True), n_jobs=1)),&#xA;    ])&#xA;&#xA;parameters = {&#xA;    &quot;clf__estimator__C&quot;: [0.1, 1],&#xA;    &quot;clf__estimator__kernel&quot;: ['poly', 'rbf'],&#xA;    &quot;clf__estimator__degree&quot;: [2, 3],&#xA;}&#xA;&#xA;grid_search_tune = GridSearchCV(pipeline, parameters, cv=2, n_jobs=8, verbose=10)&#xA;grid_search_tune.fit(train_x, train_y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to the documentation of SVC the &lt;code&gt;degree&lt;/code&gt; parameter is only used by the &lt;code&gt;poly&lt;/code&gt; kernel:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;degree : int, optional (default=3) &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Degree of the polynomial kernel&#xA;  function (‘poly’). Ignored by all other kernels.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but when I see the output of my &lt;code&gt;GridSearchCV&lt;/code&gt; it seems it's computing a different run for each &lt;code&gt;SVC&lt;/code&gt; configuration with a &lt;code&gt;rbf&lt;/code&gt; kernel and different values for the &lt;code&gt;degree&lt;/code&gt; parameter.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2&#xA;[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2&#xA;[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2&#xA;[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2&#xA;[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3&#xA;[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3&#xA;[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3&#xA;[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Shouldn't all values of degree be ignored, when the kernel is set to &lt;code&gt;rbf&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="6518" LastActivityDate="2017-04-11T06:00:38.540" Title="Is GridSearchCV computing SVC with rbf kernel and different degrees?" Tags="|machine-learning|scikit-learn|svm|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18221" PostTypeId="1" CreationDate="2017-04-09T16:33:00.020" Score="2" ViewCount="3380" Body="&lt;p&gt;what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, but not sure what are the supposed use case for each. Appreciate any advice.&lt;/p&gt;&#xA;" OwnerUserId="11033" LastActivityDate="2020-08-07T11:24:12.100" Title="What are some function/package in R to find similarity of individual words not in the context of sentences?" Tags="|r|nlp|similarity|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18222" PostTypeId="2" ParentId="18221" CreationDate="2017-04-09T17:27:53.160" Score="3" Body="&lt;p&gt;If your intent is to find compare similarity in meaning, word2vec is the only appropriate choice. adist measures the edit distance between two words, and cosine similarity compares the similarity of two documents (treating the documents as bags of words). On the other hand, transforming a word into its word2vec embedding captures some of the latent meaning of the word. See &lt;a href=&quot;https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for a detailed explanation of how this is done.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastEditorUserId="98307" LastEditDate="2020-08-07T11:24:12.100" LastActivityDate="2020-08-07T11:24:12.100" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18223" PostTypeId="2" ParentId="17254" CreationDate="2017-04-09T17:53:12.657" Score="1" Body="&lt;p&gt;With CNNs, a common strategy is to visualize the weights. They are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to have a look at the activation maps, you will have to display them 1 by 1 and they will be grayscale (e.g. you would display 15 AxB grayscale filters for your layer N). Check this site for more ideas: &lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/understanding-cnn/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, it is unclear what you can expect to &quot;see&quot; on layers that are looking at a 200*200*10 cube and what action you can take afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT - Additional link &#xA;This article describe a new technique that tries to show the &quot;meaning&quot; included in the neural network layers: &lt;a href=&quot;https://distill.pub/2018/building-blocks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://distill.pub/2018/building-blocks/&lt;/a&gt; &quot;For instance, by combining feature visualization (what is a neuron looking for?) with attribution (how does it affect the output?), we can explore how the network decides between labels like Labrador retriever and tiger cat.&quot;&lt;/p&gt;&#xA;" OwnerUserId="14003" LastEditorUserId="14003" LastEditDate="2018-03-13T16:13:37.917" LastActivityDate="2018-03-13T16:13:37.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18224" PostTypeId="1" AcceptedAnswerId="18233" CreationDate="2017-04-09T18:22:35.477" Score="-1" ViewCount="1995" Body="&lt;p&gt;I am reading about artificial neural networks and it is said that ANN is used for prediction after training with training data. It is also given that ANN is used for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have data consisting of input as (&lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;, sin(&lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;)) and output as -1 if it is in upper half of sin wave and +1 if it is in lower half. Here is it guaranteed that a trained neural network will always produce output as +1 or -1 (i.e. classify as +1 or -1)? If not, then how is ANN used for classification?&lt;/p&gt;&#xA;" OwnerUserId="30925" LastEditorUserId="29169" LastEditDate="2019-06-10T17:22:26.797" LastActivityDate="2019-06-10T17:22:26.797" Title="How ANN is used for classification?" Tags="|neural-network|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="18225" PostTypeId="2" ParentId="18217" CreationDate="2017-04-09T18:40:30.640" Score="1" Body="&lt;p&gt;Yes, google the various &quot;Model Zoo&quot; (e.g. one for caffe here: &lt;a href=&quot;https://github.com/BVLC/caffe/wiki/Model-Zoo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/BVLC/caffe/wiki/Model-Zoo&lt;/a&gt;), they host a bunch of models including  &quot;tiny&quot; ones.&lt;/p&gt;&#xA;" OwnerUserId="14003" LastActivityDate="2017-04-09T18:40:30.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18226" PostTypeId="1" AcceptedAnswerId="18227" CreationDate="2017-04-09T18:40:31.313" Score="-3" ViewCount="113" Body="&lt;p&gt;I have a collection of data for a multiplayer game (2000 games, 10 players each). I would like to create clusters from this data, each containing the ids of 3 players that had played against each other.&lt;/p&gt;&#xA;" OwnerUserId="30924" LastEditorUserId="30924" LastEditDate="2017-04-18T06:18:16.447" LastActivityDate="2017-04-18T06:18:16.447" Title="Detecting patterns from a collection of data" Tags="|python|" AnswerCount="1" CommentCount="4" ClosedDate="2017-04-10T18:18:43.977" ContentLicense="CC BY-SA 3.0" />
  <row Id="18227" PostTypeId="2" ParentId="18226" CreationDate="2017-04-09T19:41:00.150" Score="1" Body="&lt;p&gt;You can use Python &lt;code&gt;networkx&lt;/code&gt; module to find all 3-cliques:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import networks&#xA;G = nx.Graph() # The clique locator does not work with digraphs&#xA;G.add_edges_from([('A','B'),('A','D'),('A','C'),('B','D'),('C','D'),('D','E')])&#xA;[clique for clique in nx.enumerate_all_cliques(G) if len(clique)==3]    &#xA;#[['B', 'A', 'D'], ['A', 'D', 'C']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finding all clique may take a lot of time and memory. Luckily, &lt;code&gt;nx.enumerate_all_cliques&lt;/code&gt; is a generator that produces smaller cliques first, so you can stop retrieving cliques after you get a clique with more than 3 nodes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cliques=[]&#xA;for c in nx.enumerate_all_cliques(G):&#xA;  if len(c) &amp;lt; 3: continue&#xA;  if len(c) &amp;gt; 3: break&#xA;  cliques.append(c)&#xA;print(cliques)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28743" LastActivityDate="2017-04-09T19:41:00.150" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18231" PostTypeId="2" ParentId="17504" CreationDate="2017-04-10T02:17:14.363" Score="1" Body="&lt;p&gt;Neural networks typically take longer to run as you increase the number of features or columns in your dataset and also when you increase the number of hidden layers. Frameworks like tensorflow or Theano enable you to run your neural networks code on GPU to especially take advantage of the parallel programming capabilities for large array multiplications typical of backpropagation algorithms. My code to train a ConvNet for the Dogs vs Cats problem from kaggle took 50 mins to train on 24000 images. You can take a look at my experiments on the CIFAR10 dataset &lt;a href=&quot;https://github.com/priteshgudge/convnet-tensorflow-cifar10/blob/master/dlnd_image_classification.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30935" LastActivityDate="2017-04-10T02:17:14.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18232" PostTypeId="2" ParentId="18224" CreationDate="2017-04-10T02:27:11.177" Score="-1" Body="&lt;p&gt;If you train your network using a large enough dataset and the design of your model is good for your data, the prediction accuracy can get very high. But like all machine learning algorithms, even deep learning algorithms have the limitation of being able to only predict the outcomes of your use case. Deterministic solutions are not possible due to the inherent design properties of ANNs. You can use a tanH activation function to get outputs in the [-1,+1] range. After a  lot of training it is possible that your network will predict correct results in almost all the cases but it again depends on what threshold you set for your outputs. &lt;/p&gt;&#xA;" OwnerUserId="30935" LastActivityDate="2017-04-10T02:27:11.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18233" PostTypeId="2" ParentId="18224" CreationDate="2017-04-10T03:18:54.780" Score="1" Body="&lt;p&gt;In your question, you have a binary classification problem. I understand what you're asking - you want to know how exactly a network classifies the inputs. Does the network only give (-1, +1) or something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most neural networks don't just say &quot;this is upper or this is lower&quot;. The networks would give you a probability distribution [0..1] for each possible class. The most common implementation is the popular &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;softmax layer&lt;/a&gt;. You'd just choose the class with the highest probability as the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible to encode your network such that it outputs +1 or -1. In fact, we can always add a layer just after the softmax layer to do exactly that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't I recommend you to study logistic regression before tackling neural network.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-04-10T03:18:54.780" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18236" PostTypeId="1" AcceptedAnswerId="18241" CreationDate="2017-04-10T05:40:00.527" Score="2" ViewCount="2104" Body="&lt;p&gt;I want to extract the values of the below text&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Paﬁent Name : Thomas Joseph MRNO : DQ026151?&#xA;Doctor : Haneef M An : 513! Gandar : Male&#xA;Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17&#xA;Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;extract only the values of the field names for example,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thomas Joseph from the field name Pateint name, similarly for others field names and save the output to excel&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python code for the above&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My attempt -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text = pt.image_to_string(img1)&#xA;print(text)&#xA;s = re.findall(r'\s:\s(\w+)', text)&#xA;print (s)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29580" LastEditorUserId="11097" LastEditDate="2017-04-10T05:40:23.840" LastActivityDate="2017-04-10T09:53:10.273" Title="Regular expression in python -" Tags="|python|regex|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18237" PostTypeId="2" ParentId="18236" CreationDate="2017-04-10T07:45:39.947" Score="1" Body="&lt;p&gt;As @spacedman correctly mentioned, this will answered quicker at StackOverflow. But you can use this to create a dictionary like this. There might be a better way but this is a quick work around.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# -*- coding: utf-8 -*-&#xA;import re&#xA;st = '''Pafient Name : Thomas Joseph MRNO : DQ026151?&#xA;Doctor : Haneef M An : 513! Gandar : Male&#xA;Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17&#xA;Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017'''&#xA;st = st.decode('utf-8').replace('\n','')+'&amp;lt;eof&amp;gt;'&#xA;words = ['Pafient Name','MRNO','Doctor','An','Gandar','Admission Data','PM Bill No','Discharge Date','Bill Dale','&amp;lt;eof&amp;gt;']&#xA;print {words[i]:st[st.index(words[i])+len(words[i]):st.index(words[i+1])].replace(':','').strip() for i in range(len(words)-1)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-04-10T07:45:39.947" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18239" PostTypeId="2" ParentId="12851" CreationDate="2017-04-10T08:22:06.583" Score="4" Body="&lt;p&gt;Not per se nifty for papers, but very useful for showing people who don't know a lot of about neural networks what their topology may look like. This &lt;a href=&quot;https://github.com/wagenaartje/neataptic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Javascript library (Neataptic)&lt;/a&gt; lets you visualise your network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://camo.githubusercontent.com/6e033f6c37b57562bbab24f6579bc0fdcbd30544/68747470733a2f2f692e6779617a6f2e636f6d2f35333235636139323137646263613331353161383931373339353438613031642e706e67&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://camo.githubusercontent.com/6e033f6c37b57562bbab24f6579bc0fdcbd30544/68747470733a2f2f692e6779617a6f2e636f6d2f35333235636139323137646263613331353161383931373339353438613031642e706e67&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30928" LastEditorUserId="30928" LastEditDate="2017-05-15T18:41:41.730" LastActivityDate="2017-05-15T18:41:41.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18241" PostTypeId="2" ParentId="18236" CreationDate="2017-04-10T09:40:24.460" Score="1" Body="&lt;p&gt;It may not be perfect but does the job almost. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;re.findall(r'(?&amp;lt;=: )\w{2}-\w{3}-\d{4}|(?&amp;lt;=: )\d{2}-\w{3}-\w{2}|(?&amp;lt;=: )\s?\w+\s?\w+\s?\w+',data)&#xA;&#xA;#['Thomas Joseph MRNO','DQ026151','Haneef M An','513','Male','19-Feb-2V','IDOGIII','22-Feb-20','E2-Feb-2017']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29644" LastEditorUserId="29644" LastEditDate="2017-04-10T09:53:10.273" LastActivityDate="2017-04-10T09:53:10.273" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18249" PostTypeId="1" AcceptedAnswerId="20307" CreationDate="2017-04-10T16:57:01.647" Score="2" ViewCount="189" Body="&lt;p&gt;So I'm reading this paper which uses a &lt;a href=&quot;https://pdfs.semanticscholar.org/8f54/b66017b99c4ee525e91f389f9f6417cf2e88.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;max ent classifier for sentence extraction&lt;/a&gt;. The parametric form for a conditional max ent model is :&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$ P(c|s) = \frac{1}{Z(s)} \exp \sum_i \lambda_if_i(c,s)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$f_i(c,s)$&lt;/span&gt; is a feature which has a weight &lt;span class=&quot;math-container&quot;&gt;$\lambda_i$&lt;/span&gt; associated with it.&lt;/p&gt;&#xA;&lt;p&gt;Now, the paper states that &lt;em&gt;conjugate descent was used to find the optimal set of weights&lt;/em&gt; (page 3) - and this is what I'm unable to comprehend. How to use/apply &lt;strong&gt;conjugate gradient descent&lt;/strong&gt; to calculate the optimal set of weights?&lt;/p&gt;&#xA;" OwnerUserId="30965" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-07-10T15:41:20.540" Title="Constructing a Maximum Entropy Classifier for Sentence Extraction" Tags="|machine-learning|nlp|text-mining|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18250" PostTypeId="1" AcceptedAnswerId="18255" CreationDate="2017-04-10T20:00:07.573" Score="2" ViewCount="151" Body="&lt;p&gt;I am currently trying to build a self adjusting network, such that given any number of inputs, should always provide an output of shape (15,145)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network structure is pretty simple and looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    inputs = 36&#xA;    list_of_input = [Input(shape = (45,5,3)) for i in range(inputs)]&#xA;    list_of_conv_output = []&#xA;    list_of_max_out = []&#xA;    for i in range(splits):&#xA;        list_of_conv_output.append(Conv2D(filters = 145 , kernel_size = (30,3))(list_of_input[i]))&#xA;        list_of_max_out.append((MaxPooling2D(pool_size=(3,2))(list_of_conv_output[i])))&#xA;&#xA;    merge = keras.layers.concatenate(list_of_max_out)&#xA;    #reshape = Reshape((merge.shape[0],merge.shape[3]))(merge)&#xA;&#xA;    dense1 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_1&quot;)(merge)&#xA;    dense2 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;    dense3 = Dense(units = 145 , activation = 'softmax', name = &quot;dense_3&quot;)(dense2)&#xA;&#xA;&#xA;&#xA;    model = Model(inputs = list_of_input , outputs = dense3)&#xA;    model.compile(loss=&quot;sparse_categorical_crossentropy&quot;, optimizer=&quot;adam&quot;)&#xA;&#xA;    print model.summary()&#xA;&#xA;    raw_input(&quot;SDasd&quot;)&#xA;    hist_current = model.fit(x = [train_input[i] for i in range(100)],&#xA;                        y = labels_train_data,&#xA;                        shuffle=False,&#xA;                        validation_data=([test_input[i] for i in range(10)], labels_test_data),&#xA;                        validation_split=0.1,&#xA;                        epochs=150000,&#xA;                        batch_size = 15,&#xA;                        verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It been adjusted for having 36 inputs which would given it an output shape of (15,1,145) - but how can i determine the number of filters, kernel size and pooling size that would give me the desired output size.  The network is supposed to be used for classification, and the output vector of length 15 with classes for each third entry in the first axis (45 = 15*3). the total number of classes is 145, hence output dimension (15,145)&lt;/p&gt;&#xA;" OwnerUserId="26085" LastEditorUserId="29169" LastEditDate="2021-02-16T02:43:55.203" LastActivityDate="2021-02-16T02:43:55.203" Title="Self adjusting CNN network" Tags="|neural-network|classification|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18252" PostTypeId="1" CreationDate="2017-04-11T00:56:39.307" Score="1" ViewCount="23" Body="&lt;p&gt;I have a waveform which looks like this. The peaks are indicated by the points. I want a type of curve fit which best connects these peaks as smoothly as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IOGiQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IOGiQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried polynomial fit but that does not yield a smooth curve. It looks something like this.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yUNI3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yUNI3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the green line is polynomial fit. As you can see it isn't as uniform as I want it to be. Also, poly fit &lt;strong&gt;fails&lt;/strong&gt; to reach the max peak amplitude for some waveforms. So overall it is not what I am looking for. Is there a better fit that can account for these points better?&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-11T00:56:39.307" Title="Which is the best type of curve for these peaks on the waveform?" Tags="|predictive-modeling|optimization|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18254" PostTypeId="1" CreationDate="2017-04-11T01:39:22.897" Score="1" ViewCount="654" Body="&lt;p&gt;The following image shows a scatter plot of my data. The Y axis points are the labels, labeled from 1 to 6 and X-axis are dimensionally reduced values of all my features. I reduced them for better visualization. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/4wg5r.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4wg5r.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used decision Trees and Random forests both give a score of 0.74 and 0.79 respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried SVM and Naive Bayes just for seeing the score they gave 0.37 and 0.33 respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was using 75% to train and the rest to test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the best classification algorithm for this data? Multiclass Classification &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-11T07:57:14.323" Title="Which type of classification best suits this scatter plot?" Tags="|machine-learning|classification|class-imbalance|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18255" PostTypeId="2" ParentId="18250" CreationDate="2017-04-11T01:44:35.997" Score="1" Body="&lt;p&gt;Spatial pyramid pooling layers (&lt;a href=&quot;https://arxiv.org/pdf/1406.4729.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1406.4729.pdf&lt;/a&gt;) should solve this problem for you. These layers allow you to use input images of any dimension, instead of being restricted to 224x224 images, for example.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-11T01:44:35.997" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18258" PostTypeId="1" CreationDate="2017-04-11T03:02:54.080" Score="46" ViewCount="48785" Body="&lt;p&gt;I am using a standard linear regression using scikit-learn in python.&#xA;However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="43000" LastEditDate="2021-03-15T18:31:31.730" LastActivityDate="2021-09-19T01:26:15.777" Title="How to force weights to be non-negative in Linear regression" Tags="|python|scikit-learn|linear-regression|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18259" PostTypeId="2" ParentId="18258" CreationDate="2017-04-11T03:57:36.913" Score="34" Body="&lt;p&gt;What you are looking for, is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Non-negative_least_squares&quot; rel=&quot;noreferrer&quot;&gt;Non-negative least square regression&lt;/a&gt;.&#xA;It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, there is &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/8191&quot; rel=&quot;noreferrer&quot;&gt;no standard implementation&lt;/a&gt; of Non-negative least squares in Scikit-Learn. &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/pull/8428&quot; rel=&quot;noreferrer&quot;&gt;The pull request is still open&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, looks like &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.nnls.html&quot; rel=&quot;noreferrer&quot;&gt;Scipy has implemented the same&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; I haven't tried the scipy version. I found it solely by googling around.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2017-04-11T03:57:36.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18261" PostTypeId="2" ParentId="18218" CreationDate="2017-04-11T06:00:38.540" Score="3" Body="&lt;p&gt;From what I understand, you'll be able to pass different values of degree even when you're using kernels that are not the polynomial kernel, just that it will not be used. I believe the score will come out to be similar for the other kernels even with different degrees. Will you be able to confirm this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid additional computation time due to redundant searches, you can fine tune the GridSearchCV by specifying two grids. Try out the below code and passing it into the param_grid argument.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param_grid = [&#xA;  {'C': [1, 10, 100, 1000], 'kernel': ['linear']},&#xA;  {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},&#xA; ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can explore the GridSearchCV documentation for the specific examples. Take a look at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html#grid-search&quot; rel=&quot;nofollow noreferrer&quot;&gt;example here (look at 3.2.1)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30624" LastActivityDate="2017-04-11T06:00:38.540" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18262" PostTypeId="1" AcceptedAnswerId="18319" CreationDate="2017-04-11T07:18:07.640" Score="1" ViewCount="140" Body="&lt;p&gt;I would like to do classification of multi-source energy (wind/solar/teg) repersented in a time series data. &#xA;My questions are : &#xA;1- What are the most relevent feature that I should chose to do the classification (statistical ones (kurtoisis/means/ variance...) on each sliding window (for experimental purpose) or spectral ones (DWT/FFT)) and which feature selection/extraction method is the best in this case.&#xA;2-What is the best classification method should I chose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="30962" LastEditorUserId="30962" LastEditDate="2017-04-11T08:45:25.463" LastActivityDate="2017-04-14T07:12:40.057" Title="How to deal with time series of multi-source energy in order of classifcation?" Tags="|machine-learning|classification|time-series|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18263" PostTypeId="2" ParentId="18180" CreationDate="2017-04-11T07:25:36.030" Score="1" Body="&lt;p&gt;Can you use a proxy for hardware failure? Something like a state in which failure becomes likely? Then you could try to predict this state which should be more common and use a fudge factor to get to your failures.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-04-11T11:36:59.147" LastActivityDate="2017-04-11T11:36:59.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18264" PostTypeId="1" CreationDate="2017-04-11T10:25:57.020" Score="1" ViewCount="364" Body="&lt;p&gt;I am studying neural networks. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The smaller we make the learning rate, the longer the memory span over which the LMS algorithm remembers past data will be. (Section 3.5(page 103) of Simon Hykin's &quot;Neural Network and Learning Machines&quot;:)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't understand why this is the case. Could someone explain?&lt;/p&gt;&#xA;" OwnerUserId="30994" LastEditorUserId="30928" LastEditDate="2017-04-11T12:12:56.547" LastActivityDate="2017-04-11T13:09:04.487" Title="Learning rate and memory of Least Mean Square algorithm" Tags="|neural-network|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18265" PostTypeId="2" ParentId="18180" CreationDate="2017-04-11T10:39:09.500" Score="1" Body="&lt;p&gt;It depends on what you mean by failure. As el burro has mentioned, if there's a proxy that you can use to act as a label you can potentially still performing ML on your dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no mechanical engineer but I've worked in chemical plants and here are a few suggestions as proxies for pump failure. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pump speed (if it's a fixed speed pump)&lt;/li&gt;&#xA;&lt;li&gt;Rotor torque&lt;/li&gt;&#xA;&lt;li&gt;Power input? (If it's with a variable speed drive)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Perhaps you can try using the above parameters as labels and use regression to forecast the parameter entering unhealthy levels? Of course, ideally regression should be used for interpolation but you can get decent results if your data is not too far off from the failure limits.&lt;/p&gt;&#xA;" OwnerUserId="30624" LastActivityDate="2017-04-11T10:39:09.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18268" PostTypeId="1" CreationDate="2017-04-11T12:00:17.427" Score="2" ViewCount="1206" Body="&lt;p&gt;I am fitting a time series model in R. I plotted the time series to check for the pattern of it. It has 30625 observations. The observations are half hourly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;plot.ts(ts)&lt;/code&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/EFqTH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EFqTH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;I was checking &lt;strong&gt;acf&lt;/strong&gt; plot to see for auto correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;acf(ts)&lt;/code&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/k02WV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k02WV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;If ACF plot decreases slowly i.e. it has auto correlation out to higher number of lags, then it probably needs higher order of differencing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took the logarithm of the series and then checked its acf plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;acf(log(ts))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am getting an error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in plot.window(...) : need finite 'ylim' values&#xA;In addition: Warning messages:&#xA;1: In min(x) : no non-missing arguments to min; returning Inf&#xA;2: In max(x) : no non-missing arguments to max; returning -Inf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does this error means? What should I do remove this error.&#xA;I also think the significance bounds given by dotted line are not correctly calculated. What should I do in these two cases?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30635" LastEditDate="2017-04-13T14:37:00.813" LastActivityDate="2017-12-28T20:55:26.503" Title="acf function shows error while fiiting time series" Tags="|r|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18270" PostTypeId="2" ParentId="18264" CreationDate="2017-04-11T12:15:14.563" Score="2" Body="&lt;p&gt;This should be the &lt;strong&gt;weight update formula&lt;/strong&gt; your model uses.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$W_{n+1} = W_n - \mu\nabla\epsilon[n]$,&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;$\mu$ is the learning rate, and we multiply it with the change we want to do. Obviously, the smaller the learning rate you have, the less update you make, and thus the &quot;longer&quot; the model converges and also the &quot;longer&quot; the model remembers its previous states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think like this, if you set the learning rate to 0, you learn nothing and your parameters are never updated.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="836" LastEditDate="2017-04-11T13:09:04.487" LastActivityDate="2017-04-11T13:09:04.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18271" PostTypeId="1" CreationDate="2017-04-11T14:28:37.590" Score="-1" ViewCount="33" Body="&lt;p&gt;I am working on a scientific article, where I am going to work with a large SQL based database. This work consist of everything from washing data, creating sub-databases etc. How can I document this work, in a scientific matter - so I know what has been done with the data?&lt;/p&gt;&#xA;" OwnerUserId="31002" LastActivityDate="2017-04-11T16:07:23.850" Title="How to document operations on a database?" Tags="|databases|methodology|" AnswerCount="1" CommentCount="0" ClosedDate="2017-04-11T20:14:26.500" ContentLicense="CC BY-SA 3.0" />
  <row Id="18272" PostTypeId="2" ParentId="17279" CreationDate="2017-04-11T15:46:21.047" Score="1" Body="&lt;p&gt;It's important to note that before doing this, they split the frames into individual feature maps.  Note Fig. 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/d8cHl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/d8cHl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;The &quot;45 rows&quot; in the upper left of your diagram (Fig 4.) corresponds to 15 frames times 3 feature maps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The convolution is a 1D convolution, so they are not actually convolving across multiple frames, only across frequencies.  The authors then sum across feature maps in the normal way, meaning summing across frames and features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way of looking at it is in Eq. 9&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XpELW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XpELW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variable $O_i$ is a vector across frequency bands only.  The sum is over feature maps which includes frames. &lt;/p&gt;&#xA;" OwnerUserId="30810" LastActivityDate="2017-04-11T15:46:21.047" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18274" PostTypeId="2" ParentId="18271" CreationDate="2017-04-11T16:07:23.850" Score="2" Body="&lt;p&gt;You could add all SQL statements to GitHub and then use the Commit Descriptions. Depending on how extensive this is you can also write a manual change log. Or you simply excessively comment inside your sql statements. Then I would recommend to also add some kind of timestamp.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are familiar with jupyter and using python and sql together you can also use this. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-04-11T16:07:23.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18278" PostTypeId="1" CreationDate="2017-04-11T20:10:52.717" Score="6" ViewCount="701" Body="&lt;p&gt;I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.&lt;/p&gt;&#xA;&lt;p&gt;I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.&lt;/p&gt;&#xA;&lt;p&gt;The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.&lt;/p&gt;&#xA;&lt;p&gt;A bit more information:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Neural Network Structure:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).&lt;/p&gt;&#xA;&lt;p&gt;I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Genetic Algorithm&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).&lt;/p&gt;&#xA;&lt;p&gt;The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$m' = m * e^{r*0.9} $&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w' = w * m' * r$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Where &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; is the mutation parameter of that specific weight which starts off at 0.05, &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; is a random standard Gaussian number and &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; is the weight.&lt;/p&gt;&#xA;&lt;p&gt;I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.&lt;/p&gt;&#xA;" OwnerUserId="31012" LastEditorUserId="50727" LastEditDate="2020-07-13T12:06:30.703" LastActivityDate="2020-07-13T12:06:30.703" Title="Checkers playing Neural Network evolved with Genetic Algorithm becomes too sensitive to input data changes" Tags="|machine-learning|neural-network|deep-learning|overfitting|genetic-algorithms|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="18279" PostTypeId="1" AcceptedAnswerId="18338" CreationDate="2017-04-11T21:00:01.723" Score="9" ViewCount="1971" Body="&lt;p&gt;Image shows a typical layer somewhere in a feed forward network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fwC9R.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fwC9R.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W_{ij}^{(k)}$ is the weight connecting $i^{th}$ neuron in the $k^{th}$ layer to the $j^{th}$ neuron in the $(k+1)^{th}$ layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j^{(k+1)}$ is the pre-activation function value for the $j^{th}$ neuron in the $(k+1)^{th}$ layer. Sometimes this is called the &quot;logit&quot;, when used with logistic functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The feed forward equations are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_j^{(k+1)} = \sum_i W_{ij}^{(k)}a_i^{(k)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_j^{(k+1)} = f(z_j^{(k+1)})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, bias is included as a dummy activation of 1, and implied used in iterations over $i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can derive the equations for back propagation on a feed-forward neural network, using chain rule and identifying individual scalar values in the network (in fact I often do this as a paper exercise just for practice):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given $\nabla a_j^{(k+1)} = \frac{\partial E}{\partial a_j^{(k+1)}}$ as gradient of error function with respect to a neuron output.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;1. $\nabla z_j^{(k+1)} = \frac{\partial E}{\partial z_j^{(k+1)}} = \frac{\partial E}{\partial a_j^{(k+1)}} \frac{\partial a_j^{(k+1)}}{\partial z_j^{(k+1)}} = \nabla a_j^{(k+1)} f'(z_j^{(k+1)})$&lt;/h3&gt;&#xA;&#xA;&lt;h3&gt;2. $\nabla a_i^{(k)} = \frac{\partial E}{\partial a_i^{(k)}} = \sum_j \frac{\partial E}{\partial z_j^{(k+1)}} \frac{\partial z_j^{(k+1)}}{\partial a_i^{(k)}} = \sum_j \nabla z_j^{(k+1)} W_{ij}^{(k)}$&lt;/h3&gt;&#xA;&#xA;&lt;h3&gt;3. $\nabla W_{ij}^{(k)} = \frac{\partial E}{\partial W_{ij}^{(k)}} = \frac{\partial E}{\partial z_j^{(k+1)}} \frac{\partial z_j^{(k+1)}}{\partial W_{ij}^{(k)}} = \nabla z_j^{(k+1)} a_{i}^{(k)}$&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;So far, so good. However, it is often better to recall these equations using matrices and vectors to represent the elements. I can do that, but I am not able to figure out the &quot;native&quot; representation of the equivalent logic in the middle of the derivations. I can figure out what the end forms should be by referring back to the scalar version and checking that the multiplications have correct dimensions, but I have no idea &lt;em&gt;why&lt;/em&gt; I should put the equations in those forms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there actually a way of expressing the tensor-based derivation of back propagation, using only vector and matrix operations, or is it a matter of &quot;fitting&quot; it to the above derivation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using column vectors $\mathbf{a}^{(k)}$, $\mathbf{z}^{(k+1)}$, $\mathbf{a}^{(k+1)}$ and weight matrix $\mathbf{W}^{(k)}$ plus bias vector $\mathbf{b}^{(k)}$, then the feed-forward operations are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{z}^{(k+1)} = \mathbf{W}^{(k)}\mathbf{a}^{(k)} + \mathbf{b}^{(k)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{a}^{(k+1)} = f(\mathbf{z}^{(k+1)})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then my attempt at derivation looks like this:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;1. $\nabla \mathbf{z}^{(k+1)} = \frac{\partial E}{\partial \mathbf{z}^{(k+1)}} = ??? = \nabla \mathbf{a}^{(k+1)} \odot f'(\mathbf{z}^{(k+1)})$&lt;/h3&gt;&#xA;&#xA;&lt;h3&gt;2. $\nabla \mathbf{a}^{(k)} = \frac{\partial E}{\partial \mathbf{a}^{(k)}} = ??? = {\mathbf{W}^{(k)}}^{T} \nabla \mathbf{z}^{(k+1)}$&lt;/h3&gt;&#xA;&#xA;&lt;h3&gt;3. $\nabla \mathbf{W}^{(k)} = \frac{\partial E}{\partial \mathbf{W}^{(k)}} = ??? =  \nabla\mathbf{z}^{(k+1)} {\mathbf{a}^{(k)}}^T $&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Where $\odot$ represents element-wise multiplication. I've not bothered showing equation for bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where I have put ??? I am not sure of the correct way to go from the feed-forward operations and knowledge of linear differential equations to establish the correct form of the equations? I could just write out some partial derivative terms, but have no clue as to why some should use element-wise multiplication, others matrix multiplication, and why multiplication order has to be as shown, other than clearly that gives the correct result in the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not even sure if there is a purely tensor derivation, or whether it is all just a &quot;vectorisation&quot; of the first set of equations. But my algebra is not that good, and I'm interested to find out for certain either way. I feel it might do me some good comprehending work in e.g. TensorFlow if I had a better native understanding of these operations by thinking more with tensor algebra.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Sorry about ad-hoc/wrong notation. I understand now that $\nabla a_j^{(k+1)}$ is more properly written $\nabla_{a_j^{(k+1)}}E$ thanks to Ehsan's answer. What I really wanted there is a short reference variable to substitute into the equations, as opposed to the verbose partial derivatives.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-13T20:00:20.547" LastActivityDate="2017-04-19T19:56:15.057" Title="Deriving backpropagation equations &quot;natively&quot; in tensor form" Tags="|backpropagation|theory|linear-algebra|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18280" PostTypeId="1" AcceptedAnswerId="18305" CreationDate="2017-04-11T21:14:18.453" Score="2" ViewCount="188" Body="&lt;p&gt;Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My plan is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use PCA and convert this to 2 dim data. Plot this data. See if they're separable enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the above plan sound okay for testing to see if features are any good?&lt;/p&gt;&#xA;" OwnerUserId="31016" LastActivityDate="2017-04-13T00:39:34.043" Title="I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?" Tags="|dataset|dimensionality-reduction|pca|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18282" PostTypeId="2" ParentId="18280" CreationDate="2017-04-11T23:38:06.510" Score="3" Body="&lt;p&gt;By good I assume you're trying to see if the features are different enough? It depends on your data set actually. As smallchess has pointed out, PCA is commonly used and may work with your dataset. Try it out and see if the different classes are distinct? If not, you can try using higher dimensional visualisation tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, with PCA you're reducing the dimensions to new features that may or may not be intuitive in meaning. That makes interpretation more complicated. If you're looking to visualise data, why not go with higher dimension visualisations? In case you're not aware of the different available tools..&#xA;&lt;a href=&quot;https://www.quora.com/Whats-the-best-way-to-visualize-high-dimensional-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is a question with great answers on Quora about higher dimension visualisation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of the tools recommended, Andrews plot and parallel coordinates plots are very popular. I would suggest using these tools instead of using PCA and subsequently plotting for the purpose you're describing. If you have no need to visualise the data, sometimes you can just get a statistical summary/description of the datasets (mean, variance, min, max, etc). Below is an example of an Andrews Plot. You can plot the various classes into different colours to see if there's any discernible differences between the classes. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/Cw4vI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Cw4vI.png&quot; alt=&quot;Andrews Plot example on Matlab documentation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's not what you're looking for, to visualise the data to gauge the representation, just proceed to try out and build a model with the features and see if you get a good score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Here are some shortcomings of PCA..if there's information that's misrepresented, do let me know!&#xA;One thing to note is that with PCA, its transformation is orthogonal. Sometimes your data requires non-orthogonal representation. This is very dependent on how your data looks like. To give an example, take a look at the diagram below.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aft5o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aft5o.png&quot; alt=&quot;Example of PCA shortcomings&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example, PCA fails to find PCs that maximises the variance due to orthogonality. Independent Factor Analysis may work better. Also, if you're planning to use the transformed data to build your model upon, there may be limitations with the linearly transformed data. Sometimes we need non-linear PCs. See the below example from the scikit webpage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zxCuL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zxCuL.png&quot; alt=&quot;Scikit Kernel PCA&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such a case, a kernel PCA (utilizing a kernel trick similar in SVMs) may work better by getting a projection to get better linear distinction.&lt;/p&gt;&#xA;" OwnerUserId="30624" LastEditorUserId="30624" LastEditDate="2017-04-13T00:39:34.043" LastActivityDate="2017-04-13T00:39:34.043" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18283" PostTypeId="1" CreationDate="2017-04-11T23:40:54.110" Score="0" ViewCount="694" Body="&lt;p&gt;For some reason am I getting an uexpected output dimension for my classification network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network has 18 inputs of shape (45,5,3) - data formatted as (samples,45,5,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the output is a vector of length 15 - one class for each third of 45.  The extracted classes comes from a pool of 145 classes. - data formatted as (samples,15)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My network looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#stride = 2&#xA;#dim = 40&#xA;#window_height = 5&#xA;#splits = ((40-5)+1)/2 = 18&#xA;&#xA;kernel_number = int(math.ceil(splits))&#xA;list_of_input = [Input(shape = (45,5,3)) for i in range(splits)]&#xA;list_of_conv_output = []&#xA;list_of_max_out = []&#xA;for i in range(splits):&#xA;    list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (int(splits-3),3))(list_of_input[i]))&#xA;    list_of_max_out.append((MaxPooling2D(pool_size=((2,2)))(list_of_conv_output[i])))&#xA;&#xA;merge = keras.layers.concatenate(list_of_max_out)&#xA;print merge.shape&#xA;reshape = Reshape((15,324))(merge)&#xA;&#xA;dense1 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_1&quot;)(reshape)&#xA;dense2 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;dense3 = Dense(units = 145 , activation = 'softmax', name = &quot;dense_3&quot;)(dense2)&#xA;model = Model(inputs = list_of_input ,outputs = dense3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But for some reason am I getting an error when I passing my output data. &#xA;It is currently stored as numpy.ndarray of shape (16828,15) and I get an value error stating: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error when checking model target: expected dense_3 to have 3 dimensions, but got array with shape (16828, 15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why is it expecting 3 dim rather than 2 dim?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model summary clearly indicates that the output dim is (15,145) as I also would expect?  15 classes from a pool of 145 classes. right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model summary: &#xA;&lt;a href=&quot;https://pastebin.com/27YTQW2m&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pastebin.com/27YTQW2m&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26085" LastEditorUserId="75157" LastEditDate="2021-02-16T09:25:53.033" LastActivityDate="2021-02-16T09:25:53.033" Title="Pass data to CNN with multiple outputs in keras" Tags="|python|classification|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18284" PostTypeId="2" ParentId="16888" CreationDate="2017-04-12T00:58:11.217" Score="2" Body="&lt;p&gt;You can use pymc3. I am pretty sure it works for all the 3 requirements.&#xA;&lt;a href=&quot;http://pymc-devs.github.io/pymc3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pymc-devs.github.io/pymc3/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31021" LastEditorUserId="31021" LastEditDate="2019-03-05T19:05:09.680" LastActivityDate="2019-03-05T19:05:09.680" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18286" PostTypeId="1" AcceptedAnswerId="18290" CreationDate="2017-04-12T06:12:12.827" Score="2" ViewCount="3608" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If you are allowed to choose the features by hand and if you use&#xA;  enough features, you can do almost anything.For binary input vectors,&#xA;  we can have a separate feature unit for each of the exponentially many&#xA;  binary vectors and so we can make any possible discrimination on&#xA;  binary input vectors.This type of table look-up won’t generalize.But&#xA;  once the hand-coded features have been determined, there are very&#xA;  strong limitations on what a perceptron can learn.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is what Hinton explains in his Neural Networks course but I don't get the binary input example and why it is a table look-up type problem and why it won't generalize? What does he mean by hand generated features? I understand that perceptrons cannot classify non-linear data but I cannot relate this to &lt;a href=&quot;http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;his slide (slide 26)&lt;/a&gt;. It would be nice if anybody explains this with proper example.&lt;/p&gt;&#xA;" OwnerUserId="28317" LastEditorUserId="45507" LastEditDate="2018-01-31T18:35:59.000" LastActivityDate="2018-01-31T18:35:59.000" Title="Limitations of Perceptron" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18289" PostTypeId="1" AcceptedAnswerId="18307" CreationDate="2017-04-12T07:26:46.357" Score="5" ViewCount="6114" Body="&lt;p&gt;I am trying to identify a ML technique to score products based on the number of times the product was &quot;viewed&quot;, &quot;clicked&quot; and knowing the &quot;cost per click&quot; for each product. Given the product ID and category ID, how can I proceed to score each product?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure I have to coarse classify them (some have no clicks, but views, some have both, some have none)? Since there are 1000s of products... Any tip?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the technique is also used in e-commerce to design recommender systems, like based on popularity of a product. Any one can shed some light?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Edit: Though the suggestions here are interesting, still I couldn't figure out best way to do this. &lt;/p&gt;&#xA;" OwnerUserId="31017" LastEditorUserId="31017" LastEditDate="2017-04-22T10:40:26.517" LastActivityDate="2017-04-22T10:40:26.517" Title="Which machine learning technique for product ranking/scoring" Tags="|machine-learning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18290" PostTypeId="2" ParentId="18286" CreationDate="2017-04-12T07:33:44.913" Score="3" Body="&lt;p&gt;The slide explains a limitation which applies to any linear model. It would equally apply to linear regression for example. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What does he mean by hand generated features? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This means any features generated by analysis of the problem. For instance if you wanted to categorise a building you might have its height and width. A hand generated feature could be deciding to multiply height by width to get floor area, because it looked like a good match to the problem.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't get the binary input example and why it is a table look-up type problem and why it won't generalize?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A table look-up solution is just the logical extreme of this approach. If you have a really complex classification, and your raw features don't relate directly (as a linear multiple of the target), you can craft very specific manipulations of them that give just the right answer for each input example. Essentially this is the same as marking each example in your training data with the correct answer, which has the same structure, conceptually, as a table of &lt;code&gt;input: desired output&lt;/code&gt; with one entry per example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact this &lt;em&gt;might&lt;/em&gt; generalize, but only exactly as well as the crafted features do. In practice, when you have a complex problem and sample data that only partially explains your target variable (i.e. in most data science scenarios), then generating derived features until you find some that explain the data is strongly related to &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;overfitting&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your comment:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In his video lecture, he says &quot;Suppose for example we have binary input vectors. And we create a separate feature unit that gets activated by exactly one of those binary input vectors. We'll need exponentially many feature units. But now we can make any possible discrimination on binary input vectors. So for binary input vectors, there's no limitation if you're willing to make enough feature units.&quot; 1.What feature? 2.Why are we creating this feature? And why adding exponential such features we can discriminate these vectors?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is an example of the scheme that Geoffrey Hinton describes. Say you have 4 binary features, associated with one target value and see the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data 0 1 1 0 -&amp;gt; class 1&#xA;data 1 1 1 0 -&amp;gt; class 2 &#xA;data 0 1 0 1 -&amp;gt; class 1&#xA;data 1 1 1 0 -&amp;gt; class 2&#xA;data 0 1 1 1 -&amp;gt; class 2&#xA;data 0 1 0 0 -&amp;gt; class 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is possible to get a perceptron to predict the correct output values by crafting features as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data 0 1 1 0 -&amp;gt; features 1 0 0 0 0 -&amp;gt; class 1&#xA;data 1 1 1 0 -&amp;gt; features 0 1 0 0 0 -&amp;gt; class 2&#xA;data 0 1 0 1 -&amp;gt; features 0 0 1 0 0 -&amp;gt; class 1&#xA;data 1 1 1 0 -&amp;gt; features 0 1 0 0 0 -&amp;gt; class 2&#xA;data 0 1 1 1 -&amp;gt; features 0 0 0 1 0 -&amp;gt; class 2&#xA;data 0 1 0 0 -&amp;gt; features 0 0 0 0 1 -&amp;gt; class 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each unique set of original data gets a new one-hot-encoded category assigned. It is clear that ultimately if you had $n$ original features, you would need $2^n$ such derived categories - which is an exponential relationship to $n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working like this, there is no generalisation possible, because any pattern you had not turned into a derived feature and learned the correct value for would not have any effect on the perceptron, it would just be encoded as all zeroes. However, it would learn to fit the training data very well, it could just associate each unique vector with a weight equal to the training output - this is effectively a table lookup.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole point of this description is to show that hand-crafted features to &quot;fix&quot; perceptrons are not a good strategy. Even though they can be made to work for training data, ultimately you would be fooling yourself.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-13T07:10:16.783" LastActivityDate="2017-04-13T07:10:16.783" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18291" PostTypeId="2" ParentId="18280" CreationDate="2017-04-12T07:52:41.000" Score="3" Body="&lt;p&gt;I think &lt;a href=&quot;https://en.m.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt; can be useful to visualize high dimensional data by 2D or 3D projection. As opposed to PCA,  t-SNE learns a non-linear transformation which preserves the global data structure. Sklearn provides an easy to use implementation of t-SNE in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15885" LastActivityDate="2017-04-12T07:52:41.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18292" PostTypeId="1" AcceptedAnswerId="18294" CreationDate="2017-04-12T08:57:49.060" Score="-2" ViewCount="48" Body="&lt;p&gt;A data scientist cautioned me against &quot;just amplifying noise&quot; in a data analysis. What did he mean? Can you explain and/or point to resources explaining the problem? &lt;/p&gt;&#xA;" OwnerUserId="28012" LastActivityDate="2017-04-12T09:54:03.470" Title="The danger of &quot;amplifying noise&quot;" Tags="|statistics|" AnswerCount="1" CommentCount="1" ClosedDate="2017-04-12T10:04:31.047" ContentLicense="CC BY-SA 3.0" />
  <row Id="18293" PostTypeId="2" ParentId="16888" CreationDate="2017-04-12T09:01:58.287" Score="1" Body="&lt;p&gt;Not a library, but a interactive GUI based tool is &lt;a href=&quot;http://reasoning.cs.ucla.edu/samiam/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;samiam&quot;&lt;/a&gt; (Sensitivity Analysis Modeling Inference and More) from a research group at UCLA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure about your &quot;continuous PDFs&quot; requirement, whether it's possible to define them inside the samiam GUI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;samiam is free to download, but registration is required. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The size of the software is small, but java-based (ok, the jvm is not &lt;em&gt;that&lt;/em&gt; small). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For API-access, you might call functions inside the &lt;code&gt;inflib.jar&lt;/code&gt; file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There also exists a &quot;Batch tool&quot; and a &quot;Code Bandit&quot; (code generator). Haven't used any of them.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-04-12T09:01:58.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18294" PostTypeId="2" ParentId="18292" CreationDate="2017-04-12T09:54:03.470" Score="1" Body="&lt;p&gt;There is very little information in this question. I will try to answer this in the most generic sense. &lt;strong&gt;Let's start by defining Noise.&lt;/strong&gt; Noise here as you probably know is unwanted data. Any data which you are not looking for while evaluating a problem or scenario can be considered as noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Examples for amplifying noise:&lt;/strong&gt;&#xA;Amplifying noise might occur in cases and scenarios where there is a small data set and you are trying to supersample the dataset or another example could be while working with waveforms. In order to detect weaker signals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Disadvantages of Amplifying noise&lt;/strong&gt;&#xA;The biggest disadvantage of amplifying noise from a data science perspective is that the model used to perform various operations on the data such as Regression, Classification etc will be less efficient. For example having noise based on supersampling in Classification may affect the model. If we were to use decision trees for classification you might create a bias in the algorithm which just pertains to noise while training. So your accuracy for classification also decreases. Similarly, in regression when you train with noise you might choose a wrong model because the noise alters the goodness of the fit.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-12T09:54:03.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18295" PostTypeId="1" AcceptedAnswerId="18340" CreationDate="2017-04-12T09:58:27.700" Score="3" ViewCount="1722" Body="&lt;p&gt;I have used an &quot;adabag&quot;(boosting + bagging) model on an imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best results I got were: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ROC= 0.711 &lt;/li&gt;&#xA;&lt;li&gt;SENS=0.94 &lt;/li&gt;&#xA;&lt;li&gt;SPEC=0.21&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The results aren't Inhofe, especially the bad specificity.&#xA;Any suggestion on how to improve the result? Can the optimization be improved, or would the addition of a penalty term help? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ctrl &amp;lt;- trainControl(method = &quot;cv&quot;,&#xA;                     number = 5,&#xA;                     repeats = 2, &#xA;                     p = 0.80,&#xA;                     search = &quot;grid&quot;, &#xA;                     initialWindow = NULL, &#xA;                     horizon = 1,&#xA;                     fixedWindow = TRUE,&#xA;                     skip = 0,&#xA;                     verboseIter = FALSE,&#xA;                     returnData = TRUE,&#xA;                     returnResamp = &quot;final&quot;,&#xA;                     savePredictions = &quot;all&quot;,&#xA;                     classProbs = TRUE,&#xA;                     summaryFunction = twoClassSummary,&#xA;                     preProcOptions = list(thresh = 0.80, ICAcomp = 3, k = 7, freqCut = 90/10,uniqueCut = 10, cutoff = 0.2),&#xA;                     sampling = &quot;smote&quot;,&#xA;                     selectionFunction = &quot;best&quot;,&#xA;                     index = NULL,&#xA;                     indexOut = NULL,&#xA;                     indexFinal = NULL,&#xA;                     timingSamps = 0,&#xA;                     predictionBounds = rep(FALSE, 2),&#xA;                     seeds = NA,&#xA;                     adaptive = list(min = 5,alpha = 0.05, method = &quot;gls&quot;, complete = TRUE),&#xA;                     trim = FALSE,&#xA;                     allowParallel = TRUE)&#xA;&#xA;grid &amp;lt;- expand.grid(maxdepth = 25, mfinal = 4000)&#xA;&#xA;classifier &amp;lt;- train(x = training_set[,-1],y = training_set[,1], method = 'AdaBag',trControl = ctrl,metric = &quot;ROC&quot;,tuneGrid = grid)&#xA;prediction &amp;lt;- predict(classifier, newdata= test_set,'prob')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;plot from classifierplots package:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9K8CX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9K8CX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried xgboost as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gbmGrid &amp;lt;- expand.grid(nrounds = 50, eta = 0.3,max_depth = 3,gamma = 0,colsample_bytree=0.6,min_child_weight=1,subsample=0.75)&#xA;&#xA;ctrl &amp;lt;- trainControl(method = &quot;cv&quot;,&#xA;                     number = 10,&#xA;                     repeats = 2, &#xA;                     p = 0.80,&#xA;                     search = &quot;grid&quot;, &#xA;                     initialWindow = NULL, &#xA;                     horizon = 1,&#xA;                     fixedWindow = TRUE,&#xA;                     skip = 0,&#xA;                     verboseIter = FALSE,&#xA;                     returnData = TRUE,&#xA;                     returnResamp = &quot;final&quot;,&#xA;                     savePredictions = &quot;all&quot;,&#xA;                     classProbs = TRUE,&#xA;                     summaryFunction = twoClassSummary,&#xA;                     sampling = &quot;smote&quot;,&#xA;                     selectionFunction = &quot;best&quot;,&#xA;                     index = NULL,&#xA;                     indexOut = NULL,&#xA;                     indexFinal = NULL,&#xA;                     timingSamps = 0,&#xA;                     predictionBounds = rep(FALSE, 2),&#xA;                     seeds = NA,&#xA;                     adaptive = list(min = 5,alpha = 0.05, method = &quot;gls&quot;, complete = TRUE),&#xA;                     trim = FALSE,&#xA;                     allowParallel = TRUE)&#xA;&#xA;classifier &amp;lt;- train(x = training_set[,-1],y = training_set[,1], method = 'xgbTree',metric = &quot;ROC&quot;,trControl = ctrl,tuneGrid = gbmGrid)&#xA;prediction &amp;lt;- predict(classifier, newdata= test_set[,-1],'prob')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;plot from classifierplots package:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KhSlQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KhSlQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried asymmetric adaboost, this is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_weights &amp;lt;- ifelse(training_set$readmmited == &quot;yes&quot;,&#xA;                        (1/table(training_set$readmmited)[1]) * 0.4,&#xA;                        (1/table(training_set$readmmited)[2]) * 0.6)&#xA;&#xA;&#xA;ctrl &amp;lt;- trainControl(method = &quot;repeatedcv&quot;,&#xA;                     number = 5,&#xA;                     repeats = 2, &#xA;                     search = &quot;grid&quot;, &#xA;                     returnData = TRUE,&#xA;                     returnResamp = &quot;final&quot;,&#xA;                     savePredictions = &quot;all&quot;,&#xA;                     classProbs = TRUE,&#xA;                     summaryFunction = twoClassSummary,&#xA;                     selectionFunction = &quot;best&quot;,&#xA;                     allowParallel = TRUE)&#xA;&#xA;classifier &amp;lt;- train(x = training_set[,-1],y = training_set[,1], method = 'ada',trControl = ctrl,metric = &quot;ROC&quot;,weights = model_weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but the specificity is zero, what am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="31038" LastEditorUserId="14372" LastEditDate="2017-05-19T23:16:07.690" LastActivityDate="2017-05-19T23:16:07.690" Title="Improving classifier performances in R for imbalanced dataset" Tags="|machine-learning|r|xgboost|class-imbalance|ensemble-modeling|" AnswerCount="2" CommentCount="16" ContentLicense="CC BY-SA 3.0" />
  <row Id="18296" PostTypeId="2" ParentId="15583" CreationDate="2017-04-12T10:38:06.657" Score="2" Body="&lt;p&gt;I think you should have a look at t-SNE which is a visualization technique based on dimensionality reduction. &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding#cite_note-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia excerpt&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;t-distributed stochastic neighbor embedding (t-SNE) is a machine learning algorithm for dimensionality reduction developed by Geoffrey Hinton and Laurens van der Maaten. It is a nonlinear dimensionality reduction technique that is particularly well-suited for embedding high-dimensional data into a space of two or three dimensions, which can then be visualized in a scatter plot. Specifically, it models each high-dimensional object by a two- or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wzX7N.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wzX7N.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1607.00534.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper&lt;/a&gt; describes a python implementation of the technique on various word2vec datasets.&lt;/p&gt;&#xA;" OwnerUserId="14003" LastEditorUserId="98307" LastEditDate="2020-08-11T16:13:00.043" LastActivityDate="2020-08-11T16:13:00.043" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18298" PostTypeId="2" ParentId="18280" CreationDate="2017-04-12T12:37:41.990" Score="0" Body="&lt;p&gt;There are very good suggestions made above regarding the problem. In another approach, (somewhat similar but not same as PCA based menthods mentioned above) is to use an  &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;compressional autoencoder network&lt;/a&gt; and train it classwise (or one autoencoder). The compression can be from multi-dimension to 2 dimensions and can be easily plotted. Since the autoencoder(s) would learn to represent different classes, if the features selected for the original dataset can partition the data space adequately, then in a 2D scatter plot the differences could be made out.&lt;/p&gt;&#xA;" OwnerUserId="26814" LastActivityDate="2017-04-12T12:37:41.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18300" PostTypeId="1" AcceptedAnswerId="18329" CreationDate="2017-04-12T13:46:01.050" Score="3" ViewCount="1274" Body="&lt;p&gt;So I am currently working on a project for &quot;sales people&quot;. Essentially, the user is giving us a plethora of information related to deals they are trying to win. We have data related to the price of the deal, margin of the deal, number of people working on the deal, caliber of the people working on the deal, and the risk of the deal. I want to assign a &quot;score&quot; to these deals based on the given metrics and visualize how the score is calculated in a simple way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be very easy to visualize this all as a radar (spider) chart (I'm aware that a traditional radar chart has limited power since the ordering of metrics is arbitrary and the area formed by the spider chart doesn't hold much value) - but why can't I take the results from a correlation matrix or regression output to weight the metrics in some way and/or oder them based on importance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example - let's say I've built a radar chart for NBA players based on Points per game, assists per game, player efficiency, etc. If I had a logistic regression output to determine the MVP, I would use the output to order and weight the variables (based on p-values?) to build the radar chart. If I then took statistics from the most recent NBA season and plotted each NBA player, wouldn't the player with the biggest area be one of the most valuable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't seem to find much information on building weighted radar charts - although I have seen that percentiles can be used for the actual metrics. Does anybody have any advice or alternative solutions?&lt;/p&gt;&#xA;" OwnerUserId="31045" LastEditorUserId="31513" LastEditDate="2017-05-24T10:06:54.210" LastActivityDate="2017-05-24T10:06:54.210" Title="Is there a way to calculate value from a radar (spider) chart?" Tags="|statistics|regression|visualization|correlation|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18301" PostTypeId="2" ParentId="14677" CreationDate="2017-04-12T13:48:19.990" Score="3" Body="&lt;p&gt;In a similar situation, -after trying some alternatives- I had to build a language classifier in front of all learning and classification steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, for learning:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Detect the language of the input (say, an enumeration like &quot;DE&quot;, &quot;EN&quot;, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Apply language specific stemming to the words of the input.&lt;/li&gt;&#xA;&lt;li&gt;Prepend words in the learning phase with the language identifier (i.e. &quot;de_du&quot;, &quot;en_you&quot;)&lt;/li&gt;&#xA;&lt;li&gt;Use these words in a single training model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In classification phase, use the same steps.&lt;/p&gt;&#xA;" OwnerUserId="18055" LastActivityDate="2017-04-12T13:48:19.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18303" PostTypeId="1" CreationDate="2017-04-12T14:38:06.077" Score="0" ViewCount="65" Body="&lt;p&gt;I have a fixed amount of dataset, say text data with 100,000 records, to rank classifiers, such as gradient boosting, random forest, and other ones, according to their classification performance. The dataset is divided randomly for training and validation but the issue is that I get different prediction performance even if I use the same dataset overtime. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that once the classifiers are ranked after the first training and validation run on the dataset, the rank should always show the same way as the previous rank. I don't add or delete the fixed amount of dataset that I use. Even though the same dataset is divided randomly, what I think is there shouldn't be a significant impact on the performance because the ratio of data used between training and validation is 80(training):20(validation). If they change their behaviors each time I run on the same unchanged fixed amount of data, how can I tell which one shows better performance on the current dataset? I understand the training data changes but its percentage is 80% of the dataset.  &lt;/p&gt;&#xA;" OwnerUserId="30856" LastEditorUserId="30856" LastEditDate="2017-04-13T05:00:14.277" LastActivityDate="2017-04-13T07:59:49.727" Title="Is it normal for some classifiers to change their prediction performance from fixed amount of data each time I run them to train?" Tags="|classification|data-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18304" PostTypeId="2" ParentId="16455" CreationDate="2017-04-12T14:59:28.237" Score="2" Body="&lt;p&gt;Technically this is possible but most approaches combine localization with some sort of classification so that the network can get better at scoring regions, and also removes the problem of having to use the [0, 0, 0, 0] vector to encode a null region since if the classifier associated with the region gives a low probability for that object you can ignore the bounding box. That way the bounding box regression can focus on finding regions that look like they contain objects, not their type. Note that the classifier needs to be trained so that it recognizes a non-object region (ie, it must have a null/garbage/background class).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/object_localization_and_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Object localization and detection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1312.6082.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I don't have enough reputation to post more links but you should also google the R-CNN paper &quot;Rich feature hierarchies for accurate object detection and semantic segmentation&quot;)&lt;/p&gt;&#xA;" OwnerUserId="31047" LastActivityDate="2017-04-12T14:59:28.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18305" PostTypeId="2" ParentId="18280" CreationDate="2017-04-12T15:04:04.233" Score="2" Body="&lt;p&gt;This was going to be a comment but it grew to an answer.  I think there should be some clarification because the question itself is not specifically about visualization but checking the &quot;integrity&quot; of the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA will work for make a generalization of the dataset as a whole.  It is a very standard starting point for exploring data.   If the first 2 components do show a clear separation, this is a pretty solid indication that at least some projection of your data can well represent your classes.  So, &lt;strong&gt;short answer is yes&lt;/strong&gt;.  If the first 2 components don't show separation, that does not mean that the features are necessarily bad, it just means that the first two components do not explain the majority of the variability in the dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature selection is used to check the integrity/importance/value of individual features and how they affect the model.  You can use random forest gini importance to rank your features or lasso regularization to with cross-validation to find out how individual features are weighted in a log. reg. model (this does require a little more work as the weightings are not necessarily an exact measurement of variable importance). Feature selection and cross-validation are the most direct ways of determining feature integrity.  PCA is mostly a good first pass and helpful visualization.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-04-12T15:04:04.233" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18307" PostTypeId="2" ParentId="18289" CreationDate="2017-04-12T15:08:18.590" Score="5" Body="&lt;p&gt;So I am assuming you just want to be pushed in the right direction. There are 2 different ways you can go about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Netflix up until very recently did all its recommendations using classical algorithms and setups, see paper on their architecture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this type of light recommendation problem I would recommend using something from PredictionIO. It is very versatile and can be used to classify using a variety of inputs. It's also not very hard to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also solve this problem using neural nets, it can be viewed as a recommendation by classification. Deep learning is all the jazz now and you can utilize these breakthroughs in the recommender space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Youtube is the big one when it comes to deep neural nets applied to recommendations, see &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They split their system into 2 separate neural net models. One for candidate generation, and then another for producing the actual recommendations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spotify also did some awesome stuff applying Convolutional Neural Nets to the actual audio streams with some equally interesting results: &#xA;&lt;a href=&quot;http://benanne.github.io/2014/08/05/spotify-cnns.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://benanne.github.io/2014/08/05/spotify-cnns.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as implementing something like that goes I would look for examples and build in python using either tensorflow or theano and keras. It wouldn't have to be too 'deep'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helped! &lt;/p&gt;&#xA;" OwnerUserId="31046" LastEditorUserId="31046" LastEditDate="2017-04-12T20:48:35.890" LastActivityDate="2017-04-12T20:48:35.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18308" PostTypeId="1" CreationDate="2017-04-12T15:10:57.540" Score="2" ViewCount="6003" Body="&lt;p&gt;Like above, I'd like to know, what exactly a &lt;strong&gt;skewed&lt;/strong&gt; dataset is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The explanation from &lt;a href=&quot;https://stats.stackexchange.com/questions/231786/skewed-dataset-performance-measurement-in-machine-learning&quot;&gt;stats.se.com&lt;/a&gt; sounds to me more like what I call an &lt;em&gt;imbalanced&lt;/em&gt; dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the distinction?&lt;/p&gt;&#xA;" OwnerUserId="31050" LastEditorUserId="-1" LastEditDate="2017-04-13T12:44:20.183" LastActivityDate="2019-07-14T02:30:23.930" Title="What exactly is a skewed dataset ?" Tags="|dataset|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18309" PostTypeId="2" ParentId="18308" CreationDate="2017-04-12T16:07:39.410" Score="1" Body="&lt;p&gt;In the context of the link, a skewed data set is referring to a dataset with a class imbalance problem. They are trying to build a classifier, but they have many more negative examples than positive examples. It's not a very precise term, but I've heard to used in this context a few times.&lt;/p&gt;&#xA;" OwnerUserId="31052" LastActivityDate="2017-04-12T16:07:39.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18310" PostTypeId="2" ParentId="18303" CreationDate="2017-04-12T16:12:31.267" Score="2" Body="&lt;p&gt;How are you dividing? You say randomly, but is this being redone each time your analysis is run? If it is, then this is perfectly normal, expected behavior. You're getting a different prediction performance because you have a different random training set each time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also be using a stochastic method, that doesn't give the same result each time, say regression fit using gradient descent versus least squares. Since you didn't give any more details, I can't be more specific than that.&lt;/p&gt;&#xA;" OwnerUserId="31052" LastActivityDate="2017-04-12T16:12:31.267" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18312" PostTypeId="2" ParentId="18289" CreationDate="2017-04-12T21:06:45.790" Score="4" Body="&lt;p&gt;What you are looking for is called &lt;strong&gt;Collaborative Filtering&lt;/strong&gt; / &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_completion&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matrix completion&lt;/a&gt;. See &lt;a href=&quot;https://martin-thoma.com/collaborative-filtering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;my blog post&lt;/a&gt; for a short introduction.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-04-12T21:22:53.573" LastActivityDate="2017-04-12T21:22:53.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18313" PostTypeId="1" CreationDate="2017-04-12T21:30:11.127" Score="-1" ViewCount="673" Body="&lt;p&gt;Let's say I have m training examples with n continuous explanatory variables x1, x2,..., xn and a label y. I'm doing a linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to rank what combinations of explanatory variables are actually the most useful to &quot;predict&quot; y ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, is there an algorithm to tell me what are the best 2 explanatory variables if I only want to use 2 of them? Or to tell me out of the 2^n possible selections of explanatory variables, which ones are the best ? and which explanatory variables are useless / redudant ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks!&lt;/p&gt;&#xA;" OwnerUserId="14240" LastActivityDate="2017-04-12T23:21:21.860" Title="how to find / rank the most explanatory variables of a regression" Tags="|machine-learning|regression|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18314" PostTypeId="2" ParentId="18313" CreationDate="2017-04-12T21:40:24.580" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature selection&lt;/a&gt; is one of the hardest and at the same time important step in creating successful models in general. For you case, you've some number of options aside from exhaustive search (which is doable depending on the number of features and training samples) that are greedy algorithms trying to come up some decent combination of features, namely; forward/backward selection and stepwise selection. Also try Ridge/Lasso regression. Take a look at chapter chapter 6 of &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hastie-Tibshirani's: &lt;em&gt;An Introduction to Statistical Learning&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="5055" LastEditorUserId="5055" LastEditDate="2017-04-12T23:21:21.860" LastActivityDate="2017-04-12T23:21:21.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18315" PostTypeId="2" ParentId="18286" CreationDate="2017-04-12T21:46:51.593" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why doesn't table-look up generalize?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generalization means you find rules which apply to unseen situations. For example, let's say I have a function $f: \mathbb{R} \rightarrow \mathbb{R}$ and I give you the (input, output) pairs (0, 1), (1, 2), (3, 4), (3.141, 4.141).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you learn by table look-up, you know exactly those 4 tuples. But If I ask you what $f(5)$ is, you have a problem. Because you didn't find the general rule/pattern, but you simply memorized the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example: Imagine you have $n$ data points $(x, y)$ and you decide to fit a polynomial to it. As you know, you can fit any $n$ points (with the x's pairwise different) to a polynomial of degree $n-1$. But if you do that, even the slightest noise or a different unterlying model causes your predictions to be awefully wrong because your polynomial bounces like crazy.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What does he mean by hand generated features?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you have a vector of $n$ numbers $(x_1, \dots, x_n)$ as input, you might decided that the pair-wise multiplication $x_3 \cdot x_{42}$ helps the classification process. Hence you add $x_{n+1} = x_3 \cdot x_{42}$. This is a hand generated feature. In contrast, neural networks learn non-linear combinations of the input.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-04-12T21:46:51.593" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18317" PostTypeId="1" CreationDate="2017-04-13T00:00:57.133" Score="2" ViewCount="532" Body="&lt;p&gt;I'm interested in doing a mapping or inventory of all data science for social good and technology for social good organizations/initivaives. Is anyone aware of any similar efforts or places to start?&lt;/p&gt;&#xA;" OwnerUserId="31063" LastActivityDate="2017-04-13T16:44:31.010" Title="Data Science for Social Good" Tags="|machine-learning|community|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18318" PostTypeId="1" AcceptedAnswerId="18332" CreationDate="2017-04-13T01:40:46.373" Score="2" ViewCount="2647" Body="&lt;p&gt;I am trying to build a deep learning neural model using keras and tensorflow which can predict if a certain event will occur based on a set of timeseries data and some fixed data together . For example:&#xA;For a given set of entities say , Their price behavior is co-related due to rumor of some event that is going to happen in future. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) A &amp;amp; B 's fixed data like type, group etc.&#xA;2) During a certain period of time - 1 Jan 2015 - 30 Mar 2015 , their price .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Data that I have is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INPUT : Name of entity, Type Of entity , Size , Country, Specific Attributes  and time series stock data from 1 Jan 2015 - 30 Mar 2015 &#xA;OUTPUT : Y/N . Boolean output if event happend or not. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now my question is how do I build this since I have some fixed data which doesn't changes over time and some time series data which changes over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Options that I thought of are&#xA;1) LSTM - But not sure if I should feed in fixed static data.&#xA;2) CNN - Not sure if it is the right approach ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know what should be my approach to handle such a problem.&lt;/p&gt;&#xA;" OwnerUserId="18313" LastActivityDate="2017-04-13T14:25:48.713" Title="How to build a model which can predict the probability of an event based on a set of timeseries data?" Tags="|machine-learning|time-series|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18319" PostTypeId="2" ParentId="18262" CreationDate="2017-04-13T06:14:28.610" Score="0" Body="&lt;p&gt;OK, let us say you have 24 data points per day, in a year you would have 8700~ data points. As an initial analysis you would likek to classify the data points in 2 classes, 'summer energy', 'winter energy'. Notice I am refering to data points and not time series. For this analysis, on way is not to take a time series view but a collection of datapoints view. Once you do that, it becomes a unsupervised classification problem. You could use K-Mean, where the value of K is 2. You could also use neural network based models like Adaptive Resonance Theory networks. As for feature selection, take all the features that you receive from the sensor. It could additionally help to center and scale the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, if you want to classify the data into 2 classes, timeseries analysis is not immediately required, you could employ the method mentioned above and see how the results are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;EDIT 1:&lt;/em&gt;&lt;/strong&gt;&#xA;As I undestand (and assume), you have 4 sensors placed at 4 points of a single entity, e.g. an office. Each sensor output per hour is one of your dimensions. Hour of the day is also your data dimension. As an example, if you have 4 sensors, let us call them A,B,C,D, then your data point will look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Datapoint 1::&lt;br&gt;&#xA;Vector position 0: value of A sensor, 1.0&lt;br&gt;&#xA;Vector position 1: value of B sensor, 1.5&lt;br&gt;&#xA;Vector position 2: value of C sensor, 0.5&lt;br&gt;&#xA;Vector position 3: value of D sensor, 3.5&lt;br&gt;&#xA;Vector position 4: hour of day (24),  14  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have 720 such data points for one month. You should apply any clustering algorithm on the data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few suggested clustering algorithms (there are many more):&lt;br&gt;&#xA;1) K-Mean with K value as 2&lt;br&gt;&#xA;2) ANN based approach&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;EDIT 2:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could explore the ANN approach, you could make a 3 layer feed-forward network with 4 inputs and 2 outputs. The inputs would be the data point I mantioned above, however, it would be best to normalize the values ( &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;here are some techniques&lt;/a&gt;) between 0 and 1. The output in your case will be 2 neurons, one representing summer and another winter, the input vector would be the datapoint vector and the output a vector of length 2, eg [0,1] for winter ideal valu &amp;amp; [1,0] for summer ideal value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In another approach you could use &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;K- Nearest Neighbour&lt;/a&gt;, in this case data normalization may not be strictly (technicaly) required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ANN library, you could use R/Python/Java, reference to one Java library is &lt;a href=&quot;http://www.heatonresearch.com/encog/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For KNN, there are many library options, one reference is &lt;a href=&quot;https://gist.github.com/leonaburime/64dcb6876a5d3b27a9ae&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26814" LastEditorUserId="26814" LastEditDate="2017-04-14T07:12:40.057" LastActivityDate="2017-04-14T07:12:40.057" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18320" PostTypeId="2" ParentId="18303" CreationDate="2017-04-13T07:59:49.727" Score="2" Body="&lt;p&gt;Depending on a particular implementation of a classifier it may have a random initiation seed. Try to look that up in the documentation and set it to a fixed value. E.g. random_state=42. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2017-04-13T07:59:49.727" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18323" PostTypeId="1" CreationDate="2017-04-13T10:03:18.860" Score="5" ViewCount="3196" Body="&lt;p&gt;I'm starting my first real data-science project, I made a reserach and want to ask if my approach is correct:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I HAVE:&#xA;600 photos of electronic components, hundreds of components in a single photograph, components shape and distance between them may vary between photographs but there is only 1 shape in a single photo. components shapes are squares, rectangles, ovals, &quot;E&quot;-shape and similar&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I NEED: generic counter for components (no matter what is the type of elements on the photo)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PLANNED APPROACH: analyze each picture with openCV to gain predictions and then train CNN with Tensorflow and GPU (no previous experience).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is it the correct way of thinking?&lt;/p&gt;&#xA;" OwnerUserId="31069" LastEditorUserId="29169" LastEditDate="2021-02-16T02:43:33.743" LastActivityDate="2021-02-16T02:43:33.743" Title="Are CNNs best approaches to find and count objects in images?" Tags="|tensorflow|convolutional-neural-network|image-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18324" PostTypeId="1" CreationDate="2017-04-13T11:23:12.667" Score="-1" ViewCount="574" Body="&lt;p&gt;I am working with &lt;a href=&quot;https://www.nist.gov/srd/nist-special-database-4&quot; rel=&quot;nofollow noreferrer&quot;&gt;NIST Special Database 4&lt;/a&gt;, which is a database of fingerprints. The objective is to train a convolutional network (CNN) and train it to classify the fingerprints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I looked through the database, I notices some images are lighter and some are darker, an example is provided below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tYPVA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tYPVA.png&quot; alt=&quot;Darker Image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Rawfa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rawfa.png&quot; alt=&quot;Lighter (bleached) image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is if the images are used with unequal intensity (without any preprocessing), would that affect the training process for the CNN and subsequently the accuracy of the classification process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would Gamma Correction of the images to uniform intensity be better before the training &amp;amp; classification process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some best practices for Gamma correcting multiple images to similar intensity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would really appreciate some help on this.&lt;/p&gt;&#xA;" OwnerUserId="26814" LastActivityDate="2017-04-13T12:11:48.247" Title="Image Classification, Convolution Network and Gamma Correction for images" Tags="|neural-network|image-classification|preprocessing|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18325" PostTypeId="1" AcceptedAnswerId="18328" CreationDate="2017-04-13T11:52:48.737" Score="1" ViewCount="1072" Body="&lt;p&gt;I want a direction for ML technique to predict the next time you will be online in chat app&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My table contains; user id, timestamp &amp;amp; status&lt;/p&gt;&#xA;&#xA;&lt;p&gt;status is a categorical variable (online, composing, offline), I'm saving a row for each status change which means user online, maybe writes something to me (composing) and afterwards went offline, so I can extract the time online (currently I don't see a reason for this feature)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I want to export the model to PMML&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What is the best approach if I want to a service to get the &lt;strong&gt;next timestamp&lt;/strong&gt; that user will be online ?&lt;/p&gt;&#xA;" OwnerUserId="15689" LastActivityDate="2017-04-14T01:49:37.827" Title="ML technique to predict next online" Tags="|machine-learning|time-series|prediction|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18326" PostTypeId="2" ParentId="18324" CreationDate="2017-04-13T12:11:48.247" Score="3" Body="&lt;p&gt;In the &lt;a href=&quot;https://www.tensorflow.org/tutorials/deep_cnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow example&lt;/a&gt; for CNN a &lt;code&gt;distorted_inputs&lt;/code&gt; function is implemented. Besides flipping the images randomly, it's also randomly distorting &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Image brightness&lt;/li&gt;&#xA;&lt;li&gt;Image contrast&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, the intensity of the images is changed intentionally. They do it to increase the number of training images artificially. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of that knowledge I wouldn't suggest making the input similar. Moreover, I think the variability within the input dataset is essential for making the generalization, which is needed for classifications of that example.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to low reputation I am not able to post this information as a comment. Please be gently.&lt;/p&gt;&#xA;" OwnerUserId="15946" LastActivityDate="2017-04-13T12:11:48.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18327" PostTypeId="2" ParentId="18268" CreationDate="2017-04-13T13:52:59.477" Score="2" Body="&lt;p&gt;Logarithm cannot handle numbers less than and equal to zero. You can exclude them from your time-series and then add them to the graph as points with different colors (if they are not many that would make the graph ambiguous and interpretation of it hard) or just simply exclude them but have the original graph side by side as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As proposed in the comments this would be a better option to avoid zeros:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;log⁡(x+1)&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or for negative numbers as well:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;log⁡(|x|+1)&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="30635" LastEditorUserId="30635" LastEditDate="2017-06-01T14:31:43.443" LastActivityDate="2017-06-01T14:31:43.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18328" PostTypeId="2" ParentId="18325" CreationDate="2017-04-13T13:54:39.913" Score="2" Body="&lt;p&gt;That sounds like time series forecasting. Here's a related post:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/212912/forecast-time-series-data-with-external-variables&quot;&gt;https://stats.stackexchange.com/questions/212912/forecast-time-series-data-with-external-variables&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-13T13:54:39.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18329" PostTypeId="2" ParentId="18300" CreationDate="2017-04-13T13:59:00.727" Score="1" Body="&lt;p&gt;You basically want to display the importance of each feature in the model as it relates to your final score. Rather than starting with the visualization, I would figure out the data first and then see which visualization fits. Here, they use a simple bar chart:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-13T13:59:00.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18330" PostTypeId="2" ParentId="18325" CreationDate="2017-04-13T14:05:18.967" Score="1" Body="&lt;p&gt;If you are able to use python at all, I'd look into using a recurrent neural network variant such as LSTM or GRU to forecast their future timestamps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Chris Olah wrote a really nice blog post on LSTM / Recurrent nets &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the python library Keras, it is actually very easy to code an LSTM network. The general &quot;Sequential&quot; model (most straighforward / basic model type) is described here: &lt;a href=&quot;https://keras.io/getting-started/sequential-model-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/getting-started/sequential-model-guide/&lt;/a&gt; &#xA;Your model could be made through the &lt;em&gt;sequential&lt;/em&gt; model type, using recurrent layers instead of Dense ones. The only thing is your inputs need to be shaped correctly, so you'll be dealing with a 3D tensor (3d numpy array) for the time series input, where [N,T,K] is the shape of the tensor: N= # of samples, T = # of timesteps in the input and K = # of variables. You will probably have to zero pad / mask the timesteps for uneven sequences as well, which is fairly easy to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent / LSTM / GRU layers in Keras are described on this page: &#xA;&lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/layers/recurrent/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is too much then I guess you can just work with time series models. But reccurrent nets have been known to perform incredibly with an array of temporal problems / prediction. &lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2017-04-13T14:05:18.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18331" PostTypeId="1" AcceptedAnswerId="18337" CreationDate="2017-04-13T14:15:26.903" Score="0" ViewCount="1630" Body="&lt;p&gt;I would like to classify abbreviations using machine learning. For example:&#xA;I have watermel. and I ask for user what is watermel.(my application context is about food). Then He classify as watermelon.&#xA;In other time, If Other user insert waterme. Is It exist a way to infer that waterme is the same as watermelon using machine learning techniques?&lt;/p&gt;&#xA;" OwnerUserId="31083" LastActivityDate="2017-04-13T18:31:18.543" Title="Abbreviation Classification using machine learning" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18332" PostTypeId="2" ParentId="18318" CreationDate="2017-04-13T14:25:48.713" Score="4" Body="&lt;p&gt;Since you have features that would be handled best with a recurrent neural net, AND some features that would be handled best with a feedforward net, what you can actually do is &lt;strong&gt;both&lt;/strong&gt; and feed them into a main Dense layer which has a softmax output to give you the probability distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be rather hard to do by hand, but luckily you are using Keras, which allows for this kind of modeling rather easily!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the Keras functional API guide &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/getting-started/functional-api-guide/&lt;/a&gt;, there is a model actually very similar to what you are looking for, where the &quot;Main&quot; information is an LSTM layer (which you'd do for the stock prices), and the &quot;Auxiliary&quot; information would be (Name of entity, Type Of entity , Size , Country, Specific Attributes) etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model looks like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/jilDG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jilDG.png&quot; alt=&quot;Keras model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example model actually uses 2 loss functions (2 outputs), but you can easily build it to only have the one output. The code is all there so will be easy to replicate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I basically use this kind of model for almost everything now and get great results, vs just LSTM alone.&lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2017-04-13T14:25:48.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18334" PostTypeId="2" ParentId="18317" CreationDate="2017-04-13T16:44:31.010" Score="2" Body="&lt;p&gt;Sources that I've found so far;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.datakind.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;datakind&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/Data4Democracy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data4Democracy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://dssg.uchicago.edu/projects/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science for Social Good&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://escience.washington.edu/dssg/project-summaries-2016/&quot; rel=&quot;nofollow noreferrer&quot;&gt;eScience DSSG projects&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Other ideas: find a set of data that interests you depending on where you live for example and contribute to your community. Things like &lt;a href=&quot;https://www.data.gov/&quot; rel=&quot;nofollow noreferrer&quot;&gt;data.gov&lt;/a&gt; in US and &lt;a href=&quot;http://open.canada.ca/en/open-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;open-data&lt;/a&gt; in Canada.&lt;/p&gt;&#xA;" OwnerUserId="5055" LastActivityDate="2017-04-13T16:44:31.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18335" PostTypeId="2" ParentId="18331" CreationDate="2017-04-13T17:11:08.430" Score="0" Body="&lt;p&gt;Natural Language Processing/Text Mining may help you to find out a possible solution to your problem. Because, in your case, it is inevitable to handle them with lexical parsers and analysers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this may helps!!&lt;/p&gt;&#xA;" OwnerUserId="31089" LastActivityDate="2017-04-13T17:11:08.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18337" PostTypeId="2" ParentId="18331" CreationDate="2017-04-13T18:20:10.433" Score="1" Body="&lt;p&gt;Look into this package for Python:&#xA;&lt;a href=&quot;https://pypi.python.org/pypi/Distance/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pypi.python.org/pypi/Distance/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use this to generate a numeric value representing the similarity between word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a similar post that should help:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/123060/clustering-a-long-list-of-strings-words-into-similarity-groups&quot;&gt;https://stats.stackexchange.com/questions/123060/clustering-a-long-list-of-strings-words-into-similarity-groups&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, a level up in complexity would be to use t-SNE on an array generated using word2vec (this is word embedding).  Examples and resources for this are:&#xA;&lt;a href=&quot;https://www.codeproject.com/tips/788739/visualization-of-high-dimensional-data-using-t-sne&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.codeproject.com/tips/788739/visualization-of-high-dimensional-data-using-t-sne&lt;/a&gt;&#xA;&lt;a href=&quot;http://sebastianruder.com/word-embeddings-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://sebastianruder.com/word-embeddings-1/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="19161" LastEditDate="2017-04-13T18:31:18.543" LastActivityDate="2017-04-13T18:31:18.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18338" PostTypeId="2" ParentId="18279" CreationDate="2017-04-13T19:13:19.623" Score="3" Body="&lt;p&gt;Notation matters! The problem starts from:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Given &lt;span class=&quot;math-container&quot;&gt;$\nabla a_j^{(k+1)} = \frac{\partial E}{\partial a_j^{(k+1)}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I don't like your notation! it's wrong in fact, in standard mathematical notation. The correct notation is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\nabla_{a_j^{(k+1)}} E = \frac{\partial E}{\partial a_j^{(k+1)}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Then, gradient of the error &lt;span class=&quot;math-container&quot;&gt;$E$&lt;/span&gt; w.r.t a vector &lt;span class=&quot;math-container&quot;&gt;${\mathbf{a}^{(k)}}$&lt;/span&gt; is defined as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\nabla_{\mathbf{a}^{(k)}} E = \left( \frac{\partial E}{\partial a_1^{(k)}} , \cdots,  \frac{\partial E}{\partial a_n^{(k)}}\right)^T \;\;\;\; (\star)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;em&gt;side note&lt;/em&gt;: We transpose because of the &lt;em&gt;convention&lt;/em&gt; that we represent vectors as column vectors, if you'd like to represent as row vectors then the equations you want to prove will change up a transpose!)&lt;/p&gt;&#xA;&lt;p&gt;therefore with chain rule,&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial E}{\partial a_i^{(k)}}= \sum_j \frac{\partial E}{\partial z_j^{(k+1)}} \frac{\partial z_j^{(k+1)}}{\partial a_i^{(k)}}=\sum_j \frac{\partial E}{\partial z_j^{(k+1)}}W_{ij}^{(k)}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;because of &lt;span class=&quot;math-container&quot;&gt;$z_j^{(k+1)} = \sum_i W_{ij}^{(k)}a_i^{(k)}.$&lt;/span&gt; Now, you can express the above as vector (inner) product&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial E}{\partial a_i^{(k)}} = (W_{:,i}^{(k)})^T \nabla_{\mathbf{z}^{(k+1)}} E$$&lt;/span&gt; and stacking them in &lt;span class=&quot;math-container&quot;&gt;$(\star),$&lt;/span&gt; we can express &lt;span class=&quot;math-container&quot;&gt;$\nabla_{\mathbf{a}^{(k)}} E $&lt;/span&gt; as matrix-vector product&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\nabla_{\mathbf{a}^{(k)}} E = (\mathbf{W}^{(k)})^T\nabla_{\mathbf{z}^{(k+1)}} E.$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;I'll leave the rest to you :)&lt;/p&gt;&#xA;&lt;h1&gt;More vector calculusy!&lt;/h1&gt;&#xA;&lt;p&gt;Let's use the convention of vectors as column-vectors. Then &lt;span class=&quot;math-container&quot;&gt;$\mathbf{z}^{(k+1)} = (\mathbf{W}^{(k)})^T \mathbf{a}^{(k)} + \mathbf{b}^{(k)}$&lt;/span&gt; and&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\nabla_{\mathbf{a}^{(k)}} E = \frac{\partial E}{\partial \mathbf{a}^{(k)}} = \frac{\partial \mathbf{z^{(k+1)}}}{\partial \mathbf{a}^{(k)}} \frac{\partial E}{\partial \mathbf{z}^{(k+1)}}= \mathbf{W}^{(k)} \frac{\partial E}{\partial \mathbf{z}^{(k+1)}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;because&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial \mathbf{z^{(k+1)}}}{\partial \mathbf{a}^{(k)}} = \dfrac{\partial\left((\mathbf{W}^{(k)})^T \mathbf{a}^{(k)} + \mathbf{b}^{(k)}\right)}{\partial \mathbf{a}^{(k)}}=\dfrac{\partial\left((\mathbf{W}^{(k)})^T \mathbf{a}^{(k)}\right)}{\partial \mathbf{a}^{(k)}} + \dfrac{\partial\mathbf{b}^{(k)}}{\partial \mathbf{a}^{(k)}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;and &lt;span class=&quot;math-container&quot;&gt;$\dfrac{\partial\mathbf{b}^{(k)}}{\partial \mathbf{a}^{(k)}}=0$&lt;/span&gt; since &lt;span class=&quot;math-container&quot;&gt;$\mathbf{b}^{(k)}$&lt;/span&gt; doesn't depend on &lt;span class=&quot;math-container&quot;&gt;$\mathbf{a}^{(k)}.$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Thus&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\dfrac{\partial\left((\mathbf{W}^{(k)})^T \mathbf{a}^{(k)}\right)}{\partial \mathbf{a}^{(k)}} = \dfrac{\partial \mathbf{a}^{(k)}}{\partial \mathbf{a}^{(k)}} \mathbf{W}^{(k)} = \mathbf{W}^{(k)}.$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;by &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_calculus#Vector-by-vector_identities&quot; rel=&quot;nofollow noreferrer&quot;&gt;vector-by-vector&lt;/a&gt; (eight and seventh row, last column identities, respectively)&lt;/p&gt;&#xA;" OwnerUserId="5055" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-04-19T19:56:15.057" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18339" PostTypeId="1" AcceptedAnswerId="18346" CreationDate="2017-04-13T19:33:53.090" Score="40" ViewCount="36089" Body="&lt;p&gt;Consider a neural network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given set of data, we divide it into training, validation and test set. Suppose we do it in the classic 60:20:20 ratio, then we prevent overfitting by validating the network by checking it on validation set. Then what is the need to test it on the test set to check its performance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Won't the error on the test set be somewhat same as the validation set as for the network it is an unseen data just like the validation set and also both of them are same in number?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead can't we increase the training set by merging the test set to it so that we have more training data and the network trains better and then use validation set to prevent overfitting?&#xA;Why don't we do this?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastActivityDate="2023-11-21T06:04:52.017" Title="Why use both validation set and test set?" Tags="|machine-learning|neural-network|cross-validation|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18340" PostTypeId="2" ParentId="18295" CreationDate="2017-04-13T19:42:31.440" Score="4" Body="&lt;p&gt;You should try compensating for the imbalanced data and then can you try a lot of different classifiers. Either balance it out, use SMOTE to interpolate (this always struck me as too magical), or assign weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a nice article walking through it with caret, which is what it appears you're using:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dpmartin42.github.io/blogposts/r/imbalanced-classes-part-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dpmartin42.github.io/blogposts/r/imbalanced-classes-part-1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-13T19:42:31.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18341" PostTypeId="1" AcceptedAnswerId="39993" CreationDate="2017-04-13T20:56:51.787" Score="12" ViewCount="33790" Body="&lt;p&gt;I have been trying to develop a convolution neural network following some &lt;a href=&quot;https://cs231n.github.io/convolutional-networks/&quot; rel=&quot;noreferrer&quot;&gt;guides&lt;/a&gt; &lt;a href=&quot;https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/&quot; rel=&quot;noreferrer&quot;&gt;online&lt;/a&gt;. However, most guides I have encountered gloss over an important detail, which is how to programmatically represent the weights in a CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understand, in a &quot;regular&quot; neural network, the weight of a connection is a numerical value, which is adjusted in order to reduce the error; then back-propagation is used to further update the weights, reducing thus the error, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in a CNN, the input is an array of numbers (the image), and a subset of those (the filter) to calculate the mean error, by multiplying the filter pixels by the original pixels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is there a weight neuron for each filter (kernel or feature map) of the image? Or is a single weight neuron represented by the sum of all the mean error's calculated from convolving the filter over the receptive field, such that you have one value, in the end, that is the total error for the entire image?&lt;/p&gt;&#xA;" OwnerUserId="31075" LastEditorUserId="29169" LastEditDate="2021-02-16T02:43:20.800" LastActivityDate="2023-04-26T08:55:22.050" Title="How are weights represented in a convolution neural network?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="18343" PostTypeId="2" ParentId="18325" CreationDate="2017-04-14T01:49:37.827" Score="0" Body="&lt;p&gt;I agree with the two answers above me!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a time series forecasting problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python with tensorflow and keras would be the best route to build the model and have it able to be exported to train somewhere else. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start off with something nice and simple, if it doesn't do what you want add some more layers, tinker with the learning rate, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link very similar to what you want to do: &lt;a href=&quot;http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;tutorial&quot;&gt;Tutorial&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helped! Enjoy building it, this stuff is a lot of fun!!&lt;/p&gt;&#xA;" OwnerUserId="31046" LastActivityDate="2017-04-14T01:49:37.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18345" PostTypeId="2" ParentId="6345" CreationDate="2017-04-14T03:52:22.027" Score="4" Body="&lt;p&gt;Here is a more pythonic way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.pivot_table(index=[&quot;city&quot;], columns=&quot;cuisine&quot;, aggfunc=lambda x: 1, fill_value=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a late answer, but it is for people who meet the same probelm.&lt;/p&gt;&#xA;" OwnerUserId="31102" LastActivityDate="2017-04-14T03:52:22.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18346" PostTypeId="2" ParentId="18339" CreationDate="2017-04-14T06:28:38.973" Score="47" Body="&lt;p&gt;Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a choice of hyperparameter values, you use the &lt;strong&gt;training&lt;/strong&gt; set to train the model. But, how do you set the values for the hyperparameters? That's what the &lt;strong&gt;validation&lt;/strong&gt; set is for. You can use it to evaluate the performance of your model for different combinations of hyperparameter values (e.g. by means of a grid search process) and keep the best trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, how does your selected model compares to other different models? Is your neural network performing better than, let's say, a random forest trained with the same combination of training/test data? You cannot compare based on the validation set, because that validation set was part of the fitting of your model. You used it to select the hyperparameter values! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;test&lt;/strong&gt; set allows you to compare different models in an unbiased way, by basing your comparisons in data that were not use in any part of your training/hyperparameter selection process. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="2576" LastEditDate="2017-04-14T06:53:32.907" LastActivityDate="2017-04-14T06:53:32.907" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18347" PostTypeId="2" ParentId="18339" CreationDate="2017-04-14T06:43:06.780" Score="18" Body="&lt;p&gt;The test set and cross validation set have different purposes. If you drop either one, you lose its benefits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The cross validation set is used to help detect over-fitting and to assist in hyper-parameter search.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The test set is used to measure the performance of the model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You cannot use the cross validation set to measure performance of your model accurately, because you will deliberately &lt;em&gt;tune&lt;/em&gt; your results to get the best possible metric, over maybe hundreds of variations of your parameters. The cross validation result is therefore likely to be too optimistic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the same reason, you cannot drop the cross validation set and use the test set for selecting hyper parameters, because then you are pretty much guaranteed to be overestimating how good your model is. In the ideal world you use the test set just once, or use it in a &quot;neutral&quot; fashion to compare different experiments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cross validate, find the best model, then add in the test data to train, it is &lt;em&gt;possible&lt;/em&gt; (and in some situations perhaps quite likely) your model will be improved. However, you have no way to be sure whether that has actually happened, and even if it has, you do not have any unbiased estimate of what the new performance is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From witnessing many Kaggle competitions, my experience is that tuning to the test set by over-using it is a real thing, and it impacts those competitions in a large way. There is often a group of competitors who have climbed the public leaderboard and selected their best model in test (the public leaderboard is effectively a test set), whilst not being quite so thorough on their cross validation . . . these competitors drop down the leaderboard when a new test set is introduced at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach that is reasonable is to re-use (train + cv) data to re-train using the hyper-params you have found, before testing. That way you do get to train on more data, and you still get an independent measure of performance at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get more out of cross validation, the usual approach is &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation&quot; rel=&quot;noreferrer&quot;&gt;k-fold cross validation&lt;/a&gt;. A common trick in Kaggle competitions is to use k-fold cross validation, and instead of re-combining the data into a larger (train + cv) training set, to ensemble or stack the cv results into a meta-model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, always check that your splits for validation and test are robust against possible correlation within your data set.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-14T07:20:24.407" LastActivityDate="2017-04-14T07:20:24.407" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18349" PostTypeId="1" AcceptedAnswerId="18359" CreationDate="2017-04-14T10:30:43.283" Score="2" ViewCount="102" Body="&lt;p&gt;In &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap3.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author explains that for for a single neuron in neural net, by choosing cost function as cross entropy we can eliminate the derivative of activation function in the gradient term, if the activation function is choose to be sigmoid function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html#problem_337461&quot; rel=&quot;nofollow noreferrer&quot;&gt;problem&lt;/a&gt;, he asks why we cannot eliminate the input to neuron term x in gradient of cost function with respect to weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had the following reasoning, in order to compute gradient of Cost, we use chain rule, and derive cost with respect to activation, and activation with respect to  (w * x + b), and the sum with weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a weight Wi, the derivative of (Summation w * x + b) w.r.to Wi is always xi, and the derivative of activation function cannot know that, so it can never eliminate Xi unless its zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is there any other subtle reasoning?&lt;/p&gt;&#xA;" OwnerUserId="31110" LastEditorUserId="5055" LastEditDate="2017-04-15T09:21:58.040" LastActivityDate="2018-11-15T16:41:58.460" Title="Eliminate input in gradient by clever choosing of cost function in neural networks" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18359" PostTypeId="2" ParentId="18349" CreationDate="2017-04-14T16:08:40.727" Score="2" Body="&lt;p&gt;The main reason is that the chain rule is hidden in your justification. The fact that the inner function is linear w.r.t weights then derivative w.r.t weight will always result into input &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; will be somewhere in &lt;span class=&quot;math-container&quot;&gt;$\dfrac{\partial C}{\partial w}.$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="5055" LastEditorUserId="5055" LastEditDate="2018-11-15T16:41:58.460" LastActivityDate="2018-11-15T16:41:58.460" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18360" PostTypeId="1" CreationDate="2017-04-14T16:49:41.977" Score="1" ViewCount="1087" Body="&lt;p&gt;I have a large set of documents, usually 500-2,000 words each, and for several different labels, there are about 20-100 samples with those labels, and hundreds to millions more that should be labeled but do not have that label. Over time, I expect new labels to be created and new documents to be added to the data set, although relatively few old labels added to any documents. It is possible that some documents will have many labels or possibly no labels. It is also possible for a label to be a subclass of another label, though not manually indicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a keyword tagging system in place that utilizes TF-IDF metrics to extract ngrams from the documents, which is my current feature set from them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought is to use Naive Bayes classifiers (one for each label) and perform ROC analysis to see how well it classifies the category. However, if, say, 5% of the data should actually be labeled with Label A, and the amount of data supplied with Label A amounts to 1% of the data, then I would expect an ideal false discovery rate of 80%. On top of that, since the prior distribution is unknown, dealing with this issue is even more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another idea I had is to use the vectors of word/ngram counts and look at the distribution of the cosine similarities of the documents within the sample and use a simple one-sided significance test with p-values to determine if they are similar enough. This relies heavily on the idea that the ngram vectors for each label are dissimilar enough to consider this form of hypothesis testing. A big problem with this is that I am heavily relying on the assumption that similarity is a highly discriminative metric, and because labels can be extremely unbalanced (by up to 5 orders of magnitude), I may need to employ additional methods, such as limiting the number of labels (perhaps by only taking the top N labels by p-value).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training portion of the algorithm does not need to be extremely fast, but the labeling portion should be very quick once the document is parsed/tokenized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a greater penalty for mislabeling a document than leaving it unlabeled, but future, human-feedback data may be provided to indicate incorrect labels. I will not have access to anything of the sort in the meantime, though.&lt;/p&gt;&#xA;" OwnerUserId="9565" LastActivityDate="2017-04-14T18:09:19.743" Title="How can I perform multi-label classification if many labels are missing?" Tags="|text-mining|class-imbalance|multilabel-classification|missing-data|" AnswerCount="1" CommentCount="4" ClosedDate="2017-04-15T09:23:35.870" ContentLicense="CC BY-SA 3.0" />
  <row Id="18361" PostTypeId="1" CreationDate="2017-04-14T18:03:07.740" Score="2" ViewCount="1942" Body="&lt;p&gt;(Cross-posted from &lt;a href=&quot;https://stackoverflow.com/questions/43415724/reproducing-cutoff-in-xgboost-train-with-xgbclassifier&quot;&gt;https://stackoverflow.com/questions/43415724/reproducing-cutoff-in-xgboost-train-with-xgbclassifier&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've gotten xgboost to generate good predictions by using xgboost.train().&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=.6)&#xA;xgtrain = xgb.DMatrix(X_train, y_train)&#xA;&#xA;param = {'max_depth':7, 'silent':1}&#xA;bst = xgb.train(param, xgtrain, num_boost_round=2)&#xA;y_pred = bst.predict(xgtest)&#xA;y_pred = [1. if y_cont &amp;gt; .28  else 0. for y_cont in y_pred]&#xA;y_true = y_test&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This approach wasn't producing good results (I am trying to maximize the f1 score) until I realized that the f1 score increases significantly when setting a threshold on the outputs. This threshold turned out to be .28. Here are some of the predictions before I set the cutoff and convert to 0s and 1s:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.25447303  0.25383738  0.24621713 ...,  0.24621713  0.24621713 0.24621713]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But now I want to tune my parameters (using GridSearchCV()), which means that I'll need to reproduce what I did in xgboost.train() above with XGBClassifier(). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize that things might get tricky because the (default) objective function in xgboost.train() is none, and for XGBClassifier() it's 'binary:logistic'. XGBClassifier() returns the class and not the probability, which is useful in most cases but not here. I've tried predict_proba() with XGBClassifier() and then setting a cutoff, but it seemed pretty useless since the probabilities I were getting were super close to 0 and 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[  9.99445975e-01   5.54045662e-04]&#xA; [  9.89062011e-01   1.09380139e-02]&#xA; [  9.95234787e-01   4.76523908e-03]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I finish the code below do the equivalent of xgboost.train() but with XGBClassifier? When I've tried XGBClassifier without a cutoff, I would get a horrible f1 score.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=.6)&#xA;rf = XGBClassifier(max_depth=7, learning_rate=0.1, n_estimators=100, silent=True, objective='binary:logistic', nthread=-1, gamma=0, min_child_weight=1, max_delta_step=0, subsample=1, colsample_bytree=1, colsample_bylevel=1, reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5, seed=0, missing=None)&#xA;rf = rf.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31127" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2017-04-14T18:38:48.163" Title="Reproducing cutoff in xgboost.train() with XGBClassifier()" Tags="|python|scikit-learn|xgboost|" AnswerCount="0" CommentCount="8" ClosedDate="2017-04-15T09:23:10.777" ContentLicense="CC BY-SA 3.0" />
  <row Id="18362" PostTypeId="2" ParentId="18360" CreationDate="2017-04-14T18:09:19.743" Score="0" Body="&lt;p&gt;This is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Document_classification#Automatic_document_classification_.28ADC.29&quot; rel=&quot;nofollow noreferrer&quot;&gt;Automatic Document/Text Categorization&lt;/a&gt; (a &lt;a href=&quot;http://www.nactem.ac.uk/dtc/DTC-Sasaki.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;lecture note&lt;/a&gt; on this). There're some approaches you can take, such as &lt;a href=&quot;http://aircconline.com/ijdkp/V2N4/2412ijdkp02.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ATC with semi-supervised learning&lt;/a&gt; or building one-vs-all classifier and labeling a new test sample with a new label when all classifiers reject it. This was discussed &lt;a href=&quot;https://stats.stackexchange.com/questions/181132/how-to-make-new-class-from-the-test-data-in-machine-learning&quot;&gt;here&lt;/a&gt; as well. &lt;/p&gt;&#xA;" OwnerUserId="5055" LastActivityDate="2017-04-14T18:09:19.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18368" PostTypeId="1" CreationDate="2017-04-15T01:15:53.130" Score="0" ViewCount="581" Body="&lt;p&gt;For text classification with machine learning - If your training data was generated purely with regular expressions, is it possible to train a machine learning model with this training data which will be more accurate than your original regex?  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;By 'accuracy' I mean to say the percentage that are correctly labelled as compared to the human truth of the question.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(E.g. we have a set of Tweets that we've labelled as happy or sad with regex. I'm wondering if a machine learning classifier trained on this data will be able to surpass the accuracy of the regex that generated the training data)&lt;/p&gt;&#xA;" OwnerUserId="31140" LastEditorUserId="31141" LastEditDate="2017-04-15T09:21:27.470" LastActivityDate="2018-10-18T02:01:10.027" Title="Using training data generated with pure regular expressions - Can machine learning surpass the accuracy of your regular expression?" Tags="|machine-learning|training|text|regex|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18369" PostTypeId="2" ParentId="12799" CreationDate="2017-04-15T01:56:23.173" Score="8" Body="&lt;p&gt;There is some good news there is a library &lt;a href=&quot;http://pandas-ml.readthedocs.io/en/latest/xgboost.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas_ml&lt;/a&gt; which supports XGBoost. This will probably this streamline the workflow simply.&lt;/p&gt;&#xA;" OwnerUserId="2840" LastEditorUserId="29169" LastEditDate="2021-03-11T19:14:27.893" LastActivityDate="2021-03-11T19:14:27.893" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18371" PostTypeId="2" ParentId="18295" CreationDate="2017-04-15T05:58:24.327" Score="1" Body="&lt;p&gt;SMOTE is a good strategy and also I have got significant accuracy, ROC with cost-sensitive classification. In life science, we handle a lot of imbalance datasets this paper describes approach how to handle it. &lt;a href=&quot;https://jcheminf.springeropen.com/articles/10.1186/1758-2946-1-21&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://jcheminf.springeropen.com/articles/10.1186/1758-2946-1-21&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2840" LastActivityDate="2017-04-15T05:58:24.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18372" PostTypeId="2" ParentId="18368" CreationDate="2017-04-15T06:39:03.990" Score="0" Body="&lt;p&gt;It depends. :-)&#xA;Depends on many factors but first of all on your data of course. &#xA;As a corner case let's take a clustering algorithm on the perfectly separable data. Theoretically it could then reach 100% precision even without using your regex generated training data. &#xA;You need to limit your question to include specific dataset characteristics to have an actionable answer. &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2017-04-15T06:39:03.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18374" PostTypeId="1" AcceptedAnswerId="18375" CreationDate="2017-04-15T10:18:02.510" Score="8" ViewCount="10676" Body="&lt;p&gt;I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the code I'm running:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt;print clf.predict([predict_this])&#xA;['A']&#xA;&amp;gt;&amp;gt;&amp;gt;print clf.decision_function([predict_this])&#xA;[[ 185.23220833   43.62763596  180.83305074  -93.58628288   62.51448055  173.43335293]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I use the output of decision function to predict the class (A/B/C/D) with the highest probability and if possible, it's value? I have visited &lt;a href=&quot;https://stackoverflow.com/a/20114601/7760998&quot;&gt;https://stackoverflow.com/a/20114601/7760998&lt;/a&gt; but it is for binary classifiers and could not find a good resource which explains the output of decision_function for multiclass classifiers with shape ovo (one-vs-one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above example is for class 'A'. For another input the classifier predicted 'C' and gave the following result in decision_function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For another different input which the classifier predicted as 'C' gave the following result from decision_function,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 290.54180354 -133.93467605  116.37068951 -392.32251314 -130.84421412   284.87653043]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Had it been ovr (one-vs-rest), it would become easier by selecting the one with higher value, but in ovo (one-vs-one) there are &lt;code&gt;(n * (n - 1)) / 2&lt;/code&gt; values in the resulting list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to deduce which class would be selected based on the decision function?&lt;/p&gt;&#xA;" OwnerUserId="30392" LastEditorUserId="-1" LastEditDate="2017-05-23T12:38:53.587" LastActivityDate="2019-11-22T14:58:42.260" Title="Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo'" Tags="|machine-learning|python|scikit-learn|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18375" PostTypeId="2" ParentId="18374" CreationDate="2017-04-15T12:20:59.553" Score="9" Body="&lt;p&gt;Your link has sufficient resources, so let's go through:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When you call decision_function(), you get the output from each of the pairwise classifiers (n*(n-1)/2 numbers total). See pages 127 and 128 of &quot;Support Vector Machines for Pattern Classification&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Click on the &quot;page 127 and 128&quot; link (not shown here, but in the Stackoverflow answer). You should see:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YAWe9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YAWe9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python's SVM implementation uses one-vs-one. That's exactly what the book is talking about.&lt;/li&gt;&#xA;&lt;li&gt;For each pairwise comparison, we measure the decision function&lt;/li&gt;&#xA;&lt;li&gt;The decision function is the just the regular binary SVM decision boundary&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What does that to do with your question?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;clf.decision_function() will give you the $D$ for each pairwise comparison&lt;/li&gt;&#xA;&lt;li&gt;The class with the most votes win&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For instance,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is comparing:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[AB, AC, AD, BC, BD, CD]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We label each of them by the sign. We get:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[A, C, A, C, B, C]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For instance, 96.42193513 is positive and thus A is the label for AB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have three C, C would be your prediction. If you repeat my procedure for the other two examples, you will get Python's prediction. Try it!&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-04-15T13:05:39.937" LastActivityDate="2017-04-15T13:05:39.937" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18379" PostTypeId="1" CreationDate="2017-04-15T21:15:00.793" Score="1" ViewCount="183" Body="&lt;p&gt;I have some troubles in interpreting the documentation (&lt;a href=&quot;http://iepy.readthedocs.io/en/latest/tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://iepy.readthedocs.io/en/latest/tutorial.html&lt;/a&gt;) commands of IEPY package for Natural Language Processing. The first step to work with IEPY is to create an &quot;instance&quot; (as a non-programmer I have little idea what is it all about). They also provide the command to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;iepy --create &amp;lt;project_name&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My problem is that when I type the exact sentence into my command line, I get a syntax error. I suppose that I do not follow some coding conventions and misinterpret what I should really type. Would be glad to hear what I do wrong.&lt;/p&gt;&#xA;" OwnerUserId="31171" LastActivityDate="2017-04-15T21:15:00.793" Title="Interpreting IEPY documentation commands" Tags="|python|nlp|text-mining|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18380" PostTypeId="2" ParentId="10745" CreationDate="2017-04-15T21:16:50.113" Score="14" Body="&lt;p&gt;&lt;a href=&quot;https://mlr-org.com/docs/2016-08-11-benchmarking-mlr-learners-on-openml/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Benchmarking mlr (default) learners on OpenML&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jJuQd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jJuQd.png&quot; alt=&quot;Philipp Probst&amp;#39;s Ml benchmarking&quot;&gt;&lt;/a&gt;&#xA;The &lt;a href=&quot;https://www.openml.org/t/9914&quot; rel=&quot;nofollow noreferrer&quot;&gt;entire&lt;/a&gt; openml &lt;a href=&quot;https://www.openml.org/t/125923&quot; rel=&quot;nofollow noreferrer&quot;&gt;database&lt;/a&gt; of ML results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rpubs.com/Isaac/caret_reg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Test from RStudio&lt;/a&gt; suggests SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mlmastery &lt;a href=&quot;http://machinelearningmastery.com/compare-the-performance-of-machine-learning-algorithms-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;suggests LDA&lt;/a&gt; and &lt;a href=&quot;http://ttp://machinelearningmastery.com/evaluate-machine-learning-algorithms-with-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Trial and Error&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jmlr.org/papers/volume15/delgado14a/source/delgado14a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fern ́andez-Delgado et al.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Paper concludes  parallel random forest (parRF_t) is best followed by  random forest, LibSVM with Gaussian kernel (svm), extreme learning machine with Gaussian kernel,    C5.0  decision  tree  and  multi-layer  perceptron  (avNNet). &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The best boosting and bagging ensembles use LibSVM as base classifiers&#xA;  (in Weka),  being slightly better than the single LibSVM classifier,&#xA;  and adaboost R (ensemble of decision trees trained using Adaboost.M1).&#xA;  The probabilistic neural network in Matlab, tuning the Gaussian kernel&#xA;  spread (pnn m), and the direct kernel perceptron in C (dkp C), a very&#xA;  simple and fast neural network proposed by us (Fern ́andez-Delgado et&#xA;  al.,2014),  are  also  very  near  to  the  top-20.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1606.00930.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wainer, Jacques (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets&lt;/a&gt;  Based on Fernandez-Delgado et al. (2014). &quot;We have shown that random forests, RBF SVM, and gradient boosting machines are classification algorithm that most likely will result in the highest accuracy&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ttps://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rich Caruana &amp;amp; Alexandru Niculescu-Mizil () An Empirical Comparison of Supervised Learning Algorithms&lt;/a&gt; (classification) concludes with Platt-Calibrated Boosted Trees as best followed RF BagT Cal.SVM NN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many other studies include comparisons of models used. &lt;a href=&quot;https://arxiv.org/abs/1706.03014&quot; rel=&quot;nofollow noreferrer&quot;&gt;Some papers&lt;/a&gt; prefer SVM  others SVM with &lt;a href=&quot;https://www.researchgate.net/figure/51578344_fig8_Box-plot-distributions-of-Press%27-Q-see-text-for-abbreviations-X2Fr9-21582-p&quot; rel=&quot;nofollow noreferrer&quot;&gt;radial-basis&lt;/a&gt; or &lt;a href=&quot;https://www.researchgate.net/profile/Ingmar_Nitze/publication/275641579_COMPARISON_OF_MACHINE_LEARNING_ALGORITHMS_RANDOM_FOREST_ARTIFICIAL_NEURAL_NETWORK_AND_SUPPORT_VECTOR_MACHINE_TO_MAXIMUM_LIKELIHOOD_FOR_SUPERVISED_CROP_TYPE_CLASSIFICATION/links/5541238e0cf2b790436bc791.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;polynomial kernel&lt;/a&gt; for classification. (maybe same thing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ran88dom99/GeneratedRegMLBenchmark/wiki&quot; rel=&quot;nofollow noreferrer&quot;&gt;From my own regressions&lt;/a&gt; on generated data I recommend earth(MARS) Cubist SVMlinear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://minerva.usc.es/xmlui/bitstream/handle/10347/15691/rep_1428.pdf?sequence=1&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Manisha thesis&quot;&gt;Manisha Thesis&lt;/a&gt; first runs tests on UCI Machine Learning Repository then soil fertility which is the focus of the thesis. Best models on UCI were :&quot;elm-kernel is the ELM neural network but with Gaussian kernel&quot;, &quot;svr is the support vector machine for regression, with Gaussian kernel using the Lib-SVM library with the C++ interface&quot;, extraTrees and cubist. The thesis includes great descriptions of each model and links to more papers.&quot;extraTrees achieved&#xA;the best RMSE for 7 of 10 soil problems&quot;. Paper is definitely worth a read.&lt;/p&gt;&#xA;" OwnerUserId="30507" LastEditorUserId="53476" LastEditDate="2020-02-24T10:57:50.090" LastActivityDate="2020-02-24T10:57:50.090" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="18381" PostTypeId="1" CreationDate="2017-04-15T22:51:15.723" Score="1" ViewCount="5261" Body="&lt;p&gt;Just to summarize &#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/6921/understanding-dropout-and-gradient-descent&quot;&gt;Understanding dropout and gradient descent&lt;/a&gt;&#xA;and&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/207481/dropout-backpropagation-implementation&quot;&gt;https://stats.stackexchange.com/questions/207481/dropout-backpropagation-implementation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I need to implement &lt;strong&gt;inverted dropout&lt;/strong&gt; in my CNN. All the neuron outputs in dropout layer during feedforward phase are multiplied by mask/p, where mask is 0 or 1, p is retain rate. But should I apply the same operation (include &lt;strong&gt;division by p&lt;/strong&gt;) at the backpropagation phase? I suppose positive answer (see the second link above), but I need to know for sure.&lt;/p&gt;&#xA;" OwnerUserId="29891" LastEditorUserId="29891" LastEditDate="2017-04-15T23:58:44.267" LastActivityDate="2017-05-16T00:29:51.363" Title="Dropout backpropagation implementation details" Tags="|machine-learning|neural-network|keras|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18382" PostTypeId="2" ParentId="18381" CreationDate="2017-04-15T23:54:49.687" Score="3" Body="&lt;p&gt;As given in the links, the answer is yes! note that you divide the mask by p so that you won't need to multiply by p in the test time and since this is a coefficient for the new activation, it will come out of the derivative in chain rule in backprop.&lt;/p&gt;&#xA;" OwnerUserId="5055" LastActivityDate="2017-04-15T23:54:49.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18383" PostTypeId="2" ParentId="17294" CreationDate="2017-04-16T02:41:03.193" Score="2" Body="&lt;p&gt;The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-phrases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, an example adapted from the &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK Book Chapter 7&lt;/a&gt; and  &lt;a href=&quot;http://www.winwaed.com/blog/2012/01/20/extracting-noun-phrases-from-parsed-trees/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def extract_phrases(my_tree, phrase):&#xA;   my_phrases = []&#xA;   if my_tree.label() == phrase:&#xA;      my_phrases.append(my_tree.copy(True))&#xA;&#xA;   for child in my_tree:&#xA;       if type(child) is nltk.Tree:&#xA;            list_of_phrases = extract_phrases(child, phrase)&#xA;            if len(list_of_phrases) &amp;gt; 0:&#xA;                my_phrases.extend(list_of_phrases)&#xA;&#xA;    return my_phrases&#xA;&#xA;&#xA;&#xA;def main():&#xA;    sentences = [&quot;The little yellow dog barked at the cat&quot;,&#xA;                 &quot;He studies Information Technology&quot;]&#xA;&#xA;    grammar = &quot;NP: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN&amp;gt;|&amp;lt;NNP&amp;gt;*}&quot;&#xA;    cp = nltk.RegexpParser(grammar)&#xA;&#xA;    for x in sentences:&#xA;        sentence = pos_tag(tokenize.word_tokenize(x))&#xA;        tree = cp.parse(sentence)&#xA;        print &quot;\nNoun phrases:&quot;&#xA;        list_of_noun_phrases = extract_phrases(tree, 'NP')&#xA;        for phrase in list_of_noun_phrases:&#xA;            print phrase, &quot;_&quot;.join([x[0] for x in phrase.leaves()])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You defined a grammar based on regex over PoS-tags:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grammar = &quot;NP: {&amp;lt;DT&amp;gt;?&amp;lt;JJ&amp;gt;*&amp;lt;NN&amp;gt;|&amp;lt;NNP&amp;gt;*}&quot;&#xA;cp = nltk.RegexpParser(grammar)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you applied it to a tokenized and tagged sentence, generating a Tree:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentence = pos_tag(tokenize.word_tokenize(x))&#xA;tree = cp.parse(sentence)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you use &lt;code&gt;extract_phrases(my_tree, phrase)&lt;/code&gt; to recursively parse the Tree and extract sub-trees labeled as NP. The example above would extract the following noun-phrases:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Noun phrases:&#xA;(NP The/DT little/JJ yellow/JJ dog/NN) The_little_yellow_dog&#xA;(NP the/DT cat/NN) the_cat&#xA;&#xA;Noun phrases:&#xA;(NP Information/NNP Technology/NNP) Information_Technology&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a great blog post by Burton DeWilde about many more ways to extract interesting keyphrases: &lt;a href=&quot;http://bdewilde.github.io/blog/2014/09/23/intro-to-automatic-keyphrase-extraction/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Intro to Automatic Keyphrase Extraction&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6518" LastEditorUserId="64506" LastEditDate="2019-02-07T10:08:24.440" LastActivityDate="2019-02-07T10:08:24.440" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18384" PostTypeId="1" AcceptedAnswerId="18385" CreationDate="2017-04-16T03:11:27.037" Score="1" ViewCount="520" Body="&lt;p&gt;In the Cooperative stage of Kohonen's SOM, the neighborhood for a winning neuron(output node). In most cases, the neighborhood function happens to be the Gaussian Function.&#xA;For example,&#xA;$$h_j,_i = exp(-d_j,_i^2/2*\sigma^2)$$&#xA;where $h_j,_i$ is the window function ($i$ is the index of the winning neuron and $j$ is the index of the encompassing neuron), $d_j,_i$ is the lateral distance between the winning neuron $i$ and the excited neuron $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the user who is going to implement the SOM has the input data and the randomly initialized weight vectors. How is $d_j,_i$ determined? Is it randomly initialized too? Also what is the relation between the weight vectors of the output nodes and their positions?&#xA;The figure below is just to give a visual representation of the organization of the input and the output nodes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fxltr.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fxltr.jpg&quot; alt=&quot;Just to give a visual representation of the organization of the input and the output nodes&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="29564" LastEditorUserId="924" LastEditDate="2017-04-16T08:46:26.553" LastActivityDate="2023-03-16T12:30:00.733" Title="How are the positions of the output nodes determined in the Kohonen - Self Organizing Maps algorithm?" Tags="|machine-learning|neural-network|clustering|dimensionality-reduction|gaussian|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18385" PostTypeId="2" ParentId="18384" CreationDate="2017-04-16T09:45:12.927" Score="0" Body="&lt;p&gt;The distance is calculated according to a distance function (euclidian, manhattan, mahalanobis and so on. As this is an unsupervised model (hence the self-organizing part in the name) there are no output neurons. Would you clarify the term?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;I'll just recap the training a bit:&lt;/p&gt;&#xA;&lt;p&gt;Inititalizing: We have a grid (let's assume 2d) of neurons &lt;span class=&quot;math-container&quot;&gt;$n_i = (w_i, k_i)$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$w_i$&lt;/span&gt; is a randomly initialized weight and &lt;span class=&quot;math-container&quot;&gt;$k_i$&lt;/span&gt; the position on the grid.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We'll now pick a training sample &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt; randomly. For this instance &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt; we pick the neuron &lt;span class=&quot;math-container&quot;&gt;$n_m$&lt;/span&gt; (&lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; for minimum) where the distance between the weight vector and the instance  is minimal given a distance function &lt;span class=&quot;math-container&quot;&gt;$d$&lt;/span&gt;, so &lt;span class=&quot;math-container&quot;&gt;$n_m = \text{argmin}_{n_j} \; d(x_j, W(n_j))$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$W(\cdot)$&lt;/span&gt; gives the weight for the respective neuron.&lt;/li&gt;&#xA;&lt;li&gt;We now pick a set of neurons for which we will adapt the weight vector given a neighbourhood function (for example your gaussian, or a cone). The weights that should have their weights updated are given by &lt;span class=&quot;math-container&quot;&gt;$N^{+t} = \{ n_i = (w_i,k_i) \mid d_A(k_m,k_i) \leq \delta^t \}$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt; is the step in time (I'll mention that later), the &amp;quot;reach&amp;quot; &lt;span class=&quot;math-container&quot;&gt;$\delta^t$&lt;/span&gt; the neighbourhood should have and of course the position on the grid &lt;span class=&quot;math-container&quot;&gt;$k_m$&lt;/span&gt; for the winning neuron.&lt;/li&gt;&#xA;&lt;li&gt;We then update the weights of the neurons in the neighbourhood of the winning neuron &lt;span class=&quot;math-container&quot;&gt;$n_m$&lt;/span&gt; according to some updating rule. This can be interpreted as moving the weights closer to the input we currently look at (because we chose the neuron with the closest weight vector). A possible update rule could be: &lt;span class=&quot;math-container&quot;&gt;$w_s^{t+1} = w_m^t + \epsilon^t \cdot h_{mi}^t \cdot (x_j - w_m^t)$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$\epsilon^t$&lt;/span&gt; is a time-dependent learning rate and &lt;span class=&quot;math-container&quot;&gt;$h_{si}^t$&lt;/span&gt; weights the distance from the winning neuron to the neuron we are changing right now, also time-dependent.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;In order to somewhat guarantee a topological mapping two things should be considered now:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The learning rate, which is part of the update rule, has to be decreased over time, starting with a relatively high value.&lt;/li&gt;&#xA;&lt;li&gt;The neighbourhood &amp;quot;radius&amp;quot; has to be decreased as well, also starting from a relatively high value.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Let's visualize this on this picture real quick. The red nodes are inputs. The dark grey is the winning neuron for the input that is considered right now (the picture does not make it clear that we look at one example/input at a time!). The light grey nodes are the ones that are within the neighbourhood of the winning neuron and hence are updated slightly according to some updating rule.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V0ltz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V0ltz.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Disclaimer: this was from the top of my head right now, please check any formulas or hypotheses I propose here before using it in any kind of work. (Which you should always do!)&#xA;.&#xA;See also &lt;a href=&quot;https://en.wikipedia.org/wiki/Self-organizing_map&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt;, they also have some explanation of the algorithm that might better suit you.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Self-organizing-map.svg/500px-Self-organizing-map.svg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image source&lt;/a&gt;&#xA;&lt;a href=&quot;https://de.wikipedia.org/wiki/Selbstorganisierende_Karte&quot; rel=&quot;nofollow noreferrer&quot;&gt;Additional source (German Wikipedia)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29582" LastEditorUserId="29582" LastEditDate="2023-03-16T12:30:00.733" LastActivityDate="2023-03-16T12:30:00.733" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="18387" PostTypeId="2" ParentId="15094" CreationDate="2017-04-16T11:11:58.480" Score="1" Body="&lt;p&gt;The bottleneck depends on the use pattern rather than the direct number of users. If people are doing high I/O workloads, then you wont get many people on at all. Whereas if you are doing small processes or just using it as a simple data lake, you could host the sort of people you're talking about there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you considered using AWS?&lt;/p&gt;&#xA;" OwnerUserId="31154" LastActivityDate="2017-04-16T11:11:58.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18390" PostTypeId="1" CreationDate="2017-04-16T14:19:00.333" Score="-2" ViewCount="2449" Body="&lt;p&gt;I have a problem when I need to classify words in groups.&#xA;For example:&#xA;Input: apple, watermelon, knife, soap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to split the input in groups&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Food: apple, watermelon&#xA;Object: knife&#xA;Cleaning Product: Soap&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't know the best approach to do my objective.&#xA;Has anyone else ever had a similar problem?&lt;/p&gt;&#xA;" OwnerUserId="31083" LastActivityDate="2017-04-17T14:57:01.570" Title="How to classify words in groups" Tags="|classification|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18391" PostTypeId="2" ParentId="18390" CreationDate="2017-04-16T15:46:55.703" Score="1" Body="&lt;p&gt;It looks like you have a classification problem and like you already know the correct classes. So I guess that you are asking for a supervised method from the area of Machine Learning (ML). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find some ideas about ML in general and get some method you can have a look at the paper &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2347755&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Few Useful Things to Know about ML&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beside simpler methods you could have a look to implementations of neural networks. In the paper &lt;a href=&quot;https://arxiv.org/pdf/1408.5882.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Convolutional Neural Networks for Sentence Classification&lt;/a&gt;, but be aware that they classify for sentiments and also have whole sentences as input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, there are also projects with an API available, which give you a class for a given word. I am not able to remember one. Maybe some other user know more about that.&lt;/p&gt;&#xA;" OwnerUserId="15946" LastActivityDate="2017-04-16T15:46:55.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18393" PostTypeId="1" CreationDate="2017-04-16T17:40:16.257" Score="1" ViewCount="161" Body="&lt;p&gt;I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document where a &quot;keyword&quot; is more relevant, he also wants us to divide the project in 3 parts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data acquisition, preprocing (cleaning, transform, join, etc), loading. ETL PROCESS.&lt;/li&gt;&#xA;&lt;li&gt;Data processing.&lt;/li&gt;&#xA;&lt;li&gt;User friendly application.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I need to define what technologies or methods i'm gonna user for each of the parts of the project, but i have no idea what to do on the ETL part since the documents are gonna be written in legible English (they are books), i will appreciate any information you can give me on this but also about the other parts of the project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a million for reading.&lt;/p&gt;&#xA;" OwnerUserId="31182" LastActivityDate="2018-04-11T01:22:54.690" Title="what ETL technique should i use for text documents using Hadoop?" Tags="|dataset|bigdata|data-cleaning|apache-hadoop|etl|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18395" PostTypeId="2" ParentId="6854" CreationDate="2017-04-16T19:30:56.737" Score="0" Body="&lt;p&gt;As part of ELK, you will find a pretty nice time series analysis open source call timelion. Once you catch how it works, it is an excellent framework.&lt;/p&gt;&#xA;" OwnerUserId="53307" LastActivityDate="2017-04-16T19:30:56.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18396" PostTypeId="1" CreationDate="2017-04-16T20:42:19.003" Score="0" ViewCount="413" Body="&lt;p&gt;I am currently working to build a mathematical model to predict the stock market. I learned that the best way to do such thing is no longer to make one big best model, but rather to gather several models together where it follows best results. This technique is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;ensemble learning&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best &lt;code&gt;ensemble learning&lt;/code&gt; method? Bayes Optimal classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to take an ensemble learning that &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;trains several different models using training data, and picks the one that works best with the test data?&lt;/li&gt;&#xA;&lt;li&gt;runs multiple models at once on the data, and combines the results together?&lt;/li&gt;&#xA;&lt;li&gt;is an alternative technique where each ensemble boost attibutes that address data mis-classified by the previous model?&lt;/li&gt;&#xA;&lt;li&gt;...?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S.&lt;/strong&gt; Please tell me if the question is unclear.&lt;/p&gt;&#xA;" OwnerUserId="30907" LastEditorUserId="836" LastEditDate="2017-04-18T06:56:49.920" LastActivityDate="2017-04-18T07:05:37.060" Title="Ensemble learning" Tags="|machine-learning|ensemble-modeling|bayesian|" AnswerCount="2" CommentCount="3" ClosedDate="2017-04-18T10:19:56.433" ContentLicense="CC BY-SA 3.0" />
  <row Id="18398" PostTypeId="2" ParentId="18396" CreationDate="2017-04-16T21:40:06.727" Score="2" Body="&lt;p&gt;Ensemble learning is categorized into 4 different classes: bagging, boosting, stacking, hierarchy classification and sometimes they consider grading as another category. Each one of these categories has many different types. For example, in boosting you have &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adaboost&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gboost&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning#Common_types_of_ensembles&quot; rel=&quot;nofollow noreferrer&quot;&gt;many others&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is very important to understand the differences between these categories and why each one is important. Let me give you a brief idea. Bagging is very important to be used when you have one learning algorithm that is unstable , bagging can reduce the variance error. Boosting is very important when you have variations within classes, boosting can be used to generate different learners that focus on different samples ( specifically, difficult samples), stacking based classifier is a selector algorithm used to reduce bias error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is highly recommended to read a survey about ensemble learning to get a better idea which method to use. Remember to read also about cross validation that can be used to test and validate selection. Read also about imbalance data base and how over sampling, under sampling and SMOTE algorithm can be used to deal with these problems&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="836" LastEditDate="2017-04-18T07:05:37.060" LastActivityDate="2017-04-18T07:05:37.060" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18401" PostTypeId="1" CreationDate="2017-04-17T07:24:19.303" Score="2" ViewCount="52" Body="&lt;p&gt;I have been using the LDA package for R, but it is missing quite a few features especially those that can assess the output.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the any preferred packages that have some of the following?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Univariate Test Statistics&lt;/li&gt;&#xA;&lt;li&gt;Canonical Analysis&lt;/li&gt;&#xA;&lt;li&gt;Multivariate Statistics like Wilk's lambda&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28694" LastActivityDate="2017-04-17T07:24:19.303" Title="What is the favored discriminant analysis package in R?" Tags="|r|discriminant-analysis|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18402" PostTypeId="1" CreationDate="2017-04-17T07:46:11.423" Score="3" ViewCount="3084" Body="&lt;p&gt;I'm currently reading this paper : &lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, the author mentioned that LSTM learns better when the order of the source target is reversed. &quot;Mapping (c,a'), (b,b'), (a,a') (a',b',c' is a translation of a,b,c each) is better.&quot; but I don't know why reversing the sentence makes it easy for SGD to map input n output sequence better.&lt;/p&gt;&#xA;" OwnerUserId="31208" LastActivityDate="2018-04-03T09:01:37.103" Title="Why does LSTM performs better when the source target is reversed? (Seq2seq)" Tags="|machine-learning|machine-translation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18403" PostTypeId="1" AcceptedAnswerId="40586" CreationDate="2017-04-17T07:56:16.363" Score="2" ViewCount="2786" Body="&lt;p&gt;I would love to use a linear LASSO regression within statsmodels, so to be able to use the 'formula' notation for writing the model, that would save me quite some coding time when working with many categorical variables, and their interactions. However, it seems like it is not implemented yet in stats models?&lt;/p&gt;&#xA;" OwnerUserId="27768" LastActivityDate="2019-10-27T06:47:14.660" Title="Is LASSO regression implemented in Statsmodels?" Tags="|linear-regression|regularization|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18404" PostTypeId="2" ParentId="18403" CreationDate="2017-04-17T09:25:38.847" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://statsmodels.sourceforge.net/stable/genindex.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://statsmodels.sourceforge.net/stable/genindex.html&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is the index page. &quot;Lasso&quot; is not found anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why don't you use sckit-learn?&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-04-17T09:25:38.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18406" PostTypeId="2" ParentId="18396" CreationDate="2017-04-17T09:34:31.923" Score="-1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Do I have to take an ensemble learning that&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You don't have to do anything. As so often with machine learning questions, there is no obvious answer what will work better / best. Most of the time, you just have to try it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a general overview over ensembling techniques, see my blog post: &lt;a href=&quot;https://martin-thoma.com/ensembles/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://martin-thoma.com/ensembles/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keywords to look for are: Bagging, boosting, stacking&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-04-17T09:34:31.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18407" PostTypeId="2" ParentId="18403" CreationDate="2017-04-17T09:48:45.390" Score="1" Body="&lt;p&gt;Ok alternate solution: I can use Patsy with scikit-learn to obtain the same results I would obtain with the formula notation in statsmodels. See code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from patsy import dmatrices&#xA;&#xA;# create dummy variables, and their interactions&#xA;y, X = dmatrices('outcome ~ C(var1)*C(var2)', df, return_type=&quot;dataframe&quot;)&#xA;# flatten y into a 1-D array so scikit-learn can understand it&#xA;y = np.ravel(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I can now use any model implemented in scikit-learn with the usual notations having X as independent variables, and y as dependent one.&lt;/p&gt;&#xA;" OwnerUserId="27768" LastEditorUserId="27768" LastEditDate="2019-10-27T06:47:14.660" LastActivityDate="2019-10-27T06:47:14.660" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18412" PostTypeId="2" ParentId="18390" CreationDate="2017-04-17T14:57:01.570" Score="2" Body="&lt;p&gt;One possible approach is to use pretrained word embeddings like word2vec (&lt;a href=&quot;https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/&lt;/a&gt;), which map a word to a real-valued vector in a high-dimensional space, and then use clustering techniques (k-means, etc.) to assign words to categories.&lt;/p&gt;&#xA;" OwnerUserId="30408" LastActivityDate="2017-04-17T14:57:01.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18414" PostTypeId="1" AcceptedAnswerId="18415" CreationDate="2017-04-17T16:18:22.793" Score="34" ViewCount="40085" Body="&lt;p&gt;When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64, and 128 elements per mini batch.&lt;/p&gt;&#xA;&lt;p&gt;Are there any rules/guidelines on how big a mini-batch should be? Or any publications which investigate the effect on the training?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="29169" LastEditDate="2021-03-30T18:48:31.490" LastActivityDate="2021-03-30T18:48:31.490" Title="Are there any rules for choosing the size of a mini-batch?" Tags="|deep-learning|neural-network|convolutional-neural-network|optimization|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="18415" PostTypeId="2" ParentId="18414" CreationDate="2017-04-17T16:57:31.617" Score="31" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/abs/1609.04836&quot; rel=&quot;noreferrer&quot;&gt;On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima&lt;/a&gt; there are a couple of intersting statements:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It has been observed in practice that&#xA;  when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;large-batch methods tend to converge to sharp minimizers of the&#xA;  training and testing functions—and as is well known, sharp minima lead to poorer&#xA;  generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf#page=73&quot; rel=&quot;noreferrer&quot;&gt;my masters thesis&lt;/a&gt;: Hence the choice of the mini-batch size influences:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Training time until convergence&lt;/strong&gt;: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Training time per epoch&lt;/strong&gt;: Bigger computes faster (is efficient)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Resulting model quality&lt;/strong&gt;: The lower the better due to better generalization (?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is important to note &lt;strong&gt;hyper-parameter interactions&lt;/strong&gt;: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/236393/25741&quot;&gt;this nice answer / related question&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.cmu.edu/~muli/file/minibatch_sgd.pdf&quot; rel=&quot;noreferrer&quot;&gt;Efficient Mini-batch Training for Stochastic Optimization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://svail.github.io/rnn_perf/&quot; rel=&quot;noreferrer&quot;&gt;this RNN study&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-08-01T06:34:23.013" LastActivityDate="2017-08-01T06:34:23.013" CommentCount="5" CommunityOwnedDate="2017-04-17T18:16:56.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="18416" PostTypeId="1" AcceptedAnswerId="18419" CreationDate="2017-04-17T19:41:33.830" Score="4" ViewCount="2058" Body="&lt;p&gt;I am making a fun experiment in which a machine will mix different percentages of three juices – orange, apple and grape. After each mix is dispensed, a participant will taste the juice and rate it on a numeric scale, a score from 1 to 7.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the data collected, I would like to try and find the optimal juice mix programmatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this I wish to implement a machine learning algorithm, that will both generate new ratios of juices to try, and using the response will try and find to the optimal juice mixture percentages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithm would you recommend me to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I am planning on displaying this at an event, and estimate around 200 people to taste, at least.&lt;/li&gt;&#xA;&lt;li&gt;I am aware that due to the fact that different people have different taste preferences, I aim for the best tasting mixture that will fit most people.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="31229" LastEditorUserId="31229" LastEditDate="2017-04-17T20:05:32.477" LastActivityDate="2017-04-19T09:09:31.883" Title="Algorithm for finding best juice combinations" Tags="|machine-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18418" PostTypeId="1" AcceptedAnswerId="18420" CreationDate="2017-04-17T20:10:14.503" Score="2" ViewCount="78" Body="&lt;p&gt;So, I'm kinda new to machine learning and I was trying to predict the monthly sales of a business using a set of features and using a sliding window of the past sales of 12 months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used some algorithms to do it, including linear/polynomial regression, lasso/elastic and SVR. I got the best results with elastic regression resulting in the following result:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/gMy7Z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gMy7Z.png&quot; alt=&quot;Elastic&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it shows, the model fit the mean of the curve somewhat well, but I would like it to fit the variance as well. So, I've been searching what technique or feature to use could better fit my data, but I still found nothing precise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would someone knows what could I do to to take the variance of the system into account? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="30314" LastActivityDate="2017-04-17T22:43:06.863" Title="Elastic Regression fitting good mean and bad variance" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18419" PostTypeId="2" ParentId="18416" CreationDate="2017-04-17T21:44:19.533" Score="6" Body="&lt;p&gt;As described, you have no data describing individual people (such as age, sex, shoe size), but are searching for an  optimum value of the mix for the whole population. So what you want is a mix with the maximum &lt;em&gt;expected&lt;/em&gt; rating, if you chose a random person to rate it from the population. In principle, this expected rating is a function taking two parameters e.g. $f(n_{apple}, n_{orange})$ - the amount of the third juice type is a not a free choice, so you only have two dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can break down your problem into two distinct parts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Taking samples from your population in order to find approximation to the function $f(n_{apple}, n_{orange})$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the approximation as it evolves to guide the search for an optimum value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For a simple approach, you could ignore the second bullet point and just randomly sample different mixes throughout the event. Then train a regression ML on the ratings (any algorithm would do, although you'll probably want something nonlinear, otherwise you'll just predict one of the pure juices as favourite) - finally graph its predictions and find the maximum rating at the end. This would probably be fine when pitched as a fun experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is a more sophisticated approach that is well-studied and used to make decisions when you want to optimise an expected value of an action whilst exploring options - it is usually called &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-armed_bandit&quot; rel=&quot;noreferrer&quot;&gt;multi-armed bandit&lt;/a&gt;. In your case, you would need variants of it that consider an &quot;arm space&quot; or parametric choice, as opposed to a finite number of choices that represent selecting between actions. This is important to you, since splitting your mix parameters up into e.g. in 5% steps, will give you too many options to explore given the number of samples you need to make. Instead, you will need to make an assumption that the expected rating function is relatively smooth - the expected rating for 35% Apple, 10% Orange, 55% Grape is correlated with the rating for  37% Apple, 9% Orange, 54% Grape . . . that seems at least reasonable to me, but you should make clear in any write-up that this is an assumption and/or find something published that supports it. If you make this assumption, you can then use a function approximator such as a neural network, a program like xgboost or maybe some Guassian kernels to predict expected rating from mix percentages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In brief for a multi-armed bandit problem, you will use data collected as your experiment progresses to estimate the expected value for each choice, and on each step will make a new choice of mix. The choice itself will be guided by your current best approximation. However, you don't always sample the current top-rated value, you need to explore other mixes in order to refine your estimated function. You have choices here too - you could use $\epsilon$-greedy where e.g. 10% of the time you choose completely randomly to get other sample points. However, you might need something more sophisticated that explores more to start with and still converges quickly, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Gibbs_sampling&quot; rel=&quot;noreferrer&quot;&gt;Gibbs sampling&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing you don't say is at what level you are pitching this experiment. Studying the multi-armed bandit problem by yourself referring to blogs, tutorials and papers could be a bit too much work if this is for school science fair. If this all seems a bit too vague and a lot of work to study, then you can probably stick with a simple regression model from the data of a random experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest whichever approach you take, that you run some simulations of input data and see whether your approach works. Obviously there is a lot of guess work here. But the principle is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a &quot;true&quot; model function - e.g. pick an imaginary favourite mix and make it score higher. Make it a simple and probably quite subtle function - e.g. score 5 for best result, and take away euclidean distance in &quot;juice space&quot; times a small factor (maybe 1.5) from it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a noisy sampler that imitates someone in your experiment giving a rating to a specific mix. Ensure that the mean value from this matches the &quot;true&quot; function.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try out your sampling and learning strategies, see how well they find the favourite mix.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I highly recommend this kind of dry run before putting your system to real use, otherwise you will have no confidence that your ML/approximator is working.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;One more piece of advice about your estimator: You are expecting a large amount of variance in your data, and will not have a lot of samples. So to avoid over-fitting you will want to have a relatively simple ML model. For a neural network for example, you will probably want only one hidden layer with very few neurons in it (e.g. 4 or 5 might be enough). Finding a model sophisticated enough to predict a curve, but simple enough that it doesn't overfit given very noisy target outputs might take a few tries - this is the main reason why I suggest performing trial runs with simulated data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-18T06:43:43.497" LastActivityDate="2017-04-18T06:43:43.497" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18420" PostTypeId="2" ParentId="18418" CreationDate="2017-04-17T22:43:06.863" Score="1" Body="&lt;p&gt;Your model actually looks pretty good. What it sounds like you are asking to do is to overfit your model. I would not recommend that you do that. You can do that by finding more variables that you can input into the model, fitting extra polynomial terms or anything else like that, fitting a neural network will potentially do it for you too. However, you generally want to smooth your predictions out, like what you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that you could try is to add an autocorrelation term. That might cause your model to behave as you intend. With negative autocorrelation your predicted values will have a tendency to bounce back and forth around the mean. But I wouldn't recommend doing that, your performance will probably suffer, just judging by the graph that you provided. &lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2017-04-17T22:43:06.863" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18425" PostTypeId="2" ParentId="17430" CreationDate="2017-04-18T15:26:10.273" Score="0" Body="&lt;p&gt;I have few documents(about 71). So far, TF-IDF is the best way because my each documents are not that long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plus, for document, doc2vec come from word2vec is better. The reason is doc2vec has more vectors represent document (not words vector).&lt;/p&gt;&#xA;" OwnerUserId="31152" LastActivityDate="2017-04-18T15:26:10.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18427" PostTypeId="1" AcceptedAnswerId="18436" CreationDate="2017-04-18T16:28:16.577" Score="2" ViewCount="692" Body="&lt;p&gt;In a lot of cases unlabelled data needs to be transformed to labelled data. The best solution is to use (multiple) human classifiers. However, going to all the data by hand (i.e. in text-mining or image-processing) is often a daunting task. Is there software that can combine human classifiers and machine-learning techniques in real time? I am especially interested in python packages. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate, classifying images from video streams is very repetitive. After 100 images (from different streams) a machine-learning algorithm could be used to predict the labels given by the human classifier. The machine classifier might be very confident about some (un)seen samples and very uncertain about others. The human classifier can then focus on the uncertain samples helping the machine classifier to learn better what is does not yet know.&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="85045" LastEditDate="2021-02-10T16:47:58.520" LastActivityDate="2021-02-10T16:47:58.520" Title="Python package for machine-learning aided data labelling" Tags="|machine-learning|python|labels|active-learning|labelling|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18432" PostTypeId="1" AcceptedAnswerId="18438" CreationDate="2017-04-18T18:26:45.447" Score="-1" ViewCount="200" Body="&lt;p&gt;I've got a dataset of 1.5 million and am looking to train 7 different classifiers -- for each classifier I have up to 10 classes to predict. The total sample has 20K text features (more if I include bigrams). Like most distributions of text features, only 20% of them account for 80% of occurrences in the sample. I am going to manually label 10K for each prediction category, and use that to predict against the remaining 1.5 million as well as new documents that come through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how would I choose the subsample based on the features and distribution. Should I just choose a random sample (ie try to match the distribution)? Or should I try to find the 10K that maximizes the number of features represented in the sample? Whats the benefit and drawback of each?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have only one shot to label these 10K so I want to make sure I choose the right sample that maximizes my accuracy for each of the prediction categories!&lt;/p&gt;&#xA;" OwnerUserId="31266" LastEditorUserId="31266" LastEditDate="2017-04-19T15:39:11.507" LastActivityDate="2017-04-19T15:39:11.507" Title="How to Choose a Sample for Multiply Classifiers" Tags="|python|text-mining|random-forest|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18433" PostTypeId="1" AcceptedAnswerId="18434" CreationDate="2017-04-18T19:37:19.123" Score="1" ViewCount="200" Body="&lt;p&gt;I have a data set consisting of a bunch of predictors (mostly unbounded or positive real numbers) and a single response variable that I wish to predict. The response is typically exactly zero -- around 90% of the time. I have tried modelling this using standard Gaussian process methods as well as random forests. However, in both cases (although moreso when using random forests) the model seems to handle the data poorly, usually predicting a non-zero response. Now, if the predicted responses were in fact very close to zero I could just set a cut-off below which the values would be rounded to zero, but they are significantly non-zero in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea for a solution is to train two models: a classification model trained on the entire training-set that predicts whether a variable is zero or non-zero, and a regression model trained only on the rows in the training set with a non-zero response. I would then first use the classification model to predict which observations have a response that is exactly zero, and subsequently use the regression model to predict the value of the non-zero responses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a sound way to solve the described problem? Does this sort of model have a name? Are there better ways to do this?&lt;/p&gt;&#xA;" OwnerUserId="31270" LastActivityDate="2021-05-03T08:58:58.857" Title="Classification followed by regression to handle response variable that is usually zero" Tags="|machine-learning|random-forest|gaussian|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18434" PostTypeId="2" ParentId="18433" CreationDate="2017-04-18T20:38:15.710" Score="3" Body="&lt;p&gt;This sounds entirely reasonable, and the usual name for this structure I have heard for this is just &quot;pipeline&quot; which also applies to other system-feeds-next-system structures - it might also be &quot;machine learning pipeline&quot; or &quot;data processing pipeline&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are ways to assess performance of a ML pipeline:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You can of course compare the final accuracy or loss value, with the simpler model. Has turning the model into a more complex multi-stage one actually improved things? Sadly nothing is guaranteed, although I would be hopeful in your case initially - in part because you could apply adjustments available to classifier models used to deal with class imbalance issues.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can decide which part of the pipeline will gain you the most benefit by switching between pipeline-so-far input to each unit and perfect input from the training data. Then you can see how much incremental difference is possible by perfecting that unit in the pipeline.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In your case you have a two stage pipeline, so you can check whether it is worth focusing more effort on the classifier or regression parts by comparing the incremental improvements between:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The unadjusted output of the whole pipeline run end-to-end.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The output of the regression (or zero) assuming that the classifier was perfect.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A perfect score.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Whichever of the two differences gives you the largest difference (2) - (1), or (3) - (2) points at work being most rewarded for working on the classifier or regression stage respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see a worked example of this per-stage analysis in &lt;a href=&quot;http://cs229.stanford.edu/materials/ML-advice.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Advice for Applying Machine Learning&lt;/a&gt; (slides 21, 22), amongst other places.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-18T20:44:06.137" LastActivityDate="2017-04-18T20:44:06.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18435" PostTypeId="2" ParentId="18283" CreationDate="2017-04-18T20:53:37.283" Score="1" Body="&lt;p&gt;The first dimension is always the sample index. In the model summary this is written as None, so your last output (None, 15, 145) is 3D. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error message shows that your target data has dimension (:, 15), rather than the required (:, 15, 145) so you will have to reformat it.&lt;/p&gt;&#xA;" OwnerUserId="31275" LastActivityDate="2017-04-18T20:53:37.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18436" PostTypeId="2" ParentId="18427" CreationDate="2017-04-18T22:23:11.237" Score="4" Body="&lt;p&gt;It sounds like you are looking for &lt;a href=&quot;https://en.wikipedia.org/wiki/Active_learning_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;active learning&lt;/a&gt;.  In active learning, the classifier learns which samples would be most useful to have labelled by a human.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many techniques for active learning, and many ways to adapt an existing (standard) learning algorithm to the active learning setting.  The particular approach you mentioned is called &quot;uncertainty sampling&quot;, and can be applied to any standard classifier that outputs confidence/certainty scores.  There are other selection methods as well, which may perform better in some settings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also apply unsupervised methods to cluster the samples, then label one or a few samples from each cluster.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-18T22:23:11.237" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18437" PostTypeId="1" CreationDate="2017-04-18T23:34:48.323" Score="1" ViewCount="565" Body="&lt;p&gt;I am trying to use random forest to select important variables out of 15K features and fit them into logistic regression. My evaluation is based on F1 score. Dataset 2 classes ratio are around: 99.5:0.5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is where 15K features come from: &#xA;Initally I have 2000 features, after dumminize them (only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance ratio is also very small. I tried removing near zero variance features before and it significantly reduces the number of features, however the logistic regression result was also not good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, after using grid search, random forest still has very low f1 score for cross validation. (less than 0.01)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I tried undersampling for the training data to make it 1:1, random forest still has bad F1 score for cross validation. :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was thinking about not selecting the important variables and just fit all features to logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to memory issue, I am not able to fit all 15K features into logistic regression directly but if I select important variables from random forest, they do not generalize for unseen test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to solve the problem?&#xA;I know one alternative could be using hashing so I could fit all 15K features. &lt;/p&gt;&#xA;" OwnerUserId="30376" LastEditorUserId="30376" LastEditDate="2017-04-20T03:12:49.200" LastActivityDate="2021-02-09T17:01:05.120" Title="Imbalanced Data how to use random forest to select important variables?" Tags="|random-forest|logistic-regression|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18438" PostTypeId="2" ParentId="18432" CreationDate="2017-04-19T01:39:17.843" Score="2" Body="&lt;p&gt;Ideally you'll end up with a dataset where each class is fairly represented, with enough data for each class to enable suitable predictive performance. As you currently have no labels, any kind of stratified sampling is unavailable to you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Random sampling&lt;/strong&gt; will get you the 10K samples you need, but there's no guarantee you'll get fair representation for each of your classes. Assuming you were to use this approach it would make sense to continue to label the samples until you have decent representation for all classes. There is no guarantee regarding balance and also no guarantee that it will cover the majority of the variance of your dataset in feature space either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An alternative sampling approach&lt;/strong&gt; that is able to capture a majority of the variance of the data in feature space should hold up with better performance/generalisation. There are a few variations of how this could be done but you could try a clustering approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cluster your points in feature space and then instead of sampling randomly, sample from each cluster in turn. So draw a random point from the first cluster, then second, then third etc until the last cluster, and repeat if necessary. Label them in this order and keep an eye on the counts for each class as there is always the chance a class is not well represented by this approach. Another potential drawback in this approach is selecting the number of clusters. You could aim for the number of classes in the dataset or go with anything up to 10K clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An additional variant on this would be the strategy for selecting a sample from the clusters: instead of choosing a random point, choose the sample closest to the centroid; or if choosing multiple points from a cluster, order the samples via distance from centroid and sample uniformly from this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Active learning&lt;/strong&gt; is a semi-supverised approach that may be the most useful choice in this instance. When labelling, once you've got some decent coverage for each of your classes try an active learning approach to selecting which samples to label next.&lt;/p&gt;&#xA;" OwnerUserId="31282" LastEditorUserId="31282" LastEditDate="2017-04-19T02:04:18.633" LastActivityDate="2017-04-19T02:04:18.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18439" PostTypeId="2" ParentId="18437" CreationDate="2017-04-19T01:54:47.433" Score="4" Body="&lt;p&gt;15K features is a lot of features. I'd bet your 0.01 F1 score is a direct result of having so many features. Without knowing more details about your data it is hard to suggest a more sensible alternative. I'd want it to be closer to 100 features rather than 15,000 though. Are these features extracted from text? N-grams?&lt;/p&gt;&#xA;&lt;p&gt;Regarding your feature selection problem, the first step is to eliminate any features with no variance (all samples the same), then one can utilis various statistical methods for selecting features to retain those with high variance (sklearn has great libraries for this: &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature selection&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;One could also utilize Ranked Guide Iterative Feature Elimination (RGIFE) for this purpose, which uses random forests: &lt;a href=&quot;http://ico2s.org/software/rgife.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RGIFE - Ranked Guided Iterative Feature Elimination&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;P.S. Your class ratio is very skewed as well. Once you're all settled with a more reasonable dataset in feature space I'd consider either undersampling the larger class, oversampling the smaller class or a combination of both, and keep an eye on the resulting confusion matrices when performing your evaluation.&lt;/p&gt;&#xA;" OwnerUserId="31282" LastEditorUserId="85045" LastEditDate="2021-02-09T17:01:05.120" LastActivityDate="2021-02-09T17:01:05.120" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="18442" PostTypeId="1" CreationDate="2017-04-19T05:24:04.290" Score="3" ViewCount="5328" Body="&lt;p&gt;I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def tokenize_tweets():&#xA;    for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):&#xA;        tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])&#xA;        try:&#xA;            mod_text = tokenize(tweet_text)&#xA;            tokens = tknz.tokenize(mod_text)&#xA;            if len(tokens) &amp;gt; 0:&#xA;                yield tknz.tokenize(mod_text)&#xA;            else:&#xA;                yield ['NULL']&#xA;        except UnicodeEncodeError as e:&#xA;            yield ['&amp;lt;NULL&amp;gt;']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Not sure what is wrong with it.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2019-08-21T07:37:38.033" Title="Gensim word2vec training error on tweets" Tags="|word2vec|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18447" PostTypeId="1" AcceptedAnswerId="18455" CreationDate="2017-04-19T08:58:18.707" Score="8" ViewCount="922" Body="&lt;p&gt;As part of big data analysis project, I'm working on,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to perform PCA on some data, using cloud computing system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I'm using Amazon EMR for the job and Spark in particular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Leaving the &quot;How-to-perform-PCA-in-spark&quot; question aside, I want to get an understanding of how things work behind the scenes when it comes to calculating PCs on cloud-based architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How then is the covariance matrix calculated in such case when each node have only partial data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just an example. I'm trying to find some paper or documentation explaining all this behind-the-scenes voodoo, and couldn't find anything good enough for my needs (probably my poor google skills).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I can basically summarize my question(s) \ needs to be the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. How distributed PCA on cloud architecture works&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Preferably some academic paper or other sorts of explanation which also contains some visuals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Spark implementation of D-PCA&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does Spark do it? Do they have any 'twist' in their architecture to do it more efficiently, or how does the RDD objects usage contribute to improving the efficiency? etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A presentation of even an online lesson regarding it would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance to anyone who can provide some reading material.&lt;/p&gt;&#xA;" OwnerUserId="31298" LastEditorUserId="31298" LastEditDate="2017-04-19T11:24:27.260" LastActivityDate="2018-10-27T00:44:36.140" Title="Understanding how distributed PCA works" Tags="|data-mining|bigdata|apache-spark|pca|distributed|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18448" PostTypeId="2" ParentId="18416" CreationDate="2017-04-19T09:09:31.883" Score="0" Body="&lt;p&gt;If you are interested, I recommend you try the &lt;strong&gt;Ryskamp Machine Learning (RLM)&lt;/strong&gt; engine in your project. You can visit our &lt;a href=&quot;http://useaible.com/code/&quot; rel=&quot;nofollow noreferrer&quot;&gt;site&lt;/a&gt; to learn more about it and, from there, you can get the source code (hosted on github) and give it a shot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the RLM is very different from traditional machine learning. Although some basic concepts still apply such as inputs, outputs, etc. but one (out of 7 breakthroughs) of what really sets it apart is the core algorithm. The RLM does not apply the same mathematical algorithms used by traditional ML rather it goes about the problem in a logical approach. See the 7 breakthroughs on our site to learn more about the differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your project, here are some suggestions on how you can apply the RLM: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You could have it do Supervised Training. Like you will do in traditional ML, you should gather the participants data together with the ratings you mentioned. With the dataset in hand, you can then proceed training the RLM with it. You could set the input to always 1 while the output to be the three fruit ratios. The reason you have 1 as your input is because you are trying to find the best ratio for the entire population. Where you to find the best ratio for certain groups of participants, you could then have different values on your inputs (i.e., 1 = Participants Age &gt;= 30, 2 = Participants Age &amp;lt; 30, 3 = etc..). That way you can find what fruit ratio was optimal for group 1, group 2 or more.&lt;/li&gt;&#xA;&lt;li&gt;Or, you could do Unsupervised Training. Instead of preparing a predefined list of ratios to let your participants try and rate, the RLM can do that as it will try different ratios until it comes up with the optimal one. You can still follow the same inputs and outputs I mentioned in #1. Although, I worry that having only 200 participants might not be enough to fully find the optimal fruit ratio using this method.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On both options, the participant's rating (1-7 as you mentioned) is the Score you will provide to the RLM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps. If ever you need any technical assistance or feedback, please feel free to reach out to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;P.S.:&lt;/strong&gt; Although the RLM has been recently open sourced, there are some limitations to its use. Please be sure to read the &lt;a href=&quot;https://github.com/useaible/RyskampLearningMachine/blob/master/License.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;license&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31293" LastActivityDate="2017-04-19T09:09:31.883" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18450" PostTypeId="1" CreationDate="2017-04-19T10:17:18.920" Score="7" ViewCount="1411" Body="&lt;p&gt;I have a vector and want to detect outliers in it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need an outlier detection method (a non-parametric method) which can just detect red points as outliers. I tested some methods like IQR, standard deviation but they detect yellow points as outliers too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it is hard to detect just the red point but I think there should be a way (even combination of methods) to solve this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CY2rV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CY2rV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points are readings of a sensor for a day. But the values of the sensor change because of system reconfiguration (the environment is not static). The times of the reconfigurations are unknown. Blue points are for the period before reconfiguration. Yellow points are for after the reconfiguration which causes deviation in the distribution of the readings (but are normal). Red points are results of illegal modifying of the yellow points. In other words, they are anomalies which should be detected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering whether the Kernel smoothing function estimate ('pdf','survivor','cdf',etc.) could help or not. Would anyone help about their main functionality (or other smoothing methods) and justification to use in a context for solving a problem?&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="29575" LastEditDate="2017-12-18T14:30:01.130" LastActivityDate="2019-10-19T08:09:02.130" Title="Which outlier detection can detect these outliers?" Tags="|unsupervised-learning|anomaly-detection|outlier|distribution|" AnswerCount="3" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18451" PostTypeId="1" CreationDate="2017-04-19T10:55:29.983" Score="0" ViewCount="527" Body="&lt;p&gt;I have a set of 31390 individuals with an associated weight (kg) and I want to select a number of them such that the maximum weight is 3000kgs and that the weight distribution around several variables describing the sample are close to some targets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was approaching the problem by considering each individual as a gene of a chromosome and then multiplying the chromosome by the weight of the sample. In this case 1 is for being included in the sample while 0 is being excluded from the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I followed the example used in &lt;a href=&quot;https://www.r-bloggers.com/genetic-algorithms-a-simple-r-example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.r-bloggers.com/genetic-algorithms-a-simple-r-example/&lt;/a&gt; although my constraint is different as I want to minimize the difference between the weight distribution I get from the selected sample and my target. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking 1000 iterations and populations of 50 and 100 I don't get satisfactory solutions. I have read that people take populations that are (# genes)^2 but I will run out of memory if I take a population of that size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think crossover could help but it seems as if the genalg library only uses mutation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to solve the problem? Thanks. I have posted the code below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Weight.Limit&amp;lt;-30*100&#xA;&#xA;evalFunc &amp;lt;- function(x) {&#xA;  current_weight &amp;lt;- as.numeric(x %*% ind.data$weight)&#xA;  sample.dist&amp;lt;-x %*% as.matrix(ind.data[,4:ncol(ind.data)])/current_weight&#xA;  current_solution_dist &amp;lt;-  sum(sqrt(abs(constraints.data-sample.dist)))&#xA;&#xA;  if (current_weight&amp;gt; Weight.Limit) &#xA;    return(0) else return(current_solution_dist)&#xA;}&#xA;&#xA;&#xA;GAmodel &amp;lt;- rbga.bin(size = 31390, popSize = PopSize, iters = iter, mutationChance = MutationChance, elitism = T, evalFunc = evalFunc)&#xA;&#xA;cat(summary(GAmodel))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31303" LastActivityDate="2017-06-18T15:35:17.110" Title="selecting sample from population using genetic algorithm" Tags="|r|genetic-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18455" PostTypeId="2" ParentId="18447" CreationDate="2017-04-19T12:16:14.293" Score="5" Body="&lt;p&gt;The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.&lt;/p&gt;&#xA;&lt;p&gt;When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.&lt;/p&gt;&#xA;&lt;p&gt;How then is the covariance matrix calculated in such case when each node have only partial data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I shall address that, which hopefully will clear the matter to a degree.&lt;/p&gt;&#xA;&lt;p&gt;Let us look at a common form of covariance calculation, &lt;span class=&quot;math-container&quot;&gt;$\frac{1}{n}\sum(x-\bar{x})(y-\bar{y})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;This requires you to calculate the following:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\bar{x}$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\bar{y}$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$x-\bar{x}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y-\bar{y}$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;Multiply the &lt;span class=&quot;math-container&quot;&gt;$(x-\bar{x})$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$(y-\bar{y})$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;in a distributed manner. The rest is simple, let us say I have 100 datat points (x,y), which is distributed to 10 Apache Spark workers, each getting 10 data points.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Calculating the &lt;span class=&quot;math-container&quot;&gt;$\bar{x}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\bar{y}$&lt;/span&gt;&lt;/strong&gt;: Each worker will add &lt;span class=&quot;math-container&quot;&gt;$x/y$&lt;/span&gt; values of 10 data points and divide this by 10 to arrive at partial mean of &lt;span class=&quot;math-container&quot;&gt;$x/y$&lt;/span&gt; (this is the map function). Then the Spark master will run the aggregation step (in Spark DAG of the job) where the partial means from all 10 workers are taken and again added, then divided by 10 to arrive at the final &lt;span class=&quot;math-container&quot;&gt;$\bar{x}$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$\bar{y}$&lt;/span&gt; (the aggregate/reduce operation)&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Calculating the &lt;span class=&quot;math-container&quot;&gt;$(x-\bar{x}) \cdot (y-\bar{y})$&lt;/span&gt;&lt;/strong&gt;: Same way, distribute the data points, broadcast the &lt;span class=&quot;math-container&quot;&gt;$\bar{x}$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\bar{y}$&lt;/span&gt; values to all the workers and the calculate the partial &lt;span class=&quot;math-container&quot;&gt;$(x-\bar{x}) \cdot (y-\bar{y})$&lt;/span&gt;, again run aggregation to get &lt;span class=&quot;math-container&quot;&gt;$\sum (x-\bar{x})(y-\bar{y})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The above method is used for distributed calculation, you shall get the covariance, for multi-dimensional data, you can get the covariance matrix.&lt;/p&gt;&#xA;&lt;p&gt;The point is to distribute the calculation for stages that can be distributed and then centralize the calculation stages that cannot be distributed. That is in effect one of the important aspect of Spark architecture.&lt;/p&gt;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26814" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-10-27T00:44:36.140" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18456" PostTypeId="1" CreationDate="2017-04-19T13:37:26.893" Score="5" ViewCount="4831" Body="&lt;p&gt;In a classification problem when the response variable has multi-class, e.g., &quot;sunny&quot;,&quot;rainy&quot;,&quot;cloudy&quot;, how should we encode it? I know that for predictors like this, usually we do &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;noreferrer&quot;&gt;One Hot Encoding&lt;/a&gt;, and if a predictors have too many classes, then we might just use the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot; rel=&quot;noreferrer&quot;&gt;LabelEncode&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When this multi-class issue occurs in the response variable, I guess we can just use LabelEncode() rather use using One Hot encoding right? Because if we use One Hot encoding, then we will have 2 variables as the response variable, and the machine learning algorithm in sklearn usually expects the response variable not to be a vector right? (I mean it expects a long 1D vector with length equal to the number of observations but not a 2D matrix). But on the other hand, if we just map &quot;sunny&quot;, &quot;rainy&quot;,&quot;cloudy&quot; to {1,2,3} or {0,1,2}, or whatever 3 numbers, that will create a less than or greater than relationship among &quot;sunny&quot;, &quot;rainy&quot;,&quot;cloudy&quot;, which is not inherited in the original problem.&lt;/p&gt;&#xA;" OwnerUserId="30170" LastEditorUserId="30170" LastEditDate="2017-04-19T14:12:42.067" LastActivityDate="2021-01-01T17:33:17.890" Title="Encode multi-class response variable" Tags="|scikit-learn|multiclass-classification|prediction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18457" PostTypeId="1" CreationDate="2017-04-19T13:43:35.413" Score="7" ViewCount="2083" Body="&lt;p&gt;I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions.&lt;/p&gt;&#xA;&lt;p&gt;My population can be segmented into various subpopulations of interest. I've obtained the transition matrices for each of these subpopulations and would like to know if these subpopulations differ from the general population in some principled way.&lt;/p&gt;&#xA;&lt;p&gt;I don't know of any principled way of comparing transition matrices in this way. Comparing the transition matrices row-wise to that of the general population seems like one approach, but I'm not sure how to go about interpreting this. Another approach might be a spectral/eigendecomposition approach, which is much more readily interpretable to me, but might be harder to squeeze insight/stylised facts from.&lt;/p&gt;&#xA;&lt;p&gt;I've had a cursory search of the literature without much luck. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="24895" LastEditorUserId="83275" LastEditDate="2023-03-22T17:52:40.127" LastActivityDate="2023-12-23T19:02:05.310" Title="Comparing transition matrices for Markov chains" Tags="|matrix|markov-process|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="18458" PostTypeId="2" ParentId="17728" CreationDate="2017-04-19T13:48:54.117" Score="2" Body="&lt;p&gt;The problem with 1000 classes is the ILSVRC2012 challenge which is using a subset of the full imagenet with &quot;only&quot; 1000 classes. The advantage of that is that it is fixed while imagenet itself is always expanding and new classes and examples may be added. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training on the full ImageNet set is a very long task and will require either a very long time or a long time and a lot of computing powers (talking tens of GPUs). Although it's not short, training ILSVRC2012 can be done in one day up to a few days depending on your hardware resources and complexity of the model being training. &lt;/p&gt;&#xA;" OwnerUserId="31311" LastEditorUserId="31311" LastEditDate="2017-07-21T07:46:06.480" LastActivityDate="2017-07-21T07:46:06.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18459" PostTypeId="2" ParentId="18451" CreationDate="2017-04-19T14:06:21.590" Score="1" Body="&lt;p&gt;There's nothing super special about a population size of L^2. Larger populations are often better, but don't worry about trying to hit some threshold needed for the algorithm to work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of potential issues here that I see. I'm making some assumptions that when you say &quot;individuals&quot;, you mean people. Which means an average person is going to contribute something on the order of 50-100 kg toward your limit, which puts you at like 50 people in the sample in order to stay under the 3000kg limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your encoding scheme, that's going to mean only &lt;em&gt;very&lt;/em&gt; sparse vectors are feasible solutions. You can deal with that, but it likely means a standard GA pulled from a library won't work as well as one that included some prior knowledge. For example, when initializing your population, you might want to bias things pretty heavily in favor of setting bits to 0 rather than 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, you have constraints, so you'll want to include some type of constraint handling scheme. A simple thing you can do in this case is to perform a simple local search on every individual you generate. For example, suppose I have an individual &lt;em&gt;i&lt;/em&gt; selected from my population. If the total weight of the sample encoded by &lt;em&gt;i&lt;/em&gt; is less than 3000kg, iterate over every bit of &lt;em&gt;i&lt;/em&gt; (in random order) and see if you can flip a 0 bit to a 1 bit without exceeding your weight capacity. Similarly, if you generate a sample that weighs more than 3000kg, try to find 1 bits you can flip to remove weight until you're under the limit. When you're done, you have a feasible individual and you insert it into the child population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can incorporate more sophisticated measures or search schemes as well. You could have your local search operator consider those other variables. You can replace the simple hill-climbing operator with a more sophisticated method like tabu search or simulated annealing. There's a spectrum of algorithms with all sorts of different ways of combining the ideas of the global search of a GA with more targeted local search operators, and there's no general &quot;right answer&quot;. You mostly just need to experiment a bit to see what works.&lt;/p&gt;&#xA;" OwnerUserId="2975" LastActivityDate="2017-04-19T14:06:21.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18462" PostTypeId="1" CreationDate="2017-04-19T15:46:48.580" Score="-1" ViewCount="613" Body="&lt;p&gt;I have a problem I need to solve and am looking for assistance in what algorithm to use. I have a online store that I have say 10 products and I have all the order history for every order. What I am trying to find is if a customer orders product A what is the probability that they will order product A, Product B, Etc. for their second, third, etc. order. What algorithm would work best to find the probability of what product will be purchased in repeat orders? I would like to either use R or python but if it can be done with simple math and if statements that would be even better.&lt;/p&gt;&#xA;" OwnerUserId="31322" LastActivityDate="2017-05-19T19:14:50.090" Title="Probability of what product will be purchased in repeat orders" Tags="|machine-learning|r|probability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18463" PostTypeId="2" ParentId="18450" CreationDate="2017-04-19T16:25:17.453" Score="0" Body="&lt;p&gt;If you use logging you can use a running average that resets if the configuration changes. However, this will have the weakness that you need at least some data before you can detect such an outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data looks rather &quot;nice&quot; (not too much noise). I would recommend taking the average over the last 10-20 points in the same configuration. If these values are some kind of counted quantity you can take a poisson error for individual data points and calculate the error on the average. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How much historical data do you have? If you have a lot you can use it to fine tune your alarm rate in a way that you catch an acceptable ratio of all real outliers while getting a minimal number of fake warnings. What is acceptable depends on the specific problem. (Cost of False Positives or not detected outliers and their abundance).&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-04-19T16:25:17.453" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18464" PostTypeId="2" ParentId="18462" CreationDate="2017-04-19T17:34:08.587" Score="0" Body="&lt;p&gt;One way is to approach as a traditional probability problem: probability of simultaneous events.&lt;br&gt;&#xA;&lt;a href=&quot;https://math.stackexchange.com/questions/375771/how-do-you-calculate-the-probability-of-simultaneous-events&quot;&gt;https://math.stackexchange.com/questions/375771/how-do-you-calculate-the-probability-of-simultaneous-events&lt;/a&gt;&#xA;This may not apply because this is more focused on probabilities of events occurring simultaneously and it sounds like you want the probability of an event occurring (purchase of a product) given another event occurring (not independent events).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A method of handling this could therefore be categorical clustering.  This post has some very good explanations of k-modes clustering:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data/24#24&quot;&gt;K-Means clustering for mixed numeric and categorical data&lt;/a&gt;&#xA;MATLAB also has an implementation of this:&#xA;&lt;a href=&quot;https://www.mathworks.com/help/stats/kmedoids.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.mathworks.com/help/stats/kmedoids.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-04-19T17:34:08.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18465" PostTypeId="1" AcceptedAnswerId="18467" CreationDate="2017-04-19T18:04:11.283" Score="2" ViewCount="6930" Body="&lt;p&gt;My question would be about backpropagation and understanding the terms feedforward NN vs backpropagation. I have two questions really:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If I understand correctly, even a feedforward network updates its weights (via a delta rule for example). Isn't this also backpropagation? You have some random weights, run the data through the network, then cross-validate it with the desired output, then update the rules. This is backpropagation, right? So what's the difference between FFW NN and RNN? If you can't backpropagate on the other hand, how do you update the weights in a FFW NN?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've seen NN architectures looking like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/zn8uL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zn8uL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Basically all the neurons are being fed the same data, right? OK, the weights are randomized and thus different in the beginning, but how do you make sure Temp. Value #1 will be different from Temp. Value #2, if you use the same update rule?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2023-02-28T14:25:07.337" Title="Neural networks - adjusting weights" Tags="|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18467" PostTypeId="2" ParentId="18465" CreationDate="2017-04-19T19:17:03.167" Score="1" Body="&lt;p&gt;I'm not an expert on the backpropagation algorithm, however I can explain something. Every neural network can update it's weights. It may do this in different ways, but it can. This is called &lt;strong&gt;backpropagation&lt;/strong&gt;, regardless of the network architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A feed forward network is a regular network, as seen in your picture. A value is received by a neuron, then passed on to the next one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nDUJ4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nDUJ4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recurrent neural network is almost the same as a FFN, the difference being that the RNN has some connections point 'backwards'. E.g. a neuron is connecteded to a neuron that has already done his 'job' during backpropagation. Because of this, the activations of the previous output have an effect on the new output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4x8Bb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4x8Bb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On question #2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interesting question. This has to do with weight initialization. Yes, you're right, each neuron in the hidden layer accepts the same connections. However, during the initalization process, they have received a &lt;strong&gt;random weight.&lt;/strong&gt; Depending on your NN libary, the neurons might also have been initialized with a &lt;strong&gt;random bias&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So even though the same rule is applied, each neuron has different outcomes as all it's connections have different weights than the other neurons weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On your comment: just because all the neurons happen to have the same backpropagation function, doesn't mean they will end up with the same weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As they are initialized with random weights, each neurons &lt;code&gt;error&lt;/code&gt; is different. Thus they have a different gradient, and will get new weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also have to keep in mind that for a certain output to be reached, there are multiple solutions (due to non-linearity). So due to initialized random weights, one neuron might be close to a certain solution while another neuron is closer to the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, as was stated in the comments, &lt;strong&gt;a network works as a whole&lt;/strong&gt;. The output neuron is also non-linear, and for most test cases, the output should be non-linear and the output neuron most likely requires that the hidden neurons activate at different input values.&lt;/p&gt;&#xA;" OwnerUserId="30928" LastEditorUserId="30928" LastEditDate="2017-04-20T08:56:47.623" LastActivityDate="2017-04-20T08:56:47.623" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18468" PostTypeId="1" CreationDate="2017-04-19T20:50:48.347" Score="-2" ViewCount="228" Body="&lt;p&gt;I am using Python to do weather forecasting.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/drive/folders/0BxxxnsxpOeLNUHRPdHdYUW9NYkk&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the original data.&lt;/p&gt;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&lt;p&gt;The input has 180 features and the meanings are:&lt;/p&gt;&#xA;&lt;p&gt;Suppose we are doing forecast for hour k. We will use the historic weather data (past 3 hours) and the weather forecast (hour k).&lt;/p&gt;&#xA;&lt;p&gt;For the wind generator, the 180 input features (4x11x4+4) are:&lt;/p&gt;&#xA;&lt;p&gt;Station 1 Hour k-3: [temperature, humidity, wind speed, wind direction] ... Hour k: [temperature, humidity, wind speed, wind direction]&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;Station 11 Hour k-3: [temperature, humidity, wind speed, wind direction] ... Hour k: [temperature, humidity, wind speed, wind direction]&lt;/p&gt;&#xA;&lt;p&gt;Time related features [sin(hour), cos(hour), sin(day_of_year), cos(day_of_year)]&lt;/p&gt;&#xA;&lt;p&gt;4&lt;em&gt;11&lt;/em&gt;4 +4 = 180&lt;/p&gt;&#xA;&lt;p&gt;And one output is the power.&lt;/p&gt;&#xA;&lt;p&gt;The hint is,&#xA;This data is not clean, in that there is some portion of bad data in the target/output data (not the input data). However, it is hard to know which data is bad. You may consider how to clean the data first (more about sample selection, not feature selection). After data cleaning, you may then consider feature selection to reduce the input dimension. There a lot of feature selection methods available in literature.&lt;/p&gt;&#xA;&lt;p&gt;I wonder how to clean the data and then do feature selection. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="31335" LastEditorUserId="25180" LastEditDate="2021-04-09T13:19:53.947" LastActivityDate="2021-04-09T13:19:53.947" Title="clean the data with sample selection?" Tags="|machine-learning|python|feature-selection|data-cleaning|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18470" PostTypeId="1" CreationDate="2017-04-19T21:32:55.797" Score="3" ViewCount="264" Body="&lt;p&gt;In a paper titled Generative adversarial networks (link in comment - sorry I don't have enough reputation to include more than 2 links), an explanation of generative adversarial net is provided as following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Xxc8X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xxc8X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/6TlLM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6TlLM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 questions about the explanation of figure 1:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;G contracts in regions of high density and expands in regions of low density of p_g&quot;. I'm interpreting this to mean that the maximum of the green curve indicates region of contractions, while the region of curve flattening indicates region of increasing expansion. Why are the concepts of contractions and expansions visually represented in such a manner ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The blue dashed line is supposed to indicate the discriminative distribution's discrimination between samples from the data generating distribution and the generative distribution. It's unclear to me how the blue dashed line represents the discrimination (why does a straight horizontal line represents the discriminator's inability to differentiate between samples taken from real data and taken from the generative distribution?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="31179" LastActivityDate="2017-04-19T21:32:55.797" Title="Explanation of generative adversarial net in Generative adversarial networks by IJ Goodfellow and et at. (2014)" Tags="|gan|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18471" PostTypeId="2" ParentId="18468" CreationDate="2017-04-19T23:32:56.910" Score="0" Body="&lt;p&gt;Data cleaning typically requires knowledge of the application domain.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Try to identify ranges of reasonable values for each of the input features, based on your understanding of what the feature represents.  For instance, a wind speed of 2000 mph is probably not real.  A humidity value of 300% or -50% is probably not right.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, try to work out a plausible range for how much the feature value should or should not vary from one sample to the next.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;More generally, if you have a model for how these values might evolve, you can look for samples that are grotesquely implausible according to the model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Visualizing the data is also typically helpful.  For instance, you can look for outliers and examine them more closely with manual inspection.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-19T23:32:56.910" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18472" PostTypeId="2" ParentId="18456" CreationDate="2017-04-19T23:42:58.177" Score="5" Body="&lt;p&gt;It depends on the meaning of the classes, and whether they have any meaningful order.&lt;/p&gt;&#xA;&lt;p&gt;If they are &lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_data&quot; rel=&quot;nofollow noreferrer&quot;&gt;ordinal&lt;/a&gt; or a scale, then there is a meaningful ordering, and it can potentially be reasonable to order them and then assign labels &lt;span class=&quot;math-container&quot;&gt;$1, 2, \dots, n$&lt;/span&gt; in order to the classes.  This is what you call &amp;quot;LabelEncode&amp;quot;.  In some cases, some form of regression might also be appropriate and worth trying.&lt;/p&gt;&#xA;&lt;p&gt;If they are &lt;a href=&quot;https://en.wikipedia.org/wiki/Categorical_variable&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Level_of_measurement#Nominal_level&quot; rel=&quot;nofollow noreferrer&quot;&gt;nominal&lt;/a&gt;, with no meaningful ordering, then LabelEncode makes no sense and will be no better than a one-hot encoding -- and potentially worse.  So, the default is one-hot encoding.&lt;/p&gt;&#xA;&lt;p&gt;If they are categorical but have some hierarchy, there are &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification#Hierarchical_Classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;smarter methods&lt;/a&gt;.  You can build a tree that represents the hierarchy, then have a multi-hot encoding.  If there are &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; classes, you will have a tree with &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; leaves and (say) &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; nodes in total  (where &lt;span class=&quot;math-container&quot;&gt;$m &amp;gt; n$&lt;/span&gt;).  Then you can encode the class as a &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt;-vector, whose &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;th entry is 1 if the class is a descendant of node &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; in the tree, or 0 otherwise.&lt;/p&gt;&#xA;&lt;p&gt;If the classes are categorical but have various attributes, you can try an encoding that has one element for each possible attribute.  For instance, if there are &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt; binary attributes, you can have a &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt;-vector with a 0 or 1 in the &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;th entry according to whether the corresponding class has the &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;th attribute or not.  You can generalize from there for more general situations.&lt;/p&gt;&#xA;&lt;p&gt;See &lt;a href=&quot;https://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_data_type&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for more about different types of measurements.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="85045" LastEditDate="2021-01-01T13:24:33.650" LastActivityDate="2021-01-01T13:24:33.650" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18475" PostTypeId="1" AcceptedAnswerId="18476" CreationDate="2017-04-20T07:45:21.710" Score="1" ViewCount="242" Body="&lt;p&gt;I have a table like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FirstName    SecondName     Amount&#xA;Lorenzo      Perone         100&#xA;Mario        Rossi          25&#xA;...          ...            ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to create a &quot;word cloud&quot; using &quot;Amount&quot; as weight, is it possible using the &quot;word cloud&quot; tool?&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="31361" LastEditorUserId="31361" LastEditDate="2017-04-20T10:17:21.087" LastActivityDate="2018-08-24T08:01:11.463" Title="Can I create a word cloud of crowdfunding donors using word cloud?" Tags="|text-mining|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18476" PostTypeId="2" ParentId="18475" CreationDate="2017-04-20T08:38:09.423" Score="2" Body="&lt;p&gt;I will give you a simple solution using &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; which requires the &lt;a href=&quot;https://cran.r-project.org/web/packages/wordcloud/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;wordcloud&lt;/a&gt; package. Of course there are many other solutions which do not require any programming skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is a slight varient of this &lt;a href=&quot;https://www.r-bloggers.com/word-clouds-using-text-mining/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R-Bloggers tutorial&lt;/a&gt;. Feel free to have a look there for further formatting.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(wordcloud)&#xA;&#xA;words = c('Paolo Gentiloni', 'Matteo Renzi', &#xA;          'Enrico Letta', 'Mario Monti', &#xA;          'Silvio Berlusconi', 'Romano Prodi')&#xA;freq = c(100, 25, 50, 70, 95, 20)&#xA;&#xA;wordcloud(words = words, freq = freq, min.freq = 1,&#xA;          max.words=200, random.order=FALSE, rot.per=0.35, &#xA;          colors=brewer.pal(8, &quot;Dark2&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It produces the following output:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/OQ8Zj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OQ8Zj.png&quot; alt=&quot;Wordcloud of Italien prime ministers and random numbers&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw. @Lorenzo Perone: Where the ones listed Italien names? I was not sure about that.&lt;/p&gt;&#xA;" OwnerUserId="15946" LastEditorUserId="15946" LastEditDate="2017-04-20T10:34:01.223" LastActivityDate="2017-04-20T10:34:01.223" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18477" PostTypeId="1" CreationDate="2017-04-20T09:29:26.883" Score="-2" ViewCount="369" Body="&lt;p&gt;I have a question about how to derive the bottom formula under &quot;Fixing the other ai's ..&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L8Hta.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L8Hta.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the completed PDF file &lt;a href=&quot;http://web.cs.iastate.edu/~honavar/smo-svm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sequential Minimal Optimization for SVM&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The screenshot is page 9 of this file.&lt;/p&gt;&#xA;" OwnerUserId="31364" LastEditorUserId="31364" LastEditDate="2017-04-20T09:37:06.327" LastActivityDate="2021-02-06T18:35:24.563" Title="An formula derivation question about SMO algorithm of SVM" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18478" PostTypeId="1" CreationDate="2017-04-20T11:46:42.733" Score="2" ViewCount="369" Body="&lt;p&gt;I study Pattern Recognition (Classification, Clustering, NeuralNet, Decision tree, and so on. But, excluding textual analysis), and Optimization (study of computational complexity, and trade off of computation and complexity of the target hypotheses, genetic algorithm, and so on) in my university.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If somewhere else somebody studies &quot;Machine Learning&quot;, can I assume that they are actually studying Pattern Recognition + Optimization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, is machine learning the study of pattern recognition along with optimization?&lt;/p&gt;&#xA;" OwnerDisplayName="user9232" LastEditorDisplayName="user9232" LastEditDate="2017-04-20T13:55:55.113" LastActivityDate="2017-04-21T00:35:14.663" Title="Is it true that Machine Learning is basically the study of Pattern recognition along with Optimization?" Tags="|machine-learning|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18479" PostTypeId="2" ParentId="18478" CreationDate="2017-04-20T11:49:48.447" Score="2" Body="&lt;p&gt;Maybe? As Wikipedia says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The terms pattern recognition, machine learning, data mining and&#xA;  knowledge discovery in databases (KDD) are hard to separate, as they&#xA;  largely overlap in their scope.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pattern_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Pattern_recognition&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-20T11:49:48.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18481" PostTypeId="1" AcceptedAnswerId="18483" CreationDate="2017-04-20T13:24:46.320" Score="14" ViewCount="8344" Body="&lt;p&gt;Experts in my field are capable of &lt;strong&gt;predicting the likelyhood an event (binary spike in yellow)&lt;/strong&gt;  30 minutes &lt;strong&gt;before it occurs&lt;/strong&gt;. Frequency here is 1 sec, this view represents a few hours worth of data, &lt;strong&gt;i have circled in black where &quot;malicious&quot; pattern should be&lt;/strong&gt;.&#xA;Interactions between the dimensions exist, therefore dimensions cannot be studied individually (or can they?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bhiom.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bhiom.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to build a &lt;strong&gt;supervised&lt;/strong&gt; ML model using Scikit Learn &lt;strong&gt;which learns a normal rythm, and detects when symptoms might lead to a spike&lt;/strong&gt;. I am lost for which direction to take. I have tried Anomaly detection, but it only works for on the spot detection, not prior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How could I detect &quot;malicious&quot; patterns prior to those events (taking them as target variables) ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I welcome any advice on which algorithms or data processing pipeline might help, thank you :)&lt;/p&gt;&#xA;" OwnerUserId="31260" LastActivityDate="2020-08-01T13:12:34.087" Title="How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries" Tags="|machine-learning|python|predictive-modeling|time-series|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18482" PostTypeId="2" ParentId="18481" CreationDate="2017-04-20T13:39:46.813" Score="0" Body="&lt;p&gt;You need to do feature extraction or feature engineering to create variables in your training data that &quot;catch&quot; those patterns you boxed in and then have a target variable saying &quot;malice found&quot; or &quot;malice found not found&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a really simple example: predicting if it is going to rain. You could come up with a reasonably good predictor of rain in the next 30 minutes that checked every 30 minutes if 1. it suddenly got cloudy and 2. barometric pressure dropped. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-04-20T13:39:46.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18483" PostTypeId="2" ParentId="18481" CreationDate="2017-04-20T14:19:01.563" Score="13" Body="&lt;p&gt;This is a fun problem. This is a time series and from this time series you want to identify the trigger of a certain event. So it is a binary classification problem. Based on the information from the specified window will a spike occur? Yes or No.&lt;/p&gt;&#xA;&lt;p&gt;The first step is to set up your database. What you will have is a set of instances (which can have some overlap but to avoid bias it is best for them to be independently drawn) and then for each instance a human needs to label if there was a spike or if there was not a spike.&lt;/p&gt;&#xA;&lt;p&gt;Then you need to identify the time window you want to use for your time series analysis. You have done this and decided 30 minutes is a good start.&lt;/p&gt;&#xA;&lt;p&gt;Now, you have 6 waveforms in a 30 minute window from which you can extract data to get information about your classification. You can use the raw data samples as your features, but this is WAY TOO many features and will lead to poor results. Thus you need some &lt;strong&gt;feature extraction&lt;/strong&gt;, &lt;strong&gt;dimensionality reduction&lt;/strong&gt;, techniques.&lt;/p&gt;&#xA;&lt;p&gt;There are a million ways you can extract data from these waveforms. First, ask yourself, as a human &lt;strong&gt;what are the telltale signs&lt;/strong&gt; that these other waveforms should have which would mean a spike would arise. For example, in seismic data, if you see agitation in a waveform from a neighboring town then you should expect to see agitation in your town soon.&lt;/p&gt;&#xA;&lt;p&gt;In general, I like to &lt;strong&gt;extract all the basic statistics&lt;/strong&gt; from my waveforms. Get the mean, standard deviation, fluctuation index, etc. Get whatever you think might help. Check how these statistics correlate with your labels. The more correlation the better they might be. Then there are some very good techniques for extracting time and frequency information from your time-series. Look into &lt;strong&gt;envelope mode decomposition&lt;/strong&gt; and &lt;strong&gt;empirical mode decomposition&lt;/strong&gt;. I have used empirical mode decomposition successfully on some time series data and obtained far better results than I expected.&lt;/p&gt;&#xA;&lt;p&gt;Now even though you have your reduced feature space you can do better! You can apply some dimensionality reduction techniques such as &lt;strong&gt;PCA&lt;/strong&gt; or &lt;strong&gt;LDA&lt;/strong&gt; to get a lower dimensional space which may better represent your data. This might help, no guarantees.&lt;/p&gt;&#xA;&lt;p&gt;Now you have a small dataset with instances that are a Frankenstein concoction which represents your 6 waveforms across the 30 minute window. Now you are all set to select your classifier. You will want a binary classification algorithm, luckily that is the most common. There are many to choose from. How to choose?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;How many instances do you have?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;span class=&quot;math-container&quot;&gt;$\# instances &amp;gt; 100* \#features$&lt;/span&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Then you are all set to use a deep learning technique such as &lt;strong&gt;neural networks&lt;/strong&gt;, &lt;strong&gt;1D convolutional neural networks&lt;/strong&gt;, &lt;strong&gt;stacked autoencodders&lt;/strong&gt;, etc...&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Less than that!!!!&lt;/strong&gt;&#xA;The you should stick with shallow methods. Check out &lt;strong&gt;kernel support vector machines&lt;/strong&gt;, &lt;strong&gt;random forests&lt;/strong&gt;, k-nearest neighbors etc..&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Common misconception: A shallow method &lt;strong&gt;CAN&lt;/strong&gt; and &lt;strong&gt;WILL&lt;/strong&gt; perform better than a deep learning technique if you have properly selected your features. feature extraction is the most important aspect of a machine learning architecture.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;I want to use anomaly detection!&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;This would also work and there are some good techniques that would do this. However, the nature of anomaly detection is to learn the distribution of the nominal case. So you would feed your algorithm all the instances in your dataset that did not result in a spike. Then from this your algorithm would be able to identify when a novel instance is significantly different from this nominal distribution and it will flag it as a n anomaly. This would mean that a spike will occur in your context.&lt;/p&gt;&#xA;&lt;p&gt;Check out:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.stat.rice.edu/%7Ecscott/pubs/minvol06jmlr.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Minimum Volume Sets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/0910.5461&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anomaly Detection with Score functions based on Nearest Neighbor Graphs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/document/6319713/&quot; rel=&quot;nofollow noreferrer&quot;&gt;New statistic in P-value estimation for anomaly detection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="98307" LastEditDate="2020-08-01T13:12:34.087" LastActivityDate="2020-08-01T13:12:34.087" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18485" PostTypeId="2" ParentId="10228" CreationDate="2017-04-20T15:37:50.133" Score="21" Body="&lt;p&gt;For the case of a variable with two values, appearing with fractions &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$(1-f)$&lt;/span&gt;, &lt;br&gt;&#xA;the gini and entropy are given by:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$gini = 2*f(1-f)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$entropy = f*ln\big({1\over f}\big) + (1-f)*ln\big({1\over(1-f)}\big)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;These measures are very similar if scaled to &lt;span class=&quot;math-container&quot;&gt;$1.0$&lt;/span&gt;&#xA;(plotting &lt;span class=&quot;math-container&quot;&gt;$2*gini$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;${entropy\over ln(2)}$&lt;/span&gt;):&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ue6OB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ue6OB.jpg&quot; alt=&quot;Gini (y4,purple) and Entropy (y3,green) values scaled for comparison&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31384" LastEditorUserId="29169" LastEditDate="2021-09-24T05:14:12.623" LastActivityDate="2021-09-24T05:14:12.623" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18486" PostTypeId="2" ParentId="12910" CreationDate="2017-04-20T15:46:02.590" Score="1" Body="&lt;p&gt;I don't have experience with Stanford Classifier, but if it is sending output to stdout that you would like to capture, you can simply pipe the output to a file. You didn't mention an OS, but on most the &lt;code&gt;&amp;gt;&lt;/code&gt; operator is used to redirect &lt;code&gt;stdout&lt;/code&gt; to a file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you are running Standard Classifier like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;java -cp stanford-classifier.jar [option...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can instead run it like this to output to a file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;java -cp stanford-classifier.jar [option...] &amp;gt; my-output.tsv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15620" LastActivityDate="2017-04-20T15:46:02.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18487" PostTypeId="1" CreationDate="2017-04-20T16:08:32.963" Score="0" ViewCount="1961" Body="&lt;p&gt;What is the correct way to fine-tune a model's (SARSA(0), SARSA(lambda), Q(0), Q(lambda)) parameters, and how can one compare the models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read that typically one compares the number of actions needed to achieve the goal, after X episodes. However, my model is slightly different than maze/grid worlds, so I'm not sure on how to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially its a time model, with X-days, and each day an action is selected, the reward on each day depends on the action selected, and the end-state reward depends on the actions selected in the other 249 days. It is an optimization model for a regulatory situation, which implies revising the whole reward every 250 days, based on the performance (which is tied to actions) of the other 249 days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The models are being trained in a simulation, the optimal policy will be achieved once per model. The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally one would compare all possible models - as there are quite a few - with a smaller number of episodes, and the selected ones would then run with the full amount of episodes, for time-reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to compare the models? Based on average rewards with the optimal policy's actions?&lt;/p&gt;&#xA;" OwnerUserId="31385" LastEditorUserId="31513" LastEditDate="2017-05-10T08:04:02.610" LastActivityDate="2017-05-10T08:04:02.610" Title="Choosing the right parameters for SARSA and Q-Learning &amp; Comparing Models" Tags="|machine-learning|q-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18488" PostTypeId="1" AcceptedAnswerId="18519" CreationDate="2017-04-20T16:25:40.030" Score="0" ViewCount="1500" Body="&lt;p&gt;So I know about the curse of dimensionality (too many features too less data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have a 3000 sample dataset, would 3 features be too less?&lt;/p&gt;&#xA;" OwnerUserId="31346" LastActivityDate="2017-04-24T20:17:10.507" Title="Does the SVM require lots of features most of the time?" Tags="|svm|feature-construction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18489" PostTypeId="1" CreationDate="2017-04-20T17:55:36.963" Score="1" ViewCount="4575" Body="&lt;p&gt;I want to classify 8 noisy signals with &lt;em&gt;trainNetwork&lt;/em&gt;() function that is available in Matlab.  All Matlab examples were for image classification. The arguments are the data and its label, the CNN layers, in which the first layer is &lt;em&gt;imageInputLayer([32 32 3])&lt;/em&gt;. &#xA;My question is can this function be used to classify 1-D signals? if yes, how can we create such a layer?&lt;/p&gt;&#xA;" OwnerUserId="31391" LastEditorUserId="29169" LastEditDate="2021-02-16T02:43:09.023" LastActivityDate="2021-02-16T02:43:09.023" Title="Signal classification with convolution neural network" Tags="|classification|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18491" PostTypeId="2" ParentId="18450" CreationDate="2017-04-20T19:09:50.223" Score="4" Body="&lt;p&gt;You may view your data as a &lt;strong&gt;time series&lt;/strong&gt; where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are simulated sample data based on normal distribution with three different means similar to your example.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ATZlp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ATZlp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By calculating the difference to the previous value (a sort of derivation) you get the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gKffH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gKffH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My interpretation of your description is, that you tolerates re-calibration (i.e. points on greater distance from zero, red in the diagram), but they must interchange between positive and negative values (i.e. corresponding to shift from blue state to the yellow one and back). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means you &lt;strong&gt;can set up an alarm seeing a second one red point either on the negative or positive side&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2017-04-20T19:09:50.223" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18492" PostTypeId="2" ParentId="18488" CreationDate="2017-04-20T20:17:23.967" Score="0" Body="&lt;p&gt;I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it seems easier to guess to the &quot;positive&quot; side, i.e. &quot;might it work?&quot; then &quot;might it fail?&quot;. &#xA;I witnessed both small datasets (&amp;lt;1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (&amp;lt;5) features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The stage in the processing pipeline to tackle this, is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature selection&lt;/a&gt; one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You didn't mention a specific research environment, but I'd like to suggest scikit-learn &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.feature_selection.SelectKBest&lt;/a&gt; running inside a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;GridSearchCV&lt;/a&gt; pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_selection import SelectKBest&#xA;from sklearn.pipeline import make_pipeline&#xA;from sklearn.model_selection import GridSearchCV&#xA;from sklearn.svm import LinearSVC&#xA;&#xA;pipe = []&#xA;pipe.append(SelectKBest())&#xA;pipe.append(LinearSVC())&#xA;&#xA;est_pipe = make_pipeline(*pipe)&#xA;print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())&#xA;param_grid = {&#xA;  # some svc grid params..,&#xA;  'selectkbest__k': [5, 10, 50, 100]&#xA;}&#xA;&#xA;grid = GridSearchCV(est_pipe, param_grid=param_grid)&#xA;grid.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17177" LastEditorUserId="17177" LastEditDate="2017-04-20T20:43:48.413" LastActivityDate="2017-04-20T20:43:48.413" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18493" PostTypeId="2" ParentId="6052" CreationDate="2017-04-20T21:43:35.100" Score="3" Body="&lt;p&gt;If you have a premade dictionary of terms, like NLTK's &lt;code&gt;words.words()&lt;/code&gt; you can simply iterate through the string adding a space at each point and checking if they are both words. A couple possible issues come from this: 1) Compound words may be split unnecessarily and 2) tags with terms attached to compounded words would yield multiple possible results. This is where something like document distance or term frequency would come in. A simple example using a set of words called &lt;code&gt;WORDS&lt;/code&gt; would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def check_spacing(term):&#xA;    possibles = []&#xA;    for i in range(1, len(term) - 1):&#xA;        l, r = term[:i], term[i:]&#xA;&#xA;        if l in WORDS and r in WORDS:&#xA;            posibbles.append((l, r))&#xA;&#xA;    # probs_check is a theoretical function that returns a numeric value&#xA;    # which determines how likely each pair of words is to be what you want&#xA;    possibles = sorted(possibles, key=probs_check, reversed=True)&#xA;&#xA;    return ' '.join(possibles[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31399" LastActivityDate="2017-04-20T21:43:35.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18494" PostTypeId="1" CreationDate="2017-04-20T21:44:11.660" Score="0" ViewCount="4304" Body="&lt;p&gt;&lt;b&gt;Background:&lt;/b&gt; I work for a major airline in the United States and have been given a pretty significant text mining and machine learning task. Since there are several unique terms, acronyms, and abbreviations in aviation, I have written a program that reads through the documents. The program is using 4 groups of terms; &lt;code&gt;WORDS&lt;/code&gt;, &lt;code&gt;STOPS&lt;/code&gt;, &lt;code&gt;AIRPORTS&lt;/code&gt;, and &lt;code&gt;SYNONYMS&lt;/code&gt;. &lt;code&gt;WORDS&lt;/code&gt;, &lt;code&gt;STOPS&lt;/code&gt;, and &lt;code&gt;AIRPORTS&lt;/code&gt; are all sets where the first two were initialized from NLTK and the last was initialized with a manually built list of airports that my employer services. &lt;code&gt;SYNONYMS&lt;/code&gt; is a nested dictionary 'trie' style structure where the branches are misspellings and the leaves are the terms that they represent. it was initialized as an empty dictionary. I have also created a command prompt UI that informs me when it finds a word that it doesn't know and allows me to correct the word or add it to one of the aforementioned groups. The documents are being tokenized by spaCy and fed into scikit-learn machine learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Problem:&lt;/b&gt; I've started noticing that some of the words that I previously accepted in &lt;code&gt;WORDS&lt;/code&gt; or &lt;code&gt;SYNONYMS&lt;/code&gt; are showing up with alternate meanings and could have a drastic impact on how well the algorithm operates. For example the word act can be the verb act; or it can be an abbreviation for actuator or active; or it can be an acronym for auxiliary central tank or a couple other things. At present, I'm using the UI to manually correct or ignore these occurrences, which is time consuming, cumbersome, and prone to errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Desired result:&lt;/b&gt; What I'm looking for is a method of determining whether &lt;i&gt;'lt'&lt;/i&gt; means left or light depending on context (lt wing vs. warning lt) with minimal manual entry. At a minimum I would like some high level ideas/discussion on how this may be accomplished in python. Optimally, I would like some example code, preferably using the same tools I already have; spaCy, NLTK, and sklearn but I am flexible and willing to learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Notes:&lt;/b&gt; I've considered a couple methods of doing this but I haven't been able to come up with any ideas strong enough to try. For example, I've considered marking the terms with the pos inside the sets by concatenating it with the word before adding it like 'rat:noun'. The issue with this is that rat:noun could be an animal or it could be an abbreviation for ram air turbine, which is also a noun.&lt;/p&gt;&#xA;" OwnerUserId="31399" LastActivityDate="2018-10-22T19:30:47.697" Title="Determining usage/context of terms/acronyms/abbreviations in python" Tags="|python|nlp|text-mining|data-cleaning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18495" PostTypeId="1" CreationDate="2017-04-20T22:05:03.280" Score="-1" ViewCount="1482" Body="&lt;p&gt;I trained a model with results as below. It is a stacking model with base learners of random forest and gradient boosting. The mega model is a GLM. The dataset is imbalanced in the target class as shown in the confusion matrix on the right top. The target class is a default status of a single loan (Positive: default; Negative: non-default).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The AUC ROC score is quite high but the f1 score is still only 0.53. My concern is both of recall and precision were approximately 0.5, meaning that the model can only distinguish half of the bad cases and only half of bad cases it diagnosed were truly bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I adjust the probabilistic threshold, recall will increase but precision will also be sacrificed to some extent. Due to the highly imbalanced situation on the positive cases, a low precision may lead to a large proportion of False Negative among the predicted positive cases. e.g. under a lower precision, if a model tells 10 bad cases, there may be actually only 2 truly bad ones and more fake bad ones predicted which is not wanted in practice as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In predictive task on an imbalanced dataset, does f1 score matter and if so how can I further improve the score? (add new (composite) features, cost sensitive methods?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xuKF0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xuKF0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before asking this question, I have read the &lt;a href=&quot;http://www.pitt.edu/~jeffcohn/biblio/Jeni_Metrics.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; and get to know the skew of data does impact f1 score. But how to improve this case and reach a better predictive outcome for the practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HkYMe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HkYMe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2017-04-20T22:29:38.930" Title="How much should I pay attention to the f1 score on this case?" Tags="|machine-learning|predictive-modeling|prediction|model-evaluations|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18496" PostTypeId="2" ParentId="18495" CreationDate="2017-04-20T22:29:38.930" Score="0" Body="&lt;p&gt;Yes, F1 matters.  It is telling you that your classifier is basically a coin toss for true positives.  AUC and accuracy are high because of all the negatives.  So, how can you improve your score?&#xA;Try this:&#xA;&lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-04-20T22:29:38.930" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18498" PostTypeId="2" ParentId="18478" CreationDate="2017-04-21T00:35:14.663" Score="0" Body="&lt;p&gt;Machine Learning at its core is unsupervised learning. Its about making sense of unknown data. Unsupervised learning cropped up because, essentially many real world problems always have some priors associated with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At its core, data mining is about determining making sense of data. Simple data (2D) can be plotted, but what about 10,000 dimensions (sure there are means there these days but they themselves have their cons)? How would you go about it? You dont have any labels, someone just gave you a million point dataset and asked you to give them a report, what do you do then?  Would you pick up each sample and manually look at the values or would you like to use a tried and tested solution of partitioning it somehow and then looking at the classes? In a way you could say that unsupervised learning, although being the core, has these days become a pre-processor for its supervised counterpart.&lt;/p&gt;&#xA;" OwnerUserId="31346" LastActivityDate="2017-04-21T00:35:14.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18501" PostTypeId="2" ParentId="17382" CreationDate="2017-04-21T06:29:40.660" Score="1" Body="&lt;p&gt;I think brat that fit some tasks as NER, PosTag...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with classification problems, we can add keyword to text which separate with tab symbol. After, we can annotate for label.&#xA;for example: label sentence &#xA;we can annotate for word: label, with some classes&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="31409" LastActivityDate="2017-04-21T06:29:40.660" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18502" PostTypeId="2" ParentId="18477" CreationDate="2017-04-21T07:59:39.827" Score="0" Body="&lt;p&gt;Question solved.&lt;/p&gt;&#xA;&lt;p&gt;The constant in the last formula is just representing a constant which is not equivalent to the constant of &lt;span class=&quot;math-container&quot;&gt;$y_1a_1 + y_2a_2 = constant$&lt;/span&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31364" LastEditorUserId="29169" LastEditDate="2021-02-06T18:35:24.563" LastActivityDate="2021-02-06T18:35:24.563" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18504" PostTypeId="2" ParentId="18487" CreationDate="2017-04-21T09:11:07.117" Score="1" Body="&lt;p&gt;As you are building policies in simulation, and can avoid the need to use approximate methods (the state space is small enough to fit in a table in memory), then your goal is to converge on the optimal policy. With correct setup, &lt;em&gt;all&lt;/em&gt; the methods you are comparing are guaranteed to converge for your problem, provided the assumptions of reinforcement learning are not broken.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is just the reward scheme for maze solvers. For those the goal is to solve the maze quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a reward scheme. The best policy maximises the expected reward. That is the &lt;em&gt;only&lt;/em&gt; measurement that a completed RL output (your policy) cares about. As all your chosen learners should learn the optimal policy, you cannot compare them on this matter. So you maybe need to look at efficiency - how quickly the algorithm converges to optimal policy.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would characterise this as a small state space. It will fit into memory as a table. Looping through the 40 million state/action pairs should be pretty fast unless the simulation step is expensive for some reason.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, if you have a simple model for transitions and rewards at each step, I would encourage you to use &lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node19.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Value Iteration&lt;/a&gt; (a dynamic programming method) instead of the sampling based approaches in SARSA or Q-Learning. To do so, you would need to be able to define code for state transitions and rewards given current state and action. Sometimes though, the simulation is much easier to code than the transitions/rewards model. If that's the case for you, stick with SARSA or Q-Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As there are no consequences to you for bad decisions and low rewards during training stages - learning offline in simulations - then Q-Learning may be preferable as it learns the optimal policy whilst exploring. Compared to  SARSA you have to be concerned about how to reduce $\epsilon$ so as to converge on the optimal policy. For Q-Learning you can leave $epsilon$ at a relatively high value (e.g. 0.1) and still learn whilst refining estimates of alternative actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am a little concerned about this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the end-state reward depends on the actions selected in the other 249 days&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a problem, since it breaks the assumptions of the Markov Decision Process that all the algorithms are based upon. The reward at any step should &lt;em&gt;only&lt;/em&gt; be based on current state/action. By doing this, you may invalidate the policy. It definitely would defeat dynamic programming (may prevent it converging) unless you included action history in the state (which would definitely make your state space large). &lt;em&gt;However&lt;/em&gt;, it may still work for sampling methods with a high value of $\lambda$, because these are more robust when faced with environments which are not strictly MDPs.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Any suggestions on how to compare the models?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I suggest on CPU time required to learn the optimal model, assuming you intend to repeat this process on different data routinely. The way to find out is to experiment, as convergence will depend on properties of the problem, how fast your simulation code runs, and in your case you are stretching one of the assumptions behind the design of the RL solvers which could make strongly bootstrapping methods (like SARSA(0)) either fail to converge or converge very slowly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If convergence takes too long, set a cutoff at some time limit (e.g. an hour) and find out expected reward from initial conditions when following the policy learned so far. You could Monte Carlo sample that for a fair comparison (and if results are stochastic even with a deterministic policy, then give plenty of time for 1000s of runs to get a good estimate).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-04-21T11:03:43.990" LastActivityDate="2017-04-21T11:03:43.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18506" PostTypeId="1" AcceptedAnswerId="18508" CreationDate="2017-04-21T11:44:24.863" Score="0" ViewCount="1021" Body="&lt;p&gt;I have a data set like this. Here the first column is date, the second column is Temperature, third one is humidity, fourth and fifth column are two other boolean data. I have data of 6 years like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2010-01-01,25.6,59,0,1&#xA;2010-01-02,25.6,60,0,1&#xA;2010-01-03,24.2,45,1,1&#xA;2010-01-04,26.3,20,0,1&#xA;2010-01-05,26.2,17,0,1&#xA;2010-01-06,24.3,65,0,0&#xA;2010-01-07,23.1,50,0,1&#xA;2010-01-08,26.3,25,1,0&#xA;2010-01-09,26.6,23,0,1&#xA;2010-01-10,24.3,60,0,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the label (The variable I want to predict) for this data set is: (boolean) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0,0,0,1,1,0,0,0,1,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to implement Machine Learning to predict from this data set for a future time frame. I have almost no knowledge on machine learning. I want to use python to do it. Which library or methodology will be best and easiest for this? And can I have a simple sample code?&lt;/p&gt;&#xA;" OwnerUserId="31418" LastEditorUserId="14588" LastEditDate="2017-07-21T15:08:19.820" LastActivityDate="2017-07-21T15:08:19.820" Title="Simple Time Series Prediction" Tags="|machine-learning|python|time-series|forecasting|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18508" PostTypeId="2" ParentId="18506" CreationDate="2017-04-21T12:40:25.813" Score="1" Body="&lt;p&gt;Your problem looks to me more like a classification problem than a time series problem. My suggestion: Split the date into several sub-variables (year, month, day, week-of-day (maybe). Then just use this and the other values as input for a classification algorithm. Ideally you try several. I can recommend sklearn for this (&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One advice: Depending on the classification algorithm you need to first normalize your data. I find the following blog useful for entry-level problems (with code examples--  if you want to approach this from a time-series perspective he also talks about this) and documentation &lt;a href=&quot;http://machinelearningmastery.com/blog/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/blog/&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-04-21T12:40:25.813" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18511" PostTypeId="2" ParentId="15707" CreationDate="2017-04-21T21:02:10.700" Score="1" Body="&lt;p&gt;I would start with logistic regression, and get a measure of importance for each of your 20 predictor variables. It's a little tough to understand what you mean by 'variance' in your two classes of good and bad. If you mean variance in your predictor variables that lead to your response, you can calculate the variance of a linear combination of predictor variables (in your logistic regression model) using an appropriate theorem &lt;a href=&quot;https://en.wikipedia.org/wiki/Variance#Matrix_notation_for_the_variance_of_a_linear_combination&quot; rel=&quot;nofollow noreferrer&quot;&gt;from Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-04-21T21:02:10.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18513" PostTypeId="2" ParentId="17501" CreationDate="2017-04-21T21:13:46.893" Score="3" Body="&lt;p&gt;The accepted answer is correct, but it may also be helpful to think of a batch from a classification standpoint. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a binary classification problem that you are trying to solve using a multilayer perceptron, with 1000 examples of each class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When training the model, you don't want to have to wait until the model has seen all of the data before a weight update is performed. That's computationally inefficient. Instead, you take, for example, 100 random examples of each class and call it a 'batch'. You train the model on that batch, perform a weight update, and move to the next batch, until you have seen all of the examples in the training set. One pass through the training set in this manner is called an 'epoch'.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-04-21T21:13:46.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18514" PostTypeId="2" ParentId="18506" CreationDate="2017-04-21T21:24:41.693" Score="0" Body="&lt;p&gt;Welcome to the wonderful and sometimes intimidating world of machine learning! Building on El Burro's reply, I would start with a recurrent neural network, and put a logistic layer at the top. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent neural networks are times series models in machine learning. They take input (the value of a variable at each point in time), perform some transformations, and give an output (a class label, in your case, but they can also be used for sequence-to-sequence prediction). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One specifically relevant blog post for you might be &lt;a href=&quot;http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sequence classification&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-04-21T21:24:41.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18515" PostTypeId="1" AcceptedAnswerId="18517" CreationDate="2017-04-22T00:34:20.943" Score="1" ViewCount="60" Body="&lt;p&gt;I'm new to machine learning, and was working on creating an ANN which would classify each observation to a certain value. I have worked primarily with the sigmoid function up to this point to get the probability of an observation (true/false or binary output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this instance, I want each observation to be classified to one of 5 values: 0, 1, 2, 3, 4. I am using the sklearn library's StandardScalar function to scale the input data. What would be recommended in terms of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Best way to scale the output data, and&lt;/li&gt;&#xA;&lt;li&gt;Appropriate activation function to use for the output layer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="31437" LastActivityDate="2017-04-22T01:58:51.627" Title="Creating an Artifical Neural Network that produces a set of possible outputs" Tags="|machine-learning|neural-network|classification|tensorflow|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18516" PostTypeId="1" CreationDate="2017-04-22T00:37:19.153" Score="4" ViewCount="5437" Body="&lt;p&gt;When using machine learning models like gradient boosted trees and CNN, is it required (or considered as an always-do good practice) to balance the amount of positive/negative examples when learning for binary classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given &lt;code&gt;P&lt;/code&gt; positive examples and &lt;code&gt;N&lt;/code&gt; negative examples, where &lt;code&gt;P &amp;lt;&amp;lt; N&lt;/code&gt;, I can think of several choices: (Let's forget about validation set and test set)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choice A) No balancing at all, put all examples (totally &lt;code&gt;P+N&lt;/code&gt;) into the training set without weighting w.r.t. their ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choice B) Put all examples (totally &lt;code&gt;P+N&lt;/code&gt;) into the training set, but weight all positive examples &lt;code&gt;1/2P&lt;/code&gt; and all negative examples &lt;code&gt;1/2N&lt;/code&gt;, so that total weight of positive examples and negative example equal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choice C) Take all &lt;code&gt;P&lt;/code&gt; positive examples, then sample &lt;code&gt;P&lt;/code&gt; negative examples (out of &lt;code&gt;N&lt;/code&gt;), and train with these &lt;code&gt;2P&lt;/code&gt; examples with uniform weighting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the pros/cons for each of the approach and which one(s) do we usually go with?&lt;/p&gt;&#xA;" OwnerUserId="29428" LastActivityDate="2017-09-19T14:14:16.910" Title="Necessity of balancing positive/negative examples in binary classification machine learning?" Tags="|machine-learning|class-imbalance|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18517" PostTypeId="2" ParentId="18515" CreationDate="2017-04-22T01:58:51.627" Score="1" Body="&lt;p&gt;You don't need to scale the output data. For classification with a ANN the best activation is the softmax function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f(x) = e^x_i / sum e^x_j&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which normalizes an input vector by applying the exponential function element wise, then dividing by the sum. This produces a discrete probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then your output layer should have 5 output neurons, apply softmax, and you will get a discrete probability distribution over the set [0, 1, 2, 3, 4]. Then you don't need to normalize the data, just do a one-hot encoding. Note that this might differ depending on the framework used.&lt;/p&gt;&#xA;" OwnerUserId="17" LastActivityDate="2017-04-22T01:58:51.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18518" PostTypeId="1" AcceptedAnswerId="18520" CreationDate="2017-04-22T03:30:17.037" Score="0" ViewCount="2437" Body="&lt;p&gt;I was wondering does it make sense to use random forest to select most important variables then put into logistic regression for prediction? I think that it might not make sense because what's important for random forest might not be important for logistic regression?&lt;/p&gt;&#xA;" OwnerUserId="30376" LastEditorUserId="85045" LastEditDate="2021-02-09T17:01:17.703" LastActivityDate="2021-02-09T17:01:17.703" Title="Using random forest to select important variables &amp; then putting into logistic regression?" Tags="|feature-selection|random-forest|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18519" PostTypeId="2" ParentId="18488" CreationDate="2017-04-22T05:13:48.060" Score="-4" Body="&lt;p&gt;So I'll post an answer to my own question. For anyone who comes across this post during the feature selection / More features or less process, I dont know what you can do (well except if you're on python then mork's answer has a good way to do feature selection there) but I can tell you what NOT to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not under any circumstances ever &quot;try&quot; determining best features by training+testing the SVM / statistical model. That is oh this feature works because of more classification accuracy than the other one. NO. Not unless that is the &lt;em&gt;only&lt;/em&gt; way left, dont do it. That is a way, you are free to do it, but if you can try something else please do. Dont listen to anyone who tells you to do that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many features your problem requires depends on how many optimal features you can &lt;em&gt;find&lt;/em&gt;. I'll leave it at that. How to find them? That is the million dollar question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People are getting confused. When you dont know about the accuracy of your features, it is bad practice to &quot;train&quot; data to see how many features your SVM needs. For that it is better you select features on the basis of some criteria set by &lt;em&gt;your&lt;/em&gt; problem. If you want after that, &lt;em&gt;then&lt;/em&gt; you &lt;em&gt;may&lt;/em&gt; try feature selection techniques. But remember, reducing too many dimensions may also decrease accuracy sometimes.&lt;/p&gt;&#xA;" OwnerUserId="31346" LastEditorUserId="31346" LastEditDate="2017-04-24T20:17:10.507" LastActivityDate="2017-04-24T20:17:10.507" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18520" PostTypeId="2" ParentId="18518" CreationDate="2017-04-22T07:26:45.927" Score="1" Body="&lt;p&gt;There are many factors which are underlying the 'importaces' of featues obtained from random forest.&lt;/p&gt;&#xA;&lt;p&gt;For instance, features with more number of categories (unique values if its a numerical feature) would be more likely to get find splits; making it more important feature.&#xA;Having all features with same number of categories is not a common scenario.&lt;/p&gt;&#xA;&lt;p&gt;So, even though the features identified are likey to be the best predictors (in logistic regressions etc.), one need to take the inherent biases in the randome forest algorithm into consideration while utilizing the 'important' features. Therefore whether it would make sense to use the important features from random forest model into logistic regression would depend on if any and how much the importances are biased.&lt;/p&gt;&#xA;&lt;p&gt;Source for the information and further useful details on the biases in estimation of feature importances by random forest algorithm: Bias in random forest variable importance measures: Illustrations, sources and a solution &lt;a href=&quot;https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-25&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bias in random forest variable importance measures: Illustrations, sources and a solution&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31446" LastEditorUserId="85045" LastEditDate="2021-02-09T17:00:58.713" LastActivityDate="2021-02-09T17:00:58.713" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18521" PostTypeId="2" ParentId="18518" CreationDate="2017-04-22T08:33:12.827" Score="1" Body="&lt;p&gt;There is not any single good answer to this question as this thing depends on many factors and one of them is the &lt;code&gt;data&lt;/code&gt;. As per my experience, I prefer not to remove features as even little information can be very useful. &lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-04-22T08:33:12.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18522" PostTypeId="1" CreationDate="2017-04-22T11:18:10.533" Score="-1" ViewCount="1743" Body="&lt;p&gt;I am following &lt;a href=&quot;https://rpubs.com/brianzive/textmining&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt; to get a list of most recurrent n-word phrases from a dataframe that has approximately 2m records (text in multiple languages).&#xA;Everything works fine till the last couple of steps &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# tokenize into tri-grams&#xA;trigram.twitterTdm &amp;lt;- tm::TermDocumentMatrix(text.corpus[&quot;twitter&quot;], control = list(tokenize = TrigramTokenizer))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where it ends up using TrigramTokenizer (from the RWeka package) to actually create the tokens. Issue is that RWeka has an rJava dependency which is not supported on our server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there any other packages in R which I can use to get most recurrent 2/3/4 word phrases from a dataset? Any tutorials would be extremely useful (novice in this area)&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="31452" LastActivityDate="2017-04-22T13:19:31.800" Title="Text mining in R without RWeka" Tags="|r|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18523" PostTypeId="2" ParentId="18516" CreationDate="2017-04-22T12:17:01.427" Score="2" Body="&lt;p&gt;Let's start by answering your first question. &lt;strong&gt;Is it required to balance the dataset?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Absolutely, the reason is simple in failing to do so you end up with algorithmic bias. This means that if you train your classifier without balancing the classifier has a high chance of favoring one of the classes with the most examples. This is especially the case with boosted trees. Even normal decision trees, in general, have the same effect. So it is always important to balance the dataset &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's discuss the three different scenarios placed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choice A)&lt;/strong&gt;: This would be what I explained all along. I'm not saying necessarily you will have a bias. It depends on the dataset itself. If the nature of the dataset has a very fine distinction with the boundaries then the chance of misclassification is reduced, you might get a decent result but it's still not recommended. Also if the data does not have good boundaries then the rate of misclassification rises a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choice B)&lt;/strong&gt;: Since you are placing weights for each sample you are trying to overcome the bias with a penalty. This is also called as an Asymmetric method. Normally these methods increase the accuracy of a model by a slight margin but that mostly depends on the machine learning algorithm you are using. In examples like Adaboost such a model the effectivity of the model increases. This method is also called Asymmetric Adaboost. But this might not necessarily work with all algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choice C)&lt;/strong&gt;: Assuming you have weighted the samples accordingly it should do the same as either choice A or choice B. I'll leave this for you to extrapolate based on my previous explanations.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-22T12:17:01.427" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18525" PostTypeId="2" ParentId="18522" CreationDate="2017-04-22T13:19:31.800" Score="2" Body="&lt;p&gt;Since you are using the tm library NLP should be installed as well and the code below should work. Adjust the tokenizer to your needs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tm)&#xA;data(&quot;crude&quot;)&#xA;crude &amp;lt;- as.VCorpus(crude)&#xA;crude &amp;lt;- tm_map(crude, stripWhitespace)&#xA;crude &amp;lt;- tm_map(crude, removePunctuation)&#xA;crude &amp;lt;- tm_map(crude, content_transformer(tolower))&#xA;crude &amp;lt;- tm_map(crude, removeWords, stopwords(&quot;english&quot;))&#xA;crude &amp;lt;- tm_map(crude, stemDocument)&#xA;&#xA;&#xA;&#xA;# change 3 into whatever you need for a differenct n-gram&#xA;NLPtrigramTokenizer &amp;lt;- function(x) {&#xA;      unlist(lapply(ngrams(words(x), 3), paste, collapse = &quot; &quot;), use.names = FALSE)&#xA;}&#xA;&#xA;tdm_NLP &amp;lt;- TermDocumentMatrix(crude, control=list(tokenize = NLPtrigramTokenizer))&#xA;&#xA;inspect(tdm_NLP)&#xA;&amp;lt;&amp;lt;TermDocumentMatrix (terms: 2252, documents: 20)&amp;gt;&amp;gt;&#xA;Non-/sparse entries: 2485/42555&#xA;Sparsity           : 94%&#xA;Maximal term length: 32&#xA;Weighting          : term frequency (tf)&#xA;Sample             :&#xA;                       Docs&#xA;Terms                   144 236 237 242 246 248 273 489 502 704&#xA;  158 mln bpd             3   1   0   0   0   0   0   0   0   0&#xA;  ali alkhalifa alsabah   0   1   0   0   0   1   0   0   0   0&#xA;  arabian oil minist      0   0   0   0   0   1   1   0   0   0&#xA;  barrel per day          0   0   0   0   1   1   1   0   0   0&#xA;  decemb opec accord      0   0   0   0   0   1   1   0   0   0&#xA;  emerg opec meet         0   1   0   0   0   1   0   0   0   0&#xA;  hold futur posit        0   0   0   0   0   0   0   0   0   3&#xA;  industri sourc said     1   1   0   1   0   0   0   0   0   0&#xA;  sheikh abdulaziz said   0   0   0   0   4   0   0   0   0   0&#xA;  world oil price         0   1   1   0   1   1   0   0   0   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also you might be interested in reading &lt;a href=&quot;http://tidytextmining.com/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tidy text mining with R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10517" LastActivityDate="2017-04-22T13:19:31.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18528" PostTypeId="1" CreationDate="2017-04-22T14:37:33.897" Score="1" ViewCount="1057" Body="&lt;p&gt;If I have to paraphrase the current NER methodologies, it generally finds patterns in strings and creates its own &quot;vocabulary&quot;, so to speak.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naturally, it would perform like a charm with a mammoth dataset, curated carefully with hardwork and labeled entities.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if, the system is first introduced with a dictionary of named entities for respective categories and then given a sample literature or simple tweets for that category, for eg, and from them it &quot;learns&quot; how those &lt;strong&gt;named entites appear in a context.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference is subtle from regex based, in that in regex it would try to match strings and more the dictionary size, more the rules, then more its usefulness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in this system, it would actually learn how &quot;eating an apple&quot; and &quot;eating at Apple&quot; both can be accurately classified with little training set, the subtleness of grammar when mentioning about fruit and about a company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some intuition on its implementation in CRF++, CRFsuite, Stanford or any other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;General disclaimer&lt;/em&gt;&lt;br&gt;&#xA;&lt;em&gt;Not a data scientist. Just a passing thought from an ML enthusiast.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="31456" LastActivityDate="2018-03-25T16:15:42.950" Title="Dictionary based statistical NER learner" Tags="|machine-learning|named-entity-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18529" PostTypeId="2" ParentId="12964" CreationDate="2017-04-22T15:04:14.760" Score="3" Body="&lt;p&gt;In Keras, which sits on top of either TensorFlow or Theano, when you call &lt;code&gt;model.add(LSTM(num_units))&lt;/code&gt;, num_units is the dimensionality of the output space (from &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py#L602&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, line 863). To me, that means &lt;code&gt;num_units&lt;/code&gt; is the number of hidden units whose activations get sent forward to the next time step.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-04-22T15:04:14.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18531" PostTypeId="2" ParentId="18518" CreationDate="2017-04-22T17:23:28.180" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://m2.icm.edu.pl/boruta/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt; algorithm uses random forests to select relevant features. There are &lt;a href=&quot;https://github.com/scikit-learn-contrib/boruta_py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Github&lt;/a&gt; and &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;R package&lt;/a&gt; implementations.&lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2017-04-22T17:23:28.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18535" PostTypeId="1" AcceptedAnswerId="18536" CreationDate="2017-04-23T05:49:02.917" Score="0" ViewCount="61" Body="&lt;p&gt;I've been playing around with bagged trees and random forests. How can I tell what factors most influenced the categorization? Will scikitlearn just spit it out, or is it trickier than that?&lt;/p&gt;&#xA;" OwnerUserId="31474" LastActivityDate="2017-04-23T06:26:51.483" Title="How can I identify the most predictive factors?" Tags="|predictive-modeling|random-forest|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18536" PostTypeId="2" ParentId="18535" CreationDate="2017-04-23T06:26:51.483" Score="0" Body="&lt;p&gt;It basically lies within the fitted object. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(X, y)&#xA;importances = model.feature_importances_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The importances are in the same order as the columns of features ($X$).&lt;/p&gt;&#xA;" OwnerUserId="31446" LastActivityDate="2017-04-23T06:26:51.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18538" PostTypeId="2" ParentId="16448" CreationDate="2017-04-23T10:51:25.077" Score="0" Body="&lt;p&gt;The problem you're trying to solve is called binary classification. There are many algorithms and there's ton of tutorials on it. Actually it is so well studied that ready-made tools are available that try all algorithms and choose the best for your task. &lt;a href=&quot;https://github.com/paypal/autosklearn-zeroconf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/paypal/autosklearn-zeroconf&lt;/a&gt; is one of them that makes auto-sklearn binary classifier able to run on arbitrary tabular data (like what you have). &lt;/p&gt;&#xA;" OwnerUserId="15361" LastActivityDate="2017-04-23T10:51:25.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18539" PostTypeId="1" CreationDate="2017-04-23T11:47:19.023" Score="2" ViewCount="389" Body="&lt;p&gt;I have a sequence of event and would like to predict the next one. The training data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ev1,Ev2,Ev5,Ev6,Ev7&lt;/li&gt;&#xA;&lt;li&gt;Ev1,Ev6,Ev99&lt;/li&gt;&#xA;&lt;li&gt;Ev4,Ev3,Ev6&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, the idea is to get Ev7 given Ev1,Ev2,Ev5,Ev6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the number of &quot;final&quot; events is very high (100K). I tried to look into neural network but it means the last layer would need to have size of 100K.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody point to any other ways how to do that or any examples?&lt;/p&gt;&#xA;" OwnerUserId="18501" LastActivityDate="2017-06-22T18:54:27.447" Title="Next event prediction in the sequence" Tags="|classification|sequence|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18542" PostTypeId="2" ParentId="17885" CreationDate="2017-04-23T15:59:43.963" Score="0" Body="&lt;p&gt;It may be the version of the 20 newsgroups data used. There are two versions of the dataset available from &lt;a href=&quot;http://qwone.com/~jason/20Newsgroups/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://qwone.com/~jason/20Newsgroups/&lt;/a&gt;: &#xA;(1) the original &#xA;(2) another with duplicates removed and headers removed that give away the actual group. This dataset is also already separated into train and test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I load the original data set (with duplicates and headers) and use the code you reference I can get to ~93-95% after three epochs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I load the second version of the dataset (training only), it takes many more epochs to converge, with a validation accuracy of only about 55% after 10 epochs. Validation accuracy was still on an upward trajectory, so you should be able to improve on these results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, I think that that the duplicates/filtering are significantly skewing the results from the blog you mentioned. The classification is easier when the header contains which news group in which it should be classified. The lesser accuracy is a more realistic evaluation of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that if you load the dataset using scikit-learn (sklearn.datasets.fetch_20newsgroups), it loads the second version of the dataset. A tf-idf model with multinomial Naive Bayes gets to about 50% on the data, so the RNN is still out performing a baseline model. Though, 95% seems not realistic on the second version of 20 newsgroups. &lt;/p&gt;&#xA;" OwnerUserId="31488" LastActivityDate="2017-04-23T15:59:43.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18543" PostTypeId="2" ParentId="18539" CreationDate="2017-04-23T16:15:14.910" Score="1" Body="&lt;p&gt;Hidden Markov Model (HMM) is a method that you could try out. They have been used to model very large sequences (like gene sequences). &lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2017-04-23T16:15:14.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18546" PostTypeId="1" CreationDate="2017-04-23T21:50:13.603" Score="2" ViewCount="60" Body="&lt;p&gt;Consider the following problem: I want to predict the next bat of a set of baseball player. I have a training data set, where it contains the historical bat records (0-1 encoded, which is our target variable) of the set of baseball player, player's id, their name, salary, etc, as predictors. and I have an independent test data set. I am using the log loss function as the assessment of the model. I am not sure whether the problem I describe below is specific to log loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that one can just use &lt;strong&gt;All the training data set with just the player id and their bat records&lt;/strong&gt; and use empirical Bayesian method to compute the bat rate of each player, e.g., form a prior such as binomial and use the observed bat records to compute the posterior probability of each player. Then use this probability to make prediction on the test data set. This method is reasonable especially if you don't have other predictors like player's age, salary, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, now with other predictors, one can run some sophisticated machine learning models. Here for feature engineering, I have the following question. &lt;strong&gt;I could also compute the empirical Bayesian probability I just described for each player and incorporate this probability as one of the predictors.&lt;/strong&gt; I've been told that this will lead to &lt;strong&gt;information leakage&lt;/strong&gt;, since this predictor is a function of the target variable that I want to predict. Hence, this will lead to overfitting and make the generalizability of the model poor. &lt;strong&gt;I vaguely understand that but still hope someone could explain this idea clearly&lt;/strong&gt;. My puzzle is, &lt;em&gt;if just using empirical Bayesian probability alone (not using any machine learning model) to make predictions is a reasonable idea, then why put this variable in a machine learning model would cause the information leakage problem?&lt;/em&gt; In fact, I did just put this empirical Bayesian probability into my set of predictors and the model performs much worse on the test data set than I kick this predictor out of my test predictors. I checked the training error rate, which is incredibly well, which suggests strong overfitting issue. I want to understand why the overfitting occurs here, what is the underlying mechanism that drives this overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, I found a method online to deal with this problem which does not explain why but it works pretty well. The idea is the following: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;randomly divide the training data set into 5 folds (like cross validation) and label them as folder 1,2,3,4,5, then for each iteration (totally 5 iterations), we do the following: &lt;strong&gt;for iteration 1, we use the fold 2,3,4,5 to compute the empirical Bayesian probability of bat for each player, then map the results to the players in folder 1&lt;/strong&gt;. For example, player Kevin, his bat probability in &lt;strong&gt;All training data set&lt;/strong&gt; is 0.23, but using the fold 2,3,4,5, his bat probability is 0.34. Then this 0.34 is assigned to fold 1 whenever Kevin appears (assume we only have 1 Kevin). If some player only appears in fold 1 but not in the rest folds, then just assign 0.5 to this guy. In iteration 2, we use the fold 1,3,4,5 to compute the empirical Bayesian probability of bat for each player, then map the results to the players in folder 2. We do this for all iterations. Eventually, after this procedure, we will generate a new column with the bat probability for all players in the training data set using this &quot;strange procedure&quot;. &lt;strong&gt;I checked it with my model, it improves the performance on the test data set a lot&lt;/strong&gt;. And I also run this procedure for 3 fold and 10 fold, they are all pretty good comparing to (1) not include the empirical Bayesian probability as a predictor or (2) compute the empirical Bayesian probability using ALL training data set and put it into the set of predictors. But 3 and 10 folds are not as good as 5 fold. Here my question is &lt;strong&gt;why this &quot;strange procedure&quot; of computing the empirical Bayesian probability works? It seems that it overcomes the overfitting issue on the training data set by only using part of the training data set to compute the probability&lt;/strong&gt;. Note that if you just want to estimate the bat probability of each player, then using part of the data is &lt;strong&gt;definitely worse&lt;/strong&gt; than using the whole data, isn't it? So according to my experiment, it seems that a &quot;worse&quot; (or less precise) estimation of the bat probability when putting as a predictor to predict the bat probability is better than using a more precise bat probability. This seems very counter intuitive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third, note that many machine learning models have hyper parameters that needs to be set before running the model, e.g., the lambda in logistic regression. Now let's say I want to tune the lambda in logistic regression using cross validation. I also want to include the empirical Bayesian probability as a predictors. How should I conduct the cross validation procedure? I seems that we cannot just randomly divide the data into K fold and do it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, let's say we just set a small proportion of the training data as the validation data set: then I believe the correct procedure is: &lt;strong&gt;split the training data set first, then construct the empirical Bayesian probability of bat (this procedure should not be reversed)&lt;/strong&gt;. However, my question is: after I first split the data, then in the non-validation data set, how should I construct that empirical Bayesian probability bat column? apparently according to the second point of this post, we cannot just go head and compute it. We should also run that &quot;strange procedure&quot; as described in the second point on the non-validation data set? I guess I am still questioning why the &quot;strange procedure&quot; in the second point works&lt;/p&gt;&#xA;" OwnerUserId="30170" LastEditorUserId="30170" LastEditDate="2017-04-23T21:57:40.277" LastActivityDate="2017-04-23T21:57:40.277" Title="information leakage when using empirical Bayesian to generate a predictor" Tags="|classification|overfitting|bayesian|data-leakage|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18550" PostTypeId="1" CreationDate="2017-04-24T05:07:40.813" Score="1" ViewCount="368" Body="&lt;p&gt;I hope this is the right forum bcoz I couldn't find one which seems exactly relevant. Question is, are any of the following topics from Statistics really useful from point of view of being applicable while working with Machine Learning, AI and Neural Networks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A)Descriptive Statistics&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Analysis of Quantitative Data:&#xA;- Measures of central tendancy&#xA;- Measures of dispersion&#xA;- Moments&#xA;- Skewness and Kurtosis&#xA;Correlation of bivariate data:&#xA;- Fitting of curves&#xA;- Correlation coefficient&#xA;- Rank Correlation&#xA;- Intra-class Corelation&#xA;Regression and Multiple Correlations&#xA;- Linear Regression&#xA;- Plane of Regression&#xA;- Multiple Correlation&#xA;- Partial Correlation&#xA;Theory of Attributes&#xA;- Classification of Attributes&#xA;- Independence of Attributes&#xA;- Association of Attributes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;B)Probability Theory&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Basic Concepts in Probability&#xA;- Introduction to Probability&#xA;- Different Approaches to Probability Theory&#xA;- Laws of Probability&#xA;- Bayes' Theorem&#xA;Random Variables and Expectation&#xA;- Random Variables&#xA;- Bivariate Discrete Random Variables&#xA;- Bivariate Continuous Random Variables&#xA;- Mathematical Expectation&#xA;Discrete Probability Distributions&#xA;- Binomial Distributions&#xA;- Poisson Distributions&#xA;- Discrete Uniform and Hypergeometric Distributions&#xA;- Geometric and Negative Binomial Distributions&#xA;Continuous Probability Distributions&#xA;- Normal Distributions&#xA;- Area Property of Normal Distributions&#xA;- Continuous Uniform and Exponential Distributions&#xA;- Gamma and Beta Distributions&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;C)Statistical Inference&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Sampling Distributions&#xA; - Introduction to Sampling Distributions&#xA; - Sampling Distributions of Statistics&#xA; - Standard Sampling Distributions&#xA; Estimation&#xA; - Introduction to Estimation&#xA; - Point Estimation&#xA; - Interval Estimation for One Population&#xA; - Interval Estimation for Two Populations&#xA; Testing of Hypothesis&#xA; - Concepts of Testing of Hypothesis&#xA; - Large Sample Tests&#xA; - Small Sample Tests&#xA; - Chi-Square and F-Tests&#xA; Non-Parametric Tests&#xA; - One-Sample Tests&#xA; - Two-Sample Tests&#xA; - k-Sample Tests&#xA; - Analysis of Frequencies&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;D)Statistical Techniques&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sampling Designs&#xA;- Introduction to Sample Surveys&#xA;- Simple Random Sampling&#xA;- Stratified Random Sampling&#xA;- Other Sampling Schemes&#xA;Analysis of Variance&#xA;- Introduction&#xA;- One-way Analysis of Variance&#xA;- Two-way Analysis of Variance&#xA;- Two-way Analysis of Variance with m Observations per call&#xA;Design of Experiments&#xA;- Completely Randomized Design&#xA;- Randomized block Design&#xA;- Latin Square Design&#xA;- Factorial Experiments&#xA;Random Numbers Generation and Simulation Techniques&#xA;- Random Numbers Generation for Discrete Variables&#xA;- Random Numbers Generation for Continuous Variables&#xA;- Simulation Techniques&#xA;- Applications of Simulation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;E)Industrial Statistics - I&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Process Control&#xA;- Introduction to Statistical Quality Control&#xA;- Control Charts for Variables&#xA;- Control Charts for Attributes&#xA;- Control Charts for Defects&#xA;Product Control&#xA;- Acceptance Sampling Plans&#xA;- Rectifying Sampling Plans&#xA;- Single Sampling Plans&#xA;- Double Sampling Plans&#xA;Decision and Game Theory&#xA;- Introduction to Decision Theory&#xA;- Decision making Process&#xA;- Two-Person Zero-Sum Games with Saddle Point&#xA;- Two-Person Zero-Sum Games without Saddle Point&#xA;Reliability Theory&#xA;- Introduction to Reliability&#xA;- Reliability Evaluation of Simple System&#xA;- Reliability Evaluation of k-out-of-n and StandBy System&#xA;- Reliability Evaluation of Complex System&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;F)Industrial Statistics - II&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Optimisation Techniques I &#xA;- Introduction to Operations Research&#xA;- Linear Programming Problems&#xA;- Simplex Method&#xA;- Transportation Problems&#xA;Optimisation Techniques II&#xA;- Assignment Problems&#xA;- Queueing Theory&#xA;- Sequencing Problems&#xA;- Inventory Models&#xA;Regression Modelling&#xA;- Simple Linear Regression&#xA;- Statistical Inference in Simple Linear Regression&#xA;- Multiple Linear Regression&#xA;- Selection of Variables and Testing Model Assumptions&#xA;Time Series Modelling&#xA;- Trend Component Analysis&#xA;- Seasonal Component Analysis&#xA;- Stationary Processes&#xA;- Time Series Models&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: In reply to @SmallChess; honestly, both these areas are new to me but I find them really interesting. Ultimately I wish to work in Physics and be able to apply it there. Example, following links: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://physics.stackexchange.com/questions/239166/is-machine-learning-a-useful-tool-for-analyzing-experimental-data&quot;&gt;link1&lt;/a&gt;  &lt;a href=&quot;https://www.quora.com/How-is-machine-learning-used-in-physics&quot; rel=&quot;nofollow noreferrer&quot;&gt;link2&lt;/a&gt;  &lt;a href=&quot;https://www.quora.com/What-is-the-relationship-between-physics-and-machine-learning-AI&quot; rel=&quot;nofollow noreferrer&quot;&gt;link3&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps this is still quite broad. But I haven't really put my finger on anything specific as yet. Best way I can narrow it down is probably through above links and following excerpts from them:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Machine learning is used in physics just as it is used in other field of science or in industry, i.e. when dealing with complex problems and/or lots of data, having computers sharing the hard thinking part&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Nearly all of what we would call physics consists of fairly simple models, and the urge to construct and understand simple models drives much of what we do. Major exceptions to this: observational astronomy and high energy experiments like the Large Hadron Collider at CERN where machine learning can be effectively applied. Another promising application is in turbulence modeling&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps to narrow it down a bit. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="30690" LastEditorUserId="30690" LastEditDate="2017-04-26T12:34:42.600" LastActivityDate="2017-04-26T12:34:42.600" Title="Applied statistics in Machine Learning, AI, Neural Networks" Tags="|machine-learning|neural-network|statistics|" AnswerCount="2" CommentCount="1" ClosedDate="2017-04-25T10:12:32.027" ContentLicense="CC BY-SA 3.0" />
  <row Id="18553" PostTypeId="2" ParentId="18550" CreationDate="2017-04-24T06:34:19.150" Score="0" Body="&lt;p&gt;I agree @Emre. To me, they are all important to machine learning. Of course, it depends &lt;strong&gt;what exactly&lt;/strong&gt; you want to do in machine learning. For instance, if you're just doing image recognition you don't need to understand time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example, &quot;Decision and Game Theory&quot; is absolutely critical for machine learning in board games like chess (e.g reinforcement self-learning).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything you list here applies to machine learning - machine learning is really just statistical algorithms for learning an unknown function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why don't you tell us what you want to do, and we tell you what you should learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd give you the top five I think most important in your list (subjective):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Normal distribution&lt;/li&gt;&#xA;&lt;li&gt;Linear Regression&lt;/li&gt;&#xA;&lt;li&gt;Random Variables&lt;/li&gt;&#xA;&lt;li&gt;Mathematical Expectation&lt;/li&gt;&#xA;&lt;li&gt;Binomial Distributions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="9123" LastEditDate="2017-04-24T06:50:15.403" LastActivityDate="2017-04-24T06:50:15.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18554" PostTypeId="1" AcceptedAnswerId="18638" CreationDate="2017-04-24T08:00:02.167" Score="1" ViewCount="206" Body="&lt;p&gt;I need to decide between SVM (One-Class Support Vector Machine) and PCA (PCA-Based Anomaly Detection) as anomaly detection methods. Azure ML is used and provides SVM and PCA as methods - hence the choice of 2 possible methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have suggestions or a defined process for method selection? (Similar to cheat sheets you get for selecting a regression method).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use case is to detect anomalies in high frequency network traffic data (from firewalls, routers &amp;amp; switches)?&lt;/p&gt;&#xA;" OwnerUserId="31245" LastEditorUserId="31245" LastEditDate="2017-04-25T13:11:12.787" LastActivityDate="2020-08-31T12:21:31.280" Title="Anomoly detection method selection" Tags="|svm|pca|anomaly-detection|azure-ml|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18556" PostTypeId="1" CreationDate="2017-04-24T09:38:04.310" Score="-1" ViewCount="184" Body="&lt;p&gt;When dealing with the Neural Network outputs, I found two different approaches to express the output to Neural Network:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Using one column with different value as different classifications:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1        // class A&#xA;10       // class B&#xA;10       // class B&#xA;1        // class A&#xA;1        // class A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as two different class&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Using 2 columns as different classfication&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1    0     // class A&#xA;1    0     // class A&#xA;0    1     // class B&#xA;1    0     // class A&#xA;0    1     // class B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Correct me if I am wrong, or please tell me the differences or which one is better for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;MultiLayerPerceptron&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Transfer Function: &lt;strong&gt;&lt;em&gt;TANH&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;using bias neurons&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ResilentPropagation&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using Batch Mode&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="26856" LastEditDate="2017-04-25T01:48:47.703" LastActivityDate="2017-05-25T06:28:06.277" Title="Using N columns as N classification or using 1 column with multiple values" Tags="|machine-learning|neural-network|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18557" PostTypeId="1" AcceptedAnswerId="18560" CreationDate="2017-04-24T10:04:17.183" Score="3" ViewCount="347" Body="&lt;p&gt;It is commonly seen as something bad if the decision boundary of a neural network is too sharp, meaning if slight changes in the input change the class prediction completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a trained CNN, is it possible to measure / calculate the &quot;sharpness&quot; of its decision boundaries? Did somebody do that already?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-04-24T19:25:09.813" Title="Is there a way to measure the &quot;sharpness&quot; of a decision boundary of a CNN?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18558" PostTypeId="1" CreationDate="2017-04-24T12:11:35.913" Score="-1" ViewCount="184" Body="&lt;p&gt;We are currently working to replace the synonyms of certain words in a Row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are facing the problem as shown below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     List&amp;lt;Row&amp;gt; data = Arrays.asList(&#xA;        RowFactory.create(0, &quot;Allen Armstrong nishanth hemanth Allen&quot;),&#xA;        RowFactory.create(1,&quot;shivu Armstrong nishanth&quot;),&#xA;        RowFactory.create(2,&quot;shree shivu DeWALT&quot;));&#xA;&#xA;      StructType schema = new StructType(new StructField[] {&#xA;      new StructField(&quot;label&quot;, DataTypes.IntegerType, false,&#xA;        Metadata.empty()),&#xA;      new StructField(&quot;sentence&quot;, DataTypes.StringType, false,&#xA;        Metadata.empty()) });&#xA;      Dataset&amp;lt;Row&amp;gt; sentenceDataFrame = spark.createDataFrame(data, schema);&#xA;&#xA;&#xA;      List&amp;lt;Row&amp;gt; data2 = Arrays.asList(&#xA;        RowFactory.create(&quot;Allen&quot;, &quot;Apex Tool Group&quot;),&#xA;        RowFactory.create(&quot;Armstrong&quot;,&quot;Apex Tool Group&quot;),&#xA;        RowFactory.create(&quot;DeWALT&quot;,&quot;StanleyBlack&quot;));&#xA;&#xA;      StructType schema2 = new StructType(new StructField[] {&#xA;      new StructField(&quot;label2&quot;, DataTypes.StringType, false,&#xA;        Metadata.empty()),&#xA;      new StructField(&quot;sentence2&quot;, DataTypes.StringType, false,&#xA;        Metadata.empty()) });&#xA;      Dataset&amp;lt;Row&amp;gt; sentenceDataFrame2 = spark.createDataFrame(data2, schema2);&#xA;&#xA;      UDF2&amp;lt;String, String, Boolean&amp;gt; contains = new UDF2&amp;lt;String, String, Boolean&amp;gt;() {&#xA;             private static final long serialVersionUID = -5239951370238629896L;&#xA;&#xA;             @Override&#xA;             public Boolean call(String t1, String t2) throws Exception {&#xA;                 return t1.contains(t2);&#xA;             }&#xA;         };&#xA;         spark.udf().register(&quot;contains&quot;, contains, DataTypes.BooleanType);&#xA;&#xA;         UDF3&amp;lt;String, String, String, String&amp;gt; replaceWithTerm = new UDF3&amp;lt;String, String, String, String&amp;gt;() {&#xA;             private static final long serialVersionUID = -2882956931420910207L;&#xA;&#xA;             @Override&#xA;             public String call(String t1, String t2, String t3) throws Exception {&#xA;                 return t1.replaceAll(t2, t3);&#xA;             }&#xA;         };&#xA;         spark.udf().register(&quot;replaceWithTerm&quot;, replaceWithTerm, DataTypes.StringType);&#xA;&#xA;&#xA;         Dataset&amp;lt;Row&amp;gt; joined = sentenceDataFrame.join(sentenceDataFrame2, callUDF(&quot;contains&quot;, sentenceDataFrame.col(&quot;sentence&quot;), sentenceDataFrame2.col(&quot;label2&quot;)))&#xA;           .withColumn(&quot;sentence_replaced&quot;, callUDF(&quot;replaceWithTerm&quot;, sentenceDataFrame.col(&quot;sentence&quot;), sentenceDataFrame2.col(&quot;label2&quot;), sentenceDataFrame2.col(&quot;sentence2&quot;)));&#xA;         joined.show(false);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Allen Armstrong nishanth hemanth Allen&lt;br&gt;&#xA;shivu Armstrong nishanth&lt;br&gt;&#xA;shree shivu DeWALT&lt;br&gt;&#xA;&lt;br&gt;&#xA;Replacement of words&lt;br&gt;&#xA;The word in LHS has to replace with the words in RHS given in the input sentence&lt;br&gt;&#xA;Allen =&gt; Apex Tool Group&lt;br&gt;&#xA;Armstrong =&gt; Apex Tool Group&lt;br&gt;&#xA;DeWALT =&gt; StanleyBlack&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       Output&#xA;&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;      |label|sentence_replaced                                                        |&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;      |0    |Apex Tool Group Armstrong nishanth hemanth Apex Tool Group               |&#xA;      |0    |Allen Apex Tool Group nishanth hemanth Allen                             |&#xA;      |1    |shivu Apex Tool Group nishanth                                           |&#xA;      |2    |shree shivu StanleyBlack                                                 |&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;&#xA;      Expected Output&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;      |label| sentence_replaced                                                      |&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;      |0    |Apex Tool Group Apex Tool Group nishanth hemanth Apex Tool Group         | &#xA;      |1    |shivu Apex Tool Group nishanth                                           |&#xA;      |2    |shree shivu StanleyBlack                                                |&#xA;      +-----+----------------------------------+---------+---------------+------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get such output when there is multiple replacements to do in a row.&lt;br&gt;&#xA;Only Allen has been replaced with Apex Tool Group &lt;br&gt;&#xA;But not in Armstrong with Apex Tool Group in first row.&lt;br&gt;&#xA;Label 0 should merge to get a single row as an output in the data.&lt;br&gt; So that no redundancy should exist within the same Dataset Row &lt;br&gt;&#xA;Is there any other method which i must follow to get the proper output.? &lt;br&gt;Or is this is limitation of UDF&lt;/p&gt;&#xA;" OwnerUserId="30822" LastActivityDate="2017-04-25T12:24:25.923" Title="Synonym handling replacement issue with UDF in Apache Spark" Tags="|dataset|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18560" PostTypeId="2" ParentId="18557" CreationDate="2017-04-24T12:42:46.030" Score="1" Body="&lt;p&gt;You might enjoy looking into the literature on &quot;adversarial examples&quot;.  Given an instance $x$ with a label $y$, an adversarial example is a (typically carefully constructed) instance $x'$ that is very similar to $x$, but whose label differs from $y$.  The research literature suggests that it is often possible to find adversarial examples that are very close to the original $x$.  You could use the distance $d(x,x')$ as a measure of sharpness of the decision boundary near $x$; or you could average this over many $x$ to get a global measure of sharpness.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many methods for finding adversarial examples.  A standard simple one is the gradient sign method, originally described in the following paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy. &lt;a href=&quot;https://arxiv.org/pdf/1412.6572&quot; rel=&quot;nofollow noreferrer&quot;&gt;Explaining and harnessing adversarial examples&lt;/a&gt;. &lt;a href=&quot;https://arxiv.org/abs/1412.6572&quot; rel=&quot;nofollow noreferrer&quot;&gt;arXiv:1412.6572&lt;/a&gt;, 2014.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since then there have been numerous improvements that find even closer adversarial examples, e.g., iterative gradient sign (arxiv:1607.02533), Deepfool (arxiv:1511.04599), and others.  You might also be interested in  &lt;a href=&quot;https://github.com/openai/cleverhans&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cleverhans&lt;/a&gt;, a software library to assist with finding adversarial examples.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2017-04-24T19:25:09.813" LastActivityDate="2017-04-24T19:25:09.813" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18562" PostTypeId="1" AcceptedAnswerId="18600" CreationDate="2017-04-24T15:23:01.070" Score="-1" ViewCount="325" Body="&lt;p&gt;I am currently doing pattern recognition on spectograms of audio files using convolutional neural networks.&lt;/p&gt;&#xA;&lt;p&gt;The spectograms are made using &lt;code&gt;matplotlib&lt;/code&gt; &lt;code&gt;cm.jet&lt;/code&gt; colormaps.  Problem with this color map is that it auto ranges its colors based on the min and max value of the input it is given.&lt;/p&gt;&#xA;&lt;p&gt;so an example:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/y3Ger.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y3Ger.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AjmKP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AjmKP.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Spectrograms of two different version of audio file.&#xA;One with static filter output, and the other with delta filter outputs.&lt;/p&gt;&#xA;&lt;p&gt;RGB values show no difference in ranges for both delta and normal, but the db, scale shows a big difference.&lt;/p&gt;&#xA;&lt;p&gt;My input consist of one column of the static and one column of delta, or a matrix (40,2,3), but since these ranges are very different kinda make me suspect that this would not work very well.&lt;/p&gt;&#xA;&lt;p&gt;Am I right or wrong?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-04-26T16:24:34.883" Title="Does the input data representation matter while training CNN for speech recognition?" Tags="|visualization|data|image-recognition|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18565" PostTypeId="2" ParentId="16958" CreationDate="2017-04-24T17:21:16.370" Score="0" Body="&lt;p&gt;Take a look if you have the widget &quot;Map&quot; in your Visualization group. It might be the same widget, just with the different name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks &quot;Geo Map&quot; widget appears for Orange 3.4.2 for Linux, in Visualize group:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SOSKQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SOSKQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, from this &lt;a href=&quot;https://github.com/biolab/orange3/search?q=geo%20map&amp;amp;type=Commits&amp;amp;utf8=%E2%9C%93&quot; rel=&quot;nofollow noreferrer&quot;&gt;commit&lt;/a&gt; in Orange source code, it looks the previous name for this widget was just &quot;Map&quot;.&lt;/p&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="31513" LastEditDate="2017-04-24T18:09:22.900" LastActivityDate="2017-04-24T18:09:22.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18566" PostTypeId="1" CreationDate="2017-04-24T19:07:55.993" Score="-1" ViewCount="30" Body="&lt;p&gt;I'm currently using Weka to prototype some possible applications for machine learning. One such application is a grade based on the performance of a user's mailing list. I have the following attributes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;   Attribute  |   Type&#xA;-----------------------------&#xA;List Name     | Nominal&#xA;Total Sent    | Numeric&#xA;Total Opened  | Numeric&#xA;Total Clicked | Numeric&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My goal is to break up the data set into 9 categories, one for each grade (A+ to C-), but I'm not sure which algorithm I would use, or the proper parameter to begin prototyping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what some of the data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA; List   |   Sent  |  Opened | Clicked &#xA;--------------------------------------&#xA;list 1  |  51587  |  17251  |  1787&#xA;list 2  |  22128  |  8538   |  815&#xA;list 3  |  211007 |  24803  |  1719&#xA;list 4  |  12910  |  1530   |  183&#xA;list 5  |  148054 |  41171  |  0&#xA;list 6  |  182392 |  17579  |  1749&#xA;list 7  |  68212  |  4765   |  447&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="31538" LastEditorUserId="924" LastEditDate="2017-04-24T20:59:43.017" LastActivityDate="2017-04-24T20:59:43.017" Title="What algorithm would I use to create a grade based on 3 numeric attributes?" Tags="|ranking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18568" PostTypeId="2" ParentId="18566" CreationDate="2017-04-24T19:46:16.520" Score="1" Body="&lt;p&gt;You'll have to decide how you want to assign the grades.  You might want to think about what properties do you think should correspond to higher grades.  This doesn't seem like a machine learning problem in its current form, it's just a matter of choosing an evaluation metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had labelled data (for some mailing lists you had grades assigned) then you consider consider applying supervised learning methods.  Without that, there's nothing for a ML algorithm to get a foothold with.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-24T19:46:16.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18571" PostTypeId="2" ParentId="18550" CreationDate="2017-04-25T01:56:52.030" Score="0" Body="&lt;p&gt;I agree with @SmallChess, But I have something to add.(since I can't comment yet)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two concepts would help you to understand most of the Learning Algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Gradient descent&lt;/li&gt;&#xA;&lt;li&gt;Monte Carlo Tree search&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;These two concepts helped me understanding most other concepts. Hope I helped.&lt;/p&gt;&#xA;" OwnerUserId="26856" LastActivityDate="2017-04-25T01:56:52.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18572" PostTypeId="1" AcceptedAnswerId="18574" CreationDate="2017-04-25T04:19:37.280" Score="2" ViewCount="356" Body="&lt;p&gt;This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption that the training, validation and test data sets share the same distribution. They can work to some extent if the distributions differ but not by a lot. Here &quot;can work&quot; means that they work sub-optimally (i.e. can work better if the distributions are the same), not that their theory behind is supposed to deal w/ the distribution difference and can handle them like &quot;nailing it&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence my question: &lt;strong&gt;is there work on predicting based on the assumption that the data sets are actually moving through a series of distribution changes?&lt;/strong&gt; A crazy thought would be to observe the distribution difference between training and validation sets, and assume that the same diff will exist between validation and test sets and learn to predict well on test set. This will work great on time series where the nature of the data might change over time.&lt;/p&gt;&#xA;" OwnerUserId="29428" LastActivityDate="2017-07-17T23:09:46.000" Title="Machine learning learn to work well on future data distribution?" Tags="|machine-learning|distribution|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18573" PostTypeId="2" ParentId="18556" CreationDate="2017-04-25T05:47:25.707" Score="0" Body="&lt;p&gt;Using one column but different values for different classes will fall under regression. The objective would be to predict a value as close as possible to target value. Mean square error, mean absolute error etc. are used as loss functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using two columns with class indicators (1 for the target class and zero for rest) falls under classification. It can be seen as maximizing the probability for target class. Binary cross entropy, categorical cross entropy are used as loss function.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It mostly depends on the application on hand. If your end goal is just to know the class labels then use the second approach. Cross entropy is very sensitive and error will be large if the predicted and target class do not match.  &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-04-25T05:47:25.707" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18574" PostTypeId="2" ParentId="18572" CreationDate="2017-04-25T05:58:55.133" Score="1" Body="&lt;p&gt;It has been studied under various names likes Domain Adaptation,   Sample Selection Bias, Co-variate Shift. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please go trough this &lt;a href=&quot;https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;survey paper&lt;/a&gt; on Transfer learning. It covers all the possible combination like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)same distribution for train and test data &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Gradual change between train and test distribution &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Different but related distribution for train and test &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It'll also give you all the necessary resources required to study further on this topic.  &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-04-25T05:58:55.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18577" PostTypeId="1" AcceptedAnswerId="19210" CreationDate="2017-04-25T08:08:12.643" Score="1" ViewCount="274" Body="&lt;p&gt;I have time series data out of which only 10% is labeled into 10 classes. What should be the methods or models that I should look for to analyze this problem? I know the question is a bit vague but I am not looking for exact answers. I would appreciate any pointers or resources on how to approach these types of problems. &lt;/p&gt;&#xA;" OwnerUserId="31556" LastActivityDate="2017-05-25T16:02:17.370" Title="Methods to analyze time series data when only small amount of it is labeled" Tags="|machine-learning|dataset|time-series|multilabel-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18578" PostTypeId="1" AcceptedAnswerId="18579" CreationDate="2017-04-25T08:56:19.040" Score="0" ViewCount="1294" Body="&lt;p&gt;I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?&lt;/p&gt;&#xA;" OwnerUserId="31475" LastActivityDate="2017-04-25T17:35:06.010" Title="how to do the imputation for categorical feature with a missing rate?" Tags="|missing-data|data-imputation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18579" PostTypeId="2" ParentId="18578" CreationDate="2017-04-25T09:54:45.853" Score="0" Body="&lt;p&gt;You can turn it into a one-hot encoded feature with an added class of 'Missing', depending on the cardinality (how many categories are there). If the cardinality is too high, you will need to use other techniques for high cardinality features but you can still have 'Missing' as an additional category.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-04-25T09:54:45.853" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18581" PostTypeId="1" AcceptedAnswerId="18585" CreationDate="2017-04-25T10:19:25.220" Score="1" ViewCount="6463" Body="&lt;p&gt;I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2&lt;/p&gt;&#xA;&lt;p&gt;The dataset looks like:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;id|question1|question2|is_duplicate&lt;/p&gt;&#xA;&lt;p&gt;0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I am trying to refer to &lt;a href=&quot;https://www.linkedin.com/pulse/duplicate-quora-question-abhishek-thakur&quot; rel=&quot;nofollow noreferrer&quot;&gt;Abhishek Thakur's&lt;/a&gt; feature to get started.&#xA;It says:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/I5ZLw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/I5ZLw.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;As per my understanding the python code for sklearn would be:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    from sklearn.feature_extraction.text import TfidfVectorizer&#xA;    import pandas as pd&#xA;    tfidf_vectorizer = TfidfVectorizer()&#xA;    data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1)&#xA;    data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;data['tf_idf_q1]&lt;/strong&gt; and &lt;strong&gt;data['tf_idf_q2]&lt;/strong&gt; will refer to 2 models for each question as in 1st part of the image.&lt;/p&gt;&#xA;&lt;p&gt;I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question?&#xA;Or do I merge 2 questions and then get a vectorizer?&#xA;Something like below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str))&#xA;data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Any inputs are greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="23773" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2022-03-20T02:52:35.573" Title="Same TF-IDF Vectorizer for 2 data inputs" Tags="|python|nlp|scikit-learn|feature-selection|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18582" PostTypeId="2" ParentId="17972" CreationDate="2017-04-25T10:48:08.877" Score="1" Body="&lt;p&gt;I think it could be that the model is being used and not the data, try: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;hoslem.test(data$y, fitted(model), g = 10)&lt;/p&gt;&#xA;" OwnerUserId="31564" LastActivityDate="2017-04-25T10:48:08.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18583" PostTypeId="1" AcceptedAnswerId="18589" CreationDate="2017-04-25T11:58:13.553" Score="60" ViewCount="61414" Body="&lt;p&gt;I thought both, PReLU and Leaky ReLU are&#xA;&lt;span class=&quot;math-container&quot;&gt;$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Keras, however, has both functions in &lt;a href=&quot;https://keras.io/layers/advanced_activations/&quot; rel=&quot;noreferrer&quot;&gt;the docs&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2&gt;Leaky ReLU&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/layers/advanced_activations.py#L41&quot; rel=&quot;noreferrer&quot;&gt;Source of LeakyReLU&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;return K.relu(inputs, alpha=self.alpha)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Hence (see &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/backend/tensorflow_backend.py#L2529&quot; rel=&quot;noreferrer&quot;&gt;relu code&lt;/a&gt;)&#xA;&lt;span class=&quot;math-container&quot;&gt;$$f_1(x) = \max(0, x) - \alpha \max(0, -x)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h2&gt;PReLU&lt;/h2&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/keras-team/keras/blob/50d83f7ab80c04283236a986a1c99550cecc5140/keras/layers/advanced_activations.py#L121&quot; rel=&quot;noreferrer&quot;&gt;Source of PReLU&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def call(self, inputs, mask=None):&#xA;    pos = K.relu(inputs)&#xA;    if K.backend() == 'theano':&#xA;        neg = (K.pattern_broadcast(self.alpha, self.param_broadcast) *&#xA;               (inputs - K.abs(inputs)) * 0.5)&#xA;    else:&#xA;        neg = -self.alpha * K.relu(-inputs)&#xA;    return pos + neg&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Hence&#xA;&lt;span class=&quot;math-container&quot;&gt;$$f_2(x) = \max(0, x) - \alpha \max(0, -x)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&lt;p&gt;Did I get something wrong? Aren't &lt;span class=&quot;math-container&quot;&gt;$f_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$f_2$&lt;/span&gt; equivalent to &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; (assuming &lt;span class=&quot;math-container&quot;&gt;$\alpha \in (0, 1)$&lt;/span&gt;?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="16338" LastEditDate="2020-07-15T05:53:43.323" LastActivityDate="2020-08-21T09:25:58.537" Title="What is the difference between LeakyReLU and PReLU?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18584" PostTypeId="2" ParentId="18558" CreationDate="2017-04-25T12:18:44.967" Score="0" Body="&lt;p&gt;your &quot;&lt;strong&gt;contains&lt;/strong&gt;&quot; function will create 2 rows out of &quot;Allen Armstrong nishanth hemanth Allen&quot;, since 2 matches will be found: one on Allen, second one on Armstrong. That is why output has 4 instead of 3 expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;strong&gt;replacewith&lt;/strong&gt;&quot; reiterates after that over those 4 but with different replace string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. After &lt;strong&gt;contains&lt;/strong&gt; you are getting:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Allen Armstrong nishanth hemanth Allen&quot;, &quot;Allen&quot;, &quot;Apex Tool Group&quot;&#xA;&quot;Allen Armstrong nishanth hemanth Allen&quot;, &quot;Armstrong&quot;,&quot;Apex Tool Group&quot;&#xA;&quot;shivu Armstrong nishanth&quot;, &quot;Armstrong&quot;,&quot;Apex Tool Group&quot;&#xA;&quot;shree shivu DeWALT&quot;, &quot;DeWALT&quot;,&quot;StanleyBlack&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And after &lt;strong&gt;replacewith&lt;/strong&gt; - you receive your current output respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logic of contains needs to be reworked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider deleting contains function at all, since &lt;code&gt;t1.replaceAll(t2, t3);&lt;/code&gt; does the matching automatically.&lt;/p&gt;&#xA;" OwnerUserId="29534" LastEditorUserId="29534" LastEditDate="2017-04-25T12:24:25.923" LastActivityDate="2017-04-25T12:24:25.923" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18585" PostTypeId="2" ParentId="18581" CreationDate="2017-04-25T12:36:41.400" Score="2" Body="&lt;p&gt;You can use something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;import pandas as pd&#xA;tfidf_vectorizer = TfidfVectorizer()&#xA;raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])&#xA;data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data['q1'])&#xA;data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data['q2'])&#xA;data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]&#xA;X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()&#xA;Y = data_for_model['is_identical'].as_matrix()&#xA;model = Sklearn.LogisticRegression()&#xA;model.fit(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Combined Model - here you actually learn the transformation for all questions. Then transform each one(question) separately to create features for your model training.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;import pandas as pd&#xA;tfidf_vectorizer = TfidfVectorizer()&#xA;raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])&#xA;tf_train_data = pd.concat([data['q1'], data['q2']])&#xA;trained_tf_idf_transformer = tfidf_vectorizer.fit_transform(tf_train_data)&#xA;data['tf_idf_q1'] = trained_tf_idf_transformer.transform(data['q1'])&#xA;data['tf_idf_q2'] = trained_tf_idf_transformer.transform(data['q2'])&#xA;data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]&#xA;X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()&#xA;Y = data_for_model['is_identica'l].as_matrix()&#xA;model = Sklearn.LogisticRegression()&#xA;model.fit(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="27983" LastEditDate="2019-05-16T08:37:58.460" LastActivityDate="2019-05-16T08:37:58.460" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18586" PostTypeId="2" ParentId="18577" CreationDate="2017-04-25T13:05:38.337" Score="0" Body="&lt;p&gt;I'm not an expert, but here my 2 preliminary thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Clustering whole set over those 10 classes:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/K-means_clustering&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29534" LastActivityDate="2017-04-25T13:05:38.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18588" PostTypeId="1" CreationDate="2017-04-25T14:57:26.047" Score="1" ViewCount="248" Body="&lt;p&gt;I have data for 20 different people and am training a model (e.g. a neural network with the same hyperparameters) on the data from each person; so this gives me 20 models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose to use RMSE to assess the performance. However, since the training data is shuffled, the computed RMSE is nondeterministic and so oscillates. So I thought running each model 10 times and averaging the results, i.e. the RMSE's, would give me a better estimate of performance. But this is for a single person/model. How do I combine the performance of everything, i.e. all 20 models, into a single measure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run each of the 20 models 10 times for a total of 200 RMSE values, and average all? Or first compute the average for each person, and then the average of these averages?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe a different method is better? The end goal is to compare a couple of models (e.g. NN vs SVM).&lt;/p&gt;&#xA;" OwnerUserId="31570" LastActivityDate="2017-08-23T18:12:52.970" Title="ML models: average of all versus average of averages?" Tags="|neural-network|performance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18589" PostTypeId="2" ParentId="18583" CreationDate="2017-04-25T15:05:41.500" Score="74" Body="&lt;p&gt;Straight from &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)&quot; rel=&quot;noreferrer&quot;&gt;wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BpZ0e.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BpZ0e.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Leaky ReLU&lt;/strong&gt;s allow a small, non-zero gradient when the unit is not active.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Parametric ReLU&lt;/strong&gt;s take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30928" LastEditorUserId="35644" LastEditDate="2018-02-28T11:04:05.640" LastActivityDate="2018-02-28T11:04:05.640" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18590" PostTypeId="2" ParentId="18588" CreationDate="2017-04-25T16:03:05.393" Score="1" Body="&lt;p&gt;It shouldn't matter if you average all of them or take the average of the average for each person. It turns out that it will give you the same number. This is known as the law of total expectation. Essentially, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E(X)=E(E(X|Y)) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the wikipedia article for more details:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Law_of_total_expectation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Law_of_total_expectation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2017-04-25T16:03:05.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18591" PostTypeId="2" ParentId="18588" CreationDate="2017-04-25T17:07:16.853" Score="0" Body="&lt;p&gt;My two cents&lt;br&gt;&#xA;If your intention is training a different model for each individual, then you should consider using more than just average for between model comparison.&lt;br&gt;&#xA; Imagine one model performance is great(low RMSE) for 70% of individuals and performs poorly for the other 30% and the second model has mediocre performance across all individuals.&lt;br&gt;&#xA; Both models could have the same average but in reality would behave quite differently.&lt;br&gt;&#xA;Moreover, you should consider using some sort of statistical measure for the RMSE of each individual model(say a confidence interval of the RMSE).&#xA;Maybe one model is more volatile than the other.&lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-04-25T17:07:16.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18592" PostTypeId="2" ParentId="18578" CreationDate="2017-04-25T17:35:06.010" Score="0" Body="&lt;p&gt;I have read the comments of another answer and seems like you have lots of missing data. I would then in this case recommend the mice imputation (multiple imputation with chained equation). It deals with all type of different variable types (numerical, categorical, binary) and fill the NA values depends on the type of the variable. If you use R, you can check &lt;a href=&quot;https://cran.r-project.org/web/packages/mice/mice.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/mice/mice.pdf&lt;/a&gt;. It contains detailed package and function information and examples.&lt;/p&gt;&#xA;" OwnerUserId="14152" LastActivityDate="2017-04-25T17:35:06.010" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18595" PostTypeId="1" AcceptedAnswerId="18604" CreationDate="2017-04-25T18:58:16.953" Score="-1" ViewCount="63" Body="&lt;p&gt;i am working on my final year project that is a social network. Based on user interest, i have to add him in groups based on area he lives in, age group, interest type, gender and some other features.i have to use machine learning to predict in which group should i place him. I am thinking of using classification but i need data set to train which i don't have at all nor there is any data set related to this problem, so if any one of you guide me in right direction how can i do this ?&lt;/p&gt;&#xA;" OwnerUserId="31580" LastActivityDate="2017-04-26T06:52:11.400" Title="using classification when there is no dataset + guidance" Tags="|machine-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18596" PostTypeId="2" ParentId="17641" CreationDate="2017-04-25T21:31:02.920" Score="2" Body="&lt;p&gt;I have written some code for this. It can be found &lt;a href=&quot;https://stats.stackexchange.com/questions/270201/pooling-levels-of-categorical-variables-for-regression-trees/&quot;&gt;here&lt;/a&gt;. In answer to your specific questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have tried to optimize for speed. What I did should be a little faster than the code above.&lt;/li&gt;&#xA;&lt;li&gt;I do not use out of bag records. In fact the original documentation does not suggest this. I created another &lt;a href=&quot;https://stats.stackexchange.com/questions/274857/proximity-matrix-should-i-include-in-bag-records&quot;&gt;post&lt;/a&gt; to see if the consequences are understood.&lt;/li&gt;&#xA;&lt;li&gt;This is handled in my code by normalizing by the total possible leaves that could be matched.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2017-04-25T21:31:02.920" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18597" PostTypeId="1" CreationDate="2017-04-25T22:43:43.510" Score="1" ViewCount="744" Body="&lt;p&gt;I want to combine the following vectors in a way that just the red point (number 7) becomes inconsistent with other points( become an outlier and become distant from other points) and other points become consistent with each other. Please note that I have tested mahalanobis distance and Kullback-Leibler divergence between two vectors but they were not so good and detects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a=[1.3269   1.3354  1.3318  1.3282  1.34666 1.3460  1.36084 1.3526  1.3539  1.3510  1.3480  1.3479  1.34893]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b=[0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/E5Spn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/E5Spn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/6srGb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6srGb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="12345" LastActivityDate="2017-04-26T21:13:32.287" Title="Detecting outlier with combining two vectors" Tags="|data-mining|feature-extraction|feature-engineering|anomaly-detection|outlier|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18598" PostTypeId="2" ParentId="18597" CreationDate="2017-04-25T23:09:32.923" Score="1" Body="&lt;h2&gt;Autoencoder Solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You could try an autoencoder. The autoencoder would take an input vector and it would try to recreate it as an output. So you take your input, and measure the distance between the input and the predicted output variable, using your metric of choice (euclidean should work but try various). Larger distances can be thought of as more abnormal. So you can stack rank your observations from weirdest to most normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure that you are only training the autoencoder on normal data though. This would of course assume that you have more than the 13 samples that you are looking at. If not this probably isn't going to work very well, just because of the small sample.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;KDE Solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The idea is to use Kernel Density Estimation to generate a non-parametric joint density of your data set. You then find what the probability of finding a value that extreme would be. Here is some code using python's sklearn package:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.neighbors.kde import KernelDensity&#xA;import numpy as np&#xA;X=np.matrix([[1.3269, 1.3354, 1.3318, 1.3282, 1.34666, 1.3460, 1.36084, 1.3526, 1.3539, 1.3510, 1.3480, 1.3479, 1.34893],[0.0352, 0.0992, 0.1570, 0.1431, 0.1634, 0.1629, 0.1046, 0.1655, 0.1635, 0.1642, 0.1658, 0.1666, 0.15735]])&#xA;kde = KernelDensity(kernel='gaussian', bandwidth=.45).fit(X.T)&#xA;score=kde.score_samples(X.T)&#xA;prob=np.exp(score)&#xA;print(prob/prob[6])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code shows that the observations in the lowest probability density areas are observations 1,2 and 7. Of course this would work better with a larger sample, and you need to fuss with the bandwidth to calibrate it, but that this should do it.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastEditorUserId="16538" LastEditDate="2017-04-26T21:13:32.287" LastActivityDate="2017-04-26T21:13:32.287" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18599" PostTypeId="2" ParentId="18597" CreationDate="2017-04-26T00:07:46.130" Score="1" Body="&lt;p&gt;From the second graph it seems pretty easy to identify the outlier. You could probably just fit a simple polynomial (or some other function) and then flag all points that have a distance greater than 2 standard deviations (or whatever seems appropriate) from the fitted curve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0HEzS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0HEzS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;b=np.array([0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046&#xA;            ,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735])&#xA;x = np.arange(13)&#xA;p = np.poly1d(np.polyfit(x,b,4))&#xA;y = p(x)&#xA;&#xA;plt.plot(x,b,'ro')&#xA;plt.plot(x,y,'b-')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-04-26T00:15:31.117" LastActivityDate="2017-04-26T00:15:31.117" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18600" PostTypeId="2" ParentId="18562" CreationDate="2017-04-26T03:33:59.680" Score="1" Body="&lt;p&gt;Data representation does matter because this is all the information that you pass to a learning algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is normal for static and delta (delta-delta) to have different range (I have worked with mfccs). They represent different information.&#xA;Static features can be small but they may change rapidly making delta large or vice-versa. The blue regions in the first spectrogram (low magnitude) becomes red in second( high magnitude). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as all the input are processed in the same manner (static followed by delta followed by delta-delta or any order), it won't be a problem. &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2017-04-26T03:33:59.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18601" PostTypeId="2" ParentId="18597" CreationDate="2017-04-26T05:11:26.767" Score="1" Body="&lt;p&gt;I don't really agree with the idea of &quot;wanting&quot; a point to be an outlier and then massaging the algorithm to make it so. You have 2 dimensions and its either an outlier or its not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one standardizes the data and then deciphers the Mahalanobis distances, point 6 is only one of two points that sit outside of a certain threshold (point 0 being the other point). Beyond that theres not much you can do beyond some sort of nonlinear transformation which you know to be true due to some deterministic knowledge you have about a particular phenomenon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways... here's the two outlier version in case you weren't standardizing your data first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame, read_csv&#xA;dfR = read_csv('~/Machine_Learning/ipython_notebooks/AB_outlier.csv')&#xA;df=(dfR-dfR.mean())/dfR.std()&#xA;&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.covariance import EmpiricalCovariance, MinCovDet&#xA;emp_cov = EmpiricalCovariance().fit(df)&#xA;&#xA;fig = plt.figure()&#xA;plt.subplots_adjust(hspace=-.1, wspace=.4, top=.95, bottom=.05)&#xA;&#xA;# Show data set&#xA;subfig1 = plt.subplot(3, 1, 1)&#xA;my_plot = subfig1.scatter(df.A,df.B)&#xA;subfig1.set_xlim(subfig1.get_xlim()[0], 11.)&#xA;subfig1.set_title(&quot;Mahalanobis distances&quot;)&#xA;&#xA;# Show contours of the distance functions&#xA;xx, yy = np.meshgrid(np.linspace(plt.xlim()[0], plt.xlim()[1], 100),&#xA;                 np.linspace(plt.ylim()[0], plt.ylim()[1], 100))&#xA;zz = np.c_[xx.ravel(), yy.ravel()]&#xA;&#xA;mahal_emp_cov = emp_cov.mahalanobis(zz)&#xA;mahal_emp_cov = mahal_emp_cov.reshape(xx.shape)&#xA;emp_cov_contour = subfig1.contour(xx, yy, np.sqrt(mahal_emp_cov),&#xA;                              cmap=plt.cm.PuBu_r,&#xA;                              linestyles='dashed')&#xA;&#xA;subfig1.legend([emp_cov_contour.collections[1],&#xA;            my_plot],&#xA;           ['MLE dist'],&#xA;           loc=&quot;upper right&quot;, borderaxespad=0)&#xA;plt.xticks(())&#xA;plt.yticks(())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xUdbm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xUdbm.png&quot; alt=&quot;Plot of standardized data with Mahalanobis distance lines&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2017-04-26T05:11:26.767" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18604" PostTypeId="2" ParentId="18595" CreationDate="2017-04-26T06:52:11.400" Score="1" Body="&lt;p&gt;As far as I know you shouldn't use ML at this stage. There are two problems:&lt;br&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can not get enough data related to your task.&lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Even anyhow you manage to get the data it will be either to general or&lt;br&gt;&#xA;will be some other domain based so relying on the result would be &#xA;difficult.&lt;br&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Instead you can write some rule based or weighted value solutions like depending upon the type of group what age-group should be part of the group. Regular expressions to extract some keywords from interest tags and other text features and these can be used to decide the probability of a person falling into a group. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once if you have enough data and their labels using this method you can start trying ML algorithms and they will give the results.&lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2017-04-26T06:52:11.400" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18605" PostTypeId="1" CreationDate="2017-04-26T07:15:11.383" Score="0" ViewCount="59" Body="&lt;p&gt;How to deal with videos where the frame sizes are not the same frame to frame?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example &lt;a href=&quot;https://www.youtube.com/watch?v=5cKpzp358F4&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt; moves up and down and when it does, the video part of the screen has a different amount of pixels vertically.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to deal with different frame sizes in a CNN?&lt;/p&gt;&#xA;" OwnerUserId="23821" LastEditorUserId="29169" LastEditDate="2021-02-16T02:42:59.470" LastActivityDate="2021-02-16T02:42:59.470" Title="How to deal with changing video frame sizes in a CNN?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18608" PostTypeId="2" ParentId="18605" CreationDate="2017-04-26T09:11:38.693" Score="1" Body="&lt;p&gt;In &lt;a href=&quot;https://www.youtube.com/watch?v=5cKpzp358F4&quot; rel=&quot;nofollow noreferrer&quot;&gt;your example&lt;/a&gt;, you are seeing the effect of a stabilisation algorithm applied to otherwise jerky movement - and the effect is a moving &quot;window&quot; inside the video frame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual video file has a fixed frame size. As far as I know there are no video formats that switch size frame to frame. You may have different areas of focus to worry about, but should never need to worry about frame size switching on you mid video. You can rely on your video processing library to present a consistent frame size for the whole of a video.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find that the border of the view inside the video frame interferes with the performance of your CNN once trained, then you could deal with it in a few ways. A couple of simple approaches might be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Crop out a centre portion that always reliably has content in, and use just that portion for your input features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Detect and flood-fill the black border with the mean pixel value, using an image/video library (like OpenCV) so that edges around it cause less activity in your CNN.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-04-26T09:11:38.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18609" PostTypeId="1" CreationDate="2017-04-26T09:27:40.393" Score="4" ViewCount="4326" Body="&lt;p&gt;I am trying to merge two address columns into one and separate the resulting string with '--'. The dataset has 10 million rows and 33 columns - but the number of rows grows for a million or so a month. This line in Pandas/Python is very slow. Any ideas of how can I make it faster and scalable for future use? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31602" LastEditorUserId="31513" LastEditDate="2017-06-05T19:30:11.103" LastActivityDate="2017-06-05T19:30:11.103" Title="Performance issues when merging two dataframe columns into one on millions rows with Pandas" Tags="|python|pandas|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18610" PostTypeId="2" ParentId="18609" CreationDate="2017-04-26T12:03:47.853" Score="1" Body="&lt;p&gt;With my solution, you have to parse your column with string type:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[[&quot;address1&quot;, &quot;address2&quot;]] = df[[&quot;address1&quot;, &quot;address2&quot;]].dropna().astype(str)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then define concatenation fonction to concatenate two strings&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def concat_string(a, b):&#xA;    return a + '--' +  b&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally I advise you to work with pandas series, it will be improve your operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I apply &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.combine.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;combine&lt;/a&gt; (from pandas series) with in param the function define above concat_string. concat_string takes in param element from df['address1'] and combine them with df['address2'].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Combine vectorize the operation, it &quot;replaces&quot; traditional loop.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['address'] = df['address1'].combine(df['address2'], concat_string)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Benchmark:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.shape =&amp;gt; (10000, 2)&#xA;&#xA;%time df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)&#xA;CPU times: user 4.6 s, sys: 18.8 ms, total: 4.62 s&#xA;Wall time: 4.64 s&#xA;&#xA;%time df['address_Test'] = df['address1'].combine(df['address2'], concat_string)&#xA;CPU times: user 302 ms, sys: 27.6 ms, total: 329 ms&#xA;Wall time: 321 ms&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31600" LastEditorUserId="31600" LastEditDate="2017-04-26T13:00:38.670" LastActivityDate="2017-04-26T13:00:38.670" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18614" PostTypeId="1" CreationDate="2017-04-26T14:01:07.020" Score="-3" ViewCount="84" Body="&lt;p&gt;Looking for Data science training with Python in Bangalore location with real time scenarios &amp;amp; Project preparations.&lt;/p&gt;&#xA;" OwnerUserId="31614" LastActivityDate="2017-04-26T15:06:22.713" Title="Looking for Data science training with Python" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="0" ClosedDate="2017-04-26T19:21:02.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="18615" PostTypeId="2" ParentId="18614" CreationDate="2017-04-26T15:00:02.227" Score="-1" Body="&lt;p&gt;Sorry for answering it as I don't have required reputation to comment.&lt;br&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This site is meant for problem discussion and possible approaches to solve them. It is not portal to ask the question you have asked. You can google for the problem that you have asked and I am sure you will get an answer. You should have read the instruction and read community guidelines before asking this question.&lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2017-04-26T15:00:02.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18616" PostTypeId="2" ParentId="18614" CreationDate="2017-04-26T15:06:22.713" Score="1" Body="&lt;p&gt;There are a ton of great resources out there online, many free. I personally have experience with and recommend edX and DataCamp&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.datacamp.com/courses/intro-to-python-for-data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.datacamp.com/courses/intro-to-python-for-data-science&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edx.org/course/introduction-python-data-science-microsoft-dat208x-5&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.edx.org/course/introduction-python-data-science-microsoft-dat208x-5&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31618" LastActivityDate="2017-04-26T15:06:22.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18617" PostTypeId="2" ParentId="17776" CreationDate="2017-04-26T15:48:01.367" Score="2" Body="&lt;p&gt;PyTorch should work fine in WSL (CPU only).&lt;/p&gt;&#xA;" OwnerUserId="24724" LastActivityDate="2017-04-26T15:48:01.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18618" PostTypeId="1" CreationDate="2017-04-26T15:53:56.830" Score="2" ViewCount="193" Body="&lt;p&gt;Does anyone know of any libraries I could use to convert a word into a semantic prime, given a list of approved primes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, given a sentence or a word, I would like to find the combination of &quot;prime&quot; words that best match the input semantically. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good example of this would be a function that takes in a Wikipedia article and outputs something like Simple Wikipedia - it has the same semantic content, but restricts word usage to a number of prime words (in this case, simpler words).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example: say my primes are &lt;code&gt;[&quot;person&quot;, &quot;move&quot;, &quot;tool&quot;, &quot;nice&quot;, &quot;building&quot;]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I would like to have some function, library, or NLP algorithm that converts &lt;code&gt;&quot;hammer&quot;&lt;/code&gt; into &lt;code&gt;[&quot;building&quot;, &quot;tool&quot;]&lt;/code&gt;, &lt;code&gt;&quot;car&quot;&lt;/code&gt; into &lt;code&gt;[&quot;moving&quot;, &quot;tool&quot;]&lt;/code&gt;, and  &lt;code&gt;&quot;friend&quot;&lt;/code&gt; into &lt;code&gt;[&quot;nice&quot;, &quot;person&quot;]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first intuition was to use word2vec: loop through every combination of the primes and find the combination of primes who's vector most closely matches the original word. However, I suspect this will not work out very well (partially because vector addition doesn't seem to be accurate enough). Is there a more robust way to do this?&lt;/p&gt;&#xA;" OwnerUserId="31620" LastActivityDate="2017-04-27T15:19:57.827" Title="Convert Word to Semantic Prime" Tags="|nlp|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18621" PostTypeId="2" ParentId="6838" CreationDate="2017-04-26T21:52:03.773" Score="3" Body="&lt;p&gt;It really depends what you want to achieve, what your data look like and etc. SVM will generally perform better on linear dependencies, otherwise you need nonlinear kernel and choice of kernel may change results. Also, SVM are less interpretable - for e.g if you want to explain why the classification was like it was - it will be non-trivial. Decision trees have better interpretability, they work faster and if you have categorical/numerical variables its fine, moreover: non-linear dependencies are handled well (given N large enough). Also they train faster than SVM in general, but they have tendency to overfit...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also try Logistic Regression - great interpretable classifier)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum it up - the rule of thumb is try anything and compare what gives you best results/interpretation.&lt;/p&gt;&#xA;" OwnerUserId="31627" LastActivityDate="2017-04-26T21:52:03.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18622" PostTypeId="2" ParentId="12281" CreationDate="2017-04-26T22:01:07.747" Score="0" Body="&lt;p&gt;Basically if I rephrase your task - you have a large document which you want to summarize. &#xA;Text mining is your tool - you can either choose traditional approaches such as tf-idf, tf and etc. &#xA;I would recommend use holmertz technique - in such framework it makes stuff easier as it can detect stopwords on its own, extract features and etc.&#xA;Hierarchical clustering can also work, check if you will not get obvious words as cluster centers - filtering them require subject matter knowledge and additional time.&lt;/p&gt;&#xA;" OwnerUserId="31627" LastActivityDate="2017-04-26T22:01:07.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18623" PostTypeId="1" CreationDate="2017-04-27T00:40:18.787" Score="-1" ViewCount="30" Body="&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a binary classifier that tries to predict when -- if ever -- a user goes bad, a terminal state from which a user cannot recover. This phenomenon is tricky becuase a user might start off good, go bad suddenly and only get caught after a long time. Currently I've defined my target variable at the user-month level, so for each user there is one training record for every month that the user existed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; Should I only include a user's first bad month, all bad months, or something else? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional Context:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I'm facing is that some users go bad and remain undetected for many, many months. If I include all of a user's bad months, a small percentage of the users dominate and the model focuses on them at the expense of other bad users. When I include all bad months, I have one feature in particular that performs extremely well (causes the model to have a high AUC value). When I include only a user's first bad month, this same feature has little to no importance and many of the dominant bads go undetected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that today there are some users that exhibit the same signs of the dominant bads; and I suspect that they are in fact bad. If I train my model on only the first bad month and deploy it into production, I'm worried my model will likely miss the obvious. What's the best way to tackle this problem? I'm also open to using something other than client-months, e.g., &quot;first bad occurrence in the next 6 months&quot;. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-04-27T00:57:41.197" Title="Time-based over-sampling dilemma" Tags="|machine-learning|classification|predictive-modeling|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18624" PostTypeId="2" ParentId="18623" CreationDate="2017-04-27T00:57:41.197" Score="1" Body="&lt;p&gt;If you have time series for good clients and time series for bad clients, I would suggest using something along the lines of dynamic time warping to get out of a time domain, and into a dissimilarity domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From there you could just use just about any classifier that you can think of, I have a silly sort of toy example that detects whether or not it is a superbowl sunday based on traffic to a website. &lt;a href=&quot;https://github.com/Rblivingstone/SuperBowl&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Rblivingstone/SuperBowl&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the nice thing is that now that you are in dissimilarity space you can classify based on a few exemplars, say two or three columns in dissimilarity matrix, and feed that into your classifier. It should be relatively easy to implement for your particular use case.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2017-04-27T00:57:41.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18626" PostTypeId="1" CreationDate="2017-04-27T04:49:28.193" Score="7" ViewCount="17842" Body="&lt;p&gt;What is the best GPU for deep learning currently available on the market?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard that Titan X Pascal from NVidia might be the most powerful GPU available at the moment, but would be interesting to learn about other options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And additionally - what might be considered a good machine for data-intensive computing, in terms of processor parameters, RAM etc. ?&lt;/p&gt;&#xA;" OwnerUserId="88" LastActivityDate="2017-09-06T07:22:29.133" Title="What is the best hardware/GPU for deep learning?" Tags="|machine-learning|deep-learning|tensorflow|gpu|" AnswerCount="2" CommentCount="0" ClosedDate="2017-05-06T09:25:06.887" ContentLicense="CC BY-SA 3.0" />
  <row Id="18628" PostTypeId="2" ParentId="18626" CreationDate="2017-04-27T05:19:21.610" Score="7" Body="&lt;p&gt;I would recommend you read this article carefully: &lt;a href=&quot;http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/&quot; rel=&quot;noreferrer&quot;&gt;http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The most important feature for judging deep learning performance is memory bandwidth. GTX 1080 Ti's memory bandwidth is 484 GB/s and TITAN X is 480 GB/s. Read this Quora answer to understand why memory bandwidth is the most important feature better: &lt;a href=&quot;https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning&quot; rel=&quot;noreferrer&quot;&gt;https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tim has also given a &quot;theoretical&quot; rough comparison between GPUs based on their tech specs. You can also see that in his comparison 1080 Ti and TITAN X are roughly at the same performance level.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use 2 1080 Ti instead of a TITAN X since GTX 1080 Ti is definitely cheaper.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've personally evaluated the performance of 1080 Ti vs. TITAN X and they are roughly the same.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28556" LastActivityDate="2017-04-27T05:19:21.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18629" PostTypeId="1" CreationDate="2017-04-27T05:58:35.980" Score="8" ViewCount="3696" Body="&lt;p&gt;How to use Naive Bayes for multi-label text classification in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using naiveBayes() from e1071 library but it seems that while training, it doesn't accept multi-label class variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created TermDocumentMatrix using the text document corpus and tried to create the model using this matrix and class variable(list of topics a particular document belongs to). Below is the code that I used.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trainvector &amp;lt;- as.vector(traindata$bodyText)&#xA;&#xA;trainsource &amp;lt;- VectorSource(trainvector)&#xA;&#xA;traincorpus &amp;lt;- Corpus(trainsource)&#xA;&#xA;trainmatrix &amp;lt;- t(TermDocumentMatrix(traincorpus))&#xA;&#xA;model &amp;lt;- naiveBayes(as.matrix(trainmatrix), as.factor(traindata$topics))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last line gives below error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in sort.list(y) : 'x' must be atomic for 'sort.list'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Have you called 'sort' on a list?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- naiveBayes(as.matrix(trainmatrix), as.factor(unlist(traindata$topics)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but got error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in tapply(var, y, mean, na.rm = TRUE) : &#xA;  arguments must have same length&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31637" LastEditorUserId="11097" LastEditDate="2017-04-27T07:02:43.893" LastActivityDate="2018-11-04T14:26:52.760" Title="Naive Bayes for Multi label text classification" Tags="|multilabel-classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18636" PostTypeId="2" ParentId="11025" CreationDate="2017-04-27T12:44:31.750" Score="0" Body="&lt;p&gt;Besides the excellent references given by &lt;code&gt;sebap123&lt;/code&gt;, from the &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning Book&lt;/a&gt; by Ian Goodfellow et.al,&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The recurrent neural network [given] is universal in the&#xA;sense that any function computable by a Turing machine can be computed by such&#xA;a recurrent network of a finite size. &lt;strong&gt;The output can be read from the RNN after&#xA;a number of time steps that is asymptotically linear in the number of time steps&#xA;used by the Turing machine and asymptotically linear in the length of the input&lt;/strong&gt;&#xA;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="9388" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-04-27T12:44:31.750" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18638" PostTypeId="2" ParentId="18554" CreationDate="2017-04-27T14:26:02.007" Score="4" Body="&lt;p&gt;Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best thing to do is try both methods and then CV and compare performances. &lt;a href=&quot;https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="31245" LastEditDate="2020-08-31T12:21:31.280" LastActivityDate="2020-08-31T12:21:31.280" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18639" PostTypeId="2" ParentId="14291" CreationDate="2017-04-27T15:14:38.197" Score="1" Body="&lt;p&gt;I think you might want to check Andrej Karpathy's work around charRNNs some pretty cool work is being done. The github link has all the relevant code too.&#xA;If you are looking for a more applied way - you can check my blog on deep learning gender from name which effectively uses character level LSTM RNNs to learn patterns : &#xA;&lt;a href=&quot;http://medium.com/@prdeepak.babu/deep-learning-gender-from-name-lstm-recurrent-neural-networks-448d64553044&quot; rel=&quot;nofollow noreferrer&quot;&gt;medium.com/@prdeepak.babu/deep-learning-gender-from-name-lstm-recurrent-neural-networks-448d64553044&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this help.&lt;/p&gt;&#xA;" OwnerUserId="31663" LastEditorUserId="43472" LastEditDate="2018-07-31T20:50:43.493" LastActivityDate="2018-07-31T20:50:43.493" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18640" PostTypeId="2" ParentId="18618" CreationDate="2017-04-27T15:19:57.827" Score="1" Body="&lt;p&gt;Finding a robust model for what you are looking or trying to build is quite difficult at this point of time and as per I know there are no any such existing algorithms to do this.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach that you can follow is make a Knowledge base for your primes containing info/word it can be related with.&lt;br&gt;&#xA;To handle most of the relation you will need a word dictionary which contains all the possible related words. Making such an exhaustive dictionary is not completely possible/feasible. But if you have some domain specific work you can make a dictionary which can handle 80% of you cases. You can use wordnet/word2vec to find the most similar words of your base primes and can extrapolate those words.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from these you would need a set of possible relations that the word can have. For this you can use models like Open IE to extract the Subject-Object-Relation and you can associate these relations with your primes. But for this you will require a dataset of your domain from which you can get the possible relations. &lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2017-04-27T15:19:57.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18641" PostTypeId="1" AcceptedAnswerId="18651" CreationDate="2017-04-27T16:14:50.870" Score="0" ViewCount="508" Body="&lt;p&gt;I have a C5.0 model that is trained to predict binary class (C1/C2) on a dataset with 20 features. The model is configured to perform boosting (10 trials) and it has a &lt;code&gt;miss-classification cost function&lt;/code&gt; (&lt;code&gt;100:1&lt;/code&gt; where 100 is the cost for miss-classifying a Negative sample as Positive and 1 is the cost for miss-classifying Negative as Positive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at predicted probabilities generated by the model, I can see that it ranges from 0 to 1 for each class. i.e, I have instances where the predicted class (C1) has a probability lower than 0.5 ( for example: predicted class=C1 and predicted probability=0.1 ) - This is where the question arises: if &lt;code&gt;P(C1) &amp;lt; 50%&lt;/code&gt; why is it classified as C1 (there are only two classes, C1 and C2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my understanding of decision trees, the predicted probabilities are often generated based on the percentage of test cases on the leaf node that were labeled in each class divided by the total number of instances hitting that leaf node. This method will dictate the probability for two classes must sum up to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is why does the model classify an instance in class C1 if it has only got 0.1 confidence in it. Does a predicted probability of 0.1 on class C1 mean that there is a 0.9 confidence in it belonging to class C2? If so, why does it classify an instance in class C1 is it has less than 0.5 confidence in it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own theory is that this might be due to boosting and miss-classification cost and the way they are influencing the predicted class.&lt;/p&gt;&#xA;" OwnerUserId="17777" LastEditorUserId="17777" LastEditDate="2017-07-25T21:41:48.927" LastActivityDate="2017-07-25T21:41:48.927" Title="Why predicted proababilities from this binary classifier does not sum up to 1?" Tags="|classification|predictive-modeling|probability|binary|" AnswerCount="1" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18642" PostTypeId="1" CreationDate="2017-04-27T17:37:47.823" Score="0" ViewCount="102" Body="&lt;p&gt;What techniques are suitable and what do I need to learn in order to&#xA;detect and count the number of &quot;events&quot; (pic) that consist of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) shape 1 3 and 4 and not of (2 and 5) or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) of all shapes above the blue line with a certain minimum &quot;volume&quot; like shapes 1 and 4?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first idea is to use a window function and somehow integrate the area but i want to be sure to use an appropriate procedure. I havent worked with time series before and I dont know were to get startet. I have to use python for this project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NiOhp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NiOhp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31668" LastActivityDate="2017-04-27T19:47:23.357" Title="How to &quot;count&quot; certain events in a time series" Tags="|time-series|sequential-pattern-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18643" PostTypeId="1" CreationDate="2017-04-27T17:39:03.073" Score="1" ViewCount="160" Body="&lt;p&gt;During the past few years several important areas of image processing and image classification or generation became dominated by convolutional neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested if there are any methods coming from mathematical physics context (like methods designed for solving ill-posed problems, spectral analysis, image deblurring and deringing), that outperform neural network-based approaches in 2017 for some common computer vision or image processing problem. Or methods that don't have any neural network-based rivals. Maybe in the field of biomedical image analysis (just a guess)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The deeper and more specific the answer is, the better.&lt;/p&gt;&#xA;" OwnerUserId="31667" LastActivityDate="2017-04-27T17:39:03.073" Title="Mathematical physics applications in present-day image processing" Tags="|image-classification|computer-vision|image-recognition|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18644" PostTypeId="2" ParentId="18642" CreationDate="2017-04-27T18:45:51.390" Score="1" Body="&lt;p&gt;Try generating a dictionary of patterns you want to identify.  You can then use convolutions/ cross-correlations to identify where these patterns appear in your data.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Convolution&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Convolution&lt;/a&gt;&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Cross-correlation&lt;/a&gt;&#xA;&lt;a href=&quot;http://paulbourke.net/miscellaneous/correlate/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://paulbourke.net/miscellaneous/correlate/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method is also called 'matched filter'.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-04-27T18:45:51.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18647" PostTypeId="1" CreationDate="2017-04-27T19:37:42.813" Score="5" ViewCount="6317" Body="&lt;p&gt;Scikit-learn lists these as the implemented activation functions for it's multi-layer perceptron classifier:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x&#xA;‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).&#xA;‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).&#xA;‘relu’, the rectified linear unit function, returns f(x) = max(0, x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Does anyone know if it is possible to implement a custom activation function? If not, can someone point me to a library where this is possible?&lt;/p&gt;&#xA;" OwnerUserId="31676" LastEditorUserId="69840" LastEditDate="2021-09-09T16:15:10.040" LastActivityDate="2021-09-09T16:15:10.040" Title="Is it possible to customize the activation function in scikit-learn's MLPClassifier?" Tags="|neural-network|scikit-learn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="18648" PostTypeId="2" ParentId="18642" CreationDate="2017-04-27T19:47:23.357" Score="1" Body="&lt;p&gt;I suggest to start with &quot;outlier detection&quot;, &quot;anomaly detection&quot;, filtering methods. Its pretty wide topic to cover but you need to start from somewhere.&lt;/p&gt;&#xA;" OwnerUserId="31627" LastActivityDate="2017-04-27T19:47:23.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18651" PostTypeId="2" ParentId="18641" CreationDate="2017-04-27T23:42:54.567" Score="2" Body="&lt;p&gt;You are absolutely right to be skeptical about the results you are getting. I've been using boosted trees (not necessarily C5.0) for over 5 years, and I've never encountered results like what you've described here. Excluding the possibility that you simply had a typo in your code, I can think of two reasons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Low decision threshold&lt;/strong&gt;: As is often the case in class imbalance problems (e.g., fraud detection, marketing response, bankruptcy, etc) a low probability could still indicate the minority class if the decision threshold is also low. In your case, if the model might have automatically decided to pick a threshold of 0.05, then 0.1 would be flagged as C1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cost function:&lt;/strong&gt; This is what you mentioned -- if your model has a high cost associated with missing true positives then 0.1 could be classified as C1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-04-27T23:42:54.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18652" PostTypeId="2" ParentId="6798" CreationDate="2017-04-28T01:26:56.810" Score="5" Body="&lt;p&gt;List of NLP &lt;a href=&quot;https://www.kaggle.com/competitions&quot; rel=&quot;nofollow noreferrer&quot;&gt;competitions&lt;/a&gt; on &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt; by popularity [number of teams]:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Two Sigma Connect: Rental Listing Inquiries [2709 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Home Depot Product Search Relevance [2125 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/home-depot-product-search-relevance&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/home-depot-product-search-relevance&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Quora Question Pairs [2123 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/quora-question-pairs&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/quora-question-pairs&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;What's Cooking? [1388 teams] &lt;a href=&quot;https://www.kaggle.com/c/whats-cooking&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/whats-cooking&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Crowdflower Search Results Relevance [1326 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/crowdflower-search-relevance&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/crowdflower-search-relevance&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Bag of Words Meets Bags of Popcorn [578 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/word2vec-nlp-tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/word2vec-nlp-tutorial&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Transfer Learning on Stack Exchange Tags [380 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/transfer-learning-on-stack-exchange-tags/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/transfer-learning-on-stack-exchange-tags/data&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;Facebook Recruiting III - Keyword Extraction [367 teams] &lt;a href=&quot;https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction&lt;/a&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;New entrants are prohibited, you won't be able to download data for this competition&lt;/em&gt;   &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Dato: Truly Native? [274 teams] &lt;a href=&quot;https://www.kaggle.com/c/dato-native&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/dato-native&lt;/a&gt;   &lt;/li&gt;&#xA;&lt;li&gt;The Allen AI Science Challenge [170 teams]&#xA;&lt;a href=&quot;https://www.kaggle.com/c/the-allen-ai-science-challenge&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/the-allen-ai-science-challenge&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="31151" LastEditorUserId="29575" LastEditDate="2018-08-18T03:11:08.190" LastActivityDate="2018-08-18T03:11:08.190" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18654" PostTypeId="2" ParentId="758" CreationDate="2017-04-28T02:32:51.470" Score="5" Body="&lt;p&gt;&lt;strong&gt;DISCLAIMER: I work at a company, &lt;a href=&quot;http://datmo.io&quot; rel=&quot;noreferrer&quot;&gt;Datmo&lt;/a&gt;, that creates an open-source tool to do this.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best practice for reproducibility is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) First containerize your environment into a Docker environment by creating a Dockerfile and ensuring that all dependencies are covered in that file. I found this resource to be best (&lt;a href=&quot;https://arxiv.org/pdf/1410.0846.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1410.0846.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Once you have that you'll want to decide where you can keep track of all of the performance metrics and configurations (so that you can revisit it for future experimentation) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Finally, write some documentation so that a new experimenter/developer is able to revisit your code, replicate it with the environment and see where you have kept your configurations and performance metrics.&lt;/p&gt;&#xA;" OwnerUserId="30099" LastActivityDate="2017-04-28T02:32:51.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18655" PostTypeId="2" ParentId="11619" CreationDate="2017-04-28T03:13:29.370" Score="24" Body="&lt;p&gt;Difference between CNN and RNN are as follows :&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;CNN:&lt;/h1&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;CNN take a fixed size input and generate fixed-size outputs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CNN is a type of feed-forward artificial neural network - are variations of multilayer perceptrons which are designed to use minimal amounts of preprocessing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CNNs use connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CNNs are ideal for images and videos processing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h1&gt;RNN:&lt;/h1&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;RNN can handle arbitrary input/output lengths.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RNN, unlike feedforward neural networks, can use their internal memory to process arbitrary sequences of inputs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recurrent neural networks use time-series information (i.e. what I spoke last will impact what I will speak next.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RNNs are ideal for text and speech analysis.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30986" LastEditorUserId="36938" LastEditDate="2018-10-15T19:47:57.897" LastActivityDate="2018-10-15T19:47:57.897" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18656" PostTypeId="2" ParentId="12528" CreationDate="2017-04-28T06:09:22.403" Score="1" Body="&lt;p&gt;Latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. LDA does exactly the same thing you wanted to do.&lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2017-04-28T06:09:22.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18663" PostTypeId="1" AcceptedAnswerId="18669" CreationDate="2017-04-28T13:12:01.817" Score="0" ViewCount="2244" Body="&lt;p&gt;I'm using caffenet for fine-tuning. &#xA;I'm doing cross validation (15 vs all) with a very small data set of about 250 images. I'm testing every 10 iterations (~2 epochs). My batch size is 50. With some sets I'm getting very unstable accuracy - Can jump from 70% to 90% and back to 70% and back and fourth. My question is: Let's say I hit 90% accuracy after 40 iterations (~8 epochs) - Does this mean that the net had reached an optimal state or could it be that it just had a lucky guess on the validation set? My final question is: Should I stop training and save the net? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="30182" LastEditorUserId="30182" LastEditDate="2017-05-01T10:05:56.080" LastActivityDate="2017-05-01T10:05:56.080" Title="Unstable accuracy of CNN - When should I stop training?" Tags="|neural-network|classification|gradient-descent|accuracy|overfitting|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18664" PostTypeId="1" CreationDate="2017-04-28T14:51:18.340" Score="5" ViewCount="4165" Body="&lt;p&gt;I am building a deep learning model for NLP. I am pretty comfortable with adding word embedding from word2vec or Glove vectors as extra word features but I wanted to add other word features like POS tag of a word, NER tag of word along with embedding as features. How can I do this. Should I give these word features by concatenating their vector with the word vectors. Or is there some other method. Please suggest.&lt;/p&gt;&#xA;" OwnerUserId="27873" LastActivityDate="2019-09-16T21:01:10.363" Title="How to add extra word features other then word Embedding in Recurrent Neural Network model" Tags="|deep-learning|nlp|rnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18665" PostTypeId="2" ParentId="18494" CreationDate="2017-04-28T17:10:41.033" Score="1" Body="&lt;p&gt;After rubber ducking with a friend of mine, I decided to take a two stage approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, whenever I see a word I've recognized as a multiple use word, I flag it in the system and give it the appropriate correction. Once it evaluates that part, it stores the corrected word and enough words such that there is sufficient context in a tuple... inside a list... inside a dictionary. It all looks something like &lt;code&gt;use_case = {'abbr': ('five word abbr context string', 'correction')&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, I've created a dictionary of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn LinearSVCs&lt;/a&gt; that looks like &lt;code&gt;svm_dict = {'abbr': LinearSVC}&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once these things are established and the program is run again, when it finds one of the multiple usage words it first checks &lt;code&gt;use_case&lt;/code&gt; to see if that exact set of 5 words surround this case. If it does it swaps it out for the original correction, otherwise it tries to predict the usage with the appropriate SVC and asks the user if the prediction is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it's not fully automated, but neither is the program in which it runs. I figured the occasional yes/no question to the user was easier than leaving it off the lists and rewriting it every... single... time...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, I have it all in place but haven't gotten to the point where it's being used yet. So, I'll either be editing, deleting, or accepting this answer sometime next week when I've had some time to test'r out.&lt;/p&gt;&#xA;" OwnerUserId="31399" LastActivityDate="2017-04-28T17:10:41.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18667" PostTypeId="1" AcceptedAnswerId="41128" CreationDate="2017-04-29T00:38:12.613" Score="11" ViewCount="6872" Body="&lt;p&gt;PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I simply change the activation function from sigmoid to relu, it drops to less than 50%. Is there a theoretical reason why this happens?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found the following example online:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.datasets import mnist&#xA;from keras.models import Sequential &#xA;from keras.layers.core import Dense, Activation&#xA;from keras.utils import np_utils&#xA;&#xA;(X_train, Y_train), (X_test, Y_test) = mnist.load_data()&#xA;&#xA;X_train = X_train.reshape(60000, 784)     &#xA;X_test = X_test.reshape(10000, 784)&#xA;&#xA;Y_train = np_utils.to_categorical(Y_train, classes)     &#xA;Y_test = np_utils.to_categorical(Y_test, classes)&#xA;&#xA;batch_size = 100      &#xA;epochs = 15&#xA;&#xA;model = Sequential()     &#xA;model.add(Dense(100, input_dim=784)) &#xA;model.add(Activation('sigmoid'))     &#xA;model.add(Dense(10)) &#xA;model.add(Activation('softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')&#xA;&#xA;model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)&#xA;&#xA;score = model.evaluate(X_test, Y_test, verbose=1)&#xA;print('Test accuracy:', score[1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that? &lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2018-03-17T19:17:30.677" LastActivityDate="2020-08-09T09:54:28.697" Title="ReLU vs sigmoid in mnist example" Tags="|python|neural-network|keras|image-classification|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18668" PostTypeId="1" AcceptedAnswerId="18686" CreationDate="2017-04-29T00:49:01.570" Score="1" ViewCount="204" Body="&lt;p&gt;I'm modeling a regression problem. An initial attempt yields the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labels.mean(): 0.00018132978443886167&#xA;labels.std(): 0.013450786078937208&#xA;&#xA;predictions.mean(): 0.0005549060297198594&#xA;predictions.std(): 0.00430255476385355&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the mean is off, and the standard deviation is totally different. I wonder what does it indicate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess: does it mean that my features are not discriminative enough, so that the model see examples w/ positive and negative labels alike, hence the small variance in the output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm running the regression using &lt;code&gt;XGBRegressor&lt;/code&gt;, with early-stopping. I have 1M training examples, 100K validation examples (for early-stopping), and another 100K for testing purpose (for which the mean and the standard deviation are shown above).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also checked that the label distribution of the three sets are mostly basically the same.&lt;/p&gt;&#xA;" OwnerUserId="29428" LastActivityDate="2017-04-30T05:53:15.693" Title="Regression yields much smaller standard deviation and the mean is off, what could be wrong?" Tags="|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18669" PostTypeId="2" ParentId="18663" CreationDate="2017-04-29T01:03:16.880" Score="1" Body="&lt;p&gt;If accuracy regresses something is wrong in either the network, or (more likely here) the meta-parameters (probably learning rate.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be difficult to tell when a model converges. I'd recommend looking at diagnostic graphs (typically training loss, training/validation accuracy, and ratio of weights:updates) over epochs. Typically convergence is considered when loss and accuracy level out and show diminishing returns beyond some threshold (your tolerance for 1.0e-x% improvements.) So, stop training/validation when it's improving less than what you care about. &lt;/p&gt;&#xA;" OwnerUserId="30675" LastActivityDate="2017-04-29T01:03:16.880" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18670" PostTypeId="1" AcceptedAnswerId="18761" CreationDate="2017-04-29T01:58:47.287" Score="8" ViewCount="1450" Body="&lt;p&gt;I am currently studying this &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;, in which CNN is applied for phoneme recognition using visual representation of log mel filter banks, and limited weight sharing scheme. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The visualisation of log mel filter banks is a way representing and normalizing the data. They suggest to visualize as a spectogram with RGB colors, which the closest I could come up with would be to plot it using &lt;code&gt;matplotlibs&lt;/code&gt; colormap &lt;code&gt;cm.jet&lt;/code&gt;. They (being the paper) also suggest each frame should be stacked with its  [static delta delta_delta] &#xA;filterbank energies. This looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o0qpt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o0qpt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input of the consist of an image patch of 15 frames set [static delta delta_detlta] input shape would be (40,45,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The limited weight sharing consist of limiting the weight sharing to a specific filter bank area, as speech is interpreted differently in different frequency area, thus will a full weight sharing as normal convolution apply, would not work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their implementation of limited weight sharing consist of controlling the weights  in the weight matrix associated with each convolutional layer. So they apply a convolution on the complete input. &#xA;The paper only applies only one convolutional layer as using multiple would destroy the locality of the feature maps extracted from the convolutional layer. &#xA;The reason why they use filter bank energies rather than the normal MFCC coefficient&#xA;is because DCT destroys the locality of the filter banks energies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MaZ2h.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MaZ2h.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of controlling the weight matrix associated with convolution layer, I choose to implement the CNN with multiple inputs. so each input consist of a (small filter bank range, total_frames_with_deltas, 3). So for instance the paper state that a filter size of 8 should be good, so I decided a filter bank range of 8. So each small image patch is of size (8,45,3). Each of the small image patch is extracted with a sliding window of with a stride of 1 - so there is a lot of overlap between each input - and each input has its own convolutional layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RuPsv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RuPsv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(input_3 , input_3, input3, should have been input_1, input_2, input_3 ...)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing this way makes it possible to use multiple convolutional layer, as the locality is not a problem any more, as it applied inside a filter bank area, this is my theory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper don't explicitly state it but i guess the reason why they do phoneme recognition on multiple frames is to have some some of left context and right context, so only the middle frame is being predicted/trained for. So in my case is the first 7 frames set the left context window - the middle frame is being trained for and last 7 frames set is the right context window.  So given multiple frames, will only one phoneme be recognised being the middle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My neural network currently looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def model3():&#xA;&#xA;    #stride = 1&#xA;    #dim = 40&#xA;    #window_height = 8&#xA;    #splits = ((40-8)+1)/1 = 33&#xA;    next(test_generator())&#xA;    next(train_generator(batch_size))&#xA;&#xA;    kernel_number = 200#int(math.ceil(splits))&#xA;    list_of_input = [Input(shape = (window_height,total_frames_with_deltas,3)) for i in range(splits)]&#xA;    list_of_conv_output = []&#xA;    list_of_conv_output_2 = []&#xA;    list_of_conv_output_3 = []&#xA;    list_of_conv_output_4 = []&#xA;    list_of_conv_output_5 = []&#xA;    list_of_max_out = []&#xA;    for i in range(splits):&#xA;        #list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (15,6))(list_of_input[i]))&#xA;        #list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (window_height-1,3))(list_of_input[i]))&#xA;        list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (window_height,3), activation = 'relu')(list_of_input[i]))&#xA;        list_of_conv_output_2.append(Conv2D(filters = kernel_number , kernel_size = (1,5))(list_of_conv_output[i]))&#xA;        list_of_conv_output_3.append(Conv2D(filters = kernel_number , kernel_size = (1,7))(list_of_conv_output_2[i]))&#xA;        list_of_conv_output_4.append(Conv2D(filters = kernel_number , kernel_size = (1,11))(list_of_conv_output_3[i]))&#xA;        list_of_conv_output_5.append(Conv2D(filters = kernel_number , kernel_size = (1,13))(list_of_conv_output_4[i]))&#xA;        #list_of_conv_output_3.append(Conv2D(filters = kernel_number , kernel_size = (3,3),padding='same')(list_of_conv_output_2[i]))&#xA;        list_of_max_out.append((MaxPooling2D(pool_size=((1,11)))(list_of_conv_output_5[i])))&#xA;&#xA;    merge = keras.layers.concatenate(list_of_max_out)&#xA;    print merge.shape&#xA;    reshape = Reshape((total_frames/total_frames,-1))(merge)&#xA;&#xA;    dense1 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_1&quot;)(reshape)&#xA;    dense2 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;    dense3 = Dense(units = 145 , activation = 'softmax', name = &quot;dense_3&quot;)(dense2)&#xA;    #dense4 = Dense(units = 1, activation = 'linear', name = &quot;dense_4&quot;)(dense3)&#xA;&#xA;&#xA;    model = Model(inputs = list_of_input , outputs = dense3)&#xA;    model.compile(loss=&quot;categorical_crossentropy&quot;, optimizer=&quot;SGD&quot; , metrics = [metrics.categorical_accuracy])&#xA;&#xA;    reduce_lr=ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1, mode='auto', epsilon=0.001, cooldown=0)&#xA;    stop  = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')&#xA;&#xA;    print model.summary()&#xA;&#xA;    raw_input(&quot;okay?&quot;)&#xA;    hist_current = model.fit_generator(train_generator(batch_size),&#xA;                        steps_per_epoch=10,&#xA;                        epochs = 10000,&#xA;                        verbose = 1,&#xA;                        validation_data = test_generator(),&#xA;                        validation_steps=1)&#xA;                        #pickle_safe = True,&#xA;                        #workers = 4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So.. now comes the issue..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I been training the network and have only been able to get a validation_accuracy of highest being 0.17, and the accuracy after a lot of epochs end up being 1.0. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/Kk1pC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Kk1pC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;(Plot is currently being made)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fixed frame:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8i3LP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8i3LP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;(plot being still made)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure why I am not getting better results.. Why this high error rate?&#xA;I am using the TIMIT dataset which the other ones also use.. so why am I getting worse results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And sorry for the long post - hope more information of my design decision could be useful - and help understand how I understood the paper versus how i've applied would help pinpoint where my mistake would be. &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T02:42:49.363" LastActivityDate="2021-02-16T02:42:49.363" Title="CNN for phoneme recognition" Tags="|classification|keras|training|convolutional-neural-network|audio-recognition|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18671" PostTypeId="1" CreationDate="2017-04-29T03:53:24.780" Score="1" ViewCount="49" Body="&lt;p&gt;As someone who is about to finish a bachelor in Software Engineering still learning the basics of machine learning but would like to enroll in a master's course focused on machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would you recommend to learn before considering applying for a master?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to hear experiences of people who have gone through ML or computer science with a focus on ML or have taught at that level, what is the expected knowledge you are suppose to have?&lt;/p&gt;&#xA;" OwnerUserId="31688" LastEditorUserId="21" LastEditDate="2017-05-19T16:08:28.027" LastActivityDate="2017-05-19T16:08:28.027" Title="What would you recommend to know before considering applying for a master in Machine Learning?" Tags="|self-study|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18674" PostTypeId="1" CreationDate="2017-04-29T12:33:47.360" Score="1" ViewCount="2833" Body="&lt;p&gt;I am trying to compute the total number of parameters in Ilya Sutskever's &lt;a href=&quot;https://arxiv.org/pdf/1409.3215.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;seq2seq&lt;/a&gt; model.&#xA;In the paper, the authors mention a total number of 384M trainable parameters, or to be exact:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In short, their model has:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;one encoder LSTM and one decoder LSTM&lt;/li&gt;&#xA;&lt;li&gt;each LSTM has four layers&lt;/li&gt;&#xA;&lt;li&gt;each LSTM layer has 1000 units&lt;/li&gt;&#xA;&lt;li&gt;input: 1000-dimensional word embedding of 160K possible input words&lt;/li&gt;&#xA;&lt;li&gt;output: naive softmax over 80K possible words&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Following &lt;a href=&quot;https://datascience.stackexchange.com/questions/10615/number-of-parameters-in-an-lstm-model&quot;&gt;this post&lt;/a&gt; and using the formula &#xA;$$ &#xA;num~params = 4 (nm + n^2)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to compute:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;For the encoder LSTM&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;n = 1000 for all layers&lt;/li&gt;&#xA;&lt;li&gt;m = 1000 for the first hidden layer, because of 1000 dimensional embedding&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;m = 1000 for all other layers, because that's the number of units in each hidden layer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which yields:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;input -&gt; first hidden layer: 8M&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;first -&gt; second            : 8M&lt;/li&gt;&#xA;&lt;li&gt;second -&gt; third            : 8M&lt;/li&gt;&#xA;&lt;li&gt;third -&gt; last              : 8M&lt;/li&gt;&#xA;&lt;li&gt;total                      = 32M&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;which I am guessing is what they mean when they say 32M recurrent connections for the LSTM?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;For the decoder LSTM&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;computations are exactly the same as above, with the caveat that the input size is 1000 because that's the output size of the encoder LSTM.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;total                      = 32M&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Where are the other 320M parameters?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The computations made so far account for 64M parameters.&#xA;We still haven't accounted for the final softmax layer, which admittedly could be pretty large since its output is 80K, and 320M can be suspiciously decomposed as 80K * 4K, but I'm not sure about this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;can the naive softmax layer alone account for the 320M missing parameters?&lt;/li&gt;&#xA;&lt;li&gt;shouldn't we also be using the fact that this is a recurrent neural networks to compute additional weights, as in &lt;a href=&quot;https://datascience.stackexchange.com/questions/10589/number-of-parameters-in-an-rnn&quot;&gt;this post&lt;/a&gt;?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="31723" LastActivityDate="2017-05-02T13:34:18.000" Title="Number of parameters in seq2seq model" Tags="|machine-learning|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18676" PostTypeId="1" CreationDate="2017-04-29T15:46:22.493" Score="2" ViewCount="65" Body="&lt;p&gt;I'm now transitioning to data scientist as bioinformatics PhD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What fields need lots of data scientists? Or offer more opportunities?&#xA;I guess business/finance and internet?&lt;/p&gt;&#xA;" OwnerUserId="31590" LastEditorUserId="27768" LastEditDate="2017-05-04T09:29:06.800" LastActivityDate="2017-05-04T09:29:06.800" Title="What fields offer most data science job opporunities?" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ClosedDate="2017-05-01T16:15:38.657" ContentLicense="CC BY-SA 3.0" />
  <row Id="18677" PostTypeId="2" ParentId="18626" CreationDate="2017-04-29T16:23:59.977" Score="5" Body="&lt;p&gt;Look at the CUDA compute capability. They are a mixture of hardware and software features a GPU has (&lt;a href=&quot;http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities&quot; rel=&quot;nofollow noreferrer&quot;&gt;see guide&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I benchmarked the GTX 1070, Titan Black, GTX 970, GTX 980, GTX 980Ti. The numbers can be found in &lt;a href=&quot;https://arxiv.org/abs/1707.09725&quot; rel=&quot;nofollow noreferrer&quot;&gt;my masters thesis&lt;/a&gt; (Table 5.3 and Table 5.16), but the gist is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;GTX 1070 is by far the fastest&lt;/li&gt;&#xA;&lt;li&gt;GTX 980 and 980 Ti are pretty much the same, but only ~half as fast as the GTX 1070,&lt;/li&gt;&#xA;&lt;li&gt;GTX 970 needs 1.5 the time of the 980&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You should look at how much memory you need. This is sometimes a limiting factor.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-09-06T07:22:29.133" LastActivityDate="2017-09-06T07:22:29.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18679" PostTypeId="1" CreationDate="2017-04-29T16:50:48.953" Score="0" ViewCount="73" Body="&lt;p&gt;I want to study in field of &quot; Detecting fake account on online social network by content/sentiment analysis &quot; for Master's Thesis.is this possible?&lt;/p&gt;&#xA;" OwnerUserId="31731" LastActivityDate="2017-05-01T14:00:01.430" Title="Can I use sentiment analysis techniques and text processing for detection social bot in online social networks?" Tags="|machine-learning|social-network-analysis|" AnswerCount="1" CommentCount="0" ClosedDate="2017-05-01T16:15:33.667" ContentLicense="CC BY-SA 3.0" />
  <row Id="18683" PostTypeId="2" ParentId="18676" CreationDate="2017-04-29T21:25:10.897" Score="1" Body="&lt;p&gt;What difference does it make? Nothing stops you from applying to all of them. The real question is, can you and do you want to write software or do you just want to analyze data? Pay attention to the job descriptions in this regard. Data analysts are typically expected to be strong in classical statistics, while machine engineers are expected to know ... machine learning. Then every field has certain problems that you need to be familiar with before taking the technical interview, such as recommender systems for media companies, and time series forecasting for business/finance. You can find out what they are by reading around and actually applying to companies. Expect several rounds of interviews, including mathematical and programming challenges (real-time or off-line), and questions about your research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once again, every field needs data scientists, so this is not a criterion.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-04-29T21:25:10.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18684" PostTypeId="1" AcceptedAnswerId="18692" CreationDate="2017-04-29T21:25:47.450" Score="0" ViewCount="187" Body="&lt;p&gt;Customer often send currupt data for analysis. I spent a lot of time in cleaning the data or waiting for a correct dataset.&#xA;Can you recommend a tool that can handle the most common curruptions (eg. wrong set quotes)...&lt;/p&gt;&#xA;" OwnerUserId="31719" LastActivityDate="2017-04-30T23:43:59.503" Title="What tools do you use to clean corrupt data?" Tags="|dataset|data-cleaning|" AnswerCount="2" CommentCount="1" ClosedDate="2017-05-01T16:15:50.623" ContentLicense="CC BY-SA 3.0" />
  <row Id="18685" PostTypeId="2" ParentId="18684" CreationDate="2017-04-30T05:21:22.607" Score="0" Body="&lt;p&gt;If you know the nature of data you require as suggested by @Emre for manual cleaning you can use Apache Spark or Pandas. Utilizing the data frame aspects of both Spark and Pandas will help you preprocess data with a much easier convenience. Because data frames account for any data type it's easy to access and clean specifically the segments you require. If the corrupt data is of repetitive nature you can use several Map reduce techniques to clean it fast as well. If it's the case of big data.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-04-30T05:21:22.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18686" PostTypeId="2" ParentId="18668" CreationDate="2017-04-30T05:53:15.693" Score="3" Body="&lt;p&gt;The difference in standard deviation is nothing suspicious.  It is only to be expected, if you have a weak correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the regression is &lt;code&gt;y ~ x&lt;/code&gt;, i.e., &lt;code&gt;y = ax + b&lt;/code&gt;.  Suppose that &lt;code&gt;x&lt;/code&gt; only explains a small fraction of the variability in &lt;code&gt;y&lt;/code&gt;, i.e., &lt;code&gt;y&lt;/code&gt; is scattered all over the place and the least regression line only weakly fits the data.  Suppose also that the line is nearly horizontal (i.e., &lt;code&gt;a&lt;/code&gt; is small).  Then the standard deviation of the predicted &lt;code&gt;y&lt;/code&gt;-values will be small (since the line is nearly horizontal) but the standard deviation of the actual &lt;code&gt;y&lt;/code&gt;-values might be large.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But really, the way to figure out what is going on is to &lt;em&gt;visualize the data&lt;/em&gt;.  You should always start by visualizing the data.  Plot a scatterplot, and superimpose the least squares fit line on top of it.  I bet you'll immediately have a better sense of what might be going on.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-04-30T05:53:15.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18687" PostTypeId="1" AcceptedAnswerId="18732" CreationDate="2017-04-30T07:14:00.260" Score="2" ViewCount="6914" Body="&lt;p&gt;I have a data frame like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;timestamp                  vote_count&#xA;00:12:30                   350 &#xA;05:05:00                   50 &#xA;01:10:35                   200 &#xA;05:05:00                   175 &#xA;..............................&#xA;..............................&#xA;00:12:30                   900&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now I want to visualize the vote_count for the timestamps and do some analysis on that further. How can I plot the two columns against each other using matplotlib or seaborn?&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: The timestamp is in 24hr format. Also, at any timestamp, there can be multiple vote counts.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-24T21:25:22.770" LastActivityDate="2020-12-24T21:25:22.770" Title="Plotting two pandas data frame columns against each other" Tags="|python|pandas|visualization|dataframe|data-analysis|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="18690" PostTypeId="1" AcceptedAnswerId="18790" CreationDate="2017-04-30T13:14:48.587" Score="3" ViewCount="1978" Body="&lt;p&gt;I am currently looking for a way in which a network with multiple inputs can optimise its hyper parameter... scikit-learn has gridsearch CV but Keras only supports single inputs using the scikit-wrapper.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Emre -  Minimal working example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train_generator():&#xA;    while True:&#xA;        train_input = np.random.randint(100,size=(1,33,8,45,3))&#xA;        train_input_list = np.split(train_input,33,axis=1)&#xA;&#xA;        for i in range(len(train_input_list)):&#xA;            train_input_list[i] = train_input_list[i].reshape(1,8,45,3)&#xA;&#xA;        train_output = np.random.randint(100,size=(1,3,50))&#xA;        yield (train_input_list, train_output)&#xA;&#xA;def test_generator():&#xA;    while True:&#xA;        test_input = np.random.randint(100,size=(1,33,8,45,3))&#xA;        test_input_list = np.split(test_input,33,axis=1)&#xA;&#xA;        for i in range(len(test_input_list)):&#xA;            test_input_list[i] = test_input_list[i].reshape(1,8,45,3)&#xA;&#xA;        test_output = np.random.randint(100,size=(1,3,50))&#xA;&#xA;        yield (test_input_list, test_output)&#xA;&#xA;def data():&#xA;    X_train, Y_train = next(train_generator())&#xA;    X_test, Y_test = next(test_generator())&#xA;&#xA;    datagen = ImageDataGenerator()&#xA;    train_list = []&#xA;    for input in X_train:&#xA;        train_list.append(datagen.fit(input))&#xA;&#xA;    return datagen, train_list, Y_train, X_test, Y_test&#xA;&#xA;def fws(datagen, X_train, Y_train, X_test, Y_test):&#xA;    #Input shape: (batch_size,40,45,3)&#xA;    #output shape: (1,15,50)&#xA;    # number of unit in conv_feature_map = splitd&#xA;    filter_size = 8&#xA;    pooling_size = 28&#xA;    stride_step = 2&#xA;    pool_splits = ((splits - pooling_size)+1)/2&#xA;    temp_list = []&#xA;    sun_temp_list = []&#xA;    conv_featur_map = []&#xA;    pool_feature_map = []&#xA;    print &quot;Printing shapes&quot;&#xA;&#xA;&#xA;    list_of_input = [Input(shape = (8,45,3)) for i in range(33)]&#xA;&#xA;&#xA;    #convolution&#xA;    shared_conv = Conv2D(filters = 150, kernel_size = (filter_size,45), activation='relu')&#xA;    for i in range(splits):&#xA;        conv_featur_map.append(shared_conv(list_of_input[i]))&#xA;&#xA;    #Pooling&#xA;    input = Concatenate()(conv_featur_map)&#xA;    input = Reshape((splits,-1))(input)&#xA;    pooled = MaxPooling1D(pool_size = pooling_size, strides = stride_step)(input)&#xA;&#xA;    #reshape = Reshape((3,-1))(pooled)&#xA;&#xA;    #fc&#xA;    dense1 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_1&quot;)(pooled)&#xA;    dense2 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;    dense3 = Dense(units = 50 , activation = 'softmax', name = &quot;dense_3&quot;)(dense2)&#xA;&#xA;&#xA;    model = Model(inputs = list_of_input , outputs = dense3)&#xA;    sgd = keras.optimizers.SGD(lr = {{uniform(0, 1)}}, decay = {{uniform(0, 1)}}, momentum = {{uniform(0, 1)}}, nesterov = True)&#xA;    model.compile(loss=&quot;categorical_crossentropy&quot;, optimizer=sgd , metrics = [metrics.categorical_accuracy])&#xA;    #log= CSVLogger('/home/keerthikan/kaldi-trunk/dnn/original/training_total_frames_'+str(total_frames)+&quot;_dim_&quot;+str(dim)+&quot;_window_height_&quot;+str(window_height)+&quot;_batch_&quot;+str(batch)+&quot;_lr_start_&quot;+str(lr)+&quot;_decay_&quot;+str(decay)+&quot;_mom_&quot;+str(momentum)+&quot;_fws.csv&quot;)&#xA;    #plot_model(model, to_file='model.png')&#xA;&#xA;&#xA;    hist_current = model.fit_generator(datagen.flow(X_train, Y_train),&#xA;                        steps_per_epoch=32,&#xA;                        epochs = 1000,&#xA;                        verbose = 1,&#xA;                        validation_data = (X_test, Y_test),&#xA;                        validation_steps=32,&#xA;                        pickle_safe = True,&#xA;                        workers = 4)&#xA;&#xA;    score, acc = model.evaluate(X_test, Y_test, verbose=0)&#xA;&#xA;    return {'loss': -acc, 'status': STATUS_OK, 'model': model}&#xA;&#xA;if __name__ == '__main__':&#xA;&#xA;    datagen, X_train, Y_train, X_test, Y_test = data()&#xA;&#xA;    best_run, best_model = optim.minimize(model=fws,&#xA;                                          data=data,&#xA;                                          algo=tpe.suggest,&#xA;                                          max_evals=5,&#xA;                                          trials=Trials())&#xA;&#xA;    print(&quot;Evalutation of best performing model:&quot;)&#xA;    print(best_model.evaluate(X_test, Y_test)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And full error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;minimal_hyperas.py&quot;, line 167, in &amp;lt;module&amp;gt;&#xA;    trials=Trials())&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/optim.py&quot;, line 43, in minimize&#xA;    notebook_name=notebook_name, verbose=verbose)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/optim.py&quot;, line 63, in base_minimizer&#xA;    model_str = get_hyperopt_model_string(model, data,functions,notebook_name, verbose, stack)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/optim.py&quot;, line 130, in get_hyperopt_model_string&#xA;    imports = extract_imports(cleaned_source, verbose)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/utils.py&quot;, line 44, in extract_imports&#xA;    import_parser.visit(tree)&#xA;  File &quot;/usr/lib/python2.7/ast.py&quot;, line 241, in visit&#xA;    return visitor(node)&#xA;  File &quot;/usr/lib/python2.7/ast.py&quot;, line 249, in generic_visit&#xA;    self.visit(item)&#xA;  File &quot;/usr/lib/python2.7/ast.py&quot;, line 241, in visit&#xA;    return visitor(node)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/utils.py&quot;, line 14, in visit_Import&#xA;    if (self._import_asnames(node.names)!=''):&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/hyperas/utils.py&quot;, line 36, in _import_asnames&#xA;    return ''.join(asname)&#xA;TypeError: sequence item 0: expected string, NoneType found&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="25429" LastEditDate="2017-05-06T22:55:27.300" LastActivityDate="2017-05-06T22:55:27.300" Title="Efficient way to optimise hyper parameter for network with multiple inputs?" Tags="|keras|optimization|hyperparameter|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18692" PostTypeId="2" ParentId="18684" CreationDate="2017-04-30T23:37:37.913" Score="0" Body="&lt;p&gt;Weka has built in preprocessing techniques &#xA;Also u may need to check the powerful tool named dataPreparator as it provides a variety of techniques for data cleaning, transformation, and exploration&#xA;Chaining of preprocessing operators into a flow graph (operator tree) and it can&#xA;Handle large volumes of data (since data sets are not stored in the computer memory) through a user friendly graphical user interface&lt;/p&gt;&#xA;" OwnerUserId="31722" LastEditorUserId="31722" LastEditDate="2017-04-30T23:43:59.503" LastActivityDate="2017-04-30T23:43:59.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18695" PostTypeId="1" CreationDate="2017-05-01T02:28:06.313" Score="3" ViewCount="556" Body="&lt;p&gt;I am trying to understand the training phase of the tutorial &lt;a href=&quot;https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Using Keras and Deep Deterministic Policy Gradient to play TORCS&lt;/a&gt; (&lt;a href=&quot;https://web.archive.org/web/20161213122738/https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;mirror&lt;/a&gt;, &lt;a href=&quot;https://github.com/yanpanlau/DDPG-Keras-Torcs&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt;) by Ben Lau published on October 11, 2016.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tutorial says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Then the actor policy is updated using the sampled policy gradient:&#xA;  $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which in the code corresponds to: &lt;code&gt;actor.train(states, grads)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://github.com/yanpanlau/DDPG-Keras-Torcs/blob/572b92cff2e3b7be3b84e92f83a65c06c6fbad1a/ActorNetwork.py#L33&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;actor.train()&lt;/code&gt;&lt;/a&gt; method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did &lt;a href=&quot;https://github.com/yanpanlau/DDPG-Keras-Torcs/blob/572b92cff2e3b7be3b84e92f83a65c06c6fbad1a/ActorNetwork.py#L28&quot; rel=&quot;nofollow noreferrer&quot;&gt;read&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;self.action_gradient&lt;/code&gt; corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and &lt;code&gt;tf.gradients(self.model.output, self.weights)&lt;/code&gt; corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no  multiplication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-05-01T02:34:11.110" LastActivityDate="2017-05-01T18:39:48.757" Title="Understanding the training phase of the tutorial &quot;Using Keras and Deep Deterministic Policy Gradient to play TORCS&quot; tutorial" Tags="|neural-network|deep-learning|tensorflow|reinforcement-learning|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18696" PostTypeId="2" ParentId="17506" CreationDate="2017-05-01T05:03:10.443" Score="1" Body="&lt;p&gt;I would separate the creation of the data from the plotting operation, if I were you. &lt;code&gt;ggplot2::facet_wrap&lt;/code&gt; allows you to plot each of the cumulative distributions in each own panel. Here is how I would do it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pgammaX &amp;lt;- function(x) pgamma(x, shape = 64.57849, scale = 0.08854802)&#xA;&#xA;Gamma &amp;lt;- pgammaX(seq(from=4, to = 8, by = 0.1))&#xA;Normal &amp;lt;- pnorm(seq(from=-4, to = 3.9, by = 0.1))&#xA;&#xA;data &amp;lt;- data_frame(&#xA;  x = seq(-4,8,by=0.1),&#xA;  y = c(Normal, Gamma),&#xA;  # use factor to make the plot for Normal to appear first&#xA;  distribution = factor(c(rep(c(&quot;Normal&quot;, &quot;Gamma&quot;), &#xA;                          times = c(length(Normal), length(Gamma)))),&#xA;                          levels = c(&quot;Normal&quot;, &quot;Gamma&quot;))&#xA;)&#xA;&#xA;library(dplyr)    &#xA;library(ggplot2)&#xA;&#xA;data %&amp;gt;%  ggplot(aes(x, y)) + &#xA;  geom_line() + &#xA;  # use facet_wrap to plot each cummulative distribution in each own panel      &#xA;  facet_wrap(~distribution, scales = &quot;free_x&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e3Y7B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e3Y7B.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31775" LastActivityDate="2017-05-01T05:03:10.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18698" PostTypeId="2" ParentId="12017" CreationDate="2017-05-01T05:54:50.893" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://CRAN.R-project.org/package=GGally&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;GGally&lt;/code&gt;&lt;/a&gt; package, which is an extension of &lt;code&gt;ggplot2&lt;/code&gt;, might be what you need. Here are the explanations for the code below.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;upper = &quot;blank&quot;&lt;/code&gt; removes the plots on the upper right &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;lower = list(combo = &quot;box&quot;, continuous = &quot;blank&quot;)&lt;/code&gt; plots a boxplot for ever pair of variables where one is categorical and the other is continuous, but does not print the plot when both variables are categorical. (Try removing &lt;code&gt;continous = &quot;blank&quot;&lt;/code&gt; to see what happens.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;diag = NULL&lt;/code&gt; removes the plots in the diagonals.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You may tailor this for your particular use.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(GGally)&#xA;mtcars[1:4] %&amp;gt;% mutate(cyl = factor(cyl)) %&amp;gt;% &#xA;  ggpairs(upper = &quot;blank&quot;,&#xA;          lower = list(combo = &quot;box&quot;, continuous = &quot;blank&quot;),&#xA;          diag = NULL&#xA;          # remove the column labels, but why would you do that?&#xA;          # columnLabels = NULL)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dcN4J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dcN4J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31775" LastActivityDate="2017-05-01T05:54:50.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18701" PostTypeId="2" ParentId="17854" CreationDate="2017-05-01T08:23:36.763" Score="1" Body="&lt;p&gt;R+Hadoop themselves do not actually give you any massive direct benefit. You could use Hadoop streaming to run parallel R jobs across all the nodes on your Hadoop cluster, but that is reliant on your problem being inherently parallelisable. You need to take advantage of a distributed programming language which will let you run a single program over all the nodes on the Hadoop cluster. There is where something like &lt;em&gt;Apache Spark&lt;/em&gt; comes into play. It is important to be pedantic here, as Apache Spark is not INHERENTLY reliant on Hadoop (you can build and compile it standalone without any Hadoop features). Thus why some people argue it is a replacement FOR Hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Through the SparkR interface, you can access it directly from R just creating the necessary tools (i.e. importing &lt;code&gt;library(sparkR)&lt;/code&gt; if I recall correctly, then setting up your &lt;code&gt;sparkContext&lt;/code&gt; and &lt;code&gt;SQLContext&lt;/code&gt;). This then means, if properly set up, all you have to do to scale your code from 1 machine to 100 machines is add more nodes pointed to the &lt;code&gt;SparkMaster&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31154" LastActivityDate="2017-05-01T08:23:36.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18702" PostTypeId="2" ParentId="18442" CreationDate="2017-05-01T09:05:37.983" Score="2" Body="&lt;p&gt;In newer W2V version it is not enough just to write:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_name.train(sentences)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have to write inside &lt;code&gt;word_count&lt;/code&gt; or &lt;code&gt;total_examples&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I write:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;token_count = sum([len(sentence) for sentence in sentences])&lt;/code&gt;. And here is how I get sentences:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentences = []&#xA;for raw_sentence in raw_sentences:&#xA;&#xA;    if len(raw_sentence) &amp;gt; 0:&#xA;        sentences.append(sentence_to_wordlist(raw_sentence))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More documentation &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31785" LastEditorUserId="29575" LastEditDate="2018-03-06T13:28:07.993" LastActivityDate="2018-03-06T13:28:07.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18703" PostTypeId="2" ParentId="18679" CreationDate="2017-05-01T14:00:01.430" Score="1" Body="&lt;p&gt;Yes, because it has been done before:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2808779&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dl.acm.org/citation.cfm?id=2808779&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-01T14:00:01.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18704" PostTypeId="1" CreationDate="2017-05-01T15:04:59.837" Score="3" ViewCount="7942" Body="&lt;p&gt;I am studying &quot;Data Mining: Concepts and Techniques&quot; by Han, Kamber &amp;amp; Pei. In Chapter 12 &quot;Outlier Detection&quot;, they have stated that there are 3 types of outliers:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Global Outlier - deviates significantly from the rest of the data set&lt;/li&gt;&#xA;&lt;li&gt;Contextual Outlier - deviates significantly with respect to a specific context of the object&lt;/li&gt;&#xA;&lt;li&gt;Collective Outlier - the individual data objects may not be outliers, but the objects as a whole deviate significantly from the entire data set. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;According to the authors:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;“The temperature today is 28 C. Is it exceptional (i.e., an outlier)?” It depends, for example, on the time and location! If it is in winter in Toronto, yes, it is an outlier. If it is a summer day in Toronto, then it is normal. Unlike global outlier detection, in this case, whether or not today’s temperature value is an outlier depends on the context—the date, the location, and possibly some other factors. In a given data set, a data object is a contextual outlier if it deviates significantly with respect to a specific context of the object. Contextual outliers are also known as conditional outliers because they are conditional on the selected context. Therefore, in contextual outlier detection, the context has to be specified as part of the problem definition.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My question : Is not Contextual Outliers the same as Global Outliers where the original feature vector describing each data object may have to be extended to incorporate some other attributes (information regarding the &quot;context&quot;) ?&#xA;Obviously, if the data objects do not contain enough attributes, they can never be classified as outlier ! This holds true for global outliers as well. So it makes no sense to me to form a new category of outliers as Contextual Outliers.&lt;/p&gt;&#xA;" OwnerDisplayName="user31795" LastActivityDate="2020-10-29T13:31:27.540" Title="Difference between Global Outlier and Contextual Outlier?" Tags="|classification|data-mining|clustering|outlier|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18705" PostTypeId="2" ParentId="18695" CreationDate="2017-05-01T15:11:55.683" Score="3" Body="&lt;p&gt;Look at the &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/gradients&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation of &lt;code&gt;tf.gradients&lt;/code&gt;&lt;/a&gt; and you will see that the third parameter is used to weight the gradient calculation of first param wrt to second param. This is phrased in the documentation as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A user can provide their own initial grad_ys to compute the derivatives using a different initial gradient for each y (e.g., if one wanted to weight the gradient differently for each value in each y).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So it essentially uses the &lt;code&gt;grad_ys&lt;/code&gt; parameter as an element-wise multiplication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameter update gradient calculation in the project code is feeding in the already-calculated value of $\frac{\partial \mu(s|\theta)}{\partial \theta}$ as a placeholder, and using it for this third param.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the multiplication of the two gradient factors occurs &lt;em&gt;within&lt;/em&gt; &lt;code&gt;tf.gradients&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-01T18:39:48.757" LastActivityDate="2017-05-01T18:39:48.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18712" PostTypeId="1" AcceptedAnswerId="18714" CreationDate="2017-05-02T00:41:19.067" Score="-1" ViewCount="61" Body="&lt;p&gt;I have to build a classifier that classifies samples to one of thirteen classes, but the training dataset I have contains only 10 classes (the dataset is not balanced and some classes does not have any sample) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it right to build a Neural network classifier that has thirteen outputs although I don't have thirteen classes in my training set? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it affect the accuracy to have more classes in the output that the classes in the training set? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was considering to put the full number of classes just in case in the future I can retrain the model again with better dataset so I don't need to change the code of the classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&lt;/p&gt;&#xA;" OwnerUserId="30884" LastActivityDate="2017-05-02T06:55:10.987" Title="Number of outputs exceeds the number of classes in the training set" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18713" PostTypeId="1" AcceptedAnswerId="18717" CreationDate="2017-05-02T05:40:10.713" Score="1" ViewCount="247" Body="&lt;p&gt;We have a click-model which is currently being used for search ranking in production, and I want to create a new model which takes the old model click probability as one input and adds some other variables in too. The problem is that training data will positionally biased by the fact that the probability of a click is correlated to the old model's prediction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My plan is to introduce a penalty factor on the original model's prediction to ensure that it doesn't dominate the new model (eyeballing the results to decide on an appropriate penalty factor). Is this approach valid or would there be a better way to approach this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I don't want to rebuild the old model with the new variables because&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The existing model takes a long time (days) to build&lt;/li&gt;&#xA;&lt;li&gt;The new model and old model will be deployed separately, i.e. the old model will be scored offline/batch whereas the new model will be scored real-time&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10975" LastEditorUserId="10975" LastEditDate="2017-05-02T06:33:12.437" LastActivityDate="2017-05-02T09:09:40.503" Title="How to use an existing model as in input into a new model" Tags="|classification|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18714" PostTypeId="2" ParentId="18712" CreationDate="2017-05-02T06:55:10.987" Score="0" Body="&lt;p&gt;In theory, this should not do any harm to the accuracy of the trained network on the data that you have. However, of course the network will have no ability to predict the three classes it has not been shown. If it is well trained, it should predict close to zero probability for all three unseen classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be a reasonable structure to have all thirteen classes defined for a case where you continue to train online, as data arrives - although the network might have unknown problems adjusting to new classes when they appear, you will have no measure of its ability to do this. Also, if your goal is to produce a model for someone else's use and then leave it with some simple notes on how to re-train it, then it could make sense to have known future requirements already coded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you intend to re-train from scratch with new data, or if you will be building any new model when there is new data, then it is not strictly necessary to add unseen classes in advance. Depending on training time for your network, you might view any hyper-parameter in the code as something you can adjust quickly. If you write the code with a constant defining the number of classes, and refer that in all code that needs to know this number, then it should only take seconds to adapt your model - that's no time at all compared to time spent on other parts of the problem.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-02T06:55:10.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18715" PostTypeId="1" AcceptedAnswerId="18768" CreationDate="2017-05-02T07:52:39.083" Score="0" ViewCount="1255" Body="&lt;p&gt;I'm doing credit risk modelling and the data have large number of features.I am using boruta package for feature selection. The package is too computationally expensive, I cannot run it on the complete training dataset. What i'm trying to do is take a subset of the training data(let's say about 20-30%) and run the boruta package on that subsetted data and get the important features. But when i use random forest to train the data I have too use the full dataset. My question is, Is it right to select features only on a part of train data but then build the model on whole of training data?&lt;/p&gt;&#xA;" OwnerUserId="31822" LastEditorUserId="31822" LastEditDate="2017-05-02T10:38:24.527" LastActivityDate="2017-05-05T08:40:15.223" Title="Feature selection in R too large dataset" Tags="|feature-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18716" PostTypeId="2" ParentId="18713" CreationDate="2017-05-02T08:50:17.047" Score="1" Body="&lt;p&gt;This depends a  bit on the used case but you could also create the 2nd model independent and then use a third model to combine them. This has the advantage that there should be no leakage between the two models (if variables are independent) and you can merge them in a very controlled way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is unclear for me is what advantage you would have from taken input from a model that can not be used in the same context. (offline vs real time)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use them combined and Model A as input for B you would need to guess what the output for Model A is.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-05-02T08:50:17.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18717" PostTypeId="2" ParentId="18713" CreationDate="2017-05-02T09:09:40.503" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The problem is that training data will positionally biased by the fact that the probability of a click is correlated to the old model's prediction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It should be the case that many of your input variables are correlated in some way with the output, otherwise your model could not work. The main difference here is you are expecting a strong correlation from a single feature. This is not a problem - you could think of it as a complex form of feature engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are essentially &lt;a href=&quot;https://stats.stackexchange.com/questions/18891/bagging-boosting-and-stacking-in-machine-learning&quot;&gt;stacking the old model&lt;/a&gt; with some new variables which you hope are predictive. You &lt;em&gt;should probably&lt;/em&gt; in this case include all the existing/old variables so that the new model can more easily spot mistakes made by the old model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My plan is to introduce a penalty factor on the original model's prediction to ensure that it doesn't dominate the new model &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I doubt this would be useful. However the correct way to assess this plan is to try it and measure the performance compared to the simpler version without any penalty.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-02T09:09:40.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18718" PostTypeId="2" ParentId="18674" CreationDate="2017-05-02T13:34:18.000" Score="2" Body="&lt;p&gt;I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The reason for the remaining 320M parameters&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in the &lt;strong&gt;input of the encoder&lt;/strong&gt;, computing the word embeddings requires taking a one-hot vector of 160K words and transforming it into a 1K dimensional word embedding. Thus this accounts for &lt;strong&gt;160M&lt;/strong&gt; parameters;&lt;/li&gt;&#xA;&lt;li&gt;the &lt;strong&gt;softmax layer&lt;/strong&gt; of the decoder takes a 1K dimensional input (the hidden state of the last layer of LSTM cells) and transforms it into a vector of 80K probabilities. This accounts for &lt;strong&gt;80M&lt;/strong&gt; parameters;&lt;/li&gt;&#xA;&lt;li&gt;the &lt;strong&gt;input of the encoder&lt;/strong&gt; is given not only by the hidden state of the decoder, but also by the previous output of the encoder. This requires taking a 80K dimensional vector of possible words and transforming it into a 1K dimensional word embedding. Thus this accounts for &lt;strong&gt;80M&lt;/strong&gt; parameters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does this explanation make sense?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Recurrent connections&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As far as I understand, once the cell state of the LSTM is computed, it is transferred &lt;em&gt;as is&lt;/em&gt; to the next timestep. Thus there are &lt;strong&gt;no additional parameters&lt;/strong&gt; from the recurrent network (such as, e.g., was the case in this post).&lt;/p&gt;&#xA;" OwnerUserId="31723" LastActivityDate="2017-05-02T13:34:18.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18719" PostTypeId="1" CreationDate="2017-05-02T13:48:48.113" Score="2" ViewCount="225" Body="&lt;p&gt;Can anyone please tell me how is clustering is used in data management?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was recently asked this question and was left totally stumped.&lt;/p&gt;&#xA;" OwnerUserId="31831" LastEditorUserId="8432" LastEditDate="2017-06-05T19:30:20.687" LastActivityDate="2017-12-03T00:27:11.630" Title="How clustering is used in data management?" Tags="|data|management|" AnswerCount="1" CommentCount="7" ClosedDate="2017-12-03T14:04:49.017" ContentLicense="CC BY-SA 3.0" />
  <row Id="18722" PostTypeId="2" ParentId="13490" CreationDate="2017-05-02T20:07:54.823" Score="234" Body="&lt;p&gt;You could simply implement the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;class_weight&lt;/code&gt;&lt;/a&gt; from &lt;code&gt;sklearn&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Let's import the module first&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.utils import class_weight&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In order to calculate the class weight do the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class_weights = class_weight.compute_class_weight('balanced',&#xA;                                                 np.unique(y_train),&#xA;                                                 y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Thirdly and lastly add it to the model fitting&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(X_train, y_train, class_weight=class_weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Attention&lt;/strong&gt;: I edited this post and changed the variable name from &lt;em&gt;class_weight&lt;/em&gt; to &lt;em&gt;class_weight&lt;strong&gt;s&lt;/em&gt;&lt;/strong&gt; in order to not to overwrite the imported module. Adjust accordingly when copying code from the comments.&lt;/p&gt;&#xA;" OwnerUserId="31846" LastEditorUserId="41927" LastEditDate="2018-06-11T13:18:47.880" LastActivityDate="2018-06-11T13:18:47.880" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="18724" PostTypeId="2" ParentId="16994" CreationDate="2017-05-02T20:44:29.283" Score="3" Body="&lt;p&gt;There are many methods of performing this optimization- namely, choosing the optimal number of topics to supply for LDA and many papers have been authored on the topic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several of note, which each define metrics by which to evaluate LDA models for quality of topics are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rajkumar Arun, V. Suresh, C. E. Veni Madhavan, and M. N. Narasimha Murthy. 2010. On finding the natural number of topics with latent dirichlet allocation: Some observations. In Advances in knowledge discovery and data mining, Mohammed J. Zaki, Jeffrey Xu Yu, Balaraman Ravindran and Vikram Pudi (eds.). Springer Berlin Heidelberg, 391–402. &lt;a href=&quot;http://doi.org/10.1007/978-3-642-13657-3_43&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://doi.org/10.1007/978-3-642-13657-3_43&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Cao Juan, Xia Tian, Li Jintao, Zhang Yongdong, and Tang Sheng. 2009. A density-based method for adaptive lDA model selection. Neurocomputing — 16th European Symposium on Artificial Neural Networks 2008 72, 7–9: 1775–1781. &lt;a href=&quot;http://doi.org/10.1016/j.neucom.2008.06.011&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://doi.org/10.1016/j.neucom.2008.06.011&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Romain Deveaud, Éric SanJuan, and Patrice Bellot. 2014. Accurate and effective latent concept modeling for ad hoc information retrieval. Document numérique 17, 1: 61–84. &lt;a href=&quot;http://doi.org/10.3166/dn.17.1.61-84&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://doi.org/10.3166/dn.17.1.61-84&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences 101, suppl 1: 5228–5235. &lt;a href=&quot;http://doi.org/10.1073/pnas.0307752101&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://doi.org/10.1073/pnas.0307752101&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As luck would have it, if you're using R, these metrics have already been compiled for you in a convenient package called &lt;strong&gt;ldatuning&lt;/strong&gt; which provides a  set of utilities and metrics to help tune the correct number of topics within LDA models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you're using Python, the &lt;strong&gt;gensim&lt;/strong&gt; package can provide you with a lot of utilities to assist. For example, the package implements a metric that they call &quot;topic coherence&quot; that they claim corresponds roughly to &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/topic_coherence_tutorial.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;clarity of  topic distinction for a human&lt;/a&gt; among many &lt;a href=&quot;https://markroxor.github.io/gensim/tutorials/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;other utilities for tuning&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-05-02T20:44:29.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18727" PostTypeId="2" ParentId="15923" CreationDate="2017-05-03T09:28:33.827" Score="0" Body="&lt;p&gt;You could also try a bootstrapping method in which you make the fit to only a subset of data points. This subset is created by randomly picking some of the data points. By repeating this many times you can take the uncertainty from the distribution of the fit results. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-05-03T09:28:33.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18728" PostTypeId="2" ParentId="16027" CreationDate="2017-05-03T13:04:08.013" Score="2" Body="&lt;p&gt;$f$ is not defined for $\mathbb{R}^M$, so as well as reducing $x \in \mathbb{R}^N$ to $y \in \mathbb{R}^M$ you are creating a function approximator $g(y) \approx f(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural network autoencoder&lt;/a&gt; with N dimension inputs and M dimension &quot;bottleneck&quot; layer. You will need to scale inputs and outputs. Using a standard auto-encoder, you will have to measure the fraction of preserved variance in $f(x)$ afterwards. The auto-encoder is not &lt;em&gt;directly&lt;/em&gt; going to do that for you, instead it will, like PCA, attempt to use y to encode x (unlike PCA, it may do so nonlinearly). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;could&lt;/em&gt; take this further if $f(x)$ is differentiable. Instead of usual mean-squared error as the loss function, you could use $\frac{1}{k}\sum_{i=1}^{k} (f(x_{i}) - f(\hat{x_i}))^2$ - that will encourage the NN to preserve variance in $x$ that matters to $f(x)$. You will have to figure out the gradient function for this analytically. Also bear in mind I have not tried this, just attempted to match your requirement to some theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, you will now have $g(y)$ as it will be $f(\hat{x})$. You can generate any $y$ from $x$ by running the first half of the autoencoder, and can generate any $\hat{x}$ from $y$ by running second half of the autoencoder.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;might&lt;/em&gt; also be able to adapt t-SNE by using your function $f(x)$ to generate distances that need to be preserved when reducing dimensions.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-03T13:32:10.543" LastActivityDate="2017-05-03T13:32:10.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18731" PostTypeId="1" AcceptedAnswerId="18745" CreationDate="2017-05-03T17:06:39.453" Score="1" ViewCount="5198" Body="&lt;p&gt;Any Machine Learning algorithm to know the meaning of sentences? Specifically, I have sentences like, &quot;we do not allow managers to trade derivatives&quot;, &quot;We do not have policy on hedging&quot;, &quot;our policy do not permit trading on stock&quot;, &quot;trading is not allowed&quot;... The actual sentence is longer. Any algorithm I can know whether trading on stock is allowed or not by programming? Thank you for your help!!!&lt;/p&gt;&#xA;" OwnerUserId="31874" LastActivityDate="2017-05-04T11:40:53.210" Title="Any Machine Learning algorithm to know a yes/no answer?" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18732" PostTypeId="2" ParentId="18687" CreationDate="2017-05-03T17:29:48.230" Score="2" Body="&lt;p&gt;You need to groupby to deal with multiple vote counts:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-05-03T17:29:48.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18733" PostTypeId="2" ParentId="6165" CreationDate="2017-05-03T18:40:08.387" Score="1" Body="&lt;p&gt;My implementation of Complement Naive Bayes was merged into scikit-learn and can be found &lt;a href=&quot;https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.ComplementNB.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10556" LastEditorUserId="10556" LastEditDate="2023-08-31T11:49:37.587" LastActivityDate="2023-08-31T11:49:37.587" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18735" PostTypeId="2" ParentId="11930" CreationDate="2017-05-04T02:49:00.947" Score="2" Body="&lt;p&gt;I read the previous answers, and Neil Slater's comment to Emre's post, copied again below, hits the nail. &quot;Dimension hopping&quot; is a term created by Dr. Hinton of machine learning pioneer fame in the context of view point.  To quote Dr. Hinton &quot;So, typically envision the input dimensions correspond to pixels, and, if an object moves in the world and you don't move your eyes to follow it, the information about the object will occur on different pixels.&quot;  Age and weight are input dimension that are not easily confused.  Dr. Hinton used this obviously NOT likely dimension hopping situation of age and weight of patients to mean we would certainly be able to spot and fix any erroneous between these types of data (It's hard not to notice most adults are under 100 years old and more than 100 pounds).  The likely problem of dimension hopping, which Dr. Hinton was addressing, is pixels could be displaced because we have a different view point (e.g. the object could have moved or we are looking at it from a different angle).  Linear neural networks would not be able to detect this, whereas convolutional neural networks by design would.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The age example is supposed to highlight a dataset that does not have dimension-hopping. Age and weight do not &quot;hop&quot; or swap values randomly between examples - they are not interchangeable and the example is showing how odd that would be (and how difficult it would make simple tasks such as linear regression). Pixel values in images (and similar data in many signal processing tasks) do interchange or move easily due to the nature of the problem. – Neil Slater May 29 '16 at 18:01&quot;&lt;/p&gt;&#xA;" OwnerUserId="31889" LastEditorUserId="31889" LastEditDate="2017-05-04T04:35:42.067" LastActivityDate="2017-05-04T04:35:42.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18739" PostTypeId="2" ParentId="18731" CreationDate="2017-05-04T08:37:03.690" Score="2" Body="&lt;p&gt;I sounds like your need to consider Language Understanding services instead. There are a couple of providers of this type of service. The one I have used before is called LUIS from Microsoft:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.luis.ai/home/index&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.luis.ai/home/index&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not the algorithm you are looking for per se, but to solve your problem by implementing an algorithm, you need to develop a language understanding system using models based on deep learning neural nets, which is not a trivial exercise.&lt;/p&gt;&#xA;" OwnerUserId="31245" LastActivityDate="2017-05-04T08:37:03.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18741" PostTypeId="2" ParentId="17648" CreationDate="2017-05-04T09:58:04.127" Score="1" Body="&lt;p&gt;I would recommend you to have a look at &lt;a href=&quot;https://gym.openai.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI gym&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It will be easier for you to implement a new environment describing your problem. Regarding the agents, there are many implementations around that are compatible with OpenAI.&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="98307" LastEditDate="2020-08-11T16:13:15.740" LastActivityDate="2020-08-11T16:13:15.740" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18742" PostTypeId="1" CreationDate="2017-05-04T10:01:12.457" Score="-1" ViewCount="121" Body="&lt;p&gt;I am starting with machine learning and I am currently trying to understand artificial neural networks. In a multi-layer perceptron one minimizes some cost function (it can e.g. cross-entropy), but I have to feed the neural network with all the training data I have. This is not what a typical human learning process looks like. Rather, the experience changes by providing new examples through time and the prediction may change due to novel informations. In other words, the machine/net has to response to the changing environment by refining its predictions. I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Isn't it more natural to use Bayesian techniques for optimization where the next training example improves the experience (my neural net will improve through time by providing a new data)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does a neural net needs all the training data from the start or can I keep improving it by providing a new information (e.g. after some time) and without referring to the previous training data (I don't have to optimize everything once again but with larger collection of data)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="31905" LastActivityDate="2017-05-04T11:02:16.930" Title="Is learning process of artificial neural networks natural?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18744" PostTypeId="2" ParentId="18742" CreationDate="2017-05-04T11:02:16.930" Score="0" Body="&lt;p&gt;In my opinion the learning of MLPs is pretty damn &quot;natural&quot;. To answer your questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You would usually train the network's weights in random minibatches of data using a process called stochastic gradient descent (or some variant such as Adam, RMSprop, Adagrad etc...), rather than general gradient descent over the entire dataset at once. This is because each epoch is too expensive to calculate the gradients of weights over all of the data, vs a smaller random sample. Neural nets are trained iteratively and therefore kind of do improve through time (to a point, then they start overfitting). It is also common not to just run over the data once to get a decent fit, often you must run through a few times (each run through of data is called an &quot;epoch&quot;). Unlike Bayesian you don't really have a prior as the weights are usually randomly initialized, but the model (usually) keeps improving with more data / observations. There are Bayesian approaches to neural networks where you assume that the weights come from a distribution (say Gaussian prior over the weight space), then you can use MCMC - I don't think this is very popular anymore though.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can keep training by iterating over batches of different training data. In fact when data sets are large you basically have to break them up into different batches. So yes you can keep training them with different bits of data. Once some new data comes in you can then continue training, presumably you have saved the weights / model parameters. This won't start you from scratch again obviously.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think if you code a network up or even just start with some &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; or &lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/beginners&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow&lt;/a&gt; implementation (on MNIST) you'll get a feel for what's going on and how they learn / behave in practice.&lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2017-05-04T11:02:16.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18745" PostTypeId="2" ParentId="18731" CreationDate="2017-05-04T11:40:53.210" Score="1" Body="&lt;p&gt;You could do sentiment analysis for this task. Since the output is binary (yes/no) and there are obviously key words that will let you know whether a stock is available or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could get very good results using an LSTM for the sentences that feeds into a final logistic regression layer to answer whether the answer is yes &quot;1&quot; or no &quot;0&quot;. So you'd end up with an output probability of the question being answered &quot;yes&quot; or &quot;no&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This guy did the exact same model (with a different dataset basically) using theano, I highly recommend looking at this tutorial (code is provided as well):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/lstm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/tutorial/lstm.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All you need to do is build a dataset and get your hands a little dirty with python &amp;amp; theano. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to go deeper, look into word embeddings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively if that is too deep, you could just use a naive bayes algorithm. &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&lt;/a&gt; &#xA;It has good results on spam detection &amp;amp; probably might be able to help with your task as well!&lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2017-05-04T11:40:53.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18746" PostTypeId="1" CreationDate="2017-05-04T12:38:56.150" Score="0" ViewCount="484" Body="&lt;p&gt;I'm working on an experiment which is essentially a content recommendation service.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of content items in the form of articles, tweets, blog posts etc which have had a set of tags associated with them based on their contents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have a set of user profiles with information on the users personality, interests, dislikes, activities etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each user profile then has a column for liked and disliked tags to show which content items they would enjoy reading. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to be able build a service that when passed a new user profile, returns a set of tags (generated from similar existing user profiles) that can be used to find content items that the new user would enjoy reading. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my first experiment with machine learning so I was wondering if someone could give me some advice on how to achieve this, or an approach to get started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advanced! &#xA;Harry &lt;/p&gt;&#xA;" OwnerUserId="31911" LastActivityDate="2017-05-04T13:31:50.797" Title="Matching content item to a persons profile" Tags="|machine-learning|text-mining|recommender-system|prediction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18747" PostTypeId="2" ParentId="18746" CreationDate="2017-05-04T13:31:50.797" Score="1" Body="&lt;p&gt;Look into collaborative filtering and content based filtering.&lt;br&gt;&#xA;Essentially, build a user-space - using their personality, etc features. build an item space in a similar manner- using their tags, what not.&lt;br&gt;&#xA;Decide on a metric for each space, so you can compare how much each item is similar to each other(same for users).&lt;br&gt;&#xA;Then look at each user preferences and say, this user is similar to that user who liked this item who is close-like to that item.&lt;br&gt;&#xA;If user A liked item 1 and user B is similar to user A then its probable he will like item 1 as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://ashokharnal.wordpress.com/2014/12/18/worked-out-example-item-based-collaborative-filtering-for-recommenmder-engine/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is a simple enough example&lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-05-04T13:31:50.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18748" PostTypeId="1" CreationDate="2017-05-04T14:11:07.260" Score="-1" ViewCount="139" Body="&lt;p&gt;I have times series data with demand observations during months. I was wondering if, when computing demand predictions, I need to consider the outliers of the observations or not. What is your opinion on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;month   demand&#xA;1       450&#xA;2       670&#xA;3       332&#xA;4       123&#xA;5       343&#xA;6       437&#xA;7       456&#xA;8        -&#xA;9       231&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to predict the demand for the 10th month. I have computed the value &lt;code&gt;670&lt;/code&gt; as outlier (after having removed the 8th observation). I really don't know what I have to do with this outlier: should I consider it?&lt;/p&gt;&#xA;" OwnerUserId="31918" LastEditorUserId="29169" LastEditDate="2019-06-10T18:34:14.410" LastActivityDate="2019-06-10T18:34:14.410" Title="Considering outliers in demand predictions" Tags="|predictive-modeling|prediction|outlier|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="18749" PostTypeId="2" ParentId="18748" CreationDate="2017-05-04T14:16:23.947" Score="0" Body="&lt;p&gt;That depends ;). Without seeing the data I would say its very hard to say. It could be for example that you have the same demand every month but on December due to Christmas it is double. In this case just excluding the outlier would lead to loss of important information. Maybe you can expand your case/ show some data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Given the extreme limitations I personally would not give any forecasts. If you have to: Stick with some very naive approach as I can not imagine anything sophisticated will perform better. You can find a nice tutorial here: &lt;a href=&quot;http://machinelearningmastery.com/simple-time-series-forecasting-models/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/simple-time-series-forecasting-models/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the outlier question: If you have a strong reason from the outside to exclude a certain month (for example you know that in a certain month something exceptional happened which had a clear impact on the demand) you can exclude this month. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-05-04T14:47:23.200" LastActivityDate="2017-05-04T14:47:23.200" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18750" PostTypeId="1" CreationDate="2017-05-04T15:17:16.413" Score="2" ViewCount="1034" Body="&lt;p&gt;I have a dataset of 4 classes with the following number of instances:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class 0:  13175 &lt;/li&gt;&#xA;&lt;li&gt;Class 1:  82 &lt;/li&gt;&#xA;&lt;li&gt;Class 2:  75 &lt;/li&gt;&#xA;&lt;li&gt;Class 3:  121&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Have have applied several subsampling and oversampling methods from the &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python imbalance-learn API&lt;/a&gt; but none of them had a good performance for all classes. I have applied:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Undersampling: CondensedNearestNeighbour, EditedNearestNeighbours, NeighbourhoodCleaningRule, RandomUnderSampler.&lt;/li&gt;&#xA;&lt;li&gt;Oversampling: SMOTE, ADASYN&lt;/li&gt;&#xA;&lt;li&gt;class_weight:['balanced'] parameter option in my grid search &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/albahnsen/CostSensitiveClassification&quot; rel=&quot;nofollow noreferrer&quot;&gt;costcla&lt;/a&gt; library, but it does not work with more than two classes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And I was not successful. Could you suggest a solution for this problem?&lt;/p&gt;&#xA;" OwnerUserId="8710" LastEditorUserId="31513" LastEditDate="2017-05-10T08:05:03.860" LastActivityDate="2017-08-09T08:01:15.087" Title="Imbalance classes problem" Tags="|python|multiclass-classification|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18751" PostTypeId="1" CreationDate="2017-05-04T18:53:44.157" Score="11" ViewCount="4885" Body="&lt;p&gt;I am using package &lt;code&gt;Keras&lt;/code&gt; in R to do a neural network. How may I extract the output from a hidden layer? I found an &lt;a href=&quot;https://github.com/fchollet/keras/issues/588&quot; rel=&quot;noreferrer&quot;&gt;example&lt;/a&gt; in python, but it is just I have no idea how to do that in R. &lt;/p&gt;&#xA;" OwnerUserId="25867" LastActivityDate="2023-06-01T08:48:57.443" Title="R, keras: How to get output of a hidden layer?" Tags="|neural-network|r|keras|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18752" PostTypeId="1" CreationDate="2017-05-04T19:29:49.630" Score="1" ViewCount="712" Body="&lt;p&gt;Hi I have fairly very short time series data. The data set has number of systems $s_{1}, s_{2}, s_{3},..s_{n}$. For each $s_{i}$ we have recorded number of failures on each day. As of now, we have recorded 30 days. I would like to know can I use LSTM for predicting number of system failures for next day.&#xA;How about using Vector Auto Regression?. Any starting pointers and code references will be useful. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="31929" LastActivityDate="2017-10-30T14:44:01.237" Title="time series prediction using LSTM" Tags="|machine-learning|deep-learning|time-series|tensorflow|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18754" PostTypeId="1" CreationDate="2017-05-04T22:55:59.240" Score="1" ViewCount="322" Body="&lt;p&gt;I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if the observed states were sampled using different time intervals. How can I find these probabilities?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I will try to make the question a more clear. I have samples collected in random intervals during a 6-month period. This samples represent the &quot;quality&quot; of a system, which is ranked from 0 to 15 in discrete intervals i.e. (0,1,2...15). I need to model de system using a FSMC to mimic the system's behavior. So far, I have estimated the transition probabilities between states using only the frequencies of those transitions using all the samples. I am not interested in modeling the time, however, I am not sure If I can estimate the transition probabilities in such a simple way or if I have to take the time between samples (which in my case is random) into consideration when estimating those probabilities.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="31936" LastEditorUserId="31936" LastEditDate="2017-05-05T17:18:51.763" LastActivityDate="2017-05-05T17:18:51.763" Title="How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced" Tags="|probability|markov-process|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18760" PostTypeId="1" CreationDate="2017-05-05T05:22:06.990" Score="0" ViewCount="713" Body="&lt;p&gt;I am trying to understand kmeans clustering and I read a article where kmeans is used for clustering the features generated in network logs. This clustering is followed by a supervised classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They mention in article :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Due to extensive repetition of input and output network ip addresses, portnumbers, and protocol information in the collected data, these were deemed scalar entities for analysis and excluded while clustering. The remaining attributes  were chosen as input to clustering  algorithm&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fail to understand what scalar entities mean here? Does this mean that these input and output ip address and port numbers are not considered as features during feature extraction stage? If the ip addresses arent considered, how are the features mapped to the input data of network logs?&lt;/p&gt;&#xA;" OwnerUserId="28832" LastActivityDate="2017-05-12T14:28:16.317" Title="Scalar entities for k means clustering" Tags="|classification|clustering|k-means|feature-extraction|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18761" PostTypeId="2" ParentId="18670" CreationDate="2017-05-05T05:35:22.033" Score="2" Body="&lt;p&gt;Could be your network structure: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper states that their experiment are done using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conv&#xA;pool&#xA;dense&#xA;dense&#xA;dense(softmax)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So something like this for fws:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fws():&#xA;    #Input shape: (batch_size,40,45,3)&#xA;    #output shape: (1,15,50)&#xA;    # number of unit in conv_feature_map = splitd&#xA;&#xA;    filter_size = 8&#xA;    pooling_size = 28&#xA;    stride_step = 2&#xA;    pool_splits = ((splits - pooling_size)+1)/2&#xA;    conv_featur_map = []&#xA;    pool_feature_map = []&#xA;    print &quot;Printing shapes&quot;&#xA;&#xA;&#xA;    list_of_input = [Input(shape = (window_height,total_frames_with_deltas,3)) for i in range(splits)]&#xA;&#xA;&#xA;    #convolution&#xA;    shared_conv = Conv2D(filters = 150, kernel_size = (filter_size,45), activation='relu')&#xA;    for i in range(splits):&#xA;        conv_featur_map.append(shared_conv(list_of_input[i]))&#xA;&#xA;    #Pooling&#xA;    input = Concatenate()(conv_featur_map)&#xA;    input = Reshape((splits,-1))(input)&#xA;    pooled = MaxPooling1D(pool_size = pooling_size, strides = stride_step)(input)&#xA;&#xA;    #reshape = Reshape((3,-1))(pooled)&#xA;&#xA;    #fc&#xA;    dense1 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_1&quot;)(pooled)&#xA;    dense2 = Dense(units = 1000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;    dense3 = Dense(units = 50 , activation = 'softmax', name = &quot;dense_3&quot;)(dense2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Changing the position of the for loop in fws (moving it 2 lines up) makes it to a lws (Plus some adjustments with the pooling layer). &lt;/p&gt;&#xA;" OwnerUserId="29208" LastActivityDate="2017-05-05T05:35:22.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18762" PostTypeId="1" CreationDate="2017-05-05T05:36:05.747" Score="0" ViewCount="102" Body="&lt;p&gt;I want to create a deep learning model for a binary classification problem. I have 16 features and 50k samples. How would I select number of layers and number of neurons of each layer  for my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please share the links which explains the same.&lt;/p&gt;&#xA;" OwnerUserId="31947" LastActivityDate="2017-05-05T12:19:23.860" Title="Building a deep learning model" Tags="|classification|deep-learning|model-selection|" AnswerCount="1" CommentCount="1" ClosedDate="2017-05-06T09:25:37.093" ContentLicense="CC BY-SA 3.0" />
  <row Id="18764" PostTypeId="1" CreationDate="2017-05-05T07:34:16.573" Score="2" ViewCount="248" Body="&lt;p&gt;I started learning Time series modeling and bit confused when I started learning the advanced techniques like Box-Jenkins and Holts Winter techniques. &lt;strong&gt;What is the difference between ARIMA, Box-Jenkins and Holt's Winter Models.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="31949" LastActivityDate="2017-05-05T13:55:40.003" Title="Different Time series Modeling techniques?" Tags="|data-mining|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18766" PostTypeId="2" ParentId="18760" CreationDate="2017-05-05T08:24:24.263" Score="1" Body="&lt;p&gt;While doing clustering analysis, you would prefer to have features which discriminate data points and help you discover more homogeneous segments. However, if the value of a particular feature or variable is same for large percentage of the data, then that feature would not add any information for clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us take an example, say you are dealing with bank customers data and there's a feature called 'country' in it. If in the given data, most of the customers have country as USA, then you would want to simply drop that feature as it adds no useful information. This is the extreme case in which there is single unique value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your network data logs example, port numbers and protocols would be mostly the same. The web server might be listening on port 443 for HTTPS protocol connection. So in that case, the two columns can be excluded from clustering analysis.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2017-05-05T08:24:24.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18767" PostTypeId="2" ParentId="18754" CreationDate="2017-05-05T08:32:11.953" Score="1" Body="&lt;p&gt;With the limited information that you have given, it is not possible to judge the trade-off between different approaches of handling time here. I can propose the following alternatives.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Check if you can discretize the time information and then use transition to same state assumption. For example, let us say that transition from A to B took place at 30 minutes. If you have standard time interval of 10 minutes, you can say that for interval 1 &amp;amp; 2 (10 minutes, 20 minutes), the transition was from state A to A whereas in the third interval the transition was from A to B. There will be problems though when multiple transitions take place in a given interval. You can either make the interval small enough or distribute the transition probability equally in such case.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the time information significant to whatever that you are trying to model? Otherwise you can ignore the time information.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2017-05-05T08:32:11.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18768" PostTypeId="2" ParentId="18715" CreationDate="2017-05-05T08:40:15.223" Score="2" Body="&lt;p&gt;Technically, it will be alright to use a sample of the data. One important assumption is that most of the features don't have outlier. In such case your sampling might miss out on those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, &lt;em&gt;I don't see a necessity of sampling with the size of data that you have&lt;/em&gt; given. 150mb is not large data at all, given that you have 8 GB ram at disposal. Rather than sampling the data, first check if there is alternate way of selecting the features. Since there are only 18 columns, look at the summaries for each columns, how do the features correlate with your outcome variable, which variables add very little information. If you have not done more exploratory analysis of the 18 variables, I'll suggest you do that first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, why not build your random forest directly with 18 features? If you had, say, 180 features, it would have made sense to do automated feature selection first using Boruta etc. If there are features that do not add information, their variable importance in forest model will be low. You can drop them in subsequent analysis&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2017-05-05T08:40:15.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18769" PostTypeId="1" CreationDate="2017-05-05T09:16:30.670" Score="2" ViewCount="997" Body="&lt;p&gt;I run through the &lt;a href=&quot;https://www.tensorflow.org/get_started/tflearn&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quickstart&lt;/a&gt; tutorial of the tf.contrib.learn, which uses the Iris data set. After training the DNN model I get an accuracy of ~0.967.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an attempt to improve the accuracy, I tried to change the number and size of the hidden layers, I doubled the steps, and I changed optimizer and learning rates. However, the accuracy does not increase (or decrease).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could that might be a dataset limitation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how to improve the performance? I want to be able to dive into a NN and try to optimize it, before I go blindly into another tutorial.&lt;/p&gt;&#xA;" OwnerUserId="30051" LastActivityDate="2017-06-04T15:13:14.520" Title="Improve classification accuracy on Iris data set - tf.contrib.learn Quickstart tutorial" Tags="|python|classification|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18770" PostTypeId="1" AcceptedAnswerId="18773" CreationDate="2017-05-05T11:37:52.707" Score="2" ViewCount="249" Body="&lt;p&gt;I'm attempting a rather simple exercise in machine learning and trying to classify samples of text as either plaintext or ciphertext (encrypted).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are two samples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Plaintext: &lt;code&gt;This is a sentence in plaintext which any human person can read&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Ciphertext: &lt;code&gt;5oXbLiEZbMUgOOdYy+q4+rsDaqUngBrrUbpVeuu2ggvP6hHObC4GgTLhq&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The specific encryption used doesn't have any special attributes I can use to classify (for example, the ciphertext isn't guaranteed to be significantly longer than plaintext), so the task is all about figuring out what text is indistinguishable from random characters and what text is readable plaintext.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current heuristic involves counting whitespaces and assuming anything with a whitespace ratio above a certain threshold is plaintext, but I'm trying to find a better robust algorithm.&lt;/p&gt;&#xA;" OwnerUserId="31960" LastEditorUserId="31960" LastEditDate="2017-05-05T11:42:59.953" LastActivityDate="2017-05-05T12:49:43.427" Title="Classify ciphertext vs. plaintext" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18771" PostTypeId="2" ParentId="18762" CreationDate="2017-05-05T12:19:23.860" Score="3" Body="&lt;p&gt;There are no rules to infer neural network hyper-parameters from a problem description. With only number of features, number of examples and the fact you have a binary classification problem, it is far too little to even make an educated guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, are you sure you are ready to build a deep learning model for your data? Have you looked at the data, or scatter plotted it reduced to two dimensions using PCA or t-SNE, to get a feel for how easy your data is to separate (easy to separate raw data implies using simpler/shallower model)? You could try a basic model such as logistic regression or SVM in order to establish a benchmark so you can tell whether the deep model was doing anything useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you are ready to go ahead, the time-honoured approach is to try out variations, and measure the results using cross-validation. You can do this methodically, by for instance starting with one hidden layer with 64 neurons, and either adding/removing layers or adding/removing neurons in each layer. Generally these searches don't cover all possible variations, but stop once you have tried those that seem interesting and have reached a reasonable result for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do note that other hyper-parameters will affect results and can interact with your choices for numbers of neurons and layers. You cannot isolate your choice of network depth from other choices, such as optimisation method, activation functions, regularisation. When starting to build a model, it is just as reasonable to spend a long time exploring these other factors as it is to look at network size/shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is very easy to get a deep NN to over-fit your data. Cross-validation is therefore a necessity as you explore hyper-parameters for your problem. One reasonable method of searching against number of neurons is to increase them until you start to notice over-fitting, then adjust regularisation to stop the over-fit. At that point, with all other factors remaining the same, there is probably no need to explore networks with many more neurons (although deeper/shallower networks may still be worth exploring, and if you do so, you will once again want to explore number of neurons per layer).&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-05T12:19:23.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18772" PostTypeId="2" ParentId="16309" CreationDate="2017-05-05T12:33:40.090" Score="1" Body="&lt;p&gt;Shiny App, will be the place to display the graphics, however the animation you are looking for is not something that &lt;code&gt;plotly&lt;/code&gt; would create, to the best of my knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;d3&lt;/code&gt; however is a different beast. It can create aesthetically pleasing graphics. For your specific case, here is the example&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://bl.ocks.org/mbostock/1346410&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://bl.ocks.org/mbostock/1346410&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you can look at the entire &lt;a href=&quot;https://d3js.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gallery&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2017-05-05T12:33:40.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18773" PostTypeId="2" ParentId="18770" CreationDate="2017-05-05T12:49:43.427" Score="2" Body="&lt;p&gt;Simply counting the frequency of characters ought to easily distinguish between English language and ciphertext, because they're so obviously different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can just count the frequency of characters in a big corpus of English, and a big corpus of ciphertext, and apply a chi-squared test to each to figure out which one matches the counts in a new chunk of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if you can assume ciphertext has a roughly uniform distribution over characters, that alone lets you construct a good test for whether new text is unlikely to be ciphertext.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did a short blog post on something similar. &lt;a href=&quot;https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-05-05T12:49:43.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18775" PostTypeId="2" ParentId="18769" CreationDate="2017-05-05T13:29:44.180" Score="1" Body="&lt;p&gt;An accuracy of 0.967 is a good score if you achieved it on the test set, so make sure that you are talking about the test set here. Secondly, increasing the number of iterations on the dataset won't do anything if your solution has already converged. Thirdly, adding more layers is not a very good idea, since you are already using a DNN classifier and could lead to overfitting. That being said, try to caculate a cross_validation score for your dataset, as then run it on a test set. In corss-validation you basically train your model on different chunks of your dataset, and validate on the chunk you leave out, then test it on a separate test set. This way you can actually see how your hyperparameters are behaving and you won't have to guess. Once you calculate the test score this way, it means you have good values of the parameters. If you still don't feel satisfied with the accuracy, its time to try a different algorithm, using more units in each layer and trying different activation functions. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-05-05T13:29:44.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18776" PostTypeId="2" ParentId="18719" CreationDate="2017-05-05T13:52:29.847" Score="0" Body="&lt;p&gt;This is a wide open question, so I am not sure which aspects you are most interested in.  Here are some Wikipedia articles you can get started with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Protocol_for_Metadata_Harvesting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Metadata Harvesting&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Metadata_discovery&quot; rel=&quot;nofollow noreferrer&quot;&gt;Metadata Discovery&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-05-05T13:52:29.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18777" PostTypeId="2" ParentId="18764" CreationDate="2017-05-05T13:55:40.003" Score="2" Body="&lt;p&gt;&lt;strong&gt;ARIMA&lt;/strong&gt; is a parameterized model that consists of a few parts. First the &lt;strong&gt;I&lt;/strong&gt; is for integrated, and it has to do with a technique to remove non-stationarity from the data, by looking at differences between sequential values instead of the values itself, the parameter indicates how often we want to do this. &lt;strong&gt;AR&lt;/strong&gt; stands for auto-regressive, it's a regression model that regresses on previous values of itself (lag features), the order determines how far back we look. The &lt;strong&gt;MA&lt;/strong&gt; means Moving Average, which is referring to a linear combination of error terms, this is also parameterized by an order. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Box-Jenkins&lt;/strong&gt; is a method that uses ARIMA, but it first involves a number of checks before fitting the model, and afterwards does criticism to see how well the model works. This allows for iteration into a better fitting model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Holt-Winters methods&lt;/strong&gt; are used for smoothing out time series. Sometimes there is a lot of noise involved in time series, while you are only interested in trends and how values relate to each other. By smoothing it out via for example moving averages or exponential smoothing hopefully you gain more information by trying to filter out noise than you lose by throwing away relevant information.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-05T13:55:40.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18778" PostTypeId="1" CreationDate="2017-05-05T16:07:43.490" Score="0" ViewCount="704" Body="&lt;p&gt;I build a classification model. The results below when implemented Decision Tree Classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction accuracy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.785813630042&lt;/p&gt;&#xA;&#xA;&lt;p&gt;confusion matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[[ 1302  1581]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ 2577 13953]]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         precision    recall  f1-score   support&#xA;&#xA;    2.0       0.34      0.45      0.39      2883&#xA;    3.0       0.90      0.84      0.87     16530&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;avg / total       0.81      0.79      0.80     19413&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Neural Network---&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction accuracy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.863132952145&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confusion Matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[[  718  2165]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[  492 16038]]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         precision    recall  f1-score   support&#xA;&#xA;    2.0       0.59      0.25      0.35      2883&#xA;    3.0       0.88      0.97      0.92     16530&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;avg / total       0.84      0.86      0.84     19413&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Gradient Boosting Classifier---&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction accuracy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.870035543193&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Confusion Matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[[  971  1912]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[  611 15919]]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         precision    recall  f1-score   support&#xA;&#xA;    2.0       0.61      0.34      0.43      2883&#xA;    3.0       0.89      0.96      0.93     16530&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;avg / total       0.85      0.87      0.85     19413&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient boosting classifier gives me good accuracy score but it gives bad recall for class 2 compared with Decision Tree classifier. &#xA;Train Data for both classes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Class 2: 2908&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Class 3: 16673&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is recall depend on training data? How do I improve recall for class 2 along with accuracy?&lt;/p&gt;&#xA;" OwnerUserId="29824" LastActivityDate="2017-05-07T15:11:20.210" Title="Is this a good classified model based confusion matrix and classification report?" Tags="|machine-learning|classification|accuracy|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18780" PostTypeId="1" AcceptedAnswerId="18806" CreationDate="2017-05-05T16:58:43.873" Score="3" ViewCount="2002" Body="&lt;p&gt;After performing a PCA and studying the proceeding i ask myself what the result is good for in the next step. From the PCA i learned how to visualize the dataset by lowering the dimension, i got handy new vectors to describe the members of the population in a more efficient way and i learned which original predictors correlate and contribute more than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask myself questions like, is there more to learn from PCA, is it a good idea to use the PCA model for a learning algorithms to perform better and is it possible to respect nomial or ordinal predictors in the PCA somehow?&lt;/p&gt;&#xA;" OwnerUserId="31668" LastActivityDate="2017-05-07T06:15:36.993" Title="What can i do after a PCA with the results?" Tags="|data-mining|data|pca|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18782" PostTypeId="1" AcceptedAnswerId="18808" CreationDate="2017-05-05T18:44:39.317" Score="4" ViewCount="5514" Body="&lt;p&gt;As a newbie, I am a little confused. I have a dataset for binary classification with 11 features and 102 sample data. I have seen in most places (e.g., kaggle competions), the dataset may have hundreds of thousands of data samples for tens of features. On the other hand, &lt;a href=&quot;https://academic.oup.com/bioinformatics/article/21/8/1509/249540/Optimal-number-of-features-as-a-function-of-sample&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; says (at least for LDA classifier) optimal number of features is n-1 for a sample size n. My question is, if small no. of samples is enough (or even optimal), why care for larger samples? What am I missing here?&lt;/p&gt;&#xA;" OwnerUserId="31971" LastActivityDate="2017-05-08T11:55:17.620" Title="Number of features vs. number of samples : if small sample size is sufficient, why take large number of samples?" Tags="|classification|dataset|feature-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18783" PostTypeId="1" CreationDate="2017-05-05T19:05:56.013" Score="1" ViewCount="700" Body="&lt;p&gt;I have developed online neural network based one-class classifier and also enabled it for forgetting mechanism. So, now online learning with forgetting mechanism is possible. But how to handle if data is non-stationary. As it is one-class classification so, suppose we have two class data i.e. normal and outlier class then trained by only normal class data in online fashion. But training data might have different distribution compared to distribution of normal and outlier data in testing as their distribution are changing. So, how algorithm will handle this drift/shift.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more doubt regarding non-stationary data handling: whether we will pass trained labelled data once and further labelled data for training are collected from unlabelled testing data?&lt;/p&gt;&#xA;" OwnerUserId="18818" LastEditorUserId="18818" LastEditDate="2017-05-07T03:15:25.243" LastActivityDate="2017-12-03T06:29:32.577" Title="How to handle non-stationary data in online neural network based one-class classifier for anomaly detection?" Tags="|machine-learning|neural-network|data-mining|time-series|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18788" PostTypeId="1" AcceptedAnswerId="18795" CreationDate="2017-05-05T20:45:36.120" Score="-1" ViewCount="82" Body="&lt;p&gt;This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;BlueMix is essentially an all-in-one cloud platform for accessing various IBM Analytics APIs and you can code in various languages, while &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data Science Experience is sort of an RStudio on steroids (it even allows you to use RStudio), where I can process massive data sets using IBM's resources instead of my own, but I can also code things in Python if I want.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For my needs, I will be using several different types of data, including time-series physiological data and natural language text (sounds like I'll likely need Watson). I would like to be able to use TensorFlow for my work as well.&lt;/p&gt;&#xA;" OwnerUserId="31975" LastActivityDate="2021-04-16T10:26:59.303" Title="What are the differences between IBM BlueMix and IBM Data Science Experience?" Tags="|machine-learning|bigdata|time-series|tensorflow|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18790" PostTypeId="2" ParentId="18690" CreationDate="2017-05-05T22:13:15.593" Score="1" Body="&lt;p&gt;Have you looked into &lt;a href=&quot;https://en.wikipedia.org/wiki/Differential_evolution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Differential Evolution&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is similar to a genetic algorithm and appears to be available with &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.optimize.differential_evolution.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SciPy&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31975" LastActivityDate="2017-05-05T22:13:15.593" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18791" PostTypeId="2" ParentId="11485" CreationDate="2017-05-05T23:12:26.297" Score="9" Body="&lt;p&gt;&lt;strong&gt;nstart&lt;/strong&gt; option attempts multiple initial configurations and reports on the best one. For example, adding nstart=25 will generate 25 initial random centroids and choose the best one for the algorithm. Hope this helps!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.r-statistics.com/2013/08/k-means-clustering-from-r-in-action/&quot; rel=&quot;noreferrer&quot;&gt;You can read more here...&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31977" LastEditorUserId="31977" LastEditDate="2017-05-05T23:31:46.550" LastActivityDate="2017-05-05T23:31:46.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18793" PostTypeId="2" ParentId="18783" CreationDate="2017-05-06T06:44:58.833" Score="2" Body="&lt;p&gt;Usually, non-stationary series can be dealt by differencing the time series once or twice. You can add verification to your tool - checking auto-correlation plot (or partial correlation plot).&lt;/p&gt;&#xA;" OwnerUserId="31627" LastActivityDate="2017-05-06T06:44:58.833" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18795" PostTypeId="2" ParentId="18788" CreationDate="2017-05-06T09:32:15.943" Score="1" Body="&lt;p&gt;If you are comfortable with advanced coding and familiar with modeling technics in R and Python, I would recommend to use the datascience experience, since you will do all the coding.&#xA;But if you want to have ready-to-use models and API's, and wont be bothered to tweak or modify some advanced parameters, Bluemix is a good choice for you :)&lt;/p&gt;&#xA;" OwnerUserId="7714" LastEditorUserId="7714" LastEditDate="2021-04-16T10:26:59.303" LastActivityDate="2021-04-16T10:26:59.303" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18797" PostTypeId="1" CreationDate="2017-05-06T10:22:02.717" Score="8" ViewCount="1762" Body="&lt;p&gt;I'm trying to analyze the architecture of a CNN and I was expecting it to be shaped:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input Image: 28 x 28&lt;/li&gt;&#xA;&lt;li&gt;1st conv.: 28 x 28 x nb_filters&lt;/li&gt;&#xA;&lt;li&gt;2nd conv.: 28 x 28 x nb_filters x nb_filters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But apparently it goes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input Image: 28 x 28&lt;/li&gt;&#xA;&lt;li&gt;1st conv.: 28 x 28 x nb_filters&lt;/li&gt;&#xA;&lt;li&gt;2nd conv.: 28 x 28 x nb_filters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I thought that every filter in the second layer would be applied to every activation map in the first layer resulting in:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Activation maps after second layer = nb_filters_1st_layer x nb_filters_2nd_layer&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If it's not working that way, how do I know what filters are applied to what activation maps?&lt;/p&gt;&#xA;" OwnerUserId="31988" LastActivityDate="2017-05-06T16:53:15.217" Title="How do convolutional layers get connected to each other?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18799" PostTypeId="2" ParentId="18797" CreationDate="2017-05-06T12:46:58.480" Score="6" Body="&lt;p&gt;There is more than one way to do this in principle, but most CNNs, and most CNN libraries will do the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Each layer has a &lt;em&gt;target&lt;/em&gt; number of feature maps, which is what you have labelled &lt;code&gt;nb_filters&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each feature map is derived from &lt;em&gt;all&lt;/em&gt; previous layer's feature maps.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The weights of the locally connected kernels &lt;em&gt;do&lt;/em&gt; follow a &lt;code&gt;width x height x nb_input_features x nb_output_features&lt;/code&gt; structure similar to your first example (with width and height being the convolutional kernel size), because each output feature in layer b is derived as the sum of convolutions over every feature in layer a.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So if your output feature map with (channel) index $j$ in layer $k+1$ is $I^{(j, k+1)}$, and you are looking at connection to all input feature maps $I^{(i, k)}_i$ in layer $k$ with filters connecting each map $F^{(i,j,k)}$, then the maths looks a bit like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I^{(j,k+1)} = f(b^{(j,k)} + \sum_i I^{(i,k)} \ast F^{(i,j,k)})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . .where $f(x)$ is the transfer or activation function e.g. ReLU. The bias is $b^{(j,k)}$ and the $\ast$ represents the convolution function. Summing over $i$ means iterating over all the input channels. NB I have indexed the items all with a superscript like this to more clearly show that all the variables (apart from $b$) are matrices &lt;em&gt;after&lt;/em&gt; indexing - i.e. this is not the equation for a specific pixel, but for a whole feature map. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the kernel $F$ is indexed by $(i,j,k)$ i.e. there are &lt;code&gt;nb_input_features&lt;/code&gt; times &lt;code&gt;nb_output_features&lt;/code&gt; filters connecting each layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In comparison, your expected scheme would keep the derived features &lt;em&gt;separate&lt;/em&gt; layer-by-layer, and create an exponential growth in number of feature maps as the layers increased, with each feature &quot;type&quot; being handled separately. Whereas what happens in practice is each feature map is a &quot;re-mix&quot; of all the feature maps in the previous layer, convolved each with its own filter then summed up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This cross-referencing of features between layers helps create richer complex feature maps - e.g. it can combine edge and corner detectors to detect corners with specific angles. Whilst a scheme that refined each feature into multiple separate sub-features would not be able to do this mixing between features as freely (until the later fully-connected layers) - although it &lt;em&gt;might&lt;/em&gt; do well at other tasks - e.g. detecting variations of extended structures of the same type longer lines, curves etc (NB this is just a guess on my part, I don't know for certain).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If it's not working that way, how do I know what filters are applied to what activation maps?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You cannot say something like &quot;this feature map in layer N is derived only from this other feature map in layer N-1, using this single filter&quot;, because each feature map in layer N is derived from &lt;em&gt;all&lt;/em&gt; feature maps in layer N-1 and results of the filters are combined in a way that you cannot easily reverse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the weights array will be arranged so that you can find which filters are connecting each input feature map to a specific output feature map. Which dimension you would use to isolate those depends on the library. If your library stores weights in a &lt;code&gt;[kernel_width, kernel_height, input_feature_map_id, output_feature_map_id]&lt;/code&gt; form, then your selection (using Numpy-style syntax might be) &lt;code&gt;[:,:,:,n]&lt;/code&gt; where &lt;code&gt;n&lt;/code&gt; is the id of the output feature map that you want to fetch the filters for.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-06T16:53:15.217" LastActivityDate="2017-05-06T16:53:15.217" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18801" PostTypeId="1" CreationDate="2017-05-06T17:52:12.990" Score="2" ViewCount="768" Body="&lt;p&gt;I'm currently working with a dataset where each row contains 2 things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Text&lt;/li&gt;&#xA;&lt;li&gt;Related tags&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Both are dutch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to build a model that allows me to assign tags to new text based on what it learned from the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I think I'm having is that the dataset itself is very small (around 2000 messages) but the tags are also very unevenly distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, the most used tag occurs 121 times, but there are around 40 tags that only occur 5 times in a different article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to solve the data quantity problem by using dutch word-vectors and use them in an Embedding layer. This allows to give more context to the final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tags problem I haven't got a solution for yet. I think I should make a custom loss function that lets text with tags occurring more frequently weigh less when they are valid, and tags occurring less will score higher. But I have no idea how to get started on that. If you can help me, thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MAX_SEQUENCE_LENGTH = 100&#xA;EMBEDDING_DIM = 300&#xA;&#xA;embedding_layer = Embedding(num_words,&#xA;                            EMBEDDING_DIM,&#xA;                            weights=[embedding_matrix],&#xA;                            input_length=MAX_SEQUENCE_LENGTH,&#xA;                            trainable=False)&#xA;&#xA;sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')&#xA;embedded_sequences = embedding_layer(sequence_input)&#xA;x = Dropout(0.1)(embedded_sequences)&#xA;x = Conv1D(128, 5, activation='relu')(x)&#xA;x = MaxPooling1D(5)(x)&#xA;x = Conv1D(128, 5, activation='relu')(x)&#xA;x = MaxPooling1D(5)(x)&#xA;x = Flatten()(x)&#xA;x = Dense(128, activation='relu')(x)&#xA;preds = Dense(len(labels_index), activation='sigmoid')(x)&#xA;&#xA;model = Model(sequence_input, preds)&#xA;model.compile(loss='binary_crossentropy',&#xA;              optimizer='adam',&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Data input is a padded sequence as input and output is a binary list of tags that are related to the input (made using MultiLabelBinarizer).&lt;/p&gt;&#xA;" OwnerUserId="31995" LastActivityDate="2017-05-06T17:52:12.990" Title="Multi-label classification of text with variable tag distribution in Keras" Tags="|python|keras|multilabel-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18802" PostTypeId="1" AcceptedAnswerId="18803" CreationDate="2017-05-06T20:38:26.903" Score="6" ViewCount="6865" Body="&lt;p&gt;In &lt;code&gt;linear regression&lt;/code&gt; we use the following cost function which is a convex function:&lt;br /&gt; &lt;a href=&quot;https://i.stack.imgur.com/aFBeP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aFBeP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;br /&gt;&#xA;We Use the following cost function&lt;br /&gt; &#xA;&lt;a href=&quot;https://i.stack.imgur.com/JX0jv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JX0jv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;br /&gt;&#xA;in &lt;code&gt;logistic regression&lt;/code&gt; because the preceding cost function is not &lt;code&gt;convex&lt;/code&gt; whenever the hypothesis (h) is logistic function. We have changed the equation of cost function to have a convex shape to find its global (the only one which exists). There is a fact that I can not understand. In &lt;code&gt;Multi Layer Perceptrons&lt;/code&gt; ANNs I have seen a lot that they can be stuck in local minimums. Why is that? We have used this cost function for each perceptron and gotten the rules for updating the values for the weights in back propagation algorithm; So why do we stuck?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-05-06T20:43:36.590" LastActivityDate="2017-08-09T12:00:46.680" Title="Does MLP always find local minimum" Tags="|neural-network|cost-function|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18803" PostTypeId="2" ParentId="18802" CreationDate="2017-05-06T21:00:38.957" Score="8" Body="&lt;p&gt;The loss functions are only simple convex functions with respect to the weight parameters (and specific data) when there is a single layer. More exactly, they can proven to be always convex with respect to the weights in the simple models (linear or logistic regression), but &lt;em&gt;not&lt;/em&gt; with respect to weights of deeper networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can prove that there must be more than one minimum in a network with 2 or more layers - and thus the loss function cannot be convex - by considering swapping the weights around when you have found a minimum value. Unlike with a single layer network, it is possible to swap the weights around that feed into the hidden layer &lt;strong&gt;whilst maintaining the same output&lt;/strong&gt;. For example, you can swap the weights between input and hidden layer so that values of neuron output 1 and neuron output 2 are reversed. Then you can also swap the weights feeding &lt;em&gt;out&lt;/em&gt; of those neurons to the output so that the network still outputs the same value. The network would have a different set of weights, but generate the same outputs, and so this new permutation of weights is &lt;em&gt;also&lt;/em&gt; at a minimum for the loss function. It is the &quot;same&quot; network, but the weight matrices are different. It is clear that there must be very many fully equivalent solutions all at the true minimum.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a worked example. If you have a network with 2 inputs, 2 neurons in the hidden layer, and a single output, and you found that the following weight matrices were a minimum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W^{(1)} = \begin{bmatrix} -1.5 &amp;amp; 2.0 \\ 1.7 &amp;amp; 0.4 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W^{(2)} = \begin{bmatrix} 2.3 &amp;amp; 0.8 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the following matrices provide the same solution (the network outputs the same values for all inputs):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W^{(1)} = \begin{bmatrix} 1.7 &amp;amp; 0.4 \\ -1.5 &amp;amp; 2.0 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W^{(2)} = \begin{bmatrix} 0.8 &amp;amp; 2.3 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we said the first set of 6 parameters was a solution/minimum, then the &#xA;second set of 6 parameters must also be a solution (because it outputs the same). The loss function therefore has 2 minima with respect to the weights. In general for a MLP with one hidden layer containing $n$ neurons, there are $n!$ permutations of weights that produce identical outputs. That means that there are at least $n!$ minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although this does not prove that there are worse local minima, it definitely shows that the loss surface must be much more complex than a simple convex function.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-06T21:52:17.970" LastActivityDate="2017-05-06T21:52:17.970" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18804" PostTypeId="1" AcceptedAnswerId="18807" CreationDate="2017-05-06T21:27:35.243" Score="2" ViewCount="1268" Body="&lt;p&gt;pretty simple question here but just can't seem to find the answer in the normally great documentation for sklearn.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working with binary classifiers, but we can just assume i am using &lt;code&gt;LogisticRegression&lt;/code&gt;, and I was wondering if there is a general way to have the classifier select, say only 10 best (most sure) data points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say I train a set with 500K data points, and my test set has 10K lines, and out of the 10K, I just want to choose 10 that have the highest chance of being true positives.  Does this make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read about, and have been playing with the &lt;code&gt;class_weights&lt;/code&gt; attribute, which works well for giving more/less weight to each of the binary outcome classes, but its not quite working for what I want in that it always give different number of position predictions, and I can't really tell how sure the classifier is about each one of those.  &lt;/p&gt;&#xA;" OwnerUserId="31781" LastEditorUserId="31781" LastEditDate="2017-05-07T13:41:00.383" LastActivityDate="2017-05-07T14:23:27.340" Title="sklearn select N best using classifier" Tags="|python|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18806" PostTypeId="2" ParentId="18780" CreationDate="2017-05-07T06:15:36.993" Score="1" Body="&lt;p&gt;Well PCA, as suggested above by @CarltonBanks, does help you remove features with the least correlation and use mash the features together such that they have the highest correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question, &lt;strong&gt;how to visualize higher dimensions using PCA&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Transform the feature matrix with the number of components of your data set to 2 or 3 &lt;/li&gt;&#xA;&lt;li&gt;This ensures you can represent your dataset in 2 or 3 dimensions. To simply see your answer just plot this transformed matrix into a 2d or 3d plot respectively. &lt;/li&gt;&#xA;&lt;li&gt;This helps you visualize a higher dimensionality data as a 2d or 3d entity so while using regression or some predictive modeling technique you can assess the trend of data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Should we use PCA in machine learning algorithms more often?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, that strictly depends, using PCA reduces the accuracy of your data set so unless you need to save up some space caused due to a lot of features with bad correlation and the overall accuracy doesn't matter. If your machine learning model scenario is similar to this then it is ok to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, most use of PCA is as you asked before for visualizing higher dimensionality data to determine the data trend and to check which model fits best. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-05-07T06:15:36.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18807" PostTypeId="2" ParentId="18804" CreationDate="2017-05-07T06:21:22.230" Score="6" Body="&lt;p&gt;&lt;code&gt;class_weight&lt;/code&gt; is used in the process of model-training to train(=fit) a better model (let us call it &lt;code&gt;clf&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question is is about choosing the most sure predictions.&#xA;You just need to predict the probability (for binary classification this will be the probability of positive class)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_test_predicted_probability = clf.predict_proba(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then choose 10 points with the highest &lt;code&gt;y_test_predicted_probability&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#some code to do this&#xA;top_picks_indexes = y_test_predicted_probability[:,1].argsort()[-10:] # chose top 10 probabilities for class = 1&#xA;&#xA;# create a vector, Y_top_picks, with all zeros except ones for the selected top probabilities&#xA;Y_top_picks = np.zeros(len(X_test))&#xA;Y_top_picks[top_picks_indexes] = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2017-05-07T14:23:27.340" LastActivityDate="2017-05-07T14:23:27.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18808" PostTypeId="2" ParentId="18782" CreationDate="2017-05-07T07:50:50.493" Score="2" Body="&lt;p&gt;Bounds on the needed amount of samples are very common in &lt;a href=&quot;https://en.wikipedia.org/wiki/Probably_approximately_correct_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;PAC learning&lt;/a&gt;.&#xA;When you define a concept class you can compute a minimal size set of sample that will enable learning. &#xA;However, &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;More samples will allow improving accuracy&lt;/li&gt;&#xA;&lt;li&gt;More samples will enable learning more complex concepts, that might fit your data better.&lt;/li&gt;&#xA;&lt;li&gt;As @Emre wrote, real life data sets usually are not clean as in PAC learning. The concept class is not given to you, the data has noise and a given distribution is not guaranteed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Showing that a classifier can be learnt with a small amount of data is great. &#xA;It is a big advantage of the learner.&#xA;However, more data usually helps and if it helps more than expected for such a classifier, it is possible that the classifier requirements doesn't hold.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-05-07T07:50:50.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18810" PostTypeId="1" CreationDate="2017-05-07T10:53:26.297" Score="13" ViewCount="8758" Body="&lt;p&gt;Background:&#xA;While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as stated here. (&lt;a href=&quot;https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks&quot;&gt;What is the &amp;quot;dying ReLU&amp;quot; problem in neural networks?&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&#xA;What im hoping to do is to implement a check in the code itself to check if the neurons are dead. After that, the code could refit the network if needed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, what is a good citeria to check for dead neurons? Currently im thinking of checking for low variance in the prediction as a citeria. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it helps, im using keras.&lt;/p&gt;&#xA;" OwnerUserId="32011" LastActivityDate="2019-11-27T16:36:39.817" Title="How to check for dead relu neurons" Tags="|machine-learning|neural-network|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18811" PostTypeId="2" ParentId="18810" CreationDate="2017-05-07T11:52:22.830" Score="7" Body="&lt;p&gt;A dead ReLU pretty much just means that its argument value is negative such that the gradient stays at 0; no matter how you train it from that point on. You can simply have a look at the gradient during training to see whether a ReLU is dead or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice you may simply want to use leaky ReLUs, i.e. instead of f(x) = max(0,x) you set f(x) = x if x &gt; 0 and f(x) = 0.01x if x &amp;lt;= 0. This way you always allow a small non-zero gradient and the unit should not get fully stuck in training anymore.&lt;/p&gt;&#xA;" OwnerUserId="32012" LastActivityDate="2017-05-07T11:52:22.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18813" PostTypeId="2" ParentId="18778" CreationDate="2017-05-07T15:11:20.210" Score="1" Body="&lt;p&gt;The question 'is this a good model?' is essentially a business question.&#xA;There is always a precision/recall tradeoff and you decide &lt;em&gt;based&lt;/em&gt; &lt;em&gt;on&lt;/em&gt; &lt;em&gt;your&lt;/em&gt; &lt;em&gt;business&lt;/em&gt; &lt;em&gt;goal&lt;/em&gt; what model to choose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data science can help you with the question on how to compare classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us start, that each (or nearly each) classifier can predict not only the outcome (Class2 or Class3), but also the &lt;em&gt;probability&lt;/em&gt; of this outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your &lt;code&gt;confusion matrix&lt;/code&gt; and all your metrics are suitable for making business decision, but, from data science point of view, have two &lt;em&gt;problems&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) they do not take into account the agreement by chance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) they are based on the threshold probability 0.5, which is not necessarily the optimal probability for deciding about the class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To deal with (1), have a look at &lt;a href=&quot;https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english&quot;&gt;Cohen's kappa&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To deal with (2), use metrics based on probability. The simpler one is &lt;a href=&quot;http://gim.unmc.edu/dxtests/roc3.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Area Under the ROC Curve&lt;/a&gt;, see also &lt;a href=&quot;https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it&quot;&gt;here&lt;/a&gt;. Even finer, but not so straightforward to interpret is &lt;a href=&quot;http://wiki.fast.ai/index.php/Log_Loss&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logarithmic Loss&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is recall depend on training data? How do I improve recall for class 2 along with accuracy?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Recall depends on the quality of your model (and thus also on the quality of your training data) and on the chosen probability threshold. By decreasing probability threshold, you improve recall but worsen precision. Which tradeoff to choose is again a business decision.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2017-05-07T15:11:20.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18814" PostTypeId="1" CreationDate="2017-05-07T19:19:10.267" Score="0" ViewCount="846" Body="&lt;p&gt;I am new to ANNs. How to handle features whicvh consist of strings? E.g. if I have group featire which are different groups conatining differnt capital Leters and numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have the train data set containing all such important features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first guess is just to replace stings with numbers. But you have different uncorrelated groups of strings like the category or the manufacturer.&lt;/p&gt;&#xA;" OwnerUserId="32027" LastEditorUserId="-1" LastEditDate="2017-05-09T08:18:01.313" LastActivityDate="2017-05-09T08:18:01.313" Title="How to handle string features in a neural network" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ClosedDate="2017-05-07T20:39:41.523" ContentLicense="CC BY-SA 3.0" />
  <row Id="18815" PostTypeId="2" ParentId="18814" CreationDate="2017-05-07T19:47:21.960" Score="1" Body="&lt;p&gt;Does the string add any useful data? No. You should replace categories by id's for sure, and manufacturers as well. Why would having uncorrelated groups of strings stop you from giving each an id? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;code&gt;Google = 0&lt;/code&gt;, &lt;code&gt;Apple = 1&lt;/code&gt;, etc. But that doesn't mean that a category can't have id &lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;. This is a different input, so these should start at &lt;code&gt;0&lt;/code&gt; too: &lt;code&gt;Books = 0&lt;/code&gt;, &lt;code&gt;Electronics = 1&lt;/code&gt;. You still have to normalize these id's for your network ofcourse (&lt;code&gt;value = id / maxId&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might be useful: &lt;a href=&quot;https://github.com/cazala/synaptic/wiki/Normalization-101&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/cazala/synaptic/wiki/Normalization-101&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-05-07T19:47:21.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18817" PostTypeId="1" CreationDate="2017-05-07T22:49:18.537" Score="1" ViewCount="3625" Body="&lt;p&gt;I  am working on a classification problem and I found my data having  a lot of outliers which has resulted in reduction in my recognition rate. I have tried rescaling, normalization techniques like min max, box cox and even log transformation. &#xA;I am considering of eliminating outliers from box plots but I am afraid I might be eliminating useful features/data required to define the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any suggestion on how to deal with such cases.&#xA;Also further analysis of data revealed that my data constitutes of features belong to dfferent process like web application, apps. I segregated the data based on the processes and I  do see that large variation of process  resulted in different accuracy ranging from 60-95%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any tips on how to deal with such cases? In the end I want my classifier to classify irrespective of the process type. So with my current issue, does this imply that my features defined are not good enough or is there something else I can do?&lt;/p&gt;&#xA;" OwnerUserId="32029" LastActivityDate="2017-05-08T11:44:11.977" Title="Classification affected by a lot of outliers in features? How do you deal with outliers?" Tags="|multiclass-classification|outlier|normalization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18818" PostTypeId="1" AcceptedAnswerId="18819" CreationDate="2017-05-08T05:13:24.117" Score="31" ViewCount="25994" Body="&lt;p&gt;I was going through an &lt;a href=&quot;http://blog.kaggle.com/2015/11/09/profiling-top-kagglers-gilberto-titericz-new-1-in-the-world/&quot; rel=&quot;noreferrer&quot;&gt;article&lt;/a&gt; on kaggle blogs. Repeatedly, the author mentions 'LB score' and 'LB fit') as a metric for effectiveness of machine learning (along with cross validation (CV) score).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a research for the meaning of 'LB' I spent quite a bit of time, I realised that generally people directly refer it as LB without much background. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is - What is a 'LB'?&lt;/p&gt;&#xA;" OwnerUserId="31446" LastActivityDate="2018-11-06T04:32:03.940" Title="What is a LB score in machine learning?" Tags="|machine-learning|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18819" PostTypeId="2" ParentId="18818" CreationDate="2017-05-08T05:15:36.853" Score="35" Body="&lt;p&gt;In the context of Kaggle, it means &lt;strong&gt;L&lt;/strong&gt;eader&lt;strong&gt;B&lt;/strong&gt;oard (emphasis mine).&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-05-08T05:15:36.853" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="18820" PostTypeId="1" AcceptedAnswerId="19959" CreationDate="2017-05-08T06:20:23.597" Score="0" ViewCount="2224" Body="&lt;p&gt;I'm using Tensorflow 1.1.0 with GPU support and I have this function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):&#xA;    &quot;&quot;&quot;&#xA;    Create an RNN Cell and initialize it.&#xA;    :param batch_size: Size of batches&#xA;    :param rnn_size: Size of RNNs&#xA;    :return: Tuple (cell, initialize state)&#xA;    &quot;&quot;&quot;&#xA;    # Basic LSTM cell&#xA;    # lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)&#xA;    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, forget_bias=0.0, state_is_tuple=True, reuse=tf.get_variable_scope().reuse)&#xA;&#xA;    # Add drop to the cell&#xA;    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)&#xA;&#xA;    # Stack multiple LSTM layers&#xA;    rnn_cell = tf.contrib.rnn.MultiRNNCell([drop for _ in range(layers)], state_is_tuple=True)&#xA;&#xA;    # Getting an initial state of zeros&#xA;    initial_state = rnn_cell.zero_state(batch_size, tf.float32)&#xA;&#xA;    # Set the name&#xA;    initial_state = tf.identity(initial_state, 'initial_state')&#xA;&#xA;    return rnn_cell, initial_state&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The previous code works perfectly with Tensorflow 1.0.0 if I use &lt;code&gt;lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)&lt;/code&gt; but now I have changed the way I get the lstm cell. And I get this error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Attempt to reuse RNNCell&#xA;   with a different variable scope than its&#xA;  first use.  First use of cell was with scope&#xA;  'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with&#xA;  scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'.  Please create a&#xA;  new instance of the cell if you would like it to use a different set&#xA;  of weights.  If before you were using:&#xA;  MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to:&#xA;  MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]).  If&#xA;  before you were using the same cell instance as both the forward and&#xA;  reverse cell of a bidirectional RNN, simply create two instances (one&#xA;  for forward, one for reverse).  In May 2017, we will start&#xA;  transitioning this cell's behavior to use existing stored weights, if&#xA;  any, when it is called with scope=None (which can lead to silent model&#xA;  degradation, so this error will remain until then.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have changed &lt;code&gt;get_init_cell&lt;/code&gt; with this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def lstm_cell(rnn_size, keep_prob=0.75):&#xA;    # Basic LSTM cell&#xA;    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)&#xA;&#xA;    # Add drop to the cell&#xA;    return tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)&#xA;&#xA;def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):&#xA;    &quot;&quot;&quot;&#xA;    Create an RNN Cell and initialize it.&#xA;    :param batch_size: Size of batches&#xA;    :param rnn_size: Size of RNNs&#xA;    :return: Tuple (cell, initialize state)&#xA;    &quot;&quot;&quot;    &#xA;    # Stack multiple LSTM layers&#xA;    rnn_cell = tf.contrib.rnn.MultiRNNCell([lstm_cell(rnn_size, keep_prob) for _ in range(layers)])&#xA;&#xA;    # Getting an initial state of zeros&#xA;    initial_state = rnn_cell.zero_state(batch_size, tf.float32)&#xA;&#xA;    # Set the name&#xA;    initial_state = tf.identity(initial_state, 'initial_state')&#xA;&#xA;    return rnn_cell, initial_state&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I still getting the same error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also try it with the same error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_init_cell(batch_size, rnn_size, keep_prob=0.75, layers=2):&#xA;    &quot;&quot;&quot;&#xA;    Create an RNN Cell and initialize it.&#xA;    :param batch_size: Size of batches&#xA;    :param rnn_size: Size of RNNs&#xA;    :return: Tuple (cell, initialize state)&#xA;    &quot;&quot;&quot;    &#xA;&#xA;    def lstm_cell():&#xA;        # Basic LSTM cell&#xA;        return tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)        &#xA;&#xA;    def attn_cell():&#xA;        return tf.contrib.rnn.DropoutWrapper(lstm_cell(), output_keep_prob=keep_prob)&#xA;&#xA;    # Stack multiple LSTM layers&#xA;    rnn_cell = tf.contrib.rnn.MultiRNNCell([attn_cell() for _ in range(layers)], state_is_tuple=True)&#xA;&#xA;    # Getting an initial state of zeros&#xA;    initial_state = rnn_cell.zero_state(batch_size, tf.float32)&#xA;&#xA;    # Set the name&#xA;    initial_state = tf.identity(initial_state, 'initial_state')&#xA;&#xA;    return rnn_cell, initial_state&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any idea?&lt;/p&gt;&#xA;" OwnerUserId="31726" LastActivityDate="2017-06-26T01:42:18.093" Title="Attempt to reuse RNNCell with a different variable scope" Tags="|tensorflow|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18822" PostTypeId="1" CreationDate="2017-05-08T08:27:23.280" Score="2" ViewCount="478" Body="&lt;p&gt;I am working on Single Document Summarization task on News datasets. I do some experiments in this task. A simple experiment that I make and has a good result is extracting sentences just from beginning of news. Now I want to find any paper or research result about this type of sentence selection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any research to show how good is to choose sentences just from beginning of text without any reordering?&lt;/p&gt;&#xA;" OwnerUserId="23377" LastEditorUserId="13727" LastEditDate="2017-05-11T09:17:35.310" LastActivityDate="2020-07-09T03:33:57.133" Title="Extract sentences from beginning of news in single document summarization" Tags="|nlp|text-mining|automatic-summarization|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18823" PostTypeId="2" ParentId="16342" CreationDate="2017-05-08T09:42:40.947" Score="30" Body="&lt;p&gt;&lt;code&gt;scale_pos_weight&lt;/code&gt; is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to &lt;code&gt;scale_pos_weight&lt;/code&gt; is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sum(negative instances) / sum(positive instances)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your specific case, there is another option in order to weight individual data points and take their weights into account while working with the booster, and let the optimization happen regarding their weights so that each point is represented equally. You just need to simply use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgboost.DMatrix(..., weight = *weight array for individual weights*)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can define the weights as you like and by doing so, you can even handle imbalances within classes as well as imbalances across different classes.&lt;/p&gt;&#xA;" OwnerUserId="32045" LastEditorUserId="560" LastEditDate="2017-08-26T08:40:53.430" LastActivityDate="2017-08-26T08:40:53.430" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18825" PostTypeId="2" ParentId="18817" CreationDate="2017-05-08T11:09:59.380" Score="0" Body="&lt;p&gt;I would try classifying using tree based models(&lt;a href=&quot;https://stats.stackexchange.com/questions/59124/random-forest-assumptions&quot;&gt;Random Forest short explanation&lt;/a&gt;) since they are less sensitive to outliers than linear models.&lt;br /&gt;&#xA;Addressing the different data-sources can be done by a creation of a categorical feature that describes data source.&lt;br /&gt;&#xA;Use this feature and train a tree, analyze if the the tree use this feature in its first steps(In &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sklearn&lt;/a&gt; its quiet simple).&lt;br /&gt;&#xA;If it is high, look at the sub-trees after each split, if the features used in each sub-tree are different than you should consider training a different model per each data-source.&lt;/p&gt;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-05-08T11:09:59.380" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18826" PostTypeId="2" ParentId="18817" CreationDate="2017-05-08T11:44:11.977" Score="1" Body="&lt;p&gt;If the number of outliers is small and you are concerned that they will destabilize your solution, you could attempt a random forest classifier. The RF fits trees to random selections of data and variables, and collects &quot;votes&quot; from each, thus reducing the impact of outlier valuers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if the number of outliers is fairly substantital, you might want to create a new class called &quot;outlier&quot;. In the training set, apply this label to those values you have deemed to be outliers and then fit the model with the augmented class. Check if the model correctly identifies outliers in the test set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is equivalent to removing the outliers, only it creates a repeatable and machine learning way of doing so.&lt;/p&gt;&#xA;" OwnerUserId="3077" LastActivityDate="2017-05-08T11:44:11.977" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18827" PostTypeId="2" ParentId="18782" CreationDate="2017-05-08T11:55:17.620" Score="1" Body="&lt;p&gt;Just to amplify @Dal's excellent answer, you certainly can fit models with small sample sizes. That's exactly what classical statistics attempts to do, often with success. But there is a price in terms of quality of the data &#xA;, simplicity of the model and experimental design. For example, to estimate interaction terms efficiently and credibly, you want to randomly apply the treatment effects. Machine learning typically occurs in observational data where none of these assumptions are correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, 102 samples is way to small in a problem with 11 variables and a binary outcome. Classifications typically take more data than regression problems (continuous outcome). That's why you often hear pollsters using 1000 respondents to predict a categorical outcome on the basis of one or two features (questions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an optimistic rule of thumb that one needs 10 variables for each parameter that one wants to estimate. I have always thought this was a bit thin, but even on that measure, your sample is too small. &lt;/p&gt;&#xA;" OwnerUserId="3077" LastActivityDate="2017-05-08T11:55:17.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18828" PostTypeId="1" CreationDate="2017-05-08T13:27:45.267" Score="6" ViewCount="10120" Body="&lt;p&gt;I have to solve a ranking ML issue. To start with, I have successfully applied the pointwise ranking approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm playing around with pairwise ranking algorithms. I've created the pairwise probabilities (i.e. probability of item i being above item j) but I'm not sure how I can transform this to rankings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the historical data (let's assume these are queries), I have their pairwise probs AND the actual ranking (the ideal one). I want a solution that will provide a ranking for a new query as well (i.e. the ideal ranking is what I'm looking for here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any python package that has, at least partially, the functionality I'm looking for?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I have the pairwise probs for for &lt;strong&gt;all&lt;/strong&gt; possible pairs of i and j.&lt;/p&gt;&#xA;" OwnerUserId="13833" LastEditorUserId="13833" LastEditDate="2017-05-16T15:08:46.550" LastActivityDate="2017-05-19T10:40:10.920" Title="From pairwise comparisons to ranking - python" Tags="|machine-learning|python|ranking|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18829" PostTypeId="1" CreationDate="2017-05-08T13:29:19.747" Score="1" ViewCount="1047" Body="&lt;p&gt;I have a list of some 100 millions of strings, each of different length. Examples:&#xA;    &lt;code&gt;nsdgnlnesef&lt;/code&gt;   &lt;code&gt;ngmrlxkvgrmksefsfnlj&lt;/code&gt;   &lt;code&gt;rnrfnmsbanana&lt;/code&gt;   &lt;code&gt;housemgslen&lt;/code&gt;    &lt;code&gt;assistremovecouch&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see the strings vary in length and some can include english words, some not, some both words and random characters. All use the characters [a-z]. It is possible to characterise the strings are being humanly created or by a algorithm with a reasonable certainty. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later I aim to also classify between different sub-types of algorithms generating these strings (&lt;em&gt;multiclass classification&lt;/em&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 100 millions strings I have are already labeled, based on other analysis. But because this is a very slow task, I will use Machine Learning to try and predict the origin of new strings as we get new data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The methods I have started with are &lt;code&gt;bigrams&lt;/code&gt;, &lt;code&gt;Long Short Term Memory (LSTM)&lt;/code&gt; and &lt;code&gt;Gated Recurrent Unit (GRU)&lt;/code&gt;. I will eventually also run trigrams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As both time and accuracy are very important factors of the method I eventually choose, I want to make sure I test as many as possible. But since there is a forest of methods, I hope to get some helpful advice here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imagine methods based on features, such as &lt;code&gt;Random Forest&lt;/code&gt; and &lt;code&gt;Support Vector Machine&lt;/code&gt; could be useful, but I haven't them used before, and I am not certain if they or other methods could handle 100 millions of strings in an efficient way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions for methods to look at are very welcomed. &lt;/p&gt;&#xA;" OwnerUserId="31948" LastEditorUserId="31948" LastEditDate="2017-05-08T14:11:32.397" LastActivityDate="2017-05-08T14:11:32.397" Title="Methods for string classifications" Tags="|machine-learning|classification|multiclass-classification|text|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18830" PostTypeId="1" CreationDate="2017-05-08T14:10:17.860" Score="6" ViewCount="4912" Body="&lt;p&gt;I have large set of data records looking like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;text&quot;, &quot;category&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per category, like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;ngram1&quot;, &quot;category1&quot;, 1000&#xA;&quot;ngram1&quot;, &quot;category2&quot;, 20&#xA;&quot;ngram1&quot;, &quot;category3&quot;, 15&#xA;&quot;ngram2&quot;, &quot;category1&quot;, 25&#xA;&quot;ngram2&quot;, &quot;category2&quot;, 550&#xA;&quot;ngram2&quot;, &quot;category3&quot;, 600&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a clustering or classification algorithm that could help with finding groups within categories, based on counts of same n-grams within categories?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above example that algorithm should be able to show that &quot;category2&quot; and &quot;category3&quot; are part of the same group.&lt;/p&gt;&#xA;" OwnerUserId="32048" LastEditorUserId="6433" LastEditDate="2020-03-19T21:03:22.207" LastActivityDate="2020-04-19T02:01:00.120" Title="Clustering or classifing n-gram-based text categories" Tags="|classification|clustering|text-mining|ngrams|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18831" PostTypeId="1" CreationDate="2017-05-08T14:21:35.473" Score="-2" ViewCount="823" Body="&lt;p&gt;I'm training a XGBoost regressor in Python on a data set with a large number of indicator variables (one-hot-encoded from categorical variables) and a few numerical variables.The dataset size is over a million rows with a total column number of ~1000. The parameter I used was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param = {&#xA;    'objective':            'reg:linear',&#xA;    'bst:max_depth':        6, &#xA;    &quot;min_child_weight&quot;:     1,&#xA;    'gamma':                5,&#xA;    'max_delta_step':       1,&#xA;    'bst:eta':              0.01,&#xA;    'nthread':              16,&#xA;    'verbose':              1&#xA;}&#xA;num_rounds = 1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I checked the param of fitted trees and seems like most of them have a depth of 1 or 2. This happened for this feature size of ~1000; for a smaller feature size, the prediction result seems OK (but theoretically ~1000 would provide a better prediction result, so that's why I want it to work).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RHWqc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RHWqc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that there's only very few variables are used in the tree construction and results in most of the trees being identical? Or am I doing anything very stupid?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is quite large and I'm not sure how to provide a runnable sample. Any possible suggestions or general discussions are welcomed!&lt;/p&gt;&#xA;" OwnerUserId="14152" LastActivityDate="2017-10-05T20:44:50.627" Title="Q: xgboost regressor training on a large number of indicator variables results in same prediction for all rows in test" Tags="|machine-learning|python|xgboost|prediction|parameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18832" PostTypeId="2" ParentId="18831" CreationDate="2017-05-08T16:10:11.637" Score="0" Body="&lt;p&gt;Did you check the separation gain of individual variables? It might be that most of them have no separation power. This could result in the same prediction for everything. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-05-08T16:10:11.637" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18833" PostTypeId="2" ParentId="18828" CreationDate="2017-05-08T17:37:08.977" Score="0" Body="&lt;p&gt;One option is to create a &lt;a href=&quot;https://en.wikipedia.org/wiki/Directed_acyclic_graph&quot; rel=&quot;nofollow noreferrer&quot;&gt;directed acyclic graph (DAG)&lt;/a&gt; from the pairwise probabilities, where the nodes are the items and the direction of the connections are driven by the pairwise probabilities (the connection goes from item A to item B if &lt;code&gt;p(A &amp;gt; B) &amp;gt; 0.5&lt;/code&gt;, else the connection goes from B to A), and then compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Topological_sorting&quot; rel=&quot;nofollow noreferrer&quot;&gt;topological sorting&lt;/a&gt; of the graph. This would give you a sequence of nodes that respects the pairwise orderings derived from the probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The python code to implement topological sort can be implemented from the algorithm, but there are python packages like &lt;a href=&quot;https://pypi.python.org/pypi/toposort/1.0&quot; rel=&quot;nofollow noreferrer&quot;&gt;toposort&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-08T17:37:08.977" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18836" PostTypeId="1" AcceptedAnswerId="18837" CreationDate="2017-05-08T18:56:04.020" Score="3" ViewCount="316" Body="&lt;p&gt;Machine Learning contests like Kaggle usually layout the machine learning task in a human-understandable way. E.g. they might tell you the meaning of the input (features). But what if a Machine Learning contest doesn't want to expose the meaning of its input data? One approach I can think of is to apply a (random) rotation matrix to the features, so that each resulting feature doesn't have obvious meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A rotation on the input space shouldn't change a model's ability to separate the positives from the negatives (using binary classification as an example) -- after all the same hyper plan (when applied the same rotation) can be used to separate the examples. What could be changed by the rotation is the distribution of each feature (i.e. when looking at a single feature's values across all examples) if a contestant cares about them. However, rotation is PCA-invariant, so if a contestant decides to work on the PCA-ed version of the input then the rotation doesn't change anything there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features? Is there any (other) thing that I should be aware of that a rotation can change for a contestant during such a contest?&lt;/p&gt;&#xA;" OwnerUserId="29428" LastActivityDate="2017-05-08T20:39:45.030" Title="How will a rotation matrix affect contestants in machine learning contests?" Tags="|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18837" PostTypeId="2" ParentId="18836" CreationDate="2017-05-08T19:28:53.710" Score="5" Body="&lt;p&gt;Kaggle competitions with clean, anonymised and opaque numerical features are often popular. My opinion is they are popular because they are more universally accessible - all you need is to have studied at least one ML supervised learning approach, and maybe have a starter script that loads the data, and it is very easy to make a submission. The competitions become very focused on optimising parameters, picking best model implementations and ensembling techniques. The more advanced competitors will also refine and check their CV approaches very carefully, trying to squeeze the last iota of confidence out of them in order to beat the crowd climbing the public leaderboard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples of historic Kaggle competitions with obfuscated data might be &lt;a href=&quot;https://www.kaggle.com/c/otto-group-product-classification-challenge/data&quot; rel=&quot;noreferrer&quot;&gt;Otto Group Product Classification&lt;/a&gt; or &lt;a href=&quot;https://www.kaggle.com/c/bnp-paribas-cardif-claims-management/data&quot; rel=&quot;noreferrer&quot;&gt;BNP Paribas Cardif CLaims Management&lt;/a&gt;. For some of these competitions the data is adjusted for anonymity of the users who might otherwise be identified from the records. In other cases it is less clear what the sponsor's motivation is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are negative consequences (you will find these complained about in the same competitions):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use of insight from domain knowledge, or exploration/study of the underlying principles from the subject being predicted are effectively blocked. It is hard to assess the impact of this, but it is possible that the sponsors miss out on potentially better models.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Doing &quot;just&quot; the machine learning side can be a bit too mechanical and boring for some competitors, who may not not try as hard.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are always data explorations and views of data published in forums (and Kaggle's scripts - called kernels), and many people view, upvote and presumably use the insights from them. I recall at least one competition forum thread where there was a lot of discussion about weird patterns appearing in data, which were probably an artefact of obfuscation (sorry I cannot find the thread now).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With obfuscated data, there can be attempts to de-obfuscate, and they have sometimes been partially successful.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-08T20:39:45.030" LastActivityDate="2017-05-08T20:39:45.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18839" PostTypeId="1" CreationDate="2017-05-08T20:11:01.717" Score="3" ViewCount="3908" Body="&lt;p&gt;I learn about neural networks with a heavy excercise. I have products and want to forecast the revenue. I have 10 features. But 4 features  have large numbers of expressions. So my &quot;group&quot; features has over 500 different groups (of products). If I do binarization with the features I end up with vectors of over 1000 dimensions as input.&#xA;All the product classes are names with letters and numbers such as &quot;100XA9&quot;. Now there are 500 of those. For a neural network I have to do binarization with those names. Is that right?&#xA;Then my input layer is 1000, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this &quot;normal&quot;? And is this problematic for computation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean this is a typical question. A store with many products (like over 500) will forecast its revenue based on the prices and other features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope you can help me to untestand how to use neural networks for classification.&lt;/p&gt;&#xA;" OwnerUserId="32068" LastEditorUserId="995" LastEditDate="2017-05-09T08:17:58.717" LastActivityDate="2017-05-09T18:51:37.413" Title="neural network with large numbers of inputs" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18840" PostTypeId="1" CreationDate="2017-05-08T22:08:23.433" Score="3" ViewCount="498" Body="&lt;p&gt;LDA (linear discriminant analysis), SVMs with a linear kernel, and perceptrons are linear classifiers. Is there any other relationship between them, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by LDA can be found by linear SVM&lt;/li&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by linear SVM can be found by LDA.&lt;/li&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by LDA can be found by a perceptron&lt;/li&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by linear SVM can be found by a perceptron.&lt;/li&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by a perceptron can be found by LDA&lt;/li&gt;&#xA;&lt;li&gt;Every decision boundary that can be found by a perceptron can be found by an SVM with a linear kernel&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Always on the same data, of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I think the linear SVM can find more decision boundaries than a perceptron due to slack variables. While the perceptron finds just an arbitrary hyperplane which separates the data linearly (if such a hyperplane exists), the linear SVM will always find the same hyperplane due to the optimality criterium.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-05-08T22:14:14.843" LastActivityDate="2018-02-02T14:20:27.727" Title="Is there a relationship between LDA, linear SVMs and Perceptron?" Tags="|classification|svm|lda-classifier|perceptron|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18841" PostTypeId="2" ParentId="18828" CreationDate="2017-05-08T23:08:58.193" Score="0" Body="&lt;p&gt;Use a framework for ranking that is designed to supporting ranking, such as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bradley-Terry model&lt;/a&gt;.  See also &lt;a href=&quot;https://en.wikipedia.org/wiki/Elo_rating_system#Theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elo rankings&lt;/a&gt; and all the general statistical theory on &lt;a href=&quot;https://en.wikipedia.org/wiki/Pairwise_comparison&quot; rel=&quot;nofollow noreferrer&quot;&gt;pairwise comparison&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-08T23:08:58.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18842" PostTypeId="1" CreationDate="2017-05-08T23:55:19.620" Score="3" ViewCount="8461" Body="&lt;p&gt;The scikit-learn docs say it is the signed distance of that sample to the hyperplane.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've taken the sum of the weights and their corresponding coefficient and added the intercept to that sum but this does not return the value given by the &lt;code&gt;decision_function&lt;/code&gt; method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing classification tasks how is the signed distance different from taking the linear summation of the weights, feature values, and intercept?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, If I have the following feature values, intercept, and weights I would calculate the distance from the point to the hyperplane as...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_1 = 3, x_2 = 2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_1 = -0.28, w_2 = -0.20$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b = -0.68$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta^T \cdot \textbf{X} + b$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When sklearn calculates the &lt;code&gt;decision_function&lt;/code&gt; it returns&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return safe_sparse_dot(X, self.coef_.T,&#xA;                               dense_output=True) + self.intercept_ &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;self&lt;/code&gt; is our fitted classifier and X is the example for which we are calculating the decision function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I replicate what &lt;code&gt;decision_function&lt;/code&gt; is doing?&lt;/p&gt;&#xA;" OwnerUserId="32071" LastEditorUserId="32071" LastEditDate="2017-05-09T14:47:34.213" LastActivityDate="2018-03-02T12:16:54.560" Title="How does scikit-learn decision function method work?" Tags="|scikit-learn|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18843" PostTypeId="1" CreationDate="2017-05-08T23:59:22.497" Score="1" ViewCount="211" Body="&lt;p&gt;I have a dataframe with 20 columns. Data is integrated from five different data sources. I have shown 4 columns from each dataset. 4/5 sources have discrete data (either 1 or 0). The last section has continuous values which is my target (Contains NA). I intend to implement Random Forest algorithm to predict the missing values. I have seen tutorials which predict only one Y variable. Is this a regression tree for missing value prediction task for continuous multioutput Y? kindly share some tutorial links which could provide some reference code in R.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Input File Sample&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  D000022 D000038 D000152 D000163 A01AB06 A01AB07 A01AB08 A01AB09 76 77 80 84 fp131 fp132 fp133 fp134 C0000727 C0000731 C0000737 C0000768&#xA;1       1       0       0       1       0       1       0       0  1  0  1  0     0     1     0     1     0.03       0    0.0406    0.26&#xA;2       1       1       0       0       0       0       0       0  0  1  0  1     1     1     0     0       NA       NA      NA     NA&#xA;3       0       0       0       0       0       0       0       0  1  0  1  0     0     1     1     0        1        0   0.1900    0.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="31599" LastEditDate="2017-05-11T14:21:52.250" LastActivityDate="2017-05-11T14:21:52.250" Title="R tutorial for multitask regression tree for missing value prediction?" Tags="|machine-learning|r|" AnswerCount="0" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="18844" PostTypeId="1" AcceptedAnswerId="18846" CreationDate="2017-05-09T03:46:27.270" Score="1" ViewCount="1399" Body="&lt;p&gt;I wonder what are the recommended &lt;code&gt;eval_metric&lt;/code&gt;s for &lt;code&gt;count:poisson&lt;/code&gt; as objective in &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost&lt;/a&gt; in R?&lt;/p&gt;&#xA;" OwnerUserId="29204" LastEditorUserId="29204" LastEditDate="2017-05-09T04:19:02.470" LastActivityDate="2017-05-09T06:21:37.210" Title="R, xgboost: eval_metric for count:poisson" Tags="|r|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18846" PostTypeId="2" ParentId="18844" CreationDate="2017-05-09T06:21:37.210" Score="1" Body="&lt;p&gt;This is from XGBOOST documents:&#xA;“poisson-nloglik”: negative log-likelihood for Poisson regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might want to go and read more about poisson n loglikelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link for you to get started:&#xA;&lt;a href=&quot;https://onlinecourses.science.psu.edu/stat504/node/31&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://onlinecourses.science.psu.edu/stat504/node/31&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this can be helpful&lt;/p&gt;&#xA;" OwnerUserId="32045" LastActivityDate="2017-05-09T06:21:37.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18847" PostTypeId="2" ParentId="18830" CreationDate="2017-05-09T07:49:50.407" Score="2" Body="&lt;p&gt;If your ultimate goal is to cluster similar categories and assuming that you have labels of each text as category1, category2,...,categoryN from 1 to N, a bag of words method would be sufficient in order to create features so that you can run multiple desired clustering algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-means can be a good starting point for getting similar groups of text document categories, you can try out different k for more sensible outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Methodology that I would adapt here, forgive me if understood your problem wrong, would be to concatenate all the textual data for a given category and create 1,2,3,..n gram vectors. The values of these vectors can be TF-IDF of a given n-gram, and so on. Still different feature matrices can be formed to compare category vectors but TF-IDF is a pretty common way to do it in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After acquiring n-gram feature vectors for each text category, you can apply k-means algorithm with several distance metrics to find similar categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have 5 categories of text as given:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Soccer Games&lt;/li&gt;&#xA;&lt;li&gt;Basketball Games&lt;/li&gt;&#xA;&lt;li&gt;Politics&lt;/li&gt;&#xA;&lt;li&gt;Movies&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The proposed method can capture the similarity between soccer and basketball games as they are both sports. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this can help&lt;/p&gt;&#xA;" OwnerUserId="32045" LastActivityDate="2017-05-09T07:49:50.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18848" PostTypeId="1" AcceptedAnswerId="24746" CreationDate="2017-05-09T08:33:11.113" Score="10" ViewCount="12605" Body="&lt;p&gt;I have a set of data composed of time series (8 points) with about 40 dimensions (so each time series is 8 by 40). The corresponding ouput (the possible outcomes for the categories ) is eitheir 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What would be the best approach to design a classifier for time series with multiple dimensions ?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial strategy was to extract features from those time series : mean, std, maximum variation for each dimension. I obtained a dataset which I used to train a RandomTreeForest. Being aware of the total naivety of this, and after obtaining poor results, I am now looking for a more improved model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My leads are the following :&#xA;classify the series for each dimension (using KNN algorithm and DWT), reduce the dimensionality with PCA and use a final classifier along the multidimensions categories. &#xA;Being relatively new to ML, I don't know if I am totally wrong.&lt;/p&gt;&#xA;" OwnerUserId="32083" LastActivityDate="2018-01-12T20:23:05.697" Title="Classify multivariate time series" Tags="|classification|time-series|pca|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18849" PostTypeId="1" CreationDate="2017-05-09T08:57:50.920" Score="2" ViewCount="81" Body="&lt;p&gt;I have reached this point in my data sceince journey that I am left wondering if jobs are the only way out for me...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am highly skilled in Hadoop, Machine Learning and now getting into Deep Learning algorithms...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am bored with having to be employed, I want a way out...any ideas? I would still like to use my skills in my life but what other options exist out here...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open to ideas and collaborations....&lt;/p&gt;&#xA;" OwnerUserId="32084" LastActivityDate="2017-05-09T08:57:50.920" Title="Are jobs the only way out for data scientists?" Tags="|machine-learning|deep-learning|apache-hadoop|" AnswerCount="0" CommentCount="3" ClosedDate="2017-05-09T09:17:40.220" ContentLicense="CC BY-SA 3.0" />
  <row Id="18851" PostTypeId="2" ParentId="18839" CreationDate="2017-05-09T09:38:35.870" Score="1" Body="&lt;p&gt;One-hot encoding is the normal approach, and yes you would end up with 500 features for your group alone. Depending on how much training data you have this does not need to be a problem. If you have a lot of other features without direct interactions between the individual products you could use an Embedding layer before adding them to the rest, which maps your sparse categorical one-hot encoded features into a dense space via backpropagation. This will reduce the amount of parameters significantly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have enough training data to do this, you could look at gathering statistics about your product, that say something about other features for the group, or about the target for other rows (be sure not to use the current row, which would introduce target leakage) which would allow you to drop the category all together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turning them into IDs and then using this as a numerical feature is a bad idea, since there is no inherent structure in the numbers, which means there is no signal in that and it would need a very high complexity to learn how to distinguish 1 and 5 being very similar but 2 and 4 very different.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-09T09:38:35.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18852" PostTypeId="2" ParentId="11159" CreationDate="2017-05-09T11:57:29.407" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;em&gt;(Easier if followed)&lt;/em&gt; The response variable should be one team's outcome. E.g. In India vs. Australia, your response variable can be either India or Australia depicting whether that team won or not&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;(Easier if followed)&lt;/em&gt; Once you've done so, check for class populations so as to avoid predicting on unbalanced classes. E.g. If you select Australia, you need to make sure that they have equal wins and losses or somewhat balanced outcome. If Australia wins 90 out of 100 matches, it will add more steps to developing your prediction model&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;(Imbalanced classes)&lt;/em&gt; Check out class &lt;strong&gt;undersampling&lt;/strong&gt; or &lt;strong&gt;oversampling&lt;/strong&gt; techniques. R provides packages such as &lt;strong&gt;ROSE&lt;/strong&gt; and &lt;strong&gt;SMOTE&lt;/strong&gt; which handle this imbalance based on defined parameters.&lt;/li&gt;&#xA;&lt;li&gt;I haven't looked at the dataset that you are using but if there are a lot of predictors, you can use &lt;strong&gt;PCA&lt;/strong&gt; to reduce the number of predictors and make the computation quicker and less prone to erroneous interactions and predictors. Logically evaluate correlation results to detect highly correlated variables (Correlation does not imply causation)&lt;/li&gt;&#xA;&lt;li&gt;Try using &lt;strong&gt;logistic regression&lt;/strong&gt; to start your model building process. You can also adopt the &lt;strong&gt;decision tree&lt;/strong&gt; algorithm followed by &lt;strong&gt;random forests&lt;/strong&gt; which use voting mechanism to classify outcomes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="18369" LastActivityDate="2017-05-09T11:57:29.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18853" PostTypeId="1" CreationDate="2017-05-09T12:35:15.617" Score="1" ViewCount="254" Body="&lt;p&gt;The data table below contains cash flows going into an account. At index 0 the account starts with 50 in debt, is charged a monthly Rent of 90, and pays a certain amount every month, as well as getting additional supporting payments every 4 periods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cYdOS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cYdOS.png&quot; alt=&quot;Initial Data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming we know the first 8 periods (in order to form an opinion on the account, and to notice the seasonality of &quot;Support&quot;), how do we define the true balance based on previous and potentially future figures? The goal is to state how much someone actually has to pay, based on their current (and future forecasted) payment patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, as below, if a &quot;Support&quot; transaction is missed in a week where it was expected, the true balance spikes up, but then if that payment was received, the true balance goes back down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DIdSc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DIdSc.png&quot; alt=&quot;Missed Support&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've managed to confirm the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rent is a known array (for past and future)&lt;/li&gt;&#xA;&lt;li&gt;Payment is historically known and can be forecasted using ARIMA&lt;/li&gt;&#xA;&lt;li&gt;Support is historically known and can be forecasted using Exponential Smoothing&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've tried to find a way to create a regression equation based on historical and forecasted values (so that the true balance takes into accounts expected payments), but so far I've not managed to do so. It feels like I'm hitting a brick wall and I would appreciate information if the way I'm attempting to do so is correct or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The information provided here contains very simplified data. In reality, &quot;Payment&quot; can be highly irregular, while &quot;Support&quot; can either be a seasonal transaction that can be delayed by a number of periods, or can periodically stop and start again. I have so far been prototyping in Excel, as I'm unsure what to use to find a solution in Python.&lt;/p&gt;&#xA;" OwnerUserId="24830" LastEditorUserId="24830" LastEditDate="2017-05-09T15:02:03.947" LastActivityDate="2017-05-09T15:02:03.947" Title="Regression (and Forecasting) on Rent" Tags="|python|predictive-modeling|regression|forecast|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="18854" PostTypeId="1" AcceptedAnswerId="19666" CreationDate="2017-05-09T13:03:41.660" Score="6" ViewCount="2155" Body="&lt;p&gt;I understand that BatchNorm (Batch Normalization) centers to (mean, std) = (0, 1) and potentially scales (with $ \gamma $) and offsets (with $ \beta $) the data which is input to the layer. BatchNorm follows this formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MZbH7.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MZbH7.png&quot; alt=&quot;Vanilla BatchNorm&quot;&gt;&lt;/a&gt;&#xA;(retrieved from arxiv-id 1502.03167)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when it comes to 'adaptive BatchNorm', I don't understand what the difference is. What is adaptive BatchNorm doing differently? It is described as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a92TL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a92TL.png&quot; alt=&quot;adaptive BatchNorm&quot;&gt;&lt;/a&gt;&#xA;(retrieved from arxiv-id 1603.04779)&lt;/p&gt;&#xA;" OwnerUserId="31810" LastEditorUserId="31810" LastEditDate="2017-05-09T13:17:13.817" LastActivityDate="2017-06-13T07:22:12.913" Title="What is the difference between BatchNorm and Adaptive BatchNorm (AdaBN)?" Tags="|machine-learning|neural-network|deep-learning|normalization|domain-adaptation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18856" PostTypeId="1" CreationDate="2017-05-09T15:47:42.607" Score="1" ViewCount="1017" Body="&lt;p&gt;I am reading the paper with the title &quot;Classification of indoor actions through deep neural networks&quot;. And I came across this statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;With this mechanism very complex functions can be learned combining&#xA;  these modules: the resulting networks are often very sensitive to&#xA;  minute details and insensitive to large irrelevant variations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have read a lot about deep neural network. However, I might have missed the logic behind why deep neural networks are insensitive to large irrelevant features.&lt;/p&gt;&#xA;" OwnerUserId="26522" LastActivityDate="2017-05-09T17:13:13.220" Title="How neural networks handle large variations in the input?" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18857" PostTypeId="1" CreationDate="2017-05-09T16:33:29.800" Score="2" ViewCount="82" Body="&lt;p&gt;The aspect of over-fitting is typically viewed from the perspective of both- accuracy and model complexity. &#xA;To mitigate over-fitting, we usually have the practical approach of having k-fold validations, training-validation-test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: theoretically, can we leverage the Statistical Learning theory (COLT) to draw bounds on the confidence of how well learning has happened, &#xA;and how well classification can happen over unseen examples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, many a time we consider the minimum number of samples (upper bound/sufficient samples) needed to learn (VC Dimension). &#xA;Agreed, it is an overestimate, plus |H| or VC Dimension may be unknown in practice. &#xA;The other perspective wherein, given the number of samples 'm', and 'delta' (probability of failure) -&#xA;we find error bound (on unseen examples)- Can we theoretically interpret this error bound to be an estimate of over-fitting? I get this intuition from Nello Cristianini's book on SVMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate any pointers.&lt;/p&gt;&#xA;" OwnerUserId="32095" LastActivityDate="2017-05-09T16:33:29.800" Title="Overfitting and COLT/Statistical Learning Theory" Tags="|overfitting|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18858" PostTypeId="2" ParentId="18856" CreationDate="2017-05-09T17:13:13.220" Score="1" Body="&lt;p&gt;They can handle large variations of inputs because the neurons have weights and those weights get optimized as part of learning a good model. So even though the model might take in a value that ranges from 0 to 100,000, if it isn't relevant to predicting the outcome, it will have a very small effect.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-09T17:13:13.220" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18860" PostTypeId="1" CreationDate="2017-05-09T21:07:28.877" Score="3" ViewCount="10111" Body="&lt;p&gt;how to interpret agglomerative coefficient in agnes() function of cluster package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res.agnes &amp;lt;- agnes(dataset,method = 'ward')  &#xA;res.agnes$ac  &#xA;0.934  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31977" LastActivityDate="2018-03-14T10:30:23.040" Title="How to interpret agglomerative coefficient agnes() function?" Tags="|r|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18862" PostTypeId="1" CreationDate="2017-05-09T21:53:14.403" Score="1" ViewCount="145" Body="&lt;p&gt;I’m attempting to implement the modified Kneser-Ney smoothing algorithm on  a 4-gram model.  This smoothing algorithm discounts the higher order probabilities and redistributes the discount among the lower order probabilities by multiplying the lower order probabilities by a weighting factor lambda.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda_{low}(w_{i-n+1}^{i-1}) = \frac{D_{n,1} N_1(w_{i-n+1}^{i-1}\bullet) + D_{n,2} N_2(w_{i-n+1}^{i-1}\bullet) + D_{n,3+} N_{3+}(w_{i-n+1}^{i-1}\bullet)}{N_{1+}(\bullet w_{i-n+1}^{i-1}\bullet)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenging i'm facing is that the lambda weighting factor is indeed zero at the higher orders, thus producing a zero probability for the n-Gram. For instance, given the following n-Gram &quot;only seeing last word&quot; in which &quot;only seeing last&quot;, &quot;seeing last&quot;, and &quot;last&quot; are unseen, I get a zero probability regardless of the continuation count for &quot;word&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read mounds of literature on the Kneser-Ney Smoothing algorithm, but I've not found anything that specifically addresses this conundrum. Any insights on the implementation of the Kneser-Ney algorithm in such situations would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="30898" LastActivityDate="2017-05-09T21:53:14.403" Title="Modified Kneser-Ney Zero Weighting Factor" Tags="|nlp|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18863" PostTypeId="2" ParentId="17885" CreationDate="2017-05-09T23:07:57.877" Score="5" Body="&lt;p&gt;The code has been changed to remove headers.  See comment on github:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Newsgroups message contains header like 'Newsgroups: alt.atheism', which inflates the accuracy to 0.95 (2 epochs).&#xA;After removing the header, the val accuracy is 0.47 (2 epochs) and 0.71 (10 epochs).&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/pull/5585&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/fchollet/keras/pull/5585&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This confused me for days!&lt;/p&gt;&#xA;" OwnerUserId="32104" LastActivityDate="2017-05-09T23:07:57.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18864" PostTypeId="1" CreationDate="2017-05-10T04:39:03.917" Score="1" ViewCount="1311" Body="&lt;p&gt;I am reading this paper regarding action recognition based on sensor data. And this paper first transforms the input to spectrogram (time vs frequency, it's like the Fourier Transform of each segment of the input time series but square it). So instead of the time series you are given a matrix of st x sf (st is number of time windows, sf is number of frequency components considered) as input to you CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is this algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iZMna.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iZMna.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain this? Especially the formula for CNN, I don't get why you have to convolve the (t-1) and dw.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/wnRO2.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wnRO2.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26522" LastEditorUserId="29169" LastEditDate="2021-02-16T02:42:31.300" LastActivityDate="2021-02-16T02:42:31.300" Title="Convolutional Neural Networks for Sensor Data" Tags="|deep-learning|neural-network|convolutional-neural-network|convolution|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="18865" PostTypeId="1" CreationDate="2017-05-10T06:14:28.907" Score="4" ViewCount="1217" Body="&lt;p&gt;Lets say I have a dataset like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word    label_numeric&#xA;0   active  0&#xA;1   adventurous 0&#xA;2   aggressive  0&#xA;3   aggressively    0&#xA;4   ambitious   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I use a word2Vec trained model and convert each word into their word vector of 300 dimensions. This is how it looks now.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99  100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 label&#xA;0   0.058594    -0.016235   -0.174805   0.072266    -0.201172   0.073242    -0.074219   -0.149414   0.245117    -0.050049   -0.016357   -0.147461   -0.003311   0.071289    -0.008545   -0.179688   0.001686    -0.009949   -0.036621   0.048096    -0.033447   0.105957    -0.490234   0.249023    -0.199219   -0.025635   -0.248047   0.136719    -0.068848   -0.320312   0.259766    -0.053223   0.154297    -0.050537   0.110840    0.027100    0.000412    -0.133789   0.077148    0.058838    0.230469    -0.033203   -0.179688   -0.125977   -0.166992   -0.110352   -0.365234   -0.330078   -0.021729   -0.076660   0.124023    -0.107910   -0.051758   0.127930    0.192383    0.025024    0.033691    -0.386719   -0.006195   -0.074219   -0.175781   -0.088379   -0.341797   0.145508    -0.051758   0.099609    0.020874    -0.042969   -0.145508   0.090332    0.096191    0.061768    0.209961    0.314453    -0.080078   -0.304688   0.238281    -0.060791   0.146484    0.041504    -0.113281   0.019409    0.328125    0.300781    -0.153320   -0.174805   -0.347656   -0.002167   0.115723    0.104004    0.012817    -0.175781   0.088867    -0.291016   -0.092773   0.144531    -0.006256   -0.066406   -0.145508   -0.182617   -0.144531   0.074707    -0.157227   -0.025513   -0.013977   -0.289062   0.051514    -0.010559   0.121582    0.072754    0.005188    -0.162109   -0.246094   0.002014    -0.072266   -0.026733   0.143555    0.067383    0.398438    -0.212891   0.029663    -0.041748   -0.005157   0.337891    -0.192383   -0.135742   0.226562    -0.033691   -0.188477   0.322266    0.136719    -0.058594   -0.068359   0.136719    0.029175    -0.152344   -0.086426   0.021729    -0.005524   0.115723    0.106445    0.257812    0.000546    -0.161133   -0.046875   -0.049805   -0.058594   -0.110840   0.029907    -0.322266   -0.032715   -0.136719   -0.148438   0.125977    -0.205078   0.027222    -0.005219   -0.188477   0.318359    0.002792    0.155273    0.261719    -0.043457   0.113281    0.142578    0.170898    -0.202148   0.028687    0.239258    0.033203    -0.330078   -0.003647   -0.054199   -0.142578   0.201172    0.053467    -0.249023   -0.180664   0.147461    -0.036865   -0.015259   -0.107910   -0.134766   0.052002    0.109863    0.067871    0.022705    0.058838    -0.189453   -0.093262   -0.043945   -0.009216   0.020386    -0.232422   -0.083008   0.062500    0.016479    0.033936    0.041016    0.049805    0.071289    0.076660    -0.003937   -0.261719   -0.198242   -0.269531   -0.035889   -0.249023   -0.023071   -0.091797   -0.093750   0.192383    -0.376953   0.170898    0.027832    0.023438    0.047363    -0.051270   0.020386    -0.029663   0.128906    0.044434    -0.199219   0.060547    0.138672    0.104980    0.314453    -0.125000   -0.075684   0.088379    0.109863    -0.058594   0.063477    -0.120117   -0.177734   0.017700    0.112793    -0.161133   -0.188477   -0.102051   -0.068848   -0.073730   0.168945    -0.042236   -0.024536   0.128906    -0.066406   -0.020996   0.087891    -0.224609   0.025146    -0.054932   -0.102539   -0.020142   0.123047    -0.171875   0.195312    -0.203125   -0.265625   -0.026367   0.154297    -0.235352   0.092773    0.032715    0.177734    0.063477    -0.168945   0.153320    -0.182617   0.101074    0.074219    0.031250    -0.038086   0.037598    0.035400    -0.150391   -0.108398   -0.071289   -0.080078   0.078613    0.022705    0.148438    -0.098633   -0.032471   0.083984    0.031494    -0.052002   -0.062988   0.316406    -0.105957   0.026733    0.018921    0.026855    -0.176758   -0.088379   0.127930    -0.104980   0.206055    -0.003296   0.184570    0&#xA;1   -0.068359   0.076660    -0.224609   0.292969    0.054688    -0.069824   0.028809    0.090332    -0.160156   0.080566    0.289062    -0.005615   0.074219    -0.071289   0.069824    0.032715    -0.036133   0.043457    0.084961    0.224609    -0.001160   0.100098    -0.090820   0.209961    0.101074    0.009949    0.038818    0.151367    0.209961    -0.157227   0.118652    0.247070    0.090332    0.244141    0.125000    -0.253906   0.204102    -0.234375   0.118652    -0.000603   0.253906    -0.146484   -0.077148   0.180664    -0.110840   0.018677    -0.113770   0.159180    0.245117    -0.033447   -0.041748   0.246094    0.018677    0.034180    0.103516    0.087891    0.339844    -0.357422   -0.230469   -0.051758   -0.038574   -0.281250   -0.218750   -0.210938   -0.150391   -0.040283   -0.049072   -0.292969   0.151367    0.143555    0.048340    -0.194336   -0.027344   0.038574    -0.086426   -0.003036   -0.095215   0.062500    -0.098145   0.085938    -0.099609   0.046875    0.039551    0.182617    -0.142578   0.189453    -0.261719   0.030273    0.056152    0.123535    -0.082520   -0.075684   -0.267578   0.014832    0.047852    -0.012451   0.131836    0.240234    -0.107910   -0.316406   0.081055    0.092285    0.014771    0.211914    0.062500    -0.143555   0.412109    -0.210938   -0.064453   -0.193359   0.051025    0.027954    0.026367    -0.109375   0.020752    -0.124512   0.198242    -0.105469   0.250000    -0.071289   -0.065430   -0.139648   -0.032959   0.386719    -0.185547   -0.166992   0.036621    0.001389    -0.090820   0.030396    -0.249023   -0.047363   -0.013245   0.318359    -0.150391   0.048340    -0.037354   0.125000    -0.053711   0.562500    0.005463    -0.067383   -0.345703   0.214844    0.044678    0.170898    -0.218750   0.243164    -0.165039   -0.259766   -0.158203   -0.275391   -0.138672   0.080566    -0.212891   -0.238281   -0.075684   0.015320    0.089844    -0.052490   0.031738    0.339844    0.035400    0.212891    0.127930    -0.033447   0.234375    0.130859    -0.209961   -0.106445   -0.236328   0.047607    -0.153320   -0.075195   0.048340    0.133789    -0.085449   0.122070    -0.187500   -0.172852   -0.137695   -0.392578   -0.028809   -0.177734   -0.131836   -0.141602   0.071777    -0.118652   -0.072754   -0.081543   -0.070312   0.033447    0.124023    -0.088379   -0.130859   0.131836    -0.010437   0.247070    -0.287109   0.077637    0.033203    0.032959    -0.136719   -0.079590   0.051758    -0.045898   -0.131836   -0.326172   -0.202148   -0.033203   -0.176758   0.180664    -0.148438   0.227539    -0.212891   -0.143555   0.273438    0.134766    -0.261719   0.073242    -0.054688   0.027466    0.126953    0.234375    0.097168    0.259766    0.253906    -0.170898   -0.189453   0.239258    -0.173828   0.024536    0.002090    0.101074    0.351562    0.174805    0.162109    -0.146484   -0.103516   -0.037354   0.065430    -0.104004   0.108398    0.296875    0.172852    0.078613    -0.209961   -0.133789   0.037354    -0.125977   0.172852    -0.102539   0.034424    0.095215    0.158203    -0.291016   -0.047852   -0.161133   -0.024414   -0.162109   -0.161133   0.109375    0.003372    0.218750    -0.022339   0.057861    -0.351562   -0.113770   -0.247070   -0.108398   0.097656    0.083008    0.357422    0.347656    0.341797    -0.031006   0.056885    0.114746    0.083008    0.192383    0.335938    0.154297    -0.244141   -0.445312   0.166992    0.396484    -0.132812   0.077148    -0.108398   0.131836    0.063477    0.001221    -0.219727   -0.062988   -0.137695   -0.133789   0.223633    -0.069336   0.163086    0.236328    0&#xA;2   -0.003067   0.219727    -0.082520   0.255859    -0.209961   -0.117188   0.109863    0.107422    0.059570    0.007233    0.059082    -0.152344   0.208984    -0.095703   -0.096680   -0.312500   -0.154297   0.024780    0.032471    0.250000    0.090820    0.017944    0.105957    0.133789    -0.122070   0.199219    -0.073730   -0.142578   0.203125    0.047607    0.222656    0.019531    0.026123    -0.138672   0.061768    0.120605    -0.008789   -0.047852   0.269531    -0.182617   0.566406    -0.218750   -0.043457   -0.051270   -0.273438   -0.084961   -0.240234   -0.158203   0.221680    -0.043457   0.308594    0.221680    -0.112305   -0.014343   0.070312    0.174805    -0.090332   -0.384766   0.003281    -0.002808   -0.273438   -0.116211   -0.542969   -0.008057   -0.137695   0.209961    0.231445    -0.008484   -0.092285   0.226562    -0.021851   -0.083984   0.069336    0.277344    -0.217773   0.057129    0.269531    0.218750    0.137695    0.093750    -0.101562   0.281250    0.029785    0.126953    0.066406    -0.019775   -0.287109   0.267578    0.195312    -0.135742   0.012207    0.048828    -0.237305   0.101562    0.206055    -0.091309   -0.085938   0.112305    -0.008423   -0.037109   0.099121    0.018433    -0.108398   0.031982    0.202148    -0.273438   -0.007874   -0.179688   0.025879    -0.046387   -0.172852   -0.202148   -0.086426   -0.028564   -0.033447   -0.047852   0.184570    -0.146484   0.109863    -0.243164   -0.251953   -0.000456   -0.073730   0.199219    -0.248047   -0.265625   0.261719    0.003693    0.092285    -0.111816   -0.118652   -0.320312   0.121582    0.127930    -0.127930   -0.087402   0.229492    0.040527    -0.121094   0.233398    0.052734    0.213867    -0.111328   -0.030884   -0.084961   0.054932    -0.068848   0.133789    -0.121582   -0.235352   -0.031982   0.062500    -0.137695   0.244141    -0.070312   -0.090820   -0.050781   0.041748    0.166992    0.200195    0.016724    0.292969    0.023682    -0.232422   -0.113281   -0.032959   0.038330    -0.357422   0.187500    -0.034180   -0.157227   -0.213867   0.007233    0.136719    0.018433    0.040771    0.089355    0.162109    -0.051514   -0.109863   -0.142578   -0.292969   -0.043945   0.200195    -0.079102   -0.007172   0.131836    0.206055    -0.125977   -0.092285   0.118652    -0.042236   -0.054443   -0.082520   -0.238281   -0.078125   0.052979    0.003601    -0.045166   0.126953    0.064453    0.296875    0.145508    -0.006378   0.015869    -0.070312   0.036377    -0.277344   0.038574    -0.112793   -0.224609   0.171875    -0.184570   0.062500    0.142578    -0.170898   0.189453    -0.067871   -0.239258   -0.110840   -0.043213   0.089844    0.069824    0.012512    0.162109    -0.194336   0.419922    -0.116699   0.170898    0.119141    -0.189453   0.102051    0.055420    0.026245    0.008545    0.052246    -0.088379   -0.236328   -0.041016   -0.125000   -0.051514   0.020020    0.051758    -0.137695   0.206055    -0.029297   -0.106445   -0.039062   0.285156    -0.018677   0.265625    -0.072266   -0.090820   -0.030640   -0.112793   -0.181641   -0.000690   -0.171875   -0.115234   -0.179688   0.114746    0.032227    -0.016235   -0.063477   0.054688    -0.033691   -0.242188   -0.292969   -0.229492   0.067871    0.006378    0.345703    0.024780    0.148438    0.119629    0.121582    0.024780    0.086914    0.066895    0.181641    0.120605    0.234375    0.034180    -0.306641   -0.124512   0.145508    0.025269    -0.138672   0.353516    -0.227539   -0.082520   -0.035645   0.066895    -0.085938   -0.159180   -0.087402   0.186523    0.289062    -0.075195   0.050781    0&#xA;In [223]:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have two labels 0 and 1. I am now doing a Binary classification with 300 dimension word vectors as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the details of training and testing count:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Splitting the dataset to train test&#xA;from sklearn.cross_validation import train_test_split&#xA;train_X, test_X,train_Y,test_Y = train_test_split(jpsa_X_norm,jpsa_Y, test_size=0.30, random_state=42)&#xA;&#xA;print(&quot;Total Sample size in Training {}\n&quot;.format(train_X.shape[0]))&#xA;print(&quot;Total Sample size in Test {}&quot;.format(test_X.shape[0]))&#xA;​&#xA;​&#xA;Total Sample size in Training 151&#xA;&#xA;Total Sample size in Test 65&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now my label ratio in training data is as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    87&#xA;1    64&#xA;dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So it's slightly imbalanced class dataset with ratio of 0:1=1:35&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now do a GridSearchCV for both SVM and Random Forest. In both the algo, i put&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class_weights={1:1.35,0:1}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to take into account the class imbalance problem in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my GridSearchCV function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def grid_search(self):&#xA;&#xA;    &quot;&quot;&quot;This function does Cross Validation using Grid Search&#xA;&#xA;    &quot;&quot;&quot;&#xA;&#xA;    from sklearn.model_selection import GridSearchCV&#xA;    self.g_cv = GridSearchCV(estimator=self.estimator,param_grid=self.param_grid,cv=5)&#xA;    self.g_cv.fit(self.train_X,self.train_Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following as result for SVM.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The mean train scores are [ 0.57615906  0.57615906  0.57615906  0.57615906  0.93874475  0.57615906&#xA;  0.57615906  0.57615906  1.          0.94867633  0.57615906  0.57615906&#xA;  1.          1.          0.950343    0.57615906  0.81777921  0.99668044&#xA;  1.          1.        ]&#xA;&#xA;The mean validation scores are [ 0.57615894  0.57615894  0.57615894  0.57615894  0.87417219  0.57615894&#xA;  0.57615894  0.57615894  0.8807947   0.8807947   0.57615894  0.57615894&#xA;  0.86754967  0.87417219  0.88741722  0.57615894  0.70860927  0.90728477&#xA;  0.87417219  0.87417219]&#xA;&#xA;The score on held out data is: 0.9072847682119205&#xA; Parameters for Best Score : {'C': 1, 'kernel': 'linear'}&#xA;&#xA;The accuracy of svm on test data is: 0.8769230769230769&#xA;&#xA;Classification Metrics for svm :&#xA;             precision    recall  f1-score   support&#xA;&#xA;          0       0.87      0.92      0.89        37&#xA;          1       0.88      0.82      0.85        28&#xA;&#xA;avg / total       0.88      0.88      0.88        65&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The parameter grid for hyperparamter values passed to GridSearchCV for SVM is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid_svm=[{'kernel': ['rbf'], 'gamma': [1e-1,1e-2,1e-3,1e-4],\&#xA;                     'C': [0.1, 1, 10, 100]},\&#xA;                    {'kernel': ['linear'], 'C': [0.1,1,10,100]}]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I ran Random Forest also:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The mean train scores are [ 0.99009597  1.          0.99833333  1.          0.99833333  1.&#xA;  0.99834711  1.          1.          1.          1.          1.          1.&#xA;  1.          1.          1.          1.          1.          1.          1.&#xA;  1.          1.          1.          1.          1.          1.          1.&#xA;  1.          1.          1.          1.          1.          1.          1.&#xA;  1.          1.          1.          1.          1.          1.          1.&#xA;  1.        ]&#xA;&#xA;The mean validation scores are [ 0.79470199  0.85430464  0.8807947   0.87417219  0.8807947   0.85430464&#xA;  0.83443709  0.82781457  0.86754967  0.84768212  0.88741722  0.87417219&#xA;  0.81456954  0.86092715  0.85430464  0.83443709  0.8410596   0.8410596&#xA;  0.83443709  0.86092715  0.85430464  0.83443709  0.84768212  0.82781457&#xA;  0.82781457  0.82119205  0.85430464  0.81456954  0.82781457  0.85430464&#xA;  0.82781457  0.84768212  0.83443709  0.86092715  0.87417219  0.86754967&#xA;  0.86092715  0.86092715  0.8410596   0.86754967  0.86754967  0.8410596 ]&#xA;&#xA;The score on held out data is: 0.8874172185430463&#xA; Parameters for Best Score : {'max_depth': 4, 'n_estimators': 600}&#xA;&#xA;The accuracy of rf on test data is: 0.8307692307692308&#xA;&#xA;Classification Metrics for rf :&#xA;             precision    recall  f1-score   support&#xA;&#xA;          0       0.77      1.00      0.87        37&#xA;          1       1.00      0.61      0.76        28&#xA;&#xA;avg / total       0.87      0.83      0.82        65&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I had 42 combination of hyper parameter values for RF as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid_rf={'n_estimators': [30,100,250,500,600,900], 'max_depth':[2,4,7,8,9,10,13]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if you look at both the outputs for SVM and RF, my training accuracy is like close to 99% but test accuracy and validation accuracy is not close to training accuracy. This should suggest Overfitting, but I did the hyper parameter tuning using Grid Search and Random Forest generally doesn't overfit too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what could be causing this low test/validation accuracy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the AUC of both from ROC plot is very good close to 0.96. So AUC is good, and accuracy is bad I can understand class imbalance issue might be in play. But then I took care of that using class weights parameter in both. So then also my test and validation accuracy is not comparable to training?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kuQgO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kuQgO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also added more data so now I have 2000 0's and 1000 1's. I use the option &quot;balanced&quot; in scikit learn class_weight option in each algorithm for class imbalance &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now if I take more data, here is the result of random forest on more data. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The mean train scores are [ 0.70347493  0.73347328  0.74070792  0.74368715  0.74609988  0.74772955&#xA;  0.7476584   0.78035322  0.80624038  0.81432687  0.8194324   0.81581485&#xA;  0.81773002  0.81929078  0.9497877   0.96858105  0.97283788  0.97524883&#xA;  0.9759579   0.97567365  0.9751775   0.97851051  0.99099354  0.99248265&#xA;  0.99489341  0.99468108  0.99538994  0.99595762  0.98999975  0.99794336&#xA;  0.99872325  0.99893632  0.99872348  0.99914909  0.99907804  0.99687948&#xA;  0.99957447  0.99978721  0.99957452  0.99978728  0.99971639  0.99978728&#xA;  0.99985806  1.          1.          1.          1.          1.          1.        ]&#xA;&#xA;The mean validation scores are [ 0.68765957  0.71460993  0.7222695   0.71829787  0.71744681  0.72453901&#xA;  0.71971631  0.7248227   0.73191489  0.74439716  0.74638298  0.74524823&#xA;  0.74695035  0.74297872  0.75716312  0.77730496  0.78468085  0.78382979&#xA;  0.79120567  0.78609929  0.7906383   0.75120567  0.77531915  0.78808511&#xA;  0.78780142  0.79035461  0.79234043  0.78808511  0.75716312  0.7693617&#xA;  0.78297872  0.78553191  0.78609929  0.77957447  0.78269504  0.75234043&#xA;  0.77673759  0.77021277  0.7764539   0.76879433  0.77134752  0.77673759&#xA;  0.74241135  0.75148936  0.75375887  0.75375887  0.75432624  0.75829787&#xA;  0.75205674]&#xA;&#xA;The score on held out data is: 0.7923404255319149&#xA; Hyper-Parameters for Best Score : {'max_depth': 8, 'n_estimators': 700}&#xA;&#xA;The accuracy of rf on test data is: 0.8022486772486772&#xA;&#xA;Classification Metrics for rf :&#xA;             precision    recall  f1-score   support&#xA;&#xA;          0       0.83      0.90      0.86       956&#xA;          1       0.71      0.64      0.67       433&#xA;          2       0.92      0.62      0.74       123&#xA;&#xA;avg / total       0.80      0.80      0.80      1512&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seem to have decreased the accuracy from 82 to 80%. Why cold that be? If data is increasing then why for more data accuracy goes down? The result show that training accuracy is 1 but validation and test is close t0 0.8. Why is that? Is there something overfitting since validation error is high and train error is low but then Random Forest generally don't overfit that well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or Can this be, because the new data added might be noise and not true labels? So it reduces the accuracy measure on old small data?. &lt;/p&gt;&#xA;" OwnerUserId="30406" LastActivityDate="2021-01-29T02:21:54.767" Title="Possible Reason for low Test accuracy and high AUC" Tags="|machine-learning|python|data-mining|scikit-learn|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18866" PostTypeId="1" CreationDate="2017-05-10T06:50:55.653" Score="1" ViewCount="306" Body="&lt;p&gt;Currently I'm doing some simple feature selection based on correlation between features and variance within one feature. I'm applying this on the whole dataset used for model building before creating the cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question now is if this is acceptable workflow or can significantly affect the CV stats suggesting a better model than it actually is? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it technically better to do the CV-split and only then select features on the training set for that split to not leak information?&lt;/p&gt;&#xA;" OwnerUserId="32110" LastActivityDate="2017-05-10T14:35:52.517" Title="Feature selection: Information leaking if done before CV-split?" Tags="|feature-selection|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18869" PostTypeId="2" ParentId="17650" CreationDate="2017-05-10T09:27:40.023" Score="1" Body="&lt;p&gt;You decide what the output layer represents, so it should be OK (and probably easier to implement) to have a single fully connected layer of size $m_x + m_y + m_z$ and to interpret the output in your own code. E.g. $n_x = N_{(x)}, n_y = N_{(m_x + y)}, n_z = N_{(m_x + m_y + z)}$, where $N_{(i)}$ is the output of the $i^{th}$ neuron. In the case of regression, this should have very similar behaviour to 3 separate fully connected layers for each dimension. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, once you have either structure, it is not clear that you will be estimating reward directly any more. There is definitely no guarantee that the rewards will be the same between $n_x, n_y, n_z$, in fact they will not be in most cases. You will need to define a function $\hat{r}(x, y, z)$ that combined the outputs of the network and have your loss and error gradient calculated from that. The function could just be $\hat{r}(x, y, z) = n_x + n_y + n_z$ or maybe $\hat{r}(x, y, z) = n_x \times n_y \times n_z$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly this approximation has limits built in - it will likely over-estimate and under-estimate rewards due to its structure. If the simplified structure matches your problem, then this will be fine - it may even speed convergence. But if not, it will limit performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative, if you don't want to build a network that has a large number of outputs, is to have a single reward estimate as the output, and have the action choice as an input. That has the advantage of leaving the details of the approximation to the neural network. It has the disadvantage that you would need to run the network up to $m_x * m_y * m_z$ times in order to decide the policy in each state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of the large number of possible actions in your case, you may want to look at a policy-based learning method, rather than DQN. There are deep NN-based versions of those algorithms. A recently-published &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2&quot; rel=&quot;nofollow noreferrer&quot;&gt;algorithm called Asynchronous Advantage Actor-Critic (A3C)&lt;/a&gt; has performed well in computer gaming tasks and might be appropriate for your problem.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-06T16:31:58.620" LastActivityDate="2017-11-06T16:31:58.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18870" PostTypeId="2" ParentId="18860" CreationDate="2017-05-10T09:34:45.837" Score="4" Body="&lt;p&gt;Not sure if you are facing a concrete problem. Anyway, here it goes a definition of the agglomerative coefficient, from &lt;a href=&quot;http://onlinelibrary.wiley.com/book/10.1002/9780470316801&quot; rel=&quot;nofollow noreferrer&quot;&gt;Finding Groups in Data: An Introduction to Cluster Analysis&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Generally speaking, the AC describes the strength of the clustering&#xA;  structure that has been obtained by group average linkage. However,&#xA;  the AC tends to become larger when n increases, so it should not be&#xA;  used to compare data sets of very different sizes.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also, if you are familiar with the silhouette,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The AC can also be compared to the silhouette coefficient (SC)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;First sight, the coefficient you get points to a pretty reasonable cluster structure in your data, since it is closed to 1: the coefficient takes values from 0 to 1, and it is actually the mean of the normalised lengths at which the clusters are formed. That is, the lengths you see when you look at your dendogram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT after comments&lt;/strong&gt;: indeed it can be misleading that the text in the book I referenced (&quot;Finding Groups in Data...&quot;) refers in my copy-paste to the group average linkage. However, the coefficient refers to the banner (which is like a kind of simpler or binary dendogram) and just validates the structure that the clustering produces, regardless of the method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/cluster/html/coef.hclust.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is another reference to the agglomerative coefficient with no reference to the method used (although of course always in the context of hierarchical clustering).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2 after comments&lt;/strong&gt;: I explain below my understanding on how the coefficient is calculated, always using the same reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula for the coefficient is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$AC = \frac{1}{n} \sum_{i} l(i)$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. the mean of the &lt;code&gt;l(i)&lt;/code&gt;, calculated as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For each object i, we look at the line containing its label and&#xA;  measure its length I(i) on the 0-1 scale above or below the banner.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although I could not make the plot with 0-1 scale, it is just a normalization of the lengths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The banner is a binary representation of the dendogram, where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in the left side, when all objects are separated in the initial stage of the agglomerative algorithm, the banner is just blank&lt;/li&gt;&#xA;&lt;li&gt;as soon as 2 objects get together in the same cluster, they appear in the banner&lt;/li&gt;&#xA;&lt;li&gt;the &lt;code&gt;l(i)&lt;/code&gt; are marked by the points in the x axis in which the objects get together, and are the lengths of the horizontal bars of the plots&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Always clearer with an example, actually taken to a similar one in the same book.&#xA;Let's imagine 5 objects in a space of 4 dimensions; 4 of the objects very closed together, and one of them far away.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(&quot;c1&quot;=c(2, 2, 2, 2, 10), &quot;c2&quot;=c(2, 2, 2, 2, 10),&#xA;                 &quot;c3&quot;=c(2, 2, 2, 2, 10), &quot;c4&quot;=c(2, 2, 2, 2, 10))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's do the clustering,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(cluster)&#xA;r &amp;lt;- agnes(df, method=&quot;ward&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The bannerplot actually looks a bit ugly:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/YryX5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YryX5.png&quot; alt=&quot;bannerplot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but anyway, in there the 4 objects that are at the same position are joined at the beginning in a cluster, and the 5th is at distance 20. Personally, it is much easier to look at the dendogram:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/7cwSy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7cwSy.png&quot; alt=&quot;dendogram&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;however the information is pretty much the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the AC calculation. The lengths of the bars are 1 (normalized) for the first 4 objects, and 0 for the last one, thus the AC is 0.8:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; r$ac&#xA;[1] 0.8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the dendogram, one should imagine the bars rotated 90 degrees, and they go from the root of the tree to the point in which the groups are formed. &lt;/p&gt;&#xA;" OwnerUserId="9584" LastEditorUserId="9584" LastEditDate="2017-05-11T17:58:48.503" LastActivityDate="2017-05-11T17:58:48.503" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18871" PostTypeId="1" CreationDate="2017-05-10T10:55:08.880" Score="0" ViewCount="619" Body="&lt;p&gt;I am trying to use a neural network to predict final concrete strength. I have used material properties of concrete as input data. Now if I want to add some additional information like early age strength, is there any way to put them in the hidden layer? I mean other than using as input data?&lt;/p&gt;&#xA;" OwnerUserId="32116" LastEditorUserId="24000" LastEditDate="2018-04-13T14:28:03.260" LastActivityDate="2018-04-13T14:28:03.260" Title="How to add additional information in a neural network other than putting as input data?" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18873" PostTypeId="2" ParentId="18750" CreationDate="2017-05-10T11:35:06.810" Score="0" Body="&lt;p&gt;If all above fails, I'd try this: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fit model for each class. So you'd have four models, each specialized for that particular class.&lt;/li&gt;&#xA;&lt;li&gt;I'd give a shot to any ensemble modeling technique - fit multiple models and combine them in a smart way.&lt;/li&gt;&#xA;&lt;li&gt;Complex methods such as neural networks (I heard that GAN's are popular these days with impressive results). But that can be overkill.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In case non of that works, it could be just that your data cannot be separated in any way. I'd suggest to make some basic plots and calculate some simple statistics to get better understanding of the data and its separability.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-05-10T11:35:06.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18874" PostTypeId="2" ParentId="18871" CreationDate="2017-05-10T11:36:51.127" Score="1" Body="&lt;p&gt;If your inputting data, it will always be an input :). But it is definitely possible to add in certain inptus after the first hidden layer. But what is your reasoning behind this? Why would you want this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should just input all your data in the first layer, let the neural network figure out how to use the data effectively itself. The neural network is like a black box, you put something in and something comes out - don't interfere too much by adding seperate inputs to the hidden layers.&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-05-10T11:36:51.127" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18875" PostTypeId="2" ParentId="16699" CreationDate="2017-05-10T13:56:58.040" Score="1" Body="&lt;p&gt;By default, for any methods that use gradient descent or feature combination (e.g. PCA), I scale my data if the orders of magnitude in the features are different from each other.&#xA;The optimization is easier  when the fitted parameters are not too far from zero in any direction.&#xA;Scaling you data doesn't matter for tree-based methods&lt;/p&gt;&#xA;" OwnerUserId="32119" LastActivityDate="2017-05-10T13:56:58.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18876" PostTypeId="2" ParentId="12797" CreationDate="2017-05-10T14:04:57.283" Score="0" Body="&lt;p&gt;Instead of filling missing categorical value with median, I would use the mode. Doing this, you are sure to fill with existing modality even if you modalities are strings. Depending on the process behind the missing value, as said Ricardo Cruz, you could also add a new modality for the missing one&lt;/p&gt;&#xA;" OwnerUserId="32119" LastActivityDate="2017-05-10T14:04:57.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18877" PostTypeId="2" ParentId="12129" CreationDate="2017-05-10T14:23:34.227" Score="1" Body="&lt;p&gt;I will be using these facts without proof (but the proofs either follow directly from definitions or are straightforward): The vector of the centroid of a set of points is arithmetic mean of all vectors of the points in the set. Also, the centroid of the union of two sets of point is on the straight line connecting centroids of these two sets, and it divides that straight line into two parts whose ratio is determined by the ratio of number of points of in these sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider following series of identities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{i\in A} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 =$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \sum_{i\in A} \Big\Vert \vec{x}_i-\Big(\vec{m}_{A} - \frac{n_B}{n_A+n_B} (\vec{m}_{A} - \vec{m}_{B})\Big)\Big\Vert^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big) + \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + 2 \cdot \sum_{i\in A}\big(\vec{x}_i-\vec{m}_{A}\big) \cdot \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big) + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$= \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +  \frac{n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i\in B} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 +  \frac{n_A^2 n_B}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By adding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +\sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 + \frac{n_A^2 n_B + n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 - \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 - \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 = \frac{n_A n_B (n_A + n_B)}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta \big(A, B\big) = \frac{n_A n_B}{n_A+n_B} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$&lt;/p&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="31513" LastEditDate="2017-05-15T11:23:44.863" LastActivityDate="2017-05-15T11:23:44.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18878" PostTypeId="2" ParentId="13066" CreationDate="2017-05-10T14:32:05.087" Score="1" Body="&lt;p&gt;I don't know if it is still an open question, anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is painful when using TFidf, it is the size of the ouput sparse matrix. II think it is your problem here. In order to pass this, you should limit the amount of word you want to treat. I didn't try with your case but you should try CountVectorizer before TfidfTransformer or directly TfidfVectorizer.&#xA;In CountVectorizer and TfidfVectorizer from sklearn, you can specify the minimum and maximum number of document any term can be found in (min_df and max_df).&#xA;This will reduce the size of the output A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps a bit&lt;/p&gt;&#xA;" OwnerUserId="32119" LastActivityDate="2017-05-10T14:32:05.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18879" PostTypeId="2" ParentId="18866" CreationDate="2017-05-10T14:35:52.517" Score="3" Body="&lt;p&gt;If you do &lt;em&gt;not&lt;/em&gt; use target for feature selection, then there is no leakage and you can apply this on the whole dataset. The methods that you mention: checking correlation between features and checking variance of each feature belong here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you &lt;em&gt;do&lt;/em&gt; use target for feature selection, then you should use 'train' subset only. Otherwise there is a leakage. The examples here would be (i) you check correlation between each feature and target (ii) use build Random Forest model and take best features&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2017-05-10T14:35:52.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18880" PostTypeId="1" AcceptedAnswerId="18896" CreationDate="2017-05-10T15:05:24.943" Score="7" ViewCount="10883" Body="&lt;p&gt;I already used a custom transformation function in a scikit-learn pipeline. In this function I only added features to my data frame. It works great.&lt;/p&gt;&#xA;&lt;p&gt;Below is a working example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.datasets import make_blobs&#xA;&#xA;x, y = make_blobs(n_samples=300, n_features=2, centers=1)&#xA;x_train = pd.DataFrame(x[:150,:], columns=['x1','x2'])&#xA;x_test = pd.DataFrame(x[150:,:], columns=['x1','x2'])&#xA;&#xA;class myTransformation(object) :&#xA;  def __init__(self, colname):&#xA;    self.colname = colname&#xA;&#xA;  def transform(self, x) :&#xA;    dat = x.copy()&#xA;    squared = dat.loc[:,self.colname]**2&#xA;    squared.name = &amp;quot;%s_sqre&amp;quot;%self.colname&#xA;    dat.loc[:,squared.name] = squared&#xA;    dat.loc[:, self.colname+'_2'] = dat[self.colname]&#xA;    return dat&#xA;&#xA;  def fit(self, dat, y=None) :&#xA;    return self&#xA;&#xA;makePipe = Pipeline([('makeTransfo', myTransformation(colname=&amp;quot;x2&amp;quot;))])&#xA;fittedPipe = makePipe.fit(x_train)&#xA;x_1 = fittedPipe.transform(x_train)&#xA;x_2 = fittedPipe.transform(x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Now I would like to be able to add the ability to remove the equal columns in the data frames.&#xA;For now, I have the following function:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def delSameCols(df) :&#xA;  cols = []&#xA;  for i in range(df.shape[1]) :&#xA;    for j in range(i+1, df.shape[1]) :&#xA;      if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :&#xA;        if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :&#xA;          cols.append(df.columns[j])&#xA;  cols = list(set(cols))&#xA;  print( u'      -%s features removed'%len(cols) )&#xA;  return df.drop(cols, axis=1), cols&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have no idea how to deal with this/how to add a new function in the pipeline or directly in the existing function?&#xA;Does anyone have any idea?&lt;/p&gt;&#xA;" OwnerUserId="32119" LastEditorUserId="29169" LastEditDate="2021-07-16T18:35:35.503" LastActivityDate="2021-07-16T18:36:19.320" Title="How to remove columns in Transformer function in Pipeline?" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18882" PostTypeId="2" ParentId="18489" CreationDate="2017-05-10T18:03:51.530" Score="2" Body="&lt;p&gt;A CNN usually requires a lot of data. And of course even more data if you have  a very large feature set. And even more data if you have many output classes. The number of output classes makes the necessary amount of data samples to increase exponentially. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most important part of any machine learning project is to extract the most telling features from your raw data. Ask yourself in your signal what are the telltale signs that should tell your model what class the signal was drawn from. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should try to use some feature extraction techniques, then you should try to use a model better suited for the size of your dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data pre-processing and feature extraction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. n your case if you have a signal you might want to extract some information about the temporal and frequency domain transforms. You can try something like empirical mode decomposition (EMD).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In your case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a general rule of thumb, the amount of data that I suggest to have for shallow machine learning models is $10 \times \#features$. For deep models I usually suggest $100 \times \#features \times \#outputs$. So you are limited by the size of your dataset. In your case a CNN is not recommended. Only recently a group published a quite controversial result of training a CNN with 100,000 examples. The community generally believes that this is way too little and it is not feasible to expect good results when training a CNN with very little data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you use a general machine learning technique such as SVM. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-05-10T18:03:51.530" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18886" PostTypeId="2" ParentId="13948" CreationDate="2017-05-10T21:11:37.917" Score="1" Body="&lt;p&gt;If we want to check the imputation efficiency, we should probably compute the performance for the imputed values only. Indeed, the performance depends on the percentage of missing values.&lt;/p&gt;&#xA;" OwnerUserId="32119" LastActivityDate="2017-05-10T21:11:37.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18887" PostTypeId="1" CreationDate="2017-05-10T23:15:56.937" Score="1" ViewCount="147" Body="&lt;p&gt;I was going through some lectures from the Deep Learning course that Geoffrey Hinton taught on Coursera and I came across this statement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea what he meant nor seen/figured out any examples of this idea (and I'm sure other people are wondering too.) So if somebody would care to demonstrate, it would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="32130" LastActivityDate="2017-06-10T00:57:05.207" Title="What did Geoffrey Hinton mean when he said this?" Tags="|machine-learning|neural-network|deep-learning|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18888" PostTypeId="5" CreationDate="2017-05-10T23:23:42.330" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-05-10T23:23:42.330" LastActivityDate="2017-05-10T23:23:42.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18889" PostTypeId="4" CreationDate="2017-05-10T23:23:42.330" Score="0" Body="WEKA is a GUI based data mining suite written in Java, developed at the University of Waikato" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-04-19T19:17:38.317" LastActivityDate="2018-04-19T19:17:38.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18890" PostTypeId="1" CreationDate="2017-05-10T23:26:44.077" Score="0" ViewCount="1775" Body="&lt;p&gt;I'm pretty new to data science. I am working on a model to identify customers who are more likely to purchase a given product category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did try market basket using arules package from R. &#xA;I also used MatchIt, Matching algorithm and did try Propensity score matching to find similar customers who tend purchase in the given product category. But I'm not quite sure the model is good enough for this kinda problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just wondering what kind of model will be useful to find customers who are more likely to purchase a given product group. &#xA;Any help is greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="32131" LastActivityDate="2018-11-25T20:01:30.247" Title="Identifying Customers who are more likely to purchase a given product category- Which model to use?" Tags="|python|r|predictive-modeling|market-basket-analysis|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18892" PostTypeId="2" ParentId="18890" CreationDate="2017-05-11T00:48:23.507" Score="1" Body="&lt;p&gt;That is called a &quot;recommender system&quot; and is related to the associative rules/market basket stuff you said you tried. There is a package for it in R called recommenderlab. Here is a nice walk-through:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bigdata-doctor.com/recommender-systems-101-practical-example-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://bigdata-doctor.com/recommender-systems-101-practical-example-in-r/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-11T00:48:23.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18893" PostTypeId="2" ParentId="18887" CreationDate="2017-05-11T00:51:04.093" Score="1" Body="&lt;p&gt;Creating a model &lt;strong&gt;is&lt;/strong&gt; having the computer write a small program. If you have a multi-layer network, then multiple nuggets might be interacting to produce complicated results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That its pretty much how things like face detection work - you start with a series of really dumb &quot;detectors&quot; that look like basic rectangles and they combine in ways so that at the top, you're recognizing Brad Pitt. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-11T00:51:04.093" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18894" PostTypeId="2" ParentId="18629" CreationDate="2017-05-11T06:31:26.487" Score="1" Body="&lt;p&gt;Since generally, Naive bayes alogorithms implemented by most libraries don't support multilabel classification. Although You can devise your own algorithm taking inspirations from naive bayes approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a particular class c, and document d (with 3 words w1,w2,w3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'/' stands for 'given' p(a/b) = probability(a given b)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per bayes theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(c/d) = p(c,d) / p(d)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(c/d) = (p(c)*p(d/c)) / p(d)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p(d/c) = p(w1/c)*p(w2/c)*p(w3/c)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since words are assumed to be independent of each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And p(w1/c) can be obtained using your code, calculating count of w1  inside class c documents divided by overall count of w1 in all documents or you can use your own logic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But If want to avoid writing detailed code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can restructure your input data to achieve multilabel classification. such that a given document d with n labels/classes (eg d labeled with c1,c2,c3) get expanded into data of n samples (3 here) of same document d with different label each time ((d,c1),(d,c2),(d,c3)). Now you can pass this data into any R/python library supporting multinomial naive bayes. Same needs to be done in your dataset as well. Currently, you are passing traindata$topics which is y variable (training labels) as it is, without modification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even after you train your data using this approach. You need to use a probability threshold e.g 0.2, so that class labels with probability above 0.2 will be assigned to that test document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More better Approach which requires restructuring your input&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have 'n' class labels then you can train 'n' different binary naive bayes classifier for each of the class. For example, for training a classifier for class 'c1', you train a naive bayes classifier with dependent  variable Y denoting the presence of class c1 on that document as '1' and absence as '0'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you train 'n' binary naive bayes classifier using this approach. You will now use the output of these n classifiers. e.g if out of these, n classifiers, if a particular classifier, which corresponds to class 'c1' has output probability above 0.5, then class label 'c1' will be assigned to this test document.&lt;/p&gt;&#xA;" OwnerUserId="9497" LastEditorUserId="9497" LastEditDate="2018-11-04T14:26:52.760" LastActivityDate="2018-11-04T14:26:52.760" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18896" PostTypeId="2" ParentId="18880" CreationDate="2017-05-11T07:40:29.730" Score="2" Body="&lt;p&gt;I succeeded in getting a satisfying solution. I posted an entire working script. What do you think about it? Especially the creation of an attribute (self.lstRemCols) not initialized in the &lt;strong&gt;init&lt;/strong&gt; function?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.datasets import make_blobs&#xA;&#xA;class myTransfo(object) :&#xA;  def __init__(self, colname):&#xA;    self.colname = colname&#xA;&#xA;  def transform(self, x) :&#xA;    dat = x.copy()&#xA;    squared = dat.loc[:,self.colname]**2&#xA;    squared.name = &amp;quot;%s_sqre&amp;quot;%self.colname&#xA;    dat.loc[:,squared.name] = squared&#xA;    dat.loc[:, self.colname+'_2'] = dat[self.colname]&#xA;    return dat&#xA;&#xA;  def fit(self, dat, y=None) :&#xA;    return self&#xA;&#xA;class removeSameCols(object) :&#xA;  def __init__(self) :&#xA;    pass&#xA;&#xA;  def _delSameCols(self, df) :&#xA;    cols = []&#xA;    for i in range(df.shape[1]) :&#xA;      for j in range(i+1, df.shape[1]) :&#xA;        if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :&#xA;          if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :&#xA;            cols.append(df.columns[j])&#xA;    cols = list(set(cols))&#xA;    print( u'      - %s features to be removed'%len(cols) )&#xA;    return cols&#xA;&#xA;  def transform(self, x) :&#xA;    dat = x.copy()&#xA;    lstcols = list(set(dat.columns) - set(self.lstRemCols))&#xA;    return dat.loc[:, lstcols]&#xA;&#xA;  def fit(self, x, y=None) :&#xA;    dat = x.copy()&#xA;    self.lstRemCols = self._delSameCols(dat)&#xA;    return self&#xA;&#xA;&#xA;x, y = make_blobs(n_samples=300, n_features=5)&#xA;x_train = pd.DataFrame(x[:150,:], columns=['x1','x2','x3','x4','x5'])&#xA;x_test = pd.DataFrame(x[150:,:], columns=['x1','x2','x3','x4','x5'])&#xA;&#xA;makePipe2 = Pipeline([('makeCols', myTransfo(colname=&amp;quot;x2&amp;quot;)),&#xA;                      ('remCols', removeSameCols())])&#xA;makePipe2.fit(x_train)&#xA;x_1 = makePipe2.transform(x_train)&#xA;# test if only same columns in x_train are removed.&#xA;x_test.x4 = x_test.x5&#xA;x_2 = makePipe2.transform(x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32119" LastEditorUserId="29169" LastEditDate="2021-07-16T18:36:19.320" LastActivityDate="2021-07-16T18:36:19.320" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18897" PostTypeId="2" ParentId="18822" CreationDate="2017-05-11T09:15:36.487" Score="0" Body="&lt;p&gt;Before evaluating how good is &lt;a href=&quot;https://en.wikipedia.org/wiki/Automatic_summarization&quot; rel=&quot;nofollow noreferrer&quot;&gt;automatic summarization&lt;/a&gt; by taking the first sentence, you should decide how to evaluate summarization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In supervised learning typically it is easy to know if the prediction match the concept - they should be identical. Upon that, you can choose a metric that fits your needs (e.g., accuracy, precision, recall) and compare classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem of evaluating text summarization is that deciding whether a summarization is good is subjective and prone to error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A possible metric is &lt;a href=&quot;https://en.wikipedia.org/wiki/ROUGE_(metric)&quot; rel=&quot;nofollow noreferrer&quot;&gt;ROUGE&lt;/a&gt; which is a set of heuristics (e.g., common longest subsequence) that compare the summary to the original text.&#xA;Note that having a good ROUGE score is an estimation but an estimation that will enable you to benchmark your algorithm agains others. See &lt;a href=&quot;http://www.cai.sk/ojs/index.php/cai/article/viewFile/37/24&quot; rel=&quot;nofollow noreferrer&quot;&gt;EVALUATION MEASURES FOR TEXT&#xA;SUMMARIZATION&lt;/a&gt; by Josef Steinberger, Karel Jeˇzek for a discussion of metrics and other algorithms.&#xA;Showing that you get a good score with respect to an algorithm that is based on cue words will be a good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possibility is building a gold standard by comparing the first sentence to the text and manually labeling it for a good summary.&#xA;While manually labeling will give you good estimates of the performance of your algorithm, it costly in term of time.&#xA;A more severe disadvantage id that this gold standard is fitted to your algorithm and hard to use for other algorithms. Suppose that the second sentence is just as good algorithm as the first one. The gold standard built for the first sentence won't be able to show that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a good estimation I suggest that you will use ROUGE for comparison and a gold standard for absolute results. If you have the resources to create a gold standard for the benchmark algorithms, the comparison will become much more robust.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2017-05-11T09:20:48.727" LastActivityDate="2017-05-11T09:20:48.727" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18898" PostTypeId="1" AcceptedAnswerId="18899" CreationDate="2017-05-11T09:38:57.690" Score="1" ViewCount="330" Body="&lt;p&gt;I have created chatbot on Cornell movie dataset and it's working fine. I have trained chatbot application up to global step 330000. I am using tensorflow library. I am using ||source|| chatbot. I am also receiving output fine.  Then I created this new dataset file by modifying original dataset and i wish to train chatbot application with updated files. Now should I delete previously saved checkpoints and saved data and start training from zero or should i train from 330000 onwards without worrying changes in dataset.&#xA;           I wish to modify database daily basis and train on modified database. &#xA;Please kindly let me know if you have a suggestion for me as I am stuck on this issue. I will really appreciate if you help me with this issue.&lt;/p&gt;&#xA;" OwnerUserId="32145" LastActivityDate="2017-05-11T10:07:09.770" Title="how to train tensorflow chat application on updated dataset everyday" Tags="|machine-learning|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18899" PostTypeId="2" ParentId="18898" CreationDate="2017-05-11T10:07:09.770" Score="2" Body="&lt;p&gt;The distribution of your data won't change too much over a smaller period of time, retraining from scratch seems like a waste. Just add the new data to your set and do one or more epochs on all your data, not just the new ones. It can readjust the weights a little bit for changes in the distribution which is exactly what you want. Make sure you are not overreliant on early stopping as regularization method against overfitting because you continuously train like this, use some other regularizer. I would suggest periodically to retrain from scratch but this doesn't have to happen very regularly, maybe once every two months.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-11T10:07:09.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18901" PostTypeId="2" ParentId="761" CreationDate="2017-05-11T11:31:11.890" Score="9" Body="&lt;p&gt;I am probably very late with my answer, but if you are still dealing with geo clustering, you may find &lt;a href=&quot;https://cybergeo.revues.org/27035&quot; rel=&quot;noreferrer&quot;&gt;this study&lt;/a&gt; interesting. It deals with comparison of two fairly different approaches to classifying geographic data: K-means clustering and latent class growth modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the images from the study:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DsOxx.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DsOxx.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors concluded that the end results were overall similar, and that there were some aspects where LCGM overperfpormed K-means.&lt;/p&gt;&#xA;" OwnerUserId="31513" LastActivityDate="2017-05-11T11:31:11.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18902" PostTypeId="2" ParentId="17500" CreationDate="2017-05-11T12:00:36.457" Score="1" Body="&lt;p&gt;Yes, SQL Server does support it, as a part of &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_analysis_expressions&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Analysis Expressions (DAX)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation for the function for getting Poisson distribution data is &lt;a href=&quot;https://msdn.microsoft.com/en-us/library/mt171492.aspx&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31513" LastActivityDate="2017-05-11T12:00:36.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18903" PostTypeId="1" AcceptedAnswerId="19366" CreationDate="2017-05-11T12:12:06.923" Score="44" ViewCount="23426" Body="&lt;p&gt;I'm trying to understand which is better (more accurate, especially in classification problems)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been searching articles comparing LightGBM and XGBoost but found only two:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://medium.com/implodinggradients/benchmarking-lightgbm-how-fast-is-lightgbm-vs-xgboost-15d224568031&quot; rel=&quot;noreferrer&quot;&gt;https://medium.com/implodinggradients/benchmarking-lightgbm-how-fast-is-lightgbm-vs-xgboost-15d224568031&lt;/a&gt; - which is only about speed but not accuracy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/Microsoft/LightGBM/wiki/Experiments&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/Microsoft/LightGBM/wiki/Experiments&lt;/a&gt; - which is from the authors of LightGBM and no surprise LightGBM wins there.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my tests I get pretty the same AUC for both algorithms, but LightGBM runs form 2 to 5 times faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If LGBM is so cool, why don't I hear so much about it here and on Kaggle :)&lt;/p&gt;&#xA;" OwnerUserId="32148" LastActivityDate="2018-02-01T12:11:34.500" Title="LightGBM vs XGBoost" Tags="|xgboost|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18904" PostTypeId="1" CreationDate="2017-05-11T12:27:16.947" Score="4" ViewCount="111040" Body="&lt;p&gt;I'm trying to create a contour map from two variables which store some temperature values and a third variable which is the time stamp.&#xA;I used this notebook as a tutorial&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://plot.ly/pandas/contour-plots/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://plot.ly/pandas/contour-plots/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not able to convert the pandas dataframe created, into a 1d array. And the kde_scipy doesn't work with a nd-array. I tried converting the dataframe into a 1d array using .as_matrix() but this is the error I am receiving.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Degrees of freedom &amp;lt;= 0 for slice&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I convert this CSV file (with 3 columns of data) imported as a dataframe into individual columns of data? Or can I directly import each column of data into a 1d array and use it in the function kde_scipy?&lt;/p&gt;&#xA;" OwnerUserId="29509" LastActivityDate="2022-06-16T04:37:07.540" Title="How do I convert a pandas dataframe to a 1d array?" Tags="|visualization|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18905" PostTypeId="2" ParentId="18904" CreationDate="2017-05-11T12:55:14.757" Score="5" Body="&lt;p&gt;You can try this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;filename = 'data.csv'&#xA;df1 = pd.read_csv(filename)&#xA;#convert dataframe to matrix&#xA;conv_arr= df1.values&#xA;&#xA;#split matrix into 3 columns each into 1d array&#xA;&#xA;arr1 = np.delete(conv_arr,[1,2],axis=1) &#xA;arr2 = np.delete(conv_arr,[0,2],axis=1) &#xA;arr3 = np.delete(conv_arr,[0,1],axis=1) &#xA;&#xA;#converting into 1D array&#xA;arr1 = arr1.ravel()&#xA;arr2 = arr2.ravel()&#xA;arr3 = arr3.ravel()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should mostly do the job. Use the arr1 ,arr2,arr3 in the function you mentioned. They are the 1d array of the columns you split &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-05-11T12:55:14.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18906" PostTypeId="2" ParentId="18904" CreationDate="2017-05-11T13:42:20.403" Score="5" Body="&lt;p&gt;You can first convert the &lt;code&gt;DataFrame&lt;/code&gt; to NumPy format by calling &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.values.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;.values&lt;/code&gt;&lt;/a&gt;, after which the resulting &lt;code&gt;numpy.ndarray&lt;/code&gt; has the same dimensions as your original &lt;code&gt;DataFrame&lt;/code&gt;. Then, run &lt;a href=&quot;https://numpy.org/doc/1.18/reference/generated/numpy.ndarray.flatten.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;.flatten()&lt;/code&gt;&lt;/a&gt; to collapse it into one dimension.&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;my_dataframe.values.flatten()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="11007" LastEditorUserId="24000" LastEditDate="2020-05-19T17:14:31.267" LastActivityDate="2020-05-19T17:14:31.267" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18907" PostTypeId="1" AcceptedAnswerId="18908" CreationDate="2017-05-11T14:26:16.190" Score="3" ViewCount="2654" Body="&lt;p&gt;I've been reviewing performance of several NVIDIA GPU's and I see that typically results are presented in terms of &quot;images per second&quot; that can be processed. Experiments are typically being performed on classical network architectures such as Alex Net or GoogLeNet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?. I suppose that if I have 15000 images and want to calculate how fast will a given GPU train that network, I would have to multiply by certain values of my specific configuration (for example number of iterations). In case this is not true, is there a default configuration being used for this tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here an example of benchmark &lt;a href=&quot;http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2017/03/22/deep-learning-inference-on-p40-gpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning Inference on P40 GPUs&lt;/a&gt; (&lt;a href=&quot;https://web.archive.org/web/20170511144434/http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2017/03/22/deep-learning-inference-on-p40-gpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;mirror&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="5143" LastEditorUserId="843" LastEditDate="2017-05-11T14:46:32.933" LastActivityDate="2018-10-14T23:50:15.533" Title="What does images per second mean when benchmarking Deep Learning GPU?" Tags="|deep-learning|gpu|nvidia|alex-net|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18908" PostTypeId="2" ParentId="18907" CreationDate="2017-05-11T14:44:03.237" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Typically they specify somewhere whether they talk about the forward (a.k.a. inference a.k.a. test) time, e.g. from the page you mentioned in your question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vmgn7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vmgn7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example from &lt;a href=&quot;https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/soumith/convnet-benchmarks&lt;/a&gt; (&lt;a href=&quot;https://web.archive.org/web/20170511161150/https://github.com/soumith/convnet-benchmarks&quot; rel=&quot;nofollow noreferrer&quot;&gt;mirror&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QEadO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QEadO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="843" LastEditDate="2017-05-11T16:10:19.323" LastActivityDate="2017-05-11T16:10:19.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18909" PostTypeId="2" ParentId="8983" CreationDate="2017-05-11T15:20:05.090" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But the problem of this dataset is that we have unbalanced data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I think that the way to fix your problem is to use something like SMOTE or one of its variants. (My favorite is SMOTE-ENN, but go with what works the best). &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is an implementation that you can use to fix the class imbalances&lt;/a&gt;. I don't know if this will solve your specific problem, but it is worth a shot.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2017-05-11T15:20:05.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18911" PostTypeId="1" AcceptedAnswerId="18914" CreationDate="2017-05-11T15:45:55.200" Score="0" ViewCount="49" Body="&lt;p&gt;I am trying to pull data from my .csv file. I am using this command: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;mydata[mydata$Model_Data&gt;3.0,c(&quot;Model_Data&quot;,&quot;Date&quot;)]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you can see from the results below, it is counting ALL instances, however, I wish to set this to a 64 line ignore upon first matching instance. How would you suggest I go about adding this 'ignore 'n' line function' ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Snippet of data here: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Model_Data     Date&#xA;1       8.788927 19670103&#xA;2       5.625603 19670104&#xA;3       4.853577 19670105&#xA;4       4.558040 19670106&#xA;5       4.322114 19670109&#xA;6       3.011257 19670110&#xA;7       6.234991 19670111&#xA;8       3.970446 19670112&#xA;9       3.144710 19670113&#xA;11      3.121524 19670117&#xA;15      3.659759 19670123&#xA;314     5.034324 19680401&#xA;316     4.395672 19680403&#xA;320     4.042018 19680410&#xA;485     3.647299 19690113&#xA;750     3.632671 19700203&#xA;785     4.167759 19700325&#xA;809     4.520325 19700429&#xA;829     4.116661 19700527&#xA;1138    7.950606 19710816&#xA;1139    3.260332 19710817&#xA;1493    3.929633 19730111&#xA;1502    3.094216 19730124&#xA;1515    3.728929 19730213&#xA;1570    3.369889 19730503&#xA;1934    3.254718 19741010&#xA;2008    3.845721 19750127&#xA;2021    3.039563 19750213&#xA;2714    4.134147 19771110&#xA;2820    6.223156 19780414&#xA;2821    7.745218 19780417&#xA;2827    4.743293 19780425&#xA;2828    3.033731 19780426&#xA;2896    4.192446 19780802&#xA;2897    4.422611 19780803&#xA;2958    4.189009 19781030&#xA;2960    4.180385 19781101&#xA;3183    3.427686 19790920&#xA;3196    4.392758 19791009&#xA;3197    6.126659 19791010&#xA;3259    3.585480 19800109&#xA;3264    3.165421 19800116&#xA;3275    3.521842 19800131&#xA;3314    3.699859 19800327&#xA;3468    5.436180 19801105&#xA;3510    4.302425 19810107&#xA;3917    3.917657 19820817&#xA;3918    4.391777 19820818&#xA;3920    3.173933 19820820&#xA;3921    3.354431 19820823&#xA;3924    3.382543 19820826&#xA;3930    3.257510 19820903&#xA;3953    3.201376 19821007&#xA;3955    3.558906 19821011&#xA;3957    3.060809 19821013&#xA;3972    3.596346 19821103&#xA;4414    4.728832 19840802&#xA;4415    6.362526 19840803&#xA;4416    3.995445 19840806&#xA;4419    3.081986 19840809&#xA;4420    3.271267 19840810&#xA;4468    3.220568 19841018&#xA;4510    3.585172 19841218&#xA;4759    3.046736 19851213&#xA;4775    5.164241 19860108&#xA;4818    3.460404 19860311&#xA;4899    3.051578 19860707&#xA;4946    5.638514 19860911&#xA;4947    3.806834 19860912&#xA;5039    6.066431 19870123&#xA;5095    3.536616 19870414&#xA;5224    5.730824 19871016&#xA;5225   11.180750 19871019&#xA;5226    6.897399 19871020&#xA;5227    4.537756 19871021&#xA;5228    3.229374 19871022&#xA;5582    3.155970 19890317&#xA;5728    6.600688 19891013&#xA;5729    6.704284 19891016&#xA;5931    4.177266 19900803&#xA;6046    4.380562 19910117&#xA;6257    3.193782 19911115&#xA;6480    3.983774 19921005&#xA;6533    3.139862 19921218&#xA;6572    4.023227 19930216&#xA;6574    3.056474 19930218&#xA;6605    3.035592 19930402&#xA;6637    3.505759 19930519&#xA;6819    3.674848 19940204&#xA;6857    3.853518 19940331&#xA;6858    3.048781 19940404&#xA;7150    3.038139 19950531&#xA;7184    5.645942 19950719&#xA;7242    3.910097 19951010&#xA;7301    3.255512 19960104&#xA;7305    3.574967 19960110&#xA;7346    5.189287 19960308&#xA;7435    5.112708 19960716&#xA;7568    3.238173 19970123&#xA;7577    3.007577 19970205&#xA;7760    5.930298 19971027&#xA;7761    7.836272 19971028&#xA;7953    3.613173 19980804&#xA;7970    3.899849 19980827&#xA;7972    5.255132 19980831&#xA;7973    5.310272 19980901&#xA;7999    3.806758 19981008&#xA;8130    3.407432 19990419&#xA;8265    3.204466 19991028&#xA;8311    3.483312 20000104&#xA;8314    3.101274 20000107&#xA;8324    3.273686 20000124&#xA;8361    3.905785 20000316&#xA;8374    5.991309 20000404&#xA;8382    4.642381 20000414&#xA;8563    4.287756 20010103&#xA;8635    3.294676 20010418&#xA;8736    3.260459 20010917&#xA;8738    3.009326 20010919&#xA;8947    3.005049 20020719&#xA;8950    4.684101 20020724&#xA;9830    3.024372 20060120&#xA;9926    3.253976 20060608&#xA;10106   6.153445 20070227&#xA;10108   3.018874 20070301&#xA;10208   3.016551 20070724&#xA;10210   5.810003 20070726&#xA;10213   3.048170 20070731&#xA;10214   3.093512 20070801&#xA;10216   3.203759 20070803&#xA;10217   3.358139 20070806&#xA;10220   3.897666 20070809&#xA;10225   4.290941 20070816&#xA;10333   3.479401 20080122&#xA;10334   4.699377 20080123&#xA;10497   3.575420 20080915&#xA;10498   3.417303 20080916&#xA;10499   3.578305 20080917&#xA;10500   4.763519 20080918&#xA;10501   3.362684 20080919&#xA;10507   4.354753 20080929&#xA;10512   3.611450 20081006&#xA;10515   3.786056 20081009&#xA;10516   4.791725 20081010&#xA;10517   3.406449 20081013&#xA;10910   7.382690 20100506&#xA;10911   3.423236 20100507&#xA;10912   3.022318 20100510&#xA;11225   4.906076 20110804&#xA;11226   4.635050 20110805&#xA;11227   6.636959 20110808&#xA;11228   5.464225 20110809&#xA;11229   4.077466 20110810&#xA;11230   3.956520 20110811&#xA;11235   3.315989 20110818&#xA;11852   3.357452 20140203&#xA;12029   4.794625 20141015&#xA;12243   3.784646 20150821&#xA;12244   6.559546 20150824&#xA;12245   4.181079 20150825&#xA;12246   3.869772 20150826&#xA;12344   3.089881 20160115&#xA;12346   3.519323 20160120&#xA;12455   3.676909 20160624&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30339" LastActivityDate="2017-08-09T17:49:38.863" Title="Pull data from R and set lines to ignore" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18912" PostTypeId="1" CreationDate="2017-05-11T16:34:07.200" Score="-3" ViewCount="127" Body="&lt;p&gt;I would like to know if my recent work, which is essentially free to implement, has data applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More obvious applications include:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Guessing questions.&lt;/li&gt;&#xA;&lt;li&gt;Logic / objectivity.&lt;/li&gt;&#xA;&lt;li&gt;Psychic prediction.&lt;/li&gt;&#xA;&lt;li&gt;Problem-solving.&lt;/li&gt;&#xA;&lt;li&gt;Finding core content of text based on title alone.&lt;/li&gt;&#xA;&lt;li&gt;A deduction that can be used for finding new theories of physics.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;GUESSING QUESTIONS&#xA;Performing some associations on the neutrals between an original term and it's opposite, and then posing the opposite and neutral as a question may be a general formula for the core questions of any given initial topic. For example, a child may want to know who they will be when they grow up (who they will be is opposite to them, and growing up is neutral to being grown up or being a child).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LOGIC / OBJECTIVITY&#xA;The formula AB:CD AND AD:CB using polar opposites opposed on the diagonal may be the first formula for objectivity, a metaphor for math, and extensible to n-dimensional typologies, and exponentially efficient (deductions equal in number to the square root of the number of categories in the case of mod 4 sets).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PSYCHIC PREDICTIONS&#xA;The most memorable method may be choosing the opposite of a thing or person's category, and the opposite of what concerns them at the time, and combining them. For example, a motorcyclist who is mostly about himself may be concerned about meeting someone, about someone else's motorcycle, or about losing their motorcycle. However, other formulas also exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PROBLEM-SOLVING&#xA;Finding the opposite of every term in the best definition of the problem in the same order as the original corresponding words will solve any paradox, and if the problem includes the word 'problem' and the solution includes the word 'solution' then it also solves any problem that actually is a problem. Care should be taken to know what the problem actually is before solving it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FORMULA FOR CORE CONTENT&#xA;Title='[quality of X] [opposite of qualifier mentioned in core content]'.&#xA;Core Content='If you [X] qualifier [subject of X and qualifier] [opposite of X clarified]'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FORMULA FOR FINDING NEW THEORIES OF PHYSICS&#xA;'X --&gt; String Theory --&gt; Space Time'. I have found this formula effective in reaching new theories of physics after studying intuitive physics and intuitive calculus. A string-theoretic concept such as singularity or superstrings is developed from an initial concept into a further theory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is any of this exciting for app designers or big data scientists in any way? If so, I will try to make the work more helpful for programmers in the future, perhaps by linking to sample code or suggesting more specific applications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I didn't want to mention is I have very little programming experience, just a little object-oriented coding experience, so applying these methods seems to be entirely on the shoulders of other programmers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am actually afraid the methods, as valuable as they are, will die. Of course, I want to prevent that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further information, see link: &lt;a href=&quot;https://nathancoppedge.quora.com/Programmable-Heuristics&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nathancoppedge.quora.com/Programmable-Heuristics&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32156" LastEditorUserId="32156" LastEditDate="2017-05-11T18:16:54.097" LastActivityDate="2017-05-11T19:36:49.823" Title="Are there any programmers who might find programmable heuristics useful?" Tags="|bigdata|" AnswerCount="1" CommentCount="9" ClosedDate="2017-05-16T08:44:35.197" ContentLicense="CC BY-SA 3.0" />
  <row Id="18913" PostTypeId="1" CreationDate="2017-05-11T16:36:09.250" Score="1" ViewCount="171" Body="&lt;p&gt;I am currently trying to train my backpropagation to classify 150k training pairs. Each training pair is a vector of 18 Bipolar numbers and it runs through 2 hidden layers with a final output of 1 number(18-18-18-1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I feed my neural network with only a couple thousand training pairs it can classify them 100% but when I try to feed more it makes many errors. Does anyone have any recommendations for what can help a back prop net handle more training pairs? Should I do batch updating of weights or update them for every training pair(what I do now)? Also whats the best way to test for convergence with so many training pairs(when summing up error it's ~40k).Any help is much appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(FYI: each training pair is a loan account I am trying to evaluate with data on balance, age, how long it's been open, average income of zipcode and the target is whether or not it was paid)&lt;/p&gt;&#xA;" OwnerUserId="32157" LastActivityDate="2017-07-11T16:11:30.570" Title="Help training Back-propagation Neural Network With 150k training pairs" Tags="|machine-learning|neural-network|backpropagation|training|error-handling|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18914" PostTypeId="2" ParentId="18911" CreationDate="2017-05-11T17:13:01.607" Score="1" Body="&lt;p&gt;You almost had everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking example from &lt;code&gt;iris&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- 10&#xA;iris[iris$Sepal.Length&amp;gt;5.5, c(&quot;Sepal.Length&quot;, &quot;Species&quot;)][1:n,]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2017-05-11T17:13:01.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18915" PostTypeId="1" AcceptedAnswerId="18918" CreationDate="2017-05-11T17:53:52.963" Score="2" ViewCount="2139" Body="&lt;p&gt;I am reading the book The Data Warehouse Lifecycle Toolkit by Ralph Kimball. I come across the term &lt;em&gt;Stovepipes&lt;/em&gt; fairly often. After doing some research I read that Stovepipes are when you don't have conformed dimensions to link data marts. This short description was all I could find every other resource I looked at simply said that Stovepipe models are bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is is meant by &quot;conformed dimensions&quot;? What exactly are these &lt;em&gt;Stovepipes&lt;/em&gt; and can somebody provide me with an example?&lt;/p&gt;&#xA;" OwnerUserId="32161" LastEditorUserId="31513" LastEditDate="2017-05-16T08:52:54.627" LastActivityDate="2017-05-16T08:52:54.627" Title="What are stovepipes?" Tags="|terminology|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18917" PostTypeId="2" ParentId="18912" CreationDate="2017-05-11T19:08:22.660" Score="2" Body="&lt;p&gt;The area of computer science this most closely relates to is &lt;a href=&quot;https://en.wikipedia.org/wiki/Natural_language_processing&quot; rel=&quot;nofollow noreferrer&quot;&gt;natural language processing&lt;/a&gt;. There is a large body of work and ongoing research in this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a few barriers to achieving interest in your ideas from people studying the subject:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The methods as you present them, are not tightly defined or easy to comprehend. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Where the methods are comprehensible, they appear to be a form of &lt;a href=&quot;https://en.wikipedia.org/wiki/Expert_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;expert system&lt;/a&gt; which attempt to address questions about a piece of text. Expert systems are not new to NLP and are used in many text comprehension systems, but generally work at the level of text adventures, or in very limited domains (e.g. Amazon's Alexa might use such rules after analysing input, when deciding between playing some music that you requested or booking you a table at a local restaurant)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some of the terms you use in your heuristics (such as &quot;opposite&quot;) are not well defined for all inputs, and could be the subject of a large piece of research in their own right. A computer programmer would likely halt and ask you to define how to extract &quot;quality of X&quot; from a piece of content, before they could apply your rule.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Claims such as &quot;FORMULA FOR FINDING NEW THEORIES OF PHYSICS&quot; will not be well received by any research or academic community. Unless you have actually discovered, published and had verified a new formula for physics using your heuristics, you are most likely to receive negative attention for making such a claim. If you cannot defend this claim about the effectiveness of your methods &lt;em&gt;with proof&lt;/em&gt;, I'd advise not making such a claim.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Probably you could have some of your rules encoded as an expert system (perhaps with some existing NLP analysis to help with identifying the loose concepts in your rule system), and they would generate output for a range of inputs. They might be interesting in some contexts e.g. these rules, or ones like them, might be useful topic generators for &lt;a href=&quot;https://en.wikipedia.org/wiki/Chatbot&quot; rel=&quot;nofollow noreferrer&quot;&gt;chatbots&lt;/a&gt;. However, they don't appear to represent a novel approach to NLP, nor do they align well with open research topics in the subject.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-11T19:36:49.823" LastActivityDate="2017-05-11T19:36:49.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18918" PostTypeId="2" ParentId="18915" CreationDate="2017-05-11T20:38:57.993" Score="2" Body="&lt;p&gt;It seems &quot;stovepiping&quot; is a term that legendary author on the subject of data warehousing &lt;a href=&quot;https://en.wikipedia.org/wiki/Ralph_Kimball&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ralph Kimball&lt;/a&gt; adopted, and it looks that it actually originates from intelligence community. In its original (intelligence community) meaning, it is mostly related to the usage of raw data, instead of processed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://en.wikipedia.org/wiki/Stovepiping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Stovepiping (also stove piping) is a metaphorical term which recalls a stovepipe's function as an isolated vertical conduit, and has been used, in the context of intelligence, to describe several ways in which raw intelligence information may be presented without proper context. It is a system created to solve a specific problem. The lack of context may be due to the specialized nature, or security requirements, of a particular intelligence collection technology. It also has limited focus and data within is not easily shared.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;http://rationalwiki.org/wiki/Stovepiping&quot; rel=&quot;nofollow noreferrer&quot;&gt;RationalWiki&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Stovepiping is a term originating in the intelligence community to describe a process by which raw data is funneled directly to high-ranking officials or the media. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also, there is an interesting article in &lt;a href=&quot;http://www.phrases.org.uk/bulletin_board/34/messages/842.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PhraseFinder&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Even better explanation from &lt;a href=&quot;http://www.relationalsolutions.com/blog/data-warehouse-or-data-mart-whats-the-difference&quot; rel=&quot;nofollow noreferrer&quot;&gt;Relational Solutions Blog&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Most data marts are designed as one off reporting solutions, designed to solve an immediate problem that the business users need to solve. When designed as a stand-alone, they are often referred to as “stove pipes” or “silos” of information.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Analysts new to the space think these are new terms, but Data warehousing consultants have used these terms since the 90’s. They’re used to describe stand-alone reporting solutions. Typically these stand-alone solutions are developed by individual teams or departments.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;These groups develop “silo’s” or “stove pipe” reporting databases to achieve a specific goal that they were unable to get financial approval for. If they have a need for something that you can't get approval for, you resort to building something on your own. It happens in every company and every department.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="31513" LastEditDate="2017-05-15T16:54:56.777" LastActivityDate="2017-05-15T16:54:56.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18919" PostTypeId="1" AcceptedAnswerId="18928" CreationDate="2017-05-12T03:18:10.817" Score="8" ViewCount="4019" Body="&lt;p&gt;When evaluating a trained binary classification model we often evaluate the misclassification rates, precision-recall, and AUC. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, one useful feature of classification algorithms are the probability estimates they give, which support the label predictions made by the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These probabilities can be useful for a variety of reasons depending on the use case. When using these probabilities it would be useful to have a confidence interval rather than a single point estimate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can we estimate a probability confidence interval given that the misclassification error may not always serve as a proxy for the error between the estimated probability and the actual probability (which is often unknown)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've considered using brier score but I'm sure there is a better way. Can anyone point me in the right direction or offer your own insight?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, If I have classes &lt;code&gt;[C0, C1]&lt;/code&gt; and my probabilities for a given  instance $(x^{(i)}, y^{(i)})$&#xA; are &lt;code&gt;{C0: 80, C1:20}&lt;/code&gt; then I will classify this instance as &lt;code&gt;C0&lt;/code&gt;. Let's suppose that &lt;code&gt;C0&lt;/code&gt; is the correct class label, at this point the model has done it's job and made the correct classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to go another step further and use the probabilities &lt;code&gt;{C0:80, C1:20}&lt;/code&gt; which could be useful for a variety of reasons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say &lt;code&gt;C0&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; respectively represent a customer keeping and closing their account with a bank. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we wanted to create an expected value $EV$ of dollars at risk of leaving the bank we could calculate $EV$ as $P(C1) * account \space balance$. This would give us a point estimate of $EV$, which is fine but may not tell the whole story given our uncertainty in the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can we provide a lower limit and upper limit of the probability this instance is of class &lt;code&gt;C1&lt;/code&gt; with 95% confidence? &lt;/p&gt;&#xA;" OwnerUserId="32071" LastEditorUserId="32071" LastEditDate="2017-05-12T15:37:03.573" LastActivityDate="2019-08-29T09:10:26.923" Title="Confidence intervals for binary classification probabilities" Tags="|machine-learning|classification|probability|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18920" PostTypeId="1" CreationDate="2017-05-12T03:42:22.460" Score="2" ViewCount="840" Body="&lt;p&gt;Why we prefer VIF if we can find multicollinearity from the correlation matrix as well? What is the exact logic behind it?&lt;/p&gt;&#xA;" OwnerUserId="32167" LastEditorUserId="83275" LastEditDate="2021-03-12T05:49:44.667" LastActivityDate="2021-03-12T05:49:44.667" Title="Why we prefer VIF if we can find multicollinearity from correlation matrix as well?" Tags="|predictive-modeling|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18921" PostTypeId="2" ParentId="18920" CreationDate="2017-05-12T07:46:46.503" Score="2" Body="&lt;p&gt;If you have 10000 rows and 100 columns which have all 100 1s and the rest 0s. Every row has a 1 and the rest 0, a classic one hot encoded matrix. The correlations between two random columns is -0.01010101, which means the correlation matrix has a diagonal of 1s and the rest is -0.01010101. However this matrix is perfectly multicolinear.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-12T07:46:46.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18925" PostTypeId="2" ParentId="18913" CreationDate="2017-05-12T13:51:30.130" Score="1" Body="&lt;p&gt;Typical things to look for when training neural networks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Learning Curve: a plot of the error on the validation set X size of training set. The error must decrease as the network is trained, so if you see fluctuations or an increase something is wrong.&lt;/li&gt;&#xA;&lt;li&gt;Learning Rate: the learning rate influences the convergence of gradient descent. A small learning rate will make the model overfit the data while a big one may make it underfit. &lt;/li&gt;&#xA;&lt;li&gt;Precision: of the classified examples in the training set, how many did the network got right, or more precisely, how many did it classify as expected?&lt;/li&gt;&#xA;&lt;li&gt;Recall: of all the positive examples, in your case the paid ones, how many did it classify as &lt;code&gt;paid&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you notice the performance of the network is bad you may:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Add other features that make the model fit the data better - for example add average family income to the features&lt;/li&gt;&#xA;&lt;li&gt;Get more training examples&lt;/li&gt;&#xA;&lt;li&gt;Some papers suggest deeper networks are able to catch subtleties about the model they are trying to fit, so experimenting with the network architecture is an option.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="32177" LastActivityDate="2017-05-12T13:51:30.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18926" PostTypeId="2" ParentId="18760" CreationDate="2017-05-12T14:28:16.317" Score="1" Body="&lt;p&gt;k-means tries to minimize a sum-of-squares function, looking roughly like this:&#xA;$$\sum_x \sum_i (x_i - \mu_{xi})^2$$ where $\mu_x$ is the centroid of the closest cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This whole expression is only &lt;em&gt;sensible&lt;/em&gt; to use on &lt;em&gt;continuous variables&lt;/em&gt;. It makes no sense to use it on IP adresses, or port numbers, or binary numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example on port, port 80 (http) and 443 (https) are &lt;em&gt;closely&lt;/em&gt; related, but their &quot;distance&quot; is $363^2 = 131769$. The port 88, (kerberos), which is used for a single-sign-on type of authentication system and &lt;em&gt;not&lt;/em&gt; related to http has a distance of only $8^2=64$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, while the attribute is &quot;numbers&quot;, it nevertheless must not be used with the k-means objective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even with &quot;real&quot; numbers, this objective may be a bad idea. For example, the number of connections made, or the number of bytes transferred are not linear. The difference between 0 connection attempts (or 0 bytes transferred) and 1 connection attempt (or 1 megabyte transferred) is &lt;em&gt;much&lt;/em&gt; more important than the difference between 1000 vs. 1001 connection attempts (or 1000 vs. 1001 megabytes transferred) - but with the k-means objective, these difference are equal.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Therefore, use a &lt;em&gt;different&lt;/em&gt; clustering algorithm, not $k$-means.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;k-means isn't suitable for this kind of data. K-means assumes continuous variables (if you observe a difference of x, you may also observe 2*x or 0.5*x), translation invariance (a difference of 1 is the same at 0 and at any other location x), and that larger errors are much more severe (a difference of 2 is 4 times as important as a difference of 1).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-05-12T14:28:16.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18928" PostTypeId="2" ParentId="18919" CreationDate="2017-05-12T16:57:37.133" Score="5" Body="&lt;p&gt;I don't think there is a good way to do this for all models, however for a lot of models it's possible to get a sense of uncertainty (this is the keyword you are looking for) in your predictions. I'll list a few:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bayesian logistic regression gives a probability distribution over probabilities. MCMC can sample weights from your logistic regression (or more sophisticated) model which in turn predict different probabilities. If the variance in these probabilities are high you are less certain about the predictions, you could empirically take the 5% quantile or something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With neural networks you could train them with dropout (not a bad idea in general) and then instead of testing without the dropout, you do multiple forward passes per prediction and this way you sample from different models. If the variance is high, again you are more uncertain. Variational Inference is another way to sample networks and sample from these different networks to get a measure of uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know from the top of my head but I'm sure you could do something with random forests with the variance between the different end nodes where your features end up, assuming they are not deep, but this is just something I thought of.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-12T16:57:37.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18929" PostTypeId="2" ParentId="12575" CreationDate="2017-05-12T17:49:22.970" Score="14" Body="&lt;p&gt;Apart from very good responses here, you may try SequenceMatcher in difflib python library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.python.org/2/library/difflib.html&quot; rel=&quot;noreferrer&quot;&gt;https://docs.python.org/2/library/difflib.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import difflib&#xA;&#xA;a = 'Thanks for calling America Expansion'&#xA;b = 'Thanks for calling American Express'&#xA;&#xA;seq = difflib.SequenceMatcher(None,a,b)&#xA;d = seq.ratio()*100&#xA;print(d) &#xA;### OUTPUT: 87.323943&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now Consider the below code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = 'Thanks for calling American Expansion'&#xA;b = 'Thanks for calling American Express'&#xA;&#xA;seq = difflib.SequenceMatcher(None,a,b)&#xA;d = seq.ratio()*100&#xA;print(d)&#xA;### OUTPUT: 88.88888&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you may compare the d value to evaluate the similarity.&lt;/p&gt;&#xA;" OwnerUserId="32189" LastEditorUserId="67883" LastEditDate="2019-02-18T16:05:33.037" LastActivityDate="2019-02-18T16:05:33.037" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="18930" PostTypeId="1" CreationDate="2017-05-12T18:00:38.533" Score="1" ViewCount="647" Body="&lt;p&gt;Usually people construct neural nets with fewer nodes in the higher layers than the lower layers (excluding the input layer, which could sometimes have fewer columns than the lowest hidden layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there situations where it makes sense to have a diamond-shaped network, instead of a pyramid shaped network?  I suppose that such a network could be viewed as a basis expansion in the hidden units, much as a network with a few inputs and many hidden layers could be used to model nonlinearities in those hidden units.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computationally, a diamond-shaped network has many fewer weights as well.  This is because thousands of variables become many many weights when each one feeds into several nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, all information from those thousands of variables needs to pass through a few nodes.  This might not be rich enough for some problems.  But to what degree can this be mitigated by just adding lots and lots of layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively consider the question this way:  given that neural nets with one layer and enough nodes are universal approximators, is it true that a neural net with a small, number of nodes per layers, but many layers, will also be a universal approximator?&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-05-12T18:00:38.533" Title="Does it ever make sense for upper layers to have more nodes than lower layers?" Tags="|neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18931" PostTypeId="1" CreationDate="2017-05-12T19:18:16.590" Score="-1" ViewCount="2031" Body="&lt;p&gt;I have the following simple weka code to use a simple decision tree, train it, and then make predictions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import java.util.logging.Level;&#xA;import java.util.logging.Logger;&#xA;import weka.classifiers.Classifier;&#xA;import weka.classifiers.trees.SimpleCart;&#xA;import weka.core.Instances;&#xA;import weka.core.converters.ConverterUtils;&#xA;&#xA;public class weka_test {&#xA;&#xA;    public static String labelFeature = &quot;LABEL&quot;;&#xA;&#xA;    public Instances readData(String file) throws Exception {&#xA;        ConverterUtils.DataSource source = new ConverterUtils.DataSource(file);&#xA;        return source.getDataSet();&#xA;    }&#xA;&#xA;    public double accuracy(Classifier myClass, Instances testData) {&#xA;        float correctPreds = 0;&#xA;        for (int i = 0; i &amp;lt; testData.numInstances(); i++){&#xA;            String actual = testData.instance(i).stringValue(testData.classAttribute());&#xA;&#xA;            String label = null;&#xA;            try {&#xA;                double pred = myClass.classifyInstance(testData.instance(i));&#xA;                label = testData.classAttribute().value((int) pred);&#xA;            } catch (Exception ex) {&#xA;                Logger.getLogger(CategorizationTrain.class.getName()).log(Level.SEVERE, null, ex);&#xA;            }&#xA;&#xA;            if (label.equals(actual)) {&#xA;                correctPreds++;&#xA;            }&#xA;        }       &#xA;        return correctPreds / testData.numInstances();&#xA;    }&#xA;&#xA;    public static void main(String[] args) throws Exception {&#xA;        CategorizationTrain t = new CategorizationTrain();&#xA;&#xA;        Instances data = null;&#xA;        data = t.readData(&quot;train.csv&quot;);&#xA;&#xA;        if (data == null) {&#xA;            System.exit(-1);&#xA;        }&#xA;        else {&#xA;             System.out.println(&quot;data was read and the size is: &quot; + data.numAttributes() + &quot;x&quot; + data.numInstances());&#xA;        }&#xA;&#xA;        // Set label&#xA;        data.setClass(data.attribute(labelFeature));&#xA;&#xA;        System.out.println(&quot;label: &quot; + data.classAttribute().name() + &quot; --- index: &quot; + data.classIndex());&#xA;&#xA;&#xA;        // Try SimpleCart decision tree/classifier&#xA;        SimpleCart tree = new SimpleCart(); &#xA;&#xA;        tree.buildClassifier(data);&#xA;&#xA;        double acc = t.accuracy(tree, data);&#xA;        System.out.println(&quot;Accuracy = &quot; + acc);  &#xA;&#xA;&#xA;        System.out.println(&quot;tree(SimpleCart): &quot; + tree);    &#xA;        weka.core.SerializationHelper.write(&quot;tree.model&quot;, tree);&#xA;&#xA;        SimpleCart newTree = null;&#xA;        newTree = (SimpleCart) weka.core.SerializationHelper.read(&quot;tree.model&quot;);&#xA;&#xA;        acc = t.accuracy(newTree, data);&#xA;        System.out.println(&quot;Accuracy = &quot; + acc);         &#xA;&#xA;        Instances test = null;&#xA;&#xA;        test = t.readData(&quot;test.csv&quot;);        &#xA;        test.setClass(test.attribute(labelFeature));        &#xA;        System.out.println(test.attribute(labelFeature));&#xA;&#xA;        double newPred = -1.0;&#xA;        for (int i=0; i&amp;lt;test.numInstances(); i++) {&#xA;            newPred = newTree.classifyInstance(test.instance(i));&#xA;            System.out.println(test.instance(i));&#xA;            String label = data.classAttribute().value((int) newPred);&#xA;            System.out.println(&quot;Index: &quot; + newPred + &quot; Prediction = &quot; + label);  &#xA;            newPred = -1.0;&#xA;        }&#xA;        System.out.println(&quot;-----------------------------&quot;);&#xA;        for (int i=0; i&amp;lt;data.numInstances(); i++) {&#xA;            newPred = newTree.classifyInstance(data.instance(i));&#xA;            System.out.println(data.instance(i));&#xA;            String label = data.classAttribute().value((int) newPred);&#xA;            System.out.println(&quot;Index: &quot; + newPred + &quot; Prediction = &quot; + label);  &#xA;            newPred = -1.0;&#xA;        }&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My train data is just:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;VALUE   LABEL   KEYWORD&#xA;300     x28x    John&#xA;-71.46  x1x     Mary&#xA;-22     x28x    John&#xA;-2207.99    x26x    Mary&#xA;193.33  x28x    John&#xA;12200   x28x    Mary&#xA;-16     x7x     James&#xA;-1.29   x7x     James&#xA;-120    x1x     Mary&#xA;-6.7    x28x    John&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and my test data is the same without the &quot;label&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;VALUE   LABEL   KEYWORD&#xA;300             John&#xA;-71.46          Mary&#xA;-22             John&#xA;-2207.99        Mary&#xA;193.33          John&#xA;12200           Mary&#xA;-16             James&#xA;-1.29           James&#xA;-120            Mary&#xA;-6.7            John&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since test and train are the same, I was expecting the same predictions, but I have different results. What am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predictions on test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;300,?,John&#xA;Index: 1.0 Prediction = x1x&#xA;-71.46,?,Mary&#xA;Index: 0.0 Prediction = x28x&#xA;-22,?,John&#xA;Index: 1.0 Prediction = x1x&#xA;-2207.99,?,Mary&#xA;Index: 0.0 Prediction = x28x&#xA;193.33,?,John&#xA;Index: 1.0 Prediction = x1x&#xA;12200,?,Mary&#xA;Index: 1.0 Prediction = x1x&#xA;-16,?,James&#xA;Index: 0.0 Prediction = x28x&#xA;-1.29,?,James&#xA;Index: 0.0 Prediction = x28x&#xA;-120,?,Mary&#xA;Index: 0.0 Prediction = x28x&#xA;-6.7,?,John&#xA;Index: 1.0 Prediction = x1x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Predictions on train:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;300,x28x,John&#xA;Index: 0.0 Prediction = x28x&#xA;-71.46,x1x,Mary&#xA;Index: 1.0 Prediction = x1x&#xA;-22,x28x,John&#xA;Index: 0.0 Prediction = x28x&#xA;-2207.99,x26x,Mary&#xA;Index: 1.0 Prediction = x1x&#xA;193.33,x28x,John&#xA;Index: 0.0 Prediction = x28x&#xA;12200,x28x,Mary&#xA;Index: 0.0 Prediction = x28x&#xA;-16,x7x,James&#xA;Index: 3.0 Prediction = x7x&#xA;-1.29,x7x,James&#xA;Index: 3.0 Prediction = x7x&#xA;-120,x1x,Mary&#xA;Index: 1.0 Prediction = x1x&#xA;-6.7,x28x,John&#xA;Index: 0.0 Prediction = x28x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2017-05-13T13:35:20.027" LastActivityDate="2017-05-13T13:35:20.027" Title="Simple Weka classification example in Java gives inconsistent answers" Tags="|decision-trees|java|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18932" PostTypeId="2" ParentId="18931" CreationDate="2017-05-12T19:57:18.013" Score="0" Body="&lt;p&gt;Just because you trained the model on a dataset, it does not mean it will have 100% accuracy in predicting on it. I ran a similar test today on another dataset I was working with and only got 90% accuracy when I ran the model on training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;100% is actually really uncommon and usually means you are overfitting. Try showing the confusion matrix for your training run and see how much Weka claimed it got right. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-12T19:57:18.013" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18933" PostTypeId="1" AcceptedAnswerId="19067" CreationDate="2017-05-12T22:40:59.083" Score="4" ViewCount="9013" Body="&lt;p&gt;For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better interpret the feature (named &lt;code&gt;ipc&lt;/code&gt; here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GpQbK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GpQbK.png&quot; alt=&quot;Distribution of ipc feature&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regular log transformation does not work here because of the (x-axis) spread.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I transform this data to a normal distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A related answer has been pointed out in the comment but I am looking for some Python code excerpt as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="24651" LastEditorUserId="24651" LastEditDate="2017-05-13T00:01:02.257" LastActivityDate="2019-06-16T21:12:09.940" Title="Convert exponential to normal distribution" Tags="|machine-learning|preprocessing|normalization|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18934" PostTypeId="1" CreationDate="2017-05-12T23:33:40.157" Score="1" ViewCount="73" Body="&lt;p&gt;Suppose I want (the machine) to learn the &quot;&gt;&quot; function, and my training set is a collection of pairs of numbers &lt;span class=&quot;math-container&quot;&gt;$(n_1, n_2)$&lt;/span&gt; with the output &lt;code&gt;True&lt;/code&gt; if &lt;span class=&quot;math-container&quot;&gt;$n_1 &amp;gt; n_2$&lt;/span&gt; and &lt;code&gt;False&lt;/code&gt; otherwise. Which method is the most likely to work (actually, my secret goal is a form of the Netflix challenge - I have a set of &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt;-tuples of real and boolean values, and I want to rank them based on some training rank data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has surely been studied a lot...&lt;/p&gt;&#xA;" OwnerUserId="32196" LastEditorUserId="29169" LastEditDate="2019-06-07T06:02:28.490" LastActivityDate="2019-06-07T06:02:28.490" Title="Learning comparisons" Tags="|ranking|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18935" PostTypeId="1" AcceptedAnswerId="18937" CreationDate="2017-05-13T01:44:01.140" Score="0" ViewCount="110" Body="&lt;p&gt;Which is better for beginner in machine learning: The deep learning book written by Yoshua Bengio or the videos and notes in CS231n from Stanford?&lt;/p&gt;&#xA;" OwnerUserId="32198" LastActivityDate="2017-05-13T06:08:53.680" Title="Advice on the learning resources for deep learning" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2017-05-16T08:51:10.950" ContentLicense="CC BY-SA 3.0" />
  <row Id="18936" PostTypeId="2" ParentId="18920" CreationDate="2017-05-13T06:06:44.420" Score="4" Body="&lt;p&gt;The correlation matrix is not a reliable measurement for multicollinearity  because it only considers the pairwise effects. Unfortunately, multicollinearity is defined as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Phenomenon in which &lt;strong&gt;two or more&lt;/strong&gt; predictor variables in a multiple regression model are highly correlated,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Do you see the point? You'll need to consider the correlation with all other variables in your data set, not just 1-to-1 pairwise comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;VIF addresses the issue.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-05-13T06:06:44.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18937" PostTypeId="2" ParentId="18935" CreationDate="2017-05-13T06:08:53.680" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.deeplearningbook.org/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;They are the two very popular online free books. The first link even has working deep learning code.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-05-13T06:08:53.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18938" PostTypeId="1" CreationDate="2017-05-13T07:13:45.503" Score="0" ViewCount="96" Body="&lt;p&gt;I'm currently thinking of the system that would compare 2 texts (source, target) and suggest changes to the source text to match more the style of the target. It should not convert it to the target but more suggest to rephrase some sentences, changes words, add some key words and so on.&#xA;Any ideas how this could be done?&#xA;Thanks a lot for any suggestions!&lt;/p&gt;&#xA;" OwnerUserId="18501" LastActivityDate="2017-05-26T13:30:07.773" Title="How to programmatically suggest changes to a text to match some other text?" Tags="|neural-network|deep-learning|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18940" PostTypeId="1" AcceptedAnswerId="18948" CreationDate="2017-05-13T11:17:52.367" Score="4" ViewCount="700" Body="&lt;p&gt;I have a simple neural network with one hidden layer and &lt;em&gt;softmax&lt;/em&gt; as the activation function for the output layer. The hidden layer uses various activation functions since I am testing and implementing as many of them as I can. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For training and testing for the moment I am using the MNIST dataset of handwritten digits so my input data is a matrix that in each row has a different image and in each column a pixel of that image that has been reshaped as a vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use a &lt;em&gt;sigmoid&lt;/em&gt; activation function for both layers the computed gradients and analytical gradients seem to agree but when I try something else like &lt;em&gt;tanh&lt;/em&gt; or &lt;em&gt;softplus&lt;/em&gt; for the hidden layer and &lt;strong&gt;softmax&lt;/strong&gt; for the output layer there are big differences as can be seen from the data below (Left-&gt;Numerical Gradient, Right-&gt;Analytical Gradient)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(1)sigmoid (2)softmax&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  -9.4049e-04  -6.4143e-04&#xA;  -6.2623e-05  -2.5895e-05&#xA;   1.0676e-03   6.9474e-04&#xA;  -2.0473e-03  -1.3471e-03&#xA;   2.9846e-03   1.9716e-03&#xA;   4.0945e-05   2.7627e-05&#xA;  -2.5102e-05  -1.7017e-05&#xA;   8.8054e-06   6.0967e-06&#xA;   7.8509e-06   5.0682e-06&#xA;  -2.4561e-05  -1.6270e-05&#xA;   5.6108e-05   3.8449e-05&#xA;   2.0690e-05   1.2590e-05&#xA;  -9.7665e-05  -6.3771e-05&#xA;   1.7235e-04   1.1345e-04&#xA;  -2.4335e-04  -1.6071e-04&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(1)tanh (2)softmax&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  -3.9826e-03  -2.7402e-03&#xA;   4.6667e-05   1.1115e-04&#xA;   3.9368e-03   2.5504e-03&#xA;  -7.7824e-03  -5.1228e-03&#xA;   1.1451e-02   7.5781e-03&#xA;   1.5897e-04   1.0734e-04&#xA;  -9.6886e-05  -6.5701e-05&#xA;   3.3560e-05   2.3153e-05&#xA;   3.3344e-05   2.1786e-05&#xA;  -1.0282e-04  -6.8409e-05&#xA;   2.1185e-04   1.4774e-04&#xA;   9.0293e-05   5.3752e-05&#xA;  -4.0012e-04  -2.6047e-04&#xA;   6.9648e-04   4.5839e-04&#xA;  -9.7518e-04  -6.4468e-04&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(1)sigmoid (2)sigmoid&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-9.2783e-03  -9.2783e-03&#xA;   8.8991e-03   8.8991e-03&#xA;  -8.3601e-03  -8.3601e-03&#xA;   7.6281e-03   7.6281e-03&#xA;  -6.7480e-03  -6.7480e-03&#xA;  -3.0498e-06  -3.0498e-06&#xA;   1.4287e-05   1.4287e-05&#xA;  -2.5938e-05  -2.5938e-05&#xA;   3.6988e-05   3.6988e-05&#xA;  -4.6876e-05  -4.6876e-05&#xA;  -1.7506e-04  -1.7506e-04&#xA;   2.3315e-04   2.3315e-04&#xA;  -2.8747e-04  -2.8747e-04&#xA;   3.3532e-04   3.3532e-04&#xA;  -3.7622e-04  -3.7622e-04&#xA;  -9.6266e-05  -9.6266e-05&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The way I implement backpropagation is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variables-&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Theta1&lt;/strong&gt;, &lt;strong&gt;Theta2&lt;/strong&gt;: &lt;em&gt;tables with the various weights for the first and second layer.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;m&lt;/strong&gt;: &lt;em&gt;size of my training set&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;y&lt;/strong&gt;: &lt;em&gt;a vector with the correct category for every input sample&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Y&lt;/strong&gt;: &lt;em&gt;a matrix with the one hot encoding for the category for every input sample&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;X&lt;/strong&gt;: &lt;em&gt;a matrix with input data, each row is a different training sample&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% Feedforward&#xA;a1 = [ones(m, 1) X];&#xA;z2 = a1*Theta1';&#xA;a2 = [ones(m, 1) activation(z2, activation_type)];&#xA;z3 = a2*Theta2';&#xA;a3 = activation(z3, 'softmax');&#xA;h = a3;&#xA;&#xA;% Calculate J&#xA;J = sum(sum((-Y).*log(h) - (1-Y).*log(1-h), 2))/m + lambda*p/(2*m); # sigmoid&#xA;%J = -(sum(sum((Y).*log(h))) + lambda*p/(2*m)); # softmax&#xA;&#xA;% Calculate sigmas&#xA;sigma3 = a3.-Y;&#xA;sigma2 = (sigma3*Theta2).*activationGradient([ones(m, 1) z2], 'sigmoid');&#xA;sigma2 = sigma2(:, 2:end);&#xA;&#xA;% Accumulate gradients&#xA;delta_1 = (sigma2'*a1);&#xA;delta_2 = (sigma3'*a2);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first cost calculation &lt;strong&gt;J&lt;/strong&gt; was computed for &lt;strong&gt;sigmoid&lt;/strong&gt; and the one below it for &lt;strong&gt;softmax&lt;/strong&gt; (see comments) so I switch between the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have I missed something during backpropagation, why is it working as expected with &lt;strong&gt;sigmoids&lt;/strong&gt; but not as expected with &lt;strong&gt;sofmax&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="28600" LastEditorUserId="28600" LastEditDate="2017-05-14T17:36:03.640" LastActivityDate="2017-05-14T17:36:03.640" Title="Issue with backpropagation using a 2 layer network and softmax" Tags="|machine-learning|neural-network|matlab|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18942" PostTypeId="1" CreationDate="2017-05-13T14:46:41.980" Score="-1" ViewCount="3126" Body="&lt;p&gt;I have a spreadsheet of banking information and one of the goals is to find out the failure rate of an advertising campaign. I think I need to get a count of the total number of entries vs. the total number of non-subscribers. The column for subscribers is a simple &quot;yes&quot; or &quot;no&quot; instead of an integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, getting the total number of entries was easy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; val input = sc.textFile(&quot;project_1_data.csv&quot;)&#xA;input: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at &amp;lt;console&amp;gt;:27&#xA;&#xA;scala&amp;gt; input.count()&#xA;res0: Long = 45211&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, I filter for &quot;no&quot; in the subscriber column (called &quot;y&quot;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scala&amp;gt; val sub = bankDF.filter($&quot;y&quot; === &quot;no&quot;)&#xA;sub: org.apache.spark.sql.DataFrame = [age: int, job: string, marital: string, education: string, default: string, balance: int, housing: string, loan: string, contact: string, day: string, month: string, duration: int, campaign: int, pdays: string, previous: int, poutcome: string, &#xA;y: string]&#xA;&#xA;scala&amp;gt; sub.count()&#xA;res2: Long = 39922&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So now I have two numbers to work with, but they're the results of two different operations. How can I work out a percentage of these two numbers in a single pass?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if I may add a second question, is there maybe a reference page on how to do averages, medians, means, etc. - so I don't have to ask you guys questions every time? Thanks in advance for any assistance!&lt;/p&gt;&#xA;" OwnerUserId="32153" LastActivityDate="2017-05-18T23:41:38.060" Title="Spark: calculating percentage from a non-Int field" Tags="|apache-spark|sql|scala|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18944" PostTypeId="1" AcceptedAnswerId="18945" CreationDate="2017-05-13T16:03:22.243" Score="0" ViewCount="2740" Body="&lt;p&gt;As in &lt;a href=&quot;https://arxiv.org/abs/1403.6382&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, the author is using CNN to extract features of the images, and then doing SVM for further analysis. My question is how to extract features in CNN?&lt;/p&gt;&#xA;&lt;p&gt;E.g., here is a CNN code I'm using:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;%matplotlib inline&#xA;import matplotlib.pyplot as plt&#xA;import tensorflow as tf&#xA;import numpy as np&#xA;from sklearn.metrics import confusion_matrix&#xA;import time&#xA;from datetime import timedelta&#xA;import math&#xA; &#xA;# Convolutional Layer 1.&#xA;filter_size1 = 5          # Convolution filters are 5 x 5 pixels.&#xA;num_filters1 = 16         # There are 16 of these filters.&#xA; &#xA;# Convolutional Layer 2.&#xA;filter_size2 = 5          # Convolution filters are 5 x 5 pixels.&#xA;num_filters2 = 36         # There are 36 of these filters.&#xA; &#xA;# Fully-connected layer.&#xA;fc_size = 128             # Number of neurons in fully-connected layer.&#xA; &#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;data = input_data.read_data_sets('data/MNIST/', one_hot=True)&#xA; &#xA;data.test.cls = np.argmax(data.test.labels, axis=1)&#xA; &#xA;# We know that MNIST images are 28 pixels in each dimension.&#xA;img_size = 28&#xA; &#xA;# Images are stored in one-dimensional arrays of this length.&#xA;img_size_flat = img_size * img_size&#xA; &#xA;# Tuple with height and width of images used to reshape arrays.&#xA;img_shape = (img_size, img_size)&#xA; &#xA;# Number of colour channels for the images: 1 channel for gray-scale.&#xA;num_channels = 1&#xA; &#xA;# Number of classes, one class for each of 10 digits.&#xA;num_classes = 10&#xA; &#xA;def new_weights(shape):&#xA;    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))&#xA; &#xA;def new_biases(length):&#xA;    return tf.Variable(tf.constant(0.05, shape=[length]))&#xA; &#xA;def new_conv_layer(input,              # The previous layer.&#xA;                   num_input_channels, # Num. channels in prev. layer.&#xA;                   filter_size,        # Width and height of each filter.&#xA;                   num_filters,        # Number of filters.&#xA;                   use_pooling=True):  # Use 2x2 max-pooling.&#xA; &#xA;    # Shape of the filter-weights for the convolution.&#xA;    # This format is determined by the TensorFlow API.&#xA;    shape = [filter_size, filter_size, num_input_channels, num_filters]&#xA; &#xA;    # Create new weights aka. filters with the given shape.&#xA;    weights = new_weights(shape=shape)&#xA; &#xA;    # Create new biases, one for each filter.&#xA;    biases = new_biases(length=num_filters)&#xA; &#xA;    # Create the TensorFlow operation for convolution.&#xA;    # Note the strides are set to 1 in all dimensions.&#xA;    # The first and last stride must always be 1,&#xA;    # because the first is for the image-number and&#xA;    # the last is for the input-channel.&#xA;    # But e.g. strides=[1, 2, 2, 1] would mean that the filter&#xA;    # is moved 2 pixels across the x- and y-axis of the image.&#xA;    # The padding is set to 'SAME' which means the input image&#xA;    # is padded with zeroes so the size of the output is the same.&#xA;    layer = tf.nn.conv2d(input=input,&#xA;                         filter=weights,&#xA;                         strides=[1, 1, 1, 1],&#xA;                         padding='SAME')&#xA; &#xA;    # Add the biases to the results of the convolution.&#xA;    # A bias-value is added to each filter-channel.&#xA;    layer += biases&#xA; &#xA;    # Use pooling to down-sample the image resolution?&#xA;    if use_pooling:&#xA;        # This is 2x2 max-pooling, which means that we&#xA;        # consider 2x2 windows and select the largest value&#xA;        # in each window. Then we move 2 pixels to the next window.&#xA;        layer = tf.nn.max_pool(value=layer,&#xA;                               ksize=[1, 2, 2, 1],&#xA;                               strides=[1, 2, 2, 1],&#xA;                               padding='SAME')&#xA; &#xA;    # Rectified Linear Unit (ReLU).&#xA;    # It calculates max(x, 0) for each input pixel x.&#xA;    # This adds some non-linearity to the formula and allows us&#xA;    # to learn more complicated functions.&#xA;    layer = tf.nn.relu(layer)&#xA; &#xA;    # Note that ReLU is normally executed before the pooling,&#xA;    # but since relu(max_pool(x)) == max_pool(relu(x)) we can&#xA;    # save 75% of the relu-operations by max-pooling first.&#xA; &#xA;    # We return both the resulting layer and the filter-weights&#xA;    # because we will plot the weights later.&#xA;    return layer, weights&#xA; &#xA;def flatten_layer(layer):&#xA;    # Get the shape of the input layer.&#xA;    layer_shape = layer.get_shape()&#xA; &#xA;    # The shape of the input layer is assumed to be:&#xA;    # layer_shape == [num_images, img_height, img_width, num_channels]&#xA; &#xA;    # The number of features is: img_height * img_width * num_channels&#xA;    # We can use a function from TensorFlow to calculate this.&#xA;    num_features = layer_shape[1:4].num_elements()&#xA;    &#xA;    # Reshape the layer to [num_images, num_features].&#xA;    # Note that we just set the size of the second dimension&#xA;    # to num_features and the size of the first dimension to -1&#xA;    # which means the size in that dimension is calculated&#xA;    # so the total size of the tensor is unchanged from the reshaping.&#xA;    layer_flat = tf.reshape(layer, [-1, num_features])&#xA; &#xA;    # The shape of the flattened layer is now:&#xA;    # [num_images, img_height * img_width * num_channels]&#xA; &#xA;    # Return both the flattened layer and the number of features.&#xA;    return layer_flat, num_features&#xA; &#xA; &#xA;def new_fc_layer(input,          # The previous layer.&#xA;                 num_inputs,     # Num. inputs from prev. layer.&#xA;                 num_outputs,    # Num. outputs.&#xA;                 use_relu=True): # Use Rectified Linear Unit (ReLU)?&#xA; &#xA;    # Create new weights and biases.&#xA;    weights = new_weights(shape=[num_inputs, num_outputs])&#xA;    biases = new_biases(length=num_outputs)&#xA; &#xA;    # Calculate the layer as the matrix multiplication of&#xA;    # the input and weights, and then add the bias-values.&#xA;    layer = tf.matmul(input, weights) + biases&#xA; &#xA;    # Use ReLU?&#xA;    if use_relu:&#xA;        layer = tf.nn.relu(layer)&#xA; &#xA;    return layer&#xA; &#xA;x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')&#xA;x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])&#xA;y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')&#xA;y_true_cls = tf.argmax(y_true, dimension=1)&#xA; &#xA;layer_conv1, weights_conv1 = \&#xA;    new_conv_layer(input=x_image,&#xA;                   num_input_channels=num_channels,&#xA;                   filter_size=filter_size1,&#xA;                   num_filters=num_filters1,&#xA;                   use_pooling=True)&#xA; &#xA;layer_conv2, weights_conv2 = \&#xA;    new_conv_layer(input=layer_conv1,&#xA;                   num_input_channels=num_filters1,&#xA;                   filter_size=filter_size2,&#xA;                   num_filters=num_filters2,&#xA;                   use_pooling=True)&#xA; &#xA;layer_fc1 = new_fc_layer(input=layer_flat,&#xA;                         num_inputs=num_features,&#xA;                         num_outputs=fc_size,&#xA;                         use_relu=True)&#xA; &#xA;layer_fc2 = new_fc_layer(input=layer_fc1,&#xA;                         num_inputs=fc_size,&#xA;                         num_outputs=num_classes,&#xA;                         use_relu=False)&#xA; &#xA;y_pred = tf.nn.softmax(layer_fc2)&#xA; &#xA;y_pred_cls = tf.argmax(y_pred, dimension=1)&#xA; &#xA;cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,&#xA;                                                        labels=y_true)&#xA; &#xA;cost = tf.reduce_mean(cross_entropy)&#xA; &#xA;optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)&#xA; &#xA;correct_prediction = tf.equal(y_pred_cls, y_true_cls)&#xA; &#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA; &#xA;session = tf.Session()&#xA; &#xA;session.run(tf.global_variables_initializer())&#xA; &#xA;train_batch_size = 64&#xA; &#xA;# Counter for total number of iterations performed so far.&#xA;total_iterations = 0&#xA; &#xA;def optimize(num_iterations):&#xA;    # Ensure we update the global variable rather than a local copy.&#xA;    global total_iterations&#xA; &#xA;    # Start-time used for printing time-usage below.&#xA;    start_time = time.time()&#xA; &#xA;    for i in range(total_iterations,&#xA;                   total_iterations + num_iterations):&#xA; &#xA;        # Get a batch of training examples.&#xA;        # x_batch now holds a batch of images and&#xA;        # y_true_batch are the true labels for those images.&#xA;        x_batch, y_true_batch = data.train.next_batch(train_batch_size)&#xA; &#xA;        # Put the batch into a dict with the proper names&#xA;        # for placeholder variables in the TensorFlow graph.&#xA;        feed_dict_train = {x: x_batch,&#xA;                           y_true: y_true_batch}&#xA; &#xA;        # Run the optimizer using this batch of training data.&#xA;        # TensorFlow assigns the variables in feed_dict_train&#xA;        # to the placeholder variables and then runs the optimizer.&#xA;        session.run(optimizer, feed_dict=feed_dict_train)&#xA; &#xA;        # Print status every 100 iterations.&#xA;        if i % 100 == 0:&#xA;            # Calculate the accuracy on the training-set.&#xA;            acc = session.run(accuracy, feed_dict=feed_dict_train)&#xA; &#xA;            # Message for printing.&#xA;            msg = &amp;quot;Optimization Iteration: {0:&amp;gt;6}, Training Accuracy: {1:&amp;gt;6.1%}&amp;quot;&#xA; &#xA;            # Print it.&#xA;            print(msg.format(i + 1, acc))&#xA; &#xA;    # Update the total number of iterations performed.&#xA;    total_iterations += num_iterations&#xA; &#xA;    # Ending time.&#xA;    end_time = time.time()&#xA; &#xA;    # Difference between start and end-times.&#xA;    time_dif = end_time - start_time&#xA; &#xA;    # Print the time-usage.&#xA;    print(&amp;quot;Time usage: &amp;quot; + str(timedelta(seconds=int(round(time_dif)))))&#xA; &#xA;optimize(num_iterations=900)&#xA;&#xA;&#xA;&#xA;print_test_accuracy(show_example_errors=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In this case, how to extract the features? Also, I want to know that are the extracted features different filters that we used or the final updated weights that we get after completion of CNN?&lt;/p&gt;&#xA;" OwnerUserId="10810" LastEditorUserId="29169" LastEditDate="2021-02-16T02:42:19.220" LastActivityDate="2021-02-16T02:42:19.220" Title="Feature Extraction from Convolutional neural network (CNN) and using this feature to other classification algorithm" Tags="|neural-network|tensorflow|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18945" PostTypeId="2" ParentId="18944" CreationDate="2017-05-13T17:14:57.073" Score="1" Body="&lt;p&gt;Before trying to extract features, you need to define your network. Suppose your network has an architecture like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Conv1 layer&#xA;Conv2 layer&#xA;Conv3 layer&#xA;Dense1 layer&#xA;Dense2 layer &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can extract features for each input for any layer (say for Conv2) in the following way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conv2_tensor = sess.graph.get_tensor_by_name('Conv2')&#xA;_, conv_val = sess.run([conv2_tensor],&#xA;                                  {'x': image_data})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-05-13T17:14:57.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18947" PostTypeId="2" ParentId="18942" CreationDate="2017-05-13T21:05:41.397" Score="1" Body="&lt;p&gt;&lt;strong&gt;EDIT, in light of your comment and closer reading of your code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, so it looks like in your code you used the RDD api to count the&#xA; total number of rows in the file (treating it as just a plain &lt;code&gt;RDD[String]&lt;/code&gt;), whereas for counting the number of rows where &lt;code&gt;$&quot;y&quot; === &quot;no&quot;&lt;/code&gt; you used the Spark-SQL API to read in the data as a &lt;code&gt;DataFrame&lt;/code&gt;. I'd advise sticking to one API if you're just starting out, otherwise things will likely feel inconsistent and confusing. Since your problem lends itself nicely to operations that &lt;code&gt;DataFrame&lt;/code&gt; is designed to perform, I'd recommend you stick to Spark-SQL's API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to simply calculate the ratio (as a &lt;code&gt;Double&lt;/code&gt;) of rows where &lt;code&gt;$&quot;y&quot; === &quot;no&quot;&lt;/code&gt;, then you could do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;val ratio = bankDF.filter($&quot;y&quot; === &quot;no&quot;).count().toDouble / bankDF.count()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still strongly recommend you read (and run for yourself) the examples in the Spark-SQL documentation. That'll give you a sense of the sorts of operations you can perform on these dataframes (e.g. means, sums, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Original Answer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all you want is the proportion of rows where &lt;code&gt;y == &quot;no&quot;&lt;/code&gt;, then simply dividing the two &lt;code&gt;Long&lt;/code&gt;'s you've calculated is a really straightforward and natural way to get what you want. However, I'd highly recommend you take a look at the Spark-SQL API instead of the lower-level RDD API that you used here. For example, using Spark-SQL you can read a CSV file directly with &lt;code&gt;spark.read.csv()&lt;/code&gt;, and then treat your dataset like a database table, on which you can do &lt;code&gt;groupBy()&lt;/code&gt;, &lt;code&gt;filter()&lt;/code&gt;, and aggregate functions like &lt;code&gt;mean()&lt;/code&gt;, &lt;code&gt;sum()&lt;/code&gt;, etc. The &lt;a href=&quot;http://spark.apache.org/docs/latest/sql-programming-guide.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; is quite good, so give it a look.&#xA;Spark takes some getting used to, but its really powerful once you get the hang of it. Keep at it, and good luck!&lt;/p&gt;&#xA;" OwnerUserId="23581" LastEditorUserId="23581" LastEditDate="2017-05-18T23:41:38.060" LastActivityDate="2017-05-18T23:41:38.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18948" PostTypeId="2" ParentId="18940" CreationDate="2017-05-13T21:37:13.143" Score="2" Body="&lt;p&gt;I think it might be a relatively trivial bug in your cost function for softmax: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;J = -(sum(sum((Y).*log(h))) + lambda*p/(2*m)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;J = -sum(sum((Y).*log(h)))/m + lambda*p/(2*m) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I.e. for softmax only, you have effectively subtracted the regularisation term from the cost function instead of adding it. Also, you forgot to divide the error term by the number of examples in the batch (and you are taking this average when calculating the gradients)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your back propagation calculations look correct to me if you correct this miscalculation for &lt;code&gt;J&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-13T21:45:02.347" LastActivityDate="2017-05-13T21:45:02.347" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18949" PostTypeId="2" ParentId="5178" CreationDate="2017-05-13T23:07:41.153" Score="9" Body="&lt;p&gt;This is a pretty common problem. I had this pain when I did research projects for a university and now - in industrial data science projects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created and recently released an open source tool to solve this problem - &lt;a href=&quot;https://dvc.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;DVC&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It basically combines your code in Git and data in your local disk or clouds (S3 and GCP storage). DVC tracks dependency between data and code and builds the dependency graph (DAG). It helps you to make your project reproducible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DVC project could be easily shared - sync your data to a cloud (dvc sync command), share your Git repository and provide access to your data bucket in the cloud.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;learnable in a few hours&quot; - is a good point. You should not have any issues with DVC if you are familiar with Git. You really need to learn only three commands:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;dvc init&lt;/code&gt; - like &lt;code&gt;git init&lt;/code&gt;. Should be done in an existing Git repository.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;dvc import&lt;/code&gt; - import your data files (sources). Local file or URL.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;dvc run&lt;/code&gt; - steps of your workflow like &lt;code&gt;dvc run python mycode.py data/input.jpg data/output.csv&lt;/code&gt;. DVC derives the dependency between your steps automatically, builds DAG and keeps it in Git.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;dvc repro&lt;/code&gt; - reproduce your data file. Example: &lt;code&gt;vi mycode.py&lt;/code&gt; - change code, and then &lt;code&gt;dvc repro data/output.csv&lt;/code&gt; will reproduce the file (and all the dependencies.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You need to learn a couple more DVC commands to share data through the cloud and basic S3 or GCP skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dvc.org/doc/tutorials&quot; rel=&quot;nofollow noreferrer&quot;&gt;DVC tutorials&lt;/a&gt; is the best starting point.&lt;/p&gt;&#xA;" OwnerUserId="32215" LastEditorUserId="10683" LastEditDate="2020-02-05T03:22:42.210" LastActivityDate="2020-02-05T03:22:42.210" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="18950" PostTypeId="2" ParentId="758" CreationDate="2017-05-13T23:59:37.350" Score="6" Body="&lt;p&gt;I've created and recently released an open source tool &lt;a href=&quot;http://dvc.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dvc.org&lt;/a&gt; or DVC that does exactly what you are trying to reach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;[Tools for data version control.] DVC works on top of Git, adds data file version control (files are stored outside of Git) and tracks the dependencies between the code and the data files. DVC automatically derives the dependency graph (DAG) for code and data.&lt;/li&gt;&#xA;&lt;li&gt;[Tools enabling to reproduce stages and experiments.] &lt;code&gt;dvc repro data/scores.csv&lt;/code&gt; reproduces all the required steps regarding DAG.&lt;/li&gt;&#xA;&lt;li&gt;[Protocol and suggested directory structure for such a project.] DVC required a data directory (&lt;code&gt;data&lt;/code&gt; by default) where you supposed to store all data files. However, DVC transparently moves the actual content to &lt;code&gt;.cache&lt;/code&gt; directory and creates the symlinks (yeah, I made it to work on Windows as well). The &lt;code&gt;.cache&lt;/code&gt; directory is not synced to Git but it could be synced through the cloud (S3 or GCP) by command &lt;code&gt;dvc sync data/scores.csv&lt;/code&gt; (it syncs corresponded data file from cache like &lt;code&gt;.cache/scores.csv_29de545&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;[Automated build/run tools.] See from the above.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;DVC tutorial is a good starting point - &lt;a href=&quot;https://blog.dataversioncontrol.com/data-version-control-tutorial-9146715eda46&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Data Version Control: iterative machine learning&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32215" LastEditorUserId="32215" LastEditDate="2018-10-16T00:05:54.290" LastActivityDate="2018-10-16T00:05:54.290" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18951" PostTypeId="2" ParentId="14998" CreationDate="2017-05-14T00:16:35.687" Score="1" Body="&lt;p&gt;What I found - sharing notebooks for data scientists is a not a desirable format for communication. Many of them prefer IDE like Spider/RStudio or just a text editors (I know a few data scientists who use &lt;code&gt;vi&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might just share code by your source control and data by cloud storages. It will increase flexibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've recently open sourced a tool which combines code, data, and the dependencies between data and code to a single environment and makes your data science project reproducible: DVC or dataversioncontrol.com (there is a tutorial).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With DVC tool you can just share your project by Git, sync data to S3 by a single DVC command. If some of your data scientists decide to change the code at any stage of your project then the final result could be easily reproduced by a single command &lt;code&gt;dvc repro data/target_metrics.txt&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32215" LastActivityDate="2017-05-14T00:16:35.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18952" PostTypeId="1" AcceptedAnswerId="18962" CreationDate="2017-05-14T05:19:15.343" Score="0" ViewCount="116" Body="&lt;p&gt;I'm trying to experiment if an opportunity will win or lose in Azure Machine Learning studio. However, am still in Data preparation method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my Data base I have opportunity table and products table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one opportunity has multiple products. Should I deal with the many products and put them in one record?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will it affect the prediction if we have duplicate records for an opportunity like (a) or it’s better to have one record per opportunity in order to feed it to ML studio. And if yes which one will be better approach (b) or (c).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Approach a&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CxANt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CxANt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Approach b&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YXNsl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YXNsl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Approach c&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;oppid |first product |first technology|2nd product |2nd technology&#xA;1      out-services   active directory   TRN-items Adobe Acrobat &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32220" LastEditorUserId="32220" LastEditDate="2017-05-16T04:58:47.997" LastActivityDate="2017-05-16T04:58:47.997" Title="Best way for data preparation to have accurate prediction" Tags="|machine-learning|azure-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18956" PostTypeId="1" AcceptedAnswerId="18958" CreationDate="2017-05-14T14:47:40.710" Score="4" ViewCount="17844" Body="&lt;p&gt;I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the test dataset however, there are less cabins, so I'll end up with fewer attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did something like this to make them equal (create columns that are in the training set and delete those that aren't):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for column in X.columns:&#xA;    if column not in X_test.columns:&#xA;        X_test[column] = 0&#xA;&#xA;for column in X_test.columns:&#xA;    if column not in X.columns:&#xA;        X_test.drop([column], axis=1, inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I know it is not a good thing. So how else should I approach it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried removing the cabin column altogether but my model performs better on test data with that column. &lt;/p&gt;&#xA;" OwnerUserId="32164" LastActivityDate="2022-03-30T21:08:10.933" Title="Different number of features in train vs test" Tags="|python|pandas|categorical-data|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18958" PostTypeId="2" ParentId="18956" CreationDate="2017-05-14T15:07:43.307" Score="5" Body="&lt;p&gt;You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_objs_num = len(train)&#xA;dataset = pd.concat(objs=[train, test], axis=0)&#xA;dataset = pd.get_dummies(dataset)&#xA;train = copy.copy(dataset[:train_objs_num])&#xA;test = copy.copy(dataset[train_objs_num:])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-05-14T15:07:43.307" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18962" PostTypeId="2" ParentId="18952" CreationDate="2017-05-14T15:41:07.043" Score="0" Body="&lt;p&gt;The simplification of the data may make the model more stable, but it will also remove its ability to use more specific input criteria. For example, the way you move from Approach A to Approach B you are aggregating specific products into product categories. This means that if your model is successful in Approach A, it will be able to predict based on specific products. On the other hand, if your model training succeeds in Approach B it will only be able to predict on product category (and you will have to convert products into its categories before supplying it to the model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your questions, the number of data samples you have determine how much you have to aggregate and simplify your data. The data itself in its most detailed form could also fail to train the model properly, in which case the approach taken in Approach B is the best next step.&lt;/p&gt;&#xA;" OwnerUserId="31245" LastActivityDate="2017-05-14T15:41:07.043" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18963" PostTypeId="1" CreationDate="2017-05-14T15:57:34.433" Score="2" ViewCount="351" Body="&lt;p&gt;This is a sort of a follow-up to &lt;a href=&quot;https://datascience.stackexchange.com/questions/18934/learning-comparisons&quot;&gt;this newbie question:&lt;/a&gt;Suppose I want learn &lt;em&gt;ranking&lt;/em&gt; (so, I have a bunch of data points, ranked $1, 2, 3, ...$ Now, one way is to use something like logistic regression with training data pairs of data points, and labels like $(a, b) \to \mathbf{TRUE}.$ The other way is to have single datapoints and labels the actual rank value (like $a\to 1, b\to 17, \dots.$) Both methods have obvious drawbacks: the first that there is a quadratic amount of somewhat redundant data, the second that the rank is somehow not really a real number, and the actual differences of the scoring function between adjacent ranks could be wildly variable. Is there any prevailing wisdom on this?&lt;/p&gt;&#xA;" OwnerUserId="32196" LastActivityDate="2017-05-14T15:57:34.433" Title="Learning ranking" Tags="|logistic-regression|ranking|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="18968" PostTypeId="1" CreationDate="2017-05-15T08:54:41.607" Score="12" ViewCount="13924" Body="&lt;p&gt;Let's assume that we are using a batch size of 100 samples for learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in every batch, the weight of every neuron (and bias, etc) is being updated by adding the minus of the learning rate * the average error value that we found using the 100 samples * the derivative of the error function with the respect to the current neuron weight that is being updated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, when we use a Max Pool layer, how can we compute the derivative over this layer? In every sample that we feed forward, a different pixel (let's say) is chosen as the max, so when we backpropagate over 100 samples in which every time a different path was chosen, how can we do it? A solution I have in mind is to remember every pixel that was chosen as the maximum, and then maybe split the derivative over all the max-pixels. Is this what's being done?&lt;/p&gt;&#xA;" OwnerUserId="32042" LastActivityDate="2021-06-28T03:24:52.803" Title="How does backpropagation works through Max Pooling layer when doing a batch?" Tags="|neural-network|deep-learning|backpropagation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18970" PostTypeId="1" CreationDate="2017-05-15T10:34:02.930" Score="1" ViewCount="1595" Body="&lt;p&gt;&lt;em&gt;Note: I believe this question is not off-topic because it meets all of the &lt;a href=&quot;https://datascience.stackexchange.com/help/dont-ask&quot;&gt;criteria for subjective questions that&lt;/em&gt; are &lt;em&gt;allowed&lt;/a&gt;. I would be happy to rephrase or clarify if others disagree&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm about to begin a two-year project which will predominantly involve longitudinal panel data.&#xA;I've found numerous questions and answers (&lt;a href=&quot;https://datascience.stackexchange.com/questions/14972/pros-and-cons-of-python-and-r-for-data-science&quot;&gt;Pros and Cons of Python and R for Data Science&lt;/a&gt;), and blog posts (&lt;a href=&quot;https://www.quora.com/Which-is-better-for-data-analysis-R-or-Python&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/Which-is-better-for-data-analysis-R-or-Python&lt;/a&gt; | &lt;a href=&quot;http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html&lt;/a&gt;), about the relative merits of R and Python+pandas for data science but nothing that discusses longitudinal data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is therefore:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which environment (Python+pandas or R) would you use for longitudinal data analysis, and &lt;em&gt;why&lt;/em&gt;?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I would love to see answers that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tell me how you have used one or both environments to solve a particular problem with longitudinal data&lt;/li&gt;&#xA;&lt;li&gt;Which environment or package(s) you found easier to work with, and why&lt;/li&gt;&#xA;&lt;li&gt;If you used a notebook such as &lt;a href=&quot;http://beakernotebook.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;beaker&lt;/a&gt; to work with multiple environments simultaneously.&lt;/li&gt;&#xA;&lt;li&gt;If so, which environment did you use for which step(s) in the data analysis pipeline, and why?&lt;/li&gt;&#xA;&lt;li&gt;If so, did this confer advantages over just using one language: would you use such an approach again?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am not asking which one is best (we all know that such questions are never constructive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I &lt;em&gt;am&lt;/em&gt; more familiar with one environment than the other, but I'm not averse to learning new skills (and both share similar syntax anyway), so I'm not going to say which one: I want answers based on &lt;em&gt;your&lt;/em&gt; experience, not my abilities.&lt;/p&gt;&#xA;" OwnerUserId="20524" LastActivityDate="2017-05-16T09:54:21.940" Title="Pros and cons of pandas or R for longitudinal data?" Tags="|python|r|time-series|pandas|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18971" PostTypeId="1" CreationDate="2017-05-15T10:50:23.733" Score="3" ViewCount="473" Body="&lt;p&gt;I want to understanding the meaning of the difference of two information entropy values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, we may have the sets:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_{A} = \{x_1,x_2,\cdots,x_{n_A}\}$, and&#xA;$X_{B} = \{x_1,x_2,\cdots,x_{n_B}\}$ that represent the number of hours the users of $A$ and $B$ spent on the websites $A$ and $B$, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we can calculate the &lt;em&gt;Cumulative distribution function&lt;/em&gt;&#xA; (CDF) probability values, described &lt;a href=&quot;https://en.wikipedia.org/wiki/Cumulative_distribution_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, as follows:&#xA;For a real-valued random variable X, the CDF is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F_A(x) = P(X\leq x),\ \forall x \in X_A$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, we find $F_B(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we calculate the entropy values $E_A$ and $E_B$ for the probabilities  $F_A(x)$ and $F_B(x)$ obtained above (as described &lt;a href=&quot;https://en.wikipedia.org/wiki/Entropy_(information_theory)&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$E_A = - \sum\limits_{i=1}^{n_A}P(x_i)log(P(x_i))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, we find $E_B$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is, what do $E_A$ and $E_B$ mean in this scenario? and what their difference also means?&lt;/p&gt;&#xA;" OwnerUserId="4968" LastEditorUserId="31513" LastEditDate="2017-05-15T17:25:18.340" LastActivityDate="2021-07-03T20:24:29.867" Title="The meaning of the difference of two entropy values" Tags="|information-theory|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18972" PostTypeId="1" CreationDate="2017-05-15T11:38:34.333" Score="-1" ViewCount="93" Body="&lt;p&gt;Trying to solve a classification problem using a large number of features, some are individual numbers from a signal while others are aggregates of values to create a feature (median, mean etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I take multiple aggregations(m values) of the same signals (n values) to create a relatively large set of ~100 features (m*n values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it okay to have multiple aggregations of the same data as features? If not, which one should I pick, and how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data has numerical and categorical features and I use one-hot-encoding for the categorical ones before feeding them to scikit-learn's RandomForestClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="32259" LastActivityDate="2017-05-15T11:55:52.157" Title="Suitable aggregations (mean, median or something else) to make features?" Tags="|classification|feature-engineering|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18973" PostTypeId="2" ParentId="18970" CreationDate="2017-05-15T11:47:07.250" Score="2" Body="&lt;p&gt;Let me answer this from the point of data in general. I understand you want the answer for longitudinal data but the reason I don't want to specifically answer that is because pretty much all these frameworks treat data in similar ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What you should be looking for?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. How proficient are you in coding?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you code well enough and you can tackle most problems by coding the I suggest Python + Pandas. The reason is simple since Python is a programming language you can use the language outside just the normal ecosystem of running your task. So if you want to integrate other components such as maybe a web server, store in a database etc. Python will be most useful. &#xA;            where as if you are not comfortable with coding then I suggest R is the way to go since it is much more easy to learn but the disadvantage comes when you are trying to do things other than just perform your daily data analysis. It might be limiting&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. In terms of efficiency&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now pretty much all these tools work pretty fast and the differences are hardly noticeable. Usually, you would have a better edge with R in just the way computations are performed. It is a bit more optimized and it utilizes CPU cores to perform a tad bit faster computation than python does. However these days with the heavy intensive RAM etc it is not really that big of a difference. Also, most libraries for heavy matrix calculations are present in both these toolkits. So not really much difference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In my opinion&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would go for Python just because of the diverse uses it has. Also has a great supporting environment for libraries etc. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-05-15T11:47:07.250" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="18974" PostTypeId="2" ParentId="18972" CreationDate="2017-05-15T11:55:52.157" Score="0" Body="&lt;p&gt;It depends on the type of model you use whether highly correlated features are problematic or not. In general they of course add less information than  low correlated features but as long as you have enough rows (10:1 rows vs columns as a very loose guideline) and a slightly more complex model to train, it generally helps your cause in my experience. As always you can test performance using a validation set.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-15T11:55:52.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18976" PostTypeId="1" CreationDate="2017-05-15T12:23:21.620" Score="-1" ViewCount="49" Body="&lt;p&gt;I'm currently trying to apply clustering algorithm to data on callcenter employee KPIs. My dataset contains daily observations per employee on 3 KPIs. Yet the challenge is that I have a random number of observations per employee and at random intervals, as not every employee works every day. I would like to use the same approach as described &lt;a href=&quot;http://ftp.cse.buffalo.edu/users/azhang/disc/disc01/cd1/out/papers/kdd/p140-cadez.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cadez, Gaffney &amp;amp; Smith (2000)&lt;/a&gt; and I am trying to find a package in R that could help me do that. Popular EM cluster packages such as Mclust do not allow to specify which observation belongs to which individual (as far as I know). Does anyone know of an R implementation for such an algorithm? I've added a small example of my data below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;ID  Date   KPI1  KPI2  KPI3&#xA;1   1/1    1.2    1.3  2.0&#xA;2   2/1    1.4    1    2.2&#xA;1   2/1    1.2    1    2.5&#xA;2   2/1    1.4    1.2  2.4&#xA;1   3/1    1.2    1    2.3&#xA;3   3/1    1.5    1.3  2.2&#xA; &lt;/pre&gt;&#xA;" OwnerUserId="32262" LastEditorUserId="381" LastEditDate="2017-05-16T18:40:29.320" LastActivityDate="2017-05-16T18:40:29.320" Title="Clustering individuals with random observations" Tags="|machine-learning|r|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18978" PostTypeId="2" ParentId="18976" CreationDate="2017-05-15T13:09:25.203" Score="0" Body="&lt;p&gt;How random is the distribution? You could &quot;smooth&quot; the problem away by taking daily, weekly, or monthly averages of those KPIs per employee and clustering on &lt;em&gt;that&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-15T13:09:25.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18979" PostTypeId="2" ParentId="18968" CreationDate="2017-05-15T13:51:06.713" Score="11" Body="&lt;p&gt;When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During back propagation, those activation values are used as one of the numerical sources to calculate gradients, along with gradients calculated so far working backwards and the connecting weights. Like forward propagation, back propagation is applied per example, it does not work with averaged or summed values. Only when all examples have been processed do you work with the summed or averaged gradients for the batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This applies equally to max pool layers. Not only do you know what the output from the pooling layer for each example in the batch was, but you can look at the preceding layer and determine &lt;em&gt;which&lt;/em&gt; input to the pool was the maximum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically, and avoiding the need to define indices for NN layers and neurons, the rule can be expressed like this&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The forward function is $m = max(a,b)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We know $\frac{\partial J}{\partial m}$ for some target function J (in the neural network that will be the loss function we want to minimise, and we are assuming we have backpropagated to this point already)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We want to know $\frac{\partial J}{\partial a}$ and $\frac{\partial J}{\partial b}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $a &amp;gt; b$&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Locally&lt;/em&gt;,* $m = a$. So $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Locally&lt;/em&gt;,* $m$ does not depend on $b$. So $\frac{\partial J}{\partial b} = 0$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$ if $a &amp;gt; b$, else $\frac{\partial J}{\partial a} = 0$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;and $\frac{\partial J}{\partial b} = \frac{\partial J}{\partial m}$ if $b &amp;gt; a$, else $\frac{\partial J}{\partial b} = 0$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When back propagation goes across a max pooling layer, the gradient is processed &lt;em&gt;per example&lt;/em&gt; and assigned only to the input from the previous layer that was the maximum. Other inputs get zero gradient. When this is batched it is no different, it is just processed per example, maybe in parallel. Across a whole batch this can mean that more than one, maybe all, of the input activations to the max pool get some share of the gradient - each from a different subset of examples in the batch.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Locally -&gt; when making only infinitesimal changes to $m$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;** Technically, if $a=b$ exactly then we have a discontinuity, but in practice we can ignore that without issues when training a neural network.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-17T13:50:07.260" LastActivityDate="2017-05-17T13:50:07.260" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="18982" PostTypeId="1" CreationDate="2017-05-15T19:12:31.113" Score="2" ViewCount="2397" Body="&lt;p&gt;When I was learning about classification models, it came to my mind that if there is any difference between &quot;categories&quot; and &quot;classes&quot; on the basis of machine learning terminology? If there is no difference then why do we say classification models, why not categorization models? &lt;/p&gt;&#xA;" OwnerUserId="31610" LastEditorUserId="31513" LastEditDate="2017-05-15T19:26:23.743" LastActivityDate="2018-05-04T10:13:28.420" Title="Is there any difference between 'classification' and 'categorization' based on machine learning terminology?" Tags="|classification|terminology|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="18983" PostTypeId="1" AcceptedAnswerId="19017" CreationDate="2017-05-15T19:52:03.967" Score="0" ViewCount="612" Body="&lt;p&gt;I have trained RNN's before &quot;by hand&quot; using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. &lt;a href=&quot;https://github.com/aidan-plenert-macdonald/lstm-practice/blob/master/lstm.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Full Code&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using &quot;Back-Propagation Through Time&quot;, which I can't tell if it is trying to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)&#xA;cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)&#xA;x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))&#xA;outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)&#xA;&#xA;W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))&#xA;b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))&#xA;y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b&#xA;&#xA;loss = tf.reduce_mean(tf.square(y - y_))&#xA;opt = tf.train.MomentumOptimizer(1e-3, 0.9)&#xA;grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)&#xA;        for g, v in opt.compute_gradients(loss)]&#xA;train = opt.apply_gradients(grad)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: Loss plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XjLNK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XjLNK.png&quot; alt=&quot;Loss vs. Iteration&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Loss:  11.9126&#xA;Actual:  bcdefghijklabcdefghijklabcdefghijklabcde&#xA;Pred:    ffffffffffffffffffffffffffffffffffffffff&#xA;&#xA;Loss:  11.9092&#xA;Actual:  bcdefghijklabcdefghijklabcdefghijklabcde&#xA;Pred:    ffffffffffffffffffffffffffffffffffffffff&#xA;&#xA;Loss:  11.918&#xA;Actual:  bcdefghijklabcdefghijklabcdefghijklabcde&#xA;Pred:    ffffffffffffffffffffffffffffffffffffffff&#xA;&#xA;Loss:  11.9274&#xA;Actual:  bcdefghijklabcdefghijklabcdefghijklabcde&#xA;Pred:    ffffffffffffffffffffffffffffffffffffffff&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; s = 'bcdefghijklabcdefghijklabcdefghijklabcde'&#xA;&amp;gt;&amp;gt;&amp;gt; n = map(ord, s)&#xA;&amp;gt;&amp;gt;&amp;gt; chr(sum(n)/len(n))&#xA;'f'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user32280" LastEditorDisplayName="user32280" LastEditDate="2017-05-15T21:54:53.780" LastActivityDate="2017-05-16T21:25:31.170" Title="Training LSTM Recurrent Network in TensorFlow" Tags="|tensorflow|rnn|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18984" PostTypeId="1" CreationDate="2017-05-15T21:50:24.600" Score="1" ViewCount="95" Body="&lt;p&gt;I am using one text dataset for regression task. The baseline regression method (name concealed) that I employ will work well if all the input features are continuous.  However, the text dataset that I am considering has 200 bag-of-word features that are binary (0, or 1), i.e the first feature will be 1 if the associated word appearing on the text and 0 otherwise. I am just curious how can I convert those 200 binary to one or several continuous features such that my regressor can work with.&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="32284" LastActivityDate="2017-05-15T22:22:11.587" Title="How to convert 200 bag of word features to several continuous features" Tags="|regression|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18985" PostTypeId="2" ParentId="18984" CreationDate="2017-05-15T22:22:11.587" Score="2" Body="&lt;p&gt;Latent dirichlet allocation (LDA) would be a good one to try, as it is tailored for bag-of-words representations. LDA is a &lt;em&gt;topic model&lt;/em&gt;, which basically means that it assigns $n$ topics, and figures out the words that associate with each topic. The output is a vector of $n$ continuous values where each value indicates membership to the corresponding topic. You should be able to find an implementation of LDA for whatever language you're using easily (but don't confuse it with linear discriminant analysis, also commonly referred to as LDA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also try principal components analysis or singular value decompositions, which are other standard dimensionality reduction techniques.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-05-15T22:22:11.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18986" PostTypeId="1" CreationDate="2017-05-15T22:45:20.663" Score="1" ViewCount="1202" Body="&lt;p&gt;I have a time series data (counts of the number of infectious individuals per day), and I am using a SimpleRNN from Keras to classify each of these epidemics into one of eight possible classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used cross-validation to select the best number of hidden units, etc. My problem is that when I run the chosen model multiple times (that is when I train the model multiple times with the same architecture), I get very different results. I'm guessing this is a problem with many local minima, but I'm not sure - it could just be a problem with training. How can I diagnose this? I've done the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Removed the patience term - I thought I might not be training long enough, and the wildly different classification errors were due to that.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Changed my activation functions to ReLUs instead of using the default tanh units - maybe there's some saturation going on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Changed my &lt;code&gt;load_data&lt;/code&gt; function to scale the input data (which are integers).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can anyone think of anything else that I could do to try to figure out why the same code is giving me different classification errors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: More information - I'm using an RNN for an 8-class classification task. There are 200 observations per class in the test set. I obtain confusion matrices for each of the 10 runs shown in the code below. Here are two of the confusion matrices:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;189,0,0,0,0,11,0,0&#xA;0,197,0,0,0,3,0,0&#xA;0,0,199,0,0,1,0,0&#xA;1,0,0,195,0,0,0,4&#xA;0,0,0,0,200,0,0,0&#xA;26,0,0,0,0,174,0,0&#xA;0,0,0,0,0,0,200,0&#xA;0,0,0,0,0,0,5,195&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That one's pretty good. It has an average classification error of 0.032. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(EDIT 2) Here is a plot of the training and validation accuracy for the well-performing run:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hz6VF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hz6VF.png&quot; alt=&quot;trainval0&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This one, however, is much worse (average classification error of 0.389):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;192,0,0,0,0,8,0,0&#xA;0,181,0,0,0,19,0,0&#xA;44,3,106,39,3,5,0,0&#xA;0,0,3,168,29,0,0,0&#xA;0,7,0,0,193,0,0,0&#xA;62,2,0,0,0,136,0,0&#xA;32,33,49,35,31,20,0,0&#xA;27,20,48,46,37,20,0,2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(EDIT 2) Here is a plot of the training and validation accuracy for the badly performing run (looks awful):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Di3i8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Di3i8.png&quot; alt=&quot;trainval1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(EDIT 2) New thought: could this be because of bad initial conditions after all? Why else would some runs be good and some be so bad?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code (EDITED to show the last two changes in the bullet points):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    #!/usr/bin/env python&#xA;&#xA;    import numpy as np&#xA;    import pandas, math, sys, keras&#xA;    from keras import optimizers&#xA;    from keras.models import Sequential&#xA;    from keras.layers import Dense, SimpleRNN&#xA;    from keras.regularizers import l2&#xA;    from keras.utils import np_utils&#xA;    from keras.utils.np_utils import to_categorical&#xA;&#xA;  def load_data(train_file, test_file):&#xA;        trainset = np.loadtxt(train_file, delimiter=&quot;,&quot;)&#xA;&#xA;        # split into input (X) and output (Y) variables&#xA;        X = trainset[:, 0:(trainset.shape[1]-2)] &#xA;        Y = (trainset[:,trainset.shape[1]-1]).astype(int) &#xA;&#xA;        scaler = MinMaxScaler(feature_range=(0, 1))&#xA;        X_scaled = scaler.fit_transform(X)&#xA;&#xA;        y_binary = to_categorical(Y)&#xA;&#xA;        testset = np.loadtxt(test_file, delimiter=&quot;,&quot;)&#xA;        X_test = testset[:,0:(testset.shape[1]-2)]&#xA;        X_test_scaled = scaler.fit_transform(X_test)&#xA;&#xA;        Y_test = (testset[:,testset.shape[1]-1]).astype(int)&#xA;&#xA;        X_train = np.reshape(X, (X.shape[0], X.shape[1], 1))&#xA;        X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))&#xA;&#xA;        ytest_binary = to_categorical(Y_test)&#xA;&#xA;        return (X_train, y_binary, X_test, ytest_binary)&#xA;&#xA;    def get_confusion_matrix_one_hot(model_results, truth):&#xA;        '''model_results and truth should be for one-hot format, i.e, have &amp;gt;= 2 columns,&#xA;        where truth is 0/1, and max along each row of model_results is model result&#xA;        '''&#xA;        assert model_results.shape == truth.shape&#xA;        num_outputs = truth.shape[1]&#xA;        confusion_matrix = np.zeros((num_outputs, num_outputs), dtype=np.int32)&#xA;        predictions = np.argmax(model_results,axis=1)&#xA;        assert len(predictions)==truth.shape[0]&#xA;&#xA;        for actual_class in range(num_outputs):&#xA;            idx_examples_this_class = truth[:,actual_class]==1&#xA;            prediction_for_this_class = predictions[idx_examples_this_class]&#xA;            for predicted_class in range(num_outputs):&#xA;                count = np.sum(prediction_for_this_class==predicted_class)&#xA;                confusion_matrix[actual_class, predicted_class] = count&#xA;        assert np.sum(confusion_matrix)==len(truth)&#xA;        assert np.sum(confusion_matrix)==np.sum(truth)&#xA;        return confusion_matrix&#xA;&#xA;    def rnn_repeat(X, Y, Xtest, Ytest, params_to_use, save_file=&quot;rnn_twolayer_acc_loss&quot;, num_reps=10):&#xA;&#xA;        def rnn_model(hid_dim=10, ker_reg=0.01, rec_reg=0.01, optimizer=&quot;sgd&quot;):&#xA;                model = Sequential()&#xA;                model.add(SimpleRNN(units=hid_dim, activation='relu', input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))&#xA;                model.add(Dense(Y.shape[1], activation='softmax'))&#xA;                model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])&#xA;                print 'fitting a model'&#xA;                return model&#xA;&#xA;        optim = params_to_use['optimizer']&#xA;        ker_reg_best = params_to_use['ker_reg']&#xA;        hid_val = params_to_use['hid_dim']&#xA;        rec_reg_best = params_to_use['rec_reg']&#xA;&#xA;        #Need this for all classification errors&#xA;        class_error = []&#xA;&#xA;        #X has 6400 examples, 800 of each class (8 classes). Take 20% of each class and make this the validation set.&#xA;        to_take = np.random.choice(800, int(800*0.2), replace=False)&#xA;        class_split = np.array_split(X, 8) #16 equal slices of 800 elements each&#xA;        val_list = [x[to_take] for x in class_split]&#xA;        big_list = [item for sublist in val_list for item in sublist]&#xA;        val_X = np.asarray(big_list)&#xA;        label_set = np.arange(0, 8) #0 to 15&#xA;        val_Y = np.repeat(label_set, int(800*0.2))&#xA;        val_Y = to_categorical(val_Y)&#xA;&#xA;        setdiffval = set(range(800)) - set(to_take)&#xA;        setdiffval = list(setdiffval)&#xA;&#xA;        X_train_vals = [x[setdiffval] for x in class_split]&#xA;        X_train = [item for sublist in X_train_vals for item in sublist]&#xA;        X_train = np.asarray(X_train)&#xA;        Y_train = np.repeat(label_set, int(800*0.8))&#xA;        Y_train = to_categorical(Y_train)&#xA;&#xA;        for j in range(num_reps):&#xA;            model = rnn_model(hid_dim=hid_val, ker_reg=ker_reg_best, rec_reg=rec_reg_best, optimizer=optim)&#xA;            hist = model.fit(X_train, Y_train, validation_data=(val_X, val_Y), epochs=200)&#xA;            h1 = hist.history&#xA;            acc_ = np.asarray(h1['acc'])&#xA;            loss_ = np.asarray(h1['loss'])&#xA;            val_loss_ = np.asarray(h1['val_loss'])&#xA;            val_acc_ = np.asarray(h1['val_acc'])&#xA;&#xA;            #Save the accuracy and loss&#xA;            acc_and_loss = np.column_stack((acc_, loss_, val_acc_, val_loss_))&#xA;            save_file_rnn = save_file + '_' + str(j) + '.txt'&#xA;            with open(save_file_rnn, 'w') as f:&#xA;                    np.savetxt(save_file_rnn, acc_and_loss, delimiter=&quot; &quot;)&#xA;            print 'saved file', save_file_rnn&#xA;&#xA;            #Run the final, trained model on the test set and return a confusion matrix&#xA;            test_scores = model.evaluate(Xtest, Ytest) #evaluate returns the metric 'accuracy' on the test set&#xA;            print 'eval_scores', test_scores[1]*100&#xA;            predict = model.predict(Xtest) #I think predict returns the class probabilities&#xA;            con_mat = get_confusion_matrix_one_hot(predict, Ytest)&#xA;            print con_mat&#xA;&#xA;            #Save the confusion matrix&#xA;            save_con_mat = 'confusionmatrix_rnn' + '_' + str(j) + '.txt'&#xA;            np.savetxt(save_con_mat, con_mat, fmt='%i', delimiter=&quot;,&quot;)&#xA;&#xA;            #Get the classification error per class, and the average classification error&#xA;            class_error_j = []&#xA;            for i in range(0, con_mat.shape[1]): #for the number of classes&#xA;                class_i_correct = float(con_mat[i][i])/float(sum(con_mat[i]))&#xA;                class_error_j.append(1. - class_i_correct)&#xA;&#xA;            class_error_j.append(sum(class_error_j)/con_mat.shape[1]) #the average classification error&#xA;            class_error.append(class_error_j) #each class's classification error, and the average classification error for this run&#xA;&#xA;        np.savetxt('class_error.txt', class_error, fmt = '%1.3f', delimiter=' ')&#xA;&#xA;        return 0&#xA;&#xA;    if __name__=='__main__':&#xA;&#xA;        X, Y, Xtest, Ytest = load_data('train.csv', 'test.csv')&#xA;        best_params = {'rec_reg': 0.01, 'optimizer': 'adam', 'ker_reg': 0.1, 'hid_dim': 15} &#xA;        rnn_repeat(X, Y, Xtest, Ytest, best_params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="83275" LastEditDate="2019-11-28T14:25:45.130" LastActivityDate="2019-11-28T14:25:45.130" Title="RNN for classification giving vastly different results (Keras)" Tags="|classification|keras|rnn|" AnswerCount="0" CommentCount="15" ContentLicense="CC BY-SA 4.0" />
  <row Id="18989" PostTypeId="1" CreationDate="2017-05-15T23:33:44.637" Score="12" ViewCount="11871" Body="&lt;p&gt;I often find myself writing code like the following (oversimplfied example)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = read_csv('customer_data_export.csv')&#xA;df2 = df.query(&quot;date &amp;gt; '2017-01-10'&quot;)&#xA;data = df_filtered.groupby('transaction_id').sum()&#xA;plot_data = pivot_table(data, columns='weekday', rows='n_items')&#xA;# Etc etc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically the problem is that while it's relatively easy to come up with semantic names for columns (as random variables) it's hard for me to come up with meaningful names for each step of transformed dataframe. Additionally I prefer to have short names to make the code easier to type. (Working in Jupyter notebook, the tab-completion isn't the best).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some best practices that people follow with this kind of thing? &lt;/p&gt;&#xA;" OwnerUserId="10975" LastActivityDate="2022-09-18T11:12:34.210" Title="Naming conventions for dataframes" Tags="|r|pandas|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="18991" PostTypeId="1" AcceptedAnswerId="19198" CreationDate="2017-05-16T01:04:18.037" Score="0" ViewCount="4519" Body="&lt;p&gt;I notice that most of my successfully trained CNNs initiate (i.e. first epoch) loss is in the 1-2 range. Typically, these are image classifiers, either for discrete classes (cat, dog, ship, etc) or semantic (pixelwise) segmentation. Occasionally, either because of poor design or architecture misunderstand on my part, I see an initial loss in the 60s; the other day one started at 200 (and didn't converge).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: is there a heuristic that would allow me to determine a ball park value for my initial loss value? If so, on what aspects of my trained model is it contingent?&lt;/p&gt;&#xA;" OwnerUserId="20374" LastActivityDate="2017-05-25T11:13:10.950" Title="Is there a rule of thumb for the initial value of loss function in a CNN?" Tags="|machine-learning|deep-learning|loss-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="18994" PostTypeId="2" ParentId="18848" CreationDate="2017-05-16T07:10:32.867" Score="0" Body="&lt;p&gt;I do agree with Jan van der Vegt, standardization (e.g, [-1, 1]) or normalization N(0, 1) combined with the activation function can be very important with neural networks. I would check the dissertation of Pichaid Varoonchotikul: “Flood forecasting using artificial neural networks” for the ins and outs of ANNs. It has very interesting caveats. Anyway, I'm use to try first without, but when results are unsatisfactory, I'm use to made trials with either both. &#xA;Not sure it will help but I would check the R package &lt;em&gt;TSclust&lt;/em&gt; and related docs. The authors are very kindly and they will help you to find specific models to do so. They are experts on time series analyses!&#xA;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="32292" LastActivityDate="2017-05-16T07:10:32.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="18996" PostTypeId="1" AcceptedAnswerId="18997" CreationDate="2017-05-16T07:42:27.920" Score="1" ViewCount="2325" Body="&lt;p&gt;I am doing a &lt;code&gt;regression model&lt;/code&gt;. And I was wondering what would be the consequence if we have two or more &lt;code&gt;Highly correlated&lt;/code&gt; columns in the dataset? Is that something that can decrease the accuracy of the model?&lt;/p&gt;&#xA;&lt;p&gt;Answering this question would help decide how to deal with it. Would &lt;code&gt;PCA&lt;/code&gt; be the best option here?&lt;/p&gt;&#xA;" OwnerUserId="29810" LastEditorUserId="85045" LastEditDate="2020-12-31T12:42:29.663" LastActivityDate="2020-12-31T12:42:29.663" Title="What happens when you have highly correlated columns in a dataset?" Tags="|python|regression|linear-regression|correlation|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="18997" PostTypeId="2" ParentId="18996" CreationDate="2017-05-16T08:14:49.677" Score="4" Body="&lt;p&gt;Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given &lt;a href=&quot;https://stats.stackexchange.com/questions/86269/what-is-the-effect-of-having-correlated-predictors-in-a-multiple-regression-mode&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is a nice choice when it comes to dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-05-16T08:14:49.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19000" PostTypeId="2" ParentId="18970" CreationDate="2017-05-16T09:54:21.940" Score="0" Body="&lt;p&gt;Answers coming in from twitter&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;R + packages&lt;/h1&gt;&#xA;&#xA;&lt;a href=&quot;https://twitter.com/philmikejones&quot; rel=&quot;nofollow noreferrer&quot;&gt;@philmikejones&lt;/a&gt; R all the way! (both are good lets be honest, so just follow what you have greater knowledge of)&lt;/p&gt;&amp;mdash; Mark Green (@markalangreen) &lt;a href=&quot;https://twitter.com/markalangreen/status/864072522727387139&quot; rel=&quot;nofollow noreferrer&quot;&gt;May 15, 2017&lt;/a&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;a href=&quot;https://twitter.com/markalangreen&quot; rel=&quot;nofollow noreferrer&quot;&gt;@markalangreen&lt;/a&gt; Have you used a specific package for setting up your longitudinal data, or just base R?&lt;/p&gt;&amp;mdash;  Phil (@philmikejones) &lt;a href=&quot;https://twitter.com/philmikejones/status/864095140201234433&quot; rel=&quot;nofollow noreferrer&quot;&gt;May 15, 2017&lt;/a&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;a href=&quot;https://twitter.com/philmikejones&quot; rel=&quot;nofollow noreferrer&quot;&gt;@philmikejones&lt;/a&gt; Depends on what you are doing but I tend to stick to data.table - I don&amp;#39;t think there is that one &amp;#39;golden&amp;#39; package&lt;/p&gt;&amp;mdash; Mark Green (@markalangreen) &lt;a href=&quot;https://twitter.com/markalangreen/status/864104771904471043&quot; rel=&quot;nofollow noreferrer&quot;&gt;May 15, 2017&lt;/a&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;a href=&quot;https://twitter.com/markalangreen&quot; rel=&quot;nofollow noreferrer&quot;&gt;@markalangreen&lt;/a&gt; &lt;a href=&quot;https://twitter.com/philmikejones&quot; rel=&quot;nofollow noreferrer&quot;&gt;@philmikejones&lt;/a&gt; agree - in my experience people using the forecast package by &lt;a href=&quot;https://twitter.com/robjhyndman&quot; rel=&quot;nofollow noreferrer&quot;&gt;@robjhyndman&lt;/a&gt; usually use ts(), otherwise zoo or xts objects&lt;/p&gt;&amp;mdash; andrea panizza (@unsorsodicorda) &lt;a href=&quot;https://twitter.com/unsorsodicorda/status/864161222614278145&quot; rel=&quot;nofollow noreferrer&quot;&gt;May 15, 2017&lt;/a&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, recommendations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;data.table&lt;/code&gt; (which I'm not a big fan of, but &lt;code&gt;tidyverse&lt;/code&gt; is fine as my data isn't 'big')&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;forecast&lt;/code&gt; package, particularly &lt;code&gt;ts()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;zoo&lt;/code&gt; or &lt;code&gt;xts&lt;/code&gt; objects&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Prophet&lt;/h1&gt;&#xA;&#xA;&lt;a href=&quot;https://twitter.com/philmikejones&quot; rel=&quot;nofollow noreferrer&quot;&gt;@philmikejones&lt;/a&gt; What abt Prophet? Based on &lt;a href=&quot;https://twitter.com/mcmc_stan&quot; rel=&quot;nofollow noreferrer&quot;&gt;@mcmc_stan&lt;/a&gt;, available in R &amp;amp; Python, didn&amp;#39;t use much but got great results when I did: &lt;a href=&quot;https://t.co/Pqat2KevCj&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://t.co/Pqat2KevCj&lt;/a&gt;&lt;/p&gt;&amp;mdash; andrea panizza (@unsorsodicorda) &lt;a href=&quot;https://twitter.com/unsorsodicorda/status/864091058958893056&quot; rel=&quot;nofollow noreferrer&quot;&gt;May 15, 2017&lt;/a&gt;&#xA;&#xA;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://research.fb.com/prophet-forecasting-at-scale/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://research.fb.com/prophet-forecasting-at-scale/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20524" LastActivityDate="2017-05-16T09:54:21.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19001" PostTypeId="1" CreationDate="2017-05-16T09:57:14.963" Score="1" ViewCount="62" Body="&lt;p&gt;I'm a newbie to machine learning. I'm working on a dataset to predict a target variable in terms of the independent variables. In the dataset the independent variables few are very highly correlated . correlation coefficient is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used ridge regression since it will take care of the correlated variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is penalizing all the correlated variables and taking only one among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone throw light on how the variable selection is done for ridge regression in presence of correlated variables.&lt;/p&gt;&#xA;" OwnerUserId="18315" LastActivityDate="2017-05-16T09:57:14.963" Title="Selection of co related variables for ridge regression" Tags="|r|regression|linear-regression|regularization|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19002" PostTypeId="1" CreationDate="2017-05-16T11:23:41.233" Score="2" ViewCount="731" Body="&lt;p&gt;I have a big table (150 Million rows and ~ 70 columns).&#xA;In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM &amp;amp; Decision Forest algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you,&#xA;Lisa&lt;/p&gt;&#xA;" OwnerUserId="32307" LastEditorUserId="32307" LastEditDate="2017-05-16T13:57:49.737" LastActivityDate="2018-01-07T19:41:29.293" Title="Text mining of big data using R Server" Tags="|machine-learning|r|bigdata|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19003" PostTypeId="2" ParentId="19002" CreationDate="2017-05-16T12:16:45.617" Score="0" Body="&lt;p&gt;Do you mean 150 million &lt;strong&gt;rows&lt;/strong&gt; and 70 columns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yeah, it is easy to run out of memory creating the DTMs. This is a similar question to here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/10970/text-mining-on-large-dataset&quot;&gt;Text Mining on Large Dataset&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like they talk about, I would create in splits (batches) and save the chunks out to a file or a DB. When it is time to do classification, see how it works if you pull 10,000 randomly sampled rows. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-16T12:16:45.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19004" PostTypeId="2" ParentId="18989" CreationDate="2017-05-16T12:23:30.307" Score="5" Body="&lt;p&gt;Why not give them a name describing their purpose?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df_csv = read_csv('customer_data_export.csv')&#xA;df_date_filtered = df.query(&quot;date &amp;gt; '2017-01-10'&quot;)&#xA;df_grouped_by_trans_id = df_date_filtered.groupby('transaction_id').sum()&#xA;&#xA;#cleanup&#xA;rm(df_csv, df_date_filtered)&#xA;&#xA;plot_data = pivot_table(df_grouped_by_trans_id, columns='weekday', rows='n_items')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-16T12:23:30.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19005" PostTypeId="1" CreationDate="2017-05-16T13:37:53.937" Score="4" ViewCount="9952" Body="&lt;p&gt;I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;False Positives&lt;/strong&gt; are much worse for me than &lt;strong&gt;False Negatives&lt;/strong&gt;, how can I take this into account?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The API confuses me a bit and I found two arguments that might be related to this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;objective&lt;/code&gt; &amp;amp; &lt;code&gt;eval_metric&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which one is it? And how to use it?&lt;/p&gt;&#xA;" OwnerUserId="32311" LastEditorUserId="32311" LastEditDate="2017-05-16T18:26:34.063" LastActivityDate="2017-05-16T20:48:25.277" Title="XGBoost change loss function" Tags="|xgboost|loss-function|cost-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19006" PostTypeId="2" ParentId="19002" CreationDate="2017-05-16T14:08:06.977" Score="1" Body="&lt;p&gt;Thank you for your reply.&#xA;By accident I found out that MicrosoftML (&lt;strong&gt;&lt;a href=&quot;https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction&lt;/a&gt;&lt;/strong&gt;) offers just what I need - by using the featurizeText functions offered within Microsoft R client and R server.&#xA;Maybe it helps others - I found their example  and could it translate to my data (featurizeText() help under Microsoft ML).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trainReviews &amp;lt;- data.frame(review = c( &#xA;        &quot;This is great&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;Love it&quot;,&#xA;        &quot;Do not like it&quot;,&#xA;        &quot;Really like it&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;I like it a lot&quot;,&#xA;        &quot;I kind of hate it&quot;,&#xA;        &quot;I do like it&quot;,&#xA;        &quot;I really hate it&quot;,&#xA;        &quot;It is very good&quot;,&#xA;        &quot;I hate it a bunch&quot;,&#xA;        &quot;I love it a bunch&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;I like it very much&quot;,&#xA;        &quot;I hate it very much.&quot;,&#xA;        &quot;I really do love it&quot;,&#xA;        &quot;I really do hate it&quot;,&#xA;        &quot;Love it!&quot;,&#xA;        &quot;Hate it!&quot;,&#xA;        &quot;I love it&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;I love it&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;I love it&quot;),&#xA;     like = c(TRUE, FALSE, TRUE, FALSE, TRUE,&#xA;        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,&#xA;        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, &#xA;        FALSE, TRUE, FALSE, TRUE), stringsAsFactors = FALSE&#xA;    )&#xA;&#xA;    testReviews &amp;lt;- data.frame(review = c(&#xA;        &quot;This is great&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;Love it&quot;,&#xA;        &quot;Really like it&quot;,&#xA;        &quot;I hate it&quot;,&#xA;        &quot;I like it a lot&quot;,&#xA;        &quot;I love it&quot;,&#xA;        &quot;I do like it&quot;,&#xA;        &quot;I really hate it&quot;,&#xA;        &quot;I love it&quot;), stringsAsFactors = FALSE)&#xA;&#xA;&#xA;outModel &amp;lt;- rxLogisticRegression(like ~ reviewTran, data = trainReviews,&#xA;    mlTransforms = list(featurizeText(vars = c(reviewTran = &quot;review&quot;),&#xA;    stopwordsRemover = stopwordsDefault(), keepPunctuations = FALSE)))&#xA;# 'hate' and 'love' have non-zero weights summary(outModel)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32307" LastActivityDate="2017-05-16T14:08:06.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19008" PostTypeId="1" CreationDate="2017-05-16T16:02:00.083" Score="1" ViewCount="21" Body="&lt;p&gt;Assuming no particular knowledge, let us consider a system where a mapping $F:\mathbb{R}^n\times \mathbb{R}^k \to \mathbb{R}^{n} \times \{-1,1\}$ describes the transformations of $\mathbb{R}^n$ entities. Additionally, we are provided with a white box model for finding $\mathbf{y}\in \mathbb{R}^n$ such that $F(\mathbf{x},\mathbf{u}) = (\mathbf{y},c)$ for any input $(\mathbf{x},\mathbf{u})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given $M$ training samples, $(\mathbf{x}_m,\mathbf{u}_m,c_m)$ with $\mathbf{x}_m \in \mathbb{R}^n$, $\mathbf{u}_m \in \mathbb{R}^k$, $c_m \in \{\pm 1\}$ such that $F(\mathbf{x}_m,\mathbf{u}_m) = (\mathbf{y}_m,c_m)$ for $m \in \overline{1\ldots M}$, what methods are there for performing the following tasks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;find $c^\ast$ such that $F(\mathbf{x}^\ast,\mathbf{u}^\ast) = (\mathbf{y}^\ast,c^\ast)$ for known $\mathbf{x}^\ast,\mathbf{u}^\ast$&lt;/li&gt;&#xA;&lt;li&gt;find $\mathbf{u}^\ast$ such that $F(\mathbf{x}^\ast, \mathbf{u}^\ast) = (\mathbf{y}^\ast, 1)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the first task, I suppose that SVMs are one possible option, while for the second one regression could be tried. &#xA;Are there other classes of solutions that seem more appropriate for this type of problem?&lt;/p&gt;&#xA;" OwnerUserId="32314" LastEditorUserId="32314" LastEditDate="2017-05-16T16:37:48.210" LastActivityDate="2017-05-16T16:37:48.210" Title="Method(s) suited for finding model parameters in supervised contexts" Tags="|predictive-modeling|regression|svm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19010" PostTypeId="1" AcceptedAnswerId="19038" CreationDate="2017-05-16T17:16:50.230" Score="5" ViewCount="4444" Body="&lt;p&gt;I have created a neural network to classify the MNIST handwritten numbers dataset. It is using &lt;strong&gt;softmax&lt;/strong&gt; as the activation function for the output layer and various other functions for the hidden layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My implementation with the help of &lt;a href=&quot;https://datascience.stackexchange.com/questions/18940/issue-with-backpropagation-using-a-2-layer-network-and-softmax/18948?noredirect=1#comment21871_18948&quot;&gt;this&lt;/a&gt; question seems to be passing the gradient checks for all activation functions but when it comes to the actual run with my training data for an exemplary run of 10 iterations I get an accuracy of about 87% if I use &lt;strong&gt;sigmoid&lt;/strong&gt; or &lt;strong&gt;tanh&lt;/strong&gt; as the activation function for the hidden layer, but if I use &lt;strong&gt;cosine&lt;/strong&gt; it returns an accuracy of 9%. Training the network with more iterations (100, 200, 500) does not have any effect either and in fact my minimization function does not manage to move below 2.18xxx for the cost function no matter how many epochs pass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some pre-processing step that I need to perform before using &lt;strong&gt;cosine&lt;/strong&gt; if not why is it that this activation function works so badly?&lt;/p&gt;&#xA;" OwnerUserId="28600" LastEditorUserId="28600" LastEditDate="2017-05-17T16:18:03.823" LastActivityDate="2017-05-17T18:36:17.577" Title="Using the cosine activation function seems to perform very badly" Tags="|machine-learning|neural-network|matlab|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="19012" PostTypeId="2" ParentId="18991" CreationDate="2017-05-16T19:23:44.410" Score="1" Body="&lt;p&gt;The initial loss is purely dependent on your weight initialization and your data normalization. Random weights = random initial loss. If you're interested in the variance of the loss, this will roughly relate to the number of layers you have and number of weights in each layer. For a good example, check out this paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding the difficulty of training deep feedforward neural networks&lt;/a&gt; -- Glorot, Bengio&lt;/p&gt;&#xA;" OwnerUserId="12241" LastActivityDate="2017-05-16T19:23:44.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19013" PostTypeId="2" ParentId="19005" CreationDate="2017-05-16T19:31:32.730" Score="0" Body="&lt;p&gt;Based on &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page's&lt;/a&gt; description of what &lt;code&gt;objective&lt;/code&gt; and &lt;code&gt;eval_metric&lt;/code&gt; do, I don't think those are exactly right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would try playing with scale_pos_weights in this case. Go to &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/parameter.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt; and Ctrl+F for &quot;scale_pos_weights&quot;.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-16T19:31:32.730" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19014" PostTypeId="2" ParentId="19005" CreationDate="2017-05-16T20:36:45.073" Score="3" Body="&lt;p&gt;In a scenario where &lt;em&gt;consequences&lt;/em&gt; of prediction errors are not equivalent, you are usually still interested in training a model to predict accurately from the data set, and would not change the objective function in supervised learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically when consequences of FP and FN differ, you would:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the &lt;em&gt;confidence&lt;/em&gt; of the prediction given by the model. In XGBoost, that is &lt;code&gt;objective: &quot;binary:logistic&quot;&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve&quot; rel=&quot;nofollow noreferrer&quot;&gt;area under ROC&lt;/a&gt; as a base metric for deciding best tuned model (this is &lt;code&gt;eval_metric: &quot;auc&quot;&lt;/code&gt; for XGBoost). This metric is a measure of how well sorted your classes are - the higher the value, the easier and more effective it is to tune the confidence level later.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decide a weighting for your costs FP and FN - this is entirely up to you, you say that FP is worse than FN, but in order to make an optimal decision you have to turn that into a numerical statement. If you can assign a &lt;em&gt;relative&lt;/em&gt; financial cost to a business for each kind of mistake, then that would be a good start.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the best tuned model, use the weightings to calculate costs at different cutoffs of class confidence before assigning positive class that you wish to act upon in some way. Predict against a test set to get class probabilities. Then use different cutoff points for positive class, count your FP and FN given that cutoff, and multiply the totals by the costs you chose later. The lowest-scoring cutoff point should be the one to use in production. If FP has a higher cost than FN, then you will likely find that a confidence level &gt; 0.5 is required.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also look into machine learning approaches that consider the consequences of actions - e.g. reinforcement learning - in order to achieve a similar result within a single learning framework. However, that is not something you can do solely within XGBoost, and as long as your problem remains balancing between FP and FN in a single step prediction, then the above approach should be OK.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-16T20:48:25.277" LastActivityDate="2017-05-16T20:48:25.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19015" PostTypeId="2" ParentId="17791" CreationDate="2017-05-16T21:10:54.813" Score="5" Body="&lt;p&gt;Yes, there is now a port to R, which is available &lt;a href=&quot;https://github.com/thomasp85/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It purports to provide LIME explanations for any classifier that implements a &lt;code&gt;predict()&lt;/code&gt; method accepting a &lt;code&gt;type = 'prob'&lt;/code&gt; argument. I have not yet tested it myself.&lt;/p&gt;&#xA;" OwnerUserId="7829" LastEditorUserId="98307" LastEditDate="2020-08-12T15:32:37.743" LastActivityDate="2020-08-12T15:32:37.743" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="19016" PostTypeId="2" ParentId="4836" CreationDate="2017-05-16T21:15:36.407" Score="3" Body="&lt;p&gt;If you are looking for something simple to start with and easy to implement, I would recommend this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Beginners Guide to Topic Modeling&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32189" LastActivityDate="2017-05-16T21:15:36.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19017" PostTypeId="2" ParentId="18983" CreationDate="2017-05-16T21:20:07.373" Score="1" Body="&lt;p&gt;I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The moral is that optimizing neural networks is still an art.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/C0P3i.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/C0P3i.png&quot; alt=&quot;Plot every 100 iterations&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. It seems weird to use a regression loss with a classification problem.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-05-16T21:25:31.170" LastActivityDate="2017-05-16T21:25:31.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19019" PostTypeId="1" CreationDate="2017-05-17T04:33:17.987" Score="5" ViewCount="19809" Body="&lt;p&gt;From the comments in &lt;a href=&quot;https://datascience.stackexchange.com/questions/18986/rnn-for-classification-giving-vastly-different-results-keras&quot;&gt;my previous question&lt;/a&gt;, I'm trying to build my own custom weight initializer for an RNN. Based on the code given &lt;a href=&quot;https://groups.google.com/forum/#!topic/keras-users/kc0Jsw1vTqY&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; (careful - the updated version of Keras uses 'initializers' instead of 'initializations' according to &lt;a href=&quot;https://github.com/fchollet/keras/issues/6278&quot; rel=&quot;noreferrer&quot;&gt;fchollet&lt;/a&gt;), I've put together an attempt. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas, math, sys, keras&#xA;from keras import optimizers&#xA;from keras import initializers&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, SimpleRNN&#xA;from keras.regularizers import l2&#xA;import numpy as np&#xA;&#xA;def rnn_model(hid_dim=10, ker_reg=0.01, rec_reg=0.01, optimizer=&quot;sgd&quot;, w_mean = 0., w_var = 0.05):&#xA;        my_init = lambda shape: initializers.TruncatedNormal(mean=w_mean, stddev=w_var)&#xA;        model = Sequential()&#xA;        model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))&#xA;        model.add(Dense(Y.shape[1], activation='softmax'))&#xA;        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])&#xA;        print 'fitting a model'&#xA;        return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I call &lt;code&gt;rnn_model&lt;/code&gt; later, I get an error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    model = rnn_model(hid_dim=hid_val, ker_reg=ker_reg_best, rec_reg=rec_reg_best, optimizer=optim, w_mean=ave_weights, w_var=var_weights)&#xA;  File &quot;rnn.py&quot;, line 187, in rnn_model&#xA;    model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py&quot;, line 430, in add&#xA;    layer(x)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py&quot;, line 257, in __call__&#xA;    return super(Recurrent, self).__call__(inputs, **kwargs)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 551, in __call__&#xA;    self.build(input_shapes[0])&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py&quot;, line 478, in build&#xA;    constraint=self.kernel_constraint)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 384, in add_weight&#xA;    weight = K.variable(initializer(shape), dtype=K.floatx(), name=name)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py&quot;, line 288, in variable&#xA;    v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py&quot;, line 197, in __init__&#xA;    expected_shape=expected_shape)&#xA;  File &quot;/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py&quot;, line 274, in _init_from_args&#xA;    initial_value(), name=&quot;initial_value&quot;, dtype=dtype)&#xA;TypeError: __call__() takes at least 2 arguments (1 given)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to initialize a Keras model using custom parameters for the initializer?&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-05-17T04:47:29.513" LastActivityDate="2023-03-04T16:57:29.490" Title="Custom weight initialization in Keras" Tags="|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19020" PostTypeId="1" AcceptedAnswerId="19023" CreationDate="2017-05-17T04:56:08.497" Score="3" ViewCount="849" Body="&lt;p&gt;I'm still new to machine learning and just came across powerful deep learning library, Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read Keras document and tried few Keras examples on Github &lt;a href=&quot;https://github.com/fchollet/keras/tree/master/examples&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. I've also studied some basic knowledge and concepts of deep learning from several sources but still haven't really had solid understanding in CNN and RNN which look to be very powerful networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to prove my assumption, I downloaded reuters_mlp.py example from Keras Github which originally uses simple MLP networks as a model. I combined the idea of CNN which I got from imdb_cnn.py example to reuters_mlp.py example and then observed the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surprisingly, the result didn't come out like I expected. CNN performed worst than simple MLP networks. Can someone please explain why the accuracy of CNN is lower than the simple MLP networks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the outputs (Tensorflow as backend)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;8982 train sequences, 2246 test sequences, 46 classes, num_words=1000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MLP (sequences_to_matrix, mode=bianry):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch 1/5&#xA;8982/8982 [==============================] - 3s - loss: 1.3236 - acc: 0.6984&lt;br&gt;&#xA;Epoch 2/5&#xA;8982/8982 [==============================] - 2s - loss: 0.7182 - acc: 0.8250&lt;br&gt;&#xA;Epoch 3/5&#xA;8982/8982 [==============================] - 2s - loss: 0.4544 - acc: 0.8864&lt;br&gt;&#xA;Epoch 4/5&#xA;8982/8982 [==============================] - 2s - loss: 0.3197 - acc: 0.9192&lt;br&gt;&#xA;Epoch 5/5&#xA;8982/8982 [==============================] - 2s - loss: 0.2511 - acc: 0.9356&lt;br&gt;&#xA;1920/2246 [========================&gt;.....] - ETA: 0s&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test loss: 1.05213204963 Test accuracy: 0.785396260071&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN (pad_sequences):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch 1/5&#xA;8982/8982 [==============================] - 81s - loss: 1.9794 - acc: 0.5181&lt;br&gt;&#xA;Epoch 2/5&#xA;8982/8982 [==============================] - 78s - loss: 1.4289 - acc: 0.6591&lt;br&gt;&#xA;Epoch 3/5&#xA;8982/8982 [==============================] - 79s - loss: 1.1546 - acc: 0.7175&lt;br&gt;&#xA;Epoch 4/5&#xA;8982/8982 [==============================] - 78s - loss: 0.9639 - acc: 0.7663&lt;br&gt;&#xA;Epoch 5/5&#xA;8982/8982 [==============================] - 77s - loss: 0.8378 - acc: 0.7935&lt;br&gt;&#xA;2240/2246 [============================&gt;.] - ETA: 0s&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test loss: 0.960687935512, Test accuracy: 0.764470169243&lt;/p&gt;&#xA;" OwnerUserId="32343" LastActivityDate="2017-05-17T07:27:23.183" Title="Why CNN doesn't give higher accuracy over simple MLP network? [From Keras examples]" Tags="|machine-learning|deep-learning|nlp|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19023" PostTypeId="2" ParentId="19020" CreationDate="2017-05-17T07:27:23.183" Score="4" Body="&lt;p&gt;CNN (and RNN) models are not &lt;em&gt;general&lt;/em&gt; improvements to the MLP design. They are specific choices that match certain types of problem. The CNN design works best when there is some local pattern in the data (which may repeat in other locations), and this is often the case when the inputs are images, audio or other similar signals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reuters example looks like a &quot;bag of words&quot; input. There is no local pattern or repeating relationships in that data that a CNN can take advantage of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your results with a CNN on this data set look reasonable to me. You have not made a mistake, but learned how a CNN really works on this data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-17T07:27:23.183" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19024" PostTypeId="2" ParentId="18822" CreationDate="2017-05-17T07:55:14.877" Score="1" Body="&lt;p&gt;Keselman, Schubert&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;https://web.stanford.edu/class/cs224n/reports/2749103.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Computational models for text summarization&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The paper deals with methods (models) for text summarization. The reference (base) model was &quot;first sentence model&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As a baseline for our models we used a trivial model that repeats the&#xA;  first sentence of the input document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then, various experiments and results are presented, like this one: (notice that &quot;first sentence model&quot; is always present as &quot;baseline&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/M4E6j.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/M4E6j.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, one of the datasets for training and evaluation of models in this paper is &lt;a href=&quot;http://duc.nist.gov/data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DUC&lt;/a&gt;, which may be interesting to you.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Steinberger &lt;em&gt;(doctoral Thesis, 2005)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;https://www.google.rs/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwib9NzHvPTTAhVDWCwKHdhnCWUQFggtMAE&amp;amp;url=http%3A%2F%2Ftextmining.zcu.cz%2Fpublications%2FPhDThesis-Steinberger.pdf&amp;amp;usg=AFQjCNGCVVwQRrkYAnYNp5uWs3h_Sc51nQ&amp;amp;sig2=Q1pgh3Lpx_1NY7d0VJFvtg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text Summarization within the LSA Framework&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In section 2.1, the author discusses document summarization approaches based on sentence extraction. He identifies five approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Surface Level Approaches&lt;/li&gt;&#xA;&lt;li&gt;Corpus-based Approaches&lt;/li&gt;&#xA;&lt;li&gt;Cohesion-based Approaches&lt;/li&gt;&#xA;&lt;li&gt;Rhetoric-based Approaches&lt;/li&gt;&#xA;&lt;li&gt;Graph-based Approaches&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(The &quot;First Sentence Approach&quot; belongs to the *Surface Level Approaches&quot;) The author further describes these approaches and compares them.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Khodra, Widyantoro, Aziz, Trilaksono &lt;em&gt;(Journal of ICT Research and Applications, 2011)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://journals.itb.ac.id/index.php/jictra/article/download/205/199&quot; rel=&quot;nofollow noreferrer&quot;&gt;Free Model of Sentence Classifier for Automatic Extraction of Topic Sentences&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The author identifies and tests methods for identifying the most important sentences in a text (see list of 58 items below, called features). Surprisingly, in the conclusion, it is said that position of the sentence is a dominant feature, meaning that including all other features into consideration leads only to small improvement.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;position&lt;/li&gt;&#xA;&lt;li&gt;sentence length&lt;/li&gt;&#xA;&lt;li&gt;number of words before a main verb&lt;/li&gt;&#xA;&lt;li&gt;adjective incidence&lt;/li&gt;&#xA;&lt;li&gt;existential there incidence&lt;/li&gt;&#xA;&lt;li&gt;incidence of 3rd person singular grammatical form&lt;/li&gt;&#xA;&lt;li&gt;anaphora incidence&lt;/li&gt;&#xA;&lt;li&gt;coordinators incidence&lt;/li&gt;&#xA;&lt;li&gt;cardinal number incidence&lt;/li&gt;&#xA;&lt;li&gt;incidence of past tense endings&lt;/li&gt;&#xA;&lt;li&gt;Hypernymy&lt;/li&gt;&#xA;&lt;li&gt;Polysemy&lt;/li&gt;&#xA;&lt;li&gt;concreteness index&lt;/li&gt;&#xA;&lt;li&gt;affect_formulai&lt;/li&gt;&#xA;&lt;li&gt;bad_formulaic&lt;/li&gt;&#xA;&lt;li&gt;comparison_formulaic&lt;/li&gt;&#xA;&lt;li&gt;continue_formulaic&lt;/li&gt;&#xA;&lt;li&gt;contrast_formulaic&lt;/li&gt;&#xA;&lt;li&gt;detail_formulaic&lt;/li&gt;&#xA;&lt;li&gt;future_formulaic&lt;/li&gt;&#xA;&lt;li&gt;gap_formulaic&lt;/li&gt;&#xA;&lt;li&gt;good_formulaic&lt;/li&gt;&#xA;&lt;li&gt;here_formulaic&lt;/li&gt;&#xA;&lt;li&gt;in_order_to_formulaic&lt;/li&gt;&#xA;&lt;li&gt;method_formulaic&lt;/li&gt;&#xA;&lt;li&gt;no_textstructure_formulaic&lt;/li&gt;&#xA;&lt;li&gt;similarity_formulaic&lt;/li&gt;&#xA;&lt;li&gt;them_formulaic&lt;/li&gt;&#xA;&lt;li&gt;textstructure_formulaic&lt;/li&gt;&#xA;&lt;li&gt;tradition_formulaic&lt;/li&gt;&#xA;&lt;li&gt;us_previous_formulaic&lt;/li&gt;&#xA;&lt;li&gt;affect&lt;/li&gt;&#xA;&lt;li&gt;argumentation&lt;/li&gt;&#xA;&lt;li&gt;better_solution&lt;/li&gt;&#xA;&lt;li&gt;change&lt;/li&gt;&#xA;&lt;li&gt;comparison&lt;/li&gt;&#xA;&lt;li&gt;continue&lt;/li&gt;&#xA;&lt;li&gt;contrast&lt;/li&gt;&#xA;&lt;li&gt;interest&lt;/li&gt;&#xA;&lt;li&gt;need&lt;/li&gt;&#xA;&lt;li&gt;presentation&lt;/li&gt;&#xA;&lt;li&gt;problem&lt;/li&gt;&#xA;&lt;li&gt;research&lt;/li&gt;&#xA;&lt;li&gt;solution&lt;/li&gt;&#xA;&lt;li&gt;textstructure&lt;/li&gt;&#xA;&lt;li&gt;use&lt;/li&gt;&#xA;&lt;li&gt;copula&lt;/li&gt;&#xA;&lt;li&gt;aim_ref_agent&lt;/li&gt;&#xA;&lt;li&gt;gap_agent&lt;/li&gt;&#xA;&lt;li&gt;general_agent&lt;/li&gt;&#xA;&lt;li&gt;problem_agent&lt;/li&gt;&#xA;&lt;li&gt;ref_agent&lt;/li&gt;&#xA;&lt;li&gt;ref_us_agent&lt;/li&gt;&#xA;&lt;li&gt;solution_agent&lt;/li&gt;&#xA;&lt;li&gt;textstructure_agent&lt;/li&gt;&#xA;&lt;li&gt;them_agent&lt;/li&gt;&#xA;&lt;li&gt;them_pronoun_agent&lt;/li&gt;&#xA;&lt;li&gt;us_agent&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For you, the most important part of the paper may be table 5:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kMmsg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kMmsg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read carefully the explanation of the table in the paper, and the whole Section 4.3.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Other papers worth examining:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Luhn &lt;em&gt;(1958)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://courses.ischool.berkeley.edu/i256/f06/papers/luhn58.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Automatic Creation of Literature Abstracts&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Kupiec, Pedersen, Chen &lt;em&gt;(1995)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://courses.ischool.berkeley.edu/i256/f06/papers/kupiec95.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Trainable Document Summarizer&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Yang, Pedersen &lt;em&gt;(1997)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://courses.ischool.berkeley.edu/i256/f06/papers/yang97comparative.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Comparative Study on Feature Selection in Text Categorization&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Sebastiani &lt;em&gt;(2002)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;http://courses.ischool.berkeley.edu/i256/f06/papers/sebastiani02.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning in Automated Text Categorization&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="31513" LastEditDate="2017-05-17T15:42:45.693" LastActivityDate="2017-05-17T15:42:45.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19025" PostTypeId="1" CreationDate="2017-05-17T12:58:12.303" Score="10" ViewCount="2172" Body="&lt;p&gt;I have 200 data points that have the &lt;strong&gt;same&lt;/strong&gt; values on &lt;strong&gt;all&lt;/strong&gt; features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After t-SNE dimension reduction they doesn't look so equal anymore, just like this: &lt;a href=&quot;https://i.stack.imgur.com/QnGOo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QnGOo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why aren't they on the same point in the visualization and even seems to be distributed in two different clusters?&lt;/p&gt;&#xA;" OwnerUserId="32357" LastEditorUserId="32357" LastEditDate="2017-05-18T09:30:19.353" LastActivityDate="2017-07-17T22:22:38.893" Title="t-SNE: Why equal data values are visually not close?" Tags="|visualization|dimensionality-reduction|tsne|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="19028" PostTypeId="1" CreationDate="2017-05-17T13:36:53.513" Score="7" ViewCount="2841" Body="&lt;p&gt;I would like to keep the objective as &amp;quot;reg:linear&amp;quot; and &lt;code&gt;eval_metric&lt;/code&gt; as customized RMSE as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def customised_rmse(preds, dtrain):&#xA;    N = len(preds)&#xA;    preds = np.array(preds)&#xA;    actual = np.array(dtrain.get_label())&#xA;    crmse = np.sqrt(np.sum(np.power((preds+1)*1.0/(actual+1) - 1.1, 2))/N)&#xA;    return &amp;quot;custom-rmse&amp;quot;, crmse&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;When I ran, train function as follows -&amp;gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = xgb.train(param_list, xgb_train, num_rounds, watchlist, None, customised_rmse, early_stopping_rounds=30)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output I am getting is this -&amp;gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[0] train-rmse:15.1904  val-rmse:15.2102    train-custom-rmse:0.607681  val-custom-rmse:0.610993&#xA;Multiple eval metrics have been passed: 'val-custom-rmse' will be used for early stopping.&#xA;Will train until val-custom-rmse hasn't improved in 30 rounds.&#xA;[1] train-rmse:14.4936  val-rmse:14.5103    train-custom-rmse:0.588831  val-custom-rmse:0.589902&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;and so on...&lt;/p&gt;&#xA;&lt;p&gt;My question is, it is getting optimized using &amp;quot;rmse&amp;quot; or &amp;quot;custom-rase&amp;quot; or both? What I have to do to remove &amp;quot;rmse&amp;quot; as it comes by default with &amp;quot;reg:linear&amp;quot;?&lt;/p&gt;&#xA;" OwnerUserId="32359" LastEditorUserId="29169" LastEditDate="2021-11-16T21:27:06.213" LastActivityDate="2023-04-23T21:05:05.553" Title="In XGBoost, how to change eval function and keeping same objective?" Tags="|python|regression|xgboost|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19029" PostTypeId="1" AcceptedAnswerId="19032" CreationDate="2017-05-17T13:43:34.910" Score="1" ViewCount="41" Body="&lt;p&gt;Are any of the open deep learning toolkits targeted to certain areas, or a all toolkits all purpose toolkits, meaning it is a blackbox for deep learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question comes in regards to Microsoft's CNTK which seem to contain examples of speech and text classification, where others usually just have MNIST or CIFAR... &lt;/p&gt;&#xA;" OwnerUserId="25429" LastActivityDate="2017-05-17T14:25:56.440" Title="Are deep - learning toolkits targeted for certain areas or all-purpose tool kits?" Tags="|deep-learning|tools|open-source|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19030" PostTypeId="1" AcceptedAnswerId="25835" CreationDate="2017-05-17T13:46:30.053" Score="7" ViewCount="1268" Body="&lt;p&gt;We are using Orange and have 2 files (training and testing). We apply different Learners (kNN, AdaBoost...) and get the Evaluation Results. But we have some doubts about some options in the Test &amp;amp; Score window.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LYTQh.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LYTQh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/f8Ot6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f8Ot6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we apply the option &quot;Test on train data&quot; does it generate for each Learner a model and then applies it to the training population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the same about the &quot;Test on test data&quot;, does it generate the model with the training data and this option applies it to the testing population?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because when we get the confusion matrices and we calculate accuracy, sensitivity and specificity the values differ from the ones in the Evaluation Results.&lt;/p&gt;&#xA;" OwnerUserId="32360" LastEditorUserId="32360" LastEditDate="2017-05-17T16:10:34.427" LastActivityDate="2018-09-19T15:05:22.160" Title="Differences between Test &amp; Score results and calculated by Confusion Matrix (accuracy, sensitivity and specificity)" Tags="|machine-learning|orange|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19032" PostTypeId="2" ParentId="19029" CreationDate="2017-05-17T14:25:56.440" Score="2" Body="&lt;p&gt;Yes, different toolkits are suited for different purposes given they contain different algorithms. This is due to the lack of a general AI (an AI that is intelligent in all respects). Modern AI has different algorithms which are better suited for different tasks. For example a CNN framework is much better for images than a NN would be, and much better than encoders. However, a LSTM framework would be better suited to capture temporal dependencies in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if two frameworks are using the same algorithms they should both be relatively the same. Differences will arise due to the underlining codes being different as can be seen between Keras and TensorFlow. Although, this should be limited.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Worry more about the algorithm you are choosing for your task than the framework. Coding up most machine learning algorithms is not that hard if you want to customize it. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-05-17T14:25:56.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19035" PostTypeId="1" CreationDate="2017-05-17T15:56:01.743" Score="-1" ViewCount="169" Body="&lt;p&gt;I found the following very surprising.&#xA;I trained different machine learning classifiers on a data set that includes $15$ attributes using $k$-fold cross validation. Then test on a different validation data set.&#xA;The result were fine, accuracy was about $81\%$ and the f1-measure was about $83\%$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out of curiosity, I tried to train and test on one single attribute, and I found that there is one attribute that outperform the previous result, accuracy was about $85\%$ and the f1-measure was about $86\%$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This results of the one attribute was always better than all attribute when using different binary classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How common is this? Is there any interpretations for the reasons behind this?&lt;/p&gt;&#xA;" OwnerUserId="4968" LastActivityDate="2017-05-18T10:04:49.210" Title="binary classification: 1 attribute vs many" Tags="|classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19036" PostTypeId="2" ParentId="19035" CreationDate="2017-05-17T16:56:27.840" Score="1" Body="&lt;p&gt;If all the attributes are highly correlated with the best attribute that you down-selected to, I guess that could happen as having too many correlated variables is known to cause problems in certain cases. The model might have settled on a local maximum and you happened to know the global maximum. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-17T16:56:27.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19038" PostTypeId="2" ParentId="19010" CreationDate="2017-05-17T18:29:43.167" Score="5" Body="&lt;p&gt;Cosine is not a commonly used activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Activation_function&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia page describing common activation functions&lt;/a&gt;, it is not listed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And one of the &lt;em&gt;desirable properties&lt;/em&gt; of activation functions described on that page is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;$cos(0) = 1$, a basic cosine function does not have this property. Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest trying with a low learning rate, and initialising all the bias values to $-\frac{\pi}{2}$. Maybe reduce the variance in initial weights a little too, just to start off with things close to zero. Essentially this is starting with $sin()$. Caveat: not that I have tried this myself, just an educated guess, so I would be interested to know if that helps at all with stability.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-17T18:36:17.577" LastActivityDate="2017-05-17T18:36:17.577" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19039" PostTypeId="2" ParentId="12851" CreationDate="2017-05-17T19:02:16.710" Score="7" Body="&lt;p&gt;There are some novel alternative efforts on neural network visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see these articles:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wired.co.uk/article/ai-machine-learning-brain-scan&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stunning 'AI brain scans' reveal what machines see as they learn new skills&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.graphcore.ai/posts/what-does-machine-learning-look-like&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inside an AI 'brain' - What does machine learning look like?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These approaches are more oriented towards visualizing neural network operation, however, NN architecture is also somewhat visible on the resulting diagrams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qd5rF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qd5rF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/q80fj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q80fj.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LQQuG.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LQQuG.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t1xZk.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t1xZk.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="54504" LastEditDate="2018-07-03T15:10:18.497" LastActivityDate="2018-07-03T15:10:18.497" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="19040" PostTypeId="1" AcceptedAnswerId="19103" CreationDate="2017-05-17T19:11:57.737" Score="0" ViewCount="6566" Body="&lt;p&gt;I am new to clustering.i have data from quality testing of an automobile manufacturing company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 100000 datasets.each dataset has 4 variables force, voltage, current, distance. each variable is a continuous time-series with 8000 data points each(1 to 17000 milliseconds). the length of time series differs from on dataset to another. all variables in one dataset has to be compared with another dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to find clusters in the 100000 datasets based on similarities in shape of each variable in a dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which type of measuring best suits, in this case, to find similarity in shape of time-series&lt;/p&gt;&#xA;" OwnerUserId="32367" LastEditorUserId="32367" LastEditDate="2017-05-21T11:01:13.507" LastActivityDate="2017-11-07T09:18:16.527" Title="clustering multivariate time-series datasets" Tags="|machine-learning|data-mining|clustering|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="19041" PostTypeId="2" ParentId="13073" CreationDate="2017-05-17T19:12:55.050" Score="0" Body="&lt;p&gt;This picture from &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; may help you get insight what methods would yield good result in your case, and what wouldn't, and why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tji0q.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tji0q.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31513" LastActivityDate="2017-05-17T19:12:55.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19042" PostTypeId="1" CreationDate="2017-05-17T19:32:21.607" Score="2" ViewCount="1231" Body="&lt;p&gt;For vanilla K-Means clustering algorithm I know that the time complexity is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Time complexity: O(tknm),&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where n is the number of data points, k is the number of clusters, and&#xA;t is the number of iterations, m is the dimensionality of the vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, when I studied about Mini-batch K-Means to make the algorithm converge faster, I wanted to find out what is the Space &amp;amp; Time complexity of it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially so that I understand well, how much we are optimizing over vanilla K-Means.&lt;/p&gt;&#xA;" OwnerUserId="32373" LastActivityDate="2017-05-17T21:25:35.407" Title="What is the Space &amp; Time Complexity of Mini-Batch K-Means clustering algorithm?" Tags="|classification|clustering|algorithms|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19043" PostTypeId="1" CreationDate="2017-05-17T20:01:00.590" Score="11" ViewCount="43071" Body="&lt;p&gt;I wish to better understand the difference between lazy and eager learning. I am having difficulty conceptualising what the &quot;abstraction&quot; refers to between the two. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the text book I am reading it says, &quot;The distinction between easy learners and lazy learners is based on when the algorithm abstracts from the data.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lazy learner delays abstracting from the data until it is asked to make a prediction while an eager learner abstracts away from the data during training and uses this abstraction to make predictions rather than directly compare queries with instances in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that KNN algorithm loads all the data into memory so depending on the size of the dataset, the computing requirements could be enormous as well as the fact it's a non-parametric algorithm so there is no training to find parameters, the classification is done on a per query requirement??&lt;/p&gt;&#xA;" OwnerUserId="30959" LastActivityDate="2018-12-27T14:57:40.573" Title="Lazy vs Eager Learning" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19045" PostTypeId="2" ParentId="19042" CreationDate="2017-05-17T21:25:35.407" Score="2" Body="&lt;p&gt;Infinite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mini-batch k-means never converges, you need to use an iteration limit or similar heuristic, and you can never guarantee to have found a local optimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, mini-batch k-means is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;draw a random sample&lt;/li&gt;&#xA;&lt;li&gt;perform one iteration of k-means using this sample&lt;/li&gt;&#xA;&lt;li&gt;repeat&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Assuming that your sample size is N, 2 takes O(k N m t) time.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-05-17T21:25:35.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19046" PostTypeId="2" ParentId="11047" CreationDate="2017-05-17T21:30:49.853" Score="1" Body="&lt;p&gt;Arsenal,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have to reshape the input data to fit in the LSTM's input layer like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   trainX= np.reshape(trainX, (-1, 16, 4096))&#xA;   testX = np.reshape(testX, (-1, 16, 4096))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When one layer's output goes to the following layer's input, the earlier layer must produce a sequence like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    net = tflearn.lstm(net,128, dropout=0.2, return_seq=True)&#xA;    net = tflearn.lstm(net,128)&#xA;    net = tflearn.fully_connected(net, num_classes, activation='softmax')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="32377" LastEditorUserId="381" LastEditDate="2017-05-18T05:11:54.547" LastActivityDate="2017-05-18T05:11:54.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19048" PostTypeId="1" AcceptedAnswerId="19050" CreationDate="2017-05-17T22:12:24.887" Score="5" ViewCount="134" Body="&lt;p&gt;I am trying to entice my lab to transition from Matlab and R to Python. The main objection at this point seems to be that Python's analytical libraries are not sufficiently well documented. Given how prolific Python is, I suspect that sufficiently detailed documentation exists and we simply cannot find it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently, I needed to upsample a signal (1D vector) and found a Python function called &lt;code&gt;resample&lt;/code&gt;. In contrast to Matlab and R, &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html&quot; rel=&quot;noreferrer&quot;&gt;pandas documentation&lt;/a&gt; doesn't tell me what kind of interpolation resample uses (linear, cubic splines, pchip?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any place where I could find such information about this and other libraries, preferably with equations or references to papers? I understand that I could analyze source code but this isn't most time efficient. If you know R, I am basically looking for a Python equivalent of &lt;a href=&quot;https://cran.r-project.org/web/packages/pracma/pracma.pdf&quot; rel=&quot;noreferrer&quot;&gt;CRAN&lt;/a&gt; (PDF warning). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="32379" LastActivityDate="2017-05-23T17:33:25.097" Title="Where to find statistically relevant documentation of common Python packages?" Tags="|python|statistics|pandas|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19049" PostTypeId="2" ParentId="19048" CreationDate="2017-05-17T23:34:25.247" Score="4" Body="&lt;p&gt;I think the documentation, at least for your example, is not too obscure. It doesn't tell you what kind of interpolation it uses because it doesn't use any: &lt;code&gt;resample&lt;/code&gt; is a deferred operation. In order for it to work you have to call it in conjunction with a function that performs the interpolation, e.g. &lt;code&gt;series.resample(frequency).mean()&lt;/code&gt; or &lt;code&gt;series.resample(frequency).interpolation('cubic')&lt;/code&gt; (as can be seen from the examples in the documentation). Sometimes, it can be helpful to also look into the release notes for additional information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scipy and Numpy are sometimes not very detailed, but they contain at least a decent number of references with more information. (There is also a section on Matlab versus Numpy &lt;a href=&quot;https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If you are familiar with &lt;a href=&quot;https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;dplyr&lt;/code&gt; in R&lt;/a&gt; , then the &lt;a href=&quot;https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas cheat sheet&lt;/a&gt; can help as a quick comparison between R and pandas for typical data wrangling operations.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not aware of a &quot;second more detailed documentation&quot; that encompasses the entire libraries. You probably have to fall back to books (which, of course, aren't complete or always up to date).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-05-23T17:33:25.097" LastActivityDate="2017-05-23T17:33:25.097" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19050" PostTypeId="2" ParentId="19048" CreationDate="2017-05-17T23:34:35.797" Score="4" Body="&lt;p&gt;Switching to Python from Matlab is going to be somewhat dependent on the field you are in (for example, if you are in image processing/ imaging, Matlab is pretty well solid and so swithcing will be more difficult) and the stubbornness of your coworkers/boss.  While documentation for Matlab is easy to find, it's also all you get, whereas Python packages generally have decent documentation as well as countless posts on SE in which someone has likely already answered your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the greatest aspects of Python is that it is open source, but it can make finding the right tool slightly more time-consuming.  Your example is about resampling.  Pandas &lt;code&gt;resample&lt;/code&gt; is designed for time-series (see &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.resample.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas doc&lt;/a&gt;).  All the arguments are described, but I can also look at the source code if I really need to understand anything more in depth.  Or I search for some examples online: &lt;a href=&quot;http://benalexkeen.com/resampling-time-series-data-with-pandas/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://benalexkeen.com/resampling-time-series-data-with-pandas/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it seems like you are actually interested in interpolation functions.  If we search for &quot;Python interpolation&quot; we will discover a few methods. Looks like Scipy has an extensive package for interpolating: &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/interpolate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.scipy.org/doc/scipy/reference/interpolate.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It takes some time to figure out how to search for what you need in Python, but once you get the hang of it, there are a lot of great examples online of all sorts of application.  Plus, plotting in Python (and R) is way beyond Matlab (c'mon mathworks).  In the end, you can do what I did which is flat out refuse to use Matlab unless it is actually necessary and hope they don't fire you!&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-05-17T23:34:35.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19051" PostTypeId="1" AcceptedAnswerId="19052" CreationDate="2017-05-18T00:14:06.673" Score="2" ViewCount="3203" Body="&lt;p&gt;My question is focused around how to appropriately update an encoded feature set when a new category is introduced by the test data. I use the data in logistic regression and I know it is not a 'live' model (i.e. gradient descent is performed whenever new data is introduced) but do I have to retrain the model to account for added features or do I just add it to subsequent test set values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To exemplify the problem consider a TV Show training set where each show has a 'networks' feature set that includes one or more of the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&quot;abc&quot;,&quot;cbs&quot;,&quot;nbc&quot;] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, in the testing set there is a TV Show with the feature set: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&quot;abc&quot;, &quot;hulu&quot;] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would I have to add the new feature retroactively to the training data and retrain the model eventhough it will never occur? Wouldn't this introduce 'look-ahead-bias'? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I account for the added feature in the encoder going forward?&lt;/p&gt;&#xA;" OwnerUserId="29801" LastEditorUserId="29801" LastEditDate="2017-05-18T01:34:32.537" LastActivityDate="2017-05-18T01:34:32.537" Title="Updating One-Hot Encoding to account for new categories" Tags="|machine-learning|logistic-regression|categorical-data|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19052" PostTypeId="2" ParentId="19051" CreationDate="2017-05-18T01:16:20.407" Score="2" Body="&lt;p&gt;I think you have two options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Automate your train/test pipeline so that one-hot encoding is part of&#xA;it. If new categorical variables are introduced, they can be featured&#xA;in the training dataset even if not very prevalent. This would&#xA;introduce some bias if the nature of the TV show distribution has&#xA;changed over time (e.g. 20 years ago there weren't as many options)&#xA;but I don't necessarily think it is a show stopper.&lt;/li&gt;&#xA;&lt;li&gt;If new possibilities are introduced over time but for whatever reason you can't  retrain, then you should omit using that new value. This has its own disadvantages because in your example, it would be a TV show with no network.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-18T01:16:20.407" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19053" PostTypeId="2" ParentId="4967" CreationDate="2017-05-18T01:28:53.943" Score="2" Body="&lt;p&gt;A few things,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Have you determined whether the relationship between hr_of_day and power_used is statistically significant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend doing a kendall's tau correlation if you haven't already. I like kendall's tau because it handles non-linear relationships and can be considered as the probability that the probability that the two are related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Also, I would check whether temperature and hour are related. If there is multicollinearity you might need to reexamine the factors applied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) If you know that the independent fits a bimodal distribution then run some P-P plots on known bimodal distributions (such as 'beta distributions'). It would also be interesting to extract the potential gaussian distributions underlying the population - running two OLS models sequentially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, you may be better off working from time-series analysis where the function is expressed in terms of hour. But I wouldn't go adding your own factors most regression analyses will &quot;tell&quot; you that something is missing. The way to tell is if the model coefficients are not statistically significant or the amount of explained variance is small. But I reiterate that coercing the hour to a cosine or sine function implies a relationship between the dependent and independent variable that may not be real. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like you might want to go back and do some (more) due diligence before throwing it into an ML model.&lt;/p&gt;&#xA;" OwnerUserId="29801" LastActivityDate="2017-05-18T01:28:53.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19054" PostTypeId="1" CreationDate="2017-05-18T05:59:25.140" Score="3" ViewCount="320" Body="&lt;p&gt;I am doing credit risk modelling on costumer transaction data a part of which looks like this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;str(x)&#xA;'data.frame':   412516 obs. of  26 variables:&#xA; $ Tenure           : num  1.26 1.25 1.26 1.31 1.32 ...&#xA; $ Product          : Factor w/ 24 levels &quot;BACKHOE LOADER&quot;,..: 4 4 4 9 9 9 9 9 9 9 ...&#xA; $ Net.Exposure     : num  333339 528049 327335 350000 460000 ...&#xA; $ OD.On.31.01.2017 : num  0 90386 0 0 1099692 ...&#xA; $ LM.Bucket        : Ord.factor w/ 11 levels &quot;0&quot;&amp;lt;&quot;1 TO  30&quot;&amp;lt;..: 1 1 1 1 11 11 11 11 11 11 ...&#xA; $ Bucket           : Ord.factor w/ 11 levels &quot;0&quot;&amp;lt;&quot;1 TO  30&quot;&amp;lt;..: 1 3 1 1 11 11 11 11 11 11 ...&#xA; $ Billing          : num  65380 0 8800 6339 8331 ...&#xA; $ Fin.IRR          : num  13.5 14.6 14.6 18.1 23.3 ...&#xA; $ NPA.Flag         : Factor w/ 2 levels &quot;No&quot;,&quot;Yes&quot;: 1 1 1 1 2 2 2 2 2 2 ...&#xA; $ Inst.Due         : num  0 0.85 0 0 3 3 3 3 3 3 ...&#xA; $ FR.On.31.01.2017 : num  65380 0 38940 35043 499860 ...&#xA; $ POS.On.31.01.2017: num  56453 0 32920 33368 293943 ...&#xA; $ Del.String       : int  2 1 1 1 53720 53720 53720 53720 53720 53720 ...&#xA; $ Territory        : Factor w/ 43 levels &quot;AGRA&quot;,&quot;AHMEDABAD&quot;,..: 41 41 41 41 41 41 41 41 41 41 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The variables like OD(Overdue) and LM.Bucket( How many months he has been due on his loan payment till last month) change every month .I have 2 tasks :Predict Bucket and NPA Flag(Non performing asset)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I built a model for this based only on the Jan data(x). But my question is since these variables change every month, should i treat this as a sequential data and build a deep learning model(HMM/NN) on it? If i should what should I do with the static variables like Product type etc.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked my boss regarding the same and he said it shouldn't be done because external economy factor change with time. Is that a reason for concern?&lt;/p&gt;&#xA;" OwnerUserId="31822" LastEditorUserId="32177" LastEditDate="2017-05-18T08:59:54.773" LastActivityDate="2017-05-18T09:47:10.090" Title="Supervised Learning on Sequential data" Tags="|neural-network|classification|deep-learning|sequential-pattern-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19055" PostTypeId="2" ParentId="19054" CreationDate="2017-05-18T09:07:35.937" Score="0" Body="&lt;p&gt;The objective with supervised learning is to try to create a model of your data that helps you predict future values. You do that by selecting &lt;code&gt;features&lt;/code&gt; on your data set - what you call variables - that you believe represent well the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no expert but I understand economy is affected by an enormous number of variables, so even if you create a model that fits the data you currently have based on some of those variables, it might become obsolete the moment variables you did not consider start affecting the end result. That's what I believe your boss was talking about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you do decide to train a neural network in order to predict Bucket and NPA your first step will be to choose which variables you'll consider in your model. Keeping the 'static' variables will likely make your network have different predictions for, for example, different Product types but that depends on how the data is distributed across this static variables. If you choose to not use the static variables your model will completely ignore them when making predictions, which might not be what you want.&lt;/p&gt;&#xA;" OwnerUserId="32177" LastActivityDate="2017-05-18T09:07:35.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19057" PostTypeId="2" ParentId="18828" CreationDate="2017-05-18T09:15:04.867" Score="1" Body="&lt;p&gt;I believe you can find some material in David Barber's book &lt;a href=&quot;http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Reasoning and Machine Learning&lt;/a&gt;. Check out chapter 22 for 'rankings from pairwise comparisons'. The book has a MATLAB toolbox with a Rasch model function implemented there. Ranking models such as the  Bradley-Terry-Luce are modifications from the Rasch model, so I believe this code can provide you a head start. The routines are small, so converting from MATLAB to Python will not be very difficult.&lt;/p&gt;&#xA;" OwnerUserId="8792" LastActivityDate="2017-05-18T09:15:04.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19059" PostTypeId="2" ParentId="19054" CreationDate="2017-05-18T09:36:59.217" Score="1" Body="&lt;p&gt;This basically asks for a recurrent network, like the &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM&lt;/a&gt;. But if you only have 2 properties that are dynamic, I don't think you will have as much luck because they might be affected from external factors as your boss said. However, this will happen &lt;em&gt;regardless&lt;/em&gt; of the model you're using. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should not throw away static properties, unless they are the same for every test case. E.g. farmer/politician/baker category should always be included, but you call this 'static' but it's not completely static as it is not the same for every sample in your test cases.&lt;/p&gt;&#xA;" OwnerUserId="30928" LastEditorUserId="30928" LastEditDate="2017-05-18T09:47:10.090" LastActivityDate="2017-05-18T09:47:10.090" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19061" PostTypeId="2" ParentId="19035" CreationDate="2017-05-18T10:04:49.210" Score="1" Body="&lt;p&gt;What different classfiers did you use? Because rf and gbm are pretty robust to  multicollinearity(Not always tho!) . Also it may be because that one variable has much more predictive power than the others(e.g Others might have less variance in them or highly imbalanced classes in cases of factor featues). Did you check variable importance using information gain or chi squared tests? Do these and check if that one variable you used is much better scored than the rest. That may be a reason. &lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-05-18T10:04:49.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19062" PostTypeId="1" CreationDate="2017-05-18T11:28:59.690" Score="4" ViewCount="857" Body="&lt;p&gt;I've been wondering what the residual blocks &lt;em&gt;actually do&lt;/em&gt; and why they are a necessity for deep convolutional networks. Specifically, why are they so useful in some generator networks? To better extract features? I want an intuition for this concept, but can't find an understandable explanation online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what does the summation at the end of the block mean? It shows skipping around the block but in the end it adds it separately anyway? What does x and F(x) really represent in this context? Is F(x) an image feature of a given input and x is the input?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me it looks like it's for feature detection/recognition because of the way it individually adds that separately calculated sub-results, but I could be wrong. And it doesn't entirely make sense to me this way either, because how does this particular method help to achieve that end? Is it a principle of reaffirmation - the net separately calculates some of the features and through these means knows &lt;em&gt;for sure&lt;/em&gt; that a certain texture is what it is instead of something else (less mistakes everywhere)?&#xA;But then - why not have this for the discriminator as well? It fundamentally needs this same tech, doesn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it just that the generator needs to improve more than the discriminator does? In other words - if the discriminator had these blocks it would start classifying pretty much only the exact replicas of the input image and the generator would never get fool the discriminator? But then - what importance would that be anyway - the generator would get just as good at image generation anyway.&lt;/p&gt;&#xA;" OwnerUserId="32398" LastEditorDisplayName="user380" LastEditDate="2017-07-13T15:49:50.983" LastActivityDate="2017-07-13T15:49:50.983" Title="High level understanding of residual blocks" Tags="|deep-learning|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19067" PostTypeId="2" ParentId="18933" CreationDate="2017-05-18T18:25:03.220" Score="1" Body="&lt;p&gt;The following code works:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import scipy&#xA;import numpy as np&#xA;&#xA;ey = np.random.exponential(size=100)&#xA;cdfy = scipy.stats.expon.cdf(np.sort(ey))&#xA;invcdf = scipy.stats.norm.ppf(cdfy) # a normal distribution&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;" OwnerUserId="24651" LastActivityDate="2017-05-18T18:25:03.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19072" PostTypeId="1" CreationDate="2017-05-19T04:01:55.777" Score="-1" ViewCount="1051" Body="&lt;p&gt;I'm building a model that predicts the subreddit of a given reddit submission. I have a question regarding pre-processing my text (submissions). The order I followed when pre-processing my text was:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove punctuations&lt;/li&gt;&#xA;&lt;li&gt;Remove stopwords&lt;/li&gt;&#xA;&lt;li&gt;Lemmatize the remaining text&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I then fed the text into a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CountVectorizer&lt;/a&gt;.&#xA;While this got me decent accuracy on my naive bayes model, I discovered I was getting much better results (about ~5% gains) when I skipped #2 altogether. What are some reasons why this could be happening? &lt;/p&gt;&#xA;" OwnerUserId="31424" LastActivityDate="2017-05-19T06:54:03.533" Title="Stopwords removal (surprisingly) decreases accuracy of naive bayes model" Tags="|machine-learning|nlp|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19074" PostTypeId="1" CreationDate="2017-05-19T06:40:09.947" Score="1" ViewCount="330" Body="&lt;p&gt;It's not entirely clear what pre-trained networks are out there in deep learning. Of particular interest to me are the deep convolutional networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm only aware of the VGG series like VGG16/VGG19. I searched the internet for Microsoft's famous ResNet that won ImageNet 2015, but couldn't find it. I suppose the corporate giants don't really share their models, eh?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers to maybe a comprehensive list that I have simply overlooked or there isn't one?&lt;/p&gt;&#xA;" OwnerUserId="32398" LastEditorUserId="29169" LastEditDate="2021-02-16T02:42:08.200" LastActivityDate="2021-02-16T02:42:08.200" Title="What pre-trained models are available?" Tags="|neural-network|image-classification|convolutional-neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19075" PostTypeId="2" ParentId="17540" CreationDate="2017-05-19T06:50:33.463" Score="4" Body="&lt;p&gt;This would also work.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(20,15))&#xA;ax=subplot(111)&#xA;sns.heatmap(corr,ax=ax)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31446" LastActivityDate="2017-05-19T06:50:33.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19076" PostTypeId="2" ParentId="19072" CreationDate="2017-05-19T06:54:03.533" Score="0" Body="&lt;p&gt;It is likely that at least one of the stop words you have removed is predictive of the subreddit category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could be a post style thing - e.g. in a simple case the posts have different typical lengths depending on class, and counting high frequency words gives a good proxy for article length. You could test that by using article length as an additional feature whilst still removing the stop words, and see whether any of the 5% improvement is accounted for by that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible that certain stop words are really more associated with certain topics. Perhaps they occur in common phrases associated with the topic. You could check feature importance from the weights and see which of the stop words were being used most heavily by the model.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-19T06:54:03.533" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19077" PostTypeId="1" CreationDate="2017-05-19T07:31:17.630" Score="13" ViewCount="1704" Body="&lt;p&gt;Is there any difference between machine learning and artificial intelligence? Or do these terms refer to the same thing?&lt;/p&gt;&#xA;" OwnerUserId="23850" LastEditorUserId="85045" LastEditDate="2020-12-23T03:57:34.590" LastActivityDate="2020-12-23T03:57:34.590" Title="Difference between machine learning and artificial intelligence" Tags="|machine-learning|ai|theory|" AnswerCount="8" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="19078" PostTypeId="2" ParentId="19077" CreationDate="2017-05-19T09:33:23.957" Score="5" Body="&lt;p&gt;Deep Learning is a subset of Machine Learning which is a subset of Artificial Intelligence. Machine learning is a particular approach for AI but not the only one. Symbolic Logic, Bayersian Statistics are a few examples of AI approaches which do not use any kind of machine learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-05-19T09:33:23.957" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19079" PostTypeId="2" ParentId="18828" CreationDate="2017-05-19T10:40:10.920" Score="6" Body="&lt;p&gt;Pairwise comparison models such as that of Bradley and Terry can easily be extended to your case, when you have pairwise comparison &lt;em&gt;probabilities&lt;/em&gt; instead of &lt;em&gt;binary outcomes&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $N$ be the number of items, and let $p_{ij}$ be the probability that query $j$ is better than query $i$.&#xA;Then, the log-likelihood of the Bradley-Terry parameters $\lambda_1, \ldots, \lambda_N$ given probabilities $\{ p_{ij} \}$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\sum_{i,j} p_{ij} [\log(\lambda_j) - \log(\lambda_i + \lambda_j)]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be reparametrized into a convex function, and the maximum-likelihood parameters can be found by one of many convex opimization methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a simple Python algorithm that will find the ML estimate, using a &lt;a href=&quot;http://sites.stat.psu.edu/~dhunter/papers/bt.pdf&quot; rel=&quot;noreferrer&quot;&gt;minorization-maximization approach&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;def mle(pmat, max_iter=100):&#xA;    n = pmat.shape[0]&#xA;    wins = np.sum(pmat, axis=0)&#xA;    params = np.ones(n, dtype=float)&#xA;    for _ in range(max_iter):&#xA;        tiled = np.tile(params, (n, 1))&#xA;        combined = 1.0 / (tiled + tiled.T)&#xA;        np.fill_diagonal(combined, 0)&#xA;        nxt = wins / np.sum(combined, axis=0)&#xA;        nxt = nxt / np.mean(nxt)&#xA;        if np.linalg.norm(nxt - params, ord=np.inf) &amp;lt; 1e-6:&#xA;            return nxt&#xA;        params = nxt&#xA;    raise RuntimeError('did not converge')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example usage:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import itertools&#xA;&#xA;# Generating pairwise probability matrix.&#xA;pmat = np.zeros((10, 10))&#xA;for i, j in itertools.permutations(range(10), r=2):&#xA;    pmat[i][j] = (j + 1) / (i + j + 2)&#xA;&#xA;# Estimating Bradley-Terry model parameters.&#xA;params = mle(pmat)&#xA;&#xA;# Ranking (worst to best).&#xA;ranking = np.argsort(params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Source: I am an author of a Python library for parameter inference in various statistical comparison models, &lt;a href=&quot;https://github.com/lucasmaystre/choix&quot; rel=&quot;noreferrer&quot;&gt;choix&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32417" LastActivityDate="2017-05-19T10:40:10.920" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19081" PostTypeId="1" CreationDate="2017-05-19T14:04:59.290" Score="-1" ViewCount="398" Body="&lt;p&gt;I have a &lt;code&gt;data.set&lt;/code&gt; that contain around 3000 observations. Every observation falls in one of the five categories (these are pre-defined). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;code&gt;randomForest&lt;/code&gt; and &lt;code&gt;gbm&lt;/code&gt; from the &lt;code&gt;h2o&lt;/code&gt; package for classification. Before I run the algorithms i optimize the &lt;code&gt;hyperparameters&lt;/code&gt; of each algorithm using the &lt;code&gt;h2o.grid&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For model validation I am having a look at the &lt;code&gt;confussion matrix&lt;/code&gt; for both algorithms, for both &lt;code&gt;in-sample&lt;/code&gt; and &lt;code&gt;out-of-sample&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is that in sample the algorithms perform relatively well for all 5 categories but out of sample they they only perform well for one category. I guess that is very common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 explanations: the 5 categories are not equally divided, this means that for a couple categories I have a few observations.&#xA;Second, is that the algorithms are overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;1m dollar question&quot; is how can someone avoid overfitting ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of running a preliminary analysis on my data set, to see which variables (in my case most of them &lt;strong&gt;dummy&lt;/strong&gt; variables) in my data set are the &quot;most important&quot; for the classification, and then use those, instead of the whole data set. This could be tested by taking the means for the variables per category and which variable has the highest difference across the categories. Could this be a good idea ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the subject is too broad, but I think some ideas from experienced people on this topic could be useful for everyone !&lt;/p&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2017-05-19T20:42:32.320" Title="accuracy prediction out of sample gone bad ! over fitting ?" Tags="|machine-learning|classification|r|random-forest|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19083" PostTypeId="1" CreationDate="2017-05-19T15:03:45.007" Score="2" ViewCount="218" Body="&lt;p&gt;Lets assume I have an image which only has white background and black points, all same size. I need to randomly sample crops with a &lt;strong&gt;hardcoded size&lt;/strong&gt;. The condition is that all the crops need to contain &lt;strong&gt;at least&lt;/strong&gt;  &lt;em&gt;n&lt;/em&gt; points inside. Also, ideally the crops can be rotated any number of degrees. In the following sample image I have manually drawn what it would be considered as valid crops (green) and non-valid crops (red). The condition in this example is that a valid crop must have at least 5 points inside.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YA3iK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YA3iK.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note that I have made the image with paint so the crops are not exactly same size (they should), an also there are only two rotations (there should be any type of rotation)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an efficient algorithm for doing this and my idea is to implement it in Python. Any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="17484" LastEditDate="2017-05-19T15:18:16.497" LastActivityDate="2017-05-19T15:18:16.497" Title="How to randomly sample crops from plain image with points only if crop contains n points inside?" Tags="|sampling|geospatial|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19085" PostTypeId="2" ParentId="19077" CreationDate="2017-05-19T15:51:33.837" Score="10" Body="&lt;p&gt;The subject areas Artifical Intelligence and Machine Learning (plus Data Science) are loosely defined, such that it is hard to make strict statements about how they relate. In the general case, it seems that there are parts that overlap, but that they are quite far from being &quot;the same subject with two different names&quot; as suggested in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;Artifical Intelligence&lt;/em&gt; has many possible meanings and interpretations - which version to refer varies by time and by the source using it. Textbooks on artificial intelligence will often cover topics such as search algorithms, logical deduction and other things which are clearly not machine learning as it is practised today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, we could take it to refer to &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_general_intelligence&quot; rel=&quot;noreferrer&quot;&gt;Artificial General Intelligence&lt;/a&gt; (or &quot;hard AI&quot;), and it should be clear in this case that at least some form of learning algorithm(s) would be required to meet the goals of AGI. However, it is far less clear how much of AGI can be solved by combining machine learning into complex structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &lt;em&gt;Machine Learning&lt;/em&gt; has a few different working definitions, but this is a popular one:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is far more tightly defined than Artificial Intelligence, but still has a lot of scope. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trend to conflate AI and ML appears to be a media and marketing issue, not a technical one. I suspect this is in part due to advances in the last 5-10 years in neural networks. Neural network models have made strong progress, especially in signal processing of images, video, audio. There is also an analogy with biological brains which can be compelling - especially when the subject matter is simplified for consumption by mainstream media. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth mentioning Data Science too. Like Artificial Intelligence, the term is somewhat fuzzily defined. Also like AI, Data Science has more to it than just Machine Learning. To Data Science practitioners, ML is part of a toolkit to achieve goals - for some people it is a large part of what they do, for others it is just one part of a wider scope (actually training and refining a ML model might take only a small fraction of a professional data scientist, analyst or statistician's time). I think it is reasonable to state that Artificial Intelligence and Data Science relate to Machine Learning in a similar way.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-05-19T15:51:33.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19086" PostTypeId="2" ParentId="19081" CreationDate="2017-05-19T15:55:57.840" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Try solving the imbalance problem by using something like&#xA;SMOTE to make each class roughly proportional.&lt;/li&gt;&#xA;&lt;li&gt;Make sure you are splitting your in sample and out of sample data randomly.&#xA;Sometimes people will take the first x% and call that training, but&#xA;if your data has any order to it, the sample may not be&#xA;representative.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would not try removing variables from the data set until you've addressed the above.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-19T15:55:57.840" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19087" PostTypeId="1" CreationDate="2017-05-19T18:23:12.040" Score="-1" ViewCount="614" Body="&#xA;&#xA;&lt;p&gt;I'm classifying data into 2 classes, by Logistic Regression from python scikit-learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying different types of feature vectors:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Binary&lt;/li&gt;&#xA;&lt;li&gt;Frequency&lt;/li&gt;&#xA;&lt;li&gt;TF-IDF&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In Binary and TF-IDF feature vectors, I get great results. However, in Frequency feature vectors, I get poor results unless I multiply the values by some factor (for example 10, or 100). The multiplication improves the results significantly (the higher the factor, the better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a normal behavior or is it more likely that I'm doing something wrong? I suspected that the values are too small and my array rounded some of them down to zero because its type was merely &lt;code&gt;float&lt;/code&gt; (unspecified bits number). But I tried changing it to &lt;code&gt;np.float64&lt;/code&gt;, and the results didn't change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What other factors can be causing this?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Learning Curves:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binary | TF-IDF | Frequency X 10 (from left to right):&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZcHQN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZcHQN.png&quot; alt=&quot;binary_tfidf_freq10&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Normal frequency (3 attempts):&lt;/strong&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/0e9Hg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0e9Hg.png&quot; alt=&quot;frequency&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2: (data and code)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequency Data: X axis: a dictionary of 1K words , Y axis: 1K samples - 500 each class&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/FZUBX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FZUBX.png&quot; alt=&quot;feature_vector&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Labels (1K binary labels):&lt;br&gt;&#xA;&lt;em&gt;note that it's a transpose vector. so X here is the samples&lt;/em&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xVF4z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xVF4z.png&quot; alt=&quot;labels&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code for frequency:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#loop...&#xA;vec[i] = np.divide(sample_vals, len(sample.split()))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code for frequency X 10:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#loop...&#xA;vec[i] = np.multiply(np.divide(sample_vals, len(sample.split())), 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Classification code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#in `X` I have the data, and in `y` I have the labels&#xA;scores = [clf.fit(X[train], y[train]).score(X[test], y[test])&#xA;          for train, test in kfold.split(X)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32433" LastEditorUserId="32433" LastEditDate="2017-05-22T06:16:16.977" LastActivityDate="2017-05-23T06:17:41.387" Title="Classification with frequency feature vector produces poor results" Tags="|python|scikit-learn|logistic-regression|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="19090" PostTypeId="2" ParentId="19062" CreationDate="2017-05-19T18:59:12.230" Score="4" Body="&lt;p&gt;Suppose we want to fit a function $f(x)$.  We can either try to learn a neural network model $F(\cdot)$ so that $F(x) \approx f(x)$.  Or, in the residual network approach, we try to learn a neural network model $R(\cdot)$ so that $x+R(x) \approx f(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the latter easier to learn?  There's no fundamental reason why it should necessarily be so, in general.  It depends on the specific data set.  However, one possible intuition is that we might expect that in some settings, $f(\cdot)$ might be approximately linear: i.e., as a first-order approximation, $f(x) \approx x$ might be a reasonable first-order approximation.  Then we want to model the error term: i.e., suppose $f(x)=x+r(x)$; then we want to model $r(x)$.    In some settings, the error term $r(x)$ might be simpler to model or smaller than $f(x)$.  In that case, a residual network architecture might work better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you like, you can think of this as being akin to a Taylor-series approximation.  The Taylor series for $f(x)$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = c_0 + c_1 x + c_2 x^2 + \cdots$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $x$ is small (i.e., $|x| \ll 1$).  Then $f(x) = c_0$ is a zero-th order approximation; $f(x) = c_0 + c_1 x$ is a first-order approximation; and so on.  At each step, we expect that the residual/error term is probably smaller than the approximation.  You can think of a residual network as learning a Taylor series for $f(x)$, in the special case where $c_0 = 0$ and $c_1 = 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other more sophisticated explanations / intuitions, but hopefully this gives one possible perspective.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-19T18:59:12.230" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19091" PostTypeId="2" ParentId="17036" CreationDate="2017-05-19T19:18:03.640" Score="1" Body="&lt;p&gt;Since apparently each feature is encoding something about two different categories, I would suggest that you should replace that with &lt;em&gt;two features&lt;/em&gt;.  Your two features would be $(x,y)$ where $x$ is 0 or 1 according to whether it exists in the first category, and $y$ is 0 or 1 according to whether it exists in the second category.  In other words, instead of -1, 0, and 1, you would use $(1,0)$, $(0,0)$, and $(0,1)$, respectively.  I think that is closer to the true data and might give better results.  It might also make your results easier to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can try both distance metrics and see which you find more helpful -- but it might not make a large difference.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-19T19:18:03.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19092" PostTypeId="1" AcceptedAnswerId="30518" CreationDate="2017-05-19T20:07:05.323" Score="5" ViewCount="73" Body="&lt;p&gt;Suppose I have a dataset $X$ and two different binary labels $y_{1}$ and $y_{2}$. The classes are very imbalanced - 3% of true in $y_{1}$  and 2% in $y_{2}$. Moreover, there are no pairs of (0,1), so, if the  $y_{1} = 0$, then  $y_{2}=0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to choose between:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Associating the labels with each other, so it's a 3-class classification problem&lt;/li&gt;&#xA;&lt;li&gt;Build a model for  $y_{1}$ scoring, then joining to $X$ and making scoring of  $y_{2}$, using $X$ and  $y_{1}$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So the first idea is formalised as :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ y_{f} = y_{1} \bullet y_{2}; \,\,\,\, y_{f} = \{(1,1),(0,0),(1,0) \}; \,\,\, y_{f} = \Phi_{f}(X)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the second road my be formalised like this:&#xA;$$ y_{1} = \Phi_{1}(X); \,\,\, \, \, X=X\cup y_{1};  \,\,\,\, y_{2}=\Phi_{2}(X) $$ &#xA;Taking into account all condition, what is the possible theoretical background of projecting the best algorithm?  I'am trying to give estimates to an &lt;strong&gt;&lt;em&gt;error&lt;/em&gt;&lt;/strong&gt;, which is :&lt;strong&gt;&lt;em&gt;bias + variance + noise&lt;/em&gt;&lt;/strong&gt;. E.g.:&#xA;For both: $\Phi_{f}$ and $\Phi_{2}$, which are &lt;code&gt;RandomForest&lt;/code&gt;s, &lt;strong&gt;&lt;em&gt;bias&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;noise&lt;/em&gt;&lt;/strong&gt; will be the same, &lt;strong&gt;&lt;em&gt;variance&lt;/em&gt;&lt;/strong&gt; will be higher for  $\Phi_{f}$(just intuition till now). Can somebody point me the right way?&lt;/p&gt;&#xA;" OwnerUserId="29931" LastEditorUserId="50727" LastEditDate="2018-06-06T08:23:34.300" LastActivityDate="2018-07-06T08:42:31.140" Title="Theoretical background for model arhitecture choosing" Tags="|machine-learning|predictive-modeling|prediction|error-handling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19093" PostTypeId="2" ParentId="19081" CreationDate="2017-05-19T20:42:32.320" Score="1" Body="&lt;p&gt;You will need to provide the exact distribution of the 5 levels so I can comment if its a case of overfitting. Let's say one of your class is 90% of the data, and the model is giving 90% accuracy on out of sample data (100% for the majority class) and 0% for rest of the classes. This is NOT a case of overfitting, the best the model could come up with was to put all in one class, it gave decent accuracy too. In general to avoid overfitting crossvalidation is really important. I suggest using mlr package rather than h20 because the crossvalidation there is easier. As suggested by CalZ you can use SMOTE to make your training classes in better proportion(but remember to not over/under sample the out of sample data).&#xA;As far as your feature selection technique goes,You can use chi-squared tests or information gain to plot the top features, also there are many advanced feature selection algorithms like Boruta etc. You can use them, but remember don't blindly trust these algorithms, they're not magic. Only do feature selection if lets say you have 100 variables( for 3000 datapoints). For 20-30 features, you can apply the model on all of them, plot the importance plot for gbm and then try to remove the less important variable and see if the accuracy increases.&lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-05-19T20:42:32.320" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19097" PostTypeId="1" CreationDate="2017-05-20T08:10:55.557" Score="-2" ViewCount="254" Body="&lt;p&gt;I am currently working with classifying patterns, and though at good place to start would be using already established models such as VGG16, INCEPTION models and so on.. Problem that my images are pretty small.. They are in shape of (8,15,3)... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any well established models able to handle this?&lt;/p&gt;&#xA;" OwnerUserId="27966" LastEditorUserId="29169" LastEditDate="2021-02-16T02:41:58.700" LastActivityDate="2021-02-16T02:41:58.700" Title="Deep learning models for classification, able to classify small image patches?" Tags="|deep-learning|classification|keras|convolutional-neural-network|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19099" PostTypeId="1" CreationDate="2017-05-20T21:40:08.787" Score="28" ViewCount="54999" Body="&lt;p&gt;I'm starting to learn Machine learning from Tensorflow website. I have developed a very very rudimentary understanding of the flow a deep learning program follows (this method makes me learn fast instead of reading books and big articles). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few confusing things that I have come across, 2 of them are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Bias&lt;/li&gt;&#xA;&lt;li&gt;Weight&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In the MNIST tutorial on tensorflow website, they have mentioned that we need bias and weight to find the evidence of the existence of a particular pattern in an image. What I don't understand is, where and how the values for Bias and Weight are determined? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you could provide some suggestions on how to accelerate my pace in deep learning, that would be great!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/beginners&quot; rel=&quot;noreferrer&quot;&gt;Tensorflow Beginners Tutorial&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32459" LastActivityDate="2021-03-06T14:50:36.663" Title="What is weight and bias in deep learning?" Tags="|machine-learning|deep-learning|tensorflow|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19101" PostTypeId="2" ParentId="19099" CreationDate="2017-05-21T07:16:18.930" Score="14" Body="&lt;p&gt;I agree with the comments on your question that you should look into a course, maybe &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;noreferrer&quot;&gt;Andrew Ng's Machine Learning on Coursera&lt;/a&gt;, which is a highly regarded, free introductory course. This is a basic question about fundamentals of machine learning. As such I am not covering the maths in this answer, you can get that from many places, including that course.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;where and how the values for Bias and Weight are determined?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Weights and biases are the learnable parameters of your model. As well as neural networks, they appear with the same names in related models such as linear regression. Most machine learning algorithms include some learnable parameters like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values of these parameters before learning starts are initialised randomly (this stops them all converging to a single value). Then when presented with data during training, they are adjusted towards values that have correct output.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You do not need to provide values before training, although you may want to decide things such as how many parameters there should be (in neural networks that is controlled by the size of each layer). TensorFlow calculates the values automatically, during training. When you have an already-trained model and want to re-use it, then you will want to set the values directly e.g. by loading them from file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The specific code that handles changes to weights and biases from the tutorial is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first line defines how the weights and values will be changed. You can read this almost literally as &quot;define a training function that uses the gradient descent optimizer to reduce the cross entropy of the supplied data&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second line invokes that function with a specific piece of data. Each time this second line is run, the weight and bias values are adjusted so that neural network outputs $y$ values a little bit closer to the correct association for each $x$ value.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-21T08:40:55.300" LastActivityDate="2017-05-21T08:40:55.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19102" PostTypeId="1" AcceptedAnswerId="19115" CreationDate="2017-05-21T13:02:29.050" Score="0" ViewCount="147" Body="&lt;p&gt;I scraped my Facebook chat history and wanted to try out some basic machine learning stuff with word2vec. However, the data has all sorts of stuff - links, emoji, Cyrillic alphabet, etc. Even if I manage to clear some of those, would it be possible to process the Cyrillic alphabet with word2vec?&lt;/p&gt;&#xA;" OwnerUserId="16764" LastActivityDate="2017-05-22T12:54:48.403" Title="Using word2vec with mixed language data" Tags="|machine-learning|word2vec|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19103" PostTypeId="2" ParentId="19040" CreationDate="2017-05-21T16:36:31.383" Score="5" Body="&lt;p&gt;For most clustering approaches, first you need to choose a similarity measure. Some common &lt;em&gt;default&lt;/em&gt; ones for raw time series are Euclidean distance and &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Time Warping&lt;/a&gt; (DTW).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have computed the similarity measure for every pair of time series, then you can apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;hierarchical clustering&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/K-medoids&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-medoids&lt;/a&gt; or any other clustering algorithm that is appropriate for time series (not &lt;a href=&quot;https://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means&lt;/a&gt;!, see &lt;a href=&quot;https://stats.stackexchange.com/a/131337/40048&quot;&gt;this&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: if the number of time series (along with their size) makes it computationally not acceptable to compute pairwise distances, then one option can be to extract features from each time series, and then use such features as proxies for the time series in the clustering process. Some examples of such features are maximum value, number of peaks, mean value. There are libraries like &lt;a href=&quot;https://github.com/blue-yonder/tsfresh&quot; rel=&quot;nofollow noreferrer&quot;&gt;tsfresh&lt;/a&gt; in Python that are meant to easily extract such kind of features from time series. With these features, then any clustering approach like &lt;a href=&quot;https://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means&lt;/a&gt; can be applied.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-11-07T09:18:16.527" LastActivityDate="2017-11-07T09:18:16.527" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19105" PostTypeId="1" CreationDate="2017-05-22T04:35:15.383" Score="3" ViewCount="253" Body="&lt;p&gt;Assume $D$ is the training data set with both the value of the predictors $\mathbf{X}$ and the value of the response variable $Y$. I have a loss function $L$ and two models $f(\mathbf{X};\beta)$ and $g(\mathbf{X};\lambda)$, where $\beta$ and $\lambda$ are model parameters. Our goal is to estimate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;e(f)=\mathbb{E}[L(Y,f(\mathbf{X};\beta))|D]~\mbox{and } e(g)=\mathbb{E}[L(Y,g(\mathbf{X};\lambda))|D]&#xA;\end{equation}&#xA;Note that it is the expectation of the generalization error of the model $f$ (and $g$) that is trained on the specific training data set $D$, where the expectation is taken based on the same distribution that generates $D$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if we do a leave-one-out procedure, specifically: let $N$ be the total number of observations in $D$, let $D_{-j}$ be the data set that removes the $j^{th}$ observation. Then,$L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an &quot;almost&quot; unbiased estimator of $e(f)$ right? Theoretically, to get $e(f)$, once should generate infinite new $(Y_i,\mathbf{X}^{(i)})$ from the distribution of $D$, then train the model on $D$ and use that model to make prediction on the new infinite data set and take the average. Now we fit the model on $D_{-j}$, which is only slightly different from $D$. So $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an &quot;almost&quot; unbiased estimator of $e(f)$. Then you go through all $N$ data points in $D$ to obtain the value of $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$, where $j=1,2,...,N$ (assume $N$ is large), then you take the average, then the result should be very close to $e(f)$ right? Then we do the same thing to model $g$. Then in this case, we get very good estimates of $e(f)$ and $e(g)$ so we can do model selection based on $e(f)$ and $e(g)$. Specifically, if $e(f)&amp;lt;e(g)$, then I should expect that on a LARGE new independent data set $T$, $f$ should perform better than $g$ correct? Also, the quantity $e(f)$ and $e(g)$ computed from $T$ should be very close to the quantity computed from $D$, assuming both $D$ and $T$ are large. Is that correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all the above are correct, then it seems that I did model selection and model assessment in one step. But should I partition the data set into 3 pieces, i.e., first train 2 models on one piece, then apply the 2 models on another piece to do model selection, then apply the 2 models on the 3rd piece to do model assessment?&lt;/p&gt;&#xA;" OwnerUserId="30170" LastEditorUserId="1330" LastEditDate="2021-10-26T14:08:49.257" LastActivityDate="2021-10-26T14:08:49.257" Title="Model selection and assessment using leave-one-out cross validation" Tags="|cross-validation|model-selection|mathematics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19106" PostTypeId="2" ParentId="19105" CreationDate="2017-05-22T05:36:22.587" Score="0" Body="&lt;p&gt;What you just said is dividing the data into three parts: training, validating and testing. &#xA;This is a very common practice in machine learning.&#xA;We use validation to help in selecting hyper parameters. &#xA;Going with this option vs. Leave one out vs. cross validation depends mainly on how many number of samples you have. If you have a lot of samples, then going with the option of splitting the data into three parts can be more efficient.&#xA;I am not sure about your statement of unbiased. You need to be careful when you make judgment &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-05-22T05:36:22.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19107" PostTypeId="1" AcceptedAnswerId="19298" CreationDate="2017-05-22T06:13:09.000" Score="4" ViewCount="3125" Body="&lt;p&gt;Of all the examples I've found for doc2vec training, the documents are uniquely labeled. What happens when many documents share the same label?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TaggedDocument of gensim accepts a list labels for the same text. It implies we can have multiple labels for the same text. However, it's not clear to me if it's a good practice to have fragmented texts under the same label. You can still train and get the embeddings. But are they good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the question I am posting here has a title, a detailed description, and a list of tags. How do I model it for doc2vec to find similar questions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that some of the tags are not in the title nor the description. What's the best way to include them in the doc2vec trainings. Shuffle them and concatenate with title and description? Or keep them as separate entries under the same label?&lt;/p&gt;&#xA;" OwnerUserId="15624" LastEditorUserId="15624" LastEditDate="2017-05-24T05:19:25.537" LastActivityDate="2017-05-29T11:32:33.210" Title="In doc2vec, how to model correctly when many documents share the same label?" Tags="|word2vec|word-embeddings|gensim|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19108" PostTypeId="1" AcceptedAnswerId="19122" CreationDate="2017-05-22T08:51:18.527" Score="1" ViewCount="1795" Body="&lt;p&gt;I want to start with machine learning with a small prediction problem but I'm not sure I chose the right approach. I want to make a program that gets data of mechanical failures on cars (manufactured time, failure time, reason, and different characteristics of the car). Then I would give the data of new cars that will be released to market and I would try to predict when would they fail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was reading that the best approach is using survival analysis with R but since I'm not really familiar with this algorithm, I was wondering If there's any other approach.&lt;/p&gt;&#xA;" OwnerUserId="32490" LastEditorDisplayName="user380" LastEditDate="2017-05-29T05:00:07.730" LastActivityDate="2020-03-17T09:53:08.593" Title="Predicting car failures with machine learning" Tags="|machine-learning|r|prediction|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19112" PostTypeId="1" CreationDate="2017-05-22T10:29:21.267" Score="14" ViewCount="1217" Body="&lt;p&gt;In Decision Trees, we can understand the output of the tree structure and we can also visualize how the Decision Tree makes decisions. So decision trees have explainability (their output can be explained easily.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do we have explainability in Neural Networks like with Decision Trees?&lt;/p&gt;&#xA;" OwnerUserId="30056" LastEditorUserId="8501" LastEditDate="2018-05-08T01:27:32.780" LastActivityDate="2022-01-15T16:42:18.043" Title="Do neural networks have explainability like decision trees do?" Tags="|neural-network|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="19113" PostTypeId="2" ParentId="19112" CreationDate="2017-05-22T10:43:30.187" Score="7" Body="&lt;p&gt;No. Neural network is generally difficult to understand. You trade predictive power for model complexity. While it's possible to visualize the NN weights graphically, they don't tell you exactly how a decision is made. Good luck trying to understanding a deep network.&lt;/p&gt;&#xA;&lt;p&gt;There is a popular Python package (and it has a paper) that can model a NN locally with a simpler model. You may want to &lt;a href=&quot;https://github.com/marcotcr/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;take a look&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="98307" LastEditDate="2020-08-05T16:11:57.993" LastActivityDate="2020-08-05T16:11:57.993" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="19114" PostTypeId="2" ParentId="19112" CreationDate="2017-05-22T12:37:42.010" Score="9" Body="&lt;p&gt;I disagree with the previous answer and with your suggestion for two reasons:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Decision trees are based on simple logical decisions which combined together can make more complex decisions. &lt;strong&gt;BUT&lt;/strong&gt; if your input has 1000 dimensions, and the features learned are highly non linear, you get a really big and heavy decision tree which you won't be able to read/understand just by looking at the nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Neural networks are similar to that in the sens that the function they learn is understandable only if they are very small. When getting big, you need other tricks to understand them. As @SmallChess suggested, you can read this &lt;a href=&quot;https://arxiv.org/pdf/1311.2901.pdf&quot; rel=&quot;noreferrer&quot;&gt;article&lt;/a&gt; called &lt;em&gt;Visualizing and Understanding Convolutional Networks&lt;/em&gt; which explains for the particular case of convolutional neural networks, how you can read the weights to understand stuff like &quot;it detected a car in this picture, mainly because of the wheels, not the rest of the components&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These visualizations helped a lot of researchers to actually understand weaknesses in their neural architectures and helped to improve the training algorithms.    &lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-05-22T12:43:18.433" LastActivityDate="2017-05-22T12:43:18.433" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19115" PostTypeId="2" ParentId="19102" CreationDate="2017-05-22T12:54:48.403" Score="1" Body="&lt;p&gt;Word2Vec uses the bag-of-word model as input which means that you can use whatever alphabet. Building a bag of word just consists of attributing one feature per word in the entire dictionary of your corpus. You can do this with a cyrillic alphabet as with other alphabet. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-05-22T12:54:48.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19116" PostTypeId="1" AcceptedAnswerId="19121" CreationDate="2017-05-22T13:44:38.583" Score="0" ViewCount="61" Body="&lt;p&gt;I am totally new to Artificial Neural Networks. Let’s say that the model you are trying to turn into an artificial neural network has an output that is triggered &lt;em&gt;only&lt;/em&gt; by the exceedance of a threshold: $y\geq y_{1}. $Therefore, you need to find a way to use this inequality as an activation function. Is this feasible?&lt;/p&gt;&#xA;" OwnerUserId="13481" LastActivityDate="2017-05-22T19:38:33.547" Title="How can you map the exceedance of a threshold into an activation function of a Neural Network?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19119" PostTypeId="1" CreationDate="2017-05-22T17:26:48.557" Score="0" ViewCount="354" Body="&lt;p&gt;I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my client. My biggest concern at this point is whether my models are reproducible or not. I understand that LDA models are probabilistic, so separate models run on the same data under the same parameters are not guaranteed to be identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for a model to be considered reliable you would expect duplicate models to come out looking roughly the same. A more reproducible model would be a more accurate representation of the text it is modeling. I wrote a script to find the average similarity between two distinct models (found here: &lt;a href=&quot;https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models&lt;/a&gt;). The idea behind it is that I try to force a one to one relationship between two models' topics, and then find the average similarity of each match. The assumption is that a reproducible model would have a clear bijection between a duplicate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my early research it seems like training a model for longer increases the similarity of duplicate models. (Models trained under 500 iterations were more similar than those trained under 150 passes). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm wondering is if there's been any papers or studies done on the reproducibility of LDA models, or if anyone has any ideas. I do not want to fix a seed. I don't want to be able to output an identical model--I want to be confident that for any model I create, a model trained under the same data and parameters will be almost the same. &lt;/p&gt;&#xA;" OwnerUserId="32508" LastActivityDate="2017-05-26T13:43:42.453" Title="Quantifying the Reproducibility of LDA Models" Tags="|topic-model|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19121" PostTypeId="2" ParentId="19116" CreationDate="2017-05-22T19:38:33.547" Score="1" Body="&lt;p&gt;This is feasable. This is also called a &lt;code&gt;Binary/Step&lt;/code&gt; activation function. You must only use this activation function on the output neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;Step&lt;/code&gt; function will round down an answer that is lower than 0.5 to &lt;code&gt;0&lt;/code&gt;, and an answer that is higher than 0.5 to &lt;code&gt;1&lt;/code&gt;. However, please note that you do not need to use a binary activation function to output 1 - I advise you to just use TanH or sigmoid and backpropagate a whole bunch of iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in other comments, you mentioned that you know what &lt;code&gt;y1&lt;/code&gt; is. &lt;em&gt;That is not of importance&lt;/em&gt;, the network will act as a black box and will figure out a treshold itself. Don't set up your own activation function just to get the right output - that avoids the whole point of backpropagation.&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-05-22T19:38:33.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19122" PostTypeId="2" ParentId="19108" CreationDate="2017-05-22T19:57:55.743" Score="0" Body="&lt;p&gt;I'm also just a beginner in ML (who is however not familiar with survival analysis w/ R), but has tackled a couple of ML projects. Based on my knowledge, you could use &lt;strong&gt;supervised learning&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Store data, preferably in CSV format, (one column about the duration between buying the car and the car's mechanical breakdown), and the rest about the car's data/characteristics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, you can run a neural network through your data, and use your NN's library's predict() method to predict the duration before breakdown based on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could then theoretically (assuming that there is a logical correlation between the data) see which characteristics are most prone to make a car break down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for implementing your program, I use &lt;strong&gt;Python&lt;/strong&gt; with the &lt;strong&gt;Keras&lt;/strong&gt; library, which is simple enough for any programmer to use, but there exist many other great ML libraries, notably &lt;strong&gt;TensorFlow&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do note that I am also just a beginner, and that my approach might be erroneous, yet I do wish you good luck on your future ML projects!&lt;/p&gt;&#xA;" OwnerUserId="32511" LastActivityDate="2017-05-22T19:57:55.743" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19124" PostTypeId="1" CreationDate="2017-05-22T21:00:19.033" Score="12" ViewCount="11859" Body="&lt;p&gt;I hope the following excerpts will provide an insight into what my question is going to be. These are from &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting &amp;quot;better&amp;quot;. But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost.&lt;/p&gt;&#xA;&lt;p&gt;Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this.&lt;/p&gt;&#xA;&lt;p&gt;We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a  better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?&lt;/p&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="98307" LastEditDate="2020-08-05T17:27:17.537" LastActivityDate="2020-08-05T17:27:17.537" Title="How to know the model has started overfitting?" Tags="|neural-network|overfitting|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="19125" PostTypeId="1" AcceptedAnswerId="19126" CreationDate="2017-05-22T22:48:48.713" Score="0" ViewCount="2225" Body="&lt;p&gt;I want to calculate the frequency of the words in obama['text'] (obama is the variable where i have stored this series element ) in a dictionary  and store it in another column . Without using Counter library , how do i do that . The data is in this format :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; URI                                      |              name                |  text &#xA;&amp;lt;http://dbpedia.org/resource/Barack_Obama&amp;gt;           Barack Obama               barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureateduring his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output should be in the format in a new column obama['word count']:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{ 2009:4 , the :40 , chicago :10&lt;br&gt;&#xA;and so on  } &lt;/p&gt;&#xA;" OwnerUserId="32510" LastActivityDate="2017-05-23T00:39:49.417" Title="Count the frequncy of words in a cell of a column in a series" Tags="|machine-learning|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19126" PostTypeId="2" ParentId="19125" CreationDate="2017-05-22T23:22:20.587" Score="1" Body="&lt;p&gt;Why shouldn't you use the Counter class; it's exactly what you need?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import Series&#xA;from collections import Counter&#xA;text=&quot;barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureate during his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief&quot;&#xA;df = Series(text).to_frame()&#xA;newdf = df.assign(word_count = lambda x: x[0].str.split(' ').apply(Counter)[0])&#xA;newdf['word_count']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0    {'44th': 1, 'born': 2, 'november': 1, 'running...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-05-23T00:39:49.417" LastActivityDate="2017-05-23T00:39:49.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19127" PostTypeId="1" AcceptedAnswerId="19320" CreationDate="2017-05-23T00:04:38.397" Score="1" ViewCount="11207" Body="&lt;p&gt;I'm trying to use doc2vec(gensim) to identify the most similar sentence and get its label. That is, for example, when the data is composed of 36 types of TVs (each sentence explains a specific product and its labeled to that product), the doc2vec categorizes the user input and decides what TV the user is referring to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I only know how to get the most similar word: model.most_similar('red/noun')&#xA;How can you, instead of words, get the most similar sentence and its label?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/10216/doc2vec-how-to-label-the-paragraphs-gensim&quot;&gt;Doc2Vec - How to label the paragraphs (gensim)&lt;/a&gt; (this tells that the above method is actually possible in doc2vec)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you :)&lt;/p&gt;&#xA;" OwnerUserId="32521" LastActivityDate="2017-12-03T16:29:27.543" Title="Doc2vec(gensim) - How to calculate the most similar sentence and get its label?" Tags="|machine-learning|text-mining|similarity|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19129" PostTypeId="1" AcceptedAnswerId="19686" CreationDate="2017-05-23T03:03:01.957" Score="1" ViewCount="248" Body="&lt;p&gt;I am annotating a new corpus with BRAT. I have a set of &lt;code&gt;.txt&lt;/code&gt; files to annotate. Is it possible to configure BRAT so that if no &lt;code&gt;.ann&lt;/code&gt; file is found, then an empty &lt;code&gt;.ann&lt;/code&gt; file is created? Or am I supposed to myself provide the empty &lt;code&gt;.ann&lt;/code&gt; files?&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-06-13T21:19:54.150" Title="Configure BRAT so that if no `.ann` file is found, then an empty `.ann` file is created" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19131" PostTypeId="1" CreationDate="2017-05-23T06:12:28.017" Score="0" ViewCount="3634" Body="&lt;p&gt;I have created a NMF topic model in python the code snippet for which is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def select_vectorizer(req_ngram_range=[1,2]):&#xA;    ngram_lengths = req_ngram_range&#xA;    vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(ngram_lengths), stop_words='english', min_df=2)&#xA;    #print(&quot;User specified custom stopwords: {} ...&quot;.format(str(custom_stopwords)[1:-1]))&#xA;    return vectorizer&#xA;&#xA;vectorizer = select_vectorizer([2,5])&#xA;X = vectorizer.fit_transform(new_review_list)&#xA;&#xA;&#xA;clf = decomposition.NMF(n_components=20, random_state=3, alpha = .1).fit(X)&#xA;vocab = vectorizer.get_feature_names()&#xA;print_top_words(clf, vocab, num_top_words)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which created 20 topics like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Topic #0:&#xA;[u'blocks available', u'delivery blocks available', u'notifications blocks', u'notifications blocks available', u'new blocks', u'know blocks available', u'new blocks available', u'know blocks', u'open blocks available', u'available work', u'zero blocks', u'like blocks', u'notification blocks', u'day blocks', u'slow blocks', u'10 blocks', u'option set', u'logged 10', u'notification blocks available', u'day blocks available']&#xA;Topic #1:&#xA;['something', 'something'] and so on..&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to supply new text data to this fitted NMF model such that the model tells us which topic this new text belongs to. I have searched a lot, somebody suggested using &quot;transform&quot;, but I do not know how to use it. For eg: if I have a new text, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_feedback = &quot;This is a new tester feedback&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then how do I find which topic out of the 20 topics created earlier, does this &lt;code&gt;new_feedback&lt;/code&gt; fall into? Please help!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE: On trying to do what's mentioned in the below comment, I receive the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.transform(vectorizer.transform(new_feedback))&#xA;  File &quot;/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py&quot;, line 1380, in transform&#xA;    X = super(TfidfVectorizer, self).transform(raw_documents)&#xA;  File &quot;/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py&quot;, line 884, in transform&#xA;    &quot;Iterable over raw text documents expected, &quot;&#xA;ValueError: Iterable over raw text documents expected, string object received.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32248" LastEditorUserId="32248" LastEditDate="2017-05-23T16:46:07.023" LastActivityDate="2017-05-24T15:58:51.767" Title="How to use Non Negative Matric Factorization (NMF)'s transform method to project topics on new text data" Tags="|python|nlp|scikit-learn|topic-model|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19132" PostTypeId="2" ParentId="19087" CreationDate="2017-05-23T06:17:41.387" Score="0" Body="&lt;p&gt;The problem is solved if I let &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; do the job:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tfidf_vec = TfidfVectorizer(vocabulary=dictionaryList, use_idf=False)&#xA;final_vec = tfidf_vec.fit_transform(df[DF_X_DATA])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;use_idf=False&lt;/code&gt; is the solution. It lets it calculate frequency automatically. I don't have to manually divide by the sample's length this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results now are ~90% and the learning curve looks normal.&lt;/p&gt;&#xA;" OwnerUserId="32433" LastActivityDate="2017-05-23T06:17:41.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19135" PostTypeId="1" CreationDate="2017-05-23T07:41:52.600" Score="2" ViewCount="163" Body="&lt;p&gt;So I trained CNN model for people detection on caltech-pedestrian dataset:&#xA;Then I was curious and evaluated the model in every 1000th iteration on Evaluation toolbox(I guarantee, there is no bug in evaluation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, plot of the performance does not look so good. The miss rate spikes between 20K(20,000) and 30K iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am confused what does that mean. I mean usually we would expect the miss rate to decrease as we train the model more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using yolo object detection network &lt;a href=&quot;https://github.com/AlexeyAB/darknet/blob/master/cfg/tiny-yolo-voc.cfg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/AlexeyAB/darknet/blob/master/cfg/tiny-yolo-voc.cfg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So learning options are the same as &lt;a href=&quot;https://github.com/AlexeyAB/darknet/blob/master/src&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/AlexeyAB/darknet/blob/master/src&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SPj98.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SPj98.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried 10x smaller learning rate.&#xA;However, no luck:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/FV77c.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FV77c.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32535" LastEditorUserId="29169" LastEditDate="2021-02-16T02:41:45.443" LastActivityDate="2021-08-04T11:08:00.717" Title="What does it mean if performance of two different iterations of the same network (CNN model) varies a lot?" Tags="|machine-learning|convolutional-neural-network|object-recognition|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19136" PostTypeId="1" CreationDate="2017-05-23T08:41:14.777" Score="1" ViewCount="50" Body="&lt;p&gt;For a sub problem I need a network that produces a single continuous number $y$, however during training I have more information available about the output, the output is a weighted sum between a 4 other numbers $y_{1..4}$, the weights are known and constant. All of $y_{1..4}$ are dependent on the input $x$ and I feel like incorporating this knowledge into the network architecture might be beneficial for training. My current idea is to have a number of shared layers, then have 4 individual intermediate layers that all regress to the specific $y_{1..4}$ and then have $y=\sum_{i=1}^4a_iy_i$, with $a_i$ being the weight for $y_i$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is about a good loss function, I'm interested in regressing the mean so using $L_2$ loss makes sense but only doing it at the end doesn't incorporate the prior knowledge. You could add all the $L_2$ losses for the individual $y_i$, they should probably be weighted by how much weight they have in the total sum. However then we don't directly optimize $y$, sometimes the network could be more biased than normal by making a mistake in the same direction for all $y_{1..4}$. My idea is now to do it on the individual $y_i$ but also on $y$, weighing the sum of the $L_2$ of the individual $y_i$ equally to $L_2$ of $y$. The weight between these two parts of the loss function could be different and I guess is a hyperparameter of the model. My question is if there are any papers that have worked with a similar idea to see which of the options work better or if there is another approach to do this. Splitting the value and advantage streames in deep reinforcement learning is what inspired me to look at this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: The goal is indeed to get the lowest $L_2$ loss on the final $y$, but I feel like the network should be able to learn better because I have more information with regards to how $y$ is built up. My question is how to best incorporate this additional information, I know I can test some different scenarios but was wondering if there is any literature about similar approaches before implementing things that end up inferior.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-05-23T09:57:37.447" LastActivityDate="2017-05-23T09:57:37.447" Title="Decomposable output regression neural network" Tags="|neural-network|deep-learning|regression|loss-function|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19137" PostTypeId="1" AcceptedAnswerId="19173" CreationDate="2017-05-23T09:30:01.837" Score="1" ViewCount="82" Body="&lt;p&gt;I'm asking here to get some advice. My goal is to detect similar patterns in an n-dimensional data. For example (extreme simple) you have two axes: X-Axis is Time (in seconds) and Y-Axis is Power (in Watt). So you get time-series with a specific pattern. My goal is to select a subset of the time-series and calculate the similarity of other input data to the selected subset. &#xA;In the end, I want to define multiple patterns and the output predicts which pattern comes most closely to the given input data.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it suitable to use Machine Learning e.q. pattern recognition for that problem or is it a pure overkill? (I got the advice to use ML; also under the condition that I have an generic n-dimensional data)&lt;/li&gt;&#xA;&lt;li&gt;If ML is suitable for this problem, can you give me some advice where I could start? I don't ask for a solution in any way, I just want to have some suggestions in which direction I should look, because ML is a very new topic for me and I don't have much experience in practice.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="32543" LastEditorUserId="8878" LastEditDate="2017-05-24T08:01:41.257" LastActivityDate="2017-05-24T08:56:57.770" Title="Similarity between an n-dimensional curve and selected subsets of them" Tags="|machine-learning|clustering|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19138" PostTypeId="1" CreationDate="2017-05-23T09:43:58.043" Score="0" ViewCount="60" Body="&lt;p&gt;Assume that I have a single feature dataset X comprising of say a 1000 samples.  The feature is an integer that may take up any value upto 2^32. Among these, one particular value alone is considered malicious. The target/lable vector flags out the malicious value as 1 and others as 0.  What would be the right approach to train a model that could learn this single value and classify it as 1 and all else as 0? &lt;/p&gt;&#xA;" OwnerUserId="32542" LastActivityDate="2017-06-22T10:52:38.817" Title="Supervised learning: classifying/learning a single discrete value" Tags="|classification|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19139" PostTypeId="2" ParentId="19138" CreationDate="2017-05-23T10:23:23.333" Score="2" Body="&lt;p&gt;I don't really see this as a learning problem really, if it's the &lt;em&gt;same malicious value&lt;/em&gt; you have to watch out for..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only one &quot;positive&quot; sample (corresponding to the malicious value) would be enough to find the malicious integer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have I misunderstood something in your question?&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-05-23T10:23:23.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19140" PostTypeId="1" CreationDate="2017-05-23T10:33:11.993" Score="2" ViewCount="5603" Body="&lt;p&gt;I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, which fold should I use? Is averaging all 5 of the folds the best choice? Secondly, why is the data-set split ratio (70/30) different in 5 fold Cross validation and 10 fold cross validation? Shouldn't it be the same?&lt;/p&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="29169" LastEditDate="2019-06-07T04:25:37.180" LastActivityDate="2020-03-04T13:22:39.333" Title="K-Fold Cross validation confusion?" Tags="|classification|cross-validation|matlab|sampling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="19141" PostTypeId="1" CreationDate="2017-05-23T11:24:23.360" Score="2" ViewCount="839" Body="&lt;p&gt;This is related to Sports prediction (Cricket). I am new to Machine Learning and learning it through TensorFlow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am focusing only on one topic which is &lt;strong&gt;&quot;How many runs &lt;code&gt;Player A&lt;/code&gt; will score in today's match?&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My raw data includes all cricket player's score in all matches. It also includes more finer details like the ground the match happened, day/night match, opposition team etc., I need to use this data and &lt;strong&gt;predict four possible ranges&lt;/strong&gt; for a player's score with &lt;strong&gt;all 4 range having equal probability&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am planning to keep 4 simple &lt;code&gt;FeatureColumn&lt;/code&gt; to start with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Average of runs of the given player in the past 10 matches (relative to the match that is being trained)&lt;/li&gt;&#xA;&lt;li&gt;Batting order (1 to 10) (in the match being trained)&lt;/li&gt;&#xA;&lt;li&gt;Average of runs since debut (relative to the match that is being trained)&lt;/li&gt;&#xA;&lt;li&gt;Average no. of balls faced in the past 10 matches (relative to the match that is being trained)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My actual output will be the runs scored by that player in the match being trained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can consider 100 recent matches of a player for this training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tutorial &lt;a href=&quot;https://www.tensorflow.org/tutorials/wide&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/wide&lt;/a&gt; from TensorFlow seems to match with what I am approaching. But the final output is a binary value (whether salary is &gt;50K or not). But I need to generate output with probability of a given player getting a run. For eg.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Run     | Probability of getting this run&#xA;-----------------------------------------&#xA;0 runs  :  0.01&#xA;1 run   :  0.01&#xA;2 runs  :  0.02&#xA;....&#xA;13 runs :  0.5&#xA;14 runs :  0.04&#xA;....&#xA;56 runs :  0.08&#xA;57 runs :  0.03&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sum of Probability should add up to 1.&#xA;I'll then group all these values into 4 ranges with each range having an equal probability (0.25 for each range) like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YwaXe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YwaXe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which approach can I use for solving this problem? I read about Linear regression and that seems to be the best fit for me, but I am not clear on how to generate probability of all possible outcomes.&lt;/p&gt;&#xA;" OwnerUserId="32546" LastEditorUserId="32546" LastEditDate="2017-06-03T23:38:20.400" LastActivityDate="2019-04-13T00:03:09.193" Title="Linear Model to generate probability of each possible output" Tags="|python|predictive-modeling|tensorflow|probability|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19143" PostTypeId="1" CreationDate="2017-05-23T11:35:44.843" Score="4" ViewCount="4886" Body="&lt;p&gt;I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean and median are not the solution here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to research other methods for that but none of them works since I have many categorical variables. I also tried to use one hot encoding to convert categorical variables to integers but I am not sure if that is a solution in my case since from only 1 categorical column I would get 600 new columns. If I do the same with other categorical columns, I would get many millions of new columns. One of the categorical columns is URL string and it is different for every row, so I have 10 million different URLs for example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other categorical column is a description and it is also different for every row. I could probably remove the URL column, but I can't remove description, title, location and others for example. I tried PCA, but it also doesn't work with categorical data. I have missing data for both categorical and integers/floats values. Would get_dummies method be a good approach to deal with this? For missing values imputation I tried KNN and maximum likelihood but I am getting errors due to categorical variables. Missing data is completely randomly missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions how to approach this problem and also which packages should I use? &lt;/p&gt;&#xA;" OwnerUserId="32550" LastEditorUserId="-1" LastEditDate="2017-05-25T05:00:08.423" LastActivityDate="2023-10-14T12:11:44.533" Title="Imputation of missing values and dealing with categorical values" Tags="|python|scikit-learn|pandas|categorical-data|data-imputation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19144" PostTypeId="1" CreationDate="2017-05-23T11:41:10.857" Score="1" ViewCount="67" Body="&lt;p&gt;I am looking at &lt;a href=&quot;https://arxiv.org/pdf/1609.08227.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;density trees&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition about the y-axis is clear: the tree indicates the modes which then merge at &lt;em&gt;merge height&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_p(x,y) = \sup{t: \exists C \in \textit{C} \quad s.t. \quad x,y \in C }$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of execution, everything goes through the use of a density estimator, as clearly described in the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am however not sure about what goes in the x-axis.&lt;a href=&quot;https://i.stack.imgur.com/gM4IE.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gM4IE.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that the x-axis ranges between 0 and 1 makes me suspect that the x-axis is somehow related with the distribution of the probability; but I am missing the details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also checked the paper which is the &lt;a href=&quot;https://arxiv.org/pdf/1605.06416.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;original source of the picture above&lt;/a&gt;. There, the 1-dimensional case is somehow covered: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/XJ7YT.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XJ7YT.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;. But I am still missing the details for n-dimensional cases, such as the Yingyang data.&lt;/p&gt;&#xA;" OwnerUserId="1244" LastActivityDate="2017-06-07T13:36:06.460" Title="Density Tree - What is the x axis?" Tags="|clustering|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19146" PostTypeId="1" AcceptedAnswerId="19152" CreationDate="2017-05-23T13:34:21.700" Score="2" ViewCount="1068" Body="&lt;p&gt;I want to choose the &quot;optimal&quot; hyperparameters for gbm. So I run the following code using the &lt;code&gt;h2o&lt;/code&gt; package&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create hyperparameter grid&#xA;hyper_params = list(ntrees = c(10,20,50,100,200,500),&#xA;                    max_depth = c(5,10,20,30),&#xA;                    min_rows = c(5,10,20,30),&#xA;                    learn_rate = c(0.01,0.05,0.08,0.1),&#xA;                    balance_classes=c(T,F))&#xA;&#xA;# random subset of hyperparameters&#xA;search_criteria = list(strategy = &quot;RandomDiscrete&quot;, max_models = 192, seed = 42,&#xA;                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = &quot;mean_per_class_error&quot;)&#xA;&#xA;# run grid&#xA;gbm.grid &amp;lt;- h2o.grid(&quot;gbm&quot;, grid_id = &quot;gbm.grid&quot;, x = names(td.train.h2o)[!names(td.train.h2o)%like%&quot;s_sd_segment&quot;], y = &quot;s_sd_segment&quot;, &#xA;                     seed = 42, distribution = &quot;multinomial&quot;,&#xA;                     training_frame = td.train.hyper.h2o, nfolds = 3,&#xA;                     hyper_params = hyper_params, search_criteria = search_criteria)&#xA;&#xA;# sort results&#xA;gbm.sorted.grid &amp;lt;- h2o.getGrid(grid_id = &quot;gbm.grid&quot;, sort_by = &quot;mean_per_class_error&quot;, decreasing=FALSE)&#xA;&#xA;hyperparameters_gbm &amp;lt;- as.data.table(gbm.sorted.grid@summary_table)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives as optimal combination for the hyperparameters &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;learn_rate  max_depth   min_rows    ntrees&#xA;0.08    10  5   200&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I am trying to do the same but with different &lt;code&gt;stopping_metric&lt;/code&gt;. So in the above i use &lt;code&gt;mean_per_class_error&lt;/code&gt;, and in the following i use &lt;code&gt;logloss&lt;/code&gt;, so i run the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create hyperparameter grid&#xA;hyper_params = list(ntrees = c(10,20,50,100,200,500),&#xA;                    max_depth = c(5,10,20,30),&#xA;                    min_rows = c(5,10,20,30),&#xA;                    learn_rate = c(0.01,0.05,0.08,0.1),&#xA;                    balance_classes=c(T,F))&#xA;&#xA;# random subset of hyperparameters&#xA;search_criteria = list(strategy = &quot;RandomDiscrete&quot;, max_models = 192, seed = 42,&#xA;                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = &quot;logloss&quot;)&#xA;&#xA;# run grid&#xA;gbm.grid &amp;lt;- h2o.grid(&quot;gbm&quot;, grid_id = &quot;gbm.grid&quot;, x = names(td.train.h2o)[!names(td.train.h2o)%like%&quot;s_sd_segment&quot;], y = &quot;s_sd_segment&quot;, &#xA;                     seed = 42, distribution = &quot;multinomial&quot;,&#xA;                     training_frame = td.train.hyper.h2o, nfolds = 3,&#xA;                     hyper_params = hyper_params, search_criteria = search_criteria)&#xA;&#xA;# sort results&#xA;gbm.sorted.grid &amp;lt;- h2o.getGrid(grid_id = &quot;gbm.grid&quot;, sort_by = &quot;logloss&quot;, decreasing=FALSE)&#xA;&#xA;hyperparameters_gbm &amp;lt;- as.data.table(gbm.sorted.grid@summary_table)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives as optimal combination for hyperparameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;learn_rate  max_depth   min_rows    ntrees&#xA;0.1 20  5   500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that i use as argument &lt;code&gt;strategy = &quot;RandomDiscrete&quot;&lt;/code&gt;, but still for instance the optimal combination for gbm using &lt;code&gt;stopping_metric = &quot;mean_per_class_error&quot;&lt;/code&gt; is the &quot;50th optimal combination&quot; for the gbm using &lt;code&gt;stopping_metric = &quot;logloss&quot;&lt;/code&gt;, and the &quot;2nd optimal combination&quot; for gbm using &lt;code&gt;stopping_metric = &quot;logloss&quot;&lt;/code&gt;, is the &quot;14th optimal combination&quot; for gbm using &#xA;&lt;code&gt;stopping_metric = &quot;mean_per_class_error&quot;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why could that happen ?&lt;/p&gt;&#xA;" OwnerUserId="19212" LastActivityDate="2017-05-23T17:58:28.000" Title="h2o, different stopping metric leads to different optimal for hyperparameters" Tags="|machine-learning|r|random-forest|optimization|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19150" PostTypeId="1" AcceptedAnswerId="19169" CreationDate="2017-05-23T15:43:30.287" Score="2" ViewCount="1265" Body="&lt;p&gt;I'm developing a project on .net (using web API and MVC). now I have to add another module which users can make queries of Data we have in the DB and retrieve the relevant answer.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is their a easy way to Do NLP developments in .net&lt;/li&gt;&#xA;&lt;li&gt;If yes, It would be a great help if someone can give a brief idea on how to get started&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;any suggestion is greatly appreciated. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="32554" LastActivityDate="2019-05-25T04:03:50.820" Title="Natural Language Processing (NLP) in .net environment" Tags="|nlp|.net|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19151" PostTypeId="2" ParentId="19143" CreationDate="2017-05-23T15:54:37.133" Score="2" Body="&lt;p&gt;I think like @El Burro suggested, you I believe you should focus on &lt;strong&gt;feature transformation&lt;/strong&gt; mainly. Use different techniques for different features. For straightforward features, such as occupation or gender for example, use one-hot encoding, while for others  you can use some kind of hierarchical mapping-clustering (e.g. map values to groups defined by you, for example if those urls linked to products, make 30 different groups with similar types of products and map the urls to these groups. Then you can use again one-hot encoding for these mapped features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the textual features you mentioned, I'm pretty confident you could drop some of them. If you really want to exploit some kind of information generated from text though, for starters you could either do some &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;tf-idf feature extraction&lt;/strong&gt;&lt;/a&gt;, so as to generate features for each text-snippet or consider &lt;strong&gt;topic modeling&lt;/strong&gt; (take a look at &lt;a href=&quot;http://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim for python if interested&lt;/a&gt;) and represent each text as a mixture of topics.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-05-23T15:54:37.133" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19152" PostTypeId="2" ParentId="19146" CreationDate="2017-05-23T17:58:28.000" Score="4" Body="&lt;p&gt;First of all, you are using different metrics to determine how well you are doing, that means it's not weird that different metrics find different hyperparameter settings that work better. Second of all, some hyperparameters might not matter for the problem you are solving, which means all the signal you are getting from those hyperparameters is noise. Third of all, most of the machine learning algorithms are stochastic, meaning there is randomness involved in training them and sometimes in evaluating them, this means even starting the same grid or random search could lead to different hyperparameters. That said the probability of that is only high if the real performances are close to each other.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-23T17:58:28.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19153" PostTypeId="1" CreationDate="2017-05-23T19:11:58.487" Score="2" ViewCount="1367" Body="&lt;p&gt;I'm doing unsupervised learning (clustering and DR) on multinomial time series. I need to reduce dimensions for my data, which is sparse and has a lot of dimensions. I realized that some form of tensor decomposition may be the best way to go about this. What I have is a set of samples each with a set of features in time. Each time length can different, but there are the same features for each sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Want to end up with is a set of samples with a single feature in time. With that, I can do clustering using distance determined by dynamic time warping. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this entire scheme sound reasonable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TensorFlow has a feature &lt;em&gt;tf.qr&lt;/em&gt; (&lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/qr&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/api_docs/python/tf/qr&lt;/a&gt;) which can do tensor decomposition, but I don't understand the output or math behind it well enough to utilize it. I have tried running it with what seems like success, but I don't understand what is stored in the &lt;em&gt;q&lt;/em&gt; and &lt;em&gt;r&lt;/em&gt; output tensors. If someone could explain this to me and if this even applies to what I am trying to do, I would really appreciate that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, I tried using scikit tensor's &lt;em&gt;cp&lt;/em&gt; decomposition (&lt;a href=&quot;https://github.com/mnick/scikit-tensor/blob/master/sktensor/cp.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/mnick/scikit-tensor/blob/master/sktensor/cp.py&lt;/a&gt;) which  returns a single tensor. I also do not understand what is stored in this output or if it does something similar to TensorFlow's tf.qr function or related to what I want to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can probably tell by now, I am quite lost in the math. I've tried to find explanations of how these functions work, but I can't without jumping into multilinear algebra which I don't understand. If someone could point me in the right direction, I would greatly appreciate it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if &lt;em&gt;s1&lt;/em&gt; and &lt;em&gt;s2&lt;/em&gt; are two multidimensional time series, then this should computer the distance between them. I tried it in python and it does in fact output a number, but I don't know if I am understanding this correctly. It behaves in the same way DTW would with one dimension but takes the sum of the MSE of all features instead of just one of them when computing the distance at a given point in time. Also, this DTW function has a time window which is set to 10 but of course can change...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def MDTWDistance(s1, s2, window=10, num_columns=1):&#xA;&#xA;    DTW={}&#xA;    w = max(window, abs(len(s1)-len(s2)))&#xA;&#xA;    for i in range(-1,len(s1)):&#xA;        for j in range(-1,len(s2)):&#xA;            DTW[(i, j)] = float('inf')&#xA;    DTW[(-1, -1)] = 0&#xA;&#xA;    for i in range(len(s1)):&#xA;        for j in range(max(0, i-w), min(len(s2), i+w)):&#xA;            dist= mdist(s1.loc[i], s2.loc[j], num_columns)&#xA;            print i, j, dist&#xA;            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])&#xA;&#xA;    return np.sqrt(DTW[len(s1)-1, len(s2)-1]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With... &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def mdist(a, b, num_col):&#xA;&#xA;    dist = 0&#xA;    for col in range(num_col):&#xA;        dist = dist + (a[col]-b[col])**2&#xA;        print a[col], b[col]&#xA;    return dist&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this doing what I think it does/what the papers say is appropriate? &lt;/p&gt;&#xA;" OwnerUserId="31676" LastEditorUserId="31676" LastEditDate="2017-05-25T01:48:22.567" LastActivityDate="2017-09-22T03:53:13.567" Title="Tensor Decomposition in TensorFlow for multinomial time series dimensionality reduction" Tags="|python|tensorflow|dimensionality-reduction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19155" PostTypeId="1" CreationDate="2017-05-23T22:33:11.657" Score="5" ViewCount="953" Body="&lt;p&gt;I have some JSON data to be transformed to a machine-learning friendly format. Every object in my data, which will eventually become an &lt;strong&gt;instance&lt;/strong&gt; in my dataset, has the exact same fields (in this case, &lt;code&gt;foo&lt;/code&gt;, &lt;code&gt;bar&lt;/code&gt; and &lt;code&gt;array&lt;/code&gt;). The &lt;code&gt;array&lt;/code&gt; field contains a &lt;strong&gt;variable number of sub-objects&lt;/strong&gt; (from 0 to 10). &lt;strong&gt;Each&lt;/strong&gt; of these sub-objects has &lt;strong&gt;one categorical field&lt;/strong&gt; with a massive range and some other fields which can be safely ignored. All categorical fields in these sub-objects belong to the &lt;strong&gt;same range&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example (massively simplified object):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;&quot;foo&quot;: 1&#xA;&quot;bar&quot;: 0.5&#xA;&quot;array&quot; : [&#xA;    {&#xA;        &quot;categorical&quot;: &quot;Lorem&quot;&#xA;        &quot;other&quot;: 34&#xA;        &quot;stuff&quot;: 56&#xA;    },&#xA;    {&#xA;        &quot;categorical&quot;: &quot;Ipsum&quot;&#xA;        &quot;other&quot;: 53&#xA;        &quot;stuff&quot;: 12&#xA;    },&#xA;    {&#xA;        &quot;categorical&quot;: &quot;Dolore&quot;&#xA;        &quot;other&quot;: 6&#xA;        &quot;stuff&quot;: 101&#xA;    }&#xA;]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously &lt;code&gt;foo&lt;/code&gt; and &lt;code&gt;bar&lt;/code&gt; are easily represented as numeric attributes. I would now like to represent this array (&lt;code&gt;array&lt;/code&gt;) as one large one-hot (or several-hot) vector. Assuming I don't care about any other field of the sub-objects except for the category field &lt;code&gt;categorical&lt;/code&gt;, here is my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it valid and possible &lt;strong&gt;to set more than one bit in a one-hot vector&lt;/strong&gt; (which would make it, I assume, a several-hot vector) to represent all categories present in this instance? If not, how could it be done?&lt;/p&gt;&#xA;" OwnerUserId="32580" LastEditorUserId="32580" LastEditDate="2017-05-24T10:53:01.627" LastActivityDate="2018-02-18T22:56:58.317" Title="Array of categorical variables vs one-hot encoding" Tags="|classification|bigdata|categorical-data|feature-engineering|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19156" PostTypeId="2" ParentId="19153" CreationDate="2017-05-24T00:18:10.050" Score="1" Body="&lt;p&gt;&quot;Each time length can different, but there are the same features for each sample.&quot; You can make them all the same length, see Myth 1 of &#xA;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Want to end up with is a set of samples with a single feature in time&quot; This might work, if the multinomial time series are &quot;Dependent&quot; See fig 1 of&#xA;&lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/Multi-Dimensional_DTW_Journal.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.ucr.edu/~eamonn/Multi-Dimensional_DTW_Journal.pdf&lt;/a&gt;&#xA;Otherwise, this is a very very bad idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;With that, I can do clustering using distance determined by dynamic time warping.&quot; Sure, but you can do clustering with multinomial time series under DTW. See [a].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[a] Nurjahan Begum, Liudmila Ulanova, Jun Wang, Eamonn Keogh (2015). Accelerating Dynamic Time Warping Clustering with a Novel Admissible Pruning Strategy  SIGKDD 2015&lt;/p&gt;&#xA;" OwnerUserId="32584" LastActivityDate="2017-05-24T00:18:10.050" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19157" PostTypeId="2" ParentId="19150" CreationDate="2017-05-24T00:57:14.110" Score="1" Body="&lt;p&gt;There are plenty of NLP libraries for .Net if you Google that. Please be more specific as to what you want to accomplish when they query the DB.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-24T00:57:14.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19160" PostTypeId="1" CreationDate="2017-05-24T03:47:29.183" Score="1" ViewCount="9716" Body="&lt;p&gt;I'm following this &lt;a href=&quot;http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;guide&lt;/a&gt; to try creating both binary classifier and multi-label classifier using &lt;strong&gt;MeanEmbeddingVectorizer&lt;/strong&gt; and &lt;strong&gt;TfidfEmbeddingVectorizer&lt;/strong&gt; shown in the guide above as inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both embedding vectorizers are created by first, initiating w2v from documents using &lt;a href=&quot;https://radimrehurek.com/gensim/models/word2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt; library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this step has completed, I tried using these embedding vectorizers as inputs into several model such as &lt;strong&gt;OneVsRestClassifier(SVC)&lt;/strong&gt;, &lt;strong&gt;RandomForestClassifier&lt;/strong&gt; and &lt;strong&gt;ExtraTreesClassifier&lt;/strong&gt;. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the accuracies from each models (binary classifer) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;randomF_countVect: 0.8898&#xA;extraT_countVect: 0.8855&#xA;extraT_tfidf: 0.8766&#xA;randomF_tfidf: 0.8701&#xA;svc_tfidf: 0.8646&#xA;svc_countVect: 0.8604&#xA;ExtraTrees_w2v: 0.7285&#xA;ExtraTrees_w2v_tfidf: 0.7241&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Multi-label classifier also produced similar result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure that I've done wrong. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Word2vec initiation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)&#xA;w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Models&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svc = Pipeline([(&quot;count_vectorizer&quot;, vectorizer), (&quot;OneVSRest svc linear&quot;, OneVsRestClassifier(SVC(kernel='linear')))])&#xA;svc_tfidf = Pipeline([(&quot;tfidf_vectorizer&quot;, tf_vectorizer), (&quot;OneVSRest svc linear&quot;, OneVsRestClassifier(SVC(kernel='linear')))])&#xA;randomF = Pipeline([(&quot;count_vectorizer&quot;, vectorizer), (&quot;RandomForestClassifier&quot;, RandomForestClassifier(n_estimators=100))])&#xA;randomF_tfidf = Pipeline([(&quot;tfidf_vectorizer&quot;, tf_vectorizer), (&quot;RandomForestClassifier&quot;, RandomForestClassifier(n_estimators=100))])&#xA;extraT = Pipeline([(&quot;count_vectorizer&quot;, vectorizer), (&quot;ExtraTreesClassifier&quot;, ExtraTreesClassifier(n_estimators=100))])&#xA;extraT_tfidf = Pipeline([(&quot;tfidf_vectorizer&quot;, tf_vectorizer), (&quot;ExtraTreesClassifier&quot;, ExtraTreesClassifier(n_estimators=100))])&#xA;etree_w2v = Pipeline([(&quot;word2vec vectorizer&quot;, MeanEmbeddingVectorizer(w2v)), (&quot;word2vec extra trees&quot;, ExtraTreesClassifier(n_estimators=100))])&#xA;etree_w2v_tfidf = Pipeline([(&quot;tfidf word2vec vectorizer&quot;, TfidfEmbeddingVectorizer(w2v)), (&quot;tfidf word2vec extra trees&quot;, ExtraTreesClassifier(n_estimators=100))])&#xA;&#xA;&#xA;all_models = [&#xA;    (&quot;svc&quot;, svc),&#xA;    (&quot;svc_tfidf&quot;, svc_tfidf),&#xA;    (&quot;randomF&quot;, randomF), &#xA;    (&quot;randomF_tfidf&quot;, randomF_tfidf),&#xA;    (&quot;extraT&quot;, extraT), &#xA;    (&quot;extraT_tfidf&quot;, extraT_tfidf),&#xA;    (&quot;etree_w2v&quot;, etree_w2v), &#xA;    (&quot;etree_w2v_tfidf&quot;, etree_w2v_tfidf)&#xA;]&#xA;scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean()) &#xA;             for name, model in all_models], &#xA;            key=lambda args: -(args[1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32343" LastEditorUserId="40853" LastEditDate="2017-12-07T06:11:48.140" LastActivityDate="2018-09-01T09:26:00.710" Title="Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification]" Tags="|machine-learning|text-mining|scikit-learn|word2vec|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19162" PostTypeId="1" CreationDate="2017-05-24T04:01:52.443" Score="0" ViewCount="131" Body="&lt;p&gt;I'm using NVIDIA DIGITS 3.0 to do training for detecting direction of a ball throw.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My &lt;code&gt;Dataset&lt;/code&gt; contains 400+ binary images each for left and right throw with the following specs in DIGITS:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Image Type: Grayscale, JPG&lt;/li&gt;&#xA;&lt;li&gt;Image Size: 256 x 256&lt;/li&gt;&#xA;&lt;li&gt;Resize Transformation: Fill&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My &lt;code&gt;Classification Model&lt;/code&gt; specs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Solver Type: NAG&lt;/li&gt;&#xA;&lt;li&gt;Networks: GoogLeNet&lt;/li&gt;&#xA;&lt;li&gt;The rest are default values&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I separated out 8 images (4 for left, 4 for right) from my 400+ training dataset to do testing on my model. The results returned from my classification model always classifies all 8 are left throws at 90%+ accuracy even though 4 of which are clearly right throws. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 of the test images are shown below. My subject always stands at the same position to throw the ball, which is more or less constant. I've tried to change various parameters in the classification model like no. of epochs, subtract image mean, etc. to no avail. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a previous dataset, my model classifies all throws as right throw, similarly only one direction is detected. I'm unsure whether the problem lies in my dataset images or the training model. I would greatly appreciate any advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Left Throw:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/0rDYz.png&quot; alt=&quot;Left&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right Throw:&#xA;&lt;img src=&quot;https://i.stack.imgur.com/OWp49.png&quot; alt=&quot;Right&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="32586" LastActivityDate="2017-05-29T22:00:36.323" Title="Machine Learning using NVIDIA DIGITS - Can't Classify Left or Right Direction of a Ball Throw" Tags="|machine-learning|nvidia|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19163" PostTypeId="1" CreationDate="2017-05-24T04:23:45.173" Score="3" ViewCount="24618" Body="&lt;p&gt;I need to convert a datetime.date(2007, 4, 30) to a Unix timestamp in pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my datable:    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0     2007-04-30&#xA;1     2007-05-31&#xA;2     2007-06-30&#xA;3     2007-07-31&#xA;4     2007-08-31&#xA;Name: datetime, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32587" LastActivityDate="2020-11-03T06:51:47.623" Title="pandas datetime to Unix" Tags="|time-series|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19164" PostTypeId="1" CreationDate="2017-05-24T04:42:21.473" Score="1" ViewCount="255" Body="&lt;p&gt;I find a powerful tool in kaggle which is &lt;a href=&quot;http://blog.kaggle.com/2017/04/10/exploring-the-structure-of-high-dimensional-data-with-hypertools-in-kaggle-kernels/&quot; rel=&quot;nofollow noreferrer&quot;&gt;hypertools&lt;/a&gt;. I find that it can plot dynamic graph. Just a single line code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hyp.plot(temps, normalize='across', animate=True, chemtrails=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QHo21.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QHo21.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is pretty cool. However, I do not know how use this tools to plot for my data. My data is a list of tuple (x, y, t) like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[  353.,  2607.,   349.],&#xA;       [  367.,  2607.,   376.],&#xA;       [  388.,  2620.,   418.],&#xA;       [  416.,  2620.,   442.],&#xA;       [  500.,  2620.,   493.],&#xA;       [  584.,  2620.,   547.],&#xA;       [  675.,  2620.,   592.],&#xA;       [  724.,  2620.,   643.],&#xA;       [  780.,  2620.,   694.],&#xA;       [  822.,  2620.,   742.],&#xA;       [  850.,  2633.,   793.],&#xA;       [  885.,  2633.,   844.],&#xA;       [  934.,  2633.,   895.],&#xA;       [  983.,  2633.,   946.],&#xA;       [ 1060.,  2633.,  1006.],&#xA;       [ 1144.,  2633.,  1063.],&#xA;       [ 1235.,  2633.,  1093.],&#xA;       [ 1284.,  2633.,  1144.],&#xA;       [ 1312.,  2633.,  1210.],&#xA;       [ 1326.,  2633.,  1243.],&#xA;       [ 1333.,  2633.,  1354.],&#xA;       [ 1354.,  2633.,  1408.],&#xA;       [ 1375.,  2646.,  1450.],&#xA;       [ 1452.,  2659.,  1492.],&#xA;       [ 1473.,  2672.,  1543.],&#xA;       [ 1480.,  2672.,  1954.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I use this powerful tool to plot mouse trajectory?&lt;/p&gt;&#xA;" OwnerUserId="32588" LastEditorUserId="381" LastEditDate="2017-05-24T06:25:21.887" LastActivityDate="2017-05-24T06:25:21.887" Title="How can I use hypertools plot the dynamic graphic about mouse trajectory according to time?" Tags="|python|visualization|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19165" PostTypeId="2" ParentId="19163" CreationDate="2017-05-24T04:51:46.740" Score="4" Body="&lt;p&gt;finally worked something out though I wonder if it is the best solution?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dt = pd.DatetimeIndex ( dt ).astype ( np.int64 )/1000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;cheers.&lt;/p&gt;&#xA;" OwnerUserId="32587" LastActivityDate="2017-05-24T04:51:46.740" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19166" PostTypeId="2" ParentId="19124" CreationDate="2017-05-24T05:19:26.057" Score="0" Body="&lt;p&gt;To understand what overfitting means and how it affects the accuracy of the model, you need to understand the bias - variance tradeoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under-fitting as well as overfitting are two different problems that are directly related to biased- variance problems. It is always important to understand the relation between three different factors and how these factors are connected to bias-variance ( overfitting- under-fitting) problems:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- the size of the model.&#xA;Number of parameters&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- the amount of data available for training.&#xA;Number of training samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- the number of iterations. &#xA;training iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Making a direct connection between any of these factors to overfitting- under-fitting problems without looking at the others will always lead to wrong conclusions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of understanding these factors and linking theme by using mathematical equations to avoid overfitting and under-fitting problems is a difficult task, more over it is a task dependent, people use simple methods to discover and avoid overfitting. &#xA;The easy way is to divide the data into three different parts, training, validation and testing. Testing should not be touched. Use training set to train the network and validation set to test the network after each iteration or a number of iterations. Theoretically, you will see that the error on the validation set decreases gradually for the first N iterations and then will be stable for very few iterations and then starts increasing. When the error starts increasing, your network starts overfitting the training data and the training process should be stopped.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: the value N is very related to the three factors I listed above. It is always a good practice to have a demo training set and test with different models, training data. You will see that the larger the model - the less training data the smaller the N. The smaller the model - the more training data the larger the N. &#xA;Note: be careful when using small models of having under-fitting problem.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2017-05-24T05:32:50.217" LastActivityDate="2017-05-24T05:32:50.217" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19167" PostTypeId="1" CreationDate="2017-05-24T05:21:39.753" Score="3" ViewCount="676" Body="&lt;p&gt;While working with 3 classes,my dataset contains different proportions of all classes. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Class &quot;0&quot;  has  11098 cases.....Normal&#xA;class  &quot;1&quot; has  2369 cases.....&quot;Abnormal&quot;&#xA;class  &quot;2&quot; has  3396 cases.....&quot;May be Normal or Abnormal&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I trained the model with similar proportions. I have two questions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it necessary to balance all three classes?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the impact of class imbalance on classification performance?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="14372" LastEditDate="2017-05-25T17:26:29.720" LastActivityDate="2017-07-25T15:33:55.563" Title="Class imbalance problem?" Tags="|machine-learning|deep-learning|multiclass-classification|class-imbalance|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19168" PostTypeId="1" CreationDate="2017-05-24T05:45:10.267" Score="-1" ViewCount="86" Body="&lt;p&gt;How do I setup an environment where I can coalesce data from various sources, like mysql databases and csv files, on my macbook? Is there some opensource tool like informatica? or... ?&lt;/p&gt;&#xA;" OwnerUserId="27976" LastActivityDate="2017-08-23T16:12:52.817" Title="How do I coalesce data from various sources?" Tags="|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19169" PostTypeId="2" ParentId="19150" CreationDate="2017-05-24T05:45:54.810" Score="1" Body="&lt;p&gt;It seems like for my requirement  &lt;a href=&quot;https://stanfordnlp.github.io/CoreNLP/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Stanford CoreNLP&lt;/strong&gt;&lt;/a&gt; would do the trick. and there are so many services provided by &lt;strong&gt;azure&lt;/strong&gt; for the same purpose. Sorry if my question offended someone in any way.&lt;/p&gt;&#xA;" OwnerUserId="32554" LastActivityDate="2017-05-24T05:45:54.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19170" PostTypeId="1" AcceptedAnswerId="19171" CreationDate="2017-05-24T06:13:56.507" Score="2" ViewCount="5863" Body="&lt;p&gt;I'm now learning about deep learning with Keras, and to implement a deep learning model at Keras, you set the initializer to set its initial weights on.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;model = Sequential()&#xA;model.add(Dense(12, input_dim=8, kernel_initializer=&quot;random_uniform&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;kernel_initializer&lt;/code&gt; can take something others, such as &lt;code&gt;random_normal&lt;/code&gt;, which uses Gaussian, not uniform distribution, and &lt;code&gt;zero&lt;/code&gt;, which literally sets all weights to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't understand why you like to set different weights at the initializer. Specifically, what advantages does it have over setting all initial weights to 0, which sounds more natural for novices like me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, should the initial weights, if needed, be &lt;strong&gt;always&lt;/strong&gt; set a tiny value (e.g. &lt;code&gt;0.05&lt;/code&gt;) to?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-05-24T14:47:51.773" Title="What is the purpose of setting an initial weight on deep learning model?" Tags="|deep-learning|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19171" PostTypeId="2" ParentId="19170" CreationDate="2017-05-24T06:33:21.097" Score="7" Body="&lt;p&gt;This is greatly addressed in the &lt;a href=&quot;http://cs231n.github.io/neural-networks-2/#init&quot; rel=&quot;noreferrer&quot;&gt;Stanford CS class CS231n&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are several weight initialization strategies; each one is best suited for a type of activation function. For instance, &lt;a href=&quot;http://www.jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;noreferrer&quot;&gt;Glorot's initialization&lt;/a&gt; aims at not saturating sigmoid activations, while &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf&quot; rel=&quot;noreferrer&quot;&gt;He's initialization&lt;/a&gt; is meant for Rectified Linear Units (ReLUs).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-05-24T13:44:27.617" LastActivityDate="2017-05-24T13:44:27.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19172" PostTypeId="1" CreationDate="2017-05-24T07:09:27.113" Score="2" ViewCount="798" Body="&lt;p&gt;I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, Cosine Similarity and Manhatten Distance, and get quite different values:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cosine Similarity: ~0.8 to ~0.6&lt;/li&gt;&#xA;&lt;li&gt;Manhattan Distance: ~0.3 to ~0.2&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can these values be compared directly? (Does Manhattan really performs a lot better here?) Or is there another way to compare the clustering results of the two different measures?&lt;/p&gt;&#xA;" OwnerUserId="32595" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:30.170" LastActivityDate="2017-05-29T04:59:30.170" Title="How to compare performance of Cosine Similarity and Manhatten Distance?" Tags="|clustering|k-means|performance|model-evaluations|distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19173" PostTypeId="2" ParentId="19137" CreationDate="2017-05-24T08:50:45.567" Score="1" Body="&lt;p&gt;So let's start! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all please have a look at the edit I made to your original question. It was not just an edit but implied important conceptual thing so I need to start with them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graphs&lt;/a&gt;: &lt;em&gt;Graph&lt;/em&gt; is a reserved term in mathematics for network-like objects. You better use the term &lt;em&gt;Plot&lt;/em&gt; for sake of clarification. So the question is absolutely not about graphs.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time-series data&lt;/a&gt;: The example you made i.e. a temporal sequence of data points is called &lt;em&gt;Time-series&lt;/em&gt;. A n-dimensional data is not necessarily a time-series and however both seem to inhibit the same structure, they are conceptually different (but not necessarily irrelevant) types of data and different sets of algorithms and problems are defined for either, so please comment here if the data is time-series or not.&lt;/li&gt;&#xA;&lt;li&gt;Curve Plots: The last but not the least is that a plot is used to &lt;em&gt;just illustrate&lt;/em&gt; the data and all Data Analysis algorithms and approaches use that data for inference. So you never infer anything from the plot itself but from the data that it depicts. If you have any plot-like data which you don't know the data behind you need to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Image_processing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image Processing&lt;/a&gt; which I'm pretty sure is not the topic of this question but wanted to mention as you explicitly asked how to do the analysis with plots themselves.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After these points I get back to your question. Yes, what you are looking for is Pattern Recognition and you can use many Machine Learning approaches to solve your problem. If you already knew what are different classes of patterns, it would be called &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Classification&lt;/a&gt;&lt;/em&gt; but now, as I understood, you don't know the classes and you want to find &lt;em&gt;similar patterns&lt;/em&gt;. Finding patterns of similarity in data is called &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Clustering&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I don't know if your data is a time-series (e.g. amount of raining vs time) or just a structured n-dimensional data (e.g. amount of raining vs humidity) I try both. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Time-Series&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;When you have different segments of time-series and you need their similarity you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow noreferrer&quot;&gt;Correlation Analysis&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Time Warping&lt;/a&gt;. If the time series is high frequency e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Speech_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;speech data&lt;/a&gt; or &lt;a href=&quot;https://imotions.com/blog/5-basics-eeg-data-processing/&quot; rel=&quot;nofollow noreferrer&quot;&gt;EEG&lt;/a&gt;, then you better convert data to &lt;a href=&quot;https://en.wikipedia.org/wiki/Fourier_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;frequency&lt;/a&gt; (or &lt;a href=&quot;https://en.wikipedia.org/wiki/Discrete_wavelet_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;time-frequency&lt;/a&gt;) domain and then &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_extraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;extract features&lt;/a&gt; from those segment and then use those features to determine &lt;a href=&quot;https://en.wikipedia.org/wiki/Similarity_measure&quot; rel=&quot;nofollow noreferrer&quot;&gt;similarity&lt;/a&gt;. Please note that the term similarity is pretty wide and it's defined according to the nature of data and problem. You may search for &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0031320305001305&quot; rel=&quot;nofollow noreferrer&quot;&gt;time-series clustering&lt;/a&gt; for more approaches.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Non-temporal Data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Similarity between the parameters of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Regression_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;regression model&lt;/a&gt; fitted to those subsets of data might help. Another approach is to apply clustering algorithms to those segments. These algorithms can be applied to the original data subsets or some features extracted from those. &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dimensionality Reduction&lt;/a&gt; algorithms like &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt; can be used for feature extraction and might be helpful for a better clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote my answer in rush so I would appreciate any comment or further questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2017-05-24T08:56:57.770" LastActivityDate="2017-05-24T08:56:57.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19174" PostTypeId="1" AcceptedAnswerId="19188" CreationDate="2017-05-24T08:56:18.580" Score="3" ViewCount="697" Body="&lt;p&gt;I have a regression use case where I am supposed to estimate a value based on 3-4 features. Using random forest, I was able to get ~20% error. However, I have a constraint now. I can overestimate but not underestimate. So, at the cost of improving the error I am allowed to overestimate. What is the right approach of handling this constraint? Is it okay to just go with an approach like 1.2x the estimation provided by the model?&lt;/p&gt;&#xA;" OwnerUserId="32602" LastActivityDate="2017-05-24T16:55:26.080" Title="Always overestimate" Tags="|regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19177" PostTypeId="2" ParentId="17134" CreationDate="2017-05-24T10:50:29.960" Score="-1" Body="&lt;p&gt;Remember that CNNs are feature detectors. The output of a convolutional layer is a matrix that signals where certain feature was detected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, recurrent CNNs are recurrent neural networks that learn sequences of features, where those features are also learnt during the training. &lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-24T10:50:29.960" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19180" PostTypeId="2" ParentId="19155" CreationDate="2017-05-24T14:30:19.107" Score="1" Body="&lt;p&gt;Yes, it will work. Basically by creating the encoding, scikit-learn's label encoder does the same thing, you are creating such more features each representing the presence or absence of that level in your sample. To represent d levels you need d-1 variables, so in your case that is what is exactly happening. However since you said, the number of levels can be massive it will not be a good idea to use a vector that massive. It would be a good idea to run some preliminary analysis and check if some levels are totally useless and dont relate to the response as well, some might have the exact same effect and hence those levels can be combined.  &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-05-24T14:30:19.107" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19182" PostTypeId="2" ParentId="19170" CreationDate="2017-05-24T14:47:51.773" Score="1" Body="&lt;p&gt;That has to do with how forward and backpropagation works. Remember that forward propagation is done by applying the activation function to the result of multiplying the activations of layer &lt;code&gt;al&lt;/code&gt; by a weight matrix &lt;code&gt;Wl&lt;/code&gt; plus the bias vector &lt;code&gt;bl&lt;/code&gt;  for each layer of the network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;zl = activation(Wl * al-1 + bl)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;zl&lt;/code&gt; is the output of layer &lt;code&gt;l&lt;/code&gt; and &lt;code&gt;a0&lt;/code&gt; would be the input layer. It is easy to see from there that if the &lt;code&gt;Wl&lt;/code&gt; is set to zero the output &lt;code&gt;zl&lt;/code&gt; will depend only on &lt;code&gt;bl&lt;/code&gt; and the activation values will be the same for every neuron in the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The backpropagation step of gradient descent uses &lt;code&gt;zl&lt;/code&gt; calculate the gradient step and parameter updates so if every neuron in the layer has the same value those steps will also output equal values and all the neurons will learn the same parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By initializing the outputs to random values in the range you'll make each neuron learn differently breaking the symmetry.&lt;/p&gt;&#xA;" OwnerUserId="32177" LastActivityDate="2017-05-24T14:47:51.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19183" PostTypeId="2" ParentId="19167" CreationDate="2017-05-24T14:48:10.663" Score="2" Body="&lt;p&gt;As mentioned in the comments, your data doesn't have that much of an imbalance. A good way to check how the imbalance is affecting your model, is by calculating precision and recall. These values will be quite low while your accuracy will shoot high because it will predict the majority class for every sample. However the 'maybe normal' case is dangerous. It would be better to set a different threshold , probably a lower one so that even the marginally abnormal cases will be classified as abnormal. A precision recall curve or a ROC curve is a good way to decide on the cutoff. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="14372" LastEditDate="2017-05-25T17:26:27.000" LastActivityDate="2017-05-25T17:26:27.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19184" PostTypeId="1" CreationDate="2017-05-24T15:38:40.683" Score="1" ViewCount="308" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1aXF4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1aXF4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The atmosphere is made up of 4 distinct layers: the troposphere, stratosphere, mesosphere, and thermosphere. Different types of clouds form in the different layers depending on the pressure in the atmosphere and the ambi- ent temperature. The cloud types include nimbus, stratus, cumulus, and cirrus. Figure 1 shows the location of the&#xA;barriers of the atmosphere when the temperature and pressure are at an ideal condition for cloud formation. It also shows the different types of clouds formed at the different levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-&gt;How do you interpret this kind of graph? I sort of get it that as we go higher, pressure increases, while temperature decreases. But the graph looks weird to me and I can't seem to understand the right side of the graph, which sort of &quot;shows that temperature also increases&quot;. Can anyone help me explain the concept behind this type of graphical representation? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="32619" LastEditorUserId="23305" LastEditDate="2017-05-29T05:00:05.380" LastActivityDate="2017-05-29T05:00:05.380" Title="Data Interpretation with a double line graph" Tags="|visualization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19185" PostTypeId="2" ParentId="19131" CreationDate="2017-05-24T15:58:51.767" Score="1" Body="&lt;p&gt;This is how you should get your topics and corresponding words. You first get the NMF transformation of your new input and sort its components. The biggest components tell you which features to look at in nmf.components_, which links topics to words (your tfidf features). Finally, you print the k most important words of the topics. Topics and words scores are just calculated by dividing their value by the sum of all values.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import numpy as np&#xA;    # Transform new_feedback to NMF space&#xA;    nmf_new_feedback = clf.transform(vectorizer.fit(new_feedback))&#xA;    # top 10 topics of the new feedback&#xA;    compo = np.argsort(nmf_new_feedback)[-10:][::-1]&#xA;    # Connection between indices and words of tfidf&#xA;    feature_names = vectorizer.get_feature_names()&#xA;    for topic_idx in compo:&#xA;        # Current nmf topic&#xA;        topic = nmf.components_[topic_idx]&#xA;        print(&quot;Topic number: &quot;, topic_idx)&#xA;        print(&quot;Topic score: &quot;, nmf_new_feedback[topic_idx]/sum(nmf_new_feedback))&#xA;        # Top 15 words of current topic&#xA;        for k in topic.argsort()[-15:][::-1]]&#xA;            print(&quot;Word: &quot;,feature_names[k], &quot;  Score: &quot;, topic[k]/ sum(topic))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-05-24T15:58:51.767" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19186" PostTypeId="1" CreationDate="2017-05-24T16:13:03.890" Score="9" ViewCount="2895" Body="&lt;p&gt;I have training data that is labelled with binary values. I also have collected the confidence of each of these labels i.e. 0.8 confidence would mean that 80% of the human labellers agree on that label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use this confidence data to improve the accuracy of my classifier? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would the following work?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;1a) If the label is 0 and the confidence data for that label is 0.8&#xA;then give the training data a new label of 0.2&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;1b) If the label is 1 and the confidence data for that label is 0.8&#xA;then give the training data a new label of 0.8&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;2) Calculate the new label using this method for every entry in the&#xA;training set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;3) Treat the problem as a regression problem (where the labels range&#xA;between 0 and 1)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;4) Classify unlabelled data based on whether the new labels are above&#xA;or below a specific value. i.e Give all predicted labels class 0 if below X and class 1 if above X.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am currently using a SVM with an RBF kernel for my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="32622" LastEditorUserId="32622" LastEditDate="2017-05-24T17:01:09.550" LastActivityDate="2019-06-18T08:42:02.380" Title="Can training label confidence be used to improve prediction accuracy?" Tags="|machine-learning|classification|regression|scikit-learn|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19187" PostTypeId="1" CreationDate="2017-05-24T16:14:52.983" Score="0" ViewCount="62" Body="&lt;p&gt;I am new to predictive analytics but I am good at programming (like Spark, R). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 10 variables and I know their score's for over a long period of time. I want to know which variable will have high score in the future? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is a linear plot (plotted between score and time for a variable), we can extrapolate the values and find the max(future_score) among the 10 variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the plot is non-linear (although continuous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help?&lt;/p&gt;&#xA;" OwnerUserId="32623" LastActivityDate="2018-04-13T19:21:23.237" Title="Predicting the variable's score using predictive analytics" Tags="|predictive-modeling|time-series|apache-spark|pyspark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19188" PostTypeId="2" ParentId="19174" CreationDate="2017-05-24T16:55:26.080" Score="4" Body="&lt;p&gt;Use an asymmetric loss function with a cliff at your margin; e.g.,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal L (x) \equiv \begin{cases}&#xA;x^2 &amp;amp;&amp;amp; x &amp;gt; 0\\ &#xA;c x^2 &amp;amp;&amp;amp; -m &amp;lt; x &amp;lt; 0 \\&#xA;d x^{2p} &amp;amp;&amp;amp; x &amp;lt; -m&#xA;\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $c&amp;gt;1$, $p&amp;gt;1$, $d\equiv cm^{2-2p}$. The idea is to encode your constraints in your loss function.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-05-24T16:55:26.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19190" PostTypeId="2" ParentId="19140" CreationDate="2017-05-24T19:09:51.810" Score="4" Body="&lt;p&gt;The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't select a fold yourself.  K-Fold cross-validation is used to test the general accuracy of your model based on how you setup the parameters and hyper-parameters of your model fitting function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you do select is the number of folds, so in your example of 5 folds, it will do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;split up your training set into 5 different subsets (folds) &lt;/li&gt;&#xA;&lt;li&gt;create a classifier for each of the 5 folds by using k-1 folds for&#xA;fitting the model, and test the classifier accuracy using the fold&#xA;left out&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After it's done you can see how your classifier fared over the average of these folds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're trying to find the optimal parameters to configure your model for the best accuracy you should be using grid search.  Depending on your language the implementation will be different: python using sklearn.model_selection.GridSearchCV&lt;/p&gt;&#xA;&#xA;&lt;p&gt;, and R uses the 'carat' library and the train() function.  Once you've run grid search you can either use the resulting model if you're programming in R or in python, you can add the new hyperparameters to your model fitting function and re-fit your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your train/test ratio will depend on the number of folds. If you have 100 rows in your training set and you have 5 folds, then you'll have an 80/20 split train/test.  If you have 10 folds it will be a 90/10 split.  The cross-validation function for k-folds uses k-1 folds for fitting the model, and the fold left out is used for testing.&lt;/p&gt;&#xA;" OwnerUserId="32627" LastEditorUserId="27432" LastEditDate="2020-03-04T13:22:39.333" LastActivityDate="2020-03-04T13:22:39.333" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="19191" PostTypeId="1" CreationDate="2017-05-24T19:46:03.913" Score="5" ViewCount="2242" Body="&lt;p&gt;I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider predicting machine failure based on its last repair report. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A machine has many (many) different parts.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Every part could have been damaged or not.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If damaged, it could have been inspected, repaired or replaced etc.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;These operations have associated costs and the part itself has a money value and other attributes (like the time it took to repair it...). &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I want to make predictions on the level of a machine, I need to aggregate all this information. I could one-hot encode all the parts indicating whether they have been damaged or not. But this still leaves me with the other attributes on the lower part level, like price and performed operation. I could probably further expand them into individual columns by considering all the combinations like &lt;code&gt;part_X-repaired-..-price, part_X-replaced-..-price, ... part_Z-replaced-..-price&lt;/code&gt; but this seems to get out of hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to handle this type of data? I was thinking maybe some clustering technique but when I try to set it up I run into the same problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is somewhat a reversed hierarchical model structure (in hierarchical (linear) models the outcome variable is on the lowest level if I'm not mistaking).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-07-13T16:42:52.717" LastActivityDate="2017-12-11T03:10:56.297" Title="Nested features with one to many relationships" Tags="|machine-learning|dataset|categorical-data|preprocessing|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19194" PostTypeId="2" ParentId="19187" CreationDate="2017-05-25T08:04:02.143" Score="1" Body="&lt;p&gt;Can you show us some plots of them ? it really depends on what they look. There is a lot of temporal series forecast. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will depends on : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any seasonality ?&lt;/li&gt;&#xA;&lt;li&gt;Is there a trend ?&lt;/li&gt;&#xA;&lt;li&gt;Is there a high volatility ?&lt;/li&gt;&#xA;&lt;li&gt;On how long do you want do to your forecast ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20945" LastActivityDate="2017-05-25T08:04:02.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19196" PostTypeId="1" AcceptedAnswerId="19269" CreationDate="2017-05-25T08:51:13.597" Score="14" ViewCount="7254" Body="&lt;p&gt;I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;Colah's blog post &quot;Understanding LSTM Networks&quot;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bdKbh.pnghttps://www.tensorflow.org/versions/r0.11/api_docs/python/rnn_cell/rnn_cells_for_use_with_tensorflow_s_core_rnn_methods#BasicRNNCell&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bdKbh.pnghttps://www.tensorflow.org/versions/r0.11/api_docs/python/rnn_cell/rnn_cells_for_use_with_tensorflow_s_core_rnn_methods#BasicRNNCell&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$x_t$ is input of size $m*1$ vector&lt;/li&gt;&#xA;&lt;li&gt;$h_{t-1}$ is hidden state of size $n*1$ vector&lt;/li&gt;&#xA;&lt;li&gt;$[x_t, h_{t-1}]$ is a concatenation (for example, if $x_t=[1, 2, 3], h_{t-1}=[4,&#xA;   5, 6]$, then $[x_t, h_{t-1}]=[1, 2, 3, 4, 5, 6]$)&lt;/li&gt;&#xA;&lt;li&gt;$w_f$ is weights of size $k*(m+n)$ matrix, where $k$ is the number of&#xA;cell states (if $m=3$, and $n=3$ in the above example, and if we have&#xA;3 cell states, then $w_f=3*3 $ matrix)&lt;/li&gt;&#xA;&lt;li&gt;$b_f$ is bias of size $k*1$ vector, where $k$ is the number of cell&#xA;states (since $k=3$ as the above example, then $b_f$ is a $3*1$&#xA;vector).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If we set $w_f$ to be:&#xA;        \begin{bmatrix}&#xA;    1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 \\&#xA;    5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 &amp;amp; 9 &amp;amp; 10 \\&#xA;    3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 \\&#xA;    \end{bmatrix}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And $b_f$ to be: $[1, 2, 3]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then $W_f . [h_{t-1}, x_t] =$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;    \begin{bmatrix}&#xA;    1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 \\&#xA;    5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 &amp;amp; 9 &amp;amp; 10 \\&#xA;    3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 \\&#xA;    \end{bmatrix}&#xA;.        \begin{bmatrix}&#xA;    1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \\&#xA;    \end{bmatrix}&#xA;=\begin{bmatrix} 91 &amp;amp; 175 &amp;amp; 133\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we can add the bias, $W_f . [h_{t-1}, x_t] + b_f=$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{bmatrix} 91 &amp;amp; 175 &amp;amp; 133\end{bmatrix} + \begin{bmatrix} 1 &amp;amp; 2 &amp;amp; 3\end{bmatrix}=\begin{bmatrix} 92 &amp;amp; 177 &amp;amp; 136\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we feed them into a sigmoid function: $\frac{1}{1+e^{-x}}$, where $x=\begin{bmatrix} 92 &amp;amp; 177 &amp;amp; 136\end{bmatrix}$, hence we perform this function element wise, and get \begin{bmatrix} 1 &amp;amp; 1 &amp;amp; 1\end{bmatrix}. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means for each cell state, $C_{t-1}$, (there are $k=3$ cell states), we allow it to pass to the next layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the above assumption correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also means that the number of cell state and hidden state is the same?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="31513" LastEditDate="2017-05-25T12:23:48.947" LastActivityDate="2020-04-27T03:37:40.030" Title="Forget Layer in a Recurrent Neural Network (RNN) -" Tags="|neural-network|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19197" PostTypeId="1" CreationDate="2017-05-25T08:57:16.483" Score="1" ViewCount="2255" Body="&lt;p&gt;I am currently trying to classify cifar10 data using the vgg16 network on Keras, but seem to get pretty bad result, which I can't quite figure out...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vgg16 is designed for performing classification on 1000 class problems. Why is it not applicable in a small problem setting like cifar10?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.utils import np_utils&#xA;&#xA;from keras import metrics&#xA;import keras&#xA;from keras import backend as K&#xA;from keras.layers import Conv1D,Conv2D,MaxPooling2D, MaxPooling1D, Reshape&#xA;from keras.models import Model&#xA;from keras.layers import Input, Dense&#xA;import tensorflow as tf&#xA;from keras.datasets import mnist,cifar10&#xA;&#xA;&#xA;&#xA;WEIGHTS_PATH = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5'&#xA;WEIGHTS_PATH_NO_TOP = 'https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5'&#xA;&#xA;batch_size = 42&#xA;num_classes = 10&#xA;epochs = 100&#xA;&#xA;# input image dimensions&#xA;img_rows, img_cols = 32, 32&#xA;&#xA;# the data, shuffled and split between train and test sets&#xA;(x_train, y_train), (x_test, y_test) = cifar10.load_data()&#xA;&#xA;#print('x_train shape:', x_train.shape)&#xA;#print(x_train.shape[0], 'train samples')&#xA;#print(x_test.shape[0], 'test samples')&#xA;&#xA;if K.image_data_format() == 'channels_first':&#xA;    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)&#xA;    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)&#xA;    input_shape = (1, img_rows, img_cols)&#xA;else:&#xA;    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 3)&#xA;    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 3)&#xA;    input_shape = (img_rows, img_cols, 1)&#xA;&#xA;x_train = x_train.astype('float32')&#xA;x_test = x_test.astype('float32')&#xA;&#xA;x_train /= 255&#xA;x_test /= 255&#xA;&#xA;&#xA;&#xA;y_train = keras.utils.to_categorical(y_train, num_classes)&#xA;y_test = keras.utils.to_categorical(y_test, num_classes)&#xA;&#xA;&#xA;print('x_train shape:', x_train.shape)&#xA;print(x_train.shape[0], 'train samples')&#xA;print(x_test.shape[0], 'test samples')&#xA;&#xA;print('y_train shape:', y_train.shape)&#xA;print(y_train.shape, 'train samples')&#xA;print(y_test.shape, 'test samples')&#xA;def fws():&#xA;    #print &quot;Inside&quot;&#xA;    #   Params:&#xA;    #   batch ,  lr, decay , momentum, epochs&#xA;    #&#xA;    #Input shape: (batch_size,40,45,3)&#xA;    #output shape: (1,15,50)&#xA;    # number of unit in conv_feature_map = splitd&#xA;    input = Input(shape=(img_rows,img_cols,3))&#xA;    zero_padded_section = keras.layers.convolutional.ZeroPadding2D(padding=(96,96), data_format='channels_last')(input)&#xA;    print zero_padded_section&#xA;    model = keras.applications.vgg16.VGG16(include_top = False,&#xA;                    weights = 'imagenet',&#xA;                    input_shape = (224,224,3),&#xA;                    pooling = 'max',&#xA;                    classes = 10)&#xA;&#xA;    model_output = model(input)&#xA;    print model_output&#xA;&#xA;    #FC&#xA;    dense1 = Dense(units = 4000, activation = 'relu',    name = &quot;dense_1&quot;)(model_output)&#xA;    dense2 = Dense(units = 4000, activation = 'relu',    name = &quot;dense_2&quot;)(dense1)&#xA;    dense3 = Dense(units = 10 , activation = 'softmax', name = &quot;dense_3&quot;)(dense1)&#xA;&#xA;&#xA;    model = Model(inputs = input , outputs = dense3)&#xA;    #sgd = SGD(lr=0.08,decay=0.025,momentum = 0.99,nesterov = True)&#xA;    model.compile(loss=&quot;categorical_crossentropy&quot;, optimizer='adam' , metrics = [metrics.categorical_accuracy])&#xA;&#xA;    model.fit(x_train[:500], y_train[:500],&#xA;              batch_size=batch_size,&#xA;              epochs=epochs,&#xA;              verbose=1,&#xA;              validation_data=(x_test[:10], y_test[:10]))&#xA;    score = model.evaluate(x_test, y_test, verbose=0)&#xA;    print('Test loss:', score[0])&#xA;    print('Test accuracy:', score[1])&#xA;&#xA;&#xA;fws()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Current results: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;500/500 [==============================] - 166s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 36/100&#xA;500/500 [==============================] - 163s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 37/100&#xA;500/500 [==============================] - 166s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 38/100&#xA;500/500 [==============================] - 168s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 39/100&#xA;500/500 [==============================] - 167s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 40/100&#xA;500/500 [==============================] - 163s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 41/100&#xA;500/500 [==============================] - 167s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 42/100&#xA;500/500 [==============================] - 165s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 43/100&#xA;500/500 [==============================] - 178s - loss: 14.6030 - categorical_accuracy: 0.0940 - val_loss: 12.8945 - val_categorical_accuracy: 0.2000&#xA;Epoch 44/100&#xA;462/500 [==========================&amp;gt;...] - ETA: 16s - loss: 14.6179 - categorical_accuracy: 0.0931 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and still constantly outputting the same output...&lt;/p&gt;&#xA;" OwnerUserId="27966" LastEditorUserId="29169" LastEditDate="2021-02-16T02:41:31.697" LastActivityDate="2021-02-16T02:41:31.697" Title="Cifar10 classified using VGG16 keep showing the same output?" Tags="|deep-learning|classification|keras|convolutional-neural-network|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19198" PostTypeId="2" ParentId="18991" CreationDate="2017-05-25T11:13:10.950" Score="2" Body="&lt;p&gt;In the &lt;a href=&quot;http://cs231n.github.io/neural-networks-3/#baby&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS231n coursework&lt;/a&gt;, Andrej Karpathy suggests the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Look for correct loss at chance performance. Make sure you’re getting&#xA;  the loss you expect when you initialize with small parameters. It’s&#xA;  best to first check the data loss alone (so set regularization&#xA;  strength to zero). For example, for CIFAR-10 with a Softmax classifier&#xA;  we would expect the initial loss to be 2.302, because we expect a&#xA;  diffuse probability of 0.1 for each class (since there are 10&#xA;  classes), and Softmax loss is the negative log probability of the&#xA;  correct class so: -ln(0.1) = 2.302.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="20374" LastActivityDate="2017-05-25T11:13:10.950" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19199" PostTypeId="1" AcceptedAnswerId="19248" CreationDate="2017-05-25T11:47:36.777" Score="2" ViewCount="81" Body="&lt;p&gt;I want to solve the problem of finding a parameter vector for an image filter (let us assume we know nothing about how the filter works, but we can feed it an input image and a set of parameters to produce an output image).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, having a set $\{{I_k, J_k:=F_{\alpha}(I_k)}\}_{k\in\overline{1,N}}$ of $I_k$ images together with their filtered counterparts, $J_k$, what solutions would you recommend for finding $\alpha^\ast$ such that given $I^\ast$ the result $F_{\alpha^\ast}(I^\ast)$ is in the same &quot;style&quot; as the one of the $N$ training correspondence pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose one option  is to use a convnet to transform $I_k$ into a feature vector, $v_k$, and then concatenate $\alpha_k$ to obtain $u_k =(v_k,\alpha_k)$. Once this is done, use a regression method to estimate the $\alpha^\ast$ part of $u^\ast$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find an alternative solution to what seems like a candidate for the &lt;em&gt;style transfer&lt;/em&gt; approach (e.g. &lt;a href=&quot;https://arxiv.org/pdf/1703.07511.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1703.07511.pdf&lt;/a&gt;). That approach seems to solve the problem differently, and I envision situations where I need to simply use a filter rather than let a network &quot;guess the style of that filter&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Additional details and possible assumptions&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the invoked &lt;em&gt;no free lunch&lt;/em&gt; prospects, let us assume, for a paeticular problem from this class, that $F$ is a non-linear kernel-based filter that maps $I$ to $J$ as a result of an iterative and convergent process. More specifically, let $F$ be a mean shift filter with the $\alpha=(\rho, \sigma_s,\sigma_r)$ using a concatenated Gaussian kernel and a Parzen window of size $\rho$. Intuitively, I would be tempted to guess that this filter is not smooth w.r.t. $\alpha$, but a formal investigation is required (I suspect it is not smooth given that infinitesimal changes in the size of the window could shift the output towards another mode, indicating a step function behaviour).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, it is correct to assume that $\alpha \in \mathbb{R}^d$, with $d \ll N$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the goal of finding $\alpha$ when both the filter action is known (either via numerical computation in general, or, in closed-form if the filter is a gaussian blur, for example), we can be confident that the $N$ input samples have non-constant $\alpha_k$ vector values to start with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for sake of generalizability, it would be more elegant to pursue a solution that does not need to know how the filter operates without actually applying it to an input. The first approach suggested in the comments and based on convnets seems to fit this scenario and the optimization problem is taking into account the filter error. However, it would be interesting to hear more opinions, perhaps involving shallow approaches, even at the expense of designing the solution to address the concrete mean shift filter example from above.&lt;/p&gt;&#xA;" OwnerUserId="32314" LastEditorUserId="29169" LastEditDate="2021-02-16T02:41:18.863" LastActivityDate="2021-02-16T02:41:18.863" Title="Finding parameters of image filter using classified pairs" Tags="|regression|convolutional-neural-network|supervised-learning|parameter-estimation|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="19200" PostTypeId="2" ParentId="19191" CreationDate="2017-05-25T12:40:25.093" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I could probably further expand them into individual columns by&#xA;  considering all the combinations like part_X-repaired-..-price,&#xA;  part_X-replaced-..-price, ... part_Z-replaced-..-price but this seems&#xA;  to get out of hand.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is the way I would go if the approach you are using doesn't like high cardinality categorical variables. There might be some interesting relationships that fall out, like part A tends to need a repair when part B is replaced with a cheap version. If you end up with too many variables, you could try something like PCA to &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction#Principal_component_analysis_.28PCA.29&quot; rel=&quot;nofollow noreferrer&quot;&gt;reduce them&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[Edit]&lt;/strong&gt; Thought of another thing to look into: &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vowpal Wabbit&lt;/a&gt;. It is an algorithm targeted almost exactly at your problem - &lt;em&gt;lots&lt;/em&gt; of sparse variables. They claim it can handle 1 billion.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="30212" LastEditDate="2017-10-11T19:55:21.500" LastActivityDate="2017-10-11T19:55:21.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19201" PostTypeId="2" ParentId="19168" CreationDate="2017-05-25T13:02:50.820" Score="0" Body="&lt;p&gt;Options include: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do it in your code like @Emre said. Pandas (if you're using Python)&#xA;has lots of options for joining and unioning data sets. If you are using R, look up dataframe operations.&lt;/li&gt;&#xA;&lt;li&gt;Create yourself a small &quot;data mart&quot; or &quot;data lake&quot; in a DB like PostgreSQL or MySQL whose sole purpose is to coalesce things. This is what I typically do, because databases can handle way more data than you could fit in memory and SQL is very good at filtering, joining, and subsetting. Some people may use an ETL took like Informatica (which is neither open source nor cheap) but it is generally not necessary for ad-hoc analysis. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-25T13:02:50.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19202" PostTypeId="1" CreationDate="2017-05-25T13:35:41.380" Score="0" ViewCount="72" Body="&lt;p&gt;Let $D={D_k}_1^N$ denotes a set of i.i.d observations, and x denotes random variables, p() denotes probability, whether $p(D|x)$ equals to $p(D)$? Thank you for any help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: To be more clear, I talk this problem in Bayesian inference context where we seek the posterior distribution $p(x|D)~p(x)p(D|x)$, so D is dependent on x.&lt;/p&gt;&#xA;" OwnerUserId="32655" LastEditorUserId="32230" LastEditDate="2017-06-10T06:20:40.090" LastActivityDate="2017-06-10T06:20:40.090" Title="A question on data distribution probability" Tags="|machine-learning|dataset|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19205" PostTypeId="2" ParentId="19202" CreationDate="2017-05-25T14:39:00.337" Score="1" Body="&lt;p&gt;You stated that D are i.i.d. so they are independent to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The crucial question is whether D are all independent of x.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p(D|x) = p(D)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;only if &lt;strong&gt;D are all independent of x&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="31216" LastActivityDate="2017-05-25T14:39:00.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19206" PostTypeId="2" ParentId="19202" CreationDate="2017-05-25T14:48:27.797" Score="0" Body="&lt;p&gt;The general rule is the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X and Y are independent if either $P(X|Y)=P(X)$ or $P(Y|X)=P(Y)$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So if both, D and x, are independent, then $P(D|x)=P(D)$.&#xA;I hope this answers your question. However, if you meant that D and x are i.i.d., then $P(D|x)=P(D)$ is true.&lt;/p&gt;&#xA;" OwnerUserId="32660" LastEditorUserId="32660" LastEditDate="2017-06-09T10:46:08.797" LastActivityDate="2017-06-09T10:46:08.797" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19207" PostTypeId="1" CreationDate="2017-05-25T14:58:57.187" Score="7" ViewCount="6611" Body="&lt;p&gt;I am trying to build a CART model using  rpart on a data set with around 7k rows and 456 columns &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cmodel2=rpart(DV ~ .,data=teltrain2,method=&quot;class&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This has never returned a result yet. I am using a 16GB ram machine with R 3.2.3. Every time on running this, R execution goes on and on, I have to kill the R GUI process to finally stop it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Description of the columns&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;id --&gt; unique id (which I have removed from the training set)  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Location --&gt; number   &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;col3 to col400 --&gt; these have been hot encoded by me and include just 0 or 1 as values&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;col400-col456 --&gt; these are features which have numeric values like 23,144 etc, all less than 1000 &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;DV --&gt; dependent variable is categorical with more than 2 values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have tried running rpart with a smaller set of 1000 rows and 456 cols, still no luck. I don't want to do any kind of feature elimination on the first run as as of now I have no idea whether they affect the dependent variable or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried mentioning &lt;code&gt;minbucket=25,20,15,10&lt;/code&gt; with no luck either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My objective is to get the resulting model of rpart on the entire data set. What are my options?&lt;/p&gt;&#xA;" OwnerUserId="32661" LastEditorUserId="29575" LastEditDate="2018-01-24T14:37:17.237" LastActivityDate="2018-01-30T03:48:47.563" Title="R Rpart taking too much time on data set" Tags="|r|multiclass-classification|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19209" PostTypeId="2" ParentId="14466" CreationDate="2017-05-25T15:40:41.013" Score="0" Body="&lt;p&gt;I think you'll need to use the &lt;code&gt;set.seed()&lt;/code&gt; function before you fit the model.&lt;/p&gt;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;ts=nnetar(Time_Series,P=3,lambda = 0,decay=0.001,maxit=150)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32664" LastEditorUserId="29169" LastEditDate="2020-11-27T23:47:45.663" LastActivityDate="2020-11-27T23:47:45.663" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19210" PostTypeId="2" ParentId="18577" CreationDate="2017-05-25T16:02:17.370" Score="2" Body="&lt;p&gt;As mentioned in the comments semi-supervised methods are worth having a read. If the time-series are labelled per time-step you can make use of dynamical systems and Gaussian processes. Some methods that could be useful using NN's are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pretraining a neural network using an auto encoder. You can use an RNN to encode the temporal aspect of the series (&lt;a href=&quot;https://papers.nips.cc/paper/5271-pre-training-of-recurrent-neural-networks-via-linear-autoencoders.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://papers.nips.cc/paper/5271-pre-training-of-recurrent-neural-networks-via-linear-autoencoders.pdf&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;GAN's (Generative adversarial networks) currently very popular way of making use of your copious amounts of unlabeled data by jointly training two networks (supervised vs unsupervised) to play a zero sum game vs each other. &lt;a href=&quot;https://arxiv.org/pdf/1611.09904.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1611.09904.pdf&lt;/a&gt; (some work on RNN-GAN's again if you want to account on the temporal structure of your data). Note these are very hard to train.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ladder Network &lt;a href=&quot;https://arxiv.org/pdf/1507.02672.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1507.02672.pdf&lt;/a&gt; . A very interesting semi-supervised method for regular fully connected networks. Which yielded very interesting results on the MNIST data set (only 300 labelled samples were used to reach state of the art results).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CNN's are known to be effective with time series too. Since they implement adaptive FIR filters on the data. It can be worth trialing in conjunction to RNN's.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All the methods above still require some reasonable amount of labelled data.&lt;/p&gt;&#xA;" OwnerUserId="32650" LastActivityDate="2017-05-25T16:02:17.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19211" PostTypeId="1" CreationDate="2017-05-25T16:32:41.547" Score="2" ViewCount="3464" Body="&lt;p&gt;I want to know if there is any way that I can crawl customer reviews for particular products from amazon without being blocked. At the moment, my crawler is blocked after a few times. Any idea will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="16872" LastActivityDate="2018-12-11T10:29:30.000" Title="Crawling customer reviews from Amazon" Tags="|scraping|crawling|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19217" PostTypeId="2" ParentId="19211" CreationDate="2017-05-25T19:20:12.410" Score="3" Body="&lt;p&gt;You are getting blocked because people do not want to waste server bandwidth on someone who is trying to exploit it without bringing significant profits. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to make your crawling less predictable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slow down the frequency with which you ping the server and vary the actions of your crawler. This will make it harder to detect as it will act less predictably and may be wrongfully identified as being a very quick human. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-05-25T19:20:12.410" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19218" PostTypeId="2" ParentId="19186" CreationDate="2017-05-25T20:58:54.877" Score="4" Body="&lt;p&gt;Yes, it's possible to use this confidence data.  However, I wouldn't recommend the approach you mention.  Instead, let me suggest a different approach.  Actually, I'll suggest two.  The first one is conceptually clean; the second is probably easier to implement; and they'll probably be approximately equivalent in practice.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Adjust the loss function&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You can adjust the loss function to reflect the confidence scores you have on the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, if you are using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross-entropy loss&lt;/a&gt;, there's a particularly clean way to do this.  Let me explain some background on the cross-entropy loss, then explain how.  We think of the label on the training instance as a probability distribution on labels.  In binary classification, such a distribution can be represented as a vector $(p_0,p_1)$ where $p_0$ represents the probability that the label is 0 and $p_1$ the probability that the label is 1.  Normally, we are given &quot;hard labels&quot;: if we know that the correct label on instance $x$ is 0, then that corresponds to the probability distribution $(1,0)$; if the correct label is 1, that's the distribution $(0,1)$.  The cross-entropy loss then compares the prediction from the classifier to this distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice thing about the cross-entropy loss is that it generates readily to comparing any two distributions.  Thus, if you have a confidence $0.8$ that the correct label for instance $x$ is 0, then that corresponds to a probability distribution $(0.8,0.2)$.  Now, you can compute the cross-entropy of the classifier's prediction with respect to the distribution $(0.8,0.2)$, and that is the contribution to the loss from training instance $x$.  Sum this over all instances in the training set, and you get an adjusted loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can train a classifier by minimizing this adjusted loss function, and that will directly incorporate all of the information in your confidence scores.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Use weights&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can use weights to reflect the confidence information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some classifiers allow you to specify a &lt;em&gt;weight&lt;/em&gt; for each instance in the training set.  The idea is that a misprediction for a particular instance is penalized proportionality to its weight, so instances with a high weight are more important to get right and instances with a low weight are less important.  Or, equivalently, the training procedure tries harder to avoid errors on instances with a high weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use weights to reflect confidence information.  Suppose you have an instanced $x$ in the training set that you think should have label 0, with confidence $0.8$.  You would add one copy of $(x,0)$ to the training set with weight $0.8$ (i.e., instance is $x$ and label is 0), and add one copy of $(x,1)$ to the training set with weight $0.2$ (i.e., instance is $x$ and label is 1).  Build up the training set in this way.  This doubles the size of your training set.  Now train a classifier, using these weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For classifiers that support weights, this should be easy to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can also show that it is effective and reasonable.  For instance, when using the cross-entropy loss to train a classifier, this approach using weights is equivalent to adjusting the loss function as highlighted above.  So, in that particular context, the two approaches are actually equivalent.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-25T20:58:54.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19220" PostTypeId="1" AcceptedAnswerId="19372" CreationDate="2017-05-25T23:48:26.343" Score="41" ViewCount="49594" Body="&lt;p&gt;I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is meant by 'small'? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, would a single-layer MLP with 100 hidden units be 'small'? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does our definition of 'small' change for recurrent architectures?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just &lt;a href=&quot;http://timdettmers.com/2014/09/21/how-to-build-and-use-a-multi-gpu-system-for-deep-learning/&quot; rel=&quot;noreferrer&quot;&gt;found a blog post&lt;/a&gt; (possibly outdated? It's from 2014):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;...Most network card[s] only work with memory that is registered with the CPU and so the GPU to GPU transfer between two nodes would be like this: GPU 1 to CPU 1 to Network Card 1 to Network Card 2 to CPU 2 to GPU 2. What this means is, if one chooses a slow network card then there might be no speedups over a single computer. Even with fast network cards, if the cluster is large, one does not even get speedups from GPUs when compared to CPUs as the GPUs just work too fast for the network cards to keep up with them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the reason why many big companies like Google and Microsoft are using CPU rather than GPU clusters to train their big neural networks. &quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at some point, according to this post, it could have been faster to use CPUs. Is this still the case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 2: Yes, that blog post may very well be outdated because:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it seems that GPUs within a node are connected via PCIe bus, so communication can happen at about 6GiB/s. (For example: &lt;a href=&quot;https://www.youtube.com/watch?v=el1iSlP1uOs&quot; rel=&quot;noreferrer&quot;&gt;https://www.youtube.com/watch?v=el1iSlP1uOs&lt;/a&gt;, about 35 minutes in). The speaker implies that this is faster than going from GPU1 to CPU to GPU2. It would mean the network card is no longer the bottleneck.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-05-31T20:09:35.133" LastActivityDate="2017-06-07T00:32:16.053" Title="Choosing between CPU and GPU for training a neural network" Tags="|neural-network|deep-learning|gpu|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19221" PostTypeId="2" ParentId="19207" CreationDate="2017-05-26T00:18:04.110" Score="1" Body="&lt;p&gt;Have you tried using &lt;code&gt;randomForest&lt;/code&gt; instead of &lt;code&gt;rpart&lt;/code&gt;? For example, let's assume you have two data.frames:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;train_data and test_data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my example, the last column is the class (and is a &lt;code&gt;factor&lt;/code&gt; variable), and all other variables (all other columns) are &lt;code&gt;numeric&lt;/code&gt;. For example, one training example might look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 5 10 12 1 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where 0 is the class label. You can do the following to run a random forest with 50 trees:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(randomForest)&#xA;model_pl_1 &amp;lt;- randomForest(train_data[,-ncol(train_data)], train_data[,ncol(train_data)], test_data[,-ncol(test_data)], test_data[,ncol(test_data)], ntree=50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-05-26T00:18:04.110" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19223" PostTypeId="1" CreationDate="2017-05-26T04:47:19.253" Score="1" ViewCount="738" Body="&lt;p&gt;I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to determine using exactly which algorithms in real life test cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if a particular algorithm is selected, how do we know other algorithm might have better accuracy and precision than this one &lt;/p&gt;&#xA;" OwnerUserId="31696" LastActivityDate="2017-05-28T16:51:49.383" Title="How to choose an appropriate Machine Learning algorithm?" Tags="|machine-learning|data|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ClosedDate="2017-05-30T14:49:12.257" ContentLicense="CC BY-SA 3.0" />
  <row Id="19224" PostTypeId="2" ParentId="19223" CreationDate="2017-05-26T05:22:35.720" Score="5" Body="&lt;p&gt;Try as many models as you have time for and pick the best one, otherwise use your intuition to narrow the candidates. If you have no intuition, start with a simple model and move to more complex models as performance requirements dictate. Often there are reasons beyond pure performance metrics to choose a model in the real world, such as simplicity, interpretability, and speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will not, however, have to pick between supervised and unsupervised learning, for these are drastically different things. If I ask you to group all similar items, how are you going to apply supervised learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might like &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/machine_learning_map/&quot; rel=&quot;noreferrer&quot;&gt;this map about choosing the right estimator&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-05-26T05:22:35.720" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19225" PostTypeId="1" CreationDate="2017-05-26T05:37:16.603" Score="1" ViewCount="273" Body="&lt;p&gt;I am trying to develop a Speech to Text conversion system using the Hidden Markov Model Toolkit (HTK). There are several possible methods to model HMMs for isolated word recognition:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using Phones (monophones and Triphones)&lt;/li&gt;&#xA;&lt;li&gt;Using Syllables&lt;/li&gt;&#xA;&lt;li&gt;Using Words&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;http://speech.ee.ntu.edu.tw/homework/DSP_HW2-1/htkbook.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;HTK Manual&lt;/a&gt; and &lt;a href=&quot;http://www.voxforge.org/home/dev/acousticmodels/linux/create/htkjulius/tutorial/data-prep/step-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;VoxForge Tutorial on HTK&lt;/a&gt; give us step by step instructions on how to model HMMs for every monophone or triphone. I have implemented this for an Indian Langauge (Kannada) with success. However, neither source tells us how the same can be implemented using Syllable-based HMM modelling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://dsc.ijs.si/files/papers/S304%20Hejtmanek.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a paper on ASR using context-dependent syllables&lt;/a&gt; :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Neither HTK nor LASER had the direct support for working with&#xA;  syllables we had to implement a transformation algorithm. This only&#xA;  transforms configuring files for monophone ASR into the form for&#xA;  syllable ASR.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They say we need to modify the configuration files to suit syllables instead of monophones. Following the &lt;a href=&quot;http://www.voxforge.org/home/dev/acousticmodels/linux/create/htkjulius/tutorial/data-prep/step-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;10 steps in the VoxForge Tutorial&lt;/a&gt; , The configuration files are as follows :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;config, wav_config&lt;/em&gt; : Specifies the configuration parameters for generating feature vectors from training wav files&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;sample.jconf&lt;/em&gt; : Configuration for using julius for Speech Recognition. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, changing these files in any way doesn't help in recognising syllables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I performed the steps similar to the phone based modelling i.e. create a monophone pronunciation dictionary and HMM training with 9 rounds. This is equivalent to the 1st 8 steps of the voxforge tutorial. In case of triphone modelling however, instead of creating triphones for every single word, I split every word into syllables and created trihphones (and biphones) from every syllable. You may not be able to read this language Kannada, but I'll show a few samples so you can see what I'm dealing with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During triphone modelling while I was just modelling for normal triphones, a part of my triphone dictionary &lt;code&gt;dict-tri&lt;/code&gt; was the following (which worked perfectly):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ಅತಿ            ಅ+ತ್ ಅ-ತ್+ಇ ತ್-ಇ sp&#xA;ಅದರ        ಅ+ದ್ ಅ-ದ್+ಅ ದ್-ಅ+ರ್ ಅ-ರ್+ಅ ರ್-ಅ sp&#xA;ಅದು            ಅ+ದ್ ಅ-ದ್+ಉ ದ್-ಉ sp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the new method of breking down the words into syllables and converting each syllable to mono, bi &amp;amp; triphones, the same dictionary is like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ಅತಿ          ಅ ತ್+ಇ ತ್-ಇ sp&#xA;ಅದರ         ಅ ದ್+ಅ ದ್-ಅ ರ್+ಅ ರ್-ಅ sp&#xA;ಅದು          ಅ ದ್+ಉ ದ್-ಉ sp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even though you may not be able to read the language, it should be clear that I aim to reduce the number of HMM models significantly. Yet while running julius, I got the following error : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    STAT: include config: sample.jconf&#xA;    STAT: jconf successfully finalized&#xA;    STAT: *** loading AM00 _default&#xA;    Stat: init_phmm: Reading in HMM definition&#xA;    Stat: rdhmmdef: ascii format HMM definition&#xA;    Stat: rdhmmdef: limit check passed&#xA;    Stat: check_hmm_restriction: an HMM with several arcs from initial state found: &quot;sp&quot;&#xA;    Stat: rdhmmdef: this HMM requires multipath handling at decoding&#xA;    Stat: rdhmmdef: no &amp;lt;SID&amp;gt; embedded&#xA;    Stat: rdhmmdef: assign SID by the order of appearance&#xA;    Stat: init_phmm: defined HMMs:   200&#xA;    Stat: init_phmm: loading ascii hmmlist&#xA;    Stat: init_phmm: logical names:   562 in HMMList&#xA;    Stat: init_phmm: base phones:    48 used in logical&#xA;    Stat: init_phmm: finished reading HMM definitions&#xA;    STAT: making pseudo bi/mono-phone for IW-triphone&#xA;    Stat: hmm_lookup: 5 pseudo phones are added to logical HMM list&#xA;    STAT: *** AM00 _default loaded&#xA;    STAT: *** loading LM00 _default&#xA;    STAT: reading [sample.dfa] and [sample.dict]...&#xA;    Error: voca_load_htkdict: line 3: triphone &quot;*-ಅ+ತ್&quot; or biphone &quot;ಅ+ತ್&quot; not found&#xA;    Error: voca_load_htkdict: line 3: triphone &quot;ಅ-ತ್+ಇ&quot; not found&#xA;    Error: voca_load_htkdict: the line content was: 2   [ಅತಿ]   ಅ ತ್ ಇ &#xA;    Error: voca_load_htkdict: line 4: triphone &quot;*-ಅ+ತ್&quot; or biphone &quot;ಅ+ತ್&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ಅ-ತ್+ಯ್&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ತ್-ಯ್+ಅ&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ಯ್-ಅ+ದ್&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ಅ-ದ್+ಭ್&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ದ್-ಭ್+ಉ&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ಭ್-ಉ+ತ್&quot; not found&#xA;    Error: voca_load_htkdict: line 4: triphone &quot;ಉ-ತ್+ಅ&quot; not found&#xA;    Error: voca_load_htkdict: the line content was: 2   [ಅತ್ಯದ್ಭುತ] ಅ ತ್ ಯ್ ಅ ದ್ ಭ್ ಉ ತ್ ಅ &#xA;    Error: voca_load_htkdict: line 5: triphone &quot;*-ಅ+ಥ್&quot; or biphone &quot;ಅ+ಥ್&quot; not found&#xA;    Error: voca_load_htkdict: line 5: triphone &quot;ಅ-ಥ್+ಅ&quot; not found&#xA;    Error: voca_load_htkdict: line 5: triphone &quot;ಥ್-ಅ+ವ್&quot; not found&#xA;    Error: voca_load_htkdict: line 5: triphone &quot;ಅ-ವ್+ಆ&quot; not found&#xA;    .&#xA;    .&#xA;    .&#xA;    (over 200 more lines of the same)&#xA;    .&#xA;    .&#xA;    Error: voca_load_htkdict: ಹ್-ಒ+ಯ್&#xA;    Error: voca_load_htkdict: ಹ್-ಒ+ರ್&#xA;    Error: voca_load_htkdict: ಹ್-ಒ+ಳ್&#xA;    Error: voca_load_htkdict: ಹ್-ಓ+ಗ್&#xA;    Error: voca_load_htkdict: ಹ್-ಓ+ರ್&#xA;    Error: voca_load_htkdict: ಹ್-ಯ್+ಅ&#xA;    Error: voca_load_htkdict: ಹ್-ಯ್+ಆ&#xA;    Error: voca_load_htkdict: end missing phones&#xA;    Error: init_voca: error in reading sample.dict: 748 words failed out of 7 words&#xA;    ERROR: failed to read dictionary &quot;sample.dict&quot;&#xA;    ERROR: m_fusion: some error occured in reading grammars&#xA;    ERROR: Error in loading model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What exactly have I done wrong? From the looks of it, the error states I have fed it triphones that have not been modeled. But all the &lt;code&gt;Error&lt;/code&gt;s displayed are triphones that I haven not even used! For reference, here is the list of &lt;strong&gt;ALL&lt;/strong&gt; mono, bi and triphones used to create syllables (triphones1 replacement): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ಲ್-ಅ+ಮ್&#xA;ತ್-ಇ+ಮ್&#xA;ಘ್-ಅ+ಮ್&#xA;ನ್-ಊ+ಮ್&#xA;ಛ್-ಈ&#xA;ಛ್-ಅ&#xA;ಛ್-ಆ&#xA;ಛ್-ಇ&#xA;ಭ್-ಆ&#xA;ಭ್-ಇ&#xA;ಭ್-ಅ&#xA;ಭ್-ಊ&#xA;ಭ್-ಈ&#xA;ಭ್-ಉ&#xA;ಕ್-ಒ+ಮ್&#xA;ಭ್-ಏ&#xA;ಬ್+ಓ&#xA;ಬ್+ಒ&#xA;ಞ್+ಆ&#xA;ಬ್-ಅ+ಮ್&#xA;ಬ್+ಔ&#xA;ಬ್+ಇ&#xA;ಬ್+ಆ&#xA;ಬ್+ಅ&#xA;ಬ್+ಉ&#xA;ಬ್+ಈ&#xA;ಬ್+ಏ&#xA;ಬ್+ಎ&#xA;ರ್-ಒ+ಮ್&#xA;ತ್+ಊ&#xA;ತ್+ಉ&#xA;ಲ್-ಆ+ಮ್&#xA;ತ್+ಏ&#xA;ತ್+ಎ&#xA;ತ್+ಇ&#xA;ತ್+ಆ&#xA;ತ್+ಅ&#xA;ತ್+ಐ&#xA;ಕ್-ಇ+ಮ್&#xA;ಹ್-ಒ&#xA;ಹ್-ಓ&#xA;ಹ್-ಎ&#xA;ಹ್-ಈ&#xA;ಹ್-ಉ&#xA;ಹ್-ಆ&#xA;ಹ್-ಇ&#xA;ಖ್-ಅ+ಮ್&#xA;ಹ್-ಅ&#xA;ವ್-ಒ&#xA;ವ್-ಐ&#xA;ಖ್+ಊ&#xA;ಖ್&#xA;ವ್-ಉ&#xA;ವ್-ಎ&#xA;ವ್-ಏ&#xA;ಢ್+ಇ&#xA;ಢ್+ಆ&#xA;ವ್-ಆ&#xA;ವ್-ಇ&#xA;ವ್-ಅ&#xA;ಕ್-ಆ+ಮ್&#xA;ಹ್-ಅ+ಮ್&#xA;ಠ್-ಈ&#xA;ಠ್-ಏ&#xA;ಠ್-ಅ&#xA;ಳ್-ಎ+ಮ್&#xA;ಸ್-ಎ+ಮ್&#xA;ಸ್-ಅ&#xA;ಸ್-ಇ&#xA;ಸ್-ಆ&#xA;ಸ್-ಉ&#xA;ಸ್-ಈ&#xA;ದ್-ಅ+ಮ್&#xA;ಸ್-ಊ&#xA;ಶ್-ಇ&#xA;ಶ್-ಆ&#xA;ಶ್-ಅ&#xA;ಸ್-ಎ&#xA;ಸ್-ಐ&#xA;ಸ್-ಓ&#xA;ಸ್-ಒ&#xA;ಸ್-ಔ&#xA;ಗ್-ಅ+ಮ್&#xA;ಡ್-ಉ&#xA;ಶ್+ಅ&#xA;ಶ್+ಇ&#xA;ಶ್+ಆ&#xA;ಡ್-ಇ&#xA;ದ್&#xA;ಡ್-ಅ&#xA;ಡ್-ಎ&#xA;ಪ್-ಔ&#xA;ಪ್-ಓ&#xA;ಪ್-ಐ&#xA;ಧ್+ಊ&#xA;ಪ್-ಎ&#xA;ಧ್+ಈ&#xA;ಧ್+ಉ&#xA;ಪ್-ಊ&#xA;ಪ್-ಉ&#xA;ಪ್-ಇ&#xA;ಪ್-ಆ&#xA;ಪ್-ಅ&#xA;ಯ್-ಅ&#xA;ನ್&#xA;ಧ್+ಇ&#xA;ಧ್+ಅ&#xA;ಜ್&#xA;ಧ್+ಆ&#xA;ಷ್-ಈ&#xA;ತ್-ಅ+ಮ್&#xA;ಷ್-ಏ&#xA;ಷ್-ಅ&#xA;ಷ್-ಆ&#xA;ಷ್-ಇ&#xA;ಛ್+ಇ&#xA;sp&#xA;ಹ್&#xA;ಞ್-ಆ&#xA;ಥ್-ಈ&#xA;ಥ್-ಎ&#xA;ಥ್-ಏ&#xA;ಥ್-ಆ&#xA;ಥ್-ಇ&#xA;ಥ್-ಅ&#xA;ಚ್&#xA;ಯ್&#xA;ಮ್&#xA;ಟ್&#xA;ಳ್+ಎ&#xA;ಜ್+ಇ&#xA;ಜ್+ಆ&#xA;ಜ್+ಅ&#xA;ಳ್+ಉ&#xA;ಳ್+ಆ&#xA;ಳ್+ಇ&#xA;ಜ್+ಉ&#xA;ಳ್+ಅ&#xA;ಜ್+ಎ&#xA;ಖ್+ಐ&#xA;ಜ್+ಓ&#xA;ಜ್+ಐ&#xA;ಊ+ಮ್&#xA;ಳ್&#xA;ಶ್&#xA;ರ್-ಎ+ಮ್&#xA;ರ್-ಓ&#xA;ಆ-ಮ್&#xA;ರ್-ಉ&#xA;ಗ್+ಒ&#xA;ರ್-ಊ&#xA;ರ್-ಏ&#xA;ರ್-ಎ&#xA;ರ್-ಅ&#xA;ಟ್-ಊ&#xA;ರ್-ಇ&#xA;ರ್-ಆ&#xA;ಚ್+ಉ&#xA;ಚ್+ಈ&#xA;ಟ್-ಋ&#xA;ಬ್-ಎ+ಮ್&#xA;ಚ್+ಎ&#xA;ಚ್+ಅ&#xA;ಚ್+ಇ&#xA;ಚ್+ಆ&#xA;ದ್-ಒ+ಮ್&#xA;ನ್+ಓ&#xA;ನ್+ಒ&#xA;ನ್+ಐ&#xA;ನ್+ಇ&#xA;ನ್+ಆ&#xA;ನ್+ಅ&#xA;ಗ್-ಒ+ಮ್&#xA;ನ್+ಏ&#xA;ನ್+ಎ&#xA;ನ್+ಊ&#xA;ನ್+ಉ&#xA;ನ್+ಈ&#xA;ತ್-ಉ+ಮ್&#xA;ಧ್-ಈ&#xA;ಧ್-ಉ&#xA;ಧ್-ಊ&#xA;ಟ್+ಆ&#xA;ಟ್+ಇ&#xA;ಟ್+ಅ&#xA;ಟ್+ಊ&#xA;ಟ್+ಋ&#xA;ಟ್+ಈ&#xA;ಟ್+ಉ&#xA;ಧ್-ಅ&#xA;ಧ್-ಆ&#xA;ಧ್-ಇ&#xA;ಗ್+ಊ&#xA;ಚ್-ಎ&#xA;ಗ್+ಈ&#xA;ಗ್+ಉ&#xA;ಗ್+ಎ&#xA;ಗ್+ಏ&#xA;ಚ್-ಉ&#xA;ಚ್-ಈ&#xA;ಚ್-ಇ&#xA;ಚ್-ಆ&#xA;ಚ್-ಅ&#xA;ಗ್+ಆ&#xA;ಗ್+ಇ&#xA;ಖ್+ಓ&#xA;ಗ್+ಅ&#xA;ಖ್+ಏ&#xA;ತ್+ಈ&#xA;ಖ್+ಉ&#xA;ಟ್-ಅ&#xA;ಟ್-ಆ&#xA;ಟ್-ಇ&#xA;ಖ್+ಅ&#xA;ಟ್-ಉ&#xA;ಖ್+ಇ&#xA;ಖ್+ಆ&#xA;ನ್-ಅ+ಮ್&#xA;ಬ್-ಔ&#xA;ವ್+ಐ&#xA;ವ್+ಒ&#xA;ಒ-ಮ್&#xA;ಫ್-ಎ&#xA;ಸ್-ಅ+ಮ್&#xA;ವ್+ಉ&#xA;ವ್+ಎ&#xA;ವ್+ಏ&#xA;ವ್+ಅ&#xA;ವ್+ಆ&#xA;ಯ್-ಅ+ಮ್&#xA;ಹ್+ಉ&#xA;ಕ್-ಅ+ಮ್&#xA;ದ್-ಇ+ಮ್&#xA;ವ್+ಇ&#xA;ಖ್-ಇ&#xA;ಡ್-ಅ+ಮ್&#xA;ಥ್+ಅ&#xA;ಖ್-ಅ&#xA;ಸ್+ಇ&#xA;ಸ್+ಆ&#xA;ಸ್+ಅ&#xA;ಫ್+ಅ&#xA;ರ್+ಎ&#xA;ಫ್+ಎ&#xA;ಸ್+ಎ&#xA;ಎ+ಮ್&#xA;ಸ್+ಉ&#xA;ಸ್+ಈ&#xA;ಖ್-ಆ&#xA;ಸ್+ಔ&#xA;ಸ್+ಓ&#xA;ಸ್+ಒ&#xA;ಸ್+ಐ&#xA;ಗ್-ಆ+ಮ್&#xA;ಖ್-ಓ&#xA;ದ್+ಋ&#xA;ಠ್+ಏ&#xA;ಠ್+ಈ&#xA;ಠ್+ಅ&#xA;ಣ್+ಎ&#xA;ಣ್+ಉ&#xA;ಣ್+ಆ&#xA;ಣ್+ಇ&#xA;ಣ್+ಅ&#xA;ಏ-ಮ್&#xA;ಲ್+ಅ&#xA;ಲ್+ಇ&#xA;ಲ್+ಆ&#xA;ಲ್+ಉ&#xA;ರ್-ಇ+ಮ್&#xA;ಲ್+ಊ&#xA;ಲ್+ಏ&#xA;ಲ್+ಎ&#xA;ನ್-ಇ+ಮ್&#xA;ಬ್-ಅ&#xA;ಬ್-ಇ&#xA;ಬ್-ಆ&#xA;ಬ್-ಏ&#xA;ಬ್-ಎ&#xA;ಬ್-ಉ&#xA;ಬ್-ಈ&#xA;ಛ್+ಆ&#xA;ರ್-ಆ+ಮ್&#xA;ಛ್+ಅ&#xA;ಧ್&#xA;ಬ್-ಓ&#xA;ಬ್-ಒ&#xA;ರ್-ಅ+ಮ್&#xA;ಕ್-ಏ+ಮ್&#xA;ಛ್+ಈ&#xA;ಷ್+ಏ&#xA;ದ್-ಒ&#xA;ದ್-ಋ&#xA;ದ್-ಊ&#xA;ದ್-ಉ&#xA;ದ್-ಈ&#xA;ದ್-ಏ&#xA;ದ್-ಎ&#xA;ದ್-ಇ&#xA;ದ್-ಆ&#xA;ದ್-ಅ&#xA;ಗ್&#xA;ತ್-ಐ&#xA;ತ್-ಏ&#xA;ತ್-ಎ&#xA;ತ್-ಉ&#xA;ತ್-ಈ&#xA;ತ್-ಊ&#xA;ತ್-ಅ&#xA;ತ್-ಇ&#xA;ತ್-ಆ&#xA;ಝ್+ಓ&#xA;ಯ್-ಆ+ಮ್&#xA;ಘ್-ಅ&#xA;ಘ್-ಆ&#xA;ತ್-ಎ+ಮ್&#xA;ಘ್-ಓ&#xA;ಢ್+ಅ&#xA;ಛ್-ಅ+ಮ್&#xA;ಮ್+ಔ&#xA;ಸ್-ಇ+ಮ್&#xA;ಮ್+ಐ&#xA;ಮ್+ಒ&#xA;ಮ್+ಅ&#xA;ಮ್+ಇ&#xA;ಮ್+ಆ&#xA;ಮ್+ಏ&#xA;ಮ್+ಎ&#xA;ಮ್+ಉ&#xA;ಮ್+ಈ&#xA;ಮ್+ಊ&#xA;ಪ್+ಐ&#xA;ಪ್+ಓ&#xA;ಇ-ಮ್&#xA;ನ್-ಒ+ಮ್&#xA;ಪ್+ಉ&#xA;ಪ್+ಊ&#xA;ಪ್+ಎ&#xA;ಪ್+ಅ&#xA;ಪ್+ಇ&#xA;ಪ್+ಆ&#xA;ಘ್+ಆ&#xA;ಘ್+ಅ&#xA;ಮ್-ಅ+ಮ್&#xA;ಇ+ಮ್&#xA;ಘ್+ಓ&#xA;ಒ+ಮ್&#xA;ಔ-ಮ್&#xA;ಪ್-ಉ+ಮ್&#xA;ಬ್&#xA;ಳ್+ಊ&#xA;ಭ್+ಏ&#xA;ಡ್&#xA;ಬ್-ಇ+ಮ್&#xA;ಟ್-ಈ&#xA;ಢ್-ಇ&#xA;ಢ್-ಆ&#xA;ಢ್-ಅ&#xA;ಯ್-ಇ+ಮ್&#xA;ಜ್+ಈ&#xA;ಷ್-ಎ&#xA;ಫ್&#xA;ತ್-ಊ+ಮ್&#xA;ಥ್+ಈ&#xA;ಖ್-ಊ&#xA;ಖ್-ಉ&#xA;ಖ್-ಏ&#xA;ಥ್+ಎ&#xA;ಥ್+ಏ&#xA;ರ್+ಓ&#xA;ರ್+ಒ&#xA;ಥ್+ಆ&#xA;ಥ್+ಇ&#xA;ರ್+ಏ&#xA;ನ್-ಎ+ಮ್&#xA;ಕ್&#xA;ರ್+ಊ&#xA;ರ್+ಉ&#xA;ರ್+ಈ&#xA;ರ್+ಇ&#xA;ರ್+ಆ&#xA;ರ್+ಅ&#xA;ಖ್-ಐ&#xA;ಳ್-ಅ&#xA;ಳ್-ಆ&#xA;ಳ್-ಇ&#xA;ಲ್-ಇ+ಮ್&#xA;ಳ್-ಎ&#xA;ಹ್-ಇ+ಮ್&#xA;ಳ್-ಉ&#xA;ಳ್-ಊ&#xA;ಸ್+ಊ&#xA;ಭ್+ಈ&#xA;ಲ್&#xA;ಔ&#xA;ಐ&#xA;ಓ&#xA;ಒ&#xA;ಎ-ಮ್&#xA;ಅ&#xA;ಇ&#xA;ಆ&#xA;ಉ-ಮ್&#xA;ಸ್+ಏ&#xA;ಏ&#xA;ಎ&#xA;ಉ&#xA;ಈ&#xA;ಋ&#xA;ಊ&#xA;ಷ್&#xA;ನ್-ಓ&#xA;ನ್-ಒ&#xA;ಭ್+ಅ&#xA;ಭ್+ಆ&#xA;ಭ್+ಇ&#xA;ತ್&#xA;ಭ್+ಉ&#xA;ಭ್+ಊ&#xA;ದ್+ಈ&#xA;ನ್-ಐ&#xA;ದ್+ಏ&#xA;ದ್+ಎ&#xA;ದ್+ಉ&#xA;ಜ್-ಐ&#xA;ಜ್-ಓ&#xA;ದ್+ಊ&#xA;ದ್+ಅ&#xA;ದ್+ಇ&#xA;ದ್+ಆ&#xA;ಜ್-ಅ&#xA;ಜ್-ಇ&#xA;ಜ್-ಆ&#xA;ನ್-ಅ&#xA;ನ್-ಇ&#xA;ನ್-ಆ&#xA;ನ್-ಉ&#xA;ನ್-ಈ&#xA;ನ್-ಊ&#xA;ಜ್-ಉ&#xA;ಜ್-ಈ&#xA;ನ್-ಏ&#xA;ನ್-ಎ&#xA;ಲ್-ಇ&#xA;ಲ್-ಆ&#xA;ಲ್-ಅ&#xA;ಯ್+ಆ&#xA;ಯ್+ಇ&#xA;ಯ್+ಅ&#xA;ಲ್-ಏ&#xA;ಲ್-ಎ&#xA;ಯ್+ಉ&#xA;ಸ್&#xA;ಲ್-ಊ&#xA;ಲ್-ಉ&#xA;ಯ್+ಓ&#xA;ವ್&#xA;ಆ+ಮ್&#xA;ಕ್-ಎ+ಮ್&#xA;ಝ್-ಓ&#xA;ಊ-ಮ್&#xA;ವ್-ಒ+ಮ್&#xA;ಗ್-ಈ&#xA;ಗ್-ಉ&#xA;ಗ್-ಊ&#xA;ಜ್-ಎ&#xA;ರ್&#xA;ಗ್-ಎ&#xA;ಗ್-ಏ&#xA;ಕ್-ಒ&#xA;ಕ್-ಓ&#xA;ಗ್-ಅ&#xA;ಗ್-ಆ&#xA;ಗ್-ಇ&#xA;ಕ್-ಊ&#xA;ಕ್-ಋ&#xA;ಕ್-ಈ&#xA;ಕ್-ಉ&#xA;ಕ್-ಎ&#xA;ಗ್-ಒ&#xA;ದ್+ಒ&#xA;ಕ್-ಆ&#xA;ಕ್-ಇ&#xA;ಕ್-ಅ&#xA;ಣ್&#xA;ಳ್-ಇ+ಮ್&#xA;ಪ್+ಔ&#xA;ವ್-ಅ+ಮ್&#xA;ಲ್-ಏ+ಮ್&#xA;ಹ್+ಒ&#xA;ಹ್+ಓ&#xA;ಪ್-ಆ+ಮ್&#xA;ಹ್+ಅ&#xA;ಹ್+ಆ&#xA;ಹ್+ಇ&#xA;ಫ್-ಅ&#xA;ಹ್+ಎ&#xA;ಹ್+ಈ&#xA;ಕ್-ಉ+ಮ್&#xA;ರ್-ಒ&#xA;ಯ್+ಎ&#xA;ಕ್+ಒ&#xA;ಕ್+ಓ&#xA;ಯ್+ಏ&#xA;ಕ್+ಈ&#xA;ಕ್+ಉ&#xA;ಕ್+ಊ&#xA;ಕ್+ಋ&#xA;ಕ್+ಎ&#xA;ಕ್+ಏ&#xA;ಕ್+ಅ&#xA;ಕ್+ಆ&#xA;ಕ್+ಇ&#xA;ಷ್+ಈ&#xA;ಷ್+ಎ&#xA;ಮ್-ಅ&#xA;ರ್-ಈ&#xA;ಹ್-ಒ+ಮ್&#xA;ಷ್+ಆ&#xA;ಷ್+ಇ&#xA;ಷ್+ಅ&#xA;ಣ್-ಅ&#xA;ಣ್-ಆ&#xA;ಣ್-ಇ&#xA;ಮ್-ಉ+ಮ್&#xA;ಮ್-ಊ&#xA;ಣ್-ಎ&#xA;ಣ್-ಉ&#xA;ಡ್+ಎ&#xA;ಡ್+ಉ&#xA;ಡ್+ಅ&#xA;ಡ್+ಇ&#xA;ಪ್&#xA;ಅ-ಮ್&#xA;ಟ್-ಉ+ಮ್&#xA;ಪ್-ಅ+ಮ್&#xA;ಸ್-ಔ+ಮ್&#xA;ಮ್-ಐ&#xA;ಸ್-ಏ&#xA;ಮ್-ಇ&#xA;ಮ್-ಆ&#xA;ಯ್-ಆ&#xA;ಯ್-ಇ&#xA;ಯ್-ಉ&#xA;ಮ್-ಉ&#xA;ಮ್-ಈ&#xA;ಮ್-ಏ&#xA;ಮ್-ಎ&#xA;ಯ್-ಎ&#xA;ಯ್-ಏ&#xA;ಮ್-ಒ&#xA;ಯ್-ಓ&#xA;ಮ್-ಔ&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Regardless, The error I recieved should be common to every langauge. What is wrong exactly? &lt;strong&gt;What should I do to implement Syllable modeling in HTK?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="30435" LastActivityDate="2017-05-26T05:37:16.603" Title="How to train Syllables instead of phones using HTK?" Tags="|machine-learning|python|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19226" PostTypeId="2" ParentId="9131" CreationDate="2017-05-26T06:58:23.517" Score="0" Body="&lt;p&gt;One thing you can do is calculate the time interval i.e Current time - Observation Time. Let's say this is t. Now create a weight vector as c/t or c*exp(-t). You can use weights in caret using &lt;code&gt;caret::train &amp;gt; weights&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-05-26T06:58:23.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19227" PostTypeId="1" CreationDate="2017-05-26T09:04:20.050" Score="5" ViewCount="55404" Body="&lt;p&gt;I have one column in the first dataframe called 'id' and another column in the second dataframe called 'first_id' which refers to the id from the first dataframe. I need to create a new column which has value 1  if the id and first_id match, otherwise it is 0.  I tried this but I am getting an error &lt;code&gt;ValueError: Length of values does not match length of index&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df2['new'] = np.where(df2[df2.first_id.isin(df1.id.values)], 1, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand why is it happening, because &lt;code&gt;df2&lt;/code&gt; and &lt;code&gt;df2[df2.first_id.isin(df1.id.values)]&lt;/code&gt; are of different lengths but I can't make them the same. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="32690" LastActivityDate="2018-01-22T01:59:20.637" Title="Create a new column based on two columns from two different dataframes" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19228" PostTypeId="2" ParentId="19227" CreationDate="2017-05-26T10:15:56.590" Score="1" Body="&lt;p&gt;Something like this maybe?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1 = pd.DataFrame(np.random.randint(0,5,size=(100, 1)), columns=list('A')) # random 1 column df&#xA;df2 = pd.DataFrame(np.random.randint(0,5,size=(100, 1)), columns=list('B')) # random 1 column df&#xA;df2[&quot;new&quot;] = df2.apply(lambda row: 1 if row[0] == df1[&quot;A&quot;][row.name] else 0, axis = 1) # lambda function to check if they match. row.name gets the index&#xA;df2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32164" LastActivityDate="2017-05-26T10:15:56.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19231" PostTypeId="2" ParentId="19184" CreationDate="2017-05-26T11:51:59.607" Score="1" Body="&lt;p&gt;That looks a double axis or dual axis graph but isn't well done or I don't understand clouds well enough. It is almost like a &lt;a href=&quot;https://en.wikipedia.org/wiki/Sankey_diagram&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sankey&lt;/a&gt; diagram. Problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What does the X axis convey, if anything?&lt;/li&gt;&#xA;&lt;li&gt;It appears that the cloud types are relatively stratified but I'm not sure what causes the waviness over the X axis. Is it related to the two lines for temperature and pressure?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Compare that to a phase diagram like below. I'm not a chemist, but this is a lot easier for me to interpret. The x axis is the temperature, the y axis is pressure, and the lines clearly show where the material switches phases, similar to cloud types. There isn't ambiguity at any point about which phase would be in place. Now try figuring out what type of cloud you would have in your diagram at 30C and 8psi.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YIkj1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YIkj1.png&quot; alt=&quot;phase diagram&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-26T11:51:59.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19232" PostTypeId="2" ParentId="18938" CreationDate="2017-05-26T13:30:07.773" Score="2" Body="&lt;p&gt;Some time ago, I stumbled upon the &lt;a href=&quot;http://pan.webis.de/clef16/pan16-web/author-obfuscation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Author Obfuscation&lt;/a&gt; task of PAN16 workshop. The task at hand is given a document, paraphrase it so that its writing style does not match that of its original author.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although, it's not exactly what you want, you can take a look at their proceedings and find useful tips regarding authors style and how to &quot;change&quot; it (i.e. in your case, an &quot;author&quot; is a collection of documents from the same source).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;a href=&quot;http://www.uni-weimar.de/medien/webis/events/pan-16/pan16-papers-final/pan16-author-obfuscation/mihaylova16-notebook.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;one approach&lt;/a&gt;&#xA;denotes specific metrics that define an author's style (average sentence word count, use of stopwords, pos-type frequencies). You could use the target's scores in these metrics as your gold standard and try to modify the source text, in ways discussed in the proceedings of the workshop, so as to mask your source text to match the target one (in terms of these metrics).&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-05-26T13:30:07.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19233" PostTypeId="2" ParentId="19119" CreationDate="2017-05-26T13:43:42.453" Score="1" Body="&lt;p&gt;As you already noticed, Topic Models &lt;strong&gt;aren't reproducible&lt;/strong&gt; due to the probabilistic algorithm. I did some topic models on a large corpus (10k documents, 30M words), and generally, between two different topic models about 10% of the topics were incomparable. My corpus is mainly English, but contains some Latin texts, and for a given number of topics I had between 1 and 3 topics consisting of Latin language words predominantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What to do now depends on what you want to learn from your Topic Model. When you want to &quot;interpret&quot; the Topic Model (using methods from the humanities), the best thing is to commit to one topic model (i.e., fixing the random seed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you want to study some derived measures (like topic diversity), make sure that the derived measures taken from different topic models are &quot;stable&quot;, i.e., centred around one value with small statistical fluctuations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try to create an ensemble of topic models and try some techniques to get an &quot;average topic model&quot; out of it. I have no experience yet in this business and cannot suggest techniques here.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2017-05-26T13:43:42.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19234" PostTypeId="2" ParentId="19220" CreationDate="2017-05-26T14:27:47.140" Score="3" Body="&lt;p&gt;I'll first reference some quotes from similar questions:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When it comes to matrix operations, you don't think twice, you always opt for GPUs. &lt;a href=&quot;https://stackoverflow.com/a/32426899/4576519&quot;&gt;source&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&amp;zwnj;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The parallel architecture in a GPU is well adapted for vector and matrix operations.  &lt;a href=&quot;https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning/answer/Chomba-Bupe?srid=uwZSp&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So if you read through these questions, you'll see that they advise to use GPU regardless of the case; it will always provide some improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason you may have read that 'small' networks should be trained with CPU, is because implementing GPU training for just a small network might take more time than simply training with CPU - that doesn't mean GPU will be slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 100-hidden unit network is &lt;em&gt;kind of small&lt;/em&gt;, i'd call it a small network relative to the &lt;em&gt;big&lt;/em&gt; deep networks out there. Recurrent architectures (mostly) have more synapses thant feed forward networks, so a 100-hidden units RNN is 'bigger' than a 100-hidden unit FFN. &lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-05-26T14:27:47.140" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19235" PostTypeId="2" ParentId="19220" CreationDate="2017-05-26T14:36:51.297" Score="5" Body="&lt;p&gt;The CPU is the manager of the branch, he can do a bit of everything, but he is not great at much except delegating tasks. However, the GPU is a dedicated mathematician hiding in your machine. If you are doing any math heavy processes then you should use your GPU. Always. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using any popular programming language for machine learning such as python or MATLAB it is a one-liner of code to tell your computer that you want the operations to run on your GPU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also make sure to use all the cores of your machine. This means making use of parallel computing. Especially for neural networks where operations can be done independently, this is going to increase your speed immensely. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-05-26T14:36:51.297" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19236" PostTypeId="2" ParentId="17868" CreationDate="2017-05-26T14:44:37.137" Score="2" Body="&lt;p&gt;You may use logistic regression if your dataset follows some clear polynomial curve, which you can verify by plotting the data and looking how it is distributed. In that case, even if the data is skewed you might get a good classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that you achieve higher accuracy - 99% as you mentioned - by always predicting one of the values is the reason why classifier accuracy is not used to evaluate a logistic regression model. Instead Precision and Recall give you much better insight into the quality of the classifier because they measure both how many of the examples it classified as positive were actually positive and how many of the positive examples in the training set it classified correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tradeoff between both metrics help you find a classifier which is both precise and can generalize.&lt;/p&gt;&#xA;" OwnerUserId="32177" LastEditorUserId="32177" LastEditDate="2020-03-29T17:43:06.590" LastActivityDate="2020-03-29T17:43:06.590" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19237" PostTypeId="2" ParentId="17868" CreationDate="2017-05-26T14:57:43.073" Score="4" Body="&lt;p&gt;When evaluating your algorithms, especially when your dataset is unbalanced, you should use more metrics than just accuracy. The accuracy is how many examples you have correctly identified in total. As you have seen if you have an unbalanced dataset where 0.5% of your instances are 1's then this will result in 99.5% accuracy if you blindly set all your outputs as zeroes. This is obviously wrong albeit the high accuracy. The accuracy is calculated as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Accuracy = \frac{\sum{TP} + \sum{TN}}{\sum{TP} + \sum{TN} + \sum{FP} + \sum{FN}}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where TP is true positive, TN is true negatives, FP is false positives and FN is false negatives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to capture the performance of your unbalanced dataset you should look into the percentage of FP and FN you are calculating. You can do this using the sensitivity and the specificity. Calculate the sensitivity as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Sensitivity = \frac{\sum{TP} }{\sum{TP}  + \sum{FN} }$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the specificity as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Specificity = \frac{\sum{TN} }{\sum{TN}  + \sum{FP} }$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An ideal classifier should have the accuracy, specificity and sensitivity all be 1. This would mean every sample is correctly classified. In your case where you are getting very high false negatives, you will see that your sensitivity will be very low. This is a measure with which you can state that your algorithm is performing poorly. It is good form to always include these metrics in any statistical study you are doing. Accuracy alone is not sufficient to prove that you are obtaining good results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, there is the receiver-operator curve (ROC). This will tell you your false positive rate for any true positive rate. You can then calculate the area under this curve (AUC) to get a comparable metric of performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these should be used together when exclaiming the performance of your algorithm. The ROC and AUC can be omitted however leaving out the sensitivity and specificity of your algorithm is unwise.   &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2020-02-29T22:54:39.993" LastActivityDate="2020-02-29T22:54:39.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19238" PostTypeId="2" ParentId="19107" CreationDate="2017-05-26T14:58:14.110" Score="2" Body="&lt;p&gt;For the example you pose, the answers and discussion on this question might be helpful - some good points on options around labelling:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/10216/doc2vec-how-to-label-the-paragraphs-gensim&quot;&gt;Doc2Vec - How to label the paragraphs (gensim)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2017-05-26T14:58:14.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19240" PostTypeId="1" CreationDate="2017-05-26T16:01:45.647" Score="1" ViewCount="865" Body="&lt;p&gt;I used logistic regression to classify 3 classes of images: cars, dogs, cows. The accuracy is 100%, perfectly separated in clusters (scatter plots).&#xA;When I try to use the learner to predict 3 images (1 dog, 1 car, 1 cow), the predictor simply assign the probability of 1/3 to each image. How can I fix it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/j96m0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/j96m0.png&quot; alt=&quot;orange3 flows&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32701" LastEditorUserId="31513" LastEditDate="2017-05-29T04:59:44.793" LastActivityDate="2019-07-12T14:05:09.277" Title="Using Orange3 to predict image class" Tags="|image-classification|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19241" PostTypeId="1" CreationDate="2017-05-26T16:41:32.213" Score="2" ViewCount="5843" Body="&lt;p&gt;I believe that I implemented MDTW in python here but I don't know if I did it correctly. The results seem intuitive. Can someone look at this code and tell me if you see anything wrong? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of the papers I read on the subject kinda go over my head, but this just takes the mse of all column vectors in matrix &lt;em&gt;s1&lt;/em&gt; and &lt;em&gt;s2&lt;/em&gt; at each time point inside a given window &lt;em&gt;w&lt;/em&gt; instead of single dimensional dynamic time warping which just takes the mse error of each value between two vectors and finds the smallest one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use panda's Panel, which is essentially a 3D version of a DataFrame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, after I have this, I want to do clustering with some multidimensional time series. Thoughts on which clustering algorithm to run? Kmeans? Hierarchical? I'll start by building a dendrogram at least. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def MDTWDistance(s1, s2, window=10, num_columns=1):&#xA;    DTW={}&#xA;&#xA;    w = max(window, abs(len(s1)-len(s2)))&#xA;&#xA;    for i in range(-1,len(s1)):&#xA;        for j in range(-1,len(s2)):&#xA;            DTW[(i, j)] = float('inf')&#xA;    DTW[(-1, -1)] = 0&#xA;&#xA;    for i in range(len(s1)):&#xA;        for j in range(max(0, i-w), min(len(s2), i+w)):&#xA;            #print &quot;Finding Distance of&quot;, s1.loc[i], s2.loc[j]&#xA;            dist= mdist(s1.loc[i], s2.loc[j], num_columns)&#xA;            #print &quot;Dist&quot;, dist&#xA;            #print i, j, dist&#xA;            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])&#xA;&#xA;    return np.sqrt(DTW[len(s1)-1, len(s2)-1])&#xA;&#xA;def mdist(a, b, num_col):&#xA;    dist = 0&#xA;    for col in range(num_col):&#xA;        #print &quot;Finding Distance of&quot;, a[col], b[col]&#xA;        dist = dist + (a[col]-b[col])**2&#xA;    return dist&#xA;&#xA;x=np.linspace(0,50,100)&#xA;ts1=pd.Series(3.1*np.sin(x/1.5)+3.5)&#xA;ts2=pd.Series(2.2*np.sin(x/3.5+2.4)+3.2)&#xA;ts3=pd.Series(0.04*x+8.0)&#xA;ts4=pd.Series(0.048*x+8.6)&#xA;ts5=pd.Series(-0.17*x+4.1)&#xA;ts6=pd.Series(-0.14*x+4.5)&#xA;&#xA;ts1.plot()&#xA;ts2.plot()&#xA;ts3.plot()&#xA;ts4.plot()&#xA;ts5.plot()&#xA;ts6.plot()&#xA;&#xA;plt.ylim(-4,12)&#xA;plt.legend(['ts1','ts2','ts3','ts4','ts5','ts6'])&#xA;plt.show()&#xA;&#xA;timeSeries = pd.Panel({0:pd.DataFrame(np.transpose([ts1, ts2])),&#xA;                 1:pd.DataFrame(np.transpose([ts3, ts4])),&#xA;                   2:pd.DataFrame(np.transpose([ts5, ts6]))&#xA;                  })&#xA;&#xA;print &quot;0 and 1:&quot;,MDTWDistance(timeSeries[0], timeSeries[1],window=10, num_columns=2)&#xA;print &quot;0 and 2:&quot;,MDTWDistance(timeSeries[0], timeSeries[2],window=10, num_columns=2)&#xA;print &quot;1 and 2:&quot;,MDTWDistance(timeSeries[1], timeSeries[2],window=10, num_columns=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is as follow: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 and 1: 89.354619036&#xA;0 and 2: 58.8268328591&#xA;1 and 2: 133.434513377&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With the graph: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vxHZh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vxHZh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I found the distance using an MDTW package in R with the following code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import rpy2.robjects.numpy2ri&#xA;from rpy2.robjects.packages import importr&#xA;rpy2.robjects.numpy2ri.activate()&#xA;&#xA;# Set up our R namespaces&#xA;R = rpy2.robjects.r&#xA;DTW = importr('dtw')&#xA;&#xA;# Generate our data&#xA;idx = np.linspace(0, 2*np.pi, 100)&#xA;template = np.cos(idx)&#xA;query = np.sin(idx) + np.array(R.runif(100))/10&#xA;&#xA;for i,j in ([0,1], [0,2], [1,2]):&#xA;    print &quot;From&quot;, i, &quot;to&quot;, j, R.dtw(np.transpose(np.array(timeSeries[i])), np.transpose(np.array(timeSeries[j])), keep=True).rx('distance')[0][0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which output: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;From 0 to 1 186.623310713&#xA;From 0 to 2 119.769089068&#xA;From 1 to 2 272.849560995&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So these numbers are about double the distances I have... so I'm not far off, but there's something wrong with what I'm doing. Halp.&lt;/p&gt;&#xA;" OwnerUserId="31676" LastEditorUserId="31676" LastEditDate="2017-05-26T19:04:27.163" LastActivityDate="2017-12-01T17:16:18.127" Title="Multidimensional Dynamic Time Warping Implementation in Python - confirm?" Tags="|python|clustering|time-series|unsupervised-learning|distance|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19242" PostTypeId="2" ParentId="17286" CreationDate="2017-05-26T17:27:17.310" Score="9" Body="&lt;p&gt;I will assume by &lt;code&gt;C1&lt;/code&gt;, &lt;code&gt;C2&lt;/code&gt;, etc, you mean convolutional layers, and by &lt;code&gt;P1&lt;/code&gt; ,&lt;code&gt;P2&lt;/code&gt; you mean pooling layers, and &lt;code&gt;FC&lt;/code&gt; means fully connected layers.&lt;/p&gt;&#xA;&lt;p&gt;We can calculate the memory required for a forward pass like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;One image&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;If you're working with float32 values, then following the link provided above by @Alexandru Burlacu you have:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Input&lt;/code&gt;: 50x50x3 = 7,500 = 7.5K&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;C1&lt;/code&gt;: 50x50x32 = 80,000 = 80K&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;P1&lt;/code&gt;: 25x25x32 = 20,000 = 20K&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;C2&lt;/code&gt;: 25x25x64 = 40,000 = 40K&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;P2&lt;/code&gt;: 12x12x64 = 9,216 = 9.2K &amp;lt;- This is a problem (and my approximation is a very hand-wavy guess here). Instead of working with 50, 25, '12.5', it would make more sense to work with multiples of 32. I've heard working with multiples of 32 is also more efficient from a memory standpoint. The reason this is a bad idea is 2x2 pooling doesn't divide the space properly, as far as I can tell. Feel free to correct me if I'm wrong.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;FC&lt;/code&gt;: 1x500 = 500 = 0.5K&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Output&lt;/code&gt;: 1 x 10 = 10 = 0.01K (next to nothing)&lt;/p&gt;&#xA;&lt;p&gt;Total memory: 7.5K + 80K + 20K + 40K + 0.5K = 157.2K * 4 bytes = 628.8 KB&lt;/p&gt;&#xA;&lt;p&gt;That's for one image.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Minibatch&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;If you're working with a minibatch size of 64, then you're reading 64 of these into memory at once and performing the operations all together, scaling everything up like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Input&lt;/code&gt;: 64x50x50x3 = 480,000 = 480K = 0.48M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;C1&lt;/code&gt;: 64x50x50x32 = 5,120,000 = 5.12M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;P1&lt;/code&gt;: 64x25x25x32 = 1,280,000 = 1.28M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;C2&lt;/code&gt;: 64x25x25x64 = 2,560,000 = 2.56M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;P2&lt;/code&gt;: 64x12x12x64 = 589,824 = 590K = 0.59M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;FC&lt;/code&gt;: 64x500 = 32,000 = 32K = 0.032M&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Output&lt;/code&gt;: 1x10x64 = 640 = 0.64K = 0.00064M (we don't care, this is tiny)&lt;/p&gt;&#xA;&lt;p&gt;Total memory: 10M x 4 bytes ~ 40MB (I'm saying approximate because the website also says an approximate value)&lt;/p&gt;&#xA;&lt;p&gt;EDIT: I misread the website, sorry.&lt;/p&gt;&#xA;&lt;p&gt;According to the website, a backward pass requires about triple this, because of the need to store:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;the activations and associated gradients for each neuron - these are of equal size;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the gradients of the weights (parameters) which are the same size as the parameters;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the value of the momentum, if you're using it;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;some kind of miscellaneous memory (I don't understand this part)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-05-26T17:34:33.437" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19243" PostTypeId="1" AcceptedAnswerId="19246" CreationDate="2017-05-26T17:43:57.113" Score="2" ViewCount="1431" Body="&lt;p&gt;I have a data frame with following structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.columns&#xA;&#xA;Index(['first_post_date', 'followers_count', 'friends_count', 'last_post_date','min_retweet', 'retweet_count', 'screen_name', 'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],&#xA;      dtype='object')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Inside the &lt;em&gt;tweets&lt;/em&gt; series, each cell is another data frame containing all the tweets of an user.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.tweets[0].columns&#xA;&#xA;Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to perform calculation on the &lt;em&gt;tweets&lt;/em&gt; of each users, for example, finding average number of retweets of each user, the tweet with maximum retweets etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do these?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&#xA;&lt;a href=&quot;https://www.dropbox.com/s/67we61wydrrdhyk/subsampled.pickle?dl=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;link to sample data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31180" LastEditorUserId="31513" LastEditDate="2017-05-28T17:03:53.533" LastActivityDate="2017-05-29T05:00:02.910" Title="Calculating mean of data frame inside a series object" Tags="|pandas|ipython|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19244" PostTypeId="2" ParentId="12649" CreationDate="2017-05-26T18:04:28.227" Score="5" Body="&lt;p&gt;I think you're on the right track. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you will need to store the derivatives of the activations and of the parameters for backpropagation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, your choice of optimization may matter. Are you training using SGD, or Adam, or Adagrad? These will all have different memory requirements. For example, you're going to have to store the step size cache for a momentum-based method, although that should be secondary compared to the other memory considerations you mention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So all in all, you seem to have calculated the memory requirements for a forward pass. &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;noreferrer&quot;&gt;Andrej Karpathy&lt;/a&gt; mentions that the backward pass could take up to 3x the memory of the forward pass, so this might be why you see such a difference (scroll down to 'Case Studies' on the website to see an example for VGGNet).&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-05-26T18:04:28.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19246" PostTypeId="2" ParentId="19243" CreationDate="2017-05-26T20:09:52.250" Score="4" Body="&lt;p&gt;In the absence of a &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/advanced.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MultiIndex&lt;/a&gt; (the Right Way$^\mathrm{TM}$), the &lt;code&gt;apply&lt;/code&gt; method can do what you want; e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.assign(&#xA;    max_retweet=df.tweets.apply(lambda x: x.retweet_count.argmax('retweet_count')),&#xA;    avg_retweet=df.tweets.apply(lambda x: x.retweet_count.mean())&#xA;)&#xA;&#xA;&#xA;       avg_retweet  max_retweet  &#xA;26662     0.045476          187  &#xA;32316     0.821538          427  &#xA;25879     0.633681          583  &#xA;43411     0.112465          421  &#xA;28840     0.472222            8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, don't share data with pickle; it's &lt;a href=&quot;https://www.kevinlondon.com/2015/08/15/dangerous-python-functions-pt2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;dangerous&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="695" LastEditDate="2017-05-29T05:00:02.910" LastActivityDate="2017-05-29T05:00:02.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19247" PostTypeId="1" AcceptedAnswerId="19255" CreationDate="2017-05-26T20:47:24.983" Score="1" ViewCount="324" Body="&lt;p&gt;I am working on developing a data dashboard / app for a Microsoft SQL database. Currently, I am developing the dashboard using Shiny and R. The app is mostly for exploratory analysis to allow people to filter out some subset of data, build some plots, and export data / plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what are some other options for creating this type of dashboard? I am aware of Tableau. &lt;/p&gt;&#xA;" OwnerUserId="27702" LastActivityDate="2017-05-27T10:00:40.780" Title="Data dashboard for SQL server database" Tags="|visualization|sql|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19248" PostTypeId="2" ParentId="19199" CreationDate="2017-05-26T21:03:22.980" Score="0" Body="&lt;p&gt;Your parameter $\alpha$ has fairly low dimension.  Therefore, I recommend that you apply optimization methods directly to try to find the best $\alpha$ (without trying to use convolutional neural networks and regression for this purpose).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define a distance measure on images, $\|I-J\|$, to represent how dissimilar images $I,J$ are.  You might use the squared $L_2$ norm for this, for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the loss for a particular parameter choice $\alpha$ is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L(\alpha) = \sum_{k=1}^N \|F_\alpha(I_k)-J_k\|.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can now formulate your problem as follows: given a training set of images $(I_k,J_k)$, find the parameter $\alpha$ that minimizes the loss $L(\alpha)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A reasonable approach is to use some optimization procedure to solve this problem.  You might use stochastic gradient descent, for instance.  Because there might be multiple local minima, I would suggest that you start multiple instances of gradient descent from different starting points: use a grid search over the starting point.  Since your $\alpha$ is only three-dimensional, it's not difficult to do a grid search over this three-dimensional space and then start gradient descent from each point in the grid.  Stochastic gradient descent will allow you to deal with fairly large values of $N$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does require you to be able to compute gradients for $L(\alpha)$.  Depending on the filter $\alpha$, it might be possible to symbolically calculate the gradients (perhaps with the help of a framework for this, such as Tensorflow); if that's too hard, you can use black-box methods to estimate the gradient by evaluating $L(\cdot)$ at multiple points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $L_2$ distance doesn't capture similarity in your domain, you could consider other distance measures as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect this is likely to be a more promising approach than what you sketched in the question, using convolutional networks and a regression model.  (For one thing, there's no reason to expect the mapping from &quot;features of $I_k$&quot; to &quot;features of $J_k$&quot; to be linear, so there's no reason to expect linear regression to be effective here.)&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-26T21:03:22.980" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="19249" PostTypeId="2" ParentId="19227" CreationDate="2017-05-26T21:25:29.393" Score="4" Body="&lt;p&gt;You were almost there!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample DFs:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [387]: df1&#xA;Out[387]:&#xA;   id&#xA;0   1&#xA;1   2&#xA;2   3&#xA;3   4&#xA;4   5&#xA;&#xA;In [388]: df2&#xA;Out[388]:&#xA;   first_id&#xA;0         7&#xA;1         6&#xA;2         5&#xA;3         1&#xA;4         3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Solution:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [389]: df2['new'] = df2.first_id.isin(df1.id).astype(np.int8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Result:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [390]: df2&#xA;Out[390]:&#xA;   first_id  new&#xA;0         7    0&#xA;1         6    0&#xA;2         5    1&#xA;3         1    1&#xA;4         3    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-05-26T21:25:29.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19250" PostTypeId="1" CreationDate="2017-05-26T21:57:35.990" Score="1" ViewCount="170" Body="&lt;p&gt;I'm obscuring the data I'm working on, but I think this should get the point across. I'm trying to design a test for the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A client sells various popsicles. They think that on days when it is very hot, one particular kind of popsicle sells more than on days when it's not hot. How can I prove if he's correct or not? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thought is that I could do an A/B test, splitting up between not hot &amp;amp; hot days. For each of those groups, I could find the proportion of sales for that popsicle from all popsicles sold that day. If that proportion rises on hot days, then I can do a test for statistical significance and come to a conclusion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I think there have to be other tests/models I can apply based on sales counts alone right? What tests can I run that only involve things like the raw counts of sales on days when it was hot and not? Note that I am planning to use the temperature as a categorical variable but I can also make it continuous if that makes more sense. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2017-05-27T04:30:07.683" Title="A/B Testing Suitability for Hypothesis" Tags="|statistics|prediction|ab-test|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19251" PostTypeId="2" ParentId="12956" CreationDate="2017-05-27T04:12:06.237" Score="31" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Layer normalization&lt;/strong&gt; (&lt;em&gt;Ba 2016&lt;/em&gt;): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Recurrent Batch Normalization (BN)&lt;/strong&gt; (&lt;em&gt;Cooijmans, 2016&lt;/em&gt;; also proposed concurrently by &lt;a href=&quot;https://arxiv.org/abs/1604.03640&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;Qianli Liao &amp;amp; Tomaso Poggio&lt;/em&gt;&lt;/a&gt;, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Batch Normalized Recurrent Neural Networks&lt;/strong&gt; (&lt;em&gt;Laurent, 2015&lt;/em&gt;): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://arxiv.org/abs/1610.06160&quot; rel=&quot;noreferrer&quot;&gt;Streaming Normalization&lt;/a&gt;&lt;/strong&gt; (&lt;em&gt;Liao et al. 2016&lt;/em&gt;) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time): &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Weight Normalization&lt;/strong&gt; (&lt;em&gt;Salimans and Kingma 2016&lt;/em&gt;): whenever a weight is used, it is divided by its $L2$ norm first, such that the resulting weight has $L2$ norm $1$. That is, output $y = x*(w/|w|)$, where $x$ and $w$ denote the input and weight respectively. A scalar scaling factor $g$ is then multiplied to the output $y = y*g$. But in my experience $g$ seems not essential for performance (also downstream learnable layers can learn this anyway). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cosine Normalization&lt;/strong&gt; (&lt;em&gt;Luo et al. 2017&lt;/em&gt;): weight normalization is very similar to cosine normalization, where the same $L2$ normalization is applied to both weight and input: $y = (x/|x|)*(w/|w|)$. Again, manual or automatic differentiation can compute appropriate gradients of $x$ and $w$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that both &lt;em&gt;Weight&lt;/em&gt; and &lt;em&gt;Cosine Normalization&lt;/em&gt; have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref: &lt;a href=&quot;http://maxlab.neuro.georgetown.edu/hmax.html&quot; rel=&quot;noreferrer&quot;&gt;The HMAX Model Reference&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref: &lt;a href=&quot;http://cbcl.mit.edu/jmutch/cns/&quot; rel=&quot;noreferrer&quot;&gt;Cortical Network Simulator Reference&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ref: &lt;a href=&quot;https://arxiv.org/pdf/1702.05870.pdf&quot; rel=&quot;noreferrer&quot;&gt;Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks&lt;/a&gt;,  Luo Chunjie, Zhan jianfeng, Wang lei, Yang Qiang&lt;/p&gt;&#xA;" OwnerUserId="32711" LastEditorUserId="29575" LastEditDate="2018-03-12T01:06:02.507" LastActivityDate="2018-03-12T01:06:02.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19252" PostTypeId="2" ParentId="19250" CreationDate="2017-05-27T04:30:07.683" Score="1" Body="&lt;p&gt;I think you need to start by thinking very carefully about what &lt;em&gt;exactly&lt;/em&gt; your client wants to know, because there are two different things you could be testing:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is each individual customer is more likely to buy a popsicle of the designated type on a hot day than on a cool day?&lt;/li&gt;&#xA;&lt;li&gt;Is the client likely to sell more popsicles of the designated type on hot days than on cool days?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the first one, you would analyze proportions; for the second, you would analyze absolute sales counts for the designated type of popsicle, and in fact you could probably ignore sales of all other types of popsicles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that latter question is the one you want to analyze, I do think running a regression analysis on sales of the particular type of popsicle vs temperature (both as continuous variables) is a reasonable place to start. In fact you could start by just plotting them and looking at the graph, to see if any trend (linear, logarithmic, logistic, etc.) jumps out at you. I wouldn't go for A/B testing right off the bat because you're not dealing with a categorical variable, and there's not a single natural way to convert it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course it could make sense to do that conversion, but in doing so, I'd suggest looking at overall sales figures and other indicators of whether people perceive a day to be hot or cool to try to pin down the boundaries. Something like logistic regression (possibly multinomial) might come in handy, though I could easily see it giving worthless results if the data aren't clean enough. Perhaps you could exploit a clustering algorithm to identify groups of temperatures that have similar sales figures and use that to determine your categories. You could also try to correlate external data sets like community pool attendance, ice cream sales, electricity usage (for air conditioners), etc., if those are available, to further inform your analysis of what temperature class each day would fall in. (Maybe that's overkill, but it's an idea at least.)&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Incidentally, A/B testing typically designates a process where you control the assignment of the classes A and B. Like in web design, you choose (or program a computer to choose, perhaps at random) whether each visitor gets version A or version B. You don't get to choose whether each day is hot or not, so it's not &lt;em&gt;quite&lt;/em&gt; the same thing. That being said, you can probably use most of the same methods of analysis you would on a proper A/B test.&lt;/p&gt;&#xA;" OwnerUserId="695" LastActivityDate="2017-05-27T04:30:07.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19253" PostTypeId="2" ParentId="19197" CreationDate="2017-05-27T07:05:38.473" Score="3" Body="&lt;p&gt;From the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/vgg16.py#L59&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras VGG16 Documentation&lt;/a&gt; it says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;    input_shape: optional shape tuple, only to be specified&#xA;        if `include_top` is False (otherwise the input shape&#xA;        has to be `(224, 224, 3)` (with `channels_last` data format)&#xA;        or `(3, 224, 224)` (with `channels_first` data format).&#xA;        It should have exactly 3 inputs channels,&#xA;        and width and height should be no smaller than 48.&#xA;        E.g. `(200, 200, 3)` would be one valid value.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So I would worry about what's happening in your code with respect to data shapes and input shapes. I would further worry about the applicability of vgg16 to the 32x32 cifar images therefore.&lt;/p&gt;&#xA;" OwnerUserId="32723" LastActivityDate="2017-05-27T07:05:38.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19254" PostTypeId="1" CreationDate="2017-05-27T08:31:42.030" Score="0" ViewCount="58" Body="&lt;p&gt;I want to predict the grade of a (german) health insurance by some features it offers and I am not sure about the best way to define these features. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How should I treat a variable which can be &quot;80%&quot;, &quot;90%&quot; or &quot;100%&quot;. If I use it categorical I loose the fact that 100% is better than 90%. &#xA;If numerical is better, should I use 80,90,100 or 0.8, 0.9, 1 ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Categorical Variables like &quot;multi bed&quot;, &quot;double bed&quot;, &quot;singe bed&quot; also have a &quot;is-better-than&quot; relation but how should I transform them into numerical values? Ascending integers, or from -1 to 1 or from 0 to 1. Or does all this make no sense at all. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I treat grade as a numerical target or as categorical (1.0, 1.1 ...) as only these values are allowed outcomes. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What method of prediction do U suggest? I have around 80 features, most of the categorical (mostly yes/no). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="27045" LastActivityDate="2017-06-26T11:53:16.777" Title="How do I define my feature variables" Tags="|regression|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19255" PostTypeId="2" ParentId="19247" CreationDate="2017-05-27T10:00:40.780" Score="1" Body="&lt;p&gt;You could consider PowerBI. It offers a couple of useful licensing scenarios (bundled with Office 365 etc.) and also has connectors to source data that already works well with SQL Server. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You did not say if SQL Server is running on premises or in the cloud, but PowerBI should work in both cases.&lt;/p&gt;&#xA;" OwnerUserId="31245" LastActivityDate="2017-05-27T10:00:40.780" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19256" PostTypeId="1" CreationDate="2017-05-27T11:09:55.030" Score="-1" ViewCount="245" Body="&lt;p&gt;I am currently working with the famous titanic dataset from Kaggle. Now I want to explore the influence on different features on the chance of survival. I use the random fores classifier for an accuracy score output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know whether people with sibling have a bigger chance of survival than people without. I could slice my dataset into one with siblings and one without.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However these might probably not be comparable either because of the size or e.g. that people with sibblings are more often female. The comparison is distorted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I account for that and answer my question correctly.&lt;/p&gt;&#xA;" OwnerUserId="32726" LastEditorUserId="8432" LastEditDate="2017-05-29T05:00:17.683" LastActivityDate="2017-05-29T05:00:17.683" Title="Explore Influence of features on titanic dataset" Tags="|machine-learning|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19257" PostTypeId="2" ParentId="19256" CreationDate="2017-05-27T11:39:47.807" Score="0" Body="&lt;p&gt;Not sure if this is what you're using already, but SKLearn has a feature importance function that uses random forests to do something like what you're saying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your question about the survival of those with siblings, you need to think of it is an experiment with controls. Compare the survival rate of females with siblings vs. the rate for all those with siblings.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-27T11:39:47.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19258" PostTypeId="2" ParentId="19254" CreationDate="2017-05-27T11:40:40.817" Score="2" Body="&lt;p&gt;I think that before anything else it's important to explore the data to identify features that aren't bringing anything worthwhile (v. low variance, duplicates, collinearity) and to see if you've got any issues with missing values/outliers in your numeric data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, I think you'd want to keep your numeric variables as such (initially at least), but how many do you have? You may wish to normalise them so that the scale of one variable doesn't dominate (eg. is income included?). Or if you have time, try both? Look at the distribution and see what bins might be appropriate to discriminate by:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Normalization_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Normalization_(statistics)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ordinal data is tricky; in your case you can't have 1.5 beds, but that's what you might end up with if you treat them as numeric. I would initially run with one hot encoding, and when you get some results you can look at the impact of your features to decide if you want to keep them that way:&#xA;&lt;a href=&quot;https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing the full range of your grade values or how granular/continuous your prediction needs to be, you could easily do either - I'd say think about what the ideal outcome would be and build towards that. If you've got time try both, try different bins for a multi-class classifier; but if you have a massive range of target outputs then regression is probably preferable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally - start with something simple and interpretable, and build from there. If you run with classification, can a decision tree deal with your dataset, and how does it do/where does it fall down? You might get lucky, but if not you can try an SVM or random forest. Given the number of variables you might want to consider some dimensionality reduction first; you might get some good results from clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2017-05-27T11:40:40.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19259" PostTypeId="1" CreationDate="2017-05-27T11:49:55.000" Score="3" ViewCount="509" Body="&lt;p&gt;What are the cases when we should not use PCA for dimensionality reduction and what to use in such cases?&lt;/p&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="20517" LastEditDate="2017-05-27T12:51:04.470" LastActivityDate="2018-08-10T11:56:52.903" Title="Dimensionality reduction with PCA limitations" Tags="|dimensionality-reduction|pca|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19260" PostTypeId="2" ParentId="19256" CreationDate="2017-05-27T11:59:00.673" Score="0" Body="&lt;p&gt;While random forest will probably give the most accurate answers, it's difficult to interpret.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just making a start on the titanic data set - see you on Kaggle :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm going to start with some exploratory analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This plot indicates that there is some connection between number of siblings &amp;amp; chance of survival, taking sex into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JPhif.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JPhif.png&quot; alt=&quot;plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31216" LastActivityDate="2017-05-27T11:59:00.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19263" PostTypeId="2" ParentId="19259" CreationDate="2017-05-27T13:51:38.853" Score="2" Body="&lt;p&gt;You should not use PCA if you only have categorical variables, and thus the distance function in PCA is invalid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Correspondence_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Correspondence analysis&lt;/a&gt; is a common alternative.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-05-27T13:51:38.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19266" PostTypeId="2" ParentId="19172" CreationDate="2017-05-27T16:23:14.467" Score="1" Body="&lt;p&gt;Comparing distance values of different distances is nonsense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this distance function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d(x,y)=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, this gives smaller values, but it also is useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this trivial variation of Manhattan distance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d(x,y) = \sum_i |x_i-y_i|/100$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, this distance is equivalent to Manhattan, but will yield much smaller values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose the distance because it is &quot;the right thing to use&quot;, not because of some number. Cosine is good if you desire length normalization and have sparse vectors.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-05-27T16:23:14.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19267" PostTypeId="1" CreationDate="2017-05-27T16:35:14.157" Score="1" ViewCount="995" Body="&lt;p&gt;In most cases I frequently heard that to make a deep learning experiment, it is highly recommended to use GPU. It makes the computation blazingly faster than CPU, and sounds like a magical tool (although I can't use it since I use MacBook...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, is there any single disadvantage to use GPU, possibly except the higher monetary cost? For example the higher likelihood of getting crashed, etc...?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2020-09-23T12:01:35.163" Title="Is there any single disadvantage to use GPU in deep learning?" Tags="|deep-learning|gpu|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19268" PostTypeId="1" AcceptedAnswerId="19347" CreationDate="2017-05-27T19:16:24.263" Score="1" ViewCount="900" Body="&lt;p&gt;I have been having my hands full with training a model to classify web pages. This is the first time ever that I am doing this, so I know &lt;em&gt;very&lt;/em&gt; little about ML. I'm here to learn. &lt;code&gt;:-)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am using a Pipeline with FeatureUnion to run a a GridSearchCV to tune my parameters. However, I am slightly confused about how this process ought to go. For the whole process of creating a predictive model I read &lt;a href=&quot;https://datascience.stackexchange.com/questions/10085/machine-learning-steps/10087#10087&quot;&gt;this answer&lt;/a&gt; on &quot;Machine Learning Steps&quot; question, which was helpful, but I still have some questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all: &lt;strong&gt;are parameters independent of your model&lt;/strong&gt;? The answer linked above suggests first testing different models, and then do parameter tuning. However, I started with tuning my parameters on a Naive Bayes multinomial model, with the idea to test the models (e.g. svm, kNN, NBmn) afterwards with the 'optimal' parameters. But the answer above seems to suggest otherwise. Does this mean that the parameter tuning depends on the best model you choose, which makes it useless to first tune the parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, I am also not sure how to &lt;strong&gt;&lt;em&gt;correctly&lt;/em&gt; tune my parameters per feature&lt;/strong&gt;. I have 13 features, and put them all in the Pipeline. For things like &lt;code&gt;TfidfVectorizer&lt;/code&gt; for eight features, I'd like to tune max_features, ngram_range, and max_df. However, because this takes quite a long time to run I thought I'd start with two features, make sure I get the optimal value, set them in the Pipeline (not in the parameter tuning), and add a feature. However, I feel that this is not the way at all. Is it required to the complete tuning as a single step, or is it possible to do it in steps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, I noticed that you can also pass the feature weights of FeatureUnion as a parameter, and tune it. But as far as I have found you can only add specific values to test (example below) and not a range (e.g. 0-1 with steps of 0.1), which makes it hard to set the parameter values as you've got to guess which could be a good value. &lt;strong&gt;Is there a better way?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;parameters = {&#xA;    'union__transformer_weights': [&#xA;        {'f1':1, 'f2':0.4, 'f3': 0.2,'f4': 1,'f5': 0.4,'f6': 0.1},&#xA;        {'f1':0.4, 'f2':0.9, 'f3': 0.2,'f4': 0.4,'f5': 0.4,'f6': 0.2},&#xA;        {'f1':0.8, 'f2':0.7, 'f3': 0.2,'f4': 0.7,'f5': 0.7,'f6': 0.4}&#xA;    ]&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32669" LastEditorUserId="31513" LastEditDate="2017-05-28T18:12:34.157" LastActivityDate="2017-05-31T08:47:29.397" Title="How does one fine-tune parameters and weights at the same time?" Tags="|predictive-modeling|scikit-learn|supervised-learning|parameter-estimation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19269" PostTypeId="2" ParentId="19196" CreationDate="2017-05-27T19:43:12.073" Score="17" Body="&lt;p&gt;Great question! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;tl;dr: The cell state and the hidden state are two different things, but the hidden state is dependent on the cell state and they do indeed have the same size.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Longer explanation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between the two can be seen from the diagram below (part of the same blog):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ay51D.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ay51D.png&quot; alt=&quot;The cell state is the bold line travelling west to east across the top.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cell state is the bold line travelling west to east across the top. The entire green block is called the 'cell'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hidden state from the previous time step is treated as part of the input at the current time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it's a little harder to see the dependence between the two without doing a full walkthrough. I'll do that here, to provide another perspective, but heavily influenced by the blog. My notation will be the same, and I'll use images from the blog in my explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to think of the order of operations a little differently from the way they were presented in the blog. Personally, like starting from the input gate. I'll present that point of view below, but please keep in mind that the blog may very well be the best way to set up an LSTM computationally and this explanation is purely conceptual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what's happening:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The input gate&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qQlpQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qQlpQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input at time $t$ is $x_t$ and $h_{t-1}$. These get concatenated and fed into a nonlinear function (in this case a sigmoid). This sigmoid function is called the 'input gate', because it acts as a stopgap for the input. It decides stochastically which values we're going to update at this timestep, based on the current input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, (following your example), if we have an input vector $x_t = [1, 2, 3]$ and a previous hidden state $h_t = [4, 5, 6]$, then the input gate does the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Concatenate $x_t$ and $h_{t-1}$ to give us $[1, 2, 3, 4, 5, 6]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) Compute $W_i$ times the concatenated vector and add the bias (in math: $W_i \cdot [x_t, h_{t-1}] + b_i$, where $W_i$ is the weight matrix from the input vector to the nonlinearity; $b_i$ is the input bias).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume we're going from a six-dimensional input (the length of the concatenated input vector) to a three-dimensional decision on what states to update. That means we need a 3x6 weight matrix and a 3x1 bias vector. Let's give those some values:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W_i = \begin{bmatrix} 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 \\ 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 \\ 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3\end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$b_i = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The computation would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{bmatrix} 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 \\ 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 &amp;amp; 2 \\ 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3 &amp;amp; 3\end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\5 \\6 \end{bmatrix} + \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c) Feed that previous computation into a nonlinearity: $i_t = \sigma (W_i \cdot [x_t, h_{t-1}] + b_i)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(x) = \frac{1}{1 + exp(-x)}$ (we apply this elementwise to the values in the vector $x$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(\begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}) = [\frac{1}{1 + exp(-22)}, \frac{1}{1 + exp(-42)}, \frac{1}{1 + exp(-62)}] = [1, 1, 1]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In English, that means we're going to update all of our states.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input gate has a second part:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d) $\tilde{C_t} = tanh(W_C[x_t, h_{t-1}] + b_C)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point of this part is to compute how we would update the state, if we were to do so. It's the contribution from the new input at this time step to the cell state. The computation follows the same procedure illustrated above, but with a tanh unit instead of a sigmoid unit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output $\tilde{C_t}$ is multiplied by that binary vector $i_t$, but we'll cover that when we get to the cell update.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Together, $i_t$ tells us which states we want to update, and $\tilde{C_t}$ tells us how we want to update them. It tells us what new information we want to add to our representation so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then comes the forget gate, which was the crux of your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The forget gate&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H48IQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H48IQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of the forget gate is to remove previously-learned information that is no longer relevant. The example given in the blog is language-based, but we can also think of a sliding window. If you're modelling a time series that is naturally represented by integers, like counts of infectious individuals in an area during a disease outbreak, then perhaps once the disease has died out in an area, you no longer want to bother considering that area when thinking about how the disease will travel next. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just like the input layer, the forget layer takes the hidden state from the previous time step and the new input from the current time step and concatenates them. The point is to decide stochastically what to forget and what to remember. In the previous computation, I showed a sigmoid layer output of all 1's, but in reality it was closer to 0.999 and I rounded up.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The computation looks a lot like what we did in the input layer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f_t = \sigma(W_f [x_t, h_{t-1}] + b_f)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give us a vector of size 3 with values between 0 and 1. Let's pretend it gave us:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$[0.5, 0.8, 0.9]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we decide stochastically based on these values which of those three parts of information to forget. One way of doing this is to generate a number from a uniform(0, 1) distribution and if that number is less than the probability of the unit 'turning on' (0.5, 0.8, and 0.9 for units 1, 2, and 3 respectively), then we turn that unit on. In this case, that would mean we forget that information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quick note: the input layer and the forget layer are independent. If I were a betting person, I'd bet that's a good place for parallelization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updating the cell state&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kZm6J.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kZm6J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have all we need to update the cell state. We take a combination of the information from the input and the forget gates:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C_t = f_t \circ C_{t-1} + i_t \circ \tilde{C_t}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is going to be a little odd. Instead of multiplying like we've done before, here $\circ$ indicates the Hadamard product, which is an entry-wise product. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aside: Hadamard product&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we had two vectors $x_1 = [1, 2, 3]$ and $x_2 = [3, 2, 1]$ and we wanted to take the Hadamard product, we'd do this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_1 \circ x_2 = [(1 \cdot 3), (2 \cdot 2), (3 \cdot 1)] = [3, 4, 3]$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;End Aside.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this way, we combine what we want to add to the cell state (input) with what we want to take away from the cell state (forget). The result is the new cell state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The output gate&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jB3EI.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jB3EI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give us the new hidden state. Essentially the point of the output gate is to decide what information we want the next part of the model to take into account when updating the subsequent cell state. The example in the blog is again, language: if the noun is plural, the verb conjugation in the next step will change. In a disease model, if the susceptibility of individuals in a particular area is different than in another area, then the probability of acquiring an infection may change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output layer takes the same input again, but then considers the updated cell state:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$o_t = \sigma(W_o [x_t, h_{t-1}] + b_o)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, this gives us a vector of probabilities. Then we compute:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_t = o_t \circ tanh(C_t)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the current cell state and the output gate must agree on what to output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, if the result of $tanh(C_t)$ is $[0, 1, 1]$ after the stochastic decision has been made as to whether each unit is on or off, and the result of $o_t$ is $[0, 0, 1]$, then when we take the Hadamard product, we're going to get $[0, 0, 1]$, and only the units that were turned on by both the output gate and in the cell state will be part of the final output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[EDIT: There's a comment on the blog that says the $h_t$ is transformed again to an actual output by $y_t = \sigma(W \cdot h_t)$, meaning that the actual output to the screen (assuming you have some) is the result of another nonlinear transformation.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The diagram shows that $h_t$ goes to two places: the next cell, and to the 'output' - to the screen. I think that second part is optional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of variants on LSTMs, but that covers the essentials! &lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-05-27T23:29:07.833" LastActivityDate="2017-05-27T23:29:07.833" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="19270" PostTypeId="2" ParentId="18013" CreationDate="2017-05-27T20:30:59.953" Score="4" Body="&lt;p&gt;I can speak from a more theoretical point of view, but honestly I haven't had much success with VAEs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) How deep should my encoder and decoder network be? Are there any good guidelines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That depends entirely on your dataset. If you have highly nonlinear data, then a deep network should do well. The successive nonlinearities allow you to capture higher-order correlations in the input, which might be good for your situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any idea of the number of truly independent dimensions in your dataset? Have you done something like PCA (a linear autoencoder)? That might tell you where to aim your dimensionality of your bottleneck layer, and that might help you determine your layer sizes to get there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Should one use fully connected dense networks, or stacked conv1Dnets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say try both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) What activation functions are good choices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hugo Larochelle once said that you should always start with ReLUs. See if you get a good enough result with them, as they aren't as prone to the problem of exploding/vanishing gradients, which is something you're going to face with time series data. Remember to initalize them with small positive values, though, to avoid dead neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[EDIT: actually, vanishing and exploding gradients can still be an issue with ReLUs, it's just mitigated a bit. Also, initialize the &lt;em&gt;bias&lt;/em&gt; with small positive values. &lt;a href=&quot;https://www.youtube.com/watch?v=gYpoJMlgyXA&amp;amp;feature=youtu.be&amp;amp;t=14m14s&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrej Karpathy said so in a lecture&lt;/a&gt;]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Can we say anything about the 'best' dimensionality of the latent dimension?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some contexts yes, we can. The 'best' dimensionality will be the one that results in the highest lossless compression. That is, the one that compresses your input data the most without losing information when you reconstruct. Good luck finding that optimum, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could incorporate various types of regularization that will modify your loss function. You could use dropout, for example, or L1 or L2 regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: you could also consider an attention-based model, for example see &lt;a href=&quot;http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this discussion&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'Always' is a strong term. I would say not always, but batch normalization ought to be a good choice. Have another look at your implementation before you give up on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7) In the decoder layer, is it better to up-sample before returning to the original input dimension?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's the default presentation - an autoencoder is usually shown as a symmetric construct from the input to the hidden to the reconstruction. I would say experiment with not doing so, but I don't know if one is better than the other.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-05-29T15:55:22.513" LastActivityDate="2017-05-29T15:55:22.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19271" PostTypeId="1" CreationDate="2017-05-27T23:48:28.543" Score="4" ViewCount="3768" Body="&lt;p&gt;In the linear regression, when we have a categorical explanatory variable with $n$ levels, we usually remove one level and call it a baseline level and fit the model on the remaining levels. And the final intercept is the intercept plus the coefficient of baseline level. Now my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does it matter which level I choose to remove? I am working on a dataset to predict house price. When I use linear model with intercept on the testset for prediction, I get negative values for some houses which is not correct, as house price cannot be negative. But when I fit a regression without an intercept all the prediction values are positive as expected. I guess it has to do something with the baseline level I chose. &lt;/li&gt;&#xA;&lt;li&gt;Do I have to remove one level from all my categorical variables in the dataset before fit a random forest, KNN, Ridge, Lasso,... too?!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30381" LastEditorUserId="31513" LastEditDate="2017-05-28T02:42:36.347" LastActivityDate="2017-06-01T05:54:51.320" Title="Handling categorical variables in linear regression and random forest" Tags="|random-forest|linear-regression|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19272" PostTypeId="1" AcceptedAnswerId="19332" CreationDate="2017-05-28T09:06:20.633" Score="25" ViewCount="47129" Body="&lt;p&gt;I'm having some difficulty in deriving back propagation with ReLU, and I did some work, but I'm not sure if I'm on the right track.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cost Function: $\frac{1}{2}(y-\hat y)^2$ where $y$ is the real value, and $\hat y$ is a predicted value. Also assume that $x$ &gt; 0 always.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;1 Layer ReLU, where the weight at the 1st layer is $w_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oNeWz.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oNeWz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{dw_1}=\frac{dC}{dR}\frac{dR}{dw_1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;2 Layer ReLU, where the weights at the 1st layer is $w_2$, and the 2nd layer is $w_1$ And I wanted to updated the 1st layer $w_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ic6L6.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ic6L6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{dw_2}=\frac{dC}{dR}\frac{dR}{dw_2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $ReLU(w_1*ReLU(w_2x))=w_1w_2x$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;3 Layer ReLU, where the weights at the 1st layer is $w_3$, 2nd layer $w_2$ and 3rd layer $w_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XIeyw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XIeyw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{dw_3}=\frac{dC}{dR}\frac{dR}{dw_3}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since $ReLU(w_1*ReLU(w_2(*ReLU(w_3))=w_1w_2w_3x$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the chain rule only lasts with 2 derivatives, compared to a sigmoid, which could be as long as $n$ number of layers.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Say I wanted to updated all 3 layer weights, where $w_1$ is the 3rd layer, $w_2$ is the 2nd layer, $w_1$ is the 3rd layer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this derivation is correct, how does this prevent vanishing? Compared to sigmoid, where we have a lot of multiply by 0.25 in the equation, whereas ReLU does not have any constant value multiplication. If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="24119" LastEditDate="2017-05-30T05:50:44.793" LastActivityDate="2019-01-03T11:28:52.830" Title="Deep Neural Network - Backpropogation with ReLU" Tags="|neural-network|backpropagation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19273" PostTypeId="1" CreationDate="2017-05-28T09:37:54.573" Score="1" ViewCount="600" Body="&lt;p&gt;I'm fine tuning with caffenet for a 3 classes classification problem. I have 100 instances of class A, 90 instances of class B and 30 instances of class C. I thought that my net would be biased toward classes A and B but I'm actually getting quite good results. I know that caffenet doesn't take care of imbalance data for me. Maybe it has to do with the fact that my entire train set fits into one batch? Or maybe it has to do with the fact that I'm not really training from scratch but mostly using caffenet's already learned weights? Thanks&lt;/p&gt;&#xA;" OwnerUserId="30182" LastEditorUserId="50727" LastEditDate="2018-05-17T10:49:26.967" LastActivityDate="2018-05-17T10:49:26.967" Title="Fine tuning CNN with imbalance data gives good results - not sure why" Tags="|deep-learning|cnn|class-imbalance|caffe|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19274" PostTypeId="2" ParentId="19273" CreationDate="2017-05-28T10:56:08.163" Score="2" Body="&lt;p&gt;Are you getting good results on your training or your test set? If it's only the first then you are overfitting, but if it's the second it will not have to do with the fact that your whole training set fits in your batch, the gradients still have a strong pull towards the dominating class(es). But having the pretrained weights will help a lot if the classes are relatively easy to seperate. It will also help if your dataset is similar to the one it was pretrained on, because the intermediate layers that do feature extractions have learned valuable filters for your problem instance.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-05-28T11:45:30.523" LastActivityDate="2017-05-28T11:45:30.523" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19275" PostTypeId="1" CreationDate="2017-05-28T12:21:00.257" Score="2" ViewCount="347" Body="&lt;p&gt;I am a beginner in R. I am a bit confused about the output of the neuralnet() function in R, from package &lt;a href=&quot;https://cran.r-project.org/web/packages/neuralnet/neuralnet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;neuralnet&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here it shows steps equal to thirteen. I would like to inquire about the understanding of these steps. What is the total of all the steps that occur? The data I used as many as seven before the output as shown. Then my question is, how to display the details of each step that occurs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oyhjd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oyhjd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32747" LastEditorUserId="31513" LastEditDate="2017-05-29T05:00:10.200" LastActivityDate="2017-05-29T05:00:10.200" Title="Need explanation about output of R function neuralnet()" Tags="|neural-network|r|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19276" PostTypeId="1" CreationDate="2017-05-28T12:56:36.267" Score="9" ViewCount="23261" Body="&lt;p&gt;I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, economics, etc., but are very closely related and in some cases even partially overlapping. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Preparation -&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Broke the documents in list of words.&lt;/li&gt;&#xA;&lt;li&gt;Removed stop words, punctuations.&lt;/li&gt;&#xA;&lt;li&gt;Performed stemming.&lt;/li&gt;&#xA;&lt;li&gt;Replaced numerical values with '#num#' to reduce vocabulary size.&lt;/li&gt;&#xA;&lt;li&gt;Transformed the documents into TF-IDF vectors.&lt;/li&gt;&#xA;&lt;li&gt;Sorted all words based on their TF-IDF value and selected the top 20K words, these will be used a feature list for the classification algorithm.&lt;/li&gt;&#xA;&lt;li&gt;Used SVM.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have &lt;strong&gt;4,500&lt;/strong&gt; categorized documents with &lt;strong&gt;17 categories&lt;/strong&gt;, and I used &lt;strong&gt;80:20&lt;/strong&gt; ration for training and test dataset. I used Sklearn python library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best classification &lt;strong&gt;accuracy&lt;/strong&gt; I have managed to get is &lt;strong&gt;61%&lt;/strong&gt; and I need it to be at least &lt;strong&gt;85%&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.&lt;/p&gt;&#xA;" OwnerUserId="30363" LastActivityDate="2020-08-29T17:24:04.267" Title="Improving accuracy of Text Classification" Tags="|machine-learning|nlp|feature-selection|svm|sentiment-analysis|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19277" PostTypeId="2" ParentId="19276" CreationDate="2017-05-28T13:52:07.160" Score="7" Body="&lt;p&gt;First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Improve your model my adding bigrams and tri-grams as features. &lt;/li&gt;&#xA;&lt;li&gt;Try doing some topic modelling like latent Dirichlet allocation or Probabilistic latent Semantic Analysis for the corpus using a specified number of topics - say 20.  You would get a vector of 20 probabilities corresponding to the 20 topics for each document. You could use that vector as input for your classification or use it as additional features on top of what you already have from your base model enhanced with bigrams and trigrams. &lt;/li&gt;&#xA;&lt;li&gt;Another thing I would say is try using a tree based classifier ensemble to capture non-linearity and interactions between the features. Either of random forest or gradient boosting would be fine. In gradient boosting you can use  xgboost as its a pretty good package that gives good classification. &lt;/li&gt;&#xA;&lt;li&gt;If you are familiar with Deep Learning you can give a try with a Recurrent Neural network architecture(mostly the LSTM versions).  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this helps and let me know if this improves your classification accuracy.&lt;/p&gt;&#xA;" OwnerUserId="28337" LastEditorUserId="31513" LastEditDate="2017-05-29T04:10:22.263" LastActivityDate="2017-05-29T04:10:22.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19278" PostTypeId="2" ParentId="19019" CreationDate="2017-05-28T14:34:14.467" Score="1" Body="&lt;p&gt;Check this &lt;a href=&quot;https://github.com/fchollet/keras/blob/befbdaa076eedb2787ce340b1a5b4accf93d123d/examples/mnist_irnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; (a reproduction of the IRNN experiment with pixel-by-pixel sequential MNIST in &quot;A Simple Way to Initialize Recurrent Networks of Rectified Linear Units&quot; by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relevant code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(SimpleRNN(hidden_units,&#xA;                    kernel_initializer=initializers.RandomNormal(stddev=0.001),&#xA;                    recurrent_initializer=initializers.Identity(gain=1.0),&#xA;                    activation='relu',&#xA;                    input_shape=x_train.shape[1:]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32749" LastEditorUserId="31513" LastEditDate="2017-05-28T18:21:32.190" LastActivityDate="2017-05-28T18:21:32.190" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19279" PostTypeId="2" ParentId="19223" CreationDate="2017-05-28T16:51:49.383" Score="1" Body="&lt;p&gt;Basically it all depends on , which type of problem you are trying to solve.&#xA;&quot;Even the most experienced data scientists can’t tell which algorithm will perform best before trying them.&quot;&#xA;So Here some parameter on which you can decide which algorithm you pick up for your machine learning problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Accuracy&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Training time&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Linearity&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. Number of parameters&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5. Number of features&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Microsoft released a PDF cheatsheet of what machine learning algorithms to use, when.&#xA;Download the cheatsheet (PDF) from its companion blog post titled &lt;a href=&quot;https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-algorithm-cheat-sheet&quot; rel=&quot;nofollow noreferrer&quot;&gt; “Machine learning algorithm cheat sheet for Microsoft Azure Machine Learning Studio“&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-05-28T16:51:49.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19280" PostTypeId="1" AcceptedAnswerId="19286" CreationDate="2017-05-28T19:25:35.930" Score="4" ViewCount="6262" Body="&lt;p&gt;I've been working on a classification problem and have some good results, but now I struggle with trying to put together a good plot to illustrate the probabilities for each prediction.  Here is my current data:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;&#xA;df_voting_output.head(n=5)&#xA;    prob    actual  pred    correct&#xA;0   0.460200    0   0   1&#xA;1   0.548478    1   1   1&#xA;2   0.270609    0   0   1&#xA;3   0.686557    0   1   0&#xA;4   0.527935    0   1   0&#xA;5   0.098687    0   0   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I've been able to create a bar chart with probabilites using this code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;plt.bar(np.arange(len(voting_predictions[:,1])), voting_predictions[:,1])&#xA;plt.xlabel(&amp;quot;record number&amp;quot;)&#xA;plt.ylabel(&amp;quot;probability&amp;quot;)&#xA;plt.title(&amp;quot;Classification Probabilities&amp;quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SbJA2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SbJA2.png&quot; alt=&quot;current chart&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;However, I'm thinking there has to be a better way to include more information and a key.  I'd like the &amp;quot;correct&amp;quot; labels to be clear to see, so from a visual standpoint you can see how the probabilities relate to a correct classification.&lt;/p&gt;&#xA;" OwnerUserId="32754" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-05-29T06:20:31.260" Title="Trying to make compelling plot for classification results with python" Tags="|python|classification|visualization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19281" PostTypeId="1" CreationDate="2017-05-28T20:16:40.103" Score="1" ViewCount="234" Body="&lt;p&gt;I'm reading a paper by &lt;a href=&quot;http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bergstra and Bengio (2012)&lt;/a&gt; on random search for hyperparameter optimization. I'm confused by their graph and explanation in Section 2.2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Figure 2 illustrates the results of a random experiment: an experiment of 256 trials training neural networks to classify the rectangles data set. Since the trials of a random experiment are i.i.d., a random search experiment involving $S$ i.i.d. trials can also be interpreted as N independent experiments of $s$ trials, as long as $sN \le S$.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That part I think I understand. Here's my take on it: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I draw from a uniform(0, 1) random variable 100 times, then I have 100 i,.i.d. samples. That's the same as if I had drawn 10 times each from 10 separate uniform(0, 1) random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In their figure (below), they say they had 256 trials. I &lt;em&gt;think&lt;/em&gt; they chose 256 possible values of a hyperparameter and trained a neural network for each value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JwlE5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JwlE5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the caption for the figure, they say they had &quot;256 trials of a random experiment...optimizing a neural network....At horizontal axis position 8, we consider 256 trials to be 32 experiments of 8 trials each&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This implies that at horizontal axis position 1, they would have 1 experiment of 256 trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: did they train those 256 models and then break up the results into smaller chunks? Like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment 1: $[\lambda^1, \ldots, \lambda^{256}]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment 2: $[\lambda^1, \ldots, \lambda^{128}], [\lambda^{129}, \ldots, \lambda^{256}]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with value $\lambda^{129}$ from Experiment 1 equal to value $\lambda^{129}$ in Experiment 2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or did they do something else?&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-05-30T02:37:46.293" LastActivityDate="2017-05-30T02:37:46.293" Title="Experimental design - hyperparameter optimization" Tags="|neural-network|hyperparameter|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19282" PostTypeId="2" ParentId="19280" CreationDate="2017-05-28T21:19:24.710" Score="6" Body="&lt;p&gt;It's really rare that you'd show a plot of the probabilities for each example in your set. Are you sure you want to do this? A better presentation might be a confusion matrix. Here's how it works:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The columns are the true class labels&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The rows are the predicted classes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Along the right hand side of the plot you can show the probability of correctly assigning to a class (or the classification error, if you prefer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say I have three classes in my dataset. &#xA;I have 10 examples of each class, so 30 examples total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My classification results from my model are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;8 of the 10 examples in class 1 were correctly labelled; 1 was misclassified to class 2 and 1 was misclassified to class 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7 of the 10 examples in class 2 were correctly labelled; 3 were misclassified to class 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;9 of the 10 examples in class 3 were correctly labelled; 1 was misclassified to class 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then my confusion matrix looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{bmatrix} 0.8 &amp;amp; 0.1 &amp;amp; 0.1 \\ 0.0 &amp;amp; 0.7 &amp;amp; 0.3 \\ 0.0 &amp;amp; 0.1 &amp;amp; 0.9 \end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And along the right hand side of my confusion matrix I can include the classification error: 0.2 for class 1; 0.3 for class 2; 0.1 for class 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice the rows must add to 1.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-05-28T21:19:24.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19283" PostTypeId="1" AcceptedAnswerId="19321" CreationDate="2017-05-28T21:22:52.203" Score="2" ViewCount="3097" Body="&lt;p&gt;I just started training a MLP model on a data set which has the following statistics. Notice that both train and validation sets are unbalanced (88.4% negatives).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note&lt;/em&gt;: The unbalance isn't caused by sampling and is because that's the natural ratio of the classes. E.g. when I look back in a 3 month period, I see I have eaten 88 apples and 12 oranges; and looking forward, I might still favor apples more than oranges by the same ratio. I may be wrong here but I think I shouldn't give class weights nor resample to balance the two classes, because that's their natural ratio.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train data:&#xA;    Shape (features): (891473, 122)&#xA;    Shape (labels): (891473,)&#xA;    # of classes: [(0, 788118), (1, 103355)]&#xA;    % of classes: [(0, 0.88406266931247501), (1, 0.11593733068752503)]&#xA;Validation data:&#xA;    Shape (features): (251141, 122)&#xA;    Shape (labels): (251141,)&#xA;    # of classes: [(0, 222009), (1, 29132)]&#xA;    % of classes: [(0, 0.8840014175303913), (1, 0.1159985824696087)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using three 8-sized dense layers each following a ReLU layer, finally a sigmoid since it's binary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the training process converged after only 2 epochs and basically only learned the bias term and and ended up predicting everything to be negative. The confusion matrix stays unchanged like this since the 2nd epoch till the end (20 epochs):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train confusion matrix:&#xA;           P = 0   P = 1 | Total&#xA;    L = 0  0.884   0.000 | 0.884&#xA;    L = 1  0.116   0.000 | 0.116&#xA;    ----------------------------&#xA;    Total  1.000   0.000 | 0.884&#xA;Validation confusion matrix:&#xA;           P = 0   P = 1 | Total&#xA;    L = 0  0.884   0.000 | 0.884&#xA;    L = 1  0.116   0.000 | 0.116&#xA;    ----------------------------&#xA;    Total  1.000   0.000 | 0.884&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried to increase the size of the dense layer (e.g. from 32, 128, 512, 1024). They all yielded the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd expect that with 3 x 1024 neurons the model would be complex enough to learn something beyond a single bias term. So I'm quite confused as what I am doing wrong to make it not learning anything other than the bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What could be the possible reasons for the model to learn not much?&lt;/li&gt;&#xA;&lt;li&gt;What should I try and/or what kind of info should I be looking for at this point to move this forward?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time in advance~!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is a snippet of my code to show what I was trying to do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;learning_rate = 1e-4&#xA;batch_size = 10000&#xA;epochs = 5&#xA;&#xA;DENSE_SIZE = 1024&#xA;&#xA;train = (training_features.values, training_labels.values)&#xA;validation = (holdout_features.values, holdout_labels.values)&#xA;test = (eval_features.values, eval_labels.values)&#xA;eval_sets = [&#xA;    ('Train', train),&#xA;    ('Validation', validation),&#xA;    ('Test', test),&#xA;]&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(DENSE_SIZE, activation='relu', input_shape=train[0].shape[1:]))&#xA;model.add(Dense(DENSE_SIZE, activation='relu'))&#xA;model.add(Dense(DENSE_SIZE, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;# initiate optimizer&#xA;opt = RMSprop(lr=learning_rate)&#xA;&#xA;# Let's train the model&#xA;model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])&#xA;&#xA;model.fit(&#xA;    train[0],&#xA;    train[1],&#xA;    batch_size=batch_size,&#xA;    epochs=epochs,&#xA;    validation_data=validation,&#xA;    shuffle=True,&#xA;    verbose=2,&#xA;    callbacks=callbacks,&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The three answers as of writing all suggest that the inefficiency in learning is associated w/ the imbalanced classes and I should try to balance the classes by either using class weights or resampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I'm happy to see an apparent agreement of what the problem is what , could anyone provide more insights about why imbalanced classes cause difficulty even for a complex model? In theory a complex enough model can at least memorize all the rarely positive samples and achieve a higher-than-bias accuracy on the training set.&lt;/p&gt;&#xA;" OwnerUserId="29428" LastEditorUserId="29428" LastEditDate="2017-05-31T05:23:22.663" LastActivityDate="2017-05-31T05:23:22.663" Title="Why MLP only learns bias for unbalanced binary classification?" Tags="|classification|class-imbalance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19284" PostTypeId="1" CreationDate="2017-05-28T21:23:23.203" Score="1" ViewCount="192" Body="&lt;p&gt;After reading this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/#comment-129382&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; I have got a question about PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Author was talking about whether to use test set while computing PCA.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But, few important points to understand:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1) We should not combine the train and test set to obtain PCA components&#xA;  of whole data at once. Because, this would violate the entire&#xA;  assumption of generalization since test data would get ‘leaked’ into&#xA;  the training set. In other words, the test data set would no longer&#xA;  remain ‘unseen’. Eventually, this will hammer down the generalization&#xA;  capability of the model. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) We should not perform PCA on test and train&#xA;  data sets separately. Because, the resultant vectors from train and&#xA;  test PCAs will have different directions ( due to unequal variance).&#xA;  Due to this, we’ll end up comparing data registered on different axes.&#xA;  Therefore, the resulting vectors from train and test data should have&#xA;  same axes.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Author mentioned &quot;Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it... test data can only affect which PC(s) we will choose ( because of changed variance) but data can't leak really( let's say only metadata leak). For each observation we still have data only from it's own previous variables(predictors) just in lower dimensionality(projected). Am I right? or during calculations of matrixes data from test set can leak into? &lt;/p&gt;&#xA;" OwnerUserId="32757" LastActivityDate="2022-01-10T11:42:43.130" Title="Could data from a test set 'leak' into predictor during PCA?" Tags="|dimensionality-reduction|pca|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19285" PostTypeId="2" ParentId="9945" CreationDate="2017-05-28T21:54:20.747" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://www.h2o.ai/deep-water/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is really a wrapper over tensorflow, caffe, mxnet, but may be useful to you.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="29169" LastEditDate="2021-03-11T20:15:13.997" LastActivityDate="2021-03-11T20:15:13.997" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19286" PostTypeId="2" ParentId="19280" CreationDate="2017-05-28T21:57:03.803" Score="6" Body="&lt;p&gt;What piece of information are you trying to convey by presenting this plot? That determines what an appropriate plot type would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bar chart you show in your question would be useful if the specific indices of the probabilities are important - for example, if there could be something special about e.g. indices 504-559 and a person would want to see what probabilities go with those indices - but I suspect that's not the case. It's more likely that you're interested in how the probabilities correlate with actual results. To show that, you could start with a simple scatter plot with probability on the horizontal axis and actual result on the vertical axis. It also helps if you use different colors (and perhaps slightly different symbols) for actual results of 0 and 1. Here's a primitive example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LNG32.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LNG32.png&quot; alt=&quot;scatter plot of actual result vs probability&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option to show nearly the same information would be to group the probabilities into ranges and plot a histogram showing how many 1 results fall in each range. This better conveys how narrow the crossover from 0 to 1 predictions is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2L2BC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2L2BC.png&quot; alt=&quot;histogram of 1 results vs probability&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all you want to convey is how many predictions of each type were correct vs incorrect, then the confusion matrix &lt;a href=&quot;https://datascience.stackexchange.com/a/19282/695&quot;&gt;recommended by StatsSorceress&lt;/a&gt; might be a better choice, since it more directly presents that information. You could &quot;decorate&quot; it as a heatmap, if you think that would enhance the impact (i.e. if this is targeted at less numerically-minded people or it fits with an overall graphical theme), but that probably doesn't make much of a difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L1Erg.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L1Erg.png&quot; alt=&quot;heatmap confusion matrix&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="695" LastActivityDate="2017-05-28T21:57:03.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19290" PostTypeId="1" CreationDate="2017-05-29T02:58:22.033" Score="1" ViewCount="55" Body="&lt;p&gt;I have implemented a simple 1 hidden layer feed forward neural network in torch to learn X-OR operation. Below is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require 'torch'&#xA;require 'nn'&#xA;&#xA;m = nn.Sequential()&#xA;m:add(nn.Linear(2,2))&#xA;m:add(nn.Linear(2,1))&#xA;m:add(nn.Sigmoid())&#xA;&#xA;torch.manualSeed(1)&#xA;&#xA;m.modules[1].weights = torch.rand(2,2)&#xA;m.modules[2].weights = torch.rand(2,1)&#xA;&#xA;--print(parax_m)&#xA;&#xA;criterion = nn.BCECriterion()&#xA;&#xA;inputs = torch.Tensor(4,2)&#xA;inputs[1][1] = 0&#xA;inputs[1][2] = 0&#xA;&#xA;inputs[2][1] = 0&#xA;inputs[2][2] = 1&#xA;&#xA;inputs[3][1] = 1&#xA;inputs[3][2] = 0&#xA;&#xA;inputs[4][1] = 1&#xA;inputs[4][2] = 1&#xA;&#xA;targets = torch.Tensor(4,1)&#xA;targets[1][1] = 0&#xA;targets[2][1] = 1&#xA;targets[3][1] = 1&#xA;targets[4][1] = 0&#xA;&#xA;function trainEpoch(m,criterion,inputs,targets)&#xA;    for i=1,inputs:size(1) do&#xA;        local input = inputs[i]&#xA;        local target = targets[i]&#xA;        local output = m:forward(input)&#xA;        --print(output)&#xA;        local loss = criterion:forward(output,target)&#xA;        print(loss)&#xA;&#xA;            -- backward&#xA;        local gradOutput = criterion:backward(output,target)&#xA;        m:zeroGradParameters()&#xA;        local gradInput = m:backward(input,gradOutput)&#xA;        --update&#xA;        --module:updateGradParameters(0.9) -- momentum (require dpnn)&#xA;        m:updateParameters(0.01) -- W = W -0.1*dL/dW&#xA;    end&#xA;end&#xA;&#xA;for i=1,10000 do&#xA;    trainEpoch(m,criterion,inputs,targets)&#xA;end&#xA;&#xA;-- prediciton&#xA;testinput = torch.Tensor(4,2)&#xA;testinput[1][1] = 0&#xA;testinput[1][2] = 0&#xA;&#xA;testinput[2][1] = 0&#xA;testinput[2][2] = 1&#xA;&#xA;testinput[3][1] = 1&#xA;testinput[3][2] = 0&#xA;&#xA;testinput[4][1] = 1&#xA;testinput[4][2] = 1&#xA;&#xA;for i=1,testinput:size(1) do&#xA;    local output = m:forward(testinput[i])&#xA;    print(output)&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run the above code, there is no decay in loss(almost same in all iterations) so it does not predict the correct output. Can anyone help me to find the mistakes what I am doing wrong here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried with different manual seed value, different initialisation of weights but still loss remains same in all iterations.&lt;/p&gt;&#xA;" OwnerUserId="32764" LastActivityDate="2017-11-26T04:02:55.220" Title="No loss decay while learning Neural Network for X-OR operation in torch" Tags="|neural-network|torch|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19292" PostTypeId="2" ParentId="17230" CreationDate="2017-05-29T05:22:49.057" Score="3" Body="&lt;p&gt;In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix vector to generate the output sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answers of your questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you use 1 recurrent unit, it outputs the value at each time instant which is not suitable for machine translation as translation of first word may be in the last or somewhere else in target language.(I am not sure how will you map the output and input in that case.)&lt;/li&gt;&#xA;&lt;li&gt;Regarding synonyms, system can handle synonyms as words which are synonyms have very close word vectors.&lt;/li&gt;&#xA;&lt;li&gt;Regarding Word-Order, it can learn the word order if you have enough training data. Decoder also uses beam search to find the best possible translation. I will suggest you to read this paper thoroughly.&#xA;&lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Regarding evaluation, there are various metrics which are designed to evaluate the machine translation system. BLEU is one of the most famous one out of all. It uses various reference translations. You can read more about BLEU in the paper by Kishore Papineni et. al.(I can not post more than two link in answer as my reputation is less than 10.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I will also suggest you to read the below paper regarding neural machine translation.&#xA;&lt;a href=&quot;https://arxiv.org/pdf/1409.0473.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1409.0473.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!!&lt;/p&gt;&#xA;" OwnerUserId="32764" LastActivityDate="2017-05-29T05:22:49.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19293" PostTypeId="1" CreationDate="2017-05-29T05:55:41.147" Score="0" ViewCount="3855" Body="&lt;p&gt;I am curious how &lt;a href=&quot;http://seaborn.pydata.org/generated/seaborn.pairplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pairplot in seaborn library&lt;/a&gt; is constructed. I have seen similar plot, such as Pearson Correlation plot. What is the difference between these two plots? What is exact the use case of them?&lt;/p&gt;&#xA;" OwnerUserId="32770" LastEditorUserId="31513" LastEditDate="2017-05-29T12:05:29.933" LastActivityDate="2019-12-07T01:00:45.103" Title="How pairplot is constructed? Based on what rule? Why people use it?" Tags="|visualization|plotting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19294" PostTypeId="2" ParentId="19284" CreationDate="2017-05-29T06:09:03.557" Score="1" Body="&lt;p&gt;The author is right.  You are wrong.  There's no such thing as &quot;only metadata leak&quot; or &quot;can't leak really&quot;.  It's like saying &quot;not pregnant really&quot; -- either you're pregnant, or you're not.  Same here -- either data can leak, or it can't.  In this case, data can leak.  Maybe only partial data, but it's hard to know just how partial and just how bad the impact of that might be.  It might be very bad, or it might not; but since you can't know, testing with a methodology where you know the results might be meaningless isn't a good idea.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-29T06:09:03.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19295" PostTypeId="2" ParentId="19268" CreationDate="2017-05-29T06:13:13.287" Score="0" Body="&lt;p&gt;You can test each model with default parameters, and keep the model that seems best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, for each model, you can tune the parameters for that model, to see what is the best that's achievable for each model (by separately tuning its parameters) -- then keep the best model.  This takes more computation power but might be slightly better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, parameter tuning depends on the model.  Each model may have a very different set of parameters, with entirely different meaning. You can't take parameters for a random forest and then try to use them as parameters for a SVM (for instance); that doesn't even make sense, as they have different parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there are many possibilities on what order you do feature selection vs choosing a model vs parameter tuning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what the &quot;feature weights of FeatureUnion&quot; are, so I can't answer that part.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-05-29T06:13:13.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19298" PostTypeId="2" ParentId="19107" CreationDate="2017-05-29T11:32:33.210" Score="6" Body="&lt;p&gt;I've tried to explain the logic behind labels used in Document vectors in &lt;a href=&quot;https://datascience.stackexchange.com/questions/10216/doc2vec-how-to-label-the-paragraphs-gensim/10227#10227&quot;&gt;Doc2Vec - How to label the paragraphs (gensim)&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) when two documents share the same label, then doc2vec algorithm determines the semantic meaning of the label from both the documents. Note that doc2vec learns the semantic meanings of labels not individual documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Again, you are not learning documents. you are instructing doc2vec to learn the embeddings for the labels. So, if multiple labels are given for a document, all receives the same semantic meaning from the document and when a part of labels when used in other documents, keep on learning more semantic meaning from them. &#xA;For instance. doc1-&gt; &lt;code&gt;hunt, bite, eat, flesh&lt;/code&gt; doc2-&gt; &lt;code&gt;life, love, eat, money&lt;/code&gt;. It is clear that doc1 is about animals and doc2 is about human, the label &lt;code&gt;eat&lt;/code&gt; will have semantic meanings from both of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) If your goal is to find similar question such as this, then you should probably give just a single label for the whole question and then find a question with a label having closer cosine distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Never confuse labels with words. In word2vec, words learn embeddings and in doc2vec, labels learn embeddings from the words used in documents. If you would like to add some more semantic meanings to a document, then you could add it to the particular document as words. If you want to add semantic meanings to a label, the words have to be added to each document which carries that label if you want the label to have strong affinity (but adding words manually, doesnt sound like a good option to me personally).&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-05-29T11:32:33.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19301" PostTypeId="1" CreationDate="2017-05-29T18:21:27.990" Score="2" ViewCount="664" Body="&lt;p&gt;Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of the toolsets on the same boat as Python, R, and SQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But these days I have not heard its story. In fact, &lt;a href=&quot;https://trends.google.com/trends/explore?date=all&amp;amp;q=hadoop&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Google Trend also agrees with me&lt;/a&gt;, that the popularity is peak at May, 2015.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why has Hadoop failed to become one of the necessary tools as a Data Scientist, like Python or R?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-06-29T08:15:33.340" Title="Why has Hadoop failed to become popular?" Tags="|apache-hadoop|tools|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19302" PostTypeId="1" CreationDate="2017-05-29T20:37:52.753" Score="1" ViewCount="61" Body="&lt;p&gt;Ok, maybe this is a really dumb question, but has anyone ever considered extending some of the statistical learning methods to live in some space &lt;em&gt;other&lt;/em&gt; that $\mathbb R^n$? My guess is that you're just shuffling parameters around, but I wonder if there's any intuition to be gained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question came up as I was working through some course notes on neural nets, and realized that computer scientists may be even more cavalier with tensor indices than physicists :)&lt;/p&gt;&#xA;" OwnerUserId="32791" LastActivityDate="2017-05-29T21:44:51.463" Title="Machine Learning on Curved Background" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19303" PostTypeId="2" ParentId="19302" CreationDate="2017-05-29T21:39:43.070" Score="3" Body="&lt;p&gt;Essentially, each non-linear layer in a neural network is a map from $\mathbb{R}^n$ input to a $\mathbb{R}^m$ output. There is no requirement for these dimensions to be separate and uncorrelated, nor for the map to be reversible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best performance is usually found if each layer is roughly scaled to approximate a normal distribution with mean 0 and standard deviation 1. However, that is an independent issue to any kind of space-like mapping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know a suitable mapping that relates to your problem and would simplify its expression, then you can apply a spacial transform between co-ordinate systems (typically by mapping the input features) and use it within the neural network. You could map co-ordinates to any curved space if you think it might help. Choosing between polar or cartesian co-ordinates for features is one choice that may crop up for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, even without making this helpful mapping, you can analyse a neural network layer-by-layer, and see that it can effectively learn these kind of spacial mappings from the training data anyway. This is an intuition that has been called the &quot;&lt;a href=&quot;http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/&quot; rel=&quot;nofollow noreferrer&quot;&gt;manifold hypothesis&lt;/a&gt;&quot; - the link has some images showing how a neural network might learn to separate classes depending on how they are arranged in feature space, and &lt;a href=&quot;https://machinelearning.technicacuriosa.com/2017/03/19/a-darpa-perspective-on-artificial-intelligence/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article from a DARPA researcher&lt;/a&gt; has a nice series of images showing successive transformations of a 2D spiral that allow the last layer to perform linear separation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-05-29T21:44:51.463" LastActivityDate="2017-05-29T21:44:51.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19304" PostTypeId="2" ParentId="19162" CreationDate="2017-05-29T22:00:36.323" Score="1" Body="&lt;p&gt;a few things,&#xA;these images look huge. Are they resized or cropped before going into the net. Try resizing instead of cropping. Also, if you use flip/rotation augmentation that will mess up your network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, in the training data what % are right and what % are left throws? ideally, you have about 50:50 in the training set. If images in training are mostly left, it might learn to guess left preferentially. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, there might be an easier way to solve this problem. First, extract small patches and classify each patch as: 1 (contains a ball) or 0 (doesn't contain a ball) - this is segmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After segmentation, a simple algorithm that tells you if the ball is left of center or right of center will give you the correct answer. &lt;/p&gt;&#xA;" OwnerUserId="32793" LastActivityDate="2017-05-29T22:00:36.323" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19305" PostTypeId="1" CreationDate="2017-05-29T22:21:52.047" Score="3" ViewCount="2005" Body="&lt;p&gt;I am trying to write a question answer intent classification program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task is given a set of unlabelled question and answers, I have&#xA;to write a program where I may group all the similar questions &#xA;and identify their answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the answers for a group of similar questions are done, I have&#xA;to identify the intent or focus of answers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I have a set of questions like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a) Q: where is Texas? A : It is in USA.&#xA;b) Q: where is California? A: It is in USA.&#xA;c) Q: where is NASA? A: It is in USA.&#xA;d) Q: who is Queen Elizabeth II? A: Queen of England.&#xA;e) Q: who is Donald Trump? A : President of USA.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thus, &#xA;I am trying to group questions a, b &amp;amp; c as Location oriented question,&#xA;and d &amp;amp; e as Official/Person oriented question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To solve this problem, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use a standard classifier, and as it identifies 'It is in USA' as the class,&#xA;I am trying to tag it tag it as &quot;It/NA is/NA in/NA USA/LOC&quot; to identify intent/focus &#xA;of answer as Location. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a standard classifier like Naive Bayes and a standard Hidden Markov Model&#xA;based tagger. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result is more or less fine.&#xA;I am using two training set one for classifier and the other for tagging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any one of the esteemed scientists may kindly suggest how I am trying&#xA;to solve the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, Q means question, A means answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apology for cross-posting&lt;/p&gt;&#xA;" OwnerUserId="32792" LastEditorUserId="88379" LastEditDate="2020-03-03T11:28:09.560" LastActivityDate="2020-03-03T11:58:42.193" Title="Intent Classification in Question Answering" Tags="|machine-learning|classification|nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19306" PostTypeId="2" ParentId="19305" CreationDate="2017-05-30T00:12:47.407" Score="1" Body="&lt;p&gt;I don't know about the granularity of the intents. If they are just Person/Loc/Official, why not use question words(where,who,what etc.) to identify the intent? For example, &quot;where&quot; corresponds to &quot;Loc&quot;, &quot;Who&quot; corresponds to &quot;Person&quot; etc. You can also use this as a feature in you classifier. &#xA;Hope it helps!!&lt;/p&gt;&#xA;" OwnerUserId="32764" LastActivityDate="2017-05-30T00:12:47.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19307" PostTypeId="2" ParentId="19290" CreationDate="2017-05-30T01:45:53.157" Score="1" Body="&lt;p&gt;Finally I found the error in my Network.&#xA;1. I haven't added the non-linear layer after the first linear layer.&#xA;2. No randomisation while running stochastic gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By updating these two things, now it is working fine.&#xA;Updated Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require 'torch'&#xA;require 'nn'&#xA;&#xA;m = nn.Sequential()&#xA;m:add(nn.Linear(2,2))&#xA;m:add(nn.Tanh())&#xA;m:add(nn.Linear(2,1))&#xA;m:add(nn.Sigmoid())&#xA;&#xA;--print(parax_m)&#xA;&#xA;criterion = nn.BCECriterion()&#xA;&#xA;inputs = torch.Tensor(4,2)&#xA;inputs[1][1] = 0&#xA;inputs[1][2] = 0&#xA;&#xA;inputs[2][1] = 0&#xA;inputs[2][2] = 1&#xA;&#xA;inputs[3][1] = 1&#xA;inputs[3][2] = 0&#xA;&#xA;inputs[4][1] = 1&#xA;inputs[4][2] = 1&#xA;&#xA;targets = torch.Tensor(4,1)&#xA;targets[1][1] = 0&#xA;targets[2][1] = 1&#xA;targets[3][1] = 1&#xA;targets[4][1] = 0&#xA;&#xA;function trainEpoch(m,criterion,inputs,targets)&#xA;    for i=1,inputs:size(1) do&#xA;        local idx = math.random(1,4)&#xA;        local input = inputs[idx]&#xA;        local target = targets[idx]&#xA;        local output = m:forward(input)&#xA;        --print(output)&#xA;        local loss = criterion:forward(output,target)&#xA;        print(loss)&#xA;&#xA;            -- backward&#xA;        local gradOutput = criterion:backward(output,target)&#xA;        m:zeroGradParameters()&#xA;        local gradInput = m:backward(input,gradOutput)&#xA;        --update&#xA;        --module:updateGradParameters(0.9) -- momentum (require dpnn)&#xA;        m:updateParameters(0.01) -- W = W -0.1*dL/dW&#xA;    end&#xA;end&#xA;&#xA;for i=1,10000 do&#xA;    trainEpoch(m,criterion,inputs,targets)&#xA;end&#xA;&#xA;-- prediciton&#xA;testinput = torch.Tensor(4,2)&#xA;testinput[1][1] = 0&#xA;testinput[1][2] = 0&#xA;&#xA;testinput[2][1] = 0&#xA;testinput[2][2] = 1&#xA;&#xA;testinput[3][1] = 1&#xA;testinput[3][2] = 0&#xA;&#xA;testinput[4][1] = 1&#xA;testinput[4][2] = 1&#xA;&#xA;for i=1,testinput:size(1) do&#xA;    local output = m:forward(testinput[i])&#xA;    print(output)&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32764" LastActivityDate="2017-05-30T01:45:53.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19308" PostTypeId="1" CreationDate="2017-05-30T02:36:49.133" Score="0" ViewCount="146" Body="&lt;p&gt;I am having the problem with the following python code which uses the library keras, and I have not been able to find a solution online:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       def model_build():&#xA;        model_mag=Sequential()&#xA;        model_mag.add(Dense(64, activation='relu', input_dim=5))  &#xA;        model_mag.add(Dropout(0.5))&#xA;        model_mag.add(Dense(64, activation='relu'))&#xA;        model_mag.add(Dropout(0.5))&#xA;        model_mag.add(Dense(64, activation='relu'))&#xA;        model_mag.add(Dense(1))&#xA;        model_mag.compile(loss=losses.mean_squared_error,optimizer='sgd')&#xA;        return model_mag&#xA;       clf = KerasRegressor(build_fn=model_build(),nb_epoch=100)&#xA;       # here is the error&#xA;       clf.fit(X_train,y_train, nb_epoch=20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that X_train and y_train are well defined above and I can run other classifier methods just fine with them so they are not the problem. It keeps throwing me the following error in the fit line and I can't figure out why:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TypeError: &lt;strong&gt;call&lt;/strong&gt;() takes exactly 2 arguments (1 given)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am giving two arguments though. &lt;/p&gt;&#xA;" OwnerUserId="32795" LastActivityDate="2017-06-16T16:42:58.260" Title="Error in Keras Fitting" Tags="|neural-network|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19310" PostTypeId="2" ParentId="19276" CreationDate="2017-05-30T05:57:44.537" Score="1" Body="&lt;p&gt;When considering how to clean the text, we should think about the data problem we are trying to solve. Here are few more step for preprocessing which can improve your features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.) Use Good tokenizer(textblob,stanford tokenizer)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.) Try Lemmatization , stemming always not perform well in case news article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.) word segmentation &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4.) Normalization (equivalence classing of terms)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For selecting model&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.) In your example above, we classified the document by comparing the number of matching terms in the document vectors. In the real world numerous more complex algorithms exist for classification such as &lt;strong&gt;Support Vector Machines (SVMs)&lt;/strong&gt;, &lt;strong&gt;Naive Bayes&lt;/strong&gt; and &lt;strong&gt;Decision Trees&lt;/strong&gt; , &lt;strong&gt;Maximum Entropy&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.) You can think your problem as making clusters of news and getting semantic relationship of source news from these cluster. You can try topic modelling(&lt;strong&gt;LDA and LSA&lt;/strong&gt;) and &lt;strong&gt;Doc2vec/word2vec&lt;/strong&gt; technique for getting vector for document/word and then use these vectors for classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further if you are confuse to select a appropriate model for problem , you can read from this link &lt;a href=&quot;http://machinelearningmastery.com/choosing-machine-learning-algorithms-lessons-from-microsoft-azure/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Choosing Machine Learning Algorithms: Lessons from Microsoft Azure&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32686" LastEditorUserId="32686" LastEditDate="2017-07-30T13:43:08.077" LastActivityDate="2017-07-30T13:43:08.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19311" PostTypeId="2" ParentId="19108" CreationDate="2017-05-30T06:10:06.977" Score="2" Body="&lt;p&gt;I think you should first clearly specify what the covariates are, what is the target variable is and what is your goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if you have attributes about the car as covariates and target variable is failure time (car failed in 1y, 2y...), then the best approach is indeed &lt;strong&gt;Survival analysis&lt;/strong&gt;, because you try to model time to failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other side, if you target variable is just a failure - yes or no, then its a &lt;strong&gt;classification problem&lt;/strong&gt;. For that, simple models such as Decision trees or Logistic regression are very well suited. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dont use algorithm just because its fancy or dont dislike other approaches just because they are not &quot;Machine Learning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You said you want to start with Machine learning, so go ahead. Dont blindly fit whatever blackbox models, start with simple ones and look inside, how they work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, pick something more simple. Because Survival analysis requires knowledge or regression and a bit more of statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-05-30T06:10:06.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19312" PostTypeId="2" ParentId="19301" CreationDate="2017-05-30T06:22:05.267" Score="3" Body="&lt;p&gt;I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python or R are handy at the beginning when you just need to try out things but when it comes to putting things in production, Hadoop is the way to go. It does not directly provide any data scientist tools what it does provide is the base where data would be stored, processed and can be used to apply machine learning algorithms using Spark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to summarize, I see Hadoop being used in the following contexts&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;For Data Storage in Production Systems&lt;/li&gt;&#xA;&lt;li&gt;Hadoop YARN as a cluster manager for tools like Spark, Flink&lt;/li&gt;&#xA;&lt;li&gt;If someone still wants to code in R/Python/SAS - Hadoop being used as backend. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.  &lt;/p&gt;&#xA;" OwnerUserId="32806" LastActivityDate="2017-05-30T06:22:05.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19313" PostTypeId="2" ParentId="19293" CreationDate="2017-05-30T06:25:23.073" Score="0" Body="&lt;p&gt;I can advise only for the usage. For the construction, you can simply look into the source code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pairplot and Pearson are actually almost same. Both plot two variables on a 2-dimensional plot, usually referred as X and Y to observe the relationship. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However Person correlation plot is suited only for continuous variables (such as age, amount of money, height, weight etc) because it calculates the correlation coefficient (which is incorrect for non-continuous data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, pairplot is made when you only want to visualize the relationship between two variables, where the variables can be continous, categorical (product type or part of day for example) or booleans (yes/no - 1/0). Furthermore, pairplot is usually a grid of plots for each variable in your dataset. Hence you can quickly see how all the variables are related. This can help to infere which variables are useful, which have skewed distribution etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You usually do this as a first step of your analysis - to understand better your data so you can afterwards choose the right model.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="676" LastEditDate="2017-05-30T07:12:35.240" LastActivityDate="2017-05-30T07:12:35.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19314" PostTypeId="2" ParentId="19301" CreationDate="2017-05-30T08:04:56.677" Score="3" Body="&lt;p&gt;You are comparing apples with oranges. Hadoop is one of the backend of big data platform and Python/R are programming languages which are used built prediction models and data pipeline. Hadoop still can be used as data storage however more robust and faster distributed data storage frameworks are gaining popularity such as Apache Spark hence Hadoop lost its charm.&lt;/p&gt;&#xA;" OwnerUserId="32805" LastActivityDate="2017-05-30T08:04:56.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19315" PostTypeId="2" ParentId="19283" CreationDate="2017-05-30T08:55:30.867" Score="2" Body="&lt;p&gt;Though you know that dataset is naturally balanced but your model will still consider it as unbalanced dataset. If you take a example financial fraud detection, number of fraudulent transactions are always less than the ligit translations, to handle such such cases data scientists will have to balance the classes for better results.&#xA;You will have to balance your dataset to get the better results from model.&lt;/p&gt;&#xA;" OwnerUserId="32805" LastEditorUserId="32805" LastEditDate="2017-05-30T11:58:00.240" LastActivityDate="2017-05-30T11:58:00.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19316" PostTypeId="2" ParentId="10523" CreationDate="2017-05-30T09:42:23.970" Score="18" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse data such as tf-idf, etc. Adagrad can be useful.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;AdaDelta, RMSProp almost works on similar lines with the only difference in Adadelta you don't require an initial learning rate constant to start with.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Adam combines the good properties of Adadelta and RMSprop and hence tend to do better for most of the problems.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Stochastic gradient descent is very basic and is seldom used now. One problem is with the global learning rate associated with the same. Hence it doesn't work well when the parameters are in different scales since a low learning rate will make the learning slow while a large learning rate might lead to oscillations. Also Stochastic gradient descent generally has a hard time escaping the saddle points. Adagrad, Adadelta, RMSprop, and ADAM generally handle saddle points better. SGD with momentum renders some speed to the optimization and also helps escape local minima better.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28337" LastEditorUserId="29169" LastEditDate="2021-02-09T01:30:47.640" LastActivityDate="2021-02-09T01:30:47.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19317" PostTypeId="1" CreationDate="2017-05-30T09:43:38.663" Score="0" ViewCount="286" Body="&lt;p&gt;I understand stochastic gradient descent does. any other alorithm which supports incremental training preferably updating on a real time basis?&lt;/p&gt;&#xA;" OwnerUserId="30693" LastActivityDate="2017-05-30T10:10:14.883" Title="Which machine learning algorithms support a feedback ingestion loop?" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19318" PostTypeId="2" ParentId="19317" CreationDate="2017-05-30T10:10:14.883" Score="1" Body="&lt;p&gt;A few other methods support incremental learning. (Sorry it should be a comment instead but I don't have enough reputation ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32816" LastActivityDate="2017-05-30T10:10:14.883" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19319" PostTypeId="1" CreationDate="2017-05-30T11:49:24.857" Score="1" ViewCount="1260" Body="&lt;p&gt;In caret package of R, there is a method 'xgblinear'. What is the working algorithm behind this method.&lt;/p&gt;&#xA;" OwnerUserId="32172" LastActivityDate="2024-04-05T10:01:10.160" Title="On which algorithm for boosting is the method xgbLinear of the xgboost/caret- package based on?" Tags="|machine-learning|r|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19320" PostTypeId="2" ParentId="19127" CreationDate="2017-05-30T12:54:31.893" Score="1" Body="&lt;p&gt;As far as I understood you are using type of TV as &lt;strong&gt;tag&lt;/strong&gt; of particular sentence , and you are using &lt;strong&gt;doc2vec model&lt;/strong&gt; for future classification . So As above answer is suggesting that model will learn &lt;strong&gt;semantic meaning&lt;/strong&gt; of &lt;strong&gt;type of TV(tag)&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;let's suppose &lt;strong&gt;s&lt;/strong&gt; is your future sentence for prediction. then you use &lt;strong&gt;infer vector&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;model = Load_model('model.doc2vec')&lt;/p&gt;&#xA;&lt;p&gt;infer_vector = model.infer_vector(s)&lt;/p&gt;&#xA;&lt;p&gt;similar_documents = model.docvecs.most_similar([infer_vector], topn = 1)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;here similar document is list of tuples. where first element is &lt;strong&gt;label&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;let me know if this help you.&lt;/p&gt;&#xA;" OwnerUserId="32686" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-03T16:29:27.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19321" PostTypeId="2" ParentId="19283" CreationDate="2017-05-30T12:58:46.497" Score="2" Body="&lt;p&gt;Generally there are a number of ways to deal with class imbalance (&lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;more details here fore example&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically in your case, I would either try training with the training samples evenly distributed in classes or I would play around changing the loss function, to take into account the different classes' support. For the latter, you can check &lt;a href=&quot;https://stackoverflow.com/questions/35155655/loss-function-for-class-imbalanced-binary-classifier-in-tensor-flow&quot;&gt;this discussion also&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-05-30T12:58:46.497" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19322" PostTypeId="1" CreationDate="2017-05-30T13:18:37.667" Score="1" ViewCount="337" Body="&lt;p&gt;The Yelp Dataset Challenge (&lt;a href=&quot;https://www.yelp.com/dataset_challenge&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.yelp.com/dataset_challenge&lt;/a&gt;) releases data for a handful of cities each year. I'd like to analyze some cities from past years. Is this data archived anywhere?&lt;/p&gt;&#xA;" OwnerUserId="10799" LastActivityDate="2017-06-08T20:57:09.837" Title="Yelp Dataset Archives?" Tags="|data-mining|dataset|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19323" PostTypeId="1" AcceptedAnswerId="19327" CreationDate="2017-05-30T13:58:08.027" Score="0" ViewCount="166" Body="&lt;p&gt;Is there a way or tools available to generate or retrieve cypher query from a Neo4j database ? Should we need to store cypher quries along with the graph data for regeneration ?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-05-30T14:33:34.173" Title="Neo4j graph to cypher conversion" Tags="|graphs|neo4j|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19325" PostTypeId="2" ParentId="19319" CreationDate="2017-05-30T14:24:21.387" Score="0" Body="&lt;p&gt;I am not sure if this is helpful or not but here is a link to an academic paper detailing both the algorithm behind XGBoost and some use cases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dmlc.cs.washington.edu/data/pdf/XGBoostArxiv.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://dmlc.cs.washington.edu/data/pdf/XGBoostArxiv.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32664" LastActivityDate="2017-05-30T14:24:21.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19327" PostTypeId="2" ParentId="19323" CreationDate="2017-05-30T14:33:34.173" Score="1" Body="&lt;p&gt;This has already been answered in &lt;a href=&quot;https://stackoverflow.com/q/18830686/674487&quot;&gt;stackoverflow&lt;/a&gt;. There is no need to save the queries you used to populate the database, you only need to dump the contents of the database to a file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;db1/neo4j-shell -path db1/data/graph.db/ -c dump &amp;gt; export_data.cypher&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In order to load the database dump into another database, you just supply it to neo4j shell through the standard intput:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;db2/bin/neo4j-shell -path db2/data/graph.db/ &amp;lt; export_data.cypher&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-30T14:33:34.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19330" PostTypeId="2" ParentId="19283" CreationDate="2017-05-30T15:20:58.570" Score="1" Body="&lt;p&gt;It is not a matter of model complexity. In order to address class imbalance, you should specify class weights as argument to the &lt;code&gt;fit&lt;/code&gt; function in order for the logits of the cross entropy to be scaled accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find further info in &lt;a href=&quot;https://stackoverflow.com/q/35155655/674487&quot;&gt;stackoverflow&lt;/a&gt; and examples on how to provide the weights to &lt;code&gt;fit&lt;/code&gt; &lt;a href=&quot;https://datascience.stackexchange.com/q/13490/14675&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-30T15:20:58.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19332" PostTypeId="2" ParentId="19272" CreationDate="2017-05-30T15:51:15.217" Score="25" Body="&lt;p&gt;Working definitions of ReLU function and its derivative:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$ReLU(x) = \begin{cases}&#xA;  0, &amp;amp; \text{if } x &amp;lt; 0, \\&#xA;  x, &amp;amp; \text{otherwise}.&#xA;\end{cases}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{d}{dx} ReLU(x) = \begin{cases}&#xA;  0, &amp;amp; \text{if } x &amp;lt; 0, \\&#xA;  1, &amp;amp; \text{otherwise}.&#xA;\end{cases}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The derivative is the unit &lt;a href=&quot;https://en.wikipedia.org/wiki/Step_function&quot; rel=&quot;noreferrer&quot;&gt;step function&lt;/a&gt;. This does ignore a problem at &lt;span class=&quot;math-container&quot;&gt;$x=0$&lt;/span&gt;, where the gradient is not strictly defined, but that is not a practical concern for neural networks. With the above formula, the derivative at 0 is 1, but you could equally treat it as 0, or 0.5 with no real impact to neural network performance.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Simplified network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With those definitions, let's take a look at your example networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are running regression with cost function &lt;span class=&quot;math-container&quot;&gt;$C = \frac{1}{2}(y-\hat{y})^2$&lt;/span&gt;. You have defined &lt;span class=&quot;math-container&quot;&gt;$R$&lt;/span&gt; as the output of the artificial neuron, but you have not defined an input value. I'll add that for completeness - call it &lt;span class=&quot;math-container&quot;&gt;$z$&lt;/span&gt;, add some indexing by layer, and I prefer lower-case for the vectors and upper case for matrices, so &lt;span class=&quot;math-container&quot;&gt;$r^{(1)}$&lt;/span&gt; output of the first layer, &lt;span class=&quot;math-container&quot;&gt;$z^{(1)}$&lt;/span&gt; for its input and &lt;span class=&quot;math-container&quot;&gt;$W^{(0)}$&lt;/span&gt; for the weight connecting the neuron to its input &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; (in a larger network, that might connect to a deeper &lt;span class=&quot;math-container&quot;&gt;$r$&lt;/span&gt; value instead). I have also adjusted the index number for the weight matrix - why that is will become clearer for the larger network. NB I am ignoring having more than neuron in each layer for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at your simple 1 layer, 1 neuron network, the feed-forward equations are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$z^{(1)} = W^{(0)}x$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\hat{y} = r^{(1)} = ReLU(z^{(1)})$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The derivative of the cost function w.r.t. an example estimate is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial \hat{y}} = \frac{\partial C}{\partial r^{(1)}} = \frac{\partial}{\partial r^{(1)}}\frac{1}{2}(y-r^{(1)})^2 = \frac{1}{2}\frac{\partial}{\partial r^{(1)}}(y^2 - 2yr^{(1)} + (r^{(1)})^2) = r^{(1)} - y$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the chain rule for back propagation to the pre-transform (&lt;span class=&quot;math-container&quot;&gt;$z$&lt;/span&gt;) value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial z^{(1)}} = \frac{\partial C}{\partial r^{(1)}} \frac{\partial r^{(1)}}{\partial z^{(1)}} = (r^{(1)} - y)Step(z^{(1)}) = (ReLU(z^{(1)}) - y)Step(z^{(1)})$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial z^{(1)}}$&lt;/span&gt; is an interim stage and critical part of backprop linking steps together. Derivations often skip this part because clever combinations of cost function and output layer mean that it is simplified. Here it is not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get the gradient with respect to the weight &lt;span class=&quot;math-container&quot;&gt;$W^{(0)}$&lt;/span&gt;, then it is another iteration of the chain rule:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial W^{(0)}} = \frac{\partial C}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial W^{(0)}} = (ReLU(z^{(1)}) - y)Step(z^{(1)})x = (ReLU(W^{(0)}x) - y)Step(W^{(0)}x)x$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . . because &lt;span class=&quot;math-container&quot;&gt;$z^{(1)} = W^{(0)}x$&lt;/span&gt; therefore &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial z^{(1)}}{\partial W^{(0)}} = x$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is the full solution for your simplest network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in a layered network, you also need to carry the same logic down to the next layer. Also, you typically have more than one neuron in a layer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More general ReLU network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we add in more generic terms, then we can work with two arbitrary layers. Call them Layer &lt;span class=&quot;math-container&quot;&gt;$(k)$&lt;/span&gt; indexed by &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;, and Layer &lt;span class=&quot;math-container&quot;&gt;$(k+1)$&lt;/span&gt; indexed by &lt;span class=&quot;math-container&quot;&gt;$j$&lt;/span&gt;. The weights are now a matrix. So our feed-forward equations look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$z^{(k+1)}_j = \sum_{\forall i} W^{(k)}_{ij}r^{(k)}_i$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$r^{(k+1)}_j = ReLU(z^{(k+1)}_j)$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the output layer, then the initial gradient w.r.t. &lt;span class=&quot;math-container&quot;&gt;$r^{output}_j$&lt;/span&gt; is still  &lt;span class=&quot;math-container&quot;&gt;$r^{output}_j - y_j$&lt;/span&gt;. However, ignore that for now, and look at the &lt;em&gt;generic&lt;/em&gt; way to back propagate, assuming we have already found &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial r^{(k+1)}_j}$&lt;/span&gt; - just note that this is ultimately where we get the output cost function gradients from. Then there are 3 equations we can write out following the chain rule:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we need to get to the neuron input before applying ReLU: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j} \frac{\partial r^{(k+1)}_j}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j}Step(z^{(k+1)}_j)$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;We also need to propagate the gradient to previous layers, which involves summing up all connected influences to each neuron:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} W^{(k)}_{ij}$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And we need to connect this to the weights matrix in order to make adjustments later:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} r^{(k)}_{i}$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can resolve these further (by substituting in previous values), or combine them (often steps 1 and 2 are combined to relate pre-transform gradients layer by layer). However the above is the most general form. You can also substitute the &lt;span class=&quot;math-container&quot;&gt;$Step(z^{(k+1)}_j)$&lt;/span&gt; in equation 1 for whatever the derivative function is of your current activation function - this is the only place where it affects the calculations.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Back to your questions:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If this derivation is correct, how does this prevent vanishing?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your derivation was not correct. However, that does not completely address your concerns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between using sigmoid versus ReLU is just in the step function compared to e.g. sigmoid's &lt;span class=&quot;math-container&quot;&gt;$y(1-y)$&lt;/span&gt;, applied once per layer. As you can see from the generic layer-by-layer equations above, the gradient of the transfer function appears in one place only. The sigmoid's best case derivative adds a factor of 0.25 (when &lt;span class=&quot;math-container&quot;&gt;$x = 0, y = 0.5$&lt;/span&gt;), and it gets worse than that and saturates quickly to near zero derivative away from &lt;span class=&quot;math-container&quot;&gt;$x=0$&lt;/span&gt;. The ReLU's gradient is either 0 or 1, and in a healthy network will be 1 often enough to have less gradient loss during backpropagation. This is not guaranteed, but experiments show that ReLU has good performance in deep networks.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes this can have an impact too. This can be a problem regardless of transfer function choice. In some combinations, ReLU may help keep exploding gradients under control too, because it does not saturate (so large weight norms will tend to be poor direct solutions and an optimiser is unlikely to move towards them). However, this is not guaranteed.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2019-01-03T11:28:52.830" LastActivityDate="2019-01-03T11:28:52.830" CommentCount="11" ContentLicense="CC BY-SA 4.0" />
  <row Id="19337" PostTypeId="2" ParentId="6744" CreationDate="2017-05-30T19:21:47.303" Score="0" Body="&lt;p&gt;As far as PCA components are concerned you should use the same number of PCA for test data. The logic behind this is that same transformation should happen to test data that happened to train data. I am assuming here that your train and test data are independently drawn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, it is important to normalize your data before applying PCA. There is a science behind it where it calculates the importance of a variable and if you won't normalize, it will give high weightage to values which are numerically high in value so it's important to normalize before PCA. If you want to apply SVM on top of it normalization is fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic says that scale should remain same for any process thus after normalization you should use the same scale that is same max and min.&lt;/p&gt;&#xA;" OwnerUserId="20517" LastActivityDate="2017-05-30T19:21:47.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19339" PostTypeId="2" ParentId="19308" CreationDate="2017-05-30T20:11:38.137" Score="1" Body="&lt;p&gt;You have put &lt;code&gt;nb_epoch&lt;/code&gt; in &lt;code&gt;fit&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have to remove &lt;code&gt;nb_epoch&lt;/code&gt; from &lt;code&gt;fit&lt;/code&gt;, and let it only in &lt;code&gt;KerasRegressor&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2017-05-30T20:11:38.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19340" PostTypeId="1" CreationDate="2017-05-30T20:16:15.677" Score="1" ViewCount="512" Body="&lt;p&gt;I am trying to get an intuitive idea of RBMs out of curiosity, and using a &lt;a href=&quot;https://youtu.be/-l1QTbgLTyQ?t=14m10s&quot; rel=&quot;nofollow noreferrer&quot;&gt;simple example on youtube&lt;/a&gt; based on preferences for different sports, which denote profiles roughly corresponding to &lt;code&gt;outdoor&lt;/code&gt; and &lt;code&gt;indoor&lt;/code&gt; predilections. The idea is to target ads. Here is the model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7obR4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7obR4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The examples in the sample are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example Football Hockey Cricket Table-tennis Chess Badminton      CLASS&#xA;&#xA;P1          1       1       1         0        0      0           out&#xA;P2          1       0       0         1        1      1           in&#xA;P3          0       0       0         1        1      0           in&#xA;P4          1       1       0         0        0      0           out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The starting values are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W_i = 0.5 \quad A_i=0.2 \quad B_i=0.3.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the first example,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{matrix}V_1 =V_2=V_3=1;\quad V_4=V_5=V_6=0 \end{matrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the first step (updating hidden unit) with the first example $P1$:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;$\begin{matrix}\text{Example}&amp;amp;\text{football}&amp;amp;\text{hockey}&amp;amp;\text{cricket}&amp;amp;\text{tt}&amp;amp;\text{chess}&amp;amp;\text{badm.}\\\text{P1}&amp;amp;\color{blue}1&amp;amp;\color{blue}1&amp;amp;\color{blue}1&amp;amp;\color{blue}0&amp;amp;\color{blue}0&amp;amp;\color{blue}0\end{matrix}$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;is calculated (on the video) as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{align}\small\text{net}(H_1)&amp;amp;=(V_1H_1W_{11} + V_2H_1W_{12} + V3H_1W_{13} + V_4H_1W_{14} + V_5H_1W_{15} + V_6H_1W_{16})+B_1\\[2ex]&#xA;&amp;amp;=\color{blue}1\times 1\times 0.5 +\color{blue}1\times 1\times 0.5 +\color{blue}1\times 1\times 0.5 +\underset{\text{whether the 0 term is } V \text{or} H \text{ or both is not specified}}{\underbrace{0+0+0}}+0.3&#xA;\end{align}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{align}\small\text{net}(H_2)&amp;amp;=(V_1H_2W_{21} + V_2H_2W_{22} + V3H_2W_{23} + V_4H_2W_{24} + V_5H_2W_{25} + V_6H_2W_{26})+B_2\\[2ex]&amp;amp;=\small \color{blue}0\times 1\times 0.5 +\color{blue}0\times 1\times 0.5 +\color{blue}0\times 1\times 0.5 +\color{blue}1\times 1 \times 0.5+\color{blue}1\times 1 \times 0.5+\color{blue}1\times 1 \times 0.5+0.3\end{align}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the actual screen capture (slightly modified):&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e8r7b.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e8r7b.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;which seems to imply a change in the $V_i$ values for &lt;code&gt;table-tennis&lt;/code&gt;, &lt;code&gt;chess&lt;/code&gt; and &lt;code&gt;badminton&lt;/code&gt; when transitioning from the update of $H_1$ to the update of $H_2$. Although I'm sure erroneously, I was expecting the following calculation, which would leave the $V_i$ values unchanged (blue for $V_i$). The  red integers corresponding to $H_i$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{align}\small\text{net}(H_1)&amp;amp;=(V_1H_1W_{11} + V_2H_1W_{12} + V3H_1W_{13} + V_4H_1W_{14} + V_5H_1W_{15} + V_6H_1W_{16})+B_1\\[2ex]&#xA;&amp;amp;=\small \color{blue} 1\times \color{red}1\times 0.5 +\color{blue}1\times \color{red}1\times 0.5 +\color{blue}1\times \color{red}1\times 0.5 &#xA;+\color{blue}0\times \color{red}{0}\times 0.5 +\color{blue}0\times \color{red}{0}\times 0.5+\color{blue}0\times \color{red}{0}\times 0.5 +0.3\end{align}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{align}\small\text{net}(H_2)&amp;amp;=(V_1H_2W_{21} + V_2H_2W_{22} + V3H_2W_{23} + V_4H_2W_{24} + V_5H_2W_{25} + V_6H_2W_{26})+B_2\\[2ex]&#xA;&amp;amp;=\small \color{blue}1\times \color{red}0\times 0.5 +\color{blue}1\times \color{red}0\times 0.5 +\color{blue}1\times \color{red}0\times 0.5 &#xA;+\color{blue}0\times \color{red}1\times 0.5 +\color{blue}0\times\color{red} 1\times 0.5+\color{blue}0\times\color{red} 1\times 0.5 +0.3\end{align}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can I get an explanation as to how the example $P1$ is not ignored in the first updated of the hidden unit?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Evidently following the example would be much more self-explanatory if $P1$ wasn't such a pure &lt;code&gt;outdoor&lt;/code&gt; case...&lt;/p&gt;&#xA;" OwnerUserId="17370" LastEditorUserId="17370" LastEditDate="2017-05-30T20:41:03.950" LastActivityDate="2017-05-30T20:41:03.950" Title="Understanding a simple example of Restricted Boltzmann Machine (RBM)" Tags="|neural-network|rbm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19341" PostTypeId="1" CreationDate="2017-05-30T20:21:54.660" Score="-1" ViewCount="369" Body="&lt;p&gt;I have the head of my dataframe here &#xA;data  tMax  tMin  tMed precipitacao&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   &amp;lt;chr&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;        &amp;lt;dbl&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;1 30/09/2016    29  17.8  23.4          0.0&#xA;2 29/09/2016    28  16.2  22.1          0.0&#xA;3 28/09/2016    24  17.2  20.6         11.4&#xA;4 27/09/2016    24  16.6  20.3         43.4&#xA;5 26/09/2016    26  17.2  21.6          0.0&#xA;6 25/09/2016    31  14.4  22.7          0.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I have this date format(xx/yy/zz). I'd like to aggregate my other variables(tMax,tMin,tMax and precipitacao) by year and by month.&#xA;Someone pls ;D&lt;/p&gt;&#xA;" OwnerUserId="32835" LastActivityDate="2017-07-29T21:51:21.490" Title="Aggregate function by date" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19342" PostTypeId="2" ParentId="19341" CreationDate="2017-05-30T20:51:31.953" Score="1" Body="&lt;p&gt;First result on Google :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/33221425/how-do-i-group-my-date-variable-into-month-year-in-r&quot;&gt;https://stackoverflow.com/questions/33221425/how-do-i-group-my-date-variable-into-month-year-in-r&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Get dplyr&lt;/li&gt;&#xA;&lt;li&gt;Add columns to your dataframe for year and month.&lt;/li&gt;&#xA;&lt;li&gt;Group_by year and month and take averages.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-05-30T20:51:31.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19343" PostTypeId="1" CreationDate="2017-05-30T22:16:54.357" Score="1" ViewCount="67" Body="&lt;p&gt;Let me get straigt into it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of an scheduled operation. There is a surgeon and, say, three other medical personel that carry out various tasks. For instance, a nurse, an anesthesiologist, and a backup.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each one reports to duty at a certain time. However, with some probability p (to be predicted) one or many member(s) of the team won't show up for work. If an individual does not show up for work, but his particular skill set is still covered due to the backup, then the operation is done. Otherwise, it is cancelled. For instance, if the backup is a certified anesthesiologist, absence of the scheduled anesthesiologist does not prevent the surgery from taking place since a minimum required skill mix is given. If, however, the backup is a nurse, and the anesthesiologist does not show up, then the surgery has to be cancelled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem involves two layers: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does an individual team member show up for work? &lt;/li&gt;&#xA;&lt;li&gt;Is an individual team member's skill critical for the execution of the surgery. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My goal is to predict cancellation of the surgery, which involves step 1 and step 2 simulataneously (alternatively, you could think of step 2 being conditional on step 1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a good understanding of how to use standard machine learning algorithms to predict 1. But how do I generate final predictions taking both layers into account? The problem set up seems reasonably general so that I would assume there exists a &quot;standard&quot; method? &lt;/p&gt;&#xA;" OwnerUserId="13021" LastActivityDate="2017-06-06T17:50:29.903" Title="Problem with two layers: how to predict?" Tags="|machine-learning|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19344" PostTypeId="1" CreationDate="2017-05-31T00:51:38.923" Score="5" ViewCount="786" Body="&lt;p&gt;Let us say we are using a neural network with &lt;span class=&quot;math-container&quot;&gt;$4$&lt;/span&gt; layers with &lt;span class=&quot;math-container&quot;&gt;$50,30,20,10$&lt;/span&gt; neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier layer (say first) would be significantly lower than the rate of change of the parameters associated with the later on layers (say 4th).&lt;/p&gt;&#xA;&lt;p&gt;Now the cost function spans across &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt;-dimensions where &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; is the total number of the parameters and we try to find the minimum of this cost function by varying it across these dimensions.&lt;/p&gt;&#xA;&lt;p&gt;My question is why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.&lt;/p&gt;&#xA;&lt;p&gt;Here is the help I can provide on this: this is from &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap5.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;neuralnetworksanddeeplearning.com&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function &lt;span class=&quot;math-container&quot;&gt;$f(x)$&lt;/span&gt; of a single variable. Wouldn't it be good news if the derivative &lt;span class=&quot;math-container&quot;&gt;$f′(x)$&lt;/span&gt; was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?&lt;/p&gt;&#xA;&lt;p&gt;Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a &lt;span class=&quot;math-container&quot;&gt;$[784,30,30,30,10]$&lt;/span&gt; network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Specific problems from the excerpt:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. &lt;strong&gt;Why? Why cant the cost function be nearly flat in the dimensions created by the earlier weights. To be precise why is the explanation not the the last 3 layers are enough to do such a bangup job that they always find weights corresponding to a very small proximity of whatever random weights we initialized the first layer to. Potato pothatoe. i am asking the same thing in different words to make sure i convey myself&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To be concrete, consider the first layer of weights in a &lt;span class=&quot;math-container&quot;&gt;$[784,30,30,30,10]$&lt;/span&gt; network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. &lt;strong&gt;What does this mean? Kindly explain. from what i understand it is that we cant trace back the second layer inputs unambiguously to the first layer inputs. but, even if we are initializing the weights randomly, we still know what those weights are so how is it that we are throwing away the information?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-10-02T12:15:16.850" Title="Why is vanishing gradient a problem?" Tags="|neural-network|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="19345" PostTypeId="2" ParentId="19271" CreationDate="2017-05-31T05:18:41.977" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For linear regression, we have to do one hot encoding and it creates one less number of variables then levels of the categorical variable. In latest tools you don't have to do it manually it automatically does I have tried in R. One hot encoding has no impact on which level you choose, it is basically a binary way of representing the level True or False. This will create whole new variables which with just 1 and 0.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For tree-based algorithms, you basically don't need to apply the one hot encoding. These algorithms are capable enough to handle the categorical variables. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;https://gerardnico.com/wiki/data_mining/dummy&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gerardnico.com/wiki/data_mining/dummy&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20517" LastEditorUserId="20517" LastEditDate="2017-06-01T05:54:51.320" LastActivityDate="2017-06-01T05:54:51.320" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19346" PostTypeId="1" CreationDate="2017-05-31T05:31:19.587" Score="6" ViewCount="2135" Body="&lt;p&gt;In the paper, &quot;&lt;a href=&quot;https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hierarchical Attention Networks for Document Classification&lt;/a&gt;&quot;, the authors use attention to compose words to sentences and then sentences to a document representation. They make use of a context vector $u_w$ to compute the attention weights for the annotation of each word in the sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper states in section 2.2, &quot;The context vector $u_w$ can be seen as a high level representation of a fixed query “what is the informative word” over the words. The word context vector $u_w$ is randomly initialized and jointly learned during the training process.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This implies that the context vector is independent of the sentence input. It remains the same for every sentence and is learned as a parameter of the neural network. If this is the case, how will $u_w$ accurately provide attention weights for words in a random sentence, given that sentences are so diverse in meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not understand the workings of $u_w$, since it is independent of the sentence input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain?&lt;/p&gt;&#xA;" OwnerUserId="32850" LastEditorUserId="29575" LastEditDate="2017-11-20T12:28:59.523" LastActivityDate="2020-11-27T03:45:11.970" Title="Attention using Context Vector: Hierarchical Attention Networks for Document Classification" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19347" PostTypeId="2" ParentId="19268" CreationDate="2017-05-31T06:22:07.220" Score="0" Body="&lt;p&gt;Start each model with default parameters,use the evaluation metric to record error of each algorithm with default parameter. For majority of classification modeling AUC metrics gives the better results &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/02/7-important-model-evaluation-error-metrics/&quot; rel=&quot;nofollow noreferrer&quot;&gt;read here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not necessary that all features are important, do the feature selection. You can get the important features using random forest/xgboost algorithms, try to remove the least important features and check if evaluation score is improving. .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you think of tuning the parameters and yes each model have different parameters. Most common parameter is alpha(learning rate), to start with you can hand tuned alpha with values in range 0.0001 to 10 &lt;a href=&quot;http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;amp;doc=exercises/ex3/ex3.html.&quot; rel=&quot;nofollow noreferrer&quot;&gt;read here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use more sophisticated methods to tuned parameters such Randomized and Grid search &#xA;&lt;a href=&quot;http://www.pyimagesearch.com/2016/08/15/how-to-tune-hyperparameters-with-python-and-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;read here&lt;/a&gt;.&#xA;Feature wights is way to tell your model that which is most/least important feature. You can get the idea of feature weights from the feature importance I explained at the top. &lt;/p&gt;&#xA;" OwnerUserId="32805" LastEditorUserId="32805" LastEditDate="2017-05-31T08:47:29.397" LastActivityDate="2017-05-31T08:47:29.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19348" PostTypeId="1" CreationDate="2017-05-31T08:38:30.790" Score="-1" ViewCount="76" Body="&lt;p&gt;I have been tasked with comparing the capabilities of different startups offering AI-assisted data preprocessing. Due to legal reasons I cannot offer company data for the benchmarking, not even simulated using the same structures and patterns, which means that I either create a dataset with not-company-related simulated data myself or find one on the internet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify things, by data I mean timeseries. Some simple examples of what I would expect: centering/aligning the timeseries using e.g. cross-correlation, denoising them, finding out correlations, outlier detection, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any (free) datasets designed to be hard to preprocess the data? I could not find any adequate. If not, what is the best way of constructing an own challenging, simulated dataset for this purpose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of the preprocessing step is to clean the timeseries data before it is fed into a neural network in order to be classified and tagged.&lt;/p&gt;&#xA;" OwnerUserId="32854" LastActivityDate="2017-07-30T19:10:30.310" Title="Creating a dataset for benchmarking of timeseries preprocessing capabilities" Tags="|dataset|time-series|data-cleaning|correlation|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19349" PostTypeId="1" CreationDate="2017-05-31T08:40:01.647" Score="-1" ViewCount="18223" Body="&lt;p&gt;I am trying to compare the accuracy of my XGBoost model output to that of a test set (data encoded in binary). My data is stored in a Pandas DataFrame. I am doing this with SKLearn by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;accuracy = accuracy_score(testData,predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I am getting the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Can't handle mix of multilabel-indicator and binary&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting this despite both DataFrames containing the same columns and datatypes. I am not sure what to do next. Any advice would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Upon inspection, it seems that my predictions DataFrame is instead a Numpy array. I believe this may be causing the problem.&lt;/p&gt;&#xA;" OwnerUserId="32856" LastEditorUserId="29169" LastEditDate="2019-06-12T03:25:31.320" LastActivityDate="2019-06-12T03:32:09.823" Title="Determine accuracy of model on train data with Pandas DataFrame" Tags="|python|scikit-learn|xgboost|prediction|accuracy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19350" PostTypeId="2" ParentId="19349" CreationDate="2017-05-31T09:58:11.867" Score="0" Body="&lt;p&gt;Basically, the original &lt;code&gt;accuracy_score()&lt;/code&gt; function takes two arguments. First, is your test data labels (actual labels). Second, it takes your model's predictions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt;&amp;gt;&amp;gt; accuracy = accuracy_score(y_test, predictions)&#xA; &amp;gt;&amp;gt;&amp;gt; type(y_test)&#xA; &amp;gt;&amp;gt;&amp;gt; &amp;lt;type 'numpy.ndarray'&amp;gt;&#xA; &amp;gt;&amp;gt;&amp;gt; type(predicions)&#xA; &amp;gt;&amp;gt;&amp;gt; &amp;lt;type 'list'&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32686" LastEditorUserId="29169" LastEditDate="2019-06-12T03:27:07.340" LastActivityDate="2019-06-12T03:27:07.340" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19351" PostTypeId="2" ParentId="19348" CreationDate="2017-05-31T11:45:51.350" Score="0" Body="&lt;p&gt;You could try &lt;a href=&quot;https://en.wikipedia.org/wiki/Homomorphic_encryption&quot; rel=&quot;nofollow noreferrer&quot;&gt;homomorphic encryption&lt;/a&gt; of your data and then deliver that to the different startups. This is the business model of &lt;a href=&quot;https://medium.com/numerai/encrypted-data-for-efficient-markets-fffbe9743ba8#.63mxt8do2&quot; rel=&quot;nofollow noreferrer&quot;&gt;Numer.ai&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2017-05-31T11:45:51.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19352" PostTypeId="1" AcceptedAnswerId="19356" CreationDate="2017-05-31T13:58:05.197" Score="0" ViewCount="770" Body="&lt;p&gt;I am trying to execute this Tensorflow tutorial &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/examples/learn/wide_n_deep_tutorial.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/examples/learn/wide_n_deep_tutorial.py&lt;/a&gt; but I have the following error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ArgumentError: argument --model_dir: conflicting option string: --model_dir&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that appears in line:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;flags.DEFINE_string(&quot;model_dir&quot;, &quot;&quot;, &quot;Base directory for output models.&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know how I can solve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keira&lt;/p&gt;&#xA;" OwnerUserId="8710" LastActivityDate="2017-05-31T14:49:30.083" Title="Tensorflow ArgumentError: argument --model_dir: conflicting option string: --model_dir" Tags="|machine-learning|python|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19353" PostTypeId="2" ParentId="16306" CreationDate="2017-05-31T14:03:48.327" Score="6" Body="&lt;p&gt;There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only interest I have found using Spark with pandas is when you want to load a local CSV / Excel dataset and then transform it into a spark dataframe. The &quot;createDataFrame&quot; method handles this approach.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; sqlContext.createDataFrame(pandas.DataFrame([[1, 2]])).collect()  &#xA;[Row(0=1, 1=2)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-05-31T14:03:48.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19355" PostTypeId="2" ParentId="19349" CreationDate="2017-05-31T14:29:42.170" Score="0" Body="&lt;p&gt;Are you passing the test data to accuracy score? Accuracy score takes the validation labels and predicted labels as parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that test data is you validation set, result of your test data will be passed to the accuracy score.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/*y_true holds values of testData target variable,&#xA;y_pred holds the prediction values */&#xA;&#xA;accuracy=accuracy_score(y_true.values,y_pred.values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If y_true and y_preds are pandas series type then &quot;.values&quot; to convert them into list, so format of both arguments will be consistent. &lt;/p&gt;&#xA;" OwnerUserId="32805" LastEditorUserId="29169" LastEditDate="2019-06-12T03:32:09.823" LastActivityDate="2019-06-12T03:32:09.823" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19356" PostTypeId="2" ParentId="19352" CreationDate="2017-05-31T14:49:30.083" Score="1" Body="&lt;p&gt;The example you linked is from version 0.12 but the current tensorflow version is 1.2. The example underwent a lot of changes from 0.12 to 1.0. Actually, the very line you are referencing is no longer there in the last version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, it's possible that you are using a version of tensorflow that is no longer compatible with the APIs used by the example. Ensure that the version of tensorflow you are using is the same as the example.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-31T14:49:30.083" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19357" PostTypeId="1" AcceptedAnswerId="19358" CreationDate="2017-05-31T15:07:40.167" Score="14" ViewCount="13809" Body="&lt;p&gt;I have a large multi dimensional dataset that is generated each day.&lt;/p&gt;&#xA;&lt;p&gt;What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could be addressed with neural networks?&lt;/p&gt;&#xA;&lt;p&gt;Any suggestions are appreciated.&lt;/p&gt;&#xA;&lt;p&gt;Additional information: there are no examples, so the method should detect the anomalies itself&lt;/p&gt;&#xA;" OwnerUserId="32860" LastEditorUserId="29169" LastEditDate="2021-03-13T20:08:23.557" LastActivityDate="2021-03-13T20:08:23.557" Title="Detecting anomalies with neural network" Tags="|tensorflow|keras|anomaly-detection|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19358" PostTypeId="2" ParentId="19357" CreationDate="2017-05-31T15:22:18.150" Score="13" Body="&lt;p&gt;From the formulation of the question, I assume that there are no &quot;examples&quot; of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;autoencoders&lt;/strong&gt;&lt;/a&gt;: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a type of autoencoder called &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder#Denoising_autoencoder&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;denoising autoencoder&lt;/strong&gt;&lt;/a&gt;, which is trained with &lt;em&gt;corrupted versions&lt;/em&gt; of the original data as input and with the &lt;em&gt;uncorrupted&lt;/em&gt; original data as output. This delivers a network that can &lt;strong&gt;remove noise&lt;/strong&gt; (i.e. data corruptions) from the inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may train a denoising autoencoder with the daily data. Then use it on new daily data; this way you have the original daily data and an &lt;em&gt;uncorrupted version&lt;/em&gt; of those very same data. You can then compare both to detect &lt;em&gt;significant&lt;/em&gt; differences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key here is which definition of &lt;em&gt;significant difference&lt;/em&gt; you choose. You could compute the euclidean distance and assume that if it surpasses certain arbitrary threshold, you have an anomaly. Another important factor is the kind of &lt;em&gt;corruptions&lt;/em&gt; you introduce; they should be as close as possible to reasonable abnormalities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_adversarial_networks&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Generative Adversarial Networks&lt;/strong&gt;&lt;/a&gt;. The byproduct of the training is a discriminator network that tells apart &lt;em&gt;normal&lt;/em&gt; daily data from abnormal data. &lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-05-31T15:22:18.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19359" PostTypeId="1" AcceptedAnswerId="19360" CreationDate="2017-05-31T15:37:29.743" Score="4" ViewCount="889" Body="&lt;p&gt;While using support vector machines (SVM), we encounter 3 types of lines (for a 2D case). One is the decision boundary and the other 2 are margins:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/agvlL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/agvlL.png&quot; alt=&quot;decision boundary of SVM&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do we use $+1$ and $-1$ as the values after the $=$ sign while writing the equations for the SVM margins? What's so special about $1$ in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if $x$ and $y$ are two features then the decision boundary is: $ax+by+c=0$.&#xA;Why are the two marginal boundaries represented as $ax+by+c=+1$ and $ax+by+c=-1$?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="14906" LastEditDate="2018-04-19T01:01:50.833" LastActivityDate="2022-05-18T21:36:37.537" Title="Why do we use +1 and -1 for marginal decision boundaries in SVM" Tags="|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19360" PostTypeId="2" ParentId="19359" CreationDate="2017-05-31T16:25:08.590" Score="3" Body="&lt;p&gt;It's important for the optimization formulation of the SVM that $y_i=\{-1,1\}$ which is why it makes sense to also output $y=\{-1,1\}$. If we look at the soft-margin linear SVM we want to minimize:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\left[\frac{1}{n}\sum_{i=1}^n\max{(0,1-y_i(w\cdot x_i+b))}\right]+\lambda\| w\| ^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $y_i$ is either +1 or -1 which flips the hyperplane in the soft-margin definition of the problem.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-31T16:25:08.590" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19362" PostTypeId="1" CreationDate="2017-05-31T16:55:43.837" Score="2" ViewCount="5673" Body="&lt;p&gt;I'm creating a neural architecture using the functional API as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x2 = layer1(x1, name='layer1')&#xA;x3 = layer2(x2, name='layer2')&#xA;m1 = Model(x1, x3)&#xA;x5 = layer3(x4, name='layer3')&#xA;x6 = TimeDistributed(m1)(x5)&#xA;m2 = Model(x5, x4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've trained m2 (with m1 getting implicitly trained). I can access the output of the layers of m2 (say layer3) by creating an intermediate model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_name = 'layer3'&#xA;intermediate_layer_model = Model(input=model.input, output=model.get_layer(layer_name).output)&#xA;intermediate_output = intermediate_layer_model.predict(input)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since m1 is a nested model, I'm not able to access the layers of m1 through m2. How do I obtain the output x3 (layer2's output) when you use the trained m2 to predict on a new data point.&lt;/p&gt;&#xA;" OwnerUserId="32850" LastEditorUserId="32850" LastEditDate="2017-05-31T17:06:03.837" LastActivityDate="2020-02-16T19:33:16.720" Title="How to Obtain Output of Intermediate Model in Keras" Tags="|machine-learning|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19363" PostTypeId="2" ParentId="10932" CreationDate="2017-05-31T16:57:42.337" Score="10" Body="&lt;p&gt;Here is my concise thought process in understanding the two different networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which is definitely challenging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Per position or state, there will be N moves possible, and on each move there will be its own depth D in a full search tree. It is theoretically or mathematically possible to walk through all paths and find an optimal solution(s). However, we don't want to do a full search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we got two separate questions for developing an approximation approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q1. How can we skip or disregard some moves out of N per position? (i.e., breath reduction)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2. How can we stop at an intermediate depth in a search tree rather than walking through until the end of game, without failing to find an optimal solution? (i.e., depth reduction)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The policy network is mainly designed for filtering out useless moves out of N, yet without failing to find an optimal solution. Here this network initially relies on human expert moves, i.e., SL, and improved by RL later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value network is mainly designed for finding the winning probability without a full search. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two networks have a common goal of finding an optimal solution, However, in each strategic choice of move, each network plays a different role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just hope this helps. I know it'd be still at a high level.&lt;/p&gt;&#xA;" OwnerUserId="32872" LastActivityDate="2017-05-31T16:57:42.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19364" PostTypeId="2" ParentId="19348" CreationDate="2017-05-31T17:16:51.207" Score="0" Body="&lt;p&gt;Some useful free real world time series data for testing and benchmarking are: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/sunspot.month.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The sunspot time series&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datamarket.com/data/list/?q=provider:tsdl&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hyndman's data library&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I don't know the technical details, but you can download all sorts of financial times series (trading and stock indexes) from Yahoo and Google. &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2017-05-31T17:16:51.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19365" PostTypeId="1" AcceptedAnswerId="19368" CreationDate="2017-05-31T17:22:55.253" Score="2" ViewCount="8182" Body="&lt;p&gt;I have a very simple feed forward neural network with keras that should learn a sinus. Why is the predictive power so bad and what is generally the best way to pinpoint issues with a network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the code below, I have one input neuron, 10 in the hidden layer, and one output. I would expect the network to perform much more accurately.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.layers import Dense, Activation&#xA;from keras.models import Sequential&#xA;&#xA;x = np.arange(100)&#xA;y = np.sin(x)&#xA;&#xA;model = Sequential([&#xA;    Dense(10, input_shape=(1,)),&#xA;    Activation('sigmoid'),&#xA;    Dense(1),&#xA;    Activation('sigmoid')&#xA;])&#xA;&#xA;model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['accuracy'])&#xA;model.fit(x, y, epochs=10, batch_size=1)&#xA;&#xA;scores = model.evaluate(x, y, verbose=0)&#xA;print(&quot;Baseline Error: %.2f%%&quot; % (100-scores[1]*100))&#xA;&#xA;print(model.predict(np.array([.5])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  1/100 [..............................] - ETA: 0s - loss: 1.2016 - acc: 0.0000e+00&#xA; 79/100 [======================&amp;gt;.......] - ETA: 0s - loss: 0.4665 - acc: 0.0127    &#xA;100/100 [==============================] - 0s - loss: 0.5044 - acc: 0.0100     &#xA;Baseline Error: 99.00%&#xA;[[ 0.35267803]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32860" LastEditorUserId="29169" LastEditDate="2019-06-08T03:13:17.513" LastActivityDate="2019-06-08T03:13:17.513" Title="Predict sinus with keras feed forward neural network" Tags="|tensorflow|keras|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="19366" PostTypeId="2" ParentId="18903" CreationDate="2017-05-31T17:54:40.287" Score="30" Body="&lt;p&gt;LightGBM is a great implementation that is similar to XGBoost but varies in a few specific ways, especially in how it creates the trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It offers some different parameters but most of them are very similar to their XGBoost counterparts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the same parameters, you almost always get a very close score. In most cases, the training will be 2-10 times faster though.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Why don't more people use it then?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBoost has been around longer and is already installed on many machines. LightGBM is rather new and didn't have a Python wrapper at first. The current version is easier to install and use so no obstacles here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of the more advanced users on Kaggle and similar sites already use LightGBM and for each new competition, it gets more and more coverage. Still, the starter scripts are often based around XGBoost as people just reuse their old code and adjust a few parameters. I'm sure this will increase once there are a few more tutorials and guides on how to use it (most of the non-ScikitLearn guides currently focus on XGBoost or neural networks).&lt;/p&gt;&#xA;" OwnerUserId="32873" LastEditorUserId="32873" LastEditDate="2018-02-01T12:11:34.500" LastActivityDate="2018-02-01T12:11:34.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19367" PostTypeId="1" CreationDate="2017-05-31T18:07:49.097" Score="1" ViewCount="84" Body="&lt;p&gt;Can someone please explain this one?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For output units a good trick is to obtain the output non-linearity&#xA;  and the loss by considering the associated negative log-likelihood and&#xA;  choosing an appropriate (conditional) output probability model,&#xA;  usually in the exponential family. For example, one can typically take&#xA;  squared error and linear outputs to correspond to a Gaussian output&#xA;  model, cross-entropy and sigmoids to correspond to a binomial output&#xA;  model, and − log output[target class] with softmax outputs to&#xA;  correspond to multinomial output variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="32361" LastActivityDate="2017-08-29T21:05:51.667" Title="Loss given Activation Function and Probability Model" Tags="|machine-learning|deep-learning|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19368" PostTypeId="2" ParentId="19365" CreationDate="2017-05-31T18:41:25.207" Score="4" Body="&lt;p&gt;Accuracy is a metric meant for classification problems, look at the mean squared error instead. Your network is too small for a highly fluctuating function that you want to learn, if you divide your x by a smaller amount it would be easier to learn. Second of all, adding another layer and having an identity activation at the end will help quite a bit. Also taking batches bigger than 1 will make the gradient more stable. With a 1000 epochs I get to 0.00167 as mean squared error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.arange(200).reshape(-1,1) / 50&#xA;y = np.sin(x)&#xA;&#xA;model = Sequential([&#xA;Dense(40, input_shape=(1,)),&#xA;Activation('sigmoid'),&#xA;Dense(12),&#xA;Activation('sigmoid'),&#xA;Dense(1)&#xA;    ])&#xA;&#xA;model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])&#xA;&#xA;for i in range(40):&#xA;    model.fit(x, y, nb_epoch=25, batch_size=8, verbose=0)&#xA;    predictions = model.predict(x)&#xA;    print(np.mean(np.square(predictions - y)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The biggest issue is that the signal in your original small dataset is very difficult to learn, you can see this when you plot it, it will just collapse to the mean.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-31T18:41:25.207" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19370" PostTypeId="1" AcceptedAnswerId="22924" CreationDate="2017-05-31T18:51:16.227" Score="10" ViewCount="10689" Body="&lt;p&gt;From what I can see most object detection NNs (&lt;code&gt;Fast(er) R-CNN&lt;/code&gt;, &lt;code&gt;YOLO&lt;/code&gt; etc) are trained on data &lt;em&gt;including&lt;/em&gt; &lt;code&gt;bounding boxes&lt;/code&gt; indicating where in the picture the objects are localised. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there algos that simply take the &lt;code&gt;full picture + label&lt;/code&gt; annotations, and then on top of determining whether an image contain certain object(s) also indirectly&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;learn to understand the appropriate bounding box(es) for objects?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="32875" LastEditorUserId="29169" LastEditDate="2021-02-16T00:46:52.183" LastActivityDate="2021-02-16T00:46:52.183" Title="Train object detection without annotated data/bounding boxes" Tags="|neural-network|training|convolutional-neural-network|object-recognition|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19371" PostTypeId="2" ParentId="19367" CreationDate="2017-05-31T19:01:45.310" Score="1" Body="&lt;p&gt;At the output we can have an activation that takes a weighted linear sum from the previous layer and transforms it into a new range. Here are a few activations you could use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x = x$, the identity activation which is what is usually used at the end for regression models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(x) = \frac{1}{1+e^{-x}}$ which maps to $(0, 1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$exp(x) = e^x$ which maps to $(0,\infty)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These activations are differentiable which means backpropagation will keep working. Now that we can manipulate our output to turn into ranges that we want, we can use them to use them for probability distributions. In the $\sigma$ case we can use them directly as the probability at event $Y$ as opposed to not $Y$, so $P(Y|X)$. We can also use multiple output nodes to parameterize a probability distribution. For example let's try to approximate a Gaussian distribution conditioned on our input features $x$. Instead of regressing directly on $y$ we can regress $\mu$ and $\sigma$ that define the Gaussian distribution. For $\mu$ we can use the identity activation because $\mu$ is not constrained, $\sigma$ needs to be strictly positive so we could use $e^x$ for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in our training data, we don't have $\mu$ and $\sigma$, we only have a $y$, a sample from this probability distribution. What we do know is that given a specific $\mu$ and $\sigma$ what the likelihood is of that $y$. A value of 100 is extremely unlikely from a normal distribution with $\mu = 20$ and $\sigma=20$, but a value of $20$ has a much higher density. The density of a Gaussian is $f(y|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y-\mu)^2}{2\sigma^2}}$. This means we can check the likelihood of a $y$ given our current estimate of $\mu$ and $\sigma$ given $x$ and try and make the likelihood higher. Or minimize the negative log-likelihood, which has less numerical issues as opposed to normal likelihood.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-05-31T19:01:45.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19372" PostTypeId="2" ParentId="19220" CreationDate="2017-05-31T19:29:18.247" Score="35" Body="&lt;p&gt;Unlike some of the other answers, I would highly advice against always training on GPUs without any second thought. This is driven by the usage of deep learning methods on images and texts, where the data is very rich (e.g. a lot of pixels = a lot of variables) and the model similarly has many millions of parameters. For other domains, this might not be the case.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, that is definitely very small by modern standards. Unless you have a GPU suited perfectly for training (e.g. NVIDIA 1080 or NVIDIA Titan), I wouldn't be surprised to find that your CPU was faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the complexity of your neural network also depends on your number of input features, not just the number of units in your hidden layer. If your hidden layer has 100 units and each observation in your dataset has 4 input features, then your network is tiny (~400 parameters). If each observation instead has 1M input features as in some medical/biotech contexts, then your network is pretty big in terms of number of parameters. For the remainder of my answer I'm assuming you have quite few input features pr. observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One good example I've found of comparing CPU vs. GPU performance was when I trained a poker bot using reinforcement learning. For reinforcement learning you often don't want that many layers in your neural network and we found that we only needed a few layers with few parameters. Moreover, the number of input features was quite low. Initially I trained on a GPU (NVIDIA Titan), but it was taking a long time as reinforcement learning requires a lot of iterations. Luckily, I found that training on my CPU instead made my training go 10x as fast! This is just to say that CPU's can sometimes be better for training.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's important to note that while on a GPU you will always want to fill up the entire GPU memory by increasing your batch size, that is not the case on the CPU. On the CPU an increase in batch size will increase the time pr. batch. Therefore, if it's important for you to have a very large batch size (e.g. due to a very noisy signal), it can be beneficial to use a GPU. I haven't experienced this in practice though and normally small batch sizes are preferred.&lt;/p&gt;&#xA;" OwnerUserId="3044" LastEditorUserId="3044" LastEditDate="2017-06-07T00:32:16.053" LastActivityDate="2017-06-07T00:32:16.053" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19373" PostTypeId="2" ParentId="9036" CreationDate="2017-05-31T19:51:22.433" Score="1" Body="&lt;p&gt;This is not an answer but rather a comment. I would need 50 reputation to comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think in its nature, a recommendation problem is a greedy algorithm to a search/optimization problem. However, the process is biased and local. The users see and react to what they can see, and the algorithm will only reinforce it. Dithering is a way to break up the bias and make the search more global.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have been doing is a bit different than dithering, but similar idea. Instead of ranking, I converted the matching score to log linear probability, and &quot;sort&quot; the item/person by probability. An example is, P(a)=0.2, P(b)=0.8, then b has 80% of chance on top, while a has 20% of chance on top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like to hear how others hear about it too.&lt;/p&gt;&#xA;" OwnerUserId="32877" LastActivityDate="2017-05-31T19:51:22.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19374" PostTypeId="1" CreationDate="2017-05-31T20:36:29.973" Score="1" ViewCount="1703" Body="&lt;p&gt;{ Figured this is the most germane place for this question, but definitely let me know if not! }&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Two Asks:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How would I uniformly standardize the size and scale for each of the bivariate (i.e., scatter plots) &quot;subplots&quot; when using the PairPlot functionality in Seaborn (if the data is already standardized)? By default it creates the plots which are sized the same, but scaled differently.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once the scale is uniform across all plots, I want all of the axes to be formatted such that each plot looks like a coordinate plane (i.e.,4 quadrants; both axes spines centered).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here's some added context on my specific use case:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have z-scores for the World Series winning teams for a set of variables (ERA, R, Batting Avg., etc.)&lt;/li&gt;&#xA;&lt;li&gt;I want to plot all pairwise comparisons for this selected set of variables, and because all the data are already standardized, the axes should range roughly from [-3.5, 3.5].&lt;/li&gt;&#xA;&lt;li&gt;I'm less interested in regressions, and more interested in &quot;archetypes&quot; (e.g. 1995 Atlanta Braves had the best pitching in baseball but weren't exactly stellar hitters -- they still won the World Series! How many other WS winning teams would be in Quadrant III with them?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here's code I've used to make one such plot (not using Seaborn), but I want to take advantage of the power of doing as many pairwise comparison's as possible:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = zscores_wswinners[&quot;teamBA&quot;]&#xA;y = zscores_wswinners[&quot;ERA&quot;]&#xA;&#xA;fig = plt.figure()&#xA;ax = fig.add_subplot(111)&#xA;ax.scatter(x, y)&#xA;plot_axes_range = [-3, 3, -3, 3]&#xA;ax.axis(plot_axes_range, 'equal')&#xA;&#xA;ax.spines['left'].set_position('center')&#xA;ax.spines['right'].set_color('none')&#xA;ax.spines['bottom'].set_position('zero')&#xA;ax.spines['top'].set_color('none')&#xA;ax.xaxis.set_ticks_position('bottom')&#xA;ax.yaxis.set_ticks_position('left')&#xA;&#xA;ax.axhline(linewidth=1, color='black')&#xA;ax.axvline(linewidth=1, color='black')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(This seems pretty tedious, and PairPlot gets me 90%+ of the way to what I desire!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, thanks, in advance!&lt;/p&gt;&#xA;" OwnerUserId="32881" LastEditorUserId="8878" LastEditDate="2017-06-01T09:38:16.293" LastActivityDate="2017-06-01T09:38:16.293" Title="SeaBorn: Custom axes scaling &amp; spines using PairPlot?" Tags="|python|visualization|plotting|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19377" PostTypeId="1" AcceptedAnswerId="20263" CreationDate="2017-06-01T06:19:09.410" Score="6" ViewCount="1780" Body="&lt;p&gt;What is the current state-of-the art within document layout analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. detecting columns, separating images from text, distinguishing between page numbering and text and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for papers and algorithms on the topic.&lt;/p&gt;&#xA;" OwnerUserId="32901" LastActivityDate="2019-11-19T12:51:19.557" Title="Document Layout Analysis - state-of-the-art?" Tags="|ocr|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19378" PostTypeId="1" CreationDate="2017-06-01T07:08:37.510" Score="2" ViewCount="5545" Body="&lt;p&gt;You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/371fb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/371fb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one pattern of assigning weights to neural network. See the another way of assigning the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Wj9RZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Wj9RZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please tell which one of the following is the correct way of assigning the weights to the neural network?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30006" LastEditDate="2017-06-01T07:29:02.980" LastActivityDate="2018-04-27T20:27:13.447" Title="How the vector of weights assigned to a neural network?" Tags="|neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19379" PostTypeId="2" ParentId="19378" CreationDate="2017-06-01T07:57:30.967" Score="3" Body="&lt;p&gt;It looks like when you say &quot;way of assigning the weights&quot;, that you mean &quot;what order are weights counted in, so that I know which weights connect between which neurons&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no formal &quot;correct&quot; way of doing this for all neural networks. However, in practice for feed-forward networks like your diagram, you would choose to use a matrix, not a vector, to represent the weights connecting layers. That is how pretty much all standard libraries will represent weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A matrix uses two indices (call them $i,j$ in this case) to identify a single scalar value. If we call the weight matrix $W$, then an individual weight is $W_{ij}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To determine how the weights connect between neurons, then you index the input layer neuron with $i$ and the output layer neuron with $j$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In math notation, looking at your diagram, call each input value $x_i$ and each hidden value $h_j$, then the formula for calculating a single $h_j$ would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_j = f(b_j + \sum_{i=1}^{N} W_{ij}x_{i})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $f()$ is a transfer function (such as sigmoid), $N$ is the number of input features, and $b_j$ is the bias for hidden layer neuron $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will also often see this written using matrix notation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{h} = f(W\mathbf{x} + \mathbf{b})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . . this is not only clear and simple notation, but using matrix maths like this to describe a neural network is what allows us to use high performance libraries on GPUs such as TensorFlow.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-01T08:02:52.887" LastActivityDate="2017-06-01T08:02:52.887" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19380" PostTypeId="1" CreationDate="2017-06-01T09:10:50.823" Score="6" ViewCount="2606" Body="&lt;p&gt;I was just wondering what the best approach is for training a neural network (or any other machine learning algorithm) where the order of the inputs does not matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;f(x1,x2,x3,x4) = f(x2,x1,x3,x4) = f(x2,x4,x1,x3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current approach is to just randomize the order of every training sample (my network has 44 inputs). It kind of works but it causes the result of the validation loss to jump around every epoch a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe something related to embedding?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other questions that refer to this but with a variable number of inputs and typically in regards to a RNN. I'm talking about the simple case of a fixed number of inputs where order doesn't matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="32905" LastActivityDate="2017-07-01T13:52:43.243" Title="Unordered Input" Tags="|machine-learning|neural-network|keras|embeddings|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19381" PostTypeId="1" CreationDate="2017-06-01T09:28:16.383" Score="2" ViewCount="3524" Body="&lt;p&gt;I am trying to use the XGBoost model to perform a multi-class classification over 40 classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgb_params = list(colsample_bytree= 0.7,&#xA;                  subsample = 0.7,&#xA;                  eta = 0.05,&#xA;                  objective= 'multi:softmax',&#xA;                  max_depth= 5,&#xA;                  min_child_weight= 1,&#xA;                  eval_metric= &quot;mlogloss&quot;, num_class = categoryclassnos,&#xA;                  nthread=4)&#xA;&#xA;fit.xgb = xgb.train(params = xgb_params,&#xA;                    data = dtrain,&#xA;                    nrounds = 500,&#xA;                    watchlist = list(train = dtrain, test=dtest),&#xA;                    print_every_n = 50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I am getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Check failed: (info.labels.size()) != (0) label set cannot be empty&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have reproduced the dataset and the R script &lt;a href=&quot;https://github.com/abhishekkant/healthcarequeryclassifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help/ pointers are deeply appreciated.&lt;/p&gt;&#xA;" OwnerUserId="32904" LastEditorUserId="29575" LastEditDate="2018-04-27T14:09:58.443" LastActivityDate="2018-06-26T15:07:11.030" Title="XGBoost (R Interface) throwing &quot;label set cannot be empty&quot; error" Tags="|r|xgboost|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19384" PostTypeId="2" ParentId="14368" CreationDate="2017-06-01T10:19:36.793" Score="2" Body="&lt;p&gt;Here is my list with collection of benchmarks: &lt;a href=&quot;https://github.com/mrgloom/Deep-Benchmark-List&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/mrgloom/Deep-Benchmark-List&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dividiti/ck-caffe&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dividiti/ck-caffe&lt;/a&gt; looks more promissing, but complicated.&lt;/p&gt;&#xA;" OwnerUserId="17122" LastActivityDate="2017-06-01T10:19:36.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19385" PostTypeId="1" AcceptedAnswerId="19387" CreationDate="2017-06-01T10:44:17.530" Score="2" ViewCount="7857" Body="&lt;p&gt;I am using loss='binary_crossentropy' here is my code:&lt;a href=&quot;https://i.stack.imgur.com/GN3an.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GN3an.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to increase number of training image and Epoch ,but that did not help me.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_img = Input(shape=(28, 28, 1))&#xA;&#xA;x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(input_img)&#xA;x = MaxPooling2D((2, 2), border_mode='same')(x)&#xA;x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)&#xA;x = MaxPooling2D((2, 2), border_mode='same')(x)&#xA;x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)&#xA;&#xA;encoded = MaxPooling2D((2, 2), border_mode='same')(x)&#xA;&#xA;x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(encoded)&#xA;x = UpSampling2D((2, 2))(x)&#xA;x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)&#xA;x = UpSampling2D((2, 2))(x)&#xA;x = Convolution2D(16, 3, 3, activation='relu', border_mode='valid')(x) &#xA;x = UpSampling2D((2, 2))(x)&#xA;&#xA;decoded = Convolution2D(1, 3, 3, activation='sigmoid', border_mode='same')(x)&#xA;&#xA;autoencoder = Model(input_img, decoded)    &#xA;autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')    &#xA;autoencoder.fit(x_train, x_train, nb_epoch=10, batch_size=500,&#xA;               shuffle=True, validation_data=(x_test, x_test), verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32908" LastEditorUserId="836" LastEditDate="2017-06-01T12:02:33.723" LastActivityDate="2017-06-01T12:17:31.213" Title="Why my loss is negative while training SAE?" Tags="|unsupervised-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19386" PostTypeId="2" ParentId="19380" CreationDate="2017-06-01T11:04:36.040" Score="3" Body="&lt;p&gt;The suggestion of ncasas is a good one but not very clean. This ordering makes a lot of sense when it's 1-dimensional, but when you introduce more features the ordering will become more and more arbitrary. This is a problem I have come accross multiple times. This paper (&lt;a href=&quot;https://arxiv.org/pdf/1612.04530.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1612.04530.pdf&lt;/a&gt;) tries to tackle permutation equivariance, which is not exactly the same as your problem, where you would want permutation invariance. With the equivariance the output from the layer still gets shuffled if you shuffle your inputs, but the values will be the same due to the weight sharing. You could extend this idea fairly easily by using a pooling operation (max pooling or mean pooling) on the output from this permutation layer. This means you get one number output out of your layer. Repeating this with &lt;strong&gt;n&lt;/strong&gt; different weight matrices you can get &lt;strong&gt;n&lt;/strong&gt; outputs, which can then be fed into a normal feed forward network. Caution: This is something I just thought of based on your question and their paper, I never tried it but I don't see why it wouldn't work. In the case of sports teams you would use two lists of inputs, one for each team. Based on what you want to predict you could even fit these permutation layers into a siamese network, introducing even more weight sharing!&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-06-01T11:10:01.403" LastActivityDate="2017-06-01T11:10:01.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19387" PostTypeId="2" ParentId="19385" CreationDate="2017-06-01T11:58:45.720" Score="3" Body="&lt;p&gt;Use a linear output and mean squared error loss, assuming you are predicting normalised pixel intensity values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-entropy over sigmoid output layer activations can do odd things when the values are not strictly in $\{0,1\}$, depending on implementation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-01T12:17:31.213" LastActivityDate="2017-06-01T12:17:31.213" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19388" PostTypeId="2" ParentId="14864" CreationDate="2017-06-01T12:26:58.193" Score="0" Body="&lt;p&gt;I think you could do this fairly automatically if you're open to using Python. A library called &lt;a href=&quot;https://github.com/ClimbsRocks/auto_ml&quot; rel=&quot;nofollow noreferrer&quot;&gt;auto_ml&lt;/a&gt;* has a feature called &lt;a href=&quot;http://auto-ml.readthedocs.io/en/latest/categorical_ensembling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical ensembling&lt;/a&gt;, where you can explicitly say &quot;I want a model built for each level of this feature&quot;. If you made a feature that was country-product type and used that as your category, the rest should be pretty easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Disclosure: I've made minor contributions to auto_ml. It is FOSS under the MIT license.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-01T12:26:58.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19390" PostTypeId="1" CreationDate="2017-06-01T13:02:23.683" Score="8" ViewCount="4503" Body="&lt;p&gt;Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email message from a thread of emails.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example consider the below message.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hi,&#xA;&#xA;Can you offer me a better discount.&#xA;&#xA;Thanks,&#xA;Mr.X&#xA;Customer Relations.&#xA;&#xA;---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------&#xA;&#xA;Hello Mr.X,&#xA;&#xA;Does the below work out. Do let us know your thoughts.&#xA;&#xA;Thanks,&#xA;Mr.Y&#xA;&#xA;Sales Manager.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the reason why we want to split the emails is because we want to do sentiment analysis. When we fail to split the email then the results will be wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I searched around and found this &lt;a href=&quot;http://www.cs.cmu.edu/~wcohen/postscript/email-2004.pdf&quot; rel=&quot;noreferrer&quot;&gt;very comprehensive research paper&lt;/a&gt;. Also found an implementation by mail gun called &lt;a href=&quot;https://github.com/mailgun/talon&quot; rel=&quot;noreferrer&quot;&gt;mail gun talon&lt;/a&gt;. But unfortunately it does not work well for certain kind of patters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example when the second message in the email thread breaks like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------- Forwarded message ---------- &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead of the above&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is many people who are trying to do such stuff would have definitely faced such problems, but yet the area remains pretty shady. Is there any pretty solid implementation of the paper or something else that splits email pretty well.&lt;/p&gt;&#xA;" OwnerUserId="30292" LastActivityDate="2018-12-02T12:05:28.730" Title="Extracting individual emails from an email thread" Tags="|classification|scikit-learn|apache-spark|preprocessing|sentiment-analysis|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19391" PostTypeId="1" CreationDate="2017-06-01T13:33:51.617" Score="3" ViewCount="344" Body="&lt;p&gt;Can someone direct me to research papers that have tried to understand word embeddings?&lt;/p&gt;&#xA;" OwnerUserId="32913" LastActivityDate="2023-05-03T08:06:45.880" Title="Understanding Word Embeddings" Tags="|neural-network|word2vec|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19392" PostTypeId="1" CreationDate="2017-06-01T13:52:10.327" Score="8" ViewCount="3695" Body="&lt;p&gt;I have the following data for a little side project. It's from an accelerometer sitting on top of a washer/dryer and I'd like it to tell me when the machine has finished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BRIO1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BRIO1.png&quot; alt=&quot;data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;x is the input data (x/y/z movement as one value), y is the label on/off&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the x values overlap for y=1 and y=0, I was thinking of using x and a rolling 3 minute window as inputs for a SVM:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xyz60=res.xyz.resample(&quot;60S&quot;).max()&#xA;X[&quot;x&quot;]=xyz60&#xA;X[&quot;max3&quot;]=xyz60.rolling(window=3, min_periods=1).max()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VD3T4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VD3T4.png&quot; alt=&quot;data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a good approach for this kind of problem? Are there alternatives that might produce better results?&lt;/p&gt;&#xA;" OwnerUserId="32916" LastEditorUserId="836" LastEditDate="2017-06-01T14:15:01.180" LastActivityDate="2017-06-01T15:53:22.790" Title="Using time series data from a sensor for ML" Tags="|machine-learning|time-series|feature-engineering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19393" PostTypeId="2" ParentId="19391" CreationDate="2017-06-01T14:07:20.747" Score="1" Body="&lt;p&gt;The original papers written by Thomas Mikolov are &lt;a href=&quot;http://web2.cs.columbia.edu/~blei/seminar/2016_discrete_data/readings/MikolovSutskeverChenCorradoDean2013.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/pdf/1405.4053v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lecture notes&lt;/a&gt; given by Richard Socher is simple to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1402.3722.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec explained&lt;/a&gt; gives an intuition how the negative sampling in word embedding works. Not the whole algorithm though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, there are implementations in various languages and libraries explaining how it works in their docs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python - &lt;a href=&quot;https://rare-technologies.com/making-sense-of-word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt;, &lt;a href=&quot;https://www.tensorflow.org/versions/r0.11/tutorials/word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;java - &lt;a href=&quot;https://deeplearning4j.org/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;deeplearning4j&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-06-01T14:07:20.747" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19394" PostTypeId="2" ParentId="19392" CreationDate="2017-06-01T15:53:22.790" Score="7" Body="&lt;p&gt;You have time series data which is used to measure the acceleration. You which to identify when the machine is in its nominal state (OFF) and anomalous state (ON). This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Preparing you data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;All of the methods will rely on the feature extraction method you select. Assuming we continue to use the 3 sample time window as you suggested. In this algorithm you will calculate a statistic for this nominal state $y = 0$. I would suggest the mean as I assume you are already doing, take the average of the three sample resultant accelerations. You will then be left with a large number of values in a training set $S$ defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$S = \{s_0, s_1, ..., s_n \}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $s$ is the mean of the tree samples in a window. $s$ is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s_i = \frac{1}{3} \sum_{k=i-2}^{i} x_k$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x$ is your sample observations and $i\geq2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then collect more data if it is possible with the machine active such that $y = 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you can choose if you want to train your algorithm on a one-class dataset (pure anomlay detection). A biased dataset (anomaly detection) or a well-balanced dataset. The balance of the dataset is the ratio between the two classes in your dataset. A perfect dataset for a 2-class classifier would be 1:1. 50% of the data belonging to each class. You seem to have a biased dataset, assuming you don't want to waste a lot of electricity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do note that there is nothing stopping you from keeping the neighboring samples split as an instance in your dataset. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_i$ $x_{i-1}$ $x_{i-2}$ | $y_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would make a 3-dimensional input space for a specific output which is defined for the currently taken sample.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;A Biased Dataset&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Easy Solution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way that i would suggest. Assume you are using a single statistic to define what is happening throughout the 3 sample window. From the collected data get the maximum $s$ of your nominal points ($y=0$) and the minimum $s$ of your anomalous points ($y=1$). Then take the halfway mark between these two and use that as your threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a new test sample $\hat{s}$ is larger than the threshold then assign $y=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can extend this by calculating the mean $s$ for all of your nominal samples $y=0$. Then calculate the mean for your anomalous samples $y=1$. If a new sample falls closer to the mean of the anomalous samples then classify it as $y=1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But I want to get fancy!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of other techniques you can use to do this exact task.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;k-Nearest Neighbors&lt;/li&gt;&#xA;&lt;li&gt;Neural Networks&lt;/li&gt;&#xA;&lt;li&gt;Linear Regression&lt;/li&gt;&#xA;&lt;li&gt;SVM&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Simply put, almost every machine learning algorithm is well suited for this purpose. It just depends on how much data is available to you and it's distribution. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;I really want to use SVM&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If this is the case keep the three samples completely separated. Your training matrix will have 3 columns as discussed above. And then you will have your outputs $y$. Using SVM in python is very easy: &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/svm.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;&#xA;X = [[0, 0, 0], [1, 1, 1], ..., [1, 0, 1]] &#xA;y = [0, 1, ..., 1]&#xA;clf = svm.SVC()&#xA;clf.fit(X, y)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This trains your model. Then you will want to predict the outcome for a new sample. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.predict([[2., 2., 1]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-06-01T15:53:22.790" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="19395" PostTypeId="1" CreationDate="2017-06-01T16:04:32.907" Score="-1" ViewCount="5654" Body="&lt;p&gt;I am trying to plot a contour map with the following data I have in 3 columns&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;one for the date and time (T)&lt;/li&gt;&#xA;&lt;li&gt;one for the temperature - T1&lt;/li&gt;&#xA;&lt;li&gt;last one with the temperature - T2&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am intending to make a contour map which shows the temperature T2 at (T,T1).&lt;br&gt;&#xA;From what I have read and understood, I have to create a meshgrid with T and T1 and create a contour map with the meshgrid created and T2.&#xA;But most of the tutorials I have seen show it when T2 is a result of a function.&lt;br&gt;&#xA;The temperatures T1 and T2 are observed at time T and they are not a result of any function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT&#xA;For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.arange(-5, 5, 0.1)&#xA;y = np.arange(-5, 5, 0.1)&#xA;xx, yy = meshgrid(x, y, sparse=True)&#xA;z = np.sin(xx**2 + yy**2) / (xx**2 + yy**2)&#xA;h = plt.contourf(x,y,z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But in this case, z is another column of values and not a result of the function.&lt;/p&gt;&#xA;" OwnerUserId="32918" LastEditorUserId="-1" LastEditDate="2017-06-02T07:57:39.727" LastActivityDate="2017-08-01T11:46:35.970" Title="How to plot a contour map in python using matplotlib?" Tags="|python|visualization|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19396" PostTypeId="1" AcceptedAnswerId="19404" CreationDate="2017-06-01T16:14:15.830" Score="0" ViewCount="378" Body="&lt;p&gt;I'm using Sklearn to build a classifier in which my client wants a predicted probability for each row of data.  The default of let's say Random Forest is if &gt; 50%, then classifies as TRUE, but using the predict_proba function I'm able to get the probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I'm given has 10k rows, which all are labeled TRUE or FALSE.  If it's my job to provide a predicted probability by row, should I still use a 70/30 Train/Test split.  In which I create my best model on the Train 70% using a 10-fold CV.  But then when I need to output for all 10k rows, I would actually be make a prediction on the training data and the test data, which doesn't seem correct to predict on training data.  Any thoughts on how to approach this?&lt;/p&gt;&#xA;" OwnerUserId="32754" LastActivityDate="2017-06-01T20:47:49.510" Title="Propensity Modeling, still use Test/Train Split?" Tags="|scikit-learn|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19397" PostTypeId="1" CreationDate="2017-06-01T16:50:37.010" Score="0" ViewCount="98" Body="&lt;p&gt;I have an application of a straightforward MLP, for which the cost function is a function of both the network output, in addition to another value calculated from the network weights (actually the partial derivative of the output with respect to one of the inputs). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any flexible NN packages where I can code this? The only ones I've seen (Keras, nnet in R) require that the cost function is just parameterised by the network output and the target value.&lt;/p&gt;&#xA;" OwnerUserId="32921" LastActivityDate="2017-06-01T16:57:26.553" Title="Any Neural Network implementations that allow for a cost function of more than just the network output?" Tags="|neural-network|deep-learning|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19398" PostTypeId="2" ParentId="19397" CreationDate="2017-06-01T16:57:26.553" Score="1" Body="&lt;p&gt;You should be able to do that in Keras, actually you should be able to do that in almost any flexible package. For example regularization does exactly that, it adds a penalty term to your loss function that punishes high weights.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-01T16:57:26.553" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19399" PostTypeId="2" ParentId="19365" CreationDate="2017-06-01T16:59:28.610" Score="13" Body="&lt;p&gt;Keep in mind that the &lt;strong&gt;Accuracy&lt;/strong&gt; measure is measuring whether the values are &lt;em&gt;Exactly The Same&lt;/em&gt;. I.e. it is a classification measure, whereas approximating a sin curve is much better suited for measurement as a regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In evaluating the network's performance, what is the network actually doing? Let's take the network and perform a little visual analysis on its performance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;preds = model.predict(x)&#xA;plt.plot(x, y, 'b', x, preds, 'r--')&#xA;plt.ylabel('Y / Predicted Value')&#xA;plt.xlabel('X Value')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e0acv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e0acv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hmm. The model seems to be minimizing error by simply getting closer and closer to guessing 0 for every value, rather than approximating the function. There are several hypotheses here to explain this. One is that the network is not complex enough to model the function. In order to test this, let's simplify the function- that is, let's bring the range down to one sine cycle:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.arange(0, math.pi*2, 0.1)&#xA;y = np.sin(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And try to train the network again:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8HA7X.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8HA7X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not wonderful, but a better fit, certainly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about with &lt;strong&gt;100 epochs&lt;/strong&gt; instead of 10?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0Q6eS.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0Q6eS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about with &lt;strong&gt;1000 epochs&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Wk7qL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Wk7qL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is, of course, very interesting. After 1000 epochs, our networks is able to roughly approximate the downward curve from 1:0 ($\pi/2$: $\pi$) of the sine response, but not the initial upward curve 0:1 (0:$\pi/2$) or the region in which the function is negative ($\pi$:$2\pi$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This result begs the question- what will it look like after &lt;strong&gt;10000 epochs&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sxf00.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sxf00.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not significantly better. It looks like we'll have to change the architecture of the network (more layers, more neurons, and/or different activation functions) to improve beyond this point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To inform this architecture change, let's take a look at the sigmoid activation function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pp6kl.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pp6kl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Uh-oh. The value of this sigmoid function can only ever be in the range 0:1, and the range of the sin function is -1:1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To correct this, let's just normalize the sin response between 0 and 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = (np.sin(x)+1)/2 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, the network performs much better than before after 1000 epochs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/X0Nrk.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0Nrk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And 10000:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IxcxR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IxcxR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 100000 epochs, it's roughly perfect:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dmbZb.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dmbZb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even still, this advancement doesn't help much on the larger sin range (after 1000 epochs):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.arange(0, 100, 1)&#xA;y = (np.sin(x)+1)/2 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6smI5.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6smI5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we, however, take the model trained on a single sin curve and further train it on the larger range, we begin to see progress after 1000 epochs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.arange(0, 100, .1)&#xA;y = (np.sin(x)+1)/2 &#xA;&#xA;model_copy = model&#xA;model_copy.fit(x, y, epochs=1000, batch_size=8, verbose=0)&#xA;model_copy_preds = model_copy.predict(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UW8H2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UW8H2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And more so after 10000 epochs:&lt;a href=&quot;https://i.stack.imgur.com/cmQjw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cmQjw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And more so, well into the third repetition after 100000 epochs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zDPgJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zDPgJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, with careful training, our network with a single layer of sigmoid activations appears to be learning to generalize the sin curve. Further investigation could find a limit to that generalization, certainly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reproduction:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.layers import Dense&#xA;from keras.models import Sequential&#xA;import matplotlib.pyplot as plt&#xA;import math&#xA;&#xA;x = np.arange(0, math.pi*2, .1)&#xA;y = (np.sin(x)+1)/2 &#xA;&#xA;model = Sequential([&#xA;    Dense(10, input_shape=(1,)),&#xA;    Activation('sigmoid'),&#xA;    Dense(1)&#xA;])&#xA;&#xA;model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])&#xA;model.fit(x, y, epochs=100000, batch_size=8, verbose=0)&#xA;&#xA;preds = model.predict(x)&#xA;&#xA;plt.plot(x, y, 'b', x, preds, 'r--')&#xA;plt.ylabel('Y / Predicted Value')&#xA;plt.xlabel('X Value')&#xA;plt.show()&#xA;&#xA;x = np.arange(0, 100, .1)&#xA;y = (np.sin(x)+1)/2 &#xA;&#xA;model_copy = model&#xA;model_copy.fit(x, y, epochs=10000, batch_size=8, verbose=0)&#xA;model_copy_preds = model_copy.predict(x)&#xA;&#xA;plt.plot(x, y, 'b', x, model_copy_preds, 'r--')&#xA;plt.ylabel('Y / Predicted Value')&#xA;plt.xlabel('X Value')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-06-05T14:11:15.347" LastActivityDate="2017-06-05T14:11:15.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19400" PostTypeId="1" CreationDate="2017-06-01T17:04:43.293" Score="2" ViewCount="997" Body="&lt;p&gt;I am looking to build a model for specific news and blog articles which merge fashion with patterns in biology. I have 35 websites that I read daily (it is exhausting). I am wondering how to approach creating such a model so that I can send articles to it daily and it can predict whether or not they deserve reading (ie relevant vs irrelevant). For example, in a perfect world I send it 230 articles of which 12 are returned as relevant based on previous training and testing articles. Assuming all 12 are relevant I save them to the training dir and recompile  to strengthen the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After reading &lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/deep-learning-keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning with Keras&lt;/a&gt; I was hoping Chapters 5 and 6 on Word Embeddings and RNNs(simple, LSTM, GRU) would point me to how to develop such a model. I was also reading how to implement a &lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN for text classification&lt;/a&gt; but cannot seem to construct a basic conceptual framework for how to start with a few articles and adding to the training data every day to strengthen the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a term for this type of strategy? Has it been done in some form on Kaggle, Github, etc?&lt;/p&gt;&#xA;" OwnerUserId="32628" LastEditorUserId="32628" LastEditDate="2017-06-01T17:29:55.467" LastActivityDate="2017-06-01T18:35:45.520" Title="Deep Learning Binary Text Classification" Tags="|neural-network|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19401" PostTypeId="2" ParentId="19400" CreationDate="2017-06-01T18:35:45.520" Score="1" Body="&lt;p&gt;In a broader sense, what you're referring to here is known as &lt;strong&gt;text mining&lt;/strong&gt;. This is where information is extracted from text according to certain criteria in order to form specific patterns and meanings, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How you would go about implementing this depends on the criteria you have in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, let's suppose you have a blog article in text format (let's call it filepath.txt for this example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You decide that you wish to sort words into specific categories. For instance, you could categorise the keywords &lt;strong&gt;hotel&lt;/strong&gt;, &lt;strong&gt;flights&lt;/strong&gt;, &lt;strong&gt;countries&lt;/strong&gt;, as &lt;strong&gt;travel&lt;/strong&gt;, i.e. each incidence of the three words would be replaced with data. Let's assume data is a relevant category of interest. Any article that would fall under a travel category would be included in your articles of interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Python, you can then scan your file and use filedata.replace to replace your keywords with categories:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Read file&#xA;with open('filepath.txt', 'r') as file :&#xA;filedata = file.read()&#xA;&#xA;# Replace keyword&#xA;filedata = filedata.replace('How ', ' ')&#xA;filedata = filedata.replace('Why ', ' ')&#xA;filedata = filedata.replace('of ', ' ')&#xA;filedata = filedata.replace('to ', ' ')&#xA;filedata = filedata.replace('you ', ' ')&#xA;filedata = filedata.replace('all ', ' ')&#xA;filedata = filedata.replace('and ', ' ')&#xA;filedata = filedata.replace('be ', ' ')&#xA;filedata = filedata.replace(' a ', ' ')&#xA;filedata = filedata.replace(' for ', ' ')&#xA;filedata = filedata.replace(' in ', ' ')&#xA;filedata = filedata.replace(' is ', ' ')&#xA;filedata = filedata.replace(' the ', ' ')&#xA;filedata = filedata.replace(' about ', ' ')&#xA;filedata = filedata.replace(' an ', ' ')&#xA;filedata = filedata.replace('Data', ' data ')&#xA;filedata = filedata.replace('Python', ' data ')&#xA;filedata = filedata.replace('R', ' data ')&#xA;filedata = filedata.replace('machine', ' data ')&#xA;filedata = filedata.replace('Linux', ' data ')&#xA;filedata = filedata.replace('technology', ' data ')&#xA;filedata = filedata.replace('flights', 'travel')&#xA;filedata = filedata.replace('countries', 'travel')&#xA;filedata = filedata.replace('hotel', 'travel')&#xA;filedata = filedata.replace('analytics', 'data')&#xA;filedata=  filedata.replace('CNN', 'news')&#xA;filedata=  filedata.replace('weather', 'news')&#xA;filedata=  filedata.replace('Trump', 'news')&#xA;filedata=  filedata.replace('market', 'business')&#xA;filedata=  filedata.replace('entrepreneur', 'business')&#xA;filedata=  filedata.replace('financial', 'business')&#xA;&#xA;# Write to file&#xA;with open('filepath2.txt', 'w') as file:&#xA;file.write(filedata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could then use a library such as &lt;strong&gt;stringr&lt;/strong&gt; in R to see how frequently the category (which would have the keywords replaced with the category) appears:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(stringr)&#xA;WordList &amp;lt;- str_split(readLines(&quot;filepath2.txt&quot;), pattern = &quot; &quot;)&#xA;searchqueries&amp;lt;-sort(table(WordList),decreasing=TRUE)[1:100]&#xA;searchqueries&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;e.g. if the category appears X number of times or more, then the article is deemed relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just one example of how you could use text mining. Again, much of it hinges on your specific criteria and how you would go about determining relevancy. But that's one way you could approach it.&lt;/p&gt;&#xA;" OwnerUserId="32926" LastActivityDate="2017-06-01T18:35:45.520" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19402" PostTypeId="1" CreationDate="2017-06-01T19:24:30.423" Score="0" ViewCount="3696" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HgkIx.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HgkIx.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;I'm trying to read a csv file into a dataframe in Python but when examining the file I noticed they are many misaligned rows . Does anyone know how I can shift these rows so my Dataframe's size is consistent. &#xA;data= pd.read_csv('MyData.csv')&lt;/p&gt;&#xA;" OwnerUserId="32931" LastEditorUserId="32931" LastEditDate="2017-06-01T19:49:18.300" LastActivityDate="2018-05-08T22:48:33.933" Title="Fixing misaligned rows using Python" Tags="|python|pandas|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19404" PostTypeId="2" ParentId="19396" CreationDate="2017-06-01T20:47:49.510" Score="0" Body="&lt;p&gt;You are right; you shouldn't predict on the same data you have used for training your model. If your goal is to only output probabilities and you don't mind achieving this by having not a single classifier but a number of them, you can potentially use nested cross-validation to achieve that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the outer cross-validation steps, you break your data in $N_{out}$ folds, where one fold is kept for testing the model and the remaining $N_{out} - 1$ folds are used to train. You then feed this training dataset (which is similar, in nature, to that 70% you have mentioned in OP) to the inner cross-validation step with $N_{in}$ folds. Here, you will compare different models and choose the best performing one. This chosen model is then passed to the outer fold (see above) and can be used to obtain probabilities for the 1 fold that was left out. You then repeat the same procedure for other folds in the outer fold. At the end of this nested cross-validation process, you will have probabilities for all your rows (but they will have come from $N_{out}$ different classifiers, each corresponding to one of the outer folds).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the ultimate purpose of nested cross-validation is not to do this but it will give you what you want as a by-product.&lt;/p&gt;&#xA;" OwnerUserId="21433" LastActivityDate="2017-06-01T20:47:49.510" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19405" PostTypeId="1" AcceptedAnswerId="19426" CreationDate="2017-06-01T20:47:59.267" Score="1" ViewCount="5937" Body="&lt;p&gt;I have a python numpy array of size (800,800,4) which is my hyperspectral camera data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I try standard GMM methods from scikit-learn I get an error saying that the expected dimension of the data should be less than or equal to 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Current method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;gmm=GaussianMixture(covariance_type=&quot;full&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;gmm=gmm.fit(data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any python packages or functions available to process higher dimensional data using gmm?&lt;/p&gt;&#xA;" OwnerUserId="32936" LastActivityDate="2017-09-14T15:15:38.390" Title="Implementation of Gaussian Mixture Model for clustering when dealing with multidimensional hyperspectral data in python" Tags="|python|clustering|image-classification|image-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19406" PostTypeId="1" CreationDate="2017-06-01T20:49:36.550" Score="3" ViewCount="8377" Body="&lt;p&gt;I have a data frame with following structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.columns&#xA;Index(['first_post_date', 'followers_count', 'friends_count',&#xA;       'last_post_date','min_retweet', 'retweet_count', 'screen_name',&#xA;       'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],&#xA;        dtype='object')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Inside the &lt;em&gt;tweets&lt;/em&gt; series, each cell is another data frame containing all the &lt;em&gt;tweets&lt;/em&gt; of an user.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.tweets[0].columns&#xA;Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to convert this data frame to a multi-index frame, essentially by breaking the cell containing tweets. One index will be the &lt;em&gt;uid&lt;/em&gt;, and another will be the &lt;em&gt;id&lt;/em&gt; inside &lt;em&gt;tweet&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.dropbox.com/s/9uaki7ks6gv01ie/sample.json?dl=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;link to sample data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31180" LastActivityDate="2017-06-02T00:24:48.140" Title="convert single index pandas data frame to multi-index" Tags="|python|pandas|indexing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19407" PostTypeId="1" AcceptedAnswerId="19420" CreationDate="2017-06-01T22:55:49.190" Score="1" ViewCount="3980" Body="&lt;h1&gt;Problem&lt;/h1&gt;&#xA;&lt;p&gt;In Keras for Python, I have to use multiple lines of code for a &lt;a href=&quot;https://gist.github.com/stewartpark/187895beb89f0a1b3a54&quot; rel=&quot;nofollow noreferrer&quot;&gt;simple XOR neural network&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers.core import Dense, Dropout, Activation&#xA;from keras.optimizers import SGD&#xA;import numpy as np &#xA;&#xA;X = np.array([[0,0],[0,1],[1,0],[1,1]])&#xA;y = np.array([[0],[1],[1],[0]])&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(8, input_dim=2))&#xA;model.add(Activation('tanh'))&#xA;model.add(Dense(1))&#xA;model.add(Activation('sigmoid'))&#xA;&#xA;sgd = SGD(lr=0.1); model.compile(loss='binary_crossentropy', optimizer=sgd)&#xA;&#xA;model.fit(X, y, show_accuracy=True, batch_size=1, nb_epoch=1000)&#xA;print(model.predict_proba(X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Although the network above is quite small, the current implementation may become frustrating with deeper networks.&lt;/p&gt;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&lt;p&gt;Is there a &lt;strong&gt;built-in&lt;/strong&gt; shortcut in Keras that quickly adds multiple layers?&#xA;For example:&lt;br /&gt;&#xA;&lt;code&gt;[2,3,4,5]&lt;/code&gt; would be a network with 2 input neurons, 3 neurons in the first hidden layer, 4 neurons in the second hidden layer, and 5 output neurons. I use the term &lt;strong&gt;built-in&lt;/strong&gt; because otherwise I can loop through the example array, and add a layer for each element.&lt;/p&gt;&#xA;" OwnerUserId="32941" LastEditorUserId="98307" LastEditDate="2020-08-11T16:28:34.883" LastActivityDate="2020-08-11T16:28:34.883" Title="Keras: Built-In Multi-Layer Shortcut" Tags="|machine-learning|python|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19410" PostTypeId="2" ParentId="19406" CreationDate="2017-06-02T00:24:48.140" Score="2" Body="&lt;p&gt;One way to pull the embedded dataframe up into the main dataframe and build a multi index is like:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def expand_tweets(tweets_df):&#xA;    tweets = []&#xA;    for uid, sub_df in tweets_df.set_index('uid').tweets.iteritems():&#xA;        sub_df['uid'] = uid&#xA;        tweets.append(sub_df)&#xA;    return pd.concat(tweets).merge(&#xA;        tweets_df.drop('tweets', axis=1).reset_index(),&#xA;        how='outer', on='uid').set_index(['uid', 'id'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;How:&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Pull all of the tweet dataframes out of main dataframe using &lt;code&gt;uid&lt;/code&gt; as an index, and &lt;code&gt;concat()&lt;/code&gt; them together with their &lt;code&gt;uid&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then merge the main dataframe into the concatenated tweets dataframe.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Set the desired index.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import pandas as pd&#xA;with open('test.json') as f:&#xA;    df = pd.DataFrame(json.load(f))&#xA;df['tweets'] = df.tweets.apply(lambda x: pd.DataFrame(x))&#xA;&#xA;print(expand_tweets(df).text.head())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;uid         id                &#xA;1153859336  655060275025047552    Article on my new Haunted Stevenage book Paran...&#xA;            653912439940120576    Big thank you to @bobfmuk for interviewing me ...&#xA;            643709869908996096    Another interesting non-toadstool tweet today,...&#xA;            547107275681579008    @sisax67 Thanks, Simon. All the best to you &amp;amp;a...&#xA;            546693940024733696    Paul Adams @SkySportsDarts The Wanderer from W...&#xA;Name: text, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-06-02T00:24:48.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19411" PostTypeId="2" ParentId="17514" CreationDate="2017-06-02T03:40:05.763" Score="2" Body="&lt;p&gt;Actually, there is a formula which can easily convert character based PPL and word based PPL. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$PPL = 2^{(BPC*Nc/Nw)}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $BPC$ is character based $PPL$, $Nc$ and $Nw$ are the number of characters and words in a test set, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula is not completely fair, but it at least offers a way to comparing them. The following are some reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Hwang K, Sung W. Character-Level Language Modeling with Hierarchical Recurrent Neural Networks[J]. 2016.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Graves A. Generating Sequences With Recurrent Neural Networks[J]. Computer Science, 2013.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky.Subword language modeling with neural networks. Technical report, Un-published Manuscript, 2012.&lt;/p&gt;&#xA;" OwnerUserId="32946" LastEditorUserId="22012" LastEditDate="2017-08-06T18:51:39.147" LastActivityDate="2017-08-06T18:51:39.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19412" PostTypeId="2" ParentId="19390" CreationDate="2017-06-02T03:42:57.037" Score="1" Body="&lt;p&gt;First you can read about the &lt;a href=&quot;http://www.faqs.org/rfcs/rfc822.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;format of ARPA internet text messages&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You're going to have to one up on Regex parsing if you are going to want to extract any meaningful info from the emails. I'd suggest the Oreilly book on regex, or reading over &lt;a href=&quot;http://www.regular-expressions.info/&quot; rel=&quot;nofollow noreferrer&quot;&gt;regular expressions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also a tool to &lt;a href=&quot;https://www.emailparser.com/e/basic-examples/capture-text-incoming-emails-write-text-file&quot; rel=&quot;nofollow noreferrer&quot;&gt;capture text from incoming emails and write to a text file&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32686" LastEditorUserId="29575" LastEditDate="2018-03-02T14:35:18.797" LastActivityDate="2018-03-02T14:35:18.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19419" PostTypeId="2" ParentId="19402" CreationDate="2017-06-02T07:53:31.503" Score="1" Body="&lt;p&gt;What is your seperator in the csv- a comma? It would help if you could open the raw csv file in a text editor and post the result here. You could then check if this seperator can appear in the input data. You then have two options: Change the seperator (if possible) or create a small python script/ manually replace all cases in which this can happen. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-06-02T07:53:31.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19420" PostTypeId="2" ParentId="19407" CreationDate="2017-06-02T08:35:06.803" Score="1" Body="&lt;p&gt;There is no shortcut syntax that goes as far as accepting &lt;code&gt;[2,3,4,5]&lt;/code&gt; as a param and create a model. However, it would be very simple for you to create this as a Python function yourself, provided in your case you already have made the decisions about activation functions, the types of layer etc. The need to make those decisions and have them available in the Keras API means that Keras itself does not offer such a short form build function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can make the model building slightly less verbose by using a list of layers when you instantiate the model, instead of adding them afterwards:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential([&#xA;  Dense(8, input_dim=2),&#xA;  Activation('tanh'),&#xA;  Dense(1),&#xA;  Activation('sigmoid')&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if you want to try variations of a model, where the only things you change are the number and size of hidden layers, then you can write a short Python function to encapsulate that requirement. Here's an example (I'm sure you already know how to do this, just included for completeness):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def build_model(hidden_layer_sizes):&#xA;  model = Sequential()&#xA;&#xA;  model.add(Dense(hidden_layer_sizes[0], input_dim=2))&#xA;  model.add(Activation('tanh'))&#xA;&#xA;  for layer_size in hidden_layer_sizes[1:]:&#xA;    model.add(Dense(layer_size))&#xA;    model.add(Activation('tanh'))&#xA;&#xA;  model.add(Dense(1))&#xA;  model.add(Activation('sigmoid'))&#xA;&#xA;  return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you take this approach, you may find you end up parametrising other choices such as input size (as you try some feature engineering), hidden layer activation function, whether to use a Dropout layer etc. Again, it is this need to define all the other choices in a typical network that lead to Keras' design. The best you can do is compress down the choices for your case, with a custom function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to address this comment in your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Although the network above is quite small, the current implementation may become frustrating with deeper networks.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In practice, I have not found Keras' design difficult to use for deep networks. I typically write a separate build function, and parametrise a few things, such as input dimensions. However, I typically &lt;em&gt;don't&lt;/em&gt; loop through a list of different layer sizes as the main param of the build function. Instead, I find the more verbose approach just fine, even when trying variations of network size/shape (I guess that might change if I wanted to grid search including layer sizes). I think that is because I find the function names very easy to read - even with a screen full of &lt;code&gt;.add()&lt;/code&gt; functions, I can see quite quickly what the NN structure is.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-02T09:17:21.330" LastActivityDate="2017-06-02T09:17:21.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19421" PostTypeId="2" ParentId="19305" CreationDate="2017-06-02T09:57:56.580" Score="0" Body="&lt;p&gt;you can use named entity recognition from &lt;em&gt;python nltk.&lt;/em&gt; for identifying whether you question has - person or organisation or place. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk import word_tokenize, pos_tag, ne_chunk&#xA;&#xA;Q1 = &quot;where is Texas ?&quot;&#xA;A1 = &quot;It is in USA.&quot;&#xA;Q2 = &quot;where is California?&quot;&#xA;A2 = &quot;It is in USA&quot;&#xA;Q3 = &quot;where is NASA?&quot;&#xA;A3 = &quot;It is in USA.&quot;&#xA;Q4 = &quot;who is Queen Elizabeth II?&quot;&#xA;A4 = &quot;Queen of England&quot;&#xA;Q5 = &quot;who is Donald Trump?&quot;&#xA;A5 = &quot;President of USA.&quot;&#xA;&#xA;print(ne_chunk(pos_tag(word_tokenize(Q1))))&#xA;print(ne_chunk(pos_tag(word_tokenize(Q2))))&#xA;print(ne_chunk(pos_tag(word_tokenize(Q3))))&#xA;print(ne_chunk(pos_tag(word_tokenize(Q4))))&#xA;print(ne_chunk(pos_tag(word_tokenize(Q5))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;code output below&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(S where/WRB is/VBZ (&lt;strong&gt;PERSON&lt;/strong&gt; Texas/NNP) ?/.) &lt;br/&gt;&#xA;(S where/WRB is/VBZ (&lt;strong&gt;GPE&lt;/strong&gt; California/NNP) ?/.) &lt;br/&gt;&#xA;(S where/WRB is/VBZ (&lt;strong&gt;ORGANIZATION&lt;/strong&gt; NASA/NNP) ?/.) &lt;br/&gt;&#xA;(S who/WP is/VBZ (&lt;strong&gt;PERSON&lt;/strong&gt; Queen/NNP Elizabeth/NNP II/NNP) ?/.) &lt;br/&gt;&#xA;(S who/WP is/VBZ (&lt;strong&gt;PERSON&lt;/strong&gt; Donald/NNP Trump/NNP) ?/.) &lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can group output based on person or organisation or location. Also note Texas is shown as person here, so all answers will not be correct. But should be a good start.&lt;/p&gt;&#xA;" OwnerUserId="32848" LastActivityDate="2017-06-02T09:57:56.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19422" PostTypeId="2" ParentId="19395" CreationDate="2017-06-02T10:24:59.177" Score="1" Body="&lt;p&gt;Firstly, you need to specify meshgrid as &lt;strong&gt;np.meshgrid&lt;/strong&gt;, since it is part of the numpy library. You might find the following link helpful for more information (&lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;numpy.meshgrid&lt;/a&gt;). Using the below code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.mlab as mlab&#xA;import matplotlib.pyplot as plt&#xA;x = np.arange(-5, 5, 0.1)&#xA;y = np.arange(-5, 5, 0.1)&#xA;xx, yy = np.meshgrid(x, y, sparse=True)&#xA;z = np.sin(xx**2 + yy**2) / (xx**2 + yy**2)&#xA;h = plt.contourf(x,y,z)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you simply use plt.contourf, the plot will be loaded but then the plt.show() command must be used to actually display the graph:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dSiwO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dSiwO.png&quot; alt=&quot;Contour Map&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32926" LastActivityDate="2017-06-02T10:24:59.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19425" PostTypeId="1" AcceptedAnswerId="19428" CreationDate="2017-06-02T13:18:13.650" Score="1" ViewCount="351" Body="&lt;p&gt;I have an understanding problem. I am a beginner in machine learning and have also a little experience in modelling NNs but not for time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I cannot imagine how to use Neiural Networks for time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if I want to train a Multilayer NN what is the input?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read several papers about such ANN to predict time series. But they did not explain how actually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I imagine it as follows: Is every input node a specific time stamp?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to predict the value for time t+1, and I use 15 input neurons, so I use the values at each time stamp from t-15 upt to t?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I train such a NN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a little confused.&lt;/p&gt;&#xA;" OwnerUserId="32963" LastActivityDate="2017-06-02T13:38:12.583" Title="Neural Networks for time series" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19426" PostTypeId="2" ParentId="19405" CreationDate="2017-06-02T13:20:18.327" Score="3" Body="&lt;p&gt;If I understand your type of data correctly, what you have is essentially an image with $800\times800 =640000$ pixels and each pixel has a vector of 4 values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this stands, I suppose you could then transform your data to a $640000\times4$ matrix, so as to conform with scikit-learn's data representation schema of inputting matrices of shape ($\#samples\times\#features$) and then you could use the GMM class implemented by the package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you could transform the data back if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: On a second thought, I don't know if the spatial distribution of pixels plays a role on the generation of the GMM components. Maybe try it out and let us know?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, after you have trained your model you can ask for it to predict the mixture model from which each pixel in your dataset has been generated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is if you run again:&#xA;$clusters=gmm.predict(newdata)$ you will get an array with values $0-2$, denoting the cluster that each pixel belongs to. You can, then map these values in the image at hand, applying a mask of different color for each pixel in order to see if your model correctly identifies the three classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of what I mean:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.misc import imread, imshow&#xA;from sklearn.datasets import load_sample_image&#xA;&#xA;X = load_sample_image('china.jpg')&#xA;old_shape = X.shape&#xA;X = X.reshape(-1,3)&#xA;gmm = mixture.GMM(covariance_type='full', n_components=5)&#xA;gmm.fit(X)&#xA;clusters = gmm.predict(X)&#xA;clusters = clusters.reshape(old_shape[0], old_shape[1])&#xA;imshow(clusters)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-06-02T14:40:39.247" LastActivityDate="2017-06-02T14:40:39.247" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19427" PostTypeId="1" CreationDate="2017-06-02T13:21:58.993" Score="0" ViewCount="182" Body="&lt;p&gt;I have a database of text files and I would like to classify each section(i.e references, abstract, etc...) in the text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Abstract Indexes: 100-300&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;References Indexes: 9000-10000&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is there a package that is doing something like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a recommended approach for doing so?&lt;/p&gt;&#xA;" OwnerUserId="32961" LastActivityDate="2018-05-02T17:30:08.997" Title="A Text Sections Classifier" Tags="|nlp|training|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19428" PostTypeId="2" ParentId="19425" CreationDate="2017-06-02T13:38:12.583" Score="2" Body="&lt;p&gt;The type of neural networks you are looking for to predicting timeseries are called &lt;strong&gt;recurrent&lt;/strong&gt; networks; the previous neuron states in the network affect the new neuron states. Examples of such recurrent networks are &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Gated_recurrent_unit&quot; rel=&quot;nofollow noreferrer&quot;&gt;GRU&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonlinear_autoregressive_exogenous_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;NARX&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have the data from t&lt;sub&gt;0&lt;/sub&gt; to t&lt;sub&gt;10&lt;/sub&gt;, and you want to predict t&lt;sub&gt;11&lt;/sub&gt;, then you need to input the values from t&lt;sub&gt;0&lt;/sub&gt;- t&lt;sub&gt;10&lt;/sub&gt; &lt;em&gt;one by one&lt;/em&gt; into the network. You have to train the network so that the output of t&lt;sub&gt;n&lt;/sub&gt; is t&lt;sub&gt;n+1&lt;/sub&gt;. So after you have inputted t&lt;sub&gt;10&lt;/sub&gt;, the output will be the values predicted for t&lt;sub&gt;11&lt;/sub&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Real life example: I want a network to keep on decreasing it's output until it has reached 1, and then make it output 0 to start over again:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;in: 0.0, out: 0.2&#xA;in: 0.2, out: 0.4&#xA;in: 0.4, out: 0.6&#xA;in: 0.6, out: 0.8&#xA;in: 0.8, out: 1.0&#xA;in: 1.0, out: 0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;And that is what your training data should look like&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is still to complicated, I recommend you to play around with this front-end neural network library for the browser: &lt;a href=&quot;https://github.com/wagenaartje/neataptic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neataptic&lt;/a&gt;. It is very easy to fiddle around with. &lt;a href=&quot;https://jsfiddle.net/9t2787k5/4/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Example&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-06-02T13:38:12.583" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19429" PostTypeId="1" CreationDate="2017-06-02T14:59:17.030" Score="0" ViewCount="77" Body="&lt;p&gt;I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word &quot;hello&quot; -- not &quot;Hello&quot;, &quot;hello&quot;, &quot;hello.&quot;, and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)?&lt;/strong&gt; I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.&lt;/p&gt;&#xA;" OwnerUserId="13087" LastActivityDate="2017-06-04T10:57:56.120" Title="Searching documents at scale: how to maintain cleaned documents?" Tags="|text-mining|data-cleaning|recommender-system|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19430" PostTypeId="1" AcceptedAnswerId="19434" CreationDate="2017-06-02T17:52:18.740" Score="5" ViewCount="7541" Body="&lt;p&gt;I am looking to employ &lt;a href=&quot;https://deeplearning4j.org/word2vec&quot; rel=&quot;noreferrer&quot;&gt;Word2Vec&lt;/a&gt; to cluster documents and classify them by topic. However, it seems I need a significant data set &lt;a href=&quot;https://datascience.stackexchange.com/questions/9722/applying-word2vec-on-small-text-files&quot;&gt; [1&lt;/a&gt;,&lt;a href=&quot;https://datascience.stackexchange.com/questions/6553/how-much-training-data-does-word2vec-need&quot;&gt;2]&lt;/a&gt; to achieve such a task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set of thousands (not millions) of documents where my total word count is somewhere in the tens of thousands or maybe low hundreds of thousands.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there another ML methodology that allows me to attain my goal? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see &lt;a href=&quot;http://www.markhneedham.com/blog/2015/02/15/pythonscikit-learn-calculating-tfidf-on-how-i-met-your-mother-transcripts/&quot; rel=&quot;noreferrer&quot;&gt;how&lt;/a&gt; to use TF/IDF to generate words and phrases from a corpus  but output as expected is a list of common words and phrases along a flat dimension: &lt;a href=&quot;https://i.stack.imgur.com/xNjJx.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xNjJx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is something more along the lines of a high level cluster of vectors in space:&lt;a href=&quot;https://i.stack.imgur.com/F0vrB.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F0vrB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;http://benschmidt.org/word2vec_map/&quot; rel=&quot;noreferrer&quot;&gt;[source]&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32628" LastEditorUserId="32628" LastEditDate="2017-06-02T17:58:11.670" LastActivityDate="2019-05-20T08:28:30.913" Title="Word2Vec Alternative for Smaller Datasets" Tags="|machine-learning|neural-network|word2vec|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19432" PostTypeId="1" AcceptedAnswerId="19518" CreationDate="2017-06-02T18:15:24.010" Score="6" ViewCount="3137" Body="&lt;p&gt;When training an XGboost model some of the information printed regards &quot;extra nodes&quot;. I can't find an explanation of these anywhere in the documentation. What exactly are extra nodes?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 54 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 58 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 60 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5&#xA;[14:13:11] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2017-07-11T12:00:46.987" Title="What are the &quot;extra nodes&quot; in XGboost?" Tags="|machine-learning|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19433" PostTypeId="2" ParentId="19430" CreationDate="2017-06-02T18:20:34.443" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2vec&lt;/a&gt; (aka paragraph2vec, aka sentence embeddings) modifies the word2vec algorithm to unsupervised learning of continuous representations for larger blocks of text, such as sentences, paragraphs or entire documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will going to cluster each documents topics in vector space , learn it's semantic meaning. It will perform good with your given size dataset size.check this too &lt;a href=&quot;https://datascience.stackexchange.com/questions/10216/doc2vec-how-to-label-the-paragraphs-gensim&quot;&gt;Doc2Vec - How to label the paragraphs (gensim)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-06-02T18:20:34.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19434" PostTypeId="2" ParentId="19430" CreationDate="2017-06-02T18:23:55.617" Score="4" Body="&lt;p&gt;You can side step the paucity of training data, and indeed training altogether, by using &lt;a href=&quot;http://ahogrammer.com/2017/01/20/the-list-of-pretrained-word-embeddings/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pre-trained embeddings in numerous languages&lt;/a&gt;. After that you can calculate your document embeddings using one of these simple algorithms, which basically amount to running dimensionality reduction on the matrix of stacked word embeddings for each sentence using PCA/SVD:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://openreview.net/forum?id=SyK00v5xx&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Simple but Tough-to-Beat Baseline for Sentence Embeddings&lt;/a&gt; (&lt;a href=&quot;https://github.com/YingyuLiang/SIF&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1704.05358v1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Representing Sentences as Low-Rank Subspaces&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that word embeddings themselves emerge from similar calculations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/papers/NIPS2014_5477&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Word Embedding as Implicit Matrix Factorization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-06-02T18:23:55.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19435" PostTypeId="1" CreationDate="2017-06-02T19:16:19.723" Score="1" ViewCount="3128" Body="&lt;p&gt;I have some sensor data (actually many different sensors about 90 of them) and I have their readings over some time course. I was wondering on a couple of things:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1: What metrics can I use to assess if the measurements are stable across the time period. Is there something more I can do besides calculating the standard deviation which might give me an indication of the stability of the measurement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2: What is the best way to visualise the data. So, I have about 90 sensors and I can compute something like box-plot for the various sensor measurements to visualise the distributions but my issue is that the sensors have different scale and I was wondering what might be an effective visualisation tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I could have one sensor measuring temperature and one measuring pressure and the reading could be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;temp        pressure&#xA;51          9.3&#xA;49          9.2&#xA;50          9.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the scales are different. How can one visualise these sort of different  data types in one graphic as this could be much more efficient in communicating the information. I wonder if there is a way to use this graphic and the measurement stability metrics to indicate quickly if one or more sensors are misbehaving.&lt;/p&gt;&#xA;" OwnerUserId="8337" LastActivityDate="2018-11-23T07:28:39.140" Title="analysing stability of a measurement over time" Tags="|statistics|visualization|descriptive-statistics|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19436" PostTypeId="1" AcceptedAnswerId="19440" CreationDate="2017-06-02T19:26:33.870" Score="0" ViewCount="510" Body="&lt;p&gt;It is usually better if you have a not so large but balanced dataset and you are performing classification to apply stratification in order to split it in a training and testing datasets which are both balanced as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there is a notion of having the training and testing datasets &lt;strong&gt;represent adequately&lt;/strong&gt; the overall dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you expand this notion to regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the idea is that instead of just shuffling and splitting the dataset, you could group the targets in bins for example [0, 10] [10, 20] etc.&#xA;And then have the training and testing dataset be also an adequate representation of the whole dataset by having targets with all kinds of values. (Otherwise, you could end up leaving out some part of the range)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Makes sense? :)&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-06-02T21:48:59.040" Title="Is Stratification applicable to both Classification and Regression?" Tags="|classification|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19437" PostTypeId="1" CreationDate="2017-06-02T20:28:51.137" Score="1" ViewCount="408" Body="&lt;p&gt;I want to build a feedforward network with keras. &#xA;My dataset consists of youtube comments with labels 1 : spam and 0 : no spam and I use the Bag-of-words model.&#xA;The amount of the spam comments is ca. 5%.&#xA;Here is the model I use right now:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(64, input_dim=len(txt_dict), activation='relu'))&#xA;model.add(Dense(32, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics ['accuracy', precision, recall])&#xA;model.fit(x_train, y_train, epochs=100, batch_size=batch_size, verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now my problem is the model always predict 0, because of the small amount of spam labels.&#xA;I had the idea that a low recall must be punished harder, but how can I implement this? &lt;/p&gt;&#xA;" OwnerUserId="32979" LastActivityDate="2017-06-02T20:28:51.137" Title="Binary text classification problem with small label-dataset using keras" Tags="|machine-learning|python|neural-network|classification|keras|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19438" PostTypeId="1" AcceptedAnswerId="19441" CreationDate="2017-06-02T21:02:49.763" Score="0" ViewCount="580" Body="&lt;p&gt;I would like to know the salary for Data Scientists by countries, but failed to find any resources on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that US gives data scientists an insanely high amoung of money (well, it might not be &quot;insanely high&quot; on the standard of US), but I rarely hear about any other countries, possibly except UK, Canada, and Australia in some cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any research or survey or something that shows the diversity of salary for Data Scientists by countries, something like &lt;a href=&quot;http://swexperts.com/news/software-engineer-salaries-by-country/&quot; rel=&quot;nofollow noreferrer&quot;&gt;for software engineers&lt;/a&gt;? &lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-06-02T22:51:57.153" Title="How is the salary for Data Scientist different among countries?" Tags="|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19439" PostTypeId="1" CreationDate="2017-06-02T21:18:38.477" Score="0" ViewCount="59" Body="&lt;p&gt;As per my understanding, while learning the features, the Convolution part of the CNN works-on and preserves the 'spatial relationship' between the pixels. &lt;br&gt;&lt;br&gt;&#xA;But, I want to identify object(s) in my image whose definition might be all scattered throughout the image.&lt;br&gt;&#xA;By scattered I mean - The identity of object may exist in separate disconnected islands of pixel-sets. Where each of these island in itself is a connected component of pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words- my 'Cat' exists, not as a whole, but is broken up into pieces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would traditional CNN still identify my 'Cat'? &lt;br&gt;&lt;br&gt;&#xA;If yes, upto what level? If my Cat is broken into 4 pieces? 100 pieces?&lt;br&gt;&#xA;If not, what adjustment(s) do I need to make so that I can solve this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="23733" LastEditorUserId="29169" LastEditDate="2021-02-16T00:46:39.750" LastActivityDate="2021-02-16T00:46:39.750" Title="CNNs for identifying scattered object?" Tags="|machine-learning|deep-learning|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19440" PostTypeId="2" ParentId="19436" CreationDate="2017-06-02T21:48:59.040" Score="1" Body="&lt;p&gt;Yes that makes sense. Your training and testing datasets should both be similar distributions to the whole dataset in both classification and regression, and in the regression case, binning the target variable is one way to achieve that. You need to make sure that you choose good bin sizes -- you can completely skew how the distribution looks in a histogram based on the bin sizes. Too small and you only have one or two examples per bin, resulting in a very erratic histogram. Too large and you lose a lot of information about the shape of the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binning is just one way to approximate the &lt;em&gt;density function&lt;/em&gt; of the distribution. Depending on how your data looks, you might be able to fit a Gaussian (or any other distribution) curve to your target variable and use that instead to sample your train/test split.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-06-02T21:48:59.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19441" PostTypeId="2" ParentId="19438" CreationDate="2017-06-02T22:51:57.153" Score="2" Body="&lt;p&gt;The biggest one I know is O'Reilly's survey &lt;a href=&quot;http://www.oreilly.com/data/free/2016-data-science-salary-survey.csp&quot; rel=&quot;nofollow noreferrer&quot;&gt;(here the last available from 2016)&lt;/a&gt;. There are also polls on KDNuggets &lt;a href=&quot;http://www.kdnuggets.com/2014/03/data-scientists-salary-survey-us-canada-australia-lead.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;(older version)&lt;/a&gt; and other websites. But, in general, most of them are not very representative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also go look at the national averages on &lt;a href=&quot;https://www.glassdoor.com/Salaries/india-data-scientist-salary-SRCH_IL.0,5_IN115_KO6,20.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;glassdoor.com (example India)&lt;/a&gt; or other job sites.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-06-02T22:51:57.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19442" PostTypeId="1" CreationDate="2017-06-03T08:29:19.370" Score="1" ViewCount="583" Body="&lt;p&gt;Needed for dating portal. I currently use &lt;a href=&quot;https://cmusatyalab.github.io/openface/&quot; rel=&quot;nofollow noreferrer&quot;&gt; OpenFace&lt;/a&gt; to extract features with DNN. (I don't know whether they have a usable meaning) Then compare them to some model faces' features using euclidian distance and rate according to these values. Result is below satisfactory. I'm thinking about SVM, but I need to distinguish between a few levels (rank 1-5). OneVsAll? Not sure if I need 5 classifiers just to get the measure of similarity. Any ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What have been tried:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Extracted features for a set of model faces (photos chosen by me from google); extracted features from a test set and sort photos according to average (or minimum) euclidian distance to these model features; Visual estimation: many from those which are in top 10's are noticeably not similar (and not attractive) compared to a model set&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Labeled test set (according to my taste) as 1 and 0 classes (70 and 1640 samples) and CV set (14 and 194). Extracted features, trained RBF SVM, performed parameter search.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="32989" LastEditorUserId="32989" LastEditDate="2017-06-07T15:42:56.270" LastActivityDate="2017-06-07T15:42:56.270" Title="Human face attractiveness estimation" Tags="|neural-network|multiclass-classification|object-recognition|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="19443" PostTypeId="1" CreationDate="2017-06-03T09:09:39.733" Score="0" ViewCount="66" Body="&lt;p&gt;I need to find a good clustering for this data using sci-kit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AZnRG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AZnRG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KNN is not appropriate as it creates blobs although these data are linearly separated. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import hypertools as hyp&#xA;hyp.plot(tocluster,'o', n_clusters=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/K0Lkt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K0Lkt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27347" LastActivityDate="2017-06-03T11:37:51.367" Title="Appropriate Clustering Algorithm" Tags="|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19445" PostTypeId="2" ParentId="19443" CreationDate="2017-06-03T11:37:51.367" Score="1" Body="&lt;p&gt;I'd rather not rely on clustering (clearly, DBSCAN would be the first method to try).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I'd look for a &lt;em&gt;projection&lt;/em&gt; that removes the correlation (x - y maybe?), then define a grid to separate the data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-06-03T11:37:51.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19446" PostTypeId="2" ParentId="19439" CreationDate="2017-06-03T14:10:12.580" Score="1" Body="&lt;p&gt;This should not be a problem, the convolutional kernels will have bigger and bigger receptive fields when you get deeper, and either there will be kernels at the end that will combine all the components over the whole input if the receptive field is big enough, or if you use a fully connected layer at the end it will learn that specific combinations of components combine into a specific class. The number of pieces it will be able to recognize depends on the expressiveness of your network, the amount of training samples and on the difficulty of your problem.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-03T14:10:12.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19449" PostTypeId="1" CreationDate="2017-06-03T17:08:58.647" Score="-1" ViewCount="122" Body="&lt;p&gt;I'd like to train a Decision Tree using  the Classification Learner App. I have a range of IP addresses, and a country that the IP address range belongs to. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;1.0.0.0&quot;,&quot;1.0.0.255&quot;,&quot;AU&quot;&#xA;&quot;1.0.1.0&quot;,&quot;1.0.3.255&quot;,&quot;CN&quot;&#xA;&quot;1.4.0.0&quot;,&quot;1.4.0.255&quot;,&quot;AU&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've been thinking the whole day, but I can't figure out how to represent the range in the Matlab. The first step should be to convert the IP address to an integer. Yes, that's easy, but I'll have a range of numbers again. I don't want to replace the range with 255 lines (in the first case) or with even more lines. How to tell Matlab to handle the address range in an appropriate way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of this project is to train a tree using the test data (IP address range), then use the tree to determine the coutry of the given (single) IP address, not a range.&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="33001" LastActivityDate="2017-09-03T20:42:28.993" Title="How to train Matlab on a range of IP addresses?" Tags="|data-mining|decision-trees|matlab|preprocessing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19452" PostTypeId="1" AcceptedAnswerId="19457" CreationDate="2017-06-03T20:12:19.593" Score="10" ViewCount="1946" Body="&lt;p&gt;I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small research in a classroom of 60 students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have mined over hundred thousand sentences from Wikipedia for various English words (Including Barrons'800 words and 1000 most common English words)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Entire data is available at &lt;a href=&quot;https://buildmyvocab.in&quot; rel=&quot;noreferrer&quot;&gt;https://buildmyvocab.in&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to maintain the quality of content, I filtered out sentences which were longer than 160 characters since they might be difficult to understand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a next step, I want to be able to automate the process of sorting this content in the order of ease of understanding. I myself am a non-native English speaker. I want to know what features I can use to separate easy sentences from difficult ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, do you think this is possible?&lt;/p&gt;&#xA;" OwnerUserId="33005" LastEditorUserId="33005" LastEditDate="2017-06-03T21:00:31.850" LastActivityDate="2017-06-03T23:48:08.467" Title="How to determine the complexity of an English sentence?" Tags="|machine-learning|classification|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19455" PostTypeId="1" CreationDate="2017-06-03T22:15:35.817" Score="8" ViewCount="9396" Body="&lt;p&gt;On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either &lt;code&gt;2&lt;/code&gt; or &lt;code&gt;3&lt;/code&gt; for the window size. For example, &lt;a href=&quot;https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks-Part-2/&quot; rel=&quot;noreferrer&quot;&gt;in this tutorial&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h1&gt;Pooling Layers&lt;/h1&gt;&#xA;&lt;p&gt;After some ReLU layers, programmers may choose to apply a pooling layer. It is also referred to as a downsampling layer. In this category, there are also several layer options, with maxpooling being the most popular. This basically takes a filter (&lt;strong&gt;normally of size 2x2&lt;/strong&gt;) and a stride of the same length. It then applies it to the input volume and outputs the maximum number in every subregion that the filter convolves around.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I make the relevant part bold. But why is it usually set to &lt;code&gt;2&lt;/code&gt; (or in some cases &lt;code&gt;3&lt;/code&gt;)? How can I know what size is appropriate? Should I just tweak the parameter via a trial and error like a brute-force way, or is there any tips on deciding the window size?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-06-03T22:48:35.223" Title="How can you decide the window size on a pooling layer?" Tags="|deep-learning|hyperparameter|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19456" PostTypeId="1" CreationDate="2017-06-03T22:19:58.023" Score="0" ViewCount="4233" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Cross-entropy tends to allow errors to change weights even when nodes&#xA;  saturate (which means that their derivatives are asymptotically close&#xA;  to 0.)&#xA;  &lt;a href=&quot;https://www.cse.unsw.edu.au/~billw/cs9444/crossentropy.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why is the above statement true? Figures and examples if possible.&lt;/p&gt;&#xA;" OwnerUserId="33011" LastEditorUserId="33011" LastEditDate="2017-06-04T02:07:00.033" LastActivityDate="2017-06-04T07:33:34.107" Title="A practical reason to use Cross-entropy as a error-function in Neural networks?" Tags="|machine-learning|neural-network|tensorflow|information-theory|" AnswerCount="1" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="19457" PostTypeId="2" ParentId="19452" CreationDate="2017-06-03T22:40:53.040" Score="8" Body="&lt;p&gt;Yes. There are various metrics, such as the fogg index.&#xA;&lt;a href=&quot;https://github.com/chartbeat-labs/textacy&quot; rel=&quot;noreferrer&quot;&gt;Textacy&lt;/a&gt; in python has a nice list and implementations.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; ts.flesch_kincaid_grade_level&#xA;10.853709110179697&#xA;&amp;gt;&amp;gt;&amp;gt; ts.readability_stats&#xA;{'automated_readability_index': 12.801546064781363,&#xA; 'coleman_liau_index': 9.905629258346586,&#xA; 'flesch_kincaid_grade_level': 10.853709110179697,&#xA; 'flesch_readability_ease': 62.51222198133965,&#xA; 'gulpease_index': 55.10492845786963,&#xA; 'gunning_fog_index': 13.69506833036245,&#xA; 'lix': 45.76390294037353,&#xA; 'smog_index': 11.683781121521076,&#xA; 'wiener_sachtextformel': 5.401029023140788}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="555" LastEditorUserId="381" LastEditDate="2017-06-03T23:48:08.467" LastActivityDate="2017-06-03T23:48:08.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19458" PostTypeId="2" ParentId="19455" CreationDate="2017-06-03T22:48:35.223" Score="9" Body="&lt;p&gt;When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and see what works better. In the case of pooling layers it is actually relatively interpretable. Why do we use pooling? To downsample our feature maps. This is done to decrease the resolution of our processed image but also to increase the receptive fields of our following layers. Because it is a form of downsampling we throw away all the information that was not the maximum in case of max-pooling, or we throw away information about the spread with mean-pooling. By increase our pooling filter size, we decrease our resolution even further and lose more information, but we will also need less parameters in our network if we have a fully connected layer at the end and our next convolutional layers will have a bigger receptive field, but with less attention to detail.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-03T22:48:35.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19459" PostTypeId="2" ParentId="19141" CreationDate="2017-06-04T04:36:38.300" Score="1" Body="&lt;p&gt;From what I understood - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Train a classifier with the features as you mentioned and the output probabilities like you have shown - &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;So, for &lt;code&gt;Player A&lt;/code&gt;, you can use all the features (average runs scored overall, average score in past 10 innings etc.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Treat the runs scored as a categorical variable and try to learn a classifier&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Put it through the network with a softmax layer at the end - that will give you &lt;em&gt;Sum of Probability should add up to 1&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My recommendation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Each player will have scored different runs so you will not get enough data points in a particular category (so no. of people who score 83 will be really less for example)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pre-group the ranges in your training data and then do the classifier. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Having a softmax layer at then end won't give you the &quot;linear model&quot; you want, but would definitely solve the problem with a lot more clarity.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2017-06-04T04:36:38.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19460" PostTypeId="2" ParentId="19456" CreationDate="2017-06-04T04:50:04.053" Score="4" Body="&lt;ul&gt;&#xA;&lt;li&gt;During back-propagation training, you want to drive output node values to either 1.0 or 0.0 depending on the target values. &lt;/li&gt;&#xA;&lt;li&gt;If you use MSE, the weight adjustment factor (the gradient) contains a term of (output) * (1 – output). As the computed output gets closer and closer to either 0.0 or 1.0 the value of (output) * (1 – output) gets smaller and smaller. &lt;/li&gt;&#xA;&lt;li&gt;For example, if output = 0.6 then (output) * (1 – output) = 0.24 but if output is 0.95 then (output) * (1 – output) = 0.0475. &lt;/li&gt;&#xA;&lt;li&gt;As the adjustment factor gets smaller and smaller, the change in weights gets smaller and smaller and training can stall out, so to speak.&lt;/li&gt;&#xA;&lt;li&gt;But if you use cross-entropy error, the (output) * (1 – output) term goes away (the math is very cool). So, the weight changes don’t get smaller and smaller and so training isn’t s likely to stall out. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that this argument assumes you’re doing neural network classification, with &lt;em&gt;either&lt;/em&gt; softmax output node activation plus &lt;a href=&quot;http://wiki.fast.ai/index.php/Log_Loss#Multi-class_Classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multiclass logloss&lt;/a&gt;, &lt;em&gt;or&lt;/em&gt; sigmoid output node activation plus &lt;a href=&quot;http://wiki.fast.ai/index.php/Log_Loss#Binary_Classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary  logloss&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href=&quot;https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="30578" LastEditorUserId="836" LastEditDate="2017-06-04T07:33:34.107" LastActivityDate="2017-06-04T07:33:34.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19462" PostTypeId="1" CreationDate="2017-06-04T07:30:58.100" Score="1" ViewCount="579" Body="&lt;p&gt;i am trying to train a deep neural network to figure out that if there is a 1 and 0 present in the first two columns of X that the output is 1 otherwise its 0. but im only getting a 75% accuracy on the model!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;import numpy as np&#xA;import tflearn&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = [[0, 0, 1],&#xA;    [0, 1, 1],&#xA;    [1, 0, 1],&#xA;    [1, 1, 1]]&#xA;&#xA;Y = [[0, 1],&#xA;    [1, 1],&#xA;    [1, 0],&#xA;    [0, 1]]&#xA;&#xA;Xtest = np.array([[1, 1, 1],&#xA;                  [0, 1, 1],&#xA;                  [1, 0, 1],&#xA;                  [0, 1, 1]])&#xA;&#xA;# Build neural network&#xA;net = tflearn.input_data(shape=[None, 3])&#xA;net = tflearn.fully_connected(net, 32, activation='sigmoid')&#xA;net = tflearn.fully_connected(net, 32, activation='sigmoid')&#xA;net = tflearn.fully_connected(net, 2, activation='softmax')&#xA;net = tflearn.regression(net, optimizer='adam')&#xA;&#xA;# Define model&#xA;model = tflearn.DNN(net)&#xA;# Start training (apply gradient descent algorithm)&#xA;model.fit(X, Y, n_epoch=5000, batch_size=16, show_metric=True)&#xA;&#xA;pred = model.predict(Xtest)&#xA;for i in range(4):&#xA;    print(pred[i][0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output should be: [0, 1, 1, 1]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training Step: 4999  | total loss: 0.50493 | time: 0.004s&#xA;| Adam | epoch: 4999 | loss: 0.50493 - acc: 0.7813 -- iter: 4/4&#xA;--&#xA;    0.01631585881114006&#xA;    0.4872587323188782&#xA;    0.9684665203094482&#xA;    0.019177177920937538&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33016" LastActivityDate="2017-06-04T07:43:36.907" Title="Max 75% accuracy! help!" Tags="|machine-learning|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19463" PostTypeId="2" ParentId="19462" CreationDate="2017-06-04T07:43:36.907" Score="1" Body="&lt;p&gt;It looks like you are training this as a multiclass classifier, to represent a binary choice. In which case, your &lt;code&gt;Y&lt;/code&gt; value is wrong:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = [[0, 1],&#xA;    [1, 1],&#xA;    [1, 0],&#xA;    [0, 1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here your second label is not self-consistent, and thus it is impossible to predict using a softmax output layer (where the sum of all outputs must equal 1). The best it can do is &lt;code&gt;[0.5, 0.5]&lt;/code&gt; to match that label and you can see actually it got close to that in your test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want this instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y = [[0, 1],&#xA;    [1, 0],&#xA;    [1, 0],&#xA;    [0, 1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A few asides . . .&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your example inputs all have the same third column (=1). This is redundant data, and you could drop it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your network is more complex than it needs to be for this task. A single hidden layer with only a few neurons in it should be sufficient.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For this specific task you could have chosen a single output neuron using a sigmoid activation (and need only one column in Y).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-06-04T07:43:36.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19465" PostTypeId="1" AcceptedAnswerId="19470" CreationDate="2017-06-04T09:12:08.460" Score="1" ViewCount="12837" Body="&lt;p&gt;I have been trying to implement logistic regression in python. Basically the code works and it gives the accuracy of the predictive model at a level of 91% but for some reason the AUC score is 0.5 which is basically the worst possible score because it means that the model is completely random. Also the classification report returns error: &quot;UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)&quot;. Does anyone know what should I change so it works properly?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; import numpy as np&#xA; import pandas as pd&#xA; from sklearn.cross_validation import train_test_split&#xA; from sklearn.linear_model import LogisticRegression&#xA; from sklearn.metrics import accuracy_score  &#xA; from sklearn.preprocessing import StandardScaler&#xA; from sklearn.metrics import roc_auc_score&#xA; from sklearn.metrics import classification_report&#xA;&#xA; data_file = pd.read_csv('loan.csv', delimiter=',')&#xA;&#xA; # variable preprocessing&#xA;&#xA; data_file['loan_status'] = np.where(data_file['loan_status'].isin(['Fully &#xA; Paid', 'Current']), 1, 0)&#xA; loan_stat=data_file['loan_status']&#xA; loan_stat=loan_stat.astype(np.float64)&#xA;&#xA; m = {&#xA;    'n/a': 0,     &#xA;    '&amp;lt; 1 year': 0,&#xA;    '1 year': 1,&#xA;    '2 years': 2,&#xA;    '3 years': 3,&#xA;    '4 years': 4,&#xA;    '5 years': 5,&#xA;    '6 years': 6,&#xA;    '7 years': 7,&#xA;    '8 years': 8,&#xA;    '9 years': 9,&#xA;    '10+ years': 10&#xA; }&#xA; emp_length=data_file.emp_length.map(m)&#xA; emp_length.astype(np.float64)&#xA;&#xA; annual_inc=data_file['annual_inc']&#xA; delinq_2yrs=data_file['delinq_2yrs']&#xA; dti=data_file['dti']&#xA; loan_amnt=data_file['loan_amnt']&#xA; installment=data_file['installment']&#xA; int_rate=data_file['int_rate']&#xA; total_acc=data_file['total_acc']&#xA; open_acc=data_file['open_acc']&#xA; pub_rec=data_file['pub_rec']&#xA; acc_now_delinq=data_file['acc_now_delinq']&#xA;&#xA; #variables combined into one dataframe&#xA;&#xA; X=pd.DataFrame()&#xA;&#xA; X['annua_inc']=annual_inc&#xA; X['delinq_2yrs']=delinq_2yrs&#xA; X['dti']=dti&#xA; X['emp_length']=emp_length&#xA; X['loan_amnt']=loan_amnt&#xA; X['installment']=installment&#xA; X['int_rate']=int_rate&#xA; X['total_acc']=total_acc&#xA; X['open_acc']=open_acc&#xA; X['pub_rec']=pub_rec&#xA; X['acc_now_delinq']=acc_now_delinq&#xA; X['loan_stat']=loan_stat&#xA;&#xA; X=X.dropna(axis=0)&#xA; y=X['loan_stat']&#xA; X=X.drop(['loan_stat'], axis=1)&#xA;&#xA; scaler=StandardScaler()&#xA; X=scaler.fit_transform(X)&#xA;&#xA; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, &#xA; random_state=42)&#xA;&#xA; model=LogisticRegression(penalty='l2', C=1)&#xA; model.fit(X_train, y_train)&#xA; score=accuracy_score(y_test, model.predict(X_test))&#xA; roc=roc_auc_score(y_test, model.predict(X_test))&#xA; cr=classification_report(y_test, model.predict(X_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the link to the data: &lt;a href=&quot;https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33018" LastEditorUserId="14372" LastEditDate="2017-06-04T15:07:33.320" LastActivityDate="2017-06-04T15:07:33.320" Title="AUC and classification report in Logistic regression in python" Tags="|python|scikit-learn|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19466" PostTypeId="1" AcceptedAnswerId="19474" CreationDate="2017-06-04T09:58:20.137" Score="2" ViewCount="866" Body="&lt;p&gt;I have been doing a problem in which I have to predict probabilities for each of the labels in a multi-label (four to be precise) classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of a solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Id, North,          East,            West,           South&#xA;1,  0.71663940211,  0.037567315693,  0.03525987339,  0.0021068944991&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The training data is of the form where each y(i) is labelled either 0,1,2 or 3 (encoded for N,E,W &amp;amp; S respectively)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be grateful if you can just tell me how to approach this problem.&#xA;Links giving direct insight to the problem will also sufficient.&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="33020" LastEditDate="2017-06-05T04:37:45.240" LastActivityDate="2017-06-05T04:37:45.240" Title="Question Regarding Multi-label probability predictions" Tags="|machine-learning|multiclass-classification|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19467" PostTypeId="2" ParentId="19429" CreationDate="2017-06-04T10:57:56.120" Score="1" Body="&lt;p&gt;Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Read in raw documents.&lt;/li&gt;&#xA;&lt;li&gt;Stem words, remove stop words.&lt;/li&gt;&#xA;&lt;li&gt;Perform TF-IDF&lt;/li&gt;&#xA;&lt;li&gt;Train model on cleaned up data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now when it comes time to score:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load your saved model.&lt;/li&gt;&#xA;&lt;li&gt;Repeat steps #2 and #3 from above.&lt;/li&gt;&#xA;&lt;li&gt;Send the &lt;em&gt;cleaned&lt;/em&gt; data into your model for a prediction/recommendation.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is an example in Python of something like that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-04T10:57:56.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19468" PostTypeId="1" CreationDate="2017-06-04T11:05:14.133" Score="4" ViewCount="5589" Body="&lt;p&gt;LDA: linear discriminant analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose our data contains continuous feature values. Then we can apply Naive Bayes using a distribution. Lets assume the data to be normally distributed and so use Naive Bayes with normal distribution. We can also apply LDA which also uses Normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Naive Bayes we assume the features to be independent and by using LDA we assume the covariance to be same for all the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does these assumptions make these 2 models perform differently and which is a better model and in which conditions?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="27916" LastEditDate="2017-06-05T13:01:54.607" LastActivityDate="2020-07-09T14:43:47.837" Title="Difference between LDA and Naive Bayes" Tags="|naive-bayes-classifier|lda|lda-classifier|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19470" PostTypeId="2" ParentId="19465" CreationDate="2017-06-04T14:13:00.460" Score="6" Body="&lt;p&gt;In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give you the probability for each sample in X_test having label 1. &lt;/p&gt;&#xA;" OwnerUserId="14372" LastActivityDate="2017-06-04T14:13:00.460" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="19471" PostTypeId="1" CreationDate="2017-06-04T19:41:12.880" Score="-1" ViewCount="110" Body="&lt;p&gt;I just started to learn Convolutional Neural Network, and like to predict a Pokémon type by its apperance (the input is image).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, while many Pokémon has only one type, some Pokémon have two types (no Pokémon has more than 2 types). I'm not sure the proportion, but it is something like 60% for only one type vs 40% for two types, I guess.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, how should I classify it? Should I mark all the output probability that has &gt; 33% as predicted types, or is there anything better way? Also, is the convolutional neural network suitable in these cases?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-08-05T01:50:30.437" Title="Classification problem on potentially more than 1 labels" Tags="|classification|deep-learning|convolution|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19472" PostTypeId="2" ParentId="19471" CreationDate="2017-06-04T21:10:58.830" Score="0" Body="&lt;p&gt;Convolutional networks are perfect for images. There are two realistic approaches to take with the problem. The first approach, which I think is the weaker of the two is to define every combination of types as a class. This will probably amount to around 50-70 classes which is quite a bit, depending on how many samples you have. Another disadvantage is that you don't get the synergy of having an (A, B) type as an example and another one that is an (A, C). By encoding them as two completely different classes you miss out on the overlap of type A. The other approach is to have a binary output for every type individually. Then you can use some threshold on the sigmoid output of each label to determine if it's that type or not. You can also use the knowledge that it will never have more than two types to not classify a pokemon as type C if A and B have higher activation values.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-04T21:10:58.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19474" PostTypeId="2" ParentId="19466" CreationDate="2017-06-05T00:27:15.783" Score="1" Body="&lt;p&gt;A question is a bit broaden as you do not specify if you do not know how to do it in theory or how to tackle it with a ML method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of ML methods:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LogisticRegression&lt;/a&gt; in sklearn handles multiple class&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lr = LogisticRegression()&#xA;lr.fit(X, y)&#xA;class_probabilities = lr.predict_proba(X)  # outputs the probabilities&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might also want to consider Support Vector Machines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theory: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do a &quot;one vs rest&quot; when you train a single classifier per class taking the sample of all other classes as negative example. (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;wiki article&lt;/a&gt; for that) &lt;/p&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-06-05T00:27:15.783" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19475" PostTypeId="1" CreationDate="2017-06-05T10:27:54.443" Score="1" ViewCount="605" Body="&lt;p&gt;Lets say I have the cifar 10 dataset and I manipulate it in such a way that I reduce the number of frogs in the dataset from 6000 to say 600. I now consider frog as an outlier and train a two class classification problem on this modified cifar 10 dataset. All the other 9 classes as one group and the frog as a outlier. What architecture is best to detect this frog outlier? Where can I find reading materials to learn about these?&lt;/p&gt;&#xA;" OwnerUserId="33046" LastEditorUserId="29169" LastEditDate="2021-02-16T00:46:30.473" LastActivityDate="2021-02-16T00:46:30.473" Title="Convolutional Neural Network for outlier detection" Tags="|neural-network|image-classification|convolutional-neural-network|anomaly-detection|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19476" PostTypeId="1" AcceptedAnswerId="19491" CreationDate="2017-06-05T10:30:41.437" Score="3" ViewCount="374" Body="&lt;p&gt;I have a dataframe containing several features of form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Id, Acol,                   Bcol,   Ccol,           Dcol,&#xA;1,  X:0232,Y:10332,Z:23891, E:1222, F:12912,G:1292, V:1281&#xA;2,  X:432,W:2932            R:2392, T:292,U:29203   Q:2392&#xA;3,  Y:29320,W:2392          R:2932, G:239,T:2392    Q:2391&#xA;&#xA;...about 10,000 Id's&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;where 1,2,3 are the Id's.&lt;/li&gt;&#xA;&lt;li&gt;Acol, Bcol, Ccol, and Dcol are the feature columns,&lt;/li&gt;&#xA;&lt;li&gt;X, Y, Z, W are sub-features of feature &quot;Acol&quot; and so on...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How can I extract the sub-features/features from this sort of dataframe?&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="29575" LastEditDate="2017-06-05T22:18:04.500" LastActivityDate="2017-06-05T22:18:04.500" Title="Extracting sub features from inside a df cell?" Tags="|machine-learning|python|pandas|feature-extraction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19477" PostTypeId="1" CreationDate="2017-06-05T10:50:51.147" Score="1" ViewCount="49" Body="&lt;p&gt;Given a webpage url and the extracted article text from the given page, I want to calculate a probability of the article to be content marketing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I consider content marketing as differentiated from spam, because it is to admit, content marketing makes some serious efforts to deliver real information to the reader. But the true reason behind the effort of writing an article in the context of content marketing is to sooner or later guide the reader to a newsletter subscription, a product landingpage or to contact the author. In the language of content marketers this kind of qualified reader is considered a &quot;lead&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. call to actions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could by using Knn or Bayes simply test the scentences in the article text against a training set of known CTAs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, this where NLP comes in and this is all nothing else than spam detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. link structure&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I possibly can find strong evidence on content marketing motivation, in the outgoing links of a certain page or article. If it links to a landing page styled product page (some heuristics to identify these kind of webpage are needed as well) and a lot of other sites of the website link to a &quot;near by&quot; page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By &quot;near by&quot; I mean text similarity, optimized for a (and the same) keyword.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;question&lt;/strong&gt;&#xA;So what do you think about this? How could scoring look like. How should a probability model look like? Are there aspects, that I miss? Does anyone have some experiences in this field? Is a paper out there? Does anyone know some resources for this? &lt;/p&gt;&#xA;" OwnerUserId="33047" LastActivityDate="2017-06-05T17:00:58.523" Title="Probability or score on how likely an article is motivated by content marketing" Tags="|machine-learning|nlp|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19478" PostTypeId="1" CreationDate="2017-06-05T11:32:57.420" Score="0" ViewCount="1480" Body="&lt;p&gt;I'm currently learning to build a DNN binary classification model on some dataset. But when I analysed the results, I found it counter-intuitive that the training accuracy changed almost at the same rate as the validation accuracy. So did the losses. And they all reached a plateau at some point. How could I possibly improve the model performance at this stage? Here's a snippet of the results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fold: 0 epoch: 0 batch: 0&#xA;training loss: 0.674389 validation loss: 0.67371&#xA;training accuracy: 0.656331 validation accuracy: 0.656968&#xA;Fold: 0 epoch: 0 batch: 500&#xA;training loss: 0.527997 validation loss: 0.527813&#xA;training accuracy: 0.734021 validation accuracy: 0.733639&#xA;Fold: 0 epoch: 0 batch: 1000&#xA;training loss: 0.526019 validation loss: 0.526277&#xA;training accuracy: 0.736067 validation accuracy: 0.735111&#xA;Fold: 0 epoch: 0 batch: 1500&#xA;training loss: 0.525466 validation loss: 0.525721&#xA;training accuracy: 0.736672 validation accuracy: 0.73532&#xA;Fold: 1 epoch: 0 batch: 0&#xA;training loss: 0.525416 validation loss: 0.525261&#xA;training accuracy: 0.735794 validation accuracy: 0.736488&#xA;Fold: 1 epoch: 0 batch: 500&#xA;training loss: 0.525751 validation loss: 0.525623&#xA;training accuracy: 0.735472 validation accuracy: 0.735887&#xA;Fold: 1 epoch: 0 batch: 1000&#xA;training loss: 0.525579 validation loss: 0.525459&#xA;training accuracy: 0.735538 validation accuracy: 0.736077 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33050" LastActivityDate="2017-06-05T14:58:42.383" Title="Need help with how to diagnose training and validation metrics" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19479" PostTypeId="1" CreationDate="2017-06-05T13:45:51.543" Score="3" ViewCount="324" Body="&lt;p&gt;&lt;strong&gt;Problem description&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set about 10000 patients in a study. For each patient, I have a list of various measurements. Some information is scalar data (e.g. age), some information is time series of measurements, some other information can be even a bitmap. The individual record itself can be quite thick (10kB to 10MB). The data is to be processed practically in two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Preprocessing at the level of individual records (patients), i.e. to extract some features in raw data, store them. Calculate some slopes in time series etc. All this can be done on individual level and it can be very easily distributed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On top of the preprocessed data (extracted features), I will need to calculate some aggregated things such as e.g. average age, but also some machine learning tasks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, this is very suitable to be addressed in Apache Spark (or any map-reduce architecture). At the most general level, my question is: what is the most appropriate NoSQL database for this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have considered two basic options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;MongoDB - to take advantage of the document oriented storage where everything is on the same place. However, I am not sure about the performance on the larger binary data (pictures, time-series).&lt;/li&gt;&#xA;&lt;li&gt;Cassandra -  this may have some better storage of binary data, but the joins will be necessary (even if optimized by indexing all data by &quot;patient id&quot;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="3397" LastEditorUserId="8432" LastEditDate="2017-06-05T19:30:08.443" LastActivityDate="2017-06-05T19:30:08.443" Title="Data representation (NoSQL database?) for a medical study" Tags="|machine-learning|nosql|mongodb|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19481" PostTypeId="1" AcceptedAnswerId="19500" CreationDate="2017-06-05T14:16:31.977" Score="2" ViewCount="2254" Body="&lt;p&gt;I am learning deep learning, and as a first exercise to myself I am trying to build a system that learns a very simple task - capitalize the first letter of each word.  As a first step, I am tried to create &quot;character embeddings&quot; - a vector for each character. I am using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import gensim&#xA;model = gensim.models.Word2Vec(sentences)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where sentences is a list of lists of chars which I took from &lt;a href=&quot;https://en.wikipedia.org/wiki/2016_in_aviation&quot; rel=&quot;nofollow noreferrer&quot;&gt;this long Wikipedia page&lt;/a&gt;. For example, sentences[101] is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[' ', ' ', ' ', ' ', 'S', 'p', 'e', 'a', 'k', 'i', 'n', 'g', ' ', 'a', 't', ' ', 't', 'h', 'e', ' ', 'c', 'o', 'n', 'c', 'l', 'u', 's', 'i', 'o', 'n', ' ', 'o', 'f', ' ', 'a', ' ', 'm', 'i', 's', 's', 'i', 'l', 'e', ' ', 'e', 'x', 'e', 'r', 'c', 'i', 's', 'e', ... ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To test the model, I did:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.most_similar(positive=['A', 'b'], negative=['a'], topn=3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hoped to get 'B' at the top, since 'A'-'a'+'b'='B', but I got:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[('D', 0.5388374328613281),&#xA; ('N', 0.5219535827636719),&#xA; ('V', 0.5081528425216675)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(also, my capitalization application did not work so well, but this is probably because of the embeddings).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do to get embeddings that identify capitalization?&lt;/p&gt;&#xA;" OwnerUserId="33051" LastActivityDate="2017-06-06T08:27:13.763" Title="Learning character embeddings with GenSim" Tags="|word2vec|word-embeddings|gensim|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19482" PostTypeId="1" AcceptedAnswerId="19484" CreationDate="2017-06-05T14:26:11.303" Score="3" ViewCount="798" Body="&lt;p&gt;I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?&lt;/p&gt;&#xA;" OwnerUserId="33057" LastEditorUserId="8432" LastEditDate="2017-06-05T19:30:05.400" LastActivityDate="2017-06-06T13:10:52.033" Title="Is it always possible for validation accuracy to be as high as training accuracy?" Tags="|deep-learning|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19483" PostTypeId="2" ParentId="19478" CreationDate="2017-06-05T14:58:42.383" Score="1" Body="&lt;p&gt;Generally, the fact that your training and validation performance are improving at the same rate is a good thing- this (usually) means that the algorithm is learning generalizable features of your problem space rather than overfitting to the noise of your training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reaching a plateau in performance is also to be expected- it's very rare that a real-world machine learning problem can be perfectly solved, and a perfectly solved problem would be the only type that didn't reach a plateau in testing and validation performance (before it eventually did reach a plateau at 100% accuracy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of the plateau as the maximum performance that can be achieved given the particular parameter values, features, and architecture of the solution. In order to achieve performance beyond your plateau values, one of these considerations will need to be adjusted. &lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-06-05T14:58:42.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19484" PostTypeId="2" ParentId="19482" CreationDate="2017-06-05T15:05:21.897" Score="3" Body="&lt;p&gt;Yes, it is possible to have a situation in which validation accuracy cannot be as high as training accuracy. Any situation in which noise (as opposed to generalizable properties of the feature set) in the training set is more predictive of the target variable within the training set would produce this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a situation in which a random property of the training sample considered is perfectly predictive, but this is found not to be true of all other examples outside the sample. The predictive power in the training set would be perfect, but outside the training set, less than perfect. This phenomenon is broadly referred to as &quot;overfitting&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's consider a case where you have a set of fruit data and you're trying to establish whether a given fruit is an orange or a tangerine. You have 4 features- the circumference of the fruit around the stem-medial axis, the height of the fruit across the stem, a numerical value of the hue of the fruit skin, and the first letter of the amateur baseball team whose home park is closest to the field the fruit was grown in. Let's imagine that by some baffling coincidence, the baseball team letter in the training set was perfectly predictive of whether the fruit would be a tangerine or orange. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can imagine that this would not hold true across the country or the world, which would produce a situation where the training set accuracy would be perfect, but the validation set accuracy would not be able to approach that using the same methods.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-06-06T13:10:52.033" LastActivityDate="2017-06-06T13:10:52.033" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="19485" PostTypeId="1" CreationDate="2017-06-05T15:21:27.283" Score="1" ViewCount="157" Body="&lt;p&gt;I f have a time series of long time horizon (like a stock price), how do I train in the simplest way the echo state network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I chose the training set to be 2 years e.g. then I could chose every day in those 2 years as input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I train the network for 1 month? So i divide the 2 years in parts of 1 month. And for every month in the past I train the network and do average of all neworks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, waht are common methods for time series analysis with echo state networks for training? Do I rather train the whole time series or do I divide it supparts to train?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastActivityDate="2017-06-05T15:21:27.283" Title="How to train an echo state network" Tags="|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19486" PostTypeId="2" ParentId="19479" CreationDate="2017-06-05T16:56:00.973" Score="3" Body="&lt;p&gt;I think you need to define the process and desired outcome a little more. It sounds like you need to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Define what features you want.&lt;/li&gt;&#xA;&lt;li&gt;Figure out how to extract those features.&lt;/li&gt;&#xA;&lt;li&gt;Figure out how to store those features.&lt;/li&gt;&#xA;&lt;li&gt;Pass that dataset to a ML model for training.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would figure out what exactly you want for #3 in terms of the number and type of data elements and then choose a data storage method only then.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you are going to be passing the unstructured documents directly to your model, you don't need MongoDB's capabilities. 10,000 records is small, but since you mentioned you want to calculate some aggregated statistics on your patient-level data, you could likely get by with something as simple as MySQL or SQLite. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spark and Map/Reduce are actually competitors, with Spark stealing MR's spotlight lately. You might need one or the other for feature extraction but they're probably overkill for the rest of the stuff you described.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="30212" LastEditDate="2017-06-05T17:02:00.150" LastActivityDate="2017-06-05T17:02:00.150" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="19487" PostTypeId="2" ParentId="19477" CreationDate="2017-06-05T17:00:58.523" Score="1" Body="&lt;p&gt;&lt;strong&gt;If&lt;/strong&gt; you have a broad set of testing data, I think this is feasible. I've had luck in getting basic models to identify abstract concepts like someone's politics or real news vs fake news, so I think this could work if people write content marketing different from normal news.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just used this &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; in another post, but here is a tutorial in Python of taking unstructured articles, passing them through an NLP pipeline, and classifying them into multiple groups.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-05T17:00:58.523" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19489" PostTypeId="2" ParentId="19449" CreationDate="2017-06-05T18:31:43.213" Score="0" Body="&lt;p&gt;There might be a more principled way to go about this but I don't know how IP assignment works. That said,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All your information content appears to be in the second 2 numbers. The last 0-255 might be assumed and the leading 1 can be dropped. You might have something more complicated but as you lay it out, you can leave them as strings and assign to 3 groups: &#xA;'0.0', &#xA;'0.1 or '0.3'&#xA;'4.0'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may not need to spell out all possible IPs. Define your problem space first. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-06-05T18:31:43.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19490" PostTypeId="1" AcceptedAnswerId="21848" CreationDate="2017-06-05T19:32:57.950" Score="11" ViewCount="2857" Body="&lt;p&gt;I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could use that to speed up some of the processing for some of the stuff my lecturers have me doing, but it really doesn't seem easy to do this at all. I've installed gpuR, CUDA, Rtools, and a few other bits and bobs, and I can get it to create gpuMatrix objects from genomic expression data, for example, but I have yet to find a function which both works with the gpuMatrix objects and also provides any noticeable difference in performance. Perhaps this just relates to limitations inherent to the gpuR package - some other packages do seem to talk about having functions which sound like they would be more like the sort of thing I'm looking for, which brings me to the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost all of those packages are exclusively for Linux, is it particularly hard to implement GPU support for R in windows? Or is there some other reason that there are so few packages available to do this in Windows? In some sense I'm just curious, but it would also be very cool to really get it working. It surprises me that there is so little available for Windows, usually it's the other way around.&lt;/p&gt;&#xA;" OwnerUserId="33072" LastEditorUserId="33072" LastEditDate="2017-06-05T20:19:33.077" LastActivityDate="2019-09-11T09:31:09.027" Title="GPU Accelerated Data Processing for R in Windows" Tags="|r|gpu|parallel|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19491" PostTypeId="2" ParentId="19476" CreationDate="2017-06-05T19:44:10.880" Score="2" Body="&lt;p&gt;You were not specific where you wanted to end up with the data in this frame, so I will simply show how to break out the features and sub features into a format that can be pivoted into a form as needed:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Most important element is taking a feature column and breaking out the sub features.  That can be done like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_sub_features(feature_col):&#xA;    # split on commas and then colons&#xA;    feature_df = feature_col.str.split(',').apply(&#xA;        lambda feature: pd.Series(&#xA;            dict([sub_feature.split(':') for sub_feature in feature]),&#xA;            name=feature_col.name), 1)&#xA;&#xA;    # add a feature name column to use as an index&#xA;    feature_df['feature'] = feature_col.name&#xA;&#xA;    # name the columns as sub-feature for later stacking&#xA;    feature_df.columns.names = ['sub-feature']&#xA;&#xA;    # return dataframe with id/feature_name index&#xA;    new_index = [feature_df.index.name, 'feature']&#xA;    return feature_df.reset_index().set_index(new_index)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_fwf(StringIO(u&quot;&quot;&quot;&#xA;    Id  Acol                    Bcol    Ccol            Dcol&#xA;    1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281&#xA;    2   X:432,W:2932            R:2392  T:292,U:29203   Q:2392&#xA;    3   Y:29320,W:2392          R:2932  G:239,T:2392    Q:2391&quot;&quot;&quot;&#xA;                          ), header=1).set_index(['Id'])&#xA;print(df)&#xA;&#xA;feature_cols = ['Acol', 'Bcol', 'Ccol', 'Dcol']&#xA;stacked = pd.concat(get_sub_features(df[f]).stack() for f in feature_cols)&#xA;print(stacked)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                      Acol    Bcol            Ccol    Dcol&#xA;Id                                                        &#xA;1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281&#xA;2             X:432,W:2932  R:2392   T:292,U:29203  Q:2392&#xA;3           Y:29320,W:2392  R:2932    G:239,T:2392  Q:2391&#xA;&#xA;Id  feature  sub-feature&#xA;1   Acol     X               0232&#xA;             Y              10332&#xA;             Z              23891&#xA;2   Acol     W               2932&#xA;             X                432&#xA;3   Acol     W               2392&#xA;             Y              29320&#xA;1   Bcol     E               1222&#xA;2   Bcol     R               2392&#xA;3   Bcol     R               2932&#xA;1   Ccol     F              12912&#xA;             G               1292&#xA;2   Ccol     T                292&#xA;             U              29203&#xA;3   Ccol     G                239&#xA;             T               2392&#xA;1   Dcol     V               1281&#xA;2   Dcol     Q               2392&#xA;3   Dcol     Q               2391&#xA;dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Accessing Data:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Some examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(stacked.xs('T', level=2))&#xA;print(stacked.iloc[stacked.index.get_level_values('sub-feature') == 'T'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               0&#xA;Id feature      &#xA;2  Ccol      292&#xA;3  Ccol     2392&#xA;&#xA;                           0&#xA;Id feature sub-feature      &#xA;2  Ccol    T             292&#xA;3  Ccol    T            2392    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-06-05T19:44:10.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19493" PostTypeId="1" CreationDate="2017-06-05T21:48:08.433" Score="2" ViewCount="18669" Body="&lt;p&gt;Kolmogorov-Smirnov (KS) statistics is one of the commonly used measures to assess predictive power for marketing or credit risk models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The KS statistic is usually published for logistic regression problems to give an indication of the quality of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Wikipedia page (&lt;a href=&quot;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&lt;/a&gt;) gives this explanation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YPZfL.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YPZfL.jpg&quot; alt=&quot;KS statistic&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The KS statistic calculation below is from page 5 on &lt;a href=&quot;https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(ROCR)&#xA;set.seed(7)&#xA;prd=runif(1000)&#xA;act=round(prd)&#xA;prd[sample(1000,500)]=runif(500) #noise&#xA;&#xA;pred&amp;lt;-prediction(prd,act)&#xA;perf &amp;lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)&#xA;#this code builds on ROCR library by taking the max delt&#xA;#between cumulative bad and good rates being plotted by&#xA;#ROCR see https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf&#xA;ks=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])&#xA;plot(perf,main=paste0(' KS=',round(ks*100,1),'%'))&#xA;lines(x = c(0,1),y=c(0,1))&#xA;print(ks);&#xA;auc &amp;lt;- performance(pred, measure = &quot;auc&quot;)&#xA;auc &amp;lt;- auc@y.values[[1]]&#xA;print(auc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It gives a KS of 0.4939511 and an AUC of 0.7398465. Area under the curve is obviously correct as one can see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7cwnR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7cwnR.png&quot; alt=&quot;From R code&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the R code for the KS statistic correct?&#xA;If not, what should it be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Update&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Warning:&lt;/em&gt; From a sample decision tree output which is very clumpy, the Sharma method gives a better estimate of the KS score while the ks.test gives a very bad one.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     auc         ks.score.D^+ sharma.ks&#xA;    0.6153846    0.7045455    0.2307692 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The ks.test function gives this error: &lt;code&gt;1: In ks.test(prd, act, alternative = &quot;greater&quot;) :  cannot compute exact p-value with ties&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rb94c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rb94c.png&quot; alt=&quot;Dec Tree&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;act=c(1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, &#xA;  0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, &#xA;  0, 0, 1)&#xA;prd=c(0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.352941176470588, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, &#xA;  0.914893617021277, 0.352941176470588, 0.352941176470588, 0.914893617021277, &#xA;  0.914893617021277) ##this only has two values&#xA;ks.test(prd, act, alternative='greater')$statistic&#xA;#gives 0.7045455 which clearly isn't correct&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8650" LastEditorUserId="8650" LastEditDate="2017-06-24T13:14:05.573" LastActivityDate="2018-07-17T08:51:19.510" Title="What is a good method to generate the KS-statistic in R?" Tags="|r|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19494" PostTypeId="2" ParentId="19471" CreationDate="2017-06-05T23:53:19.120" Score="0" Body="&lt;p&gt;For multi-label image classification, you want to make sure that your output activation is a sigmoid function (like Jan said) and train with one hot encoded labels for each of your images. For your specific problem, I would recommend adding a &lt;code&gt;None&lt;/code&gt; label or choice so that each image doesn't need to have two types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a web page that could possibly help you understand and implement this:&#xA;&lt;a href=&quot;https://suraj-deshmukh.github.io/Multi-Label-Image-Classification/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://suraj-deshmukh.github.io/Multi-Label-Image-Classification/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32130" LastActivityDate="2017-06-05T23:53:19.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19495" PostTypeId="2" ParentId="8358" CreationDate="2017-06-06T05:15:34.697" Score="5" Body="&lt;p&gt;Try this：&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pickle&#xA;with open('rforest.pickle', 'wb') as f:&#xA;    pickle.dump(rforest, f, -1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice: with the parameter &quot;-1&quot; the model file size will largely be reduced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the documentation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;pickle.dump(obj, file[, protocol])&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Protocol version 0 is the original ASCII protocol and is backwards compatible with earlier versions of Python. &#xA;  Protocol version 1 is the old binary format which is also compatible with earlier versions of Python. &#xA;  Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;If the protocol parameter is omitted, protocol 0 is used. If protocol&#xA;  is specified as a negative value or HIGHEST_PROTOCOL, the highest&#xA;  protocol version will be used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="27265" LastActivityDate="2017-06-06T05:15:34.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19496" PostTypeId="2" ParentId="19427" CreationDate="2017-06-06T07:39:57.073" Score="1" Body="&lt;p&gt;You might try to use the metadata of the individual sections for classification: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Length -&gt; Abstracts are usually short. &lt;/li&gt;&#xA;&lt;li&gt;Existence of Figures -&gt; Do not usually appear in abstract of references&lt;/li&gt;&#xA;&lt;li&gt;Relative Abundance of quotations. The references should have a significant higher share of reference keys than the others. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You could then use the metadata as input for any supervised learning approach. For example a Neural Network or a (boosted) decision tree.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-06-06T07:39:57.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19497" PostTypeId="2" ParentId="19097" CreationDate="2017-06-06T07:48:17.967" Score="0" Body="&lt;p&gt;There is no model able to handle all the problem. For small images classification, you can just use model working on bigger ones.&#xA;For instance, I'm working with (20,20,3) images and Inception is my best model for now (Xception is about to be better when I'll have finish with it I guess). The only thing you have to do is to adapt your model to your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're working with small images, you don't have to use a 256 convolutional layer, it's just overkill and can make your model weaker. If you're using inception, you must consider reduce the convolutional kernel numbers, maybe the number of different branches in each block and the number of block.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it has been usefull for you !&lt;/p&gt;&#xA;" OwnerUserId="33091" LastActivityDate="2017-06-06T07:48:17.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19498" PostTypeId="2" ParentId="4977" CreationDate="2017-06-06T08:14:04.777" Score="1" Body="&lt;p&gt;There are a lot of choices based on what you are looking to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do you want to write the logic?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking to write code for your logic ( and not looking for a SQL like declarative language) then you can use one of Stream Processing Engines: Apache Storm, Apache Spark, Apache Fink, Apache Samza or Kafka Streams. The good news is all of them are open source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among them, my bet is Fink. It has a pretty nice architecture which handles many pitfalls others stepped into. Especially, if you want to do exactly once processing, Fink is the best bet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, writing java code to do this is fine if you do simple counting, but a nightmare if you want to do serious things like time windows, join and temporal event sequence patterns etc. ( see Patterns for Streaming Realtime Analytics for details on this argument). If you want to express the logic using a declarative language, then you will want to use Complex Event Processing, which let you write SQL-like queries and do complex things. Specifically, these kinds of queries support temporal operators like sliding windows, joins, temporal sequence patterns etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for an Open source solution, the among the choices are &lt;a href=&quot;http://wso2.com/smart-analytics&quot; rel=&quot;nofollow noreferrer&quot;&gt;WSO2 DAS (Data Analytics Server)&lt;/a&gt; formally known as WSO2 CEP ( Apache License) and Esper (GPL). ( Disclaimer I work for WSO2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are OK to pay, then there is Tibco Stream Base, IBM Infosphere Streams, and SQLStreams. Both WSO2 DAS and Esper have commercial support if you need it. Paul Vincent is maintaining CEP market Survey which provides historical overview of almost all CEP engines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recent Forrester report surveyed the choices and you can find the details from 15 &quot;True&quot; Streaming Analytics Platforms For Real-Time Everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Above stream processing engines are adding support for SQL like query languages. However, they still lag behind with the features ( e.g. temporal sequence patterns). As I discussed in &lt;a href=&quot;https://softwareengineeringdaily.com/2016/02/04/stream-processing-vs-complex-event-processing/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://softwareengineeringdaily.com/2016/02/04/stream-processing-vs-complex-event-processing/&lt;/a&gt;, CEP and Stream processing engines are merging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scalability&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you plan to process millions of events per second, then you need to setup a complex distributed processing graph. On this topic, stream processing has an advantage where they are historically built to scale. However, as I mentioned, these two models are merging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IoT Analytics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing an IoT analytics, then the data creates a time series. In that case, often you need to do temporal operations such as sliding windows. On that use case, you are better off with an SQL like language hence I recommend to go for CEP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Streaming + Batch&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final twist is whether you want to do batch ( MapReduce style processing) as well. If yes, read about Lambda Architecture and Kappa architecture. Apache Fink and WSO2 DAS supports this scenario. ( and there may be others).&lt;/p&gt;&#xA;" OwnerUserId="33092" LastActivityDate="2017-06-06T08:14:04.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19499" PostTypeId="1" AcceptedAnswerId="19505" CreationDate="2017-06-06T08:22:08.483" Score="0" ViewCount="197" Body="&lt;p&gt;0&#xA;down vote&#xA;favorite&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an imbalanced dataset to work with (with about 20-fold positive examples than negative). I know several solutions to deal with this type of data (under/oversampling, optimizing for AUC, etc.) but I would like to try changing the misclassification costs for the two groups. However, I cannot figure out the proper way to do this in caret. Do you have any idea how to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any help!&lt;/p&gt;&#xA;" OwnerUserId="33095" LastActivityDate="2017-06-06T12:11:19.900" Title="Is it possible to change misclassification cost in caret?" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19500" PostTypeId="2" ParentId="19481" CreationDate="2017-06-06T08:27:13.763" Score="3" Body="&lt;p&gt;I believe that you misunderstood the word2vec concept. Basically for words, the feature vector for a word is learnt from the surrounding words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;You shall know a word by the company it keeps&lt;/code&gt;- Firth.J.R&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case characters have been used, so the feature vector for each character depends upon the adjacent characters present. your example might work, if you have the following training sentences.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ABCDEFGHI &lt;/li&gt;&#xA;&lt;li&gt;abcdefghi &lt;/li&gt;&#xA;&lt;li&gt;AbcDEFghi &lt;/li&gt;&#xA;&lt;li&gt;aBcdEfgHI &lt;/li&gt;&#xA;&lt;li&gt;abcDEFgHi &lt;/li&gt;&#xA;&lt;li&gt;ABcdEFGHi&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With these training sentences, the characters 'A','a','B','b' will preserve capital features and english alphabet order feature. But whereas, when trained with wikipedia sentences, the characters will preserve the probability of being present in a meaningful word. For instance, the closest letter to 'C' would be 'a','o','e' but hardly 'x' or 'd' because there would be words like 'covenant','country','cat' as no common words would be 'cx..'&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-06-06T08:27:13.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19502" PostTypeId="2" ParentId="12500" CreationDate="2017-06-06T09:09:58.823" Score="6" Body="&lt;p&gt;A genetic algorithm is an algorithm based on biological evolution, on how nature evolved. It does exactly that, evolve the algorithm so that &quot;it&quot; finds the best solution to the problem at hand. You can use a genetic algorithm to find a solution to a problem which you don't know the answer, you know the answer but want to know a different one, or you are just lazy. The common steps for a genetic algorithm are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate a random population of elements&lt;/li&gt;&#xA;&lt;li&gt;Evaluate fitness of each element (how good are they against the&#xA;solution)&lt;/li&gt;&#xA;&lt;li&gt;Take the best elements for the next generation&lt;/li&gt;&#xA;&lt;li&gt;Generate child elements using the above elements &lt;/li&gt;&#xA;&lt;li&gt;Mutate (randomly) each child. This can also be done before step 4 with parents.&lt;/li&gt;&#xA;&lt;li&gt;Repeat from step 2 for n number of generations until the solution is found, the maximum number of generations have been reached or the fitness is not changing anymore (local minima).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Two basic problems with GAs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To get trapped in a local minima (or local maximum depending on the point of view). This means that you might find a good answer (or not even a good one) but it will never reach the best answer because fitness values are not changing or not getting better. This is fought using Mutation in step 5 to keep diversity in the population so it doesn't get stuck.&lt;/li&gt;&#xA;&lt;li&gt;They are generally slower than other approaches.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you can, take a look at &lt;a href=&quot;http://www.springer.com/gb/book/9781484203293&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; book. It is not free, but worth a look.&#xA;Alternatively, take a look at &lt;a href=&quot;http://natureofcode.com/book/chapter-9-the-evolution-of-code/&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; online book for free, it is a great source and he has his own youtube channel. It's more basic than the other book I recommend, but will help you get started with GAs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the other part of your question, the GA should be used to find a model, but will not act as a model. For example, if you have a neural network you can train it using the backpropagation method, but you could also train it using a GA. The GA will not use anything of the backpropagation mathematics, it could be used to generate weights in its neurons, and evaluate the answer (last layer). Weights will evolve to get closer to the solution. In this scenario, the model will still be the NN, but you used a different algorithm to find the best NN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26700" LastActivityDate="2017-06-06T09:09:58.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19503" PostTypeId="2" ParentId="16088" CreationDate="2017-06-06T09:16:06.623" Score="2" Body="&lt;p&gt;I recommend these books:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://natureofcode.com/book/chapter-9-the-evolution-of-code/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The nature of code&lt;/a&gt;: This book is a good introduction to GAs in general, and he has his own youtube channel with explanations and examples. A good starting point.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springer.com/gb/book/9781484203293&quot; rel=&quot;nofollow noreferrer&quot;&gt;Genetic Algorithms in Java Basics&lt;/a&gt;: More in depth but very well explained and easy to understand, focused on java programming.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can also see my answer &lt;a href=&quot;https://datascience.stackexchange.com/a/19502/26700&quot;&gt;here&lt;/a&gt; to have an initial and very brief general introduction of what Genetic Algorithms are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="26700" LastEditorUserId="26700" LastEditDate="2018-08-22T14:17:16.423" LastActivityDate="2018-08-22T14:17:16.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19504" PostTypeId="1" AcceptedAnswerId="19506" CreationDate="2017-06-06T10:55:16.847" Score="1" ViewCount="169" Body="&lt;p&gt;I am trying to stitch together multiple packages and tools from multiple languages (R, python, C etc.) in a single analysis workflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard way to do it? Preferably (but not necessarily) in python.&lt;/p&gt;&#xA;" OwnerUserId="31446" LastActivityDate="2017-06-06T12:39:45.867" Title="Standard method to integrate tools coded in multiple languages in an analysis workflow" Tags="|methods|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19505" PostTypeId="2" ParentId="19499" CreationDate="2017-06-06T12:11:19.900" Score="1" Body="&lt;p&gt;I would have to read more carefully: &lt;code&gt;train&lt;/code&gt; has a &lt;code&gt;weights&lt;/code&gt; parameter that do this.&lt;/p&gt;&#xA;" OwnerUserId="33095" LastActivityDate="2017-06-06T12:11:19.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19506" PostTypeId="2" ParentId="19504" CreationDate="2017-06-06T12:34:27.543" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/spotify/luigi&quot; rel=&quot;nofollow noreferrer&quot;&gt;Luigi&lt;/a&gt; is an open source python package by Spotify that does exactly what you described: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Its philosophy is similar to GNU Make, letting you define tasks and their dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also &lt;a href=&quot;https://airflow.incubator.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Airflow&lt;/a&gt; (originally from Airbnb), another python solution:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Airflow is a platform to programmatically author, schedule and monitor workflows.&#xA;  Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You may found a complete comparative table &lt;a href=&quot;http://bytepawn.com/luigi-airflow-pinball.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-06-06T12:39:45.867" LastActivityDate="2017-06-06T12:39:45.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19508" PostTypeId="2" ParentId="19427" CreationDate="2017-06-06T13:34:07.867" Score="0" Body="&lt;p&gt;A simple approach might use &lt;a href=&quot;https://arxiv.org/abs/1507.07998&quot; rel=&quot;nofollow noreferrer&quot;&gt;document vectors&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA&lt;/a&gt; representing the text of each section (possibly chunking longer sections) with a simple classifier trained on those vectors to predict section. These can be found in &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/decomposition.html#latent-dirichlet-allocation-lda&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A harder to implement but likely more powerful technique would be to train a recurrent neural network on the text of each section (again likely breaking it into smaller chunks) so that the representation your model learns is learned specifically to predict Abstract/Indices/etc ...&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2017-06-06T13:34:07.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19509" PostTypeId="2" ParentId="15393" CreationDate="2017-06-06T14:43:55.277" Score="1" Body="&lt;p&gt;I think in a lot of cases, when people are using pre-trained models, they force their words to lower and singular case and then test to see if the lowercase and singular (not plural) version were trained. Sometimes, you have to check yourself like A.M. Bittlingmayer said. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To provide some insight into why sometimes people remove stopwords versus not --- I think it has to do with whether they want to value infrequent words more or less. Because you don't want to value too highly very frequent words, the algorithm 'punishes' extremely frequent words. So, one could argue that because this already happens, you don't need to remove stopwords since they will be 'punished' anyways. So, not everyone removes stopwords. &lt;/p&gt;&#xA;" OwnerUserId="33106" LastActivityDate="2017-06-06T14:43:55.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19510" PostTypeId="2" ParentId="19493" CreationDate="2017-06-06T14:58:00.667" Score="2" Body="&lt;p&gt;Yes, this is the correct way to use the KS statistic to evaluate the performance of a model. As the true positive and false positive rates are already cumulative density functions over the range 0:1, they reasonably can fit into the paradigm of the more general use of the K-S statistic as it applies to comparison of these types of functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WxNwm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WxNwm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-06-06T15:56:34.247" LastActivityDate="2017-06-06T15:56:34.247" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19514" PostTypeId="1" CreationDate="2017-06-06T16:54:32.303" Score="1" ViewCount="684" Body="&lt;p&gt;I have a Data Set of 170 images in total. of each image i want to create 36 images like this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of the image. Then, the new image is scaled back to the original size of &lt;code&gt;188x188&lt;/code&gt;. This cropping method could be also done from three other corners. The same thing could be done with 10 percent cropping. This results in 8 images besides the original image. Afterward, these 9 images are rotated by 0, 90, 180 and 270 degrees. Therefore, 36 synthesized images are derived from each input image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need python code for this dataset augmentation.&lt;/p&gt;&#xA;" OwnerUserId="33110" LastEditorUserId="29575" LastEditDate="2017-06-07T09:29:57.243" LastActivityDate="2017-07-07T09:42:20.303" Title="Images Dataset Augmentation with fixed parameters of Crop &amp; Rotation Angle" Tags="|python|data|image-classification|data-augmentation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19516" PostTypeId="2" ParentId="19343" CreationDate="2017-06-06T17:50:29.903" Score="1" Body="&lt;p&gt;This problem is best addressed with discrete-event, stochastic simulation modeling.  The prediction you get from such a model is a probability distribution on the surgery being canceled.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy to code up such a model given your problem statement.  You need:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A boolean function for each kind of surgery, which, given the cast of characters and skill sets as input, determines True/False if the surgery can proceed.&lt;/li&gt;&#xA;&lt;li&gt;A probability distribution for the appearance of each kind of character, when scheduled.&lt;/li&gt;&#xA;&lt;li&gt;An object which is initialized with the schedule.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The idea is to do this check (surgery/surgery canceled) many times, thereby growing a set of outcomes for each policy you wish to explore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want to roll your own, there are packages for many languages such as &lt;a href=&quot;https://simpy.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SimPy&lt;/a&gt; for Python and &lt;a href=&quot;http://r-simmer.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Simmer&lt;/a&gt; for R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice thing about discrete event simulation is that you don't need to make any simplifying assumptions about probabilities, and the business logic can be arbitrarily complex.  For example, if the surgeon and one of the nurses are having an affair, the chance that one cancels on the same day as the other can be increased.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2017-06-06T17:50:29.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19517" PostTypeId="1" CreationDate="2017-06-06T18:12:53.580" Score="4" ViewCount="359" Body="&lt;p&gt;I am learning OCR and reading &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/3319502514&quot; rel=&quot;nofollow noreferrer&quot;&gt;this book&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors define 8 processes to implement OCR that follow one by one (2 after 1, 3 after 2 etc):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Optical scanning&lt;/li&gt;&#xA;&lt;li&gt;Location segmentation&lt;/li&gt;&#xA;&lt;li&gt;Pre-processing&lt;/li&gt;&#xA;&lt;li&gt;Segmentation&lt;/li&gt;&#xA;&lt;li&gt;Representation&lt;/li&gt;&#xA;&lt;li&gt;Feature extraction&lt;/li&gt;&#xA;&lt;li&gt;Recognition&lt;/li&gt;&#xA;&lt;li&gt;Post-processing&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is what they write about representation (#5)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The fifth OCR component is representation. The image representation&#xA;  plays one of the most important roles in any recognition system. In&#xA;  the simplest case, gray level or binary images are fed to a&#xA;  recognizer. However, in most of the recognition systems in order to&#xA;  avoid extra complexity and to increase the accuracy of the algorithms,&#xA;  a more compact and characteristic representation is required. For this&#xA;  purpose, a set of features is extracted for each class that helps&#xA;  distinguish it from other classes while remaining invariant to&#xA;  characteristic differences within the class.The character image &#xA;  representation methods are generally categorized into three major&#xA;  groups: (a) global transformation and series expansion (b) statistical&#xA;  representation and (c) geometrical and topological representation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is what they write about feature extraction (#6)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The sixth OCR component is feature extraction. The objective of&#xA;  feature extraction is to capture essential characteristics of symbols.&#xA;  Feature extraction is accepted as one of the most difficult problems&#xA;  of pattern recognition. The most straight forward way of describing&#xA;  character is by actual raster image. Another approach is to extract&#xA;  certain features that characterize symbols but leaves the unimportant&#xA;  attributes. The techniques for extraction of such features are divided&#xA;  into three groups’ viz. (a) distribution of points (b) transformations&#xA;  and series expansions and (c) structural analysis.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am totally confused. I don't understand what is representation. As I understand after segmentation we must take from image some features, for example topological structure like Freeman chain code and must match to some saved at the learning stage model - i.e. to do recognition. By other words - segmentation - feature extraction - recognition. I don't understand what must be done on representation stage. Please, explain.&lt;/p&gt;&#xA;" OwnerUserId="33118" LastEditorUserId="11097" LastEditDate="2017-08-17T15:56:36.950" LastActivityDate="2018-05-14T21:05:07.857" Title="What is representation in optical character recognition?" Tags="|feature-selection|feature-extraction|feature-engineering|ocr|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19518" PostTypeId="2" ParentId="19432" CreationDate="2017-06-06T19:10:12.677" Score="4" Body="&lt;p&gt;Backtracking the updater source code, it looks like &quot;extra nodes&quot; are calculated this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each boosting stage, looking at the gradient boost tree, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extra Nodes = (the total number of nodes) - (the number of start roots) - (the number of deleted nodes)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each boosting stage, there might be different starting roots (sub trees) and different deleted (so far) nodes. The extra nodes can provide some intuition into how much your processing tree is utilized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/blob/72451457120ac9d59573cf7580ccd2ad178ef908/src/tree/updater_prune.cc&quot; rel=&quot;nofollow noreferrer&quot;&gt;updater_prune.cc&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/blob/6bd1869026eb66a542ccd8573f1e4b098a8cf7c6/include/xgboost/tree_model.h&quot; rel=&quot;nofollow noreferrer&quot;&gt;tree_model.h&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost &quot;train&quot; api&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17177" LastEditorUserId="2723" LastEditDate="2017-06-07T01:53:35.873" LastActivityDate="2017-06-07T01:53:35.873" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19519" PostTypeId="1" CreationDate="2017-06-07T03:21:22.027" Score="4" ViewCount="551" Body="&lt;p&gt;I have set of images which they look like the below sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D6RRw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D6RRw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of it as a person's 2-week schedule represented by different color. Inside the images. However, there are some small missing areas shown in black. I'm interested in inpaint/fill the missing areas using ML methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I have already tried classical methods like some those mentioned in the paper: &quot;Image Inpainting, Christine Guillemot and Olivier Le Meur&quot;, but the result was not satisfactory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since almost all images have the same pattern, I'd like to know if anyone can suggest the ML-based approaches tackle this problem.&lt;/p&gt;&#xA;" OwnerUserId="33128" LastActivityDate="2018-03-30T23:05:21.383" Title="Machine Learning Based Algorithm for Image Inpainting?" Tags="|machine-learning|image-recognition|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19520" PostTypeId="2" ParentId="19514" CreationDate="2017-06-07T04:46:08.243" Score="3" Body="&lt;p&gt;If you need to work on images using Python, the preferred library is &lt;a href=&quot;http://www.effbot.org/imagingbook/pil-index.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;PIL&lt;/a&gt;.  Here I show a function to do the modifications you have delimited.  This code makes no effort to manage multiple files, or how to name the files converted.  But it does show how to modify the images in the ways you have asked.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import Image&#xA;&#xA;def modify_image(im, crop_corner=None, crop_percent=5, rotate=0):&#xA;    &quot;&quot;&quot;&#xA;    Crop and rotate an image&#xA;&#xA;    :param im: base image &#xA;    :param crop_corner: crop corner 0-3 clockwise from top left &#xA;    :param crop_percent: crop percentage of size &#xA;    :param rotate: rotate degrees (rotate after crop) &#xA;    :return: a modified image &#xA;    &quot;&quot;&quot;&#xA;    x, y = im.size&#xA;    if crop_corner is not None:&#xA;        crop_x, crop_y = [int(i * crop_percent / 100.0) for i in (x, y)]&#xA;        size_x = x - crop_x&#xA;        size_y = y - crop_y&#xA;        if crop_corner in (1, 2):&#xA;            crop_x = 0&#xA;        if crop_corner in (2, 3):&#xA;            crop_y = 0&#xA;        im = im.crop((crop_x, crop_y, crop_x + size_x-1, crop_y + size_y-1))&#xA;        im = im.resize((x, y), Image.BICUBIC)&#xA;&#xA;    im = im.rotate(rotate, expand=True)&#xA;    return im&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_image = Image.open('test_image.png')&#xA;new_image = modify_image(test_image, rotate=90)&#xA;new_image.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To save the image see the (&lt;a href=&quot;http://www.effbot.org/imagingbook/image.htm#tag-Image.Image.save&quot; rel=&quot;nofollow noreferrer&quot;&gt;Docs&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-06-07T04:46:08.243" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19521" PostTypeId="1" CreationDate="2017-06-07T05:33:53.713" Score="3" ViewCount="211" Body="&lt;p&gt;I am trying to come up with a platform which can Synthesize Quality Content among many articles found about some particular &quot;topic&quot; on the internet, the algorithm should be able to say recommend top 10 articles found on the internet about the topic. What algorithms and methods should be used to evaluate writing quality and give some kind of a score on many other parameters like &quot;source credibility&quot;. I have taken care of crawling and finding out the content, looking for algorithms and methods to evaluate this. Please suggest resources(course,research papers etc..) for the same.&lt;/p&gt;&#xA;" OwnerUserId="33130" LastActivityDate="2017-08-20T17:52:12.287" Title="How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?" Tags="|machine-learning|nlp|cosine-distance|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="19523" PostTypeId="1" AcceptedAnswerId="19526" CreationDate="2017-06-07T07:39:26.060" Score="7" ViewCount="7480" Body="&lt;p&gt;I'm exploring options for recommender systems optimized for the insurance industry, which would take into account&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i) product holdings&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ii) user characteristics (segment, age, affluence, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to stress that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) there are no product ratings available, thus collaborative filtering is not an option&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) recommended products don't have to be similar to products that have already been purchased, thus item-to-item recommendations are most probably not relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that in insurance you rarely want to recommend similar products to those already purchased ones, as someone with the Car insurance is unlikely to want to buy another Motor product, rather Home or maybe Travel, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;That's why I want to develop recommendations on similarities between the users based on their purchase history and/or demographics&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like to be able to implement it in R, if not possible, then in Python. Thanks for help and suggestions!&lt;/p&gt;&#xA;" OwnerUserId="28291" LastEditorUserId="28291" LastEditDate="2017-06-07T10:47:16.913" LastActivityDate="2019-04-19T08:11:34.893" Title="Recommender system based on purchase history, not ratings" Tags="|machine-learning|python|r|recommender-system|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19526" PostTypeId="2" ParentId="19523" CreationDate="2017-06-07T10:24:45.650" Score="6" Body="&lt;p&gt;You could use Content based filtering but then you have to intelligently pre process the data to extract all the contents of the products. Also, that might lead to leaving a some features, &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2015/08/beginners-guide-learn-content-based-recommender-systems/&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; article is a great head start after you preprocess all the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you could make pseudo ratings for product vs a customer. That would depend on your problem statement. Some few suggestions could be the number of times the customer bought the particular product in last one month or you could also take an index which would define how frequently the customer buys that product which, mathematically would be last_two_purchases/interval_of_purchase or could also take an average of last few purchases and intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After making this pseudo rating you could convert all the content based features into numerical ones and use Latent factor model for collaborative filtering. Refer &lt;a href=&quot;http://course.fast.ai/lessons/lesson4.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; video. Python could be used for this.&lt;/p&gt;&#xA;" OwnerUserId="23380" LastActivityDate="2017-06-07T10:24:45.650" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19527" PostTypeId="1" AcceptedAnswerId="28652" CreationDate="2017-06-07T11:51:19.770" Score="3" ViewCount="1823" Body="&lt;p&gt;I have a labeled dataset of two classes. Let’s say sick and healthy patients. My features are patient data as well as diagnostic data such as blood-test results. It’s categorical as well as continuous data. In total several thousand features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I want to identify the features that best differentiate the sick patients from the healthy ones.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not really an anomaly detection problem, because the sickness is not &lt;em&gt;directly&lt;/em&gt; indicated by anything in my features. Meaning that for none of my features the sick patients would be indicated by outliers. For example a result could be that the risk of sickness is higher for patients over the age of 50. But being over 50 is not generally uncommon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious approach would be to use a dedicated feature selection technique. For example a chi-square test, variable importance in decision trees or backward elimination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem about all of these approaches is that they are usually preprocessing steps before another learning algorithm is deployed. I on the other hand don’t want to use another learning algorithm. The entire task is to figure out the most important features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are feature selection algorithms really the best way to proceed, or are there any better approaches? Maybe some techniques especially designed for my kind of problems?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="28937" LastActivityDate="2018-03-05T16:27:59.510" Title="Determining Important Atrributes with Feature Selection" Tags="|r|feature-selection|correlation|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19530" PostTypeId="1" AcceptedAnswerId="19551" CreationDate="2017-06-07T12:26:49.823" Score="1" ViewCount="1550" Body="&lt;p&gt;I am unable to understand what is the adjusted term in ARI. The expected index term in the ARI is from a prior. Kindly explain  &lt;/p&gt;&#xA;" OwnerUserId="31599" LastActivityDate="2017-06-08T07:52:29.760" Title="Difference between rand index and adjusted rand index?" Tags="|machine-learning|r|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19531" PostTypeId="2" ParentId="19527" CreationDate="2017-06-07T12:29:09.943" Score="2" Body="&lt;p&gt;Sounds like pretty orthodox feature importance analysis. Easy option:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.rdocumentation.org/packages/caret/versions/6.0-76/topics/featurePlot&quot; rel=&quot;nofollow noreferrer&quot;&gt;FeaturePlot&lt;/a&gt; from caret&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It basically creates a correlation matrix for you to see which seem to separate most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Slightly fancier:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.rdocumentation.org/packages/randomForest/versions/4.6-12/topics/varImpPlot&quot; rel=&quot;nofollow noreferrer&quot;&gt;VarImpPlot&lt;/a&gt; from randomForest&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This uses the random forest ML technique to directly tell you which features were most important. It sounds very similar to what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&#xA;&lt;a href=&quot;https://discuss.analyticsvidhya.com/t/how-to-extract-important-variables-from-random-forest-model-using-varimpplot-in-r/1325&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a more detailed discussion of VarImpPlot.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-07T12:29:09.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19532" PostTypeId="2" ParentId="19523" CreationDate="2017-06-07T13:43:11.543" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;there are no product ratings available, thus collaborative filtering&#xA;  is not an option&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Wrong. You can do collaborative filtering with holdings. Just use the numbers/duration of holdings instead of ratings.    &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;That's why I want to develop recommendations on similarities between&#xA;  the users based on their purchase history and/or demographics&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then any content-based approach should be fine. I can thing of a good article called &lt;a href=&quot;http://ai2-s2-pdfs.s3.amazonaws.com/21e3/b91de1c0ccfb1b8570ccb1cd3094018b31b7.pdf&quot; rel=&quot;noreferrer&quot;&gt;TrustWalker&lt;/a&gt; using trust between users (you create links between similar users and propagate their tastes in the network).  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-06-07T13:43:11.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19536" PostTypeId="1" CreationDate="2017-06-07T19:16:25.427" Score="1" ViewCount="169" Body="&lt;p&gt;I am trying to fit a regression model to predict the revenue generated from the sales of a particular product. I have found out seasonality both in the outcome(sales revenue) as well as in the input variables. &lt;br/&gt;&lt;br/&gt;&#xA;Sales Revenue&#xA;&lt;a href=&quot;https://i.stack.imgur.com/scrzP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/scrzP.png&quot; alt=&quot;Revenue&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 'Seasonally Adjusted Annual Rate' and the 'Non Seasonally Adjusted' version of each of the input variables are available. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What version should be used to training the model ?  &lt;/li&gt;&#xA;&lt;li&gt;Should 'month' be included as a categorical variable as a means to capture&#xA;seasonality ?&lt;/li&gt;&#xA;&lt;li&gt;Should the seasonality be removed from revenue if the Seasonally Adjusted version of inputs are used? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30492" LastActivityDate="2017-06-08T08:03:47.697" Title="Regression for forecasting" Tags="|time-series|regression|forecast|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19537" PostTypeId="2" ParentId="19536" CreationDate="2017-06-07T19:20:29.657" Score="1" Body="&lt;p&gt;If the goal is to predict &lt;em&gt;actual&lt;/em&gt; revenue, you want to predict the unadjusted rate. You should include month as a variable so your model has a chance of getting the seasonality right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the goal was more performance monitoring (e.g. &quot;We did better in March than February&quot;), then you would want to be analyzing the adjusted rate. You see this all the time in the US news when they forecast adjusted home sales, car sales, or unemployment.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-07T19:20:29.657" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19538" PostTypeId="1" AcceptedAnswerId="19539" CreationDate="2017-06-07T20:41:44.300" Score="5" ViewCount="3240" Body="&lt;p&gt;Can anyone provide specific techniques with using ICD-10 codes in Machine Learning? I have usually used a simply approach of creating multiple binary column representing ICD-10 codes… which can get extremely long. Or I have used Hashing features. Are there other techniques or ways to use ICD-10 codes in ML? Anyone can provide a useful link to see various approaching to modeling such features?&lt;/p&gt;&#xA;" OwnerUserId="33158" LastActivityDate="2019-11-25T17:09:34.670" Title="ICD-10 codes in Machine Learning" Tags="|machine-learning|r|cross-validation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19539" PostTypeId="2" ParentId="19538" CreationDate="2017-06-07T21:16:53.230" Score="4" Body="&lt;p&gt;Most machine learning around ICD codes deals with auto-encoding documents or NLP to extract ICD codes automatically from documents. Once you have them, most applications I have seen use them as is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One simple alternative would be to assign the codes to relevant categories to reduce the number of levels and make them more interpretable. For example &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;injury types (e.g. F00–F99 for mental disorders)&lt;/li&gt;&#xA;&lt;li&gt;body parts (e.g. S10-S19 &amp;amp; .. for neck injuries)&lt;/li&gt;&#xA;&lt;li&gt;severity of injury etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;depending on your application. Of course, if you don't want to define the categories beforehand, this can be done using some clustering or embedding method etc. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2019-11-25T17:09:34.670" LastActivityDate="2019-11-25T17:09:34.670" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="19541" PostTypeId="1" CreationDate="2017-06-07T23:57:35.997" Score="3" ViewCount="315" Body="&lt;p&gt;I am extremely new to this data science world so bear with me if my question is not very clear, I'd be glad to clarify. What I am looking for is simple: train a program with a set of values (5 ordered integer inputs, 1 boolean output). Then I would give it 5 inputs where the outcome is not known, and it has to tell me the outcome. &lt;/p&gt;&#xA;" OwnerUserId="33168" LastEditorUserId="8432" LastEditDate="2017-06-08T01:17:13.930" LastActivityDate="2017-12-11T11:37:26.033" Title="Machine-learning algorithm / library for outcome prediction" Tags="|machine-learning|deep-learning|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19543" PostTypeId="1" AcceptedAnswerId="19544" CreationDate="2017-06-08T02:40:39.720" Score="0" ViewCount="64" Body="&lt;p&gt;A paper I read called &lt;em&gt;Preprocessing Techniques for Context Recognition from  Accelerometer Data&lt;/em&gt; refers to sample difference as the delta value between signals in a pairwise arrangement of samples that allows a basic comparison between the intensity of user activity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you do the pairwise arrangement? Would it require you to have different files of data representing different classes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have a CSV:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 1495573445.162, 0, 0.021973, 0.012283, -0.995468, 1&#xA; 1495573445.172, 0, 0.021072, 0.013779, -0.994308, 1&#xA; 1495573445.182, 0, 0.020157, 0.015717, -0.995575, 1&#xA; 1495573445.192, 0, 0.017883, 0.012756, -0.993927, 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the second, third, and fourth columns are the axes of accelerometer data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have several files named for one gesture and several others for another gesture and would like to use this sample difference statistic to help classify the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also as a secondary question, this was listed as a preprocessing technique but it sounds like it's more of a feature. Could I get clarification on that as well?&lt;/p&gt;&#xA;" OwnerUserId="33171" LastActivityDate="2017-06-08T03:52:06.663" Title="How do you calculate sample difference in terms of sensor signals?" Tags="|feature-selection|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19544" PostTypeId="2" ParentId="19543" CreationDate="2017-06-08T03:52:06.663" Score="0" Body="&lt;p&gt;Differencing is a common preprocessing step for time series. Here's an example in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame&#xA;data=[[1495573445.162, 0, 0.021973, 0.012283, -0.995468, 1],&#xA; [1495573445.172, 0, 0.021072, 0.013779, -0.994308, 1],&#xA; [1495573445.182, 0, 0.020157, 0.015717, -0.995575, 1],&#xA; [1495573445.192, 0, 0.017883, 0.012756, -0.993927, 1]]&#xA;df = DataFrame(data, columns=['timestamp', 'foo', 'x', 'y', 'z', 'bar']).set_index('timestamp')&#xA;df.assign(dx=df.x.diff(), dy=df.y.diff(), dz=df.z.diff())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;               foo         x           y            z   bar        dx         dy          dz&#xA;timestamp                               &#xA;1.495573e+09    0   0.021973    0.012283    -0.995468   1         NaN        NaN         NaN&#xA;1.495573e+09    0   0.021072    0.013779    -0.994308   1   -0.000901   0.001496    0.001160&#xA;1.495573e+09    0   0.020157    0.015717    -0.995575   1   -0.000915   0.001938   -0.001267&#xA;1.495573e+09    0   0.017883    0.012756    -0.993927   1   -0.002274  -0.002961    0.001648&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-06-08T03:52:06.663" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19545" PostTypeId="1" AcceptedAnswerId="19547" CreationDate="2017-06-08T04:31:04.110" Score="1" ViewCount="1646" Body="&lt;p&gt;I was reading through many blogs and understood the relevance and scenario of having merging two model. But, how to decide the Merge mode for two models. ie. concat, sum, dot, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For eg. I am working on the task of Auto Image captioning. So, captions and Images are 2 kinds of input that I need to handle and merge them at certain point for model to let know which caption is for which image. I am learning the text representation and Image representation by 2 different network designs. Now, at the state after learning representation for both the inputs, how do i decide what ways(concat, add, etc) to use to join/merge two representations.&lt;/p&gt;&#xA;" OwnerUserId="33175" LastEditorUserId="33175" LastEditDate="2017-06-13T04:51:19.040" LastActivityDate="2017-06-13T04:51:19.040" Title="Merge two models - Keras" Tags="|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="1" ClosedDate="2017-06-09T11:12:04.803" ContentLicense="CC BY-SA 3.0" />
  <row Id="19546" PostTypeId="1" CreationDate="2017-06-08T05:51:28.763" Score="0" ViewCount="137" Body="&lt;p&gt;Example:&lt;br&gt;&#xA;Given n number of images marked 1 to n where n is unknown, I can calculate a property of every image which is a scalar quantity. Now I have to represent this property of all images in a fixed size vector (say 5 or 10).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One naive approach can be this vector- [avg max min std_deviation]&lt;br&gt;&#xA;And I also want to include the effect of relative positions of those images.&lt;/p&gt;&#xA;" OwnerUserId="33176" LastActivityDate="2017-09-06T13:30:29.240" Title="How to reduce an unknown size data into a fixed size data? Please read details" Tags="|dataset|statistics|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19547" PostTypeId="2" ParentId="19545" CreationDate="2017-06-08T05:59:33.673" Score="0" Body="&lt;p&gt;Generally, I'd say its the same way as finding your &quot;best&quot; feature list, pre-processing methods and algorithm - you'd find most answers starting with &quot;give it a try and compare&quot;, &quot;check which combination gives you a better cross-validation?&quot; etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, when there is no farther documentation (and its an open source project) I usually take a look at the code for more intuitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case concat, dot, average and the like where strait forward implemented as it sounds (concatenating date, averaging, etc..), but it does imply about the amount of processing needed: some of the merging functions &lt;strong&gt;adds&lt;/strong&gt; to the input needed to be processed (concat, add) and some &lt;strong&gt;reduces&lt;/strong&gt; it like an aggregation function (average, min, max)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond that the resulted tensor object will perform a bit different for different datasets. Please give it a try and let us know what merging combination worked best for you, and the dataset type :)&lt;/p&gt;&#xA;" OwnerUserId="17177" LastActivityDate="2017-06-08T05:59:33.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19548" PostTypeId="2" ParentId="19546" CreationDate="2017-06-08T06:23:19.413" Score="0" Body="&lt;p&gt;It looks like you're looking for a smart aggregation function, which aggregate &lt;strong&gt;n folded rows&lt;/strong&gt; instead of the usual 1 (out of the many falling to the group-by criteria).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the aggregation function you suggested like min and max can be adjusted as the &lt;strong&gt;n-smallest&lt;/strong&gt; or the &lt;strong&gt;n-biggest&lt;/strong&gt; values. Alternatively, the min/max n-values can be a skip list of the sorted value (ascending for min, descending for max), choosing every &lt;strong&gt;n/5-th item&lt;/strong&gt; value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a wider approach, we can borrow the concept from &lt;strong&gt;time-series&lt;/strong&gt;, where at each time interval a set of parameters are sampled. Instead of the built in time-stamp buckets (say every 1 sec, 10 min, etc..) you can use your images index, thus preserving the order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As with time-series, that can be farther more reduced into smaller time-series (say from a 1 sec interval into 1 day interval), write a custom group-by function that &lt;strong&gt;&quot;folds&quot; every n/5 image-rows into 1 row&lt;/strong&gt; (so that it'll leave you with 5) and apply your min/max/custom aggregation function on every n/5 values of your magic-image-property&lt;/p&gt;&#xA;" OwnerUserId="17177" LastActivityDate="2017-06-08T06:23:19.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19549" PostTypeId="2" ParentId="19493" CreationDate="2017-06-08T06:35:17.917" Score="3" Body="&lt;p&gt;Also consider using the builtin &lt;code&gt;ks.test&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ks.test(prd, act, alternative='greater')&#xA;&#xA;        Two-sample Kolmogorov-Smirnov test&#xA;&#xA;data:  prd and act&#xA;D^+ = 0.494, p-value &amp;lt; 2.2204e-16&#xA;alternative hypothesis: the CDF of x lies above that of y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4881" LastActivityDate="2017-06-08T06:35:17.917" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19550" PostTypeId="2" ParentId="16762" CreationDate="2017-06-08T07:11:32.357" Score="2" Body="&lt;p&gt;Coming from the related field of measuring and predicting network security, I'd strongly suggest trying a time-series forecasting. I assume your data is time-stamp based (network congestion values, sampled at some interval? if not skip to the second idea)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1st idea:&lt;/strong&gt;&#xA;I'd borrow from time-series the concept of flattening the 100 measurements into 1 datum.&#xA;So instead of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[t1+delta1, location1, messurement1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[t1+delta2, location2, messurement2]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[t1+delta3, location3, messurement3]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fold into:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[t1-bucketed,  loc1, mess1, loc2, mess2, loc3, mess3.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would help the model &quot;grasp&quot; the relation between the different measurements, with emphasis on the time axis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2nd idea&lt;/strong&gt;&#xA;Flatten into each measurement row the closest measurements by topology (or even the entire 100 neighbors) into:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[mess, topol-1-mess, topol-1-dist, topol-1-other, topol-2-mess, topol-2-dist, topol-2-other, ..]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would help the model &quot;grasp&quot; the relation between a specific measurement and its neighborhood measurements, with emphasis on the topology features from each measurement&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let us know if that helped :)&lt;/p&gt;&#xA;" OwnerUserId="17177" LastActivityDate="2017-06-08T07:11:32.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19551" PostTypeId="2" ParentId="19530" CreationDate="2017-06-08T07:52:29.760" Score="3" Body="&lt;p&gt;The adjustment is simply&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Rand index - Expected value)/(Optimal value - Expected value)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose is to scale it in an &lt;em&gt;interpretable&lt;/em&gt; way. 0 is &quot;as good as random&quot;, less than 0 is worse, and close to 1 is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with the non adjusted Rand index is that a random result on certain data sets can achieve a high score otherwise.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-06-08T07:52:29.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19552" PostTypeId="2" ParentId="19536" CreationDate="2017-06-08T08:03:47.697" Score="0" Body="&lt;p&gt;Maybe its the wording you are using, but this is not in fact regression task, but job for time series. Your data (and time series in general) are correlated with itself, hence invalidating assumption of regression (independency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I'd suggest to step back and rethink the approach. Try to fit simple ARMA model and see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time series will elegantly deal with your questions since you will be able to take the trends into account and actually model them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use regression, but then the data must be stationary (thus fit regression on differences). However, I would advice to use the reqular time-series approach.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-06-08T08:03:47.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19553" PostTypeId="1" CreationDate="2017-06-08T08:28:21.097" Score="1" ViewCount="1338" Body="&lt;p&gt;I have a highly unbalanced text classification data. I am trying to over-sample through SMOTE. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a doubt that applying SMOTE over sequence of word indices will give me valid data points or not (since SMOTE used nearest neighbor like method)?&lt;/p&gt;&#xA;" OwnerUserId="13518" LastEditorUserId="25849" LastEditDate="2020-10-03T05:38:38.377" LastActivityDate="2020-10-03T05:38:38.377" Title="Can SMOTE be applied over sequence of words (sentences)?" Tags="|class-imbalance|text-classification|smote|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19554" PostTypeId="2" ParentId="15733" CreationDate="2017-06-08T10:22:21.717" Score="1" Body="&lt;p&gt;I've tried &lt;a href=&quot;https://www.tutorialspoint.com/opennlp/index.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; one, and is very straight forward and easy to follow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Java is very similar to python in some sense so it shouldn't be a problem for you if you have done OO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get deeper inside natural language understanding and processing I recommend &lt;a href=&quot;https://link.springer.com/book/10.1007/978-3-319-32967-3&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; book.&lt;/p&gt;&#xA;" OwnerUserId="26700" LastActivityDate="2017-06-08T10:22:21.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19555" PostTypeId="1" CreationDate="2017-06-08T10:52:24.790" Score="2" ViewCount="416" Body="&lt;p&gt;I am currently studying the online material of Stanford CS229 and I came across the likelihood function for discriminative(for example, regression) and generative algorithms(for example, naive bayes):&#xA;+Discriminative:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Bcl8y.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Bcl8y.gif&quot; alt=&quot;Discriminative likelihood function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;+Generative:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GwAOU.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GwAOU.gif&quot; alt=&quot;Generative likelihood function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases, m is the number of training examples and all training examples are independent of each other. What I am wondering is why in discriminative likelihood function, the formula is the product of conditional probability of y given x and in generative likelihood function, the formula is the product of joint probability of x and y? Is there some reasoning behind this choice?&lt;/p&gt;&#xA;" OwnerUserId="33083" LastEditorUserId="33083" LastEditDate="2017-06-08T18:14:46.060" LastActivityDate="2018-02-03T22:37:24.560" Title="intuition behind the difference between likelihood function of discriminative and generative algorithms" Tags="|logistic-regression|probability|naive-bayes-classifier|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19557" PostTypeId="1" AcceptedAnswerId="19564" CreationDate="2017-06-08T11:15:58.780" Score="0" ViewCount="193" Body="&lt;p&gt;I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of Raman spectra data and and most of them differ between both classes even to the naked eye (characteristic peaks or the lack of).&#xA;While I do have many datasets I know were not recorded under exact same conditions, I also have a few sets for which exact same conditions apply for both classes. They still look obviously different from each other. Since my primary goal is to develop a robust algorithm with an unsupervised classificator, I now start to wonder: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Assuming the setup and thus noise and any sort of distortion is the same in both classes, even if dominant, do I need to bother with pre-processing when various classifiation models already return satisfying results with the raw data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most (if not all) papers I read about Raman spectra pre-processing (there's a lot on that topic) gave me the impression, the authors are always interested in separating the Raman spectrum of interest from all the unwanted other signals (fluorescent background, white noise, cosmic spikes etc). While this certainly would be interesting, I actually only want to sort them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking this question from a rather hypothetical standpoint. I know that in reality you can hardly guarantee exact same conditions (especially with random occurrences like cosmic spikes).&lt;/p&gt;&#xA;" OwnerUserId="14657" LastActivityDate="2017-06-08T14:38:37.010" Title="Is pre-processing always neccessary?" Tags="|data-cleaning|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19558" PostTypeId="1" CreationDate="2017-06-08T12:01:17.580" Score="1" ViewCount="76" Body="&lt;p&gt;I have time series for which I used autocorrelation property and made autoregression model for prediction, with different time delays 0-30. I got diagram which is dependency MSE on time delay 0-30 &lt;a href=&quot;https://i.stack.imgur.com/JK41F.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://i.stack.imgur.com/JK41F.jpg&lt;/a&gt; and I don't know how to explain this discontinuity with the first dot for which delay is 0 and MSE is equal 0?&lt;/p&gt;&#xA;" OwnerUserId="27460" LastActivityDate="2018-05-04T18:34:43.160" Title="MSE and time delay relation" Tags="|predictive-modeling|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19559" PostTypeId="2" ParentId="19558" CreationDate="2017-06-08T12:14:21.613" Score="1" Body="&lt;p&gt;The zero time delay seems wrong there. Suppose you have an autoregressive model. A general formula could be:&#xA;$$y_t = \alpha + \sum_{i}\beta_i y_{t-i} + w_t$$&#xA;Now, if $i$ is zero then you will have&#xA;$$y_t = \alpha +\beta_0 y_t + w_t$$&#xA;which is really weird. It looks like you predict values at time $t$ from the values at time $t$. In plain English is like you predict the future values from the real values. In that case the error is zero, since it is not very hard to predict something from the real values. So, I think the offsets should start at $1$. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2017-06-08T12:14:21.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19560" PostTypeId="1" CreationDate="2017-06-08T13:07:09.133" Score="3" ViewCount="397" Body="&lt;p&gt;Is there such a global flight network dataset that gives you:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;flight route (the connection between each airports or cities by some flight company)&lt;/li&gt;&#xA;&lt;li&gt;the number of passengers (per year) taking each route.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In other words the dataset is a directed flight graph with passenger volume as the edge weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest thing I can find is &lt;a href=&quot;https://openflights.org/data.html#route&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenFlights&lt;/a&gt; route data.&lt;/p&gt;&#xA;" OwnerUserId="33194" LastActivityDate="2017-06-08T13:07:09.133" Title="global flight network (with number of pessengers per year) dataset" Tags="|dataset|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19562" PostTypeId="1" AcceptedAnswerId="19572" CreationDate="2017-06-08T13:38:03.513" Score="1" ViewCount="517" Body="&lt;p&gt;I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to learn a sequence of numbers (our input) from 1 to 16:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;        \begin{bmatrix}&#xA;        1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8 &amp;amp; 9 &amp;amp; 10 &amp;amp; 11 &amp;amp; 12 &amp;amp; 13 &amp;amp; 14 &amp;amp; 15 &amp;amp; 16&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Batches: 2, Sequence Length: 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, we can divide the data to 2 batches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;        \begin{bmatrix}&#xA;        1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8\\&#xA;        9 &amp;amp; 10 &amp;amp; 11 &amp;amp; 12 &amp;amp; 13 &amp;amp; 14 &amp;amp; 15 &amp;amp; 16&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we can divide this into mini batches:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;        \begin{bmatrix}&#xA;        1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4\\&#xA;        9 &amp;amp; 10 &amp;amp; 11 &amp;amp; 12&#xA;        \end{bmatrix}&#xA;$$&#xA;$$&#xA;        \begin{bmatrix}&#xA;        5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 8\\&#xA;        13 &amp;amp; 14 &amp;amp; 15 &amp;amp; 16&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;        \begin{bmatrix}&#xA;        2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5\\&#xA;        10 &amp;amp; 11 &amp;amp; 12 &amp;amp; 13&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;        \begin{bmatrix}&#xA;        2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 1\\&#xA;        10 &amp;amp; 11 &amp;amp; 12 &amp;amp; 9&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is the intuition in doing so? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="24119" LastEditDate="2017-06-08T20:42:02.247" LastActivityDate="2017-06-08T22:45:53.333" Title="Sequence Batching in RNNs" Tags="|rnn|sequence|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19563" PostTypeId="2" ParentId="19555" CreationDate="2017-06-08T13:52:21.287" Score="2" Body="&lt;p&gt;Discriminative model learn to classify an x into class y: the conditional probability distribution p(y|x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generative models learn the joint probability distribution p(x,y).It can be transformed into p(y|x) by a Bayes rule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You get different likelihood functions because the way those models classify data is different.&lt;/p&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-06-08T13:52:21.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19564" PostTypeId="2" ParentId="19557" CreationDate="2017-06-08T14:38:37.010" Score="1" Body="&lt;p&gt;[Not specific for Raman spectroscopy]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a trend in some areas of machine learning to aim at &lt;strong&gt;end-to-end learning&lt;/strong&gt;, that is, devise machine learning algorithms that take raw input data and gives the desired output (e.g. classification label, regression values). This way, end-to-end learning avoids introducing expert knowledge or preprocessing, usually relying only on huge amounts of data. This has been specially true in the last years in the field of &lt;strong&gt;deep learning&lt;/strong&gt;, where images are taken with no preprocessing, audio is not &lt;em&gt;filter-banked&lt;/em&gt;, and games are played by receiving only raw pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is certainly &lt;strong&gt;possible&lt;/strong&gt; to skip data preprocessing completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We should nevertheless take into account that most of the successful end-to-end machine learning approaches impose constraints that enable the learning to exploit the very aspects of the input data that were previously defined by experts. An example for this are Convolutional Networks, that impose a &lt;em&gt;feature locality&lt;/em&gt; constraint that enables the learning to identify features, which could have been pre-defined as &lt;em&gt;gabor filters&lt;/em&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BUT, in more traditional data science, &lt;strong&gt;feature engineering&lt;/strong&gt; is a key part of the process of devising ML systems and data exploration and expert knowledge are normally used to identify the kind of preprocessing to be applied to data.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-06-08T14:38:37.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19565" PostTypeId="1" CreationDate="2017-06-08T15:07:48.850" Score="1" ViewCount="180" Body="&lt;p&gt;I'm looking at the correlation between day of week and number of page views my website gets.  Because day of week is a discrete, non-numeric variable, I've mapped Sunday to 0, Monday to 1, ..., Saturday to 6.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my concern / question: Isn't it possible to get a low Pearson coefficient when, in fact, there is a relationship between day of week and page views, it's just a bit more complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, it might be that Mondays and Fridays are highly correlated with page views and the middle of the week is negatively correlated.  So this might average out to a correlation near 0 when there's a real pattern there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way of trying to figure out what's going on here?  My mapping of days to numbers seems simplistic...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="33199" LastActivityDate="2017-06-08T15:07:48.850" Title="Misleading Pearson Correlation Coefficient?" Tags="|correlation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19567" PostTypeId="2" ParentId="12775" CreationDate="2017-06-08T18:50:31.890" Score="3" Body="&lt;p&gt;Just take a dataset that has a large number of features and ignore as much data as you need to get the ratio you desire. E.g. take imagenet, but randomly select 100 samples per class.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2017-06-08T18:50:31.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19568" PostTypeId="2" ParentId="19322" CreationDate="2017-06-08T20:57:09.837" Score="1" Body="&lt;p&gt;It is my understanding that the dataset gets updated / extended every year. It started with Phoenix, then they added Las Vegas, Madison, Waterloo and Edinburgh, and so on, for a total of 11 cities by 2017. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, this year's dataset should also include all cities from previous years.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-06-08T20:57:09.837" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19569" PostTypeId="1" CreationDate="2017-06-08T21:16:55.867" Score="1" ViewCount="50" Body="&lt;p&gt;I'm looking for an algorithm, that classifies a webpage. But not the content. I know the readability algorithm (by arc90) to extract a main text from an webpage. That for instance can decide, wether a webpage contains an article (is a news or blog article). Thus it can decide wether a webpage is an article page or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I want to go further and classify, wether a non-article-webpage is a product page or a landingpage promoting a product or (!) wether a page has some purpose to motivate the visitor to take some action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third case has to be anything else. Whatever: It could be an overview site or else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know such a classifier or has any idea how to analyse the DOM-tree of a webpage in terms of similarity of striking container constellations?&lt;/p&gt;&#xA;" OwnerUserId="33047" LastEditorUserId="381" LastEditDate="2017-06-08T21:36:11.037" LastActivityDate="2017-06-08T21:36:11.037" Title="Analyzing Web page structure" Tags="|machine-learning|classification|similarity|information-retrieval|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19570" PostTypeId="2" ParentId="16762" CreationDate="2017-06-08T21:23:29.977" Score="1" Body="&lt;p&gt;A simple approach would be to use &lt;strong&gt;k-nearest-neighbors&lt;/strong&gt;, where the distance metric is, in your case, &quot;the number of road links away.&quot; The technique is described in chapters 2 and 13 of &lt;a href=&quot;https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;The Elements of Statistical Learning&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;. Basically it would take the average traffic volume for the &lt;code&gt;k&lt;/code&gt; nearest traffic nodes. There is essentially no training involved, other than cross validation and tuning to find the optimal &lt;code&gt;k&lt;/code&gt;. The trade-off is that it's computationally heavy at query time (when you want to make a prediction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note: If you aren't already, I highly recommend using a database like &lt;a href=&quot;https://neo4j.com/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Neo4j&quot;&gt;Neo4j&lt;/a&gt; to make querying the link distance much easier. You could probably code the KNN &quot;model&quot; yourself using just 3-4 lines of code if you use a graph database. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-06-08T21:23:29.977" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19571" PostTypeId="1" CreationDate="2017-06-08T21:57:16.490" Score="2" ViewCount="8029" Body="&lt;p&gt;I am using an imbalanced dataset (rare positive cases) to learn models for prediction and the final good AUC is 0.92 but the F1 score is very low0.2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to add some key features which will change the class probabilistic distribution and thus we can get a threshold to generate a higher F1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my original model, I get a class probabilistic distributions shown below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aUkaN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aUkaN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can adjust the threshold to make a better precision but meanwhile cut some recall off. It is due to the large overlapping area between two distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I use an extreme dataset, i.e. include the target itself as a feature to learn. As a result, I can see I split the distribution completely disjointed. &lt;a href=&quot;https://i.stack.imgur.com/nayVc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nayVc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dose it mean if I introduce a strong feature, I can split the distribution to some extent and thus promote the precision and thus f1 score? Or please advise how to improve precision under imbalanced classification issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="1330" LastEditDate="2017-12-07T03:07:08.320" LastActivityDate="2017-12-07T03:07:08.320" Title="How to improve precision under imbalanced classification" Tags="|machine-learning|predictive-modeling|anomaly-detection|class-imbalance|anomaly|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19572" PostTypeId="2" ParentId="19562" CreationDate="2017-06-08T22:45:53.333" Score="2" Body="&lt;p&gt;It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen the input &lt;em&gt;reversed&lt;/em&gt;, which is a less arbitrary transformation than the one you cite, to improve prediction in &lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;sequence-to-sequence models&lt;/a&gt;, though that's not set in stone, either.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-06-08T22:45:53.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19575" PostTypeId="1" AcceptedAnswerId="28434" CreationDate="2017-06-09T04:03:45.400" Score="4" ViewCount="15009" Body="&lt;p&gt;I have a data set given as follows:&#xA;target shape &lt;code&gt;(200000, 1)&lt;/code&gt;&#xA;train_data.shape &lt;code&gt;(200000, 48)&lt;/code&gt;&#xA;test_data.shape&lt;code&gt;(100000, 48)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;I had used the data to &lt;code&gt;predict_proba&lt;/code&gt; using &lt;code&gt;RandomForestClassifier&lt;/code&gt; &lt;code&gt;ExtraTreesClassifier&lt;/code&gt; &lt;code&gt;RandomForestClassifier&lt;/code&gt; &lt;code&gt;AdaBoostClassifier&lt;/code&gt; &lt;code&gt;GradientBoostingClassifier&lt;/code&gt; and &lt;code&gt;SVC&lt;/code&gt;&#xA;and got results without an error.&lt;/p&gt;&#xA;&lt;h1&gt;BUT&lt;/h1&gt;&#xA;&lt;p&gt;&lt;code&gt;clf = XGBClassifier()&lt;/code&gt; is showing following errors (with the same code except &lt;code&gt;clf&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;ValueError: feature_names mismatch:&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-01T02:12:40.393" Title="XGBClassifier error! ValueError: feature_names mismatch:" Tags="|machine-learning|python|xgboost|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19576" PostTypeId="2" ParentId="19575" CreationDate="2017-06-09T04:21:56.707" Score="2" Body="&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;train_data = np.array(train_data)&#xA;test_data = np.array(test_data)&#xA;clf = XGBClassifier()&#xA;clf.fit(train,y_train.ravel())&#xA;pred = clf.predict_proba(test_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use this instead of your original code.&lt;/p&gt;&#xA;" OwnerUserId="32686" LastEditorUserId="32686" LastEditDate="2017-06-09T05:37:01.857" LastActivityDate="2017-06-09T05:37:01.857" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="19577" PostTypeId="1" CreationDate="2017-06-09T05:12:48.650" Score="1" ViewCount="1548" Body="&lt;p&gt;Is there any ML framework that readily supports LSTM for multi-dimentionsal data ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best would be in Pytorch. But I don't think the official API supports it and I have found an implementation.&lt;/p&gt;&#xA;" OwnerUserId="31179" LastActivityDate="2018-10-08T12:35:44.250" Title="LSTM for multidimensional data" Tags="|rnn|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19578" PostTypeId="1" AcceptedAnswerId="19581" CreationDate="2017-06-09T05:16:41.053" Score="9" ViewCount="43381" Body="&lt;p&gt;I used MSE loss function, SGD optimization:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xtrain = data.reshape(21168, 21, 21, 21,1)    &#xA;inp = Input(shape=(21, 21, 21,1))&#xA;x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(inp)&#xA;x = MaxPool3D(pool_size=(3, 3, 3),padding='same')(x)&#xA;x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;encoded = Conv3D(filters=128, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;&#xA;print (&quot;shape of decoded&quot;, K.int_shape(encoded))&#xA;&#xA;x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(encoded)&#xA;x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)&#xA;x = UpSampling3D((3, 3, 3))(x)&#xA;&#xA;decoded = Conv3D(filters=1, kernel_size=(3, 3, 3), activation='relu', &#xA;padding='same')(x)&#xA;&#xA;print (&quot;shape of decoded&quot;, K.int_shape(decoded))&#xA;&#xA;autoencoder = Model(inp, decoded)&#xA;autoencoder.compile(optimizer='sgd', loss='mse')&#xA;autoencoder.fit(xtrain, xtrain,&#xA;                epochs=30,&#xA;                batch_size=32,&#xA;                shuffle=True,&#xA;                validation_split=0.2&#xA;                )  &#xA;&#xA;Epoch 1/30&#xA;16934/16934 [==============================] - 446s - loss: 34552663732314849715                                                                                                                     15904.0000 - val_loss: 1893.9425&#xA;Epoch 2/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 3/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 4/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 5/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 6/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 7/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 8/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 9/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 10/30&#xA;16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 11/30&#xA;16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 12/30&#xA;16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 13/30&#xA;16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 14/30&#xA;16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;Epoch 15/30&#xA;16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32908" LastEditorUserId="381" LastEditDate="2017-06-09T05:46:08.307" LastActivityDate="2019-10-16T08:34:59.717" Title="Why my training and validation loss is not changing?" Tags="|machine-learning|deep-learning|autoencoder|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19579" PostTypeId="2" ParentId="19575" CreationDate="2017-06-09T05:40:39.533" Score="2" Body="&lt;p&gt;i used this to resolve the issue.&#xA;as the order of the columns in dataframe were not same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_data.sort_index(axis=1, inplace=True)&#xA;test_data.sort_index(axis=1,inplace=True)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="33020" LastEditDate="2017-06-09T05:59:12.037" LastActivityDate="2017-06-09T05:59:12.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19581" PostTypeId="2" ParentId="19578" CreationDate="2017-06-09T06:58:19.060" Score="14" Body="&lt;p&gt;Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out most of the ReLU activations. This is known as the &lt;a href=&quot;https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks&quot;&gt;dying ReLU problem&lt;/a&gt;, although the issue here is not necessarily the choice of ReLU, you will probably get similar problems with other activations in the hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to tone down some of the numbers that might be causing such a large initial loss, and maybe also make the weight updates smaller:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Normalise your input data. The autoencoder is trying to match the input, and if the numbers are large here, this multiplies up to a large loss. If the input can have negative values (either naturally or due to the normalisation) then you should not have ReLU activation in the output layer otherwise it is not possible for the autoencoder to match the input and output values - in that case just have a linear output layer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reduce the learning rate - in Keras SGD has default &lt;code&gt;lr=0.01&lt;/code&gt;, try lower e.g. &lt;code&gt;lr=0.0001&lt;/code&gt;. Also consider a more sophisticated optimiser than plain SGD, maybe Adam, Adagrad or RMSProp.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Add some conservative weight initialisations. In Keras you can set the weight initialiser - see &lt;a href=&quot;https://keras.io/initializers/&quot; rel=&quot;noreferrer&quot;&gt;https://keras.io/initializers/&lt;/a&gt; - however, the default &lt;code&gt;glorot_uniform&lt;/code&gt; should already be OK in your case, so maybe you will not need to do this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-09T07:11:54.347" LastActivityDate="2017-06-09T07:11:54.347" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19583" PostTypeId="1" AcceptedAnswerId="19703" CreationDate="2017-06-09T10:43:47.313" Score="0" ViewCount="898" Body="&lt;p&gt;It's a generic question on tuning hyper-parameters for &lt;code&gt;XGBClassifier()&lt;/code&gt;&#xA;I have used &lt;code&gt;gridsearch&lt;/code&gt; but as my training set is around &lt;code&gt;2,00,000&lt;/code&gt; it's taking huge time and heats up my laptop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need codes for efficiently tuning my classifier's parameters for &lt;strong&gt;best&lt;/strong&gt; performance.&lt;/p&gt;&#xA;" OwnerUserId="33020" LastActivityDate="2017-06-14T11:34:52.010" Title="How to tune XGBClassifier's each and every hyper-parameter &quot;efficiently&quot;?" Tags="|machine-learning|python|xgboost|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19584" PostTypeId="1" CreationDate="2017-06-09T10:48:19.590" Score="1" ViewCount="253" Body="&lt;p&gt;I have two lists to compare. &lt;code&gt;List 1&lt;/code&gt; contains strings which represent the strings in &lt;code&gt;List2&lt;/code&gt;. Sometimes, they are direct matches, some times, one word match, sometimes there are acronyms etc. An example is as below: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   List1           |  List2&#xA;  ------------------------&#xA;   great hall town | Great Hall&#xA;   grt_town        | Great Hall      &#xA;   gHall           | Great Hall&#xA;   great town      | Great Hall&#xA;   mind great town | Great Hall&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are two main problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;List2&lt;/code&gt; is very large ( millions of records, all short in length)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;List1&lt;/code&gt; records are very unrelated sometimes. There are longer prefixes, suffixes which are not in &lt;code&gt;List2&lt;/code&gt;. Some words in &lt;code&gt;List2&lt;/code&gt; are reduced to a letter. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I vaguely have some thoughts around clustering &lt;code&gt;List2&lt;/code&gt; and trying to find distance between centroids and the elements from &lt;code&gt;List1&lt;/code&gt;. Do you think this as a starting point?&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="6514" LastActivityDate="2017-06-10T06:18:25.453" Title="What could be a right datascience approach of doing fuzzy string matching in large amount of short text data?" Tags="|clustering|bigdata|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19586" PostTypeId="1" AcceptedAnswerId="19590" CreationDate="2017-06-09T12:06:36.027" Score="1" ViewCount="11174" Body="&lt;p&gt;I am totally new to the topic of Data Science. With the help of the following sources, I &lt;em&gt;think&lt;/em&gt; I have managed to do a very &lt;strong&gt;simple and basic Linear regression&lt;/strong&gt; on a &lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques/download/train.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;train dataset&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SkLearn documentation - Linear regression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/eliekawerk/regression-to-predict-house-prices&quot; rel=&quot;nofollow noreferrer&quot;&gt;Some Kernel, that I percieved as intuitive&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques/download/test.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;the test dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My &lt;strong&gt;Python code&lt;/strong&gt; (written as an iPython notebook) that actually does the computation looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;### Stage 0: &quot;Import some stuff&quot;&#xA;%matplotlib inline&#xA;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn import datasets, linear_model&#xA;from sklearn.linear_model import LinearRegression&#xA;&#xA;### Stage 1: &quot;Prepare train dataset&quot;&#xA;my_train_dataset = pd.read_csv(&quot;../train.csv&quot;)&#xA;&#xA;### remove categorical cols&#xA;only_numerical_train_dataset = my_train_dataset.loc[:, my_train_dataset.dtypes!=object]&#xA;&#xA;### remove 'Id' and 'SalePrice' columns&#xA;my_train_dataset_X = only_numerical_train_dataset.drop(['Id','SalePrice'], axis = 1)&#xA;&#xA;### insert median into cells with missing values&#xA;print(&quot;Before: Number of cells with missing values in train data: &quot; + str(np.sum(np.sum(my_train_dataset_X.isnull()))))&#xA;null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)&#xA;cols_to_impute = []&#xA;for key in null_values_per_col.keys():&#xA;    if null_values_per_col.get(key) != 0: &#xA;        cols_to_impute.append(key)&#xA;print(&quot;Before: Need to replace values in the columns in train data: &quot; + str(cols_to_impute) + &quot;\n&quot;)&#xA;imputation_val_for_na_cols = dict()&#xA;for col in cols_to_impute:&#xA;    if (my_train_dataset_X[col].dtype == 'float64' ) or  (my_train_dataset_X[col].dtype == 'int64'):&#xA;        #numerical col&#xA;        imputation_val_for_na_cols[col] = np.nanmedian(my_train_dataset_X[col]) #with median&#xA;for key, val in imputation_val_for_na_cols.items():&#xA;    my_train_dataset_X[key].fillna(value= val, inplace = True)&#xA;print(&quot;After: Number of cells with missing values in train data: &quot; + str(np.sum(np.sum(my_train_dataset_X.isnull()))))&#xA;null_values_per_col = np.sum(my_train_dataset_X.isnull(), axis=0)&#xA;cols_to_impute = []&#xA;for key in null_values_per_col.keys():&#xA;    if null_values_per_col.get(key) != 0: &#xA;        cols_to_impute.append(key)&#xA;print(&quot;After: Need to replace values in the columns in train data: &quot; + str(cols_to_impute) + &quot;\n&quot;)&#xA;&#xA;### Stage 2: &quot;Sanity Check - the better the quality, the higher the price?&quot;&#xA;plt.scatter(my_train_dataset.OverallQual, my_train_dataset.SalePrice)&#xA;plt.xlabel(&quot;Overall Quality of the house&quot;)&#xA;plt.ylabel(&quot;Price of the house&quot;)&#xA;plt.title(&quot;Relationship between Price and Quality&quot;)&#xA;plt.show()&#xA;&#xA;### Stage 3: &quot;Prepare the test dataset&quot;&#xA;my_test_dataset = pd.read_csv(&quot;../test.csv&quot;)&#xA;&#xA;### remove categorical cols&#xA;only_numerical_test_dataset = my_test_dataset.loc[:, my_test_dataset.dtypes!=object]&#xA;&#xA;### remove 'Id' column&#xA;my_test_dataset_X = only_numerical_test_dataset.drop(['Id'], axis = 1)&#xA;&#xA;### insert median into cells with missing values&#xA;print(&quot;Before: Number of cells with missing values in test data: &quot; + str(np.sum(np.sum(my_test_dataset_X.isnull()))))&#xA;null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)&#xA;cols_to_impute = []&#xA;for key in null_values_per_col.keys():&#xA;    if null_values_per_col.get(key) != 0: &#xA;        cols_to_impute.append(key)&#xA;print(&quot;Before: Need to replace values in the columns in test data: &quot; + str(cols_to_impute) + &quot;\n&quot;)&#xA;imputation_val_for_na_cols = dict()&#xA;for col in cols_to_impute:&#xA;    if (my_test_dataset_X[col].dtype == 'float64' ) or  (my_test_dataset_X[col].dtype == 'int64'):&#xA;        #numerical col&#xA;        imputation_val_for_na_cols[col] = np.nanmedian(my_test_dataset_X[col]) #with median&#xA;for key, val in imputation_val_for_na_cols.items():&#xA;    my_test_dataset_X[key].fillna(value= val, inplace = True)&#xA;print(&quot;After: Number of cells with missing values in test data: &quot; + str(np.sum(np.sum(my_test_dataset_X.isnull()))))&#xA;null_values_per_col = np.sum(my_test_dataset_X.isnull(), axis=0)&#xA;cols_to_impute = []&#xA;for key in null_values_per_col.keys():&#xA;    if null_values_per_col.get(key) != 0: &#xA;        cols_to_impute.append(key)&#xA;print(&quot;After: Need to replace values in the columns in test data: &quot; + str(cols_to_impute) + &quot;\n&quot;)&#xA;&#xA;### Stage 4: &quot;Apply the model&quot;&#xA;lm = LinearRegression()&#xA;lm.fit(my_train_dataset_X, my_train_dataset.SalePrice)&#xA;&#xA;### Stage 5: &quot;Sanity Check - the better the quality, the higher the predicted SalesPrice?&quot;&#xA;plt.scatter(my_test_dataset.OverallQual, lm.predict(my_test_dataset_X))&#xA;plt.xlabel(&quot;Overall Quality of the house in test data&quot;)&#xA;plt.ylabel(&quot;Price of the house in test data&quot;)&#xA;plt.title(&quot;Relationship between Price and Quality in test data&quot;)&#xA;plt.show()&#xA;&#xA;### Stage 6: &quot;Check the performance of the Prediction&quot;&#xA;from sklearn.model_selection import cross_val_score&#xA;scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)&#xA;print(&quot;scores = &quot; + str(scores))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;My questions are:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Why am I getting an error in Stage 6 and how to fix it?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError Traceback (most recent call last)&#xA;&amp;lt;ipython-input-2-700c31f0d410&amp;gt; in &amp;lt;module&amp;gt;()&#xA;     85 ### test the performance of the model&#xA;     86 from sklearn.model_selection import cross_val_score&#xA;---&amp;gt; 87 scores = cross_val_score(lm, my_train_dataset_X,  lm.predict(my_test_dataset_X), cv=10)&#xA;     88 print(&quot;scores = &quot; + str(scores))&#xA;     89 &#xA;ValueError: Found input variables with inconsistent numbers of samples: [1460, 1459]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Is there something &lt;em&gt;fundamentally wrong&lt;/em&gt; with my approach to a simple and basic Linear Regression?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Edits for comments:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;@CalZ - First comment:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_test_dataset_X.shape = (1459, 36)&#xA;my_train_dataset_X.shape = (1460, 36)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;@CalZ - Second comment:&lt;/strong&gt;&#xA;I will consider refactoring the code as soon as I am sure that my approach is not fundamentally wrong.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="33221" LastEditorUserId="33266" LastEditDate="2017-06-11T17:21:36.377" LastActivityDate="2017-06-11T17:21:36.377" Title="Linear Regression and k-fold cross validation" Tags="|python|scikit-learn|linear-regression|cross-validation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19587" PostTypeId="2" ParentId="19584" CreationDate="2017-06-09T12:25:16.143" Score="1" Body="&lt;p&gt;You could try this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take distinct records from List2&lt;/li&gt;&#xA;&lt;li&gt;Clean List1 of any extraneous characters or symbols&lt;/li&gt;&#xA;&lt;li&gt;Use a fuzzy logic package like Python's &lt;a href=&quot;https://github.com/seatgeek/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;fuzzywuzzy&lt;/a&gt;. It has partial matching options.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33226" LastEditorUserId="30212" LastEditDate="2017-06-10T06:18:25.453" LastActivityDate="2017-06-10T06:18:25.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19588" PostTypeId="2" ParentId="19571" CreationDate="2017-06-09T12:36:37.997" Score="1" Body="&lt;p&gt;Introducing a Strong Feature would definitely help as it is &quot;Strong&quot; :-) .. if you do not have a sure-success feature then you may try changing penalty of miss-classification to start with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may try synthetic (i.e. SMOTE) or non-synthetic (domain based) approaches to bulk up the lower class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if this is a very very rare class the repeated sampling techniques may work &lt;/p&gt;&#xA;" OwnerUserId="33226" LastActivityDate="2017-06-09T12:36:37.997" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19589" PostTypeId="1" CreationDate="2017-06-09T12:46:24.557" Score="3" ViewCount="186" Body="&lt;p&gt;I am aware of the existence of semi-supervised learning approaches, such as the &lt;a href=&quot;https://arxiv.org/abs/1507.02672&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ladder Network&lt;/a&gt;, where only a subset of the data is labeled. Are there any methods or papers which consider correctness probabilities for the labels of that training data subset? That is, some labels may be correct with 100% probability, while others may have only 70% or 45% probability of being correct. Any links to papers or work in this direction are highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="32854" LastActivityDate="2021-09-05T11:02:25.220" Title="Probability for label correctness in semi-supervised learning" Tags="|supervised-learning|unsupervised-learning|labels|semi-supervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19590" PostTypeId="2" ParentId="19586" CreationDate="2017-06-09T12:53:44.530" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;As the error message states, the invocation to &lt;code&gt;cross_val_score&lt;/code&gt; fails because the shape arguments differ in their first dimension (1460 vs. 1459). This is consistent with the number of lines in the CSV files. However, the underlying problem is that you are mixing the test and the training sets. You should invoke it only with the test set: &lt;code&gt;cross_val_score(lm, my_test_dataset_X,  lm.predict(my_test_dataset_X), cv=10)&lt;/code&gt;. &lt;strong&gt;Update&lt;/strong&gt;: My initial suggestion was NOT correct, you cannot use your own predictions to validate! You should leave a subset of the labeled data for hold out on which to compute the cross validation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yours is not only a linear regression. The bulk of your code is in charge of data manipulation (feature selection, data imputation) and not linear regression. Actually, you are reusing scikit-learn's implementation of linear regresion, not coding your own. If you want a code review of your snippet, maybe you should try in &lt;a href=&quot;http://codereview.stackexchange.com&quot;&gt;http://codereview.stackexchange.com&lt;/a&gt; (I don't know if this fits there either, you'd better check their &lt;a href=&quot;https://codereview.stackexchange.com/help&quot;&gt;help center&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;UPDATE: About whether your code is sound from a data science point of view, it seems to me (after only a quick review) that you are doing reasonable things. There are some things that could be improved, like only handling float64 and int64 (while you can do as described &lt;a href=&quot;https://stackoverflow.com/q/25039626/674487&quot;&gt;here&lt;/a&gt;), only imputing NaNs and Nones (while there can be other values that should be imputed in certain cases, like outliers),  or imputing blindly with the median (which is a &lt;em&gt;safe&lt;/em&gt; decision but should be assessed taking into account the nature of each variable). But generally speaking seems Ok.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-06-09T13:28:03.813" LastActivityDate="2017-06-09T13:28:03.813" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19591" PostTypeId="1" AcceptedAnswerId="19593" CreationDate="2017-06-09T14:06:43.857" Score="0" ViewCount="167" Body="&lt;p&gt;I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Assumed data type (numerical vs string)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming it's determined to be a numerical column:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean and Std Deviation&lt;/li&gt;&#xA;&lt;li&gt;Max and min values&lt;/li&gt;&#xA;&lt;li&gt;Number of NaN's&lt;/li&gt;&#xA;&lt;li&gt;% of NaN's&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assuming it's determined to be a string column:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Number of distinct string values&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So the tool takes the matrix and outputs some kind of report along these lines. The goal is to do some sanity checking on my input data set as well as checking data transformations along the way. For example, if I'm doing some kind of transformation and values in column A go from 1% NaN to 60% NaN, then I did something really bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does such a tool exist?&lt;/p&gt;&#xA;" OwnerUserId="33230" LastActivityDate="2017-06-09T14:26:24.240" Title="Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.)" Tags="|python|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19593" PostTypeId="2" ParentId="19591" CreationDate="2017-06-09T14:20:06.903" Score="1" Body="&lt;p&gt;You should take a look at pandas_profiling, I don't think it works with numpy arrays but it does exactly what you want for Pandas dataframes. It can output to PDF or to a nice looking HTML format within your Jupyter Notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/JosPolfliet/pandas-profiling&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/JosPolfliet/pandas-profiling&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-09T14:20:06.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19594" PostTypeId="2" ParentId="19591" CreationDate="2017-06-09T14:26:24.240" Score="2" Body="&lt;p&gt;You could use &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas.describe&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;df  = pd.DataFrame({'first': [1,2,3,4,np.NaN], 'second': [100,200,np.NaN, np.NaN,4]})&#xA;df.describe(include='all')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Regarding the percentage of NAs, you could use just a small formula for that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;count_nan = len(df) - df.count()&#xA;count_nan / df.shape[1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-06-09T14:26:24.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19595" PostTypeId="1" AcceptedAnswerId="19596" CreationDate="2017-06-09T14:38:52.130" Score="-1" ViewCount="166" Body="&lt;p&gt;I have some views per day of week data.  It's something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mondays: 100k views&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tuesday: 110k views&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wednesday: 140k views&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sunday: 80k views&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So naively, it seems like Wednesday is a better day for website traffic.  However, I also understand that it's possible these differences are attributable to variance (maybe each day has an equal probability of getting a given viewer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I determine that Wednesday (for instance) is a better day for website traffic in a statistically significant fashion?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happy to clarify if this question is unclear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="33199" LastActivityDate="2017-06-09T15:27:54.167" Title="Determining Statistically Significant Differences in Views per Day of Week?" Tags="|statistics|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19596" PostTypeId="2" ParentId="19595" CreationDate="2017-06-09T15:20:30.357" Score="1" Body="&lt;p&gt;First, you need a sample of the views data per day, that is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;views in Mondays: 120, 111, 90, 150, ..&#xA;views in Tuesdays: 50, 50, 57, 21, ..&#xA;...&#xA;views in Sundays: 201, 184, 126, 191, ..&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These are &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_variable&quot; rel=&quot;nofollow noreferrer&quot;&gt;random variables&lt;/a&gt;. Let's refer to them as $V_d$, where $d$ is the week day, e.g. $V_{monday}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With those data, one option is to perform a &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_hypothesis_testing&quot; rel=&quot;nofollow noreferrer&quot;&gt;hypothesis test&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the distribution of $V_d$, you may do different things. If they are normal (which can be also tested by means of a hypothesis test) (so that $V_d \sim \mathcal{N}(\mu_d, \sigma_d)$), you can compare the means of two days, having a &lt;a href=&quot;https://en.wikipedia.org/wiki/Null_hypothesis&quot; rel=&quot;nofollow noreferrer&quot;&gt;null hypothesis&lt;/a&gt; $H_0: \mu_{wednesday}  = \mu_{sunday}$ and the alternative hypothesis $H_1: \mu_{wednesday}  &amp;gt; \mu_{sunday}$ and choose an appropriate two sample test (depending on the variances of the distributions, sample size, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your distributions are not gaussian, you may use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kolmogorov-Smirnov test&lt;/a&gt; or a &lt;a href=&quot;https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mann-Whitney U test&lt;/a&gt;. To know how to choose between them, you can check &lt;a href=&quot;https://stats.stackexchange.com/questions/127027/kolmogorov-smirnov-vs-mann-whitney-u-when-there-are-ties&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-06-09T15:27:54.167" LastActivityDate="2017-06-09T15:27:54.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19600" PostTypeId="2" ParentId="19589" CreationDate="2017-06-09T16:21:50.223" Score="1" Body="&lt;p&gt;I do not know any papers, I would be greatly appreciated if someone would link some. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I always test by un-labelling my already known labelled Data by using a &quot;traditional&quot; 67% - 33% (train - test) split and checking how the labelling performs in various metrics (Accuracy, Logloss, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, there are various categories of Semi-Supervised Learning. For instance, if you would use Active Learning (pool based approach, includes incrementally sample/s each time), you could view how the performance ranges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you may use as well Cross-Validation in the same sense for hyper parameter tuning for your Semi-Supervised Learning algorithm. All in all, in my perspective Semi-Supervised Learning, has a lot yet to offer and your question may be rather more domain, data specific and visualsing the Data and Clustering may give you new insights.&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2017-06-09T16:21:50.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19602" PostTypeId="1" CreationDate="2017-06-09T16:38:38.900" Score="-1" ViewCount="62" Body="&lt;p&gt;I got data like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Student age&lt;/li&gt;&#xA;&lt;li&gt;Student study subject&lt;/li&gt;&#xA;&lt;li&gt;Student gender&lt;/li&gt;&#xA;&lt;li&gt;Student academic result&lt;/li&gt;&#xA;&lt;li&gt;Student joined actives&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and I got the student graduation income. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I would like to train a model, so that I can guest a student with some background and find back his / her projected income. If I would like to use machine learning to deal with this case, any ideas on how to implement? Please advise. &lt;/p&gt;&#xA;" OwnerUserId="33238" LastActivityDate="2017-06-09T16:51:47.023" Title="Which algorithm should I use for this use cases?" Tags="|supervised-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2017-06-10T08:16:04.123" ContentLicense="CC BY-SA 3.0" />
  <row Id="19603" PostTypeId="2" ParentId="19602" CreationDate="2017-06-09T16:44:02.513" Score="2" Body="&lt;p&gt;Features: &lt;em&gt;age&lt;/em&gt;, &lt;em&gt;gender&lt;/em&gt; and &lt;em&gt;academic result&lt;/em&gt;: take as numeric values&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;study subject&lt;/em&gt; : encode the subjects &lt;em&gt;like 0 for maths, 1 for physics etc..&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for &lt;em&gt;joined actives&lt;/em&gt; encode and use features like number of activities plus other columns as &lt;strong&gt;highly important&lt;/strong&gt; activities having values encoded as 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;code&gt;&#xA;age, gender, academic result, study subject, total_actives, debate, music``&#xA;24,  0,    , 90,            , 3            , 4            ,      1,     0&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Train your &lt;code&gt;XGBoost&lt;/code&gt; regressor using the above features and voila!&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="33020" LastEditDate="2017-06-09T16:51:47.023" LastActivityDate="2017-06-09T16:51:47.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19605" PostTypeId="1" AcceptedAnswerId="19657" CreationDate="2017-06-09T21:14:49.103" Score="6" ViewCount="2644" Body="&lt;p&gt;I'm trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an analogy, imagine I had survey data of superheros liked by thousands of school children, such that &lt;em&gt;Adam Agner&lt;/em&gt; only likes Batman, &lt;em&gt;Brian Agosto&lt;/em&gt; likes Superman and Batman, &lt;em&gt;Cian Ailiff&lt;/em&gt; likes Wonderwoman, Superman and the Flash, etc. The index would be the list of children's names (tens of thousands), and the variables would be each superhero (from a list of 50) each with a True or False value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than just the amount of children who like each superhero, if I can see the overlap information in some easy way, I might find that an unusually large number only like Batman, or that most of those who like Superman are likely to also like the Flash.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest would be to do it visually, but Venn diagrams wouldn't be practical with a large number (50 here) of variables. So it would seem like a correlation matrix with a heat-map base would be the way to go, with something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/glQAw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/glQAw.png&quot; alt=&quot;Source: http://www.sthda.com/english/wiki/correlation-matrix-a-quick-start-guide-to-analyze-format-and-visualize-a-correlation-matrix-using-r-software&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine a heat map with superheros plotted on both axes could work for seeing interesting matches of one variable against all other ones, but it would still require some less ideal extra steps to see when three or more superheros matched, like seeing the colour overlap of superhero1 with superhero2 &amp;amp; superhero3 both be high, and that's kind of guesswork, because the actual children who like all 3 could still be low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment, the best solution I can think of is to try to reduce my variables down into categories, so the example could be &lt;em&gt;Marvel&lt;/em&gt;, &lt;em&gt;DC&lt;/em&gt;, &lt;em&gt;male&lt;/em&gt;, &lt;em&gt;female&lt;/em&gt;, but that loses some potential data that could be helpful from overlaps within those categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe if I did something like the image above, the circle size would be the number for that overlap, and the colour could be number matches with other variables (without listing them, but I could further investigate). That kind of coding is a little out of my comfort zone, but I could try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideas appreciated! I'd ideally do this in matplotlib or some other means within python, but if I have to use Matlab or another tool then I will consider that. Ideas or suggestions appreciated! Hopefully this was clear, thank you!&lt;/p&gt;&#xA;" OwnerUserId="32266" LastEditorUserId="32266" LastEditDate="2017-06-11T22:54:08.200" LastActivityDate="2017-06-12T20:19:11.030" Title="How to more simply see overlapping data for dozens of variables?" Tags="|python|visualization|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19606" PostTypeId="1" CreationDate="2017-06-10T00:12:35.460" Score="2" ViewCount="431" Body="&lt;p&gt;So I am computing the training accuracy of my CNN on a batch training data and it almost always gives 100% accuracy. But I found from somewhere that in order to get the training accuracy, it is tested against the entire training data and not the specific batch that you used for your training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is training accuracy even computed? I wonder if you don't use batches for training. Then it will be 100% accuracy always?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pbFbJ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pbFbJ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm suspecting that this is due to using a small batch size.&lt;/p&gt;&#xA;" OwnerUserId="32361" LastEditorUserId="32361" LastEditDate="2017-06-10T15:47:36.237" LastActivityDate="2017-06-10T15:47:36.237" Title="Training accuracy with SGD" Tags="|machine-learning|deep-learning|training|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19611" PostTypeId="1" CreationDate="2017-06-10T14:20:24.813" Score="1" ViewCount="168" Body="&lt;p&gt;I am using SVM with linear kernel for one of my multiclass text classification problem and getting accuracy of 0.78. I also tried Random Forest but the best accuracy I could get was 0.72. This set me thinking what if I could simulate Random Forest but with a SVM classifier and not decision tree. I am not sure whether it will result in improved accuracy or not. I wanted to have something in favor before I implemented this and a quick google search didn't help either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on this?&lt;/p&gt;&#xA;" OwnerUserId="30363" LastActivityDate="2017-06-13T03:41:16.913" Title="&quot;Random Forest&quot; variant of other classifiers" Tags="|machine-learning|nlp|random-forest|svm|decision-trees|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19612" PostTypeId="1" AcceptedAnswerId="19831" CreationDate="2017-06-10T14:43:24.360" Score="2" ViewCount="135" Body="&lt;p&gt;I have a training set with columns as follows:&#xA;$x_1,x_2,x_3,x_4, x_3/x_4$. &#xA;I want to know if I should consider $x_3$,$x_4$ and $x_3/x_4$ as separate features (&lt;em&gt;as $x_3/x_4$ is derived from other features&lt;/em&gt;)? Also I found that dropping $x_3$ as a feature increases my &lt;code&gt;k-fold&lt;/code&gt; &lt;code&gt;neg-loss score&lt;/code&gt; (&lt;em&gt;marginally!&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The underlying question is how would I know if entertaining $1/x_1$ or $x_1/x_2$ or even $\sqrt{x_4}x_3^4/x_1$... (&lt;em&gt;you get the idea&lt;/em&gt;) as another feature column will/won't increase my score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any method/algorithm to know which &lt;strong&gt;&quot;modified&quot;&lt;/strong&gt; features will increase my score?&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorDisplayName="user380" LastEditDate="2017-07-14T08:35:22.017" LastActivityDate="2017-07-14T08:35:22.017" Title="Regarding &quot;modification&quot; of feature columns in supervised learning" Tags="|machine-learning|feature-selection|supervised-learning|feature-engineering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19613" PostTypeId="2" ParentId="19606" CreationDate="2017-06-10T14:54:47.793" Score="1" Body="&lt;p&gt;Training accuracy is not considered because it can give falsified evaluation value for over-fitting &lt;em&gt;100% in your case&lt;/em&gt;. Hence it is always computed against the batch not entertained for training. This corresponds to the &lt;em&gt;generalized error&lt;/em&gt;.&#xA;For more information see &lt;code&gt;K-Fold&lt;/code&gt; cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This might help: &lt;a href=&quot;https://thebayesianobserver.files.wordpress.com/2012/02/debugging-ml.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://thebayesianobserver.files.wordpress.com/2012/02/debugging-ml.png&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33020" LastActivityDate="2017-06-10T14:54:47.793" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19614" PostTypeId="1" AcceptedAnswerId="23042" CreationDate="2017-06-10T15:39:55.707" Score="7" ViewCount="4035" Body="&lt;p&gt;I'm having a hard time trying to derive the maths behind LSTMs and vanishing gradients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had a of help from &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html&quot; rel=&quot;noreferrer&quot;&gt;LSTM forward and backward pass&lt;/a&gt;, but I got stuck in &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/11&quot; rel=&quot;noreferrer&quot;&gt;page 11 from LSTM forward and backward pass&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CP06d.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CP06d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can form system of equations,&#xA;$$&#xA;        \begin{bmatrix}&#xA;        a^t \\&#xA;        i^t \\&#xA;        f^t \\&#xA;        o^t \\&#xA;        \end{bmatrix}&#xA;=&#xA;        \begin{bmatrix}&#xA;        tanh(W_cx^t+U_ch^{t-1}) \\&#xA;        \sigma(W_iX^t+U_ih^{t-1}) \\&#xA;        \sigma(W_fx^t+U_fh^{t-1}) \\&#xA;        \sigma(W_ox^t+U_oh^{t-1}) \\&#xA;        \end{bmatrix}&#xA;=&#xA;        \begin{bmatrix}&#xA;        tanh(\hat  a^t) \\&#xA;        \sigma (\hat i^t) \\&#xA;        \sigma (\hat f^t) \\&#xA;        \sigma (\hat o^t) \\&#xA;        \end{bmatrix}&#xA;$$&#xA;We can then represent this as $z$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;z=&#xA;        \begin{bmatrix}&#xA;        \hat  a^t \\&#xA;        \hat i^t \\&#xA;        \hat f^t \\&#xA;        \hat o^t \\&#xA;        \end{bmatrix}&#xA;=&#xA;        \begin{bmatrix}&#xA;        W^c &amp;amp; U^c \\&#xA;        W^i &amp;amp; U^i \\&#xA;        W^f &amp;amp; U^f \\&#xA;        W^o &amp;amp; U^o \\&#xA;        \end{bmatrix}&#xA;*&#xA;        \begin{bmatrix}&#xA;        x^t \\&#xA;        h^{t-1} \\&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can find out the backprop derivation for $z$ from &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/10&quot; rel=&quot;noreferrer&quot;&gt;page 10 from LSTM forward and backward pass&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\delta z=&#xA;        \begin{bmatrix}&#xA;        \delta \hat  a^t \\&#xA;        \delta \hat i^t \\&#xA;        \delta \hat f^t \\&#xA;        \delta \hat o^t \\&#xA;        \end{bmatrix}&#xA;=&#xA;        \begin{bmatrix}&#xA;        \delta a^t \odot (1-tanh^2(\hat a^t)) \\&#xA;        \delta i^t \odot i^t \odot (1-i^t) \\&#xA;        \delta f^t \odot f^t \odot (1-f^t) \\&#xA;        \delta o^t \odot o^t \odot (1-o^t) \\&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the next part at &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/11&quot; rel=&quot;noreferrer&quot;&gt;page 11 from LSTM forward and backward pass&lt;/a&gt; is where I'm confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given $\delta z$, we need to find $\delta W$, $\delta h^{t-1}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The author wrote down $\delta I^t = W^T * \delta z$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we do some linear algebra variables moves:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z = W^T * I^t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiply both sides with $I^{t^{-1}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I^{t^{-1}} z = W^T$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiply both sides with $z^{-1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$I^{t^{-1}} = z^{-1} W^T$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somehow this doesn't match with the author's formula?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Let's ignore 1), and try to solve for $\delta I$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\delta I =&#xA;        \begin{bmatrix}&#xA;        \delta x^t \\&#xA;        \delta h^{t-1} \\&#xA;        \end{bmatrix}&#xA;$$&#xA;$$&#xA;\delta I = \frac{dE}{dI} =&#xA;        \begin{bmatrix}&#xA;        \frac{dE}{dx^t} \\&#xA;        \frac{d}{dh^{t-1}} \\&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But $\frac{d}{dh^{t-1}}$ depends on a lot of the equations in $z$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I solve for them individually and add them up?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: $\frac{dE}{d\hat i_t}$ can be found at &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/10&quot; rel=&quot;noreferrer&quot;&gt;page 10 from LSTM forward and backward pass&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h_{t-1}^{i_t}=\frac{dE}{dh_{t-1}^{i_t}}=\frac{dE}{d\hat i_t}\frac{d\hat i_t}{h_{t-1}}=\frac{dE}{d\hat i_t}\frac{d}{dh_{t-1}}i_t(1-i_t)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replace $i_t$ with $\sigma(W_iX^t+U_ih^{t-1})$, replace $\frac{dE}{d\hat i_t}$ with $\delta \hat i_t \delta i_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$=\delta \hat i_t \delta i_t \frac{d}{dh_{t-1}} \sigma(W_iX^t+U_ih^{t-1}) (1-\sigma(W_iX^t+U_ih^{t-1}))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks solvable, then my question is I would get 4 equations like the above, do I add them all together in the end to get $\delta h^{t-1}$? For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\delta I = \frac{dE}{dI} =&#xA;        \begin{bmatrix}&#xA;        \frac{dE}{dx^t} \\&#xA;        \frac{d}{dh^{t-1}} \\&#xA;        \end{bmatrix}&#xA;=&#xA;        \begin{bmatrix}&#xA;        ignore \\&#xA;        h_{t-1}^{i_t}+h_{t-1}^{a_t}+h_{t-1}^{f_t}+h_{t-1}^{o_t} \\&#xA;        \end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my logic was to find the total error contributed from $h_{t-1}$ so you need to add them together?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Finding for $W$ looks even like a bigger task, however, I'm not sure where to start on this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) How does this relate to the error carousel? I mean after derivation of all the weights and $h_{t-1}$, I'm not sure how this leads to avoidance of vanishing gradients? I read somewhere that the weights are constant 1 or something along the lines like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is kinda long, but feel free to ask for clarification if my question does not make sense. I think I've tried to solve for this almost for half a month now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate any sort of guidance. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="24119" LastActivityDate="2017-09-14T18:19:20.687" Title="Need help understanding LSTMs' backpropagation and carousel of error" Tags="|rnn|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19615" PostTypeId="1" AcceptedAnswerId="19746" CreationDate="2017-06-10T16:42:14.897" Score="3" ViewCount="8664" Body="&lt;p&gt;I'm a beginner in machine learning and I want to build a model to predict the price of houses. I prepared a dataset by crawling a local housing website and it consists 1000 samples and only 4 features (latitude, longitude, area and number of rooms).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried RandomForestRegressorand LinearSVR models in sklearn, but I can't train the model properly and the MSE is super high.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MSE almost equals 90,000,000 (the true values of prices' range are between 5,000,000 - 900,000,000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.svm import LinearSVR&#xA;import pandas as pd&#xA;import csv&#xA;from sklearn.preprocessing import normalize&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;df = pd.read_csv('dataset.csv', index_col=False)&#xA;X = df.drop('price', axis=1)&#xA;&#xA;X_data = X.values&#xA;Y_data = df.price.values&#xA;X_train, X_test, Y_train, Y_test = train_test_split(X_data, Y_data, test_size=0.2, random_state=5)&#xA;&#xA;rgr = RandomForestRegressor(n_estimators=100)&#xA;svr = LinearSVR()&#xA;&#xA;rgr.fit(X_train, Y_train)&#xA;svr.fit(X_train, Y_train)&#xA;&#xA;MSEs = cross_val_score(estimator=rgr,&#xA;                         X=X_train,&#xA;                         y=Y_train,&#xA;                         scoring='mean_squared_error',&#xA;                         cv=5)&#xA;&#xA;MSEsSVR = cross_val_score(estimator=svr,&#xA;                         X=X_train,&#xA;                         y=Y_train,&#xA;                         scoring='mean_squared_error',&#xA;                         cv=5)&#xA;&#xA;MSEs *= -1&#xA;RMSEs = np.sqrt(MSEs)&#xA;&#xA;print(&quot;Root mean squared error with 95% confidence interval:&quot;)&#xA;print(&quot;{:.3f} (+/- {:.3f})&quot;.format(RMSEs.mean(), RMSEs.std()*2))&#xA;print(&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is the problem with my dataset and count of features? How can I build a prediction model with this type of dataset?&lt;/p&gt;&#xA;" OwnerUserId="33188" LastActivityDate="2017-06-15T13:19:11.383" Title="Large mean squared error in sklearn regressors" Tags="|machine-learning|python|regression|scikit-learn|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19617" PostTypeId="2" ParentId="14581" CreationDate="2017-06-10T18:22:56.903" Score="73" Body="&lt;p&gt;*To complement already great answers above.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;From my experience, &lt;strong&gt;GRUs train faster&lt;/strong&gt; and perform better than LSTMs on &lt;strong&gt;less training data&lt;/strong&gt; if you are doing language modeling (not sure about other tasks). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;GRUs are simpler&lt;/strong&gt; and thus easier to modify, for example adding new gates in case of additional input to the network. It's just less code in general.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;LSTMs&lt;/strong&gt; should in theory &lt;strong&gt;remember longer sequences&lt;/strong&gt; than GRUs and outperform them in tasks requiring modeling long-distance relations.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;*Some additional papers that analyze GRUs and LSTMs.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Neural GPUs Learn Algorithms&quot; (Łukasz Kaiser, Ilya Sutskever, 2015)&#xA;&lt;a href=&quot;https://arxiv.org/abs/1511.08228&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1511.08228&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Comparative Study of CNN and RNN for Natural Language Processing&quot;&#xA;(Wenpeng Yin et al. 2017) &lt;a href=&quot;https://arxiv.org/abs/1702.01923&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1702.01923&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="22012" LastEditorUserId="836" LastEditDate="2017-09-27T13:06:32.997" LastActivityDate="2017-09-27T13:06:32.997" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19618" PostTypeId="1" CreationDate="2017-06-10T19:42:19.260" Score="2" ViewCount="78" Body="&lt;p&gt;i am using the KEEL gui for my experiments . I want to plot a ROC curve of the results produced by KEEL rather than a just a roc score value &lt;a href=&quot;https://i.stack.imgur.com/lptPY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lptPY.png&quot; alt=&quot;demo xml of my experiment&quot;&gt;&lt;/a&gt; .  &lt;/p&gt;&#xA;" OwnerUserId="31072" LastEditorUserId="31072" LastEditDate="2017-06-15T18:55:20.710" LastActivityDate="2017-06-15T18:55:20.710" Title="Is there any way to generate ROC curves from KEEL?" Tags="|machine-learning|data-mining|algorithms|java|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19620" PostTypeId="1" CreationDate="2017-06-11T02:39:27.017" Score="2" ViewCount="101" Body="&lt;p&gt;I am trying to use &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Credit+Approval&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI credit approval dataset&lt;/a&gt; to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to use dataset to decide whether new credit requesters request is approved or not, base on the data provided by them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I see this dataset is widely use, but I do not know use It without knowing the attribute names?&lt;/strong&gt;(I am aware that some attribute values can be changed to protect confidentiality)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What piece of knowledge am I missing here to use this dataset?&lt;/p&gt;&#xA;" OwnerUserId="33272" LastActivityDate="2017-06-12T12:08:54.507" Title="How to use a dataset where attribute names are changed?" Tags="|data-mining|dataset|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19621" PostTypeId="2" ParentId="19620" CreationDate="2017-06-11T08:05:20.443" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What piece of knowledge am I missing here to use this dataset?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By anonymising the data and altering the attribute names, the providers of this dataset have made it into an abstract machine learning exercise, where data has similar &lt;em&gt;qualities&lt;/em&gt; to real-world credit approval data, so that approaches to train models that work well on it are likely to work well in a real-world scenario. However, those models could not be used in a production system. You are not in a position to train a model based on it and then input real world data you have collected elsewhere to make a prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can still train a model on this data to predict A16 (the class attribute). You can measure the accuracy - or any other metric - by holding out a test set. You can experiment with feature engineering, feature selection etc, in an abstract way without being able to apply much domain knowledge. You can try out different model classes, different hyper-parameters, different approaches to imputing missing values or cross-validation etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your project can draw conclusions about the approach you have taken, based on test results. What you cannot do is explore real-world scenarios with simulated customer data. This might make it less compelling as a demonstration - for instance users could not explore exactly what factors might lead to an application being approved or not, instead you would just have to show some graphs that demonstrate you have selected a model with good performance.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-12T12:08:54.507" LastActivityDate="2017-06-12T12:08:54.507" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19624" PostTypeId="1" CreationDate="2017-06-11T10:56:58.927" Score="2" ViewCount="1694" Body="&lt;p&gt;I'm trying to set up a CNN in Keras, and I don't have much experience outside of simple NNs as far as the actual setup. I understand how the CNN is going to work on my data, but I need advice on how to set it up in Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my dataset, each datapoint is structured as follows. There are &lt;em&gt;n&lt;/em&gt; columns, each of which is sensor data of people performing an action. There are 10 rows where each row is a single person's sensor data for that event. The label column is a score for how well they performed. Since each of their performances is related, each datapoint is a 10 by &lt;em&gt;n&lt;/em&gt; matrix, and the label column for each data point is a 10 by 1 vector of scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, I want to set up my NN so that it treats:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Num|Sens0|Sens1|Sens2|Win&#xA;---|-----|-----|-----|---&#xA;A  |data |data |data | 0&#xA;B  |data |data |data | 0&#xA;C  |data |data |data | 0&#xA;D  |data |data |data | 0&#xA;E  |data |data |data | 0&#xA;F  |data |data |data | 0&#xA;G  |data |data |data | 0&#xA;H  |data |data |data | 1&#xA;I  |data |data |data | 0&#xA;J  |data |data |data | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the same as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Num|Sens0|Sens1|Sens2|Win&#xA;---|-----|-----|-----|---&#xA;J  |data |data |data | 0&#xA;F  |data |data |data | 0&#xA;D  |data |data |data | 0&#xA;B  |data |data |data | 0&#xA;G  |data |data |data | 0&#xA;H  |data |data |data | 1&#xA;C  |data |data |data | 0&#xA;A  |data |data |data | 0&#xA;I  |data |data |data | 0&#xA;E  |data |data |data | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my first layer (at the least), I want to set up convolutional neurons (I'm starting with 5, and moving on from there) so that each row shares weights amongst the observations, in such a way that the order I put the row in the datapoint doesn't matter (aside from it's correspondence with the label). Anyone know if this is possible? Otherwise I'll have to feed in each datapoint in all 10! orders, and I would prefer not to...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated, thank you!&lt;/p&gt;&#xA;" OwnerUserId="32888" LastActivityDate="2017-06-11T10:56:58.927" Title="How to Create Shared Weights Layer in Keras" Tags="|neural-network|keras|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19625" PostTypeId="1" AcceptedAnswerId="19627" CreationDate="2017-06-11T16:51:45.380" Score="2" ViewCount="319" Body="&lt;p&gt;Let's say that we have a CNN with two convolutional layers (&lt;a href=&quot;https://www.tensorflow.org/tutorials/layers&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/layers&lt;/a&gt;). My question regards the dimension of the tensor, which is the output of the pooling layer 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first convolutional layer, we apply $32$ filters to the input image (let's say that the output will be $28\times 28 \times 32)$, so as far as I can understand we will get $32$ separate feature maps, because of the number of filters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the next step, we can apply an activation function, which does not change the dimensionality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $\max$ pooling layer takes as input a tensor of $28\times 28\times 32$ and the output is going to be a $14 \times 14 \times \color{red}{1}$ tensor (according to the link above).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I cannot understand the unit as the depth, since we apply $32$ filters and we apply the $\max$ pooling layer to every feature map. So, why is the output tensor $14 \times 14 \times 1$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to my understanding the input in the second convolutional layer should be a $14 \times 14 \times 32$ tensor. Probably, I am missing something here.&lt;/p&gt;&#xA;" OwnerUserId="33150" LastEditorUserId="33150" LastEditDate="2017-06-11T18:37:14.150" LastActivityDate="2017-06-11T18:37:14.150" Title="Depth of the first pooling layer outcome in tensorflow documentation" Tags="|neural-network|deep-learning|image-recognition|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19626" PostTypeId="1" AcceptedAnswerId="19632" CreationDate="2017-06-11T17:22:32.393" Score="2" ViewCount="162" Body="&lt;p&gt;I have a set of user sessions. Session consists of an ordered list of types of actions that user made (for example, bought a gun, played a mission, etc).&#xA;I want to create/calculate session that have most possible similarity to all provided sessions (most common types of actions users make in order they make them)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, I know nothing about data science but I tried to google a way to do that.&#xA;I've found this document: &lt;a href=&quot;https://cran.r-project.org/web/packages/TraMineR/vignettes/TraMineR-state-sequence.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/TraMineR/vignettes/TraMineR-state-sequence.pdf&lt;/a&gt;&#xA;And it looks like 9.1 and 9.2 describe things similar to what I want. But I dont know this for certain and even if it's true I stil don't know how to use it for my scenario.&lt;/p&gt;&#xA;" OwnerUserId="33281" LastActivityDate="2019-04-23T20:08:28.480" Title="Find average sequence from a set of sequences" Tags="|clustering|sequence|" AnswerCount="1" CommentCount="1" ClosedDate="2017-06-11T20:01:10.460" ContentLicense="CC BY-SA 3.0" />
  <row Id="19627" PostTypeId="2" ParentId="19625" CreationDate="2017-06-11T17:25:46.003" Score="2" Body="&lt;p&gt;I kept spinning my head around this question because I seem to come along the same conclusion as you do. However, it appears to be a mistake in the documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/43453712/what-is-output-tensor-of-max-pooling-2d-layer-in-tensorflow&quot;&gt;https://stackoverflow.com/questions/43453712/what-is-output-tensor-of-max-pooling-2d-layer-in-tensorflow&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-06-11T17:25:46.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19629" PostTypeId="2" ParentId="19620" CreationDate="2017-06-11T17:40:17.947" Score="0" Body="&lt;p&gt;I'll start with what you want to know, and move to the caveats. You are probably safe to use any number of decision tree algorithms to do some black box analysis, given the peer-reviewed publications on the dataset using them (though as any academic will tell you, this is often a stretch too). As the papers mention, C4.5, CART, Random Forrest, adaboost, and the like are all candidate models. Most can be used to gather a variable importance ranking for predicting the response variable you chose, and you should be able to examine confusion matrix metrics for evaluating performance if you chose to predict class variables. Try different pruning methods on the tree(s) to prevent overfit, and maybe even nest models to build an ensemble. Don't forget to rigorously validate the model (I recommend cross-validate, which is baked into most frameworks that handle decision trees).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;However&lt;/em&gt;, I urge you against using this data. Here are a few reasons why:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Diagnostics nightmare. Just about all modeling techniques rely on certain assumptions. Some common ones that can break most models are the presence of autocorrelation, heteroskedasticity, and multicollinearity. Violate these, and most models will become unstable. You can try controlling for this by plotting ACFs and PACFs, throwing out highly correlated variables or lumping them together some way, just to name a few approaches. But these all have problems. The problem with throwing out highly correlated variables is that you might throw out the most important relationships that in fact don't violate anything! Or, say your data turns out to be a multivariate time series? The PACF no longer interprets the same way. Trees, used with this data set, are fairly robust, but not perfect. Their use in literature likely means that you can ignore this, but be cautious, as some papers explicitly use data that don't work to make a point to about how algorithms can break. Remember the cardinal rule: garbage in, garbage out. Best advice here is to read the papers (though this is generally not something you can fall back on with other data). &lt;/li&gt;&#xA;&lt;li&gt;Causality is impossible to detect. Blind data makes it near-impossible to track things temporally, which means that the model is likely to end up using variables in it's predictions that are only known after the event. Conceptually, this isn't very useful unless your model will only ever be used to fill missing bits in already collected data. &lt;/li&gt;&#xA;&lt;li&gt;Interpretation. This might pass for a school project, but in most cases interpretation is everything. Ask this question: what can the client (in this case, the credit company) do with your analysis? There is no way to know with masked data. Say you built a black box model with a decisin tree. It does a fantastic job of predicting the outcome. So, you put a bow on it and send it to the client. But what happens when they unwrap it, renaming the variables? It can potentially be useless, and you have no way of knowing. You may be predicting an unimportant variable, or drawing attention to &quot;insights&quot; that are completely obvious. An analyst's worth is in what he brings to the table, not the models he uses. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In summary, read the papers, use trees, and fear unknown data. There be dragons.&lt;/p&gt;&#xA;" OwnerUserId="30036" LastEditorUserId="30036" LastEditDate="2017-06-11T17:46:45.867" LastActivityDate="2017-06-11T17:46:45.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19630" PostTypeId="1" AcceptedAnswerId="19808" CreationDate="2017-06-11T18:31:58.770" Score="6" ViewCount="1768" Body="&lt;p&gt;I'm training a convoluted neural net to drive a toy car, and no matter what I do the &lt;strong&gt;&lt;em&gt;training&lt;/em&gt;&lt;/strong&gt; accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Notes:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records. &lt;/li&gt;&#xA;&lt;li&gt;Three equally proportioned classes: turn left, go straight, turn right&lt;/li&gt;&#xA;&lt;li&gt;Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6&lt;/li&gt;&#xA;&lt;li&gt;Simple NN uses the sigmoid activation function, the convnet is all relu activations&lt;/li&gt;&#xA;&lt;li&gt;Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs &lt;/li&gt;&#xA;&lt;li&gt;My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion&lt;/li&gt;&#xA;&lt;li&gt;Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%&lt;/li&gt;&#xA;&lt;li&gt;I'm using &lt;code&gt;initial = tf.truncated_normal(shape, stddev=0.1)&lt;/code&gt; to initialize all weights&lt;/li&gt;&#xA;&lt;li&gt;I'm using &lt;code&gt;initial = tf.constant(0.1, shape=shape)&lt;/code&gt; to initialize all biases&lt;/li&gt;&#xA;&lt;li&gt;It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Convnet Notes&lt;/strong&gt; (all yield same poor results):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Batch normalization&lt;/li&gt;&#xA;&lt;li&gt;Max pooling&lt;/li&gt;&#xA;&lt;li&gt;50% drop-out probability&lt;/li&gt;&#xA;&lt;li&gt;Various stride sizes&lt;/li&gt;&#xA;&lt;li&gt;Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons. &lt;/li&gt;&#xA;&lt;li&gt;3D convolution (took too much memory and caused my GPU to crash)&lt;/li&gt;&#xA;&lt;li&gt;1x1 convolutions&lt;/li&gt;&#xA;&lt;li&gt;The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/RyanZotti/Self-Driving-Car/blob/80d36fc705d74f5ba9fc320d524ea4b3e45587b0/train_ANN.py&quot; rel=&quot;noreferrer&quot; title=&quot;Simple Neural Net&quot;&gt;Simple neural net&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/RyanZotti/Self-Driving-Car/blob/80d36fc705d74f5ba9fc320d524ea4b3e45587b0/train_deep_convnet.py&quot; rel=&quot;noreferrer&quot; title=&quot;asdf&quot;&gt;Deep Convnet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29169" LastEditDate="2021-02-16T00:46:21.810" LastActivityDate="2021-02-16T00:46:21.810" Title="Convnet training error does not decrease" Tags="|machine-learning|deep-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19631" PostTypeId="1" CreationDate="2017-06-11T18:34:29.853" Score="0" ViewCount="163" Body="&lt;p&gt;This is a follow-up question regarding this &lt;a href=&quot;https://datascience.stackexchange.com/questions/19625/depth-of-the-first-pooling-layer-outcome-in-tensorflow-documentation/19627#19627&quot;&gt;question&lt;/a&gt;. I managed to verify in &lt;code&gt;python&lt;/code&gt; that the output of the first pooling layer will be a $14 \times 14\times 32$ tensor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I interpret this as having 32 $14 \times 14$ images. Maybe my interpretation is mistaken. In the second convolutional layer we apply $64$ filters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I type in &lt;code&gt;python&lt;/code&gt; &lt;code&gt;conv2.shape&lt;/code&gt; I get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;TensorShape([Dimension(55000), Dimension(14), Dimension(14), Dimension(64)])&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How that comes? I mean I am in interested in the outcome of the depth, which is $64.$&lt;/p&gt;&#xA;" OwnerUserId="33150" LastEditorUserId="33150" LastEditDate="2017-06-11T20:17:52.063" LastActivityDate="2017-06-11T20:48:17.753" Title="Connection between the first pooling layer and the second convolutional layer" Tags="|deep-learning|tensorflow|computer-vision|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19632" PostTypeId="2" ParentId="19626" CreationDate="2017-06-11T18:38:59.037" Score="4" Body="&lt;p&gt;One way would be not to approach this as a calculation per session. Most data science solutions like to end up with a number, probability or classification. I suggest you structure your data differently so that you try to answer the question - what next action is likely given the last action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to do this you would have to restructure your session data and use information from across all your sessions. For example, if you compare in how many sessions a player 'buys a gun', and if so record over all those sessions what their next action is, e.g. in 60% they 'play a mission' next. You will then have a probability of their next action based on the number of choices players made in all those sessions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have those probabilities, you will be able to answer the question, 'What comes next?'. This will in turn enable you to build that most average session that you are after by stepping through a session and building it by the most probable next step.&lt;/p&gt;&#xA;" OwnerUserId="31245" LastEditorUserId="31245" LastEditDate="2019-04-23T20:08:28.480" LastActivityDate="2019-04-23T20:08:28.480" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19633" PostTypeId="2" ParentId="19631" CreationDate="2017-06-11T20:48:17.753" Score="1" Body="&lt;p&gt;The first dimension is the batch size. From &lt;a href=&quot;https://www.tensorflow.org/tutorials/layers&quot; rel=&quot;nofollow noreferrer&quot;&gt;the tutorial you're linking to&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We store the training feature data (the raw pixel values for 55,000 images of hand-drawn digits)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So it looks like your code is performing each gradient update step using all 55,000 training examples. That's unorthodox, but it's fine as long as your computer can handle it. See &lt;a href=&quot;https://www.youtube.com/watch?v=nhqo0u1a6fw&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt; for an explanation of batching in gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your interpretation of the shape as being 32 14x14 images is fine for now, but it is more proper to refer to these 32 matrices as &quot;feature maps&quot;. A grayscale image is/contains a single feature map, while a color image would consist of three of them, one for each red, green, and blue (assuming RGB color space). Also, note that at this point in the network the feature maps are no longer &quot;images&quot; as any human would understand the word.&lt;/p&gt;&#xA;" OwnerUserId="33237" LastActivityDate="2017-06-11T20:48:17.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19638" PostTypeId="1" AcceptedAnswerId="19639" CreationDate="2017-06-12T06:06:45.403" Score="1" ViewCount="2766" Body="&lt;p&gt;Let's say I have 3 classes and 1 score for each data point&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;score z&lt;/li&gt;&#xA;&lt;li&gt;class 1&lt;/li&gt;&#xA;&lt;li&gt;class 2&lt;/li&gt;&#xA;&lt;li&gt;class 3&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;E.g. the input looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.529 5 7 4&#xA;0.310 3 4 2&#xA;0.774 10 7 6&#xA;0.774 10 8 5&#xA;0.172 3 0 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt;&amp;gt; df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0.31, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [0.172, 3.0, 0.0, 2.0]])&#xA;&amp;gt;&amp;gt;&amp;gt; df&#xA;       0     1    2    3&#xA;0  0.529   5.0  7.0  4.0&#xA;1  0.310   3.0  4.0  2.0&#xA;2  0.774  10.0  7.0  6.0&#xA;3  0.774  10.0  8.0  5.0&#xA;4  0.172   3.0  0.0  2.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row represents a data point and column 0 is the score and column 1,2,3 are some sort of coefficients for the classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The score is some sort of computed entity independent of the coefficients and the goal is to visualize the interaction between the coefficients and the score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to visualize the interaction between the 3 classes (column 1,2,3) and the score (column 0)?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastActivityDate="2017-06-12T09:40:46.143" Title="How to visualize multi-instance multiclass classification?" Tags="|visualization|multiclass-classification|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19639" PostTypeId="2" ParentId="19638" CreationDate="2017-06-12T09:40:46.143" Score="2" Body="&lt;p&gt;You could use a 3d-scatter plot, where each class would correspond to one axis and the color-intensity of the point would indicate the score value(e.g. for a grayscale colormap, the whiter the closer to 1 the value of the score).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the above format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from mpl_toolkits.mplot3d import Axes3D&#xA;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;&#xA;&#xA;df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [1, 3.0, 0.0, 2.0]])&#xA;fig = plt.figure()&#xA;ax = fig.add_subplot(111, projection='3d')&#xA;ax.scatter(df[1].values, df[2].values, df[3].values, c=df[0].values, cmap='gray', s=50, vmin=0.,vmax=1)&#xA;&#xA;ax.set_xlabel('X Label')&#xA;ax.set_ylabel('Y Label')&#xA;ax.set_zlabel('Z Label')&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/axPuu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/axPuu.png&quot; alt=&quot;Example 3d&quot;&gt;&lt;/a&gt;&#xA;You can change the scatter plot attributes for marker-size, colormap etc. &lt;a href=&quot;https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.scatter.html#matplotlib.axes.Axes.scatter&quot; rel=&quot;nofollow noreferrer&quot;&gt;according to the documentation&lt;/a&gt; to match your tastes.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-06-12T09:40:46.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19641" PostTypeId="1" AcceptedAnswerId="19695" CreationDate="2017-06-12T12:06:43.070" Score="1" ViewCount="10890" Body="&lt;p&gt;I have a pandas dataframe with binary value columns. I would like to replace values in each cell with its frequency in rspective column in place. My question is how to keep track of the current column while using apply on the subset of columns like here: (to be applied from 8th columns to the end) : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_data.ix[:,8:] = train_data.ix[:,8:].apply(x: what should come here?)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that &lt;code&gt;train_data.ix[:,col_number].value_counts()[0]&lt;/code&gt; will return number of zeros in col_number but how can I use it inside apply function?&lt;/p&gt;&#xA;" OwnerUserId="15064" LastEditorUserId="32230" LastEditDate="2017-06-12T20:53:44.860" LastActivityDate="2017-06-14T10:11:06.443" Title="How to change a cell in Pandas dataframe with respective frequency of the cell in respective column" Tags="|python|pandas|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19643" PostTypeId="1" CreationDate="2017-06-12T14:05:54.140" Score="1" ViewCount="1159" Body="&lt;p&gt;I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&#xA;Even though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$H(p, q) = -\sum_x p(x)\, \log q(x).$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?&lt;/p&gt;&#xA;" OwnerUserId="33306" LastEditorUserId="33306" LastEditDate="2017-06-13T14:44:59.690" LastActivityDate="2017-06-13T14:44:59.690" Title="Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class?" Tags="|machine-learning|neural-network|classification|tensorflow|cost-function|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19644" PostTypeId="2" ParentId="19605" CreationDate="2017-06-12T14:20:00.583" Score="0" Body="&lt;p&gt;t-Distributed Stochastic Neighbor Embedding (&lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt;) might be exactly what you need. The linked page has code and links to various blogs describing its use.&lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2017-06-12T14:20:00.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19646" PostTypeId="2" ParentId="19643" CreationDate="2017-06-12T15:29:40.163" Score="1" Body="&lt;p&gt;Cross entropy is indeed appropriate for multiclass classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the tensorflow documentation states about cross_entropy that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;it is referring to the fact that, at a conceptual level, classes are not expected to overlap. An example of non-overlapping classes are &quot;cat&quot; and &quot;dog&quot;. An example of overlapping classes are &quot;Elephant&quot; and &quot;African Elephant&quot;; this type of problem is called multilabel classification, because instead of classes, you have labels and each individual can be assigned many labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: with the new information about the problem, we can tell that the problem being faced is not a classification one, as the desired output are specific probability values. This implies that the problem is a regression one. As the outputs are probabilities, it is appropriate to use softmax to ensure they add up to 1. As it is regression, it would be appropriate to use MSE as loss function.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-06-13T08:19:36.533" LastActivityDate="2017-06-13T08:19:36.533" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19647" PostTypeId="1" AcceptedAnswerId="22370" CreationDate="2017-06-12T15:43:18.167" Score="1" ViewCount="1397" Body="&lt;p&gt;In the original paper, the author says that the annotation are the concatenation fo the forward states and the backward states at each time step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tfq1N.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tfq1N.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the tensorflow &lt;a href=&quot;https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/seq2seq/BahdanauAttention&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation&lt;/a&gt; (memory param), the memory field is said to be populated with the output (not hidden state) of an RNN encoder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I missing?&lt;/p&gt;&#xA;" OwnerUserId="33312" LastActivityDate="2017-08-18T09:21:52.423" Title="Bahdanau Attention" Tags="|tensorflow|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19648" PostTypeId="1" AcceptedAnswerId="19656" CreationDate="2017-06-12T16:40:59.607" Score="3" ViewCount="495" Body="&lt;p&gt;I'm trying to detect duplicates in a data set of about 34k distinct items. When I say &quot;duplicate,&quot; I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the items and return a sparse matrix of similar items with a similarity from 0.0 to 1.0. This matrix has about 5k non-zero entries in it, so it's very sparse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already know what the similar items are by pairs. What I need is a good way to cluster them together. I expect there to be many clusters with only 2 items in them, and a couple thousand with more than 2 items, with the vast majority of items being unclustered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a clustering algorithm that fits this scenario well? I've tried several, but they either give poor results, or are designed with few large clusters in mind instead of many small clusters, and so tend to crash.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is clustering the wrong approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Spark, but the sparse matrix is small enough that I don't mind exporting to something else if necessary.&lt;/p&gt;&#xA;" OwnerUserId="20407" LastEditorUserId="20407" LastEditDate="2017-06-13T13:54:12.457" LastActivityDate="2017-06-13T16:28:25.120" Title="Clustering a very large number of very small clusters with most data unrelated" Tags="|clustering|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19649" PostTypeId="1" AcceptedAnswerId="19651" CreationDate="2017-06-12T16:46:14.290" Score="-2" ViewCount="3778" Body="&lt;p&gt;I am doing an experiment on Azure ML. While pre processing my data, there is an option to clean missing data using either PCA or MICE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide me an example of how I can decide on which option to choose.&lt;/p&gt;&#xA;" OwnerUserId="32220" LastEditorUserId="29169" LastEditDate="2019-06-10T15:45:28.617" LastActivityDate="2019-06-10T15:45:28.617" Title="What is PCA and MICE" Tags="|machine-learning|correlation|pca|azure-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19651" PostTypeId="2" ParentId="19649" CreationDate="2017-06-12T17:50:25.243" Score="1" Body="&lt;p&gt;I don't know about Azure ML.  But:&lt;/p&gt;&#xA;&lt;p&gt;PCA is &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;principal components analysis&lt;/a&gt;.  It takes a dataset and &amp;quot;rotates&amp;quot; it, taking the original axes defined by the original variables, and creating new axes that are linear combinations of the old data.  The precise linear combinations are chosen such that each successive component maximizes variance along that new dimensions.  A quick google search turns up lots of tutorials.&lt;/p&gt;&#xA;&lt;p&gt;Here is a snipped of Hastie &amp;amp; Tibshirani's lecture on PCA&#xA;&lt;a href=&quot;https://www.youtube.com/watch?v=ipyxSYXgzjQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=ipyxSYXgzjQ&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;MICE is &amp;quot;multiple imputaiton by chained equations&amp;quot;.  Basically, missing data is predicted by observed data, using a sequential algorithm that is allowed to proceed to convergence.  (1) Start by filling in the missing data with plausible guesses at what the values might be.  (2) for each variable, predict the missing values by modeling the observed values as a function of the other variables.  At each step, update the predictions of the missing values.  There are many tricky details, and many online tutorials.  here is an article aimed at biostat practitioners:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30335" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-06-12T17:50:25.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19653" PostTypeId="1" CreationDate="2017-06-12T18:23:30.260" Score="4" ViewCount="143" Body="&lt;p&gt;I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases.  Now I have a problem where I am dealing with model output that I need to save to disk, and then query for another process.  If the data were smaller, I'd store everything in a &lt;code&gt;list&lt;/code&gt;, with hierarchical elements.  For example, if my object is called &lt;code&gt;output.OLS&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; summary(output.OLS)&#xA;        Length Class  Mode&#xA;SEP0307 3      -none- list&#xA;SEP0308 3      -none- list&#xA;SEP0306 3      -none- list&#xA;SEN0308 3      -none- list&#xA;&amp;lt;&amp;lt;snip -- hundreds of sub-lists&amp;gt;&amp;gt;&#xA;SEN0307 3      -none- list&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the second level of hierarchy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; summary(output.OLS[[1]])&#xA;      Length Class  Mode&#xA;c1  11     -none- list&#xA;c2  11     -none- list&#xA;c2  11     -none- list&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the third:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; summary(output.OLS[[1]][[1]])&#xA;         Length Class      Mode&#xA;Baseline 1008   data.frame list&#xA;CanSEM45 1008   data.frame list&#xA;CanSEM85 1008   data.frame list&#xA;CCSM45   1008   data.frame list&#xA;CCSM85   1008   data.frame list&#xA;GISS45   1008   data.frame list&#xA;GISS85   1008   data.frame list&#xA;HadGEM45 1008   data.frame list&#xA;HadGEM85 1008   data.frame list&#xA;MIROC45  1008   data.frame list&#xA;MIROC85  1008   data.frame list&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and we can see that now we're looking at &lt;code&gt;data.frame&lt;/code&gt;s&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1&amp;gt; dim(output.OLS[[1]][[1]][[1]])&#xA;[1]   33 1008&#xA;1&amp;gt; colnames(output.OLS[[1]][[1]][[1]])[1:10]&#xA; [1] &quot;scen&quot;     &quot;year&quot;     &quot;region&quot;     &quot;size&quot;    &quot;amount&quot;     &quot;mean&quot; &#xA; [7] &quot;semean&quot;   &quot;varres&quot;   &quot;result.1&quot; &quot;result.2&quot; &#xA;&amp;lt;&amp;lt;snip&amp;gt;&amp;gt; &#xA;[1006] &quot;result.998&quot;  &quot;result.999&quot;  &quot;result.1000&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are four such output objects, for a total of about 6GB.  They don't all have the columns &lt;code&gt;mean&lt;/code&gt;, &lt;code&gt;semean&lt;/code&gt; and &lt;code&gt;varres&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This list is about 1.5 gigs, which is unwieldy given what I want to do with it.  It'd be much better to read subsets of it from a database.  I can think of two potential architectures:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Architecture 1&lt;/strong&gt;: Single, huge table&#xA;The variable names would be &lt;code&gt;region&lt;/code&gt; (the outermost hierarchy), &lt;code&gt;cx&lt;/code&gt; (i.e.: c1, c2, c3), &lt;code&gt;model&lt;/code&gt; (Baseline, CanSEM, etc), and then all of the results: &lt;code&gt;result.1, result.2, ..., result.1000&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be most convenient.  The problem is that the table would be both deep and wide.  Too wide for SQLLite, which is the simplest database, as far as I can tell.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Architecture 2:&lt;/strong&gt;: one keyvalue table and many data tables.  The keyvalue table would have the following columns:&#xA;&lt;code&gt;cx, scenario, year, place, keyvalue = coerce_to_integer(unique(scenario, year, place, cx))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data tables would look like this&#xA;&lt;code&gt;keyvalue, result_number, result&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This option uses hierarchy and is actually a relational database (relational via the keyvalues, right)?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it has the drawback that it nearly triples the size of my data:  for each datum, I need to store a keyvalue and an index (result_number).  This seems like it wouldn't make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there other options&lt;/strong&gt;&#xA;For example, are there databases similar to R's &lt;code&gt;lists&lt;/code&gt; that can be stored in memory, and read from memory?  Is that sort of like what NoSQL databases are?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So, my question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What database architectures would make sense for this problem?&lt;/li&gt;&#xA;&lt;li&gt;Where should I start learning about these sorts of databases?&lt;/li&gt;&#xA;&lt;li&gt;How does R interface with them?  Can I create the database from R?  If so, how?  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-06-15T13:39:32.613" Title="Seeking advice on database architecture -- given my problem, what tools should I learn?" Tags="|r|databases|sql|nosql|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19654" PostTypeId="1" AcceptedAnswerId="29233" CreationDate="2017-06-12T19:07:57.633" Score="6" ViewCount="1241" Body="&lt;p&gt;I'm wondering how Word2Vec is being constructed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read tutorials simply stating that we can train a skip grams neural network model and use the weights that are trained as word vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I've also seen this picture:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8xOi7.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8xOi7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If reading this diagram correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The CBOW and Skip Grams model are both trained with some inputs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The output of CBOW is used as an input to the middle neural network&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) The output of skip grams is used as an output to the middle neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output of CBOW is the a prediction of the center word given a context, and the output of skip grams is the prediction of the surrounding center word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These outputs are then used to train another set of neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence we first train the CBOW, and Skip-gram neural network, then train the middle neural network afterwards? And the input to the middle neural network is one hot encoded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the above interpretation correct?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="24119" LastEditDate="2017-06-13T10:09:17.487" LastActivityDate="2021-06-16T16:16:43.663" Title="Word2Vec - CBOW and Skip-Grams" Tags="|neural-network|nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19656" PostTypeId="2" ParentId="19648" CreationDate="2017-06-12T19:56:46.750" Score="3" Body="&lt;p&gt;At this data size, you can still use hierarchical clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can stop the clustering early when the similarity is too low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But all of these approaches are pretty inefficient. The proper approach is to not use clustering at all. Instead, use similarity search or, e.g., minhash.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-06-12T19:56:46.750" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19657" PostTypeId="2" ParentId="19605" CreationDate="2017-06-12T20:05:20.927" Score="3" Body="&lt;p&gt;For this example specifically, I would suggest visualizing the data using a &lt;a href=&quot;http://www.datablick.com/blog/2015/08/27/diy-chord-diagrams-in-tableau-by-noah-salvaterra&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chord Diagram&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Chord diagram from &lt;a href=&quot;http://www.delimited.io/blog/2013/12/8/chord-diagrams-in-d3&quot; rel=&quot;nofollow noreferrer&quot;&gt;Delimited.io&lt;/a&gt;:&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Vvili.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Vvili.png&quot; alt=&quot;Chord Diagram From http://www.delimited.io/blog/2013/12/8/chord-diagrams-in-d3]&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A chord diagram would allow you to see interactions and co-occurrences of likes between each of the heroes directly, including relative magnitude of the effects intuitively and immediately. You could also include other properties of the character (universe of origin, gender, time period of introduction etc) by use of color and/or positioning of the hero in question on the circumference of the diagram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As chord diagrams are a graph-based approach, you'd want to transform the individual observations you have into what would be a (this has to be a first for this term) hero-like-cofrequency matrix formatted as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----------+--------+------------+----------+&#xA;|           | Batman | Spiderrman | Superman |  &#xA;+-----------+--------+------------+----------+&#xA;| Batman    | 0      | 2          | 3        |&#xA;+-----------+--------+------------+----------+&#xA;| Spiderman | 2      | 0          | 1        | &#xA;+-----------+--------+------------+----------+&#xA;| Superman  | 3      | 1          | 0        |&#xA;+-----------+--------+------------+----------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this represents an undirected graph, and that it is symmetric about the identity column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you're using Python, you may want to look into using Bokeh to create the chord diagram. One tutorial for doing so is available &lt;a href=&quot;http://bokeh.pydata.org/en/latest/docs/gallery/chord_chart.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck, true believer!&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-06-12T20:19:11.030" LastActivityDate="2017-06-12T20:19:11.030" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19658" PostTypeId="2" ParentId="19653" CreationDate="2017-06-12T20:35:20.627" Score="3" Body="&lt;p&gt;I used R for several years but have since moved to Python and so I have a hard time understanding output.OLS's data hierarchy. Nonetheless, here are my thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;In-Memory Databases:&lt;/strong&gt; If you're struggling to fit your R object in memory, then I'm guessing the memory requirement is too great for your laptop (i.e., the problem isn't due a 32-bit installation of R). If that's the case, putting an in-memory db, like MongoDB, on your laptop will only transfer the memory problem to a new technology. In other words, 6 GB in R will be 6 GB in Mongo, so if your laptop simply can't handle 6 GB, then it doesn't matter how you put it there. You could set up a MongoDB on AWS, which could hold 100s of GB, but this can be expensive and also difficult if you don't have AWS experience. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, MongoDB is good at storing hierarchical data like lists and hash tables / dictionaries (I forgot what R's term for that is) and is fast if you have enough memory. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Relational Databases (RDMS):&lt;/strong&gt; I would recommend going this route and creating a wide and deep table. You can still get value out of an RDBMS even if your machine has limited memory because everything is efficiently stored on disk. If you create table indexes on the columns you expect to use/filter/select most, you can very quickly query the data that you want even if your data is multiple GB. If you don't create table indexes, your queries will be horribly slow, and you'll probably abandon the RDBMS effort entirely (so make sure to index). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A long time ago my go-to package for accessing MySQL through R was &lt;a href=&quot;https://cran.r-project.org/web/packages/RMySQL/index.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;RMySQL&quot;&gt;RMySQL&lt;/a&gt;. MySQL has been around for many years, so you'll be able to find most answers to your problems on Goggle, but MySQL user interfaces like Sequel Pro (for Mac only) have sort of fallen behind on maintenance, so setting up tables and databases can be a little more challenging than they used to be. You'll now have to go the command-line route. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Postgres has now replaced MySQL as the open-source DB of choice, but it lacks good user-interface support and so you'll probably have to set everything up via the command line. I started using Postgres after I stopped using R, so I never used a Postgres R package, but it looks like there is one &lt;a href=&quot;https://cran.r-project.org/web/packages/RPostgreSQL/index.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;adfad&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you go with MySQL or Postgres, you'll probably have to use the command-line outside of R to set up the database, but after that creating, dropping, joining, or querying tables can all be done through R. RDBMS systems are also great because you can guarantee data quality (no duplicates) by creating primary keys -- I found this feature very helpful for my R modeling. In any case, SQL is a very basic language that every data-oriented person should know so setting up an RDBMS and possibly learning SQL are skills worth learning. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2017-06-12T20:45:48.393" LastActivityDate="2017-06-12T20:45:48.393" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19660" PostTypeId="2" ParentId="19381" CreationDate="2017-06-13T03:37:04.670" Score="3" Body="&lt;p&gt;In python I needed to set the &lt;code&gt;label&lt;/code&gt; param in  &lt;code&gt;dtrain = xgb.DMatrix(X_train,label=y_train,feature_names=cfg_col_X)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS I was going to comment but missing rep points.&lt;/p&gt;&#xA;" OwnerUserId="33326" LastActivityDate="2017-06-13T03:37:04.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19661" PostTypeId="2" ParentId="19611" CreationDate="2017-06-13T03:41:16.913" Score="2" Body="&lt;p&gt;The notion of ensembles of models leading to better outcomes is widespread in the Data Science and Machine Learning communities. For example, the widely read text 'The Elements of Statistical Learning' by Hastie, Tibshirani and Friedman, devotes space to the general notion of model averaging (section 8.7-8.8) with further discussion of boosting in relation to trees (e.g. chapter 10). Hence, it is common to see examples of ensembled models in the wild, using any conceivable algorithm as the base, even to the point of effectively seeing ensembles of ensembles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically with respect to Support Vector Machines, there are at least a few attempts at improving SVM performance by using ensembles. A recent example is EnsemblesSVM (Claesen,De Smet, Suykens, De Moor - see homes.esat.kuleuven.be/~claesenm/ensemblesvm/). The authors promise that this algorithm will lead to better training times by ensembling lightly trained SVMs, and distribute a free implementation via the website above.&lt;/p&gt;&#xA;" OwnerUserId="18843" LastActivityDate="2017-06-13T03:41:16.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19666" PostTypeId="2" ParentId="18854" CreationDate="2017-06-13T07:22:12.913" Score="3" Body="&lt;p&gt;I think the original batch normalization paper proposes to use mean and standard deviation estimated on the train set. The adaptive batch normalization simply re-estimates them on the target domain (could be the test set, or some unlabeled data from the target domain). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I am wrong.&lt;/p&gt;&#xA;" OwnerUserId="33333" LastActivityDate="2017-06-13T07:22:12.913" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19670" PostTypeId="1" CreationDate="2017-06-13T09:10:08.140" Score="3" ViewCount="272" Body="&lt;p&gt;when using matlab command 'fitctree' for classification purpose,  and I change the order of the attributes I do not find the same Tree and thus the same classificaiton error? why? CART algorithm does take account on the attributes firstly introduced ? &lt;/p&gt;&#xA;" OwnerUserId="30962" LastActivityDate="2024-01-27T13:00:35.053" Title="Does CART algorithm takes into account in the order of the set of attributes?" Tags="|machine-learning|algorithms|random-forest|decision-trees|matlab|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19671" PostTypeId="1" AcceptedAnswerId="19677" CreationDate="2017-06-13T09:31:58.620" Score="2" ViewCount="552" Body="&lt;p&gt;I am using XGBoost for a time-series regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During development, i choose my validation set on last %10 percentage of data. Using timeseries split cross validation and grid-search, I got my best model on this with corresponding xgb hyperparameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how to choose validation set (for early stopping) on my production environment? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) i have chosen last %10 percentage of my data as validation set, but this set is also included on training data. therefore overfit. very sensitive to noisy data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) my predicted data (lets say Y) changes over time, when i choose random rows within last year (%10 amount) and dont include them in training set, it gave me worse results on production than first option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) when i choose last week's data as validation data, not included in training set, it gave better result on 2. option. but i am not including last week's data to training procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Or do i need a validation set on my production environment? should i set iteration count from the experiments done in development stage? (e.g i got best result on 10k th iteration, so i should limit my production-setup iteration count with 10k without using validation set at all?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can i choose validation set for my production environment? Best practices, or are there any tips/tricks for this? &lt;/p&gt;&#xA;" OwnerUserId="33340" LastEditorUserId="33340" LastEditDate="2017-07-17T10:27:11.620" LastActivityDate="2017-07-17T10:27:11.620" Title="How to choose validation set for production environment?" Tags="|time-series|regression|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19672" PostTypeId="2" ParentId="19670" CreationDate="2017-06-13T10:27:25.817" Score="0" Body="&lt;p&gt;A possible answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check to see if you have nominal variables with more than a reasonable number of levels. If that is the case then a heuristic is applied to select the best split. One such heuristic is to group levels on left to right, and that might affect the order. Anyway, it is reasonable to expect that the order of levels should not affect the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See this link for detalied explanation on how this is implemented on matlab&#xA;&lt;a href=&quot;https://uk.mathworks.com/help/stats/splitting-categorical-predictors-for-multiclass-classification.html#btt1xp_&quot; rel=&quot;nofollow noreferrer&quot;&gt;matlab documentation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2017-06-13T10:27:25.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19673" PostTypeId="1" CreationDate="2017-06-13T12:15:10.897" Score="4" ViewCount="285" Body="&lt;p&gt;I'm looking at implementing &lt;a href=&quot;https://stats.stackexchange.com/q/284217/29949&quot;&gt;some&lt;/a&gt; &lt;a href=&quot;https://arxiv.org/abs/1705.02445&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural networks&lt;/a&gt; from research papers. However, I'm concerned about which framework I should use, because I'm uncertain if the networks form static computation graphs (making them suitable for TensorFlow/Keras) or if they are dynamic (making them more suitable for PyTorch).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some rules of thumb to determine if a network requires a dynamic computation graph?&lt;/p&gt;&#xA;" OwnerUserId="26686" LastActivityDate="2018-10-26T09:06:01.007" Title="How to determine if a neural-network has a static computation graph?" Tags="|neural-network|tensorflow|keras|pytorch|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19674" PostTypeId="1" AcceptedAnswerId="29474" CreationDate="2017-06-13T12:25:49.140" Score="0" ViewCount="422" Body="&lt;p&gt;I have a binary feature that i want to use it with textual features i.e. unigrams. I use logistic regression and TF/IDF for representing text. So i simply add a unique feature, say ss or oo, to text of each instances. But in practice, i see adding more number of these features to instances, say two oo or ss or more get me a better results. What is the reason? How these weights improve the classification results? Should not logistic regression can get more weights to this features instead of weighting them by hand?&lt;/p&gt;&#xA;" OwnerUserId="33245" LastEditorUserId="2932" LastEditDate="2017-06-13T14:04:34.500" LastActivityDate="2018-03-23T14:04:38.277" Title="Combining binary features with textual ones" Tags="|classification|text-mining|logistic-regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19675" PostTypeId="1" CreationDate="2017-06-13T12:33:37.307" Score="2" ViewCount="1982" Body="&lt;p&gt;i’am a beginner in scikit-learn and i’ve a little problem when using feature selection module VarianceThreshold, the problem is when i set the variance &lt;code&gt;Var[X]=.8*(1-.8)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;it is supposed to remove all features (that have the same value in all samples) which have the probability $p&amp;gt;0.8$.&#xA;in my case the fifth column should be removed, &lt;code&gt;p=8/10&amp;gt;(threshold=0,7)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#####################################&#xA;&#xA;from sklearn.feature_selection import VarianceThreshold&#xA;X=[[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,2,1,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0.00,0.00,0.00,0.00,0.50,1.00,0.00,10,3,0.30,0.30,0.30,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,253,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],&#xA;[0,2,3,1,223,185,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,4,0.00,0.00,0.00,0.00,1.00,0.00,0.00,71,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00]]&#xA;sel=VarianceThreshold(threshold=(.7*(1-.7)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this is what i get when running the script&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; sel.fit_transform(X)&#xA;&#xA;array([[ 1., 105., 146., 1., 1., 255., 254.],&#xA;[ 1., 105., 146., 1., 1., 255., 254.],&#xA;[ 1., 105., 146., 1., 1., 255., 254.],&#xA;[ 1., 105., 146., 2., 2., 255., 254.],&#xA;[ 1., 105., 146., 2., 2., 255., 254.],&#xA;[ 1., 105., 146., 2., 2., 255., 255.],&#xA;[ 2., 29., 0., 2., 1., 10., 3.],&#xA;[ 1., 105., 146., 1., 1., 255., 253.],&#xA;[ 1., 105., 146., 2., 2., 255., 254.],&#xA;[ 3., 223., 185., 4., 4., 71., 255.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the second column here should not apear.&#xA;thanks;)&lt;/p&gt;&#xA;" OwnerUserId="33349" LastEditorUserId="2932" LastEditDate="2017-06-13T18:06:53.623" LastActivityDate="2024-02-15T02:01:30.147" Title="Variance threshold with python problem" Tags="|machine-learning|python|scikit-learn|feature-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19676" PostTypeId="1" CreationDate="2017-06-13T13:27:09.750" Score="1" ViewCount="167" Body="&lt;p&gt;What is meant by &lt;em&gt;energy spectrum&lt;/em&gt; in LSI(Latent Semantic Indexing)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing topic modeling with &lt;a href=&quot;https://radimrehurek.com/gensim/models/lsimodel.html#gensim.models.lsimodel.LsiModel&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;gensim's&lt;/strong&gt; LsiModel&lt;/a&gt;, and part of the output per chunk is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INFO : preparing a new chunk of documents&#xA;INFO : using 100 extra samples and 2 power iterations&#xA;INFO : 1st phase: constructing (100000, 600) action matrix&#xA;INFO : orthonormalizing (100000, 600) action matrix&#xA;INFO : 2nd phase: running dense svd on (600, 20000) matrix&#xA;INFO : computing the final decomposition&#xA;INFO : keeping 500 factors (discarding 6.560% of energy spectrum)&#xA;INFO : merging projections: (100000, 500) + (100000, 500)&#xA;INFO : keeping 500 factors (discarding 0.843% of energy spectrum)&#xA;INFO : processed documents up to #1400000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above output is from 1,400,000 documents into the process, (out of aproxx. 3,500,000), and it appears to discard less and less for each chunk. The 2nd chunk was higher:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INFO : keeping 500 factors (discarding 6.556% of energy spectrum)&#xA;INFO : merging projections: (100000, 500) + (100000, 500)&#xA;INFO : keeping 500 factors (discarding 13.469% of energy spectrum)&#xA;INFO : processed documents up to #40000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure whether the &lt;code&gt;discarding X % of energy spectrum&lt;/code&gt; is better with high or low numbers. Is the &quot;energy&quot; analogous to entropy? Does discarding mean it looses information, or does it mean the Singular Value Decomposition is getting better and better with more information? (Or none of the above)&lt;/p&gt;&#xA;" OwnerUserId="33352" LastEditorUserId="33352" LastEditDate="2017-06-13T13:32:35.507" LastActivityDate="2017-06-13T13:32:35.507" Title="What is &quot;energy spectrum&quot; in Latent Semantic Indexing (LSI)?" Tags="|topic-model|gensim|lsi|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19677" PostTypeId="2" ParentId="19671" CreationDate="2017-06-13T13:45:48.947" Score="2" Body="&lt;p&gt;If you're simply re-training the XGBoost model periodically in order to account for the changing nature of your data, the best option is to hold out a recent set of data for testing (some variation of your option 3). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you mention, option 1- that is, training on the entire available set and validating on the most recent 10 percent of the data, is very likely to overfit (and therefore overestimate its performance on future data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 2 has the downfall that you're attempting to predict information that you have post-hoc (after the fact) information about- that is, you know what subsequent observation values were, which is very relevant to a time-series prediction. For example, if you're trying to predict what the value is tomorrow, and you know what the value is for the day after tomorrow already, you can do a much better job of predicting tomorrow. Unfortunately, you can never know this before the day after tomorrow begins. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, option 3 is the most valuable practice to determine the likelihood of your model to accurately predict future observations- you're using only data that would be available from the point of prediction (in the past relative to the predicted period) and holding that data out of the training set for the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best practice in these circumstances is to hold yourself accountable for &quot;not cheating&quot;- that is, try your best to provide a fair and honest experiment on the prediction of your model on data it hasn't seen and would be available in the context of a future use situation.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-06-13T13:45:48.947" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19678" PostTypeId="2" ParentId="19648" CreationDate="2017-06-13T16:28:25.120" Score="3" Body="&lt;p&gt;Clustering does not seem like the right approach here. Following on Anony-Mousse, a similarity matrix doesn't seem right either. Locality Sensitive Hashing (ie minhash) would be a good way to group all similar items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Creating and searching through a similarity matrix is much more inefficient (you are computing the similarity  of $\binom{n}{2} $ pairs ), so I would suggest going right to LSH rather than generating the matrix. This will group the similar items as you desire. &lt;/p&gt;&#xA;" OwnerUserId="33353" LastActivityDate="2017-06-13T16:28:25.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19679" PostTypeId="1" AcceptedAnswerId="19683" CreationDate="2017-06-13T16:38:14.560" Score="4" ViewCount="92" Body="&lt;p&gt;We are capturing emotions as survey responses. We need to assign values for the responses(emotions) for analysis purposes. Is there an optimum range that can be assigned to achieve this? (like from -100 to 100). An example of a question and a set of answers are as follow.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Question: &amp;quot;How are you feeling today?&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;Answers: Terrible, Sad, Ok, Good, Great&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A suitable approach we could think of is to assign values from 1 to 100 with equal distance. Is this statistically valid? What are the things that we should consider when achieving this? In this case, only positive integers are assigned because we need to calculate statistics like weighted average. Can't we assign negative numbers as well?&lt;/p&gt;&#xA;" OwnerDisplayName="user33293" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-06-13T17:45:13.607" Title="Range to define emotions" Tags="|sentiment-analysis|weighted-data|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19681" PostTypeId="1" CreationDate="2017-06-13T17:31:28.477" Score="1" ViewCount="2537" Body="&lt;p&gt;As per the Orange video tutorial: &lt;em&gt;youtu.be/D6zd7m2aYqU?list=PLmNPvQr9Tf-ZSDLwOzxpvY-HrE0yv-8Fy&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I'm trying to make predictions&lt;/strong&gt; on a new dataset based upon one that's already been 'coded' - both data sets are formatted exactly the same, where the 'training' set has all entries marked as either 1 or 0, see below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6eF6J.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6eF6J.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whereas the second set contains no values, as I hope the 'predictions' feature will try to guess the status:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NcZde.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NcZde.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trouble is, when I look at the predictions widget, it does nothing, displaying &lt;strong&gt;'Data does not have the same target as predictors'&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've only been using Orange for two days, so I imagine it's a really simple error on my part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;John&lt;/p&gt;&#xA;" OwnerUserId="33361" LastActivityDate="2018-12-03T02:39:21.233" Title="Using Predictions: Data does not have the same target as predictors" Tags="|orange|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19682" PostTypeId="1" CreationDate="2017-06-13T17:39:16.070" Score="1" ViewCount="3292" Body="&lt;p&gt;I have used pip to install orange and the installation seemed fine.  There was not Orange icon on my ubuntu 16.04 desktop and tried orange3 in the command line, it didn't work, and tried python3 -m orange and python3 -m orange 3 and neither worked as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use orange?&lt;/p&gt;&#xA;" OwnerUserId="33362" LastActivityDate="2021-06-23T15:20:48.523" Title="How do I access orange on Ubuntu 16.04" Tags="|orange|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19683" PostTypeId="2" ParentId="19679" CreationDate="2017-06-13T17:45:13.607" Score="5" Body="&lt;p&gt;The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Asking if equally distancing the emotions is statistically correct does not make sense. This entirely depends on your use case and opinions. Also, there is absolutely no reason why you can't use negative numbers in a weighted average. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you mentioned what you are doing and how you are evaluating emotion, there might be more to say.&lt;/p&gt;&#xA;" OwnerUserId="33353" LastActivityDate="2017-06-13T17:45:13.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19685" PostTypeId="1" CreationDate="2017-06-13T21:01:39.193" Score="1" ViewCount="207" Body="&lt;p&gt;I am currently working on text classifier with some pretty unique characteristics. The data is composed of about 2K categories but 98% of the data lives in just one of those 2K categories. However, our main problem is that we have a list of additional categories but have yet to see any transactions that fall into those categories (we know they will eventually).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Does it make sense to &quot;prime&quot; the model with these additional categories? For example, I could just add examples of these categories to my training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If priming is an acceptable are there any best practices or guides in doing so? I am wondering if I should just add the additional transactions to my training corpus, randomize them with other text, how many examples of the training set they should represent, etc..&lt;/p&gt;&#xA;" OwnerUserId="33367" LastEditorUserId="33367" LastEditDate="2017-06-14T16:25:07.497" LastActivityDate="2017-07-14T17:49:41.283" Title="Prime model with mock data, good idea or not?" Tags="|machine-learning|classification|text-mining|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19686" PostTypeId="2" ParentId="19129" CreationDate="2017-06-13T21:19:54.150" Score="1" Body="&lt;p&gt;I run into the same problem with BRAT from time to time. The way I solved it for a folder with &lt;code&gt;.txt&lt;/code&gt; files is to run this Bash one-liner directly in the shell when inside the directory with these files:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for f in *.txt; do touch $(basename $f .txt)&quot;.ann&quot;; done&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is does is the following: With a for-loop, it loops over every &lt;code&gt;.txt&lt;/code&gt; file, and creates an empty file with &lt;code&gt;touch&lt;/code&gt; having the same basename, but a &lt;code&gt;.ann&lt;/code&gt; extension instead of a &lt;code&gt;.txt&lt;/code&gt; extension.&lt;/p&gt;&#xA;" OwnerUserId="33368" LastActivityDate="2017-06-13T21:19:54.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19687" PostTypeId="1" CreationDate="2017-06-14T05:33:03.170" Score="1" ViewCount="227" Body="&lt;p&gt;I need to extract the source and destination terms from the text&#xA; documents using text mining / NLP / Information Retrieval ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example input:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;I am travelling from New York to London.&lt;/li&gt;&#xA;  &lt;li&gt;I am heading towards playground from home.&lt;/li&gt;&#xA;  &lt;li&gt;I will be going to Sweden from Boston.&lt;/li&gt;&#xA;  &lt;li&gt;I was flying from School to Home.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The output can be as follows :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S. No. |  source    | Destination&#xA;------ |  ----------|------------&#xA;      1| New York   | London&#xA;      2| playground | home&#xA;      3| Sweden     | Boston&#xA;      4| School     | Home&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33379" LastEditorUserId="17564" LastEditDate="2017-06-21T11:40:26.883" LastActivityDate="2017-06-21T11:40:26.883" Title="Extracting Part of Speech (Source and Destinations) using text mining/NLP?" Tags="|machine-learning|nlp|text-mining|nltk|stanford-nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19689" PostTypeId="1" CreationDate="2017-06-14T07:05:19.683" Score="2" ViewCount="3517" Body="&lt;p&gt;I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked better for me. I'd like to evaluate both but I can'f find any useful validation or metric to compare these two. How is it possible to compare these two with a useful metric? &lt;/p&gt;&#xA;" OwnerUserId="33382" LastActivityDate="2017-06-14T11:22:46.663" Title="How to compare LDA and TF-IDF?" Tags="|python|text-mining|k-means|model-evaluations|lda|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19692" PostTypeId="1" AcceptedAnswerId="19694" CreationDate="2017-06-14T08:12:41.713" Score="6" ViewCount="258" Body="&lt;p&gt;In stacked generalization, if I understood well, we divide the training set into train/test set. We use train set to train M models, and make predictions on test set. Then we use the predictions as input of a new model. Thus, the new training set will have M features corresponding to the M models predictions. Finally, we use the last model to make the final predictions. First, is my understanding correct ? If so, how is it possible to use the last model to make predictions as it has different features. &lt;/p&gt;&#xA;" OwnerUserId="33386" LastActivityDate="2017-06-14T10:20:46.337" Title="How predictions of level 1 models become training set of a new model in stacked generalization." Tags="|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19693" PostTypeId="2" ParentId="19687" CreationDate="2017-06-14T08:50:53.203" Score="2" Body="&lt;p&gt;Named Entity Recognition is technique which can be used here. Location is one of the 3 most studied classes (with Person and Organization). Stanford NLP has an &lt;a href=&quot;http://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;open source&lt;/a&gt; Java implementation that is extremely powerful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Example  let say sentence is &quot;&lt;strong&gt;i will be going to Sweden from Boston.&lt;/strong&gt;&quot; &#xA;&lt;a href=&quot;https://i.stack.imgur.com/BlSsB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BlSsB.jpg&quot; alt=&quot;Stanford NER Demo&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now here you can use &lt;a href=&quot;https://lost-contact.mit.edu/afs/cs.pitt.edu/projects/nltk/docs/tutorial/regexps/nochunks.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;regular expression&lt;/a&gt; to detect these &lt;strong&gt;LOCATION&lt;/strong&gt; tags.&lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-06-14T08:50:53.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19694" PostTypeId="2" ParentId="19692" CreationDate="2017-06-14T09:31:21.803" Score="4" Body="&lt;p&gt;You have created a model pipeline and must run all trained models (&quot;lower level&quot; ones first) in order to make a prediction on new data using the stack.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With test data set, it is slightly easier, since you can store the predictions from the &quot;level 1&quot; models when testing them, and only run the final model across this stored data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to your brief description, &lt;em&gt;usually&lt;/em&gt; to avoid bias from re-using training data, you would use k-fold cross-validation or similar mechanism, and your training data for the final model should be the cv predictions from each model. You do not want to use the training predictions from those models, because they are likely to be overfit whilst &quot;level 1&quot; test and production predictions will not be, and this would introduce population differences between train and test data in your &quot;level 2&quot; model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also quite a common variation to use the M new features from your &quot;level 1&quot; models alongside some or all of the original features. This gives the meta model more data to base its decision on when deciding the relative weights between the first stage models (assuming this top-level model is non-linear).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-14T10:20:46.337" LastActivityDate="2017-06-14T10:20:46.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19695" PostTypeId="2" ParentId="19641" CreationDate="2017-06-14T10:04:45.710" Score="1" Body="&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;df = pd.DataFrame(np.random.randint(2,size=(10, 4)), columns=list('ABCD'))&#xA;df&#xA;&#xA;&#xA;    A   B   C   D&#xA;0   0   1   1   1&#xA;1   1   0   1   0&#xA;2   1   0   0   1&#xA;3   0   1   1   1&#xA;4   0   0   1   1&#xA;5   0   1   1   1&#xA;6   0   0   0   0&#xA;7   1   1   1   0&#xA;8   1   1   1   0&#xA;9   1   0   1   0&#xA;&#xA;values = df.apply(pd.value_counts)&#xA;values&#xA;&#xA;    A   B   C   D&#xA;0   5   5   2   5&#xA;1   5   5   8   5&#xA;&#xA;new_df = pd.DataFrame()&#xA;for x in df.columns:&#xA;    new_df[x] = df[x].apply(lambda row: values[x][row])&#xA;new_df&#xA;&#xA;&#xA;    A   B   C   D&#xA;0   5   5   8   5&#xA;1   5   5   8   5&#xA;2   5   5   2   5&#xA;3   5   5   8   5&#xA;4   5   5   8   5&#xA;5   5   5   8   5&#xA;6   5   5   2   5&#xA;7   5   5   8   5&#xA;8   5   5   8   5&#xA;9   5   5   8   5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I created a df with random integers 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then count them by column in values df.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then loop through each column and replace each cell with their respective count. Being random it is close to 5/5 split, but you can see the C column has 8/2 split.&lt;/p&gt;&#xA;" OwnerUserId="32164" LastEditorUserId="32164" LastEditDate="2017-06-14T10:11:06.443" LastActivityDate="2017-06-14T10:11:06.443" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19696" PostTypeId="2" ParentId="19682" CreationDate="2017-06-14T10:25:10.137" Score="1" Body="&lt;p&gt;You should fire up a python console and try &lt;code&gt;import orange&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it doesn't give you any errors then the installation succeeded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the GUI, try something like:&lt;code&gt;python /usr/local/lib/python2.7/dist-packages/Orange/OrangeCanvas/orngCanvas.pyw&lt;/code&gt; (change python version)&lt;/p&gt;&#xA;" OwnerUserId="32164" LastActivityDate="2017-06-14T10:25:10.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19697" PostTypeId="1" CreationDate="2017-06-14T10:32:36.433" Score="2" ViewCount="1323" Body="&lt;p&gt;My question is rather simple what does the parameter scale_pos_weight in xgboost do? I know typically it should be $\frac{sum(negative cases)}{sum(positive cases)}$.&lt;br&gt;&#xA;Does it oversample the minority class by that ratio or does it undersample the majority class by inverse of that ratio? Or something else?&lt;br&gt;&#xA;Also I would like to know if during cross validation in xgbcv, does the sampling happen on the test part of the cross-validation also or only the train part is affected by scale_pos_weight? Because i've heard that sampling should never be applied to test as it gives over-optimistic results. &lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2019-08-12T19:02:05.257" Title="scale_pos_weight Xgboost" Tags="|classification|xgboost|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19698" PostTypeId="2" ParentId="19682" CreationDate="2017-06-14T10:37:46.747" Score="2" Body="&lt;p&gt;From the &lt;a href=&quot;https://pypi.python.org/pypi/Orange3&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange3 Pypi documentation&lt;/a&gt; :&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;To start Orange GUI from the command line, assuming it was successfully installed, run:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;orange-canvas # or python3 -m Orange.canvas&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33390" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-06-15T12:40:22.897" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19699" PostTypeId="1" CreationDate="2017-06-14T10:52:19.077" Score="-1" ViewCount="97" Body="&lt;p&gt;It is about insurance fraud. The image 1 shows the result of analysis with likelihood of claim referral, which is the target variable, from the previous analysis, image 2, on historical referral data. How should I interpret the likelihood in the image?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Link to the document: &lt;a href=&quot;http://support.sas.com/resources/papers/proceedings14/1837-2014.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://support.sas.com/resources/papers/proceedings14/1837-2014.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nFg7B.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nFg7B.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QFXB3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QFXB3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Image 2&lt;/p&gt;&#xA;" OwnerUserId="30856" LastEditorUserId="30856" LastEditDate="2017-06-14T11:10:35.207" LastActivityDate="2017-10-12T16:50:47.433" Title="What does &quot;likelihood&quot; mean in the image?" Tags="|machine-learning|classification|data-mining|predictive-modeling|statistics|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19700" PostTypeId="1" AcceptedAnswerId="19706" CreationDate="2017-06-14T11:21:20.230" Score="4" ViewCount="187" Body="&lt;p&gt;According to article about LSTM &lt;a href=&quot;http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, I know that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;it allows us to score arbitrary sentences based on how likely they are to occur in the real world. This gives us a measure of grammatical and semantic correctness. Such models are typically used as part of Machine Translation systems. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But, it seems that this article doesn't point out how to compute the score with &lt;code&gt;LSTM&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any way to compute the score?&lt;/p&gt;&#xA;" OwnerUserId="33392" LastActivityDate="2017-06-14T12:10:12.067" Title="How to score arbitrary sentences based on how likely they are to occur in the real world?" Tags="|neural-network|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19701" PostTypeId="2" ParentId="19689" CreationDate="2017-06-14T11:22:46.663" Score="2" Body="&lt;p&gt;If you have the ground truth value of the documents (their topics) all you gotta do is pick a metric and compare results. For classification problems, as yours, a common metric would be f1_score; reference: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26899" LastActivityDate="2017-06-14T11:22:46.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19702" PostTypeId="2" ParentId="19615" CreationDate="2017-06-14T11:29:24.650" Score="1" Body="&lt;p&gt;that's possibly due to poor parameter tuning. &lt;br&gt;&#xA;Try reducing C for SVR and increasing n_estimators for RFR. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice approach is to gridsearch through the parameter, and plot the metric result. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing that might help is to normalize the parameters (sklearn.preprocessing.StandardScaler) and to remove the skew from the target (usually log transform or 1/target transform works better)&lt;/p&gt;&#xA;" OwnerUserId="26899" LastActivityDate="2017-06-14T11:29:24.650" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19703" PostTypeId="2" ParentId="19583" CreationDate="2017-06-14T11:34:52.010" Score="0" Body="&lt;p&gt;Its kind of difficult to avoid a large processing time when you have many parameters and a large sample. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better method than gridsearch is Bayesian Optimization, this library even has done examples for xgboost ( &lt;a href=&quot;https://github.com/fmfn/BayesianOptimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fmfn/BayesianOptimization&lt;/a&gt; ) &lt;br&gt;&lt;br&gt;&#xA;A good trick as well is to take a reduced sample of your data set, if you have over 2 million samples, you might acquire very similar results with smaller subsets (given that they have a similar distribution).&lt;/p&gt;&#xA;" OwnerUserId="26899" LastActivityDate="2017-06-14T11:34:52.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19704" PostTypeId="1" CreationDate="2017-06-14T11:36:49.617" Score="-1" ViewCount="1419" Body="&lt;p&gt;How is a time series like the Rossmann Kaggle competition used to forecast sales? The simplest solution  I saw is a random forest. There the data for every time point is used to feed the random forests. So there is no time dependence because I use every data at each time as independent. Thats also good. But what I a doing if I want to solve this with a neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my input layer corresponds to every time point (for the same time independence)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the data mining Cup 2017 there is a similiar competition. there are over 5000 products with attributes like if it was clicked or drawn to the basket or ordered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since there are intreractions between the user actions like clicking at the product, history come important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to train a neural network for each product to forecast revenue?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastEditorUserId="32963" LastEditDate="2017-06-14T11:55:27.940" LastActivityDate="2017-08-23T21:33:27.153" Title="Sales prediction with neural networks" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19706" PostTypeId="2" ParentId="19700" CreationDate="2017-06-14T11:58:32.407" Score="6" Body="&lt;p&gt;Typically you would use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Perplexity&quot; rel=&quot;noreferrer&quot;&gt;perplexity&lt;/a&gt; value. For example, if your LSTM model is word-based and you have a sentence $[x_1, x_2 . . . x_N]$, and your model predicts the words that appear in that sentence with probability $p(x_i|x_0..x_{i-1})$ (where $x_0$ is a &quot;start token&quot; or whatever you use to start your RNN prediction sequence). Then you might quote a per-word perplexity for that sentence under your model as&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$- \frac{1}{N}\sum_{i=1}^N log_2(p(x_i|x_0..x_{i-1}))$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Using an LSTM to predict consecutive words, it is practical to construct an array of probabilities $[p_1, p_2 . . . p_N]$ by running the network on the sentence and noting the probabilities for the correct matching word - i.e. $p_i = $ the predicted probability of the correct class $x_i$ at each step, which simplifies the expression:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$- \frac{1}{N}\sum_{i=1}^N log_2(p_i)$&lt;/h2&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-14T12:10:12.067" LastActivityDate="2017-06-14T12:10:12.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19707" PostTypeId="1" AcceptedAnswerId="19710" CreationDate="2017-06-14T12:24:38.110" Score="3" ViewCount="27641" Body="&lt;p&gt;I have a pandas data frame like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(index) 0           sie&#xA;&#xA;0       1997-01-01  11.2&#xA;1        1997-01-03  12.3&#xA;2        1997-01-04  11.5&#xA;...&#xA;12454    2017-02-01  13.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to use SVM to predict the future values of the &lt;code&gt;sie&lt;/code&gt;. How can I implement python code to predict these values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model = svm.SVR().fit(df[0],df['sie'])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it is giving me this error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ValueError: Found input variables with inconsistent numbers of samples: [1, 12455]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although both &lt;code&gt;df[0]&lt;/code&gt;and&lt;code&gt;df['sie']&lt;/code&gt; have same shape of &lt;code&gt;(12455,)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: &lt;em&gt;I don't have continuous data (some dates, in between, are missing), also values in &lt;code&gt;0&lt;/code&gt; are &lt;code&gt;datetime.date()&lt;/code&gt; objects.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="29169" LastEditDate="2019-06-07T16:55:26.940" LastActivityDate="2019-06-07T16:55:26.940" Title="Time series forecast using SVM?" Tags="|python|predictive-modeling|time-series|svm|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="19708" PostTypeId="2" ParentId="19707" CreationDate="2017-06-14T13:11:51.537" Score="1" Body="&lt;p&gt;I used to solve the value error:&#xA;&lt;code&gt;model = svm.SVR().fit(np.transpose(np.matrix(df['Dates'])),np.transpose(np.matrix(df['sie'])))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More Info: &lt;a href=&quot;https://stackoverflow.com/questions/30813044/sklearn-found-arrays-with-inconsistent-numbers-of-samples-when-calling-linearre&quot;&gt;https://stackoverflow.com/questions/30813044/sklearn-found-arrays-with-inconsistent-numbers-of-samples-when-calling-linearre&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33020" LastActivityDate="2017-06-14T13:11:51.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19709" PostTypeId="2" ParentId="19699" CreationDate="2017-06-14T13:13:22.820" Score="0" Body="&lt;p&gt;I would imagine likelihood is the same as it is in statistics:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Likelihood_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Likelihood_function&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33353" LastActivityDate="2017-06-14T13:13:22.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19710" PostTypeId="2" ParentId="19707" CreationDate="2017-06-14T13:18:58.310" Score="4" Body="&lt;p&gt;Here, a very good article: &lt;a href=&quot;http://machinelearningmastery.com/time-series-forecasting-supervised-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/time-series-forecasting-supervised-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a few words, define a window of size n and that is the size of your feature vector. Reshape the dataset and play. &lt;/p&gt;&#xA;" OwnerUserId="11007" LastActivityDate="2017-06-14T13:18:58.310" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19711" PostTypeId="1" AcceptedAnswerId="46692" CreationDate="2017-06-14T13:46:30.580" Score="5" ViewCount="2039" Body="&lt;p&gt;I'm busy with a supervised machine learning problem where I am predicting contract cancellation. Although a lengthy question, I do hope someone will take the time as I'm convinced it will help others out there (I've just been unable to find ANY solutions that have helped me)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I have the following two datasets:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1) &quot;Modelling Dataset&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contains about 400k contracts (rows) with 300 features and a single label (0 = &quot;Not Cancelled&quot;, 1 = &quot;Cancelled&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row represents a single contract, and each contract is only represented once in the data. There are 350k &quot;Not Cancelled&quot; and 50k &quot;Cancelled&quot; cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features are all extracted as at a specific date for each contract. This date is referred to as the &quot;Effective Date&quot;. For &quot;Cancelled&quot; contracts, the &quot;Effective Date&quot; is the date of cancellation. For &quot;Not Cancelled&quot; contracts, the &quot;Effective Date&quot; is a date say 6 months ago. This will be explained in a moment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;2) &quot;Live Dataset&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contains 300k contracts (rows) with the same list of 300 features. All these contracts are &quot;Not Cancelled&quot; of course, as we want to predict which of them will cancel. These contracts were followed for a period of 2 months, and I then added a Label to this data to indicate whether it actually ended up cancelling in those two months: 0 = &quot;Not Cancelled&quot;, 1 = &quot;Cancelled&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get amazing results on the &quot;Modelling Dataset&quot; (random train/test split) (eg Precision 95%, AUC 0.98), but as soon as that model is applied to the &quot;Live Dataset&quot;, it performs poorly (cannot predict well which contracts ends up cancelling) (eg Precision 50%, AUC 0.7). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the Modelling Dataset, the results are great, almost irrespective of model or data preparation. I test a number of models (E.g. SkLearn random forest, Keras neural network, Microsoft GbmLight, SkLearn Recursive feature elimination). Even with default settings, the models generally perform well. I've standardized features. I've binned features to attempt improving how well it will generalize. Nothing has help it generalize to the &quot;Live Dataset&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My suspicion:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my mind, this is not an over-training issue because I've got a test set within the &quot;Modelling Dataset&quot; and those results are great on the test set. It is not a modelling or even a hyper-parameter optimization issue, as the results are already great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also investigated whether there are significant differences in the profile of the features between the two datasets by looking at histograms, feature-by-feature. Nothing is worryingly different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect the issue lies therein that the same contracts that are marked as &quot;Not Cancelled&quot; in the &quot;Modelling Dataset&quot;, which the model trains to recognize &quot;Not Cancelled&quot; of course, is basically the exact same contracts in the &quot;Live Dataset&quot;, except that 6 months have now passed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that the features for the &quot;Not Cancelled&quot; cases has not changed enough to now make the model recognize some of them as about to be &quot;Cancelled&quot;. In other words, the contracts have not moved enough in the feature space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, does my suspicion sound correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, if I've stated the problem to be solved incorrectly, how would I then set up the problem-statement if the purpose is to predict cancellation of something like contracts (when the data on which you train will almost certainly contain the data one which you want to predict)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the record, the problem-statement I've used here is similar to the way others have done this. And they reported great results. But I'm not sure that the models were ever tested in real live. In other cases, the problem to be solved was lightly different, e.g. hotel booking cancellations, which is different because there a stream of new incoming bookings and booking duration is relatively short, so no bookings in common between the modelling and live dataset. Contracts on the other hand have long duration and can cancel at any time, and sometimes never.&lt;/p&gt;&#xA;" OwnerUserId="33393" LastActivityDate="2019-03-05T09:04:35.350" Title="Predicting contract churn/cancellation: Great model results does not work in the real world" Tags="|machine-learning|python|classification|scikit-learn|churn|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19713" PostTypeId="1" AcceptedAnswerId="20048" CreationDate="2017-06-14T13:53:36.757" Score="-2" ViewCount="356" Body="&lt;p&gt;Why is training classifier on extracted features (from inceptionV3) is so much faster than simply stack classifier on topless inception model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my experience extracting features + training classifier takes roughly 3-10 times faster. &lt;/p&gt;&#xA;" OwnerUserId="31595" LastActivityDate="2017-08-27T23:44:28.733" Title="Transfer learning inceptionv3" Tags="|deep-learning|keras|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19714" PostTypeId="1" AcceptedAnswerId="19716" CreationDate="2017-06-14T14:03:55.650" Score="5" ViewCount="1582" Body="&lt;p&gt;I'm trying to classify several websites by category (finance, health-care, IT, etc...). I have at my disposal the content of the pages of the websites, and I use the words to classify. For now, I have manually classified some website to train a naive bayes model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm mostly interested in a high precision (e.g a classified website must be in the right category), I would like to add an &quot;undefined&quot; category in which a website would end up if it is not close enough to the other categories. To be clear, it's not a problem for me if a website is not classified, it's a problem if it is misclassified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there algorithms that would allow this, or a way to train an &quot;undefined&quot; category ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My best guess for now is to train something like a random forest and define a &quot;minimum score&quot; below which a website is &quot;undefined&quot;.&lt;/p&gt;&#xA;" OwnerUserId="23679" LastActivityDate="2017-06-14T17:32:52.183" Title="How can I have an &quot;undefined&quot; category in multi-class classification" Tags="|classification|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19715" PostTypeId="1" CreationDate="2017-06-14T14:38:14.857" Score="2" ViewCount="44" Body="&lt;p&gt;I want to perform feature selection, having 128 real-valued standardized features and 1/0 labels. Below are feature a5 density histograms for Classes 1 and 0. The data is skewed, so that Class 1 is about 5% weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/n6UmU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/n6UmU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next I subtract right curve from left, so that to remove &quot;normal&quot; (relative to this feature) distribution.  I assume that if 50% of feature values lay in e.g. [0.1-0.2] in BOTH classes this means this range is nothing special, regarding my classes. That's why I make subtraction. This hopefully gives me ranges (positive values of resulting curve) where this feature contribute to choosing Class0. Is this assumption correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/87Eoq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/87Eoq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I build graphs this way for all features. Below are two different features. I suppose that leftmost is better than rightmost, giving clearer distinction in correlating to one class or another (right is noisier) and the rightmost could be removed, if need to reduce feature number. Is this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/152YT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/152YT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32989" LastActivityDate="2017-06-14T14:38:14.857" Title="Am I correct in finding correlations" Tags="|feature-selection|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19716" PostTypeId="2" ParentId="19714" CreationDate="2017-06-14T14:59:17.057" Score="6" Body="&lt;p&gt;Absolutely you can create an approach that forces high-precision class tagging algorithm (at the natural cost of recall). What's more- you can do this with (at least) any method that provides a percentage calue for predictions, which is the vast majority of classifiers. The key is, as you mention, to find the minimum acceptable value of precision and cut the predictions at that value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a minimum precision is your only constraint and your solution is not sensitive to the recall (getting all or the highest possible proportion of the websites correctly classified), this is a very simple matter. Some lower percentage of your observations will be classified, but those that are will be more likely to be correctly classified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example- if your Precision floor is 70%, your cut could look something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HbXkC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HbXkC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obeservation predictions falling into the green segment could be classified as positive examples, and predictions falling into the red segment would be unclassified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach would be sufficient for Naive Bayes. Some other approaches (SVM, gradient boosting machines etc) may benefit from a custom loss function definition, in which you define a function that disproportionately punishes false positive predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(y_i = 0) \rightarrow (d = s)\wedge (y_i = 1) \rightarrow (d=1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(p) = \frac {\sum_i\frac{(p_i-y_i)^2}{d}}{n_i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(p)$ = loss function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$p_i$ = predicted class likelihood&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y_i$ = actual class (0 for negative example, 1 for positive)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_i$ = number of observations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$s$ = Penalty parameter for false positives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would heavily punish false positives relative to false negatives. It can also be adjusted to your needs, to more or less heavily punish false positives. For $s = .5$, the function looks something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nT0r9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nT0r9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that this function does &lt;em&gt;not necessarily&lt;/em&gt; define the approach you should take, but only one &lt;em&gt;possible&lt;/em&gt; approach. To create a custom loss function perfectly tailored to your use case, you'd want to know the relative cost of a false positive and a false negative, and customize the function accordingly.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-06-14T16:15:25.450" LastActivityDate="2017-06-14T16:15:25.450" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19717" PostTypeId="2" ParentId="19517" CreationDate="2017-06-14T15:08:37.427" Score="1" Body="&lt;p&gt;The representation step is before the feature extraction step for exactly the reasons that they state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you take the full image representation and go right to feature extraction, you have much more data to extract features from (added complexity) and the features you do extract will be much more noisy. This is why one at least reduces the representation to grey level (as the authors suggest). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature extraction is choosing properties of the image which you will then perform recognition on. It is important to already have thrown out unimportant details of the image before doing this. That is why the authors, and people in general, reduce to a proper representation before performing feature extraction. &lt;/p&gt;&#xA;" OwnerUserId="33353" LastActivityDate="2017-06-14T15:08:37.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19718" PostTypeId="2" ParentId="19711" CreationDate="2017-06-14T15:55:37.073" Score="2" Body="&lt;p&gt;Its hard to answer without a good look at the data. &lt;br&gt; But if I had to guess, your point seems valid. (considering there's no problem with cross validation methods or leaks)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are &quot;measuring&quot; the contract features at different points in time, there might be a high bias that the features of the cancelled contracts, measured at the point in time in which they were cancelled, might be very different from the &quot;initial&quot; features of those same contracts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, your modelling wld be learning how to predict a contract is being cancelled at the given date it is cancelled, and not prior to it, thats why it wouldnt be working properly on your &quot;real world data&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can, try using the data from the moment the contract was set (initialized), to build your model. &lt;/p&gt;&#xA;" OwnerUserId="26899" LastActivityDate="2017-06-14T15:55:37.073" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19720" PostTypeId="2" ParentId="18368" CreationDate="2017-06-14T16:46:13.490" Score="1" Body="&lt;p&gt;No- not if your regex is actually generating the data. If this is the case, the regex (or a complementary one) should be able to classify the resulting text perfectly. That is, you know what the rules for creating a 'happy' or 'sad' piece of text are within the regex, and so you simply apply those rules backwards to classify it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you take what the regex says when it creates the data as ground truth, then there's no reason to create a machine learning classifier to classify it- the creation mechanism itself already does so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you have created a regex that will produce &quot;happy&quot; or &quot;sad&quot; tweets, and you want to know if the regex itself or a machine learning algorithm will better conform to the opinions of, say, &lt;strong&gt;&lt;em&gt;human observers&lt;/em&gt;&lt;/strong&gt;, you would have a very interesting question to evaluate and a relatively straightforward test case for sentiment analysis.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-06-14T16:46:13.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19721" PostTypeId="2" ParentId="19714" CreationDate="2017-06-14T17:32:52.183" Score="3" Body="&lt;p&gt;Thomas Cleberg's approach sounds reasonable, but another very simple approach would be to explicitly code an &quot;undefined&quot; category. This is common when dealing with text datasets where a word might be new or too rare to stand on its own. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a large enough collection of websites, surely there are website that are not cleanly classified in one of your categories. Worst case you could simply search for such websites manually and augment your dataset. This approach wouldn't require any changes to how you train your models. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-06-14T17:32:52.183" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19722" PostTypeId="2" ParentId="19685" CreationDate="2017-06-14T17:39:27.340" Score="2" Body="&lt;p&gt;This is not the answer you'll want to hear, but I would say, &quot;no.&quot; Priming is not a good idea. A model is only as good as its inputs. If you're making up the inputs yourself, then your model isn't learning real patterns and you might as well just hard code a set of rules that represent how you would have created the primed data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A fancier way might be to make a bayesian model where you create priors based on your assumptions. The bayesian model would then evolve as real data becomes available. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-06-14T17:39:27.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19726" PostTypeId="1" CreationDate="2017-06-14T20:55:09.300" Score="0" ViewCount="3390" Body="&lt;p&gt;I understand precision at k and recall at k.  It is a more useful metric for evaluating the success of a binary classifier when the positive class is overwhelmingly out-weighed by the negative class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering how to choose an appropriate &quot;k&quot; value.  According to resources like &lt;a href=&quot;https://ils.unc.edu/courses/2013_spring/inls509_001/lectures/10-EvaluationMetrics.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; the recall at &quot;k&quot; is bounded by the number of positive examples, so it is not a useful metric to use when evaluating the success of severely imbalanced classes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also seem to me like this:  Precision at K is also limited by the number of positive examples at K.  Saw we have 100 examples total, and only 3 are positive examples.  Say we rank these   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Scenario 1&lt;/em&gt;: we choose k=10.  Then, the precision at K can at most be 3/10 = 0.3. And, the recall at k will be 0.03 because there are 3 in the entire dataset of 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Scenario 2&lt;/em&gt;: we choose k=3.  Then, the precision at K would be 3/3 = 1.0 !!!And, the recall at k will STILL be 3/100 = 0.03.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though our binary classifier is performing perfectly, it's perfect performance only is reflected when we choose k=3?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question how do I choose K correctly?&lt;/p&gt;&#xA;" OwnerUserId="10512" LastActivityDate="2022-10-26T14:43:52.757" Title="How do I pick &quot;K&quot; for precision at K and recall at K?" Tags="|accuracy|binary|classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19727" PostTypeId="2" ParentId="19711" CreationDate="2017-06-14T21:05:23.377" Score="3" Body="&lt;p&gt;If your model makes a prediction 6 months into the future, then it doesn't make sense to judge its performance before 6 months. If only 2 months have passed, then possibly 2/3 of the true positives have yet to reveal their true nature and you are arriving a premature conclusion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test this theory, I would train a new model to predict 2 months out and use that to get an approximation of live accuracy while your wait 4 more months for the first model. Of course, there could be other problems, but this is what I would try first. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-06-14T21:05:23.377" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19728" PostTypeId="2" ParentId="19726" CreationDate="2017-06-14T21:50:43.570" Score="1" Body="&lt;p&gt;Precision and Recall can be dubious at times. It depends on the point on the ROC Curve. Therefore, pick the point as per your need. The best way is to give more information about your results; i.e., Accuracy measure and Confusion matrix. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2017-06-14T21:50:43.570" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19729" PostTypeId="2" ParentId="19681" CreationDate="2017-06-14T23:13:29.747" Score="1" Body="&lt;p&gt;this may be a bug in orange tool because the data sets look well formatted. I suggest you to put all data (both data sets) in one file and load it onto orange, if it is OK then split the file (separate the data) and reload the two new files and it should work.&lt;/p&gt;&#xA;" OwnerUserId="33407" LastActivityDate="2017-06-14T23:13:29.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19730" PostTypeId="2" ParentId="19240" CreationDate="2017-06-14T23:22:27.320" Score="1" Body="&lt;p&gt;It means that the model you are using fails at classifying the images. Logistic regression may not be the good one for your classification. Try using another model. &lt;/p&gt;&#xA;" OwnerUserId="33407" LastActivityDate="2017-06-14T23:22:27.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19732" PostTypeId="1" AcceptedAnswerId="19733" CreationDate="2017-06-15T04:10:56.000" Score="1" ViewCount="149" Body="&lt;p&gt;[Note : There is some serious problem with logic used to get the best banner. I got it late. Directly read the answer to get general info, or you can also try to the find the mistake.]  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem: Given a set of user features, select an ad with the highest probability to be clicked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset - &lt;a href=&quot;https://www.kaggle.com/c/avazu-ctr-prediction/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/avazu-ctr-prediction/data&lt;/a&gt; (First 100000 tuples from training set and split that it 80:20 training:testing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tutorial followed - &lt;a href=&quot;https://turi.com/learn/gallery/notebooks/click_through_rate_prediction_intro.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://turi.com/learn/gallery/notebooks/click_through_rate_prediction_intro.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;C14 is my ad id.&lt;br&gt;&#xA;Problem :- Given ('device_type', 'C1', 'C15') return ad id.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training:-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have taken 'click' as my target and ('device_type', 'C1', 'C15', 'C14') as my input features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used logistic regression classifier in graphlab library to train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing ad selection in the following way:-&lt;br&gt;&#xA;Given a set of features ('device_type', 'C1', 'C15', X) Iterate X over all possible values of C14 and pass the features to predictor to get the probability that X ad will be clicked. Return the ad with maximum click probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MY PROBLEM IS LOGISTIC CLASSIFIER IS ONLY RETURNING ONE AD for every test tuple, though with different click probability, it means only one ad is getting the highest probability to be clicked.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain this observation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using boosted tree classifier instead of logistic classifier for the above prediction, I am able to get different ads as my prediction and hence getting better results.&lt;/p&gt;&#xA;" OwnerUserId="33417" LastEditorUserId="33417" LastEditDate="2017-07-07T10:45:28.080" LastActivityDate="2018-01-05T23:05:31.613" Title="Explain output of logistic classifier" Tags="|machine-learning|predictive-modeling|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19733" PostTypeId="2" ParentId="19732" CreationDate="2017-06-15T05:32:34.150" Score="1" Body="&lt;p&gt;It means your logistic classifier is biased towards one class, this could be because of below reasons that I can think of.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class Imbalance:  &lt;a href=&quot;http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; article explains how to identify and overcome the class imbalance problem. &lt;/li&gt;&#xA;&lt;li&gt;Overfitting: &lt;a href=&quot;http://machinelearningmastery.com/overfitting-and-underfitting-with-machine-learning-algorithms/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; article explains how to tackle over fitting.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Logistic classifier works better if data is linearly related, if you find non-linear relationships in data I would suggest use better algorithms like GBM/SVM/Random forest, which will give you much better and accurate results.  &lt;/p&gt;&#xA;" OwnerUserId="32805" LastEditorUserId="33245" LastEditDate="2018-01-05T23:05:31.613" LastActivityDate="2018-01-05T23:05:31.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19735" PostTypeId="1" CreationDate="2017-06-15T06:19:23.970" Score="0" ViewCount="152" Body="&lt;p&gt;I have two data sets one is cross sectional census data with 10 years interval and another one is time series data  (monthly) for several years. Now I want to perform statistical time series analysis on this both dataset in a single model. Is it possible? &lt;/p&gt;&#xA;" OwnerUserId="26105" LastActivityDate="2024-03-18T18:05:35.760" Title="Which time series analysis is appropriate for pooled time series data analysis?" Tags="|data-mining|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19736" PostTypeId="2" ParentId="19735" CreationDate="2017-06-15T06:48:35.640" Score="0" Body="&lt;p&gt;It really depends on the outcome you want from your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most straightforward approaches might be &lt;strong&gt;resampling&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Interpolate the 10-year census data to month granularity. The interpolation might be linear, polynomial, etc. This effectively imposes a strong assumption in the dynamics of the data. This may make sense for some variables but not for others. And you are probably introducing a lot of variance in the obtained conclusions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Aggregate the monthly time series into 10-year granularity. This may not be very useful if you want to predict something with an horizon less than 10 years...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-06-15T06:48:35.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19737" PostTypeId="1" CreationDate="2017-06-15T08:03:58.687" Score="2" ViewCount="132" Body="&lt;p&gt;I have statistical background now I want to learn machine learning. I have experience in R, SPSS, Stata. Necessary resources and tutorials which are effective not only waste my time are needed for my early research career. I have searched several tutorials but I need effective one.&lt;/p&gt;&#xA;" OwnerUserId="26105" LastEditorUserId="26105" LastEditDate="2017-08-10T06:01:23.890" LastActivityDate="2017-08-10T06:01:23.890" Title="How can learn machine learning with Statistics background?" Tags="|machine-learning|r|data-mining|stata|spss|" AnswerCount="1" CommentCount="2" ClosedDate="2017-06-15T15:21:20.647" ContentLicense="CC BY-SA 3.0" />
  <row Id="19738" PostTypeId="2" ParentId="19737" CreationDate="2017-06-15T08:53:31.770" Score="3" Body="&lt;p&gt;I would recommend using Kaggle or a similar website. There are a lot of competition so you probably can find one that motivates you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One point that is imho often overseen is their very large collection of non-competition datasets. I am rather confident that you will find one that meets a personal interest of yours. You can then start to explore it and discuss results with others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really think that machine learning like other programming skills is best learned on practical examples. You will fail, learn from your mistakes only to fall into the next trap and this is very important as these experiences will help you later on!&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-06-15T08:53:31.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19739" PostTypeId="2" ParentId="19653" CreationDate="2017-06-15T08:55:12.920" Score="2" Body="&lt;p&gt;As the user above, I am more in the Python side, so I am not aware of what the best solution for R would be, but I suggest using HDF5 files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can store different objects in different tables inside the file, and each table can have its hierarchy. Volume and memory should also not be a limitation, as you don't need to load the whole file into memory.&lt;/p&gt;&#xA;" OwnerUserId="1289" LastEditorUserId="1289" LastEditDate="2017-06-15T13:39:32.613" LastActivityDate="2017-06-15T13:39:32.613" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19740" PostTypeId="2" ParentId="11404" CreationDate="2017-06-15T08:57:32.853" Score="2" Body="&lt;p&gt;Yes, this is a fine technique to tackle the problem of class-imbalance. However, under-sampling methods do lead to the loss of information in the data set (say, you just removed an interesting pattern among the remaining variables, which could have contributed to a better training of the model). This is why over-sampling methods are preferred, specifically in case of smaller data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In response to your query regarding Python packages, the &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/api.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;imbalanced-learn&lt;/a&gt; toolbox is specially dedicated for the same task. It provides several under-sampling and over-sampling methods. I would recommend trying the &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/generated/imblearn.over_sampling.SMOTE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt; technique.&lt;/p&gt;&#xA;" OwnerUserId="33429" LastActivityDate="2017-06-15T08:57:32.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19742" PostTypeId="1" CreationDate="2017-06-15T11:17:24.037" Score="0" ViewCount="108" Body="&lt;p&gt;I would like to understand the behaviour of stochastic gradient descent (SGD) over long time periods for a fixed learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume that after a number of epochs we expect no further improvement to our cost function.  If we continue to train over more epochs then one of two things can happen:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The solution keeps changing but remains close to a local minimum; in the space of weights it remains in a local region.  Or,&lt;/li&gt;&#xA;&lt;li&gt;The solution keeps changing but explores a large region of the space of weights.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let's make this precise.  Take a fixed shape of neural network and use the Euclidean distance metric on the space of weights.  Now train this network using SGD on a dataset, e.g. MNIST, and for a fixed choice of learning rate.  After a large number of epochs, N start recording the weights of the network,&#xA;$$ {\bf w_N}, {\bf w_{N+1}} \ldots $$&#xA;How does $d({\bf w_N}, {\bf w_M})$ behave?  To gauge whether these distances are large or small, they can be compared to the distances you get by comparing ${\bf w_N}$ with equivalent networks given by permuting the neurons within a layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ofcourse the answer may vary depending on the neural network, the data, the learning rate, etc.  I suppose I would like to know the answer for typical &quot;real world&quot; networks.  Unfortunately I lack the processing power (and expertise) to adequately investigate this myself, and am confident this will be known to experts.&lt;/p&gt;&#xA;" OwnerUserId="33432" LastActivityDate="2017-06-15T12:38:42.627" Title="Does a neural network continue to change after SGD stops improving?" Tags="|machine-learning|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19743" PostTypeId="2" ParentId="19742" CreationDate="2017-06-15T12:06:04.767" Score="1" Body="&lt;p&gt;The evolution of the &lt;em&gt;improvement&lt;/em&gt; of the performance of the network if you keep training it depends on:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) the surface of the loss function with respect to the network parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) the learning rate, which is the scale factor you apply to the gradient in order to compute the next set of weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the learning rate is too high (with respect to the shape of the loss surface) and you keep training, you may escape a local minimum you just reached.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that you don't know the loss surface and so we cannot choose the &lt;em&gt;perfect&lt;/em&gt; learning rate for it. There are a lot of heuristic tricks to try to improve the exploration behaviour of SGD, like &lt;a href=&quot;http://distill.pub/2017/momentum/&quot; rel=&quot;nofollow noreferrer&quot;&gt;momentum&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the answer is: &lt;strong&gt;it depends&lt;/strong&gt;. The network could stabilize around a local minimum and so the weights would only suffer small oscillations, or it might ruin everything by escaping the local minimum.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-06-15T12:38:42.627" LastActivityDate="2017-06-15T12:38:42.627" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19744" PostTypeId="1" CreationDate="2017-06-15T12:36:50.613" Score="0" ViewCount="275" Body="&lt;p&gt;In this link: &lt;a href=&quot;http://optimalprediction.com/files/pdf/V2A17.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://optimalprediction.com/files/pdf/V2A17.pdf&lt;/a&gt;&lt;br&gt;&#xA;Ipsative Standardization is explained intuitively and also the why it is useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any python code or math formulas online someone could use to calculate the z1 score?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2018-04-14T20:11:38.873" Title="How Ipsative Standardization actually works for time series?" Tags="|time-series|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19746" PostTypeId="2" ParentId="19615" CreationDate="2017-06-15T13:19:11.383" Score="1" Body="&lt;p&gt;I removed 20 percent of most expensive houses from dataset and divided the prices by 1.000.000 and the result got much better.&lt;/p&gt;&#xA;" OwnerUserId="33188" LastActivityDate="2017-06-15T13:19:11.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19747" PostTypeId="1" CreationDate="2017-06-15T14:23:18.720" Score="4" ViewCount="9194" Body="&lt;p&gt;I am attempting to load the &lt;a href=&quot;https://insights.stackoverflow.com/survey/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stack Overflow 2017 developer survey results&lt;/a&gt; into Orange. The dataset has ~51.4K rows with 156 features. It is comma separated and is about 91MB in size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I attempt to load this file into Orange, I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cannot parse dataset H:/survey_results_public.csv: too many indices for array&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I overcome this error?&lt;/p&gt;&#xA;" OwnerUserId="6431" LastActivityDate="2018-02-20T15:48:59.270" Title="Overcoming &quot;too many indices for array&quot; error when loading file into Orange" Tags="|orange|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19748" PostTypeId="1" AcceptedAnswerId="26562" CreationDate="2017-06-15T15:09:41.933" Score="1" ViewCount="1965" Body="&lt;p&gt;I'm working on a project to try and predict which users would be most likely to subscribe to our services. I'm using R and am working with a data frame that has a user's email address, information about their past purchases (order count and revenue generated), how they have interacted with our website (different events triggered), and whether or not they have subscribed. I'm using this data to train and test different algorithms. I've used caret with knn, lda, cart, and nnet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been able to get how accurate my models are but that doesn't help me know who to market to. When I plug new data into my model I would like to know which users it thinks are the most likely to subscribe and also give me their associated email addresses. I've looked all over but haven't been able to find anything that helps me pair a prediction to an email address. Any help would be much appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not using the email addresses to classify the data. I simply have them to associate a row of data to a user. I had to get rid of them to run it through the training and testing algorithms and that is where I am having the issue of knowing what prediction data is associated with which email address and in turn which user.&lt;/p&gt;&#xA;" OwnerUserId="33404" LastEditorUserId="29575" LastEditDate="2018-01-12T13:56:01.503" LastActivityDate="2018-01-12T13:56:01.503" Title="Using Machine Learning to create marketing segments" Tags="|machine-learning|r|" AnswerCount="5" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19749" PostTypeId="2" ParentId="19748" CreationDate="2017-06-15T15:53:08.743" Score="0" Body="&lt;p&gt;It would be a good idea to create some visualizations of your data before choosing any particular method. Start with this and you might see some useful trends!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For starters, I would probably make a binary classifier from logistic regression. Feed a fraction of the data you have for training and see how it performs on the rest. This is one of the simplest approaches and could work very well.&lt;/p&gt;&#xA;" OwnerUserId="33353" LastActivityDate="2017-06-15T15:53:08.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19750" PostTypeId="2" ParentId="19748" CreationDate="2017-06-15T15:54:02.940" Score="0" Body="&lt;p&gt;You might want to extract certain features from an email address:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like provider, name etc. Only email address meta infromation *might be too though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do not want to use an email address but just map it you can simply use a map of entries to email addresses. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-06-15T15:54:02.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19751" PostTypeId="2" ParentId="19748" CreationDate="2017-06-15T16:17:58.490" Score="0" Body="&lt;p&gt;It would make sense if you cluster your Data and check each Cluster's RFM (Recency, Frequency, Monetary) value or the ratio of Subscriptions. Then depending on your budget target the best Clusters and check their email addresses domains or locations (from IP). Then it comes upon discussion and Optimisation on your Marketing plan.&lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2017-06-15T16:17:58.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19753" PostTypeId="2" ParentId="711" CreationDate="2017-06-15T18:57:58.747" Score="11" Body="&lt;p&gt;In answering this question one significant distinction to make is whether we are talking about &lt;strong&gt;linear&lt;/strong&gt; Support Vector Machines or &lt;strong&gt;non-linear&lt;/strong&gt;, that is, kernelized Support Vector Machines.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Linear SVMs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Linear SVMs are both in theory and practice very good models when your data can be explained by linear relations of your features. They are superior over classic methods such as linear (a.k.a. least-squares) regression because they are &lt;strong&gt;robust&lt;/strong&gt;, in the sense that small perturbations in the input data do not produce significant changes in the model. This is attained by trying to find the line (hyperplane) that maximizes the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperplane_separation_theorem&quot; rel=&quot;noreferrer&quot;&gt;margins&lt;/a&gt; between your data points. This maximum margin hyperplane has been shown to give guarantees on the generalization ability of the model over unseen data points, a theoretical property other machine learning methods lack of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear SVMs are also interpretable as any other linear model, since each input feature has a weight that directly influences the model output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also linear SVMs are &lt;strong&gt;very fast&lt;/strong&gt; to train, showing sublineal training times for very large datasets. This is achieved by making use of stochastic gradient descent techniques, much in the fashion of current deep learning methods.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Non-linear SVMs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Non-linear SVMs are still linear models, and boast the same theoretical benefits, but they employ the so called &lt;strong&gt;kernel trick&lt;/strong&gt; to build this linear model over an enlarged space. The visible result is that the resultant model can make non-linear decisions on your data. Since you can provide a &lt;strong&gt;custom kernel encoding similarities&lt;/strong&gt; between data points, you can make use of problem knowledge to make such kernel focus in the relevant parts of your problem. Doing this effectively, however, can be difficult, so in general almost everybody uses the plug-and-play gaussian kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-linear SVMs are partially interpretable, as they tell you which training data are relevant for prediction, and which aren't. This is not possible for other methods such as Random Forests or Deep Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately non-linear SVMs are slow. The state-of-the-art algorithm is &lt;strong&gt;Sequential Minimal Optimization&lt;/strong&gt;, which shows quadratic performance, and is widely implemented through the &lt;a href=&quot;https://www.csie.ntu.edu.tw/~cjlin/libsvm/&quot; rel=&quot;noreferrer&quot;&gt;LIBSVM&lt;/a&gt; library in a number of machine learning libraries, scikit-learn included.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Popularity of these methods&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It is true that SVMs are not so popular as they used to be: this can be checked by googling for research papers or implementations for SVMs vs Random Forests or Deep Learning methods. Still, they are useful in some practical settings, specially in the linear case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, bear in mind that due to the &lt;strong&gt;no-free lunch theorem&lt;/strong&gt; no machine learning method can be shown to be superior to any other over all problems. While some methods do work better in general, you will always find datasets where a not so common method will achieve better results.&lt;/p&gt;&#xA;" OwnerUserId="33447" LastActivityDate="2017-06-15T18:57:58.747" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19755" PostTypeId="1" AcceptedAnswerId="19756" CreationDate="2017-06-15T20:15:12.750" Score="5" ViewCount="13004" Body="&lt;p&gt;I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another team. They use the Gini metric to measure the performance of the model. They achieved 87%. Upon evaluation, I found that the recall was 51% whilst the error rate of the non rare event class (do not default) was 0.9%. Am I correct in thinking that the Gini is actually a misleading metric in this case because it doesn't really show the extremely poor predictive performance of the rare event class? I have questioned them about this and tried to recommend them to use precision/recall metrics as well as confusion matrices and a precision-recall trade-off graph but they quickly dismissed me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice would be much appreciated. &lt;/p&gt;&#xA;" OwnerUserId="33451" LastEditorUserId="33478" LastEditDate="2017-06-16T17:50:18.250" LastActivityDate="2018-11-25T05:32:00.247" Title="Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data" Tags="|predictive-modeling|accuracy|model-evaluations|model-selection|class-imbalance|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19756" PostTypeId="2" ParentId="19755" CreationDate="2017-06-15T21:01:01.597" Score="6" Body="&lt;p&gt;The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): &lt;code&gt;G = 2*AUC -1&lt;/code&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate &lt;code&gt;FP/(FP+TN)&lt;/code&gt;. If the number of negatives is a lot larger, this could be a potential issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, the Gini Coefficient has similar pros and cons as the AUC ROC metric. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-06-16T18:04:01.230" LastActivityDate="2017-06-16T18:04:01.230" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19758" PostTypeId="1" CreationDate="2017-06-15T22:20:26.217" Score="0" ViewCount="475" Body="&lt;p&gt;I am trying to fit a keras classifier on a data matrix X_train &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dummy_y = np_utils.to_categorical(y_train)&#xA;dummy_y_val = np_utils.to_categorical(y_val)&#xA;&#xA;def baseline_model(optimizer='rmsprop', init='normal', dropout_rate =0.0):&#xA;    model = Sequential()&#xA;    model.add(Dense(100, input_dim = X_train.shape[1], activation='relu', init=init))&#xA;    model.add(Dropout(dropout_rate))&#xA;    model.add(Dense(50, activation = 'relu', init = init))&#xA;    model.add(Dropout(dropout_rate))&#xA;    model.add(Dense(15, activation = 'sigmoid', init = init))&#xA;    model.compile(loss = 'binary_crossentropy', optimizer = optimizer, metrics =['accuracy'])&#xA;    return model&#xA;&#xA;   estimator = KerasClassifier(build_fn=baseline_model, nb_epoch=200, batch_size=5, verbose=0)&#xA;   model = baseline_model()&#xA;&#xA;model.fit(X_train, dummy_y, batch_size=32,  verbose=1, callbacks=None, \&#xA;     validation_data=(X_val, dummy_y_val), shuffle=True, class_weight=None, \&#xA;    sample_weight=None, initial_epoch=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's what I get as an error :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Train on 582 samples, validate on 290 samples Epoch 1/10&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;ValueErrorTraceback (most recent call last)&#xA;   in ()&#xA;        1 model = baseline_model()&#xA;  ----&gt; 2 model.fit(X_train, dummy_y, batch_size=32,  verbose=1, &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;....&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;ValueError: Cannot feed value of shape (582, 32) for Tensor&#xA;  u'dense_input_28:0', which has shape '(?, 18760)'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;NOTE : X_train.shape = (580,18760) and y_train has 15 classes&lt;/p&gt;&#xA;" OwnerUserId="33453" LastEditorUserId="33453" LastEditDate="2017-06-15T22:34:35.123" LastActivityDate="2017-07-16T17:12:17.670" Title="Keras : problem in fitting model" Tags="|python|keras|classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19760" PostTypeId="1" CreationDate="2017-06-16T04:37:17.533" Score="1" ViewCount="1741" Body="&lt;p&gt;I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What size neurons to include in the search&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of hidden layers, and the width of the layers &lt;/li&gt;&#xA;&lt;li&gt;The&#xA;optimizer and learning rates &lt;/li&gt;&#xA;&lt;li&gt;Activation functions and loss functions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thank you so much!&lt;/p&gt;&#xA;" OwnerUserId="33160" LastActivityDate="2017-08-20T17:10:54.690" Title="Bounds for MLP hyperparameter search" Tags="|neural-network|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19765" PostTypeId="1" CreationDate="2017-06-16T10:39:45.207" Score="1" ViewCount="81" Body="&lt;p&gt;How would I train an HMM using the EM algorithm so the transition matrix is upper diagonal?&lt;/p&gt;&#xA;" OwnerUserId="28600" LastActivityDate="2018-02-02T20:23:04.217" Title="Training HMMs using the EM algorithm" Tags="|probability|markov|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19767" PostTypeId="1" AcceptedAnswerId="28033" CreationDate="2017-06-16T12:49:38.433" Score="1" ViewCount="3833" Body="&lt;p&gt;I'm beginner in python so please bare with me. I'm trying to solve one machine learning problem using GaussianNB. I've certain fields which are not in proper date format, so I converted it into UNIX format. For example column &lt;code&gt;state_changed_at&lt;/code&gt; has value in csv as &lt;code&gt;1449619185&lt;/code&gt;. I'm converting it into proper date format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the problem is, when I'm selecting those date features to train my model, it gives me an error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Could not convert string to float: 'Thu Apr 16 23:58:58 2015'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn import metrics&#xA;from sklearn.naive_bayes import BernoulliNB&#xA;from sklearn.naive_bayes import MultinomialNB&#xA;import time&#xA;from sklearn.naive_bayes import GaussianNB&#xA;&#xA;train = pd.read_csv(&quot;datasets/train2.csv&quot;)&#xA;test = pd.read_csv(&quot;datasets/test.csv&quot;)&#xA;train.head()&#xA;&#xA;import time&#xA;&#xA;#   state_changed_at,deadline,created_at,launched_at are date time fields&#xA;# and I'm converting it into unix format&#xA;unix_cols = ['deadline','state_changed_at','launched_at','created_at']&#xA;&#xA;for x in unix_cols:&#xA;    train[x] = train[x].apply(lambda k: time.ctime(k))&#xA;    test[x] = test[x].apply(lambda k: time.ctime(k))&#xA;&#xA;&#xA;#   state_changed_at,deadline,created_at,launched_at are date time fields.&#xA;cols_to_use = ['keywords_len' ,'keywords_count','state_changed_at','deadline','created_at','launched_at']&#xA;&#xA;target = train['final_status']&#xA;&#xA;# data for modeling&#xA;k_train = train[cols_to_use]&#xA;k_test = test[cols_to_use]&#xA;&#xA;&#xA;gnb = GaussianNB()&#xA;&#xA;model = MultinomialNB()&#xA;model.fit(k_train, target)  # this lines gives me error saying: could not convert string to float: 'Thu Apr 16 23:58:58 2015'&#xA;&#xA;expected = target&#xA;predicted = model.predict(k_test)&#xA;print(model.score(k_test, predicted, sample_weight=None))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help would be really appreciated.&#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="33440" LastActivityDate="2018-07-21T08:10:10.643" Title="GausianNB: Could not convert string to float: 'Thu Apr 16 23:58:58 2015'" Tags="|machine-learning|python|gaussian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19768" PostTypeId="1" AcceptedAnswerId="19771" CreationDate="2017-06-16T14:13:34.620" Score="6" ViewCount="37134" Body="&lt;p&gt;I am trying to implement Python's &lt;strong&gt;&lt;em&gt;MLPClassifier&lt;/em&gt;&lt;/strong&gt; with &lt;strong&gt;&lt;em&gt;10 fold&lt;/em&gt;&lt;/strong&gt; cross-validation using &lt;strong&gt;&lt;em&gt;gridsearchCV&lt;/em&gt;&lt;/strong&gt; function. Here is a chunk of my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;parameters={&#xA;'learning_rate': [&quot;constant&quot;, &quot;invscaling&quot;, &quot;adaptive&quot;],&#xA;'hidden_layer_sizes': [(100,1), (100,2), (100,3)],&#xA;'alpha': [10.0 ** -np.arange(1, 7)],&#xA;'activation': [&quot;logistic&quot;, &quot;relu&quot;, &quot;Tanh&quot;]&#xA;}&#xA;&#xA;clf&#xA;= gridSearchCV(estimator=MLPClassifier,param_grid=parameters,n_jobs=-1,verbose=2,cv=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Though,I am not sure if &lt;code&gt;hidden_layer_sizes: [(100,1), (100,2), (100,3)]&lt;/code&gt; is correct. Here, I am trying to tune &lt;strong&gt;&lt;em&gt;'hidden layer size'&lt;/em&gt;&lt;/strong&gt; &amp;amp; &lt;strong&gt;&lt;em&gt;'number of neurons'&lt;/em&gt;&lt;/strong&gt;. I would like to give this &lt;strong&gt;&lt;em&gt;'tuple'&lt;/em&gt;&lt;/strong&gt; parameter for &lt;strong&gt;&lt;em&gt;hidden_layer_sizez: 1, 2, 3&lt;/em&gt;&lt;/strong&gt;, and &lt;strong&gt;neurons: 10, 20, 30,...,100&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But&#xA;I do not know if it is the correct way to do it. Therefore, I am choosing default neurons to be &lt;strong&gt;&lt;em&gt;100&lt;/em&gt;&lt;/strong&gt; in each layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can&#xA;anyone advise please?&lt;/p&gt;&#xA;" OwnerUserId="33476" LastEditorUserId="23305" LastEditDate="2017-12-20T16:17:37.167" LastActivityDate="2021-06-01T03:11:03.437" Title="How to implement Python's MLPClassifier with gridsearchCV?" Tags="|python|neural-network|scikit-learn|cross-validation|hyperparameter|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19770" PostTypeId="1" AcceptedAnswerId="19777" CreationDate="2017-06-16T15:05:49.527" Score="5" ViewCount="1033" Body="&lt;p&gt;Am I doing regression task with 7 dependent variables and 10000 data points. The SGD gives me 22% of mean absolute percentage error on test and train dataset. And Least Squares method using numpy &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.optimize.least_squares.html&quot; rel=&quot;noreferrer&quot;&gt;scipy.optimize.least_squares&lt;/a&gt; give me only 58% (I have tried different settings.). I thought that Least Squares should give the same or better performance with such size dataset. What can be the reasons?&lt;/p&gt;&#xA;" OwnerUserId="21319" LastActivityDate="2017-06-16T19:58:03.080" Title="Why does gradient descent gives me much better Relative Squared Error then the Least Squares approach?" Tags="|machine-learning|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19771" PostTypeId="2" ParentId="19768" CreationDate="2017-06-16T16:41:52.413" Score="8" Body="&lt;p&gt;A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(100,1)$ would mean that the second hidden layer only has one neuron.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-06-16T16:41:52.413" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19773" PostTypeId="2" ParentId="19758" CreationDate="2017-06-16T16:45:14.510" Score="3" Body="&lt;p&gt;The problem was : X_train and y_train were not numpy array. I transformed them to numpy arrays and it solved the problem. &lt;/p&gt;&#xA;" OwnerUserId="33453" LastActivityDate="2017-06-16T16:45:14.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19774" PostTypeId="1" CreationDate="2017-06-16T17:50:25.023" Score="3" ViewCount="474" Body="&lt;p&gt;I have a Python DataFrame that looks something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pageviews, type&#xA;100         0&#xA;50          1&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.corr()&#xA;&#xA;            pageviews   type&#xA;pageviews   1.000000    -0.009611&#xA;type       -0.009611    1.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So there's less than a 0.01 correlation between type and pageviews.  However, when I do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('type')['pageviews'].mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get back:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;type&#xA;0    1421.406621&#xA;1     885.092874&#xA;Name: pageviews, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So type 0 has almost twice the number of average pageviews but there's very little correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is this possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="33199" LastActivityDate="2017-11-16T08:09:56.840" Title="Strange Pearson Correlation Coefficient Given DataFrame" Tags="|python|pandas|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19777" PostTypeId="2" ParentId="19770" CreationDate="2017-06-16T19:36:24.613" Score="5" Body="&lt;p&gt;The reason is that the two metrics, &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_absolute_percentage_error&quot; rel=&quot;noreferrer&quot;&gt;mean absolute percentage error&lt;/a&gt; (MAPE) and &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;noreferrer&quot;&gt;mean square error&lt;/a&gt; (MSE) are optimising to different targets. Improving one can be done at the expense of the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple example, consider this data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = [ 0,  1,  2,  3,  4,  5]&#xA;y = [ 3,  5, 10, 10, 11, 15]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The best fit mean squared error (MSE) for a line on this data is $\hat{y} = 2.23x + 3.43$, which has MSE of $1.18$, and a mean absolute percentage error (MAPE) of $11.0$%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best fit mean absolute percentage loss for a line on this data is $\hat{y} = 2.35x + 2.99$, which has a MSE of $1.24$, and a MAPE of $8.34$%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that optimising for MAPE gives a worse MSE, and vice-versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference can get extreme when there is a large range for y values (in terms of orders of magnitude covered), because optimising for MAPE will favour being more accurate on small values at the expense of larger ones. So if we change y to be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = [ 1,  2, 10, 10, 11, 20]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then optimising mean abs percentage gives the line $\hat{y} = 3.78x + 1.05$ with MSE $7.09$ and MAPE $21.9$%. But optimising for mean square error gives $\hat{y} = 3.49x + 0.286$ with MSE  $4.56$ and MAPE $62.9$% - this is a larger difference, and I suspect that your data has a large range of target variable causing a similar effect.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;You can potentially get closer results using the Least Squares regressor by using a transformed target variable $z = log(y)$ and transforming back at the end. This still won't be quite the same, but it does reduce the difference significantly - in my last example if I try this, I get MAPE $24.2$% - compared to $21.9$% for optimising MAPE directly.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-16T19:58:03.080" LastActivityDate="2017-06-16T19:58:03.080" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19778" PostTypeId="2" ParentId="10557" CreationDate="2017-06-16T20:46:32.440" Score="2" Body="&lt;p&gt;I just saw your question. It is COMPLETELY WRONG to do feature selection first and then tune the model using cross-validation. In elements of statistical learning and &lt;a href=&quot;https://www.nodalpoint.com/not-perform-feature-selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; blog post, it is clearly mentioned that:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;What has happened? The problem is the predictors have an unfair advantage, as they were chosen in step (1) on the basis of &lt;em&gt;all of the samples&lt;/em&gt;. Leaving samples out &lt;em&gt;after&lt;/em&gt; the variables have been selected does not correctly mimic the application of the classifier to a completely independent test set, since these predictors &amp;quot;have already seen&amp;quot; the left-out samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;CV methods are unbiased only if all your model building is done inside the CV loop.&#xA;So do feature selection inside the CV loop for parameter tuning. It can be done easily using the filter wrapper in the MLR package in R.&lt;/p&gt;&#xA;" OwnerUserId="31822" LastEditorUserId="69263" LastEditDate="2023-02-25T05:52:21.990" LastActivityDate="2023-02-25T05:52:21.990" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19781" PostTypeId="2" ParentId="19755" CreationDate="2017-06-17T04:05:01.730" Score="4" Body="&lt;p&gt;Gini coefficient shouldn't be to my understanding a bad mertric for imbalanced classification, because it is related to AUC, which works just fine. Maybe it was gini impurity not coefficient. Check your AUC of the predictions once. Also Area under the PR curve is a better metric for imbalanced classification than AUC, maybe you should see that too. &lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-06-17T04:05:01.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19782" PostTypeId="1" CreationDate="2017-06-17T04:23:17.953" Score="9" ViewCount="6655" Body="&lt;p&gt;Continous feature discretization usually leads to lose of information due to the binning process. However most of the Top solutions for Kaggle Titanic are based on discretization(age,fare).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When should continuous features be discretized ? Is there any criteria and pros and cons on accuracy.&lt;/p&gt;&#xA;" OwnerUserId="33495" LastEditorUserId="26562" LastEditDate="2017-10-16T16:46:56.753" LastActivityDate="2023-08-20T20:17:03.370" Title="What is the rationale for discretization of continuous features and when should it be done?" Tags="|machine-learning|statistics|feature-selection|algorithms|feature-extraction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19784" PostTypeId="1" CreationDate="2017-06-17T09:36:09.793" Score="1" ViewCount="1037" Body="&lt;p&gt;I am working on this course on Machine Learning 2012 from UBC (CPSC 340). I am stuck on a Homework code problem which shows the following &lt;code&gt;RuntimeError&lt;/code&gt; in &lt;code&gt;RuntimeWarning: invalid value encountered in add, C = np.diag(np.dot(X.T, (ytile.T - protile.T + third.T)))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me check what the problem in the code is? Also, when I print the $\theta$, they are exponentially increasing and the loop keeps on executing.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def ridge(X, y, d2):&#xA;    one = np.linalg.inv(np.dot(X.T, X) + d2 * (np.eye(8)))&#xA;    two = np.dot(X.T, y)&#xA;    theta = np.dot(one, two)&#xA;    return theta&#xA;&#xA;def lasso(X, y, d2):&#xA;    theta = ridge(X, y, d2)&#xA;    k = len(theta)&#xA;    theta_ = np.zeros(k)&#xA;    A = 2 * (np.sum(X**2, axis = 0))&#xA;    while np.sum(np.abs(theta_ - theta)) &amp;gt; 1e-5:&#xA;&#xA;        theta_ = theta.copy()&#xA;&#xA;        ytile = np.tile(y, (k, 1))&#xA;        pro = np.dot(X, theta)&#xA;        protile = np.tile(pro, (k,1))&#xA;        third = np.dot(np.diag(theta), X.T)&#xA;        ########################################################&#xA;        C = np.diag(np.dot(X.T, (ytile.T - protile.T + third.T)))&#xA;        ########################################################&#xA;        for i in range(k):&#xA;            if C[i] &amp;lt; -d2 :&#xA;                theta[i] = ( C[i] + d2 ) / A[i]&#xA;            elif C[i] &amp;gt; d2 :&#xA;                theta[i] = ( C[i] - d2 ) / A[i]&#xA;            else:&#xA;                theta[i] = 0&#xA;    return theta&#xA;&#xA;theta = lasso(Xtrain, ytrain, 1)&#xA;print(theta)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27958" LastActivityDate="2017-06-17T09:36:09.793" Title="Lasso implementation in Python" Tags="|python|regularization|parameter-estimation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19786" PostTypeId="1" CreationDate="2017-06-17T15:28:01.133" Score="2" ViewCount="194" Body="&lt;p&gt;I am looking for books/tutorials that help you gain the insight into the thought process behind data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the books I've read were mostly documentation - the author shows you a function and some data that he applies that function to. They also show how to use graphs and how histogram/boxplots etc. work. They go through popular libraries like numpy, pandas and how to use them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in how people on (for example) Kaggle in their kernels on Titanic get their ideas from, on what to apply to datasets. Those people know which columns to plot as a function of each other, when to plot histograms, when to plot density functions and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am somewhat experienced in Machine Learning. It's pretty obvious which algorithm can be applied to what situation. Data exploring seems like a very ambiguous exercise with many solutions/ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to put is: Where to get the ideas for data exploration from?&lt;/p&gt;&#xA;" OwnerDisplayName="user25935" LastActivityDate="2024-03-02T12:02:05.203" Title="Thought process behind data analysis" Tags="|dataset|books|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19787" PostTypeId="2" ParentId="19782" CreationDate="2017-06-17T15:46:50.903" Score="3" Body="&lt;p&gt;I can think of three reasons why discretization might help in some problems.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;It makes sense for your problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Continuous variables such as age are better understood when discretized into meaningful groups: infants, youngsters, young adults, adults, senior, ... this is common in the field of marketing, because a small number of years do not really make much different in one person's interests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give another example, when working on a dataset with GPS locations, it might be more useful to discretize those into contry/state locations.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Interpretability&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;A continuous feature might not linearly correlate with your target but have a more complex non-linear correlation. In that case, obtaining an interpretable explanation of such feature won't be easy. However it you discretize it into a set of groups or levels, you might find that some of them correlate (or anticorrelate) with your target, giving you some interpretability.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Model limitations&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Some machine learning models and feature selection methods can't handle continuous features, such as entropy-based methods, or some variants of decision trees or neural networks. Either you discretize your features or forget about using such models.&lt;/p&gt;&#xA;" OwnerUserId="33447" LastActivityDate="2017-06-17T15:46:50.903" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19788" PostTypeId="2" ParentId="19786" CreationDate="2017-06-17T17:26:21.457" Score="0" Body="&lt;p&gt;I guess it is more about &quot;What would be the most informative (and easy readable) way to visualize one specific problem?&quot;. Having a specific question in mind you can explore galleries of the libraries with their examples, for instance,  &lt;a href=&quot;https://seaborn.pydata.org/examples/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;seaborn&lt;/a&gt;. You could also explore another kernels to see how people visualized it. For some question there are pretty trivial solutions. Other questions can be visualized in very many different ways but what counts is how readable and informative your plot is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of titanic imagine three questions that you want to visualize. &#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What if survival rate depends somehow on the age and/or sex? &#xA;&lt;br&gt;&#xA;For this you might want to choose a density or histogram with having two curves on the plot: for survival and non-survival. Maybe you want to split the plot into two plots for female and male. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/V9Hgh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V9Hgh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;What if survival rate depends on a Pclass? Or maybe Emarked as well? And maybe sex plays also an important role?&#xA;&lt;br&gt; &#xA;I am a fan of factor plots as you can introduce lots of information into it and it is quite understandable from the first sight. So, you see that 1st and 2nd class had a higher survival rate comparing to the 3rd class. (The next question can arise: For each embark what is the relative proportion of the pclass? What would be the better visualization?)&#xA;&lt;a href=&quot;https://i.stack.imgur.com/swEhW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/swEhW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;What if a survival rate depends on the pclass and age? Additionally you could see what is the age distribution of each class.&#xA;&lt;br&gt;&#xA;Violin plots might be good for that.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/iXiKV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iXiKV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-06-17T17:26:21.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19789" PostTypeId="1" CreationDate="2017-06-17T17:42:44.323" Score="1" ViewCount="160" Body="&lt;p&gt;I am currently training a pattern classification network, and seem to get very inconsistent result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network seem to overfit,  but the validation accuracy is very inconsistent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently trying to do pattern recognition on audio spectrograms. &#xA;The audio consist of utterances of yes/no, and acoustic model (gmm-hmm) has been trained to detect these.  Given i already have a gmm-hmm, I can from the alignment decode which state obervation has the highest probability, and use that as output target for my neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using different network structure such as Densenet, Resnet. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I once had a run using densenet which gave me a validation accuracy of 0.8, but this seem to only have happened once.  When i tries to train the network now i get validation scores around 0.3, which annoys me... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general does it seem like the the validation score is very inconsistent,  it loop around 0.2 - 0.4, which while the system keep overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is training really supposed to be so inconsistent?.. &lt;/p&gt;&#xA;" OwnerUserId="25429" LastActivityDate="2017-06-17T17:42:44.323" Title="Inconsistent validation accuracy? is that expected?" Tags="|training|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19790" PostTypeId="1" AcceptedAnswerId="19845" CreationDate="2017-06-17T18:05:48.523" Score="9" ViewCount="460" Body="&lt;p&gt;Suppose that I am interested in three classes $c_1$, $c_2$, $c_3$. But my dataset actually contains several more real classes $(c_j)_{j=4}^n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious answer is to define a new class $\hat c_4$ that refers to all classes $c_j$, $j&amp;gt;3$ but I suspect this is not a good idea since the samples in $\hat c_4$ will be rare and not very similar to each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To visualize what I'm trying to say, suppose I have the following two variable space and the classes $c_1$, $c_2$, $c_3$, $\hat c_4= \bigcup_{j=4}^n c_j$ are depicted in red, til, green and black respectively. This is how I suspect my data would look like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/g93jH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/g93jH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard way to approach this problem? What would be the most efficient classifier and why?&lt;/p&gt;&#xA;" OwnerUserId="31736" LastEditorDisplayName="user380" LastEditDate="2017-10-13T12:37:27.977" LastActivityDate="2017-10-13T12:37:27.977" Title="How do I approach a classification problem where one of the classes is defined by 'not any of the others'" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19791" PostTypeId="2" ParentId="18258" CreationDate="2017-06-17T18:37:00.970" Score="27" Body="&lt;p&gt;I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter &lt;code&gt;positive&lt;/code&gt; which can be set to &lt;code&gt;True&lt;/code&gt; and force the coefficients to be positive. Further, setting the Regularization coefficient &lt;code&gt;alpha&lt;/code&gt; to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import Lasso&#xA;lin = Lasso(alpha=0.0001,precompute=True,max_iter=1000,&#xA;            positive=True, random_state=9999, selection='random')&#xA;lin.fit(X,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="358" LastActivityDate="2017-06-17T18:37:00.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19793" PostTypeId="2" ParentId="17430" CreationDate="2017-06-17T19:06:43.897" Score="2" Body="&lt;p&gt;Here some points on which we can focus - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.) Averaging the &lt;strong&gt;words vectors&lt;/strong&gt; lose the &lt;strong&gt;order&lt;/strong&gt; of words, making it very similar to the concept of Bag of Words. That is why with less Data Bag of words approach perform better than word2vec.&#xA;it has been found that training word2vec with potentially large data , &lt;strong&gt;outperform&lt;/strong&gt; Bag of Words models. Google's results are based on word vectors that were learned out of more than a billion-word corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.) Published literature, distributed word vector techniques have been shown to outperform Bag of Words models. In this paper, an algorithm called &lt;strong&gt;Paragraph Vector&lt;/strong&gt; is used on the IMDB dataset to produce some of the most state-of-the-art results to date.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.) In your case Document size increase with time. So a good approach would be not to cluster each word rather than clustering documents. &lt;strong&gt;Word2vec&lt;/strong&gt; algorithm has power to learn embedding for each document , which we call &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=3&amp;amp;ved=0ahUKEwjuwt7CycXUAhXLQ48KHZFfAXAQFggxMAI&amp;amp;url=https%3A%2F%2Fcs.stanford.edu%2F~quocle%2Fparagraph_vector.pdf&amp;amp;usg=AFQjCNE-2SsR07iQm78dQqVV-6_Y3ERurA&amp;amp;sig2=u5cpzsjjCg9Wi8wGvMjKXg&amp;amp;cad=rja&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Doc2vec&lt;/strong&gt;&lt;/a&gt;. it will cluster similar documents. and using cosine similarity you can find similar documents. &lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-06-17T19:06:43.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19794" PostTypeId="1" CreationDate="2017-06-17T21:36:01.840" Score="1" ViewCount="53" Body="&lt;p&gt;I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear to ultimately land up in universities of their choice in future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to this project for the process of learning and i need some starting points about how should i go about doing this .&lt;/p&gt;&#xA;" OwnerUserId="33509" LastActivityDate="2017-07-19T08:52:47.373" Title="System to provide guide to students about getting admissions to universities of their choice or some specific courses" Tags="|data-mining|dataset|data|graphs|graphical-model|" AnswerCount="1" CommentCount="0" ClosedDate="2017-07-20T11:30:18.250" ContentLicense="CC BY-SA 3.0" />
  <row Id="19797" PostTypeId="1" CreationDate="2017-06-18T02:19:06.207" Score="0" ViewCount="906" Body="&lt;p&gt;&lt;code&gt;By default this script will run 4,000 training steps. Each step chooses ten images at random from the training set, finds their bottlenecks from the cache, and feeds them into the final layer to get predictions. Those predictions are then compared against the actual labels to update the final layer's weights through the back-propagation process.&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to this paragraph from &lt;a href=&quot;https://www.tensorflow.org/tutorials/image_retraining&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article from tensorflow , once the bottlenecks features are calculated for newer classes, backpropagation algorithm is used to retrain the last layer given the output labels. My question is, are the bottleneck features of the 10,000 classes of imagenet present while this retraining progresses - or the retraining is done for the newer class labels only.&lt;/p&gt;&#xA;" OwnerUserId="33515" LastActivityDate="2017-06-18T21:23:12.090" Title="Retraining last layer of inception" Tags="|tensorflow|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19798" PostTypeId="1" CreationDate="2017-06-18T03:15:03.540" Score="0" ViewCount="2391" Body="&lt;p&gt;In this post, &lt;a href=&quot;https://cs231n.github.io/linear-classify/#webdemo&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cs231n.github.io/linear-classify/#webdemo&lt;/a&gt; it mentions &quot;Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities.&quot; Aren't we taking the log afterward and f is equal to just Wx+b?&lt;/p&gt;&#xA;" OwnerUserId="33011" LastActivityDate="2017-06-18T14:12:27.683" Title="Softmax cost function" Tags="|machine-learning|deep-learning|information-theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19800" PostTypeId="2" ParentId="19612" CreationDate="2017-06-18T06:11:16.563" Score="2" Body="&lt;p&gt;If you are considering the information theoretical point of view, given &lt;code&gt;x3&lt;/code&gt;and &lt;code&gt;x4&lt;/code&gt; then &lt;code&gt;x3/x4&lt;/code&gt; adds no information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, before we rush into conclusions, one must recall that there are more aspects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one is the ability to represent the concept. Let's consider as case in which your features are &lt;code&gt;x3&lt;/code&gt;and &lt;code&gt;x4&lt;/code&gt; and the concept is &lt;code&gt;x3/x4&lt;/code&gt;.&#xA;The only problem is that I force you to use a decision tree as the model.&#xA;How can you represent that? &#xA;Note that even if I allow you to choose any tree, the representation will be bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second problem is that the concept is not given in advance and therefor the algorithm should search for it. Most supervised learning algorithms are actually looking for correlations between the concept and the features. A relation like &lt;code&gt;1/x&lt;/code&gt; usually cause them problem since the behavior is not linear in x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in case that you have ideas for features that don't add information but ease representation or search, adding them might help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right asking where should I stop? It might be that there is a different combination of features that will lead to a better model.&#xA;Unfortunately, we don't have a good solution for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the problem of building good features, let's say features that reduce the size of the model. We can restrict our self to features that can be represented by trees. However, building a small tree for a data set is an &lt;a href=&quot;https://en.wikipedia.org/wiki/NP-completeness&quot; rel=&quot;nofollow noreferrer&quot;&gt;NP-complete&lt;/a&gt; problem, which means that we don't have an efficient algorithm for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More than that, you can claim that given a feature building problem you can reduce classification into it (e.g., build a feature that represents the concept). Therefore, feature building is as hard as classification.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-06-18T06:11:16.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19801" PostTypeId="1" AcceptedAnswerId="22423" CreationDate="2017-06-18T08:14:53.820" Score="9" ViewCount="12632" Body="&lt;p&gt;I am dealing with imbalanced dataset and I try to make a predictive model using MLP classifier. Unfortunately the algorithm classifies all the observations from test set to class &quot;1&quot; and hence the f1 score and recall values in classification report are 0. Does anyone know how to deal with it?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model= MLPClassifier(solver='lbfgs', activation='tanh')&#xA;model.fit(X_train, y_train)&#xA;score=accuracy_score(y_test, model.predict(X_test), )&#xA;fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])&#xA;roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])&#xA;cr=classification_report(y_test, model.predict(X_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33018" LastActivityDate="2017-08-20T16:51:34.723" Title="Imbalanced dataset in MLP classifier in python" Tags="|python|neural-network|classification|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19802" PostTypeId="1" CreationDate="2017-06-18T09:03:57.243" Score="38" ViewCount="30154" Body="&lt;p&gt;What are the best practices to save, store, and share machine learning models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ~100Mo large. Also, joblib can save one model to multiple files unless you set &lt;code&gt;compress=1&lt;/code&gt; (&lt;a href=&quot;https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre&quot;&gt;https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, if you want to control access rights to models, and be able to use models from different machines, what's the best way to store them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a few choices:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Store them as files, and then put them in a repository using Git LFS&lt;/li&gt;&#xA;&lt;li&gt;Store them in an SQL database as binary files:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For instance in Postgresql &lt;a href=&quot;https://wiki.postgresql.org/wiki/BinaryFilesInDB&quot; rel=&quot;noreferrer&quot;&gt;https://wiki.postgresql.org/wiki/BinaryFilesInDB&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;This is also the method recommended by the SQL Server team:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model&quot; rel=&quot;noreferrer&quot;&gt;https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html&quot; rel=&quot;noreferrer&quot;&gt;https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system&quot; rel=&quot;noreferrer&quot;&gt;https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;HDFS&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33519" LastEditorUserId="33519" LastEditDate="2017-06-18T13:37:49.890" LastActivityDate="2021-02-28T00:23:08.017" Title="Best practices to store Python machine learning models" Tags="|python|databases|binary|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19804" PostTypeId="2" ParentId="19798" CreationDate="2017-06-18T14:12:27.683" Score="3" Body="&lt;p&gt;No, that is not what is happening, the definition of the Softmax classifier is: $$z \in \mathbb{R}^K; \sigma(z)_j = \frac{e^{z_j}}{\sum_{k = 1}^{K} e^{z_k}} $$ Let's talk first about what the softmax function aims to do, but before we can understand that, we need to understand the aim of classification in general. Classifiers are functions which take a vector in $\mathbb{R^n}$ and output one of $k$ classes. The softmax function takes in a vector $z$ and outputs a value in $\mathbb{R^+}$ whose values sum to one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's get to your question, if I were to just take $\log(Wx + b)$. It is actually almost exactly that, we just take the exponent so that our values range from $0$ to $1$. However, these values are not normalized and in order to normalize it we need to divide by the sum and we take the exponents to maintain a positive value.  &lt;/p&gt;&#xA;" OwnerUserId="27114" LastActivityDate="2017-06-18T14:12:27.683" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19805" PostTypeId="1" CreationDate="2017-06-18T14:20:55.720" Score="0" ViewCount="4759" Body="&lt;p&gt;Is there any cloud developer platform which provides free access to an NVIDIA GPU instance, maybe only for a limited trial period? I'd like to work on it before committing to a paid option. I have tried AWS, Google Cloud Developer Platform and IBM Bluemix, but none of them have this option.&lt;/p&gt;&#xA;" OwnerUserId="27616" LastActivityDate="2017-07-19T07:03:22.910" Title="Does any cloud service provide free GPU access?" Tags="|gpu|aws|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19806" PostTypeId="2" ParentId="19612" CreationDate="2017-06-18T14:34:20.180" Score="4" Body="&lt;p&gt;I think what you're talking about is called compound features, and it's extremely important because sometimes it captures interactions between certain features which are not readily apparent in considering each column vector (or column) independently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer to your question is, it may or may not, there's really no way to know without looking at the data, for example if we have a dataset consisting of the following features $x_1, x_2, x_3, x_4$. You could create many datasets consisting of mathematical operations between these variables, like you have mentioned in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for the reason as to why someone would want to do this. Consider a supervised learning scenario where you're trying to classify a nominal feature with respect to some other features. This is a purely hypothetical example, but let's say we're predicting skin color which has $3$ or $4$ distinct classes based on facial features (absolute nose position in a grid, position of left eye, etc). While you may attain low error, it may be worthwhile considering compound features like position of right eye - position of left eye which could give us more information than their absolute positions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the latter question, there isn't any real algorithm which does this because they are feature combinations which are based on the data and anything that does this would have to be a trial and error mechanism. Because underlying data similarities may not be apparently visible. One way you could solve this is in your elementary data processing phase, consider the actual possible relationships between data, or at least, what you perceive them to be.&lt;/p&gt;&#xA;" OwnerUserId="27114" LastEditorUserId="27114" LastEditDate="2017-06-19T01:52:49.533" LastActivityDate="2017-06-19T01:52:49.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19808" PostTypeId="2" ParentId="19630" CreationDate="2017-06-18T15:48:37.227" Score="7" Body="&lt;p&gt;In your convnet code, you compute the cross entropy manually:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))&#xA;train_step = tf.train.AdamOptimizer(1e-5).minimize(cross_entropy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you do that, you may run into numerical stability issues. Instead, you should use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.nn.softmax_cross_entropy_with_logits()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If using tensorflow version 1.8 or later, use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.nn.softmax_cross_entropy_with_logits_v2()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and weep at the state of tensorflow's API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See this issue for more information: &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/2462&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/issues/2462&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="33202" LastEditDate="2018-07-30T18:42:21.877" LastActivityDate="2018-07-30T18:42:21.877" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="19809" PostTypeId="1" AcceptedAnswerId="19824" CreationDate="2017-06-18T16:13:12.683" Score="2" ViewCount="3970" Body="&lt;p&gt;i am using some algorithms from weka . i was willing to plot some algorithms' roc curve for comparison . Is it possible and how ? &lt;/p&gt;&#xA;" OwnerUserId="31072" LastActivityDate="2017-06-19T05:26:45.297" Title="is there any way to plot ROC curves from weka" Tags="|machine-learning|data-mining|dataset|java|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19812" PostTypeId="2" ParentId="19805" CreationDate="2017-06-18T16:47:52.637" Score="2" Body="&lt;p&gt;Nvidia has it's own online labs where they can help you learn GPU based data processing. Might be worth checking out for you if you want to learn and play around within the online lab environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://developer.nvidia.com/dli/onlinelabs&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://developer.nvidia.com/dli/onlinelabs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the best.&lt;/p&gt;&#xA;" OwnerUserId="33530" LastActivityDate="2017-06-18T16:47:52.637" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19813" PostTypeId="1" CreationDate="2017-06-18T17:02:55.620" Score="1" ViewCount="537" Body="&lt;p&gt;I transformed the existing code which was in python pasted below was in pyspark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import csv&#xA;&#xA;&#xA;def main():&#xA;    # create a simple JSON array&#xA;    with open('paytm_tweets_data_1495614657.json') as str:&#xA;&#xA;        tweetsList = []&#xA;        # change the JSON string into a JSON object&#xA;        jsonObject = json.load(str)&#xA;&#xA;        #print(jsonObject)&#xA;&#xA;        # # print the keys and values&#xA;        for i in range(len(jsonObject)):&#xA;            tweetsList.insert(i,jsonObject[i][&quot;text&quot;])&#xA;&#xA;        #print(tweetsList)&#xA;    displaySentiment(tweetsList)&#xA;&#xA;&#xA;&#xA;def displaySentiment(tweetsList):&#xA;    aDict = {}&#xA;&#xA;    from sentiment import sentiment_score&#xA;&#xA;    for i in range(len(tweetsList)):&#xA;        aDict[tweetsList[i]] = sentiment_score(tweetsList[i])&#xA;    print (aDict)&#xA;&#xA;&#xA;    with open('PaytmtweetSentiment.csv', 'w') as csv_file:&#xA;        writer = csv.DictWriter(csv_file, fieldnames = [&quot;Tweets&quot;, &quot;Sentiment Value&quot;])&#xA;        writer.writeheader()&#xA;        writer = csv.writer(csv_file)&#xA;        for key, value in aDict.items():&#xA;            writer.writerow([key, value])&#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Converted Pyspark Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import csv&#xA;import os&#xA;from pyspark import SparkContext, SparkConf&#xA;from pyspark.python.pyspark.shell import spark&#xA;&#xA;os.environ['PYSPARK_PYTHON'] = &quot;/usr/local/bin/python3&quot;&#xA;&#xA;&#xA;def main():&#xA;    path = &quot;/Users/i322865/DeepInsights/bitbucket-code/ai-engine/twitter-sentiment-analysis/flipkart_tweets_data_1495601666.json&quot;&#xA;    peopleDF = spark.read.json(path).rdd&#xA;    df = peopleDF.map(lambda row: row['text'])&#xA;    print(df.collect())&#xA;    displaySentiment(df.collect())&#xA;&#xA;&#xA;&#xA;def displaySentiment(tweetsList):&#xA;    from sentiment import sentiment_score&#xA;&#xA;    aDict = sentiment_score(tweetsList)&#xA;&#xA;    #&#xA;    with open('paytmtweetSentiment.csv', 'w') as csv_file:&#xA;        writer = csv.DictWriter(csv_file, fieldnames = [&quot;Tweets&quot;, &quot;Sentiment Value&quot;])&#xA;        writer.writeheader()&#xA;        writer = csv.writer(csv_file)&#xA;        for i in range(len(tweetsList)):&#xA;            writer.writerow([tweetsList[i], aDict[i]])&#xA;            print([tweetsList[i], aDict[i]])&#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    conf = SparkConf().setAppName(&quot;Test&quot;).setMaster(&quot;local&quot;)&#xA;    sc = SparkContext.getOrCreate(conf=conf)&#xA;    main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I ran both programs but didn't see any significant performance improvement. What am I missing? Please could you shed some thoughts?&lt;/p&gt;&#xA;" OwnerUserId="33531" LastEditorUserId="33531" LastEditDate="2017-06-18T17:09:49.970" LastActivityDate="2018-05-24T08:15:51.607" Title="Pyspark code is not performant enough when compared to pure python alternative" Tags="|python|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19814" PostTypeId="1" AcceptedAnswerId="19816" CreationDate="2017-06-18T17:24:03.913" Score="4" ViewCount="3252" Body="&lt;p&gt;I have a categorical variable, country which takes on values like India, US, Pakistan etc. I am currently using a linear SLM for a classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my country value varies from 1-20. How should this be a feature in the classification task. Should i have a one hot vector like (1,0,0..) for us and assign this vector 20 weights, or should i have integer from 1_20 and assign a single weight? I am using scikit learn. Does the answer depend on classifier?&lt;/p&gt;&#xA;" OwnerUserId="33532" LastActivityDate="2017-06-19T16:13:03.343" Title="Categorical Variables - Classification" Tags="|machine-learning|scikit-learn|categorical-data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19816" PostTypeId="2" ParentId="19814" CreationDate="2017-06-18T19:10:30.847" Score="2" Body="&lt;p&gt;The answer depends less on the classifier and more on the nature of the variable. In your case One Hot Encoding might be the best answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Label Encoding (Replacing categorical variables with integers) is useful when the variable is &lt;strong&gt;ordinal&lt;/strong&gt;, i.e. it has a sense of order. For example the days of the week or the months of the year. Since they follow a fixed order, you can encode January as 1 and February as 2 and so on. The classifier would interpret Feb as being greater than Jan in some way (which is okay for a task like weather prediction and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can your countries be considered to be ordinal? If not, One Hot Encode them. &lt;/p&gt;&#xA;" OwnerUserId="358" LastActivityDate="2017-06-18T19:10:30.847" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19817" PostTypeId="2" ParentId="19814" CreationDate="2017-06-18T21:09:35.320" Score="0" Body="&lt;p&gt;Make a label for all your available countries. Your Y(predictive) value will be a vector of the length = no of countries. Having just one label and varying its value will affect the accuracy as it is not a linear regression problem, you need the class labels instead of the values. Country1 - Country2 != Country3 - Country4&lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2017-06-18T21:09:35.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19818" PostTypeId="2" ParentId="19797" CreationDate="2017-06-18T21:23:12.090" Score="1" Body="&lt;p&gt;I'm running the script now.   It's just creating bottleneck files for the flowers images, i.e. the old bottleneck files are not present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the bottleneck files are all created in the /tmp directory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes sense as the point of the script is only to retrain the final layer of the network.   The idea is that the previously trained layers will work for an arbitrary set of images.&lt;/p&gt;&#xA;" OwnerUserId="33538" LastActivityDate="2017-06-18T21:23:12.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19819" PostTypeId="1" CreationDate="2017-06-18T21:51:23.687" Score="2" ViewCount="2747" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Analytically, the logarithm of the sigmoid is always defined and&#xA;  finite, because the sigmoid returns values restricted to the open&#xA;  interval (0, 1), rather than using the entire closed interval of valid&#xA;  probabilities [0, 1]. In software implementations, to avoid numerical&#xA;  problems, it is best to write the negative log-likelihood as a&#xA;  function of z, rather than as a function of ˆy = σ( z). If the sigmoid&#xA;  function underflows to zero, then taking the logarithm of ˆy yields&#xA;  negative infinity.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sigmoid function's range is closed interval [0,1]? &lt;a href=&quot;https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/1200px-Logistic-curve.svg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/1200px-Logistic-curve.svg.png&lt;/a&gt; Why does it mention as open?&lt;/li&gt;&#xA;&lt;li&gt;How does write function of z rather y affect it from not reaching 0? What are those forms?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33011" LastActivityDate="2017-06-19T10:52:59.837" Title="Sigmoid's stability" Tags="|machine-learning|neural-network|information-theory|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19820" PostTypeId="2" ParentId="11751" CreationDate="2017-06-18T22:57:32.810" Score="1" Body="&lt;p&gt;Python has similar tools to Shiny now, which make it easy to create dashboards with different widgets including sliders, plots, and tables. For example, &lt;a href=&quot;https://github.com/jwkvam/bowtie&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bowtie&lt;/a&gt; let's you do this. It requires more setup than Jupyter widgets but it has more power in my opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a decent list of similar tools for python &lt;a href=&quot;https://www.quora.com/Is-there-something-similar-to-R-Shiny-for-Python-users-in-the-scientific-community&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33540" LastActivityDate="2017-06-18T22:57:32.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19822" PostTypeId="2" ParentId="19819" CreationDate="2017-06-19T04:16:18.830" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You are correct, the sigmoid's range is $(0,1)$. The quote you shared is just saying, trivially, that anything in $[0,1]$ is a valid probability.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The sigmoid function gets very small, very quickly (because $e^{-z}$ grows exponentially as $z \to - \infty$ by definition). So the computer will round $\sigma(z)$ to zero for values of $z$ that aren't even that large in magnitude. This means you shouldn't use the output $y$ of the sigmoid in the log-likelihood; you should just build the sigmoid into the likelihood in a way that avoids numerical issues. Then, modestly large inputs $z$ won't give you numerical problems.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16212" LastActivityDate="2017-06-19T04:16:18.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19824" PostTypeId="2" ParentId="19809" CreationDate="2017-06-19T05:26:45.297" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;In the Weka explorer, go to the classify tab and train/test your algorithm.&lt;/li&gt;&#xA;&lt;li&gt;The result buffer appears in the bottom left box under the section labeled &quot;result list&quot;&lt;/li&gt;&#xA;&lt;li&gt;Right click the result buffer and click visualize threshold curve, then select the class you want to analyze&lt;/li&gt;&#xA;&lt;li&gt;to save the ROC curve as an image, hold shift + alt and left click on the graph&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2017-06-19T05:26:45.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19825" PostTypeId="1" CreationDate="2017-06-19T05:29:30.593" Score="1" ViewCount="37" Body="&lt;p&gt;I have been working on a project where I am supposed to combine a generative and a discriminative model by exploiting their geometrical feature.&#xA;There is a method where you encode the generative information in the kernel and then use this kernel in a discriminative method.&#xA;I can't find a method on how to create a generative kernel.Can someone help me how to get generative kernel?I am using&lt;i&gt;&lt;b&gt; EEG Eye State&lt;/i&gt;&lt;/b&gt; from UCI repository.&lt;/p&gt;&#xA;" OwnerUserId="33547" LastEditorUserId="33547" LastEditDate="2017-06-19T20:27:29.553" LastActivityDate="2017-06-19T20:27:29.553" Title="Extraction of kernel from Generative model?" Tags="|algorithms|" AnswerCount="0" CommentCount="0" ClosedDate="2017-06-19T10:09:07.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="19826" PostTypeId="2" ParentId="19819" CreationDate="2017-06-19T07:29:17.203" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The sigmoid range $(0,1)$ is technically open, because no input value maps to $0$ or $1$. You can get arbitrarily close to $0$ or $1$ but never equal them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Taking an exponent followed by a log can cause overflow issues during computation, because exponents grow quickly. Sigmoid also gets close to zero quickly and this can underflow (potentially rounding to 0). However, the end result of the combined function is likely to be something within normal calculation range, because of the log (this is similar to - but even more extreme - multiplying and dividing by a very large number). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Although your quote suggests:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;it is best to write the negative log-likelihood as a function of z&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is no analytical form where $E = -log(\sigma(z))$ can be re-written as a simple function of z. In theory, it can be calculated numerically by expanding terms carefully (beyond me, so won't show here), and some libraries may include this kind of expansion. In practice, to avoid numerical instability, many machine learning libraries will simply cap the value like this: $E = -log( max(\epsilon, \sigma(z)) )$ with $\epsilon$ a small number, maybe $10^{-15}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more common issue where the calculations are done accurately in neural networks is when considering the gradient of a loss function. If you use a sigmoid output layer alongside a binary cross-entropy cost function $E = -(y log(\sigma(z)) + (1-y)log(1-\sigma(z))) $, then some of the terms cancel out, and then the gradient contribution from training is trivially $\frac{\partial{E}}{\partial z} = y - \sigma(z)$ - there is no need in that case to calculate any log values despite them being in the loss function.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-19T10:52:59.837" LastActivityDate="2017-06-19T10:52:59.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19827" PostTypeId="2" ParentId="19794" CreationDate="2017-06-19T08:18:30.867" Score="0" Body="&lt;p&gt;First of all, interesting question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most important thing to recommend you a starting point is to know what kind of data you have in hand. So my answer is regardless of that fact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a road-map of studies of students who got into University XXX then you can use model it with an &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;HMM&lt;/a&gt; (Hidden Markov Model). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case the observations are the sequence of courses one passed and you want to calculate the probability of landing in desired university. Transition matrix is the probability of getting course A after getting course B. Emission matrix might be something like how successful they were in the course. Transition and emission matrices can be scored using facts like dropping a course or failing or the education track they exhibit in.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-06-19T08:18:30.867" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19828" PostTypeId="2" ParentId="4944" CreationDate="2017-06-19T11:24:33.477" Score="0" Body="&lt;p&gt;If the data set is highly imbalanced i would suggest you yo use Structural SVM instead of basic classification model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From section 3.3 of the paper - &lt;strong&gt;predicting structured objects with support vector machines&lt;/strong&gt;&lt;a href=&quot;http://www.yisongyue.com/publications/cacm2009_structsvm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;&#xA;What does this mean for learning? Instead of optimiz-&#xA;ing some variant of error rate during training, which is what&#xA;conventional SVMs and virtually all other learning algo-&#xA;rithms do, it seems like a natural choice to have the learning&#xA;algorithm directly optimize, for example, the F 1 -Score (i.e.,&#xA;the harmonic mean of precision and recall). This is the point&#xA;where our binary classification task needs to become a struc-&#xA;tured output problem, since the F 1 -Score (as well as many&#xA;other IR measures) is not a function of individual examples&#xA;(like error rate), but a function of a set of examples. In partic-&#xA;ular, we arrive at the structured output problem of predict-&#xA;ing an array of labels y = ( y 1 , ..., y n ), y i Î {−1, +1}, for an array&#xA;of feature vectors x = (x 1 , ..., x n ), x i Î Â N . Each possible array&#xA;of labels y– now has an associated F 1 -Score F 1 (y, – y ) w.r.t. the&#xA;true labeling y, and optimizing F 1 -Score on the training set&#xA;becomes a well-defined problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest you to read the entire paper.&#xA;For implementation you can use python pystruct library&lt;a href=&quot;https://pystruct.github.io/user_guide.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1257" LastActivityDate="2017-06-19T11:24:33.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19829" PostTypeId="2" ParentId="5226" CreationDate="2017-06-19T11:43:20.267" Score="3" Body="&lt;p&gt;You can use dummy variables in such scenarios. With panda's &lt;code&gt;panda.get_dummies&lt;/code&gt; you can create dummy variables for strings you want to put in Decision Tree or Random Forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;d = {'one' : pd.Series([1., 2., 3.,4.], index=['a', 'b', 'c','d']),'two' :pd.Series(['Paul', 'John', 'Micheal','George'], index=['a', 'b', 'c', 'd'])}&#xA;df = pd.DataFrame(d)&#xA;&#xA;df_with_dummies= pd.get_dummies(df,columns=[&quot;two&quot;],drop_first=False)&#xA;df_with_dummies&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33567" LastActivityDate="2017-06-19T11:43:20.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19831" PostTypeId="2" ParentId="19612" CreationDate="2017-06-19T12:49:22.873" Score="2" Body="&lt;p&gt;I agree mostly with what was already said regarding feature engineering and just to provide you with more material &lt;a href=&quot;http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt; has a nice analysis regarding different stages in feature engineering, with many links and references to papers and specific challenges. I think it's worth checking out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, to some extend you could automate the task of finding &quot;good&quot; compound features, using &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_method&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernel methods&lt;/a&gt;. You could use some of the standard kernel methods implemented in many libraries (e.g. in &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#svm-kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;) as feature extractors and feed these higher-level features as input to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest#Variable_importance&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest model&lt;/a&gt; that inherently uses feature-importance while training. Then keep only most-informative of these higher-dimension features, as scored by the trained model, as compound features alongside the raw data.  &lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-06-19T12:49:22.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19833" PostTypeId="2" ParentId="19814" CreationDate="2017-06-19T16:13:03.343" Score="0" Body="&lt;p&gt;Another possibility is to order you country-variable by the rate of the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For ex. in the case of the binary classification problem (0-1), you can order your countries by rate of the majority class and then assign the number from 1 to 20. This coding allow you to make the logical operations with this variable.&lt;/p&gt;&#xA;" OwnerUserId="33478" LastActivityDate="2017-06-19T16:13:03.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19835" PostTypeId="2" ParentId="14998" CreationDate="2017-06-19T17:13:22.987" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://www.dominodatalab.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Domino Data Lab&lt;/a&gt; offers premises, SaaS, and VPC-based notebook hosting (Jupyter, Zeppelin, RStudio), git integration, scalable compute, environment templates, and a bunch of other useful things. The premises/ VPC offerings may be overkill and too pricey if you're a small team, but the SaaS plans are pretty reasonably priced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ Full disclosure: I'm a former Domino employee ]&lt;/p&gt;&#xA;" OwnerUserId="5233" LastActivityDate="2017-06-19T17:13:22.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19836" PostTypeId="1" CreationDate="2017-06-18T04:47:17.503" Score="1" ViewCount="4853" Body="&lt;p&gt;Can someone please refer a good article explaining why we use a Sigmoid activation in the final layer of a neural network and a ReLu activation in the middle layers and input layer while building a Convolutional Neural Network? I am not getting how that results in the right output.&lt;/p&gt;&#xA;" OwnerDisplayName="Venkatesh Tata" LastActivityDate="2017-06-19T17:24:38.453" Title="When to use a Sigmoid function and when to use ReLu in a Convolutional Neural Network" Tags="|machine-learning|deep-learning|keras|" AnswerCount="1" CommentCount="0" ClosedDate="2017-06-24T09:12:36.427" ContentLicense="CC BY-SA 3.0" />
  <row Id="19837" PostTypeId="2" ParentId="19836" CreationDate="2017-06-18T06:55:36.150" Score="3" Body="&lt;p&gt;&lt;code&gt;Relu&lt;/code&gt; solves the gradient vanishing problem and stops the inactive neurons. Using &lt;code&gt;Sigmoid&lt;/code&gt; in output layer because its range is (0, 1) and it can represent the probability of binary class. When you are doing multi-classification, it is more appropriate to use &lt;code&gt;Softmax&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you need to read more materials, you can refer to &lt;a href=&quot;http://cs231n.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n&lt;/a&gt;. Thanks.&lt;/p&gt;&#xA;" OwnerDisplayName="danche" LastActivityDate="2017-06-18T06:55:36.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19839" PostTypeId="2" ParentId="17070" CreationDate="2017-06-19T17:52:17.307" Score="1" Body="&lt;p&gt;Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QDHkn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QDHkn.png&quot; alt=&quot;w1 and w2 are the base area and E the height&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In 3D space things can be perpendicular to each other in two ways)&#xA;So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D&lt;/p&gt;&#xA;" OwnerUserId="33584" LastActivityDate="2017-06-19T17:52:17.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19840" PostTypeId="1" CreationDate="2017-06-19T18:57:28.623" Score="7" ViewCount="9348" Body="&lt;p&gt;What is Python's alternative to missing data imputation with &lt;code&gt;mice&lt;/code&gt; in R? Imputation using median/mean seems pretty lame, I'm looking for other methods of imputation, something like &lt;code&gt;randomForest&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user25935" LastActivityDate="2018-07-19T01:02:28.983" Title="R's mice imputation alternative in Python" Tags="|python|r|data-imputation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19842" PostTypeId="1" CreationDate="2017-06-19T19:40:30.820" Score="6" ViewCount="5851" Body="&lt;p&gt;I am building a standard RandomForest classifier (named model, see the code below) using scikit-learn package. Now, I want to get all parameters of one Randomforest classifier (including its trees (estimators)), so that I can manually draw the flow chart for each tree of the RandomForest classifier. I wonder if anyone knows how it can be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Manh&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Import Library&#xA;from sklearn.ensemble import RandomForestClassifier #use RandomForestRegressor for regression problem&#xA;#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset&#xA;# Create Random Forest object&#xA;model= RandomForestClassifier(n_estimators=10, max_depth=5) #n_estimators=1000 oob_score = True&#xA;#====&#xA;#X, y = input_X, input_y&#xA;&#xA;from sklearn.cross_validation import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 4)&#xA;&#xA;# Train the model using the training sets and check score&#xA;model.fit(X_train, y_train)&#xA;&#xA;#Predict Output&#xA;y_pred_train = model.predict(X_train)&#xA;y_pred_test = model.predict(X_test)&#xA;&#xA;#accuracy&#xA;from sklearn.metrics import accuracy_score&#xA;print(accuracy_score(y_train,y_pred_train))&#xA;print(accuracy_score(y_test,y_pred_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33587" LastActivityDate="2019-06-04T11:01:13.397" Title="Anyway to know all details of trees grown using RandomForestClassifier in scikit-learn?" Tags="|scikit-learn|random-forest|decision-trees|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19844" PostTypeId="1" AcceptedAnswerId="19846" CreationDate="2017-06-20T07:29:09.913" Score="3" ViewCount="99" Body="&lt;p&gt;I often see in a well-known datascience competition platform, that a lot of people apply some dimension reductions techniques, but instead of using it to reduce the number of features (complexity) of their models, they append the resulting features to their datasets. Is'nt that adding complexity to their model rather than simplifying it ? &lt;/p&gt;&#xA;" OwnerUserId="33386" LastActivityDate="2017-06-20T10:16:34.987" Title="Why some people add results from PCA and other dimensional reductions techniques as features" Tags="|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19845" PostTypeId="2" ParentId="19790" CreationDate="2017-06-20T09:34:39.063" Score="4" Body="&lt;p&gt;I would use a two-step approach, using the idea of the $\hat{c_4}$ class you mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first step, use a binary classifier(trained on the whole dataset) to decide if a sample belongs to the class $\hat{c_4}$ (i.e. in any non-interesting class). For this, step you could also take a look in &lt;a href=&quot;http://machinelearningmastery.com/how-to-identify-outliers-in-your-data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;outlier detection&lt;/a&gt; methods, if the samples belonging in the &quot;interesting&quot; classes are much different than the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the result is negative, move on to the next step, a new classifier trained only on samples belonging in the classes $c_1,c_2,c_3$ and use that prediction as your final one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that even using a simple clustering approach as a first step (e.g. 4-clustering &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means&lt;/a&gt; using as initial centroid values the average centroid $cent_j = \frac{\sum\limits_{x_i\in D: y_i=j}x_i}{\sum\limits_{x_i\in D: y_i=j}1}$  for each $c_1,c_2,c_3, \hat{c_4}$), would still be useful.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-06-20T11:28:43.900" LastActivityDate="2017-06-20T11:28:43.900" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19846" PostTypeId="2" ParentId="19844" CreationDate="2017-06-20T10:16:34.987" Score="3" Body="&lt;p&gt;This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's assume you want to learn the US road directions to a simple ML model. You give him all examples from roads 1 to 100 mapping to 0 (if the road is east-west), or 1 (road is north-south), except for the number 47. Then you ask the model to predict 47 and it will answer 0 because it's between 46 and 48 who are labelled 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you give the model another point of view (here, the parity of the road number), then it will obviously learn efficiently the road directions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can view the results of the PCA as the parity, just another view of the data to understand it better.&lt;/p&gt;&#xA;" OwnerUserId="33598" LastActivityDate="2017-06-20T10:16:34.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19847" PostTypeId="1" CreationDate="2017-06-20T10:49:11.520" Score="1" ViewCount="117" Body="&lt;p&gt;I am looking to get my Masters in Science in Data Science or Analytics. I will continue working during the program so it needs to be part-time. I would prefer to do an online course, but if in-person is much better, I could be open to it. So far, I found Georgia Tech and Johns Hopkins for online, part-time programs. For in-person, I could attend Georgetown part-time. Any insight you can provide regarding which of these programs would be best, if there is another school that could be better for my situation, or if any school is essentially the same would be very helpful! Thanks!&lt;/p&gt;&#xA;" OwnerUserId="33604" LastActivityDate="2017-06-20T17:17:45.143" Title="What are the top schools for online part-time Masters in data science/analytics programs? Is it much better to do the program in person?" Tags="|education|" AnswerCount="1" CommentCount="0" ClosedDate="2017-06-24T09:11:10.937" ContentLicense="CC BY-SA 3.0" />
  <row Id="19848" PostTypeId="1" CreationDate="2017-06-20T11:36:08.213" Score="2" ViewCount="496" Body="&lt;p&gt;When searching for some pretrained models for object detection with bounding boxes, I was wondering if there are also pretrained models on the ImageNet dataset for bounding boxes. Most of the time, I find models trained on the VOC or COCO dataset. Is there a reason for this? I'd like to have a model that is pretrained on many categories. If I saw correctly, the ImageNet dataset is much richer than the VOC or COCO dataset.&lt;/p&gt;&#xA;" OwnerUserId="27237" LastEditorUserId="28175" LastEditDate="2018-01-10T16:34:56.213" LastActivityDate="2018-02-01T12:18:47.170" Title="Are there pretrained models on the ImageNet Bounding Boxes dataset?" Tags="|machine-learning|deep-learning|tensorflow|image-recognition|cnn|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19849" PostTypeId="1" AcceptedAnswerId="19851" CreationDate="2017-06-20T12:39:59.643" Score="6" ViewCount="5806" Body="&lt;p&gt;Suppose you have some training dataset that you want to use to train some ML models, where targets are comprised between let's say 1 and 100. However, from the 4000 samples, there are few of them (less than 10) which have values out of the previous range, much higher than 100, let's say 300. Is it reasonable to ignore these samples and remove them from the data set, or should they be kept ? I saw people react differently, some of them say that they may hurt the model, while others say no as these samples give additional information to the model. &lt;/p&gt;&#xA;" OwnerUserId="33386" LastActivityDate="2021-12-14T10:59:25.140" Title="Is it good to remove outliers from the dataset?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19850" PostTypeId="1" AcceptedAnswerId="19855" CreationDate="2017-06-20T13:03:28.923" Score="3" ViewCount="879" Body="&lt;p&gt;Following &lt;a href=&quot;https://github.com/chiphuyen/tf-stanford-tutorials/blob/master/examples/03_logistic_regression_mnist_sol.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; tutorial, I have a doubt about the evaluation part in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# test the model&#xA;n_batches = int(mnist.test.num_examples/batch_size)&#xA;total_correct_preds = 0&#xA;for i in range(n_batches):&#xA;    X_batch, Y_batch = mnist.test.next_batch(batch_size)&#xA;    _, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch}) &#xA;    preds = tf.nn.softmax(logits_batch)&#xA;    correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y_batch, 1))&#xA;    accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) # need numpy.count_nonzero(boolarr) :(&#xA;    total_correct_preds += sess.run(accuracy)   &#xA;&#xA;print 'Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this is done on the test set, so the goal is purely to obtain the accuracy using a previously trained model. However isn't calling the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch}) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;equivalent to re-optimizing the model using the test data (and labels)?&#xA;Shouldn't we avoid re-running optimizer and loss and just compute the predictions?&lt;/p&gt;&#xA;" OwnerUserId="28685" LastActivityDate="2017-06-20T15:54:46.117" Title="Evaluating Logistic Regression Model in Tensorflow" Tags="|machine-learning|tensorflow|logistic-regression|training|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19851" PostTypeId="2" ParentId="19849" CreationDate="2017-06-20T13:10:40.597" Score="8" Body="&lt;p&gt;It mostly depends on what you are trying to achieve with your model.&#xA;Sometimes the information carried by outliers is indeed &lt;em&gt;negligible&lt;/em&gt; if not of interest (say, for example, that those high values are caused by data collection/input &lt;strong&gt;errors&lt;/strong&gt;), and may affect your model performance. In some other cases though, outliers carry a lot of &lt;strong&gt;meaning&lt;/strong&gt; and you might want your model to be &lt;strong&gt;aware&lt;/strong&gt; of their existence/possibility. In other scenarios, the outliers are what you actually care about (see &lt;strong&gt;Anomaly/Novelty detection&lt;/strong&gt; e.g.).&lt;/p&gt;&#xA;&lt;p&gt;Long story short, if these outliers are really such (i.e. they appear with a very &lt;strong&gt;low frequency&lt;/strong&gt; and very likely are bad/random/&lt;strong&gt;corrupted measurements&lt;/strong&gt;) and they do not correspond to potential events/failures that your model should be aware of, you can safely &lt;strong&gt;remove&lt;/strong&gt; them. In all other cases you should &lt;strong&gt;evaluate&lt;/strong&gt; case by case what those outliers represent.&lt;/p&gt;&#xA;" OwnerUserId="28685" LastEditorUserId="28685" LastEditDate="2021-12-14T10:59:25.140" LastActivityDate="2021-12-14T10:59:25.140" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19852" PostTypeId="1" CreationDate="2017-06-20T14:05:58.707" Score="1" ViewCount="16" Body="&lt;p&gt;I have a regression problem with a number of inputs. The problem also has several &quot;configurations&quot; which change the inputs drastically. But within the same configuration the inputs are changing much less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Older research has dealt with the problem by training a separate neural network for each configuration. I have used the approach to one-hot encode the configuration and feed them into the neural network as inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works well, but i wonder if there might be a better way of doing this. I have only tried a multi-layer perceptron. Would a CNN, RNN, LSTM be of use to inherently separate the different configurations?&lt;/p&gt;&#xA;" OwnerUserId="33609" LastActivityDate="2017-06-20T14:05:58.707" Title="How to reflect different &quot;configurations&quot; of a problem for training an ANN" Tags="|neural-network|regression|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19853" PostTypeId="1" CreationDate="2017-06-20T14:32:55.467" Score="0" ViewCount="738" Body="&lt;p&gt;I have a list of survey responses and want to analyze the frequency of words used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Orange, I've loaded the file into a Corpus, then Preprocess Text, then Word Cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Word Cloud does not show the cloud image, and I can't figure out why. Everything appears to be set up just like the Orange documentation shows online. Word Cloud does show a nice list of words and their frequencies, but I can't copy/paste that into Excel or export that list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I export that list of word frequencies into a table where one column is the word, and the other column is the count? Is there an easier way to do this?&lt;/p&gt;&#xA;" OwnerUserId="33610" LastActivityDate="2022-06-30T20:12:17.590" Title="How do I get a word frequency list? Can't export from Word Cloud" Tags="|orange|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19854" PostTypeId="1" AcceptedAnswerId="19918" CreationDate="2017-06-20T15:54:36.217" Score="0" ViewCount="5910" Body="&lt;p&gt;I'm just doing a simple linear regression with gradient descent in the multivariate case. Feature normalization/scaling is a standard pre-processing step in this situation, so I take my original feature matrix $X$, organized with features in columns and samples in rows, and transform to $\tilde{X}$, where, on a column-by-column basis, &#xA;$$\tilde{X}=\frac{X-\bar{X}}{s_{X}}.$$&#xA;Here, $\bar{X}$ is the mean of a column, and $s_{X}$ is the sample standard deviation of a column. Once I've done this, I prepend a column of $1$'s to allow for a constant offset in the $\theta$ vector. So far, so good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I did &lt;strong&gt;&lt;em&gt;not&lt;/em&gt;&lt;/strong&gt; do feature normalization, then my prediction, once I found my $\theta$ vector, would simply be $x\cdot\theta$, where $x$ is the location at which I want to predict the outcome. But now, if I &lt;strong&gt;&lt;em&gt;am&lt;/em&gt;&lt;/strong&gt; doing feature normalization, what does the prediction look like? I suppose I could take my location $x$ and transform it according to the above equation on an element-by-element basis. But then what? The outcome of $\tilde{x}\cdot\theta$ would not be in my desired engineering units. Moreover, how do I know that the $\theta$ vector I've generated via gradient descent is correct for the un-transformed locations? I realize all of this is a moot point if I'm using the normal equation, since feature scaling is unnecessary in that case. However, as gradient descent typically works better for very large feature sets ($&amp;gt; 10k$ features), this would seem to be an important step. Thank you for your time!&lt;/p&gt;&#xA;" OwnerUserId="32440" LastActivityDate="2017-06-22T15:51:32.843" Title="Feature Normalization/Scaling: Prediction Step" Tags="|linear-regression|gradient-descent|prediction|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19855" PostTypeId="2" ParentId="19850" CreationDate="2017-06-20T15:54:46.117" Score="4" Body="&lt;p&gt;I think you are correct. The line should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss_batch, logits_batch = sess.run([loss, logits], feed_dict={X: X_batch, Y:Y_batch}) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19046" LastActivityDate="2017-06-20T15:54:46.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19856" PostTypeId="2" ParentId="14331" CreationDate="2017-06-20T16:37:47.973" Score="1" Body="&lt;p&gt;One way to look for number of clusters is through VAT (Visual Assessment of Tendency), the reference you have provided: Alboukadel Kassambara also talks about that. This is a more frequently used measure by the machine learning community.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also explore Hierarchical tree and look for patterns of your interest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end, it is subjective and application dependent.&lt;/p&gt;&#xA;" OwnerUserId="33612" LastActivityDate="2017-06-20T16:37:47.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19857" PostTypeId="1" CreationDate="2017-06-20T16:43:00.147" Score="1" ViewCount="789" Body="&lt;p&gt;I am currently experimenting with user agent strings. My current plan is to tokenize the user agent string and split it in its important parts. Like OS, Browser Name, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, does it make sense to just create some sort of map where iOS = 0.1 and Windows = 0.2 where i map the strings to numbers for the operating system feature of my network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that the current best approach to work with such data, or should I try a different approach? The network should predict which category the user will click, and than move that category to a more prominent place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I expect that an iOS user prefers an iPhone mobile case more than an android one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for helping with my experiments.&lt;/p&gt;&#xA;" OwnerUserId="33613" LastActivityDate="2021-03-13T17:00:09.167" Title="How to quantify a tokenized user agent string for a neural network?" Tags="|machine-learning|neural-network|feature-extraction|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19858" PostTypeId="1" CreationDate="2017-06-20T16:47:24.890" Score="2" ViewCount="1044" Body="&lt;p&gt;I do not know if this is the right place to ask such a generic question, if not please let me know and I will remove it asap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, I am a Software Engineering student (BEng hons) and this September my 3rd year starts and therefore I have to start my final year project. I want to do something in &lt;strong&gt;Machine Learning&lt;/strong&gt; and more specifically in &lt;strong&gt;Prediction Algorithms&lt;/strong&gt;. &#xA;I would like to do predictions for &lt;strong&gt;&lt;em&gt;stock prices&lt;/em&gt;&lt;/strong&gt; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been reading a book provided by my university online the last days, but it doesnt make that much sense as there are a lot of algorithms and different approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithms would work for the project I want to do????&#xA;Do you know any specific books or online courses that you would recommend?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any guidance would be really much appreciated! &#xA;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="33614" LastActivityDate="2017-06-23T11:00:17.893" Title="Intro to Machine Learning" Tags="|machine-learning|" AnswerCount="5" CommentCount="5" ClosedDate="2017-06-23T22:01:50.123" ContentLicense="CC BY-SA 3.0" />
  <row Id="19859" PostTypeId="2" ParentId="19858" CreationDate="2017-06-20T16:50:12.437" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng's course on Machine Learning on Coursera&lt;/a&gt; is terrific. I'm doing it right now, and he has wonderfully gauged student understanding. He helps you along and doesn't skip lots of steps. It apparently is &lt;em&gt;not&lt;/em&gt; his goal to convince you of how smart he is - he's more concerned with explaining everything clearly. Highly recommended!&lt;/p&gt;&#xA;" OwnerUserId="32440" LastActivityDate="2017-06-20T16:50:12.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19860" PostTypeId="1" CreationDate="2017-06-20T17:01:30.177" Score="1" ViewCount="224" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The constraint that the n outputs must sum to $1$ means that only $n−1$&#xA;  parameters are necessary; the probability of the $n^{th}$ value may be&#xA;  obtained by subtracting the first $n−1$ probabilities from $1$. We can&#xA;  thus impose a requirement that one element of $z$ be fixed. For example,&#xA;  we can require that $z_n = 0$. Indeed, this is exactly what the sigmoid&#xA;  unit does. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What does it mean by &quot;impose a requirement that one element of $z$ be fixed.&quot; I understand you can get the $n^{th}$ one if you know n-1 but &quot;fixed&quot;?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Defining $P (y = 1 | x ) = \sigma(z )$ is equivalent to defining $P&#xA;(y = 1 | x) = softmax(z)_1$ with a two-dimensional $z$ and $z_1 = 0$. Both&#xA;  the $n − 1$ argument and the $n$ argument approaches to the softmax can&#xA;  describe the same set of probability distributions, but have different&#xA;  learning dynamics.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;How is it equivalent to having $z_1=0$?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33011" LastEditorUserId="836" LastEditDate="2017-06-20T17:49:11.177" LastActivityDate="2017-08-19T20:35:23.670" Title="Softmax function forms" Tags="|neural-network|deep-learning|probability|theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19861" PostTypeId="2" ParentId="19847" CreationDate="2017-06-20T17:17:45.143" Score="0" Body="&lt;p&gt;Michigan University has a nice and good dual program that you get your Master in Data Science and also a MBA.&#xA;I'd take a look at MIT's Master of Business Analytics as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get a job, I don't think a Master is necessary. This Forbes article explains that pretty well &lt;a href=&quot;https://www.forbes.com/sites/metabrown/2016/07/29/4-reasons-not-to-get-that-masters-in-data-science/#160f195a40c0&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.forbes.com/sites/metabrown/2016/07/29/4-reasons-not-to-get-that-masters-in-data-science/#160f195a40c0&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="7939" LastActivityDate="2017-06-20T17:17:45.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19862" PostTypeId="2" ParentId="19860" CreationDate="2017-06-20T19:23:04.960" Score="1" Body="&lt;p&gt;The book quotes in your question are the outline of a proof that shows single output sigmoid represents the same class probabilities as a two dimensional softmax.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;What does it mean by &quot;impose a requirement that one element of $z$ be fixed.&quot; I understand you can get the $n^{th}$ one if you know n-1 but &quot;fixed&quot;?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It doesn't mean the value has to be fixed. It means that we &lt;em&gt;can&lt;/em&gt; choose any $z_n$ value, and have the other's adjusted to represent any valid target probabilities. So for argument's sake, &lt;em&gt;we&lt;/em&gt; can pick any fixed value.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;How is it equivalent to having $z_1=0$?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is an equivalence between $\sigma(z)$ and $softmax(z)_1$ with $z_1 = 0$:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$\sigma(z) = \frac{1}{1 + e^{-z}}$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For softmax, this is the function for the first element from a 2-dimensional output:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$softmax(z)_1 = \frac{e^{z_1}}{\sum_{i=1}^{2}e^{z_i}} = \frac{e^{z_1}}{e^{z_1} + e^{z_2}}$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;But we have set $z_1 = 0$, and $e^0 = 1$ so:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$softmax(z)_1 = \frac{1}{1 + e^{z_2}}$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the same function as $\sigma(-z_2)$, so the two functions have identical form (if you plot them, they overlap), although the parameters drive the specific value differently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that as per step 1, we have not lost anything about the general behaviour by setting $z_1 = 0$, we just did it to simplify the equation. In fact it is possible to make a &lt;a href=&quot;https://stats.stackexchange.com/a/254071/43775&quot;&gt;very similar argument without fixing any $z$ values&lt;/a&gt;, and you end up with a more generic linear combination of $z_1$ and $z_2$ in the softmax function being equivalent to just $z$ in $\sigma(z)$. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-20T19:43:27.967" LastActivityDate="2017-06-20T19:43:27.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19864" PostTypeId="2" ParentId="19858" CreationDate="2017-06-20T22:24:52.957" Score="2" Body="&lt;p&gt;Try &lt;a href=&quot;https://www.datacamp.com/home&quot; rel=&quot;nofollow noreferrer&quot;&gt;Datacamp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a fantastic site with video tutorials and an online editor so you don't need to bother messing around installing an IDE and the right packages and getting them all to work, you can just work in your browser! They have lectures from industry leaders and professors, each short course is a nice bitesize chunk of 4 hours estimated including the interactive exercises.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best part is that because each course is only 4 hours long, you can pick and choose which ones are relevant to you, such as diving straight into machine learning if you already have a good python or R background, and finding out how to properly clean and prepare your data, or engineer new features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-06-20T22:24:52.957" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19865" PostTypeId="1" AcceptedAnswerId="19920" CreationDate="2017-06-20T22:32:53.760" Score="3" ViewCount="1444" Body="&lt;p&gt;Forgive me if this is a duplicate question, I haven't found anything that answers my question specifically after searching for a while.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset which I'm using to predict mobile app user retention, using the RandomForestClassifier in the SciKit Learn package. I'm pleased with the accuracy I'm getting and I'm planning on including a number of other metrics including precision, recall, Matthew's Corr Coefficient etc. I'm pretty sure the model is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key thing that I'm interested in here are the features themselves. I want to know what is contributing to my user churn. I have extracted the feature importances and plotted a nice looking graph, but now I'm stuck. I'd ideally like to know &lt;em&gt;how&lt;/em&gt; each variable influences the churned/not churned outcome. The problem with GINI feature importances is that I can see which ones are most influential, but for example with continuous variables I want to know at which value the RF found best to split on. I don't need to see this for every feature as I have 70+ only the most 'important' ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw a very nice decision tree plot &lt;a href=&quot;https://datascience.stackexchange.com/questions/10193/how-to-interpret-a-decision-tree-correctly&quot;&gt;here&lt;/a&gt;, but cannot find any way of reproducing something similar using scikit learn. I'm open to other suggestions. Thanks in advance :)&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-06-22T16:48:27.677" Title="How to further Interpret Variable Importance?" Tags="|machine-learning|feature-selection|random-forest|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19866" PostTypeId="2" ParentId="19853" CreationDate="2017-06-21T06:32:00.653" Score="1" Body="&lt;p&gt;Ad 1) Word Cloud has a silly bug that causes the visualization sometimes not displaying. Press Regenerate word cloud and the visualization will appear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ad 2) This is currently not possible in Orange, but it's surprising how we haven't thought about it yet. We will make necessary modifications and the export should be available in the next release.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2017-06-21T06:32:00.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19869" PostTypeId="1" CreationDate="2017-06-21T07:00:51.717" Score="12" ViewCount="5804" Body="&lt;p&gt;I'm a beginner in neural networks and currently I'm exploring the word2vec model. However I'm having a tough time to understand what the feature matrix exactly is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gdhMo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gdhMo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can understand that the first matrix is a one-hot encoding vector for a given word, but what does the second matrix signify? More specifically, what does each of those values (ie. 17, 24, 1 etc) mean?&lt;/p&gt;&#xA;" OwnerUserId="33637" LastActivityDate="2017-11-23T10:44:05.073" Title="What is the feature matrix in word2vec?" Tags="|machine-learning|neural-network|word2vec|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19870" PostTypeId="1" CreationDate="2017-06-21T07:27:09.520" Score="1" ViewCount="139" Body="&lt;p&gt;I have a lot of missing values for some variables in my data (70-80%). I have seen some people deal with missing values this way: encode the variable with missing values as 0 or 1. Where 0 is the value is missing and 1 as non missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if that technique is of any use, because I don't see any valuable information algorithm would able to pick from such variables. Also I am thinking of imputing them using mice but the problem is that in future use, we may not be able to get those variables with missing data, so the train and test set will have different number of columns &lt;/p&gt;&#xA;" OwnerUserId="31822" LastEditorUserId="69109" LastEditDate="2019-04-19T23:54:09.600" LastActivityDate="2019-04-19T23:54:09.600" Title="Does encoding missing data with fixed values help in classification?" Tags="|classification|missing-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19871" PostTypeId="1" CreationDate="2017-06-21T09:02:45.410" Score="2" ViewCount="715" Body="&lt;p&gt;I have an problem with tensorflow. I created a class for fully-connected layer with parameter in its constructor &lt;code&gt;is_hidden_layer&lt;/code&gt; by which I decide to use or not to use relu and batch normalization on layer output. Code goes like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class FullLayer(object):&#xA;  def __init__(self, neurons, is_hidden_layer):&#xA;    self.is_hidden_layer = is_hidden_layer&#xA;    self.neurons = neurons&#xA;&#xA;  def build(self, input):&#xA;    # ... uninteresting initialization code ... &#xA;    W = tf.Variable(tf.truncated_normal([input_shape, self.neurons], stddev=0.1))&#xA;    b = tf.Variable(tf.truncated_normal([self.neurons], stddev=0.1))&#xA;    hidden = tf.Variable(tf.constant(self.is_hidden_layer), name=&quot;is_hidden&quot;)&#xA;    output = tf.matmul(input, W) + b&#xA;    output = self.batch_norm(output)&#xA;    return tf.cond(tf.equal(hidden, True), lambda: tf.nn.relu(output), lambda: output)&#xA;&#xA;  def batch_norm(self, input):&#xA;    with tf.variable_scope('batch_normalize'):&#xA;      mean = tf.Variable(tf.constant(0.0, shape=[input.shape[-1]]),&#xA;                        trainable=True)&#xA;      # ... not interesting batch normalization code ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code works fine. But if change last two lines of &lt;code&gt;build&lt;/code&gt; to this one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return tf.cond(tf.equal(hidden, True), lambda: tf.nn.relu(self.batch_norm(output)), lambda: output)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I put &lt;code&gt;batch_norm&lt;/code&gt; under &lt;code&gt;cond&lt;/code&gt;) I get an error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;FailedPreconditionError (see above for traceback): Attempting to use&#xA;  uninitialized value fc1/is_hidden      [[Node: fc1/is_hidden/read =&#xA;  IdentityT=DT_BOOL, _class=[&quot;loc:@fc1/is_hidden&quot;],&#xA;  _device=&quot;/job:localhost/replica:0/task:0/cpu:0&quot;]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Exception is thrown here:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with tf.Session() as session:&#xA;  session.run(tf.global_variables_initializer())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I replace &lt;code&gt;hidden&lt;/code&gt; by &lt;code&gt;hidden.initialized_value()&lt;/code&gt; I get the same error for variables in &lt;code&gt;batch_norm&lt;/code&gt; method. I would make the same change there as well, but other side effects take place, e.g. &lt;code&gt;global_step&lt;/code&gt; variable (variable I pass to AdamOptimizer to count steps) is not incremented and remains equal to 0 all the time. This is the thing I do not understand completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a reason behind this?&lt;/p&gt;&#xA;" OwnerUserId="33643" LastEditorUserId="33643" LastEditDate="2017-06-21T09:13:15.753" LastActivityDate="2017-06-21T09:13:15.753" Title="Tensorflow - uninitialized variable under tf.cond(...)" Tags="|machine-learning|neural-network|tensorflow|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19872" PostTypeId="1" CreationDate="2017-06-21T09:06:25.870" Score="3" ViewCount="701" Body="&lt;p&gt;I'd like to predict how many students will enroll in a college in September, based on independent variables which are known earlier in the annual student recruitment cycle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, one independent variable would be the number of accepted offers; I know how many students have accepted an offer from the college for each week from January to August for the last five years, and I know how many of these students actually enrolled in September (generally about 80% of students who accept an offer actually enroll, but this varies year-on-year).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other independent variables might include: how many students have already paid a deposit on their tuition fee, or even the current exchange rate for students travelling from abroad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I've used historical conversion rates to predict enrollment, but I am wondering whether I can do something more sophisticated which would allow me to add in other independent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas of which approaches I should investigate would be very welcome. I'd be particularly interested to hear of any Python libraries which might be relevant.&lt;/p&gt;&#xA;" OwnerUserId="33642" LastEditorUserId="29575" LastEditDate="2017-07-09T00:34:28.273" LastActivityDate="2017-07-09T00:34:28.273" Title="How can I predict student enrolment in September based on independent data available earlier in the year?" Tags="|python|predictive-modeling|time-series|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19873" PostTypeId="2" ParentId="19869" CreationDate="2017-06-21T09:56:47.170" Score="9" Body="&lt;p&gt;The idea behind word2vec is to represent words by a vector of real numbers of dimension &lt;em&gt;d&lt;/em&gt;. Therefore the second matrix is the representation of those words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;i&lt;/em&gt;-th line of this matrix is the vector representation of the &lt;em&gt;i&lt;/em&gt;-th word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that in your example you have 5 words : [&quot;Lion&quot;, &quot;Cat&quot;, &quot;Dog&quot;, &quot;Horse&quot;, &quot;Mouse&quot;], then the first vector [0,0,0,1,0] means you're considering the word &quot;Horse&quot; and so the representation of &quot;Horse&quot; is [10, 12, 19]. Similarly, [17, 24, 1] is the representation of the word &quot;Lion&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my knowledege, there are no &quot;human meaning&quot; specifically to each of the numbers in these representations. One number is not representing if the word is a verb or not, an adjective or not... It's just the weights that you change to solve your optimization problem to learn the representation of your words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tutorial may help : &lt;a href=&quot;http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&quot; rel=&quot;noreferrer&quot;&gt;http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&lt;/a&gt; even though I think the image you put was from this link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also check this, which may help you get started with word vectors with TensorFlow : &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/word2vec&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33598" LastActivityDate="2017-06-21T09:56:47.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19874" PostTypeId="1" CreationDate="2017-06-21T10:11:25.860" Score="8" ViewCount="4720" Body="&lt;p&gt;I have a simple neural network(NN) for MNIST classification.  It includes 2 hidden layers, each with 500 neurons.  Hence the dimensions of the NN are: 784-500-500-10.  ReLU is used in all neurons, softmax is used at the output, and cross-entropy is the loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What puzzles me is why overfitting doesn't appear to devastate the NN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the number of parameters (weights) of the NN.  It's approximately&#xA;$$784\times500+500\times 500+500\times 10=647000.$$&#xA;However, in my experiment, I used only $6000$ examples (a tenth of the MNIST training set) to train the NN. (This is just to keep the run time short.  The training &amp;amp; test error would both go down considerably if I used more training examples.)  I repeated the experiment 10 times.  Plain stochastic gradient descent is used (no RMS prop or momentum); no regularization/drop-out/early-stopping was used.  The training error and test error reported were:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{|l|c|c|c|c|c|c|c|c|c|c|}&#xA;\hline&#xA;\textrm{No.} &amp;amp; 1 &amp;amp; 2 &amp;amp; 3 &amp;amp;4 &amp;amp;5&amp;amp;6&amp;amp;7&amp;amp;8&amp;amp;9&amp;amp;10\\ \hline&#xA;E_{train}(\%) &amp;amp; 7.8 &amp;amp; 10.3 &amp;amp;  9.1 &amp;amp; 11.0 &amp;amp; 8.7 &amp;amp; 9.2 &amp;amp; 9.3 &amp;amp; 8.3 &amp;amp;10.3&amp;amp; 8.6\\ \hline&#xA;E_{test}(\%) &amp;amp; 11.7 &amp;amp; 13.9 &amp;amp; 13.2 &amp;amp; 14.1 &amp;amp;12.1 &amp;amp;13.2 &amp;amp;13.3 &amp;amp;11.9 &amp;amp;13.4&amp;amp;12.7\\ \hline&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that in all 10 experiments (each with independent random parameter initialization), the test error differed from the training error only by approx. 4%, even though I used 6K examples to training 647K parameters.  The VC dimension of the neural network is on the order of $O(|E|log(|E|))$ at least, where $|E|$ is the number of edges (weights).  So why wasn't the test error miserably higher (e.g. 30% or 50%) than the training error?  I'd greatly appreciate it if someone can point out where I missed.  Thanks a lot!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[ EDITS 2017/6/30]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify the effects of early stopping, I did the 10 experiments again, each now with 20 epochs of training.  The error rates are shown in the figure below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\qquad\qquad\qquad$ &lt;a href=&quot;https://i.stack.imgur.com/WkvVr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WkvVr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gap between test and training error did increase as more epochs are used in the training.  However, the tail of the test error stayed nearly flat after the training error is driven to zero.  Moreover, I saw similar trends for other sizes of the training set.  The average error rate at the end of 20 epochs of training is plotted against the size of the training set below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\qquad\qquad\qquad$ &lt;a href=&quot;https://i.stack.imgur.com/rTxW7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rTxW7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So overfitting does occur, but it doesn't appear to devastate the NN.  Considering the number of parameters (647K) we need to the train and the number of training examples we have (&amp;lt;60K), the question remains: why doesn't overfitting easily render the NN useless?  Moreover, is this true for the ReLU NN for all classification tasks with softmax output and cross-entropy objective function?  Have someone seen a counter-example?&lt;/p&gt;&#xA;" OwnerUserId="33377" LastEditorUserId="33377" LastEditDate="2017-06-30T10:28:55.420" LastActivityDate="2017-07-01T15:50:42.683" Title="Why doesn't overfitting devastate neural networks for MNIST classification?" Tags="|machine-learning|neural-network|classification|overfitting|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19875" PostTypeId="2" ParentId="19857" CreationDate="2017-06-21T10:46:18.163" Score="1" Body="&lt;p&gt;iOS, Windows and Android should not be viewed as strings but as &lt;em&gt;categories&lt;/em&gt;. Categories should be one-hot encoded, and should not share the same input as they are not related to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let's assume we work with two operating systems: iOS and Windows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for iOS, input is: &lt;code&gt;1 0&lt;/code&gt; and for Windows input is: &lt;code&gt;0 1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now we want to add another OS: Android.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;iOS becomes: &lt;code&gt;1 0 0&lt;/code&gt;, Windows &lt;code&gt;0 1 0&lt;/code&gt;, Android: &lt;code&gt;0 0 1&lt;/code&gt;. Note that by adding android, you do not actually change the input! You only add an extra 0 for iOS and Windows, and 0's don't get computed in neural networks &lt;code&gt;0*weight = 0&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So by adding an OS, you do not affect what the network has learned for the previously trained OS's.&lt;/p&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-06-21T10:46:18.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19876" PostTypeId="2" ParentId="19870" CreationDate="2017-06-21T11:36:36.840" Score="1" Body="&lt;p&gt;You can treat the mere &lt;em&gt;presence&lt;/em&gt; of any value as signal - hence the 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What help this would be to your project depends on your dataset.&lt;/p&gt;&#xA;" OwnerUserId="33646" LastEditorUserId="33646" LastEditDate="2017-06-22T17:17:50.840" LastActivityDate="2017-06-22T17:17:50.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19878" PostTypeId="2" ParentId="19872" CreationDate="2017-06-21T13:24:05.410" Score="3" Body="&lt;p&gt;I would try out regression in Python's scikit-learn library to predict the September headcount given all those other variables you have. Here is a basic example using a linear model. Once you have that working, you could try a more sophisticated algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-21T13:24:05.410" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19879" PostTypeId="2" ParentId="19858" CreationDate="2017-06-21T14:02:52.480" Score="0" Body="&lt;p&gt;Analytics Vidya is of the best sites that contribute to Machine Learning.&#xA;Provides tutorials , tutorial links and also solutions for challanges&lt;/p&gt;&#xA;" OwnerUserId="33658" LastActivityDate="2017-06-21T14:02:52.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19880" PostTypeId="2" ParentId="17479" CreationDate="2017-06-21T14:59:50.593" Score="1" Body="&lt;p&gt;First add the File widget, then create a xls file with 2 columns:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;name of image&lt;/li&gt;&#xA;&lt;li&gt;image path&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Orange will import the images by itself.&lt;/p&gt;&#xA;" OwnerUserId="33663" LastEditorUserId="26562" LastEditDate="2017-06-24T06:39:37.237" LastActivityDate="2017-06-24T06:39:37.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19881" PostTypeId="2" ParentId="19874" CreationDate="2017-06-21T15:07:09.710" Score="3" Body="&lt;p&gt;In general, people think about overfitting as a function of the model complexity. Which is great, because model complexity is one of the things that you can control. In reality, there are many other factors that are related to the overfitting problem: &#xA;- number of training samples &#xA;- number of iterations &#xA;- the dimension of the input ( in your case , I believe this is the reason why you are not overfitting) &#xA;- the difficulty of the problem:if you have simple problem, linearly separable, then you do not to worry much about overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a visual demo provided by google tensorflow that allows you to change all these parameters. &#xA;&lt;a href=&quot;http://playground.tensorflow.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://playground.tensorflow.org&lt;/a&gt;&#xA;You can change you input problem, the number of samples, the dimension of your input, the network, the number of iterations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to think about overfitting as&#xA;Overfitting = large models + unrelated features &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-06-21T15:07:09.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19882" PostTypeId="1" AcceptedAnswerId="19936" CreationDate="2017-06-21T15:33:34.167" Score="12" ViewCount="40490" Body="&lt;p&gt;How could we get &lt;code&gt;feature_importances&lt;/code&gt; when we are performing regression with &lt;code&gt;XGBRegressor()&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is something like &lt;code&gt;XGBClassifier().feature_importances_&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2019-08-09T09:59:50.960" Title="Xgboost - How to use feature_importances_ with XGBRegressor()?" Tags="|python|regression|xgboost|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19883" PostTypeId="2" ParentId="19874" CreationDate="2017-06-21T15:42:02.207" Score="7" Body="&lt;p&gt;I have replicated your results using Keras, and got very similar numbers so I don't think you are doing anything wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out of interest, I ran for many more epochs to see what would happen. The accuracy of test and train results remained pretty stable. However, the loss values drifted further apart over time. After 10 epochs or so, I was getting 100% train accuracy, 94.3% test accuracy - with loss values around 0.01 and 0.22 respectively. After 20,000 epochs, the accuracies had barely changed, but I had training loss 0.000005 and test loss 0.36. The losses were also still diverging, albeit very slowly. In my opinion, the network is clearly over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question could be re-phrased: Why, despite over-fitting, does a neural network trained to the MNIST data set still generalise apparently reasonably well in terms of accuracy? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth comparing this 94.3% accuracy with what is possible using more naive approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, a simple linear softmax regression (essentially the same neural network without the hidden layers), gives a quick stable accuracy of 95.1% train, and 90.7% test. This shows that a lot of the data separates linearly - you can draw hyperplanes in the 784 dimensions and 90% of the digit images will sit inside the correct &quot;box&quot; with no further refinement required. From this, you might expect an overfit non-linear solution to get a worse result than 90%, but maybe no worse than 80% because intuitively forming an over-complex boundary around e.g. a &quot;5&quot; found inside the box for &quot;3&quot; will only incorrectly assign a small amount of this naive 3 manifold. But we're better than this 80% lower bound guesstimate from the linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible naive model is template matching, or nearest-neighbour. This is a reasonable analogy to what the over-fitting is doing - it creates a local area close to each training example where it will predict the same class. Problems with over-fitting occur in the space in-between where the values of activation will follow whatever the network &quot;naturally&quot; does. Note the worst case, and what you often see in explanatory diagrams, would be some highly curved almost-chaotic surface which travels through other classifications. But actually it may be more natural for the neural network to more smoothly interpolate between points - what it actually does depends on the nature of the higher order curves that the network combines into approximations, and how well those already fit to the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I borrowed the code for a KNN solution from &lt;a href=&quot;https://medium.com/towards-data-science/mnist-with-k-nearest-neighbors-8f6e7003fab7&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog on MNIST with K Nearest Neighbours&lt;/a&gt;. Using k=1 - i.e. choosing the label of the nearest from the 6000 training examples just by matching pixel values, gives an accuracy of 91%. The 3% extra that the over-trained neural network achieves does not seem quite so impressive given the simplicity of pixel-match counting that KNN with k=1 is doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried a few variations of network architecture, different activation functions, different number and sizes of layers - none using regularisation. However, with 6000 training examples, I could not get any of them to overfit in a way where test accuracy dropped dramatically. Even reducing to just 600 training examples just made the plateau lower, at ~86% accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My basic conclusion is that MNIST examples have relatively smooth transitions between classes in feature space, and that neural networks can fit to these and interpolate between the classes in a &quot;natural&quot; manner given NN building blocks for function approximation - without adding high frequency components to the approximation that could cause issues in an overfit scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be an interesting experiment to try with a &quot;noisy MNIST&quot; set where an amount of random noise or distortion is added to both training and test examples. Regularized models would be expected to perform OK on this dataset, but perhaps in that scenario the over-fitting would cause more obvious problems with accuracy.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;This is from before the update with further tests by OP.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your comments, you say that your test results are all taken after running a single epoch. You have essentially used early stopping, despite writing that you have not, because you have stopped the training at the earliest possible point given your training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest running for many more epochs if you want to see how the network is truly converging. Start with 10 epochs, consider going up to 100. One epoch is not many for this problem, especially on 6000 samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although increasing number of iterations is not guaranteed to make your network overfit worse than it already has, you haven't really given it much of a chance, and your experimental results so far are not conclusive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact I would half expect your test data results to &lt;em&gt;improve&lt;/em&gt; following a 2nd, 3rd epoch, before starting to fall away from the training metrics as the epoch numbers increase. I would also expect your training error to approach 0% as the network approached convergence.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-01T15:50:42.683" LastActivityDate="2017-07-01T15:50:42.683" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19888" PostTypeId="1" AcceptedAnswerId="19890" CreationDate="2017-06-21T16:48:25.713" Score="5" ViewCount="3915" Body="&lt;p&gt;How to transform raw data to fixed-frequency time series?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I have the following raw data in DataFrame&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                       A    B&#xA;2017-01-01 00:01:01    0    100&#xA;2017-01-01 00:01:10    1    200&#xA;2017-01-01 00:01:16    2    300&#xA;2017-01-01 00:02:35    3    100&#xA;2017-01-01 00:02:40    4    100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to transform it into a time series:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;1 minute frequency&lt;/li&gt;&#xA;&lt;li&gt;column A should have sum of values in time interval&lt;/li&gt;&#xA;&lt;li&gt;column B should have mean of values in time interval&lt;/li&gt;&#xA;&lt;li&gt;possibly other functions over other columns&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: Raw data is not periodic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transformed data should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                       A    B&#xA;2017-01-01 00:01:00    3    200&#xA;2017-01-01 00:02:00    7    100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33668" LastEditorUserId="29575" LastEditDate="2017-06-22T06:14:30.927" LastActivityDate="2017-06-22T06:14:30.927" Title="How to transform raw data to fixed-frequency time series?" Tags="|time-series|pandas|feature-extraction|feature-construction|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19889" PostTypeId="1" CreationDate="2017-06-21T18:01:47.127" Score="0" ViewCount="811" Body="&lt;p&gt;Hello data scientists (I'm not),&#xA;I adapted a multi-class-text-classification-cnn from &lt;a href=&quot;https://github.com/jiegzhan/multi-class-text-classification-cnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jiegzhan/multi-class-text-classification-cnn&lt;/a&gt; (thanks Jie Zhang)&#xA;It works ! 90% accuracy !&#xA;Now my issue ;-)&#xA;I really need to load my model with java in order to allow my existing java service to predict categories.&#xA;According to documentations, I need to use SavedModelBuilder which is the new way of modele saving. I tried for hours but I misse a lot of things... About signature, assets. &#xA;If someone could help me with that ? =&gt; replace the old saver.save by tf.saved_model.builder.SavedModelBuilder into &lt;a href=&quot;https://github.com/jiegzhan/multi-class-text-classification-cnn/blob/master/train.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jiegzhan/multi-class-text-classification-cnn/blob/master/train.py&lt;/a&gt;&#xA;He or she will have my infinite gratitude&lt;/p&gt;&#xA;" OwnerUserId="33672" LastActivityDate="2018-04-26T12:58:10.390" Title="tensorflow saved model to java" Tags="|tensorflow|java|classifier|" AnswerCount="1" CommentCount="1" ClosedDate="2020-06-16T08:40:05.070" ContentLicense="CC BY-SA 3.0" />
  <row Id="19890" PostTypeId="2" ParentId="19888" CreationDate="2017-06-21T18:01:58.910" Score="5" Body="&lt;p&gt;This sort of effect can be achieved with &#xA;&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;pandas.DataFrame.resample()&lt;/code&gt;&lt;/a&gt; combined with &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.resample.Resampler.aggregate.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;Resampler.aggregate()&lt;/code&gt;&lt;/a&gt; like:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.resample(&quot;1Min&quot;).agg({'A': sum, 'B': np.mean})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_fwf(StringIO(u&quot;&quot;&quot;&#xA;                           A    B&#xA;    2017-01-01T00:01:01    0    100&#xA;    2017-01-01T00:01:10    1    200&#xA;    2017-01-01T00:01:16    2    300&#xA;    2017-01-01T00:02:35    3    100&#xA;    2017-01-01T00:02:40    4    100&quot;&quot;&quot;),&#xA;                  header=1, parse_dates=[0], index_col=0)&#xA;&#xA;print(df)    &#xA;print(df.resample(&quot;1Min&quot;).agg({'A': sum, 'B': np.mean}))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                     A    B&#xA;2017-01-01 00:01:01  0  100&#xA;2017-01-01 00:01:10  1  200&#xA;2017-01-01 00:01:16  2  300&#xA;2017-01-01 00:02:35  3  100&#xA;2017-01-01 00:02:40  4  100&#xA;&#xA;                     A    B&#xA;2017-01-01 00:01:00  3  200&#xA;2017-01-01 00:02:00  7  100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-06-21T18:01:58.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19892" PostTypeId="1" AcceptedAnswerId="19916" CreationDate="2017-06-21T19:50:20.193" Score="1" ViewCount="2436" Body="&lt;p&gt;If you run the example of the LSTM sentiment classification task example in keras&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and add&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p = model.predict(x_test[0])&#xA;print(p.shape)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you get:&#xA;&lt;code&gt;(80,1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.. why? I thought the point of the model was to classify each sample as 0 or 1 as per each of the y_train elements (binary classification) &lt;/p&gt;&#xA;" OwnerUserId="31457" LastActivityDate="2017-06-25T20:19:45.280" Title="why is the model prediction output in keras lstm imdb, a vector?" Tags="|neural-network|tensorflow|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19893" PostTypeId="1" CreationDate="2017-06-21T19:58:53.100" Score="2" ViewCount="1372" Body="&lt;p&gt;I am trying to model a binary outcome in R that has many independent variables. 5 of the Ivs are factors with more than two levels. When I try to remove the intercept it only does it for one of the independent variables. I would like to see if any of the coefficients for the levels are significantly different from any of the others within that variable(and it would be nice to see if any of the levels are significantly different from the average). When I do a summary of the glm it only shows the values of the coefficient as it relates to the first level(Except for the first IV). &#xA;To recreate it I have the R code for something similar using mtcars:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;md &amp;lt;- datasets::mtcars&#xA;md &amp;lt;- mutate(md, ncyl = ifelse(cyl==6,&quot;Six Cyl&quot;, cyl))&#xA;md &amp;lt;- mutate(md, ncyl = ifelse(cyl==4,&quot;Four Cyl&quot;,ncyl))&#xA;md &amp;lt;- mutate(md, ncyl = ifelse(cyl==8,&quot;Eight Cyl&quot;,ncyl))&#xA;md &amp;lt;- mutate(md, wgear = ifelse(gear==3,&quot;Three G&quot;,gear))&#xA;md &amp;lt;- mutate(md, wgear = ifelse(wgear==4,&quot;Four G&quot;,wgear))&#xA;md &amp;lt;- mutate(md, wgear = ifelse(wgear==5,&quot;Five G&quot;,wgear))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==1,&quot;One Carb&quot;,carb))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==2,&quot;Two Carb&quot;,wcarb))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==3,&quot;Three Carb&quot;,wcarb))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==4,&quot;Four Carb&quot;,wcarb))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==6,&quot;Six Carb&quot;,wcarb))&#xA;md &amp;lt;- mutate(md, wcarb = ifelse(carb==8,&quot;Eight Carb&quot;,wcarb))&#xA;md &amp;lt;- md[,-c(2,10,11)]&#xA;model &amp;lt;- glm(mpg~., data = md)&#xA;summary(model)&#xA;modelint &amp;lt;- glm(mpg~.-1, data = md)&#xA;summary(modelint)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So for &lt;code&gt;summary(modelint)&lt;/code&gt; I can see the values for the coefficients of all 3 possible values ncyl, and with &lt;code&gt;confint(modelint)&lt;/code&gt; I can at least see confidence intervals for all of the possible values. But wgear and wcarb are just compared to their first level alphabetically and the t values are just if it is different from the first level. I know if I just did &lt;code&gt;mpg~wgear-1&lt;/code&gt; I could see their comparisons, but I think this does not take into account all of the other variables. Ideally I would like to see how 1 carb compares to two carb, three, four,...  and 1 carb vs not 1 carb while taking into account the other variables. Is there a better function to use than glm? Any advice would be greatly appreciated!&lt;/p&gt;&#xA;" OwnerUserId="33677" LastEditorUserId="381" LastEditDate="2017-06-21T20:23:42.693" LastActivityDate="2017-06-22T23:10:16.067" Title="How do I compare coefficients from my glm when I have more than one factor variable in my formula?" Tags="|r|regression|glm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19894" PostTypeId="1" CreationDate="2017-06-21T20:16:45.803" Score="3" ViewCount="1334" Body="&lt;p&gt;This question is about detection of a number having multiple digits in a single image.&lt;/p&gt;&#xA;&lt;p&gt;I have a trained model in tensorflow using Deep CNN for image recognition. the training was done on cropped images which only had a digit and its label.&lt;/p&gt;&#xA;&lt;p&gt;Now, the task is to identify any number from the image using this trained model.&lt;/p&gt;&#xA;&lt;p&gt;I would like this model to work on arbitrary sized number. so how do i extract certain features from the image to be predicted so that individual digit can be detected.&lt;/p&gt;&#xA;&lt;p&gt;For reference &lt;a href=&quot;http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; is the link to the google research paper though it does not elaborate on the issue&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://ufldl.stanford.edu/housenumbers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is the data&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/11091/how-does-deep-learning-helps-in-detecting-multiple-objects-in-single-image&quot;&gt;this question&lt;/a&gt; is similar but it does not answer fully as digits nay be too small and close to each other.&lt;/p&gt;&#xA;" OwnerUserId="32667" LastEditorUserId="29169" LastEditDate="2021-02-16T00:46:08.193" LastActivityDate="2021-02-16T00:46:08.193" Title="multiple digit detection" Tags="|deep-learning|tensorflow|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19895" PostTypeId="1" CreationDate="2017-06-21T20:19:26.293" Score="1" ViewCount="676" Body="&lt;p&gt;If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional unsupervised LDA didn't help much.&lt;/p&gt;&#xA;" OwnerUserId="33680" LastActivityDate="2019-12-18T15:41:29.117" Title="News topic detection and categorization" Tags="|machine-learning|python|unsupervised-learning|topic-model|lda|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19896" PostTypeId="1" CreationDate="2017-06-21T21:10:08.353" Score="4" ViewCount="5511" Body="&lt;p&gt;I am testing the machine learning waters and used &lt;a href=&quot;https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#4&quot; rel=&quot;nofollow noreferrer&quot;&gt;TS inception&lt;/a&gt; model to retrain the network to classify my desired objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially, my predictions were run on locally stored images and I realized that it took anywhere between 2-5 seconds to unpersist the graph from a file and around the same time to run the actual predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thereafter, I adapted my code to incorporate the camera feed from OpenCV but with the above noted times, video lags are inevitable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A time hit was expected during initial graph load; which is why &lt;code&gt;initialSetup()&lt;/code&gt; is ran beforehand, but 2-5 seconds is just absurd.&#xA;I feel like with my current application; real-time classification, this is not the best way of loading it. Is there another way of doing this? I know with mobile versions TS recommends trimming down the graph. Would slimming it down be the way to go here? In case it matters my graph is currently 87.4MB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along with this, is there a way of speeding up the prediction process?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;import cv2&#xA;import timeit&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;&#xA;camera = cv2.VideoCapture(0)&#xA;&#xA;# Loads label file, strips off carriage return&#xA;label_lines = [line.rstrip() for line&#xA;               in tf.gfile.GFile('retrained_labels.txt')]&#xA;&#xA;def grabVideoFeed():&#xA;    grabbed, frame = camera.read()&#xA;    return frame if grabbed else None&#xA;&#xA;def initialSetup():&#xA;    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'&#xA;    start_time = timeit.default_timer()&#xA;&#xA;    # This takes 2-5 seconds to run&#xA;    # Unpersists graph from file&#xA;    with tf.gfile.FastGFile('retrained_graph.pb', 'rb') as f:&#xA;        graph_def = tf.GraphDef()&#xA;        graph_def.ParseFromString(f.read())&#xA;        tf.import_graph_def(graph_def, name='')&#xA;&#xA;    print 'Took {} seconds to unpersist the graph'.format(timeit.default_timer() - start_time)&#xA;&#xA;def classify(image_data):&#xA;    print '********* Session Start *********'&#xA;&#xA;    with tf.Session() as sess:&#xA;        start_time = timeit.default_timer()&#xA;&#xA;        # Feed the image_data as input to the graph and get first prediction&#xA;        softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')&#xA;&#xA;        print 'Tensor', softmax_tensor&#xA;&#xA;        print 'Took {} seconds to feed data to graph'.format(timeit.default_timer() - start_time)&#xA;&#xA;        start_time = timeit.default_timer()&#xA;&#xA;        # This takes 2-5 seconds as well&#xA;        predictions = sess.run(softmax_tensor, {'Mul:0': image_data})&#xA;&#xA;        print 'Took {} seconds to perform prediction'.format(timeit.default_timer() - start_time)&#xA;&#xA;        start_time = timeit.default_timer()&#xA;&#xA;        # Sort to show labels of first prediction in order of confidence&#xA;        top_k = predictions[0].argsort()[-len(predictions[0]):][::-1]&#xA;&#xA;        print 'Took {} seconds to sort the predictions'.format(timeit.default_timer() - start_time)&#xA;&#xA;        for node_id in top_k:&#xA;            human_string = label_lines[node_id]&#xA;            score = predictions[0][node_id]&#xA;            print('%s (score = %.5f)' % (human_string, score))&#xA;&#xA;        print '********* Session Ended *********'&#xA;&#xA;initialSetup()&#xA;&#xA;while True:&#xA;    frame = grabVideoFeed()&#xA;&#xA;    if frame is None:&#xA;        raise SystemError('Issue grabbing the frame')&#xA;&#xA;    frame = cv2.resize(frame, (299, 299), interpolation=cv2.INTER_CUBIC)&#xA;&#xA;    # adhere to TS graph input structure&#xA;    numpy_frame = np.asarray(frame)&#xA;    numpy_frame = cv2.normalize(numpy_frame.astype('float'), None, -0.5, .5, cv2.NORM_MINMAX)&#xA;    numpy_final = np.expand_dims(numpy_frame, axis=0)&#xA;&#xA;    classify(numpy_final)&#xA;&#xA;    cv2.imshow('Main', frame)&#xA;&#xA;    if cv2.waitKey(1) &amp;amp; 0xFF == ord('q'):&#xA;        break&#xA;&#xA;camera.release()&#xA;cv2.destroyAllWindows()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After debugging my code, I realized that session creation is a both resource and time consuming operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the prior code, a new session was created for each OpenCV feed on top of running the predictions.&#xA;Wrapping the OpenCV operations inside a single session provides a massive time improvement but this still adds a massive overhead on the initial run; prediction takes 2-3 seconds. Afterwards, the prediction takes around 0.5s which makes the camera feed still laggy.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;import cv2&#xA;import timeit&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;&#xA;camera = cv2.VideoCapture(0)&#xA;&#xA;# Loads label file, strips off carriage return&#xA;label_lines = [line.rstrip() for line&#xA;               in tf.gfile.GFile('retrained_labels.txt')]&#xA;&#xA;def grabVideoFeed():&#xA;    grabbed, frame = camera.read()&#xA;    return frame if grabbed else None&#xA;&#xA;def initialSetup():&#xA;    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'&#xA;    start_time = timeit.default_timer()&#xA;&#xA;    # This takes 2-5 seconds to run&#xA;    # Unpersists graph from file&#xA;    with tf.gfile.FastGFile('retrained_graph.pb', 'rb') as f:&#xA;        graph_def = tf.GraphDef()&#xA;        graph_def.ParseFromString(f.read())&#xA;        tf.import_graph_def(graph_def, name='')&#xA;&#xA;    print 'Took {} seconds to unpersist the graph'.format(timeit.default_timer() - start_time)&#xA;&#xA;initialSetup()&#xA;&#xA;with tf.Session() as sess:&#xA;    start_time = timeit.default_timer()&#xA;&#xA;    # Feed the image_data as input to the graph and get first prediction&#xA;    softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')&#xA;&#xA;    print 'Took {} seconds to feed data to graph'.format(timeit.default_timer() - start_time)&#xA;&#xA;    while True:&#xA;        frame = grabVideoFeed()&#xA;&#xA;        if frame is None:&#xA;            raise SystemError('Issue grabbing the frame')&#xA;&#xA;        frame = cv2.resize(frame, (299, 299), interpolation=cv2.INTER_CUBIC)&#xA;&#xA;        cv2.imshow('Main', frame)&#xA;&#xA;        # adhere to TS graph input structure&#xA;        numpy_frame = np.asarray(frame)&#xA;        numpy_frame = cv2.normalize(numpy_frame.astype('float'), None, -0.5, .5, cv2.NORM_MINMAX)&#xA;        numpy_final = np.expand_dims(numpy_frame, axis=0)&#xA;&#xA;        start_time = timeit.default_timer()&#xA;&#xA;        # This takes 2-5 seconds as well&#xA;        predictions = sess.run(softmax_tensor, {'Mul:0': numpy_final})&#xA;&#xA;        print 'Took {} seconds to perform prediction'.format(timeit.default_timer() - start_time)&#xA;&#xA;        start_time = timeit.default_timer()&#xA;&#xA;        # Sort to show labels of first prediction in order of confidence&#xA;        top_k = predictions[0].argsort()[-len(predictions[0]):][::-1]&#xA;&#xA;        print 'Took {} seconds to sort the predictions'.format(timeit.default_timer() - start_time)&#xA;&#xA;        for node_id in top_k:&#xA;            human_string = label_lines[node_id]&#xA;            score = predictions[0][node_id]&#xA;            print('%s (score = %.5f)' % (human_string, score))&#xA;&#xA;        print '********* Session Ended *********'&#xA;&#xA;        if cv2.waitKey(1) &amp;amp; 0xFF == ord('q'):&#xA;            sess.close()&#xA;            break&#xA;&#xA;camera.release()&#xA;cv2.destroyAllWindows()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After fiddling around, I stumbled into &lt;a href=&quot;https://www.tensorflow.org/performance/quantization&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph quantization&lt;/a&gt; and &lt;a href=&quot;https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph transformation&lt;/a&gt; and these were the attained results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Original Graph: 87.4MB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quantized Graph: 87.5MB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transformed Graph: 87.1MB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eight Bit Calculation: 22MB but ran into &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/10739&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; upon use.&lt;/p&gt;&#xA;" OwnerUserId="33685" LastEditorUserId="33685" LastEditDate="2017-06-27T16:03:58.287" LastActivityDate="2017-06-27T16:03:58.287" Title="Tensorflow and OpenCV real-time classification" Tags="|python|classification|tensorflow|inception|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19897" PostTypeId="1" AcceptedAnswerId="19899" CreationDate="2017-06-21T21:26:09.167" Score="2" ViewCount="707" Body="&lt;p&gt;I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my &quot;career&quot; in Data Science? What's the best programming language for processing data?&#xA;P.S. I love statistics and programming so I think this will be fun.&lt;/p&gt;&#xA;" OwnerUserId="33686" LastActivityDate="2017-06-22T01:52:24.860" Title="Best Programming Language for Data Science" Tags="|data|programming|" AnswerCount="2" CommentCount="1" ClosedDate="2017-06-24T09:10:23.170" ContentLicense="CC BY-SA 3.0" />
  <row Id="19898" PostTypeId="2" ParentId="15328" CreationDate="2017-06-21T22:40:31.917" Score="3" Body="&lt;p&gt;beside what was mentioned by daoliker&lt;/p&gt;&#xA;&#xA;&lt;p&gt;inception v2 utilized separable convolution as first layer of depth 64&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tensorflow/models/blob/master/slim/nets/inception_v2.py#L93&quot; rel=&quot;nofollow noreferrer&quot;&gt;function usage&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/layers/python/layers/layers.py#L1947&quot; rel=&quot;nofollow noreferrer&quot;&gt;function definition&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1502.03167&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;quote from paper&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Our model employed &lt;em&gt;separable convolution&lt;/em&gt; with depth&#xA;  multiplier 8 on the first convolutional layer. This reduces&#xA;  the computational cost while increasing the memory consumption&#xA;  at training time.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;why this is important? because it was dropped in v3 and v4 and inception resnet, but re-introduced and heavily used in &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/slim/nets/mobilenet_v1.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;mobilenet&lt;/a&gt; later.&lt;/p&gt;&#xA;" OwnerUserId="33689" LastActivityDate="2017-06-21T22:40:31.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19899" PostTypeId="2" ParentId="19897" CreationDate="2017-06-21T23:04:31.187" Score="2" Body="&lt;p&gt;This is no doubt a duplicate, but here's how I'd weigh in on the major languages:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;R:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fantastic support for packages and specialised stats analysis&#xA;community, you can find a package to do just about anything you need&#xA;and it will be relatively easy to use.&lt;/li&gt;&#xA;&lt;li&gt;Is good for throwing together prototypes and performing exploratory analysis.&lt;/li&gt;&#xA;&lt;li&gt;Is Free and Open Source.&lt;/li&gt;&#xA;&lt;li&gt;Slower than Python. Basically don't loop over anything. It's an odd language for a programmer to use (coming from a software dev&#xA;background). Clearly designed by mathematicians.&lt;/li&gt;&#xA;&lt;li&gt;Relatively little choice of good IDEs&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Python:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fast.&lt;/li&gt;&#xA;&lt;li&gt;Also very good as a general purpose language so has 'broader' package support. Free and Open Source.&lt;/li&gt;&#xA;&lt;li&gt;Easy to use for Big Data applications. &lt;/li&gt;&#xA;&lt;li&gt;Not as streamlined for analysis as R.&lt;/li&gt;&#xA;&lt;li&gt;Syntax can be difficult to read (no surrounding braces to make it obvious where functions/ if statements end).&lt;/li&gt;&#xA;&lt;li&gt;Can be particularly tedious working with Dataframes compared to R.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;MATLAB:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generally slower.&lt;/li&gt;&#xA;&lt;li&gt;Has very impressive packages for signal processing/image recognition and all the cool stuff.&lt;/li&gt;&#xA;&lt;li&gt;Is very readable and easy to comprehend generally.&lt;/li&gt;&#xA;&lt;li&gt;Is NOT free. Student licenses are available. Was quite complicated for me to get my hands on one though...&lt;/li&gt;&#xA;&lt;li&gt;Has very good support for mathematical analysis similar to R, but much better matrix functions.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Personal recommendation: Python. Kill two birds with one stone, learn good general to advanced programming concepts and data science at the same time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good article: &lt;a href=&quot;https://www.linkedin.com/pulse/r-vs-python-matlab-octave-julia-who-winner-siva-prasad-katru&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.linkedin.com/pulse/r-vs-python-matlab-octave-julia-who-winner-siva-prasad-katru&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33603" LastEditorUserId="33603" LastEditDate="2017-06-21T23:26:20.833" LastActivityDate="2017-06-21T23:26:20.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19900" PostTypeId="2" ParentId="19893" CreationDate="2017-06-22T01:18:45.670" Score="2" Body="&lt;p&gt;R chooses the baseline level by itself unless you specify it. You can read here how to specify it: &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/3872070/how-to-force-r-to-use-a-specified-factor-level-as-reference-in-a-regression&quot;&gt;https://stackoverflow.com/questions/3872070/how-to-force-r-to-use-a-specified-factor-level-as-reference-in-a-regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can change the reference (baseline) level to get all the possible results. This is easy to do but I am not sure if it is the best way to go, since you need to repeat yourself many times. But I think this might be better if you could apply it on your own data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/60352/comparing-levels-of-factors-after-a-glm-in-r&quot;&gt;https://stats.stackexchange.com/questions/60352/comparing-levels-of-factors-after-a-glm-in-r&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the link to the R package documentation: &#xA;&lt;a href=&quot;https://cran.r-project.org/web/packages/multcomp/multcomp.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/multcomp/multcomp.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30381" LastEditorUserId="30381" LastEditDate="2017-06-22T01:26:18.717" LastActivityDate="2017-06-22T01:26:18.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19901" PostTypeId="1" AcceptedAnswerId="19931" CreationDate="2017-06-22T01:43:13.350" Score="7" ViewCount="9882" Body="&lt;p&gt;I'm using some ML algorithms (from sklearn lib) and on most of them there is a parameter n_estimators which is (if I understood well) the number of used trees. Intuitevely I would say that the more trees you use, the more accurate results you get. It turned out to be not exactly true, sometimes, a very few number of trees give much better results, but I can't figure out why ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some precisions: this is a regression problem, with a dataset containing &#xA;about 4000 samples and 500 features. I'm using GradientBoostingRegressor, ExtraTreeRegressor, AdaBoost, RandomForest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional info: I use a cross-validation with KFold=10 to evaluate the accuracy of the algorithms. The best n_estimators value seems to be 50, which give a R2 score of ~56/57% +- 8% for all above cited algo. When I try to increase it, the score quickly decreases. I tried several values, from 100 to 500, it keeps decreasing even reaching 52%. &lt;/p&gt;&#xA;" OwnerUserId="33386" LastEditorUserId="33386" LastEditDate="2017-06-23T07:07:54.737" LastActivityDate="2017-06-23T07:58:48.440" Title="Does increasing the n_estimators parameter in decision trees always increase accuracy" Tags="|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19902" PostTypeId="2" ParentId="19897" CreationDate="2017-06-22T01:52:24.860" Score="0" Body="&lt;p&gt;Python is a great choice if u are from programming developer&lt;br&gt;&#xA;This article will be good if u want more details&lt;br&gt;&#xA;&lt;a href=&quot;http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html&lt;/a&gt;&lt;br&gt;&#xA;Also u've mentioned to recommend u to  start your career in data science.I think &lt;code&gt;Kaggle.com&lt;/code&gt; will be the best choice for the beginners.It will help u to know whether this field suits u or not.&lt;br&gt;&#xA;And for blog i think &lt;code&gt;kdnuggets.com&lt;/code&gt;will be a good choice.&lt;br&gt;&#xA;Hope this answer is helpful..&lt;/p&gt;&#xA;" OwnerUserId="33693" LastActivityDate="2017-06-22T01:52:24.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19905" PostTypeId="2" ParentId="16948" CreationDate="2017-06-22T05:59:26.503" Score="1" Body="&lt;p&gt;Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nb_classes = 3 # the number of categories you have&#xA;x_train = []&#xA;y_train = []&#xA;&#xA;with open('train.csv', 'r') as train_file:&#xA;    reader = csv.reader(train_file)&#xA;    for row in reader:&#xA;       sentence = row[0]&#xA;       category = row[1]&#xA;&#xA;       x_train.append(sentence)&#xA;       y_train.append(category)&#xA;&#xA;Y_train = np_utils.to_categorical(y_train, nb_classes)&#xA;X_train = ? # your choice of tokanization&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You should do the same with your test dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should also change the loss to &lt;code&gt;categorical_crossentropy&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30519" LastEditorUserId="30519" LastEditDate="2017-12-17T11:16:41.767" LastActivityDate="2017-12-17T11:16:41.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19908" PostTypeId="1" CreationDate="2017-06-22T07:49:38.733" Score="-1" ViewCount="515" Body="&lt;p&gt;I try to construct a classic querying system where I find the most probable candidate text for a query by computing cosine similarities of TFIDF vectors of normalized text of possible answers. This works quite well if the query and the candidate texts contain identical (normalized) words. So, the following question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;When was the King of England assassinated?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my system correctly finds the following closest answer from the corpus:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The King of England was assassinated in 1234.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far, so good. My problem is that I want to find this answer in the case of questions with the same meaning but slightly different synonym wording, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;When was the British Monarch killed?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How could I compute similarity taking semantic distance into account? Shall I use word2vec representation instead of TFIDF?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2017-08-21T22:46:31.870" Title="How to improve Vector Space Models with semantic similarity?" Tags="|nlp|word2vec|vector-space-models|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19909" PostTypeId="2" ParentId="19908" CreationDate="2017-06-22T08:30:43.880" Score="0" Body="&lt;p&gt;You are right. TF-IDF is not suitable for your problem statement because TF-IDF tends to assign lower weight to common words appearing in two documents while giving importance to important words. So while &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;When was the King of England assassinated?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;The King of England was assassinated in 1234.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will match because of words like King and England but &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;When was the British Monarch killed?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;wont match as the words British and Monarch are one of its kind in the corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach is doc2vec technique which converts documents into vectors and predicts cosine similarity. It works very well in a very large number of documents or corpus, containing tens of thousands of documents. Doc2Vec is a generalization of Word2Vec which can give very interesting relationship between words. For example, &lt;code&gt;king - man + woman = queen&lt;/code&gt; without being implicitly or explicitly told.&lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-06-22T08:30:43.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19910" PostTypeId="2" ParentId="19858" CreationDate="2017-06-22T08:51:40.577" Score="0" Body="&lt;p&gt;Apart from Andrew Ng's Machine Learning course and Datacamp I will also add the 10 course specialization - Data Science Specialization by John Hopkins University offered on Coursera. It covers pretty much everything from importing and cleansing of data, to the application of Machine Learning algorithms and showcasing your results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your project, the first step is ensure what you want to do, what is the problem statement or the right question. Second step, get the data. Stock market data is mostly open for use. Third step, feature selection and engineering which is essentially selecting or building features which will help in determining relationship and finding structure/pattern. There are many machine learning algorithms which you can use. Time series analysis is used mostly for this problem domain but since it is little complex you can try out different classification techniques like kNN, Naive Bayes, SVM to determine if the stock will be good or bad. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose a data science tool where you can implement your algorithm. It could be either R, Python, SAS, MATLAB to name a few. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the best. &lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-06-22T08:51:40.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19911" PostTypeId="1" CreationDate="2017-06-22T10:09:46.247" Score="2" ViewCount="196" Body="&lt;p&gt;Correct me if I am wrong, but while business analysts are often an important role in software delivery teams, but it seems not the case in the data science field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Commonly available data scientist skill sets state also importance of story-telling skills and understanding customer's business, but is that enough?&lt;/p&gt;&#xA;" OwnerUserId="33183" LastActivityDate="2017-06-22T17:18:40.200" Title="What could be the value of having a business analyst in a data science team?" Tags="|beginner|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19912" PostTypeId="2" ParentId="19911" CreationDate="2017-06-22T11:16:04.467" Score="2" Body="&lt;p&gt;I'm not a business analyst so I guess you'll have to take what I have to say with a pinch of salt. From my understanding, Business Analysts responsibilities are focused around improving processes within a company, for example how certain technologies could be implemented to improve a workflow, they are expected to understand how these technologies might improve the workflow, or a product etc, and manage these improvement projects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems to differ from Data Science on an abstract level in that it is exploring known unknowns (&quot;can our process be improved? what technologies/methods exist that could improve it?&quot;), whereas Data Science is great for exploring unknown unknowns. For example, why is this better for &lt;em&gt;our&lt;/em&gt; workflow/product specifically? Data Science is great at throwing up results you don't expect, which is one reason why it is so valuable. I may be wrong in saying this, but Business Analysis seems to be relatively free-form depending on the company and the needs, whereas Data Science has a less subjective methodology.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this in mind, perhaps Data Scientists could be used to better inform your Business Analysts decisions? But the other way around, BAs could perhaps be used to better inform Data Scientists of business processes, or maybe your BA could focus on improving processes to make life easier for your Data Scientists such as data pipelines in a non-automated environment (example: how can we gather more useful data from our vehicle showrooms? What technologies do the DSs need and how can we implement them?).&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-06-22T11:16:04.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19914" PostTypeId="1" CreationDate="2017-06-22T12:15:25.957" Score="1" ViewCount="319" Body="&lt;p&gt;I am using spark ml classifier to create a model to predict whether a user will buy a product or not and I am using features both continuous and categorical. The model is predicting the outcome with some accuracy. Is there a way to see how much each feature is contributing to the prediction (i.e) the weight of each feature in the model created from training data? Because I need to know why a user will buy a product or not. I am using spark 2.0.0 and all the classifiers available in it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in Advance&lt;/p&gt;&#xA;" OwnerUserId="33708" LastActivityDate="2017-06-22T12:15:25.957" Title="Is there a way to find the weights of every feature in spark ml model?" Tags="|classification|logistic-regression|apache-spark|prediction|naive-bayes-classifier|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19915" PostTypeId="2" ParentId="14142" CreationDate="2017-06-22T12:56:37.737" Score="1" Body="&lt;p&gt;Almost correct, the only problem is that you are using np.random to randomly generate points in feature space. This means that you assume that the points will be in the range of [0,1]. I've tweaked your code to check the min/max range of input data and adjust accordingly. The you can find the updated version &lt;a href=&quot;https://matevzkunaver.wordpress.com/2017/06/20/hopkins-test-for-cluster-tendency/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; (with link to this post of course)&lt;/p&gt;&#xA;" OwnerUserId="33713" LastActivityDate="2017-06-22T12:56:37.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19916" PostTypeId="2" ParentId="19892" CreationDate="2017-06-22T14:39:50.867" Score="2" Body="&lt;p&gt;&lt;code&gt;x_test&lt;/code&gt; is of shape &lt;code&gt;(25000, 80)&lt;/code&gt;, that means 25000 samples with each sample corresponding to a sentence with a maximum of 80 tokens (see &lt;code&gt;maxlen&lt;/code&gt;).&#xA;Accordingly &lt;code&gt;x_test[0]&lt;/code&gt; is a vector of size 80.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.predict&lt;/code&gt; does work on batches of samples, so you can call all &lt;code&gt;model.predict(x_test)&lt;/code&gt; you will get a vector of &lt;code&gt;n&lt;/code&gt; predictions where n is the number of test samples, i.e. 25000 in that case.&#xA;You could also predict the outcomes for just a part of the test set with slicing, e.g. &lt;code&gt;model.predict(x_test[:10])&lt;/code&gt; for the 10 first samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens when you are calling this function on just one sample &lt;code&gt;x_test[0]&lt;/code&gt; is that the input is interpreted as a batch of 80 samples each with just one token.&#xA;(To the best of my knowledge, this &quot;interpretation&quot; happens somewhere down the function tree in &lt;code&gt;keras.engine.training._standardize_inpupt_data&lt;/code&gt;.)&#xA;The model &quot;doesn't know&quot; the length of the input samples beforehand and thus doen't complain about these very short samples.&#xA;So you get 80 predictions for all these samples as if they were all sentences with just one token.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a prediction for the first sample, you could use &lt;code&gt;model.predict(x_test[:1])&lt;/code&gt;. So you predict a batch containing only one sample and get the corresponding batch of one prediction.&lt;/p&gt;&#xA;" OwnerUserId="25454" LastEditorUserId="25454" LastEditDate="2017-06-25T20:19:45.280" LastActivityDate="2017-06-25T20:19:45.280" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19917" PostTypeId="2" ParentId="19911" CreationDate="2017-06-22T15:37:05.467" Score="3" Body="&lt;p&gt;I believe the integration of Business Analyst(s) into data science team makes great sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a number of reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;there will be a lot of interaction between business proper and the data science team. BAs are used to acting as an interface layer between IT and business functions. This is much harder than it seems, as IT and (for example) Sales &amp;amp; Marketing people are by nature extremely different. It takes a lot of patience and experience to get them both on the same page. Hint: when talking to Sales or Accounting do not start with an icebreak about the virtues of Open Source!&lt;/li&gt;&#xA;&lt;li&gt;BAs usually have better domain knowledge than DSs. They can be of great help in initial phases of the project as they tend to have a more natural &quot;feeling&quot; for the business (e.g. which outcomes are possible, and which not, are we forecasting a rare event and some under/over sampling is in order, what sort of results are no news and what kind are so surprising that they should be brought to board attention immediately etc.). Data Science types are good at finding the correct answers once a problem is formulated, but asking the right question is arguably more important. And it is a task a BA can help with.&lt;/li&gt;&#xA;&lt;li&gt;Business Analysis is around for longer, so there is a deep and broad pool of experienced people around. Data Science is young but sexy and looks nice in a resume, so it will be easy to persuade a smart guy/gal with strong understanding of the underlying business and good relationship with the key players in the company to become a project member. It is a win win situation. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33646" LastEditorUserId="33646" LastEditDate="2017-06-22T17:18:40.200" LastActivityDate="2017-06-22T17:18:40.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19918" PostTypeId="2" ParentId="19854" CreationDate="2017-06-22T15:45:01.893" Score="1" Body="&lt;p&gt;I have learned what the correct answer is: you have to transform your prediction location in precisely the same way you do for the columns of matrix $X$: first, subtract the means element-wise from each component of the prediction location $x$. Second, divide element-wise by the standard deviation. Third, prepend a $1$ to allow for the bias. Finally, perform the dot product with your $\theta$ vector. Since the $\theta$ vector was calculated on transformed data, it's meant to operate on transformed data. It will contain units and scaling appropriate to produce an answer in engineering units. Reference: see &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/discussions/weeks/2/threads/VjP4bCRmEeeuuQ7tmHEh3A&quot; rel=&quot;nofollow noreferrer&quot;&gt;Week 2, FAQ, Question 8&lt;/a&gt; in Andrew Ng's Machine Learning course on Coursera. (Login is probably necessary.)&lt;/p&gt;&#xA;" OwnerUserId="32440" LastEditorUserId="32440" LastEditDate="2017-06-22T15:51:32.843" LastActivityDate="2017-06-22T15:51:32.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19920" PostTypeId="2" ParentId="19865" CreationDate="2017-06-22T16:48:27.677" Score="5" Body="&lt;p&gt;The documentation offers a couple options. To plot the individual trees in your forest, one can access them like &lt;code&gt;model.estimators_[n].tree_&lt;/code&gt; and then plot them with &lt;code&gt;export_graphviz&lt;/code&gt; as explained in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html#classification&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;, or you can follow &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html&quot; rel=&quot;noreferrer&quot;&gt;this example&lt;/a&gt; that directly prints the structure in text format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would say this is not the best idea, because a feature can occur in different trees and nodes with different split points. You probably get a better intuition about your features from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html&quot; rel=&quot;noreferrer&quot;&gt;partial dependence plots&lt;/a&gt; that try to isolate the effect of one variable on your response variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a bonus, &lt;a href=&quot;https://www.oreilly.com/ideas/ideas-on-interpreting-machine-learning?imm_mid=0ef03f&amp;amp;cmp=em-data-na-na-newsltr_20170320&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; is a good article about more alternatives to gain insight into your model (not all applicable to random forests).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-06-22T16:48:27.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19922" PostTypeId="1" AcceptedAnswerId="35643" CreationDate="2017-06-22T20:04:18.330" Score="8" ViewCount="9952" Body="&lt;p&gt;When training a neural network with &lt;a href=&quot;https://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;keras&lt;/a&gt; for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of&#xA;$$\textstyle\text{loss}(p^\text{true}, p^\text{predict}) = -\sum_i p_i^\text{true} \log p_i^\text{predict}$$&#xA;but couldn't find a definitive answer in the &lt;a href=&quot;https://keras.io/losses/#categorical_crossentropy&quot; rel=&quot;noreferrer&quot;&gt;docs&lt;/a&gt; nor in the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/losses.py&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;. An authoritative reference is desirable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the code I'm not sure if the computation is delegated to &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits&quot; rel=&quot;noreferrer&quot;&gt;tensorflow&lt;/a&gt;/&lt;a href=&quot;http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#theano.tensor.nnet.nnet.categorical_crossentropy&quot; rel=&quot;noreferrer&quot;&gt;theano&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(There is an &lt;a href=&quot;https://datascience.stackexchange.com/questions/14415/how-does-keras-calculate-accuracy&quot;&gt;analogous question&lt;/a&gt; concerning the accuracy; the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/metrics.py&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt; is clearer, but I don't see a call to mean()?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS.&#xA;From this code, it appears that loss and accuracy are computed as in &lt;code&gt;loss_and_acc(...)&lt;/code&gt; but &lt;strong&gt;before&lt;/strong&gt; the last training epoch (keras version 2.0.4, same results for tensorflow and theano backend).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/python3&#xA;&#xA;import numpy as np&#xA;from numpy.random import randint, seed&#xA;&#xA;from keras import __version__ as keras_version&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;&#xA;N = 4 # Classes&#xA;S = 10 # Samples&#xA;&#xA;nn = Sequential()&#xA;nn.add(Dense(input_dim=1, units=N, kernel_initializer='normal', activation='softmax'))&#xA;nn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&#xA;seed(7)&#xA;X = np.random.random((S, 1))&#xA;Y = np.vstack([np.eye(1, N, k=randint(0, N)) for _ in range(S)])&#xA;#for (x, y) in zip(X, Y) : print(x, y)&#xA;&#xA;def loss_and_acc(NN, X, Y) :&#xA;    loss = []&#xA;    acc  = []&#xA;    for (p, q) in zip(Y, NN.predict(X)) :&#xA;        loss += [ -sum(a*np.log(b) for (a, b) in zip(p, q) if (b != 0)) ]&#xA;        acc  += [ np.argmax(p) == np.argmax(q) ]&#xA;    return (np.mean(loss), np.mean(acc))&#xA;&#xA;print(&quot;Keras version: &quot;, keras_version)&#xA;&#xA;for _ in range(10) :&#xA;    print(&quot;Before:  loss = {}, acc = {}&quot;.format(*loss_and_acc(nn, X, Y)))&#xA;    H = nn.fit(X, Y, epochs=1, verbose=0).history&#xA;    print(&quot;History: loss = {}, acc = {}&quot;.format(H['loss'][-1], H['acc'][-1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Using Theano backend.&#xA;Keras version:  2.0.4&#xA;Before:  loss = 1.3843669414520263, acc = 0.2&#xA;History: loss = 1.3843669891357422, acc = 0.20000000298023224&#xA;Before:  loss = 1.3834303855895995, acc = 0.2&#xA;History: loss = 1.3834303617477417, acc = 0.20000000298023224&#xA;Before:  loss = 1.3824962615966796, acc = 0.3&#xA;History: loss = 1.3824962377548218, acc = 0.30000001192092896&#xA;Before:  loss = 1.381564486026764, acc = 0.3&#xA;History: loss = 1.3815644979476929, acc = 0.30000001192092896&#xA;Before:  loss = 1.380635154247284, acc = 0.3&#xA;History: loss = 1.380635142326355, acc = 0.30000001192092896&#xA;Before:  loss = 1.3797082901000977, acc = 0.3&#xA;History: loss = 1.3797082901000977, acc = 0.30000001192092896&#xA;Before:  loss = 1.378783941268921, acc = 0.2&#xA;History: loss = 1.378783941268921, acc = 0.20000000298023224&#xA;Before:  loss = 1.3778621554374695, acc = 0.2&#xA;History: loss = 1.3778622150421143, acc = 0.20000000298023224&#xA;Before:  loss = 1.3769428968429565, acc = 0.2&#xA;History: loss = 1.3769428730010986, acc = 0.20000000298023224&#xA;Before:  loss = 1.3760262489318849, acc = 0.3&#xA;History: loss = 1.3760262727737427, acc = 0.30000001192092896&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33728" LastEditorUserId="33728" LastEditDate="2017-07-03T15:05:36.633" LastActivityDate="2018-08-17T08:01:37.200" Title="Keras categorical_crossentropy loss (and accuracy)" Tags="|keras|categorical-data|reference-request|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19923" PostTypeId="2" ParentId="19896" CreationDate="2017-06-22T20:51:18.143" Score="2" Body="&lt;p&gt;I have my own project that works similarly (but much simpler model), and it takes me about 0.1s to run my predictions in real time. You did the right thing by re-using the session -- that's what I did too. My guess is that your bottleneck is the size of the model. As far as I'm aware, the Inception model is huge. There will always be a tradeoff between model complexity and runtime prediction speed. The Inception model is accurate, but no one ever said it was fast. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2017-06-22T20:51:18.143" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19924" PostTypeId="2" ParentId="19893" CreationDate="2017-06-22T23:10:16.067" Score="0" Body="&lt;p&gt;You can test the following R code, using the R package &lt;code&gt;multcomp&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;md &amp;lt;- datasets::mtcars&#xA;md$cyl &amp;lt;- factor(md$cyl)&#xA;md$gear &amp;lt;- factor(md$gear)&#xA;md$carb &amp;lt;- factor(md$carb)&#xA;my.mod1  &amp;lt;- glm(mpg~., data=md)&#xA;library(multcomp)&#xA;summary(glht(my.mod1, mcp(cyl=&quot;Tukey&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8904" LastActivityDate="2017-06-22T23:10:16.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19925" PostTypeId="1" CreationDate="2017-06-23T02:09:35.357" Score="9" ViewCount="21799" Body="&lt;p&gt;I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical).&#xA;I searched about this topic on the web, but no luck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I would like to know what kind of data performs best according to given machine learning algorithm?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;It is better to explain briefly why certain kinds of data suitable for certain machine learning algorithms?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Hope answer to this question will help beginners in data science.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: It is better if you can explain what are the types of data most suitable for following algorithms.&#xA;Naive Bayes, SVM, Regression, K-Means, Deep Neural networks.&lt;/p&gt;&#xA;" OwnerUserId="33272" LastEditorUserId="33272" LastEditDate="2017-06-24T02:56:07.430" LastActivityDate="2017-06-24T02:56:07.430" Title="What are the most suitable machine learning algorithms according to type of data?" Tags="|machine-learning|algorithms|data|" AnswerCount="1" CommentCount="3" ClosedDate="2017-06-24T09:10:16.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="19928" PostTypeId="2" ParentId="19925" CreationDate="2017-06-23T06:05:28.130" Score="7" Body="&lt;p&gt;Given the list of algorithms you provided, these falls under 3 major classification of ML algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Classification Algorithms - Naive Bayes Classification, Decision Tree, Random Forest, kNN, Support Vector Machine (SVM), Neural Networks, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Regression Algorithms - Linear Regression, Logistic Regression, Lasso Regression, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note: Although Logistic Regression has Regression in its name, it is essentially a classification algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Clustering Algorithms - K-Means Clustering, Fuzzy C Means, Mixture of Gaussian, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also be knowing there are 4 types of ML algorithms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Supervised Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Unsupervised Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Reinforcement Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Semi-supervised Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Among these 4. first and second are the most important ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supervised Learning is applied when we have a labelled data set i.e., we already our output variable/dependent variable. For example, a data set which contains the size of the house (independent variable) and corresponding house price (dependent variable). We can predict the house price of new data points with respect to the size of the house. Another example is, determining if a tumor is harmful or not harmful when we already have a list of tumors which are harmful or not. &#xA;In supervised learning we know the problem statement and have all the necessary features to get answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Unsupervised Learning, we do not have labelled data. We do not have any output variable. We do not know the problem statement. It is applied when we need to find a structure in the data set and extract meaningful insights out of it. For example, a data set of Walmart containing its customer's buying pattern. Given this, Walmart will ask its data scientists to extract some meaning. The data scientist may choose to apply K-Means Clustering and find how customers are segmented. Group A customers -- buys X,Y,Z products; Group B customers -- buys U,V,X products. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classification and regression algorithms are used when dealing with a supervised learning problem and clustering algorithms are used when dealing with unsupervised learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now coming back to your original query, &#xA;1) Naive Bayes -- best applied to a data set containing multiple features (independent variable) and an output variable which takes two discrete value (Yes/No). Thus, categorical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) SVM -- best applied to a data set containing infinite number of features and you need to reduce these features down to a number so that it can be computed. Since it's a classification algorithm so it best works upon categorical data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Regression -- Linear Regression is applied to a continuous numerical data set in which the dependent and independent variable exhibits linear relationship. For example, size of the house vs house price.  Logistic Regression is a classification algorithm so it is best applied to categorical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) K-Means -- K-Means can applied to many types of data sets. What it does is segmenting data points into clusters. Data points with similar features are clustered together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Neural Networks -- Neural Networks can be shallow neural networks and deep neural networks and both of these could be applied to supervised or unsupervised problem as it has separate algorithms for both the cases. It is the most powerful and popular class of ML algorithms. It can be used in every problem statement. Main intuition behind it learning from its own error. I do not have much knowledge about neural network so I will not write further more. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to learn more about ML and Neural Network you can apply for &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;noreferrer&quot;&gt;Machine Learning by Andrew Ng&lt;/a&gt; class. It's the best course out there for beginners like us. &lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-06-23T06:05:28.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19930" PostTypeId="2" ParentId="19163" CreationDate="2017-06-23T07:49:04.983" Score="4" Body="&lt;p&gt;Use the &lt;code&gt;view&lt;/code&gt; method (&lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/timeseries.html#from-timestamps-to-epoch&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import Series&#xA;from datetime import date&#xA;df = Series([date(2007,4,30), date(2007,5,31), date(2007,6,30), date(2007,7,31), date(2007,8,31)], dtype='datetime64')&#xA;df.view('int64')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    1177891200000000000&#xA;1    1180569600000000000&#xA;2    1183161600000000000&#xA;3    1185840000000000000&#xA;4    1188518400000000000&#xA;dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-06-23T07:49:04.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19931" PostTypeId="2" ParentId="19901" CreationDate="2017-06-23T07:58:48.440" Score="7" Body="&lt;p&gt;There are a lot of misconceptions about regression random forest. Those misconceptions about regression rf are seen also in classification rf, but are less visible. The one I will present here is that regression random forests do not overfit. Well, this is not true. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Studying the statistical properties of the random forests shows that the bootstrapping procedure decreases the variance and maintain the bias. This property should be understood under the bias-variance tradeoff framework. It is clear that the random forests approximate an expectation, which means the mean of the true structure remains the same, while the variance is reduced. From this perspective, the random forests do not overfit. There is a problem, however, that problem is the sample itself which is used for training. The expectation is taken conditional on data. And if the data is not representative of the problem, it is normal that in the limit when the number of the tree grows to infinity. In plain English, this means that the regression forest will learn too well the data and if the data is not representative, then the results are bad. In which way the data might not be representative? In many ways, one would be that you do not have enough data points in all region of interests for example. This problem is seen often with testing error, so it might not affect you so much, but is possible to see that in CV also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another issue with regression trees is the number of significant variables and the number of nonsignificant variables in your data set. It is known that when you have few interesting input variables and a large number of noise variables the regression forests does not behave well. Boosting procedures does not have this behavior. There is a good reason for that. Regression forests produce more uninteresting trees which have the potential to move the learned structure away from the true underlying structure. For boosting this does not happen since at each iteration only the region of interests have large weight, so the already learned regions are affected less. The remedy would be to play with the number of variables selected on learning time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a drawback however even if you increase the number of variables take into account at learning time. Consider two randomly grown trees. If you would have 100 input variables and select 10 of them for learning, there are small chances that the trees look similar. If instead you would select 50 variables for learning then your trees have better chances to look similar. This is translated into the fact that if you increase the number of variables for testing candidates at learning time, then the correlations between them increases. If the correlation increases, then they will not be able to learn a more complex structure, because of their correlations. If the number of variables selected is small you have the potential to learn more due to diversity, but if the significant variables are small compared to nonsignificant, this would lead to learn noise, too much noise. This affects most of the time the CV performance. &lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2017-06-23T07:58:48.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19933" PostTypeId="1" CreationDate="2017-06-23T09:52:04.020" Score="7" ViewCount="12203" Body="&lt;p&gt;tl;dr I'm building a binary classifier that always eventually predicts all &quot;0&quot; or all &quot;1&quot; after some number of epochs and I'm looking for possible reasons why/how to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is all just more details on my approach and thoughts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I'm doing&lt;/strong&gt;: I am building a binary classification basic feed-forward neural network for signal processing with tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I pre-process my data by separating it into &quot;windows&quot; of length window_size points (say like 50), and that window/example can be considered either a &quot;1&quot; (positive) or a &quot;0&quot; (negative). The official, manually annotated labels are located at specific points (e.g. 403, 875, 1450, etc.) and are relatively evenly spaced. I decided that my example/window is positive if the window contains a label, negative otherwise. Multiple examples are generated from moving the window by some stride_length (say 1, 2, or 10 points, etc.), until my moving window reaches the end of the signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the training data, since most of my examples are negative, I've tried to normalize the ratio of neg to pos examples so it's close to 1:1. I do this by deleting a calculated ratio of negative examples. For test data I skip this last step, but everything else is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run it through a 3 layer NN with RELU's on hidden layers and sigmoid on the sole output, with &amp;lt;0.5 predicting a negative and &gt;0.5 predicting a positive. Cost is sum of (label - prediction)^2, using AdamOptimizer at default LR (0.001).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My issue:&lt;/strong&gt;&#xA;No matter what parameters I adjust, my NN seems to predict everything as either negative or positive after a certain number of epochs (sometimes even immediately at the first epoch). It's pretty random too. I'll run the same thing multiple times and I'll get something different every time, about half of the predicting all positive, and the other half all negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Things I have tried/thought about:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I shuffled my training examples before training my model, with not much improvement.&lt;/li&gt;&#xA;&lt;li&gt;I cut down on my negative examples as I stated above, since I had this problem before I cut down (was predicting basically all negative immediately). But now that I have close to a 1:1 ratio of pos/neg examples, it's flip-flopping between predicting all negative and all positive, so it's extremely sensitive to this ratio.&lt;/li&gt;&#xA;&lt;li&gt;I originally had a softmax_cross_entropy_with_logits cost function with one-hot encoding, so 0 1 was positive and 1 0 was negative. (I thought maybe the cost function was rewarding extreme predictions, so I switched to sum of squares which is harsher)&lt;/li&gt;&#xA;&lt;li&gt;I'm thinking of trying batch normalization and dropout, but I don't expect those to fix my problem.&lt;/li&gt;&#xA;&lt;li&gt;I think the problem might be in how I am pre-processing the data. My thought process is this: if the ratio of pos/neg examples is so important that it is making my model predict all positive or all negative, then maybe there is no real trend in my data (which is wrong, since anyone with eyes can see the difference). My thought was that because I have a moving window, sometimes the label is near the left edge of my moving window and sometimes the right edge, which means the actual region of interest is getting input into different neurons each time, which confuses the neurons so it feels like there is no real trend.&lt;/li&gt;&#xA;&lt;li&gt;However, I took a look at the weights and biases and they seem to be stabilizing after a few epochs. Shouldn't they still be jumping around wildly if there really is no trend in the data? Maybe that is wrong.&lt;/li&gt;&#xA;&lt;li&gt;I was thinking about testing my hypothesis out by running the left-leaning label examples and right-leaning label examples through different neural networks and seeing if I get better results, but maybe it's not worth the effort it will take.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've been trying this for a while and can't seem to think of a good explanation or approach. I would really appreciate any help I can get on this, and I'm happy to provide more information. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="33135" LastActivityDate="2017-06-28T19:17:47.677" Title="Binary Classifier making only one prediction" Tags="|classification|deep-learning|tensorflow|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19934" PostTypeId="2" ParentId="19858" CreationDate="2017-06-23T11:00:17.893" Score="0" Body="&lt;p&gt;Since you mentioned about in &lt;strong&gt;&lt;em&gt;stock market predictions&lt;/em&gt;&lt;/strong&gt;. I suggest you to checkout the &lt;a href=&quot;https://www.udacity.com/course/machine-learning-for-trading--ud501&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning for trading course at Udacity.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33272" LastActivityDate="2017-06-23T11:00:17.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19935" PostTypeId="2" ParentId="17114" CreationDate="2017-06-23T11:54:11.947" Score="1" Body="&lt;p&gt;You could call it &quot;axis scaling&quot;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the first is ordinal scale (values are clearly ordered, but that is it)&lt;/li&gt;&#xA;&lt;li&gt;the second is interval scale (values are ordered, and there is a meaningful unit of measurement - a year in this case)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find it discussed in more detail on &lt;a href=&quot;https://en.wikipedia.org/wiki/Level_of_measurement&quot; rel=&quot;nofollow noreferrer&quot;&gt;scales of measurement&lt;/a&gt; article on Wikipedia.&lt;/p&gt;&#xA;" OwnerUserId="33646" LastEditorUserId="33646" LastEditDate="2017-06-23T12:12:59.923" LastActivityDate="2017-06-23T12:12:59.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19936" PostTypeId="2" ParentId="19882" CreationDate="2017-06-23T15:18:54.590" Score="4" Body="&lt;p&gt;Finally I have solved this issue by: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.booster().get_score(importance_type='weight')&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2017-06-23T15:18:54.590" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19940" PostTypeId="1" CreationDate="2017-06-23T19:54:18.457" Score="-1" ViewCount="130" Body="&lt;p&gt;I want to change 8 attributes which are numeric into nominal. I used equal width binning to specify intervals. Does the bin for each attribute need to be equal? For example, when I discretize attribute A into 3 bins, do the bins of the remaining attributes need to be 3 as well, or not?&lt;/p&gt;&#xA;" OwnerUserId="33763" LastEditorUserId="23305" LastEditDate="2017-06-23T21:07:54.220" LastActivityDate="2017-07-23T23:02:24.530" Title="Do I need equal number of bins for all attributes?" Tags="|machine-learning|data-mining|predictive-modeling|data-cleaning|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19941" PostTypeId="2" ParentId="19940" CreationDate="2017-06-23T21:05:09.217" Score="0" Body="&lt;p&gt;If there is no reason specific to your problem to have equal bins, then I'd say no. In fact, binning attributes is often criticized for being &quot;arbitrary&quot;. You should not select the number of bins based on some arbitrary rule. Binning usually makes only sense if you expect your response to be a step function. If this is not the case, you should consider carefully how you select your bins and whether binning is appropriate for every variable individually. See &lt;a href=&quot;http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for a list of potential problems and &lt;a href=&quot;https://stats.stackexchange.com/a/68839/127417&quot;&gt;this answer on crossvalidated&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-06-23T21:05:09.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19943" PostTypeId="1" CreationDate="2017-06-24T14:38:50.670" Score="1" ViewCount="4049" Body="&lt;p&gt;I have 600 customers per week. How many do I need to survey to get a good sample to work with?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm very new to stats.&lt;/p&gt;&#xA;" OwnerUserId="32763" LastActivityDate="2017-06-24T21:01:01.493" Title="What is a large enough sample size?" Tags="|statistics|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19945" PostTypeId="1" CreationDate="2017-06-24T17:58:28.370" Score="2" ViewCount="5354" Body="&lt;p&gt;I'm working on a multi-digit image recognition project, but I'm stumbling across a certain error no matter what I do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a sample of my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;graph = tf.Graph()&#xA;with graph.as_default():&#xA;    x = tf.placeholder(tf.float32, shape = (None, 66, 66, 1), name = 'x')&#xA;    y = tf.placeholder(tf.int64, shape = (None, 5), name = 'y')&#xA;    keep_prob = tf.placeholder(tf.float32, name = 'keep_prob')&#xA;&#xA;...&#xA;&#xA;with tf.Session(graph = graph) as sess:&#xA;    sess.run(tf.global_variables_initializer())&#xA;&#xA;    for step in range(num_steps):&#xA;        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)&#xA;        batch_images = train_images[offset:(offset + batch_size), :, :, :]&#xA;        batch_labels = train_labels[offset:(offset + batch_size), :]&#xA;        if step % 100 == 0:&#xA;            acc = accuracy.eval(feed_dict = {x: batch_images, y: batch_labels, keep_prob: 1.0})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When trying to calculate &lt;code&gt;acc&lt;/code&gt;, I get the error &lt;code&gt;TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a float into a Tensor&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know why I'm getting this error. My variables are defined in the same graph and I am not overriding my variables.&#xA;Any ideas on what is causing this error?&lt;/p&gt;&#xA;" OwnerUserId="33784" LastActivityDate="2017-06-24T17:58:28.370" Title="Tensorflow: can not convert float into a tensor?" Tags="|python|tensorflow|image-recognition|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19946" PostTypeId="2" ParentId="19943" CreationDate="2017-06-24T18:06:26.173" Score="8" Body="&lt;p&gt;It highly depends, If you want to make inferences on the total population you need to determine what margin of error and what confidence level you can allow for. These will then help you determine the needed sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we would want to estimate the population mean $\mu$. The maximum difference between the sample mean $\bar x$ and $\mu$ is given by: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \epsilon = z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}  $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $n$ is the sample size, $\sigma$ is the population standard deviation and $z_{\frac{\alpha}{2}}$ is commonly called the critical value which is derived from the standard normal distribution. These values can be found in tables or using r or python, e.g 1.96 corresponds to a 95% confidence level. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing $\sigma$ (you usually don't) will allow you to determine the sample size needed to approximate $\mu$ within $\pm \epsilon $ with a confidence level of $1-\alpha$. You can try using $\sigma =  \frac{1}{2}$ which is usually enough. There exists methods for determining $\sigma$ as well. Anyhow, you may rearrange the above relation as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ n = \bigg ( \frac{z_\frac{\alpha}{2}\sigma }{\epsilon}  \bigg )^2 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, this question is quite hard to answer. There are many factors that determine the sample size needed for your study. You should definitely go through some material on basic statstics starting with &#xA;&lt;a href=&quot;https://www.youtube.com/watch?v=KbB0FjPg0mw&amp;amp;list=PL2SOU6wwxB0uwwH80KTQ6ht66KWxbzTIo&quot; rel=&quot;noreferrer&quot;&gt;Awesome probability theory&lt;/a&gt; and go on studying inferential statistics.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth taking some time to understand what the total population is in your case and with regard to the inferential question, does the sample include enough relevant observations? I do not feel that enough info is given in your question to provide a good answer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may look at similar surveys conducted and see what sample sizes they have used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another common approach, when you don't know much about your population is using &lt;em&gt;Slovin's formula&lt;/em&gt; : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$n = \frac{N}{(1+N \epsilon ^2)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $N$ is the total population. As an example given a confidence level of 98% $\rightarrow \epsilon = 0.02$  and a total population of 600. we would get  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$n = \frac{600}{1+600\times0.02^2} = 484.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;em&gt;Slovin's formula&lt;/em&gt;  is easily misused but can give you a good starting point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would highly recommend studying statistics if you are planning to continue working with DS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this provided some help at least. &lt;/p&gt;&#xA;" OwnerUserId="33781" LastEditorUserId="33781" LastEditDate="2017-06-24T21:01:01.493" LastActivityDate="2017-06-24T21:01:01.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19947" PostTypeId="1" CreationDate="2017-06-24T21:23:10.777" Score="2" ViewCount="4972" Body="&lt;p&gt;I'm trying to understand what level of measurement is best for describing the 'number of rooms in a flat' feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, I think it's not a continuous feature, because rational values like 1.42, for example, don't make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To decide whether a feature is categorical or nominal, we should try to find an ordering between values. And here is my question: should we look for an order with respect to the response feature (in my case 'Price of a property')?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can say a '1-room flat' is cheaper than a '2-room flat', and so on. But that is not always the case. In general, maybe it's true, but there are cases when a '1-room flat' in the city centre is way more expensive than the one further away.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a can't decide which representation to choose - categorical or nominal.&lt;/p&gt;&#xA;" OwnerUserId="32757" LastEditorUserId="13413" LastEditDate="2017-06-27T02:51:34.993" LastActivityDate="2017-07-27T14:45:37.197" Title="Categorical, nominal or continuous variable?" Tags="|categorical-data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19948" PostTypeId="2" ParentId="16045" CreationDate="2017-06-24T22:57:01.680" Score="2" Body="&lt;p&gt;As you mentioned, the mask matrix is sampled and multiplied with the activations in the feature map at layer $l$ to produce dropped out modified activations which are then convolved with the filter at the next layer $W^{(l+1)}$. (3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details, I think section 3 in this paper might help you out: &lt;a href=&quot;https://arxiv.org/ftp/arxiv/papers/1512/1512.00242.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Max-pooling &amp;amp; Convolutional dropout&lt;/a&gt;. Specifically 3.2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you test you use all nodes of the network but with the filter’s weights scaled by the retaining probability, as explained in the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please feel free to refine or correct my answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps at least a little. &lt;/p&gt;&#xA;" OwnerUserId="33781" LastEditorUserId="33781" LastEditDate="2017-08-19T22:17:30.180" LastActivityDate="2017-08-19T22:17:30.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19950" PostTypeId="2" ParentId="19167" CreationDate="2017-06-25T15:02:02.733" Score="1" Body="&lt;p&gt;Do you have data imbalance?&#xA;The answer is yes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a problem?&#xA;It depends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data imbalance ratio is quite high (1:10), is it a problem or not, it depends on three things: &#xA;- what is the used classifier?&#xA;- are these separable classes or not ?&#xA;- are you interested in overall accuracy or average accuracy ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The impact is to have bias classifier! If you have a classifier that predicts all the samples as class one , you will have overall accuracy of ~ 70%.&#xA;But it is a biased classifier the results in average accuracy of ~ 33%&lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-06-25T15:02:02.733" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19951" PostTypeId="1" CreationDate="2017-06-25T15:16:46.130" Score="2" ViewCount="394" Body="&lt;p&gt;I need to design a controller that tries to collect as much falling objects (raindrops) from an unknown source. Beneficial to solving the problem could be the recognition of the seemingly random generation of the raindrops.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y  ^&#xA;   | 0 0 1 0 0 0 0&#xA;   | 0 0 1 0 0 1 0&#xA;   | 1 0 0 0 0 1 0&#xA;   | 0 0 0 1 1 0 0  &#xA;   -----------------&amp;gt;&#xA;                  t       &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, I have a binary vector signal (raindrop generated yes/no over 4 positions) over discrete time steps. I can only see the raindrops when they are 10 steps away from hitting the ground. The basket has a capacity of one raindrop. The control actions available are step left, step right or do nothing. Any source of pattern recognition could help me design a more advanced controller. Are there methods that can recognize periodic patterns in this type of data?&lt;/p&gt;&#xA;" OwnerUserId="33807" LastEditorUserId="33807" LastEditDate="2017-06-26T21:30:42.447" LastActivityDate="2017-06-27T11:59:09.860" Title="Recognize and predict a periodic pattern" Tags="|prediction|sequential-pattern-mining|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="19952" PostTypeId="2" ParentId="13438" CreationDate="2017-06-25T16:29:47.117" Score="3" Body="&lt;p&gt;I am affraid this dataset does not include labels, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use datasets that are described here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://patrykchrabaszcz.github.io/Imagenet32/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://patrykchrabaszcz.github.io/Imagenet32/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those files will have the same format as CIFAR dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And they will contain labels according to the map file that is available in original Imagenet devkit.&lt;/p&gt;&#xA;" OwnerUserId="33811" LastActivityDate="2017-06-25T16:29:47.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19953" PostTypeId="2" ParentId="19951" CreationDate="2017-06-25T16:53:46.993" Score="2" Body="&lt;p&gt;If it is really close to random (background noise, bits from random number generators or hash codes), I don't expect there is much you can do, except create a large collector. (Or are you only allowed one collector?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may want to throw in a few random checks. Those won't prove anything but may give you some insight into the pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to create a successful model, you'd have to come up with some assumptions about parameters that could affect the success of your model, and create and test that model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, one of my assumptions would be that there are rows with higher probability.&#xA;One other assumption would be that there is some hysteresis effect (either positive or negative), so the probability may depend on the actual outcome of the previous event(s).&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-06-25T16:53:46.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19954" PostTypeId="2" ParentId="19947" CreationDate="2017-06-25T17:07:58.260" Score="0" Body="&lt;p&gt;I'd stick with continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A number of 1.42 could mean that you may have a single room, but with significantly more area than the average room.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may even have some fraction if you need to share kitchen and bathroom with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those things may be easier to fit in a number than in yet another category you don't know how to order.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-06-25T17:07:58.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19957" PostTypeId="1" CreationDate="2017-06-25T20:30:43.327" Score="3" ViewCount="45" Body="&lt;p&gt;I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When doing model selection (selecting hyperparameters), I sometimes come across these cases:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;hyperparameter configuration 1: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training set score: 0.6&#xA;validation set score: 0.6&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;hyperparameter configuration 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training set score: 0.9&#xA;validation set score: 0.65&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now if you look at the raw numbers, it does look like &lt;strong&gt;configuration 2 generalizes better&lt;/strong&gt; than configuration 1 but I'm a little bit worried about the large difference (0.9 to 0.65) between the scores for the training data and the validation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;strong&gt;should I just consider the validation score when choosing the best model for actually using in production&lt;/strong&gt; or does the &lt;strong&gt;ratio&lt;/strong&gt; between training set and validation set scores carry some information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm tempted to believe that cases where the training set and validation set score are more or less similar (as in configuration 1 above) are somehow more &lt;em&gt;stable&lt;/em&gt; than cases where there is a large difference, even though, in absolute terms, the generalization score is better for configuration 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any actual basis for this feeling? Could someone shed some light into this?&lt;/p&gt;&#xA;" OwnerUserId="413" LastActivityDate="2017-06-25T20:30:43.327" Title="Intuitive interpretation of ratios between training set scores and validation set scores" Tags="|machine-learning|cross-validation|model-selection|scoring|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19958" PostTypeId="1" CreationDate="2017-06-25T22:56:54.923" Score="3" ViewCount="585" Body="&lt;p&gt;I am new to recommender systems and am trying to find similar users of base users for user-based collaborative filtering.&lt;/p&gt;&#xA;&lt;p&gt;When I calculated the similarity score between two users (based on their ratings with Pearson algorithm [or Resnick's weighted Pearson algorithm]) I get a similarity score from -1 to 1.&lt;/p&gt;&#xA;&lt;p&gt;Is it a good idea to normalize these values to 0 to 1 (-1 would become 0 and 1 would be 1) to make it comparable to other algorithms?&lt;/p&gt;&#xA;&lt;p&gt;In fact, I tried to build recommendations with a negative similarity score of a user, the calculated/predicted rating could be negative as well which makes no sense.&lt;/p&gt;&#xA;&lt;p&gt;Should I normalize/scale &amp;quot;-1 to 1&amp;quot; to &amp;quot;0 to 1&amp;quot; or cut off all users with similarity scores below 0?&lt;/p&gt;&#xA;&lt;p&gt;(maybe the question also could be: &amp;quot;Which users should be taken as a mentor to recommend new items on a similarity score from -1 to 1? Or should I take the top n users with highest similarity score?&amp;quot;)&lt;/p&gt;&#xA;" OwnerUserId="33806" LastEditorUserId="29169" LastEditDate="2021-06-15T07:11:58.310" LastActivityDate="2023-12-17T22:01:55.233" Title="Use of negative correlation coefficient in pearson correlation algorithm for recommender systems" Tags="|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19959" PostTypeId="2" ParentId="18820" CreationDate="2017-06-26T01:42:18.093" Score="1" Body="&lt;p&gt;It looks like you might be trying to use &lt;code&gt;MultiRNNCell&lt;/code&gt; as a recurrent neural network. This isn't really its purpose; that is what dynamic_rnn, static_rnn, and raw_rnn are for. &lt;code&gt;MultiRNNCell&lt;/code&gt; is intended for creating a feedforward, multilayer network which is the &quot;cell&quot; to be iterated to produce a recurrent neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The old version of your code, if I understand it correctly, creates a network consisting of a sequence of LSTM layers, each with its own set of weights. If your goal is to make the weights shared between layers (so that your network is recurrent), you want to instantiate a &lt;code&gt;dynamic_rnn&lt;/code&gt; and pass it a single instance of one of the RNNCell subclasses. That instance generates all of the layers of the network by having its &lt;code&gt;__call__&lt;/code&gt; method invoked repeatedly, once for each layer. The &lt;code&gt;dynamic_rnn&lt;/code&gt; instance handles the scoping and variable reuse issues for you and connects the layers together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have at least three options for the cell to pass to the &lt;code&gt;dynamic_rnn&lt;/code&gt; constructor:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) A &lt;code&gt;DropoutWrapper&lt;/code&gt; which was created using a &lt;code&gt;BasicLSTMCell&lt;/code&gt; (essentially, the output of &lt;strong&gt;one&lt;/strong&gt; call to &lt;code&gt;attn_cell&lt;/code&gt;, but get rid of that &lt;code&gt;reuse&lt;/code&gt; option). Do this option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) A &lt;code&gt;MultiRNNCell&lt;/code&gt; consisting of an LSTM layer, followed by a dropout layer (unneeded here, but this is what you would do in some more general case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Subclass &lt;code&gt;rnn_cell&lt;/code&gt; and create your own type of layer.&lt;/p&gt;&#xA;" OwnerUserId="33825" LastActivityDate="2017-06-26T01:42:18.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19960" PostTypeId="2" ParentId="223" CreationDate="2017-06-26T02:31:18.360" Score="1" Body="&lt;p&gt;To describe all existed data it is so difficult task, but we can use a data model: &lt;a href=&quot;http://schema.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://schema.org/&lt;/a&gt;, where are structural types of the information. The prior execution was targeted to implement MarkUp technology, so, it seems can be useful for your task.&lt;/p&gt;&#xA;" OwnerUserId="33258" LastActivityDate="2017-06-26T02:31:18.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19962" PostTypeId="2" ParentId="11930" CreationDate="2017-06-26T06:15:24.287" Score="1" Body="&lt;p&gt;Explanation straight from Hinton's course on Neural Networks for Machine Learning ....&#xA;&lt;br/&gt;&#xA;&lt;br/&gt;&#xA;&quot; Dimension hopping occurs when &lt;strong&gt;one can take the information contained in the dimensions of some input, and move this between dimensions while not changing the target&lt;/strong&gt;. The canonical example is taking an image of a handwritten digit and translating it within the image. The dimensions that contain &quot;ink&quot; are now different (they have been moved to other dimensions), however the label we assign to the digit has not changed. Note that this is not something that happens consistently across the dataset, that is we may have a dataset containing two handwritten digits where one is a translated version of the other, however this still does not change the corresponding label of the digits.&quot;&lt;/p&gt;&#xA;" OwnerUserId="31188" LastActivityDate="2017-06-26T06:15:24.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19964" PostTypeId="1" CreationDate="2017-06-26T06:33:19.600" Score="-1" ViewCount="44" Body="&lt;p&gt;I want to create a model to determine gender from the device information and what apps they have used. Right Now I have A set of data approximately 100000 that looks in approximately like this for each device that we get data from.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;- Device ID: 3hfs83478hdfvsdfn (a hash)  &#xA;- Apps Used: com.uber:4,com.amazon.purchase:5(url of the app plus the amount of &#xA;  times the app sent a package with information.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have thought about using a simHash and feeding it into a Boosting model but the results were not great. I think simHash might be a little bit to brutal, does anyone have any suggests about how I can better manipulate the data to get more information out of it?&lt;/p&gt;&#xA;" OwnerUserId="33831" LastEditorUserId="29169" LastEditDate="2021-02-06T18:40:02.530" LastActivityDate="2021-02-06T18:41:28.373" Title="Any Ideas or suggests on data transformation to better suite a boosting or ML model" Tags="|data-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="19965" PostTypeId="1" AcceptedAnswerId="19967" CreationDate="2017-06-26T06:58:49.207" Score="0" ViewCount="57" Body="&lt;p&gt;In stacked generalization, several algorithms (I use some random trees, booster trees, etc.) are first trained and used to make the predictions which are used as input for another algorithm. However, can I use any kind of algorithms, or is there a preference?&lt;/p&gt;&#xA;&lt;p&gt;P.S.: I often see people using linear models in this case.&lt;/p&gt;&#xA;" OwnerUserId="33386" LastEditorUserId="85045" LastEditDate="2020-12-31T12:42:26.587" LastActivityDate="2020-12-31T12:42:26.587" Title="What kind of algorithms can be used as a stacker in stacked generalization?" Tags="|algorithms|ensemble-modeling|stacking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19966" PostTypeId="1" CreationDate="2017-06-26T07:42:43.780" Score="0" ViewCount="155" Body="&lt;p&gt;Hey mates I have the following project:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine having two datasets &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt;. Each dataset consits of 101 time series with the same lenght and identical time stamps. The two datasets where taken from the same experiment, therefore the data structure is identical. From the 101 time series there is one particulary signal that is of interest in both datasets. Lets call that signal &lt;code&gt;X(t)_101&lt;/code&gt;. Now we have the following case that the signal &lt;code&gt;X(t)_101&lt;/code&gt; from dataset &lt;code&gt;A&lt;/code&gt; (good dataset) is significantly different than from dataset &lt;code&gt;B&lt;/code&gt; (bad dataset). The goal now is to analyse the remaining 100 signals and find the signals that have the biggest effect on making our signal of interest &lt;code&gt;X(t)_101&lt;/code&gt; go bad. So in the end I would get some sort of ranking, on which signal is responsible for the change of our signal &lt;code&gt;X(t)_101&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I am looking into novelty detection and correlation, are there any other keywords I should consider for this topic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="29277" LastActivityDate="2024-01-03T07:03:40.440" Title="Alternative methods for novelty detection and correlations" Tags="|time-series|correlation|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19967" PostTypeId="2" ParentId="19965" CreationDate="2017-06-26T09:41:34.817" Score="1" Body="&lt;p&gt;There is no preference for stacked generalization. You can use any algorithm whether it be a &lt;code&gt;NN, RF model, XGB model&lt;/code&gt;, etc. The only thing that you need to take care of is the fact that the algorithm which you are applying to your model is useful to your data or not. Also, model stacked at level 1 may not be a very good model for level 2. For example, suppose at level 1 you stacked an XGB model with &lt;code&gt;max_depth=12, eta=0.01, colsample_bytree=0.7&lt;/code&gt;, but at second level it might happen that the same depth is just an overkill and hence you  should discard using that model at second level. So, in short there is no preference for stacked models, the preference is for the data on which you are going to train your model.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-06-26T09:41:34.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19968" PostTypeId="2" ParentId="19964" CreationDate="2017-06-26T10:12:08.697" Score="0" Body="&lt;p&gt;My two cents:&lt;/p&gt;&#xA;&lt;p&gt;I would have first created an ontology holding a list of all apps used in your data set. Using the ontology and the data I would have vectorized each device into categorical features(has/has not) of each of the apps.&lt;/p&gt;&#xA;&lt;p&gt;Then, I can use a KNN based on a cosine-similarity metric-space of prediction.&lt;/p&gt;&#xA;&lt;p&gt;I am not sure how many apps (bits) the vector will be comprised of, so you might need dimensionality reduction or a more robust classification algorithm (maybe SVM since it can handle lots of features).&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="29169" LastEditDate="2021-02-06T18:41:28.373" LastActivityDate="2021-02-06T18:41:28.373" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19970" PostTypeId="2" ParentId="11220" CreationDate="2017-06-26T11:30:35.943" Score="0" Body="&lt;p&gt;~7000 sample data entries is definitely not enough considering that a more-or-less reliable dictionary of sentiment-loaded words consists of several thousand words each for both positive and negative sentiments. Basically, when you train the model you build such a dictionary in some sense.&lt;/p&gt;&#xA;&lt;p&gt;There is, however, an &lt;a href=&quot;http://ai.stanford.edu/%7Eamaas/data/sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;existing training/test dataset&lt;/a&gt; consisting of 50000 reviews which is a bit better than what you have.&lt;/p&gt;&#xA;&lt;p&gt;At the same time, while the amount of training sample data  contributes to the quality of the classifier, it's also important that the style and dictionary used by authors of texts in the training set was similar to your test texts. In addition, text processing tricks like stemming may increase the training efficiency.&lt;/p&gt;&#xA;&lt;p&gt;For more information, you might want to look at &lt;a href=&quot;https://blog.griddynamics.com/creating-training-and-test-data-sets-and-preparing-the-data-for-twitter-stream-sentiment-analysis-of-social-movie-reviews&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog posts&lt;/a&gt; several colleagues and I wrote about creating training and test data sets.&lt;/p&gt;&#xA;" OwnerUserId="33841" LastEditorUserId="98307" LastEditDate="2020-08-17T18:55:09.717" LastActivityDate="2020-08-17T18:55:09.717" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="19971" PostTypeId="1" CreationDate="2017-06-26T12:32:09.107" Score="5" ViewCount="205" Body="&lt;p&gt;I have a dataset which is probabilistic in nature. For instance, if a record has d features, then values of $x_1$, ..., $x_d$ are such that their sum adds to 1. In other words, for each $x_i$ data record (row)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(x_{i,1}) + P(x_{i,2}) + ...+ P(x_{i,d}) = 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to apply Linear Regression Model to predict a variable $y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;can Linear Regression work for such datasets? What are other models I can use?&lt;/li&gt;&#xA;&lt;li&gt;Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures). &lt;/li&gt;&#xA;&lt;li&gt;As I use Linear Regression all independent variables&#xA;($x_{i,0}$, $x_{i,d}$)'s are &amp;lt; 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $\mathbf{w}$ for&#xA;$\hat{y}_i = \mathbf{w} \cdot \mathbf{x}_i$ calculated would be very large)? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Ideally, I am applying gradient descent for a convex optimization problem. It can be that $x_0...x_d$ belong to some unknown distribution and in my case, I know that distribution. But if there is an alternate model them pls suggest.&lt;/p&gt;&#xA;" OwnerUserId="33844" LastEditorUserId="836" LastEditDate="2017-06-26T19:09:14.690" LastActivityDate="2017-06-26T19:09:14.690" Title="Linear regression on probabilistic data" Tags="|dataset|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19972" PostTypeId="1" AcceptedAnswerId="19986" CreationDate="2017-06-26T13:44:18.403" Score="0" ViewCount="67" Body="&lt;p&gt;I have set of technical sentences extracted from few research papers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g., Via analyzing the peer feedback content, it was found that the feedback provided by the mixed mode group was of better quality than that provided by the &quot;peer comments&quot; group; that is, the former provided more detailed feedback to individuals than the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to extract the entities in the sentences and the relationships that take part with the identified. (e.g., SOV relationships)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the tools that I can use to efficiently extract them?&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-06-26T23:44:12.943" LastActivityDate="2017-06-27T05:27:11.197" Title="Tools to extract information" Tags="|machine-learning|data-mining|nlp|stanford-nlp|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="19973" PostTypeId="2" ParentId="19971" CreationDate="2017-06-26T14:01:43.877" Score="6" Body="&lt;p&gt;The issue with linear regression in your case is not whether it can be used - it can nearly always be used to build a predictive regression model - it is whether the resulting model is useful, usually whether it is accurate enough for the intended use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy and utility of your model does not depend only on the nature of $\mathbf{x}$. Instead it depends on the true nature of the $\mathbf{x} \rightarrow y$ relationship. Your measurements and model are an approximation of that relationship. Linear models are fast and stable to compute, but can be limited if the true relationship being approximated is non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some basic thoughts/feedback on your questions:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Can Linear Regression work for such datasets? &lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes it can work. Will it be good enough? You will know after testing.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are other models I can use?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Almost any regression model class could be applied to this problem. For a quick comparison, to see if a non-linear model will make more accurate predictions for you, then you could try an easy-to-apply model from an existing library, &lt;a href=&quot;https://github.com/dmlc/xgboost/tree/master/demo/regression&quot; rel=&quot;noreferrer&quot;&gt;such as XGBoost&lt;/a&gt; (which has a stand-alone command-line version).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For a linear model, your intuition is correct that one feature is redundant, as it is itself a linear combination of the other features, so there is no impact to removing it. You could remove any single column. You should not need to though, and some model classes might work better if left in. As always, experiment if you are not sure, and use hold-out test data to verify your ideas (or even better a separate cross-validation set before final test stage)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;3&quot;&gt;&#xA;  &lt;li&gt;As I use Linear Regression all independent variables ($x_{i,0}$, $x_{i,d}$)'s are &amp;lt; 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $W$ for $\hat{y}_i = W\mathbf{x}_i$ calculated would be very large)? &lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For linear regression this scaling is not very important. However, there are a few caveats:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you measure success by an error metric (such as mean square error), then to compare different ideas fairly, you should report the test error in the same way - e.g. you should scale back the $log(y)$-based predictions and calculate error on the same $(y - \hat{y})^2$ each time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For some model classes, they could work better if the target variable is within a certain range. You should check documentation for this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How useful the mapping is depends on the true nature of the relationship between $\mathbf{x}$ and $y$. Specifically for a linear model, you have completely changed the line you are trying to fit - it could as a result be a lot better or lot worse, but that is not dependent just on $\mathbf{x}$. So again you will have to give it a try and test the results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-26T16:13:05.763" LastActivityDate="2017-06-26T16:13:05.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19975" PostTypeId="2" ParentId="989" CreationDate="2017-06-26T15:01:54.220" Score="11" Body="&lt;p&gt;Did you include scaling in your pre-processing step? I had this issue when running my SVM. My dataset is ~780,000 samples (row) with 20 features (col). My training set is ~235k samples. It turns out that I just forgot to scale my data! If this is the case, try adding this bit to your code:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;scale data to [-1,1] ; increase SVM speed:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import MinMaxScaler&#xA;scaling = MinMaxScaler(feature_range=(-1,1)).fit(X_train)&#xA;X_train = scaling.transform(X_train)&#xA;X_test = scaling.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33854" LastEditorUserId="98307" LastEditDate="2020-08-02T12:49:50.630" LastActivityDate="2020-08-02T12:49:50.630" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="19976" PostTypeId="1" CreationDate="2017-06-26T15:31:43.870" Score="2" ViewCount="2818" Body="&lt;p&gt;&lt;strong&gt;Objective&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to visualize a typical customer's journey using Python or R. As usual, customers buy different products at different points of time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I checked every customers journey and chose &lt;strong&gt;product A&lt;/strong&gt; as a starting point. 10,000 customers bought &lt;strong&gt;product A&lt;/strong&gt; as their first product. Now I want to show the &quot;flow&quot; of their  journey. After 10 minutes, 2,500 of these 10,000 customers,  bought &lt;strong&gt;product B&lt;/strong&gt; as their second product. &lt;strong&gt;Product C&lt;/strong&gt; was bought by 7,500 people after 25 minutes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a more complex sankey chart. You start of with a bar chart with the height of 10,000 on the left. There are two bar charts on the right and you see a connection between the left and right charts. The longer it took the customers to buy the next product, the further away are the bars on the right from the left one. All the bars have titles with their product names. Quick infos on the charts would also be nice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final chart would visualize a longer customer journey (more than two products). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have tried&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the networkD3 R package to visualize a sankeyNetwork, however I could not place the bar charts according to my desires. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any recommendations for Python or R packages, or is there a way to draw it by myself (in a reasonable amount of time)?   &lt;/p&gt;&#xA;" OwnerUserId="26687" LastActivityDate="2017-10-30T20:07:08.720" Title="Visualizing Customer journeys" Tags="|python|r|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19977" PostTypeId="1" CreationDate="2017-06-26T19:13:59.323" Score="3" ViewCount="755" Body="&lt;p&gt;&lt;strong&gt;Situation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be a data scientist, but I have no programming or maths background.&#xA;I have more than the basic understanding of python which I taught myself over a couple of months. Right now I am stranded, because I do not know how to frame a study plan?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working in a marketing firm with lots of data, so I have an income for the time that I study and I have data to apply my knew knowledge on. I am not trying to rush up and want to build up a strong base and grow upon it over a longer period of time. I know it is not going to be easy, but I feel strongly motivated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you point out comprehensive learning plans with references to books, courses, and learning materials for self-teaching beginners? Could you also point out any articles or presentations in which former beginners and transitioners share their experiences about successfully becoming data scientists?&lt;/p&gt;&#xA;" OwnerUserId="33868" LastEditorUserId="42142" LastEditDate="2017-11-30T19:50:41.287" LastActivityDate="2017-11-30T19:50:41.287" Title="Is there a comprehensive learning plan for a self-teaching beginner in data science with no programming background?" Tags="|beginner|career|" AnswerCount="1" CommentCount="3" ClosedDate="2017-06-26T22:15:54.990" ContentLicense="CC BY-SA 3.0" />
  <row Id="19979" PostTypeId="2" ParentId="19977" CreationDate="2017-06-26T20:36:42.573" Score="4" Body="&lt;h1&gt;Short Answer:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Practice.  You already see it's both math and programming, so start practicing your math and your programming and eventually you'll be there.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Long Answer:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Data science happens at the intersection of math and programming.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of people start with some intense math (biotech, physics, engineering, or if you're lucky statistics) and thus have to learn the programming from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of others start with being some professional developer, and then have to learn the math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as you said, you know neither.  So check your motivations.  Do you really want to be a data scientist, or are you just caught up in the buzz that still surrounds it?  You really have to love it to endure the massive learning curve.  And if you love it, figure out why you're only now getting started.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alright.  Still on board?  If so, you really have three paths as I see it.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;you can follow along with some mooc or series of tutorials to spoon feed you some basic project.  It is a start.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can first try to break into being an analyst first.  Could be a much shorter path.  Best case scenario, analysts can turn directly into data scientist.  Worst case scenario, you get stuck not quite living your dream.  If you take this route with the intention of eventually becoming a data scientist, be verrry wary of the tools you use.  Highly automated analyst tools keep you locked into what the tools can do.  And if the tools don't have any statistics or machine learning baked in, then you'll get stuck.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The hard way.  If you're serious about it, then know you need to learn both overlapping languages - math and programming.  Ask around and do your research.  Find what programming languages will work best in the niche you expect you'll most like.  A part of you &quot;do I really want to be a data scientist?&quot; should includes the types of things you want to do as a data scientist.  Out of your answer, pull a few examples of actual projects.  Of those projects, pick the easiest and learn the math behind it.  Like financial stuff?  Perhaps some simple time series prediction on some open financial data.  Like marketing?  Perhaps start with looking up A/B tests.  Yes, this one may include also doing #1, but with the very important extra layer of sincerity around why you're doing it.  And a LOT of the job of a data scientist is being able to take a question nobody's asked yet, and figure out how to apply your tools to solve it.  Many moocs rob you of that most-necessary layer of first framing the problem before trying to solve it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2017-06-26T20:36:42.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19980" PostTypeId="1" CreationDate="2017-06-26T21:26:04.680" Score="16" ViewCount="14792" Body="&lt;p&gt;I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My colleague (who is a machine learning expert) said that, in his opinion, the arguably most important practical aspect of machine learning is &lt;strong&gt;how to understand whether you have collected enough data to train your machine learning model&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This statement surprised me, because I had never given that much importance to this aspect...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then looked for more information on the internet, and I found this post on &lt;a href=&quot;http://fastml.com/how-much-data-is-enough/&quot; rel=&quot;noreferrer&quot;&gt;FastML.com&lt;/a&gt; reporting as rule of thumb that you need roughly &lt;strong&gt;10 times as many data instances as there are features&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Is this issue really particularly &lt;strong&gt;relevant&lt;/strong&gt; in machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - &lt;strong&gt;Is the 10 times rule working?&lt;/strong&gt; Are there any other relevant sources for this theme?&lt;/p&gt;&#xA;" OwnerUserId="17305" LastEditorUserId="21" LastEditDate="2017-07-07T14:43:27.587" LastActivityDate="2019-11-11T17:11:34.960" Title="How much data are sufficient to train my machine learning model?" Tags="|machine-learning|data-mining|dataset|data-cleaning|data|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="19981" PostTypeId="2" ParentId="19980" CreationDate="2017-06-26T23:39:47.647" Score="6" Body="&lt;ol&gt;&#xA;&lt;li&gt;Yes, the issue is certainly relevant, since your ability to fit the model will depend on the amount of data you have, but more importantly, it depends on the quality of the predictors. &lt;/li&gt;&#xA;&lt;li&gt;A 10-times rule might be a rule of thumb (and there are many others), but it really depends on the predictive utility of your features.  E.g., the iris dataset is fairly small but easily solved, because the features yield good separation of the targets.  Conversely, you could have 10 million examples and fail to fit if the features are weak. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-06-26T23:39:47.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19983" PostTypeId="1" CreationDate="2017-06-27T00:49:23.143" Score="-1" ViewCount="111" Body="&lt;p&gt;Given a word I want to find the closely related words (e.g., finding correlations). Can you please suggest accurate tools to start with&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2017-06-27T04:16:42.707" Title="Most closely related words to a certain word" Tags="|machine-learning|data-mining|statistics|correlation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19984" PostTypeId="1" CreationDate="2017-06-27T02:14:16.700" Score="1" ViewCount="27" Body="&lt;p&gt;I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2017-06-27T02:14:16.700" Title="E-Learning related terminology" Tags="|machine-learning|data-mining|dataset|data-cleaning|data|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19985" PostTypeId="2" ParentId="19983" CreationDate="2017-06-27T04:16:42.707" Score="1" Body="&lt;p&gt;Please refine your question.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a pre-trained word2vec model e.g gensim, you can compute the cosine similarity between the vector representations of words. &lt;/p&gt;&#xA;" OwnerUserId="33781" LastActivityDate="2017-06-27T04:16:42.707" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19986" PostTypeId="2" ParentId="19972" CreationDate="2017-06-27T05:27:11.197" Score="1" Body="&lt;p&gt;Open Calais is a free-to-use tool for entity recognition and relationship mapping. It's from Thompson-Reuters so may not be wholly suitable for technical language but worth a try. Has python bindings&lt;/p&gt;&#xA;" OwnerUserId="33878" LastActivityDate="2017-06-27T05:27:11.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19987" PostTypeId="1" CreationDate="2017-06-27T07:08:55.867" Score="2" ViewCount="1586" Body="&lt;p&gt;I have around 9 string features which I have indexed using string indexer and used vector assembler to get the feature vector and used a normalizer to normalize across features . These are the transformation I have made and used naive Bayes classifier as the estimator , all the above operations after enclosed in a pipeline model. I trained on around 3 lakh samples and tested on 20k samples  the model predicted same label for all of them. I handpicked the features is that the problem? Or any tuning parameters available ? I found out a smoothing parameter which I set it to 1.0 should I increased or decrease to improve the model accuracy? Please help&lt;/p&gt;&#xA;" OwnerUserId="33460" LastEditorUserId="33460" LastEditDate="2017-06-28T05:06:49.113" LastActivityDate="2017-07-30T11:59:07.697" Title="How to improve naive Bayes multiclass classification accuracy?" Tags="|machine-learning|apache-spark|multiclass-classification|naive-bayes-classifier|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19988" PostTypeId="1" CreationDate="2017-06-27T07:12:22.333" Score="0" ViewCount="1293" Body="&lt;p&gt;This is the first time I am using pandas and iPython notebook and was not able to figure out the correct search terms for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a &lt;code&gt;.xls&lt;/code&gt; file for compile time data for 3 build-servers located at 3 sites &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt; and &lt;code&gt;C&lt;/code&gt;. These build servers compile multiple projects, so i will pick any specific project. Hence I need to plot data like this (for a specific project - not all in one graph, to keep it simple): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X-axis = date&#xA;Y-axis = average build time on that date&#xA;&#xA;3 lines for sites A, B and C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I have done so far : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import matplotlib as plt &#xA;&#xA;file=  r'/home/abc/Downloads/request.xls'&#xA;df = pd.read_excel(file,parse_dates=['Date'])&#xA;&#xA;build_times = df[['Date','site','project','Duration']]&#xA;build_group = build_times.groupby(['Date','site','project']).mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need help on following : &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;how i select only successful builds&#xA;if there is a column &lt;code&gt;status&lt;/code&gt; with 0 and 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to plot the lines for sites &lt;code&gt;A&lt;/code&gt;,&lt;code&gt;B&lt;/code&gt; and &lt;code&gt;C&lt;/code&gt; (for specific project) with above mentioned X and Y axes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33881" LastEditorUserId="55122" LastEditDate="2021-02-26T15:43:37.223" LastActivityDate="2021-02-26T15:43:37.223" Title="How to group data and plot line graphs" Tags="|pandas|visualization|ipython|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19989" PostTypeId="2" ParentId="19951" CreationDate="2017-06-27T07:20:38.923" Score="2" Body="&lt;p&gt;There is unlikely to be any useful pattern analysis for this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot prove it, but I think it highly likely that the raindrops are being generated using a pseudo-random process. Even if they are not, you have been given more than enough information in what your controller can &quot;sense&quot; in this simulation, so that predicting future raindrops is not necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, this is a problem of optimal control and planning. In machine learning, often reinforcement learning can be used to solve problems like this - potentially &lt;a href=&quot;https://en.wikipedia.org/wiki/Q-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Q-Learning&lt;/a&gt; would work here, but it would be quite hard to implement it so that it learned quickly due to the large number of states. You would need to use a function approximator like a neural network, and although that can be made to work nicely here, it does not seem necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think you need any Data Science technique here, this is closer to more traditional &lt;a href=&quot;https://en.wikipedia.org/wiki/Automated_planning_and_scheduling&quot; rel=&quot;nofollow noreferrer&quot;&gt;AI planning&lt;/a&gt;. However, there are strong links between planning and learning models, so what I am going to suggest is also something you might see in game-playing systems that use a combination of machine learning (reinforcement learning) and planning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick analysis shows that you have a large number of visible states ($2^{42}$) which would take a long time to produce optimal rules for using the simplest reinforcement learning algorithms - although it is feasible. However, you also have perfect knowledge of the dynamics and a fully deterministic system within what you can see. In addition, the branching factor of your action decisions is not high, just 3 (and sometimes 2) per step - so for instance looking 8 steps ahead will involve checking under 10,000 scenarios. That seems possible to do on every time step, which immediately suggests a simple search-based planning algorithm - similar to &lt;a href=&quot;https://en.wikipedia.org/wiki/Monte_Carlo_tree_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;Monte Carlo Tree Search&lt;/a&gt;, except in your case you can just brute-force all combinations up to an arbitrary horizon. As long as the horizon is far enough ahead, you can achieve optimal results this way, because it only takes 3 steps to fully traverse the different positions the collector can be in.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;On each time step:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Observe the current state and position of incoming raindrops&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;One by one, generate feasible chains of actions, working through all possibilities, some number of steps ahead (I'm suggesting 8 steps, but wonder if you can get away with less, e.g. just 5).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If the sequence of actions would take the controller outside of the allowed area, discard it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Score the sequence of actions by counting how many raindrops you predict it will collect, based on the information about raindrops at each timestep.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Keep a record of the best sequence of actions so far.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the best sequence of actions so far, choose the first action in that sequence, and take that action &quot;for real&quot; in the simulation, advancing by one time step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Discard the rest of the sequence, even though your controller really might take that path, ready for next phase of planning. In the simplest cases you don't revise the plan using clever tree pruning techniques, you just brute force another search after each update. A more advanced algorithm may be able to preserve the most likely sequences to go forward to next time step, and save computation (at the cost of complexity and memory used to store candidates).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-06-27T11:59:09.860" LastActivityDate="2017-06-27T11:59:09.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19990" PostTypeId="1" AcceptedAnswerId="19994" CreationDate="2017-06-27T07:31:03.580" Score="1" ViewCount="46" Body="&lt;p&gt;Let say I have a data set where every sample is an image of a landscape and a temperature associated with the landscape. How do I incorporate the temperature into my convolutional neural network for classifying if the data is e.g a winter or summer landscape? Can I simply add the temperature as a feature after the feature learning part of the network? I can not find similar questions but maybe this has a name that I am not aware of?&lt;/p&gt;&#xA;" OwnerUserId="33884" LastEditorUserId="29169" LastEditDate="2021-02-16T00:45:35.730" LastActivityDate="2021-02-16T00:45:35.730" Title="How can I deal with data that is on the format &quot;Image + single number&quot;?" Tags="|machine-learning|feature-selection|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="19991" PostTypeId="1" AcceptedAnswerId="19999" CreationDate="2017-06-27T07:47:35.093" Score="3" ViewCount="1529" Body="&lt;p&gt;I have set of documents where I have assigned topics per each document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., Topics of document 1 -&gt; 1.0 Science, 1.0 politics, 0.8 History, 0. 8 Information and Technology&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to cluster these documents and find what are the documents that share similar like topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please suggest/recommend suitable clustering technique for me to start with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I am interested in something like this -&gt; &lt;a href=&quot;https://www.quora.com/How-do-I-cluster-documents-using-topic-models&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-do-I-cluster-documents-using-topic-models&lt;/a&gt;&#xA;However, the topics assigned for my documents varies&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-06-27T09:55:32.310" LastActivityDate="2017-06-27T18:27:12.490" Title="Cluster documents based on topic similarity" Tags="|machine-learning|classification|data-mining|clustering|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19994" PostTypeId="2" ParentId="19990" CreationDate="2017-06-27T11:09:51.427" Score="1" Body="&lt;p&gt;After the convolutional part you will need to add a normal, dense layer. Concatenate it to this layer and add some more layers if necessary, to add more interactions between the temperature and the image. This wouldn't necessarily need to be too deep because it can learn to represent the image features in a way that it will combine nicely with the temperature already, hopefully.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-06-27T11:09:51.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19995" PostTypeId="1" AcceptedAnswerId="20008" CreationDate="2017-06-27T12:14:09.267" Score="2" ViewCount="387" Body="&lt;p&gt;In stacked generalization, several algorithms are trained on the training set (i.e. at layer 1) and their predictions are then stacked using a layer 2 model. In many documentations, it is said that it is better that the layer 1 algorithms should be of low correlation. How can one compute this correlation between algorithms ? &lt;/p&gt;&#xA;" OwnerUserId="33386" LastActivityDate="2017-06-27T20:27:51.520" Title="How to measure the correlation of different algorithms" Tags="|machine-learning|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="19997" PostTypeId="1" CreationDate="2017-06-27T14:23:57.873" Score="12" ViewCount="4516" Body="&lt;p&gt;I've read &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolution&quot; rel=&quot;noreferrer&quot;&gt;explanation of convolution&lt;/a&gt; and understand it to some extent. Can somebody help me understand how this operation relates to convolution in Convolutional Neural Nets? Is filter like function &lt;code&gt;g&lt;/code&gt; which applies weight?&lt;/p&gt;&#xA;" OwnerUserId="32989" LastEditorUserId="28175" LastEditDate="2018-04-17T23:24:30.040" LastActivityDate="2018-05-18T02:38:32.573" Title="Relation between convolution in math and CNN" Tags="|machine-learning|neural-network|deep-learning|cnn|convolution|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="19998" PostTypeId="1" CreationDate="2017-06-27T14:42:33.113" Score="1" ViewCount="749" Body="&lt;p&gt;I also asked here: &lt;a href=&quot;https://stats.stackexchange.com/questions/287587/reason-of-getting-only-one-node-while-implementing-classification-tree-in-matlab&quot;&gt;https://stats.stackexchange.com/questions/287587/reason-of-getting-only-one-node-while-implementing-classification-tree-in-matlab&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hello, I have $N=1520=19*80$ samples, each of which are $p=1764$-dimensional. So $N &amp;lt; p$ here. I want to perform a $19$-class classification of these samples. The training data consists of $80$ samples from each of the $19$ classes. I performed classification tree in matlab (sorry but this is the only language I know well) using fitctree$(X,Y)$ where $X=$matrix of covariates of size $N \times p$, $Y=$ vector of responses of size $N \times 1$.I'm getting only one node each time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to see if the problem solves when I reduce the number of features significantly, I tried to take the first $100$ features by $X=X(:, 1:100)$, but still got only one node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, my knn and naive Bayes classifiers are also predicting all samples to be in one class, just like the tree above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be the problem, and how could I possibly solve it? Am I making a syntactical mistake?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot in advance!&lt;/p&gt;&#xA;" OwnerUserId="33198" LastEditorUserId="33198" LastEditDate="2017-06-28T12:01:28.570" LastActivityDate="2017-06-28T12:01:28.570" Title="Why is my classification tree is showing only one node, and how to resolve this issue?" Tags="|random-forest|decision-trees|multiclass-classification|matlab|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="19999" PostTypeId="2" ParentId="19991" CreationDate="2017-06-27T15:17:31.627" Score="3" Body="&lt;p&gt;As @Emre suggested, if you already have the distribution of topics in each document, you can represent each document as a vector $x_d \in \mathbb{R}^N$, where $N$ is the number of the unique topics in your collection. For documents, not exhibiting specific topics just fill the specific cells in each feature vector with zeros. Then, you can use some clustering algorithm such as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html#sklearn.neighbors.NearestNeighbors&quot; rel=&quot;nofollow noreferrer&quot;&gt;nearest neigbors&lt;/a&gt;, using those feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example usage code in python below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.metrics import pairwise_distances&#xA;&#xA;# Initialize some documents&#xA;doc1 = {'Science':0.7, 'History':0.05, 'Politics':0.15, 'Sports':0.1}&#xA;doc2 = {'News':0.3, 'Art':0.5, 'Politics':0.1, 'Sports':0.1}&#xA;doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}&#xA;doc4 = {'Science':0.2, 'Weather':0.2, 'Art':0.6, 'Sports':0.1}&#xA;collection = [doc1, doc2, doc3, doc4]&#xA;df = pd.DataFrame(collection)&#xA;# Fill missing values with zeros&#xA;df.fillna(0, inplace=True)&#xA;# Get Feature Vectors&#xA;feature_matrix = df.as_matrix()&#xA;&#xA;# Get cosine similarity (i.e. 1 - cosine_distance) between pairs&#xA;sims = 1-pairwise_distances(feature_matrix, metric='cosine')&#xA;&#xA;# Get the ranking of the documents given document 0&#xA;# from most similar to least similar. Don't take into account&#xA;# the first document, because it will be the same that the query was about&#xA;ranking_1 = np.argsort(sims[0,:])[::-1][1:2]&#xA;print ranking_1&#xA;&#xA;ranking_2 = np.argsort(sims[1,:])[::-1][1:2]&#xA;print ranking_2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This takes 4 documents with $N=7$ unique topics, fills missing values with zeros and creates a similarity matrix between all documents. Then querying for documents 1(Science=0.7) and 2(Art:0.5) the most similar other document in the collection, we surely get documents 3(Science:0.8) and 4(Art:0.6) correspondingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try more sophisticated approaches regarding clustering and other &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;distance metrics&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-06-27T15:17:31.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20000" PostTypeId="2" ParentId="19995" CreationDate="2017-06-27T15:52:11.793" Score="2" Body="&lt;p&gt;I'm not aware of a simple way to compare. I've more read that you want a diverse set across different types of algorithms to reduce &quot;group think&quot;, so you would choose an SVM, a NN, a decision tree, etc. If at layer 2 they tend to vote together, that either means they were all fooled for the same reason, or you found predictable data.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-27T15:52:11.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20001" PostTypeId="1" CreationDate="2017-06-27T17:35:47.610" Score="-1" ViewCount="153" Body="&lt;p&gt;I am trying to create a model that predicts / classifies the response variable with an input space of size &lt;code&gt;n x 1&lt;/code&gt;, which is essentially a single feature. To be more precise, I have to predict which class in among 0~9 (y) would it be according to a probability value (X).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, I am thinking that y and X will have negative correlation. However, I cannot really think of a model that would perform well under the condition provided above since there is only one feature to consider when creating a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried XGBoost and RandomForest, but I feel like I used those models mostly as blackboxes. Please do suggest a fine machine learning model to conduct such multi-class classification with input space of size &lt;code&gt;n x 1&lt;/code&gt;. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="33905" LastActivityDate="2017-07-27T19:03:39.013" Title="Multi-class Classification Task with Input space size n x 1" Tags="|machine-learning|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20002" PostTypeId="2" ParentId="19869" CreationDate="2017-06-27T17:52:31.383" Score="6" Body="&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first matrix represents the input vector in one hot format&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second matrix represents the synaptic weights from the input layer neurons to the hidden layer neurons&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer Version&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;what the feature matrix exactly is&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It seems you have not understood the representation correctly.&#xA;That matrix is not a feature matrix but a weight matrix for the neural network.&#xA;Consider the image given below. Especially notice the left top corner where the Input Layer matrix is multiplied with the Weight matrix.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yn1my.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yn1my.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now look at the top right.&#xA;This matrix multiplication InputLayer dot-producted with Weights Transpose is just a handy way to represent the neural network at the top right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to answer your question, the equation you have posted is just the mathematical representation for the neural network which is used in the Word2Vec algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first part, [0 0 0 1 0 ... 0] represents the input word as a one hot vector and the other matrix represents the weight for the connection of each of the input layer neurons to the hidden layer neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Word2Vec trains, it backpropogates into these weights and changes them to give better representations of words as vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once training is complete, you use only this weight matrix, take [0 0 1 0 0 ... 0] for say 'dog' and multiply it with the improved weight matrix to get the vector representation of 'dog' in a dimension = no of hidden layer neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the diagram you have presented, the number of hidden layer neurons is 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the right hand side is basically the word vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image Credits : &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/matrix-multiplication-in-neural-networks&quot; rel=&quot;noreferrer&quot;&gt;http://www.datasciencecentral.com/profiles/blogs/matrix-multiplication-in-neural-networks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-06-27T17:52:31.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20004" PostTypeId="2" ParentId="19991" CreationDate="2017-06-27T18:27:12.490" Score="3" Body="&lt;h3&gt;Topics are clusters&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There is next to no difference between subspace clustering and topic modeling, except maybe that text is sparse and integer while subspace clusterers usually assume dense and continuous data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So rather than trying to cluster &lt;em&gt;again&lt;/em&gt;, just use your topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, documents can belong to multiple topics. That is because text usually is this way, and forcing everything to have a unique label reduces the quality, because it cannot reflect reality anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you insist every document should have a unique cluster, just use argmax(topic weights).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-06-27T18:27:12.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20005" PostTypeId="2" ParentId="20001" CreationDate="2017-06-27T18:33:56.790" Score="0" Body="&lt;p&gt;The method to use depends on the problem at hand. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is your data linearly separable? In the case of a single feature, this corresponds to all instances of a class being within a lower and an upper bound. If this is the case, then a simple model like logistic regression should do the trick.&lt;/li&gt;&#xA;&lt;li&gt;If your data is not linearly separable, then a decision tree could be enough. The optimization would take care of finding the optimal cuts in the values of the feature to split the data into the different classes.&lt;/li&gt;&#xA;&lt;li&gt;Another option for non-linearly separable data is to use an SVM with some kernel that makes your data linearly separable.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;More complex methods are designed to capture patterns in the data when multiple features are available, and as such shouldn't be expected to perform well under these conditions.&lt;/p&gt;&#xA;" OwnerUserId="33852" LastActivityDate="2017-06-27T18:33:56.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20006" PostTypeId="1" CreationDate="2017-06-27T19:48:44.913" Score="0" ViewCount="2313" Body="&lt;p&gt;I'm building an NLP question-answering application using Doc2Vec technique in &lt;code&gt;gensim&lt;/code&gt; package of Python. My training questions is very small, only 20 documents and I am getting very inaccurate and different similarities even for same document while running at multiple instances. Almost all the sources which I referred trained data set containing thousands of documents. So I infer the reason behind my model's inaccuracy is the size of my data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to improve the similarity between documents, maybe by changing parameters or feature engineering? If yes, what are those parameters and by what ratio should I change them? If no, what are other ways or perhaps other neural network models to tackle the problem?&lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-08-27T12:38:58.167" Title="How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set?" Tags="|neural-network|nlp|word2vec|gensim|similar-documents|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20008" PostTypeId="2" ParentId="19995" CreationDate="2017-06-27T20:27:51.520" Score="3" Body="&lt;p&gt;For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass classification you can find correlation between predicted factor variables using the &lt;code&gt;hetcor&lt;/code&gt; package in R&lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-06-27T20:27:51.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20011" PostTypeId="1" CreationDate="2017-06-28T00:49:26.380" Score="1" ViewCount="66" Body="&lt;p&gt;I have a bunch of time series data I'm doing classification on. I used TPOT with a custom cv (walkforward time series split) to find the best performing model. Logistic regression was selected. I'm a bit surprised by this, given that most kaggle competitions are won by ensembles of newer models like XGB, etc. In fact, not only was logistic regression the best model but it was actually the only one that learned something meaningful (outperformed the others like RF, XGB, etc. by &gt; 5% where 60% is the best accuracy it achieved).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My expertise is mostly in deep networks, not as much the stuff dealt with by sklearn, but in this case the training data is very limited and quite noisy. What should I take away from logistic regression being the best classifier? Can I use this knowledge to inform my design of other classifiers that could perform better? Again, I have a lot less experience with this side of ML, so I'm really looking for an intuition as to how I can take this knowledge and use apply it to build better models or ensembles.&lt;/p&gt;&#xA;" OwnerUserId="33921" LastActivityDate="2017-06-28T00:49:26.380" Title="Intuition for Logistic Regression Performance" Tags="|time-series|logistic-regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20012" PostTypeId="1" AcceptedAnswerId="22511" CreationDate="2017-06-28T01:11:05.907" Score="13" ViewCount="12172" Body="&lt;p&gt;Batch Normalization is described in &lt;a href=&quot;https://arxiv.org/abs/1502.03167&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; paper as a normalization of the input to an activation function with scale and shift variables $\gamma$ and $\beta$. This paper mainly describes using the sigmoid activation function, which makes sense. However, it seems to me that feeding an input from the normalized distribution produced by the batch normalization into a ReLU activation function of $max(0,x)$ is risky if $\beta$ does not learn to shift most of the inputs past 0 such that the ReLU isn't losing input information. I.e. if the input to the ReLU were just standard normalized, we would lose a lot of our information below 0. Is there any guarantee or initialization of $\beta$ that will guarantee that we don't lose this information? Am I missing something with how the operation of BN and ReLU work?&lt;/p&gt;&#xA;" OwnerUserId="33922" LastEditorUserId="836" LastEditDate="2017-08-22T18:06:20.433" LastActivityDate="2021-07-05T22:54:06.807" Title="Does Batch Normalization make sense for a ReLU activation function?" Tags="|machine-learning|neural-network|deep-learning|batch-normalization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20013" PostTypeId="1" AcceptedAnswerId="20065" CreationDate="2017-06-28T01:47:14.527" Score="1" ViewCount="39" Body="&lt;p&gt;I have set of topics as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;web based&quot;, &quot;web-based&quot; -&gt; &lt;strong&gt;with surplus symbols&lt;/strong&gt;&#xA;&quot;technology&quot;,&quot;technologies&quot; -&gt; &lt;strong&gt;with singular and plural&lt;/strong&gt;&#xA;&quot;learned&quot;, &quot;learnt&quot;, &quot;learning&quot; -&gt;&lt;strong&gt;suffix stripping&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please recommend an accurate tool to perform aforementioned tasks.&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2017-06-29T16:38:39.030" Title="Techniques to clean the topics" Tags="|data-mining|dataset|bigdata|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20016" PostTypeId="1" CreationDate="2017-06-28T07:22:57.710" Score="0" ViewCount="2009" Body="&lt;p&gt;I am searching for Fashion Clothing image dataset where each image is associated with description. Basically description should contains all information about cloth. like -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;image_01.jpg - red shirt has collar with long-sleeves.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any good source to collect these type of dataset ?? &lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-11-05T21:33:02.867" Title="Fashion Clothing images Dataset with image description" Tags="|dataset|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20019" PostTypeId="1" CreationDate="2017-06-28T08:46:51.793" Score="1" ViewCount="6500" Body="&lt;p&gt;I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering specific rows, I would like to visualize it by plotting 2 columns (latitude and longitude) using matplotlib.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I've done so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;import packages:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql import SparkSession&#xA;from pyspark.conf import SparkConf&#xA;from pyspark.sql.functions import *&#xA;import matplotlib.pyplot as plt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;build spark session:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;spark = SparkSession.builder \&#xA;     .master(&quot;master:IP&quot;) \&#xA;     .appName(&quot;plot track&quot;) \&#xA;     .config(conf=SparkConf()) \&#xA;     .getOrCreate()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;read .csv file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = spark.read.csv('data.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;filter rows according to values in a specific column (&quot;_c1&quot;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;one_ID = df.filter(&quot;_c1 == 1234&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;save column data? (Latitude data is in column &quot;_c3&quot; and longitude data is in column &quot;_c4&quot;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;latitude = one_ID.rdd.map(lambda y: y._c3)&#xA;longitude = one_ID.rdd.map(lambda x: x._c4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot(longitude,latitude, 'o-')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I thought this would work but it does not. I get the error -&#xA;- 'TypeError: float() argument must be a string or a number'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="33941" LastActivityDate="2017-11-07T10:00:30.333" Title="Plot RDD data using a pyspark dataframe from csv file" Tags="|dataset|bigdata|apache-spark|pyspark|plotting|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-10T16:30:03.063" ContentLicense="CC BY-SA 3.0" />
  <row Id="20020" PostTypeId="2" ParentId="19987" CreationDate="2017-06-28T09:06:02.693" Score="0" Body="&lt;p&gt;I think that you should read more on the theory of Naive Bayes classifier (&lt;a href=&quot;http://scikit-learn.org/stable/modules/naive_bayes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/naive_bayes.html&lt;/a&gt;) as it is very crucial that you need to choose the right likelihood distribution, $P(x_i|y)$. If it is hard to choose the right likelihood distribution, you may just try to use a few different distributions to see if the result can be improved.&lt;/p&gt;&#xA;" OwnerUserId="33877" LastEditorUserId="33877" LastEditDate="2017-06-28T09:14:47.930" LastActivityDate="2017-06-28T09:14:47.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20021" PostTypeId="2" ParentId="8775" CreationDate="2017-06-28T09:48:34.990" Score="0" Body="&lt;p&gt;You can explore Joint modeling for Longitudinal and time to event data. Here survival model will be built for rare events and the linear models for longitudinal data. &lt;/p&gt;&#xA;" OwnerUserId="32906" LastActivityDate="2017-06-28T09:48:34.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20023" PostTypeId="2" ParentId="18528" CreationDate="2017-06-28T10:38:49.420" Score="1" Body="&lt;p&gt;It sounds like you're describing using a &lt;strong&gt;gazetteer&lt;/strong&gt; and training, which is not particularly unusual. See &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How do we go about identifying named entities? One option would be to look up each word in an appropriate list of names. For example, in the case of locations, we could use a gazetteer, or geographical dictionary, such as the Alexandria Gazetteer or the Getty Gazetteer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using a gazetteer has tradeoffs; see &lt;a href=&quot;https://datascience.stackexchange.com/questions/9950/nlp-is-gazetteer-a-cheat&quot;&gt;here&lt;/a&gt; for some discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/33195322/how-to-represent-gazetteers-or-dictionaries-as-features-in-crf&quot;&gt;Here&lt;/a&gt;'s an example of someone using it in CRF++ as a feature.&lt;/p&gt;&#xA;" OwnerUserId="15496" LastActivityDate="2017-06-28T10:38:49.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20024" PostTypeId="1" AcceptedAnswerId="20046" CreationDate="2017-06-28T11:01:06.213" Score="5" ViewCount="2759" Body="&lt;p&gt;Possibly similar question: &lt;a href=&quot;https://datascience.stackexchange.com/questions/16858/is-it-ok-to-collect-data-using-algorithm-to-train-another&quot;&gt;Is it ok to collect data using algorithm to train another?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a model that accurately describes an underlying physical, complex, system. The model is basically a set of ODEs based on the physics of the system, validated against measurements. When a system perturbation occurs, I can run thousands of simulations to assess if the new system status is secure or not. This is basically a classification procedure (yes/no). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This procedure is &lt;strong&gt;very&lt;/strong&gt; time consuming and has to be performed in real-time (thus, requiring huge computational resources). There are thousands of possible perturbations and infinite number of initial points. The same perturbation from 1 initial point can lead to stable system, while from another, to an unstable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)? Any references to such examples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologise if this is a basic question. I am new to data science techniques with a more physics/engineering background.&lt;/p&gt;&#xA;" OwnerUserId="33948" LastActivityDate="2017-06-28T20:34:20.060" Title="Using simulations to train ML algorithms" Tags="|classification|anomaly-detection|simulation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20025" PostTypeId="1" CreationDate="2017-06-28T11:08:06.307" Score="2" ViewCount="751" Body="&lt;p&gt;I have a clinical trial dataset where the patient details are recorded at certain time intervals(visits) over period of time. Each visit will contain data recordings of all the adverse events experienced prior to the visit, any medicines taken and the dosage of experimental drug in no particular order. for example a visit could have recordings of 5 adverse events (nausea, headache, hypertension etc.) experienced in the week prior to the visit, dosage of experimental drug and some medicines given to avoid the adverse events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now i need to predict the occurrence of adverse events for a patient before his next visit and also need to know which medicines cause some of these adverse effects.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I've clearly explained the problem. What Machine learning/Statistical methods would best solve this?&lt;/p&gt;&#xA;" OwnerUserId="32906" LastActivityDate="2017-09-27T02:14:09.323" Title="Predicting next adverse event based on previous events" Tags="|machine-learning|r|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20026" PostTypeId="1" CreationDate="2017-06-28T11:31:31.937" Score="-1" ViewCount="386" Body="&lt;p&gt;What is the state of the art of transforming input data for neural networks? They need to have constant width, and that's why I'm trying to wrap my head around that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that we want to classify some books (for example, into some categories).&#xA;Books have many attributes of different type, like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;short strings (title)&lt;/li&gt;&#xA;&lt;li&gt;longer strings/documents (description)&lt;/li&gt;&#xA;&lt;li&gt;dates (publishing date, author's birth date)&lt;/li&gt;&#xA;&lt;li&gt;simple arrays (authors)&lt;/li&gt;&#xA;&lt;li&gt;longitude/latitude (place where the book was finished, author's birth place)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How can one handle these attributes? I've read already a little about handling long strings &lt;a href=&quot;https://datascience.stackexchange.com/questions/869/neural-network-parse-string-data&quot;&gt;here&lt;/a&gt;, but the rest, in especially small arrays of attributes are a mystery for me.&lt;/p&gt;&#xA;" OwnerUserId="33947" LastActivityDate="2017-06-28T16:26:47.540" Title="Transforming the input data for neural networks" Tags="|neural-network|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20029" PostTypeId="2" ParentId="20006" CreationDate="2017-06-28T12:01:44.510" Score="1" Body="&lt;p&gt;20 is a very very small number, even performing the following might not help.&#xA;However, you can try changing the parameters such as the window size, and the number of iterations. However, increasing the number of iterations a lot can cause the model to over fit. &#xA;In my case, decreasing the window size helped. &lt;/p&gt;&#xA;" OwnerUserId="33955" LastEditorUserId="33955" LastEditDate="2017-06-28T12:17:49.260" LastActivityDate="2017-06-28T12:17:49.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20030" PostTypeId="1" CreationDate="2017-06-28T12:34:17.797" Score="1" ViewCount="168" Body="&lt;p&gt;I want to construct a python micro service to generate predictions based on inputs from web via an API.&#xA;I have trained a logistic regression model, but need to create a python script to run in AWS lambda or a container with minimum set of libraries.&#xA;How do I see and possibly export the model parameters?&lt;/p&gt;&#xA;" OwnerUserId="33954" LastActivityDate="2018-01-09T17:08:11.487" Title="exporting the model parameters after training and testing" Tags="|machine-learning|orange|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20031" PostTypeId="1" CreationDate="2017-06-28T15:45:43.750" Score="1" ViewCount="264" Body="&lt;p&gt;I am new to Statistics and was reading about Decision errors in Hypothesis Testing. My question is that why is Type-II error an error at all? From what I understand, it arises when we fail to reject a false null hypothesis. When we fail to reject null hypothesis, it simply means that we do not have strong evidence to reject it. We are not making any comment about which of the two hypothesis is true (or false) . Either can be true. We are not saying that the null hypothesis is true. Then, why is such a conclusion called an error?&lt;/p&gt;&#xA;" OwnerUserId="33926" LastActivityDate="2017-08-31T09:58:38.513" Title="Type-II error in Hypothesis Testing" Tags="|statistics|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20032" PostTypeId="1" CreationDate="2017-06-28T15:50:58.170" Score="1" ViewCount="679" Body="&lt;p&gt;in my free time I am working on a little machine learning project with focus on NeuralNetworks. It is all about Chord-Classification from an acoustic guitar through a microphone. So here is what I got so far:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a simple FeedForward-Network (for first) and train it using the Backpropagation algorithm. First I record some sounds of me strumming simple chords (once, no complex strumming). This data is transformed in realtime with the FourierTransformation into a nice spectrum (this is done every 10ms) and saved in a file. After that I load these files containing the information about which chord was played and the spectrum data through time (every 10ms I got a full spectrum). This data then is normalized by the highest amplitude of all spectrum data (of all chord recordings). The now normalized data is used to train a simple neural network using batch training and backpropagation (with momentum). The activation function is the default sigmoid-function. After some epochs (about 30) the accuracy is mostly around 95%, enough for me! So then I export the network into a file, just to keep it stored. Now when I play some chords and want them to be classified, I load the NeuralNetwork. Before I can start playing, I have to calibrate the program for normalization reasons: I play a few chords and the program transforms it into a spectrum just like the training data. But this information is now used to normalize the real time data of me playing chords recorded through the mic. This basically works well. The program does what it should: Classifies the chords I play (mostly) correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So whats my problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't like the way I am normalizing my data, as for now it depends on how loud I play the chords (highest amplitude). The problem is, if I train the network for example with chords played in a mid range volume it has problems classifing me playing very loud. Same problem would be with more quiet playing. Training the Network with loud and quiet data doesn't solve this either as the more quiet data (smaller amplitudes) is kinda getting lost by the loud data (high amplitudes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where you come in: I want to know how I could optimize that. How can I normalize the spectrum data so that my network will classify loud data as well as quiet data correctly. (I know there may be better techniques like LSTM-ReccurentNetworks, but I want this to work properly, before I go onto that topic)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="33963" LastEditorUserId="33963" LastEditDate="2017-06-28T16:45:53.630" LastActivityDate="2017-06-28T16:45:53.630" Title="Audio Spectrum Normalization for NeuralNetwork Classification" Tags="|neural-network|classification|normalization|audio-recognition|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20033" PostTypeId="1" CreationDate="2017-06-28T16:02:51.433" Score="3" ViewCount="23102" Body="&lt;p&gt;I am interested to use multivariate regression with LSTM (Long Short Term Memory).&#xA;As an example, we can take the stock price prediction problem, where the price at time t is based on multiple factors (open price, closed price, etc.). Using this information we need to predict the price for t+1.&#xA;Another example can be the house price prediction (which depends on factors like, age of house, area, no. of beds, no. of baths, etc). Using this information, we can predict the price of a given house.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found many examples where they just used a single variable regression, but I am interested to use multiple features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone tried this and can anyone point out the right direction for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried the following but have no idea if this is the efficient way.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Importing the training set&#xA;training_set = pd.read_csv('Google_Stock_Price_Train.csv').iloc[:,1:3].values#has fields: Date,Open,High,Low,Close,Volume&#xA;# Feature Scaling&#xA;from sklearn.preprocessing import MinMaxScaler&#xA;sc = MinMaxScaler()&#xA;training_set_scaled = sc.fit_transform(training_set)&#xA;X_train = training_set_scaled[0:1257]&#xA;y_train = training_set_scaled[1:1258]&#xA;# Reshaping&#xA;X_train = np.reshape(X_train, (1257, 1, 2))&#xA;X_train, y_train = np.array(X_train), np.array(y_train)&#xA;X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 2))&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, LSTM&#xA;regressor = Sequential()&#xA;regressor.add(LSTM(units = 4, activation = 'sigmoid', input_shape = (None, 2)))&#xA;regressor.add(Dense(units = 2))&#xA;regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')&#xA;regressor.fit(X_train, y_train, batch_size = 32, epochs = 100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="33966" LastEditorUserId="33966" LastEditDate="2017-06-28T16:26:15.893" LastActivityDate="2022-11-13T15:12:50.523" Title="MultiVariate Regression with LSTM" Tags="|machine-learning|neural-network|deep-learning|regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20034" PostTypeId="2" ParentId="20026" CreationDate="2017-06-28T16:26:47.540" Score="-1" Body="&lt;p&gt;There is a trend towards implementations that don't need input sizes known in advance. Check out DyNet or Chainer for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From DyNet's &lt;a href=&quot;https://arxiv.org/abs/1701.03980&quot; rel=&quot;nofollow noreferrer&quot;&gt;technical paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In DyNet's dynamic declaration strategy, computation graph&#xA;  construction is mostly transparent, being implicitly constructed by&#xA;  executing procedural code that computes the network outputs, and the&#xA;  user is free to use different network structures for each input.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ronen&lt;/p&gt;&#xA;" OwnerUserId="33967" LastActivityDate="2017-06-28T16:26:47.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20035" PostTypeId="2" ParentId="19933" CreationDate="2017-06-28T16:33:23.573" Score="4" Body="&lt;p&gt;I suspect a couple issues:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The cost function is unusual for classification. You would typically&#xA;use something like cross entropy which you mentioned. Make sure not to threshold the outputs when you input to the cross entropy loss. Also, I wouldn't characterize L2 loss as being more &quot;harsh&quot;. If your predicted probability for a particular data point is 0, but its label is 1, L2 loss would be 1. But for cross entropy, the loss would be very large.&lt;/li&gt;&#xA;&lt;li&gt;It predicts all one class on the validation set or just any particular batch?&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If it is just a particular batch, maybe how you batch the data is off and all of one class is in that batch. Look at metrics like accuracy and loss.&lt;/li&gt;&#xA;&lt;li&gt;If it is the validation set, I would look into if the training loss is consistently going down. Maybe the learning rate is too high. You are using Adam which is based off of momentum so it does not necessarily need as high of a learning rate as standard SGD.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;If the weights are stabilized, it means that the network is saturated or you have 0 loss and your optimization problem is solved (probably not the latter). If the network did saturate too fast, batch norm would definitely help.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33934" LastActivityDate="2017-06-28T16:33:23.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20036" PostTypeId="1" AcceptedAnswerId="20152" CreationDate="2017-06-28T17:12:51.587" Score="2" ViewCount="205" Body="&lt;pre&gt;&lt;code&gt;Who teaches English?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, after tokenizing, stemming..&#xA;    it gives me&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Who, teach, English&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my list of word, I have a word called&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;teacher&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lemmatizing, stemming &lt;em&gt;teacher&lt;/em&gt; gives &lt;strong&gt;teacher&lt;/strong&gt; and lemmatizing, stemming &lt;em&gt;teaches&lt;/em&gt; gives &lt;strong&gt;teach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even, calculating edit_distance will not solve this.. As, edit_distance comes out to be 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, What do I do to have  &lt;strong&gt;teacher&lt;/strong&gt; and &lt;strong&gt;teach&lt;/strong&gt; treated as similar?&#xA;Similarly, there may be other cases with extra 's' at the end. Is there some stemmer that solves this problem? Is there any solution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other similar example can be: &lt;strong&gt;instructor&lt;/strong&gt; and &lt;strong&gt;instructs&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="33970" LastEditorUserId="33970" LastEditDate="2017-06-28T17:31:01.513" LastActivityDate="2021-05-05T20:06:32.493" Title="What can be done so that 'teacher' and 'teaches' are treated similar?" Tags="|nlp|nltk|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20037" PostTypeId="2" ParentId="20036" CreationDate="2017-06-28T17:26:22.080" Score="1" Body="&lt;p&gt;Check out &lt;a href=&quot;https://research.fb.com/projects/fasttext/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fasttext&lt;/a&gt;.  Fasttext works similarly to word2vec in that you can create word embeddings, however, it actually analyzes character n-grams, to force the syntactic similarity to what you're thinking about.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2017-06-28T17:26:22.080" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20038" PostTypeId="2" ParentId="20031" CreationDate="2017-06-28T17:57:42.263" Score="1" Body="&lt;p&gt;Basically, it's called an error because you're making the wrong decision. Like for instance, let's say your null hypothesis is something like: The patient does not have cancer. And the Alternative: The patient does have cancer. A type 2 error in this regard would be like saying &quot;Oh, this patient does not have cancer&quot; while they actually do have cancer. We are FAILING to REJECT a FALSE null. So I guess the most simple way to explain this is, we aren't catching a mistake, and letting it slide by. Of course this isn't all there is to it, but it should help kind of show what a type 2 error is and why it's something you want to avoid.&lt;/p&gt;&#xA;" OwnerUserId="33287" LastActivityDate="2017-06-28T17:57:42.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20043" PostTypeId="2" ParentId="19933" CreationDate="2017-06-28T19:17:47.677" Score="3" Body="&lt;p&gt;Rather than sampling your negative values to achieve a 1:1 ratio, you should try weighting your classes. You can simply use the existing positive:negative ratio you have already calculated and pass this to the class_weights parameter in skflow (assuming you are using skflow, if not, there are almost always equivalent ways of doing this in any ML package/language).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;class_weight = tf.constant([0.9, 0.1]))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;skflow.models.logistic_regression(X, y, class_weight=class_weight) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I suggest this is that your random all positive or all negative outputs may be down to the way you're down-sampling the negative examples. If you're doing this at random, you may be removing some key examples in the dataset that the classifier would have used to learn to strongly distinguish between the classes (for example: stock data, where two candles show a drop, but one shows &lt;em&gt;much&lt;/em&gt; more of a drop). By weighting you can train on all of your data and remove this potential problem, whilst still 'learning' both classes equally.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-06-28T19:17:47.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20045" PostTypeId="1" CreationDate="2017-06-28T19:55:44.117" Score="3" ViewCount="2971" Body="&lt;p&gt;Lets say I have a screenshot like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/528g2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/528g2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to be able to detect/localize each item on the floor, however, there 1) can be any number of items in the image and 2) each item is different&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a candidate list of all possible items. In reality each one is labelled and separated into individual image files:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t9DED.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t9DED.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've thought about training something like a convnet, but that feels like it might be slow because I'd need to segment each screenshot with multiple sliding windows and feed each one forward through the net. Creating those sliding window segments for each screenshot will likely take a long time. I'd like the entire detection process to be completed quickly (&amp;lt;1sec)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whats the most efficient way of doing this detection task? I will be implementing this using Javascript&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problems are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The item count is unknown. There could be 0 items in the screenshot, or there could be many&lt;/li&gt;&#xA;&lt;li&gt;There are a lot of possible targets, each one of which are different in shape, colour, and structure&lt;/li&gt;&#xA;&lt;li&gt;Javascript isn't the fastest of languages for this type of work. This is a semi strict requirement unless its absolutely impossible to do this using JS. The fallback language would be Python&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2017-07-29T22:27:13.367" Title="Computer vision: Identifying different items in screenshot" Tags="|machine-learning|classification|image-classification|computer-vision|javascript|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20046" PostTypeId="2" ParentId="20024" CreationDate="2017-06-28T20:34:20.060" Score="6" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, it is always possible to train a classification algorithm when you have labeled &lt;a href=&quot;https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables&quot; rel=&quot;noreferrer&quot;&gt;i.i.d.&lt;/a&gt; training data, and there is no hard reason why you cannot use a simulator to generate that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether or not such a trained model is fit for purpose is hard to say in advance of trying it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a simulation as your data source has some benefits:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Generating more training and test data is straightforward. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You will automatically have high quality ground truth labels (assuming your goal is to match the simulation).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you find a problem with certain parameter values, you can target them when collecting more training data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just as with data taken from real world measurements, you will need to test your results to get a sense of how accurate your model is. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your model is a function approximator. At best it will match the output of the simulator. In practice it will usually fall short of it by some amount. You will have to measure this difference by testing the model, and decide whether the cost of occasional false negative or false positive is outweighed by the performance improvement. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Statistical machine learning models perform best when interpolating between data points, and often perform badly at extrapolating. So when you say that inputs can vary infinitely, hopefully that is within some constrained parameter space of real values, as opposed to getting inputs that are completely different from anything you have considered before - the simulation would cope with such inputs, but a statistics-based function approximator most likely would not.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If your simulation has areas where the class switches rapidly with small changes in parameter values, then you will need to sample densely in those areas. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If your simulation produces near chaotic behaviour in any region (class value varies a lot and is highly sensitive to small changes in value of one or more parameters), then this is something that is very hard to approximate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you have some natural scale factor, dimensionless number or other easy to compute summary of behaviour in your physical system, it may be worth using it as an engineered feature instead of getting the machine learning code to figure that out statistically. For instance, in fluid dynamics, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Reynolds_number&quot; rel=&quot;noreferrer&quot;&gt;Reynolds number&lt;/a&gt; can characterise flow, and could be useful feature for neural network predicting vortex shedding.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Any references to such examples?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The examples I have found here are about are in renderings of fluid simulations and other complex physical systems where a full simulation can be approximated and they all use neural networks to achieve a speed improvement over full simulation.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://cims.nyu.edu/~schlacht/CNNFluids.htm&quot; rel=&quot;noreferrer&quot;&gt;Accelerating Eulerian Fluid Simulation With Convolutional Networks&lt;/a&gt; - I saw a video about this on YouTube's &lt;a href=&quot;https://www.youtube.com/user/keeroyz/featured&quot; rel=&quot;noreferrer&quot;&gt;Two Minute Papers&lt;/a&gt; channel.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://polar.ncep.noaa.gov/mmab/papers/tn265/MMAB265.pdf&quot; rel=&quot;noreferrer&quot;&gt;Using neural networks in weather prediction ensembles to improve performance&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://papers.nips.cc/paper/1562-fast-neural-network-emulation-of-dynamical-systems-for-computer-animation.pdf&quot; rel=&quot;noreferrer&quot;&gt;Fast Neural Network Emulation of Dynamical Systems for Computer Animation&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, I don't think any of these are classifiers.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-06-28T20:34:20.060" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20047" PostTypeId="2" ParentId="20031" CreationDate="2017-06-28T22:13:19.697" Score="0" Body="&lt;p&gt;You'll never actually &lt;em&gt;know&lt;/em&gt; if you've made a Type-II (or type-I) error in practice. As you surmise, during the hypothesis testing stage you either do or do not reject H0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type-I and type-II errors are more useful in terms of error &lt;em&gt;rate&lt;/em&gt;.  This is a function of your alpha level and statistical power. So for instance you can set alpha to 0.05, 0.01, 0.001, etc based on your tolerance for such errors. But again you can't know after the fact. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's called an 'error' from a pedagogical standpoint. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-06-28T22:13:19.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20048" PostTypeId="2" ParentId="19713" CreationDate="2017-06-28T22:31:20.407" Score="1" Body="&lt;p&gt;When you extract the features, I'm assuming the features are stored somewhere. This means only the computation for each image is done only once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you stack layers on top of the inception model, even if the inception weights are frozen, the forward pass for the activations or features still need to be computed. This is extra computation time, since the inception weights are not changing.&lt;/p&gt;&#xA;" OwnerUserId="33934" LastActivityDate="2017-06-28T22:31:20.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20049" PostTypeId="2" ParentId="20025" CreationDate="2017-06-28T23:01:52.930" Score="2" Body="&lt;p&gt;What you're describing is known as a multilabel classification. You want to predict some output labels (adverse effects) in a given set of possible output labels  (all possible adverse effects), using a given a set of input indicators (previous adverse effects, drugs prescribed, dosage levels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For those final two features, I'd combine them into one feature where if the drug is not prescribed obviously dosage is 0, and higher otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In R, you can use the &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/devel/html/multilabel/index.html#performance&quot; rel=&quot;nofollow noreferrer&quot;&gt;mlR package&lt;/a&gt; for this prediction stage. You may also be able to use the &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/release/html/feature_selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;generateFilterValues&lt;/a&gt;() function in this package to extract feature importances, however I'm not sure if this will work with multilabel and additionally this will only tell you how much your classifier believes your input features contribute to the output labels relative to all the other input features. It won't give you other valuable information for example at which dosage levels of a particular drug does the probability of negative symptoms increase most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, I'm unsure how this would work with multilabel classifications as I don't have any experience with them, but take a look at plotting &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/release/html/partial_dependence/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;partial dependency plots&lt;/a&gt; to get this last bit of information on how each input feature contributes to your output labels.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-06-28T23:01:52.930" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20051" PostTypeId="1" CreationDate="2017-06-29T00:34:27.867" Score="-1" ViewCount="511" Body="&lt;p&gt;I have a lot of data and manually extracted annotations for the text. I was looking for any advice to automate the annotation extraction up to a good level of accuracy. Any advice is welcome.&lt;/p&gt;&#xA;" OwnerUserId="26444" LastEditorUserId="85045" LastEditDate="2020-12-06T22:06:12.737" LastActivityDate="2021-08-01T12:53:42.157" Title="Building a text extractor to extract particular type of text from unstructured text data" Tags="|machine-learning|python|data-mining|nlp|nltk|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="20052" PostTypeId="1" CreationDate="2017-06-29T01:47:28.240" Score="1" ViewCount="390" Body="&lt;p&gt;I've a &lt;a href=&quot;http://www.golfforecast.co.uk&quot; rel=&quot;nofollow noreferrer&quot;&gt;golf website&lt;/a&gt; that pulls a load of statistical data and makes forecasts for upcoming events. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to improve the algorithm and have been looking into Azure's ML Studio. I've lots of data about previous events and stats the competing golfers had and the time of the event so I'm confident I need to be using some sort of regression based algorithm but I'm not sure which one, there's a few that look appropriate? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice would be greatly appreciated! &#xA; Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5WET8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5WET8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33986" LastActivityDate="2017-06-29T11:56:02.253" Title="Which Regression Machine Learning algorithm to use in Azure for statistical prediction/forecasting" Tags="|machine-learning|regression|azure-ml|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20054" PostTypeId="1" AcceptedAnswerId="23035" CreationDate="2017-06-29T02:56:16.950" Score="1" ViewCount="196" Body="&lt;p&gt;Is there any other materials that derives the LSTM back propagation and carousel of error except the original paper? (I could not understand it, sorry).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried deriving and got stuck, and asked the following question: &lt;a href=&quot;https://datascience.stackexchange.com/questions/19614/need-help-understanding-lstms-backpropagation-and-carousel-of-error&quot;&gt;LSTMs - Data Science Stack Exchange question&lt;/a&gt;, however, it doesn't seems that there are much people interested in hand derivation of LSTMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="24119" LastActivityDate="2017-09-14T16:19:30.947" Title="LSTMs - Carousel of Error" Tags="|rnn|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20056" PostTypeId="2" ParentId="20030" CreationDate="2017-06-29T05:42:19.513" Score="1" Body="&lt;p&gt;I'm not sure if that's what you are looking for - but you should check out the package 'pickle'. You can use it to easily dump your model into a file and load it again.&lt;/p&gt;&#xA;" OwnerUserId="30538" LastActivityDate="2017-06-29T05:42:19.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20058" PostTypeId="1" CreationDate="2017-06-29T11:42:16.360" Score="1" ViewCount="3397" Body="&lt;p&gt;I am an absolute newbie in TensorFlow but have a fair understanding of ML algorithms. I have a project to model the error characteristic of time of flight cameras. As ground truth, I have acquired a set of depth images(512x424 pixels) from a stereo set up. The the range images(512x424 pixels) from the ToF camera needs to be compared with the reference depth images. In order to learn the error characteristic I am implementing a deep neural network with the reference image pixels as training input data(median pixels as features) and the difference of reference image and range camera image as training output value. There are 3 pairs of images to train and 1 pair of images to test. I have flattened the image matrices so the training input data are 3-element lists of 217088 sized arrays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code works without any errors but the result is ugly. Please look into the following issues:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The cost reduces nicely after the first epoch but does not change much after the second epoch.&lt;/li&gt;&#xA;&lt;li&gt;The accuracy of the test phase is horrendous.&lt;/li&gt;&#xA;&lt;li&gt;The code is extremely slow. It takes almost 2 hours for a complete run. May be it has to do with the hardware. I am running it on core i3.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy &#xA;import cv2&#xA;import matplotlib.pyplot as plt&#xA;import glob&#xA;&#xA;refDepthImgLoc = 'M:\Internship\Scan\png\scan_dist*.png'&#xA;tofDepthImgLoc = 'M:\Internship\Scan\png\kinect_distance*.png'&#xA;&#xA;numImg = 4&#xA;&#xA;refDepthImg = []&#xA;tofDepthImg = []&#xA;refLoc = glob.glob(scanDistImgLoc)&#xA;tofLoc = glob.glob(tofDistImgLoc)&#xA;&#xA;for refImg, tofImg in zip(refLoc, tofLoc) :&#xA;    img1 = cv2.imread(refImg, 0)&#xA;    refDepthImg.append(img1)&#xA;&#xA;    img2 = cv2.imread(tofImg, 0)&#xA;    tofDepthImg.append(img2)&#xA;&#xA;trainData_median = []&#xA;trainLabel = []&#xA;for i in range(len(refDepthImg)):&#xA;    tempData = cv2.medianBlur(refDepthImg[i], 3)&#xA;    trainData_median.append(tempData.ravel())&#xA;    tempLabel = refDepthImg[i] - tofDepthImg[i]&#xA;    trainLabel.append(tempLabel.ravel())&#xA;&#xA;n_nodes_hl1 = 100&#xA;n_nodes_hl2 = 100&#xA;n_nodes_hl3 = 100&#xA;&#xA;n_input = 1;&#xA;n_output = 1;&#xA;learning_rate = 0.01&#xA;&#xA;x = tf.placeholder('float')&#xA;y = tf.placeholder('float')&#xA;&#xA;def neural_network_model(data):&#xA;    hidden_1_layer = {'weights':tf.Variable(tf.random_normal([n_input, n_nodes_hl1])),&#xA;                      'biases':tf.Variable(tf.random_normal([n_nodes_hl1]))}&#xA;&#xA;    hidden_2_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),&#xA;                      'biases':tf.Variable(tf.random_normal([n_nodes_hl2]))}&#xA;&#xA;    hidden_3_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])),&#xA;                      'biases':tf.Variable(tf.random_normal([n_nodes_hl3]))}&#xA;&#xA;    l1 = tf.add(tf.matmul(data,hidden_1_layer['weights']), hidden_1_layer['biases'])&#xA;    l1 = tf.nn.relu(l1)&#xA;&#xA;    l2 = tf.add(tf.matmul(l1,hidden_2_layer['weights']), hidden_2_layer['biases'])&#xA;    l2 = tf.nn.relu(l2)&#xA;&#xA;    l3 = tf.add(tf.matmul(l2,hidden_3_layer['weights']), hidden_3_layer['biases'])&#xA;    l3 = tf.nn.relu(l3)&#xA;&#xA;    output = tf.reduce_sum(l3)&#xA;&#xA;    return output&#xA;&#xA;def train_neural_network(x):&#xA;    prediction = neural_network_model(x)&#xA;    cost = tf.reduce_sum(tf.square(prediction-y))/((numImg-1)*len(trainLabel[0]))&#xA;    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)&#xA;&#xA;    hm_epochs = 10&#xA;&#xA;    with tf.Session() as sess:&#xA;        sess.run(tf.global_variables_initializer())&#xA;&#xA;        for epoch in range(hm_epochs):&#xA;            tempLoss = 0&#xA;&#xA;            for i in range(numImg - 1):&#xA;                 for (X, Y) in zip(trainData_median[i], trainLabel[i]):&#xA;                    _, c = sess.run([optimizer, cost], feed_dict={x: [[X]], y: [[Y]]})&#xA;                    tempLoss += c&#xA;&#xA;            print('Epoch', (epoch+1), 'completed out of',hm_epochs,'loss:',tempLoss)&#xA;&#xA;        print(&quot;Testing starts now&quot;)&#xA;&#xA;        test = tf.abs(prediction - y)&#xA;        i = 0;&#xA;        pred = numpy.zeros(len(trainLabel[0]));&#xA;        result = numpy.zeros(len(trainLabel[0]));&#xA;        for (X, Y) in zip(trainData_median[numImg - 1], trainLabel[numImg - 1]):&#xA;            correct, pred[i] = sess.run([test, prediction], feed_dict={x: [[X]], y: [[Y]]}) &#xA;&#xA;            if (correct &amp;lt; 0.5):&#xA;                result[i] = 1&#xA;            i += 1&#xA;&#xA;        accuracy = tf.reduce_mean(tf.cast(result, 'float'))&#xA;        print('Accuracy:', accuracy.eval())&#xA;&#xA;train_neural_network(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1 completed out of 10 loss: 204681865.46&#xA;Epoch 2 completed out of 10 loss: 3188.81297796&#xA;Epoch 3 completed out of 10 loss: 3183.35926716&#xA;Epoch 4 completed out of 10 loss: 3181.37895241&#xA;Epoch 5 completed out of 10 loss: 3179.95276242&#xA;Epoch 6 completed out of 10 loss: 3178.51366003&#xA;Epoch 7 completed out of 10 loss: 3177.6227609&#xA;Epoch 8 completed out of 10 loss: 3176.69995104&#xA;Epoch 9 completed out of 10 loss: 3176.85162593&#xA;Epoch 10 completed out of 10 loss: 3177.04338937&#xA;Testing starts now&#xA;Accuracy: 0.00301721&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please comment if something is inherently wrong in the code or the entire approach to the problem is incorrect. Should I try implementing it using CNNs? Please help me in making this work. Please let me know if any further information is required.&lt;/p&gt;&#xA;" OwnerUserId="34002" LastActivityDate="2017-06-29T11:42:16.360" Title="TensorFlow: Regression using Deep Neural Network" Tags="|machine-learning|python|deep-learning|tensorflow|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20059" PostTypeId="2" ParentId="20052" CreationDate="2017-06-29T11:56:02.253" Score="-1" Body="&lt;p&gt;If I had to choose, it would be one of the last three as they are more sophisticated. If you have some time, try them all out, sweep the hyperparameters, and compare the results. This free EdX course has a segment where you do that comparison of different models in Azure ML:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edx.org/course/principles-machine-learning-microsoft-dat203-2x-4&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.edx.org/course/principles-machine-learning-microsoft-dat203-2x-4&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-29T11:56:02.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20060" PostTypeId="2" ParentId="20045" CreationDate="2017-06-29T11:58:15.467" Score="0" Body="&lt;p&gt;You really have two problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Object detection  / bounding boxes to locate an icon you want to recognize.&lt;/li&gt;&#xA;&lt;li&gt;Image recognition of what each icon is. Because you have labeled training data already and they are all roughly the same size, this part should be relatively easy as compared to #1.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You need to get #1 working and passing a subset of the pixels to #2.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-29T11:58:15.467" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20061" PostTypeId="1" CreationDate="2017-06-29T13:53:09.927" Score="-1" ViewCount="1601" Body="&lt;p&gt;I built a pipeline for an LDA model using pyspark's machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; tokenizer = Tokenizer(inputCol='formatted_transcript',&#xA; outputCol='words')&#xA;&#xA; stopWordsRemover = StopWordsRemover(inputCol=&quot;words&quot;,&#xA; outputCol=&quot;filtered&quot;)&#xA;&#xA; vectorizer = CountVectorizer(inputCol=&quot;filtered&quot;,&#xA; outputCol=&quot;features&quot;, vocabSize=40000, minDF=5.0)&#xA;&#xA; lda = LDA(k=10, maxIter=10)&#xA;&#xA; pipeline = Pipeline(stages=[tokenizer, stopWordsRemover, vectorizer,&#xA; lda])&#xA;&#xA; pipelineModel = pipeline.fit(DF_with_formatted)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I print the results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(pipelineModel.stages[3].describeTopics(5).show(truncate=False))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I see the termIndices and termWeights, but not the actual words. How do I obtain the words corresponding to the termIndices?&lt;/p&gt;&#xA;" OwnerUserId="34006" LastEditorUserId="29169" LastEditDate="2019-06-12T03:21:30.060" LastActivityDate="2019-06-12T03:21:30.060" Title="Words from LDA output pyspark machine learning" Tags="|nlp|lda|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20062" PostTypeId="2" ParentId="20061" CreationDate="2017-06-29T16:18:04.310" Score="0" Body="&lt;p&gt;With help from &lt;a href=&quot;https://stackoverflow.com/questions/44233862/visualizing-topics-with-spark-lda&quot;&gt;here&lt;/a&gt;, I came up with the following solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vocab = pipelineModel.stages[2].vocabulary &#xA;topics = pipelineModel.stages[3].describeTopics(5)  &#xA;topics_rdd = topics.rdd&#xA;topics_words = topics_rdd\&#xA;       .map(lambda row: row['termIndices'])\&#xA;       .map(lambda idx_list: [vocab[idx] for idx in idx_list])\&#xA;       .collect()&#xA;&#xA;for idx, topic in enumerate(topics_words):&#xA;    print(&quot;topic: &quot;, idx)&#xA;    print(&quot;----------&quot;)&#xA;&#xA;    for word in topic:&#xA;        print(word)&#xA;        print(&quot;----------&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="34006" LastEditorUserId="381" LastEditDate="2017-06-29T18:57:55.343" LastActivityDate="2017-06-29T18:57:55.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20063" PostTypeId="1" CreationDate="2017-06-29T16:21:31.230" Score="1" ViewCount="185" Body="&lt;p&gt;By BVLC/Caffe examples cifar10, it has 3 convolution layers with pool/norm and its accuracy increases to around 82 easily. Now I want to increase my resolution by downloading the 10 classes of images from imagenet and make the image resolution to 55. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The scenario is that the images have higher resolution and less data: 5000 per-class while cifar-10 has 6000 per class. What are good design strategies for this new scenario? Is the performance going to drop a lot?&lt;/p&gt;&#xA;" OwnerUserId="33816" LastEditorUserId="29169" LastEditDate="2021-02-16T00:45:25.357" LastActivityDate="2021-02-16T00:45:25.357" Title="Design strategies for higher resolution images for Convolutional neural network?" Tags="|deep-learning|convolutional-neural-network|caffe|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20064" PostTypeId="1" AcceptedAnswerId="20067" CreationDate="2017-06-29T16:26:25.203" Score="1" ViewCount="4629" Body="&lt;p&gt;I am using Python/Scikit to do data encoding before I go ahead and train my Neural Network. I have a few columns that look like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;07:05:00&#xA;08:41:00&#xA;17:25:00&#xA;12:58:00&#xA;08:56:00&#xA;11:59:00&#xA;17:25:00&#xA;15:24:00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to encode this? Or is leaving it like this fine?&lt;/p&gt;&#xA;" OwnerUserId="31093" LastActivityDate="2017-06-29T19:11:35.487" Title="Encoding Time Values" Tags="|python|neural-network|scikit-learn|data-cleaning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20065" PostTypeId="2" ParentId="20013" CreationDate="2017-06-29T16:38:39.030" Score="1" Body="&lt;p&gt;For text processing, try using Python and the &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK package&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For removing surplus symbols, you can use regular expressions. Install package 're' and use the built-in function re.sub to substitute symbols like '-' with empty character.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For suffix stripping, you can either use regular expressions again or use the built-in word stemming functionality in the NLTK package. This &lt;a href=&quot;https://pythonspot.com/en/nltk-stemming/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; should help.&lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2017-06-29T16:38:39.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20067" PostTypeId="2" ParentId="20064" CreationDate="2017-06-29T18:39:59.660" Score="1" Body="&lt;p&gt;I have decided to convert the strings into seconds. Since these are all Time of Day values, I will convert them to contiguous seconds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/10663720/converting-a-time-string-to-seconds-in-python&quot;&gt;https://stackoverflow.com/questions/10663720/converting-a-time-string-to-seconds-in-python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31093" LastActivityDate="2017-06-29T18:39:59.660" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20068" PostTypeId="2" ParentId="20064" CreationDate="2017-06-29T19:11:35.487" Score="2" Body="&lt;p&gt;A common step in &quot;feature engineering&quot; is parsing these out into multiple values that might give you additional information during ML. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Hour of the day&lt;/li&gt;&#xA;&lt;li&gt;Minute of the hour&lt;/li&gt;&#xA;&lt;li&gt;Hours after sunrise&lt;/li&gt;&#xA;&lt;li&gt;Hours before sunset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you also had day, that would open a lot more to detect seasonality like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Month of the year&lt;/li&gt;&#xA;&lt;li&gt;Month of the quarter&lt;/li&gt;&#xA;&lt;li&gt;Day of the week&lt;/li&gt;&#xA;&lt;li&gt;Holiday Y/N&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-29T19:11:35.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20071" PostTypeId="1" AcceptedAnswerId="20082" CreationDate="2017-06-30T02:14:39.717" Score="36" ViewCount="77042" Body="&lt;p&gt;I tried to load fastText pretrained model from here &lt;a href=&quot;https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md&quot; rel=&quot;noreferrer&quot;&gt;Fasttext model&lt;/a&gt;. I am using &lt;a href=&quot;https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.simple.zip&quot; rel=&quot;noreferrer&quot;&gt;wiki.simple.en&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from gensim.models.keyedvectors import KeyedVectors&#xA;&#xA;word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, it shows the following errors&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;nltk_check.py&quot;, line 28, in &amp;lt;module&amp;gt;&#xA;    word_vectors = KeyedVectors.load_word2vec_format('wiki.simple.bin', binary=True)&#xA;  File &quot;P:\major_project\venv\lib\sitepackages\gensim\models\keyedvectors.py&quot;,line 206, in load_word2vec_format&#xA;     header = utils.to_unicode(fin.readline(), encoding=encoding)&#xA;  File &quot;P:\major_project\venv\lib\site-packages\gensim\utils.py&quot;, line 235, in any2unicode&#xA;    return unicode(text, encoding, errors=errors)&#xA;UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1&lt;/strong&gt; How do I load fasttext model with Gensim? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2&lt;/strong&gt; Also, after loading the model, I want to find the similarity between two words &lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt; model.find_similarity('teacher', 'teaches')&#xA; # Something like this&#xA; Output : 0.99&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I do this?&lt;/p&gt;&#xA;" OwnerUserId="33970" LastEditorUserId="65719" LastEditDate="2019-01-14T18:40:31.920" LastActivityDate="2021-04-30T19:47:44.947" Title="How do I load FastText pretrained model with Gensim?" Tags="|nlp|gensim|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20074" PostTypeId="1" AcceptedAnswerId="20155" CreationDate="2017-06-30T05:54:53.330" Score="1" ViewCount="201" Body="&lt;p&gt;I am a college student(rising senior) and became interested in Natural Language Processing last semester. I decided to focus on studying this area this summer and become skilled in this area. I wanted to get some advice for studying this particular subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, I am taking Andrew Ng's Machine Learning course on Coursera to get a sense of how Machine Learning works. After finishing this course, I am planning to take Standford's CS224n NLP course on Youtube and do its class activities. I am assuming AWS and Tenserflow is also important since they are included as topics of CS224n.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if my summer plan sounds reasonable. If not, could you please give me an advice of how to make a better plan. If this sounds reasonable, it would be great if you can add more or specify which part is particularly important in these areas.&lt;/p&gt;&#xA;" OwnerUserId="34037" LastEditorUserId="31513" LastEditDate="2017-06-30T14:45:32.110" LastActivityDate="2017-07-04T00:24:33.380" Title="Please let me know if I am on the right track to being an NLP Expert" Tags="|machine-learning|nlp|tensorflow|" AnswerCount="2" CommentCount="5" ClosedDate="2017-07-15T10:11:00.013" ContentLicense="CC BY-SA 3.0" />
  <row Id="20075" PostTypeId="1" CreationDate="2017-06-30T06:28:15.000" Score="43" ViewCount="69210" Body="&lt;p&gt;I am trying to look for a good argument on why one would use the &lt;strong&gt;Manhattan distance&lt;/strong&gt; over the &lt;strong&gt;Euclidean distance&lt;/strong&gt; in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest thing I found to a good argument so far is on &lt;a href=&quot;https://www.youtube.com/watch?v=h0e2HAPTGF4&quot; rel=&quot;noreferrer&quot;&gt;this MIT lecture&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At 36:15 you can see on the slides the following statement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;Typically use Euclidean metric; &lt;strong&gt;Manhattan may be appropriate if different dimensions are not comparable.&lt;/strong&gt;&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shortly after, the professor says that, because the number of legs of a reptile varies from 0 to 4 (whereas the other features are binary, only vary from 0 to 1), the &quot;number of legs&quot; feature will end up having a much higher weight if the Euclidean distance is used. Sure enough, that is indeed right. But one would also have that problem if using the Manhattan distance (only that the problem would be slightly mitigated because we don't square the difference like we do on the Euclidean distance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better way to solve the above problem would be to normalize the &quot;number of legs&quot; feature so its value will always be between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, since there is a better way to solve the problem, it felt like the argument of using the Manhattan distance in this case lacked a stronger point, at least in my opinion.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does anyone actually know why and when someone would use Manhattan distance over Euclidean? Can anyone give me an example in which using the Manhattan distance would yield better results?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="34036" LastEditorUserId="43623" LastEditDate="2020-04-06T16:01:18.987" LastActivityDate="2022-03-29T07:20:45.987" Title="When would one use Manhattan distance as opposed to Euclidean distance?" Tags="|machine-learning|classification|distance|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20076" PostTypeId="1" AcceptedAnswerId="20078" CreationDate="2017-06-30T07:05:33.707" Score="29" ViewCount="36996" Body="&lt;p&gt;I recently came across the terms &lt;strong&gt;Word2Vec&lt;/strong&gt;, &lt;strong&gt;Sentence2Vec&lt;/strong&gt; and &lt;strong&gt;Doc2Vec&lt;/strong&gt; and kind of confused as I am new to vector semantics. Can someone please elaborate the differences in these methods in simple words. What are the most suitable tasks for each method?&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2021-07-20T03:25:35.853" Title="Word2Vec vs. Sentence2Vec vs. Doc2Vec" Tags="|machine-learning|data-mining|clustering|nlp|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20077" PostTypeId="1" AcceptedAnswerId="20085" CreationDate="2017-06-30T07:18:51.260" Score="1" ViewCount="1848" Body="&lt;p&gt;I'm facing the following problem of integrating data from another company  (data base) to an internal one. It's about personal core data, i.e. name address etc. I would like to come up with a mapping of these keys in an automated way. I've read about HMM to use for this. However, I'm still gathering some information of some feasible and standard ways to do it. I'm looking therefore for some reference which describes a possible solution to this. &lt;/p&gt;&#xA;" OwnerUserId="33231" LastActivityDate="2017-06-30T12:51:44.647" Title="Schema matching using machine learning" Tags="|machine-learning|" AnswerCount="1" CommentCount="3" ClosedDate="2017-07-11T22:30:47.040" ContentLicense="CC BY-SA 3.0" />
  <row Id="20078" PostTypeId="2" ParentId="20076" CreationDate="2017-06-30T07:25:12.913" Score="27" Body="&lt;p&gt;Well the names are pretty straight-forward and should give you a clear idea of vector representations.&lt;/p&gt;&#xA;&lt;p&gt;The  Word2Vec Algorithm builds distributed semantic representation of words. There are two main approaches to training, Continuous Bag of Words and The skip gram model. One involves predicting the context words using a centre word, while the other involves predicting the word using the context words. You can read about it in much detail in Mikolov's  &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=http://web2.cs.columbia.edu/%7Eblei/seminar/2016_discrete_data/readings/MikolovSutskeverChenCorradoDean2013.pdf&amp;amp;ved=0ahUKEwiQpu_XguXUAhWIpo8KHQuHDNUQFggcMAA&amp;amp;usg=AFQjCNG3xiM98LkL7kkCQwCHfj0Iu8Ngdg&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The same idea can be extended to sentences and complete documents where instead of learning feature representations for words, you learn it for sentences or documents. However, to get a general idea of a SentenceToVec, think of it as a mathematical average of the word vector representations of all the words in the sentence. You can get a very good approximation just by averaging and without training any SentenceToVec but of course, it has its limitations.&lt;/p&gt;&#xA;&lt;p&gt;Doc2Vec extends the idea of SentenceToVec or rather Word2Vec because sentences can also be considered as documents. The idea of training remains similar. You can read Mikolov's Doc2Vec &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://cs.stanford.edu/%7Equocle/paragraph_vector.pdf&amp;amp;ved=0ahUKEwiQob-GhOXUAhWCqY8KHaiHCnMQFggcMAA&amp;amp;usg=AFQjCNE-2SsR07iQm78dQqVV-6_Y3ERurA&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; for more details.&lt;/p&gt;&#xA;&lt;p&gt;Coming to the applications, it would depend on the task. A Word2Vec effectively captures semantic relations between words hence can be used to calculate word similarities or fed as features to various NLP tasks such as sentiment analysis etc. However words can only capture so much, there are times when you need relationships between sentences and documents and not just words. For example, if you are trying to figure out, whether two stack overflow questions are duplicates of each other.&lt;/p&gt;&#xA;&lt;p&gt;A simple google search will lead you to a number of applications of these algorithms.&lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="21526" LastEditDate="2021-07-20T03:25:35.853" LastActivityDate="2021-07-20T03:25:35.853" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="20079" PostTypeId="1" CreationDate="2017-06-30T08:18:43.760" Score="1" ViewCount="138" Body="&lt;p&gt;I am trying to write a complete neural network from scratch. However, the results are not very encouraging. My prediction output shows one for one single row only in the 3 class classifier. I will post my code and then I will talk about the problem in the output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#labels or classes&#xA;#1=iris-setosa&#xA;#2=iris-versicolor&#xA;#0=iris-virginica&#xA;&#xA;#features&#xA;#sepallength&#xA;#sepalwidth&#xA;#petallengthcm&#xA;#petalwidth&#xA;&#xA;&#xA;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;import csv&#xA;import numpy as np&#xA;df=pd.read_csv('Iris.csv')&#xA;&#xA;df.convert_objects(convert_numeric=True)&#xA;df.fillna(0,inplace=True)&#xA;&#xA;df.drop(['Id'],1,inplace=True)&#xA;#function to convert three labels into values 0,1,2&#xA;def handle_non_numericaldata(df):&#xA;    columns=df.columns.values&#xA;    for column in columns:&#xA;        text_digit_vals={}&#xA;        def convert_to_int(val):&#xA;            return text_digit_vals[val]&#xA;        if df[column].dtype!=np.int64 and df[column].dtype!=np.float:       &#xA;            column_contents=df[column].values.tolist()&#xA;            unique_elements=set(column_contents)&#xA;            x=0&#xA;            for unique in unique_elements:&#xA;                if unique not in text_digit_vals:&#xA;                    text_digit_vals[unique]=x&#xA;                    x+=1&#xA;            df[column]=list(map(convert_to_int,df[column]))&#xA;    return(df)&#xA;handle_non_numericaldata(df)    &#xA;&#xA;x=np.array(df.drop(['Species'],1).astype(float))&#xA;c=np.array(df['Species'])&#xA;&#xA;&#xA;n_values=(np.max(c)+1)&#xA;#to make the vector one hot encode&#xA;y=(np.eye(n_values)[c])&#xA;(np.shape((y)))&#xA;m1=np.size(c)&#xA;&#xA;&#xA;theta=np.ones(shape=(4,21))&#xA;theta2=np.ones(shape=(21,3))&#xA;#no of examples &quot;m&quot;&#xA;&#xA;#learning rate alpha&#xA;alpha=0.05&#xA;#regularization parameter&#xA;lamda=0.05&#xA;for i in range(1,5000):&#xA;    z1=np.dot(x,theta)  &#xA;    sigma=1/(1+np.exp(-z1))&#xA;&#xA;    #activation layer 2.&#xA;    a2=sigma&#xA;    z2=np.dot(a2,theta2)&#xA;&#xA;    probs=np.exp(z2)&#xA;&#xA;    softmax=probs/np.sum(probs,axis=1,keepdims=True)&#xA;    delta3=softmax&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;    delta3=delta3-y&#xA;    A2=np.transpose(a2)&#xA;    dw2 = (A2).dot(delta3)&#xA;    W2=np.transpose(theta2)&#xA;&#xA;    delta2=delta3.dot(W2)*sigma*(1-sigma)&#xA;    X2=np.transpose(x)&#xA;    dw1=np.dot(X2,delta2)&#xA;&#xA;&#xA;    dw2=dw2-lamda*theta2&#xA;    dw1=dw1-lamda*theta&#xA;&#xA;    theta =theta -alpha* dw1&#xA;&#xA;    theta2= theta2-alpha * dw2&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;final1=x.dot(theta)&#xA;sigma=1/(1+np.exp(-final1))&#xA;z2=sigma.dot(theta2)&#xA;exp_scores=np.exp(z2)&#xA;probs=exp_scores/np.sum(exp_scores,axis=1,keepdims=True)&#xA;a=np.zeros_like(probs)&#xA;a[np.arange(len(a)),a.argmax(1)]=1&#xA;print(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output according to me, as I have written this, all the output probabilities are same hence  it cannot classify properly.output should be 'a' array with 50 1's in the first row, 50 in the second, and 50 in the third. But all the ones are in first row. I have been meddling with this since last three days. Please help me figure out mistake I have made in writing this neural network.&lt;/p&gt;&#xA;" OwnerUserId="34042" LastEditorUserId="34042" LastEditDate="2017-07-01T14:55:32.147" LastActivityDate="2017-07-01T14:55:32.147" Title="wrong output for neural network written from scratch" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="0" CommentCount="4" ClosedDate="2017-06-30T21:28:06.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="20080" PostTypeId="2" ParentId="20075" CreationDate="2017-06-30T08:21:00.760" Score="5" Body="&lt;p&gt;I can suggest a couple ideas, from &lt;a href=&quot;https://en.wikipedia.org/wiki/Least_absolute_deviations&quot; rel=&quot;noreferrer&quot;&gt;wikipedia&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you want to place less emphasis on outliers, manhattan distance will try to reduce all errors equally since the gradient has constant magnitude.&lt;/li&gt;&#xA;&lt;li&gt;If your noise is distributed Laplacian, the MLE is found by minimizing the manhattan estimate.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33540" LastActivityDate="2017-06-30T08:21:00.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20081" PostTypeId="1" CreationDate="2017-06-30T09:47:06.240" Score="0" ViewCount="1253" Body="&lt;p&gt;I need some help understanding my partial dependence plots for binary features passed to a GradientBoostClassifier when comparing them to the feature importances. For some background, my goal here is to investigate user churn. My classifications are: 0 = not churned, 1 = churned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1maFx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1maFx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;First of all when I plot the feature importances I get some interesting results which do not agree with the feature importances from my random forest. I put this down to algorithmic differences since many do align or are similar, but my specific concern is the binary feature 'registeredEmail', which is clearly &lt;em&gt;very&lt;/em&gt; important in the random forest model, but &lt;em&gt;not&lt;/em&gt; in the other (gradient boost, as shown). This is not my main concern, just adding in case relevant to my question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, and this is the part I am most confused by. Why does my feature importance chart show that 'playerInAlliance' (another binary feature) is significantly more important than 'registeredEmail', since when I check the partial dependencies there is a much steeper slope for 'registeredEmail'. I would interpret this as the churned prediction being highly influenced by a player not having a registered email, compared to being less likely to predict churn if the player does have their email registered. Is this correct?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/RruHX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RruHX.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note these plots are in descending order of feature importance, from left to right. Also ignore the title.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Comparing this to 'playerInAlliance' I can see that no matter if the player is in an alliance or not, this feature is influential in predicting a churned player. I don't understand how to interpret this, as if there's no real difference between players churning when in an alliance compared to having no alliance, then why is my GradientBoostClassifier considering it to be a highly important and highly dependent feature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EsCOj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EsCOj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if I plot both of these binary features against each other on one plot, we can see that dependence on a player not having a registered email is massively higher than if the player is in an alliance or not. So why is the registeredEmail feature less significant in terms of importance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TL;DR: How do I interpret Binary features in partial dependence plots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help appreciated.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastEditorUserId="33603" LastEditDate="2017-06-30T13:02:40.750" LastActivityDate="2017-06-30T13:02:40.750" Title="Feature Importance and Partial Dependence plots seem to disagree?" Tags="|machine-learning|python|visualization|feature-selection|scikit-learn|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20082" PostTypeId="2" ParentId="20071" CreationDate="2017-06-30T10:44:03.403" Score="27" Body="&lt;p&gt;Here's the link for the methods available for fasttext implementation in gensim &lt;a href=&quot;https://github.com/piskvorky/gensim/blob/develop/gensim/models/wrappers/fasttext.py&quot; rel=&quot;noreferrer&quot;&gt;fasttext.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;from gensim.models.wrappers import FastText&#xA;&#xA;model = FastText.load_fasttext_format('wiki.simple')&#xA;&#xA;print(model.most_similar('teacher'))&#xA;# Output = [('headteacher', 0.8075869083404541), ('schoolteacher', 0.7955552339553833), ('teachers', 0.733420729637146), ('teaches', 0.6839243173599243), ('meacher', 0.6825737357139587), ('teach', 0.6285147070884705), ('taught', 0.6244685649871826), ('teaching', 0.6199781894683838), ('schoolmaster', 0.6037642955780029), ('lessons', 0.5812176465988159)]&#xA;&#xA;print(model.similarity('teacher', 'teaches'))&#xA;# Output = 0.683924396754&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33970" LastEditorUserId="33970" LastEditDate="2019-02-13T16:33:21.297" LastActivityDate="2019-02-13T16:33:21.297" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="20084" PostTypeId="1" CreationDate="2017-06-30T10:58:07.127" Score="2" ViewCount="1225" Body="&lt;p&gt;I understand that this is broad, but I merely require a few pointers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to implement a text summarization system. I have already tried out an abstractive approach, now I am trying an extractive approach. The current extractive approaches out there(Textrank, Lexrank etc) are not giving optimal performance. It was suggested to me that I take it as an unsupervised learning task, using Autoencoders/RBM's or K means or any other unsupervised learning algorithm to classify sentences as important or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is something like this feasible? (I know that nothing can be said for sure in data science unless tried out, but is it worth the shot?)&lt;/li&gt;&#xA;&lt;li&gt;What kind of features should I feed into the classifier? I already know of word frequency distribution(TF-IDF), sentence position, co-occurance stats, but will these be enough? What additional features should I consider? Or should I consider directly feeding in word vectors?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have done my bit of work on the same and have read several papers, but none of them offer clarity in terms of feature computation, but only explain network architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any form of help will be appreciated, thank you!&lt;/p&gt;&#xA;" OwnerUserId="34048" LastEditorUserId="34048" LastEditDate="2017-06-30T11:06:40.663" LastActivityDate="2017-07-17T19:04:19.903" Title="Extractive text summarization, as a classification problem using deep networks" Tags="|deep-learning|nlp|svm|automatic-summarization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20085" PostTypeId="2" ParentId="20077" CreationDate="2017-06-30T12:51:44.647" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Michael_Stonebraker&quot; rel=&quot;nofollow noreferrer&quot;&gt;Michael Stonebraker&lt;/a&gt; started a company that claims to do just that, schema matching using machine learning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tamr.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tamr.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their site no longer has many details on their approach but &lt;a href=&quot;http://www.kdnuggets.com/2014/05/tamr-new-frontier-big-data-curation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article talks about some of the techniques they used, like :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Perform fuzzy string comparisons over attribute names using trigram cosine    similarity. &lt;/li&gt;&#xA;  &lt;li&gt;Treats a column of data as a document and tokenize its    values with a standard full text parser. Then, measure TF-IDF cosine&lt;br&gt;&#xA;  similarity between columns. This method is suitable for text fields.&lt;/li&gt;&#xA;  &lt;li&gt;Use minimum description length (MDL) to compare values of two    attributes. &lt;/li&gt;&#xA;  &lt;li&gt;Compute the ratio of the size of the intersection of two    columns' data to the size of their union. This method is well suited&lt;br&gt;&#xA;  for categorical fields with small number of values. &lt;/li&gt;&#xA;  &lt;li&gt;Compute Welch's t-test for a pair of columns that contain numeric values and get the probability the columns were drawn from the same&#xA;  distribution.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-30T12:51:44.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20086" PostTypeId="1" CreationDate="2017-06-30T13:20:46.723" Score="0" ViewCount="3877" Body="&lt;p&gt;I apologize if this has been addressed before, but I did not find a solution for this issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently trying to solve a problem with four dependent variables and four independent variables, all non-categorial. They seem to have non-linear dependence and multicolinearity - standard multivariate regression yielded reasonable results, but I am trying to compare them with results from Random Forest and Neural Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://scikit-learn.org/stable/modules/neural_networks_supervised.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt;, it should be possible to get multi-output with the MLPRegressor. However, when I try to apply the .fit method, I get an error message:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;fit() missing 1 required positional argument: 'y'&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code that I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Data Preprocessing - Importing the Libraries&#xA;import pandas as pd&#xA;&#xA;# Importing the dataset&#xA;dataset = pd.read_csv('Rock_Properties_4T.csv')&#xA;&#xA;y = dataset.iloc[:, 4:8].values&#xA;X = dataset.iloc[:, 0:4].values&#xA;&#xA;# ANN&#xA;from sklearn.neural_network import MLPRegressor&#xA;regressor_ANN = MLPRegressor(tol = 1e-16)&#xA;MLPRegressor.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There you go... Any idea of what could be the problem?&lt;/p&gt;&#xA;" OwnerUserId="34052" LastActivityDate="2020-04-17T14:42:11.890" Title="Multioutput regression with MLPRegressor - Does it work?" Tags="|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20087" PostTypeId="1" CreationDate="2017-06-30T13:24:59.487" Score="2" ViewCount="6798" Body="&lt;p&gt;Apologies for this basic question. I don't have a deep understanding of this function but I was pretty sure it would be zero if logits and label inputs are identical. I found out my MLP model solves my classification problem pretty well but the cost function is never actually 0 (rather it approaches something like -2.14442e+07 but never is actually 0). I have tested what happens if set logits and labels to the same input and still its the same. I use the tensorflow tf.nn.softmax_cross_entropy_with_logits as cost function. &#xA;Thank you for your help. &lt;/p&gt;&#xA;" OwnerUserId="34053" LastEditorUserId="34053" LastEditDate="2017-06-30T13:38:10.640" LastActivityDate="2018-11-07T19:54:03.607" Title="Should softmax cross entropy with logits always be zero if logits and labels are identical?" Tags="|machine-learning|classification|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20088" PostTypeId="2" ParentId="20086" CreationDate="2017-06-30T13:54:24.447" Score="0" Body="&lt;p&gt;I've never encountered that but am guessing from how the documentation is phrased later on with a single target (not target&lt;strong&gt;s&lt;/strong&gt;), you might need to specifically use the multioutput regressor in sklearn like here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-06-30T13:54:24.447" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20091" PostTypeId="2" ParentId="20087" CreationDate="2017-06-30T15:26:45.893" Score="3" Body="&lt;p&gt;Logits are the pre-transform values in a layer, and are not compared directly to the labels when calculating the cost function. In fact, with softmax layer, setting the matching logit to 1, and the rest to zero isn't even that good, because (using &lt;span class=&quot;math-container&quot;&gt;$z_i$&lt;/span&gt; as logit &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$a_i$&lt;/span&gt; as activation &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Substitute in your 0 and 1 values for &lt;span class=&quot;math-container&quot;&gt;$z_i$&lt;/span&gt; and you will see the &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; values are not that close to 0 and 1 (they would be 0.0853 and 0.2319 respectively for 10 classes). It is these activation values that feed into the loss calculation. So with typical multiclass loss function &lt;span class=&quot;math-container&quot;&gt;$l = -\sum_{\forall i} y_i log(a_i)$&lt;/span&gt;, you would score a loss of 1.4611 for such an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a very low cost, instead set the logit matching the true label (or 1) to a higher value, e.g. 10, and the logits matching the false value (or zero) to a low value e.g. -10. This would lead to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$z_i = \{10\; if\;1\;\; |\;\;-10\;if\;0\}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And varies between 10 and -10 for values between 1 and 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should significantly reduce the calculated loss (1.855e-08 for those values in a 10 class prediction). The logit allows us to 'stretch' the values for correct and incorrect predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting an actual 0 loss when using softmax or sigmoid outputs and logloss is not technically possible. You might see it due to rounding in computation. However, neither sigmoid nor softmax equals exactly 0 or 1, except in the limit as input values tend to &lt;span class=&quot;math-container&quot;&gt;$\pm \infty$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2018-11-07T19:54:03.607" LastActivityDate="2018-11-07T19:54:03.607" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20094" PostTypeId="2" ParentId="17471" CreationDate="2017-06-30T17:55:25.400" Score="5" Body="&lt;p&gt;Typically to extract features, you can use the top layer of the network before the output. The intuition is that these features are linearly separable because the top layer is just a logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For GANs, you can use the features from the discriminator. These features are supposed to give a probability if the input came from the training dataset, &quot;real images&quot;. In Radford's DCGAN &lt;a href=&quot;https://arxiv.org/abs/1511.06434&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;, they use all the convolutional layers of the discriminator and run a max pooling layer extract features for CIFAR-10.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To evaluate the quality of the representations learned by DCGANs for supervised tasks,&#xA;  we train on Imagenet-1k and then use the discriminator’s convolutional features from all layers,&#xA;  maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then&#xA;  flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM&#xA;  classifier is trained on top of them.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33934" LastActivityDate="2017-06-30T17:55:25.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20095" PostTypeId="1" CreationDate="2017-06-30T18:21:35.317" Score="0" ViewCount="896" Body="&lt;p&gt;One of my friends was asked this question in an interview. A clue/restriction is given: Do NOT use semi-supervised learning techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a binary classification problem: There are ~10K labeled points and ~1M unlabeled points. What are the steps in solving this problem?&lt;/p&gt;&#xA;" OwnerUserId="34063" LastActivityDate="2017-07-03T15:59:55.807" Title="Binary classification of partially labeled data" Tags="|classification|binary|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20098" PostTypeId="1" CreationDate="2017-07-01T13:59:39.613" Score="16" ViewCount="14239" Body="&lt;p&gt;I'm trying to understand the &lt;a href=&quot;http://adv-ml-2017.wdfiles.com/local--files/course-schedule/rl_class1.pdf&quot; rel=&quot;noreferrer&quot;&gt;policy gradient&lt;/a&gt; approach for solving the &lt;a href=&quot;https://gym.openai.com/envs/CartPole-v0&quot; rel=&quot;noreferrer&quot;&gt;cartpole&lt;/a&gt; problem.&#xA;In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation of the sum of gradients of our policy gradient for all actions in a sequence, weighted by the sum of discounted rewards in that sequence:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\nabla_\theta L(\theta) = E[ G(S_{0:T}, A_{0:T})\sum_{t=0}^{T}\nabla_\theta log\pi_\theta (A_t|S_t) ]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we estimate it using an empirical average across all samples in an episode - which makes sense intuitively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;BUT the less intuitive part is that I saw a common practice to normalize advantages in between episodes in several implementations (and indeed it works better). So after they calculate the they wouldn't directly use the advantage, but rather would normalize it, e.g. &lt;a href=&quot;https://github.com/ashutoshkrjha/Cartpole-OpenAI-Tensorflow/blob/master/cartpole.py&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; they do after every episode:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;discounted_epr = discount_rewards(epr)&#xA;discounted_epr -= np.mean(discounted_epr)&#xA;discounted_epr /= np.std(discounted_epr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;what's the justification for that - both in theory and in intuition? It seems to me that if an episode is long and as such has large advantages, it's worth learning more from that episode than from a 3 moves episode. What am I missing? &lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2019-06-05T20:47:02.703" Title="Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20099" PostTypeId="2" ParentId="5443" CreationDate="2017-07-01T14:21:09.920" Score="2" Body="&lt;p&gt;&lt;strong&gt;Excel can be an excellent tool for exploratory data analysis it really depends on your needs and of course it has its limitations like any tool, but excel definitely deserves a place in the data science hall of fame.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Worth remembering that in practice most users will be exploring a heavily reduced data set anyway (created from an SQL query). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excel is powerful for exploring data when you use the &quot;table&quot; object in combination with pivot tables, visualising is all 1-2 clicks max and a lot of excel charts in powerpoint look great, unless your looking to create something very bespoke e.g. in a scientific computing context. The interactive nature means you can explore rapidly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benefits of the &quot;table&quot; object is that as you transform the data further in excel to enable you to explore new distributions the pivot tables all remember the variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where excel is weak is that the formula list is arguably limiting, for instance a SQL case statement or python statment is way more flexible than an endless chain of if functions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It really depends on your needs but excel definitely deserves a place in the data science hall of fame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interesting anecdote, the team who work on the Facebook newsfeed algorithm can all regularly be seen to be playing with excel and lots of spreadsheets. &lt;/p&gt;&#xA;" OwnerUserId="34083" LastActivityDate="2017-07-01T14:21:09.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20100" PostTypeId="1" CreationDate="2017-07-01T15:23:31.770" Score="0" ViewCount="201" Body="&lt;p&gt;I want to compare my proposed method with traditional machine learning classifiers like Multilayer Perceptron(MLP) and SVM to check the classification accuracy.How do I compare different classifiers?. I want to draw a comparison between these three terms&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;1).Proposed method (Based on DNN) &lt;/li&gt;&#xA;&lt;li&gt;2).Shallow neural networks&lt;/li&gt;&#xA;&lt;li&gt;3).SVM&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20618" LastActivityDate="2017-07-03T10:02:45.643" Title="How do I compare traditional classifiers performance with my proposed method?" Tags="|machine-learning|classification|deep-learning|performance|matlab|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20102" PostTypeId="1" CreationDate="2017-07-01T18:46:11.710" Score="1" ViewCount="65" Body="&lt;p&gt;Have a dataset of a E-learning company  which have information related to the students demographics, package, payment info, services used, class or grade (KG to 12), login details. &#xA; If you want i can share the sample data set.&#xA;Could anyone help me out in understanding what business value can be derived from this data. I am really confused. I did some analysis was able to find out the max service consumption. Any help would be appreciated&lt;/p&gt;&#xA;" OwnerUserId="34090" LastActivityDate="2017-09-29T20:52:53.313" Title="Analysis of data E-learning website" Tags="|machine-learning|r|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20103" PostTypeId="2" ParentId="20102" CreationDate="2017-07-01T18:51:13.610" Score="1" Body="&lt;p&gt;Read news articles on schools for inspiration.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Student achievement compared by demographic&lt;/li&gt;&#xA;&lt;li&gt;Student advancement (or lack therefore) across grade&lt;/li&gt;&#xA;&lt;li&gt;Class popularity by enrollment&lt;/li&gt;&#xA;&lt;li&gt;Average cost per student&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-01T18:51:13.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20104" PostTypeId="1" CreationDate="2017-07-01T18:59:04.033" Score="3" ViewCount="13771" Body="&lt;p&gt;I'm working on a Dogs vs. Cats Redux: Kernels Edition project from Kaggle and I'm new for this area. So far, I couldn't be able to improve my test rate over %50. I'm wondering what can i do to improve my test accuracy or is there something wrong about my code? By the way, below code is use tflearn library but i also tried with Tensorflow directly, still i can not go over %50.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import division, print_function, absolute_import&#xA;&#xA;import csv&#xA;&#xA;from skimage import color, io&#xA;from scipy.misc import imresize&#xA;import numpy as np&#xA;from sklearn.model_selection import train_test_split&#xA;import os&#xA;from glob import glob&#xA;import tflearn&#xA;from tflearn.data_utils import shuffle, to_categorical&#xA;from tflearn.layers.core import input_data, dropout, fully_connected&#xA;from tflearn.layers.conv import conv_2d, max_pool_2d&#xA;from tflearn.layers.estimator import regression&#xA;from tflearn.data_preprocessing import ImagePreprocessing&#xA;from tflearn.data_augmentation import ImageAugmentation&#xA;from tflearn.metrics import Accuracy&#xA;###################################&#xA;### Import picture files&#xA;###################################&#xA;&#xA;files_path = 'dataset/train'&#xA;cat_files_path = os.path.join(files_path, 'cat*.jpg')&#xA;dog_files_path = os.path.join(files_path, 'dog*.jpg')&#xA;test_files_path = os.path.join('dataset/test1', '*.jpg')&#xA;&#xA;cat_files = sorted(glob(cat_files_path))&#xA;dog_files = sorted(glob(dog_files_path))&#xA;test_files = sorted(glob(test_files_path))&#xA;&#xA;n_files = len(cat_files) + len(dog_files)&#xA;test_n = len(test_files)&#xA;print(test_n)&#xA;size_image = 64&#xA;allX = np.zeros((n_files, size_image, size_image,3), dtype='float64')&#xA;ally = np.zeros(n_files)&#xA;testX = np.zeros((test_n, size_image, size_image, 3), dtype='float64')&#xA;count = 0&#xA;for f in test_files:&#xA;    try:&#xA;        img = io.imread(f)&#xA;        new_img = imresize(img, (size_image,size_image,3))&#xA;        testX[count] = np.array(new_img)&#xA;        count += 1&#xA;    except:&#xA;        continue&#xA;count = 0&#xA;for f in cat_files:&#xA;    try:&#xA;        img = io.imread(f)&#xA;        new_img = imresize(img, (size_image,size_image,3))&#xA;        allX[count] = np.array(new_img)&#xA;        ally[count] = 0&#xA;        count += 1&#xA;    except:&#xA;        continue&#xA;for f in dog_files:&#xA;    try:&#xA;        img = io.imread(f)&#xA;        new_img = imresize(img, (size_image,size_image,3))&#xA;        allX[count] = np.array(new_img)&#xA;        ally[count] = 1&#xA;        count += 1&#xA;    except:&#xA;        continue&#xA;&#xA;###################################&#xA;# Prepare train &amp;amp; test samples&#xA;###################################&#xA;&#xA;# test-train split&#xA;X, X_test, Y, Y_test = train_test_split(allX,ally,test_size=0.1, random_state=42)&#xA;&#xA;#encode the Ys&#xA;Y = to_categorical(Y,2)&#xA;Y_test = to_categorical(Y_test, 2)&#xA;&#xA;###################################&#xA;# Image transformations&#xA;###################################&#xA;&#xA;# normalisation of images&#xA;img_prep = ImagePreprocessing()&#xA;img_prep.add_featurewise_zero_center()&#xA;img_prep.add_featurewise_stdnorm()&#xA;&#xA;# Create extra synthetic training data by flipping &amp;amp; rotating images&#xA;img_aug = ImageAugmentation()&#xA;img_aug.add_random_flip_leftright()&#xA;img_aug.add_random_rotation(max_angle=25.)&#xA;&#xA;###################################&#xA;# Define network architecture&#xA;###################################&#xA;&#xA;# Input is a 32x32 image with 3 color channels (red, green and blue)&#xA;network = input_data(shape=[None, 64, 64, 3],&#xA;                     data_preprocessing=img_prep,&#xA;                     data_augmentation=img_aug)&#xA;&#xA;# 1: Convolution layer with 32 filters, each 3x3x3&#xA;conv_1 = conv_2d(network, 32, [3,3], activation='relu', name='conv_1')&#xA;&#xA;# 2: Max pooling layer&#xA;network = max_pool_2d(conv_1, [3,3])&#xA;&#xA;# 3: Convolution layer with 64 filters&#xA;conv_2 = conv_2d(network, 64, [3,3], activation='relu', name='conv_2')&#xA;&#xA;# 2: Max pooling layer&#xA;network = max_pool_2d(conv_2, [3,3])&#xA;&#xA;# 4: Convolution layer with 64 filters&#xA;conv_3 = conv_2d(network, 32, [3,3], activation='relu', name='conv_3')&#xA;&#xA;# 5: Max pooling layer&#xA;network = max_pool_2d(conv_3, [3,3])&#xA;&#xA;# 5: Convolution layer with 64 filters&#xA;conv_4 = conv_2d(network, 64, [3,3], activation='relu', name='conv_4')&#xA;&#xA;# 6: Max pooling layer&#xA;network = max_pool_2d(conv_4, [3,3])&#xA;&#xA;# 7: Convolution layer with 64 filters&#xA;conv_5 = conv_2d(network, 32, [3,3], activation='relu', name='conv_5')&#xA;&#xA;# 8: Max pooling layer&#xA;network = max_pool_2d(conv_5, [3,3])&#xA;&#xA;# 9: Convolution layer with 64 filters&#xA;conv_6 = conv_2d(network, 64, [3,3], activation='relu', name='conv_6')&#xA;&#xA;# 10: Max pooling layer&#xA;network = max_pool_2d(conv_6, [3,3])&#xA;&#xA;# 11: Fully-connected 512 node layer&#xA;network = fully_connected(network, 1024, activation='relu')&#xA;&#xA;# 12: Dropout layer to combat overfitting&#xA;network = dropout(network, 0.8)&#xA;&#xA;# 13: Fully-connected layer with two outputs&#xA;network = fully_connected(network, 2, activation='softmax')&#xA;&#xA;# Configure how the network will be trained&#xA;acc = Accuracy(name=&quot;Accuracy&quot;)&#xA;network = regression(network, optimizer='adam',&#xA;                     loss='categorical_crossentropy',&#xA;                     learning_rate=0.0005, metric=acc)&#xA;&#xA;# Wrap the network in a model object&#xA;model = tflearn.DNN(network, checkpoint_path='model_cat_dog_6.tflearn', max_checkpoints = 3,&#xA;                    tensorboard_verbose = 3, tensorboard_dir='tmp/tflearn_logs/')&#xA;&#xA;model.fit(X, Y, validation_set=(X_test, Y_test), batch_size=500,&#xA;      n_epoch=10, run_id='cat_dog_model', show_metric=True)&#xA;model.save('cat_dog_model_final.tflearn')&#xA;# choose images &amp;amp; plot the first one&#xA;result = []&#xA;logloss = 0&#xA;&#xA;with open('deneme.csv','wb') as cs:&#xA;    writer = csv.writer(cs, delimiter=',',quoting=csv.QUOTE_NONE)&#xA;    for i in range(0, len(testX)):&#xA;        im = [testX[i]]&#xA;        a = model.predict(im)&#xA;        if a[0][0]&amp;lt;a[0][1]:&#xA;            writer.writerow([str(i+1), 1])&#xA;        else:&#xA;            writer.writerow([str(i + 1), 0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17036" LastActivityDate="2017-07-04T20:00:42.560" Title="How to improve my test accuracy using CNN in Tensorflow" Tags="|machine-learning|python|neural-network|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20105" PostTypeId="1" CreationDate="2017-07-01T19:25:43.600" Score="-1" ViewCount="659" Body="&lt;p&gt;I'm trying to build a model that would predict the caco-2 coefficient of a molecule given its smiles string representation.&#xA;My solution is based on &lt;a href=&quot;http://www.rdkit.org/docs/Cookbook.html#using-scikit-learn-with-rdkit&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; example.&#xA;Since I need to predict a real value, I use a &lt;code&gt;RandomForestRegressor&lt;/code&gt;.&#xA;With some molecules added to the code manually, everything works (although the predictions themselves are wildly wrong):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from rdkit import Chem, DataStructs     #all the nice chemical stuff, ConvertToNumpyArray&#xA;from rdkit.Chem import AllChem&#xA;from sklearn.ensemble import RandomForestRegressor      #our regressor&#xA;from sklearn.model_selection import train_test_split    &#xA;import numpy as np&#xA;&#xA;&#xA;&#xA;# generate molecules&#xA;m1 = Chem.MolFromSmiles('Cc1ccc(NNC(=O)c2ccc(CN3C(=O)CCC3=O)cc2)cc1Cl')&#xA;m2 = Chem.MolFromSmiles('Nc1ccc(C(=O)N2CCN(c3cc[nH+]cc3)CC2)cc1[N+](=O)[O-]')&#xA;m3 = Chem.MolFromSmiles('CN(Cc1[nH+]ccn1C)C(=O)CCc1ccsc1')&#xA;m4 = Chem.MolFromSmiles('COc1ccc([N+](=O)[O-])cc1C(=O)NCCC[NH+]1CCCC1')&#xA;m5 = Chem.MolFromSmiles('C[NH+]1CCN(S(=O)(=O)c2ccc(NC(=O)Cc3ccc([N+](=O)[O-])cc3)cc2)CC1')&#xA;m6 = Chem.MolFromSmiles('CCc1ccc(S(=O)(=O)Nc2ccc(NC(C)=O)cc2)cc1')&#xA;m7 = Chem.MolFromSmiles('O=C(COC(=O)c1ccc(S(=O)(=O)N2CCCCC2)cc1)c1ccc(F)cc1')&#xA;m8 = Chem.MolFromSmiles('COC(=O)c1ccc(S(=O)(=O)NCc2csc3ccc(Cl)cc23)n1C')&#xA;m9 = Chem.MolFromSmiles('CCC(C)N1C(=O)C(=CNc2ccccc2C(=O)[O-])C(=O)NC1=S')&#xA;m10 = Chem.MolFromSmiles('Cn1c(CNC(=O)C(=O)Nc2cccc(Cl)c2Cl)nc2ccccc21')&#xA;mols = [m1, m2, m3, m4, m5 ,m6, m7, m8, m9, m10]&#xA;&#xA;&#xA;# generate fingeprints: Morgan fingerprint with radius 2&#xA;fps = [AllChem.GetMorganFingerprintAsBitVect(m, 2) for m in mols]&#xA;&#xA;# convert the RDKit explicit vectors into numpy arrays&#xA;np_fps = []&#xA;for fp in fps:&#xA;  arr = np.zeros((1,))&#xA;  DataStructs.ConvertToNumpyArray(fp, arr)&#xA;  np_fps.append(arr)&#xA;&#xA;# get a random forest regressor with 100 trees&#xA;rndf_rgsr = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1, warm_start=False)&#xA;&#xA;&#xA;#train the random forest&#xA;#ys are the caco-2 coefficients we wish to predict&#xA;ys_fit = [379.724, 101.644, 3154.167, 97.437, 21.152, 569.981, 150.55, 690.843, 78.866, 984.371]&#xA;&#xA;rndf_rgsr.fit(np_fps, ys_fit)&#xA;&#xA;&#xA;#use the random forest to predict a new molecule&#xA;m_new = Chem.MolFromSmiles('Cc1n[nH]c(Cc2ccc(-n3cnnc3)cc2)n1')      #actual caco2 is 410.037&#xA;fp = np.zeros((1,))&#xA;DataStructs.ConvertToNumpyArray(AllChem.GetMorganFingerprintAsBitVect(m_new, 2), fp)&#xA;&#xA;print(rndf_rgsr.predict((fp,)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I try to work with a lot of molecules imported from a file, which contains a lot of lines that look like &lt;code&gt;Cc1ccc(NNC(=O)c2ccc(CN3C(=O)CCC3=O)cc2)cc1Cl,379.724&lt;/code&gt;, using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from rdkit import Chem, DataStructs     &#xA;from rdkit.Chem import AllChem&#xA;from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor     #our regressors&#xA;from sklearn.tree import DecisionTreeRegressor&#xA;from sklearn.model_selection import train_test_split    &#xA;import numpy as np&#xA;import pandas as pd&#xA;from pandas import DataFrame, read_csv&#xA;&#xA;&#xA;&#xA;#import our data from file&#xA;df = pd.read_csv('test_db.csv', delimiter=',' )     #a pandas DataFrame&#xA;&#xA;&#xA;#get the values of variables and targets&#xA;X = df[&quot;smiles&quot;].values&#xA;y = df[&quot;Caco2&quot;].values&#xA;&#xA;&#xA;&#xA;#split our data set into two parts&#xA;x_train, x_eval, y_train, y_eval = train_test_split(X, y, test_size = 0.2, random_state = 42)   &#xA;&#xA;&#xA;#convert our smiles string into actual molecular graphs&#xA;mols_ready_train = [Chem.MolFromSmiles(x_train[i]) for i in range(len(x_train))]&#xA;mols_ready_eval = [Chem.MolFromSmiles(x_eval[i]) for i in range(len(x_eval))]&#xA;&#xA;# generate fingeprints: Morgan fingerprint with radius 2    &#xA;fing_prints_train = [AllChem.GetMorganFingerprintAsBitVect(m, 2) for m in mols_ready_train]&#xA;fing_prints_eval = [AllChem.GetMorganFingerprintAsBitVect(m, 2) for m in mols_ready_eval]&#xA;&#xA;&#xA;# convert the RDKit explicit vectors into numpy arrays&#xA;np_fps_train = []&#xA;for fp in fing_prints_train:&#xA;  arr = np.zeros((1,))&#xA;  DataStructs.ConvertToNumpyArray(fp, arr)&#xA;  np_fps_train.append(arr)&#xA;&#xA;np_fps_eval = []&#xA;for fp in fing_prints_eval:&#xA;  arr = np.zeros((1,))&#xA;  DataStructs.ConvertToNumpyArray(fp, arr)&#xA;  np_fps_eval.append(arr)  &#xA;&#xA;&#xA;# get a random forest regressor with 100 trees&#xA;&#xA;rndf_rgsr = RandomForestRegressor(n_estimators=1000, random_state=42, n_jobs=-1, warm_start=False)&#xA;&#xA;&#xA;#train our random forest regressor&#xA;rndf_rgsr.fit(np_fps_train, y_train)&#xA;&#xA;&#xA;&#xA;# use the random forest to predict a new molecule&#xA;m_new = Chem.MolFromSmiles('Cc1n[nH]c(Cc2ccc(-n3cnnc3)cc2)n1')&#xA;&#xA;fp = numpy.zeros((1,))&#xA;DataStructs.ConvertToNumpyArray(AllChem.GetMorganFingerprintAsBitVect(m_new, 2), fp)&#xA;&#xA;print(rndf_rgsr.predict((fp,)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it crashes with the following error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;File &quot;/home/me/predictor.py&quot;, line 55, in &#xA;      rndf_rgsr.fit(np_fps_train, y_train)   File &quot;/usr/local/lib/python2.7/dist-packages/sklearn/ensemble/forest.py&quot;,&#xA;  line 248, in fit&#xA;      y = check_array(y, accept_sparse='csc', ensure_2d=False, dtype=None)   File&#xA;  &quot;/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py&quot;,&#xA;  line 407, in check_array&#xA;      _assert_all_finite(array)   File &quot;/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py&quot;,&#xA;  line 58, in _assert_all_finite&#xA;      &quot; or a value too large for %r.&quot; % X.dtype) ValueError: Input contains NaN, infinity or a value too large for dtype('float64').&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've checked that no vectors I use contain &lt;code&gt;nan&lt;/code&gt;s or &lt;code&gt;inf&lt;/code&gt;s. The fingerprints used here are 2048 bits long, but I doubt they're the source of the problem.&#xA;Something is going wrong with validation, but I can't really see what.&#xA;Could you provide any hints?&lt;/p&gt;&#xA;" OwnerUserId="8214" LastActivityDate="2021-02-06T18:42:27.963" Title="ValueError when doing validation with random forests" Tags="|python|random-forest|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20108" PostTypeId="2" ParentId="20100" CreationDate="2017-07-01T21:36:37.337" Score="3" Body="&lt;p&gt;What about doing cross validation on your training set? Once you have the different train/test splits I would start by printing the accuracy (number of correct predictions / total predictions) and the confussion matrix for each method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using python sklearn.metrics offers a wide variety of useful functions.&lt;/p&gt;&#xA;" OwnerUserId="31736" LastActivityDate="2017-07-01T21:36:37.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20111" PostTypeId="1" CreationDate="2017-07-02T08:17:26.043" Score="1" ViewCount="73" Body="&lt;p&gt;Suppose initial data is &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; and need to calculate the inverse of covariance of matrix &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; i.e.&#xA;&lt;span class=&quot;math-container&quot;&gt;$C = cov(D,D)$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$cov$&lt;/span&gt; represents covariance.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$B = inv(C)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now, new data &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; appears. So matrix D and C both will updated as follows:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$D^{new} = \begin{bmatrix}&#xA;D\\ &#xA;N&#xA;\end{bmatrix}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C^{new} = \begin{bmatrix}&#xA;cov(D,D) &amp;amp; cov(D,D^{new})\\ &#xA;cov(D^{new},D) &amp;amp; cov(D^{new},D^{new})&#xA;\end{bmatrix} = \begin{bmatrix}&#xA;C &amp;amp; cov(D,D^{new})\\ &#xA;cov(D,D^{new})^T &amp;amp; cov(D^{new},D^{new})&#xA;\end{bmatrix}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Similarly, data will be updated continuously. &lt;strong&gt;Now, inverse of &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; (i.e. &lt;span class=&quot;math-container&quot;&gt;$B$&lt;/span&gt;) is given and need to calculate inverse of &lt;span class=&quot;math-container&quot;&gt;$B^{new}$&lt;/span&gt;&lt;/strong&gt;. However, I don't want to use direct or general inverse as it will be very expensive if matrix size will be large and also want to utilize the inverse calculated for old data i.e. &lt;span class=&quot;math-container&quot;&gt;$B$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;P.S: I have tried inverse of block or partitioned matrix but it doesn't yield similar result as general inverse in all cases (&lt;a href=&quot;https://math.stackexchange.com/questions/2341605/why-i-am-getting-difference-between-block-inverse-and-general-inverse&quot;&gt;for more detail&lt;/a&gt;). If someone needs then I can provide the MATLAB code to verify it.&lt;/p&gt;&#xA;" OwnerUserId="18818" LastEditorUserId="98307" LastEditDate="2020-08-11T07:09:08.520" LastActivityDate="2020-08-11T07:09:08.520" Title="How to calculate inverse of square matrix for streaming or online data as all data are not available at once?" Tags="|machine-learning|neural-network|online-learning|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="20112" PostTypeId="2" ParentId="10025" CreationDate="2017-07-02T11:37:14.137" Score="5" Body="&lt;p&gt;RNNs were not producing good enough results and are also hard to train so I went with CNNs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because a specific animal sound is only a few seconds long we can divide the spectrogram into chunks. I used a length of 3 seconds. We then perform classification on each chunk and average the outputs to create a single prediction per audio file. This works really well and is also simple to implement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more in-depth explanation can be found here:&#xA;&lt;a href=&quot;http://ceur-ws.org/Vol-1609/16090547.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://ceur-ws.org/Vol-1609/16090547.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15847" LastActivityDate="2017-07-02T11:37:14.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20113" PostTypeId="1" AcceptedAnswerId="20124" CreationDate="2017-07-02T12:19:20.067" Score="4" ViewCount="1114" Body="&lt;p&gt;I know the concept of a neural network, and I followed the Machine Learning course by Andrew Ng on Coursera, so I also coded some simple ones. However, I miss all the new tricks which are useful to prevent overfitting of Deep networks, such as for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using ReLU instead than tanh neurons&lt;/li&gt;&#xA;&lt;li&gt;using dropout&lt;/li&gt;&#xA;&lt;li&gt;more advanced learning methods than just vanilla stochastic gradient descent&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;etc. I would like to follow a MOOC which tells me how to use Keras for Deep Learning (I like Keras very much because IMO is much easier to understand than other packages, but I'm open to suggestions). I would also be content with a book, but I'd really prefer a MOOC. Doesn't have to be free. Can you indicate me one? The application is Data Science, but generic Deep Learning would do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: to provide more context to the question, my main applications would be Internet of Things Analytics, i.e., applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc. I &lt;strong&gt;don't need&lt;/strong&gt; to develop the cloud platform: I just need to develop the &quot;core&quot; Analytics. Think of it as just applying Deep Learning to Time Series or to Classification problems. However, methods which can easily be retrained when new data arrive, without having to go through the full dataset again, would be preferred.&lt;/p&gt;&#xA;" OwnerUserId="27526" LastEditorUserId="31513" LastEditDate="2017-07-11T12:37:30.083" LastActivityDate="2017-07-11T12:37:30.083" Title="MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks" Tags="|python|keras|reference-request|books|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20114" PostTypeId="1" AcceptedAnswerId="20135" CreationDate="2017-07-02T12:45:00.977" Score="-1" ViewCount="855" Body="&lt;p&gt;I have a dataset with some features. Over one feature, there are some values that appear only once. Can I safely remove these rows, or do they need to be kept ? &lt;/p&gt;&#xA;" OwnerUserId="33386" LastActivityDate="2017-08-02T16:06:16.117" Title="Do unique values has to be removed ?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20115" PostTypeId="2" ParentId="20105" CreationDate="2017-07-02T15:43:39.550" Score="0" Body="&lt;p&gt;Ok, I've been wrong in my assumptions.&lt;/p&gt;&#xA;&lt;p&gt;A simple:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(len(X)):&#xA;    if np.isnan(y[i]):&#xA;        print (&amp;quot;Here it is: &amp;quot;, i,X[i],y[i])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;has revealed about 200 &amp;quot;bad&amp;quot; lines in my dataset. Cleaning those up solved that particular problem.&lt;/p&gt;&#xA;" OwnerUserId="8214" LastEditorUserId="29169" LastEditDate="2021-02-06T18:42:27.963" LastActivityDate="2021-02-06T18:42:27.963" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20116" PostTypeId="2" ParentId="20113" CreationDate="2017-07-02T16:44:02.787" Score="3" Body="&lt;p&gt;Since you like Keras, the main author has written a &lt;a href=&quot;https://www.manning.com/books/deep-learning-with-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;book&lt;/a&gt; on deep learning with Python. You can check out the sections to see if it covers all the topics you want.&lt;/p&gt;&#xA;" OwnerUserId="33540" LastActivityDate="2017-07-02T16:44:02.787" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20117" PostTypeId="1" AcceptedAnswerId="22010" CreationDate="2017-07-02T17:08:52.097" Score="1" ViewCount="421" Body="&lt;p&gt;We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For images, papers we evaluate utilizes SIFT feature extraction, we use &lt;a href=&quot;http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;NUSWIDE&lt;/a&gt; for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/tutorials/image/imagenet&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt; and keras to extract features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so any idea why would resnet/inception perfrom worse than SIFT?&lt;/p&gt;&#xA;" OwnerUserId="34110" LastActivityDate="2017-08-06T13:48:16.287" Title="Inception/ResNet doing worse than SIFT in feature extraction" Tags="|deep-learning|feature-extraction|inception|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20118" PostTypeId="1" AcceptedAnswerId="20141" CreationDate="2017-07-02T20:02:23.657" Score="14" ViewCount="19797" Body="&lt;p&gt;In Data Science, many seem to be using &lt;a href=&quot;http://pandas.pydata.org/index.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;pandas&lt;/strong&gt;&lt;/a&gt; dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to &lt;strong&gt;regular relational databases&lt;/strong&gt; like &lt;a href=&quot;https://www.mysql.com/&quot; rel=&quot;noreferrer&quot;&gt;MySQL&lt;/a&gt;, which are used to store data in many other fields of programming?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While pandas does provide some useful functions for data exploration, you can't use SQL and you lose features like query optimization or access restriction.&lt;/p&gt;&#xA;" OwnerUserId="34116" LastEditorUserId="29575" LastEditDate="2017-07-03T06:05:09.213" LastActivityDate="2020-02-20T11:41:01.923" Title="Advantages of pandas dataframe to regular relational database" Tags="|pandas|databases|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20119" PostTypeId="2" ParentId="20118" CreationDate="2017-07-02T22:29:39.787" Score="2" Body="&lt;p&gt;From the pandas (&lt;a href=&quot;http://pandas.pydata.org/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Main Page&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Python Data Analysis Library¶&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While pandas can certainly access data via SQL, or from several other data storage methods, its primary purpose is to make it easier when using Python to do data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To that end pandas has various methods available that allow some &lt;a href=&quot;https://en.wikipedia.org/wiki/Relational_algebra&quot; rel=&quot;nofollow noreferrer&quot;&gt;relational algebra&lt;/a&gt; operations that can be compared to SQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also Pandas provides easy access to &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NumPy&lt;/a&gt;, which&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;is the fundamental package for scientific computing with Python. It contains among other things:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;a powerful N-dimensional array object&lt;/li&gt;&#xA;  &lt;li&gt;sophisticated (broadcasting) functions&lt;/li&gt;&#xA;  &lt;li&gt;tools for integrating C/C++ and Fortran code&lt;/li&gt;&#xA;  &lt;li&gt;useful linear algebra, Fourier transform, and random number capabilities&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-07-02T22:29:39.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20120" PostTypeId="1" AcceptedAnswerId="20122" CreationDate="2017-07-03T00:32:38.827" Score="0" ViewCount="171" Body="&lt;p&gt;I have sparse vectors and found that cosine similarity is very efficient to to measure the similarity. Now I want to cluster these vectors based on similarity. Hence, can someone please suggest/recommend clustering algorithms that make use of cosine similarity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: I do not have a predefined number of clusters beforehand and want the clustering algorithm itself to decide it.&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-07-03T02:13:53.387" LastActivityDate="2017-07-03T06:13:35.843" Title="Clustering algorithms" Tags="|machine-learning|data-mining|clustering|" AnswerCount="1" CommentCount="7" ClosedDate="2017-07-04T12:10:00.547" ContentLicense="CC BY-SA 3.0" />
  <row Id="20122" PostTypeId="2" ParentId="20120" CreationDate="2017-07-03T06:13:35.843" Score="2" Body="&lt;p&gt;You can see your affinity matrix as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Glossary_of_graph_theory_terms#weight&quot; rel=&quot;nofollow noreferrer&quot;&gt;weighted adjacency matrix&lt;/a&gt; of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph&lt;/a&gt; and apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Modularity_(networks)&quot; rel=&quot;nofollow noreferrer&quot;&gt;modularity&lt;/a&gt;-based &lt;a href=&quot;https://en.wikipedia.org/wiki/Louvain_Modularity&quot; rel=&quot;nofollow noreferrer&quot;&gt;community detection algorithms&lt;/a&gt; on that. Just note that modularity based algorithms have resolution problem i.e. finding very small communities is difficult in presence of large ones.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-07-03T06:13:35.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20123" PostTypeId="1" CreationDate="2017-07-03T06:15:34.423" Score="1" ViewCount="182" Body="&lt;p&gt;The dataset has 5 labels A,B,C,D,E&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A,B,C are majority and D&amp;amp;E are minority.The penalty to misclassify D&amp;amp;E are huge. How can I implement a cost sensitive learning. The input to the model will be english sentences like user comments.&lt;/p&gt;&#xA;" OwnerUserId="34127" LastActivityDate="2017-07-03T15:36:08.767" Title="How to overcome the class imbalance issue in a text classification problem?" Tags="|machine-learning|deep-learning|svm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20124" PostTypeId="2" ParentId="20113" CreationDate="2017-07-03T07:10:19.807" Score="2" Body="&lt;p&gt;These 2 are pretty popular:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://de.udacity.com/course/deep-learning--ud730/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://de.udacity.com/course/deep-learning--ud730/&lt;/a&gt; - Tensorflow-Course by Google&#xA;&lt;a href=&quot;http://www.fast.ai/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.fast.ai/&lt;/a&gt; - Deep Learning by a Kaggle hero&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is easy to follow and nicely presented, the second takes quite some time.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-03T07:10:19.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20125" PostTypeId="1" AcceptedAnswerId="20151" CreationDate="2017-07-03T07:52:09.843" Score="0" ViewCount="304" Body="&lt;p&gt;I have a CSV file having a bunch of sentences related to science. Before I do sentiment analysis on the sentences I want to programatically decide whether sentiment analysis is required on the sentence or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically some of the sentences are opinions of a particular topic in which case doing sentiment analysis makes sense. However some of the sentences are just definitions and sentiment analysis is not required on such sentences. So is there any way to detect the presence of a sentiment in a sentence?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that the length of a sentence varies from 9 to 30 words.)&lt;/p&gt;&#xA;" OwnerUserId="34106" LastActivityDate="2017-07-17T19:01:03.923" Title="How to check if sentiment analysis is required?" Tags="|machine-learning|nlp|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20127" PostTypeId="2" ParentId="20098" CreationDate="2017-07-03T10:02:43.900" Score="17" Body="&lt;p&gt;In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its values in a specific convenient range. We don't follow this practice for theoretical guarantees but for practical reasons. The same goes with clipping $Q$ value functions in Q-learning combined with NNs. Of course, there are some drawbacks with these approaches but in general the algorithm behaves better as the backpropagation does not lead your network weights to extreme values. Please take a look at this excellent post by &lt;a href=&quot;http://karpathy.github.io/2016/05/31/rl/&quot; rel=&quot;noreferrer&quot;&gt;Andrej Karpathy&lt;/a&gt; (I attach the part related to your question as a blockquote) which gives additional insights:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;More general advantage functions. I also promised a bit more discussion of the returns. So far we have judged the goodness of every individual action based on whether or not we win the game. In a more general RL setting we would receive some reward $r_t$ at every time step. One common choice is to use a discounted reward, so the “eventual reward” in the diagram above would become $R_t=∑^∞_{k=0}γ^kr_{t+k}$, where $\gamma$ is a number between 0 and 1 called a discount factor (e.g. 0.99). The expression states that the strength with which we encourage a sampled action is the weighted sum of all rewards afterwards, but later rewards are exponentially less important. In practice it can can also be important to normalize these. For example, suppose we compute $R_t$ for all of the 20,000 actions in the batch of 100 Pong game rollouts above. One good idea is to “standardize” these returns (e.g. subtract mean, divide by standard deviation) before we plug them into backprop. This way we’re always encouraging and discouraging roughly half of the performed actions. Mathematically you can also interpret these tricks as a way of controlling the variance of the policy gradient estimator. A more in-depth exploration can be found here.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2017-07-04T13:09:40.803" LastActivityDate="2017-07-04T13:09:40.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20128" PostTypeId="2" ParentId="20100" CreationDate="2017-07-03T10:02:45.643" Score="1" Body="&lt;p&gt;There are other metrics you can use to directly compare classifiers instead of accuracy, such as Precision and Recall to compare how your classifiers catch all possible true and false cases individually. This gives you a much better idea of &lt;em&gt;how&lt;/em&gt; your classifier performs on different cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;F-Measure&lt;/a&gt;(can be used for multiclass too!) and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matthews' Correlation Coefficient&lt;/a&gt; (binary classifications) to get a single metric that 'combines' the performance of your classifiers in several areas such as precision and recall to get a single comparable metric. &lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-07-03T10:02:45.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20129" PostTypeId="1" AcceptedAnswerId="20150" CreationDate="2017-07-03T11:37:52.103" Score="1" ViewCount="200" Body="&lt;p&gt;Thanks also to SE, I've recently changed job and now I'm working in Data Science, mainly on Analytics for the IoT (Internet of Things). Analytics are applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Until now, I've been using R to build Statistical Learning models on datasets which would fit in the memory of my workstation, so I'm not a novice for what it concerns Statistical modeling and Data Science. However, I'm a novice with Python, and I need to learn it, expecially the part of the Python ecosystem related to Data Science, because that's what my team uses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I &lt;strong&gt;don't need&lt;/strong&gt; to develop the cloud platform: I just need to develop the &quot;core&quot; Analytics. I got the book by Jake Van der Plas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1491912057&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.amazon.com/Python-Data-Science-Handbook-Essential/dp/1491912057&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but I would like to also follow a MOOC on using Python for Data Science. Can you suggest me one?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;DISCLAIMER&lt;/strong&gt; I already asked this question on CV and it wasn't considered very appropriate there. Since here my other question on MOOCs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/20113/mooc-or-book-on-deep-learning-in-python-for-someone-with-a-basic-knowledge-of-ne&quot;&gt;MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;was well-received on this site, I thought of asking again here, &lt;strong&gt;after&lt;/strong&gt; deleting the one on CV (no cross-posting). Hope this is fine.&lt;/p&gt;&#xA;" OwnerUserId="27526" LastActivityDate="2017-07-03T23:06:07.607" Title="MOOCs for Python in Data Science" Tags="|python|reference-request|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20130" PostTypeId="1" CreationDate="2017-07-03T12:12:43.060" Score="1" ViewCount="35" Body="&lt;p&gt;Imagine I setup a Kaggle competition with normalized stock data (e.g. price, volume, etc) &lt;em&gt;plus&lt;/em&gt; a random rotation matrix (i.e. so that it's less obvious what the features are). Is it possible for a contestants to figure out that the data is stock-related? If yes, how?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The contest will be presented as a general machine learning contest with no specific background info. The data will &lt;em&gt;not&lt;/em&gt; be time-series - just a set of samples, each containing a list of unnamed features plus a binary label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I asked the question, I was wondering more about whether contestants might figure out by examine the data using data/statistical analysis (e.g. PCA) or unsupervised machine learning methods (e.g. clustering).&lt;/p&gt;&#xA;" OwnerUserId="29428" LastEditorUserId="29428" LastEditDate="2017-07-04T20:59:37.573" LastActivityDate="2017-07-04T20:59:37.573" Title="Is it possible to detect which field does a rotated &quot;kaggle&quot; contest data come from?" Tags="|kaggle|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20131" PostTypeId="1" CreationDate="2017-07-03T12:42:49.527" Score="1" ViewCount="68" Body="&lt;p&gt;When I deploy my web service in AZURE Machine learning, is it possible to have NULL or empty in my test data set? Or I have to fill all the information to get the results of my prediction?&lt;/p&gt;&#xA;" OwnerUserId="32220" LastActivityDate="2018-01-03T14:05:27.763" Title="Deployed Web service" Tags="|azure-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20132" PostTypeId="1" AcceptedAnswerId="20137" CreationDate="2017-07-03T15:03:38.360" Score="1" ViewCount="4972" Body="&lt;p&gt;I have a dataset containing several features, which have class values (e.g. DBF4, JUL23, ...). In a classification problem and when using decision trees, is it better to convert these values as new binary features: so DBF4 will become a feature, which value is either 0 or 1, or is it better to keep them as they are. Knowing that there are a lot of values (more than the number or rows actually). If this is the case, is there a reason why is it better ?  &lt;/p&gt;&#xA;" OwnerUserId="33386" LastEditorUserId="33386" LastEditDate="2017-07-04T07:52:57.757" LastActivityDate="2017-07-04T07:52:57.757" Title="Is it better to have binary features rather than class ones" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20133" PostTypeId="1" CreationDate="2017-07-03T15:14:24.617" Score="7" ViewCount="940" Body="&lt;p&gt;I’ve playing with the MovieLens ratings dataset under Spark’s ALS and a manual implementation of ALS and comparing results with the same hyperparameters. I’d like to know this exactly in order to make some experiments that could be reproducible. I’ve seen several papers describing the exact loss formula for ALS in different ways. These seem to be quite common:&#xA;$$1)\: min \sum (r_{ui} - \mu – b_u – b_i – p_u^Tq_i)^2 + \lambda (\lVert \mathbf{p_u} \rVert^2  + (\lVert \mathbf{q_i} \rVert^2 + b_u^2 + b_i^2)$$&#xA;(The ratings are centered by subtracting an overall bias $\mu$, and for each user, a bias $b_u$ which is a part of the model, and part of the regularization, and a similar bias for each movie $b_i$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2)\: min \sum (r_{uj} – u_i^Tm_j)^2 + \lambda (n_{u_{i}} \lVert \mathbf{ u_i } \rVert^2  + n_{m_{j}} \lVert \mathbf{m_j} \rVert^2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The penalization for each user-feature and movie-feature vector are multiplied by the number of ratings from each user/movie, and the ratings might be centered beforehand, but the biases are not model parameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I’ve also seen other variations where the l2-norms of the vectors are not squared.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’ve playing with different variations of these formulas and comparing the results on the same data with Spark’s implementation and my manual implementation, and I cannot find any variation which would make the same hyperparameters in both implementations (the number of latent factors and the value of lambda) give similar results with the same data (e.g. under some formulas, with the same number of latent factors and using the same value for lambda that in Spark gives good cross-validated results, in my implementation results in either overfitting badly or making all the predictions close to zero, but a totally different value of lambda gives comparable results).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What formula does Spark use? Also, how are the initial matrices initialized? I’ve seen some suggestions saying to let the first column of the movie-factor matrix be the average rating for the movie and the rest random-normal numbers, while others suggest random uniform numbers with quite big signs. How is the first movie-factor matrix initialized?&lt;/p&gt;&#xA;" OwnerUserId="34153" LastEditorUserId="34153" LastEditDate="2017-07-03T16:26:27.143" LastActivityDate="2017-07-03T16:26:27.143" Title="ALS in Spark: what loss function is it minimizing?" Tags="|apache-spark|matrix-factorisation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20134" PostTypeId="2" ParentId="20123" CreationDate="2017-07-03T15:24:53.250" Score="2" Body="&lt;p&gt;Option 1: play with the thresholds for classifying something – perhaps you could set a threshold for classifying something as a minority class should even if it doesn't have the highest probability/score, or a ratio of probabilities/thresholds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 2: upsample your data (take resamples with replacement of the minority classes until they have as many observations as the others).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 3: use some cost-sensitive algorithm (e.g. some forms of classification trees) - probably won't translate into very accurate results.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastEditorUserId="34153" LastEditDate="2017-07-03T15:36:08.767" LastActivityDate="2017-07-03T15:36:08.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20135" PostTypeId="2" ParentId="20114" CreationDate="2017-07-03T15:42:59.223" Score="1" Body="&lt;p&gt;If you have a categorical feature of which one level appears only once in the data, then an algorithm that estimates its parameters with that data will assign all the residual for that observation to that feature (unless you have some regularization in place).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, that would probably be a very incorrect estimate, as it will just try to correct all the errors from the other parameters, and as such will probably not generalize to newer observations. This of course depends on the data that you have, the number of features, the goodness of fit (e.g. if your goodness of fit or classification accuracy is something like 99% then perhaps it might not be so bad to include that feature/row, especially if you are using high regularization parameters) and other factors, but in most cases you’d be better off leaving that feature out, or if possible, merging that into another category.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-07-03T15:42:59.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20136" PostTypeId="1" CreationDate="2017-07-03T15:43:13.003" Score="0" ViewCount="1093" Body="&lt;p&gt;I have implemented a neural network with 1 hidden layer using sigmoid activation unit but after watching a video on how relu activation function can be much faster a tried implementing it but the cost function is either &lt;code&gt;Nan&lt;/code&gt; or &lt;code&gt;inf&lt;/code&gt; in python. I have found that many people have the same problem but I couldn't find any solution on the internet.&#xA;I have added '#changed' comment to the lines that I have changed from the previous version in which I used sigmoid activation function.&#xA;Related python code snippets:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def relu(arg):  #I have tried both relu and leaky relu&#xA;    return 1*(arg&amp;lt;0)*0.0001*arg + (arg&amp;gt;=0)*arg&#xA;&#xA;def reluGrad(arg):&#xA;    for i in range(arg.shape[0]):&#xA;        for j in range(arg.shape[1]):&#xA;            if arg[i][j]&amp;gt;0:&#xA;                arg[i][j]=1&#xA;            else:&#xA;                arg[i][j]=0&#xA;    return arg&#xA;def softmax(x):&#xA;    x = x.transpose()&#xA;    e_x = np.exp(x - np.max(x))&#xA;    return (e_x / e_x.sum(axis=0)).transpose()&#xA;&#xA;#forward prop:&#xA;a1 = np.insert(data,0,np.ones(len(data)),1).astype(np.float64)&#xA;    z2 = a1.dot(theta1)&#xA;    a2 = relu(z2) #changed&#xA;    a2 = np.insert(a2,0,np.ones(len(a2)),1)&#xA;    z3 = a2.dot(theta2)&#xA;    a3 = softmax(z3) #changed&#xA;#compute the cost:&#xA;    cost = -(output*(np.log(a3))+(1-output)*(np.log(1-a3))).sum()&#xA;    cost = (1/len(data))*cost + (lamb/(2*len(data)))*((np.delete(theta1,0,0)**2).sum() + (np.delete(theta2,0,0)**2).sum())&#xA;&#xA;backProp:&#xA;    sigma3 = a3-output&#xA;    sigma2 = (sigma3.dot(np.transpose(theta2)))* reluGrad(np.insert(z2,0,np.ones(len(z2)),1)) #changed&#xA;    sigma2 = np.delete(sigma2,0,1)&#xA;    delta2 = (np.transpose(a2)).dot(sigma3)&#xA;    delta1 = (np.transpose(a1)).dot(sigma2)&#xA;&#xA;    grad1 = delta1/len(data) + (lamb/len(data))*np.insert(np.delete(theta1,0,0),0,np.zeros(len(theta1[0])),0)&#xA;    grad2 = delta2/len(data) + (lamb/len(data))*np.insert(np.delete(theta2,0,0),0,np.zeros(len(theta2[0])),0)&#xA;&#xA;#update theta&#xA;    theta1 = theta1 - alpha*grad1&#xA;    theta2 = theta2 - alpha*grad2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is causing this problem ? How can it be fixed ?&lt;/p&gt;&#xA;" OwnerUserId="24077" LastEditorUserId="24077" LastEditDate="2017-07-04T01:15:46.800" LastActivityDate="2017-07-04T01:15:46.800" Title="Cost tends to infinity when relu activation is used" Tags="|machine-learning|python|neural-network|classification|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20137" PostTypeId="2" ParentId="20132" CreationDate="2017-07-03T15:52:13.120" Score="5" Body="&lt;p&gt;In general, it’s better NOT to binarize them if the decision tree algorithm that you are using supports categorical features, as variables with fewer levels are less likely to be used in splits. Note however that not all implementations in all software are able to use categorical features without one-hot encoding – for example, most R implementations of decision trees and tree ensembles support categorical features natively, whereas scikit-learn’s and spark’s do not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here’s a blog post with a comparison of categorical as-is vs. one-hot encoded for random forests:&#xA;&lt;a href=&quot;https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/&quot; rel=&quot;noreferrer&quot;&gt;https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-07-03T15:52:13.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20138" PostTypeId="2" ParentId="20095" CreationDate="2017-07-03T15:59:55.807" Score="1" Body="&lt;p&gt;You of course cannot use them as training data, but they can still have some potential uses, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-You can make predictions on that data and see if the distribution of the classes are similar to those in the labeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-You can use them for feature engineering – e.g. if you are using something like principal components, or if you are using averages for some categorical variable (e.g. you have a variable like state and another like income, and you augment the state variable by creating a feature that is the average income for that state).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-If you need to impute missing values in the predictors, it’s better to use the whole dataset for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-You can see if the distributions of the predictor variables are similar, and if you see that, for example, in your test data you tend to predict poorly for observations with a low value of variable ‘x1’ and the data with no labels has mostly low values of ‘x1’, then you should perhaps change your model.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-07-03T15:59:55.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20139" PostTypeId="1" AcceptedAnswerId="20142" CreationDate="2017-07-03T17:03:24.397" Score="27" ViewCount="42546" Body="&lt;p&gt;I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation  when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z1=x.dot(theta1)+b1&#xA;&#xA;h1=1/(1+np.exp(-z1))&#xA;z2=h1.dot(theta2)+b2&#xA;h2=1/(1+np.exp(-z2))&#xA;&#xA;dh2=h2-y&#xA;#back prop&#xA;&#xA;dz2=dh2*(1-dh2)&#xA;H1=np.transpose(h1)&#xA;dw2=np.dot(H1,dz2)&#xA;db2=np.sum(dz2,axis=0,keepdims=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I looked up online for the code, and i want to know &#xA;why do we add up the matrix and then the scalar &lt;code&gt;db2=np.sum(dz2,axis=0,keepdims=True)&lt;/code&gt; is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because  &lt;code&gt;z2=h1.dot(theta2)+b2&lt;/code&gt; h1 and theta will be 0 and b2 will be 1. So the upper term will be left.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b2+=-alpha*db2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="34042" LastEditorUserId="836" LastEditDate="2017-07-03T19:21:35.803" LastActivityDate="2022-03-02T17:43:16.703" Title="Gradients for bias terms in backpropagation" Tags="|python|neural-network|backpropagation|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20141" PostTypeId="2" ParentId="20118" CreationDate="2017-07-03T17:35:16.983" Score="8" Body="&lt;p&gt;I think the premise of your question has a problem. Pandas is not a &quot;datastore&quot; in the way an RDBMS is. Pandas is a Python library for manipulating data that will fit in memory. Disadvantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pandas does not persist data. It even has a (slow) function called TO_SQL that will persist your pandas data frame to an RDBMS table.&lt;/li&gt;&#xA;&lt;li&gt;Pandas will only handle results that fit in memory, which is easy to fill. You can either use dask to work around that, or you can work on the data in the RDBMS (which uses all sorts of tricks like temp space) to operate on data that exceeds RAM. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-03T17:35:16.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20142" PostTypeId="2" ParentId="20139" CreationDate="2017-07-03T19:13:00.133" Score="12" Body="&lt;p&gt;The bias term is very simple, which is why you often don't see it calculated. In fact&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;db2 = dz2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So your update rules for bias on a &lt;em&gt;single item&lt;/em&gt; are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;b2 += -alpha * dz2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;b1 += -alpha * dz1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of the maths, if your loss is $J$, and you know $\frac{\partial J}{\partial z_i}$ for a given neuron $i$ which has bias term $b_i$ . . .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial J}{\partial b_i} = \frac{\partial J}{\partial z_i} \frac{\partial z_i}{\partial b_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial z_i}{\partial b_i} = 1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;because $z_i = (\text{something unaffected by } b_i) + b_i$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It looks like the code you copied uses the form&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;db2=np.sum(dz2,axis=0,keepdims=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;because the network is designed to process examples in (mini-)batches, and you therefore have gradients calculated for more than one example at a time. The sum is squashing the results down to a single update. This would be easier to confirm if you also showed update code for weights.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-03T19:28:40.337" LastActivityDate="2017-07-03T19:28:40.337" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20143" PostTypeId="1" CreationDate="2017-07-03T19:30:54.510" Score="5" ViewCount="5894" Body="&lt;p&gt;It seems to me that machine learning (especially deep learning) can work with thousands (even millions) of different inputs. After training an ML model on the inputs, the computer has hopefully &quot;learned&quot; something about the connections between these millions of inputs, and can hopefully make predictions regarding new unseen inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the difference between machine learning and data analysis?&lt;/p&gt;&#xA;" OwnerUserId="34160" LastEditorDisplayName="user10640" LastEditDate="2018-03-19T01:03:35.320" LastActivityDate="2019-11-11T09:01:50.470" Title="What is the difference between data analysis and machine learning?" Tags="|machine-learning|deep-learning|data-analysis|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20145" PostTypeId="2" ParentId="20143" CreationDate="2017-07-03T20:00:32.003" Score="1" Body="&lt;p&gt;For me, data analysis represents something like....&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What is an insightful KPI to get out of our data? If we are an&#xA;eComm retailer, for example, that might be what % of page visits did&#xA;we have to pages which we still had in stock?    or&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What percentage of orders are dispatched within 2 days?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Machine learning, is a whole different game. You might re-state Q1 and Q2, for example:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Can we build a predictive algorithm that predicts what pages&#xA;will be out of stock?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can we build an optimizer which helps us to dispatch orders faster?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In some deeper level, data analysis you are typically using the machine to automate some fairly trivial tasks. With machine learning, you are letting the machine automate a lot of the learning process - where you would be typically passing in that knowledge as a data analyst.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that help clarify at all?&lt;/p&gt;&#xA;" OwnerUserId="31154" LastEditorUserId="40853" LastEditDate="2018-03-19T01:29:24.587" LastActivityDate="2018-03-19T01:29:24.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20146" PostTypeId="2" ParentId="20118" CreationDate="2017-07-03T20:01:51.593" Score="2" Body="&lt;p&gt;Pandas is an in-memory data storage tool. This allows you to do very rapid calculations over large amounts of data very quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SQL (usually) persistently stores data and is a database. It is also possible to run an in-memory SQL db which may be faster than using pandas, such as SQLite.&lt;/p&gt;&#xA;" OwnerUserId="31154" LastEditorUserId="-1" LastEditDate="2019-08-14T13:51:29.603" LastActivityDate="2019-08-14T13:51:29.603" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20147" PostTypeId="2" ParentId="20104" CreationDate="2017-07-03T20:30:57.513" Score="2" Body="&lt;p&gt;I am not familiar with tensorflow, but I can tell you what I know from your question. To start off, the problem is most likely how you're training, not your model itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two possible problems you may have: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - You are overfitting to the train data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - You haven't trained your model enough&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you either use a pretrained model and finetune it to achieve better results or train your existing model on more data before going back to cats and dogs. Furthermore, it helps to augment your data so your network has more images to train on. A few more things that help would be using dropout, batchnorm, and regularization (more helpful if you are overfitting). If none of these things work, then you may have a problematic model and may need to change it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really hope this helps you in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If you're a beginner to ML, I recommend you start of with Fast.AI's MOOC on machine learning and learn with the libraries they use, keras and theano. Although, they may seem much simpler, they are definitely good enough to achieve incredible results on a number of things.)&lt;/p&gt;&#xA;" OwnerUserId="32130" LastActivityDate="2017-07-03T20:30:57.513" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20148" PostTypeId="2" ParentId="20104" CreationDate="2017-07-03T21:53:25.603" Score="4" Body="&lt;p&gt;50% is no better than random. What was your training score?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have 6 max pooling layers with 3x3 size for a 64 x 64 image:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;After 1st: size is 21 x 21 x 32&lt;/li&gt;&#xA;&lt;li&gt;After 2nd: size is 7 x 7 x 64&lt;/li&gt;&#xA;&lt;li&gt;After 3rd: size is 2 x 2 x 32&lt;/li&gt;&#xA;&lt;li&gt;After 4th: size is 1 x 1 x 64 (?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then there is two more conv/pool layers after that. It is too much pooling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, try 3 5x5 conv layers, max pool in 2x2 blocks, then dropout before the fully connected layer. Each time there is [2,2] max pool, the depth of the conv layer doubles.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Input is a 64x64 image with 3 color channels (red, green and blue)&#xA;network = input_data(shape=[None, 64, 64, 3],&#xA;                     data_preprocessing=img_prep,&#xA;                     data_augmentation=img_aug)&#xA;&#xA;# 1: Convolution layer with 32 filters, each 5x5&#xA;conv_1 = conv_2d(network, 32, [5,5], activation='relu', name='conv_1')&#xA;# 2: Max pooling layer, out size is 32x32x32&#xA;network = max_pool_2d(conv_1, [2,2])&#xA;# 3: Convolution layer with 64 filters&#xA;conv_2 = conv_2d(network, 64, [5,5], activation='relu', name='conv_2')&#xA;# 2: Max pooling layer, out size is 16x16x64&#xA;network = max_pool_2d(conv_2, [2,2])&#xA;# 4: Convolution layer with 128 filters&#xA;conv_3 = conv_2d(network, 128, [5,5], activation='relu', name='conv_3')&#xA;# 5: Max pooling layer, out size is 8x8x128&#xA;network = max_pool_2d(conv_3, [2,2])&#xA;# 12: Dropout layer to combat overfitting&#xA;network = dropout(network, 0.5)&#xA;# 11: Fully-connected 1024 node layer&#xA;network = fully_connected(network, 1024, activation='relu')&#xA;# 13: Fully-connected layer with two outputs&#xA;network = fully_connected(network, 2, activation='softmax')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, you might have to turn dropout &lt;em&gt;off&lt;/em&gt; before testing if tflearn doesn't do this automatically. In vanilla Tensorflow this is done by using a placeholder for the dropout keep probability and feeding 1.0 to it when testing.&lt;/p&gt;&#xA;" OwnerUserId="34165" LastEditorUserId="34165" LastEditDate="2017-07-04T20:00:42.560" LastActivityDate="2017-07-04T20:00:42.560" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20149" PostTypeId="2" ParentId="20136" CreationDate="2017-07-03T22:03:05.227" Score="1" Body="&lt;p&gt;If &lt;code&gt;a3&lt;/code&gt; is 0 or 1, &lt;code&gt;np.log(a3)&lt;/code&gt; or &lt;code&gt;np.log(1-a3)&lt;/code&gt; is going to give an error as $\log(0)$ is undefined, and $\lim_{x -&amp;gt; 0} \log(x) = -\infty$&lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-07-03T22:03:05.227" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20150" PostTypeId="2" ParentId="20129" CreationDate="2017-07-03T23:06:07.607" Score="2" Body="&lt;p&gt;Check out &lt;a href=&quot;https://www.coursera.org/specializations/data-science-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Applied Data Science with Python Specialization&lt;/a&gt; from Coursera. It is a series of 5 courses from the University of Michigan.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-03T23:06:07.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20151" PostTypeId="2" ParentId="20125" CreationDate="2017-07-03T23:12:05.233" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://textblob.readthedocs.io/en/dev/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TextBlob&lt;/a&gt;, a Python package, does this (and much more). It uses a pre-trained model, thus requires no training. Given a sentence, TextBlob will return the &lt;a href=&quot;https://textblob.readthedocs.io/en/dev/quickstart.html#sentiment-analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;polarity&lt;/a&gt; from -1 to 1. -1 is negative, 1 is positive, 0 is neutral. It will also return subjectivity from 0 (very objective) to 1 (very subjective). &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-03T23:12:05.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20152" PostTypeId="2" ParentId="20036" CreationDate="2017-07-03T23:21:57.650" Score="2" Body="&lt;p&gt;Use an aggressive stemmer. The Lancaster Stemmer is one the most aggressive and popular stemmers around.&lt;/p&gt;&#xA;&lt;p&gt;Here is the Python code:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;from nltk.stem.lancaster import LancasterStemmer&#xA;&#xA;lancaster_stemmer = LancasterStemmer()&#xA;assert 'teach' == lancaster_stemmer.stem('teacher') == lancaster_stemmer.stem('teaches')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2021-05-05T20:06:32.493" LastActivityDate="2021-05-05T20:06:32.493" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20153" PostTypeId="2" ParentId="20074" CreationDate="2017-07-03T23:33:09.570" Score="0" Body="&lt;p&gt;Here are a couple of general NLP resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://nbviewer.jupyter.org/url/norvig.com/ipython/How%20to%20Do%20Things%20with%20Words.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;How To Do Things With Words&lt;/a&gt; by &lt;a href=&quot;http://norvig.com/ipython/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Peter Norvig&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://classroom.udacity.com/courses/cs271/lessons/48641663/concepts/487136530923&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLP sections&lt;/a&gt; of &lt;a href=&quot;https://classroom.udacity.com/courses/cs271&quot; rel=&quot;nofollow noreferrer&quot;&gt;Intro to Artificial Intelligence&lt;/a&gt; from Udacity&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/1627052984&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Network Methods in Natural Language Processing&lt;/a&gt; by Yoav Goldberg &amp;amp; Graeme Hirst&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-03T23:33:09.570" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20154" PostTypeId="2" ParentId="20051" CreationDate="2017-07-03T23:40:38.803" Score="0" Body="&lt;p&gt;There is a spectrum of methods. On one end is the tried n' true method of performing pattern matching with rules. Write a series of regular expressions (i.e., regex). On the other end is the newer methods of weak supervision, a system that learns over time to get better. &lt;a href=&quot;http://hazyresearch.github.io/snorkel/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Snorkel&lt;/a&gt; one of implementation of weak supervision. &lt;a href=&quot;https://github.com/snorkel-team/snorkel-tutorials&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here are some snorkel demos&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="29575" LastEditDate="2021-08-01T12:53:42.157" LastActivityDate="2021-08-01T12:53:42.157" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20155" PostTypeId="2" ParentId="20074" CreationDate="2017-07-04T00:24:33.380" Score="0" Body="&lt;p&gt;You are definitely doing a great job of getting your basics down. I &lt;em&gt;really&lt;/em&gt; like Patrick Winston's &lt;a href=&quot;https://www.youtube.com/watch?v=TjZBTDzGeGg&quot; rel=&quot;nofollow noreferrer&quot;&gt;AI Course&lt;/a&gt;, he does a great job of conceptualizing the math behind these problems, which is the only place I think Ng lacks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find a ton of papers you think are interesting, and read them top to bottom. Here is one from spotify on &lt;a href=&quot;http://benanne.github.io/2014/08/05/spotify-cnns.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLP&lt;/a&gt;(super awesome)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most importantly, IMO, the thing you need to start doing is applying the stuff you learn, to problems you think are interesting. Do a few run throughs of other stuff on github and then start doing your own!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck, hope that was helpful:)&lt;/p&gt;&#xA;" OwnerUserId="31046" LastActivityDate="2017-07-04T00:24:33.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20156" PostTypeId="1" CreationDate="2017-07-04T01:37:32.483" Score="-1" ViewCount="70" Body="&lt;p&gt;I have N (movement detection) sensors. (N is determined. Usually five.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All sensors are connected to one wireless AP (access point/network router).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each sensor sends two types of data to the wireless AP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One is heartbeat data with a constant frequency, and the other is sensed data (if a movement is detected).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each sensor has a different frequency and I know the frequencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The AP encrypts each data and outputs a fixed size 512 bytes of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As soon as it encrypts one data, the AP sends it to a next router.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The source address represents the AP not the sensor's address)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I observe the packets (encrypted data) in the next router.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(the data have some noises, because there are some delays in the AP)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;What I want to do is to differentiate heartbeat packets from the mixed packets.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the heartbeat packets are a mixture of N periodic packets, so I think it is possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Is there any tool or scheme for it??&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, an observed data (of two sensors with frequency 3 and 2) is {1, 2, 3.9, 4, 6, 7, 8, 9.9, 10, 11, 12, 13, 14, 16, 16.1, ...}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the output might be {sensor1: (1, 4, 7, 10, 13, 16, ...), sensor2: (2, 3.9, 4, 6, 8, 9.9, 12, 14, 16.1), sensed data (not heartbeat): (11)}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Or maybe probabilities.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="33301" LastActivityDate="2017-10-04T08:11:48.443" Title="Filter out periodic data" Tags="|machine-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20157" PostTypeId="1" CreationDate="2017-07-04T01:46:15.247" Score="0" ViewCount="11849" Body="&lt;p&gt;I am working on a three class problem. How do you calculate precision, recall, f-score, and MCC for each class while using  MATLAB? Here is my confusion matrix:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;2775  0    0&#xA;1    591   0&#xA;4     0   845&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am calculating Accuracy from the Confusion matrix in this way:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Accuracyy = 100*sum(diag(confusionMat))./sum(confusionMat(:));  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I would like to measure the below performance measures for each class. I know the formulas, but not how to execute this in MATLAB. Please help. I would really appreciate it. Thank you.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Precision = {{TP} \over {TP \ + \ FP}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Recall = {{TP} \over {TP \ + \ FN}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$F-score = {{2 \ * \ TP} \over {2 \ * \ TP \ + FP \ + \ FN}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Matthews Correlation Coefficient (MCC)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="29169" LastEditDate="2021-07-16T18:30:39.813" LastActivityDate="2022-03-26T07:27:33.783" Title="How do you calculate Precision and Recall using a confusion matrix in Matlab?" Tags="|classification|matlab|multilabel-classification|confusion-matrix|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20159" PostTypeId="1" CreationDate="2017-07-04T05:50:27.637" Score="1" ViewCount="906" Body="&lt;p&gt;I've been trying to make an image classifier base on &lt;a href=&quot;https://www.tensorflow.org/tutorials/layers&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have changed a few things in it, which I didn't think would completely break the learning pattern but they did... The accuracy calculations are consistently about as good as random guesses, with no upward curve, not even a slow one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you could see in the code, the only major changes I made was using my own data-set instead of the MNIST data-set, and my data is pictures of characters of 89 classes, not just 10. I apply random noise and distortions before batching them to the trainer, and I have about 100,000 examples of each of the 89 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have commented next to the only changed lines to help you read the changes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code (python):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def model(features, labels, mode):&#xA;&#xA;    #Change from the tutorial - 36*36 images instead of 28*28&#xA;    input_layer = tf.reshape(features, [-1, 36, 36, 1])&#xA;&#xA;    conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, kernel_size=[5, 5], padding='same', activation=tf.nn.relu)&#xA;    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)&#xA;    conv2 = tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5], padding=&quot;same&quot;, activation=tf.nn.relu)&#xA;    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)&#xA;&#xA;    # Change From the tutorial - 9*9*64 instead of 7*7*64 to match different image size&#xA;    pool2_flat = tf.reshape(pool2, [-1, 9 * 9 * 64])&#xA;&#xA;    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)&#xA;    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == learn.ModeKeys.TRAIN)&#xA;&#xA;    # Change from tutorial = 89 classes instead of 10&#xA;    logits = tf.layers.dense(inputs=dropout, units=89)&#xA;    onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=89)&#xA;&#xA;    loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)&#xA;    train_op = tf.contrib.layers.optimize_loss(&#xA;            loss=loss,&#xA;            global_step=tf.contrib.framework.get_global_step(),&#xA;            learning_rate=0.001,&#xA;            optimizer='SGD'&#xA;    )&#xA;    # Generate Predictions&#xA;    predictions = {&#xA;        &quot;classes&quot;: tf.argmax(input=logits, axis=1),&#xA;        &quot;probabilities&quot;: tf.nn.softmax(logits, name=&quot;softmax_tensor&quot;)&#xA;    }&#xA;&#xA;    return model_fn.ModelFnOps(mode=mode, predictions=predictions, loss=loss, train_op=train_op)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I thought that the network parameters are maybe to small to handle 36*36 pictures and 89 classes, so I tried giving the conv layers more features, or increase the dense layer nodes, but nothing seems to change. The only results in tensorboard I see are of 1%-1.5% accuracy, which fit well with random guesses on 89 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a real newbie in tensorflow and machine learning so please be gentle with me, and tell me please what i'm doing wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Training code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train(batch_size, steps):&#xA;    config = get_config() &#xA;    config.mode = 'train' # just for the example&#xA;    # Generate training data&#xA;    if config.mode == learn.ModeKeys.TRAIN:&#xA;        x, y = generate(batch_size * steps)&#xA;    print(y) # prints correct labels in range 0-89&#xA;    logging_hook = tf.train.LoggingTensorHook(tensors={&quot;Probabilities&quot;: &quot;softmax_tensor&quot;}, every_n_iter=50)&#xA;    char_classifier = learn.Estimator(model_fn=model, model_dir=mydir)&#xA;&#xA;    # Train!&#xA;    if config.mode == learn.ModeKeys.TRAIN: #assume this is always true in the example here&#xA;        char_classifier.fit(x=x, y=y, batch_size=batch_size, steps=steps, monitors=[logging_hook])&#xA;    accuracy = learn.MetricSpec(metric_fn=tf.metrics.accuracy, prediction_key=&quot;classes&quot;)&#xA;&#xA;    # Evaluate!&#xA;    x, y = generate(1000)&#xA;    eval_results = char_classifier.evaluate(x=x, y=y, metrics={'accuracy':accuracy})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, sample images:&lt;a href=&quot;https://i.stack.imgur.com/KyYcV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KyYcV.png&quot; alt=&quot;fletter&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/QLxYZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QLxYZ.png&quot; alt=&quot;yletter&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/WDrBc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WDrBc.png&quot; alt=&quot;eletter&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34092" LastEditorUserId="29169" LastEditDate="2021-02-16T00:45:15.400" LastActivityDate="2021-02-16T00:45:15.400" Title="Why is my CNN model not learning anything? (tensorflow)" Tags="|python|classification|tensorflow|image-classification|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20165" PostTypeId="1" AcceptedAnswerId="23094" CreationDate="2017-07-04T12:31:06.583" Score="2" ViewCount="933" Body="&lt;p&gt;Many tutorials suggest that after training a RBM, one can have a good reconstruction of training data just like an autoencoder.&lt;/p&gt;&#xA;&lt;p&gt;An example &lt;a href=&quot;https://deeplearning4j.org/restrictedboltzmannmachine&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But the training process of RBM is essentially to maximize the likelihood of the training data. We usually use some technique like CD-K or PCD, so it seems that we can only say that a trained RBM has high probability to generate data which is like training data (digits if we use MNIST), but not correspond to reconstruction. Or are these two things just equivalent in some way?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.toronto.edu/%7Ehinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hinton said&lt;/a&gt; that it is not a good idea to use reconstruction error for monitoring the progress of training, this is why I have this question.&lt;/p&gt;&#xA;" OwnerUserId="34184" LastEditorUserId="98307" LastEditDate="2020-08-01T10:50:08.977" LastActivityDate="2020-08-01T10:50:08.977" Title="Why training a Restricted Boltzmann Machine corresponds to having a good reconstruction of training data?" Tags="|machine-learning|neural-network|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20166" PostTypeId="1" CreationDate="2017-07-04T12:35:41.023" Score="5" ViewCount="599" Body="&lt;p&gt;I read about &#xA;&lt;a href=&quot;https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html&quot; rel=&quot;noreferrer&quot;&gt;https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but couldn't find a spark library for this implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have columnar string dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with around data of 15-20 million users with their show_watched, times_watched, genre, channel and some more columns, I need to calculate lookalike/s for a user(or 100k users).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I find lookalikes for them within less time,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried by indexing data in Solr, and then using Solr MLT for finding similar users, but that takes a lot of time, also it uses TF-IDF for MLT and I need users which have times_show_watched close to that user's times_show_watched.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone recommend a better approach for this, maybe using any other framework for faster processing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried to implement clustering using Spark MLLIB and later search in which cluster a user belongs so that search space is less, but I couldn't get this approach finished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open to any approaches which would be efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="20766" LastActivityDate="2018-01-29T16:40:28.370" Title="Using Spark for finding similar users to a user?" Tags="|apache-spark|apache-mahout|similar-documents|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20168" PostTypeId="1" AcceptedAnswerId="20171" CreationDate="2017-07-04T14:09:23.730" Score="0" ViewCount="4871" Body="&lt;p&gt;I want to lemmatize set of plural keywords automatically such as 'Web based technologies', 'Information systems' etc. I want to transform them to to 'Web based technology', 'Information system' respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried NLTK as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.stem import WordNetLemmatizer&#xA;&#xA;lemmatizer = WordNetLemmatizer()&#xA;&#xA;print(lemmatizer.lemmatize(&quot;Web based technologies&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even though this perform fine for words such as 'cats' to 'cat', for the keywords I have mentioned I get the same plural form. Any idea how to solve this? Or are there any other tools/APIs that I can make use of?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. Given the keywords I only want to get the singular term of it&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2017-07-04T15:56:00.770" Title="Issues with NLTK lemmatizer (WordNet)" Tags="|machine-learning|python|data-mining|" AnswerCount="1" CommentCount="1" ClosedDate="2017-07-09T19:15:19.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="20169" PostTypeId="1" AcceptedAnswerId="20175" CreationDate="2017-07-04T15:11:15.910" Score="2" ViewCount="825" Body="&lt;p&gt;i have written this neural network for XOR function.the output is not correct.it is not classifying the test inputs correctly.can anyone please let me the reason why.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;x=np.array([[0,0],[0,1],[1,0],[1,1]])&#xA;y=np.array([[0],[1],[1],[0]])&#xA;np.random.seed(0) &#xA;theta1=np.random.rand(2,8)&#xA;theta2=np.random.rand(8,1)&#xA;np.random.seed(0)&#xA;b1=np.random.rand(4,8)&#xA;b2=np.random.rand(4,1)&#xA;alpha=0.01&#xA;lamda=0.01&#xA;&#xA;for i in range(1,2000):&#xA;    z1=x.dot(theta1)+b1&#xA;&#xA;    h1=1/(1+np.exp(-z1))&#xA;    z2=h1.dot(theta2)+b2&#xA;    h2=1/(1+np.exp(-z2))&#xA;&#xA;    dh2=h2-y&#xA;    #back prop&#xA;&#xA;    dz2=dh2*(1-dh2)&#xA;    H1=np.transpose(h1)&#xA;    dw2=np.dot(H1,dz2)&#xA;    db2=np.sum(dz2)&#xA;&#xA;    W2=np.transpose(theta2)&#xA;    dh1=np.dot(dz2,W2)&#xA;    dz1=dh1*(1-dh1)&#xA;&#xA;    X=np.transpose(x)&#xA;    dw1=np.dot(X,dz1)&#xA;&#xA;    db1=np.sum(dz1)&#xA;&#xA;&#xA;    dw2=dw2-lamda*theta2&#xA;&#xA;    dw1=dw1-lamda*theta1&#xA;    theta1=theta1-alpha*dw1&#xA;    theta2=theta2-alpha*dw2&#xA;    b1+=-alpha*db1&#xA;&#xA;    b2+=-alpha*db2&#xA;&#xA;&#xA;#prediction&#xA;#test inputs&#xA;input1=np.array([[0,0],[1,1],[0,1],[1,0]])&#xA;z1=np.dot(input1,theta1)&#xA;&#xA;h1=1/(1+np.exp(-z1))&#xA;z2=np.dot(h1,theta2)&#xA;&#xA;h2=1/(1+np.exp(-z2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;expected output=[0],[0],[1],[1]&#xA;&lt;code&gt;actual output=[[ 0.95678049]&#xA; [ 0.99437206]&#xA; [ 0.98686979]&#xA; [ 0.98628204]]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;all are ones here.&lt;/p&gt;&#xA;" OwnerUserId="34042" LastActivityDate="2017-07-04T21:51:46.397" Title="neural network for binary classification of xor gate" Tags="|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20170" PostTypeId="1" AcceptedAnswerId="20259" CreationDate="2017-07-04T15:50:47.963" Score="7" ViewCount="771" Body="&lt;p&gt;I recently completed &lt;a href=&quot;https://github.com/kohaugustine/Machine-Learning-Coursera/blob/master/machine-learning-ex3/Programming%20Exercise%203%20-%20Multi-class%20Classification%20and%20Neural%20Networks.ipynb&quot; rel=&quot;noreferrer&quot;&gt;exercise 3 of Andrew Ng's Machine Learning on Coursera using Python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my trained model has the accuracy that matches the expected 94.9%. Even after debugging and ensuring that my cost and gradient functions were bug free, and that my predictor code was working correctly, I was still getting only 90.3% accuracy. I was using the conjugate gradient (CG) algorithm in &lt;code&gt;scipy.optimize.minimize&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out of curiosity, I decided to try another algorithm, and used Broyden–Fletcher–Goldfarb–Shannon (BFGS). To my surprise, the accuracy improved drastically to 96.5% and thus exceeded the expectation. The comparison of these two different results between CG and BFGS can be viewed in my &lt;a href=&quot;https://github.com/kohaugustine/Machine-Learning-Coursera/blob/master/machine-learning-ex3/Programming%20Exercise%203%20-%20Multi-class%20Classification%20and%20Neural%20Networks.ipynb&quot; rel=&quot;noreferrer&quot;&gt;notebook&lt;/a&gt; under the header &lt;strong&gt;Difference in accuracy due to different optimization algorithms&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the reason for this difference in accuracy due to the different choice of optimization algorithm? If yes, then could someone explain why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would greatly appreciate any review of my code just to make sure that there isn't a bug in any of my functions that is causing this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; Here below I added the code involved in the question, on the request in the comments that I do so in this page rather than refer readers to the links to my Jupyter notebooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model cost functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sigmoid(z):&#xA;    return 1 / (1 + np.exp(-z))&#xA;&#xA;def compute_cost_regularized(theta, X, y, lda):&#xA;    reg =lda/(2*len(y)) * np.sum(theta[1:]**2) &#xA;    return 1/len(y) * np.sum(-y @ np.log(sigmoid(X@theta)) &#xA;                             - (1-y) @ np.log(1-sigmoid(X@theta))) + reg&#xA;&#xA;def compute_gradient_regularized(theta, X, y, lda):&#xA;    gradient = np.zeros(len(theta))&#xA;    XT = X.T&#xA;    beta = sigmoid(X@theta) - y&#xA;    regterm = lda/len(y) * theta&#xA;    # theta_0 does not get regularized, so a 0 is substituted in its place&#xA;    regterm[0] = 0 &#xA;    gradient = (1/len(y) * XT@beta).T + regterm&#xA;    return gradient&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Function that implements one-vs-all classification training:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.optimize import minimize&#xA;&#xA;def train_one_vs_all(X, y, opt_method):&#xA;    theta_all = np.zeros((y.max()-y.min()+1, X.shape[1]))&#xA;    for k in range(y.min(),y.max()+1):&#xA;        grdtruth = np.where(y==k, 1,0)&#xA;        results = minimize(compute_cost_regularized, theta_all[k-1,:], &#xA;                           args = (X,grdtruth,0.1),&#xA;                           method = opt_method, &#xA;                           jac = compute_gradient_regularized)&#xA;        # optimized parameters are accessible through the x attribute&#xA;        theta_optimized = results.x&#xA;        # Assign thetheta_optimized vector to the appropriate row in the &#xA;        # theta_all matrix&#xA;        theta_all[k-1,:] = theta_optimized&#xA;    return theta_all&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Called the function to train the model with different optimization methods:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta_all_optimized_cg = train_one_vs_all(X_bias, y, 'CG')  # Optimization performed using Conjugate Gradient&#xA;theta_all_optimized_bfgs = train_one_vs_all(X_bias, y, 'BFGS') # optimization performed using Broyden–Fletcher–Goldfarb–Shanno&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We see that prediction results differ based on the algorithm used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def predict_one_vs_all(X, theta):&#xA;    return np.mean(np.argmax(sigmoid(X@theta.T), axis=1)+1 == y)*100&#xA;&#xA;In[16]: predict_one_vs_all(X_bias, theta_all_optimized_cg)&#xA;Out[16]: 90.319999999999993&#xA;&#xA;In[17]: predict_one_vs_all(X_bias, theta_all_optimized_bfgs)&#xA;Out[17]: 96.480000000000004&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For anyone wanting to get any data to try the code, they can find it in my Github as linked in this post. &lt;/p&gt;&#xA;" OwnerUserId="33716" LastEditorUserId="33716" LastEditDate="2017-07-06T15:06:52.373" LastActivityDate="2017-07-11T12:07:47.747" Title="Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?" Tags="|machine-learning|python|logistic-regression|accuracy|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="20171" PostTypeId="2" ParentId="20168" CreationDate="2017-07-04T15:56:00.770" Score="1" Body="&lt;p&gt;As far as I know the nltk lemmatizer works on words or rather ngrams. Your example is a trigram, an easier way to work through this is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word=&quot;web based technologies&quot;&#xA;splits=word.split()&#xA;word=&quot; &quot;.join(lemmatizer.lemmatize(w) for w in splits)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-07-04T15:56:00.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20172" PostTypeId="1" AcceptedAnswerId="20254" CreationDate="2017-07-04T18:36:55.897" Score="-1" ViewCount="776" Body="&lt;p&gt;I have seen many examples of different health care chatbots examples like &lt;a href=&quot;https://www.messenger.com/t/109653792416142&quot; rel=&quot;nofollow noreferrer&quot;&gt;HealthTap&lt;/a&gt; , &lt;a href=&quot;https://www.messenger.com/t/YourDotMD&quot; rel=&quot;nofollow noreferrer&quot;&gt;your.md&lt;/a&gt;  who help people find a solution to the most common &lt;strong&gt;symptoms through AI&lt;/strong&gt;. The real benefit of these chatbots is the ability to provide advice and informations for an healthy life because some groups of people don’t have a good basic knowledge of health. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build a basic model of these kind of chat bot. &#xA;I have worked with &lt;a href=&quot;http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural conversational model&lt;/a&gt; . but these kind of chatbots need pre-defined dataset of &lt;strong&gt;conversation&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;how can I get start with &lt;strong&gt;basic health care chatbot&lt;/strong&gt; ??? I know there are many API available out there on internet but it's not good idea use pre built API.&lt;/p&gt;&#xA;" OwnerUserId="32686" LastActivityDate="2017-08-07T00:18:02.190" Title="Health Care ChatBots" Tags="|machine-learning|deep-learning|nlp|" AnswerCount="1" CommentCount="0" ClosedDate="2017-08-07T17:11:48.300" ContentLicense="CC BY-SA 3.0" />
  <row Id="20173" PostTypeId="1" CreationDate="2017-07-04T20:41:28.407" Score="0" ViewCount="174" Body="&lt;p&gt;I am looking for an example or tutorial of a system predicting numeric values by the use of various classifiers like SVM, Decision trees, ANN or KNN which optimises its choice of algorithm and parameter settings at runtime. Assuming there is unlimited input data and the system can train itself nonstop, the system should be able to adapt to different input data and choose the most accurate prediction setting as possible. Examples in Python, C++ or JAVA would be preferred. Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="34201" LastActivityDate="2017-12-02T00:21:09.507" Title="Classifier runtime evaluation" Tags="|machine-learning|classification|data-mining|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20174" PostTypeId="2" ParentId="20173" CreationDate="2017-07-04T21:12:45.633" Score="1" Body="&lt;p&gt;I don't have much hands-on experience in this, but nevertheless I think this kind of processing of streamingdata is called &quot;online learning&quot;; and finding optimal hyperparameters for any type of learning (online or batch) automatically such that the system is self-tuning is an area of active research. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prof. George Hinton talks about it generally in this video &quot;Lecture 16C : Bayesian optimization of neural network hyperparameters&quot; in this playlist: &lt;a href=&quot;https://www.youtube.com/playlist?list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/playlist?list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0&lt;/a&gt;&#xA;(13min) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you like audio, a listener asked a similar question in this  &lt;a href=&quot;http://www.thetalkingmachines.com/blog/2016/8/18/eric-lander-and-restricted-boltzmann-machines&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Eric Lander and Restricted Boltzmann Machines&quot;&gt;Episode of the Talking Machines podcast:&lt;/a&gt; &#xA;(about 8 minutes in)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clever switching of the learning &lt;em&gt;algorithm&lt;/em&gt; also has been explored, see &quot;Bagging&quot; or &quot;Boosting&quot; on Wikipedia, or &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Ensemble learning&quot;&gt;Ensemble learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a relatively easy-to-use Java-based GUI tool you might try &lt;a href=&quot;http://moa.cms.waikato.ac.nz/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;MOA&quot;- Massive ONline Analysis&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2017-07-04T21:34:27.397" LastActivityDate="2017-07-04T21:34:27.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20175" PostTypeId="2" ParentId="20169" CreationDate="2017-07-04T21:30:44.200" Score="0" Body="&lt;p&gt;There are a few mistakes in the code, so I am going to present a revised version here with comments.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Setup&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;x=np.array([[0,0],[0,1],[1,0],[1,1]])&#xA;y=np.array([[0],[1],[1],[0]])&#xA;np.random.seed(0)&#xA;&#xA;# Optional, but a good idea to have +ve and -ve weights&#xA;theta1=np.random.rand(2,8)-0.5&#xA;theta2=np.random.rand(8,1)-0.5&#xA;&#xA;# Necessary - the bias terms should have same number of dimensions&#xA;# as the layer. For some reason you had one bias vector per example.&#xA;# (You could still use np.random.rand(8) and np.random.rand(1))&#xA;b1=np.zeros(8)&#xA;b2=np.zeros(1)&#xA;&#xA;alpha=0.01&#xA;# Regularisation not necessary for XOR, because you have a complete training set.&#xA;# You could have lamda=0.0, but I have left a value here just to show it works.&#xA;lamda=0.001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Training - Forward propagation&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# More iterations than you might think! This is because we have&#xA;# so little training data, we need to repeat it a lot.&#xA;for i in range(1,40000):&#xA;    z1=x.dot(theta1)+b1&#xA;    h1=1/(1+np.exp(-z1))&#xA;    z2=h1.dot(theta2)+b2&#xA;    h2=1/(1+np.exp(-z2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Training - Back propagation&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    # This dz term assumes binary cross-entropy loss&#xA;    dz2 = h2-y &#xA;    # You could also have stuck with squared error loss, the extra h2 terms&#xA;    # are the derivative of the sigmoid transfer function. &#xA;    # It converges slower though:&#xA;    # dz2 = (h2-y) * h2 * (1-h2)&#xA;&#xA;    # This is just the same as you had before, but with less temp variables&#xA;    dw2 = np.dot(h1.T, dz2)&#xA;    db2 = np.sum(dz2, axis=0)&#xA;&#xA;    # The derivative of sigmoid is h1 * (1-h1), NOT dh1*(1-dh1)&#xA;    dz1 = np.dot(dz2, theta2.T) * h1 * (1-h1)&#xA;    dw1 = np.dot(x.T, dz1)&#xA;    db1 = np.sum(dz1, axis=0)&#xA;&#xA;    # The L2 regularisation terms ADD to the gradients of the weights&#xA;    dw2 += lamda * theta2&#xA;    dw1 += lamda * theta1&#xA;&#xA;    theta1 += -alpha * dw1&#xA;    theta2 += -alpha * dw2&#xA;&#xA;    b1 += -alpha * db1&#xA;    b2 += -alpha * db2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Prediction&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is where you can kick yourself, you forgot to use the biases!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input1=np.array([[0,0],[1,1],[0,1],[1,0]])&#xA;z1=np.dot(input1,theta1)+b1&#xA;h1=1/(1+np.exp(-z1))&#xA;z2=np.dot(h1,theta2)+b2&#xA;h2=1/(1+np.exp(-z2))&#xA;&#xA;print(h2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run the above code I get a correct-looking output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 0.01031446]&#xA; [ 0.0201576 ]&#xA; [ 0.9824826 ]&#xA; [ 0.98584079]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In summary your three big errors were the wrong dimension for the bias vectors in setup, incorrect derivatives for sigmoid function (using correct form, but with wrong variable) and forgetting to use bias at all when predicting at the end. Other details are still worth noting, but would not have prevented you getting something working.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-04T21:51:46.397" LastActivityDate="2017-07-04T21:51:46.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20176" PostTypeId="2" ParentId="6107" CreationDate="2017-07-04T22:09:00.617" Score="55" Body="&lt;p&gt;Step by step math explaining how transpose convolution does 2x upsampling with 3x3 filter and stride of 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GlqLM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GlqLM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest TensorFlow snippet to validate the math:&#xA;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;&#xA;def test_conv2d_transpose():&#xA;    # input batch shape = (1, 2, 2, 1) -&amp;gt; (batch_size, height, width, channels) - 2x2x1 image in batch of 1&#xA;    x = tf.constant(np.array([[&#xA;        [[1], [2]], &#xA;        [[3], [4]]&#xA;    ]]), tf.float32)&#xA;&#xA;    # shape = (3, 3, 1, 1) -&amp;gt; (height, width, input_channels, output_channels) - 3x3x1 filter&#xA;    f = tf.constant(np.array([&#xA;        [[[1]], [[1]], [[1]]], &#xA;        [[[1]], [[1]], [[1]]], &#xA;        [[[1]], [[1]], [[1]]]&#xA;    ]), tf.float32)&#xA;&#xA;    conv = tf.nn.conv2d_transpose(x, f, output_shape=(1, 4, 4, 1), strides=[1, 2, 2, 1], padding='SAME')&#xA;&#xA;    with tf.Session() as session:&#xA;        result = session.run(conv)&#xA;&#xA;    assert (np.array([[&#xA;        [[1.0], [1.0],  [3.0], [2.0]],&#xA;        [[1.0], [1.0],  [3.0], [2.0]],&#xA;        [[4.0], [4.0], [10.0], [6.0]],&#xA;        [[3.0], [3.0],  [7.0], [4.0]]]]) == result).all()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="34204" LastEditorUserId="34204" LastEditDate="2017-11-19T09:45:23.800" LastActivityDate="2017-11-19T09:45:23.800" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20177" PostTypeId="1" CreationDate="2017-07-05T04:42:21.287" Score="-1" ViewCount="2332" Body="&lt;p&gt;I have a dataframe like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Col1     Col2       Col3  &#xA;1     1998-09-02     A  &#xA;2     2011-10-29     B  &#xA;3     2000-12-24     C  &#xA;4     2008-09-12     D  &#xA;5     2004-07-09     E  &#xA;6     2013-01-31     F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to make Col2 into year instead of the exact dates, something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Col1    Col2     Col3  &#xA;1       1998      A  &#xA;2       2011      B  &#xA;3       2000      C  &#xA;4       2008      D  &#xA;5       2004      E  &#xA;6       2013      F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At the same time, I want the strings in Col2 be integers so that I can subset my dataframe for later programming.&#xA;How should I do that?&lt;/p&gt;&#xA;" OwnerUserId="34213" LastEditorUserId="44139" LastEditDate="2018-07-18T11:18:49.163" LastActivityDate="2018-07-18T11:18:49.163" Title="Remove part of a string in a dataframe column in R" Tags="|r|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20178" PostTypeId="1" AcceptedAnswerId="20180" CreationDate="2017-07-05T05:39:22.090" Score="4" ViewCount="1484" Body="&lt;p&gt;CNN are (I think) invariant to small translations of the input image, i.e., they will classify to the same class an image $X$ and an image $X'$ such that all pixels have been translated along a vector $\mathbf{v}$ by some &quot;small&quot; distance $d$. They're not, however, invariant to rotation. I.e., if $X'$ is obtained by $X$ applying a rotation of arbitrary degree $\theta$ around an axis $z$ orthogonal to the plane of the image, they won't necessarily classify it to the same class as $X$. Practitioners usually solve this by data augmentation, but this is unnecessarily wasteful and anyway not an option in my case. Is there a way to make the NN architecture invariant to both isometries? I don't have to use a CNN, but since CNN already enjoy some sort of translation invariance, I figured it would be easier to modify them to get rotational invariance , than to use a completely different architecture.&lt;/p&gt;&#xA;" OwnerUserId="27526" LastActivityDate="2018-09-02T16:13:21.750" Title="How do you make an NN for image classification invariant to translation **and** rotation?" Tags="|machine-learning|image-classification|convolution|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20179" PostTypeId="1" AcceptedAnswerId="20193" CreationDate="2017-07-05T05:43:20.287" Score="57" ViewCount="41201" Body="&lt;p&gt;While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://datascience.stackexchange.com/questions/12532/does-batch-size-in-keras-have-any-effects-in-results-quality/12533#12533&quot;&gt;answer&lt;/a&gt; claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?&lt;/p&gt;&#xA;" OwnerUserId="27616" LastActivityDate="2021-03-15T07:55:52.583" Title="What is the advantage of keeping batch size a power of 2?" Tags="|machine-learning|training|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20180" PostTypeId="2" ParentId="20178" CreationDate="2017-07-05T07:11:08.310" Score="4" Body="&lt;p&gt;It is not possible to have general rotationally-invariant neural network architecture for a CNN*. In fact CNNs are not strongly translation invariant, except due to pooling - instead they combine a little bit of translation invariance with &lt;a href=&quot;https://datascience.stackexchange.com/questions/16060/difference-between-equivariant-to-translation-and-invariant-to-translation&quot;&gt;translation equivariance&lt;/a&gt;. There is no equivalent to pooling layers that would reduce the effect of rotation this way (although for very small rotations the translation invariance will still help).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can however construct features and create a pipeline that reduces incoming rotation differences in your inputs. For example, &lt;a href=&quot;https://dsp.stackexchange.com/questions/4893/scale-and-rotation-invariant-feature-descriptors&quot;&gt;this answer on Signal Processing Stack Exchange&lt;/a&gt; suggests calculating dominant gradient of an image, then rotating the image so that this is always oriented the same way before further processing. If your image has strong straight edges, you could do similar by detecting those (e.g. by using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Hough_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hough transform&lt;/a&gt;) and rotating the input so that these are always oriented the same way. These approaches work only within certain image tasks, but can save on processing time and potentially increase accuracy if they are possible. Effectively they are a form of input normalisation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more radical idea might be to perform a map to polar co-ordinates in your image before processing. This would effectively convert rotational (and radial) variance into translation variance in your image. A CNN processing this mapped image would effectively convert its translation invariance into rotational invariance on the original unmapped image. But the cost would be losing all translation invariance, so only worth considering if your inputs have high variance in rotation but low variance in translation.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Never say never. This is caveated by special cases, for example the paper &lt;a href=&quot;https://arxiv.org/abs/1602.07576&quot; rel=&quot;nofollow noreferrer&quot;&gt;Group Equivariant Convolutional Networks&lt;/a&gt; explains an architecture that adds support for multiples of 90 degree rotation (taking advantage of grid structure in computer images and the weight matrices that construct neural network layers). However, if you want to support free rotation of values other than 90, 180, 270 degrees, then as far as I know, there is no way to do that architecturally &lt;em&gt;within&lt;/em&gt; the network.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-05T12:01:33.723" LastActivityDate="2017-07-05T12:01:33.723" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20181" PostTypeId="2" ParentId="20084" CreationDate="2017-07-05T07:16:56.973" Score="2" Body="&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is something like this feasible? (I know that nothing can be said for sure in data science unless tried out, but is it worth the shot?)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What kind of features should I feed into the classifier? I already know of word frequency distribution(TF-IDF), sentence position, co-occurance stats, but will these be enough? What additional features should I consider? Or should I consider directly feeding in word vectors?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Feed in word vectors. Deep learning has it's way of figuring out good features like tf-idf,co-occurence etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer Version:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convert all the words to word vectors with a word embedding algorithm like Word2Vec or Glove.&#xA;This is usually a good idea since this allows you to represent words in a better semantic (dog, cat, cow,etc will be close) and syntactic (November,December,etc will be close) sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Using tf-idf as a feature hasn't worked for me. I used it to supervise extractive summarization with a neural net.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to manage inputs:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This depends very much on how your data is. (Please let me know which dataset)&#xA;I assume you have something like [sentence1, sentence2,...] as inputs and an output like [True, False, ...] which implies sent1 is in the summary and sent2 isn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data isn't in this form, you should convert it into this form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You haven't specified whether it's Unsupervised or Supervised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;for Supervised:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use an LSTM. Feed the LSTM the sentence word by word. Once the sentence has ended (marked by a token say  or a full stop) make the LSTM make a prediction whether the sentence should be in the summary or not. Train it based on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is kind of similar to how you would do &lt;a href=&quot;http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sentiment analysis&lt;/a&gt;: feed in the sentence and then ask if the sentiment is positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can experiment with stacking LSTMs or changing hyper parameters to get better results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;for Unsupervied:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a bit difficult.&#xA;You could convert the sentences to vectors (Sent2Vec) and hope that important sentences cluster together because they contain important words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could train an RNN to predict the next character and hope that one of the neurons learns to predict important words/sentences as done by &lt;strong&gt;&lt;a href=&quot;https://blog.openai.com/unsupervised-sentiment-neuron/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI's unsupervised sentiment neuron&lt;/a&gt;&lt;/strong&gt; which learnt to predict the sentiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general &quot;Deep&quot; architecture followed these days is : &lt;a href=&quot;https://explosion.ai/blog/deep-learning-formula-nlp&quot; rel=&quot;nofollow noreferrer&quot;&gt;Embed, encode, attend, predict&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out the links I have posted (in blue), they might work for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: If possible, please mention your dataset and also how you managed to make abstractive summaries (considering that it is a lot harder than extractive)&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-07-05T07:16:56.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20183" PostTypeId="2" ParentId="9850" CreationDate="2017-07-05T10:48:04.713" Score="3" Body="&lt;p&gt;&lt;strong&gt;Regrading your question&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Where is the (1 - y_train) * log(1 - y_output) part in most&#xA;TensorFlow examples? Isn't it missing?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The answer is that most output functions are softmax. That means you don't necessarily need to reduce all the probabilities in wrong cases as they will automatically be reduced when you increase probability of the right one&lt;/p&gt;&#xA;&lt;p&gt;For Example:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;before optimisation&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;y_output = [0.2, 0.2, 0.6]&lt;/code&gt; and &lt;code&gt;y_train = [0, 0, 1]&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;after optimisation&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;y_output = [0.15, 0.15, 0.7]&lt;/code&gt; and &lt;code&gt;y_train = [0, 0, 1]&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;here observe that even though we just increased third term, all the other terms automatically reduced&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="34223" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-07-05T10:48:04.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20184" PostTypeId="2" ParentId="20177" CreationDate="2017-07-05T10:48:59.827" Score="0" Body="&lt;p&gt;More a question for stackoverflow.com.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# using only base R:&#xA;as.integer(format(format=&quot;%Y&quot;, as.Date( &quot;1998-09-02&quot;)))&#xA;# returns 1998 for example&#xA;# gets the month &#xA; as.integer(format(format=&quot;%m&quot;, as.Date( &quot;1998-09-02&quot;)))&#xA;&#xA;library(lubridate) # on CRAN&#xA;&#xA;# convert to Date datatype with lubridate&#xA;dfr$Col2     &amp;lt;- year( as.Date(dfr$Col2))&#xA;dfr$Col2_mon &amp;lt;- month(as.Date(dfr$Col2))  # lubridate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-07-05T10:48:59.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20186" PostTypeId="1" CreationDate="2017-07-05T12:06:00.290" Score="1" ViewCount="653" Body="&lt;p&gt;I've got about 5 million JSON files, about 50GB in total. They do not have a consistent schema (they're broadly the same format, but some have extra extension fields, some have missing fields, etc - the schema is quite complexly nested). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to run SQL-like queries across these files - e.g. finding the count of files with a certain property, finding the count of files where property is in a numeric or time range, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the files locally, and in an S3 bucket. I would ideally like to store the data in the cloud, so that others can run queries against them without having to have the files locally.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my options? Here are some solutions I've evaluated so far:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Apache Drill.&lt;/strong&gt; &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pros: Great for SQL-like queries and 'on-discovery' schemas. &lt;/li&gt;&#xA;&lt;li&gt;Cons: Ruled out because it struggles with a dataset this size both locally (runs out of memory) and hooked up to S3 (simply doesn't return results, I think because it has to get a local copy of the data). &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Google BigQuery.&lt;/strong&gt; &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pros: Great for SQL-like queries, stores in the cloud, large datasets, plus the friendly user interface is a bonus. JSON can be loaded from command line. &lt;/li&gt;&#xA;&lt;li&gt;Cons: I'd need to define a schema, and remove or fill any data that doesn't conform to it. This is a hassle, and a problem because it means I can't query some data that I would like to (e.g. extension fields).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Amazon Athena.&lt;/strong&gt; &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pros: Cloud support (runs against S3), and (I think) has SQL-like queries. &lt;/li&gt;&#xA;&lt;li&gt;Cons: Seems to require a defined schema (one you define at runtime, rather than in advance)? Also seems to need the files in the query to match the schema, which wouldn't work for my data. However, I'm not sure this is correct, because the documentation is quite confusing. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Google Cloud Datastore.&lt;/strong&gt; &#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pros: SQL-like queries with GQL, stores in the cloud. NoSQL, so schemaless. &lt;/li&gt;&#xA;&lt;li&gt;Cons: Seems to require pre-defined indexes in order to run queries (?), limited query support (e.g. no aggregation or joins). No documentation on how to load JSON files into it (possibly needs Cloud Dataflow setup).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm leaning towards BigQuery, mainly because it's much easier to get up and running than any other solution. However, the requirement for a predefined schema is a major hassle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any solutions I've missed? Have I misunderstood Athena's requirements about schemas?&lt;/p&gt;&#xA;" OwnerUserId="34227" LastActivityDate="2017-07-05T13:01:25.923" Title="Running SQL-like queries over large schemaless JSON dataset in the cloud?" Tags="|bigdata|nosql|json|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20187" PostTypeId="2" ParentId="20186" CreationDate="2017-07-05T13:01:25.923" Score="1" Body="&lt;p&gt;You want to write structured queries (have your cake) but have unstructured files (eat it too). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way you can play with the files a bit and have a common schema between them, with some fields just empty? That would make it easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, BigQuery is probably your least bad solution as you found. Amazon's new Redshift Spectrum is kind of similar but will also require a schema to be defined.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-05T13:01:25.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20188" PostTypeId="1" AcceptedAnswerId="62105" CreationDate="2017-07-05T14:07:20.350" Score="3" ViewCount="1991" Body="&lt;p&gt;I'm interested in interaction effects in Random Forest. I try to implement Friedman H'statistic basically developped for Gradient boosting &lt;a href=&quot;http://statweb.stanford.edu/~jhf/ftp/RuleFit.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Friendman, 2001&lt;/a&gt;. &#xA;But I have no idea how calculate (44) on the real data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kOZbB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kOZbB.png&quot; alt=&quot;Part 8.1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anybody has some examples or explanations?&lt;/p&gt;&#xA;" OwnerUserId="33478" LastActivityDate="2019-10-23T17:04:44.247" Title="Friedman H'statistic for interaction" Tags="|predictive-modeling|random-forest|xgboost|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20190" PostTypeId="2" ParentId="9588" CreationDate="2017-07-05T16:42:21.460" Score="0" Body="&lt;p&gt;If your RDD happens to be in the form of a dictionary, this is how it can be done using PySpark:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define the fields you want to keep in here:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;field_list =[]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Create a function to keep specific keys within a dict input&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def f(x):&#xA;    d = {}&#xA;    for k in x:&#xA;        if k in field_list:&#xA;            d[k] = x[k]&#xA;    return d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And just map after that, with x being an RDD row&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;rdd_subset = rdd.map(lambda x: f(x))&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="34238" LastEditorUserId="31818" LastEditDate="2017-11-27T16:26:01.727" LastActivityDate="2017-11-27T16:26:01.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20192" PostTypeId="2" ParentId="13746" CreationDate="2017-07-05T17:45:52.590" Score="22" Body="&lt;p&gt;You have to use &lt;a href=&quot;https://keras.io/backend/#backend-functions&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras backend functions&lt;/a&gt;. Unfortunately they do not support the &lt;code&gt;&amp;amp;&lt;/code&gt;-operator, so that you have to build a workaround: We generate matrices of the dimension &lt;code&gt;batch_size x 3&lt;/code&gt;, where (e.g. for true positive) the first column is the ground truth vector, the second the actual prediction and the third is kind of a label-helper column, that contains in the case of true positive only ones. Then we check which instances are positive instances, are predicted as positive and the label-helper is also positive. Those are the true positives.&lt;/p&gt;&#xA;&lt;p&gt;We can make this analog with false positives, false negatives and true negatives with some reverse-calculations of the labels.&lt;/p&gt;&#xA;&lt;p&gt;Your f1-metric may look as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def f1_score(y_true, y_pred):&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    f1 score&#xA;&#xA;    :param y_true:&#xA;    :param y_pred:&#xA;    :return:&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    tp_3d = K.concatenate(&#xA;        [&#xA;            K.cast(y_true, 'bool'),&#xA;            K.cast(K.round(y_pred), 'bool'),&#xA;            K.cast(K.ones_like(y_pred), 'bool')&#xA;        ], axis=1&#xA;    )&#xA;&#xA;    fp_3d = K.concatenate(&#xA;        [&#xA;            K.cast(K.abs(y_true - K.ones_like(y_true)), 'bool'),&#xA;            K.cast(K.round(y_pred), 'bool'),&#xA;            K.cast(K.ones_like(y_pred), 'bool')&#xA;        ], axis=1&#xA;    )&#xA;&#xA;    fn_3d = K.concatenate(&#xA;        [&#xA;            K.cast(y_true, 'bool'),&#xA;            K.cast(K.abs(K.round(y_pred) - K.ones_like(y_pred)), 'bool'),&#xA;            K.cast(K.ones_like(y_pred), 'bool')&#xA;        ], axis=1&#xA;    )&#xA;&#xA;    tp = K.sum(K.cast(K.all(tp_3d, axis=1), 'int32'))&#xA;    fp = K.sum(K.cast(K.all(fp_3d, axis=1), 'int32'))&#xA;    fn = K.sum(K.cast(K.all(fn_3d, axis=1), 'int32'))&#xA;&#xA;    precision = tp / (tp + fp)&#xA;    recall = tp / (tp + fn)&#xA;    return 2 * ((precision * recall) / (precision + recall))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Since the Keras-backend calculator returns nan for division by zero, we do not need the if-else-statement for the return statement.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;I have found a pretty good idea for a exact implementation. The problem with our first approach is, that it is only &amp;quot;approximated&amp;quot;, since it is computed batchwise and subsequently averaged. One could also calculate this after each epoch with the &lt;code&gt;keras.callback&lt;/code&gt;s. Please find the idea &lt;a href=&quot;https://github.com/fchollet/keras/issues/5794&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;An example implementation would be:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;import numpy as np&#xA;import sklearn.metrics as sklm&#xA;&#xA;&#xA;class Metrics(keras.callbacks.Callback):&#xA;    def on_train_begin(self, logs={}):&#xA;        self.confusion = []&#xA;        self.precision = []&#xA;        self.recall = []&#xA;        self.f1s = []&#xA;        self.kappa = []&#xA;        self.auc = []&#xA;&#xA;    def on_epoch_end(self, epoch, logs={}):&#xA;        score = np.asarray(self.model.predict(self.validation_data[0]))&#xA;        predict = np.round(np.asarray(self.model.predict(self.validation_data[0])))&#xA;        targ = self.validation_data[1]&#xA;&#xA;        self.auc.append(sklm.roc_auc_score(targ, score))&#xA;        self.confusion.append(sklm.confusion_matrix(targ, predict))&#xA;        self.precision.append(sklm.precision_score(targ, predict))&#xA;        self.recall.append(sklm.recall_score(targ, predict))&#xA;        self.f1s.append(sklm.f1_score(targ, predict))&#xA;        self.kappa.append(sklm.cohen_kappa_score(targ, predict))&#xA;&#xA;        return&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;To make the network to call this function you simply add it to you callbacks like&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;metrics = Metrics()&#xA;model.fit(&#xA;    train_instances.x,&#xA;    train_instances.y,&#xA;    batch_size,&#xA;    epochs,&#xA;    verbose=2,&#xA;    callbacks=[metrics],&#xA;    validation_data=(valid_instances.x, valid_instances.y),&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Then you can simply access the members of the &lt;code&gt;metrics&lt;/code&gt; variable.&lt;/p&gt;&#xA;" OwnerUserId="30835" LastEditorUserId="98307" LastEditDate="2020-08-02T10:29:00.353" LastActivityDate="2020-08-02T10:29:00.353" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20193" PostTypeId="2" ParentId="20179" CreationDate="2017-07-05T18:31:09.060" Score="59" Body="&lt;p&gt;This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.&lt;br /&gt;&#xA;You can see the mapping of the VP onto the PP as a pile of slices of size &lt;em&gt;the number of PP&lt;/em&gt;.&lt;br /&gt;&#xA;Say you've got 16 PP.&lt;br /&gt;&#xA;You can map 16 VP on them : 1 VP is mapped onto 1 PP.&lt;br /&gt;&#xA;You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.&lt;br /&gt;&#xA;Etc.&#xA;During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.&lt;br /&gt;&#xA;If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th &lt;strong&gt;AND&lt;/strong&gt; the other ones will do nothing (precised below).&lt;br /&gt;&#xA;This is due to the SIMD paradigm (called &lt;em&gt;vector&lt;/em&gt; in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See &lt;a href=&quot;https://en.wikipedia.org/wiki/SIMD&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;br /&gt;&#xA;More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), &lt;strong&gt;but only one has some data to work on&lt;/strong&gt;.&lt;br /&gt;&#xA;Nothing to do with learning. This is only programming stuff.&lt;/p&gt;&#xA;" OwnerUserId="33499" LastEditorUserId="98307" LastEditDate="2020-08-20T20:05:55.437" LastActivityDate="2020-08-20T20:05:55.437" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="20198" PostTypeId="1" AcceptedAnswerId="20209" CreationDate="2017-07-06T01:27:15.243" Score="2" ViewCount="1718" Body="&lt;p&gt;I have a set of documents as given in the example below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;doc1 = {'Science': 0.7, 'History': 0.05, 'Politics': 0.15, 'Sports': 0.1}&#xA;doc2 = {'Science': 0.3, 'History': 0.5, 'Politics': 0.1, 'Sports': 0.1}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to cluster the documents and identify the most prominent document within the cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g, cluster 1 includes = {doc1, doc4, doc5. doc8} and I want to get the most prominent document that represents this cluster (e.g., doc8). (or to identify the main theme of the cluster)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know a suitable approach to achieve this :)&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-07-06T09:01:36.917" LastActivityDate="2017-07-06T13:53:45.310" Title="Cluster documents and identify the prominent document in the cluster?" Tags="|machine-learning|data-mining|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20199" PostTypeId="1" AcceptedAnswerId="20203" CreationDate="2017-07-06T05:17:55.947" Score="54" ViewCount="502304" Body="&lt;p&gt;Fairly new to Python but building out my first RF model based on some classification data. I've converted all of the labels into int64 numerical data and loaded into X and Y as a numpy array, but I am hitting an error when I am trying to train the models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what my arrays look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; X = np.array([[df.tran_cityname, df.tran_signupos, df.tran_signupchannel, df.tran_vmake, df.tran_vmodel, df.tran_vyear]])&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; Y = np.array(df['completed_trip_status'].values.tolist())&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X&#xA;array([[[   1,    1,    2,    3,    1,    1,    1,    1,    1,    3,    1,&#xA;            3,    1,    1,    1,    1,    2,    1,    3,    1,    3,    3,&#xA;            2,    3,    3,    1,    1,    1,    1],&#xA;        [   0,    5,    5,    1,    1,    1,    2,    2,    0,    2,    2,&#xA;            3,    1,    2,    5,    5,    2,    1,    2,    2,    2,    2,&#xA;            2,    4,    3,    5,    1,    0,    1],&#xA;        [   2,    2,    1,    3,    3,    3,    2,    3,    3,    2,    3,&#xA;            2,    3,    2,    2,    3,    2,    2,    1,    1,    2,    1,&#xA;            2,    2,    1,    2,    3,    1,    1],&#xA;        [   0,    0,    0,   42,   17,    8,   42,    0,    0,    0,   22,&#xA;            0,   22,    0,    0,   42,    0,    0,    0,    0,   11,    0,&#xA;            0,    0,    0,    0,   28,   17,   18],&#xA;        [   0,    0,    0,   70,  291,   88,  234,    0,    0,    0,  222,&#xA;            0,  222,    0,    0,  234,    0,    0,    0,    0,   89,    0,&#xA;            0,    0,    0,    0,   40,  291,  131],&#xA;        [   0,    0,    0, 2016, 2016, 2006, 2014,    0,    0,    0, 2015,&#xA;            0, 2015,    0,    0, 2015,    0,    0,    0,    0, 2015,    0,&#xA;            0,    0,    0,    0, 2016, 2016, 2010]]])&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; Y&#xA;array(['NO', 'NO', 'NO', 'YES', 'NO', 'NO', 'YES', 'NO', 'NO', 'NO', 'NO',&#xA;       'NO', 'YES', 'NO', 'NO', 'YES', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO',&#xA;       'NO', 'NO', 'NO', 'NO', 'NO', 'NO', 'NO'], &#xA;      dtype='|S3')&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Traceback (most recent call last):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  File &quot;&amp;lt;stdin&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;  File &quot;/Library/Python/2.7/site-packages/sklearn/cross_validation.py&quot;, line&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;2039, in train_test_split&#xA;          arrays = indexable(*arrays)&#xA;        File &quot;/Library/Python/2.7/site-packages/sklearn/utils/validation.py&quot;, line&#xA;  206, in indexable&#xA;          check_consistent_length(*result)&#xA;        File &quot;/Library/Python/2.7/site-packages/sklearn/utils/validation.py&quot;, line&#xA;  181, in check_consistent_length&#xA;          &quot; samples: %r&quot; % [int(l) for l in lengths])&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Found input variables with inconsistent numbers of samples: [1, 29]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="34253" LastEditorUserId="67971" LastEditDate="2019-03-26T06:42:28.280" LastActivityDate="2019-09-09T20:22:03.303" Title="train_test_split() error: Found input variables with inconsistent numbers of samples" Tags="|python|scikit-learn|sampling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20200" PostTypeId="1" AcceptedAnswerId="20212" CreationDate="2017-07-06T05:21:25.523" Score="2" ViewCount="873" Body="&lt;p&gt;I'm reading a book titled &lt;a href=&quot;https://www.packtpub.com/big-data-and-business-intelligence/python-deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Deep Learning&lt;/a&gt;, and in &lt;em&gt;Convolutional layers in deep learning&lt;/em&gt; on the chapter 5, the following is written:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One more important point to make is that convolutional networks should generally have a depth equal to a number which is iteratively divisible by 2, such as 32, 64, 96, 128, and so on. This is important when using pooling layers, such as the max-pool layer, since the pooling layer (if it has size (2,2)) will divide the size of the input layer, similarly to how we should define &quot;stride&quot; and &quot;padding&quot; so that the output image will have integer dimensions. In addition, padding can be added to ensure that the output image size is the same as the input.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As far as I know, the width and the height should be a figure divisible by 2, in order to use a pooling layer. However, I don't understand why the depth must be a figure divisible by 2. The pooling layer just operates on a 2-dimensional screen based on width and height, and it operates on each filter (depth) separately, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why should the depth also be set to a figure divisible by 2?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-08-05T11:56:58.467" Title="Should the depth on convolutional layers be set to a figure divisible by 2?" Tags="|deep-learning|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20201" PostTypeId="1" AcceptedAnswerId="20211" CreationDate="2017-07-06T05:22:21.060" Score="6" ViewCount="8207" Body="&lt;p&gt;Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example yields an r2 value of 0.93, which seems unrealistic. How can I assess if these are indeed realistic results or simply model overfitting?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(datasets)&#xA;library(h2o)&#xA;&#xA;# Get the iris dataset&#xA;df &amp;lt;- iris&#xA;&#xA;# Convert to h2o&#xA;df.hex &amp;lt;- as.h2o(df)&#xA;&#xA;# Initiate h2o&#xA;h2o.init()&#xA;&#xA;# Train GBM model&#xA;gbm_model &amp;lt;- h2o.gbm(x = 2:5, y = 1, df.hex, &#xA;                     ntrees=100, max_depth=4, learn_rate=0.1)&#xA;&#xA;# Check Accuracy&#xA;perf_gbm &amp;lt;- h2o.performance(gbm_model) &#xA;rsq_gbm &amp;lt;- h2o.r2(perf_gbm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;----------&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; rsq_gbm&#xA;[1] 0.9312635&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17539" LastActivityDate="2019-01-11T09:08:01.917" Title="How to determine if my GBM model is overfitting?" Tags="|machine-learning|r|accuracy|overfitting|gbm|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20202" PostTypeId="2" ParentId="20199" CreationDate="2017-07-06T06:21:07.693" Score="3" Body="&lt;p&gt;Isn't train_test_split expecting both &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; to be a list of same length? Your X has length of 6 and Y has length of 29. May be try converting that to pandas dataframe (with 29x6 dimension) and try again? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your data, it looks like you have 6 features. In that case, try to convert your &lt;code&gt;X&lt;/code&gt; to have 29 rows and 6 columns. Then pass that dataframe to &lt;code&gt;train_test_split&lt;/code&gt;. You can convert your list to dataframe using &lt;code&gt;pd.DataFrame.from_records&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26127" LastActivityDate="2017-07-06T06:21:07.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20203" PostTypeId="2" ParentId="20199" CreationDate="2017-07-06T06:21:24.263" Score="43" Body="&lt;p&gt;You are running into that error because your &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt; don't have the same length (which is what &lt;code&gt;train_test_split&lt;/code&gt; requires), i.e., &lt;code&gt;X.shape[0] != Y.shape[0]&lt;/code&gt;. Given your current code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; X.shape&#xA;(1, 6, 29)&#xA;&amp;gt;&amp;gt;&amp;gt; Y.shape&#xA;(29,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To fix this error:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove the extra list from inside of &lt;code&gt;np.array()&lt;/code&gt; when defining &lt;code&gt;X&lt;/code&gt; or remove the extra dimension afterwards with the following command: &lt;code&gt;X = X.reshape(X.shape[1:])&lt;/code&gt;. Now, the shape of &lt;code&gt;X&lt;/code&gt; will be (6, 29).&lt;/li&gt;&#xA;&lt;li&gt;Transpose &lt;code&gt;X&lt;/code&gt; by running &lt;code&gt;X = X.transpose()&lt;/code&gt; to get equal number of samples in &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Y&lt;/code&gt;. Now, the shape of &lt;code&gt;X&lt;/code&gt; will be (29, 6) and the shape of &lt;code&gt;Y&lt;/code&gt; will be (29,).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2019-01-06T23:46:01.440" LastActivityDate="2019-01-06T23:46:01.440" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20205" PostTypeId="2" ParentId="20156" CreationDate="2017-07-06T07:48:09.683" Score="0" Body="&lt;p&gt;If both frequency and shift (delta_t) are known you can use something like an abs( (sin(phi*t + delta_t)) to be used as probability density function PDF for background (periodically recurrent data). If you then have a data point with a low background probability you can select it. This has the caveat that it does not work if data is recorded at the same time but then I guess you have to check the content anyway. The sinus function might not be the optimal implementation but is more too help you visualize it. I would just use gaussians with an appropriate width and stack them for the points when you expect data. So your global PDF could be a stack of many small gaussians. If you do not know the initial shift you probably you have to fit it on data.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-07-06T07:48:09.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20207" PostTypeId="1" CreationDate="2017-07-06T10:10:32.100" Score="3" ViewCount="1093" Body="&lt;p&gt;On a daily basis there is a sequence of events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each event may or may not occur on a given day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the times of the preceding events for the current day I want to be able to say whether there is a high likelihood of an event occurring right now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are around 20 events that may occur in a day.  I have captured only a small number of days worth of observations (say 20-40) to start but will be building up as time goes by.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of model should I use to predict this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is picture of the data to help clarify:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vh4T8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vh4T8.png&quot; alt=&quot;sample data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE/CLARIFICATION&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As each day time passes during the day I always want to know if each event is likely to happen at that moment of time. By the nature of the data it is possible that on a given day only a single event happens, but when it does it will be likely to happen at around the same time it has happened in the past. &lt;/p&gt;&#xA;" OwnerUserId="34267" LastEditorUserId="34267" LastEditDate="2017-07-06T14:35:21.370" LastActivityDate="2017-07-06T14:35:21.370" Title="Predict event likelihood given time of other events" Tags="|time-series|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20208" PostTypeId="1" CreationDate="2017-07-06T10:10:57.973" Score="11" ViewCount="2558" Body="&lt;p&gt;I'm in the process of preparing to teach&#xA;an introductory course on data science using the R programming language.&#xA;My audience is undergraduate students majoring in business subjects.&#xA;A typical business undergrad does not have any computer programming experience,&#xA;but has taken a few classes which use Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I am very comfortable with R (or other programming languages)&#xA;because I majored in computer science.&#xA;However, I have the feeling that many of my students&#xA;will feel wary of learning a programming language&#xA;because it may seem difficult to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do have some familiarity with Excel,&#xA;and it is my belief that while Excel can be useful for simple data science,&#xA;it is necessary for students to learn&#xA;a serious programming language for data science (e.g., R or Python).&#xA;How do I convince myself and the students&#xA;that Excel is insufficient&#xA;for a serious business student studying data science,&#xA;and that it is necessary for them to learn some programming?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Edited in response to comment&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here are some of the topics that I will be covering:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data processing and data cleaning&lt;/li&gt;&#xA;&lt;li&gt;How to manipulate a data table,&#xA;e.g., select a subset of rows (filter),&#xA;add new variables (mutate), &#xA;sort rows by columns &lt;/li&gt;&#xA;&lt;li&gt;SQL joins using the &lt;a href=&quot;https://cran.r-project.org/web/packages/dplyr/dplyr.pdf&quot; rel=&quot;noreferrer&quot;&gt;dplyr&lt;/a&gt; package&lt;/li&gt;&#xA;&lt;li&gt;How to draw plots (scatter plots, bar plots, histograms etc.)&#xA;using the &lt;a href=&quot;https://cran.r-project.org/web/packages/ggplot2/index.html&quot; rel=&quot;noreferrer&quot;&gt;ggplot2&lt;/a&gt; package&lt;/li&gt;&#xA;&lt;li&gt;How to estimate and interpret statistical models such as&#xA;linear regression, logistic regression,&#xA;classification trees, and k-nearest neighbors&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Because I don't know Excel very well,&#xA;I don't know whether all of these tasks can be done easily in Excel.&lt;/p&gt;&#xA;" OwnerUserId="33777" LastEditorUserId="33777" LastEditDate="2017-07-06T14:09:07.390" LastActivityDate="2018-07-02T10:32:19.490" Title="Is Excel sufficient for data science?" Tags="|programming|excel|" AnswerCount="6" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="20209" PostTypeId="2" ParentId="20198" CreationDate="2017-07-06T10:48:19.250" Score="3" Body="&lt;p&gt;A very simple approach would be to find some kind of centroid for each cluster (e.g. averaging the distributions of the documents belonging to each cluster respectively) and then calculating the cosine distance of each document within the cluster from the corresponding centroid. The document with the shorter distance will be the closest to the centroid, hence the most &quot;representative&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Continuing &lt;a href=&quot;https://datascience.stackexchange.com/questions/19991/cluster-documents-based-on-topic-similarity/19999?noredirect=1#comment23532_19999&quot;&gt;from the previous example&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.metrics import pairwise_distances&#xA;from scipy.spatial.distance import cosine&#xA;from sklearn.cluster import DBSCAN&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;&#xA;# Initialize some documents&#xA;doc1 = {'Science':0.8, 'History':0.05, 'Politics':0.15, 'Sports':0.1}&#xA;doc2 = {'News':0.2, 'Art':0.8, 'Politics':0.1, 'Sports':0.1}&#xA;doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}&#xA;doc4 = {'Science':0.1, 'Weather':0.2, 'Art':0.7, 'Sports':0.1}&#xA;collection = [doc1, doc2, doc3, doc4]&#xA;df = pd.DataFrame(collection)&#xA;# Fill missing values with zeros&#xA;df.fillna(0, inplace=True)&#xA;# Get Feature Vectors&#xA;feature_matrix = df.as_matrix()&#xA;&#xA;# Fit DBSCAN&#xA;db = DBSCAN(min_samples=1, metric='precomputed').fit(pairwise_distances(feature_matrix, metric='cosine'))&#xA;labels = db.labels_&#xA;n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)&#xA;print('Estimated number of clusters: %d' % n_clusters_)&#xA;&#xA;# Find the representatives&#xA;representatives = {}&#xA;for label in set(labels):&#xA;    # Find indices of documents belonging to the same cluster&#xA;    ind = np.argwhere(labels==label).reshape(-1,)&#xA;    # Select these specific documetns&#xA;    cluster_samples = feature_matrix[ind,:]&#xA;    # Calculate their centroid as an average&#xA;    centroid = np.average(cluster_samples, axis=0)&#xA;    # Find the distance of each document from the centroid&#xA;    distances = [cosine(sample_doc, centroid) for sample_doc in cluster_samples]&#xA;    # Keep the document closest to the centroid as the representative&#xA;    representatives[label] = cluster_samples[np.argsort(distances),:][0]&#xA;&#xA;for label, doc in representatives.iteritems():&#xA;    print(&quot;Label : %d -- Representative : %s&quot; % (label, str(doc)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-07-06T13:53:45.310" LastActivityDate="2017-07-06T13:53:45.310" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20210" PostTypeId="2" ParentId="20208" CreationDate="2017-07-06T11:22:29.990" Score="8" Body="&lt;p&gt;First of all check out &lt;a href=&quot;https://datascience.stackexchange.com/questions/5443/do-data-scientists-use-excel?rq=1&quot;&gt;this post&lt;/a&gt;. It has many reasons why Excel is inferior to other solutions, regarding data science tasks. Excel also can't handle large datasets (hundreds of thousands of records - not to mention anything in the vicinity of &lt;em&gt;Big Data&lt;/em&gt;), image and sound data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excel is good for simple tasks concerning spreadsheets; it emphasizes more on &lt;strong&gt;presentation&lt;/strong&gt; and &lt;strong&gt;ease of use&lt;/strong&gt;, while having minimal support for actually analysing the data. Unless all you want to do is to calculate simple statistical measures (mean, average, etc) or building a &lt;strong&gt;very simple&lt;/strong&gt; model (e.g. linear regression), Excel is inefficient. That being said, 99% the work a company has to deal with concerning data is simple enough to be manageable through Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However &lt;em&gt;Data Science&lt;/em&gt; mainly deals with regression, classification and complex models that excel isn't equipped to handle! If your students want to have a look at data science you need to teach them a tool that will be useful to them (R, python, etc.). These languages also have libraries with tons of built in models to &quot;play with&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another really huge reason I would go with the latter options is that they are &lt;strong&gt;open source&lt;/strong&gt;. I personally feel that open source software should be preferred from an educational standpoint to proprietary solutions (this is also why I suggest python and R over Matlab)!&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2017-07-06T11:22:29.990" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20211" PostTypeId="2" ParentId="20201" CreationDate="2017-07-06T11:23:39.977" Score="6" Body="&lt;p&gt;The term &lt;strong&gt;overfitting&lt;/strong&gt; means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data.&#xA;Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see if you are overfitting, split your dataset into two separate sets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a &lt;strong&gt;train&lt;/strong&gt; set (used to train the model)&lt;/li&gt;&#xA;&lt;li&gt;a &lt;strong&gt;test&lt;/strong&gt; set (used to test the model accuracy) &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A 90% train, 10% test split is very common. Train your model on the train test and evaluate its performance both on the test and the train set. If the accuracy on the test set is much lower than the models accuracy on the train set, the model is overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use cross-validation (e.g. splitting the data into 10 sets of equal size, for each iteration use one as test and the others as train) to get a result that is less influenced by irregularities in your splits.&lt;/p&gt;&#xA;" OwnerUserId="34116" LastEditorUserId="34116" LastEditDate="2017-07-06T11:33:52.113" LastActivityDate="2017-07-06T11:33:52.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20212" PostTypeId="2" ParentId="20200" CreationDate="2017-07-06T11:49:02.220" Score="2" Body="&lt;p&gt;You are right, the way it says it, it means that the height and width should be divisible by 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While the argument he makes is invalid, it is a good practice for the sakes of &lt;strong&gt;memory efficiency&lt;/strong&gt; to make some parameters like the batch size divisible by 2. I'm guessing you want to make more efficient use of memory blocks. The same logic &lt;em&gt;could&lt;/em&gt; apply to the depth of convolutional layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is &lt;strong&gt;no way&lt;/strong&gt; as important as the author of the book says!&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2017-07-06T11:49:02.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20213" PostTypeId="1" CreationDate="2017-07-06T12:08:47.840" Score="4" ViewCount="7663" Body="&lt;p&gt;As a part of a statistical analysis engine, I need to figure out a way to identify the &lt;strong&gt;presence&lt;/strong&gt; or &lt;strong&gt;absence&lt;/strong&gt; of trends and seasonality patterns in a given set of time series data. While most answers and tutorials in the Internet outlines methods to predict or forecast time series data using machine learning models, my objective is simply to identify the presence any such pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: daily sales data through a year&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This data set could show upward trends with monthly seasonality&#xA;or no actual trend with yearly seasonality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am not to manually inspect the scatter plot of the distribution, in what ways can I determine the presence of those patterns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at the following approaches so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using moving average or exponential smoothing to smooth the time series curve and then check if the resulting line can be approximated to a linear curve, which is supposed to provide an upwards or downward trend if any.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using auto correlation to check for seasonality, which I am yet to confirm if it's a possible solution or not (any ideas?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="34270" LastEditorUserId="29575" LastEditDate="2017-09-05T03:27:20.820" LastActivityDate="2018-08-01T13:56:53.543" Title="Identifying trend and seasonality of time series data" Tags="|machine-learning|statistics|time-series|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20217" PostTypeId="1" CreationDate="2017-07-06T16:02:52.200" Score="4" ViewCount="1909" Body="&lt;p&gt;The documentation says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The loss function to be used. Defaults to ‘hinge’, which gives a&#xA;  linear SVM. The ‘log’ loss gives logistic regression, a probabilistic&#xA;  classifier. ‘modified_huber’ is another smooth loss that brings&#xA;  tolerance to outliers as well as probability estimates.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When we use 'modified_huber' loss function, which classification algorithm is used? Is it SVM? If yes, how come it is able to give probability estimates, which is something it can't do with hinge loss?  &lt;/p&gt;&#xA;" OwnerUserId="10671" LastActivityDate="2024-03-20T17:07:50.963" Title="Which algorithm is used in sklearn SGDClassifier when modified huber loss is used?" Tags="|scikit-learn|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20218" PostTypeId="1" CreationDate="2017-07-06T18:02:16.953" Score="3" ViewCount="230" Body="&lt;p&gt;I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not interested in performances, but only to practice their settigs/functionalities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder which is the easiest solution to make them work on a cluster of a few nodes (can 3 nodes be enough?). I hope it's possible to create a cluster of nodes which can be used for all the frameworks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen some tutorials on creating clusters which involve for example virtual machines or dockers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly, docker-swarm project seems interesting for my task: is this a good way to start? Or there already-built solutions for those frameworks which don't require the manual creation of a cluster?&lt;/p&gt;&#xA;" OwnerUserId="34281" LastActivityDate="2017-07-06T18:02:16.953" Title="How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark" Tags="|apache-spark|apache-hadoop|clustering|" AnswerCount="0" CommentCount="0" ClosedDate="2017-07-07T14:42:07.177" ContentLicense="CC BY-SA 3.0" />
  <row Id="20219" PostTypeId="1" AcceptedAnswerId="20280" CreationDate="2017-07-06T18:19:04.370" Score="2" ViewCount="1066" Body="&lt;p&gt;I am currently working on a speech recognition task, on applying deep learning onto the standard acoustic model (gmm-hmm). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've currently generated a spectrogram of my utterances, and using simple pattern recognition managed to receive a 40%WER on a yes/no dataset.. Not great but a start.  The CNN is being fed a context window of 40 frames, in which the center frame is being detected for, my question is whether the use of an RNN could benefit here? so that the RNN handles the context, and the CNN does &quot;image analysis&quot; on one frame spectrogram.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and if so will it implementation wise cause some problems, When I was doing it with a CNN, was the context dependency resolved by doing pattern recognition on a bigger part of the spectrogram, depending of the context window size, but introducing RNN, CNN only has to do analysis on one frame (which I see can become to even get a proper results from) at the time, and can the one framed information be piped to the RNN until a certain context size has reached? and if so how? &lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T00:45:02.227" LastActivityDate="2021-02-16T00:45:02.227" Title="(CNN+)RNN-HMM hybrid for learning phonemes from a spectogram" Tags="|keras|rnn|convolutional-neural-network|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20220" PostTypeId="2" ParentId="12330" CreationDate="2017-07-06T18:24:31.783" Score="1" Body="&lt;p&gt;You are looking for either &quot;online&quot; or &quot;streaming&quot; topic modeling.  A &lt;a href=&quot;http://proceedings.mlr.press/v15/wang11a/wang11a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;hierarchical Dirichlet processes&lt;/a&gt; can automatically choose the number of topics. &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=2939781&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a paper on streaming/online Latent Dirichlet Allocation model (LDA). A Python implementation of streaming/online can be found &lt;a href=&quot;https://github.com/kzhai/InfVocLDA&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If your scale warrants it, Apache Spark has an &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation of online LDA&lt;/a&gt; also.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-06T18:24:31.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20221" PostTypeId="1" AcceptedAnswerId="102242" CreationDate="2017-07-06T18:33:40.553" Score="2" ViewCount="252" Body="&lt;p&gt;I have researched and am familiar with a few optimization problems. However, I can't seem to find material on what I'm attempting to solve in python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have 8 items of varying weight.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1;128&#xA;2;0&#xA;3;130&#xA;4;186&#xA;5;0&#xA;6;0&#xA;7;12&#xA;8;12&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's say I have an unlimited amount of 'knapsacks'. I can only use 2 different items to fill each knapsack which can only hold a value of 90. Each time I load the knapsacks, I would like to then come back and make another pass with the remaining values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;On the first pass, I can fit item 1 into a knapsack, item 3 in a knapsack, and item 4 in a knapsack.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the next pass, even though I took 90 away from item 4, item 4 still holds a value of over 90, and I can put item 4 in another knapsack.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there is a specific term for this particular problem or a common solution? I would greatly appreciate any direction. &lt;/p&gt;&#xA;" OwnerUserId="34284" LastEditorUserId="381" LastEditDate="2017-07-06T18:46:41.030" LastActivityDate="2021-09-20T03:24:50.137" Title="Optimization Problem Python" Tags="|python|optimization|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20222" PostTypeId="1" AcceptedAnswerId="20230" CreationDate="2017-07-06T19:05:44.447" Score="42" ViewCount="37912" Body="&lt;p&gt;I was wondering how do we have to decide how many nodes in hidden layers, and how many hidden layers to put when we build a neural network architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the input and output layer depends on the training set that we have but how do we decide the hidden layer and the overall architecture in general?&lt;/p&gt;&#xA;" OwnerUserId="34037" LastActivityDate="2019-03-17T00:28:47.373" Title="How to decide neural network architecture?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20223" PostTypeId="2" ParentId="16637" CreationDate="2017-07-06T19:48:13.883" Score="1" Body="&lt;p&gt;Mathematically, SFA differs from moving average by the fact that the instant output $y_t$ can only be a function of instant input $x_t$. On the other hand, the output of moving $y_t$ is a function of the past history of $x_{t'}$ with $t' \leq t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One immediate result due to this difference, is that if you have a flat constant input, with only a pulse at time $t$, then the pulse will be smoothed if you use a moving average, but its shape will be kept better if you use SFA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can always try both moving average and SFA for your task, although the moving average requires some input parameters such as window width while SFA does not.&lt;/p&gt;&#xA;" OwnerUserId="27342" LastActivityDate="2017-07-06T19:48:13.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20225" PostTypeId="2" ParentId="20208" CreationDate="2017-07-06T21:11:59.447" Score="7" Body="&lt;p&gt;I just got done with a Masters in Business Analytics and was faced with the same problem you are describing. Luckily I am a technical person and was able to teach myself R and Python, but I was stuck teaching the rest of the class how to use R and Python. The classes I had that used R/Python were handicapped by the lack of technical understanding by the students and so too much time was spent covering how to just open R/Python. The classes that went the other route were underwhelming and not very practical. I wanted to do for a class project something that ended up not being able to be done in Excel because of its limitations but the teacher wouldn't accept any other tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may not be something you can do right away but I would highly recommend that you try and get the department to require a programming course prior to taking your course. Data Science and Business Analytics IMHO should be cross discipline degree paths that require a good bit of Computer Science, but until the programs mature and the university system gets better it might not happen for a while. &lt;/p&gt;&#xA;" OwnerUserId="28523" LastActivityDate="2017-07-06T21:11:59.447" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20226" PostTypeId="2" ParentId="20213" CreationDate="2017-07-06T22:10:28.500" Score="0" Body="&lt;p&gt;You could calculate the &lt;a href=&quot;http://cns.bu.edu/~gsc/CN710/fincast/Technical%20_indicators/Relative%20Strength%20Index%20(RSI).htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;RSI&lt;/a&gt; (relative strength index) of your sales, over 1 month, 3 months, a year or however long of a time interval you wanted to measure if the current positive trend is outweighing the negative trend over that amount of time (or vice versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also try &lt;a href=&quot;https://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:average_directional_index_adx&quot; rel=&quot;nofollow noreferrer&quot;&gt;ADX&lt;/a&gt; (average directional movement index) combined with +- directional indicators to again detect the strength of the trend, and also identify if it is a positive or negative one depending on which directional indicator outweighs the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure there are many other traditionally stock oriented statistics you could use too.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-07-06T22:10:28.500" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20227" PostTypeId="2" ParentId="20208" CreationDate="2017-07-06T22:23:56.790" Score="3" Body="&lt;p&gt;I think you need to be teaching them a popular Data Science language like Python or R. Excel is not going to help them in a real job, and isn't practical for data science purposes.&#xA;I would probably say Python would be most valuable to them in the long run, and with packages like scikit-learn your regressions and classifications can be demonstrated in very few lines of code which they can read and understand more easily. It is not always easy to understand what R is doing by just reading it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another word of advice: Don't waste time forcing your students to set up an IDE and download the necessary packages, if you use python create a virtual environment for them with all the necessary packages, and set up an IDE like pycharm(they can get this and most other IDEs under a student/academic license) where then can develop and run their code through UI rather than console which they may find daunting and confusing. If you go down the R route then make sure you have an IDE like RStudio set up for them and make sure all of the includes and package installs are either included in your example code or fully described.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-07-06T22:23:56.790" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20228" PostTypeId="2" ParentId="20201" CreationDate="2017-07-06T22:31:23.430" Score="3" Body="&lt;p&gt;I suggest training/testing your classifier on separate splits of the original dataset, and then printing a confusion matrix:&#xA;&lt;a href=&quot;https://topepo.github.io/caret/measuring-performance.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://topepo.github.io/caret/measuring-performance.html#&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a way of seeing how many of the 'true' classifications your classifier predicted correctly or incorrectly, and the same for 'false' classifications. This will give you more information than just 'accuracy', because a model trained on data where most of the classes are 1, for example, will predict 1 most of the time because it will probably report reasonably high accuracy in doing so. A confusion matrix is like a sanity check for this.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2017-07-06T22:31:23.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20230" PostTypeId="2" ParentId="20222" CreationDate="2017-07-07T06:33:34.667" Score="37" Body="&lt;p&gt;Sadly there is no generic way to determine &lt;em&gt;a priori&lt;/em&gt; the best number of neurons and number of layers for a neural network, given just a problem description. There isn't even much guidance to be had determining good values to try as a starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common approach seems to be to start with a rough guess based on prior experience about networks used on similar problems. This could be your own experience, or second/third-hand experience you have picked up from a training course, blog or research paper. Then try some variations, and check the performance carefully before picking a best one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The size and depth of neural networks interact with other &lt;em&gt;hyper-paramaters&lt;/em&gt; too, so that changing one thing elsewhere can affect where the best values are. So it is not possible to isolate a &quot;best&quot; size and depth for a network then continue to tune other parameters in isolation. For instance, if you have a very deep network, it may work efficiently with the ReLU activation function, but not so well with sigmoid - if you found the best size/shape of network and then tried an experiment with varying activation functions you may come to the wrong conclusion about what works best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may sometimes read about &quot;rules of thumb&quot; that researchers use when starting a neural network design from scratch. These things might work for your problems or not, but they at least have the advantage of making a start on the problem. The variations I have seen are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a network with hidden layers similar size order to the input, and all the same size, on the grounds that there is no particular reason to vary the size (unless you are creating an autoencoder perhaps).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Start simple and build up complexity to see what improves a simple network.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try varying depths of network if you expect the output to be explained well by the input data, but with a complex relationship (as opposed to just inherently noisy).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try adding some dropout, it's the closest thing neural networks have to magic fairy dust that makes everything better (caveat: adding dropout may improve generalisation, but may also increase required layer sizes and training times). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you read these or anything like them in any text, then take them with a pinch of salt. However, at worst they help you get past the blank page effect, and write some kind of network, and get you to start the testing and refinement process.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As an aside, try not to get too lost in tuning a neural network when some other approach might be better and save you lots of time. Do consider and use other machine learning and data science approaches. Explore the data, maybe make some plots. Try some simple linear approaches first to get benchmarks to beat, linear regression, logistic regression or softmax regression depending on your problem. Consider using a different ML algorithm to NNs - decision tree based approaches such as XGBoost can be faster and more effective than deep learning on many problems.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-07T08:36:50.163" LastActivityDate="2017-07-07T08:36:50.163" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="20231" PostTypeId="2" ParentId="20075" CreationDate="2017-07-07T08:46:05.720" Score="12" Body="&lt;p&gt;According to &lt;a href=&quot;https://bib.dbvis.de/uploadedFiles/155.pdf&quot; rel=&quot;noreferrer&quot;&gt;this interesting paper&lt;/a&gt;, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data.&lt;/p&gt;&#xA;&lt;p&gt;The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to improve the results of distance-based algorithms, like clustering.&lt;/p&gt;&#xA;" OwnerUserId="2576" LastEditorUserId="98307" LastEditDate="2020-08-06T08:59:30.947" LastActivityDate="2020-08-06T08:59:30.947" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20232" PostTypeId="2" ParentId="19980" CreationDate="2017-07-07T09:07:34.967" Score="8" Body="&lt;p&gt;The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/haBYt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/haBYt.png&quot; alt=&quot;Learning curve&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning curve represents the evolution of the training and test errors as you increase the size of your training set. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The training error increases as you increase the size of your dataset, because it becomes harder to fit a model that accounts for the increasing complexity/variability of your training set.&lt;/li&gt;&#xA;&lt;li&gt;The test error decreases as you increase the size of your dataset, because the model is able to generalise better from a higher amount of information.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you can see on the rightmost part of the plot, the two lines in the plot tend to reach and asymptote. Therefore, you eventually will reach a point in which increasing the size of your dataset will not have an impact on your trained model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distance between the test error and training error asymptotes is a representation of your model's overfitting. But more importantly, this plot is saying whether you need more data. Basically, if you represent test and training error for increasing larger subsets of your training data, and the lines do not seem to be reaching an asymptote, you should keep collecting more data. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-07-07T09:07:34.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20234" PostTypeId="2" ParentId="20075" CreationDate="2017-07-07T10:11:59.333" Score="7" Body="&lt;p&gt;I found something which might be &lt;strong&gt;intuition&lt;/strong&gt; about this problem in &lt;a href=&quot;http://shop.oreilly.com/product/0636920052289.do&quot; rel=&quot;noreferrer&quot; title=&quot;Hands-On Machine Learning with Scikit-Learn and TensorFlow&quot;&gt;Hands-On Machine Learning with Scikit-Learn and TensorFlow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Both the RMSE and the MAE are ways to measure the distance between two&#xA;  vectors: the vector of predictions and the vector of target values.&#xA;  Various distance measures, or norms, are possible:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;Computing the root of a sum of squares (RMSE) corresponds to the Euclidian norm: it is the notion of distance you are familiar with. It&#xA;  is also called the ℓ2 norm(...)&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;Computing the sum of absolutes (MAE) corresponds to the ℓ1 norm,(...). It is sometimes called the Manhattan norm because it&#xA;  measures the distance between two points in a city if you can only&#xA;  travel along orthogonal city blocks.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;More generally, (... )ℓ 0 just gives the number of non-zero elements in the vector, and ℓ∞ gives the maximum absolute value in the vector.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;The higher the norm index, the more it focuses on large values and neglects small ones. This is why the RMSE is more sensitive to&#xA;  outliers than the MAE. But when  outliers are exponentially rare (like&#xA;  in a bell-shaped curve), the RMSE performs very well and is generally&#xA;  preferred.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="82" LastActivityDate="2017-07-07T10:11:59.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20235" PostTypeId="1" CreationDate="2017-07-07T10:33:35.520" Score="3" ViewCount="250" Body="&lt;p&gt;I have done a Logistic regression in r with 3 variables involving 1 interation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm(predicthead~vol+weekday*head, mydata)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;weekday is a factor - 7 days of the week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The coef values are :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Estimate&#xA;(Intercept) -14.81119721&#xA;Vol 0.118819705&#xA;weekday2    -24.30795615&#xA;weekday3    -4.634180612&#xA;weekday4    -21.64229775&#xA;weekday5    0.607492058&#xA;weekday6    42.80131468&#xA;weekday7    19.16098601&#xA;heads   0.059934818&#xA;weekday2:heads  0.026481088&#xA;weekday3:heads  0.006665015&#xA;weekday4:heads  0.022825105&#xA;weekday5:heads  0.005045576&#xA;weekday6:heads  -0.041724212&#xA;weekday7:heads  -0.019254475&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From my Understanding below is what I do in excel:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;intercept +&#xA;(coef of vol * vol)+&#xA;(coef of corresponding week days) + #there no is value for weekday 1&#xA;(coef * head)+&#xA;(coef of corresponding weekday:head * head) #no value for weekday1 so it will be zero.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The sum of the above should be equal to predict() value - but I am not able to match it.&#xA;Could you please explain where I have done wrong in calculating predicthead value using excel?&lt;/p&gt;&#xA;" OwnerUserId="23731" LastEditorUserId="23731" LastEditDate="2017-07-13T23:50:15.970" LastActivityDate="2017-07-13T23:50:15.970" Title="logistic regression coefs to excel formulation" Tags="|r|regression|linear-regression|excel|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20236" PostTypeId="1" AcceptedAnswerId="20238" CreationDate="2017-07-07T10:40:30.133" Score="1" ViewCount="1006" Body="&lt;p&gt;I am using logistic regression to train a model to predict 'click/non-click' using ['browser info', 'publisher info', , 'location', 'time', 'day'].  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to know the ways in which I can use the new live data to improve the improve the already trained model.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does a solution exist which takes into account - change in feature set?&lt;/p&gt;&#xA;" OwnerUserId="33417" LastActivityDate="2017-07-07T11:58:54.540" Title="How to use live data to improve a existing model?" Tags="|machine-learning|logistic-regression|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20237" PostTypeId="1" AcceptedAnswerId="20428" CreationDate="2017-07-07T11:35:05.640" Score="30" ViewCount="20685" Body="&lt;p&gt;I was going through a solution of the Housing prices competition on Kaggle (&lt;a href=&quot;https://www.kaggle.com/humananalog/xgboost-lasso/code&quot; rel=&quot;noreferrer&quot;&gt;Human Analog's Kernel on House Prices: Advance Regression Techniques&lt;/a&gt;) and came across this part:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Transform the skewed numeric features by taking log(feature + 1).&#xA;# This will make the features more normal.&#xA;from scipy.stats import skew&#xA;&#xA;skewed = train_df_munged[numeric_features].apply(lambda x: skew(x.dropna().astype(float)))&#xA;skewed = skewed[skewed &amp;gt; 0.75]&#xA;skewed = skewed.index&#xA;&#xA;train_df_munged[skewed] = np.log1p(train_df_munged[skewed])&#xA;test_df_munged[skewed] = np.log1p(test_df_munged[skewed])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure of what is the need for converting a skewed distribution into a normal distribution. Please, can someone explain in detail: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why is this being done here? or How is this helpful?&lt;/li&gt;&#xA;&lt;li&gt;How is this different from feature-scaling?&lt;/li&gt;&#xA;&lt;li&gt;Is this a necessary step for feature-engineering? What is likely to happen if I skip this step?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23644" LastEditorUserId="23644" LastEditDate="2017-07-07T11:52:05.460" LastActivityDate="2019-10-29T10:21:21.417" Title="Why do we convert skewed data into a normal distribution" Tags="|regression|feature-extraction|feature-engineering|kaggle|feature-scaling|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20238" PostTypeId="2" ParentId="20236" CreationDate="2017-07-07T11:58:54.540" Score="2" Body="&lt;p&gt;Suppose you have a model that has been trained on $N$ data over $E$ epochs. This means that the model has seen each of the $N$ examples, $E$ times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now say you got $M$ more training data. Normally you would want to train the new ones for $E$ epochs as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if $N$ and $M$ don't come from the same underlying distribution (or don't represent it adequately), this would result in the model &quot;forgetting&quot; the first $N$ examples and &quot;paying more attention&quot; to the latter $M$ ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try training your model for $&amp;lt;E$ epochs, so that it learns the latter but doesn't forget the former, but that is purely empirical and very hard to achieve in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can a few things to avoid this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Retrain&lt;/strong&gt; your whole model using both the $N+M$ examples (which you would shuffle). This would require a new complete training of the model on regular occasions and would be &lt;strong&gt;increasingly&lt;/strong&gt; difficult to train (due to the ever increasing size of the training data). This is a &lt;strong&gt;very inefficient&lt;/strong&gt; solution and wouldn't work for any on-line training application&lt;/li&gt;&#xA;&lt;li&gt;Make use of a model that supports &lt;strong&gt;on-line training&lt;/strong&gt;. Some algorithms support incremental (on-line) training, without you needing to retrain the whole thing. A &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn comparison&lt;/a&gt; is available.&lt;/li&gt;&#xA;&lt;li&gt;Customize an algorithm so that is has the desired effect. For example, you could train a linear SVM incrementally, with a large regularization penalty and an SGD classifier. This is discussed in more detail for scikit-learn &lt;a href=&quot;https://stackoverflow.com/questions/23056460/does-the-svm-in-sklearn-support-incremental-online-learning&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2017-07-07T11:58:54.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20239" PostTypeId="2" ParentId="20208" CreationDate="2017-07-07T12:30:35.103" Score="1" Body="&lt;p&gt;Excel and Data Science - sounds really strange to me. Maybe Excel and 'Data Analysis'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, I think a good compromise between Excel and R is: KNIME (&lt;a href=&quot;http://www.knime.org/knime-analytics-platform&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.knime.org/knime-analytics-platform&lt;/a&gt;). It's free on the desktop and much easier to get started. You can import / export to Excel but also use R, Python or Java if the ~ 1.000 nodes miss some functionality that you need. Since the workflows are visually created, it's also much easier to show them to someone who doesn't know any programming languages - which is quite an advantage in some companies.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-07T12:30:35.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20241" PostTypeId="2" ParentId="20237" CreationDate="2017-07-07T13:20:46.323" Score="3" Body="&lt;p&gt;The skewed data here is being normalised by adding one(one added so that the zeros are being transformed to one as log of 0 is not defined) and taking natural log. The data can be nearly normalised using the transformation techniques like taking square root or reciprocal or logarithm. Now, why it is required. Actually many of the algorithms in data assume that the data science is normal and calculate various stats assuming this. So the more the data is close to normal the more it fits the assumption.&lt;/p&gt;&#xA;" OwnerUserId="34177" LastActivityDate="2017-07-07T13:20:46.323" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20242" PostTypeId="2" ParentId="18087" CreationDate="2017-07-07T13:26:50.030" Score="3" Body="&lt;p&gt;As the name implies, &lt;em&gt;doc2vec&lt;/em&gt; generates vectors representing documents (sentences, paragraphs) but not single words. So training &lt;em&gt;doc2vec&lt;/em&gt; won't give you word vectors but document vectors. This means you can't replace &lt;em&gt;word2vec&lt;/em&gt; by &lt;em&gt;doc2vec&lt;/em&gt; at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's how the authors of the &lt;a href=&quot;https://arxiv.org/abs/1405.4053&quot; rel=&quot;nofollow noreferrer&quot;&gt;underlying paper&lt;/a&gt; describe what doc2vec does:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Our algorithm represents each document by a dense vector which is trained to predict words in the document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="34310" LastActivityDate="2017-07-07T13:26:50.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20243" PostTypeId="2" ParentId="20237" CreationDate="2017-07-07T14:57:57.647" Score="1" Body="&lt;p&gt;Because data science is just statistics at the end of the day, and one of the key assumptions of statistics is the &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Central_limit_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Central Limit Theorem&lt;/a&gt;. So this step is being done because some &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Parametric_statistics&quot; rel=&quot;nofollow noreferrer&quot;&gt;subsequent step&lt;/a&gt; uses stats techniques that rely on it.&lt;/p&gt;&#xA;" OwnerUserId="32402" LastActivityDate="2017-07-07T14:57:57.647" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20245" PostTypeId="1" CreationDate="2017-07-07T16:20:05.663" Score="0" ViewCount="134" Body="&lt;p&gt;We have a large touchscreen kiosk in local malls where people can go up to it and play a game under different categories. We want to implement ML to build a recommended list of games for each player. For now, let's assume there are 5 different games under each of the 4 categories and the games can be either available or unavailable at any given time. Also, there is a table containing a log of each play and a table of all the games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, we would want to pass 2 different data sets; example: a list of all the plays in the last 30 days with any necessary data, and a list of all the currently available games to play with the necessary &quot;relatable&quot; data. Out of this, we would then end up with a list of the top 10 recommended games for each player based on all that data. There will be more data than just the game's category to tie it together to the player/person, but I want to get a simple demo working first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am guessing I need to look into some type of recommendation system but I'm not sure which would best to try out. For now, we are looking at using Python; AWS's ML tool will be too costly. I am open to recommendations on other languages/tools.&lt;/p&gt;&#xA;" OwnerUserId="34315" LastActivityDate="2017-12-15T19:00:18.530" Title="What ML algorithm can I use for building a &quot;recommended&quot; list for players?" Tags="|machine-learning|python|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20246" PostTypeId="2" ParentId="18186" CreationDate="2017-07-07T17:34:09.137" Score="3" Body="&lt;p&gt;Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my &lt;a href=&quot;https://github.com/pexmar/DSCNN_document&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github page&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;To refer directly to my questions:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt;&#xA;The proposed code actually makes partially sense:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One should use the shared-embedding layer to save memory usage, so this part makes sense&lt;/li&gt;&#xA;&lt;li&gt;In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;karpathy's blog&lt;/a&gt; and &lt;a href=&quot;http://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Brownlees blog&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;So far the code looks like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name=&amp;quot;input_&amp;quot; + str(i))&#xA;                   for i in range(max_sentences_per_doc)]&#xA;&#xA;# LSTMs and Average Pooling (sentence-level)&#xA;shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')&#xA;shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)&#xA;sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]&#xA;sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]&#xA;&#xA;doc_modeling = Concatenate(axis=1)(sentence_modeling)&#xA;doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The convolutional layer with multiple filter sizes has to be applied &amp;quot;manually&amp;quot;, therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;conv_blocks = []&#xA;for sz in kernel_sizes:&#xA;    conv = Convolution1D(filters=filters,&#xA;                         kernel_size=sz,&#xA;                         padding=&amp;quot;valid&amp;quot;,&#xA;                         activation=&amp;quot;relu&amp;quot;,&#xA;                         strides=1)(doc_modeling)&#xA;    conv = MaxPooling1D(pool_size=2)(conv)&#xA;    conv = Flatten()(conv)&#xA;    conv_blocks.append(conv)&#xA;doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) &amp;gt; 1 else conv_blocks[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;Question 2 and 3:&lt;/strong&gt; We need either an embedding directly in the batch-data or the embedding layer.&#xA;If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Question 4:&lt;/strong&gt;&#xA;It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Question 5:&lt;/strong&gt; No :-), see question 4&lt;/p&gt;&#xA;" OwnerUserId="30835" LastEditorUserId="98307" LastEditDate="2020-08-07T11:04:51.323" LastActivityDate="2020-08-07T11:04:51.323" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="20247" PostTypeId="1" AcceptedAnswerId="20253" CreationDate="2017-07-07T18:59:50.893" Score="0" ViewCount="970" Body="&lt;p&gt;I created the following Neural Network in Python. It uses weights and biases which should follow standard procedure.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Define size of the layers, as well as the learning rate alpha and the max error&#xA;inputLayerSize = 2&#xA;hiddenLayerSize = 3&#xA;outputLayerSize = 1&#xA;alpha = 0.5&#xA;maxError = 0.001&#xA;&#xA;# Import dependencies&#xA;import numpy&#xA;from sklearn import preprocessing&#xA;&#xA;# Make random numbers predictable&#xA;numpy.random.seed(1)&#xA;&#xA;# Define our activation function&#xA;# In this case, we use the Sigmoid function&#xA;def sigmoid(x):&#xA;    output = 1/(1+numpy.exp(-x))&#xA;    return output&#xA;def sigmoid_derivative(x):&#xA;    return x*(1-x)&#xA;&#xA;# Define the cost function&#xA;def calculateError(Y, Y_predicted):&#xA;    totalError = 0&#xA;    for i in range(len(Y)):&#xA;        totalError = totalError + numpy.square(Y[i] - Y_predicted[i])&#xA;    return totalError&#xA;&#xA;# Set inputs&#xA;# Each row is (x1, x2)&#xA;X = numpy.array([&#xA;            [7, 4.7],&#xA;            [6.3, 6],&#xA;            [6.9, 4.9],&#xA;            [6.4, 5.3],&#xA;            [5.8, 5.1],&#xA;            [5.5, 4],&#xA;            [7.1, 5.9],&#xA;            [6.3, 5.6],&#xA;            [6.4, 4.5],&#xA;            [7.7, 6.7]&#xA;            ])&#xA;&#xA;# Normalize the inputs&#xA;#X = preprocessing.scale(X)&#xA;&#xA;# Set goals&#xA;# Each row is (y1)&#xA;Y = numpy.array([&#xA;            [0],&#xA;            [1],&#xA;            [0],&#xA;            [1],&#xA;            [1],&#xA;            [0],&#xA;            [0],&#xA;            [1],&#xA;            [0],&#xA;            [1]&#xA;            ])&#xA;&#xA;# Randomly initialize our weights with mean 0&#xA;weights_1 = 2*numpy.random.random((inputLayerSize, hiddenLayerSize)) - 1&#xA;weights_2 = 2*numpy.random.random((hiddenLayerSize, outputLayerSize)) - 1&#xA;&#xA;# Randomly initialize our bias with mean 0&#xA;bias_1 = 2*numpy.random.random((hiddenLayerSize)) - 1&#xA;bias_2 = 2*numpy.random.random((outputLayerSize)) - 1&#xA;&#xA;# Loop 10,000 times&#xA;for i in xrange(100000):&#xA;&#xA;    # Feed forward through layers 0, 1, and 2&#xA;    layer_0 = X&#xA;    layer_1 = sigmoid(numpy.dot(layer_0, weights_1)+bias_1)&#xA;    layer_2 = sigmoid(numpy.dot(layer_1, weights_2)+bias_2)&#xA;&#xA;    # Calculate the cost function&#xA;    # How much did we miss the target value?&#xA;    layer_2_error = layer_2 - Y&#xA;&#xA;    # In what direction is the target value?&#xA;    # Were we really sure? if so, don't change too much.&#xA;    layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)&#xA;&#xA;    # How much did each layer_1 value contribute to the layer_2 error (according to the weights)?&#xA;    layer_1_error = layer_2_delta.dot(weights_2.T)&#xA;&#xA;    # In what direction is the target layer_1?&#xA;    # Were we really sure? If so, don't change too much.&#xA;    layer_1_delta = layer_1_error * sigmoid_derivative(layer_1)&#xA;&#xA;    # Update the weights&#xA;    weights_2 -= alpha * layer_1.T.dot(layer_2_delta)&#xA;    weights_1 -= alpha * layer_0.T.dot(layer_1_delta)&#xA;&#xA;    # Update the bias    &#xA;    bias_2 -= alpha * numpy.sum(layer_2_delta, axis=0)&#xA;    bias_1 -= alpha * numpy.sum(layer_1_delta, axis=0)&#xA;&#xA;    # Print the error to show that we are improving&#xA;    if (i% 1000) == 0:&#xA;        print &quot;Error after &quot;+str(i)+&quot; iterations: &quot; + str(calculateError(Y, layer_2))&#xA;&#xA;    # Exit if the error is less than maxError&#xA;    if(calculateError(Y, layer_2)&amp;lt;maxError):&#xA;        print &quot;Goal reached after &quot;+str(i)+&quot; iterations: &quot; + str(calculateError(Y, layer_2)) + &quot; is smaller than the goal of &quot; + str(maxError)&#xA;        break&#xA;&#xA;# Show results&#xA;print &quot;&quot;&#xA;print &quot;Weights between Input Layer -&amp;gt; Hidden Layer&quot;&#xA;print weights_1&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Bias of Hidden Layer&quot;&#xA;print bias_1&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Weights between Hidden Layer -&amp;gt; Output Layer&quot;&#xA;print weights_2&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Bias of Output Layer&quot;&#xA;print bias_2&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Computed probabilities for SALE (rounded to 3 decimals)&quot;&#xA;print numpy.around(layer_2, decimals=3)&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Real probabilities for SALE&quot;&#xA;print Y&#xA;print &quot;&quot;&#xA;&#xA;print &quot;Final Error&quot;&#xA;print str(calculateError(Y, layer_2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using 32,000 epochs I manage to get on average a final error of 0.001.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, compared to the MLPClassifier (Scikit-Learn package) using the same parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mlp = MLPClassifier(&#xA;     hidden_layer_sizes=(3,),&#xA;     max_iter=32000,&#xA;     activation='logistic',&#xA;     tol=0.00001,&#xA;     verbose='true')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My result is pretty bad. The MLPClassifier gets a final error of 0 when I run it on the same data, after about 10,000 epochs. For both networks I use an input layer size of 2, hidden layer size of 3 and an output layer of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does my network need that many more epochs to train? Am I missing an important part?&lt;/p&gt;&#xA;" OwnerUserId="34322" LastEditorUserId="34322" LastEditDate="2017-07-07T22:56:13.357" LastActivityDate="2017-07-07T22:56:13.357" Title="How to improve my self-written Neural Network?" Tags="|python|neural-network|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="20248" PostTypeId="1" CreationDate="2017-07-07T20:03:58.573" Score="1" ViewCount="2593" Body="&lt;p&gt;I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="34324" LastActivityDate="2018-06-25T12:50:34.217" Title="Agglomerative Clustering without knowing number of clusters" Tags="|clustering|nlp|stanford-nlp|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20249" PostTypeId="1" CreationDate="2017-07-07T20:37:56.003" Score="2" ViewCount="832" Body="&lt;p&gt;I am currently studying this &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_cvpr_2015/app/2B_004.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; for implementing it into a speech recognition application, but seem to have some problems understanding the concepts of time step.  What is a time step?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't worked with RNN before but would guess that in a continuous stream of data time step would be the data present at time t. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;but how do one even implement/train something like that in an iteration based system?... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean something like keras, one is not able to involve output received from t-1 to t... or if so, how?, I seem to be unfamiliar with this... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In another paper i found this &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To understand the essen ce ofthe RCL, it is useful to c1arifY the&#xA;  concept oftime step in RCL. It is not identical to the time associated&#xA;  with the sequential data, and instead it refers to an iteration during&#xA;  processing the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;from &lt;a href=&quot;http://ieeexplore.ieee.org/document/7953168/&quot; rel=&quot;nofollow noreferrer&quot;&gt;RECURRENT CONVOLUTIONAL NEURAL NETWORK FOR SPEECH PROCESSING&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which then further conforms my theory of it being being about the data that was involved/processed as iteration t-1.&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T00:44:51.473" LastActivityDate="2021-02-16T00:44:51.473" Title="RCNN - recurrent convolutional neural networks - what is a time step?" Tags="|python|keras|rnn|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20251" PostTypeId="2" ParentId="20235" CreationDate="2017-07-07T21:36:29.053" Score="1" Body="&lt;p&gt;By running a logistic regression model, the objective is to get the chance of a binary outcome based on the different predictor variables. Since the result is a chance, what is modeled is actually:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;log \left(\frac{P_{+}}{1 - P_{+}}\right) = \beta_{0} + \beta_{1}x_{1} + ... + \beta_{n}x_{n}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the coefficients you get from R are the $\beta$ values.&#xA;The function you have so far in Excel computes correctly the right hand side (RHS) of the above equation, but to get the probability $P_{+}$, you need to solve the equation for it. Let's call the RHS, $\mathbf{Bx}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;log \left(\frac{P_{+}}{1 - P_{+}}\right) = \mathbf{Bx} \\&#xA;\frac{P_{+}}{1 - P_{+}} = e^{\mathbf{Bx}} \\&#xA;\frac{1}{e^{\mathbf{Bx}}} = \frac{1}{P_{+}} - 1 \\&#xA;\frac{e^{\mathbf{Bx}} + 1}{e^{\mathbf{Bx}}} = \frac{1}{P_{+}} \\&#xA;P_{+} = \frac{1}{1 + e^{-\mathbf{Bx}}}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To your results, apply this transformation and you should get the same results as the predict function in R.&lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2017-07-07T21:36:29.053" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="20252" PostTypeId="1" CreationDate="2017-07-07T21:46:15.563" Score="1" ViewCount="288" Body="&lt;p&gt;I have a short time series of daily counts ( 6 days = 6 counts : 12, 15, 69, 35, 97, 107). This looks like a rapid increase from its initial count of 12.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the statistical techniques to detect such large increases in short time period? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;rapid increases&quot; and &quot;short time period&quot; are rather loose terms, but, I am looking for techniques where these can be 'parameters' that I can provide.  For example, &quot;short period&quot; could be 5 days. 'Rapid increase' could be 20% increase day-to-day or 90% increase in 5 days...something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suggestions are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="17271" LastActivityDate="2017-07-07T21:46:15.563" Title="Detect rapid increase in time series" Tags="|time-series|anomaly-detection|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20253" PostTypeId="2" ParentId="20247" CreationDate="2017-07-07T22:09:20.427" Score="2" Body="&lt;h3&gt;A note on gradient direction&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;As an aside, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_2_error = Y - layer_2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_2_error = layer_2 - Y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And all your update functions should be gradient &lt;em&gt;descent&lt;/em&gt; e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weights_2 += -alpha * layer_1.T.dot(layer_2_delta)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Scaling Input to Normalised Values&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;One clue to performance problems was in your first version which included the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = preprocessing.scale(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Bias Gradients&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Update the bias    &#xA;bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)&#xA;bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;NB - I have assumed you have fixed the gradient direction here.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Classification Loss Function&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_2_delta = layer_2_error&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the correct delta value to match the loss function  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def calculateError(Y, Y_predicted):&#xA;    return -numpy.mean( (Y * numpy.log(Y_predicted) + &#xA;       (1-Y)* numpy.log(1 - Y_predicted)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. &lt;code&gt;alpha = 0.01&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;note you can still &lt;em&gt;report&lt;/em&gt; your other loss function, for comparison with previous results. Just be aware that you are optimising the log loss&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Difference in Optimisers&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adam&lt;/a&gt;, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum&quot; rel=&quot;nofollow noreferrer&quot;&gt;momentum&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-07T22:25:28.550" LastActivityDate="2017-07-07T22:25:28.550" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="20254" PostTypeId="2" ParentId="20172" CreationDate="2017-07-08T00:17:35.403" Score="1" Body="&lt;p&gt;From what you said, you are constrained by the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No pre-defined dataset of conversation to train a model&lt;/li&gt;&#xA;&lt;li&gt;No use of APIs for pragmatic reasons - presumably to have better control over what goes on under the hood (my guess).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To be honest, you may as well just try putting together an &lt;a href=&quot;https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligence_expert_systems.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;expert system&lt;/a&gt; with a programming language you feel comfortable with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For what you hope to do in chatbot form, you would basically prompt people for answers to various &lt;em&gt;yes/no&lt;/em&gt; or &lt;em&gt;multiple choice&lt;/em&gt; questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could have it ask questions to determine someone's BMI for instance, as well as any other questions you may find relevant to determining someone's health.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at the quizzes on this site for inspiration: &lt;a href=&quot;http://counsellingresource.com/quizzes/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://counsellingresource.com/quizzes/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This isn't as sophisticated as the Neural Conversational Model you mention, but this approach does not require any training data - which from what you say is a major constraint to what you're doing.&lt;/p&gt;&#xA;" OwnerUserId="19000" LastActivityDate="2017-07-08T00:17:35.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20256" PostTypeId="1" CreationDate="2017-07-08T06:12:49.910" Score="2" ViewCount="1519" Body="&lt;p&gt;In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference of 5% between the accuracy of training and testing set okay? Or does it signify overfitting?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="8013" LastEditDate="2017-07-08T10:36:07.293" LastActivityDate="2024-02-04T13:03:50.483" Title="Ideal difference in the training accuracy and testing accuracy" Tags="|classification|data|supervised-learning|training|accuracy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20257" PostTypeId="2" ParentId="20248" CreationDate="2017-07-08T07:31:35.883" Score="1" Body="&lt;p&gt;A minimum cluster size will not generally be satisfiable in hierarchical clustering. Instead, you have to expect many clusters with just a single point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ELKI has some fairly interesting techniques to cut a dendrogram. Check the &lt;code&gt;clustering.hierarchical.extraction&lt;/code&gt; (or so) package. If I remember correctly, some allow you to set a minimum size (but there will be a &quot;noise&quot; cluster with all the leftovers).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-07-08T07:31:35.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20258" PostTypeId="1" CreationDate="2017-07-08T09:49:53.300" Score="1" ViewCount="47" Body="&lt;p&gt;I have some 3D particle data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(x0,y0,z0)&#xA;(x1,y1,z1)&#xA;(x2,y2,z2)&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to find the irregular bounding shape of the distribution. The image below shows an example distribution from three directions (x/y, y/z, x/z), as well as a spherical bounding box, which is a poor approximation to the true distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bvPlK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bvPlK.png&quot; alt=&quot;3D particle distribution&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this particular use case, I wish to throw away particles near the edge of the distribution, since in my application these are subject to boundary effects. So, if I have a rough approximation of the bounding shape in 3D, I can scale this down and discard particles outside the re-scaled shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Solutions in interpreted languages (Python, R) preferable.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="378" LastEditorUserId="378" LastEditDate="2017-11-01T16:11:06.540" LastActivityDate="2017-11-01T16:11:06.540" Title="Find irregular bounding shape of 3D particle distribution" Tags="|clustering|algorithms|data-cleaning|data|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20259" PostTypeId="2" ParentId="20170" CreationDate="2017-07-08T10:04:17.367" Score="3" Body="&lt;p&gt;&lt;strong&gt;Limits of numerical accuracy and stability are causing the optimisation routines to struggle.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see this most easily by changing the regularisation term to 0.0 - there is no reason why this should not work in principle, and you are not using any feature engineering that particularly needs it. With regularisation set to 0.0, then you will see limits of precision reached and attempts to take log of 0 when calculating the cost function. The two different optimisation routines are affected differently, due to taking different sample points on route to the minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that with regularisation term set high, you remove the numerical instability, but at the expense of not seeing what is really going on with the calculations - in effect the regularisation terms become dominant for the difficult training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can offset some of the accuracy problems by modifying the cost function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def compute_cost_regularized(theta, X, y, lda):&#xA;    reg =lda/(2*len(y)) * np.sum(theta[1:]**2) &#xA;    return reg - 1/len(y) * np.sum(&#xA;      y @ np.log( np.maximum(sigmoid(X@theta), 1e-10) ) &#xA;      + (1-y) @ np.log( np.maximum(1-sigmoid(X@theta), 1e-10) ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also to get some feedback during the training, you can add&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                       options = {&#xA;                           'disp': True&#xA;                       }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To the call to &lt;code&gt;minimize&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this change, you can try with regularisation term set to zero. When I do this, I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predict_one_vs_all(X_bias, theta_all_optimized_cg)&#xA;Out[156]:&#xA;94.760000000000005&#xA;In [157]:&#xA;&#xA;predict_one_vs_all(X_bias, theta_all_optimized_bfgs)&#xA;/usr/local/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: overflow encountered in exp&#xA;  from ipykernel import kernelapp as app&#xA;Out[157]:&#xA;98.839999999999989&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The CG value of 94.76 seems to match the expected result nicely - so I wonder if this was done without regularisation. The BFGS value is still &quot;better&quot; although I am not sure how much I trust it given the warning messages during training and evaluation. To tell if this apparently better training result really translates into better digit detection, you would need to measure results on a hold-out test set.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-07-08T10:04:17.367" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20261" PostTypeId="1" CreationDate="2017-07-08T12:26:33.613" Score="2" ViewCount="4712" Body="&lt;p&gt;I have around 72,000 Multivariate time series (MTS) with four Dimensions and of Length around 3000 milliseconds(not constant). It is (4*3000*72000) DLN. This MTS has tow possible outcomes either Pass or Fail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to select a better classifier that suits above criteria?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read some implementations of MTS classification. I cannot apply distance measure techniques for classification as the data is too lengthy and is not constant when compare to other MTS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am considering to apply below techniques for classification &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Extract global statistical features from data and apply any classifier(KNN or Random Forest or neural networks)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what would be the best approach. Please guide me better technique and I am also open for other techniques to classify MTS &lt;/p&gt;&#xA;" OwnerUserId="32367" LastEditorUserId="32367" LastEditDate="2017-07-10T10:14:30.900" LastActivityDate="2018-04-30T01:08:50.173" Title="Multivariate time series classification" Tags="|machine-learning|neural-network|classification|deep-learning|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20262" PostTypeId="2" ParentId="20248" CreationDate="2017-07-08T12:48:12.187" Score="1" Body="&lt;p&gt;If you don t know the number of clusters, i encourage you to look at those density based algorithm : &lt;a href=&quot;https://github.com/beckgael/Mean-Shift-LSH&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mean Shift&lt;/a&gt;, DBSCAN, OPTICS.&#xA;They don t presume of the cluster number and are able to find random shape clusters.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastActivityDate="2017-07-08T12:48:12.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20263" PostTypeId="2" ParentId="19377" CreationDate="2017-07-08T15:03:29.367" Score="5" Body="&lt;p&gt;After some research, I came across &lt;a href=&quot;http://u-pat.org/ICDAR2017/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ICDAR&lt;/a&gt; (International Conference on Document Analysis and Recognition), which is taking place biannually and seems to be the most complete and up-to-date source for state-of-the-art document analysis algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ICDAR is hosting a competition in different subfields each year. At the time of this writing, the latest results from ICDAR 2015 have been published as &lt;em&gt;Competition on Recognition of Documents with Complex Layouts –&#xA;RDCL2015&lt;/em&gt;. (Search online for a PDF version.) This paper describes state-of-the-art document layout analysis algorithms, and compares their performance to both open-source and commercial solutions.&lt;/p&gt;&#xA;" OwnerUserId="32901" LastEditorUserId="32901" LastEditDate="2017-07-10T06:55:36.463" LastActivityDate="2017-07-10T06:55:36.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20271" PostTypeId="1" CreationDate="2017-07-09T11:16:24.993" Score="4" ViewCount="1051" Body="&lt;p&gt;Are there publications which mention numerical problems in neural network optimization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Blog posts, articles, workshop notes, lecture notes, books - anything?)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Background of the question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I've recently had a strange phenomenon: When I trained a convolutional network on the GTSRB dataset with a given script on my machine, it got state of the art results (99.9% test accuracy). 10 times. No outlier. When I used the same scripts on another machine, I got much worse results (~ 80% test accuracy or so, 10 times, no outliers). I thought that I probably didn't use the same scripts and as it was not important for my publication I just removed all results of that dataset. I thought I probably made a mistake one one of the machines (e.g. using different pre-processed data) and I couldn't find out where the mistake happened.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now a friend wrote me that he has a network, a training script and a dataset which converges on machine A but does not converge on machine B. Exactly the same setup (a fully connected network trained as an autoencoder).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have only one guess what might happen: The machines have different hardware. It might be possible that Tensorflow uses different algorithms for matrix multiplication / gradient calculation. Their numeric properties might be different. Those differences might cause one machine to be able to optimize the network while the other can't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this needs further investigation. But no matter what is happening in those two cases, I think this question is interesting. Intuitively, I would say that numeric issues should not be important as sharp minima are not desired anyway and differences in one multiplication are less important than the update of the next step.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-07-09T11:21:37.760" LastActivityDate="2017-11-07T14:05:34.003" Title="Do numerical inaccuracies play any role in training neural networks?" Tags="|neural-network|numerical|" AnswerCount="1" CommentCount="8" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20274" PostTypeId="1" CreationDate="2017-07-09T13:28:57.820" Score="3" ViewCount="512" Body="&lt;p&gt;What would be appropriate models/algorithms/strategies for predicting best individual send times for marketing campaigns based on past response timestamps? &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Data:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Given for example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;===========================================================&#xA;  customer campaign    campaign_time       response_time   &#xA;-----------------------------------------------------------&#xA;1   100       a     2017-01-01 06:50:01 2017-01-01 08:02:21&#xA;2   101       a     2017-01-01 06:50:01 2017-01-01 16:45:31&#xA;3   101       a     2017-01-01 06:50:01 2017-01-02 07:20:00&#xA;4   100       b     2017-01-07 06:30:21 2017-01-08 08:15:21&#xA;5   101       b     2017-01-07 06:30:21 2017-01-07 17:00:12&#xA;6   100       c     2017-01-14 06:43:55 2017-01-14 07:59:44&#xA;7   101       d     2017-01-21 14:02:01 2017-01-21 16:50:01&#xA;-----------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;two customers 100&amp;amp;101,  &lt;/li&gt;&#xA;&lt;li&gt;four past campaigns a-d. &lt;/li&gt;&#xA;&lt;li&gt;with each campaign having different times of dispatch,   &lt;/li&gt;&#xA;&lt;li&gt;and multiple,one or no response time(s) (e.g. buying a product) for customers and campaigns&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Goal:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Assuming that &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;campaign_time&lt;/code&gt; can vary for &lt;code&gt;100&lt;/code&gt; and &lt;code&gt;101&lt;/code&gt; (&lt;strong&gt;personalized times&#xA;of dispatch&lt;/strong&gt;), and&lt;/li&gt;&#xA;&lt;li&gt;past response times are an indicator for when&#xA;customers are most receptive for a campaign&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would like to predict the best next &lt;code&gt;campaign_time&lt;/code&gt; ( &lt;code&gt;2017-01-28 ??:??:??&lt;/code&gt;) for each customer based on past &lt;code&gt;response_time&lt;/code&gt;s, so that the number of respondents per campaign is maximized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone having any experience with something similar or any ideas where to start? I'd be happy to hear some ideas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To simplify things, I'd consider the first response_time the most valuable one (=&gt; should be predicted) and I'd also abstract from weekdays (=&gt; it's about predicting time 0:00-23:59, marked by the &lt;code&gt;?&lt;/code&gt; above); however it would be nice to have a continous prediction instead of a discretized one (like &lt;a href=&quot;https://datascience.stackexchange.com/questions/14124/predict-the-best-time-of-call&quot;&gt;suggested here&lt;/a&gt;). &lt;/p&gt;&#xA;" OwnerUserId="35383" LastEditorUserId="35384" LastEditDate="2017-07-10T06:03:09.283" LastActivityDate="2018-04-29T05:08:40.547" Title="Predict time of dispatch for marketing campaign" Tags="|machine-learning|predictive-modeling|marketing|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20275" PostTypeId="1" AcceptedAnswerId="24389" CreationDate="2017-07-09T13:45:44.073" Score="3" ViewCount="3371" Body="&lt;p&gt;I am using the HDBSCAN algorithm so as to perform unsupervised clustering and detect outliers. Based on the documentation there are two outputs from the clustering process that can give insight on which points are outliers.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/outlier_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The GLOSH outlier detection algorithm&lt;/a&gt; that gives a degree of certainty of whether a point is an outlier or not.&lt;/li&gt;&#xA;&lt;li&gt;The HDBSCAN labels that if an element in not part of a cluster is considered as noise and has the corresponding label.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have been working on some data, and I have noticed that these two approaches do not give the same results. That means that there are points in my dataset that are labeled as not part of a cluster but are not detected by the GLOSH outlier algorithm. Should I consider the union of these two approaches as my outliers or I am missing something in their interpretation? &lt;/p&gt;&#xA;" OwnerUserId="4668" LastEditorUserId="4668" LastEditDate="2017-07-10T15:15:02.183" LastActivityDate="2018-10-02T12:00:02.773" Title="HDBSCAN Outlier Detection and labeling" Tags="|python|clustering|outlier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20277" PostTypeId="1" AcceptedAnswerId="20282" CreationDate="2017-07-09T15:59:19.490" Score="1" ViewCount="105" Body="&lt;p&gt;I want to run several Association rule mining techniques such as Apriori, Eclat and FP Growth. I want to know the &lt;strong&gt;format of the data&lt;/strong&gt; to run these algorithms as they are disabled (marked in grey)to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I don't see the algorithm &lt;strong&gt;Eclat&lt;/strong&gt; in the 'Association' tab of Weka. Please recommend me a suitable tool/approach to perform this.&lt;/p&gt;&#xA;" OwnerUserId="33838" LastActivityDate="2017-07-09T20:54:48.433" Title="Data Type required in Weka" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20280" PostTypeId="2" ParentId="20219" CreationDate="2017-07-09T16:45:30.027" Score="1" Body="&lt;p&gt;Have you seen this... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/document/7953168/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/7953168/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... We propose to use a recently developed deep learning model, recurrent convolutional neural network (RCNN), for speech processing, which inherits some merits of recurrent neural network (RNN) and convolutional neural network (CNN). The core module can be viewed as a convolutional layer embedded with an RNN, which enables the model to capture both temporal and frequency dependance in the spectrogram of the speech in an efficient way.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sound pretty accurately to what you are searching for, or trying to implement. &lt;/p&gt;&#xA;" OwnerUserId="29208" LastActivityDate="2017-07-09T16:45:30.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20281" PostTypeId="2" ParentId="13513" CreationDate="2017-07-09T17:22:02.853" Score="3" Body="&lt;p&gt;At least for my current team (~80 data scientists and engineers), we don't have such preference. Half of the data scientists here use R and another half use Python. Many can code in both. We do deploy Python and R code in production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think any of our data scientists uses Java at all. If they need to deal with big data, they can use SparkSQL or PySpark. The data engineering team uses a mix of Java/Scala/Python/Go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are one of few data people in a small company, I can understand why they require certain language skills so you can do both data science and engineering. But tbh, I think most small companies won't have data big enough that Python or R can't handle in production.&lt;/p&gt;&#xA;" OwnerUserId="16922" LastActivityDate="2017-07-09T17:22:02.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20282" PostTypeId="2" ParentId="20277" CreationDate="2017-07-09T20:54:48.433" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to run several Association rule mining techniques such as&#xA;  Apriori, Eclat and FP Growth. [...] Please recommend me a suitable&#xA;  tool/approach to perform this&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.borgelt.net/fpm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Christian Borgelt's website&lt;/a&gt; offers several tools for frequent pattern mining: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;FPgrowth,&lt;/li&gt;&#xA;&lt;li&gt;Eclat, &lt;/li&gt;&#xA;&lt;li&gt;Apriori, and some others.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Check out the full descriptions to see how to use each one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/14293121/what-are-data-requirements-for-fp-growth-in-weka&quot;&gt;Here&lt;/a&gt; are some requirements for Weka's FPgrowth algorithm. &lt;/p&gt;&#xA;" OwnerUserId="35384" LastActivityDate="2017-07-09T20:54:48.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20283" PostTypeId="1" AcceptedAnswerId="20284" CreationDate="2017-07-09T22:41:46.557" Score="-1" ViewCount="97" Body="&lt;p&gt;I have tried to debug this, but have not made any headway.  Any ideas on how to proceed?  I believe I am invoking everything correctly.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a snippet of the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if _trainWithModel:&#xA;        print(&quot;Training using model&quot;)&#xA;        svc = svm.SVC(C=0.001, verbose=10, kernel='rbf', gamma=0.00000001)&#xA;        fittedSvc = svc.fit(trainingData.data, trainingData.target)&#xA;        print(&quot;Scoring training results = &quot;, fittedSvc.score(trainingData.data, trainingData.target))&#xA;        print(&quot;Sanity check.  Predicting using training data model&quot;)&#xA;        temp_predictedTarget = fittedSvc.predict(trainingData.data)&#xA;        realTarget = np.array([trainingData.target])&#xA;        predictedTarget = np.array([temp_predictedTarget])&#xA;        print&quot;Prediction results shapes:  trainingData.data=&quot;, trainingData.data.shape, &quot;, temp_predictedTarget=&quot;, temp_predictedTarget.shape, \&#xA;            &quot;, realTarget=&quot;, realTarget.shape, &quot;, predictedTarget=&quot;, predictedTarget.shape&#xA;        print&quot;Prediction sanity checks  predictData.target sum=&quot;, sum(trainingData.target), \&#xA;            &quot;, temp_predictedTarget sum=&quot;, sum(temp_predictedTarget), \&#xA;            &quot;, predictedTarget sum=&quot;, np.sum(predictedTarget), \&#xA;            &quot;, realTarget sum=&quot;, np.sum(realTarget)&#xA;        confusion = confusion_matrix(trainingData.data, temp_predictedTarget)&#xA;        print &quot;Confusion matrix:  \n&quot;, confusion&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the key output from this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training using model [LibSVM]..&#xA;Warning: using -h 0 may be faster&#xA;*&#xA;optimization finished, #iter = 2430&#xA;obj = -4.860000, rho = -1.000054&#xA;nSV = 4860, nBSV = 4860&#xA;Total nSV = 4860&#xA;('Scoring training results = ', 0.95715721363211625)&#xA;Sanity check.  Predicting using training data model&#xA;&#xA;Prediction results shapes:  &#xA;trainingData.data= (56719L, 108L) , &#xA;temp_predictedTarget= (56719L,) , &#xA;realTarget= (1L, 56719L) , &#xA;predictedTarget= (1L, 56719L)&#xA;&#xA;Prediction sanity checks  &#xA;predictData.target sum= 2430.0 , &#xA;temp_predictedTarget sum= 0.0 , &#xA;predictedTarget sum= 0.0 , &#xA;realTarget sum= 2430.0&#xA;&#xA;Confusion matrix:  &#xA;[[54289     0]&#xA; [ 2430     0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35392" LastEditorUserId="381" LastEditDate="2017-08-09T06:00:33.090" LastActivityDate="2017-09-08T09:41:55.180" Title="I trained my data and obtained a training score of 0.957. Why can't I get the data to provide a prediction even against the same training data?" Tags="|svm|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20284" PostTypeId="2" ParentId="20283" CreationDate="2017-07-10T02:21:47.693" Score="0" Body="&lt;p&gt;Assuming the training data has some noise in it, you don't want a fit of 1.0 because that would mean you've overfit the model to the noise. Instead you want your model to be capable of generalizing so it can make the best possible predictions on new data.&lt;/p&gt;&#xA;" OwnerUserId="35394" LastActivityDate="2017-07-10T02:21:47.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20286" PostTypeId="1" CreationDate="2017-07-10T05:12:42.967" Score="5" ViewCount="9468" Body="&lt;p&gt;I am trying to understand if over-fitting can happen in an unsupervised technique like kmeans clustering. Could someone help me understand if and how this would happen?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="24575" LastEditorUserId="29169" LastEditDate="2019-06-07T06:10:41.907" LastActivityDate="2019-06-07T06:10:41.907" Title="Overfitting in an unsupervised technique" Tags="|clustering|overfitting|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="20289" PostTypeId="1" AcceptedAnswerId="20312" CreationDate="2017-07-10T06:24:39.933" Score="1" ViewCount="2569" Body="&lt;p&gt;So I have a scraper that gets articles. However, it doesn't always work properly. I want to get better at checking when it doesn't work. For example, the following is something like I want it to scrape: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hello. This is a sequence of sentences that are put together. They don't have to follow this exact format, but something very close to this would be nice! Just basically stuff like this put together with the occasional weird formatting, which depends on what is scraped. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I might also get something that is obviously not text: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;REGISTER | LOGIN | LOGOUT | Sign in to your account Forgot your password?  {* #signInForm *}.... &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is there any python library that checks the general format of strings? Basically, I am scraping articles and want to see if the text scraped is article-y. If there isn't a python library, would the best way to go be some sort of regex matching? Is this possible to do reasonably well? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated, thanks!! &lt;/p&gt;&#xA;" OwnerUserId="35399" LastActivityDate="2017-07-10T17:14:38.357" Title="Python - Check if text is sentences?" Tags="|python|nlp|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20290" PostTypeId="2" ParentId="20286" CreationDate="2017-07-10T06:58:00.193" Score="3" Body="&lt;p&gt;I'm not sure if this is valid but how about two trivial clustering examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Every object belongs to cluster which contain only this object. So for example if you would like to cluster N cars, there will be N clusters - one for each car.&lt;/li&gt;&#xA;&lt;li&gt;On the other hand there could be case when algorithm will pick one cluster which will contain all elements inside it - one cluster with all N cars.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those will be valid clusters but obviously they will not give you any useful information.&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2017-07-10T06:58:00.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20291" PostTypeId="1" AcceptedAnswerId="20299" CreationDate="2017-07-10T07:27:38.650" Score="3" ViewCount="2140" Body="&lt;p&gt;Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself? &lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2018-03-19T13:56:11.517" Title="Which algorithm Doc2Vec uses?" Tags="|python|nlp|word2vec|gensim|similar-documents|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20292" PostTypeId="2" ParentId="20286" CreationDate="2017-07-10T07:37:50.863" Score="3" Body="&lt;h2&gt;Yes, overfitting occurs in unsupervised learning as well&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Overfitting means your algorithm is finding patterns in attributes that only exist in this dataset and don't generalize to new, unseen data. In addition to finding real patterns, when overfitting, the algorithm is also finding &quot;patterns&quot; that are only stochastic noise. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example for clustering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For clustering this means the clusters you are finding only exist in your dataset and can't be seen in new data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your algorithm might find two clusters in the dataset that don't exist for new data, because both clusters are actually subset of one bigger cluster. Your algorithm is overfitting, your clustering is too fine (e.g. your &lt;code&gt;k&lt;/code&gt; is too small for k-means) because you are finding groupings that are only noise.&lt;/p&gt;&#xA;" OwnerUserId="34116" LastActivityDate="2017-07-10T07:37:50.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20293" PostTypeId="2" ParentId="20291" CreationDate="2017-07-10T08:29:57.923" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Documentation&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Deep learning via the distributed memory and distributed bag of words&#xA;  models from &lt;a href=&quot;https://arxiv.org/pdf/1405.4053v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1]&lt;/a&gt;, using either hierarchical softmax or negative&#xA;  sampling &lt;a href=&quot;https://arxiv.org/abs/1301.3781&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/1310.4546&quot; rel=&quot;nofollow noreferrer&quot;&gt;[3]&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="82" LastActivityDate="2017-07-10T08:29:57.923" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20295" PostTypeId="1" CreationDate="2017-07-10T09:51:22.730" Score="1" ViewCount="82" Body="&lt;p&gt;I am going to calibrate two monitors. Two variables &quot;Brightness&quot; and &quot;Contrast&quot; can be adjusted to obtain the monitor's color properties (Its pixel values). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I use some calibration equipment to measure the first monitor's color space property. Then, I use a regression method to obtain a straight line representing the gray level change of that panel. Hence, I find the slope of that line representing the contrast and the y-intercept representing the brightness correlated to that monitor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to apply my prior knowledge to adjust another monitor's brightness and contrast as same as my first monitor? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume that I also use calibration equipment to measure the second monitor's color property. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use neural network to obtain the true value of contrast and brightness of another monitor?&lt;/p&gt;&#xA;" OwnerUserId="35405" LastActivityDate="2017-07-10T09:51:22.730" Title="How do I use neural network to predict variables in linear equations?" Tags="|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20296" PostTypeId="1" AcceptedAnswerId="20301" CreationDate="2017-07-10T10:26:39.450" Score="75" ViewCount="149579" Body="&lt;p&gt;Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the &lt;code&gt;true label&lt;/code&gt; is &lt;code&gt;[1 0 0 0 0]&lt;/code&gt; while the predictions be &lt;code&gt;[0.1 0.5 0.1 0.1 0.2]&lt;/code&gt;. How would I calculate the cross entropy loss for this example?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastEditorUserId="85045" LastEditDate="2020-12-24T21:25:12.647" LastActivityDate="2021-11-25T18:27:44.103" Title="Cross-entropy loss explanation" Tags="|machine-learning|neural-network|deep-learning|softmax|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20297" PostTypeId="1" CreationDate="2017-07-10T10:46:30.430" Score="1" ViewCount="1070" Body="&lt;p&gt;I am applying H2o autoencoder for anomaly detection for multivariate time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have many time series data for different metrics of network elements which are recorded every 15 minutes. I want to find out that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;At which timestamp that the anomaly happens&lt;/li&gt;&#xA;&lt;li&gt;What is/are the main metric(s) causing the anomaly&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I form the data matrix with columns as metrics and rows as the values of the metrics at a specific timestamp. For example, [[9:15, v1, v2, v3...], [9:30, v4, v5, v6, ...], ...]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then train the data with autoencoder and use h2o.anomaly to check which data points have reconstruction MSE greater than the 99% percentile as the suspected anomalies. And these give me the timestamp when the possible anomalies happen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using this method, can I get the information of the main metrics causing the anomaly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="23983" LastActivityDate="2017-08-31T01:29:35.160" Title="H2o autoencoder anomaly detection for multivariate time series data" Tags="|time-series|anomaly-detection|autoencoder|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20299" PostTypeId="2" ParentId="20291" CreationDate="2017-07-10T11:38:07.867" Score="4" Body="&lt;p&gt;Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context words and the target word, these variants arised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: the name of the model maybe confusing &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Distriubted Bag of words&lt;/code&gt; is similar to &lt;code&gt;Skip-gram model&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;Distributed Memory&lt;/code&gt; is similar to &lt;code&gt;Continuous bag of words model&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-07-10T11:38:07.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20300" PostTypeId="1" CreationDate="2017-07-10T11:54:00.393" Score="1" ViewCount="34" Body="&lt;p&gt;I need to create some labels with &lt;code&gt;labelImg&lt;/code&gt; for specific objects, but there are two main factors, which make the process of creating bounding boxes a difficult task.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The objects of interest are small compared to the size of the image and stacked together, think like two or more rectangles stacked together.&lt;/li&gt;&#xA;&lt;li&gt;The quality of the image is not very good, which means that if I zoom in a very high level, the quality of the image worsens, thus I am not even able to distinguish their boundaries.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Has anyone encountered a similar problem?&lt;/p&gt;&#xA;" OwnerUserId="33150" LastEditorUserId="33150" LastEditDate="2017-07-10T12:50:43.410" LastActivityDate="2017-07-10T12:50:43.410" Title="Establishing the ground truth for difficult cases when it comes to images" Tags="|image-recognition|labels|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20301" PostTypeId="2" ParentId="20296" CreationDate="2017-07-10T11:56:25.243" Score="99" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy&quot; rel=&quot;noreferrer&quot;&gt;cross entropy&lt;/a&gt; formula takes in two distributions, &lt;span class=&quot;math-container&quot;&gt;$p(x)$&lt;/span&gt;, the true distribution, and &lt;span class=&quot;math-container&quot;&gt;$q(x)$&lt;/span&gt;, the estimated distribution, defined over the discrete variable &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and is given by &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;span class=&quot;math-container&quot;&gt;$$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$&lt;/span&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For a neural network, the calculation is independent of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What kind of layer was used.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What kind of activation was used - although many activations will not be compatible with the calculation because their outputs are not interpretable as probabilities (i.e., their outputs are negative, greater than 1, or do not sum to 1). Softmax is often used for multiclass classification because it guarantees a well-behaved probability distribution function.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For a neural network, you will usually see the equation written in a form where &lt;span class=&quot;math-container&quot;&gt;$\mathbf{y}$&lt;/span&gt; is the ground truth vector and &lt;span class=&quot;math-container&quot;&gt;$\mathbf{\hat{y}}$&lt;/span&gt; (or some other value taken direct from the last layer output) is the estimate. For a single example, it would look like this:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;span class=&quot;math-container&quot;&gt;$$L = - \mathbf{y} \cdot \log(\mathbf{\hat{y}})$$&lt;/span&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$\cdot$&lt;/span&gt; is the inner product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your example ground truth &lt;span class=&quot;math-container&quot;&gt;$\mathbf{y}$&lt;/span&gt; gives all probability to the first value, and the other values are zero, so we can ignore them, and just use the matching term from your estimates &lt;span class=&quot;math-container&quot;&gt;$\mathbf{\hat{y}}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$L = -(1\times log(0.1) + 0 \times \log(0.5) + ...)$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$L = - log(0.1) \approx 2.303$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An important point from comments&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;That means, the loss would be same no matter if the predictions are &lt;span class=&quot;math-container&quot;&gt;$[0.1, 0.5, 0.1, 0.1, 0.2]$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$[0.1, 0.6, 0.1, 0.1, 0.1]$&lt;/span&gt;? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, this is a key feature of multiclass logloss, it rewards/penalises probabilities of correct classes only. The value is independent of how the remaining probability is split between incorrect classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will often see this equation averaged over all examples as a &lt;em&gt;cost&lt;/em&gt; function. It is not always strictly adhered to in descriptions, but usually a &lt;em&gt;loss&lt;/em&gt; function is lower level and describes how a single instance or component determines an error value, whilst a &lt;em&gt;cost&lt;/em&gt; function is higher level and describes how a complete system is evaluated for optimisation. A cost function based on multiclass log loss for data set of size &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; might look like this:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;span class=&quot;math-container&quot;&gt;$$J = - \frac{1}{N}\left(\sum_{i=1}^{N} \mathbf{y_i} \cdot \log(\mathbf{\hat{y}_i})\right)$$&lt;/span&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Many implementations will require your ground truth values to be one-hot encoded (with a single true class), because that allows for some extra optimisation. However, in principle the cross entropy loss can be calculated - and optimised - when this is not the case.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2020-01-19T08:35:00.267" LastActivityDate="2020-01-19T08:35:00.267" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="20302" PostTypeId="2" ParentId="974" CreationDate="2017-07-10T12:46:30.723" Score="2" Body="&lt;p&gt;You should use : &lt;a href=&quot;https://github.com/facebookresearch/pysparnn&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;PySparNN&quot;&gt;PySparNN&lt;/a&gt;, a recent implementation by Facebook in python which is bloody fast. It is also easy to use. &lt;/p&gt;&#xA;" OwnerUserId="21755" LastActivityDate="2017-07-10T12:46:30.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20303" PostTypeId="1" CreationDate="2017-07-10T12:57:12.920" Score="1" ViewCount="54" Body="&lt;p&gt;I am working with Multi variate time series Analysis using different Models in R. I used &lt;code&gt;Arima&lt;/code&gt;, &lt;code&gt;glm&lt;/code&gt;, &lt;code&gt;H2O.gbm&lt;/code&gt;, &lt;code&gt;randomforest&lt;/code&gt; and &lt;code&gt;Garch&lt;/code&gt; Models for the same set of data. I did predictions and calculated &lt;code&gt;RMSE&lt;/code&gt; value for each Models. I have ranked these models according to &lt;code&gt;RMSE&lt;/code&gt; value. But i would like to provide &lt;code&gt;ratings&lt;/code&gt; like movie ratings to these models based on their RMSE and rank value. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; df &amp;lt;- transform(df,rank = ave(RMSE, &#xA;           FUN = function(x) rank(x, ties.method = &quot;first&quot;)))&#xA;&amp;gt; df&#xA;         Model      RMSE  MAPE rank&#xA;1        arima  410770.6 0.510    1&#xA;2          glm  571215.2 0.841    3&#xA;3      H2O.gbm  501895.3 0.449    2&#xA;4 randomforest 1157233.0 1.746    4&#xA;5        Garch 1161791.0 1.840    5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to provide rating to these Models? for example, i would like to score all the models out of 10. Any method or process to provide ratings to these Models? Any help is much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&#xA;Magg&lt;/p&gt;&#xA;" OwnerUserId="35414" LastActivityDate="2017-11-17T21:28:36.427" Title="Rating the Models" Tags="|r|predictive-modeling|time-series|ranking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20304" PostTypeId="1" AcceptedAnswerId="20355" CreationDate="2017-07-10T13:20:41.123" Score="4" ViewCount="6520" Body="&lt;p&gt;I understand that random forest is a stylized version of bagging of trees. We choose randomly data points as well as random features for constructing random forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if we use just plain version of bagging by choosing only data points randomly then we have trees which have trained on more number of features unlike the random forest in the stylized version. Since learning with more features every individual tree has more information about the data points and so are more 'intelligent' in some sense than the individual trees in the random forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why does the random forest using stylized version of bagging performs better than the random forest using plain implementation of bagging?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the random forest using the stylized version gives a model lower variance but since each of its trees is trained on some of the features shouldn't it make the model a bit  high biased?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="27916" LastEditDate="2017-07-11T10:27:23.127" LastActivityDate="2017-07-11T16:25:19.050" Title="Why do we pick random features in random forest" Tags="|machine-learning|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20307" PostTypeId="2" ParentId="18249" CreationDate="2017-07-10T15:41:20.540" Score="1" Body="&lt;p&gt;&lt;strong&gt;Conjugate gradient descent&lt;/strong&gt; is a variation on gradient descent. &lt;strong&gt;Gradient descent&lt;/strong&gt; is a method of finding the minimum of a function by taking steps that reduce the error between the data and the model parameters. &lt;strong&gt;Conjugate gradient descent&lt;/strong&gt; extends gradient descent by searching a plane, instead of a line. The plane is defined as a linear combination of the gradient vector and the previous descent step vector. &lt;strong&gt;Conjugate gradient descent&lt;/strong&gt; is very good at finding the solutions to a set of sparse linear equations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gradient descent&lt;/strong&gt; and variations are general methods for finding the best parameters. The best practice is to define your specific model and then call on a separate gradient descent package to search for optimal values (i.e., hyperparameters) for the model.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-10T15:41:20.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20308" PostTypeId="1" AcceptedAnswerId="20561" CreationDate="2017-07-10T15:47:32.287" Score="27" ViewCount="91515" Body="&lt;p&gt;I have a Pandas DataFrame like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame({&#xA;    'Date': ['2017-1-1', '2017-1-1', '2017-1-2', '2017-1-2', '2017-1-3'],&#xA;    'Groups': ['one', 'one', 'one', 'two', 'two'],&#xA;    'data': range(1, 6)})&#xA;&#xA;    Date      Groups     data  &#xA;0  2017-1-1    one       1&#xA;1  2017-1-1    one       2&#xA;2  2017-1-2    one       3&#xA;3  2017-1-2    two       4&#xA;4  2017-1-3    two       5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I generate a new DataFrame like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Date       one     two &#xA;0  2017-1-1    3        0&#xA;1  2017-1-2    3        4&#xA;2  2017-1-3    0        5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35420" LastEditorUserId="134" LastEditDate="2018-12-24T02:49:45.303" LastActivityDate="2021-11-01T00:14:57.663" Title="How to sum values grouped by two columns in pandas" Tags="|python|pandas|dataframe|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20309" PostTypeId="2" ParentId="989" CreationDate="2017-07-10T16:07:51.187" Score="3" Body="&lt;p&gt;I recently encountered similar problem because forgot to scale features in my dataset which was earlier used to train ensemble model kind. Failure to scale the data may be the likely culprit as pointed by Shelby Matlock. You may try different scalers available in  sklearn, such as &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/preprocessing/plot_robust_scaling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RobustScaler&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import RobustScaler&#xA; scaler = RobustScaler()&#xA; X = scaler.fit_transfrom(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;X is now transformed/scaled and ready to be fed to your desired model.&lt;/p&gt;&#xA;" OwnerUserId="35421" LastEditorUserId="29169" LastEditDate="2021-02-07T20:23:37.430" LastActivityDate="2021-02-07T20:23:37.430" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20312" PostTypeId="2" ParentId="20289" CreationDate="2017-07-10T17:14:38.357" Score="1" Body="&lt;p&gt;I would try a semi-supervised learning technique where it passes you scraps and asks you to label them. What you're looking for will likely be kind of domain specific depending on the type of site. In the end you'll probably have a bunch of heuristics like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If length &amp;lt; 50 and contains &quot;LOGOUT&quot;, &quot;REGISTER&quot;,&quot;SIGN IN&quot;,&quot;LOGIN&quot;&lt;/li&gt;&#xA;&lt;li&gt;If count of &quot;|&quot; &gt; 1&lt;/li&gt;&#xA;&lt;li&gt;If count of all upper case words &gt; 1&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-10T17:14:38.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20314" PostTypeId="2" ParentId="10057" CreationDate="2017-07-10T18:07:28.253" Score="0" Body="&lt;p&gt;You are describing a problem that can be solved with Probabilistic Graphical Models (PGMs). PGMs use graphs to expresses the conditional dependence structure between random variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/pgmpy/pgmpy&quot; rel=&quot;nofollow noreferrer&quot;&gt;pgmpy&lt;/a&gt; is a Python package for PGMs.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-10T18:07:28.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20315" PostTypeId="2" ParentId="11524" CreationDate="2017-07-10T18:19:53.723" Score="1" Body="&lt;p&gt;You are describing the &quot;Language Modeling&quot; problem in NLP. Language modeling finds the probability distribution over sequences of words - given the words I have seen so far, what are the most likely words that will come next or are missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;word2vec can find reasonable collocations/cooccurrence for individual words. However, word2vec will do a poor job with sentence completion because it does not model grammatical dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find a word2vec nearest individual word demo &lt;a href=&quot;http://bionlp-www.utu.fi/wv_demo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-10T18:19:53.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20316" PostTypeId="2" ParentId="20308" CreationDate="2017-07-10T18:24:56.983" Score="9" Body="&lt;p&gt;Pandas black magic:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.groupby(['Date', 'Groups']).sum().sum(&#xA;    level=['Date', 'Groups']).unstack('Groups').fillna(0).reset_index()&#xA;&#xA;# Fix the column names&#xA;df.columns = ['Date', 'one', 'two']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Resulting &lt;code&gt;df&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       Date  one  two&#xA;0  2017-1-1  3.0  0.0&#xA;1  2017-1-2  3.0  4.0&#xA;2  2017-1-3  0.0  5.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-07-11T08:55:17.317" LastActivityDate="2017-07-11T08:55:17.317" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20317" PostTypeId="2" ParentId="9472" CreationDate="2017-07-10T18:31:43.293" Score="1" Body="&lt;p&gt;That is called &quot;entity resolution&quot; or &quot;record linkage&quot;. It is a very hard problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it all possible, I would try to add &quot;type ahead&quot; to the form. That would encourage users to select already used categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pypi.python.org/pypi/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;fuzzywuzzy&lt;/a&gt; is a Python package for approximate string matching.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dedupeio/dedupe&quot; rel=&quot;nofollow noreferrer&quot;&gt;dedupe&lt;/a&gt; is &quot;a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution quickly on structured data.&quot;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-08-09T20:33:02.733" LastActivityDate="2017-08-09T20:33:02.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20318" PostTypeId="1" CreationDate="2017-07-10T19:00:35.390" Score="2" ViewCount="335" Body="&lt;p&gt;I have about 20 categorical variables to predict another categorical variable. One of the variables can have as many as 12000 levels (O/P will be one of the 12000) and another one can have about 8000 levels. Rest will have less than 100 levels. Random Forest was the first thing that came to my mind, but Python’s Random Forest implementation doesn’t support categorical variables. If I one-hot encoded all these variables, then I will end up with thousands of variables for Millions of records which will throw me into memory issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my options?&lt;/p&gt;&#xA;" OwnerUserId="33907" LastEditorUserId="836" LastEditDate="2017-07-11T06:57:02.577" LastActivityDate="2018-09-26T14:09:22.987" Title="Classification with millions of records, thousands of categories - keep memory use efficient?" Tags="|machine-learning|statistics|algorithms|random-forest|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20319" PostTypeId="1" CreationDate="2017-07-10T19:49:05.150" Score="0" ViewCount="69258" Body="&lt;p&gt;I want to begin exploring OpenCV in Python but I'm stuck at importing the package &lt;code&gt;cv2&lt;/code&gt;. I have installed the package through &lt;code&gt;pip3 install opencv-python&lt;/code&gt; and it got installed at this location - &lt;code&gt;C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I'm trying to import &lt;code&gt;cv2&lt;/code&gt; using this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sys&#xA;sys.path.append('C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages')&#xA;import cv2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It gives the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;&amp;lt;stdin&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;  File &quot;C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages\cv2\__init__.py&quot;, line 7, in &amp;lt;modul&#xA;e&amp;gt;&#xA;    from . import cv2&#xA;ImportError: cannot import name 'cv2'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have searched a lot but cannot find anything relevant. Please suggest what needs to be done.&lt;/p&gt;&#xA;" OwnerUserId="33701" LastEditorUserId="8432" LastEditDate="2017-08-12T10:14:54.840" LastActivityDate="2019-02-07T09:14:28.317" Title="Import Error: cannot import name 'cv2'" Tags="|python|computer-vision|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20320" PostTypeId="2" ParentId="20319" CreationDate="2017-07-10T20:12:07.083" Score="0" Body="&lt;p&gt;If you are using &lt;a href=&quot;https://github.com/conda/conda&quot; rel=&quot;nofollow noreferrer&quot;&gt;conda&lt;/a&gt; package manager, it's very straightforward to install opencv by executing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install -c menpo opencv3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2017-07-10T20:12:07.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20321" PostTypeId="1" CreationDate="2017-07-10T20:20:18.740" Score="1" ViewCount="226" Body="&lt;p&gt;I have some excessive amount of data for the size of NN I am able to teach in a reasonable time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I feed all the data into the network it stops learning at some point and a resulting model shows all signs of being overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively if I increase dropout prob the model should learn less aggressively from data and gain from more data being fed into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my logic sound?&lt;/p&gt;&#xA;" OwnerUserId="16162" LastEditorUserId="50727" LastEditDate="2020-07-23T15:13:35.283" LastActivityDate="2020-07-23T15:13:35.283" Title="Should I set higher dropout prob if there are plenty of data?" Tags="|neural-network|deep-learning|overfitting|regularization|dropout|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20322" PostTypeId="1" CreationDate="2017-07-10T20:20:40.053" Score="1" ViewCount="96" Body="&lt;p&gt;&lt;strong&gt;Situation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing a data science project for my client. Due to NDA I can't disclose anything to the nature of the project, but let's imagine that they manufacture and sell slow cookers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I faced the need to get the dataset which will cover all of the 'slow cooker' products in the selected market. The information we want is the list of features of each product. For example a record should have product name and brand + capacity, material, color, cord length etc., everything you could see on a product listing page in something like Amazon. The more detailed it is the better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond that we want either a comprehensive set of product review texts from leading ecommerce platforms. If not available - at least average (also min and max) customer sentiment of such reviews.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Issue&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already did a prototype research by just doing a web-scraping of one of eCommerce web-sites. The issue is that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;It took quite a while&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is not really comprehensive - I want to cover all leading eCommerce platforms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Scraping more platforms will take a lot of time, and normalizing and merging scraping results will be a big pain.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the other hand, client is OK to just pay some money to buy clean dataset from company which already did that - data brokers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the issue I faced is that even though I spent a lot of time googling data brokers I wasn't able to come up with a good list and clearly understand if these companies actually sell the kind of data I want. And honestly if you google 'data broker' absolute majority of results will give you some angry articles from newspapers about 'companies who know more about you than google', not the links to actual companies and the description of what kinds of data they provide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bottom line - is there any advice on how to approach the issue of looking for a good data broker and choosing it? Is there any good resource which covers market leaders in this area with a comprehensive information on which kinds of data they sell?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extra question: I remember some time ago Microsoft launched so-called Azure Data Mart which supposed to be a one-stop shop for datasets sold by various companies, but seems like the idea dies without further development. I wonder if anyone knows of such successful data marts which can be used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why I didn't post it on opendata&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer is simple - I am not looking for open (free) data. Our clients are OK to pay for it. At the same time, OpenData community requires questions to cover only OpenData.&lt;/p&gt;&#xA;" OwnerUserId="7969" LastEditorUserId="7969" LastEditDate="2017-07-10T20:27:52.350" LastActivityDate="2017-07-10T20:27:52.350" Title="Data brokers for market-related data, how to choose, where to look for?" Tags="|dataset|bigdata|marketing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20326" PostTypeId="1" CreationDate="2017-07-11T05:38:04.183" Score="0" ViewCount="562" Body="&lt;p&gt;I have an input dataset (matrix 25x2275) which is normalized to values between 0 and 1. I also have a binary formatted output matrix (3x2275) like 0 0 0 0 0 0 0 0 1, 1 0 0 1 1 1 0 0 1 ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imported both files in matlab nntool and it automatically created a network with 25 input and 3 output nodes as I wanted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After I trained this network using feed-forward backProp, I tested the model in its training data and each output nodes returns a decimal value like &lt;code&gt;(0.9999 0.978 1 0 0.99 0.59368 0.38359 0.31435 1.0604).&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why it doesn't return discrete values like 1 0 0 1 1 1 0 0 1? Is there any thing that I must set in nntool to get such values?&lt;/p&gt;&#xA;" OwnerUserId="20618" LastActivityDate="2017-07-11T09:16:22.973" Title="Why Matlab neural network classification returns decimal values on prediction dataset?" Tags="|machine-learning|neural-network|classification|deep-learning|matlab|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20327" PostTypeId="1" CreationDate="2017-07-11T06:31:51.767" Score="6" ViewCount="2451" Body="&lt;p&gt;I have word images as below:&lt;br&gt;&#xA;&lt;img src=&quot;https://raw.githubusercontent.com/synckey/tensorflow_lstm_ctc_ocr/master/00000007_73791096754314441539_1.png&quot; alt=&quot;Word Image&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say it's a &lt;code&gt;256x64&lt;/code&gt; image. My aim is to extract the text from the image as &lt;code&gt;73791096754314441539&lt;/code&gt; which is basically what an OCR does.&lt;br&gt;&#xA;I am trying to build model which can recognise word from images.&lt;br&gt;&#xA;When I am saying word it can be any of the follwing:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Any dictionary word, non-dictionary word&lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;a-z,A-Z, special characters including &lt;code&gt;spaces&lt;/code&gt; &lt;br&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have built a model (snippet because of company policies) in tensorflow as below:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = tf.placeholder(tf.float32, [common.BATCH_SIZE, common.OUTPUT_SHAPE[1], common.OUTPUT_SHAPE[0], 1])&#xA;# Here we use sparse_placeholder that will generate a&#xA;# SparseTensor required by ctc_loss op.&#xA;targets = tf.sparse_placeholder(tf.int32)&#xA;&#xA;# 1d array of size [batch_size]&#xA;seq_len = tf.placeholder(tf.int32, [common.BATCH_SIZE])&#xA;&#xA;model = tf.layers.conv2d(inputs, 64, (3,3),strides=(1, 1), padding='same', name='c1')&#xA;model = tf.layers.max_pooling2d(model, (3,3), strides=(2,2), padding='same', name='m1')&#xA;model = tf.layers.conv2d(model, 128,(3,3), strides=(1, 1), padding='same', name='c2')&#xA;model = tf.layers.max_pooling2d(model, (3,3),strides=(2,2), padding='same', name='m2')&#xA;model = tf.transpose(model, [3,0,1,2])&#xA;shape = model.get_shape().as_list()&#xA;model = tf.reshape(model, [shape[0],-1,shape[2]*shape[3]])&#xA;&#xA;cell = tf.nn.rnn_cell.LSTMCell(common.num_hidden, state_is_tuple=True)&#xA;cell = tf.nn.rnn_cell.DropoutWrapper(cell, input_keep_prob=0.5, output_keep_prob=0.5)&#xA;stack = tf.nn.rnn_cell.MultiRNNCell([cell]*common.num_layers, state_is_tuple=True)&#xA;&#xA;outputs, _ = tf.nn.dynamic_rnn(cell, model, seq_len, dtype=tf.float32,time_major=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My current approach is to use take input a word image pass it through a &lt;code&gt;CNN&lt;/code&gt; extract high level image features, convert the image features to sequential data as below &lt;br&gt; &lt;code&gt;[[a1,b1,c1],[a2,b2,c2],[a3,b3,c3]] -&amp;gt; [[a1,a2,a3],[b1,b2,b3],[c1,c2,c3]]&lt;/code&gt; &lt;br&gt;&#xA;then pass it through a RNN(LSTM or BiLSTM), then use &lt;code&gt;CTC&lt;/code&gt; (Connectionist Temporal Loss) to find the loss and train network. &lt;br&gt;&#xA;I am not getting results as expected, I wanted to know if:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There is some other better way to do this task&lt;/li&gt;&#xA;&lt;li&gt;If I am converting features to sequence correctly&lt;/li&gt;&#xA;&lt;li&gt;Any research paper where something like this is done.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33704" LastActivityDate="2017-07-11T14:33:22.257" Title="How to pass features extracted using CNN into RNN?" Tags="|neural-network|deep-learning|tensorflow|rnn|ocr|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20329" PostTypeId="1" CreationDate="2017-07-11T07:17:05.203" Score="0" ViewCount="2799" Body="&lt;p&gt;Deep Learning (DL) is now days are used in various fields. I may like to know any idea/approach on usin DL for Signature extraction from emails. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some papers which uses ML like&lt;br&gt; &lt;a href=&quot;http://keg.cs.tsinghua.edu.cn/jietang/publications/f142-tang.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://keg.cs.tsinghua.edu.cn/jietang/publications/f142-tang.pdf&lt;/a&gt;&#xA;&lt;a href=&quot;http://www.cs.cmu.edu/~vitor/papers/sigFilePaper_finalversion.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.cmu.edu/~vitor/papers/sigFilePaper_finalversion.pdf&lt;/a&gt;  &amp;nbsp;&amp;nbsp; &lt;em&gt;there are few more&lt;/em&gt;.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some open source libraries in Python like talon which uses NLP (Regular expressions and other techniques) and uses ML algorithms ( SVM ) for the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have done a little research but I am not getting any resource or any idea on how to proceed with DL for this task.&lt;br&gt;&#xA;I thought of using seq2seq models but I am not sure it can be of much help for this task. I am not sure whether summarization techniques can be useful.&#xA;Any idea or resources for the same can be helpful.&lt;/p&gt;&#xA;" OwnerUserId="27873" LastEditorUserId="27873" LastEditDate="2017-07-12T04:20:01.700" LastActivityDate="2021-09-27T18:07:43.347" Title="Deep Learning approach for Signature extraction from emails" Tags="|machine-learning|deep-learning|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20330" PostTypeId="2" ParentId="20319" CreationDate="2017-07-11T07:24:06.360" Score="3" Body="&lt;p&gt;According to the official &lt;a href=&quot;https://pypi.python.org/pypi/opencv-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;, if you had previous versions of opencv-python and opencv-contrib-python installed, you should consider removing them first:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; pip uninstall opencv-python&#xA; opencv-contrib-python&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, can you try installing them as sudo?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sudo -H pip install opencv-python&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17177" LastActivityDate="2017-07-11T07:24:06.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20333" PostTypeId="2" ParentId="20326" CreationDate="2017-07-11T09:16:22.973" Score="1" Body="&lt;p&gt;Even for classification problem, the output activation can be any slashing function like sigmoid, tanh, using which the error is back-propagated.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess, its trying to give you continuous value for a labels more like probabilities, you use a cut off (say 0.5) and convert the values to 0 and 1 labels&lt;/p&gt;&#xA;" OwnerUserId="18641" LastActivityDate="2017-07-11T09:16:22.973" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20336" PostTypeId="2" ParentId="10885" CreationDate="2017-07-11T09:56:01.340" Score="0" Body="&lt;p&gt;This really depends on how comprehensive a model you use. In most cases, the model for sentiment analysis is pretty simple and based on the &quot;bag of words.&quot; In this case the positive and negative samples in the training set allow you, basically, to mine  both positive and negative emotionally-loaded words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now imagine you have this job done perfectly but you have some text which contains neither positive nor negative words. This text can be considered neutral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another case is when the text contains positive and negative words in equal numbers. Potentially, this means the text contains both negative sentences and positive ones. Or there might be negations in the text that reverse the sentiment of some words. Could this text be considered neutral? Maybe. But simple &quot;bag of words&quot; models can't distinguish this case from the previous one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this means is that training data containing only positives and negatives could still recognize &quot;neutral&quot; texts. But you'll need to think carefully about neutrality criteria acceptable for your use case, and about the model you are going to use so that you get the exact behavior you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information on this topic, you might want to look at this blog post several colleagues and I wrote about creating training and test data sets: &lt;a href=&quot;https://blog.griddynamics.com/creating-training-and-test-data-sets-and-preparing-the-data-for-twitter-stream-sentiment-analysis-of-social-movie-reviews&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.griddynamics.com/creating-training-and-test-data-sets-and-preparing-the-data-for-twitter-stream-sentiment-analysis-of-social-movie-reviews&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33841" LastEditorUserId="33841" LastEditDate="2017-07-21T13:46:54.233" LastActivityDate="2017-07-21T13:46:54.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20337" PostTypeId="2" ParentId="12226" CreationDate="2017-07-11T10:12:15.297" Score="1" Body="&lt;p&gt;You'll need labeled data. It’s best if you have your own, but if you don’t, make sure you take your labeled data from the &quot;Twitter&quot; domain.  This is important because tweets have a very specific style of writing, a specific lexicon, and use many emoticons. To help, here’s a database of 5K manually labeled tweets from Nick Sanders:  &lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.sananalytics.com/lab/twitter-sentiment/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run the labeled data through several models and calculate a metric. It could be a confusion matrix if you like. If you really don't need to have very precise classifications for all three classes (positive, neutral, negative), and if you need to identify, say, negative tweets, you can use metrics like precision and recall. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want more information about this, my colleagues and I have written a blog post describing how the performance of several models was compared in Twitter sentiment analysis: &lt;a href=&quot;https://blog.griddynamics.com/selecting-training-evaluating-and-tuning-the-model-for-twitter-stream-sentiment-analysis-of-social-movie-review&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.griddynamics.com/selecting-training-evaluating-and-tuning-the-model-for-twitter-stream-sentiment-analysis-of-social-movie-review&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33841" LastActivityDate="2017-07-11T10:12:15.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20339" PostTypeId="1" CreationDate="2017-07-11T10:54:01.270" Score="3" ViewCount="11231" Body="&lt;p&gt;What is meant by frequency of a time series? I am having data observations of energy consumption from 01/05/2016 - 31/05/2017 such that there are 96 values per day. What will be the frequency of the time series in such case?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastActivityDate="2023-02-03T11:25:48.277" Title="Frequency for a time series" Tags="|time-series|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20340" PostTypeId="1" AcceptedAnswerId="101712" CreationDate="2017-07-11T10:55:04.823" Score="6" ViewCount="6278" Body="&lt;p&gt;I'm trying to create a custom evaluation metric (feval) function for xgboost.cv. It should process some of the training features, however I can't find a way to extract features from &lt;code&gt;xgboost.DMatrix()&lt;/code&gt; object (only labels)..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I was suggested it can be done with first making a DMatrix slice then saving it to a svmlight file and finally reading it with scikit-learn, but is there a more elegant way if doing it?&lt;/p&gt;&#xA;" OwnerUserId="35461" LastEditorUserId="29575" LastEditDate="2017-09-28T05:56:05.933" LastActivityDate="2021-09-03T05:52:35.650" Title="python xgboost DMatrix - get feature values or convert to np.array" Tags="|python|xgboost|cross-validation|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20341" PostTypeId="2" ParentId="20304" CreationDate="2017-07-11T11:55:12.350" Score="5" Body="&lt;p&gt;My intuitions is that training each tree on a subset of all variables helps the less useful variable to be used at all. Since often there are some features highly correlated to the target and then all trees will just use these very good features and never ever use the weak ones. By working on random subsets the weak features are used sometimes and contribute to the result a little.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-11T11:55:12.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20342" PostTypeId="2" ParentId="20339" CreationDate="2017-07-11T12:00:26.627" Score="0" Body="&lt;p&gt;I don't think that this is defined in general. So you should be more specific about where you are using this.&lt;/p&gt;&#xA;&lt;p&gt;I could guess, maybe you mean the Time-Series Object in R?&#xA;Then take a look at &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/ts.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;And here the definition is:&lt;/p&gt;&#xA;&lt;p&gt;frequency -&amp;gt; the number of observations per unit of time periods.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastEditorUserId="85045" LastEditDate="2023-02-03T11:25:48.277" LastActivityDate="2023-02-03T11:25:48.277" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20343" PostTypeId="2" ParentId="20339" CreationDate="2017-07-11T12:00:28.223" Score="1" Body="&lt;p&gt;It's mainly defined by how you perceive your data and the way it's sampled, but basically you can define a time-series &lt;strong&gt;frequency&lt;/strong&gt; as:&#xA;(&lt;strong&gt;the number of samples&lt;/strong&gt;) per your (&lt;strong&gt;sample unit&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case: 96 per day, since it seems that a &quot;day&quot; is your meaningful sample-unit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you had several months, or even better - years, you could argue your frequency to be 96*30 (monthly # of samples) and the sample unit as months&lt;/p&gt;&#xA;" OwnerUserId="17177" LastActivityDate="2017-07-11T12:00:28.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20345" PostTypeId="1" CreationDate="2017-07-11T12:03:45.057" Score="1" ViewCount="571" Body="&lt;p&gt;I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those 100 documents or not.Which is the best model for this problem?&lt;/p&gt;&#xA;" OwnerUserId="35449" LastActivityDate="2017-07-11T18:15:44.217" Title="How to Classify Documents whether they are similar to previous documents or not?" Tags="|machine-learning|text-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20346" PostTypeId="2" ParentId="11340" CreationDate="2017-07-11T12:06:44.393" Score="1" Body="&lt;p&gt;I don't completely understand your example, but:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you want to put your model into operation/production environment at some day ... then you should not allow 'looking into the future' at training time.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-11T12:06:44.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20347" PostTypeId="2" ParentId="20170" CreationDate="2017-07-11T12:07:47.747" Score="2" Body="&lt;p&gt;&lt;strong&gt;CG does not converge to the minimum as well as BFGS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I may add an answer here to my own question too, credits given to a good friend who volunteered to look at my code. He's not on Data Science stackexchange, and didn't feel the need to create an account just to post the answer up, so he's passed this chance to post over to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also reference @Neil Slater, as there is chance his analysis on the numerical stability issue could account for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the main premise behind my solution is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that the cost function is convex, meaning it has no locals, and only a global minimum. Since the prediction using parameters trained with BFGS is better than those trained using CG, this implies that BFGS converged closer to the minimum than CG did. Whether or not BFGS converged to the global minimum, we can't say for sure, but we can definitely say that it is closer than CG is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we take the parameters that were trained using CG, and pass them through the optimization routine using BFGS, we should see that these parameters get further optimized, as BFGS brings everything closer to the minimum. This should improve the prediction accuracy and bring it closer to the one obtained using plain BFGS training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here below is code that verifies this, variable names follow the same as in the question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Copy the old array over, else only a reference is copied, and the &#xA;# original vector gets modified&#xA;theta_all_optimized_bfgs_from_cg = np.copy(theta_all_optimized_cg)&#xA;&#xA;for k in range(y.min(),y.max()+1):&#xA;    grdtruth = np.where(y==k, 1,0)&#xA;    results = minimize(compute_cost_regularized,theta_all_optimized_bfgs_from_cg[k-1,:], &#xA;                       args = (X_bias,grdtruth,0.1),&#xA;                       method = &quot;BFGS&quot;, &#xA;                       jac = compute_gradient_regularized, options={&quot;disp&quot;:True})&#xA;    # optimized parameters are accessible through the x attribute&#xA;    theta_optimized = results.x&#xA;    # Assign thetheta_optimized vector to the appropriate row in the &#xA;    # theta_all matrix&#xA;    theta_all_optimized_bfgs_from_cg[k-1,:] = theta_optimized&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;During execution of the loop, only one of the iterations produced a message that showed a non-zero number of optimization routine iterations, meaning that further optimization was performed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Optimization terminated successfully.&#xA;         Current function value: 0.078457&#xA;         Iterations: 453&#xA;         Function evaluations: 455&#xA;         Gradient evaluations: 455&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the results were improved:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In[19]:  predict_one_vs_all(X_bias, theta_all_optimized_bfgs_from_cg)&#xA;Out[19]:  96.439999999999998&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By further training the parameters, which were initially obtained from CG, through an additional BFGS run, we have further optimized them to give a prediction accuracy of &lt;code&gt;96.44%&lt;/code&gt; which is very close to the &lt;code&gt;96.48%&lt;/code&gt; that was obtained by directly using just only BFGS!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I updated my notebook with this explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this raises more questions, such as why CG did not work as well as BFGS did on this cost function, but I guess those are questions meant for another post. &lt;/p&gt;&#xA;" OwnerUserId="33716" LastActivityDate="2017-07-11T12:07:47.747" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20349" PostTypeId="1" AcceptedAnswerId="20366" CreationDate="2017-07-11T13:02:25.317" Score="1" ViewCount="120" Body="&lt;p&gt;I need to merge data from 1000s of excel files provided by different operations managers on productivity and other reports. The excel files have similarity of data but the headers are all custom since being different manager and different clients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example manager A will have a.xlx and manager b will have a.xlx, but the headers for each will be different though the data inside will be usually the same. Each day 100 different excel files are updated by all team members via new files e.g a_todays_date.xlx and manager b using /a.xlx.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something that can be handled via python ML libraries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to merge all these data and save to the DB on a per day basis. The average data would per day would be around 15GB. The end goal is to create dashboard.&lt;/p&gt;&#xA;" OwnerUserId="35468" LastActivityDate="2017-07-12T05:32:05.170" Title="Excel file merge with different headers but same data" Tags="|text-mining|excel|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20350" PostTypeId="2" ParentId="11661" CreationDate="2017-07-11T13:44:58.277" Score="2" Body="&lt;p&gt;R is a powerful tool but you need to be able to program a bit. If you are looking for an alternative to @RISK, there are several risk analysis software packages reviewed on &lt;a href=&quot;https://en.wikipedia.org/wiki/Comparison_of_risk_analysis_Microsoft_Excel_add-ins&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt; . Many are cheap or free, but they are written using VBA, pretty slow and report using Excel charts. The neaarest package to @RISK from Palisade is ModelRisk from &lt;a href=&quot;http://www.vosesoftware.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vose Software&lt;/a&gt;. It's about 1/3 the price, performs better, has more features, a better help file and is much easier to use. I used it for my PhD because it was the only tool that had the technical features I needed.&lt;/p&gt;&#xA;" OwnerUserId="35472" LastActivityDate="2017-07-11T13:44:58.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20351" PostTypeId="2" ParentId="20327" CreationDate="2017-07-11T14:33:22.257" Score="3" Body="&lt;p&gt;1 and 2. You are in the right direction, you need to extract the features using a CNN, then instead of predicting the class you want to reshape the last layer of features and feed it directly into the RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of things to pay attention to: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With a rather shallow CNN you aren't taking advantage of the &lt;strong&gt;higher-level feature extraction&lt;/strong&gt; these architectures can offer. If all your images are as simple as the example you've shown it is adequate.&lt;/li&gt;&#xA;&lt;li&gt;If you are considering a larger CNN, along with the RNN there is a substantial number of parameters to be trained. For this you need &lt;strong&gt;lots&lt;/strong&gt; of data and &lt;strong&gt;lots&lt;/strong&gt; of computational resources (either very strong GPUs or time).&lt;/li&gt;&#xA;&lt;li&gt;In order for you to get the best of the two, I would suggest incorporating a &lt;strong&gt;pre-trained&lt;/strong&gt; CNN into your model (and just fine tune the latter layers). This  pre-trained model can even be trained in generic images (i.e. ImageNet) and will substantially increase the performance of the CNN without computational cost. You then can train the latter layers of this CNN jointly with the RNN. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;3.&lt;a href=&quot;https://ai2-s2-pdfs.s3.amazonaws.com/8e91/49ab00236d04db23394774e716c4f1d89231.pdf&quot; rel=&quot;noreferrer&quot;&gt;This&lt;/a&gt; is a good example of what you are trying to do. They basically try to recognize text from street photographs among other things with the same methodology you describe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar methodologies can be found in other research domains such as &lt;a href=&quot;https://www.ics.uci.edu/~yyang8/research/cnn-rnn/cnn-rnn-cvpr2016.pdf&quot; rel=&quot;noreferrer&quot;&gt;multi-label image classification&lt;/a&gt;, &lt;a href=&quot;http://www.aclweb.org/anthology/P16-1101&quot; rel=&quot;noreferrer&quot;&gt;sequence labelling&lt;/a&gt;, &lt;a href=&quot;http://www.cs.nott.ac.uk/~pszmv/Documents/WACV2016_shashank.pdf&quot; rel=&quot;noreferrer&quot;&gt;facial expression recognition&lt;/a&gt;, etc&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2017-07-11T14:33:22.257" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20353" PostTypeId="2" ParentId="20345" CreationDate="2017-07-11T15:57:17.657" Score="1" Body="&lt;p&gt;Only way to check which is best model is to try different in practise and compare. Do you have any documents to train with that are not similar? Do you have labels or topics assigned to these 100 ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you have some irrelevant documents you can use a &quot;spam&quot; approach to this type of problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would begin with Bag of Words representing of your documents for a Naive Bayes model generation (common, good performing, baseline here) - this link &lt;a href=&quot;http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification&lt;/a&gt; is a good explanation of this for documents .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also try using a library like Python gensim that will allow you to try its various methods, and you could try that library's Doc2Vec method. Then train a linear discriminant classifier like Log. Regression with the document vectors it gives you, and try see if its resulting accuracy is better than the Naive Bayes method.  &lt;/p&gt;&#xA;" OwnerUserId="26559" LastActivityDate="2017-07-11T15:57:17.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20354" PostTypeId="1" CreationDate="2017-07-11T16:08:00.073" Score="1" ViewCount="249" Body="&lt;p&gt;I am using the scipy.stats.t.fit function, and I am surprised by the results. If I fit on some bimodal data, say&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data=[1,1,1,1,5,5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df=0.39126249808550329&#xA;loc=1.0&#xA;scale=5.7172845190830792e-21&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That is, the scale is effectively zero, and I will never be able to sample anything near 5, just the more frequent data point 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess you really can't fit on data that is too different from a t-distribution - but is scipy really giving the best t dist fit to the data? I would think that if I compute a sample mean and variance myself, i.e, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = 5&#xA;loc = 2.33&#xA;scale = 1.88&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That I'd have a better fit, although I haven't computed the likelihood of sampling &lt;code&gt;[1,1,1,5,5]&lt;/code&gt; from these two t distributinos.&lt;/p&gt;&#xA;" OwnerUserId="33544" LastActivityDate="2017-07-11T18:00:16.773" Title="scipy fit for t distribution seems broken for bi-modal data" Tags="|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20355" PostTypeId="2" ParentId="20304" CreationDate="2017-07-11T16:25:19.050" Score="8" Body="&lt;p&gt;The idea of random forests is basically to build many decision trees (or other weak learners) that are &lt;em&gt;decorrelated&lt;/em&gt;, so that their average is less prone to overfitting (reducing the variance). One way is subsampling of the training set. The reason why subsampling features can further decorrelate trees is, that if there are few dominating features, these features will be selected in many trees even for different subsamples, making the trees in the forest similar (correlated) again. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The lower the number of sampled features, the higher the decorrelation effect. On the other hand, the bias of a random forest is the same as the bias of any of the sampled trees (see for example Elements of Statistical Learning), but the randomization of random forests restrict the model, so that the bias is usually higher than a fully-grown (unpruned) tree. You are correct in that you can expect a higher bias if you sample fewer features. So, &quot;feature bagging&quot; really gives you a classical trade-off in bias and variance.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-07-11T16:25:19.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20356" PostTypeId="1" CreationDate="2017-07-11T17:42:22.097" Score="2" ViewCount="42" Body="&lt;p&gt;I have a database that I send to an excel  sheet that auto populates some field in a report with calculations that I need. There are several users and the excel sheet cannot be open at multiple times which has become an issue, after the data is submitted in the database the excel doesn't come up which the users need to print out every time. &#xA;My question is, how can I make the excel read only so that they can just print and save the document while multiple users being in the excel since it is read only. &#xA;The problem with this has become the fact that access sees the file as already open and doesn't write the data. Is there a solution without me making a report in access.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="32374" LastActivityDate="2017-07-12T05:08:17.913" Title="MS Access issue" Tags="|databases|excel|" AnswerCount="1" CommentCount="1" ClosedDate="2017-07-12T20:49:47.080" ContentLicense="CC BY-SA 3.0" />
  <row Id="20357" PostTypeId="2" ParentId="20354" CreationDate="2017-07-11T18:00:16.773" Score="1" Body="&lt;p&gt;The t-test has many assumptions. That dataset violates several of them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data should be sufficiently large (&gt;30 independent points)&lt;/li&gt;&#xA;&lt;li&gt;Data should be approximately normally distributed&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Given that the assumptions are violated, you can &lt;strong&gt;not&lt;/strong&gt; expect to valid results. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-11T18:00:16.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20358" PostTypeId="2" ParentId="20345" CreationDate="2017-07-11T18:15:44.217" Score="1" Body="&lt;p&gt;The simplest method to test if a new document is like previous documents is to hash them and look for collisions. Hashing items put them into &quot;buckets&quot;. If two items are in the same &quot;bucket&quot; (aka, collide), they are similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, people use &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow noreferrer&quot;&gt;locality-sensitive hashing (LSH)&lt;/a&gt; for document similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Shingling - Pick a window size and slide the window over a document&lt;/li&gt;&#xA;&lt;li&gt;MinHash - Compute the hashes of the text shingles a number of times and pick the minimum to reduce data size&lt;/li&gt;&#xA;&lt;li&gt;LSH - Bucket items with the same MinHash values&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can compute the Jaccard Similarity between hashed documents and choose a similarity threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is covered in detail in &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chapter 3/&quot;Finding Similar Items&quot;&lt;/a&gt; of &lt;a href=&quot;http://www.mmds.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mining of Massive Datasets&lt;/a&gt; book.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-11T18:15:44.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20359" PostTypeId="1" CreationDate="2017-07-11T18:44:23.277" Score="3" ViewCount="1635" Body="&lt;p&gt;I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something to be an anomaly when it is within the decision function space for inliers? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've attached my results here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PMgky.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PMgky.png&quot; alt=&quot;figure1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could the size of the decision function space be due to my input dimension vs this 2 dimensional projection?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also made a quick plot of anomaly score vs. prediction (0 = inlier, 1 = anomaly):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/24aPK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/24aPK.png&quot; alt=&quot;anomalies&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As seen, there exists outliers above the threshold score, which doesn't make sense to me. Can someone explain?&lt;/p&gt;&#xA;" OwnerUserId="35481" LastEditorUserId="35481" LastEditDate="2017-07-12T13:43:27.387" LastActivityDate="2024-01-06T06:03:45.407" Title="IsolationForest Decision Function vs. Anomaly Prediction Question" Tags="|machine-learning|scikit-learn|random-forest|anomaly-detection|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="20360" PostTypeId="1" CreationDate="2017-07-11T19:27:12.433" Score="1" ViewCount="818" Body="&lt;p&gt;I know that the algorithmic complexity of CNNs and other methods of deep learning can't be fully expressed in simple terms, like big oh complexity. That said, how would one go about comparing the efficiency/complexity of CNNs to standard machine learning methods (say decision trees, LDAs, Naive Bayes, etc.)? Deep learning methods are expensive, but how do we know how expensive they are comparatively? &lt;/p&gt;&#xA;" OwnerUserId="35484" LastActivityDate="2017-07-12T09:58:35.970" Title="Comparing Efficiency of Machine Learning Algorithms" Tags="|machine-learning|deep-learning|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20363" PostTypeId="1" CreationDate="2017-07-12T01:37:41.657" Score="3" ViewCount="196" Body="&lt;p&gt;how is countvectorizer used in real production environment?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;do you use a pipeline to streamline the process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is the best practice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we are going to implement a combination of countvectorizer,tfidf and some machine learning algo in production system soon and any tips or practical experiences will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks!&lt;/p&gt;&#xA;" OwnerUserId="31563" LastActivityDate="2017-07-12T01:37:41.657" Title="how is countvectorizer used in real production environment?" Tags="|nlp|preprocessing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20364" PostTypeId="2" ParentId="20360" CreationDate="2017-07-12T03:18:10.087" Score="1" Body="&lt;p&gt;One deep learning method can apply to multiple problems (for example computer vision, natural language processing etc), the performance of the method may be different depends on the problem that the method handles, but there is &quot;popular dataset&quot; that usually use by deep learning researchers to benchmark their method. Let say you have deep learning method, if you want to know how efficient your method for handwritten classification you can benchmark your method by classifying &lt;strong&gt;mnist dataset&lt;/strong&gt; and compute its accuracy. If you want to compare the performance of your method with another method you can search on the internet another method that uses the same dataset for benchmark, &lt;a href=&quot;http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here is an example&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24696" LastEditorUserId="24696" LastEditDate="2017-07-12T09:24:58.057" LastActivityDate="2017-07-12T09:24:58.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20365" PostTypeId="2" ParentId="20356" CreationDate="2017-07-12T05:08:17.913" Score="1" Body="&lt;p&gt;Instead  of sending data to Excel, Try to create  a Query that populates data in Access&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then From Excel  use Data-From Access to import the data into Excel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Users can hit refresh data (or you can put a Macro to do that every time user opens the file) data from Access  will be in Excel. &lt;/p&gt;&#xA;" OwnerUserId="35491" LastActivityDate="2017-07-12T05:08:17.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20366" PostTypeId="2" ParentId="20349" CreationDate="2017-07-12T05:32:05.170" Score="1" Body="&lt;p&gt;I suggest SSIS (SQL Server Integration Services) . &#xA;It is designed for handling collecting data  from different sources and export  to database.&#xA;You can design a simple data flow and run it from SQL serevr on schedule (daily in your case)&lt;/p&gt;&#xA;" OwnerUserId="35491" LastActivityDate="2017-07-12T05:32:05.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20367" PostTypeId="1" AcceptedAnswerId="20386" CreationDate="2017-07-12T06:56:09.017" Score="1" ViewCount="1088" Body="&lt;p&gt;When doing Data Science, we concentrate on feature engineering first - check on correlations, imputations, tranformations etc.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do we have to follow the same steps while feeding the inputs to a neural net? Or what should be the steps taken?&lt;/p&gt;&#xA;" OwnerUserId="24575" LastEditorUserId="24000" LastEditDate="2017-07-12T09:35:19.623" LastActivityDate="2020-08-08T08:58:29.323" Title="Feature engineering while using neural networks" Tags="|neural-network|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20371" PostTypeId="1" CreationDate="2017-07-12T12:43:22.993" Score="5" ViewCount="17233" Body="&lt;p&gt;There are 96 observations of energy consumption per day from 01/05/2016 - 31/05/2017. I am trying an ARIMA model in R to be fitted to these time series observations. I have chosen the frequency of time series as 96. In total, there are 38016 observations.&#xA;I have cleaned the series using &lt;code&gt;tsclean&lt;/code&gt; command in R to remove the outliers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timeseries &amp;lt;- ts(full$consumption, frequency = 96)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vJsDA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vJsDA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cleansed time series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timeseries &amp;lt;- tsclean(timeseries)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SVKnR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SVKnR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have then differenced the series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diffts &amp;lt;- diff(timeseries)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PlMpc.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PlMpc.png&quot; alt=&quot;differenced timeseries&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have then decomposed the series to obtain the seasonal component from it. Subtracted the seasonal component from it to deseasonalize it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;difftscomponent &amp;lt;- decompose(diffts)&#xA;adjusted_diffts &amp;lt;- diffts - difftscomponent$seasonal&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The ACF plot of final time series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;acf(adjusted_diffts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xCmJr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xCmJr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The PACF of the final time series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pacf(adjusted_diffts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wCuXO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wCuXO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Normally, the X-axis of ACF and the PACF plot of the time series will show lag order from 1 to ... . There will be integer values indicating the number of lags. Then why in my case there are decimal values from 0.1.... ? What does that indicate? What to do to get number of lags?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What will be the probable order of AR and MA in my case by looking at ACF and PACF plot?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have transformed the original time series i.e. made it stationary and deseasonalized it in order to fit an ARIMA model. By looking at the final adjusted time series, can it be said that it is fit to be modelled OR will it require any further transformation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30006" LastEditDate="2017-07-12T15:23:32.607" LastActivityDate="2024-01-28T12:04:15.557" Title="Plot of ACF &amp; PACF" Tags="|r|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20372" PostTypeId="1" CreationDate="2017-07-12T12:45:25.890" Score="5" ViewCount="479" Body="&lt;p&gt;I am long-time engineer with almost zero machine learning experience, who is trying to determine a good starting point to solve my problem (hopefully using machine learning).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem (I'll keep it simple):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ultimately, I wish to be able to automatically assign a category to a financial transaction description&lt;/li&gt;&#xA;&lt;li&gt;For example: &quot;121217 POST XX123 TONYS COFFEE&quot; with an amount of $5, should map to &quot;Food &amp;amp; Drink&quot;&lt;/li&gt;&#xA;&lt;li&gt;Transaction descriptions are unstructured and often consist of unnatural language; also sometimes words can be truncated, or concatenated to others words&lt;/li&gt;&#xA;&lt;li&gt;There may or may not be &quot;features&quot; such as country/region codes, dates, etc., in a description&lt;/li&gt;&#xA;&lt;li&gt;Overall, there is no guarantee to the order or structure of the tokens in a description&lt;/li&gt;&#xA;&lt;li&gt;A transaction will always have an amount&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Assumptions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I maintain a list of potential categories (maybe, 20 or 30 in total)&lt;/li&gt;&#xA;&lt;li&gt;I could maintain a huge list of business names mapped to their distinctive category (may not be necessary to use, though?)&lt;/li&gt;&#xA;&lt;li&gt;There is an existing set of training labelled data (raw descriptions/amounts and the category they belong to; in the thousands, not millions)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A best effort to extract a business name from a description, could be considered, but it would be great if that was not necessary for the accuracy of a final system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was originally thinking about NLP somewhat, but as this is fairly non-natural language, with no semantics, I believe there is no use of NLP. NER (named entity recognition) is maybe not really helpful either, as it is generally required to understand a text somewhat in order to determine entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am toying with the idea of downloading GloVes pre-trained word vectors (&lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nlp.stanford.edu/projects/glove/&lt;/a&gt;), to help determine words related to categories, though I am unsure how as of now or how well that might work. The idea might be if I trained something to say &quot;Jimmy's Diner&quot; -&gt; &quot;Food &amp;amp; Drink&quot;, then, for example, &quot;Bobby's Rest&quot; might map to &quot;Food &amp;amp; Drink&quot; too, as that is the nearest category to it in terms of word relationships/distance. It's depends on the possibility of being able to query the word embedding in such a way, as well as train it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess in other to train a system using labelled data, I'd need to extract features from the description. The problem is, what features? Some features might be useless (unique identifiers, concatenated words, etc.). I'd need the system to be somewhat forgiving in terms of polluting it with useless features (avoidable to a degree, but probably not inevitable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, it would be great to hear how some of you experts might begin to approach this: what ML techniques would you deem most suitable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have researched machine learning and deep learning &amp;amp; the associated frameworks quite a bit over the last few days, but there is so many areas, with so much potential, that it is hard to to know where to begin.&lt;/p&gt;&#xA;" OwnerUserId="35508" LastEditorUserId="8432" LastEditDate="2017-08-12T10:14:45.200" LastActivityDate="2017-08-12T10:14:45.200" Title="Which machine (or deep) learning methods could suit my text classification problem?" Tags="|machine-learning|classification|deep-learning|nlp|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20373" PostTypeId="1" AcceptedAnswerId="54274" CreationDate="2017-07-12T15:14:32.200" Score="3" ViewCount="1298" Body="&lt;p&gt;I'm trying to investigate the ways in which FPGAs differ to GPUs for the purpose of deep learning. I understand this is a complex question and not necessarily easy to answer in one go, however what I'm looking for are the key differences between the two technologies for this domain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, it would be nice to see the benefits and cons of both technologies with regards to Deep Learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, from my understanding, and please correct me if I'm wrong, is that with FPGAs, those undertaking a DL project would need someone who is able to configure the FPGA according to the type of project they want through the use of languages such as Verilog or VDHL. Furthermore, if they want to change the type of DL project they want to do, they have to reconfigure the FPGA to follow suit. Have I got this correct, or is something like Intel's Deep Learning Inference Accelerator (&lt;a href=&quot;https://www-ssl.intel.com/content/www/us/en/servers/accelerators/deep-learning-inference-accelerator-product-detail.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www-ssl.intel.com/content/www/us/en/servers/accelerators/deep-learning-inference-accelerator-product-detail.html&lt;/a&gt;) a lot more similar to GPUs than I thought in terms of configuration?&lt;/p&gt;&#xA;" OwnerUserId="35512" LastActivityDate="2019-06-22T07:52:51.677" Title="What are the key differences between FPGA and GPUs for Deep Learning?" Tags="|deep-learning|gpu|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20376" PostTypeId="2" ParentId="20372" CreationDate="2017-07-12T17:08:19.373" Score="2" Body="&lt;p&gt;You state that some of the words may occasionally be truncated or concatenated. Thus I would extract &lt;strong&gt;n-grams&lt;/strong&gt; from your Strings and then use that into a &lt;strong&gt;bag-of-words&lt;/strong&gt; vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does this work?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;n-grams&lt;/strong&gt; is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The segmentation works as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The String: Hello World&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2-gram: &quot;He&quot;, &quot;el&quot;, &quot;ll&quot;, &quot;lo&quot;, &quot;o &quot;, &quot; W&quot;, &quot;Wo&quot;, &quot;or&quot;, &quot;rl&quot;, &quot;ld&quot;&#xA;3-gram: &quot;Hel&quot;, &quot;ell&quot;, &quot;llo&quot;, &quot;lo &quot;, &quot;o W&quot;, &quot; Wo&quot;, &quot;Wor&quot;, &quot;orl&quot;, &quot;rld&quot;&#xA;4-gram: &quot;Hell&quot;, &quot;ello&quot;, &quot;llo &quot;, &quot;lo W&quot;, &quot;o Wo&quot;, &quot; Wor&quot;, &quot;Worl&quot;, &quot;orld&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bag-of-Words&lt;/strong&gt; builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Due to the high number of grams that will result, you will want to do some feature dimensionality reduction. You can use techniques such as PCA and LDA to determine which features (grams) are most pertinent to your decision boundary. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-07-12T17:08:19.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20377" PostTypeId="2" ParentId="20329" CreationDate="2017-07-12T17:14:16.633" Score="2" Body="&lt;p&gt;Detecting a signature in an email is more like a detection problem in an image rather than a natural language processing (NLP) problem. It does not particularly matter what the signature says, you simply want to be able to detect it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such a case, detection in images is currently dominated by &lt;strong&gt;convolutional neural networks (CNN)&lt;/strong&gt;. This is a very powerful technique which has proved to be the best for a multitude of image-based tasks. However, it &lt;strong&gt;requires a very large dataset&lt;/strong&gt;. So you will need to collect a lot of varied examples and annotate them yourself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output of the CNN can provide either a binary output 0/1 determining if a signature exists. Or it can provide an output which is the &lt;strong&gt;coordinates of the center of the signature (X, Y coordinates)&lt;/strong&gt; such that you can know where it is placed in the document. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-07-12T17:14:16.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20378" PostTypeId="2" ParentId="11661" CreationDate="2017-07-12T18:25:51.007" Score="0" Body="&lt;p&gt;I have not tried it, but &lt;a href=&quot;https://boozallen.github.io/argo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Argo&lt;/a&gt; seems to do Monte Carlo simulations and is open source and thus free. It claims to be fast, full feature and integrated with Excel. It looks like it is more like using @Risk than the would be working in R. The screen shots on the website suggest it is worth trying.&lt;/p&gt;&#xA;" OwnerUserId="27581" LastActivityDate="2017-07-12T18:25:51.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20379" PostTypeId="2" ParentId="20372" CreationDate="2017-07-12T19:21:05.120" Score="1" Body="&lt;p&gt;Referencing @JahKnows answer, I think what he may be trying to refer to is &lt;strong&gt;stemming&lt;/strong&gt; not &lt;strong&gt;n-grams&lt;/strong&gt;. (I would've commented but not enough reputation). Using the python package nltk it should allow you to stem the words given that you want to only get the roots of each word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I've worked on, &lt;strong&gt;n-grams&lt;/strong&gt; are sequences of words with length &lt;em&gt;n&lt;/em&gt;. They are still very helpful for the &lt;strong&gt;bag of words&lt;/strong&gt; suggestion. The python library scikit-learn has a class &lt;strong&gt;CountVectorizer&lt;/strong&gt; which lets you create this model. In addition, it also lets you set a &lt;strong&gt;max&lt;/strong&gt; number of features which can act as your feature reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, if you were going this route, I would continue and use &lt;strong&gt;Tf-idf&lt;/strong&gt; which will find out 'how important' a word is to a document. You can read more on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;wiki&lt;/a&gt;. Scikit-learn has a great tutorial on working with text data which utilizes both &lt;strong&gt;CountVectorizer&lt;/strong&gt; and &lt;strong&gt;Tf-idf&lt;/strong&gt;: &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Working With Text Data&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last steps would be is to choose a multi-class classification model and feed it into a &lt;strong&gt;OneVsRestClassifier&lt;/strong&gt; (all in scikit-learn) and choose which one works the best. (As a start you can try &lt;strong&gt;Naive Bayes, SVM&lt;/strong&gt;, etc.) This should work well since you already have a lot (I'm assuming) of labeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="35481" LastActivityDate="2017-07-12T19:21:05.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20380" PostTypeId="2" ParentId="20319" CreationDate="2017-07-12T19:34:58.640" Score="1" Body="&lt;p&gt;Although you'll find a lot of tutorials that help you install opencv and ffmpeg (both go hand in hand almost always) out there, I will post what worked for me here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps for installing ffmpeg correctly&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Download the latest static version of ffmpeg from the download page depending on your os.&lt;/li&gt;&#xA;&lt;li&gt;Extract the installed zip file using 7-zip. The folder will be named something like &quot;ffmpeg-20130731-git-69fe25c-win32-static&quot;. &lt;/li&gt;&#xA;&lt;li&gt;Create a new folder named &lt;code&gt;ffmpeg&lt;/code&gt; in your &lt;code&gt;C:&lt;/code&gt; and copy the contents of the extracted folder into this new one. &lt;/li&gt;&#xA;&lt;li&gt;Now click the start button, right-click on computer. Select Properties from the right-click menu. In the System window, click on the “Advanced system settings” link.&lt;/li&gt;&#xA;&lt;li&gt;Click the Environmental Variables button in the System Properties window. It will be located at the bottom of the window.&lt;/li&gt;&#xA;&lt;li&gt;Select the PATH entry in the &quot;User variables&quot; section. Click the Edit button.&#xA;In the “Variable value” field, enter &lt;code&gt;;c:\ffmpeg\bin&lt;/code&gt; after anything that's already written there. If you copied it to a different drive, change the drive letter.&#xA;Click OK to save your changes. &#xA;If there is no PATH entry in the &quot;User variables&quot; setting, click the New button and create one. Enter PATH for the variable name.&#xA;This method will enable FFmpeg for the current user. Other Windows users will not be able to run it from the command line. To enable it for everyone, enter &lt;code&gt;;c:\ffmpeg\bin&lt;/code&gt; in the PATH entry in &quot;System variables&quot;. Be very careful not to delete anything that is already in this variable.&lt;/li&gt;&#xA;&lt;li&gt;Open the command prompt. Enter the command &lt;code&gt;ffmpeg –version&lt;/code&gt;. If the command prompt returns the version information for FFmpeg, then the installation was successful, and FFmpeg can be accessed from any folder in the command prompt.&#xA;If you receive a &lt;code&gt;libstdc++ -6 is missing&lt;/code&gt; error, you may need to install the Microsoft Visual C++ Redistributable Package, which is available for free from Microsoft.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I followed these steps from a site I don't remember but this worked just as fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to install opencv and get it working, I followed the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Download the self-extracting executable from the OpenCV website.&lt;/li&gt;&#xA;&lt;li&gt;Run the Executable which will in turn extract the executable.&lt;/li&gt;&#xA;&lt;li&gt;Create a folder in &lt;code&gt;C:\&lt;/code&gt; called &lt;code&gt;opencv&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Copy the contents of the extracted files into &lt;code&gt;opencv&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Edit your PATH environment variable. This will be dependent on the version you want installed (i.e. 32 or 64 bit) and also the version of visual studio you have installed (express editions are fine).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;----Installing OpenCV Python&#xA;I use Anaconda. So, I copied the &lt;code&gt;cv2.pyd&lt;/code&gt; file from this OpenCV directory (the beginning part might be slightly different on your machine):&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Python 2.7 and 32-bit machine:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;C:\opencv\build\python\2.7\x84&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Python 2.7 and 64-bit machine:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;C:\opencv\build\python\2.7\x64&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To this Anaconda directory (the beginning part might be slightly different on your machine):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;C:\Users\xxx\Anaconda\Lib\site-packages&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OpenCV also requires that numpy be installed as well. matplolib is also recommended to be installed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll also have to locate the two &lt;code&gt;.dll&lt;/code&gt; files in the &lt;code&gt;bin&lt;/code&gt; folder of your &lt;code&gt;opencv&lt;/code&gt; folder. And copy and paste them into the &lt;code&gt;ffmpeg/bin&lt;/code&gt; folder as well as into the site-packages folder in your Anaconda folder. When I installed opencv, I didn't have the bin folder installed (weird, I know). So, I copied my friend's &lt;code&gt;.dll&lt;/code&gt; files and renamed them to the version of opencv that I was using and it worked!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After installing both OpenCV and ffmpeg you'll need to restart your computer to add the paths to the system path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps! For a lot of people, most of the steps above were not necessary. But for me, this worked! I use Windows 10, Opencv 3.0, Python 2.7!&lt;/p&gt;&#xA;" OwnerUserId="34333" LastActivityDate="2017-07-12T19:34:58.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20382" PostTypeId="2" ParentId="12727" CreationDate="2017-07-12T22:21:42.613" Score="5" Body="&lt;p&gt;There is a recently-added feature in Spark 2.2.0...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;spark.read.csv(file, multiLine=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://issues.apache.org/jira/browse/SPARK-19610&quot; rel=&quot;noreferrer&quot;&gt;https://issues.apache.org/jira/browse/SPARK-19610&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://issues.apache.org/jira/browse/SPARK-20980&quot; rel=&quot;noreferrer&quot;&gt;https://issues.apache.org/jira/browse/SPARK-20980&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35521" LastActivityDate="2017-07-12T22:21:42.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20386" PostTypeId="2" ParentId="20367" CreationDate="2017-07-13T02:35:08.670" Score="0" Body="&lt;p&gt;If you are talking about tabular data (not images, video, sounds etc) then yes. All of preparations you mentioned are applicable to Artificial neural networks.&lt;/p&gt;&#xA;&lt;p&gt;For instance, data normalization is very important since neurons activations are calculated applying some activation function on a weighted sum(linear combination) of it's inputs. A value of 0.01 and a value of 10000000 in a weighted sum will always have a large value if the weights are at the same scale. Theoretically the network could learn how to deal with such discrepancies on the input values but in a practical sense normalization can make learning faster.&lt;/p&gt;&#xA;&lt;p&gt;There is a classical paper on efficient backprop, with many tips and tricks, by &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Yann LeCun&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15915" LastEditorUserId="98307" LastEditDate="2020-08-08T08:58:29.323" LastActivityDate="2020-08-08T08:58:29.323" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20389" PostTypeId="2" ParentId="17921" CreationDate="2017-07-13T04:58:24.937" Score="4" Body="&lt;p&gt;Perhaps I can answer the first and second one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1)According to (93) on his book:&#xA;$w \rightarrow w=\left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}&#xA;  \sum_x \frac{\partial C_x}{\partial w}$, &#xA;there is an item $(1-\frac{\eta \lambda}{n})$on the weight, so weights will shrink every epoch. As weights were random chose at the beginning, it seems reasonable that the changes(the item $\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$) will be chaotic on first epoch. Maybe that is why the weight decay will dominate the first epoch.&#xA;And there is another explanation &lt;a href=&quot;https://datascience.stackexchange.com/questions/15016/speed-decay-proof-for-l2-regularization-and-non-normalizied-weight-initiation?answertab=active#comment16776_15016&quot;&gt;here&lt;/a&gt;, which may be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2)At each epoch, the item $(1-\frac{\eta \lambda}{n})$ will be multiplied $\frac{n}{m} $times, where $n$ is the number of training data, $m$ is the size of minibatch. So after each epoch, the weight will decay by a factor of$(1-\frac{\eta \lambda}{n})^\frac{n}{m}$. According to the definition of $e$:$e=\lim_{n\to \infty}(1+\frac{1}{n})^n$, lets change the format a little bit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(1-\frac{\eta \lambda}{n})^\frac{n}{m}=[(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}}]^{-\frac{\eta \lambda}{m}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, when $\eta \lambda \ll n$,  $(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}} \approx e$.Then we can come to the conclusion that provided $ηλ≪n$ the weights will decay by a factor of $exp(−ηλ/m)$ per epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3)I have been through this for two days and have no idea about it,but I think it should be related to the formula(85):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{eqnarray} C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln&#xA;(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2.&#xA;\end{eqnarray}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the weights were summed on last item.&lt;/p&gt;&#xA;" OwnerUserId="35496" LastActivityDate="2017-07-13T04:58:24.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20392" PostTypeId="1" AcceptedAnswerId="20401" CreationDate="2017-07-13T08:27:26.707" Score="0" ViewCount="804" Body="&lt;p&gt;I am reading the excellent &lt;em&gt;Hands-on Machine Learning with Scikit-Learn and TensorFlow&lt;/em&gt; and in chapter 10, the author says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;For the output layer, the softmax activation function is generally a&#xA;  good choice for classification tasks (when the classes are mutually&#xA;  exclusive). For regression tasks, you can simply use no activation&#xA;  function at all.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;All classification problems I can think of (binary classification, image classification, etc) generate mutually exclusive classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone give me a few examples of non-mutual exclusive classification problems?&lt;/p&gt;&#xA;" OwnerUserId="34036" LastEditorUserId="29169" LastEditDate="2019-06-06T08:05:14.550" LastActivityDate="2020-04-27T16:17:47.503" Title="Non-mutal exclusive classification task examples" Tags="|classification|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20393" PostTypeId="1" AcceptedAnswerId="20395" CreationDate="2017-07-13T08:45:26.030" Score="1" ViewCount="40" Body="&lt;p&gt;A common question I face is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a stream of incoming data. Let's call it a vector of entries where each entry represents a value. As I'm getting this stream of entries that is getting added to the vector, I want to figure out if one of them is an outlier. So my question is two essentially:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How do you establish a good baseline as you consistently keep getting data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How do you determine if a value that is appendede to the vector is significantly different than the baseline?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35542" LastActivityDate="2017-07-13T10:10:38.333" Title="Determine when entry/series of entries are outliers" Tags="|time-series|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20394" PostTypeId="2" ParentId="20367" CreationDate="2017-07-13T09:25:18.033" Score="0" Body="&lt;p&gt;Cleaning data and bringing it into a proper format is very often necessary but I would call this step &lt;em&gt;data preparation&lt;/em&gt; rather than &lt;em&gt;feature engineering&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may totally make sense to apply normalization techniques as Felipe Cruz already explained.&#xA;However, neural networks basically can approximate any function (see &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap4.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;) and thus this is also not really necessary.&#xA;It may still help especially in terms of training time, though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A strength of neural networks comes from them learning the relevant features themselves.&#xA;You might develop a feature based on the combination of two or more properties of your input but a neural net with proper architecture would also be able to &quot;come up&quot; with this feature on its own if it sees enough samples for this during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will go with the example of text here but analogous steps could basically be taken for any kind of input.&#xA;Lets suppose we're training a sentiment classifier for tweets.&#xA;The data could be cleaned by applying for example a spelling correction algorithm. Another preparation step then would be the vectorization of the input. We could tokenize the tweets and based on a vocabulary transform each word into a &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot vector&lt;/a&gt;.&#xA;As for feature engineering, one important feature might be negation, i.e. the difference between &quot;happy&quot; and &quot;not happy&quot;. It would take great effort to &quot;manually&quot; solve this problem as there are many ways to express negation (and then there's also irony...).&#xA;Suppose now we use a recurrent neural network that reads in the tweets word by word to predict the class in the end.&#xA;In the optimal case the network would learn this feature (and others) itself based on examples in the training data where negation influences the outcome.&#xA;I.e. in a simple case if there was a negating word like &lt;em&gt;not&lt;/em&gt; in the sequence before reading the word &lt;em&gt;happy&lt;/em&gt; the output is altered accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then again, applying some domain knowledge to extract features could help the model to identify important patterns more easily and thus speed up learning (if the features you develop are really relevant and sufficiently represented in your training data).&#xA;But rather than focusing on feature engineering, it is the architecture of your net that you normally focus on to make the net learn the features itself.&#xA;In the end you only will know what works best by trying it out.&lt;/p&gt;&#xA;" OwnerUserId="25454" LastActivityDate="2017-07-13T09:25:18.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20395" PostTypeId="2" ParentId="20393" CreationDate="2017-07-13T10:10:38.333" Score="0" Body="&lt;p&gt;Look into six-sigma techniques: e.g.: &lt;a href=&quot;http://www.whatissixsigma.net/imr/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.whatissixsigma.net/imr/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short: create a moving range (moving average +/- 3 times moving std.deviation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points outside that range can be considered outliers and need inspection (not just discarding).&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-07-13T10:10:38.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20397" PostTypeId="1" AcceptedAnswerId="20399" CreationDate="2017-07-13T11:55:59.080" Score="5" ViewCount="16455" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NxvUB.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NxvUB.png&quot; alt=&quot;Distribution of the target to be learned&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to regress on this target, have tried multiple transformations to bring it to normal but its not helping, read some stuffs online but none of the suggestions have worked till now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attaching the residual histogram as well, somehow the residuals are normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BwiEw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BwiEw.png&quot; alt=&quot;Distribution of Residuals&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in Advance.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastActivityDate="2017-07-13T13:03:34.607" Title="How to model a Bimodal distribution of target variable" Tags="|machine-learning|predictive-modeling|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20399" PostTypeId="2" ParentId="20397" CreationDate="2017-07-13T12:53:34.630" Score="4" Body="&lt;p&gt;To my understanding you should be looking for something like a &lt;a href=&quot;https://en.wikipedia.org/wiki/Mixture_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian Mixture Model - GMM&lt;/a&gt; or a &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kernel Density Estimation - KDE&lt;/a&gt; model to fit to your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many implementations of these models and once you've fitted the GMM or KDE, you can generate new samples stemming from the same distribution or get a probability of whether a new sample comes from the same distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In python an example would be like this:(directly taken from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/neighbors/plot_kde_1d.html#sphx-glr-auto-examples-neighbors-plot-kde-1d-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from scipy.stats import norm&#xA;from sklearn.neighbors import KernelDensity&#xA;&#xA;# Plot a 1D density example&#xA;N = 100&#xA;np.random.seed(1)&#xA;X = np.concatenate((np.random.normal(0, 1, int(0.3 * N)),&#xA;                    np.random.normal(5, 1, int(0.7 * N))))[:, np.newaxis]&#xA;&#xA;X_plot = np.linspace(-5, 10, 1000)[:, np.newaxis]&#xA;&#xA;true_dens = (0.3 * norm(0, 1).pdf(X_plot[:, 0])&#xA;             + 0.7 * norm(5, 1).pdf(X_plot[:, 0]))&#xA;&#xA;fig, ax = plt.subplots()&#xA;ax.fill(X_plot[:, 0], true_dens, fc='black', alpha=0.2,&#xA;        label='input distribution')&#xA;&#xA;for kernel in ['epanechnikov', 'tophat', 'gaussian']:&#xA;    kde = KernelDensity(kernel=kernel, bandwidth=0.5).fit(X)&#xA;    log_dens = kde.score_samples(X_plot)&#xA;    ax.plot(X_plot[:, 0], np.exp(log_dens), '-',&#xA;            label=&quot;kernel = '{0}'&quot;.format(kernel))&#xA;&#xA;ax.text(6, 0.38, &quot;N={0} points&quot;.format(N))&#xA;&#xA;ax.legend(loc='upper left')&#xA;ax.plot(X[:, 0], -0.005 - 0.01 * np.random.random(X.shape[0]), '+k')&#xA;&#xA;ax.set_xlim(-4, 9)&#xA;ax.set_ylim(-0.02, 0.4)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WrQlO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WrQlO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end the &lt;code&gt;kde&lt;/code&gt; model, could be used for sampling new data points or predicting the probability of a new sample to have been generated from this distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should play around with different kernels in KDE models or number of base distributions in GMMs, along with other parameters to get optimal results for your data.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-07-13T13:03:34.607" LastActivityDate="2017-07-13T13:03:34.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20401" PostTypeId="2" ParentId="20392" CreationDate="2017-07-13T13:57:37.993" Score="3" Body="&lt;p&gt;For example, due to the complexity of the images in the ImageNet database. Algorithms will often use hundreds or thousands of output nodes to be capable of classifying a large array of different things. Researchers also relax the cost function and allow the $k$ highest outputs to be considered. If one of these is correct then the example is considered to have been correctly classified. Furthermore with the existence of such complex data, often times a certain object might be a subset of another object. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider the picture of a human being. If the algorithm detects a human with the highest activation at the output node, but also has high activations for hands, eyes, face. This is not wrong! The image does in fact contain these objects as well and them being detected by the model is encouraging. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-07-13T13:57:37.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20403" PostTypeId="1" AcceptedAnswerId="23738" CreationDate="2017-07-13T14:59:05.267" Score="1" ViewCount="342" Body="&lt;p&gt;I am using gensim library to find most similar words to some words that I have. Using 10000 data samples (short text mainly 1-2 sentences) to train, I get really bad results! Why this is so? Also by repeating train and test it get different results. Should not it be deterministic?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = gensim.models.Word2Vec(senforword2vec)&#xA;print(model.most_similar(positive=['bad'], topn=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[('amount', 0.9987443089485168)]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In another train and&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[('yet', 0.9984885454177856)]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33245" LastEditorUserId="29575" LastEditDate="2017-10-13T12:37:41.127" LastActivityDate="2017-10-13T12:37:41.127" Title="Word2vec using gensim" Tags="|text-mining|word2vec|similarity|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20404" PostTypeId="1" AcceptedAnswerId="20437" CreationDate="2017-07-13T15:00:27.443" Score="2" ViewCount="707" Body="&lt;p&gt;There is quite a &lt;a href=&quot;https://www.quora.com/How-is-a-convolutional-neural-network-able-to-learn-invariant-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;good explanation&lt;/a&gt; which fully comply with my vision. But seems it lacks one final step. As Jean states, moving an object &lt;em&gt;significantly&lt;/em&gt; in the input image will cause the change in which neuron is activated in the yellow layer (the one previous to the first fully connected layer). So that we see that the part of the network &lt;strong&gt;before FCL&lt;/strong&gt; is  position equivariant. Then  author says that &lt;strong&gt;because&lt;/strong&gt; the network detects an object at any location, the FCLs should have taken care of it.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does the equivariance property hold for the whole network, including trailing classifier? (I've read &lt;a href=&quot;https://datascience.stackexchange.com/questions/16060/difference-between-equivariant-to-translation-and-invariant-to-translation&quot;&gt;Difference between &amp;quot;equivariant to translation&amp;quot; and &amp;quot;invariant to translation&amp;quot;&lt;/a&gt; but not sure I apply it to this case correctly)&lt;/li&gt;&#xA;&lt;li&gt;Should a network detect an object in right-top corner if only trained with images with that object at left-bottom corner?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Tried  &lt;a href=&quot;http://scs.ryerson.ca/~aharley/vis/conv/flat.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this demo&lt;/a&gt; but it doesn't seem to address the second question)&lt;/p&gt;&#xA;" OwnerUserId="32989" LastEditorUserId="32989" LastEditDate="2017-07-14T15:29:48.097" LastActivityDate="2017-07-14T15:47:53.747" Title="What exactly means CNN is position equivariant" Tags="|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20405" PostTypeId="1" CreationDate="2017-07-13T16:13:53.097" Score="4" ViewCount="1409" Body="&lt;p&gt;I'm naively thinking it would be better to normalize L2 regularization to the number of elements in a tensor, but I don't see anyone doing that. What am I missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say in a multi layer fully connected NN:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x  = input            # vector length n_in&#xA;w1 = weights          # shape n_in x n_1&#xA;b_1 = bias            # shape n_1&#xA;h_1 = Relu(w1 * x + b_1)&#xA;&#xA;w2 = weights          # shape n_1 x n_2&#xA;b_2 = bias            # shape n_2&#xA;h_2 = Relu(w2 * h_1 + b_2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the L2 loss is normally defined as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L2 = lambda * ( sum((w1) ^2) + sum((w2) ^2) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the first layer is much bigger than the second layer, than the first term in above L2 definition is much bigger, effectively making the second term useless. I'm thinking this would be better:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L2 = lambda * ( sum((w1) ^2)/(n_in*n_1) + sum((w2) ^2)/(n_1*n_2) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would also make the hyper parameter lambda independent of the number of nodes you choose. However, I don't see anybody doing this so I suspect I'm missing something. What is it?&lt;/p&gt;&#xA;" OwnerUserId="35562" LastActivityDate="2017-12-11T20:12:41.510" Title="Shouldn't L2 regularization be normalized for the number of nodes in a layer?" Tags="|neural-network|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20406" PostTypeId="1" AcceptedAnswerId="20443" CreationDate="2017-07-13T16:45:04.317" Score="0" ViewCount="160" Body="&lt;p&gt;For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally transformation would allow to revert and reconstruct original data with some noise(or to save the relationship between time series values with no option to revert to original values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of: multiplying each value of series by some number, which will do shift and scale operation or I can compute some continuous function(like log), where argument is my time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, I believe there should be more advanced method. &#xA;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&#xA;After careful considerations, and answer of @D.W. below and trying to transform data, I found that I spend more time on this than on main problem I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initial problem was twofold: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) to send real data to home computer to play at home any time I want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) to send real data to home computer, transform it a bit, to share with people in the web, to get some advise on my main problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After thinking, stressing, askying, googling I decided to drop the idea to do anything. &#xA;Few good links to people who have the same problem and are intended to transfer, transform data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For problem 1) I believe stream cipher is a an option to read more about ( as it transforms your data to a random string). Result is bad, as if one will look at your working computer logs, you are busted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For problem 2) read &lt;a href=&quot;https://stats.stackexchange.com/questions/223494/what-is-meant-by-laplace-noise&quot;&gt;about laplace noise&lt;/a&gt;. Also, believe to be bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regards.&lt;/p&gt;&#xA;" OwnerUserId="35563" LastEditorUserId="35563" LastEditDate="2017-07-16T12:50:51.830" LastActivityDate="2017-07-16T12:50:51.830" Title="How to distort data in a clever way?" Tags="|data|" AnswerCount="1" CommentCount="5" ClosedDate="2017-07-15T10:11:14.333" ContentLicense="CC BY-SA 3.0" />
  <row Id="20407" PostTypeId="1" CreationDate="2017-07-13T17:04:25.490" Score="0" ViewCount="2422" Body="&lt;p&gt;This is a really simple neural network with backprop. If one had to apply &lt;em&gt;bayesian &quot;inferences&quot;&lt;/em&gt; to update the weights and biases, what would change in the code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Forward Propogation&#xA;hidden_layer_input1=np.dot(X,wh)&#xA;hidden_layer_input=hidden_layer_input1 + bh               # linear transformation&#xA;hiddenlayer_activations = sigmoid(hidden_layer_input)     # non-linear transformation&#xA;output_layer_input1=np.dot(hiddenlayer_activations,wout)&#xA;output_layer_input= output_layer_input1+ bout             # linear transformation&#xA;output = sigmoid(output_layer_input)&#xA;&#xA;#Backpropagation&#xA;E = y-output&#xA;slope_output_layer = derivatives_sigmoid(output)&#xA;slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)&#xA;derivative_output = E * slope_output_layer&#xA;Error_at_hidden_layer = derivative_output.dot(wout.T)&#xA;derivative_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer&#xA;wout += hiddenlayer_activations.T.dot(d_output) *lr&#xA;bout += np.sum(derivative_output, axis=0,keepdims=True) *lr&#xA;wh += X.T.dot(d_hiddenlayer) *lr                          # update weight&#xA;bh += np.sum(d_hiddenlayer, axis=0,keepdims=True) *lr     # update bias&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If my understanding is correct of using bayes method to derive the weights and biases, it would help in getting to those values faster, leaving behind unnecessary loops for weights which would not converge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, coding and observing it is the best bet to get a real perception.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would like an understanding.&lt;/p&gt;&#xA;" OwnerUserId="31456" LastActivityDate="2017-08-16T03:35:33.507" Title="Applying bayesian methods to a simple neural network" Tags="|python|neural-network|bayesian-networks|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20409" PostTypeId="1" AcceptedAnswerId="20431" CreationDate="2017-07-13T18:54:38.357" Score="0" ViewCount="7660" Body="&lt;p&gt;Suppose if I delete all the objects from current session:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rm(list=ls())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any way in base R or using a function from a package which lets me restore the deleted objects from the current session?&lt;/p&gt;&#xA;" OwnerUserId="33448" LastEditorUserId="710" LastEditDate="2017-07-13T22:18:38.223" LastActivityDate="2020-02-26T02:38:17.940" Title="How to restore deleted objects in R?" Tags="|r|data-mining|statistics|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20410" PostTypeId="1" CreationDate="2017-07-13T19:23:07.900" Score="1" ViewCount="73" Body="&lt;p&gt;I have a tab delimited data set with 23 columns.  The header for this file names the 23 different features in the dataset.  The column names are...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;file  yval    size    err a-value b-value c-value d-value e-value f-value intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  second&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This file does not open/parse properly in orange canvas.  The File widget shows this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pxHnY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pxHnY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if I remove any three characters from the columns row it works.  For example if I change second to sec like this...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;file  yval    size    err a-value b-value c-value d-value e-value f-value intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  &lt;strong&gt;sec&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Or change f-value to f-va like this...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;file  yval    size    err a-value b-value c-value d-value e-value &lt;strong&gt;f-va&lt;/strong&gt;    intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  second&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Orange canvas properly parses the file and I get this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WJg24.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WJg24.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has to be a bug.  Why would orange canvas have a character limit on the header of the file?  It makes no sense.  Is there a fix for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should have mentioned I'm on Ubuntu 16.04 using Orange version 3.5.0.dev0+feea5ef&lt;/p&gt;&#xA;" OwnerUserId="35570" LastEditorUserId="35570" LastEditDate="2017-07-14T11:37:35.390" LastActivityDate="2017-07-14T11:37:35.390" Title="Orange canvas will not properly parse/open a file with a long header (column) line" Tags="|orange|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20411" PostTypeId="2" ParentId="13727" CreationDate="2017-07-13T20:24:24.673" Score="0" Body="&lt;p&gt;&lt;code&gt;model.fit()&lt;/code&gt; accepts only two parameters. Remove the third one in the last line of your code.&lt;/p&gt;&#xA;" OwnerUserId="35573" LastEditorDisplayName="user380" LastEditDate="2017-07-14T14:05:37.603" LastActivityDate="2017-07-14T14:05:37.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20413" PostTypeId="1" AcceptedAnswerId="32755" CreationDate="2017-07-13T20:53:38.567" Score="8" ViewCount="7630" Body="&lt;p&gt;I paste below the Keras documentation on Recurrent layer&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model= Sequential()&#xA;model.add(LSTM(32, input_shape=(10, 64)))&#xA;&#xA;# now model.output_shape == (None, 32)&#xA;# note: `None` is the batch dimension.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does the first argument of LSTM 32 mean here? Is it creating 32 LSTM blocks (By block I mean consisting of input, forget and output gate)? Could you please explain the meaning of the first argument and how that contributes to the output dimension of the LSTM both when return sequences is True and when return sequences is False&lt;/p&gt;&#xA;" OwnerUserId="34017" LastActivityDate="2020-02-04T14:42:56.850" Title="Clarification on the Keras Recurrent Unit Cell" Tags="|python|deep-learning|nlp|keras|rnn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20414" PostTypeId="1" CreationDate="2017-07-13T21:59:28.730" Score="-1" ViewCount="3262" Body="&lt;p&gt;I have non numeric data such as (city, province, gender etc) and numeric data (transaction amount etc). I ran the K-means on the continuous variable and now i want to map the non numeric variables on to these clusters. How do I do that in R? &#xA;How do I tell R that these are my non numeric data and use it over clustering. &lt;/p&gt;&#xA;" OwnerUserId="35579" LastActivityDate="2017-07-16T16:44:14.307" Title="K-means with categorical data" Tags="|r|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20415" PostTypeId="1" AcceptedAnswerId="23023" CreationDate="2017-07-13T23:47:15.267" Score="8" ViewCount="13892" Body="&lt;p&gt;I am trying a simple example with sklearn decision tree. I am giving &quot;number,is_power2,is_even&quot; as features and the class is &quot;is_even&quot; (of course this is stupid) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import tree&#xA;&#xA;features =[[1,0,0] , [2,1,1] , [3,0,0] , [4,1,1] , [5,0,0] , [6,0,1] ,  [900,0,1] , [1001,0,0]] #val,pow2,even&#xA;labels =  ['o'     ,  'e'    , 'o'     ,  'e'    ,  'o'    ,  'e'    ,   'e'      ,  'o'] #is even&#xA;&#xA;clf = tree.DecisionTreeClassifier()&#xA;clf = clf.fit(features,labels)&#xA;&#xA;print (clf.predict([[203,0,0]]))&#xA;&#xA;import pydot&#xA;&#xA;import pydotplus&#xA;from IPython.display import Image  &#xA;dot_data = tree.export_graphviz(clf, out_file=None, &#xA;                     feature_names=['number','pow2','even'],  &#xA;                     class_names=['o','e'],  &#xA;                     filled=True, rounded=True,  &#xA;                     special_characters=True)  &#xA;graph = pydotplus.graph_from_dot_data(dot_data)  &#xA;# Image(graph.create_png())  &#xA;graph.write_pdf(&quot;1.pdf&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The decision tree correctly identifies even and odd numbers and the predictions are working properly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The decision tree is basically like this (in pdf)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     is_even&amp;lt;=0.5&#xA;        /\&#xA;       /  \&#xA;   label1  label2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem is this. The label1 is marked &quot;o&quot; and not &quot;e&quot;.&lt;/strong&gt; However if I put class_names in export function as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;class_names=['e','o']&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then, the result is correct. I thought the output should be independent of class_names order. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I doing something wrong, or does the class_names order matter. If the latter is true, what is the right order (for an arbitrary problem) &lt;/p&gt;&#xA;" OwnerUserId="35584" LastActivityDate="2020-09-10T13:54:18.477" Title="what should be the order of class names in sklearn tree export function (Beginner question on python sklearn)" Tags="|python|scikit-learn|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20416" PostTypeId="1" CreationDate="2017-07-14T00:30:08.840" Score="4" ViewCount="5511" Body="&lt;p&gt;I want to perform DBSCAN on my datapoints, but I don't have access to the data, I just have the pairwise distance of datapoints. Additionally, I have no idea about the number of clusters but I do want that each cluster contains at least 40 data points. Does DBSCAN work with these conditions? For instance, can I have something like this? Or is more information needed? I want to emphasize that I have computed the pairwise distance and this is not the result of Euclidean or some other method.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import DBSCAN&#xA;&#xA;db = DBSCAN(min_samples=40, metric=&quot;precomputed&quot;)&#xA;&#xA;y_db = db.fit_predict(my_pairwise_distance_matrix)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not sure what is &lt;code&gt;eps&lt;/code&gt; parameter of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;DBSCAN()&lt;/code&gt;&lt;/a&gt;. How should I set that? &lt;/p&gt;&#xA;" OwnerUserId="34324" LastEditorUserId="24000" LastEditDate="2017-07-14T08:35:34.490" LastActivityDate="2019-08-26T20:02:00.247" Title="How to use precomputed distance matrix and min_sample for DBSCAN clustering method?" Tags="|machine-learning|clustering|scikit-learn|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20418" PostTypeId="1" AcceptedAnswerId="20424" CreationDate="2017-07-14T06:05:29.897" Score="1" ViewCount="638" Body="&lt;p&gt;Most of the examples I have found online for LSTMs refer to &quot;random&quot; text &lt;a href=&quot;http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;generation&lt;/a&gt;. &#xA;One of the problems that I'm trying to solve is to generate a &quot;summary&quot; of many docs into 1 doc. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;News Article: Barack Obama life history (2K words)&lt;/li&gt;&#xA;&lt;li&gt;Wikipedia: Barack Obama  (10K words)&lt;/li&gt;&#xA;&lt;li&gt;Barack Obama biography (10K words)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Will generate an article about Obama in X words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is LSTM the right approach?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2017-07-14T11:08:17.337" Title="LSTM text generation" Tags="|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20419" PostTypeId="2" ParentId="18026" CreationDate="2017-07-14T06:44:43.380" Score="6" Body="&lt;p&gt;After having used MITIE for a few weeks, I feel like I at least have enough to answer my basic questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;(and 3.) All models need to be trained from scratch - there is no online method to add new samples to the model as they come in. This is unfortunate because MITIE takes at least 45 minutes to an hour to train on a ~20k-sized dataset.&lt;/li&gt;&#xA;&lt;li&gt;The datasets I used were &lt;a href=&quot;https://github.com/mesnilgr/is13#atis-data&quot; rel=&quot;noreferrer&quot;&gt;ATIS&lt;/a&gt;, &lt;a href=&quot;https://github.com/glample/tagger/tree/master/dataset&quot; rel=&quot;noreferrer&quot;&gt;CoNLL 2003&lt;/a&gt;, and &lt;a href=&quot;http://dbpedia.org/&quot; rel=&quot;noreferrer&quot;&gt;DBpedia&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I've found MITIE to be quite good as far as classification accuracy goes, although it takes a bit of work to prepare datasets for it.&lt;/p&gt;&#xA;" OwnerUserId="7922" LastActivityDate="2017-07-14T06:44:43.380" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20420" PostTypeId="2" ParentId="20416" CreationDate="2017-07-14T06:59:10.387" Score="2" Body="&lt;p&gt;DBSCAN does &lt;em&gt;not&lt;/em&gt; guarantee a minimum cluster size. There are known situations, c.f. Wikipedia, where a cluster &lt;em&gt;can&lt;/em&gt; have fewer than &quot;minPts&quot; points. Furthermore, it has the concept of noise: points that do not have enough neighbors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For epsilon, also see the Wikipedia article. As you don't specify the number of clusters, this parameter is what mostly controls how many clusters you get. Set it to 0, and everything will be noise. Set it to the maximum distance, and everything will be in one cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really read the article. It's about &lt;em&gt;density&lt;/em&gt;, not about cluster sizes.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-07-14T06:59:10.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20421" PostTypeId="1" AcceptedAnswerId="23717" CreationDate="2017-07-14T08:09:30.777" Score="-1" ViewCount="2175" Body="&lt;p&gt;I am working on a problem where my input feature vector is of dimension 2 and I have to predict labels corresponding to each input. The set of labels is quite huge, I have about potential 1500 labels for each input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I approach it now is that I make vector representation for labels as 1's and 0's, with 1 corresponding if that label is present in the input example. However, the size of the vector would be about 1500.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using Logistic Regression on this and it gave very bad results, with precision being 0.0006 and recall being 0.0011. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also used a deep learning approach, where I passed the input feature of dimension 2 through some embedding layers and then predict/classify labels with these complex features. Loss function I used for deep learning model is binary_crossentropy. But still, it does not perform better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please direct me to some resources that discuss the problem of multi-label classification with a very large number of labels.   &lt;/p&gt;&#xA;" OwnerUserId="33179" LastActivityDate="2017-10-12T16:07:45.893" Title="Multi Label Classification for a large number of labels" Tags="|deep-learning|logistic-regression|multilabel-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20422" PostTypeId="1" CreationDate="2017-07-14T09:02:14.973" Score="1" ViewCount="3452" Body="&lt;p&gt;I've read and heard about the mighty XGBoost which is one of the most famous models people are using today to solve Kaggle challenge. This makes me interest to develop my own intuition about the model so I decided to try XGBoost on my dataset, but unfortunately, I experienced a lot of dependency related problems when I tried to install XGBoost on mac :(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I did a bit more researches and found out another related library called LightGBM from Microsoft which they claim to achieve better result (or at least, equivalent) over XGBoost. Therefore, I decided to go with the LightGBM instead since I didn't encounter any problems during the library installation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expected that the model should outperform, or at least, perform similarly compared to other models on the same dataset. My dataset contains 14k of text documents and has 0 or 1 values which refer to positive and negative sentiment respectively as target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a result showing below, LightGBM with count vectorizer as input got the 6th place with an accuracy of 94.60%. I'm quite curious since the accuracy of LightGBM is not only lower than SVC but also lower than Extra Tree model (300 esitimators). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have enough foundation and solid understanding behind these models so I couldn't come up with an answer about these outcomes. Can someone please give me an idea or assumption according to the result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. All models was performed with 5-fold cross validation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model             score&#xA;--------------  -------&#xA;extraT           0.9528&#xA;svc              0.9514&#xA;sgd_elas         0.9481&#xA;extraT_tfidf     0.9476&#xA;svc_tfidf        0.9473&#xA;lightGBM         0.9460&#xA;sgd_elas_tfidf   0.9458&#xA;lightGBM_tfidf   0.9420&#xA;randomF          0.9409&#xA;randomF_tfidf    0.9345&#xA;mulNB            0.9307&#xA;berNB            0.9087&#xA;berNB_tfidf      0.9087&#xA;mulNB_tfidf      0.9036&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32343" LastEditorUserId="32343" LastEditDate="2017-07-14T09:08:08.287" LastActivityDate="2017-07-14T15:17:13.777" Title="Why doesn't LightGBM perform better than SVC (linear kernel)? [sentiment analysis]" Tags="|machine-learning|svm|xgboost|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20423" PostTypeId="1" AcceptedAnswerId="20429" CreationDate="2017-07-14T09:22:48.587" Score="-1" ViewCount="71" Body="&lt;p&gt;I want to ensemble　three or four regression model: like GBDT, XGBDT, SVM. I know there are votingC = VotingClassifier() for classifier. I want to know are there some methods or function for ensemble regression model. Now, I just know set the simple weight for each model. how about other good method for ensemble regression model by python. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="35597" LastActivityDate="2017-07-18T10:01:15.470" Title="python: Are there are some class like Voting classifier for three or four regression model" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="0" ClosedDate="2017-07-14T14:05:27.107" ContentLicense="CC BY-SA 3.0" />
  <row Id="20424" PostTypeId="2" ParentId="20418" CreationDate="2017-07-14T09:29:18.357" Score="1" Body="&lt;p&gt;Although you can generate text in this way - sampling from a RNN trained to predict next character or word - it will not be meaningful. At best it will be grammatically accurate (in terms of nouns, verbs, adjectives etc) but semantic nonsense. It would not be a summary, except by lucky accident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To generate summaries using LSTM directly, you would need to train a network with real examples of summary inputs and outputs. This would be similar task to machine translation, but much harder due to variability in size of the inputs. It is unlikely that you will find enough training data to test the idea fully, and it is not clear that such a direct approach &lt;em&gt;can&lt;/em&gt; yield acceptable results even with large amounts of training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact text summarisation is not a solved problem by any means. There are deep learning approaches, such as &lt;a href=&quot;https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Brain team's effort using TensorFlow&lt;/a&gt;, which you could study to get some sample code and a sense of state-of-the-art. This approach uses an &lt;em&gt;attention model&lt;/em&gt; to extract apparently informational content (i.e. content that would have low probability of appearing in some assumed &lt;em&gt;generic&lt;/em&gt; document, thus is assumed to be interesting due to standing out). It is possible to use a trained LSTM to build such an attention-based model - the intuition is that the parts of the document that the already-trained LSTM is least able to predict are likely to contain noteworthy information.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-14T11:08:17.337" LastActivityDate="2017-07-14T11:08:17.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20426" PostTypeId="2" ParentId="20405" CreationDate="2017-07-14T09:52:25.187" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;However, I don't see anybody doing this so I suspect I'm missing something. What is it?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are concentrating on how this extra loss term affects the absolute value of the loss function. This is not really relevant. Many forms of regularisation are not expressed as changes to loss function at all - framing L2 regularisation as part of the loss function is convenient because it allows us to re-use the existing weight adjustment logic. However, functionally it is almost identical to a weight decay term applied per batch independently of loss function, e.g. multiply all weights by 0.99 every batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use L2 regularisation loss, then you will almost certainly want to track a separate metric such as mean square error (i.e. your original loss function before considering regularisation), when comparing test results between different levels of regularisation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, there is no reason that you cannot have a different L2 weight param per layer, and many frameworks will support this. Sometimes this could result in better generalisation. It is not done much in practice because it adds yet more dimensions to search when optimising hyper-parameters to a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no reason expect better performance by aiming for roughly equal loss values from regularisation per layer. However, if that is your goal, then a naive scaling by inverse layer size $\frac{1}{N_{in} \times N_{out}}$ would not do it. That is because the weights in a trained network will already tend to scale due to the training targets so that the mean squared value is proportional to $\frac{1}{N_{in} + N_{out}}$. So your actual scaling factor to make the L2 regularisation loss per layer roughly the same would be $\frac{N_{in} + N_{out}}{N_{in} \times N_{out}}$. In code this might look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;L2 = lambda * (&#xA;  sum((w1) ^2)*(n_in+n_1)/(n_in*n_1) + sum((w2) ^2)*(n_1+n_2)/(n_1*n_2)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(and of course matching changes to loss gradients). Although I will stress again that I do not expect this change to result in better generalisation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-14T16:28:46.277" LastActivityDate="2017-07-14T16:28:46.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20427" PostTypeId="2" ParentId="11852" CreationDate="2017-07-14T09:56:40.993" Score="1" Body="&lt;p&gt;The full answer to your question is &lt;strong&gt;&lt;em&gt;yes&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;no&lt;/em&gt;&lt;/strong&gt;. Let me elaborate:&lt;/p&gt;&#xA;&lt;h3&gt;Yes, you can train such a model with SGD:&lt;/h3&gt;&#xA;&lt;p&gt;&lt;em&gt;Yes&lt;/em&gt;, you can train a model like you described. This is well known under the name &amp;quot;Autoencoder&amp;quot;. The goal of an autoencoder is to find a (usually smaller) representation of the input data, which can then be used to reconstruct the data. This looks as follows:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hRb6U.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hRb6U.png&quot; alt=&quot;Structure of an autoencoder&quot; /&gt;&lt;/a&gt;Image taken from &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/&quot; rel=&quot;nofollow noreferrer&quot;&gt;standford.edu&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;We try to make &lt;span class=&quot;math-container&quot;&gt;$\hat{x}_1$&lt;/span&gt; equal to &lt;span class=&quot;math-container&quot;&gt;$x_1$&lt;/span&gt;. In your code, you add the additional constraint that &lt;span class=&quot;math-container&quot;&gt;$W_1$&lt;/span&gt; (the weights from the input layer to the hidden layer) are equal to &lt;span class=&quot;math-container&quot;&gt;$W_2$&lt;/span&gt; (the weights from the hidden layer to the output layer).&#xA;The reconstruction error is usually either the squared error&#xA;or the cross-entropy.&#xA;Such an autoencoder can be trained with SGD, as you proposed.&lt;/p&gt;&#xA;&lt;p&gt;So, the answer is: yes, what you propose does work and is known as Autoencoder.&lt;/p&gt;&#xA;&lt;h3&gt;No, you can't train an RBM with SGD:&lt;/h3&gt;&#xA;&lt;p&gt;In the previous section, I said that training an RBM with SGD, as described in your question, works. This is not the full truth: this is &lt;strong&gt;&lt;em&gt;NOT&lt;/em&gt;&lt;/strong&gt; an RBM anymore!&lt;/p&gt;&#xA;&lt;p&gt;As described in the &lt;a href=&quot;https://datascience.stackexchange.com/a/14043/14906&quot;&gt;answer by Quittend&lt;/a&gt; a restricted Boltzmann machine models the probabilistic distribution. The goal of RBM training is &lt;em&gt;not&lt;/em&gt; to exactly reconstruct the input, but to learn the probability distribution of the data. The reconstruction is thus not exactly the same as the input, but is a sample from the same probability distribution.&lt;/p&gt;&#xA;&lt;p&gt;The goal is to assign a high probability to the input data we train on. The probability of an input vector &lt;span class=&quot;math-container&quot;&gt;$\mathbf{v}$&lt;/span&gt; is given by&#xA;&lt;span class=&quot;math-container&quot;&gt;$$p(\mathbf{v}) = \frac 1Z \sum_{\mathbf{h}} e^{-E(\mathbf{v}, \mathbf{h})}$$&lt;/span&gt;&#xA;where &lt;span class=&quot;math-container&quot;&gt;$E$&lt;/span&gt; is the energy function, and &lt;span class=&quot;math-container&quot;&gt;$Z$&lt;/span&gt; is the partition function (to normalize &lt;span class=&quot;math-container&quot;&gt;$p$&lt;/span&gt; such that it is a valid probability).&#xA;During training, we want to increase &lt;span class=&quot;math-container&quot;&gt;$p(\mathbf v)$&lt;/span&gt; for all &lt;span class=&quot;math-container&quot;&gt;$\mathbf v$&lt;/span&gt; in our training data, so we have to calculate the derivative&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial \log p(\mathbf v)}{\partial w_{ij}}$$&lt;/span&gt;&#xA;and change the weights &lt;span class=&quot;math-container&quot;&gt;$w_{ij}$&lt;/span&gt; accordingly.&lt;/p&gt;&#xA;&lt;p&gt;This is what we do with the contrastive divergence (CD-&lt;em&gt;k&lt;/em&gt;) algorithm,&#xA;and this is &lt;em&gt;not&lt;/em&gt; possible with gradient descent (SGD).&lt;/p&gt;&#xA;&lt;h3&gt;tl;dr:&lt;/h3&gt;&#xA;&lt;p&gt;Training an unsupervised neural network with SGD exists and is known as autoencoder. An RBM is a different concept, which models the probability distribution and does not strictly reconstruct the input. RBMs can not be trained with SGD.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-07-14T09:56:40.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20428" PostTypeId="2" ParentId="20237" CreationDate="2017-07-14T10:12:46.670" Score="20" Body="&lt;p&gt;You might want to interpret your coefficients. That is, to be able to say things like &quot;if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your coefficients to be interpretable, linear regression assumes a bunch of things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of these things is no multicollinearity. That is, your $X$ variables should not be correlated against each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another is &lt;a href=&quot;https://en.wikipedia.org/wiki/Homoscedasticity&quot; rel=&quot;noreferrer&quot;&gt;Homoscedasticity&lt;/a&gt;. The errors your model commits should have the same variance, i.e. you should ensure the linear regression does not make small errors for low values of $X$ and big errors for higher values of $X$. In other words, the difference between what you predict $\hat Y$ and the true values $Y$ should be constant. You can ensure that by making sure that $Y$ follows a Gaussian distribution. (The proof is highly mathematical.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your data, you may be able to make it Gaussian. Typical transformations are taking the inverse, the logarithm or square roots. Many others exist of course, it all depends on your data. You have to look at your data, and then do a histogram or run a &lt;a href=&quot;https://en.wikipedia.org/wiki/Normality_test&quot; rel=&quot;noreferrer&quot;&gt;normality test&lt;/a&gt;, such as the Shapiro-Wilk test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are all techniques to build an &lt;a href=&quot;https://en.wikipedia.org/wiki/Bias_of_an_estimator&quot; rel=&quot;noreferrer&quot;&gt;unbiased estimator&lt;/a&gt;. I don't think it has anything to do with convergence as others have said (sometimes you may also want to normalize your data, but that is a different topic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression#Assumptions&quot; rel=&quot;noreferrer&quot;&gt;linear regression assumptions&lt;/a&gt; is important if you want to either interpret the coefficients or if you want to use statistical tests in your model. Otherwise, forget about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying the logarithm or normalizing your data, is also important because linear regression optimization algorithms typically minimize $\|\hat y - y\|^2$, so if you have some big $y$ outliers, your estimator is going to be VERY concerned about minimizing those, since it is concerned about the squared error, not absolute error. Normalizing your data is important in those case and this is why scikit-learn has a &lt;code&gt;normalize&lt;/code&gt; option in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html&quot; rel=&quot;noreferrer&quot;&gt;LinearRegression&lt;/a&gt; constructor.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2017-07-14T10:12:46.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20429" PostTypeId="2" ParentId="20423" CreationDate="2017-07-14T10:21:51.297" Score="1" Body="&lt;p&gt;There are many ways in which you could create an ensemble from your base models. Some resources to take a look at are the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, I would point your attention towards &lt;a href=&quot;https://stats.stackexchange.com/questions/139042/ensemble-of-different-kinds-of-regressors-using-scikit-learn-or-any-other-pytho&quot;&gt;this question&lt;/a&gt;, that has a lot of answers and interesting ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Regarding implementation&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used in the past the &lt;a href=&quot;https://github.com/viisar/brew&quot; rel=&quot;nofollow noreferrer&quot;&gt;brew&lt;/a&gt; module that has extensive implementations of different stacking-blending techniques etc, while using different combination rules and diversity metrics. It is also compatible with sklearn models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another interesting project is &lt;a href=&quot;https://github.com/reiinakano/xcessiv&quot; rel=&quot;nofollow noreferrer&quot;&gt;xcessiv&lt;/a&gt;, which I haven't used myself but provides a helpful gui for managing your ensemble's details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, &lt;em&gt;regarding theoretical details&lt;/em&gt;, I would suggest you take a look into &lt;a href=&quot;http://www.leg.ufpr.br/~eder/Artigos/Wather/Moreira_2007.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this survey&lt;/a&gt; that focuses on ensembles in regression tasks.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-07-18T10:01:15.470" LastActivityDate="2017-07-18T10:01:15.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20430" PostTypeId="1" CreationDate="2017-07-14T10:46:25.583" Score="0" ViewCount="204" Body="&lt;p&gt;I noticed that after each time I execute the following lines of code, my results are different. Any idea why?&#xA;I think  that the main issue here is this line of code&#xA;But I dont understand why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;test_data['prediction']=sentiment_model.predict_proba(test_matrix)[:,0]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to add new column with .predict_prob()?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_data['prediction'] = sentiment_model.predict_proba(test_matrix)[:,0]&#xA;test_data['prediction_label'] = sentiment_model.predict(test_matrix) &#xA;test_data['prediction'] = test_data['prediction'].apply(lambda x: round(x,2)) &#xA;test_data.sort_values(by='prediction', ascending=False, inplace=True) &#xA;test_data[test_data['name'] == 'Britax Decathlon Convertible Car Seat, Tiffany']&#xA;test_data.head() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32500" LastEditorUserId="32500" LastEditDate="2017-07-20T07:22:32.340" LastActivityDate="2017-10-18T09:07:42.043" Title="Logistic regression prediction changed after executed couple of time" Tags="|python|scikit-learn|logistic-regression|pandas|probability|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20431" PostTypeId="2" ParentId="20409" CreationDate="2017-07-14T11:35:35.703" Score="2" Body="&lt;p&gt;The answer is unfortunately no. There is no handy ctrl-z method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A tip to avoid these situations: I suggest you always save either the 'environment' or, as I prefer to do, the scripts with the codes for the desired objects, and save them regularly. I never type any commands directly into a work space, but always in a script which I save. This is so I can always look back at which steps I took in my coding.&lt;br&gt; Personally, I save my scripts with date notations in the file name - so when I change a script I'll still have access to the old version of the scripts. It has happened quite a few times that I've changed a code and later realized that the old code was better. It's quite a hassle when you don't have the old file saved.&lt;/p&gt;&#xA;" OwnerUserId="35602" LastActivityDate="2017-07-14T11:35:35.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20432" PostTypeId="2" ParentId="20421" CreationDate="2017-07-14T12:27:43.640" Score="0" Body="&lt;p&gt;In this abstract version of your problem, you could as well be solving the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have two parameters, e.g. longitude, latitude on Earth, and want a single model to be able to tell what model car is closest to that location. I wouldn't expect good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you have to give us more detail to help you solve the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably you have to go back to the drawing board and partition your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. how are the results on 1-2 labels, or about 10?&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-07-14T12:27:43.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20433" PostTypeId="2" ParentId="20430" CreationDate="2017-07-14T13:22:21.947" Score="2" Body="&lt;p&gt;There is some randomness in the results from selecting/shuffling data that is used in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't want that, you could set a fixed random_state (seed) in your model.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-07-14T13:22:21.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20434" PostTypeId="2" ParentId="20407" CreationDate="2017-07-14T14:45:29.993" Score="2" Body="&lt;p&gt;Bayesian models specify priors to inform and constrain the models and get uncertainty estimation in form of a posterior distribution. Non-Bayesian Deep Learning computes a scalar value for weights and biases at each layer. Bayesian Deep Learning calculates a posterior distribution of weights and biases at each layer which better estimates uncertainty but increases computational cost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://edwardlib.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Edward&lt;/a&gt; is a Python package for Bayesian inference, including Deep Learning.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-14T14:45:29.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20435" PostTypeId="1" CreationDate="2017-07-14T15:16:40.437" Score="2" ViewCount="161" Body="&lt;p&gt;Greetings data scientists,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am dealing with a complex classification/prediction problem and I am finding very difficult to separate the classes at all.&#xA;Even after PCA, my data (over the two PCs), look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DjaI7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DjaI7.png&quot; alt=&quot;PC1 and PC2 plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given this dataset, what approach could be best to accurately predict the classes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Background: I am trying to classifying modelled problems via the best approach/algorithm to solve them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks,&lt;/p&gt;&#xA;" OwnerUserId="35605" LastActivityDate="2018-01-14T04:05:47.680" Title="What are possible approaches to deal with unseparable data even after PCA?" Tags="|classification|dataset|feature-selection|prediction|pca|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20436" PostTypeId="2" ParentId="20422" CreationDate="2017-07-14T15:17:13.777" Score="2" Body="&lt;p&gt;Xgboost usually gives a higher accuracy(this is just an observation, not a fact). But the trick with xgboost is people dont know how to best use it. The most common mistake made using xgboost is not letting it train enough. Xgboost is an algorithm with overfits more than the other algorithms(well overfit is not the right term, it should be xgboost has high variance than the others). Do not tune the other parameters until you have found a converging nrounds for appropriate learning rate(0.01 or 0.1 or 0.3 etc). If you see that the cross val accuracy is much much lower than train accuracy then tune gamma(the second order derivative) first, then other parameters. If the cross val and train accuracy have a significant but not a lot difference(lets say train acc goes from 90 to 99 and cross val is stuck at 85 after significant iterations) then try reducing the max depth of trees. If the cross val and train accuracy are running side by side then just let the algorithm converge for a particular eta and then tune the other parameters(all except gamma).&lt;/p&gt;&#xA;" OwnerUserId="31822" LastActivityDate="2017-07-14T15:17:13.777" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20437" PostTypeId="2" ParentId="20404" CreationDate="2017-07-14T15:41:07.310" Score="1" Body="&lt;p&gt;In your link, the author Jean states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Additionally, I believe that if a CNN is trained showing faces only at one corner, during the learning process, the fully-connected layer may become insensitive to faces in other corners.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I also believe this to be correct. The FCN does not in any way add or improve translation invariance. Instead it will treat all outputs - each individual pixel of the &quot;feature maps&quot; - of the last convolutional layer as entirely different features. It must be trained with enough examples in order to generalise well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the feature maps are not themselves simple, clean detectors of objects as the simplified explanation of CNNs might imply. In a deep network they can be very complex and respond to a wide range of stimuli. They will also respond somewhat fuzzily, so that e.g. an eye or the side of a head showing an ear can trigger multiple feature map pixels (to the feature map's kernels, the same object slightly translated will look like distorted version of the same feature, and will still match enough sub-components of the object to trigger a positive response). The last layer will not necessarily detect full objects, but significant chunks of objects, areas of important texture etc. So position can still be quite fluid, and the &quot;head detector one pixel off in last convolutional layer&quot; scenario is not particularly realistic - although it may affect relative strength/confidence of predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can still be a problem if you need your network to generalise. If you suspect that your training data might not be covering enough variations in position, orientation etc of images, then a common approach is data augmentation. As a reflected, rotated or cropped image of an object should usually be classified as the same object, then you can pre-process your training data using those transforms to make many random variations of the input images. Some deep learning frameworks will allow you to do this continuously, generating fresh images for each and every batch.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-14T15:47:53.747" LastActivityDate="2017-07-14T15:47:53.747" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20440" PostTypeId="2" ParentId="20435" CreationDate="2017-07-14T16:31:19.143" Score="2" Body="&lt;p&gt;Is your problem unsupervised? If not, the data does not need to be &quot;visually&quot; separable in two dimensions, you can use classifiers in multiple dimensions to create separations for you (linear or nonlinear, just depends on the classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your problem is unsupervised try other methods for clustering data such a K-Means or tree-based clustering (i.e. hierarchical). &lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-07-14T16:31:19.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20442" PostTypeId="1" AcceptedAnswerId="20490" CreationDate="2017-07-14T19:53:45.710" Score="4" ViewCount="8228" Body="&lt;p&gt;I am trying to train a recurrent neural network that I built in keras on timeseries data to predict number of sales for next 10 days. For this, I've created my dataset as -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var(t) -&amp;gt; var(t+1)&#xA;var(t+1) -&amp;gt; var(t+2)&#xA;var(t+2) -&amp;gt; var(t+3)&#xA;var(t+3) -&amp;gt; var(t+4) and so on &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I did Min-Max scaling on this data and the RNN code is as follows - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(LSTM(20, input_shape=(1, look_back),activation='tanh',bias_initializer='ones'))&#xA;model.add(Dense(1, activation='linear',bias_initializer='ones'))&#xA;opt=adam(lr=0.1)&#xA;model.compile(loss='mean_squared_error',optimizer=opt)&#xA;model.fit(xtrain, ytrain, epochs=100, batch_size=1, verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the plot I am getting is when I did predictions on xtrain (green = ytrain, blue =  ypred) - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/790ph.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/790ph.png&quot; alt=&quot;Observed vs expected signals&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rnn isn't learning anything at all. Its producing same results for each dataset. I've tried adding hidden layers, increasing number of neurons, changing parameters (learning rate, momentum), optimizers (sgd, adam, adagrad, rmsprop), lstm activation fxn (tanh, softsign). I got little fluctuations in some cases in the graph. But the output is mostly constant.&#xA;Also, I've only 200 datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please guide me what I am doing wrong here. What else I can try. Will small sized data not work using RNN at all ? If so, is there any other way to solve this problem (except ARIMA model) ?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT - Increased batch size to 100 and epochs to 1000. Received some better results. Also, I did mean normalization [(x-mean)/std_dev] instead of MinMax scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ONRfh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ONRfh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23138" LastEditorUserId="23138" LastEditDate="2017-07-14T20:26:24.040" LastActivityDate="2017-07-17T16:32:46.710" Title="Recurrent neural network producing same predictions" Tags="|python|neural-network|time-series|keras|rnn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20443" PostTypeId="2" ParentId="20406" CreationDate="2017-07-15T00:49:15.037" Score="3" Body="&lt;p&gt;The short answer is: don't do that; anything you come up with on your own is likely to leak information in subtle ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The longer answer is: talk to your data protection authority.  Only use a method if they consider it acceptable.  (Hint: it's very likely that the answer you will get back is &quot;no, don't do that&quot;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The technical details are: there is an entire body of work on how to anonymize or de-identify data, so it can be revealed in a way that can't be linked to the people whom the data is about.  However, there's a long history of attempts to do this that failed; where it was actually possible to re-identify the people, violating their privacy.  See the following research paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.utexas.edu/users/shmat/shmat_cacm10.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Myths and Fallacies of &quot;Personally Identifiable Information&quot;&lt;/a&gt;.  Arvind Narayanan and Vitaly Shmatikov. Communications of the ACM, 53:6, June 2010.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to consider a scheme to do what you want, you'd need to tell us a lot about the nature of the data, what it reflects, what the privacy interests are, how the data is distributed, what other external sources might be available that might enable re-identification, and many other details.  Since you haven't provided any of that information, your specific question isn't answerable in its current form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But ultimately, I hope I've convinced that what you're trying to do is a bad idea.  Don't endanger people's data by going &quot;cowboy&quot; with some homebrew scheme to obfuscate the data; instead, talk to your data protection authority about what their requirements are.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-07-15T00:49:15.037" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20445" PostTypeId="2" ParentId="20414" CreationDate="2017-07-15T10:16:44.860" Score="0" Body="&lt;p&gt;Not sure what you mean by “map the non numeric variables on to these clusters”. You made your clusters without those variables, therefore clusters are defined by the variables you used and any cluster assignment would be done ignoring additional variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you might want to look at instead is clustering methods based on distances between points (such as K-Medoids). You’d probably want to look for some way of defining the similarity or distance between two points in your data (e.g. perhaps you can compare the similarity between provinces according to their average income, or whatever is relevant for your purposes), then clustering based on that.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-07-15T10:16:44.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20451" PostTypeId="1" AcceptedAnswerId="21917" CreationDate="2017-07-15T20:40:16.150" Score="5" ViewCount="1268" Body="&lt;p&gt;I am trying to reproduce &lt;a href=&quot;https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf&quot; rel=&quot;noreferrer&quot;&gt;results&lt;/a&gt; obtained on the &lt;a href=&quot;https://nlp.stanford.edu/sentiment/&quot; rel=&quot;noreferrer&quot;&gt;Stanford Sentiment Analysis&lt;/a&gt; dataset, which contains sentiment annotations for syntactic components of sentences taken from Rotten Tomatoes reviews.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is partitioned into training, development, and test sets by sentence in the &lt;code&gt;datasetSentences.txt&lt;/code&gt; and &lt;code&gt;datasetSplit.txt&lt;/code&gt; files. Components within these sentences are mapped to sentiment annotations in the &lt;code&gt;dictionary.txt&lt;/code&gt; and &lt;code&gt;sentiment_labels.txt&lt;/code&gt; files. The README states that the sentence and phrase IDs are different, but I can't find any mapping between them, so I don't know how to split the sentiment annotations up into the partitions used in the experiment I am trying to reproduce.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course I would expect the mapping from phrases to sentences to be many-to-one (because the same phrase might appear in multiple sentences) but I'd still expect to be given an explicit mapping instead of having to compare substrings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anybody worked with this data set? Is there something I am overlooking?&lt;/p&gt;&#xA;" OwnerUserId="35650" LastActivityDate="2017-12-03T19:47:26.370" Title="How do you map phrase IDs to sentence IDs in the Stanford Sentiment Analysis dataset?" Tags="|dataset|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20452" PostTypeId="1" CreationDate="2017-07-16T02:05:15.960" Score="1" ViewCount="619" Body="&lt;p&gt;I have the following dataset that I was thinking of using RNN/LSTM for classifying the protocol. Data contains features from packet capture that has only two provided fields: OUI of the MAC address of the device and the inter-arrival time (IAT) of the packets for a specific protocol. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the labeled data for http and ntp from the point-of-view of a specific device type is provided below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# MAC, ConnectivityIAT  -&amp;gt; protocol     &#xA;ff:f1:f2, 10, -&amp;gt; http&#xA;ff:f1:f2, 20, -&amp;gt; http&#xA;ff:f1:f2, 30, -&amp;gt; http&#xA;ff:f1:f2, 0, -&amp;gt; http&#xA;&#xA;ff:f1:f2, 3, -&amp;gt; ntp&#xA;ff:f1:f2, 6, -&amp;gt; ntp&#xA;ff:f1:f2, 9, -&amp;gt; ntp&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The task is to predict the correct class when something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ff:ff1:f2, 10 -&amp;gt; ?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do I convert the categorical value of OUI to numerical entities &lt;/li&gt;&#xA;&lt;li&gt;How to approch the solution using a NN ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35587" LastActivityDate="2019-04-10T00:01:45.333" Title="Multiclass classification of timeseries data using NN" Tags="|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20453" PostTypeId="2" ParentId="16532" CreationDate="2017-07-16T04:26:31.143" Score="0" Body="&lt;p&gt;If you're trying to do dimensionality reduction, you should use Mahout- it is best in class, and the only open source project afaik to offer truly distributed versions of PCA / SVD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mahout.apache.org/docs/0.13.1-SNAPSHOT/algorithms/linear-algebra/d-spca.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mahout.apache.org/docs/0.13.1-SNAPSHOT/algorithms/linear-algebra/d-spca.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mahout.apache.org/docs/0.13.1-SNAPSHOT/algorithms/linear-algebra/d-ssvd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mahout.apache.org/docs/0.13.1-SNAPSHOT/algorithms/linear-algebra/d-ssvd.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mahout also has  DBSCAN implementation, WIP as part of a Google Summer of Code project, worth keeping an eye on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing Anony's &quot;Mahout is Bad&quot; remark (I agree Spark is) is relevant to the deprecated MapReduce version of Mahout (as the new version hasn't implemented K-Means out of the box yet, though there is a discussion on the mailing list about how to do this fairly easily).&lt;/p&gt;&#xA;" OwnerUserId="5247" LastActivityDate="2017-07-16T04:26:31.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20454" PostTypeId="2" ParentId="20166" CreationDate="2017-07-16T04:35:38.690" Score="1" Body="&lt;p&gt;PMC from Mahout here- we're in the middle of a site re-org at the moment, and things are... well they're a mess. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a link to something I think is more useful. A tutorial on Co-Occurance in Spark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mahout.apache.org/docs/latest/tutorials/cco-lastfm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mahout.apache.org/docs/latest/tutorials/cco-lastfm/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Re &quot;A Spark Library&quot; well, mahout IS the spark library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use Mahout (Scala only, sorry if you're a Python-phile, however the syntax, especially for Mahout is very pleasant), you either need to download mahout and run &lt;code&gt;./mahout spark-shell&lt;/code&gt; from the &lt;code&gt;bin/&lt;/code&gt; directory. Or if you like GUIs notebooks and Apache Zeppelin, check out this tutorial for setting up Mahout+Spark on Zeppelin &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://mahout.apache.org/docs/latest/tutorials/misc/mahout-in-zeppelin/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mahout.apache.org/docs/latest/tutorials/misc/mahout-in-zeppelin/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If you are compiling a Jar, you just add Mahout as a dependency.)&lt;/p&gt;&#xA;" OwnerUserId="5247" LastEditorUserId="5247" LastEditDate="2018-01-29T16:40:28.370" LastActivityDate="2018-01-29T16:40:28.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20455" PostTypeId="2" ParentId="14885" CreationDate="2017-07-16T04:47:39.857" Score="1" Body="&lt;p&gt;This error is common when the &lt;code&gt;SPARK_HOME&lt;/code&gt; environment variable is not set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the shell type &lt;code&gt;export SPARK_HOME=/path/to/your/spark&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="5247" LastActivityDate="2017-07-16T04:47:39.857" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20456" PostTypeId="2" ParentId="14187" CreationDate="2017-07-16T04:48:24.843" Score="5" Body="&lt;p&gt;In simplified words,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model Parameters are something that a model learns on its own. &#xA;For example, &#xA;1) Weights or Coefficients of independent variables in Linear regression model. &#xA;2) Weights or Coefficients of independent variables SVM. &#xA;3) Split points in Decision Tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model hyper-parameters are used to optimize the model performance.&#xA;For example,&#xA;1)Kernel and slack in SVM.&#xA;2)Value of K in KNN.&#xA;3)Depth of tree in Decision trees. &lt;/p&gt;&#xA;" OwnerUserId="35657" LastActivityDate="2017-07-16T04:48:24.843" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20457" PostTypeId="1" CreationDate="2017-07-16T09:15:02.327" Score="1" ViewCount="38" Body="&lt;p&gt;I have a problem called problem 2.19 from Mathematische statistiek by A. van der Vaart &lt;a href=&quot;https://staff.fnwi.uva.nl/b.j.k.kleijn/AsympStat-LecNotes2010.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://staff.fnwi.uva.nl/b.j.k.kleijn/AsympStat-LecNotes2010.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I state it, $P$ means the orthogonal projection onto $H$, $P_0$ same onto $H_0$ and $I-P$ same onto $H^{\bot}$. And $H_0$, $H$ are subspaces with $H_0\subseteq H\subseteq \mathbb{R}^n$. Now the problem is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Suppose that $Y\sim N_n(\mu,{\sigma}^2I)$, where $\mu$ and $\sigma^2$ are parameters that are known to belong a linear subspace $H$ and $(0,\infty)$, respectively. We wish to test $H_0: \mu\in H_0$ for a linear subspace $H_0\subseteq H$. As a test statistic we use $T=||P_0Y-PY||^2/||(I-P)Y||^2$. Show how an appropriate critical value can be obtained from a table of an $F$-distribution.&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a theorem saying if $X\sim N(0,\Sigma)$ then $|X|^2\sim \sum_i \lambda_i Z_i^2$ with $\lambda_i$ all the eigenvalues of $\Sigma$ and $Z_i\sim N(0,1)$. By looking at the proof it seems like it's also possible to make a version for general $\mu$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that we can make a basis of $H_0$ and by extending it in a right way we can get a basis of $H$. I also have an ugly calculation to get the distribution of $|P_0Y-PY|^2$ and $|(I-P)Y|^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't see how this statistic works. Does anyone have a comment or answer? Thanks in advance (it's written that I'd better not thank in the comments).&lt;/p&gt;&#xA;" OwnerUserId="35663" LastActivityDate="2017-07-16T09:15:02.327" Title="A statistic for testing if $\mu$ which is known to be in subspace $H$, is also in subspace $H_0\subseteq H$" Tags="|statistics|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20463" PostTypeId="2" ParentId="20414" CreationDate="2017-07-16T16:44:14.307" Score="0" Body="&lt;p&gt;Use the &lt;code&gt;table()&lt;/code&gt; function, like this, in principle:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nclust  &amp;lt;- 2&#xA;# Create a k-means model on my.data: my.km&#xA;my.km &amp;lt;- kmeans(scale(my.data), centers = nclust)&#xA;&#xA;# Compare k-means to actual categorical values&#xA;table(my.km$cluster , my.data$categval1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then YOU decide wether the cluster assignments make sense in this context/with this particular variable.  &lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-07-16T16:44:14.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20464" PostTypeId="2" ParentId="11220" CreationDate="2017-07-16T17:12:41.673" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://nlp.stanford.edu/sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Sentiment Analysis&lt;/a&gt; dataset is based on Rotten Tomatoes reviews, has parses and sentiment annotation down to the syntactic component level.&lt;/p&gt;&#xA;" OwnerUserId="35650" LastActivityDate="2017-07-16T17:12:41.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20465" PostTypeId="2" ParentId="11247" CreationDate="2017-07-16T17:18:15.190" Score="0" Body="&lt;p&gt;Two starting places:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do a Google Scholar search for &quot;sentiment analysis&quot; and read the papers from the past few years.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Work through Scikit-Learn's &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;text classification tutorial&lt;/a&gt;. Copy the 20 Newsgroups classification code there and modify it for your task. This will give you a baseline to work from.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35650" LastActivityDate="2017-07-16T17:18:15.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20466" PostTypeId="2" ParentId="13756" CreationDate="2017-07-16T21:04:57.627" Score="1" Body="&lt;p&gt;The case that you are describing is referred in the literature as sample selection bias [1]. This case is a part of the area of Transfer Learning/Domain Adaptation. The training set does not represent the real world data-set well, which means that there is a difference between the distributions of the training and test sets. Another term from the Domain Adaptation area that is referred to the same problem is the Covariate Shift. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;B. Zadrozny, “Learning and Evaluating Classifiers under Sample Selection Bias,” Proc. 21st Int’l Conf. Machine Learning, July 2004. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2017-07-16T21:04:57.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20467" PostTypeId="2" ParentId="13422" CreationDate="2017-07-16T22:23:13.167" Score="0" Body="&lt;p&gt;This could happen for several reasons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a disrcepancy between the distribution of the features of the two samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a disrcepancy between the distribution of the labels of the two samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another issue is the sample sizes. If one sample is much larger than the other one, and also there is a discrepancy between the distributions, this could affect the final performance of the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, according to the Simpson's Paradox there is a possibility that a trend appears in different groups of data, but disappears when these groups are combined. This could be a reason for observing a worse performance when you combine the data. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2017-07-16T22:23:13.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20468" PostTypeId="2" ParentId="20413" CreationDate="2017-07-17T00:43:06.703" Score="1" Body="&lt;p&gt;The value '32' in this case is the size of the cell state and the size of the hidden state being sent forward in the network. Please see my answer &lt;a href=&quot;https://datascience.stackexchange.com/questions/19196/forget-layer-in-a-recurrent-neural-network-rnn/19269#19269&quot;&gt;here&lt;/a&gt; for more information. &lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-07-17T00:43:06.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20469" PostTypeId="1" AcceptedAnswerId="20489" CreationDate="2017-07-17T00:54:22.920" Score="6" ViewCount="22389" Body="&lt;p&gt;I have &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-visualize-the-output-of-an-intermediate-layer&quot; rel=&quot;noreferrer&quot;&gt;read the docs here&lt;/a&gt; and I understand the general idea. I am able to visualize the weights of the intermediate layers. However, I'm having trouble visualize the activations. Here's what I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained my model and saved the weights in a file called &lt;code&gt;weights_file&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to this &lt;a href=&quot;https://github.com/julienr/ipynb_playground/blob/master/keras/convmnist/keras_cnn_mnist.ipynb&quot; rel=&quot;noreferrer&quot;&gt;jupyter notebook&lt;/a&gt;, I got the values of the weights. First I defined my model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def mlp_model(hid_dim=10):&#xA;&#xA;        model = Sequential()&#xA;        model.add(Dense(units=hid_dim, input_dim=X.shape[1], activation='relu'))&#xA;        model.add(Dense(Y.shape[1], activation='softmax'))&#xA;        model.load_weights(weights_file)&#xA;        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])&#xA;&#xA;        return model&#xA;&#xA;model_created = mlp_model(hid_dim=15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get the weights, I did this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W = model_created.layers[0].kernel.get_value(borrow=True)&#xA;W = np.squeeze(W)&#xA;print(&quot;W shape : &quot;, W.shape) #(153, 15)&#xA;&#xA;W_out = model_created.layers[1].kernel.get_value(borrow=True)&#xA;W_out = np.squeeze(W_out)&#xA;print('W_out shape : ', W_out.shape) #(15, 8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From there I could create Hinton diagrams &lt;a href=&quot;https://matplotlib.org/devdocs/gallery/specialty_plots/hinton_demo.html#sphx-glr-gallery-specialty-plots-hinton-demo-py&quot; rel=&quot;noreferrer&quot;&gt;using this&lt;/a&gt;. However, when I try to work with the activations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;get_first_output = theano.function([model_created.layers[0].input], [model_created.layers[1].output])&#xA;layer_out = get_first_output([X[0,:]])[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError: ('Bad input argument to theano function with name &quot;mlp1_visualize_weights.py:131&quot; at index 0 (0-based).  \nBacktrace when that variable is created:\n\n  File &quot;mlp1_visualize_weights.py&quot;, line 213, in &amp;lt;module&amp;gt;\n    mlp_repeat(X, Y, Xtest, Ytest, params_to_use, weights_file)\n  File &quot;mlp1_visualize_weights.py&quot;, line 125, in mlp_repeat\n    model_created = mlp_model(hid_dim=hid_val, lr=lrate, reg_val=reg, momentum=moment, nest=nestval, optimizer=optim)\n  File &quot;mlp1_visualize_weights.py&quot;, line 105, in mlp_model\n    model.add(Dense(units=hid_dim, input_dim=X.shape[1], kernel_initializer=\'he_uniform\', activation=\'relu\', W_regularizer=l2(reg_val), b_regularizer=l2(reg_val)))\n  File &quot;/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py&quot;, line 426, in add\n    dtype=layer.dtype, name=layer.name + \'_input\')\n  File &quot;/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 1392, in Input\n    input_tensor=tensor)\n  File &quot;/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 1303, in __init__\n    name=self.name)\n  File &quot;/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py&quot;, line 184, in placeholder\n    x = T.TensorType(dtype, broadcast)(name)\n', 'TensorType(float32, matrix) cannot store accurately value [array([ 0.        ,  0.2037037 ,  0.20138889,  0.21100917,  0.62962963,\n        0.6875    ,  0.61206897,  0.44660194,  0.31168831,  0.17391304,\n        0. ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to look at just one input example, and find the activation and the weights from just that input example. Essentially I'm trying to figure out which features of the data each hidden unit is picking up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain how to get the activations of intermediate layers in Keras?&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-07-17T17:06:48.033" LastActivityDate="2018-10-22T11:23:17.677" Title="Keras: visualizing the output of an intermediate layer" Tags="|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20470" PostTypeId="2" ParentId="14331" CreationDate="2017-07-17T02:01:32.150" Score="1" Body="&lt;p&gt;The book &quot;Data mining concepts and techniques&quot;(by Jiawei Han) in the chapter of clustering talks about Hopkins Statistic(value range=(0,1)):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) high score--&gt;uniform distribution--&gt;no cluster&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2)low score--&gt;not uniform distribution-/-&gt;(may be not)cluster. &lt;/p&gt;&#xA;" OwnerUserId="35685" LastActivityDate="2017-07-17T02:01:32.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20471" PostTypeId="1" AcceptedAnswerId="20494" CreationDate="2017-07-17T08:52:11.083" Score="1" ViewCount="137" Body="&lt;p&gt;I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main feautures can have quite varying values. Therefore, my focus has been on density based algorithms with quite some success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am unable to grasp the idea of cluster comparison in such algorithms since the notion of cluster centers cannot be properly defined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset constists of network flows and I split the dataset in subsets based on an identifier. After applying clustering on each subset I want to be able to compare the clusters that are created on each subset so that I can compare the subsets themselves in some context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would appreciate some help from data scientist gurus on how to approach the concept of cluster comparison  or cluster center in such algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks all!&lt;/p&gt;&#xA;" OwnerUserId="4668" LastActivityDate="2021-02-18T20:33:52.473" Title="Notion of cluster centers and cluster comparison in Density Based Algorithms" Tags="|python|outlier|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20472" PostTypeId="1" AcceptedAnswerId="20473" CreationDate="2017-07-17T09:24:28.500" Score="13" ViewCount="1703" Body="&lt;p&gt;I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed.&lt;/p&gt;&#xA;&lt;p&gt;I did some google searches before asking my question and found that the RNN may help me with my problem. But, all examples which I've found somehow have a &lt;em&gt;defined&lt;/em&gt; number of input and output nodes.&lt;/p&gt;&#xA;&lt;p&gt;So, I'm looking for a strategy, how to make it real or at least some examples, preferable in Keras or PyTorch.&lt;/p&gt;&#xA;&lt;p&gt;More details about my issue:&lt;/p&gt;&#xA;&lt;p&gt;I have two inputs lists, where the length of the first one is fixed and equals two, e.g.:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;in_1 = [2,2] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;but the length of the second list is flexible, the length can be from three to inf, e.g.:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;in_2 = [1,1,2,2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;in_2 = [1,1,1,2,2,2,3,3,3]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Also, input lists depend on each other. The first list shows the dimension of the output list. So if in_1 = [2,2], means the output must have a possibility to be reshaped to [2,2] form.&lt;/p&gt;&#xA;&lt;p&gt;Currently, I'm thinking to combine two input list into one:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;in = in_1 + in_2 = [2, 2, 1, 1, 2, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Moreover, the output has the same length as the &lt;strong&gt;in_2&lt;/strong&gt; list, e.g.:&lt;/p&gt;&#xA;&lt;p&gt;if input lists are:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;in_1 = [2, 2]&#xA;in_2 = [1, 1, 2, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Output should be:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;out = [1, 2, 1, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Any ideas are welcome!&lt;/p&gt;&#xA;" OwnerUserId="29767" LastEditorUserId="71781" LastEditDate="2021-05-03T16:52:08.340" LastActivityDate="2021-05-03T16:52:08.340" Title="An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs" Tags="|python|neural-network|keras|rnn|pytorch|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20473" PostTypeId="2" ParentId="20472" CreationDate="2017-07-17T09:46:20.350" Score="8" Body="&lt;p&gt;The answer may depend on the significance of the length of the input vector or how it originates.&lt;/p&gt;&#xA;&lt;p&gt;However, the simplest solution is usually to know the largest size input and use that as number of vectors. If the given input has lesser length, you can do the padding with zeros or appropriate symbols. So instead of having a vector &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 3]$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 2, 3]$&lt;/span&gt; you can have vectors &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 3, 0]$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 2, 3]$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The same can apply for output. If output expected is &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 1]$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$[1, 3, 4, 1]$&lt;/span&gt; you can treat the first output as &lt;span class=&quot;math-container&quot;&gt;$[1, 2, 1, 0]$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Isn't this just a hack?&lt;/p&gt;&#xA;&lt;p&gt;Typically, Neural networks does function approximation. Ideally, it represents vectors (matrices) as input and vectors (matrices) as output. This is why it is always desirable that size of your input vector be fixed.&lt;/p&gt;&#xA;" OwnerUserId="30536" LastEditorUserId="98307" LastEditDate="2020-08-09T08:47:24.210" LastActivityDate="2020-08-09T08:47:24.210" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20474" PostTypeId="1" CreationDate="2017-07-17T09:53:11.937" Score="3" ViewCount="2027" Body="&lt;p&gt;I was wondering if anyone has experience with time series prediction for data from multiple sources. So for instance, time series $a,b,..,z$ each have their own shape, some may be correlated with others. The ultimate goal is to have a model trained such that the value at time $t+1$ for any given data source can be predicted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally have two solutions that in theory could work, but was wondering if anyone knew of other frequently used methods.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Multi-task learning with LSTM &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use feature engineering to model properties of each time series source as features along with usual features and use these with LSTM&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35697" LastEditorUserId="29575" LastEditDate="2018-02-12T23:04:38.527" LastActivityDate="2024-01-11T18:05:00.593" Title="Multi-Source Time Series Data Prediction" Tags="|time-series|forecast|multitask-learning|lstm|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20476" PostTypeId="2" ParentId="8278" CreationDate="2017-07-17T11:30:21.867" Score="1" Body="&lt;p&gt;The problem of covariate shift ultimately results in datasets with different underlying mathematical structure. Now, &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Manifold_learning_algorithms&quot; rel=&quot;nofollow noreferrer&quot;&gt;Manifold Learning&lt;/a&gt; estimates a low dimensional representation of high-dimensional data thereby revealing the underlying structure. Often Manifold Learning techniques are not projections -- therefore, different and more powerful, than standard PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used Manifold Learning techniques (for eg: IsoMap, MDS, etc) to visualize (and, if possible, quantify) the &quot;(dis)similarity&quot; between train and test datasets. &lt;/p&gt;&#xA;" OwnerUserId="35701" LastActivityDate="2017-07-17T11:30:21.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20480" PostTypeId="2" ParentId="20474" CreationDate="2017-07-17T13:55:12.033" Score="0" Body="&lt;p&gt;I'm only aware (and using) a RNN which gets multiple time-series in it's first layer and then mixes those in the following layers. Let me know if you have a question on this approach.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-17T13:55:12.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20483" PostTypeId="2" ParentId="20472" CreationDate="2017-07-17T14:46:42.223" Score="3" Body="&lt;p&gt;Knowing that the first list is pretty much invariant (just describing a certain geometry) you could also try creating many different, specialized NN for every distinct &lt;em&gt;in_1&lt;/em&gt; configuration and use only &lt;em&gt;in_2&lt;/em&gt; for feeding the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;em&gt;in_1&lt;/em&gt; could drive different networks.i.e. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;in_1=[1,1]? --&amp;gt; NN #1 (n1) --&amp;gt; (o1)&#xA;in_1=[2,1]? --&amp;gt; NN #2 (n1,n2) --&amp;gt; (o1,o2)&#xA;in_1=[2,2]? --&amp;gt; NN #3 (n1,n2,n3,n4) -&amp;gt; (o1,o2,o3,o4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a first step you determine the configuration (i.e. creating a dict) and then train/feed the specialized networks accordingly. &lt;/p&gt;&#xA;" OwnerUserId="35714" LastActivityDate="2017-07-17T14:46:42.223" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20484" PostTypeId="2" ParentId="20474" CreationDate="2017-07-17T15:06:28.373" Score="0" Body="&lt;p&gt;You can certainly use an LSTM for this approach, as well as VAR, or potentially a more 'traditional' model (random forest). I have implemented multivariate LSTM for t+x prediction in Keras.  Feature engineering may definitely help improve results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also used VAR and was surprised at the effectiveness of this when compared against the LSTM.  There are packages in Python and R for VAR.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if your features are effectively describing a pattern that is not necessarily dependent on the time component, a more traditional method such as random forest could be used.  Probably not as effective as VAR or LSTM, but it's fast.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-07-17T15:06:28.373" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20485" PostTypeId="1" CreationDate="2017-07-17T15:12:22.487" Score="0" ViewCount="548" Body="&lt;p&gt;I am trying to build a model that can be used to identify the word sales whenever a group of text are passed through the model. Should I use Azure or Python for this? I have done the text preprocessing and extracted the n-gram features in Azure, what should I do next and which model should I use?&lt;/p&gt;&#xA;" OwnerUserId="35710" LastActivityDate="2017-07-17T16:53:14.690" Title="How can i build a machine learning model to identify a specific word?" Tags="|machine-learning|text-mining|" AnswerCount="1" CommentCount="2" ClosedDate="2017-07-17T23:08:29.993" ContentLicense="CC BY-SA 3.0" />
  <row Id="20486" PostTypeId="2" ParentId="20472" CreationDate="2017-07-17T15:39:24.917" Score="4" Body="&lt;p&gt;I think you might have misunderstood the fixed number of inputs for the RNN. This is the number of inputs &lt;em&gt;per timestep&lt;/em&gt;. All your examples have a fixed number of inputs per timestep: 1! You feed them one at a time to your neural network, finishing with a special &quot;end&quot; token (you could always have a second input for this). Teach it to give no output until it sees the end token, and then to output the components of the result one at a time, ending with a special end output token.&lt;/p&gt;&#xA;" OwnerUserId="35716" LastActivityDate="2017-07-17T15:39:24.917" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20487" PostTypeId="2" ParentId="20485" CreationDate="2017-07-17T15:49:08.270" Score="1" Body="&lt;p&gt;It is always best to approach a problem using the simplest possible tools. In your case there is no need to use a machine learning (AI) model to detect the word sale. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find the word sale within a long datastream, I would do the following: parse through the entire String while looking at segments of size 4 at a time, convert the substring to lowercase. Determine if this substring is &quot;sale&quot;. If it is save the index of the &quot;s&quot; into a list. Return the list which will include the index of each start of the word sale within your text.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public ArrayList&amp;lt;Integer&amp;gt; getIndexSale(String inputString){&#xA;   ArrayList&amp;lt;Integer&amp;gt; indexes = new ArrayList&amp;lt;&amp;gt;();&#xA;   String[] queue = new String[4];&#xA;&#xA;   queue[0] = 0;&#xA;   queue[1] = 0;&#xA;   queue[2] = 0;&#xA;   queue[3] = 0;&#xA;&#xA;   for(int i = 0; i&amp;lt;inputString.length(); i++){&#xA;      queue[0] = queue[1];&#xA;      queue[1] = queue[2];&#xA;      queue[2] = queue[3];&#xA;      queue[3] = inputString.charAt(i);&#xA;&#xA;      String temp = java.util.Arrays.toString(queue).toLowerCase();&#xA;&#xA;      if(temp.equals(&quot;sale&quot;))  indexes.add(i);&#xA;   }&#xA;&#xA;   return indexes;&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As mentioned by Dan Carter: if you do not care about the indices of the word &quot;sale&quot; within your string then it is simpler to use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public boolean getIndexSale(String inputString){&#xA;&#xA;   if(inputString.toLowerCase().contains(&quot;sale&quot;)) return true;&#xA;   return false;&#xA;} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-07-17T16:53:14.690" LastActivityDate="2017-07-17T16:53:14.690" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20489" PostTypeId="2" ParentId="20469" CreationDate="2017-07-17T16:12:41.240" Score="7" Body="&lt;p&gt;Consider this network&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(Convolution2D(32, 3, 3, input_shape=(1,28,28))) &#xA;convout1 = Activation('relu')&#xA;model.add(convout1)&#xA;convout2 = MaxPooling2D()&#xA;model.add(convout2)&#xA;&#xA;model.add(Flatten())&#xA;&#xA;model.add(Dense(128))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(10))&#xA;model.add(Activation('softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can visualize the activation using this function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def layer_to_visualize(layer):&#xA;    inputs = [K.learning_phase()] + model.inputs&#xA;&#xA;    _convout1_f = K.function(inputs, [layer.output])&#xA;    def convout1_f(X):&#xA;        # The [0] is to disable the training phase flag&#xA;        return _convout1_f([0] + [X])&#xA;&#xA;    convolutions = convout1_f(img_to_visualize)&#xA;    convolutions = np.squeeze(convolutions)&#xA;&#xA;    print ('Shape of conv:', convolutions.shape)&#xA;&#xA;    n = convolutions.shape[0]&#xA;    n = int(np.ceil(np.sqrt(n)))&#xA;&#xA;    # Visualization of each filter of the layer&#xA;    fig = plt.figure(figsize=(12,8))&#xA;    for i in range(len(convolutions)):&#xA;        ax = fig.add_subplot(n,n,i+1)&#xA;        ax.imshow(convolutions[i], cmap='gray')&#xA;&#xA;# Specify the layer to want to visualize&#xA;layer_to_visualize(convout1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want you can refer my &lt;a href=&quot;https://github.com/yashk2810/Visualization-of-Convolutional-Layers/blob/master/Visualizing%20Filters%20Python3%20Theano%20Backend.ipynb&quot; rel=&quot;noreferrer&quot;&gt;jupyter notebook&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastEditorUserId="35718" LastEditDate="2017-07-17T16:46:45.907" LastActivityDate="2017-07-17T16:46:45.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20490" PostTypeId="2" ParentId="20442" CreationDate="2017-07-17T16:32:46.710" Score="5" Body="&lt;p&gt;Try increasing your batch size. If your batch size is small, the gradients are a rough approximation of the true gradients.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastActivityDate="2017-07-17T16:32:46.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20491" PostTypeId="1" CreationDate="2017-07-17T17:10:00.743" Score="-1" ViewCount="1068" Body="&lt;p&gt;So I've this model that simulates an ecosystem and outputs its attributes, like its chemistry, temperature etc. There are lots of input parameters to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My job is to write a program to figure out the values of those parameters automatically using machine learning techniques. i.e to make a guess, run the simulation, then check the results against actual historically observed field data. If the results are very close to the field data, then the parameters are probably correct. If they are off, then I make some adjustment of the parameters and run again. Every parameter has a default value, and can be varied only by +\- 30% of its default value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There around 30 input parameters to the simulation. However, only 8-10 are candidates for estimation. The simulation takes around 5 minutes to run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a Parameter estimation problem? I know of few algorithms meant for parameter estimation, like MCMC &amp;amp; Simulated annealing. Are they suitable in this case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could easily come up with a naive implementation for varying the parameters' values. Could someone guide/suggest me an approach to come up with an efficient solution? &lt;/p&gt;&#xA;" OwnerUserId="35723" LastEditorUserId="836" LastEditDate="2017-07-17T18:25:15.703" LastActivityDate="2017-09-15T22:43:55.393" Title="Parameter estimation for a model with multiple input parameters" Tags="|machine-learning|parameter-estimation|monte-carlo|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20493" PostTypeId="1" AcceptedAnswerId="20498" CreationDate="2017-07-17T18:37:25.337" Score="1" ViewCount="5477" Body="&lt;p&gt;I'm trying to implement the &lt;strong&gt;One Hidden Layer Model&lt;/strong&gt; presented in &lt;a href=&quot;http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article using Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten, Activation&#xA;from keras import optimizers&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(100, input_dim=9216))&#xA;model.add(Activation('relu'))&#xA;model.add(Dense(30))&#xA;model.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy'])&#xA;&#xA;sgd = optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)&#xA;hist = model.fit(X_train, y_train, epochs=10, verbose=0, validation_split=0.2)&#xA;&#xA;y_pred = model.predict_classes(X_valid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X_train shape is (2140, 9216)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;y_train shape is (2140, 30)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;X_valid shape is (1783, 9216)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But &lt;code&gt;y_valid&lt;/code&gt; shape is (1783,). I'm trying to understand why I'm not getting a (1783, 30) output. Am I missing something?&lt;/p&gt;&#xA;" OwnerUserId="35729" LastActivityDate="2017-07-17T20:15:32.773" Title="Multidimensional regression in Keras" Tags="|python|neural-network|regression|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20494" PostTypeId="2" ParentId="20471" CreationDate="2017-07-17T18:51:26.980" Score="1" Body="&lt;p&gt;You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-07-17T18:51:26.980" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20495" PostTypeId="1" AcceptedAnswerId="21962" CreationDate="2017-07-17T18:56:00.993" Score="1" ViewCount="8185" Body="&lt;p&gt;I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find keywords arranged in a cluster. I tried to auto summarize using markovify library in python but the summary it created or the sentences did not make much sense. Any suitable suggestions are welcome. Thank you&lt;/p&gt;&#xA;" OwnerUserId="24011" LastActivityDate="2018-12-27T00:01:25.260" Title="finding themes from text documents" Tags="|python|nlp|text-mining|markov-process|automatic-summarization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20497" PostTypeId="2" ParentId="20084" CreationDate="2017-07-17T19:04:19.903" Score="1" Body="&lt;p&gt;I have tried markovify, a markov chain library in python for automatic text summarization for text documents. You can check that out for a simple unsupervised approach. For me it did not give satisfactory results as I was using abstracts from medical journals as my dataset, but if you try to use it on a different themed dataset, it may help you. &lt;/p&gt;&#xA;" OwnerUserId="24011" LastActivityDate="2017-07-17T19:04:19.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20498" PostTypeId="2" ParentId="20493" CreationDate="2017-07-17T20:15:32.773" Score="3" Body="&lt;p&gt;The problem is using &lt;code&gt;predict_classes&lt;/code&gt; in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.predict_classes(X_valid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this is designed to select the argmax (index of the maximum output) and choose it as the predicted class, for a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a regression problem, and just want the raw output from the network. So instead you should call:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_pred = model.predict(X_valid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-07-17T20:15:32.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20499" PostTypeId="1" AcceptedAnswerId="20523" CreationDate="2017-07-17T21:10:22.493" Score="1" ViewCount="2188" Body="&lt;p&gt;I am a little new at this - I am used to just querying data and not so much analysis of the data so any help would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some data that is trending month over month. Some months with an increase in volume and others with a decrease. The goal is to identify a &quot;spike&quot; in the volume. Basically, is the increase abnormally large. I was trying to figure out the best way to do this. I thought the best thing to do was take the standard deviation and anything outside of two deviations - BAM!, that's a spike. Then I thought about it some and I got concerned about what happens next month. That spike is still in that data, it happened, but doesn't it shift the standard deviation so that the next month may not show a &quot;spike&quot; when without the prior &quot;spike&quot; it could have? For continuous data maybe this is less an issue but my data is rolling, I only have the last 6 months of data. If there is a &quot;spike&quot; this month and every one is put on notice, and the level never changes, then in six months wouldn't it seem like there is no spike?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess my question is, how would you quantify a measure for a &quot;spike&quot; given this limitation?&lt;/p&gt;&#xA;" OwnerUserId="35738" LastActivityDate="2017-07-18T14:42:44.417" Title="Identfying spikes in data" Tags="|data-mining|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20500" PostTypeId="2" ParentId="20491" CreationDate="2017-07-17T21:29:17.120" Score="0" Body="&lt;p&gt;Strategies for evaluating models similar to yours (from Ecology) are discussed in this paper &lt;a href=&quot;http://jasss.soc.surrey.ac.uk/17/3/11.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Facilitating Parameter Estimation and Sensitivity Analysis of&#xA;Agent-Based Models: A Cookbook Using NetLogo and R&quot;&lt;/a&gt; by Thiele et al, 2014.&#xA;It is about, well, Agent-Based Models and R, but the theoretical aspects are generally applicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just by chance I have read this last weekend.  Found it easy to read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a fast but maybe suboptimal strategy, what about &lt;a href=&quot;https://en.wikipedia.org/wiki/Latin_hypercube_sampling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Latin Hypercube Sampling?&lt;/a&gt;, see sect 2.25.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have not tried it myself, though.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-07-17T21:29:17.120" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20502" PostTypeId="2" ParentId="19025" CreationDate="2017-07-17T21:47:06.280" Score="5" Body="&lt;p&gt;You're correct that the same values in T-SNE can be distributed across different points, the reason this happens is clear if you take a look at the algorithm that T-SNE runs across.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address your first concern about the points actually not being the same after the algorithm has been applied to the dataset. I will leave you with an exercise to verify it for yourself, consider a simple array $x_1 =&#xA; [0,1]$ and $x_2 = [0,1]$ and run actual algorithm against it and see for yourself that the resultant points are not actually identical.You can cross reference your answer against this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;import numpy as np&#xA; from sklearn.manifold import TSNE&#xA; m = TSNE(n_components=2, random_state=0)&#xA; m.fit_transform(np.array([[0,1],[0,1]]))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd also observe that changing the &lt;code&gt;random_state&lt;/code&gt; actually modifies the output co-ordinates of the model. There isn't any real correlation between the actual co-ordinates and their output. Since the first step of TSNE computes the conditional likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us now try to rationalize, using the algorithm the reason this happens, just using math, without any intuition, for the moment. Note, that $x_i$ and $x_j$ are both vectors in this situation. $p_{j | i} = \frac{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}{\sum_{k \neq i}{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}}$. Now, if we calculate the $p_{ij} = \frac{p_{i|j} + p_{j | i}}{2N}$, we can see that the value is 1. After applying KL divergence, we obtain the values stated above. Now, let us apply some intuition to this. $p_{ij}$ is, informally, the conditional probability that $x_i$ would pick $x_j$ as it's neighbor. This justifies the result, 1, for two reasons. The first, being that there is no other neighbor, so it must pick the only other vector in the list of co-ordinates. In addition, the points are identical and the chances that they are picked as the others neighbor &lt;em&gt;should&lt;/em&gt; be high, as we see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now coming to whether the absolute co-ordinates in $\mathbb{R}^2$ have any significance. They really don't. The randomness can redistribute the points wherever you want them to go. However, what is more interesting is the ratios of the distances between the points and these are relative and are relative even when we project it onto higher dimensions which is pretty interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the truth is, rather than looking at the two clusters, look at the distances between them, because that conveys more information than the co-ordinates themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this answered your question :)&lt;/p&gt;&#xA;" OwnerUserId="27114" LastEditorUserId="27114" LastEditDate="2017-07-17T22:22:38.893" LastActivityDate="2017-07-17T22:22:38.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20503" PostTypeId="2" ParentId="18572" CreationDate="2017-07-17T23:09:46.000" Score="0" Body="&lt;p&gt;A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. This assumption is weak and in many cases may not hold. For example, imagine that we have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find a lot of methods in the literature. Some of them are trying to tackle the difference in the marginal distributions p(X) of the train and test set, while others are trying to tackle the conditional distributions p(y|x). Some methods are taking into account both differences among the train and test set distributions. Almost all methods are trying to bring the two distributions closer, using weights on the train instances (importance sampling) so that the new weighted train set will be close to the test set. Other methods transform the feature spaces so that the distributions the train and test sets will be close enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information about this topic you can check the survey&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pan, Sinno Jialin, and Qiang Yang. &quot;A survey on transfer learning.&quot; IEEE Transactions on knowledge and data engineering 22, no. 10 (2010): 1345-1359.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2017-07-17T23:09:46.000" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20504" PostTypeId="1" AcceptedAnswerId="20519" CreationDate="2017-07-18T01:23:22.087" Score="1" ViewCount="430" Body="&lt;p&gt;I have a dataset that looks like,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;order datetime, customer id, product name, type of product, quantity sold&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want a model to forecast sales for each individual item.  I'm thinking of using one of the following but would like some advice, literature, or other options.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One time series forecast (ARIMA) for each product&lt;/li&gt;&#xA;&lt;li&gt;RNN&lt;/li&gt;&#xA;&lt;li&gt;HMM&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What's the best method to predict sales of each individual item into the future?&lt;/p&gt;&#xA;" OwnerUserId="776" LastActivityDate="2017-07-18T12:53:08.693" Title="How do I forecast sales data down to the individual item?" Tags="|time-series|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20505" PostTypeId="1" CreationDate="2017-07-18T01:36:05.170" Score="1" ViewCount="628" Body="&lt;p&gt;My question is what kind of machine learning models could be used in the case we need to predict something from a large pool of possibilities. For example in a kaggle competition( Instacart challenge ) , they ask to predict what products will a user reorder , based on prior orders. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case the spectrum of possible outputs is huge ( there are +40000 possible products ) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have studied some models like CNN and usually the output for this models, is a logits tensor holding the probabilities, the size of the tensor usually corresponds to the possibilities in the classification problem. That are generally small ( maximum something like a hundred in cifar-100 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in this case, seems to me that this is not a good approach, because of the number of possible items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this CNN models are appropriated in this kind of tasks? what other approaches could be used? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="35746" LastActivityDate="2017-10-16T17:28:25.850" Title="Models to deal with huge possible outputs" Tags="|neural-network|model-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20506" PostTypeId="1" AcceptedAnswerId="20512" CreationDate="2017-07-18T01:58:30.540" Score="2" ViewCount="9816" Body="&lt;p&gt;I have a general question regarding TensorFlow's saver function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The saver class allows us to save a session via:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;saver.save(sess, &quot;checkpoints.ckpt&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And allows us to restore the session:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;saver.restore(sess, tf.train.latest_checkpoint(&quot;checkpoints.ckpt&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Inside the TensorFlow documentation, there is an example code (with an added epoch loop, and restore):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Create a saver.&#xA;saver = tf.train.Saver(...variables...)&#xA;# Launch the graph and train, saving the model every 1,000 steps.&#xA;sess = tf.Session()&#xA;saver.restore(sess, tf.train.latest_checkpoint(&quot;checkpoints.ckpt&quot;))&#xA;for epoch in xrange(25):&#xA;    for step in xrange(1000000):&#xA;        sess.run(..training_op..)&#xA;        if step % 1000 == 0:&#xA;            # Append the step number to the checkpoint name:&#xA;            saver.save(sess, 'my-model', global_step=step)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is that if we stopped the training loop at &lt;code&gt;epoch=15&lt;/code&gt;, and execute again, then if we would start at &lt;code&gt;epoch=0&lt;/code&gt; again, but the model is trained up to &lt;code&gt;epoch=15&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to resume from &lt;code&gt;epoch=15&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="24119" LastEditDate="2017-07-18T05:28:50.560" LastActivityDate="2017-07-18T08:45:39.143" Title="TensorFlow - Resume training in middle of an epoch?" Tags="|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20507" PostTypeId="1" CreationDate="2017-07-18T02:24:57.473" Score="0" ViewCount="120" Body="&lt;p&gt;In more traditional statistical learning methods, such as logistic regression, the coefficients on the pre-selected features can be interpreted as increasing or decreasing the log-odds of success. For example, say we want to use logistic regression with a single predictor variable &lt;code&gt;studying_hours&lt;/code&gt; to predict whether a student will pass the exam. We can estimate the parameters in the model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0 + \beta_1$studying_hours&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Y = 1$ if the student passes the exam and &#xA;$Y = 0$ otherwise, and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;studying_hours is a variable with values 1, 2, ..., 100 hours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can say that a one-unit increase (a one hour increase) in the number of hours studying for the exam increases the log-odds of passing the exam by $\beta_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It stands to reason that we should be able to make a similar argument when the features are learned instead of pre-selected. But then we would need to understand exactly what the learned feature is. In my example above, instead of pre-selecting &lt;code&gt;studying_hours&lt;/code&gt; to be the feature we want to use to predict whether a student passes the course, we would learn some informative feature instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've heard of Hinton diagrams, and of visualizing the feature map for CNNs, but I don't think either of those techniques really works here. A Hinton diagram would just tell me the magnitude and sign of the weights. A feature map would not really be informative. I don't expect to understand what all of the features are, because the whole point is that the algorithm can do a better job of learning features than I can do designing features. But even so, at least some of the features should be human-intepretable, like a 'straight line detector' in a CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we interpret the features being learned by a multilayer perceptron for logistic regression? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not really sure what to tag here, so please, if there's an edit you would recommend, go ahead!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my attempt at starting an answer, but clarification/other opinions would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a case of binary classification, and start from the simplest possible case: one scalar input, one hidden unit with a relu activation, and one output unit. Assume the hidden unit has a bias of 0. We can express the function that describes this network as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0 + \beta_1 max\{wx, 0\}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\beta_0$ is the bias on the output unit&lt;/li&gt;&#xA;&lt;li&gt;$\beta_1$ is the weight on the output unit&lt;/li&gt;&#xA;&lt;li&gt;$w$ is the weight from the input to the hidden unit&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then we can expand that function to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0 + \beta_1wx$, if $|wx| &amp;gt; 0$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0$ otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means that if $|wx| &amp;gt; 0$, then the hidden unit is giving us a factor by which to multiply the slope of the line separating the two classes. We can say that a one-unit increase in $\beta_1 w$ implies a one-unit increase in the log-odds of success. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $|wx| &amp;lt;= 0$, then the best separating line between the two classes has 0 slope, regardless of the magnitude of the weight (I think this is related to the dying relu problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we expand this to include a bias on the output, we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0 + \beta_1(wx + b)$, if $|wx + b| &amp;gt; 0$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0$ otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Through arithmetic, that first part can be expressed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$log(Y) = \beta_0 + \beta_1b + \beta_1wx$, if $|wx + b| &amp;gt; 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which means to me that the weight $w$ from the input to the hidden unit tells us both how to modify the y-intercept (which is now $\beta_0 + \beta_1b$) and how to modify the slope. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that make sense?&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-07-19T00:51:19.233" LastActivityDate="2017-07-19T00:51:19.233" Title="Explaining what a neural network is learning" Tags="|machine-learning|neural-network|visualization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20510" PostTypeId="1" CreationDate="2017-07-18T06:40:34.683" Score="1" ViewCount="45" Body="&lt;p&gt;I am implementing a module which finds based on user interactions on an online portal to find which attributes of a certain product and what values for those products influence the buyer to choose the said Item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My employers have suggested me to use Genetic Algorithm to achieve this. I have started implementing and found even the initial population generation to be a bit ineffective. Before I go further with this I would like your thoughts whether 'Genetic Algorithm' is the right way or is there a better way to go about this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me each genome is a list of attributes and their corresponding values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg. genome 1: Color: Black, Material: wool, thread count:5&#xA;     genome 2: Color: white, Handwash: yes, Dry clean only: No&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its a mixture of different attributes and their values. Am I going about this the wrong way?  &lt;/p&gt;&#xA;" OwnerUserId="35751" LastActivityDate="2017-07-18T06:40:34.683" Title="How effective is Genetic Algorithm for finding Attribute-Value relationships" Tags="|data-mining|genetic-algorithms|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20511" PostTypeId="1" CreationDate="2017-07-18T06:56:52.160" Score="4" ViewCount="463" Body="&lt;p&gt;I have texts similar to the ones below, and I want to find semantic similarity between these texts and the intent.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) what are the different steps to resign ?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1) procedure to resign ?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1) what are the process do i have to follow to resign ?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1) what are the common schemes followed for resignation by any company !!&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) to whom i need to contact after i resign ?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3) i did resign today.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;4) if you misbehave again , i'll resign !!&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;4) your rude behavior made me to resign. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;5) different company have different policy on resignation !!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To tackle above problem i found &quot; Quoras Model to handle Duplicate Questions &quot; interesting . so i thought let's try this but with small changes. and changes is instead doing binary classification , let's do multi class classification. so i was trying to do multi class classification on a &quot;Quora Data Set&quot;. my intention was to classify each question with their duplicate questions into one class on a vector space using LSTM, CNN models. LSTM worked if sequence remain unchanged. but as you know questions structure can changes keeping intent same (or we can call them as duplicate questions.). so to handle such variation of question i tried with CNN. CNN worked for some small data set but for large data set it become very sensitive or overlapping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i am thinking &quot; can i leverage NLP with deep learning to find intent and semantic relation ?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have POS tags, NER, SRL, LST, entity type, relation type, etc features. How can i leverage these features of NLP in deep learning to achieve a state-of-the-art result?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a paper, &quot;&lt;a href=&quot;http://nlp.stanford.edu/pubs/emnlp2015_2_jiwei.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;When Are Tree Structures Necessary for Deep Learning of Representations?&lt;/a&gt;&quot;, that uses parse tree as input for recursive neural network model. Is there any similar work? Can any one give me data sets used in this paper?&lt;/p&gt;&#xA;" OwnerUserId="35698" LastEditorUserId="35698" LastEditDate="2017-07-19T07:15:19.800" LastActivityDate="2017-07-19T07:15:19.800" Title="how can i leverage NLP features like SRL, LSA, POS, NER, entity type , relation type with deep learning to find semantic similarity of texts?" Tags="|deep-learning|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20512" PostTypeId="2" ParentId="20506" CreationDate="2017-07-18T07:03:40.027" Score="4" Body="&lt;p&gt;The network doesn't store its training progress with respect to training data - this is not part of its state, because at any point you could decide to change what data set to feed it. You could maybe modify it so that it knew about the training data and progress, stored in some tensor somewhere, but that would be unusual. So, in order to do this, you will need to save and make use of additional data &lt;em&gt;outside&lt;/em&gt; of the TensorFlow framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably the simplest thing to do is add the epoch number to the filename. You are already adding the current step within the epoch, so just add in the epoch multiplied:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;saver.save(sess, 'my-model', global_step=epoch*1000000+step)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you load the file, you can parse the filename to discover what epoch and step you were on and use those as the start point for the &lt;code&gt;xrange&lt;/code&gt; functions. To make this easier to re-start from any given checkpoint, you could use &lt;code&gt;argparse&lt;/code&gt; to allow your script to take the name of the checkpoint file you want to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In brief, it might look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Near top of script&#xA;import argparse&#xA;import re&#xA;&#xA;# Before main logic&#xA;parser = argparse.ArgumentParser()&#xA;parser.add_argument('checkpoint')&#xA;args = parser.parse_args()&#xA;&#xA;start_epoch = 0&#xA;start_step = 0&#xA;if args.checkpoint:&#xA;    saver.restore(sess, tf.train.latest_checkpoint(args.checkpoint))&#xA;    found_num = re.search(r'\d+', args.checkpoint)&#xA;    if found_num:&#xA;        checkpoint_id = int(found_num.group(0))&#xA;        start_epoch = checkpoint_id // 1000000&#xA;        start_step = checkpoint_id % 1000000&#xA;&#xA;# Change to xrange:&#xA;for epoch in xrange(start_epoch, 25):&#xA;    for step in xrange(start_step, 1000000):&#xA;        sess.run(..training_op..) # etc&#xA;&#xA;    # At end of epoch loop, you need to re-set steps:&#xA;    start_step = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You may want to reduce the number of checkpoints you are creating - as it stands you would have 25,000 checkpoint files generated by your code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be use a single checkpoint file, and  to save and restore a Python pickle of a simple &lt;code&gt;dict&lt;/code&gt; containing the state at the time you made the checkpoint, with a similar name.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-07-18T08:45:39.143" LastActivityDate="2017-07-18T08:45:39.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20513" PostTypeId="2" ParentId="20505" CreationDate="2017-07-18T07:13:44.877" Score="0" Body="&lt;p&gt;As in your example, +40000 possible products or outputs are kind of very large for all types of algorithms. You probably need to be more innovative than using only one approach for that. e.g. you need to define a distance function for such outputs according to your problem. As it is clear, regular distance functions such as Euclidean, doesn't seem to make sense in the context of large vectors and your specific problem. But as a suggestion you may want to read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Structured_support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;Structured support vector machine&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random forest&lt;/a&gt;. Also you may find other probabilistic approaches interesting too, e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Hidden_Markov_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;HMM&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, we may not search for an absolute answer specially in real-world problems and datasets as in your defined problem. Understanding the data and the related features will help you in order to find the best solution.&lt;/p&gt;&#xA;" OwnerUserId="35656" LastActivityDate="2017-07-18T07:13:44.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20517" PostTypeId="2" ParentId="15032" CreationDate="2017-07-18T11:38:46.403" Score="3" Body="&lt;p&gt;Taking the diagonal of &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/tensordot&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf.tensordot&lt;/a&gt; also does what you want, if you set axis to e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[1], [1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have adapted Neil Slater's example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;# Arbitrarity, we'll use placeholders and allow batch size to vary,&#xA;# but fix vector dimensions.&#xA;# You can change this as you see fit&#xA;a = tf.placeholder(tf.float32, shape=(None, 3))&#xA;b = tf.placeholder(tf.float32, shape=(None, 3))&#xA;&#xA;c = tf.diag_part(tf.tensordot( a, b, axes=[[1],[1]]))&#xA;&#xA;with tf.Session() as session:&#xA;    print( c.eval(&#xA;        feed_dict={ a: [[1,2,3],[4,5,6]], b: [[2,3,4],[5,6,7]] }&#xA;    ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which now also gives:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 20.  92.]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This might be suboptimal for large matrices though (see discussion &lt;a href=&quot;https://stackoverflow.com/questions/26168363/elegant-expression-for-row-wise-dot-product-of-two-matrices&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="35764" LastEditorUserId="35764" LastEditDate="2017-07-18T12:02:46.770" LastActivityDate="2017-07-18T12:02:46.770" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20518" PostTypeId="2" ParentId="20507" CreationDate="2017-07-18T12:19:01.360" Score="1" Body="&lt;p&gt;That's a popular question, since many use cases require to understand what's going on in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest to do just some simulation with the resulting NN. Like in your example you could have a test-dataset and den modify the 'studying_hours' by +5%, +10%, +15%, ... and then plot the result over this modification factor. This way you can see by an simulation what the NN is thinking of the 'studying_hours' and how it influences the result.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-07-18T12:19:01.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20519" PostTypeId="2" ParentId="20504" CreationDate="2017-07-18T12:53:08.693" Score="2" Body="&lt;p&gt;Do you want to forecast sales by day over a series of days? That would be more like ARIMA. Do you want to forecast &quot;How many widgets will we sell in the next month&quot;? That would be more like a regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, if the retailer you're working with has a large assortment of products that changes over time (e.g. seasonal clothing) then you should also look into forecasting by product category and not an individual product. It will be much more accurate.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-18T12:53:08.693" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20520" PostTypeId="1" AcceptedAnswerId="20524" CreationDate="2017-07-18T14:07:08.507" Score="1" ViewCount="1334" Body="&lt;p&gt;I would like to use scikit-learn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;GridSearchCV()&lt;/code&gt;&lt;/a&gt; to do a grid search on custom parameters in a kernel I have specified. Specifically, the kernel is of the form&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SeqKernel(x, y, orig_kernel, cut_off, order)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, &lt;code&gt;orig_kernel&lt;/code&gt; is a kernel typically used in SVM learning (such as linear, polynomial, RBF, or sigmoid). I wish to perform a grid search over values of &lt;code&gt;cut_off&lt;/code&gt; and &lt;code&gt;order&lt;/code&gt;, with the additional caveat that only the pairs such that &lt;code&gt;order&lt;/code&gt; $\le$ &lt;code&gt;cut_off&lt;/code&gt; are considered. What would be the best way to implement this?&lt;/p&gt;&#xA;" OwnerUserId="35772" LastEditorUserId="24000" LastEditDate="2017-07-18T14:23:48.257" LastActivityDate="2017-07-18T15:29:43.183" Title="Using GridSearchCV for custom kernel SVM in scikit-learn" Tags="|python|scikit-learn|svm|cross-validation|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20521" PostTypeId="1" CreationDate="2017-07-18T14:18:19.850" Score="4" ViewCount="20882" Body="&lt;p&gt;I have data that looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 1495573445.162, 0, 0.021973, 0.012283, -0.995468, 1&#xA; 1495573445.172, 0, 0.021072, 0.013779, -0.994308, 1&#xA; 1495573445.182, 0, 0.020157, 0.015717, -0.995575, 1&#xA; 1495573445.192, 0, 0.017883, 0.012756, -0.993927, 1&#xA; 1495573445.202, 0, 0.021194, 0.012161, -0.994705, 1&#xA; 1495573445.212, 0, 0.019638, 0.013718, -0.994019, 1&#xA; 1495573445.222, 0, 0.019440, 0.010803, -0.994476, 1&#xA; 1495573445.232, 0, 0.018112, 0.010849, -0.993073, 1&#xA; 1495573445.242, 0, 0.020157, 0.011154, -0.994644, 1&#xA; 1495573445.252, 0, 0.020340, 0.010040, -0.995804, 1&#xA; 1495573445.262, 0, 0.017792, 0.009857, -0.996078, 1&#xA; 1495573445.272, 0, 0.020538, 0.010239, -0.994858, 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is accelerometer data where the data frame columns are labeled &lt;code&gt;&quot;Time stamp&quot;&lt;/code&gt;, &lt;code&gt;&quot;Time skipped&quot;&lt;/code&gt;, &lt;code&gt;&quot;x&quot;&lt;/code&gt;, &lt;code&gt;&quot;y&quot;&lt;/code&gt;, &lt;code&gt;&quot;z&quot;&lt;/code&gt;, and &lt;code&gt;&quot;label&quot;&lt;/code&gt; with the index set to &lt;code&gt;&quot;Time stamp&quot;&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sampling rate is around 100Hz. How should I create a sliding window in this case? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came up with this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sliding_window(data, window_size, step_size):&#xA;    data = pd.rolling_window(data, window_size)&#xA;    data = data[step_size - 1 :: step_size]&#xA;    print data&#xA;    return data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I doubt this is the correct answer, and I don't know what to set window_size and step_size given that I have a 100Hz sampling rate.&lt;/p&gt;&#xA;" OwnerUserId="33171" LastActivityDate="2017-07-18T18:07:45.763" Title="Sliding window time series data with Python Pandas data frame" Tags="|python|time-series|pandas|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20522" PostTypeId="1" CreationDate="2017-07-18T14:35:42.103" Score="0" ViewCount="3004" Body="&lt;p&gt;I'm working on a project and want to use decisions tree (because I have both catgorical and numerical values in my input and don't want to transform the categoricals variables) to predict an output varaible , but the problem is that I don't have only one output variable but 4 !&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/acISz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/acISz.png&quot; alt=&quot;here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;that picture is a look for my output dataset:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The first one will can be predict with a regression decision tree .&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the the rest if can be predict with a classification decision tree but with more than 2 classes so a multi-class.&lt;/p&gt;&#xA;&lt;p&gt;So I want to know what is the best solution to approach this problem:&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using 4 differents decisions trees one tree for each variables ?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or using 1 multi-output decision tree?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What are the avantages and inconvenients of each approach?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30464" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-09-17T09:00:16.470" Title="Muti-Output Decision tree with classification and regression in output" Tags="|machine-learning|classification|data-mining|regression|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20523" PostTypeId="2" ParentId="20499" CreationDate="2017-07-18T14:42:44.417" Score="1" Body="&lt;p&gt;Your intuition about the algorithm is about right and is a known technique from six-sigma.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you may expect from six-sigma, take 3 times deviation. You could also take the running average and deviation to compare the new value to if you expect trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to inspect the spike but can consider removing it, which will reduce variations again.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-07-18T14:42:44.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20524" PostTypeId="2" ParentId="20520" CreationDate="2017-07-18T15:29:43.183" Score="0" Body="&lt;p&gt;Look at the source code and modify it to meet your conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/ab93d65/sklearn/model_selection/_search.py#L685&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/blob/ab93d65/sklearn/model_selection/_search.py#L685&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-07-18T15:29:43.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20525" PostTypeId="1" AcceptedAnswerId="20558" CreationDate="2017-07-18T15:45:06.017" Score="6" ViewCount="1628" Body="&lt;p&gt;Recently I am dealing a classification problem with some algorithms, say logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I preprocess my data, I standardize all my features and then generate polynomial features based on them.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import PolynomialFeatures, StandardScaler&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# features is my entire features dataset, labels excluded &#xA;features = poly.fit_transform(features)&#xA;features = std.fit_transform(features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After finishing training my model, the accuracy is, say about 80%. Then I invert the two line of preprocessing code to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;features = std.fit_transform(features)&#xA;features = poly.fit_transform(features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have read &lt;a href=&quot;https://datascience.stackexchange.com/questions/9020/do-i-have-to-standardize-my-new-polynomial-features&quot;&gt;this post&lt;/a&gt; but it seems the answers is not strong enough to help me figure it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I standardize my data first or generate polynomials from original features first?&lt;/p&gt;&#xA;" OwnerUserId="26125" LastActivityDate="2017-07-19T18:01:57.517" Title="Should I standardize first or generate polynomials first?" Tags="|classification|logistic-regression|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20527" PostTypeId="2" ParentId="20521" CreationDate="2017-07-18T18:07:45.763" Score="4" Body="&lt;p&gt;You can use the built-in Pandas functions to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[&quot;Time stamp&quot;] = pd.to_datetime(df[&quot;Time stamp&quot;]) # Convert column type to be datetime&#xA;indexed_df = df.set_index([&quot;Time stamp&quot;])           # Create a datetime index&#xA;indexed_df.rolling(100)                             # Create rolling windows&#xA;indexed_df.rolling(100).mean()                      # Then apply functions to rolling windows&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code is not 100% correct because the datetime conversion is not correct but it should help you get started.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-18T18:07:45.763" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20528" PostTypeId="2" ParentId="20511" CreationDate="2017-07-18T18:23:11.893" Score="0" Body="&lt;p&gt;One promising algorithm is &lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Word Mover's Distance&quot;&lt;/a&gt;. It treats document similarity as an optimization problem between word embedding vectors. The goal is to minimize the cost of &quot;travel&quot; between sets of word vectors. Given the semantic encoding inherent in word vectors, it can model aspects of intention and meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word Mover's Distance (WMD) is relatively simple to implement. However, it is computationally expensive because of the number of comparisons it makes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;WMD completely ignores POS, NER, parse trees, and all other related NLP techniques. Those restrictions make it easier to apply, however, limits the properties of language it can model.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-18T18:23:11.893" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20529" PostTypeId="1" CreationDate="2017-07-18T18:36:17.073" Score="4" ViewCount="2240" Body="&lt;p&gt;I'm familiar with traditional reinforcement learning where the algorithm must choose a categorical action (e.g., best move in a game or the highest click-through-rate ad among a set of ads). Does reinforcement learning work for problems with continuous actions? Are there any easy to understand references that you recommend? I was hoping to be able to use reinforcement learning to assign credit limits, which are represented in dollars (i.e., continuous amounts).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I've come across &lt;a href=&quot;http://arxiv.org/pdf/1509.02971v2.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;this paper&quot;&gt;this paper&lt;/a&gt; from Google's DeepMind, which so far seems relevant. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="12515" LastEditDate="2017-07-18T19:47:55.987" LastActivityDate="2017-08-03T10:21:35.643" Title="Reinforcement learning for continuous (rather than discrete) actions" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="20533" PostTypeId="1" CreationDate="2017-07-19T01:09:35.337" Score="2" ViewCount="697" Body="&lt;h1&gt;What I want to do&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and &lt;strong&gt;&lt;em&gt;photorealistically&lt;/em&gt;&lt;/strong&gt; replace this Person, Person A (can be anyone, I also want this to work cross-gender), with the face/hair region of another pre-defined Person B. Then output the resulting image-sequence of Person B seemingly mimicking Person A's face pose and expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The project should preferably be available open source and on github for everyone to use and improve. It would also be nice if I could do it in Python 3.6.x.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What I have&lt;/h1&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A &lt;em&gt;very&lt;/em&gt; large dataset of images with Person B's face from all different angles and with all kinds of different facial expressions, aligned and centered (suitable for generative adversarial network training)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Ideas on how to do this&lt;/h1&gt;&#xA;&#xA;&lt;h2&gt;1. Database Approach&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Process&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Extract a multitude of landmarks of every image in the dataset of Person B and save them in a table with the reference to the image. &lt;/li&gt;&#xA;&lt;li&gt;Extract a multitude of landmarks for each image in the sequence of Person A and per-image select the closest matches in the table of Person B, then use these matching images as replacements.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Concerns&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I can see this approach resulting in a very long and resource intensive calculation with frames that just can not find a good match and may seem off or just plain fake.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;2. ML Approach&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Process&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use a BEGAN or WGAN process that will be trained similar to the famous male-&gt;female converters and then interpolate&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Concerns&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This technique will result in a very low resolution output. All in all from the results I have seen in different github projects, it's only photorealistic up to 125x125 pixels&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Why I'm writing this question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;To ask all of you what direction you would recommend me to take =) &lt;/p&gt;&#xA;" OwnerUserId="35795" LastEditorUserId="35795" LastEditDate="2017-07-19T01:32:49.930" LastActivityDate="2017-07-19T01:32:49.930" Title="2D Image-based Avatar Animation With Expressions / Image-To-Image Translation" Tags="|machine-learning|deep-learning|image-recognition|databases|distance|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20534" PostTypeId="2" ParentId="13216" CreationDate="2017-07-19T04:01:11.700" Score="2" Body="&lt;p&gt;Basically, this is selecting a sample from the true distribution which consists of the true class and some other noisy class labels. Then taking the softmax over it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is based on sampling words from true distribution and noise distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here the basic Idea is to train logistic regression classifier which can separate the samples obtained from true distribution and sample obtained from noise distribution. Remember When we are talking about the samples obtained from the true distribution we are talking about only one sample which is the true class obtained from the model distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I have explained about NCE loss and how it differ from the NCE loss . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.linkedin.com/pulse/heavy-softmax-use-nce-loss-shamane-siriwardhana&quot; rel=&quot;nofollow noreferrer&quot;&gt;Noise Contrastive Estimation : Solution for expensive Softmax .&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="24918" LastEditDate="2019-02-19T17:42:50.163" LastActivityDate="2019-02-19T17:42:50.163" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="20535" PostTypeId="1" AcceptedAnswerId="20542" CreationDate="2017-07-19T04:15:22.443" Score="51" ViewCount="78358" Body="&lt;p&gt;I've been reading Google's DeepMind Atari &lt;a href=&quot;http://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf&quot; rel=&quot;noreferrer&quot; title=&quot;RL&quot;&gt;paper&lt;/a&gt; and I'm trying to understand the concept of &quot;experience replay&quot;. Experience replay comes up in a lot of other reinforcement learning papers (particularly, the AlphaGo paper), so I want to understand how it works. Below are some excerpts.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;First, we used a biologically inspired mechanism termed experience replay that randomizes over the data, thereby removing correlations in the observation sequence and smoothing over changes in the data distribution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The paper then elaborates as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;While other stable methods exist for training neural networks in the reinforcement learning setting, such as neural fitted Q-iteration, these&#xA;  methods involve the repeated training of networks &lt;em&gt;de novo&lt;/em&gt; hundreds&#xA;  of iterations. Consequently, these methods, unlike our algorithm, are&#xA;  too inefficient to be used successfully with large neural networks. We&#xA;  parameterize an approximate value function &lt;span class=&quot;math-container&quot;&gt;$Q(s, a; \theta_i)$&lt;/span&gt; using the deep&#xA;  convolutional neural network shown in Fig. 1, in which &lt;span class=&quot;math-container&quot;&gt;$\theta_i$&lt;/span&gt; are the parameters (that is, weights) of the Q-network at iteration &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt;. To perform&#xA;  experience replay, we store the agent's experiences &lt;span class=&quot;math-container&quot;&gt;$e_t = (s_t, a_t, r_t, s_{t+1})$&lt;/span&gt; at each time-step &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt; in a data set &lt;span class=&quot;math-container&quot;&gt;$D_t = \{e_1, \dots, e_t \}$&lt;/span&gt;. During learning, we apply Q-learning updates, on samples (or mini-batches) of experience &lt;span class=&quot;math-container&quot;&gt;$(s, a, r, s') \sim U(D)$&lt;/span&gt;, drawn uniformly at random from the pool of stored samples. The Q-learning update at iteration &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; uses the following loss&#xA;  function: &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is experience replay, and what are its benefits, in laymen's terms?&lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="63366" LastEditDate="2018-11-29T03:00:37.877" LastActivityDate="2023-02-03T21:05:11.813" Title="What is &quot;experience replay&quot; and what are its benefits?" Tags="|reinforcement-learning|q-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20536" PostTypeId="1" CreationDate="2017-07-19T04:45:01.953" Score="2" ViewCount="4556" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LubPB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LubPB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;Below is text output obtained after ocr image to string of medical discharge summary report.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;XXXXXXXX  T D.0.A'. 20.05. 2017&#xA;13.0,? ; 20.05.2017&#xA;AGE / sax; 43 YEAR(S] / MALE&#xA;CODE: IP1’7- 14041] FHL33350709 D.0.D:22.05.2017&#xA;ROOM NO: 1309F&#xA;CONSULTANT: XXXXXXXXX [CARDIOLOGIST]&#xA;CO v CONSULTANT: XXXXXXXX [GEN . PHYSICIAN]&#xA;DIAGNOSIS : ACUTE CORONARY SYNDROME&#xA;UNSTABLE ANGINA&#xA;MILD CORONARY ARTERY DISEASE&#xA;PLAN : MEDICAL MANAGEMENT&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The output I obtained is not accurate enough as the input image&lt;/strong&gt;.I want to cleanse the text output and extract the values of the fields such as DOA,DOD,AGE/SEX,CODE,ROOM NO,CONSULTANT,DIAGNOSIS etc.&#xA;For ex , i need the extract the value of the field name &quot;consultant&quot; as &quot;XXXXXXXXX [CARDIOLOGIST]&quot; as output in excel format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Below is my attempt&lt;/strong&gt; :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pytesseract as pt&#xA;import re&#xA;import pandas as pd&#xA;import cv2&#xA;import numpy as np&#xA;from matplotlib import pyplot as plt&#xA;&#xA;from PIL import Image&#xA;&#xA;img1 = Image.open('/root/regex_try/DS_Sample.jpg')&#xA;&#xA;text = pt.image_to_string(img1)&#xA;&#xA;print(text)&#xA;&#xA;Fieldnames_dict = {'Hospital Name':['Hospital Name', 'Hospital_Name', &#xA;'hospital name'],&#xA;'Hospital Address':['Address', 'city', 'Hospital Address'],&#xA;'Patient Name' : ['Name', 'Patient_Name', 'Patient Name', 'patient name'],&#xA;'Address' : ['Patient Address', 'Address','Amy“'],&#xA;'Age' : ['Age', 'age'],&#xA;'Sex': ['Sex', 'sex', 'gender', 'Gender'],&#xA;'Doctor Name' : ['Doctor Name', 'doctor name', 'doctor', 'treating doctor', &#xA;'Consultants','CONSULTANT','Consultant Name'],&#xA;'Inpatient Number' :['Inpatient Number', 'IP Number'],&#xA;'Admission Date' : ['Date of Admission', 'Admitted Date', 'D.0.A'],&#xA;'Discharge Date' : ['Date of Discharge', 'Discharged Date', 'D.0.D'],&#xA;'Diagnosis' : ['Diagnosis', 'Final Diagnosis', 'Principal/Secondary &#xA; Diagnosis'],&#xA;'Treatment Given' : ['Treatment Given', 'On Examination', 'Examination'],&#xA;'Follow up' : ['Follow Up', 'Follow up', 'Review &amp;amp; Advise', 'Condition on &#xA; Discharge', 'Advise on Discharge'],&#xA;'Summary' : ['Summary', 'Past Treatment Given'],&#xA;'Bed No' : ['Bed Number', 'Bed / Ward Details'],&#xA;'SS Number' : ['SS Number', 'SS No.'],&#xA;'UHID Number' : ['UHID Number', 'UHID No'],&#xA;'MR Number' : ['MR No', 'MR Number','MRNO '],&#xA;}&#xA;&#xA;def preprocess_formtext(field_dict,text):&#xA;   values_dict=field_dict&#xA;   values_dict=values_dict.fromkeys(values_dict, '')&#xA; for i in field_dict.keys(): #reaching the keys of dict&#xA;    for x in field_dict[i]: #reaching every element in tuples&#xA;        q=0 &#xA;        p=[x.upper()+r&quot;:(.*) &quot;,x.upper()+r&quot; : (.*) &quot;,x.upper()+r&quot;: (.*)&quot;,x.upper()+r&quot;: (.*)&quot;]&#xA;        for j in p:&#xA;            #print(j)&#xA;            match = re.search(j, text)&#xA;            if match:&#xA;                result = match.group(1)&#xA;                q=1&#xA;                values_dict[i]=result&#xA;                break&#xA;            else:&#xA;                result = &quot;&quot;&#xA;        if q==1:&#xA;            break&#xA;return values_dict&#xA;&#xA;values_dict=preprocess_formtext(Fieldnames_dict,text)&#xA;&#xA;formdata_df=pd.DataFrame([values_dict])                                                        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29580" LastEditorUserId="29580" LastEditDate="2017-07-19T07:15:56.723" LastActivityDate="2018-11-12T06:15:38.350" Title="How to improve OCR (Scanning) results?&quot;" Tags="|python|regex|ocr|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20537" PostTypeId="2" ParentId="20511" CreationDate="2017-07-19T05:25:40.617" Score="1" Body="&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Represent words as word vectors. Then add extra dimensions to the word vectors. In these extra dimensions, include POS, NER, etc. features in a numeric form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer Version:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a word2vec/Glove model with each word represented by a 100 dimensional vector. Additionally, you have features like POS, NER, etc. for each word.&#xA;Instead of representing each word with just 100 dimensions, represent it with &lt;code&gt;100 + n&lt;/code&gt; dimensions and fill these &lt;code&gt;n&lt;/code&gt; slots with the nlp features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will give the vectors an additional meaning which you have defined manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Suggestions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could benefit from using other models like Doc2Vec/Sent2Vec which can represent a whole sentence as vector. With these vectors you could query similarity of sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So say you've trained it on your data, just query :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.most_similar_cosmul(SENT_42)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where SENT_42 represents &quot;to whom i need to contact after i resign ?&quot;,&#xA;you will get a list of sentences similar to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you use &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim's doc2vec&lt;/a&gt; model in python.&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-07-19T05:25:40.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20538" PostTypeId="2" ParentId="20522" CreationDate="2017-07-19T06:33:21.910" Score="1" Body="&lt;p&gt;I check and found this in &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html#multioutput-classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn &lt;/a&gt; documentation and I'm sure that is the answer I was looking for!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Multioutput classification support can be added to any classifier with MultiOutputClassifier. This strategy consists of fitting one classifier per target. This allows multiple target variable classifications. The purpose of this class is to extend estimators to be able to estimate a series of target functions (f1,f2,f3...,fn) that are trained on a single X predictor matrix to predict a series of reponses (y1,y2,y3...,yn).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="30464" LastActivityDate="2017-07-19T06:33:21.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20539" PostTypeId="2" ParentId="19805" CreationDate="2017-07-19T07:03:22.910" Score="1" Body="&lt;p&gt;Try out &lt;a href=&quot;https://www.floydhub.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Floydhub&lt;/a&gt;. They give 100 free GPU hours.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastActivityDate="2017-07-19T07:03:22.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20540" PostTypeId="2" ParentId="17036" CreationDate="2017-07-19T07:03:46.040" Score="0" Body="&lt;p&gt;I think here the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hamming_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hamming distance&lt;/a&gt; can also be considered. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In information theory, the &lt;em&gt;Hamming distance&lt;/em&gt; between two strings of&#xA;  equal length is the number of positions at which the corresponding&#xA;  symbols are different.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-07-19T07:03:46.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20541" PostTypeId="1" AcceptedAnswerId="20557" CreationDate="2017-07-19T07:16:19.960" Score="4" ViewCount="1195" Body="&lt;p&gt;When I was reading this &lt;a href=&quot;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper, &lt;em&gt;Fully Convolutional Networks for Semantic Segmentation&lt;/em&gt;, I found that they use an up-sampling layer to classify each pixel in to a class. I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How do you understand the mathematics behind the de-convolution operation? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why do we use an upsampling layer? Is it for extract more global context?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="29169" LastEditDate="2021-02-16T00:44:40.237" LastActivityDate="2021-02-16T00:44:40.237" Title="What is deconvolution operation used in Fully Convolutional Neural Networks?" Tags="|deep-learning|convolutional-neural-network|supervised-learning|convolution|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="20542" PostTypeId="2" ParentId="20535" CreationDate="2017-07-19T07:25:32.017" Score="55" Body="&lt;p&gt;The key part of the quoted text is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This means instead of running Q-learning on state/action pairs as they occur during simulation or actual experience, the system stores the data discovered for [state, action, reward, next_state] - typically in a large table. Note this does not store associated values - this is the raw data to feed into action-value calculations later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning phase is then logically separate from gaining experience, and based on taking random samples from this table. You still want to interleave the two processes - acting and learning - because improving the policy will lead to different behaviour that should explore actions closer to optimal ones, and you want to learn from those. However, you can split this how you like - e.g. take one step, learn from three random prior steps etc. The Q-Learning targets when using experience replay use the same targets as the online version, so there is no new formula for that. The loss formula given is also the one you would use for DQN without experience replay. The difference is only &lt;em&gt;which&lt;/em&gt; s, a, r, s', a' you feed into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In DQN, the DeepMind team also maintained two networks and switched which one was learning and which one feeding in current action-value estimates as &quot;bootstraps&quot;. This helped with stability of the algorithm when using a non-linear function approximator. That's what the bar stands for in ${\theta}^{\overline{\space}}_i$ - it denotes the alternate &lt;em&gt;frozen&lt;/em&gt; version of the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advantages of experience replay:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;More efficient use of previous experience, by learning with it multiple times. This is key when gaining real-world experience is costly, you can get full use of it. The Q-learning updates are incremental and do not converge quickly, so multiple passes with the same data is beneficial, especially when there is low variance in immediate outcomes (reward, next state) given the same state, action pair.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Better convergence behaviour when training a function approximator. Partly this is because the data is more like &lt;a href=&quot;https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables&quot; rel=&quot;noreferrer&quot;&gt;i.i.d.&lt;/a&gt; data assumed in most supervised learning convergence proofs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disadvantage of experience replay:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It is harder to use multi-step learning algorithms, such as Q($\lambda$), which can be tuned to give better learning curves by balancing between bias (due to bootstrapping) and variance (due to delays and randomness in long-term outcomes). Multi-step DQN with experience-replay DQN is one of the extensions explored in the paper &lt;a href=&quot;https://arxiv.org/abs/1710.02298&quot; rel=&quot;noreferrer&quot;&gt;Rainbow: Combining Improvements in Deep Reinforcement Learning&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The approach used in DQN is briefly &lt;a href=&quot;https://www.youtube.com/watch?v=UoPei5o4fps&amp;amp;index=6&amp;amp;list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT&quot; rel=&quot;noreferrer&quot;&gt;outlined by David Silver in parts of this video lecture&lt;/a&gt; (around 01:17:00, but worth seeing sections before it). I recommend watching the whole series, which is a graduate level course on reinforcement learning, if you have time.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-02-15T12:08:59.457" LastActivityDate="2018-02-15T12:08:59.457" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="20543" PostTypeId="1" CreationDate="2017-07-19T07:39:15.570" Score="-1" ViewCount="5500" Body="&lt;p&gt;Update 1:&#xA;I know that I can seperate animal,bird object annotations/images from imagenet. In fact I already filtered annotations/images that are my interested and downloaded them by URL. I searched for another dataset for the following reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have roughly around 30K animal 30K bird objects. I included children synsets of animal synset as animal and children synsets of bird synset as bird. However, after training 100K iteration, YOLO object detector gives me only 45mAP which is not desirable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for this low performance is has to do with imagenet annotations: &#xA;Image that belongs animal category only annotated animals and takes people as background. However, my dataset contains annotation of people in other images. Hence, this conflict is making hard for detector to learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XwEyJ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XwEyJ.jpg&quot; alt=&quot;imagenet annotation of animal(horse) &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am considering COCO dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Original Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to train a model that detects following objects:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;People  &lt;/li&gt;&#xA;&lt;li&gt;Animal&lt;/li&gt;&#xA;&lt;li&gt;Bird&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I already have enough images and annotations for people. &#xA;Could anybody kindly suggest a dataset for that purpose which contains any class of animal or bird? &lt;/p&gt;&#xA;" OwnerUserId="32535" LastEditorUserId="32535" LastEditDate="2017-07-20T01:34:31.110" LastActivityDate="2017-07-20T01:34:31.110" Title="Animal Detection dataset" Tags="|deep-learning|dataset|computer-vision|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20544" PostTypeId="2" ParentId="11467" CreationDate="2017-07-19T07:41:01.513" Score="1" Body="&lt;p&gt;In a way, OLS is a model to estimate the regression line based on training data. While, RSS is a parameter to know the accuracy of model for both testing and training data.&lt;/p&gt;&#xA;" OwnerUserId="35807" LastActivityDate="2017-07-19T07:41:01.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20545" PostTypeId="2" ParentId="20430" CreationDate="2017-07-19T08:58:38.883" Score="1" Body="&lt;p&gt;My guess - the order of labels entered as training set is different.&lt;br&gt;&#xA;From the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression.predict_proba&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt; -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The returned estimates for all classes are ordered by the label of classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So make sure you know which class prediction probability your slicing when using the [:, 0] in the first row. &lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-07-19T08:58:38.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20546" PostTypeId="1" CreationDate="2017-07-19T11:30:55.513" Score="5" ViewCount="12045" Body="&lt;p&gt;There are lots of examples of classifier using deep learning techniques with &lt;a href=&quot;http://www.cs.toronto.edu/~kriz/cifar.html&quot; rel=&quot;noreferrer&quot;&gt;CIFAR-10&lt;/a&gt; datasets. The way it works is that, train thousands of images of cat, dog, plane etc and then classify an image as dog, plane or cat. But I want to do the reverse thing. I want to train dog, cat, planes and it should output images. Here is my idea&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Group similar/clear images of cat as value 1, a less similar/blur images as 1.1 and so on. &#xA;Similarly, group similar images of dog as value 2 and less similar images as 2.1 and so on ...&#xA;Do the same for all types of images.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Generated dataset should look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input    output&#xA; 1      pixels(24*24) of a cat images(clear)&#xA; 1      pixels(24*24) of a cat images(clear)&#xA; .&#xA; .&#xA; 1.1      pixels(24*24) of another cat images(blur) &#xA; 1.1      pixels(24*24) of another cat images(blur) &#xA;  .&#xA;  .&#xA;  and so on&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Now train values of input and label of images as output. Input will be just 1 dimensional, may be I will think of some other data. Output layer will have 24*24 i.e 576 units or neurons.&lt;/li&gt;&#xA;&lt;li&gt;At the end of training, I want something like this, if I give a input for example 1.15 it should output a new image since we trained with values 1.1 and 1.2 but we didn't train with 1.11 or 1.115.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please give me some idea how can I do this? Any link to example or papers would be nice.&lt;/p&gt;&#xA;" OwnerUserId="35812" LastEditorUserId="35812" LastEditDate="2017-07-19T11:39:32.053" LastActivityDate="2017-07-20T05:36:36.870" Title="How to generate image using deep learning" Tags="|neural-network|deep-learning|regression|linear-regression|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20547" PostTypeId="1" CreationDate="2017-07-19T11:39:13.780" Score="0" ViewCount="57" Body="&lt;p&gt;I want to find keywords for a business idea. The problem is that I have a hard time summarizing the business idea to one to three terms that would describe it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a (machine learning) tool that takes a description as an input and outputs - intelligently - categories and/or explicit search terms that fit the business idea most likely?&lt;/p&gt;&#xA;" OwnerUserId="35814" LastActivityDate="2017-08-18T15:11:01.240" Title="Verifying business ideas based on keywords derived from a description of the business idea" Tags="|machine-learning|neural-network|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20548" PostTypeId="1" AcceptedAnswerId="20549" CreationDate="2017-07-19T12:13:18.233" Score="5" ViewCount="9454" Body="&lt;p&gt;I was going through a Udacity tutorial wherein a few data points were given and the exercise was to test which of the following models best fit the data: linear regression, decision tree, or SVM. Using &lt;code&gt;sklearn&lt;/code&gt;, I was able to determine that that SVM is the best fit followed by decision tree. I got a very distinct decision boundary when these two algorithms were applied:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/06jWu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/06jWu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any specific reason for the said shapes or does it just depend on the data sets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code was quite straightforward; just reading the CSV, separating the features and then applying the algorithms as shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.ensemble import GradientBoostingClassifier&#xA;from sklearn.svm import SVC&#xA;&#xA;import pandas&#xA;import numpy&#xA;&#xA;# Read the data&#xA;data = pandas.read_csv('data.csv')&#xA;&#xA;# Split the data into X and y&#xA;X = numpy.array(data[['x1', 'x2']])&#xA;y = numpy.array(data['y'])&#xA;&#xA;# import statements for the classification algorithms&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.ensemble import GradientBoostingClassifier&#xA;from sklearn.svm import SVC&#xA;&#xA;    # Logistic Regression Classifier&#xA;    classifier = LogisticRegression()&#xA;    classifier.fit(X,y)&#xA;&#xA;    # Decision Tree Classifier&#xA;    classifier = GradientBoostingClassifier()&#xA;    classifier.fit(X,y)&#xA;&#xA;    # Support Vector Machine Classifier&#xA;    classifier = SVC()&#xA;    classifier.fit(X,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35816" LastEditorUserId="24000" LastEditDate="2017-07-20T02:18:31.797" LastActivityDate="2019-12-25T16:58:51.193" Title="Why Decision Tree boundary forms a square shape and SVM a circular/oval one?" Tags="|machine-learning|classification|svm|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20549" PostTypeId="2" ParentId="20548" CreationDate="2017-07-19T12:28:11.287" Score="6" Body="&lt;p&gt;Shape of the SVM decision boundary depends on the kernel (similarity function) used. The &quot;standard&quot; version of SVM has linear decision boundary. The one displayed could be using Gaussian kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision boundary of a decision tree is determined by overlapping orthogonal half-planes (representing the result of each subsequent decision) and can end up as displayed on the pictures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See more here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://shapeofdata.wordpress.com/2013/07/02/decision-trees/&quot; rel=&quot;noreferrer&quot;&gt;https://shapeofdata.wordpress.com/2013/07/02/decision-trees/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM&quot; rel=&quot;noreferrer&quot;&gt;https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35817" LastActivityDate="2017-07-19T12:28:11.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20550" PostTypeId="2" ParentId="20546" CreationDate="2017-07-19T13:09:27.130" Score="1" Body="&lt;p&gt;Generative Adversarial Networks (GANs) can generate novel, related images to a given training set. A general introduction (with code) can be found &lt;a href=&quot;https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-19T13:09:27.130" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20551" PostTypeId="2" ParentId="20543" CreationDate="2017-07-19T13:14:08.697" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://www.image-net.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageNet&lt;/a&gt; is the gold standard for labeled image training data. It contains both &lt;a href=&quot;http://www.image-net.org/search?q=animal&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;animal&amp;quot;&lt;/a&gt; and &lt;a href=&quot;http://www.image-net.org/search?q=bird&quot; rel=&quot;nofollow noreferrer&quot;&gt;&amp;quot;bird&amp;quot;&lt;/a&gt; categories.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-07-19T13:14:08.697" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20553" PostTypeId="2" ParentId="20547" CreationDate="2017-07-19T15:07:26.147" Score="1" Body="&lt;p&gt;You need keyword extraction technics. Basically they try to find the most important words in the text. There are many types of such methods.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-07-19T15:07:26.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20554" PostTypeId="1" AcceptedAnswerId="20571" CreationDate="2017-07-19T15:36:51.797" Score="-1" ViewCount="1048" Body="&lt;p&gt;I have a bar chart that looks like this (truncated for space, orange label obscured for privacy):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5lxc8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5lxc8.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, it is ordered by US state.  The color distinction is for something called &quot;Entity&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the set ups for Marks, Columns, and Rows, and the legend:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2uNyd.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2uNyd.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to do is sort the order in which these appear, not by state, but rather by the volume measure for the blue Entity (&quot;Industry&quot;).  So, in the example above, the states would be re-ordered to NY, NJ, NV, NM.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do this in Tableau? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2862" LastActivityDate="2017-07-20T11:32:39.283" Title="Sorting bar chart in Tableau by one category of bars" Tags="|visualization|tableau|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20555" PostTypeId="1" CreationDate="2017-07-19T16:06:48.320" Score="1" ViewCount="794" Body="&lt;p&gt;I have a dataset that would be useful for training future machine learning models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it contains sensitive information (personal identifiers like names and addresses) that make it difficult to store or retain longterm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if there are transformations that could be applied that would remove the sensitive information, while retaining it's usefulness?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is specifically useful for training NLP related models, but I'm also interested in a general solution to this for general machine learning tasks if one exists.&lt;/p&gt;&#xA;" OwnerUserId="35829" LastActivityDate="2017-07-19T18:56:21.077" Title="Removing sensitive information from a dataset while retaining its usefulness for machine learning tasks?" Tags="|machine-learning|dataset|data-cleaning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20557" PostTypeId="2" ParentId="20541" CreationDate="2017-07-19T16:57:51.980" Score="8" Body="&lt;p&gt;Upsampling layer is used to increase the resolution of the image. In segmentation, we first downsample the image to get the features and then upsample the image to generate the segments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For deconvolution operation we pad the image with zeroes and then do a convolution operation on that, hence it is upsampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For eg: - If after downsampling the images becomes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[1, 1]&#xA;[1, 1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then if we pad it with zeroes, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[0, 0, 0, 0, 0, 0]&#xA; [0, 0, 0, 0, 0, 0]&#xA; [0, 0, 1, 1, 0, 0]&#xA; [0, 0, 1, 1, 0, 0]&#xA; [0, 0, 0, 0, 0, 0]&#xA; [0, 0, 0, 0, 0, 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, if we do convolution on the above image with a 3 x 3 filter, we will get an image of shape (4, 4). Thus we upsampled an image of shape (2, 2) to (4, 4).&lt;/p&gt;&#xA;" OwnerUserId="35718" LastEditorUserId="11097" LastEditDate="2017-08-19T08:34:43.033" LastActivityDate="2017-08-19T08:34:43.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20558" PostTypeId="2" ParentId="20525" CreationDate="2017-07-19T18:01:57.517" Score="3" Body="&lt;p&gt;Depends on what are you using the standardization for and how the features relate to your problem, but I guess for most situations you’re better off standardizing them afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for some standardization in order to make optimization of some algorithm faster (e.g. neural networks rather than logistic regression), it might be a better idea to standardize afterwards, as otherwise it defies the purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing it in order to introduce some regularization, then standardizing them to 0-1 first will make the polynomials smaller overall and thus more likely to be given weights compared to the original features, which is probably not what you want.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-07-19T18:01:57.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20559" PostTypeId="2" ParentId="20555" CreationDate="2017-07-19T18:56:21.077" Score="1" Body="&lt;p&gt;There are lots of solutions to this problem and both of them would allow you to release the dataset. There are in general, two factors that are of importance, the first, whether the method is error prone or not and secondly, whether it is time consuming or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first, being a simpler and dependent solution. If you have a dictionary of names that you maintain, in your database, what you can do is search for whether the given word is a substring of a name (probably through a trie search) and then, check to see if it's a name in your database. If it is, replace it with some string, something even as easy as &quot;Name101&quot; would be easy to accomplish. The same can be done for geographical locations and other possibly sensitive data. This is a non-error prone and time saving method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you haven't done the former, there is a solution, however, it's a natural language processing technique called Named Entity Recognition (&lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Named-entity_recognition&lt;/a&gt;), where you can detect if a given word is a name, location or anything that could be worth hiding. However, please understand that this method is not guaranteed to succeed, it may hide most of your data, but it is not guaranteed to hide all of it. This is time saving but certainly error prone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last possible alternative is to actually comb through the data yourself and make sure that you've changed all of the names. This is an error prone and time consuming technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, you can use any of these to accomplish the tasks, but some of them are possible given your method of data collection, or your data pipeline.&lt;/p&gt;&#xA;" OwnerUserId="27114" LastActivityDate="2017-07-19T18:56:21.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20561" PostTypeId="2" ParentId="20308" CreationDate="2017-07-19T21:29:25.477" Score="20" Body="&lt;p&gt;&lt;a href=&quot;https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot_table.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pivot_table&lt;/code&gt;&lt;/a&gt; was made for this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.pivot_table(index='Date',columns='Groups',aggfunc=sum)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;results in&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;         data&#xA;Groups    one  two&#xA;Date&#xA;2017-1-1  3.0  NaN&#xA;2017-1-2  3.0  4.0&#xA;2017-1-3  NaN  5.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Personally I find this approach much easier to understand, and certainly more pythonic than a convoluted groupby operation.  Then if you want the format specified you can just tidy it up:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.fillna(0,inplace=True)&#xA;df.columns = df.columns.droplevel()&#xA;df.columns.name = None&#xA;df.reset_index(inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;which gives you&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;       Date  one  two&#xA;0  2017-1-1  3.0  0.0&#xA;1  2017-1-2  3.0  4.0&#xA;2  2017-1-3  0.0  5.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25426" LastEditorUserId="112944" LastEditDate="2021-11-01T00:14:57.663" LastActivityDate="2021-11-01T00:14:57.663" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="20563" PostTypeId="2" ParentId="8129" CreationDate="2017-07-20T00:39:50.633" Score="-1" Body="&lt;p&gt;Yes, there are plenty ways to do this, what you basically want is to interpolate between the two. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is current state-of-the-art for solving problems like yours: &lt;a href=&quot;https://github.com/carpedm20/BEGAN-tensorflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/carpedm20/BEGAN-tensorflow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also keep an eye out for my question here: &lt;a href=&quot;https://datascience.stackexchange.com/questions/20533/2d-image-based-avatar-animation-with-expressions-image-to-image-translation&quot;&gt;2D Image-based Avatar Animation With Expressions / Image-To-Image Translation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I end up using the ML approach, you could very well also use it with tiles instead of faces and be able to adjust the likeness. I will link to my project in my question. &lt;/p&gt;&#xA;" OwnerUserId="35795" LastActivityDate="2017-07-20T00:39:50.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20564" PostTypeId="2" ParentId="20546" CreationDate="2017-07-20T05:13:13.680" Score="4" Body="&lt;p&gt;DCGAN(Deep convolutioanl GAN) or WGAN can be used to generate images by training them on already existing images. GAN are very tricky to train. DCGAN and WGAN are the techniques that make it somewhat easier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A GAN consists of a Generator and a Discriminator. Generator's job is to generate images and Discriminator's job is to predict whether the image generated by the generator is fake or real. If the generator succeeds in fooling the discriminator, we can say that generator has succeeded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have generated MNIST images using &lt;a href=&quot;https://github.com/yashk2810/DCGAN-Keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;DCGAN&lt;/a&gt;, you can easily port the code to generate dogs and cats images.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastEditorUserId="35718" LastEditDate="2017-07-20T05:36:36.870" LastActivityDate="2017-07-20T05:36:36.870" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20565" PostTypeId="1" AcceptedAnswerId="37861" CreationDate="2017-07-20T06:03:15.540" Score="4" ViewCount="1023" Body="&lt;p&gt;I am trying to visualization CNN by the method in this paper &quot;&lt;a href=&quot;https://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Visualizing and Understanding Convolutional Networks&lt;/a&gt;&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to this tutorial &lt;a href=&quot;https://arxiv.org/pdf/1603.07285.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A guide to convolution arithmetic for deep&#xA;learning&lt;/a&gt; (P.19)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can always rewrite convolution as a matrix multiplication&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, $Y$ is a feature map &amp;amp;  $X$ is input &amp;amp; $W$ is a filter &amp;amp; $\otimes$ is convolution operation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{bmatrix}&#xA;y_{11} &amp;amp; y_{12} \\\\&#xA;y_{21} &amp;amp; y_{22}&#xA;\end{bmatrix}&#xA;=&#xA;\begin{bmatrix}&#xA;x_{11} &amp;amp; x_{12} &amp;amp; x_{13} &amp;amp; x_{14}\\\\&#xA;x_{21} &amp;amp; x_{22} &amp;amp; x_{23} &amp;amp; x_{24}\\\\&#xA;x_{31} &amp;amp; x_{32} &amp;amp; x_{33} &amp;amp; x_{34}\\\\&#xA;x_{41} &amp;amp; x_{42} &amp;amp; x_{43} &amp;amp; x_{44}&#xA;\end{bmatrix}&#xA;\otimes&#xA;\begin{bmatrix}&#xA;w_{11} &amp;amp; w_{12} &amp;amp; w_{13} \\\\&#xA;w_{21} &amp;amp; w_{22} &amp;amp; w_{23} \\\\&#xA;w_{31} &amp;amp; w_{32} &amp;amp; w_{33}&#xA;\end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the matrix multiplication form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{bmatrix}&#xA;y_{11} \\\\&#xA;y_{12} \\\\&#xA;y_{21} \\\\&#xA;y_{22}&#xA;\end{bmatrix}&#xA;=&#xA;C&#xA;\begin{bmatrix}&#xA;x_{11}\\\\&#xA;x_{12}\\\\&#xA;\vdots \\\\&#xA;x_{44}&#xA;\end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;C=&#xA;\left[\begin{smallmatrix}&#xA;w_{11} &amp;amp; w_{12} &amp;amp; w_{13} &amp;amp; 0 &amp;amp; w_{21} &amp;amp; w_{22} &amp;amp; w_{23} &amp;amp; 0 &amp;amp; w_{31} &amp;amp; w_{32} &amp;amp; w_{33} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0\\\\&#xA;0 &amp;amp; w_{11} &amp;amp; w_{12} &amp;amp; w_{13} &amp;amp; 0 &amp;amp; w_{21} &amp;amp; w_{22} &amp;amp; w_{23} &amp;amp; 0 &amp;amp; w_{31} &amp;amp; w_{32} &amp;amp; w_{33} &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0\\\\&#xA;0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; w_{11} &amp;amp; w_{12} &amp;amp; w_{13} &amp;amp; 0 &amp;amp; w_{21} &amp;amp; w_{22} &amp;amp; w_{23} &amp;amp; 0 &amp;amp; w_{31} &amp;amp; w_{32} &amp;amp; w_{33} &amp;amp; 0\\\\&#xA;0&amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; 0 &amp;amp; w_{11} &amp;amp; w_{12} &amp;amp; w_{13} &amp;amp; 0 &amp;amp; w_{21} &amp;amp; w_{22} &amp;amp; w_{23} &amp;amp; 0 &amp;amp; w_{31} &amp;amp; w_{32} &amp;amp; w_{33}\\\\&#xA;\end{smallmatrix}&#xA;\right]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we can visualize filter by multiplying $C^{-1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OK, my question is why we use $C^{T}$ instead of the pseudo inverse $(C^{T}C)^{-1}C^{T}$?&lt;/p&gt;&#xA;" OwnerUserId="35847" LastActivityDate="2018-09-06T02:28:44.193" Title="Why we use transposed filter as the deconvolution operation instead of the pseudo inverse of filter?" Tags="|deep-learning|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20566" PostTypeId="2" ParentId="20536" CreationDate="2017-07-20T06:44:06.690" Score="4" Body="&lt;p&gt;Build a dictionary of common words that frequently appear in these documents (e.g., MEDICAL, SEX, AGE, etc).  Then, for each word in the output from OCR, check whether it is similar to a word in your dictionary; if so, then replace it with the dictionary word.  &quot;Similar&quot; might be defined as &quot;edit distance &amp;lt;= 1&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, your sample output has the word &quot;sax&quot;.  If your dictionary contains the word &quot;SEX&quot;, &quot;sax&quot; would be detected as a misspelling of &quot;SEX&quot; (edit distance = 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This post-processing will help you detect some of the common words and help you identify the fields.  What it won't do is clean up errors in the OCR of codes, as those can be anything.  There's probably no good way to handle that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, do a Google search on how to use Tesseract.  There are some best practices that seem to improve its output (e.g., convert to greyscale TIFF format, deskew text, binarize, and &lt;a href=&quot;https://stackoverflow.com/q/9480013/781723&quot;&gt;more&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, some commercial OCR software is significantly better than Tesseract or any other free OCR.  e.g., Abbyy seems to be well-regarded.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-07-20T06:44:06.690" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20569" PostTypeId="2" ParentId="9222" CreationDate="2017-07-20T08:35:04.760" Score="1" Body="&lt;p&gt;Upsampling or deconvolution layer is used to increase the resolution of the image. In segmentation, we first downsample the image to get the features and then upsample the image to generate the segments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For deconvolution operation we pad the image with zeroes and then do a convolution operation on that, hence it is upsampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For eg: - If after downsampling the images becomes [[1, 1] [1, 1]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then if we pad it with zeroes, [[0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0] [0, 0, 1, 1, 0, 0] [0, 0, 1, 1, 0, 0] [0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if we do convolution on the above image with a 3 x 3 filter, we will get an image of shape (4, 4). Thus we upsampled an image of shape (2, 2) to (4, 4).&lt;/p&gt;&#xA;" OwnerUserId="35718" LastActivityDate="2017-07-20T08:35:04.760" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20571" PostTypeId="2" ParentId="20554" CreationDate="2017-07-20T09:00:00.633" Score="1" Body="&lt;p&gt;To my knowledge you have to create an auxiliary variable&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If [Entity] == &quot;Industy&quot; THEN [Volume] ELSE 0 END&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can sort by this. Right click on State then select sort and choose sort by auxiliary variable.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-07-20T11:32:39.283" LastActivityDate="2017-07-20T11:32:39.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20572" PostTypeId="1" AcceptedAnswerId="21672" CreationDate="2017-07-20T10:22:01.873" Score="0" ViewCount="5352" Body="&lt;p&gt;It is not very clear what is the difference between the following two schemes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jBqXT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jBqXT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From help docs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Test &amp;amp; Score&lt;/strong&gt; widget: tests learning algorithms on data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Predictions&lt;/strong&gt; widget: shows models’ predictions on the data. Which model? Pretrained one? Where is it taken from?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously the results of &lt;strong&gt;Confusion Matrix&lt;/strong&gt; are different. So how should I train and test my model is not clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If can easily do this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N6svk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N6svk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the point of a separate single &lt;strong&gt;Predictions&lt;/strong&gt; widget connected to some learner? What does it actually test? &lt;/p&gt;&#xA;" OwnerUserId="22012" LastEditorUserId="22012" LastEditDate="2017-07-25T11:32:25.460" LastActivityDate="2017-07-25T11:32:25.460" Title="Why Orange &quot;Predictions&quot; and &quot;Test &amp; Score&quot; produce different results on the same data?" Tags="|orange|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20573" PostTypeId="1" AcceptedAnswerId="20575" CreationDate="2017-07-20T10:45:20.630" Score="2" ViewCount="3331" Body="&lt;p&gt;I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy of string as features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a &lt;code&gt;multiclass&lt;/code&gt; data set (40-50 classes), but it is imbalanced. Some classes can be 1000x smaller compared to the largest class. I have restricted the data to 1 million strings per class as maximum, otherwise the imbalance could be even larger. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of this I want to use over-sampling to improve the data for the underrepresented classes. I have looked into &lt;code&gt;ADASYN&lt;/code&gt; and &lt;code&gt;SMOTE&lt;/code&gt; from the python &lt;code&gt;imblearn&lt;/code&gt; package. But when I run it the process eats up all my &lt;code&gt;RAM&lt;/code&gt; in the swap memory, and soon after the process gets killed. I assume because the memory is not enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is now how to best proceed. Obviously my data is too large to be over-sampled as it is. I have thought of two options, but I cannot make out which is the most &quot;correct&quot;. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;I sent in only one underrepresented class and the largest class, and repeated this for each underrepresented class. I am not sure if this could mean that classes might start to overlap though. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I instead under-sample the data, maybe down to 100k samples per class. This might reduce the data enough such that I can run oversampling on the less represented classes (with 1k-10k samples). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any other options that are more appropriate that I have missed? &lt;/p&gt;&#xA;" OwnerUserId="31948" LastActivityDate="2021-08-19T10:00:23.383" Title="Handling large imbalanced data set" Tags="|python|bigdata|multiclass-classification|class-imbalance|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20575" PostTypeId="2" ParentId="20573" CreationDate="2017-07-20T11:28:47.113" Score="4" Body="&lt;p&gt;There are multiple options, depending on your problem and the algorithms you want to use. The most promising (or closest to your original plan) is to use a generator to prepare batches of training data. This is only useful for models that allow for partial fits, like neural networks. Your generator can just stratify examples by for example generating a batch that includes exactly one of each target. One epoch would be when you served all the samples from the biggest class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Downsampling is not a bad idea but it depends on the difficulty of your task, because you do end up throwing away information. You could look at some curves depending on the amount of samples for your model, if it looks relatively capped this wouldn't be a big issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of models allow for weighting classes in your loss function. If we have 10,000 of class A and 1,000 of class B, we could weight class B 10x, which means mistakes that way count much harder and it will focus relatively more on samples from class B. You could try this but I could see this going wrong with extreme imbalances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can even combine these methods, downsample your biggest classes, upsample your smaller classes and use weights to balance them perfectly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Example of the batch options:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have 4x A, 2x B and 1x C, so our set is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A1 A2 A3 A4 B1 B2 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regular upsampling would go to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A1 A2 A3 A4 B1 B2 B1 B2 C1 C1 C1 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this will not fit in our memory in a big data setting. What we do instead is only store our original data in memory (could even be on disk) and keep track where we are for each class (so they are seperated on target).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: A1 A2 A3 A4&#xA;B: B1 B2&#xA;C: C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our first batch takes one of each class:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A1 B1 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now our C class is empty, which means we reinitialize it, shuffle them (in this case it's only one example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: A2 A3 A4&#xA;B: B2&#xA;C: C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next batch:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A2 B2 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B and C are empty, reinitialize them and shuffle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A: A3 A4&#xA;B: B2 B1&#xA;C: C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next batch is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A3 B2 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And our last one of the epoch would be A4 B1 C1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, we have the same distribution as the full memory option, but we never keep more in memory than our original ones, and the model always gets balanced, stratified batches.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-07-20T12:11:11.810" LastActivityDate="2017-07-20T12:11:11.810" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20578" PostTypeId="1" CreationDate="2017-07-20T12:49:22.490" Score="1" ViewCount="121" Body="&lt;p&gt;I have trained two boosted classifiers on same data with same features, but I am getting two different set of feature importance from the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few high ranked features (suppose top 10) from one boosted classifier is almost 5-6 ranks below when compared with the top 10 of other boosted classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything wrong with this, or its totally okay to have this kind of behavior as it depends on how a model learns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastActivityDate="2017-07-20T12:49:22.490" Title="Feature importance varying a lot using same data with same features" Tags="|feature-selection|feature-engineering|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20580" PostTypeId="1" CreationDate="2017-07-20T13:52:15.260" Score="1" ViewCount="22" Body="&lt;p&gt;I have some data as below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample_id  start end class&#xA;   1        3     7   1&#xA;   1        10    12  1&#xA;   1        14    18  1&#xA;   2        5     7   2&#xA;   2        9     11  2&#xA;   2        14    20  2&#xA;   2        21    26  2&#xA;   3        4     6   1&#xA;   3        8     11  1&#xA;   .....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It could be viewed as below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;reference    0.............................................................30&#xA;    sample 1:  3------7     10-----12   14--------18&#xA;    sample 2:     5---7   9-----11      14----------------20  21---------26&#xA;    sample 3:   4---6   8-------11   13---------------19  20----22  24-------28&#xA;    sample 4:   4------------------12       15--17  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In short, it could be described like this. Suppose, there is a 1000m long straight line. I randomly select a number n between 20-100. Then randomly select n segments from the reference line and create a sample. Similarly, for second sample, select a random number between 20-100 and take that many random segments from the reference line. Then randomly select 50% of the samples as class 1 and rest of the samples as class 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, there are two types of samples, class 1 and class 2. Each sample consists of n number of segments. In above example, sample 1 consists of 3 intervals and second sample consists of 4 intervals. Number of intervals vary from sample to sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to use a deep learning algorithm to classify sample type as either class 1 or 2. Which deep learning algorithm should I use? Or what would be the solution for the problem using deep learning technique.&lt;/p&gt;&#xA;" OwnerUserId="35812" LastActivityDate="2017-07-20T13:52:15.260" Title="Solving problem with variational dimension with deep learning" Tags="|machine-learning|neural-network|classification|data-mining|deep-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20582" PostTypeId="1" AcceptedAnswerId="20583" CreationDate="2017-07-20T16:47:57.383" Score="3" ViewCount="2733" Body="&lt;p&gt;I have three dataframes. Their shapes are (2656, 246), (2656, 2412) and (2656, 7025). I want to merge dataframes as above:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yTWn6.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yTWn6.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So It will result a (2656, 9683) Dataframe. Thanks for any help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typo on image: on Dataframe 3, it will 7025, not 5668.&lt;/p&gt;&#xA;" OwnerDisplayName="user34075" LastEditorDisplayName="user34075" LastEditDate="2017-07-20T17:11:28.537" LastActivityDate="2017-07-20T17:37:51.023" Title="Concatenate dataframes Pandas" Tags="|pandas|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20583" PostTypeId="2" ParentId="20582" CreationDate="2017-07-20T17:37:51.023" Score="4" Body="&lt;p&gt;Assuming that the rows are in same order that you wish to merge all of the dataframes, you can use the concat command specificying axis=1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;new_df = pd.concat([df1, df2, df3], axis=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the row index for each of the data frames are different and you want to merge them in the current order, you can also apply ignore_index:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;new_df = pd.concat([df1, df2, df3], ignore_index=True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For details on the merge, join &amp;amp; concatenation operations, please refer to the &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/merging.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas docs&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2017-07-20T17:37:51.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20584" PostTypeId="1" CreationDate="2017-07-20T18:29:39.897" Score="1" ViewCount="1063" Body="&lt;p&gt;I have a somewhat strange case that I can't find an answer to anywhere. It is really only applicable if you have a legitimately large data set. Let me describe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBoost returns feature importance for all variables in the model. I have been using a data set with over 1000 features and typically about 25-50% of them come back with non-zero feature importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to make predictions with this model. Imagine I am making predictions on A LOT of observations (over 1m, imagine predicting on incoming tweets or something). If I re-train the model, giving it only the columns that the original training picked out as valuable (non-zero importance), then the re-trained model makes predictions MUCH FASTER than the original, and WITH THE EXACT SAME ACCURACY. This last part makes sense, because it is building the exact same trees. Also, not only is it faster, but it takes up a lot less space in memory because you are passing it something like a 1m x 250 matrix instead of a 1m x 1000 matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is there a way to train a model, and then pass the predict function only the features it actually uses &lt;em&gt;without having to re-train the model&lt;/em&gt; to except only those features instead of the full matrix? This &lt;a href=&quot;http://machinelearningmastery.com/feature-importance-and-feature-selection-with-xgboost-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; discusses a similar concept of using XGBoost for feature selection, but it does it by &lt;em&gt;retraining&lt;/em&gt; the model each time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious answer is &quot;just re-train it&quot; but if I am using a lot of training observations then training the model takes awhile, so I would rather not have to retrain it if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing this in R, but solutions in Python/sklearn would also be helpful.&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="35872" LastActivityDate="2017-07-20T18:29:39.897" Title="XGBoost: predict on only valuable features" Tags="|r|scikit-learn|bigdata|xgboost|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="20586" PostTypeId="1" CreationDate="2017-07-20T19:19:54.807" Score="2" ViewCount="1407" Body="&lt;p&gt;I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Preprocessing performed : Converted IP addresses to integers. Removed rows with NA's. Dis not perform scaling as it doesn't make any sense with categorical data and also since I'm computing Hamming distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modelling and Implementation : I have implemented 'Hamming Distance' to compute the distance matrix for the set. I have tried converting the categorical variables to numerical by converting them to factors and perform Kmeans on the data, however the accuracy I'm getting is very low as expected. I have used KModes as well and I have got cluster labels to the data, I'm having challenge proceeding further from here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question : Once the distance matrix is computed, how can I go about detecting the outliers with such data type? I really appreciate any recommendations. Thanks in advance &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data Snippet :  2887562076 2344654028        6   41940      80&#xA;                2344654028 2887562076        6      80   41940&#xA;                2344654028 2887562076        6      80   41940&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31135" LastActivityDate="2017-08-24T01:52:26.123" Title="Outlier detection on categorical network log data" Tags="|r|clustering|categorical-data|unsupervised-learning|outlier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="20587" PostTypeId="1" AcceptedAnswerId="22105" CreationDate="2017-07-20T19:43:25.967" Score="12" ViewCount="21186" Body="&lt;p&gt;I have a dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Sample Dataframe&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame({&#xA;    'names': ['A','B','C','D','E','F','G','H','I','J','K','L'],&#xA;    'col1': [0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0],&#xA;    'col2': [0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to replace some of the &lt;code&gt;0&lt;/code&gt;'s in &lt;code&gt;col1&lt;/code&gt; and &lt;code&gt;col2&lt;/code&gt; with &lt;code&gt;1&lt;/code&gt;'s, but not replace the &lt;code&gt;0&lt;/code&gt;'s if three or more &lt;code&gt;0&lt;/code&gt;'s are consecutive in the same column.  How can this be done with pandas?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Original Dataset:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names   col1    col2&#xA;A   0   0&#xA;B   1   0&#xA;C   0   0&#xA;D   1   0&#xA;E   1   1&#xA;F   1   0&#xA;G   0   1&#xA;H   0   0&#xA;I   0   1&#xA;J   1   0&#xA;K   0   0&#xA;L   0   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Desired Dataset:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names   col1    col2&#xA;A   1   0&#xA;B   1   0&#xA;C   1   0&#xA;D   1   0&#xA;E   1   1&#xA;F   1   1&#xA;G   0   1&#xA;H   0   1&#xA;I   0   1&#xA;J   1   0&#xA;K   1   0&#xA;L   1   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35420" LastEditorUserId="35420" LastEditDate="2017-08-06T03:41:56.140" LastActivityDate="2017-08-09T12:06:11.903" Title="Find the consecutive zeros in a DataFrame and do a conditional replacement" Tags="|python|pandas|dataframe|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="20588" PostTypeId="1" CreationDate="2017-07-21T03:24:55.810" Score="1" ViewCount="429" Body="&lt;p&gt;I don't know which the update statement for Stochastic Gradient descent is the right one for the Newton update of the coefficients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it use the sum of the derivatives for ALL variables to update all the  coefficients or is it iterating through each coefficient and using the sum of the derivatives for only that particular variable to update that coefficient?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;On page 120 (pg 139 on pdf) of the Elements of Statistical Learning&#xA;book &lt;a href=&quot;https://web.stanford.edu/~hastie/Papers/ESLII.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://web.stanford.edu/~hastie/Papers/ESLII.pdf&lt;/a&gt; The Newton&#xA;update statement (4.23) looks like all coefficients are updated with the sum of the&#xA;derivatives of all rows Xi for ALL the variables&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In the update statement in this link &lt;a href=&quot;http://machinelearningmastery.com/logistic-regression-tutorial-for-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://machinelearningmastery.com/logistic-regression-tutorial-for-machine-learning/&lt;/a&gt;&#xA;the coefficients are updated with the sum of the derivatives for&#xA;each that variable and not the sum of the derivatives for all&lt;br&gt;&#xA;variables.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The update statement on slide # 23, pg 21 for &lt;a href=&quot;http://www.hlt.utdallas.edu/~vgogate/ml/2015s/lectures/lr-nb-lec6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.hlt.utdallas.edu/~vgogate/ml/2015s/lectures/lr-nb-lec6.pdf&lt;/a&gt;&#xA;I read the update statement to say the new coefficient for the given&lt;br&gt;&#xA;variable is the old weight + the derivative of that given variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27394" LastActivityDate="2017-07-21T03:24:55.810" Title="Update statement for Logistic Regression" Tags="|logistic-regression|gradient-descent|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20592" PostTypeId="1" CreationDate="2017-07-21T09:56:13.730" Score="1" ViewCount="777" Body="&lt;p&gt;I am trying to use xgboost in R for pairwise ranking for an implicit dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, let's assume that I am dealing with a search problem, where I want to rank documents relative to a given query. Each query has an ID and the documents that were clicked also have an ID. My dataset only consists of the positive feedback and looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;queryID    documentID    features    target&#xA;   1            1            ...        1&#xA;   1            3            ...        1&#xA;   1            13           ...        1                    &#xA;   2            3            ...        1&#xA;   2            4            ...        1 &#xA;   3            990          ...        1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have read, that for grouping my queries together, I have to use setinfo(group, &lt;em&gt;list&lt;/em&gt;). But what do I do with my documentID, do I implement it as just another feature in my feature list?&lt;/p&gt;&#xA;" OwnerUserId="35906" LastActivityDate="2017-07-21T09:56:13.730" Title="xgboost - How do I treat document ID in pairwise ranking" Tags="|xgboost|ranking|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20593" PostTypeId="1" AcceptedAnswerId="20595" CreationDate="2017-07-21T10:29:59.200" Score="0" ViewCount="73" Body="&lt;p&gt;When training neural network, is it better to randomly choose data for every batch or feed the data sequentially? Why?&lt;/p&gt;&#xA;" OwnerUserId="35908" LastActivityDate="2017-07-21T12:32:33.467" Title="SGD data should be randomly selected or sequentially feed?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20594" PostTypeId="1" CreationDate="2017-07-21T11:19:48.160" Score="4" ViewCount="227" Body="&lt;p&gt;I want to build a &lt;strong&gt;Recommendation System&lt;/strong&gt; to recommend products to users.&#xA;This is for research purposes. &#xA;The context-system the engine will be integrated in is also not build yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So right now I am starting the project, building kind of a &lt;strong&gt;E-Commerce &amp;amp; Social-Network-Platform&lt;/strong&gt; for research purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To realize the recommendation system, I want to integrate &lt;strong&gt;Neo4j&lt;/strong&gt; and &lt;strong&gt;Elasticsearch&lt;/strong&gt; with each other. For the &lt;strong&gt;content based filtering&lt;/strong&gt; part Elasticsearch should do its job nicely. For the &lt;strong&gt;collaborative filtering&lt;/strong&gt; part I want to use the Graph in Neo4j.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to ask you for some experience or suggestions about the following questions regarding this topic:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Should I use another database as the &lt;strong&gt;main system storage&lt;/strong&gt; and just use Neo4j to store recommendation data? Or is it a suggestive way to just store everything in the same graph?&lt;/li&gt;&#xA;&lt;li&gt;How would you determine which part of the recommendation computation should happen &lt;strong&gt;online&lt;/strong&gt; in Realtime and which &lt;strong&gt;offline&lt;/strong&gt; as a precomputation?&lt;/li&gt;&#xA;&lt;li&gt;Does anyone here have some experience with this kind of realization? What did your architecture look like?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sorry, if this is a little vague described here and there. I am also new to this and want to expand my horizon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks alot for your help. I would be really happy to get some input here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers TJ&lt;/p&gt;&#xA;" OwnerUserId="35910" LastActivityDate="2017-07-21T12:55:05.203" Title="Graph-Document-Recommendations" Tags="|machine-learning|predictive-modeling|recommender-system|neo4j|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20595" PostTypeId="2" ParentId="20593" CreationDate="2017-07-21T12:32:33.467" Score="1" Body="&lt;p&gt;If you feed the data randomly, there are fewer chances of overfitting than if you feed the data sequentially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data is fed sequentially, then the network will not learn to generalize as the weights will be updated for that batch according to that same image again and again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is always recommended to feed the randomly.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastActivityDate="2017-07-21T12:32:33.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20596" PostTypeId="2" ParentId="20594" CreationDate="2017-07-21T12:55:05.203" Score="1" Body="&lt;p&gt;Why not using a Multi-Model DBMS like OrientDB (I'm the author) or any others? In this way, with just one system you can store documents and graphs together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be:&#xA;- much faster (because you avoid the double call to 2 systems where most of the times the 2nd call needs the output of the 1st call, so you pay a lot of latency)&#xA;- easier to manage: one DBMS, one set of skills, in the case of commercial support, only one vendor&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following your use case, with a Multi-Model DBMS like OrientDB, you could start with a FULL-TEXT query (the underlying technology is Lucene, the same of Elastic Search) and then navigate the resulting graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the Social Travel Agency example about how you can mix multiple models at the same time: &lt;a href=&quot;https://orientdb.com/docs/3.0.x/gettingstarted/demodb/DemoDB-DataModel.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://orientdb.com/docs/3.0.x/gettingstarted/demodb/DemoDB-DataModel.html&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="6477" LastActivityDate="2017-07-21T12:55:05.203" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="20601" PostTypeId="1" AcceptedAnswerId="21611" CreationDate="2017-07-21T17:10:21.067" Score="4" ViewCount="633" Body="&lt;p&gt;I use Python for data munging, R for data analysis, and I combine both by running the munging/analysis sequentially in a Makefile. My Python and R scripts depend on various constants. I typically store my Python constants in a settings.py file, and I import this file across several Python scripts which depend on those constants. My R constants are typically not used in more than one script, so I store those constants in the script which uses them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some constants are shared by both the Python scripts and R scripts. Right now, I am defining them separately in R/Python files. However, it would be better if there was some way of defining constants in common across Python and R files, since this redundancy can lead to mistakes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best way for me to define constant in common across files? Obviously, there is the problem that these are two different languages; so, for example, a list in Python is not quite the same as a list in R. However, there might be some solution which is more elegant than simply repeating constants across files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe this would be the best place for this question, but if it belongs in the Python or R StackExchanges, please let me know.&lt;/p&gt;&#xA;" OwnerUserId="35919" LastActivityDate="2017-07-22T03:27:51.473" Title="Importing Common Constants to R and Python" Tags="|python|r|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="20604" PostTypeId="1" AcceptedAnswerId="20606" CreationDate="2017-07-21T21:15:44.587" Score="5" ViewCount="134" Body="&lt;p&gt;I am trying to predict an output value based on several continuously-valued inputs using a regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what approach is appropriate to scale/transform the input data for the regression. Let's just pretend that it is unlabeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My most naive approach would be to add each input multiple times:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;input&lt;/li&gt;&#xA;&lt;li&gt;log(input)&lt;/li&gt;&#xA;&lt;li&gt;sqrt(input)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and then let the regression model worry about finding which flavor of each input (if any) is significant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the risks with using this approach?&lt;/p&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2017-07-21T22:34:25.813" Title="Why is duplicating inputs bad?" Tags="|regression|feature-selection|linear-regression|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="20606" PostTypeId="2" ParentId="20604" CreationDate="2017-07-21T22:06:18.967" Score="7" Body="&lt;p&gt;The issue with building a regression model on all 3 of these is that you are potentially introducing &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;noreferrer&quot;&gt;multicollinearity&lt;/a&gt; into the model. Although log(input) and sqrt(input) are not linear functions of the input a quick test (using Matlab) shows they are still highly correlated (depending on the range)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input=rand(1,100);&#xA;input_log=log(input);&#xA;input_rt=sqrt(input);&#xA;corrcoef(input, input_log) %-&amp;gt;0.8549&#xA;corrcoef(input, input_rt) %-&amp;gt;0.9779&#xA;corrcoef(input_sq, input_rt) %0&amp;gt;0.9407&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Multicollinearity will not reduce the predictive power of the model, but it will make the regression coefficients of these variables difficult to interpret since small adjustments in the data may cause the regression model to switch which of the 3 input variations it finds significant. It also adds unnecessary complexity to the model. I would train the models with the 3 separately if you want to observe the effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, applying things like sqrt and log to your output can be useful if the data currently can't be described by a linear relationship.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastEditorUserId="33922" LastEditDate="2017-07-21T22:34:25.813" LastActivityDate="2017-07-21T22:34:25.813" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21610" PostTypeId="1" AcceptedAnswerId="21620" CreationDate="2017-07-22T03:16:08.207" Score="8" ViewCount="6526" Body="&lt;p&gt;I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried several variations of the code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input = Input(shape=(1036800,), name='image_input')&#xA;initial_model = VGG16(weights='imagenet', include_top=False)&#xA;x = Flatten()(initial_model(input).output)&#xA;x = Dense(200, activation='relu')(x)&#xA;x = Dense(1)(x)&#xA;model = Model(inputs=input, outputs=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Previous variations of the above code yielded errors related to the input being the wrong dimensions, &lt;code&gt;input_shape&lt;/code&gt; needing to have 3 channels (when using (1036800,) for that parameter in the initialization of VGG16), and the most recent error that results from running the above code is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;model_alex.py&quot;, line 57, in &amp;lt;module&amp;gt;&#xA;    model = initialize_model()&#xA;  File &quot;model_alex.py&quot;, line 20, in initialize_model&#xA;    x = Flatten()(initial_model(input).output)&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 596, in __call__&#xA;    output = self.call(inputs, **kwargs)&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 2061, in call&#xA;    output_tensors, _, _ = self.run_internal_graph(inputs, masks)&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 2212, in run_internal_graph&#xA;    output_tensors = _to_list(layer.call(computed_tensor, **kwargs))&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/layers/convolutional.py&quot;, line 164, in call&#xA;    dilation_rate=self.dilation_rate)&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py&quot;, line 3156, in conv2d&#xA;    data_format='NHWC')&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py&quot;, line 639, in convolution&#xA;    input_channels_dim = input.get_shape()[num_spatial_dims + 1]&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py&quot;, line 500, in __getitem__&#xA;    return self._dims[key]&#xA;IndexError: list index out of range&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the &lt;a href=&quot;https://pastebin.com/9EwQy5tV&quot; rel=&quot;noreferrer&quot;&gt;full code&lt;/a&gt;. Here is the &lt;a href=&quot;https://drive.google.com/open?id=0B-i1XzvYHh2ZaHVSVmhNSlNLc0E&quot; rel=&quot;noreferrer&quot;&gt;sample data file&lt;/a&gt; used in the script.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the possible approach towards fixing this might be to resize the raw image files to 224x224 and turn them into numpy arrays of shape (224, 224, 3) so they can be plugged into the pre-trained model's first layer. However, I don't want to warp the images or waste another night pre-processing data when I should already be training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides that, I all I can think to do is Google my problem and try to adapt the found solutions or aimlessly tweak various shape related parameters and functions -- neither of which has gotten me very far over the past 4 hours.&lt;/p&gt;&#xA;" OwnerUserId="36935" LastEditorUserId="24000" LastEditDate="2017-07-22T21:07:18.757" LastActivityDate="2017-07-22T21:07:18.757" Title="Prepending Input layer to pre-trained model" Tags="|python|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21611" PostTypeId="2" ParentId="20601" CreationDate="2017-07-22T03:27:51.473" Score="6" Body="&lt;p&gt;You should consider storing this sort of data into a format that is more universally accessible.  I would suggest either JSON or YAML. JSON can be readily parsed by both &lt;a href=&quot;https://stackoverflow.com/questions/2617600/importing-data-from-a-json-file-into-r&quot;&gt;R&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/questions/7771011/parse-json-in-python&quot;&gt;Python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your data would look something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;    &quot;FIRST_YEAR&quot;: 1952, &#xA;    &quot;IV_TYPES&quot;: [&#xA;        [&quot;p&quot;, 6], &#xA;        [&quot;n&quot;, 6], &#xA;        [&quot;w&quot;, 4]&#xA;    ]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-07-22T03:27:51.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21612" PostTypeId="2" ParentId="20587" CreationDate="2017-07-22T04:45:28.260" Score="5" Body="&lt;p&gt;You should use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.shift.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;pandas.DataFrame.shift()&lt;/code&gt;&lt;/a&gt; to find the pattern you need.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fill_zero_not_3(series):&#xA;    zeros = (True, True, True)&#xA;    runs = [tuple(x == 0 for x in r)&#xA;            for r in zip(*(series.shift(i)&#xA;                           for i in (-2, -1, 0, 1, 2)))]&#xA;    need_fill = [(r[0:3] != zeros and r[1:4] != zeros and r[2:5] != zeros)&#xA;                 for r in runs]&#xA;    retval = series.copy()&#xA;    retval[need_fill] = 1&#xA;    return retval&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame({&#xA;    'names': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L'],&#xA;    'col1': [0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0],&#xA;    'col2': [0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]}).set_index('names')&#xA;&#xA;df['col1'] = fill_zero_not_3(df['col1'])&#xA;df['col2'] = fill_zero_not_3(df['col2'])&#xA;print(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       col1  col2&#xA;names            &#xA;A         1     0&#xA;B         1     0&#xA;C         1     0&#xA;D         1     0&#xA;E         1     1&#xA;F         1     1&#xA;G         0     1&#xA;H         0     1&#xA;I         0     1&#xA;J         1     0&#xA;K         1     0&#xA;L         1     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-07-22T04:45:28.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21618" PostTypeId="1" AcceptedAnswerId="21619" CreationDate="2017-07-22T10:08:34.107" Score="0" ViewCount="1777" Body="&lt;p&gt;I have downloaded a dataset from Amazon. &lt;a href=&quot;http://jmcauley.ucsd.edu/data/amazon/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://jmcauley.ucsd.edu/data/amazon/&lt;/a&gt; Dataset involves feature vectors of images. There are around 1.5 M feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset consists of 10 characters (the product ID), followed by 4096 floats (repeated for every product). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every product image involves feature vectors with (4096x1) size. Feature vectors involve float numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do these float numbers mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I understood is, there are at total 4096 features, and each index of feature vectors indicate a specific feature. The values in feature vectors indicate the frequency of regarding feature in all specific image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it so? Or, if it is not, what might be the right explanation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks, &lt;/p&gt;&#xA;" OwnerUserId="28138" LastEditorUserId="28138" LastEditDate="2017-07-22T10:42:35.167" LastActivityDate="2017-07-22T12:02:27.087" Title="Image Feature Vectors" Tags="|image-classification|feature-extraction|image-recognition|computer-vision|amazon-ml|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21619" PostTypeId="2" ParentId="21618" CreationDate="2017-07-22T10:59:41.730" Score="1" Body="&lt;p&gt;The same link shows how these features are extracted, with a deep look into the cited article &lt;a href=&quot;http://cseweb.ucsd.edu/~jmcauley/pdfs/sigir15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Image-based recommendations on styles and substitutes&quot;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Features are calculated from the original images using the&#xA;  Caffe deep learning framework [11]. In particular, we used a&#xA;  Caffe reference model with 5 convolutional layers followed&#xA;  by 3 fully-connected layers, which has been pre-trained on&#xA;  1.2 million ImageNet (ILSVRC2010) images. We use the&#xA;  output of FC7, the second fully-connected layer, which results&#xA;  in a feature vector of length F = 4096.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The reference neural network that was mentioned is the &lt;em&gt;BAIR Reference CaffeNet&lt;/em&gt; at the &lt;a href=&quot;http://caffe.berkeleyvision.org/model_zoo.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe Model Zoo&lt;/a&gt;, which is a slightly modified version of AlexNet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the model was trained over &lt;a href=&quot;http://www.image-net.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageNet&lt;/a&gt;, which contains a wide variety of photographs of various categories (1000 of them, if I recall correctly), retrieving the neural codes of one of the layers (obtained just by forward propagation) will give you visual features with a fair representation of the images, even if the network was not specifically trained for Amazon's tasks (such as product recommendation). What these values actually mean is not something that tangible: it is the outcome of multiple 2D convolutions and other normalization and regularization functions, the parameters of which were adjusted specifically for classifying photographs from ImageNet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The FC7 layer has a rectified linear unit activation (ReLU), which means that they are all non-negative numbers (potentially with several zeros). And since it's a fully connected layer that follows several convolutions, there is no intuitive mapping between a feature index and a certain characteristic of the image. You may picture the network as a highly complex function that yields a high-level representation of the image, under the form of a vector of numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also the paper &lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-319-10590-1_38&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Codes for Image Retrieval&lt;/a&gt;, where the authors retrieve features from a pre-trained neural network in this fashion, for retrieving images in a different image domain.&lt;/p&gt;&#xA;" OwnerUserId="17538" LastEditorUserId="17538" LastEditDate="2017-07-22T12:02:27.087" LastActivityDate="2017-07-22T12:02:27.087" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="21620" PostTypeId="2" ParentId="21610" CreationDate="2017-07-22T11:11:07.180" Score="5" Body="&lt;p&gt;The issue is that you shouldn't flatten the images into 1-dimensional vector because the VGG16 contains 2D convolution layers (e.g. spatial convolution over images), which require the input to have the shape of &lt;code&gt;(number_of_images, image_height, image_width, image_channels)&lt;/code&gt;, given that &lt;code&gt;keras.backend.image_data_format()&lt;/code&gt; returns &lt;code&gt;'channels_last'&lt;/code&gt;. If your &lt;code&gt;image_data_format&lt;/code&gt; is &lt;code&gt;'channels_first'&lt;/code&gt;, change the input data shape to &lt;code&gt;(number_of_images, image_channels, image_height, image_width)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is your fixed code (tested with Keras 2.0.4):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_train = x_train.reshape((x_train.shape[0], 480, 720, 3))&#xA;x_test = x_test.reshape((x_test.shape[0], 480, 720, 3))&#xA;&#xA;initial_model = VGG16(weights='imagenet', include_top=False)&#xA;input = Input(shape=(480, 720, 3), name='image_input')&#xA;x = Flatten()(initial_model(input))&#xA;x = Dense(200, activation='relu')(x)&#xA;x = BatchNormalization()(x)&#xA;x = Dropout(0.5)(x)&#xA;x = Dense(1)(x)&#xA;model = Model(inputs=input, outputs=x)&#xA;model.compile(loss='mse', optimizer='adam')&#xA;&#xA;model.fit(x_train, y_train, epochs=20, batch_size=16)&#xA;score = model.evaluate(x_test, y_test, batch_size=16)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2017-07-22T11:11:07.180" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21622" PostTypeId="1" AcceptedAnswerId="21623" CreationDate="2017-07-22T14:45:33.643" Score="2" ViewCount="616" Body="&lt;p&gt;Convolutional artificial neural networks work well in particular with images. Why?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="29169" LastEditDate="2021-02-16T00:44:31.450" LastActivityDate="2021-02-16T00:44:31.450" Title="Why do convolutional networks work so well for images?" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="4" ClosedDate="2017-07-22T21:06:46.373" ContentLicense="CC BY-SA 3.0" />
  <row Id="21623" PostTypeId="2" ParentId="21622" CreationDate="2017-07-22T14:45:33.643" Score="3" Body="&lt;p&gt;The convolution neural networks take into consideration that an image already has a two-dimensional structure. This is a domain knowledge provided by humans and not something the neural network can pick up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider if I would give you the raw pixels shuffled in a flattened array (or any other signal for that matter). Would you be able to reconstruct its 2D structure? Probably this would be almost impossible (unless someone builds an algorithm to solve jigsaw puzzles at the pixel level!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying convolution with various strides we are looking for patterns in the patches of the image and not on the individual pixels. Using different strides we are explicitly expressing the amount of resolution that we are investigating the patterns of the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So given that these convolution filters are trainable we are able to have an architecture that works well with images. In fact, it would work well with any signal that would have a 2D structure.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T14:45:33.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21625" PostTypeId="1" AcceptedAnswerId="21626" CreationDate="2017-07-22T16:17:11.350" Score="0" ViewCount="118" Body="&lt;p&gt;Does a single word vector, trained using Word2Vec or a similar approach, carries information or meaning?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T16:17:11.350" Title="What does a Word2Vec represents?" Tags="|word2vec|" AnswerCount="1" CommentCount="0" ClosedDate="2017-07-22T21:06:52.247" ContentLicense="CC BY-SA 3.0" />
  <row Id="21626" PostTypeId="2" ParentId="21625" CreationDate="2017-07-22T16:17:11.350" Score="1" Body="&lt;p&gt;So we have that a word to vector model has been trained on a certain corpus to be able given words as inputs (one hot encoding is one way for the input) to represent the word as a vector of a usually high dimensionality of numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it carry information? Yes, we could say that it carries the information about the words. Similar words have similar vectors. The point is that us as humans we do not define the similarity but the neural network defines the similarity for us based on the corpus that we are providing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it carry meaning? No, because this representation does include grammatical or syntactical information and therefore the answer would be that it does not carry any meaning.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T16:17:11.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21627" PostTypeId="1" AcceptedAnswerId="21628" CreationDate="2017-07-22T16:43:05.140" Score="3" ViewCount="1624" Body="&lt;p&gt;We have heard a lot about the advantages that artificial neural networks have over other models but what are the disadvantages of them in comparison to the simplest case of a linear model?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2020-09-14T11:51:27.810" Title="What are the disadvantages of using deep neural networks compared to a linear model?" Tags="|deep-learning|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21628" PostTypeId="2" ParentId="21627" CreationDate="2017-07-22T16:43:05.140" Score="3" Body="&lt;p&gt;First of all, we should say that a single affine layer of a neural network without any non-linearities/activations is practically the same as a linear model. Here we are referring to deep neural networks that have multiple layers and activation functions (non-linearities as relu, elu, tanh, sigmoid etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second of all nonlinearities and multiple layers introduce a nonconvex and usually rather complex error space which means that we have many local minimums that the training of the deep neural network can converge to. This means that a lot of hyperparameters have to be tuned in order to get to a place in the error space where the error is small enough so that the model will be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of hyper parameters which could start from 10 and reach up to 40 or 50 are dealt with bayesian optimization using Gaussian processes to optimize them which still does not guarantee good performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their training is very slow and adding the tuning of the hyperparameters into that makes it even slower where in comparison the linear model would be much faster to be trained. This introduces a serious cost-benefit tradeoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A trained linear model has weights which are interpretable and give useful information to the data scientist onto how various features play a role for the task at hand.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="24885" LastEditDate="2017-09-13T08:47:08.663" LastActivityDate="2017-09-13T08:47:08.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21629" PostTypeId="1" AcceptedAnswerId="21630" CreationDate="2017-07-22T17:32:42.783" Score="0" ViewCount="179" Body="&lt;p&gt;Typically all the layers of an artificial neural network are trainable. But what are the hidden layers trying to learn?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T18:25:17.373" Title="What do the hidden layers of a neural network try to learn?" Tags="|deep-learning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21630" PostTypeId="2" ParentId="21629" CreationDate="2017-07-22T17:32:42.783" Score="2" Body="&lt;p&gt;Each hidden layer represents a nonlinear application of a function on the inputs of the previous layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first layers perform a function on our data-inputs and the last layer is responsible for either using softmax for classification purposes or be an affine layer for regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The in-between layers are necessary in order to stack complexity and therefore avoid having to create a layer of very high complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, in theory, two level deep neural network can represent any function. But how complex would this function need to be?&lt;br&gt;&#xA;A layer of a very high complexity would be able to map our raw inputs into our final classification or regression task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However what would be simpler to do is to let the intermediate hidden layers learn useful features regarding the given task and then the next hidden layer could exploit these features in its own way. And then the next hidden layer could use these newly transformed features in its own way and so on. So stacking nonlinear functions one on top of the other is more clever than requiring a single nonlinear function to do all the hard work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a limit on how many layers you can stack together for the main reason that these hidden layers get trained by backpropagation which uses the chain rule of derivation in order to pass the deltas of the error back to the weights of the layers and tune them but a long series of these operations could end-up with exploding or vanishing gradients which means really poor performance or a model that cannot be trained.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T17:32:42.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21632" PostTypeId="1" AcceptedAnswerId="21633" CreationDate="2017-07-22T17:56:52.403" Score="3" ViewCount="703" Body="&lt;p&gt;The state of the art of nonlinearity is to use rectified linear units (ReLU) instead of a sigmoid function in deep neural networks. What are the advantages?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T18:22:08.873" Title="What are the advantages of the rectified linear (relu) over the sigmoid activation function in networks with many layers?" Tags="|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21633" PostTypeId="2" ParentId="21632" CreationDate="2017-07-22T17:56:52.403" Score="4" Body="&lt;p&gt;The sigmoid function becomes asymptotically either zero or one which means that the gradients are near zero for inputs with a large absolute value.&lt;br&gt;&#xA;This makes the sigmoid function prone to vanishing gradient issues which the ReLU does not suffer as much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, ReLU has an attribute which can be seen both as positive and negative depending on which angle you are approaching it. The fact that ReLU is effectively a function that is zero for negative inputs and identity for positive inputs means that it is easy to have zeros as outputs and this leads to dead neurons. However, dead neurons might sound bad but in many cases, it is not because it allows for sparsity. In a way, the ReLU does a similar job of what an L1 regularization would do which would bring some weights to zero which in turn means a sparse solution.&lt;br&gt;&#xA;Sparsity is something that, lots of times, leads to a better generalization of the model but there are times which has a negative impact on performance so it depends.&lt;br&gt;&#xA;A good practice when using ReLU is to initialize the bias to a small number rather than zero so that you avoid dead neurons at the beginning of the training of the neural network which might prevent training in general.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-07-22T17:56:52.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21634" PostTypeId="2" ParentId="21632" CreationDate="2017-07-22T18:22:08.873" Score="0" Body="&lt;p&gt;To add to George Pligor's comment, it is a good idea to use Xavier weight initialization while using ReLU. Here is a description of this idea: &lt;a href=&quot;http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-07-22T18:22:08.873" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21635" PostTypeId="2" ParentId="21629" CreationDate="2017-07-22T18:25:17.373" Score="0" Body="&lt;p&gt;Hidden layers try to find some kind of structure in the data. If you are working with images, visualizing hidden layers can show you what they are trying to learn. Here is an example: &lt;a href=&quot;https://qph.ec.quoracdn.net/main-qimg-85bdd2809fb1d58c65262c909d22df59&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://qph.ec.quoracdn.net/main-qimg-85bdd2809fb1d58c65262c909d22df59&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-07-22T18:25:17.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21637" PostTypeId="1" CreationDate="2017-07-22T23:35:36.983" Score="2" ViewCount="1872" Body="&lt;p&gt;I am trying to code multivariate (or Multi output dx input features and dy outputs) Random Forest Regressor algorithm for a project, i.e. the algorithm can be used to predict more than one dependent variables using same features and training the model only once to account for correlation between the response variables. Does anyone know or have an already coded algorithm in R or python that I can use as a reference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that Sklearn Randomforest can be applied in Multivariate settings. And there is a Package in R called Mutlivariate Random Forest for such use. But, I need an amateur level from scratch implementation that I can understand and learn from about how to code GINI gain function and prediction function for the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried scouting the Github, but haven't found anything useful yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance to whoever is helping. &lt;/p&gt;&#xA;" OwnerUserId="36958" LastActivityDate="2017-07-22T23:35:36.983" Title="Code for Multivariate Random Forest in Python/R?" Tags="|machine-learning|python|regression|random-forest|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21638" PostTypeId="1" AcceptedAnswerId="25308" CreationDate="2017-07-23T05:00:29.247" Score="4" ViewCount="90" Body="&lt;p&gt;I have what seems to me to be a slightly complex version of a decision tree problem, that can't figure out how to model, and I'm trying to avoid the &quot;just dump it into an NN&quot; solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a bunch of expert opinions (from either people or algorithms) where the features dimensions overlap. Both features (inputs) and outputs (decision) are categorical, but I also have a certainty in the decision (let's say 0-1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say that the input features are a,b,c,d,e and there is one output (decisions) from among v,w,x,y,z:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Features   Decision  certainty&#xA; a,b,c       w          0.5&#xA; b,c,d,e     z          0.4&#xA; a,c,e,f     v          0.75&#xA; ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Without the certainty, this would obviously just be a trivial categorical decision tree problem. However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features-&gt;decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The certainties suggest Bayesian combination, and if the data was unidimensional this would be trivial as well. So I'm kind of caught half way between a decision tree and a bayesian model. An obvious cop out is just to dump it into an NN (or even just an NB or regression), using the certainties either as outputs on the categories, or doing something dumb like replicating the I/O pairings per the (un)certainty. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="32947" LastActivityDate="2018-03-07T16:41:00.940" Title="Bayesian combination of multi-dimensional experts?" Tags="|decision-trees|categorical-data|naive-bayes-classifier|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21639" PostTypeId="2" ParentId="17850" CreationDate="2017-07-23T06:59:46.910" Score="0" Body="&lt;p&gt;After you train a model, you can save the weights that it has learned. Next time when you want to train some more, then you can load those weights and start training from that point again. There is no need to train the model from scratch everytime.&lt;/p&gt;&#xA;" OwnerUserId="35718" LastActivityDate="2017-07-23T06:59:46.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21640" PostTypeId="2" ParentId="21638" CreationDate="2017-07-23T08:35:36.990" Score="1" Body="&lt;p&gt;Just so you know, in this case, even the neural network can't help you. For NNs to work, you need to have lots of learnable patterns in the data (meaning data without uncertainity/ irreducible error). If the data itself has a lot of uncertainty in the outcome( i.e. probabilistic outcomes) itself, its very hard of NNs to optimize that. (For more on this, consider modeling following problems with NNs: XOR(with probability =1) vs XOR(with probabilities &amp;lt;1)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, your best bet might be to use regression. Let us know how it folds out, thanks.&lt;/p&gt;&#xA;" OwnerUserId="36971" LastActivityDate="2017-07-23T08:35:36.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21643" PostTypeId="1" AcceptedAnswerId="39444" CreationDate="2017-07-23T17:35:35.327" Score="2" ViewCount="6512" Body="&lt;p&gt;I came across &lt;a href=&quot;https://datascience.stackexchange.com/questions/17099/adding-features-to-time-series-model-lstm&quot;&gt;this&lt;/a&gt; excellent question and answer by Adam Sypniewski. Here method of using RNNs (e.g. LSTMs and GRUs) for predicting time series where the dependent variable depends on several independent variables. Just wondering, is anyone aware of some Keras code for this? &lt;/p&gt;&#xA;" OwnerUserId="32192" LastActivityDate="2021-04-22T13:28:36.520" Title="time series prediction with several independent variables using RNNs and Keras" Tags="|machine-learning|python|neural-network|deep-learning|keras|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21645" PostTypeId="1" CreationDate="2017-07-23T19:28:30.260" Score="1" ViewCount="3271" Body="&lt;p&gt;Let's say this: &#xA;I'm working on a machine learning project and I'm working on a dataset with a 4250,13 as shape and it is already grouped in 7 categories! &#xA;Note that those categories can't be considered as predictor.&#xA;Here is how my data is grouped in the categories feature 'FAC':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FSEG       32.852598 % of the dataset&#xA;FSTA       19.151644 % of the dataset&#xA;FD       19.003181 % of the dataset&#xA;FM      16.076352 % of the dataset&#xA;FT      5.132556 % of the dataset&#xA;PPSE       4.814422 % of the dataset&#xA;FSDC    2.969247 % of the dataset&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a continuous output that I want to predict 'CGPA', so the task is a regression, my goal is to predict it in each category, and the final decision will be the category where the predicted output is maximized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach to deal with this problem is to sample my dataset into 7 sub-datasets and train the model in all those 7 datasets. And for a new input, predict the output in each category, and the final category will be where the predicted output is maximal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to know, is there any way to do it in one dataset and automatically predict the category where my output is max? With a single model instead of 7?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I'm using python and scikit learn&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sound like random-forest but not sure that is it ...&#xA;can someone help? Any help will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my dataset with samples collected in each category. My final output is called CGPA and the category label is 'FAC'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DKWXQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DKWXQ.png&quot; alt=&quot;the sample dataset&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="30464" LastEditorUserId="37035" LastEditDate="2017-07-25T05:14:39.847" LastActivityDate="2017-07-25T05:14:39.847" Title="Predicting a Continuous output in a dataset with categories" Tags="|machine-learning|python|classification|scikit-learn|regression|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="21646" PostTypeId="2" ParentId="21645" CreationDate="2017-07-23T20:26:57.797" Score="2" Body="&lt;p&gt;A random forest will work, however standard regression will also work with categorical variables as predictors. You will have to &quot;one-hot&quot; encode your categorical predictors into 6 &quot;dummy&quot; variables (classes-1 = 7-1 = 6). The first dummy variable will encode 0/1 for whether or not the observation is class A, second dummy variable as 0/1 for class B, etc. You only need 6 dummy variables because if all of them are 0 for a given observation, that means the observation is in group 7 (G).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some languages, such as R, the regression command will automatically do this one-hot conversion for you. For python, the pandas package can do this for you with &lt;code&gt;pd.get_dummies(data)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-07-23T20:26:57.797" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21647" PostTypeId="2" ParentId="20572" CreationDate="2017-07-23T21:00:27.897" Score="1" Body="&lt;p&gt;I guess &lt;em&gt;Test &amp;amp; Score&lt;/em&gt; is used to score models on labeled, &lt;em&gt;known&lt;/em&gt; data, e.g. via cross-validation, whereas &lt;em&gt;Predictions&lt;/em&gt; is used to have trained model make predictions on new, &lt;em&gt;unlabeled&lt;/em&gt; data.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastEditorUserId="15527" LastEditDate="2017-07-24T22:48:46.383" LastActivityDate="2017-07-24T22:48:46.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21650" PostTypeId="1" CreationDate="2017-07-24T03:44:17.010" Score="25" ViewCount="13111" Body="&lt;p&gt;I was reading about the solution to this &lt;a href=&quot;https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335&quot; rel=&quot;noreferrer&quot;&gt;OTTO Kaggle challenge&lt;/a&gt; and the first place solution seems to use several transforms for the input data X, for example Log(X+1), sqrt( X + 3/8), etc. Is there a general guideline on when to apply which kind transformations to various classifiers? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do understand the concepts of mean-var and min-max normalization. However, for the above transformations, my guess is that Log and Sqrt are used to compress the dynamic range of the data. And the x-axis shift is just to recenter the data. However, the author chooses to use different methods of normalization for the same input X when feeding into different classifiers. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="35549" LastEditorUserId="35549" LastEditDate="2017-07-24T10:36:52.470" LastActivityDate="2017-11-23T20:46:53.420" Title="Feature Transformation on Input data" Tags="|machine-learning|feature-extraction|feature-scaling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21652" PostTypeId="2" ParentId="761" CreationDate="2017-07-24T05:12:53.400" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://commons.apache.org/proper/commons-math/javadocs/api-3.1/org/apache/commons/math3/stat/clustering/DBSCANClusterer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Java Apache&lt;/a&gt; commons-math does this pretty easily.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;List&amp;lt;Cluster&amp;lt;T&amp;gt;&amp;gt;    cluster(Collection&amp;lt;T&amp;gt; points) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37002" LastEditorUserId="98307" LastEditDate="2020-08-02T13:23:58.763" LastActivityDate="2020-08-02T13:23:58.763" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21655" PostTypeId="1" AcceptedAnswerId="21694" CreationDate="2017-07-24T08:55:54.533" Score="1" ViewCount="3688" Body="&lt;p&gt;I am doing XGBoost classification on a huge data set and its showing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Tree method is automatically selected to be 'approx' for faster speed. to use old behavior(exact greedy algorithm on single machine), set tree_method to 'exact'&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I shift it to be exact?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried &lt;code&gt;clf = xgb(tree_method='exact')&lt;/code&gt; and it returned error!&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="33701" LastEditDate="2017-07-25T23:18:20.170" LastActivityDate="2018-04-24T21:56:23.107" Title="How can I do tree_method ='exact' in XGBoost classifier?" Tags="|machine-learning|python|xgboost|ensemble-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21658" PostTypeId="1" CreationDate="2017-07-24T13:35:42.317" Score="2" ViewCount="179" Body="&lt;p&gt;Assume the existence of a collection of physical parameters and a collection of output variables which may depend on the physical parameters. An example in the training dataset consists of a vector with the actual measurement of the physical parameters, and another vector with the output values which were measured. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my understanding, I can train a neural network such as a MLP to learn the mapping from the measured input parameters to the measured output values. If a model could be learned and assuming the learned relations are due to causality and not some unfortunate correlation, then for an arbitrary input vector the model could predict the vector of output values. Note that &#xA;the relationship between input and output is not a 1:1 map, rather n:1. Different parameter choices may thus lead to the same output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to go one step further and define an interval for each output value. I then want to know how I must set the ideal input parameter values to ensure that the output values are in the corresponding intervals, i.e. how to alter inputs for the output values to be within the ranges. I tried feeding the output values as input and trying to predict the physical parameters, but I have problems due to the ratio input/output. I have 130 physical parameters and only 10 output variables. How can I achieve this with a neural network?&lt;/p&gt;&#xA;" OwnerUserId="32854" LastEditorUserId="32854" LastEditDate="2017-07-25T04:58:54.063" LastActivityDate="2024-03-03T11:03:00.367" Title="Model building with neural networks" Tags="|neural-network|predictive-modeling|tensorflow|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21659" PostTypeId="1" CreationDate="2017-07-24T14:26:50.540" Score="3" ViewCount="107" Body="&lt;p&gt;Let me describe my problem:&#xA;I've got around 1500 items, all of them described by 15 numeric (rational numbers, positive and negative) attributes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an minimalized example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item A:&#xA;- value I: 34&#xA;- value II: 12335&#xA;- value III: -10&#xA;- value IV: 0&#xA;....&#xA;&#xA;Item B:&#xA;- value I: 500&#xA;- value II: -2332&#xA;- value III: 0&#xA;- value IV: 9&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No i have to combine them and find the best combination.&#xA;There is space for about 70 items and each item can be used n times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The performance of each combination can be determinated by a simple math formular using 3 of the attributes, while the other attributes has to match min or max requirements to determinate if a combination is valid or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;because some items have good values for a single attribute, but negative for other attributes, other items have to be taken to compensate these negative aspects&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm looking for an approach/algorithm/tooling to find the best combination or at least get near to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approaches were to (in pseudocode)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//approach A&#xA;while(true)&#xA;  var best = generateRandomCombination()&#xA;  var challenger = generateRandomCombination()&#xA;  best = betterPerformanceIndicator(best, challenger)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;at some point i have to stop it and use the result.&#xA;tried it, let it run for an hour and loggt every single improvement. horrific performance, to much randomness means repetition. and of course it does not came near to a result i'd produce on my own. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//approach B&#xA;var best = generateRandomCombination();&#xA;while(true)&#xA;  var challenger = replaceSingleItemWithRandomOne(best)&#xA;  while (!challenger.isBetter(best) &amp;amp;&amp;amp; untestedItemsLeft()){&#xA;    challenger = replaceWithOtherItem(best);&#xA;  }&#xA;  best = betterPerformanceIndicator(best, challenger)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This approach is some kind of approximation, but i think because of the needed compensations it won't be very efficient to change only one item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So i see two possible ways to improve:&#xA;- perhaps i should give the items some kind of weighting&#xA;- replace more than on item&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions to improve my approaches or replace it with better ones?&#xA;Is this some kind of data-mining problem? What should i learn to deal with it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(if my question is wrong in here, i'm sorry. would be nice if you would redirect me to the correct stack exchange section)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;added information due to comments:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in the beginning i have to fix the container properties, to determinate if there is space for 70, 71 or an other amout of items. And to to determinate the multiplicator of each attribute. As an example: the containers multiplicator of &quot;value I&quot; would be 1,1 - adding Item A and Item B wour result in (34 + 500) * 1,1 = 587,4. &lt;/li&gt;&#xA;&lt;li&gt;So you see, adding the items is simple math (addition and multiplication)&#xA;Concerning speed, my first approaches were written in java using Lists, but i have not tracked the speed of these single operations.&lt;/li&gt;&#xA;&lt;li&gt;the order of the item does not effect the outcome, so it's about building an unordered set, just like Neil asked.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The formula to compute the total value is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;square root of ( (sum of all valueI) multiplied by (sum of all valueII and valueIII) )&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;min and max requiremants are just natural numbers, that the sum of all valueN should (not) reach at least&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37020" LastEditorUserId="37020" LastEditDate="2017-07-25T16:16:06.983" LastActivityDate="2022-10-30T15:00:45.113" Title="big combination with repetition problem" Tags="|algorithms|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="21660" PostTypeId="1" CreationDate="2017-07-24T15:25:26.330" Score="1" ViewCount="1919" Body="&lt;p&gt;I am getting the following error when running a Gaussian Mixture Model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Fitting the mixture model failed because some components have ill-defined empirical covariance (for instance caused by singleton or collapsed samples). Try to decrease the number of components, or increase reg_covar.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The matrix that I am using has a relatively large shape so it will be hard to display on this page, however, here is an overview&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[  6.10086000e+05   1.58787000e+05   0.00000000e+00 ...,   8.00000000e+00&#xA;    0.00000000e+00   0.00000000e+00]&#xA; [  2.36273000e+05   1.48953000e+05   0.00000000e+00 ...,   5.00000000e+00&#xA;    0.00000000e+00   0.00000000e+00]&#xA; [  1.70486000e+05   1.53083000e+05   0.00000000e+00 ...,   3.50000000e+01&#xA;    0.00000000e+00   0.00000000e+00]&#xA; ..., &#xA; [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00&#xA;    0.00000000e+00   0.00000000e+00]&#xA; [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   2.00000000e+01&#xA;    0.00000000e+00   0.00000000e+00]&#xA; [  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,   2.00000000e+00&#xA;    0.00000000e+00   1.00000000e+00]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code snippet for running the GMM:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def perform_gaussian_mixture():&#xA;    colors = [&quot;navy&quot;, &quot;cyan&quot;, &quot;darkorange&quot;, &quot;orchid&quot;, &quot;lime&quot;]&#xA;&#xA;    def make_ellipses(gmm, ax):&#xA;        for n, color in enumerate(colors):&#xA;            if gmm.covariance_type == 'full':&#xA;                covariances = gmm.covariances_[n][:2, :2]&#xA;            elif gmm.covariance_type == 'tied':&#xA;                covariances = gmm.covariances_[:2, :2]&#xA;            elif gmm.covariance_type == 'diag':&#xA;                covariances = np.diag(gmm.covariances_[n][:2])&#xA;            elif gmm.covariance_type == 'spherical':&#xA;                covariances = np.eye(gmm.means_.shape[1]) * gmm.covariances_[n]&#xA;            v, w = np.linalg.eigh(covariances)&#xA;            u = w[0] / np.linalg.norm(w[0])&#xA;            angle = np.arctan2(u[1], u[0])&#xA;            angle = 180 * angle / np.pi  # convert to degrees&#xA;            v = 2. * np.sqrt(2.) * np.sqrt(v)&#xA;            ell = mpl.patches.Ellipse(gmm.means_[n, :2], v[0], v[1],&#xA;                                      180 + angle, color=color)&#xA;            ell.set_clip_box(ax.bbox)&#xA;            ell.set_alpha(0.5)&#xA;            ax.add_artist(ell)&#xA;&#xA;    data = matricize()&#xA;&#xA;    x_train, x_test, y_train, y_test = train_test_split(data[&quot;data&quot;], data[&quot;target&quot;], test_size=0.25, random_state=42)&#xA;&#xA;    n_classes = len(np.unique(data[&quot;target&quot;]))&#xA;&#xA;    # Try GMMs using different types of covariances.&#xA;    estimators = dict((cov_type, mixture.GaussianMixture(n_components=n_classes,&#xA;                                                         covariance_type=cov_type, max_iter=20, random_state=0))&#xA;                      for cov_type in [&quot;full&quot;])&#xA;&#xA;    n_estimators = len(estimators)&#xA;&#xA;    plt.figure(figsize=(3 * n_estimators // 2, 6))&#xA;    plt.subplots_adjust(bottom=.01, top=0.95, hspace=.15, wspace=.05,&#xA;                        left=.01, right=.99)&#xA;&#xA;    for index, (name, estimator) in enumerate(estimators.items()):&#xA;&#xA;        # Since we have class labels for the training data, we can&#xA;        # initialize the GMM parameters in a supervised manner.&#xA;        estimator.means_init = np.array([x_train[np.where(y_train == i)].mean(axis=0)&#xA;                                         for i in range(n_classes)])&#xA;&#xA;        # Train the other parameters using the EM algorithm.&#xA;        estimator.fit(x_train)&#xA;&#xA;        h = plt.subplot(2, n_estimators // 2, index + 1)&#xA;        make_ellipses(estimator, h)&#xA;&#xA;        for n, color in enumerate(colors):&#xA;            d = data[&quot;data&quot;][np.where(data[&quot;target&quot;] == n)]&#xA;            plt.scatter(d[:, 0], d[:, 1], s=0.8, color=color,&#xA;                        label=data[&quot;target_names&quot;][n])&#xA;        # Plot the test data with crosses&#xA;        for n, color in enumerate(colors):&#xA;            d = x_test[np.where(y_test == n)]&#xA;            plt.scatter(d[:, 0], d[:, 1], marker='x', color=color)&#xA;&#xA;        y_train_pred = estimator.predict(x_train)&#xA;        train_accuracy = np.mean(y_train_pred.ravel() == y_train.ravel()) * 100&#xA;        plt.text(0.05, 0.9, 'Train accuracy: %.1f' % train_accuracy,&#xA;                 transform=h.transAxes)&#xA;&#xA;        y_test_pred = estimator.predict(x_test)&#xA;        test_accuracy = np.mean(y_test_pred.ravel() == y_test.ravel()) * 100&#xA;        plt.text(0.05, 0.8, 'Test accuracy: %.1f' % test_accuracy,&#xA;                 transform=h.transAxes)&#xA;&#xA;        plt.xticks(())&#xA;        plt.yticks(())&#xA;        plt.title(name)&#xA;&#xA;    plt.legend(scatterpoints=1, loc='lower right', prop=dict(size=12))&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please let me know if more information is required.&lt;/p&gt;&#xA;" OwnerUserId="37024" LastActivityDate="2023-12-18T11:05:30.683" Title="ValueError ill-defined empirical covariance when running a Gaussian Mixture Model" Tags="|machine-learning|scikit-learn|gaussian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21662" PostTypeId="2" ParentId="10612" CreationDate="2017-07-24T17:40:47.973" Score="14" Body="&lt;p&gt;The title of this question is a separate question to its text so I will answer both separately (given that one leads into the other).&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I infer unseen sentences:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;# ... trained model stored in var model&#xA;list_of_words = [&amp;quot;this&amp;quot;, &amp;quot;is&amp;quot;, &amp;quot;a&amp;quot;, &amp;quot;new&amp;quot;,&amp;quot;unseen&amp;quot;, &amp;quot;sentence&amp;quot;]&#xA;inferred_embedding = model.infer_vector(list_of_words)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How does this work ? As per the original paper (&lt;a href=&quot;https://cs.stanford.edu/%7Equocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&lt;/a&gt;) we have two weight matrices initialized at random &lt;span class=&quot;math-container&quot;&gt;$W \in \mathbb{R}^{N\times P}$&lt;/span&gt; which is the same matrix from Word2Vec in which every column or row maps to a word vector and &lt;span class=&quot;math-container&quot;&gt;$D \in \mathbb{R}^{M \times R}$&lt;/span&gt; which is the document matrix where each column or row maps to a sentence/document. During training a softmax classifier of fixed window &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt; size moves (in a moving window like fashion) minimizing the following log likelihood (multi-class cross-entropy):&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\frac{1}{M}\sum_{i=1}^{M}\frac{1}{|D_{i}|}\sum_{t=k}^{|D_{i-1}|-k}log(p(w_{t}^{i} | w_{t-k}^{i}, ..., w_{t+k}^{i},D_{i} ))&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Where &lt;span class=&quot;math-container&quot;&gt;$D_{i}$&lt;/span&gt; corresponds to the vector representing the &lt;span class=&quot;math-container&quot;&gt;$i^{th}$&lt;/span&gt; sentence, &lt;span class=&quot;math-container&quot;&gt;$|D_{i}|$&lt;/span&gt; its number of words in that document and &lt;span class=&quot;math-container&quot;&gt;$w_{t}^{i}$&lt;/span&gt; is the &lt;span class=&quot;math-container&quot;&gt;$t^{th}$&lt;/span&gt; word in the &lt;span class=&quot;math-container&quot;&gt;$i^{th}$&lt;/span&gt; document. All back-propagation remembers is the document we are currently moving our windowed softmax over and only updates that row in matrix &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; along with the words in that window.&lt;/p&gt;&#xA;&lt;p&gt;Furthermore when we want to infer something not in the training set we fix &lt;span class=&quot;math-container&quot;&gt;$W$&lt;/span&gt; so that it is not updated and augment matrix &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; with the new randomly initialised row and just train for several iterations (with the new row holding the embedding for the inferred vector). This leads into question 2.&lt;/p&gt;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Can we infer that a possibly unseen sentence exactly corresponds to a sentence in the training set space (and therefore to its label)?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The short answer is no and this is not what Doc2Vec is for. Because of the random initialization + the complexity of convergence and training your inferred vector will never be exactly the same as its corresponding vector in &lt;span class=&quot;math-container&quot;&gt;$D$&lt;/span&gt; (i.e. if it had been included in the training set). This is why Gensim has not built in a function to support this. However, given that the model has been well trained, these two vectors should be arbitrarily close to each other, thus you can conclude that they are extremely similar.&lt;/p&gt;&#xA;&lt;p&gt;Even fixing the random seed may not work; there are so many other variables that can affect its convergence please see first answer on &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/issues/374&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/RaRe-Technologies/gensim/issues/374&lt;/a&gt; .&lt;/p&gt;&#xA;&lt;p&gt;In any case, you can find the most similar label in your data set to an inferred sentence just by iterating over your training set and comparing the similarity to the inferred vector. But why would you want to do an exact match to something in the training set? that's what regular expressions are for. The purpose of these document embeddings is for supervised or unsupervised learning tasks (i.e. classification and clustering).&lt;/p&gt;&#xA;" OwnerUserId="32650" LastEditorUserId="-1" LastEditDate="2021-07-12T06:51:22.930" LastActivityDate="2021-07-12T06:51:22.930" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21665" PostTypeId="1" CreationDate="2017-07-24T19:14:30.023" Score="1" ViewCount="703" Body="&lt;p&gt;In the Config file there are some parameters which I don't understand properly . I will mention them here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;first_stage_features_stride&lt;/strong&gt; - Is this the ration of &lt;strong&gt;input/output&lt;/strong&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;height_stride&lt;/strong&gt; (In the first_stage_anchor_generator) - What is this ?&lt;/p&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2017-07-24T19:14:30.023" Title="What are the model configuration parameters in tensorflow object detection API?" Tags="|tensorflow|object-recognition|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21666" PostTypeId="1" AcceptedAnswerId="21675" CreationDate="2017-07-25T02:35:31.657" Score="3" ViewCount="1080" Body="&lt;p&gt;This is specific to the generative adversarial network (GAN) proposed in A. Makhzani et al. &quot;Adversarial Autoencoders&quot;. In a traditional GAN, the discriminator is trained to distinguish real samples in $p(x)$ from fake generated samples output by the generator. On the other hand, the variational autoencoder uses discriminators over the latent codes $z$ instead of the original samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I cannot understand is how the discriminator is able to distinguish between the latent code's prior distribution $p(z)$ and the posterior distribution $q(z)$. What is the reasoning behind the discriminator model attempting to discriminate something that is not a sample $x$, as in the traditional GAN? And should this discriminator be trained before the rest of the autoencoder?&lt;/p&gt;&#xA;" OwnerUserId="32882" LastEditorUserId="17538" LastEditDate="2017-07-25T16:30:26.890" LastActivityDate="2017-07-25T16:30:26.890" Title="What is the purpose of the discriminator in an adversarial autoencoder?" Tags="|unsupervised-learning|autoencoder|gan|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21667" PostTypeId="1" CreationDate="2017-07-25T04:21:00.600" Score="1" ViewCount="70" Body="&lt;p&gt;I have a binomial outcome that I am trying to predict using a gbm in h2o.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have set quite a low min_rows value for each node and it appears to be overfitting.  See plots below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/nq25n.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I group the results into percentiles and plot Observed vs Predicted I get good results for Validation but Training shows a higher proportion of observed outcomes in the upper percentiles and lower observed outcomes in the lower percentiles.  See images below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/pAO0H.png&quot; alt=&quot;Training&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have thought training would have predicted more accurately and the Validation and Hold Out plots would have been more inaccurate.  Is this the expected result and why does it occur?&lt;/p&gt;&#xA;" OwnerUserId="37044" LastActivityDate="2017-07-25T04:21:00.600" Title="Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome" Tags="|machine-learning|predictive-modeling|overfitting|gbm|binary|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21668" PostTypeId="1" AcceptedAnswerId="21669" CreationDate="2017-07-25T06:05:06.517" Score="-1" ViewCount="238" Body="&lt;p&gt;I have data set of client profile and mutual funds , now the problem is there is huge numbers of different mutual funds available and based on history I can see for certain type of client profile more than 20 different funds have been suggested ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have no idea which algorithm should I use to come up with most recommended fund for certain client profile ?  &lt;/p&gt;&#xA;" OwnerUserId="33137" LastEditorUserId="33701" LastEditDate="2017-07-25T12:59:54.510" LastActivityDate="2017-07-25T12:59:54.510" Title="Data not consistent" Tags="|machine-learning|predictive-modeling|recommender-system|" AnswerCount="1" CommentCount="2" ClosedDate="2017-09-02T09:32:14.443" ContentLicense="CC BY-SA 3.0" />
  <row Id="21669" PostTypeId="2" ParentId="21668" CreationDate="2017-07-25T07:06:03.293" Score="0" Body="&lt;p&gt;I assume you want to build a Recommendation Engine which will recommend mutual funds to the clients based on their profile. Recommender System is a special class of Machine Learning and make use of Supervised/Unsupervised Machine Learning techniques. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Wikipedia, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A recommender system or a recommendation system (sometimes replacing&#xA;  &quot;system&quot; with a synonym such as platform or engine) is a subclass of&#xA;  information filtering system that seeks to predict the &quot;rating&quot; or&#xA;  &quot;preference&quot; that a user would give to an item.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So essentially, the item here in your problem statement is the mutual fund of which you need to predict the preference a client would give. Higher rating/preference of a Mutual Fund, let's call it X for a client A will mean X should be recommended to A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, how do you give rating/preference to a list of items? Recommender Systems uses a number of approaches listed below:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Popularity Based Approach&lt;/li&gt;&#xA;&lt;li&gt;Content Based Approach&lt;/li&gt;&#xA;&lt;li&gt;Collaborative Filtering Based Approach which composes of Nearest Neighbor and Matrix Factorization&lt;/li&gt;&#xA;&lt;li&gt;Hybrid Approach&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Popularity Based Approach&lt;/strong&gt; - Popularity Based Approach will recommend items, in your case, mutual funds, based on the total count of purchases. It will rank the mutual fund with maximum purchase in the past at the top. The item at the top will be recommended to every user/client. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Advantages of Popularity Based Approach is that it is very easy to&#xA;implement. The approach is used to list out the most popular items in&#xA;e-commerce websites, online news (Google News). It can use context (for example, most popular products at this time of day). It can use user and item features (for example, most popular products with feature F1, most popular products among user with feature UF1). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Disadvantage of Popularity Based Approach: No personalized&lt;br&gt;&#xA;recommendation. Every one is recommended the same item.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Content Based Approach&lt;/strong&gt; - In Content Based Approach, features of both the item as well as the user/client is used to offer personalized recommendation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Wikipedia,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Content-based filtering methods are based on a description of the item&#xA;  and a profile of the user’s preferences.  In a content-based&#xA;  recommender system, keywords are used to describe the items and a user&#xA;  profile is built to indicate the type of item this user likes. In&#xA;  other words, these algorithms try to recommend items that are similar&#xA;  to those that a user liked in the past.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, let's say if one of the client with features UF1, UF2 purchased a mutual fund X with a set of features F1, F2 and F3 in the past. New mutual fund products, lets say Y and Z will only be recommended to this client if their features matches with that of F1, F2 and F3. &#xA;It is also called Classification Based Approach because you are feeding in list of features of products and user into a classifier to predict what item should be recommended. To build a Content Based Approach model, you can use Machine Learning models like Naive Bayes Classifier or a Decision Tree. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Advantage of Content Based Approach over Popularity Based Approach is&#xA;that it gives personalized recommendation. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Disadvantage: It is not scalable. It is difficult to collect high&lt;br&gt;&#xA;quality information on user and product.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Collaborative Filtering Approach&lt;/strong&gt; - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to Wikipedia,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Collaborative filtering methods are based on collecting and analyzing&#xA;  a large amount of information on users’ behaviors, activities or&#xA;  preferences and predicting what users will like based on their&#xA;  similarity to other users.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Without going into technical jargons, Collaborative Filtering Approach is of two types -- &lt;strong&gt;User-based Collaborative Filtering&lt;/strong&gt; &amp;amp; &lt;strong&gt;Item-based Collaborative Filtering&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In User-based Collaborative Filtering, let's say user/client A bought two items X and Y and user/client B bought the same items X and Y. So these two users are similar to each other based on their purchase history. Now if the user/client B purchases a third item Z, Z will also be recommended to user/client A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Item-based Collaborative Filtering, let's say a user/client A bought items W, Y and Z. User/client B bought W and Y. User/client C bought only Y. &#xA;What will be recommended to user/client C? Answer is item W because item W was co-purchased with item Y by user/client A and B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These comes under Nearest Neighbor techniques of Collaborative Filtering because we are trying to classify a user or an item by determining its nearest neighbor. An algorithm known as kNN (k-nearest Neighbor) is used to do the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is one another technique known as Matrix Factorization as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hybrid Based Approach&lt;/strong&gt; - Hybrid Based Approach, as name says, is a combination of Content Based Approach and Collaborative Filtering Approach. It is used largely used by Netflix and Amazon to give best recommendations. &lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-07-25T07:06:03.293" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="21672" PostTypeId="2" ParentId="20572" CreationDate="2017-07-25T08:36:35.533" Score="2" Body="&lt;p&gt;Perhaps I can elaborate a bit on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test&amp;amp;Score is used for evaluating a model. You provide T&amp;amp;S with training data and the learner (e.g. Random Forest) and then the widget performs 10-fold cross-validation on training data, leaving out 1/10 of a data for testing. Results of all 10 fold are then combined into a single evaluation result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predictions, however, does not perform cross-validation. It doesn't test the data provided. Instead, Random Forest passes a model to Prediction, not a Learner. A model is built on train data. Predictions widget does not consider the real class, it uses the model from RF and applies it to the (new) data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are two different procedures, thus you are getting different results.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2017-07-25T08:36:35.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21673" PostTypeId="2" ParentId="17479" CreationDate="2017-07-25T08:43:41.377" Score="2" Body="&lt;p&gt;As r.kfr said: Import Images works with folders, not individual images. Place all images in a folder and select it for import.&#xA;Note that if you have images in different folders, Orange will consider each folder name as a class label for images.&lt;/p&gt;&#xA;&lt;p&gt;Alternatively, you can create an Excel/CSV file and define the path to the image. Mark it with &lt;em&gt;meta type=image&lt;/em&gt; in the third header row. Here is how to define the &lt;a href=&quot;http://orange-visual-programming.readthedocs.io/loading-your-data/index.html#header-with-attribute-type-information&quot; rel=&quot;nofollow noreferrer&quot;&gt;header&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastEditorUserId="85045" LastEditDate="2021-09-21T05:17:07.863" LastActivityDate="2021-09-21T05:17:07.863" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21675" PostTypeId="2" ParentId="21666" CreationDate="2017-07-25T10:39:23.463" Score="2" Body="&lt;p&gt;The purpose of having a prior distribution $p(z)$ in any generative adversarial network is to be able to smoothly match a latent code $z$ in a known distribution to an input $x$ in the domain and vice versa. The encoder of a simple autoencoder, without any additional measures other than the in typical pipeline&#xA;$$x \rightarrow E \rightarrow z \rightarrow D \rightarrow x'$$&#xA;would only require $x$ to approach $x' = D(E(x))$, and for that purpose the decoder may simply learn to reconstruct $x$ regardless of the distribution obtained from $E$. This means that $p(z)$ can be very irregular, making generation of new samples less feasible. Even with slight changes to the bottleneck vector, we cannot be sure that the encoder would ever be able to produce that code with any $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In an adversarial autoencoder (AAE) however, the encoder's job is two-fold: it encodes inputs in $p(x)$ to the respective code in $q(z)$ so that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it minimizes the reconstruction cost $f(x, D(E(x)))$ (where $f$ is a distance metric between samples, such as the mean squared error);&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;while&lt;/em&gt; it learns to adapt $q(z)$ to the prior distribution $p(x)$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The latter task is effectively enforced because the discriminator receives:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;positive feedback from codes in $p(z)$;&lt;/li&gt;&#xA;&lt;li&gt;and negative feedback from codes in $q(z)$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Even if the discriminator might not know anything about any of the two distributions at the beginning, it is only a matter of making enough iterations before it does. The ideal encoder will manage to trick the discriminator to the point of having approximately 50% accuracy in the discrimination process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that $p(x)$ may not be just a Gaussian or uniform distribution (as in, some sort of noise). &#xA;Quoting from Goodfellow's Deep Learning book (&lt;a href=&quot;http://www.deeplearningbook.org/contents/generative_models.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;chapter 20&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When developing generative models, we often wish to extend neural networks to implement stochastic transformations of $x$. One straightforward way to do this is to augment the neural network with extra inputs $z$ that are sampled from some simple probability distribution, such as a uniform or Gaussian distribution. The neural network can then continue to perform deterministic computation internally, but the function $f(x, z)$ will appear stochastic to an observer who does not have access to $z$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although denoising autoencoders rely on this aspect to learn a model that ignores noise from a sample, the same paper on AAEs (section 2.3) shows that combining noise with a one-hot encoded vector of classes can be used to incorporate label information about the sample. This information is only provided to the discriminator, but it still influences how the encoder produces $q(z)$.&lt;/p&gt;&#xA;" OwnerUserId="17538" LastEditorUserId="17538" LastEditDate="2017-07-25T14:56:40.450" LastActivityDate="2017-07-25T14:56:40.450" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21676" PostTypeId="1" CreationDate="2017-07-25T10:58:28.337" Score="3" ViewCount="107" Body="&lt;p&gt;I need help in the analyse of a categorization problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a set of &lt;strong&gt;dates&lt;/strong&gt; (small set: 20 elements maximum), I would like to group dates which are equally distributed (with a tolerance). &#xA;It can be, for instance, monthly or weekly separated dates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example. Given this repartition : &lt;a href=&quot;https://i.stack.imgur.com/AVtyf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AVtyf.png&quot; alt=&quot;Repartition of the dates&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to categorize into these two groups: &lt;a href=&quot;https://i.stack.imgur.com/jhZ5q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jhZ5q.png&quot; alt=&quot;Groups of data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I am a developer, not a data scientist. I have an intuition that it should be possible to do a kind of regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no clue how to analyse this problem. Can you help me with that, please ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I have already seen this thread (&lt;a href=&quot;https://datascience.stackexchange.com/questions/12115/recurring-events-finding-in-a-time-series&quot;&gt;Recurring events - finding in a time series&lt;/a&gt;) but I have not helped me.&lt;/p&gt;&#xA;" OwnerUserId="37054" LastActivityDate="2024-01-28T10:02:05.250" Title="Find recurrent dates in a small set (and get rid of non recurrent ones)" Tags="|classification|time-series|regression|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21677" PostTypeId="1" AcceptedAnswerId="21683" CreationDate="2017-07-25T11:55:54.600" Score="1" ViewCount="3420" Body="&lt;p&gt;I am trying to build a decision tree using python and sklearn DecisionTreeClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the data_type used for splitting the tree  is Boolean(let it be x).However the tree that is generated contains comparisons like x&amp;lt;=0.5 . &#xA;This does not make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody suggest how to use boolean values in Decision trees.&lt;/p&gt;&#xA;" OwnerUserId="37063" LastEditorUserId="33701" LastEditDate="2017-07-25T12:18:31.423" LastActivityDate="2017-07-25T14:43:36.630" Title="How to use boolean data in DecisionTreeClassifier in sklearn?" Tags="|python|data-mining|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21678" PostTypeId="1" CreationDate="2017-07-25T12:15:35.197" Score="0" ViewCount="1312" Body="&lt;p&gt;I want to train an Artificial Neural Network on some data however some of the fields are optional, dependent upon the values contained in other fields. The data in these optional fields is missing, but correctly so. I am wondering how I should factor this into my neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like what I might need to do is turn certain inputs on/off dependent upon the values contained in other inputs but then I am wondering how this would affect weighting in the hidden layer. Could I have some kind of conditional hidden layer or something for only when these optional values are 'activated'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the standard approach to this? It seems surprisingly hard to find any material on this online!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="37059" LastEditDate="2017-07-25T12:21:38.823" LastActivityDate="2017-10-23T07:52:26.633" Title="Dealing with optional data in an Artificial Neural Network" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21679" PostTypeId="1" AcceptedAnswerId="21688" CreationDate="2017-07-25T12:26:32.043" Score="1" ViewCount="6052" Body="&lt;p&gt;I'm looking for a rather large amount of pdf files for testing my text processing program. Tried looking for an open site to get like some thousand pdfs, but wasn't able to find anything. I don't really know if that is the right place to ask (probably not) but maybe one has a good tip for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="37022" LastEditorUserId="33701" LastEditDate="2017-07-25T12:43:26.570" LastActivityDate="2017-07-25T18:14:45.507" Title="Dataset - Sample pdfs for text processing?" Tags="|machine-learning|nlp|dataset|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21680" PostTypeId="2" ParentId="21678" CreationDate="2017-07-25T12:29:48.350" Score="1" Body="&lt;p&gt;Several options, depending on your algorithms: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Default to 0&lt;/li&gt;&#xA;&lt;li&gt;Default to 1&lt;/li&gt;&#xA;&lt;li&gt;Default to Average&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37022" LastActivityDate="2017-07-25T12:29:48.350" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="21681" PostTypeId="2" ParentId="21676" CreationDate="2017-07-25T12:53:35.837" Score="0" Body="&lt;p&gt;If it is a categorization problem then you should look for a classification algorithm, not a regression technique. The simplest classification algorithm is Logistic Regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But by the looks of it, seems like you do not have a labelled data-set and if that's the case you should look for Clustering techniques. Clustering is a part of Unsupervised Learning Technique in ML which create clusters or groups of similar data points. &lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-07-25T12:53:35.837" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21683" PostTypeId="2" ParentId="21677" CreationDate="2017-07-25T14:43:36.630" Score="2" Body="&lt;p&gt;Why would this not make sense? It tries to create a split with maximal separation between signal and background. Any float &gt;0 and &amp;lt; 1 will result in the same split. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-07-25T14:43:36.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21685" PostTypeId="2" ParentId="12498" CreationDate="2017-07-25T17:42:49.940" Score="1" Body="&lt;p&gt;One solution is binning the data into groups. For example, create three groups - &quot;High&quot;, &quot;Medium&quot;, &quot;Low&quot;. The numerical splits groups can be handpicked to emphasize the signal in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another solution is to threshold the data and not show data above or below the threshold.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-25T17:42:49.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21688" PostTypeId="2" ParentId="21679" CreationDate="2017-07-25T18:14:45.507" Score="4" Body="&lt;p&gt;A GitHub repo with ~1,000 pdfs is &lt;a href=&quot;https://github.com/tpn/pdfs&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another GitHub repo has a corpus of pdf examples, including &lt;a href=&quot;https://github.com/openpreserve/format-corpus/tree/master/pdfCabinetOfHorrors&quot; rel=&quot;nofollow noreferrer&quot;&gt;edge-cases&lt;/a&gt;, is &lt;a href=&quot;https://github.com/openpreserve/format-corpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-25T18:14:45.507" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21689" PostTypeId="2" ParentId="19840" CreationDate="2017-07-25T19:07:59.217" Score="1" Body="&lt;p&gt;You can use the library &lt;a href=&quot;https://github.com/jeweinberg/Pandas-MICE&quot; rel=&quot;nofollow noreferrer&quot;&gt;HERE&lt;/a&gt;. You are able to impute the values of your dataset using linear models with an Sklearn type interface.&lt;/p&gt;&#xA;" OwnerUserId="37086" LastEditorUserId="37086" LastEditDate="2017-07-27T20:12:39.217" LastActivityDate="2017-07-27T20:12:39.217" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21693" PostTypeId="1" CreationDate="2017-07-25T19:48:51.740" Score="10" ViewCount="10879" Body="&lt;p&gt;When using VC-dimensions to estimate the capability of a binary classifier, you can find 3 points in R&lt;sup&gt;2&lt;/sup&gt; that can be &lt;a href=&quot;https://en.wikipedia.org/wiki/Shattered_set&quot; rel=&quot;noreferrer&quot;&gt;shattered&lt;/a&gt;, e.g.:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/z5DAn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/z5DAn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you can &lt;strong&gt;not&lt;/strong&gt; shatter &lt;strong&gt;any&lt;/strong&gt; 4 points with a circle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is stated in &lt;a href=&quot;http://www.cs.ubc.ca/~murphyk/Teaching/CS340-Fall06/lectures/L4pac.pdf&quot; rel=&quot;noreferrer&quot;&gt;these lecture notes&lt;/a&gt;. Can anyone give me an intuitive explanation?&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-08-13T23:23:27.697" LastActivityDate="2018-08-13T23:23:27.697" Title="With regards to VC-dimension, why can you shatter 3 points with circles but not 4 points?" Tags="|machine-learning|classification|vc-theory|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="21694" PostTypeId="2" ParentId="21655" CreationDate="2017-07-25T19:53:48.677" Score="2" Body="&lt;p&gt;Indeed &lt;code&gt;tree_method&lt;/code&gt; is a parameter for Tree Booster. There are 4 choices, namely, &lt;code&gt;auto&lt;/code&gt;, &lt;code&gt;exact&lt;/code&gt;, &lt;code&gt;approx&lt;/code&gt; and &lt;code&gt;hist&lt;/code&gt;. The default is set to &lt;code&gt;auto&lt;/code&gt; which  heuristically chooses a faster algorithm based on the size of your dataset. Approximate algorithm (&lt;code&gt;approx&lt;/code&gt;) is chosen in your case since your dataset is huge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To change the tree construction algorithm, you have to pass &lt;code&gt;tree_method&lt;/code&gt; in the &lt;code&gt;train&lt;/code&gt; function as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xgb&#xA;...&#xA;parameters = {'tree_method': 'exact'} #You can add other parameters as well&#xA;model = xgb.train(parameters, x_train, num_round) #num_round is the number of rounds for boosting&#xA;prediction = model.predict(x_test)&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="33701" LastActivityDate="2017-07-25T19:53:48.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21695" PostTypeId="2" ParentId="17830" CreationDate="2017-07-25T20:36:23.583" Score="1" Body="&lt;p&gt;Based on the data seen in your graphs, according to me, this is a time series modelling problem and a model like ARIMA (Autoregressive Integrated Moving Model Average) would be a better fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you mentioned that you're starting (you've probably done a lot by now) here's a tutorial by Dr Jason Brownlee on implementing ARIMA on Python: &#xA;&lt;a href=&quot;https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ARIMA for time series forecasting with Python&lt;/a&gt;. This is for in sample prediction, where you want to predict values for which you already have to test the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For out of sample Prediction see: &lt;a href=&quot;https://machinelearningmastery.com/make-sample-forecasts-arima-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time Series ARIMA model for forecasting with Python&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37088" LastEditorUserId="29575" LastEditDate="2018-02-05T01:24:33.323" LastActivityDate="2018-02-05T01:24:33.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21696" PostTypeId="2" ParentId="9762" CreationDate="2017-07-25T20:52:35.583" Score="0" Body="&lt;p&gt;In some cases, neural networks trained with the back-propagation algorithm have shown better results than time series models. The back propagation algorithm searches for weight values that minimize the total error of the network over the set of training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may take a look at these links which describes what I've said in more detail : &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Frank M&#xA;Thiesing and Oliver Vornberger   &lt;a href=&quot;http://www2.inf.uos.de/papers_pdf/icnn_97.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Sales Forecasting Using Neural Networks&quot;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Backpropagation Algorithm&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37088" LastActivityDate="2017-07-25T20:52:35.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21699" PostTypeId="2" ParentId="11091" CreationDate="2017-07-25T22:35:30.433" Score="0" Body="&lt;p&gt;Your question explicitly states that your are only looking for multiple cars rather than multiple objects, so the answer is in the question. You are not looking for multiple objects, rather multiple occurrences of the same object.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Providing you trained the system well enough to recognise both types of car then they should both be detected using standard cascading filter approaches ... This is like asking how can I detect 2 faces in one photograph?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were looking for a car and a monkey then the situation is very different and using common approaches with tools like open CV you would generally train 2 classifiers (one for cars and one for monkeys) the iterate over the image twice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more different classes of object you want to detect the more classifiers and iterations you would need.&lt;/p&gt;&#xA;" OwnerUserId="37096" LastActivityDate="2017-07-25T22:35:30.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21701" PostTypeId="1" AcceptedAnswerId="21705" CreationDate="2017-07-26T04:14:32.640" Score="0" ViewCount="3169" Body="&lt;p&gt;It is a very basic question, but cannot find a satisfactory answer to. When we do logistic regression, what are the inputs? Suppose we have a dataset of students giving number of hours each student spent studying, and the end result of whether they passed or failed. The wikipedia article on LR goes to do a fitting with the logistic curve based on probability as a function of number of hours. But how do I get the probabilities if all I have is whether they passed or failed? &lt;/p&gt;&#xA;" OwnerUserId="37045" LastEditorUserId="33701" LastEditDate="2017-07-26T10:19:12.770" LastActivityDate="2019-04-26T03:09:29.597" Title="What are the inputs to a logistic regression? Probability or trial result?" Tags="|machine-learning|classification|logistic-regression|probability|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21702" PostTypeId="2" ParentId="21701" CreationDate="2017-07-26T04:57:20.370" Score="0" Body="&lt;p&gt;I think this is your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Q1: What to give to logistic regression?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Q2: I just want to predict whether a student pass or not, I don't care about anything else?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is able to handle categorical and continuous variables. In your example, number of hours for each student in your training set is your inputs. Of course, you'll also need a binary response variable (pass or failed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Q2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression is not a classifier, the model gives you fitted probabilities conditional to the number of hours. You can set a threshold to your model (many posts exist, please search). Or you can apply a classifier such as &lt;code&gt;linear discriminant analysis&lt;/code&gt; and many others.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-07-26T04:57:20.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21705" PostTypeId="2" ParentId="21701" CreationDate="2017-07-26T07:27:37.223" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;From Wiki: The graph shows the probability of passing the exam versus the number of hours studying, with the logistic regression curve fitted to the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The probability curve is the output of the LR, not the input. Typically during training, the output class (or target class) will be discrete class labels with 1 or 0. During inferencing, the output will be a continuous value between 0 and 1. To generate the probability curve, just feed in different values of &quot;hours studying&quot; into the trained model.&lt;/p&gt;&#xA;" OwnerUserId="35549" LastEditorUserId="35549" LastEditDate="2019-04-26T03:09:29.597" LastActivityDate="2019-04-26T03:09:29.597" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21706" PostTypeId="2" ParentId="10529" CreationDate="2017-07-26T07:46:14.037" Score="1" Body="&lt;p&gt;These are some suggestions you can try:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Just use 1 LSTM layer instead of 2, there might be less problem of vanishing gradient for shallower networks. In fact the dense layer of 4 is optional, since return_sequences = True. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find a way to normalize the input in the range 0 to 1. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try different optimizers like SGD or Adam.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reduce your learning rate, the default learning rate might be too large for an input dimension of 4. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The loss should not be categorical_crossentropy, since this is a regression problem. The loss should mean squared error. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In order of importance 4 &gt; 5 &gt; 3 &gt; 1 &gt; 2 (i.e point 4 is most important)&lt;/p&gt;&#xA;" OwnerUserId="35549" LastActivityDate="2017-07-26T07:46:14.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21707" PostTypeId="1" AcceptedAnswerId="21711" CreationDate="2017-07-26T08:03:16.380" Score="-1" ViewCount="276" Body="&lt;p&gt;Not sure if this is the right forum, but currently i have a dataset which contains a list of TV shows. Each record contains pricing between competitors (price in provider 1. Example: Itunes) TV show cover image, synopsis, country of origin, language, etc. Looking for ideas what project is suggested that i can prototype it, i want to learn ML and this may be a useful dataset.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2017-07-26T08:17:03.387" LastActivityDate="2017-07-26T08:54:47.847" Title="ML project ideas for dataset" Tags="|data-mining|dataset|text|" AnswerCount="1" CommentCount="0" ClosedDate="2017-07-26T10:18:57.650" ContentLicense="CC BY-SA 3.0" />
  <row Id="21710" PostTypeId="2" ParentId="12237" CreationDate="2017-07-26T08:19:31.450" Score="-1" Body="&lt;p&gt;if the data type of your &quot;tz&quot; column is a string, then you can do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.select(from_utc_timestamp(df.start_time,tz).alias('start_time')).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37108" LastActivityDate="2017-07-26T08:19:31.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21711" PostTypeId="2" ParentId="21707" CreationDate="2017-07-26T08:43:08.283" Score="1" Body="&lt;p&gt;If this is your first ML project, you should try to predict one feature with the other features; i.e., see if highest bidder can be predicted based on cover image, synopsis, country of origin, etc. For this you can use a basic NN as a baseline, and keep adding layers until it's powerful enough. To choose the feature to be predicted, you can make a correlation matrix or use random forests to make sure there are some relationships within the data and you're not simply trying to predict an uncorrelated variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A little bit more difficult would be to predict the precise pricing a provider would give a movie based on the price it's given other movies (of course, without looking at the other competitors' pricing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hardest, I think, would be to use the cover image as input to a CNN in order to predict the genre of movie, country of origin, etc. &lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="37109" LastEditDate="2017-07-26T08:54:47.847" LastActivityDate="2017-07-26T08:54:47.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21713" PostTypeId="1" CreationDate="2017-07-26T09:17:42.760" Score="2" ViewCount="1488" Body="&lt;p&gt;I am trying to construct a machine learning model that predicts the difference in price from tomorrow to the day after tomorrow, using yesterday's OHLCV (open, high, low, close, volume).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My models (LSTM and DNNs), however, give me very poor outputs. I think it is because the data needs to be reformatted in some way, shape, or form (i.e. normalized, log-pct change, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to manipulate the input? Should I use a sci-kit 'scaler', or any other method to normalize the data? If so, what range should the scaler be within?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the output? If I want the output to be linear (continuous, non-softmax value), should I manipulate the output data as well to be scaled, and then unscale the predicted output to match the original labels? Does it matter?&lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="37109" LastEditDate="2017-07-27T04:19:35.293" LastActivityDate="2017-09-14T21:29:10.250" Title="Stock Price Data Manipulation for LSTM" Tags="|machine-learning|python|neural-network|lstm|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21716" PostTypeId="1" AcceptedAnswerId="21727" CreationDate="2017-07-26T11:43:23.170" Score="1" ViewCount="805" Body="&lt;p&gt;In the context of discrete choice models, what difference does it make in segmenting my sample based on a particular &quot;criteria&quot; and study the effects of explanatory variables on each segment VERSUS Just adding the &quot;criteria&quot; as another explanatory variable in the unsegmented population&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, I want to build a commuter mode choice model using MNL, what difference does it make when I split my population via gender and study the effects of various explanatory variables VERSUS adding gender as another explanatory variable in the entire unsegmented population.&lt;/p&gt;&#xA;" OwnerUserId="37116" LastActivityDate="2017-07-26T18:01:06.210" Title="Difference between segmentation and effect of explanatory variables" Tags="|logistic-regression|methodology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21719" PostTypeId="1" CreationDate="2017-07-26T15:56:01.783" Score="2" ViewCount="3138" Body="&lt;p&gt;I have a situation where I have to cluster word2vec vectors (200 length dimension vectors on a very large corpus). I decided to use Density based clustering (DBSCAN, HDBSCAN) because my dataset is very high in noise, and I do not want it to be part of my clusters. My knowledge on Cosine distance is limited but I find that Density based clustering algorithms do not have a direct implementation using cosine distance (&lt;code&gt;pairwise_distance&lt;/code&gt; calculation is too memory intensive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question here is can I normalize the Word2vec vectors using L2 normalization using: &lt;code&gt;norm_data = normalize(vector_array, norm='l2')&lt;/code&gt; from the sklearn library to normalize these vectors, and then use euclidean distance over the normalized vectors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest any other better technique to cluster word vectors when there is noise in the dataset?&lt;/p&gt;&#xA;" OwnerUserId="37128" LastEditorUserId="52196" LastEditDate="2019-07-04T14:38:16.647" LastActivityDate="2019-07-04T14:38:16.647" Title="Can you l2 normalize word2vec vectors for density clustering?" Tags="|python|clustering|word2vec|normalization|dbscan|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="21720" PostTypeId="1" AcceptedAnswerId="21721" CreationDate="2017-07-26T16:05:46.343" Score="3" ViewCount="8242" Body="&lt;p&gt;I am totally beginner in Python and after using seasonal_decompose for time series decomposition &lt;code&gt;result=seasonal_decompose(series, model='additive', freq=365)&lt;/code&gt; I got plotted results with commands &lt;code&gt;result.plot()&lt;/code&gt; and &lt;code&gt;pyplot.show()&lt;/code&gt;, but I cannot understand how to print this results values on screen or how to see decomposed time series values?&#xA;I have plot, but I need to have values in console or some data file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Complete code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import read_csv&#xA;series=read_csv('B.csv', header=0, parse_dates=[0], index_col=0, squeeze=True)&#xA;from random import randrange&#xA;from matplotlib import pyplot&#xA;from statsmodels.tsa.seasonal import seasonal_decompose&#xA;result=seasonal_decompose(series, model='additive', freq=365)&#xA;result.plot()&#xA;pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27460" LastActivityDate="2019-07-01T16:07:28.597" Title="python print values seasonal_decomposition" Tags="|python|time-series|pandas|ipython|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21721" PostTypeId="2" ParentId="21720" CreationDate="2017-07-26T16:15:59.220" Score="5" Body="&lt;p&gt;seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.'  We can access the data by calling the object:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;res = seasonal_decompose(series, model='additive', freq=365)&#xA;residual = res.resid&#xA;seasonal = res.seasonal &#xA;trend = res.trend&#xA;print trend&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc...&lt;/p&gt;&#xA;" OwnerUserId="19161" LastEditorUserId="19161" LastEditDate="2017-07-26T16:42:36.197" LastActivityDate="2017-07-26T16:42:36.197" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="21722" PostTypeId="1" CreationDate="2017-07-26T16:43:05.053" Score="0" ViewCount="223" Body="&lt;p&gt;Supposing I have a dataset that I assume that have instances generated by two different distributions, is it possible to separate these instances based on the underlying generating distribution?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if I relax this hypothesis, and suppose I know one of the distributions beforehand, is this task possible? The second distribution could he chaotic, and also it is impossible to know it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, is there any result concerning the distance between the two distributions? Is there any theoretical proof of correctness, if there are algorithms for realizing that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you! &lt;/p&gt;&#xA;" OwnerUserId="37132" LastEditorUserId="37132" LastEditDate="2017-07-26T22:19:30.157" LastActivityDate="2017-07-26T22:19:30.157" Title="Detecting Different Distributions in data" Tags="|clustering|statistics|distribution|" AnswerCount="2" CommentCount="0" ClosedDate="2017-07-30T20:04:19.970" ContentLicense="CC BY-SA 3.0" />
  <row Id="21723" PostTypeId="2" ParentId="12384" CreationDate="2017-07-26T16:48:29.970" Score="1" Body="&lt;p&gt;You can never get 100% coverage for real-world grammar extraction. Grammar is complex and undefined for real-world data. In addition, it is an &quot;open world&quot; data problem because novel grammar phrases can always be created.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might get 100% coverage for a small data set through overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://Statistical%20learning%20theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Statistical learning&lt;/a&gt; theory provides a framework for predicting the limits of machine learning.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-26T16:48:29.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21724" PostTypeId="2" ParentId="21719" CreationDate="2017-07-26T16:57:55.323" Score="1" Body="&lt;p&gt;Normalizing the vectors maps them all down to a unit sphere. That definitely changes their Euclidean distances (not cosine distances though), so, no I don't think that's valid, nor does it help the dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering in high dimensions is indeed problematic and I agree with @Emre that t-SNE is an option to get more meaningful clusters out for visualization purposes. Simple PCA followed by clustering might be effective too.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-07-26T16:57:55.323" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21725" PostTypeId="2" ParentId="21722" CreationDate="2017-07-26T17:10:49.337" Score="0" Body="&lt;p&gt;I think you're looking for a measure of difference between two probability distributions. In that case, if you have two continuous distributions, try measuring the KL divergence (&lt;a href=&quot;https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained&quot; rel=&quot;nofollow noreferrer&quot;&gt;more information is available here&lt;/a&gt;). You may also be interested in a chi-squared test, depending on your application (have a look at &lt;a href=&quot;https://stats.stackexchange.com/questions/88764/test-for-difference-between-2-empirical-discrete-distributions&quot;&gt;Glen_b's answer here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think there are any theoretical proofs of correctness. All you can really do is compare your empirical distribution (your instances) to your theoretical distribution (the distribution you know they came from). Sometimes there's going to be a good chance that an instance that came from one distribution looks very much like an instance from another distribution - the instances would not be 'identifiable'. For example, if I'm generating data from a binomial distribution with large n and small p, those instances are going to look like they come from a Poisson distribution (&lt;a href=&quot;http://courses.wcupa.edu/rbove/Berenson/10th%20ed%20CD-ROM%20topics/section5_6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;see more detail here&lt;/a&gt;). In that case, the instances themselves are not really distinguishable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;** MAJOR EDIT **&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I misunderstood the question, as you can see from the comments below. If you have one distribution that is made up of two underlying distributions, and you want to label them, I think clustering may be the best way to go. Try something like the k-means algorithm. I'm pretty sure that doesn't require any distributional assumptions.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-07-26T21:28:26.733" LastActivityDate="2017-07-26T21:28:26.733" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21726" PostTypeId="2" ParentId="11975" CreationDate="2017-07-26T17:27:25.227" Score="1" Body="&lt;p&gt;One solution to this problem can be found in &lt;em&gt;&lt;a href=&quot;https://arxiv.org/abs/1412.7449&quot; rel=&quot;nofollow noreferrer&quot;&gt;Grammar as a Foreign Language&lt;/a&gt;&lt;/em&gt;. The paper outlines a system that &quot;translates&quot; English sentences into syntactic constituency parse trees using a sequence-to-sequence LSTM (Long Short-Term Memory model). A TensorFlow implementation can be found &lt;a href=&quot;https://github.com/ofirnachum/grammar&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-26T17:27:25.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21727" PostTypeId="2" ParentId="21716" CreationDate="2017-07-26T18:01:06.210" Score="0" Body="&lt;p&gt;If you choose to build a model where one of the categorical features such as a gender plays a big role in the outcome, then the overall model results will give a large weight to the gender, and other features will have a smaller weight. While your model will be applicable for the unsegmented population, the predictive power might not be very good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose that you wanted to identify if the weights of the other features differed for males vs. females and if the segmented models had better predictive power, then by segmenting the population and creating models for each gender, you could explore changes in the magnitude or direction of the weights of each feature &amp;amp; evaluate the goodness of fit of your models to see if there is an improvement in the segmented population models vs. the overall model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If performing logistic regression, you could scale the coefficients or Z-values returned in the model parameters for each model and check if the relative magnitude &amp;amp; direction of the overall vs. the segmented models gave you similar or different results. You could also look at the confusion matrix and use various accuracy measures to assess if the segmented models performed better, worse or the same as the overall model. In this way, you could determine whether for the outcome independent gender-based models are more appropriate for your question than a single overall model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, this leads to the question of if I have &quot;n&quot; features, should I create multiple models by segmenting the &quot;x&quot; categories in each of the &quot;n&quot; features?  This really depends on the data question being asked. In some cases, it is valuable to run multiple models where each feature is dropped one-by-one to assess feature importance. In some cases, just evaluating the weight &amp;amp; relevance of each feature from the overall model is sufficient. Going beyond, one could look at optimizing models by choosing which of all &quot;n&quot; features should be segmented to yield the model with the best predictive power. This paper from &lt;a href=&quot;http://www.fico.com/en/node/8140?file=9737&quot; rel=&quot;nofollow noreferrer&quot;&gt;FICO&lt;/a&gt; describes one methodology. This &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/02/guide-build-predictive-models-segmentation/&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; details an approach and comparison of various models. &lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2017-07-26T18:01:06.210" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21729" PostTypeId="1" CreationDate="2017-07-26T19:22:59.953" Score="1" ViewCount="221" Body="&lt;p&gt;The term &quot;Data Cleaning&quot; is used to describe outlier checking, date parsing, missing value imputation to structuring datasets (organizing data values within a dataset) to facilitate analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter is commonly referred to as &quot;&lt;strong&gt;&lt;em&gt;Data Tidying&lt;/em&gt;&lt;/strong&gt;&quot; but what about the former? I have seen outlier checking, date parsing and missing value imputation referred to as &quot;&lt;em&gt;soft processing&lt;/em&gt;&quot;, however I wonder if this is a commonly used term.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I describe a component of &lt;em&gt;data cleaning&lt;/em&gt; as &quot;&lt;strong&gt;soft processing&lt;/strong&gt;&quot; is it reasonable to assume that knowledgeable people will know that I am referring to outlier checking, date parsing and missing value imputation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the field of Data Science, is there a commonly accepted phrase or term that umbrellas &lt;em&gt;outlier checking, date parsing and missing value imputation&lt;/em&gt;. &lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2017-07-26T19:56:27.700" Title="Describing the data cleaning process" Tags="|data-cleaning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21730" PostTypeId="2" ParentId="21729" CreationDate="2017-07-26T19:56:27.700" Score="2" Body="&lt;p&gt;If you're not modifying anything you can call it &lt;em&gt;validation&lt;/em&gt;. If you are, you can call it &lt;em&gt;preprocessing&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-07-26T19:56:27.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21731" PostTypeId="2" ParentId="21722" CreationDate="2017-07-26T21:39:53.757" Score="1" Body="&lt;p&gt;It's not really feasible to label points as coming from distribution A or B if there is be overlap amongst the distributions. You can't know whether the point fell near the mean of A or the tail of B. In the simplest case, consider two normally distributed datasets with overlap - how can you know which generated it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;could&lt;/em&gt; estimate the probability of each point falling under A or B, but that doesn't really solve the problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different approach, but one I'm not sure will help, would be to fit a new distribution defined as the convolution of A and B. The ex-gaussian distribution is one such example. But you'd have to do the maths to figure out the PDF equation, and that still won't necessarily label your points. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-07-26T21:39:53.757" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21733" PostTypeId="1" CreationDate="2017-07-27T02:55:17.843" Score="0" ViewCount="4706" Body="&lt;p&gt;I am a data-science rookie and I would like to use Python/ R to create a correlation matrix (something like this: &lt;a href=&quot;http://www.marketcalls.in/python/quick-start-guide-compute-correlation-matrix-using-nsepy-pandas-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.marketcalls.in/python/quick-start-guide-compute-correlation-matrix-using-nsepy-pandas-python.html&lt;/a&gt;) and build a machine learning model. However, I have some questions and would really appreciate some guidance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: Although the data file is pretty big and have more than 350,000 entries, some columns missed many values (i.e., 60%/ 70% of the values are missing). I am wondering should I abandon those columns/ delete those rows/ any other great recommendations? And what is a good threshold, is it okay to proceed with columns that 20%? 30%? 40% values are missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much. Greatly appreciated your help!!&lt;/p&gt;&#xA;" OwnerUserId="37149" LastActivityDate="2018-11-20T17:33:24.810" Title="Correlation matrix with a dataset with many missing value" Tags="|data-cleaning|correlation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21734" PostTypeId="1" CreationDate="2017-07-27T03:20:21.703" Score="24" ViewCount="29029" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/42187425/how-to-change-input-shape-in-sequential-model-in-keras&quot;&gt;This post&lt;/a&gt; seems to indicate that what I want to accomplish is not possible. However, I'm not convinced of this -- given what I've already done, I don't see why what I want to do can not be achieved...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two image datasets where one has images of shape (480, 720, 3) while the other has images of shape (540, 960, 3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initialized a model using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input = Input(shape=(480, 720, 3), name='image_input')&#xA;&#xA;initial_model = VGG16(weights='imagenet', include_top=False)&#xA;&#xA;for layer in initial_model.layers:&#xA;    layer.trainable = False&#xA;&#xA;x = Flatten()(initial_model(input))&#xA;x = Dense(1000, activation='relu')(x)&#xA;x = BatchNormalization()(x)&#xA;x = Dropout(0.5)(x)&#xA;x = Dense(1000, activation='relu')(x)&#xA;x = BatchNormalization()(x)&#xA;x = Dropout(0.5)(x)&#xA;x = Dense(14, activation='linear')(x)&#xA;&#xA;model = Model(inputs=input, outputs=x)&#xA;model.compile(loss='mse', optimizer='adam', metrics=['mae'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now that I've trained this model on the former dataset, I'd like to pop the input tensor layer off and prepend the model with a new input tensor with a shape that matches the image dimensions of the latter dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = load_model('path/to/my/trained/model.h5')&#xA;old_input = model.pop(0)&#xA;new_input = Input(shape=(540, 960, 3), name='image_input')&#xA;x = model(new_input)&#xA;m = Model(inputs=new_input, outputs=x)&#xA;m.save('transfer_model.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which yields this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;&amp;lt;stdin&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 2506, in save&#xA;    save_model(self, filepath, overwrite, include_optimizer)&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/models.py&quot;, line 106, in save_model&#xA;    'config': model.get_config()&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 2322, in get_config&#xA;    layer_config = layer.get_config()&#xA;  File &quot;/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py&quot;, line 2370, in get_config&#xA;    new_node_index = node_conversion_map[node_key]&#xA;KeyError: u'image_input_ib-0'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the post that I linked, maz states that there is a dimension mismatch that prevents changing the input layer of a model -- if this was the case, then how is that I put a (480, 720, 3) input layer in front of the VGG16 model which expects (224, 224, 3) images? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a more likely issue is that my former model's output is expecting something different than what I'm giving it based on what fchollet is saying in &lt;a href=&quot;https://github.com/fchollet/keras/issues/2790&quot; rel=&quot;noreferrer&quot;&gt;this post&lt;/a&gt;. I'm syntactically confused, but I believe the whole &lt;code&gt;x = Layer()(x)&lt;/code&gt; segment is constructing the layer piece by piece from input-&gt;output and simply throwing a different input in front is breaking it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really have no idea though...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody please enlighten me how to accomplish what I'm trying to do or, if it's not possible, explain to me why not?&lt;/p&gt;&#xA;" OwnerUserId="36935" LastActivityDate="2021-03-11T18:57:30.317" Title="Keras -- Transfer learning -- changing Input tensor shape" Tags="|keras|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21735" PostTypeId="2" ParentId="21733" CreationDate="2017-07-27T07:41:42.467" Score="1" Body="&lt;p&gt;It is impossible to say whether or not you should trim your data, without doing some &lt;strong&gt;&lt;em&gt;Exploratory Data Analysis&lt;/em&gt;&lt;/strong&gt; first. Those missing values may be important to your understanding of the dataset, so you have to look at your data comparing variable to variable to determine if the missing values hold information.&lt;br&gt;&#xA;For example, if your data is about vehicles, there could be a variable representing &lt;em&gt;number of doors&lt;/em&gt;.  If your missing data is structural an observation of a motorcycle may have a missing value for that variable.  In which case, the observation for that variable could be discarded as meaningless.  Or, you could impute the missing value to something meaningful like the number 0. It depends on what insight you are ultimately trying extract from you data.  Handling missing values is quite a complex subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest that you &quot;&lt;a href=&quot;https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tidy the data&lt;/a&gt;&quot; first.  From the Wickham paper&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Tidy datasets are easy to manipulate, model and visualize, and have a specific structure:&#xA;  each variable is a column, each observation is a row, and each type of observational unit&#xA;  is a table.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Initially this can help you do some Exploratory Analysis to determine what is important and what is not.  Once you have made those distinction you can find and implement strategies to handle the missing values appropriately.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2017-07-27T07:41:42.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21742" PostTypeId="1" CreationDate="2017-07-27T09:46:24.933" Score="0" ViewCount="112" Body="&lt;p&gt;I am reading about k-means clustering method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the method calculate the mean between two coordination points?&#xA;How does the method work with more than two dimensions? How can calculate the distance and the mean if there are more than two dimension?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any packages about k-means clustering in R?&lt;/p&gt;&#xA;" OwnerUserId="37164" LastEditorUserId="35902" LastEditDate="2017-07-28T06:56:17.513" LastActivityDate="2017-07-28T06:56:17.513" Title="K mean clustering method of data mining" Tags="|machine-learning|r|data-mining|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21743" PostTypeId="1" AcceptedAnswerId="21753" CreationDate="2017-07-27T10:09:35.183" Score="0" ViewCount="33" Body="&lt;p&gt;I have the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Goal       Achieved&#xA;&#xA;100        90&#xA;150        130&#xA;200        175&#xA;&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first column &quot;Goal&quot; is the number which should be done that day, and the column &quot;Achieved&quot; is the number that was actually done that day. Each row stands for a day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use this data to assess the performance of a worker. Here below I have two ideas:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Method 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A600s.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A600s.png&quot; alt=&quot;https://i.stack.imgur.com/A600s.png&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Method 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/u9DYg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/u9DYg.png&quot; alt=&quot;https://i.stack.imgur.com/u9DYg.png&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which one of the methods is reasonable and mathematically right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if neither is good, what should I do to evaluate the performance of a worker?&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="24000" LastEditDate="2017-07-27T15:16:56.793" LastActivityDate="2017-07-27T15:16:56.793" Title="How to evaluate the performance based on rate data" Tags="|performance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21744" PostTypeId="1" CreationDate="2017-07-27T11:50:45.587" Score="0" ViewCount="112" Body="&lt;p&gt;I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with inconsistent or absent in-bound taxonomy terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Primary source of profile info is biography pages on people's employer websites. Some of those sites choose to state employees' multiple specialist topics, some make only narrative biographies available, some both. I have collected all available info, using Python's Scrapy, in to CSV files - one per company, people are rows  - where available, topics at my end now themselves reside in a comma-separated field/string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: in one sheet, cell S7 is: &quot;Analytics Applications,Big Data,Cognitive Computing,Competitive Intelligence,eDiscovery,Enterprise Content Management (ECM),Information Architecture,Market Research,Product Information Management (PIM)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is severalfold:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Taxonomy terms across companies are inconsistent (eg. &quot;Cognitive Computing&quot; in the above example may, to another company, be &quot;AI&quot;).&lt;/li&gt;&#xA;&lt;li&gt;Some companies use far too many terms in total (eg. one company alone uses approx 450 tags in total).&lt;/li&gt;&#xA;&lt;li&gt;Often, none are available at all.&lt;/li&gt;&#xA;&lt;li&gt;As biography narratives describe more than just employees' specialist topics (eg. education and upbringing background), their usefulness in automation may be questionable.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My goal is to create a taxonomy that categorises all the collected person bios in a much more harmonious, consistent and briefer fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;System setup is PHP/MySQL/WordPress. Profile CSVs are imported in to WordPress, and the system has the ability to perform PHP functions on imported content (not just on the info in WordPress after import, but during import via PHP).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total profiles count is approx 4,500, so manual taxonomisation is unappealing. So I have examined AI/machine learning techniques. I am not strictly a developer and certainly not a data scientist or mathematician.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have found text classification tests carried out using Aylien and Monkey Learn to yield poor results. In each case, output results are not granular enough, ie. turning in-bound terms of biogs about granular topics like cloud computing infrastructure and data centres in to overly basic terms like &quot;Computers &amp;amp; Internet&quot;. Aylien uses the off-the-shelf IPTC NewsCodes taxonomy, and I understand I can use Monkey Learn to train. I like the idea of using a standardised off-the-shelf taxonomy like NewsCodes, but a) the results are questionable, and b) it may not be granular enough for my needs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, I have decided to draw up my preferred hierarchy of taxonomy terms, approx 230, which should each speak roughly to the swathe of inconsistent in-bound terms and profiles (in other words, correlate to the people's topics). That seemed like an important step, assuming I need to steer this manually. But I'm struggling to grasp how to actually implement that correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I am looking for some guidance on best methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea I am toying with is to put my own preferred taxonomy in to WordPress as taxonomy terms, and, alongside each, put a cluster of terms from the actual source material so that, if one of the related terms is found in a user's inbound data, the term from my preferred taxonomy should be assigned. But I'm not sure whether this is particularly efficient, or even wise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my first time on the Data Science group at StackExchange. I apologise if I have shot wide of the mark here at all.&lt;/p&gt;&#xA;" OwnerUserId="37169" LastEditorUserId="37169" LastEditDate="2017-07-27T12:41:15.767" LastActivityDate="2024-01-29T04:04:50.590" Title="What methods to create singular content classification from inconsistent inbound info?" Tags="|classification|text|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21748" PostTypeId="1" CreationDate="2017-07-27T12:24:49.060" Score="0" ViewCount="59" Body="&lt;p&gt;Here is all 34 rows of my training data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(20, [0, 0, 0, 4, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(15, [0, 0, 0, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(9, [0, 0, 0, 4, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 6, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(16, [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(17, [0, 0, 0, 3, 4, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 4, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(14, [0, 0, 0, 1, 0, 0, 0, 3, 4, 0, 2, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(13, [0, 0, 0, 5, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(17, [0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(19, [0, 0, 0, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 4, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(15, [0, 0, 0, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 6, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 3, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(13, [0, 0, 0, 1, 2, 2, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1, 4, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 5, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(11, [0, 0, 0, 6, 3, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(21, [0, 0, 0, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 3, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(21, [0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 3, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(23, [0, 0, 0, 4, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(20, [0, 0, 0, 4, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 2, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(24, [0, 0, 0, 1, 4, 6, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 6, 1, 0, 0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1])&#xA;(23, [0, 0, 0, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 2, 7, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 2, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 6, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(24, [0, 0, 0, 4, 5, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(23, [0, 0, 0, 6, 5, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(22, [0, 0, 0, 5, 3, 0, 3, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(24, [0, 0, 0, 5, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(23, [0, 0, 0, 1, 0, 0, 0, 0, 6, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;(21, [0, 0, 0, 4, 5, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to create a number sequence generator that, given the first number in the tuple (like &lt;code&gt;22&lt;/code&gt;), it produces a list of numbers that look like the second:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0, 0, 0, 6, 1, 0, 0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like it to automatically discover the constraints (the sum of the list equals the first number, some positions in the list are always zero, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there something that could produce this backward classifier? I would prefer it to be some library in Python which doesn't require GPUs to run.&lt;/p&gt;&#xA;" OwnerUserId="27226" LastEditorUserId="24000" LastEditDate="2017-07-27T12:33:01.087" LastActivityDate="2017-07-27T16:03:48.627" Title="Simplest way to train a number sequence generator" Tags="|feature-extraction|sequential-pattern-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21749" PostTypeId="1" AcceptedAnswerId="21755" CreationDate="2017-07-27T14:23:57.807" Score="3" ViewCount="830" Body="&lt;p&gt;I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While the last one would be the field I like to predict at some point.&lt;/p&gt;&#xA;" OwnerUserId="37153" LastEditorUserId="24000" LastEditDate="2017-07-27T14:42:48.470" LastActivityDate="2018-06-25T17:42:37.510" Title="How to handle missing data for machine learning" Tags="|machine-learning|missing-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21750" PostTypeId="2" ParentId="21744" CreationDate="2017-07-27T14:27:18.943" Score="0" Body="&lt;p&gt;You could import each companies data into a specific  table and then develop regular expression Scripts two change specific Expressions into your own taxonomy.&#xA;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Regular_expression&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Regular_expression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2017-07-27T14:27:18.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21752" PostTypeId="2" ParentId="19947" CreationDate="2017-07-27T14:45:37.197" Score="2" Body="&lt;p&gt;I don’t know where you’re getting this particular classification scheme from, (continuous, categorical, nominal), but it’s worth mentioning that this isn’t a great scheme for classifying data types. You’ve already encountered an example of a data type (count data/natural numbers) that doesn’t fit neatly into them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also worth mentioning that a lot of people will use “nominal” and “categorical” interchangeably to describe a discrete data type with no natural ordering. Probably the most widely-accepted term to describe discrete data with a natural ordering is &lt;em&gt;ordinal&lt;/em&gt; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, &quot;number of rooms in a flat&quot; has an unambiguous ordering. In a very real and intuitive sense, a five-room flat has more rooms than a two-room flat. A one-room flat has fewer rooms than a three-room flat. It makes sense to talk about one flat having a greater or lesser number of rooms than another, and if we want to know if that ordering matters in some way, we need to preserve the ordering in our analysis.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-07-27T14:45:37.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21753" PostTypeId="2" ParentId="21743" CreationDate="2017-07-27T15:13:20.373" Score="2" Body="&lt;p&gt;It depends on what you want to achieve, and how you'd define performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose it is a running exercise program with 3 exercises in a week with increasing targets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first formula calculates the average exercise success, the second formula defines the weekly success, where the exercises are weighted with the goals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no right or wrong, just interpretation of the numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to confuse you a bit more: suppose:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Goal: 8, Achieved 1, Goal: 2, achieved 1&lt;/li&gt;&#xA;&lt;li&gt;Goal: 2, Achieved 0, Goal: 8, achieved 3.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Case 1. beats Case  2. on both instances in formula 1, but loses in formula 2.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-07-27T15:13:20.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21754" PostTypeId="2" ParentId="21749" CreationDate="2017-07-27T15:46:38.850" Score="3" Body="&lt;p&gt;Interpolation seems like it would make sense in this case: any time you miss a day, take an average of the before and after.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, I don't think you have to give up on the missing weather values so easily. There are a variety of R packages that simplify getting weather for an arbitrary location from someone like Weather Underground with only a couple lines of code. &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-07-27T15:46:38.850" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21755" PostTypeId="2" ParentId="21749" CreationDate="2017-07-27T15:54:04.860" Score="6" Body="&lt;p&gt;There are three main approaches to handling missing data.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ignore - some methods can just ignore missing data, and not use it in the model at all&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37184" LastActivityDate="2017-07-27T15:54:04.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21756" PostTypeId="2" ParentId="21748" CreationDate="2017-07-27T16:03:48.627" Score="1" Body="&lt;p&gt;You could reframe the problem as a regression - prediction of a single real-valued dependent variable based on several independent variables. If you choose to model with regression, then it technically a &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Calibration_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;calibration&lt;/a&gt;&quot; problem. A calibration problem takes a known observation of the dependent variables is used to predict corresponding explanatory variables.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-27T16:03:48.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21757" PostTypeId="2" ParentId="15477" CreationDate="2017-07-27T16:19:21.287" Score="-1" Body="&lt;p&gt;Deep learning extends the ability of information retrieval (IR) systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep learning has proved to be a powerful tool for feature engineering. This improvement in feature engineering, over manual construction, has improved the quality of any machine learning system, including information retrieval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, word embeddings create a dense, vectorized representation of words that encodes the semantic relationships based on co-occurrence. Both documents and queries can be projected into the same latent space and nearest-neighbors can be found. Word embeddings can be used to extend the posting index to by finding related terms. They can also be used for query-expansion / query rewriting. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-27T16:19:21.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21758" PostTypeId="1" CreationDate="2017-07-27T16:32:01.820" Score="1" ViewCount="2117" Body="&lt;p&gt;Something that has always bothered me is summarizing distributions when feature engineering for a machine learning process. Does anybody have best practices for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: Imagine a dataset showing orders of products by customers. You want to summarize customer behaviour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that example, if we're to only focus on customers' order value, the order value for each customer would have some distribution D(x).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, can create variables describing features of each customer's distributions (min, mean, median, max, quartiles, IQR, etc.), but are there best practices around which features tend to provide the most information upon extraction? Moreover, is there some way to contain the information of a distribution in a single variable?&lt;/p&gt;&#xA;" OwnerUserId="37186" LastActivityDate="2018-02-23T15:35:58.480" Title="Feature engineering on distributions" Tags="|machine-learning|data-mining|statistics|feature-selection|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21759" PostTypeId="1" AcceptedAnswerId="23849" CreationDate="2017-07-27T16:51:52.807" Score="1" ViewCount="231" Body="&lt;p&gt;I'm reading an old book called &quot;Data Mining Practical Machine Learning and Techniques&quot;;  It uses a tool called Weka to automate examples as well, but I'd prefer to use Python to do so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new to Machine Learning, and I'm just trying to line up examples with a computer API; I'm not certain of the type of Naive Bayes being used in the example (which I implemented in a spreadsheet from the book).  From the look of it, based on other examples in the book, it doesn't look to be multinomial-bayes, but I don't seem to be able to find an API that lines up with it either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example tries to determine if a sports team will go outside and play or not (yes, no) based on the Outlook (sunny, overcast, rainy), Temperature Classification (hot, mild, cold), Humidity Classification (high, normal) and if it is Windy or not (TRUE, FALSE).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values are INPUT from data rows in the Weather Data Table 1.2 on page 11.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The processing of the INPUT begins and then in &lt;code&gt;Table 4.2 The weather data with counts and probabilities&lt;/code&gt; they tally up probabilities and totals based on possible attribute values and totals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They then add a new day row to the input and process which does not have a Will the team play value, but which does have all the other attributes with a single value for each.  The likelyhood is calculated for yes and no, and then in the Output a probability value for &lt;code&gt;yes&lt;/code&gt; and a value for &lt;code&gt;no&lt;/code&gt; are calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've included the spreadsheet &lt;a href=&quot;https://drive.google.com/open?id=0B4UUQH5An0uuaHlPOGhoUXdXVms&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize this is really basic Machine Learning, but I just wondered if anyone could point me towards a python library or python algorithm that would handle such a problem...I've actually also tried to get it to work in Weka, and run into a similar issue in which I appear to be able to find a Naive Bayes algorithm without churning out the correct result presented in the book and in my spreadsheet.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So does this particular Naive Bayes method have a name and equivalent API?&lt;/p&gt;&#xA;" OwnerUserId="14301" LastActivityDate="2017-10-16T17:40:38.807" Title="What Naive Bayes method is being used in this example?" Tags="|python|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21760" PostTypeId="1" CreationDate="2017-07-27T16:51:55.287" Score="-3" ViewCount="1798" Body="&lt;p&gt;I have a very large panel data set (contains around 50M observations, size around 3G). I would to run an algorithm on it. The algorithm basically just loops over observations. Ideally, I would like to use functions in numpy, but I guess this would be really slow. Would R or matlab be good for this? Are there any other python packages I could use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help.&lt;/p&gt;&#xA;" OwnerUserId="37187" LastEditorUserId="29169" LastEditDate="2019-06-10T15:55:15.727" LastActivityDate="2019-06-10T15:55:15.727" Title="Which coding language to use for very large datasets?" Tags="|python|r|bigdata|matlab|programming|" AnswerCount="1" CommentCount="2" ClosedDate="2017-07-28T06:55:35.613" ContentLicense="CC BY-SA 4.0" />
  <row Id="21761" PostTypeId="2" ParentId="21758" CreationDate="2017-07-27T16:54:38.607" Score="1" Body="&lt;p&gt;At the end each distribution can be described by a function with parameters. Can be a Gaussian, polynomial etc. In principle you can choose functions that only have one free parameter and fit this one. Depending on your data you might be able to guess a function class that seems to fit the underlying distributions. You can then use the fit parameters as input for your data set. Example: Fit a Gaussian and use mean, normalization and sigma = 3 parameters for your model.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-07-27T16:54:38.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21762" PostTypeId="2" ParentId="21760" CreationDate="2017-07-27T17:09:30.450" Score="1" Body="&lt;p&gt;Python is highly efficient for large scale datasets.&#xA;Second choince will be R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try python scikit-learn if you want to solve a machine learning problem.&#xA;Pandas for manipulating and extracting data.&lt;/p&gt;&#xA;" OwnerUserId="33020" LastActivityDate="2017-07-27T17:09:30.450" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21765" PostTypeId="1" AcceptedAnswerId="21777" CreationDate="2017-07-27T20:02:01.067" Score="0" ViewCount="1280" Body="&lt;p&gt;I'm not gonna lie, I'm very new to neural networks but am also so interested in them and learning the way they work and what can be made from them. So in my endeavors for learning, I stumbled upon Siraj Ravel's youtube channel and furthermore, his Github where he posted this 1 neuron neural network that predicts the output given inputs.* I looked at it and his accompanying video and got the basic gist of how it works. After running it a few times making minor adjustments to say the inputs, I decided why not try to add numbers other than 1 and 0. I made a pattern with the output being the inputs added, however, I ran into an error where the output of its test was '1.'. I assumed this was because of the sigmoid function forcing it to be in between 1 and 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary:&lt;/strong&gt; How can I change that program so I can give it an input of patterns with different numbers than 1 and 0 and it will output correctly (examples below)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*In his program inputs were like [1, 0, 1] = 1, [0, 0, 1] = 0 and [1, 0, 0] = 1 (pattern is first column of matrix is answer) and he asked the program to give output for [1, 1, 0] and it correctly outputted 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*My goal for it was so given &quot;[2, 2, 2] = 6, [1, 2, 0] = 3, and [1, 1, 2] = 4 (more training included if necassary)&quot; as input could it get 4 as output for &quot;[2, 2, 0]?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/llSourcell/Make_a_neural_network/blob/master/demo.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;&lt;em&gt;Code for program&lt;/em&gt;&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37196" LastActivityDate="2017-07-28T11:24:33.540" Title="Shallow Neural Net for predicting numbers other then 1 or 0?" Tags="|neural-network|prediction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21766" PostTypeId="1" CreationDate="2017-07-27T20:13:15.053" Score="1" ViewCount="487" Body="&lt;p&gt;So far, I have stumbled upon many advices and papers on PU Learning and Unary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TLDR: Does anyone have suggestions for specific algorithm or implementation for labeled data of only one class and unlabeled data that can be from either classes? And I'm unsure what is the proportion of Class A to B that exists within the unlabeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest answer I have found has been one-class SVM (Binary semi-supervised classification with positive only and unlabeled data set), but I have so many unlabeled examples compared to how many labeled ones I can find. And I am unsure if either the positive class or negative class are rare enough for anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the other suggested methods is the two-step process where I can figure out a set of reliable negative class data, but I cannot really identify a set of data as reliably negative (&lt;a href=&quot;https://www.cs.uic.edu/~liub/publications/ICDM-03.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.uic.edu/~liub/publications/ICDM-03.pdf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And another method suggests a weighted SVM (&lt;a href=&quot;http://users.csc.tntech.edu/~weberle/Spring2011/CSC6910/Papers/posonly.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://users.csc.tntech.edu/~weberle/Spring2011/CSC6910/Papers/posonly.pdf&lt;/a&gt;), but I am unsure if I can make the same assumption as the authors in that my positive data is a random subset of all the positive data, as I used a criteria to figure out which ones were positive, so I assume there is bias in the labeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, I have a lot of labeled data of positive class, that is to say the data of what I am looking for, but then I have many more unlabeled data. (Though in a way, the labeled data could also be considered data of a negative class.) And I am unsure what proportion of positive data and negative data exists within the unlabeled data, as there could be an equal distribution between the two classes. Or who knows, maybe data of the positive class could be rarer than the data of the negative class.&lt;/p&gt;&#xA;" OwnerUserId="37200" LastActivityDate="2023-12-25T20:02:05.233" Title="How should I construct a binary classifier for thousands of positive data and millions of unlabeled data?" Tags="|python|classification|tensorflow|svm|semi-supervised-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21768" PostTypeId="2" ParentId="6789" CreationDate="2017-07-27T20:50:57.193" Score="1" Body="&lt;p&gt;What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;delta3 = x3 - Y[:,j];&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;similar to the way you calculate &lt;code&gt;delta2&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26239" LastActivityDate="2017-07-27T20:50:57.193" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21769" PostTypeId="1" AcceptedAnswerId="21772" CreationDate="2017-07-27T21:01:05.057" Score="-2" ViewCount="121" Body="&lt;p&gt;Given data table with inconsistent item descriptions, how could I most effectively assign an item category using R (i.e. &lt;code&gt;dplyr&lt;/code&gt;), MySQL, or Python? An R based solution is preferred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MySQL is the data source. As is, case-when logic assigns an item category based on an item description. There is no common key or category to aggregate the items, hence the need to create one. However, not all like items are named the same. These are the same items, but sold across different locations. The is little to no consistency for item descriptions between locations. Real world data provides plenty of unstructured learning opportunities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, consider an item like 'Whole Cheese Pizza'. This item exists with multiple descriptions such as 'Whl Cheese Pizza', 'Pizza Chs Whole', 'Cheese Pizzza Whole', 'Whole Cheese' etc. Ideally these all roll into one category named 'Pizza'. Case-when logic uses finds item descriptions like '%Pizza%' or 'Whl Cheese%' or 'Whole Cheese%' to assign the 'Pizza' category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly not ideal nor really scaleable. More pragmatic than programmatic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without seeing the catalog, are there any suggestions on how to apply a more programmatic method to catalog items with inconsistent text descriptions for aggregation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can provide additional context or details as needed. Thank you for time and expertise!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: &lt;a href=&quot;https://drive.google.com/open?id=0B5LCwsERNvi6R2NwY2dmS1lqbE0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sample file&lt;/a&gt; here with item descriptions and rollups attached. Goal is to recreate the rollup category without case-when logic. Note the missing rollup category is the default case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit: fixed link to file&lt;/p&gt;&#xA;" OwnerUserId="31092" LastEditorUserId="35902" LastEditDate="2017-07-28T06:55:57.483" LastActivityDate="2017-07-28T06:55:57.483" Title="Categorize observations with inconsistent text descriptions" Tags="|python|r|text-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21771" PostTypeId="2" ParentId="21742" CreationDate="2017-07-28T02:24:22.907" Score="2" Body="&lt;p&gt;To answer simply, Euclidean distance is generally used:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ d = |\mathbf{x}-\mathbf{y}|=\sqrt{\sum_{i=1}^n|x_i-y_i|^2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Read more about k-means over here:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.datascience.com/blog/introduction-to-k-means-clustering-algorithm-learn-data-science-tutorials&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to K-means&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://sites.google.com/site/dataclusteringalgorithms/k-means-clustering-algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means clustering algorithm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Using &lt;code&gt;kmeans()&lt;/code&gt; in &lt;code&gt;R&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/k-means-clustering-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;K Means Clustering in R&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35902" LastActivityDate="2017-07-28T02:24:22.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21772" PostTypeId="2" ParentId="21769" CreationDate="2017-07-28T02:48:33.430" Score="0" Body="&lt;p&gt;Thanks for the real-world problem! Interesting challenge there. Some thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;regex (regular expressions) may not work because &quot;&lt;em&gt;Whole Cheese&lt;/em&gt;&quot; (the description) doesn't even contain &quot;&lt;em&gt;pizza&lt;/em&gt;&quot; (the category)!&lt;/li&gt;&#xA;&lt;li&gt;the &quot;case-when logic&quot; you mentioned is SQL's equivalent of handling conditional (if/else) logic using SQL's &lt;code&gt;CASE&lt;/code&gt; statement. This can be helpful but super-tedious due to the huge number of rules required. (And then what happens when you have misspelt descriptions e.g. &lt;em&gt;pizzza&lt;/em&gt;?)&lt;/li&gt;&#xA;&lt;li&gt;wrt the comment by @Spaceman, my thought is that using &lt;code&gt;stringdist&lt;/code&gt; to calculate string distance to then perform clustering, which is unsupervised, wastes all the useful labelled data which can be used!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My suggestion&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a combination of &quot;case-when logic&quot; and classification algorithms, to generate a set of labelled data (the labels being the rollups) in which then you can perform some supervised classification for the unlabelled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your sample file, some categories have already been labelled. You can use that as training data for your supervised learning (e.g. SVM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may require some text mining knowledge (e.g. &lt;code&gt;DocumentTermMatrix&lt;/code&gt;, tokenizers). I refer you to the wonderful &lt;code&gt;tm&lt;/code&gt; and &lt;code&gt;RTextTools&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="35902" LastActivityDate="2017-07-28T02:48:33.430" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21774" PostTypeId="5" CreationDate="2017-07-28T06:07:55.463" Score="0" Body="&lt;p&gt;The &lt;em&gt;k&lt;/em&gt;-nearest neighbors algorithm (&lt;em&gt;k&lt;/em&gt;-NN) is one of the simplest machine learning algorithms. It is non-parametric (makes no assumptions about the population distribution or sample size) and memory-based method that can be used for classification and regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we want to train &lt;em&gt;k&lt;/em&gt;-NN with a set of $N$ feature vectors $\boldsymbol{x}$ and class labels $y$: $\{(\boldsymbol{x}_1,y_1),(\boldsymbol{x}_2,y_2),\dots,(\boldsymbol{x}_N,y_N)\}$. In the training phase, no model is fitted but the feature vectors and class labels are stored in memory. In the classification/regression phase, a distance between a given query feature vector $\boldsymbol{x}_u$ and each training example is calculated, and &lt;em&gt;k&lt;/em&gt; (user-defined constant) closest training examples in the feature space are selected. If it is a classification task, the predicted class label $\hat{y}_u$ is selected by a majority vote amongst $\boldsymbol{x}_u$'s &lt;em&gt;k&lt;/em&gt; nearest neighbors. If it is a regression task, the predicted value $\hat{y}_u$ is the average of the values of $\boldsymbol{x}_u$'s &lt;em&gt;k&lt;/em&gt; nearest neighbors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A downside of the majority voting is that if the class distribution is skewed,   the frequent class examples can dominate the voting. A way to try to solve this issue is to weight points by the inverse of their distance, which leads to closer neighbors having greater influence than those further away.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;References&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Friedman, J., Hastie, T., &amp;amp; Tibshirani, R. (2009). &lt;em&gt;The elements of statistical learning&lt;/em&gt; (2nd edition). New York: Springer Series in Statistics.&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikpedia: k-nearest neighbors algorithm&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-07-31T20:23:30.290" LastActivityDate="2017-07-31T20:23:30.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21775" PostTypeId="4" CreationDate="2017-07-28T06:07:55.463" Score="0" Body="k-nearest neighbors algorithm (k-NN) is a non-parametric method used for classification and regression. Training the algorithm consists of simply storing training examples in the memory. The algorithm makes predictions based on the characteristics of the k closest training examples in the feature space." OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-07-30T20:23:37.827" LastActivityDate="2017-07-30T20:23:37.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21777" PostTypeId="2" ParentId="21765" CreationDate="2017-07-28T08:13:06.673" Score="1" Body="&lt;p&gt;Well, the main problem is that the existing code and the desired codes solve different problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The existing code solves a classification problem, when you predict one of certain classes. And you want to solve a regression problem, so that neural net will predict an arbitrary number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sigmoid function is necessary when you predict classes, you need to remove it for regression. Or in other words you need not Logistic Regression, but Linear Regression.&#xA;Here is a great article with explanations on this: &lt;a href=&quot;http://peterroelants.github.io/posts/neural_network_implementation_part01/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://peterroelants.github.io/posts/neural_network_implementation_part01/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have written an example of implementation: &lt;a href=&quot;https://gist.github.com/Erlemar/6a5cfcca423ef3b5f6e890c6bef6d5ed&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/Erlemar/6a5cfcca423ef3b5f6e890c6bef6d5ed&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35541" LastEditorUserId="35541" LastEditDate="2017-07-28T11:24:33.540" LastActivityDate="2017-07-28T11:24:33.540" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="21779" PostTypeId="1" CreationDate="2017-07-28T08:48:02.773" Score="0" ViewCount="244" Body="&lt;p&gt;ARIMA cannot model large lags(obtained from autocorrelation plot) and long range dependency(hurst exponent H &gt; 0.5). Why is it so?&lt;/p&gt;&#xA;" OwnerUserId="37221" LastActivityDate="2019-04-20T02:03:15.633" Title="Why can't ARIMA model large lags and/or long range dependence?" Tags="|time-series|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21780" PostTypeId="1" CreationDate="2017-07-28T09:32:13.880" Score="11" ViewCount="13533" Body="&lt;p&gt;I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the &quot;rule if ten&quot; I need at least 10 events for each feature to be included. Though, I have an imbalanced dataset, with 20% o positive class and 80% of negative class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That gives me only 70 events, allowing approximately only 7/8 features to be included in the Logistic model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to evaluate all the features as predictors, I don't want to hand pick any features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what would you suggest? Should I make all possible 7 features combinations? Should I evaluate each feature alone with an association model and then pick only the best ones for a final model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also curious about the handling of categorical and continuous features, can I mix them? If I have a categorical [0-1] and a continuous [0-100], should I normalize?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently working with Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help!&lt;/p&gt;&#xA;" OwnerUserId="37225" LastActivityDate="2017-07-29T21:34:26.290" Title="How to perform Logistic Regression with a large number of features?" Tags="|machine-learning|python|predictive-modeling|logistic-regression|data|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21782" PostTypeId="1" AcceptedAnswerId="21785" CreationDate="2017-07-28T14:36:29.707" Score="2" ViewCount="3825" Body="&lt;p&gt;I am trying to use &lt;code&gt;train_test_split&lt;/code&gt; to split my data. However, I am getting an index error. I pasted part of the error message below. I am using Python 3.5 version and &lt;code&gt;sklearn&lt;/code&gt; 0.18.1. The code worked with my previous dataset that was different. Features here are in Pandas &lt;code&gt;DataFrame&lt;/code&gt; and labels are in Pandas&#xA;&lt;code&gt;Series&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.4, random_state=1)&#xA;&#xA;KeyError                                  &#xA;Traceback (most recent call last)&#xA;/apps/anaconda/anaconda-3.5/lib/python3.5/site-&#xA;    packages/pandas/indexes/base.py in get_loc(self, key, method, tolerance)&#xA;   2133             try:&#xA;-&amp;gt; 2134                 return self._engine.get_loc(key)&#xA;   2135             except KeyError:&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;&#xA;pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4443)()&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;&#xA;pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4289)()&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;&#xA;pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13733)()&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;&#xA;pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13687)()&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;&#xA;KeyError: 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28584" LastEditorUserId="381" LastEditDate="2017-07-28T16:13:59.290" LastActivityDate="2017-07-28T21:01:55.630" Title="Pandas index error" Tags="|machine-learning|python|scikit-learn|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21783" PostTypeId="2" ParentId="19747" CreationDate="2017-07-28T15:52:08.677" Score="1" Body="&lt;p&gt;This error basically means that your data is incorrectly formatted and Orange cannot read it. Somewhere there is either a redundant column delimiter or it might be absent. Either way, you should carefully check that all your columns are correctly separated by tabs or any other consistent separator. &lt;/p&gt;&#xA;" OwnerUserId="22012" LastActivityDate="2017-07-28T15:52:08.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21784" PostTypeId="2" ParentId="21780" CreationDate="2017-07-28T17:08:11.770" Score="6" Body="&lt;p&gt;In order to reduce your model down to 7 variables there are a few approaches you could take:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;noreferrer&quot;&gt;PCA&lt;/a&gt; (unsupervised): this creates &quot;new&quot; linear combinations of your data where each proceding component explains as much variance in the data as possible. So the first 7 components (out of 27) should be able to explain a good percentage of the variation in your data. You can then plug these seven components into your logistic regression equation. The disadvantage here is that because the components are combinations of your original variables you lose some interpretability with your regression model. It should however produce very good accuracy. This same technique applied to other dimension reduction methods such as &lt;/li&gt;&#xA;&lt;li&gt;Another common method in regression is forward stepwise where you start with one variable and add on another each step, which is either kept or dropped based on some criteria (usually a BIC or AIC score). Backwards stepwise regression is the same thing but you start with all variables and remove one each time again based on some criteria. Based on a brief search it doesn't seem that python has a stepwise regression but they do a similar feature elimination algorithm described in this &lt;a href=&quot;https://datascience.stackexchange.com/questions/937/does-scikit-learn-have-forward-selection-stepwise-regression-algorithm&quot;&gt;Data Science post&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Lasso Regression uses an $L_{1}$ penalization norm that shrinks the coefficients of features effectively eliminating some of them.You can include this $L_1$ norm into your logistic regression model. It seems &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression&quot; rel=&quot;noreferrer&quot;&gt;sklearn's LogisticRegression&lt;/a&gt; allows you do assign the penalization you want in order to achieve this. Note: Lasso will not explicitly set variable coefficients to zero, but will shrink them allowing you to select the 7 largest coefficients.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As @E_net4 commented, your continuous question is addressed in another post.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-07-28T17:08:11.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21785" PostTypeId="2" ParentId="21782" CreationDate="2017-07-28T21:01:55.630" Score="3" Body="&lt;p&gt;Pandas indexes differently:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X[some_slice]   # in Numpy is NOT equal to&#xA;df[some_slice]  # in Pandas, but is instead equal to&#xA;df.iloc[some_slice]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can cast your &lt;code&gt;features&lt;/code&gt; dataframe into numpy array by calling &lt;code&gt;.values&lt;/code&gt; on them right before splitting:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = \&#xA;    train_test_split(features.values, labels.values, test_size=0.4, random_state=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-07-28T21:01:55.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21786" PostTypeId="2" ParentId="18000" CreationDate="2017-07-28T23:42:52.627" Score="2" Body="&lt;p&gt;Recommender Systems are a huge topic of its own right and goes without saying, with a lot of research going on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://www.springer.com/us/book/9780387858203&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;book&quot;&gt;book&lt;/a&gt; does a deep-dive into recommender systems and may not be something you want, but it's helpful as a reference. It seems like you were unsure what those terms mean. The &lt;a href=&quot;http://ai.berkeley.edu/home.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Berekely AI Course&lt;/a&gt; covers most of these topics and their lectures are available for free.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be able to make better choices after going through some of the above materials.&lt;/p&gt;&#xA;" OwnerUserId="37258" LastActivityDate="2017-07-28T23:42:52.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21788" PostTypeId="1" CreationDate="2017-07-29T00:07:40.637" Score="4" ViewCount="493" Body="&lt;p&gt;As context, I am relatively new to the world of machine learning and I am attempting a project with a goal of classifying plays in an NBA game. My inputs are a sequence of 40 frames from each play in an NBA game and my labels are 11 all-encompassing classifications for a given play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plan is to take each sequence of frames and pass each frame into a CNN to extract a set of features. Then each sequence of features from a given video would be passed onto an RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using Keras for most of my implementation and I chose to use a VGG16 model for my CNN. Here is some of the relevant code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;video = keras.Input(shape = (None, 255, 255, 3), name = 'video')&#xA;cnn = keras.applications.VGG16(include_top=False, weights = None, input_shape=&#xA;(255,255,3), pooling = 'avg', classes=11)&#xA;cnn.trainable = True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is - would it still be beneficial for me to initialize the weights of the VGG16 ConvNet to 'imagenet' if my goal is to classify video clips of NBA games? If so, why? If not, how can I train the VGG16 ConvNet to get my own set of weights and then how can I insert them into this function? I have had little luck finding any tutorials where someone included their own set of weights when using the VGG16 model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I apologize if my questions seem naive but I would really appreciate any help in clearing this up.&lt;/p&gt;&#xA;" OwnerUserId="37259" LastActivityDate="2017-12-29T15:38:58.963" Title="Do I Need Pretrained Weights For Keras VGG16?" Tags="|machine-learning|python|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21789" PostTypeId="2" ParentId="21788" CreationDate="2017-07-29T00:17:58.460" Score="2" Body="&lt;p&gt;It would be beneficial to use pretrained weights trained on ImageNet because ImageNet is a huge dataset, and is fairly general. So you're more likely to get good image features from it. You might have to play around with from which layer you want to extract the features from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to create your own embeddings, you need to use the same code and instead of loading the weights, train it instead on the images you have. I do not recommend this for the following reasons&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You most likely have too small a dataset for the model to converge  &lt;/li&gt;&#xA;&lt;li&gt;VGG16 does classification, what will be the labels for the individual frames is a design choice and being a novice, it might be frustrating for you to get it right&lt;/li&gt;&#xA;&lt;li&gt;It takes a lot of time and resources, not an ideal situation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;P.S, if you still feel that ImageNet does not sufficiently cover your use case, you can try the pretrained Inception V3 model trained on the &lt;a href=&quot;https://github.com/openimages/dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open Images dataset&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="37258" LastEditorUserId="37258" LastEditDate="2017-11-27T07:56:49.270" LastActivityDate="2017-11-27T07:56:49.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21791" PostTypeId="2" ParentId="19987" CreationDate="2017-07-29T04:01:51.093" Score="1" Body="&lt;p&gt;If you are hand-coding, and found that you got the same prediction for everything in your test set, it is possible that you are multiplying feature probabilities until you hit the floating point limitations of your environment, ending up with a zero value, which probably matches one of your labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;e.g.,&lt;/em&gt; $0.001 * 0.0001 * 0.0002 * 0.0003 ... = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try adding log values instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;e.g.,&lt;/em&gt; $log(0.001) + log(0.0001) + log(0.0002) ... $  &lt;/p&gt;&#xA;" OwnerUserId="37262" LastEditorUserId="24000" LastEditDate="2017-07-30T11:59:07.697" LastActivityDate="2017-07-30T11:59:07.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21792" PostTypeId="2" ParentId="11589" CreationDate="2017-07-29T08:05:27.427" Score="9" Body="&lt;p&gt;Yes, there is a reason. It has to do with how you initialize your weights.&lt;/p&gt;&#xA;&lt;p&gt;There are 16 local minimums that have the highest probability of converging between 0.5 - 1.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1icYN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1icYN.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here is a paper that analyses the xor problem: &lt;a href=&quot;https://www.maths.stir.ac.uk/%7Ekjt/techreps/pdf/TR148.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning XOR: exploring the space of a&#xA;classic problem&lt;/a&gt;, Bland 1998.&lt;/p&gt;&#xA;" OwnerUserId="37268" LastEditorUserId="55122" LastEditDate="2023-04-02T14:22:17.460" LastActivityDate="2023-04-02T14:22:17.460" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="21794" PostTypeId="1" CreationDate="2017-07-29T08:37:45.703" Score="8" ViewCount="9591" Body="&lt;p&gt;I have trained decision tree . I also have a graph of the tree (&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HfLaS.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HfLaS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;) . Now i want to see which samples (red circled ones ) are under which leafs . I am using sklearn's implantation .  Is there any way to get the samples under a leaf ? ?    &lt;/p&gt;&#xA;" OwnerUserId="31072" LastActivityDate="2018-08-05T21:43:58.997" Title="Is there any way to get samples in under each leaf of a decision tree in Sklearn ?" Tags="|machine-learning|python|classification|scikit-learn|decision-trees|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21795" PostTypeId="1" CreationDate="2017-07-29T08:54:34.883" Score="1" ViewCount="419" Body="&lt;p&gt;I'm looking for a neural network architecture, which gives pretty good performance on an image classification challenge and is still simple to implement in a framework and also reasonably fast to train. Do you have suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new the neural network implementation, and in the end, I'd like to do some &quot;research-style&quot; playing with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, a pointer to an image classification data set which is not too tough for a simple network would be great. However, not something like MNIST which is so easy that almost any classifier does a good job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Which neural network framework do you recommend if I'd like to modify the way the network is trained?&lt;/p&gt;&#xA;" OwnerUserId="723" LastActivityDate="2017-07-29T10:59:23.823" Title="What is a simple image classification ANN architecture and a simple data set?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21798" PostTypeId="2" ParentId="21795" CreationDate="2017-07-29T10:59:23.823" Score="2" Body="&lt;p&gt;I'd advice going through cs231n course, which covers this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple but good architecture is using a feed forward net with one hidden layer and ReLU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for dataset - CIFAR-10 is good, if you don't like MNIST.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow is quite easy to use framework.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-07-29T10:59:23.823" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="21800" PostTypeId="1" CreationDate="2017-07-29T14:23:37.060" Score="1" ViewCount="592" Body="&lt;p&gt;I have set up a Q-learning problem in R, and would like some help with the theoretical correctness of my approach in framing the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem structure For this problem, the environment consists of 10 possible states. When in each state, the agent has 11 potential actions which it can choose from (these actions are the same regardless of the state which the agent is in). Depending on the particular state which the agent is in and the subsequent action which the agent then takes, there is a unique distribution for transition to a next state i.e. the transition probabilities to any next state are dependant on (only) the previous state as well as the action then taken.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each episode has 9 iterations i.e. the agent can take 9 actions and make 9 transitions before a new episode begins. In each episode, the agent will begin in state 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each episode, after each of the agent's 9 actions, the agent will get a reward which is dependant on the agent's (immediately) previous state and their (immediately) previous action as well as the state which they landed on i.e. the agent's reward structure is dependant on a state-action-state triplet (of which there will be 9 in an episode).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The transition probability matrix of the agent is static, and so is the reward matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have set up two learning algorithms. In the first, the q-matrix update happens after each action in each episode. In the second, the q-matrix is updated after each episode. The algorithm uses an epsilon greedy learning formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The big problem is that in my Q-learning, my agent is not learning. It gets less and less of a reward over time. I have looked into other potential problems such as simple calculation errors, or bugs in code, but I think that the problem lies with the conceptual structure of my q-learning problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have set up my Q-matrix as being a 10 row by 11 column matrix i.e. all the 10 states are the rows and the 11 actions are the columns. Would this be the best way to do so? This means that an agent is learning a policy which says that &quot;whenever you are in state x, do action y&quot;&#xA;Given this unique structure of my problem, would the standard Q-update still apply? i.e. &lt;code&gt;Q[cs,act]&amp;lt;&amp;lt;-Q[cs,act]+alpha*(Reward+gamma*max(Q[ns,])-Q[cs,act])&lt;/code&gt; Where &lt;code&gt;cs&lt;/code&gt; is current state; &lt;code&gt;act&lt;/code&gt; is action chosen; &lt;code&gt;Reward&lt;/code&gt; is the reward given your current state, your action chosen and the next state which you will transition to; &lt;code&gt;ns&lt;/code&gt; is the next state which you will transition to given your last state and last action (note that you transitioned to this state stochastically).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an open AI gym in R? Are there Q-learning packages for problems of this structure?&lt;/p&gt;&#xA;" OwnerUserId="37278" LastEditorUserId="836" LastEditDate="2017-07-29T17:05:21.143" LastActivityDate="2017-08-03T09:43:55.407" Title="Q-learning with a state-action-state reward structure and a Q-matrix with states as rows and actions as columns" Tags="|r|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21803" PostTypeId="2" ParentId="17227" CreationDate="2017-07-29T18:23:40.467" Score="1" Body="&lt;p&gt;I'm a neuroscientist, and I encountered what I think is a similar issue. Just to give a little background: the brain is divided in ~ 200k voxels. Using a technique known as DTI, fibers are sought. Either a fiber passes through a certain voxels (1), or it doesn't (0). Therefore, given a certain voxel, the connections pattern with the rest of the brain is a 200k vector; its components being mostly 0s, sometimes 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) either calculate the cross-correlation matrix between all vectors of your dataset, or the distance matrix; in both cases, you'll get a square matrix.&#xA;Which distance to choose: I would suggest the Jaccard distance, eminently suitable for binary, sparse feature vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b1) use the spectral reordering algorithm, as described in &lt;a href=&quot;http://www.pnas.org/content/suppl/2004/08/20/0403743101.DC1/03743SuppText.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.pnas.org/content/suppl/2004/08/20/0403743101.DC1/03743SuppText.pdf&lt;/a&gt; (read the paper too, it is wonderful) to separate your data into clusters (each square is going to be one), or at least to get an idea about the number of your clusters. Optionally, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;b2) apply k-means to cluster the rows (or columns; it doesn't matter, being the aforementioned cc/distance matrices symmetrical) of your matrix of choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HTH,&#xA;kind regards,&#xA;Luca&lt;/p&gt;&#xA;" OwnerUserId="37280" LastActivityDate="2017-07-29T18:23:40.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21804" PostTypeId="2" ParentId="21780" CreationDate="2017-07-29T21:34:26.290" Score="6" Body="&lt;p&gt;You're taking the &quot;Rule of 10&quot; too seriously.  It's a very rough rule of thumb.  It's not intended to be used like you are using it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you are thinking: &quot;I have only 70 positive instances, so by the Rule of 10, I'm only allowed to use 7 features; how do I choose which 7 features to use?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's not what the Rule of 10 means.  It's not some rule that specifies how many features you are permitted to use.  The Rule of 10 is descriptive, not prescriptive, and it's an approximate guideline: if the number of instances is much fewer than 10 times the number of features, you're at especially high risk of overfitting, and you might get poor results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what should you do?  You should do what you'd do anyway: use regularization, and use cross-validation to select the regularization hyper-parameters.  Also, it's important to have a hold-out test set that you don't touch until you've finalized everything about the classifier, to avoid overfitting and biased accuracy estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you can get more data, that would really help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, since you have imbalanced classes, you might consider reading about class imbalance and methods for dealing with it.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-07-29T21:34:26.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21805" PostTypeId="2" ParentId="20045" CreationDate="2017-07-29T22:27:13.367" Score="2" Body="&lt;p&gt;This is a good case for computer vision.  You know exactly what the icon looks like (basically pixel-perfect), and you want to find whether it exists anywhere in the screenshot.  This is an image matching problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-correlation#Normalized_cross-correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Normalized cross-correlation&lt;/a&gt; (NCC) is a standard way to check whether small image $I$ (your icon)  appears anywhere in the large image $I'$: you compute the NCC and then look for a location where the value of the NCC is large; that is your candidate match.  You can now apply this to the set of candidate icons: for each candidate icon, search whether it appears anywhere in the screenshot, using NCC.  OpenCV and other computer vision libraries should include normalized implementations of the NCC computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For speed, you can also try using &lt;a href=&quot;https://en.wikipedia.org/wiki/Pyramid_(image_processing)&quot; rel=&quot;nofollow noreferrer&quot;&gt;image pyramids&lt;/a&gt;, but try using NCC directly on the full screenshot first.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-07-29T22:27:13.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21809" PostTypeId="1" CreationDate="2017-07-30T13:03:37.460" Score="4" ViewCount="268" Body="&lt;p&gt;I am now searching for a long time on the internet and on papers for an answers of simple questions. Am I able to train a Support Vector Regression algorithm with different data sets? If yes, how is the approach called?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 10 times the same battery with different usage, temperature and capacity.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Usage and temperature are features &lt;code&gt;(x_i,i)&lt;/code&gt; and capacity is the output &lt;code&gt;(y_i,i)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Battery_1 till timepoint n: &lt;code&gt;[x_1,1 y_1,1; ... ;x_1,n y_1,n]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;li&gt;Battery_10 till timepoint n: &lt;code&gt;[x_10,1 y_10,1; ... ;x_10,n y_10,n]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now I want to train my SVR with these sets, where the samples within a set belong together. I want give the algorithm a set of usage and temperature where I don't know the capacity and my SVR should predict it as such : &lt;code&gt;x_d&lt;/code&gt; --&gt; &lt;code&gt;y_d&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your help and input.&lt;/p&gt;&#xA;" OwnerUserId="37298" LastEditorUserId="37298" LastEditDate="2017-07-31T09:54:21.313" LastActivityDate="2024-01-19T08:01:50.093" Title="Support Vector Regression trained with data sets" Tags="|machine-learning|predictive-modeling|svm|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21810" PostTypeId="1" CreationDate="2017-07-30T14:42:24.763" Score="1" ViewCount="220" Body="&lt;p&gt;I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I am intent on having a career in data science/analysis.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do have a history in mathematics; I finished calculus 3 in high school and finished a set theory course in college as well as advanced logic courses (symbolic, modal, predicate, etc), but I have never studied how to code, nor have I taken any statistics courses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any suggestions on how to best proceed?  I of course want to be as attractive to employers as possible.  I have no debt or family and am open and willing to take multiple years (and take on loans) studying for this profession.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would getting a math degree (with an emphasis in statistics and taking courses like linear algebra) be the right (or at least, good) way to go?  A local university offers a concentration in data science within the mathematics department, so I could do that as well. And in addition, taking courses in computer science to learn how to code?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I'm looking to push a reset button (well not entirely since I have a background in mathematics, and I believe my background in philosophy will help too) on my career path and will do whatever it takes.  I want a career with a future and with hopefully opportunity to choose where I work (city), and a career that fits with how my mind works (I believe I could thrive in this field).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Jobs that I have had since college will not help in any way towards future jobs in this field, so I will be fully leaning on future classes/preparation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance and for reading this partial ramble!  I'm motivated but don't know exactly what I should do.  I realize that there are many paths people take to get into this profession but would love advice for someone like me in my situation.&lt;/p&gt;&#xA;" OwnerUserId="37288" LastEditorUserId="37288" LastEditDate="2017-07-30T15:23:32.620" LastActivityDate="2017-07-31T15:46:55.020" Title="Advice on how to get started for someone a bit later in life?" Tags="|statistics|data|beginner|career|" AnswerCount="2" CommentCount="4" ClosedDate="2017-07-31T20:22:45.597" ContentLicense="CC BY-SA 3.0" />
  <row Id="21814" PostTypeId="2" ParentId="21810" CreationDate="2017-07-30T16:56:34.963" Score="2" Body="&lt;p&gt;I wouldn't advice you to go that route. You're saying you'd like to study for a few years and then start as a beginner (and what other than a beginner can you start at? or a graduate programme). But bear in mind by the time you're 37, there'll be 22 year olds who'll be starting as a beginner just like you, with perhaps comparable skills (though perhaps not your educational background). Why would an employer take on someone fifteen years older?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would you like also to be as attractive to employers as possible? Do you think that leads to anything other than employers wanting to &lt;em&gt;use&lt;/em&gt; you? And what do you think happens after you're used--unless you'll make damn sure it will not? You probably want to be attractive to either get approval, remuneration, or advance in your career, but these three don't go hand in hand. It's difficult to get them in equal measure, e.g., often to advance in your career you have to take extra risks, like resigning, or conversely, to get accepted for more jobs lower your expectations. I don't see why you'd like to be attractive to employers and assume that that'll take care of your success, it won't, or it might not. Why not, rather, think of working for yourself? Find a career in which you can achieve independence. Specially as you get older.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I think it's tricky to try and build a career in a field like data science that's new, 'moving ground' so to speak, and not quite well defined (for some), essentially by way of courses and university studies. If you don't have the computer industry background to know the [future] value of what you're sold, how are you gonna choose? There're plenty of places ready to sell you a course, and many will use vivid depictions of future success or 'professionalism' in order to attract you (because, for these academies and schools, you're the customer, and of course they want / need customers). How do you know you're not going to be spending time and money taking lessons in something that'll be perfectly useless? And why take courses? Since many of these topics, disciplines, and programming languages, are free to learn online? Or do you assume that studying at a certain place means that that place will take care of your career, or necessarily give you a good foundation? Why not, rather, find out how present-day data scientists got to where they are? I think, if you interview some, you'll probably find no two paths were alike. And you can't emulate any of them (because what worked when they got started is not necessarily what will work for you tomorrow).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me it sounds like you'd be more adept to teaching than at data science. This is because you sound like you like to learn, like to teach (to advance yourself and others, I'd like to think), and you say you have time to do it. I'd try and get into teaching through some way other than that PhD in Philosophy you mentioned. Maybe start your own academy, alone or with a partner, and organize yourself to teach others what you already know, or what you're capable of learning, or teach them how to learn, or at least how to strive (motivation). That would sound to me possibly like a better investment.&lt;/p&gt;&#xA;" OwnerUserId="3546" LastActivityDate="2017-07-30T16:56:34.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21817" PostTypeId="5" CreationDate="2017-07-31T02:16:32.240" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-07-31T02:16:32.240" LastActivityDate="2017-07-31T02:16:32.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21818" PostTypeId="4" CreationDate="2017-07-31T02:16:32.240" Score="0" Body="ggplot2 is an actively maintained open-source chart-drawing package for R, written by Hadley Wickham, based upon the principles of &quot;Grammar of Graphics&quot;.  It partially replaces R's basic plot and the lattice package, while providing a clean, powerful, orthogonal and fun API." OwnerUserId="35902" LastEditorUserId="35902" LastEditDate="2017-07-31T03:15:24.017" LastActivityDate="2017-07-31T03:15:24.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21820" PostTypeId="1" CreationDate="2017-07-31T03:13:10.383" Score="0" ViewCount="1102" Body="&lt;p&gt;In a Variational Autoencoders, z cannot be simply sampled from the output of the autoencoder directly since the network would not be differentiable. Instead, we have to sample from a normal distribution who's parameters are the output of the autoencoder, mu and stddev. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I So cannot understand how the network isn't differentiable (lets say we take z = mu + srd). So the reparameterization is not clear to me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be much appreciated!!&lt;/p&gt;&#xA;" OwnerUserId="32882" LastEditorUserId="32882" LastEditDate="2017-07-31T13:00:17.727" LastActivityDate="2017-08-29T07:38:10.880" Title="Training the Variational Autoencoder After applying the reparameterization trick" Tags="|deep-learning|training|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21821" PostTypeId="1" AcceptedAnswerId="21833" CreationDate="2017-07-31T03:21:08.783" Score="0" ViewCount="495" Body="&lt;p&gt;Here is my understanding of those 2 terms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hyper-parameter:&lt;/strong&gt; A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the number of neurons in each layer, etc. Some models don't have any hyper-parameters, like the linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Parameter:&lt;/strong&gt; A variable in which the training process will update. For instance, the weights of a Neural Network are parameters as they are updated as we train the network and there is no human intervention on the process. Another example would be the slope and the y-intercept in a simple linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, what would the &lt;em&gt;learning rate parameter&lt;/em&gt; ($\eta$) be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\Theta_{i+1} = \Theta_{i} + \eta \nabla J(\Theta_{i} )&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?&lt;/p&gt;&#xA;" OwnerUserId="34036" LastActivityDate="2017-07-31T12:46:47.873" Title="Can the learning rate be considered both a parameter AND a hyper-parameter?" Tags="|hyperparameter|parameter|definitions|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21828" PostTypeId="1" CreationDate="2017-07-31T11:45:38.337" Score="7" ViewCount="485" Body="&lt;p&gt;Given a binary classifier, is it always possible to explain why it has classified some input as a positive class ? And by that I mean, if we have a big set of features, is there a tool that says : &lt;em&gt;'For this output, these are the features that were the most responsible for labeling it as a positive' ?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks  ! &lt;/p&gt;&#xA;" OwnerUserId="33242" LastActivityDate="2020-08-20T17:06:04.107" Title="Explain output of a given classifier w.r.t features" Tags="|machine-learning|classification|feature-extraction|" AnswerCount="3" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21829" PostTypeId="2" ParentId="21828" CreationDate="2017-07-31T11:50:39.690" Score="3" Body="&lt;p&gt;Try out some of the examples in &lt;a href=&quot;https://github.com/TeamHG-Memex/eli5&quot; rel=&quot;nofollow noreferrer&quot;&gt;this library&lt;/a&gt;, which attempts to use machine learning to understand black box machine learning models in Python.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="98307" LastEditDate="2020-08-20T17:06:04.107" LastActivityDate="2020-08-20T17:06:04.107" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="21830" PostTypeId="2" ParentId="21828" CreationDate="2017-07-31T11:54:36.277" Score="0" Body="&lt;p&gt;Interesting, I haven't heard of such thing yet but correlation matrix between features and the target variable will give you idea to certain extend. If correlation is on higher side then most likely those feature will have higher say while predicting the target. If you are using python I would suggest you to visualize correlation matrix using seaborn here is sample code &lt;a href=&quot;http://seaborn.pydata.org/examples/many_pairwise_correlations.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32805" LastActivityDate="2017-07-31T11:54:36.277" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21833" PostTypeId="2" ParentId="21821" CreationDate="2017-07-31T12:46:47.873" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No it does not, because you are at all times controlling the learning rate without reference to the data. Typically both the learning rate &lt;em&gt;and&lt;/em&gt; the learning rate schedule are considered hyper-parameters, so by adding a learning rate schedule you have not transformed learning rate into a parameter, but instead &lt;em&gt;added&lt;/em&gt; a new hyper-parameter that may need tuning!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate is also is not &lt;em&gt;part of&lt;/em&gt; the model as it would be used in production, so it is a &lt;em&gt;training&lt;/em&gt; hyper-parameter. Although that is not a strict separation, this also applies to things that are neither hyper-parameters nor parameters, such as the current &lt;em&gt;&lt;a href=&quot;https://www.willamette.edu/~gorr/classes/cs449/momrate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;momentum&lt;/a&gt;&lt;/em&gt; values, which are used to improve training rate, and are usually stored in a some data structures alongside the model being trained. Likewise most adaptive learning rate structures (used in e.g. RMSProp, Adagrad etc) are in this category - neither model parameter nor hyper-parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be possible to argue that for a continuous online model, the learning rate could be a model parameter, as it is part of the production model being used to make predictions - provided it was still being used, but adaptive, depending on data seen to date, and/or current learning progress. Then if you had to feed in an &lt;em&gt;initial&lt;/em&gt; learning rate you could maybe say that the learning rate was both a parameter and hyper-parameter of the ANN model. Although you would likely have to explain the scenario in full to be understood, it would be more usual to just say you were using an adaptive learning rate.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-07-31T12:46:47.873" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21838" PostTypeId="2" ParentId="21794" CreationDate="2017-07-31T14:05:44.567" Score="1" Body="&lt;p&gt;Yes,&lt;br&gt;&#xA;Use the apply function on the fitted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.apply&quot; rel=&quot;nofollow noreferrer&quot;&gt;Documents&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mentioned leaf-index as the same as in the underlining _tree object(used for visualization). &lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-07-31T14:05:44.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21839" PostTypeId="2" ParentId="21810" CreationDate="2017-07-31T14:24:43.147" Score="4" Body="&lt;p&gt;I for myself wouldn't advise you to not take that route. I do agree with most of the points that were mentioned before but my conclusion is a slightly different one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to get into data science without any preceding field knowledge deserves respect and it is not impossible. Also, there is still a lack of qualified data scientist and I would predict that the demand is growing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But you should consider the following points:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Data scientist is probably the job in the statistics/IT area that requires the highest level of education. Ask yourself if this is really what your aiming at. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you choose to go on that route be aware that it will be extremely hard. Data science is so multifaceted and interdisciplinary that there is a lot to learn. Even if you'd have a couple of hours to study each day I'm afraid it would take years. Being able to work as a data scientist needs far more than an academical math knowledge.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You will have to compete with PhDs in their mid-twenties who already have years of experience in publishing and conducting machine-learning/math/programming projects. Employers will probably favor them over you.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If you don't care that it will be painful and want to start learning, I'd do the following:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Get basic knowledge of statistics, programming, and IT-infrastructure. You should be familiar with t-tests, ANOVAs, regressions and cluster-/classification techniques. You should start learning a statistical programming language. For starters I'd recommend &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; or &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python&lt;/a&gt;. Basic knowledge of another higher programming language is useful. Knowing basic programming principles such as object-oriented programming doesn't hurt either. I'd recommend to take a look at Java since a lot of big data tools such as &lt;a href=&quot;https://hive.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hive&lt;/a&gt;, &lt;a href=&quot;http://hadoop.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hadoop&lt;/a&gt; or &lt;a href=&quot;https://spark.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spark&lt;/a&gt; are Java-based frameworks. Look at these frameworks, too. Understand what they are used for. At last you should have at least basic knowledge of databases and know the difference between a relational and a non-relational database.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Don't spend money on courses. The internet is full of free sources. Just take into account who offers them. There are a lot of lecturers and professors from very good universities who provide online courses or share their lectures.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Read books. Again: Don't spend money. Search the internet for free high-quality materials (&lt;a href=&quot;http://www.kdnuggets.com/2017/04/10-free-must-read-books-machine-learning-data-science.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;see here for example&lt;/a&gt;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check out community sites such as &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt; and join competitions. This is probably as close as you will get to real-job tasks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Continue doing that for a couple of month and ask yourself again if this is really where you want to go. If yes, continue.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But realistically, here is what I would try to do:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Don't aim at becoming a data scientist right away.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider to start in a statistics-related job (e.g. controller, business analyst, marketing manager, game designer etc.) and work your way up. I strongly believe that such a strategy is more likely to be successful. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Because no matter what you do before starting in the industry, you will have to learn on the job.&lt;/p&gt;&#xA;" OwnerUserId="37345" LastEditorUserId="37345" LastEditDate="2017-07-31T15:46:55.020" LastActivityDate="2017-07-31T15:46:55.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21840" PostTypeId="1" CreationDate="2017-07-31T14:46:57.500" Score="3" ViewCount="243" Body="&lt;p&gt;As mentioned in the title I am attempting to search through 10,000 vectors with 8000 features each, all in Python. Currently I have the vectors saved in their own directories as pickled numpy arrays. The features were pulled from &lt;a href=&quot;https://arxiv.org/pdf/1604.00449.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; deep neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to this space, but I have heard about M-Trees, R-Trees, inverted tables, and hashing. Is any one of them better for such a large number of features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This implementation needs to be done quite rapidly and is just a prototype, so simplicity is valuable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="37346" LastActivityDate="2017-07-31T17:51:47.927" Title="Best method for similarity searching on 10,000 data points with 8,000 features each in Python?" Tags="|python|search|search-engine|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21841" PostTypeId="2" ParentId="21809" CreationDate="2017-07-31T15:04:19.227" Score="0" Body="&lt;p&gt;Why not concatenating the 10 datasets you have and simply creating one big dataset?&#xA;Alternatively you could train 10 seperate SVMs and let them vote (for examples: output = mean of the 10 SVMs).&lt;/p&gt;&#xA;" OwnerUserId="37330" LastActivityDate="2017-07-31T15:04:19.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21845" PostTypeId="2" ParentId="21840" CreationDate="2017-07-31T17:51:47.927" Score="2" Body="&lt;p&gt;There are two main paths:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Load all vectors into memory. If you are able to load vectors into memory, then you might be able to search the space with &quot;clever&quot; brute force. One such method is found in this &lt;a href=&quot;https://arxiv.org/abs/1102.3828&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Keep vectors on disk. If you follow this path, then you have to index the vectors. You are basically building a search engine. Common open source search engines are: &lt;a href=&quot;http://lucene.apache.org/solr/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Solr&lt;/a&gt; and &lt;a href=&quot;https://github.com/elastic/elasticsearch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elasticsearch&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It also depends on what kind of what kind of search is needed. A working definition of &quot;close&quot; vector is needed. The most common definitions of &quot;close&quot; vectors can be found &lt;a href=&quot;https://softwareengineering.stackexchange.com/questions/328714/how-to-efficiently-search-a-set-of-vectors-for-a-vector-which-is-the-closest-mat&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/facebookresearch/faiss&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fais&lt;/a&gt; is a new library for efficient similarity search of vectors. It is designed for many vectors (&gt; 1,000,000) each being relatively small (10s to 100s  of dimensions). It may or may not scale to your problem. If it does not scale because of high dimensionality, you can reduce dimensionality with &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction#Principal_component_analysis_.28PCA.29&quot; rel=&quot;nofollow noreferrer&quot;&gt;Principal component analysis (PCA)&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-07-31T17:51:47.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21848" PostTypeId="2" ParentId="19490" CreationDate="2017-07-31T19:23:48.337" Score="4" Body="&lt;p&gt;From my experience setting up GPU processing for R is hard, setting it up on a Windows machine is even harder. Additionally, GPU processing can only be used for very specific types of calculations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to setup GPU processing for the sake of it, then my answer is quite useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you however care about general performance optimization of your system and your code, I advise to check the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use Microsoft R Open instead of Base R because it automatically enables multicore processing on your machine.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Vectorize your code&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use libraries such as data.table instead of dataframes&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Avoid growing objects&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, R's performance depends strongly on your code quality. A very good summary on what you can and should do is provided in &lt;a href=&quot;http://www.burns-stat.com/pages/Tutor/R_inferno.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;R Inferno&lt;/a&gt; by Patrick Burns.&lt;/p&gt;&#xA;" OwnerUserId="37345" LastEditorUserId="37345" LastEditDate="2019-09-11T09:31:09.027" LastActivityDate="2019-09-11T09:31:09.027" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="21849" PostTypeId="1" CreationDate="2017-07-31T19:40:45.117" Score="0" ViewCount="3920" Body="&lt;p&gt;How can I calculate support/confidence/lift on a dataset in order to find frequent itemsets and determine association rules, in python? What would be the most effective method for predicting and offering recommendations on a test set of incomplete &quot;shopping carts&quot;? I am limited to the Anaconda distribution so I cant use packages such as orange3, etc. &lt;/p&gt;&#xA;" OwnerUserId="37301" LastActivityDate="2018-04-28T21:08:32.613" Title="Market-basket: calculating support/confidence/lift/rules" Tags="|python|market-basket-analysis|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21850" PostTypeId="7" CreationDate="2017-07-31T20:01:36.930" Score="0" Body="&lt;p&gt;&#xA;    On Stack Exchange, we believe the core moderators should come &lt;i&gt;from&lt;/i&gt; the community, and be elected by the community itself through popular vote. We hold regular elections to determine who these community moderators will be.&lt;/p&gt;&#xA;&lt;p&gt;&#xA;    Community moderators are &lt;a href=&quot;https://stackoverflow.blog/2009/05/18/a-theory-of-moderation/&quot;&gt;accorded the highest level of privilege on our community&lt;/a&gt;, and should themselves be exemplars of positive behavior and leaders within the community.&lt;/p&gt;&#xA;&lt;p&gt;&#xA;    Our general criteria for moderators is as follows:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;    &lt;li&gt;patient and fair&lt;/li&gt;&#xA;    &lt;li&gt;leads by example&lt;/li&gt;&#xA;    &lt;li&gt;shows respect for their fellow community members in their actions and words&lt;/li&gt;&#xA;    &lt;li&gt;open to some light but firm moderation to keep the community on track and resolve (hopefully) uncommon disputes and exceptions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&#xA;    Every election has three phases:&#xA;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;    &lt;li&gt;Nomination&lt;/li&gt;&#xA;    &lt;li&gt;Primary&lt;/li&gt;&#xA;    &lt;li&gt;Election&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&#xA;    Please participate in the moderator elections by voting, and perhaps even by nominating yourself to be a community moderator!&#xA;&lt;/p&gt;&#xA;" OwnerUserId="-1" LastEditorUserId="8879" LastEditDate="2022-02-15T15:47:31.820" LastActivityDate="2022-02-15T15:47:31.820" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21851" PostTypeId="5" CreationDate="2017-07-31T21:51:19.090" Score="0" Body="&lt;p&gt;Hyperparameters are model parameters that are defined before training begins whereas regular parameters are learned during the training process.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2020-06-01T11:36:06.337" LastActivityDate="2020-06-01T11:36:06.337" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21852" PostTypeId="4" CreationDate="2017-07-31T21:51:19.090" Score="0" Body="Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn." OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-08-01T02:32:36.343" LastActivityDate="2017-08-01T02:32:36.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21853" PostTypeId="6" CreationDate="2017-07-31T22:12:37.527" Score="0" Body="&lt;p&gt;Hi All,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My name is Kasra, originally from Iran and currently living in Germany. I am a Machine Learning Scientist by training and my research has been focused on Graphs and Complex Networks during the last 3 years of my academic activity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did my masters in Machine Learning at University of Bonn, Germany and did 2 years of my PhD at Space Physics Group, Institute of Geophysics at University of Cologne in Germany. At the moment I work as a senior Machine Learning Scientist at areto-consulting in Cologne.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to contribute to the community as a moderator as I think I can bring practice and theory together in SE discussions and help the community stand at a rational place between practical Data Science and theoretical ML/PR/DM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more info you may check my &lt;a href=&quot;http://kasramsh.wixsite.com/kasra&quot; rel=&quot;nofollow noreferrer&quot;&gt;website&lt;/a&gt; or my &lt;a href=&quot;https://www.linkedin.com/in/kasramani84/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LinkedIn&lt;/a&gt; profile which are a bit out-dated but still says something about me! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish for final elected candidates a bright, joyful and fruitful way ahead in advance!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2017-07-31T22:12:37.527" LastActivityDate="2017-07-31T22:12:37.527" CommentCount="2" CommunityOwnedDate="2017-07-31T22:12:37.527" ContentLicense="CC BY-SA 3.0" />
  <row Id="21854" PostTypeId="5" CreationDate="2017-07-31T22:31:31.130" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-07-31T22:31:31.130" LastActivityDate="2017-07-31T22:31:31.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21855" PostTypeId="4" CreationDate="2017-07-31T22:31:31.130" Score="0" Body="Feature scaling is a data pre-processing step where the range of variable values is standardized. Standardization of datasets is a common requirement for many machine learning algorithms. Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value." OwnerUserId="24000" LastEditorUserId="24000" LastEditDate="2017-08-01T07:45:09.210" LastActivityDate="2017-08-01T07:45:09.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21856" PostTypeId="2" ParentId="21849" CreationDate="2017-08-01T00:52:36.180" Score="0" Body="&lt;p&gt;Python package &lt;a href=&quot;https://orange3-associate.readthedocs.io/en/latest/scripting.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange3-Associate&lt;/a&gt;, which contains functions for mining association rules and seems to be what you are referring to, should be able to be installed on Anaconda's Python distribution with Python's internal &lt;code&gt;pip&lt;/code&gt; command, i.e.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install orange3-associate&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-08-01T00:52:36.180" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21857" PostTypeId="6" CreationDate="2017-08-01T02:42:47.613" Score="0" Body="&lt;p&gt;Hi, my name is Raj and I'm running to become one of your mods. I work as a Sr. Data Scientist at my day job, while leading communities like Pydata Delhi for helping and educate Data Science and ML enthusiasts via talks and workshops.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I would be a good choice as the moderator of Data Science SE, as I have been one of the most active members on this site since the early beta days. I am &lt;a href=&quot;https://datascience.stackexchange.com/users?tab=Reputation&amp;amp;filter=all&quot;&gt;one of the top contributors&lt;/a&gt; in this community [#4 currently], and one of the most active members on the Meta site, helping users with their queries and suggestions about the main site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been the top reviewer in all the review queues after our current moderator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also been moderating the beta site of &lt;a href=&quot;https://devops.stackexchange.com/&quot;&gt;Devops StackExchange&lt;/a&gt; as their Moderator Pro-Tempore. So, I already have experience handling the diamond responsibilities as the moderator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.meta.stackexchange.com/a/2294/11097&quot;&gt;Link to my answers&lt;/a&gt; for Moderator questionnaire.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-08-01T09:27:32.190" LastActivityDate="2017-08-01T09:27:32.190" CommentCount="0" CommunityOwnedDate="2017-08-01T02:42:47.613" ContentLicense="CC BY-SA 3.0" />
  <row Id="21859" PostTypeId="2" ParentId="16843" CreationDate="2017-08-01T03:57:12.710" Score="1" Body="&lt;p&gt;To clarify (for people like myself who are learning from scratch and need basic explanations), what Wikipedia means (if you look through the source) is that the learning rate does not affect eventual convergence, assuming the learning rate is between 0 and 1. A learning rate too large (example: consider an infinite learning rate where the weight vector immediately becomes the training case) can fail to converge to a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate can, however, affect the speed at which you reach convergence (as mentioned in the other answers).&lt;/p&gt;&#xA;" OwnerUserId="37371" LastActivityDate="2017-08-01T03:57:12.710" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21862" PostTypeId="2" ParentId="410" CreationDate="2017-08-01T06:33:15.327" Score="10" Body="&lt;p&gt;Copy-pasted from &lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf&quot; rel=&quot;noreferrer&quot;&gt;my masters thesis&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the loss does not decrease for several epochs, the learning rate might be too low.&#xA;The optimization process might also be stuck in a local minimum.&lt;/li&gt;&#xA;&lt;li&gt;Loss being NAN might be due to too high learning rates. Another reason is division&#xA;by zero or taking the logarithm of zero.&lt;/li&gt;&#xA;&lt;li&gt;Weight update tracking: Andrej Karpathy proposed in the 5th lecture of CS231n to track weight updates to check if&#xA;the learning rate is well-chosen. He suggests that the weight update should be in the order&#xA;of 10−3. If the weight update is too high, then the learning rate has to be decreased. If the&#xA;weight update is too low, then the learning rate has to be increased.&lt;/li&gt;&#xA;&lt;li&gt;Typical learning rates are in [0.1, 0.00001]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-08-01T06:33:15.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21863" PostTypeId="1" CreationDate="2017-08-01T07:39:28.510" Score="0" ViewCount="967" Body="&lt;p&gt;I have build a naive bayes model for text classification.It is predicting correctly.But it is returning 'NA' in prediction results if i put 'type = raw'.i have seen some results in stackoverflow to add some noise.when i do that i am getting all A category as 0's and all B category as 1's.How can i get correct probabilities in naive bayes?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library('tm');&#xA;library('e1071');&#xA;library('SparseM');&#xA;Sample_data &amp;lt;- read.csv(&quot;products.csv&quot;);&#xA;traindata &amp;lt;- as.data.frame(Sample_data[1:60,c(1,2)]);&#xA;testdata &amp;lt;- as.data.frame(Sample_data[61:80,c(1,2)]);&#xA;trainvector &amp;lt;- as.vector(traindata$Description);&#xA;testvector &amp;lt;- as.vector(testdata$Description);&#xA;trainsource &amp;lt;- VectorSource(trainvector);&#xA;testsource &amp;lt;- VectorSource(testvector);&#xA;traincorpus &amp;lt;- Corpus(trainsource);&#xA;testcorpus &amp;lt;- Corpus(testsource);&#xA;traincorpus &amp;lt;- tm_map(traincorpus,stripWhitespace);&#xA; traincorpus &amp;lt;- tm_map(traincorpus,tolower);&#xA; traincorpus &amp;lt;- tm_map(traincorpus, removeWords,stopwords(&quot;english&quot;));&#xA;traincorpus&amp;lt;- tm_map(traincorpus,removePunctuation);&#xA; testcorpus &amp;lt;- tm_map(testcorpus,stripWhitespace);&#xA; testcorpus &amp;lt;- tm_map(testcorpus,tolower);&#xA; testcorpus &amp;lt;- tm_map(testcorpus, removeWords,stopwords(&quot;english&quot;));&#xA; testcorpus&amp;lt;- tm_map(testcorpus,removePunctuation);&#xA;trainmatrix &amp;lt;- t(TermDocumentMatrix(traincorpus));&#xA;testmatrix &amp;lt;- t(TermDocumentMatrix(testcorpus));&#xA;model &amp;lt;- naiveBayes(as.matrix(trainmatrix),as.factor(traindata$Group));&#xA;results &amp;lt;- predict(model,as.matrix(testmatrix))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35449" LastEditorUserId="35449" LastEditDate="2017-08-02T05:37:47.953" LastActivityDate="2024-01-21T12:01:35.220" Title="Naive Bayes Predict type = 'raw' returning NA" Tags="|machine-learning|r|naive-bayes-classifier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21864" PostTypeId="1" CreationDate="2017-08-01T08:25:35.373" Score="1" ViewCount="174" Body="&lt;p&gt;In Software Engineering, when an application gets developed it would be pushed into the testing environment to make sure that the application is reliable and doesn't have any bugs. Mostly this will be done by another User other than the Developer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does it work in Data Science? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do we have a testing Environment in industries as well before the model goes into Production? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, what will a sample test case look like and will there be a separate tester or is it the developer doing different tests like Gini Test or the T-test and evaluate his results against AUC, Recall and Precision ?&lt;/p&gt;&#xA;" OwnerUserId="37014" LastActivityDate="2017-08-01T08:25:35.373" Title="Testing Environment for Data Science Models in Industries?" Tags="|data-mining|predictive-modeling|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21866" PostTypeId="1" CreationDate="2017-08-01T08:47:11.537" Score="1" ViewCount="54" Body="&lt;p&gt;I'm just starting out with learning about Data Science and programming through various online course. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was thinking a good first project would be a to create a log of what I'm learning and how long for and then work out ways that I can visualise this for use in a CV further down the road.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was wanting to get a rough outline for how you would set a project like this up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was thinking of something like a simple site where I click on the learning resource I used and it asks for the amount of time I studied for and then records the data and time. Maybe more inputs for tags and percentage complete etc... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently learning R and Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So grateful for any ideas or advise you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers &lt;/p&gt;&#xA;" OwnerUserId="37380" LastActivityDate="2017-08-31T10:20:13.573" Title="Recording Data of my learning progress" Tags="|python|r|data|beginner|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21868" PostTypeId="2" ParentId="21866" CreationDate="2017-08-01T09:05:27.000" Score="1" Body="&lt;p&gt;I think the first point to address is how employers will know you are being honest about your logging. Maybe a better use of your time is to not put in number of hours, but projects and code references you've made so that the output is more concrete rather than just a number of hours.  The problem is I know plenty of people who put in 1 hour and learn what a normal person would learn in 5 hours. Think about the WOW factor in this.. what would be convincing to employers?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if its only you that are logging in hours and nobody else is going to see or use that program, you might as well use excel while focusing on whatever project you're passionate about.&lt;/p&gt;&#xA;" OwnerUserId="37109" LastActivityDate="2017-08-01T09:05:27.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21869" PostTypeId="2" ParentId="21809" CreationDate="2017-08-01T09:06:14.820" Score="0" Body="&lt;p&gt;Add one more feature in dataset which will identify battery, with this your dataset will have 3 features battery,usage, temperature and one target variable capacity. Test this out if you are losing the information then create different models for each data set.&lt;/p&gt;&#xA;" OwnerUserId="32805" LastActivityDate="2017-08-01T09:06:14.820" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21870" PostTypeId="1" AcceptedAnswerId="21881" CreationDate="2017-08-01T09:16:43.810" Score="1" ViewCount="100" Body="&lt;p&gt;I have a biological unbalanced dataset on which I have applied deep learning, Support Vector Machine (all the kernel functions) and Artificial Neural network for multiclass classification (size: 139 samples , 5 attributes) in python. Unfortunately the accuracy is not exceeding 55%. What can be done to increase the accuracy? If a dataset cannot ever go beyond such an average accuracy, what is the solution?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2021-02-19T07:46:11.533" Title="What can be done to increase the accuracy of a biological dataset?" Tags="|classification|deep-learning|scikit-learn|multiclass-classification|class-imbalance|" AnswerCount="2" CommentCount="2" ClosedDate="2017-08-01T23:28:12.197" ContentLicense="CC BY-SA 3.0" />
  <row Id="21871" PostTypeId="1" CreationDate="2017-08-01T09:22:53.650" Score="8" ViewCount="1959" Body="&lt;p&gt;With a non-differentiable operation, such as a minimization, how does &lt;strong&gt;Tensorflow&lt;/strong&gt; compute the gradients? Some kind of &lt;em&gt;soft-minimum approximation&lt;/em&gt;? &#xA;If so, can I retrieve the &lt;strong&gt;analytical computation&lt;/strong&gt; for a specific gradient?&lt;/p&gt;&#xA;" OwnerUserId="28685" LastActivityDate="2017-08-01T14:32:43.960" Title="How does Tensorflow compute gradients of reduce_min operation?" Tags="|machine-learning|python|tensorflow|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21872" PostTypeId="1" CreationDate="2017-08-01T11:24:54.050" Score="-1" ViewCount="1174" Body="&lt;p&gt;I am going throught GAN for image generation and I am using this &lt;a href=&quot;https://medium.com/towards-data-science/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; for reference. The author is creating a generator model which does this. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/YxaAT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YxaAT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the generator model code is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.G = Sequential()&#xA;dropout = 0.4&#xA;depth = 64+64+64+64&#xA;dim = 7&#xA;&#xA;# In: 100&#xA;# Out: dim x dim x depth&#xA;self.G.add(Dense(dim*dim*depth, input_dim=100))&#xA;self.G.add(BatchNormalization(momentum=0.9))&#xA;self.G.add(Activation('relu'))&#xA;self.G.add(Reshape((dim, dim, depth)))&#xA;self.G.add(Dropout(dropout))&#xA;&#xA;&#xA;# In: dim x dim x depth&#xA;# Out: 2*dim x 2*dim x depth/2&#xA;self.G.add(UpSampling2D())&#xA;self.G.add(Conv2DTranspose(int(depth/2), 5, padding='same'))&#xA;self.G.add(BatchNormalization(momentum=0.9))&#xA;self.G.add(Activation('relu'))&#xA;self.G.add(UpSampling2D())&#xA;self.G.add(Conv2DTranspose(int(depth/4), 5, padding='same'))&#xA;self.G.add(BatchNormalization(momentum=0.9))&#xA;self.G.add(Activation('relu'))&#xA;self.G.add(Conv2DTranspose(int(depth/8), 5, padding='same'))&#xA;self.G.add(BatchNormalization(momentum=0.9))&#xA;self.G.add(Activation('relu'))&#xA;&#xA;# Out: 28 x 28 x 1 grayscale image [0.0,1.0] per pix&#xA;self.G.add(Conv2DTranspose(1, 5, padding='same'))&#xA;self.G.add(Activation('sigmoid'))&#xA;self.G.summary()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understood most of the codes but I have doubts in these two functions&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.G.add(UpSampling2D())&#xA;self.G.add(Conv2DTranspose(int(depth/number), 5, padding='same'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is happeing in those layers?&lt;/p&gt;&#xA;" OwnerUserId="21471" LastEditorUserId="29169" LastEditDate="2021-02-16T00:44:20.437" LastActivityDate="2021-02-16T00:44:20.437" Title="How is the generator code works in a GAN?" Tags="|machine-learning|keras|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21875" PostTypeId="2" ParentId="21872" CreationDate="2017-08-01T12:48:55.063" Score="1" Body="&lt;p&gt;You should either use UpSampling2D or Conv2DTranspose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upsampling2D just repeats the input. After Upsampling you should add a normal convolutional layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conv2DTranspose is the backpropagated Conv2D, and there you do not neet any upsampling &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For both cases read the documentary.&#xA;&lt;a href=&quot;https://arxiv.org/pdf/1603.07285v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1603.07285v1.pdf&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="37330" LastActivityDate="2017-08-01T12:48:55.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21876" PostTypeId="2" ParentId="21870" CreationDate="2017-08-01T12:55:47.010" Score="1" Body="&lt;p&gt;There are many possible explanations.&#xA;Maybe just mistakes in the code or maybe there is just no better separation possible.&#xA;When usign SVM the features are extremely important and you should most of your time invest in choosing/designing them.&#xA;The statement &amp;quot;I have applied Deep learning&amp;quot; ist just to vague and therefore I can not give any tip.&#xA;Third when dealing with unbalanced data you have many possible actions.&#xA;I recommend reading:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Oversampling and undersampling in data analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37330" LastEditorUserId="85045" LastEditDate="2021-02-19T07:46:11.533" LastActivityDate="2021-02-19T07:46:11.533" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="21877" PostTypeId="1" AcceptedAnswerId="21888" CreationDate="2017-08-01T13:20:51.307" Score="37" ViewCount="83844" Body="&lt;p&gt;I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My methodology is this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split my data into training/test.&lt;/li&gt;&#xA;&lt;li&gt;Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.&lt;/li&gt;&#xA;&lt;li&gt;I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It is at this stage that I see strange behaviour and I'm unsure how to proceed. &lt;strong&gt;Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search&lt;/strong&gt;, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. &lt;strong&gt;Or, do I simply take the output GridSearchCV object as the new estimator&lt;/strong&gt;? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&#xA;So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?&lt;/p&gt;&#xA;" OwnerUserId="33603" LastEditorUserId="33603" LastEditDate="2017-08-01T13:58:29.793" LastActivityDate="2020-11-04T10:06:52.947" Title="How to use the output of GridSearch?" Tags="|machine-learning|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21878" PostTypeId="2" ParentId="21877" CreationDate="2017-08-01T13:39:32.960" Score="3" Body="&lt;p&gt;GridSearchCV lets you combine an estimator with a grid search preamble to tune hyper-parameters. The method picks the optimal parameter from the grid search and uses it with the estimator selected by the user. GridSearchCV inherits the methods from the classifier, so yes, you can use the .score, .predict, etc.. methods directly through the GridSearchCV interface. If you wish to extract the best hyper-parameters identified by the grid search you can use .best_params_ and this will return the best hyper-parameter. You can then pass this hyper-parameter to your estimator separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using .predict directly will yield the same results as getting the best hyper-parameter through .best_param_ and then using it in your model. By understanding the underlining workings of grid search we can see why this is the case.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;Grid Search&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This technique is used to find the optimal parameters to use with an algorithm. This is NOT the weights or the model, those are learned using the data. This is obviously quite confusing so I will distinguish between these parameters, by calling one hyper-parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hyper-parameters are like the k in k-Nearest Neighbors (k-NN). k-NN requires the user to select which neighbor to consider when calculating the distance. The algorithm then tunes a parameter, a threshold, to see if a novel example falls within the learned distribution, this is done with the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do we choose k?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people simply go with recommendations based on past studies of the data type. Others use grid search. This method will be able to best determine which k is the optimal to use for your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How does it work?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First you need to build a grid. This is essentially a set of possible values your hyper-parameter can take. For our case we can use $[1, 2, 3, ..., 10]$. Then you will train your k-NN model for each value in the grid. First you would do 1-NN, then 2-NN, and so on. For each iteration you will get a performance score which will tell you how well your algorithm performed using that value for the hyper-parameter. After you have gone through the entire grid you will select the value that gave the best performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This goes against the principles of not using test data!!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would be absolutely right. That is the reason grid search is often mixed with cross-validation. So that we keep the test data completely separate until we are truly satisfied with our results and are ready to test. $n$-fold cross-validation takes a training set and separates it into $n$ parts. It then trains on $n-1$ folds and tests on the fold which was left out. For each value in the grid, the algorithm will be retrained $n$ times, for each fold being left out. Then the performance across each fold is averaged and that is the achieved performance for that hyper-parameter value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The selected hyper-parameter value is the one which achieves the highest average performance across the n-folds. Once you are satisfied with your algorithm, then you can test it on the testing set. If you go straight to the testing set then you are risking overfitting. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2017-08-01T18:12:35.423" LastActivityDate="2017-08-01T18:12:35.423" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="21880" PostTypeId="2" ParentId="11077" CreationDate="2017-08-01T14:12:29.353" Score="4" Body="&lt;p&gt;There are a number of implementations of Word2Vec, but most assume the basic unit to be 'sentences' - though they don't care what those sentences look like. If you are using something like &lt;code&gt;gensim&lt;/code&gt; you will need each sentence in its own list, and each sentence will be a list of tokens. If you are using another package you may be reading from the disk in which case you likely need all documents concatenated with each newline representing a new sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fundamental consideration in deciding on the size of a 'sentence' (whether it's the entire note, all a patient's notes, or a single sentence) is what surrounding words should be used to train the model. The W2V model will consider all words within a certain distance of the target word when tuning its vector representation (regardless of if it's &lt;code&gt;cbow&lt;/code&gt; or &lt;code&gt;skipgram&lt;/code&gt;), but will not look beyond a 'sentence' boundary. So if your fundamental unit is a document not a sentence, associations will bleed across sentences (this may or may not be what you want).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, consider the nature of the documents first. Patient notes can be messy and full of automatically generated text, so it can be important to strip out or replace certain strings. Similarly, they often use whitespace and newlines rather than punctuation. You might want to consider treating existing whitespace as the end of a sentence and you might want to use a parser to replace dates, names, etc with default tokens to increase generalizability.&lt;/p&gt;&#xA;" OwnerUserId="12473" LastActivityDate="2017-08-01T14:12:29.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21881" PostTypeId="2" ParentId="21870" CreationDate="2017-08-01T14:19:29.610" Score="1" Body="&lt;p&gt;&lt;strong&gt;A shallow neural network is the wrong approach for a problem with a small training set.&lt;/strong&gt; Deep learning is even worse for small training sets. 139 samples is severely insufficient to train any deep learning model, or even a shallow neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a very general rule of thumb I use 100 examples for each feature in my dataset for deep learning. This then increases exponentially with every single different class you expect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you use a machine learning technique such as SVM. This will likely result in better results given the size of your dataset. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have an unbalanced dataset then you would want to use anomaly detection algorithms which can be trained on a single distribution. You can learn the distribution of each output class you want. From there, novel examples can be classified based on the likelihood they fit within a given distribution. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-08-01T14:19:29.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21882" PostTypeId="2" ParentId="21871" CreationDate="2017-08-01T14:32:43.960" Score="3" Body="&lt;p&gt;A minimum operation &lt;em&gt;is&lt;/em&gt; differentiable, or at least you can easily express the partial derivatives w.r.t. its inputs:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;$f = min(x_1, x_2, x_3 ... x_n)$&lt;/h2&gt;&#xA;&#xA;&lt;h2&gt;$ \frac{\partial f}{\partial x_i} = &#xA;\begin{cases}&#xA;    1,&amp;amp; \text{if } argmin_i(x_i) = i\\&#xA;    0,              &amp;amp; \text{otherwise}&#xA;\end{cases}$&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This does not hold strictly when multiple values share the same minimum value. However, that is not a problem in practice for gradient-based optimisers in TensorFlow, which can simply set all tied indices to have partial derivative of 1 (or a fraction $\frac{1}{n_{min}}$), with little impact to the eventual result, because ties for values will happen rarely. Ties may happen frequently enough in a ReLU-based network that the TensorFlow developers have considered a best response for them - I don't know specifically what TensorFlow does for that situation.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-08-01T14:32:43.960" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21883" PostTypeId="1" AcceptedAnswerId="21884" CreationDate="2017-08-01T15:06:11.937" Score="1" ViewCount="2137" Body="&lt;p&gt;I am using numpy to implement some neural network tutorials. There is a constant(&quot;eta&quot;) used in codes. What does it means and what it stands for?&lt;/p&gt;&#xA;" OwnerUserId="37395" LastActivityDate="2017-08-01T15:08:52.977" Title="what does eta constant mean in numpy" Tags="|python|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21884" PostTypeId="2" ParentId="21883" CreationDate="2017-08-01T15:08:52.977" Score="2" Body="&lt;p&gt;Usually &quot;eta&quot; means learning rate, but it would be better if you could show an example.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-08-01T15:08:52.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21887" PostTypeId="1" AcceptedAnswerId="21899" CreationDate="2017-08-01T16:54:59.527" Score="6" ViewCount="1561" Body="&lt;p&gt;Keras has a problem with the input dimension. My first layer looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(128, batch_size=1, input_shape=(150,), kernel_initializer=&quot;he_uniform&quot;, kernel_regularizer=regularizers.l2(0.01), activation=&quot;elu&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see the input dimension should be (150,) and with the fixed batch_size it is (1, 150)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data has dimension (150,) and could be for example a numpy array with 150 zeros.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;old_qval = model.predict(old_state_m)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I call the model to make a prediction. Normally Keras should automatically add the batch size as an extra dimension so I should end up with (1, 150) which would work. But Keras adds the dimension for the batch size at the wrong place and I end up with (150, 1). I tried tensorflow and theano backend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have a bug in my code or is it a problem with Keras?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I fix the problem? I could reshape my input data but it already has the needed shape of (150,) and should be fine. What else could I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I should provide more data or code feel free to ask.&lt;/p&gt;&#xA;" OwnerUserId="37300" LastEditorUserId="836" LastEditDate="2017-08-02T11:29:50.797" LastActivityDate="2017-08-02T12:17:05.573" Title="Keras input dimension bug?" Tags="|python|neural-network|deep-learning|keras|q-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21888" PostTypeId="2" ParentId="21877" CreationDate="2017-08-01T17:20:30.727" Score="45" Body="&lt;p&gt;Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.&lt;/p&gt;&#xA;&lt;p&gt;The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;clf = GridSearchCV(estimator=RandomForestClassifier(), &#xA;                    param_grid=parameter_candidates,&#xA;                    cv=5,&#xA;                    refit=True,&#xA;                    error_score=0,&#xA;                    n_jobs=-1)&#xA;&#xA;clf.fit(training_set, training_classifications)&#xA;optimised_random_forest = clf.best_estimator_&#xA;return optimised_random_forest&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Will return a RandomForestClassifier. This is all pretty clear from the [documentation][1]. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;clf = GridSearchCV(estimator=RandomForestClassifier(), &#xA;                    param_grid=parameter_candidates,&#xA;                    cv=5,&#xA;                    refit=True,&#xA;                    error_score=0,&#xA;                    n_jobs=-1)&#xA;&#xA;clf.fit(training_set, training_classifications)&#xA;return clf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Optimised Random Forest Accuracy:  0.916970802919708&#xA;[[139  47]&#xA; [ 44 866]]&#xA;GridSearchCV Accuracy:  0.916970802919708&#xA;[[139  47]&#xA; [ 44 866]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function.&#xA;[1]: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit&quot; rel=&quot;noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33603" LastEditorUserId="33603" LastEditDate="2020-11-04T10:06:52.947" LastActivityDate="2020-11-04T10:06:52.947" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21889" PostTypeId="1" CreationDate="2017-08-01T18:45:53.433" Score="2" ViewCount="2435" Body="&lt;p&gt;I am building a neural network to learn to recognize handwritten digits from MNIST. I have confirmed that backpropagation calculates the gradients perfectly (gradient checking gives error &amp;lt; 10 ^ -10).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears that no matter how I train the weights, the cost function always tends towards around 3.24-3.25 (never below that, just approaching from above) and the training/test set accuracy is very low (around 11% for the test set). It appears that the h values in the end are all very close to 0.1 and to each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot find why my program cannot produce better results. I was wondering if anyone could maybe take a look at my code and please tell me any reasons for this occurring. Thank you so much for all your help, I really appreciate it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my Python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import math&#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;&#xA;# Neural network has four layers&#xA;# The input layer has 784 nodes&#xA;# The two hidden layers each have 5 nodes&#xA;# The output layer has 10 nodes&#xA;num_layer = 4&#xA;num_node = [784,5,5,10]&#xA;num_output_node = 10&#xA;&#xA;# 30000 training sets are used&#xA;# 10000 test sets are used&#xA;# Can be adjusted&#xA;Ntrain = 30000&#xA;Ntest = 10000&#xA;&#xA;# Sigmoid Function&#xA;def g(X):&#xA;    return 1/(1 + np.exp(-X))&#xA;&#xA;# Forwardpropagation&#xA;def h(W,X):&#xA;    a = X&#xA;    for l in range(num_layer - 1):&#xA;        a = np.insert(a,0,1)&#xA;        z = np.dot(a,W[l])&#xA;        a = g(z)&#xA;    return a      &#xA;&#xA;# Cost Function&#xA;def J(y, W, X, Lambda):&#xA;    cost = 0&#xA;    for i in range(Ntrain):&#xA;        H = h(W,X[i])&#xA;        for k in range(num_output_node):            &#xA;            cost = cost + y[i][k] * math.log(H[k]) + (1-y[i][k]) * math.log(1-H[k])&#xA;    regularization = 0&#xA;    for l in range(num_layer - 1):&#xA;        for i in range(num_node[l]):&#xA;            for j in range(num_node[l+1]):&#xA;                regularization = regularization + W[l][i+1][j] ** 2&#xA;    return (-1/Ntrain * cost + Lambda / (2*Ntrain) * regularization)&#xA;&#xA;# Backpropagation - confirmed to be correct&#xA;# Algorithm based on https://www.coursera.org/learn/machine-learning/lecture/1z9WW/backpropagation-algorithm&#xA;# Returns D, the value of the gradient&#xA;def BackPropagation(y, W, X, Lambda):&#xA;    delta = np.empty(num_layer-1, dtype = object)&#xA;    for l in range(num_layer - 1):&#xA;        delta[l] = np.zeros((num_node[l]+1,num_node[l+1]))&#xA;    for i in range(Ntrain):&#xA;        A = np.empty(num_layer-1, dtype = object)&#xA;        a = X[i]&#xA;        for l in range(num_layer - 1):&#xA;            A[l] = a&#xA;            a = np.insert(a,0,1)&#xA;            z = np.dot(a,W[l])&#xA;            a = g(z)&#xA;        diff = a - y[i]&#xA;        delta[num_layer-2] = delta[num_layer-2] + np.outer(np.insert(A[num_layer-2],0,1),diff)&#xA;        for l in range(num_layer-2):&#xA;            index = num_layer-2-l&#xA;            diff = np.multiply(np.dot(np.array([W[index][k+1] for k in range(num_node[index])]), diff), np.multiply(A[index], 1-A[index])) &#xA;            delta[index-1] = delta[index-1] + np.outer(np.insert(A[index-1],0,1),diff)&#xA;    D = np.empty(num_layer-1, dtype = object)&#xA;    for l in range(num_layer - 1):&#xA;        D[l] = np.zeros((num_node[l]+1,num_node[l+1]))&#xA;    for l in range(num_layer-1):&#xA;        for i in range(num_node[l]+1):&#xA;            if i == 0:&#xA;                for j in range(num_node[l+1]):&#xA;                    D[l][i][j] = 1/Ntrain * delta[l][i][j]&#xA;            else:&#xA;                for j in range(num_node[l+1]):&#xA;                    D[l][i][j] = 1/Ntrain * (delta[l][i][j] + Lambda * W[l][i][j])&#xA;    return D&#xA;&#xA;# Neural network - this is where the learning/adjusting of weights occur&#xA;# W is the weights&#xA;# learn is the learning rate&#xA;# iterations is the number of iterations we pass over the training set&#xA;# Lambda is the regularization parameter&#xA;def NeuralNetwork(y, X, learn, iterations, Lambda):&#xA;&#xA;    W = np.empty(num_layer-1, dtype = object)&#xA;    for l in range(num_layer - 1):&#xA;        W[l] = np.random.rand(num_node[l]+1,num_node[l+1])/100&#xA;    for k in range(iterations):&#xA;        print(J(y, W, X, Lambda))&#xA;        D = BackPropagation(y, W, X, Lambda)&#xA;        for l in range(num_layer-1):&#xA;            W[l] = W[l] - learn * D[l]&#xA;    print(J(y, W, X, Lambda))&#xA;    return W&#xA;&#xA;mnist = input_data.read_data_sets(&quot;MNIST_data/&quot;, one_hot=True)&#xA;&#xA;# Training data, read from MNIST&#xA;inputpix = []&#xA;output = []&#xA;&#xA;for i in range(Ntrain):&#xA;    inputpix.append(2 * np.array(mnist.train.images[i]) - 1)&#xA;    output.append(np.array(mnist.train.labels[i]))&#xA;&#xA;np.savetxt('input.txt', inputpix, delimiter=' ')&#xA;np.savetxt('output.txt', output, delimiter=' ')&#xA;&#xA;# Train the weights&#xA;finalweights = NeuralNetwork(output, inputpix, 2, 5, 1)&#xA;&#xA;# Test data&#xA;inputtestpix = []&#xA;outputtest = []&#xA;&#xA;for i in range(Ntest):&#xA;    inputtestpix.append(2 * np.array(mnist.test.images[i]) - 1)&#xA;    outputtest.append(np.array(mnist.test.labels[i]))&#xA;&#xA;np.savetxt('inputtest.txt', inputtestpix, delimiter=' ')&#xA;np.savetxt('outputtest.txt', outputtest, delimiter=' ')&#xA;&#xA;# Determine the accuracy of the training data&#xA;count = 0&#xA;for i in range(Ntrain):&#xA;    H = h(finalweights,inputpix[i])&#xA;    print(H)&#xA;    for j in range(num_output_node):&#xA;        if H[j] == np.amax(H) and output[i][j] == 1:&#xA;            count = count + 1&#xA;print(count/Ntrain)&#xA;&#xA;# Determine the accuracy of the test data&#xA;count = 0&#xA;for i in range(Ntest):&#xA;    H = h(finalweights,inputtestpix[i])&#xA;    print(H)&#xA;    for j in range(num_output_node):&#xA;        if H[j] == np.amax(H) and outputtest[i][j] == 1:&#xA;            count = count + 1&#xA;print(count/Ntest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37406" LastEditorUserId="37407" LastEditDate="2017-08-01T21:11:32.650" LastActivityDate="2018-01-30T10:14:28.157" Title="Neural Network: Backpropagation Works on MNIST, but Training/Test Set Accuracy Very Low" Tags="|machine-learning|neural-network|classification|deep-learning|backpropagation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21892" PostTypeId="2" ParentId="21889" CreationDate="2017-08-02T03:16:05.090" Score="2" Body="&lt;p&gt;If you are sure that the code for forward and backward passes is correct, then it seems that the problem is with model's hyperparameters as well as in using only 30k images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If accuracy is ~11%, is means that the model always predicts one and the same value.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Learning rate is 2. This is very high. Usually it is less than 0.1, or even 10-100 smaller.&lt;/li&gt;&#xA;&lt;li&gt;Number of iterations is 5. While this could be ok, it seems low.&lt;/li&gt;&#xA;&lt;li&gt;Regularization strength is 1. Quite high, try 0.1-0.001&lt;/li&gt;&#xA;&lt;li&gt;Hidden layers. There are 2 hidden layers with 5 nodes. 5 nodes is quite low. While there are a lot of theories about the number of neurons in hidden layers, it shouldn't be lower than the number of outputs. Try at least 10, or maybe better 64 or 128.&lt;/li&gt;&#xA;&lt;li&gt;You use 30000 training samples. Is your PC not powerfull enough or is there any other reason? Anyway, the problem is that classes maybe disbalanced in these 30000, some classes may have too much samples and som - too few. If you take only 30000 samples, you need to make sure that the classes are almost equally distributed.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-08-02T03:16:05.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21895" PostTypeId="1" AcceptedAnswerId="21903" CreationDate="2017-08-02T07:53:38.927" Score="0" ViewCount="68" Body="&lt;p&gt;If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with Mysql -&gt; Hadoop -&gt; Redshift and then analytics and visualizations on top of redshift if my database only has about 10million transactions per month. Would it be better to just run something like &lt;a href=&quot;https://redash.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ReDash&lt;/a&gt; visualizations on top of my unaggregated data in MySql?&lt;/p&gt;&#xA;" OwnerUserId="37432" LastActivityDate="2017-08-02T13:23:57.540" Title="How much data warrants building a pipeline?" Tags="|bigdata|visualization|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21896" PostTypeId="1" AcceptedAnswerId="22083" CreationDate="2017-08-02T10:47:37.473" Score="10" ViewCount="3933" Body="&lt;p&gt;I know that &lt;code&gt;Polynomial Logistic Regression&lt;/code&gt; can easily learn a typical data like the following image: &lt;br /&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/AM2zD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AM2zD.png&quot; alt=&quot;first image&quot;&gt;&lt;/a&gt;&#xA;&lt;br /&gt;&#xA;I was wondering whether the following two data also can be learned using &lt;code&gt;Polynomial Logistic Regression&lt;/code&gt; or not. &lt;br /&gt;&#xA;&lt;hr /&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/M6FjC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/M6FjC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &#xA;&lt;hr /&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/pQwKQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pQwKQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I have to add more explanation. Assume the first shape. If we add extra polynomial features for this 2-D input (like x1^2 ...) we can make a decision boundary which can separate the data. Suppose I choose X1^2 + X2^2 = b. This can separate the data. If I add extra features I will get a wavy shape (maybe a wavy circle or wavy ellipsis) but it still can not separate the data of the second graph, can it? &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-09T08:20:50.183" LastActivityDate="2017-11-09T08:20:50.183" Title="Can The linearly non-separable data be learned using polynomial features with logistic regression?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="21897" PostTypeId="1" CreationDate="2017-08-02T11:34:37.350" Score="2" ViewCount="143" Body="&lt;p&gt;My data is a matrix with 725 features and 667 observations which have binary data(either 0 or 1). my target variable is a univariate which has two classes (either 0 or 1). I removed the features which had zero as column sums, so features reduced to 652. since I have a binary data I did not do the preprocessing (Mean = 0 and s.d=1). But, My data is highly sparse; there is 99% sparsity. If I want to use LDA model in a 10 c.v setup. Then I am required to remove the colinear features (this means that all the features in a training set whose column sum is zero should be removed) to predict small set of test data in every fold. If this is the case then in every fold I will have to remove a different set of features which are zero in that training set for the test set. Is it an acceptable practice to complete my y.predict after 10 c.v (each fold used different features to predict the y) and do a rowmean for all the folds in the end and then show the accuracy of a classifier? I cant have the correct feature selection from this strategy but I can get a classifier prediction performance.&lt;/p&gt;&#xA;" OwnerUserId="31599" LastActivityDate="2017-08-02T11:34:37.350" Title="Linear Discriminant Analysis (LDA) , Removing colinear features in cross validation set up is correct or not?" Tags="|machine-learning|r|lda-classifier|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21899" PostTypeId="2" ParentId="21887" CreationDate="2017-08-02T11:39:50.797" Score="2" Body="&lt;p&gt;When you do &lt;code&gt;model.predict(X)&lt;/code&gt; the first axis in &lt;code&gt;X&lt;/code&gt; is always an index in a batch. So if you want to predict on one sample, do something like &lt;code&gt;X = np.expand_dims(X, axis=0)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, this should work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;old_qval = model.predict( np.expand_dims(old_state_m, axis=0) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-02T12:17:05.573" LastActivityDate="2017-08-02T12:17:05.573" CommentCount="2" CommunityOwnedDate="2017-08-02T11:39:50.797" ContentLicense="CC BY-SA 3.0" />
  <row Id="21900" PostTypeId="1" CreationDate="2017-08-02T12:30:38.813" Score="0" ViewCount="473" Body="&lt;p&gt;How can I concatenate matrices with unequal rows by intercalating the rows? For example, I want to concatenate A matrix ( dim(A) = 15 20 ) and B matrix (dim(B) = 17 20) and I want the result to be a matrix of 32 rows by 20 columns. I want result matrix with one row from matrix A and the second row from matrix B. Simple rbind just stacks up one matrix on top of other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following function works in case of equal number of rows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# drop false ensures that coercion does not change the data structure.&#xA;interleave(A, B, drop=FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I want the rows to be interleaved and then the remaining rows from B matrix to be added at the end. If the number of rows from one matrix is added and no more are remaining then the remaining rows should get rowbind (normally stacked). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way that I can tweak this interleave function from gdata package in R? or is there any other way?&lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="29575" LastEditDate="2017-08-02T16:18:24.827" LastActivityDate="2017-12-31T12:04:34.907" Title="intercalate matrices by Rows in R" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21902" PostTypeId="1" CreationDate="2017-08-02T12:41:49.033" Score="3" ViewCount="15598" Body="&lt;p&gt;I have the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;userid itemid&#xA;  1       1&#xA;  1       1&#xA;  1       3&#xA;  1       4&#xA;  2       1&#xA;  2       2&#xA;  2       3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I drop duplicated items (not the index) to make the following &lt;code&gt;DataFrame&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;userid itemid&#xA;  1       1&#xA;  1       3&#xA;  1       4     &#xA;  2       1&#xA;  2       2&#xA;  2       3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone help me?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd    &#xA;df = pd.DataFrame({'userid':[1,1,1,1, 2,2,2],&#xA;                   'itemid':[1,1,3,4, 1,2,3] })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33137" LastEditorUserId="28175" LastEditDate="2017-08-03T17:20:52.483" LastActivityDate="2023-10-25T03:08:37.960" Title="How duplicated items can be deleted from dataframe in pandas" Tags="|pandas|dataframe|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21903" PostTypeId="2" ParentId="21895" CreationDate="2017-08-02T13:23:57.540" Score="0" Body="&lt;p&gt;You should only complicate it as much as necessary to meet your requirements. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What kind of aggregations are you doing? Most DBs can handle basic counts pretty easily. 10 million in a month sounds like something a large MySQL instance could handle.&lt;/li&gt;&#xA;&lt;li&gt;How quickly does everything have to update? If the users only expect the summarized data to be updated monthly, you have a lot of time. If you have to update it more often, it might make sense to have MySQL collecting it but Redshift presenting it. Redshift's analytical focus and columnar storage means you may not need pre-aggregated data until your volume gets much higher than what would require aggregation in MySQL. I was using a 4 node Redshift cluster with the smallest instance types and it could do a count across &gt;300 million rows in 17 seconds.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can you not build aggregate tables in MySQL? What is making you want to put Hadoop in between MySQL and Redshift?&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-02T13:23:57.540" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21904" PostTypeId="2" ParentId="21902" CreationDate="2017-08-02T13:32:50.607" Score="1" Body="&lt;pre&gt;&lt;code&gt;df.groupby(df.index).first() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;worked for me.&lt;/p&gt;&#xA;" OwnerUserId="33137" LastEditorUserId="29575" LastEditDate="2017-08-02T15:42:19.053" LastActivityDate="2017-08-02T15:42:19.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21905" PostTypeId="2" ParentId="10165" CreationDate="2017-08-02T14:12:42.947" Score="0" Body="&lt;p&gt;You could also try to replace bins with Weight of Evidence values, and use those as your training set. See package &lt;em&gt;woe&lt;/em&gt; if you are using R (&lt;em&gt;smbinning&lt;/em&gt; or &lt;em&gt;woeBinning&lt;/em&gt; are available as well, but don't catch non-linearities as well in my experience).&lt;/p&gt;&#xA;" OwnerUserId="37447" LastActivityDate="2017-08-02T14:12:42.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21909" PostTypeId="2" ParentId="21902" CreationDate="2017-08-02T17:14:58.613" Score="11" Body="&lt;p&gt;The best way would be to use &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html&quot; rel=&quot;noreferrer&quot;&gt;drop_duplicates()&lt;/a&gt;. If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.drop_duplicates()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.drop_duplicates(subset=['userid', 'itemid'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37453" LastActivityDate="2017-08-02T17:14:58.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21911" PostTypeId="1" CreationDate="2017-08-02T17:25:38.250" Score="6" ViewCount="709" Body="&lt;p&gt;I have started to look at Radial-Basis-Function Neural Networks (RBF-NN), and would like to solve the following task as an exercise: Given is the data in the image below. Each output $Y_j$ of the network is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(*)$  $Y_j = \sum_{i}^N w_{ij}* exp(-\frac{||x-\mu_i||^2}{2\sigma_i^2})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for the -ith neuron. &lt;strong&gt;The task&lt;/strong&gt; is to draw an RBF network that &lt;em&gt;perfectly&lt;/em&gt; classifies the data, with suitable means, covariances and weights. In a second step a point has to be taken and classified in the worked-out model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My ideas&lt;/strong&gt; We have two nodes in the input layer, one for each dimension. The hidden-layer has as many neurons as there are training samples. Each of these calculates the activation given by the exponential above. The output layer has three output nodes as there are three classes. I would determine $\Sigma$ and $\mu$ as follows: For each class and training-sample $x_i$ of this class, $\mu_j$ is just the centroid of the training samples, $\sigma = \frac{1}{m}\sum_i ||x_i-\mu||$, and $\Sigma = \sigma*I_d$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt; That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $\sigma_i$? Also, for &lt;em&gt;perfect&lt;/em&gt; classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/I90YM.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/I90YM.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37455" LastEditorUserId="37455" LastEditDate="2017-08-03T06:48:14.493" LastActivityDate="2018-05-21T10:09:30.297" Title="Step-by-step construction of an RBF neural network" Tags="|machine-learning|neural-network|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21912" PostTypeId="1" CreationDate="2017-08-02T18:16:59.670" Score="5" ViewCount="4182" Body="&lt;p&gt;I have scoured the internet and books, but everything seems to use &lt;code&gt;num_steps&lt;/code&gt; and &lt;code&gt;batch_size&lt;/code&gt; or similar terms interchangeably and I can not get a grasp on their specific use, or where &lt;em&gt;batches&lt;/em&gt; fit in when talking about RNNs ( I understand their use with Gradient Descent, and maybe I am misunderstanding RNNs and I need to be set straight at the fundamental level- if that is the case, please do so. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So where does batching apply in Recurrent Neural Networks? At the level of instances within time steps or at the level of timesteps themselves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By my current understanding the data fed into the model is structured like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = [&#xA;    &quot;time_step_1&quot;: [&#xA;        &quot;instance_1&quot;: [&#xA;            &quot;feature_1&quot;: &quot;some_value1&quot;,&#xA;            &quot;feature_2&quot;: &quot;some_value2&quot;,&#xA;            &quot;feature_3&quot;: &quot;some_value3&quot;,&#xA;            &quot;feature_4&quot;: &quot;some_value4&quot;&#xA;        ],&#xA;    &quot;time_step_2&quot;: [&#xA;        &quot;instance_2_but_first_in_time_step_2&quot;: [&#xA;            &quot;feature_1&quot;: &quot;some_value1&quot;,&#xA;            &quot;feature_2&quot;: &quot;some_value2&quot;,&#xA;            &quot;feature_3&quot;: &quot;some_value3&quot;,&#xA;            &quot;feature_4&quot;: &quot;some_value4&quot;&#xA;        ],&#xA;    &quot;time_step_3&quot;: [&#xA;        &quot;instance_3_but_first_in_time_step_3&quot;: [&#xA;            &quot;feature_1&quot;: &quot;some_value1&quot;,&#xA;            &quot;feature_2&quot;: &quot;some_value2&quot;,&#xA;            &quot;feature_3&quot;: &quot;some_value3&quot;,&#xA;            &quot;feature_4&quot;: &quot;some_value4&quot;&#xA;        ],&#xA;    &quot;time_step_4&quot;: [&#xA;        &quot;instance_4_but_first_in_time_step_4&quot;: [&#xA;            &quot;feature_1&quot;: &quot;some_value1&quot;,&#xA;            &quot;feature_2&quot;: &quot;some_value2&quot;,&#xA;            &quot;feature_3&quot;: &quot;some_value3&quot;,&#xA;            &quot;feature_4&quot;: &quot;some_value4&quot;&#xA;        ],&#xA;    &quot;time_step_5&quot;: [&#xA;        &quot;instance_5_but_first_in_time_step_5&quot;: [&#xA;            &quot;feature_1&quot;: &quot;some_value1&quot;,&#xA;            &quot;feature_2&quot;: &quot;some_value2&quot;,&#xA;            &quot;feature_3&quot;: &quot;some_value3&quot;,&#xA;            &quot;feature_4&quot;: &quot;some_value4&quot;&#xA;        ],&#xA;&#xA;    ]            &#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Except&lt;/strong&gt;: That many people (and examples) have more &lt;em&gt;instances&lt;/em&gt; per time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A typical training loop looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for epoch in range(do_num_epochs):&#xA;    for time_step_count in range(len(data)):&#xA;        run(model, data[time_step_count])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And that is if you had (for example) 10,000 instances at each time step, then batching would allow the weights to be updated more often than having to wait for the entire 10,000 instances to be processed. Then the loop would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = len(data) // do_num_batches&#xA;&#xA;for epoch in range(do_num_epochs):&#xA;    for time_step_count in range(len(data)):&#xA;        for batch_count in range(do_num_batches):&#xA;            batch_data = data[time_step_count][batch_count * batch_size : (batch_count + 1) * batch_size]&#xA;            run(model, batch_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But there is no point in processing the data in batches if there is only a single instance of data per time step is there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made a chart to try to explain ( to myself as well ) what the flow and data structure would look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZUy6N.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZUy6N.png&quot; alt=&quot;Data Structure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xnJ38.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xnJ38.png&quot; alt=&quot;Model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am just trying to understand the role of the batch in all of this, how it fits in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;strong&gt;Edit/Addition&lt;/strong&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some more research on the subject, I am fairly certain that batches are a way to feed in sequences in chunks, and control the frequency of weight updates. By utilizing batches, the weights are updated at batch level, thus letting you group time-series together in groups. I illustrated what I &lt;em&gt;believe&lt;/em&gt; to be right below, in the below example there are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;4 instances,&lt;/li&gt;&#xA;&lt;li&gt;each with 5 features&lt;/li&gt;&#xA;&lt;li&gt;across 30 time steps&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oB17f.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oB17f.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first example (mini-batch), there are 3 batches, of &lt;code&gt;batch_size&lt;/code&gt;= 10 in that example, the weights would be updated 3 times, once after the conclusion of each batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second example, is &lt;em&gt;online&lt;/em&gt; learning with an effective &lt;code&gt;batch_size&lt;/code&gt;=1 and in that example, the weights would be updated 30 times, once after each &lt;code&gt;time_series&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the final example (Batch learning), the entire data set is processed at once, the weights are updated only once (1) at the conclusion of the 'batch'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each of these example, the entire data set would be run through many multiple of times, each one of these cycles is an &lt;strong&gt;epoch&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="37460" LastEditorUserId="37460" LastEditDate="2017-08-04T15:29:29.383" LastActivityDate="2023-11-22T20:19:53.980" Title="Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?" Tags="|neural-network|time-series|rnn|prediction|training|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21914" PostTypeId="2" ParentId="21800" CreationDate="2017-08-02T19:11:13.190" Score="2" Body="&lt;p&gt;There is a problem in your definition of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Q(s,a)&lt;/code&gt; is the expected utility of taking action &lt;code&gt;a&lt;/code&gt; in state &lt;code&gt;s&lt;/code&gt; and following the optimal policy afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The expected utility of a certain state (based on your definition) is different after taking 1, 2 or 9 steps. That means that the reward of being in state &lt;code&gt;s_0&lt;/code&gt; and taking action &lt;code&gt;a_0&lt;/code&gt; is different in step &lt;code&gt;0&lt;/code&gt; from what you get in step &lt;code&gt;9&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To adequate model the problem, you should reframe it and consider the state to be both the 'position'+'step'. You will now have 90 states (10pos*9steps).&lt;/p&gt;&#xA;" OwnerUserId="5041" LastEditorUserId="5041" LastEditDate="2017-08-03T09:43:55.407" LastActivityDate="2017-08-03T09:43:55.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21916" PostTypeId="2" ParentId="21902" CreationDate="2017-08-02T19:23:17.593" Score="1" Body="&lt;p&gt;Use &lt;code&gt;drop_duplicates()&lt;/code&gt; of &lt;code&gt;pandas&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd    &#xA;df = pd.DataFrame({'userid':[1,1,1,1, 2,2,2],&#xA;                   'itemid':[1,1,3,4, 1,2,3] })&#xA;print(df)&#xA;print()&#xA;print(df.drop_duplicates())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Consider that drop won't change the df itself and just pass a new data frame which has dropped the specified row(s). If you want to change the &lt;code&gt;df&lt;/code&gt; itself set &lt;code&gt;inside&lt;/code&gt; parameter to &lt;code&gt;True&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-08-08T19:44:07.383" LastActivityDate="2017-08-08T19:44:07.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21917" PostTypeId="2" ParentId="20451" CreationDate="2017-08-02T19:38:16.170" Score="3" Body="&lt;p&gt;Only full sentences are used for testing and validation, though sentences and phrases are used for training. See &lt;a href=&quot;http://www.aclweb.org/anthology/D14-1181&quot; rel=&quot;nofollow noreferrer&quot;&gt;Yoon Kim 2014, Convolutional Neural Networks for Sentence Classification&lt;/a&gt; for an overview of various text classification experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is code that creates training, dev, and test .CSV files from the various text files in the dataset download.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;&quot;&quot;&#xA;Put all the Stanford Sentiment Treebank phrase data into test, training, and dev CSVs.&#xA;&#xA;Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., &amp;amp; Potts, C. (2013). Recursive Deep Models&#xA;for Semantic Compositionality Over a Sentiment Treebank. Presented at the Conference on Empirical Methods in Natural&#xA;Language Processing EMNLP.&#xA;&#xA;https://nlp.stanford.edu/sentiment/&#xA;&quot;&quot;&quot;&#xA;&#xA;import os&#xA;import sys&#xA;&#xA;import pandas&#xA;&#xA;&#xA;def get_phrase_sentiments(base_directory):&#xA;    def group_labels(label):&#xA;        if label in [&quot;very negative&quot;, &quot;negative&quot;]:&#xA;            return &quot;negative&quot;&#xA;        elif label in [&quot;positive&quot;, &quot;very positive&quot;]:&#xA;            return &quot;positive&quot;&#xA;        else:&#xA;            return &quot;neutral&quot;&#xA;&#xA;    dictionary = pandas.read_csv(os.path.join(base_directory, &quot;dictionary.txt&quot;), sep=&quot;|&quot;)&#xA;    dictionary.columns = [&quot;phrase&quot;, &quot;id&quot;]&#xA;    dictionary = dictionary.set_index(&quot;id&quot;)&#xA;&#xA;    sentiment_labels = pandas.read_csv(os.path.join(base_directory, &quot;sentiment_labels.txt&quot;), sep=&quot;|&quot;)&#xA;    sentiment_labels.columns = [&quot;id&quot;, &quot;sentiment&quot;]&#xA;    sentiment_labels = sentiment_labels.set_index(&quot;id&quot;)&#xA;&#xA;    phrase_sentiments = dictionary.join(sentiment_labels)&#xA;&#xA;    phrase_sentiments[&quot;fine&quot;] = pandas.cut(phrase_sentiments.sentiment, [0, 0.2, 0.4, 0.6, 0.8, 1.0],&#xA;                                           include_lowest=True,&#xA;                                           labels=[&quot;very negative&quot;, &quot;negative&quot;, &quot;neutral&quot;, &quot;positive&quot;, &quot;very positive&quot;])&#xA;    phrase_sentiments[&quot;coarse&quot;] = phrase_sentiments.fine.apply(group_labels)&#xA;    return phrase_sentiments&#xA;&#xA;&#xA;def get_sentence_partitions(base_directory):&#xA;    sentences = pandas.read_csv(os.path.join(base_directory, &quot;datasetSentences.txt&quot;), index_col=&quot;sentence_index&quot;,&#xA;                                sep=&quot;\t&quot;)&#xA;    splits = pandas.read_csv(os.path.join(base_directory, &quot;datasetSplit.txt&quot;), index_col=&quot;sentence_index&quot;)&#xA;    return sentences.join(splits).set_index(&quot;sentence&quot;)&#xA;&#xA;&#xA;def partition(base_directory):&#xA;    phrase_sentiments = get_phrase_sentiments(base_directory)&#xA;    sentence_partitions = get_sentence_partitions(base_directory)&#xA;    # noinspection PyUnresolvedReferences&#xA;    data = phrase_sentiments.join(sentence_partitions, on=&quot;phrase&quot;)&#xA;    data[&quot;splitset_label&quot;] = data[&quot;splitset_label&quot;].fillna(1).astype(int)&#xA;    data[&quot;phrase&quot;] = data[&quot;phrase&quot;].str.replace(r&quot;\s('s|'d|'re|'ll|'m|'ve|n't)\b&quot;, lambda m: m.group(1))&#xA;    return data.groupby(&quot;splitset_label&quot;)&#xA;&#xA;&#xA;base_directory, output_directory = sys.argv[1:3]&#xA;os.makedirs(output_directory, exist_ok=True)&#xA;for splitset, partition in partition(base_directory):&#xA;    split_name = {1: &quot;train&quot;, 2: &quot;test&quot;, 3: &quot;dev&quot;}[splitset]&#xA;    filename = os.path.join(output_directory, &quot;stanford-sentiment-treebank.%s.csv&quot; % split_name)&#xA;    del partition[&quot;splitset_label&quot;]&#xA;    partition.to_csv(filename)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a gist for the code &lt;a href=&quot;https://gist.github.com/wpm/52758adbf506fd84cff3cdc7fc109aad&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="35650" LastEditorUserId="35650" LastEditDate="2017-12-03T19:47:26.370" LastActivityDate="2017-12-03T19:47:26.370" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21918" PostTypeId="1" AcceptedAnswerId="21921" CreationDate="2017-08-02T20:37:06.863" Score="7" ViewCount="19519" Body="&lt;p&gt;I've looked at hmmlearn but I'm not sure if it's the best one. &lt;/p&gt;&#xA;" OwnerUserId="33171" LastActivityDate="2019-07-23T22:38:51.983" Title="What's a good Python HMM library?" Tags="|python|markov|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ClosedDate="2017-08-03T11:11:22.300" ContentLicense="CC BY-SA 3.0" />
  <row Id="21919" PostTypeId="1" CreationDate="2017-08-02T20:48:12.253" Score="1" ViewCount="885" Body="&lt;p&gt;Suppose I have a movies dataframe in pandas. One of the features is Genre. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has a list of genre names. for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Movie_ID    Genre&#xA;&#xA;1        [Action, Thriller, Drama]&#xA;2        [Romance, Comedy]&#xA;3        [Action, Romance]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I use this column for a clustering problem, say K-means clustering?&lt;/p&gt;&#xA;" OwnerUserId="37471" LastEditorUserId="8878" LastEditDate="2017-08-03T14:28:03.407" LastActivityDate="2017-08-06T19:18:04.487" Title="Handling features with multiple values for clustering" Tags="|clustering|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21920" PostTypeId="2" ParentId="21863" CreationDate="2017-08-02T20:50:37.780" Score="0" Body="&lt;p&gt;I am assuming that you are referring to this &lt;a href=&quot;https://stackoverflow.com/questions/17904190/why-does-naivebayes-return-all-nas-for-multiclass-classification-in-r&quot;&gt;Stackoverflow post&lt;/a&gt; that mentions to add noise to the data since the error seems to be coming when there is one (or small) instance of a class in the dataset. Is that the case with the training data? If what you're trying to predict is a rare-event, then a suggestion might be to balance the training data by oversampling the rare class (hence adding noise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided the above is not working, another suggestion is to remove infrequent terms in your term-document-matrix using the function &lt;code&gt;removeSparseTerms&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going beyond, given the amount of training data you have, it would be good to evaluate if the term document matrix with the words it contains or frequency of specific words is sufficient to differentiate the classes. If not, you should consider adding new features to describe the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;count of positive/negative words or a sentiment index that ranges from -1 to 1, if relevant for your data&lt;/li&gt;&#xA;&lt;li&gt;types of words in dataset (index or count of adjectives or nouns or verbs), again depending on your problem &amp;amp; data&lt;/li&gt;&#xA;&lt;li&gt;rather than using term-document-matrix, try noun-phrases&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, I'm assuming that your test data contains records for both classes. If not, it is difficult to evaluate the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps. If you could formulate your question more clearly with the data problem and provide some examples of the data, that would help.&lt;/p&gt;&#xA;" OwnerUserId="6648" LastActivityDate="2017-08-02T20:50:37.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21921" PostTypeId="2" ParentId="21918" CreationDate="2017-08-03T02:14:59.747" Score="7" Body="&lt;p&gt;SKLearn has an amazing array of HMM implementations, and because the library is very heavily used, odds are you can find tutorials and other StackOverflow comments about it, so definitely a good start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.sourceforge.net/stable/modules/hmm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.sourceforge.net/stable/modules/hmm.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Right now its outdated in June 2019&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As of July 2019 you can use hmmlearn (&lt;code&gt;pip3 install hmmlearn&lt;/code&gt;) &lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="26562" LastEditDate="2019-07-23T22:38:51.983" LastActivityDate="2019-07-23T22:38:51.983" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="21924" PostTypeId="2" ParentId="21719" CreationDate="2017-08-03T06:57:53.653" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can do DBSCAN, OPTICS, HDBSCAN with cosine similarity - they do not expect or require Euclidean distances.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, &lt;em&gt;one&lt;/em&gt; version of cosine distance corresponds nicely to Euclidean distance of L2 normalized vectors. So there is some theoretical support to L2 normalize the vectors and try Euclidean then.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Given how word2vec is trained, I don't think Euclidean nor cosine distance is the right thing to do. Instead, use the raw dot product as a similarity measure (beware that it is not limited to [0;1] but allows negative similarity - values may be too spread out to use them intuitively).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;t-SNE, which was discussed in the comments and another answer, also relies on distances to the nearest neighbors. So it does suffer from the same problem - cosine, Euclidean, dot product? The fast ELKI implementation does support cosine distance as well as Euclidean, but you appear to be set on Python already. Also, if you have a large vocabulary, t-SNE may give you some scalability trouble. The standard algorithm will need O(n²) memory and time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-08-03T07:08:18.607" LastActivityDate="2017-08-03T07:08:18.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21926" PostTypeId="1" CreationDate="2017-08-03T07:44:58.330" Score="2" ViewCount="1552" Body="&lt;p&gt;I recently learned about face recognition with deep learning &lt;a href=&quot;https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. One of the approach involved is &lt;a href=&quot;https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients&quot; rel=&quot;nofollow noreferrer&quot;&gt;Histogram of oriented gradients&lt;/a&gt; which is used for face detection as follows (short summary) :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;convert image to gray scale&lt;/li&gt;&#xA;&lt;li&gt;look at every pixel in image and detect surrounding pixel&lt;/li&gt;&#xA;&lt;li&gt;draw and arrow in direction where surrounding pixels are getting darker&lt;/li&gt;&#xA;&lt;li&gt;repeat the whole process &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;this results in HOG version of image ,something like this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qpgms.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qpgms.png&quot; alt=&quot;hog_image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt; Can we use similar method in colored images like how does rgb varies at pixel level &lt;code&gt;or&lt;/code&gt; how does pixels vary in colored images &lt;strong&gt;to&lt;/strong&gt; improve the accuracy of face detection and similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Purpose&lt;/strong&gt; face comparison is a bit tedious in this approach in terms of accuracy. So, i want to find some sort of &lt;code&gt;hash function/value&lt;/code&gt;(just using as layman term) to derive unique value from every image. This might improve face comparison easier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally , sharing any already implemented approach in this direction will be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="31696" LastEditorUserId="25157" LastEditDate="2017-10-05T00:51:08.693" LastActivityDate="2017-10-05T00:51:08.693" Title="application of Histogram of oriented gradients in colored image" Tags="|machine-learning|deep-learning|image-classification|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21928" PostTypeId="2" ParentId="9865" CreationDate="2017-08-03T08:14:01.927" Score="1" Body="&lt;p&gt;Just another point of view, Dig the topic names a bit deeper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Text mining&lt;/code&gt; - mining of text (just as &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_mining&quot; rel=&quot;nofollow noreferrer&quot;&gt;data mining&lt;/a&gt;, and the data is text data). mining is about extracting useful information from the available data. information could be patterns in text or matching structure but the semantics in the text is not considered. The goal is not about making the system understand what does the text conveys, rather about providing information to the user based on a certain step by step process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Natural language processing&lt;/code&gt; - Natural language is what humans use for communication. processing such a data is NLP. The data could be speech or text. Thus, the main goal is towards understanding what is the semantic meaning conveyed in it. Now you know why we care about grammatical part of speeches and the lexical relations among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;speech recognition systems could be a part of NLP, but it has nothing to do with text mining. And, it seems like NLP is the bigger fish and it uses text-mining, but its actually the other way around. text-mining uses NLP, because it makes sense to mine the data when you understand the data semantically.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="16024" LastEditDate="2017-08-03T10:58:49.500" LastActivityDate="2017-08-03T10:58:49.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21929" PostTypeId="1" AcceptedAnswerId="21940" CreationDate="2017-08-03T08:26:08.010" Score="4" ViewCount="1125" Body="&lt;p&gt;I read about the &lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rosenblatt Perceptron Learning Algorithm&lt;/a&gt;. Often there is an &lt;a href=&quot;http://sebastianraschka.com/Articles/2015_singlelayer_neurons.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;explicit note&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is important to note that all weights in the weight vector are being updated simultaneously&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But why are all weights updated simultaneously? I tried another approach where I iterated over all weights and updated them in different iterations. It also worked on some simple test cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone explain to me, why they are updated simultaneously and why should this approach be better?&lt;/p&gt;&#xA;" OwnerUserId="37483" LastEditorUserId="29575" LastEditDate="2017-08-03T15:41:55.183" LastActivityDate="2017-08-04T16:11:06.990" Title="Perceptron weight vector update" Tags="|perceptron|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21931" PostTypeId="2" ParentId="21900" CreationDate="2017-08-03T09:24:15.623" Score="1" Body="&lt;p&gt;The following approach works perfectly, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l &amp;lt;- list(a=A,b=B)&#xA;dataset &amp;lt;- do.call(rbind, l)[order(sequence(sapply(l, nrow))), ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31599" LastActivityDate="2017-08-03T09:24:15.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21933" PostTypeId="2" ParentId="20529" CreationDate="2017-08-03T10:21:35.643" Score="2" Body="&lt;p&gt;The DDPG described in that paper is very good to start working in control problems. There are many implementations of this algorithm online. However, bare in mind though that the algorithm is very unstable and needs many hours of monitoring and tuning the parameters of the networks. I would suggest you to create a very simple scenario (simulator) and put the DDPG to interact with it. It is very important to understand what your networks are learning and which state features are important or no. Furthermore the reward function will play a very important role in the performance of your algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another important point is the way you formalize your problem will determine if you are trying to solve a one-step decision making problem (e.g. given state features ---&gt; credit limit value, reward) or a sequential decision making problem (e.g. given state features ---&gt; credit limit value, reward, new state features and so on). This will specify if you need to modify the Q learning part of the Critic and remove the $maxQ(s',a')$ part (one step decision case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding pretraining, as Neil suggested you can use regression to pretrain Actor and Critic networks so the weights are initialized to better than random values. A very interesting approach from Deepmind is the one that is described in this paper &lt;a href=&quot;https://www.google.gr/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjU87TO6LrVAhXKKsAKHbQgDFgQFggnMAA&amp;amp;url=https%3A%2F%2Farxiv.org%2Fabs%2F1704.03732&amp;amp;usg=AFQjCNGnywW3NNumXbfe6ViyrA6cwQVPYw&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning from Demonstrations for real-world RL&lt;/a&gt;, and most likely helped them optimize the energy consumption at their server centers.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-08-03T10:21:35.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21934" PostTypeId="1" CreationDate="2017-08-03T10:49:30.910" Score="3" ViewCount="3990" Body="&lt;p&gt;Currently, I am doing a project with the aim of classifying potholes through machine learning. The data collected is from an accelerometer in which the z-axis measures the &quot;vertical&quot; acceleration of the car, when a pothole is struck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried to deconstruct the signals and create features using two methods:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Partial_autocorrelation_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;PACF&lt;/a&gt; along with a moving average to combat the noise&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculated a periodogram for spectral analysis&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there any other ways to differentiate a pothole from &quot;rough&quot; road surfaces, as I am unsure which approach to take?&lt;/p&gt;&#xA;" OwnerUserId="37484" LastEditorUserId="29575" LastEditDate="2017-08-03T17:20:49.117" LastActivityDate="2019-10-04T05:59:27.793" Title="Feature extraction of accelerometer data for machine learning" Tags="|machine-learning|classification|feature-extraction|feature-engineering|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21935" PostTypeId="2" ParentId="21934" CreationDate="2017-08-03T11:34:26.753" Score="1" Body="&lt;p&gt;Actually you could treat your acceloremeter signal like normal audio signals. There are endless possibilities for processing audio data (e.g. chroma features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way would be to directly process your raw signal with the help of a neural network (1D-convolution or LSTM).&lt;/p&gt;&#xA;" OwnerUserId="37330" LastEditorUserId="37330" LastEditDate="2017-08-04T08:56:20.060" LastActivityDate="2017-08-04T08:56:20.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21936" PostTypeId="2" ParentId="20143" CreationDate="2017-08-03T14:18:27.827" Score="1" Body="&lt;p&gt;The following is how I understand the distinction; it's based on my own experience engaging with various communities over the last few years, as I've been teaching myself statistics/data science/ML.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The term &quot;machine learning&quot; comes out of a fairly coherent academic/research community, centered in computer science but with moderately strong connections to a few other fields, such as statistics, and weaker connections to fields like genomics.  For one example, take a look at the research areas of the faculty in Carnegie Mellon's &lt;a href=&quot;https://www.ml.cmu.edu/people/core-faculty.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning Department&lt;/a&gt;.  For another example, all three of the authors of &lt;a href=&quot;http://rads.stackoverflow.com/amzn/click/0387848576&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Elements of Statistical Learning&lt;/em&gt;&lt;/a&gt; — which seems to be one of the standard textbooks in machine learning — are statisticians. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, the term &quot;data analysis&quot; is used in radically different ways in different sectors.  Data analysts in established companies (especially 10+ years ago) might use Excel and Tableau to plot earning trends over time and do some simple financial modeling.  The data analysts in a biology lab might be a couple postdocs and grad students who have taught themselves enough R or Python to run statistical tests and generate plots.  Web- or app-based companies might have data analysts building predictive machine learning models of audience/consumer behavior, in order to find ways to marginally increase sales or ad revenue.  Data journalists at news media outlets might focus more on building visualizations and interactives of data aggregates and summaries using D3, with little or no deep quantitative analysis (i.e., using neither classical statistics nor predictive models).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because &quot;data analysis&quot; has different aims and goals in different sectors — and is done by people with very different kinds of formal training — different kinds of &quot;data analysts&quot; will use different software tools and different methods.  Deep learning and other predictive models will be appropriate for some kinds of data analysis (web- and app-based companies; certain data-cleaning purposes in scientific research) but irrelevant for others (data journalism).  &lt;/p&gt;&#xA;" OwnerUserId="26961" LastActivityDate="2017-08-03T14:18:27.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21937" PostTypeId="2" ParentId="21919" CreationDate="2017-08-03T14:24:52.657" Score="2" Body="&lt;p&gt;You can easily use &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;get_dummy function&lt;/a&gt; in Pandas to convert them to numerical vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that categorical variables do not have a numerical intuition e.g. when it comes to the definition of &lt;strong&gt;Distance&lt;/strong&gt;. But just imagine you have one feature &lt;em&gt;Genre&lt;/em&gt; with 3 values &lt;em&gt;Comedy&lt;/em&gt;, &lt;em&gt;Romance&lt;/em&gt; and &lt;em&gt;Crime&lt;/em&gt;. Then you can model them in a 3-dimensional space by saying &lt;em&gt;Comedy = (1,0,0)&lt;/em&gt;, &lt;em&gt;Crime = (0,1,0)&lt;/em&gt; and &lt;em&gt;Romance = (0,0,1)&lt;/em&gt;. It replaces 1 feature with three but intuitively works well.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Update&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I just understood your question after editing it! It was a bit fuzzy previously. But I keep my initial answer and add an update.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case use the values of the feature Genre (unique values of union of all genre sets in that column) as new features and determine their presence with 1 and 0 otherwise. Should work.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Movie_ID    Action  Thriller  Drama  Romance  Comedy&#xA;&#xA;1             1       1         1       0       0&#xA;2             0       0         0       1       1&#xA;3             1       0         0       1       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2017-08-03T14:34:55.587" LastActivityDate="2017-08-03T14:34:55.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21938" PostTypeId="2" ParentId="21934" CreationDate="2017-08-03T16:20:48.913" Score="0" Body="&lt;p&gt;Instead of spectral features and moving average, I would recommend wavelet features. You could either do a continuous wavelet transform (CWT) or a Short Wavelet Transform (SWT) and identify the peaks / drop where the potholes show up. The advantage with wavelet is that it is well resistant to noise and you can also preserve the time axis information to pin point the location of peaks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trick is just to choose an appropriate wavelet. Those which have spike shapes like Daubechies and Symlets would be ideal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With proper feature extraction, you can even do the detection without machine learning.&lt;/p&gt;&#xA;" OwnerUserId="18465" LastActivityDate="2017-08-03T16:20:48.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21939" PostTypeId="1" CreationDate="2017-08-03T17:47:53.160" Score="1" ViewCount="48" Body="&lt;p&gt;I'm plotting fluctuations in a trinomial distribution across two dimensions.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colorspace_plot &amp;lt;- function(zz) {&#xA;    zz$z2 &amp;lt;- zz$x * zz$y&#xA;	zz$z3 &amp;lt;- zz$x * (1-zz$y)&#xA;    zz &amp;lt;- cbind(zz, t(apply(zz[,c(&quot;z1&quot;,&quot;z2&quot;,&quot;z3&quot;)], 1, function(x) x/sum(x)) ))&#xA;    setnames(zz, c(&quot;x&quot;, &quot;y&quot;, &quot;z1&quot;, &quot;z2&quot;, &quot;z3&quot;, &quot;p1&quot;, &quot;p2&quot;, &quot;p3&quot;))&#xA;    zz$pcolor &amp;lt;- with(zz, rgb(p1, p2, p3, 1))&#xA;	pcolors &amp;lt;- zz$pcolor&#xA;    names(pcolors) &amp;lt;- pcolors&#xA;    return( ggplot(zz, aes(x,y)) + geom_bin2d(aes(fill=pcolor), binwidth=1/11) +  scale_fill_manual(values = pcolors ) + guides(fill=&quot;none&quot;) )&#xA;}&#xA;&#xA;clear &amp;lt;- data.frame(x=rep(seq(from=0, to=1,length.out=11),times=11), y=rep(seq(from=0, to=1,length.out=11),each=11), z1=rep(1/3,11*11))&#xA;colorspace_plot(clear)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e0cs4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e0cs4.png&quot; alt=&quot;good colorspace plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what its called, but this kind of plot has been used for this kind of thing before (Figure 4 &lt;a href=&quot;https://science.sciencemag.org/content/347/6218/145/tab-figures-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my actual data doesn't have the same clean separation of values as the ideal example. It's dominated by one of the three categories, so it comes out varying between clean and muddy variations of just one color. It doesn't communicate.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;muddy &amp;lt;- data.frame(x=rep(seq(from=0.6, to=1,length.out=11),times=11), y=rep(seq(from=0.6, to=1,length.out=11),each=11), z1=rep(1/3,11*11))&#xA;colorspace_plot(muddy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AAF3B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AAF3B.png&quot; alt=&quot;bad colorspace plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this plot still works, it doesn't &quot;work&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question&lt;/strong&gt;: is there another way to represent trinomial data within-bin? For example: can I insert into each bin little pie charts or histograms, or something?  A solution that stays within &lt;code&gt;R&lt;/code&gt; and &lt;code&gt;ggplot&lt;/code&gt; (and even with &lt;code&gt;geom_bin2d()&lt;/code&gt;—I think I could figure this out with &lt;code&gt;facet_grid()&lt;/code&gt;) would be ideal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One promising idea that came up was to represent the distribution in each bin by filling each bin with a 10x10 grid of appropriately colored dots: a pointillist approach to my muddy plots.  But I have no idea how to pull that off, or if it would even look good. Things I'll ask you to disregard: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rescaling the green or the mapping so that the other percentages come off larger than they really are.&lt;/li&gt;&#xA;&lt;li&gt;breaking this one panel into three panels, one for each category. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37499" LastActivityDate="2017-08-03T17:47:53.160" Title="Infoviz for multinomials: miniature plots within plots in R's ggplot" Tags="|r|visualization|ggplot2|plotting|infographics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21940" PostTypeId="2" ParentId="21929" CreationDate="2017-08-03T19:35:09.627" Score="3" Body="&lt;p&gt;The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the right direction, which could mess with the convergence of the procedure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that in the weight space every input vector is a hyperplane. You need to find a weight vector that is on the correct side of &lt;strong&gt;all&lt;/strong&gt; the hyperplanes of your data inputs. The correct weight therefore is in a convex cone. If you observe a misclassification, that means your weight vector is on the wrong side of the hyperplane and therefore outside the convex cone of possible solutions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gXRRB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gXRRB.png&quot; alt=&quot;Example of two example input (blue lines) with perpendicular vectors indicating on which side the output is correct&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, by adding/subtracting the input vector to the weight vector you make sure that this data input vector is now correctly classified. You also make sure, that you decrease the distance of your weight vector by a sufficient margin (at least the length of the input vector) towards the cone of all possible solutions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AGFwF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AGFwF.png&quot; alt=&quot;Example of a weight update&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only add parts of the vector (i.e. not update all weights) at each iteration, you cannot be sure that you make sufficient progress or even move in the right direction of the solution space.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-08-04T16:11:06.990" LastActivityDate="2017-08-04T16:11:06.990" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21942" PostTypeId="1" AcceptedAnswerId="21943" CreationDate="2017-08-03T21:37:31.397" Score="3" ViewCount="2941" Body="&lt;p&gt;I am implementing Linear Discriminant Analysis in R, which parameters can be tunned in cross validation set up? In regularized mode called penalizedLDA there are parameters which are optimised but I want to know which parameters are turned in case of simple LDA method?&lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="31599" LastEditDate="2017-08-04T15:00:53.773" LastActivityDate="2017-08-04T15:00:53.773" Title="Linear Discriminant Analysis, which parameters can be tunned in cross validation set up?" Tags="|machine-learning|r|lda-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21943" PostTypeId="2" ParentId="21942" CreationDate="2017-08-03T21:49:46.093" Score="4" Body="&lt;p&gt;LDA has a closed-form solution and therefore has no hyperparameters. The solution can be obtained using the empirical sample class covariance matrix. Shrinkage is used when there are not enough samples. In that case the empirical covariance matrix is often not a very good estimator.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-08-03T21:49:46.093" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21944" PostTypeId="2" ParentId="21828" CreationDate="2017-08-03T22:00:45.403" Score="2" Body="&lt;p&gt;It's hard to give a very good answer to such a broad question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model interpretability is a big topic, and usually depends on the model. Simpler models, such as &lt;strong&gt;logistic regressions,&lt;/strong&gt; are easier to interpret than &lt;strong&gt;neural networks.&lt;/strong&gt; It's easy to say things like &quot;if I can increase feature $X_i$ by 1, then the odds of $y$ happening will increase by $\beta_i$&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise, individual &lt;strong&gt;decision trees&lt;/strong&gt; are easier to interpret than &lt;strong&gt;random forests.&lt;/strong&gt; Yet, some people try to interpret random forests by computing &quot;feature importance&quot;, which can be computed several ways, one of which is the number of splits that include the feature relative to the number of samples it splits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want a way to treat your model as a black box and be able to interpret &lt;em&gt;any&lt;/em&gt; model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of two ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Manipulate your inputs and see what happens to the ouput, using your sample&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One typical way is to change the input and see the impact on the model performance. In images, you can black out parts of the image, and see which parts contributes most to the accuracy. This is widely used for convolutional neural networks, which are hard to interpret otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For numerical variables, you can zero out or add some noise to each feature and see what the impact of that individual feature has on the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen these kind of things widely used.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Train with and without the feature&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Similar to the previous one, except you train with and without the feature, and see the impact it has on the model accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has the added benefit that you don't have to think about hyperparameters such as how much noise you'll add to each feature like the previous approach. Furthermore, you can better understand the impact of several features in the output by trying with and without those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't seen this approach being used, but apparently it's also &lt;a href=&quot;https://github.com/TeamHG-Memex/eli5&quot; rel=&quot;nofollow noreferrer&quot;&gt;being used,&lt;/a&gt; as another person replied.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Anyway, I would avoid such hacks. Most models can be made somehow interpretable. I have seen people even making recurrent neural networks interpretable. And, if interpretability is a concern, just use a simpler model that is easier to interpret and explain.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2017-08-03T22:00:45.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21945" PostTypeId="1" CreationDate="2017-08-03T23:09:46.617" Score="2" ViewCount="3366" Body="&lt;p&gt;hope someone kindly put time here,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my approach is like this:&#xA;TFIDF -&gt; LDA -&gt; SVM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that I have to send the feature vector to SVM,&#xA;but my question is that how can I make this feature vector to send to svm?&#xA;Is featureValue here is the probability assigned to each word?&#xA;My question is kind of the step 3 in the below link that I dont know how to do it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/138352/is-this-a-correct-way-to-do-document-classification-using-topic-modeling&quot;&gt;svm on lda&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but in this link there is no implementation, that is just explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for your time:)&lt;/p&gt;&#xA;" OwnerUserId="37507" LastActivityDate="2019-02-15T13:48:50.027" Title="Apply SVM on LDA in python" Tags="|machine-learning|text-mining|svm|topic-model|lda|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21950" PostTypeId="1" CreationDate="2017-08-04T03:56:48.463" Score="17" ViewCount="24413" Body="&lt;p&gt;Can someone explain why we can not feed &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;noreferrer&quot;&gt;LDA&lt;/a&gt; topic model with &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;TFIDF&lt;/a&gt;?  What is wrong with this approach conceptually?&lt;/p&gt;&#xA;" OwnerUserId="37512" LastEditorUserId="53476" LastEditDate="2023-08-10T07:26:13.540" LastActivityDate="2023-08-10T07:44:39.120" Title="Why should we not feed LDA with TF-IDF input?" Tags="|machine-learning|python|topic-model|lda|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="21951" PostTypeId="1" CreationDate="2017-08-04T04:46:32.483" Score="1" ViewCount="48" Body="&lt;p&gt;As the question states I would like to be able to get a formula based on the relationship between potential values (from a set of values) and a result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result: 13&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set of values: 3, 5, 4, 2, 1, 7, 9, 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find what equation of the values would give the result (ie. 3 * 4 + 2 - 1), keeping in mind some of the values may be irrelevant in the final equation. The equation would not have to 100% accurate, finding something about 95-98% accurate would be fit for my purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume this is a task that regression would be used for however I'm not sure where to begin. Ideally I would like to achieve this in Python or R and I'm guessing there would be some sort of packages that could be leveraged. I have many sets of values and results to run through to hopefully improve the accuracy of the model/equation, I'm not sure how many would be required to get a statistically significant accuracy.&lt;/p&gt;&#xA;" OwnerUserId="37516" LastActivityDate="2017-08-04T04:46:32.483" Title="How to get the relationship/formula between a result and a set of values?" Tags="|python|r|regression|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21952" PostTypeId="1" AcceptedAnswerId="22739" CreationDate="2017-08-04T05:40:43.620" Score="3" ViewCount="1071" Body="&lt;p&gt;totally new to larger than ram datasets but I have csv files that are about 100 gb each with around 300 million rows each  (just two of them). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am looking for is something like: don't go analyzing more than 1 terabyte of data on a laptop. Or don't analyze data that is more than 10x your ram or the wait times will prove frustrating. This is what I mean by rules of thumb. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a mac laptop with an i5, 8gb ram, ssd is it reasonable to process this data (100gb per file and I have 2 files) with either dask or blaze in python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried and could read in the csv but when doing simple trials like dropping one column or finding the length of the data frame it takes at least an hour For the drop (I gave up waiting) and half an hour for the length() (that did finish). Way too slow to be productive. I changed the data format to parquet and amazingly the data is down to just 10 gb. This is the only way they the length() command ever even finished. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: are these response times normal experience in general given my hardware and csv size? Are my expectations for what dask can do way too high? Any rough guidelines on how to troubleshoot or is this clearly underpowered hardware？&lt;/p&gt;&#xA;" OwnerUserId="32637" LastEditorUserId="32637" LastEditDate="2017-08-04T15:21:18.043" LastActivityDate="2017-08-31T10:01:17.463" Title="Need rules of thumb for out of core larger than ram dataset on a laptop" Tags="|python|pandas|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="21953" PostTypeId="1" CreationDate="2017-08-04T07:36:29.367" Score="1" ViewCount="195" Body="&lt;p&gt;We have a large user-base, within which we want to find lookalikes, around 25-30 Million, for the users, we have data such as show liked by the user, genre liked etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we need to do is extrapolate our custom list (ex: find lookalikes for a list of 100,000 and extend it to 300,000) from our final result of lookalike_job_data, when we need to use the IDs somewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is written in Java and we use Threadpool for parallelizing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our current approach is, we have all our data in Apache Solr, as our data is updated incrementally, we query for 'term-frequency' of a shows_watched by a user, sort them as per 'term-frequency', we don't take into consideration the shows which have 'term-frequency' as 1 &#xA;-&gt; regard them as new users&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or users which only have 2-3 shows in their list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we query for users who have watched the same shows in pairs.&#xA;For ex. if we get 5 shows which have 'term-frequency' &#xA;such as sh1-&gt; 7, sh2-&gt;4, sh3-&gt;5, sh4-&gt;6, sh5-&gt;14 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We make pairs of 3 among them which are in 'AND' with 'OR' over all of the pairs as if our query will be as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find me users, that have watched :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(sh1 AND sh2 AND sh3) OR (sh2 AND sh3 sh4) and so on &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our query also uses 'frange'( a feature in '&lt;strong&gt;SOLR&lt;/strong&gt;' to limit a value) which limits the results to a distance from the base user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is applied to 'dist' (feature in '&lt;strong&gt;SOLR&lt;/strong&gt;') function, which gets the distance of other's shows to the base user &#xA;(same shows for which we the term-frequency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this will be our cluster for that user, and we ignore the rest of the user's in this cluster but still, this approach is slow for our usage, We'll probably run this job for full data once/twice a week and incrementally assign the new user's to existing clusters b/w them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This process is still slow, can someone recommend a better approach, I am open to anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 2 months our data size is around 140GBs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our SOLR runs on an AWS &lt;em&gt;i3.2xlarge&lt;/em&gt;(8 core, 60GB ram, 2TB instance store SSD) machine,&#xA;and we fire queries from 4core, 16 GB ram machine&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We initially have our data in S3.&lt;/p&gt;&#xA;" OwnerUserId="20766" LastEditorUserId="21" LastEditDate="2017-08-05T09:20:15.863" LastActivityDate="2017-08-22T11:44:02.080" Title="Finding lookalike for large number of users" Tags="|clustering|apache-spark|similarity|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21954" PostTypeId="1" AcceptedAnswerId="21971" CreationDate="2017-08-04T10:50:38.383" Score="6" ViewCount="6015" Body="&lt;p&gt;I have seen other posts in this forum but didn't find any convincing answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is calculated based on a portion of trees in Forest rather by full Forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what are the advantages and disadvantages of using OOB instead of a CV?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is getting to train on more data by using OOB correct to say?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="23305" LastEditDate="2017-08-04T18:59:55.833" LastActivityDate="2017-08-04T18:59:55.833" Title="Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?" Tags="|random-forest|cross-validation|sampling|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21955" PostTypeId="1" AcceptedAnswerId="21956" CreationDate="2017-08-04T11:22:50.340" Score="9" ViewCount="16185" Body="&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;x = tf.placeholder(tf.float32, [None,4])    # input vector    &#xA;w1 = tf.Variable(tf.random_normal([4,2]))   # weights between first and second layers&#xA;b1 = tf.Variable(tf.zeros([2]))             # biases added to hidden layer&#xA;w2 = tf.Variable(tf.random_normal([2,1]))   # weights between second and third layer&#xA;b2 = tf.Variable(tf.zeros([1]))             # biases added to third (output) layer&#xA;&#xA;def feedForward(x,w,b):                     # function for forward propagation&#xA;    Input = tf.add(tf.matmul(x,w), b)&#xA;    Output = tf.sigmoid(Input)&#xA;    return Output&#xA;&#xA;Out1 = feedForward(x,w1,b1)                # output of first layer&#xA;Out2 = feedForward(Out1,w2,b2)             # output of second layer&#xA;MHat = 50*Out2                             # final prediction is in the range (0,50)&#xA;&#xA;M = tf.placeholder(tf.float32, [None,1])   # placeholder for actual (target value of marks)&#xA;J = tf.reduce_mean(tf.square(MHat - M))    # cost function -- mean square errors                          &#xA;train_step = tf.train.GradientDescentOptimizer(0.05).minimize(J)     # minimize J using Gradient Descent&#xA;sess = tf.InteractiveSession()             # create interactive session &#xA;&#xA;tf.global_variables_initializer().run()    # initialize all weight and bias variables with specified values&#xA;xs = [[1,3,9,7],    &#xA;      [7,9,8,2],                           # x training data&#xA;      [2,4,6,5]]&#xA;&#xA;Ms = [[47],&#xA;      [43],                                # M training data&#xA;      [39]]&#xA;&#xA;for _ in range(1000):                      # performing learning process on training data 1000 times&#xA;    sess.run(train_step, feed_dict = {x:xs, M:Ms})&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; print(sess.run(MHat, feed_dict = {x:[[1,15,9,7]]}))&#xA;[[50.]]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; print(sess.run(MHat, feed_dict = {x:[[3,8,1,2]]}))&#xA;[[50.]]&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; print(sess.run(MHat, feed_dict = {x:[[6,7,10,9]]}))&#xA;[[50.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this code, I am trying to predict the marks M obtained by a student in a test out of 50 given how many hours he/she slept, studied, used electronics and played the day before the test. These 4 features come under the input feature vector x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To solve this regression problem, I am using a deep neural network with an input layer with 4 perceptrons (the input features), a hidden layer with two perceptrons and an output layer with one perceptron. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used sigmoid as the activation function. But, I am getting the exact same prediction([[50.0]]) for M for all possible input vectors I feed in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please tell me what is wrong with the code above, and why I get the same result each time?&lt;/p&gt;&#xA;" OwnerUserId="37532" LastEditorUserId="29575" LastEditDate="2017-08-04T18:25:59.983" LastActivityDate="2017-11-01T19:19:52.647" Title="Tensorflow regression model giving same prediction every time" Tags="|neural-network|deep-learning|regression|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21956" PostTypeId="2" ParentId="21955" CreationDate="2017-08-04T11:48:22.323" Score="8" Body="&lt;p&gt;Your network design/logic is basically correct, but you are seeing some very common problems with neural network numerical stability. This results in your weights diverging and not training accurately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the fixes, any one of them might help a little, but the first two should be used for nearly all neural network projects.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;1. Inputs need to be scaled to work with neural networks.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is called input normalisation. Usually you would do this in data preprocessing, but for your simple network we can include the scaling at the input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_normalised = x * 0.2 - 0.5             # Arbitrary scaling I just made up&#xA;Out1 = feedForward(x_normalised,w1,b1)   # output of first layer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The most common scaling operation is to take all the training data and convert it so that it has mean $0$ and standard deviation $1$ (typically &lt;em&gt;per&lt;/em&gt; feature, as opposed to global scaling for all data as I have done here) - store the values used to achieve that and apply them to all following data for training, tests etc.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;2. Adjust learning rate until you get a working value.&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_step = tf.train.GradientDescentOptimizer(0.001).minimize(J) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A value that is too high will cause your training to fail. A value that is to low will take ages to learn anything.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;3. For small training sets, use more iterations than you might think&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is not a general thing, but specific to demos with tiny amounts of training data like your example, or the commonly-used &quot;learning XOR function&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for _ in range(10000):                      # performing learning process&#xA;&#xA;   sess.run(train_step, feed_dict = {x:xs, M:Ms})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With your very simple network actually this may cause over-fitting to the training data, so you will have to play with a value that gives you &quot;sensible&quot; results. However, in general how to spot and minimise over-fitting is a whole broad subject in itself, based on how you test and measure generalisation. This will need other questions if you are not sure when you learn it. It should be high on your list of things to learn though . . . it is a critical skill in producing useful neural networks that solve real problems.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-04T12:01:37.350" LastActivityDate="2017-08-04T12:01:37.350" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21958" PostTypeId="1" CreationDate="2017-08-04T13:33:15.220" Score="3" ViewCount="4966" Body="&lt;p&gt;I am trying to develop a model based on one-class classification approach. For example, the model should to identify if a given picture contains a cat or not. Keep in mind that my training dataset only contains pictures of cats and nothing else. Sort of like a anomaly detection problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen some suggestions on using autoencoders for unary classification but I couldn't find a concrete example on the net? Can someone point to an implementation of this approach using Keras. Is there any other way to about solving this problem?&lt;/p&gt;&#xA;" OwnerUserId="37537" LastActivityDate="2020-07-23T11:39:50.700" Title="One-class classification in Keras using Autoencoders?" Tags="|deep-learning|keras|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21959" PostTypeId="1" CreationDate="2017-08-04T14:25:26.683" Score="9" ViewCount="1644" Body="&lt;p&gt;I am trying to implement the &lt;a href=&quot;https://en.wikipedia.org/wiki/AdaBoost&quot; rel=&quot;nofollow noreferrer&quot;&gt;AdaBoost.M1&lt;/a&gt; algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in &lt;code&gt;R&lt;/code&gt;. There exists a variety of different packages for this purpose; &lt;code&gt;AdaBag&lt;/code&gt;, &lt;code&gt;Ada&lt;/code&gt; and &lt;code&gt;gbm&lt;/code&gt;. &lt;code&gt;gbm()&lt;/code&gt; (from the &lt;code&gt;gbm&lt;/code&gt;-package) appears to be my only available option, as &lt;code&gt;stack.overflow&lt;/code&gt; is a problem in the others, and though it works, it is very time-consuming.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there any way to overcome the &lt;code&gt;stack.overflow&lt;/code&gt; the problem in the other packages, or have the &lt;code&gt;gbm()&lt;/code&gt; run faster? I have tried converting the &lt;code&gt;data.frame&lt;/code&gt; into a matrix without success.&lt;/li&gt;&#xA;&lt;li&gt;When performing &lt;code&gt;AdaBoost&lt;/code&gt; in &lt;code&gt;gbm()&lt;/code&gt; (with distribution set to &amp;quot;AdaBoost&amp;quot;), An &lt;a href=&quot;http://www-bcf.usc.edu/%7Egareth/ISL/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Statistical Learning&lt;/a&gt; (Hastie et al.) mentions the following parameters needed for tuning:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;Needs:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The total number of trees to fit.&lt;/li&gt;&#xA;&lt;li&gt;The shrinkage parameter denoted lambda.&lt;/li&gt;&#xA;&lt;li&gt;The number of splits in each tree, controlling the complexity of the boosted ensemble.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;As the algorithm is very time consuming to perform in &lt;code&gt;R&lt;/code&gt;, I need to find literature on what tuning parameters are within a suitable range for this kind of large feature space data, before performing cross-validation over the range to estimate the test error rate.&lt;/p&gt;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="37539" LastEditorUserId="89090" LastEditDate="2020-12-01T09:18:00.157" LastActivityDate="2020-12-01T09:18:00.157" Title="AdaBoost implementation and tuning for high dimensional feature space in R" Tags="|machine-learning|r|boosting|gbm|adaboost|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="21961" PostTypeId="1" CreationDate="2017-08-04T15:35:15.427" Score="7" ViewCount="1224" Body="&lt;p&gt;My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of &gt;100k samples. Classifying foreground class as background is quite OK, while classifying background as foreground should be penalised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Scikit-learn's RandomForests, and I was experimenting with SVM and OneVsRest classifiers as well. I would like to specify the scoring metrics used for the method &lt;code&gt;fit()&lt;/code&gt; of the model, so it will correspond to my goal (I imagine something like fitness function with evolution algorithms). However, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.fit&quot; rel=&quot;noreferrer&quot;&gt;API&lt;/a&gt; does not allow something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I tried:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use &lt;code&gt;class_weight&lt;/code&gt; parameter of the model. If I set it so it represents the real world, then the classifier learns to classify everything as background having accuracy &gt;99 %. If I set &lt;code&gt;class_weight = 'balanced'&lt;/code&gt;, then it seems better, but it has high false positive rate.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html&quot; rel=&quot;noreferrer&quot;&gt;scoring method&lt;/a&gt; for &lt;code&gt;GridSearchCV&lt;/code&gt;, which outputs values specified by me (even F1-score makes more sense than simple accuracy), but it is used only for the parameter selection and the final model is learnt by &lt;code&gt;fit()&lt;/code&gt; method, which again ignores my scoring.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there a way to provide my own custom scoring function for the &lt;code&gt;fit()&lt;/code&gt; method?&lt;/p&gt;&#xA;" OwnerUserId="37544" LastEditorUserId="29575" LastEditDate="2017-08-04T16:59:29.787" LastActivityDate="2018-07-18T17:25:27.160" Title="Custom metrics for unbalanced classes problem in RandomForest or SVM" Tags="|python|scikit-learn|random-forest|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21962" PostTypeId="2" ParentId="20495" CreationDate="2017-08-04T16:35:48.250" Score="2" Body="&lt;p&gt;The best method to find themes in a collection of documents is &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;topic modeling&lt;/a&gt;. Topic modeling finds the hidden (aka, latent) themes beyond just keyword counts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many approaches to topic modeling. &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Latent Dirichlet allocation (LDA)&lt;/a&gt; is a standard topic modeling approach. LDA is a probabilistic graphical model that assumes that each document is a mixture of a small number of topics and that each word's creation is attributable to one of the document's topics. The number of topics is a selected hyperparameter.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-04T16:35:48.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21963" PostTypeId="1" AcceptedAnswerId="45910" CreationDate="2017-08-04T16:44:24.353" Score="9" ViewCount="1858" Body="&lt;p&gt;If I have a learning problem that should have an inherent symmetry, is there a way to subject my learning problem to a symmetry constraint to enhance learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I am doing image recognition, I might want 2D rotational symmetry. Meaning that the rotated version of an image should get the same result as the original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if I am learning to play tic-tac-toe, then rotating by 90deg should yield the same game play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has any research been done on this?&lt;/p&gt;&#xA;" OwnerDisplayName="user32280" LastActivityDate="2019-02-20T18:57:40.577" Title="&quot;Deep Noether's Theorem&quot;: Building in Symmetry Constraints" Tags="|machine-learning|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="21966" PostTypeId="1" AcceptedAnswerId="21992" CreationDate="2017-08-04T16:59:57.457" Score="0" ViewCount="52" Body="&lt;p&gt;Sorry if my question is kind of dumb, I am very new to this field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to create a CNN that plays a variant of chess (for the examples, we'll use chess as it is close enough).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My network , which is a policy network, outputs a vector of planes of probabilities. (e.g 1st pawn layer, 2nd rook layer etc..) each layer contains scalars determining how &quot;good&quot; would a move of the piece to this case be, according to the network. My question is, given the input channels &lt;strong&gt;I&lt;/strong&gt;. And expected outpus (0s everywhere and 1 for the move that was played), how to calculate loss for gradient descent ? (e.g : how to evaluate the &quot;closeness&quot; of a move to another.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My wild guess is to weight each layer a &quot;wrongness&quot; factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really &quot;wrong&quot; and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ? And in general , how to compute the loss of a non classifying convolutional neural network ? &lt;/p&gt;&#xA;" OwnerUserId="33973" LastActivityDate="2017-08-05T20:55:14.203" Title="Evaluating loss for non classifying convolutional neural network" Tags="|machine-learning|neural-network|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21971" PostTypeId="2" ParentId="21954" CreationDate="2017-08-04T18:36:26.663" Score="4" Body="&lt;p&gt;OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it holds that if the number of bootstrap samples is large enough, CV and OOB samples will produce the same (or very similar) error estimates. Thus, if you perform many bootstrap samples, I would recommend performing cross-validation along the way with OOB samples until the OOB error converges.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-08-04T18:36:26.663" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="21972" PostTypeId="1" CreationDate="2017-08-04T20:54:57.027" Score="1" ViewCount="568" Body="&lt;p&gt;I'm working on a Reinforcement learning project where the agent needs to navigate itself around the maze and get to the goal. (I used Q Learning as my algorithm) The agent found the optimal path in 50 episodes when using TensorFlow, and consistently obtained the optimum reward. The graph's something like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yLxq0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow Version&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I wanted to use Keras for fast prototyping in the future, I tried coding the exact same algorithm with Keras. However, even though I'm using the same optimization functions(cost functions are slightly different), the AI written on Keras fluctuates a lot. Something like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/5qyxY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras Version&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My backend is TensorFlow, and I tried switching to Theano, but no luck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My TensorFlow code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import random&#xA;import tensorflow as tf&#xA;import matplotlib.pyplot as plt&#xA;&#xA;tf.reset_default_graph()&#xA;&#xA;def action(env, s, a):&#xA;    env[s] = &quot;_&quot;&#xA;    c = True&#xA;    if a == 1 and s &amp;gt;= 5:&#xA;        s1 = s-5&#xA;    elif a == 3 and s &amp;lt;= 19:&#xA;        s1 = s+5&#xA;    elif a == 2 and s%5 != 4:&#xA;        s1 = s+1&#xA;    elif a == 4 and s%5 != 0 and s &amp;gt; 0:&#xA;        s1 = s-1&#xA;    else:&#xA;        s1 = s&#xA;        r = -1&#xA;        d = False&#xA;    if env[s1] == &quot;0&quot;:&#xA;        r = -5&#xA;        d = False&#xA;        c = False&#xA;    elif env[s1] == &quot;G&quot;:&#xA;        r = 100&#xA;        d = True&#xA;        c = False&#xA;    else:&#xA;        r = -1&#xA;        d = False&#xA;    return s1, r, d, c&#xA;&#xA;def reset_env():&#xA;#The maze&#xA;    default_env = [&quot;_&quot;] * 25&#xA;    default_env[0] = &quot;A&quot;&#xA;    default_env[1] = &quot;0&quot;&#xA;    default_env[7] = &quot;0&quot;&#xA;    default_env[8] = &quot;0&quot;&#xA;    default_env[11] = &quot;0&quot;&#xA;    default_env[24] = &quot;G&quot;&#xA;    return default_env&#xA;&#xA;inputs1 = tf.placeholder(shape=[1,25],dtype=tf.float32)&#xA;W = tf.Variable(tf.random_uniform([25,4],0,0.01))&#xA;Qout = tf.matmul(inputs1,W)&#xA;predict = tf.argmax(Qout,1)&#xA;&#xA;nextQ = tf.placeholder(shape=[1,4],dtype=tf.float32)&#xA;loss = tf.reduce_sum(tf.square(nextQ - Qout))&#xA;trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1)&#xA;updateModel = trainer.minimize(loss)&#xA;&#xA;init = tf.global_variables_initializer()&#xA;&#xA;y = 0.5&#xA;e = 0&#xA;num_episodes = 500&#xA;&#xA;rList = []&#xA;with tf.Session() as sess:&#xA;    sess.run(init)&#xA;    success = 0&#xA;    for i in range(num_episodes):&#xA;        rAll = 0&#xA;        s = 0&#xA;        d = False&#xA;        env = reset_env()&#xA;        j = 0&#xA;        c = True&#xA;        print &quot;------------------START-------------------&quot;&#xA;        while j &amp;lt; 100:&#xA;            j += 1&#xA;            a,allQ = sess.run([predict,Qout],feed_dict=    {inputs1:np.identity(25)[s:s+1]})&#xA;            if np.random.rand(1) &amp;lt; e:&#xA;                a[0] = random.randint(0,3)&#xA;            s1,r,d,c = action(env, s, int(a[0]+1))&#xA;            #print s1, r, d, c&#xA;            env[s1] = &quot;A&quot;&#xA;            Q1 = sess.run(Qout,feed_dict={inputs1:np.identity(25)[s1:s1+1]})&#xA;            maxQ1 = np.max(Q1)&#xA;            targetQ = allQ&#xA;            targetQ[0,a[0]] = r + y*maxQ1&#xA;            _,W1 = sess.run([updateModel,W],feed_dict={inputs1:np.identity(25)[s:s+1],nextQ:targetQ})&#xA;            rAll += r&#xA;            s = s1&#xA;            if c == False:&#xA;                break&#xA;            if d == True:&#xA;                e = 1./((i/50) + 10)&#xA;                success += 1&#xA;                break&#xA;        rList.append(rAll)&#xA;plt.plot(rList)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My Keras code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import random&#xA;import matplotlib.pyplot as plt&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras import optimizers&#xA;import numpy&#xA;&#xA;def action(env, s, a):&#xA;    env[s] = &quot;_&quot;&#xA;    c = True&#xA;    if a == 1 and s &amp;gt;= 5:&#xA;        s1 = s-5&#xA;    elif a == 3 and s &amp;lt;= 19:&#xA;        s1 = s+5&#xA;    elif a == 2 and s%5 != 4:&#xA;        s1 = s+1&#xA;    elif a == 4 and s%5 != 0 and s &amp;gt; 0:&#xA;        s1 = s-1&#xA;    else:&#xA;        s1 = s&#xA;        r = -1&#xA;    if env[s1] == &quot;0&quot;:&#xA;        r = -10&#xA;        d = False&#xA;        c = False&#xA;    elif env[s1] == &quot;G&quot;:&#xA;        r = 100&#xA;        d = True&#xA;        c = False&#xA;    else:&#xA;        r = -1&#xA;        d = False&#xA;    return s1, r, d, c&#xA;&#xA;def reset_env():&#xA;    default_env = [&quot;_&quot;] * 25&#xA;    default_env[0] = &quot;A&quot;&#xA;    default_env[1] = &quot;0&quot;&#xA;    default_env[7] = &quot;0&quot;&#xA;    default_env[8] = &quot;0&quot;&#xA;    default_env[11] = &quot;0&quot;&#xA;    default_env[24] = &quot;G&quot;&#xA;    return default_env&#xA;&#xA;sgd = optimizers.SGD(lr=0.1)&#xA;model = Sequential()&#xA;model.add(Dense(4, input_dim=25, kernel_initializer='uniform'))&#xA;model.compile(loss='mean_squared_error',&#xA;          optimizer=sgd,&#xA;          metrics=['accuracy'])&#xA;y = 0.5&#xA;e = 0&#xA;num_episodes = 100&#xA;&#xA;rList = []&#xA;success = 0&#xA;for i in range(num_episodes):&#xA;    rAll = 0&#xA;    s = 0&#xA;    d = False&#xA;    env = reset_env()&#xA;    j = 0&#xA;    c = True&#xA;    print &quot;---------------START-------------------&quot;&#xA;    while j &amp;lt; 100:&#xA;        j += 1&#xA;        allQ = model.predict(np.identity(25)[s:s+1])[0]&#xA;        a = allQ.argmax()&#xA;        if np.random.rand(1) &amp;lt; e:&#xA;           a = random.randint(0,3)&#xA;        s1,r,d,c = action(env, s, int(a+1))&#xA;        env[s1] = &quot;A&quot;&#xA;        Q1 = model.predict(np.identity(25)[s1:s1+1])[0]&#xA;        maxQ1 = np.max(Q1)&#xA;        targetQ = allQ&#xA;        targetQ[a] = r + y*maxQ1&#xA;        model.fit(np.identity(25)[s:s+1], np.array([targetQ]), epochs=1, verbose=0)&#xA;        rAll += r&#xA;        s = s1&#xA;        if c == False:&#xA;            break&#xA;        if d == True:&#xA;            e = 1./((i/50) + 10)&#xA;            success += 1&#xA;            break&#xA;    rList.append(rAll)&#xA;plt.plot(rList)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="37548" LastEditDate="2017-08-11T22:51:52.847" LastActivityDate="2017-08-11T22:51:52.847" Title="Keras not converging to optimum while TensorFlow does" Tags="|machine-learning|tensorflow|keras|reinforcement-learning|q-learning|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="21973" PostTypeId="1" CreationDate="2017-08-04T21:30:02.827" Score="0" ViewCount="98" Body="&lt;p&gt;I have trained a model on a training set, which is not that big (overall around 120 true positives, and of course lots of negative examples). What I am trying to do is to improve the results by increasing the data size. I tried two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I added data from a different domain and concatenated the data with the existing one. It increased the F-score from 0.13 to 0.14.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I added the same extra data instances, but this time with a domain adaptation technique (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=6CA64C881358DBBF8B4D53FE84DB1253?doi=10.1.1.110.2062&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature augmentation&lt;/a&gt;). This time, I got an improvement of around 6 percent which was significant.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is why does the first approach help a little bit?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The additional data set was around 10 times bigger than the main data set. That is, adding around 1000 instances from a different domain increased the results 1 percent. If it didn't help at all, I think, that would be easier to explain. But now, I don't understand. If the data is biased and test set and training set are so close to each other such that the additional data set can not help, why do I get 1 percent improvement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, how do I get almost the same result when the main data set in this situation is just around 1/10 of the data? I don't understand the behaviour of the model. I am using a &lt;code&gt;Naive Bayse&lt;/code&gt; classifier.&lt;/p&gt;&#xA;" OwnerUserId="37549" LastEditorUserId="29575" LastEditDate="2017-08-06T10:25:03.673" LastActivityDate="2017-08-06T10:25:03.673" Title="Why does increasing the training set size not improve the results?" Tags="|classification|naive-bayes-classifier|domain-adaptation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21974" PostTypeId="1" CreationDate="2017-08-04T21:59:31.523" Score="0" ViewCount="420" Body="&lt;p&gt;As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix for that. A few steps before the Q-matrix, I need the similarity matrices made using the un-normalized Student-t kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using r, not sure if that's relevant to an answer.&lt;/p&gt;&#xA;" OwnerUserId="37550" LastActivityDate="2024-01-15T01:07:40.907" Title="How do I calculate a similarity matrix with a Student-t kernel?" Tags="|r|dataset|visualization|similarity|tsne|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21975" PostTypeId="1" CreationDate="2017-08-04T22:18:11.417" Score="1" ViewCount="651" Body="&lt;p&gt;Imagine a fully-connected neural network with its last two layers of the following structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Dense]&#xA;    units = 612&#xA;    activation = softplus&#xA;&#xA;[Dense]&#xA;    units = 1&#xA;    activation = sigmoid&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output value of the net is 1, but I'd like to know what the input x to the sigmoidal function was (must be some high number, since sigm(x) is 1 here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Folllowing &lt;a href=&quot;https://stackoverflow.com/a/41712013/1922302&quot;&gt;indraforyou's&lt;/a&gt; answer I managed to retrieve the output and weights of Keras layers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;outputs = [layer.output for layer in model.layers[-2:]]&#xA;functors = [K.function( [model.input]+[K.learning_phase()], [out] ) for out in outputs]&#xA;&#xA;test_input = np.array(...)&#xA;layer_outs = [func([test_input, 0.]) for func in functors]&#xA;&#xA;print layer_outs[-1][0]  # -&amp;gt; array([[ 1.]])&#xA;&#xA;dense_0_out = layer_outs[-2][0]                           # shape (612, 1)&#xA;dense_1_weights = model.layers[-1].weights[0].get_value() # shape (1, 612)&#xA;dense_1_bias = model.layers[-1].weights[1].get_value()&#xA;&#xA;x = np.dot(dense_0_out, dense_1_weights) + dense_1_bias&#xA;print x # -&amp;gt; -11.7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can x be a negative number? In that case the last layers output should be a number closer to 0.0 than 1.0. Are &lt;code&gt;dense_0_out&lt;/code&gt; or &lt;code&gt;dense_1_weights&lt;/code&gt; the wrong outputs or weights?&lt;/p&gt;&#xA;" OwnerUserId="35917" LastActivityDate="2017-08-04T22:18:11.417" Title="Keras retrieve value of node before activation function" Tags="|python|neural-network|keras|logistic-regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21976" PostTypeId="2" ParentId="21973" CreationDate="2017-08-04T22:46:01.017" Score="1" Body="&lt;p&gt;The model that you are generating is most probably under-fitting the model. Imagine, you are fitting a straight line to a non-linear data. Certainly even if you are increasing your sample size, the model is still not getting any better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Study about learning curves and how they can be used to solve the under-fitting!&lt;/p&gt;&#xA;" OwnerUserId="29805" LastActivityDate="2017-08-04T22:46:01.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21978" PostTypeId="2" ParentId="21650" CreationDate="2017-08-05T01:26:39.747" Score="2" Body="&lt;p&gt;These specific ones could be pure heuristic. For images though it is pretty standard: change RGB to BGR and subtract the mean from every pixel. This is used in all contests/datasets like Imagenet, Pascal VOC, MS COCO. The reason is that the network is presented with a standardized dataset, as all images can be very different. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2017-08-05T01:26:39.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21980" PostTypeId="1" CreationDate="2017-08-05T08:07:12.770" Score="1" ViewCount="564" Body="&lt;p&gt;I want to convert the input data available as C4.5 file format into a R data frame. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data set can be downloaded from the following link&#xA;&lt;a href=&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/communities/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://archive.ics.uci.edu/ml/machine-learning-databases/communities/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and is avaialble as .NAMES and .DATA files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach was to convert it to a csv file using WEKA and then import the csv file into R. Could not do it as WEKA detected a error while parsing the C4.5 file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can some one point me in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="37561" LastEditorUserId="1330" LastEditDate="2020-04-22T22:01:35.727" LastActivityDate="2024-02-19T14:07:55.993" Title="Importing C4.5 file to R dataframe" Tags="|r|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21982" PostTypeId="1" CreationDate="2017-08-05T12:44:43.043" Score="1" ViewCount="1783" Body="&lt;p&gt;I am working on developing an algorithm which will predict the future traffic for the restaurant. I am confuse that which of the two: Linear regression or time series analysis I should use as the base for my algorithm. The features I am using are: Day,whether there was festival,temperature,climatic condition , current rating,whether there was holiday,service rating,number of reviews etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please guide me how should I proceed . Also how can I optimize my algorithm so that it can learn with time.&lt;/p&gt;&#xA;" OwnerUserId="37502" LastActivityDate="2022-08-04T19:09:44.260" Title="Time series analysis vs linear regression" Tags="|machine-learning|deep-learning|time-series|logistic-regression|linear-regression|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21983" PostTypeId="2" ParentId="5226" CreationDate="2017-08-05T12:57:12.647" Score="7" Body="&lt;p&gt;You should usually &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;noreferrer&quot;&gt;one-hot encode&lt;/a&gt; categorical variables for scikit-learn models, including random forest. Random forest will often work ok without one-hot encoding but usually performs better if you do one-hot encode. One-hot encoding and &quot;dummying&quot; variables mean the same thing in this context. Scikit-learn has &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;noreferrer&quot;&gt;sklearn.preprocessing.OneHotEncoder&lt;/a&gt; and Pandas has &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;noreferrer&quot;&gt;pandas.get_dummies&lt;/a&gt; to accomplish this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are alternatives. The article &lt;a href=&quot;http://www.kdnuggets.com/2015/12/beyond-one-hot-exploration-categorical-variables.html&quot; rel=&quot;noreferrer&quot;&gt;&quot;Beyond One-Hot&quot; at KDnuggets&lt;/a&gt; does a great job of explaining why you need to encode categorical variables and alternatives to one-hot encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are alternative implementations of random forest that do not require one-hot encoding such as R or H2O. The implementation in R is &lt;a href=&quot;https://stats.stackexchange.com/questions/49243/rs-randomforest-can-not-handle-more-than-32-levels-what-is-workaround/96442#96442&quot;&gt;computationally expensive&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/questions/49243/rs-randomforest-can-not-handle-more-than-32-levels-what-is-workaround&quot;&gt;will not work if your features have many categories&lt;/a&gt;. H2O will work with large numbers of categories. Continuum has made &lt;a href=&quot;https://www.continuum.io/blog/developer-blog/using-anaconda-h2o-supercharge-machine-learning-predictive-analytics&quot; rel=&quot;noreferrer&quot;&gt;H2O available in Anaconda Python.&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/pull/3346&quot; rel=&quot;noreferrer&quot;&gt;ongoing effort to make scikit-learn handle categorical features directly&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.revolutionanalytics.com/2013/07/rxdtree-a-new-type-of-tree-algorithm.html&quot; rel=&quot;noreferrer&quot;&gt;This article&lt;/a&gt; has an explanation of the algorithm used in H2O. It references the academic paper &lt;a href=&quot;http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf&quot; rel=&quot;noreferrer&quot;&gt;A Streaming Parallel Decision Tree Algorithm&lt;/a&gt; and a &lt;a href=&quot;http://largescale.ml.tu-berlin.de/media/abstracts/73_Ben-Haim_Parallel_Decision_Tree.pdf&quot; rel=&quot;noreferrer&quot;&gt;longer version&lt;/a&gt; of the same paper.&lt;/p&gt;&#xA;" OwnerUserId="26121" LastActivityDate="2017-08-05T12:57:12.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21984" PostTypeId="1" AcceptedAnswerId="25343" CreationDate="2017-08-05T14:40:09.643" Score="7" ViewCount="867" Body="&lt;p&gt;Right now, I am working on preparing a small dataset for release to the public by getting rid of sensitive information.  While working on it, I wondered... what are the best practices of dealing private or sensitive polynomial attributes in a dataset?(*) I have heard to create anonymity or to permute is achieved by De-identification, obscurification,  anonymization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would like to learn more about this topic in data science/analysis.   I am particularly interested in the packages and concepts that can be use in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(*)Well besides the obvious of solutions of completely removing the sensitive attribute or using hashcode encryption.  I am a little bit familiar with how this problem can be complicated by the ability to correlate attributes&lt;/p&gt;&#xA;" OwnerUserId="37571" LastEditorUserId="20372" LastEditDate="2017-12-06T10:36:13.920" LastActivityDate="2018-01-05T11:08:18.640" Title="How do we make data Obfuscate or &quot;De-identificate&quot; to make it anonymous and share it publicly?" Tags="|r|dataset|anonymization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="21986" PostTypeId="1" CreationDate="2017-08-05T16:11:53.327" Score="0" ViewCount="2553" Body="&lt;p&gt;Why it's so that in convolutional neural networks we generally take the image dimensions of input image to be generally a &lt;em&gt;square?&lt;/em&gt; We even do padding to make it happen. Why not different dimensions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I understood is that computer computes multiplication and division (by 2) much faster than the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone shed some light on this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any link or reference will be appreciated. I already have CS231n notes and lecture slides. &lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="24000" LastEditDate="2017-08-06T03:10:26.093" LastActivityDate="2017-08-06T05:52:18.923" Title="Why CNN input images are often square shaped?" Tags="|neural-network|deep-learning|convolution|" AnswerCount="1" CommentCount="0" ClosedDate="2017-08-06T09:06:56.923" ContentLicense="CC BY-SA 3.0" />
  <row Id="21987" PostTypeId="2" ParentId="21982" CreationDate="2017-08-05T16:20:44.127" Score="0" Body="&lt;p&gt;Why do you not try both? Test the accuracy of the methods in your test and cross validation set. Use learning curves related techniques to come to a experimental logical conclusion. Remember, this is data &quot;science&quot;!&lt;/p&gt;&#xA;" OwnerUserId="29805" LastActivityDate="2017-08-05T16:20:44.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21988" PostTypeId="1" CreationDate="2017-08-05T17:05:09.833" Score="1" ViewCount="1801" Body="&lt;p&gt;New to data science and I am working on a personal project in sports analytics. &#xA;I have data in the form of one dataset for each year/season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each dataset has the team's wins, losses, and various offensive and defensive stats.&#xA;These will be &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pandas.DataFrames&lt;/code&gt;&lt;/a&gt; indexed by team name. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to predict wins and losses based on this data for each year, and then create a more generalized model to predict next season using all of the previous years data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having difficulty deciding how best to combine the data. I think that it would be very inefficient to combine all of the datasets into 1 and have variables for each stat in the year prior, 2 years prior, etc.&#xA;How is this type of sequential data usually handled?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For clarification, I plan on using &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;nofollow noreferrer&quot;&gt;gradient boosting&lt;/a&gt; via xgboost package and an &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM&lt;/a&gt; via the Keras package)&lt;/p&gt;&#xA;" OwnerUserId="37574" LastEditorUserId="29575" LastEditDate="2017-08-06T10:59:48.557" LastActivityDate="2019-12-27T12:01:10.103" Title="How to handle multiple datasets of sequential data" Tags="|machine-learning|dataset|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21990" PostTypeId="2" ParentId="21980" CreationDate="2017-08-05T18:43:38.727" Score="0" Body="&lt;pre&gt;&lt;code&gt;URL &amp;lt;- 'https://archive.ics.uci.edu/ml/machine-learning-databases/communities/communities.data'&#xA;data &amp;lt;- read.table(URL, sep = ',',  na.strings = '?')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2968" LastActivityDate="2017-08-05T18:43:38.727" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="21992" PostTypeId="2" ParentId="21966" CreationDate="2017-08-05T20:21:07.317" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My wild guess is to weight each layer a &quot;wrongness&quot; factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really &quot;wrong&quot; and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can relatively simply teach a policy network to predict human-like moves in your scheme using a database of moves. Actually your &quot;wrongness&quot; is probably well-enough represented by classification (your positive class might be &quot;This is a good move&quot;) and the usual log loss that goes with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you have trained a policy network, you will want to look in depth at the literature for game-playing bots. Your policy network might work quite well alongside &lt;a href=&quot;https://en.wikipedia.org/wiki/Monte_Carlo_tree_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;Monte Carlo Tree Search&lt;/a&gt;, provided you have some kind of evaluation heuristic for the resulting position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;reinforcement learning&lt;/a&gt; approach to learn from self-play would take you further, enabling the bot to teach itself about good and bad moves/positions, but is too complex to explain in an answer here. I suggest look into the subject after training your network and seeing how good a player you can create using just the policy network and a move search algorithm.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And in general , how to compute the loss of a non classifying convolutional neural network ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are a few common options available for regression, such as mean square error (MSE) i.e. $\frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$ where $\hat{y}_i$ is your prediction and $y_i$ is the ground truth for each example. If you use this loss function, and want to predict values outside of range 0-1, remember to use a linear output layer (i.e. no activation function after the last layer), so that the network can actually output close to the values you need - that's about the only difference in network architecture you need to care about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the more general case of game-playing bots, it is usual (but not required) to calculate a predicted &quot;return&quot; or &quot;utility&quot; which is the sum of all rewards that will be gained by continuing to act in a certain way. MSE loss is a good choice for that. Although for zero-sum two player games where the reward is simply win/lose, you can use a sigmoid output layer (predicting chance of a win) and cross-entropy loss, much like a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your specific case, you &lt;em&gt;can&lt;/em&gt; treat your initial policy network as a classifier. This immediately gives you some probability weightings for the predicted move, which you can use to pick the predicted best play, or maybe to guide &lt;a href=&quot;https://en.wikipedia.org/wiki/Monte_Carlo_tree_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;Monte Carlo Tree Search&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The kind of network that predicts &lt;em&gt;utility&lt;/em&gt; or &lt;em&gt;return&lt;/em&gt; from a position (or combination of position and action) is called a value network (or an action-value network) in reinforcement learning . If you have &lt;em&gt;both&lt;/em&gt; a policy network and a value network, then you are on the way to creating an &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2&quot; rel=&quot;nofollow noreferrer&quot;&gt;actor-critic&lt;/a&gt; algorithm.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-05T20:55:14.203" LastActivityDate="2017-08-05T20:55:14.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21994" PostTypeId="1" AcceptedAnswerId="21999" CreationDate="2017-08-05T22:28:15.163" Score="2" ViewCount="720" Body="&lt;p&gt;My data is coming from a CSV, which should be visualized in Tableau.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the data contains the column &lt;code&gt;category_list&lt;/code&gt;, which consists of values separated by a vertical bar (&lt;code&gt;|&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Tableau can't handle arrays inside of attributes, I used Python (Pandas) to load the CSV and manipulate the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;companies = pd.read_csv(&quot;companies.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I assume that the &lt;code&gt;category_list&lt;/code&gt; column needs to be broken down and stored into another CSV (containing the &lt;code&gt;permalink&lt;/code&gt; (unique ID) and &lt;code&gt;category&lt;/code&gt; pairs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;permalink,category&#xA;/organization/-qounter,Application Platforms&#xA;/organization/-qounter,Real Time&#xA;/organization/-qounter,Social Network Media&#xA;/organization/-the-one-of-them-inc-,Apps&#xA;/organization/-the-one-of-them-inc-,Games&#xA;/organization/-the-one-of-them-inc-,Mobile&#xA;/organization/1-4-all,Entertainment&#xA;/organization/1-4-all,Games&#xA;/organization/1-4-all,Software&#xA;/organization/1-800-publicrelations-inc-,Internet&#xA;/organization/1-800-publicrelations-inc-,Marketing&#xA;/organization/1-800-publicrelations-inc-,Media&#xA;/organization/1-800-publicrelations-inc-,Public Relations&#xA;/organization/1-mainstream,Apps&#xA;/organization/1-mainstream,Cable&#xA;/organization/1-mainstream,Distribution&#xA;/organization/1-mainstream,Software&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to achieve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excerpt of the original CSV:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;permalink,category_list,...&#xA;/organization/-qounter,Application Platforms|Real Time|Social Network Media,...&#xA;/organization/-the-one-of-them-inc-,Apps|Games|Mobile,...&#xA;/organization/1-4-all,Entertainment|Games|Software,...&#xA;/organization/1-800-publicrelations-inc-,Internet|Marketing|Media|Public Relations,...&#xA;/organization/1-mainstream,Apps|Cable|Distribution|Software,...&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13235" LastActivityDate="2018-11-10T10:18:11.587" Title="Breaking down a column in Pandas into a separate CSV for display in Tableau" Tags="|python|pandas|csv|tableau|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21995" PostTypeId="2" ParentId="21988" CreationDate="2017-08-06T00:06:21.523" Score="1" Body="&lt;p&gt;Generally, to perform machine learning all data needs to be in a single dataframe. Team name, better yet team id, should be the primary key. Then modeling each year requires selecting the relevant columns (e.g., wins_year_1, loses_year_1, …). Finally, the general model would select all the columns. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is best practice to transform all data before modeling. This avoids making transformation errors on some of the data which can lead to errors in modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pandas support this type of combining through the &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;merge&lt;/a&gt; command.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, this data is not called sequential data. Sequential data implies a notion of state. The state changes values over time. In most sports, recent performance is not a strong indicator of how likely a team will the next game. This is often referred to as the Hot Hand Fallacy. Thus, current &quot;win/lose&quot; state of a team is not a strong feature in modeling.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-06T00:06:21.523" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="21996" PostTypeId="2" ParentId="5427" CreationDate="2017-08-06T00:15:03.323" Score="2" Body="&lt;p&gt;Data augmentation is the process of synthetically creating samples based on existing data. Existing data is slightly perturbed to generate novel data that retains many of the original data properties. For example, if the data is images. Image pixels can be swapped. Many examples of data augmentation techniques can be found &lt;a href=&quot;https://arxiv.org/pdf/1609.08764.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-06T00:15:03.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21997" PostTypeId="2" ParentId="10219" CreationDate="2017-08-06T00:35:20.780" Score="2" Body="&lt;p&gt;One method is using &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing&quot; rel=&quot;nofollow noreferrer&quot;&gt;locality-sensitive hashing&lt;/a&gt; (LSH), an approximate nearest neighbor search method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each document, both resumes and job descriptions, is hashed into the same space. There are several ways to perform the hashing. An older method is &lt;a href=&quot;https://en.wikipedia.org/wiki/W-shingling&quot; rel=&quot;nofollow noreferrer&quot;&gt;shingling&lt;/a&gt;. The entire process is outlined in &lt;a href=&quot;http://infolab.stanford.edu/~ullman/mmds/ch3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chapter 3: Finding Similar Items&lt;/a&gt; in &lt;a href=&quot;http://www.mmds.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mining of Massive Datasets&lt;/a&gt;. A newer method is &lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once all the documents are in the same space, LSH maps similar items to the same “buckets” with high probability. It is a type of hashing where collisions are features, not bugs. The collisions are a variation of clustering. Given a new document, all similar documents are retrieved also. Given a resume, retrieve similar job descriptions. Given a job description, retrieve similar resumes. (After retrieval, filter out documents in the same document class)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since LSH is an approximate algorithm, it scales well to millions of documents and handles noisy data. Resumes and job descriptions are noisy descriptions so it is appropriate to have an algorithm that can handle noise.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-06T00:35:20.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21998" PostTypeId="2" ParentId="20587" CreationDate="2017-08-06T03:35:39.867" Score="3" Body="&lt;p&gt;@Stephen Rauch 's answer is very smart, but it's slow when I applied it to a large dataset. Inspired by &lt;a href=&quot;https://stackoverflow.com/questions/24885092/finding-the-consecutive-zeros-in-a-numpy-array&quot;&gt;this post&lt;/a&gt;, I think I got a more efficient way to achieve the same goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame({&#xA;    'names': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L'],&#xA;    'col1': [0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0],&#xA;    'col2': [0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]}).set_index('names')&#xA;&#xA;for i in range(df.shape[1]):&#xA;    iszero = np.concatenate(([0], np.equal(df.iloc[:, i].values, 0).view(np.int8), [0]))&#xA;    absdiff = np.abs(np.diff(iszero))&#xA;    zerorange = np.where(absdiff == 1)[0].reshape(-1, 2)&#xA;    for j in range(len(zerorange)):&#xA;        if zerorange[j][1] - zerorange[j][0] &amp;lt; 3:&#xA;            df.iloc[zerorange[j][0]:zerorange[j][1], i] = 1&#xA;print(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Results:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        col1  col2&#xA;names            &#xA;A         1     0&#xA;B         1     0&#xA;C         1     0&#xA;D         1     0&#xA;E         1     1&#xA;F         1     1&#xA;G         0     1&#xA;H         0     1&#xA;I         0     1&#xA;J         1     0&#xA;K         1     0&#xA;L         1     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35420" LastActivityDate="2017-08-06T03:35:39.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="21999" PostTypeId="2" ParentId="21994" CreationDate="2017-08-06T03:55:06.407" Score="4" Body="&lt;p&gt;If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with &quot;permalink&quot; column&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.set_index('permalink').category_list.str.split('|', expand=True).stack().reset_index('permalink').rename(columns={0:'category'})&#xA;&#xA;permalink   category&#xA;0   /organization/-qounter  Application Platforms&#xA;1   /organization/-qounter  Real Time&#xA;2   /organization/-qounter  Social Network Media&#xA;0   /organization/-the-one-of-them-inc- Apps&#xA;1   /organization/-the-one-of-them-inc- Games&#xA;2   /organization/-the-one-of-them-inc- Mobile&#xA;0   /organization/1-4-all   Entertainment&#xA;1   /organization/1-4-all   Games&#xA;2   /organization/1-4-all   Software&#xA;0   /organization/1-800-publicrelations-inc-    Internet&#xA;1   /organization/1-800-publicrelations-inc-    Marketing&#xA;2   /organization/1-800-publicrelations-inc-    Media&#xA;3   /organization/1-800-publicrelations-inc-    Public Relations&#xA;0   /organization/1-mainstream  Apps&#xA;1   /organization/1-mainstream  Cable&#xA;2   /organization/1-mainstream  Distribution&#xA;3   /organization/1-mainstream  Software&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can save it to csv.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-08-06T03:55:06.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22000" PostTypeId="1" CreationDate="2017-08-06T04:26:40.267" Score="2" ViewCount="117" Body="&lt;p&gt;I have a long data frame having 96 consecutive observation belonging to same variable.&#xA;I have to two such variables. One variable is Energy denoted by &quot;param2(kWh)&quot; and another is demand denoted by &quot;param2(kW)&quot;. So data frame is like--- &lt;/p&gt;&#xA;&#xA;&lt;p&gt;first row has variables described as &quot;Interval&quot; and &quot;param2(kWh)&quot; and then 96 observations belonging to Energy variable, followed by 98th row having variables described a &quot;Interval&quot; and &quot;param2(kW)&quot; and then 96 observations belonging to Demand variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again after that there is row described as &quot;Interval&quot; and &quot;param2(kWh)&quot; and then 96 observations belonging to Energy variable. After that again there is row describing variables &quot;Interval&quot; and &quot;param2(kW)&quot; and 96 observations belonging to Demand variable and such intertwining of these two variables i.e. Energy and Demand continues for long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to have two separate data frames one for Energy i.e. &quot;param2(kWh)&quot; and one for Demand i.e. &quot;param2(kW)&quot;. &#xA;How these two data frames be formed from the above single data frame?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am attaching the internet site of the data frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://1drv.ms/f/s!Apm2LEjQkhz3aNBBoUygHPMz2_E&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://1drv.ms/f/s!Apm2LEjQkhz3aNBBoUygHPMz2_E&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="30006" LastEditDate="2017-08-06T12:09:56.883" LastActivityDate="2017-08-10T15:12:26.763" Title="Data manipulation in R" Tags="|r|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22001" PostTypeId="2" ParentId="21986" CreationDate="2017-08-06T05:52:18.923" Score="1" Body="&lt;p&gt;I'm not sure, (also studied the CS231n class), but I'm guessing it's something to do with:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;matrix operations, &lt;/li&gt;&#xA;&lt;li&gt;convention, and/or &lt;/li&gt;&#xA;&lt;li&gt;control size of image out.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Not a lot of evidence points towards (1), cause we have insane libraries to do this (ATLAS, MKL, or OpenBLAS). (2) makes more sense practically. (3), because you mentioned zero padding (hyperparameter), the CS231n course says this is done to control the size of output from the current CNN layer and control the spatial size so that input (h x w) = output (h x w). Also, I guess it also reduces errors in matrix operations when you have the same image size (h = w) instead of (h &gt; w or h &amp;lt; w).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely, (3) is the predominant reason. This is just my thoughts. Someone please confirm.&lt;/p&gt;&#xA;" OwnerUserId="37495" LastActivityDate="2017-08-06T05:52:18.923" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22002" PostTypeId="1" CreationDate="2017-08-06T07:52:00.267" Score="3" ViewCount="685" Body="&lt;p&gt;I have a dataset of shape &lt;code&gt;105 x 501 x 266&lt;/code&gt; where &lt;code&gt;105&lt;/code&gt; is the number of data and &lt;code&gt;501 x 266&lt;/code&gt; is the shape of &lt;code&gt;1&lt;/code&gt; data i.e. The labels_dataset is of shape &lt;code&gt;105 x 1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each value of the &lt;code&gt;501 x 266 matrix&lt;/code&gt; is a complex number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it essentially becomes &lt;code&gt;501 * 266 * 2&lt;/code&gt;(real and imaginary part of the number)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now I have to feed this data to a CNN.&#xA;I 'm new to training networks. So need to know whether my data is in best possible form for the CNN or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've printed out max, min, sd, mean of real part, imaginary part and magnitude of the dataset for more info:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;max real = 0.186396, min real = -0.204375&#xA;max imag = 0.166608, min imag = -0.159017&#xA;max abs = 0.219019, min abs = 2.33527e-10&#xA;mean real = 4.01718e-10, complex = 6.79294e-15, abs = 8.82916e-05&#xA;std dev real = 0.000442753, complex = 0.000400677, abs = 0.000590573&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this a good form of data for input to a CNN. What are the options to make it more suitable?&lt;/p&gt;&#xA;" OwnerUserId="37584" LastEditorUserId="28175" LastEditDate="2018-05-20T14:09:02.120" LastActivityDate="2018-05-20T20:06:30.923" Title="What pre processing should I use on data to feed into a CNN?" Tags="|machine-learning|neural-network|deep-learning|cnn|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22003" PostTypeId="1" CreationDate="2017-08-06T08:11:00.520" Score="1" ViewCount="109" Body="&lt;p&gt;I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (&lt;a href=&quot;https://github.com/CamDavidsonPilon/lifetimes&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/CamDavidsonPilon/lifetimes&lt;/a&gt;). The lifetimes lib in python uses a minimize function that has the data arrays as input parameters. The format is something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date,id&#xA;2014-03-08 00:00:00,0&#xA;2014-05-21 00:00:00,1&#xA;2014-03-14 00:00:00,2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We need to implement this same function in scala, but the implementation available here (&lt;a href=&quot;https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala&lt;/a&gt;) has a sligthly different format for the data input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   * @param data - Input data for L-BFGS. RDD of the set of data examples, each of&#xA;   *               the form (label, [feature values]).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Question is, how could I change the scala implementation to accept the data as just plain arrays or in the format given by the lifetimes lib in python?&lt;/p&gt;&#xA;" OwnerUserId="31307" LastActivityDate="2017-08-06T08:11:00.520" Title="How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?" Tags="|apache-spark|optimization|scala|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22004" PostTypeId="1" AcceptedAnswerId="22017" CreationDate="2017-08-06T08:45:17.383" Score="0" ViewCount="152" Body="&lt;p&gt;I'm reading page 495 of &lt;a href=&quot;http://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elements of Statistical Learning&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here we discuss a technique for transforming the density estimation problem&#xA;into one of supervised function approximation. This forms the basis&#xA;for the generalized association rules described in the next section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $g(x)$ be the unknown data probability density to be estimated, and&#xA;$g_0(x)$ be a specified probability density function used for reference. For example,&#xA;$g_0(x)$ might be the uniform density over the range of the variables.&#xA;Other possibilities are discussed below. The data set $x_1, x_2, . . . , x_N$ is presumed&#xA;to be an i.i.d. random sample drawn from $g(x)$. A sample of size $N_0$&#xA;can be drawn from $g_0(x)$ using Monte Carlo methods. Pooling these two&#xA;data sets, and assigning &lt;strong&gt;mass&lt;/strong&gt; $w = N_0/(N + N_0)$ to those drawn from $g(x)$,&#xA;and $w_0 = N/(N + N_0)$ to those drawn from $g_0(x)$, results in a random&#xA;sample drawn from the mixture density $(g(x) + g_0(x)) /2$. If one assigns&#xA;the value $Y = 1$ to each sample point drawn from $g(x)$ and $Y = 0$ those&#xA;drawn from $g_0(x)$, then&#xA;$$μ(x) = E(Y | x) =\frac{g(x)}&#xA;{g(x) + g_0(x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can be estimated by supervised learning using the combined sample&#xA;$(y_1, x_1), (y_2, x_2), . . . , (y_{N+N_0}, x_{N+N_0})$&#xA;as training data.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I don't understand what are &lt;strong&gt;mass&lt;/strong&gt; and &lt;strong&gt;mixture density&lt;/strong&gt;. And I want to know &#xA;why $μ(x) = E(Y | x) = \frac{g(x)}{g(x) + g_0(x)}$.&lt;/p&gt;&#xA;" OwnerUserId="37590" LastEditorUserId="37590" LastEditDate="2017-08-06T11:34:33.120" LastActivityDate="2017-08-06T23:29:45.250" Title="About A technique mentioned on ESL to transform an unsupervised into supervised" Tags="|machine-learning|supervised-learning|unsupervised-learning|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22005" PostTypeId="2" ParentId="21994" CreationDate="2017-08-06T09:51:07.780" Score="0" Body="&lt;p&gt;If it is a one-time transformation, you can simply separate the field's contents in Excel itself.&#xA;Execute the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Open the file in MS Excel&lt;/li&gt;&#xA;&lt;li&gt;Go to the Data tab&lt;/li&gt;&#xA;&lt;li&gt;Click on the Text to Columns option&lt;/li&gt;&#xA;&lt;li&gt;Select Delimited&lt;/li&gt;&#xA;&lt;li&gt;Set | or any other desired character to be the field separator / splitter&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, this is only in case it is not a periodic transformation requirement. If so, @Andrey Lukyanenko's solution is a better one.&lt;/p&gt;&#xA;" OwnerUserId="25742" LastActivityDate="2017-08-06T09:51:07.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22006" PostTypeId="2" ParentId="13059" CreationDate="2017-08-06T09:58:19.980" Score="3" Body="&lt;p&gt;When faced with such situations (loading &amp;amp; appending multi-GB csv files), I found @user666's option of loading one data set (e.g. DataSet1) as a Pandas DF and appending the other (e.g. DataSet2) in chunks to the existing DF to be quite feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code I implement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;amgPd = pd.DataFrame()&#xA;for chunk in pd.read_csv(path1+'DataSet1.csv', chunksize = 100000, low_memory=False):&#xA;    amgPd = pd.concat([amgPd,chunk])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25742" LastActivityDate="2017-08-06T09:58:19.980" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22008" PostTypeId="1" AcceptedAnswerId="22011" CreationDate="2017-08-06T12:18:19.507" Score="2" ViewCount="2370" Body="&lt;p&gt;I have been trying around the &lt;strong&gt;&lt;em&gt;EXTERNAL&lt;/em&gt;&lt;/strong&gt; table concepts in &lt;strong&gt;HIVE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CREATE EXTERNAL TABLE IF NOT EXISTS MovieData&#xA;(id INT, title STRING,releasedate date, videodate date,&#xA;URL STRING,unknown TINYINT, Action TINYINT, Adventure TINYINT, &#xA;Animation TINYINT,Children TINYINT, Comedy TINYINT, Crime TINYINT, &#xA;Documentary TINYINT, Drama TINYINT, Fantasy TINYINT,&#xA;Film-Noir TINYINT, Horror TINYINT, Musical TINYINT, &#xA;Mystery TINYINT, Romance TINYINT, Sci-Fi TINYINT,&#xA;Thriller TINYINT, War TINYINT, Western TINYINT)&#xA;COMMENT 'This is a list of movies and its genre'&#xA;ROW FORMAT DELIMITED&#xA;FIELDS TERMINATED BY '|'&#xA;LINES TERMINATED BY '\n' &#xA;STORED AS TEXTFILE; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Created a table using the above statement and then used the &lt;em&gt;LOAD&lt;/em&gt; statement to get the data populated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LOAD DATA LOCAL INPATH '/home/ubuntu/MovieLens.txt' INTO TABLE MovieData;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next time I &lt;strong&gt;DROP&lt;/strong&gt; the table in HIVE and recreate it again and LOAD the data... But when I do a &lt;strong&gt;COUNT&lt;/strong&gt; operation on the table I get double the values that's present in the file that I loaded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read through few articles that &lt;strong&gt;EXTERNAL&lt;/strong&gt; table does not delete the data but the schema alone from the HIVE metastore... &lt;a href=&quot;https://acadgild.com/blog/managed-and-external-tables-in-hive/&quot; rel=&quot;nofollow noreferrer&quot;&gt;External Table&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please advise why does HIVE behave this way...&lt;/p&gt;&#xA;" OwnerUserId="37397" LastActivityDate="2018-05-24T10:01:35.820" Title="COUNT on External Table in HIVE" Tags="|hive|" AnswerCount="2" CommentCount="0" ClosedDate="2018-08-05T17:50:48.443" ContentLicense="CC BY-SA 3.0" />
  <row Id="22009" PostTypeId="1" CreationDate="2017-08-06T12:39:26.153" Score="1" ViewCount="704" Body="&lt;p&gt;I'm training an &lt;code&gt;xgboost&lt;/code&gt; model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of &lt;code&gt;0.90&lt;/code&gt;. I'm using a special value &lt;code&gt;-65336&lt;/code&gt; for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC &lt;code&gt;0.84&lt;/code&gt;, and my prediction means is a reasonable &lt;code&gt;0.14&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;THEN, I tell XGBOOST that &lt;code&gt;missing_value=-65336&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the cross-validated performance on the train population stays the same.&lt;/li&gt;&#xA;&lt;li&gt;BUT the performance on the general population degrades from AUC &lt;code&gt;0.84&lt;/code&gt; to AUC &lt;code&gt;0.79&lt;/code&gt;. Also, I noticed that the prediction mean is changing from &lt;code&gt;0.14&lt;/code&gt; to &lt;code&gt;0.40&lt;/code&gt;(!)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?&lt;/p&gt;&#xA;" OwnerUserId="16050" LastEditorUserId="83275" LastEditDate="2019-11-30T09:43:05.447" LastActivityDate="2019-11-30T14:38:32.657" Title="XGBOOST missing_value feature degrades my performance?" Tags="|xgboost|missing-data|auc|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22010" PostTypeId="2" ParentId="20117" CreationDate="2017-08-06T13:48:16.287" Score="4" Body="&lt;p&gt;After examining the dataset, we found that the problem was in NUSWIDE dataset itself. Almost half of the dataset didn't have supervised labels (81 labels entered by humans). Also we weren't training on the whole dataset, rather, we were taking random samples because some methods required that the whole dataset was loaded in memory which wasn't feasible. &#xA;After knowing that, we deleted the tuples that didn't have supervised labels, extracted features and done PCA and got a comparable results to SIFT (improved, in some cases). &#xA;So my advice if you face a similar problem is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Make sure that your data labels are valid.&lt;/li&gt;&#xA;&lt;li&gt;Use multiple layers for feature extraction, not just the deeper ones in the network.&lt;/li&gt;&#xA;&lt;li&gt;Using PCA instead of random sampling dataset worked very well for us.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="34110" LastActivityDate="2017-08-06T13:48:16.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22011" PostTypeId="2" ParentId="22008" CreationDate="2017-08-06T13:48:21.043" Score="2" Body="&lt;p&gt;In Hadoop framework, there are multiple way to analyze the data. This depends on your use case, expertise and preference. Hive &lt;code&gt;EXTERNAL&lt;/code&gt; tables are designed in a such way that other programmer can also share the same data location from other data processing model like Pig, MapReduce Programming, Spark and other without affecting each other work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of external table you will not loose the data if you have accidentally dropped your table  as you already know it only drops the meta data and delete the schema and data remain untouched.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method is useful if there is already legacy data in &lt;code&gt;HDFS&lt;/code&gt; on which the user wants to put some metadata so that the data can be queried and manipulated using Hive. Or you are loading data on &lt;code&gt;HDFS&lt;/code&gt; from other ETL tools.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since EXTERNAL table doesn't delete the data and you are loading file again you are getting the count difference. if you are on your own to do all operation like load, analysis, drop etc, Hive support the &lt;code&gt;INTERNAL&lt;/code&gt; table as well. If you want to delete the data when you drop table you can use Hive &lt;code&gt;INTERNAL&lt;/code&gt; table. To create internal table you just need to remove the &lt;code&gt;EXTERNAL&lt;/code&gt; keyword from your query and when you will drop this table it will delete the data as well.&lt;/p&gt;&#xA;" OwnerUserId="37596" LastActivityDate="2017-08-06T13:48:21.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22013" PostTypeId="2" ParentId="21945" CreationDate="2017-08-06T15:23:29.970" Score="1" Body="&lt;p&gt;Following your code in your other thread:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Use tf (raw term count) features for LDA.&#xA;print(&quot;Extracting tf features for LDA...&quot;)&#xA;tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,&#xA;                                max_features=n_features,&#xA;                                stop_words='english')&#xA;t0 = time()&#xA;tf = tf_vectorizer.fit_transform(data_samples)&#xA;print(&quot;done in %0.3fs.&quot; % (time() - t0))&#xA;&#xA;print(&quot;Topic modelling with LDA...&quot;)&#xA;lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,&#xA;                                learning_method='online',&#xA;                                learning_offset=50.,&#xA;                                random_state=0)&#xA;&#xA;lda_x = lda.fit_transform(tf)&#xA;# so lda_x is your doc-topic distribution that you can use for feature vector to your SVM model.&#xA;# lda.components_ is your topic-word distribution.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2017-08-06T15:23:29.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22014" PostTypeId="1" CreationDate="2017-08-06T15:54:43.137" Score="5" ViewCount="414" Body="&lt;p&gt;I am working on project related to prediction of sales quantity  based on previous data.&#xA;Data consists of attributes like  .MRP, Size , Discount,Color,Fabric type e.t.c&#xA;I am  having both sales quantity and Sock on hand in data .Currently I am trying to predict sales quantity using &lt;strong&gt;Random forest Algorithm&lt;/strong&gt;(in R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some basic feature transformations like one hotted categorical variables and performed Normalization on data and training the model.&#xA;   Here I am having some issues in data.I performed some manipulations listed below&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I am having two different  datasets  one contains sales records and other has stock on hand.&#xA;we are matching and aggregating  the records week wise. In some cases stock on hand is not available but sales happened .For those records making sales quantity as stock on hand.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For unsold records we are not having any discount .To keep discount we are taking average of discounts over the same period for particular item from other store (Suggestion by client).&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Some of the  records are having negative sales quantity after aggregation. we are making those negative sales to zero .&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To train the model I am having sales quantity &quot;0&quot;(Zero) for unsold items.In data unsold items are more than sold items (like if sold items are 20  unsold are around  300)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am not getting any considerable results .I tried the same thing with **Neural Network **  also in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am beginner in predictive modelling.This is my first project .Please  correct me if anything wrong in my approach&lt;/p&gt;&#xA;" OwnerUserId="37604" LastActivityDate="2017-08-07T15:18:23.267" Title="Sales Prediction for Fashion Retail Data" Tags="|neural-network|r|predictive-modeling|regression|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22015" PostTypeId="2" ParentId="21919" CreationDate="2017-08-06T19:18:04.487" Score="0" Body="&lt;p&gt;K-means will work really bad on such data, because the method is &lt;em&gt;designed&lt;/em&gt; to process continuous values, where squared errors need to be optimized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than trying to find a hammer that matches your &quot;nail&quot;, you first need to understand your &quot;nail&quot; as it might be a screw! So &lt;em&gt;what&lt;/em&gt; is your objective, &lt;em&gt;what&lt;/em&gt; is an answer result, and &lt;em&gt;when&lt;/em&gt; is a result good? Only then you can find an algorithm to optimize this problem. If you simply try random algorithms, forcing your data into some unnatural form that doesn't preserve the relevant properties, this is a waste of time. They will literally be solving a different problem.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-08-06T19:18:04.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22016" PostTypeId="1" CreationDate="2017-08-06T21:31:54.857" Score="1" ViewCount="2412" Body="&lt;p&gt;When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by rerunning all the code. Is there a better way?&lt;/p&gt;&#xA;" OwnerUserId="37546" LastActivityDate="2021-02-19T13:42:04.643" Title="Unable to immediately continue Jupyer Notebook after closing then reopening the project" Tags="|machine-learning|python|data|ipython|jupyter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22017" PostTypeId="2" ParentId="22004" CreationDate="2017-08-06T23:29:45.250" Score="1" Body="&lt;p&gt;Fascinating problem, welcome to the site! In the second edition, this is covered in section &lt;em&gt;14.2.4 Unsupervised as Supervised Learning&lt;/em&gt;. The idea here is to design a classifier (that's supervised learning) between the desired but unknown distribution, and a known reference distribution (14.10), then to use this classifier along with the reference distribution to estimate the unknown distribution at any point (14.12). You can see that when the classifier deems a point to have come from $g$ ($\hat \mu \to 1$), the density $g_0 \hat \mu /(1-\hat \mu)$ increases. The converse is also true. This is what you would expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mass refers to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Probability_measure&quot; rel=&quot;nofollow noreferrer&quot;&gt;probability measure&lt;/a&gt; of the sample. The idea is to redress by difference in sample sizes by weighting the probability of each sample inversely by its size (or equivalently, the size of the other sample). That's why the density of the mixture is the average of the components. A mixture density is simply a distribution formed by a mixture or sum (of two distributions in this case). Finally, $\mu$ is commonly used in statistics to denote the mean. The conditional mean of $Y$ is simply the probability that $x$ is drawn from $g$ (because the $Y=0$ term does not contribute to the expectation), thus&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu(x) \equiv \mathbb E(Y|x) = \mathrm P(Y=1|x) = \frac{g}{g+g_0}(x)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This equation comes from Bayes' rule, after using $g = P(x|Y=1)$ and $g_0 = P(x|Y=0)$. This approach shows the power of using density &lt;em&gt;ratios&lt;/em&gt;. In fact, there's a &lt;a href=&quot;http://www.cambridge.org/catalogue/catalogue.asp?isbn=1139227289&quot; rel=&quot;nofollow noreferrer&quot;&gt;whole book&lt;/a&gt; devoted it.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-08-06T23:29:45.250" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22018" PostTypeId="1" AcceptedAnswerId="75916" CreationDate="2017-08-07T01:49:36.047" Score="1" ViewCount="290" Body="&lt;p&gt;Are there any Python machine learning libraries, which allow model persistence without pickling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, I want to avoid problems (and time sinks) that could arise upon sharing models with others. (In my case: academic collaborators in non-computational research fields.)&lt;/p&gt;&#xA;" OwnerUserId="23437" LastEditorUserId="23437" LastEditDate="2017-08-07T05:48:32.070" LastActivityDate="2020-06-12T20:20:44.120" Title="Python model persistence without pickling" Tags="|python|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22020" PostTypeId="1" AcceptedAnswerId="22022" CreationDate="2017-08-07T08:40:28.550" Score="1" ViewCount="55" Body="&lt;p&gt;Is there the name for an algorithm of cluster assignment that is based uniquely on the distance between the data point to classify and the center of the cluster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me be more clear:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I have two clusters $A,B$ made by $N$ points $x(i)_A$ and $x(j)_B$. I have to assign a new point $x_{new}$ to it's cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is assigned to the cluster that minimize the distance between the point and the center of the cluster:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ min_{A,B} ( d_A(x_{new}, \frac{1}{N}\sum_{i \in A}x_i), d_B(x_{new}, \frac{1}{N}\sum_{j \in B}x_j) ) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this procedure has a name? Is there an algorithm similar to this? The closest algorithm that I can think of is k-Nearest Neighbour, but it's easy to think of cases where kNN performs properly and this algorithm performs very poorly. For instance, this algorithm as such will perform badly on data shaped non-uniformly / non-gaussian. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question arise from a &lt;a href=&quot;https://arxiv.org/pdf/1307.0411.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; that offer a procedure for doing supervised quantum machine learning, and claims that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Consider the task of assigning a post-processed vector $u \in R^n$ to&#xA;  one of two sets $V, W$, given $M$ representative samples $v_j ∈ V$ and&#xA;  $M$ samples $w_k ∈ W$. A common method for such an assignment is to&#xA;  evaluate the distance $|u−\frac{1}{M}\sum_{j} v_j |$ between $u$ and&#xA;  the mean of the vectors in $V$ , and to assign $u$ to $V$ if this&#xA;  distance is smaller than the distance between $u$ and the mean of $W$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1409" LastActivityDate="2017-08-07T09:16:04.910" Title="Name of this algorithm for supervised cluster assignment" Tags="|machine-learning|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22021" PostTypeId="1" AcceptedAnswerId="22032" CreationDate="2017-08-07T08:51:20.467" Score="6" ViewCount="5345" Body="&lt;p&gt;In the following &lt;a href=&quot;https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1578-z&quot; rel=&quot;noreferrer&quot;&gt;article&lt;/a&gt;, one of the statement is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The K-means algorithm is effective only for spherical datasets&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does spherical dataset mean?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2017-09-06T16:33:03.260" Title="What is the meaning of spherical dataset?" Tags="|classification|dataset|k-means|unsupervised-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22022" PostTypeId="2" ParentId="22020" CreationDate="2017-08-07T09:16:04.910" Score="0" Body="&lt;p&gt;K means clustering kind of works like this.&lt;br&gt;&#xA;It uses the &lt;a href=&quot;https://en.wikipedia.org/wiki/Lloyd%27s_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lloyd algorithm&lt;/a&gt;, which basically has two steps.&lt;br&gt;&#xA;Per each new sample:&lt;br&gt;&#xA; 1) Choose the best cluster, by calculating the minimum distance to available clusters mean.&lt;br&gt;&#xA; 2) Recalculate chosen cluster mean.&lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-08-07T09:16:04.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22023" PostTypeId="1" CreationDate="2017-08-07T09:52:09.357" Score="4" ViewCount="4971" Body="&lt;p&gt;I have some data with around 1000 features. Problem is most of the features are 0. In each row usually around 100 features have values and rest of the features are 0. A sample example data is given below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  f1  f2  f3 ........ f1000  label&#xA; 1  45  0    0            32     1&#xA; 2  23  0    0            0      2&#xA; 3  0   40   0            0      1&#xA; 4  0   0    23           24     3&#xA; 5  78  0    64           0      2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see there are lot of 0 in each row. Which deep neural networks method should I use? I was thinking about RNN or CNN but I am not sure if these are the best choices. I was thinking about some methods which will ignore zero values i.e does not consider the zero values in the calculation. I think RNN can do this. But I am not exactly sure. &lt;/p&gt;&#xA;" OwnerUserId="29345" LastEditorUserId="29169" LastEditDate="2021-02-16T00:44:09.030" LastActivityDate="2021-02-16T00:44:09.030" Title="Deep neural networks for mostly zero valued data" Tags="|deep-learning|neural-network|classification|rnn|convolutional-neural-network|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22024" PostTypeId="1" AcceptedAnswerId="22025" CreationDate="2017-08-07T10:01:41.563" Score="0" ViewCount="978" Body="&lt;p&gt;Logistic Regression generates a binary outcome for a non-binary variable. I need a binary outcome from a binary variable. This is the requirement.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to predict &lt;strong&gt;binary&lt;/strong&gt; A using previous values of A?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to predict &lt;strong&gt;binary&lt;/strong&gt; A,B,C,D values from previous A,B,C,D values?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I know I can gather data and calculate basic probability. I that the most suitable approach? Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="37467" LastActivityDate="2017-08-07T10:12:48.290" Title="Binary predication from binary variables" Tags="|machine-learning|prediction|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22025" PostTypeId="2" ParentId="22024" CreationDate="2017-08-07T10:12:48.290" Score="2" Body="&lt;p&gt;This is really a job for Logistic Regression. Input variables can be categorical/boolean and the prediction can be categorical/boolean as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if your target variable contains multiple categories, you should use multinomial logistic regression or Ordinal Logistic Regression (if there is an order in your categories). It also changes the way you interpret the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The another approach would be to take any nonparametric method, Decision Tree for example. Where input and target can be boolean/categorical/continuous. Note that in case of a categorical variable, it is often recommended to transform it into dummy variables.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-08-07T10:12:48.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22026" PostTypeId="1" CreationDate="2017-08-07T10:13:40.480" Score="4" ViewCount="209" Body="&lt;p&gt;I am trying to predict sales quantity based on attributes of the item.Sales are aggregated by week wise and prediction is also done by week wise.I am having large number records with zero sales quantity than compared to positive sales(for 20 positive sales 250 zero sales records are there).I want to increase my training data by adding small values to sales quantity and combining with total records.&#xA;At present I am using RANDOM FOREST and NEURAL NETWORK .I am not getting any good results&#xA;Please correct me if anything is not considerable&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-08-07T20:14:25.837" Title="Artificially Increasing Training data" Tags="|neural-network|r|predictive-modeling|regression|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22027" PostTypeId="2" ParentId="19546" CreationDate="2017-08-07T11:00:54.383" Score="2" Body="&lt;p&gt;My approach would be using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Moment_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Moments&lt;/a&gt;.&lt;br&gt;&#xA;Using moments is a cool way trowing any arbitrary set of points into space, and it is used for both probability and mass calculations.&lt;br&gt;&#xA;Since you have not stated what are you going to do with this vector representation I guess that a generalized &quot;shape&quot; like representation might work for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using Python, then &lt;a href=&quot;http://scipy.stats.moment&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; can help. &lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-08-07T11:00:54.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22029" PostTypeId="2" ParentId="22026" CreationDate="2017-08-07T11:51:03.680" Score="1" Body="&lt;p&gt;I have tried SMOTE for categorical response variable scenario but never tried on Continuous response variable but I think that should not matter. You can run oversampling or undersampling based on the predictor variables. If you are using R, you can use DMwR library and use the SMOTE function. In Python you got to use imblearn.over_sampling.SMOTE&lt;/p&gt;&#xA;" OwnerUserId="37629" LastActivityDate="2017-08-07T11:51:03.680" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22030" PostTypeId="2" ParentId="19527" CreationDate="2017-08-07T12:01:27.117" Score="2" Body="&lt;p&gt;You could play a bit with the classic &quot;Pima Diabetes&quot; dataset of native american women tested negative/positive for diabetes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In R, there are a few variants (with and without missing data) of this dataset in the MASS package.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a &quot;diabetes.arff&quot; datafile it is also present as a sample dataset in the Weka software, and Weka has a few &quot;Attribute Selection&quot; algorithms built in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know Weka, I suggest that you try them yourself. With a few clicks you can learn a lot about your dataset (without getting a definitive answer, of course).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have added a screenshot illustrating my approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/86nu7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/86nu7.png&quot; alt=&quot;weka screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weka starts with the small &quot;Weka GUI chooser&quot;  window. I have opened the Weka Experimenter (1), then the big window shows up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've imported the file (in the preprocessing&quot; tab (2). I've just loaded it , didn't apply any preprocessing, as the dataset is already clean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in (3) I've chosen the &quot;Select Attributes&quot; Tab, and tried a few of Weka's algorithms. Digits 4,5,6 indicate the buttons I had to click.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(7) Shows the my last run - the &quot;GainRatioAttributeEval&quot; Method, which Weka combined with the &quot;Ranker&quot; method of weighting the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Main panel (8) shows what that algorithm considers the most important attributes/feature, ranked by importance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;=== Attribute selection 10 fold cross-validation (stratified), seed: 1 ===&#xA;&#xA;average merit      average rank  attribute&#xA; 0.104 +- 0.008     1.1 +- 0.3     2 plas   ---  Blood plasma&#xA; 0.092 +- 0.008     1.9 +- 0.3     6 mass   ---  Body mass&#xA; 0.067 +- 0.009     3.1 +- 0.3     8 age    ---  age&#xA; 0.052 +- 0.005     3.9 +- 0.3     1 preg   ---  &#xA; 0.04  +- 0.003     5   +- 0       5 insu   ---  &#xA; 0.02  +- 0.011     6.4 +- 0.66    7 pedi   --- &#xA; 0.016 +- 0.011     7.1 +- 0.7     4 skin   --- &#xA; 0.009 +- 0.009     7.5 +- 0.67    3 pres   --- &#xA;&#xA;# Column Positions in the original datatable:&#xA;%    1. Number of times pregnant&#xA;%    2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test&#xA;%    3. Diastolic blood pressure (mm Hg)&#xA;%    4. Triceps skin fold thickness (mm)&#xA;%    5. 2-Hour serum insulin (mu U/ml)&#xA;%    6. Body mass index (weight in kg/(height in m)^2)&#xA;%    7. Diabetes pedigree function&#xA;%    8. Age (years)&#xA;%    9. Class variable (0 or 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So this attribute selection algorithm thinks the top 3 important attributes (in that order) are #2, #6, #8. These are 2. blood plasma, 6. body mass index, and 8. age. Sounds pretty reasonable to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that this is an idealized example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many missing data does your dataset contain? You didn't tell us. I think your patient data will be very sparse and the dataset will contain lots of empty cells. I think that many attribute selection algorithms work best when there are few missing data values.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2017-08-07T12:07:57.633" LastActivityDate="2017-08-07T12:07:57.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22031" PostTypeId="2" ParentId="22021" CreationDate="2017-08-07T12:32:10.517" Score="1" Body="&lt;p&gt;&lt;strong&gt;spherical dataset&lt;/strong&gt; is basically a form of &lt;strong&gt;non-linear dataset&lt;/strong&gt; in which observational data are modeled by a function which is &lt;strong&gt;a non-linear combination of the model parameters and depends on one or more independent variables&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your dataset has &lt;strong&gt;high variance&lt;/strong&gt; , you need to reduce the number of features and add more dataset. After that you can then use &lt;strong&gt;non-linear methods&lt;/strong&gt; for classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, Non-linear methods typically involves applying some type of transformation to your input dataset. After the transformation, many techniques can then try to use a linear method for classification. &lt;/p&gt;&#xA;" OwnerUserId="35713" LastActivityDate="2017-08-07T12:32:10.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22032" PostTypeId="2" ParentId="22021" CreationDate="2017-08-07T12:50:12.303" Score="5" Body="&lt;p&gt;In this case, a picture is a worth a thousand words. They literally mean data whose distribution on X,Y is roughly a sphere. Different clustering algorithms work better on different distributions. For example, K means does poorly on the arrangement in the first two rows but OK on the last row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/38eBD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/38eBD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-07T12:50:12.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22033" PostTypeId="2" ParentId="11024" CreationDate="2017-08-07T12:50:39.523" Score="27" Body="&lt;p&gt;I was learning this topic too, and these are what I found:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;This type of encoding is called &lt;em&gt;likelihood encoding&lt;/em&gt;, &lt;em&gt;impact coding&lt;/em&gt; or &lt;em&gt;target coding&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you have classification task, you calculate the relative frequency of your target with respect to every category value. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read &lt;a href=&quot;https://www.kaggle.com/c/mercedes-benz-greener-manufacturing/discussion/36136#201638&quot; rel=&quot;nofollow noreferrer&quot;&gt;this comment by raddar here&lt;/a&gt;. The corresponding notebook is &lt;a href=&quot;https://www.kaggle.com/raddar/raddar-extratrees&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The quote:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Another implementation of this encoding &lt;a href=&quot;https://www.kaggle.com/tnarik/likelihood-encoding-of-categorical-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;is here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In R library &lt;strong&gt;vtreat&lt;/strong&gt; they have implementation of impact encoding. See &lt;a href=&quot;http://www.win-vector.com/blog/2014/08/vtreat-designing-a-package-for-variable-treatment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In &lt;a href=&quot;https://tech.yandex.com/catboost/doc/dg/concepts/algorithm-main-stages_cat-to-numberic-docpage/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CatBoost library&lt;/a&gt; they have a lot of options for categorical variable encoding including target encoding. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is no such encoding in sklearn yet.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;UPDATE: There is a nice package for sklearn models and pipelines! &lt;a href=&quot;https://github.com/scikit-learn-contrib/category_encoders&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn-contrib/category_encoders&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37630" LastEditorUserId="37630" LastEditDate="2020-04-27T20:41:13.493" LastActivityDate="2020-04-27T20:41:13.493" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="22036" PostTypeId="1" AcceptedAnswerId="22053" CreationDate="2017-08-07T14:46:02.643" Score="7" ViewCount="15134" Body="&lt;p&gt;I understand that the loss metric can be used as linear, or log, or other things. This is documented at &lt;a href=&quot;http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters&quot; rel=&quot;noreferrer&quot;&gt;http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to understand how LightGBM works on variables with different scale. In other words, is it necessary for me to harmonize scale when running LightGBM? (I am used to linear regression where you need to get into linear scale.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had inputs x1, x2, x3, output y and some noise N then here are a few examples of different scales.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$y = x1 + x2 + x3 + N $&lt;/li&gt;&#xA;&lt;li&gt;$y = exp(x1 + x2 + x3 + N) $&lt;/li&gt;&#xA;&lt;li&gt;$y = log(x1 + x2 + x3 + N) $&lt;/li&gt;&#xA;&lt;li&gt;$y = sqrt(x1 + x2 + x3 + N) $&lt;/li&gt;&#xA;&lt;li&gt;$y = log(x1 * x2 * x3 * N) $&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2023-10-31T08:55:40.707" Title="How does LightGBM deal with value scale?" Tags="|regression|xgboost|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22037" PostTypeId="1" CreationDate="2017-08-07T14:58:57.693" Score="10" ViewCount="1788" Body="&lt;p&gt;I now read a book titled &lt;em&gt;&quot;Hands-on Machine Learning with Scikit-Learn and TensorFlow&quot;&lt;/em&gt; and on the chapter 11, it has the following description on the explanation of ELU (Exponential ReLU).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Third, the function is smooth everywhere, including around z = 0, which helps speed up Gradient Descent, since it does not bounce as much left and right of z = 0.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/h7BrH.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h7BrH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;z&lt;/code&gt; means the x-axis on the graph above. I understand the derivative is smooth since the &lt;code&gt;z &amp;lt; 0&lt;/code&gt; line has a curve and in that realm the derivative is no longer equal to &lt;code&gt;0&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, why is it the case that if the function is &quot;smooth everywhere, including around z=0&quot;, it speeds up Gradient Descent?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-12-03T13:35:32.860" Title="Why does it speed up gradient descent if the function is smooth?" Tags="|deep-learning|gradient-descent|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22038" PostTypeId="2" ParentId="22014" CreationDate="2017-08-07T15:18:23.267" Score="1" Body="&lt;p&gt;If this is your first project start with something simple - e.g. linear regression, &lt;code&gt;lm(sales ~ data)&lt;/code&gt;. Once you get it up and running you can try something more sophisticated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest avoiding neural networks in sales prediction - neural networks are sexy, but require huge amounts of data to train properly. This is often not feasible in sales prediction environment.&lt;/p&gt;&#xA;" OwnerUserId="33646" LastActivityDate="2017-08-07T15:18:23.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22042" PostTypeId="2" ParentId="22037" CreationDate="2017-08-07T17:17:18.627" Score="1" Body="&lt;p&gt;My guess it is due to the derivative, as ReLU has a discontinuous derivative at 0. Hence if you use the definition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;f' \approx \frac{f(x+\epsilon) -f(x-\epsilon)}{2 \epsilon}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and $x$ is very close to 0, you will get many of those 'jumps'.&lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="26562" LastEditDate="2017-11-06T00:13:47.380" LastActivityDate="2017-11-06T00:13:47.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22043" PostTypeId="6" CreationDate="2017-08-07T19:54:11.533" Score="0" Body="&lt;p&gt;Hello, everybody.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is my honor to participate in DS SE moderator election.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be my mission as the moderator:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;making DS SE interesting and fun place to be.&lt;/li&gt;&#xA;&lt;li&gt;attracting different people from all walks of life to this site.&lt;/li&gt;&#xA;&lt;li&gt;making DS SE the place you want to visit every morning, every afternoon, and every evening.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you like this, vote for me!&lt;/p&gt;&#xA;" OwnerUserId="31513" LastEditorUserId="31513" LastEditDate="2017-08-07T19:54:11.533" LastActivityDate="2017-08-07T19:54:11.533" CommentCount="2" CommunityOwnedDate="2017-08-07T19:54:11.533" ContentLicense="CC BY-SA 3.0" />
  <row Id="22044" PostTypeId="2" ParentId="22026" CreationDate="2017-08-07T20:14:25.837" Score="1" Body="&lt;p&gt;Avoid neural networks for weekly sales data. There simply is not enough data points to make it work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The technique works great with image or video data (it can recognize numbers, or cat videos) but you need something in the neighborhood of 200,000 data points to train your network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine the cost and complexity of getting 200,000 &lt;em&gt;weekly&lt;/em&gt; data points (that is something between 3,000 and 4,000 years of collecting).&lt;/p&gt;&#xA;" OwnerUserId="33646" LastActivityDate="2017-08-07T20:14:25.837" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22045" PostTypeId="1" CreationDate="2017-08-07T20:40:34.110" Score="2" ViewCount="1241" Body="&lt;p&gt;I have a column as month in my excel data set, when i load the data in the file, orange changes it to some numerical values.. Help! I want the Month as it is. Tried changing the value type as short or long date in excel too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The orange loads Dec-16 as 42055.00 something.&lt;/p&gt;&#xA;" OwnerUserId="37646" LastActivityDate="2019-08-24T13:01:45.517" Title="Orange Changes the data type" Tags="|data-mining|predictive-modeling|dataset|orange|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22047" PostTypeId="1" CreationDate="2017-08-07T21:04:42.987" Score="5" ViewCount="9182" Body="&lt;p&gt;I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values.&lt;/p&gt;&#xA;&lt;p&gt;I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.&lt;/p&gt;&#xA;&lt;p&gt;Is this the correct way to forecast time series with LSTMs?&lt;/p&gt;&#xA;&lt;p&gt;How can this behaviour be explained?&lt;/p&gt;&#xA;&lt;p&gt;Will this be the case even for a time series with great seasonality?&lt;/p&gt;&#xA;&lt;p&gt;Is this behaviour expected even for a very large time series data?&lt;/p&gt;&#xA;&lt;p&gt;Should I rather try to train the network with Y matrix having 31st day to 60th day values for the same X?&lt;/p&gt;&#xA;&lt;p&gt;My train data looks something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;array([[-0.35811423, -0.22393472, -0.39437897, ..., -0.36718042, -0.37080689, -0.35267452], [-0.22393472, -0.39437897, -0.13327289, ..., -0.37080689, -0.35267452, -0.2030825 ], [-0.39437897, -0.13327289, -0.1532185 , ..., -0.35267452, -0.2030825 , -0.25294651]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Architecture: Input LSTM layer (20 neurons) 1 hidden lstm layer 20 neurons 1 output dense layer, batch size as 1. Stateful lstm, I reset model states after each epoch.&lt;/p&gt;&#xA;" OwnerUserId="37648" LastEditorUserId="98307" LastEditDate="2020-08-13T11:24:46.183" LastActivityDate="2020-08-13T11:24:46.183" Title="Time series forecasting with RNN(stateful LSTM) produces constant values" Tags="|deep-learning|time-series|tensorflow|rnn|lstm|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22048" PostTypeId="2" ParentId="21902" CreationDate="2017-08-07T21:14:24.047" Score="2" Body="&lt;p&gt;To avoid reassignment, use (inplace = True)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.drop_duplicates(inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is same as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.drop_duplicates()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30360" LastActivityDate="2017-08-07T21:14:24.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22049" PostTypeId="2" ParentId="21963" CreationDate="2017-08-08T00:22:10.043" Score="8" Body="&lt;p&gt;From Emre's comment above, Section 4.4 of &lt;a href=&quot;https://people.cs.uchicago.edu/~risi/papers/KondorThesis.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Group theoretical methods in machine learning&lt;/a&gt; by Risi Kondor has detailed information and proofs about creating kernel methods that inherently have symmetries. I will summarize it in a hopefully intuitive way (I am a physicist not a mathematician!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most ML algorithms have a matrix multiplication like,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;s_i &amp;amp;= \sum_j W_{ij}~x_j \\&#xA;  &amp;amp;= \sum_j W_{ij}~(\vec{e}_j \cdot \vec{x})&#xA;\end{align}&lt;/span&gt;&#xA;with  &lt;span class=&quot;math-container&quot;&gt;$ \vec{x} $&lt;/span&gt; being the input and &lt;span class=&quot;math-container&quot;&gt;$ W_{ij} $&lt;/span&gt; being the weights we wish to train.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Kernel Method&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Enter the realm of &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_method&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernel methods&lt;/a&gt; and let the algorithm handle input via,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;s_i &amp;amp;= \sum_j W_{ij}~k(e_j,~x)&#xA;\end{align}&lt;/span&gt;&#xA;where now we generalize to &lt;span class=&quot;math-container&quot;&gt;$ x, e_j \in \mathcal{X} $&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a group &lt;span class=&quot;math-container&quot;&gt;$ G $&lt;/span&gt; that acts on &lt;span class=&quot;math-container&quot;&gt;$ \mathcal{X} $&lt;/span&gt; via &lt;span class=&quot;math-container&quot;&gt;$ x \rightarrow T_g(x) $&lt;/span&gt; for &lt;span class=&quot;math-container&quot;&gt;$ g \in G $&lt;/span&gt;. A simple way to make our algorithm invariant under this group is to make a kernel,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;k^G(x, y) &amp;amp;= \frac{1}{|G|} \sum_{g \in G} k(x, T_g(y))&#xA;\end{align}&lt;/span&gt;&#xA;with &lt;span class=&quot;math-container&quot;&gt;$ k(x, y) = k(T_g(x), T_g(y)) $&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;k^G(x, T_h(y)) &amp;amp;= \frac{1}{|G|} \sum_{g \in G} k(x, T_{gh}(y)) \\&#xA;  &amp;amp;= \frac{1}{|G|} \sum_{g \in G} k(x, T_{g}(y)) \\&#xA;  &amp;amp;= \frac{1}{|G|} \sum_{g \in G} k(T_{g}(x), y)&#xA;\end{align}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;span class=&quot;math-container&quot;&gt;$ k(x, y) = x \cdot y $&lt;/span&gt; which works for all unitary representations,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;k^G(x, T_h(y)) &amp;amp;= \left[ \frac{1}{|G|} \sum_{g \in G} T_{g}(x) \right] \cdot y&#xA;\end{align}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which offers a transformation matrix that can symmeterize the input into the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;SO(2) Example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Actually just the group that maps to &lt;span class=&quot;math-container&quot;&gt;$ \frac{\pi}{2} $&lt;/span&gt; rotations for simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us run linear regression on data &lt;span class=&quot;math-container&quot;&gt;$ (\vec{x}_i, y_i) \in \mathbb{R}^2 \times \mathbb{R} $&lt;/span&gt; where we expect a rotational symmetry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our optimization problem becomes,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;\min_{W_{j}} &amp;amp;\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\&#xA;\tilde{y}_i &amp;amp;= \sum_j W_{j} k_G(e_j, x_i) + b_i&#xA;\end{align}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The kernel &lt;span class=&quot;math-container&quot;&gt;$ k(x, y) = \| x - y \|^2 $&lt;/span&gt; satisfies &lt;span class=&quot;math-container&quot;&gt;$ k(x, y) = k(T_g(x), T_g(y)) $&lt;/span&gt;. You could also use &lt;span class=&quot;math-container&quot;&gt;$ k(x, y) = x \cdot y $&lt;/span&gt; and a variety of kernels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;k_G(e_j, x_i) &amp;amp;= \frac{1}{4} \sum_{n=1}^4 \| R(n\pi/2)~\vec{e}_j - \vec{x}_i \|^2 \\&#xA;  &amp;amp;= \frac{1}{4} \sum_{n=1}^4 ( \cos(n\pi/2) - \vec{x}_{i1} )^2 + ( \sin(n\pi/2) - \vec{x}_{i2} )^2 \\&#xA;  &amp;amp;= \frac{1}{4} \left[ 2 \vec{x}_{i1}^2 + 2 \vec{x}_{i2}^2 + (1 - \vec{x}_{i1} )^2 + (1 - \vec{x}_{i2} )^2 + (1 + \vec{x}_{i1} )^2 + (1 + \vec{x}_{i2} )^2 \right] \\&#xA;  &amp;amp;= \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1&#xA;\end{align}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that we needn't sum over &lt;span class=&quot;math-container&quot;&gt;$ j $&lt;/span&gt; because it is the same for both. So our problem becomes,&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align}&#xA;\min_{W} &amp;amp;\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\&#xA;\tilde{y}_i &amp;amp;= W \left[ \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1 \right] + b_i&#xA;\end{align}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which yields the expected spherical symmetry!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Tic-Tac-Toe&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Example code can be seen &lt;a href=&quot;https://github.com/aidan-plenert-macdonald/tictactoe/commit/b243bbb607201932c2b1bbc7ae61bac713bfab05&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It shows how we can create a matrix that encodes the symmetry and use it. &lt;strong&gt;Note&lt;/strong&gt; that this is really bad when I actually run it! Working with other kernels at the moment.&lt;/p&gt;&#xA;" OwnerDisplayName="user32280" LastEditorDisplayName="user32280" LastEditDate="2019-02-19T18:17:47.933" LastActivityDate="2019-02-19T18:17:47.933" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="22050" PostTypeId="1" AcceptedAnswerId="22076" CreationDate="2017-08-08T01:05:06.917" Score="1" ViewCount="398" Body="&lt;p&gt;I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which parameter is the best. This appears to happen based on how the random number generator splits the data into train/test sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;CV 5-fold&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Split of 75%&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Fixing the random seed does help (or using the same test set), but it concerns me that I get such variations based on what samples are in the test set. It seems alarming that the 'best parameter' is so dependent on a particular shuffle of the data and I worry how it translates to real world use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are people's approach to situations like this? I was thinking I could just repeat each test multiple times and take the average, but that has very large computation costs and seems very inefficient. &lt;/p&gt;&#xA;" OwnerUserId="32905" LastActivityDate="2017-08-08T16:03:53.593" Title="Grid Search and High Variance" Tags="|cross-validation|hyperparameter|search|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22051" PostTypeId="1" CreationDate="2017-08-08T06:09:34.200" Score="3" ViewCount="938" Body="&lt;p&gt;By now i thought that fuzzy clustering can be applied to any kind of data-sets but now i have heard that it can be applied only to specific data-sets that involve the concept of probability, is that true?&lt;/p&gt;&#xA;" OwnerUserId="37658" LastActivityDate="2017-08-09T09:56:00.617" Title="datasets for fuzzy clustering" Tags="|clustering|fuzzy-logic|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22052" PostTypeId="2" ParentId="22050" CreationDate="2017-08-08T07:27:59.660" Score="1" Body="&lt;p&gt;I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.&lt;/p&gt;&#xA;" OwnerUserId="13833" LastActivityDate="2017-08-08T07:27:59.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22053" PostTypeId="2" ParentId="22036" CreationDate="2017-08-08T07:36:05.473" Score="9" Body="&lt;p&gt;Generally, in tree-based models the scale of the features does not matter. This is because at each tree level, the score of a possible split will be equal whether the respective feature has been scaled or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of it like here: We're dealing with a binary classification problem and the feature we're splitting takes values from 0 to 1000. If you split it on 300, the samples &amp;lt;300 belong 90% to one category while those &gt;300 belong 30% to one category. Now imaging this feature is scaled between 0 and 1. Again, if you split on 0.3, the sample &amp;lt;0.3 belong 90% to one category while those &gt;0.3 belong 30% to one category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you've changed the splitting point but the actual distribution of the samples remains the same regarding the target variable. &lt;/p&gt;&#xA;" OwnerUserId="13833" LastActivityDate="2017-08-08T07:36:05.473" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22054" PostTypeId="2" ParentId="22023" CreationDate="2017-08-08T07:46:47.547" Score="2" Body="&lt;p&gt;Why using deep learning at all? It seems you already have a big set of features. So maybe try using Random Forests? They are pretty good at handling outliers (in you ur case, useless 0 values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN are good for dealing with 2 dimensional input data, like pictures. &#xA;RNN you should use when dealing with timeseries. In my opinion deep learning should be thus used, when dealing with abstract data.&lt;/p&gt;&#xA;" OwnerUserId="37330" LastActivityDate="2017-08-08T07:46:47.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22055" PostTypeId="1" AcceptedAnswerId="22057" CreationDate="2017-08-08T08:06:09.933" Score="4" ViewCount="786" Body="&lt;p&gt;Say I've built a (completely unrealistic) classification model in Keras that gives me 1.00 accuracy.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And next, I would like to use my model on some new, unseen data, and use &lt;code&gt;model.predict_proba&lt;/code&gt; to get a probability that the observation belongs to class &quot;A&quot;. Say this returns to me a 0.75.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I interpreting this correctly in English:  &quot;100 percent of the time, the model is confident that this new observation is 75 percent likely to be class A&quot; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is correct, then let's consider if my model was not totally perfect, like in real life, and instead it gave me a 0.40 accuracy. Say my &lt;code&gt;predict_proba&lt;/code&gt; is still 0.75.  Then, is this correct:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;40 percent of the time, the model is confident that this new observation is 75 percent likely to be class A.&quot; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so...this makes it seem like &lt;code&gt;predict_proba()&lt;/code&gt; is not tell a complete story.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could mislead someone (say a journalist...or a judge, whoever) by saying, &quot;There's a 75 percent chance this unseen observation belongs to class A&quot;...and that might sound great, if I fail to reveal that this statment was based on a model that had a low accuracy like  0.40.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I stating this correctly, and does my apprehension have validity?&lt;/p&gt;&#xA;" OwnerUserId="15533" LastActivityDate="2021-07-17T06:19:22.893" Title="What does a predicted probability really mean, without considering the accuracy of the underlying model?" Tags="|deep-learning|keras|prediction|probability|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22057" PostTypeId="2" ParentId="22055" CreationDate="2017-08-08T08:50:30.680" Score="4" Body="&lt;p&gt;Accuracy is measured in classification model by comparing the predicted labels to the actual known labels.&lt;br /&gt;&#xA;The predicted labels are a function of both the predicted probabilities for each class and a predefined threshold(binary classification usually is 0.5)&lt;br /&gt;&#xA;So if sample A got predict_proba of {0: 0.2, 1: 0.8} it will be labeled as 1(since 0.8 &amp;gt; 0.5).&lt;br /&gt;&#xA;Accuracy is measure of classification correctness and predict_proba is a direct result of the model underlying function.&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="27983" LastEditDate="2021-07-17T06:19:22.893" LastActivityDate="2021-07-17T06:19:22.893" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22058" PostTypeId="1" CreationDate="2017-08-08T09:02:12.083" Score="15" ViewCount="20535" Body="&lt;p&gt;I'm working on &lt;a href=&quot;https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/&quot; rel=&quot;noreferrer&quot;&gt;Seq2Seq model&lt;/a&gt; using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is clear that GPUs are far much better in parallelization than CPUs. At the moment, I only have CPUs to work with. I could access 16 CPUs(2 Threads per core X 4 cores per socket X 2 sockets)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the doc of multi-core &lt;a href=&quot;http://deeplearning.net/software/theano/tutorial/multi_cores.html&quot; rel=&quot;noreferrer&quot;&gt;support&lt;/a&gt; in Theano, I managed to use all the four cores of a single socket. So, basically the CPU is at 400% usage with 4CPUs used and the remaining 12 CPUs remain unused. How do I make use of them too. Tensorflow could also be used instead of Theano background, if it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aFGsp.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aFGsp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16024" LastEditorUserId="2932" LastEditDate="2017-09-05T14:09:27.193" LastActivityDate="2018-05-03T22:31:04.120" Title="Make Keras run on multi-machine multi-core cpu system" Tags="|tensorflow|keras|theano|parallel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22059" PostTypeId="1" CreationDate="2017-08-08T09:45:48.753" Score="1" ViewCount="1180" Body="&lt;p&gt;If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking of starting with a model that converts the job postings into a vector space using TF-IDF and then just calculate the matrix cosine similarity. Can I improve on this?&lt;/p&gt;&#xA;" OwnerUserId="37664" LastActivityDate="2017-08-08T14:00:43.027" Title="What techniques should I use to compare the similarity between a bunch of texts?" Tags="|nlp|similarity|cosine-distance|similar-documents|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22060" PostTypeId="1" CreationDate="2017-08-08T09:51:51.520" Score="2" ViewCount="59" Body="&lt;p&gt;I want to implement Distributed operating system mutual exclusion using Machine learning algorithms but I am not getting dataset anywhere to perform.&#xA;Is there anyway i can get the required dataset?&#xA;I already tried UCI and amazon public dataset site.&lt;/p&gt;&#xA;" OwnerUserId="37665" LastActivityDate="2017-08-08T09:51:51.520" Title="Machine Learning over distributed operating system" Tags="|machine-learning|dataset|" AnswerCount="0" CommentCount="1" ClosedDate="2020-12-10T08:58:43.787" ContentLicense="CC BY-SA 3.0" />
  <row Id="22061" PostTypeId="2" ParentId="22059" CreationDate="2017-08-08T10:04:46.557" Score="0" Body="&lt;p&gt;Start with &lt;code&gt;named entity recognition&lt;/code&gt;, it gives an idea which part of posting is requirements and what part is description and then you could try just with bag of words. As the topic structure and vocabulary is standard and no weights are needed for word importance, I dont find a reason for TF-IDF to be used here.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-08-08T10:04:46.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22062" PostTypeId="1" AcceptedAnswerId="33131" CreationDate="2017-08-08T10:43:32.227" Score="6" ViewCount="1590" Body="&lt;p&gt;I am using the seaborn violin plot feature however the size of the violins aren't what I would expect them to be. Does anyone know whether this is something I am doing wrong, or as I suspect whether this is a bug in seaborn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code I am using is the following;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; sns.violinplot(x = &quot;Issue Type&quot;, y = &quot;issue_adj_risk_score&quot;, data=subset, scale=&quot;count&quot;, cut =0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the chart I can see is like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Om9ND.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Om9ND.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand it, becuase I have used the &lt;code&gt;scale =&quot;count&quot;&lt;/code&gt; parameter, the sizes of each of these violins should be proportional to an overall count of the number of things in each area. In 'category 1' however (pink) I have a total of 936 with an issue_adj_risk_score of 1 whilst in category 2 (orange) I have a total of 1212 with an issue_adj_risk_score of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am confused however as the pink is larger than the orange! :-(&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know why this might be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;EDIT&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now seeing further problems where the range of a particular category is inconsistent. I have a dataset and used violinplot to plot the below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ALMl7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ALMl7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we can see that a certain category (which I have highlighted in red) has a range from 1-4. This surprised me so I looked into the underlying data and it doesn't contain any values at 4. I then took a subset that contained this category only and plotted a violinplot with the same data and got this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tQcwj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tQcwj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is what I would have expected to see in the initial chart!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To investigate this further I plotted a boxplot based on the original data once again, and got the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6OZn9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6OZn9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is consistent with the first chart but doesn't reflect the actual data. How can all of these 3 charts come from the same data!!??&lt;/p&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="37059" LastEditDate="2017-08-16T10:54:19.070" LastActivityDate="2020-04-02T18:16:55.980" Title="Is this a bug in Seaborn?" Tags="|python|visualization|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22063" PostTypeId="2" ParentId="22023" CreationDate="2017-08-08T10:55:45.143" Score="2" Body="&lt;p&gt;I don't understand that why have you decided to use CNN or RNN? &#xA;This problem can be solved by ANN if you need to use Deep Learning. &#xA;CNNs are good at Image Recognition tasks. &#xA;RNNs are good at Time series analysis. &#xA;I'd suggest you to use a couple of Machine learning algorithms. They might be enough for this. You can also use Gradient Boosting methods. &#xA;Hope this helped. &lt;/p&gt;&#xA;" OwnerUserId="37666" LastActivityDate="2017-08-08T10:55:45.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22064" PostTypeId="2" ParentId="22023" CreationDate="2017-08-08T12:42:23.930" Score="4" Body="&lt;p&gt;There's this misunderstanding that deep learning is generally suitable if you have loads of data, which, judging from your comment, is your case. This is generally an inaccurate belief.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep learning (including CNN and RNN), are complex models with thousands of parameters that are able to model complex relationships. Such relationships are generally &quot;hidden&quot; in vast amounts of data, but this is not always the case. You may have at hand data that are generated from a simple distribution, and as such it will not need a complex model to approximate, even if your sample size is huge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a fabricated example: Let's say that you have the function &lt;code&gt;y=a*x1 + b*x2 + c*x3 + d*x4&lt;/code&gt;. This function entails linear relationships between all independent variables and the dependent variable, y. You could sample this function a million times and still you would have at hand data that could be approximated through linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to your case: to identify what kind of algorithm you would need, you first need to look at your data, perhaps through performing a statistical analysis. Then I would suggest to start simple. Try logistic regression, for starters. Is the model satisfactory in validation? If not, try perhaps decision trees. Or a shallow neural net. Always validate (you have loads of data so validation should be easy!). My (admittedly wild) guess is that your classification problem could be addressed with much simpler algorithms than DNN. But YMMV, of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FWIW, &lt;a href=&quot;https://stats.stackexchange.com/questions/215308/linear-regression-on-large-sparse-feature-set&quot;&gt;here's another answer similar to your case&lt;/a&gt;. And &lt;a href=&quot;https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix&quot;&gt;another one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2017-08-08T12:42:23.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22065" PostTypeId="2" ParentId="18750" CreationDate="2017-08-08T13:46:43.493" Score="1" Body="&lt;p&gt;I think you're in a pretty difficult state. I think the class imbalance techniques you're using are fine. I would advise you to try framing your problem in a &lt;strong&gt;hierarchical classification&lt;/strong&gt; manner.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Level 1 Classifier&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;classifier_A between class 0 &amp;amp; class (1 + 2 + 3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Note:&lt;/strong&gt; Apply all sort of class imbalance methods to get good accuracy here&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Level 2 Classifier&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;classifier_B between class 1, class 2, class 3&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Final pseudo model&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;if predict_classifier_A(x_test) == &quot;class 0&quot;:&#xA;   result = &quot;class 0&quot;&#xA; else:&#xA;   result = predict_classifier_B(x_test)&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I'm not sure will it work or not but its worth a try. Let us know if it worked better.&lt;/p&gt;&#xA;" OwnerUserId="2433" LastActivityDate="2017-08-08T13:46:43.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22066" PostTypeId="2" ParentId="18750" CreationDate="2017-08-08T13:48:43.207" Score="0" Body="&lt;p&gt;Although, sampling techniques like up sampling, down sampling exist to solve this imbalance class issue. Smote sampling is also available in several packages. Another useful technique is to assign weights to the different observations (according to the frequencies in data) so that minor classes are not modelled as noise(usually minor classes are given more weights to instruct the model the the minor classes are valuable and not outliers or noise). Although the overall accuracy would be great for a classifier but the misclassification error is very high for the minor class. CARET package in R helps you prepare a basic framework to handle this class-imbalance problem. with CreateDataPartition function you can generate stratified datasets. Kindly explore this package in R documentation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend to divide the four classes into chunks of equal sizes and make one dataset that would have equal representation of classes in one dataset. then the different datasets would have re-representation of minor classes but the dataset would have stratified sampling. These dataset can then be divided into training set and test set to do cross-validation. we can get the generalisation error by averaging out from the (say 10 fold). cross-validation. For second step for Hyperparameter tunning, internal cross fold validation can be performed to improve the prediction performance of the model. &lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="31599" LastEditDate="2017-08-09T08:01:15.087" LastActivityDate="2017-08-09T08:01:15.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22068" PostTypeId="2" ParentId="22059" CreationDate="2017-08-08T14:00:43.027" Score="1" Body="&lt;p&gt;As of now, I can think of two ways to formulate this problem:&lt;/p&gt;&#xA;&lt;h1&gt;1. Search problem&lt;/h1&gt;&#xA;&lt;p&gt;Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build capabilities like &lt;a href=&quot;https://github.com/DiceTechJobs/ConceptualSearch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Semantic search using Word2Vec&lt;/a&gt; models, etc.&lt;/p&gt;&#xA;&lt;p&gt;Now write a query engine which takes a resume and queries this Search engine. It will be blazing fast since job listing will be all indexed.&lt;/p&gt;&#xA;&lt;h1&gt;2. Similarity problem&lt;/h1&gt;&#xA;&lt;p&gt;I would have created hybrid similarity function. For example:&lt;/p&gt;&#xA;&lt;p&gt;a) How many top key words matched between resume and job listing&lt;/p&gt;&#xA;&lt;p&gt;b) Similarity of resume and job listing using &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2Vec&lt;/a&gt; (Pre compute vector for job listings)&lt;/p&gt;&#xA;&lt;p&gt;c) Using algorithms like &lt;a href=&quot;http://www.mit.edu/%7Eandoni/LSH/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Locality Sensitive Hashing&lt;/a&gt; to reduce the lookup space&lt;/p&gt;&#xA;&lt;p&gt;This approach will be slow but might yield a good result.&lt;/p&gt;&#xA;" OwnerUserId="2433" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-08-08T14:00:43.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22069" PostTypeId="2" ParentId="22050" CreationDate="2017-08-08T14:20:23.643" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Averaging might help. You can optimize your hyper parameter tuning&#xA;time using &lt;a href=&quot;http://fastml.com/optimizing-hyperparams-with-hyperopt/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Hyper-parameter tuning&lt;/a&gt; approach.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try to reduce the number of features or try algorithms like Random&#xA;Forest with other standard techniques like k-fold repeated CV.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try creating new features which might help you differentiate classes&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="2433" LastActivityDate="2017-08-08T14:20:23.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22070" PostTypeId="1" AcceptedAnswerId="22071" CreationDate="2017-08-08T14:32:30.537" Score="1" ViewCount="224" Body="&lt;p&gt;What I know so far in DCGAN is that a discriminator is trained using the labeled data (so maybe that occurs before training the generative model). Also, I know that there is race between the generator and the discriminator, so maybe training occur online. So I have some concerns here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How many outputs the discriminator should have (Is it one output that describes the probability, ex: P(x))?&lt;/li&gt;&#xA;&lt;li&gt;How do we chose its output when feeding fake data vs. real data?&lt;/li&gt;&#xA;&lt;li&gt;Is the discriminator trained before using it with the DCGAN or the training is done online (It is mentioned in the Original Paper: Generative Adversarial Nets &lt;a href=&quot;https://arxiv.org/pdf/1406.2661.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1406.2661.pdf&lt;/a&gt;, that the whole network is trained using the back propagation), hence I think its online?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help is much appreciated!!&lt;/p&gt;&#xA;" OwnerUserId="32882" LastEditorUserId="32882" LastEditDate="2017-08-08T14:41:06.733" LastActivityDate="2017-08-08T14:58:50.353" Title="Training the Discriminative Model in Generative Adversarial Neural Network" Tags="|unsupervised-learning|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22071" PostTypeId="2" ParentId="22070" CreationDate="2017-08-08T14:46:58.830" Score="1" Body="&lt;p&gt;In normal GANs, there are no labels, the training is completely unsupervised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The role of the discriminator is to tell apart samples generated by the generator from those taken from the training dataset. The training dataset is just a bunch of images. The discriminator is trained to output 0 for data generated by the generator (i.e. fake data) and 1 for real data (so &lt;strong&gt;the discriminator has a single output&lt;/strong&gt;). This should answer points 1 and 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training of discriminator and generator takes place alternatively in a loop: first we train the discriminator, then the generator, then the discriminator again, etc. It is possible (and common) to train the discriminator a few times per each time we train the generator. This should answer point 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible to use labels, but not in the way you were suggesting. When labels are used, we have Conditional GANs (&lt;a href=&quot;https://arxiv.org/abs/1411.1784&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1411.1784&lt;/a&gt;). In this case, the label is supplied &lt;strong&gt;as input&lt;/strong&gt; to both the generator and the discriminator. The generator has to generate data that is associated to the supplied label. The discriminator has to tell apart fake data from real data, given the label.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-08-08T14:58:50.353" LastActivityDate="2017-08-08T14:58:50.353" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22072" PostTypeId="1" AcceptedAnswerId="22074" CreationDate="2017-08-08T15:00:47.583" Score="12" ViewCount="7134" Body="&lt;p&gt;I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure &lt;code&gt;N/A&lt;/code&gt;s are treated as a special value rather than a number between -1 and 1, and tagging your categorical values properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past I've done plenty of linear regression analysis. So feature engineering mainly concerned with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Getting features into the correct scale using log, exponent, power transformations&lt;/li&gt;&#xA;&lt;li&gt;Multiplying features: if you have height and width, multiply to make area&lt;/li&gt;&#xA;&lt;li&gt;Selecting features: remove features based on P value&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But, for LightGBM (and Random Forest) it seems like the scale of the features doesn't matter because orderable items are ordered and then randomly bisected. Interactions of features don't matter because one of the weak classifiers should find it if it is important. And feature selection isn't important because if the effect is weak then those classifiers will be attenuated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, assuming you can't find more data to bring in, what feature engineering should be done with decision tree models?&lt;/p&gt;&#xA;" OwnerUserId="35930" LastEditorUserId="23305" LastEditDate="2017-08-08T16:42:20.580" LastActivityDate="2017-08-08T16:42:20.580" Title="What feature engineering is necessary with tree based algorithms?" Tags="|feature-selection|decision-trees|xgboost|gradient-descent|feature-engineering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22073" PostTypeId="1" CreationDate="2017-08-08T15:15:32.167" Score="5" ViewCount="4218" Body="&lt;p&gt;I now read a book titled &quot;&lt;em&gt;Hands-On Machine Learning with Scikit-Learn and TensorFlow&lt;/em&gt;&quot; and on the chapter 11, the author writes the following explanation on batch normalization:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activation function or with the ReLU activation function, since the next layer’s weights can take care of scaling, but for any other activation function, you should add &quot;scale&quot;: True to bn_params.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;batch_norm&lt;/code&gt; function is from TensorFlow, FYI. The author explains that the γ parameter should not be set on ReLU activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't understand why on ReLU, the next layer's weights can take care of scaling... I understand that the next layer takes input from the ReLU output, which is &lt;code&gt;max(0, x)&lt;/code&gt;. Why does it take care of scaling and thus no need to set γ parameter on ReLU?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastEditorUserId="836" LastEditDate="2017-08-22T18:07:10.460" LastActivityDate="2017-08-22T18:07:10.460" Title="Why is scale parameter on batch normalization not needed on ReLU?" Tags="|deep-learning|batch-normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22074" PostTypeId="2" ParentId="22072" CreationDate="2017-08-08T15:22:21.713" Score="12" Body="&lt;p&gt;Feature engineering that I would consider essential for even tree based algorithms are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your model needs to know that something happens on the third Monday of every month, it will be nearly impossible to determine this from timestamps.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On a similar vein, creating new features from the data you have available can drastically improve your predictive power. This is where domain knowledge is extremely important - if you know of, or think you know of a relationship then you can include variables that describe that relationship. This is because tree based methods can only create splits that are horizontal or vertical (i.e. orthogonal to your data).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Dimension reduction is typically performed by either feature selection or feature transformation. Reducing the dimension through feature selection will likely not help much with the models you mention, but an algorithm may or may not benefit from feature transformation (for example principal component analysis) depending on how much information is lost in the process. The only way to know for sure is to explore whether feature transformation provides better performance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30964" LastActivityDate="2017-08-08T15:22:21.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22075" PostTypeId="1" AcceptedAnswerId="22077" CreationDate="2017-08-08T15:24:06.500" Score="3" ViewCount="5203" Body="&lt;p&gt;Since the aim of a Discriminator is to output 1 for real data and 0 for fake data, hence, the aim is to increase the likelihood of true data vs. fake one. In addition, since maximizing the likelihood is equivalent to minimizing the log-likelihood, why are we updating the discriminator by ascending its stochastic gradient as mentioned in Algorithm 1 in &lt;a href=&quot;https://arxiv.org/pdf/1406.2661.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1406.2661.pdf&lt;/a&gt;. Shouldn't we update the discriminator by descending its stochastic gradient?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is much appreciated!&lt;/p&gt;&#xA;" OwnerUserId="32882" LastActivityDate="2018-03-31T15:41:23.930" Title="Loss function in GAN" Tags="|unsupervised-learning|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22076" PostTypeId="2" ParentId="22050" CreationDate="2017-08-08T15:34:27.043" Score="1" Body="&lt;p&gt;As mentioned it can be a good idea to repeat CV a few times and average the results to obtain a more reliable estimate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find many parameter constellations that are within one standard deviation (or in that neighborhood) of the best performing model, it can make sense to choose a model with a slightly worse CV performance, but with a simpler decision boundary (e.g. shallower decision trees, smaller gammas in an RBF-kernel SVM, stronger regularization parameters etc.) - something that was suggested for example &lt;a href=&quot;http://projecteuclid.org/euclid.aos/1032181158&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-08-08T16:03:53.593" LastActivityDate="2017-08-08T16:03:53.593" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22077" PostTypeId="2" ParentId="22075" CreationDate="2017-08-08T16:02:36.340" Score="4" Body="&lt;p&gt;In algorithm 1 of the original GAN article (&lt;a href=&quot;https://arxiv.org/pdf/1406.2661.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1406.2661.pdf&lt;/a&gt;), the discriminator is said to be updated by &quot;ascending its stochastic gradient&quot;. This is referring to equation 1:&#xA;$$&#xA;\min_G \max_D V(D, G)=&#xA;\mathbb{E}_{x\sim p_{data}(x)}[\log D(x)]&#xA;+ \mathbb{E}_{z\sim p_z(z)}[\log(1 - D(G(z)))]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we want to minimize something, we do grandient descent. When we want to maximize something, we do gradient ascent. In this context, we want to &lt;strong&gt;maximize&lt;/strong&gt; $V(D, G)$ with respect to the discriminator $D$, that is, the $\max_D V(D, G)$ part from equation 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend you have a look at NIPS 2016 GAN tutorial &lt;a href=&quot;https://www.youtube.com/watch?v=AJVyzd0rqdc&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/abs/1701.00160&quot; rel=&quot;nofollow noreferrer&quot;&gt;text&lt;/a&gt;. They are very enlightening.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-03-31T15:41:23.930" LastActivityDate="2018-03-31T15:41:23.930" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22078" PostTypeId="1" CreationDate="2017-08-08T16:29:05.800" Score="2" ViewCount="9482" Body="&lt;p&gt;I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided to use &lt;strong&gt;multinomial SVM&lt;/strong&gt; as the evaluator. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import itertools&#xA;from gensim.models import ldamodel&#xA;from nltk.tokenize import RegexpTokenizer&#xA;from nltk.stem.porter import PorterStemmer&#xA;from gensim import corpora, models&#xA;from sklearn.naive_bayes import MultinomialNB&#xA;&#xA;tokenizer = RegexpTokenizer(r'\w+')&#xA;&#xA;# create English stop words list&#xA;en_stop = {'a'}&#xA;&#xA;# Create p_stemmer of class PorterStemmer&#xA;p_stemmer = PorterStemmer()&#xA;&#xA;# create sample documents&#xA;doc_a = &quot;Brocolli is good to eat. My brother likes to eat good brocolli, but not my mother.&quot;&#xA;doc_b = &quot;My mother spends a lot of time driving my brother around to baseball practice.&quot;&#xA;doc_c = &quot;Some health experts suggest that driving may cause increased tension and blood pressure.&quot;&#xA;doc_d = &quot;I often feel pressure to perform well at school, but my mother never seems to drive my brother to do better.&quot;&#xA;doc_e = &quot;Health professionals say that brocolli is good for your health.&quot;&#xA;&#xA;# compile sample documents into a list&#xA;doc_set = [doc_a, doc_b, doc_c, doc_d, doc_e]&#xA;&#xA;# list for tokenized documents in loop&#xA;texts = []&#xA;&#xA;# loop through document list&#xA;for i in doc_set:&#xA;    # clean and tokenize document string&#xA;    raw = i.lower()&#xA;    tokens = tokenizer.tokenize(raw)&#xA;&#xA;    # remove stop words from tokens&#xA;    stopped_tokens = [i for i in tokens if not i in en_stop]&#xA;&#xA;    # stem tokens&#xA;    stemmed_tokens = [p_stemmer.stem(i) for i in stopped_tokens]&#xA;&#xA;    # add tokens to list&#xA;    texts.append(stemmed_tokens)&#xA;&#xA;# turn our tokenized documents into a id &amp;lt;-&amp;gt; term dictionary&#xA;dictionary = corpora.Dictionary(texts)&#xA;&#xA;# convert tokenized documents into a document-term matrix&#xA;corpus = [dictionary.doc2bow(text) for text in texts]&#xA;&#xA;&#xA;# generate LDA model&#xA;#ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word=dictionary, passes=20)&#xA;&#xA;id2word = corpora.Dictionary(texts)&#xA;# Creates the Bag of Word corpus.&#xA;mm = [id2word.doc2bow(text) for text in texts]&#xA;&#xA;# Trains the LDA models.&#xA;lda = ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=4,&#xA;                               update_every=1, chunksize=10000, passes=1)&#xA;&#xA;&#xA;# Assigns the topics to the documents in corpus&#xA;a=[]&#xA;lda_corpus = lda[mm]&#xA;for i in range(len(doc_set)):&#xA;    a.append(lda_corpus[i])&#xA;    print(lda_corpus[i])&#xA;merged_list = list(itertools.chain(*lda_corpus))&#xA;print(a)&#xA;    #my_list.append(my_list[i])&#xA;&#xA;&#xA;sv=MultinomialNB()&#xA;&#xA;yvalues = [0,1,2,3]&#xA;&#xA;sv.fit(a,yvalues)&#xA;predictclass = sv.predict(a)&#xA;&#xA;testLables=[0,1,2,3]&#xA;from sklearn import metrics, tree&#xA;#yacc=metrics.accuracy_score(testLables,predictclass)&#xA;#print (yacc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;when I run this code it throws the error mentioned in the subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also this is the output of LDA model(topic doc distribution) that I feed to SVM:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[(0, 0.95533888404477663), (1, 0.014775921798986477), (2, 0.015161897773308793), (3, 0.014723296382928375)], [(0, 0.019079556242721694), (1, 0.017932434792585779), (2, 0.94498655991579728), (3, 0.018001449048895311)], [(0, 0.017957955483631164), (1, 0.017900184473362918), (2, 0.018133572636989413), (3, 0.9460082874060165)], [(0, 0.96554611572184923), (1, 0.011407838337200715), (2, 0.011537900721487016), (3, 0.011508145219463113)], [(0, 0.023306931039431281), (1, 0.022823706054846005), (2, 0.93072240824085961), (3, 0.023146954664863096)]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my labels here are : 0,1,2,3&lt;/p&gt;&#xA;" OwnerUserId="37510" LastEditorUserId="67971" LastEditDate="2019-03-26T06:55:32.763" LastActivityDate="2019-03-26T06:55:32.763" Title="Found array with dim 3. Estimator expected &lt;= 2" Tags="|machine-learning|python|classification|svm|lda|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22080" PostTypeId="1" CreationDate="2017-08-08T17:08:21.517" Score="2" ViewCount="49" Body="&lt;p&gt;I am fairly new into Data Science but encoutered it before. The following problem troubles me and i hope you guys can point me in the right direction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input are some strings where some carry the same information others not. An unknow number of these strings are crooked* to a warrying degree. From only one letter off to complete garbage. On the output side are the corrected strings from the input. The catch is that there are only certain, already known, combinations of valid strings possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a naive approach i chained some fuzzy searches and already got some promising results. Now i don't know where to start or if there are similar problems already solved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* (are we still allowed to say this?)&lt;/p&gt;&#xA;" OwnerUserId="37680" LastEditorUserId="1330" LastEditDate="2020-12-09T11:34:15.580" LastActivityDate="2022-02-13T16:03:50.337" Title="Mapping a set of corrupted strings to the correct ones" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22081" PostTypeId="1" CreationDate="2017-08-08T18:18:09.843" Score="1" ViewCount="37" Body="&lt;p&gt;Being relatively new to machine learning I am trying to use the simplest algorithm to classify mesh objects. I can have the mesh objects in any familiar file type such as STL, OBJ, PLY, FBX etc. I should have enough training data as well as validation data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is where do I start? how is the mesh converted into some form of a shape that one of the common machine learning libraries (e.g. tensorflow, libsvm) can read?&lt;/p&gt;&#xA;" OwnerUserId="28572" LastActivityDate="2017-08-08T18:18:09.843" Title="Classifying / labeling polygonal meshes" Tags="|machine-learning|classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22083" PostTypeId="2" ParentId="21896" CreationDate="2017-08-08T20:20:05.910" Score="9" Body="&lt;p&gt;Yes &lt;em&gt;in theory&lt;/em&gt; the polynomial extension to logistic regression can approximate any arbitrary classification boundary. That is because a polynomial can approximate any function (at least of the types useful to classification problems), and this is proven by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem&quot; rel=&quot;noreferrer&quot;&gt;Stone-Weierstrass theorem&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether this approximation is practical for all boundary shapes is another matter. You may be better looking for other basis functions (e.g. Fourier series, or radial distance from example points), or other approaches entirely (e.g. SVM) when you suspect a complex boundary shape in feature space. The problem with using high order polynomials is that the number of polynomial features you need to use grows exponentially with degree of the polynomial and number of original features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could make a polynomial to classify XOR. $5 - 10 xy$ might be a start if you use $-1$ and $1$ as the binary inputs, this maps input $(x,y)$ to output as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(-1,-1): -5 \qquad (-1,1): 5 \qquad (1,-1): 5 \qquad(1, 1): -5$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Passing that into the logistic function should give you values close enough to 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar to your two circular areas is a simple figure-of-eight curve:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a(x^2 - y^2 - bx^4 + c)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $a, b$ and $c$ are constants. You can get two disjoint closed areas defined in your classifier - on opposite sides of the $y$ axis, by choosing $a, b$ and $c$ appropriately. For example try $a=1,b=0.05,c=-1$ to get a function that clearly separates into two peaks around $x=-3$ and $x=3$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dNysS.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dNysS.png&quot; alt=&quot;two separate classes&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plot shown is from &lt;a href=&quot;https://academo.org/demos/3d-surface-plotter/?expression=x%5E2-y%5E2-0.05*x%5E4-1.0%3E0&amp;amp;xRange=-5%2C%2B5&amp;amp;yRange=-5%2C%2B5&amp;amp;resolution=95&quot; rel=&quot;noreferrer&quot;&gt;an online tool at academo.org&lt;/a&gt;, and is for $x^2 - y^2 - 0.05x^4 -1&amp;gt;0$ - the positive class shown as value 1 in the plot above, and is typically where $\frac{1}{1+e^{-z}} &amp;gt; 0.5$ in logistic regression or just $z&amp;gt;0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An optimiser will find best values, you would just need to use $1, x^2, y^2, x^4$ as your expansion terms (although note these specific terms are limited to matching the same basic shape reflected around the $y$ axis - in practice you would want to have multiple terms up to fourth degree polynomial to find more arbitrary disjoint groups in a classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact any problem you can solve with a deep neural network - of any depth - you can solve with a flat structure using linear regression (for regression problems) or logistic regression (for classification problems). It is &quot;just&quot; a matter of finding the right feature expansion. The difference is that neural networks will attempt to discover a working feature expansion directly, whilst feature engineering using polynomials or any other scheme is hard work and not always obvious how to even start: Consider for example how you might create polynomial approximations to what convolutional neural networks do for images? It seems impossible. It is likely to be extremely impractical, too. But it does exist.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-09T07:36:24.300" LastActivityDate="2017-08-09T07:36:24.300" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="22084" PostTypeId="1" CreationDate="2017-08-08T18:12:10.467" Score="1" ViewCount="1161" Body="&lt;p&gt;I'm using the following code to fit a simple keras model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Prepare data&#xA;x1_dat = []&#xA;x2_dat = []&#xA;y_dat = []&#xA;cs = 3&#xA;for session in Sessions_group:&#xA;    session = session.split(',')&#xA;    for i in xrange(0,len(session)-cs+1):&#xA;        x1_dat.append(session[i])&#xA;        x2_dat.append(session[i+1])&#xA;        y_dat.append(session[i+2])&#xA;&#xA;x1_dat = np.stack(x1_dat)&#xA;x2_dat = np.stack(x2_dat)&#xA;y_dat = np.stack(y_dat)&#xA;&#xA;# Create model&#xA;n_fac = 100&#xA;def embedding_input(name, n_in, n_out):&#xA;    inp = Input(shape=(1,), dtype='int64', name=name)&#xA;    emb = Embedding(n_in, n_out, input_length=1)(inp)&#xA;    return inp, Flatten()(emb)&#xA;&#xA;vocab_size = len(uniqueArtifacts)&#xA;x1_in, x1 = embedding_input('x1', vocab_size, n_fac)&#xA;x2_in, x2 = embedding_input('x2', vocab_size, n_fac)&#xA;n_hidden = 512&#xA;dense_in = Dense(n_hidden, activation='relu')&#xA;x1_hidden = dense_in(x1)&#xA;dense_hidden = Dense(n_hidden, activation='tanh')&#xA;x2_dense = dense_in(x2)&#xA;hidden_2 = dense_hidden(x1_hidden)&#xA;x2_hidden = merge([x2_dense, hidden_2])&#xA;dense_out = Dense(vocab_size, activation='softmax')&#xA;x3_out = dense_out(x2_hidden)&#xA;&#xA;# Fit model&#xA;model = Model([x1_in, x2_in], x3_out)&#xA;model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())&#xA;model.fit([x1_dat, x2_dat], y_dat, batch_size=10, nb_epoch=1, verbose = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To briefly explain it - I'm trying to predict the third item given  the first 2 items. I pass item1 and item2 through &lt;code&gt;Dense&lt;/code&gt; layers (&lt;code&gt;dense_in&lt;/code&gt; in the code), and merge (sum) the output and pass it through another &lt;code&gt;Dense&lt;/code&gt; layer (&lt;code&gt;dense_out&lt;/code&gt; in the code) to produce the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seemingly simple code gives me the following error when I execute the &lt;code&gt;fit&lt;/code&gt; command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------------------------&#xA;IndexError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-31-1912702bd26c&amp;gt; in &amp;lt;module&amp;gt;()&#xA;----&amp;gt; 1 model.fit([x1_dat, x2_dat], y_dat, batch_size=10, nb_epoch=1, verbose = 1)&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit(self, x, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch)&#xA;   1194                               val_f=val_f, val_ins=val_ins, shuffle=shuffle,&#xA;   1195                               callback_metrics=callback_metrics,&#xA;-&amp;gt; 1196                               initial_epoch=initial_epoch)&#xA;   1197 &#xA;   1198     def evaluate(self, x, y, batch_size=32, verbose=1, sample_weight=None):&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in _fit_loop(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch)&#xA;    889                 batch_logs['size'] = len(batch_ids)&#xA;    890                 callbacks.on_batch_begin(batch_index, batch_logs)&#xA;--&amp;gt; 891                 outs = f(ins_batch)&#xA;    892                 if not isinstance(outs, list):&#xA;    893                     outs = [outs]&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.pyc in __call__(self, inputs)&#xA;    957     def __call__(self, inputs):&#xA;    958         assert isinstance(inputs, (list, tuple))&#xA;--&amp;gt; 959         return self.function(*inputs)&#xA;    960 &#xA;    961 &#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)&#xA;    896                     node=self.fn.nodes[self.fn.position_of_error],&#xA;    897                     thunk=thunk,&#xA;--&amp;gt; 898                     storage_map=getattr(self.fn, 'storage_map', None))&#xA;    899             else:&#xA;    900                 # old-style linkers raise their own exceptions&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)&#xA;    323         # extra long error message in that case.&#xA;    324         pass&#xA;--&amp;gt; 325     reraise(exc_type, exc_value, exc_trace)&#xA;    326 &#xA;    327 &#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)&#xA;    882         try:&#xA;    883             outputs =\&#xA;--&amp;gt; 884                 self.fn() if output_subset is None else\&#xA;    885                 self.fn(output_subset=output_subset)&#xA;    886         except Exception:&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/theano/gof/op.pyc in rval(p, i, o, n)&#xA;    870             # default arguments are stored in the closure of `rval`&#xA;    871             def rval(p=p, i=node_input_storage, o=node_output_storage, n=node):&#xA;--&amp;gt; 872                 r = p(n, [x[0] for x in i], o)&#xA;    873                 for o in node.outputs:&#xA;    874                     compute_map[o][0] = True&#xA;&#xA;/home/prateek/anaconda2/lib/python2.7/site-packages/theano/tensor/subtensor.pyc in perform(self, node, inputs, out_)&#xA;   2240 &#xA;   2241         if self.set_instead_of_inc:&#xA;-&amp;gt; 2242             out[0][inputs[2:]] = inputs[1]&#xA;   2243         elif config.cxx:&#xA;   2244             inplace_increment(out[0], tuple(inputs[2:]), inputs[1])&#xA;&#xA;IndexError: index 1864798 is out of bounds for axis 1 with size 2392&#xA;Apply node that caused the error: AdvancedIncSubtensor{inplace=False,  set_instead_of_inc=True}(Alloc.0, TensorConstant{1}, ARange{dtype='int64'}.0, Elemwise{Cast{int32}}.0)&#xA;Toposort index: 69&#xA;Inputs types: [TensorType(float32, matrix), TensorType(int8, scalar), TensorType(int64, vector), TensorType(int32, vector)]&#xA;Inputs shapes: [(10, 2392), (), (10,), (10,)]&#xA;Inputs strides: [(9568, 4), (), (8,), (4,)]&#xA;Inputs values: ['not shown', array(1, dtype=int8), 'not shown', 'not shown']&#xA;Outputs clients: [[GpuFromHost(AdvancedIncSubtensor{inplace=False,  set_instead_of_inc=True}.0)]]&#xA;&#xA;Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py&quot;, line 2717, in run_cell&#xA;    interactivity=interactivity, compiler=compiler, result=result)&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py&quot;, line 2827, in run_ast_nodes&#xA;    if self.run_code(code, result):&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py&quot;, line 2881, in run_code&#xA;    exec(code_obj, self.user_global_ns, self.user_ns)&#xA;  File &quot;&amp;lt;ipython-input-29-6ef574590fa0&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.py&quot;, line 667, in compile&#xA;    sample_weight, mask)&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.py&quot;, line 318, in weighted&#xA;    score_array = fn(y_true, y_pred)&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/objectives.py&quot;, line 41, in sparse_categorical_crossentropy&#xA;    return K.sparse_categorical_crossentropy(y_pred, y_true)&#xA;  File &quot;/home/prateek/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py&quot;, line 1249, in sparse_categorical_crossentropy&#xA;    target = T.extra_ops.to_one_hot(target, nb_class=output.shape[-1])&#xA;&#xA;HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further information:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The total number of values that I'm trying to predict from is 2392.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The index in the error &lt;code&gt;IndexError: index 1864798 is out of bounds for axis 1 with size 2392&lt;/code&gt; (1864798 in this case) keeps changing as I re-execute the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output to the following code (the size of my data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1_dat.shape, x2_dat.shape, y_dat.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;((6510,), (6510,), (6510,))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I go about resolving this error? Any help would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="33907" OwnerDisplayName="Patthebug" LastActivityDate="2017-08-08T20:40:17.843" Title="Index error in simple keras model" Tags="|deep-learning|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22086" PostTypeId="1" CreationDate="2017-08-09T00:19:23.887" Score="1" ViewCount="113" Body="&lt;p&gt;Neural nets are widely used (as example for the MNIST dataset). Using neural networks and convolutional neural networks, in particular, we can get over 99% accuracy. However, the MNIST dataset is a dataset where each image only has one digit cropped and centered so that it is always in the same position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to create a neural network to recognize digits where the digits may not be in the center (and possibly there may be other objects or digits in the scene), how do I go about developing it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to detect the digits and their position.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-16T00:43:58.193" LastActivityDate="2021-02-16T00:43:58.193" Title="Using neural networks to recognize digits in a scene" Tags="|neural-network|convolutional-neural-network|mnist|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22087" PostTypeId="1" AcceptedAnswerId="22116" CreationDate="2017-08-09T01:39:31.170" Score="-1" ViewCount="1601" Body="&lt;p&gt;Assuming there is a $n$ x $m$ matrix with $n$ features and $m$ samples (each row is a feature and each column is a sample). I would like to use isolation forest in sklearn to return the positions of anomaly samples, for example, it returns an array [3,9,28,66] which contains the positions of columns of anomalies. Could anyone help me to do this by giving a small example code? Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="31484" LastActivityDate="2017-08-09T15:34:35.370" Title="How to use isolation forest from sklearn to return the positions of anomalies?" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22088" PostTypeId="1" CreationDate="2017-08-09T03:50:22.753" Score="4" ViewCount="741" Body="&lt;p&gt;I am extracting tweets on brands for sentiment analysis. I am using twitteR package on R. Is there a specific way by which I can only collect tweets from a location ex NY or Paris or Canada etc. I have tried Geocode but here I have to give lat and long data and the vicinity range.&lt;/p&gt;&#xA;" OwnerUserId="37698" LastEditorUserId="1330" LastEditDate="2018-05-07T14:38:02.077" LastActivityDate="2018-05-07T14:38:02.077" Title="How to collect tweets by geo-location?" Tags="|r|nlp|sentiment-analysis|geospatial|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22089" PostTypeId="1" AcceptedAnswerId="22122" CreationDate="2017-08-09T04:51:55.183" Score="0" ViewCount="2375" Body="&lt;p&gt;I am playing with features (input data) to improve my model's accuracy.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data and feed it as input? Or will the algorithm learn these from the time-series itself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, should I create a column that is comprised of the moving average, or will the algorithm pick up on moving average from the raw data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is feature engineering just the munging of independent variables? Or is it extracting features that are dependent on other raw data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's another question: If I have a categorical feature, would it be better to have it as a one-hot vector (say, 5 binary inputs), or to have it as one input with range [0,4]?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does one intuitively know the answer to these questions??&lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="37109" LastEditDate="2017-08-09T06:03:39.757" LastActivityDate="2017-08-09T22:11:00.943" Title="Difference Between Feature Engineering and Feature Learning" Tags="|machine-learning|data|feature-engineering|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22090" PostTypeId="2" ParentId="18890" CreationDate="2017-08-09T05:15:57.917" Score="1" Body="&lt;p&gt;Market Basket Analysis can be used. Follow this link to understand more about the algorithms used and other key concepts&#xA;&lt;a href=&quot;http://www.listendata.com/2015/12/market-basket-analysis-with-r.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.listendata.com/2015/12/market-basket-analysis-with-r.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37698" LastActivityDate="2017-08-09T05:15:57.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22093" PostTypeId="1" AcceptedAnswerId="22097" CreationDate="2017-08-09T07:30:39.773" Score="15" ViewCount="11544" Body="&lt;p&gt;I read this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To train our neural network, we will initialize each parameter W(l)ijWij(l) and each b(l)ibi(l) to a small random value near zero (say according to a Normal(0,ϵ2)Normal(0,ϵ2) distribution for some small ϵϵ, say 0.01)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;from &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/&quot; rel=&quot;noreferrer&quot;&gt;Stanford Deep learning tutorials&lt;/a&gt;&#xA;at the 7th paragraph in the &lt;em&gt;Backpropagation Algorithm&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is why the initialization of the weight or bias should be &lt;strong&gt;around 0&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="-1" LastEditDate="2017-08-09T15:14:32.730" LastActivityDate="2021-03-11T20:17:02.237" Title="Why should the initialization of weights and bias be chosen around 0?" Tags="|deep-learning|stanford-nlp|randomized-algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22094" PostTypeId="1" CreationDate="2017-08-09T07:41:54.103" Score="0" ViewCount="375" Body="&lt;p&gt;I have one dataset, and decided to use XGBClassifier to get a variable importance plot from it.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was a little surprised that some features that I assumed were insightful had no appreciable value.  With this in mind, I used an sklearn RandomForestClassifier to create another variable importance plot.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my surprise the two were very different! Many variables that appear on one don't appear on the other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used as similar parameters as I could, between the two models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what they look like to give a general idea:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QeyCt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QeyCt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/IDpdO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IDpdO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone had this similar experience, and if so, what kinds of reasoning for this might there be? It's a little frightening to think that many Kagglers and others really depend on these plots and to see that they can vary so much...makes me wonder.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="15533" LastActivityDate="2017-08-09T08:15:22.343" Title="XGBClassifier and RandomForestClassifier variable importance plots are very different" Tags="|visualization|random-forest|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22095" PostTypeId="2" ParentId="22094" CreationDate="2017-08-09T08:15:22.343" Score="1" Body="&lt;p&gt;There are multiple reasons why this could happen. The first, and probably the most important one is that while both of the algorithms are tree based ensemble methods, they are very different from each other. Second of all, these feature importances are just some heuristic that do some calculations on where the features are used. I assume these heuristics are very different for both the algorithms&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case however the differences might just be too big to only point to these two reasons. What could have happened is that your features are highly correlated. If two columns are highly correlated then the algorithms become kind of indifferent to which one to use, which means the two algorithms might both pick a different one each time they come across this choice. You could take a look if column 0 and column 7 are very correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is that some of your columns are categorical and that the implementations deal with this in a different way. Python doesn't work cleanly with categorical values in Scikit-learn. One-hot encoding is not a nice representation for undeep trees but if you believe this may be causing it you could preprocess them like that to see if the feature importances become more similar.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-09T08:15:22.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22096" PostTypeId="2" ParentId="22093" CreationDate="2017-08-09T08:39:35.610" Score="4" Body="&lt;p&gt;If you set it as 0, they will all have the same error so backprop will make them all equal; therefore, you should have random initialisation.&lt;/p&gt;&#xA;&lt;p&gt;Why around 0? I think &lt;a href=&quot;https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network&quot;&gt;this post&lt;/a&gt; may answer it well:&lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="29169" LastEditDate="2021-03-11T20:17:02.237" LastActivityDate="2021-03-11T20:17:02.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22097" PostTypeId="2" ParentId="22093" CreationDate="2017-08-09T09:09:38.537" Score="12" Body="&lt;p&gt;Assuming fairly reasonable data normalization, the expectation of the weights should be zero or close to it. It might be reasonable, then, to set all of the initial weights to zero because a positive initial weight will have further to go if it should actually be a negative weight and visa versa. This, however, does not work. If all of the weights are the same, they will all have the same error and the model will not learn anything - there is no source of asymmetry between the neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we could do, instead, is to keep the weights very close to zero but make them different by initializing them to small, non-zero numbers. This is what is suggested in the tutorial that you linked. It has the same advantage of all-zero initialization in that it is close to the 'best guess' expectation value but the symmetry has also been broken enough for the algorithm to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach has additional problems. It is not necessarily true that smaller numbers will work better, especially if the neural network is deep. The gradients calculated in backpropagation are proportional to the weights; very small weights lead to very small gradients and can lead to the network taking much, much longer to train or never completing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another potential issue is that the distribution of the outputs of each neuron, when using random initialization values, has a variance that gets larger with more inputs. A common additional step is to normalize the neuron's output variance to 1 by dividing its weights by $sqrt(d)$ where $d$ is the number of inputs to the neuron. The resulting weights are normally distributed between $\left[\frac{-1}{\sqrt{d}}, \frac{1}{\sqrt{d}}\right]$&lt;/p&gt;&#xA;" OwnerUserId="30964" LastActivityDate="2017-08-09T09:09:38.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22098" PostTypeId="2" ParentId="22051" CreationDate="2017-08-09T09:56:00.617" Score="3" Body="&lt;p&gt;First I would say the fuzzy clustering is not necessarily a clustering algorithm which uses fuzzy logic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In machine learning terminology, a &quot;soft&quot; clustering algorithm is called fuzzy clustering as the intuition behind it is not &quot;crisp&quot; (like hard clustering algorithms in which a data point belongs ONLY to one cluster and not others) i.e. for each data point, they output a vector of &quot;probabilities&quot; corresponding to the membership of that point to each cluster (please note that fuzzy memberships are not necessarily probabilities). Even in &lt;a href=&quot;https://en.wikipedia.org/wiki/Fuzzy_clustering#Fuzzy_C-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;FCM algorithm&lt;/a&gt;, you are just doing a soft version of k-means and it has nothing to do with fuzzy logic (except of the very general idea i.e. not being crisp in making decision)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you can apply them to any dataset.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-08-09T09:56:00.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22100" PostTypeId="2" ParentId="18802" CreationDate="2017-08-09T10:01:56.087" Score="1" Body="&lt;p&gt;Answering to @Media's comment about why there are many local minima: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wws2P.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wws2P.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider, your randomly initialized weights to be belonging to energy level E4 in above image. Clearly, E1, E2, E3 have lesser energy. So all these 3 levels are minima w.r.t. E4. Now, with neural networks, your learning algorithms is such that it is &lt;em&gt;guaranteed&lt;/em&gt; [yes, mathematical guarantee] to take you to a lesser energy level. The trouble is you don't know which lower state it will put you into. But, oh, lower, it will be!&lt;/p&gt;&#xA;" OwnerUserId="36971" LastEditorUserId="24000" LastEditDate="2017-08-09T12:00:46.680" LastActivityDate="2017-08-09T12:00:46.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22101" PostTypeId="1" CreationDate="2017-08-09T11:01:49.043" Score="4" ViewCount="6686" Body="&lt;p&gt;As I am new to TensorFlow, I would like to do image recognition in TensorFlow using Python. For this Image Recognition I would like to train my own image dataset and test that dataset. Please answer me how to train a dataset and how to select the dataset..&lt;/p&gt;&#xA;" OwnerUserId="37715" LastActivityDate="2017-08-09T11:10:41.507" Title="How to train an image dataset in TensorFlow?" Tags="|python|dataset|tensorflow|training|image-recognition|" AnswerCount="1" CommentCount="0" ClosedDate="2017-08-09T12:00:19.067" ContentLicense="CC BY-SA 3.0" />
  <row Id="22102" PostTypeId="2" ParentId="22101" CreationDate="2017-08-09T11:10:41.507" Score="1" Body="&lt;p&gt;If you are interested in training tensorflow model for image recognition in general, it is a good idea to go through tensorflow tutorials on MNIST:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/beginners&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/get_started/mnist/beginners&lt;/a&gt;&#xA;&lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/pros&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/get_started/mnist/pros&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to collect the data yourself and train model on it, here is something which could be interesting: &lt;a href=&quot;https://digits-draw-recognize.herokuapp.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://digits-draw-recognize.herokuapp.com/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my site with handwritten digit recognition. I have collected dataset by myself - I drew ~1k digits and processed the data. After this I trained two models, one of them is CNN in Tensorflow. And these models train on the new images drawn by people. You can find some description of functionality on this site.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-08-09T11:10:41.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22103" PostTypeId="1" AcceptedAnswerId="22109" CreationDate="2017-08-09T11:37:27.223" Score="-1" ViewCount="55" Body="&lt;p&gt;I'm currently training an agent to learn how to fight in a shooting game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the bullet positions of the agent's opponent as one of the features. The features &quot;don't exist&quot; when the opponent isn't firing a bullet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I substitute the feature with when the opponent of the agent doesn't fire a bullet? Right now, I'm considering using &quot;0&quot;, but are there better alternatives?&lt;/p&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="37548" LastEditDate="2017-08-22T12:58:05.647" LastActivityDate="2017-08-22T12:58:05.647" Title="How should values that &quot;don't exist&quot; sometimes be handled as input data?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22104" PostTypeId="1" CreationDate="2017-08-09T11:46:52.777" Score="1" ViewCount="113" Body="&lt;p&gt;I am very new to machine learning.I am studying Suicide data in India from kaggle dataset.&#xA;&lt;a href=&quot;https://www.kaggle.com/rajanand/suicides-in-india/kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/rajanand/suicides-in-india/kernels&lt;/a&gt;&#xA;But I am unable to understand &lt;strong&gt;&lt;em&gt;what should be the target variable for prediction or classification&lt;/em&gt;&lt;/strong&gt;.&#xA;My immediate thought is that a socially important goal of such studies is a prevention of future suicides.For ex:&#xA;a number of suicides going to happen next year among students of a given high school. If a prediction value is high, then preventive measures need to be taken.&#xA;But I am unable to start which ML algorithm should i use or how to approach.I have completed data analysis.&#xA;Please help.&lt;/p&gt;&#xA;" OwnerUserId="37716" LastActivityDate="2017-08-09T11:46:52.777" Title="what should be the Prediction or Target variable for Suicide data in India" Tags="|classification|random-forest|decision-trees|" AnswerCount="0" CommentCount="0" ClosedDate="2017-08-09T12:00:09.290" ContentLicense="CC BY-SA 3.0" />
  <row Id="22105" PostTypeId="2" ParentId="20587" CreationDate="2017-08-09T12:00:50.987" Score="11" Body="&lt;p&gt;Consider the following approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def f(col, threshold=3):&#xA;    mask = col.groupby((col != col.shift()).cumsum()).transform('count').lt(threshold)&#xA;    mask &amp;amp;= col.eq(0)&#xA;    col.update(col.loc[mask].replace(0,1))&#xA;    return col&#xA;&#xA;In [79]: df.apply(f, threshold=3)&#xA;Out[79]:&#xA;       col1  col2&#xA;names&#xA;A         1     0&#xA;B         1     0&#xA;C         1     0&#xA;D         1     0&#xA;E         1     1&#xA;F         1     1&#xA;G         0     1&#xA;H         0     1&#xA;I         0     1&#xA;J         1     0&#xA;K         1     0&#xA;L         1     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Step by step:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [84]: col = df['col2']&#xA;&#xA;In [85]: col&#xA;Out[85]:&#xA;names&#xA;A    0&#xA;B    0&#xA;C    0&#xA;D    0&#xA;E    1&#xA;F    0&#xA;G    1&#xA;H    0&#xA;I    1&#xA;J    0&#xA;K    0&#xA;L    0&#xA;Name: col2, dtype: int64&#xA;&#xA;In [86]: (col != col.shift()).cumsum()&#xA;Out[86]:&#xA;names&#xA;A    1&#xA;B    1&#xA;C    1&#xA;D    1&#xA;E    2&#xA;F    3&#xA;G    4&#xA;H    5&#xA;I    6&#xA;J    7&#xA;K    7&#xA;L    7&#xA;Name: col2, dtype: int32&#xA;&#xA;In [87]: col.groupby((col != col.shift()).cumsum()).transform('count')&#xA;Out[87]:&#xA;names&#xA;A    4&#xA;B    4&#xA;C    4&#xA;D    4&#xA;E    1&#xA;F    1&#xA;G    1&#xA;H    1&#xA;I    1&#xA;J    3&#xA;K    3&#xA;L    3&#xA;Name: col2, dtype: int64&#xA;&#xA;In [88]: col.groupby((col != col.shift()).cumsum()).transform('count').lt(3)&#xA;Out[88]:&#xA;names&#xA;A    False&#xA;B    False&#xA;C    False&#xA;D    False&#xA;E     True&#xA;F     True&#xA;G     True&#xA;H     True&#xA;I     True&#xA;J    False&#xA;K    False&#xA;L    False&#xA;Name: col2, dtype: bool&#xA;&#xA;In [89]: col.groupby((col != col.shift()).cumsum()).transform('count').lt(3) &amp;amp; col.eq(0)&#xA;Out[89]:&#xA;names&#xA;A    False&#xA;B    False&#xA;C    False&#xA;D    False&#xA;E    False&#xA;F     True&#xA;G    False&#xA;H     True&#xA;I    False&#xA;J    False&#xA;K    False&#xA;L    False&#xA;Name: col2, dtype: bool&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2017-08-09T12:06:11.903" LastActivityDate="2017-08-09T12:06:11.903" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22107" PostTypeId="1" CreationDate="2017-08-09T12:27:16.260" Score="1" ViewCount="183" Body="&lt;p&gt;I want to predict sales quantity for items. My data is like if the item is sold in week it has number of items sold. &lt;strong&gt;If not sold sales quantity is maintained as zero. Problem is my data is having lot of unsold records compared to sold items. Is it okay to maintain zero sales record in training data?&lt;/strong&gt;&#xA;Currently I am using &lt;strong&gt;Random Forest&lt;/strong&gt; for  my prediction. &lt;/p&gt;&#xA;" OwnerUserId="37626" LastEditorUserId="16401" LastEditDate="2017-08-09T14:56:25.857" LastActivityDate="2017-08-09T14:56:25.857" Title="Sales Prediction" Tags="|r|predictive-modeling|regression|random-forest|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22109" PostTypeId="2" ParentId="22103" CreationDate="2017-08-09T12:50:28.580" Score="1" Body="&lt;p&gt;Having to input a non-existing feature is a common problem in machine learning models. Entering &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;0&lt;/code&gt; could mean the position &lt;code&gt;{ x: 0, y: 0 }&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you'd input &quot;nothing&quot;, that still would be &lt;code&gt;0&lt;/code&gt; (because &lt;code&gt;nothing * weight = 0&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best you can do is figure out what works best through trial and error. I could think of a few options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If your network supports negative activation values (range: &lt;code&gt;-1, 1&lt;/code&gt;), you should try to input it as &lt;code&gt;-1&lt;/code&gt;.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I don't think the learning model will have a hard time if you'd just input &lt;code&gt;0&lt;/code&gt; as input, like you proposed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Add an extra input feature, 'bullet exists'. &lt;code&gt;0&lt;/code&gt; if there is no bullet, &lt;code&gt;1&lt;/code&gt; if there is a bullet. Then just input &lt;code&gt;0&lt;/code&gt; for the x and y coordinates.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30928" LastActivityDate="2017-08-09T12:50:28.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22110" PostTypeId="2" ParentId="22047" CreationDate="2017-08-09T13:11:20.230" Score="0" Body="&lt;p&gt;I am working on data for 3 years(daily data) and can see the values changing for all the days. I am trying to predict for next 30 days using &lt;a href=&quot;http://machinelearningmastery.com/multi-step-time-series-forecasting-long-short-term-memory-networks-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dr Jason Brownlee's tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The above tutorial helped me predict for next 30 days, I am still trying to improve on the predictions .&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt; :While trying to improve my results I was experimenting and came across similar situation as yours.&#xA;Recommendation ---&amp;gt;&lt;strong&gt;Remove any activation functions from your network&lt;/strong&gt; and it may help.&lt;/p&gt;&#xA;" OwnerUserId="37720" LastEditorUserId="98307" LastEditDate="2020-08-13T10:29:58.103" LastActivityDate="2020-08-13T10:29:58.103" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22113" PostTypeId="2" ParentId="22000" CreationDate="2017-08-09T13:55:08.290" Score="2" Body="&lt;p&gt;Here is one way to generate two data frames based on the data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# read data&#xA;dat &amp;lt;- read.csv(&quot;sample csv1.csv&quot;, stringsAsFactors = FALSE, header = FALSE)&#xA;&#xA;# find rows containing &quot;Interval&quot;&#xA;idx_int &amp;lt;- dat[[1]] == &quot;Interval&quot;&#xA;&#xA;# calculate groups&#xA;group &amp;lt;- cumsum(idx_int) %% 2&#xA;&#xA;# remove indices for &quot;Interval&quot; rows&#xA;group2 &amp;lt;- group[!idx_int]&#xA;&#xA;# split data frame into two data frames&#xA;dat_list &amp;lt;- split(dat[!idx_int, ], group2)&#xA;&#xA;# generate two separate data frames&#xA;dat_energy &amp;lt;- dat_list[[2]]  # Energy&#xA;dat_demand &amp;lt;- dat_list[[1]]  # Demand&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="106" LastEditorUserId="106" LastEditDate="2017-08-10T15:12:26.763" LastActivityDate="2017-08-10T15:12:26.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22116" PostTypeId="2" ParentId="22087" CreationDate="2017-08-09T15:34:35.370" Score="0" Body="&lt;p&gt;You only have to call the &lt;code&gt;predict&lt;/code&gt; method on your matrix, which returns an array of &lt;code&gt;+1/-1&lt;/code&gt; for inliers/outliers, assuming your IsolationForest is already fitted. Note that the matrix should be of the form (rows=samples x columns=features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this should do it&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mdl = IsolationForest(...)&#xA;mdl.fit(...)&#xA;#your matrix &quot;X&quot; should be transposed to the form (m samples x n features)&#xA;pred = mdl.predict(X.T)&#xA;anomaly_columns = np.where(pred==-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-08-09T15:34:35.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22117" PostTypeId="1" AcceptedAnswerId="22121" CreationDate="2017-08-09T16:00:45.103" Score="2" ViewCount="508" Body="&lt;p&gt;I have been looking at autoencoders from the keras blog here: &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.keras.io/building-autoencoders-in-keras.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering, what motifications would be necessary in order to be able to give it different surfaces i.e. 2-dimensional vectors, of which some of them have large spikes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example here we see a surface that looks clean:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q6oIE.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q6oIE.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could a neural network look like, if I wanted to remove individual spikes from this surface?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I right in thinking that a normal fully connected feed forward propagation would be sufficient? If so, is there any way to control thresholds when spikes are should be eliminated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, would you agree that the training principle would still be the same as show in the keras blog? Would it work if I simply trained it with many good examples of clean surfaces to recognize themselves?&lt;/p&gt;&#xA;" OwnerUserId="32860" LastActivityDate="2017-08-09T21:37:13.017" Title="Autoencoder for cleaning outliers in a surface" Tags="|keras|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22118" PostTypeId="1" AcceptedAnswerId="22132" CreationDate="2017-08-09T19:06:20.703" Score="2" ViewCount="2410" Body="&lt;p&gt;Why do we need for Shortcut Connections to build &lt;a href=&quot;https://arxiv.org/abs/1512.03385v1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Residual Networks&lt;/a&gt;, and how it help to train neural networks for classification and detection?&lt;/p&gt;&#xA;" OwnerUserId="37736" LastActivityDate="2017-08-10T09:18:42.043" Title="Why do we need for Shortcut Connections to build Residual Networks?" Tags="|machine-learning|neural-network|deep-learning|computer-vision|caffe|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22119" PostTypeId="1" CreationDate="2017-08-09T20:03:11.423" Score="1" ViewCount="153" Body="&lt;p&gt;Let me motivate my problem with an example. Let's assume our observations concern ratings a user give to different items, while navigating through item catalog. The user begins rating item1 (may be relevance), and proceeds to item2 to give (or not) a rating to that item. The user may return back to item1 to give yet another score (recursive nature). I am interested in learning a recommender (say, collaborative) which predicts a score based on the context (traversal path) (s)he is in (or learn some kind of a general transfer function which predicts based on past context).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A use case would be to find the true user-item rating score. Another would be to recommend an item to a user with the highest rated score based on the contexts (traversal history) of that user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The observations can be encoded as:&#xA;$$&#xA;User \times Context_1 \leftarrow Score \\&#xA;User \times Context_1 \times Context_2 \leftarrow Score \\&#xA;\vdots \\&#xA;User \times Context_1 \times \ldots \times Context_N \leftarrow Score&#xA;$$&#xA;where any of the two contexts can be the same, eg. $(u_1, it_1, it_2, it_1) = 5$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been looking at Tensors to incorporate contextual information for collaborative filtering. On one hand, the resulting tensor has very high number of dimensions (longest possible traversal path of the observed data) and extremely sparse. On the other hand, since the length of traversal history for each user is different, the resulting tensor for unobserved dimensions is incomplete.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;User \times Context_1 \times Score_1 \times ... \times Context_N \times Score_N \leftarrow \{0,1\}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the tensor decomposition the right technique to tackle this problem? Are deep models such as RNNs better fit for this kind of nested recursive contextual problems.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If TD is the way to go, what is the best approach to deal with the dimensions of the problem? I do know there has been some new techniques such as Tensor-Train or Hierarchical Tucker decomposition for dealing with the `curse of dimensionality'.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any insights into this problem or pointers is highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="37739" LastActivityDate="2017-08-09T20:03:11.423" Title="Tensor Decomposition for Higher-Order Context-Aware Recommender Systems" Tags="|deep-learning|recommender-system|matrix-factorisation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22120" PostTypeId="2" ParentId="22088" CreationDate="2017-08-09T20:42:07.860" Score="1" Body="&lt;p&gt;To get location based tweets, you have to specify a location circle with center (lat and long) and radius using &lt;a href=&quot;https://dev.twitter.com/rest/reference/get/geo/reverse_geocode&quot; rel=&quot;nofollow noreferrer&quot;&gt;reverse_geocode&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no way to find tweets by setting a polygon or drawing a border.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-09T20:42:07.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22121" PostTypeId="2" ParentId="22117" CreationDate="2017-08-09T21:37:13.017" Score="5" Body="&lt;p&gt;A possible approach would be a &lt;strong&gt;denoising autoencoder&lt;/strong&gt;. It is like a normal autoencoder but instead of training it using the same input and output, you &lt;em&gt;inject&lt;/em&gt; noise on the input while keeping the expected output clean. Hence, the autoencoder learns to &lt;em&gt;remove&lt;/em&gt; it. This kind of autoencoders are also described in the blog post you linked to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you could just train your denoising autoencoder injecting to the inputs spikes of the &lt;em&gt;height&lt;/em&gt; you expect to be removed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About what kind of architecture (e.g. fully connected, convolutional), only actual tests can tell you what is appropriate and what is not.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-08-09T21:37:13.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22122" PostTypeId="2" ParentId="22089" CreationDate="2017-08-09T22:11:00.943" Score="1" Body="&lt;p&gt;Feature engineering refers to creating new information that was not there previously, often by using domain specific knowledge or by creating new features that are transformations of others you already have, such as adding interaction terms or as you state, moving averages. &#xA;A model generally cannot 'pick up' on information it doesn't have, and that is where  finesse and creativity comes into play. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether you should one-hot or leave a feature as categorical depends on the modeling approach. Some, like randomForest will do fine with categorical predictors; others prefer recoding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuition on these questions comes with practice and experience.  There's no substitute for trying out and comparing toy examples to see how your choices affect outcomes. You should take the time to do that, and intuition will follow. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-08-09T22:11:00.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22126" PostTypeId="1" AcceptedAnswerId="30579" CreationDate="2017-08-09T23:35:57.710" Score="0" ViewCount="334" Body="&lt;p&gt;When I have N samples of 2 different features (different scale, different meaning etc.), but whose values are scalar numbers of the same format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to flatten the data into a 1D array of length [N*2] and feed it in this 1D shape into the neural network. Or must I keep them separate (as columns or otherwise)? I will standardize each feature before combining them into the single array.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I want to do this for convenience reasons i.e. not deal with multidimensional arrays).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is input for a classification problem.&lt;/p&gt;&#xA;" OwnerUserId="37747" LastActivityDate="2018-05-20T18:47:01.907" Title="Does it make sense to flatten two different features into a single array" Tags="|machine-learning|neural-network|feature-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22127" PostTypeId="2" ParentId="22073" CreationDate="2017-08-10T00:35:30.000" Score="7" Body="&lt;p&gt;Scaling in batch normalization is said not to be required because of the nature of ReLU. Although it is a non-linear operation, a multiplication with a positive scalar $\gamma$ over the values of a vector $x$ which are already normalized and centered to the trained bias $\beta$ makes no influence on its non-linear outcome: for $\gamma &amp;gt; 0$, $max(0, \gamma \times x) = \gamma \times max(0, x)$. In fact, the extra parameter adds no representative power in the model, for the same reason that we must include non-linearities between other neuron layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so, the weights of the following layer would scale the values to fit accordingly. We could have $z = Wgx$, but that would be the same thing as $z = Wx$ if $W$ is scaled by $g$. The same reasoning can't be made for other non-linearities, such as the sigmoid or tanh, which have a saturating effect that is clearly influenced by a scalar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Curiously, the official documentation on TensorFlow's &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;tf.contrib.layers.batch_norm&lt;/code&gt;&lt;/a&gt; makes the same recommendation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;strong&gt;scale&lt;/strong&gt;: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. &lt;code&gt;nn.relu&lt;/code&gt;), this can be disabled since the scaling can be done by the next layer.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="17538" LastEditorUserId="17538" LastEditDate="2017-08-10T09:21:04.950" LastActivityDate="2017-08-10T09:21:04.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22128" PostTypeId="1" AcceptedAnswerId="22512" CreationDate="2017-08-10T04:38:09.490" Score="0" ViewCount="348" Body="&lt;p&gt;Currently I have a list of Books which I need to compare to a different list.&#xA;For example in my local list I have:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; - Harry Potter and the Philosopher's Stone | J.K. Rowling&#xA; - Harry Potter and the Deathly Hallows | J.K. Rowling&#xA; - Harry Potter and the Goblet of Fire | J.K. Rowling&#xA; - Harry Potter and the Prisoner of Azkaban | J.K. Rowling&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I compare I end up having a simple operation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Book Title + Author (local list) vs [Book Title + Author] (remote list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm currently using &lt;a href=&quot;https://stackoverflow.com/questions/6690739/fuzzy-string-comparison-in-python-confused-with-which-library-to-use&quot;&gt;Levenshtein&lt;/a&gt; distance, based on that I can find out if Book is in remote system or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case if Book1 vs Book2 result score exceeds X, I assume is the same Book.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;levenshtein(Book1 Title + Author, Book2 Title + Author)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I read about &lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word Embeddings&lt;/a&gt; and before I start coding want to check if converting my text information to embeddings may improve my results.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2017-08-23T12:18:19.673" Title="Text comparison using word embeddings" Tags="|nlp|word-embeddings|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22129" PostTypeId="2" ParentId="22118" CreationDate="2017-08-10T05:09:20.137" Score="6" Body="&lt;p&gt;The short answer is that when a net is &lt;strong&gt;very&lt;/strong&gt; deep it becomes very difficult for gradients to propagate backwards all the way. Skip connections offer &quot;short cuts&quot; for gradients to propagate further and allow for efficient training of very deep nets.&lt;/p&gt;&#xA;" OwnerUserId="37752" LastActivityDate="2017-08-10T05:09:20.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22131" PostTypeId="2" ParentId="12872" CreationDate="2017-08-10T06:15:46.270" Score="3" Body="&lt;p&gt;Word2vec is a good starting point for most scenarios. It &lt;em&gt;does&lt;/em&gt; capture semantics by way of prediction using CBOW method. It allows translations (as most repeated example I can put here again), V(King) - V(Queen) ~~ V(men) - V(women) and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is the problem? The issue lies in word sense ambiguity. Whenever the word itself has two different meaning in two different context, the word vector will tend to really be away from either context. Python ~ Boa (both snakes) and Python - Java (both programming languages)..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any alternative?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the very specific purpose of &quot;synonyms&quot; if you want &lt;a href=&quot;https://wordnet.princeton.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wordnet&lt;/a&gt; would be ideal place. It captures explicit relationship of two words rather than implicit relation based on usage and occurrences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wordnet is mostly crafted as a dictionary - where as word2vec is mined by usage. &lt;/p&gt;&#xA;" OwnerUserId="30536" LastActivityDate="2017-08-10T06:15:46.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22132" PostTypeId="2" ParentId="22118" CreationDate="2017-08-10T06:39:51.840" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why do we need for Shortcut Connections to build Residual Networks,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Because otherwise the network would not be a residual network.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;how [do residual connections] help to train neural networks for classification and detection?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They add shortcuts for the gradient. This way the first few layers get updates very quick, the vanishing gradient problem is no longer a problem. So you can easily make networks with a 1000 layers. See &lt;a href=&quot;https://arxiv.org/abs/1512.03385&quot; rel=&quot;nofollow noreferrer&quot;&gt;Residual Networks paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing specific about classification / detection. Its about very deep networks.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-08-10T09:18:42.043" LastActivityDate="2017-08-10T09:18:42.043" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22136" PostTypeId="1" AcceptedAnswerId="23358" CreationDate="2017-08-10T10:32:46.393" Score="3" ViewCount="322" Body="&lt;p&gt;I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I  need to  consider sales for period wise i.e week  .While aggregating I am facing  problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;on differnt days item has different discounts.&#xA;Case1: Some items are having large variations in discounts like(0% and 50%) &#xA;Case2: some are having small variation like(40% and 49%).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-In case1 If we aggregate the discount and divide by sales quantity then there will be  decrease of  discount in one record and  increase in another.&#xA;- In case2 If we aggregate discounts and take average of that .It seems there is no large variations in discount by taking average.Which is not correct for first case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my data Sales is more sensitive to discounts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to get generalized discount to predict sales(with Random forest algorithm)?&lt;/p&gt;&#xA;" OwnerUserId="37604" LastActivityDate="2017-09-28T05:56:10.490" Title="Aggregation of Discount" Tags="|machine-learning|random-forest|data-cleaning|aggregation|forecasting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22137" PostTypeId="1" AcceptedAnswerId="22151" CreationDate="2017-08-10T10:33:12.267" Score="5" ViewCount="3751" Body="&lt;p&gt;In &lt;a href=&quot;https://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper related to factorization machine, the author compares factorization machine (FM) with SVM. As FM performs better than SVM, it's considered state of the art for sparse data. Why SVM is considered so important algorithm for sparse dataset? What features of SVM makes it superior for sparse data?&lt;/p&gt;&#xA;" OwnerUserId="37760" LastEditorUserId="8432" LastEditDate="2017-08-12T10:13:45.210" LastActivityDate="2022-02-26T07:53:06.957" Title="SVM on sparse data" Tags="|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22138" PostTypeId="1" CreationDate="2017-08-10T11:00:00.873" Score="-1" ViewCount="542" Body="&lt;p&gt;I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having major difficulties visualising the data. By visualisation I mean, generating aesthetic plots. The data has attributes such as node number, name, total money going in, total money going out and edges. I'd like to use such attributes to help generate colourful plots that signify groups and communities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already tried NetworkX, Pandas, Neo4J, Gephi and R-Studio. All of which crash and are unable to do anything on my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please recommend alternatives to create such visualisations?&lt;/p&gt;&#xA;" OwnerUserId="37727" LastActivityDate="2017-08-10T12:06:58.227" Title="How to visualise very large graphs with 250M nodes and 500M+ edges?" Tags="|dataset|bigdata|visualization|data|" AnswerCount="1" CommentCount="3" ClosedDate="2017-08-12T10:13:27.930" ContentLicense="CC BY-SA 3.0" />
  <row Id="22139" PostTypeId="2" ParentId="22138" CreationDate="2017-08-10T12:06:58.227" Score="1" Body="&lt;p&gt;I'll assume you want an image that you could display on a standard 1920x1200 monitor. If every node was PERFECTLY packed and represented by only one pixel, you would have a white box that covered only 1% (2.3 million) of the nodes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Subset your data to display to something reasonable like 1000 nodes using Neo4j and then you should get more reasonable performance.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-10T12:06:58.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22140" PostTypeId="1" CreationDate="2017-08-10T12:10:07.340" Score="-1" ViewCount="173" Body="&lt;p&gt;I have a question regarding an idea. I know many ML algorithms and I know how they work and perform. But I have an idea which I believe in should be similar to one of the current available methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is there are 2 clusters of points in space with n dimensions (n large). Based on the context we expect the clusters to be fairly separated. Now the question is I need an algorithm which takes the data, number of clusters, and 1 sample per clusters as input, and then tries to cluster the rest of the data. The main feature here is the fact that the algorithm starts with 1 sample per class, and then it must go on finding very similar points first and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any idea which algorithm or method is the most similar method to the thing I'm looking for?&lt;/p&gt;&#xA;" OwnerUserId="37761" LastActivityDate="2017-08-10T15:16:54.590" Title="Learning to cluster using one sample per class" Tags="|svm|learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22141" PostTypeId="2" ParentId="22140" CreationDate="2017-08-10T12:27:40.907" Score="0" Body="&lt;p&gt;You should take a look at &lt;strong&gt;hierarchial clustering&lt;/strong&gt;, this is similar to what you want. Normally they start with either all the samples as their own clusters and then recursively find two clusters that when merged reduce the intercluster distance the least (some kind of metric that you can define) or the other way around where you have one cluster and you split a cluster in a way that increases the intercluster distance the most. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could do is start with your samples as individual clusters and keep adding the sample that is closest to one of your clusters greedily. This is also similar to &lt;strong&gt;label propagation&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way is to base it on &lt;strong&gt;k-means&lt;/strong&gt; by taking all the samples as cluster centers, assign all your other points to the closest centers and update your cluster centers. A problem with this approach is that your original cluster samples could actually end up in another cluster, which you would need to either constrain in some way or accept.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-10T12:27:40.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22142" PostTypeId="2" ParentId="22140" CreationDate="2017-08-10T12:41:50.630" Score="0" Body="&lt;p&gt;If you have 1 sample per cluster &lt;strong&gt;as input&lt;/strong&gt;, then you are &lt;strong&gt;not facing a clustering problem&lt;/strong&gt;, but a &lt;strong&gt;classification&lt;/strong&gt; one, where your training dataset is comprised of 1 sample per class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the setup you described, the most direct approach would be 1-nearest neighbor, that is, &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-nearest neighbors algorithm&lt;/a&gt; with $k=1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think any other machine learning classification algorithm you may try on your training dataset would probably end up with the similar results as 1-nn. Anyway, if you don't have have a test dataset, you cannot assess the &lt;em&gt;correctness&lt;/em&gt; of your &lt;em&gt;model&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-08-10T15:16:54.590" LastActivityDate="2017-08-10T15:16:54.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22143" PostTypeId="1" AcceptedAnswerId="22150" CreationDate="2017-08-10T12:58:56.223" Score="0" ViewCount="93" Body="&lt;p&gt;I have the following data set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(student=c(1,2,3,4,5,6,7,8,9), sat=c(365,0,545,630,385,410,0,655,0), act=c(28,20,0,0,16,17,35,29,21))&#xA;&#xA;student     sat    act&#xA;    1       365    28&#xA;    2         0    20&#xA;    3       545     0&#xA;    4       630     0&#xA;    5       385    16&#xA;    6       410    17&#xA;    7         0    35&#xA;    8       655    29&#xA;    9         0    21&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I'd like to create a new field with the following conditions&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is an SAT score &gt; 0 use SAT score&#xA;If SAT=0, then convert the ACT to an SAT score using the rubric &lt;a href=&quot;https://www.tntech.edu/admissions/actsat&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. (When there was a range in the SAT score, I just used the median. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ACT SAT&#xA;8   200&#xA;9   210&#xA;10  220&#xA;11  225&#xA;12  250&#xA;13  285&#xA;14  325&#xA;15  360&#xA;16  385&#xA;17  410&#xA;18  440&#xA;19  465&#xA;20  485&#xA;21  505&#xA;22  525&#xA;23  545&#xA;24  560&#xA;25  575&#xA;26  595&#xA;27  615&#xA;28  635&#xA;29  655&#xA;30  675&#xA;31  700&#xA;32  725&#xA;33  750&#xA;34  775&#xA;35  790&#xA;36  800&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is one heck of an ifelse statement. I've tried this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df$newgrade=-ifelse(ACT=8,200, ifelse (ACT=9,210, ifelse(ACT=10,220, ifelse (ACT=11,225, ACT=12,250, ifelse(ACT=13,285, ifelse (ACT=14,325, ACT=15,D, ifelse(ACT=16,C, ifelse (ACT=17,B, ACT=18,D, ifelse(ACT=19,C, ifelse (ACT=20,B, ACT=21,D, ifelse(ACT=22,C, ifelse (ACT=23,B, ACT=24,D, ifelse(ACT=25,C, ifelse (ACT=26,B, ACT=27,D, ifelse(ACT=28,C, ifelse (ACT=29,B, ACT=30,D, ifelse(ACT=31,C, ifelse (ACT=32,B, ACT=33,D, ifelse(ACT=34,C, ifelse (ACT=35,B, ACT=36,D))))))))))))))))))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to follow the example at the bottom of this page but it didn't work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone else on another board suggested: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df$newgrade&amp;lt;-ifelse(df$sat == 0, conversion$SAT[match(df$act, conversion$ACT)], df$sat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but then a new issue presented itself:&#xA;If there is neither an ACT nor a SAT score. How can it put a 0 in for that group??&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any assistance you may bring.&lt;/p&gt;&#xA;" OwnerUserId="37770" LastActivityDate="2017-08-11T05:32:58.580" Title="Nested IF Else in R - SAT/ACT test" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22145" PostTypeId="1" CreationDate="2017-08-10T13:52:01.700" Score="2" ViewCount="1044" Body="&lt;p&gt;I am very interested in deep learning, and I was recommended to read &lt;a href=&quot;http://web.stanford.edu/~hastie/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;The Elements of Statistical Learning&lt;/em&gt;&lt;/a&gt;, but it is very hard to get into. So my question is :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How important is learning statistics and reading books like elements of statistical learning, if my end goal is to learn deep learning? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should have mentioned that I have already done Andrew Ng's Machine Learning course.&lt;/p&gt;&#xA;" OwnerUserId="20327" LastEditorUserId="20327" LastEditDate="2018-01-28T08:38:47.120" LastActivityDate="2018-01-28T08:38:47.120" Title="Is statistics a pre-requisite to learn deep learning?" Tags="|deep-learning|" AnswerCount="3" CommentCount="4" ClosedDate="2017-08-12T10:13:18.877" ContentLicense="CC BY-SA 3.0" />
  <row Id="22146" PostTypeId="2" ParentId="22145" CreationDate="2017-08-10T14:13:50.740" Score="5" Body="&lt;p&gt;I would not call it a pre-requisite, but I would say reading books the that is necessary for knowing it all very well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you to start with Andrew Ng's Machine Learning course &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;on coursera&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you are familiar with that, move on to deep learning videos:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nando de Freitas&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=SGZ6BttHMPw&amp;amp;list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hugo Larochelle&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;While you are going through topics one after the other, go back to the books and read what they have to offer. Before you know, you would have done it all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="37771" LastEditorUserId="23305" LastEditDate="2017-08-10T15:32:19.847" LastActivityDate="2017-08-10T15:32:19.847" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22147" PostTypeId="2" ParentId="22145" CreationDate="2017-08-10T14:21:10.793" Score="4" Body="&lt;p&gt;If you don't like to start with statistics and bias variance trade off.I suggest you to go through &lt;/p&gt;&#xA;&#xA;&lt;p&gt;with Andrew Ng's class after that you can proceed to &#xA;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Networks and deep learning book&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a good starter for Deep Learning&lt;/p&gt;&#xA;" OwnerUserId="37626" LastEditorUserId="37626" LastEditDate="2017-08-14T02:28:27.330" LastActivityDate="2017-08-14T02:28:27.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22150" PostTypeId="2" ParentId="22143" CreationDate="2017-08-10T15:31:22.880" Score="1" Body="&lt;p&gt;You can indeed use the conversion table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conversion &amp;lt;- read.table(text = &quot;ACT SAT&#xA;8   200&#xA;9   210&#xA;10  220&#xA;11  225&#xA;12  250&#xA;13  285&#xA;14  325&#xA;15  360&#xA;16  385&#xA;17  410&#xA;18  440&#xA;19  465&#xA;20  485&#xA;21  505&#xA;22  525&#xA;23  545&#xA;24  560&#xA;25  575&#xA;26  595&#xA;27  615&#xA;28  635&#xA;29  655&#xA;30  675&#xA;31  700&#xA;32  725&#xA;33  750&#xA;34  775&#xA;35  790&#xA;36  800&quot;, header = TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With the help of this table and mathematical/logical operators, you can create the values:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;transform(df, newgrade = (sat | act) * &#xA;            (conversion$SAT[match(df$act, conversion$ACT)] ^&#xA;               as.logical(act) * !sat) + sat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   student sat act newgrade&#xA;1        1 365  28      365&#xA;2        2   0  20      485&#xA;3        3 545   0      545&#xA;4        4 630   0      630&#xA;5        5 385  16      385&#xA;6        6 410  17      410&#xA;7        7   0  35      790&#xA;8        8 655  29      655&#xA;9        9   0  21      505&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The value in &lt;code&gt;newgrade&lt;/code&gt; will also be 0 if both &lt;code&gt;sat&lt;/code&gt; and &lt;code&gt;act&lt;/code&gt; are 0.&lt;/p&gt;&#xA;" OwnerUserId="106" LastEditorUserId="106" LastEditDate="2017-08-11T05:32:58.580" LastActivityDate="2017-08-11T05:32:58.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22151" PostTypeId="2" ParentId="22137" CreationDate="2017-08-10T16:24:44.103" Score="8" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;Support Vector Machines&lt;/a&gt; (SVM) represent data examples as points in space and tries to create a mapping with a wide as possible gap between the separate categories. The data examples closest to the gap are called &quot;support vectors&quot;. Those support vectors define the SVM classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are able to identify the support vectors, you can ignore all other data points and dimensions. Thus SVM can efficiently handle sparse data.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-08-02T16:14:14.897" LastActivityDate="2018-08-02T16:14:14.897" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22153" PostTypeId="1" CreationDate="2017-08-10T16:32:23.513" Score="3" ViewCount="1778" Body="&lt;p&gt;I have a rectangular numeric dataset, and I'm applying a multilayer perceptron to it. I'm having success, but I'm now looking to see what other architectures I can apply.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much of deep learning seems applied to loosely-structured data -- sequences, text, images -- and everyone is having a lot of fun working with a variety of interesting models...at least when they have a problem that fits these models.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about basic, row/column datasets. What are some of the canonical models to be used with this kind of data, apart from tweaking the layers of a basic MLP?  &lt;/p&gt;&#xA;" OwnerUserId="15533" LastActivityDate="2017-08-14T13:16:03.390" Title="What are alternatives to MLP, when you have rectangular, structured data?" Tags="|deep-learning|model-selection|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="22154" PostTypeId="2" ParentId="22153" CreationDate="2017-08-10T17:19:46.237" Score="3" Body="&lt;p&gt;Fancy deep learning architectures mostly work by exploiting structure in your data. Permutation invariance or equivariance, temporal structure and spatial structure come to mind. Maybe some features are based on a set of the same objects? Most of the benefits come from sharing weights and learning a shared representation. Another potential benefit is the flexibility in output, although that is less relevant in your case. If you have a lot more labeled data on another task with the same features, you could pretrain a larger model on that task and then fine tune that network on your current task. For the rest it is difficult to say without showing some examples.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-10T17:19:46.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22163" PostTypeId="1" CreationDate="2017-08-11T01:11:51.120" Score="10" ViewCount="4256" Body="&lt;p&gt;My Q-Learning algorithm's state values keep on diverging to infinity, which means my weights are diverging too. I use a neural network for my value-mapping. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Clipping the &quot;reward + discount * maximum value of action&quot; (max/min set to 50/-50)&lt;/li&gt;&#xA;&lt;li&gt;Setting a low learning rate (0.00001 and I use the classic Backpropagation for updating the weights)&lt;/li&gt;&#xA;&lt;li&gt;Decreasing the values of the rewards&lt;/li&gt;&#xA;&lt;li&gt;Increasing the exploration rate&lt;/li&gt;&#xA;&lt;li&gt;Normalizing the inputs to between 1~100 (previously it was 0~1)&lt;/li&gt;&#xA;&lt;li&gt;Change the discount rate&lt;/li&gt;&#xA;&lt;li&gt;Decrease the layers of the neural network (just for validation)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've heard that Q Learning is known to diverge on non-linear input, but are there anything else that I can try to stop the divergence of the weights?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update #1 on August 14th, 2017:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've decided to add some specific details on what I'm doing right now due to a request to. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently trying to make an agent learn how to fight in a top-down view of a shooting game. The opponent is a simple bot which moves stochastically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Each character has 9 actions to choose from on each turn:&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;move up&lt;/li&gt;&#xA;&lt;li&gt;move down&lt;/li&gt;&#xA;&lt;li&gt;move left&lt;/li&gt;&#xA;&lt;li&gt;move right&lt;/li&gt;&#xA;&lt;li&gt;shoot a bullet upwards&lt;/li&gt;&#xA;&lt;li&gt;shoot a bullet downwards&lt;/li&gt;&#xA;&lt;li&gt;shoot a bullet to the left&lt;/li&gt;&#xA;&lt;li&gt;shoot a bullet to the right&lt;/li&gt;&#xA;&lt;li&gt;do nothing&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The rewards are:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if agent hits the bot with a bullet, +100 (I've tried many different values)&lt;/li&gt;&#xA;&lt;li&gt;if agent gets hit by a bullet shot by the bot, -50 (again, I've tried many different values)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if the agent tries to fire a bullet while bullets can't be fired(ex. when the agent just fired a bullet, etc. ), -25(Not necessary but I wanted the agent to be more efficient)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if the bot tries to go out of the arena, -20(Not necessary too but I wanted the agent to be more efficient)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The inputs for the neural network are:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Distance between the agent and the bot on the X axis normalized to 0~100&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Distance between the agent and the bot on the Y axis normalized to 0~100&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Agent's x and y positions&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Bot's x and y positions&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Bot's bullet position. If the bot didn't fire a bullet, the parameters are set to the x and y positions of the bot. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've also fiddled with the inputs too; I tried adding new features like the x value of the agent's position(not the distance but the actual position)and the position of the bot's bullet. None of them worked. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pygame import *&#xA;from pygame.locals import *&#xA;import sys&#xA;from time import sleep&#xA;import numpy as np&#xA;import random&#xA;import tensorflow as tf&#xA;from pylab import savefig&#xA;from tqdm import tqdm&#xA;&#xA;&#xA;#Screen Setup&#xA;disp_x, disp_y = 1000, 800&#xA;arena_x, arena_y = 1000, 800&#xA;border = 4; border_2 = 1&#xA;&#xA;#Color Setup&#xA;white = (255, 255, 255); aqua= (0, 200, 200)&#xA;red = (255, 0, 0); green = (0, 255, 0)&#xA;blue = (0, 0, 255); black = (0, 0, 0)&#xA;green_yellow = (173, 255, 47); energy_blue = (125, 249, 255)&#xA;&#xA;#Initialize character positions&#xA;init_character_a_state = [disp_x/2 - arena_x/2 + 50, disp_y/2 - arena_y/2 + 50]&#xA;init_character_b_state = [disp_x/2 + arena_x/2 - 50, disp_y/2 + arena_y/2 - 50]&#xA;&#xA;#Setup character dimentions&#xA;character_size = 50&#xA;character_move_speed = 25&#xA;&#xA;#Initialize character stats&#xA;character_init_health = 100&#xA;&#xA;#initialize bullet stats&#xA;beam_damage = 10&#xA;beam_width = 10&#xA;beam_ob = -100&#xA;&#xA;#The Neural Network&#xA;input_layer = tf.placeholder(shape=[1,7],dtype=tf.float32)&#xA;weight_1 = tf.Variable(tf.random_uniform([7,9],0,0.1))&#xA;#weight_2 = tf.Variable(tf.random_uniform([6,9],0,0.1))&#xA;&#xA;#The calculations, loss function and the update model&#xA;Q = tf.matmul(input_layer, weight_1)&#xA;predict = tf.argmax(Q, 1)&#xA;next_Q = tf.placeholder(shape=[1,9],dtype=tf.float32)&#xA;loss = tf.reduce_sum(tf.square(next_Q - Q))&#xA;trainer = tf.train.GradientDescentOptimizer(learning_rate=0.001)&#xA;updateModel = trainer.minimize(loss)&#xA;&#xA;initialize = tf.global_variables_initializer()&#xA;&#xA;jList = []&#xA;rList = []&#xA;&#xA;init()&#xA;font.init()&#xA;myfont = font.SysFont('Comic Sans MS', 15)&#xA;myfont2 = font.SysFont('Comic Sans MS', 150)&#xA;myfont3 = font.SysFont('Gothic', 30)&#xA;disp = display.set_mode((disp_x, disp_y), 0, 32)&#xA;&#xA;#CHARACTER/BULLET PARAMETERS&#xA;agent_x = agent_y = int()&#xA;bot_x = bot_y = int()&#xA;agent_hp = bot_hp = int()&#xA;bot_beam_dir = int()&#xA;agent_beam_fire = bot_beam_fire = bool()&#xA;agent_beam_x = bot_beam_x = agent_beam_y = bot_beam_y = int()&#xA;agent_beam_size_x = agent_beam_size_y = bot_beam_size_x = bot_beam_size_y = int()&#xA;bot_current_action = agent_current_action = int()&#xA;&#xA;def param_init():&#xA;    &quot;&quot;&quot;Initializes parameters&quot;&quot;&quot;&#xA;    global agent_x, agent_y, bot_x, bot_y, agent_hp, bot_hp, agent_beam_fire, bot_beam_fire, agent_beam_x, bot_beam_x, agent_beam_y, bot_beam_y&#xA;&#xA;    agent_x = list(init_character_a_state)[0]; agent_y = list(init_character_a_state)[1]&#xA;    bot_x = list(init_character_b_state)[0]; bot_y = list(init_character_b_state)[1]&#xA;    agent_hp = bot_hp = character_init_health&#xA;    agent_beam_fire = bot_beam_fire = False&#xA;    agent_beam_x = bot_beam_x = agent_beam_y = bot_beam_y = beam_ob&#xA;    agent_beam_size_x = agent_beam_size_y = bot_beam_size_x = bot_beam_size_y = 0&#xA;&#xA;&#xA;def screen_blit():&#xA;    global disp, disp_x, disp_y, arena_x, arena_y, border, border_2, character_size, agent_x, \&#xA;    agent_y, bot_x, bot_y, character_init_health, agent_hp, bot_hp, red, blue, aqua, green, black, green_yellow, energy_blue, \&#xA;    agent_beam_fire, bot_beam_fire, agent_beam_x, agent_beam_y, bot_beam_x, bot_beam_y, agent_beam_size_x, agent_beam_size_y, bot_beam_size_x, bot_beam_size_y, beam_width&#xA;&#xA;    disp.fill(aqua)&#xA;    draw.rect(disp, black, (disp_x / 2 - arena_x / 2 - border, disp_y /&#xA;                            2 - arena_y / 2 - border, arena_x + border * 2, arena_y + border * 2))&#xA;    draw.rect(disp, green, (disp_x / 2 - arena_x / 2,&#xA;                            disp_y / 2 - arena_y / 2, arena_x, arena_y))&#xA;    if bot_beam_fire == True:&#xA;        draw.rect(disp, green_yellow, (agent_beam_x, agent_beam_y, agent_beam_size_x, agent_beam_size_y))&#xA;        bot_beam_fire = False&#xA;    if agent_beam_fire == True:&#xA;        draw.rect(disp, energy_blue, (bot_beam_x, bot_beam_y, bot_beam_size_x, bot_beam_size_y))&#xA;        agent_beam_fire = False&#xA;&#xA;    draw.rect(disp, red, (agent_x, agent_y, character_size, character_size))&#xA;    draw.rect(disp, blue, (bot_x, bot_y, character_size, character_size))&#xA;&#xA;    draw.rect(disp, red, (disp_x / 2 - 200, disp_y / 2 + arena_y / 2 +&#xA;                            border + 1, float(agent_hp) / float(character_init_health) * 100, 14))&#xA;    draw.rect(disp, blue, (disp_x / 2 + 200, disp_y / 2 + arena_y / 2 +&#xA;                            border + 1, float(bot_hp) / float(character_init_health) * 100, 14))&#xA;&#xA;&#xA;def bot_take_action():&#xA;    return random.randint(1, 9)&#xA;&#xA;def beam_hit_detector(player):&#xA;    global agent_x, agent_y, bot_x, bot_y, agent_beam_fire, bot_beam_fire, agent_beam_x, \&#xA;    bot_beam_x, agent_beam_y, bot_beam_y, agent_beam_size_x, agent_beam_size_y, \&#xA;    bot_beam_size_x, bot_beam_size_y, bot_current_action, agent_current_action, beam_width, character_size&#xA;&#xA;    if player == &quot;bot&quot;:&#xA;        if bot_current_action == 1:&#xA;            if disp_y/2 - arena_y/2 &amp;lt;= agent_y &amp;lt;= bot_y and (agent_x &amp;lt; bot_beam_x + beam_width &amp;lt; agent_x + character_size or agent_x &amp;lt; bot_beam_x &amp;lt; agent_x + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif bot_current_action == 2:&#xA;            if bot_x &amp;lt;= agent_x &amp;lt;= disp_x/2 + arena_x/2 and (agent_y &amp;lt; bot_beam_y + beam_width &amp;lt; agent_y + character_size or agent_y &amp;lt; bot_beam_y &amp;lt; agent_y + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif bot_current_action == 3:&#xA;            if bot_y &amp;lt;= agent_y &amp;lt;= disp_y/2 + arena_y/2 and (agent_x &amp;lt; bot_beam_x + beam_width &amp;lt; agent_x + character_size or agent_x &amp;lt; bot_beam_x &amp;lt; agent_x + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif bot_current_action == 4:&#xA;            if disp_x/2 - arena_x/2 &amp;lt;= agent_x &amp;lt;= bot_x and (agent_y &amp;lt; bot_beam_y + beam_width &amp;lt; agent_y + character_size or agent_y &amp;lt; bot_beam_y &amp;lt; agent_y + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;    else:&#xA;        if agent_current_action == 1:&#xA;            if disp_y/2 - arena_y/2 &amp;lt;= bot_y &amp;lt;= agent_y and (bot_x &amp;lt; agent_beam_x + beam_width &amp;lt; bot_x + character_size or bot_x &amp;lt; agent_beam_x &amp;lt; bot_x + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif agent_current_action == 2:&#xA;            if agent_x &amp;lt;= bot_x &amp;lt;= disp_x/2 + arena_x/2 and (bot_y &amp;lt; agent_beam_y + beam_width &amp;lt; bot_y + character_size or bot_y &amp;lt; agent_beam_y &amp;lt; bot_y + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif agent_current_action == 3:&#xA;            if agent_y &amp;lt;= bot_y &amp;lt;= disp_y/2 + arena_y/2 and (bot_x &amp;lt; agent_beam_x + beam_width &amp;lt; bot_x + character_size or bot_x &amp;lt; agent_beam_x &amp;lt; bot_x + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;        elif bot_current_action == 4:&#xA;            if disp_x/2 - arena_x/2 &amp;lt;= bot_x &amp;lt;= agent_x and (bot_y &amp;lt; agent_beam_y + beam_width &amp;lt; bot_y + character_size or bot_y &amp;lt; agent_beam_y &amp;lt; bot_y + character_size):&#xA;                return True&#xA;            else:&#xA;                return False&#xA;&#xA;&#xA;def mapping(maximum, number):&#xA;    return number#int(number * maximum)&#xA;&#xA;def action(agent_action, bot_action):&#xA;    global agent_x, agent_y, bot_x, bot_y, agent_hp, bot_hp, agent_beam_fire, \&#xA;    bot_beam_fire, agent_beam_x, bot_beam_x, agent_beam_y, bot_beam_y, agent_beam_size_x, \&#xA;    agent_beam_size_y, bot_beam_size_x, bot_beam_size_y, beam_width, agent_current_action, bot_current_action, character_size&#xA;&#xA;    agent_current_action = agent_action; bot_current_action = bot_action&#xA;    reward = 0; cont = True; successful = False; winner = &quot;&quot;&#xA;    if 1 &amp;lt;= bot_action &amp;lt;= 4:&#xA;        bot_beam_fire = True&#xA;        if bot_action == 1:&#xA;            bot_beam_x = bot_x + character_size/2 - beam_width/2; bot_beam_y = disp_y/2 - arena_y/2&#xA;            bot_beam_size_x = beam_width; bot_beam_size_y = bot_y - disp_y/2 + arena_y/2&#xA;        elif bot_action == 2:&#xA;            bot_beam_x = bot_x + character_size; bot_beam_y = bot_y + character_size/2 - beam_width/2&#xA;            bot_beam_size_x = disp_x/2 + arena_x/2 - bot_x - character_size; bot_beam_size_y = beam_width&#xA;        elif bot_action == 3:&#xA;            bot_beam_x = bot_x + character_size/2 - beam_width/2; bot_beam_y = bot_y + character_size&#xA;            bot_beam_size_x = beam_width; bot_beam_size_y = disp_y/2 + arena_y/2 - bot_y - character_size&#xA;        elif bot_action == 4:&#xA;            bot_beam_x = disp_x/2 - arena_x/2; bot_beam_y = bot_y + character_size/2 - beam_width/2&#xA;            bot_beam_size_x = bot_x - disp_x/2 + arena_x/2; bot_beam_size_y = beam_width&#xA;&#xA;    elif 5 &amp;lt;= bot_action &amp;lt;= 8:&#xA;        if bot_action == 5:&#xA;            bot_y -= character_move_speed&#xA;            if bot_y &amp;lt;= disp_y/2 - arena_y/2:&#xA;                bot_y = disp_y/2 - arena_y/2&#xA;            elif agent_y &amp;lt;= bot_y &amp;lt;= agent_y + character_size:&#xA;                bot_y = agent_y + character_size&#xA;        elif bot_action == 6:&#xA;            bot_x += character_move_speed&#xA;            if bot_x &amp;gt;= disp_x/2 + arena_x/2 - character_size:&#xA;                bot_x = disp_x/2 + arena_x/2 - character_size&#xA;            elif agent_x &amp;lt;= bot_x + character_size &amp;lt;= agent_x + character_size:&#xA;                bot_x = agent_x - character_size&#xA;        elif bot_action == 7:&#xA;            bot_y += character_move_speed&#xA;            if bot_y + character_size &amp;gt;= disp_y/2 + arena_y/2:&#xA;                bot_y = disp_y/2 + arena_y/2 - character_size&#xA;            elif agent_y &amp;lt;= bot_y + character_size &amp;lt;= agent_y + character_size:&#xA;                bot_y = agent_y - character_size&#xA;        elif bot_action == 8:&#xA;            bot_x -= character_move_speed&#xA;            if bot_x &amp;lt;= disp_x/2 - arena_x/2:&#xA;                bot_x = disp_x/2 - arena_x/2&#xA;            elif agent_x &amp;lt;= bot_x &amp;lt;= agent_x + character_size:&#xA;                bot_x = agent_x + character_size&#xA;&#xA;    if bot_beam_fire == True:&#xA;        if beam_hit_detector(&quot;bot&quot;):&#xA;            #print &quot;Agent Got Hit!&quot;&#xA;            agent_hp -= beam_damage&#xA;            reward += -50&#xA;            bot_beam_size_x = bot_beam_size_y = 0&#xA;            bot_beam_x = bot_beam_y = beam_ob&#xA;            if agent_hp &amp;lt;= 0:&#xA;                cont = False&#xA;                winner = &quot;Bot&quot;&#xA;&#xA;    if 1 &amp;lt;= agent_action &amp;lt;= 4:&#xA;        agent_beam_fire = True&#xA;        if agent_action == 1:&#xA;            if agent_y &amp;gt; disp_y/2 - arena_y/2:&#xA;                agent_beam_x = agent_x - beam_width/2; agent_beam_y = disp_y/2 - arena_y/2&#xA;                agent_beam_size_x = beam_width; agent_beam_size_y = agent_y - disp_y/2 + arena_y/2&#xA;            else:&#xA;                reward += -25&#xA;        elif agent_action == 2:&#xA;            if agent_x + character_size &amp;lt; disp_x/2 + arena_x/2:&#xA;                agent_beam_x = agent_x + character_size; agent_beam_y = agent_y + character_size/2 - beam_width/2&#xA;                agent_beam_size_x = disp_x/2 + arena_x/2 - agent_x - character_size; agent_beam_size_y = beam_width&#xA;            else:&#xA;                reward += -25&#xA;        elif agent_action == 3:&#xA;            if agent_y + character_size &amp;lt; disp_y/2 + arena_y/2:&#xA;                agent_beam_x = agent_x + character_size/2 - beam_width/2; agent_beam_y = agent_y + character_size&#xA;                agent_beam_size_x = beam_width; agent_beam_size_y = disp_y/2 + arena_y/2 - agent_y - character_size&#xA;            else:&#xA;                reward += -25&#xA;        elif agent_action == 4:&#xA;            if agent_x &amp;gt; disp_x/2 - arena_x/2:&#xA;                agent_beam_x = disp_x/2 - arena_x/2; agent_beam_y = agent_y + character_size/2 - beam_width/2&#xA;                agent_beam_size_x = agent_x - disp_x/2 + arena_x/2; agent_beam_size_y = beam_width&#xA;            else:&#xA;                reward += -25&#xA;&#xA;    elif 5 &amp;lt;= agent_action &amp;lt;= 8:&#xA;        if agent_action == 5:&#xA;            agent_y -= character_move_speed&#xA;            if agent_y &amp;lt;= disp_y/2 - arena_y/2:&#xA;                agent_y = disp_y/2 - arena_y/2&#xA;                reward += -5&#xA;            elif bot_y &amp;lt;= agent_y &amp;lt;= bot_y + character_size and bot_x &amp;lt;= agent_x &amp;lt;= bot_x + character_size:&#xA;                agent_y = bot_y + character_size&#xA;                reward += -2&#xA;        elif agent_action == 6:&#xA;            agent_x += character_move_speed&#xA;            if agent_x + character_size &amp;gt;= disp_x/2 + arena_x/2:&#xA;                agent_x = disp_x/2 + arena_x/2 - character_size&#xA;                reward += -5&#xA;            elif bot_x &amp;lt;= agent_x + character_size &amp;lt;= bot_x + character_size and bot_y &amp;lt;= agent_y &amp;lt;= bot_y + character_size:&#xA;                agent_x = bot_x - character_size&#xA;                reward += -2&#xA;        elif agent_action == 7:&#xA;            agent_y += character_move_speed&#xA;            if agent_y + character_size &amp;gt;= disp_y/2 + arena_y/2:&#xA;                agent_y = disp_y/2 + arena_y/2 - character_size&#xA;                reward += -5&#xA;            elif bot_y &amp;lt;= agent_y + character_size &amp;lt;= bot_y + character_size and bot_x &amp;lt;= agent_x &amp;lt;= bot_x + character_size:&#xA;                agent_y = bot_y - character_size&#xA;                reward += -2&#xA;        elif agent_action == 8:&#xA;            agent_x -= character_move_speed&#xA;            if agent_x &amp;lt;= disp_x/2 - arena_x/2:&#xA;                agent_x = disp_x/2 - arena_x/2&#xA;                reward += -5&#xA;            elif bot_x &amp;lt;= agent_x &amp;lt;= bot_x + character_size and bot_y &amp;lt;= agent_y &amp;lt;= bot_y + character_size:&#xA;                agent_x = bot_x + character_size&#xA;                reward += -2&#xA;    if agent_beam_fire == True:&#xA;        if beam_hit_detector(&quot;agent&quot;):&#xA;            #print &quot;Bot Got Hit!&quot;&#xA;            bot_hp -= beam_damage&#xA;            reward += 50&#xA;            agent_beam_size_x = agent_beam_size_y = 0&#xA;            agent_beam_x = agent_beam_y = beam_ob&#xA;            if bot_hp &amp;lt;= 0:&#xA;                successful = True&#xA;                cont = False&#xA;                winner = &quot;Agent&quot;&#xA;    return reward, cont, successful, winner&#xA;&#xA;def bot_beam_dir_detector():&#xA;    global bot_current_action&#xA;    if bot_current_action == 1:&#xA;        bot_beam_dir = 2&#xA;    elif bot_current_action == 2:&#xA;        bot_beam_dir = 4&#xA;    elif bot_current_action == 3:&#xA;        bot_beam_dir = 3&#xA;    elif bot_current_action == 4:&#xA;        bot_beam_dir = 1&#xA;    else:&#xA;        bot_beam_dir = 0&#xA;    return bot_beam_dir&#xA;&#xA;#Parameters&#xA;y = 0.75&#xA;e = 0.3&#xA;num_episodes = 10000&#xA;batch_size = 10&#xA;complexity = 100&#xA;with tf.Session() as sess:&#xA;    sess.run(initialize)&#xA;    success = 0&#xA;    for i in tqdm(range(1, num_episodes)):&#xA;        #print &quot;Episode #&quot;, i&#xA;        rAll = 0; d = False; c = True; j = 0&#xA;        param_init()&#xA;        samples = []&#xA;        while c == True:&#xA;            j += 1&#xA;            current_state = np.array([[mapping(complexity, float(agent_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(agent_y) / float(arena_y)),&#xA;                                        mapping(complexity, float(bot_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(bot_y) / float(arena_y)),&#xA;                                        #mapping(complexity, float(agent_hp) / float(character_init_health)),&#xA;                                        #mapping(complexity, float(bot_hp) / float(character_init_health)),&#xA;                                        mapping(complexity, float(agent_x - bot_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(agent_y - bot_y) / float(arena_y)),&#xA;                                        bot_beam_dir&#xA;                                        ]])&#xA;            b = bot_take_action()&#xA;            if np.random.rand(1) &amp;lt; e or i &amp;lt;= 5:&#xA;                a = random.randint(0, 8)&#xA;            else:&#xA;                a, _ = sess.run([predict, Q],feed_dict={input_layer : current_state})&#xA;            r, c, d, winner = action(a + 1, b)&#xA;            bot_beam_dir = bot_beam_dir_detector()&#xA;            next_state = np.array([[mapping(complexity, float(agent_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(agent_y) / float(arena_y)),&#xA;                                        mapping(complexity, float(bot_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(bot_y) / float(arena_y)),&#xA;                                        #mapping(complexity, float(agent_hp) / float(character_init_health)),&#xA;                                        #mapping(complexity, float(bot_hp) / float(character_init_health)),&#xA;                                        mapping(complexity, float(agent_x - bot_x) / float(arena_x)),&#xA;                                        mapping(complexity, float(agent_y - bot_y) / float(arena_y)),&#xA;                                        bot_beam_dir&#xA;                                        ]])&#xA;            samples.append([current_state, a, r, next_state])&#xA;            if len(samples) &amp;gt; 10:&#xA;                for count in xrange(batch_size):&#xA;                    [batch_current_state, action_taken, reward, batch_next_state] = samples[random.randint(0, len(samples) - 1)]&#xA;                    batch_allQ = sess.run(Q, feed_dict={input_layer : batch_current_state})&#xA;                    batch_Q1 = sess.run(Q, feed_dict = {input_layer : batch_next_state})&#xA;                    batch_maxQ1 = np.max(batch_Q1)&#xA;                    batch_targetQ = batch_allQ&#xA;                    batch_targetQ[0][a] = reward + y * batch_maxQ1&#xA;                    sess.run([updateModel], feed_dict={input_layer : batch_current_state, next_Q : batch_targetQ})&#xA;            rAll += r&#xA;            screen_blit()&#xA;            if d == True:&#xA;                e = 1. / ((i / 50) + 10)&#xA;                success += 1&#xA;                break&#xA;            #print agent_hp, bot_hp&#xA;            display.update()&#xA;&#xA;        jList.append(j)&#xA;        rList.append(rAll)&#xA;        print winner&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that if you have pygame and Tensorflow and matplotlib installed in a python environment you should be able to see the animations of the bot and the agent &quot;fighting&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I digressed in the update, but it would be awesome if somebody could also address my specific problem along with the original general problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update #2 on August 18, 2017:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the advice of @NeilSlater, I've implemented experience replay into my model. The algorithm has improved, but I'm going to look for more better improvement options that offer convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update #3 on August 22, 2017:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've noticed that if the agent hits the bot with a bullet on a turn and the action the bot taken on that turn was not &quot;fire a bullet&quot;, then the wrong actions would be given credit. Thus, I've turned the bullets into beams so the bot/agent takes damage on the turn the beam's fired. &lt;/p&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="37548" LastEditDate="2017-08-22T11:44:46.130" LastActivityDate="2019-04-10T11:33:32.277" Title="Why does Q Learning diverge?" Tags="|machine-learning|python|reinforcement-learning|q-learning|" AnswerCount="2" CommentCount="16" ContentLicense="CC BY-SA 3.0" />
  <row Id="22164" PostTypeId="1" CreationDate="2017-08-11T03:04:02.173" Score="2" ViewCount="3346" Body="&lt;p&gt;I am new to word/paragraph embedding and I am still trying to understand it. My question may be trivial. So please bear with me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a collection of short documents (each documents contain few sentences and lots of noise). I want to cluster these documents according to the most similar documents into one cluster (soft cluster is fine for now). Now there are several techniques available (and noted tutorials such as in scikit-learn) but I would like to see if I can successfully use doc2vec (gensim implementation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To this extent, I have ran the doc2vec on the collection and I have the &quot;paragraph vector&quot;s for each document. Obviously, I can cluster these vectors using something like K-Means. I am just wondering if this is the right approach or there is something else is needed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastActivityDate="2017-08-11T03:04:02.173" Title="Clustering text documents using doc2vec" Tags="|nlp|gensim|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22168" PostTypeId="1" AcceptedAnswerId="22169" CreationDate="2017-08-11T07:17:18.987" Score="0" ViewCount="571" Body="&lt;p&gt;I have a set of unique sentences. For each sentence I calculate a semantic similarity score (between 0 to 1) with the remaining sentences as mentioned in the below example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dataset = {sen1, sen2, sen3, sen4,..., senN}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For &lt;strong&gt;sen1&lt;/strong&gt; I calculate pairwise semantic similarity scores as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sen1 and sen2 = 0.3&#xA;sen1 and sen3 = 0.7&#xA;sen1 and sen4 = 0.9&#xA;...&#xA;...&#xA;...&#xA;sen1 and senN = 1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Likewise for all the sentences I calculate pairwise semantic similarity scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, I am getting a pairwise value, is it possible to cluster these sentences? Also what is the most appropriate clustering technique in my situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I want to cluster sentences based on the similarity value I have and also I consider values above 0.5 as semantically similar sentences.)&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-08-11T07:46:05.237" LastActivityDate="2017-08-11T08:24:28.143" Title="Categorise sentences based on their semantic similarity" Tags="|machine-learning|data-mining|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22169" PostTypeId="2" ParentId="22168" CreationDate="2017-08-11T08:24:28.143" Score="1" Body="&lt;p&gt;There are several techniques that you could apply in order to cluster data if your input is a matrix of pairwise distances between elements. As usual, the best option depends on your specific data, so it is hard to answer to the question of what is the best one, but you could try any of the following ones:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/K-medoids&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-medoids&lt;/a&gt; algorithm is similar to the well-known k-means algorithm. After randomly choosing k of your sequences as initial cluster centers (initial medoids) and assigning each sequence to the closest medoid, you randomly reassign sequences to different clusters as long as the value of the cost function decreases.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hierarchical_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hierarchical clustering&lt;/a&gt; is another example of clustering algorithm whose input is a matrix of pairwise distances between sequences. In this case the output is a dendrogram.&lt;/li&gt;&#xA;&lt;li&gt;Another option is to apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Multidimensional_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;multidimensional scaling&lt;/a&gt;, a dimensionality reduction technique which input is a matrix of pairwise distances between sequences, to project your sequences into a 2D plane. Once you do that, you can apply any cluster algorithm you can think of, like for instance k-means. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As I said, there are many other options, but these ones are the simplest ones I can think of, and the ones I would start from. &lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-08-11T08:24:28.143" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22170" PostTypeId="1" CreationDate="2017-08-11T08:27:57.187" Score="6" ViewCount="11672" Body="&lt;p&gt;I'm working on with an imbalanced dataset in Keras, and would like to give a higher weight to the samples from my minority class. The &lt;code&gt;fit()&lt;/code&gt; function has a nice &lt;code&gt;sample_weight&lt;/code&gt; argument, however because of the size of my data I have to use &lt;code&gt;fit_generator()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;fit_generator()&lt;/code&gt; has a &lt;code&gt;class_weight&lt;/code&gt; argument, which seems useful for this purpose and is already discussed in &lt;a href=&quot;https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras&quot;&gt;Another question&lt;/a&gt;. However, in this case the labels are not one-hot-encoded/categorical and I could not find whether using &lt;code&gt;class_weight&lt;/code&gt; also allows for categorical data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can use the &lt;code&gt;class_weight&lt;/code&gt; argument for one-hot-encoded/categorical labels and if so how? Or do I have to resort to a custom weighted loss function?&lt;/p&gt;&#xA;" OwnerUserId="37804" LastActivityDate="2017-08-17T14:54:33.760" Title="Setting class weights for categorical labels in Keras using generator" Tags="|keras|class-imbalance|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22173" PostTypeId="1" CreationDate="2017-08-11T09:48:37.030" Score="5" ViewCount="5509" Body="&lt;p&gt;Many works use 2-hidden-layer neural networks for classifying MNIST handwritten digits sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To improve accuracy, other techniques (dropout, ReLU.. etc) have been used without increasing the number of hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any reason not to use more than three hidden layers?&#xA;for example, overfitting?&lt;/p&gt;&#xA;" OwnerUserId="37810" LastEditorUserId="37810" LastEditDate="2017-08-11T10:42:50.730" LastActivityDate="2017-08-11T15:36:15.833" Title="Why not use more than 3 hidden layers for MNIST classification?" Tags="|overfitting|mnist|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22174" PostTypeId="1" CreationDate="2017-08-11T10:06:27.863" Score="3" ViewCount="40" Body="&lt;p&gt;I know that if we input an image of shape [6,128] to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 , then there will be 5 activation maps as output from the layer ..each map has a shape of [4,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about the number of maps outputted when we use batching ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean if we are using a batch with size of 2 images (each of shape[6,128]) as input to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 also , then there will be 10 activation maps as output from the layer instead of 5 (each image in the batch has 5 maps) ?? or we will have also 5 maps but each of shape[2,4,1] ???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there will be 10 maps since if we apply then max pooling , we want to choose the max from each image independently , right? so each image must have separate maps from the maps of the other image..&lt;/p&gt;&#xA;" OwnerUserId="37811" LastEditorUserId="29169" LastEditDate="2021-02-16T00:43:47.570" LastActivityDate="2021-02-16T00:43:47.570" Title="Convnets : do we have separate activation maps for images in a batch" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22175" PostTypeId="1" CreationDate="2017-08-11T06:48:41.987" Score="2" ViewCount="39" Body="&lt;p&gt;I have a HP15-R203TX notebook with CPU i5-5200U 2.20GHzx4 / RAM 4Gb /2 Gb NVIDIA GeForce 820M. My friend has the same notebook. Another one has a dell notebook but same graphic card.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When i ran same jupyter notebook on same dataset to train the same number of layers, it takes 20 minutes on mine but 6-8 minutes on both of my friend's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried resetting the whole environment, but didn't helped.&lt;/p&gt;&#xA;" OwnerDisplayName="Ashish Agrawal" LastActivityDate="2017-08-11T14:14:38.437" Title="Why it is taking more training time on one machine comparatively to similar another?" Tags="|machine-learning|python|deep-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22176" PostTypeId="2" ParentId="22145" CreationDate="2017-08-11T10:50:14.627" Score="2" Body="&lt;p&gt;It is not a pre-requisite, and you can learn it easily once you encounter something you are not familiar with. Statistics is pretty old and there are many learning resources on the Web, which you can get to whenever you hit the wall while learning about deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to which field is pre-requisite, I think it is enough to first learn about Gaussian (normal) distribution, linear regression, and logistic regression. Then when you encounter something you don't understand, it is time to invest your time on statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more requisite fields I believe are calculus and linear algebra. If you haven't learned about them (such as partial derivative, matrix transpose, etc), it is very difficult to start to learn deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the prior exposure to some of machine learning algorithms would make your learning faster. But I'm sure you already got it given that you finished Andrew Ng's course on Coursera. &lt;a href=&quot;https://www.coursera.org/specializations/deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng starts deep learning course on Coursera from August 15th&lt;/a&gt;, so you can join it and get a grasp of what is required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The book you linked sounds more like focused on machine learning than on statistics, BTW.&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-08-11T10:50:14.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22177" PostTypeId="1" AcceptedAnswerId="22183" CreationDate="2017-08-11T11:20:38.073" Score="10" ViewCount="7688" Body="&lt;p&gt;I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape &lt;code&gt;(total_seq, 20)&lt;/code&gt;. I can reshape it into &lt;code&gt;(total_seq, 20, 1)&lt;/code&gt; for concatenation to other features.&lt;/p&gt;&#xA;&lt;p&gt;I also have news title with 10 words for each timestep. So I have 3D array of shape &lt;code&gt;(total_seq, 20, 10)&lt;/code&gt; of the news' tokens from &lt;code&gt;Tokenizer.texts_to_sequences()&lt;/code&gt; and &lt;code&gt;sequence.pad_sequences()&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I want to concatenate the news embedding to the stock price and make predictions.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;My idea is that the news embedding should return tensor of shape&#xA;&lt;code&gt;(total_seq, 20, embed_size)&lt;/code&gt; so that I can concatenate it with the&#xA;stock price of shape &lt;code&gt;(total_seq, 20, 1)&lt;/code&gt; then connect it to LSTM layers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;To do that, I should convert news embedding of shape &lt;code&gt;(total_seq, 20, 10)&lt;/code&gt; to&#xA;&lt;code&gt;(total_seq, 20, 10, embed_size)&lt;/code&gt; by using &lt;code&gt;Embedding()&lt;/code&gt; function.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;But in Keras, the &lt;code&gt;Embedding()&lt;/code&gt; function takes a 2D tensor instead of 3D tensor. How do I get around with this problem?&lt;/p&gt;&#xA;&lt;p&gt;Assume that &lt;code&gt;Embedding()&lt;/code&gt; accepts 3D tensor, then after I get 4D tensor as output, I would remove the 3rd dimension by using LSTM to return last word's embedding only, so output of shape &lt;code&gt;(total_seq, 20, 10, embed_size)&lt;/code&gt; would be converted to &lt;code&gt;(total_seq, 20, embed_size)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;But I would encounter another problem again, LSTM accepts 3D tensor not 4D so&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;How do I get around with Embedding and LSTM not accepting my inputs?&lt;/p&gt;&#xA;" OwnerUserId="37814" LastEditorUserId="98307" LastEditDate="2020-08-02T10:28:43.933" LastActivityDate="2020-08-02T10:28:43.933" Title="How to use Embedding() with 3D tensor in Keras?" Tags="|python|tensorflow|keras|rnn|lstm|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="22178" PostTypeId="2" ParentId="22173" CreationDate="2017-08-11T11:51:16.453" Score="2" Body="&lt;p&gt;Empirically, the network performance does not increase much for a fully-connected network on MNIST when you add layers, but you can &lt;em&gt;probably&lt;/em&gt; find ways to improve it on networks with 3+ hidden layers, such as data augmentation (e.g. variations of all inputs translated +-0..2 pixels in x and y, roughly 25 times the original data size, as a start).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think this idea is pursued very far in practice, because CNNs offer a much better performance increase for the effort required. You hit the point of diminishing returns earlier with a basic MLP (around 96-97% accuracy) than you can reach easily with a CNN (around 99% accuracy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The theory basis for this difference is not obvious to me, but very likely yes this is related to over-fitting. The weight sharing and feature pooling in a CNN is very effective way of processing image data for classification tasks, and avoids over-fitting by reducing the number of parameters, whilst re-using the parameters for the task in a way that makes very good sense given the nature of the inputs.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-11T15:36:15.833" LastActivityDate="2017-08-11T15:36:15.833" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22179" PostTypeId="1" CreationDate="2017-08-11T12:18:36.087" Score="6" ViewCount="238" Body="&lt;p&gt;I am currently studying this &lt;a href=&quot;https://yorkspace.library.yorku.ca/xmlui/handle/10315/29980&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; (page 53), in which the suggest convolution to be done in a special manner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation} \tag{1}\label{1}&#xA;q_{j,m} = \sigma \left(\sum_i \sum_{n=1}^{F} o_{i,n+m-1} \cdot  w_{i,j,n} + w_{0,j} \right)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is their explanation: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As shown in &lt;a href=&quot;https://i.stack.imgur.com/Dq9g8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fig. 4.2&lt;/a&gt;, all input feature maps (assume I in total), $O_i (i = 1, · · · , I)$ are mapped into a number of feature maps (assume $J$ in total), $Q_j (j = 1, · · · , J)$ in the convolution layers based on a number of local filters ($I × J$ in total), $w_{ij}$ $(i = 1, · · · , I; j = 1, · · · , J)$. The mapping can be represented as the well-known convolution operation in signal processing.&#xA;Assuming input feature maps are all one dimensional, each unit of one feature map in the convolution layer can be computed as equation $\eqref{1}$ (equation above).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $o_{i,m}$ is the $m$-th unit of the $i$-th input feature map $O_i$, $q_{j,m}$ is the $m$-&#xA;th unit of the $j$-th feature map $Q_j$ of the convolution layer, $w_{i,j,n}$ is the $n$th element of the weight vector, $w_{i,j}$, connecting the $i$th feature map of the input to the $j$th feature map of the convolution layer, and $F$ is called the filter size which is the number of input bands that each unit of the convolution layer receives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far so good:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i basically understood from this is what I've tried to illustrate in this image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/l3pFr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/l3pFr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seem to me what they are doing is actually processing all data points up to F, and across all feature maps.  Basically moving in both x-y direction, and compute on point from that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't that basically 2d- convolution on a 2d image of size $(I x F)$ with a filter equal to the image size?. &#xA;The weight doesn't seem to differ at all have any importance here..?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-16T00:43:36.293" LastActivityDate="2021-02-16T00:43:36.293" Title="Why is this not ordinary convolution?" Tags="|neural-network|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22181" PostTypeId="1" AcceptedAnswerId="22285" CreationDate="2017-08-11T13:41:33.153" Score="3" ViewCount="870" Body="&lt;p&gt;I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the learning rate for the neural network, $\eta$, and the learning rate for Q-learning, $\alpha$. They both seem to change the rate at which the neural net takes new conclusions over old ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $\eta$, or do they have ultimately different effects?&lt;/p&gt;&#xA;" OwnerUserId="37678" LastActivityDate="2017-08-15T19:03:09.790" Title="Neural Network Learning Rate vs Q-Learning Learning Rate" Tags="|machine-learning|neural-network|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22182" PostTypeId="1" CreationDate="2017-08-11T13:44:02.060" Score="1" ViewCount="230" Body="&lt;p&gt;I want to use the SQL table connector to link data in a PostGres database which is on my local machine running as 9.6. When I do this I get the following error;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;could not open extension control file &quot;.../postgressql/extension/quantile.control&quot; No such file or directory&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking so some similar issues I  tried: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;find /usr -name quantile.control&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in terminal and get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;find: /usr/sbin/authserver: Permission denied&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What can I do to resolve this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: From this &lt;a href=&quot;https://blog.biolab.si/2015/12/04/2uda/&quot; rel=&quot;nofollow noreferrer&quot;&gt;answer&lt;/a&gt; I can see tat I need to install an extension, which I do according to the instructions &lt;a href=&quot;https://pgxn.org/dist/quantile&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. But am not able to get the extension to install getting the following error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INFO: best version: quantile 1.1.5&#xA;INFO: saving /var/folders/0c/g8tsv9jx3qgdwrdpqvjx45tr0000gn/T/tmppe1b4shp/quantile-1.1.5.zip&#xA;INFO: unpacking: /var/folders/0c/g8tsv9jx3qgdwrdpqvjx45tr0000gn/T/tmppe1b4shp/quantile-1.1.5.zip&#xA;INFO: building extension&#xA;Makefile:16: /Applications/anaconda/lib/pgxs/src/makefiles/pgxs.mk: No such file or directory&#xA;make: *** No rule to make target `/Applications/anaconda/lib/pgxs/src/makefiles/pgxs.mk'.  Stop.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37786" LastEditorUserId="11097" LastEditDate="2017-08-12T04:21:56.123" LastActivityDate="2017-08-12T04:21:56.123" Title="Problem connecting to Postgres" Tags="|orange|" AnswerCount="0" CommentCount="0" ClosedDate="2017-08-12T10:13:01.450" ContentLicense="CC BY-SA 3.0" />
  <row Id="22183" PostTypeId="2" ParentId="22177" CreationDate="2017-08-11T13:45:50.777" Score="8" Body="&lt;p&gt;I'm not entirely sure if this is the cleanest solution but I stitched everything together. Each of the 10 word positions get their own input but that shouldn't be too much of a problem. The idea is to make an Embedding layer and use it multiple times. First we will generate some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_samples = 1000&#xA;time_series_length = 50&#xA;news_words = 10&#xA;news_embedding_dim = 16&#xA;word_cardinality = 50&#xA;&#xA;x_time_series = np.random.rand(n_samples, time_series_length, 1)&#xA;x_news_words = np.random.choice(np.arange(50), replace=True, size=(n_samples, time_series_length, news_words))&#xA;x_news_words = [x_news_words[:, :, i] for i in range(news_words)]&#xA;y = np.random.randint(2, size=(n_samples))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we will define the layers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Input of normal time series&#xA;time_series_input = Input(shape=(50, 1, ), name='time_series')&#xA;&#xA;## For every word we have it's own input&#xA;news_word_inputs = [Input(shape=(50, ), name='news_word_' + str(i + 1)) for i in range(news_words)]&#xA;&#xA;## Shared embedding layer&#xA;news_word_embedding = Embedding(word_cardinality, news_embedding_dim, input_length=time_series_length)&#xA;&#xA;## Repeat this for every word position&#xA;news_words_embeddings = [news_word_embedding(inp) for inp in news_word_inputs]&#xA;&#xA;## Concatenate the time series input and the embedding outputs&#xA;concatenated_inputs = concatenate([time_series_input] + news_words_embeddings, axis=-1)&#xA;&#xA;## Feed into LSTM&#xA;lstm = LSTM(16)(concatenated_inputs)&#xA;&#xA;## Output, in this case single classification&#xA;output = Dense(1, activation='sigmoid')(lstm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After compiling the model we can just fit it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit([x_time_series] + x_news_words, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After what you mentioned in the comments, you can add a dense layer that summarizes the news, and adds that to your time series (stock prices):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Summarize the news:&#xA;news_words_concat = concatenate(news_words_embeddings, axis=-1)&#xA;news_words_transformation = TimeDistributed(Dense(combined_news_embedding))(news_words_concat)&#xA;&#xA;## New concat&#xA;concatenated_inputs = concatenate([time_series_input, news_words_transformation], axis=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-08-11T19:15:18.213" LastActivityDate="2017-08-11T19:15:18.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22185" PostTypeId="2" ParentId="22175" CreationDate="2017-08-11T14:14:38.437" Score="1" Body="&lt;p&gt;In machine learning, while your choice of inputs and hyper-parameters do matter, most of the time convergence ultimately comes down to how lucky you are. If you're lucky you might reach optimum in a very short period of time, whereas if you're unlucky you might get stuck in a local optimum with a huge error and get stuck in a very long loop. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's one reason; another is possibly that if you're using TensorFlow and you installed the version for non-GPU computers, and your friends installed the version for computers with GPUs. &lt;/p&gt;&#xA;" OwnerUserId="37548" LastActivityDate="2017-08-11T14:14:38.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22186" PostTypeId="2" ParentId="22174" CreationDate="2017-08-11T14:27:32.357" Score="1" Body="&lt;p&gt;Yes, separate maps. The number of the image in the batch is another dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Max pool is performed on a per-image basis, other layers operate on an entire batch - e.g. batch normalisation.&lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-08-11T14:27:32.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22187" PostTypeId="1" CreationDate="2017-08-11T14:40:13.300" Score="1" ViewCount="152" Body="&lt;p&gt;In Rshiny i want to wrap the data in the table.How can i do that below is the code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    fluidRow(&#xA;        column(4,&#xA;               tableOutput('table')&#xA;        )&#xA;&#xA;tags$style(HTML('#table{word-wrap: break-word;}')),&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it is not reflecting.I am newbie to UI&lt;/p&gt;&#xA;" OwnerUserId="35449" LastActivityDate="2017-08-11T14:40:13.300" Title="Rshiny table row data wrap" Tags="|r|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22188" PostTypeId="2" ParentId="9819" CreationDate="2017-08-11T14:49:35.090" Score="1" Body="&lt;p&gt;Increasing the iter count (number of epochs) dramatically increases the training time. Word2Vec gives quality results only if you feed a massive amount documents, therefore looping even twice on them is not reasonable although it actually makes the resulting word embeddings more accurate.&lt;/p&gt;&#xA;" OwnerUserId="37830" LastActivityDate="2017-08-11T14:49:35.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22190" PostTypeId="1" CreationDate="2017-08-11T14:53:29.367" Score="2" ViewCount="47" Body="&lt;p&gt;I have a dataset from an operating process having 5 measurements and 1 outcome. All values are normally distributed. When I train a regression model on the dataset it performs good on the majority of the dataset - the default operating condition of the process. It performs much worse though on other than default operating conditions, values distant from the mean. If it were a classification problem I would treat this as class imbalance and perform some resampling technique to get balanced classes. How do I treat this for a regression problem?&lt;/p&gt;&#xA;" OwnerUserId="37831" LastActivityDate="2022-04-07T23:01:20.120" Title="Resampling a normally distributed dataset for regression problems?" Tags="|regression|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22191" PostTypeId="1" AcceptedAnswerId="22214" CreationDate="2017-08-11T17:50:23.280" Score="10" ViewCount="14770" Body="&lt;p&gt;I am going through &lt;a href=&quot;https://github.com/pytorch/examples/blob/master/imagenet/main.py&quot; rel=&quot;noreferrer&quot;&gt;this imagenet example&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, in &lt;a href=&quot;https://github.com/pytorch/examples/blob/86bc3e516dde92fee0ce00668a3714c03155dd44/imagenet/main.py#L88&quot; rel=&quot;noreferrer&quot;&gt;line 88&lt;/a&gt;, the module DistributedDataParallel is used. When I searched for the same in the &lt;a href=&quot;http://pytorch.org/docs/&quot; rel=&quot;noreferrer&quot;&gt;docs&lt;/a&gt;, I haven’t found anything. However, I found the &lt;a href=&quot;http://pytorch.org/docs/master/nn.html?highlight=dataparallel#dataparallel-layers-multi-gpu-distributed&quot; rel=&quot;noreferrer&quot;&gt;documentation for &lt;code&gt;DataParallel&lt;/code&gt;&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-08-14T13:52:18.060" LastActivityDate="2019-07-27T13:53:23.780" Title="What is the difference between Pytorch's DataParallel and DistributedDataParallel?" Tags="|gpu|distributed|pytorch|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22192" PostTypeId="2" ParentId="20372" CreationDate="2017-08-11T21:00:43.553" Score="1" Body="&lt;p&gt;Facebook's &lt;a href=&quot;https://github.com/facebookresearch/fastText&quot; rel=&quot;nofollow noreferrer&quot;&gt;fasttex&lt;/a&gt; is really great for loosely formatted text.  Fasttext is a library for doing several tasks, the one you'll want to use is text classification.  The best part of fasttext it uses character n-grams that instead of working with full words (which @JahKnows described well).  The hardest part of using the tool is formatting the training data properly, but just look through the provided example and you should get it pretty quickly.&lt;/p&gt;&#xA;" OwnerUserId="8041" LastActivityDate="2017-08-11T21:00:43.553" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22193" PostTypeId="1" AcceptedAnswerId="22206" CreationDate="2017-08-11T21:47:04.587" Score="2" ViewCount="737" Body="&lt;p&gt;I want to create a heat map to visualize some production data, but without geolocation. I am finishing some experiments in a greenhouse, divided in different sectors. The idea is to make a heat map to watch in which areas we are harvesting more fruits, or less, in each season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which tool, software or language do you think is better for this purpose? I was looking for information, but the main tools are for geolocation.&lt;/p&gt;&#xA;" OwnerUserId="37845" LastEditorUserId="8432" LastEditDate="2017-08-12T10:14:57.417" LastActivityDate="2022-07-14T15:47:37.503" Title="Heat map and visualization" Tags="|visualization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22194" PostTypeId="1" CreationDate="2017-08-11T21:53:51.810" Score="6" ViewCount="7803" Body="&lt;p&gt;I implemented different machine learning algorithms on a matrix with binary data to predict a univariate target with two classes. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;random forest (accuracy = 62.01)&lt;/li&gt;&#xA;&lt;li&gt;Neural Network(acc= 58.9)&lt;/li&gt;&#xA;&lt;li&gt;svm-radial kernel (accuracy = 58.02)&lt;/li&gt;&#xA;&lt;li&gt;linear discriminant analysis(accuracy = 57.9)&lt;/li&gt;&#xA;&lt;li&gt;logistic regression(accuracy = 57.6). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My baseline accuracy is 52.55. But in case of Naive Bayes in same setting gives only 48.5 accuracy that identifies only one class in y. predict. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible for a machine learning model to behave worse than a random classification?&lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="29575" LastEditDate="2017-08-12T03:29:38.417" LastActivityDate="2022-02-10T21:19:01.877" Title="Is it possible for a ML model to perform worse than random classification?" Tags="|machine-learning|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22195" PostTypeId="1" CreationDate="2017-08-11T22:02:48.103" Score="1" ViewCount="194" Body="&lt;p&gt;We have a data set of 300,000+ records that looks something like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item ID, Quantity 2017, Quantity 2016, Quantity 2015, Quantity 2014, Quantity 2013&#xA;1111, 100, 50, 25, 10, 0&#xA;2222, 0, 10, 100, 500, 1000&#xA;3333, 10, 0, 5, 2, 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We are currently attempting to find the best model to predict Quantity 2017 based on the previous quantities for each record. We have tried Decision Tree Regression, Multiple Linear Regression, and Random Forest Regression (10, 100, 1000 trees) but our results were a bit too far off for approval. We are using 80% of our data for training and 20% for testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any models that are better suited for this type of calculation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are also concerned that perhaps our data structure is part of the problem and we should re-evaluate. Is there another structure that might be better?&lt;/p&gt;&#xA;" OwnerUserId="37847" LastActivityDate="2017-09-11T16:01:18.103" Title="Choosing the right model for predicting demand" Tags="|machine-learning|model-selection|forecasting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22196" PostTypeId="2" ParentId="22194" CreationDate="2017-08-11T22:16:33.153" Score="8" Body="&lt;p&gt;Yes, it's possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just means the model is adjusting to noise, so it's valuing the &quot;wrong&quot; features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an analogy, if you were randomly guessing basketball game outcomes, you'd probably perform better than someone who thinks less points is better and is guessing based on previous games' scores&lt;/p&gt;&#xA;" OwnerUserId="37794" LastActivityDate="2017-08-11T22:16:33.153" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22197" PostTypeId="2" ParentId="22194" CreationDate="2017-08-11T22:19:47.873" Score="2" Body="&lt;p&gt;Your model can be worse than random, for example, if some fundamental assumptions are violated, in an imbalanced setting when your using accuracy as your baseline or you have noisy data etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in a binary setting, if your classes are perfectly balanced and if your classifier is consistently making false predictions (not due to randomness), you can always adjust the model to be better than random by predicting the exact opposite of your model. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-08-11T22:19:47.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22198" PostTypeId="1" CreationDate="2017-08-12T00:21:04.323" Score="2" ViewCount="1419" Body="&lt;p&gt;We have &lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;LDA&lt;/a&gt; topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics.&lt;/p&gt;&#xA;&lt;p&gt;Also, we can evaluate the model we have created. one of the approaches is using classification method like SVM.  My goal is to evaluate the created model.&lt;/p&gt;&#xA;&lt;p&gt;I am faced with two kinds of code for making the LDA model.&lt;/p&gt;&#xA;&lt;h3&gt;Approach 1:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;# generate LDA model&#xA;id2word = corpora.Dictionary(texts)&#xA;&#xA;# Creates the Bag of Word corpus.&#xA;mm = [id2word.doc2bow(text) for text in texts]&#xA;&#xA;# Trains the LDA models.&#xA;lda = ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=10,&#xA;                               update_every=1, chunksize=10000, passes=1,gamma_threshold=0.00, minimum_probability=0.00)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;this way I can not use Fit_transform&lt;/p&gt;&#xA;&lt;h3&gt;Approach 2:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;tf_vectorizer = CountVectorizer(max_features=n_features,&#xA;                                stop_words='english')&#xA;tf = tf_vectorizer.fit_transform(data_samples)&#xA;&#xA;lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,&#xA;                                learning_method='online',&#xA;                                learning_offset=50.,&#xA;                                random_state=0)&#xA;&#xA;lda_x=lda.fit_transform(tf) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;in the first approach, there is no fit_transform method for LDA model, I do not know why as I don't understand the difference between them.&lt;/p&gt;&#xA;&lt;p&gt;Anyway, I need to pass that LDA model I have created with the first approach to SVM(the reason I put these two approach here is that I know with the second approach there is no error probably because of fit_transform but for some reason I couldn't use that.&lt;/p&gt;&#xA;&lt;h3&gt;Final code:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;from gensim.models import ldamodel&#xA;from nltk.tokenize import RegexpTokenizer&#xA;from nltk.stem.porter import PorterStemmer&#xA;from gensim import corpora, models&#xA;from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.multiclass import OneVsRestClassifier&#xA;from sklearn.svm import LinearSVC&#xA;&#xA;&#xA;tokenizer = RegexpTokenizer(r'\w+')&#xA;&#xA;# create English stop words list&#xA;en_stop = {'a'}&#xA;&#xA;# Create p_stemmer of class PorterStemmer&#xA;lines=[]&#xA;p_stemmer = PorterStemmer()&#xA;lisOfFiles=[x[2] for x in os.walk(&amp;quot;data&amp;quot;)]&#xA;&#xA;fullPath = [x[0] for x in os.walk(&amp;quot;data&amp;quot;)]&#xA;for j in lisOfFiles[2]:&#xA;    with open(os.path.join(fullPath[2],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;for j in lisOfFiles[3]:&#xA;    with open(os.path.join(fullPath[3],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;for j in lisOfFiles[4]:&#xA;    with open(os.path.join(fullPath[4],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;# compile sample documents into a list&#xA;doc_set = lines&#xA;# list for tokenized documents in loop&#xA;texts = []&#xA;&#xA;# loop through document list&#xA;for i in doc_set:&#xA;    # clean and tokenize document string&#xA;    raw = i.lower()&#xA;    tokens = tokenizer.tokenize(raw)&#xA;&#xA;    # remove stop words from tokens&#xA;    stopped_tokens = [i for i in tokens if not i in en_stop]&#xA;&#xA;    # stem tokens&#xA;    stemmed_tokens = [p_stemmer.stem(i) for i in stopped_tokens]&#xA;&#xA;    # add tokens to list&#xA;    texts.append(stemmed_tokens)&#xA;&#xA;# generate LDA model&#xA;id2word = corpora.Dictionary(texts)&#xA;&#xA;# Creates the Bag of Word corpus.&#xA;mm = [id2word.doc2bow(text) for text in texts]&#xA;&#xA;# Trains the LDA models.&#xA;lda = ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=10,&#xA;                               update_every=1, chunksize=10000, passes=1,gamma_threshold=0.00, minimum_probability=0.00)&#xA;&#xA;# Assigns the topics to the documents in corpus&#xA;&#xA;dictionary = corpora.Dictionary(texts)&#xA;&#xA;# convert tokenized documents into a document-term matrix&#xA;corpus = [dictionary.doc2bow(text) for text in texts]&#xA;&#xA;&#xA;#creating the labels&#xA;lda_corpus = lda[mm]&#xA;label_y=[]&#xA;for i in lda_corpus:&#xA;    new_y = []&#xA;    for l in i:&#xA;        sorted_labels = sorted(i, key=lambda z: z[0], reverse=True)&#xA;        if l[1] &amp;gt; 0.005:&#xA;            new_y.append(l[0])&#xA;        label_y.append(new_y)&#xA;&#xA;classifier = Pipeline([&#xA;    ('vectorizer', CountVectorizer(max_df=2,min_df=1)),&#xA;    ('tfidf', TfidfTransformer()),&#xA;    ('clf', OneVsRestClassifier(LinearSVC()))])&#xA;classifier.fit(lda, label_y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As you see in my code I have used the first approach for some reasons, but in the last line, it raises an error(object of type int has no len()). It seems it can not accept LDA created in this way (I was thinking because in this way I didn't use fit_transform) how can I fix this error with my code?&lt;/p&gt;&#xA;&lt;h3&gt;Stack trace:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;/home/saria/tfwithpython3.6/bin/python /home/saria/PycharmProjects/TfidfLDA/test4.py&#xA;Using TensorFlow backend.&#xA;Traceback (most recent call last):&#xA;  File &amp;quot;/home/saria/PycharmProjects/TfidfLDA/test4.py&amp;quot;, line 92, in &amp;lt;module&amp;gt;&#xA;    classifier.fit(lda, label_y)&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/sklearn/pipeline.py&amp;quot;, line 268, in fit&#xA;    Xt, fit_params = self._fit(X, y, **fit_params)&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/sklearn/pipeline.py&amp;quot;, line 234, in _fit&#xA;    Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/sklearn/feature_extraction/text.py&amp;quot;, line 839, in fit_transform&#xA;    self.fixed_vocabulary_)&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/sklearn/feature_extraction/text.py&amp;quot;, line 760, in _count_vocab&#xA;    for doc in raw_documents:&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/gensim/models/ldamodel.py&amp;quot;, line 1054, in __getitem__&#xA;    return self.get_document_topics(bow, eps, self.minimum_phi_value, self.per_word_topics)&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/gensim/models/ldamodel.py&amp;quot;, line 922, in get_document_topics&#xA;    gamma, phis = self.inference([bow], collect_sstats=per_word_topics)&#xA;  File &amp;quot;/home/saria/tfwithpython3.6/lib/python3.5/site-packages/gensim/models/ldamodel.py&amp;quot;, line 429, in inference&#xA;    if len(doc) &amp;gt; 0 and not isinstance(doc[0][0], six.integer_types + (np.integer,)):&#xA;TypeError: object of type 'int' has no len()&#xA;&#xA;Process finished with exit code 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37510" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-08-12T03:21:21.500" Title="fitting classifier object of type 'int' has no len()" Tags="|python|classification|svm|multilabel-classification|lda|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22199" PostTypeId="1" AcceptedAnswerId="22289" CreationDate="2017-08-12T01:01:07.093" Score="4" ViewCount="2083" Body="&lt;p&gt;What is the default number of internal layers and internal nodes in training a neural network? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data has 62 observations with roughly 200 predictors. I have a target variable with two classes and implemented a neural network with one internal layer and one internal node without repeats. Also, I tried with two internal layers, with 5 internal nodes in one, and 2 internal nodes in second layer. I want to find the accuracy, first, on default values and then I will try to optimize the model performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the criterion to choose the number of layers and internal nodes in a neural network training model? In the case of Random Forest, we can choose to try to be roughly equal to the square root of the number of predictors. &lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="29169" LastEditDate="2019-06-09T16:41:19.843" LastActivityDate="2019-06-09T16:41:19.843" Title="What are the default values of nodes and internal layers in Neural Network models?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22202" PostTypeId="1" CreationDate="2017-08-12T11:50:37.757" Score="1" ViewCount="133" Body="&lt;p&gt;Is it possible for results to fluctuate a little bit even if I set set.seed? Kindly comment on the degree of freedom allowed or the possible standard deviation from the result set if seeds are fixed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I set seeds in R and I get slightly varied results ranging from 0.1 to 0.5 S.D from previous runs. I want to know if it is normal&lt;/p&gt;&#xA;" OwnerUserId="31599" LastEditorUserId="1330" LastEditDate="2020-06-12T20:16:12.463" LastActivityDate="2020-06-12T20:16:12.463" Title="reproducible results by set.seed" Tags="|machine-learning|r|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22205" PostTypeId="1" AcceptedAnswerId="22207" CreationDate="2017-08-12T14:49:18.617" Score="4" ViewCount="220" Body="&lt;p&gt;I have been trying to understand the backpropagation for a while now.&#xA;I have came across two variants of it.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;In the Andrew Ng class the derivatives of the weights of hidden layers are calculated using the error signal that is distributed back to the hidden node.&lt;/li&gt;&#xA;&lt;li&gt;In Geoffrey Hinton class the derivatives of the weights of hidden layers are calculated using the derivatives of the next layer that are already computed and from my knowledge of calculus that makes more sense.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can someone explain how the first variant works?&lt;/p&gt;&#xA;" OwnerUserId="37866" LastEditorUserId="29575" LastEditDate="2017-08-13T05:06:22.900" LastActivityDate="2017-08-14T08:37:38.063" Title="Confusion in backpropagation algorithm" Tags="|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22206" PostTypeId="2" ParentId="22193" CreationDate="2017-08-12T15:11:15.687" Score="2" Body="&lt;p&gt;I believe that the simplest is to use &lt;code&gt;Seaborn&lt;/code&gt;. Check out the example below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import seaborn as sns &#xA;&#xA;sns.set()&#xA;np.random.seed(0)&#xA;&#xA;uniform_data = np.random.rand(10, 12)&#xA;ax = sns.heatmap(uniform_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the output looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oKvwJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oKvwJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is taken from the seaborn &lt;a href=&quot;http://seaborn.pydata.org/generated/seaborn.heatmap.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="17290" LastEditDate="2017-08-21T06:03:17.583" LastActivityDate="2017-08-21T06:03:17.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22207" PostTypeId="2" ParentId="22205" CreationDate="2017-08-12T15:14:45.617" Score="4" Body="&lt;p&gt;The first variant &lt;em&gt;is&lt;/em&gt; the second variant, or more accurately there is only one type of backpropagation, and that works with the gradients of a loss function with respect to parameters of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not an uncommon point to have questions about though, the main issue that I see causing confusion is when the loss function has been cleverly constructed so that it works with the output layer activation function, and the derivative term is numerically $\hat{y} - y$, which looks the same as taking the linear error directly. People studying the code implementing a network like this can easily come to the conclusion that the initial gradient is in fact an initial error (and whilst these are numerically equal, they are different concepts, and in a generic neural network they don't have to be equal)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This situation applies for the following network architectures:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Mean squared error $\frac{1}{2N}\sum_{i=1}^N(\hat{y}_i - y_i)^2$ and linear output layer - note the multiplier $\frac{1}{2}$ is there to deliberately simplify the derivative.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Binary cross-entropy $\frac{-1}{N}\sum_{i=1}^Ny_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1-\hat{y}_i)$ and sigmoid output layer. The derivative of the loss neatly cancels out the derivative of the sigmoid, leaving you with gradient at the pre-transform stage of $\hat{y} - y$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multi-class logloss with one-hot encoding of true classes $\frac{-1}{N}\sum_{i=1}^N\mathbf{y}_i\cdot\text{log}(\hat{\mathbf{y}}_i)$ and softmax output layer. Again the derivative of the loss neatly cancels out, leaving you with gradient at the pre-transform stage of $\hat{y} - y$ for the true class.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So when you are told that backpropagation processes an &quot;error signal&quot; or &quot;the error&quot; backwards through the network, just mentally add &quot;the gradient of&quot; to the start of the phrase. Some people will say it knowingly as shorthand, others might be honestly confused. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same applies to deeper layers, although then there is no other source for the confused &quot;this is the error being distributed&quot; other than as shorthand for &quot;this is the [gradient of the] error being distributed&quot;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-14T08:37:38.063" LastActivityDate="2017-08-14T08:37:38.063" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22208" PostTypeId="2" ParentId="22195" CreationDate="2017-08-12T15:20:30.677" Score="1" Body="&lt;p&gt;It looks to me that the data are in fact a Time Serie. Therefore, you can treat them as such.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, there are many approaches with each having each own strengths/weaknesses. I'd start with simple models such as ARMA or SARIMA. Possibly also GARCH or VAR. If you wish to go for something more advanced, Recurrent Neural Networks are also used in Time Series prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beside that, I am surprised that for example Random Forest have not given sufficient performance. Have you tried thorough feature selection? It may be that you have too high expectations.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-08-12T15:20:30.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22209" PostTypeId="2" ParentId="22199" CreationDate="2017-08-12T17:54:04.023" Score="3" Body="&lt;p&gt;There are websites that explain these pretty well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Deciding on the number of neurons in the hidden layer(s)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://www.r-bloggers.com/selecting-the-number-of-neurons-in-the-hidden-layer-of-a-neural-network/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.r-bloggers.com/selecting-the-number-of-neurons-in-the-hidden-layer-of-a-neural-network/&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The most common rule of thumb is to choose a number of hidden neurons between 1 and the number of input variables.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Deciding on the number of layers of hidden layers&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw&quot;&gt;https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope that answers your question!&lt;/p&gt;&#xA;" OwnerUserId="37548" LastActivityDate="2017-08-12T17:54:04.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22211" PostTypeId="1" CreationDate="2017-08-12T21:04:27.917" Score="2" ViewCount="305" Body="&lt;p&gt;Let's assume I'm modeling a process like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y=log(x1+x2+x3+N)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $x_i$ are features and $N$ is some error/noise value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the way that decision trees work and the way that LightGBM works, will this be modeled correctly? Or will it be necessary for me to train using $e^y$?&lt;/p&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2020-03-19T21:25:44.313" Title="What scale does LightGBM use for output?" Tags="|predictive-modeling|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22212" PostTypeId="1" AcceptedAnswerId="22221" CreationDate="2017-08-13T01:05:18.227" Score="5" ViewCount="2845" Body="&lt;p&gt;I saw the term &quot;policy collapse&quot; on the comments of a &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial for  reinforcement learning&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad hyper-parameter, be it the learning rate, batch size, etc., but I couldn't find anything explaining it in clearly and in detail. &lt;/p&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="836" LastEditDate="2017-08-13T14:57:58.973" LastActivityDate="2023-07-03T13:02:53.937" Title="What is &quot;Policy Collapse&quot; and what are the causes?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22214" PostTypeId="2" ParentId="22191" CreationDate="2017-08-13T04:42:59.550" Score="9" Body="&lt;p&gt;As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's docstrings for figuring out the difference. So, the docstring of the &lt;code&gt;DistributedDataParallel&lt;/code&gt; module is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Implements distributed data parallelism at the module level.&#xA;      This container parallelizes the application of the given module by&#xA;      splitting the input across the specified devices by chunking in the batch&#xA;      dimension. The module is &lt;strong&gt;replicated on each machine and each device&lt;/strong&gt;, and&#xA;      each such replica handles a portion of the input. During the backwards&#xA;      pass, gradients from each node are averaged.&#xA;      The batch size should be larger than the number of GPUs used locally. It&#xA;      should also be an integer multiple of the number of GPUs so that each chunk&#xA;      is the same size (so that each GPU processes the same number of samples).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the docstring for the &lt;code&gt;dataparallel&lt;/code&gt; is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Implements data parallelism at the module level.&#xA;      This container parallelizes the application of the given module by&#xA;      splitting the input across the specified devices by chunking in the batch&#xA;      dimension. In the forward pass, the module is replicated on each device,&#xA;      and each replica handles a portion of the input. During the backwards&#xA;      pass, gradients from each replica are summed into the original module.&#xA;      The batch size should be larger than the number of GPUs used. It should&#xA;      also be an integer multiple of the number of GPUs so that each chunk is the&#xA;      same size (so that each GPU processes the same number of samples).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/2?u=dawny33&quot; rel=&quot;noreferrer&quot;&gt;reply in the Pytorch forums&lt;/a&gt; was also helpful in understanding the difference between the both,&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2017-08-13T04:42:59.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22215" PostTypeId="1" AcceptedAnswerId="22231" CreationDate="2017-08-13T05:22:59.297" Score="4" ViewCount="3237" Body="&lt;p&gt;I have an e-commerce website where customers can purchase items directly from the site. I have training data which includes order id, user id, order number, days since prior order, product id, add to cart order, reordered...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict, for each user, what items he will purchase on his next order. I tried to use Naive Bayes, average purchase items per user and the following equation: &lt;code&gt;posterior ~ Bayes Factor x prior&lt;/code&gt; but the prediction outcome is not good and has many false positives and/or negatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe I can try to first train on the number of items a user will purchase then train on the specific items he will get but not sure will it get better results. &#xA;I think this can go in the multi label classification but has not used multi labels in classification before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using python with sklearn, pandas...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any better models I can use and how to train and predict variable multi labels and whether I can do it in sklearn? Keep in mind that the data is large and predicting using some of the classification algorithms in sklearn unfortunately takes huge amounts of memory so, any ideas on how to reduce memory consumption would also be useful. &lt;/p&gt;&#xA;" OwnerUserId="37883" LastEditorUserId="29575" LastEditDate="2017-08-13T22:58:47.043" LastActivityDate="2017-08-14T09:15:19.650" Title="How to predict user next purchase items" Tags="|machine-learning|scikit-learn|pandas|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22217" PostTypeId="1" CreationDate="2017-08-13T09:32:13.293" Score="1" ViewCount="29" Body="&lt;p&gt;I have implemented logistic regression using glmnet library. For hyper parameter tuning internal cross-validation is performed. I can fix the internal folds using set.seed so that foldid parameters can be given. I want to compare the predictive performance in SVM and Random Forest classifier. I want to know If there is a similar parameter in Random Forest and SVM so that I can fix internal folds for hyper parameter tuning in these functions.&lt;/p&gt;&#xA;" OwnerUserId="31599" LastActivityDate="2017-08-13T09:32:13.293" Title="How to fix internal folds for parameter tuning in random forest and SVM?" Tags="|machine-learning|random-forest|svm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22218" PostTypeId="1" CreationDate="2017-08-13T11:14:35.200" Score="1" ViewCount="24" Body="&lt;p&gt;Suppose that I train my image dataset on CNN, but the resolution of the image varies significantly on the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, should I scale the images up to the image that has the maximum resolution, or scale down to the lowest resolution? In that case should I scale up/down the whole images even if the highest likelihood of the whole samples are somewhere in the middle of the distribution of the resolution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or should I use another technique to deal with the varying resolution problem?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-08-13T11:14:35.200" Title="What should I do in case the resolution of the image varies in the dataset?" Tags="|deep-learning|image-classification|convolution|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22220" PostTypeId="1" CreationDate="2017-08-13T12:25:28.767" Score="1" ViewCount="40" Body="&lt;p&gt;It has been a while that I am trying to understand the dtw (dynamic time warping) algorithm but I steel can't figure out the loginc behind this algorithm..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many references it's explained that to compute the path between to time series I need to implement this recursivity: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PAG89.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PAG89.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have many questions here and I hope some one can give me an answer:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What we mean exactly by the &lt;em&gt;path&lt;/em&gt; between two time series ?&lt;/li&gt;&#xA;&lt;li&gt;What is the (cost path matrix described here:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YeVMh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YeVMh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Why not just compute the distances between a given point of &lt;code&gt;t&lt;/code&gt; and other points of &lt;code&gt;r&lt;/code&gt; and select the closest one an repeat this until matching all points from &lt;code&gt;t&lt;/code&gt; to points from &lt;code&gt;r&lt;/code&gt; ??&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any helpfull answer / comment.&lt;/p&gt;&#xA;" OwnerUserId="24657" LastActivityDate="2017-08-13T12:25:28.767" Title="DTW - What is fan-in nodes and why not just take the nearst points from each timeserie?" Tags="|time-series|distance|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22221" PostTypeId="2" ParentId="22212" CreationDate="2017-08-13T15:16:13.267" Score="8" Body="&lt;p&gt;A web search for &lt;code&gt;&quot;policy collapse&quot; &quot;reinforcement learning&quot;&lt;/code&gt; finds this question, a related one in stats.stackexchange.com &lt;em&gt;and&lt;/em&gt; the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A google books ngrams search for &lt;code&gt;policy collapse&lt;/code&gt; finds no references at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is hard to prove a negative, but I think this is not a widely used term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the comment does appear to be referring to a real phenomenon. That is where a reinforcement agent, instead of converging on the value functions for an optimal policy as it gains experience, actually diverges (and the parameters of the approximator will diverge too).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can happen when using non-linear function approximators to estimate action-values. More generally, it tends to happen when you have the following traits in your problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Using a function approximator, especially a non-linear one (although even linear function approximators can diverge)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Using a bootstrap method, e.g. Temporal Difference (TD) Learning (including SARSA and Q-learning), where values are updated from the same value estimator applied to successive steps. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Off-policy training. Attempting to learn the optimal policy whilst not behaving optimally (as in Q-Learning).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://incompleteideas.net/sutton/book/the-book-2nd.html&quot; rel=&quot;noreferrer&quot;&gt;Sutton and Barto's book&lt;/a&gt; this is called the &quot;deadly triad&quot;. If you do a web search for &lt;code&gt;&quot;deadly triad&quot; &quot;reinforcement learning&quot;&lt;/code&gt; you will find many more results. It is an ongoing area of research how best to combat the effect. In the &lt;a href=&quot;http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html&quot; rel=&quot;noreferrer&quot;&gt;paper that introduced the DQN model learning to play Atari games&lt;/a&gt;, the researchers applied two things that help stabilise against the effect:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Experience replay, where transitions are not learned from immediately, but put into a pool from which mini-batches are sampled to train the approximator.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Bootstrap estimates are made from a &quot;frozen&quot; copy of the learning network, updated every N training steps - i.e. when calculating the TD target $R + \gamma \hat{q}(S', A', \theta)$, use this old copy of the network.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From the comment section you linked, it appears even applying these things is not a guaranteed fix and takes some judgement. In that case it was increasing the mini-batch size for experience replay that helped to stabilise an agent playing a variant of the video game &lt;a href=&quot;https://en.wikipedia.org/wiki/Pong&quot; rel=&quot;noreferrer&quot;&gt;Pong&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-13T16:42:54.553" LastActivityDate="2017-08-13T16:42:54.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22222" PostTypeId="1" CreationDate="2017-08-13T15:38:55.350" Score="3" ViewCount="1794" Body="&lt;p&gt;I would use &lt;em&gt;Statsmodels SARIMAX&lt;/em&gt; with data having seasonality greater than one year.&#xA;In my case, I have seasonality equal &lt;em&gt;4&lt;/em&gt; years, therefore I have tried:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;smt.SARIMAX(endog, order=(p, d, q), seasonal_order=(P,D,Q,4))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it correct set &lt;code&gt;seasonality = 4&lt;/code&gt; when the seasonality is equal 4 years?&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2020-06-24T13:47:13.073" Title="SARIMAX with seasonality greater than one year" Tags="|python|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22223" PostTypeId="1" CreationDate="2017-08-13T16:41:04.193" Score="3" ViewCount="1632" Body="&lt;p&gt;Let’s explain the whole goal firstly, then go through the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is PubMed, I used about three categories of this collection and went through the abstract part(in each category there is 10 abstract file so totally I have 30 abstract)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After applying LDA on my data, for the evaluation process, to see what is the accuracy of the topics generated for each document, I evaluated that with OneVsRestClassifier in sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As OneVsRestClassifier is a classification method, it needs the data to have label, so I am going to explain how I generated the label for the output of LDA model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In LDA, I used the document-topic matrix and this matrix is like for each document there are some topics. for the topics, there is probability that shows with which probability this topic belongs to this document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the label for each row is the highest probability of the topics(indixes that shows the topic e.g. the probability in the first column shows the probability for the topic0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope the explanations are clear, if not, please let me know I will elaborate more with examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is the code that doing the above approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;&#xA;from nltk.tokenize import RegexpTokenizer&#xA;from nltk.stem.porter import PorterStemmer&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn.decomposition import LatentDirichletAllocation&#xA;from sklearn.feature_extraction.text import CountVectorizer&#xA;from sklearn.multiclass import OneVsRestClassifier&#xA;from sklearn.svm import LinearSVC&#xA;import numpy as np&#xA;from sklearn import metrics&#xA;&#xA;tokenizer = RegexpTokenizer(r'\w+')&#xA;&#xA;# Create p_stemmer of class PorterStemmer&#xA;lines=[]&#xA;p_stemmer = PorterStemmer()&#xA;lisOfFiles=[x[2] for x in os.walk(&quot;data&quot;)]&#xA;&#xA;fullPath = [x[0] for x in os.walk(&quot;data&quot;)]&#xA;&#xA;for j in lisOfFiles[2]:&#xA;    with open(os.path.join(fullPath[2],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;&#xA;for j in lisOfFiles[3]:&#xA;    with open(os.path.join(fullPath[3],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;for j in lisOfFiles[4]:&#xA;    with open(os.path.join(fullPath[4],j)) as f:&#xA;                    a=f.read()&#xA;                    lines.append(a)&#xA;&#xA;# compile sample documents into a list&#xA;doc_set = lines&#xA;&#xA;tf_vectorizer = CountVectorizer(max_features=1000,&#xA;                                stop_words='english')&#xA;&#xA;tf = tf_vectorizer.fit_transform(doc_set)&#xA;&#xA;# Trains the LDA models.&#xA;lda = LatentDirichletAllocation(n_topics=10, max_iter=5,&#xA;                                learning_method='online',&#xA;                                learning_offset=50.,&#xA;                                random_state=0)&#xA;&#xA;lda_x=lda.fit_transform(tf)&#xA;&#xA;#creating the labels&#xA;&#xA;new_y = np.argmax(lda_x, axis=1)&#xA;&#xA;Xtrain, Xvalidate, ytrain, yvalidate = train_test_split(lda_x, new_y, test_size=.3)&#xA;&#xA;predictclass=OneVsRestClassifier(LinearSVC(random_state=0)).fit(Xtrain, ytrain).predict(Xvalidate)&#xA;yacc=metrics.accuracy_score(yvalidate,predictclass)&#xA;print (yacc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when I run this code, it seems like most of the time gives me 100 accuracy :|&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and when I run it for several times the accuracy is changing, and never drop below 66 but it is changing between 66 and 100, and most of the time is 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand these changing is because of the random sampling of LDA. firstly I believe something is wrong with my code that I am getting that much good accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, if the accuracy is changing like this, which one is reliable, is there any approach to fix it out in one output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same process also is happening for my NMF model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;My Doubt&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my only doubt was the data I was working on. as I explained in the first paragraph, it includes 30 abstract file which I have transferred to a list, so I have a list seperated by comma, but if the abstract file has already had the comma, how it can realize the correct document?!!!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but I tested it by iterating through the list and getting the size of the list and it was 30**&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when I changed the data from 300 abstract to 3000 abstract, Accuracy is still changing BUT just from 93 to 97, I mean the difference has decressed&lt;/p&gt;&#xA;" OwnerUserId="37510" LastEditorUserId="37510" LastEditDate="2017-08-14T04:20:56.203" LastActivityDate="2020-10-22T17:41:30.420" Title="why the accuracy of LDA model is always changing and also is high" Tags="|scikit-learn|svm|multiclass-classification|topic-model|lda|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22225" PostTypeId="1" AcceptedAnswerId="33115" CreationDate="2017-08-13T20:53:43.917" Score="1" ViewCount="534" Body="&lt;p&gt;I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. However, what I find really weird is that they are all roughly the same size! Each topic catches around 5% of tokens and 5% of texts. In terms of the tokens, the smallest topic is 4.5% the largest 5.5%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody suggest if this a 'normal' behaviour? This is the code I'm using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ldafitted &amp;lt;- LDA(sentences.tm, k = K, method = &quot;Gibbs&quot;,&#xA;             control = list(alpha = 0.1, # default is 50/k which would be 2.5.  a lower alpha value places more weight on having each document composed of only a few dominant topics&#xA;                            delta = 0.1, # default 0.1 is suggested in Griffiths and Steyvers (2004).&#xA;                            estimate.beta = TRUE,&#xA;                            verbose = 50, # print every 50th draw to screen&#xA;                            seed = 5926696,&#xA;                            save = 0,    # can save model every xth iteration&#xA;                            iter = 5000, &#xA;                            burnin = 500,&#xA;                            thin = 5000, #  every thin iteration is returned for iter iterations. Standard is same as iter&#xA;                            best = TRUE)) #only the best draw is returned&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In short: My question is if there are circumstances under which it is reasonable that Latent Dirichlet allocation will cluster text in topics of equal size? Or is it something I should be worried if it happens?&lt;/p&gt;&#xA;" OwnerUserId="37898" LastEditorUserId="37898" LastEditDate="2017-08-15T22:16:53.817" LastActivityDate="2018-09-05T21:45:57.533" Title="Equally sized topics in Latent Dirichlet allocation" Tags="|topic-model|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22226" PostTypeId="2" ParentId="5706" CreationDate="2017-08-13T23:10:53.433" Score="28" Body="&lt;p&gt;ReLU neurons output zero and have zero derivatives for all negative inputs. So, if the weights in your network always lead to negative inputs into a ReLU neuron, that neuron is effectively not contributing to the network's training. Mathematically, the gradient contribution to the weight updates coming from that neuron is always zero (see the Mathematical Appendix for some details).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the chances that your weights will end up producing negative numbers for &lt;em&gt;all&lt;/em&gt; inputs into a given neuron? It's hard to answer this in general, but one way in which this can happen is when you make too large of an update to the weights. Recall that neural networks are typically trained by minimizing a loss function $L(W)$ with respect to the weights using gradient descent. That is, weights of a neural network are the &quot;variables&quot; of the function $L$ (the loss depends on the dataset, but only implicitly: it is typically the sum over each training example, and each example is effectively a constant). Since the gradient of any function always points in the direction of steepest increase, all we have to do is calculate the gradient of $L$ with respect to the weights $W$ and move in the opposite direction a little bit, then rinse and repeat. That way, we end up at a (local) minimum of $L$. Therefore, if your inputs are on roughly the same scale, a large step in the direction of the gradient can leave you with weights that give similar inputs which can end up being negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, what happens depends on how information flows through the network. You can imagine that as training goes on, the values neurons produce can drift around and make it possible for the weights to kill all data flow through some of them. (Sometimes, they may leave these unfavorable configurations due to weight updates earlier in the network, though!). I explored this idea in a &lt;a href=&quot;https://intoli.com/blog/neural-network-initialization/&quot; rel=&quot;noreferrer&quot;&gt;blog post about weight initialization&lt;/a&gt; -- which can also contribute to this problem -- and its relation to data flow. I think my point here can be illustrated by a plot from that article:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://intoli.com/blog/neural-network-initialization/img/relu-output-progression-violinplot.png&quot; alt=&quot;Activations in a ReLU MLP with different initialization strategies&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The plot displays activations in a 5 layer Multi-Layer Perceptron with ReLU activations after one pass through the network with different initialization strategies. You can see that depending on the weight configuration, the outputs of your network can be choked off.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Mathematical Appendix&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Mathematically if $L$ is your network's loss function, $x_j^{(i)}$ is the output of the $j$-th neuron in the $i$-th layer, $f(s) = \max(0, s)$ is the ReLU neuron, and $s^{(i)}_j$ is the linear input into the $(i+1)$-st layer, then by the chain rule the derivative of the loss with respect to a weight connecting the $i$-th and $(i+1)$-st layers is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac{\partial L}{\partial w_{jk}^{(i)}} = \frac{\partial L}{\partial x_k^{(i+1)}} \frac{\partial x_k^{(i+1)}}{\partial w_{jk}^{(i)}}\,.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first term on the right can be computed recursively. The second term on the right is the only place directly involving the weight $w_{jk}^{(i)}$ and can be broken down into&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align*}&#xA;\frac{\partial{x_k^{(i+1)}}}{\partial w_{jk}^{(i)}} &amp;amp;= \frac{\partial{f(s^{(i)}_j)}}{\partial s_j^{(i)}} \frac{\partial s_j^{(i)}}{\partial w_{jk}^{(i)}} \\&#xA;&amp;amp;=f'(s^{(i)}_j)\, x_j^{(i)}.&#xA;\end{align*}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this you can see that if the outputs are always negative, the weights leading into the neuron are not updated, and the neuron does not contribute to learning.&lt;/p&gt;&#xA;" OwnerUserId="37899" LastActivityDate="2017-08-13T23:10:53.433" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22228" PostTypeId="1" AcceptedAnswerId="22232" CreationDate="2017-08-14T03:30:46.653" Score="3" ViewCount="59" Body="&lt;p&gt;I have a locus &lt;code&gt;L&lt;/code&gt; of points (lat, long). And I would like to find N=10 points (let's call them warehouses) such that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$loss = \sum_{l \in L} maximum_{w \in W}(distance(l, w))^2 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is minimized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a documented algorithm or approach that solves this problem? Right now I am thinking Excel may be able to handle this task. However I have too much data for Excel and will need to implement this in Python / Pandas. &lt;/p&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2017-08-14T12:52:27.800" Title="Find points on a map close to given points" Tags="|algorithms|feature-extraction|feature-engineering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22229" PostTypeId="2" ParentId="13465" CreationDate="2017-08-14T06:49:54.737" Score="1" Body="&lt;pre&gt;&lt;code&gt;from sklearn import tree&#xA;&#xA;features = [[2200,1], [1500,1], [1800,1], [900,2], [1000,2]]&#xA;&#xA;labels = ['SUV', 'SUV', 'SUV', 'hatchback', 'hatchback']&#xA;&#xA;clf = tree.DecisionTreeClassifier()&#xA;&#xA;clf = clf.fit(features, labels)&#xA;&#xA;print(clf.predict([[1350, 1]]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37915" LastEditorUserId="24000" LastEditDate="2017-08-14T10:21:16.280" LastActivityDate="2017-08-14T10:21:16.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22230" PostTypeId="1" CreationDate="2017-08-14T08:27:40.943" Score="4" ViewCount="1498" Body="&lt;p&gt;I am recently watching some tutorials for deep learning from Dr Andrew Ng on Youtube. Link is here&lt;a href=&quot;https://www.youtube.com/watch?v=1ZhtwInuOD0&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Youtube video&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a concept of &lt;em&gt;number of features&lt;/em&gt; in &lt;strong&gt;convolutional neural network&lt;/strong&gt; in &lt;strong&gt;TensorFlow&lt;/strong&gt;'s &lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/pros#convolution_and_pooling&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't quite understand why the &lt;em&gt;feature&lt;/em&gt; is 32 or 64 here in conv layer1 or layer2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I came to the &lt;a href=&quot;https://youtu.be/1ZhtwInuOD0?t=9m8s&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;, there is also the concept of &lt;em&gt;Quadratic features&lt;/em&gt;. It is calculated as &lt;em&gt;3 million&lt;/em&gt; however. But how is it calculated?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the two &lt;em&gt;features&lt;/em&gt; related in concept?&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="26019" LastEditDate="2020-04-14T15:44:58.240" LastActivityDate="2023-12-19T16:22:27.613" Title="how to calculate quadratic features in computer vision Neural Network" Tags="|neural-network|deep-learning|tensorflow|convolution|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22231" PostTypeId="2" ParentId="22215" CreationDate="2017-08-14T09:15:19.650" Score="2" Body="&lt;p&gt;First of all you have to realize these kind of problems have large amounts of noise compared to signal, because predicting what someone will buy based on a very small window of information is difficult. That said, you are throwing away a lot of information with your current approach. Temporal aspects include a ton of information, for example the sequence in which items were bought etcetera. While this is a lot more complicated than what you are describing now, you could look into recurrent neural networks where you feed history up to the point of prediction as a sequence and predict the item they will buy next as softmax classification. This will depend on the amount of products that you offer whether this is feasible or not. Another advantage is that so-called 'out-of-core' training is relatively easy with neural networks due to the iterative training of batches. Multi-label is also clean, you can just add a number of labels at the end of your graph if necessary.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-14T09:15:19.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22232" PostTypeId="2" ParentId="22228" CreationDate="2017-08-14T09:39:36.980" Score="2" Body="&lt;p&gt;I can tell you how I would do it, but there is almost certainly a faster implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you start with, for each point in $L$, the distances to each warehouse, $w \in W$. These distances should be calculated by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Haversine_formula&quot; rel=&quot;nofollow noreferrer&quot;&gt;haversine formula&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find the distance to the $N$th closest point in $w$ by using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Quickselect&quot; rel=&quot;nofollow noreferrer&quot;&gt;quickselect&lt;/a&gt; algorithm. This is very similar to the quicksort algorithm but only sorting the parts that you care about. The average case for quickselect is $O(N)$ but you'll need to repeat for each $l \in L$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that, since the square is monotonic for positive distances, you only need to minimise&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{l \in L} maximum_{w \in W}(distance(l, w)) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a handy implementation of the quickselect algorithm on &lt;a href=&quot;http://www.koderdojo.com/blog/quickselect-algorithm-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;KoderDojo&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30964" LastActivityDate="2017-08-14T09:39:36.980" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22233" PostTypeId="1" AcceptedAnswerId="22235" CreationDate="2017-08-14T09:42:39.843" Score="3" ViewCount="6376" Body="&lt;p&gt;I have a datset with Scores and Categories and I would like to calculate the Standard Deviation of these scores, per category. The data look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category    Score    &#xA;AAAA        1&#xA;AAAA        3&#xA;AAAA        1&#xA;BBBB        1&#xA;BBBB        100&#xA;BBBB        159&#xA;CCCC        -10&#xA;CCCC        9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I would then like is the Standard Deviation of each Category. I know that with numpy I can use the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;numpy.std(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the example I can find only have this relating to a list and not a range of different categories in a DataFame.&lt;/p&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="8432" LastEditDate="2017-08-14T12:58:01.440" LastActivityDate="2019-11-25T13:44:03.973" Title="Calculating the Standard Deviation by category using Python" Tags="|python|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22234" PostTypeId="1" AcceptedAnswerId="22237" CreationDate="2017-08-14T09:44:03.367" Score="4" ViewCount="3571" Body="&lt;p&gt;The F beta formula according to the wikipedia is &quot;The weighted harmonic mean of precision and recall&quot;. I can not understand why in the left part of equation there is beta and in the right one is beta^2:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/swW0x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/swW0x.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my mind if I claim that Precision is 5 times important than Recall:&#xA;F beta = (1+beta)/(beta/P+1/R)=(1+beta)&lt;em&gt;P&lt;/em&gt;R/(beta*R+P), where beta=0.2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this right? &lt;/p&gt;&#xA;" OwnerUserId="37921" LastEditorUserId="8432" LastEditDate="2017-08-14T13:29:08.793" LastActivityDate="2021-01-08T20:19:20.427" Title="Explanation of the F beta formula" Tags="|classification|statistics|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22235" PostTypeId="2" ParentId="22233" CreationDate="2017-08-14T10:02:54.000" Score="6" Body="&lt;p&gt;You can easily do this using pandas:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;df = pd.DataFrame([[&quot;AA&quot;, 1], [&quot;AA&quot;, 3], [&quot;BB&quot;, 3], [&quot;CC&quot;, 5], [&quot;BB&quot;, 2], [&quot;AA&quot;, -1]])&#xA;df.columns = [&quot;Category&quot;, &quot;Score&quot;]&#xA;print df.groupby(&quot;Category&quot;).apply(np.std)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13833" LastActivityDate="2017-08-14T10:02:54.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22237" PostTypeId="2" ParentId="22234" CreationDate="2017-08-14T11:04:49.873" Score="3" Body="&lt;p&gt;That's a great question, because on its face it seems like the weight should be $\beta$ alone, and, it should be in front of recall. The answer is in the text from which that reference is taken, on page 133: &lt;a href=&quot;http://www.dcs.gla.ac.uk/Keith/pdf/Chapter7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.dcs.gla.ac.uk/Keith/pdf/Chapter7.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition is designed to make the metric indifferent to a &lt;em&gt;change&lt;/em&gt; in precision or recall when $P/R = \beta$. That is, $F_\beta$ increases by the same amount when either precision or recall increases, at the point where precision is already $\beta$ times bigger than recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition does indeed weight recall more highly as you can verify. Honestly on re-reading the text above, I was confused, because I don't see how it makes sense to think of &quot;equilibrium&quot; as the point where precision is much bigger, if recall matters more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I plugged in the formula to &lt;a href=&quot;http://www.wolframalpha.com/input/?i=derivative%20((1%2BB%5E2)%20*%20P%20*%20R)%20%2F%20(B%5E2%20*%20P%20%2B%20R)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wolfram Alpha&lt;/a&gt;, and:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/sZpG2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sZpG2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hm. These are only equal if $R/P = \beta$! I think the paper may have misstated this then, or else I've really missed something. It's a formula whose value changes at the same rate with respect to precision or recall, when recall is already $\beta$ times larger, and in that sense it corresponds to treating recall as $\beta$ time more important.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-08-14T11:04:49.873" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="22239" PostTypeId="2" ParentId="17254" CreationDate="2017-08-14T11:54:16.657" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I am thus wondering how such 3D (3×3×10 in my toy example) convolution filters are usually displayed as 2D images (i.e. sum or mean, ...)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; The 3x3x10 image is displayed as 10 3x3 images&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer Version&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you start with a 32x32x3 image&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You decide to use a filter of shape 5x5x3 with an output depth of 15 (with no padding and stride 1).&#xA;i.e., you will convert the 32x32x3 image to a 28x28x15 image.&#xA;What does this depth of 15 really mean?&#xA;Each of the 15 dimensions represents one filters output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e., when we apply an output depth of 15 for a 5x5x3 filter, we take 15 5x5x3 filters and apply them to the image. From them, we get 15 28x28x1 images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of these 15 $28$$\times$$28$$\times$$1$ images are stacked on top of each other to make a 28x28x15 image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, &lt;strong&gt;to answer your question&lt;/strong&gt; : the $N$$\times$$N$$\times$$15$ images are split into 15 NxNX1 images and displayed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: you'll notice in many CNN visualisations as the depth of the convolutional pyranmid increases, the number of small images increases.&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-08-14T11:54:16.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22240" PostTypeId="2" ParentId="22228" CreationDate="2017-08-14T12:52:27.800" Score="1" Body="&lt;p&gt;Scipy has tools most most of this already.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;locations = train[['latitude', 'longitude']].values&#xA;center = locations.mean(axis=0)&#xA;warehouses = np.repeat(np.expand_dims(center, 0), 20, 0)&#xA;warehouses = warehouses.flatten()&#xA;&#xA;def Distances(warehouses):&#xA;    warehouses = np.reshape(warehouses, [20, 2])&#xA;    distances = scipy.spatial.distance.cdist(locations, warehouses)&#xA;    closests = distances.min(axis=1)&#xA;    other_way = distances.min(axis=0)&#xA;    return np.append(closests, other_way)&#xA;&#xA;x = scipy.optimize.least_squares(Distances, warehouses, verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2017-08-14T12:52:27.800" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22241" PostTypeId="2" ParentId="22153" CreationDate="2017-08-14T13:16:03.390" Score="1" Body="&lt;p&gt;Given 35 numeric features and 3 classes, perhaps first try a variant of SVM or random forest to get a handle on the data - neither of these are deep learning, but they are fast and good for benchmarking/troubleshooting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;SVMs: @Media mentions a linear support vector machine, but if your data isn't linearly separable then you can also employ a nonlinear kernel or soft margin classifier: &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/svm.html&lt;/a&gt;  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Random Forests: an ensemble method fitting decision trees to your data, there are a ton of variants (bagging, boosting):&#xA;&lt;a href=&quot;https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you've already done that, you could use a restricted boltzmann machine to build a generative model of the data, or an unsupervised algorithm like an autoencoder or self-organising map to reduce the dimensionality of the data, then use that as the input to your MLP/SVM/softmax classifier.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Restricted Boltzmann Machines: good brief description of workflow here &lt;a href=&quot;https://datascience.stackexchange.com/questions/14568/how-to-use-rbm-for-classification&quot;&gt;How to use RBM for classification?&lt;/a&gt; and more detail on training from Hinton here: &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Self-organising Maps: Good page on what: &lt;a href=&quot;http://www.ai-junkie.com/ann/som/som1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.ai-junkie.com/ann/som/som1.html&lt;/a&gt; and implemented in tensorflow here: &lt;a href=&quot;https://codesachin.wordpress.com/2015/11/28/self-organizing-maps-with-googles-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://codesachin.wordpress.com/2015/11/28/self-organizing-maps-with-googles-tensorflow/&lt;/a&gt; but of course you can choose the structure of your output.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are other options out there but it might be worth finding out what sort of results you get on the faster and more explanatory algorithms before building a full CNN etc.&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2017-08-14T13:16:03.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22242" PostTypeId="1" CreationDate="2017-08-13T23:04:42.820" Score="13" ViewCount="1559" Body="&lt;p&gt;I'm reading the cornerstone paper &lt;a href=&quot;https://arxiv.org/pdf/1409.3215v1.pdf%3B&quot; rel=&quot;noreferrer&quot;&gt;Sequence to Sequence Learning&#xA;with Neural Networks&lt;/a&gt; by Ilya Sutskever and Quoc Le. On the first page, it briefly mentions that: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A surprising example of the power of DNNs is their ability to sort&#xA;N N-bit numbers using only 2 hidden layers of quadratic size &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone briefly outline how to sort numbers using only 2 hidden layers?&lt;/p&gt;&#xA;" OwnerUserId="37932" OwnerDisplayName="Toussaint Louverture" LastActivityDate="2023-06-21T17:21:55.893" Title="Sort numbers using only 2 hidden layers" Tags="|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22244" PostTypeId="1" CreationDate="2017-08-14T14:33:54.940" Score="1" ViewCount="162" Body="&lt;p&gt;Which method is implemented when using the periodogram method from the 'TSA' package on R? Is it Welch or Bartlett?&lt;/p&gt;&#xA;" OwnerUserId="37484" LastActivityDate="2017-08-14T14:33:54.940" Title="TSA library on R, periodogram method" Tags="|r|time-series|feature-extraction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22245" PostTypeId="1" CreationDate="2017-08-14T14:54:34.497" Score="1" ViewCount="579" Body="&lt;p&gt;I have an HP au111tx laptop with Nvidia 940mx graphics card.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible for me to use this graphics card exclusively for ML. I want to use the integrated graphics card for normal display. I can use Windows or Ubuntu.&lt;/p&gt;&#xA;" OwnerUserId="37934" LastEditorUserId="8432" LastEditDate="2017-08-15T09:59:04.077" LastActivityDate="2017-08-15T09:59:04.077" Title="Trying to use GPU of laptop for TensorFlow" Tags="|tensorflow|gpu|" AnswerCount="0" CommentCount="0" ClosedDate="2017-08-14T15:01:59.233" ContentLicense="CC BY-SA 3.0" />
  <row Id="22246" PostTypeId="1" CreationDate="2017-08-14T14:57:05.480" Score="2" ViewCount="328" Body="&lt;p&gt;i am trying out multiple approaches to generate domain specific data (sentences).For e.g. if i am building a data set to train a chatbot in the travel domain, i would like to take pairs of sentences like &quot;i would like to book a return trip to Philly commencing on the 24th&quot; and &quot;i am traveling to Phialdelphia for a conference for 3 days beginning the 24th&quot; and use these as the enc and decoder inputs. So once the training is complete, i can feed it the first sentence and generate multiple samples, set a threshold (rather than argmaxing it) and have a fairly large corpus of similar sentences that i could train another classifier with. I personally think its a good analogy to machine translation tasks (where the sampled output is another language but in some sense semantically the same sentence)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While i am trying it out, can anyone tell me if they have tried this approach and seen decent results ? Also any recommendations/ pointers on better approaches? thanks a ton!&lt;/p&gt;&#xA;" OwnerUserId="29157" LastActivityDate="2017-08-14T15:20:46.030" Title="using LSTM encoder decoder framework to generate sentences" Tags="|nlp|text-mining|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22247" PostTypeId="1" AcceptedAnswerId="22484" CreationDate="2017-08-14T15:04:25.663" Score="1" ViewCount="1441" Body="&lt;p&gt;I have written a function for 10 fold crossvalidation that I want to use for different models, e.g PPR, MARS. However, I get an error when running it and I cannot figure out why it does not work? My CV function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    cv10 &amp;lt;- function(reg.fn, formula, dataset, ...)&#xA;{&#xA;  set.seed(201)&#xA;&#xA;  ### Number of observations&#xA;  nrow &amp;lt;- nrow(dataset)&#xA;&#xA;  ### Create a permutation of the observations indices&#xA;  Ind &amp;lt;- sample.int(nrow,nrow, replace = FALSE)&#xA;&#xA;  ### Compute the size of each of the 10 folds&#xA;  M &amp;lt;- nrow / 10 # 'fold size'&#xA;&#xA;  ### Initialize the score&#xA;  score &amp;lt;- 0&#xA;  ### The first fold will then contain the observations which correspond to..&#xA;  ### ..the indices of the first M elements of Ind.&#xA;&#xA;  for(i in 1:10){&#xA;    beg &amp;lt;- i*M&#xA;    end &amp;lt;- (i+1)*M&#xA;    ### Data to train the model&#xA;    data.train &amp;lt;- dataset[Ind[-beg:-end],]&#xA;&#xA;    ### Data to test the model&#xA;    data.fold &amp;lt;- dataset[Ind[beg:end],]&#xA;&#xA;    ### Fit the model&#xA;    model &amp;lt;- reg.fn(formula,data=data.train,...)&#xA;    predicted.y &amp;lt;- predict(model,data.fold)&#xA;&#xA;    ### Update the CV-score&#xA;    score &amp;lt;- sum((predicted.y - data.fold[,1])^2) / M&#xA;  }&#xA;&#xA;  return(score/10)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Testing using ppr:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    cv.scores &amp;lt;- numeric(10)&#xA;### Some code&#xA;for(i in 1:10){&#xA;  score &amp;lt;- cv10(reg.fn = ppr, formula = y~.,&#xA;                dataset = data, nterms=i)&#xA;  cv.scores[i] &amp;lt;- scores&#xA;}&#xA;cv.scores&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The traceback:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;  Error in matrix(NA, length(keep), object$q, dimnames = list(rn,&#xA;&amp;gt; object$ynames)) : &#xA;&amp;gt;       length of 'dimnames' [1] not equal to array extent &#xA;&amp;gt;     4.&#xA;&amp;gt;     matrix(NA, length(keep), object$q, dimnames = list(rn, object$ynames)) &#xA;&amp;gt;     3.&#xA;&amp;gt;     predict.ppr(model, data.fold) &#xA;&amp;gt;     2.&#xA;&amp;gt;     predict(model, data.fold) &#xA;&amp;gt;     1.&#xA;&amp;gt;     cv10(reg.fn = ppr, formula = y ~ ., dataset = data, nterms = i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The data I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; structure(list(y = c(23.0551546516262, 27.8893494373006, 3.32468370559938, &#xA;    -13.5852336127512, -5.14668013186906, -0.489523212484223, -14.328750654513, &#xA;    -4.26428395686341, -2.75486620989581, 17.3107345018601, 25.6193450849393, &#xA;    0.605103858286016, -1.30909806542865, 2.03575942172917, -19.1193524499977, &#xA;    -1.46508279385589, 2.65778970954973, 14.8513018374104, -2.87449028138997, &#xA;    1.37368992108124, -1.43518738939116, 0.0199676357940499, -1.549025998582, &#xA;    -4.06263285631006, -9.15130335901099, -2.62794216480131, -1.68473200963303, &#xA;    3.15144283445608, 7.78027589015824, 9.09732626327383), x1 = c(0.286060694657523, &#xA;    -0.344546030966432, 0.325763726232689, -1.69658096808073, -1.2854825202758, &#xA;    -0.0750318862014798, 0.266937353823139, 0.0559340444850217, -2.30403430891787, &#xA;    0.189004139305415, 0.693296170158882, 0.223809355083932, 0.398456942903131, &#xA;    1.01347438447768, -0.64785307166209, 0.648452713333917, 0.207342703528518, &#xA;    0.0643901392726141, 0.669380920067964, -0.374254446133507, -0.244000842201787, &#xA;    -0.988253138922366, 1.24206047974719, -1.68266602919039, 1.44289062580162, &#xA;    -0.465439746975312, 0.693661499094998, -0.0877255722586039, -0.955080382553146, &#xA;    0.170100884691593), x2 = c(-0.343601401483176, -0.924078839603673, &#xA;    0.973710320640175, 0.0267187344544633, -1.36283892301834, 0.105184057636645, &#xA;    -0.644019900369909, 0.960031901250783, 0.147336523178527, 0.339467057535232, &#xA;    -0.192287076626924, 0.0722969316029643, 0.389789911800799, -0.328247051156339, &#xA;    -0.090450711707476, 0.716681577815978, 0.0626860575507786, -0.69236622624416, &#xA;    0.584444051353438, -0.0911664147267412, -0.315213328094698, -0.0806856079787168, &#xA;    0.484583750517842, -0.120406402869962, 0.596077475841207, -0.36353784662963, &#xA;    -0.780093462571257, 0.324679908484668, 0.508548510215705, -0.193595813912055&#xA;    ), x3 = c(0.982327855388361, 0.624091435911063, 0.621531522270016, &#xA;    -0.902870741076395, 0.931325903563023, -1.05264178470207, 0.307132555544596, &#xA;    0.275469955530981, 2.78596687577565, -0.590390951909848, -0.0257046477898407, &#xA;    -0.122008374353289, 0.455026913225061, -0.607514744574133, 0.595817459312108, &#xA;    1.48223488775224, 0.636854208609479, 0.201054337281812, -0.716437866742046, &#xA;    -2.30960460962945, -1.11690418809942, 0.296611889529358, 0.992033628272787, &#xA;    -0.769290105905667, -1.4112664763812, 0.972758797977034, 0.680563892580633, &#xA;    0.0312007101558726, 2.40109797772769, 0.27149586035907), x4 = c(2.87744884192944, &#xA;    2.97037391737103, 2.04590974515304, -2.09065303439274, -0.886272139381617, &#xA;    0.258417838253081, -2.48789734393358, -1.14431498106569, 1.52785618370399, &#xA;    2.43856811150908, 2.88160788919777, 0.143826744519174, -1.32458955561742, &#xA;    0.850324050989002, -2.63397432630882, -0.270683331415057, 1.85416122945026, &#xA;    2.19268380571157, -1.33175755385309, 1.08762756781653, 0.7014160878025, &#xA;    0.907778979744762, -1.3183526317589, 0.718872689176351, -2.21834870846942, &#xA;    -0.750489700119942, -0.889076801016927, 1.39292777515948, 2.34955989941955, &#xA;    2.1975970286876), x5 = c(1.48984236368162, 0.869139640762428, &#xA;    0.748845036625717, 0.351786000608901, -1.47779050566991, -2.3154451409239, &#xA;    2.20221698212952, 0.414262887380592, 0.244955910040375, 0.429121363729595, &#xA;    -0.317306195296495, -1.38016320237183, 0.694020488858179, 0.305431051706151, &#xA;    -0.398558943204744, -1.00163421976715, 1.29024064725421, -0.770948417017754, &#xA;    0.741664981312622, 0.169399870781162, -1.35676745536567, 0.471865193264912, &#xA;    0.960859048309877, 1.46760491067668, 1.4378809852526, 0.0349201858899876, &#xA;    -1.42177690061078, -1.43127605517511, -0.101638629745238, 1.49972397311187&#xA;    )), .Names = c(&quot;y&quot;, &quot;x1&quot;, &quot;x2&quot;, &quot;x3&quot;, &quot;x4&quot;, &quot;x5&quot;), row.names = c(NA, &#xA;    30L), class = &quot;data.frame&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31086" LastEditorUserId="471" LastEditDate="2017-08-22T17:37:14.247" LastActivityDate="2017-08-22T17:46:46.687" Title="R - Function for 10 fold crossvalidation" Tags="|r|cross-validation|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22249" PostTypeId="2" ParentId="22246" CreationDate="2017-08-14T15:20:46.030" Score="1" Body="&lt;p&gt;I don't know about any research on this topic, but I have thought about similar ideas before. Let's assume the decoder is some RNN that outputs probabilites conditioned on the encoder vector and on what it has generated so far, $P(Word|EncodedVector, DecodedSoFar)$. What you can do is a type of beam search (there is research on this), which is basically greedily take the most likely option so far, but keep all the paths in memory while they still have the potential for a likely enough sentence. The probability of a sequence is just the product of all the predictions that we have sampled. Let's say our model has the following properties (we ignore the encoder):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A) = 0.3$,  $P(B) = 0.5$, $P(End) = 0.2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, condtioned on one sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|A) = 0.6$, $P(B|A)=0.1$, $P(End|A)=0.3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A|B)=0.7$, $P(B|B)=0.2$, $P(End|B)=0.1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's say when we have two samples the probability of ending is always 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means we get the following probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(End) = 0.2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(A) = 0.3 * 0.3 = 0.09$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(B) = 0.5 * 0.1 = 0.05$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(AA) = 0.3 * 0.6 * 1. = 0.18$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(AB) = 0.3 * 0.1 * 1. = 0.03$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(BA) = 0.5 * 0.7 * 1. = 0.35$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(BB) = 0.5 * 0.2 * 1. = 0.1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But we don't want to iterate over all possibilities, this grows way too quickly. What we can do is build a tree which we can bound by our threshold. Let's say we only want samples with at least probability $0.15$. Our root node is empty, with probability 1. Our terminal nodes are the ones that have $End$ as the last part of the sequence. We start growing our tree with the most likely scenario, which is $B$, which means we have $P(empty)=1$, $P(B)=0.5$. For our next option, we can either look at another start than B or by expanding upon B. $P(A)=0.3$ but $P(BA)=0.35$ so we first expand $B$. We now have $P(empty)=1$, $P(B)=0.5$ and $P(BA)=0.35$. The next step is the highest probability, staying at 0.35 by ending $P(AB-end)=0.35$. The next step is expanding our root by going to $P(A)=0.3$. The next step is expanding root with $P(End)=0.2$. Our next step would be $P(AA)=0.18$ but this is below our threshold. We know there are no sequences that are more likely because of the way we have built our tree, which means we have all sequences above the given threshold. You can use something similar to generate the top &lt;strong&gt;k&lt;/strong&gt; most likely samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Be aware that this does bias towards shorter sentences if your model allows for them, because every probability decreases the total probability. I have not really thought about how much of this is a problem, it also depends a lot on your generating RNN. You could think of some 'penalties' for short ones but you want the metric to be monotonically decreasing because else you cannot bound your tree properly. &lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-14T15:20:46.030" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22250" PostTypeId="1" AcceptedAnswerId="22258" CreationDate="2017-08-14T16:42:07.040" Score="20" ViewCount="25460" Body="&lt;p&gt;I'm converting a corpus of text documents into word vectors for each document. I've tried this using a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; and a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html&quot; rel=&quot;noreferrer&quot;&gt;HashingVectorizer&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that a &lt;code&gt;HashingVectorizer&lt;/code&gt; does not take into consideration the &lt;code&gt;IDF&lt;/code&gt; scores like a &lt;code&gt;TfidfVectorizer&lt;/code&gt; does. The reason I'm still working with a &lt;code&gt;HashingVectorizer&lt;/code&gt; is the flexibility it gives while dealing with huge datasets, as explained &lt;a href=&quot;https://stackoverflow.com/questions/30024122/what-is-the-difference-between-hashing-vectorizer-and-count-vectorizer-when-eac&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/questions/25145552/tfidf-for-large-dataset&quot;&gt;here&lt;/a&gt;. (My original dataset has 30 million documents)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am working with a sample of 45339 documents, so, I have the ability to work with a &lt;code&gt;TfidfVectorizer&lt;/code&gt; also. When I use these two vectorizers on the same 45339 documents, the matrices that I get are different. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;hashing = HashingVectorizer()&#xA;with LSM('corpus.db')) as corpus:&#xA;    hashing_matrix = hashing.fit_transform(corpus)&#xA;print(hashing_matrix.shape) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;hashing matrix shape (45339, 1048576)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tfidf = TfidfVectorizer()&#xA;with LSM('corpus.db')) as corpus:&#xA;    tfidf_matrix = tfidf.fit_transform(corpus)&#xA;print(tfidf_matrix.shape) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;tfidf matrix shape (45339, 663307)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I want to understand better the differences between a &lt;code&gt;HashingVectorizer&lt;/code&gt; and a &lt;code&gt;TfidfVectorizer&lt;/code&gt;, and the reason why these matrices are in different sizes - particularly in the number of words/terms. &lt;/p&gt;&#xA;" OwnerUserId="2647" LastEditorUserId="83275" LastEditDate="2021-12-12T08:25:35.313" LastActivityDate="2021-12-12T09:38:30.180" Title="What is the difference between a hashing vectorizer and a tfidf vectorizer" Tags="|nlp|scikit-learn|text-mining|tfidf|hashingvectorizer|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22251" PostTypeId="2" ParentId="15732" CreationDate="2017-08-14T17:06:09.210" Score="2" Body="&lt;p&gt;Name parsing does not appear to built-in to &lt;a href=&quot;https://stanfordnlp.github.io/CoreNLP/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CoreNLP&lt;/a&gt;.&#xA;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option is writing a series of Regular Expression using &lt;a href=&quot;https://nlp.stanford.edu/software/tokensregex.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford TokensRegex&lt;/a&gt; to parse and label name tokens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is using a third party package, such as &lt;a href=&quot;https://pypi.python.org/pypi/nameparser&quot; rel=&quot;nofollow noreferrer&quot;&gt;nameparser&lt;/a&gt; in Python.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-14T17:06:09.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22252" PostTypeId="2" ParentId="17804" CreationDate="2017-08-14T17:17:02.387" Score="0" Body="&lt;p&gt;There are several ways of approaching improving that word2vec model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try removing all the commas, creating a continuous block of text to train word2vec. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also a concept called &quot;fine tuning&quot;, it is a variation of transfer learning. Find a set of pre-trained word2vec weights and use those as the initial weights (word2vec weights are typically initialized as random). Then update the weights with your dataset. The pre-trained weights give the general patterns in words and better embed rare words. The fine tuning will improve the generic embeddings to model your particular dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly when tokenizing your dataset make sure you create the correct &lt;a href=&quot;https://en.wikipedia.org/wiki/Collocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;collocations&lt;/a&gt;. For example, &quot;Brad Pitt&quot; should be &quot;brad_pitt&quot; (instead of &quot;brad&quot; and &quot;pritt&quot; separately). You can also recode all relevant mentions of &quot;Brad&quot; into &quot;brad_pitt&quot;. Collocations carry much of the semantic meaning in proper nouns.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-14T17:17:02.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22253" PostTypeId="2" ParentId="9758" CreationDate="2017-08-14T18:21:10.090" Score="0" Body="&lt;p&gt;In your case you have &#xA;- a very small (or no) gap between train and test curves that indicates that the model has a high bias/underfit, solution: need to choose a more complex model;&#xA;- for the sake of completion, need to add an opposite case when the gap between train and test curves is very large that indicates a high variance/overfitting, solutions: a) continue increasing Data set Size; b) choose less complex model, c) do regularization.  &lt;/p&gt;&#xA;" OwnerUserId="37942" LastActivityDate="2017-08-14T18:21:10.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22254" PostTypeId="1" CreationDate="2017-08-14T19:00:59.327" Score="1" ViewCount="258" Body="&lt;p&gt;The authors of the original paper of Faster R-CNN when they refer to the positive anchors, they are labeled as 1. I guess they refer in &lt;strong&gt;binary&lt;/strong&gt; classification. What happens in the case in a task we have more than two classes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean the purpose of the RPN is to generate just region proposals (there is an object or not, regardless the label) or it generates region proposals and the respective label as well? I guess it is the second case, since the positive anchors that participate in the training process need to have a ground truth label as well, i.e. (cat, dog, e.t.c).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I might misunderstand things and interpret things in a wrong way.&lt;/p&gt;&#xA;" OwnerUserId="33150" LastActivityDate="2019-12-13T23:29:34.360" Title="Faster R-CNN: Labels regarding the positive anchors when there are many classes" Tags="|machine-learning|deep-learning|image-recognition|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22255" PostTypeId="1" AcceptedAnswerId="22257" CreationDate="2017-08-14T19:26:00.837" Score="4" ViewCount="219" Body="&lt;p&gt;I have code in Tensorflow using convolution neural network to recognize the characters in street view Text (SVT) data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the label type is string, what should I use instead of &lt;code&gt;tf.nn.sparse_softmax_cross_entropy_with_logits()&lt;/code&gt; in the loss function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot use &lt;code&gt;tf.nn.sparse_softmax_cross_entropy_with_logits()&lt;/code&gt; because the labels her must be an int dtype??&lt;/p&gt;&#xA;" OwnerUserId="37775" LastEditorUserId="29575" LastEditDate="2017-09-12T08:40:56.157" LastActivityDate="2020-08-13T13:57:02.743" Title="what is the loss function in char recognition using Tensorflow?" Tags="|deep-learning|tensorflow|convolution|ocr|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22257" PostTypeId="2" ParentId="22255" CreationDate="2017-08-14T20:30:37.480" Score="2" Body="&lt;p&gt;The loss function is correct, you just need to convert categorical variables into numerical representations using one-hot vector encoding. Please take a look at &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-categorical-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="36971" LastEditorUserId="98307" LastEditDate="2020-08-13T13:57:02.743" LastActivityDate="2020-08-13T13:57:02.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22258" PostTypeId="2" ParentId="22250" CreationDate="2017-08-14T21:45:18.297" Score="11" Body="&lt;p&gt;The main difference is that &lt;code&gt;HashingVectorizer&lt;/code&gt; applies a hashing function to term frequency counts in each document, where &lt;code&gt;TfidfVectorizer&lt;/code&gt; scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary &lt;a href=&quot;https://spark.apache.org/docs/latest/mllib-feature-extraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Hash functions are an efficient way of mapping terms to features; it doesn’t necessarily need to be applied only to term frequencies but that’s how &lt;code&gt;HashingVectorizer&lt;/code&gt; is employed here. Along with the 45339 documents, I suspect the feature vector is of length 1048576 because it’s the default 2^20 &lt;code&gt;n_features&lt;/code&gt;; you could reduce this and make it less expensive to process but with an increased risk of collision, where the function &lt;a href=&quot;http://preshing.com/20110504/hash-collision-probabilities/&quot; rel=&quot;nofollow noreferrer&quot;&gt;maps different terms to the same feature&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Depending on the use case for the word vectors, it may be possible to reduce the length of the hash feature vector (and thus complexity) significantly with acceptable loss to accuracy/effectiveness (due to increased collision). Scikit-learn has some hashing parameters that can assist, for example &lt;code&gt;alternate_sign&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the hashing matrix is wider than the dictionary, it will mean that many of the column entries in the hashing matrix will be empty, and not just because a given document doesn't contain a specific term but because they're empty across the whole matrix. If it is not, it might send multiple terms to the same feature hash - this is the 'collision' we've been talking about. &lt;code&gt;HashingVectorizer&lt;/code&gt; has a setting that works to mitigate this called &lt;code&gt;alternate_sign&lt;/code&gt; that's on by default, described &lt;a href=&quot;http://en.wikipedia.org/wiki/Feature_hashing#Properties&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;‘Term frequency - inverse document frequency’ takes term frequencies in each document and weights them by penalising words that appear more frequently across the whole corpus. The intuition is that terms found situationally are more likely to be representative of a specific document’s topic. This is different to a hashing function in that it is necessary to have a full dictionary of words in the corpus in order to calculate the inverse document frequency. I expect your tf.idf matrix dimensions are 45339 documents by 663307 words in the corpus; &lt;a href=&quot;https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Manning et al&lt;/a&gt; provide more detail and examples of calculation.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;‘Mining of Massive Datasets’ by Leskovec et al has a ton of detail on both feature hashing and tf.idf, the authors made the pdf available &lt;a href=&quot;http://www.mmds.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32697" LastEditorUserId="98307" LastEditDate="2020-08-06T16:59:44.807" LastActivityDate="2020-08-06T16:59:44.807" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="22259" PostTypeId="2" ParentId="22250" CreationDate="2017-08-14T21:53:45.843" Score="6" Body="&lt;p&gt;The &lt;code&gt;HashingVectorizer&lt;/code&gt; has a parameter &lt;code&gt;n_features&lt;/code&gt; which is &lt;code&gt;1048576&lt;/code&gt; by default. When hashing, they don't actually compute a dictionary mapping terms to a unique index to use for each one. Instead, you just hash each term and use a large enough size that you don't expect there to be too many collisions: &lt;code&gt;hash(term) mod table_size&lt;/code&gt;. You can make the returned matrix be any size you want by setting &lt;code&gt;n_features&lt;/code&gt;. You should adjust this to be in the right ballpark for your corpus if you don't think the default is reasonable (having it larger will cause less collisions though it takes more memory).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import HashingVectorizer&#xA;vectorizer = HashingVectorizer()&#xA;print(vectorizer.transform(['a very small document']).shape)&#xA;(1, 1048576)&#xA;&#xA;small_vectorizer = HashingVectorizer(n_features=5)&#xA;print(small_vectorizer.transform(['a very small document']).shape)    &#xA;(1, 5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37952" LastEditorUserId="37952" LastEditDate="2017-12-18T20:30:11.853" LastActivityDate="2017-12-18T20:30:11.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22260" PostTypeId="1" AcceptedAnswerId="22334" CreationDate="2017-08-14T22:02:23.303" Score="4" ViewCount="4497" Body="&lt;p&gt;I've seen a &lt;a href=&quot;http://biostats.bepress.com/cgi/viewcontent.cgi?article=1033&amp;amp;context=uncbiostat&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper describing a &quot;Reinforcement Learning Tree&quot;&lt;/a&gt;, which the authors say that it has a better convergence than random forests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I couldn't find anything related to utilizing random forests for a table in Reinforcement Learning. Have tests ever been conducted on using random forests in reinforcement learning?&lt;/p&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="836" LastEditDate="2017-08-16T21:56:21.910" LastActivityDate="2017-08-17T11:36:21.093" Title="Has the Random Forest algorithm ever been used in Reinforcement Learning applications?" Tags="|random-forest|reinforcement-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22261" PostTypeId="2" ParentId="16323" CreationDate="2017-08-15T01:31:38.293" Score="0" Body="&lt;p&gt;From a stats perspective, it sounds like you have a &lt;a href=&quot;https://en.wikipedia.org/wiki/Poisson_point_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Poisson process&lt;/a&gt;, where the events are user submissions.  So you might represent your dependent variable as the number of events in a unit of time (say, number of submissions per week), and set up a &lt;a href=&quot;https://en.wikipedia.org/wiki/Poisson_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Poisson regression&lt;/a&gt; or negative binomial regression.  For the independent variable, you might try the number of comments received in the previous unit of time, i.e., you might see how well the number of months received this week predicts the number of submission next week.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that there are likely to be trends over time (i.e., a new user doesn't make many submission at first, but gradually becomes a regular user) and &lt;a href=&quot;https://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow noreferrer&quot;&gt;autocorrelation&lt;/a&gt;, especially if your time scale is too small.  For example, in your example data, most days have 0 submissions, so autocorrelation will high if your units are days or smaller.  So consider using &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;time series methods&lt;/a&gt; or selecting a time scale large enough to get low autocorrelation.  &lt;/p&gt;&#xA;" OwnerUserId="26961" LastActivityDate="2017-08-15T01:31:38.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22264" PostTypeId="1" CreationDate="2017-08-15T07:14:44.480" Score="8" ViewCount="3232" Body="&lt;p&gt;In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning rate scheduler such as power scheduling and exponential scheduling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't understand at what kind of situations you should use one over the other. I feel that using adaptive learning rate optimization algorithm such as Adam is simpler and easier to implement than using learning rate scheduler. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how can you use it apart properly, depending on what kind of problems?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-08-16T18:35:19.593" Title="When should you use learning rate scheduling over adaptive learning rate optimization algorithm?" Tags="|deep-learning|learning-rate|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22266" PostTypeId="1" AcceptedAnswerId="22272" CreationDate="2017-08-15T10:17:00.043" Score="7" ViewCount="12785" Body="&lt;p&gt;I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category    Score    &#xA;AAAA        1&#xA;AAAA        3&#xA;AAAA        1&#xA;BBBB        1&#xA;BBBB        100&#xA;BBBB        159&#xA;CCCC        -10&#xA;CCCC        9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I would then like would be something like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Category    Count   Mean   Std   Min   25%   50%   75%   Max     &#xA;AAAA        &#xA;AAAA        &#xA;AAAA        &#xA;BBBB        &#xA;BBBB        &#xA;BBBB        &#xA;CCCC        &#xA;CCCC        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have been looking at using pandas with a combination of both &lt;code&gt;.groupby()&lt;/code&gt; and &lt;code&gt;.describe()&lt;/code&gt; like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('Category')['Score'].describe()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and this almost looks like what I want but when I come to view this as a Dataset, all of the stats are in the index. I would like the data to be in the form of a table so I can output it and create a visualization off of the back of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="37059" LastEditDate="2017-08-15T12:42:54.410" LastActivityDate="2017-08-15T12:42:54.410" Title="Summary statistics by category using Python" Tags="|python|pandas|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22267" PostTypeId="1" CreationDate="2017-08-15T10:20:16.817" Score="1" ViewCount="296" Body="&lt;p&gt;I was doing some digging to get some deeper understanding on the Capacity Planning done for setting up a Hadoop Cluster.&#xA;I have a daily ~100 GB of data generated and would like to find how a Capacity planning needs to be done for it&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Daily Input : 80 ~ 100 GB&#xA;Project Duration : 1 year&#xA;Block Size : 128 MB&#xA;Replication : 3&#xA;Compression : 30 %&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It would be really helpful if you can guide me through and give me a good insight on how and what factors should be taken into consideration for the planning.&lt;/p&gt;&#xA;" OwnerUserId="37397" LastEditorUserId="37397" LastEditDate="2017-08-15T15:47:51.010" LastActivityDate="2017-08-15T15:47:51.010" Title="Hadoop Cluster Capacity Planning" Tags="|apache-hadoop|" AnswerCount="0" CommentCount="0" ClosedDate="2017-08-15T10:33:49.803" ContentLicense="CC BY-SA 3.0" />
  <row Id="22269" PostTypeId="1" CreationDate="2017-08-15T10:14:09.797" Score="2" ViewCount="1399" Body="&lt;p&gt;I have trained a model for predicting the sale of items daily, such as daily car sales, with a machine learning model in Python. Now I get the new time real data (time series data). I want to retrain my mode. There are some issues:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How much data I should use for retraining model, namely what's the time &#xA;  range of data or when I should retrain my model. For example use the new one day or one week et al., data to retrain model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What's condition of new data if I want to use the new data for retraining model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to evaluate the stability of new training model, and How to know abnormal prediction for new model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="35597" OwnerDisplayName="tktktk0711" LastActivityDate="2017-08-15T11:29:27.617" Title="Machine learning mode retraining after getting new data" Tags="|python|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22270" PostTypeId="2" ParentId="22269" CreationDate="2017-08-15T10:59:22.427" Score="1" Body="&lt;p&gt;Once you trained your model, you have a measure of accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should retrain your model when the accuracy drops below a certain threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to use a Bayesian model, in which you update your model with each new observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest looking into &lt;a href=&quot;https://pymc-devs.github.io/pymc3/notebooks/stochastic_volatility.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pymc3's examples&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12942" OwnerDisplayName="Uri Goren" LastActivityDate="2017-08-15T10:59:22.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22271" PostTypeId="1" CreationDate="2017-08-15T12:18:25.633" Score="1" ViewCount="2019" Body="&lt;p&gt;I have been building a model to find explanation of Outliers in a high dimensional numerical data, generated from many sensors. The data contains more than 350 different fields and each field has numerical values (Float or Integer). It looks like: 350 columns and many rows. I want to find the outliers/ anomalies in the data and also the Explanation why those values are outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was reading about Generative models and found that &quot;They have the potential to understand and explain the underlying structure of the Input data even when there are no labels.&quot; I would like to know if it will be good to use the GANs for outlier detection and explanation on numerical data?&lt;/p&gt;&#xA;" OwnerUserId="37977" LastActivityDate="2017-08-15T13:29:52.310" Title="Can the Generative Adversarial Network useful for Outlier detection and Outlier explanation in a high dimentional numerical data?" Tags="|neural-network|data-mining|deep-learning|outlier|gan|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22272" PostTypeId="2" ParentId="22266" CreationDate="2017-08-15T12:33:51.540" Score="15" Body="&lt;p&gt;IIUC:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [80]: df.groupby(&quot;Category&quot;)['Score'].describe().reset_index()&#xA;Out[80]:&#xA;  Category  count       mean        std   min    25%    50%     75%    max&#xA;0     AAAA    3.0   1.666667   1.154701   1.0   1.00    1.0    2.00    3.0&#xA;1     BBBB    3.0  86.666667  79.839422   1.0  50.50  100.0  129.50  159.0&#xA;2     CCCC    2.0  -0.500000  13.435029 -10.0  -5.25   -0.5    4.25    9.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-08-15T12:33:51.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22273" PostTypeId="1" CreationDate="2017-08-15T12:37:23.997" Score="7" ViewCount="2274" Body="&lt;p&gt;As a part of my university project, I am researching/developing a sentiment analysis model wherein I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning trained classifier for prediction of Sentiments on Twitter data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Detailed description -&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain what I am trying to do is - Combine Machine Learning classifier and NLTK Vader sentiment analysis to get the better classification of tweets as Positive, Negative or Neutral.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have done is - &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Cleaned the data (&lt;a href=&quot;http://www.sananalytics.com/lab/twitter-sentiment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Niek Sanders twitter corpus&lt;/a&gt;) and pre-processed the tweets which include stop words removal, URL removal, User mention removal, remove the # symbol from Hashtags, lower case conversion, stemmer processing, etc.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Split into 80:20 training: test ratio&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Used TfidfVectorizer to create a sparse matrix of features with TFIDF of words. The number of columns is equal to the number of words in clean data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use this vector matrix to train and test the classifiers using scikit-learn.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Classifiers used&lt;/strong&gt; - This same vector matrix is being used to train - KNN, Random forest, Naive Bayes, SVM, Artificial Neural Network and Convolutional Neural Network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the main doubt arises when trying to combine NLTK Vader (SentimentIntensityAnalyzer results). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am doing is - From step three above, in the tfidf vector-matrix I am adding 2 columns and I am adding the Positive and Negative polarity result for the tweet given by NLTK, so now the vector-matrix has 2 new columns n+1 (total positive polarity of tweet by NLTK), n+2 (total negative polarity of tweet by NLTK) and it looks like -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    0      1     2     3     4     5     6     7  ......... |  n+1    |   n+2&#xA;------------------------------------------------------------|---------|----------&#xA;    0.4   0.3   0.4   0.1   0.5   0.3   0.2   0.4 ......... |  0.345  |  0.345&#xA;    0.5   0.3   0.2   0.8   0.3   0.6   0.4   0.5 ......... |  0.765  |  0.523&#xA;    0.6   0.4   0.1   0.7   0.8   0.8   0.2   0.2 ......... |  0.392  |  0.664&#xA;    0.2   0.9   0.7   0.4   0.9   0.9   0.8   0.5 ......... |  0.832  |  0.658&#xA;    0.9   0.5   0.9   0.7   0.3   0.2   0.2   0.5 ......... |  0.273  |  0.283&#xA;    0.5   0.2   0.2   0.7   0.2   0.1   0.6   0.6 ......... |  0.505  |  0.194&#xA;    0.4   0.3   0.2   0.3   0.3   0.9   0.5   0.5 ......... |  0.102  |  0.927&#xA;    0.1   0.8   0.1   0.2   0.1   0.5   0.2   0.7 ......... |  0.735  |  0.455&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt; - So is it correct to add it this way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though it has increased the overall accuracy of the classifiers by small margin of 1-2%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But am I doing it right?&lt;/strong&gt; &#xA;If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.&lt;/p&gt;&#xA;" OwnerUserId="37979" LastEditorUserId="83275" LastEditDate="2020-01-25T07:38:11.383" LastActivityDate="2020-06-23T11:01:15.100" Title="Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis" Tags="|machine-learning|neural-network|scikit-learn|sentiment-analysis|nltk|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22275" PostTypeId="2" ParentId="22271" CreationDate="2017-08-15T13:29:52.310" Score="2" Body="&lt;p&gt;See &lt;a href=&quot;https://www.quora.com/Do-generative-adversarial-networks-function-for-outlier-detection/answer/Ian-Goodfellow?srid=ikXo&quot; rel=&quot;nofollow noreferrer&quot;&gt;this answer&lt;/a&gt; by Ian Goodfellow (the creator of GANs) to the same question at Quora:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;There are definitely some papers about it, such as &lt;a href=&quot;https://arxiv.org/abs/1703.05921&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1703.05921] Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery&lt;/a&gt;. I don’t work on anomaly detection so I haven’t read these papers and don’t know a lot about how they work.&lt;/p&gt;&#xA;&lt;p&gt;It’s worth mentioning that there are also papers on anomaly detection using inputs that resemble adversarial examples: &lt;a href=&quot;https://arxiv.org/pdf/1706.02690.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1706.02690.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;For GANs, one thing to keep in mind is that the discriminator is not a generalized detector of weird things. It is trying to tell whether a sample came from the real data or &lt;em&gt;one specific non-data distribution: the generator&lt;/em&gt;. Because of that, it seems like the discriminator would only be useful for anomaly detection if you think you can make your generator resemble the anomalies you expect to need to detect.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Nevertheless, neural networks in general are &lt;em&gt;black boxes&lt;/em&gt; (i.e. not very interpretable), so they might detect outliers but you would not be able to tell &lt;em&gt;what&lt;/em&gt; characterized them as so.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-08-15T13:29:52.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22277" PostTypeId="1" CreationDate="2017-08-15T14:58:57.640" Score="1" ViewCount="263" Body="&lt;p&gt;I Have a variable which is Pareto-ly distributed 'x', with unknown alpha and m.&#xA;I want to find out the distribution of its mean, so I use the following model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with pymc3.Model() as model:&#xA;&#xA;        alpha = pymc3.Uniform('alpha', 0, 0.2)&#xA;        m = pymc3.Uniform('m', 0, 2)&#xA;        x = pymc3.Pareto('x', alpha = alpha, m = m, observed = obs)&#xA;        mean_x = pymc3.Deterministic('mean_x', tt.mean(x))&#xA;&#xA;        step = pymc3.Metropolis()&#xA;        trace = pymc3.sample(20000, step=step)&#xA;        burned_trace=trace[1000:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I only get a single value for 'mean_x', 'alpha' and 'm' every time I sample, and both 'alpha' and 'm' converge to the mean of their respective uniform distribution('alpha' converges to 0.1 and 'm' to 1. What is the correct way to sample the mean of a Pareto-ly distributed random variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="37984" LastEditorUserId="37984" LastEditDate="2017-08-15T15:35:08.860" LastActivityDate="2017-08-15T15:35:08.860" Title="How to use pymc3 to sample the mean of a Pareto random variable?" Tags="|python|bayesian|monte-carlo|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22280" PostTypeId="1" CreationDate="2017-08-15T16:32:01.893" Score="1" ViewCount="318" Body="&lt;p&gt;I have lots of events with timestamps like &quot;15/07/2017  12:38:20&quot; with accompanying information for each event. I want to find the length of time it takes for most events to occur on a given day, and over multiple days using averages and standard deviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could calculate the time between first and last timestamp of the day, but there are outliers which skew the data, so by using averages and standard deviations I can implement the &lt;a href=&quot;https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;68-95-99%&lt;/a&gt; statistical rule to calculate the time it takes for 68% of events to occur (or 95% or 99%), and add boxplots etc. The data is distributed normally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have extracted the time part of each timestamp (e.g. 12:38:20) and converted that into seconds (a day has 86,400) by multiplying the hour by 3,600 seconds, minutes by 60 seconds and then adding all the seconds together. 12:38:20 therefore is 45,500 seconds into the day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this works fine for a single day. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Day 1: say 100 events occur.&#xA;Mean = 33,317 seconds into the day (so around 9.15am, i.e. the mean event occurred at this time)&#xA;Stdev.p = 3,378 seconds (1 stdev away from the mean, so around 56 minutes 18 seconds)&#xA;Stdev.p*2 = 6,757 seconds (between -1 and +1 stdev's away from the mean, this is the time it takes for 68% of events to occur, so around 1 hr 52 minutes)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All good so far. I can calculate this for every single day across my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to group multiple days together, so I can show the time it takes on average for a group of days (the whole week, month, quarter or whatever). &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do I take the average of the standard deviations - i.e. stdev for day 1 + stdev for day 2, divided by the number of days?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or do I calculate the standard deviation for all events over days 1 and 2? Doing it this second way gives bigger durations across the two days. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As I'm typing this I'm feeling like its the first option. The second way mixes times from the first day with the second day, which is incorrect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I've answered my own question, but would welcome any reassurance or comments to the contrary. &lt;/p&gt;&#xA;" OwnerUserId="33298" LastActivityDate="2017-08-15T16:32:01.893" Title="Average and standard deviations around timestamps" Tags="|time-series|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22281" PostTypeId="1" CreationDate="2017-08-15T17:19:05.823" Score="0" ViewCount="42" Body="&lt;p&gt;By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small part of a file, my guess is that I have to somehow access the file, load it in memory and filter out that small part that I am interested in (Otherwise how else would I be able to query the subset of the records that I want?). In otherwards that should then mean I have anyway had to read the whole file into memory. So if it was a 20GB file, it means I have had to read 20GB so as to filter off the 1GB that I want. Where do the performance benefits come from then?&lt;/p&gt;&#xA;" OwnerUserId="37991" LastActivityDate="2017-08-16T18:14:14.663" Title="R: I need to understand the mechanism behind reading a subset of a file in R (e.g., via sqldf) or other data-science-centered programming language" Tags="|machine-learning|python|r|data-mining|bigdata|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22283" PostTypeId="1" CreationDate="2017-08-15T18:41:49.833" Score="1" ViewCount="116" Body="&lt;ol&gt;&#xA;&lt;li&gt;What are bottleneck values and how are they generated?&lt;/li&gt;&#xA;&lt;li&gt;How does the next-to-last layer of Inception use these bottlenecks to generate the accuracy?&lt;/li&gt;&#xA;&lt;li&gt;How is the final accuracy calculated?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37996" LastActivityDate="2017-08-15T18:41:49.833" Title="TensorFlow Inception V3" Tags="|tensorflow|inception|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22284" PostTypeId="2" ParentId="15085" CreationDate="2017-08-15T18:52:36.000" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1701.03682&quot; rel=&quot;nofollow noreferrer&quot;&gt;LIDE: Language Identification from Text Documents&lt;/a&gt; benchmarked the difference between using character and word n-gram information for natural language identification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S1319157815000609&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word-length algorithm for language identification of under-resourced languages&lt;/a&gt; proposed new algorithms for languages with limited available data.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-15T18:52:36.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22285" PostTypeId="2" ParentId="22181" CreationDate="2017-08-15T18:53:05.437" Score="1" Body="&lt;p&gt;There is usually only one learning rate active when using a neural network as function approximator in reinforcement learning. The different names $\eta$ and $\alpha$ are just different conventions for the same basic concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use a function approximator, other than a linear one, in reinforcement learning, then typically you would not use TD error based update like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{w} \leftarrow \mathbf{w} + \alpha[R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})]\nabla \hat{q}(S, A,\mathbf{w})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you would train your estimator in a supervised learning manner on sampled TD target (which would use $\eta$ param in a neural network):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mathbf{x} = \phi(S, A), y = R+\gamma\hat{q}(S', A',\mathbf{w})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can actually do either if your library supports it - it is certainly possible to calculate $\nabla \hat{q}(S, A,\mathbf{w})$ for a neural network for instance, instead of using an explicit training loss function. However, the two approaches are equivalent ways of expressing the same thing, there is no reason to use both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other parameters used in reinforcement learning that may affect rates of convergence and other properties of learning agents. For example with differential semi-gradient TD learning - which might be an algorithm you would look at for a continuous task - Sutton and Barto present $\beta$ as a separate learning rate for the average reward, distinct from the learning rate of the estimator.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $eta$, or do they have ultimately different effects?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They are essentially the &lt;em&gt;same&lt;/em&gt; parameter with a different name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to pass an error value like $R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})$ (whether multiplied by $\alpha$ or not) into the neural network as a target, then you have got the wrong idea. Instead you want your neural network to learn the target $R+\gamma\hat{q}(S', A',\mathbf{w})$. That's because the subtraction of current prediction and multiplication by a learning rate is built into the neural network training.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-15T19:03:09.790" LastActivityDate="2017-08-15T19:03:09.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22286" PostTypeId="1" AcceptedAnswerId="22288" CreationDate="2017-08-15T19:00:26.920" Score="1" ViewCount="55" Body="&lt;p&gt;Is there a go to source for searching for data sources? Like a way to ask google, specifically to show data files?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I'm looking for data on the number of smart phones and tablets sold in countries around the world. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas where I can look?&lt;/p&gt;&#xA;" OwnerUserId="16238" LastActivityDate="2017-08-15T19:34:55.603" Title="Resources for datasets" Tags="|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22287" PostTypeId="2" ParentId="22281" CreationDate="2017-08-15T19:16:02.903" Score="1" Body="&lt;p&gt;There are multiple aspects to performance when it comes to reading a portion of the file. I am going to discuss the two most common one based on functions that are available in the base R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is R doing at the time of reading?&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Establish a connection with the file.&lt;/li&gt;&#xA;&lt;li&gt;Open the file&lt;/li&gt;&#xA;&lt;li&gt;Read the data sequentially&lt;/li&gt;&#xA;&lt;li&gt;Load the data in memory&lt;/li&gt;&#xA;&lt;li&gt;Close the connection&lt;/li&gt;&#xA;&lt;li&gt;dump the object into a variable if assigned, else render the output in some gui.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;While it is doing all these steps, it is also trying to:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Understand how big the data is so that it can allocate appropriate memory&lt;/li&gt;&#xA;&lt;li&gt;What are the data types or &lt;code&gt;classes&lt;/code&gt; so that it can (again) allocate appropriate memory.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sub bullet &lt;code&gt;2&lt;/code&gt; is important here because unless &lt;code&gt;R&lt;/code&gt; reads the entire dataset, it cannot determine the datatype. Providing this information alone at the time of the read makes processing much faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By reading a subset of the data, R makes it easier to interpret the datatype and then assigns it to the subsequent sets without worrying about what the actual datatype is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does R really loads all the data every time?&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Functions that allow filter at the time of read, really target the third step of reading data sequentially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row is compared against the filter criteria and the result stored as a binary logical index. This index is then used to determine, what remains when the connection closes. I could not find specific reference in the documentation if the discard happens immediately or post entire read, but I will assume that the &lt;code&gt;R&lt;/code&gt; collects the garbage automatically as it reads with a filter criteria and makes the memory available to a subsequent line of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May be there are other experts who can throw more light on the real inner workings here and either validate or correct my assumptions, but hope my context helps.&lt;/p&gt;&#xA;" OwnerUserId="10345" LastActivityDate="2017-08-15T19:16:02.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22288" PostTypeId="2" ParentId="22286" CreationDate="2017-08-15T19:34:55.603" Score="3" Body="&lt;p&gt;Browse the following sources to see if you could find them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://archive.ics.uci.edu/ml/index.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI ML Repository&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://openml.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenML webpage&lt;/a&gt; (login required to download)&lt;/li&gt;&#xA;&lt;li&gt;List of datasets for ML research (&lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wiki&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="28716" LastActivityDate="2017-08-15T19:34:55.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22289" PostTypeId="2" ParentId="22199" CreationDate="2017-08-15T19:46:58.837" Score="1" Body="&lt;p&gt;One potential approach can be iterative design of a neural network architecture such as Multi-Layer Perceptron (MLP) as described in the following post:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/238637/deep-neural-network-tuning-hyperparameters&quot;&gt;https://stats.stackexchange.com/questions/238637/deep-neural-network-tuning-hyperparameters&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume &lt;a href=&quot;http://cs231n.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;recommended&lt;/a&gt; ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid overfitting on a small dataset, it is important to add l1 or l2 regularization (weight decay) and a dropout layer (e.g. with keep probability of 0.5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can then use cross validation with &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;random search&lt;/a&gt; or &lt;a href=&quot;https://github.com/fmfn/BayesianOptimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian optimization&lt;/a&gt; to choose the best architecture as described in the cross validated article above. &lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-15T19:46:58.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22290" PostTypeId="1" CreationDate="2017-08-15T19:58:41.783" Score="1" ViewCount="253" Body="&lt;p&gt;I need to compare two groups of people, where the independent variable is having a college degree, and the dependent variable is the income.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that if I divide the whole population of the study into two groups,&#xA;one of them has signally more persons, so the mean is affected by the size of each group and the value of the outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I demonstrate or reject the theory that having a college degree &#xA;among the population of the study, ensures a better income?&lt;/p&gt;&#xA;" OwnerUserId="37999" LastActivityDate="2017-08-17T21:08:05.157" Title="how to compare two groups" Tags="|dataset|statistics|distance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22292" PostTypeId="1" CreationDate="2017-08-15T20:39:22.910" Score="1" ViewCount="566" Body="&lt;p&gt;In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the disadvantage of using these packages to read a small file that one would typically read with read.csv?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If their is no disadvantage of using them to read small files, does this mean functions such as read.csv might not have much use in future given the emerging big-data-targeted readers?&lt;/p&gt;&#xA;" OwnerUserId="37991" LastEditorUserId="37991" LastEditDate="2017-08-15T20:44:57.880" LastActivityDate="2020-08-11T14:52:58.920" Title="R: reading big data files in R" Tags="|python|r|bigdata|csv|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22293" PostTypeId="2" ParentId="21713" CreationDate="2017-08-15T20:55:42.343" Score="1" Body="&lt;p&gt;I used a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MinMax scaler&lt;/a&gt; in the range between (0, 1) applied to the closing price of S&amp;amp;P500. The RNN consisted of a single LSTM layer with a lookback window of 10 days to predict the next day's closing price. The following figure shows RNN prediction of the next day's closing price (in red). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZEmG5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZEmG5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For code used to generate the figure, have a look at the following &lt;a href=&quot;https://github.com/vsmolyakov/experiments_with_python/blob/master/chp04/keras_lstm_series.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;ipython notebook&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-15T20:55:42.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22294" PostTypeId="1" CreationDate="2017-08-15T22:01:52.660" Score="3" ViewCount="1827" Body="&lt;p&gt;I am trying to replicate a network for facial key point detection like in the following link &lt;a href=&quot;http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Daniel Nouri's Blog on KFKD&lt;/a&gt;. The blog uses Lasagne but i am trying to do using Tensorflow. &lt;strong&gt;I am unable to follow the part of performing data augmentation in Tensorflow.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the methods I saw either did augmentation before training or were applied only on X values not on Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find the replacement of FlipBatchIterator class used in the blog&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nolearn.lasagne import BatchIterator&#xA;&#xA;class FlipBatchIterator(BatchIterator):&#xA;flip_indices = [&#xA;    (0, 2), (1, 3),&#xA;    (4, 8), (5, 9), (6, 10), (7, 11),&#xA;    (12, 16), (13, 17), (14, 18), (15, 19),&#xA;    (22, 24), (23, 25),&#xA;    ]&#xA;&#xA;def transform(self, Xb, yb):&#xA;    Xb, yb = super(FlipBatchIterator, self).transform(Xb, yb)&#xA;&#xA;    # Flip half of the images in this batch at random:&#xA;    bs = Xb.shape[0]&#xA;    indices = np.random.choice(bs, bs / 2, replace=False)&#xA;    Xb[indices] = Xb[indices, :, :, ::-1]&#xA;&#xA;    if yb is not None:&#xA;        # Horizontal flip of all x coordinates:&#xA;        yb[indices, ::2] = yb[indices, ::2] * -1&#xA;&#xA;        # Swap places, e.g. left_eye_center_x -&amp;gt; right_eye_center_x&#xA;        for a, b in self.flip_indices:&#xA;            yb[indices, a], yb[indices, b] = (&#xA;                yb[indices, b], yb[indices, a])&#xA;&#xA;    return Xb, yb&#xA;&#xA;&#xA;&#xA;net3 = NeuralNet(&#xA;# ...&#xA;regression=True,&#xA;batch_iterator_train=FlipBatchIterator(batch_size=128),&#xA;max_epochs=3000,&#xA;verbose=1,&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38003" LastActivityDate="2017-08-15T22:01:52.660" Title="Data Augmentation in Tensorflow" Tags="|deep-learning|tensorflow|data-augmentation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22295" PostTypeId="2" ParentId="20452" CreationDate="2017-08-16T03:14:17.250" Score="0" Body="&lt;p&gt;You can convert the categorical value of OUI using a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LabelEncoder&lt;/a&gt;. It looks like you have two features (OUI and IAT) and a fixed number of classes / outputs. You could use an MLP classifier to solve this problem. In Keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(Dense(32, input_shape=(num_features,)))&#xA;model.add(Activation('relu'))&#xA;model.add(Dense(num_classes))&#xA;model.add(Activation('softmax'))&#xA;&#xA;model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is just an example, it's better to create a deeper architecture and add regularization (e.g. l1/l2 weight penalty or drop-out) to prevent over-fitting.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-16T03:14:17.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22296" PostTypeId="2" ParentId="20407" CreationDate="2017-08-16T03:35:33.507" Score="1" Body="&lt;p&gt;I recommend getting to know &lt;a href=&quot;http://pymc-devs.github.io/pymc3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyMC library&lt;/a&gt; for Bayesian Inference applied to graphical models but also deep learning. See &lt;a href=&quot;http://twiecki.github.io/blog/2016/06/01/bayesian-deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; for an implementation of Bayesian MLP in PyMC3. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you would expect, the MLP weights are assigned a prior Gaussian distribution which gets updated to a posterior after observing the training data. Instead of Stochastic Gradient Descent or Adam optimizer, &lt;a href=&quot;https://arxiv.org/pdf/1603.00788.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ADVI&lt;/a&gt; variational inference algorithm is used to compute the posterior distributions of all latent variables.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-16T03:35:33.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22297" PostTypeId="2" ParentId="17513" CreationDate="2017-08-16T06:34:57.657" Score="4" Body="&lt;p&gt;CNN feature extraction in TensorFlow is now made easier using the tensorflow/models repository on Github. There are pre-trained VGG, ResNet, Inception and MobileNet models available &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/slim#pre-trained-models&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used the following wrapper for convenient feature extraction in TensorFlow. You can just provide the tool with a list of images. It will pre-process the images and extract the features by feed-forwarding through the network. You can specify the layer names that you want to extract and save them to HDF5 or pickle afterwards. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tomrunia/TF_FeatureExtraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tomrunia/TF_FeatureExtraction&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38014" LastActivityDate="2017-08-16T06:34:57.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22299" PostTypeId="1" AcceptedAnswerId="22489" CreationDate="2017-08-16T08:07:44.947" Score="13" ViewCount="5829" Body="&lt;p&gt;Answers on here have stated that &lt;a href=&quot;https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful&quot;&gt;the dimensions in t-SNE are meaningless&lt;/a&gt;, and that the &lt;a href=&quot;https://datascience.stackexchange.com/questions/10802/can-closer-points-be-considered-more-similar-in-t-sne-visualization&quot;&gt;distances between points are not a measure of similarity&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, can we say anything about a point based on it's nearest neighbours in t-SNE space? &lt;a href=&quot;https://datascience.stackexchange.com/questions/19025/t-sne-why-equal-data-values-are-visually-not-close&quot;&gt;This answer&lt;/a&gt; to why points that are exactly the same are not clustered suggests the ratio of distances between points is similar between lower and higher dimensional representations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the image below shows t-SNE on one of my datasets (15 classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I say that &lt;code&gt;cro 479&lt;/code&gt; (top right) is an outlier? Is &lt;code&gt;fra 1353&lt;/code&gt; (bottom left) is more similar to &lt;code&gt;cir 375&lt;/code&gt; than the other images in the &lt;code&gt;fra&lt;/code&gt; class, etc? Or could these just be artefacts, e.g. &lt;code&gt;fra 1353&lt;/code&gt; got stuck on the other side of a few clusters and couldn't force its way through to the other &lt;code&gt;fra&lt;/code&gt; class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NqNxt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NqNxt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-08-24T16:41:22.080" Title="Does nearest neighbour make any sense with t-SNE?" Tags="|tsne|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22300" PostTypeId="1" CreationDate="2017-08-16T08:29:44.117" Score="2" ViewCount="973" Body="&lt;p&gt;The network in CNN recognize one digit for one image,&#xA; how can I recognize &lt;strong&gt;two or more digits in one image&lt;/strong&gt;.&#xA;A proper way I've thought about is modifying the weights in the FC layer. &#xA;In other words, if there is two three digits in a image, the fully connected layer's weight can be expressed as (in tensorflow):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;w_output = __weight_variable([1024, number_of_digits*number_classes], stddev=0.1)&lt;/code&gt;&#xA;If we want to recognize one digit in one image, just remove the &#xA;&lt;code&gt;number_of_digits&lt;/code&gt; in the above statement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another question occurred when the number of digits is unknown(Maybe two or more). How to modify the network to recognize all the digits in the image? &lt;/p&gt;&#xA;" OwnerUserId="38019" LastEditorUserId="29169" LastEditDate="2021-02-16T00:43:24.003" LastActivityDate="2021-02-16T00:43:24.003" Title="How to detect multiple digits in one image using CNN" Tags="|neural-network|tensorflow|convolutional-neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22301" PostTypeId="1" CreationDate="2017-08-16T09:18:06.850" Score="-1" ViewCount="1988" Body="&lt;p&gt;While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen different open-source implementations using different schemes for tokenization. They handle URL-mentions, Capitalization, User-mention etc. differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I usually follow the script accompanying the GloVE code:  &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nlp.stanford.edu/projects/glove/&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any std. rules / best practices one should follow while tokenizing tweets? So much variation in different code-bases confuses me sometimes.&lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2017-09-20T17:05:30.600" Title="Best way to tokenize tweet" Tags="|nlp|text-mining|preprocessing|twitter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22302" PostTypeId="1" AcceptedAnswerId="22359" CreationDate="2017-08-16T10:41:32.463" Score="8" ViewCount="9191" Body="&lt;p&gt;Usually the neural network training has at least 2 steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;first trained on a large set of some standard data (ImageNet, ...)&lt;/li&gt;&#xA;&lt;li&gt;and then the resulting weights are trained on a small set of my data (in this step we can train all layers or only one last layer)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What is the same of 2-nd step, is it Fine-tuning or Transfer-learning? And what is the different between Fine-tuning and Transfer-learning?&lt;/p&gt;&#xA;" OwnerUserId="37736" LastActivityDate="2019-01-14T16:43:06.373" Title="What is the different between Fine-tuning and Transfer-learning?" Tags="|machine-learning|neural-network|deep-learning|computer-vision|caffe|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22303" PostTypeId="1" AcceptedAnswerId="23580" CreationDate="2017-08-16T11:43:56.423" Score="5" ViewCount="391" Body="&lt;p&gt;What is the fully-convolutional model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is fully-convolutional model a model that has only convolutional layers (with Batch-norm and Activation) and has not any: max-pool, fully-connected, and other layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is true, then why Yolo v2 neural network for object detection is named Fully-Convolutional model if it uses Max-pool layers: &lt;a href=&quot;https://arxiv.org/pdf/1612.08242v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1612.08242v1.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Table 2: The path from YOLO to YOLOv2. Most of the listed design&#xA;  decisions lead to significant increases in mAP. Two  exceptions are&#xA;  &lt;strong&gt;switching to a fully convolutional network&lt;/strong&gt; with anchor boxes and using the new network.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wxHil.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wxHil.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Also why Fully Convolutional Networks for Semantic Segmentation is named Fully Convolutional if it has Max-pool layers: &lt;a href=&quot;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V5bWN.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V5bWN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37736" LastActivityDate="2017-10-06T07:11:55.347" Title="What is the fully-convolutional model?" Tags="|machine-learning|neural-network|deep-learning|computer-vision|caffe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22306" PostTypeId="1" AcceptedAnswerId="23181" CreationDate="2017-08-16T14:13:30.537" Score="1" ViewCount="135" Body="&lt;p&gt;I want to extract the total bill from image receipts. I could extract the entire data present in the image but now I am struck with the problem of extracting only the information that I need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the image that I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/38XnB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/38XnB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pasting the extracted information from the image&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m cm lnnk 3mm: :33; no 1 z m&#xA;&#xA;x Visut all! ms“; (or nulnunn mfn an an: nan.&#xA;&#xA;&#xA;&#xA;Sub Iota] 19.56&#xA;TOTAL 19.56&#xA;VISA 1956&#xA;Fun 19.56&#xA;D!!! You Know 0&#xA;&#xA;&#xA;&#xA;For ureat-tastlru dessens under 200&#xA;cahries, try our Triple Berry Frozen&#xA;Yogurt Sunda: a dish of Frozen Yogurt.&#xA;or a Vanma rozen Vugurt Done.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this data I just want to extract the total bill. To get this I found out that I could use Ad Hoc Normalization (Adhoc retrieval). Can someone provide any insights on Adhoc retrieval. If there are any other option to extract the data from the image please let me do so. I am using tesseract to extract this information. Sometimes it is no giving the proper output. I could use some help in improvising the output given by the tesseract.&lt;/p&gt;&#xA;" OwnerUserId="38031" LastActivityDate="2017-09-20T16:59:06.353" Title="ADHoc Information Retrieval" Tags="|data-mining|nlp|information-retrieval|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22307" PostTypeId="1" CreationDate="2017-08-16T14:13:40.703" Score="0" ViewCount="1642" Body="&lt;p&gt;I am trying to perform string delivery classification (X: delivery_string, y: delivery_string_relevance (values 0 or 1)). I am using DTM (document term matrix) for the feature extraction, and experimenting with text-mining algorithms. However, I still believe that I can preprocess my raw strings better (before creating the DTM) by extracting delivery dates and obtaining one common format representation for all of them. Any suggestions on good date parsers? &lt;strong&gt;&lt;em&gt;datefinder.find_dates()&lt;/em&gt;&lt;/strong&gt; in Python is doing quite a decent job, however still fails for some cases, such like when string contains &quot;delivery time 10 - 12 days&quot; it extracts it as 10-12-2017.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sample data (language used Dutch):&lt;/strong&gt;&#xA;beschikbaar vanaf 04 okt // beschikbaar vanaf: 05-08-2017 // 8 - 9 weken // 8 - 9 werkdagen // 8 dag(en)&lt;/p&gt;&#xA;" OwnerUserId="17810" LastActivityDate="2017-08-16T15:24:25.090" Title="Extract and normalize date from string for the proper text mining" Tags="|machine-learning|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22309" PostTypeId="2" ParentId="22307" CreationDate="2017-08-16T15:24:25.090" Score="1" Body="&lt;p&gt;The best option is &lt;a href=&quot;https://github.com/scrapinghub/dateparser&quot; rel=&quot;nofollow noreferrer&quot;&gt;dateparser&lt;/a&gt;, whose goal is to parse natural language dates with support for Dutch. There is another package &lt;a href=&quot;https://github.com/kvh/recurrent&quot; rel=&quot;nofollow noreferrer&quot;&gt;recurrent&lt;/a&gt; which does natural language parsing of dates focusing on recurring events.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-16T15:24:25.090" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22310" PostTypeId="2" ParentId="22179" CreationDate="2017-08-16T15:55:30.110" Score="1" Body="&lt;p&gt;It's analogous to an &lt;em&gt;extremely specific&lt;/em&gt; convolution step on a 2D image. That is, it's analogous to an $N \times I$ image with &lt;em&gt;one&lt;/em&gt; feature map (e.g. a black &amp;amp; white image, if we're talking about the input), and you choose to use $J$ filters of size $F \times I$ which spans the entire width of the image and only strides along the length to create $J$ feature maps of size $(N - F + 1) \times 1.$ The restrictions here are that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-The &quot;image&quot; (or current layer you're working on) necessarily only has one feature map&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-The filters span the entire width of the image, so they don't stride along that direction, and the resulting feature maps have a width of $1.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could then re-interpret the $(N-F+1) \times 1$ resulting feature maps as a single $(N-F+1) \times J$ image, which then creates the exact same restrictions for the next convolutional layer (or more restrictions if you're doing pooling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, the analogy is there, but it's only analogous to a very restricted class of convolution on 2D image arrays, which I don't think is very useful. The kind of application it would be used for is black and white images for which you don't care about translation invariance along one of the dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason CNNs on images are much more flexible than this analogy allows for is because the input objects (and each respective hidden layer) of an image CNN is a 3D array, not a 2D array as it is in this case. The third dimension is the feature maps (the RGB values for the input image, for example). Then you could allow the filter to be of any size in both dimensions, and stride along both directions. I would prefer to keep the text interpretation when it comes to 2D inputs of a CNN.&lt;/p&gt;&#xA;" OwnerUserId="2841" LastActivityDate="2017-08-16T15:55:30.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22311" PostTypeId="2" ParentId="9850" CreationDate="2017-08-16T16:01:59.403" Score="4" Body="&lt;p&gt;BLUF: iterative trial-and-error with subset of data and matplotlib.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long Answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My team was struggling with this same question not that long ago. All the answers here are great, but I wanted to share with you my &quot;beginner's answer&quot; for context and as a starting point for folks who are new to machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to aim for a cost function that is smooth and convex for your specific choice of algorithm and data set. That's because you want your algorithm to be able to confidently and efficiently adjust the weights to eventually reach the global minimum of that cost function. If your cost function is &quot;bumpy&quot; with local max's and min's, and/or has no global minimum, then your algorithm might have a hard time converging; its weights might just jump all over the place, ultimately failing to give you accurate and/or consistent predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you are using linear regression to predict someone's weight (real number, in pounds) based on their height (real number, in inches) and age (real number, in years), then the mean squared error cost function should be a nice, smooth, convex curve. Your algorithm will have no problems converging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But say instead you are using a logistic regression algorithm for a binary classification problem, like predicting a person's gender based on whether the person has purchased diapers in the last 30 days and whether the person has purchased beer in the last 30 days. In this case, mean squared error might not give you a smooth convex surface, which could be bad for training. And you would tell that by experimentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could start by running a trial with using MSE and a small and simple sample of your data or with mock data that you generated for this experiment. Visualize what is going on with matplotlib (or whatever plotting solution you prefer). Is the resulting error curve smooth and convex? Try again with an additional input variable... is the resulting surface still smooth and convex? Through this experiment you may find that while MSE does not fit your problem/solution, cross entropy gives you a smooth convex shape that better fits your needs. So you could try that out with a larger sample data set and see if the hypothesis still holds. And if it does, then you can try it with your full training set a few times and see how it performs and if it consistently delivers similar models. If it does not, then pick another cost function and repeat the process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This type of highly iterative trial-and-error process has been working pretty well for me and my team of beginner data scientists, and lets us focus on finding solutions to our questions without having to dive deeply into the math theory behind cost function selection and model optimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, a lot of this trial and error has already been done by other people, so we also leverage public knowledge to help us filter our choices of what might be good cost functions early in the process. For example, cross entropy is generally a good choice for classification problems, whether it's binary classification with logistic regression like the example above or a more complicated multi-label classification with a softmax layer as the output. Whereas MSE is a good first choice for linear regression problems where you are seeking a scalar prediction instead of the likelihood of membership in a known category out of a known set of possible categories, in which case instead of a softmax layer as your output you'd could just have a weighted sum of the inputs plus bias without an activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this answer helps other beginners out there without being overly simplistic and obvious.&lt;/p&gt;&#xA;" OwnerUserId="38032" LastEditorUserId="38032" LastEditDate="2017-08-16T18:54:03.297" LastActivityDate="2017-08-16T18:54:03.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22312" PostTypeId="1" CreationDate="2017-08-16T16:34:42.037" Score="1" ViewCount="81" Body="&lt;p&gt;I have three classifiers for language identification:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A: en, de, ru, fr, ij, kl&#xA;B: en, de, ru, fr, xy&#xA;C: en, de, ru, fr, no, pq, rs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I have a balanced dataset which matches the classes of A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a fair way to compare those classifiers?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My thoughts&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Accuracy&lt;/strong&gt; on classes of A: Not fair, because B can't possible recognize ij, kl and will likely make mistakes due to the fact that it knows language xy. (Similar for C)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Smallest common subset&lt;/strong&gt;: Possible, but not so interesting as those are the &quot;easy&quot; classes.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Precision&lt;/strong&gt;: Given a language the classifier knows, how often does it actually recognize it?&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unfair for C, as it has more possibilities to make mistakes. Probably this could be somehow compensated? E.g. if it recognizes a class which is not in A, just the next best is taken (until the correct class is predicted or a wrong class from A is preditected)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There could also be a possibility to use the classifier A in combination with another classifier. The weight / importance (however that is measured) of the contribution of the other classifier is its score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any publications which do something similar?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-08-16T18:17:26.240" LastActivityDate="2017-08-16T18:17:26.240" Title="What is an appropriate way to compare classifiers with different sets of classes?" Tags="|classification|scoring|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22313" PostTypeId="1" CreationDate="2017-08-16T17:05:53.190" Score="4" ViewCount="621" Body="&lt;p&gt;I've been working through the Sutton + Barto RL text, implementing a number of the algos + running them in the OpenAI gym. One phenomenon that I seem to come across quite regularly is that agents who, at certain points during their training appear to be making good progress towards learning a plausible state-value/state-action functions, &quot;catastrophically forget&quot; the insights they glean and subsequently never recover. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make this more concrete, here's the (smoothed) reward history of my implementation of a semi-gradient expected SARSA agent with linear function approximation and binary features running on the &lt;a href=&quot;https://gym.openai.com/envs/MountainCar-v0&quot; rel=&quot;nofollow noreferrer&quot;&gt;MountainCar&lt;/a&gt; environment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/G4nlO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/G4nlO.png&quot; alt=&quot;Reward history&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem Details&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The problem has a bounded, continuous state space and a discrete, 3-action action space.&lt;/li&gt;&#xA;&lt;li&gt;The learner receives a reward of $-1$ at each timestep. The episode ends when the car makes its way all the way up the hill OR it reaches 200 timesteps without success.&lt;/li&gt;&#xA;&lt;li&gt;Prior to learning, I tile-coded the state space using an 8x8 grid with 8 overlapping tilings, generated using Sutton's own &lt;code&gt;tiles.py&lt;/code&gt; script.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Learner Details&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The learner is an implementation of the semi-gradient SARSA algorithm with linear function approximation for the Q value.&lt;/li&gt;&#xA;&lt;li&gt;The agent begins with all Q weights initialized to 0.&lt;/li&gt;&#xA;&lt;li&gt;The learning rate for the agent was set at $\frac{1}{10} \times (\text{# tiles})^{-1}$, where $\text{# tiles}$ in this case was $8 \times 8 \times 8 = 512$&lt;/li&gt;&#xA;&lt;li&gt;During learning, the agent selected its actions using an $\epsilon$-soft policy, where $\epsilon$ is set to 0.10&lt;/li&gt;&#xA;&lt;li&gt;The agent's temporal discount factor is set to $\gamma = 0.95$ (though I realize it could have been 1 given the episodic nature of the task)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possibility I want to rule out is that my implementation of the learning agent is incorrect. To that end I am curious to know whether others experience this kind of &quot;forgetting&quot; behavior (even on simple problems like this), and if so, how it might be reduced. &lt;/p&gt;&#xA;" OwnerUserId="38036" LastActivityDate="2018-04-24T22:59:49.830" Title="Catastrophic forgetting in linear semi-gradient RL agent?" Tags="|machine-learning|gradient-descent|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22314" PostTypeId="1" CreationDate="2017-08-16T17:14:43.253" Score="2" ViewCount="311" Body="&lt;p&gt;I am quite confident with using number data with a neural network, but I want to use string data. My question is where do I begin? Obviously you cant times weights and strings together (x * w) because they are of different data types, so what should I do to the string data to turn it into numbers? If you're interested, here is my dataset. It's by no means the final version but here it is just to give you an example. It's based on spelling:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;enormous..........| Difficulty: 2&lt;/li&gt;&#xA;&lt;li&gt;rhythm..............| Difficulty: 3&lt;/li&gt;&#xA;&lt;li&gt;hamster.............| Difficulty: 2&lt;/li&gt;&#xA;&lt;li&gt;walk...................| Difficulty: 1&lt;/li&gt;&#xA;&lt;li&gt;accommodate...| Difficulty: 3&lt;/li&gt;&#xA;&lt;li&gt;blue....................| Difficulty: 1&lt;/li&gt;&#xA;&lt;li&gt;projector.............| Difficulty: 1&lt;/li&gt;&#xA;&lt;li&gt;regression..........| Difficulty: 2&lt;/li&gt;&#xA;&lt;li&gt;go.......................| Difficulty: 1&lt;/li&gt;&#xA;&lt;li&gt;playwright........| Difficulty: 3&lt;/li&gt;&#xA;&lt;li&gt;weird.................| Difficulty: 3&lt;/li&gt;&#xA;&lt;li&gt;conscience.......| Difficulty: 2&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;...so after training, when I will input a word, the network should return  either 1, 2 or 3, depending on how hard the network thinks the word is to spell. To sum up, my question is: I am used to using number data with my projects instead of strings, so what steps should I take when creating a network based around strings for the first time? I use numpy only&lt;/p&gt;&#xA;" OwnerUserId="37363" LastEditorUserId="37363" LastEditDate="2017-08-16T20:33:35.263" LastActivityDate="2018-02-13T19:10:29.917" Title="How do I go from number data to string data (NN)?" Tags="|machine-learning|python|neural-network|classification|backpropagation|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22317" PostTypeId="2" ParentId="22264" CreationDate="2017-08-16T17:59:17.227" Score="4" Body="&lt;p&gt;I'm not sure about other fields but recently in the field of deep neural network training there is this arXiv submission, &lt;a href=&quot;https://arxiv.org/abs/1705.08292&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Marginal Value of Adaptive Gradient Methods in Machine Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. &lt;strong&gt;We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance.&lt;/strong&gt; These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28556" LastEditorUserId="381" LastEditDate="2017-08-16T18:35:19.593" LastActivityDate="2017-08-16T18:35:19.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22318" PostTypeId="2" ParentId="22292" CreationDate="2017-08-16T18:05:29.280" Score="1" Body="&lt;p&gt;In R many great improvements are typically made by creating new libraries rather than changing the actual functions in base R itself. datatable, read_csv in readr, tibbles, etc are very good examples.&lt;/p&gt;&#xA;&lt;p&gt;read.csv vs fread as &lt;a href=&quot;https://stackoverflow.com/a/24428572/9214357&quot;&gt;answered by Matt Dowle himself&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;read.csv coerces strings as factors by default which is not done by read_csv and fread. So if that is what you need, read.csv will work well. Now answering your question,there is no disadvantage using the fast readers to read small files apart from a small speedup that you might get.&lt;/p&gt;&#xA;" OwnerUserId="28474" LastEditorUserId="98307" LastEditDate="2020-08-11T14:52:58.920" LastActivityDate="2020-08-11T14:52:58.920" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22319" PostTypeId="2" ParentId="22281" CreationDate="2017-08-16T18:14:14.663" Score="0" Body="&lt;p&gt;It was not clear when you said &quot;reading only a subset of the records of a file into R&quot;. If head(dataset, 100) is the subset that you intended, it doesn't read the entire file into memory. It just reads the first 100 rows into memory.&lt;/p&gt;&#xA;" OwnerUserId="28474" LastActivityDate="2017-08-16T18:14:14.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22324" PostTypeId="1" AcceptedAnswerId="22326" CreationDate="2017-08-17T01:32:13.283" Score="5" ViewCount="1445" Body="&lt;p&gt;I have two sets of topics obtained from two different sets of news paper articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${y_1, y_2, ..., y_n}$ includes the main topics of 'Y' news paper set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to find clusters in the two sets that are similar/related by considering the cluster attributes as given in the example below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example 1,&#xA;**X1 in Cluster_1** is mostly similar/related to **Y2 in Cluster_2**&#xA;**X2 in Cluster_1** is mostly similar/related to **Yn in cluster_2**&#xA;and so on.&#xA;&#xA;Example 2:&#xA;News about Yet in Cluster_1 is mostly similar/related to News about Science in Cluster_2&#xA;News about Floods in Cluster_1 is mostly similar/related to News about Rains in Cluster_2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since, I am dealing with two separate sets of clusters, what would be a suitable measurement/method I can use to connect the clusters in the two different sets?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="29575" LastEditDate="2017-08-18T05:55:56.627" LastActivityDate="2017-08-18T05:55:56.627" Title="Combine two sets of clusters" Tags="|machine-learning|data-mining|unsupervised-learning|topic-model|dirichlet|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22325" PostTypeId="2" ParentId="17471" CreationDate="2017-08-17T03:47:34.967" Score="2" Body="&lt;p&gt;Kenny's answer is correct – if you're using convolutional &lt;em&gt;D&lt;/em&gt;, output of layers before dense may serve as features. My intuition is that it will work better for AC-GANs (or similar architectures, which make D classify input in addition to determining if it's fake or real).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an approach called &lt;a href=&quot;https://arxiv.org/abs/1605.09782&quot; rel=&quot;nofollow noreferrer&quot;&gt;BiGAN&lt;/a&gt; which adds an &lt;em&gt;Encoder&lt;/em&gt; component able to map generated and training samples to latent distribution &lt;em&gt;z&lt;/em&gt; used to &quot;initialize&quot; generator. Authors show that it can effectively be used as a feature set for transfer learning and other tasks.&lt;/p&gt;&#xA;" OwnerUserId="25504" LastActivityDate="2017-08-17T03:47:34.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22326" PostTypeId="2" ParentId="22324" CreationDate="2017-08-17T03:56:33.550" Score="6" Body="&lt;p&gt;To compare two LDA topics, you're really trying to compute the distance between two probability distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One such measure that's commonly used in these circumstances is the &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Hellinger_distance&quot; rel=&quot;noreferrer&quot; title=&quot;Hellinger Distance&quot;&gt;Hellinger Distance&lt;/a&gt;. To find the closest match for $x_1$ in the topics for $y$, you would calulate the Hellinger Distance between $x_1$ and each $y$ topic, then take the lowest one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that there's no guarantee whatsoever that the &quot;most similar&quot; topic in this sense would be remotely, subjectively similar.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-08-17T03:56:33.550" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22327" PostTypeId="1" AcceptedAnswerId="22353" CreationDate="2017-08-17T04:22:16.493" Score="-1" ViewCount="208" Body="&lt;p&gt;I am running a set of input parameters in SVM. One if the input contains all zero values. I know that this kind input should be omitted, but I don't know the reason why. Can anyone help me?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="38063" LastActivityDate="2017-08-17T17:05:30.697" Title="The effect of all zero value as the input of SVM" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22328" PostTypeId="2" ParentId="22314" CreationDate="2017-08-17T07:16:14.633" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;so what steps should I take when creating a network based around strings for the first time?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Neural networks work with numerical data. They also work best with relatively small floating point numbers, centred around zero. You can be less strict about that part, but you will often see the approach in neural networks of calculating the mean and standard deviation from the training data, for each feature, then converting all the features by doing &lt;code&gt;x = (x - means) / stds&lt;/code&gt; (you want to store these scaling factors you used along with the network data, because you will want to re-used the same values when you use the network to make predictions later).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what do you do if the input data is not already in this form? You prepare it in your code, just before using it to train or predict. It is a very common structure to see in machine learning scripts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;raw_features, raw_labels = load_from_disk( some_data_source )&#xA;all_features = convert_features( raw_features )&#xA;all_labels = convert_labels( raw_labels )&#xA;train_X, test_X, train_y, test_y = split_data( all_features, all_labels )&#xA;model = build_model( .... various model params ....)&#xA;model.fit( train_X, train_y )&#xA;test_predictions = model.predict( test_X )&#xA;report_accuracy( test_predictions, test_Y )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above is rough pseudo code, so typically all the functions above have different names, or are multiple lines that do the same thing that you might not bother to encapsulate into a re-usable method if you are writing a quick script. The part I have shown that splits the features might be built in to the training function, and it is also common that the training process can use the test data to help monitor progress.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the loading and conversion takes a long time, you might do it in a separate script and save the resulting NumPy array in a separate file to load it quicker next time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the part you are concerned about is how you might build a &lt;code&gt;convert_features&lt;/code&gt; section of your code from the starting strings. The answer is to use whatever values you can extract from your strings that might be relevant. The string length might be a simple start i.e. &lt;code&gt;len( text )&lt;/code&gt; - but you can also look into any other measure you can figure out (e.g. number of vowels, which uncommon bi-grams are in the word). Deciding which features to try and testing between them is &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_engineering&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature engineering&lt;/a&gt;, and this often involves some creativity. The important thing is that the features must all be numeric. For a neural network, you should also try and make them relatively small and/or convert them to have mean 0, standard deviation 1 before going to next stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use the network to make predictions later, you have to repeat most of the pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = load_model( model_file_or_identifier )&#xA;raw_features, raw_labels = fetch_data( some_data_source )&#xA;X = convert_features( raw_features )&#xA;y = convert_labels( raw_labels )&#xA;predictions = model.predict( X )&#xA;report_predictions( X, y )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-17T07:22:31.717" LastActivityDate="2017-08-17T07:22:31.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22329" PostTypeId="1" CreationDate="2017-08-17T07:46:40.723" Score="1" ViewCount="105" Body="&lt;p&gt;I recently read a post describing the Support Vector Machine theory and brought about some idea that says it throws data from &lt;code&gt;n&lt;/code&gt; dimensions to some &lt;strong&gt;higher&lt;/strong&gt; dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/15zrpp/please_explain_support_vector_machines_svm_like_i/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM description&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanna know how can we &lt;em&gt;generate&lt;/em&gt; more dimensional features from given &lt;code&gt;n&lt;/code&gt; features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whats the basis of these &lt;strong&gt;extra&lt;/strong&gt; dimensions? Won't these be dependent on the previous &lt;code&gt;n&lt;/code&gt; dimensions, and if so how can we say that these are independent &quot;dimensions&quot;??&lt;/p&gt;&#xA;" OwnerUserId="33020" LastActivityDate="2017-08-17T07:46:40.723" Title="Support Vector Machines: How can you generate higher dimensional data from lower ones?" Tags="|svm|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="22334" PostTypeId="2" ParentId="22260" CreationDate="2017-08-17T11:28:47.983" Score="7" Body="&lt;p&gt;I expect someone somewhere has used a RF estimator inside RL to approximate action values, if only to assess it as a comparison to other function approximators. However, it does look like from a web search that this is not used widely, and I could not find an example either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problem with a RL/RF hybrid with RF as the value estimator is that the random forest base algorithm is not an online one - it works across a finalised data set and processes the whole batch in order to do things like bagging. Even when used as an estimator using experience replay, support for online learning is a desirable feature, and values are typically fed into supervised learning part in small or medium batches. That is because the action values that are learned by the internal estimation function in reinforcement learning are non-&lt;a href=&quot;https://en.wikipedia.org/wiki/Stationary_process&quot; rel=&quot;noreferrer&quot;&gt;stationary&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have learned the action values for the current policy, and in most RL well before estimates for the values have converged, you change the policy. This changes the expected action values, so your estimator must be able to forget older data and bias towards most recent values. Algorithms that can be made to work online can do this, purely offline ones cannot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are &lt;a href=&quot;http://www.vision.cs.chubu.ac.jp/CV-R/pdf/AmirICCVW2009.pdf&quot; rel=&quot;noreferrer&quot;&gt;online Random Forests&lt;/a&gt; which have the necessary alterations to work with non-stationary data. I don't see any reason why they could not work. Decision trees in general have been used successfully as estimators in RL - see &lt;a href=&quot;https://pdfs.semanticscholar.org/f9b3/0e1f6d85cb77e95ff1d580ee67d7406f1dd6.pdf&quot; rel=&quot;noreferrer&quot;&gt;Reinforcement Learning with Decision Trees&lt;/a&gt;. It looks like that in principle it could be done.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-17T11:36:21.093" LastActivityDate="2017-08-17T11:36:21.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22335" PostTypeId="1" AcceptedAnswerId="22338" CreationDate="2017-08-17T11:53:15.637" Score="44" ViewCount="23911" Body="&lt;p&gt;I was reading this blog post titled: &lt;a href=&quot;https://www.technologyreview.com/s/604122/the-financial-world-wants-to-open-ais-black-boxes/&quot; rel=&quot;noreferrer&quot;&gt;The Financial World Wants to Open AI’s Black Boxes&lt;/a&gt;, where the author repeatedly refer to ML models as &quot;black boxes&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A similar terminology has been used at several places when referring to ML models. &lt;strong&gt;Why is it so?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not like the ML engineers don't know what goes on inside a neural net. Every layer is selected by the ML engineer knowing what activation function to use, what that type of layer does, how the error is back propagated, etc.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2020-08-20T18:30:24.903" Title="Why are Machine Learning models called black boxes?" Tags="|machine-learning|terminology|" AnswerCount="9" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22337" PostTypeId="1" AcceptedAnswerId="22366" CreationDate="2017-08-17T11:59:57.020" Score="12" ViewCount="3939" Body="&lt;p&gt;I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using &lt;code&gt;click&lt;/code&gt; to gather the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My concern is what will happen if the data gathering script is writing to the HDF5 file and the yet-to-be ML application tries to read data from the same file?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I took a look at The HDF Group's documentation about &lt;a href=&quot;https://support.hdfgroup.org/HDF5/PHDF5/&quot; rel=&quot;noreferrer&quot;&gt;HDF5 parallel I/O&lt;/a&gt;, but that didn't really clear things up for me.&lt;/p&gt;&#xA;" OwnerUserId="29806" LastEditorUserId="29575" LastEditDate="2017-08-18T05:43:33.937" LastActivityDate="2017-08-18T05:43:33.937" Title="Can HDF5 be reliably written to and read from simultaneously by separate python processes?" Tags="|python|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22338" PostTypeId="2" ParentId="22335" CreationDate="2017-08-17T12:47:35.030" Score="58" Body="&lt;p&gt;The &lt;em&gt;black box&lt;/em&gt; thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the &lt;em&gt;explainability&lt;/em&gt; of the function modelled by the machine learning algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly catalogued (e.g. because the value of certain component of the input vector was too low).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same applies to decision trees: you can &lt;em&gt;follow&lt;/em&gt; the logic applied by the tree and understand why a certain element was assigned to one class or the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, deep neural networks are the paradigmatic example of black box algorithms. No one, not even the most expert person in the world grasp the function that is actually modeled by training a neural network. An insight about this can be provided by &lt;a href=&quot;https://blog.openai.com/adversarial-example-research/&quot; rel=&quot;noreferrer&quot;&gt;adversarial examples&lt;/a&gt;: some slight (and unnoticeable by a human) change in a training sample can lead the network to think that it belongs to a totally different label. There are some techniques to create adversarial examples, and some techniques to improve robustness against them. But given that no one actually knows all the relevant properties of the function being modeled by the network, it is always possible to find a novel way to create them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Humans are also black boxes and &lt;a href=&quot;https://twitter.com/goodfellow_ian/status/870813726751956993?lang=en&quot; rel=&quot;noreferrer&quot;&gt;we are also sensible to adversarial examples&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-08-18T07:50:10.813" LastActivityDate="2017-08-18T07:50:10.813" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22339" PostTypeId="2" ParentId="22335" CreationDate="2017-08-17T12:48:35.517" Score="7" Body="&lt;p&gt;It comes down to model interpretability and explainability. Given the output of a simpler model, it is possible to identify exactly how each input contributes to model output, but that gets more difficult as models get more complex. For example with regression you can point to the coefficients, with a decision tree you can identify the splits. And with this information, you could derive rules to explain model behaviour. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as the number of model parameters increases, it becomes increasingly difficult to explain exactly what combinations of input lead to the final model output, or derive rules from the model's behaviour. Say in the financial industry when the COO comes over and asks 'so, why did your high frequency trading algo break the economy', he doesn't want to hear how it was built, just why it sent him bankrupt. It will be possible to state how the model was constructed, but it might not be possible to explain what combinations of factors that the model received as input led to the output, and that’s why people are talking about black boxes.&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2017-08-17T12:48:35.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22340" PostTypeId="1" CreationDate="2017-08-17T13:23:01.120" Score="5" ViewCount="6436" Body="&lt;p&gt;The question &lt;a href=&quot;https://datascience.stackexchange.com/questions/17024/rnns-with-multiple-features&quot;&gt;RNN&amp;#39;s with multiple features&lt;/a&gt; is ambiguous and not explicitly in differentiating different features. I want to understand how to use RNN to predict time-series with multiple features containing non-numeric data as well. As a deep learning model, I assume I don't need to quantify the non-numeric elements. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose sales data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sales | Weather | Holiday | Temperature&#xA;100   | Windy   | Yes     | 3&#xA;2000  | Sunny   | Yes     | 20&#xA;200   | Sunny   | No      | 30&#xA;-5    | Stormy  | No      | 3&#xA;-50   | Cold    | No      | -50&#xA;500   | Cold    | Yes     | -20&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where I want to predict the &lt;code&gt;Sales&lt;/code&gt; column with the other columns. I have found demos such as &lt;a href=&quot;http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; about using RNN with numeric data, without enriching the data with non-numeric data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;strong&gt;How can I predict time series with multiple features contain numeric features and non-numeric features?&lt;/strong&gt;&lt;/h2&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Other interesting questions related to RNN&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/34045295/multivariate-time-series-rnn-using-tensorflow-is-this-possible-with-an-lstm-cel&quot;&gt;Multivariate time-series RNN -question in SO with numeric data&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/12721/time-series-prediction-using-arima-vs-lstm&quot;&gt;Time series prediction using ARIMA vs LSTM&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="15738" LastEditorUserId="91943" LastEditDate="2020-04-11T13:32:51.693" LastActivityDate="2020-04-11T13:32:51.693" Title="RNN time-series predictions with multiple features containing non-numeric features and numeric features?" Tags="|deep-learning|time-series|rnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22341" PostTypeId="2" ParentId="22335" CreationDate="2017-08-17T14:23:51.647" Score="1" Body="&lt;p&gt;I think the &lt;strong&gt;black box&lt;/strong&gt; concept as used in this way originates from black box &lt;em&gt;testing&lt;/em&gt; in software and hardware Quality Assurance. It is when you either choose not to / or even can't look into and see the inner working of what you are testing. It could be for a reason that it would be &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;impractical or impossible to peek into it (it is in a sealed environment and we simply can't look into it) - But it might as well be&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;because there is a larger chance of writing crappy tests if one can see the inside. Larger risk of (with or without intent) &quot;writing tests designed to pass&quot;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Writing the test to fit the thing that is being tested, lowering the chances of actually finding anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be &lt;strong&gt;perfectly possible&lt;/strong&gt; for a skilled signal engineer to peek into the inner workings of a neural network and check which features are being selected for in a particular training sequence.&lt;/p&gt;&#xA;" OwnerUserId="38086" LastActivityDate="2017-08-17T14:23:51.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22342" PostTypeId="1" CreationDate="2017-08-17T14:35:49.183" Score="2" ViewCount="2114" Body="&lt;p&gt;I have a big data and want to perform some of prediction models(such as Regression,Decision Tree,..) on it in R but I have a problem. As there are too many variables, Is there any way to identify which variables should i use as predictors in the model to predict a specific variable?&#xA;Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="38089" LastActivityDate="2017-08-18T13:25:14.787" Title="How to identify the appropriate predictors in regression model" Tags="|r|predictive-modeling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22343" PostTypeId="1" CreationDate="2017-08-17T14:53:08.927" Score="1" ViewCount="1235" Body="&lt;p&gt;I am doing certification and i have a project to complete. In project, they have said &quot;Find Patterns&quot;. What does it mean? what steps should i carry out?&lt;/p&gt;&#xA;" OwnerUserId="38091" LastEditorUserId="23402" LastEditDate="2017-08-18T06:25:34.207" LastActivityDate="2017-08-18T06:25:34.207" Title="What does &quot;Find pattern&quot; mean in data science?" Tags="|data-mining|dataset|categorical-data|" AnswerCount="1" CommentCount="0" ClosedDate="2017-08-18T06:23:57.210" ContentLicense="CC BY-SA 3.0" />
  <row Id="22344" PostTypeId="2" ParentId="22170" CreationDate="2017-08-17T14:54:33.760" Score="2" Body="&lt;p&gt;For categorical data, it is best to use &lt;code&gt;sample_weight&lt;/code&gt; instead of &lt;code&gt;class_weight&lt;/code&gt; argument. This can be done by simply giving all samples of a particular class the same weight. &lt;code&gt;sample_weight&lt;/code&gt; works for categorical data because it takes a numpy array as its value as opposed to a dictionary (which won't work for categorical class labels) in case of &lt;code&gt;class_weight&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See: &lt;a href=&quot;https://keras.io/models/sequential/#sequential-model-methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras sequential model methods&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You can either pass a flat (1D) Numpy array with the same length as&#xA;  the input samples (1:1 mapping between weights and samples), or in the&#xA;  case of temporal data, you can pass a 2D array with shape (samples,&#xA;  sequence_length), to apply a different weight to every timestep of&#xA;  every sample. In this case you should make sure to specify&#xA;  sample_weight_mode=&quot;temporal&quot; in compile().&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The other way, as you mentioned is using custom weighted loss function. A detailed discussion on it can be found &lt;a href=&quot;https://github.com/fchollet/keras/issues/2115&quot; rel=&quot;nofollow noreferrer&quot;&gt;here.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37827" LastActivityDate="2017-08-17T14:54:33.760" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22346" PostTypeId="2" ParentId="22343" CreationDate="2017-08-17T15:37:25.687" Score="2" Body="&lt;p&gt;&lt;em&gt;First&lt;/em&gt;, I'm going to directly quote the two inner questions you put in this question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Find Patterns&quot;, What does it mean?:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Find patterns is the term given to the process of analyzing data(Data Analysis) or dataset for insights in it that is to say; the classification of input into per-determined categories,recognize particular categories of input by their characteristics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when they ask you to &lt;strong&gt;find patterns&lt;/strong&gt;,they assume that you don't necessarily know what is in that data(it can be structured,semi-structured and unstructured) so you can't even make decisions upon that data.therefore you have to find patterns in it. You have to look for them, which can often involve a lot of hunch-following and guess-work combined with a bit of solid testing.you tend to know the patterns already and you search for them based on their features or attributes etc. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;what steps should i carry out?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Ask Question&lt;/li&gt;&#xA;&lt;li&gt;Do some background research&lt;/li&gt;&#xA;&lt;li&gt;Construct hypothesis&lt;/li&gt;&#xA;&lt;li&gt;Test with an experiment&lt;/li&gt;&#xA;&lt;li&gt;Analyze results and draw effective conclusions.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concerning Hypothesis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If the hypothesis is true,then proceed and report results&#xA;or Else,if it's false or partially true report results.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this can give you a glimpse.&lt;/p&gt;&#xA;" OwnerUserId="23402" LastEditorUserId="23402" LastEditDate="2017-08-17T16:17:08.793" LastActivityDate="2017-08-17T16:17:08.793" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22348" PostTypeId="1" CreationDate="2017-08-17T16:10:44.037" Score="4" ViewCount="5548" Body="&lt;p&gt;I am reading the Chapter-1 of the &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning book&lt;/a&gt;, where the following appears:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A wheel has a geometric shape, but its image may be complicated by&#xA;  shadows falling on the wheel, the sun glaring off the metal parts of&#xA;  the wheel, the fender of the car or an object in the foreground&#xA;  obscuring part of the wheel, and so on.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;One solution to this problem is to use ML to discover not only the&#xA;  mapping from representation to output but also the representation to&#xA;  itself. &lt;strong&gt;This approach is called representation learning.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here, I did not understand the exact definition of representation learning. I have referred to the wikipedia page and also Quora, but no one was explaining it clearly. The lack of explanation with a proper example is lacking too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors cite the autoencoder as an example for a representation learning algorithm. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An autoencoder is the combination of an encoder function, which&#xA;  converts the input data into a different representation, and a decoder&#xA;  function, which converts the new representation back into the original&#xA;  format&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can someone explain in simpler terms what exactly representation learning is?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Would be much better if explained with an example. The car wheel example would do too.]&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2017-08-22T15:16:59.267" Title="What is representation learning?" Tags="|feature-selection|terminology|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22349" PostTypeId="2" ParentId="22335" CreationDate="2017-08-17T16:12:26.727" Score="5" Body="&lt;p&gt;Black box models refer to any mathematical models whose equations are chosen to be as general and flexible as possible without relying on any physical/scientific laws.&lt;/p&gt;&#xA;&lt;p&gt;Grey box models are mathematical models where part of the equations (mathematical function) comes from physical known laws but the remaining part is assumed general function to compensate for the unexplained part.&lt;/p&gt;&#xA;&lt;p&gt;White box models are mathematical models completely built on physical laws and understanding of the system, like for example mechanical motion laws (model of aircraft ..etc)&lt;/p&gt;&#xA;&lt;p&gt;See &lt;a href=&quot;https://en.wikipedia.org/wiki/Mathematical_model#A_priori_information&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38098" LastEditorUserId="98307" LastEditDate="2020-08-20T18:30:24.903" LastActivityDate="2020-08-20T18:30:24.903" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="22350" PostTypeId="2" ParentId="22342" CreationDate="2017-08-17T16:52:42.723" Score="1" Body="&lt;p&gt;Check the correlation of the variables or features, the ones with the least correlation can be discarded and the ones with highest must be considered to perform prediction. However, least and highest is subjective to your requirement&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-08-17T16:52:42.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22351" PostTypeId="1" CreationDate="2017-08-17T16:57:29.223" Score="2" ViewCount="1442" Body="&lt;p&gt;To analyze a dataset from banking I have both numerical and categorical values. I transform them to analyze with k-prototypes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MB3GJ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MB3GJ.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The modified dataset:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;E.g.: Job (for 1 to 12 'cos there are 12 levels)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/h0fwc.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h0fwc.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I scale the dataset before doing the k-prototypes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I determine the optimal &quot;k&quot; to choose (coding)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought to execute:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(clustMixType)&#xA;&#xA;lbd &amp;lt;- lambdaest(BPor)&#xA;&#xA;kpres &amp;lt;- kproto(BPor, 5, lambda = lbd) #Change '5' for every possible value of k.&#xA;&#xA;print(kpres)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then, calculate the sum of within cluster error (choosing the little one).&lt;/p&gt;&#xA;" OwnerUserId="38101" LastEditorUserId="24000" LastEditDate="2017-08-17T17:34:15.450" LastActivityDate="2021-10-06T10:47:25.107" Title="How to choose the optimal k in k-protoypes?" Tags="|r|clustering|k-means|categorical-data|numerical|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22352" PostTypeId="2" ParentId="22342" CreationDate="2017-08-17T17:03:24.460" Score="2" Body="&lt;p&gt;I assume by too many variables you mean the case where number of variables is greater than the number of observations ($p&amp;gt;n$) or close to that. You will need to either reduce the number of variables before feeding them in, use some kind of automatic feature selection methods as @M.Mashayekhi has pointed out or utilize learning models that are robust to high feature dimension. Keep in mind that feature selection/creation, in my opinion, is the most important part of model building. There is no point in using a complex machine learning algorithm if all your input variables are useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is always a good idea to first reduce the number of input variables by business intuition or context. By that I mean answering the questions: &quot;Do I really need all of these variables to be in the model?&quot;, &quot;What variables just wouldn't make sense to include in the model as a predictor?&quot; &quot;Are there variables that has to be in the model even if it is not the strongest predictor, simply for business reasons? If these questions are not easy to answer, or you have already reduced the number of variables but are still in the case of too many variables, then you can try the following.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Principal Component Analysis (PCA)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One suggestion would be to use PCA to reduce your feature space then run learning algorithms like regression, decision trees on the reduced feature space. You can do that in R using &lt;code&gt;pca&lt;/code&gt;. Pick the first however many principal components where the next PC has a decline in &lt;em&gt;marginal&lt;/em&gt; variance explained (Since each addition principal component always increases variance explained). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regression with Lasso ($\mathcal{L1}$) Regularization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative would be to let the model do the feature selection for you. A good starter would be a regression with the lasso ($\mathcal{L1}$) penalty, this shrinks the estimated coefficients toward zero. It does that by shrinking large effect coefficients more slowly toward zero than small effect ones. This effectively removes small effect coefficients from the model by setting them equal to zero, hence, auto feature selection. The challenge is to pick a good shrinkage parameter $\lambda$, which governs how much you are shrinking the coefficients. You can do that with cross validation. &lt;code&gt;glmnet&lt;/code&gt; in R let's you run lasso regressions, and &lt;code&gt;glmnet.cv&lt;/code&gt; let's you pick $\lambda$ using CV. Also note that by shrinking coefficients, lasso regression reduces prediction errors by balancing between variance reduction and bias increase. You will have to decide which is more important, inference or making good predictions, since unlike OLS regression, estimates by lasso regression are no longer unbiased.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Random Forest&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest is an ensemble tree learning algorithm (implemented in &lt;code&gt;randomForest&lt;/code&gt; in R) which is robust to many useless variables. Particularly, it works well in the case of $p&amp;gt;n$. RF is very popular because of its robustness and simple tuning. The downside is this is what's called a &lt;a href=&quot;https://datascience.stackexchange.com/questions/22335/why-are-machine-learning-models-called-black-boxes&quot;&gt;Black Box&lt;/a&gt; algorithm. Meaning that there is no good way to interpret the relationship between the input and dependent variables. You can however use something like the &quot;variable importance plot&quot; to give you a rough idea of what variables were most useful during the splits &lt;em&gt;on average&lt;/em&gt;. This is fine if you are only concerned about making good predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other techniques&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are certainly other techniques that goes well with high dimensional data like &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;Support Vector Machines&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting%22Gradient%20Boosting%22&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boosted Regression Trees&lt;/a&gt; (&lt;a href=&quot;https://stats.stackexchange.com/questions/173390/gradient-boosting-tree-vs-random-forest?rq=1&quot;&gt;Difference between RF and Boosted Trees&lt;/a&gt;), but they might usually require a lot more tuning than the ones mentioned above.&lt;/p&gt;&#xA;" OwnerUserId="38096" LastEditorUserId="38096" LastEditDate="2017-08-18T13:25:14.787" LastActivityDate="2017-08-18T13:25:14.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22353" PostTypeId="2" ParentId="22327" CreationDate="2017-08-17T17:05:30.697" Score="0" Body="&lt;p&gt;Your question is not clear, but I'm assuming you're trying to say one of the features you used for SVM has values containing zeros, and how does this affect SVM in terms of making a decision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about it in a Euclidean hyperplane(three dimensions for better visualization), if the input contains all zeros where will you plot a point related to that feature on the hyperplane. The value of that point will be zero, so effectively you're not using that feature at all. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-08-17T17:05:30.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22354" PostTypeId="2" ParentId="22348" CreationDate="2017-08-17T17:27:30.610" Score="7" Body="&lt;p&gt;If you look at older machine learning algorithms, they rely on the input being a feature and learn a classifier, regressor, etc on top of that. Most of these features were hand crafted, meaning, they were designed by humans. Classical examples of features in computer vision include Harris, SIFT, LBP, etc . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with these is that they were designed by humans based on heuristics. Images can be represented using these features and ML algorithms can be applied on top of that. However, they may not be the most optimal in terms of the objective function, i.e, it may be possible to design better features that can lead to lower objective function values. Instead of hand crafting these image representations, we can learn them. That is known as representation learning. We can have a neural network which takes the image as an input and outputs a vector, which is the feature representation of the image. This is the representation learner. This be followed by another neural network that acts as the classifier, regressor, etc . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in the wheel example, you can try to manually describe how a wheel should look like and how it can be represented. Say, it should be circular, be black in colour, have treads, etc. But these are all hand crafted features and may not generalize to all situations. For example, if you look at the wheel from a different angle, it might be oval in shape. Or the lighting may cause it to have lighter and darker patches. These kinds of variations are hard to account for manually. Instead, we can let the representation learning neural network learn them from data by giving it several positive and negative examples of a wheel and training it end to end. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this makes sense. Let me know if I can clarify any point I made. &lt;/p&gt;&#xA;" OwnerUserId="38105" LastActivityDate="2017-08-17T17:27:30.610" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22355" PostTypeId="1" AcceptedAnswerId="22369" CreationDate="2017-08-17T17:50:24.123" Score="5" ViewCount="10361" Body="&lt;p&gt;I am working on a clustering project where we have collected protein data from over 100 patients samples. This data is normalized and log transformed. The goal is to cluster samples based upon their similarities, I am using hierarchal clustering and trying out combinations of distance metrics and clustering algorithms. (We haven't made a decision on distance method or clustering algorithms) My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate your input on this one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="38107" LastActivityDate="2017-08-18T07:17:13.683" Title="When is centering and scaling needed before doing hierarchical clustering?" Tags="|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22356" PostTypeId="2" ParentId="22340" CreationDate="2017-08-17T18:51:58.533" Score="6" Body="&lt;p&gt;Possible ways to expose these categorical variables as part of a time-step to be fed to an RNN include producing &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/one_hot&quot; rel=&quot;noreferrer&quot;&gt;One_Hot Tensors&lt;/a&gt;, &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/lookup/HashTable&quot; rel=&quot;noreferrer&quot;&gt;Hash Table Representations&lt;/a&gt;, or &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/keras/layers/Embedding&quot; rel=&quot;noreferrer&quot;&gt;Embedding Layer Outputs&lt;/a&gt; from the categorical fields and concatenating all of the feature tensors at each time-step to feed to your RNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, instead of discrete categories, you're looking to interpret the strings that make up the fields as sequences themselves, you could encode them numerically at a word- or character level and pass them through a subsidiary RNN to produce the internal representation at each timestep. This would be appropriate only for cases where you have a very large variety among strings in those fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apologies for the Tensorflow links if you're not using that framework.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-08-17T18:51:58.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22357" PostTypeId="2" ParentId="22348" CreationDate="2017-08-17T19:16:01.907" Score="3" Body="&lt;p&gt;&quot;One solution to this problem is to use ML to discover not only the mapping from representation to output but also the representation to itself. This approach is called representation learning.&quot;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Mapping the representation to output is called &quot;Hetero association.&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mapping the representation to itself is called &quot;Auto association.&quot; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both approaches are about classification but in the first case you are associating a representation (or object, or item, or vector) to a label (for a category different from the original represented object, item, or vector). &#xA;In the second case you are associating a representation (or object, or item, or vector) to a label (for the original represented object, item, or vector). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By representation, they mean an image, object, item, vector, etc.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So an auto associative encoder can take an image and output the category for the original image, while a hetero associative encoder can take an image an output a learned association for that image (for example, the word &quot;dog&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would autoassociative learning be useful? Spell checkers are a good example. Given a small lexicon of words (&quot;dog, hog, hat, heat&quot;) each word forming its own category, we can create an auto encoder that when presented with each word, featurizes the word, and returns the original category / word. Let's featurize the words using trigrams. So we split each word into its feature set and let that feature set represent the orginal word as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dog  --&amp;gt; {'  d', ' do', 'dog', 'og ', 'g  '} --&amp;gt; dog&#xA;&#xA;hog  --&amp;gt; {'  h', ' ho', 'hog', 'og ', 'g  '} --&amp;gt; hog&#xA;&#xA;hat  --&amp;gt; {'  h', ' ha', 'hat', 'at ', 't  '} --&amp;gt; hat&#xA;&#xA;heat --&amp;gt; {'  h', ' he', 'hea', 'eat', 'at ', 't  '} --&amp;gt; heat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we present a new word, featurize it, and see where it falls:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hot --&amp;gt; {'  h', ' ho', 'hot', 'ot ', 't  '} --&amp;gt; ???&#xA;&#xA;&quot;hot&quot; matches 0% of the features in &quot;dog&quot;.&#xA;&#xA;&quot;hot&quot; matches 40% of the features in &quot;hog&quot;.&#xA;&#xA;&quot;hot&quot; matches 40% of the features in &quot;hat&quot;. &#xA;&#xA;&quot;hot&quot; matches 33% of the features in &quot;heat&quot;. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;hot would be classified as either &quot;hog&quot; or &quot;hat&quot; if the match threshold were 40% or below. If we had a higher match threshold than 40% then there would be no match and hence no proper classification for the word hot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The auto encoding of each of the lexicon entries (dog, hog, hat, heat) allowed us to take the original representation of each word and split it into features, which can be combined to point back to the original representation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dog  --&amp;gt; {'  d', ' do', 'dog', 'og ', 'g  '} --&amp;gt; dog&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A hetero encoding could take each of the lexicon entries and map it to a different category.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dog --&amp;gt; animal&#xA;&#xA;hog --&amp;gt; animal&#xA;&#xA;hat --&amp;gt; thing&#xA;&#xA;heat --&amp;gt; thing&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;a simple hash table could suffice as the hetero encoder in this example. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks take an input vector of features and output the category either in an autoassociative or heterassociative manner. &lt;/p&gt;&#xA;" OwnerUserId="38110" LastEditorUserId="11097" LastEditDate="2017-08-22T15:16:59.267" LastActivityDate="2017-08-22T15:16:59.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22359" PostTypeId="2" ParentId="22302" CreationDate="2017-08-17T20:08:50.930" Score="4" Body="&lt;p&gt;Generally, I would refer to this as transfer learning or network adaptation. That is, taking a network that has learned useful features from one domain and adapting that network and its developed features to another domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, there appear to be &lt;a href=&quot;http://blog.revolutionanalytics.com/2016/08/deep-learning-part-2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;many sources&lt;/a&gt; that closely conflate fine tuning with transfer learning. Therefore, I would say the difference in terminology is primarily opinion-based and suggest closure of this question on those grounds.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-08-17T20:08:50.930" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22360" PostTypeId="1" CreationDate="2017-08-17T20:16:08.447" Score="2" ViewCount="130" Body="&lt;p&gt;I'm writing a battery of tests (in Python) for the purpose of measuring the speed of my company's different computational instances. The goal is to see how fast different AWS EC2 instances are at running different ML models or common data science tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm reaching out to ask if anyone knows of any time-consuming (but realistic) models that can be built using only the standard Anaconda packages and either the built in datasets, or a popular dataset known in the industry? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal would be to give end users a sense of how much computational power they need, and I think using some popular data sets or models to compare relative time to completion would be the best way for them to choose the right amount of power for their needs.&lt;/p&gt;&#xA;" OwnerUserId="38112" LastEditorUserId="29575" LastEditDate="2017-08-18T05:56:03.993" LastActivityDate="2024-02-11T23:01:15.113" Title="What are some good models to test the speed of a data science machine?" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22361" PostTypeId="2" ParentId="22355" CreationDate="2017-08-17T21:05:25.030" Score="1" Body="&lt;p&gt;If your variables are of incomparable units then you should standardize variables by scaling. K-clustering is 'isotropic' in all directions, meaning that the clusters tend to be more or less round.  By not scaling, you're essentially putting a weight on certain variables.&lt;/p&gt;&#xA;" OwnerUserId="38112" LastActivityDate="2017-08-17T21:05:25.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22362" PostTypeId="2" ParentId="22335" CreationDate="2017-08-17T21:06:13.633" Score="4" Body="&lt;p&gt;&lt;strong&gt;ML engineers don't know what goes on inside a neural net&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry to contradict you, but it's true. They know how neural networks learn, but they do not know what any given neural network has learned. The logic learned by neural networks is notoriously inscrutable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point of using machine learning is usually to learn the rules that a programmer or domain expert would not think of. This is inherently difficult to figure out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's analogous to a conventional computer program written with one letter variable names, no comments, no obvious structure, using obscure mathematics, and all by someone who is now dead. You can step through it in a debugger, but it is still far from clear how it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rarely, someone does take the trouble to figure out what a neural network does. For example, the &lt;a href=&quot;https://en.wikipedia.org/wiki/Min-conflicts_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;min-conflicts algorithm&lt;/a&gt; was discovered by analyzing a neural network trained on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Eight_queens_puzzle&quot; rel=&quot;nofollow noreferrer&quot;&gt;N-queens problem&lt;/a&gt;. But it's a lot of work.&lt;/p&gt;&#xA;" OwnerUserId="38118" LastActivityDate="2017-08-17T21:06:13.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22363" PostTypeId="2" ParentId="22290" CreationDate="2017-08-17T21:08:05.157" Score="1" Body="&lt;p&gt;I assume you are performing an independent samples t-test. That the Ns are different isn't necessarily a problem -- the mean is an unbiased estimator -- but just how different are the sample sizes?&#xA;As you describe, you are probably violating the homogeneity of variance assumption. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;T-tests tend to be fairly robust to this violation, but you might consider having some outlier treatment and/or using a nonparametric test. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-08-17T21:08:05.157" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22365" PostTypeId="1" CreationDate="2017-08-18T02:02:38.580" Score="3" ViewCount="151" Body="&lt;p&gt;I have two sets of clusters (Cluster_set_1 and Cluster_set_2) obtained by analysing two different sets of news paper articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A cluster includes word/word phrases as given in the examples below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C1 in Cluster_set_1: Energy, Fuel, Oil, Mining&#xA;C2 in Cluster_set_1: school, education, students, schools, million, read&#xA;...&#xA;...&#xA;etc.&#xA;&#xA;C1 in Cluster_set_2: Gas, oil, pipeline&#xA;C2 in Cluster_set_2: program, business, management,information, reports&#xA;...&#xA;...&#xA;etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to find clusters in the two cluster sets that are similar/related by considering the word/word phrases in the cluster as given in the example below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example: &#xA;Cluster of 'Energy, Fuel, Oil, Mining' in Cluster_set_1 is mostly similar/related to &#xA;Cluster of 'Gas, oil, pipeline' in Cluster_set_2&#xA;&#xA;Reason: Because they both includes word/word phrases related to Energy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since, I am dealing with two separate sets of clusters, what would be a suitable approach that I can use to connect the clusters in the two different sets?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="29575" LastEditDate="2017-08-18T04:50:59.770" LastActivityDate="2017-12-19T14:07:09.323" Title="Map similar like clusters in two different cluster sets" Tags="|machine-learning|classification|data-mining|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22366" PostTypeId="2" ParentId="22337" CreationDate="2017-08-18T03:59:27.633" Score="11" Body="&lt;p&gt;HDF5 parallel I/O will not solve this problem.  That technology is primarily intended for performance, not for collision avoidance.&lt;/p&gt;&#xA;&lt;p&gt;What you want is know as &lt;a href=&quot;https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesSwmrDocs.html&quot; rel=&quot;noreferrer&quot;&gt;SWMR (single-writer/multiple-reader)&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Data acquisition and computer modeling systems often need to analyze and visualize data while it is being written. It is not unusual, for example, for an application to produce results in the middle of a run that suggest some basic parameters be changed, sensors be adjusted, or the run be scrapped entirely.&lt;/p&gt;&#xA;&lt;p&gt;To enable users to check on such systems, we have been developing a concurrent read/write file access pattern we call SWMR (pronounced swimmer). SWMR is short for single-writer/multiple-reader. SWMR functionality allows a writer process to add data to a file while multiple reader processes read from the file.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;SWMR was first included in &lt;a href=&quot;https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.0/src/hdf5-1.10.0-RELEASE.txt&quot; rel=&quot;noreferrer&quot;&gt;HDF5 version 1.10.0 released on 2016-03-30&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Concurrent Access to HDF5 Files - Single Writer/ Multple Reader (SWMR)&lt;/p&gt;&#xA;&lt;p&gt;The Single Writer/ Multiple Reader or SWMR feature enables users to&#xA;read data concurrently while writing it. Communications between the&#xA;processes and file locking are not required. The processes can run&#xA;on the same or on different platforms as long as they share a common&#xA;file system that is POSIX compliant.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-08-18T03:59:27.633" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22367" PostTypeId="2" ParentId="22335" CreationDate="2017-08-18T05:34:42.257" Score="23" Body="&lt;p&gt;While I agree on &lt;a href=&quot;https://datascience.stackexchange.com/a/22338/8820&quot;&gt;ncasas answer&lt;/a&gt; in most points (+1), I beg to differ on some:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Decision Trees can be used as black box models, too. In fact, I'd say in most cases they are used as black-box models. If you have 10,000 features and a tree of depth of 50 you cannot reasonably expect a human to understand it.&lt;/li&gt;&#xA;&lt;li&gt;Neural Networks can be understood. There are many analyzation techniques (see &lt;a href=&quot;https://arxiv.org/abs/1707.09725&quot; rel=&quot;noreferrer&quot;&gt;chapter 2.5 of my master thesis&lt;/a&gt; for some which are aimed at improving the model). Especially occlusion analysis (Figure 2.10), Filter visualization (Figure 2.11). Also the &lt;a href=&quot;https://arxiv.org/abs/1602.04938v3&quot; rel=&quot;noreferrer&quot;&gt;Why Should I Trust You?&lt;/a&gt; paper (&lt;a href=&quot;http://www.shortscience.org/paper?bibtexKey=journals/corr/1602.04938&quot; rel=&quot;noreferrer&quot;&gt;my notes&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Explaining the prediction of a black-box model by fancy occlusion analysis (from &quot;Why should I trust you?&quot;):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/uy6NK.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uy6NK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to point out &lt;a href=&quot;https://arxiv.org/abs/1606.03490&quot; rel=&quot;noreferrer&quot;&gt;The Mythos of Model Interpretability&lt;/a&gt;. It formulates some ideas about interpretability in a concise way.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Your question&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why are Machine Learning models called black boxes?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How people use it&lt;/strong&gt;: Because they do not model the problem in a way which allows humans to directly say what happens for any given input.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Personal thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I don't think this notion of a &quot;black box model&quot; makes much sense. For example, think of weather forecasting. You cannot expect any human to say which weather will be predicted if he is only given the data. Yet most people would not say that physical weather models are black box models. So where is the difference? Is it only the fact that one model was generated using data and the other one was generated using insights into physics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When people speak of black box models they usually say it as if it is a bad thing. But humans are black box models, too. The critical difference I see here is that the class of errors humans make is easier to predict for humans. Hence it is a training problem (adverserial examples on the NN side) and an education problem (teaching humans how NNs work).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How the term 'black-box model' should be used&lt;/strong&gt;: An approach which makes more sense to me is to call the problem a &quot;black box problem&quot;, similar to what &lt;a href=&quot;https://datascience.stackexchange.com/a/22349/8820&quot;&gt;user144410&lt;/a&gt; (+1) writes. Hence any model which only treats the problem as a black box - hence something you can put input in and get output out - is a black box model. Models which have insights (not only assume!) about the problem are not black-box models. The insight part is tricky. Every model makes restrictions on the possible function which it can model (yes, I know about the universal approximation problem. As long as you use a fixed-size NN it doesn't apply). I would say something is an insight into the problem if you know something about the relationship of input and output without poking the problem (without looking at data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What follows from this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Neural Networks can be non-blackbox (whitebox?)&lt;/li&gt;&#xA;&lt;li&gt;Logistic Regression can be a black-box model.&lt;/li&gt;&#xA;&lt;li&gt;It's more about the problem and your insights about it, less about the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-08-19T08:56:06.437" LastActivityDate="2017-08-19T08:56:06.437" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22368" PostTypeId="1" CreationDate="2017-08-18T06:34:25.383" Score="1" ViewCount="592" Body="&lt;p&gt;I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come across few articles on Feature Extraction using Scipy / Sci-kit learn in Python but I want to know is there any way to do the same in Apache Spark using some packages or modules within Spark&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas that enable me to use Pyspark is also welcomed here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciating your thoughts on this. Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="38134" LastActivityDate="2017-08-18T06:34:25.383" Title="How to do Feature Extraction using Apache Spark" Tags="|feature-selection|apache-spark|feature-extraction|feature-engineering|pyspark|" AnswerCount="0" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="22369" PostTypeId="2" ParentId="22355" CreationDate="2017-08-18T07:17:13.683" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It depends on the type of data you have. For some types of well defined data, there may be no need to scale and center. A good example is geolocation data (longitudes and latitudes). If you were seeking to cluster towns, you wouldn't need to scale and center their locations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For data that is of different physical measurements or units, its probably a good idea to scale and center. For example, when clustering vehicles, the data may contain attributes such as number of wheels, number of doors, miles per gallon, horsepower etc. In this case it may be a better idea to scale and center since you are unsure of the relationship between each attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition behind that is that since many clustering algorithms require some definition of distance, if you do not scale and center your data, you may give attributes which have larger magnitudes more importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of your problem, I would scale and center the data if it contains attributes like patient height, weight, age etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/6718/37809&quot;&gt;This answer&lt;/a&gt; on a similar question has more.&lt;/p&gt;&#xA;" OwnerUserId="37809" LastActivityDate="2017-08-18T07:17:13.683" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22370" PostTypeId="2" ParentId="19647" CreationDate="2017-08-18T09:21:52.423" Score="2" Body="&lt;p&gt;in case you havent yet found the answer, tensorflows default attention implementation doesn't perform bi-directional encoding hence you dont see the concatenation (whereas in the paper , its clearly mentioned) ..i am guessing we need to include b-directional rnn's explicitly to mimic the paper. For further proof look at line 788 &#xA;&lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the comment below the function definition clearly tells you that &quot;&lt;em&gt;Then it runs an RNN to encode&#xA;  embedded encoder_inputs into a state vector. It keeps the outputs of this&#xA;  RNN at every step to use for attention later. Next, it embeds decoder_inputs...&lt;/em&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hope that helped&lt;/p&gt;&#xA;" OwnerUserId="29157" LastActivityDate="2017-08-18T09:21:52.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22371" PostTypeId="1" AcceptedAnswerId="39749" CreationDate="2017-08-18T09:22:23.403" Score="11" ViewCount="20397" Body="&lt;p&gt;At &lt;a href=&quot;http://www.speech.zone/exercises/dtw-in-python/&quot; rel=&quot;noreferrer&quot;&gt;http://www.speech.zone/exercises/dtw-in-python/&lt;/a&gt; it says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Although it's not really used anymore, Dynamic Time Warping (DTW) is a&#xA;  nice introduction to the key concept of Dynamic Programming.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am using DTW for signal processing and are a little surprised: What is used instead?&lt;/p&gt;&#xA;" OwnerUserId="14909" LastActivityDate="2019-08-22T08:23:52.080" Title="Dynamic Time Warping is outdated?" Tags="|time-series|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22373" PostTypeId="2" ParentId="9175" CreationDate="2017-08-18T10:14:15.947" Score="9" Body="&lt;p&gt;I have just struggled with this same question for a few hours.  Thought I'd share the insite that helped me understand it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is that the filters for the second convolutional layer do not have the same dimensionality as the filters for the first layer.  In general, &lt;strong&gt;the filter has to have the same number of dimensions as its inputs&lt;/strong&gt;.  So in the first conv layer, the input has 2 dimensions (because it is an image).  Thus the filters also have two dimensions.  If there are 20 filters in the first conv layer, then the output of the first conv layer is a stack of 20 2D feature maps. So the output of the first conv layer is  3 dimensional, where the size of the third dimension is equal to the number of filters in the first layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this 3D stack forms the input to the second conv layer.  Since the input to the 2nd layer is 3D, the filters also have to be 3D.  Make the size of the second layer's filters in the third dimension equal to the number of feature maps that were the outputs of the first layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you just convolve over the first 2 dimensions; rows and columns. Thus the convolution of each 2nd layer filter with the stack of feature maps (output of the first layer) yields a single feature map.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The size of the third dimension of the output of the second layer is therefore equal to the number of filters in the second layer.&lt;/p&gt;&#xA;" OwnerUserId="38142" LastActivityDate="2017-08-18T10:14:15.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22374" PostTypeId="2" ParentId="22371" CreationDate="2017-08-18T10:17:02.360" Score="4" Body="&lt;p&gt;Dynamic Time Warping (DTW) has quadratic complexity. There are several other versions of the algorithm, like &lt;a href=&quot;http://cs.fit.edu/~pkc/papers/tdm04.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FastDTW&lt;/a&gt; (linear complexity) that decrease the complexity by computing approximations. FastDTW is implemented, for instance, &lt;a href=&quot;https://pypi.python.org/pypi/fastdtw&quot; rel=&quot;nofollow noreferrer&quot;&gt;in this Python module&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2576" LastActivityDate="2017-08-18T10:17:02.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22375" PostTypeId="2" ParentId="22371" CreationDate="2017-08-18T10:20:29.737" Score="0" Body="&lt;p&gt;As far as I know it's mostly about the computational aspect which was improved, so it's still a proper method to measure similarity between sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend &lt;a href=&quot;https://wiki.epfl.ch/edicpublic/documents/Candidacy%20exam/a12-esling%20time-series%20data%20mining.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; as a good reference, specially section 4.3. Here is the bold part of this section:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A warping path W is&#xA;  a set of contiguous matrix indices defining a mapping between two time series. Even&#xA;  if there is an exponential number of possible warping paths, the optimal path is the&#xA;  one that minimizes the global warping cost. DTW can be computed using dynamic programming&#xA;  with time complexity O(n2) [Ratanamahatana and Keogh 2004a]. However,&#xA;  several lower bounding measures have been introduced to speed up the computation.&#xA;  Keogh and Ratanamahatana [2005] introduced the notion of upper and lower envelope&#xA;  that represents the maximum allowed warping. Using this technique, the complexity&#xA;  becomes O(n). It is also possible to impose a temporal constraint on the size of the DTW&#xA;  warping window. It has been shown that these improve not only the speed but also&#xA;  the level of accuracy as it avoids the pathological matching introduced by extended&#xA;  warping [Ratanamahatana and Keogh 2004b]. The two most frequently used global&#xA;  constraints are the Sakoe-Chiba Band and the Itakura Parallelogram. Salvador and&#xA;  Chan [2007] introduced the FastDTW algorithm which makes a linear time computation&#xA;  of DTW possible by recursively projecting a warp path to a higher resolution and&#xA;  then refining it. A drawback of this algorithm is that it is approximate and therefore&#xA;  ACM Computing Surveys, Vol. 45, No. 1, Article 12, Publication date: November 2012.&#xA;  12:18 P. Esling and C. Agon&#xA;  offers no guarantee to finding the optimal solution. In addition to dynamic warping,&#xA;  it may sometimes be useful to allow a global scaling of time series to achieve meaningful&#xA;  results, a technique known as Uniform Scaling (US). Fu et al. [2008] proposed&#xA;  the Scaled and Warped Matching (SWM) similarity measure that makes it possible to&#xA;  combine the benefits of DTW with those of US.&#xA;  Other shape-based measures have been introduced such as the Spatial Assembling&#xA;  Distance (SpADe) [Chen et al. 2007b]; it is a pattern-based similarity measure. This&#xA;  algorithm identifies matching patterns by allowing shifting and scaling on both temporal&#xA;  and amplitude axes, thus being scale robust. The DISSIM [Frentzos et al. 2007]&#xA;  distance has been introduced to handle similarity at various sampling rates. It is de-&#xA;  fined as an approximation of the integral of the Euclidean distance. One of the most&#xA;  interesting recent proposals is based on the concept of elastic matching of time series&#xA;  [Latecki et al. 2005]. Latecki et al. [2007] presented an Optimal SuBsequence matching&#xA;  (OSB) technique that is able to automatically determine the best subsequence and&#xA;  warping factor for distance computation; it includes a penalty when skipping elements.&#xA;  Optimality is achieved through a high computational cost; however, it can be reduced&#xA;  by limiting the skipping range.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-08-18T10:20:29.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22379" PostTypeId="2" ParentId="9175" CreationDate="2017-08-18T12:01:43.727" Score="1" Body="&lt;p&gt;The first layer consists of $k_1$ kernels with size $3 \cdot 3 \cdot 1$ to give $k_1$ feature maps which are stacked depth-wise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second layer consists of $k_2$ kernels with size $5 \cdot 5 \cdot k_1$ to give $k_2$ feature maps which are stacked depth-wise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, the kernels in a convolutional layer span the depth of the output of the previous layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A layer with $1 \times 1$ convolutional layer actually has $k_n$ kernels of size $1 \cdot 1 \cdot k_{n-1}$.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Speculation:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Bonus question 2 is not something I'm familiar with, but I will guess the depth parameter in the convolution becomes an extra dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. If the output of a layer is size $m \cdot n \cdot k_{n}$, a 3D convolution with padding would result in an output of size $m \cdot n \cdot k_{n+1} \cdot k_{n}$&lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-08-18T12:01:43.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22382" PostTypeId="1" CreationDate="2017-08-18T13:30:33.520" Score="1" ViewCount="172" Body="&lt;p&gt;&lt;strong&gt;Dataset&lt;/strong&gt;&#xA;For an E-Commerce platform, I have a lot of products (Ad_Group_ID), for each product, I have the number of people that arrive on the landing page (Clicks) ; and the number of leads = the number of people that will buy the product (Conversions). Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date,       Ad_group_ID,Clicks,Conversions&#xA;2017-08-17, product1,   10,    1&#xA;2017-08-18, product1,   100,   10&#xA;2017-08-17, product2,   5,     0&#xA;2017-08-18, product2,   50,    5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Goal&lt;/strong&gt;&#xA;I want to predict the number of leads&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mathematical Model&lt;/strong&gt;&#xA;I imagine this is going to follow a binomiale distribution like:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/u6YQc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/u6YQc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ML-Design&lt;/strong&gt;&#xA;&lt;a href=&quot;https://github.com/gobert/data-science-notebooks/blob/master/sem-marketing/leads-prediction.ipynb#L4&quot; rel=&quot;nofollow noreferrer&quot;&gt;I wrote a class BinomialePredictor&lt;/a&gt;, that will delegate to scipy.binom. The results of the evaluation are acceptable IMHO (it has been wayyyy worse with try with Stochastic Gradient Descent and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&#xA;Is there a module of scikit-learn that do this kind of &quot;Binomial regression&quot;? I think it's very strange. Or which kind of modeling would be better fit scikit-learn&lt;/p&gt;&#xA;" OwnerUserId="38154" LastActivityDate="2017-08-18T13:30:33.520" Title="Binomial model in scikit-learn" Tags="|predictive-modeling|scikit-learn|probability|model-selection|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22385" PostTypeId="2" ParentId="12168" CreationDate="2017-08-18T13:49:05.680" Score="1" Body="&lt;p&gt;It depends on how &quot;generalized&quot; you want your methods. If you have known and fixed set of quantitative relationships (=, &amp;lt;=, &gt;=, …), then you can consider it a classification problem. Given some input string, which category does it most likely belong to? The same goes for entities. Are you assuming it will always be two entities? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the system should handle an unlimited number of entities with unknown and possibly infinite number of relationships than the problem quickly becomes intractable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you assume there will always two entities, then you can apply &lt;a href=&quot;https://en.wikipedia.org/wiki/Resource_Description_Framework&quot; rel=&quot;nofollow noreferrer&quot;&gt;Resource Description Framework (RDF)&lt;/a&gt;. RDF goal is to find semantic entities and their relationships. RDF finds triples: a subject, a predicate, and an object. The predicate is &quot;quantitative relationship&quot; between the subject and object. There are a variety of frameworks for extracting RDF from a sentence. The most effective ones depend on building a parse tree of the natural language.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-08-19T23:50:13.840" LastActivityDate="2017-08-19T23:50:13.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22386" PostTypeId="1" CreationDate="2017-08-18T13:54:28.923" Score="4" ViewCount="1424" Body="&lt;p&gt;I have generated a common Sklearn model for text classification which I want to make accessible in the cloud (there is no provider preference) as an API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far the closest solution that I managed to find is &lt;a href=&quot;https://datascience.com.co/creating-an-api-using-scikit-learn-aws-lambda-s3-and-amazon-api-gateway-d9d10317e38d&quot; rel=&quot;noreferrer&quot;&gt;in this tutorial&lt;/a&gt; but it seems quite complex (getting the venv.zip dependencies package at the beginning is unclear, for instance) and specific (in my case NLTK and an external Stanford segmenter is involved for preprocessing and I cannot figure out where to put all these modules and how to invoke them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a decent and robust way to solve this challenge?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-12-25T10:01:30.363" Title="How to Deploy a Scikit-learn Model into the Cloud?" Tags="|scikit-learn|cloud|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22387" PostTypeId="1" AcceptedAnswerId="23578" CreationDate="2017-08-18T14:09:42.870" Score="14" ViewCount="18409" Body="&lt;p&gt;These two convolution operations are very common in deep learning right now. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read about dilated convolutional layer in this paper : &lt;a href=&quot;https://arxiv.org/pdf/1609.03499.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;WAVENET: A GENERATIVE MODEL FOR RAW AUDIO&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and De-convolution is in this paper : &lt;a href=&quot;https://arxiv.org/pdf/1411.4038.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fully Convolutional Networks for Semantic Segmentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both seem to up-sample the image but &lt;strong&gt;what is the difference?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="29169" LastEditDate="2021-02-16T00:43:10.323" LastActivityDate="2021-02-16T00:43:10.323" Title="What is the difference between Dilated Convolution and Deconvolution?" Tags="|machine-learning|deep-learning|computer-vision|convolutional-neural-network|convolution|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22388" PostTypeId="2" ParentId="22387" CreationDate="2017-08-18T14:10:34.873" Score="7" Body="&lt;p&gt;Though both seem to be doing the same thing, which is up-sampling a layer, there's a clear margin between them. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;First we talk about &lt;strong&gt;Dilated Convolution&lt;/strong&gt;&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I found this nice &lt;a href=&quot;http://www.erogol.com/dilated-convolution/&quot; rel=&quot;noreferrer&quot;&gt;blog&lt;/a&gt; on above topic. So as I understood, this is more like exploring the &lt;strong&gt;input data points in a wide manner&lt;/strong&gt;. Or increasing the receptive field of the convolution operation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a dilated convolution diagram from the &lt;a href=&quot;https://arxiv.org/pdf/1511.07122.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OMnBX.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OMnBX.png&quot; alt=&quot;Image 3&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is more of normal convolution, but help to capture more and more global context from input pixels without increasing the size of the parameters. This  can also help to increase the spacial size of the output. But the main thing here is this increases the receptive field size exponentially with the number of layers. This is very common in the signal processing field.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This blog really explains what is new in the dilated convolution and how this is compared to normal convolution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Blog: &lt;a href=&quot;http://www.inference.vc/dilated-convolutions-and-kronecker-factorisation/&quot; rel=&quot;noreferrer&quot;&gt;Dilated Convolutions and Kronecker Factored Convolutions&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Now I will explain what is &lt;strong&gt;Deconvolution&lt;/strong&gt;&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This is called transposed convolution. This is equal to the function we used for convolution in the back-propagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Simply in backprop we distribute gradients from one neuron in the output feature map to all the elements in the receptive fields, then we also sum up gradients for where they coincided with same receptive elements&lt;/em&gt;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a &lt;a href=&quot;https://grzegorzgwardys.wordpress.com/2016/04/22/8/&quot; rel=&quot;noreferrer&quot;&gt;good resource with pictures&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the basic idea is deconvolution works in the output space. Not input pixels. It will try to create more broader spacial dimensions in the output map.  This is used in &lt;strong&gt;Fully Convolutional Neural Nets for Semantic Segmentation&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So more of Deconvolution is a learnable up-sampling layer.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;It's tries to learn how to up-sample while combining with the final loss&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the best explanation I found for deconvolution. Lecture 13 in cs231, &lt;a href=&quot;https://www.youtube.com/watch?v=ByjaPdWXKJ4&quot; rel=&quot;noreferrer&quot;&gt;from 21.21 onward&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="29575" LastEditDate="2017-08-19T08:08:53.593" LastActivityDate="2017-08-19T08:08:53.593" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22395" PostTypeId="1" CreationDate="2017-08-18T20:55:20.470" Score="0" ViewCount="341" Body="&lt;p&gt;I have a corpus of free form text (emails) and am trying to extract the highest degree (eg. High School, Bachelor, Master's, phD) from each of them. Is stemming the way to go? Or lemmatization? Note that the degree may not be mentioned at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some example emails that I'll need to go through: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hi. My name is XXXXX. I have bachelor's in Math from UPenn and a Master's of Education from Stony Brook. Would love to work with XXXX!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Dear XXXX, my name is XXXX and am interested in your company. Please email me for CV.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37983" LastEditorUserId="8501" LastEditDate="2017-08-21T06:06:55.457" LastActivityDate="2017-08-23T10:07:19.763" Title="Extract person's academic degree from text" Tags="|nlp|information-retrieval|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22398" PostTypeId="2" ParentId="22335" CreationDate="2017-08-18T23:03:43.723" Score="4" Body="&lt;p&gt;A black box, as you may know, refers to a function where you know the signature of the inputs and outputs, but can't know how it determines the outputs from the inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of the term is being buzz-worded incorrectly in this case.  It may be beyond the writer/author's willingness or capacity to know and understand ML models, but that does not mean it is beyond the willingness or capacities of others.  The engineers that create each ML model know exactly how it works and can pull up the decision tree at will and walk it.  Just because someone may be too lazy or it may take a while to do so does not mean the information is not readily available for consumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML models are not black boxes, they are clear boxes that are just really big.&lt;/p&gt;&#xA;" OwnerDisplayName="user38177" LastActivityDate="2017-08-18T23:03:43.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22401" PostTypeId="5" CreationDate="2017-08-19T06:35:16.150" Score="0" Body="&lt;p&gt;Apache &lt;strong&gt;Pig&lt;/strong&gt; is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization which enables them to handle very large data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the present time, Pig's infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs for which large-scale parallel implementations already exist (e.g. the Hadoop subproject). Pig's language layer currently consists of a textual language called &lt;a href=&quot;http://pig.apache.org/docs/r0.15.0/basic.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pig Latin&lt;/a&gt; which has the following key properties: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Ease of programming.&lt;/strong&gt; It is trivial to achieve parallel execution of simple, &quot;embarrassingly parallel&quot; data analysis tasks. Complex tasks comprised of multiple interrelated data transformations are explicitly encoded as data flow sequences, easy to write and understand.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Optimization opportunities.&lt;/strong&gt; The declarative way in which tasks are encoded permits the system to optimize their execution plan automatically, allowing the user to focus on semantics rather than efficiency.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Extensibility.&lt;/strong&gt; Users can create their own functions to do special-purpose processing.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Official Website:&lt;/strong&gt; &lt;a href=&quot;https://pig.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pig.apache.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Useful Links:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cwiki.apache.org/confluence/display/PIG/Index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wiki&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pig.apache.org/docs/r0.15.0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Documentation (0.15.0)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://pig.apache.org/docs/r0.15.0/basic.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pig Latin Basics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-08-19T12:00:44.407" LastActivityDate="2017-08-19T12:00:44.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22402" PostTypeId="4" CreationDate="2017-08-19T06:35:16.150" Score="0" Body="Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization which enables them to handle very large data sets." OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-08-19T12:00:35.430" LastActivityDate="2017-08-19T12:00:35.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22405" PostTypeId="1" CreationDate="2017-08-19T10:48:46.027" Score="2" ViewCount="1609" Body="&lt;p&gt;I'm working with a shell script(#!/bin/sh) and I wanted to know if there is a way to call variables with their values from an Rscript that I have called in my Shell script.&#xA;If that doesn't make sense I want to create, for example a data frame &#xA;&lt;code&gt;data=data.frame(a=seq(1,5), b=seq(1,5))&lt;/code&gt; in a script called &lt;strong&gt;test.r&lt;/strong&gt;&#xA;and then call that variable, with it's content in my shell script, i.e to print it with an echo: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;echo $data&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="38188" LastEditorUserId="471" LastEditDate="2017-08-22T17:23:06.253" LastActivityDate="2018-01-19T19:20:21.197" Title="Passing variables and values from an R script to a shell script" Tags="|r|parsing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22406" PostTypeId="1" CreationDate="2017-08-19T12:21:47.793" Score="1" ViewCount="529" Body="&lt;p&gt;I am performing &lt;em&gt;regression&lt;/em&gt; task using *random forest**.In my prediction set I am having *new levels for categorical variable** which are not present in training data .Currently I am performing one hot encoding to handle this.At present I am not getting considerable results. &lt;a href=&quot;https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This article is saying that it is not good to perform one hot encoding&lt;/a&gt;.How to handle this case?&lt;/p&gt;&#xA;" OwnerUserId="37604" LastActivityDate="2017-08-19T12:21:47.793" Title="New values for categorical variable in Prediction dataset" Tags="|machine-learning|statistics|regression|random-forest|categorical-data|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22407" PostTypeId="1" AcceptedAnswerId="22413" CreationDate="2017-08-19T13:47:09.923" Score="1" ViewCount="813" Body="&lt;p&gt;My question deals with the algorithm described in the paper:&#xA;&lt;a href=&quot;https://www.cis.upenn.edu/~mkearns/papers/rlexec.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning for Optimized Trade Execution&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper uses &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;reinforcement learning technique&lt;/a&gt; to deal with the problem of optimized trade execution. They divide the data into episodes, and then apply (on page 4 in the &lt;a href=&quot;https://www.cis.upenn.edu/~mkearns/papers/rlexec.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;) the following update rule (to the cost function) and algorithm to find an optimal policy: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;code&gt;T&lt;/code&gt; is the total time units, &lt;code&gt;I&lt;/code&gt; is the volume, &lt;code&gt;L&lt;/code&gt; is the possible number of actions, x represents the state, and &lt;code&gt;c&lt;/code&gt; represents the cost function, and &lt;code&gt;c_im&lt;/code&gt; is the immediate reward at a certain state and a certain action. &lt;code&gt;n&lt;/code&gt; is the number of times that the state-action pair were visited)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ijuhb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ijuhb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand correctly, the algorithm is basically a dynamic programming, when we move backwards in time. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Why do we need &lt;code&gt;n&lt;/code&gt; in the cost function update rule. Aren't we visiting each state exactly once?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes). In such case, how do we combine the results from all the episodes? That is, each episode provides an optimal policy. How do we combine all these policies to one final policy?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38216" LastEditorUserId="29575" LastEditDate="2017-08-19T16:25:18.647" LastActivityDate="2017-08-19T23:01:41.170" Title="Reinforcement Learning algorithm for Optimized Trade Execution" Tags="|machine-learning|algorithms|optimization|reinforcement-learning|finance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22408" PostTypeId="1" CreationDate="2017-08-19T15:36:55.523" Score="5" ViewCount="4265" Body="&lt;p&gt;I don't understand the difference between the parameter $C$ and $\lambda$ in terms of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;noreferrer&quot;&gt;SVM&lt;/a&gt;. It seems to me that they are both involved in regulating over-fitting of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What difference between $C$ and $\lambda$?&lt;/p&gt;&#xA;" OwnerUserId="38202" LastEditorUserId="29575" LastEditDate="2017-08-19T15:43:24.493" LastActivityDate="2020-04-30T09:00:23.957" Title="What is the difference between C and lambda in the context of an SVM?" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22409" PostTypeId="1" CreationDate="2017-08-19T16:22:48.267" Score="0" ViewCount="57" Body="&lt;p&gt;I have a data set of recruiting pipeline information that has 12 columns of categorical variables (they range from binary variables like gender to non-binary variables like the name of the school the applicant attended). The last column (13) contains a &quot;Yes&quot; or &quot;No&quot; value that tells if the person is still in the recruiting process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea here is to build a model that can predict (reasonably well) the likelihood of the person withdrawing from the recruiting process at any given stage (there are many stages and this field is one of the 12 independent variables captured).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of using &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;logistic regression&lt;/a&gt; to create the model but all the predictor variables are categorical, which I hear, doesn't bode well for logistic regression. Another factor is that some fields have missing values (like gender) that I cannot reasonably input/capture accurate data for atm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given my situation, what approach do you think would be a good starting point? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was thinking of &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest&lt;/a&gt; but not sure if there's a better way to go about tackling this problem.&lt;/p&gt;&#xA;" OwnerUserId="38204" LastEditorUserId="29575" LastEditDate="2017-08-19T18:57:12.697" LastActivityDate="2018-06-27T18:11:45.273" Title="Dropout classifcation" Tags="|classification|random-forest|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22410" PostTypeId="2" ParentId="22408" CreationDate="2017-08-19T18:55:15.993" Score="3" Body="&lt;p&gt;You are right to be confused. What's going on is that the hyperparameters refer to different formulations. On the one hand there is a generic empirical risk minimizer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda\lVert \mathbf w \rVert^2 + \frac 1 n \sum_{i=1}^n \max\left(0, 1 - y_i(\mathbf x_i^T \mathbf w + w_0)\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other there is the soft-margin objective&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{2} \lVert \mathbf w \rVert^2 + C \sum_{i=1}^n \xi_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;such that $$\xi_i \geq 0, y_i\left(\mathbf x_i^T \mathbf w + w_0\right) \geq 1-\xi_i, \forall i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one places SVM in the framework of empirical risk minimization, in which the objective is an expected loss plus the $L_p$-norm of the main parameter. The purpose of introducing &lt;em&gt;slack&lt;/em&gt; variables and constraints in the second formulation is to allow a fraction of points (adjustable by $C$) to lie on the wrong side of the margin, and eliminate the non-differentiable $\max$ function. Moreover, it draws attention to the fact that SVMs encode sparsity in the loss function, rather than the prior. This is called the &lt;em&gt;primal&lt;/em&gt; form and can be solved using quadratic programming. Wikipedia currently explains this in the &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=Support_vector_machine&amp;amp;oldid=796006866#Computing_the_SVM_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Computing the classifier&lt;/a&gt; section. You should be able to see that $C=1/{2n\lambda}$. Some texts omit the $1/2$ or the $n$.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-08-19T19:32:33.470" LastActivityDate="2017-08-19T19:32:33.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22412" PostTypeId="2" ParentId="22409" CreationDate="2017-08-19T21:43:25.550" Score="2" Body="&lt;p&gt;There isn't a particular approach or model that is definitively the best. You might want to try multiple types of models. A good starting point would be to visualize your data. If there are many non-linear relationships between your data and the the outputs you are predicting then you might want to start with a tree based model. It will be much less work to make it perform well. If the relationship between your variables and the outcomes you are predicting are fairly linear then a logistic regression model is a good approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to have a silver bullet model to solve all your problems then I recommend trying xgboost. Use grid search and you will have a good model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually worked on a similar project for my university. Aggressive feature engineering combined with logistic regression performed and xgboost were the two best models.&lt;/p&gt;&#xA;" OwnerUserId="33980" LastActivityDate="2017-08-19T21:43:25.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22413" PostTypeId="2" ParentId="22407" CreationDate="2017-08-19T21:47:32.057" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why do we need n in the cost function update rule. Aren't we visiting each state exactly once?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The update is assuming a static distribution and estimating the average value. As each estimate is made available, it is weighted less of the total each time. The formula means that the first sample is weighted $1$, second $\frac{1}{2}$, third $\frac{1}{3}$ which is what you need to get the mean value when you apply the changes due to the samples serially whilst maintaining the best estimate of the mean at each step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a little odd in my experience of RL, because it assumes the bootstrap values (the max over next step) come from a final distribution to weight everything equally like this. But I think it is OK due to working back from final step, hence each bootstrap value should be fully estimated before going backwards to previous time step.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This looks like an algorithm that you run on the whole data set, where each episode is the same length $T$. So you run each timestep (starting with the end time step and working backwards since the ultimate reward is established at the end of the episode, so this is more efficient), and sample from &lt;em&gt;every&lt;/em&gt; episode at that timestep in the &lt;code&gt;While (not end of data)&lt;/code&gt; loop. The values are therefore combined inside the loop at that stage, and there is no need to add anything to the algorithm to combine episodes.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-19T23:01:41.170" LastActivityDate="2017-08-19T23:01:41.170" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22414" PostTypeId="1" AcceptedAnswerId="22415" CreationDate="2017-08-19T21:54:30.713" Score="2" ViewCount="458" Body="&lt;p&gt;I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know of proprietary solutions such as Oracle extensions, but are there open source/free systems for scientific data? I would define that as a system that would have the MKSA unit system integrated, and an extensive library of scientific conversions/operations.&lt;/p&gt;&#xA;" OwnerUserId="38213" LastActivityDate="2017-08-23T12:29:55.697" Title="Are there databases specializing in scientific data" Tags="|databases|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22415" PostTypeId="2" ParentId="22414" CreationDate="2017-08-19T23:48:34.577" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://database.cs.brown.edu/projects/scidb/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SciDB&lt;/a&gt; is an open-source DBMS for scientific data.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-08-23T12:29:55.697" LastActivityDate="2017-08-23T12:29:55.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22416" PostTypeId="1" CreationDate="2017-08-20T03:52:51.523" Score="3" ViewCount="3360" Body="&lt;p&gt;I'm trying to use a custom kernel that accepts 3 arguments, with the SVM in sk-learn:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def k_gaussian(_x1, _x2, _sigma):&#xA;    normsq = np.square(np.linalg.norm(_x1-_x2))&#xA;    return np.exp(- normsq/(2 * np.square(sigma)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to the &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#custom-kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;, a custom kernel must only have two arguments, which the &lt;code&gt;svm.SVC&lt;/code&gt; class will handle automatically with the given input data. We are told to pass the custom kernel in a form like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = svm.SVC(kernel=my_kernel)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I'm working on an assignment which requires us to run experiments on SVM performance with varying values of &lt;code&gt;_sigma&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achieve that in this case? Can I pass in something like?:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = svm.SVC(kernel=k_gaussian(_sigma=2)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would things like decorators help me here?&lt;/p&gt;&#xA;" OwnerUserId="33716" LastEditorUserId="29575" LastEditDate="2017-08-20T04:59:43.773" LastActivityDate="2020-05-05T17:17:14.903" Title="Passing a custom kernel with more than two arguments into `svm.SVC` in scikit-learn" Tags="|python|scikit-learn|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22417" PostTypeId="2" ParentId="22416" CreationDate="2017-08-20T04:22:32.030" Score="7" Body="&lt;p&gt;That can be done with a closure like:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def build_k_gaussian(sigma):&#xA;&#xA;    def k_gaussian(_x1, _x2):&#xA;        diff = _x1[:, np.newaxis] - _x2&#xA;        normsq = np.square(np.linalg.norm(diff, axis = 2))&#xA;        return np.exp(- normsq / (2 * np.square(sigma)))&#xA;&#xA;    return k_gaussian&#xA;&#xA;clf = svm.SVC(kernel=build_k_gaussian(sigma=2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;How does this work?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The function &lt;code&gt;k_gaussian&lt;/code&gt; is defined when &lt;code&gt;build_k_gaussian()&lt;/code&gt; is called.  &lt;code&gt;k_gaussian&lt;/code&gt; will be able to access the value of &lt;code&gt;sigma&lt;/code&gt; from when the function was created.  This is known as a closure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in the end, &lt;code&gt;build_k_gaussian&lt;/code&gt; returns a function that takes two parameters, which is what the &lt;code&gt;kernel&lt;/code&gt; parameter required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the &lt;a href=&quot;https://scikit-learn.org/stable/modules/svm.html#using-python-functions-as-kernels&quot; rel=&quot;nofollow noreferrer&quot;&gt;Using Python functions as kernels on scikit-learn&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Your kernel must take as arguments two matrices of shape (n_samples_1, n_features), (n_samples_2, n_features) and &lt;strong&gt;return a kernel matrix of shape (n_samples_1, n_samples_2).&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So you need to apply the kernel function on all pairs of samples, therefore &#xA;&lt;code&gt;diff&lt;/code&gt; broadcasts the &lt;code&gt;_x1&lt;/code&gt; matrix and subtracts all samples in &lt;code&gt;_x2&lt;/code&gt; from all samples in &lt;code&gt;_x1&lt;/code&gt;. You have to calculate the norm along &lt;code&gt;axis=2&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="96467" LastEditDate="2020-05-05T17:17:14.903" LastActivityDate="2020-05-05T17:17:14.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22418" PostTypeId="2" ParentId="22335" CreationDate="2017-08-20T04:52:20.523" Score="3" Body="&lt;p&gt;In the blog posting cited in the question, the discussion is about the fact that the experts who develop machine learning models in finance can't explain to their customers (financiers with no training in machine learning) how the model makes the decisions that it does.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This brings out a distinction between models that are black boxes because of information that is truly secret (e.g. the coefficients are encoded in a tamper proof FPGA) and models that are open (in the sense that the coefficients are known) but not comprehensible &lt;em&gt;to a particular audience&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This latter kind of &quot;black box&quot; is problematic because customers want to reassure themselves that the model you've constructed has &quot;face validity.&quot;  With other types of models such as Logistic Regression, it's relatively easy to look at the coefficients and check that they have the expected plus or minus signs- even a mathematically illiterate MBA can understand that.     &lt;/p&gt;&#xA;" OwnerUserId="8754" LastActivityDate="2017-08-20T04:52:20.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22421" PostTypeId="1" CreationDate="2017-08-20T11:16:17.610" Score="3" ViewCount="908" Body="&lt;p&gt;Is there an algorithm or NN to match two documents? One is a claim description (e.g. a CV or product offer) and another is a requirements description (e.g. vacancy description or RFP). They are not similar, so basically it's not a docs similarity per se.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's it better embedding to use on document corps (Doc2vec, Word2vec or just TF-IDF? etc) and what kind of further NN architecture would work to basically find a matching scores vector/matrix as output on how do input claim docs match to requirement docs?&#xA;Or is there exists just any text analitics algorithm or something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for help.&lt;/p&gt;&#xA;" OwnerUserId="38078" LastEditorUserId="38078" LastEditDate="2017-08-27T16:35:43.803" LastActivityDate="2022-12-16T04:02:55.413" Title="Is there an algorithm or NN to match two documents, basically not closely similar?" Tags="|machine-learning|neural-network|deep-learning|text-mining|similarity|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22423" PostTypeId="2" ParentId="19801" CreationDate="2017-08-20T16:51:34.723" Score="3" Body="&lt;p&gt;You can try using &lt;strong&gt;data re-sampling&lt;/strong&gt; techniques. They can be divided in four categories: undersampling the majority class, oversampling the minority class, combining over and under sampling, and creating an ensemble of balanced datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above methods and more are implemented in the &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn&quot; rel=&quot;nofollow noreferrer&quot;&gt;imbalanced-learn&lt;/a&gt; library in Python that interfaces with scikit-learn. See &lt;a href=&quot;https://github.com/vsmolyakov/experiments_with_python/blob/master/chp01/imbalanced_data.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;ipython notebook&lt;/a&gt; for an example.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-20T16:51:34.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22424" PostTypeId="2" ParentId="19760" CreationDate="2017-08-20T17:10:54.690" Score="1" Body="&lt;p&gt;In designing an MLP architecture, we can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume &lt;a href=&quot;http://cs231n.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;recommended&lt;/a&gt; ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers (see &lt;a href=&quot;https://github.com/vsmolyakov/experiments_with_python/blob/master/chp03/tensorflow_optimizers.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;ipython notebook&lt;/a&gt; for comparison). Your loss function will depend on the problem: cross-entropy for classification and MSE for regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid local optima in parameter search, it's better to use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;random search&lt;/a&gt; or &lt;a href=&quot;https://github.com/fmfn/BayesianOptimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian optimization&lt;/a&gt;. For additional answers, see this &lt;a href=&quot;https://datascience.stackexchange.com/questions/22199/what-are-the-default-values-of-nodes-and-internal-layers-in-neural-network-model/22289&quot;&gt;post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-20T17:10:54.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22425" PostTypeId="2" ParentId="19654" CreationDate="2017-08-20T17:20:11.257" Score="0" Body="&lt;p&gt;To understand exactly how &lt;a href=&quot;https://arxiv.org/pdf/1301.3781.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; model is defined and trained it may be helpful to look at code based on this &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/examples/tutorials/word2vec/word2vec_basic.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/examples/tutorials/word2vec/word2vec_basic.py&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-20T17:20:11.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22426" PostTypeId="2" ParentId="19521" CreationDate="2017-08-20T17:52:12.287" Score="2" Body="&lt;p&gt;&quot;Source credibility&quot; of Internet articles is best calculated through the &lt;a href=&quot;https://en.wikipedia.org/wiki/PageRank&quot; rel=&quot;nofollow noreferrer&quot;&gt;Page Rank&lt;/a&gt; algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If an article is a hub then it is the authority on the topic and can be assumed well written (or at least very useful).&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-20T17:52:12.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22427" PostTypeId="2" ParentId="16603" CreationDate="2017-08-20T18:26:41.623" Score="2" Body="&lt;p&gt;NLP stands for Natural Language Processing. Programming language source code are synthetic (or unnatural) languages. Thus, NLP tools are not useful for processing programming language source code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Understanding programming language source code is done by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Compiler&quot; rel=&quot;nofollow noreferrer&quot;&gt;compiler&lt;/a&gt; or interpreter. Compilers and interpreters perform many functions, including lexical analysis, parsing, and semantic analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Python language, language analysis is relatively straightforward since the main Python implementation, named CPython, compiles Python programs into intermediate bytecode, which is executed by the virtual machine. The &lt;a href=&quot;https://docs.python.org/3/library/dis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;dis&lt;/a&gt; module supports the analysis of CPython bytecode by disassembling it.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-02-22T15:34:12.290" LastActivityDate="2018-02-22T15:34:12.290" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22428" PostTypeId="1" CreationDate="2017-08-20T18:56:01.743" Score="2" ViewCount="89" Body="&lt;p&gt;My dataset contains half a million unlabeled entries with over 100 binary features. A third of these features are present in less than 1000 samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to classify a few samples by hand (into one of two classes) and use them as trainings set for either an SVM or a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I select samples for classifying them by hand? Are there some preferred methods? I fear that by selecting them randomly not all possible features are present in my training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to select the training set and the dataset used for later evaluation in the same way? For example could I just use 10 examples of each feature usage for training and random samples for evaluation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a rule of thumb about how large my training set has to be?&lt;/p&gt;&#xA;" OwnerUserId="38231" LastEditorUserId="29575" LastEditDate="2017-08-20T19:43:24.940" LastActivityDate="2018-05-01T13:37:26.937" Title="How to select samples for a trainings set" Tags="|machine-learning|neural-network|svm|training|" AnswerCount="0" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22430" PostTypeId="1" CreationDate="2017-08-20T19:35:34.767" Score="3" ViewCount="101" Body="&lt;p&gt;I have a bunch of data related to my company's incoming and outgoing calls, and I'm trying to detect spam callers in order to blacklist them. The features I've extracted from the data are a bit simple right now:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Caller Number: Who is calling&lt;/li&gt;&#xA;&lt;li&gt;N_Unique: The amount of different phones called&lt;/li&gt;&#xA;&lt;li&gt;N_Answered: The amount of times the call was connected&lt;/li&gt;&#xA;&lt;li&gt;N_Missed: The amount of times the call was ignored/missed&lt;/li&gt;&#xA;&lt;li&gt;Avg_Duration: The average duration answered calls lasted&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Initially my superior sent me the dataset with a field called &quot;IsSpammer&quot;, which was binary, and after a very simple Random Forest (1000 trees, default scikit-learn everything else) I got a 100% accuracy even after cross-validation. I took a closer look at the data and realized anyone who called more than 20 different numbers inside the company or had average call duration less than 20 seconds where flagged as spammers (which may be true), but then I asked supervisor how this field was generated, and he told me he set it under that rule the random forest discovered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then realized every ML problem I've faced came from a Kaggle contest that gave me a clear field to predict, and a training set that had &quot;correct&quot; information on the fields to predict. I don't know how to proceed with ML if I don't have the information of Spammer/NotSpammer, since apparently if I come up with a clever (or not) rule, then the best I can hope to discover is that rule I'm using in the first place... I'm not sure if getting more features and making and obscure rule using all of them would help. Obviously I lack experience in this, so I figured I would ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a problem that could benefit from using machine learning? &lt;/p&gt;&#xA;" OwnerUserId="38234" LastEditorUserId="13748" LastEditDate="2017-09-20T17:43:47.170" LastActivityDate="2017-11-19T18:58:00.580" Title="Detecting spammers with artificially generated target class" Tags="|machine-learning|dataset|data-cleaning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22433" PostTypeId="2" ParentId="12253" CreationDate="2017-08-20T22:01:46.523" Score="2" Body="&lt;p&gt;&lt;strong&gt;&lt;em&gt;recall&lt;/em&gt;&lt;/strong&gt; is called 'recall' because it's the fraction of relevant (training-set) instances which were 'recalled' (yes or 'retrieved' as you suggest. 'coverage' would be more ambiguous, it could be misinterpreted as other things, e.g. the % of the training-set that you trained on (e.g. a partition. Your suggested 'coverage' would make sense wrt training-set but not the test-set, hence it's way too ambiguous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always assumed the reason 'recall' was also known as &lt;em&gt;'sensitivity'&lt;/em&gt; was due to signal-processing or medical influences: think a mine-detector, or a radar, or a test for a disease: 'sensitive' would mean it picks up most/all of the known relevant exemplars (from training-set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, most of these terms' arcane names are seriously unintuitive and are a patchwork quilt drawn from a vocabulary from disparate fields over many decades, but they're set in stone now so you just have to find a mnemonic to learn them and not get too hung up on them and get on with things...&lt;/p&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="8501" LastEditDate="2017-08-22T11:42:00.153" LastActivityDate="2017-08-22T11:42:00.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22434" PostTypeId="1" CreationDate="2017-08-21T00:25:50.023" Score="1" ViewCount="156" Body="&lt;p&gt;Very conceptual question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a TensorFlow model that works well. I have isolated about 70 features worth of data and when training, my validation accuracy stays around 32% while my training accuracy is up to 89%! However, when I use only 24 of those 70 features, I get a validation accuracy of 38% and a training accuracy of 53%. I am using regularization, heavy dropout, and early stopping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a good problem to have? Should I keep the 70 features while boosting overfitting parameters? What should I do about this? Maybe it's a good idea to keep using only the 24 best features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it generally O.K. to shed some features by just discarding them? Definitely some emotional attachment to my features here, but is it common for an ML process to simply discard these features as opposed to working with them to achieve higher accuracy?&lt;/p&gt;&#xA;" OwnerUserId="37109" LastEditorUserId="13727" LastEditDate="2017-08-21T05:56:23.017" LastActivityDate="2020-07-13T04:11:41.107" Title="Is it a good thing if my model overfits in TensorFlow?" Tags="|machine-learning|python|tensorflow|overfitting|" AnswerCount="1" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="22437" PostTypeId="2" ParentId="22395" CreationDate="2017-08-21T08:10:32.030" Score="0" Body="&lt;p&gt;I would hand label a few hundred cases just to build a test dataset to check how well your method works. I think this is a clear case where building some rules based system will work better, build a set of rules that just look for specific keywords, match all the words to your rules and pick the maximum degree that you could find. After running your new method through your test set, you can see where the model made mistakes and because you made the rules yourself, you can see why it made a mistake and how you could fix it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are set on a more automated approach, you could look into rules mining or character based classification, but you would likely need to label much more data.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-21T08:10:32.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22438" PostTypeId="1" CreationDate="2017-08-21T08:43:43.737" Score="4" ViewCount="3865" Body="&lt;p&gt;I have a series of timestamps that represent the time a user clicked a certain button.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to detect the automated clicks, so I need to find recurring patterns in the data that may point to an automated script. The majority of the data is by regular users. I don't need to detect them online, I just need to detect them in historical data. I also have user_id to make a distinction between users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a default approach to detecting recurring patterns in a largely unstructured dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most things I've found are about anomaly detection on a certain value paired with a timestamp but my problem is not really anomaly-detection as I need to find patterns amongst unpatterned data&lt;/p&gt;&#xA;" OwnerUserId="38136" LastEditorUserId="38136" LastEditDate="2017-08-21T09:25:19.687" LastActivityDate="2017-08-21T09:25:19.687" Title="How to find patterns in a series of timestamps" Tags="|python|time-series|unsupervised-learning|anomaly-detection|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22439" PostTypeId="1" CreationDate="2017-08-21T08:49:57.503" Score="1" ViewCount="353" Body="&lt;p&gt;To train a Deep neural network, why training Restricted Boltzmann Machine(RBM) layer-wise first and then apply BP algorithm through the network? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Comparing to random initiating all the weights+bias in all layers and then apply BP algorithm, is there a quantitative reason(proof) that pretrained RBM is better than random weights assignment? I found a lot of case(datasets) studies about these two training methods saying RBM is better than random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Some documents argued that RBM also could help avoiding local minimizers when running gradient descent, but didn't offer any proof)&lt;/p&gt;&#xA;" OwnerUserId="38257" LastActivityDate="2017-08-21T08:49:57.503" Title="Why Restricted Boltzmann Machine(RBM) is better than random initiating?" Tags="|machine-learning|deep-learning|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22440" PostTypeId="2" ParentId="22438" CreationDate="2017-08-21T09:23:37.230" Score="2" Body="&lt;p&gt;The question can be super difficult if you only have the number of clicks per time-stamp. The reason is that you might find many different recurrent patterns!&#xA;For instance certain time periods in year, month or day (according to the functionality of website) may attract a certain temporal pattern of clicks.&#xA;But if you have more relevant features the story is different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After all I will point out some directions and hope it helps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0013469479900750&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autocorrelation&lt;/a&gt; may initially give you a good insight about what is happening there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then go for &lt;a href=&quot;https://wiki.epfl.ch/edicpublic/documents/Candidacy%20exam/a12-esling%20time-series%20data%20mining.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time-Warping methods&lt;/a&gt; to define similarity function and seek it along time-series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Embedding in high-dimensional spaces disclose the temporal recurrent pattern of a time-series. It's mostly from physics literature and is called State-Space Recunstruction. This method transfers your time-series into a m-dimensional space using &lt;a href=&quot;https://journals.aps.org/pra/abstract/10.1103/PhysRevA.45.3403&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and there you can see the time-series as a dynamic system which may help in detecting recurrent dynamics.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-08-21T09:23:37.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22441" PostTypeId="1" CreationDate="2017-08-21T09:25:35.627" Score="6" ViewCount="7125" Body="&lt;p&gt;I have a traditional prediction setting, with a training data set &lt;code&gt;train&lt;/code&gt; and a test data set &lt;code&gt;test&lt;/code&gt;.&#xA;I do not know the outcome &lt;code&gt;y&lt;/code&gt; of the &lt;code&gt;test&lt;/code&gt; set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that tsne separates my binary classification setting quite well.&#xA;However, tsne cannot really be used for prediction, as in &lt;code&gt;predict(tsne, newdata=test)&lt;/code&gt; which can be done for PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best approach here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I combine my &lt;code&gt;train&lt;/code&gt; and &lt;code&gt;test&lt;/code&gt; set (i.e., &lt;code&gt;rbind&lt;/code&gt;) and run tsne on the whole data set?&lt;/p&gt;&#xA;" OwnerUserId="15664" LastActivityDate="2017-12-23T16:05:05.717" Title="tsne for prediction" Tags="|predictive-modeling|prediction|pca|tsne|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22442" PostTypeId="1" CreationDate="2017-08-21T10:32:15.770" Score="2" ViewCount="35286" Body="&lt;p&gt;&lt;strong&gt;Dictonary&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{(8758148.0, 'CI Alpine Growth Equity Fund'): D    0.000016&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI American Growth Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI American Small Companies Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI American Value Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Canadian Equity Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Canadian Investment Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Canadian Small Cap Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Canadian Small/Mid Cap Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Global Bond RSP Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64,&#xA; (8758148.0, 'CI Harbour Fund'): D    0.0&#xA; Name: 8758148.0, dtype: float64 }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need the dataframe to be: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id          Name                                Value &#xA;8758148.0   CI Alpine Growth Equity Fund        0.000016&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am stuck in issue where I need to convert list into such a data frame with certain name of the columns &lt;/p&gt;&#xA;" OwnerUserId="33137" LastEditorUserId="20492" LastEditDate="2017-08-21T21:40:53.023" LastActivityDate="2017-08-21T21:40:53.023" Title="Convert List to DataFrame" Tags="|python|pandas|dataframe|" AnswerCount="3" CommentCount="3" ClosedDate="2017-09-02T09:29:43.263" ContentLicense="CC BY-SA 3.0" />
  <row Id="22443" PostTypeId="2" ParentId="22441" CreationDate="2017-08-21T12:03:45.420" Score="1" Body="&lt;p&gt;You could try performing t-SNE on the combined test and train data, then assigning the class of each test point based on $k$-NN with the training data in t-SNE coordinates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't believe there are any guarantees with t-SNE, so you might need to find another method. My question on the possibility of using $k$-NN with t-SNE is &lt;a href=&quot;https://datascience.stackexchange.com/questions/22299/does-nearest-neighbour-make-any-sense-with-t-sne&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-08-21T12:03:45.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22444" PostTypeId="2" ParentId="22442" CreationDate="2017-08-21T12:11:33.663" Score="1" Body="&lt;p&gt;Let's see. I modified the string a bit, so that it can be saves as a dictionary.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = {(8758148.0, 'CI Alpine Growth Equity Fund'): 'D    0.000016 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI American Growth Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI American Small Companies Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI American Value Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Canadian Equity Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Canadian Investment Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Canadian Small Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Canadian Small/Mid Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Global Bond RSP Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',&#xA; (8758148.0, 'CI Harbour Fund'): 'D    0.0 Name: 8758148.0, dtype: float64' }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame([i[0] for i in g.keys()], columns=['id'])&#xA;df['Name'] = [i[1] for i in g.keys()]&#xA;df['Value'] = [i.split()[1] for i in g.values()]&#xA;df&#xA;&#xA;    id          Name                            Value&#xA;0   8758148.0   CI Alpine Growth Equity Fund    0.000016&#xA;1   8758148.0   CI American Growth Fund 0.0&#xA;2   8758148.0   CI American Small Companies Fund    0.0&#xA;3   8758148.0   CI American Value Fund  0.0&#xA;4   8758148.0   CI Canadian Equity Fund 0.0&#xA;5   8758148.0   CI Canadian Investment Fund 0.0&#xA;6   8758148.0   CI Canadian Small Cap Fund  0.0&#xA;7   8758148.0   CI Canadian Small/Mid Cap Fund  0.0&#xA;8   8758148.0   CI Global Bond RSP Fund 0.0&#xA;9   8758148.0   CI Harbour Fund 0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2017-08-21T12:11:33.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22445" PostTypeId="2" ParentId="8657" CreationDate="2017-08-21T12:17:55.640" Score="1" Body="&lt;p&gt;Below is how you can implement gradient descent in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sigmoid(z):&#xA;    s= 1/(1 + np.exp(-z))&#xA;    return s&#xA;&#xA;def propagate(w, b, X, Y):&#xA;&#xA;    m = X.shape[1]&#xA;&#xA;    A = sigmoid(np.dot(w.T,X)+b)                                     # compute activation&#xA;&#xA;    cost = -1/m * np.sum(Y * np.log(A) + (1-Y) * (np.log(1-A)))&#xA;&#xA;    dz= (1/m)*(A - Y)&#xA;    dw = np.dot(X, dz.T)&#xA;    db = np.sum(dz)&#xA;&#xA;&#xA;    cost = np.squeeze(cost)&#xA;    grads = {&quot;dw&quot;: dw,&#xA;             &quot;db&quot;: db}&#xA;&#xA;    return grads, cost&#xA;&#xA;&#xA;def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):&#xA;&#xA;    costs = []&#xA;&#xA;    for i in range(num_iterations):&#xA;        m = X.shape[1]&#xA;        grads,cost = propagate(w, b, X, Y)&#xA;        b = b - learning_rate*grads[&quot;db&quot;]&#xA;        w = w - learning_rate*grads[&quot;dw&quot;]&#xA;        if i % 100 == 0:&#xA;            costs.append(cost)&#xA;        if print_cost and i % 100 == 0:&#xA;            print (&quot;Cost after iteration %i: %f&quot; %(i, cost))&#xA;&#xA;    params = {&quot;w&quot;: w,&#xA;              &quot;b&quot;: b}&#xA;    return params, grads, costs&#xA;&#xA;&#xA;&#xA;def predict(w, b, X):&#xA;    m = X.shape[1]&#xA;    Y_prediction = np.zeros((1,m))&#xA;    w = w.reshape(X.shape[0], 1)&#xA;    A = sigmoid(np.dot(w.T,X)+ b)&#xA;&#xA;    for i in range(A.shape[1]):&#xA;        x_exp = np.exp(A)&#xA;        x_sum = np.sum(x_exp,axis=1,keepdims=True)&#xA;        s = np.divide(x_exp,x_sum)&#xA;&#xA;    Y_prediction = 1. * (A &amp;gt; 0.5)&#xA;&#xA;    return Y_prediction&#xA;&#xA;&#xA;&#xA;def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False):&#xA;    w, b = initialize_with_zeros(X_train.shape[0])&#xA;&#xA;    print(&quot;learning rate:&quot;,learning_rate)&#xA;&#xA;    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost = False)&#xA;&#xA;    w = parameters[&quot;w&quot;]&#xA;    b = parameters[&quot;b&quot;]&#xA;&#xA;    Y_prediction_train = predict(w,b,X_train)&#xA;    Y_prediction_test = predict(w,b,X_test)&#xA;    d = {&quot;costs&quot;: costs,&#xA;         &quot;Y_prediction_test&quot;: Y_prediction_test, &#xA;         &quot;Y_prediction_train&quot; : Y_prediction_train, &#xA;         &quot;w&quot; : w, &#xA;         &quot;b&quot; : b,&#xA;         &quot;learning_rate&quot; : learning_rate,&#xA;         &quot;num_iterations&quot;: num_iterations}&#xA;&#xA;    return d&#xA;&#xA;d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38261" LastEditorUserId="23305" LastEditDate="2017-08-21T15:16:23.293" LastActivityDate="2017-08-21T15:16:23.293" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22447" PostTypeId="2" ParentId="22430" CreationDate="2017-08-21T12:28:07.627" Score="0" Body="&lt;p&gt;You could use unsupervised learning to cluster together the types of callers. From there, maybe you'll see a group form of people that call more than 10 internal numbers (or something like that) and then can separately check to see whether those seem to be spammers.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-21T12:28:07.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22448" PostTypeId="2" ParentId="22442" CreationDate="2017-08-21T12:30:12.427" Score="0" Body="&lt;p&gt;I'm just going to make up a simple example which you can modify to your own example. This is a generic approach that can be used for any similar operations. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;d = {('a', 'b'): 1, ('c', 'd'): 2}&#xA;x=pd.DataFrame(d.items(),columns=['keys','value'])&#xA;&#xA;def extract(x):&#xA;  (key1,key2)=x['keys']&#xA;  x['key_1']=key1&#xA;  x['key_2']=key2&#xA;  del x['keys']&#xA;  return x&#xA;&#xA;x.apply(lambda x: extract(x),axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8819" LastActivityDate="2017-08-21T12:30:12.427" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22450" PostTypeId="2" ParentId="22405" CreationDate="2017-08-21T13:56:17.920" Score="1" Body="&lt;p&gt;The Bash feature you want is called command substitution, e.g:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;echo $(./test.r)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That will echo the output of your Rscript. Alternatively you could capture it in an environment variable.&lt;/p&gt;&#xA;" OwnerUserId="32402" LastActivityDate="2017-08-21T13:56:17.920" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22453" PostTypeId="1" AcceptedAnswerId="22461" CreationDate="2017-08-21T15:05:55.757" Score="1" ViewCount="76" Body="&lt;p&gt;I have a &lt;code&gt;.csv&lt;/code&gt; file with data in the following form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;moment_1;moment_2;moment_3;force_x;force_y;force_z;...&#xA;-0,02131267;-1,6032766088;5,9906811787;5,40010285;0,0203;86,44227467;...&#xA;2599;-1,70091039344;-1,3044809;-0,0406673590;-2,60896180797;43,2334;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The file is very large and I need to put it in an interactive visualization, that's why I need to reduce the data points without changing the overall structure too much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many data points are very close to each other as seen in the following image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jVE7f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jVE7f.png&quot; alt=&quot;too close&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach was to define a threshold and filter all points which have a distance to the previous point lower than the threshold. But I think that's not an optimal solution because, when I remove one index, I need to remove it from the other data array too, otherwise the structure is changed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there better approaches?&lt;/p&gt;&#xA;" OwnerUserId="32543" LastEditorUserId="29575" LastEditDate="2017-08-21T22:38:04.357" LastActivityDate="2017-08-22T11:35:29.547" Title="Feature selection on n different values" Tags="|visualization|feature-selection|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22454" PostTypeId="1" CreationDate="2017-08-21T16:12:27.800" Score="3" ViewCount="111" Body="&lt;p&gt;The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing interval and ratio scales into better digested bins. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when dealing with the data itself, the need for different &quot;exceptions&quot; arises. Sometimes it is just different kinds of null flavors, which can be written off as a meta data issue. But there are also more interesting cases. I have encountered satisfaction measurements where the overall satisfaction with a feature is considered to be &quot;excellent&quot;, &quot;good&quot; or &quot;bad&quot; if the distribution of answers peaks at the expected points in the scale, but &quot;polarising&quot; if the distribution is U-shaped, and &quot;unclear&quot; if there is a flattish or weirdly multimodal distribution. Obviously, the &quot;excellent&quot;, &quot;good&quot; and &quot;bad&quot; part is ordinal, but the last two are nominal. Other examples include a &quot;prefer not to say&quot; box in self-reported measures, if the reluctance of giving information is meaningful under the employed model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am pretty sure that we don't yet have qualitative methods to analyze these mixed scales. But I hope that the situation is at least recognized in theoretical texts, even if the only recommendation is &quot;treat them like nominal&quot;. Also, while a mixed statistical analysis is difficult, maybe there are at least visualization approaches capable of representing the true complexity of the situation. But I don't know how to look for such literature, since I have never encountered a name for this kind of scale. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does somebody know of a name (even if it is not widely established) and can point to sources which define it and deal with it?&lt;/p&gt;&#xA;" OwnerUserId="38272" LastActivityDate="2020-12-29T14:24:56.950" Title="Is there a name for a scale which mixes ordinal and nominal?" Tags="|categorical-data|theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22456" PostTypeId="1" AcceptedAnswerId="22458" CreationDate="2017-08-21T18:23:46.863" Score="4" ViewCount="9072" Body="&lt;p&gt;I used tensorflow to recognize text from natural images by using convolutional neural network; there is no specific number of characters in the text. To make a successful training I should convert the categorical labels into binary using one-hot encoding. So, for each label, I used integer encoding for each character and stored them in one numpy array in order to create TFRecords. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ '&#xA;TrainLabel  = [&quot;CNN in Tensorflow&quot;]&#xA;&#xA;# define a mapping of chars to integers&#xA;char_to_int = dict((c, i) for i, c in enumerate(alphabet))&#xA;integer_encoded = [char_to_int[char] for char in TrainLabel[0]]&#xA;&#xA;if (len(TrainLabel[0])) &amp;lt; 51:&#xA;    for j in xrange(51- (len(TrainLabel[0]))):&#xA;        integer_encoded.append(52)&#xA;&#xA;# one hot encode&#xA;onehot_encoded = []&#xA;for value in integer_encoded:&#xA;    letter = [0 for _ in range(len(alphabet))]&#xA;    letter[value] = 1&#xA;    onehot_encoded.append(letter)&#xA;&#xA;label = np.array(onehot_encoded, np.float32) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;51 is the maximum number of character in the text, so if the text has less than 51 characters, pad it to 51 characters with spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we print the label, it will be like this::&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],  &#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       ..., &#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  1.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  1.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  1.]], dtype=float32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;after creating the batch queue, the label has shape &lt;code&gt;[batch_size, 2703]&lt;/code&gt;. &lt;code&gt;2703&lt;/code&gt; is come from &lt;code&gt;51*53&lt;/code&gt; which 53 is the number of classes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is in loss function:: the label shape in &lt;strong&gt;tf.nn.sparse_softmax_cross_entropy_with_logits()&lt;/strong&gt; must be [batch_size], but the label that I used here has this shape [batch_size, 53] because I used &lt;strong&gt;one-hot encoding&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I deal with that?? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the problem::&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(labels_static_shape.ndims, logits.get_shape().ndims)) ValueError:&#xA;  Rank mismatch: Rank of labels (received 2) should equal rank of logits&#xA;  minus 1 (received 2).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="37775" LastEditorUserId="29575" LastEditDate="2017-08-22T10:00:02.127" LastActivityDate="2020-02-16T11:14:32.063" Title="how can I solve label shape problem in tensorflow when using one-hot encoding?" Tags="|neural-network|tensorflow|convolution|ocr|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22457" PostTypeId="1" CreationDate="2017-08-21T18:31:58.773" Score="5" ViewCount="4841" Body="&lt;p&gt;I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at &lt;a href=&quot;http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/&quot; rel=&quot;nofollow noreferrer&quot;&gt;kaggle&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain what are the residuals that are modeled in case of a classification scenario?&lt;/p&gt;&#xA;" OwnerUserId="38284" LastEditorUserId="29575" LastEditDate="2017-08-22T05:25:42.017" LastActivityDate="2017-10-21T09:52:03.203" Title="Residuals in a gradient boosted classification" Tags="|classification|xgboost|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22458" PostTypeId="2" ParentId="22456" CreationDate="2017-08-21T18:34:41.367" Score="2" Body="&lt;p&gt;Since your data are already in a one-hot encoding, you can use &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.nn.softmax_cross_entropy_with_logits()&lt;/code&gt;&lt;/a&gt;, which expects an input of shape &lt;code&gt;[batch_size, num_classes]&lt;/code&gt; for the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.nn.sparse_softmax_cross_entropy_with_logits()&lt;/code&gt;&lt;/a&gt; op expects the labels as a batch of integers, where each integer corresponds to the class ID for each example.)&lt;/p&gt;&#xA;" OwnerUserId="14517" LastActivityDate="2017-08-21T18:34:41.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22460" PostTypeId="2" ParentId="22442" CreationDate="2017-08-21T19:13:34.940" Score="0" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [178]: pd.DataFrame([[k[0], k[1], v.split()[1]] for k,v in d.items()],&#xA;                       columns=['id','Name','Value'])&#xA;Out[178]:&#xA;          id                              Name     Value&#xA;0  8758148.0      CI Alpine Growth Equity Fund  0.000016&#xA;1  8758148.0           CI American Growth Fund       0.0&#xA;2  8758148.0  CI American Small Companies Fund       0.0&#xA;3  8758148.0            CI American Value Fund       0.0&#xA;4  8758148.0           CI Canadian Equity Fund       0.0&#xA;5  8758148.0       CI Canadian Investment Fund       0.0&#xA;6  8758148.0        CI Canadian Small Cap Fund       0.0&#xA;7  8758148.0    CI Canadian Small/Mid Cap Fund       0.0&#xA;8  8758148.0           CI Global Bond RSP Fund       0.0&#xA;9  8758148.0                   CI Harbour Fund       0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;d&lt;/code&gt; is your dictionary&lt;/p&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-08-21T19:13:34.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22461" PostTypeId="2" ParentId="22453" CreationDate="2017-08-21T20:52:27.393" Score="2" Body="&lt;p&gt;Instead of filtering single points I would suggest that you smooth your data using established techniques, e.g. Savitzky–Golay filter. Another option would be to employ Kernel Density Estimation, where you can then visualize the curves using a reduced, regular set of supporting points.&lt;/p&gt;&#xA;" OwnerUserId="38291" LastActivityDate="2017-08-21T20:52:27.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22464" PostTypeId="2" ParentId="22016" CreationDate="2017-08-22T04:24:24.513" Score="1" Body="&lt;p&gt;If you&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;run a jupyter server&lt;/li&gt;&#xA;&lt;li&gt;open a new notebook with one cell: &lt;code&gt;x=1&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;close the notebook&lt;/li&gt;&#xA;&lt;li&gt;re-open the notebook&lt;/li&gt;&#xA;&lt;li&gt;add another cell under the 1st: &lt;code&gt;x+1&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;it would output &lt;code&gt;2&lt;/code&gt;, showing that the variable you had set is still in memory in the kernel&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if you completely shutdown the jupyter server, or if you shutdown the kernel of your notebook, then you'll need to re-run the whole thing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;a href=&quot;https://github.com/jupyterlab/jupyterlab&quot; rel=&quot;nofollow noreferrer&quot;&gt;JupyterLab&lt;/a&gt; is now in Beta and worth trying out as recommended &lt;a href=&quot;https://stackoverflow.com/a/46135233/4126114&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35596" LastEditorUserId="35596" LastEditDate="2017-09-21T15:02:01.663" LastActivityDate="2017-09-21T15:02:01.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22465" PostTypeId="1" CreationDate="2017-08-22T05:55:58.930" Score="-2" ViewCount="452" Body="&lt;p&gt;I am using Word2Vec for text vectorization. It is doing a good job but some cases it is failing. For example &quot;turn the computer off and on&quot; and the sentence &quot;restart the computer&quot; does not have a very good similarity score, even though they mean the same thing. Doc2Vec is not doing a good job as my inputs are usually a couple of sentences and not a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please suggest an approach which would give a good similarity score between &quot;turn on and off&quot; and &quot;restart&quot; and also other combinations like that?&lt;/p&gt;&#xA;" OwnerUserId="25766" LastActivityDate="2021-02-06T18:34:06.743" Title="Finding similarity between a word and a sentence (like &quot;restart&quot; and &quot;turn off and on&quot;)" Tags="|machine-learning|python|nlp|word2vec|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22468" PostTypeId="2" ParentId="22457" CreationDate="2017-08-22T08:25:54.250" Score="3" Body="&lt;p&gt;It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to a probability. The loss function used for deciding the weights of the terminal nodes is adapted from the normal sigmoid loss to not have to map directly to probabilities. I cannot find it at the moment but it should be easy to derive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I found the post that I read this from the other day, it's over at stats stackexchange: &lt;a href=&quot;https://stats.stackexchange.com/questions/204154/classification-with-gradient-boosting-how-to-keep-the-prediction-in-0-1&quot;&gt;https://stats.stackexchange.com/questions/204154/classification-with-gradient-boosting-how-to-keep-the-prediction-in-0-1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-08-22T08:25:54.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22470" PostTypeId="1" AcceptedAnswerId="22472" CreationDate="2017-08-22T09:31:03.423" Score="24" ViewCount="40229" Body="&lt;p&gt;I have a very basic question which relates to Python, numpy and multiplication of matrices in the setting of logistic regression.&lt;/p&gt;&#xA;&lt;p&gt;First, let me apologise for not using math notation.&lt;/p&gt;&#xA;&lt;p&gt;I am confused about the use of matrix dot multiplication versus element wise pultiplication. The cost function is given by:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$J = - {1\over m} \sum_{i=1}^m y^{(i)}log(a^{(i)})+(1 - y^{(i)})log(1-a^{(i)})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;And in python I have written this as&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    cost = -1/m * np.sum(Y * np.log(A) + (1-Y) * (np.log(1-A)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But for example this expression (the first one - the derivative of J with respect to w)&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;${\partial J \over{\partial w}} = {1 \over{m}} X(A-Y)^T$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;${\partial J\over{\partial b}} = {1\over{m}} \sum \limits_{i = 1}^m (a^{(i)}-y^{(i)})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;is&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;   dw = 1/m * np.dot(X, dz.T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I don't understand why it is correct to use dot multiplication in the above, but use element wise multiplication in the cost function i.e why not:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;   cost = -1/m * np.sum(np.dot(Y,np.log(A)) + np.dot(1-Y, np.log(1-A)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I fully get that this is not elaborately explained but I am guessing that the question is so simple that anyone with even basic logistic regression experience will understand my problem.&lt;/p&gt;&#xA;" OwnerUserId="38306" LastEditorUserId="29169" LastEditDate="2021-03-11T20:24:28.610" LastActivityDate="2021-06-06T18:46:11.230" Title="Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another" Tags="|python|logistic-regression|cost-function|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22471" PostTypeId="2" ParentId="22470" CreationDate="2017-08-22T10:15:04.673" Score="3" Body="&lt;p&gt;Are you asking, what's the difference between a dot product of two vectors, and summing their elementwise product? They are the same. &lt;code&gt;np.sum(X * Y)&lt;/code&gt; is &lt;code&gt;np.dot(X, Y)&lt;/code&gt;. The dot version would be more efficient and easy to understand, generally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in the cost function, $Y$ is a matrix, not a vector. &lt;code&gt;np.dot&lt;/code&gt; actually computes a matrix product, and the sum of those elements is not the same as the sum of the elements of the pairwise product. (The multiplication isn't even going to be defined for the same cases.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I guess the answer is that they're different operations doing different things, and these situations are different, and the main difference is dealing with vectors versus matrices.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-08-22T10:15:04.673" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22472" PostTypeId="2" ParentId="22470" CreationDate="2017-08-22T10:48:45.013" Score="17" Body="&lt;p&gt;In this case, the two math formulae show you the correct type of multiplication:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\text{log}(a_i)$&lt;/span&gt; in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; only interacts with its matching element in &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt;, which is basically the definition of element-wise.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The terms in the gradient calculation are matrices, and if you see two matrices &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$B$&lt;/span&gt; multiplied using notation like &lt;span class=&quot;math-container&quot;&gt;$C = AB$&lt;/span&gt;, then you can write this out as a more complex sum: &lt;span class=&quot;math-container&quot;&gt;$C_{ik} = \sum_j A_{ij}B_{jk}$&lt;/span&gt;. It is this inner sum across multiple terms that &lt;code&gt;np.dot&lt;/code&gt; is performing.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;In part your confusion stems from the &lt;em&gt;vectorisation&lt;/em&gt; that has been applied to equations in the course materials, which are looking forward to more complex scenarios. You could in fact use&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;cost = -1/m * np.sum( np.multiply(np.log(A), Y) + np.multiply(np.log(1-A), (1-Y)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;cost = -1/m * np.sum( np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y.T)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;whilst &lt;code&gt;Y&lt;/code&gt; and &lt;code&gt;A&lt;/code&gt; have shape &lt;code&gt;(m,1)&lt;/code&gt; and it should give the same result. NB the &lt;code&gt;np.sum&lt;/code&gt; is just flattening a single value in that, so you could drop it and instead have &lt;code&gt;[0,0]&lt;/code&gt; on the end. However, this does not generalize to other output shapes &lt;code&gt;(m,n_outputs)&lt;/code&gt; so the course does not use it.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="118874" LastEditDate="2021-06-06T18:46:11.230" LastActivityDate="2021-06-06T18:46:11.230" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22474" PostTypeId="2" ParentId="22453" CreationDate="2017-08-22T11:35:29.547" Score="0" Body="&lt;p&gt;You could try a linear filter like a Gaussian. It will smooth the data and has the property that no new local maxima or minima will be created (see Gaussian scale space theory). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I think a median filter (non-linear) may give you better results, as it will remove single spikes but preserve step edges. The kalman filter is also suggested in one of the answers here:  &lt;a href=&quot;https://dsp.stackexchange.com/questions/4680/what-is-smoothing-in-very-basic-terms/4682#4682&quot;&gt;https://dsp.stackexchange.com/questions/4680/what-is-smoothing-in-very-basic-terms/4682#4682&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="34165" LastActivityDate="2017-08-22T11:35:29.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22475" PostTypeId="1" CreationDate="2017-08-22T12:35:58.567" Score="1" ViewCount="170" Body="&lt;p&gt;I have a data set of some &lt;strong&gt;questions&lt;/strong&gt; and &lt;strong&gt;answers&lt;/strong&gt; that users have completed by choices. I'm trying to build a recommendation engine to find similar users based on their answers on top of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for each user I have a data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_1, {&quot;question_id&quot;: &quot;choice_id&quot;, ...}&#xA;user_2, {&quot;question_id&quot;: &quot;choice_id&quot;, ...}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've found clustering and similarity algorithms will be helpful and I've chosen Apache Spark's &lt;strong&gt;RowMatrix&lt;/strong&gt; &lt;strong&gt;columnSimilarities&lt;/strong&gt; to be very efficient.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.mllib.linalg.distributed import RowMatrix&#xA;&#xA;rows = sc.parallelize([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])&#xA;&#xA;mat = RowMatrix(rows)&#xA;&#xA;mat.columnSimilarities(threshold)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is how should I transform my data to get an input like this?&#xA;I've found &lt;strong&gt;scikit-learn's DictVectorizer&lt;/strong&gt; may be helpful but the problem is users complete questions randomly so I can't use questions as features.&lt;/p&gt;&#xA;" OwnerUserId="38322" LastActivityDate="2017-08-22T12:35:58.567" Title="Creating a user-based recommendation engine?" Tags="|python|apache-spark|pyspark|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22476" PostTypeId="1" AcceptedAnswerId="30723" CreationDate="2017-08-22T12:52:49.833" Score="5" ViewCount="365" Body="&lt;p&gt;I often have trouble deciding how to tweak my input data for the agent. I am changing my data so that values in the hundreds get changed to a value between -1 and 1, and values that are originally small get changed the same ratio. However, my agents often aren't successful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could the numeric range of the input data be a cause for the inability to learn? &lt;/p&gt;&#xA;" OwnerUserId="37548" LastEditorUserId="37548" LastEditDate="2017-08-23T00:33:23.733" LastActivityDate="2018-10-21T19:00:07.510" Title="Is feature scaling necessary in reinforcement learning for the agent to learn successfully?" Tags="|reinforcement-learning|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22477" PostTypeId="1" CreationDate="2017-08-22T12:52:51.643" Score="6" ViewCount="2089" Body="&lt;p&gt;I'm using Tensorflow for an auto-tagging task on audio clips. The problem is actually a multilabel classification problem meaning that each clip can have multiple tags at the same time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the model evaluation, I would like to be able to compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve&quot; rel=&quot;noreferrer&quot;&gt;AUC&lt;/a&gt; per clip and per tag. My problem is that AUC is basically a metric for binary classification and I don't know how it can be extended to a multilabel problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found many references of AUC computation on multilabel classification tasks, but none of these explain how to do it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://arxiv.org/abs/1508.04999&quot; rel=&quot;noreferrer&quot;&gt;An example&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="33466" LastEditorUserId="33466" LastEditDate="2017-08-22T14:29:25.277" LastActivityDate="2020-10-17T13:07:26.520" Title="AUC computation on multilabel classification" Tags="|machine-learning|tensorflow|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22478" PostTypeId="1" CreationDate="2017-08-22T13:09:19.953" Score="4" ViewCount="4131" Body="&lt;p&gt;From &lt;a href=&quot;https://datascience.stackexchange.com/questions/16342/unbalanced-multiclass-data-with-xgboost/18823&quot;&gt;this post&lt;/a&gt;, I know you can set &lt;code&gt;scale_pos_weight&lt;/code&gt; for an imbalanced dataset.  However, for the multi-classification problem in the imbalanced dataset, I don't quite understand how to set the weight parameter in &lt;code&gt;dmatrix&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use XGBoost for an imbalanced dataset in a multi-classification problem?&lt;/p&gt;&#xA;" OwnerUserId="38308" LastEditorUserId="29575" LastEditDate="2017-08-22T14:03:08.837" LastActivityDate="2020-10-16T04:26:44.400" Title="How to set weights in multi-class classification in xgboost for imbalanced data?" Tags="|xgboost|multiclass-classification|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22481" PostTypeId="1" CreationDate="2017-08-22T14:40:06.910" Score="-1" ViewCount="292" Body="&lt;p&gt;I'm building a recommendation system, but my data has high sparsity.  In most of my data, each user gives one feedback, for one item. For example, I have 10 items and 15 users, I have 3 feedbacks, from 3 different users, and the scenario for others items is like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jmwwH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jmwwH.png&quot; alt=&quot;Example&quot;&gt;&lt;/a&gt;          &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a preferred approach to building a recommendation system with this kind of data?&lt;/p&gt;&#xA;" OwnerUserId="38327" LastEditorUserId="29575" LastEditDate="2017-08-22T15:02:19.387" LastActivityDate="2017-08-23T17:25:49.367" Title="Which recommender system approach is good with high sparsity in user?" Tags="|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22483" PostTypeId="1" CreationDate="2017-08-22T15:47:36.553" Score="0" ViewCount="82" Body="&lt;p&gt;I am toying with AWS Machine Learning and I have a dataset with about 200 records with about 220,000 variables each.&lt;/p&gt;&#xA;&lt;p&gt;Apparently, AWS Machine Learning has a hard limit of &lt;a href=&quot;http://docs.aws.amazon.com/machine-learning/latest/dg/system-limits.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;1000 variables&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;How can I still analyse my dataset through AWS ML? Should I apply a MDR procedure through my &lt;em&gt;200 x 200000&lt;/em&gt; dataset or should I split my variables into chunks of 1000 and combine them somehow after processing each of those chunks through AWS ML?&lt;/p&gt;&#xA;" OwnerUserId="38314" LastEditorUserId="85045" LastEditDate="2020-11-13T22:11:36.473" LastActivityDate="2020-11-13T22:11:36.473" Title="Maximum Dimensionality of AWS Machine Learning" Tags="|machine-learning|aws|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22484" PostTypeId="2" ParentId="22247" CreationDate="2017-08-22T17:46:46.687" Score="1" Body="&lt;p&gt;Because your data has 30 rows and your loop is from 1 to 10 so when it gets to &lt;code&gt;i=10&lt;/code&gt; then &lt;code&gt;end &amp;lt;- (i+1)*M&lt;/code&gt; which is 11 * 3 which is 33 and then you do &lt;code&gt;Ind[beg:end]&lt;/code&gt; but &lt;code&gt;Ind&lt;/code&gt; is only 30 long so you get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1] 21 NA NA NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then its NA NA NA NA hey hey goodbye.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should try basic debugging techniques before posting here. Print &lt;code&gt;i&lt;/code&gt; in the loop to see where it fails. Print intermediate values to check they are as expected. Step through code using RStudio's debugger if you are into that.&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-08-22T17:46:46.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22486" PostTypeId="5" CreationDate="2017-08-22T17:58:46.647" Score="0" Body="&lt;p&gt;Batch normalization was introduced in the 2015 paper &lt;a href=&quot;https://arxiv.org/abs/1502.03167&quot; rel=&quot;nofollow noreferrer&quot;&gt;Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is conceptually an extension of the general advice to scale input values to neural networks (typically to mean 0, standard deviation 1). Batch normalization applies the same idea to re-scale neural network activations on a per-layer basis, and this can result in improved training times.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-22T17:58:46.647" LastActivityDate="2017-08-22T17:58:46.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22487" PostTypeId="4" CreationDate="2017-08-22T17:58:46.647" Score="0" Body="For questions about Batch Normalization of layer activations in theory and practice, as used in (typically deep) neural networks." OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-22T17:58:46.647" LastActivityDate="2017-08-22T17:58:46.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22488" PostTypeId="1" AcceptedAnswerId="31611" CreationDate="2017-08-22T19:48:52.230" Score="7" ViewCount="27809" Body="&lt;p&gt;I am trying an &lt;code&gt;LSTM&lt;/code&gt; model using &lt;code&gt;tensorflow&lt;/code&gt; following this &lt;a href=&quot;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&quot; rel=&quot;noreferrer&quot;&gt;tutorial&lt;/a&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having trouble understanding why am I getting an error in my test set when I try to invert scaling for forecast (line 86 in the tutorial). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After loading the dataset and created featured, following is what I did for &lt;code&gt;reframed&lt;/code&gt; dataset,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# split into train and test sets&#xA;values = reframed.values&#xA;split_point = len(reframed)- 168&#xA;&#xA;train = values[ : split_point, :]&#xA;test = values[split_point: , :]&#xA;&#xA;# split into input and outputs&#xA;train_X, train_y = train[:, :-1], train[:, -1]&#xA;test_X, test_y = test[:, :-1], test[:, -1]&#xA;&#xA;# reshape input to be 3D [samples, timesteps, features]&#xA;train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))&#xA;test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))&#xA;&#xA;print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)&#xA;&amp;gt;&amp;gt; (2399, 1, 39) (2399,) (168, 1, 39) (168,)&#xA;&#xA;&#xA;# design network&#xA;model = Sequential()&#xA;model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))&#xA;model.add(Dense(1))&#xA;model.compile(loss='mae', optimizer='adam')&#xA;&#xA;&#xA;# fit network&#xA;history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)&#xA;&#xA;# make a prediction&#xA;yhat = model.predict(test_X)&#xA;test_X = test_X.reshape((test_X.shape[0], test_X.shape[2]))&#xA;&#xA;inv_yhat = np.concatenate((yhat, test_X[:, 1:]), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this step, the following code that has &lt;code&gt;scaler.invert_transform&lt;/code&gt; throws the error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inv_yhat = scaler.inverse_transform(inv_yhat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: operands could not be broadcast together with shapes (168,39) (41,) (168,39)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Shape,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_X.shape&#xA;&amp;gt;&amp;gt; (168, 1, 39)&#xA;&#xA;yhat.shape&#xA;&amp;gt;&amp;gt;(168,1)&#xA;&#xA;inv_yhat.shape&#xA;&amp;gt;&amp;gt; (168,39)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help or sugeestions would be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-08-23T17:15:50.540" LastActivityDate="2021-01-15T21:58:41.147" Title="Value Error: Operands could not be broadcast together with shapes - LSTM" Tags="|machine-learning|python|tensorflow|lstm|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="22489" PostTypeId="2" ParentId="22299" CreationDate="2017-08-22T20:09:02.617" Score="10" Body="&lt;p&gt;No, it is not necessary that this is the case, however, this is, in a convoluted way, the goal of T-SNE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before getting into the meat of the answer, let's take a look at some basic definitions, both mathematically and intuitively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Nearest Neighbors&lt;/strong&gt;: Consider a metric space $\mathbb{R}^d$ and a set of vectors $X_1, ..., X_n \in \mathbb{R}^d$, given a new vector $x \in \mathbb{R}^d$, we want to find the points such that $|| X_1 - x || \le ... \le ||X_n - x ||$. Intuitively, it's just the minimum of the distances using a suitable definition of norm in $\mathbb{R}^d$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now coming to whether the nearest neighbors actually matter while applying dimensionality reduction. Usually in my answers, I intend to rationalize something with mathematics, code and intuition. Let us first consider the intuitive aspect of things. If you have a point that is a distance $d$ away from another point, from our understanding of the t-sne algorithm we know that this distance is preserved as we transition into higher dimensions. Let us further assume that a point $y$ is the nearest neighbor of $x$ in some dimension $d$. By definition, there is a relationship between the distance in $d$ and $d + k$. So, we have our intuition which is that the distance is maintained across different dimensions, or at least, that is what we aim for. Let's try to justify it with some mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this answer I talk about the math involved in t-sne, albeit not in detail (&lt;a href=&quot;https://datascience.stackexchange.com/questions/19025/t-sne-why-equal-data-values-are-visually-not-close&quot;&gt;t-SNE: Why equal data values are visually not close?&lt;/a&gt;). What the math here is, is basically maximizing the probability that two points remain close in a projected space as they are in the original space assuming that the distribution of the points is exponential. So, looking at this equation $p_{j | i} = \frac{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}{\sum_{k \neq i}{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}}$. Notice that the probability is dependent on the distance between the two points, so the further apart they are, the further apart they get as they get projected to lower dimensions. Notice that if they are far apart in $\mathbb{R}^k$, there is a good chance they will not be close in the projected dimension. So now, we have a mathematical justification as to why the points &quot;should&quot; remain close. But again, since this is an exponential distribution, if these points are significantly far apart, there is no guarantee that the Nearest Neighbors property is maintained, although, this is the aim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now finally a neat coding example which demonstrates this concept too.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.manifold import TSNE&#xA;from sklearn.neighbors import KNeighborsClassifier&#xA;X = [[0],[1],[2],[3],[4],[5],[6],[7],[8],[9]]&#xA;y = [0,1,2,3,4,5,6,7,8,9]&#xA;neighs = KNeighborsClassifier(n_neighbors=3)&#xA;neighs.fit(X, y)&#xA;X_embedded = TSNE(n_components=1).fit_transform(X)&#xA;neighs_tsne = KNeighborsClassifier(n_neighbors=3)&#xA;neighs_tsne.fit(X_embedded, y)&#xA;print(neighs.predict([[1.1]]))&#xA;&amp;gt;&amp;gt;&amp;gt;[0]&#xA;print(neighs_tsne.predict([[1.1]]))&#xA;&amp;gt;&amp;gt;&amp;gt;[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although this is a very naive example and doesn't reflect the complexity, it does work by experiment for some simple examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Also, adding some points with respect to the question itself, so it is not necessary that this is the case, it might be, however, rationalizing it through mathematics would prove that you have no concrete outcome (no definitive yes or no).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this cleared up some of your concerns with TSNE.&lt;/p&gt;&#xA;" OwnerUserId="27114" LastEditorUserId="27114" LastEditDate="2017-08-24T16:41:22.080" LastActivityDate="2017-08-24T16:41:22.080" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22491" PostTypeId="1" CreationDate="2017-08-22T20:23:33.073" Score="4" ViewCount="1525" Body="&lt;p&gt;I am wondering what is the best practice other devs are using for their python spark jobs. &#xA;I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to a standalone cluster and being able to run my code from PyCharm or at least from the command line onto the cluster. I am using Windows 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any success stories with such set up? what is the recommended setup/practice?&lt;/p&gt;&#xA;" OwnerUserId="32424" LastActivityDate="2018-01-12T12:51:34.173" Title="Spark development on local machine with PyCharm" Tags="|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22494" PostTypeId="1" AcceptedAnswerId="22618" CreationDate="2017-08-22T23:52:26.863" Score="24" ViewCount="11247" Body="&lt;p&gt;I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've managed to achieve around 85% classification accuracy on my test set, however I set a goal of achieving 90% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main problem is overfitting. Somehow it always ends up happening (normally after epoch 8-10). The architecture of my network is loosely inspired by VGG-16, more specifically my images are resized to &lt;span class=&quot;math-container&quot;&gt;$128x128x3$&lt;/span&gt;, and then I run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Convolution 1 128x128x32 (kernel size is 3, strides is 1)&#xA;Convolution 2 128x128x32 (kernel size is 3, strides is 1)&#xA;Max pool    1 64x64x32   (kernel size is 2, strides is 2)&#xA;Convolution 3 64x64x64   (kernel size is 3, strides is 1)&#xA;Convolution 4 64x64x64   (kernel size is 3, strides is 1)&#xA;Max pool    2 32x32x64   (kernel size is 2, strides is 2)&#xA;Convolution 5 16x16x128  (kernel size is 3, strides is 1)&#xA;Convolution 6 16x16x128  (kernel size is 3, strides is 1)&#xA;Max pool    3 8x8x128    (kernel size is 2, strides is 2)&#xA;Convolution 7 8x8x256    (kernel size is 3, strides is 1)&#xA;Max pool    4 4x4x256    (kernel size is 2, strides is 2)&#xA;Convolution 8 4x4x512    (kernel size is 3, strides is 1)&#xA;Fully connected layer 1024 (dropout 0.5)&#xA;Fully connected layer 1024 (dropout 0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All the layers except the last one have relus as activation functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I have tried different combinations of convolutions (I started with simpler convolutions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I have augmented the dataset by mirroring the images, so that in total I have 50000 images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I am normalizing the images using min max normalization, where X is the image&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$X = X - 0 / 255 - 0$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is written in tensorflow and the batch sizes are 128.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mini-batches of training data end up overfitting and having an accuracy of 100% while the validation data seems to stop learning at around 84-85%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried to increase/decrease the dropout rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimizer being used is AdamOptimizer with a learning rate of 0.0001&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment I have been playing with this problem for the last 3 weeks and 85% seems to have set a barrier in front of me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the record, I know I could use transfer learning to achieve much higher results, but I am interesting on building this network as a self-learning experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am running the SAME network with a different batch size, in this case I am using a much smaller batch size (16 instead of 128) so far I am achieving 87.5% accuracy (instead of 85%). That said, the network ends up overfitting anyway. Still I do not understand how a dropout of 50% of the units is not helping... obviously I am doing something wrong here. Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seems like the problem had to do with the batch size, as with a smaller size (16 instead of 128) I am achieving now 92.8% accuracy on my test set, with the smaller batch size the network still overfits (the mini batches end up with an accuracy of 100%) however, the loss (error) keeps decreasing and it is in general more stable. The cons are a MUCH slower running time, but it is totally worth the wait.&lt;/p&gt;&#xA;" OwnerUserId="38339" LastEditorUserId="29169" LastEditDate="2021-02-16T00:42:47.423" LastActivityDate="2021-02-16T00:42:47.423" Title="Convolutional neural network overfitting. Dropout not helping" Tags="|deep-learning|neural-network|convolutional-neural-network|image-recognition|dropout|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="22496" PostTypeId="1" AcceptedAnswerId="22497" CreationDate="2017-08-23T05:05:06.707" Score="2" ViewCount="1660" Body="&lt;p&gt;Here's the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PlayerID, Characters, Win or Lose&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can make it look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1, 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2GRG822L9,&quot;barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,&quot;,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The second column is an 8 character combination from 70+ n characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to encode the variables to be dummy variables, so each character gets its own column. Is there a way to do this in python/R? I'm assuming you have to leave the second column as a string rather than outputting a csv file that looks like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2GRG822L9,barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,1&#xA;8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It should probably look like this before dummy encoding (I can get rid of commas within the string)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2GRG822L9,&quot;barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,&quot;,1&#xA;8PYPY0LLQ,&quot;valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,&quot;,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38302" LastActivityDate="2017-08-23T05:43:27.727" Title="Converting a string to dummy encoded variables" Tags="|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22497" PostTypeId="2" ParentId="22496" CreationDate="2017-08-23T05:43:27.727" Score="3" Body="&lt;p&gt;It is simple with python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame&#xA;data = [('2GRG822L9',&quot;barbarians8,valkyrie5,chr_balloon3,fire_spirits8,minion8,firespirit_hut6,rage4,skeleton_horde3&quot;,1), ('8PYPY0LLQ',&quot;valkyrie5,chr_witch4,hog_rider5,zapMachine1,mega_minion3,baby_dragon2,bomber7,skeleton_horde1&quot;,0)]&#xA;df = DataFrame.from_records(data,columns=('PlayerID', 'Characters', 'Result'))&#xA;df = df.drop('Characters', 1).join(df.Characters.str.get_dummies(','))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    PlayerID  Result  baby_dragon2  barbarians8  bomber7  chr_balloon3  \&#xA;0  2GRG822L9       1             0            1        0             1   &#xA;1  8PYPY0LLQ       0             1            0        1             0   &#xA;&#xA;   chr_witch4  fire_spirits8  firespirit_hut6  hog_rider5  mega_minion3  \&#xA;0           0              1                1           0             0   &#xA;1           1              0                0           1             1   &#xA;&#xA;   minion8  rage4  skeleton_horde1  skeleton_horde3  valkyrie5  zapMachine1  &#xA;0        1      1                0                1          1            0  &#xA;1        0      0                1                0          1            1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-08-23T05:43:27.727" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22498" PostTypeId="1" CreationDate="2017-08-23T06:46:40.787" Score="1" ViewCount="788" Body="&lt;p&gt;I need to train a CNN (&lt;a href=&quot;https://pjreddie.com/darknet/yolo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;YOLO&lt;/a&gt; to be precise) on RGB + depth information. I have the RGB input as a separate PNG file and the depth map as a separate PNG file. These PNG files are 8 bit depth, 3 channel files. I need to create a single PNG file of 16 bit depth, 3 channels which is essentially a combination of both the original files. What is the right way to combine / encode RGB + depth information? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data 1: R, G, B channels of 8 bits each&lt;br&gt;&#xA;Data 2: D1, D2, D3 channels of 8 bits each&lt;br&gt;&#xA;Required: C1, C2, C3 channels of 16 bits each&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current idea&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using a bitwise shifting.&lt;br&gt;&#xA;C1 = R &amp;lt;&amp;lt; 8 | D1&lt;br&gt;&#xA;C2 = G &amp;lt;&amp;lt; 8 | D2&lt;br&gt;&#xA;C3 = B &amp;lt;&amp;lt; 8 | D3&lt;br&gt;&#xA;(shift by 8 bits left and then OR)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Notice that the channels left shifted (R, G and B) obtain higher significant bits and dominate over D1, D2, and D3 while forming C1, C2, C3. The CNN loses the information present in the depth map. Is there a better encoding?&lt;/li&gt;&#xA;&lt;li&gt;A small note about the depth map. The depth information requires 16&#xA;bits and is contained in D2 and D3. D1 channel is a pseudo channel&#xA;with only gray scale information of the image. The final encoding&#xA;should focus on D2 and D3 more.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Another idea&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make sure no channel dominates over the other, I was thinking of interleaving the two 8 bit sequences to form a 16 bit sequence. If channel R has &lt;code&gt;b1, b2, ..., b8&lt;/code&gt; bit sequence and channel D1 has &lt;code&gt;bb1, bb2, ..., bb8&lt;/code&gt; bit sequence then the combined 16 bit C1 channel will have &lt;code&gt;b1, bb1, b2, bb2, ..., b8, bb8&lt;/code&gt; bit sequence. Any thoughts on this?&lt;/p&gt;&#xA;" OwnerUserId="32850" LastEditorUserId="32850" LastEditDate="2017-08-25T11:46:09.023" LastActivityDate="2017-08-25T11:46:09.023" Title="RGB + Depth Encoding for CNNs" Tags="|neural-network|classification|deep-learning|feature-selection|image-recognition|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22499" PostTypeId="2" ParentId="22494" CreationDate="2017-08-23T08:00:28.403" Score="3" Body="&lt;p&gt;There are several possible solutions for your Problem.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use Dropout in the earlier layers (convolutional layers) too.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your network seems somehow quite big for such an &quot;easy&quot; task; try to reduce it. The big architectures are also trained on much bigger datasets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you want to keep your &quot;big&quot; architecture try:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Image augmentation in order to virtually increase your training data&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Try adversarial training. It sometimes helps.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37330" LastEditorUserId="29575" LastEditDate="2017-08-23T14:40:14.643" LastActivityDate="2017-08-23T14:40:14.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22501" PostTypeId="2" ParentId="22494" CreationDate="2017-08-23T09:32:50.927" Score="3" Body="&lt;p&gt;I suggest you analyze the learning plots of your validation accuracy as Neil Slater suggested. Then, if the validation accuracy drops try to reduce the size of your network (seems too deep), add dropout to the CONV layers and BatchNormalization after each layer. It can help get rid of overfitting and increase the test accuracy.  &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2017-08-23T09:32:50.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22502" PostTypeId="2" ParentId="22465" CreationDate="2017-08-23T09:53:00.503" Score="0" Body="&lt;p&gt;If you are training your word2vec by yourself than you should increase your training dataset. You can easily get the Wikipedia &lt;a href=&quot;https://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia&quot; rel=&quot;nofollow noreferrer&quot;&gt;database&lt;/a&gt;. If you are using a pretrained model, you can always fine tune it with additional data. &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2017-08-23T09:53:00.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22503" PostTypeId="2" ParentId="22395" CreationDate="2017-08-23T10:07:19.763" Score="1" Body="&lt;p&gt;I would have started with a regexp to find all degrees in the email and then compare them to take the highest one (you can order them) ex. Masters &gt; Bachelor. This might be a little bit too easy but worth try. &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2017-08-23T10:07:19.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22505" PostTypeId="2" ParentId="128" CreationDate="2017-08-23T10:17:58.587" Score="0" Body="&lt;p&gt;I have a situation where HDP works well compared to LDA. I have about 16000 documents that belong to various classes. As I am unaware of how many different topics I can gather for each class, HDP is really helpful in this case. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-08-23T10:17:58.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22506" PostTypeId="2" ParentId="22301" CreationDate="2017-08-23T10:26:33.323" Score="1" Body="&lt;p&gt;Something that has helped me when I have worked with tweets is what is the idea behind using the tweets. There is no standard way of doing it as sometimes things that are irrelevant to tokenizers are the ones you actually need. I would take a generic approach of tokenizer, modified for the use case I work with. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, ensure you keep a copy of the raw tweets after processing, so that you do not lose any information, you might need it for later use.&lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-08-23T10:26:33.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22507" PostTypeId="2" ParentId="20329" CreationDate="2017-08-23T10:32:44.663" Score="0" Body="&lt;p&gt;In this problem, the signature in each email is related to the text that is before the signature that's why I would suggest using recurrent neural networks like LSTMs or GRUs especially if it's a classification task. You can also try the LSTM seq to seq model if your task is to output the signature. &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2017-08-23T10:32:44.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22508" PostTypeId="2" ParentId="22465" CreationDate="2017-08-23T10:35:58.427" Score="0" Body="&lt;p&gt;One approach you could take is to build sentence vectors using vectors generated for Words.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/29760935/how-to-get-vector-for-a-sentence-from-the-word2vec-of-tokens-in-sentence&quot;&gt;This post&lt;/a&gt; covers the different techniques you could use to achieve it.&lt;/p&gt;&#xA;" OwnerUserId="1029" LastEditorUserId="29169" LastEditDate="2021-02-06T18:34:06.743" LastActivityDate="2021-02-06T18:34:06.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22511" PostTypeId="2" ParentId="20012" CreationDate="2017-08-23T12:12:07.157" Score="5" Body="&lt;p&gt;That is known a problem with the ReLU activation functions. It is often called a &quot;dying ReLU&quot;. Given an input over the zero boundary, the unit is now almost always closed. A closed ReLU cannot update its input parameters, a dead ReLU stays dead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is to use variants of ReLU for the activation function such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)&quot; rel=&quot;noreferrer&quot;&gt;Leaky ReLU, Noisy ReLUs, or ELUs&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-23T12:12:07.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22512" PostTypeId="2" ParentId="22128" CreationDate="2017-08-23T12:18:19.673" Score="1" Body="&lt;p&gt;You do &lt;strong&gt;not&lt;/strong&gt; need to use Word Embedding to solve this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are performing set membership check. In this problem, each book title and author combination is an &lt;a href=&quot;https://en.wikipedia.org/wiki/Element_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;element&lt;/a&gt;. You can create a &lt;a href=&quot;https://en.wikipedia.org/wiki/Hash_table&quot; rel=&quot;nofollow noreferrer&quot;&gt;hash&lt;/a&gt; for each element and perform constant time lookup. This assumes you have consistent normalization for each data source.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-23T12:18:19.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22513" PostTypeId="1" AcceptedAnswerId="22515" CreationDate="2017-08-23T12:40:16.087" Score="0" ViewCount="150" Body="&lt;p&gt;I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it worked perfectly. But, when I started installing Hadoop and set the &lt;code&gt;JAVA_HOME&lt;/code&gt; environment variable HDFS worked, but spark break down and shows: &lt;code&gt;Files was unexpected at this time&lt;/code&gt;. when I launched it. When I deleted &lt;code&gt;JAVA_HOME&lt;/code&gt;, Spark worked again but it's not the case for HDFS. What should I do in this case?&lt;/p&gt;&#xA;" OwnerUserId="38361" LastActivityDate="2017-08-23T13:49:16.773" Title="Install Spark and Hadoop in the same machine" Tags="|apache-spark|apache-hadoop|pyspark|" AnswerCount="1" CommentCount="0" ClosedDate="2017-08-23T14:39:42.370" ContentLicense="CC BY-SA 3.0" />
  <row Id="22515" PostTypeId="2" ParentId="22513" CreationDate="2017-08-23T13:49:16.773" Score="1" Body="&lt;p&gt;The Hadoop stack is difficult to setup and people complain that you can't trust any answers to problems over 6-12 months old. I would recommend getting a pre-configured Hadoop/Spark setup from Cloudera or HortonWorks. Both have free community editions.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-23T13:49:16.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22516" PostTypeId="2" ParentId="6546" CreationDate="2017-08-23T13:55:37.140" Score="7" Body="&lt;p&gt;This also returns the average of the selected column:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql.functions import mean&#xA;df.select(mean(df(&amp;quot;ColumnName&amp;quot;))).show()&#xA;+----------------+&#xA;| avg(ColumnName)|&#xA;+----------------+&#xA;|230.522453845909|&#xA;+----------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38371" LastEditorUserId="69263" LastEditDate="2023-10-07T14:22:00.120" LastActivityDate="2023-10-07T14:22:00.120" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22519" PostTypeId="1" CreationDate="2017-08-23T15:26:13.507" Score="2" ViewCount="236" Body="&lt;p&gt;Running an unsupervised plot of my data, I noticed a hyperbolic ('boomerang') shape:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dim=2&#xA;&#xA;vectorizer = TfidfVectorizer(min_df=5, max_df = 0.4, stop_words = 'english')&#xA;train_tf_idf = vectorizer.fit_transform(bunch_train.data)&#xA;svd = TruncatedSVD(n_components=dim,random_state=42)&#xA;svd_train = svd.fit_transform(train_tf_idf)&#xA;svd_train = Normalizer().fit_transform(svd_train)&#xA;&#xA;labels = {&#xA;        0:'alt.atheism',&#xA;        1:'comp.graphics',&#xA;        2:'sci.med'}&#xA;y = np.vectorize(labels.get)(bunch_train.target)&#xA;&#xA;with plt.style.context('seaborn-whitegrid'):&#xA;    plt.figure(figsize=(12, 8))&#xA;    for lab, col in zip(('alt.atheism', 'comp.graphics', 'sci.med'),&#xA;                        ('blue', 'red', 'green')):&#xA;        plt.scatter(svd_train[y==lab, 0],&#xA;                    svd_train[y==lab, 1],&#xA;                    label=lab,&#xA;                    c=col)&#xA;&#xA;    plt.title('2D SVD on TF-IDF - 3-NewsGroups',size=28)&#xA;    plt.xlabel('PC1')&#xA;    plt.ylabel('PC2')&#xA;    plt.legend(loc='upper left',prop={'size': 20})&#xA;    plt.tight_layout()&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SxF1H.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SxF1H.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect it has something to do with the Normalizer - when deleting the following line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svd_train = Normalizer().fit_transform(svd_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the data plots like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/An0pM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/An0pM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38377" LastActivityDate="2017-08-23T16:07:25.093" Title="Why is my PCA boomerang-shaped when normalizing?" Tags="|pca|normalization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22520" PostTypeId="2" ParentId="22519" CreationDate="2017-08-23T16:07:25.093" Score="1" Body="&lt;p&gt;The problem here is that you are normalizing the output of SVD. It might be clear that this output does not need to be normalized if you consider the output of each step:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Vectorizer: [n_items x n_words] Matrix. Unnormalized.&lt;/li&gt;&#xA;&lt;li&gt;TFIDF: [n_items x n_words] Matrix with rows &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting&quot; rel=&quot;nofollow noreferrer&quot;&gt;normalized&lt;/a&gt; to unit Euclidean norm.&lt;/li&gt;&#xA;&lt;li&gt;SVD: [n_items x dim] Matrix. Columns correspond to each row's loading onto two orthonormal unit vectors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In short, the output of the third step (SVD) is meaningful as-is and normalization obscures that meaning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to why the normalized output appears curved? The math to show this seems a bit involved but my intuition is that normalization is mapping the row weightings onto the &lt;a href=&quot;https://en.wikipedia.org/wiki/Singular_value_decomposition#Geometric_meaning&quot; rel=&quot;nofollow noreferrer&quot;&gt;ellipsoid manifold&lt;/a&gt; used in SVD.&lt;/p&gt;&#xA;" OwnerUserId="38379" LastActivityDate="2017-08-23T16:07:25.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22521" PostTypeId="2" ParentId="22481" CreationDate="2017-08-23T17:25:49.367" Score="0" Body="&lt;p&gt;You're dataset seems fairly small for a recommendation system so I am not sure how these approaches with scale but you are looking to solve a collaborative filtering problem which is essentially a sparse matrix completion. I popular and effective method for doing this is &lt;a href=&quot;https://web.stanford.edu/~hastie/Papers/mazumder10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;softImpute&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further resources here is a survey paper on &lt;a href=&quot;https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/11824/11581&quot; rel=&quot;nofollow noreferrer&quot;&gt;Top-N Recommender Systems via Matrix Completion&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-08-23T17:25:49.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22522" PostTypeId="1" AcceptedAnswerId="22525" CreationDate="2017-08-23T17:48:12.523" Score="6" ViewCount="815" Body="&lt;p&gt;I'm wondering about the benefits of advanced activation layers such as LeakyReLU, Parametric ReLU, and Exponential Linear Unit (ELU). What are the differences between them and how do they benefit training? &lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2020-08-07T03:15:24.627" Title="Advanced Activation Layers in Deep Neural Networks" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22523" PostTypeId="1" AcceptedAnswerId="22524" CreationDate="2017-08-23T18:12:08.653" Score="1" ViewCount="69" Body="&lt;p&gt;Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?&lt;/p&gt;&#xA;" OwnerUserId="38385" LastActivityDate="2017-08-23T22:10:31.153" Title="Is there any measure to find how much classifiers are confident?" Tags="|machine-learning|classification|data-mining|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22524" PostTypeId="2" ParentId="22523" CreationDate="2017-08-23T18:46:10.070" Score="2" Body="&lt;p&gt;There are many measures which implicitly take into account the confidence of a prediction. One very common one is &lt;a href=&quot;https://www.kaggle.com/wiki/LogLoss&quot; rel=&quot;nofollow noreferrer&quot;&gt;Log Loss&lt;/a&gt; (also called &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross_entropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cross Entropy&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any &lt;a href=&quot;https://en.wikipedia.org/wiki/Scoring_rule#ProperScoringRules&quot; rel=&quot;nofollow noreferrer&quot;&gt;proper scoring rule&lt;/a&gt; meets this criteria when applied to a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some others examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Self-information&quot; rel=&quot;nofollow noreferrer&quot;&gt;Surprisal&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Brier_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;Brier score&lt;/a&gt; (as mentioned by darXider in his comment)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://link.springer.com/article/10.1007/s11238-007-9067-x&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spherical Scoring Rule&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://web0.msci.memphis.edu/~randall/preprints/Pool3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logarithmic Scoring Rule&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And an infinite number of other, related functions.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-08-23T22:10:31.153" LastActivityDate="2017-08-23T22:10:31.153" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22525" PostTypeId="2" ParentId="22522" CreationDate="2017-08-23T18:47:06.877" Score="10" Body="&lt;h1&gt;ReLU&lt;/h1&gt;&#xA;&lt;p&gt;Simply rectifies the input, meaning positive inputs are retained but negatives give an output of zero. &lt;a href=&quot;http://invibe.net/biblio_database_dyva/woda/data/att/6525.file.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(Hahnloser et al. 2010)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;f(x) = max(0,x)&#xA;$$&lt;/span&gt;&#xA;Pros:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Eliminates the vanishing/exploding gradient problem.&lt;/strong&gt; (true for all following as well)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Sparse activation.&lt;/strong&gt; (true for all following as well)&lt;/li&gt;&#xA;&lt;li&gt;Noise-robust deactivation state (i.e. does not attempt to encode the degree of absence).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dying ReLU problem (many neurons end up in a state where they are inactive for most or all inputs).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Not differentiable.&lt;/strong&gt; (true for all following as well)&lt;/li&gt;&#xA;&lt;li&gt;No negative values means mean unit activation is often far from zero. This slows down learning.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1&gt;Leaky ReLUs&lt;/h1&gt;&#xA;&lt;p&gt;Adds a small coefficient (&lt;span class=&quot;math-container&quot;&gt;$&amp;lt;1$&lt;/span&gt;) for negative values. &lt;a href=&quot;http://web.stanford.edu/%7Eawni/papers/relu_hybrid_icml2013_final.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(Maas, Hannun, &amp;amp; Ng 2013)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;f(x) = \begin{cases}&#xA;x &amp;amp; \text{if } x \geq 0 \\&#xA;0.1 x &amp;amp; \text{otherwise}&#xA;\end{cases}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Pros:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Alleviates dying ReLU problem.&lt;/strong&gt; (true for all following)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Negative activations push mean unit activation closer to zero and thus speeds up learning.&lt;/strong&gt; (true for all following)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1&gt;PReLUs&lt;/h1&gt;&#xA;&lt;p&gt;Just like Leaky ReLUs but with a learnable coefficient. (Note that in the below equation a different &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; can be learned for different channels.) &lt;a href=&quot;https://arxiv.org/pdf/1502.01852.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(He et al. 2015)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;f(x) = \begin{cases}&#xA;x &amp;amp; \text{if } x \geq 0 \\&#xA;a x &amp;amp; \text{otherwise}&#xA;\end{cases}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Pros:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Improved performance (lower error rate on benchmark tasks) compared to Leaky ReLUs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Deactivation state is not noise-robust (i.e. noise in deactivation results in different levels of absence).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h1&gt;ELUs&lt;/h1&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;f(x) = \begin{cases}&#xA;x &amp;amp; \text{if } x \geq 0 \\&#xA;\alpha(exp(x)-1) &amp;amp; \text{otherwise}&#xA;\end{cases}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Replaces the small linear gradient of Leaky ReLUs and PReLUs with a vanishing gradient. &lt;a href=&quot;https://arxiv.org/pdf/1511.07289.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(Clevert, Unterthiner, Hochreiter 2016)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Pros:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Improved performance (lower error and faster learning) compared to ReLUs.&lt;/li&gt;&#xA;&lt;li&gt;Deactivation state is noise-robust.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="38379" LastEditorUserId="46749" LastEditDate="2020-08-07T03:15:24.627" LastActivityDate="2020-08-07T03:15:24.627" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22526" PostTypeId="2" ParentId="22523" CreationDate="2017-08-23T19:19:52.317" Score="0" Body="&lt;p&gt;In addition to comparing the log probabilities as Thomas had suggested, you can run cross validation and compare the means and standard deviations of your models' errors. Then basic statistics can be used to compare these distributions. &lt;/p&gt;&#xA;" OwnerUserId="33980" LastActivityDate="2017-08-23T19:19:52.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22529" PostTypeId="2" ParentId="19704" CreationDate="2017-08-23T21:33:27.153" Score="0" Body="&lt;p&gt;You can either work with a neural network and add chunks of data to its input layer. This could work, and it's taking the time dependency in some sort of way but not in the cleanest way (in my opinion). If it is a time-series, what I would do is use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;recurrent neural network&lt;/a&gt;, either a &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM&lt;/a&gt;, or a &lt;a href=&quot;https://en.wikipedia.org/wiki/Gated_recurrent_unit&quot; rel=&quot;nofollow noreferrer&quot;&gt;GRU&lt;/a&gt;, even a Simple RNN could work. In the end, a simple RNN is just a Neural network with layers for each timestep, meaning that if you unfold the recurrence of the RNN it will produce a normal feedforward neural network with more than one hidden layer (depending on the number of timesteps). Also, remember that in a neural network we do not know how the network came up with the decision (it's a black box) so either model should work just fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are going to use a RNN, I would train a neural network for each store for example, or make the inputs of all the stores into the input layer (depending on what you want to do). The output layer would be either one store's forecast (1 neuron in the output layer) or the forecast for each of the stores (n neurons in the output layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way of knowing what to do is to train, test, validate and compare approaches. Try both the NN and the RNN to see which one works better in this scenario, change number of layers, change number of inputs, number of outputs, etc. To my understanding there is no fixed solution when using neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps to get you going on your research.&lt;/p&gt;&#xA;" OwnerUserId="26700" LastActivityDate="2017-08-23T21:33:27.153" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22530" PostTypeId="1" CreationDate="2017-08-23T21:46:00.453" Score="1" ViewCount="79" Body="&lt;p&gt;I have an OCR that can read a given character. Now I want to build something that can take a scanned document and convert it into text. The problem is how do I segment each character? I know this can be done manually but how to automate it? I know that segmenting italics is different but since I want to go step by step I am considering normal fonts.&lt;/p&gt;&#xA;" OwnerUserId="28317" LastActivityDate="2017-08-23T21:46:00.453" Title="How to segment a scanned Page so that it can be fed into an OCR" Tags="|machine-learning|neural-network|ocr|" AnswerCount="0" CommentCount="1" ClosedDate="2017-09-17T09:45:14.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="22531" PostTypeId="2" ParentId="22488" CreationDate="2017-08-23T21:52:10.147" Score="3" Body="&lt;p&gt;I thought of posting as an answer according to the suggestion by @Seteve Kallestad in case of someone come across this issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is: The shape of the data must be the same when inverting the scale as when it was originally scaled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was making an error in my data preparation. &#xA;Initially, when I scaled the data my test set had a shape of &lt;code&gt;(168,39)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it mentions in the error &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;shapes (168,39) (41,) (168,39)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I had &lt;code&gt;41&lt;/code&gt; columns and it should have been &lt;code&gt;39&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is shape of &lt;code&gt;yhat&lt;/code&gt; is &lt;code&gt;(168,1)&lt;/code&gt; and shape of &lt;code&gt;test_X[:, 1:]&lt;/code&gt; is &lt;code&gt;168,38&lt;/code&gt; as it starts from column number 1. When they are concatenated it becomes &lt;code&gt;(168,39)&lt;/code&gt; which is exactly how it was when it was &lt;code&gt;scaled&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastActivityDate="2017-08-23T21:52:10.147" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22532" PostTypeId="2" ParentId="20586" CreationDate="2017-08-24T01:45:12.177" Score="1" Body="&lt;p&gt;Foremost, in my opinion categorical data should not be converted to continuous format at all. This is statistically incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, if a categorical data say &lt;code&gt;Sex&lt;/code&gt; with two levels &lt;code&gt;Male&lt;/code&gt;, &lt;code&gt;Female&lt;/code&gt; exists then it can be written as &lt;code&gt;1&lt;/code&gt;,&lt;code&gt;2&lt;/code&gt;, but, it should be specified that these numbers are to be treated as &lt;code&gt;nominal&lt;/code&gt; and not continuous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Continuing further, one method of detecting outliers in categorical data is to calculate the &lt;code&gt;frequency&lt;/code&gt; of the data points and then average each such obtained frequency to the total number of data points. The data points with the minimum frequencies can be considered as outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find this &lt;a href=&quot;https://link.springer.com/article/10.1007/s11047-015-9489-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; and this &lt;a href=&quot;https://stackoverflow.com/questions/32459197/attribute-value-frequency-in-r-outliers-in-categorical-variables&quot;&gt;SO post&lt;/a&gt; implementation in R interesting.&lt;/p&gt;&#xA;" OwnerUserId="11061" LastEditorUserId="29575" LastEditDate="2017-08-24T01:52:26.123" LastActivityDate="2017-08-24T01:52:26.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22534" PostTypeId="2" ParentId="326" CreationDate="2017-08-24T03:38:39.007" Score="2" Body="&lt;p&gt;I prefer Python over R because Python is a complete programming language so I can do end to end machine learning tasks such as gather data using a HTTP server written in Python, perform advanced ML tasks and then publish the results online. This can all be done in Python. I actually found R to be harder to learn and the payoffs for learning Python are much greater because it can be used for pretty much any programming task. &lt;/p&gt;&#xA;" OwnerUserId="38400" LastActivityDate="2017-08-24T03:38:39.007" CommentCount="1" CommunityOwnedDate="2017-08-24T03:38:39.007" ContentLicense="CC BY-SA 3.0" />
  <row Id="22536" PostTypeId="1" AcceptedAnswerId="22948" CreationDate="2017-08-24T05:53:34.230" Score="2" ViewCount="512" Body="&lt;p&gt;This question is related to &lt;a href=&quot;https://datascience.stackexchange.com/questions/424/how-to-grow-a-list-of-related-words-based-on-initial-keywords&quot;&gt;&quot;How to grow a list of related words based on initial keywords?&quot;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the previous question they attempt to get similar words to a given word. However, I am interested in knowing the possibility of doing it to &lt;strong&gt;sentences&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm not familiar with this area, my questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do this with sentences (without just considering words)?&#xA;What &lt;strong&gt;tools&lt;/strong&gt; are available for it? &lt;/p&gt;&#xA;" OwnerUserId="38395" LastEditorUserId="38395" LastEditDate="2017-08-25T01:33:51.063" LastActivityDate="2020-08-07T09:29:07.483" Title="Detect related sentences" Tags="|data-mining|nlp|text-mining|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22537" PostTypeId="1" AcceptedAnswerId="26062" CreationDate="2017-08-24T07:24:13.317" Score="2" ViewCount="1817" Body="&lt;p&gt;As far as I know, there is practically no limit on the number of dimensions of input feature for LSTM. And it apparently can learn the sequence of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is does LSTM by nature, also take previous output values besides feature vector as new feature element for the next output values? i.e. If we have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[t]  -&amp;gt; [x1, ..., xn] [y_t]&#xA;[t+1] -&amp;gt; [x1, ..., xn] [y_(t+1)]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it necessary to manipulate data like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[t]  -&amp;gt; [x1, ..., xn, y_(t-1)] [y_t]&#xA;[t+1] -&amp;gt; [x1, ..., xn, y_t]    [y_(t+1)]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or LSTM is already handling this for us?&lt;/p&gt;&#xA;" OwnerUserId="30043" LastActivityDate="2019-02-27T18:08:32.307" Title="LSTM: Taking previous output values as feature" Tags="|lstm|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22540" PostTypeId="2" ParentId="9131" CreationDate="2017-08-24T10:03:27.193" Score="0" Body="&lt;p&gt;I would read up on decay functions. With one of these you can choose exactly to what extent things in the past contribute:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Exponential_decay&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Exponential_decay&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2017-08-24T10:03:27.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22542" PostTypeId="2" ParentId="22537" CreationDate="2017-08-24T10:27:46.467" Score="1" Body="&lt;p&gt;LSTMs handle this for you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can search for some tutorial on the LSTMs that explains how they works. One that I like is &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="33466" LastActivityDate="2017-08-24T10:27:46.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22544" PostTypeId="1" AcceptedAnswerId="22555" CreationDate="2017-08-24T12:44:10.537" Score="1" ViewCount="227" Body="&lt;p&gt;I am trying to display a date in my result after running the below program in SAS. It runs properly but in the sas data table under DOB column I don't get anything except a period &lt;code&gt;.&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is my code what am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data sample; &#xA;Input ID name $ Dob;&#xA;Format DOB mmddyy10. ; &#xA;datalines;&#xA;1 abc 22jan1996&#xA;2 xyz 25aug1996&#xA;;&#xA;run;&#xA;Proc print data = sample;&#xA;run; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also I would like to know what does a period means at the end of this line before the semicolon:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Format DOB mmddyy10. ;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38426" LastEditorUserId="8878" LastEditDate="2017-08-24T14:14:40.253" LastActivityDate="2020-07-26T04:12:53.963" Title="Displaying date in SAS" Tags="|sas|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22545" PostTypeId="1" AcceptedAnswerId="22619" CreationDate="2017-08-24T13:26:54.070" Score="5" ViewCount="7703" Body="&lt;p&gt;We would like to implement rule engine in one of our products, but when I read on the Internet, many are suggesting that rule engines are an old way of doing things and it will be very hard to manage as we add more rules into it. To be successful in implementing a rule based system we need to know all the rules beforehand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, everyone seems to be suggesting that we use Machine Learning algorithms to predict the outcome, but I am new to Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my Question is, should I learn Machine Learning or go on implementing Drools as rule engine in my application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or can you suggest any online resources achieving the same using Machine Learning?&lt;/p&gt;&#xA;" OwnerUserId="38429" LastEditorUserId="29575" LastEditDate="2017-08-24T13:49:12.730" LastActivityDate="2017-08-27T07:05:04.067" Title="Rule engine vs machine learning" Tags="|machine-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22546" PostTypeId="2" ParentId="21650" CreationDate="2017-08-24T13:44:16.090" Score="22" Body="&lt;p&gt;We love the normal form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which &lt;strong&gt;Transformation&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes physical (biological, economic, whatever) sense, for example in terms of limiting behaviour as values get very small or very large?&#xA;This question often leads to the use of logarithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we keep dimensions and units simple and convenient? If possible, we prefer measurement scales that are easy to think about. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cube root of a volume and the square root of an area both have the dimensions of length, so far from complicating matters, such transformations may simplify them. Reciprocals usually have simple units, as mentioned earlier. Often, however, somewhat complicated units are a sacrifice that has to be made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;When to Use What&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most useful transformations in introductory data analysis are the reciprocal, logarithm, cube root, square root, and square. In what follows, even when it is not emphasised, it is supposed that transformations are used only over ranges on which they yield (finite) real numbers as results.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Reciprocal&lt;/strong&gt;: The reciprocal, x to 1/x, with its sibling the negative reciprocal, x to -1/x, is a very strong transformation with a drastic effect on distribution shape. It can not be applied to zero values.  Although it can be applied to negative values, it is not useful unless all values are positive. The reciprocal of a ratio may often be interpreted as easily as the ratio itself:    Example:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;population density (people per unit area) becomes area per person&lt;/li&gt;&#xA;&lt;li&gt;persons per doctor becomes doctors per person &lt;/li&gt;&#xA;&lt;li&gt;rates of erosion become time to erode a unit depth&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(In practice, we might want to multiply or divide the results of taking the reciprocal by some constant, such as 1000 or 10000, to get numbers that are easy to manage, but that itself has no effect on skewness or linearity.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reciprocal reverses order among values of the same sign: largest becomes smallest, etc. The negative reciprocal preserves order among values of the same sign.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Logarithm&lt;/strong&gt;: The logarithm, x log&lt;sub&gt;10&lt;/sub&gt; x, or x log &lt;sub&gt;e x or ln x, or x log 2&lt;/sub&gt; x, is a strong    transformation with a major effect on distribution shape. It is    commonly used for reducing right skewness and is   often appropriate    for measured variables. It can not be applied to zero or negative    values. One unit on a logarithmic scale means a multiplication by the    base of logarithms being used. Exponential growth or decline.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$y = a exp (bx)$&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;is made linear by&#xA;- $ln y = ln a + bx$&#xA;       so that the response variable y should be logged. (Here exp() means    raising to the power e, approximately 2.71828, that is the base of    natural logarithms).&#xA;       An aside on this exponential growth or decline equation: &#xA;       $x = 0$, and $y = a exp(0) = a$ so that a is the amount or count when    x = 0. If a and b &gt; 0, then y grows at a faster and faster rate (e.g.    compound interest or unchecked population growth), whereas if a &gt; 0    and b &amp;lt; 0, y declines at a slower and slower rate (e.g. radioactive    decay).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Power functions&lt;/strong&gt; :&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$y = ax^b$ are made linear by $log y = log a + b log x$ &#xA;so that both variables y and x should be logged. An aside on such power&lt;br&gt;&#xA;functions: put $x = 0$, and for $b &amp;gt; 0$,&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$y = ax^b = 0$    so the power function for positive b goes through the origin, which    often makes physical or biological or&#xA;economic sense. Think:    does    zero for x imply zero for y? This&lt;br&gt;&#xA;kind of power function is a    shape    that fits many data sets&lt;br&gt;&#xA;rather well.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Consider ratios y = p / q where p and q are both positive in       practice.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Examples are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Males / Females  &lt;/li&gt;&#xA;&lt;li&gt;Dependants / Workers&lt;/li&gt;&#xA;&lt;li&gt;Downstream length / Downvalley length&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then y is somewhere between 0 and infinity, or in the last case, between 1 and infinity. If p = q, then y = 1. Such definitions often lead to skewed data, because there is a clear lower limit and no clear upper limit. The logarithm, however, namely&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;log y = log p / q = log p - log q,&#xA;is somewhere between -infinity and infinity and p = q means that log y = 0. Hence the logarithm of such a ratio is likely to be more symmetrically&#xA;distributed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cube root&lt;/strong&gt;: The cube root, x &lt;sup&gt;1/3&lt;/sup&gt;. This is a fairly strong transformation with a substantial effect on distribution shape: it is weaker than the logarithm. It is also used for reducing right skewness, and has the    advantage that it can be applied to zero and negative values.  Note that the cube root of a volume has the units of a length. It is commonly applied to rainfall data.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Applicability to negative values requires a special note. Consider&lt;br&gt;&#xA;(2)(2)(2) = 8 and (-2)(-2)(-2) = -8. These examples show that the&lt;br&gt;&#xA;cube root of a negative number has negative sign and the same&lt;br&gt;&#xA;absolute value as the cube root of the equivalent positive number. A &#xA;similar property is possessed by any other root whose power is the&lt;br&gt;&#xA;reciprocal of an odd positive integer (powers 1/3, 1/5, 1/7, etc.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This property is a little delicate. For example, change the power just a&#xA;smidgen from 1/3, and we can no longer define the result as a product of&#xA;precisely three terms. However, the property is there to be exploited if&#xA;useful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Square root&lt;/strong&gt;:The square root, x to $x^(1/2)$ = sqrt(x), is a transformation with a moderate effect on distribution shape: it is&#xA;weaker than the logarithm&#xA;   and the cube root. It is also used for reducing right skewness, and also&#xA;   has the advantage that it can be applied to zero values. Note that the&#xA;   square root of an area has the units of a length. It is commonly applied&#xA;   to counted data, especially if the values are mostly rather small.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Square&lt;/strong&gt;: The square, x to $x^2$, has a moderate effect on distribution shape and it could be used to reduce left skewness. In&lt;br&gt;&#xA;practice, the main reason for   using it is to fit a response by a&lt;br&gt;&#xA;quadratic function $y = a + b x + cx^2$. Quadratics have a turning&lt;br&gt;&#xA;point, either a maximum or a minimum, although the turning point in a&#xA;function fitted to data might be far beyond the limits of the&lt;br&gt;&#xA;observations. The distance of a body from an origin is a quadratic if&#xA;that body is moving under constant acceleration, which gives a very&lt;br&gt;&#xA;clear physical justification for using a quadratic. Otherwise&lt;br&gt;&#xA;quadratics are typically used solely because they can mimic a&lt;br&gt;&#xA;relationship within the data region. Outside that region they may&lt;br&gt;&#xA;behave very poorly, because they take on arbitrarily large values for&#xA;extreme values of x, and unless the intercept a is constrained to be &#xA;0, they may behave unrealistically close to the origin.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Squaring usually makes sense only if the variable concerned is zero&#xA;or positive, given that $(-x)^2$ and $x^2$ are identical.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="29392" LastEditorUserId="40853" LastEditDate="2017-11-23T20:46:53.420" LastActivityDate="2017-11-23T20:46:53.420" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22551" PostTypeId="1" AcceptedAnswerId="22552" CreationDate="2017-08-24T16:15:20.330" Score="1" ViewCount="543" Body="&lt;p&gt;Suppose I have a data frame called quoteDF&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;quotesDF&#xA;&#xA;              volume  shares&#xA;2017-01-03   2934300     100&#xA;2017-01-04   3381400     120&#xA;2017-01-05   2682300     140  &#xA;2017-01-06   2945500     160&#xA;2017-01-09   3189900     180&#xA;2017-01-10   4118700     200&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I do,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; quotesDF.loc[1, 'shares']&#xA;120&#xA;&amp;gt; quotesDF.loc[1, ['shares']]&#xA;shares 120&#xA;Name: 1, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why the first one retunrs 120, the second one retures shares 120? In my mind, they are the same thing, except I put the second one in a vector. But, the first one is a vector stands by itself. It's just that I didn't put the square bracket on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why Python give me such a confusing time?&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2017-08-24T17:29:25.453" Title="In Python, why subsetting with or without square bracket is different?" Tags="|python|dataset|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ClosedDate="2017-08-24T22:03:43.953" ContentLicense="CC BY-SA 3.0" />
  <row Id="22552" PostTypeId="2" ParentId="22551" CreationDate="2017-08-24T17:29:25.453" Score="2" Body="&lt;p&gt;Assuming, you have a &lt;code&gt;pandas&lt;/code&gt; dataframe, &lt;code&gt;.loc&lt;/code&gt; is strictly label based. Since you're using &lt;code&gt;[]&lt;/code&gt; it accesses the column you're specifying inside the brackets and that is the reason you're getting &lt;code&gt;shares 120&lt;/code&gt;. Read &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.15.2/indexing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; for better explanation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is another &lt;a href=&quot;https://stackoverflow.com/questions/33417991/pandas-why-are-double-brackets-needed-to-select-column-after-boolean-indexing&quot;&gt;link&lt;/a&gt; that has answers similar to your question. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastActivityDate="2017-08-24T17:29:25.453" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22553" PostTypeId="1" CreationDate="2017-08-24T18:06:58.893" Score="0" ViewCount="374" Body="&lt;p&gt;I am new to machine learning, so was hoping you might have some insights one what classification algorithm might be best suited for my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set of hundreds of predictions from each of ~100 forecasters. Each prediction is binary, and the outcome variable is binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically let's say I have data on 200 events. For each event, I have 1 prediction from each of 100 forecasters. Each of the 100 forecasters predicts each of the 200 events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am hoping for an output of some formula, for which I input the predictions of the forecasters for that particular outcome, and it renders an output for the probability of that outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main interest is in predictive accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestions for what algorithms might suit this problem?&lt;/p&gt;&#xA;" OwnerUserId="38438" LastActivityDate="2017-08-24T19:10:21.380" Title="Suggestions for binary classification algorithm" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22554" PostTypeId="2" ParentId="22553" CreationDate="2017-08-24T18:45:18.027" Score="2" Body="&lt;p&gt;I would say support vector machine, support vector &lt;a href=&quot;http://users.ecs.soton.ac.uk/srg/publications/pdf/SVM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;classifier&lt;/a&gt;. This will allow you to have as many predictors as you want. Also, you can improve your accuracy by tuning your model parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is another nice link for &lt;a href=&quot;http://dataaspirant.com/2017/01/13/support-vector-machine-algorithm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;support vector classification&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you did not mention any language, I would use &lt;code&gt;python&lt;/code&gt; as there are so much online support and you resources available online. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also check &lt;a href=&quot;https://machinelearningmastery.com/develop-first-xgboost-model-python-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;xgboost classifier&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us know how did it go. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-08-24T19:10:21.380" LastActivityDate="2017-08-24T19:10:21.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22555" PostTypeId="2" ParentId="22544" CreationDate="2017-08-24T19:23:15.943" Score="2" Body="&lt;p&gt;Try the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data sample; &#xA;Input ID name $ Dob;&#xA;informat DOB date9. ; &#xA;format DOB mmddyy10. ; &#xA;datalines;&#xA;1 abc 22jan1996&#xA;2 xyz 25aug1996&#xA;;&#xA;run;&#xA;Proc print data = sample;&#xA;run; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have added the informat line. The problem with the original code is that you have not told SAS what is the format that DOB will come in. In this case: 22jan1996 is of date9. format, so I added the informat telling SAS that the data will come in this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The format DOB line tells SAS to display the data as mmddyy10. which makes 22jan1996 look like 01/22/1996.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, the &lt;code&gt;.&lt;/code&gt; before the &lt;code&gt;;&lt;/code&gt; in the format/informat lines specifies that it is a format type. There are character formats: $8. or numeric formats Best12. etc...&lt;/p&gt;&#xA;" OwnerUserId="38439" LastEditorUserId="29575" LastEditDate="2017-08-24T22:03:21.050" LastActivityDate="2017-08-24T22:03:21.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22556" PostTypeId="1" CreationDate="2017-08-24T19:33:48.480" Score="5" ViewCount="1214" Body="&lt;p&gt;Let's say that I have a directed graph, and for each node I want to have some measure of whether it is more of an &quot;upstream&quot; node (that lies at the beginning of many paths), or a &quot;downstream&quot; node (to which many paths converge). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, in a sequence A-&gt;B-&gt;C I want A to have the highest value, and C the lowest value, while B would be &quot;in between&quot;. In a perfect cycle A-&gt;B-&gt;C-&gt;A I want all nodes to have the same &quot;in between&quot; value. The metrics should tolerate a mix of cycles and acyclic elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, it would also give more extreme values to nodes from longer acyclic paths: higher values to nodes with larger estuaries (the 'mother of all' node), and lower values to nodes with larger watersheds (the 'all roads lead to Rome' node that integrates all flows).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And also ideally it should generalize to weighted graphs. And, ideally, in a way that does not depend on the scaling factor for weights (so if you multiply all weights in the graph by 2, the metrics for each node shouldn't change, as the topology clearly didn't change with this scaling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a non-weighted graph I can come with a metrics like that: find all predecessors of a node, find all children of a node, then divide the number of children to the sum of children and ancestors; this value would be 1 for any point of origin; would give 0 for dead-ends, and would give 0.5 for cycles. So it's mostly OK. What I don't like about this metrics is that 1) it does not care about the length of the path, 2) it will be computationally slow, 3) I don't know how to generalize it to weighted graphs in a scale-invariant way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was wondering whether there's a known metrics with approximately these properties that was described and studied before. It feels like a logical thing to calculate that many people would use when analyzing social networks, for example; so it feels like it should have a name and published algorithms. Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I think it's fair to say that the pagerank metrics has many of the properties I described (with the values reversed): sinks are high, sinks with larger watersheds are higher, nodes of origin are low, cycles tend to have &quot;in-between&quot; values, and the algorithm clearly supports weighted graphs. The part it does not care about is whether a node of origin has a large estuary or no estuary at all. Now I'm wondering whether I actually need 2 metrics: one page-rank, for watersheds, and a different one for estuaries. Like a weighted share of nodes visited by random walks initiated in the node of interest, or something like that. Or are there simpler metrics?&lt;/p&gt;&#xA;" OwnerUserId="38440" LastEditorUserId="38440" LastEditDate="2017-09-01T15:52:53.040" LastActivityDate="2018-01-12T16:31:48.957" Title="In a directed graph, how to measure whether a node is more &quot;upstream&quot; or &quot;downstream&quot;?" Tags="|graphs|social-network-analysis|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22559" PostTypeId="2" ParentId="13894" CreationDate="2017-08-24T20:26:45.367" Score="17" Body="&lt;p&gt;To get a confusion matrix from the test data you should go througt two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Make predictions for the test data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For example, use &lt;code&gt;model.predict_generator&lt;/code&gt; to predict the first 2000 probabilities from the test generator. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;generator = datagen.flow_from_directory(&#xA;        'data/test',&#xA;        target_size=(150, 150),&#xA;        batch_size=16,&#xA;        class_mode=None,  # only data, no labels&#xA;        shuffle=False)  # keep data in same order as labels&#xA;&#xA;probabilities = model.predict_generator(generator, 2000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Compute the confusion matrix based on the label predictions&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import confusion_matrix&#xA;&#xA;y_true = np.array([0] * 1000 + [1] * 1000)&#xA;y_pred = probabilities &amp;gt; 0.5&#xA;&#xA;confusion_matrix(y_true, y_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional note on test and validation data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="20372" LastEditDate="2018-03-11T12:19:10.400" LastActivityDate="2018-03-11T12:19:10.400" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22560" PostTypeId="1" CreationDate="2017-08-24T21:47:14.053" Score="3" ViewCount="429" Body="&lt;p&gt;I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (&lt;a href=&quot;https://gist.github.com/ychervonyi/d7d119492e09e2b89bcdae98bdb8eb6a&quot; rel=&quot;nofollow noreferrer&quot;&gt;2 layers&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that I cannot underfit or overfit the data (see the plot below). I understand why I cannot underfit: it might happen if data is easily separable, but why I cannot overfit? The dataset capacity is 600 (# of samples (150) times # of features (4)), so I should be able to overfit using a network with a capacity bigger than that. I am trying to use a multilayer perceptron with a total # of parameters ranging from 15 to ~32000, but neither under-, nor over-fitting happens. What is going on? Maybe overfitting does not happen for the same reason, because the data is easily separable? Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rlcWQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rlcWQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38446" LastActivityDate="2017-08-27T07:14:24.473" Title="Cannot underfit/overfit on the IRIS dataset" Tags="|overfitting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22564" PostTypeId="1" AcceptedAnswerId="22566" CreationDate="2017-08-25T06:32:37.883" Score="1" ViewCount="207" Body="&lt;p&gt;We have a huge dataset with us that looks like below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Factor -|- ... -|- Rank1 -|- Rank2 -|- Calls&#xA;A11-----|- ... -|-0.1234--|-3.2345--|- Cat A&#xA;A11-----|- ... -|-1.1234--|-0.2345--|- Cat B&#xA;A12-----|- ... -|-2.1234--|-3.2345--|- Cat C&#xA;A12-----|- ... -|-2.1234--|-3.2345--|- Cat C&#xA;...&#xA;A13-----|- ... -|-0.1234--|-3.2345--|- Cat A&#xA;A13-----|- ... -|-3.1234--|-0.1345--|- Cat B&#xA;A13-----|- ... -|-2.1234--|-2.2345--|- Cat C&#xA;A14-----|- ... -|-4.1234--|-4.2345--|- Cat C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and we have about 10 million of such data points.&#xA;We also have a test set with about .2 million data points where we need to accurately call them out into different classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, we are trying a mix of K-means &amp;amp; Random Forest approach coded in (python2.7-sklearn) which gives us about 90% accuracy at classification but we wanna reach for more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested to apply some kind of deep learning approach to this and that's why I'm learning about TensorFlow. But every link I've gone though in deep learning talks (tf documentation + youtube videos) about CNN and image recognition (MNIST .. etc) only and I'm not finding any idea how to start solving this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for your suggestions and guidance on how to approach this kind of problem / What neurons to stack or How to build models for this kind of data? I'm willing to burn my share of the midnight oil on any links and suggestions offered on this but I need to know if I'm even thinking right by trying to solve it with DL or if there's any other approach for this kind of data which may work better, or to solve this what should I study or learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;b&gt;Edit 1:&lt;/b&gt;&#xA;So, I managed to go through the TensorFlow documentation for MNIST an perhaps I can see some correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the sense if I pass the rank1 &amp;amp; rank2 as input &lt;code&gt;np.array&lt;/code&gt; for the 2 neuron input layer and each call as a one_hot &lt;code&gt;np.array&lt;/code&gt;, where Cat A = [1,0,0],Cat B = [0,1,0],Cat C = [0,0,1] and put 3 neuron on the outermost layer each predicting one Cat(category) through a softmax function, will that work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if so, what should I use as internal/hidden layers? Can I use the other factors as well in the input layers? Do I need to convert them into numeric(int/float)? Should I pass these rank 1 and rank 2 values to the input layer just as it is or should I engineer them anyway beforehand?&lt;/p&gt;&#xA;" OwnerUserId="33530" LastEditorUserId="24000" LastEditDate="2017-08-25T07:16:17.247" LastActivityDate="2017-08-25T07:23:04.313" Title="What ML/DL approach better suits this problem?" Tags="|machine-learning|deep-learning|tensorflow|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22565" PostTypeId="1" CreationDate="2017-08-25T06:38:51.000" Score="3" ViewCount="906" Body="&lt;p&gt;I am trying to create a model that would capture the usual behavior of a user: i.e., create a model for user profiling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do have 2 million rows of data indicating normal behavior of a user and want to define a model that can detect any variation from the normal behavior of the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am contemplating of using one class svm / isolation forest trained on normal behavior. I would like to get an opinion whether that looks as a good direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or should I venture into deep learning or unsupervised clustering based model and gather data for abnormal behavior of a user as well to possibly use supervised models.&lt;/p&gt;&#xA;" OwnerUserId="28832" LastEditorUserId="40853" LastEditDate="2018-01-25T09:03:26.767" LastActivityDate="2018-06-24T16:53:25.093" Title="ML algorithms for defining NORMAL user behavior" Tags="|deep-learning|clustering|unsupervised-learning|supervised-learning|" AnswerCount="3" CommentCount="0" ClosedDate="2018-06-24T21:15:56.797" ContentLicense="CC BY-SA 3.0" />
  <row Id="22566" PostTypeId="2" ParentId="22564" CreationDate="2017-08-25T06:56:31.757" Score="2" Body="&lt;p&gt;Yes, you can use DL network to solve this problem. It is a easy multi-classes classification task. And just a full connected network will do this work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To construct the network I recommend using &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; which is easy to use. And before training the network, it is better to preprocess the data(standardizing the numeric column, and embedding the string column).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the data normalisation you can reference this blog: &lt;a href=&quot;https://visualstudiomagazine.com/Articles/2013/07/01/Neural-Network-Data-Normalization-and-Encoding.aspx?Page=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Network Data Normalization and Encoding&lt;/a&gt;, that show the basic methods and codes.&lt;/p&gt;&#xA;" OwnerUserId="38419" LastEditorUserId="38419" LastEditDate="2017-08-25T07:23:04.313" LastActivityDate="2017-08-25T07:23:04.313" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22568" PostTypeId="1" AcceptedAnswerId="22612" CreationDate="2017-08-25T08:43:58.890" Score="2" ViewCount="727" Body="&lt;p&gt;I have a problem that I have been working on with logistic regression which involves classifying clients based on their income and socio-demographic variables, all of which are categorical into those who have defaulted on their credit and those who haven't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have recently had tensorflow recommended as a package to use for neural networks, however on their website &lt;a href=&quot;https://tensorflow.rstudio.com/tutorial_mnist_beginners.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://tensorflow.rstudio.com/tutorial_mnist_beginners.html&lt;/a&gt; they discuss tensorflow in terms of image classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My concrete question is thus:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Can tensorflow be used for any type of classification problem, for example, a binary classification of observations based upon categorical variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers,&#xA;Steve&lt;/p&gt;&#xA;" OwnerUserId="38460" LastEditorUserId="38460" LastEditDate="2017-08-25T10:33:46.230" LastActivityDate="2017-08-30T19:10:17.017" Title="Using tensorflow for any type of dataset" Tags="|machine-learning|classification|tensorflow|multilabel-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22569" PostTypeId="2" ParentId="22565" CreationDate="2017-08-25T09:05:13.633" Score="0" Body="&lt;p&gt;I think it depends mostly on how &lt;strong&gt;difficult&lt;/strong&gt;/expensive/time-consuming it is to &lt;strong&gt;gather&lt;/strong&gt; data for abnormal behavior.It also depends on whether you'd expect this abnormal behavior to belong to one (or a &lt;strong&gt;fixed number&lt;/strong&gt; of) classes or it could be anything that deviates from normal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can afford to &lt;strong&gt;collect&lt;/strong&gt; labelled data and expect the abnormal behavior to be limited to a fixed number of classes, then the &lt;strong&gt;supervised&lt;/strong&gt; approach is very likely going to give you &lt;strong&gt;better&lt;/strong&gt; results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, if any of the above hypothesis do not hold, then I think the &lt;strong&gt;semi-supervised&lt;/strong&gt; approach (as you said, One Class SVM or similar techniques) is in my opinion a &lt;strong&gt;good way&lt;/strong&gt; to go.&lt;/p&gt;&#xA;" OwnerUserId="28685" LastActivityDate="2017-08-25T09:05:13.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22571" PostTypeId="1" CreationDate="2017-08-25T09:05:47.963" Score="1" ViewCount="1084" Body="&lt;p&gt;I am doing Andrew Ng's excellent new Deeplearning course. I have an issue with implementing back propagation in a one hidden layer network that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vRi0e.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vRi0e.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to derive the derivative of Z1 using the formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TKmtP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TKmtP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that g prime Z1 in this formula is the derivative of g(z) with respect to Z which I believe is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    sigmoid(Z) * (1-sigmoid(Z))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the function sigmoid generates the sigmoid function with respect to Z.&#xA;My final equation looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  dZ1 = np.dot(W2.T, dZ2) * (sigmoid(Z1)*(1-sigmoid(Z1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where Z1 is the output of the first (and only) hidden layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where am I going wrong?&lt;/p&gt;&#xA;" OwnerUserId="38306" LastEditorUserId="38306" LastEditDate="2017-08-25T09:19:24.507" LastActivityDate="2017-08-25T09:19:24.507" Title="Neural networks: generating g prime(z) in back propagation" Tags="|neural-network|backpropagation|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22572" PostTypeId="1" AcceptedAnswerId="22577" CreationDate="2017-08-25T10:08:50.260" Score="1" ViewCount="1305" Body="&lt;p&gt;I need to build semantic word embeddings representation of compound terms like &quot;electronic engineer&quot; or &quot;microsoft excel&quot;. One approach would be to use a standard pretrained model an average the words but, since I have a corpus  of my domain, is there a possible better approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be more precise:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data I have is a corpus of millions of documents. Each document is ~ half a page and contains these compound terms. However there may be compound terms not included in the corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="17484" LastEditDate="2017-09-04T23:05:12.407" LastActivityDate="2017-09-04T23:05:12.407" Title="How can I get semantic word embneddings for compound terms?" Tags="|machine-learning|word|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22574" PostTypeId="1" CreationDate="2017-08-25T11:45:29.727" Score="3" ViewCount="110" Body="&lt;p&gt;I'm studying the theory behind &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural nets&lt;/a&gt; and I wonder if there is any actual difference between using different lost/cost functions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I could use either &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_absolute_error&quot; rel=&quot;nofollow noreferrer&quot;&gt;MAE&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_squared_error&quot; rel=&quot;nofollow noreferrer&quot;&gt;MSE&lt;/a&gt; for backpropagating loss; does this decision have any actual effect on the model efficiency? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day both functions just calculate the difference between $y$ and $\hat y$ (albeit on a different scale). But for the optimizer, only the error tendency is what really matters, not the absolute difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this question is relevant for any other model evaluation.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="29575" LastEditDate="2017-08-27T17:24:23.073" LastActivityDate="2017-09-26T19:11:20.667" Title="What is the efficiency difference between different cost functions in case of neural networks?" Tags="|machine-learning|neural-network|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22575" PostTypeId="1" CreationDate="2017-08-25T11:52:15.187" Score="-2" ViewCount="1634" Body="&lt;p&gt;Say I have this data:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;User 1  User 2  Duration&#xA;5       2       7-11 months&#xA;5       5       Less than 1 month&#xA;5       5       4-6 months&#xA;5       5       4-6 months&#xA;5       4       4-6 months&#xA;5       5       2 years&#xA;4       6       1 year&#xA;4       4       4-6 months&#xA;4       8       7-11 months&#xA;4       4       2 years&#xA;3       3       Less than 1 month&#xA;3       3       7-11 months&#xA;3       3       1-3 months&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;To sum the variable, for example, if I would like to calculate &lt;code&gt;user1&lt;/code&gt; within 4-6 months, I did that. Now, I would like to display all the individual values like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;       4-6 months  7-11 months  1-3 months&#xA;User1    values      values       values&#xA;User2    values      values       values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How do I get this kind of output?&lt;/p&gt;&#xA;&lt;p&gt;Until now I have used this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df %&amp;gt;% group_by(Duration) %&amp;gt;% summarise(count=sum(User1, na.rm = TRUE)) %&amp;gt;% &#xA;filter(Duration==&amp;quot;4-6 months&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38466" LastEditorUserId="29169" LastEditDate="2021-02-06T18:33:19.217" LastActivityDate="2021-02-06T18:33:40.853" Title="how to sum a variable by group in r and display in a tabular form?" Tags="|r|rstudio|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22576" PostTypeId="2" ParentId="22575" CreationDate="2017-08-25T11:58:04.780" Score="0" Body="&lt;p&gt;Check out the &lt;code&gt;xtabs()&lt;/code&gt; function. &lt;a href=&quot;https://www.rdocumentation.org/packages/stats/versions/3.4.1/topics/xtabs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; are some examples.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="29169" LastEditDate="2021-02-06T18:33:40.853" LastActivityDate="2021-02-06T18:33:40.853" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22577" PostTypeId="2" ParentId="22572" CreationDate="2017-08-25T12:16:36.433" Score="2" Body="&lt;p&gt;If you want an exact answer, please provide a precise question i.e. define what data you have, and what you exactly wants. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This said, in a general manner, you need a dataset of texts that contain these compound terms. How to treat compound terms is a whole scientific field in itself, but since you're talking about semantic word embeddings, I suggest you take a look at the article &lt;a href=&quot;https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Distributed Representations of Words and Phrases&#xA;and their Compositionality&lt;/a&gt;. The same guys who introduced word2vec describe here a simple method to go from word representation to phrase representation, giving btw a way to merge compound terms in single terms. The words &quot;microsoft excel&quot; become &quot;microsoft_excel&quot; and get their own unique embedding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a python implementation for that, take a look at the &lt;a href=&quot;https://radimrehurek.com/gensim/models/phrases.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim.models.phrase&lt;/a&gt; class. This does the same work as presented in the previous article. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-08-25T12:16:36.433" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22581" PostTypeId="1" CreationDate="2017-08-25T12:46:09.867" Score="3" ViewCount="258" Body="&lt;p&gt;In the following dataset, the first 4 columns are predictor variables and the engine running index is the response variable.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;   O2 level | Cylinder pressure | Fuel Flow | Engine temp | Engine running index&#xA;       5              15               3           31                7&#xA;       2              31              18            1               88&#xA;       1              22              66            4               31&#xA;      ...            ...              ...          ...             ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Situation: An engineer comes with a similar data set and asks a question: &amp;quot;What should be the setting of my O2 level, cylinder pressure etc to get the best running index?&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;Now one could take a statistical approach and try figure out the answer, but I was looking to employ some machine learning techniques. The ones I've tried, i.e. Regression, are used more to predict what the Engine running index will be, rather than suggest what the settings should be, to get the best running index. The other one I was looking at was PCA, but not sure that it will give me an answer as well.&lt;/p&gt;&#xA;&lt;p&gt;Note: the possible max value for Engine running index is not known.&lt;/p&gt;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&lt;p&gt;Are there ML techniques to help me answer the engineer's question?&lt;/p&gt;&#xA;" OwnerUserId="38467" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-24T15:47:24.733" Title="Best ML technique to suggest predictor variables" Tags="|machine-learning|scikit-learn|regression|pca|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22582" PostTypeId="2" ParentId="12886" CreationDate="2017-08-25T12:59:30.007" Score="2" Body="&lt;p&gt;Checkout &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/weighted_cross_entropy_with_logits&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.nn.weighted_cross_entropy_with_logits()&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Computes a weighted cross entropy.&lt;/p&gt;&#xA;&lt;p&gt;This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This should let you do what you want.&lt;/p&gt;&#xA;" OwnerUserId="38470" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-08-25T15:14:40.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22584" PostTypeId="2" ParentId="22581" CreationDate="2017-08-25T13:42:37.047" Score="0" Body="&lt;p&gt;You should try to define what is 'best' Engine running index. Then try to cluster your data given that measure (Engine running index) so that you will put in one cluster all the 'best' Engine running index. &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2017-08-25T13:42:37.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22587" PostTypeId="1" CreationDate="2017-08-25T14:45:24.520" Score="7" ViewCount="3409" Body="&lt;p&gt;I have a large dataset (2M entries) of people, but many people have multiple entries in the database with slightly (or significantly) different identifying information. For example, I may have J. Doe and John Doe, or I may have John Doe with an accompanying email address and John Doe without an accompanying email address.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been looking at different clustering algorithms but nothing seems well-suited to what I'm doing, which is to aggregate the entries based on rules like the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;guess first and last names based on whether one of the names is written in all capitals&lt;/li&gt;&#xA;&lt;li&gt;aggregate &quot;J. Doe&quot; and &quot;J. Doe&quot; if email addresses match&lt;/li&gt;&#xA;&lt;li&gt;aggregate &quot;J. Doe&quot; into &quot;John Doe&quot; if no other people have first name starting with &quot;J&quot; and last name &quot;Doe&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With a smaller dataset this would be a relatively straightforward task to do just with some simple rules, but with the number of entries I have the aggregation tasks can get really slow and the logic gets pretty convoluted. My current solution (based around using the fulltext search function within my database to find similar entries, adding hashes based on those results, and then aggregating based on a mix of hashes and types of ambiguity) works, but every time I try to run it or update it it just screams that it's the kind of problem somebody else has already solved. But I haven't been able to find a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there algorithms that will do what I want based on rules like this? Or specific packages or software that might be helpful? Or am I approaching this problem completely wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Please note though that I am well aware that there are many different ways to incorrectly aggregate identities (e.g. that J. Doe could mean John Doe or James Doe), so I don't need warnings against trying to aggregate things.)&lt;/p&gt;&#xA;" OwnerUserId="38472" LastActivityDate="2018-10-17T01:15:14.063" Title="Algorithms for aggregating duplicate identities based on non-numerical data?" Tags="|clustering|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22590" PostTypeId="1" CreationDate="2017-08-25T16:39:04.333" Score="2" ViewCount="51" Body="&lt;p&gt;I am trying to track users reliably on my website so that if they are abusive, they can be banned and not come back easily (obviously this can be bypassed with TOR and such, but most trolls don't care &lt;em&gt;that&lt;/em&gt; much). I have some data that can be set from my server onto the browser and is guaranteed to be unique because I have set the data myself. However, this data is also easily deletable (clearing cache, clearing cookies, clearing other site data). I have some other data (fingerprinting data) that is extremely difficult to remove or change but can easily conflict with other devices which have the same fingerprint. Here is the data I could collect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Storage/caching (guaranteed unique, deletable): Cookies, HTTP ETags,&#xA;localStorage, indexedDB, FileSystem API, Last-Modified date echoing&#xA;&#xA;Fingerprinting (possible collisions between devices, difficult to delete):&#xA;canvas2d, WebGL, &#xA;font list, screen size, &#xA;IP address, plugins installed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And a bit more. All I know so far is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&lt;li&gt;Trust the unique identifiers first&lt;/li&gt;&#xA;    &lt;li&gt;If unique identifiers conflict, choose the majority&lt;/li&gt;&#xA;    &lt;li&gt;If I identifier a user based on his/her unique identifiers but their fingerprints are different, update their default fingerprints to match&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can you help me decide what to do in any  given situation, e.g., a user has deleted all of his/her storage/caching data but the fingerprints remain? What happens when fingerprints conflict in some ways but not in others? What are the probabilities that this is a new user or a returning user?&lt;/p&gt;&#xA;" OwnerUserId="38476" LastActivityDate="2017-08-25T16:39:04.333" Title="How to track user given some guaranteed unique but deletable data and some possibly conflicting but non-deletable data?" Tags="|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22593" PostTypeId="1" AcceptedAnswerId="81405" CreationDate="2017-08-25T20:55:57.950" Score="2" ViewCount="2111" Body="&lt;p&gt;Is there a function equivalent of &lt;code&gt;R&lt;/code&gt;'s &lt;code&gt;ndiffs&lt;/code&gt; function in &lt;code&gt;Python&lt;/code&gt;? &#xA;It is used to estimate a number of diffs for creating stationary time-series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.rdocumentation.org/packages/forecast/versions/8.1/topics/ndiffs&quot; rel=&quot;nofollow noreferrer&quot;&gt;R Documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user38487" LastEditorUserId="43271" LastEditDate="2018-01-23T13:03:47.970" LastActivityDate="2020-09-08T17:45:36.233" Title="ndiffs for Python" Tags="|python|r|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22594" PostTypeId="2" ParentId="22593" CreationDate="2017-08-25T21:28:03.297" Score="1" Body="&lt;p&gt;I am not sure if there any is any library that can directly make your data stationary in python. However, you can plot your time series, which will visually tell you the components such as trend and seasonality of your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;using &lt;code&gt;statsmodel&lt;/code&gt; you can take each a look at each component of a time-series like this,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trend = decomposition.trend&#xA;seasonal = decomposition.seasonal &#xA;residual = decomposition.residual&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this you can use &lt;code&gt;shift&lt;/code&gt; function to make your time-series stationary by taking off trend and seasonality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer to this &lt;a href=&quot;http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA-model-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;, which has in detail to make your time-series stationary based on your data. Hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-08-25T23:32:16.890" LastActivityDate="2017-08-25T23:32:16.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22595" PostTypeId="2" ParentId="10005" CreationDate="2017-08-25T21:38:02.717" Score="1" Body="&lt;p&gt;Your description of the model functioning is correct. Structurally, both models are representatives of the so called kernel methods. As such they are very similar, the same in many cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is completely different between the two methods is the way the kernel centers and linear coefficients are derived. My experience with SVM is limited, so I cannot go into the details of the training method, but in summary is consists of solving the constrained optimization problem of finding the best approximation subject to maximum coefficients. This is achieved by linear programming or through a method like Sequential Minimal Optimization (SMO).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training RBFNs, on the other hand, is a different story. Identifying the kernels is done in one of two ways usually; either by using a clustering algorithm or by using Orthogonal Least Squares (OLS). In either case, the linear coefficients are identified as a second step through least squares (LS).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The differences in training mean that, even through in structure the resulting models may be the same, in functioning they will most probably be completely different, due to the different fitting procedures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some references:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;J. C. Platt, “Fast Training of Support Vector Machines Using Sequential Minimal Optimization,” Adv. kernel methods, pp. 185–208, 1998.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.&lt;/p&gt;&#xA;" OwnerUserId="1085" LastActivityDate="2017-08-25T21:38:02.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22598" PostTypeId="1" AcceptedAnswerId="22622" CreationDate="2017-08-26T05:11:57.593" Score="1" ViewCount="1181" Body="&lt;p&gt;Linear Regression cost function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J(\theta) = \frac{1}{2 m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h_{\theta}(x) = \theta_0 + \theta_1 x_1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic Regression cost function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J(\theta) = \frac{1}{m} \sum_{i=1}^m - (y^{(i)} \times log(h_{\theta}(x{(i)})) + (1-y{(i)}) \times log(1 - h_{\theta}(x{(i)})))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h_{\theta}(x) = g(\theta_0 + \theta_1 x_1 + \theta_2 x_2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively &lt;code&gt;linear regression&lt;/code&gt; is easy to understand as it optimizes the average squared distance between hypothesis and training data. But in case of &lt;code&gt;logistic regression&lt;/code&gt;, I failed to understand the cost function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does the logistic regression cost function represent?&lt;/p&gt;&#xA;" OwnerUserId="38492" LastEditorUserId="29575" LastEditDate="2017-08-26T06:34:31.997" LastActivityDate="2017-09-26T07:54:01.653" Title="Understanding Logistic Regression Cost function" Tags="|logistic-regression|linear-regression|optimization|cost-function|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22605" PostTypeId="1" AcceptedAnswerId="23083" CreationDate="2017-08-26T11:35:55.153" Score="1" ViewCount="1089" Body="&lt;p&gt;I am using sklearn affinity propagation algorithm as below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;affprop = sklearn.cluster.AffinityPropagation(affinity=&quot;precomputed&quot;, damping=0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also have a similarity matrix created for the data I am using. Now I want to use my similarity matrix to use in the affinity propagation model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; they have different methods for this such as fit, fit_predict, predict. So, I'm not sure what to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it correct if I use,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;affprop.fit(my similarity matrix)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2017-09-16T20:00:26.117" Title="Affinity propagation fit, fit_predict and predict methods" Tags="|python|data-mining|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22608" PostTypeId="2" ParentId="869" CreationDate="2017-08-26T13:22:16.677" Score="0" Body="&lt;p&gt;I've tried the following 2 ways for trial-n-test implementation of neural networks with text. The latter one works fairly well, but with limitations.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Create vocabulary using word2vect or NLTK/custom word tokens and assign an index to each word. It is this index which represents the word as number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Challenges:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The indexes must be &quot;normalized&quot; using feature scaling.&lt;/li&gt;&#xA;&lt;li&gt;If the output of the neural network has even a slight variation, then the output may be an index to unexpected word (e.g. if expected output is 250; but NN outputs 249 or 251, then it might be a close output from numeric context; but they are indexes to different words). Recurrent NN to generate output index can be leveraged here.&lt;/li&gt;&#xA;&lt;li&gt;If new words are added to the vocabulary, then the token indexes should be re-scaled. The model trained with previously scaled values may become invalid and must be re-trained.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use identity matrix e.g. for &quot;n&quot; words use &quot;n x n&quot; or (n-1 x n-1) matrix where each row and column represents a word. Put &quot;1&quot; in intersection cell and &quot;0&quot; at rest places. (&lt;a href=&quot;https://visualstudiomagazine.com/articles/2013/07/01/neural-network-data-normalization-and-encoding.aspx&quot; rel=&quot;nofollow noreferrer&quot;&gt;reference&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Challenges:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Every input and output value is &quot;n x 1&quot; vector. For a large sized vocabulary its a bulky computation and slower.&lt;/li&gt;&#xA;&lt;li&gt;If new words are added to the vocabulary, then the identity matrix (i.e. word vector) should be re-calculated. The model trained with previously calculated vectors may become invalid and must be re-trained.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38183" LastEditorUserId="29575" LastEditDate="2017-08-26T15:57:49.753" LastActivityDate="2017-08-26T15:57:49.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22612" PostTypeId="2" ParentId="22568" CreationDate="2017-08-26T20:33:27.243" Score="3" Body="&lt;p&gt;TensorFlow is a general purpose library for numerical computation using data flow graphs. It is primarily used for neural networks but can be used for any mathematical operations on multidimensional data arrays (tensors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, TensorFlow can be used to estimate binary logistic regression with explanatory categorical variables. An example can be found in the TensorFlow tutorials &lt;a href=&quot;https://www.tensorflow.org/tutorials/wide#defining_the_logistic_regression_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-08-30T19:10:17.017" LastActivityDate="2017-08-30T19:10:17.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22613" PostTypeId="1" CreationDate="2017-08-26T21:00:20.123" Score="0" ViewCount="1247" Body="&lt;p&gt;What are some of the important CNN architectures one should know about? What CNN architectures did well on the ImageNet &lt;a href=&quot;http://www.image-net.org/challenges/LSVRC/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ILSVRC&lt;/a&gt; challenge? What CNN architectures are good candidates for &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;transfer learning&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-27T09:26:48.637" Title="Important CNN architectures" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22614" PostTypeId="2" ParentId="22613" CreationDate="2017-08-26T21:18:37.993" Score="2" Body="&lt;p&gt;Here is a list of several important CNN architectures:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;LeNet-5&lt;/strong&gt; the first successful application of CNNs to digit recognition,   developed by Yann LeCun in 1990. It consists of a sequence of Convolutional, ReLU, Max Pooling layers followed by a Fully Connected layer.&#xA;Reference: Y. LeCun et. al., &quot;&lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient-Based Learning Applied to Document Recognition&lt;/a&gt;&quot;, Proceedings of the IEEE, 1998&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;AlexNet&lt;/strong&gt; popularized CNNs in computer vision, did really well on the ImageNet ILSVRC challenge in 2012 showing significant gains in performance.&#xA;The network has similar architecture to LeNet but is deeper and bigger and features convolutional layers stacked on top of each other. Reference: A. Krizhevsky, et. al. &quot;&lt;a href=&quot;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageNet Classification with Deep Convolutional&#xA;Neural Networks&lt;/a&gt;&quot;, NIPS, 2012&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;VGG16&lt;/strong&gt; demonstrated the importance of depth as a critical component to good performance, it was a runner-up in ILSVRC 2014. The architecture consists of a stacked convolutional and max pooling layers with increasing depth and &#xA;it uses a large number of parameters due to the final fully connected layers. Reference: K. Simonyan, &quot;&lt;a href=&quot;https://arxiv.org/pdf/1409.1556.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Very Deep Convolutional Networks for Large-Scale Image Recognition&lt;/a&gt;&quot;, ICLR, 2015&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Inception&lt;/strong&gt; original version called GoogLeNet from Google won the ILSVRC 2014 challenge the architecture consists of inception modules that dramatically reduce the number of parameters, it uses multi-scale 3x3, 5x5 convolutional filters including 1x1 convolutions for dimensionality reduction. Reference: C. Szegedy et al., &quot;&lt;a href=&quot;https://arxiv.org/pdf/1409.4842.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Going Deeper with Convolutions&lt;/a&gt;&quot;, CVPR, 2015&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;ResNet50&lt;/strong&gt; residual network was the winner of ILSVRC 2015, it introduces skip connections for easier training that enable very deep architectures&#xA;and makes use of batch normalization. Reference: K. He et al., &quot;&lt;a href=&quot;https://arxiv.org/pdf/1512.03385.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Residual Learning for Image Recognition&lt;/a&gt;&quot;, CoRR, 2015&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-08-26T21:18:37.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22616" PostTypeId="1" AcceptedAnswerId="22621" CreationDate="2017-08-27T03:36:03.410" Score="1" ViewCount="3153" Body="&lt;p&gt;I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical variables be done the same?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastActivityDate="2019-11-19T15:08:31.510" Title="Transformation of categorical variables" Tags="|random-forest|categorical-data|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22618" PostTypeId="2" ParentId="22494" CreationDate="2017-08-27T07:00:32.870" Score="22" Body="&lt;p&gt;Ok, so after a lot of experimentation I have managed to get some results/insights.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance&lt;/strong&gt; of the network, as a negative side, the training process is muuuuuch slower.&lt;/p&gt;&#xA;&lt;p&gt;Second point, data is important, nothing new here but as I learned while fighting this problem, more data always seems to help a bit.&lt;/p&gt;&#xA;&lt;p&gt;Third point, dropout is useful in large networks with lots of data and lots of iterations, in my network I applied dropout on the final fully connected layers only, convolution layers did not get dropout applied.&lt;/p&gt;&#xA;&lt;p&gt;Fourth point (and this is something I am learning over and over): neural networds take A LOT to train, even on good GPUs (I trained this network on  floydhub, which uses quite expensive NVIDIA cards), so &lt;strong&gt;PATIENCE is key&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Final conclusion: Batch sizes are more important that one might think, apparently it is easier to hit a local minimum when batches are larger.&lt;/p&gt;&#xA;&lt;p&gt;The code I wrote is available as a &lt;a href=&quot;https://github.com/moriano/loco-learning/blob/master/cats-vs-dogs/cats-vs-dogs.ipynb&quot; rel=&quot;noreferrer&quot;&gt;python notebook&lt;/a&gt;, I think it is decently documented.&lt;/p&gt;&#xA;" OwnerUserId="38339" LastEditorUserId="98307" LastEditDate="2020-08-05T11:06:28.110" LastActivityDate="2020-08-05T11:06:28.110" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="22619" PostTypeId="2" ParentId="22545" CreationDate="2017-08-27T07:05:04.067" Score="7" Body="&lt;p&gt;I'll try and answer your question point-by-point from an architect's point-of-view:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We would like to implement rule engine in one of our products, but&#xA;  when I read on the Internet, many are suggesting that rule engines are&#xA;  an old way of doing things&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The 'people of the internet' don't know your scale and the use-case. Only you know. So, they might be suggesting ideas from a very high-scale point of view, when your system is not necessarily a high scale one. So, pick the right idea/tool for the job.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;suggesting that rule engines are an old way of doing things and it&#xA;  will be very hard to manage as we add more rules into it&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;True&lt;/strong&gt;. Rule-based systems are not scalable, due to obvious reasons. And, that's why people consider ML as a highly scaled out rule-based engine, wherein the rules can no longer be understood by humans. Thus, &lt;a href=&quot;https://datascience.stackexchange.com/q/22335/11097&quot;&gt;the black-box name&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So my Question is, should I learn Machine Learning or go on&#xA;  implementing Drools as rule engine in my application.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;That is for you to decide.&lt;/strong&gt; We don't know your scale. But, if you think the rules would be increasing at a huge rate in the future, then ML is the best way forward for your use-case.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or can you suggest any online resources achieving the same using&#xA;  Machine Learning?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For me, every recommender system is a complex rule-based engine under the hood. So, go through &lt;a href=&quot;https://stackoverflow.com/q/2323768/4993513&quot;&gt;Amazon's&lt;/a&gt; and how they do their recommendation.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2017-08-27T07:05:04.067" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22620" PostTypeId="2" ParentId="19124" CreationDate="2017-08-27T07:06:36.967" Score="5" Body="&lt;p&gt;Something that I have learned the hard way is to plot the learning curves, I know, it is not as fun as writing the machine learning code per-se, but it is fundamental to visually understand what is happening. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A rule of thumb definition is that over fitting occurs when your train accuracy keeps improving while your validation accuracy stops improving (or even starts getting worse). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simplest solution to avoid over fitting is early stopping (stop training as soon as things look bad), of course being the simplest solution comes at a cost: it is not the best solution. Regularisation and dropout are good tools to fight over fitting, but that is a different matter :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps&lt;/p&gt;&#xA;" OwnerUserId="38339" LastActivityDate="2017-08-27T07:06:36.967" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22621" PostTypeId="2" ParentId="22616" CreationDate="2017-08-27T07:08:41.970" Score="2" Body="&lt;p&gt;Afaik, once you deal with the categorical variables you end up having several columns where the values are either true/false (or 0/1). So, I do not see how making them Gaussian would help.&lt;/p&gt;&#xA;" OwnerUserId="38339" LastEditorUserId="73851" LastEditDate="2019-11-19T15:08:31.510" LastActivityDate="2019-11-19T15:08:31.510" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22622" PostTypeId="2" ParentId="22598" CreationDate="2017-08-27T07:11:56.573" Score="1" Body="&lt;p&gt;The way my intuition works for logistic regression is simple, say you are trying to classify if you have a picture with a dog on it, if that is the case, you output 1 (true), if not, you output 0 (false)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost function essentially represents the diff between what your model output (imagine, you have a dog in a picture, and your model outputs 0.75 instead of 1, your cost is 0.25). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is not a very mathematical way to put it, but at least it has helped me a lot to understand logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="38339" LastActivityDate="2017-08-27T07:11:56.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22623" PostTypeId="2" ParentId="22560" CreationDate="2017-08-27T07:14:24.473" Score="1" Body="&lt;p&gt;IRIS is a very small data set, with not that many data points. I have found that to play with a data set, MNIST does much better job, and you can easily do over fitting and under fitting there.&lt;/p&gt;&#xA;" OwnerUserId="38339" LastActivityDate="2017-08-27T07:14:24.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22624" PostTypeId="1" CreationDate="2017-08-27T08:06:40.440" Score="1" ViewCount="138" Body="&lt;p&gt;I have multiple VMs (Virtual Machines) on one physical server (hardware machine). I have CPU utilization information (every 5 minute) for all the virtual machines and the physical server. Now I want to understand how each VM is contributing to the overall load of the physical server. I am assuming that this is a time series problem. The CPU utilization of each VM as well the physical server are separate time series. One way to approach this problem is to apply linear regression considering CPU utilization of each VM as explanatory varaibles and CPU utilization of physical server as response variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  cpu_physical_server(t) = Function(cpu_vm_one(t) + cpu_vm_two(t) + cpu_vm_three(t) + .....)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As per my understanding, I can't directly apply regression modelling (&lt;code&gt;lm()&lt;/code&gt; or &lt;code&gt;glm()&lt;/code&gt;) as these are all time series (&lt;a href=&quot;https://svds.com/avoiding-common-mistakes-with-time-series/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to calculate the regression coefficients when both explanatory and response variables are time series? What kind of correction(s) should I apply?&lt;/p&gt;&#xA;" OwnerUserId="16581" LastEditorUserId="11097" LastEditDate="2017-08-27T10:01:30.680" LastActivityDate="2017-08-29T09:48:46.353" Title="Regression coefficient(s) when explanatory &amp; response variables are time series" Tags="|r|time-series|regression|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22626" PostTypeId="2" ParentId="22613" CreationDate="2017-08-27T09:26:48.637" Score="1" Body="&lt;p&gt;See &lt;a href=&quot;https://arxiv.org/abs/1707.09725&quot; rel=&quot;nofollow noreferrer&quot;&gt;my masters thesis&lt;/a&gt;. Chapter 2.2 and 2.3 for a high-level overview over the building blocks. Appendix D describes a couple of really well known CNN architectures in detail:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Lenet-5 (historic)&lt;/li&gt;&#xA;&lt;li&gt;AlexNet (still in use, but not SotA)&lt;/li&gt;&#xA;&lt;li&gt;VGG-16 D (Still in use, but not SotA)&lt;/li&gt;&#xA;&lt;li&gt;Googlenet and the Inception-Nets (v2, v3, v4)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not described in detail, but still important&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Resnets&lt;/li&gt;&#xA;&lt;li&gt;Densenets&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-08-27T09:26:48.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22627" PostTypeId="1" AcceptedAnswerId="22649" CreationDate="2017-08-27T16:38:59.617" Score="5" ViewCount="167" Body="&lt;p&gt;I have sentences telling me to who a shop is opened to: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;cats, dogs or birds&quot; (1) &lt;/li&gt;&#xA;&lt;li&gt;&quot;young dogs with collar&quot; (2) &lt;/li&gt;&#xA;&lt;li&gt;&quot;old cats or yellow birds&quot; (3) etc...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to design an algorithm that will change this sentences to a tree representation of the logic in it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(1) = (cat) or ((dog) or (bird))&lt;/li&gt;&#xA;&lt;li&gt;(2) = (young) and (dog) and (collar)&lt;/li&gt;&#xA;&lt;li&gt;(3) = ((cat) and (old)) or ((bird) and (yellow))&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What do you think will work the best? LSTM maybe? How can I have this representation as a result?&lt;/p&gt;&#xA;" OwnerUserId="38523" LastEditorUserId="29575" LastEditDate="2017-08-27T17:24:35.653" LastActivityDate="2017-08-28T12:24:13.313" Title="Logic in sentence : tree representation" Tags="|machine-learning|neural-network|data-mining|deep-learning|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22629" PostTypeId="2" ParentId="22574" CreationDate="2017-08-27T18:51:56.727" Score="1" Body="&lt;p&gt;Think of it in terms of navigating a landscape.  The land you move across is created by your error function (that is, the realtion between your model and your data), and the way you move across the landscape is your training function.  Especially in Neural Networks, falling in 'local minima' is a big problem.  In fact, Neural Networks have been theoretially able to approximate &lt;em&gt;any&lt;/em&gt; function for... 20 years?  See &lt;a href=&quot;https://en.wikipedia.org/wiki/Universal_approximation_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Universal Approximation Theorem&lt;/a&gt;.  But the problem has always been learning how to train a given network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, in your case comaring MAE and MSE, the difference is in the 'square' part.  The steepness of the 'fitness landscape' will be much steeper where error is greater with MSE compared to MAE.  1*1 = 1.  i.e., squaring does nothing at unit-error.  10 * 10 = 100.  Absolute error of 10 will become a squared error of 100.  That is looking at one sample, and has huge implications around falling in local minima where the error is great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking across the whole dataset, another huge implication is how the NN balances the error it sees.  With MSE, samples with high error become samples with very high error.  Thus, a NN trained with MSE will 'care more' about learning samples that are very wrong at the expense of small improvements to 'easier' samples.&lt;/p&gt;&#xA;" OwnerUserId="6391" LastActivityDate="2017-08-27T18:51:56.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22630" PostTypeId="1" CreationDate="2017-08-27T19:28:11.207" Score="1" ViewCount="46" Body="&lt;p&gt;I want to analyze product sales data with &lt;a href=&quot;https://en.wikipedia.org/wiki/Weka_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;WEKA&lt;/a&gt;. My data have attributes: sell quantity and sales price. I want to add more attributes, such as mean and standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What formula I should use to create more dimensions?&lt;/p&gt;&#xA;" OwnerUserId="38527" LastEditorUserId="381" LastEditDate="2017-08-27T20:01:48.623" LastActivityDate="2017-08-27T22:13:54.743" Title="How to create more dimensions for data mining?" Tags="|data-mining|dataset|data-cleaning|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22632" PostTypeId="1" AcceptedAnswerId="22685" CreationDate="2017-08-27T21:34:07.687" Score="2" ViewCount="2383" Body="&lt;p&gt;Let's say I want to add a few hand-crafted features to a convolutional neural network &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN&lt;/a&gt; in TensorFlow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CNN can be a simple one as described &lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/pros&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naturally I'd like to add these features right after the second pooling and right before the first fully-connected layer (FC1 in the example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that easy to express my method in code? I'd have to append my features to the &lt;code&gt;h_pool2_flat&lt;/code&gt; vector/tensor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])&#xA;h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15850" LastEditorUserId="29575" LastEditDate="2017-08-27T22:25:35.430" LastActivityDate="2018-05-08T09:04:55.100" Title="Adding hand-crafted features to a convolutional neural network (CNN) in TensorFlow" Tags="|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22634" PostTypeId="2" ParentId="22630" CreationDate="2017-08-27T22:13:54.743" Score="1" Body="&lt;p&gt;In general, this is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_engineering&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature engineering&lt;/a&gt;. Expansion of dataset attributes via attribute transformations is called basis function expansion. These are often nonlinear functions of the original attributes. Popular nonlinear functions are &lt;a href=&quot;https://en.wikipedia.org/wiki/Polynomial&quot; rel=&quot;nofollow noreferrer&quot;&gt;polynomials&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-27T22:13:54.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22636" PostTypeId="1" CreationDate="2017-08-28T03:04:28.523" Score="1" ViewCount="2756" Body="&lt;p&gt;I learned that Keras doesn't have a built-in way to set a threshold for precision and accuracy when building a classifier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Courtesy of a solution &lt;a href=&quot;https://stackoverflow.com/questions/42606207/keras-custom-decision-threshold-for-precision-and-recall&quot;&gt;here&lt;/a&gt;, I wanted to see what would happen when I fit a simple 3 layer Multilayer Perceptron with different classification thresholds for these metrics.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I went ahead and fit a model with as such:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics = [precision_threshold(0.5), recall_threshold(0.5)])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and got out these results:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;val_precision: 0.7048 - val_recall: 0.5795&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, I plotted a normalized confusion matrix. And using this:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def precision(cm = confusion_matrix):&#xA;    return round(cm[1,1] / (cm[0,1] + cm[1,1]), 4)&#xA;&#xA;def recall(cm = confusion_matrix):&#xA;    return round(cm[1,1] / (cm[1,0] + cm[1,1]), 4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I calculated the resulting precision and recall:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Normalized confusion matrix&#xA;[[ 0.6026  0.3974]&#xA; [ 0.2204  0.7796]]&#xA;&#xA;Precision: 0.71&#xA;Recall:    0.78&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seems pretty reasonable so far.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I switched up the thresholds and looked at what outcomes resulted. The table below shows what I got when I changed the precision threshold to 0.1 and the recall threshold to 0.9:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|-----------------------------------------------------|&#xA;| paramter settings  for model fitting:               |&#xA;| [precision_threshold(0.1)    recall_threshold(0.9)] |&#xA;|                                                     |&#xA;| validation results                                  |&#xA;|- val_precision  0.4089     - val_recall 0.0457      |&#xA;|                                                     |&#xA;| Normalized confusion matrix                         |&#xA;| [[ 0.6148  0.3852]                                  |&#xA;| [ 0.2303  0.7697]]                                  |&#xA;|                                                     |&#xA;| Precision   0.70                                    |&#xA;| Recall      0.79                                    |&#xA;|-----------------------------------------------------|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Interestingly, if you look at the final Precision and Recall scores, I notice that they didn't change, even with the extreme parameters I set. Basically the same confusion matrix as well.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I repeated this for a large set of different settings and got generally the same final Precision and Recall scores. I could past them in here but it would take up a lot of space.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more detail then I'll switch to my actual question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model is based on the business case where a large precision value is very desirable and recall is not relevant.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is this -- given that I desire large precision -- is tuning the specific precision and recall thresholds for the model valuable at all, if the final Precision and Recall scores are left fundamentally unchanged across all configurations of the thresholds?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this makes sense. Thanks for reading this.  &lt;/p&gt;&#xA;" OwnerUserId="15533" LastActivityDate="2017-08-28T03:04:28.523" Title="Can tuning individual precision and recall classification thresholds improve deep learning models?" Tags="|keras|model-evaluations|confusion-matrix|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22638" PostTypeId="1" AcceptedAnswerId="22660" CreationDate="2017-08-28T05:21:23.170" Score="-1" ViewCount="1848" Body="&lt;p&gt;Is there any difference if we predict two variables vs predict one variable with two models (so we have two variables predicted)?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastEditorUserId="24119" LastEditDate="2017-08-28T05:51:38.717" LastActivityDate="2017-08-29T13:24:57.100" Title="Predict Two Variables vs Predict One Variable with Two Models" Tags="|machine-learning|" AnswerCount="2" CommentCount="8" ClosedDate="2017-08-29T10:27:34.493" ContentLicense="CC BY-SA 3.0" />
  <row Id="22639" PostTypeId="1" AcceptedAnswerId="22694" CreationDate="2017-08-28T05:25:51.610" Score="6" ViewCount="11243" Body="&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I picked up data-set from &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/bank+marketing&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and tried to run Logistic Regression on it. Since I am not very much aware of MATLAB, I converted &quot;Strings&quot; to &quot;Numbers&quot; with my own using &quot;NUMBERS&quot; software.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I want to achieve:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After running the LR algorithm when I tried to predict the value of existing data points, I am getting values ranging between 0-1 (as it should be), but since my job it to predict whether it is either 0 (yes) or 1 (no), that means I need to find a cut-off line (threshold) in my prediction (This could probably be done by comparing actual value by predicted value).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I figure out the threshold for predicted result such that result is assumed to be 1 if predicted value &gt; threshold, otherwise 0? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predicted values can be &lt;a href=&quot;https://github.com/guptakvgaurav/ml-practice/blob/master/result_predict_val.txt&quot; rel=&quot;noreferrer&quot;&gt;found here.&lt;/a&gt;&#xA;I am assuming predicted values are correct as &lt;a href=&quot;https://raw.githubusercontent.com/guptakvgaurav/ml-practice/master/cost_curve.png&quot; rel=&quot;noreferrer&quot;&gt;cost curve&lt;/a&gt; is showing asymptotic nature. I have pushed &lt;a href=&quot;https://github.com/guptakvgaurav/ml-practice&quot; rel=&quot;noreferrer&quot;&gt;my work here&lt;/a&gt;, you may want to cross-validate and provide me few more key points.&lt;/p&gt;&#xA;" OwnerUserId="38538" LastEditorUserId="29575" LastEditDate="2017-08-28T05:59:01.507" LastActivityDate="2017-08-29T19:10:12.120" Title="How to determine threshold in Sigmoid function" Tags="|machine-learning|classification|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22640" PostTypeId="2" ParentId="22638" CreationDate="2017-08-28T06:00:36.133" Score="1" Body="&lt;p&gt;You should specify what exactly the model you're thinking. Of course there're differences. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Multiple comparisons (inflate Type 1 error)&lt;/li&gt;&#xA;&lt;li&gt;Loss in statistical power&lt;/li&gt;&#xA;&lt;li&gt;Different probability mass function&lt;/li&gt;&#xA;&lt;li&gt;Marginal distribution vs joint distribution&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, we're talking about Apple and Orange. They have different interpretation, different equations, different formula, different way to estimate solution etc etc.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-08-28T06:00:36.133" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22641" PostTypeId="2" ParentId="22565" CreationDate="2017-08-28T06:10:55.413" Score="0" Body="&lt;p&gt;There is &lt;strong&gt;one class classification&lt;/strong&gt;. In particular OC-SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could simply anaylze the distance to the nearest sample.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-08-28T06:10:55.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22643" PostTypeId="1" AcceptedAnswerId="53501" CreationDate="2017-08-28T10:14:22.430" Score="0" ViewCount="618" Body="&lt;p&gt;Simplified question:&#xA;I have a dataset of how well certain agents perform on certain tasks, and based on this I've trained a model that can make predictions for newly incoming tasks. I'd like to make an algorithm that distributes tasks amongst agents to maximise performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way would be to make a prediction for each scenario - each agent assigned to each new set of tasks (note that one agent can work on more than one task at a time) -, thus iterating through all possible solutions, and summing performance and finding the max.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This, however, would be very complex - to iterate through every possible combination an finding a maximum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to optimise (maximise performance) without having to iterate through all possible solutions? How could I do that?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Original question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a look at the following setup:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There are a number of &quot;agents&quot; &lt;/li&gt;&#xA;&lt;li&gt;Agents work on &quot;jobs&quot; &lt;/li&gt;&#xA;&lt;li&gt;Each agent might be better at certain jobs, but the jobs get assigned by random, and not based on performance (good performance = the least amount of time spent working)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd like to know if there's any ML algorithm that can learn how to assign jobs to agents as to minimise time spent working (maximise performance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that with methods like regression or classification, or even deep learning you can teach the algorithm to predict a target variable, but in this case I do not want to reach a target variable, but want to increase performance (minimise time spent working). Is there such an algorithm/method that could &quot;learn&quot; from past performance reviews and assign new jobs to agents to maximize performance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A bit more formalised:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I teach an SVM (or a regressor, or a neural net), that will be able to predict how well a certain agent A1 performs on a certain (type of) task T1 (call this performance P1). So when a new task comes in, I'll be able to predict P1 based on A1 and T1. BUT! And here comes the question. Now that I can predict P1 based on A1 and T1, how do I use this knowledge to actually assign ALL the tasks - T1...Tn amongst all the agents A1...Am as to maximise the performance: sum(P1, P2, ..., Pi)?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="29169" LastEditDate="2019-06-09T23:29:30.243" LastActivityDate="2019-06-10T00:22:32.687" Title="ML algorithm for target minimization?" Tags="|machine-learning|algorithms|optimization|" AnswerCount="4" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="22644" PostTypeId="1" CreationDate="2017-08-28T11:18:00.613" Score="5" ViewCount="1253" Body="&lt;p&gt;I am trying to perform Hindi To English translation using s2s model, following - &lt;a href=&quot;https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/&quot; rel=&quot;noreferrer&quot;&gt;https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used &lt;a href=&quot;https://github.com/karimkhanp/Seq2Seq/tree/master/data&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/karimkhanp/Seq2Seq/tree/master/data&lt;/a&gt; dataset which contains 37726 sentences for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training took around 44 hours on my 8GB machine. I considered &lt;code&gt;3 layers, 10 epoch, 20000 vocap size&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other parameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ap.add_argument('-max_len', type=int, default=200)&#xA;ap.add_argument('-vocab_size', type=int, default=20000)&#xA;ap.add_argument('-batch_size', type=int, default=100)&#xA;ap.add_argument('-layer_num', type=int, default=3)&#xA;ap.add_argument('-hidden_dim', type=int, default=1000)&#xA;ap.add_argument('-nb_epoch', type=int, default=10)&#xA;ap.add_argument('-mode', default='train') &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But now when I test using hindi sentence, it give UNK for all words. Though I take same sentence for testing as in training even it says UNK in results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test sentences:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल&#xA;मैं भारत से प्यार करता हूँ&#xA;ज़बूर जो कि दाउद को प्रदान की गयी&#xA;प्रशासनिक विभाजन&#xA;वे इस देश के प्रथम UNK -LRB- अफ्रीकी UNK -RRB-&#xA;नेपाली विदेश&#xA;वेल्श खिलाड़ी इंग्लैंड के लिए खेलने के लिए पात्र हैं&#xA;उनके बड़े भाई अजीत तेंडुलकर ने उन्हें खेलने के लिये प्रोत्साहित किया था ।&#xA;फिर एक मिनट के बाद किताब छत की ओर उछालकर उन्होंने कहा - चलो ।&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;of of of&#xA;of of of&#xA;the of of of&#xA;&#xA;lrb lrb rrb rrb&#xA;&#xA;the the of of of&#xA;the the of of of of&#xA;the the the of of of of&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I could not understand the issue behind this result. I have used some sentences which are same as training sentences. Atleast answer should be correct for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did I do anything wrong? I really appreciate any help&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if MODE == 'train':&#xA;    k_start = 1&#xA;    t1 = time()&#xA;    # If any trained weight was found, then load them into the model&#xA;    if len(saved_weights) != 0:&#xA;        print('[INFO] Saved weights found, loading...')&#xA;        epoch = saved_weights[saved_weights.rfind('_')+1:saved_weights.rfind('.')]&#xA;        model.load_weights(saved_weights)&#xA;        k_start = int(epoch) + 1&#xA;&#xA;    i_end = 0&#xA;    for k in range(k_start, NB_EPOCH+1):&#xA;        # Shuffling the training data every epoch to avoid local minima&#xA;        indices = np.arange(len(X))&#xA;        np.random.shuffle(indices)&#xA;        X = X[indices]&#xA;        y = y[indices]&#xA;&#xA;        # Training 1000 sequences at a time&#xA;        for i in range(0, len(X), 1000):&#xA;            if i + 1000 &amp;gt;= len(X):&#xA;                i_end = len(X)&#xA;            else:&#xA;                i_end = i + 1000&#xA;            y_sequences = process_data(y[i:i_end], y_max_len, y_word_to_ix)&#xA;&#xA;            print('[INFO] Training model: epoch {}th {}/{} samples'.format(k, i, len(X)))&#xA;            model.fit(X[i:i_end], y_sequences, batch_size=BATCH_SIZE, nb_epoch=1, verbose=2)&#xA;        model.save_weights('checkpoint_epoch_{}.hdf5'.format(k))&#xA;    print(&quot;Time taken to train the data in hour=&amp;gt;&quot;, (time()-t1)/3600)&#xA;&#xA;# Performing test if we chose test mode&#xA;else:&#xA;    # Only performing test if there is any saved weights&#xA;    if len(saved_weights) == 0:&#xA;        print(&quot;The network hasn't been trained! Program will exit...&quot;)&#xA;        sys.exit()&#xA;    else:&#xA;        # import pdb&#xA;        # pdb.set_trace()&#xA;        X_test = load_test_data('test', X_word_to_ix, MAX_LEN)&#xA;        X_test = pad_sequences(X_test, maxlen=X_max_len, dtype='int32')&#xA;        model.load_weights(saved_weights)&#xA;        # print(X_test)&#xA;        # print(model.predict(X_test))&#xA;        predictions = np.argmax(model.predict(X_test), axis=2)&#xA;        # print(predictions)&#xA;        # print(y_ix_to_word)&#xA;        sequences = []&#xA;        for prediction in predictions:&#xA;            sequence = ' '.join([y_ix_to_word[index] for index in prediction if index &amp;gt; 0])&#xA;            print(sequence)&#xA;            sequences.append(sequence)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12448" LastEditorUserId="12448" LastEditDate="2017-08-28T11:54:34.130" LastActivityDate="2017-08-28T11:54:34.130" Title="Machine Translation using seq2seq model" Tags="|machine-learning|neural-network|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22646" PostTypeId="2" ParentId="22644" CreationDate="2017-08-28T11:40:38.520" Score="3" Body="&lt;p&gt;Clearly, the optimization of your loss function led to it only predicting the most common words in English. The probability of other words are obviously small. &#xA;I am no expert, but I would check three things. 1. How long does it take for the loss to converge? With the output you show it is likely it does not take a long time. 2. Maybe try to filter out so you have different words in the sentences. 3. Check if it can help to optimize another loss function which would have a bias towards more diverse sentences.&#xA;I hope this can give you a few ideas at least.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2017-08-28T11:40:38.520" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22648" PostTypeId="2" ParentId="22644" CreationDate="2017-08-28T11:50:50.603" Score="3" Body="&lt;p&gt;This is kind of a common issue in Seq2Seq model. I haven't tried machine translation, but I have tried text generation and got stuck with this problem of repeating words. The problem is in learning weights I believe. Even with GPUs it takes almost a week to get a considerable good result. I guess, you have used CPUs. for sure, CPUs takes much more time than 44 hours.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-08-28T11:50:50.603" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22649" PostTypeId="2" ParentId="22627" CreationDate="2017-08-28T12:24:13.313" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Syntaxnet&lt;/a&gt; parser could surely help you in parsing the sentences and in tree represenation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you plan on solving using RNNs, I believe &lt;a href=&quot;https://github.com/stanfordnlp/treelstm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tree LSTM&lt;/a&gt; will be a better choice than LSTM, as it also preserves dependency information. &#xA;&lt;a href=&quot;https://arxiv.org/abs/1503.00075&quot; rel=&quot;nofollow noreferrer&quot;&gt;Full paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use &lt;code&gt;Tree LSTM&lt;/code&gt;, if you need a vector embedding for the whole sentence. For use cases like classification, sentiment analysis. It works and there is a good probability that the vector could have all the information but you may not be certain and thats why its still a black box. But, if your use case is a clear representation of tree structure and logic among the terms (which you wanted), better go with parsers like &lt;code&gt;Syntaxnet&lt;/code&gt; and try rule-based models for the use case mentioned.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2017-08-28T12:24:13.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22650" PostTypeId="1" CreationDate="2017-08-28T12:35:03.890" Score="-1" ViewCount="85" Body="&lt;p&gt;I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies importing goods from a certain country. Here is an example of the records:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| ID      | buyer_string                                      |&#xA;| ------- | ------------------------------------------------- |&#xA;| 5278898 | IVY LEAGUE (UK) LIMITED,THORNFIELD                |&#xA;| 5480749 | EI CORTE INGLES S.A.                              |&#xA;| 206882  | NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... |&#xA;| 4698965 | H&amp;amp;M HENNES &amp;amp; MAURITZ GBC AB SWEDEN N/P-H&amp;amp;M HEN... |&#xA;| 3012744 | LOGIX FZCOPO BOX 261422 JEBEL ALI FREE ZONE UN... |&#xA;| 5557832 | K-MART CORPORATION 3333 BEVERLY ROAD HOFFMAN E... |&#xA;| 3000435 | TAHA KONFEKSIYON SANAYI TIC. LTD                  |&#xA;| 61883   | H &amp;amp; M (INTL) LTD SUITE 2102B                      |&#xA;| 3994401 | H&amp;amp;M HENNES &amp;amp; MAURITZ GBC AB                       |&#xA;| 2767566 | WAL-MART CANADA CORP                              |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The task is to map each entry in the dataset to the name of the company it refers to. For instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;NEXT PLCDESFORD RPAD ENDERBY LEICESTER LE19 4A... -&gt; &lt;strong&gt;Next PLC&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;H&amp;amp;M HENNES &amp;amp; MAURITZ GBC AB SWEDEN N/P-H&amp;amp;M HEN... -&gt; &lt;strong&gt;H&amp;amp;M Hennes and Mauritz&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;WAL-MART CANADA CORP -&gt; &lt;strong&gt;Walmart Corporation&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;etc...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The main problem is that I do not have a complete &quot;clean&quot; list of company names that I know are present in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a small list of the biggest 200 companies operating in the country in question. Performing fuzzy matching using this list only covers around 50% of the records that I have (there are approximately 6 million records in total).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried clustering the records using Affinity Propagation (with the Levenshtein/Edit Distance as a distance metric). The clustering approach managed to cover around 80% of the records; however, many of cluster exemplars do not map onto &quot;clean&quot; company names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current pipeline is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normalize the text in each record i.e. convert everything to lowercase, remove special characters, etc.&lt;/li&gt;&#xA;&lt;li&gt;Using regular expressions, match records with the list of 200 company names.&lt;/li&gt;&#xA;&lt;li&gt;Perform affinity propagation clustering on the remaining unmatched records; using the first 30 characters as features.&lt;/li&gt;&#xA;&lt;li&gt;Perform fuzzy matching on the cluster exemplars to match them to actual company names (from the list that I have).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The issue with my approach is that I am unable to &quot;discover&quot; new companies in the dataset. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;IVY LEAGUE (UK)&lt;/strong&gt; is not in list of companies that I have, so there is no way I can map IVY LEAGUE (UK) LIMITED,THORNFIELD to &lt;strong&gt;IVY LEAGUE (UK)&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So I was wondering if can get any other suggestions on tackling this task or at least improving my current pipeline? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help on this is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="38550" LastEditorUserId="23305" LastEditDate="2017-08-28T19:31:34.877" LastActivityDate="2017-08-28T19:31:34.877" Title="Text standardisation for manually entered data" Tags="|machine-learning|text-mining|data-cleaning|unsupervised-learning|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22651" PostTypeId="2" ParentId="22643" CreationDate="2017-08-28T13:09:11.627" Score="0" Body="&lt;p&gt;Some ideas:1. Could you assign features for jobs and assignments? Performance would be the labels. Then you could simply use an SVM for example.&#xA;2. Given a user representation matrix X, each row i is an agent, and a job representation matrix Y, each j is a job representation, and a matrix of performance P, where every row stands for a user and every column how well the agent performed on that job, you could learn X and Y through optimization. Check this: &lt;a href=&quot;https://en.wikipedia.org/wiki/Non-negative_matrix_factorization&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Non-negative_matrix_factorization&lt;/a&gt;&#xA;I hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2017-08-28T13:09:11.627" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="22653" PostTypeId="2" ParentId="22650" CreationDate="2017-08-28T17:40:08.123" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The main problem is that I do not have a complete &quot;clean&quot; list of&#xA;  company names that I know are present in the dataset.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You have a garbage in, garbage out or business process problem. I would suggest:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When users enter company information, they should be prompted first&#xA;to select from a clean list and only type in a value if they say it&#xA;is not present.&lt;/li&gt;&#xA;&lt;li&gt;After the name of the organization, enter some &lt;a href=&quot;https://www.dnb.co.uk/duns-number.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;unique ID&lt;/a&gt; at the end in []. Hopefully the imported goods have that on the package. Then parse that unique number out and use that matching.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-08-28T17:40:08.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22658" PostTypeId="1" CreationDate="2017-08-28T18:53:38.637" Score="-1" ViewCount="114" Body="&lt;p&gt;I'm currently using a regression ANN to determine the size of an object, in the range [1,100]. I get the output values (sometimes outside of the desired range), but I have no idea how confident the network is of the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking, if I set up an ANN with a 100 one-hot vector as the output would I be able to determine the most likely result and its confidence? My understanding is that a one-hot classification network gives a number of output values, the sum of which equal 1, so in essence you are getting the probability of each value and can work out some error bars.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I way off here, or is there a better way of determining the confidence of the result on a regression ANN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I should mention that I'm currently using Python 3.5 and Keras with a Tensorflow backend.&lt;/p&gt;&#xA;" OwnerUserId="38566" LastEditorUserId="29575" LastEditDate="2017-08-28T19:10:47.943" LastActivityDate="2017-08-31T22:29:24.683" Title="Would one-hot classification allow me to generate error bars of results?" Tags="|machine-learning|neural-network|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22660" PostTypeId="2" ParentId="22638" CreationDate="2017-08-28T19:10:28.367" Score="4" Body="&lt;p&gt;I think OP meant a &lt;em&gt;multi-class&lt;/em&gt; model that predicts an outcome variable with multiple classes versus building multiple separate binary classification models for &lt;em&gt;each&lt;/em&gt; class. Indeed these two modeling techniques are different, and should be used differently according to the problem.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Multi-class Classification Problems&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;These are problems where you have to assign cases to a dependent variable with multiple categories/classes/outcome. Classes are mutually exclusive, meaning that observations can be assigned to only one category at a time (estimated probability for every category of $Y$ adds up to one). Many algorithms exists for this type of problem. For example Random Forest, Multinomial Logistic Regression, Boosted Trees, Linear Discriminant Analysis, etc. Each with their own set of assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the muli-class case, multinomial logisitc regression actually picks a &quot;pivot&quot; class and runs a binary logistic regression for each class regressing on the pivot class. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Multi-label Classification Problems&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;These are problems where each case can be assigned to more than one category. The dependent variable is still multi-class, but we cannot use a multi-class classification model because the categories are &lt;em&gt;not&lt;/em&gt; mutually exclusive. There exist several ways to deal with multi-label problems, one of which is to transform a multi-label problem to multiple distinct binary classification problems. This is known as a &lt;strong&gt;binary relevance transformation&lt;/strong&gt;. This can be done by simply treating each class of the dependent variable as a binary outcome (in that specific class or not), and running a binary classification method (like binary logistic regression) on each of them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is similar to the multinomial logsitic regression case because multiple binary logistic regressions are run, but it is also different because each model in this case is independent and do not depend on a chosen pivot class.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Multi-class or Multi-label?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;To answer your question: &quot;Multi-class to predict a multi-category dependent or several binary models to predict each class?&quot;. It really depends on your problem. Do you want to assign multiple classes to each observation? If so, you have a multi-label problem and the binary relevance transformation is a good way to model. Are categories in your dependent variable instead mutually exclusive? In this case, use multi-class classification models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; The downside of binary relevance is that it treats each category of the dependent variable to be independent, thus ignoring any dependencies across different classes. A &lt;strong&gt;label powerset transformation&lt;/strong&gt; is a good alternative. This transforms the dependent variable into a multi-class variable with each class representing the occurrence of a combination of the original classes. A multi-class model is then run on these new combination classes. This takes into account the &lt;em&gt;co-occurrence&lt;/em&gt; of classes not just single occurrences.&lt;/p&gt;&#xA;" OwnerUserId="38096" LastEditorUserId="38096" LastEditDate="2017-08-29T13:24:57.100" LastActivityDate="2017-08-29T13:24:57.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22661" PostTypeId="1" AcceptedAnswerId="22671" CreationDate="2017-08-28T19:48:37.607" Score="1" ViewCount="212" Body="&lt;p&gt;The criterion for firing a perceptron is as follows &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QmxDq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QmxDq.png&quot; alt=&quot;Firing Criterion&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is it that when the function $w \cdot x + b = 0$ the output is zero as well. Why couldn't it have been set to 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one were to simulate the behavior of the perceptron with a sigmoid function, $w \cdot x + b$ can be multiplied by any arbitrary constant $C&amp;gt;0$ (as $C$ tends to infinity) as long as $w \cdot x+b$ is not equal to 0. This is because at 0 condition, the firing always gives output as 1 instead of zero it should have giving. However, changing the firing rule overcomes this problem. Then, is there any reason why 0 should be the output when $w \cdot x+b= 0$?   &lt;/p&gt;&#xA;" OwnerUserId="30397" LastEditorUserId="29575" LastEditDate="2017-08-28T20:15:49.580" LastActivityDate="2017-10-28T07:48:07.797" Title="Criterion for Firing a perceptron" Tags="|neural-network|perceptron|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22662" PostTypeId="1" CreationDate="2017-08-28T20:27:16.347" Score="-1" ViewCount="133" Body="&lt;p&gt;In the current problem I am working on I am running text classification against ~16 possible variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we assume there's equal number of every label our baseline would be &lt;code&gt;1/16 = 6.25%&lt;/code&gt;. Using scikit learn, TF-IDF, ngrams and running text normalization the best results I am yielding so far are (without running gridsearch on the parameters):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;52% for SGDClassifier&lt;/li&gt;&#xA;&lt;li&gt;49% for RandomForestClassifier&lt;/li&gt;&#xA;&lt;li&gt;49% for DecisionTreeClassifier&lt;/li&gt;&#xA;&lt;li&gt;38% for MultinomialNaiveBayes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not the best results for the final product but already a huge improvement on the baseline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are better approaches to improving the results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea is to add a label and a sub-label, break the 16 into 4 groups and then run 2 models, one against the 4 major groups, and a second against the 4 minor groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other approaches can include SVMClassifier, KNN, OnevsRestClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is better approach to dealing with an array of labels for text classification to try and improve upon a 52% accuracy?&lt;/p&gt;&#xA;" OwnerUserId="38477" LastActivityDate="2017-08-29T02:24:55.180" Title="Handling Larger number of category label in text classification" Tags="|machine-learning|python|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ClosedDate="2017-09-05T12:24:46.267" ContentLicense="CC BY-SA 3.0" />
  <row Id="22663" PostTypeId="2" ParentId="8625" CreationDate="2017-08-28T23:07:58.010" Score="2" Body="&lt;p&gt;Just for fun you can compute the feature by hand by forming tuples $seq =(d_1,...,d_N)$ such that $Sum(seq) = \sum^N_{i=1} \leq D$. Once you form those tuples each entry indicates the power the current raw feature should be raised by. So say $(1,2,3)$ would map to the monomial $x_1 x_2^2 x_3^3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code to get the tuples is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generate_all_tuples_for_monomials(N,D):&#xA;    if D == 0:&#xA;        seq0 = N*[0]&#xA;        sequences_degree_0 = [seq0]&#xA;        S_0 = {0:sequences_degree_0}&#xA;        return S_0&#xA;    else:&#xA;        # S_all = [ k-&amp;gt;S_D ] ~ [ k-&amp;gt;[seq0,...,seqK]]&#xA;        S_all = generate_all_tuples_for_monomials(N,D-1)# S^* = (S^*_D-1) U S_D&#xA;        print(S_all)&#xA;        #&#xA;        S_D_current = []&#xA;        # for every prev set of degree tuples&#xA;        #for d in range(len(S_all.items())): # d \in [0,...,D_current]&#xA;        d = D-1&#xA;        d_new = D - d # get new valid degree number&#xA;        # for each sequences, create the new valid degree tuple&#xA;        S_all_seq_for_deg_d = S_all[d]&#xA;        for seq in S_all[d]:&#xA;            for pos in range(N):&#xA;                seq_new = seq[:]&#xA;                seq_new[pos] = seq_new[pos] + d_new # seq elements dd to D&#xA;                if seq_new not in S_D_current:&#xA;                    S_D_current.append(seq_new)&#xA;        S_all[D] = S_D_current&#xA;        return S_all&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then it should be easy to do regression if you know linear algebra.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c = pseudo_inverse(X_poly)*y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;example. Probably better to do regularized linear regression though if your interested in generalization.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Acknowledgements to &lt;a href=&quot;https://cs.stackexchange.com/questions/80549/how-does-one-generate-all-the-terms-of-a-multivariate-polynomial-algorithmically/80550#comment171459_80550&quot;&gt;Yuval is CS exchange&lt;/a&gt; for the help. &lt;/p&gt;&#xA;" OwnerUserId="2447" LastActivityDate="2017-08-28T23:07:58.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22665" PostTypeId="1" CreationDate="2017-08-29T01:23:34.263" Score="0" ViewCount="4906" Body="&lt;p&gt;I have the following list of numbers (1 – 5  categories, first row) that I am trying to predict the next sequence for each column (total column sum = 6). There are over 1,500 lines of data in total. I have tried to make a column of the accumulated sum of each line and then use Linear Regression and Time Series Forecasting to predict the next accumulated sum and then subtract the previous number to predict the number (0,1,2,3,4) but I am not getting the results that I am after. &#xA;How best can I forecast the next series of numbers using a non-Linear approach ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lIi9V.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lIi9V.png&quot; alt=&quot;predict next number&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38576" LastActivityDate="2023-07-22T22:33:39.677" Title="Predict next series of numbers based on previous data" Tags="|machine-learning|time-series|linear-regression|forecasting|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22666" PostTypeId="2" ParentId="22662" CreationDate="2017-08-29T02:24:55.180" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2vec&lt;/a&gt; might help. Embed each document into a high dimensional space. Then use a classification algorithm (e.g., k-nearest neighbors) on the created vectors.  &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gensim&lt;/a&gt; is a Python package for &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/blob/develop/tutorials.md#tutorials&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-29T02:24:55.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22667" PostTypeId="1" CreationDate="2017-08-29T03:23:24.343" Score="0" ViewCount="92" Body="&lt;p&gt;I am interested in learning more about the field of ML and more specifically in text classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a piece of text, is there any technique or algorithm to determine if that piece of text refers to a job ad or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="38579" LastActivityDate="2023-12-09T15:05:40.630" Title="Determine if a piece of text is a job ad" Tags="|machine-learning|classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22668" PostTypeId="2" ParentId="22667" CreationDate="2017-08-29T03:48:21.350" Score="0" Body="&lt;p&gt;For text classification in your case, it can be solved by training a binary classes classification network.&#xA;Main steps as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Training data preparation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two classes data are needed, one class contain job ad text, and the other does not contain job ad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Building the network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Building a binary classification network with Keras, Caffe or any other DL library you like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Train the network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training the network on the data prepared above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. Text class prediction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the trained network to predict the text class you want to deal with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It’s a easy classification task for neural network: )&lt;/p&gt;&#xA;" OwnerUserId="38419" LastActivityDate="2017-08-29T03:48:21.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22669" PostTypeId="2" ParentId="18752" CreationDate="2017-08-29T06:36:32.197" Score="1" Body="&lt;p&gt;Yes, LSTMs are fairly used in time series prediction. They can even handle missing data which is quite common in timeseries: &lt;a href=&quot;https://arxiv.org/pdf/1511.03677.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning to diagnose with LSTM recurrent neural network&lt;/a&gt;. Still, I don't recommend starting directly with LSTM as the training takes time and you have to try many parameters to find the best that fits your data. You should start with some basic regressors like RandomForest or XGBoost.  &lt;/p&gt;&#xA;" OwnerUserId="38355" LastEditorUserId="38355" LastEditDate="2017-10-30T14:44:01.237" LastActivityDate="2017-10-30T14:44:01.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22670" PostTypeId="2" ParentId="22665" CreationDate="2017-08-29T06:41:24.140" Score="1" Body="&lt;p&gt;You could use LSTMs with a sequence to sequence model perhaps (using e.g. Keras). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;x lines are input, the next line is output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just an idea, your question is not that clear. &lt;/p&gt;&#xA;" OwnerUserId="37808" LastActivityDate="2017-08-29T06:41:24.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22671" PostTypeId="2" ParentId="22661" CreationDate="2017-08-29T06:49:49.423" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The perceptron algorithm was invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt (&lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia EN&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It was the first neural network like architecture. It was thought that (a combination of) perceptrons could learn anything. There was a lot of controversy about this and after a while Marvin Minsky showed that a single perceptron could not solve the XOR problem. This was the end of the first neural network hype. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After a couple of decades the multilayer perceptron was invented, and with it the backpropagation algorithm. The backpropagation algorithm can learn weights that are not directly connected to the output unit. However, it needs a differentiable activation function. Hence, the ramp activation function was not sufficient anymore and others like the sigmoid and tanh were used for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, the perceptron was first implemented using electronic boards and not with the software available to us today. Binary thresholds made this complicated process a lot easier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in short Rosenblatt could have chosen for any version of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Heaviside_step_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;heaviside step function&lt;/a&gt; but he chose for the one where $w \cdot x + b = 0$ leads to a zero output. If you think binary values this makes sense, a zero input (i.e. False) leads to a zero output. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="20372" LastEditDate="2017-08-29T06:55:02.500" LastActivityDate="2017-08-29T06:55:02.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22672" PostTypeId="2" ParentId="21820" CreationDate="2017-08-29T07:38:10.880" Score="1" Body="&lt;p&gt;Have you read Doersch's &lt;a href=&quot;https://arxiv.org/pdf/1606.05908.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial on VAE&lt;/a&gt;? In paragraph 1.1 the author shows main intuitions behind the latent model and in the following sections he goes deeper into the explanation of how a VAE works. I think it's a good starting point to approach to VAEs.&lt;/p&gt;&#xA;" OwnerUserId="38320" LastActivityDate="2017-08-29T07:38:10.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22675" PostTypeId="2" ParentId="18752" CreationDate="2017-08-29T09:14:29.493" Score="0" Body="&lt;p&gt;You have asked about &lt;em&gt;any&lt;/em&gt; pointers. What about applying some basic/intermediate probability theory? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you already have the probabilities that a machine fails, you can use the &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Geometric_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Geometric Distribution&lt;/a&gt;&lt;/em&gt; to calculate the probabilites that a machine fails on the next day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;For a machine that breaks with 10% probability on a single day:&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This example is using R's built-in &lt;code&gt;pgeom()&lt;/code&gt; function)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# probability that the machine breaks on 5th day or earlier&#xA;pgeom(4, 0.1)&#xA;# 0.4095&#xA;&#xA;# probability that the machine is still working on 20th day&#xA;1 - pgeom(19, 0.1)&#xA;# 0.1216&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Maybe you can use an educated guess for your probabilities as a baseline model for more sophisticated neural network models.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-08-29T09:14:29.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22677" PostTypeId="2" ParentId="22624" CreationDate="2017-08-29T09:48:46.353" Score="1" Body="&lt;p&gt;You should also have a look at the Cross Validated forum on StackExchange a lot questions asked there are related to time series analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are looking for seems like a &quot;Dynamic Regression or &quot;Intervention Analysis&quot; &lt;a href=&quot;https://www.otexts.org/fpp/9/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.otexts.org/fpp/9/1&lt;/a&gt; (with examples in R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An here &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat510/node/72&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://onlinecourses.science.psu.edu/stat510/node/72&lt;/a&gt; you will find a procedure which does an ordinary linear regression but investigates the autocorrelated errors caused by the time series structure of your data (also with an example in R)&lt;/p&gt;&#xA;" OwnerUserId="38583" LastActivityDate="2017-08-29T09:48:46.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22678" PostTypeId="1" CreationDate="2017-08-29T10:58:07.203" Score="0" ViewCount="94" Body="&lt;p&gt;At the moment we use different methods for record linking locations in different datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Theoretically given two locations we can give a prediction on how well they match (are the same). This is not just based on address data (street, house number, zip, city, country, latitude, longitude) but also based on the name, type of establishment and other properties like phonenumber.&#xA;Since most features are prone to fuzzy errors (different spellings, writing styles, formatting, human entry error, null-values (absent)) this means we cannot do strict comparisons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using many business rules, basic distance calculations, etc.  we can clean &amp;amp; reformat the data so we can have a pretty good comparison and high accuracy on matches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;eg. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Datlinq BV, &#xA;  Roer 266,&#xA;  2908MC, Capelle aan den IJssel &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is the same as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;datlinq,&#xA;  266 Roer,&#xA;  capelle a/d yssel&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but not as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Ferro BV,&#xA;  Roer 266,&#xA;  2908MC, Capelle aan den IJssel &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Our problem is many a scaling issue:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need smart ways to reduce the dataset so we don't compare every location in dataset A to every location dataset B. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this moment we do some pre-filtering (boxing) on latitude / longitude (if present)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I was wondering if we could somehow generate a vector for each data location and us a basic distance calculation to find matches (or match candidates) in other datasets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe another way to find very similar locations across huge datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd love to use a machine learning model for this, but I am stumped how to generate numerical features out of a string address, location type &amp;amp; telephone number, without applying to many business rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So the actual question is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there smart ways to do (parallel) record linking across big datasets (using Spark or so)&lt;/li&gt;&#xA;&lt;li&gt;Is the approach of converting locations to vectors using a ml model and then using a distance calculation to find nearby points a good approach?&lt;/li&gt;&#xA;&lt;li&gt;Is there terminology I should use to find more information about this problem? Are there papers / experiments?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-- Update 20170829 151213&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we are working on prefiltering datasets based on nearby locations, but unfortunately for many datasets we don't have reliable lat/longs. So I'm looking for a more general approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So based on the fact that we cannot positively resolve a nearby location without comparing textual addresses. A smart heuristic is something I'm looking for, but I want to incorporate weights to certain (fuzzy) features, eg. zipcodes should match more closely towards the beginning then towards the end, phonenumbers other way around. Streets and cities maybe spelled very differently including abbreviations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure I can extract all these features and create business rules with arbitrary weights, but I'd rather have an unsupervised ML model learn the most important features based on positive and negative test/trainingsets of address matches.&#xA;I would also guess that weights will vary differently by country/language so maintaining these by hands seems a suboptimal solution.&lt;/p&gt;&#xA;" OwnerUserId="38584" LastEditorUserId="38584" LastEditDate="2017-08-29T13:11:44.837" LastActivityDate="2017-08-29T14:27:04.967" Title="Distance measure calculation addresses for record linking" Tags="|feature-extraction|feature-engineering|distance|scalability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22679" PostTypeId="1" AcceptedAnswerId="22689" CreationDate="2017-08-29T11:30:04.663" Score="0" ViewCount="981" Body="&lt;p&gt;I have multi-touch attribution data like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    medium  conversion&#xA;1   organic 0&#xA;2   (none) &amp;gt; referral &amp;gt; referral &amp;gt; (none) &amp;gt; (none) &amp;gt; referral   0,0,0,0,0,0&#xA;3   (none)  0&#xA;4   organic &amp;gt; referral &amp;gt; referral   0,1,0&#xA;5   referral &amp;gt; referral &amp;gt; referral &amp;gt; referral   0,0,1,0&#xA;6   organic &amp;gt; referral &amp;gt; referral &amp;gt; (none) &amp;gt; referral   0,1,0,1,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd like to remove rows with no conversions (like rows 1, 2 and 3) and I tried grepl but couldn't make it work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the remaining rows how to split rows so it ends with a conversion, e.g. row 4 will be organic &gt; referral 0,1&#xA;and row 6 will split into organic &gt; referral 0,1 and organic &gt; referral &gt; referral &gt; (none) 0,1,0,1&lt;/p&gt;&#xA;" OwnerUserId="38589" LastActivityDate="2017-08-30T17:18:27.500" Title="Slice rows in R based on column value" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22680" PostTypeId="2" ParentId="22678" CreationDate="2017-08-29T11:37:51.207" Score="1" Body="&lt;p&gt;Is the problem that you &lt;em&gt;can&lt;/em&gt; resolve a new address B to a location reliably, but, don't know if it's the same address as an existing address A? then it seems straightforward to perform the expensive check for near-matches on only addresses that also resolve to a nearby location, and consider them in order of distance. The simple distance calculation shouldn't be expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or are you saying that you can't resolve B to a location by itself, and have to resolve it by finding a fuzzy match with some A, but you don't know where A might be? Then you need some cheap heuristic match to compute over all records. What about simply the distribution of characters/digits in every string? easy to compute, small, and easy to compare. Presumably near-matches have nearly the same letters and digits, so might correlate well with actual fuzzy matches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited to add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try to build a classifier that can tell when two different strings represent the same address. That's not your ultimate goal, I know. But along the way you might be able to learn a useful intermediate vector representation of the strings that is relevant for determining same/not same. And then use that intermediate representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory that's what a neural network could do. Inputs are two strings; they share weights/layers that transform to an intermediate rep; then the top of the network learns same/not same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Simple as that&quot; but in practice you might find that learning this way takes such a huge network to do well that it's infeasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe someone knows an equivalent idea that doesn't need the brute force of deep learning.&lt;/p&gt;&#xA;" OwnerUserId="21" LastEditorUserId="21" LastEditDate="2017-08-29T14:27:04.967" LastActivityDate="2017-08-29T14:27:04.967" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22683" PostTypeId="1" CreationDate="2017-08-29T12:51:41.803" Score="2" ViewCount="94" Body="&lt;p&gt;I trying to predict sales quantity of particular item by using attributes of item.&lt;strong&gt;Not by using Time series&lt;/strong&gt;.I have currently 3 years of past data.I am performing this by using &lt;strong&gt;Neural Networks&lt;/strong&gt;.&lt;br&gt;&#xA;&lt;strong&gt;What is the Error metric need to follow &#xA;in this case ?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-11-27T18:25:09.637" Title="Error Metric for Prediction of sales quantity (by using Attributes of item)" Tags="|machine-learning|neural-network|predictive-modeling|regression|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22684" PostTypeId="1" AcceptedAnswerId="22686" CreationDate="2017-08-29T13:46:51.160" Score="2" ViewCount="4730" Body="&lt;p&gt;From &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/object_detectionR-CNN&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow's object recognition (R-CNN)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm re-training the existing model with new categories: the types of clothes (jeans, pants, blouse, and so on). Since we don't need colors to determine the type of clothes that user is wearing, I want to re-train it with gray-scale images. Is it possible to use gray-scale images to train existing model (which are trained with color images)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm concerned because they trained their model with &lt;strong&gt;color images&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the model just consider the grayscale image as color image? And does it still work? :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s I'm generating XML and csv files to put data for training and testing. &lt;/p&gt;&#xA;" OwnerUserId="38593" LastEditorUserId="29169" LastEditDate="2021-02-15T22:24:56.607" LastActivityDate="2021-02-15T22:24:56.607" Title="Is it possible to use grayscale images to existing model?" Tags="|deep-learning|tensorflow|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22685" PostTypeId="2" ParentId="22632" CreationDate="2017-08-29T14:29:08.520" Score="2" Body="&lt;p&gt;I figured it out. If we denote the additional features as &lt;code&gt;x_feat&lt;/code&gt;, I changed the lines from&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])&#xA;h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])&#xA;h_pool2_flat = tf.concat( [h_pool2_flat, x_feat ], 1 )&#xA;h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15850" LastActivityDate="2017-08-29T14:29:08.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22686" PostTypeId="2" ParentId="22684" CreationDate="2017-08-29T14:46:56.827" Score="1" Body="&lt;p&gt;It depends on the model. You'll have to dig into the model definition and see. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's not uncommon for models to grayscale images in pre-processing. In that case, you will be fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, many recent deep convolutional models operate on RGB images as 3D tensors (4D with the batch dimension). In this case, you should consider modifying the model so that it operates on a single color channel before training it. Multi-channel convolution is discussed in the convent's chapter of Goodfellow's &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38379" LastActivityDate="2017-08-29T14:46:56.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22687" PostTypeId="1" AcceptedAnswerId="22688" CreationDate="2017-08-29T15:09:58.410" Score="-1" ViewCount="213" Body="&lt;p&gt;Today I encountered this strange behavior in Python doing data manipulation. Why changing a will affect b below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; a = ['Hello', 1, 2]&#xA;&amp;gt;&amp;gt;&amp;gt; b = a&#xA;&amp;gt;&amp;gt;&amp;gt; a[0] = 5&#xA;&amp;gt;&amp;gt;&amp;gt; b&#xA;[5, 1, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I only asked a to change, why is b changing? But the following is fine,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; a = 3&#xA;&amp;gt;&amp;gt;&amp;gt; b = 4&#xA;&amp;gt;&amp;gt;&amp;gt; a&#xA;3&#xA;&amp;gt;&amp;gt;&amp;gt; b&#xA;4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My guess is that I am doing passing by reference(?). But in both cases are passing by reference, what is going on here?&lt;/p&gt;&#xA;" OwnerUserId="386" LastActivityDate="2017-08-29T15:30:07.287" Title="Why changing one element of a vector will change all variables?" Tags="|python|data-cleaning|" AnswerCount="1" CommentCount="1" ClosedDate="2017-08-29T19:51:31.133" ContentLicense="CC BY-SA 3.0" />
  <row Id="22688" PostTypeId="2" ParentId="22687" CreationDate="2017-08-29T15:22:35.097" Score="2" Body="&lt;p&gt;In your first case, &lt;code&gt;list&lt;/code&gt; what you're doing is copying a list, which is referencing a list to another list (ex: list &lt;code&gt;b&lt;/code&gt; is referenced by list &lt;code&gt;a&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid that, you have to consider &lt;code&gt;deepcopy&lt;/code&gt; in python in order to safely make edits to the copied list without those changes being reflected in the original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; import copy&#xA; a = ['Hello', 1, 2]&#xA; b = copy.deepcopy(a)&#xA;&#xA; a[0] = 5&#xA; print(b)  # This will give the original list&#xA; &amp;gt;&amp;gt; ['Hello', 1, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;a href=&quot;https://docs.python.org/2/library/copy.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; that will be helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to get around as @Sophie mentioned in the comment is slicing,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b= a[:]&#xA;&#xA;a[0] = 5&#xA;&#xA;print(a)&#xA;&amp;gt;&amp;gt; [5, 1, 2]&#xA;&#xA;print(b)&#xA;&amp;gt;&amp;gt; ['Hello', 1, 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In your second case, it is just a reference to a variable as you mentioned in the question. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-08-29T15:30:07.287" LastActivityDate="2017-08-29T15:30:07.287" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22689" PostTypeId="2" ParentId="22679" CreationDate="2017-08-29T16:20:34.537" Score="1" Body="&lt;p&gt;This can be done with packages from the &lt;a href=&quot;http://dplyr.tidyverse.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;dplyr&lt;/a&gt; which is part of the &lt;a href=&quot;http://www.tidyverse.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tidyverse&lt;/a&gt; made by Hadley Wickham. The &lt;a href=&quot;https://github.com/tidyverse/stringr&quot; rel=&quot;nofollow noreferrer&quot;&gt;stringr&lt;/a&gt; package (also made by Hadley) is really helpfull in working with vectors of strings. Another package &lt;a href=&quot;https://github.com/tidyverse/purrr&quot; rel=&quot;nofollow noreferrer&quot;&gt;purrr&lt;/a&gt; is helpfull for applying functions t lists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, lets import the libraries and create data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse)&#xA;library(stringr)&#xA;&#xA;df = tibble(&#xA;  medium = c(&quot;organic&quot;, &quot;(none) &amp;gt; referral &amp;gt; referral &amp;gt; (none) &amp;gt; (none) &amp;gt; referral&quot;, &quot;(none)&quot;, &quot;organic &amp;gt; referral &amp;gt; referral&quot;, &quot;referral &amp;gt; referral &amp;gt; referral &amp;gt; referral&quot;, &quot;organic &amp;gt; referral &amp;gt; referral &amp;gt; (none) &amp;gt; referral&quot;),&#xA;  conversion = c(&quot;0&quot;, &quot;0,0,0,0,0,0&quot;, &quot;0&quot;, &quot;0,1,0&quot;, &quot;0,0,1,0&quot;, &quot;0,1,0,1,0&quot;)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You probably have the data as strings while a list represention would be much easier to work with. The following code converts the strings to lists by splitting on &quot;&gt;&quot; or &quot;,&quot; (resp. for medium and conversion). Also read upon the &lt;a href=&quot;https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;%&amp;gt;%&lt;/code&gt;&lt;/a&gt; which is really handy for working with data frames. After converting the strings to lists the conversion column can be made boolean by mapping the &lt;code&gt;==&lt;/code&gt; operator on each row.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- df %&amp;gt;%&#xA;  mutate(medium = str_split(medium, &quot;&amp;gt;&quot;), &#xA;         conversion = str_split(conversion, &quot;,&quot;), &#xA;         conversion = map(conversion, `==`, &quot;1&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Removing the items after the first conversion is now a matter of simple indexing. A little magic is done with &lt;code&gt;dplyr::lag&lt;/code&gt; and &lt;code&gt;cumsum&lt;/code&gt; to get a boolean list that indicates all the steps before and during conversion. Then &lt;code&gt;map&lt;/code&gt; is used to get (using the &lt;code&gt;[&lt;/code&gt; operator) all the mediums by boolean indexing with &lt;code&gt;no_conversion_yet&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- df %&amp;gt;%&#xA;  mutate(no_conversion_yet = map(conversion, function(x) dplyr::lag(cumsum(x) &amp;lt; 1, default=TRUE))) %&amp;gt;%&#xA;  mutate(medium = map(no_conversion_yet, medium, `[`))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Filtering the rows that do not have a conversion is now easy. Simple remove all the rows that do not have any &lt;code&gt;TRUE&lt;/code&gt; value in the conversion column. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- df %&amp;gt;%&#xA;  mutate(any_conversion = map_lgl(conversion, any)) %&amp;gt;%&#xA;  filter(any_conversion)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Voila, life made easy by the superb packages of Hadley Wickham! (Also check his &lt;a href=&quot;http://vita.had.co.nz/papers/tidy-data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="20372" LastEditDate="2017-08-30T17:18:27.500" LastActivityDate="2017-08-30T17:18:27.500" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22690" PostTypeId="1" CreationDate="2017-08-29T16:48:40.890" Score="19" ViewCount="13548" Body="&lt;p&gt;I'm wondering in what situations it is advantageous to stack LSTMs?&lt;/p&gt;&#xA;" OwnerUserId="18418" LastEditorUserId="843" LastEditDate="2017-08-29T21:08:20.647" LastActivityDate="2019-09-06T11:13:26.203" Title="Advantages of stacking LSTMs?" Tags="|machine-learning|neural-network|deep-learning|lstm|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22691" PostTypeId="2" ParentId="22683" CreationDate="2017-08-29T16:53:16.193" Score="1" Body="&lt;p&gt;This is a fairly common task, especially in fields like economics, but the choice of using a Neural Network makes your approach different from most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, I think you can still use common validation metrics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean Error&lt;/li&gt;&#xA;&lt;li&gt;Mean Absolute Error&lt;/li&gt;&#xA;&lt;li&gt;Absolute Error (&quot;delta&quot;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You probably don't need anything more fancy than that, especially if those are the error metrics that your business clients will be using to measure your success (which I imagine would be the case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, if you want a fancier ML metric to use whilst developing your algorithm you might try Root Mean Squared Error (RMSE). Here's an &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0895717794002075&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; on the topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to, you could also construct a pseudo-R-squared metric, but those are less used with ANN's and I wouldn't choose it myself for this particular case.&lt;/p&gt;&#xA;" OwnerUserId="2723" LastActivityDate="2017-08-29T16:53:16.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22694" PostTypeId="2" ParentId="22639" CreationDate="2017-08-29T19:10:12.120" Score="3" Body="&lt;p&gt;As per Andrew Ng's Course, if you use the sigmoid activation, the outputs represent the probability of either outcome 0 or outcome 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the decision boundary is 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if prediction &gt; 0.5 , the prediction is 1 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if prediction &amp;lt;= 0.5 , the prediction in 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a screenshot from Andrew Ng's &lt;a href=&quot;https://www.youtube.com/watch?v=UJWfVf69C4k&quot; rel=&quot;nofollow noreferrer&quot;&gt;slides&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kBQGT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kBQGT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have gone through your code and results. There seems to be something wrong with the implementation as none of your predictions give a value greater than 0.5 .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't pin down the problem. Some debugging will be needed on your side. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's paradoxical is that your loss is reducing. I suspect this might be because &lt;strong&gt;your data set is unbalanced&lt;/strong&gt; i.e., you have 221 0s and about 30 1s. This could be the reason for other problems as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt; example, where the values are correctly matching.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WuYn2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WuYn2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you manage to find the error or the solution conclusively, please post it here so that we can all learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-08-29T19:10:12.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22696" PostTypeId="1" CreationDate="2017-08-29T19:49:52.823" Score="2" ViewCount="123" Body="&lt;p&gt;What constitutes as a &quot;good enough&quot; score for a Decision Tree Regressor? The &lt;code&gt;.score()&lt;/code&gt; function gives us a general score about our model. This can be 1 if the model predicts all data with a 100% accuracy, and can be arbitrarily worse. If I understand correctly, a score of 0 means the prediction is quasi constant. But starting with what value can we say that the prediction is &quot;usable&quot; (i know this is ambiguous, but still). Is a score of 0.4-0.5 good enough to use the predictions in practice as opposed to not using any machine learning techniques?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2020-07-12T21:07:15.853" Title="SKLearn DT regressor - good enough score?" Tags="|scikit-learn|regression|decision-trees|score|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22697" PostTypeId="1" CreationDate="2017-08-29T20:15:30.437" Score="2" ViewCount="101" Body="&lt;p&gt;I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is there any algorithm/application on which you could use Pareto/weak Pareto optimal points as the optimal parameters? I was wondering whether there exists something like a multiobjective version of logistic regression or least squares. Any pointer to the literature will also be very helpful.&lt;/p&gt;&#xA;" OwnerUserId="38608" LastActivityDate="2017-09-01T06:53:38.223" Title="What are some machine learning problems that can be attacked with continuous multiobjective optimization?" Tags="|deep-learning|svm|logistic-regression|optimization|gradient-descent|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22699" PostTypeId="1" CreationDate="2017-08-30T07:25:12.133" Score="2" ViewCount="621" Body="&lt;p&gt;I am new to Data Science and I am trying to understand the 'Word2Vec' approach from a long time. Can someone please explain it in simple terms? Moreover, what are the problems that can be solved through 'Word2Vec' approach?&lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2017-08-30T11:28:58.153" Title="What is Word2Vec approach" Tags="|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22702" PostTypeId="1" AcceptedAnswerId="22704" CreationDate="2017-08-30T08:49:06.410" Score="2" ViewCount="52" Body="&lt;p&gt;Is it possible to use a trained CNN to generate data? After training on data of X and corresponding Ys, given a new Y to generate X? &#xA;Or do I have to use GANs or RNNs? I'm still pretty new to the subject so I would be happy to have some leads, if I have to go in this direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am learning tensorflow  and trying to implement the &quot;Convolutional Sequence to Sequence Learning&quot; paper&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarification: I want to use a CNN to generate sequences of text. I have a reallyb big database of descriptions and also one of 1000 classes, and wanted to try it out with CNNs (since im coming from image classification)&lt;/p&gt;&#xA;" OwnerUserId="37118" LastEditorUserId="37118" LastEditDate="2017-08-30T09:36:51.300" LastActivityDate="2017-08-30T09:36:51.300" Title="Generation/Synthesis of Data with CNNs" Tags="|convolution|gan|generative-models|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22703" PostTypeId="1" CreationDate="2017-08-30T08:54:29.443" Score="6" ViewCount="1280" Body="&lt;p&gt;Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to infer this information by looking directly into the co-occurrence matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, I cannot understand those methods aiming to capture as much relevant information from the original co-occurrence matrix as possible. Isn't it easier to work on the co-occurrence matrix directly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much in advance.&lt;/p&gt;&#xA;" OwnerUserId="13618" LastActivityDate="2017-08-30T14:19:15.570" Title="What are the advantages of Skip-Gram methods?" Tags="|word2vec|word-embeddings|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22704" PostTypeId="2" ParentId="22702" CreationDate="2017-08-30T09:17:35.803" Score="1" Body="&lt;p&gt;In general, if you want to generate data in X, or X, Y pairs, then you should start by training a generative model - as opposed to a discriminative model, which is what most NN classifiers are. There are many types of generative model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://kvfrans.com/variational-autoencoders-explained/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Variational Autoencoders&lt;/a&gt; (VAE)and &lt;a href=&quot;http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Networks&lt;/a&gt; (GAN) have been demonstrated recently with interesting results on images - although both take a lot of training data and time, and are limited to relatively small image dimensions (e.g. 128x128). There are many sub-types of these two designs, including a &lt;a href=&quot;https://github.com/anitan0925/vaegan&quot; rel=&quot;nofollow noreferrer&quot;&gt;combined VAEGAN&lt;/a&gt; which attempts to combine strengths of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a trained CNN, you &lt;em&gt;can&lt;/em&gt; generate data - sort of. What you can do is start with some arbitrary value of X and Y, then use back propagation to calculate gradients of a cost function. But instead of using the gradients to update weights, you back propagate all the way to the input, and use the gradients at the input to alter X, repeating the process multiple times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is essentially how Deep Dream and Style Transfer work (although in general these don't use a Y value, but selected activation values within layers). There is a major caveat to this approach - your generated X will not be sampled evenly from any distribution of X that the network has been trained with. Instead you will generate a &quot;super stimulus&quot; X for the given Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mention RNNs. One way these can be used to generate X is by &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sampling from their output and feeding this back into the input&lt;/a&gt;. For text sequences this tends to generate grammatically correct nonsense. I am not sure if this would be considered a strictly generative model, since it is not clear to me whether the input X is being sampled evenly. It is likely that you could use the approach to generate images too, although you would have to take care defining what the sequence is (just a sequence of pixels line-by-line will probably not produce any recognisable image).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-08-30T09:26:14.183" LastActivityDate="2017-08-30T09:26:14.183" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22705" PostTypeId="1" CreationDate="2017-08-30T10:01:36.787" Score="2" ViewCount="322" Body="&lt;p&gt;The question is about a wrongly chosen strategy for &lt;strong&gt;train/test splitting&lt;/strong&gt; in a RandomForest model. I know choosing the test set this way gives the wrong output but I would like to understand &lt;strong&gt;why&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The model looks at previous days of data and tries to predict if next day's data will be higher or lower than today, i.e. a classification problem)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I copied the train/test split code from another example and it simply sets random rows to be either train_set or test_set. (Tried to illustrate below) &#xA;&lt;a href=&quot;https://i.stack.imgur.com/9or6s.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9or6s.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Raw data is daily Close values of for example EURUSD.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then create features based on that. Every feature looks at some previous data points and comes up with a set of features which is a row in X_test. I then train a random forest model to try to predict next day's close.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy in the test_set is very high and it increases with the increasing number of historical previous points it looks at which seems to suggest overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I change train/test split model to have, for example, &lt;strong&gt;train_set: data in January-June&lt;/strong&gt; and &lt;strong&gt;test_set: data in August&lt;/strong&gt;, i.e. completely separate datasets and no possible mixing, the accuracy is a more realistic 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, I know the train/test split it is not correct, but can someone help me understand why..?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is simple: Every time I want to validate a row (i.e. one prediction in test_set) I use features that looks at previous data trying to predict tomorrow's data? How come there is overfitting?&lt;/p&gt;&#xA;" OwnerUserId="38626" LastActivityDate="2017-08-30T10:01:36.787" Title="Wrong train/test split strategy" Tags="|random-forest|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22706" PostTypeId="1" CreationDate="2017-08-30T10:03:49.240" Score="6" ViewCount="983" Body="&lt;h1&gt;The use case :&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Everyday, we have metrics that are established daily to check that our systems are doing fine. From time to times, bugs occur in the workflow building these metrics, and I have to build an algorithm that will alert us when it seems there is a problem, so that we can check where it is coming from.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What I'm trying :&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I don't know much about time series. The data I have displays a weekly seasonality, and the volume isn't constant (it's growing). See image : &lt;a href=&quot;http://www.blabala.com&quot; rel=&quot;noreferrer&quot;&gt;http://www.blabala.com&lt;/a&gt; (I don't have enough rep to post multiple images)&#xA;Therefore, from what I read, ARIMA seems to be a relevant algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I stumbled upon the &lt;code&gt;tsoutlier&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt;, and its functions &lt;code&gt;locate.outliers.oloop()&lt;/code&gt; and &lt;code&gt;discard.outliers()&lt;/code&gt;, so I had the following idea: instead of trying to train something I don't master, let's have the automatic parts of the algorithm train on their own, and use that: if it detects outliers, then throw an alert.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First question: does it seem to be a good or a bad idea?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;If it's a good approach :&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I'm having trouble getting something that works well. I've explored two ways of doing things : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) train the whole thing, and run it on itself.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cval &amp;lt;- 3&#xA;mo2 &amp;lt;- locate.outliers.oloop(TS_train, &#xA;                             arima_train, &#xA;                             maxit.iloop = 100, &#xA;                             maxit.oloop = 100, &#xA;                             cval = cval)&#xA;d2&amp;lt;-discard.outliers(mo2,TS_train, method = &quot;bottom-up&quot;,&#xA;                     tsmethod.call = arima_train$call,&#xA;                     cval = cval)&#xA;&#xA;OTL &amp;lt;-  TS_train[sort(d2$outliers$ind)]&#xA;&#xA;plot(1:length(TS_train), TS_train, type = 'p')&#xA;lines(1:length(TS_train), TS_train, col = &quot;grey&quot;)&#xA;points(sort(d2$outliers$ind), OTL, col = &quot;red&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This hasn't given very good results. See image 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) have a sliding window:&lt;/strong&gt; train the algorithm with day 1 to 60 as training data, evaluate day 61, slide one day, train the algorithm with day 2 to 61 as training data, evaluate day 62, slide one day, and so on. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plage &amp;lt;- 60&#xA;cval &amp;lt;- 3&#xA;ylim &amp;lt;-  range(DT$col1) + c(-1,1) * .05 * diff(range(DT$col1))&#xA;&#xA;for( i in (plage + 1):65 ) { &#xA;  TS_tmp &amp;lt;- ts(DT[(i-plage):i]$col1, frequency = 7)&#xA;  aa &amp;lt;- auto.arima(TS_tmp)&#xA;  mo_tmp &amp;lt;- locate.outliers.oloop(TS_tmp, aa, &#xA;                                  maxit.iloop = 100, maxit.oloop = 100, &#xA;                                  cval = cval)&#xA;  d_tmp &amp;lt;- discard.outliers(mo_tmp, TS_tmp, method = &quot;bottom-up&quot;,&#xA;                            tsmethod.call = aa$call, cval = cval)&#xA;&#xA;  Outl &amp;lt;- TS_tmp[sort(d_tmp$outliers$ind)]&#xA;  print(plot(DT[(i-plage):i]$col1, type = &quot;b&quot;, xlim = c(1, nrow(DT)), ylim = ylim))&#xA;  print(points(sort(d_tmp$outliers$ind), Outl , col = &quot;red&quot;))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems to perform worse... (See image 2) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then again, I don't know much about time series and don't really know how to evaluate how well a model performs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Are any of the two approaches a good idea ? Which one is best ? Any better idea ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm using the 9 months of data I have in approach n1. In approach n2 (if it's relevant), how much data should be included in my training set ? (currently : 60 days)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Any advice on how to best calibrate the model(s) ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0bll2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0bll2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38581" LastActivityDate="2018-07-16T13:21:02.150" Title="Anomaly detection in time series" Tags="|r|time-series|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22707" PostTypeId="1" CreationDate="2017-08-30T10:20:26.987" Score="1" ViewCount="3457" Body="&lt;p&gt;My goal is to implement an assistant for crawling web data for users that don't understand anything about HTML or DOM. I will show a web page to the user and the user has to select, what data he is interested on the page (or what data he is not interested in).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: If the user clicks on the cell inside a table, it is very likely he wants to extract all elements inside that column. He might only be interested in the table row, or he might be interested in only the one cell. So the algorithm proposes him three selectors for each possibility. The user might choose one of the proposed selector or he might click on another element to get a new proposition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far the use case. The component I want to create:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;has as input data a list of elements inside a DOM (hierarchy of HTML elements), that the user wants to crawl&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;shall output a (probability) ranked list of selectors that will fit the users selection to 100% and possibly include other elements that the user is probably interested in. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The problem here is the high variability of websites, and the possibility to create similar looking results with very differently structured elements. Manually creating rules for prediction can therefore probably cover the most basic uses (like getting data from a table column), but will fail when it comes to finding article parts on a news page. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So finally my questions. I am thinking about the following processing: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;first generate selectors that will fit the users selection &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;sort out the selectors that do not fit the users choice of elements by 100%&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;use a (machine learned?) model to predict which selector the user might be interested in&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do you think the above processing will be productive for this problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What algorithm could be most suitable for ranking the selectors in 3.?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there an idea on how to auto-magically come up with a list of selectors for the selected elements?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I am making sense - I do not have a very strong background in data analysis and machine learning so I am hoping for some directions where to look into.&lt;/p&gt;&#xA;" OwnerUserId="38630" LastActivityDate="2019-02-28T13:50:02.723" Title="Machine learning for predicting HTML Elements on a web page?" Tags="|machine-learning|data-mining|predictive-modeling|hierarchical-data-format|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22708" PostTypeId="2" ParentId="22699" CreationDate="2017-08-30T10:43:56.473" Score="3" Body="&lt;p&gt;&lt;strong&gt;What is Word2Vec?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2Vec is an implementation of word embedding techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word embedding tries to represent relationships that may exist between the individual words (those contained in your processing texts) by giving them each a vector with same predefined dimension. In this vector space words that share common contexts may be located closer. How to assign the words into the vector space? This work has often been done through neural network training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What problems can it solve?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problem solved by Word2Vec (or word embedding) is that it creates a way to represent relationships between the words in your processing texts, as opposed to just treat the words as individual symbols. This makes any following data mining or machine learning more effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides Word2Vec, you can also use many other pre-trained embeddings like &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GloVe&lt;/a&gt;. If the dataset is large, it is better to train your own word embedding which will get a better performance than using pre-trained embeddings.&lt;/p&gt;&#xA;" OwnerUserId="38419" LastEditorUserId="836" LastEditDate="2017-08-30T11:28:58.153" LastActivityDate="2017-08-30T11:28:58.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22709" PostTypeId="1" CreationDate="2017-08-30T11:59:02.383" Score="0" ViewCount="562" Body="&lt;p&gt;I am going in a very abstract layer for the problem, but I think this problem might be a common one so posting it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what I am looking for is any ML algorithm or Data Science technique is there to create a relationship factor between different sets of goods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess, its used heavily in social media, but here I am talking w.r.t. product goods context. Like suppose if we sell a bunch of products together in a store, so in collection there will be many products that are always bought together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, accordingly there will be a relationship factor between different products i.e. some product that are always bought together will have a higher factor than the one that are bought together less frequently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is there any approach where we can tackle this problem to get the factor between two products using any technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : Narrowing it down to a universe where all items are having same factors and their relationship factor to each other totally depends on their occurance together.&lt;/p&gt;&#xA;" OwnerUserId="37217" LastEditorUserId="37217" LastEditDate="2017-08-30T12:45:29.827" LastActivityDate="2017-08-31T08:04:32.617" Title="Technique/Algorithm for product categorization Machine Learning" Tags="|machine-learning|data-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22710" PostTypeId="1" AcceptedAnswerId="22744" CreationDate="2017-08-30T12:25:40.437" Score="1" ViewCount="8229" Body="&lt;p&gt;I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oQaSB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oQaSB.png&quot; alt=&quot;Network structure&quot;&gt;&lt;/a&gt; As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input &lt;code&gt;x&lt;/code&gt; will compute &lt;code&gt;c=ax+b&lt;/code&gt;, &lt;code&gt;a&lt;/code&gt; being a parameter and &lt;code&gt;b&lt;/code&gt; being its bias. Then it will feed &#xA;&lt;code&gt;c&lt;/code&gt; to Sigmoid function to compute &lt;code&gt;z=1/(1+exp(-c))&lt;/code&gt; which will be its output. Please correct me if I'm wrong in any of this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm trying to follow &lt;a href=&quot;http://karpathy.github.io/neuralnets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; to create back-propagation differential equations. It says that &lt;code&gt;dc = z * (1-z) * dz&lt;/code&gt; where dz is the gradient of the green edge and then &lt;code&gt;da = x * dc&lt;/code&gt; where &lt;code&gt;da&lt;/code&gt; is the gradient computed in this iteration of backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is that there are three &lt;code&gt;z&lt;/code&gt;s (one corresponding to each outward edge) and their value(weight) will be identical (&lt;code&gt;z=1/(1+exp(-c))&lt;/code&gt;). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute &lt;code&gt;dc&lt;/code&gt; which is simply &lt;code&gt;z * (1-z) * dz&lt;/code&gt; but there are three &lt;code&gt;z&lt;/code&gt;s and three &lt;code&gt;dz&lt;/code&gt;s?&lt;/p&gt;&#xA;" OwnerUserId="38637" LastActivityDate="2020-04-28T03:40:47.237" Title="Help with backpropagation equations for a simple neural network with Sigmoid activation" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22711" PostTypeId="1" CreationDate="2017-08-30T13:35:56.540" Score="1" ViewCount="435" Body="&lt;p&gt;I am trying to perform machine translation using &lt;a href=&quot;https://github.com/OpenNMT/OpenNMT-py/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am using &lt;a href=&quot;https://github.com/karimkhanp/Seq2Seq/tree/master/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; dataset for training.&lt;/p&gt;&#xA;&lt;p&gt;Result looks like this -&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;root@krm-datascience:/home/krm.khan/OpenNMT-py# python translate.py -model demo-model_acc_15.97_ppl_486.59_e1.pt -src data/src-test.txt -output pred.txt -replace_unk -verbose&#xA;Loading model&#xA;&#xA;SENT 1: डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल&#xA;PRED 1: This .&#xA;PRED SCORE: -8.3647&#xA;&#xA;SENT 2: मैं भारत से प्यार करता हूँ&#xA;PRED 2: This .&#xA;PRED SCORE: -8.1020&#xA;&#xA;SENT 3: पाकिस्तानी सरकार के अनुसार इसी समय पाकिस्तान की नींव डाली गई थी ।&#xA;PRED 3: This .&#xA;PRED SCORE: -7.8469&#xA;&#xA;SENT 4: ज़बूर जो कि दाउद को प्रदान की गयी&#xA;PRED 4: This .&#xA;PRED SCORE: -8.3363&#xA;&#xA;SENT 5: प्रशासनिक विभाजन&#xA;PRED 5: India .&#xA;PRED SCORE: -6.6838&#xA;&#xA;SENT 6: वे इस देश के प्रथम &amp;lt;unk&amp;gt; -LRB- अफ्रीकी &amp;lt;unk&amp;gt; -RRB-&#xA;PRED 6: This .&#xA;PRED SCORE: -8.2260&#xA;&#xA;SENT 7: नेपाली विदेश &amp;lt;unk&amp;gt;&#xA;PRED 7: Hindi of India .&#xA;PRED SCORE: -11.5715&#xA;PRED AVG SCORE: -3.6957, PRED PPL: 40.2738&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I could not understand the result. Is it correctly trained? Why result is so bad.&lt;/p&gt;&#xA;" OwnerUserId="12448" LastEditorUserId="26019" LastEditDate="2020-08-20T16:57:03.577" LastActivityDate="2020-08-20T16:57:03.577" Title="Hindi English Machine translation using OpenNMT" Tags="|machine-learning|classification|deep-learning|machine-translation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22712" PostTypeId="1" CreationDate="2017-08-30T13:50:31.837" Score="1" ViewCount="75" Body="&lt;p&gt;I have a model which calculates churn probabilities for e-commerce customers based on their historical activity data (no. of sessions, page views, purchases etc) using supervised learning with an AUC of 88%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task I have at hand is to find causes for churn. The analysis that I did has a 5 month observation period data where I look at causal metrics (service delivery, mode of service, returns made and the reasons for it, customer-support contact frequency etc) and try to predict customers who would churn (used a threshold on churn probability to convert to a binary variable). The model is a weak one with an AUC of 62% (using random forest) and there is no single variable which comes out to be a strong indicator for churn in this model. I am doing some feature engineering at present but my initial attempts haven't resulted in anything promising.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to do such an analysis?&lt;/p&gt;&#xA;" OwnerUserId="33886" LastActivityDate="2017-08-30T13:50:31.837" Title="Causes for churn" Tags="|machine-learning|predictive-modeling|churn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22713" PostTypeId="2" ParentId="22709" CreationDate="2017-08-30T14:03:21.453" Score="2" Body="&lt;p&gt;As Sean Owen said, it's a wide problem so I can't be generic on this one. I'll just give you an example on how you could tackle this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let say you sell some products on a website. If you have 100 different products, you can represent each sale by a 100 dimensional vector, where each component is the count of ordered product. &lt;em&gt;Example: sale1 = (2,0,1,0,...,0) means 2 item1 and 1 item3 sold.&lt;/em&gt; We use the same technique with words, and it is called bag of words. If you stop here, you only can perform statistical counts on individual product sales. So let's use matrix factorization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you sold products to 1000 people, then you can concat all your sales in a matrix of dimension 1000 X 100 where each row represent one sale and each column represent one product. Using decomposition algorithm such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Singular_value_decomposition&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVD&lt;/a&gt;, you can find out singular values in your data that will show you latent factors in your sales (basically trends or connections between your products).      &lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-08-31T07:13:31.777" LastActivityDate="2017-08-31T07:13:31.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22714" PostTypeId="2" ParentId="22703" CreationDate="2017-08-30T14:19:15.570" Score="6" Body="&lt;p&gt;Basically, skip-gram aim for computing the probability of a context given a word (it is CBOW that does the opposite). BUT, it does not simply learn a co-occurrence matrix, it compresses the information in a low dimensional space (for example 300 for a 100000 original dimension). By doing this, it learns continuous low dimensional representation for the words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization.pdf&quot; rel=&quot;noreferrer&quot;&gt;It is proved&lt;/a&gt; that word2vec actually factorizes a word-context pointwise mutual information matrix (close to the co-occurrence matrix). I think this article might give you a better understanding of the underlying processes. &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-08-30T14:19:15.570" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22715" PostTypeId="1" CreationDate="2017-08-30T14:46:01.283" Score="1" ViewCount="957" Body="&lt;p&gt;I am reading an &lt;a href=&quot;https://pjreddie.com/media/files/papers/yolo.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;object detection paper called YOLO (You only look once)&lt;/a&gt; and I have some questions about the architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the CNN network, the author described that 1x1 convolution reduce feature space from preceding layers. When I see the diagram below, I am confused that he stacked 3x3x256 convolution layer follow by 1x1x256.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't the number of features (256 to 256) the same before and after? How is this supposed to reduce the feature space?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B4cKy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B4cKy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4841" LastEditorUserId="29169" LastEditDate="2021-02-15T22:24:43.967" LastActivityDate="2021-02-15T22:24:43.967" Title="1x1 convolution" Tags="|deep-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22719" PostTypeId="2" ParentId="22707" CreationDate="2017-08-30T19:23:54.003" Score="2" Body="&lt;p&gt;There are at least 2 distinct problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Understanding the non-programmers intention for the webpage's content &lt;/li&gt;&#xA;&lt;li&gt;Understanding the webpage's content&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The 1st problem is a problem not best solved with data science or machine learning, hard-coded rules will work better. The 2nd problem is mostly a software engineering problem. I would look into existing solutions for inspiration, for example &lt;a href=&quot;https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd&quot; rel=&quot;nofollow noreferrer&quot;&gt;scraper&lt;/a&gt; or &lt;a href=&quot;http://import.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;import.io&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still want to sort and present the &quot;selectors&quot; to users, that is most frequently called a &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank&quot; rel=&quot;nofollow noreferrer&quot;&gt;learning to rank&quot;&lt;/a&gt; problem. Learning to rank problems are common in information retrieval systems / search engines. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2019-02-28T13:50:02.723" LastActivityDate="2019-02-28T13:50:02.723" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="22720" PostTypeId="2" ParentId="15509" CreationDate="2017-08-30T22:08:10.073" Score="1" Body="&lt;p&gt;You can use fully convolutional neural nets for image segmentation. Check out the kernels in the kaggle competition: &lt;a href=&quot;https://www.kaggle.com/c/carvana-image-masking-challenge&quot; rel=&quot;nofollow noreferrer&quot;&gt;Carvana Image Masking Challenge&#xA;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check out &lt;a href=&quot;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fully Convolutional Neural Network&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30921" LastEditorUserId="29575" LastEditDate="2017-12-29T02:54:54.917" LastActivityDate="2017-12-29T02:54:54.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22722" PostTypeId="2" ParentId="128" CreationDate="2017-08-31T01:02:16.897" Score="24" Body="&lt;p&gt;I wanted to point out, since this is one of the top Google hits for this topic, that Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), &lt;em&gt;and&lt;/em&gt; hierarchical Latent Dirichlet Allocation (hLDA) are all distinct models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LDA models documents as dirichlet mixtures of a fixed number of topics- chosen as a parameter of the model by the user- which are in turn dirichlet mixtures of words. This generates a flat, soft probabilistic clustering of terms into topics and documents into topics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HDP models topics as mixtures of words, much like LDA, but rather than documents being mixtures of a fixed number of topics, the number of topics is generated by a dirichlet process, resulting in the number of topics being a random variable as well. The &quot;hierarchical&quot; portion of the name refers to another level being added to the generative model (the dirichlet process producing the number of topics), not the topics themselves- the topics are still flat clusterings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hLDA, on the other hand, is an adaptation of LDA that models topics as mixtures of a new, distinct level of topics, drawn from dirichlet &lt;em&gt;distributions&lt;/em&gt; and not processes. It still treats the number of topics as a hyperparameter, i.e., independent of the data. The difference is that the clustering is now hierarchical- it learns a clustering of the first set of topics themselves, giving a more general, abstract relationships between topics (and hence, words and documents). Think of it like clustering the stack exchanges into math, science, programming, history, etc. as opposed to clustering data science and cross validation into an abstract statistics and programming topic that shares some concepts with, say, software engineering, but the software engineering exchange is clustered on a more concrete level with the computer science exchange, and the similarity between all of the mentioned exchanges doesn't appear as much until the upper layer of clusters.&lt;/p&gt;&#xA;" OwnerUserId="38663" LastActivityDate="2017-08-31T01:02:16.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22725" PostTypeId="1" CreationDate="2017-08-31T02:11:38.127" Score="29" ViewCount="20058" Body="&lt;p&gt;I am interested in knowing what really happens in  Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use  Hellinger Distance? What are the benefits of using  Hellinger Distance?&lt;/p&gt;&#xA;" OwnerUserId="38395" LastEditorUserId="38395" LastEditDate="2017-08-31T03:06:40.853" LastActivityDate="2018-04-16T03:20:42.863" Title="What is Hellinger Distance and when to use it?" Tags="|machine-learning|data-mining|text-mining|distance|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="22727" PostTypeId="1" CreationDate="2017-08-31T05:32:11.503" Score="4" ViewCount="1544" Body="&lt;p&gt;I have data frame having 3 columns namely index, Actual &amp;amp; Predicted (all column are numeric).&#xA;I am using 2 packages &lt;code&gt;plotly&lt;/code&gt; &amp;amp; &lt;code&gt;ggplot2&lt;/code&gt; for plotting index on x-axis &amp;amp; Actual and Predicted on y-axis.&#xA; The command for plotting the graphs goes as follows-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot&amp;lt;- ggplotly(ggplot(dataframe, aes(x = index))+ geom_line(aes(y=Predicted), colour= &quot;blue&quot;)+ geom_line(aes(y=Actual), colour=&quot;red&quot;))&#xA;&#xA;plot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The R was giving the plot smoothly without any recommendations, but suddenly it started giving recommendations like,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;We recommend that you use the dev version of ggplot2 with &lt;code&gt;ggplotly()&lt;/code&gt;&#xA;Install it with: &lt;code&gt;devtools::install_github('hadley/ggplot2')&lt;/code&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I tried installing &lt;code&gt;ggplot2&lt;/code&gt; with devtools as given in the recommendations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to plot the graph but still I am getting the above recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does that recommendation mean and what to do if i am getting the recommendation even after installing it with &lt;strong&gt;devtools&lt;/strong&gt; ?&lt;/p&gt;&#xA;" OwnerUserId="30006" LastEditorUserId="34028" LastEditDate="2017-08-31T21:09:52.703" LastActivityDate="2018-11-10T14:00:00.233" Title="R recommendations while applying ggplot2 and plotly together to plot a graph" Tags="|r|ggplot2|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22729" PostTypeId="1" CreationDate="2017-08-31T07:03:20.510" Score="1" ViewCount="46" Body="&lt;p&gt;I have a survival data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sum    Days&#xA; 4      10&#xA; 5      12&#xA; 2      7&#xA; 3       8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's say this is a ink purchase history of one credit card. Sum means how many bottles of ink the credit card buys at that time. Days means how many days are before this credit card makes another purchase. So, the first two lines means this credit card buys 4 bottles of ink at first time. Then 10 days later, this card buys 5 bottles. So, I consider this to be a survival model. All the data except for the last line is the training set. The last line is the test set. The model trained by the training set is used to predict the value of Days of the last line given the value of Sum. In this example, I think the model is satisfying if the predicted value is 9 as 9 is pretty close to the real value 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have like 1000 credit cards each with its own ink purchase history. I trained three survival models seperately: weibull, average and ctree. We can expect that one model performs better on these credit cards while the other is more suitable for other cards. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I want to use a classification method to identify which data (credit cards) suits which survival model more so that my system can automically decide which survival model will be used when a particular credit card data comes in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it I take the purchase data as a time series, then in my data, this is very sparse. Thereforely it is probably not good for time series classification. So, what I want to ask is if ther is a classification model which can do this? &lt;/p&gt;&#xA;" OwnerUserId="16911" LastActivityDate="2017-08-31T07:03:20.510" Title="How to use a classification model to choose the best one from three survival models?" Tags="|classification|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22730" PostTypeId="1" CreationDate="2017-08-31T07:35:29.927" Score="1" ViewCount="521" Body="&lt;p&gt;I have set of documents as follows where each document has set of words that represents the content of it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Doc1: {fish, moose, wildlife, hunting, bears, polar}&#xA;Doc2: {energy, fuel, costs, oil, gas}&#xA;Doc3: {wildlife, hunt, polar, fishing}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, if I look at my documents I can deduce that Doc1 and Doc3 are very much similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want distance metrics for bag-of-words. I followed some &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/distance_metrics.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorials&lt;/a&gt; in Gensim about how to do it. However, as I understand, initially they train a model and then use that model to calculate the Hellinger Distance. However, in my case, I do not have any training data. Hence, please let me know how to achieve this with no training data.&lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2024-04-01T22:01:05.120" Title="Hellinger Distance in Gensim" Tags="|machine-learning|data-mining|text-mining|topic-model|gensim|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22731" PostTypeId="1" AcceptedAnswerId="22992" CreationDate="2017-08-31T07:59:02.007" Score="2" ViewCount="447" Body="&lt;p&gt;I am dealing with acoustic data with very high sampling frequency of 2MHz and want to build a classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there are any rules of thumb for preprocessing acoustic data. Is it better to directly use raw data (timesignal) or first to construct spectrograms, and to use these?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are papers, which say raw is better, and there are papers saying spectrograms are better. It somehow seems to me, that authors already had a preferred method, even before writing the paper. I think a real comparison is difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read the paper &lt;a href=&quot;https://arxiv.org/abs/1612.07640&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Deep Learning and Its Applications to Machine Health Monitoring: A survey&quot;&lt;/a&gt;, in which a study of different methods was done. I looked up his references, but authors seemed just to pick raw or spectrograms without explaining. For example, in the paper &lt;a href=&quot;http://ieeexplore.ieee.org/document/6854950/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;End-to-end learning for music audio&quot;&lt;/a&gt; from Dieleman spectrograms are preferred. In &lt;a href=&quot;https://arxiv.org/abs/1703.01789&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Sample-Level deep convolutional neural networks for music auto-tagging using raw waveforms&quot;&lt;/a&gt;, they claim their 1D structure is better or at least comparable to 2D architectures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I had better experience with spectrograms. &lt;/p&gt;&#xA;" OwnerUserId="37330" LastEditorUserId="37330" LastEditDate="2017-08-31T08:56:43.403" LastActivityDate="2017-09-13T13:02:21.967" Title="How to preprocess Acoustic Data" Tags="|preprocessing|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22732" PostTypeId="2" ParentId="22709" CreationDate="2017-08-31T08:04:32.617" Score="1" Body="&lt;p&gt;I assume you are looking for a similarity measure between items. A quick and simple one is item-item cosine similarity. An item (product) can be represented by a vector $x$ with $x_i = 1$ if it was in the $i$th purchase, otherwise $x_i=0$. The similarity between two products is then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac {&amp;lt;x,y&amp;gt;} {\lVert x \rVert \lVert y \rVert} = \frac{x_1y_1 + \dots + x_n y_n} {\sqrt{\sum x_i^2 \sum y_i^2} } = \frac {\mbox{number of times 1 &amp;amp; 2 sold together}} {\sqrt{\mbox{number of times 1 sold} \times \mbox{ number of times 2 sold} }}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matrix decomposition techniques can probably give better results, but they might require a bit more knowledge of linear algebra or finding some ready-made products.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2017-08-31T08:04:32.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22733" PostTypeId="1" CreationDate="2017-08-31T08:08:11.763" Score="1" ViewCount="761" Body="&lt;p&gt;I have three excel sheets for training (all numerical values).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;1st excel sheets have parameters &lt;code&gt;a, b, c&lt;/code&gt; and output as &lt;code&gt;y&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;2nd excel sheet have parameters &lt;code&gt;a, b, c, d, e&lt;/code&gt; and output as &lt;code&gt;y&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;3rd excel sheet have parameters &lt;code&gt;a, b, c, d&lt;/code&gt; and output as &lt;code&gt;y&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have to predict value &lt;code&gt;y&lt;/code&gt; if values of only &lt;code&gt;a, b, c&lt;/code&gt; are given.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I apply linear regression?&lt;/p&gt;&#xA;" OwnerUserId="38673" LastEditorUserId="29575" LastEditDate="2017-09-01T08:11:19.987" LastActivityDate="2017-09-07T21:53:46.210" Title="How to predict numerical value &quot;y&quot;? if I have datasets having different number of parameters" Tags="|machine-learning|predictive-modeling|regression|linear-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22734" PostTypeId="2" ParentId="22733" CreationDate="2017-08-31T08:19:02.230" Score="1" Body="&lt;p&gt;If you have to predict $y$ only given $a,b,c$, then as a first look just discard $d,e$ etc from other data sets*. Any kind of regression might be useful, you will have to explore that - you might as well start with linear regression as it is simplest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if you are not sure about which algorithm to apply for best results, you should set up an experiment to try multiple models:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Decide on what &quot;best&quot; means by choosing a test metric (mean square error is a common choice for regression problems). &lt;/li&gt;&#xA;&lt;li&gt;Split your data into training, cross-validation and test sets.&lt;/li&gt;&#xA;&lt;li&gt;Train different types of model on the training set:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tune any hyper-parameters (of more complex models) by taking the best results from the cross-validation set. &lt;/li&gt;&#xA;&lt;li&gt;Pick your best cross-validation result as your guess at the best model. &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Finally get an unbiased estimate of its performance by using your test set. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* There might be ways to use $d,e$ values to refine some models, but if you are at the stage of asking whether or not to use linear regression, this is not worth exploring initially.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-08-31T08:19:02.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22735" PostTypeId="2" ParentId="22733" CreationDate="2017-08-31T08:21:22.877" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;Do process your data first, meaning see if you can combined, as you specified a,b,c,d,e etc these are your independent variables.&lt;/li&gt;&#xA;&lt;li&gt;you know the required output y, thats your dependent variable.&lt;/li&gt;&#xA;&lt;li&gt;you can apply rectifier and sigmoid in ANN&lt;/li&gt;&#xA;&lt;li&gt;and then check for the confusion matrix&lt;/li&gt;&#xA;&lt;li&gt;optimize if required, which i m sure will be required.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38674" LastActivityDate="2017-08-31T08:21:22.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22736" PostTypeId="1" CreationDate="2017-08-31T09:09:53.743" Score="3" ViewCount="52639" Body="&lt;p&gt;pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Code 1: Reading Excel&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pdf = pd.read_excel(Name.xlsx)&#xA;sparkDF = sqlContext.createDataFrame(pdf)&#xA;df = sparkDF.rdd.map(list)&#xA;type(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Want to implement without pandas module&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Code 2: gets list of strings from column colname in dataframe df&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stringsList = []&#xA;columnList = list(df[colname])&#xA;for i in range(len(columnList)):&#xA;    if type(columnList[i]) != float:&#xA;        text = columnList[i]&#xA;        stringsList.append(text.lower())    &#xA;    else:&#xA;        stringsList.append(u'')&#xA;return stringsList&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to implement this in pyspark.&lt;/p&gt;&#xA;" OwnerUserId="38678" LastEditorUserId="381" LastEditDate="2017-08-31T19:18:25.530" LastActivityDate="2020-09-09T08:52:09.700" Title="Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName" Tags="|python|apache-spark|pyspark|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22737" PostTypeId="1" CreationDate="2017-08-31T09:32:28.943" Score="-2" ViewCount="1212" Body="&lt;p&gt;I have deep FC model (7 layers, 500 units per layer) and I need to speed up it in the production. Memory requirements don't matter. What tools or approaches can you advise except quantization. There are several works about that with several times speed up, but I can't find any tool for Tensorflow. I found the only tool for CNN compression/speed up.&lt;/p&gt;&#xA;" OwnerUserId="21319" LastActivityDate="2017-08-31T16:11:30.237" Title="How to speed-up trained Tensorflow model with FC layers for production" Tags="|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22738" PostTypeId="2" ParentId="20031" CreationDate="2017-08-31T09:58:38.513" Score="1" Body="&lt;p&gt;This type of error happens when you say that the null hypothesis is true when it is actually false. For our null hypothesis that dogs live longer than cats, it would be like saying that dogs do live longer than cats, when in fact, they don't. To help you remember a type II error, think of two wrongs. You are wrongly thinking that the null hypothesis is wrong. The probability of making a type II error is labeled with a beta.&lt;/p&gt;&#xA;" OwnerUserId="32762" LastActivityDate="2017-08-31T09:58:38.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22739" PostTypeId="2" ParentId="21952" CreationDate="2017-08-31T10:01:17.463" Score="1" Body="&lt;p&gt;I am not an expert but this is how I see things: use spark. Pyspark seems to be the flavor to use since it's python and that's the language that data science has settled on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset sizes are expanding way faster than RAM is on laptops. What &quot;they&quot; don't tell you is that scikit learn, pandas, R, etc all require your dataset and all intermediate steps to fit inside RAM. If you have an 8gb dataset, and you pivot it, your intermediate step could easily be say 30gb. Even if it's temporary and your final cleaned dataset to be fed into. A neural network shrinks back down to say 6gb, too bad. All of your steps must fit inside RAM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the real world, and even the limitation that datasets be limited to RAM sizes is quite limiting. This is what I wished I had known earlier. It's like saying here is a grocery store which is all yours (scikit learn has tons of features ) but you can only leave the store with a single shopping bag full of items). I would much rather have a grocery store with the standard 100 items and let me have a buffet on just those 100. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there are some libraries here and there for scikit learn that read from files in chunks, and then you add the chunks together, etc. But you will quickly see that that is reinventing the wheel and distracts from doing data science. The fact remains that these frameworks were built for in RAM usage. It's not a bad thing at all per se, but quickly becomes limiting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want is to write the code once and not have it change whether the dataset is 1gb or 100gb. This is what pyspark is for. You can just learn their API and pyspark will handle chunking and distributing the workload such that it fits in the available RAM that you have. Of course if you are RAm limited computations take longer but this is ok. At least you will get an answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The downside is that if you want the very much appreciated ability to not have to worry about &quot;is this dataset too large for my ram&quot; question then you need to stay in pyspark's API. If they don't have a feature that say scikit learn has, or keras has, then too bad. Write your own or work around it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thankfully the API looks very complete. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where does it lack? Well for say neural networks you aren't going to have all kinds of special activation functions and customization that you can have with Keras. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in my experience and it really is very little, more data and better data beats the specific algorithm much of the time. I only mean to say that I would rather be able to process tons of data and use a vanilla classifier than be limited to in RAM dataset sizes and while being able to optimize 1000  hyperparameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also don't want to be switching between different aPIs all the time as a beginner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if there is any chance that your dataset will be larger than RAM (I think that likelihood is a near certainty in the years ahead) I would most definitely learn pyspark. Even before scikit learn.  &lt;/p&gt;&#xA;" OwnerUserId="32637" LastActivityDate="2017-08-31T10:01:17.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22740" PostTypeId="1" AcceptedAnswerId="22746" CreationDate="2017-08-31T10:08:51.103" Score="5" ViewCount="4524" Body="&lt;p&gt;I have experienced that most of the datasets contain missing values, which make our task bit challenging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know how to fill up those missing values in an efficient way? and is there any specific techniques to handle missing values?&lt;/p&gt;&#xA;" OwnerUserId="32762" LastEditorUserId="27768" LastEditDate="2017-09-04T20:11:23.203" LastActivityDate="2020-08-17T20:32:37.543" Title="Missing Values in Data" Tags="|data-mining|dataset|data-cleaning|data|missing-data|" AnswerCount="4" CommentCount="0" ClosedDate="2017-09-01T09:24:51.690" ContentLicense="CC BY-SA 3.0" />
  <row Id="22741" PostTypeId="2" ParentId="22740" CreationDate="2017-08-31T11:22:24.850" Score="3" Body="&lt;p&gt;Yes there are so many approaches to handle missing data or missing values depending on the task and the property of the data itself. For example in time series you can think about forward filling or even backward filling, max, mean or median over a time lapse. There are also 'smarter' ways like training a model over the available data and try to predict the missing ones. The latest requires much data so can give efficient results. You can refer to these two links for more information.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://machinelearningmastery.com/handle-missing-data-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;How to Handle Missing Data with Python&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/q/103500&quot;&gt;Machine learning algorithms to handle missing data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38355" LastEditorUserId="98307" LastEditDate="2020-08-17T20:32:37.543" LastActivityDate="2020-08-17T20:32:37.543" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22742" PostTypeId="1" CreationDate="2017-08-31T12:07:21.763" Score="1" ViewCount="504" Body="&lt;p&gt;Can someone explain some questions I have about the pseudocode in &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Wikipedia article for Dynamic Time Warping&lt;/a&gt;? (I'm talking about the implementation with the window constraint parameter &lt;code&gt;w&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the line &lt;code&gt;DTW[0, 0] := 0&lt;/code&gt;, why are we setting the first element to zero after setting the entire matrix to infinity? I'm sure there's some reason, but I can't figure out why. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the lines initializing the second set of for loops:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i := 1 to n&#xA;    for j := max(1, i-w) to min(m, i+w)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;why do we skip the first row/column? I get the bounding idea in the inner loop with the window constraint, but from what I understand, in DTW, not using the first column and first row is essential. Why?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within the for loops:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost := d(s[i], t[j])&#xA;DTW[i, j] := cost + minimum(DTW[i-1, j  ],    // insertion&#xA;                            DTW[i  , j-1],    // deletion&#xA;                            DTW[i-1, j-1])    // match&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand the cost is the current distance between two elements in the respective time series, but what are we doing when we set &lt;code&gt;DTW[i, j]&lt;/code&gt; to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost + minimum(DTW[i-1, j  ],    // insertion&#xA;               DTW[i  , j-1],    // deletion&#xA;               DTW[i-1, j-1])    // match&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not the best at Dynamic Programming, so I don't understand this line as well. Why are each cases considered an insertion, deletion, and a match?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, for the last line:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;return DTW[n, m]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are we returning the last element? I thought DTW was supposed to return the aligned time series which I figured would be represented by the warping path in the matrix. &lt;/p&gt;&#xA;" OwnerUserId="33171" LastEditorUserId="33171" LastEditDate="2017-08-31T13:23:02.930" LastActivityDate="2017-08-31T13:23:02.930" Title="Dynamic Time Warping Linear Algebra and Pseudocode Explanation" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22743" PostTypeId="2" ParentId="22740" CreationDate="2017-08-31T12:08:25.310" Score="2" Body="&lt;p&gt;One of the most common way to fill up the missing values is using &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_interpolation&quot; rel=&quot;nofollow noreferrer&quot;&gt;linear interpolation&lt;/a&gt; : The previous valid values before the missing value and the following valid values after the missing value are used for the interpolation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in &lt;code&gt;Python-Numpy&lt;/code&gt; package, you can use &lt;code&gt;numpy.interp&lt;/code&gt;for filling in missing values using linear interpolation. Here is an example for a simple array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; import numpy as np&#xA;&amp;gt;&amp;gt; data=np.array([1,2,np.nan,6,9,np.nan,15])&#xA;&amp;gt;&amp;gt; print(data)&#xA;[  1.   2.  nan   6.   9.  nan  15.]&#xA;&amp;gt;&amp;gt; nans, xf= np.isnan(data), lambda z: z.nonzero()[0]&#xA;&amp;gt;&amp;gt; data[nans]= np.interp(xf(nans), xf(~nans), data[~nans])&#xA;&amp;gt;&amp;gt; print(data)&#xA;[  1.   2.   4.   6.   9.  12.  15.]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-08-31T12:08:25.310" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22744" PostTypeId="2" ParentId="22710" CreationDate="2017-08-31T14:05:14.700" Score="1" Body="&lt;p&gt;The gradient is actually calculated w.r.t to the weights so that each neuron would have a different 'change in the weight' according to its previous weight. In other words, not all the 3 neurons in the hidden layer will be penalized with the same value because the initial weights at t=0 are different for these neurone. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is well explained here: &lt;a href=&quot;https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this short answer helped!&lt;/p&gt;&#xA;" OwnerUserId="38355" LastEditorUserId="95921" LastEditDate="2020-04-28T03:40:47.237" LastActivityDate="2020-04-28T03:40:47.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22745" PostTypeId="2" ParentId="22643" CreationDate="2017-08-31T14:53:46.097" Score="0" Body="&lt;p&gt;I just had another idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why don't you uniformly randomize the assignments. &#xA;For each randomization of assignments, calculate the cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Run this, for I don't know how many times. &#xA;Pick the configuration with the least cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of linearity of expectation, this should give you a good enough answer, might not be the optimal, but more efficient.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2017-08-31T14:53:46.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22746" PostTypeId="2" ParentId="22740" CreationDate="2017-08-31T15:42:59.930" Score="4" Body="&lt;p&gt;Various methods are available for fill missing values in data.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Ignore the tuple is the simplest and not effective method.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fill the missing value manually.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a global constant to fill the missing value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use attribute mean value to fill missing value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use attribute mean for all samples belonging to the same class as the given&#xA;tuple.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use most probable value to fill in the missing value (this may be determined&#xA;with regression, inference tool, or decision tree induction)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Reference:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Mining – Concepts and Techniques - &lt;a href=&quot;http://hanj.cs.illinois.edu/bk1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;JIAWEI HAN&lt;/a&gt; &amp;amp; MICHELINE KAMBER, ELSEVIER, 2nd Edition.&lt;/p&gt;&#xA;" OwnerUserId="38695" LastEditorUserId="29575" LastEditDate="2017-09-01T07:49:14.677" LastActivityDate="2017-09-01T07:49:14.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22747" PostTypeId="2" ParentId="22740" CreationDate="2017-08-31T15:44:02.973" Score="3" Body="&lt;p&gt;First of all, if most of your data is missing, you are in trouble anyway. You need to ask why is most of the data missing, and also, why are the data you observed not missing. Being missing is very likely telling you something in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All methods of correcting missing data, including the naive interpolation, mean replacement, and median replacement methods, assume that you can largely ignore the reason for the data being missing - These are the Missing at Random [MAR], and the (much stronger) Missing Completely at Random [MCAR], assumptions. &#xA;If one, or both, of these are not true, which is very likely if most of your data are missing, then no asymptotically reliable method of imputation is known to exist. This doesn't mean you can do nothing - see &lt;a href=&quot;http://www.stefvanbuuren.nl/mi/docs/mnar.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for some suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most circumstances, with MAR data, people use model based methods. Essentially these impute a dataset many times, filling in the missing values using plausible models of the missing data, and then run analyses on the ensemble of imputed datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've usually used the mice package in r.&#xA;A paper is &lt;a href=&quot;https://www.jstatsoft.org/article/view/v045i03/v45i03.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;A useful webpage is at web.maths.unsw.edu.au/~dwarton/missingDataLab.html.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the process which leads to your missing data is &lt;strong&gt;not ignorable&lt;/strong&gt;, in other words, if being missing tells you something, &lt;strong&gt;none of this will work&lt;/strong&gt;. It will produce nice looking numbers, but no-one, you included, will have any idea what they mean.&lt;/p&gt;&#xA;" OwnerUserId="38692" LastActivityDate="2017-08-31T15:44:02.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22748" PostTypeId="2" ParentId="22737" CreationDate="2017-08-31T16:11:30.237" Score="1" Body="&lt;p&gt;Increasing software speed depends on reducing bottlenecks. Have you benchmarked your code to find which operations are taking the longest? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In TensorFlow code, it could be &lt;a href=&quot;https://github.com/beopst/tf-performance-tips&quot; rel=&quot;nofollow noreferrer&quot;&gt;GPUs or CPUs&lt;/a&gt;. The TensorFlow tutorials have a &quot;&lt;a href=&quot;https://www.tensorflow.org/performance/performance_guide&quot; rel=&quot;nofollow noreferrer&quot;&gt;Performance Guide&lt;/a&gt;&quot; which list best practices. They include:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;• Build and install from source&lt;br&gt;&#xA;    • Utilize queues for reading data&lt;br&gt;&#xA;    • Preprocessing on the CPU&lt;br&gt;&#xA;    • Use NCHW image data format&lt;br&gt;&#xA;    • Place shared parameters on the GPU&lt;br&gt;&#xA;    • Use fused batch norm&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-31T16:11:30.237" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22749" PostTypeId="2" ParentId="22715" CreationDate="2017-08-31T17:04:42.917" Score="2" Body="&lt;p&gt;As referenced in @E_net4 's comment, 1x1 convolutions are equivalent to fully connected layers that allow for feature expansion or contraction based on the number of filters used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the architecture diagram in the picture it looks like the 1x1 convolutions are in fact reducing the feature space by cutting the number of filters in half. I.e. looking at the the 3rd block in the image, the input is 56x56x256 but the first convolution is a 1x1x128 so assuming stride 1 the output would then be 56x56x128.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-08-31T17:04:42.917" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22750" PostTypeId="1" AcceptedAnswerId="22775" CreationDate="2017-08-31T17:20:36.823" Score="1" ViewCount="2268" Body="&lt;p&gt;Having issues with the &quot;find informative projections&quot; feature in Orange 3. In order to be able to use this feature in the scatterplots, you have to select a target variable in the &quot;select columns&quot;. So I choose the value that I'm most interested in finding correlations to. However, whatever I choose as a target variable doesn't show up in the informative projections list as being correlated to other features in my data set. Why not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of features in my data and was under the impression that the &quot;find informative projections&quot; option finds pairs of features that seem to have a linear correlation and sorts them by which pairs are most correlated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's the case, why do you need a target variable? How do I choose an appropriate target variable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I misunderstanding the purpose of the &quot;find informative projections&quot; feature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm attaching a pic of my work flow.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/CLj3E.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CLj3E.png&quot; alt=&quot;Orange data workflow&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38703" LastActivityDate="2017-12-20T11:22:05.153" Title="Orange 3 &quot;Find Informative Projections&quot; and target variables?" Tags="|prediction|orange|plotting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22751" PostTypeId="2" ParentId="15892" CreationDate="2017-08-31T18:03:17.560" Score="4" Body="&lt;p&gt;I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]&lt;/p&gt;&#xA;" OwnerUserId="38704" LastActivityDate="2017-08-31T18:03:17.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22752" PostTypeId="2" ParentId="5725" CreationDate="2017-08-31T18:19:52.477" Score="1" Body="&lt;p&gt;Housing prices is a popular regression data set. &lt;a href=&quot;https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is one example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Car prices is an other popular choice. &lt;a href=&quot;https://www.kaggle.com/orgesleka/used-cars-database&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is one example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember there is &quot;no free lunch&quot; in machine learning. Models tend to &lt;strong&gt;not&lt;/strong&gt; generalize across domains.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-08-31T18:19:52.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22754" PostTypeId="2" ParentId="22658" CreationDate="2017-08-31T22:15:33.967" Score="0" Body="&lt;p&gt;Assuming the output size of your model is an integer (i.e. $\in\mathbb Z ^{[0,100]}$) a vector $\in \mathbb R^{100}$ should definitely be the output to your ANN. This is the most common approach to creating output classification for neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The values of this output vector could be all over the place depending on how your weights are initialized/trained so in order to interpret the output as a probability of belonging to one of the given 100 classes (and sum to 1 as you say) you need to feed the output through a &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;soft max function&lt;/a&gt;. You can then take the output of this to get your size class prediction and train using the cross entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your output is continuous the easiest approach is just to have one output size, train on MSE, and get your answer by rounding between 0 and 100.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastEditorUserId="33922" LastEditDate="2017-08-31T22:29:24.683" LastActivityDate="2017-08-31T22:29:24.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22755" PostTypeId="1" CreationDate="2017-09-01T01:02:14.177" Score="3" ViewCount="1227" Body="&lt;p&gt;I have a multiclass-classification dataset with the target (dependent) variable highly imbalanced. While using the randomForest package in R, I usually use the parameters &lt;code&gt;sampsize &amp;amp; strata&lt;/code&gt; to account for the imbalance in training data. Are there any similar options in xgboost package also? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary of the number of datapoints available in each class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Factor 1     : 667&lt;br&gt;&#xA; Factor 2     : 676&lt;br&gt;&#xA;Factor 3    :7807&lt;br&gt;&#xA; Factor 4       : 850 &lt;/p&gt;&#xA;" OwnerUserId="38717" LastActivityDate="2020-12-23T12:01:55.213" Title="Stratified Sampling for XGboost" Tags="|r|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22756" PostTypeId="2" ParentId="5725" CreationDate="2017-09-01T02:33:38.570" Score="0" Body="&lt;p&gt;Go to &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCL Machine Learning Repository&lt;/a&gt;. You will find the various data sets by algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Regression, Classification, Clustering etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assure you that this will be really helpful for practicing algorithms.&lt;/p&gt;&#xA;" OwnerUserId="32762" LastEditorUserId="32762" LastEditDate="2017-12-17T17:08:06.353" LastActivityDate="2017-12-17T17:08:06.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22757" PostTypeId="1" CreationDate="2017-09-01T05:38:21.160" Score="4" ViewCount="223" Body="&lt;p&gt;I am reading a paper &lt;a href=&quot;https://link.springer.com/content/pdf/10.1007%2Fs10618-006-0059-1.pdf&quot; rel=&quot;noreferrer&quot;&gt;Frequent pattern mining:current status and future directions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;From this paper I learned that a lot of people did some interesting jobs in this field. But this review is published in 2007. And as the size of data becomes bigger and bigger, in my opinion, frequent pattern mining which based on sparse data may not compete with Neural Network anymore? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Hope someone can tell some recent news about frequent pattern mining.&lt;/p&gt;&#xA;" OwnerUserId="37590" LastActivityDate="2017-09-01T05:38:21.160" Title="How's frequent pattern mining in data mining?" Tags="|data-mining|sequential-pattern-mining|" AnswerCount="0" CommentCount="5" ClosedDate="2020-08-28T15:53:28.247" ContentLicense="CC BY-SA 3.0" />
  <row Id="22759" PostTypeId="1" AcceptedAnswerId="22785" CreationDate="2017-09-01T07:15:19.223" Score="0" ViewCount="596" Body="&lt;p&gt;Sorry for the bad title, I can't find a good one. So I will try to explain what I'm looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm doing sales forecasting with a Regression Forest. (Spark - Scala for the technology) &#xA;I've worked on some test data and I did my forecast using training data.&#xA;But some of the features which I have used can't be employed to forecast the future as they would not be known to me at any given time. For example the numbers of customers of a day, their categories, what kind of advantage they have etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to find others features that will be as useful as these ones or Do I need to perform prediction on these features before my sales forecasting and use the predictions? &#xA;Are there any another solutions? Also, what kind of algorithms should I use for the &quot;features forecasting&quot;? &lt;/p&gt;&#xA;" OwnerUserId="38723" LastEditorUserId="30107" LastEditDate="2017-09-02T09:29:31.590" LastActivityDate="2017-09-02T09:29:31.590" Title="Sales Forecasting - Random Forest - Which features should I use for out of sample forecasting?" Tags="|regression|feature-selection|random-forest|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22760" PostTypeId="1" CreationDate="2017-09-01T08:05:04.107" Score="18" ViewCount="24019" Body="&lt;p&gt;Most networks I've seen have one or two dense layers before the final softmax layer.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there any principled way of choosing the number and size of the dense layers?&lt;/li&gt;&#xA;&lt;li&gt;Are two dense layers more representative than one, for the same number of parameters?&lt;/li&gt;&#xA;&lt;li&gt;Should dropout be applied before each dense layer, or just once?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="34165" LastEditorUserId="29169" LastEditDate="2021-02-15T22:24:33.723" LastActivityDate="2022-05-18T13:48:16.497" Title="Number and size of dense layers in a CNN" Tags="|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22762" PostTypeId="1" AcceptedAnswerId="22821" CreationDate="2017-09-01T10:57:57.723" Score="51" ViewCount="158736" Body="&lt;p&gt;I'm following this &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html#multioutput-classification&quot; rel=&quot;noreferrer&quot;&gt;example&lt;/a&gt; on the scikit-learn website to perform a multioutput classification with a Random Forest model.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import make_classification&#xA;from sklearn.multioutput import MultiOutputClassifier&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.utils import shuffle&#xA;import numpy as np&#xA;&#xA;X, y1 = make_classification(n_samples=5, n_features=5, n_informative=2, n_classes=2, random_state=1)&#xA;y2 = shuffle(y1, random_state=1)&#xA;Y = np.vstack((y1, y2)).T&#xA;&#xA;forest = RandomForestClassifier(n_estimators=10, random_state=1)&#xA;multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)&#xA;multi_target_forest.fit(X, Y).predict(X)&#xA;&#xA;print(multi_target_forest.predict_proba(X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;From this &lt;code&gt;predict_proba&lt;/code&gt; I get a 2 5x2 arrays:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[array([[ 0.8,  0.2],&#xA;       [ 0.4,  0.6],&#xA;       [ 0.8,  0.2],&#xA;       [ 0.9,  0.1],&#xA;       [ 0.4,  0.6]]), array([[ 0.6,  0.4],&#xA;       [ 0.1,  0.9],&#xA;       [ 0.2,  0.8],&#xA;       [ 0.9,  0.1],&#xA;       [ 0.9,  0.1]])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I was really expecting a &lt;code&gt;n_sample&lt;/code&gt; by &lt;code&gt;n_classes&lt;/code&gt; matrix. I'm struggling to understand how this relates to the probability of the classes present.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html#sklearn.multioutput.MultiOutputClassifier.predict_proba&quot; rel=&quot;noreferrer&quot;&gt;docs&lt;/a&gt; for &lt;code&gt;predict_proba&lt;/code&gt; states:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;array of shape = [n_samples, n_classes], or a list of n_outputs such arrays if n_outputs &amp;gt; 1.&lt;/p&gt;&#xA;&lt;p&gt;The class probabilities of the input samples. The order of the classes corresponds to that in the attribute classes_.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I'm guessing I have the latter in the description, but I'm still struggling to understand how this relates to my class probabilities.&lt;/p&gt;&#xA;&lt;p&gt;Furthermore, when I attempt to access the &lt;code&gt;classes_&lt;/code&gt; attribute for the &lt;code&gt;forest&lt;/code&gt; model I get an &lt;code&gt;AttributeError&lt;/code&gt; and this attribute does not exist on the &lt;code&gt;MultiOutputClassifier&lt;/code&gt;. How can I relate the classes to the output?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;print(forest.classes_)&#xA;&#xA;AttributeError: 'RandomForestClassifier' object has no attribute 'classes_'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20429" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2020-02-12T14:46:33.447" Title="Understanding predict_proba from MultiOutputClassifier" Tags="|scikit-learn|random-forest|multilabel-classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22764" PostTypeId="2" ParentId="22760" CreationDate="2017-09-01T13:00:23.347" Score="22" Body="&lt;p&gt;First of all:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There is no way to determine a good network topology just from the&#xA;  number of inputs and outputs. It depends critically on the number of&#xA;  training examples and the complexity of the classification you are&#xA;  trying to learn.[1]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and Yoshua Bengio has proposed a very simple rule:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Just keep adding layers until the test error does not improve anymore.[2]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Moreover:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The earlier features of a ConvNet contain more generic features (e.g.&#xA;  edge detectors or color blob detectors) that should be useful to many&#xA;  tasks, but later layers of the ConvNet becomes progressively more&#xA;  specific to the details of the classes contained in the original&#xA;  dataset.[3]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For example, in a method for learning feature detectors:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;first layer learns edge detectors and subsequent layers learn more complex features, and higher level layers encode more abstract features. [4]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, using two dense layers is more advised than one layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The original paper on Dropout provides a number of useful heuristics to consider when using dropout in practice. One of them is:&#xA;   &lt;strong&gt;Use dropout on incoming (visible) as well as hidden units. Application of dropout at each layer of the network has shown good results.&lt;/strong&gt; [5]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;in CNN, usually, a Dropout layer is applied after each pooling layer, and also after  your Dense layer. A good tutorial is here [6]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &lt;a href=&quot;https://www.cs.cmu.edu/Groups/AI/util/html/faqs/ai/neural/faq.html&quot; rel=&quot;noreferrer&quot;&gt;https://www.cs.cmu.edu/Groups/AI/util/html/faqs/ai/neural/faq.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Bengio, Yoshua. &quot;Practical recommendations for gradient-based training of deep architectures.&quot; Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012. 437-478.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;noreferrer&quot;&gt;http://cs231n.github.io/transfer-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4] &lt;a href=&quot;http://learning.eng.cam.ac.uk/pub/Public/Turner/Teaching/ml-lecture-3-slides.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://learning.eng.cam.ac.uk/pub/Public/Turner/Teaching/ml-lecture-3-slides.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[5] &lt;a href=&quot;https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/&quot; rel=&quot;noreferrer&quot;&gt;https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[6] &lt;a href=&quot;https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html&quot; rel=&quot;noreferrer&quot;&gt;https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-09-01T13:00:23.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22765" PostTypeId="1" AcceptedAnswerId="22772" CreationDate="2017-09-01T13:23:49.527" Score="2" ViewCount="1600" Body="&lt;p&gt;Let's say I have a set of features: a, b, c, d, e, f. I'm now interested in identifying possible interactions between these features that best predict an outcome. For example, it could be that the features a, f, and the interactions a:b:g, d:f and c:e are the 5 most important factors that predict the outcome. It is not only important for an algorithm to account for feature interactions, but I also want to be able to identify these interactions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I approach this problem with machine learning?&lt;/p&gt;&#xA;" OwnerUserId="38734" LastActivityDate="2017-09-01T15:11:05.807" Title="Identifying important interactions between features using machine learning" Tags="|machine-learning|feature-selection|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22766" PostTypeId="1" CreationDate="2017-09-01T14:24:48.433" Score="0" ViewCount="291" Body="&lt;p&gt;I have been trying to decode a captcha using CNN. &lt;br&gt;The number of training samples is 1,82,000.&lt;br&gt; So far, I could achieve 99% accuracy on training as well as test set.&lt;br&gt;&lt;br&gt;&#xA;The issue however is that LOCATION INVARIANCE is not achieved while training. The captcha text if slightly shifted horizontally gives bad prediction.&lt;br&gt;&lt;br&gt;Below image is one of the samples on which network was trained and gives good prediction accuracy for similar images (Accuracy~99%) .&lt;br&gt;&lt;br&gt; &lt;a href=&quot;https://i.stack.imgur.com/NaNvs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NaNvs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;br&gt;&lt;br&gt;&#xA;But if following image is given , it gives some random prediction.&lt;br&gt;&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LoQ3x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LoQ3x.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only difference between above two images is location of captcha text (apart from the actual captcha value.)&lt;br&gt;&lt;br&gt;&#xA;As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter. So what could be the possible reasons for this ?&lt;/p&gt;&#xA;" OwnerUserId="13155" LastEditorUserId="29169" LastEditDate="2021-02-15T22:24:25.757" LastActivityDate="2021-02-15T22:24:25.757" Title="Location Invariance not achieved in CNN in spite of 99% test accuracy" Tags="|deep-learning|neural-network|tensorflow|convolutional-neural-network|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22767" PostTypeId="2" ParentId="16318" CreationDate="2017-09-01T14:30:43.567" Score="9" Body="&lt;p&gt;For those still wondering: it's so that you can shuffle your data. With your TFrecords in one file, you can't shuffle the order. This is typically necessary with SGD. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, with shards, you can shuffle the order of the shards which allows you to approximate shuffling the data as if you had access to the individual TFRecords. This is clearly better than nothing, and clearly the more shards you have the better this approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alternative is pre-shuffle your data via duplicating it or don't use TFRecords at all.&lt;/p&gt;&#xA;" OwnerUserId="38736" LastActivityDate="2017-09-01T14:30:43.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22769" PostTypeId="1" CreationDate="2017-09-01T14:51:49.357" Score="1" ViewCount="479" Body="&lt;p&gt;Let's say, I have a Machine Learning model built and does predictions for the future. If I have some predicted values that exceed a threshold, can I set up some type of notification system? It can be in email or a text message. Is it possible to do it in &lt;code&gt;python&lt;/code&gt;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been searching online, could not find a valid answer. Any thoughts or resources would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="83275" LastEditDate="2019-11-30T09:43:02.677" LastActivityDate="2019-11-30T09:43:02.677" Title="Machine Learning Model Alert/Notification system - Python" Tags="|machine-learning|scikit-learn|keras|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22770" PostTypeId="2" ParentId="16797" CreationDate="2017-09-01T14:59:47.717" Score="7" Body="&lt;p&gt;You already have the answer from Icyblade. However, I want to point out that your Average Precision formula is wrong. The formula $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$ is the definition of precision, &lt;strong&gt;not Average Precision&lt;/strong&gt;. For object detection, AP is defined in &lt;a href=&quot;http://homepages.inf.ed.ac.uk/ckiw/postscript/ijcv_voc09.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. Briefly, it summarises the precision/recall curve hence not only precision but also recall is taken into account (hence the False Negative will be penalised too).&lt;/p&gt;&#xA;" OwnerUserId="38735" LastActivityDate="2017-09-01T14:59:47.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22771" PostTypeId="1" CreationDate="2017-09-01T15:02:30.823" Score="4" ViewCount="3772" Body="&lt;p&gt;I have 2 datasets, a continuous dataset(75 datapoints and 14 variables) and a discretized dataset which was made by placing the continuous datasets into buckets. I have built a decision tree classifier (using the &lt;code&gt;python&lt;/code&gt; &lt;code&gt;sklearn&lt;/code&gt; package) and the classifier works much better for the discrete dataset rather than the continuous dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also read in a few papers that sometimes it is preferable to use discrete datasets. But I don't know why. I Would appreciate any input or an explanation.&lt;/p&gt;&#xA;" OwnerUserId="38739" LastEditorUserId="34028" LastEditDate="2017-09-01T23:00:25.863" LastActivityDate="2019-10-07T17:00:46.490" Title="Should we use discrete or continuous input for decision trees" Tags="|python|classification|scikit-learn|dataset|decision-trees|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22772" PostTypeId="2" ParentId="22765" CreationDate="2017-09-01T15:11:05.807" Score="2" Body="&lt;p&gt;I think that it is important, in this situation, to ask yourself why you are using machine learning to detect the interactions. It feels a bit like data dredging. Using domain knowledge to think about what interactions would be likely or feasible given the phenomenon you are studying may well serve you well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, you can fit a tree model using, for instance, &lt;code&gt;R&lt;/code&gt;. This will allow you to detect complex and high order interactions. You can see this as an example on page 30 &lt;a href=&quot;http://www3.imperial.ac.uk/pls/portallive/docs/1/1171924.PDF&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Note that I would probably use &lt;code&gt;partykit&lt;/code&gt; for fitting the tree model because it allows you to use non-normal distributions. This method, though, works best if you have a very large data set and you may struggle to fit them in a linear model once you have found them (because they can only occur on one side of the tree).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative method that addresses some of the limitations above is to use a technique called &lt;a href=&quot;http://additivegroves.net/papers/Interactions.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Additive Groves&lt;/a&gt;. These work on the principle of observing the relative performance of differently restricted tree models. Because interaction effects are not additive, this technique can allow for the identification of interactions.&lt;/p&gt;&#xA;" OwnerUserId="30964" LastActivityDate="2017-09-01T15:11:05.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22773" PostTypeId="1" CreationDate="2017-09-01T15:15:00.480" Score="1" ViewCount="74" Body="&lt;p&gt;my algortithm isnt working. The code seems to make sense and everything but im just not convinced with the results. I fell like the centroids should be amongst the data more, sort of central, but they arent doing that no matter how many iterations i set the algorithm too. Could someone run my code and give me some pointers? Ive labeled the code so you can understand it.&#xA;Some main points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;k = 2; 2 centroids&lt;/li&gt;&#xA;&lt;li&gt;centroid1 = Dark red, so datapoints in this cluster are light red&lt;/li&gt;&#xA;&lt;li&gt;centroid 2 = Dark green, so datapoints in this cluster are light green&lt;/li&gt;&#xA;&lt;li&gt;dogs are either 'big' or 'small' depending on their height and weight&lt;/li&gt;&#xA;&lt;li&gt;The euclidean distance is used to decide on which cluster to belong to&lt;/li&gt;&#xA;&lt;li&gt;Raw/main data stored in dictionary labeled 'x'; they are moved about into the dictionaries 'cluster1' and 'cluster2' depending on their distance to the centroids&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import pyplot as plt&#xA;import random, math&#xA;&#xA;plt.title(&quot;Big Dogs and Small Dogs&quot;)&#xA;plt.ylabel(&quot;Height (cm)&quot;)&#xA;plt.xlabel(&quot;Weight (kg)&quot;)&#xA;&#xA;def euclideandis(x, y, a, b):   # Formula for euclidean distance&#xA;    return math.sqrt((x-a)**2 + (x-a)**2)&#xA;&#xA;x = {4.4:31,&#xA;     3.2:19,&#xA;     4.6:32,        # Data- key = weight, value = height&#xA;     4:25,&#xA;     4.1:29,&#xA;     2.90:17,&#xA;     2.9:11}&#xA;&#xA;centroid1 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]   # create random centroids that are between the datapoints&#xA;centroid2 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]&#xA;&#xA;for j in xrange(1):   # keep updating colours, position of centroids&#xA;    cluster1 = {}       # Everything in cluster 1 is closest to the dark red spot, therefore gets included in this dict, and gets scattered in light red&#xA;    cluster2 = {}       # Everything in cluster 2 is closest to the dark green spot, therefore gets included in this dict, and gets scattered in light green&#xA;    for key in x:&#xA;        temp1 = 0       # works out euclidean distance for all data points in x, then compares them&#xA;        temp2 = 0&#xA;        temp1 = euclideandis(key, x[key], centroid1[0][0], centroid1[0][1]) # Dis betw data point, red centroid&#xA;        temp2 = euclideandis(key, x[key], centroid2[0][0], centroid2[0][1]) # Dis betw data point, green centroid&#xA;        if temp1 &amp;lt; temp2:&#xA;            cluster1[key] = x[key]      # if the euclidean distance between datapoint and red spot,&#xA;                                        # smaller than the euclidean distance between datapoint and green spot,&#xA;                                        # add the point to the red cluster, else add to green cluster&#xA;        else:&#xA;            cluster2[key] = x[key]&#xA;&#xA;    centroid1 = [[0, 0]]        # Centroids reset as they will be changed  &#xA;    centroid2 = [[0, 0]]&#xA;&#xA;    iterable = 0&#xA;    for key in cluster1:        # works out mean coordinates of each cluster and changes the centroids coordinates to this&#xA;        iterable = iterable + key&#xA;    centroid1[0][0] = iterable/len(cluster1)&#xA;    iterable = 0&#xA;    for key in cluster1:&#xA;        iterable = iterable + cluster1[key]&#xA;    centroid1[0][1] = iterable/len(cluster1)&#xA;    iterable = 0&#xA;    for key in cluster2:&#xA;        iterable = iterable + key&#xA;    centroid2[0][0] = iterable/len(cluster2)&#xA;    iterable = 0&#xA;    for key in cluster2:&#xA;        iterable = iterable + cluster2[key]&#xA;    centroid2[0][1] = iterable/len(cluster2)&#xA;&#xA;&#xA;&#xA;plt.scatter(cluster1.keys(), cluster1.values(), color = &quot;red&quot;)      # scatters everythning&#xA;plt.scatter(cluster2.keys(), cluster2.values(), color = &quot;lime&quot;)&#xA;plt.scatter(centroid1[0][0], centroid1[0][1], color = &quot;maroon&quot;) &#xA;plt.scatter(centroid2[0][0], centroid1[0][1], color = &quot;green&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To sum up, there isn't a clear error with my program, only I'm not really convinced about the results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UgpZt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UgpZt.png&quot; alt=&quot;The plot after 100k iterations; shouldn&amp;#39;t the centroids be more in the middle of the datapoints?&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(above) why after 100k iterations arent the centroids in the middle of the datapoints? The red one is but the green one isnt &lt;/p&gt;&#xA;" OwnerUserId="37363" LastEditorUserId="37363" LastEditDate="2017-09-01T17:42:00.937" LastActivityDate="2017-09-01T18:21:59.560" Title="Stuck implementing k means for big and small dogs - dodgy results" Tags="|machine-learning|python|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22775" PostTypeId="2" ParentId="22750" CreationDate="2017-09-01T15:49:56.690" Score="0" Body="&lt;p&gt;As you said, &quot;find informative projections&quot; gives you the best pair of features (&quot;score plots&quot;) to explain the target variable. The two best features will be on the x- and y-axis, while your target variable will be the &lt;strong&gt;color&lt;/strong&gt; (hue for numeric, categorical otherwise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example for the Iris dataset. Where &quot;iris&quot; is the target --&gt; color, and petal length and petal width are the most informative features, followed by petal width and sepal width, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e2ykx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e2ykx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-09-01T15:49:56.690" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22776" PostTypeId="1" AcceptedAnswerId="22777" CreationDate="2017-09-01T16:36:58.420" Score="9" ViewCount="9154" Body="&lt;p&gt;Have there been any papers, or does anyone have any specific experience to know whether normalizing labels in a regression problem is likely to improve the performance of a neural network? I have labels that are in the range (0,1000) applying square loss in a ConvNet. I want to know if it might be useful to normalize these to a (0,1) range, or whether that's known to not matter.&lt;/p&gt;&#xA;" OwnerUserId="24792" LastActivityDate="2017-09-01T17:06:01.147" Title="Is it valuable to normalize/rescale labels in neural network regression?" Tags="|neural-network|normalization|labels|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22777" PostTypeId="2" ParentId="22776" CreationDate="2017-09-01T17:06:01.147" Score="12" Body="&lt;p&gt;Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-01T17:06:01.147" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22778" PostTypeId="2" ParentId="22766" CreationDate="2017-09-01T18:04:35.873" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not true. Some options used in CNN architecture, such as max pooling, or strided convolutions, can add a moderate amount of translation invariance. However this will not cover larger translations - anything that is a significant percentage of the image width/height.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNNs also support translation &lt;em&gt;equivariance&lt;/em&gt;, which is where image textures and motifs that appear repeatedly but in different locations (lines, corners, curves) are learned efficiently and appear in the feature maps. This is probably closer to the USP of CNNs, that they learn the &quot;representation language&quot; of a signal, such as photograph, where it is consistent across the dimensions that are being convolved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cause of your problem is therefore very likely that your training set does not include enough images with the more central translation, or enough variations with the separate images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To cover large translations you could look at one or more of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Image pre-processing. In your supplied examples it looks very feasible to centre the digits.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data augmentation. Generate translated versions of your training data to cover expected range of input translations in test.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Network architecture. A RNN or RNN/CNN hybrid could probably consume the images using smaller overlapped tiles of the images in sequence, and be trained to output the captcha string at the end. You might also be able to do this without a RNN, but that would require labelling each tile for the separate digits.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-01T21:06:22.580" LastActivityDate="2017-09-01T21:06:22.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22779" PostTypeId="2" ParentId="22773" CreationDate="2017-09-01T18:21:59.560" Score="3" Body="&lt;p&gt;The centroid is likely correct, you have a display error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The line&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(centroid2[0][0], centroid1[0][1], color = &quot;green&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(centroid2[0][0], centroid2[0][1], color = &quot;green&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's just one of those things that happens when you implement an algorithm from scratch in order to learn it . . . I bet you've spent hours looking at the top part of your script :-)&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-01T18:21:59.560" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22781" PostTypeId="1" CreationDate="2017-09-02T00:06:49.650" Score="2" ViewCount="9024" Body="&lt;p&gt;I am still finding confusing on look back topic when using &lt;code&gt;LSTM&lt;/code&gt; for time-series analysis.&#xA;If I have hourly data and I want to predict next 6 hours with multiple predictors, should I look back up to 6 hours when I prepare my training set? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;OR&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I look back one hour (shift 1) and then predict next hour and take that predicted value and feed it back to predict the value after until next 6 hours? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This concept is still a little fuzzy for me in &lt;code&gt;LSTM&lt;/code&gt;. Any thoughts would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-09-17T21:52:15.383" LastActivityDate="2017-10-12T13:19:17.877" Title="LSTM - How many times should I look back to predict next six hours -Multivariate Time-Series" Tags="|machine-learning|python|time-series|keras|lstm|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22782" PostTypeId="1" AcceptedAnswerId="22786" CreationDate="2017-09-02T00:18:25.147" Score="7" ViewCount="28285" Body="&lt;p&gt;Are these the features which are manually labelled by humans? or  Is there any technique for obtaining these features. Is this related to learned features?&lt;/p&gt;&#xA;" OwnerUserId="30641" LastEditorUserId="29169" LastEditDate="2021-02-15T22:24:09.170" LastActivityDate="2021-02-15T22:24:09.170" Title="What is the meaning of hand crafted features in computer vision problems?" Tags="|machine-learning|feature-selection|computer-vision|feature-engineering|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22783" PostTypeId="1" CreationDate="2017-09-02T03:34:38.433" Score="1" ViewCount="280" Body="&lt;p&gt;I'm setting up a single perceptron for doing linear classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the perceptron initialized with random weights and a random bias instead of just having all of the weights set to zero initially? Wouldn't the loss optimization happening in the back propagation work more effectively on average if these values were all set to zero, as (I think, intuitively) this would be the average of all values that these parameters could take, so starting there is essentially like starting at the &quot;middle&quot; of the space of all perceptron solutions? Why is random a better choice than zero?&lt;/p&gt;&#xA;" OwnerUserId="38758" LastActivityDate="2017-09-02T03:34:38.433" Title="Why is a perceptron initialized with a random line?" Tags="|perceptron|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22784" PostTypeId="2" ParentId="22771" CreationDate="2017-09-02T05:34:55.427" Score="1" Body="&lt;p&gt;Discrete is the way to go. The reason is simple if you visualize a decision tree it involves drawing a decision boundary based on a set of constraints which are in the form of features. It would be much easier to draw these decision boundaries based on discrete features compared to its continuous counterpart. If the values are continuous it becomes difficult for the classifier to effectively draw this boundary and might have some skew in its results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a useful video series in Udacity, Intro into Machine learning, please refer the decision tree section they show a really good visualization on how decision trees work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link: &lt;a href=&quot;https://in.udacity.com/course/intro-to-machine-learning--ud120&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://in.udacity.com/course/intro-to-machine-learning--ud120&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please check that out it may help you understand better. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-09-02T05:34:55.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22785" PostTypeId="2" ParentId="22759" CreationDate="2017-09-02T05:58:43.990" Score="1" Body="&lt;p&gt;The question is poorly phrased, I've tried to edit it to the best of my abilities. However here are the problems you've stated, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Some of the features which are being used now can't be used later because they might not be known. Will it affect the model? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If they can be used, what type of algorithm can be chosen?  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The answer to the first problem, you have to check the accuracy first before making choosing any new features if the rest of your features give a good enough accuracy then there is no need to choose new features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second problem, to predict those values of the features you are using now if they are discrete in nature try classification algorithms whichever fits the model best, else try something along the lines of regression if the input values are continuous. And then use these predicted values along with your existing model and check how the accuracy varies. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-09-02T05:58:43.990" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22786" PostTypeId="2" ParentId="22782" CreationDate="2017-09-02T07:11:16.240" Score="8" Body="&lt;p&gt;&quot;Hand Crafted&quot; features refer to properties derived using various algorithms using the information present in the image itself. For example, two simple features that can be extracted from images are edges and corners. A basic edge detector algorithm works by finding areas where the image intensity &quot;suddenly&quot; changes. To understand that we need to remember that a typical image is nothing but a 2D matrix (or multiple matrices or tensor or n-dimensional array, when you have multiple channels like Red, Green, Blue, etc). In the case of an 8-bit gray-scale image (or a &quot;black and white&quot; image, although this latter definition is not quite accurate) the image is typically a 2D matrix with values ranging from 0 to 255, with 0 being completely black and 255 being completely white. Now imagine an image of a blackboard set against a totally white wall. As we move left-to-right in the image the values in one of the rows of the matrix might look like 255-255-255... since we will be &quot;moving&quot; along the wall. However, when we are about to hit the blackboard in the image it might look like 255-255-0-0-0...As you might have guessed, the blackboard &quot;begins&quot; in the image where the zeros start. In other words, the &quot;intensity&quot; of the image along the &quot;x&quot; axis has dropped rather suddenly (or a very large negative gradient along x), which means a typical edge detector will consider it to be a good candidate for an edge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm that we just saw is only the very basic of algorithms, and others like Harris corners and Hogg detectors use slightly more &quot;sophisticated&quot; algorithms. Actually, even the Canny edge detector does a lot more than I what I just described, but that is besides the point. The point is that once you understand that an image is nothing more than a data matrix, or an n-dimensional array, the other algorithms are not that difficult to understand either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As regards your last question: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is this related to learned features?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &quot;handcrafted features&quot; were commonly used with &quot;traditional&quot; machine learning approaches for object recognition and computer vision like Support Vector Machines, for instance. However, &quot;newer&quot; approaches like convolutional neural networks typically do not have to be supplied with such hand-crafted features, as they are able to &quot;learn&quot; the features from the image data. Or to paraphrase Geoff Hinton, such feature extraction techniques were &quot;what was common in image recognition before the field became silly&quot;.&lt;/p&gt;&#xA;" OwnerUserId="34033" LastEditorUserId="34033" LastEditDate="2017-09-02T07:27:55.890" LastActivityDate="2017-09-02T07:27:55.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22788" PostTypeId="2" ParentId="20143" CreationDate="2017-09-02T18:50:30.017" Score="1" Body="&lt;p&gt;Data analysis usefully defined, via &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt;, emphasis mine:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Data analysis, also known as analysis of data or data analytics, is a process of inspecting, cleansing, transforming, and modeling data with the &lt;strong&gt;goal of discovering useful information, suggesting conclusions, and supporting decision-making&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Meaning, the focus is on deriving information, insight, or conclusions so that humans may do, understand or decide better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tom Mitchell wrote a very helpful definition of in his book &lt;em&gt;Machine Learning&lt;/em&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Machine_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;as quoted on wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here, the emphasis is on programming agents that learn to do things—predict housing prices, transcribe text—from experience, i.e. data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This, I find, is a helpful distinction: Instead of method, we can reason about goals. If we're analyzing Fourier coefficients of recorded speech to understand how the human brain parses sounds into vowels, that's analysis; if we're doing to write a program that automatically transcribes text, it's machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Very diverse field, little consensus re terms, your mileage may vary.)&lt;/p&gt;&#xA;" OwnerUserId="1154" LastActivityDate="2017-09-02T18:50:30.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22793" PostTypeId="2" ParentId="16689" CreationDate="2017-09-03T08:24:08.933" Score="1" Body="&lt;p&gt;The problem is the &lt;code&gt;relu&lt;/code&gt; unit. It is not a very good choice in such a simple network. There is a good chance that the ReLU starts off &quot;dead&quot; -&#xA;- if the weight for the neuron in the first layer is negative (a 50/50 chance), then both 0 and 1 inputs will produce a 0 output and no gradient, so the network cannot learn to separate them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change to &lt;code&gt;tanh&lt;/code&gt; instead will completely fix the problem, and the network will learn the relationship trivially. This as will also work with &quot;leaky&quot; ReLU or any other unit without the simple cutoff of ReLU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A leaky ReLU version of your model would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(keras.layers.Dense(1, input_dim=1, init='uniform' ))&#xA;model.add(keras.layers.advanced_activations.LeakyReLU(alpha=0.01))&#xA;model.add(keras.layers.Dense(1, init='uniform', activation='sigmoid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In larger/deeper networks with more complex input data, this disadvantage of ReLU units generally has lower impact and can be worked around more easily.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-03T08:24:08.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22795" PostTypeId="1" AcceptedAnswerId="22810" CreationDate="2017-09-03T14:55:47.560" Score="7" ViewCount="17640" Body="&lt;p&gt;Is feature scaling useful for &lt;a href=&quot;https://en.wikipedia.org/wiki/Cluster_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;clustering algorithms&lt;/a&gt;? What type of features, I mean numeric, categorical etc., are most efficient for clustering?&lt;/p&gt;&#xA;" OwnerUserId="38791" LastEditorUserId="29575" LastEditDate="2017-09-10T10:11:04.930" LastActivityDate="2020-01-14T15:57:55.957" Title="Do Clustering algorithms need feature scaling in the pre-processing stage?" Tags="|machine-learning|clustering|feature-engineering|feature-scaling|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22796" PostTypeId="1" CreationDate="2017-09-03T15:04:51.820" Score="9" ViewCount="528" Body="&lt;p&gt;I trained a convolutional network to classify images of a mechanical component as good or defective. Though the test accuracy was high, I realized that the model performed poorly on images which had slightly different lighting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The features that I'm trying to detect are subtle, and lighting seems to trip the model into calling a lot of false negatives. Most, if not all, of the training and test data came from images which were shot under diffused lighting. The new images that I tried the model on were shot with focused lights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hoping that histogram equalization(&lt;a href=&quot;https://en.wikipedia.org/wiki/Adaptive_histogram_equalization&quot; rel=&quot;noreferrer&quot;&gt;CLAHE&lt;/a&gt;) would help, I did equalization on &#xA;images before feeding it to the model, did this for training as well as test data. The problem then was that accuracy was high, but the model seemed to have learnt some other correlation between the images and labels. After equalization, everything in the image stands out, the defects become even more subtle and hard to detect even for human eye. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model prediction changed even when the same component was shot with the same lighting but with the light at different orientations. &#xA;So my question is, what can I do, either at the data end or with the model, so that it becomes robust to changes in lighting(or same kind of lighting but changes to orientation of lighting)?&lt;/p&gt;&#xA;" OwnerUserId="38790" LastEditorUserId="38790" LastEditDate="2017-09-03T16:07:30.113" LastActivityDate="2017-09-05T19:07:56.660" Title="Convolutional network for classification, extremely sensitive to lighting" Tags="|machine-learning|classification|deep-learning|image-classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22797" PostTypeId="1" AcceptedAnswerId="22798" CreationDate="2017-09-03T16:23:45.600" Score="1" ViewCount="671" Body="&lt;p&gt;We are trying to build a model, gathering specific hotels booking data, try to find the pattern how the hotel is booked, stayed, what type of people lived, how many bookings average per day. The bookings might vary from weekdays to weekend, also from winter to summer, normal days and vacation period. All these factors are accountable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, as the time passes by, we want to know if the booking becomes abnormal, e.g. normally, young couple books the hotel quite a lot, all the sudden, a bunch of business men checked in for a couple of days. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since in this case, we don't have samples/labels for normality and abnormality, we started thinking use unsupervised learning, like clustering for a start. Say, we construct a sample(booking features for every week) going back all the way to the beginning of the year. And then, we try to cluster them, then, every week, we calculate the current week, and see if it belongs to any clusters? or it is an abnormal point stands out required attention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a reasonable approach or there are some better ways?  &lt;/p&gt;&#xA;" OwnerUserId="38791" LastActivityDate="2017-09-03T16:51:59.817" Title="How to model to predict hotel booking abnomality?" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22798" PostTypeId="2" ParentId="22797" CreationDate="2017-09-03T16:51:59.817" Score="1" Body="&lt;p&gt;Look at the distribution of the features you want to consider for anomalies (e.g., user attributes) conditioned on the date, so you do not trigger a warning for normal seasonality. An anomaly then is when the current conditional distribution is significantly different from the historical average. For more information look into &lt;em&gt;contextual anomaly detection&lt;/em&gt;. Welcome to the site and good luck!&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-09-03T16:51:59.817" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22802" PostTypeId="2" ParentId="21719" CreationDate="2017-09-04T00:45:14.663" Score="4" Body="&lt;p&gt;Euclidean distances between l2-normalized word vectors is equivalent to the angular distance between the word vectors. This is not the same as cosine distance, but it is very similar. It is worth noting that cosine distance is not actually a metric, and that forces many algorithms, including the ones discussed here, to resort to more expensive O(n^2) asymptotic performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind I have done quite well clustering similar dimension word vectors trained with word2vec using HDBSCAN using exactly such an l2-normalizing approach as you suggest here. In general I did find it beneficial to use the leaf cluster extraction method rather than the traditional excess of mass approach to get tight clusters. I found the results to be quite good, with clusters providing very clean &quot;topics&quot; for the corpus on which word2vec was trained.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-09-04T00:45:14.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22803" PostTypeId="2" ParentId="22795" CreationDate="2017-09-04T00:48:00.393" Score="2" Body="&lt;p&gt;Feature scaling will certainly effect clustering results. Exactly what scaling to use is an open question however, since clustering is really an exploratory procedure rather than something with a ground truth you can check against. Ultimately you want to use your knowledge of the data to determine how to relatively scale features. In practice clustering is going to work best with numeric features, so if you have a mix of feature types you may want to look at embedding methods such as GLRM as a preprocessing step.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-09-04T00:48:00.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22804" PostTypeId="1" CreationDate="2017-09-04T01:22:53.530" Score="1" ViewCount="152" Body="&lt;p&gt;What is the proper procedure for finetuning a deep fully convolutional neural network with skip connections for transfering between different tasks? For a 'typical' (no skip connections, not fully convolutional) neural network, at least the last layer is &quot;fully connected&quot;, which is to say the nodes in this layer use every node in the previous layer as input, and the number of outputs is equal to the number of class labels &lt;em&gt;M&lt;/em&gt;. In &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Long_Fully_Convolutional_Networks_2015_CVPR_paper.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Shelhamer's original FCN paper&lt;/a&gt; and the follow-up PAMI paper, there are details about how to implement this type of finetuning (also sometimes referred to as feature extraction); however, there aren't any details as to how it should be done with a network with skip connections like a &lt;a href=&quot;https://arxiv.org/abs/1512.03385&quot; rel=&quot;nofollow noreferrer&quot;&gt;ResNet&lt;/a&gt; or a DenseNet (sorry linked removed due to reputation). Does anyone have any advice on best practice here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(x-posted to Reddit and Caffe forums)&lt;/p&gt;&#xA;" OwnerUserId="31776" LastActivityDate="2017-09-04T01:22:53.530" Title="Finetuning a deep fully convolutional neural network with skip connections" Tags="|machine-learning|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22805" PostTypeId="1" CreationDate="2017-09-04T02:23:08.780" Score="3" ViewCount="389" Body="&lt;p&gt;I am trying to predict the extent of suitability (0.5 to 3.5) for a section of non-fiction books based on a few parameters. This is for a set of subscribers that we want to predict books suitability: based on score I plan to display star ratings: most suitable(2.5 to 3.5), mildly suitable(1.5 to 2.5), not relevant (0.5 to 1.5).&lt;/p&gt;&#xA;&lt;h3&gt;Example dimensions:&lt;/h3&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;price: continuous&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;subject: categorical (humor, adventure, mystery, sports, non-fiction)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;hardbound: binary&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;recency of book: categorical&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;We initially start with a default profile for all user and after getting feedback from him (as 1, 2 or 3 stars), build a customized profile.&lt;/p&gt;&#xA;&lt;p&gt;Given that we will initially start with small volume of user data, looking to do a &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;k&lt;/em&gt;-NN&lt;/a&gt; to identify nearest neighbors on the dimensions, and calculate suitability score based on weighted suitability of those neighbors.&lt;/p&gt;&#xA;&lt;p&gt;Given the categorical data, I wanted to create &lt;strong&gt;custom distance matrices&lt;/strong&gt; so that I could provide the relative distance between categories. (e.g. to indicate that humor and adventure are close preferences; sports and nonfiction are far apart)&lt;/p&gt;&#xA;&lt;h3&gt;Two questions:&lt;/h3&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is my current approach valid- any feedback or pointers?&lt;/li&gt;&#xA;&lt;li&gt;What specific &lt;em&gt;k&lt;/em&gt;-NN implementation (package) would allow me to provide my custom distance matrices?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38767" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-05T02:26:07.440" Title="Developing KNN with custom distance matrices" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22807" PostTypeId="1" CreationDate="2017-09-04T07:56:26.763" Score="1" ViewCount="822" Body="&lt;p&gt;I've trained a gradient boosting classification model. But, suppose i've a set of fixed features F1,F2....Fn and new features which are added weekly (no. of actions done in that week). So, after 2 weeks dataset to be trained on is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Fixed           Dynamic&#xA;F1 ,F2 .....Fn    W1 ,W2&#xA;&#xA;After 3 weeks&#xA;&#xA;  Fixed           Dynamic&#xA;F1 ,F2 .....Fn    W1 ,W2, W3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do we approach this problem on production server, is there any approach available which allow model to be retrained on new features and not only on new observations ? &lt;/p&gt;&#xA;" OwnerUserId="33134" LastActivityDate="2022-09-25T03:04:05.407" Title="how to retrain model with periodic new features?" Tags="|machine-learning|classification|online-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22808" PostTypeId="2" ParentId="22795" CreationDate="2017-09-04T08:08:17.450" Score="5" Body="&lt;p&gt;Yes. Clustering algorithms such as K-means do need feature scaling before they are fed to the algo. Since, clustering techniques use &lt;strong&gt;Euclidean Distance&lt;/strong&gt; to form the cohorts, it will be wise e.g to scale the variables having heights in meters and weights in KGs before calculating the distance.&lt;/p&gt;&#xA;" OwnerUserId="38815" LastActivityDate="2017-09-04T08:08:17.450" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22809" PostTypeId="2" ParentId="22807" CreationDate="2017-09-04T08:30:16.413" Score="1" Body="&lt;p&gt;The way gradient boosting is constructed and trained, there is not an obvious solution for this without just training from scratch. Other models might be more suitable for this (Adding this to a neural network and retraining this will take less time than from scratch I think). Another approach would be to use these time-based lag features as a time series, that way you just input your features and it will learn relationships between the past few weeks instead.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-04T08:30:16.413" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22810" PostTypeId="2" ParentId="22795" CreationDate="2017-09-04T09:01:21.133" Score="7" Body="&lt;p&gt;Clustering algorithms are certainly effected by the feature scaling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that you have two features:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;weight (in Lbs)&lt;/li&gt;&#xA;&lt;li&gt;height (in Feet)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;... and we are using these to predict whether a person needs a 'S' or 'L' size shirt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are using weight+height for that, and in our trained set let's say we have two people already in clusters: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Adam (175Lbs+5.9ft) in 'L'&lt;/li&gt;&#xA;&lt;li&gt;Lucy (115Lbs+5.2ft) in 'S'. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;We have a new person - Alan (140Lbs+6.1ft.), and your clustering algo will put it in the cluster which is nearest.  So, if we don't scale the features here, the height is not having much effect and Alan will be allotted in 'S' cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, we need to scale it. Scikit Learn provides many functions for scaling. One you can use is &lt;code&gt;sklearn.preprocessing.MinMaxScaler&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="38818" LastEditorUserId="29575" LastEditDate="2017-09-04T17:00:33.157" LastActivityDate="2017-09-04T17:00:33.157" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22811" PostTypeId="1" CreationDate="2017-09-04T09:10:24.857" Score="3" ViewCount="1514" Body="&lt;p&gt;I want to learn more about the recommender system topic.&#xA;I am very interested in the usage of different database systems for this use case.&lt;/p&gt;&#xA;&lt;p&gt;My problem is that I cannot find a good overview of different &lt;strong&gt;architectures of recommender systems&lt;/strong&gt;, especially with the focus on the database part.&lt;/p&gt;&#xA;&lt;p&gt;Can someone help me out with a good reference or some own thoughts?&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Thanks a lot&lt;/strong&gt;. As interesting as this topic is for me as hard it seems to get some concrete information about it.&lt;/p&gt;&#xA;" OwnerUserId="35910" LastEditorUserId="83275" LastEditDate="2021-03-12T05:46:25.793" LastActivityDate="2021-03-12T05:46:25.793" Title="Is there an overview over recommender system architectures?" Tags="|machine-learning|recommender-system|databases|nosql|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22813" PostTypeId="1" AcceptedAnswerId="22817" CreationDate="2017-09-04T11:30:19.893" Score="14" ViewCount="24980" Body="&lt;p&gt;What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. scikit-learn's TF-IDF vectorizer transforms text data into sparse matrices. I can use these sparse matrices directly with a Naive Bayes classifier for example. But what's the way to also take into account the other features? Should I de-sparsify the tf-idf representation of the text and combine the features and the text into one DataFrame? Or can I keep the sparse matrix as a separate column for example? What's the correct way to do this?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="29169" LastEditDate="2021-05-06T17:38:40.067" LastActivityDate="2021-05-06T17:38:40.067" Title="Using TF-IDF with other features in scikit-learn" Tags="|python|scikit-learn|pandas|tfidf|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22814" PostTypeId="1" CreationDate="2017-09-04T12:09:46.973" Score="8" ViewCount="4247" Body="&lt;p&gt;I would like to know if the class weighting is also used in evaluating the loss in the validation data during the training. If not, is there a way to adjust the fit() function so that it takes into account the class weighting during the validation loss calculation. &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2023-04-10T15:57:10.583" Title="Class weighting during validation in Keras" Tags="|neural-network|classification|keras|class-imbalance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22815" PostTypeId="2" ParentId="22814" CreationDate="2017-09-04T12:47:30.087" Score="4" Body="&lt;p&gt;As described in the tensorflow docs[1], &lt;code&gt;validation_data&lt;/code&gt; can be a tuple &lt;code&gt;(inputs, targets, sample_weights)&lt;/code&gt;.  So, you can use this parameter for class weighting during the validation loss calculation.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;[1] &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38657" LastEditorUserId="139026" LastEditDate="2023-04-10T15:57:10.583" LastActivityDate="2023-04-10T15:57:10.583" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="22816" PostTypeId="1" CreationDate="2017-09-04T14:26:10.337" Score="1" ViewCount="482" Body="&lt;p&gt;I ran some text classification on text data. When I tried to save the &lt;code&gt;SGD&lt;/code&gt; classifier object, I was surprised to find it is more than 4GBs (even though my input is ~600 000 rows, 700MB). What's in that object that makes it so huge? Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="34028" LastEditDate="2017-09-04T17:00:30.640" LastActivityDate="2017-09-05T21:34:50.470" Title="What's in an SGD classifier object?" Tags="|scikit-learn|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22817" PostTypeId="2" ParentId="22813" CreationDate="2017-09-04T14:58:45.353" Score="12" Body="&lt;p&gt;scikit-learn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.html#sklearn.pipeline.FeatureUnion&quot; rel=&quot;noreferrer&quot;&gt;FeatureUnion&lt;/a&gt; concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found &lt;a href=&quot;https://scikit-learn.org/stable/modules/compose.html#feature-union&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2020-05-09T20:48:25.390" LastActivityDate="2020-05-09T20:48:25.390" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22819" PostTypeId="1" AcceptedAnswerId="22855" CreationDate="2017-09-04T17:37:41.517" Score="3" ViewCount="487" Body="&lt;p&gt;I'm trying to create a classifier to distinguish different boats by their trajectories. I have training data of the longitude and latitude of a boat and time in seconds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vessels like a ferry will have a straight predictable trajectory between two points, whereas fishing vessels can have zig-zag like trajectories for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial approach is to create features for example the mean speed, standard deviation of the speed, standard deviation on the course, such that each trajectory table is distilled into 1 row of features. Then I can train something like a random forest classifier on these rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a good approach, any other suggestions that could account for the characteristic trajectory shapes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="28658" LastActivityDate="2017-09-05T21:17:00.947" Title="Approach to classify spatial trajectories of vessels" Tags="|machine-learning|classification|geospatial|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22820" PostTypeId="2" ParentId="22811" CreationDate="2017-09-04T19:21:20.113" Score="4" Body="&lt;p&gt;A great source is the personal page of &lt;a href=&quot;https://xamat.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Xavier Amatriain&lt;/a&gt; (former Head of Engineering at Netflix and Quora). There are several links at publications detailing the system and architecture of recommender systems. In particular you could take a look at this &lt;a href=&quot;https://xamat.github.io/pubs/recsys12-tutorial.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Building industrial-scale real-world recommender systems&lt;/a&gt;, there is also this talk &lt;a href=&quot;https://www.youtube.com/watch?v=VJOtr47V0eo&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lessons Learned from Building Real­-Life Recommender Systems&lt;/a&gt; and this blog post &lt;a href=&quot;https://medium.com/netflix-techblog/system-architectures-for-personalization-and-recommendation-e081aa94b5d8&quot; rel=&quot;nofollow noreferrer&quot;&gt;System Architectures for Personalization and Recommendation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-09-04T19:21:20.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22821" PostTypeId="2" ParentId="22762" CreationDate="2017-09-04T22:46:29.380" Score="46" Body="&lt;p&gt;Assuming your target is (0,1), then the classifier would output a probability matrix of dimension (N,2). &#xA;The first index refers to the probability that the data belong to class 0, and the second  refers to the probability that the data belong to class 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two would sum to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then output the result by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probability_class_1 = model.predict_proba(X)[:, 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you have k classes, the output would be (N,k), you would have to specify the probability of which class you want.&lt;/p&gt;&#xA;" OwnerUserId="37924" LastEditorUserId="89923" LastEditDate="2020-02-12T14:46:33.447" LastActivityDate="2020-02-12T14:46:33.447" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22822" PostTypeId="2" ParentId="11024" CreationDate="2017-09-04T23:07:17.493" Score="1" Body="&lt;p&gt;Likelihood encoding is still not available on scikit learn. You may do it by creating a dictionary, and then do a replace function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dict1 = {'cate1':1,'cate2':1,'cate3':2,'cate4':2}&#xA;for i in df.shape[0]:&#xA;    df.iloc[i] = dict1[df.iloc[i]] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37924" LastActivityDate="2017-09-04T23:07:17.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22823" PostTypeId="2" ParentId="20213" CreationDate="2017-09-04T23:20:51.820" Score="0" Body="&lt;p&gt;Classically, autocorrelation is how you model time-series data and imply seasonality. This is mostly mathematical but not much related to machine learning. &#xA;Once you have the model, RSI/ADX, or other momentum index, EWMA or other moving average, are smoothing technique which help you to generate signals, which enable you to know where exactly you are in the time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to compare the similarity of two time series with the same time index, you may compare their euclidean distances, or compute a covariance matrix. These allow you to explore the data better and decide what you can do next.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine Learning algorithm require labelled data for training. You may provide a specific context of your use case to see if there are some mature machine learning algorithm to be applied on your time-series data&lt;/p&gt;&#xA;" OwnerUserId="37924" LastActivityDate="2017-09-04T23:20:51.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22825" PostTypeId="1" CreationDate="2017-09-05T00:50:03.123" Score="4" ViewCount="7606" Body="&lt;p&gt;Good Day,&lt;/p&gt;&#xA;&lt;p&gt;Microsoft offers their Azure Machine Learning Platform: &lt;a href=&quot;https://azure.microsoft.com/en-ca/services/machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://azure.microsoft.com/en-ca/services/machine-learning/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Azure Machine Learning is designed for applied machine learning. Use best-in-class algorithms and a simple drag-and-drop interface—and go from idea to deployment in a matter of clicks.&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;Use Azure Machine Learning to deploy your model into production as a web service in minutes—a web service that can be called from any device, anywhere, and that can use any data source.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;By their demo and their photos online it looks like a simple GUI application you can drag and drop your various pre-processing elements, estimators, and testing schemes and appears to make it easy to get started on Machine learning projects.&lt;/p&gt;&#xA;&lt;p&gt;What no website selling a product will do is advertise their downsides.  With a GUI like Weka you lack the ability to fine tune and tweek the different parameters.&lt;/p&gt;&#xA;&lt;p&gt;Azure ML seems to allow hooks into Python or R to give you more control and it sure would be appealing to have to do less work to get the same results.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;What are the known disadvantages of using Azure ML for ML projects vs writing the same process in code (e.g. Sci-kit Learn or in R)?&lt;/p&gt;&#xA;" OwnerUserId="38477" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-10-16T11:22:08.900" Title="What are the disadvantages of Azure's ML vs a pure code approach (R/SKlearn)" Tags="|machine-learning|software-recommendation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22828" PostTypeId="1" CreationDate="2017-09-05T05:02:57.140" Score="13" ViewCount="45506" Body="&lt;p&gt;I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number of clusters I expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read the sklearn documentation of DBSCAN and Affinity Propagation, where both of them requires a distance matrix (not cosine similarity matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really, I'm just looking for any algorithm that doesn't require &lt;strong&gt;a) a distance metric and b) a pre-specified number of clusters&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of an algorithm that would do that? &lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2017-11-08T23:30:10.447" Title="Clustering with cosine similarity" Tags="|machine-learning|data-mining|clustering|text-mining|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22830" PostTypeId="1" AcceptedAnswerId="22850" CreationDate="2017-09-05T06:48:06.457" Score="2" ViewCount="2470" Body="&lt;p&gt;I have created a deep neural network that solves the spiral dataset classification problem. However, when measuring the performance, the accuracy goes up and down but always stays at around 50% - which is of course very bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image below shows loss and accuracy of 100 epochs of training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/y0XyL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y0XyL.png&quot; alt=&quot;Loss and accuracy of 100 epochs of training.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I fix this? I have done research and I don't see where the error is in my code. Is the error in the architecture of my network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Make sure we have the required libraries loaded&#xA;library(keras)&#xA;library(tensorflow)&#xA;library(ggplot2)&#xA;&#xA;# Load the data&#xA;spiralData = read.table(&quot;spiral.data&quot;, header=TRUE)  &#xA;&#xA;# Visualize the data&#xA;qplot(x, y, data = spiralData, colour = label)&#xA;&#xA;# Store the data in features and labels.&#xA;x&amp;lt;-c(spiralData$x)&#xA;y&amp;lt;-c(spiralData$y)&#xA;features &amp;lt;- matrix(c(x,y),nrow=length(x))&#xA;labels &amp;lt;- matrix(spiralData$label)&#xA;&#xA;# Create model.&#xA;model &amp;lt;- keras_model_sequential()&#xA;&#xA;# Add layers and compile the model.&#xA;# Our model consists of 4 hidden layers, each with 6 neurons.&#xA;model %&amp;gt;% &#xA;  layer_dense(units = 6, activation = 'tanh', input_shape = c(2)) %&amp;gt;%  &#xA;  layer_dense(units = 6, activation = 'tanh') %&amp;gt;%   &#xA;  layer_dense(units = 6, activation = 'tanh') %&amp;gt;%  &#xA;  layer_dense(units = 6, activation = 'tanh') %&amp;gt;%   &#xA;  layer_dense(units = 1, activation = 'sigmoid') %&amp;gt;% &#xA;  compile(&#xA;    optimizer = 'rmsprop',&#xA;    loss = 'binary_crossentropy',&#xA;    metrics = c('accuracy')&#xA;  )&#xA;&#xA;# Train the model, iterating on the data in batches of 32 samples.&#xA;# Also, visualize the training process.&#xA;model %&amp;gt;% fit(features, labels, epochs=100, batch_size=32)&#xA;&#xA;# Evalute the model&#xA;score = model %&amp;gt;% evaluate(features, labels, batch_size=32)&#xA;print(score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My &quot;spiral.data&quot; dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x y label&#xA;1 0 1&#xA;-1 0 0&#xA;0.971354 0.209317 1&#xA;-0.971354 -0.209317 0&#xA;0.906112 0.406602 1&#xA;-0.906112 -0.406602 0&#xA;0.807485 0.584507 1&#xA;-0.807485 -0.584507 0&#xA;0.679909 0.736572 1&#xA;-0.679909 -0.736572 0&#xA;0.528858 0.857455 1&#xA;-0.528858 -0.857455 0&#xA;0.360603 0.943128 1&#xA;-0.360603 -0.943128 0&#xA;0.181957 0.991002 1&#xA;-0.181957 -0.991002 0&#xA;-3.07692e-06 1 1&#xA;3.07692e-06 -1 0&#xA;-0.178211 0.970568 1&#xA;0.178211 -0.970568 0&#xA;-0.345891 0.90463 1&#xA;0.345891 -0.90463 0&#xA;-0.496812 0.805483 1&#xA;0.496812 -0.805483 0&#xA;-0.625522 0.67764 1&#xA;0.625522 -0.67764 0&#xA;-0.727538 0.52663 1&#xA;0.727538 -0.52663 0&#xA;-0.799514 0.35876 1&#xA;0.799514 -0.35876 0&#xA;-0.839328 0.180858 1&#xA;0.839328 -0.180858 0&#xA;-0.846154 -6.66667e-06 1&#xA;0.846154 6.66667e-06 0&#xA;-0.820463 -0.176808 1&#xA;0.820463 0.176808 0&#xA;-0.763975 -0.342827 1&#xA;0.763975 0.342827 0&#xA;-0.679563 -0.491918 1&#xA;0.679563 0.491918 0&#xA;-0.57112 -0.618723 1&#xA;0.57112 0.618723 0&#xA;-0.443382 -0.71888 1&#xA;0.443382 0.71888 0&#xA;-0.301723 -0.78915 1&#xA;0.301723 0.78915 0&#xA;-0.151937 -0.82754 1&#xA;0.151937 0.82754 0&#xA;9.23077e-06 -0.833333 1&#xA;-9.23077e-06 0.833333 0&#xA;0.148202 -0.807103 1&#xA;-0.148202 0.807103 0&#xA;0.287022 -0.750648 1&#xA;-0.287022 0.750648 0&#xA;0.411343 -0.666902 1&#xA;-0.411343 0.666902 0&#xA;0.516738 -0.559785 1&#xA;-0.516738 0.559785 0&#xA;0.599623 -0.43403 1&#xA;-0.599623 0.43403 0&#xA;0.65738 -0.294975 1&#xA;-0.65738 0.294975 0&#xA;0.688438 -0.14834 1&#xA;-0.688438 0.14834 0&#xA;0.692308 1.16667e-05 1&#xA;-0.692308 -1.16667e-05 0&#xA;0.669572 0.144297 1&#xA;-0.669572 -0.144297 0&#xA;0.621838 0.27905 1&#xA;-0.621838 -0.27905 0&#xA;0.551642 0.399325 1&#xA;-0.551642 -0.399325 0&#xA;0.462331 0.500875 1&#xA;-0.462331 -0.500875 0&#xA;0.357906 0.580303 1&#xA;-0.357906 -0.580303 0&#xA;0.242846 0.635172 1&#xA;-0.242846 -0.635172 0&#xA;0.12192 0.664075 1&#xA;-0.12192 -0.664075 0&#xA;-1.07692e-05 0.666667 1&#xA;1.07692e-05 -0.666667 0&#xA;-0.118191 0.643638 1&#xA;0.118191 -0.643638 0&#xA;-0.228149 0.596667 1&#xA;0.228149 -0.596667 0&#xA;-0.325872 0.528323 1&#xA;0.325872 -0.528323 0&#xA;-0.407954 0.441933 1&#xA;0.407954 -0.441933 0&#xA;-0.471706 0.341433 1&#xA;0.471706 -0.341433 0&#xA;-0.515245 0.231193 1&#xA;0.515245 -0.231193 0&#xA;-0.537548 0.115822 1&#xA;0.537548 -0.115822 0&#xA;-0.538462 -1.33333e-05 1&#xA;0.538462 1.33333e-05 0&#xA;-0.518682 -0.111783 1&#xA;0.518682 0.111783 0&#xA;-0.479702 -0.215272 1&#xA;0.479702 0.215272 0&#xA;-0.423723 -0.306732 1&#xA;0.423723 0.306732 0&#xA;-0.353545 -0.383025 1&#xA;0.353545 0.383025 0&#xA;-0.272434 -0.441725 1&#xA;0.272434 0.441725 0&#xA;-0.183971 -0.481192 1&#xA;0.183971 0.481192 0&#xA;-0.0919062 -0.500612 1&#xA;0.0919062 0.500612 0&#xA;1.23077e-05 -0.5 1&#xA;-1.23077e-05 0.5 0&#xA;0.0881769 -0.480173 1&#xA;-0.0881769 0.480173 0&#xA;0.169275 -0.442687 1&#xA;-0.169275 0.442687 0&#xA;0.2404 -0.389745 1&#xA;-0.2404 0.389745 0&#xA;0.299169 -0.324082 1&#xA;-0.299169 0.324082 0&#xA;0.343788 -0.248838 1&#xA;-0.343788 0.248838 0&#xA;0.373109 -0.167412 1&#xA;-0.373109 0.167412 0&#xA;0.386658 -0.0833083 1&#xA;-0.386658 0.0833083 0&#xA;0.384615 1.16667e-05 1&#xA;-0.384615 -1.16667e-05 0&#xA;0.367792 0.0792667 1&#xA;-0.367792 -0.0792667 0&#xA;0.337568 0.15149 1&#xA;-0.337568 -0.15149 0&#xA;0.295805 0.214137 1&#xA;-0.295805 -0.214137 0&#xA;0.24476 0.265173 1&#xA;-0.24476 -0.265173 0&#xA;0.186962 0.303147 1&#xA;-0.186962 -0.303147 0&#xA;0.125098 0.327212 1&#xA;-0.125098 -0.327212 0&#xA;0.0618938 0.337147 1&#xA;-0.0618938 -0.337147 0&#xA;-1.07692e-05 0.333333 1&#xA;1.07692e-05 -0.333333 0&#xA;-0.0581615 0.31671 1&#xA;0.0581615 -0.31671 0&#xA;-0.110398 0.288708 1&#xA;0.110398 -0.288708 0&#xA;-0.154926 0.251167 1&#xA;0.154926 -0.251167 0&#xA;-0.190382 0.206232 1&#xA;0.190382 -0.206232 0&#xA;-0.215868 0.156247 1&#xA;0.215868 -0.156247 0&#xA;-0.230974 0.103635 1&#xA;0.230974 -0.103635 0&#xA;-0.235768 0.050795 1&#xA;0.235768 -0.050795 0&#xA;-0.230769 -1e-05 1&#xA;0.230769 1e-05 0&#xA;-0.216903 -0.0467483 1&#xA;0.216903 0.0467483 0&#xA;-0.195432 -0.0877067 1&#xA;0.195432 0.0877067 0&#xA;-0.167889 -0.121538 1&#xA;0.167889 0.121538 0&#xA;-0.135977 -0.14732 1&#xA;0.135977 0.14732 0&#xA;-0.101492 -0.164567 1&#xA;0.101492 0.164567 0&#xA;-0.0662277 -0.17323 1&#xA;0.0662277 0.17323 0&#xA;-0.0318831 -0.173682 1&#xA;0.0318831 0.173682 0&#xA;6.15385e-06 -0.166667 1&#xA;-6.15385e-06 0.166667 0&#xA;0.0281431 -0.153247 1&#xA;-0.0281431 0.153247 0&#xA;0.05152 -0.13473 1&#xA;-0.05152 0.13473 0&#xA;0.0694508 -0.112592 1&#xA;-0.0694508 0.112592 0&#xA;0.0815923 -0.088385 1&#xA;-0.0815923 0.088385 0&#xA;0.0879462 -0.063655 1&#xA;-0.0879462 0.063655 0&#xA;0.0888369 -0.0398583 1&#xA;-0.0888369 0.0398583 0&#xA;0.0848769 -0.018285 1&#xA;-0.0848769 0.018285 0&#xA;0.0769231 3.33333e-06 1&#xA;-0.0769231 -3.33333e-06 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Visualized, the dataset looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Y2MOg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y2MOg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34322" LastActivityDate="2017-09-05T19:42:49.917" Title="Deep Neural Network using Keras/Tensorflow solves Spiral Dataset Classification. But Accuracy is stuck around 50%" Tags="|deep-learning|tensorflow|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22831" PostTypeId="1" AcceptedAnswerId="22849" CreationDate="2017-09-05T07:19:45.160" Score="0" ViewCount="1975" Body="&lt;p&gt;I need a tool that lets me create a solution that fullfills the following requirements:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Needs to be interactive and user friendly:  Possibility to apply filters, change dimensions, etc.. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Needs to be updated online: Must be able to process data in real time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Needs to be able to handle huge ammount of data and support big data technologies.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Bonus points for python or R libraries but its definetely not a must.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking on getting started with D3py or google charts but I dont know if they are the best options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would you recommend and why?&lt;/p&gt;&#xA;" OwnerUserId="31736" LastActivityDate="2017-09-05T15:12:17.043" Title="Data Visualization Tool recomendations" Tags="|visualization|" AnswerCount="2" CommentCount="0" ClosedDate="2017-09-06T13:58:10.967" ContentLicense="CC BY-SA 3.0" />
  <row Id="22832" PostTypeId="1" CreationDate="2017-09-05T07:20:56.873" Score="2" ViewCount="759" Body="&lt;p&gt;I am trying to train a LSTM network to forecast time steps further.&#xA;I have a list of queries and the current question is based on one among them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The validation loss (using mse) is always lower than the Train loss (mse), I know I am under fitting hence, generalizing pretty badly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So atleast overfit for sometime, what changes can be brought in the network. Below is the code for the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;learning_rate = 0.001 &#xA;n_neurons = [150, 80, 60, 40, 25, 10]&#xA;dropout = 0.2&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;def fit_lstm(train, n_lag, n_seq, n_batch, nb_epoch, n_neurons, dropout=dropout, reset_state=False):&#xA;    # reshape training into [samples, timesteps, features]...&#xA;    X, y = train[:, :n_lag], train[:, n_lag:]&#xA;    X = X.reshape(X.shape[0], 1, X.shape[1])&#xA;    # design network..&#xA;    model = Sequential()&#xA;    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), &#xA;                 dropout_U=dropout, stateful=True, return_sequences=True))&#xA;    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), &#xA;                dropout_U=dropout, stateful=True, return_sequences=True))&#xA;    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), &#xA;                 stateful=True, return_sequences=False)) &#xA;    model.add(Dense(y.shape[1], activation='tanh'))&#xA;    tic = time.time() &#xA;    model.compile(optimizer=adam, loss='mse') #, metrics=[mean_absolute_percentage_error]&#xA;    # fit network..&#xA;    loss, val_loss = list(), list() &#xA;    for i in range(nb_epoch):&#xA;        print('Running Epoch ==&amp;gt; %s' %i)&lt;br&gt;&#xA;        history = model.fit(X, y, nb_epoch=1, batch_size=n_batch, validation_split=0.1,&#xA;                callbacks=[early_stop,reduce_lr], verbose=2, shuffle=False)&#xA;        loss.append(history.history['loss'])&#xA;        val_loss.append(history.history['val_loss'])&#xA;        model.reset_states() ## clears the state...&#xA;    toc = time.time()&#xA;    print('====='*10)&#xA;    print('Total computation time to train the Model : %0.2f' %((toc - tic) * 100) + ' secs')&lt;br&gt;&#xA;    return model, loss, val_loss&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be highly appreciable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastActivityDate="2017-12-14T08:25:44.297" Title="LSTM Validation MSE always lower than Train MSE" Tags="|time-series|keras|forecast|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22833" PostTypeId="2" ParentId="22831" CreationDate="2017-09-05T07:33:53.267" Score="1" Body="&lt;p&gt;If user friendly and interactive matter, then how about Tableau (or QlikView). You can work directly on the data and can connect to 'big data' sources. Also you can trigger R / Python scripts from there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want something more customizes, then maybe Shiny.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-09-05T07:33:53.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22834" PostTypeId="2" ParentId="22828" CreationDate="2017-09-05T07:48:29.723" Score="12" Body="&lt;p&gt;First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You said you have cosine similarity between your records, so this is actually a distance matrix. You can use this matrix as an input into some clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'd suggest to start with hierarchical clustering - it does not require defined number of clusters and you can either input data and select a distance, or input a distance matrix (where you calculated the distance in some way).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the hierarchical clustering is expensive to calculate, so if you have a lot of data, you can start with just sample.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-09-05T07:48:29.723" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22836" PostTypeId="2" ParentId="21982" CreationDate="2017-09-05T08:49:15.180" Score="0" Body="&lt;p&gt;I think Linear regression is more feasible than time series analysis here, becasuse  I think you have lots of categorical variables, and time series analysis works better with purely numeric data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of your categorical vriables are likely to be NA, and many might have high cardinality and thus might not be suited for one-hot-encoding. So you have to choose an algorithm that can handle NA values well and can deal with many unique categorical variables turned into indicator variables. You can also handle this with appropriate preprocessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you didn't tell use whether your dataset has sub-daily (e.g. hourly) or daily resolution. I think daily resolution is too coarse (weather may change several times per day), guest arrivals may peak in the morning or evening. So time series analysis shines when you want to determine, say, the periodicity (which is likely on an hourly scale for the workdays most restaurants), but your variables seem to be on the daily level and less predictable.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2017-09-05T09:00:53.173" LastActivityDate="2017-09-05T09:00:53.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22837" PostTypeId="2" ParentId="22832" CreationDate="2017-09-05T10:41:50.003" Score="2" Body="&lt;p&gt;One important thing to start with is to check that your targets are in [-1, 1] range because you have a 'tanh' as output function. You should also analyze the distribution of your targets (True class). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, one of the first steps that I would recommend is to try to overfit your model. Make a model 'complex enough' and overfit small proportion of your initial data set (20% maybe?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model does not overfit, there are two possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Model architecture not suited for this task (maybe lstm) or model not complex enough&lt;/li&gt;&#xA;&lt;li&gt;Your data is not adapted for prediction task.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If your model overfits (Good sign):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Try to increase the complexity of your model when increasing the size of the samples (training set). Do not use any dropout or regularization and increase the number of layers instead of the number of units/layer. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="38355" LastEditorUserId="38355" LastEditDate="2017-12-14T08:25:44.297" LastActivityDate="2017-12-14T08:25:44.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22838" PostTypeId="1" CreationDate="2017-09-05T11:42:13.790" Score="2" ViewCount="1721" Body="&lt;p&gt;I think that the vanishing gradient problem occurs when the derivative of activation function is less than 1.&#xA;The deeper the neural network is, the f' * f' * f'... operation will result in the gradient closer to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if we use hard-sigmoid as an activation function (its derivative is 0 or 1), the vanishing gradient problem can be solved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the dying problem (derivative = 0 @ ReLU) is different from the vanishing gradient problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it right?&lt;/p&gt;&#xA;" OwnerUserId="37810" LastActivityDate="2017-12-04T20:36:52.060" Title="What is the relationship between hard-sigmoid function and vanishing gradient descent problem?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22839" PostTypeId="2" ParentId="22825" CreationDate="2017-09-05T12:29:38.700" Score="5" Body="&lt;p&gt;I learned the Azure ML platform after scikit-learn (Python) and caret (R).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Low or no coding involved. It is more like Orange in that you set up the ML exercise as a large process flow. It is easier for non-technical people to get started with.&lt;/li&gt;&#xA;&lt;li&gt;Easy to convert to production-ready. You click a button and it deploys a web-service that you could send to a programmer to use. In Python, you would need to add another library and in R it is even dicier.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disadvantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fewer algorithms (e.g. no XGBoost) and other transformations (e.g. NLP) built-in. You can of course reference Python and R to do this, but that is much tougher than using the GUI. If you can figure out how to call Python/R from within Azure ML, you likely can do it on your own.&lt;/li&gt;&#xA;&lt;li&gt;Price.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-05T12:29:38.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22840" PostTypeId="2" ParentId="22058" CreationDate="2017-09-05T12:48:44.977" Score="7" Body="&lt;p&gt;In order to set the number of threads used in Theano (and, therefore, the number of CPU cores), you'll need to set a few parameters in the environment:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import os&#xA;os.environ['MKL_NUM_THREADS'] = '16'&#xA;os.environ['GOTO_NUM_THREADS'] = '16'&#xA;os.environ['OMP_NUM_THREADS'] = '16'&#xA;os.eviron['openmp'] = 'True'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should allow you to use all cores of all CPUs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can, of course, also be done in Tensorflow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;from keras.backend import tensorflow_backend as K&#xA;&#xA;with tf.Session(config=tf.ConfigProto(&#xA;                    intra_op_parallelism_threads=16)) as sess:&#xA;    K.set_session(sess)&#xA;    &amp;lt;Your Keras code&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2017-09-05T12:54:58.607" LastActivityDate="2017-09-05T12:54:58.607" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22846" PostTypeId="2" ParentId="16247" CreationDate="2017-09-05T14:23:59.657" Score="1" Body="&lt;p&gt;Adding to what &lt;a href=&quot;https://datascience.stackexchange.com/a/16250/38732&quot;&gt;Himanshu Rai&lt;/a&gt; said, you should be careful not to oversample before the StratifiedKFold, once you risk to put the same sample in both the training and testing folds &lt;a href=&quot;http://www.marcoaltini.com/blog/dealing-with-imbalanced-data-undersampling-oversampling-and-proper-cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;check this&lt;/a&gt;, (where you took your image from) and that doesn't really evaluate your model's capacity to data that it never saw. What i did was to use StratifiedKFold, and then oversample or SMOTE (or whatever you want) all the training folds separately, by only looking at that fold's training data. Then use all those folds to validate your models.&lt;/p&gt;&#xA;" OwnerUserId="38732" LastEditorUserId="38732" LastEditDate="2017-09-05T14:44:28.343" LastActivityDate="2017-09-05T14:44:28.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22849" PostTypeId="2" ParentId="22831" CreationDate="2017-09-05T15:12:17.043" Score="1" Body="&lt;p&gt;Some tools/software are listed below. Most of these can do what you want but require input in other areas such as in development of a UI or the tool may have a learning curve to overcome.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://developers.google.com/chart/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Chart&lt;/a&gt;: I guess this sets the benchmark. Really user friendly, has a large chart gallery and can process data in real time.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.tableau.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tableau&lt;/a&gt;: Has two variants &quot;server&quot; and &quot;cloud version&quot;. Can handle big data but may require a licence depending on your domain&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://d3js.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;D3&lt;/a&gt;: A JS library for data viz and is light weight. User friendliness and interactive nature depends on how you design the UI. Can also display data in real time.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.fusioncharts.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fusion Charts&lt;/a&gt;: Another JS library for web and mobile devices. Deals with data in XML of JSON format. Requires a licence.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.qlik.com/us/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Qlikview&lt;/a&gt;: One of Tableau's competitors. Highly customisable, so there may be a learning curve to get accustomed with the tool.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://powerbi.microsoft.com/en-us/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Microsoft Power BI&lt;/a&gt;: Surprisingly good. Ticks all your boxes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="20429" LastActivityDate="2017-09-05T15:12:17.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22850" PostTypeId="2" ParentId="22830" CreationDate="2017-09-05T16:05:57.810" Score="5" Body="&lt;p&gt;Your network is actually working, it just takes a lot of epochs to learn the spiral. In fact you can see from your learning curves that learning is still occurring, just not much per epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try 60,000 epochs . . . when I try your model (in Python, but still same data and model) using 60,000 epochs I get loss under 0.0001 and accuracy of 100% reliably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few factors involved in why you need this amount of iteration:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The data set size is small, which means you get less updates to weights per epoch. You need to compensate by increasing the number of iterations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You have a &quot;starved&quot; network topology that can just about learn the spiral, but needs to be quite precisely optimised before it starts performing well. You could increase the number of neurons per hidden layer slightly. Or maybe adding another hidden layer:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If I add one more hidden layer, size 6, &lt;code&gt;tanh&lt;/code&gt; activation, the network learns 100% accuracy in under 20,000 epochs.&lt;/li&gt;&#xA;&lt;li&gt;If instead, I increase the original four hidden layers to size 16, the network learns 100% accuracy in under 10,000 epochs.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;tanh&lt;/code&gt; is not optimal for a deep network, because the gradient diminishes in the deeper parts of the model. RMSProp will compensate for that, but still changing to &lt;code&gt;relu&lt;/code&gt; will improve convergence speed.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If I use the four-hidden-layer model, with layer size 16 and &lt;code&gt;relu&lt;/code&gt; activation, the network converges to 100% accuracy on the training data set in around 2000 epochs.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;np.random.seed(4375689)&#xA;&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.optimizers import RMSprop&#xA;&#xA;train_data = pd.read_csv('spirals.csv').values&#xA;train_X = train_data[:,0:2]&#xA;train_y = train_data[:,2]&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(16, activation='relu', input_shape=(2,)))&#xA;model.add(Dense(16, activation='relu'))&#xA;model.add(Dense(16, activation='relu'))&#xA;model.add(Dense(16, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;model.compile(loss='binary_crossentropy', optimizer=RMSprop(),&#xA;              metrics=['accuracy'])&#xA;&#xA;history = model.fit(train_X, train_y, batch_size=32, epochs=2000, verbose=0)&#xA;&#xA;score = model.evaluate(train_X, train_y, verbose=0)&#xA;&#xA;print(score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-05T19:42:49.917" LastActivityDate="2017-09-05T19:42:49.917" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22851" PostTypeId="2" ParentId="22838" CreationDate="2017-09-05T16:49:24.467" Score="2" Body="&lt;p&gt;The vanishing gradient problem appears with activation functions that squash their input in a very small output range, like sigmoids and tanh. The input range is large and therefore the probability for it to fall in the &lt;em&gt;squashing part&lt;/em&gt; of the sigmoid is very high. Chain several sigmoids and there you have vanishing gradients,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical examples of such a type of activations are sigmoid and hyperbolic tangent. Hard sigmoids are no exception. Their main difference with normal sigmoids is that they are computationally cheaper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ReLU's are different. Half of their range is linear. That's why they don't suffer from vanishing gradients. However, they have their own problem: dying ReLU. This happens when the input &lt;em&gt;falls&lt;/em&gt; far in the left side and &lt;em&gt;cannot escape&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-09-05T16:49:24.467" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="22852" PostTypeId="2" ParentId="22796" CreationDate="2017-09-05T18:47:02.833" Score="2" Body="&lt;p&gt;It's called overfitting, your model learned to predict labels based on features which are not actually responsible for particular predictions, so when they are not presented it fails to predict right. Although there are various ways to deal with overfitting (e.g. Dropout) what you seem to need is image augmentation. It is a simple yet very powerful way to train a robust neural network. For your case - just simulate different lighting conditions by e.g. increasing/decreasing pixel values randomly for your inputs for same labels. It's also common practice to add random noise, random rotations, flip etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more info check &lt;a href=&quot;https://stackoverflow.com/questions/36144993/data-augmentation-techniques-for-small-image-datasets&quot;&gt;this&lt;/a&gt; stackoverflow answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I recently published project where I used various of Augmentation functions which you may find useful. See: &lt;a href=&quot;https://github.com/Naurislv/P12.1-Semantic-Segmentation/blob/master/augmentation.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Naurislv/P12.1-Semantic-Segmentation/blob/master/augmentation.py&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38884" LastEditorUserId="38884" LastEditDate="2017-09-05T19:07:56.660" LastActivityDate="2017-09-05T19:07:56.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22853" PostTypeId="1" AcceptedAnswerId="24821" CreationDate="2017-09-05T19:14:30.057" Score="27" ViewCount="12357" Body="&lt;p&gt;I heard Andrew Ng (in a video I unfortunately can't find anymore) talk about how the understanding of local minima in deep learning problems has changed in the sense that they are now regarded as less problematic because in high-dimensional spaces (encountered in deep learning) critical points are more likely to be saddle points or plateaus rather than local minima. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen papers (e.g. &lt;a href=&quot;http://papers.nips.cc/paper/6112-deep-learning-without-poor-local-minima.pdf&quot; rel=&quot;noreferrer&quot;&gt;this one&lt;/a&gt;) that discuss assumptions under which &quot;every local minimum is a global minimum&quot;. These assumptions are all rather technical, but from what I understand they tend to impose a structure on the neural network that make it somewhat linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a valid claim that, in deep learning (incl. nonlinear architectures), plateaus are more likely than local minima? And if so, is there a (possibly mathematical) intuition behind it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Is there anything particular about deep learning and saddle points?&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="50727" LastEditDate="2018-05-22T08:36:09.533" LastActivityDate="2018-07-28T11:38:28.080" Title="local minima vs saddle points in deep learning" Tags="|machine-learning|deep-learning|optimization|convergence|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="22854" PostTypeId="2" ParentId="22825" CreationDate="2017-09-05T20:54:45.730" Score="4" Body="&lt;p&gt;&lt;strong&gt;Disadvantages&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Price. This is a cloud service, and therefore, it will cost you money if you wish to use it for more &quot;serious&quot; projects.&lt;/li&gt;&#xA;&lt;li&gt;Limited storage in the free version.&lt;/li&gt;&#xA;&lt;li&gt;It only has a limited number of techniques built-in, and as you mentioned with Weka, often, life can be made easier if you can simply go straight into the code to tweak things.&lt;/li&gt;&#xA;&lt;li&gt;Azure ML is a high-level tool, which therefore grants you less control over you can do. The custom code functionality is nice, but not truly what it was intended for. If you can write custom code, it probably means that you shouldn't use Azure ML anymore.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Let's not be too pessimistic, however, as it also has plenty of advantages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Advantages&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Very easy for &quot;outsiders&quot; and beginners to understand how a classification pipeline (data transformation, preprocessing, feature extraction etc.) works as it is visual and they can even build it themselves&lt;/li&gt;&#xA;&lt;li&gt;Allows you to deploy a model that you can later use in a web application later on&lt;/li&gt;&#xA;&lt;li&gt;Since this is an Azure product, it gives you a quick access to creating cloud solutions very easily&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-05T20:54:45.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22855" PostTypeId="2" ParentId="22819" CreationDate="2017-09-05T21:17:00.947" Score="0" Body="&lt;p&gt;Your data contains spatial and temporal data, which means that you can account for location and speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not sure how global the coordinates are, but perhaps grouping them per region might be interesting. It would be like putting all coordinates that you're given and defining zones of interests (zone A, B, C, etc.). Perhaps, some boats only sail in specific regions. Indeed, yachts might be found more often in the Caribbeans than in the Arctic sea. Of course, the number of zones you want to define needs to be fine-tuned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that your data accounts for an entire trip from port A to B, you could, of course, use the length of the journey as a feature, as I'm assuming that not all boats travel the same distances. You could also use the overall direction of the trip as the angle between port A and port B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You've already accounted for speed by computing the mean, standard deviation, and other metrics. What you could also do is fit a polynomial of the speed, and use its coefficients as features. Perhaps certain boats have a more steady speed than other boats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following the same thought, you could perhaps fit a polynomial describing the course of each boat and use its coefficients as features. View the course here as the change of angle. This should capture how much &quot;zig zagging&quot; a boat does. Of course, you'll need to fine-tune to pick the right polynomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-05T21:17:00.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22856" PostTypeId="2" ParentId="22816" CreationDate="2017-09-05T21:34:50.470" Score="1" Body="&lt;p&gt;I will answer, assuming that since you wanted to do text classification, your features were created using some TFIDF feature extraction technique. Next time, please specify the dimensions of your features and not only their storage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the documentation (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html&lt;/a&gt;), the object has a coef_ attributes that stores all of the weights of your model. Since for text, feature vectors are about the size of your vocabulary, it can be fairly large, especially if you do not preprocess your text using stemming or if you do not remove stopwords or irrelevant words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation also says that the size of the coef_ is number of classes * number of features. So depending on how many features and classes you have, this can get large quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could explain partially why the object is so large. Other factors could be that Python often stores the predictions in the object after you train it, so that you can access them quickly, so you would need to add another 600,000 numbers to your object. And for this object, it seems to also store the confidence scores, which is another 600,000.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-05T21:34:50.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22857" PostTypeId="2" ParentId="22813" CreationDate="2017-09-05T21:46:51.837" Score="5" Body="&lt;p&gt;Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your classifier requires you to use dense input, you might want to keep the TFIDF features as sparse, and add the other features to them in a sparse format. And then only, make the matrix dense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that, you could use &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.hstack.html&quot; rel=&quot;noreferrer&quot;&gt;scipy.sparse.hstack&lt;/a&gt;. It combines two sparse matrices together by column. &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.vstack.html&quot; rel=&quot;noreferrer&quot;&gt;scipy.sparse.vstack&lt;/a&gt; also exists. And of course, scipy also has the non-sparse version scipy.hstack and scipy.vstack&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-05T21:46:51.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22858" PostTypeId="2" ParentId="22805" CreationDate="2017-09-05T22:09:03.897" Score="1" Body="&lt;p&gt;I would personally take a look at collaborative filtering as this will take into account the information of the user itself, but also the information of similar users. If let's say, my best friend and I have the same taste in books, knowing his taste can help you guess mine! I'd recommend watching the Andrew Ng lectures on &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera about recommender systems.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;KNN might seem like a natural choice if you don't know about recommender systems (I have been there!) but please, make sure to read about them before you decide to use KNN.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-05T22:09:03.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22859" PostTypeId="1" CreationDate="2017-09-06T01:57:43.780" Score="-2" ViewCount="2396" Body="&lt;p&gt;I am trying do an Image Classification where each sample of training data contains data of the current pixel with the 8 surrounding ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where can I find examples of SVM, in python, that use 5 or more features in training?&lt;/p&gt;&#xA;" OwnerUserId="37703" LastActivityDate="2017-09-06T02:26:58.533" Title="Where can I find python code for SVM that use multiple feature data?" Tags="|svm|image-classification|libsvm|" AnswerCount="1" CommentCount="0" ClosedDate="2017-09-06T12:55:28.220" ContentLicense="CC BY-SA 3.0" />
  <row Id="22860" PostTypeId="2" ParentId="22859" CreationDate="2017-09-06T02:16:07.200" Score="1" Body="&lt;p&gt;By modifying the $2$-dimensional example in &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit Documentation about SVM&lt;/a&gt;, we have this $5$ dimensional example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;X = [[0, 0, 0, 0, 0], [1, 1, 1, 1, 1]]&#xA;y = [0, 1]&#xA;clf = svm.SVC()&#xA;clf.fit(X, y)  &#xA;print(clf.predict([[2., 2., 2., 2., 2.], [-1 for i in range(5)]]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can read more about the parameters option of SVC &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2017-09-06T02:26:58.533" LastActivityDate="2017-09-06T02:26:58.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22863" PostTypeId="1" CreationDate="2017-09-06T04:18:58.337" Score="-1" ViewCount="77" Body="&lt;p&gt;In my case, All the time-series observations are with high dimensions. Very likely, they will fall in multiple clusters(meaning multiple distribution patterns) other than a single cluster(meaning single distribution pattern). Will anomaly detection be able to cope with this scenario?&lt;/p&gt;&#xA;" OwnerUserId="38791" LastActivityDate="2017-09-06T08:04:55.890" Title="How does outlier detection work if there are multiple distribution clusters?" Tags="|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22864" PostTypeId="1" CreationDate="2017-09-06T06:13:58.977" Score="2" ViewCount="2665" Body="&lt;p&gt;After loading a dataset as DataFrame in pyspark's SQLContext, unable to use the Python DataFrame property of 'iloc' on it. Does a DataFrame created in SQLContext of pyspark behave differently and expect different methods than the python DataFrame ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error : AttributeError: 'DataFrame' object has no attribute 'iloc'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Full Code: Just loading a data_set and using a custom function to split the datset in to two sets named test_set and train_set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql import SQLContext&#xA;import numpy as np&#xA;import pandas as pd&#xA;&#xA;data_set = sqlContext.read.format('csv').options(header='true', inferSchema='true').load('/FileStore/tables/69qouxnt1504605761640/AirQualityUCI.csv')&#xA;&#xA;# Split train and test sets&#xA;def split_train_test(data, test_ratio):&#xA;    length = data.count()&#xA;    shuffled_indices = np.random.permutation(length)&#xA;    test_set_size = int(length * test_ratio)&#xA;    test_indices = shuffled_indices[:test_set_size]&#xA;    train_indices = shuffled_indices[test_set_size:]&#xA;    return data.iloc[train_indices], data.iloc[test_indices]&#xA;&#xA;train_set, test_set = split_train_test(data_set, 0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38865" LastActivityDate="2017-09-06T06:13:58.977" Title="Unable to use the Python Data Frame method &quot;iloc&quot; on a Data Frame created in pyspark's SQLContext" Tags="|python|apache-spark|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22865" PostTypeId="2" ParentId="22828" CreationDate="2017-09-06T06:32:58.663" Score="4" Body="&lt;p&gt;DBSCAN &lt;em&gt;can&lt;/em&gt; trivially be implemented with a similarity measure instead of a distance. You just need to change the &amp;lt;= epsilon into a &gt;= epsilon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HAC also works just fine with similarities (at least single-link, complete-link, UPGMA, WPGMA - don't use Ward), if you swap &quot;min&quot; and &quot;max&quot; (you want to merge with maximum similarity rather than minimum distance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are lazy, you can also just transform your similarity into a distance. If you have a fixed maximum, dist=max-sim will often do.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-06T06:32:58.663" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="22866" PostTypeId="2" ParentId="22863" CreationDate="2017-09-06T08:04:55.890" Score="1" Body="&lt;p&gt;This question is difficult to answer because anomaly detection is a whole field and not just one technique. Some simple techniques would not be able to pick up on these types of patterns but most advanced techniques would be able to pick up on them. It depends on the underlying techniques that are used, if it uses a local density estimation technique then time series that lie close to a lot of other time series will not be annotated as anomalies. If you would look at certain statistics only due to high dimensions this might not be the case.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-06T08:04:55.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22867" PostTypeId="2" ParentId="19541" CreationDate="2017-09-06T09:12:30.153" Score="0" Body="&lt;p&gt;&lt;code&gt;scikit-learn&lt;/code&gt; is a powerful and a very very simple ML library for Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your doubt is about a Classification Problem, where you're going to predict a class (True or False) given some input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html#supervised-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;classifiers&lt;/a&gt; that you can use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this example I've used &lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logistic Regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LogisticRegression&#xA;&#xA;# Input data&#xA;data = [[1.0, 2.0, 3.0, 4.0, 5.0], [6.0, 7.0, 8.0, 9.0, 10.0]]&#xA;&#xA;# Target values for input data&#xA;target = [True, False]&#xA;&#xA;# Define the model (default parameters)&#xA;model = LogisticRegression(C=1.0, penalty='l1', tol=1e-6)&#xA;&#xA;# Train the model&#xA;model.fit(data, target)&#xA;&#xA;# Once the model is trained, we make a prediction&#xA;model.predict([[1.0, 2.0, 3.0, 4.0, 5.0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you execute this code, the prediction for that data is &lt;code&gt;False&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="38901" LastActivityDate="2017-09-06T09:12:30.153" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22869" PostTypeId="2" ParentId="22616" CreationDate="2017-09-06T11:28:26.347" Score="4" Body="&lt;p&gt;You can just convert them into factors if your random forest algorithm is accepting that .Usually one hotting the categorical variables are not suggested&#xA;&lt;a href=&quot;https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is the an article gives information about categorical variables in Random Forest&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-09-06T11:28:26.347" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22871" PostTypeId="1" CreationDate="2017-09-06T13:44:30.387" Score="6" ViewCount="1879" Body="&lt;p&gt;I am attempting to train an SVM from a set of features which are both numeric and categorical, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Distance X (Numeric)&lt;/li&gt;&#xA;&lt;li&gt;Distance Y (Numeric)&lt;/li&gt;&#xA;&lt;li&gt;Font Size Difference (Numeric)&lt;/li&gt;&#xA;&lt;li&gt;Word 1 Bold (Boolean)&lt;/li&gt;&#xA;&lt;li&gt;Word 2 Bold (Boolean)&lt;/li&gt;&#xA;&lt;li&gt;Word 1 Font Size (Numeric)&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For mapping the features to a feature array I am treating true as 1 and false as 0 and then normalizing the entire feature array using Z-Scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I instead encode false as -1 so it has a numerical impact on the generation of the support vectors, with a value of 0 it does not modify the chosen vector (I assume)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should boolean features be normalized in the same way as numerics or should they be left with their encoded values?&lt;/p&gt;&#xA;" OwnerUserId="38903" LastActivityDate="2017-10-09T14:28:48.413" Title="Should boolean features be normalized and should false be -1 or 0" Tags="|feature-selection|normalization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22872" PostTypeId="2" ParentId="22828" CreationDate="2017-09-06T15:29:59.313" Score="2" Body="&lt;p&gt;All clustering methods use a distance metric of some sort. And remember that distance is essentially a dissimilarity measure. So if you normalize your similarity betwen 0 and 1, your distance is simply 1-similarity&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for algorithms that do not require a number of clusters to be specified, there are of course hierarchical clustering techniques, which essentially build a tree like structure that you can &quot;cut&quot; wherever you please (you can use some perfomance metrics to do that automatically)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X-means is a version of K-means which tries a certain number of K and picks the one that maximizes some evaluation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mean shift also &quot;finds&quot; a natural number of clusters but is sensible to other parameters such as the bandwith for instance.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-06T15:29:59.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22873" PostTypeId="1" CreationDate="2017-09-06T17:13:18.300" Score="1" ViewCount="657" Body="&lt;p&gt;I am working on evaluating when a pair of string objects can be considered equal (e.g. given that we are talking about journals, is &quot;international journal of air and water pollution&quot; the same of &quot;air and water pollution&quot;?) and I was wondering what is the proper corpus to use to build a TF IDF vectorizer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using as corpus all the distinct values of these strings belonging to the same type (in this example all the strings referring to journals). Is this a valid approach? Why? What could be other valid approaches?&lt;/p&gt;&#xA;" OwnerUserId="32503" LastEditorUserId="32503" LastEditDate="2017-09-08T15:03:29.230" LastActivityDate="2018-02-19T03:01:21.810" Title="Choosing the right corpus to build a TF IDF Vectorizer used for comparing the similarity of two strings" Tags="|text-mining|similarity|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22874" PostTypeId="1" AcceptedAnswerId="22875" CreationDate="2017-09-06T17:49:32.180" Score="-1" ViewCount="5103" Body="&lt;p&gt;I want to know who can I use the &lt;strong&gt;&lt;em&gt;tf.argmax&lt;/em&gt;&lt;/strong&gt; in &lt;em&gt;multi-dimension tensor&lt;/em&gt; in the Tensorflow using convolution neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;bellow is simple example to explain what I want exactly::&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1-D tensor example::&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [0,1,2,3]&#xA;logit_a = tf.one_hot(indices = a, depth = 4)&#xA;out_a = tf.argmax(logit_a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the &lt;strong&gt;out_a&lt;/strong&gt; will be like this [0,1,2,3]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2-D tensor example::&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b = [[0,1,2,3],[4,5,6,7]]&#xA;logit_b = tf.one_hot(indices = b, depth = 8)&#xA;out_b = tf.argmax(logit_b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;out_b&lt;/strong&gt; will give me this:: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;array([[0, 0, 0, 0, 1, 0, 0, 0],&#xA;       [0, 0, 0, 0, 0, 1, 0, 0],&#xA;       [0, 0, 0, 0, 0, 0, 1, 0],&#xA;       [0, 0, 0, 0, 0, 0, 0, 1]])&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but what I want is [0,1,2,3,4,5,6,7] or [[0,1,2,3],[4,5,6,7]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;how can I do that??&lt;/p&gt;&#xA;" OwnerUserId="37775" LastActivityDate="2017-09-06T19:08:00.427" Title="how can I used tf.argmax in multi-dimensions tensor?" Tags="|neural-network|deep-learning|tensorflow|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22875" PostTypeId="2" ParentId="22874" CreationDate="2017-09-06T19:08:00.427" Score="3" Body="&lt;p&gt;Your problem is that you are not specifying the axis that you want to convert your tensor into a one hot tensor vector with so it's defaulting to looking at all elements at once, making logic_b of shape (2, 4, 8) when really what you want it to be is of shape (8, 2, 4). See below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;session = tf.InteractiveSession()&#xA;a = tf.constant([0,1,2,3],[4,5,6,7])&#xA;logit_b = tf.one_hot(a, depth = 8, axis = 0)&#xA;out_b = tf.argmax(logit_b, axis = 0)&#xA;out_b.eval()&#xA;&#xA;#array([[0, 1, 2, 3],&#xA;#       [4, 5, 6, 7]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-09-06T19:08:00.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22878" PostTypeId="2" ParentId="19541" CreationDate="2017-09-07T05:26:19.027" Score="1" Body="&lt;p&gt;It is a basic classification model. Our focus is to train the model using training dataset and evaluate its performance with the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, assume you have 10 numbers as input and a binary output.&#xA;Consider random 8 entries as your training data and train it with any classification algorithm using R/Python. Then test it on the remaining entries which acts as a test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since it is associated with classification we need to consider performance metrics like accuracy, precision and recall based on the labeled feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on classification try &lt;a href=&quot;https://www-users.cs.umn.edu/~kumar/dmbook/ch4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27586" LastEditorUserId="29575" LastEditDate="2017-09-07T06:00:35.143" LastActivityDate="2017-09-07T06:00:35.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22881" PostTypeId="1" CreationDate="2017-09-07T13:54:57.797" Score="2" ViewCount="3786" Body="&lt;p&gt;How can one automate the extraction of a &lt;strong&gt;relevant title&lt;/strong&gt; from a given document (docx, pdf, etc..)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some thoughts:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Intuitively likely to be the first line in the text&lt;/li&gt;&#xA;&lt;li&gt;Cannot be something like a date, &quot;all rights reserved&quot;, &quot;Harvard University - Faculty of Law&quot; etc.&lt;/li&gt;&#xA;&lt;li&gt;Might be in the footer&lt;/li&gt;&#xA;&lt;li&gt;The human brain solves this immediately (by filtering out common names and numbers)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm using textract in python to extract the whole text, which limits the solution to be based on text only. Let me know if you are familiar with any other packages.&lt;/p&gt;&#xA;" OwnerUserId="38377" LastEditorUserId="35809" LastEditDate="2017-09-07T21:40:21.677" LastActivityDate="2018-11-17T14:01:55.353" Title="How to extract titles from documents?" Tags="|data-mining|nlp|text-mining|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22884" PostTypeId="1" CreationDate="2017-09-07T19:27:23.937" Score="-1" ViewCount="111" Body="&lt;p&gt;I am about to start a final year at my university and I want to do a Final Year Project based on Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to learn basics of Machine Learning by utilising sci-kit.&#xA;My project may involve predicting prices for graphics cards depending either on sales of games or prices of bitcoin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read this &lt;a href=&quot;https://webmasters.stackexchange.com/questions/73908/how-illegal-is-it-to-get-data-from-a-100-accessible-but-not-exposed-api&quot;&gt;post&lt;/a&gt; and I understand that's generally illegal to get/scrap data from APIs without permission, but is it possible to get it and use it for 'fair-use' and 'non-profit' for my project?&#xA;Thank you&lt;/p&gt;&#xA;" OwnerUserId="38960" LastActivityDate="2017-09-08T08:08:51.633" Title="Is it ok to use a dataset from public API for Final Year Project?" Tags="|machine-learning|scraping|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22888" PostTypeId="1" CreationDate="2017-09-08T07:39:48.983" Score="0" ViewCount="635" Body="&lt;p&gt;I just finished PhD and initially wished to work on data science and deep learning. However, after some rounds of interviews, I have been offered a job of web analytics and business intelligence at a medium size company. Is there any similarity with data science, and is there a future in it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of some precarious situation, I have to accept this job, but should I keep looking for another job meanwhile, or will the experience be helpful to rise up the ladder or join a big brand? &lt;/p&gt;&#xA;" OwnerUserId="37045" LastEditorDisplayName="user10640" LastEditDate="2018-03-16T05:08:04.057" LastActivityDate="2018-03-16T05:08:04.057" Title="Is web analytics similar to data science?" Tags="|machine-learning|deep-learning|career|market-basket-analysis|consumerweb|" AnswerCount="1" CommentCount="3" ClosedDate="2018-03-18T19:37:16.853" ContentLicense="CC BY-SA 3.0" />
  <row Id="22889" PostTypeId="2" ParentId="22888" CreationDate="2017-09-08T08:02:04.597" Score="1" Body="&lt;p&gt;We can see Web Analytics like an application of Data Science concepts. Almost a field of it. Usually it's very close to Business Intelligence for marketing aims and the main goal is to analyse users information.&lt;br&gt;&#xA;So it's about marketing.&lt;/p&gt;&#xA;" OwnerUserId="35809" LastEditorUserId="23305" LastEditDate="2017-09-08T15:29:25.570" LastActivityDate="2017-09-08T15:29:25.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22890" PostTypeId="2" ParentId="22884" CreationDate="2017-09-08T08:08:51.633" Score="0" Body="&lt;p&gt;I have done same thing one years ago, and I found the answer: &quot;you can do it by specifying sources and writing that you assume all responsibility&quot;.&lt;br&gt;&#xA;You have also to write that you wont use data in order to make money.&lt;/p&gt;&#xA;" OwnerUserId="35809" LastActivityDate="2017-09-08T08:08:51.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22891" PostTypeId="2" ParentId="18890" CreationDate="2017-09-08T08:12:48.370" Score="3" Body="&lt;p&gt;Actually, I'd propose to tackle it from a different angle - assuming you want to predict product categories, and those are usually few, you can see it as a regular classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every product group is a category, which you will try to predict. So you can use simple models, which are easy to train (logistic regression, decision trees etc.), easy to build, debug, explain and test. And you can experiment with various features - which ones are the most helpful when predicting product category (this is crucial, since many people will probably ask you why the model predict this category for this person).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you master this, then the regular approach to this is any type of recommender system (collaborative filtering, matrix factorization etc.). Plus you can use your simple approach as a benchmark.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-09-08T08:12:48.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22893" PostTypeId="2" ParentId="15075" CreationDate="2017-09-08T09:04:33.860" Score="2" Body="&lt;p&gt;TSFRESH is already supporting Time Series Forecast.&lt;/p&gt;&#xA;&lt;p&gt;See details and example &lt;a href=&quot;http://tsfresh.readthedocs.io/en/latest/text/forecasting.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://github.com/blue-yonder/tsfresh/blob/master/notebooks/timeseries_forecasting_google_stock.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14684" LastEditorUserId="98307" LastEditDate="2020-08-04T10:54:19.500" LastActivityDate="2020-08-04T10:54:19.500" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="22896" PostTypeId="2" ParentId="20283" CreationDate="2017-09-08T09:41:55.180" Score="0" Body="&lt;p&gt;You can do that, and you actually do that correctly. You simply misinterpret the results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the target variable, you have &lt;code&gt;54289&lt;/code&gt; instances encoded with &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;2430&lt;/code&gt; encoded with &lt;code&gt;1&lt;/code&gt;. The classifier does not find a proper separator and simply classifies everything to class &lt;code&gt;0&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obtained score is not misleading. The score is obtained as a fraction of the correct prediction count divided by the total number of predictions. Since the only correctly predicted instance count is the count of the instances labeled with $0$, the formula is simply:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\#zeroes}{\#zeroes + \#ones} = \frac{54289}{54289+2430} = 0.9571572136321163$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is clearly shown also in the confusion matrix which you plotted. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand what you might probably do is to put some more weight on penaly errors. Thus you probably have to play with 'C' parameter from the svm learner, probably by increasing the value of 'C' to much bigger values. It is pretty hard to find a proper value for this parameter, a rule of thumb would be to use first values of the form $10^k$, where $k$ is an integer in $[-7,7]$ and refine around good values.&lt;/p&gt;&#xA;" OwnerUserId="108" LastActivityDate="2017-09-08T09:41:55.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22897" PostTypeId="1" AcceptedAnswerId="22898" CreationDate="2017-09-08T10:26:29.193" Score="-2" ViewCount="2123" Body="&lt;p&gt;I am investigating some machine learning algorithms (Perceptron and KNN) and I'm confused in the methodology section of my report. I am evaluating the performance of the 2 algorithms. Which methodology should I use?&lt;/p&gt;&#xA;" OwnerUserId="38983" LastEditorUserId="13748" LastEditDate="2017-09-08T13:26:54.020" LastActivityDate="2017-09-08T13:26:54.020" Title="Which methodologies can be used in machine learning research?" Tags="|methodology|" AnswerCount="2" CommentCount="0" ClosedDate="2017-09-08T13:26:28.327" ContentLicense="CC BY-SA 3.0" />
  <row Id="22898" PostTypeId="2" ParentId="22897" CreationDate="2017-09-08T12:19:11.810" Score="0" Body="&lt;p&gt;Typically, the methodology is to determine the prediction capability of the models after the learning on a validation set. The algorithm with the highest prediction accuracy wins.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-09-08T12:19:11.810" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22899" PostTypeId="1" CreationDate="2017-09-08T12:55:52.620" Score="1" ViewCount="142" Body="&lt;p&gt;I have a multivariable regression problem which basically forms a plane in 3D space. There may be other independent variables at play, but if I can approach the problem with 2 independent variables then I should be able to scale to include the rest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Broadly, it's a resource problem. I have a required resource (z), which depends on time (x) and number of registered users (y). Registered users is a function of time, but I need the time component because the resource requirement is different at different times of the day. I want to be able to predict the resource requirement at a given time of day if our users continue to registered at a known rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a chart of the data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/w5tcP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w5tcP.png&quot; alt=&quot;Required resource as a function of date and number of registered users.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is obviously not a &lt;em&gt;linear&lt;/em&gt; regression problem, so how do I approach this?&lt;/p&gt;&#xA;" OwnerUserId="33718" LastActivityDate="2017-09-08T18:57:54.260" Title="How to solve this regression problem?" Tags="|regression|linear-regression|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22900" PostTypeId="2" ParentId="22897" CreationDate="2017-09-08T13:18:22.200" Score="1" Body="&lt;p&gt;The methodology (or Methods) section should explain to the reader how you performed the experiment, and why you chose these methods. In your case, you should ask question such as&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How the data was obtained? Why is this data good for the experiment?&lt;/li&gt;&#xA;&lt;li&gt;What pre-processing was done? Why these pre-proc techniques?&lt;/li&gt;&#xA;&lt;li&gt;How were the algorithms compared (K-Fold CV, Test/Train split)? Why use these methods? What was the comparison metrics? Why these metrics and not others?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that this section should be written clearly and without much detail. For a more comprehensive writing guide, check the &lt;a href=&quot;http://libguides.usc.edu/writingguide/methodology&quot; rel=&quot;nofollow noreferrer&quot;&gt;USC Guide&lt;/a&gt; (it's for social sciences, but still apply).&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-09-08T13:18:22.200" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22901" PostTypeId="1" CreationDate="2017-09-08T13:59:47.930" Score="2" ViewCount="2557" Body="&lt;p&gt;I have a regression problem where I am predicting a continuous variable. Loss functions used most often in these cases (RMSE, MAE, etc.) don't treat over- or under- predictions differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am in a scenario where under-predicting would be a much worse outcome than over-predicting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of loss function would appropriately capture this?&lt;/p&gt;&#xA;" OwnerUserId="21690" LastActivityDate="2017-11-07T18:06:06.357" Title="Loss function that penalizes under-predictions more than over-predictions?" Tags="|machine-learning|predictive-modeling|regression|loss-function|" AnswerCount="1" CommentCount="3" ClosedDate="2017-11-10T16:29:30.990" ContentLicense="CC BY-SA 3.0" />
  <row Id="22902" PostTypeId="2" ParentId="22873" CreationDate="2017-09-08T15:03:24.770" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word Mover’s Distance (WMD)&lt;/a&gt; is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encoded the semantic meaning of words into dense vectors.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to &quot;travel&quot; to reach the embedded words of another document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DjJW1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DjJW1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Source: &lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;From Word Embeddings To Document Distances&quot; Paper&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-08T15:03:24.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22905" PostTypeId="2" ParentId="22901" CreationDate="2017-09-08T16:52:10.947" Score="1" Body="&lt;p&gt;Pick an asymmetric loss function. One option is &lt;a href=&quot;https://en.wikipedia.org/wiki/Quantile_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;quantile regression&lt;/a&gt; (linear but with different slopes for positive and negative errors).&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-08T16:52:10.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22906" PostTypeId="2" ParentId="22899" CreationDate="2017-09-08T16:55:35.200" Score="1" Body="&lt;p&gt;Reframe the problem as &lt;a href=&quot;https://en.wikipedia.org/wiki/Time_series&quot; rel=&quot;nofollow noreferrer&quot;&gt;time series&lt;/a&gt;. Predict &quot;z&quot; over time, given some level of &quot;y&quot;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-08T16:55:35.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22908" PostTypeId="2" ParentId="22899" CreationDate="2017-09-08T18:57:54.260" Score="2" Body="&lt;p&gt;If you want to predict, you need an underlying model of the behavior. Then you can perform least-squares fitting to determine the model parameters. Linear regression just assumes that you have a linear relationship and predicts from that relationship - there's nothing magic about it, except that the model is simple and often effective. Polynomial regression gives bad behavior when extrapolating, so I wouldn't advise that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Building a model function is usually a trial-and-error process (unless you have some prior knowledge about your system, which I guess you don't). I'd suggest looking at the locus of &lt;code&gt;(x, y)&lt;/code&gt; points where &lt;code&gt;z&lt;/code&gt; is maximized. It looks like they have an exponential decay relationship. A model like &lt;code&gt;y = exp(-x/x0)&lt;/code&gt; might fit the data rather closely. Once you've determined the value of x0 from the data, you can rebuild a model for &lt;code&gt;z&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39007" LastActivityDate="2017-09-08T18:57:54.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22913" PostTypeId="2" ParentId="22795" CreationDate="2017-09-09T09:21:09.533" Score="4" Body="&lt;p&gt;In fact, most clustering algorithms are even &lt;strong&gt;highly sensitive to scaling&lt;/strong&gt;. Rescaling the data can completely ruin the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bad scaling also appears to be a key reason why people fail with finding meaningful clusters. It is just very easy to do badly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By no means rely on automatic scaling. It must fit your task and data. Preprocessing is an art, and will require most of the work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-continuous variables are big issue. While you can &quot;hack&quot; data into binary encodings and then pretend they are suitable, the discreteness poses a major issue for the algorithms. For example, many points have the same distance. And the &lt;em&gt;mean&lt;/em&gt; of such a variable doesn't make a lot of semantic sense anymore. The squared deviation (as used by k-means) is even worse. Results may often be better if you ignore such variables when clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same goes for bad attributes, such as identifiers, sequence numbers, etc.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-09T09:21:09.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22915" PostTypeId="2" ParentId="22795" CreationDate="2017-09-09T11:31:21.387" Score="3" Body="&lt;p&gt;Scaling affects Clustering Results in a way that depends by the metric used (Euclidean Distance, Squared Euclidean Distance, Manhattan Distance, …) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general when you are mixing features which have different physical measurements units, you can think of a Linear Transformation (i.e. Offset + Scale Factor) to transform them into a common space &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also think to try learning this transformation, using some more complicated model than a linear one (if you think your problem requires it) &lt;/p&gt;&#xA;" OwnerUserId="33025" LastActivityDate="2017-09-09T11:31:21.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22916" PostTypeId="1" AcceptedAnswerId="22920" CreationDate="2017-09-09T12:37:02.750" Score="0" ViewCount="324" Body="&lt;p&gt;Parametric methods make explicit assumptions about the functional form of $f$, and supervised learning are pretty the same right? It's a way to build a statistical model for predicting, or estimating, an output based on one or more inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they are the same, why would they have to create different names. Is there a reason behind this?&lt;/p&gt;&#xA;" OwnerUserId="8794" LastEditorUserId="29575" LastEditDate="2017-09-10T10:10:58.673" LastActivityDate="2017-09-10T10:10:58.673" Title="Are parametric method and supervised learning exactly the same?" Tags="|machine-learning|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22917" PostTypeId="2" ParentId="22871" CreationDate="2017-09-09T13:41:33.743" Score="1" Body="&lt;p&gt;Based on the comment by Ricardo Cruz I tried switching the value of false from 0 to -1 and also normalization on and off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neither switching the value from 0 to - 1 or normalization of features had any impact on the values predicted by the SVM. In this case I was using a Gaussian kernel and Sequential Minimal Optimization for my SVM.&lt;/p&gt;&#xA;" OwnerUserId="38903" LastActivityDate="2017-09-09T13:41:33.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22920" PostTypeId="2" ParentId="22916" CreationDate="2017-09-09T17:37:45.290" Score="1" Body="&lt;p&gt;Parametric methods in simple terms follow a particular distribution. The most common example would be that of Normal Distribution, where 64 percent of the data is situated around +-1 standard deviation from the mean. The essence of this distribution is the arrangement of values with respect to their mean. Similarly, other methods such Poisson Distribution etc have their own unique modeling technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parametric Estimation might have laid the foundation to some of the most vital parts of Machine Learning but it is an absolute mistake to think supervised learning is the same thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Supervised Learning may include approaches to fit the aforementioned parametric models but this might not always be the case. More often, the data scatter is quite spread out. It might not just be fitting one parametric model but a hybrid of more than one. Supervised Learning also takes into account the error which most parametric models don't consider unless incorporated manually. You could say that supervised learning is an evolved and more robust version of parametric methods which is highly flexible. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-09-09T17:37:45.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22924" PostTypeId="2" ParentId="19370" CreationDate="2017-09-09T23:08:29.957" Score="8" Body="&lt;p&gt;Yes, there are models that do this. This &lt;a href=&quot;http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Oquab_Is_Object_Localization_2015_CVPR_paper.html&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt; points to one of the first papers I believe. The main idea is called weakly supervised object detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper essentially makes three modifications.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;They treat the typical hidden fully connected layer as a convolutional layer. This works because convolutional layers can be thought of as convolving the same fully connected network about the image.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;They add a global max pooling later at the end of this convolutional layer. This is the operator that will &quot;highlight&quot; the area of this final conv layer that has learned the pattern of objects it is trying to classify. Using a threshold on the weights of this global max will ensure a region is significant. Then, they use an algorithm to create a bounding box from this region.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;They suggest a new loss function that lends itself to an object existing or not. I think they assume a Bernoulli for each class which lends itself to multiple logistic regression instead of softmax.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Take a look because it's pretty sweet and has been cited by a lot of other new exciting papers.&lt;/p&gt;&#xA;" OwnerUserId="39052" LastEditorUserId="29575" LastEditDate="2017-09-10T10:11:09.243" LastActivityDate="2017-09-10T10:11:09.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22929" PostTypeId="1" CreationDate="2017-09-10T14:26:10.097" Score="2" ViewCount="2887" Body="&lt;p&gt;I have a dataset that I have created in Python with a category called &lt;code&gt;change_label&lt;/code&gt; that has values &quot;buy, hold, sell&quot; (or alternatively I could use -1,0,1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering, what is the best way to fit this data so that machine learning algorithms can be applied to it to make predictions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My options are label encoding, label binzarization, or one-hot encoding. Basically, I thought that simply using sci-kit learn's &lt;code&gt;LabelBinarizer()&lt;/code&gt; would be enough. However, I want to be sure. Any advice? &lt;/p&gt;&#xA;" OwnerUserId="37301" LastActivityDate="2017-09-10T14:26:10.097" Title="OneHotEncoder vs. LabelEncoder vs. LabelBinarizer" Tags="|python|scikit-learn|encoding|" AnswerCount="0" CommentCount="1" ClosedDate="2017-09-10T21:14:54.440" ContentLicense="CC BY-SA 3.0" />
  <row Id="22930" PostTypeId="2" ParentId="16416" CreationDate="2017-09-10T16:35:31.887" Score="-1" Body="&lt;p&gt;It is just an arbitrary choice. You have to choose one number and the order of magnitude matters, but not the exact value. Powers of two just feel natural.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't think so: Evaluate it on a given architecture. Lower the number of neurons from a power of two to a smaller number. If the time increases, you've proven me wrong.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-09-10T16:35:31.887" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22931" PostTypeId="1" CreationDate="2017-09-10T17:20:28.910" Score="1" ViewCount="266" Body="&lt;p&gt;I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate documents from the corpus. &#xA;Where can I find such data?&lt;/p&gt;&#xA;" OwnerUserId="2647" LastActivityDate="2018-06-07T06:10:17.983" Title="Where can I find datasets with labeled duplicate text documents?" Tags="|data|training|text|classifier|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22933" PostTypeId="2" ParentId="10188" CreationDate="2017-09-10T19:31:19.300" Score="8" Body="&lt;p&gt;The error measure in the loss function is a 'statistical distance'; in contrast to the popular and preliminary understanding of distance between two vectors in Euclidean space. With 'statistical distance' we are attempting to map the 'dis-similarity' between estimated model and optimal model to Euclidean space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no constricting rule regarding the formulation of this 'statistical distance', but if the choice is appropriate then a progressive reduction in this 'distance' during optimization translates to a progressively improving model estimation. Consequently, the choice of 'statistical distance' or error measure is related to the underlying data distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, there are several well defined distance/error measures for different classes of statistical distributions. It is advisable to select the error measure based on the distribution of the data in hand. It just so happens that the Gaussian distribution is ubiquitous, and consequently its associated distance measure, the L2-norm is the most popular error measure. However, this is not a rule and there exist real world data for which an 'efficient'* optimization implementation would adopt a different error measure than the L2-norm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the set of &lt;a href=&quot;https://en.wikipedia.org/wiki/Bregman_divergence&quot; rel=&quot;noreferrer&quot;&gt;Bregman divergences&lt;/a&gt;. The canonical representation of this divergence measure is the L2-norm (squared error). It also includes relative entropy (Kullback-Liebler divergence), generalized Euclidean distance (Mahalanobis metric), and Itakura-Saito function. You can read more about it in this paper on &lt;a href=&quot;https://web.archive.org/web/20100812221422/http://www.ee.washington.edu/research/guptalab/publications/FrigyikSrivastavaGupta.pdf&quot; rel=&quot;noreferrer&quot;&gt;Functional Bregman Divergence and Bayesian Estimation of Distributions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take-away: The L2-norm has an interesting set of properties which makes it a popular choice for error measure (other answers here have mentioned some of these, sufficient to the scope of this question), and the squared error will be the appropriate choice most of the time. Nevertheless, when the data distribution requires it, there are alternate error measures to choose from, and the choice depends in large part on the formulation of the optimization routine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;*The 'appropriate' error measure would make the loss function convex for the optimization, which is very helpful, as opposed to some other error measure where the loss function is non-convex and thereby notoriously difficult.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-09-10T19:31:19.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22934" PostTypeId="2" ParentId="22931" CreationDate="2017-09-10T20:58:59.830" Score="1" Body="&lt;p&gt;Instead of training a classifier, detecting duplicates may advantageously be done in a direct fashion: using a similarity metric like shingle signatures or a hashing/checksum function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(A text to representation function F should preserve formal/meta elements that you already know to be identifiers or partial-identifiers, and it should also incorporate a compact representation of the content -- like sparse excerpts, checksum, or any other hashing key or canonized form. The comparison function G should then compare these text representations provided by F and define a similarity score.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd still need an evaluation corpus, so looking for a corpus is still important. I don't know of any open one, but job-boards and academic databases contain lots of duplicates so you could start there.&lt;/p&gt;&#xA;" OwnerUserId="39071" LastActivityDate="2017-09-10T20:58:59.830" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22936" PostTypeId="2" ParentId="22690" CreationDate="2017-09-11T00:07:39.897" Score="6" Body="&lt;p&gt;One situation in which it's advantageous to stack LSTMs is when we want to learn hierarchical representation of our time-series data. In stacked LSTMs, each LSTM layer outputs a sequence of vectors which will be used as an input to a subsequent LSTM layer. This hierarchy of hidden layers enables more complex representation of our time-series data, capturing information at different scales. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, stacked LSTMs can be used to improve accuracy in time-series classification, such as activity prediction, in which heart-rate, step-count, GPS and other signals can be used to predict activity such as walking, running, biking, climbing stairs or resting. For an example of time-series classification with stacked LSTMs using EEG data have a look at the following &lt;a href=&quot;https://github.com/vsmolyakov/experiments_with_python/blob/master/chp01/keras_lstm_seqclass.ipynb&quot; rel=&quot;noreferrer&quot;&gt;ipython notebook&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18418" LastActivityDate="2017-09-11T00:07:39.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22939" PostTypeId="1" CreationDate="2017-09-11T04:26:18.217" Score="3" ViewCount="157" Body="&lt;p&gt;Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular PCA?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that (hopefully) we could have new features that would improve PCA in a way that wouldn't require retraining every time we add new features. If this isn't possible as explained here, how to achieve this in another way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We add new features regularly by adding new datasources, or add some new features through feature engineering. Downflow models rely on PCA of that feature space. By training models on PCA and not on features directly simplifies workflows, and also removes burden of retraining models if it was possible to have a PCA (or another feature extraction process) that maps original features to this &quot;stable&quot; subspace.&lt;/p&gt;&#xA;" OwnerUserId="5380" LastEditorUserId="5380" LastEditDate="2017-09-11T04:53:49.137" LastActivityDate="2017-09-17T11:23:55.040" Title="stable set PCA while adding features" Tags="|feature-selection|feature-extraction|feature-engineering|pca|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22941" PostTypeId="1" CreationDate="2017-09-11T07:42:50.650" Score="2" ViewCount="1800" Body="&lt;p&gt;I have a dataset with &lt;code&gt;N&lt;/code&gt; records and &lt;code&gt;D&lt;/code&gt; numerical attributes belonign to &lt;code&gt;C&lt;/code&gt; different classes. I use a Stacked Autoencoder for feature extraction for a &lt;em&gt;classification&lt;/em&gt; task:&#xA;it takes an input vector $x \in \mathbb{R}^d$, and maps it to a hidden representation $y \in \mathbb{R}^{d^{'}}$ ($ d^{'} &amp;lt; d $).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: if we consider outputs of the middlemost layer ( i.e. $y \in \mathbb{R}^{d^{'}}$) as our feature set, for each class of data ($ c_i \in C $),  how can we realize which subset of this feature set has the greatest impact on identifying the class $m_i$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example: for MNIST we have &lt;code&gt;N=60000&lt;/code&gt;, &lt;code&gt;D=784&lt;/code&gt;, and &lt;code&gt;C=10&lt;/code&gt;. An autoencoder with this architecture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d=784&#xA;inp = Input(shape=(d,))&#xA;x = Dense(d, activation='relu')(inp)&#xA;x = Dense(d//2, activation='relu')(x)&#xA;x = Dense(d//8, activation='relu')(x)&#xA;y = Dense(d//128, activation='relu')(x)&#xA;x = Dense(d//8, activation='relu')(y)&#xA;x = Dense(d//2, activation='relu')(decoded)&#xA;x = Dense(d, activation='sigmoid')(x)&#xA;model = Model(input_img, z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;produces a $y\in \mathbb{R}^{6}$. For example, here we see the outputs of layer &lt;code&gt;y&lt;/code&gt; for digit &lt;code&gt;5&lt;/code&gt; and &lt;code&gt;9&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  class  y_1    y_2     y_3     y_4     y_5    y_6&#xA;  9      1.09   9.59    1.58    8.47    1.14   7.25&#xA;  9      2.13   1.34    4.00    8.59    1.53   1.36&#xA;  5      7.19   7.52    4.58    5.04    1.09   5.35&#xA;  5      9.80   1.55    1.46    5.06    6.49   3.51&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we connect layer y to a &lt;code&gt;softmax&lt;/code&gt; dense layer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out = Dense(num_classes, activation='softmax')(y)&#xA;encoded = Model(input_img, out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the new model &lt;code&gt;encoded&lt;/code&gt; give us a good accuracy about &lt;code&gt;98%&lt;/code&gt;. So, if we consider new representation, &lt;code&gt;y&lt;/code&gt;, as an efficiently extracted feature vector, is it reasonable to assign a subset of this feature vector to each digit?&#xA;Or, what is the right way to relate the vector &lt;code&gt;y&lt;/code&gt; to the &lt;code&gt;C&lt;/code&gt; different classes? &lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-09-11T07:42:50.650" Title="Feature extraction using autoencoder and assigning sub-features to the classes" Tags="|deep-learning|feature-selection|feature-extraction|feature-engineering|autoencoder|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22943" PostTypeId="2" ParentId="13744" CreationDate="2017-09-11T08:47:20.053" Score="0" Body="&lt;p&gt;You are asking for more ideas here, not for concrete solutions, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, maybe try &lt;code&gt;RWeka&lt;/code&gt;'s implementation of kmeans, &lt;code&gt;SimpleKMeans()&lt;/code&gt;? it has hyperparameters allowing you to implicitly preprocess the data with a &lt;a href=&quot;https://en.wikipedia.org/wiki/Canopy_clustering_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Canopy algorithm&lt;/a&gt;, before running KMeans. Canopy (has nothing to do with Python) can  run in either batch or incremental mode. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weka also has a standalone implementation of the Canopy Clusterer built-in, which you can use via the Explorer GUI. However with RWeka you have to use the &lt;code&gt;RWeka::make_Weka_clusterer()&lt;/code&gt; mechanism of calling the appropriate Java class that implements Canopy. So in R, you don't have to wrap it in SimpleKMeans() but then use &lt;code&gt;RWeka::make_Weka_clusterer()&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Canopy &quot;is intended to speed up clustering operations on large data sets&quot;. As I understand the Wikipedia article, Canopy also removes some redundancy in the dataset, but in a different manner, &lt;em&gt;not&lt;/em&gt; by building weights as you propose. See also &lt;a href=&quot;https://stats.stackexchange.com/questions/13895/how-do-i-algorithmically-determine-values-of-t1-t2-for-canopy-clustering&quot;&gt;this answer&lt;/a&gt; from stats.SE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all your attributes have low cardinality (= few unique values), I'd say just take samples (your option #1). &lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2018-05-11T08:12:58.303" LastActivityDate="2018-05-11T08:12:58.303" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22944" PostTypeId="1" AcceptedAnswerId="22955" CreationDate="2017-09-11T09:21:38.743" Score="1" ViewCount="1403" Body="&lt;p&gt;I am calculating Hellinger distance for different vectors. I initially assumed that the value returned by it in in the range of 0 to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However for the following two vectors I received Hellinger score as &lt;code&gt;1.0488088481701514&lt;/code&gt;, which is &lt;code&gt;&amp;gt; 1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vector_1 = [0.0,0.5,0.7]&#xA;vector_2 = [1.0,0.0,0.0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I am curious to know the range of Hellinger distance values. Please explain me why that value exceeded 1.&lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2017-09-11T19:16:35.050" Title="Limits of Hellinger distance values" Tags="|machine-learning|python|data-mining|text-mining|distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22945" PostTypeId="1" AcceptedAnswerId="22946" CreationDate="2017-09-11T11:06:02.337" Score="0" ViewCount="339" Body="&lt;p&gt;I know this question is rather broad but hopefully on topic. Are there useful references on classifying encrypted images. For example classifying cat/no cat on encrypted images.&#xA;Is there efficient and accurate software for this task?&#xA;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT for clarification: What I mean is the following: We have an encryption method E. A classifier is trained on encrypted images with their labels. Then at run time an image is sent in encrypted form from the client to the server and classified. This is similar to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ML Confidential:&#xA;Machine Learning on Encrypted Data&#xA;&lt;a href=&quot;https://eprint.iacr.org/2012/323.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://eprint.iacr.org/2012/323.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20884" LastEditorUserId="20884" LastEditDate="2017-09-11T12:27:38.593" LastActivityDate="2017-09-11T17:05:50.393" Title="Classifying encrypted images" Tags="|image-classification|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="22946" PostTypeId="2" ParentId="22945" CreationDate="2017-09-11T12:08:33.127" Score="2" Body="&lt;p&gt;Very briefly, homomorphic encryption as used by the linked paper in the question works as follows (excuse ad-hoc notation). Define an encryption method $E(x,k)$ that takes some data $x$ and secret key $k$ that has the following properties:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is a decryption method $D( E(x,k), k ) = x$ (optionally the two $k$ here can be different)&lt;/li&gt;&#xA;&lt;li&gt;You can define operations that work on the encrypted data in a self-consistent manner, e.g. $\text{Add}_E(E(x,k),E(y,k)) = E(x+y,k)$ - note that the result is still encrypted, and you need to know $k$ in order to use method $D()$ and find out what $x+y$ is.&lt;/li&gt;&#xA;&lt;li&gt;You may also need to support operations like $\text{Add}_E(E(x,k),y) = E(x+y,k)$ e.g. where one of the values is normal numeric variable - this might be useful for initialising a model - or using an existing trained model - but the catch is that when training you cannot keep these values unencrypted, once they use updated due to training results, they will also be encrypted.&lt;/li&gt;&#xA;&lt;li&gt;Note that no operation can output a decrypted value without using the key, otherwise the whole scheme is insecure, it implies some backdoor to get decrypted data.&lt;/li&gt;&#xA;&lt;li&gt;For ML use, you need an encryption scheme E that supports multiple basic operations, e.g. $\text{Add}_E$, $\text{Subtract}_E$, $\text{Multiply}_E$, $\text{GreaterThan}_E$ etc. You need some minimal set that is enough to build a model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can then build variant models that instead of performing arithmetic to sum/multiply etc numeric values, work with the abstract operations on encrypted data. The models would otherwise look just like their unencrypted counterparts - e.g. you could quickly build a linear regression using only a few types of operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that you are replacing simple fast floating point add, multiply operations that are built into the processor with a more complex custom operation, this significantly affects performance compared to the unencrypted model. How much so depends on the encryption scheme, and how it is implemented. The paper shows in section 5, timings on unencrypted vs encrypted data and the difference is several orders of magnitude slower when encrypted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper was published in 2012, so it is possible that some improvements have been made here. However, on a deeper read of the subject I think that this is still at the stage of research proof-of-concept. I &lt;em&gt;might&lt;/em&gt; be wrong, and there are nice workable implementations available that could be used in something as complex as a CNN, but I have not found anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also linked numer.ai from comments. Initially that looked interesting because surely they would have solved efficiency problems. But in fact their main competitions are using data obfuscation techniques, not encryption - a homomorphic encryption paper is linked from their main site, but it does not seem to be what they are using. End users are writing very familiar-looking scripts that perform logistic regression etc using regular Python (no special operators imported)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you should take a second look at your wider problem and analyse your threat model. What precisely are you trying to protect against? If it is about keeping your own operations separate from client data, then you might be looking into company process and auditing solutions, rather than purely technical (you may still want to add technical solutions such as disk-level encryption to protect customer's data at rest in you data centre, in case someone gets into the centre physically and just grabs a disk containing all the cat images) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, look at &lt;a href=&quot;https://cloudsecurityalliance.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cloud Security Alliance&lt;/a&gt; which as well as having a certification scheme, has analysis showing how their recommendations map to other schemes such as UK's ISO27001. Note that this is typically significant capital investment including 1+ year project to implement, and usually undertaken by mid-sized or larger companies, when they want to work with government or large corporate's data. However, it is probably a more reliable and maybe still cheaper route than trying to research and build a technical solution involving models that process encrypted data at this time.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-11T17:05:50.393" LastActivityDate="2017-09-11T17:05:50.393" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22947" PostTypeId="2" ParentId="22881" CreationDate="2017-09-11T12:18:12.197" Score="1" Body="&lt;p&gt;The automate the extraction of a title is an example of &lt;a href=&quot;https://en.wikipedia.org/wiki/Automatic_summarization&quot; rel=&quot;nofollow noreferrer&quot;&gt;extractive summarization&lt;/a&gt;. It is typically solved with a combination of supervised and sem-supervised learning. More recently Deep Learning models with attention have been giving state-of-the-art results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll need labeled to train your own model or used established packages such as &lt;a href=&quot;https://pypi.python.org/pypi/sumy&quot; rel=&quot;nofollow noreferrer&quot;&gt;symy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-11T12:18:12.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22948" PostTypeId="2" ParentId="22536" CreationDate="2017-09-11T12:22:02.877" Score="3" Body="&lt;p&gt;&lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word Mover’s Distance (WMD)&lt;/a&gt; is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to &quot;travel&quot; to reach the embedded words of another document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DjJW1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DjJW1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Source: &lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;From Word Embeddings To Document Distances&quot; Paper&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim package&lt;/a&gt; has a &lt;a href=&quot;https://radimrehurek.com/gensim/models/keyedvectors.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;WMD implementation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-11T12:22:02.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22950" PostTypeId="2" ParentId="22945" CreationDate="2017-09-11T13:03:34.120" Score="1" Body="&lt;p&gt;If the encryption is done &lt;em&gt;right&lt;/em&gt; without the workaround described in the comments, this is not possible. Do a thought exercise:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an encrypted file. I do not know the contents. I pass it to a program that can determine that it is an image containing a cat or no cat. Why couldn't the classifier also determine the color of the cat? The location the picture was taken? The text on the sign behind the cat?&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-11T13:03:34.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22952" PostTypeId="1" AcceptedAnswerId="22953" CreationDate="2017-09-11T17:07:28.877" Score="5" ViewCount="3021" Body="&lt;p&gt;I have &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjBkd3C0Z3WAhVMsBQKHfZ1DVQQFggnMAA&amp;amp;url=https%3A%2F%2Fwww.coursera.org%2Flearn%2Fpython-data-analysis%2Flecture%2FLn156%2Fpandas-idioms&amp;amp;usg=AFQjCNHKDkwSCZabMMM42aghkOrMrQKhsw&quot; rel=&quot;noreferrer&quot;&gt;seen&lt;/a&gt; the term &lt;code&gt;pandorable&lt;/code&gt; but I cannot find any accurate definition for that. What does that mean?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-08T21:31:03.637" LastActivityDate="2018-07-16T14:21:47.123" Title="What is pandorable?" Tags="|python|pandas|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22953" PostTypeId="2" ParentId="22952" CreationDate="2017-09-11T17:23:45.417" Score="6" Body="&lt;p&gt;It's a neologism for &quot;fitting for pandas&quot;; compare with &quot;&lt;a href=&quot;https://stackoverflow.com/questions/25011078/what-does-pythonic-mean&quot;&gt;pythonic&lt;/a&gt;&quot;. As stated in the &lt;a href=&quot;https://www.coursera.org/learn/python-data-analysis/lecture/Ln156/pandas-idioms&quot; rel=&quot;noreferrer&quot;&gt;lecture transcript&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A sort of sub-language within Python, Pandas has its own set of idioms. We've alluded to some of these already, such as using vectorization whenever possible, and not using iterative loops if you don't need to. Several developers and users within the Panda's community have used the term &lt;strong&gt;pandorable&lt;/strong&gt; for these idioms.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-09-11T17:23:45.417" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22955" PostTypeId="2" ParentId="22944" CreationDate="2017-09-11T17:46:49.783" Score="1" Body="&lt;p&gt;It is bounded by unity, but your first vector does not encode a &lt;a href=&quot;https://en.wikipedia.org/wiki/Probability_mass_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;probability mass function&lt;/a&gt;, since 0.5 + 0.7 &gt; 1.0. If the 0.7 had been 0.5 or the 0.5 a 0.3, the distance would have been 1.0 since the distributions are maximally separated, having no overlap.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-09-11T19:16:35.050" LastActivityDate="2017-09-11T19:16:35.050" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="22957" PostTypeId="1" AcceptedAnswerId="22978" CreationDate="2017-09-11T18:03:47.500" Score="8" ViewCount="45746" Body="&lt;p&gt;I'm doing some exploratory data analysis on some data and I get these histograms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KnXhq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KnXhq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[&quot;abv&quot;].apply(np.log).hist()&#xA;df[&quot;ibu&quot;].apply(np.log).hist()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I get this new plot of the transformed histograms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UjbHZ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UjbHZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct that a log transform was ok to do in this case, and if so, what's the best way to interpret the results?&lt;/p&gt;&#xA;" OwnerUserId="33510" LastActivityDate="2017-09-12T17:12:31.240" Title="Am I doing a log transformation of data correctly?" Tags="|python|statistics|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22958" PostTypeId="2" ParentId="14999" CreationDate="2017-09-11T18:11:29.357" Score="0" Body="&lt;p&gt;I've used &lt;a href=&quot;http://www.borgelt.net/pyfim.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;fim&lt;/a&gt;'s fpgrowth function in the past and it worked well.  It's kind of a pain to install on Windows machines however.  It seems to be an academic website so I'm not sure if they're doing many updates to the code over time...&lt;/p&gt;&#xA;" OwnerUserId="39107" LastActivityDate="2017-09-11T18:11:29.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22960" PostTypeId="1" AcceptedAnswerId="22965" CreationDate="2017-09-11T19:21:42.240" Score="0" ViewCount="529" Body="&lt;p&gt;I am trying to build a recommender system based on a large and very sparse matrix. Dimensions of that matrix would approximately be 12000 x 37000, possibly even more rows up to 100000. However, this matrix is extremely sparse. With the 12000x37000 version, about 0.053% of the matrix is non-NA. I've tried SVD, but alas, to no avail. To ensure I have not caused any error during my proceedings:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I created a data.table with unique triplets of &quot;User&quot; - &quot;Item&quot; - &quot;Rating&quot;. I should mention &quot;Rating&quot; can stretch anywhere from 0 to about 150.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now, I applied dcast.data.table to the triplet table, corrected the problem with the first value becoming a column, converted to a matrix. Now, I had a matrix with users as rows, items as columns and the rating as cell content.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Split into Test and Validation set, replaced &quot;NA&quot; with 0, subtracted row means for every row, applied propack.svd from the &quot;svd&quot; package to that matrix, multiplied the three matrices delivered by propack and added the row means to it. (user-means).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, I compared the values from the validation set to the corresponding values in my prediction matrix.. and no surprise, the Root mean square error was horribly high, around 6-7.(Mean of non-NA values is around 4.5). I've tried multiple variants of normalisation too, but I just could not get the RMSE below 5.8, ever.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to build a viable item recommender system for this dataset? Possibly via arules or clustering?&lt;/p&gt;&#xA;" OwnerUserId="39109" LastEditorUserId="29575" LastEditDate="2017-09-11T22:32:57.903" LastActivityDate="2017-09-11T23:16:46.213" Title="R Recommender System for very sparse matrices required" Tags="|r|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22961" PostTypeId="2" ParentId="19681" CreationDate="2017-09-11T20:38:23.793" Score="3" Body="&lt;p&gt;This is because we have to exclude the target column from the second document (testing data-set).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example remove the column 'Status' from second file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In below example, in training dataset, 'Loan_Status' column is present which is the target field, whereas, in testing dataset, 'Loan_Status' column is not present, and will be computed by the 'Prediction' as shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dvOe0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dvOe0.png&quot; alt=&quot;training and testing data set&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Prediction&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nV5nl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nV5nl.png&quot; alt=&quot;prediction value&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39110" LastActivityDate="2017-09-11T20:38:23.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22962" PostTypeId="1" CreationDate="2017-09-11T21:45:10.313" Score="0" ViewCount="159" Body="&lt;p&gt;I am working on a problem with only ~ 30 data points, I need to use these 30 points to build a machine learning model (ridge regression, random forest ... etc.), then use it to predict on another thousands of data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some experiments, I believe the data set is too small. However, I am wondering is there a standard approach to prove that whether we have enough data points? And how to determine how many data points are needed? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2017-09-11T21:59:35.817" Title="Prove data points not enough? Estimate required number of data points?" Tags="|machine-learning|data-mining|predictive-modeling|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22964" PostTypeId="2" ParentId="22962" CreationDate="2017-09-11T21:59:35.817" Score="1" Body="&lt;p&gt;It depends on what you mean by &quot;needed&quot;. More training data = lower variance with the same model complexity. So when you have a small training dataset, your models will have to be very simple (like linear regression), so to have best generalization. And very simple models tend to badly represent reality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer is: you need so many points, such that your validation error is below desired threshold. 30 is probably much too low for any more complex machine learning model than linear regression with one, maybe two features.&lt;/p&gt;&#xA;" OwnerUserId="39112" LastActivityDate="2017-09-11T21:59:35.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22965" PostTypeId="2" ParentId="22960" CreationDate="2017-09-11T23:16:46.213" Score="0" Body="&lt;p&gt;I would look into the Soft Imputation method that has an &lt;a href=&quot;https://cran.r-project.org/web/packages/softImpute/softImpute.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation in R&lt;/a&gt;. It uses iterative soft-thresholding to compute missing values. Calculations are done with a matrix class called &quot;Incomplete&quot; to deal large sparse matrices and allows for quick calculation of scaling/centering rows and columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've had good success using this completing a 10,000 by 10,000 very sparse matrix so I'd imagine it should do fairly well with your dataset.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-09-11T23:16:46.213" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22968" PostTypeId="1" CreationDate="2017-09-12T07:52:18.630" Score="3" ViewCount="1651" Body="&lt;p&gt;Let's say we have feature A, B, C, D, E to represent one observation in an outlier detection model. We are using scikit-learn outlier detection in our case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AFAIK, if we normalize all the features, they are as equally important as others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if I want to make B very important, e.g. if A, C, D, E increase 20%, the observation might still be considered as normal case while B increases 10%, the observation needs to be marked as an outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way to increase or decrease features' weight in outlier detection model? &lt;/p&gt;&#xA;" OwnerUserId="38791" LastActivityDate="2017-11-12T09:38:55.237" Title="How to increase the weight when it comes to outlier detection" Tags="|scikit-learn|outlier|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22969" PostTypeId="2" ParentId="14999" CreationDate="2017-09-12T08:56:22.020" Score="1" Body="&lt;p&gt;Have you considered to write it by yourself? Because there is probably no up-to-date maintained library right now.&lt;/p&gt;&#xA;&lt;p&gt;Check &lt;a href=&quot;http://sequenceanalysis.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; out, its the basic - PrefixSpan and Closed/Maximal patterns are actually not that hard to implement.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastEditorUserId="98307" LastEditDate="2020-08-04T17:00:08.760" LastActivityDate="2020-08-04T17:00:08.760" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22970" PostTypeId="1" CreationDate="2017-09-12T08:59:14.717" Score="5" ViewCount="120" Body="&lt;p&gt;Anonymized dumps of the stack exchange data are &lt;a href=&quot;https://archive.org/details/stackexchange&quot; rel=&quot;nofollow noreferrer&quot;&gt;available here&lt;/a&gt;. Do you know projects or article that have been using these data (for social network analysis or information retrieval) ? My little research on the subject on Google Scholar &amp;amp; co seems to indicate that this dataset has been very seldom used.  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastEditorUserId="17208" LastEditDate="2017-09-12T09:40:38.833" LastActivityDate="2018-01-10T11:15:57.113" Title="In what data science applications has the stack exchange dump been used?" Tags="|dataset|social-network-analysis|information-retrieval|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="22971" PostTypeId="2" ParentId="22970" CreationDate="2017-09-12T09:19:04.377" Score="1" Body="&lt;p&gt;This is a Community Wiki answer, everyone is welcome to add references.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Kaggle ran a competition to predict whether a question would be closed on Stack Overflow: &lt;a href=&quot;https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-12T09:19:04.377" CommentCount="0" CommunityOwnedDate="2017-09-12T09:19:04.377" ContentLicense="CC BY-SA 3.0" />
  <row Id="22974" PostTypeId="1" CreationDate="2017-09-12T12:42:34.673" Score="1" ViewCount="1186" Body="&lt;p&gt;I have a &quot;User x Item&quot; matrix as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user item1 item2 item3&#xA;u1   2     0     3&#xA;u2   1     2     0&#xA;u3   4     3     1&#xA;u4   0     2     2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to computer the similarity between items based on users.&#xA;For example, to calculate similarity between items i1 and i2, I only choose users who have assigned values to both of these items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cosine similarity between two items (i1 and i2) is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cos(v1,v2) = (1*2 + 4*3)/sqrt[(1 + 4)*(16+9)] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is for item-to-item similarity, should I consider all users, or just common users who assigned values to the items?&lt;/p&gt;&#xA;" OwnerUserId="39136" LastEditorUserId="13748" LastEditDate="2017-09-12T15:13:57.670" LastActivityDate="2017-10-12T15:50:47.170" Title="Computing Item-to-Item Similarity Using Cosine" Tags="|recommender-system|similarity|cosine-distance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22976" PostTypeId="2" ParentId="22974" CreationDate="2017-09-12T13:38:00.840" Score="1" Body="&lt;p&gt;Filtering users will create bias in your training data. This may be good or bad, depending on your data and goals. The best way to find out, for your specific system, is to try and test both methods, optimizing for whatever metric you choose is best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Honestly, I think you should consider all users. Say you have a system for recommending movies. In this system, i1 represents a children's movie, that is rated by users u1 through u5, with u5 being a parent and u1, u2, u3 and u4 children. Similarly, i2 represents a terror movie, that is rated by users u5 through u9, all adults. In such a system, u5 being the only common ground between i1 and i2, if this user rated both positively and you only consider them, then you will have a high similarity between the items; but if you consider all users that rated either movie, you will have a very low similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or in other words, I think intuitively it's best to consider a missing rating as a false/negative rating. Excluding terms where both ratings are zero makes no difference in the cosine similarity anyway. But again, you should run and test it.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-09-12T13:38:00.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22977" PostTypeId="1" CreationDate="2017-09-12T14:11:21.807" Score="2" ViewCount="71" Body="&lt;p&gt;Context: Working on Temporal Alignment for sign-language features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm comparing three methods for time-warping (Uniform Time Warping - UTW, Procrustes Dynamical Time Warping - PDTW, Procrustes Canonical Time Warping - PCTW) on some and get results, which are hard to explain for myself. I expected the PDTW and PDTW to be at least somewhat comparable, but in the results are vastly different. This is mainly, because the PCTW gets stuck on the first entry: &lt;a href=&quot;https://i.stack.imgur.com/lFILV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lFILV.jpg&quot; alt=&quot;alignments for two exemplary samples, samples 3 and 5, each with the three respective time-warping alignments&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I looked into it in some more detail, occasionaly it also might get stuck on another index later on, but only rarely, and comparably a lot shorter. I can upload graphs of that as well if needed.&#xA;I use the MATLAB implementation by Zhou and De la Torre based on &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/26761734&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm really grateful for any idea how to interprete and possibly fix this.&lt;/p&gt;&#xA;" OwnerUserId="39139" LastEditorUserId="29575" LastEditDate="2017-09-12T15:16:52.753" LastActivityDate="2017-09-12T15:16:52.753" Title="Time Warping - Procrustes Canonical Time Warping Alignment stuck on first entry" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22978" PostTypeId="2" ParentId="22957" CreationDate="2017-09-12T17:12:31.240" Score="5" Body="&lt;p&gt;Yes, log transform seems a good solution for better interpretation. Overlap between these two datasets is really small, so, only by looking at the plot, you can say with high certainty, that they are significantly different from each-other.&lt;/p&gt;&#xA;" OwnerUserId="39112" LastActivityDate="2017-09-12T17:12:31.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22981" PostTypeId="1" AcceptedAnswerId="23229" CreationDate="2017-09-12T20:02:22.807" Score="2" ViewCount="907" Body="&lt;p&gt;Hello I'm looking for some advice making a animal Classifier from trap cams, I applied SIFT to detect points of interested but it turns out that most of the points were in the background and less in the animals. My intention is to recognize animals from a dataset, any ideas that you can bring me to achieve what I'm trying to do? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Images are something like this and the keypoints : &#xA;&lt;a href=&quot;https://i.stack.imgur.com/BIufe.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BIufe.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other options or suggestions? to change the way I'm going now. &lt;/p&gt;&#xA;" OwnerUserId="39158" LastActivityDate="2017-09-22T06:29:57.227" Title="Data Preprocessing, how separate background from image to detect animals?" Tags="|classification|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22982" PostTypeId="2" ParentId="10322" CreationDate="2017-09-12T21:32:40.590" Score="7" Body="&lt;p&gt;This changed in the latest version of Bokeh (I guess 0.12.7). &lt;a href=&quot;https://bokeh.pydata.org/en/latest/docs/user_guide/categorical.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is the new way of doing it.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from bokeh.io import show, output_file&#xA;from bokeh.models import ColumnDataSource, FactorRange&#xA;from bokeh.plotting import figure&#xA;&#xA;output_file(&amp;quot;bars.html&amp;quot;)&#xA;&#xA;fruits = ['Apples', 'Pears', 'Nectarines', 'Plums', 'Grapes', 'Strawberries']&#xA;years = ['2015', '2016', '2017']&#xA;&#xA;data = {'fruits' : fruits,&#xA;        '2015'   : [2, 1, 4, 3, 2, 4],&#xA;        '2016'   : [5, 3, 3, 2, 4, 6],&#xA;        '2017'   : [3, 2, 4, 4, 5, 3]}&#xA;&#xA;# this creates [ (&amp;quot;Apples&amp;quot;, &amp;quot;2015&amp;quot;), (&amp;quot;Apples&amp;quot;, &amp;quot;2016&amp;quot;), (&amp;quot;Apples&amp;quot;, &amp;quot;2017&amp;quot;), (&amp;quot;Pears&amp;quot;, &amp;quot;2015), ... ]&#xA;x = [ (fruit, year) for fruit in fruits for year in years ]&#xA;counts = sum(zip(data['2015'], data['2016'], data['2017']), ()) # like an hstack&#xA;&#xA;source = ColumnDataSource(data=dict(x=x, counts=counts))&#xA;&#xA;p = figure(x_range=FactorRange(*x), plot_height=250, title=&amp;quot;Fruit Counts by Year&amp;quot;,&#xA;           toolbar_location=None, tools=&amp;quot;&amp;quot;)&#xA;&#xA;p.vbar(x='x', top='counts', width=0.9, source=source)&#xA;&#xA;p.y_range.start = 0&#xA;p.x_range.range_padding = 0.1&#xA;p.xaxis.major_label_orientation = 1&#xA;p.xgrid.grid_line_color = None&#xA;&#xA;show(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0YbGr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0YbGr.png&quot; alt=&quot;bokeh bar chart categorical&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39160" LastEditorUserId="98307" LastEditDate="2020-08-03T16:42:03.233" LastActivityDate="2020-08-03T16:42:03.233" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="22984" PostTypeId="1" CreationDate="2017-09-12T22:44:47.937" Score="1" ViewCount="483" Body="&lt;p&gt;Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like to classify the text manually. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider the example provided in &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&lt;/a&gt; so that it's reproducible. In the following example, a classification model is trained and is fitted for test documents. One of the test document is - 'what the heck is this?'. I understand that model is returning the class which has highest probability. However, when the model is unsure, I would like to label the text as 'unable to classify'&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.naive_bayes import MultinomialNB&#xA;clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)&#xA;&#xA;docs_new = ['God is love', 'OpenGL on the GPU is fast', 'what the heck is this?']&#xA;X_new_counts = count_vect.transform(docs_new)&#xA;X_new_tfidf = tfidf_transformer.transform(X_new_counts)&#xA;&#xA;predicted = clf.predict(X_new_tfidf)&#xA;&#xA;for doc, category in zip(docs_new, predicted):&#xA;    print('%r =&amp;gt; %s' % (doc, twenty_train.target_names[category]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Output&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'God is love' =&amp;gt; soc.religion.christian&#xA;'OpenGL on the GPU is fast' =&amp;gt; comp.graphics&#xA;'what the heck is this?' =&amp;gt; soc.religion.christian&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Predicting the probabilities&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here are the probabilities of prediction. Documents 1 and 2 have somewhat clear winner. However, the third document doesn't have one. I have about 100 classes and I would be hesitant to set a manual threshold.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.predict_proba(X_new_tfidf)&#xA;array([[ 0.16297502,  0.03828016,  0.03737814,  0.76136668],&#xA;       [ 0.16387956,  0.36874738,  0.2364763 ,  0.23089675],&#xA;       [ 0.28288106,  0.17035852,  0.2484853 ,  0.29827513]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30360" LastActivityDate="2018-10-16T04:01:28.977" Title="Eliminate low quality predictions in a classification task" Tags="|classification|scikit-learn|nlp|text-mining|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22985" PostTypeId="2" ParentId="22690" CreationDate="2017-09-13T04:53:22.667" Score="9" Body="&lt;p&gt;From &lt;a href=&quot;https://stats.stackexchange.com/q/163304/12359&quot;&gt;What are the advantages of stacking multiple LSTMs?&lt;/a&gt; (I'll only update the answer there):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;From {1}:&lt;/p&gt;&#xA;  &#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that&#xA;    deep RNNs work better than shallower ones on some tasks. In&#xA;    particular, Sutskever et al (2014) report that a 4-layers deep&#xA;    architecture was crucial in achieving good machine-translation&#xA;    performance in an encoder-decoder framework. Irsoy and Cardie (2014)&#xA;    also report improved results from moving from a one-layer BI-RNN to an&#xA;    architecture with several layers. Many other works report result using&#xA;    layered RNN architectures, but do not explicitly compare to 1-layer&#xA;    RNNs.&lt;/p&gt;&#xA;  &lt;/blockquote&gt;&#xA;  &#xA;  &lt;hr&gt;&#xA;  &#xA;  &lt;p&gt;References:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;{1} Goldberg, Yoav. &quot;A Primer on Neural Network Models for Natural Language Processing.&quot; J. Artif. Intell. Res.(JAIR) 57 (2016): 345-420.&#xA;  &lt;a href=&quot;https://scholar.google.com/scholar?cluster=3704132192758179278&amp;amp;hl=en&amp;amp;as_sdt=0,5&quot; rel=&quot;noreferrer&quot;&gt;https://scholar.google.com/scholar?cluster=3704132192758179278&amp;amp;hl=en&amp;amp;as_sdt=0,5&lt;/a&gt;&#xA;  ; &lt;a href=&quot;http://u.cs.biu.ac.il/~yogo/nnlp.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://u.cs.biu.ac.il/~yogo/nnlp.pdf&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-09-13T04:53:22.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22986" PostTypeId="2" ParentId="22968" CreationDate="2017-09-13T07:56:55.153" Score="1" Body="&lt;p&gt;Please look at my answer at &lt;a href=&quot;https://datascience.stackexchange.com/questions/20499/identfying-spikes-in-data/20523#20523&quot;&gt;Identfying spikes in data&lt;/a&gt;&#xA;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A general method could be to calculate a (moving) average and standard deviation.&#xA;Outliers are points that differ more than 3 times the standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, B will probably have a standard deviation of 3%, because 10% is considered an outlier. For A, C, D, E, the standard deviation may be 7%, because 20% is not considered an outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relative importance does not make something an outlier. A large deviation does.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-09-13T07:56:55.153" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="22988" PostTypeId="1" CreationDate="2017-09-13T08:35:04.633" Score="1" ViewCount="68" Body="&lt;p&gt;I am new to anomaly intrusion detection system. I was reading the &lt;a href=&quot;https://pdfs.semanticscholar.org/770b/ae9ed688a84cfd2705a4ae4a11f5c9701f03.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;service specific anomaly detection for network intrusion detection&lt;/a&gt; but I could not understand section &lt;strong&gt;&lt;em&gt;3.2.3 payload Distribution&lt;/em&gt;&lt;/strong&gt;, page 5. To better understand the author I tried to do the calculations by hand but I am lost. I have included a sample of my calculations.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How do we calculate the &lt;em&gt;payload distribution&lt;/em&gt; if we have many &lt;em&gt;character distribution&lt;/em&gt; (one &lt;em&gt;character distribution&lt;/em&gt; for each request in the training set)?    &lt;/li&gt;&#xA;&lt;li&gt;How do we calculate the segments for the &lt;em&gt;payload distribution&lt;/em&gt;?   &lt;/li&gt;&#xA;&lt;li&gt;Given a new request how do we calculate its anomaly score?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2YKHB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2YKHB.jpg&quot; alt=&quot;My calculations&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39179" LastEditorUserId="29575" LastEditDate="2017-09-14T04:17:10.340" LastActivityDate="2017-09-14T04:17:10.340" Title="Intrusion Detection System (IDS)" Tags="|machine-learning|dataset|anomaly-detection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22990" PostTypeId="1" CreationDate="2017-09-13T09:06:40.697" Score="2" ViewCount="116" Body="&lt;p&gt;I have some datapoints $x_i$ and errors $\sigma_i$ which are the absolute value of some datapoints $w_i$ with (supposed) gaussian errorbars (the $\sigma_i$). The $x_i$ are amplitudes so we did not care the sign.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to compute the average (and weighted by $1/\sigma_i^2$ average) of them. And I need the errorbars on this mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I compute it if I don't have the values $w_i$ but just $x_i$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean if this was just the error of gaussian distributions (i mean for the  $w_i$ ) I would use the  weighted mean for this formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \mu = \frac{\Sigma w_i \sigma_i^{-2}}{\Sigma \sigma_i^{-2}}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for the error $$ \sigma = \sqrt{\frac{1}{\Sigma \sigma_i^{-2}}}$$ (as in &lt;a href=&quot;https://ned.ipac.caltech.edu/level5/Leo/Stats4_5.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ned.ipac.caltech.edu/level5/Leo/Stats4_5.html&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about in this case with the $x_i$?&lt;/p&gt;&#xA;" OwnerUserId="39182" LastEditorUserId="29575" LastEditDate="2017-09-14T06:15:12.807" LastActivityDate="2017-09-14T06:15:12.807" Title="Errorbars on a weighted average (of a non gaussian distribution)" Tags="|statistics|descriptive-statistics|error-handling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22991" PostTypeId="1" CreationDate="2017-09-13T11:55:20.247" Score="5" ViewCount="4406" Body="&lt;p&gt;Is it real to use word's n-grams for Deep Neural Network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g., sentences list contains in X_train dataframe with &quot;sentences&quot; column.&#xA;I use tokenizer from Keras in the next manner:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tokenizer = Tokenizer(lower=True, split=' ')&#xA;tokenizer.fit_on_texts(X_train.sentences)&#xA;X_train_tokenized = tokenizer.texts_to_sequences(X_train.sentences)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And later I use padding:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train_sequence = sequence.pad_sequences(X_train_tokenized)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also I use simple LSTM network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Embedding(MAX_FEATURES, 128))&#xA;model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2,&#xA;               activation='tanh', return_sequences=True))&#xA;model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, activation='tanh'))&#xA;model.add(Dense(number_classes, activation='sigmoid'))&#xA;model.compile(loss='categorical_crossentropy', optimizer = 'rmsprop',&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, tokenizer execution.&#xA;In Keras docs: &lt;a href=&quot;https://keras.io/preprocessing/text/&quot; rel=&quot;noreferrer&quot;&gt;https://keras.io/preprocessing/text/&lt;/a&gt;&#xA;I see character processing only, but it is nt apprepriate for my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main question: Can I use n-grams for NLP tasks with deep learning (not necessary Sentiment Analysis, any abstract NLP task).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Indeed, in many tutorials or books I doesn't see any remainder n-grams for text processing, only embeddings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For clarification: I'd like to consider not just words, but combination of words - I'd like to try it for my task.&lt;/p&gt;&#xA;" OwnerUserId="988" LastActivityDate="2017-09-16T13:30:57.943" Title="N-grams in NLP deep learning" Tags="|neural-network|deep-learning|nlp|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22992" PostTypeId="2" ParentId="22731" CreationDate="2017-09-13T12:36:45.680" Score="2" Body="&lt;p&gt;As far as the paper &quot;Sample-Level deep convolutional neural networks for music auto-tagging using raw waveforms&quot;, I can give you some of my intuitions about the question since I and my colleague proceeded the experiments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, I suggest you to use spectrogram based approaches in your situations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two reasons I would like to point out,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, training raw waveform based architecture takes about 4 times longer than spectrogram based model when the sampling rate is ranging from 16kHz to 22kHz. In your case, sampling rate is even 22Mhz. I think it will take a lot more time than spectrogram based model with similar performances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, to obtain well trained raw waveform based model, we need more than 50 hours audio since the model has more parameters and deeper layers. In my opinion, the benefit of using a raw waveform-based model is not the performance improvement, but on the generative model. If we use well performing raw waveform based model, we would not need to reconstruct audio signal from spectrogram when the case is generative model. This is the main reason why we performed reported experiments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If computing power and memory improve with current trends, we expect that the raw waveform-based model will be the mainstream in the near future. But now I think the spectrogram-based model is more convenient, especially for industrial applications.&lt;/p&gt;&#xA;" OwnerUserId="39189" LastActivityDate="2017-09-13T12:36:45.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22994" PostTypeId="1" AcceptedAnswerId="22995" CreationDate="2017-09-13T12:44:58.100" Score="7" ViewCount="3795" Body="&lt;p&gt;I'm trying to follow a tutorial for Q-Table learning from &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0&quot; rel=&quot;noreferrer&quot;&gt;this source&lt;/a&gt;, and am having difficulty understanding a small piece of the code. Here's the entire block:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import gym&#xA;import numpy as np&#xA;&#xA;&#xA;env = gym.make('FrozenLake-v0')&#xA;&#xA;#Initialize table with all zeros&#xA;Q = np.zeros([env.observation_space.n,env.action_space.n])&#xA;# Set learning parameters&#xA;lr = .8&#xA;y = .95&#xA;num_episodes = 2000&#xA;#create lists to contain total rewards and steps per episode&#xA;#jList = []&#xA;rList = []&#xA;for i in range(num_episodes):&#xA;    #Reset environment and get first new observation&#xA;    s = env.reset()&#xA;    rAll = 0&#xA;    d = False&#xA;    j = 0&#xA;    #The Q-Table learning algorithm&#xA;    while j &amp;lt; 99:&#xA;        j+=1&#xA;        #Choose an action by greedily (with noise) picking from Q table&#xA;        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))&#xA;        #Get new state and reward from environment&#xA;        s1,r,d,_ = env.step(a)&#xA;        #Update Q-Table with new knowledge&#xA;        Q[s,a] = Q[s,a] + lr*(r + y*np.max(Q[s1,:]) - Q[s,a])&#xA;        rAll += r&#xA;        s = s1&#xA;        if d == True:&#xA;            break&#xA;    #jList.append(j)&#xA;    rList.append(rAll)&#xA;&#xA;&#xA;print &quot;Score over time: &quot; +  str(sum(rList)/num_episodes)&#xA;&#xA;print &quot;Final Q-Table Values&quot;&#xA;print Q&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code runs well and I'm able to print my results, but here is where I'm having difficulties:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing? Any help is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="38917" LastActivityDate="2017-09-13T13:32:41.550" Title="Simple Q-Table Learning: Understanding Example Code" Tags="|python|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22995" PostTypeId="2" ParentId="22994" CreationDate="2017-09-13T13:01:12.293" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The code looks like a relatively ad-hoc* adjustment to ensure early exploration, and an alternative to $\epsilon$-greedy action choice. The &lt;code&gt;1/(i+1)&lt;/code&gt; factor is &lt;em&gt;similar&lt;/em&gt; to decaying $\epsilon$, but not identical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\epsilon$-greedy with the same decay factor might look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = np.argmax(Q[s,:])&#xA;if epsilon/(1+math.sqrt(i)) &amp;gt; random.random():&#xA;    a = random.randrange(0, env.action_space.n)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;math.sqrt(i)&lt;/code&gt; is just a suggestion, but I feel that &lt;code&gt;epsilon/(1+i)&lt;/code&gt; is probably too aggressive and would cut off exploration too quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not something I have seen before when studying Q-Learning (e.g. in David Silver's lectures or Sutton &amp;amp; Barto's book). However, Q-Learning is not predicated on using &lt;em&gt;any&lt;/em&gt; specific action choice, it just needs enough exploration in the behaviour policy. For the given problem adding some noise to a greedy selection obviously works well enough. Technically for guaranteed convergence tabular Q-Learning needs infinite exploration over infinite time steps. The code as supplied &lt;em&gt;does indeed do that&lt;/em&gt; because the noise is unbound from the Normal distribution. So there is always some small finite chance of selecting an action with a relatively low action-value estimate and refining that estimate later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the fast decay (1/episode number) and initial scaling factor for the noise are both hyperparameters that need tuning to the problem. You might prefer something more standard from the literature such as $\epsilon$-greedy, Gibbs sampling or upper-confidence-bound action selection (the example is quite similar to UCB, in that it adds to the Q-values before taking the max). &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Perhaps the approach used in the example has a name (some variation of &quot;Noisy Action Selection&quot;) but I don't know it, and could not find it on a quick search.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-13T13:32:41.550" LastActivityDate="2017-09-13T13:32:41.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="22996" PostTypeId="2" ParentId="22731" CreationDate="2017-09-13T13:02:21.967" Score="0" Body="&lt;p&gt;I think it depends on the characteristics of your data sample, and what you need to detect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Raw data may be better if you need to find start and duration of some event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Spectral/frequency data may be better if you look for repeating patterns (heartbeat).&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-09-13T13:02:21.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="22998" PostTypeId="1" CreationDate="2017-09-13T13:55:15.433" Score="0" ViewCount="738" Body="&lt;p&gt;I am relatively new to using word2vec. I am interested in solving the topic-word intrusion introduced &lt;a href=&quot;https://www.umiacs.umd.edu/%7Ejbg/docs/nips2009-rtl.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; by using the vector spaces of words generated by word2vec and SVC.&lt;/p&gt;&#xA;&lt;p&gt;I have a corpus with a vocabulary of 8000 words, the vocabulary is perfectly contained in Google's word2vec trained model. I was wondering which model would provide a better representation of the words, the pre-trained model on 3M words or a model trained only on the 8000 words appearing in my corpus?&lt;/p&gt;&#xA;" OwnerUserId="39194" LastEditorUserId="83275" LastEditDate="2021-03-15T05:52:44.180" LastActivityDate="2021-03-15T05:52:44.180" Title="Using pre-trained word2vec vs re-training a word2vec model" Tags="|word2vec|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23002" PostTypeId="1" CreationDate="2017-09-13T15:38:39.180" Score="2" ViewCount="83" Body="&lt;p&gt;I am building a modified RNN (specifically a GRU) to classify sequences. These sequences are of variable length, and contain &quot;states&quot;. Each point in the sequence is categorical and they look like [A,B,C] or [A,A,C,B], etc. Classification will be 1 or 0 depending on wether or not the user in the sequence performed an action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I face an issue where some of these sequences appear in both classes. In addition, one class occurs less than 10% of the time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In training on a downsampled dataset, the RNN can usually reach 75+% accuracy, with similar Specificity and Sensitivities. In Testing, the sensitivity falls dramatically to under 20%, with the specificity rising to 95%+.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is less than desirable, and I believe the issue may stem from the sequences that appear in both classes. A sequence that is positive most of the time in training will usually be negative most of the time in testing (as the testing set is unbalanced). I assume that the net learns the distributions more than the sequences, which explains the stark contrast in results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To people with experience working on these kind of imbalances and dataset issues, is my assumption correct? What advice do you have to handle data where the same sequence can appear in both classes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As next steps, I am thinking of weighting the minority class, and training on a non-downsampled dataset to preserve the distribution of sequences in each class. I am also considering a &quot;representative&quot; training and testing set, where if sequence [A,B,A] appears in a 6:1 negative to positive ratio, I will downsample to the extent that I can preserve the majority. Hopefully that will lead to less imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also the option to add more &quot;depth&quot; to the training data. Each point in the sequence has multiple features (usually hierarchical) that add more specifics about each state, this may decrease the number of sequences that appear in both classes. However, I am hesitant to make things more complex if there is a chance the network isn't learning properly. &lt;/p&gt;&#xA;" OwnerUserId="39196" LastActivityDate="2017-09-13T15:38:39.180" Title="Classifying Sequences Where Some Sequences in Both Classes" Tags="|classification|class-imbalance|sequence|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23003" PostTypeId="1" AcceptedAnswerId="23007" CreationDate="2017-09-13T16:08:13.013" Score="2" ViewCount="2436" Body="&lt;p&gt;So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format.&#xA;The code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from PIL import Image&#xA;from keras.preprocessing import image&#xA;&#xA;path='data/dogscats/sample/train/dogs/dog.1402.jpg'&#xA;&#xA;imgkeras=image.load_img(path)&#xA;imgkeras=image.img_to_array(imgkeras)&#xA;&#xA;plt.imshow(imgkeras)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of the following code is&#xA;&lt;a href=&quot;https://i.stack.imgur.com/KtT2b.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KtT2b.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where as the original image is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GwYL1.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GwYL1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain why is this happening? , also when the image is loaded through opencv and fed into vgg16 the predicted label is more accurate for this particular image than when it is loaded through keras as above,is the improper color format affecting that?&lt;/p&gt;&#xA;" OwnerUserId="37866" LastActivityDate="2017-09-15T08:15:25.660" Title="Keras loading images in incorrect format" Tags="|keras|image-classification|preprocessing|computer-vision|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23004" PostTypeId="2" ParentId="23003" CreationDate="2017-09-13T16:17:35.500" Score="0" Body="&lt;p&gt;Keras might not be the issue. The issue might be with the displaying done by matplotlib. The &lt;a href=&quot;https://matplotlib.org/users/image_tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;matplotlib image tutorial&lt;/a&gt; covers different ways of displaying images.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-13T16:17:35.500" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23006" PostTypeId="1" CreationDate="2017-09-13T18:26:59.513" Score="1" ViewCount="142" Body="&lt;p&gt;I've been experimenting with RNNs for predictive typing at both the letter and word level in TensorFlow and Keras, with great success so far. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases we know about the context in which a user is typing a message. For example the time of day, the frequency at which messages are being exchanged, or the gender of the recipient. Giving the RNN this information could produce better predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the example RNNs I've been working from have only passed the relevant series - times series, text, musical notes, etc. Is it reasonable to simply pass time, gender and frequency data into the RNN mixed with the text representations? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The orignal training data would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I may admit at the outset th&#xA; may admit at the outset tha&#xA;may admit at the outset that&#xA;ay admit at the outset that &#xA;y admit at the outset that I&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could I just include contextual information with text?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;M 12:09PM 27s I may admit at the outset th&#xA;M 12:09PM 27s  may admit at the outset tha&#xA;M 12:09PM 28s may admit at the outset that&#xA;M 12:10PM 29s ay admit at the outset that &#xA;M 12:10PM 30s y admit at the outset that I&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39206" LastActivityDate="2017-09-13T18:26:59.513" Title="Passing contextual information to an RNN" Tags="|neural-network|rnn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23007" PostTypeId="2" ParentId="23003" CreationDate="2017-09-13T19:20:02.993" Score="3" Body="&lt;p&gt;This is caused due to the img_to_array method which converts the image to a float32 array.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.asarray(img, dtype=K.floatx())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Matplotlib interprets NxMx3 uint8 array as a standard image (0..255 components) in which case there is no preprocessing. Otherwise the pixels are multiplied by 255(without checking the range) and then cast into uint8, which I guess leads to this behaviour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this:&lt;a href=&quot;https://stackoverflow.com/questions/39925420/bizzare-matplotlib-behaviour-in-displaying-images-cast-as-floats&quot;&gt;https://stackoverflow.com/questions/39925420/bizzare-matplotlib-behaviour-in-displaying-images-cast-as-floats&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer the second part of your question, I guess the imagenet competitors used OpenCV to load images in BGR format to train vgg16 and hence the pretrained weights work well with images opened in BGR format. &lt;/p&gt;&#xA;" OwnerUserId="39208" LastEditorUserId="39208" LastEditDate="2017-09-15T08:15:25.660" LastActivityDate="2017-09-15T08:15:25.660" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23008" PostTypeId="1" CreationDate="2017-09-13T20:08:11.580" Score="3" ViewCount="684" Body="&lt;p&gt;I am trying to determine the apt algorithm for a ranking problem that I am working on. I have social media metrics - engagement, sentiment, audience size etc for several brands and am looking for a ranking / classification algorithm to rank them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if I have a dependent variable or label class for classical classification algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is aggregated by brands and algorithms needs to rank the brands based on the metrics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas would be much appreciated. &lt;/p&gt;&#xA;" OwnerUserId="12533" LastActivityDate="2022-04-11T00:03:55.357" Title="Ranking algorithm based on a handful of features" Tags="|machine-learning|classification|ranking|scoring|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="23012" PostTypeId="2" ParentId="19748" CreationDate="2017-09-14T01:37:27.890" Score="0" Body="&lt;p&gt;You don't provide a whole lot of detail about the different events triggered but I would imagine that the &lt;a href=&quot;https://cran.r-project.org/web/packages/arules/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;arules package&lt;/a&gt; in R would be great at identifying some patterns to examine.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the apriori algorithm in this way in the domain of marketing is often called a market basket analysis.  It can be quite powerful to highlight relationships and can often give you a start of point for more discoveries.&lt;/p&gt;&#xA;" OwnerUserId="39218" LastEditorUserId="29575" LastEditDate="2017-09-14T06:15:07.147" LastActivityDate="2017-09-14T06:15:07.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23014" PostTypeId="1" CreationDate="2017-09-14T05:04:24.750" Score="2" ViewCount="99" Body="&lt;p&gt;I've got 500 images of paper receipts scanned and OCR as one dataset. I also have a dataset of transactions from my credit card statement including amount and date. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What model is best to match the receipts to the transactions?&lt;/p&gt;&#xA;" OwnerUserId="39229" LastEditorUserId="381" LastEditDate="2017-09-14T06:04:58.477" LastActivityDate="2017-09-14T06:18:45.340" Title="What model to use for matching two datasets" Tags="|information-retrieval|ocr|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23015" PostTypeId="2" ParentId="23014" CreationDate="2017-09-14T06:18:45.340" Score="1" Body="&lt;p&gt;Let the credit card statement be the ground truth, and the receipts be the noisy inputs. For a given line item, find the receipt with the smallest distance. If the distance is small enough, declare a match. This is a threshold you will have to experimentally determine. You can let the distance be the sum of the distances for the amount and date. A heuristic for these individual distances is simply the edit distance. A more sophisticated approach would be to model the OCR error using labeled data to determine the most likely input; print something with known text in a similar typeface to the receipts so you can learn which characters are commonly confused with one another, and thereby estimate the most likely input sequence. Going another step further, you could jointly model the density between price, store, and item, so you can recognize for example that a vegetable bought at your local grocery store probably does not cost \$100 but \$1.00&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your OCR software gives you confidence estimates over the characters and possible guesses, you can use that too.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-09-14T06:18:45.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23016" PostTypeId="1" CreationDate="2017-09-14T07:23:50.240" Score="1" ViewCount="250" Body="&lt;p&gt;I am learning about the &quot;pairwise approach&quot; for learning to rank. As far as I understood, the training output is a partial ranking function $r$ that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;given given some query $q$ and two document $d_i$ and $d_j$&lt;/li&gt;&#xA;&lt;li&gt;predicts whether $$r(d_i)&amp;gt;r(d_j)$$ or not&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, for a IR system to work, the ranking should be &lt;strong&gt;absolute&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The natural question next is how to construct the absolute rank using only the partial ranks output by $r$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But partial ranks does not guarantee absolute rank. For example &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r(x)&amp;gt;r(y), r(y)&amp;gt;r(z), r(z)&amp;gt;r(x)$$ gives a cycle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I am misunderstanding about how pairwise approach works for IR system. Can anyone correct me?&lt;/p&gt;&#xA;" OwnerUserId="33194" LastActivityDate="2017-09-14T16:32:27.377" Title="Learning to rank: construct absolute ranking using pair-wise ranking approach" Tags="|information-retrieval|ranking|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23017" PostTypeId="1" AcceptedAnswerId="23022" CreationDate="2017-09-14T07:31:11.840" Score="2" ViewCount="1047" Body="&lt;p&gt;After clustering, is there a way to explain the clusters? Or get the boundaries of the clusters? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For example&lt;/strong&gt;:&#xA;If we have a data set of people's spending habits with columns for their spend in different categories like groceries, clothing, transportation, rent etc. And we applied a clustering algorithm (like k-means or agglomerative clustering) on it.&#xA;Can we get descriptions of clusters, like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Cluster 1 contains people who spend &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;More than \$500 on groceries&lt;/li&gt;&#xA;&lt;li&gt;Less than \$200 on transportation&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Cluster 2 contains people who spend &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Less than \$100 on rent&lt;/li&gt;&#xA;&lt;li&gt;Less than \$300 on transportation&lt;/li&gt;&#xA;&lt;li&gt;More than \$50 on transportation&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Basically I need an explanation which is meaningful to a layman user.&lt;/p&gt;&#xA;" OwnerUserId="35889" LastActivityDate="2017-09-17T16:24:11.903" Title="Explaination or Description of clusters after clustering" Tags="|machine-learning|clustering|k-means|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23018" PostTypeId="1" CreationDate="2017-09-14T07:52:56.543" Score="1" ViewCount="55" Body="&lt;p&gt;I have a signed bipartite graph in which the &lt;strong&gt;nodes&lt;/strong&gt; are (1) students and (2) topics. An &lt;strong&gt;edge&lt;/strong&gt; is drawn between a student and topic node if the student mentions their opinion about the topic in a short answer (i.e., some students have an opinion about one topic but not another). The &lt;strong&gt;valence&lt;/strong&gt; of the edge indicates whether the opinion is positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how do I find out how many other students agree with a particular student? In terms of not just which topics they have an opinion on, but also what that opinion is (positive/negative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Based on a comment below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What exactly do you mean by agree? Should all existing opinions align or only those on a specific topic? What if one student has additional opinions about a topic? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All existing opinions (including topics and the valence) should align. If a student gives an same opinion about the same topics as another, but happens to talk about another topic as well, that would not count as full agreement. Perhaps there is a way to calculate partial agreement as well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What exactly is your problem? Defining the characteristics is straightforward: Just count. Do you perhaps struggle with an algorithmic implementation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Algorithmic implementation is perhaps what I am going for. Since I have 100 students, it would be difficult to hand-count the number of peers that agree with them. So if there is a way to calculate a value for each student, that would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="20400" LastEditorUserId="20400" LastEditDate="2017-09-14T08:28:18.503" LastActivityDate="2017-09-14T12:56:30.303" Title="How do I calculate for each person in the network how many people agree with their opinion?" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23019" PostTypeId="2" ParentId="20033" CreationDate="2017-09-14T08:16:43.050" Score="7" Body="&lt;p&gt;Here's a great example for Multivariate Regression using LSTM:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&quot; rel=&quot;noreferrer&quot;&gt;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most Examples you will find will be for time-series prediction using LSTM, like the one you listed for Stock Prices. &lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-09-14T08:16:43.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23020" PostTypeId="1" AcceptedAnswerId="23021" CreationDate="2017-09-14T08:27:22.043" Score="3" ViewCount="818" Body="&lt;p&gt;I have been building a neural network for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To select my best model. I have been using 10-Fold cross validation. and selected the network that gives the highest mean accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that I have selected the best model, I want to use all the data I have to train this model because the amount of data I have is limited (I will merge training, dev and test data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My issue is that, when training with all the data, I don't know when to stop training. Training loss is not an indicator for sure. Usually, I have a development set that I use to monitor training. When the training loss does not improve anymore, I stop training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how to supervise a model with only training data? In other words, how to tell when the network needs to stop?&lt;/p&gt;&#xA;" OwnerUserId="39236" LastEditorUserId="33245" LastEditDate="2019-12-13T16:53:18.180" LastActivityDate="2019-12-13T16:53:18.180" Title="What to do after selecting a model with cross-validation?" Tags="|machine-learning|neural-network|classification|cross-validation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="23021" PostTypeId="2" ParentId="23020" CreationDate="2017-09-14T08:51:11.553" Score="4" Body="&lt;p&gt;When 'overtraining' is not a problem (as in it will not diverge if you use more time), just use all your data and the empirically found optimal hyper parameters. In case of neural networks this is not the case (although in my experience, a lot of architectures converge to a specific test error, and take a long time to end up diverging again). I see a few options that you could try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Most obvious one is keeping a (small) validation set around to use as indicator for early stopping (don't think of this as throwing away data, you still use it to train your network better)&lt;/li&gt;&#xA;&lt;li&gt;Use same weight initalization as one of your folds and run for the same amount of epochs, same initialization should make convergence rate more similar than new random initialization&lt;/li&gt;&#xA;&lt;li&gt;Keep all the cross validation models and use them in an ensemble instead of retraining the full model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-14T08:51:11.553" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23022" PostTypeId="2" ParentId="23017" CreationDate="2017-09-14T08:57:28.293" Score="4" Body="&lt;p&gt;It depends on the clustering technique you use. Since you tagged this post with &lt;code&gt;k-means&lt;/code&gt; I will assume this is what you are using. Cluster centers should already be somewhat informative for laymans, but since you should be/are scaling this can lose some of it's interpretation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could do is assign class labels to each sample based on in what cluster they ended up in. Then you could fit a multi-class decision tree to your data and use the decision rules for interpretation, like 60% of cluster 1 has $x_1 &amp;lt; 0.9$.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-14T08:57:28.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23023" PostTypeId="2" ParentId="20415" CreationDate="2017-09-14T09:02:15.203" Score="3" Body="&lt;p&gt;The names should be given in ascending numerical order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saw this in the code's docs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class_names : list of strings, bool or None, optional (default=None)&#xA;    Names of each of the target classes in ascending numerical order.&#xA;    Only relevant for classification and not supported for multi-output.&#xA;    If ``True``, shows a symbolic representation of the class name.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39237" LastActivityDate="2017-09-14T09:02:15.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23024" PostTypeId="2" ParentId="12343" CreationDate="2017-09-14T09:04:46.740" Score="2" Body="&lt;p&gt;The solution was posted in &lt;a href=&quot;https://stackoverflow.com/a/43684587/9214357&quot;&gt;Stack Overflow&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;You need:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.to_csv('file.csv',encoding='utf-8-sig')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28915" LastEditorUserId="50146" LastEditDate="2020-08-12T07:42:46.813" LastActivityDate="2020-08-12T07:42:46.813" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23025" PostTypeId="2" ParentId="23018" CreationDate="2017-09-14T09:36:37.403" Score="2" Body="&lt;p&gt;As a fast answer, you can represent each student as a vector with $K$ elements (where $K$ is the number of topics) and values $\{+1, 0, -1\}$, denoting positive/non-existent/negative opinion about this topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, a simple measure of agreement between two students is the element-wise product between two student-vectors. That is the product will be:&#xA;$similarity = \sum_{i=1}^{K}st_1[i]*st_2[i]$, where $st_1,st_2$ are the student-vectors. Obviously, only the topics where both students have aligned opinions will boost the total [e.g. $1*1=1$ and $(-1)*(-1)=1]$, while misaligned opinions will decrease the sum. If any of the two students haven't expressed an opinion about a topic, then this topic won't matter in the sum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that sense, you can find the most like-minded students to a specific student, as the ones with the highest $similarity$. If what you really need is a number of agreeing students for each unique student, then a threshold on the $similarity$ score can be set. The value of the threshold can be decided empirically from your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is easily implemented and if you are comfortable with coding, I could post a sample script in python. One thing to consider though, is in what format is the bipartite graph (a .csv, a graph file of some kind etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: MINOR EXAMPLE. Fetch example .csv file used from &lt;a href=&quot;https://expirebox.com/download/dc66b2ff0111bf53e327635772902c6a.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;# Change location of file according to your needs&#xA;with open('students_example.csv', 'r') as f:&#xA;    df = pd.read_csv(f)&#xA;# Print for visualization&#xA;print(df.head())&#xA;print(&quot;~&quot;*25)&#xA;&#xA;# Delete column containing the student_id&#xA;del df['Student_ID']&#xA;# Parse the pandas DataFrame as matrix&#xA;student_vectors = df.as_matrix()&#xA;# The number of students at hand, let it be N.&#xA;N_students = student_vectors.shape[0]&#xA;# Initialize empty matrix of similarity between students&#xA;# Its size will be NxN (each student with each other)&#xA;similarity_scores = np.zeros((N_students, N_students))&#xA;# Iterate over each student vector and calculate the&#xA;# similarity with all students&#xA;for i, student in enumerate(student_vectors):&#xA;    # Reshaping and transposing to get the dot product between each student&#xA;    # And all the student vectors&#xA;    similarity_scores[i,:] = np.dot(student.reshape(1,-1), student_vectors.T)&#xA;# Fill the diagonal (that is the similarity of each student with him/herself)&#xA;# with low similarity scores so as not to confuse them with other possibly&#xA;# agreeing students&#xA;np.fill_diagonal(similarity_scores, -1000)&#xA;&#xA;# Random wanted student for example purposes&#xA;wanted_id = 3&#xA;# Print Students Opinion&#xA;print(&quot;Wanted Students Opinion:&quot;)&#xA;print(df.loc[wanted_id].to_string())&#xA;print(&quot;~&quot;*25)&#xA;print(&quot;Most similar:(Student ID = %d)&quot;% np.argsort(similarity_scores[wanted_id,:])[::-1][0])&#xA;print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][0]].to_string()&#xA;print(&quot;~&quot;*25)&#xA;print(&quot;Second most similar:(Student ID = %d)&quot;% np.argsort(similarity_scores[wanted_id,:])[::-1][1])&#xA;print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][1]].to_string()&#xA;print(&quot;~&quot;*25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you follow the example, the output for the wanted student (with $student_{ID}=3$) with opinions: {Trump -1, Net Neutrality   -1,Vaccination 1, Obamacare -1}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;will give you two other students with the same opinions and their ids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can modify the script to fit your needs accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: Sorry for the messy code, it was written rather hastily. Also, i tried it with Python 2.7.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastEditorUserId="8370" LastEditDate="2017-09-14T12:56:30.303" LastActivityDate="2017-09-14T12:56:30.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23027" PostTypeId="2" ParentId="15148" CreationDate="2017-09-14T12:22:50.380" Score="1" Body="&lt;p&gt;To answer your last question, histograms will give you an idea about where your outlier is by (as mentioned previously from other users) displaying bars which are disjointed from others by a reasonable distance. You may have to play around with the bin-widths to discover it, but it should be noticeable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another tool that may be useful to you is the box-plot. This visualization will not only give you information about the distribution of the data (though, unlike the histogram, frequency won't be specified), but will also specifically identify outliers. I would recommend using a combination of the scatter-plot, histogram, and box-plot to better familiarize yourself with your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm uncertain if MATLAB has this functionality (though I image it would), but in R when you plot your data you can choose to display the index of each observation above the data point on scatter-plots. Digging into MATLAB's plot function a bit more couldn't hurt.&lt;/p&gt;&#xA;" OwnerUserId="38917" LastActivityDate="2017-09-14T12:22:50.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23029" PostTypeId="1" AcceptedAnswerId="23076" CreationDate="2017-09-14T14:15:57.707" Score="8" ViewCount="6124" Body="&lt;p&gt;I had recently a great discussion about the advantages of RNN/LSTM in time series analysis in comparison to other Neural Networks like MLP or CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other side said, that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The NN just have to be deep enough to model the time connections&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;RNNs are just used, because they are faster&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;With RNN the older (e.g. t-20) timepoints are not that relevant in comparison to the newer timepoints (e.g. t-1), because the data from older timepoints have to go through all neurons until the weight is updated in this neuron.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is easier to understand the weight matrix of normal NN in comparison to RNN to understand partly the behavior of the NN  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My answer to that was:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Everyone suggests using the LSTMs for times series analysis&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It can consider the time better in comparison to a sliding window approach especially if the length of the sequence is quite long&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To 3. That happens just if you use vanilla RNN instead of LSTM, because of the gradient explosion&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So I know that I did not answer him completely (and maybe not correctly) and I am now really interested in the real reasons why LSTMS are suggested the most of the times. Maybe he is right? What do you think?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Can a normal NN model the time connections the same way like a&#xA;RNN/LSTM does when it is just deep enough?&lt;/li&gt;&#xA;&lt;li&gt;Does an RNN need more or less data in comparison to a NN to get the&lt;br&gt;&#xA;same/ better results?&lt;/li&gt;&#xA;&lt;li&gt;Are there time series where normal NN or RNN/LSTM perform better?&lt;/li&gt;&#xA;&lt;li&gt;Is it time data depended which Model will perform the best or are there some guidelines?&lt;/li&gt;&#xA;&lt;li&gt;Can the NN behavior be understood better than the RNN behavior?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37808" LastEditorUserId="29575" LastEditDate="2017-09-15T07:39:21.547" LastActivityDate="2018-03-22T22:26:02.613" Title="Why are RNN/LSTM preferred in time series analysis and not other NN?" Tags="|neural-network|predictive-modeling|time-series|rnn|model-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23030" PostTypeId="2" ParentId="23017" CreationDate="2017-09-14T14:20:01.243" Score="1" Body="&lt;p&gt;&lt;em&gt;Perfect answer by Jan van der Vegt&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To add on, if you don't have any option to add the label to each record, you have to depend on your domain knowledge to interpret the cluster outcomes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I say in layman terms, K-Means is applied only when you know how many clusters you need to get (using Scree plot may be) to get meaningful insights for your business question. So you must be having the numbers/clusters in your mind already. Its not simply you have to apply the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example- If you decide you are going to have two clusters (as per your question), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cluster 1&lt;/strong&gt; - May be this group represents people who are in Urban areas, more money to spend and the density of grocery/commodities shops are more that resulted in less transportation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cluster 2&lt;/strong&gt; - This group may be comprises people from country side due to which their transportation cost is more than the other cluster.&lt;/p&gt;&#xA;" OwnerUserId="28038" LastActivityDate="2017-09-14T14:20:01.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23035" PostTypeId="2" ParentId="20054" CreationDate="2017-09-14T15:36:56.500" Score="2" Body="&lt;p&gt;One tool that may be helpful is &lt;a href=&quot;https://blog.aidangomez.ca/2016/04/17/Backpropogating-an-LSTM-A-Numerical-Example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Aiden Gomez's&lt;/a&gt; blog post. Its strength lies primarily in the fact that he runs through a toy/numerical example, which when paired with the &lt;a href=&quot;http://www.felixgers.de/papers/phd.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;original paper/thesis&lt;/a&gt; , serves as a great foundational tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did take a look at the &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/&quot; rel=&quot;nofollow noreferrer&quot;&gt;site you mentioned&lt;/a&gt; in your other question, it's actually an excellent resource. I'll hop on over and try and clarify what I can for you when I get a chance. It looks like you've misunderstood/overlooked notation which can happen since there are so many components involved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may also be worth taking a look at &lt;a href=&quot;https://github.com/llSourcell/LSTM_Networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;some code&lt;/a&gt;. Siraj Raval has a great video on LSTMs and includes the code in the link I've included. No libraries. I wouldn't dive too deep, but it's a great way to see the inner workings of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as the CEC goes, there is a &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/34piyi/why_can_constant_error_carousels_cecs_prevent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;reddit post&lt;/a&gt;. If you're looking for a more rigorous handling of this topic you can either reference the original paper or often cited paper: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.421.8930&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;On the Difficulty of Training Recurrent Neural Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39253" LastEditorUserId="39253" LastEditDate="2017-09-14T16:19:30.947" LastActivityDate="2017-09-14T16:19:30.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23036" PostTypeId="1" CreationDate="2017-09-14T15:52:36.603" Score="-1" ViewCount="879" Body="&lt;p&gt;I have a TF-IDF representation of labeled documents (say 5 labels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to estimate the top-n words that are most likely to contribute to the classification, before applying any model (such as NMF/LDA)? I presume it could be derived from the TF-IDF score in each label. &lt;/p&gt;&#xA;" OwnerUserId="38377" LastActivityDate="2017-09-14T23:37:20.193" Title="Most meaningful words from TF-IDF" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23037" PostTypeId="1" CreationDate="2017-09-14T16:02:34.187" Score="0" ViewCount="182" Body="&lt;p&gt;I have a dataset with n records. The input features are f1, f2, f3 ... fm, and the dependent variable (output) is just the rank (not any type of scores).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of model should I build so that I can use the model to predict the rank of a new dataset? I suppose I could build a linear regression by assuming the dependent variable is just the rank (i.e. score = rank), but I am wondering if there is a better model or approach for this problem. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2017-09-14T16:24:57.597" Title="Model with rank as the dependent variable?" Tags="|predictive-modeling|ranking|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23038" PostTypeId="1" CreationDate="2017-09-14T16:14:41.940" Score="0" ViewCount="1226" Body="&lt;p&gt;I'm trying to predict the possible diagnosis given a consultation reason. I have ID's for all the data. So my data kind of looks like below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Reason            | Diagnosis&#xA;------------------------------&#xA;448, 124          | 9&#xA;551, 448, 122     | 9, 12&#xA;111, 110          | 32&#xA;143               | 43&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There can be up to 10 reasons and upto 5 diagnoses in my training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for in the algorithm or model is that it accepts 1 - 10 reason_ids as input and returns top 5 possibilities for diagnosis with % of probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm good at python so if there is any open source model or code I can look at it will be great.&lt;/p&gt;&#xA;" OwnerUserId="39258" LastEditorUserId="38887" LastEditDate="2020-05-08T10:41:01.460" LastActivityDate="2022-11-05T02:03:10.447" Title="Multiple variable as input and output" Tags="|machine-learning|classification|categorical-data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23039" PostTypeId="2" ParentId="23037" CreationDate="2017-09-14T16:24:57.597" Score="2" Body="&lt;p&gt;Leaving aside any discussion of algorithms and statistics, you cannot predict a rank of an object without context. Are you supposed to rank it relative to your current rows, or in relationship to other new rows? For ranking algorithms based on features, you would usually want to assign an arbitrary score and then sort by it and assign ranks accordingly.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-09-14T16:24:57.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23040" PostTypeId="2" ParentId="23016" CreationDate="2017-09-14T16:32:27.377" Score="2" Body="&lt;p&gt;Creating a total ranking from pairwise comparisons that don’t necessarily follow the axioms or rational preferences would certainly require some optimization, and you would need to compute a quadratic number of such comparisons in order to produce a ranking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is however one trick to do this easily: if the function that classifies whether one element is ranked above the other is linear in all the features, you can, intuitively, compare each element to a null element (i.e. just apply the decision function to the element alone), get a score assigned to them, and sort them by this score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, intuitively, your linear classifier would be something like this: element 1 is above element 2 if:&#xA;F(el1 – el2)&gt;0 [function has to be symmetrical, thus there can't be a non-zero threshold] --&gt; F(el1) – F(el2) &gt;0 (only when F is linear!!!)&#xA;Which implies that el1 &gt; el2 iff F(el1)&gt;F(el2)&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-09-14T16:32:27.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23041" PostTypeId="2" ParentId="18934" CreationDate="2017-09-14T16:35:11.183" Score="1" Body="&lt;p&gt;The classical solution is to try a one-class SVM applied only to the elements ranked above the others - you can read some papers about it looking by SVM rank or SVM ranking that will show you why that works.&lt;/p&gt;&#xA;" OwnerUserId="34153" LastActivityDate="2017-09-14T16:35:11.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23042" PostTypeId="2" ParentId="19614" CreationDate="2017-09-14T18:19:20.687" Score="3" Body="&lt;h2&gt;Part 1: Derivation&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It's important to note that $\delta z \neq z^{-1}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So on &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/11&quot; rel=&quot;nofollow noreferrer&quot;&gt;page 11&lt;/a&gt; when he says he plans to derive $\delta I^t, \delta h^{t-1}$, and $\delta w^t$ he's invoking the chain rule rather than inverse matrix multiplication.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's derive $\delta I^t$.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$\delta I^t = \frac{\partial E}{\partial I^t}$ by definition&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\delta I^t = \frac{\partial z^t}{\partial I^t} * \frac{\partial E}{\partial z^t} $ by the chain rule&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1.1 Recall that $z^t = W * I^t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1.2 Therefore $\frac{\partial z^t}{\partial I^t} = W^T$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore $\delta I^t = W^T * \delta z^t$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The author continues by noting that since $I^t = \begin{bmatrix}x^t\\h^{t-1}\end{bmatrix}$, that $\delta h^{t-1}$ can be retrieved from $\delta I^t$ where  $\delta h^{t-1}$ would be the dx1 terms after the first nx1 terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along the same vein to derive $\delta W^t$, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$\delta W^t = \frac{\partial E}{\partial W^t}$ by definition&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\delta W^t = \frac{\partial E}{\partial z^t} * \frac{\partial z^t}{\partial W^t} $ by the chain rule&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1.1 Recall that $z^t = W * I^t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1.2 Therefore $\frac{\partial z^t}{\partial W^t} = (I^t)^T$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.2.1 $\frac{\partial E}{\partial z^t} = \delta z^t$ by definition&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore $\delta W^t = \delta z^t * (I^t)^T$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;Part 2: Constant Error Carousels&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The following explanation draws heavily from the &lt;a href=&quot;http://www.cs.toronto.edu/~rgrosse/csc321/lec10.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;14th slide of this lecture&lt;/a&gt;, and from the &lt;a href=&quot;http://harinisuresh.com/2016/10/09/lstms/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM section of this blog&lt;/a&gt;, which I would highly recommend referencing. Consider this portion of my answer the TL;DR version of these two links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Constant Error Carousel (CEC) as you might well know is the magic of the LSTM in that it prevents vanishing gradients. It's denoted as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c_{t+1} = c_t *$ forget gate + new input $*$ input gate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of regular RNNs during backpropagation, the derivative of an activation function, such as a sigmoid, will be less than one. Therefore over time, the repeated multiplication of that value against the weights $f'(x) * W$ will lead to a vanishing gradient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of an LSTM, we only multiply the cell state by a forget gate, which acts as both the weights and the activation function for the cell state. As long as that forget gate equals one, the information from the previous cell state passes through unchanged. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in our case, the parameters we derived $h_t$ and $h_{t-1}$ are the hidden states which as part of the input vector $I^t$, will be filtered through the forget gate, and affect the cell state accordingly. The $\delta W$ we solved for is the parameter we use to update all our weights/gates. &lt;/p&gt;&#xA;" OwnerUserId="39253" LastActivityDate="2017-09-14T18:19:20.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23043" PostTypeId="1" CreationDate="2017-09-14T18:31:17.630" Score="3" ViewCount="380" Body="&lt;p&gt;Lets say I am creating a fully convolutional classifier for identifying different types of animals. Now lets assume that I know what part of the animal kingdom the animal belongs to beforehand (mammal, reptile, amphibian .. etc). How would I present this extra information to the fully convolutional network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas or research would be very helpful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit*&#xA;More specifically I am concerned with domain adaptation as shown &lt;a href=&quot;https://arxiv.org/pdf/1612.07828.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, In this example I am using a fully convolutional network without any pooling or striding, where the output is the same size as the input (i.e. an image transformed from one domain to another). In this respect I want to be able run the domain adaptation on images of any size. Adding extra channels to the input seems to make the most sense, however if I am one-hot encoding a bunch of categorical variables, this will consume a lot of memory.&lt;/p&gt;&#xA;" OwnerUserId="39261" LastEditorUserId="29169" LastEditDate="2021-02-15T22:23:50.650" LastActivityDate="2021-02-15T22:23:50.650" Title="How to Precondition A Fully Convolutional Neural Network" Tags="|deep-learning|classification|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23044" PostTypeId="2" ParentId="23043" CreationDate="2017-09-14T18:41:14.667" Score="1" Body="&lt;p&gt;I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Don't make it fully convolutional but add some dense layers where you concat this information&lt;/li&gt;&#xA;&lt;li&gt;If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network&lt;/li&gt;&#xA;&lt;li&gt;Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle&lt;/li&gt;&#xA;&lt;li&gt;This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution&lt;/li&gt;&#xA;&lt;li&gt;Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let me know how much it helps!&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-14T18:41:14.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23046" PostTypeId="1" CreationDate="2017-09-14T19:51:41.493" Score="1" ViewCount="1072" Body="&lt;p&gt;I am interested in changing some weights values in certain conv layers of a &lt;code&gt;tf-slim&lt;/code&gt; model. Say &lt;code&gt;Mobilenet_v1&lt;/code&gt;. Is there any straightforward way to do that while keeping the whole architecture and the data the same as before?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.: &#xA;Suppose we have got this numpy array of weights from the last level layer of trained MobileNet_v1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; conv=sess.run('MobilenetV1/Conv2d_13_depthwise/depthwise_weights_2:0')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives me the whole array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;&#xA;&#xA;[ -7.38371849e-01],&#xA;[  1.39969051e+00],&#xA;[  1.34190738e+00],&#xA;[ -5.66634417e-01],&#xA;[ -2.53899932e-01],&#xA;[  1.18222773e-01],&#xA;[  2.57152438e-01],&#xA;[  2.43642002e-01],&#xA;[ -5.11337161e-01],&#xA;[ -5.80285192e-01],&#xA;[  3.62841874e-01],&#xA;[ -4.72957313e-01],&#xA;[  7.34595060e-01],&#xA;[ -7.35578299e-01],&#xA;[  6.73987687e-01]]]], dtype=float32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for instance, I would like to search for all values within a specific range:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; numpy.count_nonzero((conv&amp;gt;=XX) &amp;amp; (conv&amp;lt;=YY))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then set those values to zero. That means that I overwrite the new variables values, so then I use the new pre-trained model for further training, or, fine-tuning, or inference. How can I do that? &lt;/p&gt;&#xA;" OwnerUserId="31681" LastEditorUserId="31681" LastEditDate="2017-09-15T16:39:20.923" LastActivityDate="2017-09-15T16:39:20.923" Title="How can we manually change some TF slim weights and use them to re-train, fine-tune, or predict" Tags="|tensorflow|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23049" PostTypeId="2" ParentId="23036" CreationDate="2017-09-14T23:37:20.193" Score="2" Body="&lt;p&gt;The TF-IDF is a measure of the &lt;strong&gt;discriminating ability of a term in a document&lt;/strong&gt;. The TF-IDF of a specific term increases if the term is more frequent in a specific document but decreases if it is frequent in the whole corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with TF-IDF is that while it is good in distinguishing the document from the corpus, it &lt;strong&gt;isn't good in distinguishing one label from another!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could try to compute is a variation of the TF-IDF, where in the nominator you would count the term frequency, not a specific document, but in the &lt;strong&gt;set of documents under the same label&lt;/strong&gt;. The denominator would stay the same. This way, instead of getting the top terms for each document you could get the top terms for each label. This metric could give you a rough estimation of what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that while the top term for each label would be the most important in classifying examples to the specific label, it isn't necessarily the best for classification in general. The best term for classification, theoretically, would be one that could divide your data in two equal parts.&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2017-09-14T23:37:20.193" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23050" PostTypeId="1" CreationDate="2017-09-15T04:12:10.390" Score="1" ViewCount="997" Body="&lt;p&gt;When I am cleaning my data, I have some features which contain large numbers and some features that are binary. Should I scale the large features and then add the binary columns or just scale them all together? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My fear is that scaling them all together makes the binary features seem less important than they really are. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am prepping a neural network for binary classification. I am using a sigmoid output and scaling my features from [0,1]&lt;/p&gt;&#xA;" OwnerUserId="38047" LastActivityDate="2021-12-22T15:00:10.200" Title="Am I supposed to scale binary features alongside other numerical features?" Tags="|data-mining|feature-selection|data-cleaning|feature-scaling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23052" PostTypeId="2" ParentId="19357" CreationDate="2017-09-15T07:56:33.103" Score="0" Body="&lt;p&gt;I am trying to solve a similar problem. Does your dataset contain a mix of text and numerical features ? If so the complexity of detecting anomalies increases ( I don't know by what factor).&#xA;If your dataset is uniform, for example containing only numeric values, you can potentially use a RNN which still needs a labelled dataset but it can detect time series like patterns ( since you mention comparison with pervious day's values for ex)&lt;/p&gt;&#xA;" OwnerUserId="39274" LastActivityDate="2017-09-15T07:56:33.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23055" PostTypeId="1" AcceptedAnswerId="23065" CreationDate="2017-09-15T09:47:22.647" Score="1" ViewCount="1198" Body="&lt;p&gt;The MNIST handwritten digit dataset uses a file format .idx. What are the advantages of this file format over alternatives such as CSV, TSX and ODS?&lt;/p&gt;&#xA;" OwnerUserId="39282" LastEditorUserId="13748" LastEditDate="2017-09-16T20:00:43.487" LastActivityDate="2017-09-16T20:00:43.487" Title="Why use the .idx data format?" Tags="|dataset|performance|mnist|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23056" PostTypeId="1" CreationDate="2017-09-15T09:54:39.927" Score="2" ViewCount="55" Body="&lt;p&gt;Are there Deep Learning models that utilize some sort of top-down information transfer (not only during training)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the model is used for speech recognition. The lower layers would listen to the samples and say: &quot;The word 'one', the word 'two', I can't understand what this sample is&quot;. And then an upper layer would suggest: &quot;Could it be the word 'three', that is not uncommon after 'one' and 'two'&quot;. And then the lower layers would be like: &quot;Now that you said it! Of course it's 'three'&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm pretty sure that an architecture like this is a requirement for reliable speech recognition, especially in noisy environments.&lt;/p&gt;&#xA;" OwnerUserId="39281" LastActivityDate="2017-09-15T09:54:39.927" Title="Deep Learning models with top-down transfer" Tags="|machine-learning|deep-learning|audio-recognition|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23057" PostTypeId="2" ParentId="10856" CreationDate="2017-09-15T10:21:01.287" Score="0" Body="&lt;p&gt;If you want stick to clustering, then you can calculate similarity matrix based on measure suitable for categorical data (hamming, jaccard...) and fetch it to hierarchical clustering algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also suggest to look for association rules, there is very nice package in R &lt;a href=&quot;https://cran.r-project.org/web/packages/arules/arules.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ARules&lt;/a&gt;, which can show the frequent products. Moreover, it can also visualize the &quot;products&quot; used together, so you could see groups of similar products.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-09-15T10:21:01.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23058" PostTypeId="2" ParentId="19357" CreationDate="2017-09-15T10:37:34.620" Score="4" Body="&lt;p&gt;I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can compare distribution of each variable either by using quantiles or any statistical test to see whether they are significantly different&lt;/li&gt;&#xA;&lt;li&gt;You could also count occurrence of each label/category and compare them&lt;/li&gt;&#xA;&lt;li&gt;I'd also try to employ any sort of distance measure. For example you could calculate mahalanobis distance and look for big changes&lt;/li&gt;&#xA;&lt;li&gt;Or something really simple - just an absolute difference between new and old data, set a threshold and everything exceeding the threshold will be reported&lt;/li&gt;&#xA;&lt;li&gt;You can also put in place some multidimensional techniques - like correlation matrix, principal components, clustering etc. and look for changes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If none of these are suitable, then there is whole branch of stats/ML models specialized for anomaly detection. SVM, t-SNE, Isolation forests, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.25.4115&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Peer Group Analysis&lt;/a&gt;, &lt;a href=&quot;http://www.dataminingapps.com/2016/04/what-does-break-point-analysis-mean-in-the-context-of-fraud-detection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Break Point Analysis&lt;/a&gt;, Time series (where you would look for outliers outside trends).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those methods have the advantage that they are sort of white-box, so you can tell why someone is an outlier. Should this not be the thing you want, others suggested ANN approaches, which will also work.&lt;/p&gt;&#xA;" OwnerUserId="17290" LastActivityDate="2017-09-15T10:37:34.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23059" PostTypeId="1" AcceptedAnswerId="23062" CreationDate="2017-09-15T12:24:56.750" Score="1" ViewCount="5377" Body="&lt;p&gt;I have a training set and a testing set of vectors. All the vectors are labeled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each labeled vector in the testing set, there are 3 vectors in the training set with the same label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the cosine distance in order to calculate the similarity between the elements in the vectors. In the picture, we can see the results of applying the cosine distance similarity in a subset of 6 vectors from the testing set and 18 from the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/61nY8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/61nY8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to create a similarity matrix of the labels. So, in this case, I'd need a matrix of 6x6 dimension, but I am not sure how to transform this matrix of scores to a similarity matrix.&lt;/p&gt;&#xA;" OwnerUserId="39287" LastActivityDate="2017-09-15T14:38:15.910" Title="Create similarity matrix" Tags="|algorithms|beginner|similarity|cosine-distance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23060" PostTypeId="2" ParentId="17382" CreationDate="2017-09-15T14:27:33.483" Score="4" Body="&lt;p&gt;According to their &lt;a href=&quot;http://brat.nlplab.org/examples.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation &lt;/a&gt;, BRAT does a lot of things, but text classification isn't one of them. BRAT is &quot;too powerful&quot; for that. I'd recommend you use a tool like &lt;a href=&quot;https://prodi.gy/demo&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prodigy&lt;/a&gt; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should do what you're looking for.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-15T14:27:33.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23061" PostTypeId="1" CreationDate="2017-09-15T14:31:35.660" Score="4" ViewCount="2757" Body="&lt;p&gt;I believe this should be the correct place for this topic since it did not seem to fit naturally on the math Stack Exchange.&lt;/p&gt;&#xA;&lt;p&gt;I am writing an essay about data visualization, but am having trouble nailing down some terminology. I used data from the OECD to investigate trends in the data about OECD member countries. I created a graph in Tableau using data from the OECD that depicts the GDP per capita, average # of years spent in education system, satisfaction score as reported by the member country citizens, and a &amp;quot;Feel Safe&amp;quot; score as reported by the member country citizens (a percentage of the surveyed population who said they would feel safe walking home alone at night). Click the following link to &lt;a href=&quot;http://www.filehosting.org/file/details/693633/OECD_Data.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;download the full OECD data set&lt;/a&gt; I used.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ABpA8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ABpA8.png&quot; alt=&quot;tableau graph&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am confused about whether to label my graph as a multivariate visualization or a multidimensional visualization.&lt;/p&gt;&#xA;&lt;p&gt;The core of my confusion comes from someone claiming that my graph is only two-dimensional since it can be displayed on a page. However, &amp;quot;dimensions&amp;quot; usually refers to the number of independent variables in a data set and the data displayed in my graph contains four independent variables.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Could someone please tell me what my graphic should be categorized as, and possibly explain some of the differences between multivariate and multidimensional data/visualizations (if any)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39292" LastEditorUserId="29169" LastEditDate="2021-08-20T21:26:01.830" LastActivityDate="2021-08-20T21:26:01.830" Title="What are the differences between multivariate data visualizations and multidimensional visualizations?" Tags="|visualization|tableau|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23062" PostTypeId="2" ParentId="23059" CreationDate="2017-09-15T14:38:15.910" Score="0" Body="&lt;p&gt;You have two scenarios here.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The vectors with the same labels are close to each other.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then, what you can do is perhaps use the distance between the two closest/furthest elements in the groups with label x and the group with label y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also create an average vector for each label, and then simply get the distance between these.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your vectors are not close to each other&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is not an issue. Here, what you'll want to see, is if for instance, for each vector with label x, there is a vector with label y close to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that, you could use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Earth_mover%27s_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Earth Mover's Distance&lt;/a&gt; which gives you a single score when you are trying to see how far two &quot;groups of things&quot; are.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-15T14:38:15.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23063" PostTypeId="2" ParentId="23050" CreationDate="2017-09-15T15:02:20.303" Score="1" Body="&lt;p&gt;There are two important concepts that you need to understand.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Scaling your features won't affect their &quot;importance&quot; in your neural network&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Intuitively, your neural network will itself learn which feature is important or not by learning weights.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Scaling your features will speed up your convergence and will limit the risks of overshooting or being stuck in a local optimum. It also makes numerical sense.&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Intuitively, scaling features can make your training faster because your &quot;path to convergence&quot; will probably be shorter if your features are scaled (right picture).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also makes numerical sense to scale your features because if you have very large and very small values, some of your weights might drop very low and this could cause some numerical issues and hinder the performance of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q3jqF.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q3jqF.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-15T15:02:20.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23065" PostTypeId="2" ParentId="23055" CreationDate="2017-09-15T17:51:11.813" Score="2" Body="&lt;p&gt;Generally, you will find datasets being distributed in CSV format for their simplicity and human readable format that you could ingest in any programming language with just the packages that the language is shipped with. Usually, tabular data is exported in CSV format and that is one of the reasons why MNIST dataset is not provided in CSV format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the quote from &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LeCun's website&lt;/a&gt; for storing the dataset in idx format.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The data is stored in a very simple file format designed for storing&#xA;  vectors and multidimensional matrices.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In terms of performance, binary file formats fairs better compared to text file formats like CSV or rich text format like ODS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following are some of the binary file formats that are widely used.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Avro format&lt;/li&gt;&#xA;&lt;li&gt;Parquet format &lt;/li&gt;&#xA;&lt;li&gt;Optimized Row Columnar (ORC) &lt;/li&gt;&#xA;&lt;li&gt;Protocol Buffers (protobuf)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These file formats support data compression, stores data type metadata to serialize and de-serialize data effeciently.&lt;/p&gt;&#xA;" OwnerUserId="9657" LastActivityDate="2017-09-15T17:51:11.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23066" PostTypeId="1" CreationDate="2017-09-15T18:48:30.000" Score="1" ViewCount="890" Body="&lt;p&gt;Is there a set and/or a model that is available freely in the Internet that can tell man from woman based on input image?&lt;/p&gt;&#xA;" OwnerUserId="39300" LastActivityDate="2017-09-21T08:32:08.847" Title="A data set to distinguish man from woman?" Tags="|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23067" PostTypeId="1" CreationDate="2017-09-15T18:59:27.633" Score="1" ViewCount="73" Body="&lt;p&gt;I have been looking for some methods to improve my prediction model, but could not find any so far. I have data, including 10 numerical features, that I use for prediction. I used a random forest regression model and it works perfectly on the train set (some part of perfection comes from over fitting). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the model is not doing a good job for data points that some of their features are not within the range of the sample data. For example, parameter x is between 0 and 2000 in the sample but the new data point that we want to predict has a value of 3000 for x. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is due to data limitation, but I wonder if there is any way that I can generate samples or improve the prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to build a simple model excluding this parameter, but the problem is that this is the most significant parameter in my model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hints are appreciated.&lt;/p&gt;&#xA;" OwnerUserId="39301" LastEditorUserId="29575" LastEditDate="2017-09-15T21:15:01.160" LastActivityDate="2018-02-20T21:35:18.240" Title="How to improve a regression prediction model for data with parameters which are out of sample range?" Tags="|regression|prediction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23068" PostTypeId="2" ParentId="22939" CreationDate="2017-09-15T19:51:39.327" Score="1" Body="&lt;p&gt;If you're prepared to use a neural net, it might be worthwhile looking into spatial pyramid pooling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A spatial pyramid pooling layer can take inputs of varying sizes (eg from  a convolutional layer) and convert them to an output of a fixed size (to pass on to a fully connected layer for instance). This is probably most useful where features are spatially related in some way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could continue simply implementing pca on your inputs, selecting the N dimensions with the highest variance for each example, but train the model continuously through reinforcement learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As there is more to how data is distributed along an axis than the variance, it is unlikely that any one pretrained model will perform well when your preprocessing step is constantly swapping out features, which is why a reinforcement model would be needed.  Combined with the pca, hopefully that won't take up too much overhead. &lt;/p&gt;&#xA;" OwnerUserId="39302" LastEditorUserId="39302" LastEditDate="2017-09-17T11:23:55.040" LastActivityDate="2017-09-17T11:23:55.040" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23069" PostTypeId="1" AcceptedAnswerId="23301" CreationDate="2017-09-15T21:13:34.773" Score="2" ViewCount="84" Body="&lt;p&gt;I recently saw the graph of correlated technologies from the &lt;a href=&quot;https://insights.stackoverflow.com/survey/2017#correlated-technologies&quot; rel=&quot;nofollow noreferrer&quot;&gt;SO Developer Survey&lt;/a&gt;, which was pretty; reproduced below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t4A2c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t4A2c.png&quot; alt=&quot;SO Developer Survey correlated technologies&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a weighted graph that represents the amount of communication between pairs of people. I'd like to create a similar visualization as above. I want to be able to tease out &quot;islands&quot; of people with high influence among each other, but little influence elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make this concrete, let's say I have four animals: &lt;code&gt;dog, wolf, cat, lion&lt;/code&gt;. The canines talk a lot amongst themselves; the felines do too, but members of one group only rarely talk to members of the other group. So my influence graph (which is symmetric, hence the upper triangular matrix) looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+------+-----+------+-----+------+&#xA;|      | Dog | Wolf | Cat | Lion |&#xA;+------+-----+------+-----+------+&#xA;| Dog  | ∞   | 100  | 1   | 1    |&#xA;| Wolf | -   | ∞    | 3   | 2    |&#xA;| Cat  | -   | -    | ∞   | 130  |&#xA;| Lion | -   | -    | -   | ∞    |&#xA;+------+-----+------+-----+------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know how to create graphs whose edges are visually represented by the weight between two nodes, but I am not sure how to account or calculate islands of influence. Are there general visualization techniques to apply to a data set like this, and also for figuring out islands of influence? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="18984" LastActivityDate="2017-09-25T17:04:46.360" Title="Calculation and visualization of islands of influence" Tags="|visualization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23070" PostTypeId="2" ParentId="15056" CreationDate="2017-09-15T23:22:26.193" Score="5" Body="&lt;p&gt;As an alternative to using padded arrays, you can just feed all of your data as one big spaghetti string and then do origami inside the tensorflow graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;&#xA;sess = tf.InteractiveSession()&#xA;&#xA;noodle = tf.placeholder(tf.float32, [None])&#xA;chop_indices = tf.placeholder(tf.int32, [None,2])&#xA;&#xA;do_origami = lambda list_idx: tf.gather(noodle, tf.range(chop_indices[list_idx,0], chop_indices[list_idx,1]))&#xA;&#xA;print( [do_origami(list_idx=i).eval({noodle:[1,2,3,2,3,6], chop_indices:[[0,2],[2,3],[3,6]]}).tolist() for i in range(3)] )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[1.0, 2.0], [3.0], [2.0, 3.0, 6.0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you have a variable number of inner lists, though, then good luck. You can't return a list from tf.while_loop and you can't just use a list comprehension as above so you would have to do the computations separately for each inner list.&lt;/p&gt;&#xA;" OwnerUserId="39308" LastEditorUserId="39308" LastEditDate="2017-09-15T23:27:28.400" LastActivityDate="2017-09-15T23:27:28.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23072" PostTypeId="1" AcceptedAnswerId="23074" CreationDate="2017-09-16T01:39:23.753" Score="2" ViewCount="863" Body="&lt;p&gt;In every explanation of SVMs, we're shown how training finds a hyperplane that best separates the data.  Presumably then for inference, you just check which side of the plane a point is on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, all the &quot;disadvantages of SVMs&quot; posts [&lt;a href=&quot;https://stats.stackexchange.com/questions/24437/advantages-and-disadvantages-of-svm&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://www.svms.org/disadvantages.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;] lament that SVM models are large and slow because they end up storing most of the data as support vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would SVMs store any of the data, rather than just the (coefficients of the) separating hyperplane?  (And what is a &quot;support vector&quot; in the soft-margin case, when points of both classes are scattered on both sides of the hyperplane, anyway?)&lt;/p&gt;&#xA;" OwnerUserId="39311" LastActivityDate="2019-02-25T05:31:03.570" Title="Why does an SVM model store the support vectors, and not just the separating hyperplane?" Tags="|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23073" PostTypeId="2" ParentId="18848" CreationDate="2017-09-16T01:54:54.407" Score="9" Body="&lt;p&gt;If you're in Python, there are a couple of packages that can automatically extract hundreds or thousands of features from your timeseries, correlate them with your labels, choose the most significant, and train models for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/blue-yonder/tsfresh&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/blue-yonder/tsfresh&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/rtavenar/tslearn&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/rtavenar/tslearn&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39311" LastActivityDate="2017-09-16T01:54:54.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23074" PostTypeId="2" ParentId="23072" CreationDate="2017-09-16T03:17:18.340" Score="3" Body="&lt;p&gt;The hyperplane is a linear combination of the support vectors. In the soft margin case, there is only a limited amount of slack; every input does not get to be support vector. In the nonlinear case, the separating &lt;em&gt;hypersurface&lt;/em&gt; may be embedded in an &lt;strong&gt;infinite-dimensional&lt;/strong&gt; space, making it impossible to store. To borrow from &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine#Kernel_trick&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Wikipedia article&lt;/a&gt;, the normal vector $w$ is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$w = \sum_i c_i y_i \phi(x_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\phi$ is the feature embedding function, and $c_i$ is a Lagrangian dual variable that is zero for points on the correct side of the margin. Instead, test points are classified through a kernel function $k(x_i,x_j) = \left&amp;lt; \phi(x_i), \phi(x_j) \right&amp;gt;$ like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x \to \mathrm{sgn}(\left&amp;lt;w , \phi(x)\right&amp;gt; + b) \equiv \mathrm{sgn} \left( b+\sum_i c_i y_i k(x_i, x)\right)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice how we avoided explicitly calculating $w$.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-09-16T03:17:18.340" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23075" PostTypeId="1" AcceptedAnswerId="23109" CreationDate="2017-09-16T09:35:59.727" Score="2" ViewCount="3070" Body="&lt;p&gt;If I trained a network using Neural Network classifier, how can I know which feature was most important for predicting the target variable? I mean how to create a &quot;feature ranking&quot; among the features (from high importance value to low). I have seen some literature about decision trees/AdaBoost but I am typically interested in Neural networks especially for classification purpose. To make it more clear, an example is shown in the figure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PW5FI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PW5FI.png&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="44605" LastEditDate="2018-06-26T03:26:07.610" LastActivityDate="2019-06-28T06:58:33.370" Title="How to rank a feature importance?" Tags="|neural-network|feature-selection|feature-extraction|matlab|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23076" PostTypeId="2" ParentId="23029" CreationDate="2017-09-16T12:38:50.900" Score="5" Body="&lt;p&gt;I'll try to provide some insight which will hopefully help.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;em&gt;Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough?&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Every neural net gets better in theory if it gets deeper. For a regular NN to model time connections properly, you could use the last n time steps as your input and the n+1th time  step as your target. This will generate your training set and depending on your data, you could be able to model your time series fairly efficiently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the most obvious pitfalls of this approach are actually addressed by RNNs/LSTM.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;em&gt;Does an RNN need more or less data in comparison to a NN to get the&#xA;same/ better results?&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Difficult question, which would probably require some empirical results to check that theory. Also, with neural nets, sometimes, it's more about how fast it trains rather than how much training data it has that will make the biggest difference in performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me, the main difference is that your regular NN will need a fixed-size input, whereas your RNN will be able to learn with input &quot;up to&quot; a certain size, which can be a big advantage to model the entire time series well.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;em&gt;Are there time series where normal NN or RNN/LSTM perform better?&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Again, this is a difficult question as it will depend on the data, the architecture of the networks, the training time etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't done empirical research on this, but I'd say that your best guess would be time series that are rather short (less than 100 time steps for instance). If the time series is short, you might not need to model such an intricate relationship through time, which a regular NN could perhaps do as well as an RNN.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;em&gt;Is it time data depended which Model will perform the best or are there some guidelines?&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As explained above, every learning exercise will highly depend on the architecture and hyperparameter used, even the initialization of your weights, whether you use pre-training or not, and of course, your data. Again, I think that the shortest the time series, the more competitive regular NNs could be. But again, this is merely intuition-based and hasn't been checked thoroughly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;- Can the NN behavior be understood better than the RNN behavior?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It all depends on what you mean by understanding. RNN have more weights than NN so ultimately, there will be more things to analyze and eventually understand. But, with more data also comes more information, so perhaps there is more information to be gained from an RNN than a simple NN, even if it's a deep one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plus, sometimes, based on the initialization of the weights and other parameters, the interpretation of the model could vary, even if it's trained on the same data.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-16T12:38:50.900" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23077" PostTypeId="2" ParentId="23038" CreationDate="2017-09-16T12:54:47.930" Score="1" Body="&lt;p&gt;You could approach this in two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Association-rule learning&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it's like trying to infer rules such that: if the reasons are X and Y, the diagnosis will likely be Z.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common algorithm is &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apriori&lt;/a&gt;, which is easy to implement.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Classification/Supervised learning&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your features could be binary (0 if reason X is not applied, 1 if it is). You could have one for each reason, and even combination of reasons etc. You might run into some issues if you have many possible reasons (your features will be very sparse).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the case, I'd look into dimensionality reduction to make your features denser.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-16T12:54:47.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23078" PostTypeId="2" ParentId="22991" CreationDate="2017-09-16T13:30:57.943" Score="4" Body="&lt;p&gt;Theoretically, you could use n-grams to model text sequences. But there are some good reasons why it's not often mentioned in textbooks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, modeling sequences is learning that after a, b, and c comes d. So if your sequence is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;Hello, my name is Bob, what a lovely day today, how are you?&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It technically doesn't matter if a = &quot;Hello&quot;, or &quot;H&quot;, or &quot;Hello, my name is Bob&quot;. To your model, it will just learn that after a thing, comes another thing etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, you could use n-grams. But the problem is that you need to make sure that the way the ngrams are split is meaningful (i.e. &quot;Hello, my&quot; is a probably worse split than &quot;my name is&quot;, which probably happens more often)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But don't worry, the whole point of using LSTM is that the memory part of the network will mimic using n-grams, as long as it helps with the overall performance. This means that if it needs to take more words into account to predict the next one, it will do it up to the depth of your network.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-16T13:30:57.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23079" PostTypeId="2" ParentId="22998" CreationDate="2017-09-16T13:37:07.767" Score="2" Body="&lt;p&gt;As always, &quot;is A better than B&quot; always depends on what you consider better? Is it accuracy, is it speed etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dumb but correct answer to your question is: &quot;try both and see which one is better&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performance of techniques is always to some extent dependent on the data. What you need to remember with word vectors, is that they are learned based on a certain context. If the context that was used by Google's model is similar to yours, you might be better off using their model. But if it's different, you might run into some problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just imagine the following case. You have 4 words: King, Man, Queen, Woman. Which pairs of two words would you create? Depending on the context, you could make a case for several&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;King/Man and Queen/Woman because of gender&lt;/li&gt;&#xA;&lt;li&gt;Man/Woman and King/Queen because of the use of the word etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-09-16T13:37:07.767" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23080" PostTypeId="2" ParentId="23075" CreationDate="2017-09-16T13:51:04.153" Score="1" Body="&lt;p&gt;There are multiple possibilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious one is to sum weights of all connections from input layer to the first hidden layer per input neuron. The neuron with highest sum of connection weights should be the most important. This has several problems. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Weights could be negative so you should use absolute value of them.&lt;/li&gt;&#xA;&lt;li&gt;The weight depend on the scale of the input variable. You should normalize the input for example by subtracting mean and dividing by standard deviation of the input (z-normalization). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Another possibility is to selectively remove features, retrain the network and observe removal of which features caused the biggest decrease in model accuracy. These are potentially the most important features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem of this approach is, that it changes the architecture of the network, retrains it and does not report the relevance of features entering the original network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find an elaborate discussion on the problem here &lt;a href=&quot;ftp://ftp.sas.com/pub/neural/importance.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ftp://ftp.sas.com/pub/neural/importance.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are multiple other approaches, such as Garson's, using complex functions to correct for some problems. You can find several of them implemented in &lt;a href=&quot;https://cran.r-project.org/web/packages/NeuralNetTools/README.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NeuralNetTools&lt;/a&gt; package in R.&lt;/p&gt;&#xA;" OwnerUserId="39319" LastEditorUserId="39319" LastEditDate="2017-09-18T09:12:22.900" LastActivityDate="2017-09-18T09:12:22.900" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23081" PostTypeId="2" ParentId="20471" CreationDate="2017-09-16T14:32:32.153" Score="0" Body="&lt;p&gt;You could employ Gaussian Mixture Modeling (or a variant). The objective is to fit a Gaussian kernel &lt;span class=&quot;math-container&quot;&gt;$N(\mu, \sigma)$&lt;/span&gt; to each of your sub-clusters. Baseline distance measure between pair of sub-clusters, you are seeking, could be &lt;span class=&quot;math-container&quot;&gt;$L_2$&lt;/span&gt; norm of their means &lt;span class=&quot;math-container&quot;&gt;$d(\mu_1, \mu_2)$&lt;/span&gt;. The sub-clusters would typically have different standard deviations. You can factor that into your distance measure, to improve interpretation of distance measure. You can use this to identify outliers. Typically, outliers would be characterized by joint criteria of highest mean distance from all other sub-clusters, and a low variance.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastEditorUserId="2515" LastEditDate="2021-02-18T20:33:52.473" LastActivityDate="2021-02-18T20:33:52.473" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23082" PostTypeId="1" CreationDate="2017-09-16T17:20:01.747" Score="2" ViewCount="925" Body="&lt;p&gt;The wikipedia says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Extreme learning machines are feedforward neural network for classification, regression,&#xA;      clustering, sparse approximation, compression and feature learning with a single layer or &#xA;      multi layers of hidden nodes, where the parameters of hidden nodes (not just the weights&#xA;      connecting inputs to hidden nodes) need not be tuned.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How is it possible to assign values to links without actually seeing the &lt;em&gt;Training set&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Till now from my research to understand this, I've made a lead, but not satisfactory because I can't anything from this below paragraph.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For any nonconstant piecewise continuous function&#xA;  that is used as the activation function, if the parameters of&#xA;  the hidden neurons are tuned, then the function can make&#xA;  the SLFNs approximate any target continuous function f (x).&#xA;  Then, according to any continuous distribution probability, the&#xA;  function sequence {h subscript i (x)} where i goes from 1 to L   can be randomly generated, and&#xA;  it has the universal approximation capability, which means that&#xA;  $$\lim_{L→ \infty} ||L&#xA;i=1 βi hi (x) − f (x)|| = 0$$&#xA;  holds with probability&#xA;  of one with appropriate output weights $β$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;SOURCE: A Parallel Multiclassification Algorithm for Big&#xA;Data Using an Extreme Learning Machine, IEEE transactions on neural networks 2017&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone prove/explain this with an example please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also any extra knowledge on extreme learning machine is very welcomed.&lt;/p&gt;&#xA;" OwnerUserId="39325" LastEditorUserId="8820" LastEditDate="2017-09-18T19:12:41.083" LastActivityDate="2018-03-07T12:12:45.560" Title="What is Extreme Learning Machine? Why tuning of weights is not required?" Tags="|machine-learning|neural-network|bigdata|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23083" PostTypeId="2" ParentId="22605" CreationDate="2017-09-16T18:41:53.470" Score="2" Body="&lt;p&gt;It is correct to use as you indicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in below , I use levenshtein for make similarity matrix. then use this matrix for clustring by affinity algorithm.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lev_similarity = -1*np.array([[distance.levenshtein(w1,w2) for w1 in words]&#xA;                              for w2 in words])&#xA;&#xA;af = affprop.fit(lev_similarity)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39328" LastEditorUserId="29575" LastEditDate="2017-09-16T20:00:26.117" LastActivityDate="2017-09-16T20:00:26.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23087" PostTypeId="2" ParentId="130" CreationDate="2017-09-17T01:39:10.617" Score="2" Body="&lt;p&gt;A1. What is dimensionality reduction: &#xA;If you think of data in a matrix, where rows are instances and columns are attributes (or features), then dimensionality reduction is mapping this data matrix to a new matrix with fewer columns. For visualization, if you think of each matrix-column (attribute) as a dimension in feature space, then dimensionality reduction is projection of instances from the higher dimensional space (more columns) to a lower dimensional sub-space (fewer columns).&#xA;&lt;a href=&quot;https://i.stack.imgur.com/KIveI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KIveI.png&quot; alt=&quot;Dimensionality reduction is subspace projection&quot;&gt;&lt;/a&gt;&#xA;Typical objective for this transformation is (1) preserving information in the data matrix, while reducing computational complexity; (2) improving separability of different classes in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A2. Dimensionality reduction as feature selection or feature extraction:&#xA;I'll use the ubiquitous &lt;a href=&quot;https://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;nofollow noreferrer&quot;&gt;Iris dataset&lt;/a&gt;, which is arguably the 'hello world' of data science. Briefly, the Iris dataset has 3 classes and 4 attributes (columns). I'll illustrate feature selection and extraction for the task of reducing Iris dataset dimensionality from 4 to 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I compute pair-wise co-variance of this dataset using library in Python called seaborn. The code is: sns.pairplot(iris, hue=&quot;species&quot;, markers=[&quot;o&quot;, &quot;s&quot;, &quot;D&quot;]) The figure I get is&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qK5nf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qK5nf.png&quot; alt=&quot;Iris pair-plot&quot;&gt;&lt;/a&gt;&#xA;I can &lt;strong&gt;select&lt;/strong&gt; the pair of attributes (2 dimensions) that provide me the greatest separation between the 3 classes (species) in the Iris dataset. This would be a case of feature-selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next up is feature extraction. Herein, I am projecting the 4-dimensional feature space of Iris to a new 2-dimensional subspace, which is not axis aligned with the original space. These are new attributes. They are typically based on the distribution in the original high dimensional space.&#xA;The most popular method is Principal Component Analysis, which computes Eigenvectors in the original space.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/vSpgw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vSpgw.png&quot; alt=&quot;PCA using SVD&quot;&gt;&lt;/a&gt;&#xA;Obviously, we are not restricted to using only a linear and global projection to a subspace based on Eigenvectors. We can use non-linear projection methods as well.&#xA;Here is an example of non-linear PCA using neural networks&#xA;&lt;a href=&quot;https://i.stack.imgur.com/YoO9f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YoO9f.png&quot; alt=&quot;non-linear PCA using NN&quot;&gt;&lt;/a&gt;&#xA;The attributes (dimensions) in the last example are &lt;strong&gt;extracted&lt;/strong&gt; from the original 4 attributes using neural networks. You can experiment with various flavors of PCA for iris dataset youself using this &lt;a href=&quot;http://www.bioconductor.org/packages/release/bioc/html/pcaMethods.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pca methods code&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary:&#xA;While feature extraction methods may appear to be superior in performance to feature selection, the choice is predicated by the application. The attributes from feature extraction typically lose physical interpretation, which may or may not be an issue based on the task at hand. For example, if you are designing a very expensive data collection task with costly sensors and need to economize on the attributes (number of different sensors), you'd want to collect a small pilot sample using all available sensors and then &lt;strong&gt;select&lt;/strong&gt; the ones that are most informative for the big data collection task.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-09-17T01:39:10.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23088" PostTypeId="2" ParentId="61" CreationDate="2017-09-17T02:27:31.110" Score="1" Body="&lt;p&gt;You are erroneously conflating two different entities: (1) bias-variance and (2) model complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) Over-fitting is bad in machine learning because it is impossible to collect a truly unbiased sample of &lt;a href=&quot;https://en.wikipedia.org/wiki/Statistical_population&quot; rel=&quot;nofollow noreferrer&quot;&gt;population&lt;/a&gt; of any data. The over-fitted model results in parameters that are biased to the sample instead of properly estimating the parameters for the entire population. This means there will remain a difference between the estimated parameters $\hat{\phi}$ and the optimal parameters $\phi^{*}$, regardless of the number of training epochs $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$|\phi^{*} - \hat{\phi}| \rightarrow e_{\phi} \mbox{  as }n\rightarrow \infty$, where $e_{\phi}$ is some bounding value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) Model complexity is in simplistic terms the number of parameters in $\phi$. If the model complexity is low, then there will remain a regression error regardless of the number of training epochs, even when $\hat{\phi}$ is approximately equal to $\phi^{*}$. Simplest example would be learning to fit a line (y=mx+c), where $\phi = \{m,c\}$ to data on a curve (quadratic polynomial).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[|y-M(\hat{\phi})|] \rightarrow e_{M} \mbox{ as } n \rightarrow \infty$, where $e_{M}$ is some regression fit error bounding value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary: Yes, both sample bias and model complexity contribute to the 'quality' of the learnt model, but they don't directly affect each other. If you have biased data, then regardless of having the correct number of parameters and infinite training, the final learnt model would have error. Similarly, if you had fewer than the required number of parameters, then regardless of perfectly unbiased sampling and infinite training, the final learnt model would have error.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-09-17T02:27:31.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23091" PostTypeId="2" ParentId="23017" CreationDate="2017-09-17T16:24:11.903" Score="1" Body="&lt;p&gt;You can train a decision tree classifier on the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A decision tree is one of few algorithms capable of producing an &quot;interpretable&quot; result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you need to understand that the clusters &lt;em&gt;are&lt;/em&gt; much more complex than a simple if-then rule.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-17T16:24:11.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23092" PostTypeId="2" ParentId="14187" CreationDate="2017-09-17T17:37:25.897" Score="8" Body="&lt;p&gt;In machine learning, a model $M$ with parameters and hyper-parameters looks like,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y \approx M_{\mathcal{H}}(\Phi | D)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is training data and $Y$ is output data (class labels in case of classification task).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective during training is to find estimate of parameters $\hat{\Phi}$ that optimizes some loss function $\mathcal{L}$ we have specified. Since, model $M$ and loss-function $\mathcal{L}$ are based on $\mathcal{H}$, then the consequent parameters $\Phi$ are also dependent on hyper-parameters $\mathcal{H}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hyper-parameters $\mathcal{H}$ are not 'learnt' during training, but does not mean their values are immutable. Typically, the hyper-parameters are fixed and we think simply of the model $M$, instead of $M_{\mathcal{H}}$. Herein, the hyper-parameters can also be considers as a-priori parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The source of confusion stems from the use of $M_{\mathcal{H}}$ and modification of hyper-parameters $\mathcal{H}$ during training routine in addition to, obviously, the parameters $\hat{\Phi}$. There are potentially several motivations to modify $\mathcal{H}$ during training. An example would be to change the learning-rate during training to improve speed and/or stability of the optimization routine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The important point of distinction is that, the result, say label prediction, $Y_{pred}$ is based on model parameters $\Phi$ and not the hyper-parameters $\mathcal{H}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distinction however has caveats and consequently the lines are blurred. Consider for example the task of clustering, specifically Gaussian Mixture Modeling (GMM). The parameters set here is $\Phi = \{\bar{\mu}, \bar{\sigma} \}$, where $\bar{\mu}$ is set of $N$ cluster means and $\bar{\sigma}$ is set of $N$ standard-deviations, for $N$ Gaussian kernels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may have intuitively recognized the hyper-parameter here. It is the number of clusters $N$. So $\mathcal{H} = \{N \}$. Typically, cluster validation is used to determine $N$ apriori, using a small sub-sample of the data $D$. However, I could also modify my learning algorithm of Gaussian Mixture Models to modify the number of kernels $N$ during training, based on some criterion. In this scenario, the hyper-parameter, $N$ becomes part of the set of parameters $\Phi = \{\bar{\mu}, \bar{\sigma}, N \}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, it should be pointed out that result, or predicted value, for a data point $d$ in data $D$ is based on $GMM(\bar{\mu}, \bar{\sigma})$ and not $N$. That is, each of the $N$ Gaussian kernels will contribute some likelihood value to $d$ based on the distance of $d$ from their respective $\mu$ and their own $\sigma$. The 'parameter' $N$ is not explicitly involved here, so its arguably not 'really' a parameter of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summary: the distinction between parameters and hyper-parameters is nuanced due to the way they are utilized by practitioners when designing the model $M$ and loss-function $\mathcal{L}$. I hope this helps disambiguate between the two terms.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-09-17T17:37:25.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23094" PostTypeId="2" ParentId="20165" CreationDate="2017-09-17T19:06:28.560" Score="2" Body="&lt;p&gt;&lt;strong&gt;1)&lt;/strong&gt; I believe this comes from general property of Maximum Likelihood estimation, namely, that it is equivalent to minimization of &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow noreferrer&quot;&gt;KL-divergence&lt;/a&gt; between the model distribution and the (empirical) data distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Proof&lt;/strong&gt;: Suppose we have data $X=\{\mathbf{x}_1 \ldots \mathbf{x}_N\}$ drawn independently from unknown data generating process. To approximate its distribution we consider family of distributions $p_{\text{model}}(\;\cdot\;;\theta)$ parameterized by parameters $\theta$. Then&#xA;$$&#xA;\theta_{MLE}&#xA;=\text{arg}\max_{\theta}p_{\text{model}}(X;\theta)&#xA;=\text{arg}\max_{\theta}\prod_{n=1}^N p_{\text{model}}(\mathbf{x}_n;\theta)&#xA;=\\&#xA;=\text{arg}\max_{\theta}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)&#xA;=\text{arg}\max_{\theta}\underbrace{\left(\frac{1}{N}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)\right)}_{  \mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{model}}(\mathbf{x};\theta) \right]},&#xA;$$&#xA;where $p_{\text{data}}(\mathbf{x})=\frac{1}{N}\sum_{n=1}^N \delta_{\mathbf{x}-\mathbf{x}_n}$ is (empirical) data distribution. Now by definition&#xA;$$&#xA;D_{KL}(p_{\text{data}} \| p_{\text{model}}(\;\cdot\;;\theta))=\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{data}}(\mathbf{x})-\log p_{\text{model}}(\mathbf{x};\theta) \right]&#xA;$$&#xA;Observe that the first term is independent of $\theta$ thus minimizing KL-divergence is equivalent to minimizing only the second term, which is a quantity with minus sign of what Maximum Likelihood maximizes. $\blacksquare$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if training data has high probability according to the RBM $\Leftrightarrow$ model distribution of RBM is close to empirical data distribution, then taking into account that RBM is trained using MCMC-based algorithm (CD-k/PCD-k), if we assume that state of Markov Chain particles are close to their stationary distribution, then feeding training example into RBM shouldn't change it too much after reconstruction. Therefore the low reconstruction error is expected in this case, and typically this happens in practice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2)&lt;/strong&gt; But as you correctly pointed out, even though it is convenient to measure reconstruction error during the training process, it is a poor measure of what actually is going on inside RBM during learning, since this is not the function RBM is aim to optimize (RBM maximizes probability of visible units $p(\mathbf{v})$ over all training data, which is equivalent to minimizing resp. KL-divergence). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, measuring (log-)likelihood directly is intractable in this model, but there exist better proxies to the true likelihood, such as pseudo-likelihood (more &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; or in the Hinton's practical guide you mentioned).&lt;/p&gt;&#xA;" OwnerUserId="27122" LastEditorUserId="27122" LastEditDate="2017-09-18T01:19:36.120" LastActivityDate="2017-09-18T01:19:36.120" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23095" PostTypeId="2" ParentId="9909" CreationDate="2017-09-17T19:12:27.647" Score="1" Body="&lt;p&gt;RBM maximizes probability of visible units $p(\mathbf{v})$, defined by the model, over all training data. It is equivalent to minimizing the &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow noreferrer&quot;&gt;KL-divergence&lt;/a&gt; between the model distribution and the (empirical) data distribution (more details &lt;a href=&quot;https://datascience.stackexchange.com/questions/20165/why-training-a-restricted-boltzmann-machine-corresponds-to-having-a-good-reconst/23094#23094&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maximum likelihood learning typically is performed by gradient descent. There are analytical expressions for the gradients of log-likelihood w.r.t. model weights, but they are intractable for regular-sized RBMs because of the exponential number of terms in the respective sums. Therefore the gradients themselves are approximated by MCMC-based algorithm called Contrastive Divergence (or its variants).&lt;/p&gt;&#xA;" OwnerUserId="27122" LastEditorUserId="29575" LastEditDate="2017-09-17T19:55:38.970" LastActivityDate="2017-09-17T19:55:38.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23096" PostTypeId="2" ParentId="11852" CreationDate="2017-09-17T19:24:44.730" Score="1" Body="&lt;p&gt;Just some elaborations, RBM is trained to maximize probability of visible units $p(\mathbf{v})$ defined by the model using &lt;em&gt;maximum likelihood&lt;/em&gt;, &lt;strong&gt;not&lt;/strong&gt; the reconstruction error, this is true. But the &lt;em&gt;maximum likelihood estimation is performed using Gradient Descent&lt;/em&gt;. Another pitfall that analytical expressions for gradients of log-likelihood w.r.t. model parameters are typically intractable (because of exponentially many terms in respective sums), therefore gradients themselves are approximated using MCMC-based algorithm called Contrastive Divergence (or its variants). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RBM is trained using GD/SGD where gradients are approximated using Contrastive Divergence and the function to optimize is log-likelihood of the training data.&lt;/p&gt;&#xA;" OwnerUserId="27122" LastActivityDate="2017-09-17T19:24:44.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23097" PostTypeId="2" ParentId="11042" CreationDate="2017-09-17T19:29:14.690" Score="1" Body="&lt;p&gt;I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance &lt;em&gt;pseudo log-likelihood&lt;/em&gt; (more &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), and in principle you can train RBM until PLL is not changed too much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).&lt;/p&gt;&#xA;" OwnerUserId="27122" LastEditorUserId="27122" LastEditDate="2017-09-17T19:54:22.573" LastActivityDate="2017-09-17T19:54:22.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23098" PostTypeId="2" ParentId="10127" CreationDate="2017-09-17T19:41:40.710" Score="0" Body="&lt;p&gt;General BMs were trained the same way as RBMs, but it was much slower, because Gibbs sampling was much less efficient for general BM due to presence of hidden-hidden and visible-visible connections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Until &lt;a href=&quot;http://www.utstat.toronto.edu/~rsalakhu/papers/dbm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, where the authors proposed new algorithm, that also uses variational mean-field approach to approximate one of the terms in gradients of log-likelihood.&lt;/p&gt;&#xA;" OwnerUserId="27122" LastActivityDate="2017-09-17T19:41:40.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23099" PostTypeId="2" ParentId="12443" CreationDate="2017-09-17T19:46:53.017" Score="1" Body="&lt;p&gt;Since RBM has only one layer of weights, why you bother changing sigmoid to ReLU in 1-layer net? Gradient vanishing is very unlikely to happen in such a shallow net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also train Gaussian-Bernoulli or Gaussian-Gaussian RBM (more &lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), which has identity activation function, which is closer to ReLU than sigmoid, and what is more justified if you have real-valued data, not binary. However, these types of networks are a bit less stable to train because of this unconstrained activation.&lt;/p&gt;&#xA;" OwnerUserId="27122" LastEditorUserId="29575" LastEditDate="2017-09-17T20:07:42.853" LastActivityDate="2017-09-17T20:07:42.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23100" PostTypeId="1" CreationDate="2017-09-18T03:30:24.640" Score="2" ViewCount="1014" Body="&lt;p&gt;I am new to Orange 3 and while the tool looks very promising, I was wondering if there is an Orange server where the model can be deployed for a production instance.&lt;/p&gt;&#xA;" OwnerUserId="39356" LastEditorUserId="29575" LastEditDate="2017-09-18T06:55:58.237" LastActivityDate="2018-07-05T12:41:41.317" Title="Is there an Orange server to deploy the models developed in Orange 3" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23101" PostTypeId="1" CreationDate="2017-09-18T05:17:46.670" Score="1" ViewCount="389" Body="&lt;p&gt;I was trying to categorical variable engineering following &lt;a href=&quot;https://arxiv.org/abs/1604.06737&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper. The code is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import random&#xA;import pandas&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;&#xA;from tensorflow.contrib import layers&#xA;from tensorflow.contrib import learn&#xA;from __future__ import print_function&#xA;&#xA;from sklearn.preprocessing import LabelEncoder&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My dataset looks like the following. It's has 2 independent variable ('X1' &amp;amp; 'X2')and 1 dependent variable ('lable'). 'X2' is the categorical variable. I want to create an embedding vector for this variable and run the simple linear regression to predict 'label'using Tensorflow.  I could use any other method. But since linear regression is easiest to understand, I'm trying that.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame({'X1': np.array([&quot;A&quot;,&quot;A&quot;,&quot;B&quot;,&quot;C&quot;,&quot;B&quot;,&quot;C&quot;,&quot;B&quot;,&quot;C&quot;,&quot;C&quot;,&quot;B&quot;,&#xA;                         &quot;A&quot;,&quot;B&quot;,&quot;A&quot;,&quot;C&quot;,&quot;A&quot;,&quot;A&quot;,&quot;C&quot;]),'X2': np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,&#xA;                         7.042,10.791,5.313,7.997,5.654,9.27,3.1]),&#xA;                       'label': np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,&#xA;                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;For variable 'X1', I'm creating levels.&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;encoder = LabelEncoder()&#xA;encoder.fit(df.X1.values)&#xA;X = encoder.transform(df.X1.values)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Recreating dependent variable list.&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,&#xA;                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Setting Hyper-parameters&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_epochs = 5&#xA;learning_rate = 1e-3&#xA;cardinality = len(np.unique(X))&#xA;embedding_size = 2&#xA;input_X_size = 1&#xA;n_hidden = 10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Setting up variables:&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;embeddings = tf.Variable(tf.random_uniform([cardinality, embedding_size], -1.0, 1.0))&#xA;&#xA;&#xA;h = tf.Variable(tf.truncated_normal((embedding_size + len(df.X1), n_hidden), stddev=0.1))&#xA;&#xA;&#xA;W_out = tf.get_variable(name='out_w', shape=[n_hidden],&#xA;                            initializer=tf.contrib.layers.xavier_initializer())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Embedding:&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;embedded_chars = tf.nn.embedding_lookup(embeddings, x)&#xA;embedded_chars = tf.reshape(embedded_chars, [-1])&#xA;embedded_chars= embedded_chars + np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,&#xA;                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Multiplying with Hidden Layers:&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;layer_1 = tf.matmul(embedded_chars,h)&#xA;layer_1 = tf.nn.relu(layer_1)&#xA;out_layer = tf.matmul(layer_1, W_out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;# Define loss and optimizer&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost = tf.reduce_sum(tf.pow(out_layer-y, 2))/(2*n_samples)&#xA;optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Run the graph&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;init = tf.global_variables_initializer()&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Launch the graph&#xA;with tf.Session() as sess:&#xA;    sess.run(init)&#xA;&#xA;    for epoch in range(training_epochs):&#xA;        avg_cost = 0.&#xA;&#xA;        _, c = sess.run([optimizer, cost],&#xA;                        feed_dict={x: X, y: Y})&#xA;print(&quot;Ran without Error&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While running the code, I'm getting the following error.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_1' (op:&#xA;  'MatMul') with input shapes: [17], [19,10].&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not able to add the continuous variable with embedding variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please guide me how to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you! &lt;/p&gt;&#xA;" OwnerUserId="10810" LastEditorUserId="10810" LastEditDate="2017-09-18T05:36:01.633" LastActivityDate="2018-07-06T12:50:34.610" Title="Categorical Variable Reduction using NN" Tags="|neural-network|tensorflow|embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23105" PostTypeId="1" CreationDate="2017-09-18T09:04:58.073" Score="1" ViewCount="646" Body="&lt;p&gt;I just finished reading &lt;a href=&quot;https://arxiv.org/abs/1502.01852&quot; rel=&quot;nofollow noreferrer&quot;&gt;Delving Deep into Rectifiers&lt;/a&gt; paper. This paper proposes a new activation function called PReLU. Maybe it is obvious, because the paper did not mention it, but I want to know when is the parameter of a PReLU updated? Is it updated before weight update or after weight update?&lt;/p&gt;&#xA;" OwnerUserId="39363" LastEditorUserId="29575" LastEditDate="2017-09-18T18:25:54.573" LastActivityDate="2017-10-18T19:16:07.337" Title="PReLU Activation Function update rule" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23106" PostTypeId="2" ParentId="23105" CreationDate="2017-09-18T09:43:43.360" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it updated before weight update or after weight update?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It doesn't usually matter, provided your NN framework has separate gradient calculation stage from parameter update stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gradients should be collected with all weights and parameters at their current values (during back-propagation). Then the parameters are updated, using those gradients. There should be no interaction between the parameters during the update step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you have implemented the training code yourself from scratch, and have interleaved the gradient calculations and weight updates, then this only works in a pure online situation with weights updated as the gradients are calculated. In that scenario, you should probably update the weights for a layer &lt;em&gt;before&lt;/em&gt; updating the PReLU parameter for the same layer, to prevent changes in the activation function altering the gradients during your calculations (this is not a concern with usual weight updates). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, separate your gradient calculations from the weight update steps. This is more flexible and allows you to use other more advanced optimisations and layer designs.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-18T09:43:43.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23109" PostTypeId="2" ParentId="23075" CreationDate="2017-09-18T11:50:20.840" Score="1" Body="&lt;p&gt;The firs solution sevo proposes is not feasible because of a third problem that was not mentioned. The first layer only learns a first representation of the input, which is used in later layers. Even if the absolute weights of $x_1$ might be very big, if the later layers have small weights connected to these neurons the importance goes down. This is exactly why neural networks are considered to be difficult to interpret. The rest of the answer is useful, I just wanted to add this.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-18T11:50:20.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23112" PostTypeId="1" CreationDate="2017-09-18T14:35:13.653" Score="-1" ViewCount="548" Body="&lt;p&gt;From SQL server I imported multiple tables that each have multiple fields. Unfortunately the field names are not that descriptive (data is originally from SAP) but I have a separate Excel file that has description for all the field names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way how I could rename multiple fields once by using the Excel file? Renaming them one by one would be too slow way to do it.&lt;/p&gt;&#xA;" OwnerUserId="39378" LastActivityDate="2017-09-19T08:08:49.167" Title="Tableau: How to change multiple field names" Tags="|preprocessing|sql|tableau|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23113" PostTypeId="2" ParentId="23112" CreationDate="2017-09-18T14:37:23.093" Score="0" Body="&lt;p&gt;Do you have/know python pandas? An easy why would be too load it into pandas and do the modifications there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;df= pd.read(excel)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AFTER Comment &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OR:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can rename columns in SQL with AS - if that is possible with your SAP import. To create a (semi-) automated Logic should be easy with a decent Texteditor,&lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-09-19T08:08:49.167" LastActivityDate="2017-09-19T08:08:49.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23114" PostTypeId="1" CreationDate="2017-09-18T15:32:15.900" Score="1" ViewCount="817" Body="&lt;p&gt;One CLass SVM is a well known machine learning algorithm for novality detection, but it is applicable only for a set of vectors not for time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read a paper recently of how to use (One-Class SVM) OCSVM with time series data by first converting the time series into a phase space then apply the projection into OCSVM [1] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an implementation or library for such an algorithm to handle OCSVM with time series data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.653.2440&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.653.2440&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30884" LastEditorUserId="30884" LastEditDate="2017-09-18T18:34:58.537" LastActivityDate="2017-09-18T18:34:58.537" Title="One Class SVM for time series data" Tags="|time-series|svm|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23115" PostTypeId="1" CreationDate="2017-09-18T15:47:42.540" Score="0" ViewCount="467" Body="&lt;p&gt;I have millions of lat long points that have been grouped into squares. Some squares have thousands of points, others have a couple of points. The idea is that we have one set of lat long for the square with a weighting related to the square based on the number of items it has instead of having millions of rows of data to the cluster.&lt;/p&gt;&#xA;&lt;p&gt;I was originally using the &lt;a href=&quot;https://cran.r-project.org/web/packages/leaderCluster/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;leadercluster algorithim&lt;/a&gt; which allowed me to specify the distance each of my clusters should cover. This is ideal for my use case but now I would like to use it to cluster squares factoring in the weight of the squares. This is basically a learning experience for me and any help would be greatly appreciated&lt;/p&gt;&#xA;&lt;p&gt;I have seen &lt;a href=&quot;https://stats.stackexchange.com/questions/13995/weighted-clustering-on-large-spatial-dataset&quot;&gt;this&lt;/a&gt; which could be useful but I'm not really able to make headway with it&lt;/p&gt;&#xA;&lt;p&gt;Below is some sample data in R about airports and arrivals&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- read.csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_february_us_airport_traffic.csv')&#xA;&#xA;ggplot(df, aes(x = long, y = lat)) + geom_point()&#xA;&#xA;sample &amp;lt;- df %&amp;gt;% select(long, lat, arrivals = cnt)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="25320" LastEditorUserId="97640" LastEditDate="2021-06-29T16:59:17.870" LastActivityDate="2023-12-23T21:02:10.237" Title="Weighted clustering of lat lon coordinates" Tags="|r|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23119" PostTypeId="1" CreationDate="2017-09-18T17:43:23.087" Score="1" ViewCount="2095" Body="&lt;p&gt;Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, instead of plotting a scatter with lots of data points, I would like to plot just the average y value at every x value. In pandas, this is easily done with groupby().mean(). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a similar widget/feature I'm overlooking in orange?&lt;/p&gt;&#xA;" OwnerUserId="39391" LastEditorUserId="29575" LastEditDate="2017-09-18T19:00:36.243" LastActivityDate="2024-01-19T10:07:25.763" Title="Orange3 summarizing data, grouping data values" Tags="|data-mining|dataset|data-cleaning|data|orange|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23121" PostTypeId="2" ParentId="13461" CreationDate="2017-09-18T19:10:34.827" Score="9" Body="&lt;p&gt;Currently (Keras v2.0.8) it takes a bit more effort to get predictions on single rows after training in batch.&lt;/p&gt;&#xA;&lt;p&gt;Basically, the &lt;code&gt;batch_size&lt;/code&gt; is fixed at training time, and has to be the same at prediction time.&lt;/p&gt;&#xA;&lt;p&gt;The workaround right now is to take the weights from the trained model, and use those as the weights in a new model you've just created, which has a &lt;code&gt;batch_size&lt;/code&gt; of 1.&lt;/p&gt;&#xA;&lt;p&gt;The quick code for that is&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = create_model(batch_size=64)&#xA;mode.fit(X, y)&#xA;weights = model.get_weights()&#xA;single_item_model = create_model(batch_size=1)&#xA;single_item_model.set_weights(weights)&#xA;single_item_model.compile(compile_params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;'s a blog post that goes into more depth.&lt;/p&gt;&#xA;&lt;p&gt;I've used this approach in the past to have multiple models at prediction time- one that makes predictions on big batches, one that makes predictions on small batches, and one that makes predictions on single items. Since batch predictions are much more efficient, this gives us the flexibility to take in any number of prediction rows (not just a number that is evenly divisible by &lt;code&gt;batch_size&lt;/code&gt;), while still getting predictions pretty rapidly.&lt;/p&gt;&#xA;" OwnerUserId="26173" LastEditorUserId="98307" LastEditDate="2020-08-05T13:58:20.647" LastActivityDate="2020-08-05T13:58:20.647" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23123" PostTypeId="1" CreationDate="2017-09-18T20:55:19.980" Score="0" ViewCount="1042" Body="&lt;p&gt;With data that looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;blockNumber&quot;:&quot;1941895&quot;,&quot;blockHash&quot;:&quot;0x53464299a83cecc3e4d930b617c9518b8f74139265423d8110a919f5180bec79&quot;,&quot;hash&quot;:&quot;0x0abe75e40a954d4d355e25e4498f3580e7d029769897d4187c323080a0be0fdd&quot;,&quot;from&quot;:&quot;0x4586ffaf28e08b1613dd96ced9b57d52e8ad9d72&quot;,&quot;to&quot;:&quot;0x91337a300e0361bddb2e377dd4e88ccb7796663d&quot;,&quot;gas&quot;:&quot;21000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;1&quot;,&quot;value&quot;:&quot;0x22c06103f88111000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:47:25 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941645&quot;,&quot;blockHash&quot;:&quot;0x78804d09bb4e7126f53133e33e3548e0f04a691c01661ab9b719c3811e54355e&quot;,&quot;hash&quot;:&quot;0x22c2b6490900b21d67ca56066e127fa57c0af973b5d166ca1a4bf52fcb6cf81c&quot;,&quot;from&quot;:&quot;0x81bbf9f19ffe8368efe7611ccf5dcbdb4618b645&quot;,&quot;to&quot;:&quot;0xb01a7866a244dbb600a7bbd170d43d4221838868&quot;,&quot;gas&quot;:&quot;90000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;0&quot;,&quot;value&quot;:&quot;0x4563918244f40000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 19:57:50 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941910&quot;,&quot;blockHash&quot;:&quot;0xc7ba89fc0110a033c4bd03be4505014761141b956c228bc51ec49c15a4508ce4&quot;,&quot;hash&quot;:&quot;0x8570106b0385caf729a17593326db1afe0d75e3f8c6daef25cd4a0499a873a6f&quot;,&quot;from&quot;:&quot;0x91337a300e0361bddb2e377dd4e88ccb7796663d&quot;,&quot;to&quot;:&quot;0x9fde2180b544b7690c35bdc66182eb843ac38030&quot;,&quot;gas&quot;:&quot;90000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;6356&quot;,&quot;value&quot;:&quot;0x41e92b66341ef0000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:50:12 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941919&quot;,&quot;blockHash&quot;:&quot;0x4785c1b1a678cf7058e1fed3fc1c7d33c4326c2fb309f5fc75688f23d496b61c&quot;,&quot;hash&quot;:&quot;0x8adfe7fc3cf0eb34bb56c59fa3dc4fdd3ec3f3514c0100fef800f065219b7707&quot;,&quot;from&quot;:&quot;0x69ca903e87329fd63a3c7b2d3efde6a9bf3c3d45&quot;,&quot;to&quot;:&quot;0xbfc39b6f805a9e40e77291aff27aee3c96915bdd&quot;,&quot;gas&quot;:&quot;40000&quot;,&quot;gasUsed&quot;:&quot;29130&quot;,&quot;gasPrice&quot;:&quot;30000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[{&quot;address&quot;:&quot;0xbfc39b6f805a9e40e77291aff27aee3c96915bdd&quot;,&quot;topics&quot;:[&quot;0x23919512b2162ddc59b67a65e3b03c419d4105366f7d4a632f5d3c3bee9b1cff&quot;],&quot;data&quot;:&quot;AAAAAAAAAAAAAAAAwNMyg48U70L83hzyUYxCfdtnZyk=&quot;}],&quot;nonce&quot;:&quot;20&quot;,&quot;value&quot;:&quot;0x1d2eb2accbaf90800&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:52:08 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941922&quot;,&quot;blockHash&quot;:&quot;0xd46dbf526f6d7c9197e841c8a4d7b2f4abdac4a62860cffabb943a46d07a86d4&quot;,&quot;hash&quot;:&quot;0x8b0fe2b7727664a14406e7377732caed94315b026b37577e2d9d258253067553&quot;,&quot;from&quot;:&quot;0x0b2c5cba2dc240e867f7721412c20e6016596d26&quot;,&quot;to&quot;:&quot;0x9c83fe12c7575ea7350019e04253d3620957851f&quot;,&quot;gas&quot;:&quot;21000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;21000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;2&quot;,&quot;value&quot;:&quot;0x7ce66c50e2840000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:52:51 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941688&quot;,&quot;blockHash&quot;:&quot;0x86bb1e90d0fa7be11d3f196057976383bb73cbd1596992e868155a576b5ddfb9&quot;,&quot;hash&quot;:&quot;0x244b29b60c696f4ab07c36342344fe6116890f8056b4abc9f734f7a197c93341&quot;,&quot;from&quot;:&quot;0x006cdc135b4e3a89d3ac1027ec3de609b8fff500&quot;,&quot;to&quot;:&quot;0x58ae42a38d6b33a1e31492b60465fa80da595755&quot;,&quot;gas&quot;:&quot;50000&quot;,&quot;gasUsed&quot;:&quot;50000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;47&quot;,&quot;value&quot;:&quot;0xc7140013deaf40&quot;,&quot;error&quot;:&quot;invalid jump destination (PUSH1) 2&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:06:38 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941794&quot;,&quot;blockHash&quot;:&quot;0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29&quot;,&quot;hash&quot;:&quot;0xf2b5b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63&quot;,&quot;from&quot;:&quot;0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9&quot;,&quot;to&quot;:&quot;0x2ba24c66cbff0bda0e3053ea07325479b3ed1393&quot;,&quot;gas&quot;:&quot;121000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;14&quot;,&quot;value&quot;:&quot;0x24406420d09ce7440000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:28:11 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941716&quot;,&quot;blockHash&quot;:&quot;0x75e1602cad967a781f4a2ea9e19c97405fe1acaa8b9ad333fb7288d98f7b49e3&quot;,&quot;hash&quot;:&quot;0xf8f2a397b0f7bb1ff212b6bcc57e4a56ce3e27eb9f5839fef3e193c0252fab26&quot;,&quot;from&quot;:&quot;0xa0480c6f402b036e33e46f993d9c7b93913e7461&quot;,&quot;to&quot;:&quot;0xb2ea1f1f997365d1036dd6f00c51b361e9a3f351&quot;,&quot;gas&quot;:&quot;121000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;1&quot;,&quot;value&quot;:&quot;0xde0b6b3a7640000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:12:17 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941794&quot;,&quot;blockHash&quot;:&quot;0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29&quot;,&quot;hash&quot;:&quot;0xf275b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63&quot;,&quot;from&quot;:&quot;0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9&quot;,&quot;to&quot;:&quot;0x2ba24c66cbff0bda0e3053ea07325479b3ed1393&quot;,&quot;gas&quot;:&quot;121000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;14&quot;,&quot;value&quot;:&quot;0x24406420d09ce7440000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:28:11 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941794&quot;,&quot;blockHash&quot;:&quot;0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29&quot;,&quot;hash&quot;:&quot;0xf285b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63&quot;,&quot;from&quot;:&quot;0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9&quot;,&quot;to&quot;:&quot;0x2ba24c66cbff0bda0e3053ea07325479b3ed1393&quot;,&quot;gas&quot;:&quot;121000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;14&quot;,&quot;value&quot;:&quot;0x24406420d09ce7440000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:28:11 UTC&quot;}&#xA;{&quot;blockNumber&quot;:&quot;1941895&quot;,&quot;blockHash&quot;:&quot;0x53464299a83cecc3e4d930b617c9518b8f74139265423d8110a919f5180bec79&quot;,&quot;hash&quot;:&quot;0x0abg75e40a954d4d355e25e4498f3580e7d029769897d4187c323080a0be0fdd&quot;,&quot;from&quot;:&quot;0x4586ffaf28e08b1613dd96ced9b57d52e8ad9d72&quot;,&quot;to&quot;:&quot;0x91337a300e0361bddb2e377dd4e88ccb7796663d&quot;,&quot;gas&quot;:&quot;21000&quot;,&quot;gasUsed&quot;:&quot;21000&quot;,&quot;gasPrice&quot;:&quot;20000000000&quot;,&quot;input&quot;:&quot;&quot;,&quot;logs&quot;:[],&quot;nonce&quot;:&quot;1&quot;,&quot;value&quot;:&quot;0x22c06103f88111000&quot;,&quot;timestamp&quot;:&quot;2016-07-24 20:47:25 UTC&quot;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Processed as &lt;code&gt;JSON&lt;/code&gt; with the following script: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import pprint&#xA;import collections&#xA;import itertools&#xA;&#xA;with open('toy.json', 'rb') as inpt:&#xA;&#xA;    dict_hash_gas = list()&#xA;    for line in inpt:&#xA;        resource = json.loads(line)&#xA;        dict_hash_gas.append({resource['hash']:resource['gas']})&#xA;&#xA;# Count up the values&#xA;counts = collections.Counter(v for d in dict_hash_gas for v in d.values())&#xA;&#xA;counts = counts.most_common()&#xA;&#xA;# Apply a threshold&#xA;threshold = 1000&#xA;counts = [list(group) for val, group in itertools.groupby(counts, lambda x: x[1] &amp;gt; threshold) if val]&#xA;&#xA;# print(counts)&#xA;&#xA;# resort the list by the magnitude of the manifest values&#xA;# as opposed to the frequency &#xA;counts[0].sort(key=lambda x:int(x[0]))&#xA;&#xA;# print(counts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would it be possible to partition the data into three distinct clusters based on the &lt;code&gt;gas&lt;/code&gt; variable?&lt;/p&gt;&#xA;" OwnerUserId="8285" LastActivityDate="2017-09-19T07:04:51.337" Title="clustering JSON data into three clusters, based on single attribute, using python" Tags="|python|clustering|json|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23124" PostTypeId="1" CreationDate="2017-09-18T21:56:06.543" Score="0" ViewCount="80" Body="&lt;p&gt;The following &lt;a href=&quot;https://github.com/ihadanny/IANNA/blob/master/gym_do_not_repeat_yourself/envs/do_not_repeat_yourself_env.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;env&lt;/a&gt; describes a simple task: there's a bit-vector of size 10, with all bits initially set to 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The agent must learn to switch it all to 1s. It has 10 possible actions - each one switches the corresponding bit to 1. Switching a bit which was 0 to 1 gives a reward of 1, but trying to switch a bit which is already 1 gives a reward of -1 and ends the game. The agent sees the entire bit-vector after each step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why isn't it very very simple? I wrote a policy-gradient approach &lt;a href=&quot;https://github.com/ihadanny/IANNA/blob/master/do_not_repeat_yourself_agent.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;agent&lt;/a&gt; but it doesn't arrive to sufficient results (stay at around ~3 points average) after 10K episodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I missing? Is it indeed such a hard task???&lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2017-09-19T12:50:41.330" Title="suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector" Tags="|deep-learning|tensorflow|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23129" PostTypeId="1" CreationDate="2017-09-19T02:48:27.693" Score="4" ViewCount="604" Body="&lt;p&gt;I was recently asked this question in an interview and wondered what the answer would be - &quot;How do Factorization Machines get around the overfitting problem when using second-order interactions?&quot;&lt;/p&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2018-07-30T10:55:17.180" Title="Factorization Machine - prevent over fitting" Tags="|machine-learning|recommender-system|overfitting|matrix-factorisation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23130" PostTypeId="1" CreationDate="2017-09-19T06:40:41.327" Score="0" ViewCount="605" Body="&lt;p&gt;I am hoping to use &lt;strong&gt;affinity propagation&lt;/strong&gt; to cluster my data using &lt;code&gt;sklearn&lt;/code&gt;. But I came across a question whether to use a &lt;strong&gt;distance matrix&lt;/strong&gt; or &lt;strong&gt;similarity matrix&lt;/strong&gt; in the &lt;code&gt;fit&lt;/code&gt; method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know what is suitable to use?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2019-09-10T00:01:20.127" Title="What is required in Affinity Propagation" Tags="|machine-learning|data-mining|clustering|text-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23131" PostTypeId="2" ParentId="23119" CreationDate="2017-09-19T06:55:02.927" Score="0" Body="&lt;p&gt;Unfortunately, not that I know of.&#xA;You could try Feature Constructor and define custom ranges or simply create a column with mean, then plot it. But that is not so different from Python Script.&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2017-09-19T06:55:02.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23132" PostTypeId="1" CreationDate="2017-09-19T06:55:21.840" Score="2" ViewCount="1379" Body="&lt;p&gt;In the &lt;a href=&quot;http://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html#hyperparameter-optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; given for &lt;a href=&quot;https://sklearn-crfsuite.readthedocs.io/en/latest/api.html#sklearn_crfsuite.CRF&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn_crfsuite&lt;/a&gt;, the parameter space that needs to be passed to a cross_validating class like &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomizedSearchCV&lt;/a&gt; is defined as below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params_space = {&#xA;    'c1': scipy.stats.expon(scale=0.5),&#xA;    'c2': scipy.stats.expon(scale=0.05),&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the example the algorithm that is shown is &quot;lbfgs&quot;. Other supported algorithms in sklearn_crfsuite are &quot;l2sgd&quot;, &quot;ap&quot;, &quot;pa&quot;, &quot;arow&quot;. How should the variable &lt;code&gt;params_space&lt;/code&gt; be defined in those cases?&lt;/p&gt;&#xA;" OwnerUserId="39423" LastEditorUserId="11097" LastEditDate="2017-09-19T07:13:33.667" LastActivityDate="2018-02-02T09:26:37.983" Title="How should the hyper parameters be defined for the other algorithms defined for sklearn_crfsuite.CRF" Tags="|classification|scikit-learn|gradient-descent|perceptron|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23133" PostTypeId="2" ParentId="23123" CreationDate="2017-09-19T07:04:51.337" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Sort the data&lt;/li&gt;&#xA;&lt;li&gt;Perform Kernel Density Estimation&lt;/li&gt;&#xA;&lt;li&gt;Find density minima. These are the optimum split points&lt;/li&gt;&#xA;&lt;li&gt;Split the data&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-19T07:04:51.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23134" PostTypeId="2" ParentId="23115" CreationDate="2017-09-19T07:06:56.087" Score="0" Body="&lt;p&gt;Leader clustering is so simple, &lt;strong&gt;the weight does not make a difference&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It assigns points to a cluster if the distance is less than a threshold. It does not matter whether the point has 1 weight or is a &quot;square&quot; of 100.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-19T07:06:56.087" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23136" PostTypeId="1" AcceptedAnswerId="23139" CreationDate="2017-09-19T08:39:36.083" Score="5" ViewCount="344" Body="&lt;p&gt;Say I am training neural networks using a train set and set aside a validation set V. I obtain models h's after each epoch along with the validation losses(0-1 loss) $\hat{L}(h_1,V)$, $\hat{L}(h_2,V)$ ... if I use the early stopping rule suggested &lt;a href=&quot;https://www.quora.com/How-does-one-employ-early-stopping-in-TensorFlow&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;(top answer). &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?&lt;/li&gt;&#xA;&lt;li&gt;How can I bound the true loss using $\hat{L}(h_*,V)$ ?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm guessing no for the first one since the stopping rule depends on the partition of my data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Afaik the bounds that can be applied depends on the size of my hypothesis set and I'm not entirely sure if it's finite, countable or uncountable in this case. &lt;/p&gt;&#xA;" OwnerUserId="39425" LastActivityDate="2017-09-22T06:56:16.163" Title="Early stopping and bounds" Tags="|machine-learning|self-study|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23138" PostTypeId="2" ParentId="23124" CreationDate="2017-09-19T12:33:46.927" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why isn't it very very simple?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The problem description is simple, but not in a way that is accessible to the standard RL algorithms. First of all, the analogy of switches and &quot;corresponding&quot; has no meaning to the RL. Secondly, the low complexity is not detectable and is outside of the assumptions of RL agents in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reinforcement Learning is based on an internal model of the problem as a generic Markov Decision Process (MDP). The RL agent has to discover, by trial and error, what the MDP does, typically starting with no knowledge or assumptions other than the set of states and allowed actions. The agent needs to discover &lt;em&gt;enough&lt;/em&gt; about the MDP to make optimal actions, whilst the only assumption is that the environment is a MDP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple switch problem can definitely be modelled as a MDP. The agent then has to discover the simple configuration in the toy problem. What makes it harder than the problem description:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;State/action space. There are 1024 states and 10 actions. There is no a priori reason to link any action with any outcome, so to be thorough the agent has to try them all. A minimal search would be across 10,240 state/action pairs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;State transitions. There are 1024 states where an action can be taken, and a terminal state. So there are 1024 * 1025 = 1,049,600 possible state transitions for every action. In your case, each state/action pair has one outcome, and there are roughly 5,000 different transitions (because half of them are to the terminal state).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The agent does not know that the state transitions are deterministic and sparse. It can only discover that by trial and error. For instance, perhaps in a particular configuration, taking one particular action would cause some bits to re-set. Maybe that only happens one time in four attempts. Once you add the possibility of arbitrary probability distributions here, then the number of potential MDPs is infinite. RL algorithms are designed to cope with this, still learning a good approximation of an optimal policy eventually. There is no simple way to make them dumber and faster against a simpler environment without encoding some of your knowledge about that environment.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The agent does not know that the rewards are deterministic and simple. Again it can only discover this by trial and error. To the agent the different state vectors and actions are neither similar to each other (all switches and bits) nor have any association (action 0 is not associated in any way with bit 0). Even after the associations are observed, they are not encoded in a way that the agent can reason by analogy. Again, once you add the possibility of arbitrary probability distributions for reward, then the number of potential MDPs is infinite. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You have made things a lot harder for yourself by trying a policy gradient method with a neural network function approximator. Although this probably can be made to solve your problem, it is an advanced technique, and also known to be unstable and hard to train. It might be a good choice when the relationship between the state, transitions and rewards was very complex, and there were more states possible with &lt;em&gt;meaningful similarity&lt;/em&gt; between them (in a numerical/statistical sense, not in terms of analogy, the neural network still won't find the analogy with switches and numbered bits). Even then, it will learn slowly and cautiously, needing to repeat state/action pairs multiple times in order to learn that the transitions and reward are reliable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless there is some good reason for you to stick with a policy gradient method, I suggest using a tabular algorithm (i.e. no function approximation, just a table of action value estimates) and something like single step Q-Learning. That has the advantage that because you know the algorithm is deterministic, you can set a high learning rate and it will remain stable. In fact Q-learning will probably learn an optimal policy in much less than 10,000 episodes. However, initially it will learn just one of the many possible optimal policies and tend to stick with it  - i.e. some permutation of taking each action 0 to 9, once each, but always in the same order because it has learned that is safe.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-19T12:50:41.330" LastActivityDate="2017-09-19T12:50:41.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23139" PostTypeId="2" ParentId="23136" CreationDate="2017-09-19T13:25:58.090" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No. You have taken multiple measurements, each with some uncertainty, and chosen the maximum or minimum value.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;How can I bound the true loss using $\hat{L}(h_*,V)$ ?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the general case you cannot. It will depend on how much over-fitting is occurring within the model on the training set, size of cv set, amount of times it has been used, and how similar the model's performance was on each use. There is also sampling bias in the cv set, and that interacts with the selection process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is generally done if you need unbiased estimate at the end of production is a train/cv/test split. The cv set is used for model selection, and once you have a single model selected, you estimate its loss - or other key metric - on the test set. It is important to use the test set minimally and not in order to select models, if you want it to be an unbiased measure. Otherwise you repeat the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach which maintains confidence in cv-based metrics is to use k-fold cross validation. Taking taking the mean of a metric in k-fold cross validation is still biased once you have used it a few times, but the bias is reduced somewhat. You can take that idea further with &lt;em&gt;nested&lt;/em&gt; cross-validation, which allows you to get an unbiased estimate of model performance in a general fashion (i.e. using the same hyper-parameters) from more of your data.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-22T06:56:16.163" LastActivityDate="2017-09-22T06:56:16.163" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23143" PostTypeId="1" CreationDate="2017-09-19T15:43:36.133" Score="6" ViewCount="19656" Body="&lt;p&gt;I want to predict count data. In my understanding both standard classification and regression are not well suited for this. A poisson or binomial regression algorithm seems to do the trick. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am used to doing most of my ML tasks in sklearn. But on this topic I could not find an implementation. Are there any suitable options within the python universe for this? &lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29575" LastEditDate="2018-10-12T14:38:42.660" LastActivityDate="2022-06-28T11:23:43.230" Title="Poisson regression options in python" Tags="|scikit-learn|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23144" PostTypeId="2" ParentId="23143" CreationDate="2017-09-19T16:28:51.117" Score="5" Body="&lt;p&gt;&lt;code&gt;statsmodels&lt;/code&gt; has you covered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There aren't a lot of great examples of Poisson regression in the statsmodels API, but if you're happy with GLMs, statsmodels has a &lt;a href=&quot;http://www.statsmodels.org/dev/glm.html&quot; rel=&quot;noreferrer&quot;&gt;GLM API&lt;/a&gt; which lets you specify any single-parameter distribution, including Poisson.&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-09-19T16:28:51.117" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23145" PostTypeId="2" ParentId="23132" CreationDate="2017-09-19T17:47:28.413" Score="4" Body="&lt;p&gt;&lt;code&gt;C1&lt;/code&gt; and &lt;code&gt;C2&lt;/code&gt; are coefficients for L1 and L2 regularization, respectively. You can use the same definition of &lt;code&gt;params_space&lt;/code&gt; regardless of the optimization algorithm.&lt;/p&gt;&#xA;" OwnerUserId="33600" LastActivityDate="2017-09-19T17:47:28.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23146" PostTypeId="1" CreationDate="2017-09-19T18:10:30.693" Score="0" ViewCount="457" Body="&lt;p&gt;I would need to know what this widget does. What architecture does it use? What dataset has it been trained on? And so on.&lt;/p&gt;&#xA;" OwnerUserId="39440" LastEditorUserId="23305" LastEditDate="2017-09-19T18:50:02.097" LastActivityDate="2017-11-18T23:53:11.410" Title="What does the Orange widget &quot;Image Embedding&quot; do exactly?" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23148" PostTypeId="2" ParentId="761" CreationDate="2017-09-19T18:52:43.103" Score="6" Body="&lt;p&gt;You can use &lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;HDBSCAN&lt;/a&gt; for this.  The python package has support for haversine distance which will properly compute distances between lat/lon points.&lt;/p&gt;&#xA;&lt;p&gt;As the &lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/faq.html?highlight=lat%20lon%27#q-haversine-metric-is-not-clustering-my-lat-lon-data-correctly&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs mention&lt;/a&gt;, you will need to convert your points to radians first for this to work.  The following psuedocode should do the trick:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;points = np.array([[lat1, lon1], [lat2, lon2], ...])&#xA;rads = np.radians(points)&#xA;clusterer = hdbscan.HDBSCAN(min_cluster_size=N, metric='haversine')&#xA;cluster_labels = clusterer.fit_predict(rads)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39442" LastEditorUserId="86641" LastEditDate="2023-01-12T18:50:07.947" LastActivityDate="2023-01-12T18:50:07.947" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="23150" PostTypeId="1" CreationDate="2017-09-19T20:37:59.963" Score="1" ViewCount="573" Body="&lt;p&gt;I have a click-stream data of users regarding their behaviour in a platform. There are activities provided in a temporal order and users can go back and forth between these activities. Below is an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WjAIF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WjAIF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, there are 3 activities in a temporal order. Blue arrows represent one user and the green arrows represent another user. As you can see, these two users followed a different path.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my scenario I have over 500 users , which are divided into 2 categories (let's Group A and Group B), and more than 50 activities. I want to identify the common patterns for Group A and Group B users and make a comparison among them (I am hoping to see patterns distinct to each user group).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if there are some useful (free) tools out there that I can take advantage of. I prefer Python but I could not find a python library for this. I am also looking for a tool (or python library) to visualise the paths that users follow. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="28915" LastActivityDate="2017-09-20T06:04:06.447" Title="Using python to identify common paths that users follow among different activities" Tags="|python|visualization|sequential-pattern-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23154" PostTypeId="2" ParentId="23146" CreationDate="2017-09-19T23:12:32.663" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;https://github.com/biolab/orange3-imageanalytics/blob/001b8f1e01d39a484a11705dcaef3fe4e47d9fe0/orangecontrib/imageanalytics/image_embedder.py#L25-L39&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt; reveals it's likely Google's &lt;a href=&quot;https://www.kaggle.com/google-brain/inception-v3&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inception V3&lt;/a&gt; model.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-09-19T23:12:32.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23156" PostTypeId="1" AcceptedAnswerId="23162" CreationDate="2017-09-20T01:56:33.853" Score="3" ViewCount="664" Body="&lt;p&gt;I am training an RL agent on a time series (with TensorFlow in python) in the following way: to predict the quantity of interest at time period $t$, I feed a window of $W$ observations at time periods $[t-W,t)$. Throughout the training the window advances step by step until I have a minibatch of $M$ observations and rewards to train on. Repeat until you run out of historical data, that's one Epoch. I train on a few thousand epochs with small learning rate (the loss is very unstable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eventually, I want to start pulling live data from the environment to make predictions.&#xA;At this point, if I wanted to continue the training online, how should I deal with the epochs? There is no &quot;running out of data&quot; anymore.&lt;/p&gt;&#xA;" OwnerUserId="39081" LastActivityDate="2017-09-20T09:54:52.500" Title="How to transition between offline and online learning?" Tags="|python|tensorflow|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23157" PostTypeId="2" ParentId="12532" CreationDate="2017-09-20T02:54:31.833" Score="8" Body="&lt;p&gt;Oddly enough, I found that larger batch sizes with keras require more epochs to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the output of &lt;a href=&quot;https://github.com/shadiakiki1986/test-ml/blob/master/t4-LSTM%20stateful/test_temporal_regression.py&quot; rel=&quot;noreferrer&quot;&gt;this script&lt;/a&gt; based on keras' &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/tests/integration_tests/test_temporal_data_tasks.py#L78&quot; rel=&quot;noreferrer&quot;&gt;integration test&lt;/a&gt; is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs 15   , batch size 16   , layer type Dense: final loss 0.56, seconds 1.46&#xA;epochs 15   , batch size 160  , layer type Dense: final loss 1.27, seconds 0.30&#xA;epochs 150  , batch size 160  , layer type Dense: final loss 0.55, seconds 1.74&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Related&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Keras &lt;a href=&quot;https://github.com/fchollet/keras/issues/4708&quot; rel=&quot;noreferrer&quot;&gt;issue 4708&lt;/a&gt;: the user turned out to be using &lt;code&gt;BatchNormalization&lt;/code&gt;, which affected the results.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://machinelearningmastery.com/tune-lstm-hyperparameters-keras-time-series-forecasting/&quot; rel=&quot;noreferrer&quot;&gt;This tutorial&lt;/a&gt; on LSTM, section &lt;strong&gt;Tuning the Batch Size&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;search results for &lt;a href=&quot;https://duckduckgo.com/?q=keras+lstm+batch+size+effect+on+result&amp;amp;t=canonical&amp;amp;ia=web&quot; rel=&quot;noreferrer&quot;&gt;keras lstm batch size effect on result&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://theorangeduck.com/page/neural-network-not-working?imm_mid=0f6562&amp;amp;cmp=em-data-na-na-newsltr_20170920&quot; rel=&quot;noreferrer&quot;&gt;My Neural Network isn't working! What should I do?&lt;/a&gt;, point 5 (You Used a too Large Batch Size) discusses exactly this&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Using too large a batch size can have a negative effect on the&#xA;  accuracy of your network during training since it reduces the&#xA;  stochasticity of the gradient descent.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Edit: most of the times, increasing &lt;code&gt;batch_size&lt;/code&gt; is desired to speed up computation, but there are other simpler ways to do this, like using data types of a smaller footprint via the &lt;code&gt;dtype&lt;/code&gt; argument, whether in &lt;em&gt;keras&lt;/em&gt; or &lt;em&gt;tensorflow&lt;/em&gt;, e.g. &lt;code&gt;float32&lt;/code&gt; instead of &lt;code&gt;float64&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="35596" LastEditorUserId="35596" LastEditDate="2018-04-16T06:20:10.523" LastActivityDate="2018-04-16T06:20:10.523" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23158" PostTypeId="1" CreationDate="2017-09-20T04:57:55.693" Score="0" ViewCount="2780" Body="&lt;p&gt;I got call from a recruiter for an elasticsearch job. I am not sure how these 2 are related. Can anyone suggest and provide example use case?&lt;/p&gt;&#xA;" OwnerUserId="13100" LastActivityDate="2017-09-20T16:48:00.743" Title="How Natural language processing and elasticsearch are related" Tags="|nlp|beginner|career|self-study|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23159" PostTypeId="1" AcceptedAnswerId="24112" CreationDate="2017-09-20T05:53:18.477" Score="67" ViewCount="30325" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/17187507/why-use-softmax-as-opposed-to-standard-normalization&quot;&gt;Why use softmax as opposed to standard normalization?&lt;/a&gt; In the comment area of the top answer of this question, @Kilian Batzner raised 2 questions which also confuse me a lot. It seems no one gives an explanation except numerical benefits.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;I get the reasons for using Cross-Entropy Loss, but how does that relate to the softmax? You said &amp;quot;the softmax function can be seen as trying to minimize the cross-entropy between the predictions and the truth&amp;quot;. Suppose, I would use standard / linear normalization, but still use the Cross-Entropy Loss. Then I would also try to minimize the Cross-Entropy. So how is the softmax linked to the Cross-Entropy except for the numerical benefits?&lt;/p&gt;&#xA;&lt;p&gt;As for the probabilistic view: what is the motivation for looking at log probabilities? The reasoning seems to be a bit like &amp;quot;We use e^x in the softmax, because we interpret x as log-probabilties&amp;quot;. With the same reasoning we could say, we use e^e^e^x in the softmax, because we interpret x as log-log-log-probabilities (Exaggerating here, of course). I get the numerical benefits of softmax, but what is the theoretical motivation for using it?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="39461" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-10-11T12:56:08.243" Title="In softmax classifier, why use exp function to do normalization?" Tags="|machine-learning|deep-learning|" AnswerCount="5" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23160" PostTypeId="2" ParentId="23150" CreationDate="2017-09-20T06:04:06.447" Score="1" Body="&lt;p&gt;You can rank the paths by frequency using the database (SQL), constraining the start-, end- points as necessary once you fix the length of the window. If you let the path length be variable then you will not be able to do it all in SQL. In that case you can learn the transition probabilities between states then solve a weighted shortest path problem, where the distances are log likelihoods. Or you can use a heuristic like A* search. I do not know of any library that will do all this since it is very ad hoc, but you can do the visualization in &lt;a href=&quot;https://stackoverflow.com/questions/20133479/how-to-draw-directed-graphs-using-networkx-in-python&quot;&gt;networkx&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-09-20T06:04:06.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23161" PostTypeId="2" ParentId="12532" CreationDate="2017-09-20T06:32:53.973" Score="19" Body="&lt;p&gt;I feel the accepted answer is possibly wrong. There are variants in &lt;em&gt;Gradient Descent Algorithms&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Vanilla Gradient Descent&lt;/strong&gt;: Here the Gradient is being calculated on all the data points at a single shot and the average is taken. Hence we have a smoother version of the gradient takes longer time to learn. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Stochastic Gradient Descent&lt;/strong&gt; : Here one-data point at a time hence the gradient is aggressive (noisy gradients) hence there is going to be lot of oscillations ( we use Momentum parameters - e.g Nesterov to control this). So there is a chance that your oscillations can make the algorithm not reach a local minimum.(diverge).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Mini-Batch Gradient Descent&lt;/strong&gt;: Which takes the perks of both the previous ones averages gradients of a small batch. Hence not too aggressive like SGD and allows Online Learning which Vanilla GD never allowed. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The smaller the Mini-Batch the better would be the performance of your model (not always) and of course it has got to do with your epochs too faster learning. If you are training on large dataset you want faster convergence with good performance hence we pick Batch-GD's. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SGD had fixed learning parameter hence we start other Adaptive Optimizers like Adam, AdaDelta, RMS Prop etc which changes the learning parameter based on the history of Gradients. &lt;/p&gt;&#xA;" OwnerUserId="29644" LastEditorUserId="35596" LastEditDate="2017-09-20T10:03:17.783" LastActivityDate="2017-09-20T10:03:17.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23162" PostTypeId="2" ParentId="23156" CreationDate="2017-09-20T07:10:56.193" Score="1" Body="&lt;p&gt;Although RL algorithms can be run online, in practice this is not stable when learning off policy (as in Q-learning) and with a function approximator. To avoid this, new experience can be added to history and the agent learn from the history (called experience replay). You could think of this as a semi-online approach, since new data is immediately available to learn from, but depending on batch sizes and history size, it might not be used to alter parameters for a few time steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically in RL systems like DQN, you would train for some randomly sampled batch between each action, out of some window of historical data (maybe all historical data, maybe last N steps). The amount of training you perform between actions is a hyper-parameter of your model, as is any sampling bias towards newer data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example in the &lt;a href=&quot;https://arxiv.org/pdf/1312.5602.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Atari game-playing paper by Deep Mind team&lt;/a&gt;, the agent sampled a mini-batch with 32 observations (state, action, reward, next state) to train the neural network on, in between each action, whilst playing the game online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept of an epoch does not occur in online learning. If you are using each epoch to report performance metrics, and want to continue using comparable numbers, then you can pick some number of training iterations instead. There are no fixed rules for this, but you might want to consider reporting the same statistics on similar number of iterations as you trained on historical data - e.g. if you had 10,000 training samples in your history and are now training online with a mini-batch size of 50 per time step, then report progress every 10,000/50 = 200 time steps. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-20T09:54:52.500" LastActivityDate="2017-09-20T09:54:52.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23164" PostTypeId="1" AcceptedAnswerId="23167" CreationDate="2017-09-20T10:00:44.910" Score="0" ViewCount="435" Body="&lt;p&gt;&lt;strong&gt;Introduction:&lt;/strong&gt; Lately I've been looking into different machine learning methods to work around different business problems. By now I have a good, basic understanding of most regression and classification methods, and I'm able to use these methods to predict numeric values given other numeric values and/or simple categories (e.g. an employee's salary given age, years of experience and level of education) or a binary classification (e.g. will this employee leave the company based on the same variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I'm looking for:&lt;/strong&gt; However, I haven't found the right method for the problem I initially wanted to solve, which involves predicting a non-numeric, non-binary value from a mix of numeric and categorical data. I'm not looking for an in-depth explanation of how to solve the exact problem, but merely advise on which techniques/methods to look into. Ideally something that could be done with R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The business problem:&lt;/strong&gt; I have historical data on public tenders (i.e. public sector instutions buying goods/services from private contractors through calls for tenders). The data includes variables like: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Orderer&lt;/strong&gt; - i.e. who announced the tender (1 of ~150 municipalities/state insitutions) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Type of procurement&lt;/strong&gt; (1 or more of thousands of industrial classification codes) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Estimated value of contract&lt;/strong&gt; - A numeric value estimating the value of the contract (at a point before the winner is chosen).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Winner&lt;/strong&gt; - i.e. which contractor won the tender (1 of ~2000 private &#xA;companies)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I want to do is predict the winner of a tender given the three other variables. It's obviously not a regression problem, and the classification methods I know seem inadequate in handling the problem too. The data are clean and streamlined (no alternate spelling of the different orderer/contractor names, etc.). Any ideas about what to look into? &lt;/p&gt;&#xA;" OwnerUserId="39465" LastActivityDate="2017-09-20T14:13:51.843" Title="Method for predicting winner of call for tenders" Tags="|machine-learning|r|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23165" PostTypeId="1" AcceptedAnswerId="23166" CreationDate="2017-09-20T10:55:50.403" Score="1" ViewCount="12400" Body="&lt;p&gt;Recently, we were taught K-means clustering. I understood the basic idea of the algorithm and successfully implemented it for data with a single dimensional. Now we are told to implement it for two dimensional data. As far as I understood x and y can be two attributes of a dataset but our professor said otherwise. She said that the we have to determine x and y of an attribute in the data set to cluster the data. She used a simple 2D matrix as an example. This has got me confused. How can one determine x and y of an atttribute? Row number and column number seems silly to me to use it for distance calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is how does one determine x and y for 2-D k means clustering?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per this &lt;a href=&quot;https://datascience.stackexchange.com/questions/13200/k-means-clustering-algorithm-problems&quot;&gt;question&lt;/a&gt; the two attributes (weight and height) are used as x and y. Is this correct?  &lt;/p&gt;&#xA;" OwnerUserId="39468" LastEditorUserId="30212" LastEditDate="2017-09-20T12:41:43.557" LastActivityDate="2019-06-18T11:23:26.650" Title="How to determine x and y in 2 dimensional K-means clustering?" Tags="|clustering|k-means|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23166" PostTypeId="2" ParentId="23165" CreationDate="2017-09-20T12:03:29.837" Score="2" Body="&lt;p&gt;I see no problem with the example of clustering on two numerical attributes like height and weight like in the example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing I can think of is somewhere lost in translation, your professor was trying to explain the concept of reducing many dimensions (attributes) to two and then clustering on those derived dimensions. That is a common technique for trying to visually spot clusters in a complex data set.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-20T12:03:29.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23167" PostTypeId="2" ParentId="23164" CreationDate="2017-09-20T12:10:15.147" Score="2" Body="&lt;p&gt;Having the classifier try to predict one of 2000 possible values is going to be tough. A common approach is to either bucket the possible targets/labels or decompose the problem. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Instead of trying to predict the exact winning company, group the winners into similar groups and predict the group. For example, predict if the winner will be a large public company, small/medium business, or an independent consultant.&lt;/li&gt;&#xA;&lt;li&gt;The nature of procurement probably varies across the buyer and what is being bought. A model that is good at predicting which company will win the business of a Fortune 500 company probably has a different structure than who is good at winning the business of a small city. In a similar fashion, the competition for who is going to build a bridge is going to be different and not include the companies who will bid on implementing a new website. Partition your data into similar competitions and then try to predict the results. This will hopefully have the side effect of reducing the number of potential targets. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="30212" LastEditDate="2017-09-20T14:13:51.843" LastActivityDate="2017-09-20T14:13:51.843" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23169" PostTypeId="2" ParentId="22430" CreationDate="2017-09-20T14:13:58.413" Score="2" Body="&lt;p&gt;Machine learning is the process of automatically discovering (inductively) the formula/rule that models a certain random variable. If you can discover the rule by yourself, you don't need the machine to aid you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You (or your boss), as the domain expert, has written a rule to model the target class. As long as this rule is deemed good enough for the business, there's no need to spend resources to generate another model.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-09-20T14:13:58.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23170" PostTypeId="2" ParentId="23129" CreationDate="2017-09-20T14:17:09.900" Score="6" Body="&lt;p&gt;Here are some excerpts from &lt;a href=&quot;http://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf&quot; rel=&quot;noreferrer&quot;&gt;the original paper&lt;/a&gt; that I think are key to understanding the question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Instead of using an own model parameter for each interaction,&#xA;  the FM models the interaction by factorizing it. We will see later on,&#xA;  that this is the key point which allows high quality parameter&#xA;  estimates of higher-order interactions under sparsity.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;blockquote&gt;&#xA;  &lt;p&gt;Factorization machines can estimate interactions even in these&#xA;  settings (&lt;strong&gt;sparse data&lt;/strong&gt;) well because they break the independence of the interaction&#xA;  parameters by factorizing them. In general this means that the data&#xA;  for one interaction helps also to estimate the parameters for related&#xA;  interactions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In other words, instead of fitting an independent parameter for every second order interaction, it &lt;em&gt;factorizes&lt;/em&gt; the parameters which reduces the parameter space and the model complexity, and thus making the model less prone to over fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps understand some aspects of the question.&lt;/p&gt;&#xA;" OwnerUserId="18249" LastActivityDate="2017-09-20T14:17:09.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23173" PostTypeId="5" CreationDate="2017-09-20T15:17:53.797" Score="0" Body="&lt;p&gt;Principal component analysis (PCA) is a statistical technique for dimension reduction often used in clustering or factor analysis.  Given any number of explanatory or causal variables, PCA ranks the variables by their ability to explain greatest variation in the data.  It is this property that allows PCA to be used for dimension reduction, i.e. to identify the most important variables from amongst a large set possible influences.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically, principal component analysis (PCA) amounts to an orthogonal transformation of possibly correlated variables (vectors) into uncorrelated variables called principal component vectors.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-12-26T04:14:50.720" LastActivityDate="2017-12-26T04:14:50.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23174" PostTypeId="4" CreationDate="2017-09-20T15:17:53.797" Score="0" Body="Principal component analysis, a technique for dimensionality reduction." OwnerUserId="13748" LastEditorUserId="13748" LastEditDate="2017-09-20T18:10:11.340" LastActivityDate="2017-09-20T18:10:11.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23175" PostTypeId="5" CreationDate="2017-09-20T15:20:33.420" Score="0" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE algorithm&lt;/a&gt; is a nonlinear dimensionality reduction method, particularly efficient for transforming high-dimensional data into two or three dimension spaces.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastEditorUserId="13748" LastEditDate="2017-09-20T18:10:08.790" LastActivityDate="2017-09-20T18:10:08.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23176" PostTypeId="4" CreationDate="2017-09-20T15:20:33.420" Score="0" Body="t-SNE (t-distributed stochastic neighbor embedding) is a technique for dimensionality reduction." OwnerUserId="13748" LastEditorUserId="13748" LastEditDate="2017-09-20T18:10:06.303" LastActivityDate="2017-09-20T18:10:06.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23178" PostTypeId="2" ParentId="22536" CreationDate="2017-09-20T15:58:45.427" Score="2" Body="&lt;p&gt;Using the python package &lt;a href=&quot;https://github.com/seatgeek/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fuzzy Wuzzy&lt;/a&gt; is also useful&lt;/p&gt;&#xA;&lt;p&gt;It uses Levenshtein distance from the python-Levenshtein package, and gives you different options for re-arranging or using word tokens.&lt;/p&gt;&#xA;" OwnerUserId="9299" LastEditorUserId="98307" LastEditDate="2020-08-07T09:29:07.483" LastActivityDate="2020-08-07T09:29:07.483" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23179" PostTypeId="2" ParentId="23158" CreationDate="2017-09-20T16:48:00.743" Score="4" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Natural_language_processing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Natural language processing (NLP)&lt;/a&gt; is a field of computer science, artificial intelligence, and computational linguistics.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Elasticsearch&quot; rel=&quot;nofollow noreferrer&quot;&gt;ElasticSearch&lt;/a&gt; is a search engine based on Lucene.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;ElasticSearch uses parts of NLP (e.g., tokenization and stemming). ElasticSearch also includes software engineering elements to ensure a search solution is performant.&lt;/p&gt;&#xA;&lt;p&gt;Since NLP is entire discipline, there are many parts of NLP that are not related to ElasticSearch.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-09-20T16:48:00.743" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23180" PostTypeId="2" ParentId="22984" CreationDate="2017-09-20T16:53:09.307" Score="0" Body="&lt;p&gt;Model confidence is domain specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus you can manually set a threshold. For example, if P &amp;lt; .65 then manually classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you can train a second machine learning system to learn what level of confidence is required for the specific task. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-20T16:53:09.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23181" PostTypeId="2" ParentId="22306" CreationDate="2017-09-20T16:59:06.353" Score="0" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/mre/receipt-parser&quot; rel=&quot;nofollow noreferrer&quot;&gt;receipt-parser&lt;/a&gt; is receipt parser package in Python built on tesseract.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-20T16:59:06.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23182" PostTypeId="2" ParentId="22301" CreationDate="2017-09-20T17:05:30.600" Score="1" Body="&lt;p&gt;Tokenization, like all preprocessing, is application specific. It depends on the end goal. The most difficult (and interesting) parts of tweets are usernames, hashtags, URLs, and emoticons. Different applications will model these elements differently. Thus the need for different tokenizers with different parameters.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-20T17:05:30.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23183" PostTypeId="1" AcceptedAnswerId="23186" CreationDate="2017-09-20T17:53:58.877" Score="28" ViewCount="31731" Body="&lt;p&gt;If we have a look to 90-99% of the papers published using a CNN (ConvNet).&#xA;The vast majority of them use filter size of &lt;strong&gt;odd numbers&lt;/strong&gt;:{1, 3, 5, 7} for the most used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This situation can lead to some problem: With these filter sizes, usually the convolution operation is not perfect with a padding of 2 (common padding) and some edges of the input_field get lost in the process...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question1:&lt;/strong&gt; Why using only odd_numbers for convolutions filter sizes ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question2:&lt;/strong&gt; Is it actually a problem to omit a small part of the input_field during the convolution ? Why so/not ?&lt;/p&gt;&#xA;" OwnerUserId="39198" LastEditorUserId="131150" LastEditDate="2022-02-12T21:05:43.167" LastActivityDate="2022-02-12T21:05:43.167" Title="Why convolutions always use odd-numbers as filter size" Tags="|deep-learning|computer-vision|convolutional-neural-network|convolution|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23184" PostTypeId="2" ParentId="23183" CreationDate="2017-09-20T18:05:57.383" Score="5" Body="&lt;p&gt;1) Suppose &lt;code&gt;input_field&lt;/code&gt; is all zero except for one entry at index &lt;code&gt;idx&lt;/code&gt;. An odd filter size will return data with a peak centered around &lt;code&gt;idx&lt;/code&gt;, an even filter size won't - consider the case of a uniform filter with size 2. Most people want to preserve the locations of peaks when they filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) All of the &lt;code&gt;input_field&lt;/code&gt; is relevant for the convolution, but the edges of &lt;code&gt;output_field&lt;/code&gt; cannot be computed accurately since the necessary data is not contained in &lt;code&gt;input_field&lt;/code&gt;. If I want to compute an answer for the first element of &lt;code&gt;output_field&lt;/code&gt;, the filter must be centered on the first element of &lt;code&gt;input_field&lt;/code&gt;. But then there are filter elements that don't correspond to any available element of &lt;code&gt;input_field&lt;/code&gt;. There are various tricks to get a guess for the edges of &lt;code&gt;output_field&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39007" LastActivityDate="2017-09-20T18:05:57.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23185" PostTypeId="1" AcceptedAnswerId="23187" CreationDate="2017-09-20T18:22:07.657" Score="1" ViewCount="115" Body="&lt;p&gt;I have a set of sales transaction data containing more than 1 item purchased. Every item sold has a category. I would like figure out which categories are most commonly ordered together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is more or less like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Transaction ID|Item ID | Sales Quantity|Item Category&#xA;1              Apple     1              Fruit&#xA;1              Banana    1              Fruit&#xA;1              Carrot    2              Vegetable&#xA;2              Carrot    1              Vegetable&#xA;2              Ice Cream 2              Dessert&#xA;3              Squash    2              Vegetable&#xA;3              Chocolate 2              Dessert&#xA;4              Apple     1              Fruit&#xA;4              Carrot    1              Vegetable&#xA;4              Doughnut  1              Dessert&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Just eyeballing above you can see that there are a high amount of vegetable-dessert pairings on the same transaction. But now imagine that we have 250,000+ transactions in the data set and dozens of categories. I'm looking to discover cross category sales only. Not interested in Apples and Bananas (Fruit-Fruit) pairs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I can teach myself how to code the analysis, but I'm just not sure what this is called or what to Google. Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="39483" LastEditorUserId="13748" LastEditDate="2017-09-21T07:39:27.363" LastActivityDate="2017-09-21T07:39:27.363" Title="Discovering cross category sales using transactions history (Clustering?)" Tags="|r|data-mining|clustering|excel|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23186" PostTypeId="2" ParentId="23183" CreationDate="2017-09-20T19:03:57.153" Score="24" Body="&lt;p&gt;The convolution operation, simply put, is combination of element-wise product of two matrices. So long as these two matrices agree in dimensions, there shouldn't be a problem, and so I can understand the motivation behind your query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A.1. However, the intent of convolution is to encode source data matrix (entire image) in terms of a filter or kernel. More specifically, we are trying to encode the pixels in the neighborhood of anchor/source pixels. Have a look at the figure below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/YDusp.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YDusp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Typically, we consider every pixel of the source image as anchor/source pixel, but we are not constrained to do this. In fact, it is not uncommon to include a stride, where in we anchor/source pixels are separated by a specific number of pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay, so what is the source pixel? It is the anchor point at which the kernel is centered and we are encoding all the neighboring pixels, including the anchor/source pixel. Since, the kernel is symmetrically shaped (not symmetric in kernel values), there are equal number (n) of pixel on all sides (4- connectivity) of the anchor pixel. Therefore, whatever this number of pixels maybe, the length of each side of our symmetrically shaped kernel is 2*n+1 (each side of the anchor + the anchor pixel), and therefore filter/kernels are always odd sized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if we decided to break with 'tradition' and used asymmetric kernels? You'd suffer aliasing errors, and so we don't do it. We consider the pixel to be the smallest entity, i.e. there is no sub-pixel concept here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A.2 &#xA;The boundary problem is dealt with using different approaches: some ignore it, some zero pad it, some mirror reflect it. If you are not going to compute an inverse operation, i.e. deconvolution, and are not interested in perfect reconstruction of original image, then you don't care about either loss of information or injection of noise due to the boundary problem. Typically, the pooling operation (average pooling or max pooling) will remove your boundary artifacts anyway. So, feel free to ignore part of your 'input field', your pooling operation will do so for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Zen of convolution: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the old-school signal processing domain, when an input signal was convolved or passed through a filter, there was no way of judging a-prior which components of the convolved/filtered response were relevant/informative and which were not. Consequently, the aim was to preserve signal components (all of it) in these transformations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These signal components are information. Some components are more informative than others. The only reason for this is that we are interested in extracting higher-level information; Information pertinent towards some semantic classes. Accordingly, those signal components that do not provide the information we are specifically interested in can be pruned out. Therefore, unlike old-school dogmas about convolution/filtering, we are free to pool/prune the convolution response as we feel like. The way we feel like doing so is to rigorously remove all data components that are not contributing towards improving our statistical model.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastEditorUserId="2515" LastEditDate="2017-09-21T19:02:20.377" LastActivityDate="2017-09-21T19:02:20.377" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="23187" PostTypeId="2" ParentId="23185" CreationDate="2017-09-20T19:13:52.980" Score="1" Body="&lt;p&gt;Note that your data can be re-ordered to look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Transaction ID | Items&#xA;1                {Apple, Banana, Carrot}&#xA;2                {Carrot, Ice Cream}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This kind of data set is trivial for &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;association rule mining&lt;/a&gt;. A very simple and well-known algorithm of this kind is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apriori&lt;/a&gt;. I'm certain there are packages for executing this algorithm in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the restriction of &quot;discover cross category sales only&quot;, you can just post-prune the generated rules, ie. let the algorithm generated inter-category sales and then remove those later, which should be trivial.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-09-20T19:13:52.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23188" PostTypeId="2" ParentId="23185" CreationDate="2017-09-20T20:02:35.790" Score="0" Body="&lt;p&gt;Based on @Mephy's answer, I wanted to illustrate a little bit more. Association rule mining, which is also called 'Market Basket Analysis'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on your question, Googling market basket analysis will give you more results and there are &lt;a href=&quot;http://www.salemmarafi.com/code/market-basket-analysis-with-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;many&lt;/a&gt; examples and posts on getting started in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example &lt;code&gt;arules&lt;/code&gt; is a great package for this type of analysis in R. Good luck!&lt;/p&gt;&#xA;" OwnerUserId="34028" LastActivityDate="2017-09-20T20:02:35.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23189" PostTypeId="2" ParentId="23165" CreationDate="2017-09-20T20:20:35.783" Score="1" Body="&lt;p&gt;K-Means is an algorithm primarily used to find clusters in higher dimensional datasets. I have never come across K-Means being used for single dimensional datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, a 2D K-Means (or even higher dimensional) works this way:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Decide the number of clusters you want (k value).&lt;/li&gt;&#xA;&lt;li&gt;Initialize k number of random 2D (or higher dimensional) points (centroids).&lt;/li&gt;&#xA;&lt;li&gt;Each point in your dataset gets assigned the centroid it lies closest to.&lt;/li&gt;&#xA;&lt;li&gt;The centroid positions then gets updated to the average of all points that were assigned the said centroid. Repeat from step 3 till stopping condition reached.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And for your second question, yes, that is a correct method, and the answer to that question is correctly explained.&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-09-20T20:20:35.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23191" PostTypeId="2" ParentId="14222" CreationDate="2017-09-20T22:57:01.780" Score="0" Body="&lt;p&gt;You might also incorporate TF-IDF to see if there are common word frequencies with elements you intend to extract, or with those elements that you could remove.&lt;/p&gt;&#xA;" OwnerUserId="39488" LastActivityDate="2017-09-20T22:57:01.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23192" PostTypeId="2" ParentId="5893" CreationDate="2017-09-20T23:41:42.930" Score="1" Body="&lt;p&gt;An approach I have used to build a stopword list is to build and train a logistic regression model (due to its interpretability) on your text data. Take the absolute value of the coefficients for each token. Then, sort descending the absolute value of the coefficients of the tokens. Then, create a list of all the tokens with high coefficient absolute value that might lead to overfitting or that might meet some other criteria to be a stopword. That list is your stopwords list. You can then apply that stopword list to another set of documents of this type (kind of like a test set), to see if removing them increases the accuracy, precision, or recall of the test set model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This strategy is effective because it takes into account the impact of tokens when building a stopword list.&lt;/p&gt;&#xA;" OwnerUserId="39488" LastActivityDate="2017-09-20T23:41:42.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23193" PostTypeId="1" CreationDate="2017-09-21T00:58:55.183" Score="2" ViewCount="144" Body="&lt;p&gt;I have a simple game I'm building for fun, just to see how well ML can work with simple data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically it's just a game where it has turns that go like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Computer generates a random number $x$, and does not show the player.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Player wagers that they can guess a number lower than $x$. Call the wager amount $w$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Player tries to guess a low number $g$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $g \lt x$, then player gains $wg$ points.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $g \ge x$, then player loses their wager, $w$ points.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If player has funds $f_t$ at start of turn $t$ then another way to put this is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_{t+1} = \begin{cases}&#xA;    f_t + wg,&amp;amp; \text{if } g\le x\\&#xA;    f_t - w,              &amp;amp; \text{otherwise}&#xA;\end{cases}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of play:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Start with $f=50$&lt;/li&gt;&#xA;&lt;li&gt;Turn 1, $x=11$, player wagers $w=8$ and guesses $g=9$. Player gains $+72$ points, so $f=122$ at the end of the turn.&lt;/li&gt;&#xA;&lt;li&gt;Turn 2, $x=10.5, w=4, g=7$, points change by $+28$, $f=150$.&lt;/li&gt;&#xA;&lt;li&gt;Turn 3, $x=20, w=1, g=6$, points change by $+6$, $f=156$.&lt;/li&gt;&#xA;&lt;li&gt;Turn 4, $x=2, w=10, g=15$, points change by $-10$, $f=146$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I wanted to use ML to try and predict this by feeding it pre-generated turns to see if it can find any patterns. There are only a few variables involved, so I figured it shouldn't be overly complicated. Ideally I would like for the ML to learn how to play the game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering what type of ML is applicable to this kind of problem? It is not clear to me where to start, even though I have made the game simple. I have more complex games that I made in the past that I would like to try to apply ML to as well. &lt;/p&gt;&#xA;" OwnerUserId="39490" LastEditorUserId="836" LastEditDate="2017-09-22T19:03:41.263" LastActivityDate="2017-09-22T20:58:19.297" Title="ML that learns to predict and play a simple wagering game" Tags="|machine-learning|python|game|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23194" PostTypeId="1" CreationDate="2017-09-21T05:36:24.053" Score="5" ViewCount="220" Body="&lt;p&gt;I'm learning about the &lt;a href=&quot;https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;restricted Boltzmann machine&lt;/a&gt;, but I can't decide whether the hidden vectors of RBM are useless or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following are my two understandings for the role hidden vectors play. Have I misunderstood something? If not, then how could the role those vectors play  be so different?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;This is the first understanding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I learned from this video &lt;a href=&quot;https://youtu.be/NkG5HxPdjuY&quot; rel=&quot;nofollow noreferrer&quot;&gt;RBM&lt;/a&gt; that hidden units are just structural support and we don't care about what those hidden vectors really are.&lt;/strong&gt; We introduce hidden units into RBM just for it to gain more expressive power for probability distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have only one image, and I transform this image to a binary vector and feed this vector into a RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;(my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;My second understanding goes like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a RBM, each hidden units of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by this hidden unit(by the weights and bias). So, if we input an image into this RBM, what the RBM will do is to project this input vector onto those hyperplanes defined by all the hidden units. Thus for an input vector, the corresponding hidden vector is very important, it is some abstract representation. And we can further feed this representation into other models for doing classification.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, these are my understandings, if you can answer the question by explaining how RBM is used for MNIST that would be extremely helpful for me.&lt;/p&gt;&#xA;" OwnerUserId="39501" LastEditorUserId="29575" LastEditDate="2017-09-25T15:14:43.900" LastActivityDate="2018-04-08T20:03:37.577" Title="What is the role hidden vectors play in RBM?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23196" PostTypeId="1" CreationDate="2017-09-21T07:39:46.310" Score="0" ViewCount="2963" Body="&lt;p&gt;I want to use an &lt;em&gt;accelerometer&lt;/em&gt; to &lt;em&gt;detect which way a train is heading&lt;/em&gt;.&#xA;You could do this by just setting thresholds manually to detect the direction of acceleration on the x-axis (which would be aligned to the trains driving axis) after a standstill. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still want to solve this problem using machine learning, because setting the thresholds takes quite some time.&#xA;The sensor data has to be in a context, otherwise its useless. The driving direction can only be estimated if you know the n sensor readings before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I could probably use a RNN to create this context.&#xA;My ideas was it to not use a RNN (since it adds complexity) and just put several measurments in one sample. It would look like this:&#xA;accelX,accelY,accelZ,accelX1,accelY2,AccelZ2....AccelXn,AccelYn,AccelZn, label&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also measure the Y- and Z-axis because they are important to tell if you are standing still or just driving very smoothly without much acceleration (you have peaks in the Z-axis because you drive over bumps and in curves you have peaks on the Y-axis)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These samples would be taken one after another. This would create the context the net needs to judge it the sample was either standstill,accelerating positive on the X-Axis, accelerating negative on the X-Axis or if the train is just driving normally. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After my net would be trained I could just measure, pack and feed these samples in my net and in the end get a classification. If now one sample is classified as standstill and the next following one as accelerating+, I'd now that we are now driving forward. If a driving then standstill then accelerating- follows I then know, that we are now driving backward. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset will be quite unbalanced since the train is mostly driving and only accelerating once after every station. In addition, my dataset won't be too big since I can only record data when I'm in the train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use keras for building my model since its easy to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do you think my approach will work?&lt;/li&gt;&#xA;&lt;li&gt;What model architecture do you suggest me?&lt;/li&gt;&#xA;&lt;li&gt;Should I manually classify my samples or use unsupervised learning?&lt;/li&gt;&#xA;&lt;li&gt;Should I also include other features (variance of the data ..)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help&lt;/p&gt;&#xA;" OwnerUserId="39477" LastActivityDate="2017-09-21T13:10:33.447" Title="Multiclass Classification on live sensor data" Tags="|time-series|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23197" PostTypeId="2" ParentId="23066" CreationDate="2017-09-21T08:32:08.847" Score="1" Body="&lt;p&gt;You can use &lt;a href=&quot;https://www.deepdetect.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;deepdetect&lt;/a&gt; which provides image classification functionality over API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a gender classification model available on the &lt;a href=&quot;https://www.deepdetect.com/applications/model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;models page&lt;/a&gt;. I have used it without problems before.&lt;/p&gt;&#xA;" OwnerUserId="39508" LastActivityDate="2017-09-21T08:32:08.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23201" PostTypeId="1" AcceptedAnswerId="23205" CreationDate="2017-09-21T11:07:11.650" Score="1" ViewCount="102" Body="&lt;p&gt;Considering a simple ANN:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$x \rightarrow&#xA;f=(U_{m\times n}x^T)^T \rightarrow&#xA;g = g(f) \rightarrow&#xA;h = (V_{p \times m}g^T)^T \rightarrow&#xA;L = L(h,y)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x\in\mathbb{R}^n$, $U$ and $V$ are matrices, $g$ is the point-wise sigmoid function, $L$ returns a real number indicating the loss by comparing the output $h$ with target $y$, and finally $\rightarrow$ represents data flow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To minimize $L$ over $U$ and $V$ using gradient descent, we need to know $\frac{\partial L}{\partial U_{ij}}$ and $\frac{\partial L}{\partial V_{ij}}$, I know two ways to do this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;do the differentiation point wise, and having a hard time figuring out how to vectorize it&lt;/li&gt;&#xA;&lt;li&gt;flatten $U$ and $V$ into a row vector, and use multivariate calculus (takes a vector, yields a vector) to do the differentiation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the purpose of tutorial or illustration, the above two methods might be suffice, but say if you really want to implement back-prop by hand in the real world, what math will you use to do the derivative?&#xA;I mean, is there a branch, or method in meth, that teaches you how to take derivative of vector-valued function of matrices?&lt;/p&gt;&#xA;" OwnerUserId="39117" LastEditorUserId="39117" LastEditDate="2017-09-21T11:17:29.110" LastActivityDate="2017-09-21T13:02:00.717" Title="What's the math for real world back-propagation?" Tags="|machine-learning|deep-learning|backpropagation|theory|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23202" PostTypeId="1" CreationDate="2017-09-21T11:42:21.380" Score="0" ViewCount="85" Body="&lt;p&gt;I want to improve the graphic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/skWQN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/skWQN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that I created with R's ggplot2. The graphic is for a scientific publication. These are the runtime of 12 algorithms. There are four at the top, four in the middle and four at the bottom. They are not necessarily related to each other. It is very hard to discern the different colours from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main point that I want to drive home is that there are four that are slow, four that are middle fast, and four that are fast. Indeed, the groups should be more or less equally fast, and the fact that they differ is mostly due to the fact, that we talk about runtime (which always differs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I improve the plot such that I drive home the main point without being distracted by the many colour, but possibly still have differentiation within the three groups?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe it is not smart to keep the differentiation in the first place?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to keep the line plot, but I am open to alternative suggestions if there are really good ideas out there.&lt;/p&gt;&#xA;" OwnerUserId="14909" LastActivityDate="2017-11-20T21:14:20.717" Title="Improve line plot visualisation" Tags="|visualization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23203" PostTypeId="2" ParentId="23202" CreationDate="2017-09-21T11:55:32.680" Score="1" Body="&lt;p&gt;You already made it log scale which would have been my first idea. How about use lattice to make it three plots with different Y start and end? Each plot should have a center point that covers one of the three groupings you have and you can stack them 1x3 (vertically). &lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-21T11:55:32.680" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23204" PostTypeId="2" ParentId="23202" CreationDate="2017-09-21T12:01:24.043" Score="0" Body="&lt;p&gt;Three things you can try &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You can add a legend to the plot. Even clearer would be text just&#xA;above the groups of lines that say &quot;fast&quot;, &quot;medium&quot;, &quot;slow&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Make sure you mention to the audience that it is your subjective&#xA;grouping, or if you use a metric, to include the definition&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The loglog plot makes the dramatic runtime differences less pronounced.  You can try seeing what this plot looks like with linear axes&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39508" LastActivityDate="2017-09-21T12:01:24.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23205" PostTypeId="2" ParentId="23201" CreationDate="2017-09-21T13:02:00.717" Score="1" Body="&lt;p&gt;There is &lt;a href=&quot;https://atmos.washington.edu/~dennis/MatrixCalculus.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matrix Calculus&lt;/a&gt;, (and I would recommend the very useful &lt;a href=&quot;https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matrix Cookbook&lt;/a&gt; as a bookmark to keep), but for the most part, when it comes to derivatives, it just boils down to pointwise differentiation and keeping your dimensionalities in check.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to look up &lt;a href=&quot;https://en.wikipedia.org/wiki/Automatic_differentiation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autodifferentiation&lt;/a&gt;. This is sort of a generalisation of the Chain Rule, such that it's possible to decompose any composite function, i.e. $a(x) = f(g(x))$, and calculate the gradient of the loss with respect to $g$ as a function of the gradient of the loss with respect to $f$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that for every operation in your neural network, you can give it the gradient of the operation that &quot;consumes&quot; it, and it'll calculate its own gradient and propagate the error backwards (hence back-propagation)&lt;/p&gt;&#xA;" OwnerUserId="24895" LastActivityDate="2017-09-21T13:02:00.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23206" PostTypeId="2" ParentId="23196" CreationDate="2017-09-21T13:10:33.447" Score="2" Body="&lt;p&gt;I would suggest you use a simpler method first, just to have a baseline to compare from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you can adapt &lt;a href=&quot;https://github.com/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this jupyter notebook&lt;/a&gt; to fit your purpose. They train a k-NN model to compare time-series using similarity measures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you're comfortable with that, you can move on to using LSTMs for classification&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTMs for Human Activity Recognition&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/aqibsaeed/Multilabel-timeseries-classification-with-LSTM&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multilabel time series classification with LSTM&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The last one also works if you are trying to predict multiple labels per sample.&lt;/p&gt;&#xA;" OwnerUserId="39508" LastActivityDate="2017-09-21T13:10:33.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23207" PostTypeId="1" CreationDate="2017-09-21T14:18:23.027" Score="1" ViewCount="68" Body="&lt;p&gt;Can anyone explain (or refer to a great explanation of) the &lt;strong&gt;intuition&lt;/strong&gt; of how &lt;a href=&quot;http://www.cs.cornell.edu/~cnyu/latentssvm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Latent SVM&lt;/a&gt; works? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think Latent SVM should have some resemblance to CRF (Conditional Random Fields) and EM (expectation maximization) and of course standard SVM (max-margins, kernel trick) and I'll appreciate very much answers using these 'tools' to explain how Latent SVM works. &lt;/p&gt;&#xA;" OwnerUserId="39513" LastEditorUserId="39513" LastEditDate="2017-09-21T14:31:23.077" LastActivityDate="2017-09-21T14:31:23.077" Title="Intuitive explanation of how Latent SVM works?" Tags="|machine-learning|svm|graphical-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23211" PostTypeId="1" AcceptedAnswerId="23218" CreationDate="2017-09-21T14:45:54.727" Score="0" ViewCount="1286" Body="&lt;p&gt;I have a set of images which are loaded from an &lt;code&gt;h5&lt;/code&gt; file. I checked their dimensions and I get &lt;code&gt;(209, 64, 64, 3)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;read = h5py.File('datasets/train_catvnoncat.h5', 'r')&#xA;read['train_set_x'].shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(209, 64, 64, 3)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;it means that there are 209 images but the point that I cannot understand is that, what is (64, 64, 3)? &lt;br /&gt;&#xA;I have used the following code for plotting: &lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;&#xA;plt.imshow(read['train_set_x'][1])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I get a colored image which is 64 by 64. before this, I thought for (., ., .) shapes, the second number specifies the number of lines and the third one specifies the number of rows. also the first one specifies the number of the mentioned (row and column) arrays.&lt;br /&gt;&#xA;My question is that in &lt;code&gt;numpy&lt;/code&gt; if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator; Why this is different in images and rows and columns are arranged differently in images. Shouldn't it be (3, 64, 64)?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-09-21T17:20:00.677" LastActivityDate="2017-09-21T17:35:33.577" Title="What is the order of elements in an image in python?" Tags="|python|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23213" PostTypeId="1" CreationDate="2017-09-21T16:02:30.073" Score="9" ViewCount="2937" Body="&lt;p&gt;In papers such as &lt;a href=&quot;http://arxiv.org/abs/1709.01507&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; I often see training curves with this kind of shape:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UM8Kp.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UM8Kp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why is there such a large decrease in error when the learning rate is changed?&lt;/li&gt;&#xA;&lt;li&gt;Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?&lt;/li&gt;&#xA;&lt;li&gt;Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="34165" LastEditorUserId="34165" LastEditDate="2017-09-21T17:32:23.323" LastActivityDate="2017-09-22T23:19:23.967" Title="Why does decreasing the SGD learning rate cause a massive increase in accuracy?" Tags="|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23214" PostTypeId="2" ParentId="23213" CreationDate="2017-09-21T16:16:14.177" Score="6" Body="&lt;p&gt;With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution.&#xA;Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again.&#xA;To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over.&#xA;Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2017-09-21T16:16:14.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23215" PostTypeId="2" ParentId="23213" CreationDate="2017-09-21T16:17:29.880" Score="2" Body="&lt;p&gt;Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-09-22T23:19:23.967" LastActivityDate="2017-09-22T23:19:23.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23216" PostTypeId="2" ParentId="6612" CreationDate="2017-09-21T16:28:36.633" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-distributed stochastic neighbor embedding (t-SNE)&lt;/a&gt; is often used for dimensionality reduction in word embeddings. t-SNE maintains the relative relationships between the vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most often t-SNE is used for visualization, thus reducing the dimensions to 2 or 3. It could also reduce the dimensions down to 50.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-21T16:28:36.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23217" PostTypeId="2" ParentId="10028" CreationDate="2017-09-21T16:34:29.363" Score="1" Body="&lt;p&gt;There are many ways to perform chunking/phrase structure parsing. The most common methods are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Rule-based&lt;/li&gt;&#xA;&lt;li&gt;Statistical models&lt;/li&gt;&#xA;&lt;li&gt;Deep Learning&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Fixed rules are simple and fast. But they make many mistakes given the complexity of human language. The other methods are typically more useful because they generate probabilistic rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Context-free_grammar&quot; rel=&quot;nofollow noreferrer&quot;&gt;Context-free grammar (CFG)&lt;/a&gt; is a set of rules for the production of formal language and can be used for parsing.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-11-24T23:19:03.433" LastActivityDate="2017-11-24T23:19:03.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23218" PostTypeId="2" ParentId="23211" CreationDate="2017-09-21T17:35:33.577" Score="3" Body="&lt;p&gt;The &lt;code&gt;[64, 64, 3]&lt;/code&gt; shape you have found is a common convention to represent a colour image in (x, y, colour_channel) dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key word here is &lt;em&gt;convention&lt;/em&gt; - there is no inherently preferred way to represent a colour image in terms of fundamental maths or computing needs, and even within Python you will find multiple conventions, varying in the ordering within the dimension - e.g. OpenCV uses (x, y, channel) convention for the shape, but has channels in order BGR - so channel 0 is blue - whilst most other libraries will use RGB ordering (ignoring for now the alternative colour spaces).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is that in numpy if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When you have a 3-dimensional array, what you decide to call &quot;rows&quot; and &quot;columns&quot; is also a convention. It depends partly on what that array represents, and there is no single way to visualise the contents.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-21T17:35:33.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23219" PostTypeId="1" CreationDate="2017-09-21T18:00:20.947" Score="1" ViewCount="5493" Body="&lt;p&gt;Suppose I want to apply cross validation without any inbuilt function. I have multi classifier problem which is perfectly  divided in n section where n is the number of target features. If I divide my training data sets in to k folds sets then I might end up having single target feature in one set. So to solve this problem I shuffle my training set and target and I am able to achieve 0.96 accuracy compared to 0.98 when I use inbuilt method in &lt;code&gt;sklearn&lt;/code&gt;. Is there a better approach to divide your data perfectly in to k folds? &lt;/p&gt;&#xA;" OwnerUserId="39533" LastEditorUserId="39533" LastEditDate="2017-09-21T18:06:50.363" LastActivityDate="2017-09-22T01:42:12.137" Title="K fold cross validation algorithm" Tags="|machine-learning|python|classification|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23220" PostTypeId="1" CreationDate="2017-09-21T18:27:46.677" Score="1" ViewCount="87" Body="&lt;p&gt;We have some skill levels (beginner, advance, expert) which users assign themselves. Then they get some rating (2, 3, 4, 5) stars from others…. So an expert may have 2-star overall rating and an advance may have a 5-star overall rating. I think advance user with 5-star overall rating should be valued more than expert user with 2-star overall rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we compose a recommender algorithm with this reversal stuff? &lt;/p&gt;&#xA;" OwnerUserId="39534" LastEditorUserId="29575" LastEditDate="2017-09-21T18:47:18.100" LastActivityDate="2017-09-21T18:47:18.100" Title="Recommender algorithm" Tags="|machine-learning|algorithms|recommender-system|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23221" PostTypeId="1" AcceptedAnswerId="23222" CreationDate="2017-09-21T18:40:05.183" Score="7" ViewCount="21269" Body="&lt;p&gt;I have a dataframe where I need to fill in the missing values in one column (paid_date) by using the values from rows with the same value in a different column (id). There is guaranteed to be no more than 1 non-null value in the paid_date column per id value and the non-null value will always come before the null values. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;index     id       paid_date           &#xA;6         25220    2017-01-05 00:00:00&#xA;9         30847    None               &#xA;11        30847    None               &#xA;14        29369    2017-06-21 00:00:00&#xA;17        31232    2017-08-31 00:00:00&#xA;20        26196    2017-02-20 00:00:00&#xA;21        26196    None               &#xA;24        28303    2017-05-09 00:00:00&#xA;25        28303    None               &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I replace the &lt;code&gt;None&lt;/code&gt; values in the &lt;code&gt;paid_date&lt;/code&gt; column if there is a row with a &lt;code&gt;paid_date&lt;/code&gt; with a matching id?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;index     id       paid_date          &#xA;6         25220    2017-01-05 00:00:00&#xA;9         30847    None               &#xA;11        30847    None               &#xA;14        29369    2017-06-21 00:00:00&#xA;17        31232    2017-08-31 00:00:00&#xA;20        26196    2017-02-20 00:00:00&#xA;21        26196    2017-02-20 00:00:00&#xA;24        28303    2017-05-09 00:00:00&#xA;25        28303    2017-05-09 00:00:00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried using &lt;code&gt;fillna&lt;/code&gt; with a dictionary that mapped ids to paid_dates and I tried using &lt;code&gt;pd.Series.map&lt;/code&gt; but neither worked.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;paid_dates = df[pd.notnull(df['paid_date'])]&#xA;pds = pd.Series(data=paid_dates['paid_date'].values, index=paid_dates['id'])&#xA;pds_dict = pds.to_dict()&#xA;&#xA;# doesn't work&#xA;df['paid_date'].fillna(value=pds_dict)&#xA;&#xA;# also doesn't work&#xA;df['paid_date'].map(pds_dict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39529" LastEditorUserId="29575" LastEditDate="2017-09-25T15:14:50.937" LastActivityDate="2017-09-25T15:14:50.937" Title="Pandas how to fill missing values in one column if the values in another column are equal" Tags="|python|data-cleaning|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23222" PostTypeId="2" ParentId="23221" CreationDate="2017-09-21T18:45:53.637" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;There is guaranteed to be no more than 1 non-null value in the&#xA;  paid_date column per id value and the non-null value will always come&#xA;  before the null values.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [117]: df['paid_date'] = pd.to_datetime(df['paid_date'], errors='coerce')&#xA;&#xA;In [118]: df&#xA;Out[118]:&#xA;   index     id  paid_date&#xA;0      6  25220 2017-01-05&#xA;1      9  30847        NaT&#xA;2     11  30847        NaT&#xA;3     14  29369 2017-06-21&#xA;4     17  31232 2017-08-31&#xA;5     20  26196 2017-02-20&#xA;6     21  26196        NaT&#xA;7     24  28303 2017-05-09&#xA;8     25  28303        NaT&#xA;&#xA;In [119]: df.groupby('id').ffill()&#xA;Out[119]:&#xA;   index     id  paid_date&#xA;0      6  25220 2017-01-05&#xA;1      9  30847        NaT&#xA;2     11  30847        NaT&#xA;3     14  29369 2017-06-21&#xA;4     17  31232 2017-08-31&#xA;5     20  26196 2017-02-20&#xA;6     21  26196 2017-02-20&#xA;7     24  28303 2017-05-09&#xA;8     25  28303 2017-05-09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If it's not guaranteed, then we can do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [111]: df['paid_date'] = pd.to_datetime(df['paid_date'], errors='coerce')&#xA;&#xA;In [112]: df&#xA;Out[112]:&#xA;   index     id  paid_date&#xA;0      6  25220 2017-01-05&#xA;1      9  30847        NaT&#xA;2     11  30847        NaT&#xA;3     14  29369 2017-06-21&#xA;4     17  31232 2017-08-31&#xA;5     20  26196 2017-02-20&#xA;6     21  26196        NaT&#xA;7     24  28303 2017-05-09&#xA;8     25  28303        NaT&#xA;&#xA;In [113]: df.loc[df.paid_date.isnull(), 'paid_date'] = \&#xA;              df.loc[df.paid_date.isnull(), 'id'].map(df.loc[df.paid_date.notnull()] \&#xA;                .set_index('id')['paid_date'])&#xA;&#xA;In [114]: df&#xA;Out[114]:&#xA;   index     id  paid_date&#xA;0      6  25220 2017-01-05&#xA;1      9  30847        NaT&#xA;2     11  30847        NaT&#xA;3     14  29369 2017-06-21&#xA;4     17  31232 2017-08-31&#xA;5     20  26196 2017-02-20&#xA;6     21  26196 2017-02-20&#xA;7     24  28303 2017-05-09&#xA;8     25  28303 2017-05-09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2017-09-21T18:51:41.230" LastActivityDate="2017-09-21T18:51:41.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23223" PostTypeId="2" ParentId="17776" CreationDate="2017-09-21T19:23:55.933" Score="6" Body="&lt;p&gt;As of August 14, 2017, you can install Pytorch from &lt;code&gt;peterjc123&lt;/code&gt;'s fork as follows. Currently, python 3.5 and 3.6 are supported. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# If your main Python version is not 3.5 or 3.6&#xA;conda create -n test python=3.6 numpy pyyaml mkl&#xA;&#xA;# for CPU only packages&#xA;conda install -c peterjc123 pytorch&#xA;&#xA;# for Windows 10 and Windows Server 2016, CUDA 8&#xA;conda install -c peterjc123 pytorch cuda80&#xA;&#xA;# for Windows 10 and Windows Server 2016, CUDA 9&#xA;conda install -c peterjc123 pytorch cuda90&#xA;&#xA;# for Windows 7/8/8.1 and Windows Server 2008/2012, CUDA 8&#xA;conda install -c peterjc123 pytorch_legacy cuda80&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;pytorch support for windows &lt;a href=&quot;https://github.com/peterjc123/pytorch-scripts&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="16338" LastEditorUserId="41639" LastEditDate="2018-02-17T00:22:31.713" LastActivityDate="2018-02-17T00:22:31.713" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23224" PostTypeId="2" ParentId="23219" CreationDate="2017-09-21T19:25:48.063" Score="3" Body="&lt;p&gt;It seems there is a small bit of ambiguity in the question. But my best guess is that you are looking for stratifiedkfold (Emre has mentioned that in a comment) or second best guess is labelkfold.&#xA;See: &lt;a href=&quot;http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold&lt;/a&gt;&#xA;and&#xA;&lt;a href=&quot;http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.LabelKFold.html#sklearn.cross_validation.LabelKFold&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.LabelKFold.html#sklearn.cross_validation.LabelKFold&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would help if you specify which inbuilt cross_validation in sklearn you tried, since there are several to choose from:&#xA;&lt;a href=&quot;http://scikit-learn.org/0.17/modules/classes.html#module-sklearn.cross_validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/0.17/modules/classes.html#module-sklearn.cross_validation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, please also let us know about the number of data points available, since that will contribute to deciding on the numeric value of k. The point being that you don't want to bias your training due to an counter-productively high value of k for the available n data points. Herein lies the reason for the labelkfold approach since it will ensure that the same data point does not repeatedly occur in multiple folds, which would end up biasing your classifier, and we want to avoid this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if your training time is of the order of a few minutes, you could try several cross_validation schemes. It will be a good learning experience and might provide some useful insights into the data :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&#xA;Since, you are exploring the Iris dataset, I've gone ahead and created a small demo of the effect of different cross validation methods on classifier evaluation. I've put in a small range for relevant parameters for each of the methods considered. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've included my code for you to experiment with, you can try more cross-validation methods. Iris data set has small set of instances and there isn't too much 'craziness' in the distribution of the classes. If you tried it on some real-world data with a lot of bias and noise, you might see a more pronounced effect of cross-validation (use error-bars). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &quot;&quot;&quot;&#xA;Evaluate cross-validation schemes for Iris dataset&#xA;&quot;&quot;&quot;&#xA;&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import seaborn as sns&#xA;from sklearn import svm&#xA;from sklearn import datasets&#xA;from sklearn.model_selection import train_test_split&#xA;from time import time&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.model_selection import StratifiedKFold&#xA;from sklearn.model_selection import RepeatedStratifiedKFold&#xA;&#xA;iris = datasets.load_iris()&#xA;X, y = iris.data, iris.target&#xA;&#xA;&quot;&quot;&quot; visually inspect iris dataset &quot;&quot;&quot;&#xA;iris_sns = sns.load_dataset('iris')&#xA;g = sns.pairplot(iris_sns, hue='species', markers=[&quot;o&quot;, &quot;s&quot;, &quot;D&quot;])&#xA;plt.savefig(filename='iris_pairplot.png')&#xA;&quot;&quot;&quot; Dimensions of data matrix and target labels &quot;&quot;&quot;&#xA;print(X.shape, y.shape)&#xA;&#xA;&quot;&quot;&quot;&#xA;cross-validation: the apparent silver bullet to over-fitting&#xA;The available data and label matrices are split into set of training and validation matrices&#xA;We first begin with mere evaluation of classifier trained on different random sets &#xA;of training and validation data sampled from the entire available data.&#xA;By averaging the computed classification performance on each training subset,&#xA;we will acquire a statistically reliable estimate of how this classifier might perform on unseen test data.&#xA;Subsequently, we will train classifier on multiple subsets of training data to actually improve the&#xA;learnt classifier itself.&#xA;&quot;&quot;&quot;&#xA;&#xA;&quot;&quot;&quot; The effect of train-test split sizes&quot;&quot;&quot;&#xA;test_sizes = [0.1 * i for i in range(1, 10)]&#xA;scores = []&#xA;for test_size in test_sizes:&#xA;    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=round(time()))&#xA;    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)&#xA;    clf = svm.SVC(kernel='linear', C=0.1).fit(X_train, y_train)&#xA;    scores.append(clf.score(X_test, y_test))&#xA;&#xA;fig1 = plt.figure(1)&#xA;plt.plot(test_sizes, scores, 'k-d')&#xA;plt.title('Iris dataset, effect of proportion of train-test splits')&#xA;plt.xlabel('proportion of test-images')&#xA;plt.ylabel('classification score')&#xA;plt.savefig('train-test-splits.png')&#xA;&quot;&quot;&quot; The effect of number of cross-validations &quot;&quot;&quot;&#xA;cvs = [i for i in range(2, 11)]&#xA;cv_scores_means = []&#xA;cv_scores_std = []&#xA;for cv in cvs:&#xA;    clf = svm.SVC(kernel='linear', C=0.1, random_state=round(time()))&#xA;    scores = cross_val_score(clf, X, y, cv=cv)&#xA;    cv_scores_means.append(scores.mean())&#xA;    cv_scores_std.append(scores.std())&#xA;&#xA;fig2 = plt.figure(2)&#xA;plt.errorbar(cvs, cv_scores_means, yerr=cv_scores_std, fmt='k-o')&#xA;plt.title('Effect of number of cross-validations')&#xA;plt.xlabel('number of cross-validation')&#xA;plt.ylabel('mean classification score')&#xA;plt.savefig('number-of-cv.png')&#xA;&quot;&quot;&quot; The effect of k-stratified splits&quot;&quot;&quot;&#xA;num_splits = [i for i in range(2,11)]&#xA;scores_splits = []&#xA;for n_splits in num_splits:&#xA;    skf = StratifiedKFold(n_splits=n_splits)&#xA;    clf = svm.SVC(kernel='linear', C=0.1)&#xA;    scores = []&#xA;    for train, test in skf.split(X, y):&#xA;        clf.fit(X[train], y[train])&#xA;        scores.append(clf.score(X[test], y[test]))&#xA;    scores_splits.append(np.average(scores))&#xA;&#xA;fig3 = plt.figure(3)&#xA;plt.plot(num_splits, scores_splits, 'k-s')&#xA;plt.title('Effect of number of stratified splits')&#xA;plt.xlabel('number of splits')&#xA;plt.ylabel('mean classification score')&#xA;plt.savefig('num-stratified-splits.png')&#xA;&quot;&quot;&quot; the effect of k-label splits&quot;&quot;&quot;&#xA;num_folds = [i for i in range(2, 11)]&#xA;num_repeats = [i for i in range(2, 10, 2)]&#xA;line_colors = ['r', 'g', 'b', 'k', 'y']&#xA;scores_repeats = []&#xA;for n_repeat in num_repeats:&#xA;    scores_folds = []&#xA;    for n_fold in num_folds:&#xA;        rskf = RepeatedStratifiedKFold(n_splits=n_fold, n_repeats=n_repeat, random_state=round(time()))&#xA;        clf = svm.SVC(kernel='linear', C=0.1)&#xA;        scores = []&#xA;        for train, test in rskf.split(X, y):&#xA;            clf.fit(X[train], y[train])&#xA;            scores.append(clf.score(X[test], y[test]))&#xA;        scores_folds.append(np.average(scores))&#xA;    scores_repeats.append(scores_folds)&#xA;fig4 = plt.figure(4)&#xA;handles = []&#xA;for i in range(len(num_repeats)):&#xA;    h, = plt.plot(num_folds, scores_repeats[i], line_colors[i]+'-x', label=&quot;#rep: &quot;+str(num_repeats[i]))&#xA;    handles.append(h)&#xA;plt.title('Effect of number of splits and repeats for repeated stratified k fold')&#xA;plt.xlabel('number of splits')&#xA;plt.ylabel('mean classification score')&#xA;plt.legend(handles=[h for h in handles])&#xA;plt.savefig('num-split-repeat-stratified.png')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yedEw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yedEw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WYuXU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WYuXU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ZtQ5S.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZtQ5S.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/GQ075.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GQ075.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2515" LastEditorUserId="2515" LastEditDate="2017-09-22T01:42:12.137" LastActivityDate="2017-09-22T01:42:12.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23226" PostTypeId="2" ParentId="23202" CreationDate="2017-09-21T19:57:12.817" Score="0" Body="&lt;p&gt;If your main objective is to highlight/compare the speed groupings, I think a bar chart might accomplish that more clearly. You could plot 12 distinct bars, showing the total runtime@50% (or whichever % mark you like). The speed groupings would still be clearly shown, and you could label each individual algorithm in the display.&lt;/p&gt;&#xA;" OwnerUserId="39538" LastActivityDate="2017-09-21T19:57:12.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23229" PostTypeId="2" ParentId="22981" CreationDate="2017-09-22T06:29:57.227" Score="2" Body="&lt;p&gt;You need to do some Background Subtraction on the images. If you have the Background image without the animal, you can simply subtract it from the current image to get just the animal.&#xA;Once you have just the animal, you can apply SIFT or CNNs or whatever.&#xA;This is called &lt;strong&gt;frame differencing&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fvyrk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fvyrk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have the background image, you can try methods like &lt;a href=&quot;http://docs.opencv.org/3.3.0/db/d5c/tutorial_py_bg_subtraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; provided by opencv&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically what you are looking for is background subtraction/foreground detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;image source: &lt;a href=&quot;http://docs.opencv.org/3.3.0/d1/dc5/tutorial_background_subtraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://docs.opencv.org/3.3.0/d1/dc5/tutorial_background_subtraction.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-09-22T06:29:57.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23230" PostTypeId="1" CreationDate="2017-09-22T06:43:47.887" Score="1" ViewCount="119" Body="&lt;p&gt;I'm making a ad-sales predictor and try to make it fit by using the S curve's below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ ()= \frac{}{1+} (1−^{−}) $&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$f(x)$ is an ad-sales&lt;/li&gt;&#xA;&lt;li&gt;$x$ is an advertise cost&lt;/li&gt;&#xA;&lt;li&gt;both $a$ and $b$ are fitting parameters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It looks like logistic function curve. I actually don't know a name of it though I have to use it for our client order. Do you know a name of the function or any text referring it?&lt;/p&gt;&#xA;" OwnerUserId="39553" LastEditorUserId="29575" LastEditDate="2017-09-22T15:02:47.050" LastActivityDate="2017-09-22T15:02:47.050" Title="Do you know this S curve's name?" Tags="|predictive-modeling|marketing|simulation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23232" PostTypeId="2" ParentId="23230" CreationDate="2017-09-22T10:56:23.963" Score="5" Body="&lt;p&gt;From a quick search, the function you provided doesn't seem to have a particular name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are you sure the function belongs to Sigmoidal curves? The &lt;a href=&quot;https://www.desmos.com/calculator/dzrub58we2&quot; rel=&quot;noreferrer&quot;&gt;function&lt;/a&gt; does not remind too much of an S-shape.&lt;/p&gt;&#xA;" OwnerUserId="39508" LastActivityDate="2017-09-22T10:56:23.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23233" PostTypeId="1" AcceptedAnswerId="23468" CreationDate="2017-09-22T12:40:44.170" Score="10" ViewCount="4008" Body="&lt;p&gt;What is the best cost function to train a &lt;strong&gt;neural network&lt;/strong&gt; to perform &lt;strong&gt;ordinal regression&lt;/strong&gt;,  i.e. &lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_regression&quot; rel=&quot;noreferrer&quot;&gt;to predict a result whose value exists on an arbitrary scale where only the relative ordering between different values is significant&lt;/a&gt; (e.g: to predict which product size a customer will order: 'small' (coded as 0), 'medium'(coded as 1), 'large' (coded as 2) or 'extra-large'(coded as 3))? I'm trying to figure out if there are better alternatives than quadratic loss (modeling the problem as an 'vanilla' regression) or cross-entropy loss (modeling the problem as classification).&lt;/p&gt;&#xA;" OwnerUserId="39513" LastActivityDate="2020-06-17T12:22:16.543" Title="Cost function for Ordinal Regression using neural networks" Tags="|neural-network|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23236" PostTypeId="1" CreationDate="2017-09-22T14:53:42.497" Score="1" ViewCount="1082" Body="&lt;p&gt;I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consequently, I get 2 tokens for “organization” and “organisation” or “color” and “colour”.&#xA;Examples : &lt;a href=&quot;https://en.oxforddictionaries.com/spelling/british-and-spelling&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.oxforddictionaries.com/spelling/british-and-spelling&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know a (python) library converting “British English” to “American English” (or vis versa) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would be happy to that ... (but I am french and my english is not soo good)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="39564" LastActivityDate="2019-12-03T09:00:55.103" Title="Tokenize text with both American and English words" Tags="|text-mining|nltk|text-filter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23237" PostTypeId="2" ParentId="23236" CreationDate="2017-09-22T16:07:30.443" Score="1" Body="&lt;p&gt;Grouping related tokens is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Text_normalization&quot; rel=&quot;nofollow noreferrer&quot;&gt;text normalization&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is not an established Python package that does this. You could create a &lt;a href=&quot;https://stackoverflow.com/questions/42329766/python-nlp-british-english-vs-american-english&quot;&gt;custom dictionary&lt;/a&gt; or write &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_extraction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;a function to rewrite the tokens&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-09-22T16:07:30.443" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23238" PostTypeId="1" CreationDate="2017-09-22T16:32:16.160" Score="1" ViewCount="28" Body="&lt;p&gt;I have the following data frame for a certain user group. Basically, it shows the current page, and the previous page visited across all users. For example, 81 users come from BLQ2_2 to visit BLQ2_1. I have the same data set for another user group. I am trying find an effective way of visualising these data sets to spot the differences among the user groups. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PAGE    PREV_PAGE COUNT  &#xA;BLQ2_1  BLQ2_2    81&#xA;        BLQ2_4     8&#xA;        BLQ2_3     6&#xA;        BLQ2_1     3&#xA;BLQ2_2  BLQ2_3    91&#xA;        BLQ2_4    22&#xA;        BLQ2_1     9&#xA;        BLQ2_5     5&#xA;        BLQ2_2     1&#xA;BLQ2_3  BLQ2_4    87&#xA;        BLQ2_1    12&#xA;        BLQ2_5     6&#xA;        BLQ2_2     2&#xA;        BLQ2_3     1&#xA;BLQ2_4  BLQ2_5    45&#xA;        BLQ2_1    26&#xA;        BLQ2_2    17&#xA;        BLQ2_3     8&#xA;        BLQ2_4     3&#xA;BLQ2_5  BLQ2_1    19&#xA;        BLQ2_2    12&#xA;        BLQ2_4    12&#xA;        BLQ2_5     8&#xA;        BLQ2_3     6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A bar chart could help. But, I wonder if there are other options around there that I could not think of. Another option could be arranging the pages like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;BLQ2_1       BLQ2_2      BLQ2_3       BLQ2_4      BLQ2_5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and connecting them based on the visits (e.g., drawing a thick line -considering the COUNT value is 81- from BLQ2_2 to BLQ2_1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate some input.&lt;/p&gt;&#xA;" OwnerUserId="28915" LastActivityDate="2017-09-22T16:32:16.160" Title="Visualising 2-step sequence data to spot the differences among two different user groups" Tags="|python|visualization|pandas|ggplot2|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23239" PostTypeId="1" CreationDate="2017-09-22T17:37:15.360" Score="1" ViewCount="83" Body="&lt;p&gt;I'm building a Generative Adversarial Network. I need to be able to save the output of the generator in some data structure, and then at each training step, sample from the current generator, as well as from the historical images. How can I do this in TensorFlow? I don't want to use a feed_dict for performance reasons.&lt;/p&gt;&#xA;" OwnerUserId="39261" LastActivityDate="2017-09-22T17:37:15.360" Title="Tensorflow: Save History of Images from Generator" Tags="|neural-network|deep-learning|tensorflow|gan|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23243" PostTypeId="2" ParentId="23193" CreationDate="2017-09-22T20:21:04.527" Score="2" Body="&lt;p&gt;There are a few different data science and ML techniques you could throw at the numbers output by this game. You could try to analyse human players' style. You could generate a table of expected gains/losses by ignoring actual amount gained or lost and estimating the probability distribution of $x$ even though the player did not see it, just from know whether $x \gt w$ or $x \le w$ from enough examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your end goal is to find out how a computer might &lt;em&gt;play&lt;/em&gt; such a game, then one clear machine learning choice would be Reinforcement Learning (RL). It is not the only way for a computer to play a game - there are very many optimisation techniques for this. However, it is a machine learning approach - it learns from data observations - and in many variations it includes numerical analysis that you may find interesting, such as chance of winning from a certain position, or predicted future rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RL is not a single algorithm, or even a single approach. Instead it is a way of framing a problem, and all ways of solving that kind of problem are considered to be RL. For RL to work, a problem needs to be framed as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_decision_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Markov Decision Process (MDP)&lt;/a&gt;. The good news is that your example game, and similar ones, are good examples of MDPs already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In RL, the way that an agent plays a game is called its &lt;em&gt;policy&lt;/em&gt;. The usual goals of RL include measuring the performance of a specific policy, or discovering the best policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, where to start? First, if you want to try an RL process on your game you should probably do a few things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Understand what in the game is the &lt;em&gt;State&lt;/em&gt;, what are the &lt;em&gt;Actions&lt;/em&gt; and what are the &lt;em&gt;Rewards&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The state is anything that the agent knows about and can affect the outcome. Probably the only thing to worry about here is the current funds $f$, although this may not have a huge influence unless $w$ can always go up to $f$. Note each value is treated as a &lt;em&gt;different&lt;/em&gt; state.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The action is the agent's choice of both $w$ and $g$ - note each specific combination is considered a &lt;em&gt;different&lt;/em&gt; action. If there are a large number of actions, this can make learning harder, requiring more advanced RL algorithms, so I recommend initially you try variations of your game with limited number of choices.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The most obvious choice of reward is the change to $f$ at the end of each turn. It doesn't have to be though, it depends on what you consider a &quot;winning condition&quot; to be for the game. See below.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Simplify the game options a little. Set a small limit to possible ranges of $x, w, g$. This will make it easier to try things out initially.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Set a clear goal that you want to achieve, and make sure that the game structure works for that. You may need to adjust the game representation for even simple change such as &quot;get the most reward in 10 turns&quot; - because that might encourage large wagers at certain steps, and the agent will need to know how many turns it has left (so your state becomes combination of $(f, t)$. Another variation that may work is to set the goal of getting to a certain amount of funds, such as 1000. In which case, the reward would not be the increase in $f$, but actually be +1 for getting to that target, and 0 for any other result. That might radically change the behaviour of the agent, which could be interesting to experiment with.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Look up simple Reinforcement Learning algorithms. I suggest you start with tabular methods - perhaps Monte Carlo Control, or Q-Learning. They can typically be implemented in a few tens of lines in Python/Numpy.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A useful resource for RL is &lt;a href=&quot;http://incompleteideas.net/sutton/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton &amp;amp; Barto's &lt;em&gt;Reinforcement Learning: An Introduction&lt;/em&gt;&lt;/a&gt;. The draft of the second version is free to download.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-22T20:58:19.297" LastActivityDate="2017-09-22T20:58:19.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23245" PostTypeId="1" CreationDate="2017-09-22T22:15:43.750" Score="2" ViewCount="9143" Body="&lt;p&gt;Is it possible with Orange (only using its widgets, without writing Python code) to implement the following typical machine learning processes? &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train a training set,&lt;/li&gt;&#xA;&lt;li&gt;Validating a validation set (e.g. using k-fold cross validation), and&lt;/li&gt;&#xA;&lt;li&gt;Testing the model with a test set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Using the 'Test &amp;amp; Score&quot; widget, a single dataset is used. Does this mean that only validation is performed, and there is no actual test? Can in this case classification results (accuracy, AUC, etc.) considered reliable?&lt;/p&gt;&#xA;" OwnerUserId="39577" LastEditorUserId="29575" LastEditDate="2017-09-23T13:50:52.477" LastActivityDate="2019-10-22T17:47:05.887" Title="Training set, validation set, and test set with Orange" Tags="|training|orange|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23247" PostTypeId="1" CreationDate="2017-09-23T00:51:38.470" Score="1" ViewCount="521" Body="&lt;p&gt;After fitting a decision tree with some continuous variable, how do I interpret the effect that variable has on the target?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I'm predicting target Y. From sklearn random forest or Xgboost I can find out that the feature X is important. How do I determine if feature X's correlation to Y is positive or negative?&lt;/p&gt;&#xA;" OwnerUserId="39580" LastActivityDate="2023-12-04T21:04:30.603" Title="How to interpret continuous variables in a decision tree model?" Tags="|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23250" PostTypeId="1" CreationDate="2017-09-23T09:53:58.060" Score="-4" ViewCount="570" Body="&lt;p&gt;Consider an example in which one wants process a 1d list of number with a CNN network, with 1d convolution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are the weight shared across the number of filters, or are they shared across the number of input feature maps ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And are the weight used to compute the different feature maps, are they different or the same?&lt;/p&gt;&#xA;" OwnerUserId="25429" LastEditorUserId="29169" LastEditDate="2021-02-15T22:23:36.273" LastActivityDate="2021-02-15T22:23:36.273" Title="weight sharing for CNN network" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ClosedDate="2021-02-16T11:23:47.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="23253" PostTypeId="1" CreationDate="2017-09-23T14:54:30.963" Score="8" ViewCount="3699" Body="&lt;p&gt;When I read the scikit-learn user manual about Decision Trees, they mentioned that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;CART (Classification and Regression Trees) is very similar to C4.5,&#xA;  but it differs in that it supports numerical target variables&#xA;  (regression) and &lt;strong&gt;does not compute rule sets&lt;/strong&gt;. CART constructs binary&#xA;  trees using the feature and threshold that yield the largest&#xA;  information gain at each node.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't understand where we compute rule sets for the C4.5 algorithm(and I dont even know what rule sets mean). Its essentially same as the CART, except that it uses gini index instead of cross entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please explain what rule sets are and how they are used in C4.5 in detail? &lt;/p&gt;&#xA;" OwnerUserId="39599" LastEditorUserId="144646" LastEditDate="2023-07-30T13:21:33.273" LastActivityDate="2024-01-02T18:00:50.087" Title="Decision Trees - C4.5 vs CART - rule sets" Tags="|decision-trees|algorithms|cart|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23254" PostTypeId="1" AcceptedAnswerId="23255" CreationDate="2017-09-23T15:20:02.227" Score="0" ViewCount="110" Body="&lt;p&gt;If we are to stick with an X and Y axis, the X axis being Time and the Y axis being test scores. Where more time equates to High test scores. You can use a binary classifcation algorithm to predict success. Wouldn't 2 if statements do the same thing&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If (time &amp;gt; someValue)&#xA;   User will probably pass&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Another Scenario&#xA;Is I have papayas, and they have two labels, squishness and color.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If the squishness is greater then 7 //on a scale of 1-10&#xA;   AND the color is green &#xA;     Then it is a good papaya&#xA;&#xA;Else &#xA;   It isn't ripe.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I dont understand the value of a perceptron in these scenarios. &lt;/p&gt;&#xA;" OwnerUserId="39600" LastActivityDate="2017-09-24T12:07:14.387" Title="why use a perceptron when it seems multiple IF staments do the same thing" Tags="|machine-learning|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23255" PostTypeId="2" ParentId="23254" CreationDate="2017-09-23T15:35:16.647" Score="4" Body="&lt;p&gt;Machine learning (perceptrons or not) is all about &lt;strong&gt;automatically&lt;/strong&gt; finding generic but correct rules, be it in the form of If-Else-Rules, encoded formulas, closest occurrences, or others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ML algorithm is just a way to (automatically) find this knowledge, whatever the representation may be. In another words, you use it to find the &lt;code&gt;someValue&lt;/code&gt; of your example, based on your data. You don't &lt;em&gt;need&lt;/em&gt; ML if you can represent such knowledge yourself.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-09-23T15:35:16.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23256" PostTypeId="2" ParentId="23254" CreationDate="2017-09-23T15:45:10.167" Score="0" Body="&lt;p&gt;Perceptrons &lt;em&gt;can indeed&lt;/em&gt; be implemented using this kind of boolean logic. Perceptron training is about finding suitable values to go into the if/else expressions, using example data. In your first example &lt;code&gt;someValue&lt;/code&gt; has to be discovered, from known practice time and test results. In your second example &lt;code&gt;squishiness &amp;gt; 7 and colour &amp;gt;= green&lt;/code&gt; has to be discovered by measuring some papayas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the more general case, the perceptron finds a hyperplane (line in 2D, plane in 3D) that discriminates between two target classes based on the input feature vector $\mathbf{x}$. That hyperplane could be in any direction and is expressed mathematically in terms of the input features. It still can be expressed as an if/else statement though:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $\mathbf{\theta}^T \mathbf{x} + b &amp;gt; 0$:&lt;br/&gt;&#xA;$\qquad$class A&lt;br/&gt;&#xA;Else:&lt;br/&gt;&#xA;$\qquad$class B&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The perceptron training finds parameter vector $\mathbf{\theta}$ and scalar value of bias $b$.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-23T15:45:10.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23257" PostTypeId="1" AcceptedAnswerId="23258" CreationDate="2017-09-23T15:50:21.417" Score="2" ViewCount="2116" Body="&lt;p&gt;My understanding of gradient descent as an optimizer for a neural network is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n &amp;gt; 1$.&lt;/p&gt;&#xA;" OwnerUserId="39601" LastActivityDate="2017-09-23T18:22:28.643" Title="Gradient descent with vector-valued loss" Tags="|neural-network|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23258" PostTypeId="2" ParentId="23257" CreationDate="2017-09-23T16:03:06.443" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n &amp;gt; 1$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot &quot;minimise a vector&quot;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* There is a useful definition of &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-objective_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-objective optimisation&lt;/a&gt;, which effectively finds multiple &lt;em&gt;sets&lt;/em&gt; of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-23T18:22:28.643" LastActivityDate="2017-09-23T18:22:28.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23259" PostTypeId="1" CreationDate="2017-09-23T18:53:31.323" Score="5" ViewCount="2631" Body="&lt;p&gt;In the  context of text mining, I'd like to discover potential synonyms in my dataset. The current dataset is &lt;a href=&quot;https://archive.org/details/stackexchange&quot; rel=&quot;noreferrer&quot;&gt;stackexchange's stackoverflow data on archive.org&lt;/a&gt;. The result doesn't have to be perfect, I can post-process it by hand. But I need help to have a clue of which term seems to be &quot;similar&quot; to another term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here examples of synonyms I am looking for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;postgresql, postgres, pgsql, psql&lt;/li&gt;&#xA;&lt;li&gt;mobile, phone, android, iphone&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also in best case, it should be possible to also guess multi word synonyms like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;rdbms, relational database management system&lt;/li&gt;&#xA;&lt;li&gt;obama, barack obama&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The algorithm doesn't need to compute whether those are many-way synonyms (like rdbms and relational database management system) and one-way synonyms (like iphone is mobile but not all mobile are iphone).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read that word2vec can be helpful but I am not sure how to use it.&lt;/p&gt;&#xA;" OwnerUserId="23614" LastEditorUserId="23614" LastEditDate="2017-09-24T06:06:20.757" LastActivityDate="2018-03-26T17:03:05.023" Title="What algorithm can help me discover synonyms?" Tags="|data-mining|nlp|text-mining|word2vec|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23261" PostTypeId="1" CreationDate="2017-09-24T06:35:27.810" Score="1" ViewCount="94" Body="&lt;p&gt;I want to plot the following estimate  of the tail dependence coefficient:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{\lambda}_n=\frac{1}{k}\sum_{j=1}^nI_{\{X_j&amp;gt;X_{(n-k)},Y_j&amp;gt;X_{(n-k)}\}}$&#xA;where $I_{(.)}$ is an indicator function and $X_{(1)}&amp;gt;...&amp;gt;X_{(n)}$ are order statistics.&lt;br&gt;&#xA;If I consider this example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n=1000;&#xA;alpha=4;&#xA;U=runif(n);     &#xA;phi=0.8;&#xA;sigma=0.1;&#xA;X=(1-U)^{(-1/alpha)} ;&#xA;Z=rnorm(n)      ;   &#xA;Y=phi*X+sigma*abs(Z) ; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then $\lambda=(.8)^4=04.09$. When I plot $\hat{\lambda}_n$ vs $k$ then the graph seems to stabilize around $0.50$ which is not true. See below the graph.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n=length(X);&#xA;lambda_hat=c();&#xA;&#xA;lambda_hat=sapply(1:(n-1), FUN = function(i,minXY=pmin(X,Y),Xsort=sort(X)) length(minXY[minXY&amp;gt;Xsort[floor(n-i)]])/i);&#xA;k=1:(n-1)&#xA;&#xA;plot(k,lambda_hat,type = &quot;l&quot;,lwd=2,col=&quot;blue&quot;,&#xA;     main=&quot;Tdc of X and Y&quot;,ylab=&quot;Estimates&quot;,xlab=&quot;Order statistics&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39613" LastEditorUserId="18249" LastEditDate="2017-09-24T20:22:57.747" LastActivityDate="2017-09-24T20:22:57.747" Title="Plotting estimates of Tail Dependence Coefficient in R" Tags="|r|visualization|estimators|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23262" PostTypeId="1" CreationDate="2017-09-24T08:35:20.843" Score="2" ViewCount="430" Body="&lt;p&gt;Let's say I have an expertise in the domain knowledge of the dataset I am working on. I know that part of my dataset acts 100% differently than the rest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it is straightforward to check if a row belongs to one part of the dataset or the other with a couple of simple if-else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I split my dataset upfront and create two different prediction models for the two parts of the dataset? Or should I keep one model and try to improve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then when I want to use the model for predictions, I check the input values and see in which category it belongs and I call the appropriate model for it.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-09-25T06:46:09.953" Title="Should I divide my dataset in multiple prediction models?" Tags="|predictive-modeling|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23263" PostTypeId="2" ParentId="23250" CreationDate="2017-09-24T09:05:38.887" Score="1" Body="&lt;p&gt;I'm not entirely clear what you're asking, but, let's say your convolutional input is $m$ channels of $n$ numbers. Assuming a same convolution. Then you output $d$ (maybe not the same as $m$) channels of $n$ numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do each of the $d$ output channels come from different filters with different weights? yes of course or they would be redundant. Are the same weights used for each of the $m$ input channels in each filter? no, not shared, so it can learn from each channel separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the convolutional filter the same across the $n$ numbers, when mapping input to output features? yes. Same filter weights are used across the input.&lt;/p&gt;&#xA;" OwnerUserId="21" LastActivityDate="2017-09-24T09:05:38.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23264" PostTypeId="1" AcceptedAnswerId="23272" CreationDate="2017-09-24T10:50:17.553" Score="15" ViewCount="34321" Body="&lt;p&gt;I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to split the original dataset into 3 sub-dataframes based on some simple rules. However, it takes a long time to execute the code. About 15-20 seconds just for the filtering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any alternative way that will improve the performance of the code?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;#read dataset&#xA;df = pd.read_csv('myData.csv')&#xA;&#xA;#create a dataframe with col1 10 and col2 &amp;lt;= 15&#xA;df1 = df[(df.col1 == 10) &amp;amp; (df.col2 &amp;lt;= 15)]&#xA;df = df[~df.isin(df1)].dropna()&#xA;&#xA;#create a dataframe with col3 7 and col4 &amp;gt;= 4&#xA;df2 = df[(df.col3 == 7) &amp;amp; (df.col4 &amp;gt;= 4)]&#xA;df = df[~df.isin(df2)].dropna()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the end, I have the &lt;code&gt;df1, df2, df&lt;/code&gt; dataframes with the filtered data.&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="29575" LastEditDate="2017-09-25T15:15:19.023" LastActivityDate="2022-02-26T07:38:37.407" Title="Improve Pandas dataframe filtering speed" Tags="|python|pandas|performance|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23265" PostTypeId="2" ParentId="23262" CreationDate="2017-09-24T10:59:51.990" Score="2" Body="&lt;p&gt;Yes, you can divide your dataset and use different prediction models for different part of your dataset. Take advantage of your domain knowledge to build a good model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After all, you can actually count the following as a single model &#xA;$$f(x) = \begin{cases} f_1(x) &amp;amp; ,x \in A \\ f_2(x) &amp;amp; ,x \in A^c \end{cases}$$&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2017-09-24T10:59:51.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23266" PostTypeId="1" CreationDate="2017-09-24T11:22:35.857" Score="2" ViewCount="229" Body="&lt;p&gt;I have a questionnaire for different years. The questionnaire layout is the same for every year. The questionnaire has Likert-scale &lt;em&gt;definitely agree&lt;/em&gt; to &lt;em&gt;definitely disagree&lt;/em&gt;. The questions are the in one column and the number of responses is corresponding with the liker scale, which is in each row from top to bottom.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|-------------| Category1| Q1 | q2 |  Category2| Q1 | q2 |  q3&#xA;|-------------| Average  |    |    |  Average&#xA;| agree       | 5        | 5  | 5  |      8    | 9  | 6  |  1&#xA;| disagree    | 4        | 2  | 2  |      3      4    3     3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This an example of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would I use the survey package? what benefits are there? Or should I use a normal association rule and have an association between the question? This is just an example to see any trends and patterns.&lt;/p&gt;&#xA;" OwnerUserId="39620" LastEditorUserId="29575" LastEditDate="2017-09-25T15:15:34.757" LastActivityDate="2017-09-25T15:15:34.757" Title="Questionnaire data analysis" Tags="|r|data-mining|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23267" PostTypeId="2" ParentId="23254" CreationDate="2017-09-24T12:07:14.387" Score="0" Body="&lt;h2&gt;When should I use machine learning?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Machine learning should only be used if there is no reasonable other way:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Automatic Speech Recognition (ASR)&lt;/strong&gt;: We can't manually design how we recognize language by rules. It is too complex, often fuzzy and a lot depends on context. The same is true for various &lt;strong&gt;Computer Vision (CV)&lt;/strong&gt; tasks.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Natural Language Processing (NLP)&lt;/strong&gt;: Let it be automatic summarization, translation or sentiment analysis. Additionally to the problems in ASR and CV it has a lot of special cases. While we might be able to recognize those if phrased correctly, it is in many cases absolutely hopeless to write them all down and keep them up to date.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;What is the perceptron good for?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It's good for learning how multilayer perceptrons (MLPs) work. I'm not sure if there is any real use case where you would want to apply a perceptron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision stumps (if-statements) are extremely simple, just like perceptrons. If such a simple model is enough, you could probably also just set it by hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Please let me know in the comments if you know an example where a single perceptron / decision stump with the associated learning method provides value compared to an ad-hoc decision method)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-09-24T12:07:14.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23268" PostTypeId="1" AcceptedAnswerId="23346" CreationDate="2017-09-24T12:38:41.903" Score="3" ViewCount="177" Body="&lt;p&gt;I used Data Science Studio (Dataiku) to create a Decision Tree model. Then I took the pickle file and loaded into Python to continue using it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pickle&#xA;&#xA;f = open('clf.pkl', 'rb')&#xA;loaded_model = pickle.load(f, encoding='latin1')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On model settings, I used &lt;code&gt;standard rescaling&lt;/code&gt; which uses the &lt;code&gt;avgstd&lt;/code&gt;. Dataiku also exports this &lt;code&gt;json&lt;/code&gt; file with details about the rescaling:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;    &quot;shifts&quot;: [&#xA;        4.2708957215287455, &#xA;        5.582300530732055, &#xA;        4.721780769116731, &#xA;        6.309030531691733, &#xA;        4.534705132386515, &#xA;        50183.866161634876, &#xA;        4.628957297141036, &#xA;        5.931597829632046, &#xA;        1.834355009673187, &#xA;        21814.135528393213, &#xA;        0.9999925875959688, &#xA;        0.23165941222883746, &#xA;        -0.11146363232269413&#xA;    ], &#xA;    &quot;columns&quot;: [&#xA;        &quot;col1&quot;, &#xA;        &quot;col2&quot;, &#xA;        &quot;col3&quot;, &#xA;        &quot;col4&quot;, &#xA;        &quot;col5&quot;, &#xA;        &quot;col6&quot;, &#xA;        &quot;col7&quot;, &#xA;        &quot;col8&quot;, &#xA;        &quot;col9&quot;, &#xA;        &quot;col10&quot;, &#xA;        &quot;col11&quot;, &#xA;        &quot;col12&quot;, &#xA;        &quot;col13&quot;&#xA;    ], &#xA;    &quot;inv_scales&quot;: [&#xA;        0.29041217789420476, &#xA;        0.32026605114154144, &#xA;        0.3398879256267485, &#xA;        0.2539738260220278, &#xA;        0.27817344479641604, &#xA;        1.1217850173179438e-05, &#xA;        0.3181917203503525, &#xA;        0.2886476076886483, &#xA;        0.37842451508835384, &#xA;        2.329233011164756e-05, &#xA;        0.21830904186227362, &#xA;        2.003574563132119, &#xA;        1.386943696546877&#xA;    ]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a new input with the original values (before rescaling). How can I use the above information to rescale all the features on the new object I have to predict the results?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-09-27T12:58:48.737" Title="How to reapply rescaling when you want to predict your data" Tags="|predictive-modeling|feature-scaling|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23269" PostTypeId="2" ParentId="23264" CreationDate="2017-09-24T14:21:37.173" Score="6" Body="&lt;p&gt;Have you timed which line of your code is most time consuming? I suspect that the line &lt;code&gt;df = df[~df.isin(df1)].dropna()&lt;/code&gt; would take a long time. Would it be faster if you simply use the negation of the condition you applied to obtain &lt;code&gt;df1&lt;/code&gt;, when you want to filter away rows in &lt;code&gt;df1&lt;/code&gt; from &lt;code&gt;df&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, use &lt;code&gt;df = df[(df.col1 != 10) | (df.col2 &amp;gt; 15)]&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30567" LastActivityDate="2017-09-24T14:21:37.173" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23270" PostTypeId="2" ParentId="23245" CreationDate="2017-09-24T15:11:21.520" Score="2" Body="&lt;p&gt;The answer is yes. If Test &amp;amp; Score is given only one data set, then all it can do is show results of cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test the models on a separate data set, use separate File widgets to load training and test data. Connect File widget with training data to Test &amp;amp; Score, and the connect File widget with Test data to Test &amp;amp; Score. The connect whatever learner to test (in the workflow on the figure I've used logistic regression and random forest). Make sure &quot;Test on train data&quot; is selected in the Test &amp;amp; Score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lTEGn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lTEGn.png&quot; alt=&quot;Orange workflo with Test &amp;amp; Score widget&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39624" LastActivityDate="2017-09-24T15:11:21.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23271" PostTypeId="2" ParentId="23259" CreationDate="2017-09-24T16:18:22.067" Score="5" Body="&lt;p&gt;word2vec is probably the way to go. It maps words to a point in n-dimensional space. You can use Euclidean (or whatever distance) to find the nearest points to a given word. If training went well, the closest points should be a synonym.&lt;/p&gt;&#xA;" OwnerUserId="18903" LastActivityDate="2017-09-24T16:18:22.067" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23272" PostTypeId="2" ParentId="23264" CreationDate="2017-09-24T17:39:38.773" Score="20" Body="&lt;p&gt;The concept to understand is that the conditional is actually a vector.  So, you can simply define the conditions, and then combine them logically, like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;condition1 = (df.col1 == 10) &amp;amp; (df.col2 &amp;lt;= 15)&#xA;condition2 = (df.col3 == 7) &amp;amp; (df.col4 &amp;gt;= 4)&#xA;&#xA;# at this point, condition1 and condition2 are vectors of bools&#xA;&#xA;df1 = df[condition1]&#xA;df2 = df[condition2 &amp;amp; ~condition1]&#xA;df = df[~ (condition1 | condition2)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will be considerable faster as it only evaluates the conditional once.  Then it uses them to perform indexed lookup to create the new smaller dataframes.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-09-24T17:39:38.773" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23275" PostTypeId="2" ParentId="23165" CreationDate="2017-09-24T20:04:32.193" Score="1" Body="&lt;p&gt;&quot;Distance&quot; in k-means is the sum of squares over all attributes. It does not matter how many attributes you have.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d(a, b) := sum_i (a_i - b_i)²&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where i iterates over your &lt;em&gt;attributes&lt;/em&gt;, a is a data vector and b is a centroid vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't assume there is one, or two attributes. It's simply a parameter of the data set, how many variables it has. Could be 42 variables.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-24T20:04:32.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23276" PostTypeId="1" AcceptedAnswerId="23316" CreationDate="2017-09-24T22:59:55.597" Score="0" ViewCount="1351" Body="&lt;p&gt;I have a data frame, and I am only interested in the numerical variables (variable class is double):&lt;/p&gt;&#xA;&lt;h2&gt;Example&lt;/h2&gt;&#xA;&lt;pre&gt;&lt;code&gt;A    B    C    D    E&#xA;0.13 2.22 3.44 3.30 6.54&#xA;3.55 4.23 0.43 5.33 3.55&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I want to build a linear regression xgboost model, so I converted the data frame to a dense matrix.&lt;/p&gt;&#xA;&lt;p&gt;I took out all categorical variables from my data, and am left with the five variables of interest. But I can't seem to be able to use the dense matrix, despite it being a type of input that xgboost accepts. See &lt;a href=&quot;https://cran.r-project.org/web/packages/xgboost/vignettes/xgboostPresentation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. I am instead getting this error:&lt;/p&gt;&#xA;&lt;h2&gt;Error&lt;/h2&gt;&#xA;&lt;p&gt;&amp;quot;Error in xgb.DMatrix(train_mat[, -n], label = train_mat[, n]) : [23:57:13] amalgamation/../dmlc-core/src/io/local_filesys.cc:66: LocalFileSystem.GetPathInfo 2007 Error:No such file or directory&amp;quot;.&lt;/p&gt;&#xA;&lt;h2&gt;Code&lt;/h2&gt;&#xA;&lt;pre&gt;&lt;code&gt;train_mat &amp;lt;- as.matrix(subset(train, select = -c(Genre, Publisher, Rank, Name, Platform)))&#xA;&#xA;lm.boosted &amp;lt;- xgb.train(data = train_mat, booster= &amp;quot;gblinear&amp;quot;, max_depth=3, nthread=1, nrounds = 2, eval_metric=&amp;quot;error&amp;quot;, eval_metric= &amp;quot;rmse&amp;quot;, objective= &amp;quot;reg:linear&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I tried to convert to a xgb.DMatrix, but I am not clear on what it's asking for to do the conversion.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;train_mat &amp;lt;- xgb.DMatrix(data = train_mat, label= train$Global_Sales)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How do I input data into xgboost?&lt;/p&gt;&#xA;" OwnerUserId="39640" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-09-26T05:12:13.593" Title="Matrix class not being recognized" Tags="|r|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23277" PostTypeId="1" CreationDate="2017-09-25T01:15:39.177" Score="-1" ViewCount="266" Body="&lt;p&gt;I have done 10-Fold CV on my data and I have selected my model complexity from the result. Now for my best complexity, I will be having 10 different models. Should I take the average of these models for my testing or should I just fit a model of our selected complexity on our CV data and then test?&lt;/p&gt;&#xA;" OwnerUserId="28317" LastActivityDate="2017-10-25T13:46:36.690" Title="Cross-Validation model selection" Tags="|machine-learning|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23278" PostTypeId="2" ParentId="23276" CreationDate="2017-09-25T02:13:41.357" Score="0" Body="&lt;p&gt;Following this &lt;a href=&quot;https://www.rdocumentation.org/packages/xgboost/versions/0.6-4/topics/xgb.train#l_examples&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose the last column in &lt;code&gt;train_mat&lt;/code&gt; is the &lt;code&gt;label&lt;/code&gt; that needs to be predicted, i.e. &lt;code&gt;Global_Sales&lt;/code&gt;, then you construct the &lt;code&gt;xgb.DMatrix&lt;/code&gt; as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n &amp;lt;- ncol(train_mat)    # or if not the last, the position of the label column&#xA;dtrain &amp;lt;- xgb.DMatrix(train_mat[,-n], label=train_mat[,n])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To train the model, simply pass &lt;code&gt;dtrain&lt;/code&gt; to the &lt;code&gt;data&lt;/code&gt; parameter:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lm.boosted &amp;lt;- xgb.train(data = dtrain, booster= &quot;gblinear&quot;, max_depth=3, nthread=1, nrounds = 2, eval_metric=&quot;error&quot;, eval_metric= &quot;rmse&quot;, objective= &quot;reg:linear&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18249" LastEditorUserId="18249" LastEditDate="2017-09-25T02:39:28.007" LastActivityDate="2017-09-25T02:39:28.007" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23281" PostTypeId="2" ParentId="23268" CreationDate="2017-09-25T06:25:50.607" Score="0" Body="&lt;p&gt;Note that the approach is brittle as DSS makes no guarantee about the data format in the pickle. But as long as you don't upgrade it, you should be fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you tried the &quot;export to python notebook&quot; feature in the model page? This one is meant to be readable and could help you.&lt;/p&gt;&#xA;" OwnerUserId="3410" LastActivityDate="2017-09-25T06:25:50.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23282" PostTypeId="2" ParentId="23262" CreationDate="2017-09-25T06:46:09.953" Score="1" Body="&lt;p&gt;As @Siong They Goh wrote, since it is straightforward to check to which part the model belongs, the classification algorithm might find it in the combined set. If the partitioning will be very strong, this split will be important in the model (e.g., the root in the decision tree) and afterwards the classifier will have to learn the proper model for each part.&#xA;That will happen in a scenario in which you have unrelated parts.&#xA;Note that in such situation it won't matter much whether you will split the data set or not. The benefit that you will gain will be from preventing noise to mislead your classifier (e.g. an unrelated feature that by random looks informative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more interesting scenario will be when the division into parts is informative but there are other features that are even more informative. In this case given a joint data set, it will be easier for the classifier to use them. For example, if your data set is small, it will be less representative and the influence of noise will be stronger, two effect that will become stronger when splitting it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest that you will choose a small benchmark &lt;strong&gt;classifier&lt;/strong&gt; (e.g., a size bounded decision tree) and check its performance on both variants. This way you will be able to know which solution fits you more.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-09-25T06:46:09.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23285" PostTypeId="1" CreationDate="2017-09-25T10:58:33.607" Score="2" ViewCount="305" Body="&lt;p&gt;I am a newbie in machine learning but I have a coursework to create program that can extract some concrete features from the given text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;If I want to extract number of red apples and green apples, I will extract 3, 5 from &quot;On that tree 3 red apples and 5 green apples grow&quot; and 10, 1000 from &quot;In the box there are 1000 green apples and 10 of red&quot;.&#xA;Actually the real examples can be more complicated like a extracting a car specification from the ad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concrete features are known before the data set processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you suggest what kind of algorithms I should use? Previously, I used only linear regression.&lt;/p&gt;&#xA;" OwnerUserId="39656" LastEditorUserId="39656" LastEditDate="2017-09-25T11:28:59.207" LastActivityDate="2018-05-04T04:51:24.910" Title="Feature extraction from the text" Tags="|machine-learning|nlp|text-mining|feature-extraction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23286" PostTypeId="1" CreationDate="2017-09-25T11:13:05.307" Score="4" ViewCount="1646" Body="&lt;p&gt;I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do that? I tried topic modelling using MALLET. It groups similar words into a cluster. I want a heading or subject attached to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to group similar words under one topic and assign them a title?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There are a number of different features that need to get one topic like picture quality camera selfie resolution that should come under camera. Need something that groups these things I enter words and it gives me the topic.&lt;/p&gt;&#xA;" OwnerUserId="39661" LastEditorUserId="11097" LastEditDate="2017-09-26T04:58:32.637" LastActivityDate="2017-11-25T06:47:04.923" Title="Group similar words under one topic and assign them a title" Tags="|machine-learning|python|nlp|text-mining|nltk|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23287" PostTypeId="1" AcceptedAnswerId="23294" CreationDate="2017-09-25T11:28:13.940" Score="11" ViewCount="9891" Body="&lt;p&gt;Why  weights with large values cause neural networks to be overfitted, and consequently we use approaches like regularization to neutralize weights with large values?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-09-26T12:36:25.093" LastActivityDate="2017-09-26T12:36:25.093" Title="Why large weights are prohibited in neural networks?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23288" PostTypeId="2" ParentId="23286" CreationDate="2017-09-25T11:53:29.273" Score="2" Body="&lt;p&gt;If you have already selected the keywords you want grouped, why not write a function that finds all occurrences of words in the list and replaces it with your one, core word?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you're describing sounds like a more advanced version of &lt;a href=&quot;https://en.wikipedia.org/wiki/Stemming&quot; rel=&quot;nofollow noreferrer&quot;&gt;stemming&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-25T11:53:29.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23289" PostTypeId="2" ParentId="23287" CreationDate="2017-09-25T11:58:57.517" Score="1" Body="&lt;p&gt;Large weights might correlate with certain patterns in the input data (x), this means that the model almost hard codes certain values. This then makes our traing data fit well but our test data fit less well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use &lt;code&gt;dropout&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2017-09-25T11:58:57.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23290" PostTypeId="2" ParentId="23285" CreationDate="2017-09-25T12:01:03.700" Score="1" Body="&lt;p&gt;I guess if you know what you want to extract you can just find it using &lt;strong&gt;regular expressions&lt;/strong&gt; for integers or car specifications. &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2017-09-25T12:01:03.700" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23291" PostTypeId="1" AcceptedAnswerId="23296" CreationDate="2017-09-25T12:09:50.610" Score="2" ViewCount="57" Body="&lt;p&gt;I have a dataset with several individuals and features. I'm studying behavior over the year (for instance, averages or iterations of money gained, jobs, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My ultimate goal is to implement a classifier since I have a specific feature for every person (which is equal to 0, 1, or 2). When I first tried to implement a SVM, I ended up with bad results because I did not have enough data / features: I have too many number 1 individuals and not enough 0 and 2's, so my classifier almost always put people into category 1. Therefore, I tried to increase my number of lines by separating my data into quarters (i.e JAN, FEV, MAR, then APR, MAY, JUN, then JUL, AUG, SEPT, and finally OCT, NOV, DEC)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering two things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Would that be a good idea? Do I have to be cautious of a particular hypothesis that could impact my results?&lt;/li&gt;&#xA;&lt;li&gt;In case it is a good idea, I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have &quot;Age&quot; available for my last quarter) ; do I have to drop the feature ? Or would it be wiser to abandon the last quarter ? Or is it possible to make the classifier work despite that lack of information without actually deleting anything ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39664" LastActivityDate="2017-09-25T13:59:06.957" Title="Is it possible to implement a classifier according to quarters? What about missing data?" Tags="|machine-learning|classification|svm|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23294" PostTypeId="2" ParentId="23287" CreationDate="2017-09-25T13:42:11.713" Score="10" Body="&lt;p&gt;Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. Generally,&#xA;&lt;strong&gt;regularization&lt;/strong&gt; is any modification we make to a&#xA;learning algorithm that is intended to reduce its generalization error but not its training error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;L2 parameter norm penalty&lt;/strong&gt; commonly known as &lt;strong&gt;Weight decay&lt;/strong&gt; is one of the simplest and most common kinds of regularization technique which forces the weights to become smaller, by adding a parameter norm $Ω(θ) = 1/2 || w||^{2}_2$ penalty to the objective function.&#xA;For example, in linear regression, this gives us solutions that have a smaller slope, or put weight on fewer of the features. In other words, even though the model is capable of representing functions with much more complicated shape, weight decay has encouraged it to use a simpler function described by smaller coefficients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, in the feature space, only directions along which the parameters contribute significantly to reducing the objective function are preserved relatively intact. In directions that do not contribute to reducing the objective function, movement in this direction will not significantly increase the gradient.So, &lt;strong&gt;Components of the weight vector corresponding to such unimportant directions are decayed away through the use of the regularization throughout training&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another simple explanation is when your weights are large, they are more sensitive to small noises in the input data. So, when a small noise is propagated through your network with large weights, it produces much different value in the output layer of the NN rather than a network with small weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that &lt;em&gt;weight decay&lt;/em&gt; is not the only regularization technique. In the past few years, some other approaches have been introduced such as &lt;strong&gt;Dropout&lt;/strong&gt;, &lt;strong&gt;Bagging&lt;/strong&gt;, &lt;strong&gt;Early Stop&lt;/strong&gt;, and &lt;strong&gt;Parameter Sharing&lt;/strong&gt; which work very well in NNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other interesting findings in this very rich &lt;a href=&quot;http://www.deeplearningbook.org/contents/regularization.html&quot; rel=&quot;noreferrer&quot;&gt;chapter&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38657" LastEditorUserId="38657" LastEditDate="2017-09-25T14:00:19.760" LastActivityDate="2017-09-25T14:00:19.760" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23295" PostTypeId="2" ParentId="23277" CreationDate="2017-09-25T13:45:27.437" Score="0" Body="&lt;p&gt;You re-train the model based on your best complexity (or parameters) on the complete training data. Then you can test the model on un-touched test dataset and report your error measure.&lt;/p&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2017-09-25T13:45:27.437" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23296" PostTypeId="2" ParentId="23291" CreationDate="2017-09-25T13:59:06.957" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Would that be a good idea? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is hard to tell from your description. It is not an immediately bad idea. If it results in a better classifier (according to cross-validation), then it has probably worked. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main things that would concern me about splitting behaviour data by quarter and treating as independent are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your data samples will very likely be correlated when they share a person. You can work around this by careful splitting between training and cross-validation / test sets. Do &lt;em&gt;not&lt;/em&gt; make a fully random split, but split by person - any individuals records should appear only in one of the training, cross-validation or test sets (assuming your goal is to take similar data in production from users who are not in your current database, and predict their class).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There could be seasonal variation in the records that reduce the effectiveness of the split. So a &quot;type 1&quot; person's records in APR-JUN might look like a &quot;type 0&quot; person from JAN-MAR.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How will you receive data in production - when you want to classify new users? If you only want to work on single-quarter data, then your new classifier is fine. If you have more data, you have to deal with your classifier maybe predicting different target variable for the &lt;em&gt;same person&lt;/em&gt; depending on the quarter. You could combine these in some way - but if you do so, you should also do this in test to see what the impact of doing this is, which may be counter-productive (you end up with the same number of test examples as if you had not done the split). It might also be OK, perhaps it will add some regularisation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do I have to be cautious of a particular hypothesis that could impact my results?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You have to be very cautious about testing your classifier, because you could get data leakage from the cross-validation and test sets to the training set, which would make you think the classifier is generalising well when in fact it is not. The fix for this described above - split by person when deciding train/cv/test split.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have &quot;Age&quot; available for my last quarter) ; do I have to drop the feature ? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Handling missing data is a complicated topic in its own right, there are lots of options. You can start with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If data is missing at random (i.e. there is no reason to suspect it is related to the target variable, or only impacts certain types of record), you can substitute the mean value of that feature from the training set, or &lt;em&gt;impute&lt;/em&gt; it based on a statistical model from the other features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If data is missing for reasons that might impact the target variable, then you should give that information to the classifier, because it might be an important feature in its own right. You can take the mean or more complex imputed value as before for the original feature, but &lt;em&gt;also&lt;/em&gt; you should add a new boolean feature &quot;feature X was missing&quot;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Whether or not you should use the partial data or drop it is not possible to say in general. If you are not sure, then try both and pick the version with the best cross-validation result.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-25T13:59:06.957" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23297" PostTypeId="1" CreationDate="2017-09-25T14:23:45.510" Score="8" ViewCount="3138" Body="&lt;p&gt;I'm doing experiments with GAN. I've successfully trained GAN on 28x28px MNIST dataset (samples scaled to &lt;code&gt;(-1,1)&lt;/code&gt; interval)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My next experiment is to train GAN on bigger images. My dataset consists of grayscale 128x128px images. I've added one more convolutional layer to both generator and discriminator and updated dense layers accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample from MNIST dataset and sample from my custom dataset:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/8JYYL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8JYYL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I trained the model, all results from generator looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ftKEs.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ftKEs.png&quot; alt=&quot;single sample&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong? Why does the generator generate only this noise pattern instead of samples from my dataset distribution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Generator:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H = Sequential()&#xA;&#xA;H.add(Dense(1024, input_shape=(noise_dim,)))&#xA;H.add(BatchNormalization())&#xA;H.add(Activation('relu'))&#xA;&#xA;H.add(Dense(128*16*16))&#xA;H.add(BatchNormalization())&#xA;H.add(Activation('relu'))&#xA;H.add(Reshape([16, 16, 128]))&#xA;&#xA;H.add(Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same'))&#xA;H.add(BatchNormalization())&#xA;H.add(Activation('relu'))&#xA;&#xA;H.add(Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same'))&#xA;H.add(BatchNormalization())&#xA;H.add(Activation('relu'))&#xA;&#xA;H.add(Conv2DTranspose(32, (3, 3), strides=(2, 2), padding='same'))&#xA;H.add(BatchNormalization())&#xA;H.add(Activation('relu'))&#xA;&#xA;H.add(Conv2DTranspose(1, (3, 3), padding='same'))&#xA;H.add(Activation('tanh', name='out_image'))&#xA;&#xA;gen_in = Input(shape=(noise_dim,), name='in_noise')&#xA;gen_out = H(gen_in)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Discriminator:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H = Sequential()&#xA;&#xA;H.add(Conv2D(32, (3, 3), strides=(2, 2), padding='same', activation='relu', input_shape=(128, 128, 1)))&#xA;H.add(LeakyReLU(0.2))&#xA;&#xA;H.add(Conv2D(64, (3, 3), strides=(2, 2), padding='same', activation='relu'))&#xA;H.add(BatchNormalization())&#xA;H.add(LeakyReLU(0.2))&#xA;&#xA;H.add(Conv2D(128, (3, 3), strides=(2, 2), padding='same', activation='relu'))&#xA;H.add(BatchNormalization())&#xA;H.add(LeakyReLU(0.2))&#xA;H.add(Flatten())&#xA;&#xA;H.add(Dense(1024))&#xA;H.add(BatchNormalization())&#xA;H.add(LeakyReLU(0.2))&#xA;&#xA;H.add(Dense(1, activation='sigmoid', name='out_binary'))&#xA;&#xA;disc_in = Input(shape=(128, 128, 1), name='in_image')&#xA;disc_out = H(disc_in)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38054" LastEditorUserId="38054" LastEditDate="2017-09-25T14:57:38.977" LastActivityDate="2023-01-09T02:46:54.223" Title="Strange patterns from GAN" Tags="|deep-learning|keras|convolution|gan|generative-models|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23298" PostTypeId="1" CreationDate="2017-09-25T14:31:17.130" Score="2" ViewCount="33" Body="&lt;p&gt;Self-driving cars, arms that can learn ow to pick up objects, machines that can have conversations, etc.  I understand how neural nets work.  But I really don't know much about how neural nets are combined to form intelligent systems.  Is it all rule-based above a few base nets for tasks like e.g. image recognition?  Or are there meta-nets that do deep learning on the outputs from base nets?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things to read/references would be super-helpful!&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2022-03-06T16:49:11.917" Title="How are neural nets combined to form intelligent systems?" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23299" PostTypeId="1" AcceptedAnswerId="45547" CreationDate="2017-09-25T14:36:25.643" Score="2" ViewCount="5655" Body="&lt;p&gt;I am going through the tutorial at this &lt;a href=&quot;https://machinelearningmastery.com/implement-decision-tree-algorithm-scratch-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;site&lt;/a&gt;. Here, I can see the author is explaining the derivation of Gini Index. I want to understand the following terms&lt;br&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt; Group&#xA;&lt;li&gt; Classes : As far as I have understood, it represents the possible values of labels in the data which we are supposed to classify. Please correct me if I am wrong.&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The website &lt;a href=&quot;http://dni-institute.in/blogs/cart-decision-tree-gini-index-explained/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; states that it is the difference between 1 and the probabilities of the classified values within the dataset while creating the split. But the first link does add some more points to the simple derivation. Can anyone please explain in layman terms the derivation of Gini Index?&lt;/p&gt;&#xA;" OwnerUserId="32640" LastActivityDate="2019-02-14T01:32:26.163" Title="What exactly is a Gini Index" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23301" PostTypeId="2" ParentId="23069" CreationDate="2017-09-25T15:47:10.133" Score="2" Body="&lt;p&gt;As you've explained it, you want to detect &lt;em&gt;communities&lt;/em&gt;, which is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Community_structure#Algorithms_for_finding_communities&quot; rel=&quot;nofollow noreferrer&quot;&gt;common problem&lt;/a&gt; in &lt;a href=&quot;https://en.wikipedia.org/wiki/Social_network_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;social network analysis&lt;/a&gt;. If your network had been directed and temporal, I would have suggested you look into &lt;em&gt;influence&lt;/em&gt; analysis; e.g., as in &lt;a href=&quot;http://dl.acm.org/citation.cfm?id=3070658&quot; rel=&quot;nofollow noreferrer&quot;&gt;Detecting Communities of Authority and Analyzing Their Influence in Dynamic Social Networks&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-09-25T17:04:46.360" LastActivityDate="2017-09-25T17:04:46.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23306" PostTypeId="1" AcceptedAnswerId="23314" CreationDate="2017-09-25T21:08:47.300" Score="2" ViewCount="1469" Body="&lt;p&gt;Training a random forest model is inherently non-deterministic (absent control over the random number generator), but is predict() also non-deterministic? That is, if I construct randomForest (with an odd number for &lt;code&gt;ntree&lt;/code&gt; per the caveat in the doc) and save an &lt;code&gt;.rda&lt;/code&gt;, will loading that &lt;code&gt;.rda&lt;/code&gt; give me identical results given identical inputs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried answering this myself by looking through &lt;a href=&quot;https://github.com/cran/randomForest&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/cran/randomForest&lt;/a&gt; , and it seems that nobody along the predict() path is calling for a random number, but I'm very new to R and rather rusty with C and may be missing something. Pointers into the code or the docs will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="34089" LastEditorUserId="34089" LastEditDate="2017-09-25T22:13:59.940" LastActivityDate="2017-09-26T04:27:24.413" Title="In R's randomForest, is predict() non-deterministic?" Tags="|r|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23307" PostTypeId="5" CreationDate="2017-09-25T21:09:26.457" Score="0" Body="&lt;p&gt;Simple games such as tic-tac-toe (noughts and crosses) can make interesting and practical toy problems to apply a variety of data analysis. More complex games can challenge state of the art algorithms and are areas on ongoing research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Artificial intelligence that seeks to optimise game play may or may not be related to data science. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely to be on topic are questions about markov models, bandit algorithms and reinforcement learning. Reinforcement learning specifically is a machine learning approach where analysis or optimal play can be discovered through observing outcomes of actions taken in a game, without prior knowledge of how the game is supposed to behave or be played.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably off-topic are questions about designing games, implementing game rules in software, &quot;traditional AI&quot; search algorithms such as A*.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-25T21:09:26.457" LastActivityDate="2017-09-25T21:09:26.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23308" PostTypeId="4" CreationDate="2017-09-25T21:09:26.457" Score="0" Body="Questions related to analysing or optimising games, such as board games, card games or video games." OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-25T21:09:26.457" LastActivityDate="2017-09-25T21:09:26.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23309" PostTypeId="1" AcceptedAnswerId="23318" CreationDate="2017-09-25T21:43:15.577" Score="2" ViewCount="3529" Body="&lt;p&gt;I don't understand why using the &lt;em&gt;test set&lt;/em&gt; for model &lt;strong&gt;evaluation&lt;/strong&gt; is a bad idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I completely understand why you should not use your test set to &lt;strong&gt;train&lt;/strong&gt; your model (because in that case, you would be memorizing and you just cannot tell whether your model will generalize well or not if you don't have a separate test set). But why is it that simply using your test set to test (not train) your model is bad? You won't be changing any parameters of the model (because you are not training).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, &lt;a href=&quot;https://www.youtube.com/watch?v=aDW44NPhNw0&quot; rel=&quot;nofollow noreferrer&quot;&gt;at the end of this video&lt;/a&gt;, Luis says we are breaking what he calls the &quot;Golden rule&quot; (i.e. never use your testing data for training). However, all I can see he is doing is using the test set to &lt;em&gt;verify&lt;/em&gt; which model performs better to then be able to make a selection on which model he will use in the end.&lt;/p&gt;&#xA;" OwnerUserId="34036" LastEditorUserId="34036" LastEditDate="2017-09-25T21:59:26.000" LastActivityDate="2017-09-26T07:14:29.520" Title="Why exactly using a test set for model evaluation is a bad idea?" Tags="|training|model-evaluations|model-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23311" PostTypeId="2" ParentId="23309" CreationDate="2017-09-25T22:16:45.090" Score="2" Body="&lt;p&gt;In the linked video, a polynomial is fitted to some data. The degree of the polynomial has to be chosen, but it cannot be trained. It is therefore a hyperparameter. In order to choose the degree, several instances of the algorithm, each one trained with a polynomial of different degree, have to be evaluated on a data set that was not used for training. However, if we used the test set for assessing the values of hyperparameters, we would also be coupling the resulting model to the data in the test set, as the chosen hyperparameter (and therefore the whole model) would depend on the test set data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test set should only be used for the final evaluation in order to know how your algorithm is going to behave on data that has &lt;strong&gt;never&lt;/strong&gt; been seen before by it. This is why there is normally a validation set used to tune the hyperparameters.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-09-26T06:19:59.360" LastActivityDate="2017-09-26T06:19:59.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23313" PostTypeId="1" CreationDate="2017-09-26T03:58:44.170" Score="0" ViewCount="31" Body="&lt;p&gt;I have sentences similar to the following format &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This vulnerability happened in &lt;code&gt;Firefox&lt;/code&gt; &lt;code&gt;1.x&lt;/code&gt; before &lt;code&gt;1.8&lt;/code&gt;, &lt;code&gt;Safari&lt;/code&gt;&#xA;  &lt;code&gt;2.x&lt;/code&gt; before &lt;code&gt;2.8&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Given the above sentence, I want to extract a dictionary&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{Firefox: 1.0-1.8, Safari: 2.0-2.8}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem is how should I identify the version range with the software they belong to, using NLP techniques?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2017-09-26T09:33:19.390" Title="Identifying software and version range in a sentence" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23314" PostTypeId="2" ParentId="23306" CreationDate="2017-09-26T04:27:24.413" Score="4" Body="&lt;p&gt;The model will not change unless you re-train it.&#xA;The same input sample should always have the same output value for a given model. &#xA;changing the seed, saving and reloading, etc should have no impact on the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training itself is indeed non-deterministic, predict is not. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(randomForest)&#xA;# sample 80% of data to train&#xA;split = sample(1:nrow(iris), floor(nrow(iris) * 0.8))&#xA;df_train = iris[split,]&#xA;df_test = iris[-split,]&#xA;# rf model&#xA;mod = randomForest(Species ~ ., df_train)&#xA;# predictions&#xA;set.seed(123)&#xA;res1 = predict(mod, df_test)&#xA;set.seed(999)&#xA;res2 = predict(mod, df_test)&#xA;identical(res1, res2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18574" LastActivityDate="2017-09-26T04:27:24.413" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23316" PostTypeId="2" ParentId="23276" CreationDate="2017-09-26T05:01:04.457" Score="0" Body="&lt;p&gt;I got some help from the xgboost issues page on Github; thread &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/2746&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Made the following change to resolve the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_mat &amp;lt;- data.matrix(subset(train, select = -c(Genre, Publisher, Rank, Name, Platform)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39640" LastActivityDate="2017-09-26T05:01:04.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23317" PostTypeId="1" AcceptedAnswerId="23405" CreationDate="2017-09-26T07:09:57.567" Score="0" ViewCount="932" Body="&lt;p&gt;I have made a classification engine with only one independent field (Comments) and classified them in multiple dependent variables. Now I want to have multiple independent variables in training data (comments,state,age,gender etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently i am using python 3.6.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Example of what I have done:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Comments                                Classified&#xA;(The car wash service is good)          Positive&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example of what I want to do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Comments                          Gender    City   Age       Classified&#xA; (The car wash service is good)    Male      LA     40        Positive&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As  you can see in the second example we have 4 independent variables affecting the outcome(Classified sentiment). I want to implement a classification engine based on this use case. Please suggest how should i move ahead...???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:- I have used SGDClassifier from sklearn.linear_model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have first passed list of comments as vectors which i have vectorized using TfidfVectorizer. Below is the code of vectorization:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;&#xA;vectorizer = TfidfVectorizer(min_df=5,&#xA;                                 max_df = 0.8,&#xA;                                 sublinear_tf=True,&#xA;                                 use_idf=True)&#xA;&#xA;train_vectors = vectorizer.transform(Train_data_Comments_list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then I pass these vectors to train along with their classified labels. Below is the code(I have used partial_fit as i want to train again and again):-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier = SGDClassifier.partial_fit(train_vectors, Train_Labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I use the the classifier to classify rest of the vectors.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_vectors = vectorizer.transform(Test_data_comments_list)&#xA;prediction_liblinear = classifier.predict(test_vectors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it? &lt;/p&gt;&#xA;" OwnerUserId="39135" LastEditorUserId="836" LastEditDate="2017-09-28T07:04:46.980" LastActivityDate="2017-09-29T07:26:37.567" Title="Classification engine with multiple Textual independent fields" Tags="|machine-learning|python|classification|feature-construction|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23318" PostTypeId="2" ParentId="23309" CreationDate="2017-09-26T07:14:29.520" Score="2" Body="&lt;p&gt;&lt;em&gt;Choosing&lt;/em&gt; a variation of your model is a form of training. Just because you are not using gradient descent or whatever training process is core to a model class, does not mean your parameters are not influenced by this selection process. If you generated many thousands of models with random parameters and picked the best performing one on a data set, then this is also form of training. In fact, this is a  valid way of optimising, called &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Search&lt;/a&gt; - it is somewhat inefficient for large models, but it still works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may generate hundreds of models using the training data and using gradient descent or boosting (depending on what the training algorithm uses in your model), then select the one that performs best on cross-validation. In that case, then as well as the selection process that you intend to use this for, you are also effectively using the cv data set to fine-tune the training from the first step, using something quite similar to random search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main benefit of having two stages to testing (cv and test sets), is that you will get an &lt;em&gt;unbiased&lt;/em&gt; estimate of model performance from the test set. This is considered important enough that it has become standard practice.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-26T07:14:29.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23320" PostTypeId="2" ParentId="23297" CreationDate="2017-09-26T07:33:05.503" Score="1" Body="&lt;p&gt;Training GANs becomes much more difficult with increasing input dimensionality. I encountered that sometimes even small changes to your network can decide about convergence/divergence of your network (like initialization of your variables). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend following the tipps from &lt;a href=&quot;https://github.com/soumith/ganhacks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/soumith/ganhacks&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing I personally recognized is that you should not use as many Conv2DTranspose layers as you do in your current network. Try instead increasing the latent space.&lt;/p&gt;&#xA;" OwnerUserId="37330" LastActivityDate="2017-09-26T07:33:05.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23322" PostTypeId="1" CreationDate="2017-09-26T08:32:37.200" Score="2" ViewCount="1762" Body="&lt;p&gt;It is suggested in SVM to experiment with different classifiers using the various kernels available.Tuning the parameters is advised too.I am experimenting with various gamma and regularization parameters. Is there a way to print 'margin' for each experiment? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m1gSE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m1gSE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35503" LastActivityDate="2017-11-09T22:24:27.003" Title="Is there a way to print SVM margin obtained for each kernel used in sklearn?" Tags="|machine-learning|scikit-learn|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23323" PostTypeId="1" AcceptedAnswerId="26921" CreationDate="2017-09-26T08:56:30.433" Score="10" ViewCount="20073" Body="&lt;p&gt;I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use such dataset for topic detection of various sentences or paragraphs. I was planning to train a classifier with such a dataset and use it for predictions. However, I couldn't find any. Are there any such known datasets available? &lt;/p&gt;&#xA;" OwnerUserId="25183" LastEditorUserId="25183" LastEditDate="2017-09-28T10:24:30.060" LastActivityDate="2021-10-22T07:06:00.557" Title="Public dataset for news articles with their associated categories" Tags="|machine-learning|data-mining|nlp|dataset|text-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23325" PostTypeId="2" ParentId="23313" CreationDate="2017-09-26T09:33:19.390" Score="1" Body="&lt;p&gt;If all your sentences are in the following format(atleast that what I assume), you can parse it with regex instead.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;`([A-Z][a-z]+)\s+(\d+\.\d+)(?:\s+before\s+)(\d+\.\d+)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this pattern with the given sentence finds all the software and the versions in it. &lt;code&gt;re.findall(pattern, sentence)&lt;/code&gt; will give you a list of tuples like this&#xA;&lt;code&gt;[('Firefox', '2.4', '2.5'), ('Safari', '6.0', '6.8')]&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="4868" LastActivityDate="2017-09-26T09:33:19.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23326" PostTypeId="1" CreationDate="2017-09-26T10:07:27.253" Score="1" ViewCount="1555" Body="&lt;p&gt;Can anybody provide any links for any training dataset for sentiment analysis on Dutch languages? I googled trying to find maybe movie review on Dutch but didn't find website with persons review like in IMDB&lt;/p&gt;&#xA;" OwnerUserId="39706" LastActivityDate="2017-10-26T14:08:34.897" Title="Suggest DataSet for Sentiment Analysis on Dutch languages" Tags="|sentiment-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23327" PostTypeId="2" ParentId="23326" CreationDate="2017-09-26T10:29:39.970" Score="1" Body="&lt;p&gt;Check out this free Dutch &lt;a href=&quot;http://corporafromtheweb.org/nlcow14/&quot; rel=&quot;nofollow noreferrer&quot;&gt;corpora&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39663" LastActivityDate="2017-09-26T10:29:39.970" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23328" PostTypeId="1" CreationDate="2017-09-26T10:30:08.187" Score="-1" ViewCount="2429" Body="&lt;p&gt;I have a dataset with sales numbers for around 100 related products. Every day, the number of sales of each product is recorded along with other relevant information (what day of the week is it, is it a public holiday, what is the weather like etc. etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So essentially this is a time series with daily entries, and I am thinking of pushing this through an LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how do I deal with the fact that I have multiple observations at every point in time?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Day         Product        Wheather        NumberSold&#xA;1 Jan       Meat           Sunny           15&#xA;1 Jan       Apples         Sunny           211&#xA;1 Jan       Fries          Sunny           5&#xA;...&#xA;1 Jan       Carrots        Cloudy          75&#xA;2 Jan       Meat           Cloudy          10&#xA;2 Jan       Apples         Cloudy          220&#xA;...&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do I have to divide my dataset up by product so as to have only one entry per day to feed into the LSTM? Or is there a way to deal with all of the 1 Jan observations as a sort of batch that is remembered? &lt;/p&gt;&#xA;" OwnerUserId="35454" LastActivityDate="2021-11-07T23:04:10.420" Title="LSTM with multiple entries per time step" Tags="|neural-network|lstm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23330" PostTypeId="2" ParentId="23322" CreationDate="2017-09-26T10:37:06.080" Score="1" Body="&lt;p&gt;You can check out &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane.html#sphx-glr-auto-examples-svm-plot-separating-hyperplane-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; example on the &lt;code&gt;sklearn&lt;/code&gt; site.&lt;/p&gt;&#xA;" OwnerUserId="39663" LastEditorUserId="28175" LastEditDate="2017-11-09T22:24:27.003" LastActivityDate="2017-11-09T22:24:27.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23331" PostTypeId="2" ParentId="6107" CreationDate="2017-09-26T10:38:53.320" Score="6" Body="&lt;p&gt;In addition to David Dao's answer: It is also possible to think the other way around. Instead of focusing on which (low resolution) input pixels are used to produce a single output pixel, you can also focus on which individual input pixels contribute to which region of output pixels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is done in &lt;a href=&quot;https://distill.pub/2016/deconv-checkerboard&quot; rel=&quot;noreferrer&quot;&gt;this distill publication&lt;/a&gt;, including a series of very intuitive and interactive visualizations. One advantage of thinking in this direction is that explaining checkerboard artifacts becomes easy. &lt;/p&gt;&#xA;" OwnerUserId="39708" LastEditorUserId="14675" LastEditDate="2017-09-26T13:29:12.657" LastActivityDate="2017-09-26T13:29:12.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23332" PostTypeId="2" ParentId="23323" CreationDate="2017-09-26T11:38:58.943" Score="5" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/datasets/twenty_newsgroups.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This dataset&lt;/a&gt; is included with scikit-learn, a popular ML library for Python.&lt;/p&gt;&#xA;&lt;p&gt;It is postings to Usenet and categorized by the group. The group titles are not exactly &amp;quot;categories&amp;quot; like you would see on Google News, but each newsgroup is supposed to be on a specific topic as indicated by the name, so the concepts are similar. For example:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;alt.atheism, - Atheism&lt;/li&gt;&#xA;&lt;li&gt;comp.graphics, - Computer Graphics&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;li&gt;rec.autos - Automobiles&lt;/li&gt;&#xA;&lt;li&gt;rec.motorcycles - Motorcycles&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="98307" LastEditDate="2020-08-17T07:39:54.300" LastActivityDate="2020-08-17T07:39:54.300" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23334" PostTypeId="1" CreationDate="2017-09-26T14:27:35.077" Score="3" ViewCount="518" Body="&lt;p&gt;I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset of &gt; 100,000 events, where each event has a winner.&#xA;I have plenty of data points, some data on the event itself, and some data on each entrant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of entrants in each event is variable, and I want to build a neural network around picking a likely winner of the events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the number of entrants is variable, what appears to be common advice is to have enough inputs for the maximum case scenario, and 0 them out for events where there are empty slots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This feels somewhat inelegant, and I had a slightly different idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was going to have a NN where the inputs are information about the event, and information about 1 entrant. I would then have a single output (a float between 0 and 1). I would run this through, getting 1 output for each entrant in an event, then I would be left with a number of floats, equal to the number of entrants in the event. I would then select the highest value, and use the entrant that refers to as the choice for the winner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a reason I shouldn't be doing it this way? Is there a better solution I haven't yet come across?&lt;/p&gt;&#xA;" OwnerUserId="39715" LastActivityDate="2024-01-14T10:06:45.920" Title="Genetic neural network to satisfy variable number of inputs and outputs" Tags="|neural-network|dataset|genetic-algorithms|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23335" PostTypeId="2" ParentId="23334" CreationDate="2017-09-26T15:07:21.190" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a reason I shouldn't be doing it this way?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Depends on the nature of the data. There might be an element of &quot;Scissor/Paper/Stone&quot; in the competition you are scoring, where different strengths and weaknesses of competitors can combine such that Player A beats Player B, Player B beats Player C, but Player C beats Player A. In that case, you cannot produce reliable ranking between players by considering each entrant separately, and a network that rates each player individually will perform less well than one that can compare players.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If players are in more of a race-to-finish or score max points separately in a competition, then separately rating each player in each competition should be more reliable. And it is definitely easier to build and train a neural network to predict that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative, if your events are more like tournaments where entrants oppose each other (even if within some larger free-for-all), is to predict relative rank between pairs of players. This may not be consistent, so you will need to use a pairwise ranking method to resolve that for the final winner. If it really is a knockout tournament, and you know how the initial draw and team combinations will work, then you could maybe make a prediction by simulating the possible games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing preventing you from combining these approaches in some way either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whichever method you use, you will want to think a little about what your metric is going to be to select the best approach. If you only care about predicting the winner, then accuracy of that prediction might be enough. If you care about where the eventual winner is placed, perhaps &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_reciprocal_rank&quot; rel=&quot;nofollow noreferrer&quot;&gt;mean reciprocal rank&lt;/a&gt; would be better (score 1 for correct prediction, 1/2 for predicting winner as ranked second, 1/3 if third etc).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-09-26T16:03:52.337" LastActivityDate="2017-09-26T16:03:52.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23337" PostTypeId="2" ParentId="15553" CreationDate="2017-09-26T16:54:07.363" Score="0" Body="&lt;p&gt;You would have to run a set of artificial tests, trying to detect relevant features using different methods while knowing in advance which subsets of input variables affect the output variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good trick would be to keep a set of random input variables with different distributions and make sure your feature selection algos indeed tag them as not relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another trick would be to make sure that after permuting rows the variables tagged as relevant stop being classified as relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Above said applies to both filter and wrapper approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also be sure to handle cases when if taken separately (one by one) variables do not show any influence on the target, but when taken jointly reveal a strong dependence. Example would be a well-known XOR problem (check out the Python code):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.feature_selection import f_regression, mutual_info_regression,mutual_info_classif&#xA;&#xA;x=np.random.randn(5000,3)&#xA;y=np.where(np.logical_xor(x[:,0]&amp;gt;0,x[:,1]&amp;gt;0),1,0)&#xA;&#xA;plt.scatter(x[y==1,0],x[y==1,1],c='r',marker='x')&#xA;plt.scatter(x[y==0,0],x[y==0,1],c='b',marker='o')&#xA;plt.show()&#xA;print(mutual_info_classif(x, y)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/14ry4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/14ry4.png&quot; alt=&quot;Python 3 XOR illustration&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[ 0.          0.          0.00429746]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, presumably powerful (but univariate) filtering method (computing of mutual information between out- and input variables) was not able to detect any relationships in the dataset. Whereas we know for sure it's a 100% dependency and we can predict Y with 100% accuracy knowing X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good idea would be to create a kind of benchmark for features selection methods, does anyone want to participate?&lt;/p&gt;&#xA;" OwnerUserId="39722" LastEditorUserId="29575" LastEditDate="2017-09-27T08:34:53.700" LastActivityDate="2017-09-27T08:34:53.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23338" PostTypeId="1" AcceptedAnswerId="23342" CreationDate="2017-09-26T18:21:56.100" Score="0" ViewCount="172" Body="&lt;p&gt;I want to train neural network on aesthetics. I am getting confused on how to go about for training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume, I have large data set of landscapes, portraits, wildlife etc which are aesthetic according to humans. But, I want to train them for the quality, the kind of colours involved, contrast levels, background blur etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I train images for this criteria? Is there a way for doing this by unsupervised learning?&lt;/p&gt;&#xA;" OwnerUserId="9003" LastActivityDate="2017-09-26T22:56:28.643" Title="Deep learning - aesthetics data modelling" Tags="|neural-network|deep-learning|dataset|keras|data-cleaning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23339" PostTypeId="2" ParentId="9134" CreationDate="2017-09-26T18:27:03.107" Score="8" Body="&lt;p&gt;I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based on the following nice explanation of gradient boosting &lt;a href=&quot;http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example aims to predict salary per month (in dollars) based on whether or not the observation has own house, own car and own family/children. Suppose we have a dataset of three observations where the first variable is 'have own house', the second is 'have own car' and the third variable is 'have family/children', and target is 'salary per month'. The observations are &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.- (Yes,Yes, Yes, 10000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.-(No, No, No, 25)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.-(Yes,No,No,5000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose a number &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; of boosting stages, say &lt;span class=&quot;math-container&quot;&gt;$M=1$&lt;/span&gt;. The first step of gradient boosting algorithm is to start with an initial model &lt;span class=&quot;math-container&quot;&gt;$F_{0}$&lt;/span&gt;. This model is a constant defined by &lt;span class=&quot;math-container&quot;&gt;$\mathrm{arg min}_{\gamma}\sum_{i=1}^3L(y_{i},\gamma)$&lt;/span&gt; in our case, where &lt;span class=&quot;math-container&quot;&gt;$L$&lt;/span&gt; is the loss function. Suppose that we are working with the usual loss function &lt;span class=&quot;math-container&quot;&gt;$L(y_{i},\gamma)=\frac{1}{2}(y_{i}-\gamma)^{2}$&lt;/span&gt;. When this is the case, this constant is equal to the mean of the outputs &lt;span class=&quot;math-container&quot;&gt;$y_{i}$&lt;/span&gt;, so in our case &lt;span class=&quot;math-container&quot;&gt;$\frac{10000+25+5000}{3}=5008.3$&lt;/span&gt;. So our initial model is &lt;span class=&quot;math-container&quot;&gt;$F_{0}(x)=5008.3$&lt;/span&gt; (which maps every observation &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; (e.g. (No,Yes,No)) to 5008.3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next we should create a new dataset, which is the previous dataset but instead of &lt;span class=&quot;math-container&quot;&gt;$y_{i}$&lt;/span&gt; we take the residuals &lt;span class=&quot;math-container&quot;&gt;$r_{i0}=-\frac{\partial{L(y_{i},F_{0}(x_{i}))}}{\partial{F_{0}(x_{i})}}$&lt;/span&gt;. In our case, we have &lt;span class=&quot;math-container&quot;&gt;$r_{i0}=y_{i}-F_{0}(x_{i})=y_{i}-5008.3$&lt;/span&gt;. So our dataset becomes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.- (Yes,Yes, Yes, 4991.6)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.-(No, No, No, -4983.3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.-(Yes,No,No,-8.3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next step is to fit a base learner &lt;span class=&quot;math-container&quot;&gt;$h$&lt;/span&gt; to this new dataset. Usually the base learner is a decision tree, so we use this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now assume that we constructed the following decision tree &lt;span class=&quot;math-container&quot;&gt;$h$&lt;/span&gt;. I constructed this tree using entropy and information gain formulas but probably I made some mistake, however for our purposes we can assume it's correct. For a more detailed example, please check &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.saedsayad.com/decision_tree.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.saedsayad.com/decision_tree.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The constructed tree is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yRjle.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yRjle.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's call this decision tree &lt;span class=&quot;math-container&quot;&gt;$h_{0}$&lt;/span&gt;. The next step is to find a constant &lt;span class=&quot;math-container&quot;&gt;$\lambda_{0}=\mathrm{arg\;min}_{\lambda}\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$&lt;/span&gt;. Therefore, we want a constant &lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt; minimizing &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C=\frac{1}{2}(10000-(5008.3+\lambda*{4991.6}))^{2}+\frac{1}{2}(25-(5008.3+\lambda(-4983.3)))^{2}+\frac{1}{2}(5000-(5008.3+\lambda(-8.3)))^{2}$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where gradient descent comes in handy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that we start at &lt;span class=&quot;math-container&quot;&gt;$P_{0}=0$&lt;/span&gt;. Choose the learning rate equal to &lt;span class=&quot;math-container&quot;&gt;$\eta=0.01$&lt;/span&gt;. We have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial{C}}{\partial{\lambda}}=(10000-(5008.3+\lambda*4991.6))(-4991.6)+(25-(5008.3+\lambda(-4983.3)))*4983.3+(5000-(5008.3+\lambda(-8.3)))*8.3$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then our next value &lt;span class=&quot;math-container&quot;&gt;$P_{1}$&lt;/span&gt; is given by &lt;span class=&quot;math-container&quot;&gt;$P_{1}=0-\eta{\frac{\partial{C}}{\partial{\lambda}}(0)}=0-.01(-4991.6*4991.7+4983.4*(-4983.3)+(-8.3)*8.3)$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat this step &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; times, and suppose that the last value is &lt;span class=&quot;math-container&quot;&gt;$P_{N}$&lt;/span&gt;. If &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; is sufficiently large and &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; is sufficiently small then &lt;span class=&quot;math-container&quot;&gt;$\lambda:=P_{N}$&lt;/span&gt; should be the value where &lt;span class=&quot;math-container&quot;&gt;$\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$&lt;/span&gt; is minimized. If this is the case, then our &lt;span class=&quot;math-container&quot;&gt;$\lambda_{0}$&lt;/span&gt; will be equal to &lt;span class=&quot;math-container&quot;&gt;$P_{N}$&lt;/span&gt;. Just for the sake of it, suppose that &lt;span class=&quot;math-container&quot;&gt;$P_{N}=0.5$&lt;/span&gt; (so that &lt;span class=&quot;math-container&quot;&gt;$\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$&lt;/span&gt; is minimized at &lt;span class=&quot;math-container&quot;&gt;$\lambda:=0.5$&lt;/span&gt;). Therefore, &lt;span class=&quot;math-container&quot;&gt;$\lambda_{0}=0.5$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next step is to update our initial model &lt;span class=&quot;math-container&quot;&gt;$F_{0}$&lt;/span&gt; by &lt;span class=&quot;math-container&quot;&gt;$F_{1}(x):=F_{0}(x)+\lambda_{0}h_{0}(x)$&lt;/span&gt;. Since our number of boosting stages is just one, then this is our final model &lt;span class=&quot;math-container&quot;&gt;$F_{1}$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that I want to predict a new observation &lt;span class=&quot;math-container&quot;&gt;$x=$&lt;/span&gt;(Yes,Yes,No) (so this person does have own house and own car but no children). What is the salary per month of this person? We just compute &lt;span class=&quot;math-container&quot;&gt;$F_{1}(x)=F_{0}(x)+\lambda_{0}h_{0}(x)=5008.3+0.5*4991.6=7504.1$&lt;/span&gt;. So this person earns $7504.1 per month according to our model.&lt;/p&gt;&#xA;" OwnerUserId="23759" LastEditorUserId="23759" LastEditDate="2018-09-29T22:17:22.910" LastActivityDate="2018-09-29T22:17:22.910" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23340" PostTypeId="2" ParentId="23338" CreationDate="2017-09-26T18:51:25.157" Score="2" Body="&lt;p&gt;Your question is not properly framed. Deep learning optimizes a function that maps input variables to output variables. Your input variables are the pictures (specifically, the combination of RGB pixel values). What are your output variables? That is, what specifically do you want the network to predict?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to get the network to simply rate how aesthetically pleasing an image is on a scale of 1-10, that is fairly straightforward: simply have a large sample of human volunteers rate the images on this scale, and train the network to predict those ratings. If done properly, the network will then be able to give a fairly sophisticated prediction of whether or not a given image is aesthetically pleasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to get the network to &lt;em&gt;create&lt;/em&gt; aesthetically pleasing images, you have a long road ahead of you - this is an unsolved problem and an area of open research. Check out &lt;a href=&quot;https://medium.com/towards-data-science/gangogh-creating-art-with-gans-8d087d8f74a1&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to train a neural network to &lt;em&gt;understand&lt;/em&gt; what qualities make an image aesthetically by detecting underlying patterns and communicating them to humans, you are probably using the wrong tool.&lt;/p&gt;&#xA;" OwnerUserId="33237" LastActivityDate="2017-09-26T18:51:25.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23341" PostTypeId="1" AcceptedAnswerId="23377" CreationDate="2017-09-26T22:13:18.027" Score="25" ViewCount="30131" Body="&lt;p&gt;I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Adding more details from comments below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new to this so guidance is appreciated. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Memory&lt;/strong&gt;: GPU is K80&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Framework&lt;/strong&gt;: Cuda and cuDNN &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Data size per workloads&lt;/strong&gt;: 20G&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Computing nodes to consume&lt;/strong&gt;: one per job, although would like to consider a scale option &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cost&lt;/strong&gt;: I can afford a GPU option if the reasons make sense&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Deployment&lt;/strong&gt;: Running on own hosted bare metal servers, not in the cloud.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU. &lt;/p&gt;&#xA;" OwnerUserId="39733" LastEditorUserId="24000" LastEditDate="2017-09-27T10:11:41.540" LastActivityDate="2020-11-29T13:09:32.713" Title="Should I use GPU or CPU for inference?" Tags="|deep-learning|gpu|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23342" PostTypeId="2" ParentId="23338" CreationDate="2017-09-26T22:56:28.643" Score="3" Body="&lt;p&gt;Modeling aesthetics in media is an example of ordinal classification.&#xA;One of the most actively maintained datasets for this can be found is &lt;a href=&quot;http://www.inf-cv.uni-jena.de/en/jenaesthetics&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jen Aesthetics&lt;/a&gt;&#xA;A relatively recent paper using deep learning towards aesthetics modeling is&#xA;&lt;a href=&quot;http://hera.inf-cv.uni-jena.de:6680/pdf/Denzler16_CNNA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to deep learning era, research groups were trying to translate methods/guidelines used in the photography community to create/capture good quality pictures. There are several guidelines that you can explore with a bit of search online. One popular example is the 'rule of thirds'. Here the primary subject should not be centered in the image but offset and ideally centered at the intersection of 1/3 and 2/3 horizontal and vertical lines. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/uAc7c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uAc7c.png&quot; alt=&quot;rule-of-thirds&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is easy to translate into an algorithm: use salient object recognition or visual attention detection and measure the distance of the center of the salient/attention patch from the 4 rule-of-thirds points. Use this off-set as a feature. The closer the salient patch is to any one of the rule-of-third points, the higher the aesthetic ordinal score for that image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://people.csail.mit.edu/khosla/papers/www2014_khosla.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is another good paper that explores what makes images popular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some researchers have also used the tags or descriptions of photos as features. The objective here is to learn an association between lexical features and image aesthetics. They have sourced their data from online repositories like &lt;a href=&quot;http://www.dpchallenge.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Digital Photography Challenge&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This subjective task is needless to say very complex. If you plan to address it, I'd recommend beginning with a clear definition of the context within which you aim to address aesthetics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, you'd like to map any given image (media) to some value in [0, ..., 1] $\in \mathcal{R}$.&#xA;However, this is very difficult unless you have access to a lot of training data. I suggest trying instead to simplify the problem. If you can reliably map images to just two classes, good aesthetics and bad aesthetics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can successively generalize from binary classification to full fledged multi-class ordinal classification, for which you'll very likely have to keep increasing the depth of your CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck! Since, there is more to aesthetics than meets the eye! :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/70Ua2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/70Ua2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-09-26T22:56:28.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23343" PostTypeId="1" CreationDate="2017-09-27T01:05:07.040" Score="4" ViewCount="729" Body="&lt;p&gt;I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes probabilities? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say our model is two trees. A terminal node of tree 1 has example 14 in a node with 20% class 1, 60% class 2, and 20% class 3. A terminal node of tree 2 has example 14 in a node with 100% class 2. Then our prediction for training example 14 is [10%, 80%, 10%].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why use Softmax instead of this averaging approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am looking to apply this knowledge to understanding xgboost better, as well as a simple 1 tree classification model.&lt;/p&gt;&#xA;" OwnerUserId="39738" LastEditorUserId="303" LastEditDate="2020-03-19T21:57:27.703" LastActivityDate="2024-03-20T03:03:15.873" Title="Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?" Tags="|decision-trees|xgboost|softmax|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23344" PostTypeId="1" CreationDate="2017-09-27T02:06:41.013" Score="0" ViewCount="156" Body="&lt;p&gt;I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (and content not always as sentences). Is there any better approach apart from writing custom web scraper/parser? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tired using Named Entity Recognition and Part of speech tagging approach, but not sure if it helps in this case (or is there something about NER I am missing?)&lt;/p&gt;&#xA;" OwnerUserId="39741" LastEditorUserId="471" LastEditDate="2017-09-27T06:43:28.050" LastActivityDate="2017-09-27T07:00:06.957" Title="Extracting specific information from multiple unstructured website" Tags="|python|text-mining|nltk|scraping|" AnswerCount="1" CommentCount="0" ClosedDate="2017-09-27T10:33:45.457" ContentLicense="CC BY-SA 3.0" />
  <row Id="23345" PostTypeId="2" ParentId="23344" CreationDate="2017-09-27T07:00:06.957" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there any better approach apart from writing custom web scraper/parser?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Probably not. Although this problem might be tractable using machine learning and natural language processing techniques in future, it would be far more work, and a much larger challenge, than the typical 80/20 rule you get from a web scraper plus regular expressions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. you write a web scraper with your best guess at the rules for extracting names. It finds 800 names correctly out of 1000. You analyse the problem sites, and add custom CSS selectors and tweak some of the regular expressions. That finds 150 of the remaining 200. Repeat the analysis and fix, you are down to maybe 10 problem sites, each requiring an individual fix. It is dull, repetitive work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The alternative of attempting this problem using machine learning and NLP: First get the true labels for 10,000+ sites . . . &lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-27T07:00:06.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23346" PostTypeId="2" ParentId="23268" CreationDate="2017-09-27T07:10:50.600" Score="2" Body="&lt;p&gt;Adding to jrouquie's answer, the formula would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each column,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rescaled_feature = (input_feature - shift)  * inv_scale&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39653" LastActivityDate="2017-09-27T07:10:50.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23348" PostTypeId="2" ParentId="23341" CreationDate="2017-09-27T09:18:29.090" Score="5" Body="&lt;p&gt;You'd only use GPU for &lt;strong&gt;training&lt;/strong&gt; because deep learning requires massive calculation to arrive at an optimal solution. However, you don't need GPU machines for &lt;strong&gt;deployment&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take Apple's new iPhone X as an example. The new iPhone X has an advanced machine learning algorithm for facical detection. Apple employees must have a cluster of machines for training and validation. But your iPhone X doesn't need a GPU for just running the model.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-09-27T09:18:29.090" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="23350" PostTypeId="1" AcceptedAnswerId="23351" CreationDate="2017-09-27T10:18:10.290" Score="0" ViewCount="466" Body="&lt;p&gt;For example, let's say that the age (say $x$) of a person for $12&amp;lt; x&amp;lt; 25$ can be used to predict computer usage to a high degree of certainty. In a decision tree, this could be represented by a split of $x&amp;lt;25$ followed by a split of $x&amp;gt;12$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does xgboost, or other decision tree learning algorithms in general, handle multiple splits within the same branch? &lt;/p&gt;&#xA;" OwnerUserId="39755" LastEditorUserId="29575" LastEditDate="2017-09-27T15:09:25.027" LastActivityDate="2017-09-27T15:09:25.027" Title="Do xgboost and random forests in general handle multiple splits of the same numeric feature in a single branch?" Tags="|random-forest|decision-trees|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23351" PostTypeId="2" ParentId="23350" CreationDate="2017-09-27T11:13:51.180" Score="1" Body="&lt;p&gt;Yes, it can handle multiple splits withing the same branch.&lt;br&gt;&#xA;A decision tree model can use the same feature as many times as optimally needed.&lt;br&gt;&#xA;You can see an example in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn doc&lt;/a&gt;.&lt;br&gt;&#xA;The petal-length feature is used multiple times in the same branch.&lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2017-09-27T11:13:51.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23352" PostTypeId="1" CreationDate="2017-09-27T11:24:08.637" Score="2" ViewCount="806" Body="&lt;p&gt;I am using TF-IDF for text classification and have been curious about the following two concepts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;augmented term frequency&lt;/em&gt; which is basically used for weighting in order to eliminate the bias towards longer documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, there is a &lt;em&gt;cosine normalization&lt;/em&gt; which appears to be done for the same purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are they both similar? Can we use any one of them to eliminate the bias towards larger documents?&lt;/p&gt;&#xA;" OwnerUserId="39756" LastEditorUserId="29575" LastEditDate="2017-09-27T15:15:07.407" LastActivityDate="2017-09-27T15:15:07.407" Title="TF-IDF Augmented Frequency vs Cosine Normalization" Tags="|python|scikit-learn|text-mining|tfidf|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23353" PostTypeId="2" ParentId="16219" CreationDate="2017-09-27T11:48:31.920" Score="0" Body="&lt;p&gt;Adding to [G5W's answer][&lt;a href=&quot;https://datascience.stackexchange.com/a/16249/29575]&quot;&gt;https://datascience.stackexchange.com/a/16249/29575]&lt;/a&gt; that FSelector (and possibly the other implementations as well) performs a discretization of features before calculating scores. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For FSelector, this is done in the file selector.info.gain.R. You can check out the discretization with &lt;code&gt;FSelector:::discretize.all&lt;/code&gt;. This step removes information to the extent that the ordering of features is altered. &lt;/p&gt;&#xA;" OwnerUserId="39757" LastEditorUserId="29575" LastEditDate="2017-09-28T05:56:00.870" LastActivityDate="2017-09-28T05:56:00.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23354" PostTypeId="1" CreationDate="2017-09-27T11:52:14.760" Score="0" ViewCount="57" Body="&lt;p&gt;I am clustering based on my cosine similarity matrix. Now I want to plot/visualize my very large clusters. I am interested in using a tool that is better than sklearn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please recommend me.&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2017-09-28T20:10:26.923" Title="What are the tools to plot cluster results?" Tags="|machine-learning|python|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23357" PostTypeId="1" CreationDate="2017-09-27T12:40:14.377" Score="5" ViewCount="782" Body="&lt;p&gt;I am have a data set with 52 variables. Most of them have zeros, it resembles a sparse matrix. How can I cluster this kind of data and are there any special types of clustering? I am attaching pca plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bgCFJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bgCFJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38145" LastEditorUserId="23240" LastEditDate="2019-03-14T16:12:11.820" LastActivityDate="2020-02-14T10:01:35.413" Title="Clustering for high dimensional data" Tags="|machine-learning|r|data-mining|clustering|unsupervised-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="23358" PostTypeId="2" ParentId="22136" CreationDate="2017-09-27T12:44:51.050" Score="4" Body="&lt;p&gt;Aggregate discount sequences are used to define the stacking logic for aggregate discounts. This logic can define:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The order in which aggregate discounts are applied.&lt;/li&gt;&#xA;&lt;li&gt;Which aggregate discounts are mutually exclusive.&lt;/li&gt;&#xA;&lt;li&gt;Which aggregate discounts can be &quot;stacked,&quot; or applied in conjunction with one another.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can use aggregate discounts within a quote or an order only if they are part of an aggregate discount sequence. Therefore, you must define an aggregate discount sequence even if you have only one aggregate discount. An aggregate discount can be a part of many aggregate discount sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aggregate discount sequences are linked with price list and discount matrix line items (like volume discounts and attribute adjustments). At run time, the pricing procedure picks the aggregate discount sequence to be used and evaluates the aggregate discounts to be applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The order in which aggregate discounts are applied within an aggregate discount sequence can affect the resulting price. For example, if aggregate discount (a) gives a discount of $100, and aggregate discount (b) gives you 10% off the price of the product, the resulting price would be lower if aggregate discount (b) is applied before aggregate discount (a).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To apply all the aggregate discounts in a sequence, you attach the sequence to either a price list or a product with component line items. A sequence includes all the aggregate discounts that apply to one price list or to one product with components in a price list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the following procedure to define the aggregate discount sequence. This task is a step in Process of Creating Aggregate Discounts.&lt;/p&gt;&#xA;" OwnerUserId="38145" LastEditorUserId="29575" LastEditDate="2017-09-28T05:56:10.490" LastActivityDate="2017-09-28T05:56:10.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23360" PostTypeId="2" ParentId="23268" CreationDate="2017-09-27T12:58:48.737" Score="0" Body="&lt;p&gt;In neural network We use scaling  (max-min)+min&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a&amp;lt;-(max(s$Cvs)-min(s$Cvs))+min(s$Cvs)&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-09-27T12:58:48.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23361" PostTypeId="2" ParentId="6908" CreationDate="2017-09-27T13:09:10.870" Score="0" Body="&lt;p&gt;&lt;strong&gt;You can opt for Apache spark it has API for languages like JAVA,Python,R .Spark is very good at performing operations on large scale data&lt;/strong&gt;.It is having lot of inbuilt functions to clean and data is stored in spark Data frames. Machine learning algorithms are also implemented in spark&#xA;&lt;a href=&quot;https://spark.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is relevant information&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-09-27T13:09:10.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23362" PostTypeId="1" AcceptedAnswerId="23367" CreationDate="2017-09-27T13:43:42.413" Score="1" ViewCount="699" Body="&lt;p&gt;I am experimenting with BagginClassifier, but I fail to get the expected functionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the BagginClassifier should draw (bootstrapping) a new data set with replacement. For example: the following code should generate a new bootstrapped sample of the same size as the original data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sklearn.datasets as ds&#xA;import numpy as np&#xA;&#xA;&#xA;X, y = ds.load_iris().data, ds.load_iris().target&#xA;&#xA;bag = BaggingClassifier(base_estimator=LogisticRegression(), &#xA;                        n_estimators=100,&#xA;                        max_samples=1.0,&#xA;                        bootstrap=True,&#xA;                        n_jobs=1)&#xA;bag.fit(X, y)&#xA;&#xA;print X[bag.estimators_samples_[0]].shape&#xA;&amp;gt;&amp;gt; 95&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(or any other number close to 95).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naively, I would expect to get the bootstrapped sample of the same size as the original one (150), but with some random repetition of rows. However, I get a smaller sample size with unique rows. That's strange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's wrong here?&lt;/p&gt;&#xA;" OwnerUserId="14070" LastActivityDate="2017-10-01T05:22:56.963" Title="question about `sklearn.ensemble.BaggingClassifier`" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23367" PostTypeId="2" ParentId="23362" CreationDate="2017-09-27T14:40:32.530" Score="2" Body="&lt;p&gt;Found the answer hiding in lines 93-100 in the bagging.py file.&lt;br&gt;&#xA;Here is what I understand - the bootstrapping process works in three steps:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Calculate number of samples to train each estimator on (the max_samples variable in the bagging.py code). In your case its $1.0 * x.shape[0] = 150$.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Select with repetition the needed max samples(as calculated in the previous step).&lt;br&gt;&#xA;The selection is done by using &lt;code&gt;randint&lt;/code&gt; function, and it generates an array of the &lt;code&gt;x&lt;/code&gt; series indices. A given index can appear in this array more than once.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In order to account for indices(samples) that were selected more than one, a weights vector is passed into the base estimator fit function. So, for example, a sample that was selected twice will have $weight * 2$ and it will have the desired impact on the fitting algorithm.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As to my understanding, one can use the &lt;code&gt;estimators_samples_&lt;/code&gt; only to find out which samples were included and not how many times each of them did.&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="27983" LastEditDate="2017-10-01T05:22:56.963" LastActivityDate="2017-10-01T05:22:56.963" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23369" PostTypeId="1" CreationDate="2017-09-27T15:01:37.513" Score="3" ViewCount="221" Body="&lt;p&gt;My agenda is finding patterns and possible a model that describe my data.&lt;br&gt;&#xA;My data is comprised of multiple time-series uni-variate sets.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_set = [pd.Series(np.random.normal(size=10), range(10)), &#xA;            pd.Series(np.random.normal(size=10), range(10)),&#xA;            pd.Series(np.random.normal(size=10), range(10)),...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each of these series follows some time-series assumptions - meaning there is a dependency between X and X-1, X-2 .. at each series.&lt;br&gt;&#xA;All of the series represent similar process, so they have some-what similar properties(value scale, change points, seasonality) but they are not identical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to my understanding all time-series models are good for learning one time-series with no implication on other series.&#xA;Is there a way for fitting a model that is best for all series? &#xA;What is the method for learning/model across multiple related time series?&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="27983" LastEditDate="2017-09-28T05:16:39.297" LastActivityDate="2020-06-06T23:42:16.980" Title="Methods for analyzing multiple time series" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23371" PostTypeId="2" ParentId="17294" CreationDate="2017-09-27T16:40:49.670" Score="0" Body="&lt;p&gt;This &lt;a href=&quot;https://github.com/toliwa/CoreNLP&quot; rel=&quot;nofollow noreferrer&quot;&gt;extension of Stanford CoreNLP&lt;/a&gt; to capture MultiWord Expressions(MWE's) worked like a charm for one of my tasks. For Python users, gear up to write some connector code or hack the Java code. &lt;/p&gt;&#xA;" OwnerUserId="31188" LastActivityDate="2017-09-27T16:40:49.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23372" PostTypeId="1" CreationDate="2017-09-27T17:13:01.013" Score="3" ViewCount="291" Body="&lt;p&gt;Given N classes in a multi-label dataset and a trained classifier C, how would we add a new class N+1 to the dataset, and fine-tune the trained classifier C such that it now predicts N+1 labels? (lets call this fine-tuned N+1 class classifier C'). Let's use Open Images as an example, which has 9,000,000 images in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is with the data and it being a multi-label dataset. Say we have 5000 images from the N+1 class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first issue is that there may be other objects from any of the N classes in the images besides class N+1 (these images are taken from the web). So we run the trained classifier C on the 5000 images and have machine annotations for the other N classes in the 5000 images. We then use turkers to verify all the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the second (unsolved) issue is, how to find instances of the N+1 class in the original 9,000,000 training set? If we don't find these 'false negatives' in the original training set, fine-tuning could have unexpected results.&lt;/p&gt;&#xA;" OwnerUserId="39767" LastActivityDate="2017-09-27T17:13:01.013" Title="How to add a new label to a multi-label dataset (like Open Images)" Tags="|machine-learning|deep-learning|training|multilabel-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23373" PostTypeId="2" ParentId="20340" CreationDate="2017-09-27T21:17:05.927" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Currently we cannot direct extract data matrix from DMatrix, mainly due to DMatrix is a internal data structure that may or maynot sit in memory.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;--by tqchen commented on 3 Aug 2015&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://github.com/dmlc/xgboost/issues/422&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="39716" LastEditorUserId="29575" LastEditDate="2017-09-27T22:23:41.767" LastActivityDate="2017-09-27T22:23:41.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23374" PostTypeId="1" CreationDate="2017-09-28T03:00:21.633" Score="1" ViewCount="108" Body="&lt;p&gt;NIST is a metric used to measure the goodness of translation.&lt;/p&gt;&#xA;&lt;p&gt;In the paper, &lt;a href=&quot;http://www.mt-archive.info/HLT-2002-Doddington.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doddington (2002)&lt;/a&gt; introduce the notion of &lt;strong&gt;&amp;quot;Information weights&amp;quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Information weights were computed using N-gram counts over the&#xA;set of reference translations, according to the following equation:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dQsge.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dQsge.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Which Owczarzak (2008) describes as&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Here the precision is modified by the information weights Info(w1 . . . wn), computed on the basis of the frequency of a given n-gram in relation to the frequency of a ‘parent’ (n-1)-gram, in accordance with the equation (2.6)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;There's an implementation of the NIST score on &lt;a href=&quot;https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v13a.pl&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v13a.pl&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;But we can't seem to figure out how should the information weights be computed on &lt;a href=&quot;https://github.com/nltk/nltk/pull/1840&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/nltk/nltk/pull/1840&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Does anyone know the details of how this information weights work?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I've tried looking at the mteval-v13a.pl script but I'm not understanding how it works and relates to the formula above.&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="25180" LastEditDate="2023-01-17T17:28:59.177" LastActivityDate="2023-01-17T17:28:59.177" Title="What does NIST information weights refer to?" Tags="|nlp|model-evaluations|machine-translation|information-theory|ngrams|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23375" PostTypeId="2" ParentId="10000" CreationDate="2017-09-28T03:00:47.353" Score="2" Body="&lt;p&gt;From a &lt;a href=&quot;https://stats.stackexchange.com/questions/44702/definition-of-dynamic-bayesian-system-and-its-relation-to-hmm&quot;&gt;similar Cross Validation question&lt;/a&gt; follows &lt;a href=&quot;https://stats.stackexchange.com/users/17096/jerad&quot;&gt;@jerad&lt;/a&gt; answer:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;HMMs are not equivalent to DBNs, rather they are a special case of DBNs in which the entire state of the world is represented by a single hidden state variable. Other models within the DBN framework generalize the basic HMM, allowing for more hidden state variables (see the second paper above for the many varieties).&lt;/p&gt;&#xA;&lt;p&gt;Finally, no, DBNs are not always discrete. For example, linear Gaussian state models (Kalman Filters) can be conceived of as continuous valued HMMs, often used to track objects in space.&lt;/p&gt;&#xA;&lt;p&gt;I'd recommend looking through these two excellent review papers:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://select.cs.cmu.edu/class/10701-F09/readings/zoubin-hmms.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;An Introduction to Hidden Markov Models and Bayesian Networks by Zoubin Gharamani&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.ubc.ca/%7Emurphyk/Papers/dbnchapter.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Bayesian Networks by Kevin Murphy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="39513" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-28T19:19:25.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23376" PostTypeId="1" CreationDate="2017-09-28T06:04:25.437" Score="12" ViewCount="20439" Body="&lt;p&gt;I have already gone through &lt;a href=&quot;https://stackoverflow.com/a/4103234/4993513&quot;&gt;this post&lt;/a&gt; which uses &lt;code&gt;nltk&lt;/code&gt;'s &lt;code&gt;cmudict&lt;/code&gt; for counting the number of syllables in a word:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import cmudict&#xA;d = cmudict.dict()&#xA;def nsyl(word):&#xA;  return [len(list(y for y in x if y[-1].isdigit())) for x in d[word.lower()]] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, for words outside the cmu's dictionary like names for example: &lt;code&gt;Rohit&lt;/code&gt;, it doesn't give a result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So, is there any other/better way to count syllables for a word?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2017-09-28T07:16:00.627" LastActivityDate="2022-05-31T00:48:37.670" Title="How to get the number of syllables in a word?" Tags="|nlp|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23377" PostTypeId="2" ParentId="23341" CreationDate="2017-09-28T08:33:39.620" Score="19" Body="&lt;p&gt;It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?&lt;/p&gt;&#xA;&lt;p&gt;If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.&lt;/p&gt;&#xA;" OwnerUserId="25104" LastEditorUserId="43000" LastEditDate="2020-11-29T13:09:32.713" LastActivityDate="2020-11-29T13:09:32.713" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="23378" PostTypeId="1" AcceptedAnswerId="23379" CreationDate="2017-09-28T08:37:10.113" Score="1" ViewCount="1001" Body="&lt;p&gt;I am reading &lt;a href=&quot;http://ciml.info/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;A Course in Machine Learning&quot;&lt;/a&gt; and, in &lt;a href=&quot;http://ciml.info/dl/v0_99/ciml-v0_99-ch02.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;chapter 2&lt;/a&gt;, the author says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;For &lt;strong&gt;most&lt;/strong&gt; models, there will be associated parameters. These are&#xA;the things that we use the data to decide on. Parameters in a decision&#xA;tree include: the specific questions we asked, the order in which we&#xA;asked them, and the classification decisions at the leaves.&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is about the first sentence. &lt;strong&gt;Is there any model in machine learning that does not have parameters?&lt;/strong&gt; I can't think of any. For sure there are models without &lt;em&gt;hyperparameters&lt;/em&gt; (for instance, the linear model does not contain any hyperparameter, but it still contains 2 parameters, the slope and the y-intercept).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?&lt;/p&gt;&#xA;" OwnerUserId="34036" LastActivityDate="2017-10-01T13:31:15.753" Title="Do models without parameters exist?" Tags="|training|model-selection|parameter|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23379" PostTypeId="2" ParentId="23378" CreationDate="2017-09-28T08:56:24.557" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there any model in machine learning that does not have parameters?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;noreferrer&quot;&gt;k-nearest neighbors&lt;/a&gt;  is parameterless (there is only a single hyper-parameter $k$).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Exactly: such models require no training at all. k-NN in particular relies on knowing the data set upon prediction. Anything close to &quot;training&quot; this model would be pushing data points to a set, but these do not count as parameters.&lt;/p&gt;&#xA;" OwnerUserId="17538" LastActivityDate="2017-09-28T08:56:24.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23381" PostTypeId="2" ParentId="326" CreationDate="2017-09-28T09:33:47.300" Score="2" Body="&lt;p&gt;R: R is the Open source counterpart. which has traditionally been used in academics and research.&#xA;Because of its open source nature, latest techniques get released quickly.&#xA;There is a lot of documentation available over the internet and it is a very cost-effective option.&#xA;Python: With origination as an open source scripting language,&#xA;Python usage has grown over time. Today, it sports libraries (numpy, scipy and matplotlib) and functions for almost any statistical operation / model building you may want to do.&#xA;Since introduction of pandas, it has become very strong in operations on structured data.&lt;/p&gt;&#xA;&lt;p&gt;Python Code&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Import Library&#xA;# Import other necessary libraries like pandas, numpy...&#xA;from sklearn import linear_model&#xA;# Load Train and Test datasets&#xA;# Identify feature and response variable(s) and values must be numeric and numpy arrays&#xA;x_train=input_variables_values_training_datasets&#xA;y_train=target_variables_values_training_datasets&#xA;x_test=input_variables_values_test_datasets&#xA;# Create linear regression object&#xA;linear = linear_model.LinearRegression()&#xA;# Train the model using the training sets and check score&#xA;linear.fit(x_train, y_train)&#xA;linear.score(x_train, y_train)&#xA;# Equation coefficient and Intercept&#xA;print('Coefficient: \n', linear.coef_)&#xA;print('Intercept: \n', linear.intercept_)&#xA;# Predict Output&#xA;predicted= linear.predict(x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;R Code&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Load Train and Test datasets&#xA;# Identify feature and response variable(s) and values must be numeric and numpy arrays&#xA;x_train &amp;lt;- input_variables_values_training_datasets&#xA;y_train &amp;lt;- target_variables_values_training_datasets&#xA;x_test &amp;lt;- input_variables_values_test_datasets&#xA;x &amp;lt;- cbind(x_train,y_train)&#xA;# Train the model using the training sets and check score&#xA;linear &amp;lt;- lm(y_train ~ ., data = x)&#xA;summary(linear)&#xA;# Predict Output&#xA;predicted= predict(linear,x_test) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38145" LastEditorUserId="71442" LastEditDate="2020-10-03T19:22:35.207" LastActivityDate="2020-10-03T19:22:35.207" CommentCount="0" CommunityOwnedDate="2017-09-28T09:33:47.300" ContentLicense="CC BY-SA 4.0" />
  <row Id="23382" PostTypeId="1" AcceptedAnswerId="23395" CreationDate="2017-09-28T10:25:11.403" Score="1" ViewCount="117" Body="&lt;p&gt;Text generation is well studied using Markov chains or NNs, but I am not aware of any works to word sequence prediction in terms of subspace learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Treating phrases or sentences as temporal data such as time series, it is possible to represent word sequences as a tensor $T = WS \times W \times K$, where $WS$ is the set of word sequences present in the corpus, $W$ represents the set of segmented words, and $K$ is the maximum length of observed sequences&#xA; For instance, for a phrase, ws = &lt;code&gt;word sequence prediction&lt;/code&gt;, then $T(ws, ``sequence&quot;, 2) = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an incomplete tensor, where entries s.t. prediction are missing, the reconstructed tensor after decomposition can then be used to generate texts, in terms of the observed word space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Is there any works using tensor factorization or factorization machines for word sequence generation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How subspace learning models differ from those generative models, such as Recurrent Neural Networks or Belief Networks? What are the downsides of using subspace methods as compared to other established methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) How to establish the threshold for the length of the predicted sequence? For example, can one look at the $WS_r \times K_r$ space, and use cross-validation to find the threshold for each word sequences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers or answers to any of the above questions is highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="37739" LastActivityDate="2017-09-29T19:41:25.940" Title="Text generation using Tensor Factorization" Tags="|predictive-modeling|word-embeddings|matrix-factorisation|text-generation|generative-models|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23383" PostTypeId="1" AcceptedAnswerId="23397" CreationDate="2017-09-28T10:37:06.907" Score="2" ViewCount="346" Body="&lt;p&gt;I have some data that contains IDs along with an associated column containing times. What I want to do is to be able to determine which IDs are most similar, based upon the times e.g. Which IDs have correlated times for performing an action&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data looks something like this;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID      Time(secs)&#xA;AAAA    1&#xA;AAAA    6&#xA;AAAA    5&#xA;AAAA    2&#xA;AAAA    4&#xA;BBBB    2&#xA;BBBB    4&#xA;BBBB    6&#xA;BBBB    3&#xA;CCCC    3&#xA;CCCC    4&#xA;CCCC    1&#xA;CCCC    6&#xA;DDDD    7 &#xA;DDDD    4&#xA;DDDD    5&#xA;DDDD    3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Naively I initially thought this would be a simple case of plotting the values and then calculating the Correlation Coefficient but I soon realised this isn't possible;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID             CorrCoef&#xA;AAAA&amp;gt;BBBB      ????&#xA;AAAA&amp;gt;CCCC&#xA;AAAA&amp;gt;DDDD&#xA;BBBB&amp;gt;CCCC&#xA;BBBB&amp;gt;DDDD&#xA;CCCC&amp;gt;DDDD&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am now thinking I need to be able to compare 2 populations using something like a t-statistic or perhaps by using something like an AutoRegression. It is fair to say I am now feeling a little bit out of my depth with something that seemed quite basic initially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any pointers as to the best way of doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based upon a suggestion below, it seems a KS test could be useful here. What I need to do therefore is subset the data so that it can be fed into the below line of code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scipy.stats.ks_2samp(data1, data2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="37059" LastEditDate="2017-09-28T16:17:03.877" LastActivityDate="2017-09-29T22:16:15.737" Title="Correlating activity between entities using Python" Tags="|python|statistics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23384" PostTypeId="1" CreationDate="2017-09-28T10:51:29.450" Score="3" ViewCount="336" Body="&lt;p&gt;From my understanding, a stochastic process whose value at a particular instant is dependent on the previous values taken and every time the process is run, the path chosen can be different. We can know only the limits and confines the process stays after certain intial seeding value.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weight values for a neural networks are updated via the stochastic gradient descent method. What is the stochastic part about it? After some initialization of the variable, error function accumulated will be same if the same set of input data is provided after initialization in each test. Why is the relevance of the term stochastic? Where is the scope for the randomness and what is the source of this randomness? &lt;/p&gt;&#xA;" OwnerUserId="24235" LastEditorUserId="24235" LastEditDate="2017-09-28T11:15:26.917" LastActivityDate="2023-05-29T16:07:43.763" Title="What is the stochastic part in stochastic gradient descent?" Tags="|neural-network|optimization|gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23386" PostTypeId="2" ParentId="23384" CreationDate="2017-09-28T11:13:07.627" Score="4" Body="&lt;p&gt;You sample a random mini-batch as opposed to the full dataset. This means that you get a stochastic approximation of the true gradient of your loss function with regards to your dataset and weights. &lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-09-28T11:13:07.627" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23387" PostTypeId="2" ParentId="23383" CreationDate="2017-09-28T11:34:05.587" Score="1" Body="&lt;p&gt;This depends a bit if you have a lot of data per ID. If you have you can apply the Kolmogorov–Smirnov test between these distributions to check how similar they are. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-09-28T11:34:05.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23388" PostTypeId="1" CreationDate="2017-09-28T12:22:50.303" Score="3" ViewCount="377" Body="&lt;p&gt;I know that with Orange it is possible, using the 'Test &amp;amp; Score&quot; widget, to test the performance of a classifier trained on a specific training set using a different test set ('Test on test data' option).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is it possible to use a training set to train a classifier, tune its hyperparameters through cross validation on a separate validation set, and, finally, test the classification performance on a third, separate set using the already trained classifier? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How this can be achieved?&lt;/p&gt;&#xA;" OwnerUserId="39760" LastEditorUserId="29575" LastEditDate="2017-09-28T19:16:09.580" LastActivityDate="2017-09-29T19:51:35.153" Title="Using already trained classifiers in Orange" Tags="|cross-validation|training|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23390" PostTypeId="1" CreationDate="2017-09-28T13:28:24.957" Score="2" ViewCount="164" Body="&lt;p&gt;I read the &lt;a href=&quot;https://blog.acolyer.org/2017/03/24/a-miscellany-of-fun-deep-learning-papers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;wavenet&lt;/a&gt; paper and I am clear about why they used Dilated Convolution. According to it they used this for give the causal property for the model. Where they wanted to introduced a sequential behavior . But lot of posts say it can use to increase the receptive field. Can some give me simple examples where we can use this in signal processing domain CNN + LSTM) and vision domain. &lt;/p&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2017-09-28T13:28:24.957" Title="When should I use dilated convolution?" Tags="|convolution|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23392" PostTypeId="1" CreationDate="2017-09-28T16:11:36.707" Score="0" ViewCount="309" Body="&lt;p&gt;We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good tools to help us manage this process, or would you recommend writing a Python script instead?&lt;/p&gt;&#xA;" OwnerUserId="8875" LastActivityDate="2023-10-21T11:03:40.427" Title="Tool vs Python Script for Transforming Data in Mongo" Tags="|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23395" PostTypeId="2" ParentId="23382" CreationDate="2017-09-28T19:25:01.837" Score="2" Body="&lt;p&gt;Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term language dependencies, its results would be similar to low-order Markov chains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensor Factorization could be used as another signal in a larger natural language generation system, for example &lt;a href=&quot;https://arxiv.org/abs/1704.02686&quot; rel=&quot;nofollow noreferrer&quot;&gt;improving word-embeddings&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-09-29T19:41:25.940" LastActivityDate="2017-09-29T19:41:25.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23396" PostTypeId="2" ParentId="23354" CreationDate="2017-09-28T20:10:26.923" Score="1" Body="&lt;p&gt;ELKI has some very nice cluster visualizations. You could also use tSNE or MDS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But as you used cosine, I assume you have text data, and that is not easy to visualize. &lt;em&gt;You&lt;/em&gt; probably &lt;em&gt;first&lt;/em&gt; need to figure out how to visualize your &lt;em&gt;input&lt;/em&gt; data, then you can think about adding cluster information to this.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-09-28T20:10:26.923" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23397" PostTypeId="2" ParentId="23383" CreationDate="2017-09-28T20:36:47.297" Score="2" Body="&lt;p&gt;You may compute the $id$ co-occurrences frequencies in a given time-window.&#xA;Suppose (without loss of generality) your criteria for co-occurrence is that both $id$s must occur on the second &lt;code&gt;t&lt;/code&gt;, than using maximum likelihood estimate $P(id_{i}|id_{j})$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(id_{i}|id_{j}) = \frac{count(id_{i}, id_{j})}{count(id_{j})}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the maximum likelihood estimate for the joint probability is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(id_{i}, id_{j}) = \frac{count(id_{i}, id_{j})}{\sum{k \in IDs}{} &#xA; \ count(id_{k})}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $id_j, id_i \in IDs\ $ and $IDs$ is the set containing all $id$s (&lt;code&gt;AAAA,BBB,CCCC,...&lt;/code&gt;), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then calculate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Pointwise_mutual_information&quot; rel=&quot;nofollow noreferrer&quot;&gt;pointwise mutual information&lt;/a&gt; between each $id$ pair, that is, how ofter two $id$s co-occur, compared with what we would expect if they were independent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$I(id_i, id_j) = \log_{2}{\frac{P(id_i, id_j)}{P(id_i)P(id_j)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given you a estimation of how strong is the association between $id_i$ and $id_j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same strategy may be used to find similarities, you may think about each $id_i$ being a $|IDs|$-dimensional vector with the co-occurrences frequencies being the values. You can then apply &lt;a href=&quot;https://brenocon.com/blog/2012/03/cosine-similarity-pearson-correlation-and-ols-coefficients/&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine-similarity or pearson-correlation&lt;/a&gt; to find the most similar vectors ($id$s). &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Complementing my answer follows python demonstrating the ideas above for the sample dataset given in the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we create our dataframe from the data in the question&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;import pandas as pd&#xA;import numpy as np&#xA;import collections&#xA;&#xA;d = {'ID': ['AAAA', 'AAAA', 'AAAA', 'AAAA', 'AAAA', 'BBBB', 'BBBB', 'BBBB', 'BBBB', 'CCCC', 'CCCC', 'CCCC', 'CCCC','DDDD', 'DDDD', 'DDDD', 'DDDD'],&#xA;     'Time': [1, 6, 5, 2, 4, 2, 4, 6, 3, 3, 4, 1, 6, 7, 4, 5, 3]}&#xA;&#xA;df = pd.DataFrame(d)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Computes the co-occurrences&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;dfm = df.merge(df, on='Time')&#xA;dfm = dfm[dfm.ID_x != dfm.ID_y]  # ID_x and ID_y are created by the merge&#xA;df_M = pd.get_dummies(dfm.ID_x).groupby(dfm.ID_y).apply(sum)&#xA;print(df_M)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dataframe &lt;code&gt;df_M&lt;/code&gt; represents the co-occurrence matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;      AAAA  BBBB  CCCC  DDDD&#xA;ID_y                        &#xA;AAAA     0     3     3     2&#xA;BBBB     3     0     3     2&#xA;CCCC     3     3     0     2&#xA;DDDD     2     2     2     0&#xA;&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Pairwise Mutual Information&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let &lt;code&gt;N&lt;/code&gt; be the total amount of co-occurrences, than I can compute every joint probability simply dividing the co-occurrences in the &lt;code&gt;df_M&lt;/code&gt; matrix  by &lt;code&gt;N&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;N = df_M.sum().sum()&#xA;df_joint_P = df_M/N  # Computes every joint probability P(id_i, id_j)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The probability of each $id$ is the sum of all its joint probabilities&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;df_ID_P = df_joint_P.sum(axis=0) # Marginalizes to produces P(id_i)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now that we have all we need to compute a PMI dataframe&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;idx = [(r, c) for r in list(df_M) for c in list(df_M)]&#xA;pmi_dict = collections.defaultdict(dict)&#xA;for r,c in idx:&#xA;    pmi_dict[r][c] = np.log2(2*df_joint_P[r][c]/(df_ID_P[r] * df_ID_P[c])) if df_joint_P[r][c] &gt; 0 else 0&#xA;pmi_df = pd.DataFrame(pmi_dict)&#xA;print(pmi_df)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The pairwise mutual information between every pair of different $id$s is different than zero and like bellow&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;          AAAA      BBBB      CCCC      DDDD&#xA;AAAA  0.000000  1.491853  1.491853  1.321928&#xA;BBBB  1.491853  0.000000  1.491853  1.321928&#xA;CCCC  1.491853  1.491853  0.000000  1.321928&#xA;DDDD  1.321928  1.321928  1.321928  0.000000&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can see that every $id$ seems to be almost equally associated with each other.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Cosine Similarity&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now we can compute the cosine similarity, remembering that it is the $l_2$-normalized dot-product&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;df_NM = df_M.div(df_M.pow(2).sum(axis=1).pow(0.5), axis=0) &#xA;df_cos = df_NM.dot(df_NM.T)&#xA;print(df_cos)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The cosines are&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;code&gt;&#xA;          AAAA      BBBB      CCCC      DDDD&#xA;AAAA  1.000000  0.590909  0.590909  0.738549&#xA;BBBB  0.590909  1.000000  0.590909  0.738549&#xA;CCCC  0.590909  0.590909  1.000000  0.738549&#xA;DDDD  0.738549  0.738549  0.738549  1.000000&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously every $id$ is (trivially) most similar to itself but from the data above we see that every $id$ is similar to all the others (confirming the result we found using PMI) but we may notice that $id$ &lt;code&gt;DDDD&lt;/code&gt; is more similar to all the other $id$s (at least for this tiny example).  &lt;/p&gt;&#xA;" OwnerUserId="39513" LastEditorUserId="39513" LastEditDate="2017-09-29T22:16:15.737" LastActivityDate="2017-09-29T22:16:15.737" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23398" PostTypeId="2" ParentId="19124" CreationDate="2017-09-28T20:37:05.007" Score="2" Body="&lt;p&gt;As the source you are quoting says &quot;the cost on the test data is no more than a proxy for classification accuracy.&quot; You might ask, why should we use a proxy, why not use the accuracy directly? The answer is that you need to minimize the cost function with respect to the weights and biases. Therefore it has to be a differentiable function of the the weights and biases. Accuracy is not a differentiable function and therefore cannot be used directly. But since ultimately you care about accuracy, as you yourself illustrated above (...please note the classification accuracy is 100% in the first instance and yet the cost is higher... ), you determine overfitting based on the accuracy on the test set. &lt;/p&gt;&#xA;" OwnerUserId="39811" LastActivityDate="2017-09-28T20:37:05.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23399" PostTypeId="2" ParentId="9249" CreationDate="2017-09-28T21:54:42.653" Score="10" Body="&lt;p&gt;It was announced on  2017-09-28 that Theano will be discontinued:&lt;/p&gt;&#xA;&lt;p&gt;From &lt;a href=&quot;https://groups.google.com/forum/#!topic/theano-users/7Poq8BZutbY&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; (Yoshua Bengio):&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;After almost ten years of development, we have the regret to announce that we will put an end to our Theano development after the 1.0 release,  which is due in the next few weeks. We will continue minimal maintenance  to keep it working for one year, but we will stop actively implementing  new features. Theano will continue to be available afterwards, as per  our engagement towards open source software, but MILA does not commit to  spend time on maintenance or support after that time frame.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So TensorFlow is a better option.&lt;/p&gt;&#xA;" OwnerUserId="843" LastEditorUserId="98307" LastEditDate="2020-08-05T09:21:31.550" LastActivityDate="2020-08-05T09:21:31.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23400" PostTypeId="1" CreationDate="2017-09-28T22:00:24.643" Score="2" ViewCount="132" Body="&lt;p&gt;My scenario is roughly the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine 500 cars, all Toyota Corollas (or whatever). While these cars have many similarities, they are not exactly identical: some of them have 1.5 liter engines while others have 1.8 liter engines, and some have a high mileage while others have barely left their garage. Still, we can imagine that all cars are the product of one process with a bunch of parameters. Some of these parameters are observed, while others are unobserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data on the condition of each car at a set of times &lt;code&gt;t_i&lt;/code&gt;. For our purposes, &quot;condition&quot; is represented by a number between 0.0 and 1.0, where 0.0 means &quot;as good as new&quot; and 1.0 means &quot;basically a wreck&quot;. My goal is to build a model which lets me forecast the condition of each car at some future time, based on its condition at the previous time steps as well as its &quot;parameters&quot; like mileage and engine size. I could train a separate model for each of the cars, but I likely don't have enough data for each car for this model to provide accurate forecasts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already made an attempt at this using ordinary machine learning methods by transforming the full history of each car into one data point by encoding the condition at each of the previous time steps as a feature. My results are kind of so-so, and I'd like to try out LSTMs for this purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I have a hard time figuring out how to do this with LSTMs. A bit of googling has led me to lots of tutorials on how to use LSTMs to forecast two properties of the same &quot;object&quot;, say, condition and mileage for a &lt;em&gt;single&lt;/em&gt; car, but none on how to predict the same property for a bunch of &quot;different objects&quot; with additional parameters. I assume just plugging&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[conditionCar1, mileageCar1, colorCar1, conditionCar2, mileageCar2, colorCar2, (...)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;observed at a lot of time steps into an LSTM model wouldn't work very well. Is what I'm trying to do viable, and if so does anybody have any pointers to relevant information?&lt;/p&gt;&#xA;" OwnerUserId="31270" LastActivityDate="2017-11-07T03:34:49.127" Title="Using LSTMs for modelling and forecasting several time series generated by the same process" Tags="|neural-network|time-series|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23402" PostTypeId="1" AcceptedAnswerId="23479" CreationDate="2017-09-28T22:43:10.070" Score="2" ViewCount="283" Body="&lt;p&gt;I am using kmeans to cluster some data with 2 features. Not sure I understand why kmeans is producing the clusters I see:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YMdou.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YMdou.png&quot; alt=&quot;kmeans results&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would kmeans not cluster these points in a way that matches what we would expect visually looking at the data? Why are seemingly random points in the middle of visual clusters being put into a second cluster like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code I am running:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cols = ['col1', 'col2']&#xA;features = map(lambda x: df[x], cols)&#xA;input = np.matrix(list(zip(*features)))&#xA;&#xA;scaler = StandardScaler()&#xA;scaler.fit(input)&#xA;input_scaled = scaler.transform(input)&#xA;&#xA;algo = KMeans(n_clusters=2)&#xA;algo.fit(input_scaled)&#xA;df['cluster'] = pd.Series(algo.labels_)&#xA;&#xA;sns.lmplot(x=cols[0],y=cols[1],data=df, fit_reg=False, hue='cluster')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39814" LastActivityDate="2018-01-30T23:59:18.493" Title="Confused by kmeans results" Tags="|python|clustering|k-means|" AnswerCount="1" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="23403" PostTypeId="2" ParentId="8941" CreationDate="2017-09-29T00:51:13.743" Score="2" Body="&lt;p&gt;DataSciCon.Tech&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://datascicon.tech/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://datascicon.tech/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a 3-day developer-focused conference in Atlanta USA in November/December&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Full-day workshops on: &#xA;Data Science for Discovery, Innovation, and Value Creation&#xA;Data Science with R Workshop&#xA;Introduction to Machine Learning with Python and TensorFlow&#xA;Data Analytics with Tableau&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Followed by 2 days with deep dive content in 4 tracks on topics such as:&#xA;Data Science,&#xA;Data Analytics,&#xA;Artificial Intelligence,&#xA;Machine Learning,&#xA;Deep Learning,&#xA;Big Data,&#xA;Data Visualisation, &#xA;and Deep Learning.&lt;/p&gt;&#xA;" OwnerUserId="39815" LastActivityDate="2017-09-29T00:51:13.743" CommentCount="1" CommunityOwnedDate="2017-09-29T00:51:13.743" ContentLicense="CC BY-SA 3.0" />
  <row Id="23405" PostTypeId="2" ParentId="23317" CreationDate="2017-09-29T07:26:37.567" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In short, yes. You can convert each input variable into a vectorised feature separately, then concatenate those vectors into one long vector of features per example. Most statistical machine learning models don't care (or &quot;understand&quot;) that the feature data has been formed in different ways, it is just treated numerically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You do need to pay attention to each input data type in order to decide how best to convert to a usable feature. There is no single &quot;best&quot; way to do this, and this is partly an art - you add the &quot;science&quot; part of Data Science by testing your ideas and taking measurements, e.g. accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are some quick thoughts on how you might prepare the data into features from each column:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Comments&lt;/em&gt;. You have already converted this, you are using a variant of &quot;bag of words&quot; to convert text into a fixed length numerical vector. This is a common approach and hard to beat. If you have a &lt;em&gt;lot&lt;/em&gt; of data you can look into more sophisticated models that take account of word order, such as Recurrent Neural Networks, but that's a whole new subject area for now.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Gender&lt;/em&gt;. You can convert this into a simple vector, using &lt;a href=&quot;http://www.ritchieng.com/machinelearning-one-hot-encoding/&quot; rel=&quot;nofollow noreferrer&quot;&gt;one hot encoding&lt;/a&gt; - this will have two or more columns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;City&lt;/em&gt;. You could one hot encode this also. It might also be worth grouping cities and having a smaller feature vector. E.g. group by state/country before one hot encoding.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Age&lt;/em&gt;. Either group and one hot encode, or scale to a smaller number,&#xA;by dividing by e.g. 50 (this helps algorithms like SGD by keeping distance metrics similar between different feature types). I suspect grouping in typical demographic splits (e.g. 0-11, 12-17, 18-24, 25-34 etc) and one hot encoding would work well for sentiment analysis, because that would capture some generalised differences between uses of text expression.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-09-29T07:26:37.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23406" PostTypeId="2" ParentId="19124" CreationDate="2017-09-29T10:26:24.817" Score="8" Body="&lt;p&gt;Let's say we want to predict if a student will land a job interview based on her resume.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, assume we train a model from a dataset of 10,000 resumes and their outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, we try the model out on the original dataset, and it predicts outcomes with 99% accuracy… wow!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now comes the bad news.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we run the model on a new (“unseen”) dataset of resumes, we only get 50% accuracy… uh-oh!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our model doesn’t generalize well from our training data to unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is known as overfitting, and it’s a common problem in machine learning and data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Overfitting V/s Underfitting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can understand overfitting better by looking at the opposite problem, underfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Underfitting occurs when a model is too simple – informed by too few features or regularized too much – which makes it inflexible in learning from the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple learners tend to have less variance in their predictions but more bias towards wrong outcomes (see: The Bias-Variance Tradeoff).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, complex learners tend to have more variance in their predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both bias and variance are forms of prediction error in machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, we can reduce error from bias but might increase error from variance as a result, or vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This trade-off between too simple (high bias) vs. too complex (high variance) is a key concept in statistics and machine learning, and one that affects all supervised learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-09-29T10:26:24.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23407" PostTypeId="1" CreationDate="2017-09-29T10:34:46.797" Score="1" ViewCount="71" Body="&lt;p&gt;I would like to cluster nodes in a graph based on the structure of their local neighbourhood, i.e. nodes with similar graph structure around each other should be in the same cluster. My issue is basically to define a good feature set with a metric that would result in reasonable clusters. Does anyone know of any such features? :)&lt;/p&gt;&#xA;" OwnerUserId="39827" LastActivityDate="2017-09-29T10:34:46.797" Title="Clustering nodes based on structure of local neighbourhood in a graph" Tags="|clustering|" AnswerCount="0" CommentCount="6" ClosedDate="2017-10-01T11:37:41.550" ContentLicense="CC BY-SA 3.0" />
  <row Id="23408" PostTypeId="2" ParentId="155" CreationDate="2017-09-29T10:37:23.827" Score="4" Body="&lt;p&gt;Just we load MASS package in R we access multiple dataframes  or data sets .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;install.packages(&quot;MASS&quot;)&#xA;require(&quot;MASS&quot;)&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-09-29T10:37:23.827" CommentCount="0" CommunityOwnedDate="2017-09-29T10:37:23.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="23410" PostTypeId="2" ParentId="410" CreationDate="2017-09-29T10:51:54.103" Score="4" Body="&lt;p&gt;Neural networks are often trained by gradient descent on the weights. This means at each iteration we use backpropagation to calculate the derivative of the loss function with respect to each weight and subtract it from that weight. &#xA;However, if you actually try that, the weights will change far too much each iteration, which will make them “overcorrect” and the loss will actually increase/diverge. So in practice, people usually multiply each derivative by a small value called the “learning rate” before they subtract it from its corresponding weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also think of a neural networks loss function as a surface, where each direction you can move in represents the value of a weight. Gradient descent is like taking leaps in the current direction of the slope, and the learning rate is like the length of the leap you take.&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-09-29T10:51:54.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23411" PostTypeId="1" AcceptedAnswerId="23412" CreationDate="2017-09-29T11:59:11.760" Score="5" ViewCount="1586" Body="&lt;p&gt;I am currently using a random forest model for classification, however I am unsure how the feature selection technique &quot;varImp&quot; works on R. I understand the context of variable importance, however when I implement it in R it doesn't seem to produce the results I expect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When removing the most important variable (of 31 features), the model's accuracy does not decrease. I would expect it to as it should be contributing the most to the model's ability to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please explain what this function is doing?&lt;/p&gt;&#xA;" OwnerUserId="37484" LastEditorUserId="29575" LastEditDate="2017-10-01T11:36:12.643" LastActivityDate="2017-10-01T11:36:12.643" Title="Variable Importance Random Forest on R" Tags="|machine-learning|r|feature-selection|random-forest|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23412" PostTypeId="2" ParentId="23411" CreationDate="2017-09-29T12:52:03.203" Score="4" Body="&lt;p&gt;&lt;strong&gt;What the function is doing for each variable?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Record the Out-Of-Bag (OOB) accuracy for each tree.&lt;/li&gt;&#xA;&lt;li&gt;&quot;shuffle&quot; or permute the values of that variable.  This means you take all the values of that variable in the data, and assign those values randomly back out to the observations, which is a way of introducing noise and getting rid of the signal that that variable provided.&lt;/li&gt;&#xA;&lt;li&gt;Now it finds the OOB accuracy again, but this time the values for that variable are incorrect since we permuted it.  By introducing noise where your model expects signal, you should see a decrease in performance.&lt;/li&gt;&#xA;&lt;li&gt;Compare the original accuracy in (1) to the accuracy in (3) for each variable.  If the model performance decreases a lot for a variable in step (3) compared to (1), then it is deemed to have greater importance.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why does removing the most important variable not have a negative affect on accuracy? (my guess)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably because that important variable is correlated with some other variable(s) you have.  Your model can capture the information contained in that missing important variable by using a few other variables to make up for it.  When you drop the important variable, which other variables see a notable gain?&lt;/p&gt;&#xA;" OwnerUserId="14913" LastActivityDate="2017-09-29T12:52:03.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23413" PostTypeId="2" ParentId="23411" CreationDate="2017-09-29T13:15:11.393" Score="3" Body="&lt;p&gt;The VarImp works on the principle how your variables are helping  data to split with minimum error and order them by their efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per you said even after removing the important variable you are still getting same accuracy .May be it is not affecting your dependent variable to large extent or you didn't perform needed transformations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other ways to find feature importance like recursive feature elimination.&lt;a href=&quot;https://topepo.github.io/caret/recursive-feature-elimination.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is description RFE in caret&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-09-29T13:15:11.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23416" PostTypeId="1" CreationDate="2017-09-29T14:19:58.450" Score="2" ViewCount="3554" Body="&lt;p&gt;I've gone through the paper describing the &lt;a href=&quot;https://arxiv.org/pdf/1505.04597.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;UNet convolutional neural network&lt;/a&gt; a number of times, but am still having trouble figuring out how to connect the output of the network to the ground truth targets.  Below is an image depicting the architecture of the network.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EtyQs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EtyQs.png&quot; alt=&quot;&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://lmb.informatik.uni-freiburg.de/research/funded_projects/bioss_deeplearning/unet.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;uni-freiburg.de&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;&lt;p&gt;As can be seen from the figure, the output is a &lt;code&gt;388 x 388 x k&lt;/code&gt; matrix (where &lt;code&gt;k&lt;/code&gt; is the number of classes).  The target segmentation in this case should be &lt;code&gt;572 x 572&lt;/code&gt; (same spatial dimensions as the input image).  How do we match these up?  Are we suppose to perform some kind of interpolation on the output feature map to get it to match the dimensionality of the input?&lt;/p&gt;&#xA;" OwnerUserId="39442" LastEditorUserId="29169" LastEditDate="2021-02-15T22:23:23.497" LastActivityDate="2021-02-15T22:23:23.497" Title="How to map ground truth to prediction for UNet architecture" Tags="|deep-learning|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23417" PostTypeId="2" ParentId="23392" CreationDate="2017-09-29T16:47:56.743" Score="1" Body="&lt;p&gt;Many ETL tools support MongoDB and are commonly used for normalizing and standardizing data. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.pentaho.com/analytics-for-mongodb#tab-2&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.pentaho.com/analytics-for-mongodb#tab-2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-09-29T16:47:56.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23419" PostTypeId="2" ParentId="23341" CreationDate="2017-09-29T18:09:20.023" Score="8" Body="&lt;p&gt;Running inference on a GPU instead of CPU will give you close to the same speedup as it does on training, less a little to memory overhead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as you said, the application runs okay on CPU. If you get to the point where inference speed is a bottleneck in the application, upgrading to a GPU will alleviate that bottleneck. &lt;/p&gt;&#xA;" OwnerUserId="39845" LastActivityDate="2017-09-29T18:09:20.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23420" PostTypeId="2" ParentId="23416" CreationDate="2017-09-29T18:28:22.873" Score="2" Body="&lt;p&gt;It depends on how your UNet architecture is constructed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example image you've shown, the output of the first few layers of convolutions is cropped to 392x392 before it's copied over to the input of the final convolutional layers. Then, it's decreased to 388x388 by convolution without padding. For the example, you're probably best off cropping your image to 392x392, then scaling down to 388x388, as the lost resolution due to the convolution is more analogous to scaling than cropping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also construct the network to output an image of the same dimensionality to avoid this issue. The downside to this approach is that the network will produce some strange/poor results near the boundaries of the image, so you're spending computational power on pixels you will need to disregard anyway.&lt;/p&gt;&#xA;" OwnerUserId="39845" LastActivityDate="2017-09-29T18:28:22.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23422" PostTypeId="2" ParentId="23388" CreationDate="2017-09-29T19:51:35.153" Score="3" Body="&lt;p&gt;A scheme like this should work. Namely, you pass training data directly into the the learner widget and pass its trained Model into Predictions widget along with labeled test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ji4ff.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ji4ff.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2017-09-29T19:51:35.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23423" PostTypeId="1" CreationDate="2017-09-29T19:55:06.783" Score="6" ViewCount="12665" Body="&lt;p&gt;I have a couple thousand photos of whales taken from drones and I'm planning to build a simple binary classifier to run on these and future images to see if they contain a whale. I'd like to label specific pixels within the image as whale (1) or not whale (0) and I'm at a loss for a good free tool to do this. I'm imagining something where I can just zoom in and manually draw an outline around a whale but I'm open to other options. Does anyone know of a good way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="39851" LastActivityDate="2019-07-02T10:12:44.140" Title="Tool to Label Images for Supervised Classification" Tags="|python|training|labels|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23425" PostTypeId="2" ParentId="23423" CreationDate="2017-09-29T20:20:34.830" Score="5" Body="&lt;p&gt;If you are specifically looking to outline the whales, seems like FastAnnotationTool could work:&#xA;&lt;a href=&quot;https://github.com/christopher5106/FastAnnotationTool&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/christopher5106/FastAnnotationTool&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other options here:&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/List_of_Manual_Image_Annotation_Tools&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/List_of_Manual_Image_Annotation_Tools&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When had to annotate many images for a project, I built a fairly simple MATLAB gui that displayed images.  I cycled through each image and if I clicked on a point on the image, that image was annotated and the point where I clicked was saved in a corresponding .mat file (.csv would have been better). It worked... mostly.  I'd guess between python and OpenCV, this could be done easily without MATLAB.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-09-29T20:20:34.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23427" PostTypeId="1" AcceptedAnswerId="23436" CreationDate="2017-09-29T22:42:22.830" Score="7" ViewCount="2826" Body="&lt;p&gt;I'm just starting to learn about linear regressions and was wondering why it is that we opt to minimize the sum of squared errors. I understand the squaring helps us balance positive and negative individual errors (so say e1 = -2 and e2 = 4, we'd consider them as both regular distances of 2 and 4 respectively before squaring them), however, I wonder why we don't deal with minimizing the absolute value rather than the squares. If you square it, e2 has a relatively higher individual contribution to minimize than e1 compared to just the absolutely values (and do we want that?). I also wonder about decimal values. For instance, say we have e1 = 0.5 and e2 = 1.05, e1 will be weighted less when squared because 0.25 is less than 0.5 and e2 will be weighted more.   Lastly, there is the case of e1 = 0.5 and e2 = 0.2. E1 is further away to start, but when you square it 0.25 is compared with 0.4. Anyway, just wondering why we do sum of squares Erie minimization rather than absolute value.&lt;/p&gt;&#xA;" OwnerUserId="39854" LastActivityDate="2020-09-20T12:01:50.680" Title="The Why Behind Sum of Squared Errors in a Linear Regression" Tags="|linear-regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23429" PostTypeId="1" AcceptedAnswerId="23441" CreationDate="2017-09-30T04:35:22.077" Score="5" ViewCount="281" Body="&lt;p&gt;The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?&lt;/p&gt;&#xA;" OwnerUserId="39859" LastEditorUserId="18375" LastEditDate="2017-12-11T23:34:17.853" LastActivityDate="2017-12-11T23:34:17.853" Title="Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?" Tags="|machine-learning|neural-network|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23430" PostTypeId="1" CreationDate="2017-09-30T04:38:39.333" Score="3" ViewCount="459" Body="&lt;p&gt;As we know that K-Means is a powerful clustering, however it often stuck with local minimum problem because of bad initialization . One of the solution is K-Harmonic means that use Harmonic Average as the performance function instead of minimum distance btw the data and cluster's centroid. I am curious that why using harmonic function makes less sensitive to init centroids than ordinary K-Means. &lt;/p&gt;&#xA;" OwnerUserId="14357" LastActivityDate="2017-09-30T21:25:15.293" Title="Why K-harmonic means is less sensitive to random initialization than K-means?" Tags="|machine-learning|k-means|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23432" PostTypeId="1" AcceptedAnswerId="23470" CreationDate="2017-09-30T08:53:37.950" Score="1" ViewCount="213" Body="&lt;p&gt;I am doing time series forecasting for estimating monthly sales of certain consumer goods SKU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have last 3 years of actual sales data, wherein first 2.5 years of data is monthly sales data, but last 6 months data daily sales data is available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can certainly add the daily data to get monthly data, but considering that total data points are then only 36 (for 3 months) and it may be slightly challenging to get various models like ARIMA, Exponential models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to use the daily last 6 months of data to improve the accuracy of the model?&lt;/p&gt;&#xA;" OwnerUserId="39645" LastEditorUserId="29575" LastEditDate="2017-10-02T10:57:02.510" LastActivityDate="2017-10-02T13:59:41.040" Title="Time Resolution Changes in Time Series Forecasting" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23436" PostTypeId="2" ParentId="23427" CreationDate="2017-09-30T12:42:15.567" Score="3" Body="&lt;p&gt;Simple google search on &lt;strong&gt;&quot;stats why regression not absolute difference&quot;&lt;/strong&gt; would give you good answers. Try it yourself!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can quickly summarise:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your regression parameters are solutions to the maximum likelihood optimisation. That involves derivative, but absolute difference doesn't have a derivative at zero. There's no unique solution for least absolute regression.&lt;/li&gt;&#xA;&lt;li&gt;Least absolute regression is an alternative to the regular sum of squares regression, commonly classified as one of the robust statistical methods.&lt;/li&gt;&#xA;&lt;li&gt;You'd prefer least absolute regression if you care about outliers, otherwise the regular regression is generally better.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You might want to read about L1 vs L2:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models&quot;&gt;https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-09-30T12:42:15.567" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23437" PostTypeId="1" AcceptedAnswerId="24538" CreationDate="2017-09-30T15:57:53.597" Score="9" ViewCount="226" Body="&lt;p&gt;Suppose that you have a video file which pixel order has been shuffled once. That is, a random order have been defined once and applied to all frames. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it exist some known approach for retrieving the initial order of pixels?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some ideas around retrieving the initial topology by placing pixels which values are correlated in space and time closer together. I wonder if this has been studied and if efficient algorithms were published.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also this problem can be thought of as a way to project to a 2D matrix a set of values varying in time in order to be able to apply computer vision techniques (like CNN), with the assumption that these values are indeed somehow correlated.&lt;/p&gt;&#xA;" OwnerUserId="39569" LastEditorUserId="29169" LastEditDate="2021-02-15T22:23:13.443" LastActivityDate="2021-02-15T22:23:13.443" Title="Ways to reconstruct shuffled pixels of a video file?" Tags="|statistics|convolutional-neural-network|convolution|image-recognition|tsne|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23439" PostTypeId="2" ParentId="23430" CreationDate="2017-09-30T21:08:51.610" Score="4" Body="&lt;p&gt;Harmonic mean is less sensitive to outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that you have 3 numbers: &lt;code&gt;1&lt;/code&gt; &lt;code&gt;1&lt;/code&gt; and &lt;code&gt;30001&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their arithmetic mean (average) is &lt;code&gt;10001&lt;/code&gt; and their harmonic mean is &lt;code&gt;2&lt;/code&gt; that is much closer to the majority of the points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in a way, using an harmonic mean is more forgiving.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;A Bayesian look at the harmonic mean&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Assuming that the closer a point is, the more likely it is to be in a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can choose to model the distribution of distances from an origin point, as an Gamma-Exponential model with a priors $\alpha=m$ and $\beta=\frac{m}{\mu_0}$ where $\mu_0$ is the initial mean and $m$ is the strength of the prior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And given observations of $\{d_1,d_2,\dots,d_n\}$, the updated model's mean is:&#xA;$$\mu'=\frac{m+n}{\frac{m}{\mu_0}+\frac{1}{d_1}+\frac{1}{d_2}+\dots+\frac{1}{d_n}}$$&#xA;We got our beloved harmonic mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could argue that the harmonic mean in k-means context is an assumption on the distribution of distances.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastEditorUserId="12942" LastEditDate="2017-09-30T21:25:15.293" LastActivityDate="2017-09-30T21:25:15.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23441" PostTypeId="2" ParentId="23429" CreationDate="2017-09-30T22:27:14.823" Score="10" Body="&lt;p&gt;Sigmoid function is a &lt;a href=&quot;https://stats.stackexchange.com/questions/233658/softmax-vs-sigmoid-function-in-logistic-classifier&quot;&gt;partial case&lt;/a&gt; of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why do so many functions used in data science have derivatives of the&#xA;  form f(x)*(1-f(x))?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you consider the following differential equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y' = y \cdot (1-y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you will find the general solution in the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y(x) = \frac{e^x}{c + e^x} = \frac{1}{1 + c e^{-x}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in some sense, there are not that many functions with this property: they are &quot;very like&quot; sigmoid function.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2017-09-30T22:27:14.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23442" PostTypeId="2" ParentId="16584" CreationDate="2017-10-01T08:29:03.423" Score="1" Body="&lt;p&gt;Precision@K measures the proportion of positive items among the K highest-ranked items while AUC measures the quality of the overall ranking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details, you should check out this &lt;a href=&quot;https://stackoverflow.com/questions/45451161/evaluating-the-lightfm-recommendation-model/45466481#45466481&quot;&gt;answer&lt;/a&gt; from Maciej Kula.&lt;/p&gt;&#xA;" OwnerUserId="30392" LastActivityDate="2017-10-01T08:29:03.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23443" PostTypeId="1" AcceptedAnswerId="23444" CreationDate="2017-10-01T08:29:43.153" Score="1" ViewCount="803" Body="&lt;p&gt;I want to apply a CNN to a series of image sequences to classify that sequences of frames/images in two groups/categories. We are in a binary classification problem. My dataset is composed by a lot of 'batches' of frames. For example, each batch of frames could be compose of 20 frames of 64x64 pixels. One important thing is that the order of that 20 frames is important. If you shuffle the order of that 20 frames the output could change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For all this, I want to create a CNN to solve this binary classification problem. I'm usig Keras and TensorFlow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is my question? Well, I'm not sure if I have to use a &lt;a href=&quot;https://keras.io/layers/wrappers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TimeDistributed layer&lt;/a&gt; or not. The input shape of the neural network is the following one: (20, 64, 64, 1), whose meaning is: 20 frames with a 64x64 size (1 channel - grayscale). Should I use a TimeDistributed layer?&lt;/p&gt;&#xA;" OwnerUserId="39896" LastEditorUserId="836" LastEditDate="2017-10-01T08:38:40.223" LastActivityDate="2017-10-01T09:06:56.023" Title="Structure of Convolutional Neural Network to analyze a sequence of frames" Tags="|tensorflow|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23444" PostTypeId="2" ParentId="23443" CreationDate="2017-10-01T09:06:56.023" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm not sure if I have to use a TimeDistributed layer or not&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You definitely don't &lt;em&gt;have&lt;/em&gt; to use TimeDistributed. You have other choices, that may be equally valid, depending on your data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Flatten your example data into 81920 features per example and use a simple Dense layer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a Conv3D layer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use some form of RNN, such as LSTM.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From your data description, I would expect &lt;em&gt;either&lt;/em&gt; a TimeDistributed or 3D CNN based approach to be a good first bet. Intuition suggests that the CNN would work better if there was minor change between frames (because it has capability to directly find subtle frame differences), the TimeDistributed approach would work better processing larger changes (because it will ignore frame differences until the fully-connected layer).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I use a TimeDistributed layer?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Only you can answer this, by trying it and measuring performance of your classifier. However, it should function correctly, and intuition suggests it would be a good choice if your frames are in sequence but visually disjoint.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-01T09:06:56.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23445" PostTypeId="1" CreationDate="2017-10-01T12:16:04.680" Score="0" ViewCount="104" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cDQOS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cDQOS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I do not understand is that is there only one input for each weights or there is a vector of inputs from 1 to n?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also dont get the notation for inputs and outputs? Is that a notation or a power operation?&lt;/p&gt;&#xA;" OwnerUserId="39902" LastActivityDate="2017-10-02T10:57:14.050" Title="An optimization problem involving perceptron" Tags="|neural-network|optimization|perceptron|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23447" PostTypeId="2" ParentId="23378" CreationDate="2017-10-01T13:31:15.753" Score="1" Body="&lt;p&gt;Consider the case of the majority rule.&#xA;In the majority rule you go over the training set, check which concept value is the majority and returns it for every sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are no parameters, there is a training process and the classifier is has some value (mostly as a benchmark or in extreme cases). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note the the word &quot;parameter&quot; has different meanings and it is hard to know what one means without context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One might refer to hyper-parameters, like k in k-nn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One can also refer to &quot;problem definition&quot; parameters, like the distance function used in k-nn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different meaning is actually the content of the model. &#xA;I'm not sure which of the two meaning is the one the quote is using and whether the &lt;strong&gt;the order in which we&#xA;asked them&lt;/strong&gt; is the feature selection method (meaning 2) or the path to the leaf (meaning 3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last (and very common definition) is of a parameter in a &lt;a href=&quot;https://en.wikipedia.org/wiki/Parametric_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Parametric model&lt;/a&gt; which is &quot;is a family of distributions that can be described using a finite number of parameters. These parameters are usually collected together to form a single k-dimensional parameter vector θ = (θ1, θ2, …, θk).&quot;&#xA;Obviously, this is not the meaning used in the quote.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-10-01T13:31:15.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23449" PostTypeId="1" CreationDate="2017-10-01T17:43:41.877" Score="-1" ViewCount="53" Body="&lt;p&gt;I hope this is in the correct forum. I am working on a project for a &lt;a href=&quot;https://www.coursera.org/learn/statistical-inference&quot; rel=&quot;nofollow noreferrer&quot;&gt;course&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have this code (please see below) near the top of my .Rmd file and it knitr's just fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to make the output shorter (course requirement), so i was thinking of putting the code into an init function. Should one do this? Will it work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need the code to be run up front. I do want to show the code in the knitr document, but i would like to be in the appendix with the other code,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: looks like everything but the source can go into a function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;```{r}&#xA;#install.packages('e1071')  # do once in R console.&#xA;library(e1071) &#xA;source(&quot;functions.R&quot;)&#xA;set.seed(1)&#xA;initialize()&#xA;```&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit2: the code is now smaller:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;{r}&#xA;source(&quot;functions.R&quot;)&#xA;initialize()&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="39223" LastEditorUserId="39223" LastEditDate="2017-10-02T17:40:58.007" LastActivityDate="2017-10-02T17:40:58.007" Title="Where can i move this r markdown code?" Tags="|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23453" PostTypeId="2" ParentId="23445" CreationDate="2017-10-01T21:41:13.310" Score="1" Body="&lt;p&gt;The perceptron receives a $n$ number of $m$-dimensional vectors, so, $x^{i} \in \ R^{m}\  \forall\ i \in [1,n]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To iterate, the dataset has $n$ different examples and each of which has $m$ dimensions. It's not a power notation, it just denotes the $i$th example.&lt;/p&gt;&#xA;" OwnerUserId="39918" LastEditorUserId="29575" LastEditDate="2017-10-02T10:57:14.050" LastActivityDate="2017-10-02T10:57:14.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23454" PostTypeId="2" ParentId="23449" CreationDate="2017-10-01T22:03:31.220" Score="0" Body="&lt;p&gt;If you want the code to run but don't want it included set include=FALSE as shown below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;```{r pressure, include=FALSE}&#xA;TYPE CODE HERE&#xA;```&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17959" LastActivityDate="2017-10-01T22:03:31.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23455" PostTypeId="2" ParentId="14474" CreationDate="2017-10-01T22:59:33.880" Score="0" Body="&lt;p&gt;If there are factor levels in your test data that you haven't seen in training data--then, strictly speaking, you cannot predict anything for these observations.  You can overcome it by additional assumptions but I am afraid there is no other general solution here than just to give up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar considerations may apply to the other cases.  Are you willing to do out-of-sample predictions?  What are the underlying assumptions in your model than make it possible?  Which role do these assumptions play otherwise, do these influence your results a lot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, I agree with TBSRounder.  You have to use exactly the same transformation as what you applied for the training data.&lt;/p&gt;&#xA;" OwnerUserId="39920" LastActivityDate="2017-10-01T22:59:33.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23457" PostTypeId="1" CreationDate="2017-10-02T01:54:18.660" Score="1" ViewCount="788" Body="&lt;p&gt;I'm working on a Keras neural net that does key point prediction of body parts (left foot, left knee, left hip, etc.). For each image (X), the target (Y) is a list of coordinates for the keypoints (left_foot_x, left_foot_y, left_knee_x, left_knee_y), etc.). Whenever a keypoint isn't visible the X and Y coordinate equal = 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect that leaving the data in this state will lead to bad results because 0 really means NA (i.e. the keypoint isn't visible). Leaving the X and Y coordinates 0 would suggest the keypoint is actually in the corner which is false. &lt;strong&gt;What is the right way to structure data to do multi target regressions when the correct output is sometimes NA?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="39922" LastActivityDate="2017-11-01T11:56:12.947" Title="Multi target regression when some targets have correct NA labels" Tags="|regression|tensorflow|multiclass-classification|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23459" PostTypeId="2" ParentId="2618" CreationDate="2017-10-02T06:16:35.300" Score="1" Body="&lt;p&gt;I've found a fantastic way to get around IP address blocks is Amazon AWS (Or Azure or any other similar on-demand service). A t2.nano instance from AWS costs almost the same as a high quality proxy and is more than capable of handling rate-limited requests just fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say you need to scrape 100,000 pages. Using the AWS CLI, your program can automatically start 1,000 instances. Even if you need to wait 2 seconds between requests, you'd still be done in 200 seconds. And how much do you pay for it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, the price for a t2.nano instance is \$0.0058 per Hour in the Ohio region of AWS. For a thousand instances, that's only \$5.8 per hour. But you don't need the entire hour. Your job of 100,000 pages was finished in less than 200 seconds. Add some extra time for setting up the script, installing required packages, zipping the results and downloading them to your server/pc and you still used at most 10 minutes of server time per instance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or about one dollar.&#xA;100,000 pages in 200 seconds for one dollar. Not bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: When scaling like this, you have to be very careful about not accidentally overloading the scraping target. If you unleash this much firepower on a single website, that's about 1,000 requests hitting the server every alternate second. Enough to kill most webservers. So while the 1,000 servers option may be a good idea if you have a diversified list of sites, you'll probably need to use 10-20 servers at maximum if you're hitting a single site.&lt;/p&gt;&#xA;" OwnerUserId="39853" LastActivityDate="2017-10-02T06:16:35.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23461" PostTypeId="1" AcceptedAnswerId="23464" CreationDate="2017-10-02T08:37:32.003" Score="1" ViewCount="95" Body="&lt;p&gt;I was thinking about this lately. Let's say that we have a very complex space, which makes it hard to learn a classifier that can efficiently split it. But what if this very complex space is actually made up of a bunch of &quot;simple&quot; subspaces. By simple, I mean that it would be easier to learn a classifier for that subspace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this situation, would clustering my data first, in other words finding these subspaces, help me learn a better classifier? This classifier would essentially be an ensemble of each subspace's classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify, I don't want to use the clusters as additional features and feed it to a big classifier, I want to train on each cluster individually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something that's already been done/proven to work/proven to not work? Are there any papers on it? I've been trying to search for things like this but couldn't find anything relevant so I thought I'd ask here.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-10-02T09:09:10.390" LastActivityDate="2017-10-02T09:51:52.567" Title="Can clustering my data first help me learn better classifiers?" Tags="|classification|clustering|unsupervised-learning|meta-learning|ensemble|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23462" PostTypeId="1" AcceptedAnswerId="23466" CreationDate="2017-10-02T09:48:13.360" Score="5" ViewCount="15328" Body="&lt;p&gt;Why would the dimension of $w^{[2]}$ be $(n^{[2]}, n^{[1]})$ ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a simple linear equation, $z^{[n]}= W^{[n]}a^{[n-1]} + b^{[n]}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There seems to be an error in the screenshot.&#xA;the weight, $W$ should be transposed, please correct me if I am wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$W^{[2]}$ are the weights assigned to the neurons in the layer 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n^{[1]}$ is the number of neurons in layer 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Screenshot from &lt;a href=&quot;https://www.coursera.org/learn/neural-networks-deep-learning/lecture/6dDj7/backpropagation-intuition-optional&quot; rel=&quot;noreferrer&quot;&gt;Andrew Ng deeplearning coursera course video&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zK9kZ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zK9kZ.png&quot; alt=&quot;backpropagation algorithm&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39929" LastEditorUserId="29575" LastEditDate="2017-10-02T14:21:30.183" LastActivityDate="2022-09-18T03:44:14.353" Title="Dimension of weight matrix in neural network" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23463" PostTypeId="2" ParentId="23457" CreationDate="2017-10-02T09:48:40.890" Score="1" Body="&lt;p&gt;One option is to modify your network to output a binary label for each body part, signalling whether the part is visible or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would need the following changes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Modify your network architecture to return a real number in the range $[0, 1]$ for each body part. A sigmoid would be an appropriate activation for the last layer. Its semantics would be the &lt;em&gt;probability&lt;/em&gt; that the $i^{th}$ body part is visible.&lt;/li&gt;&#xA;&lt;li&gt;Extend your training and test data with appropriate values for the new labels.&lt;/li&gt;&#xA;&lt;li&gt;Change your loss to only take into account the difference between body part positions (expected vs predicted) if the part was expected to be found (e.g. by multiplying). I'm not very sure whether you should use the real labels or the predicted ones here (or even an average of both).&lt;/li&gt;&#xA;&lt;li&gt;Add a new term (added to the original loss) to your loss with the cross entropy of the body part presence &lt;em&gt;flag&lt;/em&gt;. You should probably add a weighting factor $\alpha$ to this new loss term to have a proper scale with respect to the position distance loss term.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-10-02T09:48:40.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23464" PostTypeId="2" ParentId="23461" CreationDate="2017-10-02T09:51:52.567" Score="3" Body="&lt;p&gt;It is absolutely a way to improve your classifier's accuracy. Actually a &quot;strong&quot; enough classifier such as a &lt;em&gt;neural network&lt;/em&gt; could be able to learn by itself these clusters. However, you would need a substancially deeper network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;smartest&quot; way to do this, if you know there are many groups/clusters in your data is to actually perform a 2-steps process:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cluster your data&lt;/li&gt;&#xA;&lt;li&gt;Train X models, one for each of your clusters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A nice way to visualise this is the following problem, you want to build a recommendation engine for a Netflix-like application, you don't want to build one model per person, how would you do this ?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First find clusters of similar users (geeks, SF fans, teenagers, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Fit one model for each of these clusters&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39198" LastActivityDate="2017-10-02T09:51:52.567" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23466" PostTypeId="2" ParentId="23462" CreationDate="2017-10-02T11:05:18.283" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;There seems to be an error in the screenshot. The weight, $W$ should be transposed, please correct me if I am wrong.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Matrix multiplication works so that if you multiply two matrices together, $C = AB$, where $A$ is an $i \times j$ matrix and $B$ is a $j \times k$ matrix, then C will be a $i \times k$ matrix. Note that $A$'s column count must equal $B$'s row count ($j$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the neural network, $a^{[1]}$ is a $n^{[1]} \times 1$ matrix (column vector), and $z^{[2]}$ needs to be a $n^{[2]} \times 1$ matrix, to match number of neurons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore $W^{[2]}$ has to have dimensions $n^{[2]} \times n^{[1]}$ in order to generate an $n^{[2]} \times 1$ matrix from $W^{[2]}a^{[1]}$&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-02T11:05:18.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23468" PostTypeId="2" ParentId="23233" CreationDate="2017-10-02T11:51:30.607" Score="9" Body="&lt;p&gt;Another approach was suggested in this paper for face age estimation:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The authors use a number of binary classifiers predicting whether a data point is larger than a threshold, and do this for multiple thresholds. I.e. in your case the network would have three binary outputs corresponding to&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;larger than 0&lt;/li&gt;&#xA;&lt;li&gt;larger than 1&lt;/li&gt;&#xA;&lt;li&gt;larger than 2.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For example, for 'large (2)' the ground-truth would be &lt;code&gt;[1 1 0]&lt;/code&gt;. The final cost function is a weighted sum of the individual cross-entropy cost functions for each binary classifier.&lt;/p&gt;&#xA;&lt;p&gt;This has the advantage of inherently weighting larger errors more because more of the individual cost-entropy terms will be violated. Simply doing categorical classification of the ordered outcomes doesn't inherently have this feature.&lt;/p&gt;&#xA;" OwnerUserId="29026" LastEditorUserId="29026" LastEditDate="2020-06-17T12:22:16.543" LastActivityDate="2020-06-17T12:22:16.543" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23470" PostTypeId="2" ParentId="23432" CreationDate="2017-10-02T13:59:41.040" Score="1" Body="&lt;p&gt;I would do two separate analyses:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;By day for six months&lt;/li&gt;&#xA;&lt;li&gt;By month for the last 36 months (with the most recent 6 aggregated)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The 36 month analysis will be important for catching annual trends you wouldn't be able to see with the daily data, but you'll have a lot more fine-grain with the daily analysis to catch quarterly, monthly, and weekly cycles.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-10-02T13:59:41.040" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23472" PostTypeId="1" AcceptedAnswerId="23473" CreationDate="2017-10-02T15:42:30.753" Score="2" ViewCount="47" Body="&lt;p&gt;I have been working with Tensor and matrix Non negative constrained algorithms. I have never seen a non negative constrained algorithm (ex. Non Negative Tucker Decomposition &lt;strong&gt;NTD&lt;/strong&gt;) with error that is comparable to the non constrained algorithm (ex.High Order SVD &lt;strong&gt;HOSVD&lt;/strong&gt;). Publications normally don't compare estimation error of unconstrained algorithms with constrained algorithms instead the comparison is always between different algorithms of same type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By Testing, I found that the result of constrained algorithms with noise is around $10^{-7}$ compared to unconstrained ones is around $10^{-15}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any reference that I can start from or any explanation would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="39941" LastEditorUserId="39941" LastEditDate="2017-10-04T14:29:22.503" LastActivityDate="2017-10-04T14:29:22.503" Title="Does Non Negativity Constrains increases the estimation error" Tags="|algorithms|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23473" PostTypeId="2" ParentId="23472" CreationDate="2017-10-02T15:50:03.123" Score="3" Body="&lt;p&gt;Are you talking about train error or test error?&#xA;Non negativity constrains make your search space of possible solutions much smaller. This means that the train error would be always higher or equal when you add non-negativity constraint. &#xA;But higher train error can mean that you are overfitting. If the assumption of non-negativity make sense for the solution of your problem, then it is likely that this assumption would decrease the test error as you are limiting your function space and decrease the chance of overfitting.&lt;/p&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2017-10-02T15:50:03.123" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23474" PostTypeId="1" CreationDate="2017-10-02T16:24:55.537" Score="1" ViewCount="323" Body="&lt;p&gt;This my Gradient descent function -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Take ϴ as 0,0&#xA;Till n iterations&#xA;    hypothesis = Dot product of ϴ and feature vector X&#xA;    loss = hypothesis - target vector&#xA;    descent = Dot product of Transpose of X and loss divided by the size of X&#xA;    ϴ = theta - alpha*grad&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now how to modify this to include Lasso and Ridge regression?&lt;/p&gt;&#xA;" OwnerUserId="39533" LastActivityDate="2017-10-02T16:24:55.537" Title="Algorithm to apply Lasso and Ridge in Gradient descent" Tags="|machine-learning|regression|linear-regression|gradient-descent|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23475" PostTypeId="1" CreationDate="2017-10-02T17:34:08.457" Score="10" ViewCount="643" Body="&lt;p&gt;To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where &lt;strong&gt;&lt;em&gt;variation within each group&lt;/em&gt;&lt;/strong&gt; (the filled circles) and &lt;strong&gt;&lt;em&gt;variation between the groups&lt;/em&gt;&lt;/strong&gt; (black arrows) are simple to be understood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'm wondering if it could be possible to extend the current visualization to a &lt;strong&gt;2 x 3 two-way ANOVA&lt;/strong&gt; (adding another way with two groups to the current visualization)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;strong&gt;Note:&lt;/strong&gt; the dashed vertical lines denote each group's mean)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gip67.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gip67.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39945" LastEditorUserId="13023" LastEditDate="2018-12-29T23:14:59.983" LastActivityDate="2024-02-20T22:01:35.210" Title="A clear visualization of a two-way ANOVA" Tags="|r|statistics|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23477" PostTypeId="1" AcceptedAnswerId="23482" CreationDate="2017-10-02T18:05:51.087" Score="0" ViewCount="139" Body="&lt;p&gt;I am new to Machine Learning and I have a conceptual question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a scaled dataset (scikit-learn and pandas).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will this discrepancy be a problem, if so, how should I resolve it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best,&lt;/p&gt;&#xA;" OwnerUserId="39949" LastEditorUserId="39949" LastEditDate="2017-10-02T20:18:37.650" LastActivityDate="2017-10-02T21:17:34.997" Title="Non Scaled New Actual Data" Tags="|machine-learning|python|scikit-learn|pandas|feature-scaling|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23479" PostTypeId="2" ParentId="23402" CreationDate="2017-10-02T19:47:04.883" Score="0" Body="&lt;p&gt;Got it. Thanks to everyone who helped. The issue had nothing to do with kmeans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not realize that when you do this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataframe[col] = Series, the series gets merged based on the index, as opposed to a simple appending of the column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataframe had been filtered before I generated the lists of features, so the index was not 0,1,2 but rather 0,2,5, etc. I needed to perform reset_index() on the original dataframe before assigning a new column in the dataframe to the labels from the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KOlXN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KOlXN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39814" LastActivityDate="2017-10-02T19:47:04.883" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23480" PostTypeId="2" ParentId="23427" CreationDate="2017-10-02T20:10:16.983" Score="0" Body="&lt;p&gt;A question similar to this has already been asked on Cross Validated. See:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/46019/why-squared-residuals-instead-of-absolute-residuals-in-ols-estimation&quot;&gt;Why squared residuals instead of absolute residuals in OLS estimation?&lt;/a&gt; and &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia&quot;&gt;Why square the difference instead of taking the absolute value in standard deviation?&lt;/a&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The former is actually a duplicate question from the latter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also benefit from answer to this post&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/127598/square-things-in-statistics-generalized-rationale?noredirect=1&amp;amp;lq=1&quot;&gt;square things in statistics- generalized rationale&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2017-10-02T20:10:16.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23481" PostTypeId="1" CreationDate="2017-10-02T20:17:27.253" Score="6" ViewCount="299" Body="&lt;p&gt;Take the following tensor:&#xA;$$&#xA;\left[\begin{array}{cc}&#xA;a &amp;amp; b &amp;amp; c\\&#xA;d &amp;amp; e &amp;amp; f\\&#xA;g &amp;amp; h &amp;amp; i\\&#xA;\end{array}\right] &#xA;$$&#xA;$$&#xA;\left[\begin{array}{cc}&#xA;j &amp;amp; k &amp;amp; l\\&#xA;o &amp;amp; n &amp;amp; m\\&#xA;p &amp;amp; q &amp;amp; r\\&#xA;\end{array}\right]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where each matrix represents a channel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This could be reshaped fairly easily into a vector:&#xA;$$&#xA;[a,d,g,b,e,h,c,f,i,j,o,p,k,n,q,l,m,r]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then concatenated row-wise with other vectorized tensors to form a typical flat-file dataset of dimension $N \times P$, where $N$ is the number of training samples and $P$ is the product of the tensor dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than futzing with a convolutional layer, one could simply constrain ones weights to be zero in the subsequent layer.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $X$ is a flat $N\times P$ dataset of concatenated vectorized tensors, then the convolutional weights would form a sparse matrix, with the first two columns of a $P \times 4$ convolutional &quot;layer&quot; being $2\times 2\times 2$ filter being&#xA;$$&#xA;\left[\begin{array}{c}&#xA;w 0\\ w0\\ 00\\ ww\\ ww\\ 00\\ 0w\\0w\\00\\w0\\ w0\\00\\ww\\ww\\0w\\0w\\0w\\00&#xA;\end{array}\right]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems to me more intuitive than the tensor formulation, and could be computed fairly simply using sparse matrix packages.  Perhaps it is partly matter of taste.  But I'm curious: is there anything &lt;em&gt;special&lt;/em&gt; about the tensor paradigm -- either mathematically or computationally -- that is superior to the flattened representation?  I understand that computers convert matrix algebra to for-loops &quot;under the hood&quot;, but doesn't the advent of the GPU make such explicit looping irrelevant?&lt;/p&gt;&#xA;" OwnerUserId="30335" LastEditorUserId="29169" LastEditDate="2021-02-15T22:23:02.307" LastActivityDate="2021-02-15T22:23:02.307" Title="What is the point of tensors in CNNs? Why not simply reshape the data into matrices?" Tags="|deep-learning|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23482" PostTypeId="2" ParentId="23477" CreationDate="2017-10-02T20:37:39.803" Score="0" Body="&lt;p&gt;You should save the scaler params used to fit the training set and use the &lt;em&gt;same&lt;/em&gt; ones to transform all other data used with the model from then on - whether CV, test or new unseen data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No that won't work. Once you add scaling/normalisation to the training pipeline, the exact same scaling (as in same scaling params, not re-calculated) should be applied to all input features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The scikit-learn scalers like e.g. &lt;code&gt;StandardScaler&lt;/code&gt; have two key methods:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;fit&lt;/code&gt; should be applied to your training data&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;transform&lt;/code&gt; should be applied after fit, and should be used on every data set to normalise model inputs.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;fit_transform&lt;/code&gt; can be used on the training data only to do both in a single step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need to do the training and predictions in different processes (maybe live predictions are on different devices for instance), then you need to save and restore the scaling params. One basic, simple way to do this is using &lt;code&gt;pickle&lt;/code&gt; e.g. &lt;code&gt;pickle.dump( min_max_scaler, open( &quot;scaler.p&quot;, &quot;wb&quot; ) )&lt;/code&gt; to save to a file and &lt;code&gt;min_max_scaler = pickle.load( open( &quot;scaler.p&quot;, &quot;rb&quot; ) )&lt;/code&gt; to load it back.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-02T21:17:34.997" LastActivityDate="2017-10-02T21:17:34.997" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23483" PostTypeId="2" ParentId="23481" CreationDate="2017-10-02T20:42:48.613" Score="4" Body="&lt;p&gt;Tensors come pretty natural in convolutionals networks.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This information gets lost when you reshape an image into a vector. Look how a &lt;a href=&quot;https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner&amp;#39;s-Guide-To-Understanding-Convolutional-Neural-Networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;convolutional layer&lt;/a&gt; works.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3Nvyo.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3Nvyo.gif&quot; alt=&quot;conv layer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Training usually is done in batches, which is another dimension as far as a neural network is concerned. You don't want to mix different images into a matrix, they are totally independent.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, there is depth channel. Initially it's one of R, G, B channels of the input image, and then each channel corresponds to a filter applied to the previous conv layer. Once again, filters are independent, it doesn't make sense to mix them up until the final layer.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, in total there are 4-rank tensors going through the conv net. It's not only more intuitive (each dimension has a meaning), but also results in higher accuracy, because it employs all meaningful information from the images.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2017-10-02T20:42:48.613" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23484" PostTypeId="1" CreationDate="2017-10-02T22:49:00.233" Score="0" ViewCount="2770" Body="&lt;p&gt;Ok this is extremely weird, can someone run this code and see if it crashes with that error?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import pandas  as pd&#xA;    import numpy as np&#xA;    from sklearn.covariance import MinCovDet&#xA;&#xA;    clf=MinCovDet()&#xA;&#xA;    dataF=np.array([0.5, 0.1, 0.1, 0.1, 0.957, 0.1, 0.1, 0.1, 0.4285, 0.1])&#xA;&#xA;    manual=pd.DataFrame(data=dataF)&#xA;    clf.fit(manual)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I change the array to this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Working&#xA;dataF=np.array([0.5, 0.11, 0.1, 0.1, 0.957, 0.1, 0.1, 0.1, 0.4285, 0.1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it runs fine. But it seems to crash with any array where there are too many of the same values. This array crashes as well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Not working&#xA;dataF = np.array([0.5, 0.3, 0.3, 0.3, 0.957, 0.3, 0.3, 0.3, 0.4285, 0.3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I already checked for NANs and everything, there's nothing.&lt;/p&gt;&#xA;" OwnerUserId="30712" LastEditorUserId="15946" LastEditDate="2017-10-03T14:46:58.397" LastActivityDate="2017-10-03T19:20:58.100" Title="scikit-learn minCovDet Input contains NaN, infinity or a value too large for dtype('float64')" Tags="|scikit-learn|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23486" PostTypeId="1" AcceptedAnswerId="23543" CreationDate="2017-10-03T02:40:32.240" Score="6" ViewCount="4145" Body="&lt;p&gt;For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ? Could anyone show me why $w^{[2]T}$ instead of just $w$ ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read other articles, but the transposed dimension of $w$ is ignored when chain derivation is applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Screenshot from &lt;a href=&quot;https://www.coursera.org/learn/neural-networks-deep-learning/lecture/6dDj7/backpropagation-intuition-optional&quot; rel=&quot;noreferrer&quot;&gt;Andrew Ng deeplearning coursera course video&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B0E5E.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B0E5E.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39929" LastEditorUserId="836" LastEditDate="2017-10-04T16:39:50.813" LastActivityDate="2018-08-06T13:30:15.490" Title="Proper derivation of dz[1] expression for backpropagation algorithm" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23487" PostTypeId="2" ParentId="17710" CreationDate="2017-10-03T06:27:22.397" Score="19" Body="&lt;p&gt;Let's define first Feature Engineering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Feature selection&lt;/li&gt;&#xA;&lt;li&gt;Feature extraction&lt;/li&gt;&#xA;&lt;li&gt;Adding features through domain expertise&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;XGBoost does (1) for you.&#xA;XGBoost does not do (2)/(3) for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.&lt;/p&gt;&#xA;" OwnerUserId="39963" LastEditorUserId="29575" LastEditDate="2017-10-03T14:47:00.973" LastActivityDate="2017-10-03T14:47:00.973" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23488" PostTypeId="1" CreationDate="2017-10-03T06:35:58.003" Score="3" ViewCount="812" Body="&lt;p&gt;While creating a nearest neighbor graph for isomap, there is a possibility that the graph is disconnected. In this case finding graph distances between all pairs of points will not be possible. Are there any simple methods other than iteratively changing the nearest neighbor search parameters  till we get a connected graph?&lt;/p&gt;&#xA;" OwnerUserId="17558" LastActivityDate="2023-12-30T06:00:25.057" Title="How to deal with disconnected components in isomap?" Tags="|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23489" PostTypeId="1" AcceptedAnswerId="23490" CreationDate="2017-10-03T07:05:12.847" Score="3" ViewCount="2987" Body="&lt;p&gt;I am a professional java developer with 9 years of experience. Never worked with Python but, I do not have any restraints learning it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still I need clarity on a couple of aspects. I hope if anyone can help, it will clear the cobwebs in my mind. I am looking to develop a machine learning project : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With a web interface to present interactive analytical graphs and views. &lt;/li&gt;&#xA;&lt;li&gt;Employing a web crawler collecting data, both structured and unstructured. &lt;/li&gt;&#xA;&lt;li&gt;A backend crunching the data collected using machine learning and artificial intelligence and natural language processing algorithms.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;While exploring options available, I found both Python and Java have extensive support and frameworks/libraries/modules available to leverage machine learning implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, working with Java gives me a sense of strongly typed programming and it can leverage the underlying CPU cores via its robust multi-threading APIs. Python on the other hand is scripted language and on the face value looks to lesser efficient when working on complex logic crunching a huge amount of data pertaining to Natural Language Processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might be utterly wrong as I am not very acquainted with Python ecosystem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me understand on the choice between Java and Python if I keep in mind execution speed, maintainability and community support targeted toward development of a large, scalable and robust machine learning/artificially intelligent application ?&lt;/p&gt;&#xA;" OwnerUserId="39965" LastActivityDate="2017-11-13T14:51:20.883" Title="Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?" Tags="|machine-learning|python|bigdata|data-cleaning|java|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23490" PostTypeId="2" ParentId="23489" CreationDate="2017-10-03T07:21:25.837" Score="7" Body="&lt;p&gt;I work with &lt;code&gt;python&lt;/code&gt; and &lt;code&gt;java&lt;/code&gt; in big-data settings every day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;python&lt;/code&gt; is definitely my language of choice when researching, experimenting and fitting models.&#xA;&lt;code&gt;python&lt;/code&gt; has a ton of very useful libraries such as &lt;code&gt;spacy&lt;/code&gt;, &lt;code&gt;nltk&lt;/code&gt; and &lt;code&gt;sklearn&lt;/code&gt; that makes exploration very easy, especially in within a &lt;code&gt;jupyter notebook&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When running the code in production, If performance is a consideration, we implement only the prediction phase in &lt;code&gt;java&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;java&lt;/code&gt; is more suited for production, has great support for packaging, tests, deployment.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastEditorUserId="12942" LastEditDate="2017-10-03T18:42:10.097" LastActivityDate="2017-10-03T18:42:10.097" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23491" PostTypeId="1" AcceptedAnswerId="23506" CreationDate="2017-10-03T09:54:42.617" Score="3" ViewCount="4656" Body="&lt;p&gt;I have a dataset of about 10000 unlabeled job titles (mostly very short titles) such as &lt;code&gt;head of mobile&lt;/code&gt; or &lt;code&gt;lead iOS developer&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to perform classification of those job titles in two different ways : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 1st classification is sorting job titles according to the type of job it represents (i.e. marketing, IT, healthcare, legal...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second classification would be about the level of seniority of a specific job (i.e. executive, manager, associate, trainee...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried an approach based on Word2Vec (vectors were taken from the Google Word2Vec set)  :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I cleaned and stemmed job titles&lt;/li&gt;&#xA;&lt;li&gt;I created a vector representing the job title as the average vector of the words that compose it &lt;/li&gt;&#xA;&lt;li&gt;I created a list of vectors representing the target values (executive could be the average vector of &lt;code&gt;+executive, +chief, -assistant&lt;/code&gt; for example)&lt;/li&gt;&#xA;&lt;li&gt;Find the least cosine distance between the job title vector and the target vectors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;While this approach gets decent result (about 70-80% accuracy) it is not enough for the task I'm planning to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was wondering if a better approach could be used (except labeling the data by hand and using that to train some algorithm)&lt;/p&gt;&#xA;" OwnerUserId="39974" LastActivityDate="2021-08-17T16:24:28.633" Title="Classifying job titles" Tags="|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23492" PostTypeId="1" CreationDate="2017-10-03T12:59:23.943" Score="6" ViewCount="6787" Body="&lt;p&gt;I want to use deep learning for regression. However, the number of training samples is not large. In image processing, some new samples are generated on the basis of initial data through tasks like rotation to increase the number of data sets. Can I know if there is any algorithm to generate more data based on current samples (data augmentation for regression)?&lt;/p&gt;&#xA;" OwnerUserId="39982" LastActivityDate="2018-05-19T20:01:10.577" Title="Data Augmentation for Regression" Tags="|deep-learning|regression|data-augmentation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23493" PostTypeId="1" AcceptedAnswerId="23502" CreationDate="2017-10-03T14:17:09.163" Score="23" ViewCount="38456" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/15484/sigmoid-vs-relu-function-in-convnets&quot;&gt;Here&lt;/a&gt; the answer refers to vanishing and exploding gradients that has been in &lt;code&gt;sigmoid&lt;/code&gt;-like activation functions but, I guess, &lt;code&gt;Relu&lt;/code&gt; has a disadvantage and it is its expected value. there is no limitation for the output of the &lt;code&gt;Relu&lt;/code&gt; and so its expected value is not zero. I remember the time before the popularity of &lt;code&gt;Relu&lt;/code&gt; that &lt;code&gt;tanh&lt;/code&gt; was the most popular amongst machine learning experts rather than &lt;code&gt;sigmoid&lt;/code&gt;. The reason was that the expected value of the &lt;code&gt;tanh&lt;/code&gt; was equal to zero and and it helped learning in deeper layers to be more rapid in a neural net. &lt;code&gt;Relu&lt;/code&gt; does not have this characteristic, but why it is working so good if we put its derivative advantage aside. Moreover, I guess the derivative also may be affected. Because the activations (output of &lt;code&gt;Relu&lt;/code&gt;) are involved for calculating the update rules.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-27T10:13:01.583" LastActivityDate="2022-06-14T02:42:39.797" Title="Why ReLU is better than the other activation functions" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|activation-function|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23496" PostTypeId="1" CreationDate="2017-10-03T15:28:19.903" Score="1" ViewCount="50" Body="&lt;p&gt;I have several images of one artist, but they are small size 300x200 px&#xA;can Neural Network somehow restore it to a big size like 1024x768 if, for example, I have only one big picture from this artist?&lt;/p&gt;&#xA;" OwnerUserId="39706" LastActivityDate="2017-10-03T15:28:19.903" Title="Can neural network make from Small images make big" Tags="|neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23497" PostTypeId="1" CreationDate="2017-10-03T15:39:51.400" Score="2" ViewCount="578" Body="&lt;p&gt;LSTMs and GRUs are meant to learn longer-term dependencies than basic RNN architectures that do not have gated architectures. However, both LSTMs and GRUs are still typically trained with truncated backpropagation through time. Naively, it seems that the choice of truncation length will limit how many time-steps the network can &quot;remember&quot;. Is this the case, or can an LSTM trained with BPTT truncated to n steps learn dependencies from more than n steps in the past?&lt;/p&gt;&#xA;" OwnerUserId="39993" LastActivityDate="2017-10-03T15:39:51.400" Title="What limits the long-term dependencies learned by an LSTM?" Tags="|neural-network|deep-learning|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23498" PostTypeId="1" AcceptedAnswerId="23500" CreationDate="2017-10-03T16:16:27.533" Score="1" ViewCount="88" Body="&lt;p&gt;I have a neural network that takes in roughly twelve values, and outputs a singly probability. The issue is that the network appears to be smart enough to realize there is a very significant correlation between three of the values and the output, and doesn't seem to be recognizing any correlation with the remainder of the inputs. In other words, the model has figured out that if the and of three specific inputs are non-zero, there is an ~80% chance the output is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems like expected behavior, but I am looking to have the network enhance it's 80% prediction with data from the other nine inputs. Is this possible? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My proposed solution requires a bit more information regarding the nature of the data. The twelve inputs can be categorized into four groups of three variables each. For instance, we have three metrics: purchase cost, warranty length and average maintenance costs. These three metrics are then applied to four different appliances in a household: the boiler, washing machine, dishwasher and refrigerator. The network is then supposed to output the probability of the owner of the household moving out in a few months. In this example, there happens to be a 80% correlation between refrigerators and the homeowner moving: if they own a refrigerator, there is an 80% chance they will move out within the next few months. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned before, the network is smart enough to recognize the correlation between the refrigerator and the moving out, but I am hoping to get it to augment this prediction with the inclusion of the other inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first strategy for addressing this would be to split the single network into four separate networks, each handling one appliance. These will output four probabilities total, which I can then either weight and compute by hand, or feed into a fifth network for final processing. My worry is that this would result in the same issue occurring - the final network would determine that ignoring the first three inputs and only returning the fridge-based probability is the best way to minimize error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second possible solution is to simply up the amount of iterations over the dataset, in the hope that the network happens to locate a slightly better solution by chance, then works to optimize it. This seems unlikely, and overly dependent on the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My third and final possible solution is to initially only train the network on input where the homeowner doesn't own a fridge (all fridge values are zero) such that the network optimizes it's solution without having to worry about the fridges, and then gradually introduce inputs from homeowners with fridges.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will any of these work? Is there a better solution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the Tensorflow Estimator API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="39992" LastActivityDate="2017-10-03T17:03:18.827" Title="How to prevent a neural network from choosing the 'easiest' solution" Tags="|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23500" PostTypeId="2" ParentId="23498" CreationDate="2017-10-03T17:03:18.827" Score="2" Body="&lt;p&gt;This is fairly common with scenarios like predicting fraud when overall cases of fraud are uncommon. The model will frequently just say &quot;no fraud!&quot; and be right a large percent of the time. You need to assign a penalty, cost, or weight so that output 1 isn't so easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This article describes that further in a lot of detail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.fineighbor.com/tensorflow-dealing-with-imbalanced-data-eb0108b10701&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.fineighbor.com/tensorflow-dealing-with-imbalanced-data-eb0108b10701&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-10-03T17:03:18.827" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23502" PostTypeId="2" ParentId="23493" CreationDate="2017-10-03T18:37:20.097" Score="31" Body="&lt;p&gt;The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (&lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; by Krizhevsky et al).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it's not the only advantage. &lt;a href=&quot;https://stats.stackexchange.com/a/176905/130598&quot;&gt;Here&lt;/a&gt; is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property is that compared to tanh / sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'm not convinced that great success of modern neural networks is due to ReLu &lt;em&gt;alone&lt;/em&gt;. New initialization techniques, such as Xavier initialization, dropout and (later) batchnorm also played very important role. For example, famous AlexNet used ReLu &lt;em&gt;and&lt;/em&gt; dropout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer your question: ReLu has very nice properties, though &lt;a href=&quot;https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks&quot;&gt;not ideal&lt;/a&gt;. But it truly proves itself when combined with other great techniques, which by the way solve non-zero-center problem that you've mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPD: ReLu output is not zero-centered indeed and it does hurt the NN performance. But this particular issue can be tackled by other regularization techniques, e.g. batchnorm, which &lt;a href=&quot;https://arxiv.org/abs/1502.03167&quot; rel=&quot;noreferrer&quot;&gt;normalizes the signal before activation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We add the BN transform immediately before the nonlinearity, by&#xA;  normalizing $x = Wu+ b$. ... normalizing it is likely to produce&#xA;  activations with a stable distribution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18375" LastEditorUserId="18375" LastEditDate="2017-10-03T19:23:50.057" LastActivityDate="2017-10-03T19:23:50.057" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="23503" PostTypeId="2" ParentId="23491" CreationDate="2017-10-03T18:40:55.287" Score="1" Body="&lt;p&gt;I'm surprised you got 80% accuracy and think you will have to do some labeling if you want to improve. Even people have problems with this! For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sales Executive - executive can be misleading, this might have been called &quot;Salesman&quot; 50 years ago&lt;/li&gt;&#xA;&lt;li&gt;VP at a technology company vs VP at a bank. In the latter case, VP is &lt;a href=&quot;https://en.wikipedia.org/wiki/Vice_president#Expanded_use&quot; rel=&quot;nofollow noreferrer&quot;&gt;more of an honorific&lt;/a&gt; and might not include any actual management.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you don't want to mass label data, I would try to assign weights to common words that are featured in titles. For example, replace senior/junior with +1 and -1 in a &quot;seniority&quot; value. If you do that for the top 100 words, I bet it will help over a totally unsupervised approach.&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-10-03T18:40:55.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23504" PostTypeId="1" CreationDate="2017-10-03T18:51:51.500" Score="0" ViewCount="1083" Body="&lt;p&gt;My name is Reza, a Master Student at Golestan University of Technical and engineering, Iran and my M.Sc. thesis is about detect fake account on social networks. so i need data set. how can i collect data set from social networks like instagram????&lt;/p&gt;&#xA;" OwnerUserId="31731" LastActivityDate="2017-10-05T08:33:10.010" Title="how can i collect data set from social networks like instagram?" Tags="|data-mining|dataset|social-network-analysis|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23506" PostTypeId="2" ParentId="23491" CreationDate="2017-10-03T18:53:01.860" Score="1" Body="&lt;p&gt;A reasonable assumption to make, is that titles that share indicative words, are in the same sector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Examples:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Positive example: &quot;senior IOS developer&quot; and &quot;principal web developer&quot; seem to be both in the IT sector, and the word &quot;developer&quot; is our giveaway.&lt;/li&gt;&#xA;&lt;li&gt;Negative example: &quot;chief operations officer&quot; and &quot;chief data officer&quot; share 2 words, but those words (chief,officer) are not indicative&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, I would maintain a list of non-indicative words, such as &quot;senior&quot;,&quot;chief&quot;,...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Filter them out, and then apply an hierarchical clustering algorithm on the word-distance between titles.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-03T18:53:01.860" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23507" PostTypeId="2" ParentId="23484" CreationDate="2017-10-03T19:20:58.100" Score="1" Body="&lt;p&gt;Well, it seems to be an issue with the library, I opened an issue in gitHub&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/9864&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scikit-learn/scikit-learn/issues/9864&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30712" LastActivityDate="2017-10-03T19:20:58.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23509" PostTypeId="1" AcceptedAnswerId="23585" CreationDate="2017-10-03T20:29:47.290" Score="0" ViewCount="579" Body="&lt;p&gt;I've wrote &lt;a href=&quot;https://www.kaggle.com/questions-and-answers/40231#post225658&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; about my problem. And now I have a new one.&#xA;How to use time as a feature? I have a pandas DataFrame with date-time column('date') and I don't know how to use it as a feature. For now I'm using  TfidfVectorizer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;vectorizer = TfidfVectorizer(min_df=5, max_features=10000, ngram_range=(1, 2))&#xA;vz = vectorizer.fit_transform(df['word'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;strong&gt;'word'&lt;/strong&gt; column is tokens on my text. &#xA;I'm using MiniBatchKMeans. Here is code example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import MiniBatchKMeans&#xA;&#xA;num_clusters = 32&#xA;kmeans_model = MiniBatchKMeans(n_clusters=num_clusters, init='k-means++',     n_init=1,  init_size=1000, batch_size=1000, verbose=False, max_iter=1000)&#xA;kmeans = kmeans_model.fit(vz)&#xA;kmeans_clusters = kmeans.predict(vz)&#xA;kmeans_distances = kmeans.transform(vz)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm interested at how to add my time feature(my time column) to clusterizator and is there any possibility to make some kind of &quot;weight&quot; factor for time features and for different tokens.&lt;/p&gt;&#xA;" OwnerUserId="40002" LastActivityDate="2017-10-06T16:34:28.327" Title="How to add time as a feature into clustering algorithm?" Tags="|machine-learning|python|nlp|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23510" PostTypeId="1" CreationDate="2017-10-03T20:50:58.390" Score="1" ViewCount="256" Body="&lt;p&gt;I'm reading &lt;a href=&quot;https://github.com/ckmarkoh/neuralart_tensorflow/blob/master/main.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;someone else's code&lt;/a&gt;, which uses vgg19 to transfer the style of an artist to a 'content' image. I'm wondering why they have put&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INI_NOISE_RATIO * noise_img + (1. - INI_NOISE_RATIO) * content_img)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead of just &lt;code&gt;noise_img&lt;/code&gt; (&lt;code&gt;INI_NOISE_RATIO=0.7&lt;/code&gt; here).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why use a mix of the noise image and content image as input for the white noise bit in the content loss? The paper &lt;a href=&quot;https://arxiv.org/pdf/1508.06576.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1508.06576.pdf&lt;/a&gt; by Gatys et.al doesn't mention the mixing?&lt;/p&gt;&#xA;" OwnerUserId="40003" LastEditorUserId="13748" LastEditDate="2017-12-03T14:04:34.657" LastActivityDate="2017-12-03T14:04:34.657" Title="Using a White noise image to minimise the loss in a (convolutional) neural network" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23511" PostTypeId="2" ParentId="23510" CreationDate="2017-10-03T21:35:50.533" Score="1" Body="&lt;p&gt;The loss functions are not related to the noise image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network is run twice to determine parameters to the loss functions. Once for the content image, and once for the style image. The code you are asking about occurs afterwards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First the network is run against the content image (note no noise mixing here):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess.run([net['input'].assign(content_img)])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Aside: Uses an assignment function - not a &lt;code&gt;feed_dict&lt;/code&gt; - because in order for the style transfer to work, the input has become a tf variable, not a placeholder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the first loss function is defined against some of the network parameters from this run (this extracts the activations of interest from the networks as fixed params to the loss function - because it uses output from &lt;code&gt;sess.run&lt;/code&gt;, not references to tensor variables):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost_content = sum(map(lambda l,: l[1]*&#xA;  build_content_loss(sess.run(net[l[0]]),net[l[0]]) , CONTENT_LAYERS))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The same thing is done for the style image:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess.run([net['input'].assign(style_img)])&#xA;cost_style = sum(map(lambda l: l[1]*&#xA;      build_style_loss(sess.run(net[l[0]]), net[l[0]]), STYLE_LAYERS))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And a final cost function defined:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cost_total = cost_content + STYLE_STRENGTH * cost_style&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, after the cost function is defined, the code initialises a new input (which is also our style transfer output):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess.run(net['input'].assign( INI_NOISE_RATIO* noise_img + &#xA;                   (1.-INI_NOISE_RATIO) * content_img))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This line has nothing to do with defining loss. It just initialises the network ready to generate gradients against a starting image and the loss functions that were already defined beforehand. You can start with pure noise or a copy of the content image to start the process, and each will give a slightly different result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper started from a random image. Quoting from the paper (on page 10):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Thus we can change the initially random image $\tilde{x}$ until it generates the same response in a certain layer of the CNN [. . .]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;. . . however, starting from the content image was tried and used very soon afterwards by many in practice, and it works well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This mix of both approaches is what the author has chosen, presumably because it gets pleasing results. As there are no lower layers in the network than the input, in fact you can start from an arbitrary image (e.g. all zeros), and you should still get a result, although it may not look as nice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To (sort of) answer your question as written:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why use a mix of the noise image and content image as input for the white noise bit in the content loss?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The code is not doing that. The mixing is occurring as you say. But the line you show is not part of the content loss. The mixed image is the initial solution, and iterations will turn it into the stylised image.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-03T21:51:28.100" LastActivityDate="2017-10-03T21:51:28.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23513" PostTypeId="1" CreationDate="2017-10-03T22:39:00.280" Score="-1" ViewCount="592" Body="&lt;p&gt;What is an easy way to tweak a popular NN package and use my own coded activation function, instead of its internal activation functions like ReLU or sigmoid? Preference is with R code tweaks, and if not-then in Python.&lt;/p&gt;&#xA;" OwnerUserId="40008" LastActivityDate="2017-10-04T01:36:42.487" Title="NN with different activation function" Tags="|machine-learning|python|neural-network|r|" AnswerCount="1" CommentCount="1" ClosedDate="2017-10-05T00:52:02.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="23515" PostTypeId="2" ParentId="23513" CreationDate="2017-10-04T01:36:42.487" Score="1" Body="&lt;p&gt;Here is an example in &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; and Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;&#xA;def custom_activation(x):&#xA;    return abs(x)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(64, &#xA;            input_dim=20,&#xA;            activation=custom_activation))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-10-04T01:36:42.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23516" PostTypeId="1" AcceptedAnswerId="23518" CreationDate="2017-10-04T03:17:28.380" Score="1" ViewCount="107" Body="&lt;p&gt;I'm trying to grasp the structure of this convoluted neural network.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.imgur.com/zmFjHn0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.imgur.com/zmFjHn0.png&quot; alt=&quot;Image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;a href=&quot;https://arxiv.org/pdf/1603.02199.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;I understand the first layer is a 6x6 conv with stride 2 followed by 3x3 max pool and then 6 5x5 convs and another 3x3 max pool. After this, however, outputs from a fully connected layer of 64 neurons are &amp;quot;tiled over the special dimensions of the response map of pool2&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;I don't understand what this means. The output of pool2 should be 64 (because of 64 filters) 18x18 arrays. In the first 18x18 array do I add output1 to each of the 18*18=324 values, and in the second array add output2 to each of the 324 values, etc.?&lt;/p&gt;&#xA;&lt;h3&gt;TLDR&lt;/h3&gt;&#xA;&lt;p&gt;What do I do with the 64 outputs (each is a 18x18 array) and the 64 outputs from the fully connected layer?&lt;/p&gt;&#xA;" OwnerUserId="40016" LastEditorUserId="29169" LastEditDate="2021-02-15T22:22:50.537" LastActivityDate="2021-02-15T22:22:50.537" Title="Need help understanding the structure of this convoluted neural network" Tags="|machine-learning|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23517" PostTypeId="1" CreationDate="2017-10-04T07:04:20.083" Score="-2" ViewCount="126" Body="&lt;p&gt;I am new to data science. I am confused about the differences between statistical learning and predictive analytics. To me, it looks like both are the same as we try to work on function of x(attributes) for a target value y with very low error. Is this a correct assessment?&lt;/p&gt;&#xA;" OwnerUserId="40020" LastEditorUserId="29169" LastEditDate="2019-06-11T14:50:36.747" LastActivityDate="2019-06-11T14:50:36.747" Title="What is the difference between statistical learning and predictive analytics?" Tags="|machine-learning|data-mining|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23518" PostTypeId="2" ParentId="23516" CreationDate="2017-10-04T08:04:38.310" Score="1" Body="&lt;p&gt;The so-called motor command $v_t$ (I don't know what it means but it looks to be some scalar feature although it would work the same if it's a vector) is fed into a layer that builds 64 representations of this value, one for each feature map in the convolutional layer that we are going to add it too. The conv layers have a spatial resolution however, and this representation is only one number for the corresponding feature map. What they do is tile this number so that this whole motor command representation has the same dimensions as the max pooling layer after the convolutional layer (pool2). Now that the dimensions match we can use an element-wise addition operation to inject this information into the convolutional network.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-04T08:04:38.310" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23520" PostTypeId="2" ParentId="17710" CreationDate="2017-10-04T11:42:32.283" Score="19" Body="&lt;ol&gt;&#xA;&lt;li&gt;Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model. I typically use low numbers for row and feature sampling, and trees that are not deep and only keep the features that enter to the model. Then fine tune with another model. This prevented overfitting for me when the number of features was very high.&lt;/li&gt;&#xA;&lt;li&gt;Feature generation: XGBoost (classification, booster=gbtree) uses tree based methods. This means that the model would have hard time on picking relations such as &lt;em&gt;ab&lt;/em&gt;, &lt;em&gt;a/b&lt;/em&gt; and &lt;em&gt;a+b&lt;/em&gt; for features &lt;em&gt;a&lt;/em&gt; and &lt;em&gt;b&lt;/em&gt;. I usually add the interaction between features by hand or select the right ones with some heuristics. Depending on the application, this can really boost the performance.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2017-10-04T11:42:32.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23521" PostTypeId="1" AcceptedAnswerId="23522" CreationDate="2017-10-04T11:43:18.103" Score="0" ViewCount="593" Body="&lt;p&gt;When we use kernels in SVM to linearly sperate non linear data points by mapping it to 'another dimensions', does this suitable 'another dimensions' always be a higher dimension with respect to original dimension of the data points?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And is it true that we can always find a higher dimensions that can linearly seperate data points in a training set?&lt;/p&gt;&#xA;" OwnerUserId="27916" LastActivityDate="2017-10-04T11:57:52.880" Title="Does Kernels always map data points to higher dimensions" Tags="|machine-learning|classification|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23522" PostTypeId="2" ParentId="23521" CreationDate="2017-10-04T11:57:52.880" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;'Another dimension' can be also a lower dimensional space. You are free to choose your kernel. Example is: $k\big((x_1, y_1), (x_2, y_2)\big) = x_1x_2$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it true that we can always find a higher dimensions that can linearly seperate data points? Yes. If you have a finite dataset, you can always find a higher dimension that can linearly seperate data points.  The obvious one is mapping &lt;em&gt;n&lt;/em&gt; points to a &lt;em&gt;n&lt;/em&gt;-dimensional space. If you do any non-trivial mapping (no &lt;em&gt;d&lt;/em&gt; points are on a &lt;em&gt;d-1&lt;/em&gt; dimensional hyperspace), then you can always linearly separate any two class assignments.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2017-10-04T11:57:52.880" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23523" PostTypeId="1" AcceptedAnswerId="23646" CreationDate="2017-10-04T11:57:54.557" Score="4" ViewCount="1194" Body="&lt;p&gt;Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There probably is an algorithm for decoding top-N probable hidden states sequences (&lt;em&gt;k-shortest paths&lt;/em&gt; or the like).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is there a &lt;strong&gt;good implementation&lt;/strong&gt; anywhere?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPD&lt;/strong&gt; (copy-pasted from comments): Viterbi algorithm decodes the MOST probable sequence given observations + model. That is, argmax_x p(state_1,...,state_n|obs_1,...,obs_n). What I am asking for is an implementation of how to get the 'next argmax-es' with possibly slightly smaller probabilities GIVEN the observed sequence. &lt;/p&gt;&#xA;" OwnerUserId="14048" LastEditorUserId="14048" LastEditDate="2017-10-10T23:20:51.267" LastActivityDate="2017-11-09T23:51:01.353" Title="Viterbi-like algorithm suggesting top-N probable state sequences implementation" Tags="|algorithms|graphs|markov-process|sequence|graphical-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23525" PostTypeId="2" ParentId="23504" CreationDate="2017-10-04T12:20:33.267" Score="1" Body="&lt;p&gt;I am not aware about Instagram, or if there is a ready-to-use dataset for your purposes, but in general in order to get data from (almost) any online source you should a method called &quot;web scraping&quot; (This might help you while you google).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might also be useful to you, if you decide what kind of tools you are going to use for your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried something simple, but it was for Twitter though&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a rather simple piece of code that gets twitter &lt;code&gt;500&lt;/code&gt; tweets concerning &lt;code&gt;Barbie&lt;/code&gt; using &lt;code&gt;R&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(twitteR)&#xA;&#xA;setwd(&quot;C:/Users/slin/desktop/informatie&quot;)&#xA;&#xA;ck &amp;lt;- &quot;4bKJU1Z4B60SzbPPIva7OknnQ&quot;&#xA;cs &amp;lt;- &quot;cCxAPYDN3d0APvlurEBoXMNCKl0DKgrneV0w6KQCQenNpmM7fa&quot;&#xA;at &amp;lt;- &quot;715872051559575552-qMYaugTPHJeRg9fh70ABNWCjdl0Lp6T&quot;&#xA;as &amp;lt;- &quot;zxn6lOyCw2dcPyllkiHJq4zk0wHwGQ4tuKa9IpR96TNvO&quot;&#xA;&#xA;setup_twitter_oauth(ck, cs, access_token = at, access_secret = as)&#xA;&#xA;t_stream &amp;lt;- searchTwitter('barbie', resultType=&quot;recent&quot;, n=500)&#xA;&#xA;df &amp;lt;- do.call(&quot;rbind&quot;, lapply(t_stream, as.data.frame))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;ck&lt;/code&gt;, &lt;code&gt;cs&lt;/code&gt;, &lt;code&gt;at&lt;/code&gt; and &lt;code&gt;as&lt;/code&gt; can be obtained as it is explained &lt;a href=&quot;https://nocodewebscraping.com/twitter-json-examples/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; in &lt;code&gt;step 2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in &lt;a href=&quot;https://nocodewebscraping.com/twitter-json-examples/&quot; rel=&quot;nofollow noreferrer&quot;&gt;that&lt;/a&gt; site you can see a guide of how to web scrap data from Twitter using Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also find useful &lt;a href=&quot;https://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (Python), &lt;a href=&quot;http://scrapingexpert.com/how-to-scrape-data-from-twitter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (Python) and &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (R)&lt;/p&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2017-10-05T08:08:47.283" LastActivityDate="2017-10-05T08:08:47.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23527" PostTypeId="2" ParentId="23492" CreationDate="2017-10-04T12:29:25.730" Score="0" Body="&lt;p&gt;You can try something like SMOTE and see how your newly generated data fits your requirements. If your data has a statistical model you can use an appropriate parametric model to generate data. You can even try methods like Non Parametric estimation such as Parzen windows etc. All of this depends on the statistical fit of your image data which you have processed so far. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other Methods similar to SMOTE: ADASYN, Boundary SMOTE etc (Look them up on the internet)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: Apply SMOTE on the already processed image data such as the training data you already have not directly on the image itself. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-04T12:29:25.730" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23528" PostTypeId="2" ParentId="23517" CreationDate="2017-10-04T12:37:31.950" Score="0" Body="&lt;p&gt;Both words can be used interchangeably. However in the interest of a more accurate answer to help you out, you can say statistical methods include the application of mathematical models and fits which aid in the best estimation of the trend for a set of variables in a dataset. These include Regression, Clustering etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Predictive analytics deals with the &quot;forecast&quot; aspect of statistical methods. An easy example would be that of regression as you mentioned in your question, deals with a function where x is evaluated to find y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To conclude we can say that predictive analytics is a subset of statistical methods. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-04T12:37:31.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23531" PostTypeId="1" AcceptedAnswerId="23532" CreationDate="2017-10-04T13:28:02.317" Score="0" ViewCount="100" Body="&lt;p&gt;&lt;strong&gt;My goal is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate the probability of a client take a loan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My problem is:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to take in account the date of the loan in this process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Certainly the date is important here since, let's assume my clients usually take loans closer to Christmas, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought was to use the day of the year, since this would absorb those &quot;special dates&quot; like between days 340~365 they are more likely to take the loan since this is close to Christmas and this way I don't have to &quot;manually consider&quot; those special dates and can even discover new special periods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a sound approach?&lt;/p&gt;&#xA;" OwnerUserId="40030" LastEditorUserId="29575" LastEditDate="2017-10-04T13:34:23.703" LastActivityDate="2017-10-04T13:34:23.703" Title="Using dates from predicting loan" Tags="|machine-learning|data-mining|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23532" PostTypeId="2" ParentId="23531" CreationDate="2017-10-04T13:33:31.570" Score="1" Body="&lt;p&gt;You are going in the right direction. Dates can contain a lot of information depending on the task you want to learn. A problem with your suggestion of using the day of the year straight up is that the last day of the year and the first day of the following year are very close to each other, while in your representation they are the furthest away. An alternative to this could be using the fact that this is cyclical and map the &lt;code&gt;day_of_year&lt;/code&gt; feature onto a circle and use the coordinates on this circle as features to represent this. A lot of the important features regarding dates are cyclical, like the day of the week, the day of the month, the hour of the day etcetera. An alternative which allows for easier non-linear relationships would be to one-hot encode them as classes. This has the advantage of a more free representation, but the downside that it will cost more features and it cannot generalize based on the fact that two options might be next to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regards to additional domain knowledge, like knowing that being close to christmas, or close to salary day might be relevant, it's almost always beneficial to add these features to your data. This could be done by calculating absolute distance to christmas, or days before pay day etcetera. Some of these features can be work intensive to create, for example localized holiday features, but they can significantly influence people's decisions and if that is what you are trying to predict, adding these will benefit your performance.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-04T13:33:31.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23533" PostTypeId="1" CreationDate="2017-10-04T14:17:23.950" Score="1" ViewCount="112" Body="&lt;p&gt;I am just getting into Googles Cloud services, and have not been doing much ML in Python. I have done one or two ML projects in school using Tensorflow, but not really accustomed with the other available libraries. Is it so that if I am interested in using the Google Cloud services I should definitely be using Tensorflow, or can they be integrated with Theano or other ML libraries too? If they can be used, can they only be used in part and some functionality is lost, or are they fully available?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just looking for some guidance if there is any argument for using something else than Tensorflow for someone in my shoes, and don't really find good guidance using Google as they promote their own product.&lt;/p&gt;&#xA;" OwnerUserId="39752" LastActivityDate="2017-10-04T14:17:23.950" Title="Is it required to use Tensorflow in Google ML Engine, or can Theano etc be used?" Tags="|tensorflow|cloud|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23534" PostTypeId="1" CreationDate="2017-10-04T14:19:26.723" Score="2" ViewCount="523" Body="&lt;p&gt;I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in a list (of millions) that includes strings  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['rs12346','rs1212122',...,'sxs-rs333',...,'kgp222']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;..is there a way to extract the following patterns?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;'rs' plus one or more digits&lt;/li&gt;&#xA;&lt;li&gt;'sxs-rs' plus one or more digits&lt;/li&gt;&#xA;&lt;li&gt;'kgp' plus one or more digits&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some other parameters:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;cleaning up beforehand isn't an option&lt;/li&gt;&#xA;&lt;li&gt;some manual tweaking would be possible (e.g. manually changing a pattern because of outside knowledge)&lt;/li&gt;&#xA;&lt;li&gt;exceptions (i.e. classifying less than 100% of the strings) can be tolerated &lt;/li&gt;&#xA;&lt;li&gt;an ideal solution would be using a built-in python library. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="3594" LastEditorUserId="3594" LastEditDate="2019-03-14T12:57:56.267" LastActivityDate="2019-03-14T12:57:56.267" Title="Discovering string &quot;motifs&quot; in python" Tags="|python|clustering|data-cleaning|text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23535" PostTypeId="1" AcceptedAnswerId="23536" CreationDate="2017-10-04T14:36:46.880" Score="1" ViewCount="82" Body="&lt;p&gt;We have a custom dataset of 20 thousand images with two pixel-wise labeled classes.&#xA;However we have 1 million more raw images, which we would like to label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to label the most important new images first. Importance is defined as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Images with the more new information&lt;/li&gt;&#xA;&lt;li&gt;Images helping our deep learning bounding box classifier to improve&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So instead of labeling images, where we already have about thousand similar ones, we first want to label the images which are quite different from the already-labeled ones and help more to improve our classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can we assign priorities and decide which images to label first?&lt;/p&gt;&#xA;" OwnerUserId="40035" LastActivityDate="2017-10-04T15:09:52.503" Title="How to decide which images to label next?" Tags="|image-classification|labels|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23536" PostTypeId="2" ParentId="23535" CreationDate="2017-10-04T14:45:28.150" Score="2" Body="&lt;p&gt;This type of problem is considered to be part of 'active learning'. There is a lot of research being done on this topic at the moment, but some first approaches are relatively easy, depending on the type of model that you are using. Since you mentioned that you are using deep learning bounding box detectors, I will showcase a few examples of how to approach this problem using Convolutional neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The core idea is that we want some measure of potential gain of an unlabeled sample. That way we can train our model on our labeled training set, predict the labels for our unlabeled set and measure which examples will be most useful to label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of classification you could use the sigmoid/softmax output and get some kind of uncertainty from there, however deep learning models are usually fairly certain about their predictions and a high probability doesn't automatically mean that it predicts it well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to use dropout in your model during training, and then apply dropout to your predictions on your unlabeled set as well. By sampling multiple dropout masks and comparing all the different predictions, you could measure how different the outputs are. If the outputs are very similar, it's unlikely that your model will learn much more if you label this, but if the outputs vary wildly, maybe this example lives in a part of your feature space that your model doesn't know or understand very well yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot of ways to approach this, what I have written here is just an introduction to the concept of 'active learning'. There are a lot of papers available about this topic! EDIT: I haven't actually read a lot of this research, but here are a few:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1703.02910.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1703.02910.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1707.05928.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1707.05928.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1701.03551.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1701.03551.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-10-04T15:09:52.503" LastActivityDate="2017-10-04T15:09:52.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23538" PostTypeId="1" CreationDate="2017-10-04T14:48:14.050" Score="1" ViewCount="392" Body="&lt;p&gt;How do we design a CNN for ordinal classification?&lt;/p&gt;&#xA;&lt;p&gt;I am trying to analyze plant leaf images for diseases. I've got the disease type classification working. Now we need to classify the magnitude of the disease affliction on a scale of 1 to 10 (ordinal scale), where 1 is almost no trace of diseased parts on the leaf and 10 is completely diseased.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it possible to construct a CNN for this task?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How do I deal with the imbalance in training samples for each ordinal level?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2515" LastEditorUserId="29169" LastEditDate="2021-02-15T22:22:40.623" LastActivityDate="2021-02-15T22:22:40.623" Title="CNN for ordinal classification" Tags="|machine-learning|deep-learning|image-classification|class-imbalance|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23539" PostTypeId="1" AcceptedAnswerId="23564" CreationDate="2017-10-04T15:13:46.197" Score="6" ViewCount="4203" Body="&lt;p&gt;What is the difference between Recursive Feature Elimination (RFE) function and SelectFromModel in Scikit-Learn? Both seems exactly similar.&lt;/p&gt;&#xA;" OwnerUserId="33702" LastActivityDate="2017-10-05T12:55:50.127" Title="Difference between RFE and SelectFromModel in Scikit-Learn" Tags="|python|scikit-learn|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23540" PostTypeId="1" CreationDate="2017-10-04T15:14:15.157" Score="4" ViewCount="2836" Body="&lt;p&gt;I have a regression problem where a neural network has to predict a value from $0$ to $19.999...$. I would like to convert this regression problem into a classification problem with the classes $0, 1, 2, ..., 18, 19$. Therefore I am using a $\mathrm{softmax}$-output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if I use a categorical crossentropy-loss, the loss is the same if the network classifies an input as $1$ instead of $14$ and if the input is classified as $13$ instead of $14$. I like to punish the loss more if the predicted value / class is farther away from the true value / class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better loss function than categorical crossentropy for this use-case?&lt;/p&gt;&#xA;" OwnerUserId="23909" LastEditorUserId="29575" LastEditDate="2017-10-04T15:32:13.423" LastActivityDate="2017-10-04T15:34:26.287" Title="Converting a regression problem into a classification problem" Tags="|neural-network|regression|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23541" PostTypeId="2" ParentId="23540" CreationDate="2017-10-04T15:34:26.287" Score="3" Body="&lt;p&gt;I think what you should be looking for is &lt;a href=&quot;https://en.wikipedia.org/wiki/Ordinal_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;ordinal regression/classification&lt;/a&gt; in order to utilize the ordering of the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example &lt;a href=&quot;https://pythonhosted.org/mord/&quot; rel=&quot;nofollow noreferrer&quot;&gt;python&lt;/a&gt;/&lt;a href=&quot;https://github.com/ayrna/orca&quot; rel=&quot;nofollow noreferrer&quot;&gt;matlab&lt;/a&gt; implementations, with some extra resources.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For fast integration with your system, you could create a new ordinal regression model using the outputs of the already trained network as input to the new ordinal model and the true ordinal value as the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For another quick/dirty solution you could try simple Mean Absolute Error or Mean Squared Error, &lt;a href=&quot;https://link.springer.com/content/pdf/10.1007/978-3-642-01818-3_25.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;according to this study&lt;/a&gt;, between the predicted and target classes of your system, as it is right now, instead of the categorical cross-entropy error function.&lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-10-04T15:34:26.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23542" PostTypeId="2" ParentId="23534" CreationDate="2017-10-04T15:46:59.893" Score="0" Body="&lt;p&gt;I think the answer also depends on your usecase. If you just want to detect these kind of strings I would focus on heuristic rules that suit my needs, rather than creating a system that learns to recognize the patterns of the strings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if your aim is generating similar kind of strings based on the current patterns or finding new strings in a stream of text, you should look for regex generators. There is this &lt;a href=&quot;https://github.com/aloisdg/awesome-regex&quot; rel=&quot;nofollow noreferrer&quot;&gt;list of resources&lt;/a&gt; regarding regexes in general and you should focus on the generators for the task at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a similar task in the past, I had used an &lt;a href=&quot;http://regex.inginf.units.it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;online, free tool&lt;/a&gt; that will generate a regex (if possible) that satisfies as many of your sample strings as possible. So you could give it a try, as a fast solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, I would  first do some data exploration(e.g. are numbers and letters also intermixed or do numbers always come up after or letters/symbols etc.), in order to get a better understanding for the problem at hand. Also, this could lead to simple heuristic rules that could suffice as a crude solution or at least a simple baseline system against which I would &quot;pitch&quot; a machine learning model. &lt;/p&gt;&#xA;" OwnerUserId="8370" LastActivityDate="2017-10-04T15:46:59.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23543" PostTypeId="2" ParentId="23486" CreationDate="2017-10-04T17:09:50.453" Score="8" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes it is correct. You can show it is with some different arguments:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Checking correct dimensions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Although this is not a robust theoretical argument, checking dimensions is actually what I do in practice when I am confused when implementing NNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the rough logic of a dimension check on the equation in your question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$W^{[2]}$ is a $n^{[2]} \times n^{[1]}$ matrix for the forward propagation to work.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$dz^{[2]}$ is a $n^{[2]} \times 1$ column vector (because it is the gradient of cost function with respect to $z^{[2]}$, so it has to have the same dimension as it)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$dz^{[1]}$ is a $n^{[1]} \times 1$ column vector&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is $dz^{[2]}$ that is being multiplied - so the multiplying matrix must be $\text{something} \times n^{[2]}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The output needs to be same dimensions as $dz^{[1]}$ - so the matrix must also be $n^{[1]} \times \text{something}$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;That means the multiplying matrix must have dimensions $n^{[1]} \times n^{[2]}$ - which $W^{[2]T}$ has (and no other component of the network could match)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Theory from item-by-item calculation, converted to matrix math&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;But &lt;em&gt;why&lt;/em&gt; is it $W^{[2]T}$ and not some other matrix with the desired dimensions? For that we need to go back to the chain rule. Note that the expression you have written is actually &lt;em&gt;two&lt;/em&gt; steps of the chain rule combined - first we derive $da^{[1]}$ from $dz^{[2]}$, then we derive $dz^{[1]}$ from $da^{[1]}$. It is that first step that introduces terms from $W^{[2]}$, so I will just show that. Also, &lt;a href=&quot;https://datascience.stackexchange.com/questions/18279/deriving-backpropagation-equations-natively-in-tensor-form&quot;&gt;deriving this directly using matrix notation is more complex&lt;/a&gt;, so we'll just drop into using the index values, and use the &quot;official&quot; partial derivative notation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate $da^{[1]}$ for a specific $i^{th}$ neuron, you have to sum over the gradients $dz^{[2]}$ for all the neurons it links to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \frac{\partial J}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} \frac{\partial z_j^{[2]}}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} W_{ij}^{[2]} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing this for each $i$ value in turn gets your the full $da^{[1]}$ vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compare this to how the forward propagation works in the same network using the same notation and indexing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z^{[2]}_j = b^{[2]}_j + \sum_i W_{ij}^{[2]} a_i^{[1]}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And you can see that the index used to drive the sum is different between forward and backward calculations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When we write $W^{[2]}_{ij}$, then $i$ is index of matrix row, $j$ is index of matrix column from $W^{[2]}$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When multiplying forward, we sum over $i$, i.e. each row of $W^{[2]}$ multiplied by each matching element of $a^{[1]}$, and this is summed up. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When you look at the gradient, you can see the equivalent sum is over $j$. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To turn that gradient calculation into a normal matrix multiply (just so the matrix notation works, and you don't have to write out the sum) you have to swap rows and columns - which is what a transpose is.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is partly a definitions and notation issue. We could define a different kind of matrix multiply that worked with sums column-by-column. But this is not usually done, because multiplying by the transpose does the same thing.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-06T09:44:59.807" LastActivityDate="2017-10-06T09:44:59.807" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="23545" PostTypeId="2" ParentId="23534" CreationDate="2017-10-04T17:44:10.580" Score="2" Body="&lt;p&gt;Assuming that letters are indicative of &quot;motifs&quot; and numbers are considered as digits and not exact numbers, this is what I would do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First - transform numbers into a digit placeholder (&lt;code&gt;#&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;s = re.sub(&quot;\d&quot;,s,&quot;#&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I would transform a string into a bag-of-bigrams vector in the char level&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;vecs = CountVectorizer(s, analyzer=&quot;char_wb&quot;, ngram_range=(1,3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After these 2 steps, we got a sparse vector from any string:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;'ab123' --&amp;gt; {&quot; a&quot;:1,&quot;ab&quot;:1,&quot;b#&quot;,&quot;##&quot;:2,&quot;# &quot;:1}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next we want to convert those vectors to a pairwise distance matrix and cluster by that distance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import pairwise_distances&#xA;from scipy.cluster import  hierarchy&#xA;def link_cluster(X, threshold=0.1, metric=&quot;cosine&quot;, algo=&quot;average&quot;):&#xA;    X = X.todense()&#xA;    Z = hierarchy.linkage(X, algo, metric=metric)&#xA;    C = hierarchy.fcluster(Z,threshold, criterion=&quot;distance&quot;)&#xA;    return C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your end result would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C = link_cluster(vecs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12942" LastEditorUserId="12942" LastEditDate="2017-10-04T18:15:27.823" LastActivityDate="2017-10-04T18:15:27.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23547" PostTypeId="1" CreationDate="2017-10-04T18:58:39.620" Score="1" ViewCount="180" Body="&lt;p&gt;I am working on a market mix model ad was trying to apply diminishing returns to GRP values for my media activities. I was confused on how to go about it ,i.e how to decide the parameters for the for the diminishing fn ex negative exponential function y = α⋅(1 − e−β⋅x) ; β &gt; 0 has 2 parameters alpha &amp;amp; Beta, there are any number of combinations , what is the approach to finding a reasonable estimate for these values then.&lt;/p&gt;&#xA;" OwnerUserId="40039" LastActivityDate="2017-10-04T18:58:39.620" Title="How to determine parameters for diminishing return function in a market mix model" Tags="|regression|marketing|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23548" PostTypeId="2" ParentId="23538" CreationDate="2017-10-04T19:08:08.613" Score="1" Body="&lt;p&gt;NN aren't ideal for regression tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd train 10 networks, one for each magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN #k would be a binary classifier predicting whether the magnitude is &lt;strong&gt;k or less&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, CNN #10 is a function that always returns 1, and CNN#0 is the binary classifier you've already trained.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-04T19:08:08.613" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23549" PostTypeId="1" AcceptedAnswerId="26251" CreationDate="2017-10-04T19:19:06.740" Score="6" ViewCount="3539" Body="&lt;p&gt;I've noticed in a few caffe models I've been working with that the learning rate for the bias is often set to be twice that of the one for the weights.&lt;/p&gt;&#xA;&lt;p&gt;Another user mentions that this is the case in &lt;a href=&quot;https://discuss.pytorch.org/t/how-to-set-different-learning-rates-multitude-for-weight-and-bias/4631&quot; rel=&quot;nofollow noreferrer&quot;&gt;this thread&lt;/a&gt; but does not elaborate on what the reason for doing so is.&lt;/p&gt;&#xA;&lt;p&gt;Is there some justification for having different learning rates for the bias as the weights? Did I miss some paper providing a theoretical or experimental justification of this?&lt;/p&gt;&#xA;" OwnerUserId="40042" LastEditorUserId="29169" LastEditDate="2021-02-15T22:22:31.153" LastActivityDate="2021-02-15T22:22:31.153" Title="Why is the learning rate for the bias usually twice as large as the the LR for the weights?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="23551" PostTypeId="2" ParentId="23475" CreationDate="2017-10-04T19:30:40.077" Score="1" Body="&lt;p&gt;You could go for a 3D plot, but they are only useful in specific situations (e.g. simple bar plot, a surface etc.), and definitely not here. Your best option, while keeping the overall design of your plot, seems to be to split your data by the factor with 2 levels, and plot each separately in 2 panels side by side. That way, you can compare similarities, baseline shifts, and interactions.&lt;/p&gt;&#xA;" OwnerUserId="30542" LastActivityDate="2017-10-04T19:30:40.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23553" PostTypeId="1" AcceptedAnswerId="23561" CreationDate="2017-10-04T21:01:14.180" Score="1" ViewCount="2085" Body="&lt;p&gt;I have a homework question that requires me to create a 3 level tree (root, intermediate and leaf) with majority vote and expand the tree if 3 level tree is not possible. I have to use ID3 algorithm. Its a manual question rather than coding assignment and dataset contains 16, 4 features and a class label with T, F as their values. &#xA;I started calculating entropies and information gain for different nodes. When I complete the 3 level tree, I can assign the classes at the leaf with 50%, 100%, 100% and 75% majority vote or confidence. My questions are  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;is 50% really a majority vote?&lt;/li&gt;&#xA;&lt;li&gt;Do I need to expand on that part of the tree that has 50% confidence&lt;/li&gt;&#xA;&lt;li&gt;or I can just assign the classes arbitrarily?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Regards &lt;/p&gt;&#xA;" OwnerUserId="40045" LastActivityDate="2017-10-05T09:14:54.817" Title="Decision Tree Classifier with Majority vote (Is 50% a majority vote)" Tags="|classification|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23554" PostTypeId="1" AcceptedAnswerId="23582" CreationDate="2017-10-05T06:13:47.057" Score="0" ViewCount="75" Body="&lt;p&gt;I use R to do data analysis. I have a dataset. When I use different classifying algorithms, such as random forest, SVM, etc, I have the different accuracy. So, I want to integrate all the algorithms into one framework, let's say adaboost.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that adaboost framework use multiple &quot;weak&quot; classifying algorithms to combine a strong classifier. So, can I customize the &quot;weak&quot; classifying algorithms as I want? Here is just my current idea: In this framework, I use SVM first. Then give the data that are classified incorrectly more weights. Then, I use random forest. ... In the end, all the classifiers in this framework will work together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just what I think about this issue. If there is other method working such as voting, please let me know too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated.  &lt;/p&gt;&#xA;" OwnerUserId="16911" LastEditorUserId="16911" LastEditDate="2017-10-05T22:33:14.410" LastActivityDate="2017-10-06T11:38:34.370" Title="In R, can I integrate different classifying algorithms in one bagging model?" Tags="|r|random-forest|svm|classifier|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23557" PostTypeId="2" ParentId="6107" CreationDate="2017-10-05T07:58:10.757" Score="12" Body="&lt;p&gt;We could use PCA for analogy.&lt;/p&gt;&#xA;&lt;p&gt;When using conv, the forward pass is to extract the coefficients of principle components from the input image, and the backward pass (that updates the input) is to use (the gradient of) the coefficients to reconstruct a new input image, so that the new input image has PC coefficients that better match the desired coefficients.&lt;/p&gt;&#xA;&lt;p&gt;When using deconv, the forward pass and the backward pass are reversed. The forward pass tries to reconstruct an image from PC coefficients, and the backward pass updates the PC coefficients given (the gradient of) the image.&lt;/p&gt;&#xA;&lt;p&gt;The deconv forward pass does exactly the conv gradient computation given in &lt;a href=&quot;http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;That's why in the caffe implementation of deconv (refer to Andrei Pokrovsky's answer), the deconv forward pass calls &lt;code&gt;backward_cpu_gemm()&lt;/code&gt;, and the backward pass calls &lt;code&gt;forward_cpu_gemm()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16647" LastEditorUserId="98307" LastEditDate="2020-08-05T07:30:46.170" LastActivityDate="2020-08-05T07:30:46.170" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23558" PostTypeId="2" ParentId="23504" CreationDate="2017-10-05T08:33:10.010" Score="0" Body="&lt;p&gt;I'm no expert. But not only Instagram, other social networks has an api that allows you to obtain users activity. I would suggest going with collecting the activity data for a set of actual users and some bots(which as of now you have to find it yourself). This would be enough to make a dataset. Then, proceed with an algorithm of your choice. Hope, this helps.&lt;/p&gt;&#xA;" OwnerUserId="4868" LastActivityDate="2017-10-05T08:33:10.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23560" PostTypeId="1" CreationDate="2017-10-05T09:13:47.037" Score="1" ViewCount="85" Body="&lt;p&gt;I have a dataset which contains information about objects from 20 different categories. These objects are manually labeled on 100 images. The objects are located in 3 rows on a square image, there can be multiple instances of the same class object. For a given object, I have the following data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;its class&lt;/li&gt;&#xA;&lt;li&gt;bounding box center coordinates&lt;/li&gt;&#xA;&lt;li&gt;bounding box size&lt;/li&gt;&#xA;&lt;li&gt;information whether the object is in the correct position or not&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The ultimate goal is to have a classifier, which determines if the position of the detected object is correct. I am not sure if I should treat a label or &quot;correctness&quot; as an output, what would you suggest? Also, what are the best candidates for classifiers in case of this data, given the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each training image (with labelled objects) is warped by a small angle, perspective, translations etc.&lt;/li&gt;&#xA;&lt;li&gt;There is one correct arrangement of objects on the image&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please look below for simplified illustration of the data:&#xA;number - label, &#xA;bold - incorrect position&lt;/p&gt;&#xA;&#xA;&lt;p&gt;===================&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 1 1   3 3  &lt;strong&gt;5&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6  &lt;strong&gt;7&lt;/strong&gt;   1 1  4 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;9 5 5 5 9 7 7 7&lt;/p&gt;&#xA;&#xA;&lt;p&gt;===================&lt;/p&gt;&#xA;" OwnerUserId="40063" LastActivityDate="2018-05-02T18:30:49.023" Title="What classifier is the best to determine if object was detected in the correct position?" Tags="|machine-learning|classification|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23561" PostTypeId="2" ParentId="23553" CreationDate="2017-10-05T09:14:54.817" Score="2" Body="&lt;p&gt;The goal of ID3 is to get the purest nodes possible ( ironically that is what contributes to its problem of overfitting), so 50% is not pure at all, the data under that node is equally likely to be in one of the classes which makes peedicition tricky, it would be better to grow the tree further and find nodes which are more pure than atleast 50%. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-10-05T09:14:54.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23563" PostTypeId="2" ParentId="21982" CreationDate="2017-10-05T12:15:14.797" Score="1" Body="&lt;p&gt;The data you are having is panel data which is a combination of both cross sectional data and Time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;You can try with regression models by giving time stamp to your data&lt;/strong&gt; .Like maintaining one feature based your weekday (1 to 7).or if you have trends and seasonality in your data you can go to giving week number as feature like (0 to 53) weeks.&lt;/p&gt;&#xA;" OwnerUserId="38145" LastActivityDate="2017-10-05T12:15:14.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23564" PostTypeId="2" ParentId="23539" CreationDate="2017-10-05T12:55:50.127" Score="9" Body="&lt;p&gt;They effectively try to achieve the same result but the methodology used by each technique varies a little. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;RFE removes least significant features over iterations. So basically it first removes a few features which are not important and then fits and removes again and fits. It repeats this iteration until it reaches a suitable number of features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SelectFromModel is a little less robust as it just removes less important features based on a threshold given as a parameter. There is no iteration involved. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-05T12:55:50.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23565" PostTypeId="1" CreationDate="2017-10-05T13:47:24.393" Score="1" ViewCount="75" Body="&lt;p&gt;I recently proved that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbb{E}_I[\nabla L_I(x,y,w)] = \nabla(L_I(x,y,w))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $I$ represents a randomized batch of size $m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I cannot really understand why this is good news. I know that gradient is important for optimizing a model, especially with gradient descent. I also understand that in mini-batch algorithms we average over a whole batch instead of doing it for all the data, saving a lot of computations and reducing the variance.&lt;/p&gt;&#xA;" OwnerUserId="40074" LastActivityDate="2017-10-05T15:14:17.970" Title="Why is it good news that the expected gradient of the loss for a mini batch is equal to the gradient for the whole set?" Tags="|machine-learning|gradient-descent|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23568" PostTypeId="2" ParentId="23565" CreationDate="2017-10-05T15:14:17.970" Score="2" Body="&lt;p&gt;It is usually good news when an approximating function you use to take an action to optimise something is unbiased compared to the thing being approximated. It means your actions &lt;em&gt;in aggregate&lt;/em&gt; will head towards the same optimisation point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the gradient was biased in the long term, then (depending on nature of the bias) an optimisation routine will converge incorrectly by following them in gradient descent, and give an incorrectly optimised answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is sometimes OK if a value is initially biased, but the bias reduces over time towards zero. Not that this is the case here, but you will see that in some machine learning algorithms. For instance, in a neural network using momentum, typically the initial momentum values are biased, but the bias decays exponentially (it is actually possible to make an unbiased momentum estimate, but most implementations of SGD with momentum or Nesterov momentum do not do that - Adam implementations typically do).&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-05T15:14:17.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23571" PostTypeId="1" AcceptedAnswerId="23573" CreationDate="2017-10-05T17:01:30.420" Score="2" ViewCount="80" Body="&lt;p&gt;Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) current point is over threshold - 1/0&#xA;2) current point is AM - 1/0&#xA;3) current day is weekday - 1/0&#xA;4) current percentage - 0.0-1.0&#xA;5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Label:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 if a point is over threshold at any point in time after now and before now+15 minutes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Features 5-9 are intended to capture the inertia of the current percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?&lt;/p&gt;&#xA;" OwnerUserId="37776" LastEditorUserId="37776" LastEditDate="2017-10-05T18:55:32.870" LastActivityDate="2017-10-05T18:55:32.870" Title="Should I relabel this data or remove the potentially leaky feature?" Tags="|machine-learning|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23573" PostTypeId="2" ParentId="23571" CreationDate="2017-10-05T17:23:08.583" Score="1" Body="&lt;p&gt;Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Actual / Predicted   Alarm goes off      Alarm did not go off&#xA;State changed        True positive       False positive&#xA;State stayed         False negative      True negative&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note the true negatives are useless - &lt;em&gt;we don't care&lt;/em&gt; if the state stays in the same spot (below/above threshold). Therefore, the accuracy is &lt;strong&gt;NOT&lt;/strong&gt; important for this system. Do not try to minimize the wrong metric. This extends to &quot;do not use ROC/AUC&quot; (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also would like to deal with a huge &quot;lag&quot;, trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-10-05T17:23:08.583" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23576" PostTypeId="2" ParentId="8460" CreationDate="2017-10-05T23:31:15.753" Score="8" Body="&lt;p&gt;As an update on this question, I believe the accepted answer is not the best as of 2017.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested in comments by Kyle, &lt;code&gt;hmmlearn&lt;/code&gt; is currently the library to go with for HMMs in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Several reasons for this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;a href=&quot;https://hmmlearn.readthedocs.io/en/latest/tutorial.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;up-to-date documentation&lt;/a&gt;, that is very detailed and includes tutorial &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The &lt;code&gt;_BaseHMM&lt;/code&gt; class from which custom subclass can inherit for implementing HMM variants&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compatible with the last versions of Python 3.5+&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Intuitive use&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Opposite to this, the &lt;code&gt;ghmm&lt;/code&gt; &lt;a href=&quot;http://ghmm.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;library&lt;/a&gt; does not support Python 3.x according to the current documentation. Most of the documentation pages have been generated in 2006. It does not seem at first glance a library of choice...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Still valid in 2018.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastEditorUserId="25157" LastEditDate="2018-09-21T13:44:38.710" LastActivityDate="2018-09-21T13:44:38.710" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23577" PostTypeId="1" CreationDate="2017-10-06T00:29:06.227" Score="1" ViewCount="791" Body="&lt;p&gt;I am wondering how to handle correlated fields in a dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people suggest to drop this correlated fields because they don't provide any inside, other claim that a lot of information is getting lost. I am not sure if I should keep this correlated fields or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to handle correlated data. Drop it, keep it, or does it depend on additional parameters? If yes, which? Are there rules I can follow?&lt;/li&gt;&#xA;&lt;li&gt;What if the fields are sparse? (because the dataset is incomplete)&lt;/li&gt;&#xA;&lt;li&gt;Does 1) depend on what kind of model is used? Regression, classification, NN?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a concrete example, the correlated values from the &lt;a href=&quot;https://www.kaggle.com/c/zillow-prize-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zillow competition&lt;/a&gt; on Kaggle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NOAg1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NOAg1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qEfaw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qEfaw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40093" LastActivityDate="2017-10-06T18:08:18.303" Title="How to handle correlated variables in a dataset?" Tags="|regression|linear-regression|correlation|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-06T20:50:12.837" ContentLicense="CC BY-SA 3.0" />
  <row Id="23578" PostTypeId="2" ParentId="22387" CreationDate="2017-10-06T06:50:17.527" Score="8" Body="&lt;p&gt;In sort of mechanistic/pictorial/image-based terms:&lt;/p&gt;&#xA;&lt;h3&gt;&lt;strong&gt;Dilation:&lt;/strong&gt; ### &lt;em&gt;&lt;strong&gt;SEE COMMENTS, WORKING ON CORRECTING THIS SECTION&lt;/strong&gt;&lt;/em&gt;&lt;/h3&gt;&#xA;&lt;p&gt;Dilation is largely the same as run-of-the-mill convolution (frankly so is deconvolution), except that it introduces gaps into it's kernels, i.e. whereas a standard kernel would typically slide over contiguous sections of the input, it's dilated counterpart may, for instance, &amp;quot;encircle&amp;quot; a larger section of the image --while still only have as many weights/inputs as the standard form.&#xA;&lt;br&gt;&lt;br&gt;&#xA;(Note well, whereas dilation injects zeros into it's &lt;em&gt;kernel&lt;/em&gt; in order to more quickly &lt;em&gt;decrease&lt;/em&gt; the facial dimensions/resolution of it's output, transpose convolution injects zeroes into it's &lt;em&gt;input&lt;/em&gt; in order to &lt;em&gt;increase&lt;/em&gt; the resolution of it's output.)&lt;/p&gt;&#xA;&lt;p&gt;To make this more concrete, let's take a very simple example: &lt;br&gt; Say you have a 9x9 image, &lt;strong&gt;x&lt;/strong&gt; with no padding. If you take a standard 3x3 kernel, with stride 2, the first subset of concern from the input will be &lt;strong&gt;x&lt;/strong&gt;[0:2, 0:2], and all nine points within these bounds will be considered by the kernel. You would then sweep over &lt;strong&gt;x&lt;/strong&gt;[0:2, 2:4] and so on.&lt;/p&gt;&#xA;&lt;p&gt;Clearly, the output will have smaller facial dimensions, specifically 4x4. Thus, the next layer's neurons have receptive fields in the exact size of these kernels passes. But if you need or desire neurons with more global spatial knowledge (e.g if an important feature is only definable in regions larger than this) then you will need to convolve this layer a second time to create a third layer in which the effective receptive field is some union of the previous layers r.f..&lt;/p&gt;&#xA;&lt;p&gt;But if you don't want to add more layers and/or you feel that the information being passed on is overly redundant (i.e. your 3x3 receptive fields in the second layer only actually carry &amp;quot;2x2&amp;quot; amount of distinct information), you can use a dilated filter. Let's be extreme about this for clarity and say we'll use a 9x9 3-dialated filter. Now, our filter will &amp;quot;encircle&amp;quot; the entire input, so we won't have to slide it at all. We will still however, only be taking 3x3=9 data points from the input, &lt;strong&gt;x&lt;/strong&gt;, typically:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;x&lt;/strong&gt;[0,0] U &lt;strong&gt;x&lt;/strong&gt;[0,4] U &lt;strong&gt;x&lt;/strong&gt;[0,8] U &lt;strong&gt;x&lt;/strong&gt;[4,0] U &lt;strong&gt;x&lt;/strong&gt;[4,4] U &lt;strong&gt;x&lt;/strong&gt;[4,8] U &lt;strong&gt;x&lt;/strong&gt;[8,0] U &lt;strong&gt;x&lt;/strong&gt;[8,4] U &lt;strong&gt;x&lt;/strong&gt;[8,8]&lt;/p&gt;&#xA;&lt;p&gt;Now, the neuron in our next layer (we'll only have one) will have data &amp;quot;representing&amp;quot; a much larger portion of our image, and again, if the image's data is highly redundant for adjacent data, we may well have preserved the same information and learned an equivalent transformation, but with fewer layers and fewer parameters. I think within the confines of this description it's clear that while definable as resampling, we are here &lt;em&gt;downsampling&lt;/em&gt; for each kernel.&lt;/p&gt;&#xA;&lt;br/&gt;&#xA;&lt;h3&gt;&lt;strong&gt;Fractionally-strided or transpose or &amp;quot;deconvolution&amp;quot;:&lt;/strong&gt;&lt;/h3&gt;&#xA;&lt;p&gt;This sort is very much still convolution at heart. The difference is, again, that we will be moving from smaller input volume to a larger output volume. OP posed no questions about what upsampling is, so I'll save a bit of breadth, this time 'round and go straight to the relevant example.&lt;/p&gt;&#xA;&lt;p&gt;In our 9x9 case from before, say we want to now upsample to 11x11. In this case, we have two common options: we can take a 3x3 kernel and with stride 1 and sweep it over our 3x3 input with 2-padding so that our first pass will be over the region [left-pad-2:1, above-pad-2: 1] then [left-pad-1:2, above-pad-2: 1] and so on and so forth.&lt;/p&gt;&#xA;&lt;p&gt;Alternatively, we can additionally insert padding &lt;em&gt;in between&lt;/em&gt; the input data, and sweep the kernel over it without as much padding. Clearly we will sometimes be concerning ourselves with the &lt;em&gt;exact same&lt;/em&gt; input points more than once for a single kernel; this is where the term &amp;quot;fractionally-strided&amp;quot; seems more well-reasoned. I think the following animation (borrowed from &lt;a href=&quot;https://github.com/vdumoulin/conv_arithmetic&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and based (I believe) off of &lt;a href=&quot;https://arxiv.org/pdf/1603.07285v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;this work&lt;/a&gt; will help to clear things up despite being of different dimensions. The input is blue, the white injected zeros and padding, and the output green:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://github.com/vdumoulin/conv_arithmetic/raw/master/gif/padding_strides_transposed.gif&quot; alt=&quot;transposed conv, input is blue, output green&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;Of course, we're concerning ourselves with all of the input data as opposed to dilation which may or may not ignore some regions entirely. And since we're clearly winding up with more data than we began, &amp;quot;upsampling&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;I encourage you to read the excellent document I linked to for a more sound, abstract definition and explanation of transpose convolution, as well as for learning why the examples shared are illustrative but largely inappropriate forms for actually computing the represented transformation.&lt;/p&gt;&#xA;" OwnerUserId="27038" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-07T04:27:31.377" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="23580" PostTypeId="2" ParentId="22303" CreationDate="2017-10-06T07:11:55.347" Score="4" Body="&lt;p&gt;In general, a network with CNN with no Fully connected layers is termed as Fully Convolutional Network(FCN). It can include any type of pooling layers, batch norm, dropouts etc.&lt;/p&gt;&#xA;" OwnerUserId="40058" LastActivityDate="2017-10-06T07:11:55.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23582" PostTypeId="2" ParentId="23554" CreationDate="2017-10-06T11:38:34.370" Score="1" Body="&lt;p&gt;What you're looking for is called an ensemble model which means it is a compilation of several models to improve the results. This is a very common technique for winners in Kaggle competitions. Since you're using R and caret is a popular way to do ML in R, here's a package just for that purpose on caret:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2017-10-06T11:38:34.370" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23585" PostTypeId="2" ParentId="23509" CreationDate="2017-10-06T16:34:28.327" Score="1" Body="&lt;p&gt;I've decided to try LDA:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cvectorizer = CountVectorizer(min_df=4, max_features=10000, ngram_range(1,2))&#xA;cvz = cvectorizer.fit_transform(df['word'])&#xA;n_topics = 32&#xA;n_iter = 2000&#xA;lda_model = lda.LDA(n_topics=n_topics, n_iter=n_iter)&#xA;X_topics = lda_model.fit_transform(cvz)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I've concatenated time feature to the LDA vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; X_all  = np.hstack((X_topics, X_time))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;strong&gt;X_time&lt;/strong&gt; was my time feature column.&lt;/p&gt;&#xA;" OwnerUserId="40002" LastActivityDate="2017-10-06T16:34:28.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23586" PostTypeId="1" AcceptedAnswerId="23622" CreationDate="2017-10-06T16:35:57.127" Score="5" ViewCount="4346" Body="&lt;p&gt;My problem context:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;dataset too big to fit into memory.&lt;/li&gt;&#xA;&lt;li&gt;binary classification [0,1]&lt;/li&gt;&#xA;&lt;li&gt;30 csv files in a directory with exactly 30,000 samples (rows)&lt;/li&gt;&#xA;&lt;li&gt;each file contains 15,000 &lt;code&gt;0&lt;/code&gt; class and 15,000 &lt;code&gt;1&lt;/code&gt; class (no unbalance)&lt;/li&gt;&#xA;&lt;li&gt;model is xgboost.XGBClassifier()&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;If I iterate through these 30 files, fit a model, and pickle the model to disk; is it possible to aggregate these 30 &lt;strong&gt;fitted&lt;/strong&gt; models into a single model? I am aware of &lt;code&gt;sklearn.ensemble.VotingClassifier&lt;/code&gt; but that is used for:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.ensemble.VotingClassifier&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Soft Voting/Majority Rule classifier for &lt;strong&gt;unfitted&lt;/strong&gt; estimators.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="39547" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-09T15:15:53.203" Title="Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)" Tags="|python|classification|scikit-learn|ensemble-modeling|binary|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23587" PostTypeId="2" ParentId="23577" CreationDate="2017-10-06T18:08:18.303" Score="2" Body="&lt;p&gt;If your model has dependent variables, then it's hard to interpret the coefficients down the road.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say that your model has 3 variables: &lt;code&gt;profits&lt;/code&gt;, &lt;code&gt;costs&lt;/code&gt; and &lt;code&gt;revenue&lt;/code&gt;.&#xA;You trained your linear regression model to predict &lt;code&gt;revenue&lt;/code&gt; (silly, I know).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could get the following coefficients: $c_{cost}=-1,c_{profits}=1,c_{revenue}=0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone looking at these results, could conclude that &lt;code&gt;revenue&lt;/code&gt; is not predictive of &lt;code&gt;revenue&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is kind of a trivial example because the linear dependency is very clear (&lt;code&gt;revenue=profits-costs&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But consider it a motivation for not adding dependent variables to your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlation is a measurement of pairwise linear dependency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's more of a question of what type are your variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your variables are continuous, Dimensionality reduction algorithms like &lt;code&gt;PCA&lt;/code&gt; should account for the correlation.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-06T18:08:18.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23589" PostTypeId="2" ParentId="22037" CreationDate="2017-10-06T22:50:02.437" Score="1" Body="&lt;p&gt;A preliminary: there are three attributes of a function that are relevant here: continuous, monotonic, and differentiable. The RELU is continuous and monotonic nut not differentiable at z=0. The exponential relu or ELU is all three of those attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The differential or gradient gives you a direction. When the derivative of a function is undefined at a point, then the direction of the gradient is indeterminate at that point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When applying gradient descent, we wish to continuously modify parameters such that the loss function steadily decreases, which is the same as saying we wish to keep moving down towards minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the derivative of a loss function is undefined at some point, the gradient is indeterminate. This means the gradient descent could potentially move in the wrong direction. The magnitude of delay caused by this indeterminacy depends on the learning rate and other hyper-parameters. Regardless of the hyper-parameters, statistically, the undefined derivative in RELU at z=0, does contribute to slowing convergence of gradient descent.&lt;/p&gt;&#xA;" OwnerUserId="2515" LastActivityDate="2017-10-06T22:50:02.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23591" PostTypeId="1" CreationDate="2017-10-07T08:02:06.843" Score="7" ViewCount="1359" Body="&lt;p&gt;I have a dataset with 10,000 genes like below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;person gene1 gene2 ... gene10000  ethnic&#xA;  1       0      1         1       asian&#xA;  2       1      0         1       European&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row means, whether a person has a gene in their DNA or not. We are trying to classify different ethnic groups based on the data above. But first we want to use some clustering algorithms to visualize how the cluster would look like for different ethnic groups. We are not going to use this clustering algorithms to classify groups, it will be used just to visualize how it would look like, if we have nice separate clusters or overlapping clusters etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please recommend some clustering algorithms for this type of dataset. Also, the dimension is 10000. Is this going to be a problem for clustering? Should I use some dimensionality reduction algorithms first? If so please give your recommendations. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="40134" LastActivityDate="2019-08-05T16:50:46.430" Title="Clustering algorithms for high dimensional binary sparse data" Tags="|machine-learning|clustering|unsupervised-learning|" AnswerCount="6" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23593" PostTypeId="1" CreationDate="2017-10-07T18:04:50.497" Score="1" ViewCount="74" Body="&lt;p&gt;I know that feature selection helps in removal of irrelevant features. Do they also remove redundant features(features which convey the same info as as some other feature)? If not, which methods are used for removing these highy corelated variables?&lt;/p&gt;&#xA;" OwnerUserId="40159" LastActivityDate="2017-11-07T00:03:27.713" Title="Does feature selection removes highly corelated variables?" Tags="|feature-selection|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23594" PostTypeId="2" ParentId="23591" CreationDate="2017-10-07T19:10:14.150" Score="1" Body="&lt;p&gt;How many ethnic groups did you identify?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to visualize your problem, I'd determine the key influencers for each of the ethnic groups in a Naive Bayes like approach.&#xA;These genes (gene combinations?!) (including their values) may strongly correlate to some ethnic group, while not (or inverse) correlate to another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Place them on top of a pyramid graph. Place bars to the left and right for the correlation values.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-07T19:10:14.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23595" PostTypeId="1" AcceptedAnswerId="23598" CreationDate="2017-10-07T19:31:30.960" Score="0" ViewCount="275" Body="&lt;p&gt;I am writing a thesis and don't know if the charts below should have grid lines or not. Do they need the grid lines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MtSJd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MtSJd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/669Ug.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/669Ug.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;An example without the grid lines:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gqHmF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gqHmF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user40160" LastActivityDate="2017-10-08T02:36:35.183" Title="Charts with or without grids?" Tags="|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23596" PostTypeId="1" CreationDate="2017-10-07T19:47:39.340" Score="0" ViewCount="932" Body="&lt;p&gt;I am trying to visualize data using R and scatterplot3d.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have loaded data and used:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colors &amp;lt;- c(&quot;#999999&quot;, &quot;#E69F00&quot;, &quot;#56B4E9&quot; )&#xA;scatterplot3d(output&lt;span class=&quot;math-container&quot;&gt;$X2,output$&lt;/span&gt;X6 , output$X7 , color=colors, pch=&quot;X9&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;X9 is label column in my dataset. it contains 3 categories : A , B , C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By documentation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;color : colors of points in the plot, optional if x is an appropriate&#xA;  structure. Will be ignored if highlight.3d = TRUE.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;pch: plotting  &quot;character&quot;, i.e. symbol to use.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yet I still get this error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in scatterplot3d(output\$X2, output\$X6, output\$X7, color =&#xA;  colors,  :    length(color) must be equal length(x) or 1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assumed I had to put color for every collumn in dataset, but creating such array where:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;length( colors ) = length( num of columns )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields the same error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the right way to add colors?&lt;/p&gt;&#xA;" OwnerUserId="40162" LastEditorUserId="26562" LastEditDate="2018-09-27T04:40:06.263" LastActivityDate="2024-01-24T04:08:11.160" Title="Coloring labels using scatterplot3d in R" Tags="|r|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23597" PostTypeId="2" ParentId="23593" CreationDate="2017-10-07T23:09:39.760" Score="1" Body="&lt;p&gt;Lasso is a very common strategy as it is able to handle correlated variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/7935/what-are-disadvantages-of-using-the-lasso-for-variable-selection-for-regression&quot;&gt;https://stats.stackexchange.com/questions/7935/what-are-disadvantages-of-using-the-lasso-for-variable-selection-for-regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-10-07T23:09:39.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23598" PostTypeId="2" ParentId="23595" CreationDate="2017-10-08T02:36:35.183" Score="1" Body="&lt;p&gt;I would say that it is really up to you here. I don't think that the grid lines are distracting in any way, but can be helpful if you are trying to show more exact results. For example, it if it important that the scatter plot shows exact points then keep them. However, if you are demonstrating more of a general trend, then you could go without them. I'm not sure if these are your finished graphs or not, but make sure to label the color differences in the lines and provide a title. Also, is there a reason why your x-axis is on the top of the graph? If you want to keep it there and use the grid lines, try to have the numbers be above the graph instead of being slightly blocked by them. &lt;/p&gt;&#xA;" OwnerUserId="40166" LastActivityDate="2017-10-08T02:36:35.183" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23600" PostTypeId="1" AcceptedAnswerId="23601" CreationDate="2017-10-08T03:59:03.400" Score="1" ViewCount="153" Body="&lt;p&gt;This is the trained neural network for the XOR operator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://computing.dcu.ie/~humphrys/Notes/Neural/multi.neural.html#multi.xor&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/57ruH.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I find something like a trained network for recognizing hand writing digits somewhere on the internet? Is there an &quot;official format&quot; for trained neural networks?&lt;/p&gt;&#xA;" OwnerUserId="40167" LastEditorUserId="29575" LastEditDate="2017-10-08T04:30:00.387" LastActivityDate="2017-10-08T09:31:36.710" Title="Where can I find a trained neural network data to play with?" Tags="|neural-network|dataset|training|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23601" PostTypeId="2" ParentId="23600" CreationDate="2017-10-08T06:57:34.080" Score="1" Body="&lt;p&gt;What about the super popular and classical MINIST? Tensorflow has a nice tutorial:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/beginners&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/get_started/mnist/beginners&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Just follow the instructions!&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-10-08T06:57:34.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23603" PostTypeId="1" AcceptedAnswerId="24365" CreationDate="2017-10-08T07:30:55.210" Score="2" ViewCount="507" Body="&lt;p&gt;I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular observation. We replace this missing value by the class conditional mean $E({{x}_{n}}|y=0)$ and $E({{x}_{n}}|y=1)$. How do I justify this missing value treatment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is actually an exercise. But, something that I noticed is that if $p(x|y=0)$ and $p(x|y=1)$ have means ${{\mu }_{0}}$ and ${{\mu }_{1}}$, then how is their covariance matrix the same? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming their covariance matrix is same, the only advantage I see is that if I take class conditional mean as missing value then in multivariate Gaussian, the effect of this is that the $n^{th}$ row of $\left( x-\mu  \right)$ vector will be $0$ since $\left( {{x}_{n}}-{{\mu }_{n}} \right)$ will be $0$, thus making no impact in calculating $p(y=0|x)$ and $p(y=1|x)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this explanation correct?&lt;/p&gt;&#xA;" OwnerUserId="28317" LastEditorUserId="12384" LastEditDate="2017-11-04T21:28:07.157" LastActivityDate="2017-11-05T04:34:58.040" Title="Replacing missing value by class conditional mean" Tags="|machine-learning|missing-data|bayesian|gaussian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23606" PostTypeId="2" ParentId="23600" CreationDate="2017-10-08T09:31:36.710" Score="1" Body="&lt;p&gt;There are many pre-trained models but for more complicated task, I recommend to check out &lt;a href=&quot;https://github.com/caffe2/caffe2/wiki/Model-Zoo&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe Model Zoo&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="16487" LastActivityDate="2017-10-08T09:31:36.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23607" PostTypeId="1" CreationDate="2017-10-08T10:15:40.980" Score="-1" ViewCount="489" Body="&lt;p&gt;Suppose we have a dataframe df in python, with numerical and categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;Numerical&lt;/code&gt;,&#xA;     when do we replace by mean and when by median.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;Categorical&lt;/code&gt;,&#xA;    is there an approach which needs to be followed. &lt;/p&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2017-10-08T11:04:19.363" Title="What are the best way to handle missing values" Tags="|python|data-cleaning|preprocessing|missing-data|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-09T15:30:11.637" ContentLicense="CC BY-SA 3.0" />
  <row Id="23608" PostTypeId="1" CreationDate="2017-10-08T10:48:34.747" Score="0" ViewCount="941" Body="&lt;p&gt;I am relatively new to data science and have an exercise task. This consists of the classifications of excerpts of texts. However, the texts are obfuscated such that one cannot read the words, spaces etc. But the &quot;patterns&quot; are preserved. I have a training set of the following form. One .txt file with the text excerpts looking like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;shdbcjhbjhbefbhbwhbkbehbwbwbfhwb /&#xA;wbhbwtjnwkjbrfbqlenk /&#xA;wjnfkjebrkbrghkbibgibib /&#xA;tberbtewtwbkwtjbrkbwkbtwrbt /&#xA;.&#xA;.&#xA;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and one .txt file with the labels of the texts looking like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 /&#xA;4 /&#xA;11 /&#xA;0 /&#xA;.&#xA;.&#xA;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that each string of symbols belongs to a certain text and one knows to which one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task is now to set up a classifier, to train it on above training data and to test it on test data which also consists of text excerpts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My basic idea was to interpret the data as images and to set up a deep learning NN with e.g. TensorFlow. However, I am not sure about the shape of the data which is needed to feed the data to the NN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I have to create label-folders containing all texts with a certain class-label or is there a more direct way (so far I created vectors in R consisting of the texts and the labels)? Must the excerpts all have the same length? How do I have to manipulate the data until I can feed it to the NN?&lt;/p&gt;&#xA;" OwnerUserId="40178" LastEditorUserId="14372" LastEditDate="2017-10-09T13:53:54.407" LastActivityDate="2018-08-11T16:00:35.690" Title="Classification of obfuscated text data" Tags="|neural-network|deep-learning|text-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23609" PostTypeId="2" ParentId="23607" CreationDate="2017-10-08T11:04:19.363" Score="1" Body="&lt;p&gt;In my experience, there is no one correct answer to this question. It depends mostly on your problem itself. I'll base my answer on most general cases though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most Numerical data, a parametric method is the best way to go. If you have some sample data which you can use as a baseline. Try using a parametric method such as Normal distribution, Exponential Distribution or so depending on your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For categorical data, you can try various oversampling techniques or undersampling techniques. Methods such as SMOTE, ADASYN etc can be used to fill in some missing values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, these methods are just general cases. But as aforementioned it depends more on your model and the type of data you are working with. It could be that maybe you are using scientific data and you don't need to generate synthetic data so based on how much data you have available you can discard the missing rows. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All such nuances come into play before making the most optimal decision for your data model. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-08T11:04:19.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23611" PostTypeId="2" ParentId="20271" CreationDate="2017-10-08T13:38:20.517" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there publications which mention numerical problems in neural network optimization?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Of course, there has been a lot of research on &lt;a href=&quot;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;vanishing gradients&lt;/a&gt;, which is entirely a numerical problem. There is also a fair amount of research of &lt;a href=&quot;https://stackoverflow.com/questions/46613748/float16-vs-float32-for-convolutional-neural-networks&quot;&gt;training with low precision operations&lt;/a&gt;, but the result is surprising: reduced floating point precision &lt;em&gt;doesn't&lt;/em&gt; seem to affect neural network training. This means that precision loss is pretty unlikely to be the cause of this phenomenon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, the environment can affect the computation (as suggested in the comments):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Most obviously, random-number generator. Use a seed in your script and try to make a reproducible result at least on a single machine. After that you can compute the summary of activations and gradients (e.g. via &lt;a href=&quot;https://www.tensorflow.org/get_started/summaries_and_tensorboard&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.summary&lt;/code&gt;&lt;/a&gt; in tensorflow) and compare the tensors across the machines. Clearly, basic operations such as matrix multiplication or piece-wise exponent should give very close if not identical result, no matter what hardware is used. You should be able to see if the tensors diverge immediately (which means there is another source of randomness) or gradually.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;python&lt;/code&gt; interpreter, &lt;code&gt;cuda&lt;/code&gt;, &lt;code&gt;cudnn&lt;/code&gt; driver and key libraries versions (&lt;code&gt;numpy&lt;/code&gt;, &lt;code&gt;tensorflow&lt;/code&gt;, etc). You can go as far the same linux kernel and &lt;code&gt;libc&lt;/code&gt; version, but I think you should expect reproducibility even without it. &lt;code&gt;cudnn&lt;/code&gt; version is important, because the convolution is likely to be natively optimized. &lt;code&gt;tensorflow&lt;/code&gt; is also very important, because Google rewrites the core all the time.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Environment variables, e.g. &lt;code&gt;PATH&lt;/code&gt;, &lt;code&gt;LD_LIBRARY_PATH&lt;/code&gt;, etc., linux configuration parameters, e.g. &lt;a href=&quot;https://linux.die.net/man/5/limits.conf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;limits.conf&lt;/code&gt;&lt;/a&gt;, permissions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Extra precautions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Explicitly specify the type of each variable, don't rely on defaults.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Double check that the training / test data is identical and is read in the &lt;a href=&quot;https://stackoverflow.com/questions/8977441/does-readdir-guarantee-an-order&quot;&gt;same order&lt;/a&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does your computation use any pre-trained models? Any networking involved? Check that as well.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would suspect hardware differences the last: it's an extraordinary case if the high-level computation without explicit concurrency leads to different results (except floating point precision differencies) depending on the number of cores or cache size.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2017-10-08T13:38:20.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23613" PostTypeId="1" AcceptedAnswerId="23626" CreationDate="2017-10-08T14:11:55.760" Score="3" ViewCount="1085" Body="&lt;p&gt;I have seen &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=3&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwin9efZmeHWAhWRJVAKHfX9DgoQFgguMAI&amp;amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FProbably_approximately_correct_learning&amp;amp;usg=AOvVaw0-7Yenwwy4WysCK70mZL-5&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; but I really cannot realize that.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this framework, the learner receives samples and must select a generalization function (called the hypothesis) from a certain class of possible functions. The goal is that, with high probability (the &quot;probably&quot; part), the selected function will have low generalization error.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Actually we do that in every machine learning situations and we do latter part for avoiding over-fitting. Why do we call it PAC-learning?&lt;br /&gt;&#xA;I also have not get the meaning of the math too. Is there anyone that can help?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-10-10T11:40:28.813" Title="What is PAC learning?" Tags="|machine-learning|pac-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23614" PostTypeId="1" CreationDate="2017-10-08T15:09:04.187" Score="14" ViewCount="10303" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EOQM4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EOQM4.png&quot; alt=&quot;multiple softmax in last layer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to implement mutiple softmaxes in the last layer in Keras? So the sum of Nodes 1-4 = 1; 5-8 = 1; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I go for a different network design?&lt;/p&gt;&#xA;" OwnerUserId="40180" LastActivityDate="2017-10-11T07:35:04.763" Title="Keras Multiple “Softmax” in last layer possible?" Tags="|machine-learning|keras|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23615" PostTypeId="1" AcceptedAnswerId="23747" CreationDate="2017-10-08T15:57:30.383" Score="16" ViewCount="9818" Body="&lt;p&gt;I am trying to create a neural network using time series as input, in order to train it based on the type of each series.&#xA;I read that using RNNs you can split the input into batches and use every point of the time series into individual neurons and eventually train the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do though is use multiple time series as an input. So for example you might receive input from two sensors. (So two time series), but I want to use both of them in order to get a final result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I am not trying to predict future values of the time series, I am trying to a get a classification based on all of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I approach this problem?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a way to use multiple time series as an input to an RNN?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I try to aggregate the time series into one?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or should i just use two different neural networks? And if this last approach is correct, if the number of time series increases wouldn't that be too computer intensive?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="40182" LastEditorUserId="29575" LastEditDate="2017-10-08T16:11:42.767" LastActivityDate="2017-10-13T09:04:22.370" Title="RNN using multiple time series" Tags="|time-series|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23616" PostTypeId="1" AcceptedAnswerId="23656" CreationDate="2017-10-08T16:01:40.900" Score="1" ViewCount="3006" Body="&lt;p&gt;I want to fill missing values in my dataframe.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; In [1]: df = spark.createDataFrame([[1],[1],[2],[3],[3],[None],[3],[None],[3],[2],[None],[1],[4]], ['data'])     &#xA; In [2]: df.show()&#xA; +----+&#xA; |data|&#xA; +----+&#xA; |   1|&#xA; |   1|&#xA; |   2|&#xA; |   3|&#xA; |   3|&#xA; |null|&#xA; |   3|&#xA; |null|&#xA; |   3|&#xA; |   2|&#xA; |null|&#xA; |   1|&#xA; |   4|&#xA; +----+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know I can use pyspark.ml &lt;a href=&quot;https://spark.apache.org/docs/latest/ml-features.html#imputer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Imputer&lt;/a&gt; to fill with the mean / median, or use &lt;a href=&quot;https://stackoverflow.com/questions/36343482/fill-in-null-with-previously-known-good-value-with-pyspark&quot;&gt;this&lt;/a&gt; method to fill with the last valid value. These are fine options, but I would like to impute with a random sample from the data distribution. For example, in the data provided, nulls will be filled according to these probabilities:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; P(1) = .3&#xA; P(2) = .2&#xA; P(3) = .4&#xA; P(4) = .1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What would be the best way to fill these values from a random sample?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="37931" LastEditorUserId="29575" LastEditDate="2017-10-08T16:53:14.513" LastActivityDate="2018-12-11T17:34:23.557" Title="Filling missing values with pyspark using a probability distribution" Tags="|pyspark|missing-data|data-imputation|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23617" PostTypeId="2" ParentId="12830" CreationDate="2017-10-08T17:19:49.660" Score="6" Body="&lt;p&gt;A fully connected layer (for input size $n*n$ over with $i$ channels, and $m$ output neurons) IS NOT equivalent to a 1x1 convolution layer but rather to an $n$x$n$ convolution layer (i.e. a big kernel, same size as input- no pad) with number of filters equal to the FC output/hidden layer (i.e. $m$ filters)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you asked, it has the same number of parameters as the FCN, i.e. $n*n*i*m$ (plus bias):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FCN: $n*n*i$ (weights per input layer= input$*$channels) $* m$ (times output/hidden layer width)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN: $n*n$ (each kernel) $* i $ (kernel per input channel) $* m$ (number of filters)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://www.quora.com/Is-a-fully-connected-neural-network-conceptually-similar-to-a-1x1-convolutional-neural-network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="40184" LastEditorUserId="40184" LastEditDate="2017-10-09T10:04:59.600" LastActivityDate="2017-10-09T10:04:59.600" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23618" PostTypeId="1" AcceptedAnswerId="23621" CreationDate="2017-10-08T17:34:17.187" Score="-1" ViewCount="554" Body="&lt;p&gt;Owl is the numerical library for OCaml: &lt;a href=&quot;https://github.com/ryanrhymes/owl&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ryanrhymes/owl&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is supposed to be an equivalent of numpy and also have capabilities of tensorflow.&lt;br&gt;&#xA;Any insights on why it should be used or why it shouldn't?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-10-09T01:10:05.893" Title="What are the advantages or disadvantages of Owl?" Tags="|tensorflow|numerical|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23621" PostTypeId="2" ParentId="23618" CreationDate="2017-10-09T01:10:05.893" Score="0" Body="&lt;p&gt;In a nutshell it is promising but it lacks in multiple points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The research below explains why it is better/worse than Numpy and Tensorflow:&lt;br&gt;&#xA;&lt;a href=&quot;http://pligor.tumblr.com/post/166198475026/owl-an-ocaml-numerical-library-research-by&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pligor.tumblr.com/post/166198475026/owl-an-ocaml-numerical-library-research-by&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-10-09T01:10:05.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23622" PostTypeId="2" ParentId="23586" CreationDate="2017-10-09T04:23:02.663" Score="7" Body="&lt;p&gt;Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal was to make a strong&lt;strong&gt;er&lt;/strong&gt; predictive model from multiple trained models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quote from my question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;is it possible to aggregate these 30 fitted models into a single&#xA;  model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Answer: yes but there's no good functionality that allows you to do this in sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Verbose answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have 30 CSV files that contain 15,000 &lt;code&gt;0&lt;/code&gt; class and 15,000 &lt;code&gt;1&lt;/code&gt; class samples. In other words, an equally balanced number of binary responses (no class imbalance). I generated these files myself because 1) the size of the data I'm working with is too big to fit into memory (point #1 of my original question) and 2) contains 97% class &lt;code&gt;0&lt;/code&gt; and 3% class &lt;code&gt;1&lt;/code&gt; (large class imbalance). My goal is to see if it's even possible to distinguish between a &lt;code&gt;0&lt;/code&gt; or a &lt;code&gt;1&lt;/code&gt; if the class imbalance issue was removed from the equation. If there were distinguishing features found, I'd want to know what those features were.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To generate each batch, I grabbed 15,000 &lt;code&gt;1&lt;/code&gt; responses, and randomly sampled from the 97% 15,000 more samples, joined into one dataset (30,000 samples total), then shuffled them at random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mWPkV.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mWPkV.png&quot; alt=&quot;batch files&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then went through each batch and trained an XGBClassifier() using optimal parameters found from GridSearchCV() for each. I then saved the model to disk (using Python's pickle capabilities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, you have 30 saved models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;/models/&lt;/code&gt; directory looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['model_0.pkl', 'model_1.pkl', 'model_10.pkl', 'model_11.pkl', 'model_12.pkl',&#xA; 'model_13.pkl', 'model_14.pkl', 'model_15.pkl', 'model_16.pkl', 'model_17.pkl',&#xA; 'model_18.pkl', 'model_19.pkl', 'model_2.pkl', 'model_20.pkl', 'model_21.pkl',&#xA; 'model_22.pkl', 'model_23.pkl', 'model_24.pkl', 'model_25.pkl', 'model_26.pkl',&#xA; 'model_27.pkl', 'model_28.pkl', 'model_29.pkl', 'model_3.pkl', 'model_4.pkl',&#xA; 'model_5.pkl', 'model_6.pkl', 'model_7.pkl', 'model_8.pkl', 'model_9.pkl']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again, going back to the original question - is it possible to aggregate these into a single model? I wrote a pretty basic python function for this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def xgb_predictions(X):&#xA;  ''' returns predictions from 30 saved models '''&#xA;&#xA;  predictions = {}&#xA;  for pkl_file in os.listdir('./models/'):&#xA;    file_num = int(re.search(r'\d+', pkl_file).group())&#xA;    xgb = pickle.load(open(os.path.join('models', pkl_file), mode='rb'))&#xA;    y_pred = xgb.predict(X)&#xA;    predictions[file_num] = y_pred&#xA;&#xA;  new_df = pd.DataFrame(predictions)&#xA;  new_df = new_df[sorted(new_df.columns)]&#xA;&#xA;  return new_df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It iterates through each file in a directory of saved models and loads them one at a time casting its own &quot;vote&quot; so-to-speak. The end result is a new dataframe of &lt;strong&gt;predictions&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this new dataset is smaller (X.shape x 30), it will fit into memory. I iterate through each batch file calling &lt;code&gt;xgb_predictions(X)&lt;/code&gt;, getting a dataframe of predictions, add the &lt;code&gt;y&lt;/code&gt; column to the dataframe, and append it to a new file. I then read the full dataset of predictions and create a &quot;level 2&quot; model instance where &lt;code&gt;X&lt;/code&gt; is the prediction data and &lt;code&gt;y&lt;/code&gt; is still &lt;code&gt;y&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to recap, the concept is, for binary classification, create equally balanced class datasets, train a model on each, run through each dataset and let each trained model cast a prediction. This collection of predictions is, in a way, a transformed version of your original dataset. You build another model to try and predict &lt;code&gt;y&lt;/code&gt; given the predictions dataset. Train it all in memory so you end up with a single model. Any time you want to predict, you take your dataset, transform it into predictions, then use your level 2 trained model to cast the final prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This technique performed much better than any other single model did or even stacking with UNTRAINED prior models.&lt;/p&gt;&#xA;" OwnerUserId="39547" LastEditorUserId="39547" LastEditDate="2017-10-09T15:15:53.203" LastActivityDate="2017-10-09T15:15:53.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23623" PostTypeId="1" AcceptedAnswerId="23625" CreationDate="2017-10-09T04:33:20.637" Score="1" ViewCount="235" Body="&lt;p&gt;I am building a multi-class support vector machine (8 classes to be precise) on an image dataset of pre-defined classes. And then I thought of a question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What if I have an image that doesn't belong to the set of predefined classes, what would be the outcome?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So I decided to experiment with it and the result was very bad. I got a higher accuracy for images that don't belong to any of the classes. Some images gave 98% accuracy, that they belonged to a particular class, even though my expectation should be that they should should have a very low accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried using OnceClass SVM to first predict if it's part of the class or not. If yes, then what's the label? (Meaning I have 2 models).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this doesn't seem to work as the OneClass SVM couldn't classify the &quot;other&quot; images well. Now I am running out of ideas of how to go about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I approach this problem?&lt;/p&gt;&#xA;" OwnerUserId="40197" LastEditorUserId="29575" LastEditDate="2017-10-09T05:28:01.627" LastActivityDate="2022-03-30T09:07:37.193" Title="What is the best way to classify data not belonging to set of classes?" Tags="|machine-learning|predictive-modeling|svm|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23625" PostTypeId="2" ParentId="23623" CreationDate="2017-10-09T05:25:22.900" Score="1" Body="&lt;p&gt;Just a proposal of a method to try out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stage $1$: Use one-class SVM to assign those images that do not belong to the set of predefined classes as the $9$-th class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stage $2$: For those images that passes through your filter, let the multi-class SVM assign them to one of the $8$ classes.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2017-10-09T05:53:16.770" LastActivityDate="2017-10-09T05:53:16.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23626" PostTypeId="2" ParentId="23613" CreationDate="2017-10-09T05:32:15.540" Score="2" Body="&lt;p&gt;PAC stands for &lt;a href=&quot;https://en.wikipedia.org/wiki/Probably_approximately_correct_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Probably Approximately Correct&lt;/a&gt;.&#xA;It was a very common research area in computer science looking for proof of learnability of certain hypothesis sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual hypothesis sets were not distributions (like in statistics) but more logical formulas like &lt;a href=&quot;https://en.wikipedia.org/wiki/Disjunctive_normal_form&quot; rel=&quot;nofollow noreferrer&quot;&gt;DNF&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Conjunctive_normal_form&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNF&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Deterministic_finite_automaton&quot; rel=&quot;nofollow noreferrer&quot;&gt;DFA&lt;/a&gt;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to prove learnability, you need to show that for every sample distribution, for every concept $h$ in the hypothesis class $H$,  for every $\lambda$ (the probability part) and for every $\epsilon$ (the approximately part) you can find an hypothesis $h^*$ such that with probability at least $1- \lambda$ the disagreement between $h$ and $h^*$ will be smaller than $\epsilon$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This bar for learning is very high, and some of the research proved negative results about many simple hypothesis sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you try to learn to differentiate between cats and dogs, you don't have a mathematical definition of hypothesis set. Most publications run an algorithm on a dataset and show their result. Almost nobody proves that the algorithms are correct (even given some assumptions). I find the lack of proofs quite sad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The math behind PAC is actually probability. Let's say that you have two cubes (classifier, concept) and you want to know how correlated they are. So, you toss them together $m$ times and you check the level of disagreement, epsilon. The higher the $m$ and the lower the $\epsilon$, the more confident you are that this is due to correlation and not due to error, $\lambda$. Given this intuition, PAC learning usually works in the other direction. Once you state the desired lambda and epsilon, the algorithm provides the number of samples $m$ needed to reach them.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2017-10-10T11:40:28.813" LastActivityDate="2017-10-10T11:40:28.813" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23629" PostTypeId="1" CreationDate="2017-10-09T08:57:20.657" Score="2" ViewCount="524" Body="&lt;p&gt;It's my first question here and I hope it's not too simple for a data science question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one hatched and semi-transparent and the other one non-transparent fully-colored. &lt;strong&gt;I'd like to emphasize how the hatched histograms are widening while the fully-colored ones are not.&lt;/strong&gt; I also want to keep something that shows that the data in hatched and non-hatched correspond with each other as I did in this histogram with the colors - red full-colored histogram data corresponds to red hatched data. &#xA;&lt;strong&gt;Any ideas how to visualize it differently? Is the visualization presented here good enough to show the things I described here?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XMoTY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XMoTY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39764" LastActivityDate="2017-10-09T08:57:20.657" Title="Histogram alternatives for two sets of data combined" Tags="|visualization|data|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23630" PostTypeId="2" ParentId="13461" CreationDate="2017-10-09T09:21:49.847" Score="20" Body="&lt;p&gt;You can do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;q = model.predict( np.array( [single_x_test,] )  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40210" LastEditorUserId="29575" LastEditDate="2017-10-09T13:54:01.830" LastActivityDate="2017-10-09T13:54:01.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23634" PostTypeId="1" AcceptedAnswerId="23665" CreationDate="2017-10-09T11:02:16.100" Score="-2" ViewCount="74" Body="&lt;p&gt;I am really quite new to this whole world but I keep hearing lots of talk about ElasticSearch and Kibana.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been to their website but would just like a really simple, plain English explanation of how it can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions I have in mind are;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is it part of the Apache foundation?&lt;/li&gt;&#xA;&lt;li&gt;Where does the software sit? Would it form part of an Hadoop ecosystem?&lt;/li&gt;&#xA;&lt;li&gt;When would I use it?&lt;/li&gt;&#xA;&lt;li&gt;What skills do I need in order to use it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Bit of a general question i know but something which might be useful for other people too I feel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2017-10-10T10:42:52.560" Title="What is ElasticSearch and how can it be used?" Tags="|search|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-13T15:48:14.010" ContentLicense="CC BY-SA 3.0" />
  <row Id="23636" PostTypeId="2" ParentId="23614" CreationDate="2017-10-09T12:36:55.737" Score="5" Body="&lt;p&gt;It is possible just implement your own softmax function. You can split a tensor to parts, then compute softmax separately per part and concatenate tensor parts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def custom_softmax(t):&#xA;    sh = K.shape(t)&#xA;    partial_sm = []&#xA;    for i in range(sh[1] // 4):&#xA;        partial_sm.append(K.softmax(t[:, i*4:(i+1)*4]))&#xA;    return K.concatenate(partial_sm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;concatenate&lt;/code&gt; without axis argument concatenate through last axis (in our case axis=1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can include this activation function in a hidden layer or add it to a graph. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dense(activation=custom_activation)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Activation(custom_activation))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You also need to define a new cost function.&lt;/p&gt;&#xA;" OwnerUserId="32745" LastEditorUserId="32745" LastEditDate="2017-10-11T07:35:04.763" LastActivityDate="2017-10-11T07:35:04.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23637" PostTypeId="1" AcceptedAnswerId="25515" CreationDate="2017-10-09T13:42:54.810" Score="4" ViewCount="1785" Body="&lt;p&gt;I'm currently trying to train a &lt;code&gt;RandomForestClassifier&lt;/code&gt; on a dataset consisting of 5000 instances with 12 (now) encoded features and a binary target label. Through &lt;code&gt;GridSearchCV&lt;/code&gt; I found out, that &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;best_parameters = {&#xA;    'criterion':    'gini', &#xA;    'max_depth':    12, &#xA;    'max_features': 'log2', &#xA;    'n_estimators': 300&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;works best out of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hyperparameters = {&#xA;    &quot;n_estimators&quot;: [9,10,20,30,40,50,60,100,150,200,300,1000],&#xA;    &quot;max_depth&quot;:    [3,6,9,12,20],&#xA;    &quot;criterion&quot;:    [&quot;gini&quot;, &quot;entropy&quot;],&#xA;    &quot;max_features&quot;: [&quot;log2&quot;, &quot;auto&quot;]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And gives a &lt;code&gt;mean_test_score&lt;/code&gt; of &lt;code&gt;0.8546&lt;/code&gt; which is quite good already, as I think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I would like to get some kind of visual interpretation like a ROC curve for each parameter. But does it actually make sense in the case of a &lt;code&gt;RandomForestClassifier&lt;/code&gt; to create a ROC curve for each hyperparameter? Or are there other ways to tune my classifier?&lt;/p&gt;&#xA;" OwnerUserId="40224" LastEditorUserId="40224" LastEditDate="2017-10-09T13:58:41.513" LastActivityDate="2017-12-09T07:38:59.657" Title="ROC curve for different hyperparameters of `RandomForestClassifier`?" Tags="|classification|scikit-learn|random-forest|model-selection|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23638" PostTypeId="1" AcceptedAnswerId="23640" CreationDate="2017-10-09T14:20:54.000" Score="4" ViewCount="2247" Body="&lt;p&gt;I am new to Doc2Vec. As I understand Doc2Vec group similar documents based on the context of their words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of newspaper documents and I want to identify what are the main topics of the newspapers (group 'Politics' news documents to one group, 'Sports' news documents to another group etc.) based on their content. Thus, I am interested in knowing;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What variant of Doc2Vec is more suitable for this (dbow, dm)?&lt;/li&gt;&#xA;&lt;li&gt;What are the real world applications of Doc2Vec algorithm?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2017-10-10T06:59:28.957" Title="What are real world applications of Doc2Vec?" Tags="|machine-learning|data-mining|text-mining|word-embeddings|gensim|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23639" PostTypeId="1" CreationDate="2017-10-09T14:42:18.640" Score="-1" ViewCount="125" Body="&lt;p&gt;I have time series dataset with 7 categorical features (Names) and each of them has 3-5 numerical features (results of activities of each person) for the period 1996-2017 by months. &#xA;Is it possible to create a model that will be forecasting &quot;Y&quot; on the base of values for selected Names.&#xA;For example: John + Jack = Y? in 2017-02-01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of the dataset:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/5AfxS.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5AfxS.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40219" LastActivityDate="2017-10-09T16:07:06.837" Title="Time Series forecasting model" Tags="|machine-learning|python|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23640" PostTypeId="2" ParentId="23638" CreationDate="2017-10-09T15:44:30.077" Score="3" Body="&lt;p&gt;I will answer your second question first, doc2vec and word2vec both are primarily good representations of text data that capture the semantics of words and documents. So whenever you are working with text data, you need a representation for it and that is what word2vec and doc2vec provides. Now think of any real world task on text data, like document similarity, using doc2vec you can find cosine similarity between two documents easily, now think of real world applications for it, finding duplicate question on a site like stack overflow, ranking candidate answers for a question answering model, features for text cladsification like sentiment analysis ( word2vec doesnt work well here, the context of good and bad are quite similar so it struggles to differentiate between positive and negative reviews). So these are just representations, you can apply them to any NLP and a lot of IR task.&lt;br&gt;&#xA;To answer your first question, a model is not just task dependent but also data dependent. So you can read Mikolov's paper to find out how each model works for the baseline tasks but a good idea is to try both models on your data and extrinsically evaluate which algorithm performs better. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-10-09T15:44:30.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23641" PostTypeId="2" ParentId="23639" CreationDate="2017-10-09T16:07:06.837" Score="2" Body="&lt;p&gt;Yes, you certainly can. You have to convert all your categorical variables into one numbers. This means, names in your &lt;code&gt;Name&lt;/code&gt; and &lt;code&gt;Name2&lt;/code&gt; columns can be converted using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot encoder&lt;/a&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After converting them, you may consider some feature engineering techniques to such dimensionality reduction based on your scenario. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this, you can use whatever the model is in your mind to solve your problem. Also, please take a peak at this &lt;a href=&quot;https://stackoverflow.com/questions/34007308/linear-regression-analysis-with-string-categorical-features-variables&quot;&gt;post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="34028" LastActivityDate="2017-10-09T16:07:06.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23642" PostTypeId="1" AcceptedAnswerId="23643" CreationDate="2017-10-09T16:15:21.823" Score="-1" ViewCount="129" Body="&lt;p&gt;I was trying to understand the probabilities output by a logistic regression in credit scorecard, let us say that I have performed Vintage analysis and identified the performance period as 6 months , and Bad rate is defined as 90DPD , so when I perform logistic regression for incoming new customers , the probabilities would define the probability that a customer will go Bad in the next 6 months ( which comes from performance period) ? &lt;/p&gt;&#xA;" OwnerUserId="40039" LastActivityDate="2017-10-09T16:46:29.960" Title="Interpretation of probabilities from logistic regression in Credit Score Card modelling" Tags="|r|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23643" PostTypeId="2" ParentId="23642" CreationDate="2017-10-09T16:46:29.960" Score="0" Body="&lt;p&gt;This would hold if you have the same priors in train and test population. However this is not the case for most of credit scoring cases. In the train data, you have customers that are accepted by model, so a higher good to bad ratio. In test data (when you apply your scorecard for new customers), this ratio is typically very different. Therefore the probabilities predicted by the logistic model can not directly used. &#xA;There is no easy solution for this (look at reject inference problem). A simple improvement would reweighting/downsampling the train data to match the good/bad ratio of the test data (if you have an approximation for that).&lt;/p&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2017-10-09T16:46:29.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23644" PostTypeId="2" ParentId="19099" CreationDate="2017-10-09T17:21:17.307" Score="8" Body="&lt;p&gt;&lt;strong&gt;Weight&lt;/strong&gt; - Weight is the strength of the connection. If I increase the input then how much influence does it have on the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weights near zero mean changing this input will not change the output. Many algorithms will automatically set those weights to zero in order to simplify the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bias&lt;/strong&gt; - as means how far off our predictions are from real values. Generally parametric algorithms have a high bias making them fast to learn and easier to understand but generally less flexible. In turn they are have lower predictive performance on complex problems that fail to meet the simplifying assumptions of the algorithms bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Low Bias&lt;/strong&gt;: &lt;em&gt;Suggests less assumptions about the form of the target function.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;High-Bia&lt;/strong&gt;s: &lt;em&gt;Suggests more assumptions about the form of the target function.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="32762" LastEditorUserId="-1" LastEditDate="2020-04-26T14:26:40.233" LastActivityDate="2020-04-26T14:26:40.233" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23646" PostTypeId="2" ParentId="23523" CreationDate="2017-10-09T17:54:38.277" Score="1" Body="&lt;p&gt;Apparently, I misunderstood your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several methods for finding the k-best paths with extending versions of the Viterbi algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first advice would be to look at &lt;a href=&quot;https://stackoverflow.com/questions/13768335/finding-the-top-k-viterbi-paths-in-hmm&quot;&gt;this question&lt;/a&gt; on SO that is similar to yours and has a good illustrated answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I would refer you to two articles/thesis that are publicly available and from where one can extend his/her research. &lt;em&gt;(Disclaimer: these references may not be the &quot;best&quot; one but I've chosen them because they are publicly available and provide a good number of reference to deepen research on the topic)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The k-best paths in Hidden Markov Models. Algorithms and Applications to Transmembrane Protein Topology Recognition. Thesis by Golod, available &lt;a href=&quot;https://uwspace.uwaterloo.ca/bitstream/handle/10012/4603/thesis.pdf?sequence=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. (a thesis like this gives countless references on the topic)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decoding HMMs using the k-best paths: algorithms and applications by Brown and Dolog, available &lt;a href=&quot;https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-S1-S28&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; (a partial short version of what you will find in the aforementioned thesis)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My previous answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For anyone coming across this question looking for a way of computing some of the typical state sequences of an HMM (as I thought first this question was about), just know that such a concept of most probable sequence without specifying data is not really something used in any theory about HMMs, as far as I know. However, one can follow these steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a first try, I would implement something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Get the state at time $t=0$&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Draw an initial state $s_0$ from the initial state probability mass function (pmf)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Get the state at time $t+1$&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Draw the new state $s_1$ from the pmf defined by the $s_0$-th row of the transition matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repeat this step as many times as needed to draw states up to $s_N$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can repeat the entire procedure $X$ times in order to get as many sample paths as you wish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very fast and easy to implement and it will give you what you want. Many scientific libraries in many languages have a built-in function for drawing a random sample from a pmf.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastEditorUserId="25157" LastEditDate="2017-10-10T20:55:15.253" LastActivityDate="2017-10-10T20:55:15.253" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23647" PostTypeId="2" ParentId="10558" CreationDate="2017-10-09T18:00:16.267" Score="1" Body="&lt;p&gt;Curiously, a half way authoritative answer is possible. I too wanted to know this answer. Googling found very little until I stumbled across a Data Scientist job posting on amazon's web site. At least as of August 28, 2017 they were looking for:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Solid background in statistical learning techniques for NLP (HMMs, CRFs, SVMs, LDA, LSI, MRFs etc)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So this isn't an exhaustive list, in particular there could be interesting technologies that they don't want to advertise, but the above list is at least a subset of what techniques they use.&lt;/p&gt;&#xA;" OwnerUserId="40235" LastActivityDate="2017-10-09T18:00:16.267" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23648" PostTypeId="1" AcceptedAnswerId="23649" CreationDate="2017-10-09T18:26:57.497" Score="2" ViewCount="968" Body="&lt;p&gt;I have a small dataset, so I have to use cross validation to report the test result to get a better estimate of the classification result. For some reason, I have to use neural networks to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because neural networks have their unique quirks e.g finding hyper-parameters, I am using a nested cross-validation. I am dividing up my dataset into 10 folds for cross-validation. Then I am dividing the 9 folds that are for training, again in 10 folds. From those 10 folds, I am using 9 folds to train with different hyper-parameters(in my case it is number of hidden units and dropout rate), and using the other fold to to get the accuracy with different hyper-parameters (kind of like a validation set in the deep learning literature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I am training my model again on all of the 9 folds of the first division of data with the best hyper-parameters I found. Because I am missing out on some data from the initial 9 folds for using as a validation set. Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result? Just to make it clear, I am not tuning any hyper-parameters at this stage. I am just setting the network to stop training on the training data when it reaches the best test result. I think this is a really subtle problem, if it is a problem at all. That is why I am confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am doing this whole thing 5-6 times with different seeds for different divisions of data, and I am only reporting the mean of all of these runs.&lt;/p&gt;&#xA;" OwnerUserId="13391" LastActivityDate="2017-10-10T07:00:02.893" Title="Reporting test result for cross-validation with Neural Network" Tags="|machine-learning|python|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23649" PostTypeId="2" ParentId="23648" CreationDate="2017-10-09T18:43:21.160" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Technically yes this is incorrect process for reporting an unbiased test metric. This could be a bad over-estimation of performance if cv results are noisy and vary randomly epoch-to-epoch. You should in theory treat the early stopping epoch number same as your other hyper-parameters, discover a good value from the lower-level cross-validation and stick with it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a problem with my suggestion though - the early stopping epoch number is sensitive to training data set size, and you just increased the size when you changed from lower level cross-validation to the higher level one. So you might get an unbiased measure at the expense of significantly worse results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, take a look at your learning curves. Just how sensitive are the cv results to epoch number? If they are not sensitive - no obvious over-fit over a reasonable range of epoch numbers - then just pick a mid-range fixed number of epochs that applies to all folds. That way you will have your unbiased estimate, and &lt;em&gt;probably&lt;/em&gt; not compromised on model quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you may just have to be happy knowing that your test estimate is biased, but you have reduced the variance significantly by using 10-fold CV, and have only searched one hyper-parameter at the top level (not all three). It may only be slightly biased, and still a good estimate. The smoother and less jittery the learning curves are, the more you can get away with this - but sadly you won't get a measure of the bias, so you'll never be 100% sure.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-10T07:00:02.893" LastActivityDate="2017-10-10T07:00:02.893" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23650" PostTypeId="1" CreationDate="2017-10-09T19:45:40.957" Score="2" ViewCount="47" Body="&lt;p&gt;Consider the following example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a rare disease whose occurrence seems to depend on a certain number of variables. You build a model which tries to predict patients most likely to be effected by the disease which is partially successful, that is, it predicts likely onset of the disease with some accuracy, but less than desired. Ideally, you update your model as more historical data on patients with this disease comes in, and the accuracy starts to improve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eventually you start notifying high risk patients and provide them with steps to counter this disease. Because of this, more and more patients who would have been classified as high risk are actually not catching the disease and therefore decreasing the accuracy of your model. In a sense the model was a 'victim' of its own success. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any strategies for dealing with such prediction scenarios: Where a model designed to predict an undesirable outcome looses accuracy due to it successfully averting the outcome in real world cases? &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2017-12-05T19:11:54.353" Title="How can we handle cases were the quality of the prediction undermines the quality of the data?" Tags="|classification|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23651" PostTypeId="1" CreationDate="2017-10-09T20:19:27.787" Score="11" ViewCount="24764" Body="&lt;p&gt;I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and end points; and (2) estimating the amount of crime for a specific point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to use a linear regression model. &lt;strong&gt;However, can I use these GPS coordinates directly in a linear model?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Latitude and longitude do not have an &lt;a href=&quot;https://stackoverflow.com/questions/34078894/categorical-and-ordinal-feature-data-difference-in-regression-analysis&quot;&gt;ordinal property&lt;/a&gt;, such as with a person's age. For example, the two points (40.805996, -96.681473) and (41.226682, -95.986587) do not seem to have any meaningful ordering. They are just points in space. I was thinking of replacing them with categorical US zip codes and then doing &lt;a href=&quot;https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science&quot; rel=&quot;noreferrer&quot;&gt;one-hot encoding&lt;/a&gt;, but that would result in &lt;strong&gt;a lot&lt;/strong&gt; of variables.&lt;/p&gt;&#xA;" OwnerUserId="40240" LastEditorUserId="40240" LastEditDate="2017-10-10T00:01:33.663" LastActivityDate="2018-03-07T14:23:01.663" Title="Can GPS coordinates (latitude and longitude) be used as features in a linear model?" Tags="|machine-learning|feature-selection|linear-regression|feature-extraction|geospatial|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23652" PostTypeId="1" AcceptedAnswerId="23790" CreationDate="2017-10-09T21:26:57.450" Score="1" ViewCount="634" Body="&lt;p&gt;I have a complex graph that takes an input $x_t$, it does a bunch of convolutions, tensor shape manipulations etc... until it produces $y_t$.&#xA;I want now to try a different approach, where I feed $y_t$ at some point into the graph (I concatenate it to some features after a convolution), while the net is processing $x_{t+1}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the net should do the following ($y_0$ is a fixed initial input):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(x_0,y_0)\rightarrow y_1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(x_1,y_1)\rightarrow y_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been looking at the standard RNN implementations in TensorFlow (GRU, LSTM, etc), but I don't quite understand how I can do this, because it seems that LSTM and the other RNN units have a fixed graph (like, a fully connected layer) and I cannot use the one that I need instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to subclass &lt;code&gt;tf.nn.rnn_cell.RNNCell&lt;/code&gt;? That seems problematic too because it seems to admit only 2D tensors as input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I just discovered &lt;a href=&quot;http://n-s-f.github.io/2017/07/10/rnn-tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this awesome article&lt;/a&gt; and I'm going to document myself about &lt;code&gt;tf.scan&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39081" LastEditorUserId="39081" LastEditDate="2017-10-10T16:51:35.180" LastActivityDate="2017-10-14T12:51:24.597" Title="Can TF turn a given graph into a recursive one?" Tags="|python|tensorflow|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23653" PostTypeId="2" ParentId="23651" CreationDate="2017-10-09T23:15:08.007" Score="6" Body="&lt;p&gt;You cannot use them directly, as it is unlikely there is a true linear relationship unless you're looking to predict &quot;how far east or north&quot; someone is. As mentioned in the comments, you need to convert them into zones. If you wanted to keep it really simple, you could use a kNN clustering algorithm with a low number of potential clusters and then assign each instance a new feature with the cluster ID, and then one-hot encode that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also want to read about how people interpolate coordinates to predict values across a whole map. The first example is with temperature stations, but you can also imagine it being &quot;hot zones&quot; for crime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://docs.qgis.org/2.18/en/docs/gentle_gis_introduction/spatial_analysis_interpolation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DOCS&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="30212" LastEditorUserId="29575" LastEditDate="2018-03-07T14:23:01.663" LastActivityDate="2018-03-07T14:23:01.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23656" PostTypeId="2" ParentId="23616" CreationDate="2017-10-10T04:05:00.590" Score="0" Body="&lt;p&gt;You could try modeling it as a discrete distribution and then try obtaining the random samples. Try making a function p(x) and deriving the CDF from that. In the example you gave the CDF graph would look like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DmiYM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DmiYM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you obtained your CDF you can try using &lt;strong&gt;Inverse Transform Sampling&lt;/strong&gt;. This method allows you to obtain random samples for the distribution specified. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://en.wikipedia.org/wiki/Inverse_transform_sampling&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; would be helpful for your understanding of that technique. These random samples can fill those missing values as per your requirement of probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other techniques as well, you could search and explore along the lines of &lt;em&gt;random sample generation from discrete distributions&lt;/em&gt;. It might be the case that your actual data might fit for example something like Poisson's distribution etc. For all such cases, you can find various examples on how to generate random samples for those particular distributions &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-10T04:05:00.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23657" PostTypeId="1" AcceptedAnswerId="23659" CreationDate="2017-10-10T05:37:02.307" Score="1" ViewCount="72" Body="&lt;p&gt;What are the best machine learning techniques to classify responders to a medicine if I have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Clinical data with ~200 features (age, education, marital status etc.)&lt;/li&gt;&#xA;&lt;li&gt;Gene data with around 250K features (genom data (snips) taken from the patient (DNA analysis))&lt;/li&gt;&#xA;&lt;li&gt;Number of obs. ~ 4K (the data is taken from a study on 4000 patients).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please advise.&lt;/p&gt;&#xA;" OwnerUserId="40255" LastEditorUserId="40255" LastEditDate="2017-10-10T06:04:45.613" LastActivityDate="2017-10-10T06:44:15.900" Title="What machine learning technique should I use in medical problem" Tags="|machine-learning|classification|data-mining|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23659" PostTypeId="2" ParentId="23657" CreationDate="2017-10-10T06:44:15.900" Score="1" Body="&lt;p&gt;Typically you need a much more exact phrasing (in more mathematical terms) of the question to ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will a patient respond to medicine X? What is the likelihood? To what amount will patient repond to medicine X? Is patient in the group that is expected to respond to medicine X?&#xA;Are slightly diffent questions that may impact choice of technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, your data plays an important role. Are you missing data? Have you normalized already? Do you expect 'Marital status' or 'Education level' to have an impact on medicine efficiency? (It might if certain groups take medicine at home, but it may be less likely when taking under doctor supervision)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A priori you determine how to measure or quantize the success of a model (typically prediction accuracy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then typically you try a few machine learning techniques, and make a model from the most successful one. &lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-10T06:44:15.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23661" PostTypeId="1" CreationDate="2017-10-10T08:08:26.797" Score="2" ViewCount="1298" Body="&lt;p&gt;I am looking for a dataset that has been crowdsourced, and corresponds to a multiple answer correct per question setting. I need the individual labels given by the participants from the crowd.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have searched online (such as &lt;a href=&quot;https://www.crowdflower.com/data-for-everyone/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), but all I get is datasets where the answers have been aggregated to create a ground truth. I also found sources like &lt;a href=&quot;https://github.com/ipeirotis/Get-Another-Label&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, which provide such a dataset but apply to the single label case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any help. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="40260" LastActivityDate="2018-02-03T15:45:42.093" Title="Where can I find a crowdsourced dataset for multi-label classification with individual participant labels?" Tags="|dataset|multilabel-classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23662" PostTypeId="1" CreationDate="2017-10-10T09:49:22.423" Score="0" ViewCount="104" Body="&lt;p&gt;I'm currently working on an assignment related to house prices in the UK. I have a list of prices that depend on the size of the property (small, medium, large) and the property location in terms of how close the property is to urban areas (Yes = close to Urban areas | No = far away). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data that I have is from two extremes, so all the properties are either close to urban areas or far away. As you can assume, the prices are high for large properties that are close to urban areas. In contrast, the prices are low for small properties that are far from urban areas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the data is quite overlapping and a small house in urban area might have the same price (or close) to a large property outside the urban zone. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my task is to determine whether the property is located in urban areas just by using house prices (the only available feature). But to do so I need to determine the size of the property as this has a significant affect on the price as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reality, I would like to test my algorithm(s) on a range of property prices even those that are somewhere in between that is, not far to urban areas but no so close either. Can someone share their ideas on how one would tackle such a problem using only the house prices as my features? Thanks in advance for your input!&lt;/p&gt;&#xA;" OwnerUserId="40263" LastEditorUserId="23305" LastEditDate="2017-10-10T21:45:16.117" LastActivityDate="2017-11-10T10:53:23.203" Title="Classification methods using one overlapping feature" Tags="|machine-learning|classification|data-mining|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23664" PostTypeId="2" ParentId="22828" CreationDate="2017-10-10T10:22:49.447" Score="5" Body="&lt;p&gt;I'd use sklearn's Hierarchical clustering&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer&#xA;from scipy.cluster import  hierarchy&#xA;&#xA;#Vectorizing&#xA;X = CountVectorizer().fit_transform(docs)&#xA;X = TfidfTransformer().fit_transform(X)&#xA;#Clustering&#xA;X = X.todense()&#xA;threshold = 0.1&#xA;Z = hierarchy.linkage(X,&quot;average&quot;, metric=&quot;cosine&quot;)&#xA;C = hierarchy.fcluster(Z, threshold, criterion=&quot;distance&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;C&lt;/code&gt; is your clustering of the documents &lt;code&gt;docs&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use other metrics instead of &lt;code&gt;cosine&lt;/code&gt;, and use a different threshold than &lt;code&gt;0.1&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-10T10:22:49.447" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23665" PostTypeId="2" ParentId="23634" CreationDate="2017-10-10T10:42:52.560" Score="0" Body="&lt;p&gt;The answers to the questions above are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Q. Is it part of the Apache foundation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. Quite simply. No. Elastic is Open Source but not part of the Apache foundation&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Q. Where does the software sit? Would it form part of an Hadoop ecosystem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. Elastic is entirely separate from Hadoop and requires its own cluster of machines in order to operate. The data is distributed across this cluster, as in Hadoop however the data is indexed for quick retrieval. There is then a front end visualization and analysis tool called Kibana that can be used to interrogate the data and to write queries&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Q. When would I use it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. When you have a large dataset that you want to query very quickly&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Q. What skills do I need in order to use it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. You will need to be able to write Elastic queries. An SQL-like query language&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2017-10-10T10:42:52.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23666" PostTypeId="1" AcceptedAnswerId="23677" CreationDate="2017-10-10T10:50:22.720" Score="16" ViewCount="16054" Body="&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest#From_bagging_to_random_forests&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia page&lt;/a&gt; which quotes &lt;a href=&quot;https://statistics.stanford.edu/~tibs/ElemStatLearn&quot; rel=&quot;noreferrer&quot;&gt;&quot;The Elements of Statistical Learning&quot;&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Typically, for a classification problem with $p$ features, $\lfloor \sqrt{p}\rfloor$  features are used in each split.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I understand that this is a fairly good educated guess and it was probably confirmed by empirical evidence, but are there other reasons why one would pick the square root? Is there a statistical phenomenon happening there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this somehow help decrease the variance of the errors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the same for regression and classification?&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-10-10T20:48:56.440" LastActivityDate="2017-10-10T20:48:56.440" Title="How many features to sample using Random Forests" Tags="|statistics|random-forest|optimization|model-evaluations|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23667" PostTypeId="2" ParentId="10898" CreationDate="2017-10-10T12:02:15.843" Score="5" Body="&lt;p&gt;There are various approaches.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;1. K-D Tree based on L2-normalized euclidean distance&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;K-D Tree is an efficient space split method. See &lt;a href=&quot;https://en.wikipedia.org/wiki/K-d_tree&quot; rel=&quot;noreferrer&quot;&gt;wikipedia&lt;/a&gt;.&lt;br&gt;&#xA;So it's used for euclidean distance neighbor searching intuitively.&lt;br&gt;&#xA;Why I say it also works for cosine similarity, consider  &lt;strong&gt;the equivalence between cosine similarity and L2-normalized euclidean distance someway&lt;/strong&gt;.&lt;br&gt;&#xA;When $\mathbf{x}$ and $\mathbf{y}$ are unit vectors, then:&lt;br&gt;&#xA;\begin{align}&#xA;||\mathbf{x} - \mathbf{y}||_2^2&#xA;&amp;amp;= (\mathbf{x} - \mathbf{y})^\top (\mathbf{x} - \mathbf{y}) \\&#xA;&amp;amp;= \mathbf{x}^\top \mathbf{x} - 2 \mathbf{x}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} \\&#xA;&amp;amp;= 2 - 2\mathbf{x}^\top \mathbf{y} \\&#xA;&amp;amp;= 2 - 2 \cos\angle(\mathbf{x}, \mathbf{y})&#xA;\end{align}&#xA;So, it will produce identical results for a similarity ranking between a vector &lt;strong&gt;u&lt;/strong&gt; and a set of vectors &lt;strong&gt;V&lt;/strong&gt;. &#xA;See &lt;a href=&quot;https://stats.stackexchange.com/questions/146221/is-cosine-similarity-identical-to-l2-normalized-euclidean-distance/146279#146279&quot;&gt;an stack-exchange post&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;2.The Priority Search K-Means Tree Algorithm&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;It's an approximate method.&lt;br&gt;&#xA;Use cosine similarity as distance measure.&lt;br&gt;&#xA;See the paper &lt;a href=&quot;http://www.cs.ubc.ca/research/flann/uploads/FLANN/flann_pami2014.pdf&quot; rel=&quot;noreferrer&quot;&gt;Scalable Nearest Neighbor Algorithms&#xA;for High Dimensional Data&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;3. Search within sub-search-space after The K-Means clustering&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;It's an approximate method.&lt;br&gt;&#xA;After K-Means clustering via cosine similarity, we get k subsets and respective centroid. We can let k be 1000 or larger.&lt;br&gt;&#xA;So for a given query point, our steps are:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;find the nearest centroid, and find the corresponding subset.  &lt;/li&gt;&#xA;&lt;li&gt;search and sort in this subset.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="39820" LastActivityDate="2017-10-10T12:02:15.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23668" PostTypeId="1" CreationDate="2017-10-10T13:02:22.627" Score="2" ViewCount="104" Body="&lt;p&gt;I want to be able to correlate values from various IDs, where the date is the same with one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data looks something like this;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID      Time(secs)   Date&#xA;AAAA    1            01/01/1990&#xA;AAAA    6            02/01/1990&#xA;AAAA    5            03/01/1990&#xA;AAAA    2            04/01/1990&#xA;AAAA    4            05/01/1990&#xA;BBBB    2            01/01/1990&#xA;BBBB    4            02/01/1990&#xA;BBBB    6            03/01/1990&#xA;BBBB    3            04/01/1990&#xA;CCCC    3            01/01/1990&#xA;CCCC    4            02/01/1990&#xA;CCCC    1            03/01/1990&#xA;CCCC    6            04/01/1990&#xA;DDDD    7            01/01/1990&#xA;DDDD    4            02/01/1990&#xA;DDDD    5            03/01/1990&#xA;DDDD    3            04/01/1990&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to find the correlation coefficient between each combination of these IDs, where there are matching dates. N.B. Not all of the IDs have the same date;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID             CorrCoef&#xA;AAAA&amp;gt;BBBB      ????&#xA;AAAA&amp;gt;CCCC&#xA;AAAA&amp;gt;DDDD&#xA;BBBB&amp;gt;CCCC&#xA;BBBB&amp;gt;DDDD&#xA;CCCC&amp;gt;DDDD&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think I need to feed the data from each ID into a variable and then run the following;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data1.corr(data2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37059" LastEditorUserId="20492" LastEditDate="2017-10-13T12:37:35.630" LastActivityDate="2024-04-05T17:01:20.137" Title="Reading values from a column into a variable and then correlating using Python" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23669" PostTypeId="1" CreationDate="2017-10-10T13:21:20.900" Score="5" ViewCount="8361" Body="&lt;p&gt;I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either &quot;good&quot; or &quot;bad&quot; (binary classification). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used XGBoost to classify these data, but I get to much overfitting. I do cross validation using the logloss to evaluate the performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Green : logloss validation curve&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Red : logloss training curve&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X-axis : nrounds (max number of boosting iterations)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iiumz.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iiumz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first model (first picture was generated with : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bst.res &amp;lt;- xgb.cv(nfold = 4,&#xA;               data = matrix_training_set,&#xA;               label = training_set_label,&#xA;               eta = 0.1,&#xA;               # minimum sum of instance weight(hessian) needed in a child &#xA;               min_child_weight = 4,&#xA;               # maximum depth of a tree &#xA;               max_depth = 14,&#xA;               objective = &quot;binary:logistic&quot;,&#xA;               #L2 parameter &#xA;               lambda = 1.1,&#xA;               # L1 parameter &#xA;               alpha = 0.5, &#xA;               gamma = 0,&#xA;               eval_metric = &quot;logloss&quot;,&#xA;               nrounds = 2000,&#xA;               verbose = TRUE,&#xA;               subsample = 0.7,&#xA;               print_every_n = 10,&#xA;               early_stop_round = 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I tried (second picture) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bst.res &amp;lt;- xgb.cv(nfold = 4,&#xA;               data = matrix_training_set,&#xA;               label = training_set_label,&#xA;               eta = 0.01,&#xA;               # minimum sum of instance weight(hessian) needed in a child &#xA;               min_child_weight = 4,&#xA;               # maximum depth of a tree &#xA;               max_depth = 6,&#xA;               objective = &quot;binary:logistic&quot;,&#xA;               #L2 parameter &#xA;               lambda = 1.1,&#xA;               # L1 parameter &#xA;               alpha = 0.5, &#xA;               gamma = 1,&#xA;               eval_metric = &quot;logloss&quot;,&#xA;               nrounds = 2000,&#xA;               verbose = TRUE,&#xA;               subsample = 0.7,&#xA;               print_every_n = 10,&#xA;               early_stop_round = 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, it is possible to reduce overfitting by changing max_depth, min_child_weight , gamma, subsample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below, again, I reduce the overfitting (3rd picture) : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bst.res &amp;lt;- xgb.cv(nfold = 4,&#xA;               data = matrix_training_set,&#xA;               label = training_set_label,&#xA;               eta = 0.01,&#xA;               # minimum sum of instance weight(hessian) needed in a child &#xA;               min_child_weight = 7,&#xA;               # maximum depth of a tree &#xA;               max_depth = 6,&#xA;               objective = &quot;binary:logistic&quot;,&#xA;               #L2 parameter &#xA;               lambda = 1.1,&#xA;               # L1 parameter &#xA;               alpha = 0.5, &#xA;               gamma = 5,&#xA;               eval_metric = &quot;logloss&quot;,&#xA;               nrounds = 2000,&#xA;               colsample_bytree = 0.7,&#xA;               verbose = TRUE,&#xA;               subsample = 0.3,&#xA;               print_every_n = 10,&#xA;               early_stop_round = 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now it seems that the model is biased (red and green are indeed really close, take care of the scale).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried grid searching, but the validation curve always stand really high (over 0.5 for logloss and less than 0.62 for AUC). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm wondering if I should create a model more biased (make the trees simplier), and then add more data to reduce this bias. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas to make this green curve lower ? Is that possible that there is no possible correlation between my target (&quot;good&quot; or &quot;bad&quot;) and my features, which means that it becomes impossible to create a classifier from these data ? &lt;/p&gt;&#xA;" OwnerUserId="40270" LastActivityDate="2018-01-17T17:02:45.473" Title="Overfitting XGBoost" Tags="|r|xgboost|cross-validation|overfitting|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23670" PostTypeId="2" ParentId="4957" CreationDate="2017-10-10T14:39:29.670" Score="0" Body="&lt;p&gt;You can find an Excel and VBA implementation of Random Forest using the open source ALGLIB Library &lt;a href=&quot;https://www.academia.edu/attachments/54618055/download_file?st=MTUwNzY0NjEwOCwxOTYuMjMuNTAuNzMsMTIyMTAxMg%3D%3D&amp;amp;s=profile&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40147" LastActivityDate="2017-10-10T14:39:29.670" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23671" PostTypeId="1" CreationDate="2017-10-10T14:45:32.997" Score="-1" ViewCount="398" Body="&lt;p&gt;I have data on horse racing, including 6 fetures for the race, and 27 features for each horse in the race - I also have the odds of the horse and whether it won or not for that race.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for a ML algorithm (or multiple - more ways of trying it the better) which firstly could support a varied number of horses per race such as this, (some races have as few as say 4 horses per race, some as many as 30), but also not to train it simply to find the most likely winner (That would be easy, one data feature on the horses if predicted odds, so picking the lowest value here would pretty much get a decent strike rate of picking a winner), but I want to be picking winners who would provide the most value. so for a given race, it might be a horse with expected odds of say 15, with some lower odds horses and some higher in the race, but this may be the best bet based on the likelyhood of that horse actaully winning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to train any model whilst taking into account the odds as well as the fact as whether they won or not as part of the supervised data.&lt;/p&gt;&#xA;" OwnerUserId="39715" LastActivityDate="2017-10-11T00:30:24.557" Title="Which Algorithm(s) can fit odds-effected racing, and how is that structured?" Tags="|algorithms|supervised-learning|model-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23672" PostTypeId="1" CreationDate="2017-10-10T15:29:22.913" Score="2" ViewCount="136" Body="&lt;p&gt;As the &lt;a href=&quot;https://arxiv.org/abs/1301.3764&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper by Schaul &amp;amp; LeCun&lt;/a&gt; states&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The findings are clear: in contrast to the other algorithms tested,&#xA;vSGD-fd does not require any hyper-parameter tuning to give reliably&#xA;good performance on the broad range of tests: the learning rates adapt&#xA;automatically to different curvatures and noise levels.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Auto-adapting learning rate sounds like a huge deal,&#xA;but why is everyone seem to be using Adam and other optimizers?&lt;/p&gt;&#xA;" OwnerUserId="37279" LastEditorUserId="98307" LastEditDate="2020-08-12T14:21:56.583" LastActivityDate="2020-09-15T06:01:55.217" Title="Why vSGD-fd optimization algorithm isn't popular?" Tags="|neural-network|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23674" PostTypeId="1" AcceptedAnswerId="23675" CreationDate="2017-10-10T19:02:07.180" Score="0" ViewCount="16255" Body="&lt;p&gt;I have a json file which has multiple events, each event starts with EventVersion Key. The data looks similar to the following synthesized data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;Records&quot;:[{&quot;eventVersion&quot;:&quot;1.04&quot;,&quot;userIdentity&quot;:{&quot;type&quot;:&quot;R&quot;,&quot;principalId&quot;:&quot;P:i&quot;,&quot;arn&quot;:&quot;arn:aws:sts::5&quot;,&quot;accountId&quot;:&quot;50&quot;,&quot;accessKeyId&quot;:&quot;AW&quot;,&quot;sessionContext&quot;:{&quot;attributes&quot;:{&quot;mfaAuthenticated&quot;:&quot;f&quot;,&quot;creationDate&quot;:&quot;2013-09&quot;},&quot;sessionIssuer&quot;:{&quot;type&quot;:&quot;R&quot;,&quot;principalId&quot;:&quot;WA&quot;,&quot;arn&quot;:&quot;arn:aws:iam::6&quot;,&quot;accountId&quot;:&quot;70&quot;,&quot;userName&quot;:&quot;user1&quot;}}},&quot;eventTime&quot;:&quot;2027-6&quot;,&quot;eventSource&quot;:&quot;a.com&quot;,&quot;eventName&quot;:&quot;DS&quot;,&quot;awsRegion&quot;:&quot;UZ&quot;,&quot;sourceIPAddress&quot;:&quot;2.1.3&quot;,&quot;userAgent&quot;:&quot;li&quot;,&quot;requestParameters&quot;:null,&quot;responseElements&quot;:null,&quot;requestID&quot;:&quot;OO&quot;,&quot;eventID&quot;:&quot;09&quot;,&quot;eventType&quot;:&quot;ABC&quot;,&quot;apiVersion&quot;:&quot;2010-4&quot;,&quot;recipientAccountId&quot;:&quot;78&quot;},{&quot;eventVersion&quot;:&quot;1.04&quot;,&quot;userIdentity&quot;:{&quot;type&quot;:&quot;R&quot;,&quot;principalId&quot;:&quot;P:i&quot;,&quot;arn&quot;:&quot;arn:aws:sts::5&quot;,&quot;accountId&quot;:&quot;50&quot;,&quot;accessKeyId&quot;:&quot;AW&quot;,&quot;sessionContext&quot;:{&quot;attributes&quot;:{&quot;mfaAuthenticated&quot;:&quot;f&quot;,&quot;creationDate&quot;:&quot;2013-09&quot;},&quot;sessionIssuer&quot;:{&quot;type&quot;:&quot;R&quot;,&quot;principalId&quot;:&quot;WA&quot;,&quot;arn&quot;:&quot;arn:aws:iam::6&quot;,&quot;accountId&quot;:&quot;70&quot;,&quot;userName&quot;:&quot;user1&quot;}}},&quot;eventTime&quot;:&quot;2027-6&quot;,&quot;eventSource&quot;:&quot;a.com&quot;,&quot;eventName&quot;:&quot;DS&quot;,&quot;awsRegion&quot;:&quot;UZ&quot;,&quot;sourceIPAddress&quot;:&quot;2.1.3&quot;,&quot;userAgent&quot;:&quot;li&quot;,&quot;requestParameters&quot;:null,&quot;responseElements&quot;:null,&quot;requestID&quot;:&quot;OO&quot;,&quot;eventID&quot;:&quot;09&quot;,&quot;eventType&quot;:&quot;ABC&quot;,&quot;apiVersion&quot;:&quot;2010-4&quot;,&quot;recipientAccountId&quot;:&quot;78&quot;}]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using the following code in Python to convert this to Pandas Dataframe such that Keys are columns and values of each event is a row. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with open('/Users/snehahonnappa/Documents/NLP_AWSlogs/Model/Data/505728423372_CloudTrail_ap-northeast-1_20160913T1700Z_yKA3wB5Nx6juR6Kg.json') as json_data:&#xA;    sample_object = json.load(json_data)&#xA;&#xA;df = pd.io.json.json_normalize(sample_object)&#xA;df.columns = df.columns.map(lambda x: x.split(&quot;.&quot;)[-1])&#xA;&#xA;print df.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I print shape of the dataframe its 1X1. I'm expecting (Number of unique keys X Number of records) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Snippet of how I'm expecting the dataframe to be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eventVersion               userIdentity                          eventTime&#xA;             type   principalId P   arn     accountID userName  &#xA;1.04         R      P           i   arn:aws 50        user1      2027-6&#xA;1.06         Q      O           i   arn:aws 67        u2         2027-7 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Appreciate any help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm writing the json file into a csv and then trying to convert this to dataframe on which my models can be applied on. Following is my code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import csv&#xA;import sys&#xA;&#xA;data_parsed = json.loads(open('/tmp/A.json').read())&#xA;log_data = data_parsed['Records']&#xA;&#xA;# open a CSV file for writing&#xA;data = open('/tmp/log.csv', 'w')&#xA;&#xA;# create the csv writer object&#xA;csvwriter = csv.writer(data)&#xA;count = 0&#xA;&#xA;for i in log_data:&#xA;      if count == 0:&#xA;             header = i.keys()&#xA;             csvwriter.writerow(header)&#xA;             count += 1&#xA;      csvwriter.writerow(i.values())&#xA;&#xA;data.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is writing the keys as headers and values of each record as a separate row which is as expected. However the nested json objects are being written as one value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is a snippet of my csv file which was obtained by executing the above code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;eventVersion eventID eventTime requestParameters                eventType&#xA;1.04         0       2016-20                                    AwsApiCall&#xA;1.04         8       2016-20    {u'tagKeys': [u'User Name']}    AwsApiCall&#xA;1.05         4       2016-30    {u'filterSet': {u'items': [{u'name': u'resource-type', u'valueSet': {u'items': [{u'value': u'*'}]}}, {u'name': u'tag:User Name', u'valueSet': {u'items': [{u'value': u'*'}]}}]}}    &#xA;                                                                AwsApiCall&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions to tackle this?&lt;/p&gt;&#xA;" OwnerUserId="31135" LastEditorUserId="31135" LastEditDate="2017-10-10T22:35:13.947" LastActivityDate="2017-10-10T22:35:13.947" Title="Converting Json file to Dataframe Python" Tags="|machine-learning|python|csv|json|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23675" PostTypeId="2" ParentId="23674" CreationDate="2017-10-10T19:31:56.113" Score="2" Body="&lt;p&gt;I once ran into a situation like this where i wanted a complex dataframe due to the original source having a complex data structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I solved my issue by simplifying my structure into multiple separate dataframes instead of one big complex multi structure dataframe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With simple separate dataframes i was better positioned to apply complex algorithmic operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at your specific data, you could get rid of userIdentity which results in a simple 2d dataframe. This should position you to do any complex operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand this doesn't answer your specific dataframe structure requirement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But i hope this answers the spirit of your objective.&lt;/p&gt;&#xA;" OwnerDisplayName="user40285" LastEditorDisplayName="user40285" LastEditDate="2017-10-10T19:37:31.227" LastActivityDate="2017-10-10T19:37:31.227" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23676" PostTypeId="2" ParentId="23614" CreationDate="2017-10-10T20:27:08.703" Score="10" Body="&lt;p&gt;I would use the functional interface.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Activation, Input, Dense&#xA;from keras.models import Model&#xA;from keras.layers.merge import Concatenate&#xA;&#xA;input_ = Input(shape=input_shape)&#xA;&#xA;x = input_&#xA;x1 = Dense(4, x)&#xA;x2 = Dense(4, x)&#xA;x3 = Dense(4, x)&#xA;x1 = Activation('softmax')(x1)&#xA;x2 = Activation('softmax')(x2)&#xA;x3 = Activation('softmax')(x3)&#xA;x = Concatenate([x1, x2, x3])&#xA;&#xA;model = Model(inputs=input_, outputs=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-10-10T20:27:08.703" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="23677" PostTypeId="2" ParentId="23666" CreationDate="2017-10-10T20:37:05.350" Score="23" Body="&lt;p&gt;I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees increasing the strength of the forest as a whole. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's interesting is that the authors of &lt;a href=&quot;https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf&quot; rel=&quot;noreferrer&quot;&gt;Random Forests&lt;/a&gt; (pdf) find an empirical difference between classification and regression:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An interesting difference between regression and classification is that the&#xA;  correlation increases quite slowly as the number of features used increases.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Therefore, for regression often $N/3$ is recommended, which gives larger values than $\sqrt N$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, there is no clear justification for $\sqrt N$ or $\log N$ for classification problems other than that it has shown that lower correlation among trees can decrease generalization error enough to more than offset the decrease in strength of individual trees. In particular, the authors note that the range where this trade-off can decrease the generalization error is quite large:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;The in-between&#xA;  range is usually large. In this range, as the number of features goes up, the&#xA;  correlation increases, but PE*(tree) compensates by decreasing.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(PE* being the generalization error)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As they say in Elements of Statistical Learning:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In practice the best values for these parameters will depend on the problem,&#xA;  and they should be treated as tuning parameters.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;One thing your problem can depend on is the number of categorical variables. If you have many categorical variables that are encoded as dummy-variables it usually makes sense to increase the parameter. Again, from the Random Forests paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When many of the variables are categorical, using a&#xA;  low [number of features] results in low correlation, but also low strength. [The number of features] must be&#xA;  increased to about two-three times $int(log_2M+1)$ to get enough strength to&#xA;  provide good test set accuracy.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-10-10T20:37:05.350" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23679" PostTypeId="2" ParentId="23668" CreationDate="2017-10-10T22:18:47.533" Score="0" Body="&lt;p&gt;If i understand correctly you can do something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [42]: df&#xA;Out[42]:&#xA;      ID  Time(secs)        Date&#xA;0   AAAA           1  01/01/1990&#xA;1   AAAA           6  02/01/1990&#xA;2   AAAA           5  03/01/1990&#xA;3   AAAA           2  04/01/1990&#xA;4   AAAA           4  05/01/1990&#xA;5   BBBB           2  01/01/1990&#xA;6   BBBB           4  02/01/1990&#xA;7   BBBB           6  03/01/1990&#xA;8   BBBB           3  04/01/1990&#xA;9   CCCC           3  01/01/1990&#xA;10  CCCC           4  02/01/1990&#xA;11  CCCC           1  03/01/1990&#xA;12  CCCC           6  04/01/1990&#xA;13  DDDD           7  01/01/1990&#xA;14  DDDD           4  02/01/1990&#xA;15  DDDD           5  03/01/1990&#xA;16  DDDD           3  04/01/1990&#xA;&#xA;In [43]: df.pivot(index='Date', columns='ID', values='Time(secs)').corr()&#xA;Out[43]:&#xA;ID        AAAA      BBBB      CCCC      DDDD&#xA;ID&#xA;AAAA  1.000000  0.778924 -0.336336 -0.368964&#xA;BBBB  0.778924  1.000000 -0.609449 -0.257143&#xA;CCCC -0.336336 -0.609449  1.000000 -0.609449&#xA;DDDD -0.368964 -0.257143 -0.609449  1.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Virtual helper DF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [44]: df.pivot(index='Date', columns='ID', values='Time(secs)')&#xA;Out[44]:&#xA;ID          AAAA  BBBB  CCCC  DDDD&#xA;Date&#xA;01/01/1990   1.0   2.0   3.0   7.0&#xA;02/01/1990   6.0   4.0   4.0   4.0&#xA;03/01/1990   5.0   6.0   1.0   5.0&#xA;04/01/1990   2.0   3.0   6.0   3.0&#xA;05/01/1990   4.0   NaN   NaN   NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-10-10T22:18:47.533" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23680" PostTypeId="2" ParentId="17767" CreationDate="2017-10-10T22:56:56.537" Score="1" Body="&lt;p&gt;You must attach the data frame &lt;code&gt;curved_data&lt;/code&gt; before running the commands. Then, run your command like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seg_model &amp;lt;- segmented(reg, seg.Z = ~X, psi = c(-35, 4, 30),&#xA;                       control = seg.control(display = TRUE))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40296" LastEditorUserId="24000" LastEditDate="2017-10-13T08:52:47.053" LastActivityDate="2017-10-13T08:52:47.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23681" PostTypeId="1" CreationDate="2017-10-11T00:00:51.127" Score="2" ViewCount="1062" Body="&lt;p&gt;I have a model that I train on same data, but i want a feature to have a stronger weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have three features: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Car manufacturer's name&#xA;Price&#xA;Top speed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I want to classify my cars as &quot;Sport&quot;, &quot;Luxury&quot;, &quot;Family&quot;, &quot;Compact&quot; using these indicators. However, say that I realise the manufacturer's name is the strongest indicator, so I want that to weigh more. For example, if I buy a Ferrari, that is a sport car, even though the price tag may make it look like a luxury car. So, while I want to use all features, I want one of them to have a stronger weight (this is just a silly example, my actual problem is different, but this gives an idea). If a manufacturer makes all 4 kinds, speed and price are important, otherwise the name should have a stronger weight than the price and top speed. Basically, my tree should really look at the manufacturer's name and then split depending on the other features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I set up sklearn in python to do this? How?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2023-04-27T17:26:58.790" Title="Giving more weight to a particular feature in scikit-learn decision trees" Tags="|python|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23682" PostTypeId="2" ParentId="23671" CreationDate="2017-10-11T00:30:24.557" Score="0" Body="&lt;p&gt;So you want your algorithm to pick a horse that is more likely to win than the odds would suggest in order to profit? The odds are set in a way such that, taking expert opinions and loads of data into account, the bookkeepers are minimising their expected losses. It seems unlikely that you have more information on the horses than they do, so I think it will be difficult to beat them at their own game!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, the approach I would take here would be to add a new feature into the data that looks at the horses position and odds in their most recent race(s):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;score = \frac{odds}{position}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, say the odds for three particular horses and their corresponding positions in some race are as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\begin{array}{c|c|c|} &#xA; &amp;amp; \text{Predicted Odds} &amp;amp; \text{Actual Position} &amp;amp; \text{Score} \\ \hline&#xA;\text{Horse A} &amp;amp; 9:1   &amp;amp; 2 &amp;amp; 4.5 \\ \hline&#xA;\text{Horse B} &amp;amp; 1.3:1 &amp;amp; 1 &amp;amp;  1.3\\ \hline&#xA;\text{Horse C} &amp;amp; 3:1   &amp;amp; 6 &amp;amp; 0.5 \\ \hline&#xA;\end{array}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, horses that are overperforming will have high scores, and horses that are underperforming will have low scores. You can average this score over the last $n$ races that the horse participated in to get a more stable scoring to evaluate the horses current form. This feature can then be used by any standard machine learning algorithm along with any other features you may have.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-10-11T00:30:24.557" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23685" PostTypeId="1" CreationDate="2017-10-11T05:09:26.863" Score="3" ViewCount="1599" Body="&lt;p&gt;Suppose I have want to predict how likely people in both US and Canada will buy product A in a month. Basically, it's a binary classification problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume I have 2 millions rows of historical data with people who has bought/not bought the product (label). One millions rows data are from US, the other are from Canada. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can build a machine learning models in following two ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Build a model with 2 millions rows of data. One model with all the&#xA;data.&lt;/li&gt;&#xA;&lt;li&gt;Model localization: Build a model with US data. Build another model with Canada data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Intuitively, it seems to me option 1 will always performs better than 2. Because there will be more data in one model. But in practice, option 2 seems always performs better than option 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is when should I use option 2 instead of option 1? How to make the decision?&lt;/p&gt;&#xA;" OwnerUserId="6567" LastActivityDate="2017-10-11T12:15:41.030" Title="Model localization: one big model vs two small models" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="23687" PostTypeId="1" CreationDate="2017-10-11T08:36:18.057" Score="1" ViewCount="1519" Body="&lt;p&gt;According to a benchmark of GBM vs. xgboost vs. LightGBM (&lt;a href=&quot;https://www.kaggle.com/nschneider/gbm-vs-xgboost-vs-lightgbm&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/nschneider/gbm-vs-xgboost-vs-lightgbm&lt;/a&gt;) it is possible to implenet xgboost with the argument &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tree_method = 'hist' &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However doing so gives me always an error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) : &#xA;Invalid Input: 'hist', valid values are: {'approx', 'auto', 'exact'}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I missing?&lt;/p&gt;&#xA;" OwnerUserId="27714" LastActivityDate="2017-10-11T17:42:59.513" Title="xgboost with tree_method = 'hist' in R" Tags="|r|xgboost|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23688" PostTypeId="2" ParentId="23662" CreationDate="2017-10-11T10:05:39.490" Score="2" Body="&lt;p&gt;Could you plot something like heatmap or 3D plot (x=size,y=location,z=prices) ? I would try something like a simple decision tree to fit your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using only the houses prices as feature, I would first plot prices versus location. If there are two distinct clusters slightly overlaping, maybe fixing just one threshold is good enough. &lt;/p&gt;&#xA;" OwnerUserId="40270" LastActivityDate="2017-10-11T10:05:39.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23689" PostTypeId="2" ParentId="23685" CreationDate="2017-10-11T12:15:41.030" Score="2" Body="&lt;p&gt;When and how to choose 1 vs 2? Data exploration, understanding the problem, and testing the models. Make sure you use metrics, a cut-off level and test data that is appropriate for your business problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take for example, if there is a fundamental difference between US and CA customers. Then allowing the model to focus on the signal of each country, without dealing with the noise of the other country, may produce a better model for each. By doing data exploration (plot purchases by country over time and some other important features) you might convince yourself that it is worth the effort to test multiple models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the art vs science of modeling.&lt;/p&gt;&#xA;" OwnerUserId="28780" LastActivityDate="2017-10-11T12:15:41.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23690" PostTypeId="1" AcceptedAnswerId="23695" CreationDate="2017-10-11T15:47:15.933" Score="2" ViewCount="510" Body="&lt;p&gt;This it a problem that has come on my path a few times now and I don't have a satisfying solution yet. The goal is to predict probabilities or fractions based on some $x$ where our training $y$ has these probabilties or fractions and thus is in the domain $[0,1]$ as opposed to $\{0,1\}$. My question is with regards to my loss function. In the case of fractions, if the error between 0.4 and 0.5 and the error between 0.89 and 0.99 is the same I can just use MSE if I want to predict the expected value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of probabilities where we want to approach it similarly as classification problems, where the difference between 0.89 and 0.99 is much bigger than that of 0.4 and 0.5, we want to put this in our loss function. Does cross entropy still work properly if I feed it fractions in $y$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{L}(y,\hat{y})=-y\log(\hat{y}) - (1-y)\log(1-\hat{y})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say our $y=0.5$ and our current prediction is $\hat{y}=0.6$ we would get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{L}(0.5,0.6)=-0.5\log(0.6) - 0.5\log(0.4)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't really see why this would go wrong? The function is still convex. Everywhere it says that the target should be in $\{0, 1\}$ however. Maybe my math is lacking or I'm missing something obvious, why is this a bad idea?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-11T18:54:15.270" Title="Fractions or probabilities as training labels" Tags="|machine-learning|neural-network|classification|regression|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23691" PostTypeId="1" CreationDate="2017-10-11T16:32:05.397" Score="3" ViewCount="735" Body="&lt;p&gt;I'm trying to build a program that can identify peaks in a signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's my first attempt at machine learning so I'm starting with a toy problem where I'm generating a gaussian curve of random width and position with a small amount of added noise and trying to identify what part of the signal looks like a peak but I'm facing some issues. I'm using Theano+Lasagne and adapted my code from the MNIST.py example. Here's an example of what I feed to my code and what I get as a result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/19t51.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/19t51.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is to feed the signal (1D vector of 512 values) to a simple neural network composed of 5 dense layers (3 linear followed by 2 sigmoids) of 128 neuron each. And training it on tens of thousands of generated examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the issue I'm facing is that no matter what I try regarding weight initialization or learning parameters I always converge to the same result : a network that seem to only spit out the average value of the &quot;goal output&quot; of all the examples it was trained with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would assume that given the structure of my neural net and what data I'm feeding to it, it would easily converge to something that does a bit of averaging and thresholding to give me something reasonably close to the expected output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that foolish to assume ? Or am I missing something important ?&lt;/p&gt;&#xA;" OwnerUserId="40330" LastEditorUserId="836" LastEditDate="2017-10-11T17:56:14.437" LastActivityDate="2017-10-11T17:56:14.437" Title="Neural network for peak identification problem not training successfully" Tags="|neural-network|theano|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="23692" PostTypeId="1" AcceptedAnswerId="23694" CreationDate="2017-10-11T16:54:11.067" Score="2" ViewCount="3199" Body="&lt;p&gt;A Generative Adversarial Network (GAN) consists of two sub-networks: (1) generator and (2) discriminator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does a discriminator should be able to do? Or more specifically, should it be able to distinguish (classify) a real object (for example a vector) from a generated one or should it be able to distinguish a set of generated vectors from a set of real vectors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to think that the second option is correct. However, if it is the case, how do we build a neural network that classifies a set of vectors instead of a vector?&lt;/p&gt;&#xA;" OwnerUserId="15384" LastActivityDate="2017-10-11T17:54:29.953" Title="What does discriminator of a GAN should do?" Tags="|machine-learning|neural-network|gan|generative-models|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23693" PostTypeId="2" ParentId="23687" CreationDate="2017-10-11T17:42:59.513" Score="2" Body="&lt;p&gt;The fast &lt;code&gt;hist&lt;/code&gt; mode is available in newer versions of XGBoost. You can find infos on &lt;a href=&quot;https://github.com/dmlc/xgboost/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/xgboost/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For small data sets (100k lines), you wont feel much gain in training speed.&lt;/p&gt;&#xA;" OwnerUserId="36977" LastActivityDate="2017-10-11T17:42:59.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23694" PostTypeId="2" ParentId="23692" CreationDate="2017-10-11T17:54:29.953" Score="4" Body="&lt;p&gt;The discriminator must classify individual elements as being fake (i.e. created by the generator) or real (i.e. taken from the training dataset). The discriminator generates labels (real/fake) for each element in the batch. The loss functions are computed based on those labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Elements are fed to the discriminator in batches of the same type (i.e. all elements in the batch are real or all elements in the batch are fake). This is because the fake data batch is directly generated by the generator as part of the same computational graph (i.e. the output of the generator is &#xA; directly connected to the input of the discriminator). This is so to be able to propagate gradients of the generator parameters through the discriminator.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-10-11T17:54:29.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23695" PostTypeId="2" ParentId="23690" CreationDate="2017-10-11T18:54:15.270" Score="3" Body="&lt;p&gt;Cross-entropy loss still works with probabilities in $[0,1]$ as well as $\{0,1\}$. Most importantly, $\hat{y} = y$ is still a stationary point (although it will not equal $0$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also the case that the possible improvement in loss (and immediate gradient) is larger for $\hat{y} = 0.99, y = 0.89$ than for $\hat{y} = 0.4, y = 0.5$. If you use a sigmoid output, then the gradient at the logit $\hat{y} - y$ still applies - the larger gradient of the loss function scales inversely to the lower gradient of the sigmoid at that point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in short, yes use binary cross-entropy loss for single-class probabilities, even when they are not strictly in $\{0,1\}$.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-11T18:54:15.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23696" PostTypeId="1" CreationDate="2017-10-11T22:13:45.883" Score="3" ViewCount="13103" Body="&lt;p&gt;I am trying to do a grid searching using the methodology that mentioned in this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt;. However, I found that &lt;code&gt;XGBClassifier().fit()&lt;/code&gt; is using much more memory than &lt;code&gt;xgboost.train&lt;/code&gt;. Does anyone know why? Is this related to sparse matrix?&lt;/p&gt;&#xA;" OwnerUserId="40343" LastActivityDate="2017-10-17T23:36:20.410" Title="XGBoost Fit vs Train" Tags="|python|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23697" PostTypeId="1" CreationDate="2017-10-11T22:45:47.400" Score="4" ViewCount="99" Body="&lt;p&gt;I am new to the field of machine learning and recently learned the basics and working out various algorithms in python using libraries such as pandas, numpy, matplotlib, scikitlearn , etc. &#xA;                    I started learning about working Bigdata by distributing it and using Apache Spark's library MLlib to load and apply algorithms on it. So is working with MLlib the only way on Spark or is there any other way to use pandas and other libraries on distributed data?&lt;/p&gt;&#xA;" OwnerUserId="40010" LastActivityDate="2018-06-09T06:50:46.567" Title="Is MLlib compulsory to work with distributed data?" Tags="|machine-learning|python|bigdata|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23698" PostTypeId="2" ParentId="23697" CreationDate="2017-10-11T23:36:11.663" Score="1" Body="&lt;p&gt;Yes. There are a million ways to engineer a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use pandas for data wrangling and distribute this task via parallel python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally use python for distributed - data acquisition, data wrangling, and algorithmic computation. This way i get the capability of Multi-machine, multi CPU, &amp;amp; multi GPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use python all they way, i recommend recording all the different options in terms of libraries, then order them by most active in development releases, or by whatever criterion you value the most.&lt;/p&gt;&#xA;" OwnerDisplayName="user40285" LastActivityDate="2017-10-11T23:36:11.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23703" PostTypeId="1" CreationDate="2017-10-12T09:05:14.340" Score="0" ViewCount="76" Body="&lt;p&gt;I am trying to build one risk calculator using diabetes patients comments on social media platform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have list of comments given by diabetes patients for their experience on diabetes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When any new user come on my platform, I collect what symptoms he is facing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I match user symtoms with comment data symtoms and try to calculate the risk score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;approach 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Numbers of comments in database having user entered symptoms) / (Total comments entered by diabetes patients)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;approach 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Numbers of comments in database having diabetes as a symptom and also user entered symptoms) / (Total number of comments in database having diabetes as a symptom)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any better approach to calculate the risk score using user entered symtoms and diabetes patient comment data? &lt;/p&gt;&#xA;" OwnerUserId="12448" LastActivityDate="2017-10-13T10:18:58.507" Title="diseases risk prediction using social media comments" Tags="|algorithms|probability|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23705" PostTypeId="2" ParentId="23703" CreationDate="2017-10-12T10:22:06.210" Score="1" Body="&lt;p&gt;You should also have comments/symptoms of a reference group of non-diabetes patients to be able to make a distinction yes/no diabetes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have users of which you know they don't have diabetes?&#xA;If not, &quot;Is often thirsty?&quot; will always map to diabetes, because you have no other data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-realistic example: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Suppose 99% of diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.&lt;/li&gt;&#xA;&lt;li&gt;Then suppose Suppose 99% of non-diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.&lt;/li&gt;&#xA;&lt;li&gt;There is a strong correlation between eating candy bars and having diabetes.&lt;/li&gt;&#xA;&lt;li&gt;Your data of only diabetes patients shows that both people who eat 10 candy bars and people who eat no candy bars develop diabetes.&lt;/li&gt;&#xA;&lt;li&gt;Data that includes non-diabetes patients will reveal that people that eat 0 candy bars are much more likely to not have diabetes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Or you should have comments and diabetes type data to be able to make a distinction diabetes type 1 or type 2.&#xA;&quot;How often do you use medication?&quot; is a strong indication between the two, but you'd need the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, expect a risk score with at least weighted numbers that can be either positive or negative, for weak or strong (inverse) correlation between an answer and indication for diabetes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot more advanced techniques (e.g. with neural networks) that will have better results, including finding correlations between answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are doing anything more serious than 'Go see the doctor' on &gt;50% of positive indicators, you should really do more research on data science models, but also on how to formulate the questions etc.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-13T10:18:58.507" LastActivityDate="2017-10-13T10:18:58.507" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23706" PostTypeId="1" CreationDate="2017-10-12T12:36:13.170" Score="2" ViewCount="691" Body="&lt;p&gt;I'm currently using anomaly score calculation methods which work on single dimensional data sets (Timestamp &amp;amp; Value). I would like to calculate the same single anomaly score for multidimensional data set (Timestamp &amp;amp; Value 1 &amp;amp; Value 2 &amp;amp; ..... &amp;amp; Value N). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method could compare all the values and calculate anomaly score based upon the relationship between them?&lt;/p&gt;&#xA;" OwnerUserId="40370" LastEditorUserId="29575" LastEditDate="2018-02-09T14:41:18.543" LastActivityDate="2018-02-09T14:41:18.543" Title="Anomaly score calculation for multidimensional data set" Tags="|python|anomaly-detection|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23707" PostTypeId="1" CreationDate="2017-10-12T12:44:27.850" Score="1" ViewCount="78" Body="&lt;p&gt;I've been reading a lot about computer vision lately, and while there is a huge amount of info about object classification, and a lot less on object detection, I have not found anything on class-based object detection i.e when I know what I am looking for. For example, looking for cats on a picture. Nowadays you can say if a picture is a cat (object classification), or if the picture has cats among other things. &#xA;How would I use this knowledge to improve the performance (in both accuracy and speed)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somethings that I wouldn't do (but i'm tempted to do):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Perform a brute-force solution with a binary classifier&lt;/li&gt;&#xA;&lt;li&gt;Implement a shape-based selective search and then use a binary classifier&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What i would really like to do, but haven't found in the literacy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Implement a trainable selective search to find class-based regions of interest. &lt;/li&gt;&#xA;&lt;li&gt;Use a binary classifier to check if ROI is in fact the object to be found. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40371" LastEditorUserId="29169" LastEditDate="2021-02-15T22:22:20.077" LastActivityDate="2021-02-15T22:22:20.077" Title="Is there something like class-based object detection? Or class-based selective search?" Tags="|computer-vision|convolutional-neural-network|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23708" PostTypeId="2" ParentId="22781" CreationDate="2017-10-12T12:53:30.013" Score="1" Body="&lt;p&gt;The second option sounds fun but better don't do it: errors will sum up and it's much harder to track where errors come from. Like you have some error for the 1st hour, feed it back in the net, predict the 2nd hour get an error there and can't really tell how much of that is caused by the error of the first hour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides that, I haven't seen any application which works with this feedback mechanism.&lt;/p&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-10-12T12:53:30.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23709" PostTypeId="1" AcceptedAnswerId="23711" CreationDate="2017-10-12T13:10:41.253" Score="0" ViewCount="668" Body="&lt;p&gt;Problem : Generate a text output based on input strings which will be combined using a number of rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Feature1               Feature2                    O/P&#xA;&#xA;Rule 1  Enum_Domain           Priority          /Enum_Domain/Priority&#xA;&#xA;Rule 2 Enum_Domain.EnumData   Name              /Enum_Domain/EnumData/Name&#xA;&#xA;Rule 1  Trunkgroup            Gateway            /Trunkgroup/Gateway&#xA;&#xA;Rule 2  GatewayGrp.Gateway    IP                /GatewayGrp/Gateway/IP&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a simple programming problem, but is there any machine learning algorithm that can learn these rules and generate the output based on the two inputs.&lt;/p&gt;&#xA;" OwnerUserId="40372" LastActivityDate="2017-10-12T13:22:21.850" Title="Can we use machine learning to generate a text output based on the input strings" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23710" PostTypeId="2" ParentId="22781" CreationDate="2017-10-12T13:19:17.877" Score="1" Body="&lt;p&gt;So at time $t$ we want to predict values $y_{t+1}$ up to $y_{t+6}$ correct? You should use $x_0$ up to $x_t$ as inputs and use 6 values as your target/output. Then when you get new information, you add $x_{t+1}$ and use it to update your cell state and hidden state of your LSTM and get new outputs. The problem with feeding predictions is that errors will accumulate, although this does happen with sequence generation models (like seq2seq). What you would do then is during training use two types of training, one where you use current predictions and one where you use your known training data, both a fraction of the time. This is called 'teacher forcing'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we use my first suggestion, here is an explanation how to prepare your data. To train your model you will need to use lags to prepare your targets. Let's say we have $x$ as one-dimensional and we are trying to predict the next 6 values of $x$, you would get as $x$ and $y$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| t | x   | y                     |&#xA;| 0 | 0   | [-1, 1, 2, 3, 2, 1]   |&#xA;| 1 | -1  | [1, 2, 3, 2, 1, 0]    |&#xA;| 2 | 1   | [2, 3, 2, 1, 0, -1]   |&#xA;| 3 | 2   | [3, 2, 1, 0, -1, -2]  |&#xA;| 4 | 3   | NaN                   |&#xA;| 5 | 2   | NaN                   |&#xA;| 6 | 1   | NaN                   |&#xA;| 7 | 0   | NaN                   |&#xA;| 8 | -1  | NaN                   |&#xA;| 9 | -2  | NaN                   |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last six steps don't have a target because it's not available in our training set. The loss function would be a (weighted) mean squared error over the six predictions per time step. If you really don't want to throwaway data, you could make a custom loss function that can use the fact that only 4 targets are available because we are near the end, but that will complicate things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: With regards to the question in the title, you would like to look back as far as possible, which basically means you always start at $t=0$.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-12T13:19:17.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23711" PostTypeId="2" ParentId="23709" CreationDate="2017-10-12T13:22:21.850" Score="1" Body="&lt;p&gt;Yes, sequence 2 sequence models attempt to do this. This can be used in a number of domains, from typo fixing to machine translation. They are encoder -&gt; decoder based, which means you have a part that encodes your input and then a decoder that generates a new sequence based on this encoding (and usually some attention). In this case your encoder would likely be two recurrent neural networks of which the output would be concatenated and then a decoder that takes this concatenated output and turns this into a new sequence. If you want to use attention you need to adapt the standard attention a bit because you have two textual inputs, but if you understand how it works this would not be too difficult to adapt.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-12T13:22:21.850" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23713" PostTypeId="1" AcceptedAnswerId="23719" CreationDate="2017-10-12T14:08:19.673" Score="1" ViewCount="242" Body="&lt;p&gt;I have time series $R$, which shows, how &lt;code&gt;something&lt;/code&gt; changes at the regional level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have several time series $U_i$, which show, how &lt;code&gt;something&lt;/code&gt; changes at a special unit $I$ level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many units in the region. $R$ has no missing data. Different $U_i$ have their own missing periods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I want to&lt;/strong&gt; forecast $U$ after a missing period using information of $R$ and information of $U$  when it was availible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My thoghts till now&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $R$ is known on interval $[0, 365]$. Suppose $U_i$ is known on interval $[0,300]$. Let's take $R$ and $U_i$ both on interval $[0,300]$, take difference between them and trying to predict that difference with linear regression. So for interval $[301,365]$, I will have differences and to restore $U_i$, I will just have to take out my differences from $R$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I don't like my solution&lt;/strong&gt;, because:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We need a model for each $U_i$. &lt;/li&gt;&#xA;&lt;li&gt;Because, sometimes data is more sparse and I don't even have a $[0,300]$ known interval, so not able to train regression properly.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29931" LastEditorUserId="29575" LastEditDate="2017-10-15T22:47:54.460" LastActivityDate="2017-10-15T22:47:54.460" Title="Forecasting one time series with missing data with help of other time series" Tags="|time-series|forecast|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23714" PostTypeId="2" ParentId="23713" CreationDate="2017-10-12T14:32:42.447" Score="0" Body="&lt;p&gt;Just a thought:&#xA;Why wouldn't you do the training on the difference data for 1-250 and test using 251-300 to see if the underlying pattern actually is accurate. If that was the case, you can generalize from 1-300 to 301-365.&lt;/p&gt;&#xA;" OwnerUserId="40378" LastActivityDate="2017-10-12T14:32:42.447" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23717" PostTypeId="2" ParentId="20421" CreationDate="2017-10-12T16:07:45.893" Score="0" Body="&lt;p&gt;I am sorry, but your description is lacking, you might want to follow this procedure when classifying a multi-label classifier.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 1: Make assumptions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Let's try and simplify the problem with the most basic multi-label classifier assumption: the labels are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way to tackle this problem is to classify $\{l_1,\dots,l_k\}$ labels by training $k$ classifiers independently, where classifier $C_i(x)=1$ iff $l_i\in Labels(x)$.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 2: Feature engineering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;We have no idea from your question what features are reasonable for your problem.&#xA;But applying logistic regression on $x_1$ and $x_2$ would not yield good results if their relation to the label is not linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could add any number of &quot;engineered feature&quot; such as $x_1*x_2,\frac{x_1}{x_y},\sqrt{(x_1-c_1)^2+(x_2-c_2)^2},\dots$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And bring prior knowledge to the model, for example, if $x_1$ and $x_2$ are longitude and latitude, What country / region do they represent ? climate ? earth-quake rate, etc....&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 3: Removing outliers and dont-cares&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The best way is to plot your model or apply clustering algorithms, and see what data points pull the model to undesired places/&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 4: decide on an objective function&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is a multi-label classification problem, so your input of &quot;0.0006&quot; accuracy doesn't tell me much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to measure multi-label performance, is the Jaccaard index&#xA;$$J(y_{true}, y_{pred})=\frac{|y_{pred}\cap y_{true}|}{|y_{pred}\cup y_{true}|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of other ways, and you may want to weigh the labels differently.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 5: Train the classifier&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Now you can train your logistic regression / NN model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several packages (e.g. &lt;a href=&quot;http://meka.sourceforge.net&quot; rel=&quot;nofollow noreferrer&quot;&gt;MEKA&lt;/a&gt;) that specialize on multi-label models.&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-12T16:07:45.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23718" PostTypeId="2" ParentId="23706" CreationDate="2017-10-12T16:18:39.193" Score="1" Body="&lt;p&gt;One very simple yet efficient way, is auto-regression, which means, you train a regressor on past data, and predict the future, if the prediction is off by too much - then you call it an anomaly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to make it more formal, if we denote $\vec{v}_{t0}$  as your data at timestamp $t_0$, we train a regression model $R$ that learns from $\{v_{t_0},\dots,v_{t_{n-1}}\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your anomaly score is a function of $R({v_{t_0},\dots,v_{t_{n-1}}})-v_{t_n}$&lt;/p&gt;&#xA;" OwnerUserId="12942" LastActivityDate="2017-10-12T16:18:39.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23719" PostTypeId="2" ParentId="23713" CreationDate="2017-10-12T16:46:06.573" Score="1" Body="&lt;p&gt;Not a fully complete answer, but some inputs.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your time series are correlated. &lt;/li&gt;&#xA;&lt;li&gt;I assume that the measure you want to forecast for a region is an aggregation of units forecasts.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To address the first point, I usually use Vector Autoregressive Model (VAR) that forecast all time-series at once (each one being expressed as a regression using the others)&#xA;The second point involves the concept of hierarchical forecasting and reconciliation. You can exploit the fact that the regional forecast should/must equal the unit- forecasts. There can be a process to adjust forecasts to take that into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are both packages for VaR and hierarchical reconciliation in R but as far as I know no direct code to handle both at the same time...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find this paper providing some details on the proposed approach:&#xA;&lt;a href=&quot;https://mpra.ub.uni-muenchen.de/76556/1/MPRA_paper_76556.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://mpra.ub.uni-muenchen.de/76556/1/MPRA_paper_76556.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18722" LastActivityDate="2017-10-12T16:46:06.573" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23720" PostTypeId="1" CreationDate="2017-10-12T17:00:16.400" Score="1" ViewCount="525" Body="&lt;p&gt;I saw the &lt;a href=&quot;http://labelme.csail.mit.edu/Release3.0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;labelme&lt;/a&gt; tool and was wondering if there is a similar tool for labelling short documents? Preferably in Python.&lt;/p&gt;&#xA;" OwnerUserId="40384" LastEditorUserId="29169" LastEditDate="2021-07-18T21:40:20.157" LastActivityDate="2022-12-28T04:21:43.007" Title="Are there any tools to make text labeling faster?" Tags="|python|nlp|labels|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="23721" PostTypeId="1" CreationDate="2017-10-12T17:03:35.873" Score="4" ViewCount="219" Body="&lt;p&gt;I've learned machine learning via textbooks and examples, which don't delve into the engineering challenges of working with &quot;big-ish&quot; data like Kaggle's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a specific example, I'm working on the &lt;a href=&quot;https://www.kaggle.com/c/nyc-taxi-trip-duration&quot; rel=&quot;nofollow noreferrer&quot;&gt;New York taxi trip challenge&lt;/a&gt;. It's a regression task for ~ 2 million rows and 20 columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My 4GB-RAM laptop can barely do EDA with &lt;code&gt;pandas&lt;/code&gt; and &lt;code&gt;matplotlib&lt;/code&gt; in Jupyter Notebook. However, when I try to build a random forest with 1000 trees, it hangs (e.g. Kernel restart error in Jupyter Notebook).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To combat this, I set up a 16GB-RAM desktop. I then ssh in, start a browser-less Jupyter Notebook kernel, and &lt;a href=&quot;http://amber-md.github.io/pytraj/latest/tutorials/remote_jupyter_notebook&quot; rel=&quot;nofollow noreferrer&quot;&gt;connect my local Notebook to that kernel&lt;/a&gt;. However, I still max out that machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point, I'm guessing that I need to run my model training code as a script. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Will this help with my machine hanging?&lt;/li&gt;&#xA;&lt;li&gt;What's the workflow to store the model results and use it later for prediction? Do you use a Makefile to keep this reproducible?&lt;/li&gt;&#xA;&lt;li&gt;Doing this also sacrifices the interactivity of Jupyter Notebook -- is there a workflow that maintains the interactivity?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My current toolkit is RStudio, Jupyter Notebook, Emacs, but am willing to pick up new things.&lt;/p&gt;&#xA;" OwnerUserId="9847" LastEditorUserId="9847" LastEditDate="2017-10-12T19:08:07.397" LastActivityDate="2021-06-06T19:03:05.273" Title="Avoid hardware limitation while competing in Kaggle?" Tags="|scikit-learn|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23722" PostTypeId="2" ParentId="23706" CreationDate="2017-10-12T17:12:08.963" Score="0" Body="&lt;p&gt;Assuming that your anomaly score is based on the Gaussian probability distribution, there's no reason why you could not extend it to be multivariate.&lt;/p&gt;&#xA;" OwnerUserId="40385" LastActivityDate="2017-10-12T17:12:08.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23723" PostTypeId="2" ParentId="23651" CreationDate="2017-10-12T17:17:28.143" Score="2" Body="&lt;p&gt;You could do whatever your heart desire, but unless your model predicts the temperature or time-difference, I cannot come up with any other target variable that depends solely on the coordinates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you probably want to do, is use an external data source and enrich your data with Country / Zip code / climate / other geographic features that will help your model perform.&lt;/p&gt;&#xA;" OwnerUserId="40385" LastActivityDate="2017-10-12T17:17:28.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23724" PostTypeId="2" ParentId="23651" CreationDate="2017-10-12T17:50:04.547" Score="0" Body="&lt;p&gt;GPS coordinates can be directly converted to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Geohash&quot; rel=&quot;nofollow noreferrer&quot;&gt;geohash&lt;/a&gt;. Geohash divides the Earth into &quot;buckets&quot; of different size based on the number of digits (short Geohash codes create big areas and longer codes for smaller areas). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A geohash is a single number that can be used as a feature in a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Geohash applies only to the entire world, zipcodes do not.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-10-12T17:50:04.547" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="23725" PostTypeId="1" CreationDate="2017-10-12T17:57:12.310" Score="1" ViewCount="477" Body="&lt;p&gt;I have built this network with pytorch, it's basically a modification of VGG16, I add some layers and remove some. The output of the network are meant to be softmax scores for &lt;strong&gt;F.multilabel-soft-margin-loss&lt;/strong&gt; in pytorch. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def __init__(self, bn=False):&#xA;super(WSDR, self).__init__()&#xA;# Network based on VGG16&#xA;# ConvReLU means a conv layer following a relu&#xA;self.conv1 = nn.Sequential(ConvReLU(3, 64, 3, pd=True, bn=bn),&#xA;                           ConvReLU(64, 64, 3, pd=True, bn=bn),&#xA;                           nn.MaxPool2d(2))&#xA;self.conv2 = nn.Sequential(ConvReLU(64, 128, 3, pd=True, bn=bn),&#xA;                           ConvReLU(128, 128, 3, pd=True, bn=bn),&#xA;                           nn.MaxPool2d(2))&#xA;self.conv3 = nn.Sequential(ConvReLU(128, 256, 3, pd=True, bn=bn),&#xA;                           ConvReLU(256, 256, 3, pd=True, bn=bn),&#xA;                           ConvReLU(256, 256, 3, pd=True, bn=bn),&#xA;                           nn.MaxPool2d(2))&#xA;self.conv4 = nn.Sequential(ConvReLU(256, 512, 3, pd=True, bn=bn),&#xA;                           ConvReLU(512, 512, 3, pd=True, bn=bn),&#xA;                           ConvReLU(512, 512, 3, pd=True, bn=bn),&#xA;                           nn.MaxPool2d(2))&#xA;self.conv5 = nn.Sequential(ConvReLU(512, 512, 3, pd=True, bn=bn),&#xA;                           ConvReLU(512, 512, 3, pd=True, bn=bn),&#xA;                           ConvReLU(512, 512, 3, pd=True, bn=bn))&#xA;&#xA;self.gap = nn.Sequential(ConvReLU(512, 1024, 3, pd=True, bn=bn),&#xA;                         ConvReLU(1024, 20, 3, pd=True, bn=bn),&#xA;                         nn.AvgPool2d(kernel_size=14, stride=14))&#xA;# adding this fc layer, things go right&#xA;# self.fc = nn.Linear(self.num_classes, self.num_classes)&#xA;&#xA;def forward(self, im_data):&#xA;    x = self.conv1(im_data)&#xA;    x = self.conv2(x)&#xA;    x = self.conv3(x)&#xA;    x = self.conv4(x)&#xA;    conv5features = self.conv5(x)&#xA;    gap = self.gap(conv5features)&#xA;    scores = gap.squeeze()&#xA;    # scores = self.fc(scores)&#xA;    return scores&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this network can not reduce loss correctly. The output of the network become all zeros after only 2 or 3 iterations(one forward and one backward means one iteration), so do the gradients in the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the problem may be caused by the last global average pooling layer, usually people add another fc layer after that but I didn't. I tried to add a fc, and the problem disappered, the loss start to reduce after adding fc layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What caused the vanishing gradient and zero output(gap2 feature, scores in the above code all become zeros)?&lt;/li&gt;&#xA;&lt;li&gt;Why adding a fc layer can solve that problem?&lt;/li&gt;&#xA;&lt;li&gt;I really don't want the fc layer because I want every channel of the last conv feature to represent one class(for classification), how can I make things right without adding fc layer?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please leave your advice, any hints would be helpful&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addtional codes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class ConvReLU(nn.Module):&#xA;def __init__(self, in_ch, out_ch, kernel_sz, stride=1, relu=True, pd=True, bn=False):&#xA;    super(ConvReLU, self).__init__()&#xA;    padding = int((kernel_sz - 1) / 2) if pd else 0  # same spatial size by default&#xA;    self.conv = nn.Conv2d(in_ch, out_ch, kernel_sz, stride, padding=padding)&#xA;    self.bn = nn.BatchNorm2d(out_ch, eps=0.001, momentum=0, affine=True) if bn else None&#xA;    self.relu = nn.ReLU(inplace=True) if relu else None&#xA;&#xA;def forward(self, x):&#xA;    x = self.conv(x)&#xA;    if self.bn is not None:&#xA;        x = self.bn(x)&#xA;    if self.relu is not None:&#xA;        x = self.relu(x)&#xA;    return x&#xA;&#xA;# compute output&#xA;output = model(input_var)  # the above model, input_var:image data&#xA;loss = F.multilabel_soft_margin_loss(output, target_var)&#xA;# compute gradient and do SGD step&#xA;optimizer.zero_grad()&#xA;loss.backward()&#xA;optimizer.step()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35908" LastEditorUserId="35908" LastEditDate="2017-10-15T05:47:51.573" LastActivityDate="2017-10-15T05:47:51.573" Title="Global average polling without fc layer, Vanishing gradient or other problem?" Tags="|machine-learning|classification|deep-learning|gradient-descent|pytorch|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23726" PostTypeId="1" AcceptedAnswerId="23727" CreationDate="2017-10-12T18:15:04.360" Score="3" ViewCount="13198" Body="&lt;p&gt;Does the fact that I have linearly separable data or not impact the convergence of the perceptron algorithm?&#xA;Is it always gonna converge if the data is linearly separable and not if it is not ? Is there a general rule ?&lt;/p&gt;&#xA;" OwnerUserId="40388" LastActivityDate="2017-10-17T21:10:38.957" Title="Is the percepetron algorithm's convergence dependent on the linearity of the data?" Tags="|perceptron|convergence|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23727" PostTypeId="2" ParentId="23726" CreationDate="2017-10-12T19:00:16.027" Score="5" Body="&lt;p&gt;Yes, the perceptron learning algorithm is a linear classifier. If your data is separable by a hyperplane, then the perceptron will always converge. It will never converge if the data is not linearly separable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, the perceptron learning algorithm can be used on data that is not linearly separable, but some extra parameter must be defined in order to determine under what conditions the algorithm should stop 'trying' to fit the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you could set a maximum number of iterations (or epochs) for the algorithm to run, or you could set a threshold for the maximum number of allowed misclassifications.&lt;/p&gt;&#xA;" OwnerUserId="40391" LastEditorUserId="40391" LastEditDate="2017-10-17T21:10:38.957" LastActivityDate="2017-10-17T21:10:38.957" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23728" PostTypeId="1" AcceptedAnswerId="23877" CreationDate="2017-10-12T19:54:52.020" Score="13" ViewCount="5298" Body="&lt;p&gt;Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it better at preventing overfitting? Is it deterministic (so always a unique solution)? Is it better at feature selection (because producing sparse models)? Does it disperse the weights among the features?&lt;/p&gt;&#xA;" OwnerUserId="40388" LastActivityDate="2017-10-17T19:18:17.883" Title="Why using L1 regularization over L2?" Tags="|linear-regression|regularization|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23729" PostTypeId="1" CreationDate="2017-10-12T20:00:18.570" Score="1" ViewCount="7512" Body="&lt;p&gt;How do I use the same scale used in preprocessing with new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actual code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = df.values #returns a numpy array&#xA;min_max_scaler = preprocessing.MinMaxScaler()&#xA;x_scaled = min_max_scaler.fit_transform(x)&#xA;df_scaled = pd.DataFrame(x_scaled)&#xA;&#xA;clf = tree.DecisionTreeClassifier()&#xA;clf.fit(X_train, y_train)&#xA;pred = clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I understand it correctly I should have included a scaler variable with the StandardScaler.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/38780302/predicting-new-data-using-sklearn-after-standardizing-the-training-data&quot;&gt;https://stackoverflow.com/questions/38780302/predicting-new-data-using-sklearn-after-standardizing-the-training-data&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = tree.DecisionTreeClassifier()&#xA;clf.fit(X_train, y_train)&#xA;scaler = preprocessing.StandardScaler().fit(X_train)&#xA;pred = clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What scaler parameters should I use for future data processing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="39949" LastEditorUserId="39949" LastEditDate="2017-10-12T21:04:47.743" LastActivityDate="2017-10-12T21:04:47.743" Title="How to use the same scale with new data? - scikit learn - scikit learn" Tags="|python|scikit-learn|preprocessing|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23730" PostTypeId="2" ParentId="23729" CreationDate="2017-10-12T20:23:59.303" Score="3" Body="&lt;p&gt;IMO, you don't need to use scaling if your classifiers are based on decision trees. Also, in your final piece of code, the variable &lt;code&gt;scaler&lt;/code&gt; is never used, so I am not sure at all why it is defined. Nevertheless, if you insist on using a scaler, you should &lt;code&gt;Pipeline&lt;/code&gt; it so it automatically applies to the test data the same scaling it has learned by fitting to training data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipeline = Pipeline([('scaler', StandardScaler()), ('classifier', DecisionTreeClassifier())])&#xA;pipeline.fit(X_train, y_train)&#xA;predictions = pipeline.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that you'd obviously need to &lt;code&gt;import&lt;/code&gt; the necessary modules (I was too lazy to write them here!).&lt;/p&gt;&#xA;" OwnerUserId="21433" LastActivityDate="2017-10-12T20:23:59.303" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23731" PostTypeId="1" CreationDate="2017-10-12T20:57:37.467" Score="1" ViewCount="135" Body="&lt;p&gt;I'm trying to extract NPs from transcribed spoken text, such as &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;um it's the bl- it's the blue one in the right no left hand corner&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which contains e.g. fillers (e.g. &lt;em&gt;um&lt;/em&gt;) and disfluencies (e.g. &lt;em&gt;bl-&lt;/em&gt;, &lt;em&gt;right no left hand corner&lt;/em&gt;) that are not commonly seen in written text. Ideally, I'd like to get something like the three sequences &lt;em&gt;it&lt;/em&gt;, &lt;em&gt;the blue one&lt;/em&gt; and &lt;em&gt;the left hand corner&lt;/em&gt; (or at the very least &lt;em&gt;the right no left hand corner&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using Stanford CoreNLP's pre-trained &lt;a href=&quot;https://nlp.stanford.edu/software/srparser.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;shift-reduce parser&lt;/a&gt; with a &lt;a href=&quot;https://nlp.stanford.edu/software/srparser.shtml#beam&quot; rel=&quot;nofollow noreferrer&quot;&gt;beam size of 4&lt;/a&gt; (&lt;code&gt;englishSR.beam.ser.gz&lt;/code&gt;) and bidirectional &lt;a href=&quot;https://nlp.stanford.edu/software/tagger.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;dependency network POS tagging&lt;/a&gt; (&lt;code&gt;english-bidirectional-distsim.tagger&lt;/code&gt;) after filtering out fillers and duplicated tokens (e.g. &lt;em&gt;uh it's it's that one&lt;/em&gt; &amp;rarr; &lt;em&gt;it's that one&lt;/em&gt;). This performs okay but seems to fail a lot more than I'd expect; &lt;strong&gt;Are there no chunkers or (shallow) parsers widely available which are tailored specifically to spoken English as opposed to written English?&lt;/strong&gt; The language the chunker/parser is written in is irrelevant (i.e. it needn't have a Java API). I've also tried using Stanford CoreNLP's caseless models, which actually seem to perform a bit worse (however, I haven't done any rigorous comparisons).&lt;/p&gt;&#xA;" OwnerUserId="40357" LastActivityDate="2023-12-27T09:02:29.727" Title="Chunker/shallow parser for spoken language" Tags="|nlp|preprocessing|parsing|stanford-nlp|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23733" PostTypeId="2" ParentId="6513" CreationDate="2017-10-12T23:30:07.010" Score="1" Body="&lt;p&gt;Like Emre said, if you have a certain type I and type II error rates that need to be met, you can consider the Sequential Probability Ratio Test or other methods like Distilled Sensing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were conducting the A/B test in an online fashion, i.e. decisions need to be made for each location as they arrive, then you can consider some online FDR procedures (generalized alpha-investing methods) like LORD, LOND, $\alpha$-investing or spending and GAI++.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bonferroni type corrections are for FWER control and tend to be overly conservative.&lt;/p&gt;&#xA;" OwnerUserId="40399" LastActivityDate="2017-10-12T23:30:07.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23734" PostTypeId="1" AcceptedAnswerId="23735" CreationDate="2017-10-13T00:02:32.203" Score="5" ViewCount="2428" Body="&lt;p&gt;Could someone please explain to me why the autoencoder is not converging? To me the results of the two networks below should be the same. However, the autoencoder below is not converging, whereas, the network beneath it is.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;autoencoder implementation, does not converge&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;autoencoder = Sequential()&#xA;encoder = containers.Sequential([Dense(32,16,activation='tanh')]) &#xA;decoder = containers.Sequential([Dense(16,32)])&#xA;autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, &#xA;                        output_reconstruction=True))&#xA;rms = RMSprop()&#xA;autoencoder.compile(loss='mean_squared_error', optimizer=rms)&#xA;&#xA;autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,&#xA;            validation_data=(testData, testData), show_accuracy=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;non-autoencoder implementation, converges&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(32,16,activation='tanh')) &#xA;model.add(Dense(16,32))&#xA;model.compile(loss='mean_squared_error', optimizer=rms)&#xA;&#xA;model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size,&#xA;            validation_data=(testData, testData), show_accuracy=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40400" LastActivityDate="2017-10-13T00:12:04.327" Title="Keras autoencoder not converging" Tags="|deep-learning|keras|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23735" PostTypeId="2" ParentId="23734" CreationDate="2017-10-13T00:12:04.327" Score="3" Body="&lt;p&gt;The new version (0.3.0) of Keras no longer has tied weights in AutoEncoder, and it still shows different convergence. This is because weights are initialized differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the non-AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32). In the AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32), and then when you create the AutoEncoder instance, Dense(32,16) weights are initialized again (self.encoder.set_previous(node) will call build() to initialize weights).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the following two NNs converge exactly the same:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;autoencoder = Sequential()&#xA;encoder = containers.Sequential([Dense(32,16,activation='tanh')]) &#xA;decoder = containers.Sequential([Dense(16,32)])&#xA;autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, &#xA;                        output_reconstruction=True))&#xA;rms = RMSprop()&#xA;autoencoder.compile(loss='mean_squared_error', optimizer=rms)&#xA;np.random.seed(0)&#xA;autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,&#xA;            validation_data=(testData, testData), show_accuracy=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40384" LastActivityDate="2017-10-13T00:12:04.327" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23736" PostTypeId="1" CreationDate="2017-10-13T00:28:14.627" Score="5" ViewCount="2040" Body="&lt;p&gt;I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E.g., first newspaper dataset&#xA;article_1 = {'politics': 0.1, 'nature': 0.8, ..., 'sports':0, 'wild-life':1}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again, I train my second newspaper dataset (from a different distributor) to get the topics per each newspaper article.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;E.g., second newspaper dataset (from a different distributor)&#xA;article_2 = {'people': 0.3, 'animals': 0.7, ...., 'business':0.7, 'sports':0.2}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As shown in the examples, the topics I get from the two datasets are different, thus I manually matched similar topics based on their frequent words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to identify whether the two newspaper distributors publish the same news in every week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I am interested in knowing  if there is a systematic way of comparing the topics across two corpora and measuring their similarity. Please help me.&lt;/p&gt;&#xA;" OwnerUserId="33838" LastEditorUserId="33838" LastEditDate="2017-10-15T23:48:18.120" LastActivityDate="2017-10-19T20:10:57.737" Title="Compare two topic modelling sets" Tags="|machine-learning|python|data-mining|text-mining|topic-model|" AnswerCount="2" CommentCount="21" ContentLicense="CC BY-SA 3.0" />
  <row Id="23738" PostTypeId="2" ParentId="20403" CreationDate="2017-10-13T04:50:04.503" Score="2" Body="&lt;p&gt;It seems that you are training on a very small dataset. A dataset of size 10000 samples will bring out less than 5k words after preprocessing, which I think isn't enough for your model to learn the relations. Try increasing your dataset and then train. Also, you can tune your parameters while training. Try changing window size (default is 5), embedding size and min count. Generally, 100 is a good embedding size, but you can tweak it and see what gives you better results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try using already trained word2vec model released by Google. It's 1.5GB and includes word vectors for a vocabulary of 3 million words and phrases that they trained on roughly 100 billion words from a Google News dataset.&lt;/p&gt;&#xA;" OwnerUserId="39934" LastEditorUserId="29575" LastEditDate="2017-10-13T05:01:18.517" LastActivityDate="2017-10-13T05:01:18.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23739" PostTypeId="1" AcceptedAnswerId="23744" CreationDate="2017-10-13T05:25:23.793" Score="20" ViewCount="10219" Body="&lt;p&gt;I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the purpose of dimension reduction or visualizing clusters in high dimensional data, we can use an autoencoder to create a (lossy) 2 dimensional representation by inspecting the output of the network layer with 2 nodes. For example, with the following architecture, we would inspect the output of the third layer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$[X] \rightarrow N_1=100 \rightarrow N_2=25 \rightarrow (N_3=2) \rightarrow N_4=25 \rightarrow N_5=100 \rightarrow [X]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X$ is the input data and $N_l$ is the number of nodes in the $l$th layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my question is, why do we want a symmetrical architecture? Doesn't a mirror of the deep 'compression' phase mean we might have a similarly complex 'decompression' phase resulting in a 2 node output which is not forced to be very intuitive? In other words, wouldn't having a simpler decoding phase result in the output of the layer with 2 nodes necessarily being simpler too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thinking here is that the less complex the decompression phase, the simpler (more linear?) the 2D representation has to be. A more complex decompression phase would allow a more complex 2D representation.&lt;/p&gt;&#xA;" OwnerUserId="37475" LastActivityDate="2021-03-11T20:10:32.847" Title="Why are autoencoders for dimension reduction symmetrical?" Tags="|neural-network|dimensionality-reduction|autoencoder|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23740" PostTypeId="1" AcceptedAnswerId="23741" CreationDate="2017-10-13T07:14:24.067" Score="0" ViewCount="764" Body="&lt;p&gt;my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use mse as the loss function,but the value of the loss always be 0 my code as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(cols*footsize, 400,init = &quot;glorot_uniform&quot;))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(400, 400,init = &quot;glorot_uniform&quot;))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(400, 50,init = &quot;glorot_uniform&quot;))&#xA;model.add(Activation('softmax'))&#xA;model.compile(loss='mse', optimizer='rmsprop')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And regardless of the number of epochs, the loss is always 0&lt;/p&gt;&#xA;" OwnerUserId="40400" LastActivityDate="2017-10-13T07:21:54.243" Title="Loss of MSE always be 0 when keras for topic predict" Tags="|deep-learning|keras|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23741" PostTypeId="2" ParentId="23740" CreationDate="2017-10-13T07:21:54.243" Score="1" Body="&lt;p&gt;First, is your output a one-hot vector of predicted classes? i.e: class one is &lt;code&gt;[1, 0, 0, ...]&lt;/code&gt; and class two is &lt;code&gt;[0, 1, 0, 0, ...]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, then using softmax activation at the output layer is acceptable and you are doing a classification problem. If you are doing a classification problem (one-hot output) you cannot use MSE as the loss, use categorical cross-entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Softmax scales the output so that the number given is a predicted probability of a certain class. Wikipedia here: &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Softmax_function&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are expecting the output vector to be real numbers then you need to use linear activation on your output neurons.&lt;/p&gt;&#xA;" OwnerUserId="40407" LastActivityDate="2017-10-13T07:21:54.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23742" PostTypeId="2" ParentId="23739" CreationDate="2017-10-13T07:32:18.757" Score="9" Body="&lt;p&gt;Your question is definitely in place, however I found that any question in the format of &quot;should I do X or Y in deep learning ?&quot; has only one answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Try them both&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep learning is a very empirical field, and if a non-symmetric auto-encoder works for your domain, then use it (and publish a paper)&lt;/p&gt;&#xA;" OwnerUserId="40407" LastActivityDate="2017-10-13T07:32:18.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23744" PostTypeId="2" ParentId="23739" CreationDate="2017-10-13T07:57:43.200" Score="20" Body="&lt;p&gt;There is no specific constraint on the symmetry of an autoencoder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the &lt;strong&gt;weights&lt;/strong&gt; of the layers in the encoder and decoder where &lt;strong&gt;shared&lt;/strong&gt;. This is not a requirement, but it &lt;a href=&quot;https://www.quora.com/Is-weight-sharing-required-for-an-autoencoder/answer/Ian-Goodfellow?srid=ikXo&quot; rel=&quot;noreferrer&quot;&gt;allows to use certain loss functions (i.e. RBM score matching)&lt;/a&gt; and can act as &lt;a href=&quot;https://stats.stackexchange.com/a/154867/40048&quot;&gt;regularization&lt;/a&gt;, as you effectively reduce by half the number of parameters to optimize. Nowadays, however, I think no one imposes encoder-decoder weight sharing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About architectural symmetry, it is &lt;strong&gt;common&lt;/strong&gt; to find the same number of layers, the same type of layers and the same layer sizes in encoder and decoder, but &lt;strong&gt;there is no need&lt;/strong&gt; for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, in convolutional autoencoders, in the past it was very common to find convolutional layers in the encoder and &lt;em&gt;deconvolutional&lt;/em&gt; layers in the decoder, but now you normally see upsampling layers in the decoder because they have less artefacts problems.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-10-13T11:51:15.320" LastActivityDate="2017-10-13T11:51:15.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23745" PostTypeId="1" AcceptedAnswerId="23746" CreationDate="2017-10-13T08:17:35.557" Score="-1" ViewCount="439" Body="&lt;p&gt;I would like to implement a neural network allow to make captcha recognition. Actually, I'm new in deep learning that's the first neural network I'm building. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen a another similar project on Github : &lt;a href=&quot;https://deepmlblog.wordpress.com/2016/01/03/how-to-break-a-captcha-system/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://deepmlblog.wordpress.com/2016/01/03/how-to-break-a-captcha-system/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not able to understand it. I don't know what's a vgg base neural network model, etc ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a course or courses that I could follow in order to be able to implement such a neural network. I don't know where to start ... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't aim to become an expert for the moment, I just want to first concretely discover this science that seems exciting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help&lt;/p&gt;&#xA;" OwnerUserId="40410" LastActivityDate="2017-10-13T08:36:50.623" Title="Breaking captcha with a neural network - Learning deep learning" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-19T15:14:02.203" ContentLicense="CC BY-SA 3.0" />
  <row Id="23746" PostTypeId="2" ParentId="23745" CreationDate="2017-10-13T08:36:50.623" Score="0" Body="&lt;p&gt;My recommondation is a course from Stanford, &lt;a href=&quot;http://cs231n.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Convolutional Neural Networks for Visual Recognition&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But: This will not provide you easy to understand code to break captchas. You will learn the concept of Convolutional Neural Networks (CNN), which are popular in the classification of images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a reminder: Captchas are built to prevent computers from automatically filling them in. Even if nowadays techniques might be able to solve them, your project might be a bad idea (especially from the legal perspective).&lt;/p&gt;&#xA;" OwnerUserId="15946" LastActivityDate="2017-10-13T08:36:50.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23747" PostTypeId="2" ParentId="23615" CreationDate="2017-10-13T09:04:22.370" Score="10" Body="&lt;p&gt;Multivariate time series is an active research topic you will find a lot of recent paper tackling the subject. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your questions, you can use a single RNN. You can input one value for each time step. Nothing keeps you from adding another value at each time step (if your sensor are synchronized). Your model will then learn how to classify with a two dimensional time series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You check &lt;a href=&quot;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&quot; rel=&quot;noreferrer&quot;&gt;this blog&lt;/a&gt;. In your case, only the output is different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the two last points, aggregating the time series into one is risky in the sense that you might lose important information during the process. Finally the main disadvantage of your last point is that you won't be able to use a potential correlation between the two time series for the final classification.&lt;/p&gt;&#xA;" OwnerUserId="40413" LastActivityDate="2017-10-13T09:04:22.370" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23748" PostTypeId="2" ParentId="5178" CreationDate="2017-10-13T09:22:52.507" Score="0" Body="&lt;p&gt;You may take a look at my project called DOT:&#xA;Distrubuted Object Tracker repository manager.&lt;br&gt;&#xA;It is a very simple VCS for binary files for personal use (no collaboration).&lt;br&gt;&#xA;It uses SHA1 for checksuming and block deduplication. Full P2P syncing.&lt;br&gt;&#xA;One unique feature: adhoc one time TCP server for pull/push.&lt;br&gt;&#xA;It can also use SSH for transport.&lt;br&gt;&lt;br&gt;&#xA;It is not yet released, but might be a good starting point.&lt;br&gt;&#xA;&lt;a href=&quot;http://borg.uu3.net/cgit/cgit.cgi/dot/about/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://borg.uu3.net/cgit/cgit.cgi/dot/about/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40415" LastEditorUserId="14372" LastEditDate="2017-10-15T22:48:01.150" LastActivityDate="2017-10-15T22:48:01.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23750" PostTypeId="1" AcceptedAnswerId="23751" CreationDate="2017-10-13T11:12:31.830" Score="1" ViewCount="1407" Body="&lt;p&gt;I am reading &lt;a href=&quot;https://arxiv.org/pdf/1506.02025.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Convolutional Neural Networks define an exceptionally powerful class of models,&#xA;  but are still limited by the lack of ability to be spatially invariant to the input data&#xA;  in a computationally and parameter efficient manner. In this work we introduce a&#xA;  new learnable module, the Spatial Transformer, which explicitly allows the spa-&#xA;  tial manipulation of data within the network. This differentiable module can be&#xA;  inserted into existing convolutional architectures, giving neural networks the abil-&#xA;  ity to actively spatially transform feature maps, conditional on the feature map&#xA;  itself, without any extra training supervision or modification to the optimisation&#xA;  process. We show that the use of spatial transformers results in models which&#xA;  learn invariance to translation, scale, rotation and more generic warping, result-&#xA;  ing in state-of-the-art performance on several benchmarks, and for a number of&#xA;  classes of transformations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Spatial Transformers are used in &lt;code&gt;CNNs&lt;/code&gt; to have spatial invariant transformations and consequently the process of learning would be so easier and the networks would have better performance for data with different kind of distributions (noisy data). In this paper I don't realize the meaning of &lt;code&gt;differentiable module&lt;/code&gt;. These so called differentiable modules are used in neural networks. But what is the meaning of &lt;em&gt;differentiable&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-13T12:36:13.583" LastActivityDate="2017-11-13T12:36:13.583" Title="What are differentiable modules used in deep learning" Tags="|machine-learning|deep-learning|cnn|tranformation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23751" PostTypeId="2" ParentId="23750" CreationDate="2017-10-13T11:25:02.933" Score="3" Body="&lt;p&gt;&quot;Differentiable&quot; means that you can compute the derivative of the operations in the module, and therefore you can compute the gradients of the loss function with respect to the module parameters (i.e. use backpropagation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is normally a requirement for operations involved in neural network computations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: you &lt;strong&gt;can&lt;/strong&gt; use non-differentiable operations as part of a computational graph, but you won't be able to backpropagate gradients through them, and therefore any learnable parameters involved in operations prior to the non-differentiable one would not be able to learn.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-10-13T11:25:02.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23754" PostTypeId="1" CreationDate="2017-10-13T12:08:49.437" Score="7" ViewCount="2319" Body="&lt;p&gt;Are there any tools to monitor network's training in PyTorch?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an equivalent to tensorboard in tensorflow.&lt;/p&gt;&#xA;" OwnerUserId="40424" LastActivityDate="2019-01-11T20:05:51.107" Title="Any pytorch tools to monitor neural network's training?" Tags="|deep-learning|tensorflow|pytorch|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23755" PostTypeId="2" ParentId="23728" CreationDate="2017-10-13T12:13:17.753" Score="-1" Body="&lt;p&gt;L2 has one very important advantage to L1, and that is invariance to rotation and scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is especially important in geographical / physical application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say your technician accidentally installed your sensor in 45 degree angel, L1 would be affected, while L2 (Euclidean distance) would remain the same.&lt;/p&gt;&#xA;" OwnerUserId="40424" LastActivityDate="2017-10-13T12:13:17.753" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23756" PostTypeId="2" ParentId="23754" CreationDate="2017-10-13T12:24:24.007" Score="4" Body="&lt;p&gt;I am using &lt;a href=&quot;https://github.com/lanpa/tensorboardX&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorboardX&lt;/a&gt;. It supports most of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. Haven't tried the rest like audio and graph. But the repo also contains examples for those usecases. Installation can be done easily with pip. It's all explained in the readme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also other software which implement a wrapper for PyTorch (and other languages/frameworks) of TensorBoard. As far as I know, they support fewer functionalities. But have a look at &lt;a href=&quot;https://github.com/torrvision/crayon&quot; rel=&quot;nofollow noreferrer&quot;&gt;Crayon&lt;/a&gt; and &lt;a href=&quot;https://github.com/TeamHG-Memex/tensorboard_logger&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorboard-Logger&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="40400" LastEditorDisplayName="user10640" LastEditDate="2019-01-11T20:05:51.107" LastActivityDate="2019-01-11T20:05:51.107" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23757" PostTypeId="1" CreationDate="2017-10-13T13:47:46.980" Score="0" ViewCount="594" Body="&lt;p&gt;I am currently involved in analyzing a particular dataset called &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Haberman%27s+Survival&quot; rel=&quot;nofollow noreferrer&quot;&gt;Haberman Survival Dataset&lt;/a&gt;.&#xA;The dataset contains cases from a study that was conducted between 1958 and 1970 at the University of Chicago's Billings Hospital on the survival of patients who had undergone surgery for breast cancer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Attribute Information:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Age of patient at time of operation (numerical) &lt;/li&gt;&#xA;&lt;li&gt;Patient's year of operation (year - 1900, numerical) &lt;/li&gt;&#xA;&lt;li&gt;Number of positive axillary nodes detected (numerical) &lt;/li&gt;&#xA;&lt;li&gt;Survival status (class attribute),&#xA; 1 = the patient survived 5 years or longer,&#xA; 2 = the patient died within 5 year&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For this dataset I have first processed/cleaned the dataset using R, and now I am planning to collect a sample from this population and then train the sample data (training set) with some suitable machine learning algorithm and build a suitable model (1-3 are predictor variables, 4 is the response variable), and then again collect the sample (test set) from the population and then apply the model that I have built to test set and predict the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What machine learning models can be used to achieve this objective?&lt;/p&gt;&#xA;" OwnerUserId="39503" LastEditorUserId="13748" LastEditDate="2017-10-13T15:45:46.277" LastActivityDate="2017-10-13T15:45:46.277" Title="Predicting survival on Haberman's dataset" Tags="|machine-learning|r|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23759" PostTypeId="2" ParentId="23757" CreationDate="2017-10-13T14:08:09.177" Score="1" Body="&lt;p&gt;You can use any classification algorithm. Read up on Logistic Regression and Support Vector Machines, if you have sufficient data, you can try neural networks. The threshold score is of extreme importance here for good precision and recall. So lookup, precision-recall curves and ROC curves which help in determining a reasonable threshold. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-10-13T14:08:09.177" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="23761" PostTypeId="1" CreationDate="2017-10-13T15:08:24.227" Score="-3" ViewCount="1875" Body="&lt;p&gt;Given a csv of the form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x, y, annotation&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can we efficiently (least code) generate an interactive (zoom/pan) scatterplot where the annotation appears when mousing-over or clicking on an individual point?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very helpful when evaluating clusterings.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2017-10-13T19:31:59.820" LastActivityDate="2017-10-13T22:36:32.973" Title="Fast, efficient way to zoom/pan scatterplot with mouseover data" Tags="|python|r|clustering|visualization|javascript|" AnswerCount="2" CommentCount="2" ClosedDate="2017-10-19T15:13:42.990" ContentLicense="CC BY-SA 3.0" />
  <row Id="23764" PostTypeId="1" CreationDate="2017-10-13T18:21:50.197" Score="0" ViewCount="7244" Body="&lt;p&gt;I know there is a partial answer &lt;a href=&quot;https://stats.stackexchange.com/questions/65893/maximal-depth-of-a-decision-tree&quot;&gt;here&lt;/a&gt; but my question is slightly different.&#xA;I have implemented a decision tree in sklearn. Say I have $2^n$ different values for a feature, with just &lt;strong&gt;one&lt;/strong&gt; feature. I was expecting that a good decision tree should be able to keep the depth at or below $n$. Let's say the $2^n$ values for the features are indexed ($1 \dots 2^n$), if I make a split in the middle, that is the most effective way, and I can get a leaf with just one element at the end of a a n-deep tree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, it could be less, if, for example, I only have two categories, and all the elements below $2^{n-1}$ are in one class, and the other elements in the other class. But, in the worst case, I thought I should have depth $n$. However, when I implemented a solution for just around 1 million examples with one feature that has about 100,000 values, I got a tree with depth 125! (and I was expecting about 17).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is that? &lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2017-10-13T19:08:13.573" Title="Max depth for a decision tree in sklearn" Tags="|python|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23765" PostTypeId="2" ParentId="23713" CreationDate="2017-10-13T18:26:45.800" Score="-1" Body="&lt;p&gt;Looks like casual Impact library could help you with that&#xA;&lt;a href=&quot;http://google.github.io/CausalImpact/CausalImpact.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://google.github.io/CausalImpact/CausalImpact.html&lt;/a&gt;&#xA;Or a at least principles from it&lt;/p&gt;&#xA;" OwnerUserId="40443" LastActivityDate="2017-10-13T18:26:45.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23766" PostTypeId="2" ParentId="23764" CreationDate="2017-10-13T19:08:13.573" Score="2" Body="&lt;p&gt;There is an assumption in the statement &lt;code&gt;if I make a split in the middle&lt;/code&gt;. The assumption is the tree will always split at the middle value = the median &lt;strong&gt;value&lt;/strong&gt; so half the values go right and the other half go left. The split could be anywhere. And importantly the split focuses on the number of records at each value (think histogram). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets take the 100K values and 1m records. The root split could be, the records with the max value goes right, all other records go left. Then the next split on the left says the records with the max value of those 99,999 values go right, all other records go left. And so on. In this case it peels off one value, and all records with that value, at each split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is an extreme example but it illustrates the point. The split could be the min, max or anywhere in between. The number of records at each value is what is important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can research the answer to your tree by &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html#sklearn.tree.export_graphviz&quot; rel=&quot;nofollow noreferrer&quot;&gt;exporting&lt;/a&gt; the tree which will show you the splits and the split statistic for each node. &lt;/p&gt;&#xA;" OwnerUserId="28780" LastActivityDate="2017-10-13T19:08:13.573" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23769" PostTypeId="2" ParentId="23761" CreationDate="2017-10-13T19:31:15.517" Score="3" Body="&lt;p&gt;Here's a way to do it in Python, in a Jupyter notebook, with &lt;a href=&quot;https://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;bokeh&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from bokeh.plotting import figure, output_notebook, show&#xA;from bokeh.models.sources import ColumnDataSource&#xA;from bokeh.models import HoverTool, BoxZoomTool, PanTool, WheelZoomTool&#xA;&#xA;import pandas as pd&#xA;&#xA;df = pd.read_csv(&quot;your.csv&quot;)&#xA;source = ColumnDataSource(df_plot)&#xA;hover = HoverTool(tooltips=[(&quot;info&quot;, &quot;@annotation&quot;)])&#xA;p = figure(plot_width=400, plot_height=400, tools=[hover, BoxZoomTool(), PanTool(), WheelZoomTool()])&#xA;p.circle('x', 'y', size=7, source=source)&#xA;&#xA;output_notebook()&#xA;show(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives you an interactive, exploratory chart right in the notebook.&lt;/p&gt;&#xA;" OwnerUserId="1077" LastActivityDate="2017-10-13T19:31:15.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23771" PostTypeId="1" CreationDate="2017-10-13T21:06:17.247" Score="1" ViewCount="347" Body="&lt;p&gt;I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that I have a scoring function, S(i,n) that takes and item, i, and gives is a score of, say, 0-1, where n is some sort of parameter that we'll get back to in a moment. I can run this on a set of items, i*, and thereby (ignoring ties) rank these items S(i*,n1) -&gt; {s1, s2, s3 ...} where these scores correspond to the items, and so can be ordered by simple sorting to rank the items. (It's possible that the actual scores are enough, that we don't need the rank ordering, in what is to follow.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The scoring function depends on n, so S(i*,n1) usually != S(i*,n2) (Again, ignoring the unusual case of a tie.) What I'm interested in is measuring the &quot;information utility&quot; of the ns. So, for example, in the (neglected by definition) case where so S(i*,n1) == S(i*,n2), n2 provide no additional information over n1 (and v.v.), but in the more common case where S(i*,n1) != S(i*,n2) the scores, and perhaps the rankings will differ, and so n2 provides some more information over n1 (and v.v.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Note that the reason I keep harping on the ranking and not the scores is that there is an infinitude of cases where the scores will change by the rankings NOT change! I really care a more about the rankings than the scores. Say, for example, I'm trying to choose the best drug, or the best stock, or something like that. There which comes in first matters more than the specific scores.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are, of course, tests, like the Wilcoxon's and Spearman's that tell me on a Z scale how much the rankings differ under different ns. And perhaps the simplest thing to do is just to pick the largest n and call the most informative n the one that gives the largest Z on one of those tests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is there a well understood way to directly measure the information change between two lists, that would seem to be a more theoretically well motivated way of doing this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for your suggestions.    &lt;/p&gt;&#xA;" OwnerUserId="32947" LastEditorUserId="32947" LastEditDate="2018-03-07T20:03:24.443" LastActivityDate="2018-03-07T20:03:24.443" Title="Information measure of rank changes?" Tags="|ranking|information-theory|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23772" PostTypeId="2" ParentId="23761" CreationDate="2017-10-13T22:36:32.973" Score="3" Body="&lt;p&gt;Just to add to @Pete response (I don't have enough reputation to comment), if you're not using Jupyter just remove the &lt;code&gt;output_notebook()&lt;/code&gt; line and add &lt;code&gt;output_file(&quot;csv_plot.html&quot;)&lt;/code&gt; so it looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from bokeh.plotting import figure, output_notebook, show&#xA;from bokeh.models.sources import ColumnDataSource&#xA;from bokeh.models import HoverTool, BoxZoomTool, PanTool, WheelZoomTool&#xA;&#xA;import pandas as pddf = pd.read_csv(&quot;your.csv&quot;)&#xA;&#xA;source = ColumnDataSource(df_plot)&#xA;output_file(&quot;csv_plot.html&quot;)&#xA;hover = HoverTool(tooltips=[(&quot;info&quot;, &quot;@annotation&quot;)])&#xA;p = figure(plot_width=400, plot_height=400, tools=[hover, BoxZoomTool(),     PanTool(), WheelZoomTool()])&#xA;p.circle('x', 'y', size=7, source=source)&#xA;&#xA;show(p)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will save the plot in your current working directory and then open the plot in your browser when it's done generating it.  Check out the &lt;a href=&quot;http://bokeh.pydata.org/en/latest/docs/user_guide/quickstart.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;bokeh docs&lt;/a&gt; for more info&lt;/p&gt;&#xA;" OwnerUserId="40453" LastActivityDate="2017-10-13T22:36:32.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23774" PostTypeId="1" AcceptedAnswerId="23780" CreationDate="2017-10-14T02:12:32.590" Score="-3" ViewCount="601" Body="&lt;p&gt;I want to &lt;strong&gt;test how long it takes to run an algorithm&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here is what I am doing:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;close all the other un-needed applications, run my algorithms alone&lt;/li&gt;&#xA;&lt;li&gt;considering some unstable computer system factors, run multiple times&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you can tell, I can get a set of running time &lt;strong&gt;t1, t2, t3, t4, ... , tn&lt;/strong&gt;, so I'm asking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What to choose from &lt;strong&gt;t1, t2, t3, t4, ... , tn&lt;/strong&gt; as my final result, median or average?&lt;/li&gt;&#xA;&lt;li&gt;Is this the right way to do this?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;btw:&lt;/strong&gt; my algorithm is actually a rendering algorithm, so it would be more complicated.&lt;/p&gt;&#xA;" OwnerUserId="24005" LastActivityDate="2017-10-14T08:20:28.423" Title="Median versus Average, how to choose?" Tags="|dataset|data-cleaning|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23775" PostTypeId="2" ParentId="23774" CreationDate="2017-10-14T02:33:30.737" Score="1" Body="&lt;p&gt;You should choose mean or average over median. Let me explain why, specifically in your case since you're checking for expected computational time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mean or average could be one of the three cases.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Equal to the median&lt;/li&gt;&#xA;&lt;li&gt;Greater than the median&lt;/li&gt;&#xA;&lt;li&gt;Lesser than the median &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now if it is equal to the median there is no problem in choosing whichever value but in real time it's most likely to follow the other two cases. The reason the mean is greater or lesser than the median, in this case, is because it skews to either higher computational time or lower computational time. What this skew represents is you could say where the most likely frequency of computational times would be. This implies it gives a more normalized approach and it shows where you can expect the computational times of most of your execution runs expected to be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So choose mean over median.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-10-14T02:33:30.737" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23776" PostTypeId="2" ParentId="16921" CreationDate="2017-10-14T03:38:48.650" Score="5" Body="&lt;p&gt;The original Savitzky-Golay paper addressed &lt;em&gt;smoothing&lt;/em&gt;, meaning that you estimate a value for some point in the past, using values from its past and future. It sounds like what you're interested in is &lt;em&gt;filtering&lt;/em&gt;:  at time step k, you want to reduce noise and come up with a better estimate of the value at the same time step k, without knowing the future.  That's important in many fields such as process control, and some fault diagnosis where delays can't be tolerated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SG approach of calculating the constants for least squares fitting for smoothing has also been done for filtering.  A curve is fit through n recent points, and instead of estimating the middle point, the end point is estimated by a least squares fit, reduced to simple multiplications by constants. There are tables of the necessary coefficients and an example at &#xA;&lt;a href=&quot;https://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/LeastSquares-Filter/leastsquares-filter.htm&quot; rel=&quot;noreferrer&quot;&gt;https://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/LeastSquares-Filter/leastsquares-filter.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no time delay in the calculations, unlike with smoothing.  Also, if the input signal is a ramp, the filter will converge to the correct value once all of the points in the time window reflect that ramp.  On the other hand, there are some side effects that may or may not matter to you.  Even in the linear case, for a step change in the input, there is overshoot in the filter output.  Higher order least-squares filtering can be quirkier, so it is rarely used. But the linear version overall is quite effective, assuming the little bit of overshoot isn't a real problem  (for instance, if you don't see step inputs in the data anyway).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The link above is part of a more general &quot;Guide to Filtering&quot;, that covers some other common filters as well, such as the exponential filter and moving average filters in particular.  (Filters don't have time delays, but most do lag the inputs - meaning smear out changes over time.  That's a byproduct of filtering out the noise, essentially cancelling out some new noise by a certain amount of averaging against old noise).  It also covers general issues for filtering, such as aliasing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two other filters that are not covered in the Guide that might be considered close competitors to the SG-style filtering:  the alpha-beta filter and double exponential smoothing.  These are covered in Wikipedia:  alpha-beta filtering has its own page, and double exponential smoothing is part of the page on exponential smoothing.  The goal of both is to filter/estimate both the values and the derivative, resulting in a filter that will give correct outputs to a ramp input signal.  I've long suspected that these two filters are essentially the same, but never taken the time to look into it. &lt;/p&gt;&#xA;" OwnerUserId="40457" LastActivityDate="2017-10-14T03:38:48.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23779" PostTypeId="2" ParentId="23774" CreationDate="2017-10-14T06:59:24.023" Score="0" Body="&lt;p&gt;Typically 'How long an algorithm takes' is a set of outputs: Expected value for the expected performance and Maximum value for the worst case performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A time-critical application needs the Maximum value to prevent failures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Average value is the best indication of the expected performance and it is also the easiest to calculate.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-14T06:59:24.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23780" PostTypeId="2" ParentId="23774" CreationDate="2017-10-14T08:20:28.423" Score="5" Body="&lt;p&gt;You haven't asked a proper statistical question, so the choice of mean or median as &quot;best&quot; as a measure of your runtime is unanswerable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you looked at the distribution of run times? Is the algorithm intrinsically variable in its run-time, or is it fixed in its run time but the run times differ because of noise caused by the OS doing other things? Do you want to remove that noise? What if the OS suddenly decides to swap to disk for a bit, or a big network data packet arrives, and the OS goes and does something for a few ms. You could get a long run time for one of your times, and that could pull the mean value way off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The median is a &lt;em&gt;robust estimator&lt;/em&gt; which means a single &quot;bad&quot; value can't throw it off. The mean can be thrown off by a single &quot;rogue&quot; value. Is that what you want? Maybe you do. &lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-10-14T08:20:28.423" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23783" PostTypeId="1" CreationDate="2017-10-14T09:58:21.207" Score="1" ViewCount="41" Body="&lt;p&gt;First off, I don't really know much about machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a virtual world, such as a video game like minecraft or an application like Google Street view, a user can navigate the world using the keyboard and mouse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking to predict what interaction the user will perform. For example, the user might hold down &lt;code&gt;w&lt;/code&gt; for some period of time (to move forward) and then press and hold &lt;code&gt;a&lt;/code&gt; for a period of time to move left. Alternatively, the user may hold &lt;code&gt;w&lt;/code&gt; and use the mouse to turn and look up/down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked into N-Grams, and I can predict interaction to a certain degree, but I am not considering how long a user is not performing an action. I think performing &lt;code&gt;no action&lt;/code&gt; is an important indicator for correct interaction prediction as &quot;no action&quot; could itself be classified as an interaction. The duration of an interaction should also be important, but again, my not sure how to consider that in a model. My Ngram model simply considers the past 3 or 5 inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to develop something which is very fast to predict interaction  (this is key) and learns rapidly from mistakes. Ideally, it would be &quot;Online&quot; in that it constantly &quot;Learns&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd be interested in hearing which models you think would be well suited to my task (are ngrams appropriate?), any examples which closely match my task and I can therefore adopt to suit my needs, any datasets containing user interaction and what I should look at to predict the mouse interaction.&lt;/p&gt;&#xA;" OwnerUserId="40468" LastEditorUserId="40468" LastEditDate="2017-10-14T13:31:28.347" LastActivityDate="2017-10-15T19:14:05.697" Title="Predicting interactions" Tags="|machine-learning|classification|ngrams|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-19T15:15:52.550" ContentLicense="CC BY-SA 3.0" />
  <row Id="23786" PostTypeId="2" ParentId="22491" CreationDate="2017-10-14T11:23:16.033" Score="2" Body="&lt;p&gt;In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First, develop your spark in local mode on your computer. Use a simple subset of data from your final dataset. This way allows you to work with your favorite IDE (jupiter notebook, PyCharm...) and small computing latences. This is really suitable for spark job development.&lt;/li&gt;&#xA;&lt;li&gt;Secondly, once your are done with dev in local mode, you can transfer your code on a cluster using SSH clients or whatever. You just have to change your spark context relative instructions and the input/output sources. Now test your code on a small dataset again and correct the few issues that could appear.&lt;/li&gt;&#xA;&lt;li&gt;Finally, once your code is cluster suitable, you can now use it on your whole Big Data like dataset.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note : On a Hadoop based cluster, I recommend to store your data on HDFS ORC format. You can then smoothly structure it with Hive and easily read or write these tables with Spark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note : Concerning the SSH clients, I recommend you using mobaXterm (a ssh based client for windows) to easily access cluster and drag-and-drop your code. To get some subsets of data for local mode, you can use &quot;Toad for Hadoop&quot; client which gives you the possibility  to run SQL-like queries directly on your hive tables and extract data in CSV or Excel format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note : Spark Standalone mode is not suitable for more that few machines. Prefer yarn or mesos clusters approaches for big datasets. &lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-10-14T11:23:16.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23787" PostTypeId="2" ParentId="12077" CreationDate="2017-10-14T11:37:35.530" Score="0" Body="&lt;p&gt;Using Spark, there is no notion of &quot;mappers&quot; or &quot;reducers&quot;. Each task you perform is Spark is achieved by executors (JVM with allocated ressources). Executors also have the ability to split themselves into multiple cores in order to multithread some tasks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to improve your performs with Spark in cluster mode, you can play on the following parameters :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of executors - the more you have, the more tasks you can achieve in the same time. Keep in mind executors creation is limited by hardware you have.&lt;/li&gt;&#xA;&lt;li&gt;The number of cores - this is the number of cores for each executor. An executor with 5 core is able to run 5 simple tasks at the same time.&lt;/li&gt;&#xA;&lt;li&gt;The amount of memory - for each executor and the driver.&lt;/li&gt;&#xA;&lt;li&gt;The number of partitions in your dataset.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To give you an example, suppose you have a dataset with 30 partitions. Thus, you have 30 tasks to achieve for each spark job. You have to choose the number of executors, the number of cores and the amount of memory. Keep in mind that each core in each executor is able to run a task at the same time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of configuration could be : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;6 executors.&lt;/li&gt;&#xA;&lt;li&gt;5 cores.&lt;/li&gt;&#xA;&lt;li&gt;2G per executor.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you set to few executors and cores, you will have large computing latencies. However, if you set too much executors and cores, a part of them won't compute data because there will be too few partitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some links to learn setting you Spark Context : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/&lt;/a&gt;&#xA;&lt;a href=&quot;https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-10-14T11:37:35.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23789" PostTypeId="1" AcceptedAnswerId="23913" CreationDate="2017-10-14T12:33:00.527" Score="40" ViewCount="37489" Body="&lt;p&gt;I wasn't clear on couple of concepts: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Random Forest uses various sample from tree to create a tree. What's the advantage of this method instead of just using a singular tree? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2023-03-18T12:03:14.047" Title="Why do we need XGBoost and Random Forest?" Tags="|machine-learning|data-mining|random-forest|decision-trees|xgboost|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23790" PostTypeId="2" ParentId="23652" CreationDate="2017-10-14T12:51:24.597" Score="0" Body="&lt;p&gt;I finally managed to find a solution by using &lt;code&gt;tf.scan&lt;/code&gt; (see &lt;a href=&quot;https://stackoverflow.com/questions/46735018/is-this-function-creating-a-new-tensorflow-graph-each-time&quot;&gt;this question on stackexchange&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trick is to create a model object that calls &lt;code&gt;tf.scan&lt;/code&gt; in one of its methods using a step function that operates on the elements of a list of inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function &lt;code&gt;tf.scan&lt;/code&gt; takes care of feeding the output of the step function back into itself for the next iteration, and it is quite flexible so you might be able to do what you need without the need to implement a full RNN from scratch (say, with &lt;code&gt;tf.nn.raw_rnn&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below, the minimal working example that I had posted over there:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import os&#xA;os.environ['TF_CPP_MIN_LOG_LEVEL']='2' # to avoid TF suggesting SSE4.2, AVX etc...&#xA;&#xA;class Model():&#xA;    def __init__(self):&#xA;        self._inputs = tf.placeholder(shape=[None], dtype=tf.float32)&#xA;        self._predictions = self._compute_predictions()&#xA;&#xA;    def _step(self, old_state, new_input):&#xA;        # ---- In here I will write a much more complex graph ----&#xA;        return old_state + new_input&#xA;&#xA;    def _compute_predictions(self):&#xA;        return tf.scan(self._step, self._inputs, initializer = tf.Variable(0.0))&#xA;&#xA;    @property&#xA;    def predictions(self):&#xA;        return self._predictions&#xA;&#xA;    @property&#xA;    def inputs(self):&#xA;        return self._inputs&#xA;&#xA;def test(sess, model):&#xA;    sess.run(tf.global_variables_initializer())&#xA;    print(sess.run(model.predictions, {model.inputs: [1.0, 2.0, 3.0, 4.0]}))&#xA;&#xA;test(tf.Session(), Model())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39081" LastActivityDate="2017-10-14T12:51:24.597" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23792" PostTypeId="2" ParentId="23789" CreationDate="2017-10-14T13:28:51.437" Score="10" Body="&lt;p&gt;When you build a tree, you need to define some criteria for splitting nodes. These include metrics like Information Gain and Gini Index. Those are &lt;em&gt;heuristic&lt;/em&gt; approaches, they are not guaranteed to give the &lt;em&gt;best&lt;/em&gt; possible split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weight in the fact some attributes are less relevant and/or more noisy, and many other problems that happen in real data. In short, &lt;strong&gt;you cannot build a perfect tree&lt;/strong&gt; in a decent computational time (you could of course build all possible trees and test the best, but then you'd have to wait some years for training even in a medium-sized dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we cannot have the &lt;em&gt;best&lt;/em&gt; tree, we have approximations. One approximation is to build many trees (using different data partitions or attribute partitions), since we expect most trees to be somewhat correct, and consider their classifications in a voting system; this should deal with most noise, the vertical partition can deal with irrelevant attributes, the heuristic has less importance, and maybe other advantages.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-10-14T13:28:51.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23793" PostTypeId="1" CreationDate="2017-10-14T13:37:45.867" Score="0" ViewCount="3153" Body="&lt;p&gt;For example, I have a variable Color with 3 different values. How can I transform it, so that I can use this variable in linear model?&#xA;I've read about dummy variables, but as these three will depend on each other (if the value is &quot;red&quot;, then it's not &quot;black&quot;), can I use dummy variables? Shouldn't independent variables be not dependent on each other?&lt;/p&gt;&#xA;" OwnerUserId="40480" LastActivityDate="2017-10-17T01:24:28.913" Title="non-binary nominal variable in linear regression" Tags="|machine-learning|data-mining|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23796" PostTypeId="1" CreationDate="2017-10-14T17:04:58.580" Score="0" ViewCount="1978" Body="&lt;p&gt;I am new to Data Science and have a problem with categorical variables. &#xA;My data set has 2 columns of strings - Departure City and Arrival City. It looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID  | DepCity| ArrCity |Price|Time|SomeColumn   |ColToPredict|&#xA;1   |London  | Berlin  | 300 | 95 | 220         | 4          |&#xA;2   |Dublin  | Nice    | 420 |115 | 59          | 1          |&#xA;3   |Milan   | Brussels| 150 |108 | 154         | 3          |&#xA;4   |Paris   | Rome    | 160 |120 | 200         | 4          |&#xA;250 |Madrid  | Oslo    | 290 |300 | 110         | 2          |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So there are a lot of categorical variables in both columns and these columns are important (their values depend on other columns). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use Python and sklearn. And it's not possible just to eliminate them as suggested in some tutorials. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there is a way to deal with categorical variables by creating new columns with zeros and ones for each variable. But I'm not sure that it's my case, because I have about 30 unique cities in each column. What will be the best way to convert categorical variables into numerical?&lt;/p&gt;&#xA;" OwnerUserId="40491" LastEditorUserId="40491" LastEditDate="2017-10-14T17:25:41.600" LastActivityDate="2017-10-19T14:37:31.770" Title="How to deal with categorical variables" Tags="|machine-learning|scikit-learn|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23798" PostTypeId="2" ParentId="23796" CreationDate="2017-10-14T18:01:34.913" Score="3" Body="&lt;p&gt;In your specific case you are working with cities, which can be modelled as geographic location. I've never used it be you could look at the geohash method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or could also use an identifiant for each city. An algorithm like a random forest can directly use it this way.&lt;/p&gt;&#xA;" OwnerUserId="40331" LastEditorUserId="40331" LastEditDate="2017-10-14T18:09:09.257" LastActivityDate="2017-10-14T18:09:09.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23803" PostTypeId="1" CreationDate="2017-10-15T00:35:17.697" Score="-1" ViewCount="724" Body="&lt;p&gt;I have a problem where I'm trying to estimate a numeric number given a set of features. The number ranges from 160 to 240.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could I label the data using ranges i.e. ( 160-165, 166-170, 171-175, etc, etc ) and then use a random forest to classify?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the pros and cons of that vs using a linear regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems kind of nice because I would have probabilities/confidences with the random forest approach.&lt;/p&gt;&#xA;" OwnerUserId="40502" LastActivityDate="2017-10-17T17:02:17.060" Title="Using random forest to estimate a numeric using ranges?" Tags="|random-forest|linear-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23806" PostTypeId="1" CreationDate="2017-10-15T06:51:55.593" Score="0" ViewCount="1092" Body="&lt;p&gt;Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client.For the Purpose of this Analysis i first want to download a file from the Website called UCI Machine Learning Repository and then access the file for downstream Analysis, so i used the Below shown R Script in the r-studio server in my browser&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;download.file(&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data&quot;, destfile = &quot;Haberman.txt&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This works exactly fine but since i am new to work in the Cloud Platform i couldn't understand where RStudio stores the Haberman.txt file, because when i work on my PC i used to specify the Path to store the Haberman.txt file in the destfile argument as shown below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; download.file(&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data&quot;, destfile = &quot;E:/Couresera_Datasets/Haberman.txt&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now when i Navigate to the specified Path in my PC i will be able to find the Haberman.txt file but where can i find the Haberman.txt file in the Ubuntu EC2 instance when i use the download.file()?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any Help Will be Appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in Advance&lt;/p&gt;&#xA;" OwnerUserId="39503" LastActivityDate="2017-11-15T20:41:14.927" Title="where does the downloaded file gets stored in the R-Studio Server on Ubuntu EC2 Instance when using download.file()" Tags="|r|rstudio|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23808" PostTypeId="1" CreationDate="2017-10-15T09:39:51.790" Score="0" ViewCount="333" Body="&lt;p&gt;I am trying to predict scored labels using regression. But when I am about to get the result from Azure ML Web Service in Excel 2016, there is no result appeared in the scored label column. How should I fix this? Below is all my process...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fndil.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fndil.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my problem I always get. As you are seeing now, there is no result in scored label column when I try to predict.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0tDZ4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0tDZ4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38042" LastActivityDate="2023-12-29T00:03:25.407" Title="No result for scored labels in Azure ML Web Service" Tags="|machine-learning|regression|azure-ml|excel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23809" PostTypeId="2" ParentId="10339" CreationDate="2017-10-15T13:51:35.703" Score="2" Body="&lt;p&gt;I ended up finding &lt;a href=&quot;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the paper you were referencing&lt;/a&gt;, maybe next time, add it to your post.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I can understand in this paper, &lt;em&gt;label-dropout&lt;/em&gt; means that you are &lt;em&gt;dropping&lt;/em&gt; the real labels and replace them with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The section of the paper you're referring to is explaining everything in great detail, so I'd try to read it carefully. In short, instead of using the ground-truth distribution (i.e. one 1 and a bunch of 0s) in the target layer, they are adding some uncertainty. That way, the model becomes less confident, and therefore is less prone to overfitting and becomes more adaptable.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-10-15T13:51:35.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23812" PostTypeId="1" CreationDate="2017-10-15T18:29:12.143" Score="0" ViewCount="141" Body="&lt;p&gt;Not sure if this question is OT here.  If it is, perhaps move it to meta?  If not:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning.  I've got a doctorate, but not in statistics or computer science.  I'm not a software engineer and am only really fluent in R.  Maybe I know a bit of C++ and can write a SQL query, but I wouldn't be the guy you'd hire to set up a data pipeline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of engineers who know maybe a thing or two about statistics and machine learning, but who can build apps in Java, manage a spark cluster, set up a distributed database, etc.  But they wouldn't be the person you'd hire to do any sort of analysis that isn't already canned in python.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Browsing jobs in the private sector, I see a lot of demand for the second type of data scientist.  Is there much demand for the first type?  What is the pay differential in the private sector between the two types?  Did you start as the first type and become more like the second type &quot;on the job?&quot;&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-10-16T01:44:30.610" Title="What is valued more in the data science job market, statistical analysis or data processing?" Tags="|machine-learning|data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23813" PostTypeId="2" ParentId="23783" CreationDate="2017-10-15T19:14:05.697" Score="1" Body="&lt;p&gt;Interesting question.  I haven't worked with this sort of data much, but it seems to me that the bulk of the job is likely to be feature engineering.  Every &quot;supervised&quot; statistical method that I know of requires that you shoe-horn your data into &quot;outcomes&quot; and &quot;covariates.&quot;  $\mathbf{Y}$ and $\mathbf{X}$.  Once your data is in this form you simply find an appropriate algorithm that can estimate&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mathbf{Y} = f(\mathbf{X})&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But taking data on how people play a video game and turning it into a 2-dimensional matrix isn't necessarily obvious.  One of the most difficult things is that you'll need to make observations of different users comparable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you're playing quake 3 and you're trying to predict whether the user is going to go for the BFG.  How do you define $\mathbf{Y}$?  Is it &lt;code&gt;got BFG&lt;/code&gt;? Or should it be &lt;code&gt;came within some distance of BFG&lt;/code&gt;?  (maybe people messed up the rocket jump, for example).  What if they got fragged on the way to the BFG?  The point is that defining the outcome involves choices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Likewise what are the covariates?  Say I get spawned in some location, and I know the map, and immediately move toward the BFG.  I'll have a different set of &lt;code&gt;w&lt;/code&gt;, &lt;code&gt;a&lt;/code&gt;, &lt;code&gt;s&lt;/code&gt;, and &lt;code&gt;d&lt;/code&gt; keystrokes than someone who got spawned elsewhere on the map.  So is it n-grams that you want to look at?  Maybe rather it is change in euclidian distance between spawning location and BFG location over some time interval.  if so, what is the appropriate time interval?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, what you're doing is taking raw data and turning it into abstractions of that data.  This is actually what a neural net does.  It takes raw data $\mathbf{X}$ and forms a lower-dimensional representation of that data, $\mathbf{V}$, where $\mathbf{V}$ has fewer columns than $\mathbf{X}$.  When neural nets work well, those columns of $\mathbf{V}$ can be uncannily similar to things that a human would pick out of a stream of raw data.  Akin to this:  &lt;a href=&quot;https://www.youtube.com/watch?v=3vAnuBtyEYE&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=3vAnuBtyEYE&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those $\mathbf{V}$ are then just related to your outcome $\mathbf{Y}$ in a linear model or a logistic regression.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you don't turn raw keystrokes into &lt;code&gt;blonde&lt;/code&gt;, &lt;code&gt;brunette&lt;/code&gt; and &lt;code&gt;redhead&lt;/code&gt; without a lot of data and long training times.  It'll go faster if you can prespecify functions of your raw data, which requires domain knowledge.  &lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-10-15T19:14:05.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23814" PostTypeId="1" CreationDate="2017-10-15T19:52:25.410" Score="1" ViewCount="54" Body="&lt;p&gt;I am trying to find an interesting way to interpret and display a set of data for the research I'm working on. Columns 2-4 show the net change from time 1 to time 2 in antibiotic coverage for different types of bacteria (types A-D). 1 means coverage for that type of bacteria was added, and -1 means coverage for that bacteria was stopped, and 0 means there was no change in coverage (i.e. it wasn't covered at time 1 or at time 2, or it was covered the same at both times). Column 1 shows whether bacterial cultures were positive or negative. I'm interested in showing graphically the differences in coverage for A-D when cultures are positive vs. negative. Each row represents a different case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll notice that there is no change (0) in A-D in the majority of cases. The fact that most cases had no change may be the most interesting point, but I'm not sure how to make that point stand out on a graph either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The graph's i've tried making have only included the cases in which there is at least 1 change. So far I've tried mostly mostly bar/column graphs showing the proportion of positive/negative cultures with each changes. So the column might be B - Add with a culture-positive and culture-negative bar showing the relative proportions of each to see if the culture made a difference in adding B, and the same for B - Stop, C - Add, C - Stop, etc, for each category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the most part I've seen that the culture doesn't really impact whether coverage for each category was added or stopped, but the charts I'm making don't tell that in a convincing way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;Culture     A   B   C   D&#xA;negative    0   0   0   0&#xA;positive    -1  0   0   0&#xA;positive    0   0   0   0&#xA;negative    1   1   1   0&#xA;negative    0   -1  -1  0&#xA;negative    0   0   -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   -1  0   0&#xA;negative    0   0   0   0&#xA;positive    1   0   0   1&#xA;positive    0   0   0   0&#xA;positive    0   -1  -1  0&#xA;negative    1   1   1   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   -1  0   0&#xA;negative    0   1   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;positive    0   1   -1  0&#xA;positive    0   0   1   0&#xA;negative    0   0   0   0&#xA;negative    1   1   1   0&#xA;positive    0   0   0   0&#xA;positive    1   1   1   0&#xA;negative    -1  -1  -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    -1  -1  -1  0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    1   0   0   0&#xA;negative    0   -1  0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    -1  -1  0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   -1  0&#xA;negative    0   0   1   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    1   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   -1  0&#xA;negative    0   0   0   -1&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   -1  0&#xA;negative    0   -1  -1  0&#xA;positive    -1  0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   1   1   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    -1  0   0   0&#xA;positive    0   1   0   0&#xA;negative    0   0   0   0&#xA;positive    -1  -1  0   0&#xA;positive    -1  0   0   0&#xA;positive    0   0   0   1&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   -1  0   0&#xA;positive    -1  -1  -1  0&#xA;negative    0   0   -1  0&#xA;negative    0   0   0   0&#xA;negative    0   -1  -1  0&#xA;negative    0   1   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    -1  -1  -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   1   1   0&#xA;positive    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   1   0   0&#xA;negative    -1  1   0   0&#xA;negative    0   0   0   0&#xA;positive    1   1   1   0&#xA;negative    0   0   0   1&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   0   0   0&#xA;negative    -1  0   0   0&#xA;negative    -1  0   0   0&#xA;negative    0   0   0   0&#xA;positive    -1  0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   -1  0   1&#xA;negative    -1  -1  0   0&#xA;negative    -1  0   0   0&#xA;negative    0   0   0   0&#xA;negative    -1  0   0   0&#xA;negative    -1  0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   1   1   0&#xA;negative    0   0   0   0&#xA;negative    -1  0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   1&#xA;negative    0   0   0   0&#xA;negative    0   0   1   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   -1  0&#xA;negative    0   0   0   0&#xA;negative    -1  0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   1&#xA;positive    -1  0   0   0&#xA;positive    -1  -1  -1  0&#xA;positive    1   0   1   0&#xA;negative    -1  0   0   0&#xA;positive    0   1   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   1   1   1&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   1   0   0&#xA;negative    0   0   0   0&#xA;negative    -1  -1  -1  0&#xA;negative    0   -1  0   0&#xA;negative    1   0   0   0&#xA;positive    0   0   0   0&#xA;negative    1   1   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   0   0&#xA;negative    -1  0   0   0&#xA;negative    -1  0   0   0&#xA;negative    1   1   1   0&#xA;negative    0   -1  0   0&#xA;positive    0   0   -1  0&#xA;negative    0   0   0   0&#xA;positive    1   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   1   1   0&#xA;positive    0   0   0   0&#xA;negative    -1  -1  -1  0&#xA;negative    1   1   1   0&#xA;negative    1   1   1   0&#xA;negative    0   0   0   0&#xA;negative    0   0   1   0&#xA;positive    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    -1  -1  0   0&#xA;positive    0   0   0   1&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    0   -1  0   0&#xA;positive    -1  -1  -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;positive    0   0   1   0&#xA;positive    0   -1  0   0&#xA;negative    -1  0   0   0&#xA;negative    1   1   1   0&#xA;negative    1   1   1   0&#xA;positive    1   0   0   0&#xA;positive    0   0   0   0&#xA;negative    -1  0   0   0&#xA;positive    0   -1  -1  0&#xA;negative    0   0   0   0&#xA;negative    0   0   0   0&#xA;negative    1   0   0   0&#xA;positive    0   1   1   0&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="40525" LastActivityDate="2017-11-16T21:39:13.757" Title="Graph to display differences (or lack of) in multilevel categorical data" Tags="|statistics|graphs|graphical-model|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23816" PostTypeId="2" ParentId="12830" CreationDate="2017-10-15T21:19:10.977" Score="2" Body="&lt;p&gt;The equivalent kernel simply has whatever shape the input has, and computes a tensor dot product. (I use the word &quot;shape&quot; as there seems to be some confusion over &quot;size&quot;, which often ignores the channel/depth dimension). There's no &quot;sliding the kernel across input&quot; involved, since the kernel is as large as it can be.&#xA;Quoting &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/#convert&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS 231n course notes&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;any FC layer can be converted to a CONV layer. For example, an FC&#xA;  layer with K=4096 that is looking at some input volume of size 7×7×512&#xA;  can be equivalently expressed as a CONV layer with&#xA;  F=7,P=0,S=1,K=4096,F=7,P=0,S=1,K=4096. In other words, we are setting&#xA;  the filter size to be exactly the size of the input volume, and hence&#xA;  the output will simply be 1×1×4096 since only a single depth&#xA;  column “fits” across the input volume, giving identical result as the&#xA;  initial FC layer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I believe &quot;F=7,P=0,S=1,K=4096,F=7,P=0,S=1,K=4096&quot; here means each conv kernel has shape 7x7x512, and there's 4096 such filters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The earlier answer mentioned that the last fc of AlexNet (which receives input with shape 1x1x4096 and computes 1000 class scores) is implemented as &quot;1x1 convolution&quot;. To be complete, each such conv kernel has &lt;strong&gt;shape&lt;/strong&gt; 1x1x4096, and there's 1000 of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Le Cunn also explains this in the &lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN paper&lt;/a&gt;, page 8, description of LeNet5:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Layer C5 is a convolutional layer with 120 feature maps. Each unit is&#xA;  connected to a 5x5 neighborhood on all 16 of S4's feature maps. Here&#xA;  because the size of S4 is also 5x5, the size of C5's feature maps is&#xA;  1x1; this amounts to a full connection between S4 and C5.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8660" LastActivityDate="2017-10-15T21:19:10.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23818" PostTypeId="2" ParentId="23812" CreationDate="2017-10-16T01:42:43.900" Score="2" Body="&lt;p&gt;Given your current situation, a data analyst or scientist position makes more sense, but with some practice you can acquire the software skills to become a data- or machine learning engineer. There is a glut of people from academia similar to you, and software companies want people who can actually build software, so it is not surprising you see demand for engineers. But that does not mean you &lt;em&gt;have&lt;/em&gt; to become one. Do you want to build software products?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation would be to make sure you are familiar enough with common applications of statistics in industry (e.g., A/B testing, and time series forecasting) to get a job in analysis, then learn computer science on the side. There is a wide range of salaries, depending on seniority, company and location. You can find specific figures on sites like LinkedIn, Glassdoor, Paysa, Payscale.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-10-16T01:42:43.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23819" PostTypeId="2" ParentId="23812" CreationDate="2017-10-16T01:44:30.610" Score="2" Body="&lt;p&gt;Data Science is the new &quot;sexy&quot; job. I got dozens of colleagues who ask me how to become one. Some, come to me and tell me that Andrew Ng's coursera course is hard, and what can they learn instead so they can put it on their CV. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What all this means is that there are a lot of not well prepared people that think they can become a data scientist, and advertise themselves as such, but are really not. Many companies are starting to learn to vet better their candidates because of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, I am going to answer your question: you have some real skills, you actually understand statistics, which most data scientist I work with don't. Many financial institutions will want people like you, they can find plenty of people who can write Java and set up a cluster and set up a machine learning algorithm without understanding what it does. They are technicians, but do not understand what the data implies. If you can show that you actually can &quot;think&quot; and provide insights, you can be appreciated much more. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, you can command a higher salary. What you need is to learn most used machine learning algorithms, maybe learn some python (easy if you already know R) and tailor your resume for those jobs who require actual thinking. You have real skills.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2017-10-16T01:44:30.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23820" PostTypeId="1" AcceptedAnswerId="23822" CreationDate="2017-10-16T01:47:52.517" Score="0" ViewCount="1578" Body="&lt;p&gt;I'm categorizing 30 types of clothes from the image using R-CNN Object Detection Library from tensorflow : &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/research/object_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/models/tree/master/research/object_detection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does color matter when we collect images for training and testing? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I put only purple and blue shirts, I guess it won't recognize red shirts? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I gray scale all images to detect the types of clothes? :) &lt;/p&gt;&#xA;" OwnerUserId="38593" LastEditorUserId="13736" LastEditDate="2017-10-16T03:12:11.860" LastActivityDate="2023-10-06T18:42:30.450" Title="Should I gray scale the image?" Tags="|deep-learning|image-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23822" PostTypeId="2" ParentId="23820" CreationDate="2017-10-16T07:30:44.677" Score="1" Body="&lt;p&gt;Your hypothesis about missing colours in your samples affecting results in production could be correct. However, it is trivial to convert images to greyscale as you load them from storage. So keep them in colour, and convert them as you load them if you need black and white.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That way you can try both with and without colour as input and you will have your answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is very little effort to do in practice, and allows you to do the &quot;science&quot; part of data science by comparing two approaches and measuring the difference. This is standard practice, even if you are reasonably certain one approach or another is &quot;the best&quot;, it is normal to explore a few variations. When you are not sure, then it is even more important to try it and see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test your hypothesis, you could put all the t-shirts of a particular colour in your test set. If that reduces the accuracy of your results with the colour model, it would back up your concern. One fix for that might be to remove colour information from the model, if it is not relevant to the task. An alternative is to collect more data so you have enough samples of different colour shirts. However, you might find if you are fine-tuning a neural network trained on many more images (e.g. Inception v5) that the impact of colour is less even though your samples do not cover all possible T-shirt colours.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-16T07:30:44.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23823" PostTypeId="1" CreationDate="2017-10-16T07:32:49.070" Score="1" ViewCount="205" Body="&lt;p&gt;$$ \lambda||\hat w||^2 +(1/n)\sum   max(0,1-y_i(\hat w \hat x_i -b))   $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we know that $2/||\hat w||$ is the width of the margin. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$:&#xA;$$ 1-y_0(\hat w \hat x_0 -b)=3 $$&#xA;That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term - they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Perhaps one reason is that  $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?&lt;/p&gt;&#xA;" OwnerUserId="40540" LastEditorUserId="40540" LastEditDate="2017-10-16T14:20:08.550" LastActivityDate="2018-11-14T17:03:25.790" Title="Support vector machine margin term, norm or norm squared?" Tags="|classification|svm|regularization|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23824" PostTypeId="2" ParentId="23793" CreationDate="2017-10-16T08:25:12.247" Score="0" Body="&lt;p&gt;When two variables are independent of each other, it means that no variable can be expressed as a function of the other. &quot;If value is red, then it's not black&quot; isn't an example of dependent variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am assuming that your '3 variables' are R,G and B in the range (0-255). To transform the 3 variables into a singular one for a linear model, you could use hex valued colors. Look at this one to get a better idea: &lt;a href=&quot;https://math.stackexchange.com/questions/1635999/algorithm-to-convert-integer-to-3-variables-rgb&quot;&gt;https://math.stackexchange.com/questions/1635999/algorithm-to-convert-integer-to-3-variables-rgb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-16T08:25:12.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23827" PostTypeId="2" ParentId="23591" CreationDate="2017-10-16T09:23:39.877" Score="0" Body="&lt;p&gt;'Clustering different ethnic groups for visualization' seems more like you are trying to do supervised dimensionality reduction since you already know the target variables in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you will be using it for classification later, I assume you already know the number of ethnic groups. This can be done using Linear Discriminant Analysis (LDA). Check out this post: &lt;a href=&quot;https://stats.stackexchange.com/questions/161362/supervised-dimensionality-reduction&quot;&gt;https://stats.stackexchange.com/questions/161362/supervised-dimensionality-reduction&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-16T09:23:39.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23828" PostTypeId="1" CreationDate="2017-10-16T09:32:29.497" Score="2" ViewCount="6607" Body="&lt;p&gt;I'm working with sklearn SVM and I have a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run the method &lt;code&gt;sklearn.SVM.SVC.fit()&lt;/code&gt; using a database with only a few features (&amp;lt; 10) it takes a very long time. This is weird because, when I run the same method with the same database using all of the features (&gt; 100) it takes just a few seconds. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using a polynomial kernel and this problem only appears when the degree is &gt;= 3. And it is also weird that if I use only a few features, but I add 50 features with 0s in all the samples, it works fine!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried the following experiments:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;fit()&lt;/code&gt; with all the features and any kernel degree. Works fine.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;fit()&lt;/code&gt; with a few features and kernel degree &amp;lt;= 2. Works fine.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;fit()&lt;/code&gt; with a few features and kernel degree &gt;= 3. Very slow.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;fit()&lt;/code&gt; with kernel degree = 3, a few features but adding 50 features with 0s in all the samples. Works fine!!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is, how can I solve this? Does my trick of adding features with 0s affect the classification?&lt;/p&gt;&#xA;" OwnerUserId="40545" LastEditorUserId="29575" LastEditDate="2017-12-14T12:37:22.133" LastActivityDate="2017-12-14T15:13:06.273" Title="sklearn SVM really slow" Tags="|python|classification|scikit-learn|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23830" PostTypeId="2" ParentId="23814" CreationDate="2017-10-16T10:15:09.660" Score="0" Body="&lt;p&gt;Very first input: Please confirm if you are really asking about &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graphs&lt;/a&gt; as there are tones of misunderstanding between plotting and Graph as a mathematical object. In case you mean visualization and not Graph as a math object please let me know to edit your tags and title. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will continue with my answer anyway as graphs are the right tool for visualization and moreover for analysis of your data even if you touched them accidentally! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see your problem from Graph Theory point of view in different ways. I assume cases as nodes and changes as edges in simplest way. You may continue developing the idea to more sophisticated structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume each case is a node in your graph. Then you can model each bacteria with one graph in which there is no edge between two nodes if the change is zero and an edge if there is, where the weight of edge is either 1 or -1 for different cases. Then you come up with 4 graphs on each of which you may do statistical analysis according to &lt;a href=&quot;https://en.wikipedia.org/wiki/Network_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;network topology&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a more detailed modeling, you may consider two nodes for each case corresponding to each timestamp and connect them with weights 1 or -1 or not connect them in case the change is zero. To avoid weights you can simply use directed graph in which an edge goes from node of case $X$ in time $t_0$ goes to node of case $X$ in time $t_1$ if the change is one and if it's -1, an edge goes from node of case $X$ in time $t_1$ to node of case $X$ in time $t_0$. The topology of connections also tells a lot in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases the visualization of graph might be intuitive might not. If not you can visualize different statistical measures on nodes, edges, degrees, graphs themselves, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps! &lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-10-16T10:15:09.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23831" PostTypeId="1" CreationDate="2017-10-16T10:17:21.083" Score="2" ViewCount="425" Body="&lt;p&gt;I'm creating a supervised convnet which recognizes a phrase in a regional language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I have a dataset which contains 100 sounds for labels A and another 100 sounds for labels B. Now normal operation of the network is to differentiate a sound between label A and label B.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But let's say that I give a new sound to the network to classify, which is actually neither A nor B. I want the neural net to tell me that it doesn't qualify any of the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I achieve this?&lt;/p&gt;&#xA;" OwnerUserId="40546" LastEditorUserId="29169" LastEditDate="2021-02-15T22:22:09.000" LastActivityDate="2021-02-15T22:22:09.000" Title="How to classify a data as not matching any label" Tags="|deep-learning|neural-network|convolutional-neural-network|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23832" PostTypeId="1" CreationDate="2017-10-16T10:32:30.507" Score="1" ViewCount="211" Body="&lt;p&gt;I currently have a Graph model whereby I am mapping connections of different types between entities and attributing a weight to these connections based upon my own personal experience. Also, I would like to understand the connections between these entities in relation to a particular outcome. Looking at this problem, I can't help but notice it's similarity to a typical Artificial Neural Network (ANN) and am wondering if/how I can bake some of the theory there into my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me explain further; I have a set of entities for which I have sets of connections. Each different type of connection has it's own individual weight, as shown in this diagram:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IyIDF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IyIDF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weights in this diagram are estimated based on my own experience and can then be summed to look at the strength of a relationship between these entities. It is this strength of relationship that I want to understand in relation to a particular Output/Outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't help but notice when looking at this particular application of Graph Theory that it bears a large resemblance to an Artificial Neural Network (ANN), as shown in this diagram;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/56zb6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/56zb6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference here being that whilst in the first diagram the nodes represent entities, in the second diagram the nodes represent input values. In the ANN, input values, randomized weights and the process of back-propagation are are used to help determine what the weights need to be whilst in the first diagram, these values are estimated. What I want to be able to do is to use randomized weights and the process of back-propagation to determine the weights in my Graph model that maps the connections between entities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any experience of doing anything like this or could maybe at lest discuss the possibility of doing something like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!  &lt;/p&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2017-10-16T11:38:29.763" Title="Training a Graph model like an Artificial Neural Network" Tags="|machine-learning|graphical-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23833" PostTypeId="2" ParentId="20505" CreationDate="2017-10-16T11:10:12.957" Score="3" Body="&lt;p&gt;There is something called &quot;sampled softmax&quot; (e.g. &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/sampled_softmax_loss&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow's implementation&lt;/a&gt;), which simply partitions the output space and at each training step only takes into account one of the partitions (see section 3 from &lt;a href=&quot;http://arxiv.org/abs/1412.2007&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; to learn the math). Sampled softmax is only meant to speed up training for very large output spaces; at inference time you use normal softmax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if your output is of hierarchical nature, you can use hierarchical softmax. See &lt;a href=&quot;http://ruder.io/word-embeddings-softmax/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post&lt;/a&gt; for a review of different methods to handle very large categorical space.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-10-16T17:28:25.850" LastActivityDate="2017-10-16T17:28:25.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23834" PostTypeId="2" ParentId="23832" CreationDate="2017-10-16T11:38:29.763" Score="1" Body="&lt;p&gt;First, a note: don't let the graph structure make you think that this can be addressed with neural networks; this structure only suggests that this is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Multilevel_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;hierarchical model&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, let's take into account the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In your first figure, you don't connect A, B or C to the output. I assume that there is a relationship from A,B,C to output.&lt;/li&gt;&#xA;&lt;li&gt;You mention that you have &lt;em&gt;expert a priori knowledge&lt;/em&gt; on the relationships from A to B and from A to C.&lt;/li&gt;&#xA;&lt;li&gt;I also assume that A, B, C and the output are numeric variables.&lt;/li&gt;&#xA;&lt;li&gt;We don't know whether A, B, C and the output can be observed or not.&lt;/li&gt;&#xA;&lt;li&gt;We don't know why there are 3 lines from A to C and 2 lines from A to B, so I will not take this into account.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You may want to provide more information if any of the assumptions above is not correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If only A and the output are observable, we are talking about a latent variable model. Latent variables are best suited for Hierarchical Bayesian methods. If the relations between variables are linear, you can model this as a bayesian linear model, e.g. hierarchical bayesian regression to model your variables and inject your prior knowledge as prior distributions over the weights. &lt;a href=&quot;http://twiecki.github.io/blog/2014/03/17/bayesian-glms-3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; you can see the application of hierarchical bayesian regression to a latent variable model. This kind of methods are simple and interpretable, they handle prior knowledge graciously and they handle uncertainty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all variables are observable, you can define a combination of different extremely simple &quot;neural network&quot; models to approach your problem: Network 1 would receive A and output C; Network 2 would receive A and output B; Network 3 would receive A, B and C and generate the output. If the relationships among variables are linear, Network 1 and Network 2 are simple multiplications. You can then initialize their weights with your prior knowledge. The training would &lt;em&gt;adjust&lt;/em&gt; the weights for everything to fit, but will not give you insight about the relationships except the fitted weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you think about whether bayesian hierarchical models can fit your problem.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-10-16T11:38:29.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23836" PostTypeId="2" ParentId="23831" CreationDate="2017-10-16T11:42:30.350" Score="4" Body="&lt;p&gt;Let's phrase it another way, decomposing into two problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;given a sound, we want to know if it's of class A&lt;/li&gt;&#xA;&lt;li&gt;given a sound, we want to know if it's of class B&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This way of putting it is valuable to some techniques, notably &quot;one-class classification&quot; and &quot;PU learning&quot; (learning from positive and unlabeled examples). These techniques are very relevant when you want to know &quot;does my item follow a given distribution D?&quot;, which is exactly what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, if you have (or can collect) a lot of data that is neither A nor B, you may simply label it as a garbage class C and use a common classifier - however, it may actually be harmful, because you're likely assuming wrong things about class C [Li, Liu and Ng, &quot;Negative Training Data can be Harmful to Text Classification&quot;, 2010]. It &lt;em&gt;could&lt;/em&gt; work and is much simpler to develop, may be good enough depending on your case.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-10-16T11:42:30.350" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23837" PostTypeId="1" AcceptedAnswerId="23838" CreationDate="2017-10-16T11:49:51.283" Score="-1" ViewCount="131" Body="&lt;p&gt;Let's say that I want to create a pseudorandom number generator, and I'd like to make sure that it's truly close to random. For simplicity, let's assume that I want to output a 0 or a 1 randomly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have heard of the monobit, runs, poker test etc. but are there &quot;machine learning&quot; ways to evaluate a pseudorandom number generator?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As in, one could try to predict the number that will be outputted given the first k numbers that were previously outputted, and the performance of that model would give you how well the pseudorandom generator is performing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be way over my head, but could a generative adversarial network learn a truly pseudorandom generator that way?&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2019-12-07T11:12:55.437" LastActivityDate="2019-12-07T11:12:55.437" Title="Using machine learning to evaluate a random number generator" Tags="|generative-models|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23838" PostTypeId="2" ParentId="23837" CreationDate="2017-10-16T12:56:09.063" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;are there &quot;machine learning&quot; ways to evaluate a pseudorandom number generator?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No it is the wrong tool for the job. Statistical tests, like those you mention &lt;em&gt;monobit, runs, poker test&lt;/em&gt; etc, are a way to evaluate a PRNG and search for bias or unwanted patterns which would establish that it had flaws. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you try to use ML on random data - e.g. map dependent x,y data to a label z where all are determined randomly - then one sign the data is random will be that the ML does poorly. However, this will not give any measure of quality of the randomness. Using a high variance model it will still be possible to fit the data, although cross validation and test sets will score badly.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It might be way over my head, but could a generative adversarial network learn a truly pseudorandom generator that way?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A GAN learns how to map a low dimensional space into a manifold (sub-shape) of a larger dimensional space, allowing you to sample from a population in the higher dimension. It cannot inject &quot;noise&quot; into that space other than using something provided already by a PRNG. The GAN itself cannot be the PRNG. The GAN sampling routine is typically driven by an external PRNG (i.e. not driven by the weights or processing of the GAN)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably you could construct a PRNG from a RNN, although I suspect it would be hard to create one of any quality, and even if you could get something close to statistical randomness from it, the performance is likely to be poor compared to standards such as Mersenne Twister.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-16T13:02:45.647" LastActivityDate="2017-10-16T13:02:45.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23840" PostTypeId="1" CreationDate="2017-10-16T15:24:00.517" Score="2" ViewCount="413" Body="&lt;p&gt;I am trying to build a RNN that can take time series as input for example from various sensors.&#xA;While building the RNN there are a couple of initial lengths that must be defined.&#xA;The size of the cell and the length of how much you will unroll the RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what i understand, cell size is arbitrary and can be modified independent of how the inputs are shaped. But how much you will unroll the RNN depends on your input length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If i have a time series of 1 hour consisting of 1000 values, should i choose to unroll the RNN 1000 times, or is it too much?&#xA;My goal is to classify the whole time series, and ideally i want all 1000 time steps to be taken into account for this classification.&lt;/p&gt;&#xA;" OwnerUserId="40182" LastActivityDate="2017-10-16T15:24:00.517" Title="RNN unrolling length" Tags="|time-series|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23841" PostTypeId="1" AcceptedAnswerId="23851" CreationDate="2017-10-16T15:37:15.780" Score="-1" ViewCount="594" Body="&lt;p&gt;I'm studying about K-mode and K-prototype but I cannot find any proper example on a very basic example of how it works contrary to K-means where there are quite a lot (like this one &lt;a href=&quot;http://mnemstudio.org/clustering-k-means-example-1.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;description-k-means&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know a book or website which have a similar example for K-mode and/or K-prototype ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot&lt;/p&gt;&#xA;" OwnerUserId="40557" LastActivityDate="2017-10-16T19:03:15.887" Title="K-mode or K-prototype" Tags="|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23843" PostTypeId="1" CreationDate="2017-10-16T16:10:00.417" Score="6" ViewCount="2810" Body="&lt;p&gt;I am trying to do the following:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt; vc = VotingClassifier(estimators=[('gbc', GradientBoostingClassifier()),&#xA;                                   ('rf', RandomForestClassifier()), &#xA;                                   ('svc', SVC(probability=True))],&#xA;                       voting='soft',&#xA;                       n_jobs=-1,&#xA;                       weights=[2, 3, 1])&#xA;cross_val_score(vc, X_new, y, n_jobs=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;In this, I want to tune the parameter &lt;code&gt;weights&lt;/code&gt;. If I use &lt;code&gt;GridSearchCV&lt;/code&gt;, it is taking a lot of time. Since it needs to fit the model for each iteration. Which is not required, I guess. Better would be use something like &lt;code&gt;prefit&lt;/code&gt; used in &lt;code&gt;SelectModelFrom&lt;/code&gt; function from &lt;code&gt;sklearn.model_selection&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Is there any other option or I am misinterpreting something?&lt;/p&gt;&#xA;" OwnerUserId="40558" LastEditorUserId="1330" LastEditDate="2021-01-23T14:36:35.193" LastActivityDate="2021-01-23T14:36:35.193" Title="How to tune weights in Voting Classifier (Sklearn)" Tags="|scikit-learn|model-selection|ensemble-modeling|parameter-estimation|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="23844" PostTypeId="1" AcceptedAnswerId="23845" CreationDate="2017-10-16T16:16:40.937" Score="1" ViewCount="688" Body="&lt;p&gt;I'm kinda new to this field, so I started tinkering with some models in Keras (using Tensorflow backend). But recently I started to migrate to a pure Tensorflow approach, and I'm not getting good results, what is strange, since I'm using the TF backend in Keras, so I was expecting similar results. So, most certainly, I'm getting something in the implementation wrong, but I can't figure out what it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to implement a 7 layer MLP, with one linear output using ADAM. To make it easy, I removed all regularization from the model, so I was expecting the model to overfit, what happened to the Keras model, but not to the TF model.&#xA;If someone could point what is wrong in the Tensorflow implementation, I would be very grateful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the github repository:&#xA;&lt;a href=&quot;https://github.com/makalaia/Tensorflow-Benchmark&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/makalaia/Tensorflow-Benchmark&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keras code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;import numpy as np&#xA;import time&#xA;import matplotlib.pyplot as plt&#xA;&#xA;from keras.layers import Dense&#xA;from keras.models import Sequential&#xA;from pandas import read_csv&#xA;&#xA;&#xA;def calculate_rmse(real, predict):&#xA;    m = len(real)&#xA;    return np.sqrt(np.sum(np.power((real - predict), 2)) / m)&#xA;&#xA;&#xA;test_size = 150&#xA;df = read_csv('data/mastigadin.csv', header=None)&#xA;df.set_index(list(df)[0], inplace=True)&#xA;&#xA;y_total = df.iloc[:, -1:].values&#xA;x_total = df.iloc[:, :-1].values&#xA;y_train = y_total[:-test_size, :]&#xA;x_train = x_total[:-test_size, :]&#xA;y_test = y_total[-test_size:, :]&#xA;x_test = x_total[-test_size:, :]&#xA;&#xA;tempo = time.time()&#xA;&#xA;# Neural net&#xA;epochs = 200&#xA;batch_size = 64&#xA;optmizer = keras.optimizers.Adam()&#xA;model = Sequential()&#xA;model.add(Dense(256, input_shape=(x_train.shape[1],)))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(256, activation='relu'))&#xA;model.add(Dense(1))&#xA;&#xA;# fit&#xA;model.compile(loss='mean_squared_error', optimizer=optmizer)&#xA;model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_test, y_test), verbose=2)&#xA;print('TIME: ' + str(time.time() - tempo))&#xA;&#xA;# predict&#xA;y_trained = model.predict(x_train)&#xA;y_tested = model.predict(x_test)&#xA;&#xA;# errors&#xA;error_train = calculate_rmse(y_train, y_trained)&#xA;print('TRAIN: RMSE - ' + str(error_train))&#xA;error_test = calculate_rmse(y_test, y_tested)&#xA;print('\nVAL:   RMSE - ' + str(error_test))&#xA;&#xA;# plot&#xA;plt.plot(y_total, label='REAL DATA')&#xA;plt.plot(y_trained, label='TRAINED DATA')&#xA;plt.plot(range(len(y_train), len(y_total)), y_tested, label='TEST DATA')&#xA;plt.legend()&#xA;plt.title('KERAS')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Tensorflow code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;import time&#xA;import matplotlib.pyplot as plt&#xA;from pandas import read_csv&#xA;&#xA;&#xA;def calculate_rmse(real, predict):&#xA;    m = len(real)&#xA;    return np.sqrt(np.sum(np.power((real - predict), 2)) / m)&#xA;&#xA;&#xA;test_size = 150&#xA;df = read_csv('data/mastigadin.csv', header=None)&#xA;df.set_index(list(df)[0], inplace=True)&#xA;&#xA;y_total = df.iloc[:, -1:].values&#xA;x_total = df.iloc[:, :-1].values&#xA;y_train = y_total[:-test_size, :]&#xA;x_train = x_total[:-test_size, :]&#xA;y_test = y_total[-test_size:, :]&#xA;x_test = x_total[-test_size:, :]&#xA;n_samples = x_train.shape[0]&#xA;&#xA;tempo = time.time()&#xA;epochs = 200&#xA;batch_size = 64&#xA;&#xA;n_input = 36&#xA;n_output = 1&#xA;n_hidden = 256&#xA;&#xA;# tf Graph input&#xA;X = tf.placeholder(&quot;float&quot;, [None, n_input])&#xA;Y = tf.placeholder(&quot;float&quot;, [None, n_output])&#xA;&#xA;# Store layers weight &amp;amp; bias&#xA;weights = {&#xA;    'h1': tf.get_variable('h1', shape=[n_input, n_hidden]),&#xA;    'h2': tf.get_variable('h2', shape=[n_hidden, n_hidden]),&#xA;    'h3': tf.get_variable('h3', shape=[n_hidden, n_hidden]),&#xA;    'h4': tf.get_variable('h4', shape=[n_hidden, n_hidden]),&#xA;    'h5': tf.get_variable('h5', shape=[n_hidden, n_hidden]),&#xA;    'h6': tf.get_variable('h6', shape=[n_hidden, n_hidden]),&#xA;    'h7': tf.get_variable('h7', shape=[n_hidden, n_hidden]),&#xA;    'out': tf.Variable(tf.random_normal([n_hidden, n_output]))&#xA;}&#xA;biases = {&#xA;    'b1': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b2': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b3': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b4': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b5': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b6': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'b7': tf.Variable(tf.random_normal([n_hidden])),&#xA;    'out': tf.Variable(tf.random_normal([n_output]))&#xA;}&#xA;&#xA;&#xA;# Create model&#xA;def multilayer_perceptron(x):&#xA;    layer_1 = tf.nn.relu(tf.add(tf.matmul(x, weights['h1']), biases['b1']))&#xA;    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, weights['h2']), biases['b2']))&#xA;    layer_3 = tf.nn.relu(tf.add(tf.matmul(layer_2, weights['h3']), biases['b3']))&#xA;    layer_4 = tf.nn.relu(tf.add(tf.matmul(layer_3, weights['h4']), biases['b4']))&#xA;    layer_5 = tf.nn.relu(tf.add(tf.matmul(layer_4, weights['h5']), biases['b5']))&#xA;    layer_6 = tf.nn.relu(tf.add(tf.matmul(layer_5, weights['h6']), biases['b6']))&#xA;    layer_7 = tf.nn.relu(tf.add(tf.matmul(layer_6, weights['h7']), biases['b7']))&#xA;    out_layer = tf.matmul(layer_7, weights['out']) + biases['out']&#xA;    return out_layer&#xA;&#xA;&#xA;# Construct model&#xA;pred = multilayer_perceptron(X)&#xA;&#xA;# Define loss and optimizer&#xA;cost = tf.reduce_mean(tf.squared_difference(pred, Y))&#xA;optimizer = tf.train.AdamOptimizer()&#xA;train_op = optimizer.minimize(cost)&#xA;&#xA;# Initializing the variables&#xA;init = tf.global_variables_initializer()&#xA;&#xA;display_step = 1&#xA;with tf.Session() as sess:&#xA;    sess.run(init)&#xA;&#xA;    # Training cycle&#xA;    for epoch in range(epochs):&#xA;        avg_cost = 0.&#xA;        total_batch = int(n_samples / batch_size)&#xA;        # Loop over all batches&#xA;        tp = time.time()&#xA;        for i in range(total_batch):&#xA;            batch_x = x_train[i * batch_size:(i + 1) * batch_size]&#xA;            batch_y = y_train[i * batch_size:(i + 1) * batch_size]&#xA;            # Run optimization op (backprop) and cost op (to get loss value)&#xA;            _, c = sess.run([train_op, cost], feed_dict={X: batch_x, Y: batch_y})&#xA;            # Compute average loss&#xA;            avg_cost += c / total_batch&#xA;        # Display logs per epoch step&#xA;        if epoch % display_step == 0:&#xA;            print(&quot;Epoch:&quot;, '%04d' % (epoch + 1), &quot;cost={:.9f}&quot;.format(avg_cost), &quot;TIME: %.2f&quot; % (time.time() - tp))&#xA;    print('TIME: ' + str(time.time() - tempo))&#xA;&#xA;    # Test model&#xA;    y_trained = sess.run(pred, feed_dict={X: x_train})&#xA;    y_tested = sess.run(pred, feed_dict={X: x_test})&#xA;&#xA;    error_train = calculate_rmse(y_train, y_trained)&#xA;    print('TRAIN: RMSE - ' + str(error_train))&#xA;    error_test = calculate_rmse(y_test, y_tested)&#xA;    print('\nVAL:   RMSE - ' + str(error_test))&#xA;&#xA;plt.plot(y_total, label='REAL DATA')&#xA;plt.plot(y_trained, label='TRAINED DATA')&#xA;plt.plot(range(len(y_train), len(y_total)), y_tested, label='TEST DATA')&#xA;plt.legend()&#xA;plt.title('TENSORFLOW')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30314" LastActivityDate="2017-10-16T16:35:06.090" Title="Tensorflow MLP worse than Keras(TF backend)" Tags="|tensorflow|keras|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23845" PostTypeId="2" ParentId="23844" CreationDate="2017-10-16T16:35:06.090" Score="2" Body="&lt;p&gt;This part is badly enough wrong that you will get poor results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Store layers weight &amp;amp; bias&#xA;weights = {&#xA;    'h1': tf.get_variable('h1', shape=[n_input, n_hidden]),&#xA;    'h2': tf.get_variable('h2', shape=[n_hidden, n_hidden]),&#xA;    'h3': tf.get_variable('h3', shape=[n_hidden, n_hidden]),&#xA;    'h4': tf.get_variable('h4', shape=[n_hidden, n_hidden]),&#xA;    'h5': tf.get_variable('h5', shape=[n_hidden, n_hidden]),&#xA;    'h6': tf.get_variable('h6', shape=[n_hidden, n_hidden]),&#xA;    'h7': tf.get_variable('h7', shape=[n_hidden, n_hidden]),&#xA;    'out': tf.Variable(tf.random_normal([n_hidden, n_output]))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is the initialization. Your hidden layers have no initialization at all. The output layer initializes with likely the wrong scale. To match Keras, your initialiser should be something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.random_normal([n_in, n_out]) * (math.sqrt(2.0/(n_in + n_out))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or you can use the built-in Xavier initialiser:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.contrib.layers.xavier_initializer()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In addition, you can probably drop the initializer for the bias values.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-10-16T16:35:06.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23848" PostTypeId="1" CreationDate="2017-10-16T17:23:21.237" Score="1" ViewCount="1150" Body="&lt;p&gt;I have not GPU support so it often happens that my model takes hours to train. Can I train my model in batches , for example if I want to have 100 epochs for my model,but due to power cut my training stops(at 50th epoch) but when I retrain my model I want to train it from where it was left (from 50th epoch).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be much appreciated if anyone can explain it by some example.&lt;/p&gt;&#xA;" OwnerUserId="37502" LastEditorUserId="14675" LastEditDate="2017-10-16T20:58:26.283" LastActivityDate="2019-04-07T17:23:22.627" Title="How to resume training of a model?" Tags="|machine-learning|python|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23849" PostTypeId="2" ParentId="21759" CreationDate="2017-10-16T17:40:38.807" Score="3" Body="&lt;p&gt;A basic Naive Bayes is being used in this example. Each feature can have a number of different values within the ranges of 2 or 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bernoulli Naive Bayes requires that each feature be either true or false or 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multinomial Naive Bayes allows features to be of values 0+ as it is counting occurrences of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps. The Naive Bayes wikipedia pages summarizes this pretty well as well.&lt;/p&gt;&#xA;" OwnerUserId="40563" LastActivityDate="2017-10-16T17:40:38.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23850" PostTypeId="1" CreationDate="2017-10-16T17:58:11.880" Score="0" ViewCount="1006" Body="&lt;p&gt;I have a data frame like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; id                 Date       Volume      Price     Values(Volume*Price)&#xA;56033738624803469  20170111    1          943339    943339&#xA;56033738624803469  20170111    10         919410    9194100&#xA;56033738624803469  20170112    1          919410    919410&#xA;56033738624803469  20170112    5          954999    4774955&#xA;4659957480182399   20170207    1          1000000   1000000&#xA;4659957480182399   20170208    5          1000000   5000000&#xA;4659957480182399   20170208    40         1000000   40000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to compute and plot the following computation for the first 100 days for each ID:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;compute average values per day over the first 100 days&#xA;Plot this for all of Ids&#xA;Afterwards, the plot should like this:&#xA;Desired plot (&lt;a href=&quot;https://i.stack.imgur.com/2cozR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://i.stack.imgur.com/2cozR.png&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I've done so far:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df2 = df.groupby(['Id', 'Date']).sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                           Index   Volume        Price        Values&#xA;Id               Date                                               &#xA;1745829084228393 20170207      1     1000    1000000.0  1.000000e+09&#xA;                 20170208   5151   999000  101000000.0  9.990000e+11 &#xA;                 20170403      1       12    1000100.0  1.200120e+07&#xA;                 20170408      1       12    1000000.0  1.200000e+07&#xA;                 20170417      1      500    1000000.0  5.000000e+08&#xA;                 20170423      3    14500    2000000.0  1.450000e+10&#xA;                 20170507     10    35000    4000000.0  3.500000e+10&#xA;                 20170510     21    49051    6000000.0  4.905100e+10&#xA;                 20170529      1        4    1000000.0  4.000000e+06&#xA;2888358730233310 20170212    820  2000000   40000000.0  2.000000e+12&#xA;2929948497881810 20170207   1830  1500000   60000000.0  1.500000e+12&#xA;                 20170208    903   700000   42000000.0  7.000000e+11&#xA;                 20170212   1176   800000   48000000.0  8.000000e+11&#xA;3715246194918044 20150509     66     1008   11000000.0  1.008000e+09&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to calculate sum of first , second ,... of each ID for Values, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Date_Order     Sum_Values(= summation first date of each id )&#xA;&#xA; first_Date     (1.000000e+09 + 2.000000e+12 + 1.500000e+12 + 1.008000e+09)&#xA;&#xA; second_Date    (9.990000e+11 + 7.000000e+11)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ids are important i want summation of first day of values of each id(ids are different and may start on different days)&lt;/p&gt;&#xA;" OwnerUserId="40564" LastActivityDate="2017-12-16T10:04:35.133" Title="how can i sum first day value of each id together?" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23851" PostTypeId="2" ParentId="23841" CreationDate="2017-10-16T19:03:15.887" Score="1" Body="&lt;p&gt;As these methods are designed for categoricial data, you won't find visual examples as for k-means: the data is not that visual. A realistic example run will just be a boring list of updated modes / prototypes.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-10-16T19:03:15.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23852" PostTypeId="2" ParentId="23850" CreationDate="2017-10-16T19:09:25.577" Score="1" Body="&lt;p&gt;Just a suggestion:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the length of the most transaction over all ID be $n$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prepare a list of size $n$, $A$, initialize all entries to $0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now go through each ID, suppose a particular ID has $m$ transaction.  We add the value of the $i$-th transaction to the $i$-th entry of $A$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The list $A$ would have your desired information. The $i$-entry stores the sum of values of the $i$-th day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can also introduce a new feature that encode which transaction  is it for each ID and then use groupby to sum it up again.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2017-10-16T19:17:17.783" LastActivityDate="2017-10-16T19:17:17.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23853" PostTypeId="1" CreationDate="2017-10-16T20:00:13.353" Score="0" ViewCount="1051" Body="&lt;p&gt;So I am working on a deep RL model in OpenAI Gym. I have everything else working except for my function that allows my agent experience replay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from collections import deque&#xA;from keras.optimizers import Adam&#xA;import numpy as np&#xA;import gym&#xA;import tensorflow as tf&#xA;import math&#xA;import random&#xA;&#xA;&#xA;&#xA;env = gym.make('MountainCar-v0')&#xA;&#xA;#hyperparams&#xA;gamma = .99 #discount factor for reward&#xA;alpha = .01 #learning rate&#xA;beta  = .01 #decay&#xA;batch_size = 32&#xA;memory = deque(maxlen=2000)&#xA;epsilon = 1.0&#xA;&#xA;&#xA;#building model&#xA;model = Sequential()&#xA;model.add(Dense(12, activation='sigmoid', input_dim=2))&#xA;model.add(Dense(12, activation='tanh'))&#xA;model.add(Dense(2, activation='linear'))&#xA;model.compile(loss='mse', optimizer=Adam(lr=alpha, decay=beta))&#xA;&#xA;&#xA;def preprocess_state(state):&#xA;    return np.reshape(state, [1, 2])&#xA;&#xA;def experience(state, action, reward, next_state, done):&#xA;    # memory for replay&#xA;    memory.append((state, action, reward, next_state, done))&#xA;&#xA;class agent():&#xA;&#xA;&#xA;&#xA;&#xA;def choose_action(self, state):&#xA;    rand_number = np.random.randint(2)&#xA;# choosing random action 50%, acting greedily with respect to value func/policy (naive e-greedy)&#xA;    return env.action_space.sample() if (np.random.random() &amp;lt;= epsilon) else np.argmax(model.predict(state))&#xA;&#xA;def replay(self):&#xA;    x_batch, y_batch = [], []&#xA;    minibatch = random.sample(memory, batch_size)&#xA;    for state, action, reward, next_state, done in minibatch:&#xA;        y_target = model.predict(state)&#xA;        y_target[0][action] = reward if done else reward + gamma * np.max(model.predict(next_state)[0])&#xA;        x_batch.append(state[0])&#xA;        y_batch.append(y_target[0])&#xA;&#xA;&#xA;agent = agent()&#xA;&#xA;&#xA;def run():&#xA;&#xA;for episodes in range(1000):&#xA;    state = preprocess_state(env.reset())&#xA;    tot_reward = 0&#xA;    done = False&#xA;    while not done:&#xA;        env.render()&#xA;        action = agent.choose_action(state)&#xA;        next_state, reward, done, t = env.step(action)&#xA;        next_state = preprocess_state(next_state)&#xA;        #record results&#xA;&#xA;        experience(state,action,reward,next_state,done)&#xA;&#xA;        tot_reward += reward&#xA;&#xA;        #replay for agent&#xA;        if batch_size &amp;lt; len(memory):&#xA;            agent.replay()&#xA;&#xA;        print(&quot;episode: {} | score: {} |&quot;.format(&#xA;       episodes,tot_reward))&#xA;&#xA;        if done:&#xA;            print &quot;You did it&quot;&#xA;            break&#xA;&#xA;run()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The issue is with this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  y_target[0][action] = reward if done else reward + gamma * np.max(model.predict(next_state)[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the error: &lt;a href=&quot;https://i.stack.imgur.com/2ZeGn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2ZeGn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31046" LastEditorUserId="31046" LastEditDate="2017-10-16T20:17:03.617" LastActivityDate="2017-10-16T20:17:03.617" Title="Deep Reinforcent Learning Model, experience replay" Tags="|python|tensorflow|keras|reinforcement-learning|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23854" PostTypeId="2" ParentId="23806" CreationDate="2017-10-16T20:30:37.917" Score="1" Body="&lt;p&gt;It'll put it exactly where you asked for it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; destfile = &quot;Haberman.txt&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that is, a file called &lt;code&gt;Haberman.txt&lt;/code&gt; in the current working directory at the time you ran &lt;code&gt;download.file&lt;/code&gt;. You can get the current working directory with &lt;code&gt;getwd()&lt;/code&gt; in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example if its something you can read with &lt;code&gt;read.csv&lt;/code&gt;, you can do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hdata = read.csv(&quot;Haberman.txt&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as long as you haven't changed the current working directory (like with &lt;code&gt;setwd&lt;/code&gt;).&lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2017-10-16T20:30:37.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23855" PostTypeId="2" ParentId="23848" CreationDate="2017-10-16T20:58:09.633" Score="3" Body="&lt;p&gt;With tensorflow, currently the most straightforward and easy way to get persistence for your model is to use a &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/train/MonitoredTrainingSession&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.train.MonitoredTrainingSession&lt;/code&gt;&lt;/a&gt;. You just need to use it instead the normal &lt;code&gt;tf.Session()&lt;/code&gt; that is frequently used. This an illustrative Python snippet:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with tf.train.MonitoredTrainingSession(checkpoint_dir='/tmp/mymodel',&#xA;                                       save_summaries_secs=600) as sess:&#xA;   _ = sess.run(train_op, feed_dict={...})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this, your model is automagically saved every 600 secs in &lt;code&gt;/tmp/mymodel&lt;/code&gt; and restored the next time you restart the program.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-10-16T21:04:39.497" LastActivityDate="2017-10-16T21:04:39.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23857" PostTypeId="2" ParentId="23793" CreationDate="2017-10-17T00:08:56.420" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I've read about dummy variables, but as these three will depend on each other (if the value is &quot;red&quot;, then it's not &quot;black&quot;), can I use dummy variables?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, you can use dummy variables to represent a multilevel qualitative variable like &lt;code&gt;Color&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;ISLR Example&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;An example that may be helpful to you can be found in &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ISLR&lt;/a&gt;, chapter 3 on pages 85 and 86:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WraXa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WraXa.png&quot; alt=&quot;ISLR page 85&quot;&gt;&lt;/a&gt; &#xA;&lt;a href=&quot;https://i.stack.imgur.com/TgGDD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TgGDD.png&quot; alt=&quot;ISLR page 86&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results in the book can be recreated using the &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/Credit.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Credit&lt;/code&gt;&lt;/a&gt; data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example given in the book, &lt;code&gt;balance&lt;/code&gt; is regressed onto &lt;code&gt;ethnicity&lt;/code&gt;. That is, &lt;code&gt;balance&lt;/code&gt; is the quantitative dependent variable (if it were qualitative this would be a classification problem rather than a regression problem), and &lt;code&gt;ethnicity&lt;/code&gt; is a 3-level qualitative variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Credit &amp;lt;- read.csv(&quot;http://www-bcf.usc.edu/~gareth/ISL/Credit.csv&quot;, header = TRUE)&#xA;&amp;gt; attach(Credit)&#xA;&amp;gt; summary(Balance)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;   0.00   68.75  459.50  520.01  863.00 1999.00 &#xA;&#xA;&amp;gt; summary(Ethnicity)&#xA;African American            Asian        Caucasian &#xA;              99              102              199 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The task is to predict &lt;code&gt;Balance&lt;/code&gt; based on &lt;code&gt;Ethnicity&lt;/code&gt;. Therefore &lt;code&gt;Ethnicity&lt;/code&gt; needs to be dummified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As stated in the text pictured above, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There will always be one fewer dummy variable than the number of levels. The level with no dummy variable—African American in this example—is known as the &lt;em&gt;baseline&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;is.Caucasian &amp;lt;- ifelse(Ethnicity==&quot;Caucasian&quot;, 1, 0)&#xA;is.Asian &amp;lt;- ifelse(Ethnicity==&quot;Asian&quot;, 1, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above code accomplishes creating dummy variables for Caucasian and Asian, with African American as the baseline. Now to create the linear model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;balance.lm &amp;lt;- lm(Balance ~ is.Caucasian + is.Asian, Credit)&#xA;summary(balance.lm)&#xA;&#xA;Call:&#xA;lm(formula = Balance ~ is.Caucasian + is.Asian, data = Credit)&#xA;&#xA;Residuals:&#xA;    Min      1Q  Median      3Q     Max &#xA;-531.00 -457.08  -63.25  339.25 1480.50 &#xA;&#xA;Coefficients:&#xA;             Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)    531.00      46.32  11.464   &amp;lt;2e-16 ***&#xA;is.Caucasian   -12.50      56.68  -0.221    0.826    &#xA;is.Asian       -18.69      65.02  -0.287    0.774    &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 460.9 on 397 degrees of freedom&#xA;Multiple R-squared:  0.0002188, Adjusted R-squared:  -0.004818 &#xA;F-statistic: 0.04344 on 2 and 397 DF,  p-value: 0.9575&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The results are an exact match to the information displayed in Table 3.8 on page 86.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example using 3 colors&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Let's say we wanted to predict the mass of an ball using its color. In this case quantitative variable &lt;code&gt;mass&lt;/code&gt; is the response (dependent variable) and &lt;code&gt;color&lt;/code&gt; is the independent variable.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; color &amp;lt;- c(&quot;red&quot;, &quot;black&quot;, &quot;green&quot;, &quot;red&quot;, &quot;green&quot;, &quot;red&quot;, &quot;red&quot;)&#xA;&amp;gt; mass &amp;lt;- c(   12,      40,       1,    15,       2,    14,    14)&#xA;&amp;gt; ball.data &amp;lt;- data.frame(mass, color)&#xA;&amp;gt; head(ball.data)&#xA;&#xA;  mass color&#xA;1   12   red&#xA;2   40 black&#xA;3    1 green&#xA;4   15   red&#xA;5    2 green&#xA;6   14   red&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can dummify &lt;code&gt;color&lt;/code&gt; into two variables, one for red and one for black, with green as the baseline.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ball.data$is.red &amp;lt;- ifelse(ball.data$color==&quot;red&quot;, 1, 0)&#xA;&amp;gt; ball.data$is.black &amp;lt;- ifelse(ball.data$color==&quot;black&quot;, 1, 0)&#xA;&amp;gt; head(ball.data)&#xA;&#xA;  mass color is.red is.black&#xA;1   12   red      1        0&#xA;2   40 black      0        1&#xA;3    1 green      0        0&#xA;4   15   red      1        0&#xA;5    2 green      0        0&#xA;6   14   red      1        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we fit a linear model like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ball.lm &amp;lt;- lm(mass ~ is.red + is.black, ball.data)&#xA;&amp;gt; summary(ball.lm)&#xA;&#xA;Call:&#xA;lm(formula = mass ~ is.red + is.black, data = ball.data)&#xA;&#xA;Residuals:&#xA;         1          2          3          4          5          6          7 &#xA;-1.750e+00  5.551e-17 -5.000e-01  1.250e+00  5.000e-01  2.500e-01  2.500e-01 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)   1.5000     0.8101   1.852 0.137727    &#xA;is.red       12.2500     0.9922  12.347 0.000247 ***&#xA;is.black     38.5000     1.4031  27.439 1.05e-05 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 1.146 on 4 degrees of freedom&#xA;Multiple R-squared:  0.9947,    Adjusted R-squared:  0.9921 &#xA;F-statistic: 376.7 on 2 and 4 DF,  p-value: 2.79e-05&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When we compare the coefficient estimates in the linear model summary to the values in the data set, we see they are pretty close, as expected. That is about it.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;More info&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://onlinecourses.science.psu.edu/stat501/node/301&quot; rel=&quot;nofollow noreferrer&quot;&gt;PSU STAT 501: Regression Methods, Lesson 8: Categorical Predictors&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.idre.ucla.edu/r/modules/coding-for-categorical-variables-in-regression-models/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CODING FOR CATEGORICAL VARIABLES IN REGRESSION MODELS | R LEARNING MODULES&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26788" LastEditorUserId="26788" LastEditDate="2017-10-17T01:24:28.913" LastActivityDate="2017-10-17T01:24:28.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23858" PostTypeId="1" CreationDate="2017-10-17T00:41:52.770" Score="0" ViewCount="623" Body="&lt;p&gt;As I understand we can apply community detection algorithms such as Louvain to detect communities in a social network (i.e. involves people). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am quite interested in knowing if we can use the same community detection algorithms such as Louvain to identify communities in word vector space (e.g., word2vec), instead of clustering?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="37986" LastEditDate="2017-10-17T00:55:58.213" LastActivityDate="2023-05-23T04:00:43.747" Title="Can we apply community detection algorithms for word vector space?" Tags="|machine-learning|data-mining|text-mining|social-network-analysis|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="23859" PostTypeId="2" ParentId="23803" CreationDate="2017-10-17T03:27:01.160" Score="0" Body="&lt;p&gt;A simple solution to this problem would be to first use some algorithm to perform regression to predict the dependent variable (since the dependent variable is quantitative) and then group these predictions into buckets of the desired width using a loop to iterate over the predicted values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vector of predicted values looks like this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;(165.9, 163.2, 169.1, 171.7, 178.3)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the categories are based on intervals of 5, then 165.9 -&gt; {160 - 165}, ... , 178.3 -&gt; {176 - 180}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The point is that the grouping of values into categories based on some arbitrary interval occurs separately from the regression. Doing such a thing results in significant information loss regarding the predictions.&lt;/p&gt;&#xA;" OwnerUserId="26788" LastActivityDate="2017-10-17T03:27:01.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23860" PostTypeId="2" ParentId="19782" CreationDate="2017-10-17T03:51:36.520" Score="7" Body="&lt;p&gt;One reason to discretize continuous features is to improve the signal-to-noise ratio.  Fitting a model using features that have been binned reduces the impact that small fluctuations in the data have on the model, often small fluctuates are just noise. Each bin &amp;quot;smooths&amp;quot; out the fluctuates/noises in sections of the data.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2023-08-20T20:17:03.370" LastActivityDate="2023-08-20T20:17:03.370" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="23862" PostTypeId="1" CreationDate="2017-10-17T06:18:28.893" Score="-2" ViewCount="95" Body="&lt;p&gt;Given time-series data of a set of input metrics $M_1$, $M_2$, $M_3$ ... $M_n$ and a set of observed metrics $O_1$, $O_2$, $O_3$ ... $O_m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each $M_i$, what is the best way to find equations that best describe the relationship between $M_i$ and the set of observed metrics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is given that all $M_i$ are independent of each other.&lt;/p&gt;&#xA;" OwnerUserId="40584" LastActivityDate="2017-10-17T08:27:01.030" Title="Finding relationships from group of variables" Tags="|machine-learning|time-series|statistics|data|correlation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23864" PostTypeId="2" ParentId="23862" CreationDate="2017-10-17T08:27:01.030" Score="0" Body="&lt;p&gt;It is a pretty difficult task to obtain a function $f_1$ satisfying $M_1(t) = f_1(O_1(t), O_2(t),...)$. It is mostly like a regression problem where you are trying to fit $M_1(t)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could suggest three approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Simple multiple regression. &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/linear_model.html&lt;/a&gt;. If that doesn't work, you could increase the order of your observed metric space $O$ using PolynomialFeatures &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html&lt;/a&gt; and then try multiple regression. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don't really fancy these other two approaches but will list them anyway, in case you want to have a go.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Multivariate Adaptive Regression Splines &lt;a href=&quot;http://contrib.scikit-learn.org/py-earth/content.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://contrib.scikit-learn.org/py-earth/content.html&lt;/a&gt;&#xA;You will get a good fit but the function will be discrete in nature.&lt;/li&gt;&#xA;&lt;li&gt;OpenMLC &lt;a href=&quot;https://github.com/MachineLearningControl/OpenMLC-Python/releases&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/MachineLearningControl/OpenMLC-Python/releases&lt;/a&gt; which uses Genetic Algorithms. This will be able to provide you with functions like $sin(O_i), log(O_i), exp(O_i)$ as well. But the problem is that since it uses genetic algorithms, you won't obtain a deterministic function. The function will change each time you run it with varying accuracy.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-17T08:27:01.030" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23867" PostTypeId="1" CreationDate="2017-10-17T10:33:30.893" Score="1" ViewCount="1064" Body="&lt;p&gt;In Keras I have to predict 47 columns and my query is below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense, Activation&#xA;&#xA;model_1 = Sequential([ Dense(32, input_shape=(192,)), Activation('relu'), Dense(47), Activation('softmax'), ])&#xA;&#xA;model_1.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&#xA;model_1.fit(standardized_X_train, train_y, batch_size=32, epochs=15, verbose=1, validation_data=(standardized_X_test, test_y))&#xA;&#xA;classes = model.predict(x_test, batch_size=128)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, when I try to predict all my columns are showing NaN as predictions. Any reason as to why this is happening?&lt;/p&gt;&#xA;" OwnerUserId="30000" LastEditorUserId="13748" LastEditDate="2017-10-17T16:54:34.313" LastActivityDate="2017-10-17T16:54:34.313" Title="Predicting multiple columns in Keras" Tags="|neural-network|tensorflow|keras|" AnswerCount="0" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="23868" PostTypeId="1" AcceptedAnswerId="23880" CreationDate="2017-10-17T10:43:28.863" Score="4" ViewCount="2507" Body="&lt;p&gt;I am required to implement a simple perceptron based neural network for an image classification task, with a binary output and a single layer, however I am having difficulties. I have a few problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I am required to use a tanh() axtivation function, which has the range [-1, 1], however, my training labels are 1 and 0. Should I scale the activation function, or simply change 0 labels to -1?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;So the gradient descent rule dictates that I shift the weights according to the rule: &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\Delta \omega = -\eta \frac{\delta E}{\delta \omega}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the mean square error for my error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;E = (output - label)^2&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Considering my output is $o = tanh(\omega .x)$, x is my input vector and $y_i$ is the corresponding label here. &#xA;$$&#xA;\frac{\delta E}{\delta \omega} = \frac{\delta (y_i - tanh(wx))^2}{\delta \omega} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) \frac{\delta wx}{\delta w} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) x \\= -2(y_i - o)(1 - o^2)x&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I implemented this is python, the dot product of the input vector with the weights turns out to be too large, which makes $tanh(x)=1$ and $1-o^2 = 0$, so I can't learn. How can I circumvent this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the replies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The implementation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def perc_nnet(X, y, iter = 10000, eta=0.0001):&#xA;    a, b, c = X.shape&#xA;    W_aug = np.random.normal(0, 0.01, a*b+1)&#xA;    errors = []&#xA;&#xA;    for i in range(iter):&#xA;        selector = rd.randint(0,c-1)&#xA;        x_n = X[:,:,selector].ravel() #.append(1) #has the bias as well&#xA;        x_n = np.append(x_n, 1)&#xA;        v = x_n.dot(W_aug)&#xA;        o = np.tanh(v)&#xA;        y_i = y[:,selector] if y[:,selector]==1 else -1&#xA;        MSE = 0.5*(o - y_i)**2&#xA;        errors.append(MSE)&#xA;        delta = - eta * (o - y_i) * (1 - o**2) * x_n&#xA;        W_aug = W_aug + delta&#xA;&#xA;    return W_aug, errors&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40590" LastEditorUserId="40590" LastEditDate="2017-10-17T13:05:44.287" LastActivityDate="2017-10-18T06:06:13.590" Title="How to implement gradient descent for a tanh() activation function for a single layer perceptron?" Tags="|neural-network|gradient-descent|perceptron|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23870" PostTypeId="1" CreationDate="2017-10-17T11:56:28.187" Score="1" ViewCount="268" Body="&lt;p&gt;I heard that the activation could be thought of as a unnormalized angle between input and nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How is this true?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have&#xA;$\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the further I could get. Any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input  neuron holds a vector? Or is the vector formulation a shortcut for the entire network?&lt;/p&gt;&#xA;" OwnerUserId="31316" LastActivityDate="2017-10-17T11:56:28.187" Title="neural networks: activation with tanh is an angle?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23872" PostTypeId="1" CreationDate="2017-10-17T14:56:00.873" Score="0" ViewCount="628" Body="&lt;p&gt;Let’s say I am modelling data on flights and delays. I have 14 airlines (&lt;code&gt;carrier&lt;/code&gt; variable) in the data below. I would like to know if I can create fewer categories or clusters of this categorical data, so instead of having 14 airlines maybe end up with 3 or 4 clusters of airlines. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(data.table)&#xA;# Data takes ~30 seconds to load&#xA;flights &amp;lt;- as.data.table(read.csv(url(&quot;https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv&quot;)))&#xA;length(unique(flights$carrier))&#xA;&#xA;km &amp;lt;- kmeans(flights[,.(dep_delay,carrier)], centers = 4, nstart = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any methodology that could be used? I tried k-means but it seems it is not the most appropriate. &lt;/p&gt;&#xA;" OwnerUserId="30786" LastEditorUserId="30786" LastEditDate="2017-10-19T12:48:48.173" LastActivityDate="2017-10-21T22:55:25.510" Title="Cluster a categorical variable without breaking the existing categories" Tags="|r|clustering|k-means|categorical-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23876" PostTypeId="2" ParentId="23803" CreationDate="2017-10-17T17:02:17.060" Score="0" Body="&lt;p&gt;Random forests can be used for BOTH classification and regression, so you can use this algorithm directly with your data as is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will still have the benefits you quoted, i.e a way to estimate the confidence of the outputted value. &lt;/p&gt;&#xA;" OwnerUserId="40331" LastActivityDate="2017-10-17T17:02:17.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23877" PostTypeId="2" ParentId="23728" CreationDate="2017-10-17T19:00:23.437" Score="9" Body="&lt;p&gt;Basically, we add a regularization term in order to prevent the coefficients to fit so perfectly to overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between L1 and L2 is L1 is the sum of weights and L2 is just the sum of the square of weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L1 cannot be used in gradient-based approaches since it is not-differentiable unlike L2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L1  helps perform feature selection in sparse feature spaces.Feature selection is to know which features are helpful and which are redundant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference between their properties can be summarized as :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m3otE.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m3otE.png&quot; alt=&quot;l1 vs l2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40615" LastEditorUserId="40615" LastEditDate="2017-10-17T19:18:17.883" LastActivityDate="2017-10-17T19:18:17.883" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23878" PostTypeId="2" ParentId="23696" CreationDate="2017-10-17T23:36:20.410" Score="5" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/17282/xgbregressor-vs-xgboost-train-huge-speed-difference&quot;&gt;XGBRegressor vs. xgboost.train huge speed difference?&lt;/a&gt; - see this answer, likely the defaults (num_boost_round=10 vs. n_estimators=100) are the cause of this.&lt;/p&gt;&#xA;" OwnerUserId="20894" LastActivityDate="2017-10-17T23:36:20.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23880" PostTypeId="2" ParentId="23868" CreationDate="2017-10-18T06:06:13.590" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How can I circumvent this problem?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TLDR :&lt;/strong&gt; Normalize your input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice how tanh actually works on the input data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000])) &#xA;array([-1. , -1., -1. , -0.76159416,  0.,0.76159416,  1. ,1.   , 1])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would explain&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000)&#xA;array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001,  0.00999967,  0.09966799,  0.76159416])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, tanh treats all of them differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&#xA;You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to why should be a whole different answer.&#xA;&lt;a href=&quot;https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a link for the same.&lt;/p&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-10-18T06:06:13.590" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23881" PostTypeId="1" CreationDate="2017-10-18T07:29:13.473" Score="0" ViewCount="523" Body="&lt;p&gt;So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is &gt;50k. &#xA;P.S : this is not the known ADULT dataset (which is a lot cleaner), the difference is that mine wants to predict the total person income rather than the adjusted gross income.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that some of the attributes such as : &lt;em&gt;class of worker&lt;/em&gt; (meaning if he's self employed, local government...),  &lt;em&gt;major occupation and and industry codes&lt;/em&gt; have 100 000 missing values knowing that the total length of my dataset is 150 000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now it bugs me to simply drop those column because it seems to me like an important information to know if we want to predict the total income.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do in such a position ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks ! &lt;/p&gt;&#xA;" OwnerUserId="33242" LastActivityDate="2017-10-18T18:04:30.963" Title="Filling missing values for important features" Tags="|machine-learning|data-cleaning|feature-engineering|missing-data|" AnswerCount="2" CommentCount="0" ClosedDate="2017-10-19T15:15:11.333" ContentLicense="CC BY-SA 3.0" />
  <row Id="23884" PostTypeId="1" CreationDate="2017-10-18T11:30:33.967" Score="0" ViewCount="210" Body="&lt;p&gt;So right now I am trying to create multiple variables with training data, and in the process I have reached an error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in eval(predvars, data, env) : object '1.band1' not found&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;which is a product of these lines:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;for(i in 1:length(data_split)){&#xA; assign(paste(&quot;fit.lda&quot;,i, sep = &quot;&quot;), train(class~., data=data_split[i], &#xA; method=&quot;lda&quot;, metric=metric, trControl=control))}&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is it something that I did wrong or is it something that can be fixed with another methodology&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &#xA;My dataset is a data frame which was created by the following lines&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;validation_index &amp;lt;- createDataPartition(valuetable$class, p=0.80, list=FALSE)&#xA;dataset &amp;lt;- valuetable[validation_index,]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;valuetable&lt;/code&gt; is a dataframe which containes the values of multiple rasters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I wanted to split my data so my train function can be processed faster, that is because my machine's specs are pretty low, thus allowing me to re-create the later the results that would take time using other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Let me know if you want more clarification&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38188" LastEditorUserId="38188" LastEditDate="2017-10-18T12:01:50.833" LastActivityDate="2023-12-23T17:03:35.423" Title="Spliting training data into multiple variables using R" Tags="|r|bigdata|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23885" PostTypeId="2" ParentId="23884" CreationDate="2017-10-18T11:41:48.110" Score="0" Body="&lt;p&gt;You didn't explained how is your dataset and its variables, so I'll put a generic example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say your dataset has a variable called &lt;code&gt;SET&lt;/code&gt;, whose content is &lt;code&gt;TRN&lt;/code&gt; for training and &lt;code&gt;TST&lt;/code&gt; for test. To create a dataset containing only training set you have to use the following code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;trn.ds&amp;lt;- ds[ds$SET == 'TRN',]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need a test dataset, just adapt the code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;tst.ds&amp;lt;- ds[ds$SET == 'TST']&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: It wasn't me who downvoted your question, I don't understand why people downvote almost every question here.&lt;/p&gt;&#xA;" OwnerUserId="30114" LastActivityDate="2017-10-18T11:41:48.110" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23886" PostTypeId="2" ParentId="23881" CreationDate="2017-10-18T13:37:09.983" Score="2" Body="&lt;p&gt;I would first build some models without those predictors. If 2/3 of your training set is missing so you can't have your basic model based on them. If your accuracy is too low I would try to use a smaller data set and not use the missing values. Those kind of missing values are not somthing that you can easily define (like a default job) unless you have a prior on your data (e.g most people in the data are data scientists). If you do have any kind of such prior you can use missing data methods more efficiently.&lt;/p&gt;&#xA;" OwnerUserId="38226" LastActivityDate="2017-10-18T13:37:09.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23887" PostTypeId="1" CreationDate="2017-10-18T15:04:52.723" Score="2" ViewCount="89" Body="&lt;p&gt;After applying PCA and working with the reduced dataset, I want to delete the outliers. To do this my idea is to compute the kNN-graph and delete those vertices (points) that have an inner degree of 0. That is that they are not in the neighborhood of any other point besides itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is if it is a good idea to instead of working with the $l_2$ distance, I should add weights that take into account the variance explained by each coordinate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that if the variance explained by coord 1 is 0.93, and by coordinate 2 only 0.05, then the points are expected to be &quot;close together&quot; on the projection over coord 2. It seems natural that the distance on the second coordinate should be much more important than the first one for outlier detection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First approach was to simply use the inverse of the variance explained as weights. Does that idea make sense? If so, what is the proper way to implement it?&lt;/p&gt;&#xA;" OwnerUserId="31736" LastEditorUserId="31736" LastEditDate="2017-10-18T15:13:13.727" LastActivityDate="2017-10-18T15:13:13.727" Title="Outlier detection: Should the metric used in kNN take into account variance explained by each coordinate?" Tags="|outlier|distance|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23890" PostTypeId="1" CreationDate="2017-10-18T17:25:00.960" Score="-3" ViewCount="2365" Body="&lt;p&gt;I'm currently tring to train a model with XGBoost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset has ~7 million records and 61 columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I'm currently having is that I get a MemoryError on python when I try to train the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Investigating on the XGboost documentation I can see that I can train a pre trained model, so I thought I could implement a sequential training dividing my data into smaller pieces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have anyone already done this? If so, could you please share or recommend the way to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using the SkLearn XGBClassifier model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank!&lt;/p&gt;&#xA;" OwnerUserId="40653" LastActivityDate="2017-10-27T16:20:37.027" Title="Training XGBoost sequentially" Tags="|python|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23891" PostTypeId="2" ParentId="23881" CreationDate="2017-10-18T18:04:30.963" Score="0" Body="&lt;p&gt;I would say before drop the columns if could verify the entropy (&lt;a href=&quot;http://www.math.unipd.it/~aiolli/corsi/0708/IR/Lez12.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;see here&lt;/a&gt;) of each one to see if they are important feature to your process or not. This idea is more common in classification problems, but maybe you can do some adjustments on yours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For numeric features you can try work with the simple interpolation solution (&lt;a href=&quot;https://www.google.com.br/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjZ_bKx4PrWAhXBEpAKHQbeDYMQFggzMAA&amp;amp;url=http%3A%2F%2Fwww.icmc.sc.usp.br%2F~gbatista%2Ffiles%2Faai2003.pdf&amp;amp;usg=AOvVaw0P4FJ-OK_dUDw9KbPUm5In&quot; rel=&quot;nofollow noreferrer&quot;&gt;see here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally I like a book from Dr Graham about how to handle missing data called Missing Data Analysis and Design.&lt;/p&gt;&#xA;" OwnerUserId="40656" LastActivityDate="2017-10-18T18:04:30.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23894" PostTypeId="1" CreationDate="2017-10-18T19:40:56.623" Score="8" ViewCount="23762" Body="&lt;p&gt;So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import OneHotEncoder&#xA;onehotencoder=OneHotEncoder(categorical_features=[10])&#xA;Y= onehotencoder.fit_transform(X).toarray()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However i get the following error &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: could not convert string to float:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A bit of information Y is converted to an object from a df using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y=df.iloc[:,:].values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to oneHotencode the 10th column of y which contains string values. The type of Y in variable explorer is object and if execute &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;type(Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i get numpy.ndarray&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm new to Pandas and sklearn and would really appreciate any help.&lt;/p&gt;&#xA;" OwnerUserId="40659" LastActivityDate="2020-10-22T12:48:18.180" Title="issue with oneHotEncoding" Tags="|python|scikit-learn|preprocessing|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23895" PostTypeId="1" AcceptedAnswerId="25737" CreationDate="2017-10-18T20:30:52.027" Score="44" ViewCount="57066" Body="&lt;p&gt;How we can program in the Keras library (or TensorFlow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPUs and you would like to use all of them to train faster, but your code is just for a single CPU or GPU.&lt;/p&gt;&#xA;" OwnerUserId="40665" LastEditorUserId="85045" LastEditDate="2021-01-01T06:18:06.760" LastActivityDate="2021-01-01T06:18:06.760" Title="Multi GPU in Keras" Tags="|python|deep-learning|tensorflow|keras|gpu|" AnswerCount="4" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="23897" PostTypeId="2" ParentId="11118" CreationDate="2017-10-19T02:10:55.067" Score="10" Body="&lt;p&gt;I'm no expert but it looks like AlphaGo Zero answers your question.&#xA;&lt;a href=&quot;https://deepmind.com/blog/alphago-zero-learning-scratch/&quot; rel=&quot;noreferrer&quot;&gt;https://deepmind.com/blog/alphago-zero-learning-scratch/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Previous versions of AlphaGo initially trained on thousands of human&#xA;  amateur and professional games to learn how to play Go. AlphaGo Zero&#xA;  skips this step and learns to play simply by playing games against&#xA;  itself, starting from completely random play. In doing so, it quickly&#xA;  surpassed human level of play and defeated the previously published&#xA;  champion-defeating version of AlphaGo by 100 games to 0.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="14284" LastActivityDate="2017-10-19T02:10:55.067" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="23898" PostTypeId="1" CreationDate="2017-10-19T02:32:06.927" Score="1" ViewCount="233" Body="&lt;p&gt;Chess is Exp-complete. I assume Alpha GO is also Exp complete. Correct me if I am wrong. Now deep reinforcement learning has been successful in tackling this problem. What about other more practical problems like scheduling, bin-packing, and routing? &#xA;In particular, what is the state of the art for scheduling applications using ML techniques? How should one start in this direction?&lt;/p&gt;&#xA;" OwnerUserId="26493" LastEditorUserId="26493" LastEditDate="2017-10-20T01:15:51.143" LastActivityDate="2017-10-20T01:15:51.143" Title="Is machine learning successful in solving combinatorial optimisation problems in NP-hard? Discuss problem of scheduling using machine learning" Tags="|machine-learning|optimization|research|" AnswerCount="1" CommentCount="1" ClosedDate="2017-10-19T15:14:35.647" ContentLicense="CC BY-SA 3.0" />
  <row Id="23899" PostTypeId="2" ParentId="23898" CreationDate="2017-10-19T06:33:43.390" Score="3" Body="&lt;p&gt;You don't have to get to EXP-complete in order to get a hard problem.&#xA;NP-Complete is bad enough...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cryptographic assumptions (e.g., the existence of &lt;a href=&quot;https://en.wikipedia.org/wiki/One-way_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;one way functions&lt;/a&gt;) are also a good way to create hard problems&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is research in the area lead of plenty of results like &lt;a href=&quot;https://www.cis.upenn.edu/~mkearns/papers/crypto.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cryptographic Limitations on Learning Boolean Formulae and Finite Automata&lt;/a&gt; - &lt;a href=&quot;https://en.wikipedia.org/wiki/Deterministic_finite_automaton&quot; rel=&quot;nofollow noreferrer&quot;&gt;DFA&lt;/a&gt; is not &lt;a href=&quot;https://en.wikipedia.org/wiki/Probably_approximately_correct_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;PAC&lt;/a&gt; learnable in polynomial time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is an important difference between these negative results and writing a great chess program.&#xA;The negative results means that for a given problem no algorithm will solve ALL instances (let say, in polynomial time).&#xA;That doesn't mean that you won't be able to play better the human champion on some/most/all instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this complexity observation is indifference of the technique. Machine learning is just another (useful) way to cope with these problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for machine learning ability to cope with these problems, you can find NP-Complete problems all around in machine learning (SVM, Neural Networks, Decision trees). That didn't prevent them from finding many useful models.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-10-19T06:33:43.390" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23901" PostTypeId="1" CreationDate="2017-10-19T07:30:01.827" Score="9" ViewCount="4921" Body="&lt;p&gt;Formal definition that I have seen of concept class is&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;class of all true functions&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;mathematically :&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$f:X \rightarrow\{0,1\}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;and that of hypothesis is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$h:X \rightarrow\{0,1\}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;But most of the times they are used together. For example in definition of PAC&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;A concept class  is PAC learnable by a learner  using hypothesis space  if for all concepts ∈, distributions over , true error probability 0≤≤1/2, failure probability 0≤≤1/2, learner  outputs a hypothesis ℎ∈ such that&lt;/p&gt;&#xA;&lt;p&gt;True error less than or equal to &lt;/p&gt;&#xA;&lt;p&gt;Computational time is polynomial in 1/,1/, representation size of data object, and representation size of concept&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;What is the difference?&lt;/p&gt;&#xA;" OwnerUserId="40687" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2021-10-26T06:59:13.307" Title="What is the difference between concept class and hypothesis" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23902" PostTypeId="2" ParentId="23872" CreationDate="2017-10-19T09:06:16.310" Score="1" Body="&lt;p&gt;Without looking at the data, can't you just use your Domain Knowledge to create meaningful groups/attributes for the airlines data frame? Such as &quot;low-cost carrier/premium&quot;, &quot;foreign (non-USA) airline/US-Airline&quot; , &quot;long-distance carrier/domestic only&quot;, &quot;founded in decade&quot; ,  or whatever? With only 14 airlines you can assign appropriate values yourself, manually, without relying on some cryptic algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, is your dataset identical to the dataset in the R package &quot;nycflights13&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, you could easily load it with &lt;code&gt;library(nycflights13)&lt;/code&gt;, after installing the package of course.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-10-19T09:06:16.310" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23903" PostTypeId="2" ParentId="6964" CreationDate="2017-10-19T11:53:49.823" Score="7" Body="&lt;p&gt;&lt;a href=&quot;https://keras.rstudio.com/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt; is also now available for R. Here's an &lt;a href=&quot;https://cran.r-project.org/web/packages/kerasR/vignettes/introduction.html#recurrent-neural-networks-rnn-with-imdb&quot; rel=&quot;noreferrer&quot;&gt;example&lt;/a&gt; of an LSTM with the R API.&lt;/p&gt;&#xA;" OwnerUserId="40694" LastActivityDate="2017-10-19T11:53:49.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23904" PostTypeId="2" ParentId="23894" CreationDate="2017-10-19T12:47:22.617" Score="5" Body="&lt;p&gt;There is an easy way to use one hot encoding in pandas and you can read about it in the following link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;noreferrer&quot;&gt;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;get_dummies&lt;/code&gt; method is easy and optimized for use in pandas Data Frames.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="38226" LastActivityDate="2017-10-19T12:47:22.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23905" PostTypeId="2" ParentId="23894" CreationDate="2017-10-19T13:07:46.363" Score="3" Body="&lt;p&gt;You need two steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;LabelEncoder&lt;/code&gt;&lt;/a&gt; to encode your string variables to integers&lt;/li&gt;&#xA;&lt;li&gt;Then use &lt;code&gt;OneHotEncoder&lt;/code&gt; on your integer variables&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40693" LastActivityDate="2017-10-19T13:07:46.363" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23906" PostTypeId="1" AcceptedAnswerId="26760" CreationDate="2017-10-19T13:12:57.687" Score="1" ViewCount="83" Body="&lt;p&gt;This is an issue for all Data Scientists who have worked with this stack:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;python&lt;/li&gt;&#xA;&lt;li&gt;scikit-learn&lt;/li&gt;&#xA;&lt;li&gt;scipy-stats&lt;/li&gt;&#xA;&lt;li&gt;matplotlib&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We are looking for ways to have a project already implemented in the aforementioned stack scale for very large datasets by doing the minimum amount of work&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Counter examples would be to rewrite everything in Tensorflow framework or use industry tools that are unrelated with Python.&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2018-01-17T17:07:40.397" Title="What is the easiest way to scale a data science project based on scikit stack?" Tags="|bigdata|scalability|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23907" PostTypeId="2" ParentId="23736" CreationDate="2017-10-19T13:16:45.763" Score="1" Body="&lt;p&gt;Considering you extracted the news using a TF-IDF approach what you have as a result is just one feature (frequency of terms). I think you would need to add more features to your corpus to be able to match two news as the same (or similar).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One new feature would be the temporal, where you would add a timestamp for the news. It will allow you check if two news (from different publishers) were published on the same period. (a week, 2 weeks, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second one could be a spatial, if you have geolocation of the news for example, you can add it to your train dataset. Something similar was done by &lt;a href=&quot;http://sci-hub.cc/10.1145/1998076.1998141&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chi-Chun Pan&lt;/a&gt;. It would allow you be more confident if the news happened on the same place.&lt;/p&gt;&#xA;" OwnerUserId="40656" LastActivityDate="2017-10-19T13:16:45.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23908" PostTypeId="2" ParentId="23796" CreationDate="2017-10-19T14:37:31.770" Score="0" Body="&lt;p&gt;Creating new columns for each variable with 0s and 1s is a technique called &quot;One Hot Encoding&quot; and I think it's fair to say that it is a standard way to treat categorical variables when making use of algorithms that cannot use categorical variables. As you correctly identify, it can cause problems by drastically increasing the dimensionality of the problem, especially if you have many levels in your categorical variables. Other encoding techniques commonly employed, such as &lt;a href=&quot;http://contrib.scikit-learn.org/categorical-encoding/helmert.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;helmert&lt;/a&gt;, equally increase the dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative encoding is &lt;a href=&quot;http://contrib.scikit-learn.org/categorical-encoding/binary.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary&lt;/a&gt; coding. In binary coding, the category levels are first encoded as ordinal (numbered sequentially). The ordinal codes are converted into binary and the binary digits are split into separate columns. To encode 30 unique cities, you would need 5 columns. The dimensionality is increased by the number of digits required to represent the number of levels in binary, for each categorical variable.&lt;/p&gt;&#xA;" OwnerUserId="30964" LastActivityDate="2017-10-19T14:37:31.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23909" PostTypeId="2" ParentId="23906" CreationDate="2017-10-19T15:10:22.697" Score="1" Body="&lt;p&gt;You generally don't.  Scikit-learn is primarily aimed to help new data scientists quickly get comfortable with data science&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, some strategies for scaling are discussed here: &lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/scaling_strategies.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This includes using out-of-core models, reducing data size with PCA, and various incremental learners&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides that, your best bet is to use a beefier computer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, remember that once a model is trained, it can be pickled and shared.  Training/testing is usually the time/cpu consuming process.  So, once you have a model, you should be able to implement it on machines less beefy than the train/test machine&lt;/p&gt;&#xA;" OwnerUserId="21232" LastActivityDate="2017-10-19T15:10:22.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23910" PostTypeId="1" CreationDate="2017-10-19T17:00:50.637" Score="1" ViewCount="26" Body="&lt;p&gt;Data augmentation is very standard for annotated image datasets for tasks like image labelling.  Images are flipped, rotated, pixelated and so on, to add more training data and make the system robust to noise and not overfit on irrelevant features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are for example speech recognition training datasets pre-processed?  Are they augmented with background noise or other mutations like image datasets are?  Has there been any work in this direction?&lt;/p&gt;&#xA;" OwnerUserId="12363" LastActivityDate="2017-10-19T17:00:50.637" Title="Are annotated audio datasets augmented with mutated versions the way image datasets are?" Tags="|nlp|preprocessing|audio-recognition|data-augmentation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23911" PostTypeId="1" AcceptedAnswerId="23920" CreationDate="2017-10-19T17:07:24.217" Score="2" ViewCount="62" Body="&lt;p&gt;I have an image of some data which is approximately 4,000 x 8,000 pixels.  I am interested in finding out if anyone has used a deep learning algorithm to predict what would be on the image if it extended 100 more pixels in each direction.  I would imagine that the data could be trained on smaller rectangles, and then the rules developed would be used to extend beyond the image given.  Has anyone seen a problem like this (and is there a reference)?   Even if not, what deep learning scheme would be best for this?&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2017-10-19T22:09:36.713" Title="Deep Learning to estimate what is beyond the edge" Tags="|deep-learning|reference-request|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23912" PostTypeId="1" CreationDate="2017-10-19T18:11:54.477" Score="2" ViewCount="289" Body="&lt;p&gt;I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and suggestions/critiques on my proposed strategy below. I realise that experimentation may be the best way forward but I don't want to unnecessarily duplicate somebody else's work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that a popular approach is to assume that the function of model error wrt. HPs is a bayesian process, the parameters of which are updated as new points are evaluated (ie. an expensive full training of the underlying deep learning model) so that a more rapid convergence on the optimal HPs can be achieved by quickly predicting better candidates for the next HPs to be evaluated than a simple grid search would be able to. Ilija Ilievski et al report more efficient convergence by using an RBF surrogate model (with pySOT) instead of bayesian &lt;a href=&quot;https://github.com/ilija139/HORD&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ilija139/HORD&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that changes in some HPs (let's call them cheap) eg. ADAM initial learning rate and learning rate decay won't affect training time wheras doubling some HPs (let's call them expensive) eg. SGD batch size or total number of steps will double training time, does it make sense to approach this problem by trying to quickly refine cheap HPs whilst the object function is cheap(er) to evaluate and then allowing the object function to become more expensive gradually as the expensive HPs inflate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of implementing this startegy might be to hold expensive HPs at relatively low values whilst refining cheap HPs and then allowing expensive HPs to grow and then iterating this process until some precondition is met, ie. model-error no longer goes down or processor-time budget is exceeded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative and possibly more 'organic' approach that I imagine would effectively achieve something similar would be to give each evaluation of the object function a certain processor time budget that would gradually increase. So when the object function over-ran its budget, it would be (gracefully) stopped and whatever error result that model gave, would be taken alongside whatever HPs were used (with number of steps being the actual number of steps taken, rather than what was requested). The idea being that this would pressure the HP surrogate to rapidly refine the cheap HPs initially whilst allowing the expensive HPs to be refined over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense?&lt;/p&gt;&#xA;" OwnerUserId="40711" LastActivityDate="2017-10-19T19:07:33.107" Title="Strategies for automatically tuning the hyper-parameters of deep learning models" Tags="|deep-learning|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23913" PostTypeId="2" ParentId="23789" CreationDate="2017-10-19T18:45:17.943" Score="52" Body="&lt;p&gt;It's easier to start with your second question and then go to the first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bagging&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forest is a bagging algorithm. It reduces variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust model (reduce variance) through &lt;strong&gt;bagging&lt;/strong&gt; -- bagging is when you create different models by resampling your data to make the resulting model more robust.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random forest is what we call to bagging applied to decision trees, but it's no different than other bagging algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would you want to do this? It depends on the problem. But usually, it is highly desirable for the model to be stable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Boosting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boosting reduces variance, and also reduces bias. It reduces variance because you are using multiple models (bagging). It reduces bias by training the subsequent model by telling him what errors the previous models made (the boosting part).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two main algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adaboost: this is the original algorithm; you tell subsequent models to punish more heavily observations mistaken by the previous models&lt;/li&gt;&#xA;&lt;li&gt;Gradient boosting: you train each subsequent model using the residuals (the difference between the predicted and true values)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In these ensembles, your base learner &lt;strong&gt;must&lt;/strong&gt; be weak. If it overfits the data, there won't be any residuals or errors for the subsequent models to build upon. Why are these good models? Well, most competitions in websites like Kaggle have been won using gradient boosting trees. Data science is an empirical science, &quot;because it works&quot; is good enough. Anyhow, do notice that boosting models can overfit (albeit empirically it's not very common).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another reason why gradient boosting, in particular, is also pretty cool: because it makes it very easy to use different loss functions, even when the derivative is not convex. For instance, when using probabilistic forecast, you can use stuff such as the &lt;a href=&quot;https://www.lokad.com/pinball-loss-function-definition&quot; rel=&quot;noreferrer&quot;&gt;pinball function&lt;/a&gt; as your loss function; something which is much harder with neural networks (because the derivative is always constant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Interesting historical note: Boosting was originally a theoretical invention motivated by the question &quot;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.192.8630&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;noreferrer&quot;&gt;can we build a stronger model using weaker models&lt;/a&gt;&quot;]&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Notice: People sometimes confuse random forest and gradient boosting trees, just because both use decision trees, but they are two very different families of ensembles.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2017-11-24T13:26:59.940" LastActivityDate="2017-11-24T13:26:59.940" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23915" PostTypeId="2" ParentId="23912" CreationDate="2017-10-19T19:07:33.107" Score="1" Body="&lt;p&gt;I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ymmv, perhaps.&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-10-19T19:07:33.107" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23917" PostTypeId="2" ParentId="23736" CreationDate="2017-10-19T20:10:57.737" Score="2" Body="&lt;p&gt;One method to compare the topics across two corpora and measuring their similarity is with &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kullback-Leibler divergence&lt;/a&gt;, aka relative entropy.&#xA;Kullback-Leibler divergence is a measure of how one probability distribution diverges from a second probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another, more scalable algorithm can be found in &lt;em&gt;&lt;a href=&quot;http://vis.stanford.edu/files/2013-TopicModelDiagnostics-ICML.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Topic Model Diagnostics:&#xA;Assessing Domain Relevance via Topical Alignment&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-10-19T20:10:57.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23919" PostTypeId="2" ParentId="23911" CreationDate="2017-10-19T21:30:46.543" Score="2" Body="&lt;p&gt;There are quite a few &lt;a href=&quot;https://arxiv.org/pdf/1605.07157.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;papers&lt;/a&gt; on predicting the next image in a set of video sequences. So I would familiarize yourself with those first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that being said it is definitely possible to do this sort of things using ML. There has been a lot of work on recurrent layers for Convolutional Neural Nets. These at a high level seems like it would be a good candidate to investigate for your initial architectures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some info on RCNNs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example RCNN in keras: &lt;a href=&quot;https://github.com/jacobzweig/Keras-RCNN/blob/master/Keras_RCNN.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Papers on RCNNs: &lt;a href=&quot;https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Liang_Recurrent_Convolutional_Neural_2015_CVPR_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;   &lt;a href=&quot;https://papers.nips.cc/paper/5634-convolutional-neural-networks-with-intra-layer-recurrent-connections-for-scene-labeling.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31046" LastActivityDate="2017-10-19T21:30:46.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23920" PostTypeId="2" ParentId="23911" CreationDate="2017-10-19T22:09:36.713" Score="4" Body="&lt;p&gt;I think the closest problem that has been addressed with deep learning is image inpainting, that is, filling a blacked out region in the image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://bamos.github.io/data/2016-08-09/completion.gif&quot; alt=&quot;Image&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, this paper: &lt;a href=&quot;https://arxiv.org/abs/1607.07539&quot; rel=&quot;nofollow noreferrer&quot;&gt;Semantic Image Inpainting with Perceptual and Contextual Losses&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is certainly possible to fill missing information from an image with deep learning.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-10-19T22:09:36.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23921" PostTypeId="2" ParentId="13966" CreationDate="2017-10-19T23:35:04.953" Score="2" Body="&lt;p&gt;You can obtain a master-set of cutpoints by using &lt;code&gt;arules::discretize(x, ..., onlycuts = T)&lt;/code&gt;. Then, do as @hssay does in his/her answer. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(arules)&#xA;&#xA;train &amp;lt;- data.frame(dat = runif(100))&#xA;test &amp;lt;- data.frame(dat = runif(100))&#xA;&#xA;mastercuts &amp;lt;- arules::discretize(train, method = &quot;interval&quot;, categories = 4, onlycuts = T)&#xA;&#xA;train$bin &amp;lt;- as.numeric(cut(train, breaks = mastercuts))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40723" LastEditorUserId="24000" LastEditDate="2017-10-20T07:24:18.797" LastActivityDate="2017-10-20T07:24:18.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23922" PostTypeId="1" AcceptedAnswerId="23975" CreationDate="2017-10-19T23:38:40.733" Score="10" ViewCount="1617" Body="&lt;p&gt;For some reason, AlphaGo Zero isn't getting as much publicity as the original AlphaGo, despite its incredible results. Starting from scratch, it's already beaten AlphaGo Master and has passed numerous other benchmarks. Even more incredibly, it's done this in 40 days. Google names it as &lt;a href=&quot;https://deepmind.com/blog/alphago-zero-learning-scratch/&quot; rel=&quot;noreferrer&quot;&gt;&quot;arguably the best Go player in the world&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DeepMind claims this is a &quot;novel form of reinforcement learning&quot; - is this technique truly novel? Or have there been other times when this technique was used- and if so, what were their results? I think the requirements I'm talking about are 1) no human intervention and 2) no historical play, but these are flexible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/11118/alphago-and-other-game-programs-using-reinforcement-learning-without-human-dat&quot;&gt;This&lt;/a&gt; appears to be a similar question, but all the answers seem to start from the assumption that AlphaGo Zero is the first of its kind.&lt;/p&gt;&#xA;" OwnerUserId="40720" LastEditorUserId="836" LastEditDate="2017-10-20T07:14:47.143" LastActivityDate="2017-10-26T09:59:12.987" Title="What is the &quot;novel reinforcement learning algorithm&quot; in AlphaGo Zero?" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="23923" PostTypeId="1" CreationDate="2017-10-19T23:42:08.347" Score="1" ViewCount="521" Body="&lt;p&gt;I need to train a system on a large set of images and associated captions to determine which (image, caption) pairs are correct and which are not. I don't have any labeled pairs, but I can assume that more pairs are correct than are not. Furthermore, the images and captions have small subsets that are reasonably similar, but not exactly.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Current idea&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;My current idea is to build a sort of GAN architecture where I have a generator and a discriminator, except that the generator is simply sampling from the existing corpus of images and captions to generate false pairs and the discriminator is in charge of determining which pairs are correct or not.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;images, captions = [], []&#xA;bpg = bad_pair_generator(images, captions)&#xA;&#xA;clfr = classifier()&#xA;&#xA;for epoch in range(epochs):&#xA;    good_pairs, bad_pairs = zip(images, captions), bpg.generate(N)&#xA;    X, y = good_pairs + bad_pairs, [1]*len(good_pairs) + [0]*len(bad_pairs)&#xA;&#xA;    clfr.train(X, y)&#xA;&#xA;    trickiness = [yi == yi_pred for yi, yi_pred in zip(y, clfr.predict(X))]&#xA;    bpg.fit(bad_pairs, trickiness[-len(bad_pairs):])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this, I would hope that the classifier was stressed sufficiently to learn a good model for whether the pairs are good or not.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are there better researched approaches?&lt;/li&gt;&#xA;&lt;li&gt;If not, does this approach seem sane and reasonably safe?&lt;/li&gt;&#xA;&lt;li&gt;What kind of model could I use to compared the image to the text? Clearly a CNN/RNN would be good for the image/text, but how do you combine them?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerDisplayName="user32280" LastEditorDisplayName="user32280" LastEditDate="2017-10-19T23:56:30.283" LastActivityDate="2019-02-28T12:26:40.413" Title="Unsupervised learning if existing image captions match the images" Tags="|unsupervised-learning|image-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23924" PostTypeId="2" ParentId="23823" CreationDate="2017-10-20T07:23:52.100" Score="1" Body="&lt;p&gt;It is indeed for computational tractability. You would not lose the all-important sparsity, since that is provided by the second term. The original formulation of SVM, which you can find on Wikipedia actually uses $\lVert w \rVert$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Minimize $\|{\vec {w}}\|$ subject to ${\displaystyle y_{i}({\vec {w}}\cdot {\vec {x}}_{i}-b)\geq 1}-\xi_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, minimizing $\lVert w \rVert$ is the same as $\lVert w \rVert^2$. A quantative difference arises when introducing the constraint using the Lagrangian function, but qualitatively it is the same.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-10-20T07:23:52.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23925" PostTypeId="1" AcceptedAnswerId="64336" CreationDate="2017-10-20T07:31:27.757" Score="3" ViewCount="171" Body="&lt;p&gt;Let's say that I have sparse feature vectors and I'd like to use dimensionality reduction in order to visualize them more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dimensionality reduction techniques like PCA will estimate colinearity between the features given the data. What if I have some prior knowledge on the colinearity between my features? As in, I would be able to create an approximate distance matrix between my features, and therefore between my data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that if features are actually colinear, methods like PCA will find a way to reduce them, however, I am afraid that I don't have enough data points to infer that colinearity strictly from the data and that PCA would remove meaningful features and keep uninformative ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that my data looks like something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x_0 = \{1,NaN,-1,NaN\}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x_1 = \{1,NaN,NaN,-1\}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I know that the 3rd and 4th features are colinear and that the distance between &lt;span class=&quot;math-container&quot;&gt;$x_0$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$x_1$&lt;/span&gt; is close to zero. Therefore, they should be mapped to a very similar data point in a lower-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can PCA still do it with so few data points? Are there ways to &quot;force&quot; the known colinearity or distance measures? T-SNE perhaps?&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2019-12-06T12:29:38.930" LastActivityDate="2019-12-06T15:49:34.867" Title="Dimensionality reduction with prior knowledge of colinearity between features" Tags="|dimensionality-reduction|pca|tsne|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="23927" PostTypeId="1" AcceptedAnswerId="23984" CreationDate="2017-10-20T10:24:52.740" Score="1" ViewCount="140" Body="&lt;p&gt;I've found a number of resources that mention Tufte's 'data density index' and 'data-to-ink ratio' when considering the analysis of particular visualisations and visualisation techniques, but I am yet to come across any papers that actually critique the measures themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there critiques of these principles available?&lt;/p&gt;&#xA;" OwnerUserId="40736" LastEditorUserId="29575" LastEditDate="2017-10-22T20:46:07.090" LastActivityDate="2017-10-22T20:46:07.090" Title="What are some good papers that discuss Tufte's 'data density index' and 'data-to-ink ratio' in data visualisation?" Tags="|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23932" PostTypeId="1" AcceptedAnswerId="23948" CreationDate="2017-10-20T13:18:46.373" Score="1" ViewCount="240" Body="&lt;p&gt;I'm investigating various NLP algorithms and tools to solve the following problem; NLP newbie here, so pardon my question if it's too basic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, I have a messaging app where users can send text messages to one or more people. When the user types a message, I want the app to suggest to the user who the potential recipients of the message are?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If user &quot;A&quot; sends a lot of text messages regarding &quot;cats&quot; to user &quot;B&quot; and some messages to user &quot;C&quot; and sends a lot of messages regarding &quot;politics&quot; to user &quot;D&quot;, then next time user types the message about &quot;cats&quot; then the app should suggest &quot;B&quot; and &quot;C&quot; instead of &quot;D&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm doing some research on topic modeling and word embeddings and see that LDA and Word2Vec are the 2 probable algorithms I can use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wanted to pick your brain on which one you think is more suitable for this scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea I have is, extract topics using LDA from the previous messages and rank the recipients of the messages based on the # of times a topic has been discussed (ie, the message sent) in the past. If I have this mapping of the topic and a sorted list of users who you talk about it (ranked based on frequency), then when the user types a message, I can again run topic extraction on the message, predict what the message is about and then lookup the mapping to see who can be the possible recipients and show to user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a good approach? Or else, Word2Vec (or doc2vec or lda2vec) is better suited for this problem where we can predict similar messages using vector representation of words aka word embeddings? Do we really need to extract topics from the messages to predict the recipients or is that not necessary here? Any other algorithms or techniques you think will work the best? Should I just use supervised learning instead? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are your thoughts and suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help.&lt;/p&gt;&#xA;" OwnerUserId="40751" LastActivityDate="2017-10-20T21:54:06.497" Title="LDA vs Word2Vec vs Others for predicting recipients of a message" Tags="|clustering|supervised-learning|word2vec|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23933" PostTypeId="1" AcceptedAnswerId="23937" CreationDate="2017-10-20T13:44:26.497" Score="1" ViewCount="1713" Body="&lt;p&gt;Assume I want to predict if I'm fit in the morning. One feature is the last time I was online. Now this feature is tricky: If I take the hour, then a classifier might have a difficult time with it because 23 is numerically closer to 20 than to 0, but actually the time 23 o'clock is closer to 0 o'clock.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a transformation to make this more linear? Probably into multiple features? (Well, hopefully not 60 features if I do the same for minutes)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-10-20T17:40:53.190" Title="How can I deal with circular features like hours?" Tags="|feature-engineering|" AnswerCount="1" CommentCount="5" ClosedDate="2017-10-22T08:18:24.423" ContentLicense="CC BY-SA 3.0" />
  <row Id="23935" PostTypeId="1" CreationDate="2017-10-20T15:13:45.633" Score="-1" ViewCount="561" Body="&lt;p&gt;Sorry for the long title but why do I always see in the example code that the derivative of sigmoid is x * (1 - x). While the true mathematical derivative is s(x) * (1 - s(x)) and in fact, I tried to change the derivative in the code to its true mathematical derivative but it didn't even converge so I guess maybe that's why. But then how did it comes to x * (1 - x) instead of the real one? could someone please explain this to me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code (from i am trask's blog &lt;a href=&quot;https://iamtrask.github.io/2015/07/12/basic-python-network/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])&#xA;y = np.array([[0,1,1,0]]).T&#xA;syn0 = 2*np.random.random((3,4)) - 1&#xA;syn1 = 2*np.random.random((4,1)) - 1&#xA;for j in xrange(60000):&#xA;    l1 = 1/(1+np.exp(-(np.dot(X,syn0))))&#xA;    l2 = 1/(1+np.exp(-(np.dot(l1,syn1))))&#xA;    l2_delta = (y - l2)*(l2*(1-l2))&#xA;    l1_delta = l2_delta.dot(syn1.T) * (l1 * (1-l1))&#xA;    syn1 += l1.T.dot(l2_delta)&#xA;    syn0 += X.T.dot(l1_delta)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40756" LastEditorUserId="40756" LastEditDate="2017-10-21T08:38:06.817" LastActivityDate="2021-12-10T18:53:10.907" Title="Why is different function used for the derivative of sigmoid than its true derivative?" Tags="|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23936" PostTypeId="2" ParentId="23935" CreationDate="2017-10-20T15:48:13.847" Score="3" Body="&lt;p&gt;Let's take a look at &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/math_grad.cc#L228&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow's implementation of the gradient of the sigmoid&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Status SigmoidGrad(const AttrSlice&amp;amp; attrs, FunctionDef* g) {&#xA;  return GradForUnaryCwise(g, {&#xA;      {{&quot;y&quot;}, &quot;Sigmoid&quot;, {&quot;x&quot;}},&#xA;      FDH::Const(&quot;const&quot;, 1.0f),&#xA;      {{&quot;one&quot;}, &quot;Cast&quot;, {&quot;const&quot;}, {{&quot;SrcT&quot;, DT_FLOAT}, {&quot;DstT&quot;, &quot;$T&quot;}}},&#xA;      {{&quot;a&quot;}, &quot;Sub&quot;, {&quot;one&quot;, &quot;y&quot;}, {}, {&quot;dy&quot;}},&#xA;      {{&quot;b&quot;}, &quot;Mul&quot;, {&quot;y&quot;, &quot;a&quot;}},             // y * (1 - y)&#xA;      {{&quot;dx&quot;}, &quot;Mul&quot;, {&quot;dy&quot;, &quot;b&quot;}},           // dy * y * (1 - y)&#xA;  });&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you &lt;em&gt;can&lt;/em&gt; see, the implementation is &lt;code&gt;s(x) * (1 - s(x))&lt;/code&gt;. Note that they reuse the output of the sigmoid itself &lt;code&gt;y = s(x)&lt;/code&gt;, therefore they write &lt;code&gt;y * (1 - y)&lt;/code&gt;. Maybe in your example they did the same and that's the source of the confusion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;: OP has included the blog that is the source of the confusion. There, we can see that function &lt;code&gt;nonlin&lt;/code&gt; is misleading:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# sigmoid function&#xA;def nonlin(x, deriv=False):&#xA;    if(deriv==True):&#xA;        return x*(1-x)&#xA;    return 1/(1+np.exp(-x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is misleading because argument &lt;code&gt;x&lt;/code&gt; means different things depending on the value of flag &lt;code&gt;deriv&lt;/code&gt;, and there is no comment in the code indicating so. If &lt;code&gt;deriv&lt;/code&gt; is &lt;code&gt;False&lt;/code&gt;, then &lt;code&gt;x&lt;/code&gt; is the input of the sigmoid. If &lt;code&gt;deriv&lt;/code&gt; is &lt;code&gt;True&lt;/code&gt;, &lt;code&gt;x&lt;/code&gt; is the sigmoid of the input. We can deduce it from its usage in the code. Here the author uses &lt;code&gt;nonlin&lt;/code&gt; to compute the sigmoid: &lt;code&gt;l1 = nonlin(np.dot(l0,syn0))&lt;/code&gt;; and here the author uses &lt;code&gt;nonlin&lt;/code&gt; to compute its derivative: &lt;code&gt;nonlin(l1,True)&lt;/code&gt;. As you can see, in the second case the author supplies &lt;code&gt;l1&lt;/code&gt; as input to &lt;code&gt;nonlin&lt;/code&gt;, and &lt;code&gt;l1&lt;/code&gt; is the output of the previous invocation to &lt;code&gt;nonlin&lt;/code&gt;, that is, &lt;code&gt;l1&lt;/code&gt; is the sigmoid of the input.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-10-21T09:00:14.137" LastActivityDate="2017-10-21T09:00:14.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23937" PostTypeId="2" ParentId="23933" CreationDate="2017-10-20T16:29:57.587" Score="2" Body="&lt;p&gt;The question was already posted, you can find the answer there :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/5990/what-is-a-good-way-to-transform-cyclic-ordinal-attributes&quot;&gt;What is a good way to transform Cyclic Ordinal attributes?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to transform your time feature into two feature : it's like if you represent the hour as the angle of the hand on the clock, and use the sin/cos of the angle as your features&lt;/p&gt;&#xA;" OwnerUserId="40331" LastActivityDate="2017-10-20T16:29:57.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23939" PostTypeId="2" ParentId="14273" CreationDate="2017-10-20T17:28:53.163" Score="2" Body="&lt;p&gt;Your original approach was right, and your intuition about the missing level too. To do what you want you just needed to add add the level &quot;None&quot;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Create a factor for the example&#xA;x&amp;lt;-factor(c(&quot;S&quot;,NA,&quot;M&quot;,&quot;S&quot;,&quot;S&quot;,&quot;S&quot;,NA,NA,&quot;S&quot;,&quot;M&quot;,&quot;S&quot;,NA,&quot;M&quot;,&quot;S&quot;,NA,&quot;S&quot;,&quot;S&quot;,NA,&quot;M&quot;,&quot;S&quot;,NA,&quot;M&quot;))&#xA;&#xA;levels(x)&amp;lt;-c(levels(x),&quot;None&quot;)  #Add the extra level to your factor&#xA;x[is.na(x)] &amp;lt;- &quot;None&quot;           #Change NA to &quot;None&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40760" LastActivityDate="2017-10-20T17:28:53.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23945" PostTypeId="2" ParentId="14273" CreationDate="2017-10-20T19:23:28.580" Score="9" Body="&lt;p&gt;You can use this function : &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;forcats::fct_explicit_na&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(forcats) &#xA;fct_explicit_na(DF$col, na_level = &quot;None&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Usage&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;It can be used within the mutate function and piped to edit DF directly: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse) # for tidy data packages, automatically loads dplyr&#xA;library(magrittr) # for piping&#xA;DF %&amp;lt;&amp;gt;% mutate(cols = fct_explicit_na(col, na_level = &quot;None&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that &quot;col&quot; needs to be a factor for this to work.&lt;/p&gt;&#xA;" OwnerUserId="40767" LastEditorUserId="-1" LastEditDate="2019-07-06T05:39:24.010" LastActivityDate="2019-07-06T05:39:24.010" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="23947" PostTypeId="2" ParentId="23901" CreationDate="2017-10-20T21:04:14.697" Score="2" Body="&lt;p&gt;If one requires that $H = C$, then this is called the &quot;proper PAC&quot; framework compared to &quot;PAC prediction&quot; where we don't care about the representation of $h$ as long as the prediction error is small enough (i.e. we allow $H$ to be the class of all time-polynomial programs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of a concept as the set of inputs that produce the same label (e.g. all the images that show a chair form the concept &quot;chair&quot; or all the points in the same half space form the concept &quot;true/false&quot;).&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-10-20T21:04:14.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23948" PostTypeId="2" ParentId="23932" CreationDate="2017-10-20T21:54:06.497" Score="1" Body="&lt;p&gt;Why the detour with the topic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can just learn a linear SVM to directly predict the recipient and avoid all the difficulties from topic modeling. What if a mail isn't about any of the previous topics? how many topics are there?&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-10-20T21:54:06.497" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23954" PostTypeId="1" AcceptedAnswerId="23957" CreationDate="2017-10-21T08:25:50.147" Score="7" ViewCount="5265" Body="&lt;p&gt;Sklearn has a &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;feature_importances_&lt;/code&gt;&lt;/a&gt; attribute, but this is highly model-specific and I'm not sure how to interpret it as removing the most important feature does not necessarily decrease the models quality most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a model-agnostic way to tell which features are important for a prediction problem?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only way I could see is the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use an ensemble of different models&lt;/li&gt;&#xA;&lt;li&gt;Either start with a big set of features and remove one at a time. To find the features &quot;uplift&quot;, compare the ensembles quality with the full feature set against the ensembles quality against the removed feature set.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(What this can't do is to find connected features: Some features might be not exactly the same, but have a common underlying cause which is important for the prediction. Hence removing either of them doesn't change much, but removing both might change a lot. I ask &lt;a href=&quot;https://datascience.stackexchange.com/q/23955/8820&quot;&gt;another question&lt;/a&gt; for that.)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-10-21T09:15:29.343" LastActivityDate="2020-04-23T16:48:25.263" Title="Is there a model-agnostic way to determine feature importance?" Tags="|feature-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23957" PostTypeId="2" ParentId="23954" CreationDate="2017-10-21T09:12:21.910" Score="8" Body="&lt;p&gt;Ways to &quot;determine feature importance&quot; are normally called &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection&quot; rel=&quot;noreferrer&quot;&gt;feature selection algorithms&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 3 types of feature selection algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Filter approaches: they choose variables without using a model at all, just looking at the feature values. One example is scikit-learn's variance threshold selector.&lt;/li&gt;&#xA;&lt;li&gt;Wrapper approaches: they use whatever prediction algorithm to score different subsets of features and choose the best subset based on that. These use a model but are model agnostic, as they don't care about which model you use. One example is &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_digits.html#sphx-glr-auto-examples-feature-selection-plot-rfe-digits-py&quot; rel=&quot;noreferrer&quot;&gt;recursive feature elimination&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Embedded approaches: in these approaches, the variable selection is part of a model, hence the feature selection and the model are coupled together. This is the case of the &lt;code&gt;feature_importances_&lt;/code&gt; in random forest algorithm. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From the question, I understand that both filter and wrapper approaches are suitable for the OP needs. A classic article that covers both very well is &lt;a href=&quot;http://ai.stanford.edu/~ronnyk/wrappers.pdf&quot; rel=&quot;noreferrer&quot;&gt;this one by Kovavi and John&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; you can see an overview of scikit-learn feature selection capabilities, which includes examples of the three aforementioned types of variable selection algorithms.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-10-21T09:18:11.870" LastActivityDate="2017-10-21T09:18:11.870" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="23960" PostTypeId="2" ParentId="17124" CreationDate="2017-10-21T13:21:45.080" Score="1" Body="&lt;p&gt;You definitely have &lt;strong&gt;interval&lt;/strong&gt; data, that is, data which takes on discrete values, as opposed to &lt;strong&gt;continuous&lt;/strong&gt; data, which takes on values along a continuum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be of value to additionally determine if the data is &lt;strong&gt;ordinal&lt;/strong&gt;, meaning that the order of the values is important, for example if &lt;strong&gt;[0, 1, 2]&lt;/strong&gt; signifies &lt;strong&gt;[small, medium, large]&lt;/strong&gt; or some analogous system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of ordinal data, it may be best to keep the data as exposed to the SVM training process in integer form, as the integer representation encodes some information about the relationship between the categories. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach would also be more reasonable if the values that the variable could take on in a production setting could expand beyond the values you've already observed in the training set- a categorical approach would be less able to handle new values in that context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are no ordinal relationships and you suspect all of the possible values are enumerated in the training set,treating the variable as categorical would be approriate.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-10-21T13:21:45.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23965" PostTypeId="1" CreationDate="2017-10-21T22:00:10.967" Score="1" ViewCount="2299" Body="&lt;p&gt;I am following this &lt;a href=&quot;https://medium.com/towards-data-science/reinforcement-learning-w-keras-openai-actor-critic-models-f084612cfd69&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; and am trying to use it for another OpenAI Gym environment (MountainCar-v0). This environment has a discrete action space instead of a box one. By this I mean if you were to print(env.action_space) for this particular environment you get Discrete(3) as the result as opposed to some Box(3,) action space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_critic_model(self):&#xA;    state_input = Input(shape=self.env.observation_space.shape)&#xA;    state_h1 = Dense(24, activation='relu')(state_input)&#xA;    state_h2 = Dense(48)(state_h1)&#xA;&#xA;    action_input = Input(shape=self.env.action_space.shape)&#xA;    action_h1    = Dense(48)(action_input)&#xA;&#xA;    merged    = Add()([state_h2, action_h1])&#xA;    merged_h1 = Dense(24, activation='relu')(merged)&#xA;    output = Dense(1, activation='relu')(merged_h1)&#xA;    model  = Model(input=[state_input,action_input], &#xA;        output=output)&#xA;&#xA;    adam  = Adam(lr=0.001)&#xA;    model.compile(loss=&quot;mse&quot;, optimizer=adam)&#xA;    return state_input, action_input, model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the critic model takes in both the state and action as input. Because I am working in with a discrete action space I get this error: &lt;code&gt;AttributeError: 'Discrete' object has no attribute 'shape'&lt;/code&gt;. This is because the discrete action space has no shape but I have to pass it through as an input. And when just passing env.action_space as the input, I get this error: &lt;code&gt;TypeError: 'Discrete' object is not iterable&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Critic model uses an action input which is the action space shape. Being a discrete action space, &lt;a href=&quot;https://gym.openai.com/envs/MountainCar-v0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;mountaincar&lt;/a&gt; does not have an action space shape. How would I reconfigure this code to suit my purpose(use action input for a discrete action shape space).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, on this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;future_reward = self.target_critic_model.predict([new_state, target_action])[0][0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it seems we are trying to predict on 2 diff arrays, and I get an error regarding that. So what's going on there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Error is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Error when checking : the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 arrays but instead got the following list of 2 arrays: [array([[-0.47361678, -0.00322982]]), array([[ 0.00318837, -0.05581997,  0.06615788]], dtype=float32)].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after switching to my new machine, installing fresh python, keras, tensorflow, and gym(which got rid of the other error) I am left with a new keras error: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AssertionError: Please provide to Input either a `shape` or a `batch_shape` argument. Note that `shape` does not include the batch dimension.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also still getting the array related error.&lt;/p&gt;&#xA;" OwnerUserId="31046" LastEditorUserId="31046" LastEditDate="2017-11-11T07:25:02.497" LastActivityDate="2018-09-30T11:04:45.683" Title="Actor-Critic in Discrete action spaces" Tags="|python|deep-learning|reinforcement-learning|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="23966" PostTypeId="1" CreationDate="2017-10-21T22:09:39.703" Score="0" ViewCount="428" Body="&lt;p&gt;After research, I've decided to take advice on the approach to use for our machine learning problem as I'm still not sure which to take.&lt;/p&gt;&#xA;&lt;h2&gt;Our problem:&lt;/h2&gt;&#xA;&lt;p&gt;We have our labels as a set of website names (Google, Twitter and Facebook for example). We need to train our model on a set of images that belong to each website, and later be able to give the model any image and predict which website it might be. Also note that the image could belong to none of the websites trained so I need some sort of &lt;code&gt;None&lt;/code&gt; output.&lt;/p&gt;&#xA;&lt;p&gt;One of the problems we're facing is the small number of data for each set (around 40 images for Facebook login forms that could be pushed to 200 with color transformations &lt;strong&gt;if that makes a difference while training.&lt;/strong&gt;)&lt;/p&gt;&#xA;&lt;h2&gt;What we need:&lt;/h2&gt;&#xA;&lt;p&gt;Some sort of image classifier/predictor/recognizer (still not sure which) that will detect the class of the input image (Facebook, Google or Twitter for example) or tell us that it's none.&lt;/p&gt;&#xA;&lt;h2&gt;Problems we might face:&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'm not sure if our model is subject to over-fitting since our set of input images are limited and we should already have trained the model on. Take the facebook login page for example, they all look the same with small variety in the location of the fields. So our training and out-of-sample data are almost the same.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2&gt;What I've done so far:&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I've already started working on a Convolutional Deep Network using keras, but I'm worried that deep networks would be too complex for our data, or our data is too simple(alike) for the model to be trained correctly.&lt;/li&gt;&#xA;&lt;li&gt;One Shot Learning and Siamese Networks, but this needs to have an output that must belong to the data we've trained on (which disregards the &lt;code&gt;None&lt;/code&gt; possibility)&lt;/li&gt;&#xA;&lt;li&gt;Sklearn's different classifiers, but a classifier must have an either or output for one of the labels, which again disregards the &lt;code&gt;None&lt;/code&gt; possibility.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Finally,&lt;/strong&gt; and what sounds most logical personally, is to use sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;OneClassSVM&lt;/a&gt; on each label (so if I have Google, Facebook and Twitter then I would have 3 models, one that corresponds to each label) and then pass my input image to each model to determine to which it belongs. The problem with this is if I have a lot of classes (which could be the case) then I'd have to run each one.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;One final approach could be generating probabilities for each class after inputting an image to our model, assuming that these probabilities should not sum to 1 as to keep the &lt;code&gt;None&lt;/code&gt; possibility.&lt;/p&gt;&#xA;&lt;p&gt;Any suggestions or advice would be helpful. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="40795" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-22T08:00:44.383" Title="Best approach for image recognition/classification with few training data" Tags="|machine-learning|scikit-learn|keras|svm|convolution|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-22T08:25:24.430" ContentLicense="CC BY-SA 3.0" />
  <row Id="23967" PostTypeId="2" ParentId="23872" CreationDate="2017-10-21T22:55:25.510" Score="0" Body="&lt;p&gt;With just 14 airlines, the clustering algorithm of choice is of course hierarchical clustering. But you will need to develop a good similarity measure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why don't you use e.g. airline alliances instead?&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-10-21T22:55:25.510" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="23968" PostTypeId="1" CreationDate="2017-10-22T01:44:51.807" Score="3" ViewCount="7083" Body="&lt;p&gt;I am debugging results from the UNET architecture that I am using for identifying corneal reflection in eye images. While I am getting over 99% training accuracy and also very high (over 99%) validation accuracy, when I run the validation images myself, I am getting nothing but blank images from model prediction. When I used the same architecture with exact same parameters for training with a mask set for pupil, I get again high accuracy numbers but running the validation set again in the prediction gave great results. Here is the sample output for the data set where I am having trouble, mismatch of validation and prediction results:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train on 326 samples, validate on 140 samples&#xA;Epoch 1/1&#xA;277s - loss: 0.1961 - dice_coef: 0.0012 - acc: 0.9834 - val_loss: 0.0338 - val_dice_coef: 4.8418e-11 - val_acc: 0.9979&#xA;326/326 [==============================] - 79s     &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#-------------------------------------------------------&#xA;# Define UNET model&#xA;#-------------------------------------------------------&#xA;print(&quot;Compiling UNET Model.....&quot;)&#xA;&#xA;def dice_coef(y_true, y_pred):&#xA;    y_true_f = K.flatten(y_true)&#xA;    y_pred_f = K.flatten(y_pred)&#xA;    intersection = K.sum(y_true_f * y_pred_f)&#xA;    coef = (2. * intersection + K.epsilon()) / (K.sum(y_true_f) + K.sum(y_pred_f) + K.epsilon())&#xA;    return coef&#xA;&#xA;x_data = x_data[:,:,:,np.newaxis]&#xA;y_data = y_data[:,:,:,np.newaxis]&#xA;x_train, x_val, y_train, y_val = train_test_split(x_data, y_data, test_size = 0.3)&#xA;&#xA;input_layer = Input(shape=x_train.shape[1:])&#xA;c1 = Conv2D(filters=8, kernel_size=(3,3), activation='relu', padding='same')(input_layer)&#xA;&#xA;l = MaxPool2D(strides=(2,2))(c1)&#xA;c2 = Conv2D(filters=16, kernel_size=(3,3), activation='relu', padding='same')(l)&#xA;&#xA;l = MaxPool2D(strides=(2,2))(c2)&#xA;c3 = Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same')(l)&#xA;&#xA;l = MaxPool2D(strides=(2,2))(c3)&#xA;c4 = Conv2D(filters=32, kernel_size=(1,1), activation='relu', padding='same')(l)&#xA;&#xA;l = concatenate([UpSampling2D(size=(2,2))(c4), c3], axis=-1)&#xA;l = Conv2D(filters=32, kernel_size=(2,2), activation='relu', padding='same')(l)&#xA;&#xA;l = concatenate([UpSampling2D(size=(2,2))(l), c2], axis=-1)&#xA;l = Conv2D(filters=24, kernel_size=(2,2), activation='relu', padding='same')(l)&#xA;&#xA;l = concatenate([UpSampling2D(size=(2,2))(l), c1], axis=-1)&#xA;l = Conv2D(filters=16, kernel_size=(2,2), activation='relu', padding='same')(l)&#xA;&#xA;l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)&#xA;l = Dropout(0.5)(l)&#xA;&#xA;output_layer = Conv2D(filters=1, kernel_size=(1,1), activation='sigmoid')(l)&#xA;&#xA;model = Model(input_layer, output_layer)&#xA;&#xA;model.compile(optimizer=Adam(1e-4), loss='binary_crossentropy', metrics=[dice_coef, 'acc'])&#xA;&#xA;&#xA;#-------------------------------------------------------&#xA;# Train UNET MOdel&#xA;#-------------------------------------------------------&#xA;if train_opt:&#xA;    print(&quot;Training UNET Model.....&quot;)&#xA;    weight_saver = ModelCheckpoint(weights_file, monitor='val_dice_coef', save_best_only=True, save_weights_only=True)&#xA;    annealer = LearningRateScheduler(lambda x: 1e-3 * 0.8 ** x)&#xA;    hist = model.fit(x_train, y_train, batch_size = 8, validation_data = (x_val, y_val), epochs=1, verbose=2, callbacks = [weight_saver, annealer])&#xA;    model.evaluate(x_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help!&lt;/p&gt;&#xA;" OwnerUserId="40800" LastEditorUserId="836" LastEditDate="2017-10-22T07:27:58.863" LastActivityDate="2017-10-22T09:20:18.013" Title="99% validation accuracy but 0% prediction results (UNET Architecture)" Tags="|neural-network|training|prediction|accuracy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23969" PostTypeId="1" AcceptedAnswerId="23998" CreationDate="2017-10-22T07:36:15.920" Score="37" ViewCount="84593" Body="&lt;p&gt;I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the dataset. An example would look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New sentence: &quot;&lt;code&gt;I opened a new mailbox&lt;/code&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prediction based on dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sentence                       | Similarity&#xA;A dog ate poop                   0%&#xA;A mailbox is good                50%&#xA;A mailbox was opened by me       80%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've read that &lt;a href=&quot;https://datascience.stackexchange.com/questions/16488/text-similarity-using-rnn&quot;&gt;cosine similarity&lt;/a&gt; can be used to solve these kinds of issues paired with tf-idf (and RNNs should not bring significant improvements to the basic methods), or also &lt;a href=&quot;https://datascience.stackexchange.com/questions/16713/text-processing&quot;&gt;word2vec&lt;/a&gt; is used for similar problems. Are those actually viable for use in this specific case, too? Are there any other techniques/algorithms to solve this (preferably with Python and SKLearn, but I'm open to learn about TensorFlow, too)?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2023-04-13T05:48:43.360" Title="Sentence similarity prediction" Tags="|python|nlp|scikit-learn|similarity|text|" AnswerCount="6" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23970" PostTypeId="2" ParentId="23966" CreationDate="2017-10-22T08:00:44.383" Score="2" Body="&lt;p&gt;Your data seem to be very small to be used for training a CNN. I do not fully understand what does it mean that an image &quot;belongs&quot; to &quot;Twitter&quot; and not to &quot;Google&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, my best advice to you is to use simple sklearn classifiers and gather some data that does not belong to any of your options to be used as the &lt;code&gt;None&lt;/code&gt; class. The &lt;code&gt;None&lt;/code&gt; class should include a similar amount of data as the other classes and then you can train any classier and check the scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the over-fitting situation, you should consider doing cross-validation (because your data is too small for hold out set). Try doing a 5-fold cross-validation and assess how good your model performs on average. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="38226" LastActivityDate="2017-10-22T08:00:44.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23971" PostTypeId="2" ParentId="23968" CreationDate="2017-10-22T08:10:16.090" Score="8" Body="&lt;p&gt;There is no &quot;mismatch&quot; of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts just blank output images would score roughly the same as your network has so far. Your accuracy metric is not meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The low Dice coefficient score gives you a better idea of what is going on. That covers positive case matches only, and a good score would be close to 1.0. The low score shows that the network is not focussing on getting positive pixels correct. Instead the network has learned to predict close to zero everywhere, because that minimises the loss metric to a first approximation very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to fix things?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, stop reporting accuracy. This metric is misleading you, and you need to find another so that you can assess your model fairly. As you already have Dice, you could just drop accuracy and use Dice instead. Alternatively, you could use a weighted accuracy, inversely proportional to mean number of pixels in each class of your training data. Looking at &lt;a href=&quot;https://keras.io/metrics/&quot; rel=&quot;noreferrer&quot;&gt;Keras metrics options&lt;/a&gt;, you will probably want to add another custom metric for this. You have already done this with the &lt;code&gt;dice_coef&lt;/code&gt; function, so that should not be a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, train your network as it is for longer. Your example has only one epoch. Depending on how many training examples you have, this is probably too low for an image processing network. Try increasing the number in a geometric advancement - 3, 10, 30, 100, 300, 1000 - using the new metric to see if you get any improvement during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also try altering the cost function to weight it in favour of positive pixels. Keras' &lt;a href=&quot;https://keras.io/models/sequential/#fit&quot; rel=&quot;noreferrer&quot;&gt;fit function has a class_weight parameter&lt;/a&gt; for this purpose, you could set it e.g. to &lt;code&gt;class_weight = {0:1, 1:20}&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-10-22T09:20:18.013" LastActivityDate="2017-10-22T09:20:18.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23972" PostTypeId="1" CreationDate="2017-10-22T10:35:24.227" Score="2" ViewCount="182" Body="&lt;p&gt;Framework I used: &lt;a href=&quot;https://github.com/affinelayer/pix2pix-tensorflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;pix2pix tensorflow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to train pix2pix to correct contrast on an image. However, I am getting blurred and grainy images. There is lot of black noise in the image(examples below). Is it because of the loss?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I have made no change to pix2pix-tensorflow and used it as is.&#xA;The losses I see at the end of my training are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;discriminator loss: around 0.7 &lt;/li&gt;&#xA;&lt;li&gt;generator_loss_GAN: around 2.2&lt;/li&gt;&#xA;&lt;li&gt;generator_loss_L1: around 0.16&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Sample outputs: (Input-Output-Target)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ztmfq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ztmfq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LqjFC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LqjFC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Loss information from tensorboard:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7GTLM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7GTLM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Am I first of all using correctly for my usecase? &lt;/li&gt;&#xA;&lt;li&gt;How can I make sure there is no loss of data in the image at all and only the light amount changes in the target picture? &lt;/li&gt;&#xA;&lt;li&gt;If GAN isn't right way, can you let me know how to achieve this?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="9003" LastActivityDate="2017-10-22T10:35:24.227" Title="Training tensorflow to apply contrast" Tags="|python|neural-network|deep-learning|tensorflow|gan|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23973" PostTypeId="1" CreationDate="2017-10-22T10:42:28.143" Score="1" ViewCount="148" Body="&lt;p&gt;Say I have a dataset, which consists of &lt;code&gt;n&lt;/code&gt; &lt;code&gt;customers&lt;/code&gt;. A &lt;code&gt;customer&lt;/code&gt; is represented by a &lt;code&gt;name&lt;/code&gt;, an &lt;code&gt;order&lt;/code&gt;, an &lt;code&gt;orderAmount&lt;/code&gt; and a &lt;code&gt;timestamp&lt;/code&gt; (time of day) for when the order was placed. It could look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name     |order|orderAmount|timestamp    &#xA;customer1|tv   |1          |08:30&#xA;customer2|hifi |1          |12:00&#xA;customer3|hifi |3          |12:30&#xA;customer4|tv   |2          |09:30&#xA;customer5|cd   |10         |10:00&#xA;customer1|tv   |2          |11:30&#xA;...      |...  |...        |...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I'm interested in is clustering these customers, so it would be possible for me to clearly differentiate between them. By looking at the above set, it's clear that there's a difference in what, how much and when they buy, but I would like to automate this process, and I assume clustering is one way to do it, but please correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing in particular I'm not sure of is how I would represent this dataset in a &quot;customer-matrix&quot;. I know, for example, a distance-based clustering method such as KMeans requires some sort of normalized input, but how would I get started on that with a dataset such as this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When/if I can produce good clusters, I imagine I could assign labels to these clusters and use these labels to train a classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a beginner to data science, so there might be a step or two missing in my process so please bear with me.&lt;/p&gt;&#xA;" OwnerUserId="39014" LastActivityDate="2017-10-22T11:11:31.403" Title="Clustering customer dataset to find customer patterns" Tags="|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23974" PostTypeId="2" ParentId="23973" CreationDate="2017-10-22T11:11:31.403" Score="2" Body="&lt;p&gt;Suggestion: use an indicator variable for your &quot;order&quot;+&quot;orderAmount&quot; data. The table would look like this, which seems more suitable for distance metrics:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name      | tv | hifi | cd | timestamp&#xA;customer1 | 1  | 0    | 0  | 8:30&#xA;customer2 | 0  | 1    | 0  | 12:00&#xA;customer3 | 0  | 3    | 0  | 12:30&#xA;customer4 | 2  | 0    | 0  | 9:30&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you do so, then I'd also suggest to normalize those indicator variables (make sure they are in the interval [0, 1], otherwise you could have a order of 1000 &quot;cd&quot; completely dominating the maximum of 10 &quot;tv&quot; for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing to watch out is the possibility of binning. If the time of the day doesn't matter much, you could group all the orders from the same day and customer in a single row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you can manually label &lt;em&gt;some&lt;/em&gt; examples, you could use a semi-supervised algorithm, and maybe it would have better performance than completely unsupervised clustering. Some possible algorithms are HMRF-KMeans, co-training variants, and Spy EM.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-10-22T11:11:31.403" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23975" PostTypeId="2" ParentId="23922" CreationDate="2017-10-22T12:36:32.130" Score="6" Body="&lt;p&gt;The &lt;em&gt;AlphaGo Zero&lt;/em&gt; article from &lt;em&gt;Nature&lt;/em&gt;, &quot;Mastering the Game of Go without Human Knowledge&quot;, claims four major differences from the earlier version: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Self-learning only (not trained on human games) &lt;/li&gt;&#xA;&lt;li&gt;Using only the board and stones as input (no hand-written features). &lt;/li&gt;&#xA;&lt;li&gt;Using a single neural network for policies and values&lt;/li&gt;&#xA;&lt;li&gt;A new tree-search algorithm that uses this combined policy/value network to guide where to search for good moves.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Points (1) and (2) are not new in Reinforcement learning, but improve on the previous &lt;em&gt;AlphaGo&lt;/em&gt; software as stated in the comments to your question. It just means they are now using pure Reinforcement Learning starting from randomly initialized weights. This is enabled by better, faster learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Their claim here is &lt;em&gt;&quot;Our primary contribution is to demonstrate that superhuman performance can be achieved without human domain knowledge.&quot;&lt;/em&gt; (p. 22).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Points (3) and (4) are novel in the sense that their algorithm is simpler and more general than their previous approach. They also mention that is is an improvement on previous work by Guo &lt;em&gt;et al.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unifying the policy/value network (3) enables them to implement a more efficient variant of Monte-Carlo tree search to search for good moves and simultaneous using the search tree to train the network faster (4). This is very powerful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, they describe a number of interesting implementation details like batching and reusing data-structures to optimize the search for new moves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect is that it needs less computing power, running on 4 TPUs rather than 176 GPUs and 48 TPUs for previous versions of their software.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This definitely makes it &quot;novel&quot; in the context of Go software. I believe that (3) and (4) are also &quot;novel&quot; in a broader context and will be applicable in other Reinforcement Learning domains such as &lt;em&gt;e.g.&lt;/em&gt; robotics.&lt;/p&gt;&#xA;" OwnerUserId="1424" LastEditorUserId="1424" LastEditDate="2017-10-26T09:59:12.987" LastActivityDate="2017-10-26T09:59:12.987" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="23980" PostTypeId="1" CreationDate="2017-10-22T17:05:47.893" Score="2" ViewCount="483" Body="&lt;p&gt;I have the following loss function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Kte1I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Kte1I.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I optimize it using SGD in tensorflow. The $\phi$ is a soft approximation of the minimum since the minimum is not differentiable, I used the following approximations: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H9y8u.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H9y8u.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where k is a negative integer I also tried the formulas &lt;a href=&quot;https://mathoverflow.net/questions/35191/a-differentiable-approximation-to-the-minimum-function&quot;&gt;here&lt;/a&gt;,&#xA;yet the model converges very quickly to a local minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other way to calculate the minimum?&lt;/p&gt;&#xA;" OwnerUserId="40813" LastEditorUserId="1330" LastEditDate="2022-03-06T16:43:17.090" LastActivityDate="2022-03-06T16:43:17.090" Title="Optimizing the non-differentiable minimum in tensorflow" Tags="|machine-learning|tensorflow|loss-function|optimization|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="23981" PostTypeId="2" ParentId="19538" CreationDate="2017-10-22T17:06:00.710" Score="3" Body="&lt;p&gt;I've used ICD-10 codes using one-hot encoding / dummy variables as you describe in your Q. This creates a very sparse binary design matrix which has a large number of features as you say. Some ways I've used to make it more manageable RAM-wise:&#xA;(Out of interest how long are your unique code columns, and number of row samples?) &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I find that by using parent codes (e.g. omitting the &quot;.&quot; after the ICD codes: &quot;G12.21&quot; changes to &quot;G12&quot;) there are fewer columns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As the design matrix is binary and very sparse, unless you are interested in the rare diagnoses, you can omit many co-morbidities depending on how rare the diagnoses are. e.g. discarding a few rare diagnosis for each patient (in my case with 200k rows, reduced columns from ~7,500 to ~6,500) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40818" LastActivityDate="2017-10-22T17:06:00.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23983" PostTypeId="1" CreationDate="2017-10-22T19:08:34.470" Score="6" ViewCount="688" Body="&lt;p&gt;I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512-&gt;128-&gt;128-&gt;1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the &lt;strong&gt;weights on the final layer only&lt;/strong&gt; getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any words of wisdom on how to make this network generalize better given that I've already tried:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;dropout of varying degrees&lt;/li&gt;&#xA;&lt;li&gt;l1/l2/group lasso regularization&lt;/li&gt;&#xA;&lt;li&gt;adding noise to inputs&lt;/li&gt;&#xA;&lt;li&gt;adding noise to gradients and weights&lt;/li&gt;&#xA;&lt;li&gt;feature-engineering so as to remove/re-represent highly skewed features&lt;/li&gt;&#xA;&lt;li&gt;batch normalization&lt;/li&gt;&#xA;&lt;li&gt;using a lower learning rate on the final layer&lt;/li&gt;&#xA;&lt;li&gt;simply using a smaller network (this is the best solution I've found)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.&lt;/p&gt;&#xA;" OwnerUserId="40827" LastEditorUserId="40827" LastEditDate="2017-10-27T06:56:53.793" LastActivityDate="2018-11-02T21:00:07.223" Title="Final layer of neural network responsible for overfitting" Tags="|machine-learning|neural-network|gradient-descent|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="23984" PostTypeId="2" ParentId="23927" CreationDate="2017-10-22T19:50:02.290" Score="2" Body="&lt;p&gt;There are critcism's of Tuft's principles.  Some examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.perceptualedge.com/articles/visual_business_intelligence/sometimes_we_must_raise_our_voices.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Sometimes We Must Raise Our Voices&lt;/em&gt;&lt;/a&gt;, Stephen Few, Perceptual Edge&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Even after many years of working in the field of data visualization, which has involved a great deal of experience and study that has expanded my expertise into many areas that Tufte hasn’t specifically addressed, I have only on rare occasions discovered reasons to disagree with any of his principles. The topic that I’m addressing in this article, however, deals with one of those rare disagreements.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dl.acm.org/citation.cfm?id=1362587&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Minimalism in information visualization: attitudes towards maximizing the data-ink ratio&lt;/em&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;People did not like Tufte's minimalist design of bar-graphs; they seem to prefer &quot;chartjunk&quot; instead.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-10-22T19:50:02.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23990" PostTypeId="1" AcceptedAnswerId="23992" CreationDate="2017-10-23T02:37:45.080" Score="5" ViewCount="645" Body="&lt;p&gt;I am studying the blog: &lt;a href=&quot;http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding Convolutional Neural Networks for NLP&lt;/a&gt;. It is very good blog. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I can't understand clearly about this blog. As the figure Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification as following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sGtTn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sGtTn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to ask: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I know the region sizes(2,3,4) is like 2-gram, 3-gram, 4-gram word, but what’s the meaning of number filters? Here is 2 filters for each region. Why in the author's code about sentence classification is the number of filters defined to 128? Could you give examples to explain the meaning of the number of filters? for example using the sentence of ‘I like this movie very much’ would be great.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;2) I understand the height of region size (4) is 4, but in the figure, the height of region(2, 3) are 5 and 6 respectively, I don't know why? I think the height of region is 2 and 3.&lt;/p&gt;&#xA;" OwnerUserId="35597" LastEditorUserId="35597" LastEditDate="2017-10-31T04:10:06.397" LastActivityDate="2017-10-31T04:10:06.397" Title="Understand clearly the figure: Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification" Tags="|python|deep-learning|nlp|sentiment-analysis|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23992" PostTypeId="2" ParentId="23990" CreationDate="2017-10-23T04:21:11.533" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Answering this in terms of NLP examples is quite hard, remember &quot;All models are wrong, some models are useful.&quot; First think of this in an image classification problem context, you want to use a large number of filters to collect a large number of features out of the image, one could detect edges, the other could detect densely coloured areas, one might turn a region to b&amp;amp;w. Extend  a similar logic to text, by using a lot of filters, in this case 128, you are trying to capture a lot of features. For an example like , &quot; I like movies very much&quot;, a certain filter might detect that like is a positive word and not a similarity comparison, a certain filter of size 2 might detect very much and detect that it is an expression of degeree. You can go on like that, it will be hard to come up with 128 features but the idea is to get enough features. If you think the number is unreasonable and might lead to overfitting, you can reduce the number and compare your results.       &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;No, 1- maxpool means that you take the maximum value of the output vector after applying a filter to the input. So it has nothing to do with the longest word but rather choose an element from the output that express the extracted feature to the highest amount. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-10-23T04:21:11.533" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="23995" PostTypeId="2" ParentId="21678" CreationDate="2017-10-23T07:52:26.633" Score="1" Body="&lt;p&gt;Zeros can be okay because they will have no impact on the input sum of the next layer. But you should train so the network that it has to know what to do when those are missing (no value coming from them). So you should dropout the input layer, possibly the way your data can be missing. Or you could just dropout with a probability between 0-1 the way that you make this probability random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to deal with missing data too, so I think I will try it out.&lt;/p&gt;&#xA;" OwnerUserId="40851" LastActivityDate="2017-10-23T07:52:26.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="23998" PostTypeId="2" ParentId="23969" CreationDate="2017-10-23T09:20:13.613" Score="40" Body="&lt;p&gt;Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Doc2vec solution&lt;/strong&gt;&lt;br&gt;&#xA;You can train your doc2vec model following this &lt;a href=&quot;https://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;. You may want to perform some pre-processing steps like removing all &lt;strong&gt;stop words&lt;/strong&gt; (words like &quot;the&quot;, &quot;an&quot;, etc. that don't add much meaning to the sentence). Once you trained your model, you can find the similar sentences using following code.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import gensim  &#xA;&#xA;model = gensim.models.Doc2Vec.load('saved_doc2vec_model')  &#xA;&#xA;new_sentence = &quot;I opened a new mailbox&quot;.split(&quot; &quot;)  &#xA;model.docvecs.most_similar(positive=[model.infer_vector(new_sentence)],topn=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[('TRAIN_29670', 0.6352514028549194),&#xA; ('TRAIN_678', 0.6344441771507263),&#xA; ('TRAIN_12792', 0.6202734708786011),&#xA; ('TRAIN_12062', 0.6163255572319031),&#xA; ('TRAIN_9710', 0.6056315898895264)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above results are list of tuples for &lt;code&gt;(label,cosine_similarity_score)&lt;/code&gt;. You can map outputs to sentences by doing &lt;code&gt;train[29670]&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that the above approach will only give good results if your doc2vec model contains embeddings for words found in the new sentence. If you try to get similarity for some gibberish sentence like &lt;code&gt;sdsf sdf f sdf sdfsdffg&lt;/code&gt;, it will give you few results, but those might not be the actual similar sentences as your trained model may haven't seen these gibberish words while training the model. So try to train your model on as many sentences as possible to incorporate as many words for better results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Word2vec Solution&lt;/strong&gt;&lt;br&gt;&#xA;If you are using word2vec, you need to calculate the average vector for all words in every sentence and use cosine similarity between vectors.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def avg_sentence_vector(words, model, num_features, index2word_set):&#xA;    #function to average all words vectors in a given paragraph&#xA;    featureVec = np.zeros((num_features,), dtype=&quot;float32&quot;)&#xA;    nwords = 0&#xA;&#xA;    for word in words:&#xA;        if word in index2word_set:&#xA;            nwords = nwords+1&#xA;            featureVec = np.add(featureVec, model[word])&#xA;&#xA;    if nwords&amp;gt;0:&#xA;        featureVec = np.divide(featureVec, nwords)&#xA;    return featureVec&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Calculate Similarity &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import cosine_similarity&#xA;&#xA;#get average vector for sentence 1&#xA;sentence_1 = &quot;this is sentence number one&quot;&#xA;sentence_1_avg_vector = avg_sentence_vector(sentence_1.split(), model=word2vec_model, num_features=100)&#xA;&#xA;#get average vector for sentence 2&#xA;sentence_2 = &quot;this is sentence number two&quot;&#xA;sentence_2_avg_vector = avg_sentence_vector(sentence_2.split(), model=word2vec_model, num_features=100)&#xA;&#xA;sen1_sen2_similarity =  cosine_similarity(sentence_1_avg_vector,sentence_2_avg_vector)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39934" LastEditorUserId="39934" LastEditDate="2020-02-04T21:47:29.457" LastActivityDate="2020-02-04T21:47:29.457" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="23999" PostTypeId="1" CreationDate="2017-10-23T10:15:50.337" Score="1" ViewCount="91" Body="&lt;p&gt;I'm still training a Convolutional Neural Network model in Tensorflow to recognize age groups from facial images. I'm using the Adience 3D (frontalized with a 3D model - it might not be good data) dataset, with four folds for training, and one fold for testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently using three Conv-Conv-Pool layer sets, the first one's Conv layer using 32 filters with 5x5 size, the second and third using 64 filters with 3x3 size, and to top it all off, a Dense layer with flattened input, then a Dropout layer, dropping out the outputs of the dense layer by 20%, then on to the logits layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I normalized my cropped color images with the per-channel normalization - the &lt;code&gt;(A - np.mean(A)) / np.std(A)&lt;/code&gt; method for each 64x64 array in the 64x64x3 input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm training it with Batch Size 64 and 100 steps so as to evaluate the weights every 100 steps, and while I don't see much improvement for now, I could see it increasing from steps 100 to 500 (0.27 to 0.35), before dropping on stage 600 then going up again up to step 800 (0.30 - 0.34).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a sign I have to change my layer structure?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On that note, when should I change up my layer structure/hyperparameters like training steps, batch size, filter size, etc., and how?&lt;/p&gt;&#xA;" OwnerUserId="40625" LastEditorUserId="29169" LastEditDate="2021-02-15T22:21:59.573" LastActivityDate="2021-02-15T22:21:59.573" Title="Convolutional Neural Networks - When do I adjust parts of it, and how?" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24000" PostTypeId="1" CreationDate="2017-10-23T10:52:06.303" Score="3" ViewCount="2156" Body="&lt;p&gt;I was wondering how can we use trained neural network model's weights or hidden layer output for simple classification problem, and then use those for feature engineering and implement some boosting algorithm on the new engineered features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose,if we have 100 rows with 5 features (100x5) matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data:&#xA;      X                 Y&#xA;x1,x2,x3,x4,x5          y1  y2&#xA;0,1,2,3,4               0   1&#xA;3,2,5,6,4               1   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Network:&#xA;2 layers, input and softmax output, compile using cross entropy  .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we utilise trained weights or hidden layer output of above network and use it for feature engineering on original dataset and then apply some boosting algo on modified dataset and will it increase accuracy ?&lt;/p&gt;&#xA;" OwnerUserId="33134" LastEditorUserId="33134" LastEditDate="2017-10-23T14:03:17.770" LastActivityDate="2017-12-04T06:34:01.433" Title="How to use neural network's hidden layer output for feature engineering?" Tags="|machine-learning|neural-network|feature-selection|feature-engineering|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="24002" PostTypeId="1" AcceptedAnswerId="24008" CreationDate="2017-10-23T11:40:31.693" Score="0" ViewCount="1272" Body="&lt;p&gt;I'm trying to use Speeded Up Robust Features (SURF) to get the $k$ most similar images from a set of images in my directory. I'm planning to use $k$-Nearest Neighbours ($k$-NN) for this. As far as I know, the size of SURF descriptors are $n \times 64$ or $n \times 128$ depending on how many descriptors I want. There are suggestions and I've read about the &lt;em&gt;Bag of Visual Words&lt;/em&gt; method, where these patches are converted to bags of words, similar to the common Natural Language Processing technique. I've also read that Bag of Visual Words are generated by clustering, whereas the feature patches are clustered together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is, how can I use these Bag of Visual Words to train my $k$-NN such that I can get similar images? I really can't grasp how those clusters can generate BoWs. Say I have 1000 images, if I convert them, what will they look like? Will they be 1000 BoWs that still represent the same images? &lt;/p&gt;&#xA;" OwnerUserId="40857" LastEditorUserId="17538" LastEditDate="2017-10-23T18:00:27.150" LastActivityDate="2017-10-23T18:00:27.150" Title="How do I represent SURF Features into Bag of Words to determine Nearest Neighbors?" Tags="|machine-learning|feature-extraction|k-means|image-recognition|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24003" PostTypeId="2" ParentId="17759" CreationDate="2017-10-23T13:57:47.930" Score="33" Body="&lt;p&gt;Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://medium.com/towards-data-science/top-6-errors-novice-machine-learning-engineers-make-e82273d394db&quot; rel=&quot;noreferrer&quot;&gt;More Info&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="15250" LastEditorUserId="29575" LastEditDate="2017-10-23T14:13:03.267" LastActivityDate="2017-10-23T14:13:03.267" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="24006" PostTypeId="2" ParentId="17759" CreationDate="2017-10-23T16:14:03.097" Score="8" Body="&lt;p&gt;&lt;strong&gt;I recommend using numerical features.&lt;/strong&gt; Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the issue that you raise is that you want to represent hours and months in a manner where 12 is as close to 11 as it is to 1. In order to achieve that, I recommend going with what was suggested in the comments and using a sine/cosine function before using the hours/months as numerical features.&lt;/p&gt;&#xA;" OwnerUserId="40871" LastActivityDate="2017-10-23T16:14:03.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24008" PostTypeId="2" ParentId="24002" CreationDate="2017-10-23T17:26:31.670" Score="3" Body="&lt;p&gt;The Bags of Visual Words (BoWs) approach to image retrieval, described in works such as Sivic et al. &quot;Video Google: a text retrieval approach to object matching in videos&quot; (2003) and Csurka et al. &quot;Visual categorization with bags of keypoints&quot; (2004), is composed of multiple phases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;First, a visual vocabulary, often called a &lt;strong&gt;codebook&lt;/strong&gt;, is generated. This is usually done by applying k-means clustering over the keypoint descriptors of a data set, or a sufficiently descriptive fraction of it. The vector $\mathcal{V}$ of size $k$ containing the centroids $\mathcal{V_i}$ of each cluster is your visual vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In this case, each image $x$ should yield a variable number $x_n$ of SURF keypoint descriptors, usually of size 64 each. One can aggregate all keypoint descriptors from multiple images and perform k-means clustering over all of them.&lt;/li&gt;&#xA;&lt;li&gt;The choice of the $k$ hyperparameter in clustering depends on the image domain. One may try multiple values of $k$ (10, 100, 1000) to understand which is more suitable for the intended task.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Afterwards, each image is &quot;tested&quot; against the codebook, by determining the closest visual vocabulary points and incrementing the corresponding positions in the BoW for each keypoint descriptor in the image. In other words, considering an image's BoW $B = \{ o_i \}$: for each image keypoint descriptor $d_j$, $o_i$ is incremented when the smallest distance (often the Euclidean distance) from $d_j$ to all other visual vocabulary points in $\mathcal{V}$ is the distance to $\mathcal{V}_i$.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The result is a histogram of visual descriptor occurrences of size $k$, which can be used for representing the visual content. As similar images will yield similar bags of words, one can compare images through their BoWs. The Euclidean distance between them is a commonly used metric here.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Therefore, the bag of words of each image makes a global representation of  that image. When performing content-based image retrieval, the $n$ most similar images are retrieved by fetching the $n$ closest (or approximately closest) bags of words to the query image's. No training process is required at this point (we can picture visual vocabulary generation as an offline training phase).&lt;/p&gt;&#xA;" OwnerUserId="17538" LastEditorUserId="17538" LastEditDate="2017-10-23T17:33:52.583" LastActivityDate="2017-10-23T17:33:52.583" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24009" PostTypeId="2" ParentId="23999" CreationDate="2017-10-23T18:20:43.057" Score="2" Body="&lt;p&gt;If you wanna get the optimal parameters, you can try using grid search. A &lt;em&gt;grid search&lt;/em&gt; or &lt;em&gt;parameter sweep&lt;/em&gt; does an exhaustive searching through a specified subset (you have to define this subset) of the hyperparameter space of the learning algorithm which in your case is the Convolutional Neural Network. Although this may take a long time but this is one of the ways you'll know the good parameters for your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try searching for optimization algorithms or parameter tuning techniques.&lt;/p&gt;&#xA;" OwnerUserId="40857" LastActivityDate="2017-10-23T18:20:43.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24010" PostTypeId="1" AcceptedAnswerId="24018" CreationDate="2017-10-23T19:39:51.150" Score="1" ViewCount="351" Body="&lt;p&gt;I'm trying to replicate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Forgetting_curve&quot; rel=&quot;nofollow noreferrer&quot;&gt;forgetting curve&lt;/a&gt; using the open sourced &lt;a href=&quot;https://github.com/duolingo/halflife-regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Duolingo data&lt;/a&gt; for fun.&#xA;The problem is that my finding doesn't make any sense, namely that the longer you wait the better recall value. Anyone have any pointers?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# make our plot outputs appear and be stored within the notebook.&#xA;%matplotlib inline &#xA;&#xA;import matplotlib.pyplot as plt # import matplotlib for scatterplot, use the alias plt&#xA;import numpy as np # import the numpy package with alias np&#xA;import pandas as pd&#xA;from scipy.optimize import curve_fit&#xA;&#xA;df = pd.read_csv('learning_traces.13m.csv') # Load Duolingo data&#xA;df = df[(df['history_seen'] == 1) &amp;amp; (df['session_seen'] == 1)] # Seen only once before&#xA;df = df.sort_values('delta') # sort by: time (in seconds) since the last lesson/practice that included this word/lexeme&#xA;df['delta'] = df['delta'] / 60.0 # Seconds to minutes&#xA;minInMonth = 44640&#xA;&#xA;def func(x, a):&#xA;    return np.exp2(-x / a)&#xA;&#xA;xdata = df['delta']&#xA;ydata = df['p_recall']&#xA;&#xA;plt.scatter(xdata, ydata)&#xA;&#xA;popt, pcov = curve_fit(func, xdata, ydata)&#xA;print(popt) # Show the result&#xA;plt.plot(xdata, func(xdata, *popt), 'r-', label='fit')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Expectation:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6r9SI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6r9SI.png&quot; alt=&quot;Expectation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reality&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dRRkp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dRRkp.png&quot; alt=&quot;Reality&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rolling Mean&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aU8zN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aU8zN.png&quot; alt=&quot;Rolling Mean&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13281" LastEditorUserId="13281" LastEditDate="2017-10-24T12:33:51.240" LastActivityDate="2017-10-24T12:33:51.240" Title="Forgetting curve using Duolingo data" Tags="|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24011" PostTypeId="2" ParentId="11921" CreationDate="2017-10-23T20:40:55.733" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://tensorflow.rstudio.com/tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow for R&lt;/a&gt; is available.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It provides full access to &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow API&lt;/a&gt;, the &lt;a href=&quot;https://tensorflow.rstudio.com/keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras API&lt;/a&gt;, and &lt;a href=&quot;https://tensorflow.rstudio.com/tfestimators/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow Estimators&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Installation of Tensorflow (excerpt below) -&gt; &lt;a href=&quot;https://tensorflow.rstudio.com/tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://tensorflow.rstudio.com/tensorflow/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h1&gt;Installation&lt;/h1&gt;&#xA;  &#xA;  &lt;p&gt;To get started, install the tensorflow R package from GitHub as&#xA;  follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;devtools::install_github(&quot;rstudio/tensorflow&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;Then, use the install_tensorflow() function to install TensorFlow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tensorflow)&#xA;install_tensorflow() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;You can confirm that the installation succeeded with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess = tf$Session() hello &amp;lt;- tf$constant('Hello, TensorFlow!')&#xA;sess$run(hello) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;This will provide you with a default installation of&#xA;  TensorFlow suitable for getting started with the tensorflow R package.&#xA;  See the article on installation to learn about more advanced options,&#xA;  including installing a version of TensorFlow that takes advantage of&#xA;  Nvidia GPUs if you have the correct CUDA libraries installed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4766" LastActivityDate="2017-10-23T20:40:55.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24013" PostTypeId="1" CreationDate="2017-10-23T20:54:08.883" Score="-2" ViewCount="470" Body="&lt;p&gt;I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know how to prepare the data and train the ML model. Currently, I am considering the duplicate record as issues and training the SVM model for one-class classification.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Duplicate Bill&#xA;    ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should I train the model with actual bill and duplicate bill as both issues and train the model and then predict the rest or is there any other way to consider or should I implement time-series classification?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Records &#xA;Actual Bill                                                     label&#xA;ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1&#xA;Duplicate Bill&#xA;ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40879" LastEditorUserId="85045" LastEditDate="2020-12-11T17:42:23.713" LastActivityDate="2020-12-11T17:42:23.713" Title="Training a model to predict duplicate records in machine learning" Tags="|machine-learning|dataset|svm|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24014" PostTypeId="2" ParentId="24013" CreationDate="2017-10-23T21:25:58.457" Score="1" Body="&lt;p&gt;This really isn't a machine learning task, but a database task. Unless you want indication why such records appear as duplicates, but that is more of a debugging task.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-23T21:25:58.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24015" PostTypeId="2" ParentId="21974" CreationDate="2017-10-23T21:28:38.113" Score="0" Body="&lt;p&gt;You can use &lt;code&gt;dt&lt;/code&gt; from the &lt;code&gt;stats&lt;/code&gt; to get the density of a Student-t distribution. See the &lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/TDist.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;help page&lt;/a&gt; for extra information about this, and related, functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example, showing the Student-t distrib&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(stats)&#xA;&#xA;xs = seq(-5, 5, .1)&#xA;density = dt(xs, df=1)&#xA;&#xA;plot(xs, density)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PEw15.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PEw15.png&quot; alt=&quot;Student-t distribution&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2017-10-23T21:28:38.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24018" PostTypeId="2" ParentId="24010" CreationDate="2017-10-23T23:05:32.217" Score="2" Body="&lt;p&gt;I think the most important problem you are facing is that you are trying to fit a function that somewhat erratically flips between 0 and 1 as if it were a smooth function. My Mathematical intuition says that won't be numerically stable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create a more smooth function you could calculate a running average of your y data and plot it against the corresponding time data.&#xA;It should descend as you forget more over a longer period of time and should be easy to plot to demonstrate your intuition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has hysteresis/history since initial successes will be remembered. Your fit function would be the integral: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\int_0^X e^{-x/a} dx = a - a\cdot e^{-X/a}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $a$ will be the estimator of the forgetting rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other problems could be in the time domain (just leave the unit of time at seconds), or you are mixing time scale/sample scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both your expectation and reality plot should cover the complete time range on the x-axis to make them better comparable.&#xA;Maybe you should throw away data with exceptionally large time delta values anyway, as they are likely to disturb the calculations and the plots. The programs from your references remove data &gt; 9 months and &amp;lt; 45 minutes if I recall correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another domain problem that could interfere is that the data mixed words that are easy to remember and words that are difficult to remember, e.g. 'cat' is in many languages easier to remember than 'butterfly'  (though that could also average out over much measurement data).&lt;/p&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-24T11:59:47.637" LastActivityDate="2017-10-24T11:59:47.637" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24019" PostTypeId="1" AcceptedAnswerId="24027" CreationDate="2017-10-24T00:59:04.663" Score="1" ViewCount="700" Body="&lt;p&gt;In the application I am developing, I have about 5000 product label images.(One label per product).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One functionality of my application is that user can take a picture using his camera and get a possible match(es) against the product labels registered the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since initially, my system only has one sample per product, I decided to go with traditional Computer Vision techniques. I managed to implement this using Feature extraction and Descriptor matching.(using OpenCV SIFT and FLANN techniques referring this: &lt;a href=&quot;https://github.com/kipr/opencv/blob/master/samples/cpp/matching_to_many_images.cpp&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/kipr/opencv/blob/master/samples/cpp/matching_to_many_images.cpp&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am thinking how to improve the accuracy by combining with CNN or Deep Learning techniques since when users approve matches, it gradually add more label samples for a product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to build a hybrid image matching system combining Computer Vision techniques and CNN/Deep Learning techniques?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any similar services already available as services?&lt;/p&gt;&#xA;" OwnerUserId="15897" LastActivityDate="2020-04-17T19:15:09.163" Title="Match an image from a set of images : Combine traditional Computer vision + Deep Learning/CNN" Tags="|machine-learning|deep-learning|image-classification|computer-vision|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24020" PostTypeId="1" CreationDate="2017-10-24T01:29:15.660" Score="1" ViewCount="252" Body="&lt;p&gt;Given three set of data with categorical integer x-axis with the same range (0-10):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from itertools import chain&#xA;from collections import Counter, defaultdict&#xA;&#xA;from IPython.display import Image&#xA;import pandas as pd&#xA;import numpy as np&#xA;&#xA;import seaborn as sns&#xA;import colorlover as cl&#xA;import matplotlib.pyplot as plt&#xA;&#xA;data1 = Counter({8: 10576, 9: 10114, 7: 9504, 6: 7331, 10: 6845, 5: 5007, 4: 3037, 3: 1792, 2: 908, 1: 368, 0: 158})&#xA;data2 = Counter({5: 9030, 6: 8347, 4: 8149, 7: 6822, 3: 6153, 8: 5075, 2: 3973, 9: 3205, 1: 2247, 10: 1580, 0: 1059})&#xA;data3 = Counter({4: 9884, 5: 9058, 3: 9048, 2: 6942, 6: 6630, 1: 4596, 7: 3784, 0: 2524, 8: 1912, 9: 917, 10: 345})&#xA;&#xA;df_data1 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data1.items()])))&#xA;df_data2 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data2.items()])))&#xA;df_data3 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data3.items()])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can plot three different barcharts for each set of data as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sns.set(style=&quot;white&quot;)&#xA;g = sns.factorplot(x=0, data=df_data1, kind=&quot;count&quot;, color='#4882D5', size=6, aspect=1.5, order=None)&#xA;g.set_xticklabels(step=1)&#xA;plt.show()&#xA;g = sns.factorplot(x=0, data=df_data2, kind=&quot;count&quot;, color='#9CD2D5', size=6, aspect=1.5, order=None)&#xA;g.set_xticklabels(step=1)&#xA;plt.show()&#xA;g = sns.factorplot(x=0, data=df_data3, kind=&quot;count&quot;, color='#EF988D', size=6, aspect=1.5, order=None)&#xA;g.set_xticklabels(step=1)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[out]:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jPXqP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jPXqP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/EIIsU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EIIsU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HoJFR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HoJFR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to highlight is the fact that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;for &lt;code&gt;data1&lt;/code&gt;, we see that the center of mass is inclined to the right where the x-axis is higher at 8&lt;/li&gt;&#xA;&lt;li&gt;for &lt;code&gt;data2&lt;/code&gt;, it's normally shaped where the center is at 5&lt;/li&gt;&#xA;&lt;li&gt;for &lt;code&gt;data3&lt;/code&gt;, the center of mass is at 4 on the lower side. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Imagine:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;data1&lt;/code&gt; x-axis represents the no. of &lt;strong&gt;blueberry&lt;/strong&gt; a person buys at the supermarket&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;data2&lt;/code&gt; x-axis represents the no. of &lt;strong&gt;watermelon&lt;/strong&gt; a person buys at the supermarket&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;data3&lt;/code&gt; x-axis represents the no. of &lt;strong&gt;oranges&lt;/strong&gt; a person buys at the supermarket&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The y-axis for all 3 datasets represent the number of people who buys the respective no. of fruits as per the dataset. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's a little hard to convey the message with the barcharts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a better way to represent the dataset and highlight what I want to say?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a way to present the 3 graphs as one? How would it look like?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="1330" LastEditDate="2022-09-07T12:46:00.393" LastActivityDate="2022-09-07T12:46:00.393" Title="How to better represent three sets of categorical data?" Tags="|visualization|plotting|seaborn|counts|histogram|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24022" PostTypeId="1" CreationDate="2017-10-24T02:44:50.557" Score="-2" ViewCount="141" Body="&lt;p&gt;I have a script that scans text files and searches for keywords related to cryptography. And I'm scanning source code files. A lot of the times, I get back some evidence of cryptography like &quot;AES_encrypt&quot;, but sometimes it can be a false positive, e.g. the keyword &quot;crypt&quot; in the string &quot;cryptic words&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get rid of false positives, I designed an app that allows a human expert to go in and flag each hit as positive or false-positive. I want to gather this data and train a supervised classifier that learns from the human's expertise and gives a probability that a certain future hit is a positive. My training data looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keyword | keyword_group | is_source_file | P/FP&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | True | P&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | True | P&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | False | P&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | True | P&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | True | FP&#xA;&quot;aes_encrypt&quot; | &quot;aes&quot; | False | FP&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | True | P&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | True | P&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | True | P&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | True | FP&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | False | P&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | False | FP&#xA;&quot;decrypt_aes&quot; | &quot;aes&quot; | False | FP&#xA;&quot;rsa_key&quot; | &quot;rsa&quot; | True | P&#xA;&quot;rsa_key&quot; | &quot;rsa&quot; | True | P&#xA;&quot;rsa_key&quot; | &quot;rsa&quot; | True | P&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Keywords are organized in groups. For example, &quot;aes_encrypt&quot; and &quot;aes_decrypt&quot; are both under &quot;aes&quot; group. &lt;code&gt;is_source_file&lt;/code&gt; indicates whether this hit was found in a source code file like a &quot;.cpp&quot; or a &quot;.py&quot; file as opposed to other file extensions like a &quot;.txt&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to give the classifier a keyword and its group and whether it was found in a source file, and I want back a probability that it is a positive. What kind of classifier do you recommend for this job? Would SVM be a good choice?&lt;/p&gt;&#xA;" OwnerUserId="40889" LastEditorUserId="40889" LastEditDate="2017-10-24T03:20:08.650" LastActivityDate="2017-10-24T03:42:59.000" Title="Will SVM classifier be a good choice?" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24023" PostTypeId="2" ParentId="24022" CreationDate="2017-10-24T03:42:59.000" Score="1" Body="&lt;p&gt;There are a lot of classifiers that can do this job for you. I'll name and summarize a few here and see whether it fits your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Logistic Regression&lt;/strong&gt; is a probabilistic approach where it gives informarion about the statistical significance of features but you have to account for the Logistic Regression assumptions. (as far as I can recall the assumptions are similar to that of Linear Regression. You can Google this to find out more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;K-Nearest Neighbors&lt;/strong&gt; is easy to understand and to implement. What this algorithm does is to check what group the input is nearest to. You have to choose the number of &lt;em&gt;k&lt;/em&gt; though since the algorithm determines which classification it gives the input by a majority vote.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Support Vector Machine&lt;/strong&gt; is one that you have mentioned so I'm assuming that you know how this works. The advantage of SVM is that it is not based on outliers and it is not sensitive to overfitting. However, it does not perform well on non-linear problems and it is not the best choice for data that has a lot of features&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Kernel Support Vector Machine&lt;/strong&gt; in order to address the problem that SVM only works for linear problems, what Kernel SVM does is that it uses the kernel trick. The kernel trick basically maps the data to a higher dimension such that they can be linearly separable. An example of a kernel trick is using the Gaussian or Radian Basis Function for SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Naive Bayes&lt;/strong&gt; is a probabilistic approach. It is efficient, not based on outliers and works well on non-linear problems. However the caveat here is that it assumes that all the features have the same statistical relevance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Decision Trees&lt;/strong&gt; is an approach that can be easily interpretable since it creates trees where the leaf nodes are the classes. It can work on both linear and non-linear problems but it performs better when using a large dataset as opposed to a small one. It is also easily susceptible to overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a lot more but these are some that you can choose from to have more details on these algorithms you can do a quick Google search about it. &lt;/p&gt;&#xA;" OwnerUserId="40857" LastActivityDate="2017-10-24T03:42:59.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24024" PostTypeId="1" CreationDate="2017-10-24T05:45:56.360" Score="2" ViewCount="64" Body="&lt;p&gt;Currently, I am building a credit rating model based on logistic regression and faced a problem with inserting panel variables in it. Is it possible to do so in logistic regression and if it is, what packages and functions can deal with it?&lt;/p&gt;&#xA;" OwnerUserId="40898" LastEditorUserId="29575" LastEditDate="2017-10-25T09:36:20.277" LastActivityDate="2017-10-25T09:36:20.277" Title="Is it possible to run logistic regression on data with a few longitudinal parameters?" Tags="|machine-learning|r|logistic-regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24025" PostTypeId="2" ParentId="23895" CreationDate="2017-10-24T07:45:05.563" Score="5" Body="&lt;ol&gt;&#xA;&lt;li&gt;For TensorFlow:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/tutorials/using_gpu&quot; rel=&quot;noreferrer&quot;&gt;TensorFlow Using GPUs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the sample code on how is used, so for each task is specified the list with &lt;em&gt;devices/device:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Creates a graph.&#xA;c = []&#xA;for d in ['/gpu:2', '/gpu:3']:&#xA;  with tf.device(d):&#xA;    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])&#xA;    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])&#xA;    c.append(tf.matmul(a, b))&#xA;with tf.device('/cpu:0'):&#xA;  sum = tf.add_n(c)&#xA;# Creates a session with log_device_placement set to True.&#xA;sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))&#xA;# Runs the op.&#xA;print(sess.run(sum))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;tf will use GPU by default for computation even if is for CPU (if is present supported GPU). so you can just do a for loop: &quot;for d in ['/gpu:1','/gpu:2', '/gpu:3' ... '/gpu:8',]:&quot; and in the &quot;tf.device(d)&quot; should include all your instance GPU resources. So tf.device() will actually be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://devblogs.nvidia.com/parallelforall/scaling-keras-training-multiple-gpus/&quot; rel=&quot;noreferrer&quot;&gt;Scaling Keras Model Training to Multiple GPUs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Keras&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For Keras by using Mxnet than &lt;em&gt;args.num_gpus&lt;/em&gt;, where &lt;strong&gt;num_gpus&lt;/strong&gt; is the list of your required GPUs. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def backend_agnostic_compile(model, loss, optimizer, metrics, args):&#xA;  if keras.backend._backend == 'mxnet':&#xA;      gpu_list = [&quot;gpu(%d)&quot; % i for i in range(args.num_gpus)]&#xA;      model.compile(loss=loss,&#xA;          optimizer=optimizer,&#xA;          metrics=metrics, &#xA;          context = gpu_list)&#xA;  else:&#xA;      if args.num_gpus &amp;gt; 1:&#xA;          print(&quot;Warning: num_gpus &amp;gt; 1 but not using MxNet backend&quot;)&#xA;      model.compile(loss=loss,&#xA;          optimizer=optimizer,&#xA;          metrics=metrics)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;horovod.tensorflow&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;On top of all Uber open sourced Horovod recently and I think is great:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://eng.uber.com/horovod/&quot; rel=&quot;noreferrer&quot;&gt;Horovod&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import horovod.tensorflow as hvd&#xA;&#xA;# Initialize Horovod&#xA;hvd.init()&#xA;&#xA;# Pin GPU to be used to process local rank (one GPU per process)&#xA;config = tf.ConfigProto()&#xA;config.gpu_options.visible_device_list = str(hvd.local_rank())&#xA;&#xA;# Build model…&#xA;loss = …&#xA;opt = tf.train.AdagradOptimizer(0.01)&#xA;&#xA;# Add Horovod Distributed Optimizer&#xA;opt = hvd.DistributedOptimizer(opt)&#xA;&#xA;# Add hook to broadcast variables from rank 0 to all other processes during&#xA;# initialization.&#xA;hooks = [hvd.BroadcastGlobalVariablesHook(0)]&#xA;&#xA;# Make training operation&#xA;train_op = opt.minimize(loss)&#xA;&#xA;# The MonitoredTrainingSession takes care of session initialization,&#xA;# restoring from a checkpoint, saving to a checkpoint, and closing when done&#xA;# or an error occurs.&#xA;with tf.train.MonitoredTrainingSession(checkpoint_dir=“/tmp/train_logs”,&#xA;                                      config=config,&#xA;                                      hooks=hooks) as mon_sess:&#xA; while not mon_sess.should_stop():&#xA;   # Perform synchronous training.&#xA;   mon_sess.run(train_op)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2017-10-24T07:45:05.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24027" PostTypeId="2" ParentId="24019" CreationDate="2017-10-24T08:26:25.443" Score="2" Body="&lt;p&gt;In my opinion, if you want a hybrid of Convolutional Neural Networks and the classic feature extraction techniques that would be redundant. Mainly because the architecture of a Convolutional Neural Network is composed of convolutions. I won't go much into detail of the whole architecture but these convolutions actually extract the good features for you and then those convolutions are connected to a classic Neural Network that does the classification task. Hence, extracting features using SIFT and using Convolutional Neural Networks would be redundant. In addition, the features that CNN will extract are better as compared to simple SIFT features. Though if you want to push through with combining SIFT and Deep Learning, you can instead substitute convolutions and use SIFT for feature extraction and then feed it into a Neural Network. That would work as well. &lt;/p&gt;&#xA;" OwnerUserId="40857" LastActivityDate="2017-10-24T08:26:25.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24028" PostTypeId="2" ParentId="17891" CreationDate="2017-10-24T09:53:01.640" Score="1" Body="&lt;p&gt;I had a similar issue. I tried using &lt;code&gt;train&lt;/code&gt; from the caret package to solve the issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code will look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ctrl &amp;lt;- trainControl(method = &quot;cv&quot;, &#xA;                     summaryFunction = twoClassSummary, &#xA;                     classProbs = TRUE)&#xA;&#xA;fit = train(formula,data= creditdataset, method = &quot;gbm&quot;,trControl = ctrl,metric = &quot;ROC&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Make sure you see other options in &lt;code&gt;trControl&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="40905" LastEditorUserId="29575" LastEditDate="2017-10-24T14:22:53.240" LastActivityDate="2017-10-24T14:22:53.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24029" PostTypeId="1" CreationDate="2017-10-24T10:01:13.910" Score="-2" ViewCount="380" Body="&lt;p&gt;I'm a project manager and I've been tasked to work with a data scientists. As a project manager it's my role to create a plan which I will monitor and report against.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data scientist has experience in his role however is struggling provide a breakdown of the work that he intends to do. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example the initial task is to analyse customer data held on our database to identify potential insights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can data science projects be Project managed by identifying time constrained tasks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you direct me towards some examples or resources which may help breakdown data science projects? &lt;/p&gt;&#xA;" OwnerUserId="40885" LastEditorUserId="40885" LastEditDate="2017-10-25T13:00:05.990" LastActivityDate="2017-10-25T13:00:05.990" Title="Data Science Project Manager - Planning" Tags="|data|" AnswerCount="1" CommentCount="1" ClosedDate="2017-10-24T22:02:35.613" ContentLicense="CC BY-SA 3.0" />
  <row Id="24032" PostTypeId="2" ParentId="24020" CreationDate="2017-10-24T11:06:22.793" Score="1" Body="&lt;p&gt;You shoud put the barcharts next to each other as described in &lt;a href=&quot;https://matplotlib.org/devdocs/gallery/api/barchart.html#sphx-glr-gallery-api-barchart-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;N = 5&#xA;men_means = (20, 35, 30, 35, 27)&#xA;men_std = (2, 3, 4, 1, 2)&#xA;&#xA;ind = np.arange(N)  # the x locations for the groups&#xA;width = 0.35       # the width of the bars&#xA;&#xA;fig, ax = plt.subplots()&#xA;rects1 = ax.bar(ind, men_means, width, color='r', yerr=men_std)&#xA;&#xA;women_means = (25, 32, 34, 20, 25)&#xA;women_std = (3, 5, 2, 3, 3)&#xA;rects2 = ax.bar(ind + width, women_means, width, color='y', yerr=women_std)&#xA;&#xA;# add some text for labels, title and axes ticks&#xA;ax.set_ylabel('Scores')&#xA;ax.set_title('Scores by group and gender')&#xA;ax.set_xticks(ind + width / 2)&#xA;ax.set_xticklabels(('G1', 'G2', 'G3', 'G4', 'G5'))&#xA;&#xA;ax.legend((rects1[0], rects2[0]), ('Men', 'Women'))&#xA;&#xA;&#xA;def autolabel(rects):&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    Attach a text label above each bar displaying its height&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    for rect in rects:&#xA;        height = rect.get_height()&#xA;        ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,&#xA;                '%d' % int(height),&#xA;                ha='center', va='bottom')&#xA;&#xA;autolabel(rects1)&#xA;autolabel(rects2)&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FZpii.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FZpii.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32860" LastEditorUserId="98307" LastEditDate="2020-08-10T09:17:07.703" LastActivityDate="2020-08-10T09:17:07.703" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24035" PostTypeId="2" ParentId="24029" CreationDate="2017-10-24T11:53:04.617" Score="1" Body="&lt;p&gt;See mine (and others) answers here regarding &lt;a href=&quot;https://www.quora.com/Whats-the-best-way-to-work-with-a-data-scientist&quot; rel=&quot;nofollow noreferrer&quot;&gt;how to work with a data scientist&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem you have with planning is real and severe. We all have a problem in planning research. As Albert Einstein said &quot;If we knew what it was we were doing, it would not be called research, would it?&quot;.&#xA;Though it is hard, we are not the only one having to deal with uncertainty in planning. The common methods of dealing with uncertainty are useful here too. Your idea to break down the task is great. So is performing quick tests (POC like) in order to see is a direction is valuable.&#xA;The main problem in research that it is very common that good direction will be found to be not beneficial only after a lot of work. You might be surprised from the other side, having great ideas coming when somebody takes a shower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the problem in estimation, people hesitate to do it. Though, they realize that &quot;You'll have it when it ready&quot; is not an answer that a customer will accept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is indeed not a pure data science question and so is my answer. Understand that planning is hard, do try to do so and expect your planning to be very inaccurate.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-10-24T11:53:04.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24038" PostTypeId="1" CreationDate="2017-10-24T12:33:54.673" Score="1" ViewCount="675" Body="&lt;p&gt;So I have a &lt;code&gt;DataFrame&lt;/code&gt; which consists of order data from customer on an exchange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a column &lt;code&gt;Dollars&lt;/code&gt; which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the &lt;code&gt;Dollar&lt;/code&gt; column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dollars   | Normalized dollars&#xA;181447.50 | 9.10975e-06&#xA;281885.00 | 1.41523e-05&#xA;290786.00 | 1.45992e-05&#xA;70923.00  | 3.56076e-06&#xA;1121169.54| 5.62894e-05&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These values seem &lt;em&gt;tiny&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used &lt;code&gt;sklearn.preprocessing.normalize&lt;/code&gt; for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's worth mentioning that the lowest &lt;code&gt;Dollars&lt;/code&gt; value is &lt;code&gt;0.06&lt;/code&gt; while the largest is &lt;code&gt;5,605,847,772.52&lt;/code&gt;, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is &lt;code&gt;0.157&lt;/code&gt;. Should I do more filtering on the dataset to remove extreme outliers as a general rule?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&#xA;Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small. &lt;code&gt;ScaledDollars&lt;/code&gt; is the result of using &lt;code&gt;sklearn.StandardScaler&lt;/code&gt; on &lt;code&gt;Log10Dollars&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dollars   | Normalized dollars| Log10Dollars | LogNorm   | ScaledDollars&#xA;181447.50 | 9.10975e-06       | 5.258751     | 0.00384193| -0.761916&#xA;281885.00 | 1.41523e-05       | 5.450072     | 0.00398171| -0.573336&#xA;290786.00 | 1.45992e-05       | 5.463573     | 0.00399157| -0.560028&#xA;70923.00  | 3.56076e-06       | 4.850787     | 0.00354388| -1.16404&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39014" LastEditorUserId="39014" LastEditDate="2017-10-24T13:12:17.460" LastActivityDate="2017-10-25T07:53:51.243" Title="Is my normalization off?" Tags="|classification|preprocessing|normalization|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24039" PostTypeId="2" ParentId="24038" CreationDate="2017-10-24T12:43:30.270" Score="3" Body="&lt;p&gt;You should use scaling instead of normalizing. By normalizing the column you make the &lt;code&gt;norm&lt;/code&gt; 1, while you are interested in making the expected value 0 with a standard deviation of 1. With the norm, the more samples you have the smaller your samples will become. Use the scikit-learn StandardScaler instead. Despite this, with these huge numbers you will still get weird results (although most of them will be negative with this skew). What could be an alternative is using a log transformation on your dollars first, and then scaling them. Conceptually you usually look at relative values between dollars, the difference between 10 and 20 dollars is similar to that of 10,000 and 20,000 which the log transformation will capture.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-10-24T12:43:30.270" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24040" PostTypeId="2" ParentId="24038" CreationDate="2017-10-24T12:44:07.693" Score="2" Body="&lt;p&gt;It is not surprising to obtain values that tiny because all your values are being scaled down by $5,605,847,772.46$ after being subtracted by $0.06$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about doing lognormalization? Apparently it is not uncommon to use this on financial data like explained &lt;a href=&quot;https://www.r-statistics.com/2013/05/log-transformations-for-skewed-and-wide-distributions-from-practical-data-science-with-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT (For completeness of answer)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After taking the Logarithm of each of your value, you might want to use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MinMaxScaler()&lt;/a&gt; instead of L2 Norm of scikit-learn if you want your values to lie beween 0-1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If an element '$x_i$' in &lt;code&gt;Log10Dollars&lt;/code&gt; after Normalization/Scaling is represented as '$\bar{x_i}$' then,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Using L2 Norm from sklearn: $\bar{x_i} = \frac{x_i}{\sqrt{\sum_{i=1}^{n}{x_i^2}}}$&lt;/li&gt;&#xA;&lt;li&gt;Using &lt;code&gt;MinMaxScaler()&lt;/code&gt;: $\bar{x_i} = \frac{x_i−min(x)}{max(x)−min(x)}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The second representation will give you $\bar{x_i} \in [0,1]$. &lt;/p&gt;&#xA;" OwnerUserId="39234" LastEditorUserId="39234" LastEditDate="2017-10-25T07:53:51.243" LastActivityDate="2017-10-25T07:53:51.243" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24041" PostTypeId="1" CreationDate="2017-10-24T13:37:18.193" Score="1" ViewCount="3045" Body="&lt;p&gt;If I have a dataset with events occuring at certain times of day, &lt;code&gt;Hour&lt;/code&gt;, how would I go about using this for, say, a classifier? Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hour     | Event | Item&#xA;08:45:22 | Buy   | Apple&#xA;09:03:10 | Buy   | Orange&#xA;10:00:00 | Sell  | Apple&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would I convert the &lt;code&gt;Hour&lt;/code&gt; timestamp into a numeric value, such as &lt;code&gt;Unit Timestamp&lt;/code&gt; and then do normalization on it, like I would any other numeric value? &lt;/p&gt;&#xA;" OwnerUserId="39014" LastActivityDate="2017-10-24T16:26:36.190" Title="Normalizing time data" Tags="|preprocessing|normalization|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24043" PostTypeId="2" ParentId="24041" CreationDate="2017-10-24T14:15:30.990" Score="1" Body="&lt;p&gt;Since you are dealing with cyclic events (&lt;code&gt;Hour&lt;/code&gt; column goes from 00:00:00 to 23:59:59 and then back again), you could transform the columns into 'seconds' and then use &lt;code&gt;sine, cosine&lt;/code&gt; transformation. See &lt;a href=&quot;https://datascience.stackexchange.com/questions/17759/encoding-features-like-month-and-hour-as-categorial-or-numeric/24003#24003&quot;&gt;this&lt;/a&gt; similar question.&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-24T14:15:30.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24045" PostTypeId="2" ParentId="22581" CreationDate="2017-10-24T15:47:24.733" Score="4" Body="&lt;h2&gt;STEP 1&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;When you do Regression on this dataset, what you obtain is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{60mm}f(x_1,x_2,x_3,x_4) = ERI$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where your $x_1,x_2,x_3,x_4, ERI$ are &lt;code&gt;O2 level&lt;/code&gt;, &lt;code&gt;Cylinder Pressure&lt;/code&gt;, &lt;code&gt;Fuel Flow,&lt;/code&gt;, &lt;code&gt;Engine Temp&lt;/code&gt;, and &lt;code&gt;Engine Running Index&lt;/code&gt;respecetively.&#xA;In case of &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html&quot; rel=&quot;noreferrer&quot;&gt;Multiple Regression&lt;/a&gt;, you will get 1 coefficient per feature (total 4 in &lt;code&gt;coef_&lt;/code&gt;, {$w_1,w_2,w_3,w_4$}) and one &lt;code&gt;intercept_&lt;/code&gt; ($w_0$) term for the intercept. Your equation would then come to look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{40mm}ERI = w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w_0$ - $w_4$ are fixed values. Your objective then is to find the best $x_1$ - $x_4$ which maximize $ERI$. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;STEP 2&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The above is better expressed by the optimization problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\max_{(x_1,x_2,x_3,x_4)} w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$&#xA;with constraints that $x_1,x_2,x_3,x_4 \geq 0$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I added those constraints by default as all those variables would be positive only. You could have upper limits on each of them as well. It is important to add these constraints because you don't want any of your value to reach $\infty$ or even some unrealistic values like &lt;code&gt;Engine Temp&lt;/code&gt; = $10^6$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above function is the objective function which you want to maximize, OR you can minimize the negative of the same. Both of them are equivalent. Thus your objective becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\min_{(x_1,x_2,x_3,x_4)} -(w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0)$&#xA;with constraints that $x_1,x_2,x_3,x_4 \geq 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here, you can use &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize&quot; rel=&quot;noreferrer&quot;&gt;Scipy's Minimization function&lt;/a&gt; with the Objective function we just came up with.&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-24T15:47:24.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24046" PostTypeId="2" ParentId="24041" CreationDate="2017-10-24T16:17:14.083" Score="3" Body="&lt;p&gt;My answer is that I would just normalize 0 = 00:00:00, 1 = 23:59:59+1sec by counting seconds and divide by the number of seconds in a day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I disagree with the sine/cosine transformation, it might treat 11:30 and 12:30 the same if you pick the wrong transformation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, there might be patterns like 'last minute' or 'early rise' that do not correspond to the cyclic nature of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be the responsibility of the machine learning algorithm to detect which transformation should be made, e.g. if your machine learning algorithm would be a neural network, it should/could have nodes with activation functions that may look like sine/cosine, amongst others. Those would detect and respond to such cyclic behaviors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normalized time data is also easy to pick up by tree based/svm algorithms. Maybe even easier than 'Hour' as category, e.g. if you have a shop that is open from 9:30 to 17:30, the hour category &quot;9&quot; would be ambiguous.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-24T16:26:36.190" LastActivityDate="2017-10-24T16:26:36.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24047" PostTypeId="1" CreationDate="2017-10-24T19:32:54.497" Score="4" ViewCount="71" Body="&lt;p&gt;I am using IBM Watson tool to determine tones (&lt;a href=&quot;https://tone-analyzer-demo.mybluemix.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://tone-analyzer-demo.mybluemix.net/&lt;/a&gt;) and personality scores (&lt;a href=&quot;https://personality-insights-livedemo.mybluemix.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://personality-insights-livedemo.mybluemix.net/&lt;/a&gt;) on different files containing natural language. What would be the recommended strategy and rationale for choosing between:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Getting scores from different files and averaging/manipulating them.&lt;/li&gt;&#xA;&lt;li&gt;Getting one raw score on aggregated data and using it.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40940" LastEditorUserId="13413" LastEditDate="2019-12-16T17:09:50.840" LastActivityDate="2024-03-11T22:00:55.133" Title="averaging multiple scores on small chunks of data or raw score on single collated data" Tags="|nlp|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24048" PostTypeId="2" ParentId="24047" CreationDate="2017-10-24T20:24:55.413" Score="0" Body="&lt;p&gt;I think this is 'mood' vs. 'character'. And as such, it depends on the question you ask. E.g. did I make my customer upset? Or is my customer loyal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though mood and character correlate, as someone with a certain character will be more likely to be in some mood, there is also variation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you will measure 'mood' with the 'tone' analyzer, and 'character' with the 'personality' analyzer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the 'personality' you will need all the data you can get to get the 'average', for the 'mood', a single data point can be sufficient.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-24T20:24:55.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24049" PostTypeId="2" ParentId="10301" CreationDate="2017-10-24T20:40:57.423" Score="0" Body="&lt;p&gt;This might not be of much help, but I wanted to point out that you might have to control for direction, given that the GPS co-ordinates will reduce and increase depending on the direction of travel. &lt;/p&gt;&#xA;" OwnerUserId="40942" LastActivityDate="2017-10-24T20:40:57.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24050" PostTypeId="2" ParentId="22" CreationDate="2017-10-25T02:17:35.637" Score="2" Body="&lt;p&gt;If we consider a scenario where the categorical variable cannot be hot encoded like the categorical variable has 200+ categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such cases you can use a package &#xA;&lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType/clustMixType.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;clustMixType&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can handle mixed data(numeric and categorical), you just need to feed in the data, it automatically segregates Categorical and Numeric data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find any issues like some numeric is under categorical then you can you as.factor()/ vice-versa as.numeric(), on that respective field and convert that to a factor and feed in that new data to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate lambda, so that you can feed-in as input at the time of clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we can even get a WSS(within sum of squares), plot(elbow chart) to find the optimal number of Clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this answer helps you in getting more meaningful results.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-10-25T02:17:35.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24051" PostTypeId="2" ParentId="15989" CreationDate="2017-10-25T02:33:02.213" Score="486" Body="&lt;p&gt;Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a &lt;em&gt;One-vs-All&lt;/em&gt; (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class A: 1 TP and 1 FP&lt;/li&gt;&#xA;&lt;li&gt;Class B: 10 TP and 90 FP&lt;/li&gt;&#xA;&lt;li&gt;Class C: 1 TP and 1 FP&lt;/li&gt;&#xA;&lt;li&gt;Class D: 1 TP and 1 FP&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$&lt;/li&gt;&#xA;&lt;li&gt;A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are quite different values for precision. Intuitively, in the macro-average the &quot;good&quot; precision (0.5) of classes A, C and D is contributing to maintain a &quot;decent&quot; overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$&#xA;$Pr_{macro-stdev}=0.173$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.&lt;/p&gt;&#xA;" OwnerUserId="40949" LastActivityDate="2017-10-25T02:33:02.213" CommentCount="15" ContentLicense="CC BY-SA 3.0" />
  <row Id="24052" PostTypeId="1" CreationDate="2017-10-25T02:43:49.793" Score="31" ViewCount="26746" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; is the GitHub link to the most recent &lt;code&gt;data.table&lt;/code&gt; benchmark.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;data.table&lt;/code&gt; benchmarks has not been updated since 2014. I heard somewhere that &lt;code&gt;Pandas&lt;/code&gt; is now faster than &lt;code&gt;data.table&lt;/code&gt;. Is this true? Has anyone done any benchmarks? I have never used Python before but would consider switching if &lt;code&gt;pandas&lt;/code&gt; can beat &lt;code&gt;data.table&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="40951" LastEditorUserId="29169" LastEditDate="2021-02-01T20:19:38.810" LastActivityDate="2021-03-14T18:16:24.620" Title="Is pandas now faster than data.table?" Tags="|python|r|pandas|data|data-table|" AnswerCount="4" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="24054" PostTypeId="1" CreationDate="2017-10-25T06:41:40.520" Score="-1" ViewCount="102" Body="&lt;p&gt;In a hotel booking scenario, when we are using a clustering model to cluster people's booking behaviors. We have two out of 15 features: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature1: booking_counts_yearly, which indicates how many times bookings has specific customer made. e.g. customerA booked 25 times while customerB booked 2 times in 2017. CustomerA's &lt;code&gt;booking_counts_yearly=25&lt;/code&gt; while customerB's &lt;code&gt;booking_counts_yearly=2&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature2: booking_5_stars_rate, which indicates the portion of 5 star hotel bookings has been made by specific customers. e.g. customerA booked 25 times in total, 5 out of 25 bookings are 5 star hotels, then customerA's &lt;code&gt;booking_5_stars_rate&lt;/code&gt; is &lt;code&gt;5/25=0.25&lt;/code&gt;. CustomerB booked 2 times in total, none of the bookings is 5 star hotel, then customerB's &lt;code&gt;booking_5_stars_rate&lt;/code&gt; is &lt;code&gt;0/2=0&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's discuss this case:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Customer1 with &lt;code&gt;booking_counts_yearly=1&lt;/code&gt;,&#xA;&lt;code&gt;booking_5_stars_rate=100%(1)&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;Customer2 with&#xA;&lt;code&gt;booking_counts_yearly=1&lt;/code&gt;, &lt;code&gt;booking_5_stars_rate=100%(1)&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;Customer3&#xA;with &lt;code&gt;booking_counts_yearly=1&lt;/code&gt;, &lt;code&gt;booking_5_stars_rate=100%(1)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Customer4 with &lt;code&gt;booking_counts_yearly=15&lt;/code&gt;,&#xA;&lt;code&gt;booking_5_stars_rate=100%(1)&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;Customer5 with&#xA;&lt;code&gt;booking_counts_yearly=100&lt;/code&gt;, &lt;code&gt;booking_5_stars_rate=70%(0.7)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my case, customer 4 and customer 5 should be one cluster, it looks like they are like type of person: &quot;made some bookings through the whole year and majority of times, they booked high-class hotels&quot; while customer 1, 2, 3 are more like: &quot;lived once a year, it happens to be 5 star&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I should do in feature engineering to make sure these two type of persons are well separated. Currently the cluster my clustering model gives me, I have a lot of customer 1, 2, 3 (considered as noises) mixed with customer 4 in one single cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way I can combine these two features so that my model would take the counts and rates both into consideration?&lt;/p&gt;&#xA;" OwnerUserId="38791" LastActivityDate="2017-10-29T22:47:54.137" Title="how to combine count and rate features in clustering" Tags="|clustering|feature-selection|feature-engineering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24057" PostTypeId="1" CreationDate="2017-10-25T08:04:48.440" Score="2" ViewCount="241" Body="&lt;p&gt;I am working on the problem of automatic punctuation: given a stream of words, decide for each word whether there should be a punctuation mark after it (in future work I also want to distinguish between different punctuation marks, but currently it's a binary classification problem). &#xA;My classifier uses a bidirectional LSTM whose output is fed into a multi-layer perceptron. The accuracy of the current classifier on the entire dataset is not good enough. In order to understand why, I want to create an &quot;easier&quot; dataset - a dataset on which even a simple (but not trivial) classifier would give good results. Then, I will make the dataset more and more difficult and see what improvements have to be made to the classifier. What is a good technique for creating synthetic datasets that are easier (but not trivial) to classify correctly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it is relevant, here are some details on my system:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each word is represented by an embedding vector of length 200.&lt;/li&gt;&#xA;&lt;li&gt;The embedding vector is fed into a bidirectional LSTM with an output length of 50.&lt;/li&gt;&#xA;&lt;li&gt;The LSTM output is fed into a multi-layer perceptron with two layers, the hidden layer has size 60.&lt;/li&gt;&#xA;&lt;li&gt;The dataset has about 2000 paragraphs, each of which contains about 200 words.&lt;/li&gt;&#xA;&lt;li&gt;Since the classes are imbalanced (only 20% of the words are punctuated), I check the system accuracy using &lt;a href=&quot;https://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cohen's kappa coefficient&lt;/a&gt; - a number between -1 and 1. A random classifier has  kappa around 0. A good classifier has kappa above 0.7. My current classifier has kappa around 0.4 (average of 10-fold cross-validation), that is, better than random but not very good.&lt;/li&gt;&#xA;&lt;li&gt;I train the classifier in 80 iterations. After each iteration I check the kappa of the model on a validation set. Finally I return the model with the highest kappa.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33051" LastEditorUserId="33051" LastEditDate="2017-10-25T08:24:36.103" LastActivityDate="2017-10-27T14:28:39.033" Title="Creating an easy but not trivial dataset" Tags="|deep-learning|nlp|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24059" PostTypeId="2" ParentId="24054" CreationDate="2017-10-25T08:13:41.153" Score="1" Body="&lt;p&gt;Maybe you can map the booking_counts_yearly to a number of percentile classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. low: less than 3 bookings, medium: 3-10 bookings, high: more than 10 bookings. Where 3 and 10 would be values that would split the population evenly (or by business purpose if necessary).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar you could partition the rating in percentiles, but that seems less relevant to me because of its nature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;--&#xA;Note, this could also depend on the machine learning/classification technique you are using. Decision trees may not need this at all, as they will determine the optimal partitioning by nature, while this normalization might be required for neural networks to function properly.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-26T12:39:42.977" LastActivityDate="2017-10-26T12:39:42.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24061" PostTypeId="1" CreationDate="2017-10-25T09:19:57.667" Score="0" ViewCount="39" Body="&lt;p&gt;For my classification task i have 3 features a,b,c.&#xA;Feature c is missing for some datas.&#xA;I can have already a good score for my classifier training with the two other features a,b and even better if I don't look at the missing data and I train with a,b,c.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to train my classifier with features a,b,c on all the data which don't have missing values, and in my prediction phase, if the feature c is missing, to predict the class like if I had train my classifier with a and b?&lt;/p&gt;&#xA;" OwnerUserId="40962" LastActivityDate="2017-11-24T12:42:58.590" Title="Can we look just at the other features when we have a missing vaue?" Tags="|missing-data|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24062" PostTypeId="2" ParentId="24061" CreationDate="2017-10-25T10:35:08.663" Score="1" Body="&lt;p&gt;Try a few options and pick the best.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fill in the mean/median from known $c$'s in the $a,b,c$ model&lt;/li&gt;&#xA;&lt;li&gt;Create a linear regression model to predict $c$ from $a,b$, fill the estimate $c$ from this model in the $a,b,c$ model&lt;/li&gt;&#xA;&lt;li&gt;Train both models and use the applicable one&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Personally, I think 2. will work best in most cases, but 1. can already be sufficient when $c$ is not correlated to $a$ or $b$.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-25T10:35:08.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24063" PostTypeId="1" AcceptedAnswerId="24074" CreationDate="2017-10-25T10:54:22.420" Score="1" ViewCount="7053" Body="&lt;p&gt;I'm currently reading this book and want someone to tell me if what currently I'm assuming about neural network is right or not.   &lt;a href=&quot;https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a layer has 30 neurons: In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer. That would be 30 neurons of the first layer multiplied by the 30 neurons of the second layer that would give a total of 900 (30*30 = 900) multiplications (is this is correct?) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But those 900 multiplications is only for the first layer and the second layer of the neural network(nn). So if I have a feedforward nn that has 3 layers I would have to do 900 more multiplications because they are the multiplications of the output of the first layer (that they are the input of the second layer) with the weight of the third layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to recap what I said so far I have a feedforward nn with 3 layer with 30 neurons in the first two layers multiply each neuron with all the neurons of the second layer it would be 30 * 30 = 900 multiplications for each additional layer that I add, it adds 900 additional multiplications. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then for a model with three (3) fully connected layer would be 900 multiplications in the first two layer because for what I said earlier and 900 multiplications in the second and third layer for a total of 1,800 (900 + 900)  multiplications excluding an activation function and this is only for a forward pass, is my understanding correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass. Am I correct assuming this for a Neural network?. That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.&lt;/p&gt;&#xA;" OwnerUserId="40965" LastEditorUserId="836" LastEditDate="2017-10-25T13:34:47.630" LastActivityDate="2017-10-25T13:55:22.283" Title="Amount of multiplications in a neural network model" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24064" PostTypeId="2" ParentId="23890" CreationDate="2017-10-25T11:28:08.137" Score="2" Body="&lt;p&gt;What you want to do is a sort of manual stacking. Start by training $M_1$ with $n_1$ samples. Then predict the margin for $n_2$ sample with $M_1$ (output_margin=True). Set the margin for $n_2$ samples and train $M_2$ using $n_2$ samples. &#xA;In round $k$, you need to predict with $M_i$, $i = 1,...,k-1$ using the margins from model $i-1$ and output the margin to model $i+1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- EDIT&#xA;Pseudocode in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_1 &amp;lt;- xgb.train(data = X1, watchlist = Xn1) &#xA;&#xA;X2 &amp;lt;- xgb.Dmatrix(X2)&#xA;base1 &amp;lt;- predict(model_1, X2, outputmargin=TRUE)&#xA;setinfo(X2, &quot;base_margin&quot;, base1)&#xA;&#xA;model_2 &amp;lt;- xgb.train(data = X2, watchlist = Xn2) &#xA;&#xA;X3 &amp;lt;- xgb.Dmatrix(X3)&#xA;base1 &amp;lt;- predict(model_1, X3, outputmargin=TRUE)&#xA;setinfo(X3, &quot;base_margin&quot;, base1)&#xA;base2 &amp;lt;- predict(model_2, X3, outputmargin=TRUE)&#xA;setinfo(X3, &quot;base_margin&quot;, base2)&#xA;&#xA;model_3 &amp;lt;- xgb.train(data = X3, watchlist = Xn3) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10216" LastEditorUserId="10216" LastEditDate="2017-10-27T16:20:37.027" LastActivityDate="2017-10-27T16:20:37.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24067" PostTypeId="1" AcceptedAnswerId="24089" CreationDate="2017-10-25T12:16:45.373" Score="1" ViewCount="426" Body="&lt;p&gt;I want perform demand forecast for particular item based on attributes.Did I need to train the model with unsold items ? &lt;strong&gt;by maintaining sales Quantity as zero&lt;/strong&gt; or  go with only items sold in training period.&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-10-26T11:21:38.103" Title="Input for Sales Forecasting" Tags="|machine-learning|neural-network|r|statistics|forecasting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24068" PostTypeId="1" CreationDate="2017-10-25T12:23:52.413" Score="2" ViewCount="674" Body="&lt;p&gt;How Box cox and other transformations convert data into Normal Distributions ?&lt;/p&gt;&#xA;" OwnerUserId="37604" LastActivityDate="2017-10-26T16:56:53.503" Title="How Box cox and other transformations convert data into Normal Distributions?" Tags="|machine-learning|python|r|statistics|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24069" PostTypeId="1" CreationDate="2017-10-25T12:25:16.680" Score="1" ViewCount="3156" Body="&lt;p&gt;Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? &#xA;I am interested in clustering documents using unsupervised learning and segregating it into appropriate cluster. &#xA;Any link, code example, paper will greatly be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="39932" LastActivityDate="2023-12-01T08:00:25.060" Title="How to map topic to a document after topic modeling is done with LDA" Tags="|topic-model|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24072" PostTypeId="1" AcceptedAnswerId="24073" CreationDate="2017-10-25T13:25:02.370" Score="3" ViewCount="4640" Body="&lt;p&gt;I have a financial dataset, where I'm trying to predict company types, based on the amount dollars, what time of day, and whether they buy or sell (currency pairs). It looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The features I use to predict:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X.head():&#xA;&#xA;Dollars   | Hours      | Buy | Sell&#xA;-0.761916   0.364838     1     0&#xA;-0.924413   0.377558     1     0&#xA;-0.573336   0.397836     0     1&#xA;-0.561639   0.399144     0     1&#xA;-1.164036   0.423715     1     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The features I want to predict could look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y.head()&#xA;Bank  Tech  Fund  Holding  Defence  Financial Services  Pharma &#xA;1     0     0     0        0        0                   0   &#xA;1     0     0     0        0        0                   0   &#xA;1     0     0     0        0        0                   0   &#xA;1     0     0     0        0        0                   0   &#xA;1     0     0     0        0        0                   0   &#xA;&#xA;Agriculture  Commodities  Energi  Pension  &#xA;0            0            0       0  &#xA;0            0            0       0  &#xA;0            0            0       0  &#xA;0            0            0       0  &#xA;0            0            0       0  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this snippet, the first five companies are banks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a training/test ratio of 0.25, I get an accuracy of 0.99, which seems too good to be true:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)&#xA;&#xA;rand_forest = RandomForestClassifier(max_depth = None, random_state = 0)&#xA;&#xA;rand_forest.fit(X,y)&#xA;&#xA;predictions = rand_forest.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result of the &lt;code&gt;classification_report&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         precision    recall  f1-score   support&#xA;&#xA;      0       0.98      0.95      0.97      5074&#xA;      1       0.98      0.91      0.94      2292&#xA;      2       0.98      0.82      0.89       572&#xA;      3       0.99      0.83      0.90       235&#xA;      4       0.98      0.91      0.94       261&#xA;      5       0.99      0.81      0.89       411&#xA;      6       0.98      0.83      0.90       239&#xA;      7       1.00      0.70      0.82       144&#xA;      8       1.00      0.81      0.89       384&#xA;      9       0.99      0.81      0.89       200&#xA;     10       1.00      0.81      0.90       232&#xA;&#xA;avg / total   0.98      0.90      0.94     10044&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Adjusting the &lt;code&gt;max_depth&lt;/code&gt; parameter of the classifier changes this number significantly though, but I'm still reading up on what the consequences of that parameter actually are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth mentioning that there is only 50,000 entries in this dataset, across 11 different companytypes, which might be too little?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a simpler &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; yiels an accuracy of about 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used the entire dataset for training, not the actual training set. Switching these two outs gives an accuracy of 54%, which sounds much better (or more realistic anyways).&lt;/p&gt;&#xA;" OwnerUserId="39014" LastEditorUserId="39014" LastEditDate="2017-10-25T13:49:08.907" LastActivityDate="2017-10-25T13:49:08.907" Title="Random Forest Classifier gives very high accuracy on test set - overfitting?" Tags="|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24073" PostTypeId="2" ParentId="24072" CreationDate="2017-10-25T13:31:05.873" Score="4" Body="&lt;p&gt;&lt;code&gt;rand_forest.fit(X,y)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are you using the whole data set for training? You are using the test set for training then evaluate the performance on it again?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your code, I didn't see you actually used the training set you created.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2017-10-25T13:31:05.873" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24074" PostTypeId="2" ParentId="24063" CreationDate="2017-10-25T13:44:47.930" Score="2" Body="&lt;p&gt;Essentially you are correct, there are a lot of calculations necessary to process inputs and train neural networks.&lt;/p&gt;&#xA;&lt;p&gt;You have some terminology a bit wrong or vague. E.g.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The neurons do not multiply together directly. A common way to write the equation for a neural network layer, calling input layer values &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt; and first hidden layer values &lt;span class=&quot;math-container&quot;&gt;$a_j$&lt;/span&gt;, where there are N inputs might be&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$a_j = f( b_j + \sum_{i=1}^{N} W_{ij}x_{i})$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$f()$&lt;/span&gt; is the activation function &lt;span class=&quot;math-container&quot;&gt;$b_j$&lt;/span&gt; is the bias term, &lt;span class=&quot;math-container&quot;&gt;$W_{ij}$&lt;/span&gt; is the weight connecting &lt;span class=&quot;math-container&quot;&gt;$a_j$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;So if you have &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; neurons in the hidden layer, you have &lt;span class=&quot;math-container&quot;&gt;$N\times M$&lt;/span&gt; multiplications and &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; separate sums/additions over &lt;span class=&quot;math-container&quot;&gt;$N+1$&lt;/span&gt; terms, and &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; applications of the transfer function &lt;span class=&quot;math-container&quot;&gt;$f()$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;It doesn't work quite so directly, and there as a small factor of more calculations involved (you do not calculate each derivative with a single multiplication, often there are a few, some results are re-used, and other operations may be involved). However yes you do need to calculate a derivative for each weight and bias term, and there are roughly that number of weights in your network that require the calculations done.&lt;/p&gt;&#xA;&lt;p&gt;Your suggested numbers are actually quite small compared to typical neural networks used for image problems. These typically perform millions of computations for a forward pass.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Actually that is a trivial number of calculations for a modern CPU, and would be done in less than a millisecond. But multiply this out by a few factors:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You must do this for &lt;em&gt;each and every example in the training data&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your example network is small, think bigger&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This does not include the activation function calculations - typically slower than a multiply&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your rough estimate ignores some of the necessary operations, so as a guesstimate, multiply number of CPU-level operations by 3 or 4 from your analysis.&lt;/p&gt;&#xA;&lt;p&gt;. . . and the number of operations does start to get to values where CPUs can take hours or days to perform training tasks in practice.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-25T13:55:22.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24076" PostTypeId="1" CreationDate="2017-10-25T15:10:02.370" Score="0" ViewCount="93" Body="&lt;p&gt;For example for restaurants reviews usually have suggestions like &quot;Go in the evenings&quot;, &quot;order the so and so sauce with this dish&quot;  or even &quot;TIP: ask for the blah blah blah&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I detect such sentences? How do people usually tackle similar challenges? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do they create classification rules like &lt;code&gt;&amp;lt;modal_verb&amp;gt;&amp;lt;preference_verb&amp;gt;&amp;lt;optional_window_size_of_3&amp;gt;&amp;lt;positive_sentiment_words&amp;gt;&lt;/code&gt;&#xA;Some examples of these rules are “would be great” and “could be really good”  --&#xA; found this from &lt;a href=&quot;https://www.experfy.com/blog/suggestion-and-opinion-mining-from-qualitative-surveys&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I would have to use a tagger to categorize words?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any blog that has attempted something similar step by step?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would appreciated.&lt;/p&gt;&#xA;" OwnerUserId="40983" LastActivityDate="2017-10-25T17:00:50.127" Title="How to go about text mining for suggestions/Tips in reviews for restaurants?" Tags="|text-mining|nltk|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-25T20:09:58.850" ContentLicense="CC BY-SA 3.0" />
  <row Id="24077" PostTypeId="2" ParentId="24076" CreationDate="2017-10-25T17:00:50.127" Score="1" Body="&lt;p&gt;So the way I thought about it is - abstractly you are attempting to identify which sentences are, Imperative Mood (IMP) and which are not. One option is to put together a training set/validation set, where each row is a phrase's pos tags (can use some encoder to convert to integers) with a target value of 'isImperative' (binary) then train a model and restructure each review so that they resemble the training set format and have your model assign the values. Seems the easiest and most sustainable method from where I am sitting. Sorry but I don't know any easily accessible phrase data sets but I bet there is one out there somewhere. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="29801" LastActivityDate="2017-10-25T17:00:50.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24078" PostTypeId="2" ParentId="13564" CreationDate="2017-10-25T17:50:32.400" Score="3" Body="&lt;h3&gt;Suggestion&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You could bin the 'Health' vector to make a categorical variable (ideally consisting of similar number of observations, for example : 2-bin would be median-split of 'high' and 'low', 3-bin would be tertiles of 'high', 'medium' and 'low', and so on), and then do box plots of 'Weight' per bin. You might see that the bin of 'low' is different from the bin of 'high'. &#xA;The number of bins you choose depends on the distribution of the 'Health' variable and you can play around with that.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Implementation&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr) # for modifying datasets &#xA;library(ggplot2) # for plotting &#xA;library(magrittr) # for piping&#xA;stackodato &amp;lt;- data.frame(&quot;Health&quot; = sample(0:10, 10), &quot;Weight&quot; = sample(0:200, 10)) # creating a pseudo dataset&#xA;&#xA;stackodato %&amp;gt;% &#xA;mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c(&quot;low&quot;, &quot;high&quot;))) %&amp;gt;% # add &quot;binnedHealth&quot; column which has the &quot;Health&quot; variable categorized into two factors : &quot;high&quot; and &quot;low&quot;&#xA;ggplot()+geom_boxplot(aes(x=binnedHealth, y=Weight)) # boxplot showing the distribution of &quot;Weight&quot; split by the &quot;binnedHealth&quot; factor&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LfXA1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LfXA1.png&quot; alt=&quot;Boxplot of median-split Health - x-axis is binned Health&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try this : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stackodato %&amp;gt;% mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c(&quot;low&quot;, &quot;high&quot;))) %&amp;gt;% ggplot()+geom_boxplot(aes(x=Health, y=Weight, group = binnedHealth))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Hp6MR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Hp6MR.png&quot; alt=&quot;Boxplot of median-split Health - x-axis is Health&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40767" LastEditorUserId="40767" LastEditDate="2017-10-26T20:40:57.233" LastActivityDate="2017-10-26T20:40:57.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24079" PostTypeId="1" CreationDate="2017-10-25T19:06:45.947" Score="1" ViewCount="228" Body="&lt;p&gt;I am currently working on an lbsn (localization-based social network) system and i need to predict the user's age and gender.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every time a user enters a venue, the system creates a &quot;check-in&quot; with the user, the venue and the datetime.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every venue is categorized using Foursquare Venue Categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The system generate a Weigthed Concept Hierarchy to represent the interest level between a user and a Venue Category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to predict the user's age and gender using the mentioned data?&lt;/p&gt;&#xA;" OwnerUserId="33120" LastEditorUserId="33120" LastEditDate="2017-10-26T20:44:57.270" LastActivityDate="2024-01-01T15:04:50.703" Title="Predict User Demographics from location based social networks" Tags="|machine-learning|classification|clustering|supervised-learning|" AnswerCount="1" CommentCount="4" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24080" PostTypeId="1" CreationDate="2017-10-25T19:22:39.730" Score="-3" ViewCount="762" Body="&lt;p&gt;I have estimated normal distributions for two classes (0 and 1). The distributions for the two classes are,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;X_0 &amp;amp;\sim \mathcal{N}(\mu=549.96,~~ \sigma=549.96) \\&#xA;X_1 &amp;amp;\sim \mathcal{N}(\mu=250.047, \sigma=8.33)&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I am given a sample, say $ x = 381.82 $, and I need to tell if it is more probable to belong to class 0 or it is more probable to belong to class 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I do? Calculate normal &lt;code&gt;cdf&lt;/code&gt;? Or do I need to calculate density at the given point? I am using &lt;code&gt;matlab&lt;/code&gt; for implementation but it does not matter because we are not allowed to use any builtin functions.&lt;br&gt;&#xA;Regards&#xA;Adeel &lt;/p&gt;&#xA;" OwnerUserId="40045" LastEditorDisplayName="user32280" LastEditDate="2017-10-27T02:35:12.903" LastActivityDate="2017-10-27T04:13:25.357" Title="Which Normal distribution a point belongs to?" Tags="|machine-learning|prediction|gaussian|" AnswerCount="1" CommentCount="6" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24081" PostTypeId="1" AcceptedAnswerId="24083" CreationDate="2017-10-25T22:54:50.883" Score="14" ViewCount="15981" Body="&lt;p&gt;I recently came across graph embedding such as DeepWalk and LINE. However, I still do not have a clear idea as what is meant by graph embeddings and when to use it (applications)? Any suggestions are welcome!&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="8878" LastEditDate="2017-10-26T11:58:37.140" LastActivityDate="2023-08-25T11:05:49.010" Title="What are graph embedding?" Tags="|graphs|" AnswerCount="4" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24082" PostTypeId="1" AcceptedAnswerId="24105" CreationDate="2017-10-25T23:04:17.877" Score="1" ViewCount="74" Body="&lt;p&gt;I have some data with some margin of error and am using numpy arrays to plot that data. Assuming the precision is, let's say, 0.05 or 0.03grams, then the data will be something like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4.0 $\pm$ 0.05, 3.05 $\pm$ 0.03]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I show this on my plot (either on the x axis or the y axis. If it's both, even better)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use pyplot and my OS is macOS&lt;/p&gt;&#xA;" OwnerUserId="41004" LastActivityDate="2017-10-26T16:38:48.873" Title="Matplotlib: plot indicating the precision of measurements" Tags="|error-handling|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24083" PostTypeId="2" ParentId="24081" CreationDate="2017-10-26T02:32:57.310" Score="22" Body="&lt;p&gt;Graph embedding learns a mapping from a network to a vector space, while preserving relevant network properties.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vector spaces are more amenable to data science than graphs. Graphs contain edges and nodes, those network relationships can only use a specific subset of mathematics, statistics, and machine learning. Vector spaces have a richer toolset from those domains. Additionally, vector operations are often simpler and faster than the equivalent graph operations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example is finding nearest neighbors. You can perform &quot;hops&quot; from node to another node in a graph. In many real-world graphs after a couple of hops, there is little meaningful information (e.g., recommendations from friends of friends of friends). However, in vector spaces, you can use distance metrics to get quantitative results (e.g., Euclidian distance or Cosine Similarity). If you have quantitative distance metrics in a meaningful vector space, finding nearest neighbors is straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;a href=&quot;https://arxiv.org/abs/1705.02801&quot; rel=&quot;noreferrer&quot;&gt;Graph Embedding Techniques, Applications, and Performance: A Survey&lt;/a&gt;&quot; is an overview article that goes into greater detail.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2019-02-24T16:48:47.367" LastActivityDate="2019-02-24T16:48:47.367" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24084" PostTypeId="1" CreationDate="2017-10-26T02:33:52.120" Score="-3" ViewCount="40" Body="&lt;p&gt;We are trying to implement a highly accurate search, based on user-entered search terms, into a large product database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the user searches for &quot;iPhone&quot;, then one of these is potentially what they want, but the other is not:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Apple iphone 5s Grey 32GB 4G smartphone Unlocked new 1 yr wty&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Apple iphone 5s case new 1 yr wty&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be a failure if the search returned the latter before, or instead of, the former.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The single word &quot;case&quot; in the second product title, made it 'not an iPhone', but all the other words were not relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we have no existing data of which is a 'correct' result or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which seems to mean we have a language-related problem - Which words in a product title/description are important, and which are not? - Relative to a user's search terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not knowing much about data science, I am looking for some steerage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks like a common search problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone point me to an algorithm, or better yet, an existing service of some kind, that could help us with achieving an accurate search result in this scenario?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, if this isn't sufficiently specific, but not not knowing what I don't know, I am not sure I know how to be more specific.&lt;/p&gt;&#xA;" OwnerUserId="41008" LastEditorUserId="41008" LastEditDate="2017-11-08T01:26:37.630" LastActivityDate="2017-11-08T01:26:37.630" Title="Algorithms/services to know an &quot;iPhone case&quot; is not an &quot;iPhone&quot;, in the context of complex item descriptions?" Tags="|text-mining|text|search|" AnswerCount="1" CommentCount="0" ClosedDate="2017-10-26T11:15:17.430" ContentLicense="CC BY-SA 3.0" />
  <row Id="24085" PostTypeId="2" ParentId="24084" CreationDate="2017-10-26T02:50:00.010" Score="1" Body="&lt;p&gt;The problem you are describing is called &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Query_Rewriting&quot; rel=&quot;nofollow noreferrer&quot;&gt;query rewriting&lt;/a&gt;&quot;, taking a user's string literal and processing it to find the best items in the search index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can always start with rule-based logic to parse queries, either hard-coded or regular expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then move to statistical learning methods to improve query rewriting. A user's query should be tokenized based on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Collocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;collocations&lt;/a&gt;, frequently occurring n-grams. Tokenizing on collocations would group together meaningful words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also session-based context. Create an overall session state (e.g., &quot;phones&quot; or &quot;fruit&quot;). Each token would contribute changes in session, some tokens would contribute a little and others would contribute a lot. For example, the presence of &quot;case&quot; would automatically void the &quot;phone&quot; state. The session state would then change term weightings. Session state works well because users often make several queries in a role, called query refinement.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-10-26T02:50:00.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24086" PostTypeId="2" ParentId="24067" CreationDate="2017-10-26T06:48:26.510" Score="1" Body="&lt;p&gt;If you want to estimate &lt;em&gt;demand&lt;/em&gt;, no, you do not need to train the model with unsold items for which you have supply, since they do not affect it. Having an excess supply/capacity does not affect your customer behaviour. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have supply, but no demand, it needs to be there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, also take into account that if you have had a demand greater than the supply at any moment, you have to do proper demand unconstraining to have a good forecast of your demand. &lt;/p&gt;&#xA;" OwnerUserId="37376" LastEditorUserId="37376" LastEditDate="2017-10-26T11:21:38.103" LastActivityDate="2017-10-26T11:21:38.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24088" PostTypeId="1" AcceptedAnswerId="24090" CreationDate="2017-10-26T07:37:13.673" Score="-1" ViewCount="41" Body="&lt;p&gt;How would you assign Data Science &lt;em&gt;primary&lt;/em&gt; tasks to Data Scientists in a team according to a colleague' seniority? I mean, as always you can expect different things and also level and pace of self-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example just some picks which come to my mind:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Junior/Beginner: mining technology and data cleansing?&lt;/li&gt;&#xA;&lt;li&gt;Intermediate: training models, data visualization?&lt;/li&gt;&#xA;&lt;li&gt;Senior: plan and design work for other team (not just from managerial point of view), decide on strategy and estimate project risks (with a custom data science solution, hehe)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note. I do not think this is a very opinion based question: it would be however if my premise is wrong in the way if we would argument &quot;there are junior geniuses&quot; but I try to focus rather to some mainstream rule of thumb than exceptions.&lt;/p&gt;&#xA;" OwnerUserId="33183" LastActivityDate="2017-10-26T10:14:44.567" Title="Is it possible to plan and assign Data Science tasks by complexity levels based on team members' experience?" Tags="|data|model-selection|methodology|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24089" PostTypeId="2" ParentId="24067" CreationDate="2017-10-26T08:35:48.537" Score="1" Body="&lt;p&gt;In short, if there is supply, it should be in the model to determine demand:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have XXXL pink jeans in your store and don't sell them, I think it needs to be in the model.&lt;/li&gt;&#xA;&lt;li&gt;If you don't have the pink jeans in your store for sale, it doesn't.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It gets interesting when you'd sell some XS pink jeans and blue XXXL ones, but you'd at least have to offer them in the store to be able to sell them and get measurement data to train the model.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-26T08:35:48.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24090" PostTypeId="2" ParentId="24088" CreationDate="2017-10-26T10:14:44.567" Score="1" Body="&lt;p&gt;I would divide tasks by complexity, smaller tasks for the juniors, bigger tasks for intermediates, etc. But the complete task from mining to interpreting the results of a trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would juniors otherwise be able to 'grow' from data cleansing to model training?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data cleansing and model training are probably too closely related to divide between team members. Where would you normalize data, and how do you assure that both team members use the same assumptions? And the Junior has to learn how to clean data properly for models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do think that Data Science projects, more than other projects in Engineering, have uncertain outcomes. Setting milestones, S.M.A.R.T. targets, monitoring, supervision, reviews, interpretation should be done on a daily basis by Seniors.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-26T10:14:44.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24091" PostTypeId="2" ParentId="10188" CreationDate="2017-10-26T10:54:14.260" Score="3" Body="&lt;p&gt;In your formulation, you try to obtain the mean deviation of your approximation from the observed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the mean value of your approximation is close or equal to the mean value of the observed data (something which is desirable and often happens with many approximation schemes) then the result of your formulation would be zero or negligible, because positive errors compensate with negative errors. This might lead to the conclusion that your approximation is wonderful at each observed sample, while it might not be the case. That's why you use the square of the error at each sample and you add them up (your turn each error positive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this is only a possible solution, as you could have used L1-norm (absolute value of the error at each sample) or many others, instead of L2-norm.&lt;/p&gt;&#xA;" OwnerUserId="41024" LastEditorUserId="29575" LastEditDate="2018-12-13T14:05:49.900" LastActivityDate="2018-12-13T14:05:49.900" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24093" PostTypeId="1" AcceptedAnswerId="24096" CreationDate="2017-10-26T12:36:27.727" Score="77" ViewCount="193057" Body="&lt;p&gt;I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="29169" LastEditDate="2019-06-20T00:33:49.773" LastActivityDate="2024-03-07T08:58:45.593" Title="How to clone Python working environment on another machine?" Tags="|python|anaconda|" AnswerCount="10" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24094" PostTypeId="2" ParentId="24093" CreationDate="2017-10-26T12:46:53.397" Score="6" Body="&lt;p&gt;Look into 'containers', e.g. Docker (&lt;a href=&quot;https://www.docker.com/what-container&quot; rel=&quot;noreferrer&quot;&gt;https://www.docker.com/what-container&lt;/a&gt;), a more lightweight alternative to virtualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will require some time investment but in the end will provide many benefits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the link, where I marked your specific need in &lt;strong&gt;&lt;em&gt;bold italic&lt;/em&gt;&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Package software into standardized units for development, shipment and deployment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Available for both Linux and Windows based apps, &lt;strong&gt;&lt;em&gt;containerized software will always run the same, regardless of the environment&lt;/em&gt;&lt;/strong&gt;. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-26T12:46:53.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24095" PostTypeId="2" ParentId="24093" CreationDate="2017-10-26T13:05:24.463" Score="8" Body="&lt;p&gt;If your program is mostly Python, you could rely solely on virtual environments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the working virtualenv, create a file with the version of each installed Python library :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip freeze &amp;gt; requirements.txt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the new virtualenv, ask &lt;code&gt;pip&lt;/code&gt; to install those libraries with the same version :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install -r requirements.txt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This makes sure you get the same lib versions on both machines. And since requirements.txt is tracked by your VCS, you can always recreate the environment of an old version of your code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you need a database, a production web server, etc. you end up with a few more steps and you can't rely on virtualenv to ensure both environments match. This is where Docker steps in (See &lt;a href=&quot;https://datascience.stackexchange.com/a/24094/41029&quot;&gt;Pieter21's answer&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="41029" LastEditorUserId="41029" LastEditDate="2018-06-29T19:49:32.967" LastActivityDate="2018-06-29T19:49:32.967" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24096" PostTypeId="2" ParentId="24093" CreationDate="2017-10-26T14:36:50.193" Score="92" Body="&lt;p&gt;First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As for the question itself - you can export your Anaconda environment using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda env export &amp;gt; environment.yml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And recreate it using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda env create -f environment.yml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create a virtual environment in Anaconda you can use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda create -n yourenvname python=x.x anaconda&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which you activate using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;source activate yourenvname&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28328" LastActivityDate="2017-10-26T14:36:50.193" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24097" PostTypeId="2" ParentId="24080" CreationDate="2017-10-26T14:37:27.813" Score="2" Body="&lt;p&gt;A common distance measure between a point and a normal distribution is the number of standard deviations away from the mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d_0 = |x-μ_0|/σ_0 = 15.4$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$d_1 = |x-μ_1|/σ_1 = 15.82$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://en.wikipedia.org/wiki/Mahalanobis_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mahalanobis distance&lt;/a&gt; for generalization in multiple dimensions.&lt;/p&gt;&#xA;" OwnerUserId="41031" LastEditorUserId="29575" LastEditDate="2017-10-26T22:12:47.440" LastActivityDate="2017-10-26T22:12:47.440" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24098" PostTypeId="2" ParentId="11129" CreationDate="2017-10-26T14:54:46.400" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do some feature engineering. As they told you in the comments, try&#xA;to see if the event is more likely to happen during weekends or&#xA;weekdays, during certain months, during a certain season or quarter,&#xA;etc. To read some more on this topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explore different machine learning models. I suggest to start with a Random Forest. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Check your model accuracy (confusion matrix). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Iterate. Try other features, other models...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Just to get you up and running, you can take a look at the Titanic dataset in Kaggle. It will help you understand a bit more of the problem you are trying to solve. &lt;/p&gt;&#xA;" OwnerUserId="37376" LastActivityDate="2017-10-26T14:54:46.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24099" PostTypeId="1" CreationDate="2017-10-26T15:08:47.297" Score="3" ViewCount="62" Body="&lt;p&gt;I'm trying to understand t-SNE better and I was hoping someone could elaborate on how the $\sigma _i$'s are chosen. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was also wondering why they aren't just calculated in the normal way standard deviation is calculated from the data with $x_{i} $ as the expected value for each $\sigma _i$?&lt;/p&gt;&#xA;" OwnerUserId="41033" LastEditorUserId="29575" LastEditDate="2017-10-28T18:03:42.183" LastActivityDate="2017-10-28T18:03:42.183" Title="Bandwidth of the gaussian kernels in t-sne" Tags="|dimensionality-reduction|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24100" PostTypeId="2" ParentId="1213" CreationDate="2017-10-26T15:12:58.433" Score="0" Body="&lt;p&gt;Depending on how much data you have, a good approach should be some form of a neural network based encoder-decoder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The are plenty of results for &quot;neural network image denoising&quot; on google but a common approach will be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Start with your input image&lt;/li&gt;&#xA;&lt;li&gt;Pass it through smaller and smaller convolutional layers&lt;/li&gt;&#xA;&lt;li&gt;Upscale it back through bigger and bigger deconvolutional layers (usually this part is a mirroring of the first, convolutional, phase)&lt;/li&gt;&#xA;&lt;li&gt;Compare the output to your expected results&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28328" LastActivityDate="2017-10-26T15:12:58.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24102" PostTypeId="1" AcceptedAnswerId="24240" CreationDate="2017-10-26T15:46:13.777" Score="0" ViewCount="62" Body="&lt;p&gt;Okay so, I understand that inputs are sent directly to the network (basically being multiplied to weights of the nodes, receive a bias, and gets activated) and then the network produces an output through this feed forward method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, in order for the network to &quot;learn,&quot; we determine the error by comparing the predicted output versus the &quot;known output / answer,&quot; and then back-propagate. Right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't get it is if the output goes through some activation function, say Sigmoid (results to a value between 0.0 - 1.0), how can it ever learn if the known output is a continuous value such as &quot;Salary&quot;, &quot;Number of Something&quot;, or in the use-case I have, &quot;Volume of Orders&quot; which ranges from anywhere between 300 to 1500.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;// Example I have 7 inputs&#xA;// They go through a hidden layer that consists 16 hidden neurons&#xA;// and my output is a single neuron that's supposed to be &quot;Volume of Orders&quot;&#xA;//&#xA;// MyInputLayers[] --&amp;gt; HiddenLayers[] --&amp;gt; Output=0.123&#xA;// &#xA;// Since I learn from tutorials that Output layers go through an activation,&#xA;// I think it's impossible for the network to learn because a&#xA;// prediction of 0.123 compared to the &quot;known output&quot; that is, say 615&#xA;// is almost ridiculous&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should I apply Sigmoid to the &quot;known output&quot; as well? If so, how can I remap it when the network has already learned and is ready to do some predictive magic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I must be missing / not understanding something here. Thank you in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s. My dataset is composed of 6 months of records; an Excel file that has 200,000+ rows and 8 columns (7 leading, 1 lagging).&lt;/p&gt;&#xA;" OwnerUserId="41034" LastEditorUserId="41034" LastEditDate="2017-10-27T03:14:40.903" LastActivityDate="2017-11-01T06:14:16.417" Title="Regression: Should &quot;known outputs&quot; be also activated?" Tags="|machine-learning|neural-network|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24105" PostTypeId="2" ParentId="24082" CreationDate="2017-10-26T16:38:48.873" Score="1" Body="&lt;p&gt;One can use the &lt;code&gt;matplotlib.axes.Axes.errorbar&lt;/code&gt; class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://matplotlib.org/examples/statistics/errorbar_demo_features.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt; in the documentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# example error bar values that vary with x-position&#xA;error = 0.1 + 0.2 * x&#xA;&#xA;fig, (ax0, ax1) = plt.subplots(nrows=2, sharex=True)&#xA;ax0.errorbar(x, y, yerr=error, fmt='-o')&#xA;ax0.set_title('variable, symmetric error')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Acknowledgments: Thanks to &lt;a href=&quot;https://datascience.stackexchange.com/users/381/emre&quot;&gt;Emre&lt;/a&gt;!&lt;/p&gt;&#xA;" OwnerUserId="41004" LastActivityDate="2017-10-26T16:38:48.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24106" PostTypeId="2" ParentId="24068" CreationDate="2017-10-26T16:56:53.503" Score="2" Body="&lt;p&gt;Box Cox and other transformations find the exponent needed to transform the data into a normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transformations happen by finding the estimate of $\lambda$ and some values in the neighborhood are chosen to transform the original data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the data is transformed, we assume the data is normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original formula proposed by Box &amp;amp; Cox is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;y(\lambda) = \left\{&#xA;        \begin{array}{ll}&#xA;           \frac{y^{\lambda} -1 }\lambda \quad if \lambda \neq 0; \\&#xA;            log(y), \quad  if \lambda = 0.&#xA;        \end{array}&#xA;    \right.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which basically turns into choosing $\lambda$ as a model parameter. This value is found by maximizing log-likelihood.&lt;/p&gt;&#xA;" OwnerUserId="15250" LastActivityDate="2017-10-26T16:56:53.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24108" PostTypeId="1" AcceptedAnswerId="24177" CreationDate="2017-10-26T17:22:52.393" Score="3" ViewCount="13513" Body="&lt;p&gt;I am interested in time-series forecasting with &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomForest&lt;/a&gt;. The basic approach is to use a rolling window and use the data points within the window as features for the RandomForest regression, where we regress the next values after the window on the values within the window. Just plain autoregressive model (with lags), but with Random Forest instead of linear regression.&lt;/p&gt;&#xA;&lt;h3&gt;Problem:&lt;/h3&gt;&#xA;&lt;p&gt;If I have more than one time-series (multiple time-series), how to pass them in RF regression?&lt;/p&gt;&#xA;&lt;p&gt;For example: given two time series &lt;span class=&quot;math-container&quot;&gt;$y_1(t)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y_2(t)$&lt;/span&gt;, the outcome time series is &lt;span class=&quot;math-container&quot;&gt;$z(t)$&lt;/span&gt; and I am interested in predicting the values of &lt;span class=&quot;math-container&quot;&gt;$z(t)$&lt;/span&gt; based on the combination of &lt;span class=&quot;math-container&quot;&gt;$y_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y_2$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;What I need, is to use rolling window for each &lt;span class=&quot;math-container&quot;&gt;$y_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y_2$&lt;/span&gt;, and then feed these values within the window from both time-series into RF regression, to predict the value of &lt;span class=&quot;math-container&quot;&gt;$z(t)$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&lt;p&gt;How do I incorporate the data from both rolling windows into the input for RF regression?&lt;/p&gt;&#xA;" OwnerUserId="14070" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-29T17:15:43.140" Title="Multiple time-series predictions with Random Forests (in Python)" Tags="|time-series|random-forest|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24109" PostTypeId="1" CreationDate="2017-10-26T17:56:35.197" Score="5" ViewCount="1549" Body="&lt;p&gt;I am referring Ehthem Alpaydin, &lt;a href=&quot;https://mitpress.mit.edu/books/introduction-machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;'Introduction To Machine Learning'&lt;/a&gt; book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the chapter 'Decision Trees', I need help with understanding the concept of discriminant and how it is being used in this paragraph:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Each $f_m(x)$ defines a discriminant in the d-dimensional input space dividing it into smaller regions that are further subdivided as we take a path from the root down. $f_m(.)$ is a simple function and when written down as a tree, a complex function is broken down into a series of simple decisions. Different decision tree methods assume different models for $f_m(.)$, and the model class defines the shape of the discriminant and the shape of regions. Each leaf node has an output label, which in the case of classification is the class code and in regression is a numeric value.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="40801" LastEditorUserId="29575" LastEditDate="2017-10-28T19:19:28.917" LastActivityDate="2017-11-02T18:08:53.177" Title="Understanding decision tree concept" Tags="|machine-learning|decision-trees|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24112" PostTypeId="2" ParentId="23159" CreationDate="2017-10-26T20:49:19.147" Score="79" Body="&lt;p&gt;It is more than just numerical. A quick reminder of the softmax:&#xA;$$&#xA;P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the exp you mentioned. A few important points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The loss function is not directly related to softmax. You can use standard normalization and still use cross-entropy.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A &quot;hardmax&quot; function (i.e. argmax) is not differentiable. The softmax gives at least a minimal amount of probability to all elements in the output vector, and so is nicely differentiable, hence the term &quot;soft&quot; in softmax.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now I get to your question. The $e$ in softmax is the natural exponential function. &lt;strong&gt;Before&lt;/strong&gt; we normalize, we transform $x$ as in the graph of $e^x$:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Rs2tj.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rs2tj.png&quot; alt=&quot;natural exponential function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $x$ is 0 then $y=1$, if $x$ is 1, then $y=2.7$, and if $x$ is 2, now $y=7$! A huge step! This is what's called a non-linear transformation of our unnormalized log scores. The interesting property of the exponential function combined with the normalization in the softmax is that high scores in $x$ become much more probable than low scores. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An example&lt;/strong&gt;. Say $K=4$, and your log score $x$ is vector $[2, 4, 2, 1]$. The simple argmax function outputs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;[0, 1, 0, 0]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The argmax is the goal, but it's not differentiable and we can't train our model with it :( A simple normalization, which is differentiable, outputs  the following probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;[0.2222, 0.4444, 0.2222, 0.1111]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's really far from the argmax! :( Whereas the softmax outputs:&#xA;$$&#xA;[0.1025, 0.7573, 0.1025, 0.0377]&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's much closer to the argmax! Because we use the natural exponential, we hugely increase the probability of the biggest score and decrease the probability of the lower scores when compared with standard normalization. Hence the &quot;max&quot; in softmax.&lt;/p&gt;&#xA;" OwnerUserId="41047" LastEditorUserId="41047" LastEditDate="2017-10-27T13:02:52.863" LastActivityDate="2017-10-27T13:02:52.863" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="24113" PostTypeId="2" ParentId="18903" CreationDate="2017-10-26T21:52:37.300" Score="14" Body="&lt;p&gt;XGBoost now has a histogram binning option for tree growth similar to the one LightGBM uses. It provides about the same level of speedup and similar accuracy characteristics, although the algorithms are still not exactly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some plots and tables here showing how they are right on top of each other now.&#xA;&lt;a href=&quot;https://github.com/dmlc/xgboost/issues/1950&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/dmlc/xgboost/issues/1950&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be fair, LightGBM cites their own performance tests showing them still edging out XGBoost (hist), though not by an order of magnitude any more.&#xA;&lt;a href=&quot;https://github.com/Microsoft/LightGBM/blob/master/docs/Experiments.rst#comparison-experiment&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/Microsoft/LightGBM/blob/master/docs/Experiments.rst#comparison-experiment&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41051" LastEditorUserId="41051" LastEditDate="2017-10-27T04:08:16.797" LastActivityDate="2017-10-27T04:08:16.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24115" PostTypeId="2" ParentId="24081" CreationDate="2017-10-27T01:57:28.363" Score="28" Body="&lt;p&gt;&lt;strong&gt;What are graph Embeddings ?&lt;/strong&gt;&#xA;&quot;Graph Embeddings&quot; is a hot area today in machine learning. It basically means finding &quot;latent vector representation&quot; of graphs which captures the topology (in very basic sense) of the graph. We can make this &quot;vector representation&quot; rich by also considering the vertex-vertex relationships, edge-information etc. There are roughly two levels of embeddings in the graph (of-course we can anytime define more levels by logically dividing the whole graph into subgraphs of various sizes):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Vertex Embeddings&lt;/strong&gt; - Here you find latent vector representation of every vertex in the given graph. You can then compare the different vertices by plotting these vectors in the space and interestingly &quot;similar&quot; vertices are plotted closer to each other than the ones which are dissimilar or less related. This is the same work that is done in &quot;DeepWalk&quot; by Perozzi.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Graph Embeddings&lt;/strong&gt; - Here you find the latent vector representation of the whole graph itself. For example, you have a group of chemical compounds for which you want to check which compounds are similar to each other, how many type of compounds are there in the group (clusters) etc. You can use these vectors and plot them in space and find all the above information. This is the work that is done in &quot;Deep Graph Kernels&quot; by Yanardag.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Applications -&lt;/strong&gt;&#xA;By looking carefully, embeddings are &quot;latent&quot; representations  which means if a graph has a |V| * |V| adjacency matrix where |V| = 1M, its hard to use or process a 1M * 1M numbers in an algorithm. So, latent embedding of dimension 'd', where d &amp;lt;&amp;lt; |V|, would make the adjacency matrix |V| * d and relatively easier to use. Another application could be - Consider a simple scenario where we want to recommend products to the people who have similar interests in a social network. By getting vertex embeddings (here it means vector representation of each person), we can find the similar ones by plotting these vectors and this makes recommendation easy. These are some applications and there are others. You can refer to a nice survey paper - &lt;a href=&quot;https://arxiv.org/pdf/1705.02801.pdf&quot; rel=&quot;noreferrer&quot;&gt;Graph Embedding Techniques, a Survey&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Where from it all came ?&lt;/strong&gt; There has been a lot of works in this area and almost all comes from the groundbreaking research in natural language processing field - &quot;Word2Vec&quot; by Mikolov. If you want to get started with the research on graph embeddings, I would recommend to first understand how Word2Vec works. You can find nice explanations - &lt;a href=&quot;https://arxiv.org/pdf/1411.2738.pdf&quot; rel=&quot;noreferrer&quot;&gt;Word2Vec parameter learning explained&lt;/a&gt; and &lt;a href=&quot;https://www.youtube.com/watch?v=ERibwqs9p38&quot; rel=&quot;noreferrer&quot;&gt;Stanford Lecture&lt;/a&gt;. Then you can jump to the papers that you listed. Those works can be categorized as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Works based on &quot;Vertex Embeddings&quot;: - &lt;a href=&quot;http://www.perozzi.net/publications/14_kdd_deepwalk.pdf&quot; rel=&quot;noreferrer&quot;&gt;DeepWalk&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/pdf/1607.00653.pdf&quot; rel=&quot;noreferrer&quot;&gt;Node2Vec&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/pdf/1503.03578.pdf&quot; rel=&quot;noreferrer&quot;&gt;LINE&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Works based on &quot;Graph Embeddings&quot;: - &lt;a href=&quot;https://users.soe.ucsc.edu/~vishy/pubs/YanVis15.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deep Graph Kernels&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/pdf/1606.08928.pdf&quot; rel=&quot;noreferrer&quot;&gt;Subgraph2Vec&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2017-10-27T01:57:28.363" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24120" PostTypeId="1" AcceptedAnswerId="24136" CreationDate="2017-10-27T06:12:27.310" Score="2" ViewCount="685" Body="&lt;p&gt;There is a function call TreeBagger that can implement random forest. However, if we use this function, we have no control on each individual tree. Can we use the MATLAB function fitctree, which build a decision tree, to implement random forest? Thanks a lot.&lt;/p&gt;&#xA;" OwnerUserId="20618" LastEditorUserId="20618" LastEditDate="2017-10-27T14:27:12.050" LastActivityDate="2017-10-27T15:03:09.697" Title="Can we implement random forest using fitctree in matlab?" Tags="|machine-learning|deep-learning|random-forest|decision-trees|matlab|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24122" PostTypeId="1" CreationDate="2017-10-27T08:15:45.747" Score="2" ViewCount="1533" Body="&lt;p&gt;We are trying a DBSCAN clustering model on our 30,000 samples with 15 features each. We tuned the &lt;a href=&quot;https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;epsilon parameter&lt;/a&gt; small enough to make sure the radius of the clustering circle is small while it does clustering. We expected to have small clusters with high density. After clustering and parameters tuning, we used t-SNE to plot the clustering results in 2 dimensional space, we found that we have small clusters like cluster 2,3,4,5 with high density as expected while large clusters like cluster 0,1 scattered loosely as unexpected. obviously, cluster 0, 1 looks very strange to us. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there something we are not aware of, is it something wrong with clustering algorithm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or it's something about t-SNE, when it plots the clustering results in lower dimension, it might look like cluster 0, 1 are very scattered, actually they are more close and tensed in higher dimensional space?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;cluster 0: orange&lt;/li&gt;&#xA;&lt;li&gt;cluster 1: blue&lt;/li&gt;&#xA;&lt;li&gt;cluster 2: red&lt;/li&gt;&#xA;&lt;li&gt;cluster 3: green&lt;/li&gt;&#xA;&lt;li&gt;cluster 4: light yellow&lt;/li&gt;&#xA;&lt;li&gt;cluster 5: purple&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;30,000 samples, 6 clusters case:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fhf4y.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fhf4y.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;30,000 samples, 8 clusters case:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U4ZzN.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U4ZzN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38791" LastEditorUserId="924" LastEditDate="2017-10-27T21:05:48.183" LastActivityDate="2017-10-27T21:05:48.183" Title="t-SNE plotting DBSCAN clustering results very scattered issue" Tags="|machine-learning|clustering|tsne|plotting|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24127" PostTypeId="2" ParentId="24109" CreationDate="2017-10-27T11:22:48.383" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ODrl9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ODrl9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above image the big square is your $f_m(x)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The big square is made up of small partitions. Those partitions are your smaller functions(Partitions) which performs the comparison between your variables and try to find out best possible split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here each partition has nearly same labels for classification and nearest numerical values for regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The square representation is for 2 dimensions if you want to get for higher dimensions refer the last one with 3 dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: Elements of Statistical Learning, Basics of Decision trees&lt;/p&gt;&#xA;" OwnerUserId="37626" LastEditorUserId="29575" LastEditDate="2017-10-28T18:03:18.967" LastActivityDate="2017-10-28T18:03:18.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24131" PostTypeId="1" CreationDate="2017-10-27T14:07:11.290" Score="1" ViewCount="62" Body="&lt;p&gt;I've been dealing with a dataset full of categorical variables and had some issues to apply dimensionality reduction techniques as PCA is no great fit for such datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some research, I came across Multi Correspondence Aanalysis for the first time and read several documents about it, mainly this one (&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.470.7630&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.470.7630&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt;) and they all talked about CA (Correspondence Analysis) since MCA is basically CA but when you have more than only two variables so I've been trying to read about that too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem for me is when they say &quot;MCA is simply CA applied to the indicator matrix&quot;; I really can't understnd that statement. I understand it when they say that MCA is CA applied to the &lt;strong&gt;Burt Matrix&lt;/strong&gt; and it makes sens for me, but when it's stated that MCA = CA + indicator matrix (see pages 3-4 in the document above for example), I can't see it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone shed some light on that please ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks !&lt;/p&gt;&#xA;" OwnerUserId="33242" LastActivityDate="2017-10-27T14:07:11.290" Title="Link between Correspondance Analysis and MCA" Tags="|visualization|categorical-data|dimensionality-reduction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24132" PostTypeId="2" ParentId="24057" CreationDate="2017-10-27T14:28:39.033" Score="1" Body="&lt;p&gt;You might use a Markov chain for generating a simple enough dataset. You can estimate the probabilities of punctuation by scanning a large corpus of text (e.g. wikipedia), then you can use the model to generare fictious sentences with punctuation marks. You can set the order of the Markov chain so as to make the generated dataset increasingly closer to real-world text. The Markov chain should be defined on the set of words, however, so you have to carefully choose the vocabulary. &lt;/p&gt;&#xA;" OwnerUserId="28713" LastActivityDate="2017-10-27T14:28:39.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24136" PostTypeId="2" ParentId="24120" CreationDate="2017-10-27T15:03:09.697" Score="4" Body="&lt;p&gt;I would highly recommend doing some research into the architecture of random forests.  There are many sites that provide in depth tutorials on RFs (&lt;a href=&quot;https://machinelearningmastery.com/implement-random-forest-scratch-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Implementation in Python&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Quick  explanation: take your dataset, bootstrap the samples and apply a decision tree. Within your trees, you  want to randomly sample the features at each split. You should not have to build your own RF using fitctree however.  You don't want to control each individual tree in the forest. This introduces bias, and the point of the RF is that by bagging many trees, you remove the risk of overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define your hyperparameters and let the algorithm do it's thing.  Carefully cross-validate to ensure you are not under-fitting.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2017-10-27T15:03:09.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24137" PostTypeId="1" AcceptedAnswerId="24145" CreationDate="2017-10-27T16:36:27.360" Score="1" ViewCount="6613" Body="&lt;p&gt;I'm using the &lt;strong&gt;NSL-KDD&lt;/strong&gt; data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the &lt;code&gt;get_dummies&lt;/code&gt; method in python and the &lt;code&gt;NominalToBinary&lt;/code&gt; method in WEKA, but the problem is that some nominal features contain 64 values so the conversion increases the dimensionality of the data a lot, and this can create problems for the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is if I can convert the nominal attributes by establishing a correspondence between each category of a nominal feature and a sequence of integer values, for example &lt;code&gt;protocol_type {tcp=0, udp=1, icmp=2...etc}&lt;/code&gt;? Would this alter the credibility of the resulted data set?&lt;/p&gt;&#xA;" OwnerUserId="33349" LastEditorUserId="26562" LastEditDate="2017-10-28T10:37:11.943" LastActivityDate="2017-10-29T22:14:28.077" Title="Converting a nominal attributes to numerical ones in data set" Tags="|machine-learning|dataset|weka|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24138" PostTypeId="1" CreationDate="2017-10-27T16:58:55.980" Score="3" ViewCount="257" Body="&lt;p&gt;I am new to decision tree method. For decision tree regression model, does it just fit a piece wise step function over data? When and why would people prefer it over some traditional regression like locally weighted (LOESS) regression?  &lt;/p&gt;&#xA;" OwnerUserId="41092" LastEditorUserId="41092" LastEditDate="2017-10-27T18:02:20.390" LastActivityDate="2024-02-04T03:02:40.287" Title="Is decision tree regression comparable to locally weighted regression" Tags="|regression|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24145" PostTypeId="2" ParentId="24137" CreationDate="2017-10-27T23:14:16.100" Score="3" Body="&lt;p&gt;By converting a nominal attribute to a single numeric attribute as you described, you are implicitly introducing an ordering over the nominal labels which is a bad representation of the data, and can lead to unwanted effects from a classifier. Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!) Imagine you are training a $k$-NN model on this data. It doesn't make sense to say that ICMP should be &quot;further away&quot; from TCP than UDP, but if you adopted the mapping that you suggested, the representation of the data has this assumption built-in. Alternatively, what if you are training a decision tree-based model? Usually, in decision trees, binary split points are chosen for numeric attributes. There could be some randomness in your training data where splits at certain values of the numeric attribute results in overfitting to noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically when converting a nominal attribute to numeric, one numeric attribute per nominal label is created. Each attribute is set to one if the corresponding nominal label is set, and zero otherwise. For example, if a nominal attribute called &lt;code&gt;protocol&lt;/code&gt; has labels {&lt;code&gt;tcp&lt;/code&gt;, &lt;code&gt;udp&lt;/code&gt;, &lt;code&gt;icmp&lt;/code&gt;}, then this dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{array}{ccl}&#xA;\text{inst.} &amp;amp; \text{protocol} &amp;amp; \text{other attributes} \\ \hline&#xA;1 &amp;amp; \text{tcp} &amp;amp; \dots \\&#xA;2 &amp;amp; \text{icmp}&amp;amp; \dots \\&#xA;3 &amp;amp; \text{icmp}&amp;amp; \dots  \\&#xA;\vdots &amp;amp; \vdots &amp;amp; \ddots&#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;could be converted as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{array}{ccccl}&#xA;\text{inst.} &amp;amp; \text{tcp} &amp;amp; \text{udp} &amp;amp; \text{icmp} &amp;amp; \text{other attributes} \\ \hline&#xA;1 &amp;amp; 1 &amp;amp; 0 &amp;amp; 0 &amp;amp; \dots \\&#xA;2 &amp;amp; 0 &amp;amp; 0 &amp;amp; 1 &amp;amp; \dots \\&#xA;3 &amp;amp; 0 &amp;amp; 0 &amp;amp; 1 &amp;amp; \dots \\&#xA;\vdots &amp;amp; \vdots &amp;amp; \vdots &amp;amp; \vdots &amp;amp; \ddots \\&#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what the &lt;code&gt;NominalToBinary&lt;/code&gt; filter does in WEKA. As you mention, the downside of this is that a large number of additional attributes can be introduced if the number of distinct nominal values is high. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the dimensionality is too high after the conversion, you may want to consider using a dimensionality reduction technique such as random projection, PCA, t-SNE, etc. Note that this will reduce the interpretability of your model. You could also use feature selection techniques to remove some of the less useful attributes. It is possible that some of the nominal labels are not useful for your model, and you will improve performance by removing them. Another thing you could try is to use your domain knowledge to reduce the number of categories. For example, TCP and UDP are both transport protocols, maybe for your application the distinction between TCP and UDP is not that important and you can put instances with protocol $\in$ {&lt;code&gt;tcp&lt;/code&gt;, &lt;code&gt;udp&lt;/code&gt;} into a new category, removing the old ones.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2017-10-29T22:14:28.077" LastActivityDate="2017-10-29T22:14:28.077" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24148" PostTypeId="2" ParentId="14957" CreationDate="2017-10-28T01:57:25.183" Score="0" Body="&lt;p&gt;Apache Mahout is an open-source library that provides a good recommendation engine. I've used it on two projects successfully.&lt;/p&gt;&#xA;" OwnerUserId="41109" LastActivityDate="2017-10-28T01:57:25.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24151" PostTypeId="1" AcceptedAnswerId="24152" CreationDate="2017-10-28T13:45:50.603" Score="-1" ViewCount="623" Body="&lt;p&gt;What is the generic way to preprocess data for machine learning and predictive models ?What are the sequence of steps to be taken?&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-10-28T13:45:50.603" Title="How to preprocess data?" Tags="|machine-learning|data-mining|predictive-modeling|data-cleaning|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24152" PostTypeId="2" ParentId="24151" CreationDate="2017-10-28T13:45:50.603" Score="2" Body="&lt;p&gt;You are having a dataset with both continous and categorical data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.&lt;strong&gt;Centre the data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for numerical variable&#xA;centering usually done by subtracting mean of the column and some times by minimum value of the column&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.&lt;strong&gt;Scaling&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scaling the data means converting range of data between 0 and 1.&#xA;It is done by different methods some follows by dividing with range and some follows by dividing with variance(unit var)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3.&lt;strong&gt;Skewness&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check for the skewness of the variable in given data.If the skewness factor is not around zero then try to perform data tranformations using exponential transformations , Box cox and logarithmic e.t.c&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4.&lt;strong&gt;One on n encoding&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its is also called as one hot encoding also it is used to encode categorical variables that to nominal variables&#xA;or else if you can use Equilateral encoding etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for ordinal variable you can encode them as increasing or decreasing order&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5.&lt;strong&gt;Feature Selection and Importance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to remove unwanted variables in your data then you use any feature selection algorithm like recursive feature selection or feature importance by trees etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6.&lt;strong&gt;Dimensionality Reduction&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To reduce the number of dimensions in your data go with algorithms like Principal Component Analysis (unsupervised) or Partial Least Squares(Supervised)&#xA;and select the number of dimensions which can nearly describe variance of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7.&lt;strong&gt;Removal of outliers&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Outliers are the portion of your data you are not explored in some situations&#xA;to remove outliers you can go with techniques like Spatial Sign and some other  techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;8.&lt;strong&gt;Missing Values&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Missing values are the most common problem in data science.In order to over come to that you can impute values using different approaches like knn impute and build some model to predict missing data using other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;9.&lt;strong&gt;Binning Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is like converting continous data into categorical or interval data this sounds interesting but often leads to loss of valuable information&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are some of the important and basic steps of data preprocessing for majority of the algorithms but some algorithms doesn't need some of the steps like&#xA;Random forest accepting factor values (so no need of one on n encoding)&#xA;XgBoost accepting missing Values etc.&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-10-28T13:45:50.603" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24153" PostTypeId="2" ParentId="24137" CreationDate="2017-10-28T14:55:24.700" Score="2" Body="&lt;p&gt;For encoding of the categorical variables with high cardinality (i.e. with large number of levels) you may want to try the so called &lt;em&gt;impact coding&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main idea is very simple, you just split the dataset into non-overlapping buckets by the variable of interest (“protocol” in your case) and calculate average of your response variable over each bucket. Then, the values of the categorical variable can be substituted by the average value over particular bucket. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Avg(response | protocol=”tcp”)&#xA;&#xA;Avg(response | protocol=”icmp”)&#xA;&#xA;Avg(response | protocol=”udp”)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The tricky part is to avoid data leakage, this can be done by splitting the entire dataset into several subsets (e.g. “encoding”, “training”, “validation”,…) and using only the data from “encoding” dataset for the nominal-to-numerical conversion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’ve learned about this approach from &lt;a href=&quot;http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Win-Vector&lt;/a&gt; blog and their paper: &lt;a href=&quot;https://arxiv.org/pdf/1611.09477.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;vtreat: a &lt;code&gt;data.frame&lt;/code&gt; Processor for Predictive Modeling&lt;/em&gt;&lt;/a&gt;, which I highly recommend.&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29575" LastEditDate="2017-10-28T15:11:27.580" LastActivityDate="2017-10-28T15:11:27.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24157" PostTypeId="1" AcceptedAnswerId="24163" CreationDate="2017-10-28T19:57:26.710" Score="1" ViewCount="165" Body="&lt;p&gt;I'm not really happy with the mind maps I've been able to find on Google, most of them are algorithm based.  I want to make a good one that is problem/solution domain based.  Do I have this right for my top level nodes?  Here is the general direction I am headed: &lt;a href=&quot;https://i.stack.imgur.com/fNuwv.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://i.stack.imgur.com/fNuwv.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions/doubts about what I have so far are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is my starting point below generally correct?  e.g. no high level subclass is missing, and everything presented as a subclass deserves to be here?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;is Hybrid learning always just a combination of supervised and unsupervised?  Or, are there real examples of other hybrid models (e.g. 'reinforcement' and 'supervised', etc.).  I know &lt;em&gt;theoretically&lt;/em&gt; we can combine any methods...I'm looking for what's real/applied/demonstrable today.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;does Reinforcement learning belong at this high level, or is it actually a subset of one of the others (or one I've omitted)?  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Machine Learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.1 Supervised (uses labelled data to train and validate)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.2 Unsupervised (uses unlabeled data, or ignores labels if they are present)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.3 Semi-supervised (uses partially labelled (mostly unlabeled) data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.4 Hybrid (combines a supervised method and an unsupervised method)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.5 Reinforcement Learning (uses data from the environment)  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="40559" LastEditorUserId="40559" LastEditDate="2017-10-28T20:50:40.863" LastActivityDate="2017-10-28T23:21:40.973" Title="what are the top level subsets/domains of ML?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24158" PostTypeId="2" ParentId="17099" CreationDate="2017-10-28T19:58:46.113" Score="1" Body="&lt;p&gt;There is a function in keras LSTM &lt;code&gt;reset_states(states)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the parameter states is the concatination of two states, hidden state h and cell state.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;States = [h, c]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it would be interesting to know if you should initialize &lt;code&gt;h&lt;/code&gt; or &lt;code&gt;c&lt;/code&gt; according to the approaches in the above mentioned papers.&lt;/p&gt;&#xA;" OwnerUserId="41096" LastEditorUserId="29575" LastEditDate="2017-10-29T00:29:00.383" LastActivityDate="2017-10-29T00:29:00.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24160" PostTypeId="1" CreationDate="2017-10-28T21:27:35.643" Score="3" ViewCount="202" Body="&lt;p&gt;I would like to analyze speech samples looking for speech-language pathologies.&#xA;Most of the resources I can find are about speech recognition which is a completely different problem. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I will always know what is the text that is being spoken on the given sample.&lt;/li&gt;&#xA;&lt;li&gt;I can assume that audio samples will always be of good quality with no background noises.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would like to construct a neural net(or maybe some different model) that will detect certain anomalies, but I have few questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How should I pass the input to the net? The input will have two parts: speech and text being spoken (+ label). Should I make I somehow divide these parts or just concatenate them as one sample?&lt;/li&gt;&#xA;&lt;li&gt;What can I read to better understand working with audio/speech in ML environment?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41131" LastActivityDate="2017-10-28T23:37:30.777" Title="How to approach speech analysis?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24161" PostTypeId="1" CreationDate="2017-10-28T21:46:02.473" Score="1" ViewCount="802" Body="&lt;p&gt;Can someone help me understand how to find the values of a confusion matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that essentially a confusion matrix looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;True Positive |False Positive&#xA;False Negative|True Negative&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've encountered a problem where I have around 1000 cases in my test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the approximate middle of its ROC chart there is a point where the false positive rate is 0.5, the true positive rate is 0.7, and the accuracy is 0.7.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my approach to this problem and I just want to verify if I am doing it correctly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate the count of True Positives: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$0.7 * 1000\,Cases = 700\,Cases$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To calculate the count of False Positives:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$0.5 * 1000\,Cases = 500\,Cases$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would assume calculating the True Negatives would be $1000 - 700 = 300$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then solving for False Negatives = &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$.7 (Accuracy) = 700 + 300 / 700 + 300 + 500 + FN$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone confirm if this is the right approach?&lt;/p&gt;&#xA;" OwnerUserId="41132" LastEditorUserId="29575" LastEditDate="2017-10-29T15:42:21.293" LastActivityDate="2018-12-19T15:35:01.940" Title="Calculating a Confusion Matrix" Tags="|confusion-matrix|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24162" PostTypeId="2" ParentId="24160" CreationDate="2017-10-28T22:35:13.773" Score="2" Body="&lt;p&gt;I think you are too quick to throw AI at this problem. (As you already noted, keep alternatives to neural networks open: maybe factor analysis or principal component analysis are more suitable, or support vector machines)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, you have to do more work in the problem domain, to find out that different pathologies may result in different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation would be to start small with one pathology. Use domain knowledge to determine if (and to what extent) the pathology is present in some samples, and how that could be observed from the audio samples, in a limited number of parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some pathologies, speech recognition would help. You could count differences in the text spoken and the text recognized. The count (and maybe a pattern in those counts) could be an input for the model (not necessary a neural network as you already noted).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some other pathologies would need audio spectral or temporal observations in the audio samples (unnatural pitch, unnatural pauses). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give an example: stuttering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would look in the audio samples and spectra for differences in the patterns. E.g. a spectral plot of stuttering vs. regular speech could look like: &lt;a href=&quot;https://i.stack.imgur.com/MYkpa.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MYkpa.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So typically you'd want to detect/measure relative power in some frequency regions as a measure for stuttering and learn your model with such data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An alternative that might also work is to run a stuttering sample (and a natural sample) through speech recognition, to see if differences can be observed in a countable/measurable way.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-28T23:37:30.777" LastActivityDate="2017-10-28T23:37:30.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24163" PostTypeId="2" ParentId="24157" CreationDate="2017-10-28T23:21:40.973" Score="0" Body="&lt;p&gt;Google (in images!) on 'machine learning cheat sheet', to find an example like this:&#xA;&lt;a href=&quot;https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-cheat-sheet&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-cheat-sheet&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you are well on your way in identifying all kinds of techniques, but you may want to turn your plot 'sideways', e.g. two-way / multi-class classification are problem (sub)domains and result in different recommended machine learning techniques. &lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-28T23:21:40.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24167" PostTypeId="1" AcceptedAnswerId="24204" CreationDate="2017-10-29T08:57:47.957" Score="1" ViewCount="1176" Body="&lt;p&gt;For example I have the following data structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user: Chris&#xA;age: 32&#xA;income: 60.000&#xA;basket value: 45&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want predict the basket value, and my features are the age and income.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a linear regression I get a regression function as the result of the fitting for example: $$y = 0.5x + 0.785$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I can use the function for prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the form of the result of the fitting by regression decision tree? Is it also a function?&lt;/p&gt;&#xA;" OwnerUserId="20903" LastEditorUserId="29575" LastEditDate="2017-10-30T02:37:20.117" LastActivityDate="2017-10-30T21:05:27.790" Title="Forecasting: How Decision Tree work?" Tags="|decision-trees|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24168" PostTypeId="2" ParentId="24167" CreationDate="2017-10-29T09:31:04.380" Score="0" Body="&lt;p&gt;Decision trees solve different types questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. will someone with Age A, Gender G, Income I buy product X or Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no fitting involved, but the model might internally calculate/come up with an Age boundary or Income boundary that would switch the decision from X to Y.&lt;/p&gt;&#xA;" OwnerUserId="33812" LastActivityDate="2017-10-29T09:31:04.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24169" PostTypeId="1" AcceptedAnswerId="24182" CreationDate="2017-10-29T10:38:14.657" Score="1" ViewCount="1457" Body="&lt;p&gt;I am building a Neural Network for multiclass classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset has 3 millions of observations. My features are 7 unordered categorical values. My problem is sparse as 4 features among the 7 can take more than 1500 different values. The label is a categorical variable that takes 2100 different values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model outputs a vector of 2100 probabilities through the softmax function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Tensorflow with Python and I wanted to use one-hot encoding to encode each of the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do this method suffer from the sparsity of my data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, do the One-Hot encoding transform my inputs into a vector of more than 2000*4 = 8000 features? In that case, if I use a mini-batch gradient descent with a batchsize of 50, is the fact that I have a lot more features than samples (8000&gt;&gt;&gt;50) a problem (&quot;curse of dimensionality&quot;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you have a solution, and a way to implement it on Tensorflow with Python?&lt;/p&gt;&#xA;" OwnerUserId="41128" LastEditorUserId="29575" LastEditDate="2017-10-29T15:42:17.467" LastActivityDate="2017-10-29T18:33:43.773" Title="How to treat sparse categorical features in a Neural Network for multiclass classification with Tensorflow?" Tags="|neural-network|tensorflow|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24170" PostTypeId="2" ParentId="24161" CreationDate="2017-10-29T11:56:39.077" Score="1" Body="&lt;p&gt;Your approach is not correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some hints that should help you to understand your mistakes &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the sum over &lt;strong&gt;all&lt;/strong&gt; elements of the confusion matrix should be 1000 in your case. &lt;/li&gt;&#xA;&lt;li&gt;TPR=TP / Num_positives&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.google.de/amp/www.dataschool.io/simple-guide-to-confusion-matrix-terminology/amp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Good guide to confusion matrix terminology&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-10-29T11:56:39.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24174" PostTypeId="1" CreationDate="2017-10-29T16:11:00.523" Score="4" ViewCount="3462" Body="&lt;p&gt;Given this test data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd &#xA;import numpy as np &#xA;&#xA;data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994', '2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592', '2014-05-03 18:47:05.332662', '2014-05-03 18:47:05.385109', '2014-05-04 18:47:05.436523', '2014-05-04 18:47:05.486877'], &#xA;        'battle_deaths': [34, 25, 26, 15, 15, 14, 26, 25, 62, 41], 'prisioners': [3, 4, 3, 2, 2, 6, 4, 5, 2, 8]}&#xA;df = pd.DataFrame(data, columns = ['date', 'battle_deaths', 'prisioners'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Set index datetime:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.set_index(pd.DatetimeIndex(df['date']), drop=True)&#xA;del df['date']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It works when I want to resample to the milliseconds, but it takes too long...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timeit df.resample('1L').sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess is because is aggregating all the milliseconds with NaN data, but when I drop it ..&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timeit df.resample('1L').sum().dropna()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It takes even longer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Y9nnB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y9nnB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;guessing again that the dropna is done at the end... There is any way which by dropping the NaN samples will accelerate the process?&lt;/p&gt;&#xA;" OwnerUserId="30914" LastEditorUserId="23305" LastEditDate="2017-10-30T15:13:41.313" LastActivityDate="2018-02-18T21:56:58.313" Title="Pandas dataframe resample aggregation by mills too slow" Tags="|python|pandas|dataframe|aggregation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24175" PostTypeId="2" ParentId="24174" CreationDate="2017-10-29T16:46:52.093" Score="3" Body="&lt;p&gt;It appears you don't really want to use resampling.  You are immediately throwing away the resampled data.  I think what you actually need is to simply &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;groupby&lt;/a&gt; records in the same millisecond.  That can be accomplished with:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Truncate to milliseconds and group by&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['milliseconds'] = df['date'].str[:-3]&#xA;grouped_and_summed = df.groupby(df.milliseconds).sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994',&#xA;                 '2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071',&#xA;                 '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592',&#xA;                 '2014-05-03 18:47:05.332662', '2014-05-03 18:47:05.385109',&#xA;                 '2014-05-04 18:47:05.436523',&#xA;                 '2014-05-04 18:47:05.486877'],&#xA;        'battle_deaths': [34, 25, 26, 15, 15, 14, 26, 25, 62, 41],&#xA;        'prisioners': [3, 4, 3, 2, 2, 6, 4, 5, 2, 8]}&#xA;df = pd.DataFrame(data, columns=['date', 'battle_deaths', 'prisioners'])&#xA;&#xA;df['milliseconds'] = df['date'].str[:-3]&#xA;print(df.groupby(df.milliseconds).sum())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                         battle_deaths  prisioners&#xA;milliseconds                                      &#xA;2014-05-01 18:47:05.069             34           3&#xA;2014-05-01 18:47:05.119             25           4&#xA;2014-05-02 18:47:05.178             26           3&#xA;2014-05-02 18:47:05.230             30           4&#xA;2014-05-02 18:47:05.280             14           6&#xA;2014-05-03 18:47:05.332             26           4&#xA;2014-05-03 18:47:05.385             25           5&#xA;2014-05-04 18:47:05.436             62           2&#xA;2014-05-04 18:47:05.486             41           8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-11-20T18:52:51.750" LastActivityDate="2017-11-20T18:52:51.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24176" PostTypeId="1" AcceptedAnswerId="24178" CreationDate="2017-10-29T17:12:40.017" Score="-1" ViewCount="2930" Body="&lt;p&gt;I am doing a regression and I want to use the regularizer that will be the most useful to get a sparse set of parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which regularizer should I use ? Cardinality? maximum value ? Sum of absolute values ? Euclidean norm ?&lt;/p&gt;&#xA;" OwnerUserId="40388" LastActivityDate="2017-10-31T07:56:40.447" Title="Which regularizer to use to get a sparse set of regression parameters?" Tags="|regression|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24177" PostTypeId="2" ParentId="24108" CreationDate="2017-10-29T17:15:43.140" Score="12" Body="&lt;p&gt;Random forest (as well as most of supervised learning models) accepts a vector $x=(x_1,...x_k)$ for each observation and tries to correctly predict output $y$. So you need to convert your training data to this format. The following &lt;code&gt;pandas&lt;/code&gt;-based function will help:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;def table2lags(table, max_lag, min_lag=0, separator='_'):&#xA;    &quot;&quot;&quot; Given a dataframe, return a dataframe with different lags of all its columns &quot;&quot;&quot;&#xA;    values=[]&#xA;    for i in range(min_lag, max_lag + 1):&#xA;        values.append(table.shift(i).copy())&#xA;        values[-1].columns = [c + separator + str(i) for c in table.columns]&#xA;    return pd.concat(values, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example, the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame({'y1':[1,2,3,4,5], 'y2':[10,20,40,50,30], 'z': [1,4,9,16,25]})&#xA;x = table2lags(df[['y1', 'y2']], 2)&#xA;print(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will produce output&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   y1_0  y1_1  y1_2  y2_0  y2_1  y2_2&#xA;0   1.0   NaN   NaN  10.0   NaN   NaN&#xA;1   2.0   1.0   NaN  20.0  10.0   NaN&#xA;2   3.0   2.0   1.0  40.0  20.0  10.0&#xA;3   4.0   3.0   2.0  50.0  40.0  20.0&#xA;4   5.0   4.0   3.0  30.0  50.0  40.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first two rows have missing values, because lags 1 and 2 are undefined on them. You can fill them with what you find appropriate, or simply omit them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have matrix of $x$ values, you can feed it, for example, to a &lt;code&gt;scikit-learn&lt;/code&gt; regressor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import RandomForestRegressor&#xA;rf = RandomForestRegressor().fit(x[2:], df['z'][2:])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, a piece of advice. Your model could much improve if you used not only raw lagged values as features, but also their different aggregations: mean, other linear combinations (e.g. ewm), quantiles, etc. Including additional linear combinations into a linear model is useless, but for tree-based models it can be of much help.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-10-29T17:15:43.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24178" PostTypeId="2" ParentId="24176" CreationDate="2017-10-29T17:27:15.513" Score="4" Body="&lt;p&gt;The most common sparse regularizer is &lt;strong&gt;sum of absolute values&lt;/strong&gt; (so-called Lasso regression). With carefully chosen penalty coefficient, it makes some of less useful parameters exactly zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cardinality&lt;/strong&gt; penalty exactly imposes sparsity, but it cannot be combined with gradient descent, and usually requires combinatorial optimization. Simply put, it is slow to apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Maximum value&lt;/strong&gt; and &lt;strong&gt;Euclidean norm&lt;/strong&gt; do not affect sparsity at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find more details in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Regularization_(mathematics)#Regularizers_for_sparsity&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipeda&lt;/a&gt; article.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-10-31T07:56:40.447" LastActivityDate="2017-10-31T07:56:40.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24179" PostTypeId="1" CreationDate="2017-10-29T17:32:44.413" Score="0" ViewCount="224" Body="&lt;p&gt;As a stepping stone towards a project involving classification of data points having multidimensional and imaginary features, I decided to try to solve first a simple problem: given 2 random points with coordinates (x1,y1) and (x2,y2), can a program classify the data according to the sign of the slope these two points have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Obviously, there is a link between x1 and y1 (and x2 and y2)&lt;/strong&gt;. Since all sklearn input files should be 2D (so adding a tuple or array as an input feature is not possible). I compared already different approaches to handle this (see below). &lt;strong&gt;However, here's my question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to improve classification of data with multidimensional features in python/sklearn after transformation of the datapoint from &lt;strong&gt;NxM&lt;/strong&gt; matrix to &lt;strong&gt;1xNM&lt;/strong&gt; matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;note: I have read these questions &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/11999147/is-it-possible-to-use-complex-numbers-as-target-labels-in-scikit-learn&quot;&gt;https://stackoverflow.com/questions/11999147/is-it-possible-to-use-complex-numbers-as-target-labels-in-scikit-learn&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/33483347/using-complex-valued-data-in-a-randomforestclassifier&quot;&gt;https://stackoverflow.com/questions/33483347/using-complex-valued-data-in-a-randomforestclassifier&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/1213/machine-learning-algorithms-for-2d-data&quot;&gt;machine learning algorithms for 2d data?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;so I am aware of the solution to just dissociate the data into two features (that is not my question), but as shown below, it seems hard for the learner to see a link between them without help.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Preparing the data&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is how I generate the data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generate_and_write_data(n):&#xA;    np.random.seed(358)&#xA;    data = pd.DataFrame(np.random.randn(n, 4), columns=('x1', 'x2', 'y1', 'y2'))&#xA;&#xA;    slopes = pd.Series((data.y2-data.y1)/(data.x2-data.x1))&#xA;    types = pd.Series(np.sign((data.y2-data.y1)/(data.x2-data.x1)))&#xA;    xdiff = pd.Series((data.x2-data.x1))&#xA;&#xA;    data['xdifference'] = pd.Series(xdiff, index=data.index)&#xA;    data['slopetype'] = pd.Series(types, index=data.index)&#xA;    data['slopes'] = pd.Series(slopes, index=data.index)&#xA;    return data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which results in the following plotted data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GLhQS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GLhQS.png&quot; alt=&quot;Raw data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, same data but colored according to the slope (so you can see a kind of trend)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/y0VVv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y0VVv.png&quot; alt=&quot;Colored according to slope&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two pipelines I constructed, because I initially thought an ICA could do better than a PCA. Also, since I wanted to end up with only 2 components (intuitively, one for each point?), I added these models to the comparison.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;icamodel = make_pipeline(FastICA(), svm.SVC())&#xA;pcamodel = make_pipeline(PCA(), svm.SVC())&#xA;icamodel2 = make_pipeline(FastICA(2), svm.SVC())&#xA;pcamodel2 = make_pipeline(PCA(2), svm.SVC())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I have several post-treatments of the data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sorting (changing x1 and x2, as well as y1 and y2), so the x1 is always lower than x2 (see below for an image) &lt;/li&gt;&#xA;&lt;li&gt;Using the distance between x1 and x2 as a feature, combined with y1 and y2. (so the x-values are omitted. See above in the data construction)&lt;/li&gt;&#xA;&lt;li&gt;Sorting, and only feeding the y-values to the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here is how I sort the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Xdiff = df.loc[:, 'y1':'xdifference']&#xA;&#xA;X_order = map((lambda x1,x2,y1,y2: [x1,x2,y1,y2] if x1 &amp;lt; x2 else [x2,x1,y2,y1]), X.x1, X.x2, X.y1, X.y2)&#xA;X_order = pd.DataFrame(X_order, columns=('x1', 'x2', 'y1', 'y2'))&#xA;&#xA;X2_order = map((lambda x1,x2,y1,y2: [y1,y2] if x1 &amp;lt; x2 else [y2,y1]), X.x1, X.x2, X.y1, X.y2)&#xA;X2_order = pd.DataFrame(X2_order, columns=('y1', 'y2'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sorted data:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/o0yKX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o0yKX.png&quot; alt=&quot;Sorted data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Model output&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The comparison of these two different models (PCA vs ICA) with the different data treatments yielded a huge difference in performance. I constructed the learning curves for all models, feeded with the different datasets (training score and validation score are in the same color). They are pictured below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different data inputs, training the PCA model, &lt;strong&gt;mapping to 4 vectors&lt;/strong&gt;:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/XKdLO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XKdLO.png&quot; alt=&quot;Different data inputs, training the PCA model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different data inputs, training the ICA model, &lt;strong&gt;mapping to 4 vectors&lt;/strong&gt;. The unprocessed and xdist models seem to be equivalent:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aHKfR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aHKfR.png&quot; alt=&quot;Different data inputs, training the ICA model. The unprocessed and xdist models seem to be equivalent&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different data inputs, training the PCA model, &lt;strong&gt;mapping to 2 vectors&lt;/strong&gt;.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/XY4Ik.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XY4Ik.png&quot; alt=&quot;PCA, 2 dims&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Different data inputs, training the ICA model, &lt;strong&gt;mapping to 2 vectors&lt;/strong&gt;. The unprocessed and xdist models seem to be equivalent.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/QJXkZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QJXkZ.png&quot; alt=&quot;ICA, 2 dims&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that the model where I sort, omit the x data and feed the y data to the model always performs best. &lt;strong&gt;However, this feels like an huge simplification of the model and won't help me with the complex numbers.&lt;/strong&gt; also, note that the ICA model performs as good as the PCA model when we map to 2 dimensions and feed all ordered data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I prefer the PCA mapping to 4 dimensions, with the ordered data. It does quite a good job, but still not as good as I want it. I'd welcome any suggestion for transforming the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;As a sidenote, As soon as I used the FastICA in a pipeline, I got the warning 'FastICA did not converge' as discussed here &lt;a href=&quot;https://github.com/rhiever/tpot/issues/148&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/rhiever/tpot/issues/148&lt;/a&gt; so comparing the models is probably not possible until I find the right tolerance/number of iterations for the ICA. However, this only happened when I used the ICA in a pipeline. When I used it separately, it was able to map the sorted X data to two different regions which were discernible by eye without the error.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="41155" LastEditorUserId="41155" LastEditDate="2017-10-29T17:49:15.470" LastActivityDate="2018-04-29T01:08:32.740" Title="Improve 2D data handling to classify according to sign of slope" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24180" PostTypeId="1" AcceptedAnswerId="24181" CreationDate="2017-10-29T17:52:13.113" Score="0" ViewCount="249" Body="&lt;h1&gt;Problem&lt;/h1&gt;&#xA;&lt;p&gt;I want to train HyperGAN with a set of 400+ images of people, but they aren't the specified size (32x32 pixels) for training.&lt;/p&gt;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&lt;p&gt;Is there any way/program to help cropping/resizing them so as to not do it 100% manually?&lt;/p&gt;&#xA;" OwnerUserId="41162" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-10-29T18:18:43.623" Title="Cropping Images for Dataset" Tags="|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24181" PostTypeId="2" ParentId="24180" CreationDate="2017-10-29T18:18:43.623" Score="2" Body="&lt;p&gt;Cropping and/or resizing is very trivial using &lt;a href=&quot;https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_gui/py_image_display/py_image_display.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenCV&lt;/a&gt;. You could write a 10 line script to iterate over all your images and apply the transformation you require.&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-29T18:18:43.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24182" PostTypeId="2" ParentId="24169" CreationDate="2017-10-29T18:22:24.437" Score="0" Body="&lt;p&gt;Having more features than batchsize it's not a problem indeed is the rule in many fields like computer vision. Having a very sparse dataset is also the norm in NLP. Both communities rely on &lt;a href=&quot;https://www.youtube.com/watch?v=UcKPdAM8cnI&quot; rel=&quot;nofollow noreferrer&quot;&gt;dropout&lt;/a&gt; to prevent &lt;a href=&quot;https://en.wikipedia.org/wiki/Overfitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;overfitting&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may see an example of using drouput with Tensorflow &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39513" LastEditorUserId="39513" LastEditDate="2017-10-29T18:33:43.773" LastActivityDate="2017-10-29T18:33:43.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24183" PostTypeId="1" CreationDate="2017-10-29T21:03:23.440" Score="6" ViewCount="429" Body="&lt;p&gt;&lt;a href=&quot;http://colorbrewer2.org/#type=qualitative&amp;amp;scheme=Set3&amp;amp;n=12&quot; rel=&quot;nofollow noreferrer&quot;&gt;ColorBrewer&lt;/a&gt; seems to be very useful in selecting a color pallet to represent factors that have up to 12 possible values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 28.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a horrible idea to represent 28 variables with color? If so, could you suggest an alternative visual indicator? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm using the colors for column side colors in a heatmap shown below. As you can see, the Strain column is not very informative:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CPVWe.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CPVWe.jpg&quot; alt=&quot;Strain/Condition Image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30754" LastEditorUserId="1330" LastEditDate="2020-12-10T19:03:19.787" LastActivityDate="2024-02-08T07:01:20.157" Title="Visualizing 28 different variables with 28 different colors?" Tags="|visualization|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="24184" PostTypeId="2" ParentId="24054" CreationDate="2017-10-29T22:47:54.137" Score="1" Body="&lt;p&gt;It's not just about combining different feature types count/interval/ratio. Formally, you probably &quot;only&quot; need to treat all of them as ordinal data, and then you would supposedly be fine...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pretty much any clustering algorithm is extremely sensitive to feature scaling. One star is not the same as one count. And there is no objective factor how to weight one versus the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution is to &lt;strong&gt;abandon the idea of a &quot;correct&quot; clustering&lt;/strong&gt;. Consider any result to be just a suggestion. Study it, analyze it, debate it. Take only one of the clusters (not the entire clustering). Mix clusters  of different runs. Do &lt;em&gt;not&lt;/em&gt; assume everybody is part of exactly one cluster, only some will cluster. Formalize single clusters. Verify each cluster with an experiment such as an A/B test. Etc.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-10-29T22:47:54.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24185" PostTypeId="2" ParentId="16704" CreationDate="2017-10-30T02:48:39.900" Score="3" Body="&lt;p&gt;The features for a token in a NER algorithm are usually binary. i.e The feature exists or it does not. For example, a token (say the word 'hello'), is all lower case. Therefore, that is a feature for that word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could name the feature 'IS_ALL_LOWERCASE'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for POS tags, lets take the word 'make'. It is a verb and hence the feature &quot;IS_VERB&quot; is a feature for that word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A gazetter can be used to generate features. The presence (or absence) of a word in the gazatter is a valid feature. Example: the word 'John' is present in the gazetter of Person names. so &quot;IS_PERSON_NAME&quot; can be a feature.&lt;/p&gt;&#xA;" OwnerUserId="41181" LastActivityDate="2017-10-30T02:48:39.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24186" PostTypeId="1" AcceptedAnswerId="24289" CreationDate="2017-10-30T05:10:47.697" Score="1" ViewCount="490" Body="&lt;p&gt;I am trying to classify  &lt;code&gt;heading&lt;/code&gt;, &lt;code&gt;image&lt;/code&gt; and &lt;code&gt;image caption&lt;/code&gt;  of a webpage. I am preparing data by scraping selected URLs(around 1000) using &lt;code&gt;XPath&lt;/code&gt; of &lt;code&gt;DOM elements&lt;/code&gt; I need. Each data row in the CSV file contains &lt;code&gt;tag name, x-coordinate, y-coordinate, text-size&lt;/code&gt; etc with three target labels &lt;code&gt;heading, image, image caption&lt;/code&gt; but a text simply will not become image caption until unless it is under an image. Is there any way to create this dependency among rows.&lt;/p&gt;&#xA;" OwnerUserId="35463" LastEditorUserId="1424" LastEditDate="2017-11-02T10:02:06.890" LastActivityDate="2017-11-02T10:02:06.890" Title="Classification: How to manage data sets where one data row depends on another data row" Tags="|machine-learning|svm|feature-extraction|feature-engineering|feature-construction|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24187" PostTypeId="2" ParentId="16422" CreationDate="2017-10-30T06:05:05.130" Score="5" Body="&lt;p&gt;Inspired by Einstein, &quot;“If you can't explain it to a six year old, you don't understand it yourself.”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of the above answers are very well explained but if one is looking for an easy to remember, abstract difference, here is the best one I know:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The key difference is Machine Learning only digests data, while Deep Learning can generate and enhance data. It is not only predictive but also generative.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/intuitionmachine/data-science-is-informative-but-deep-learning-is-transformative-316e61871dd8&quot; rel=&quot;noreferrer&quot;&gt;Source.&lt;/a&gt; Of course there is much more to it but for beginners it can get way too confusing. &lt;/p&gt;&#xA;" OwnerUserId="41185" LastActivityDate="2017-10-30T06:05:05.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24189" PostTypeId="1" CreationDate="2017-10-30T08:35:01.917" Score="3" ViewCount="3503" Body="&lt;p&gt;Among the different stages of creating a prediction model, when should I balance the data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should it be at the beginning before I perform any manipulation on the data and creating new features, or after, when I already selected the most important features based on the real data?&lt;/p&gt;&#xA;" OwnerUserId="33861" LastActivityDate="2022-08-02T14:31:10.440" Title="Data balance -before or after feature selection/engineering" Tags="|machine-learning|feature-selection|feature-engineering|class-imbalance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24190" PostTypeId="1" CreationDate="2017-10-30T10:45:14.430" Score="-2" ViewCount="2578" Body="&lt;p&gt;I have a data set that is formatted according to mlogit's standards using mlogit.data command in Rstudio. Here is the first three rows of the dataset (total number of rows 3000+).&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;         Trip  SevereEarthquake  Night  Age  Mode&#xA;1.NTG    1     0                 0      18   FALSE&#xA;1.TGNV   1     0                 0      18   FALSE&#xA;1.TGV    1     0                 0      18   TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;After some variable creation which I do not include here to make the question short, I ran a multinomial logit model using the code below and it worked fine.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;mlogit(Mode ~ SE.TGV + SE.TGNV + AGE.NTG, data, reflevel = &amp;quot;NTG&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;However when I define nests in the code (as below) to do a nested logit model I get an error:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;mlogit(Mode ~ SE.TGV + SE.TGNV + AGE.NTG, data, reflevel = &amp;quot;NTG&amp;quot;, &#xA;       nests = list(notrip = &amp;quot;NTG&amp;quot;, trip = c(&amp;quot;TGV&amp;quot;,&amp;quot;TGNV&amp;quot;)))&#xA;&#xA;&#xA;Error in solve.default(crossprod(attr(x, &amp;quot;gradi&amp;quot;)[, !fixed])) : &#xA;system is computationally singular: reciprocal condition number = 8.87901e-37&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;What is causing this problem? How can I solve this?&lt;/p&gt;&#xA;" OwnerUserId="38740" LastEditorUserId="38740" LastEditDate="2024-02-27T18:39:10.800" LastActivityDate="2024-02-27T18:39:10.800" Title="Error in mlogit package: system is computationally singular: reciprocal condition number = 8.87901e-37" Tags="|r|rstudio|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24191" PostTypeId="2" ParentId="24190" CreationDate="2017-10-30T10:54:25.777" Score="0" Body="&lt;p&gt;This means that your variables are 'dependent'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your NTG and TGNV have the exact same values. Your system can't tell if contributions are made because of either one of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either you should try to find more data that is different between NTG and TGNV to make the system independent, or you can just get rid of one of the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know much R (yet!), but after you combine TGV and TGNV, shouldn't your Mode relate to 'trip' instead of both TGV and TGNV?&#xA;Since TGV and TGNV are now grouped, the model matrix can't separate them or something like that. (And maybe the AGE.NTG part has to change too.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mlogit(Mode ~ ??? + AGE.NTG, data, reflevel = &quot;NTG&quot;, &#xA;   nests = list(notrip = &quot;NTG&quot;, trip = c(&quot;TGV&quot;,&quot;TGNV&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33812" LastEditorUserId="33812" LastEditDate="2017-10-30T16:04:30.890" LastActivityDate="2017-10-30T16:04:30.890" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24193" PostTypeId="1" CreationDate="2017-10-30T13:29:38.933" Score="-1" ViewCount="54" Body="&lt;p&gt;I am a newbie in Data Science, so please don't blame me for stupid questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my problem. I've got a dataset (no empty cells, only numerical values) consisted of 15 columns (1st - user id, last one - response, 2-14 - different features such as eventtype, country, browser etc., actually this dataset describes users actions on some webpage). Response value could be 1 or 0, so it's definitely a classification problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to have some practice I would like to try main algorithms on this dataset (kNN, Random Forest, SVC). The problem is that in this dataset I've got around 5000 unique id's and the total number of lines is around 17000, so different users have several lines related to their actions on the webpage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, algorithms listed above gave too good predictions, but I suppose it's not correct to use them on such data, because they make predictions for the object based on this object's historic data. Which options do I have to improve the dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I could try to leave only unique id's in dataset, but it will reduce the dataset size from 17k lines to 5k lines, which, I suppose, will have a negative effect on predictions. Which other strategis I can use? &lt;/p&gt;&#xA;" OwnerUserId="41196" LastEditorUserId="18375" LastEditDate="2017-12-11T20:33:43.310" LastActivityDate="2017-12-11T20:33:43.310" Title="Dataset processing question" Tags="|dataset|preprocessing|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24194" PostTypeId="1" AcceptedAnswerId="24198" CreationDate="2017-10-30T14:29:18.697" Score="2" ViewCount="137" Body="&lt;p&gt;I have a question regarding annotating text data for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).&lt;/li&gt;&#xA;&lt;li&gt;Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thoughts,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.&lt;/li&gt;&#xA;&lt;li&gt;will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21257" LastActivityDate="2017-10-30T15:26:02.207" Title="Text annotating process, quality vs quantity?" Tags="|machine-learning|classification|data-mining|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24196" PostTypeId="1" CreationDate="2017-10-30T14:40:57.847" Score="3" ViewCount="2736" Body="&lt;p&gt;I'm currently looking for the correct number of 2018 budget for the AWS instances in preparation of more deep neural net training, but now a bit confused because of this chart below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UEqhL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UEqhL.png&quot; alt=&quot;AWS EC2 Cost&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p2.16xlarge costs 3 times more than g3.16xlarge with the same number of processors but slightly less memory. Is this diff just because of the memory difference? or is there any other key difference between p2 and g3 instances? If I'm missing anything here, please share your wisdom!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="41200" LastEditorUserId="41200" LastEditDate="2017-10-30T14:59:07.700" LastActivityDate="2018-02-01T19:42:31.820" Title="Why is Amazon P2 instance cost much more than G3?" Tags="|deep-learning|aws|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24197" PostTypeId="2" ParentId="24193" CreationDate="2017-10-30T15:00:51.523" Score="2" Body="&lt;p&gt;What you have here is some data that's too specific. I don't know if there's a widely accepted name for that, but I've seen it be called &quot;leaking&quot; or &quot;cheating&quot; data. Using this kind of data is potentially dangerous for the algorithm, as you have seen in your experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a quick example: I want to predict whether a costumer prefers to wear white socks or black socks. I have a lot of information from this costumer, like the city he lives or how many TVs he owns (let's pretend there's some relation with preferred sock color). For some reason, the people handling the interview I use as data sorted the 200 interview responses by class - the first 120 responses are for white socks, and the next 80 are for black socks. Each response has an identifier from 1 to 200.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I run this data through a decision tree learner. It comes up with a stump (a tree with only a single node) containing the rule:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;If Id &amp;lt;= 120 Then&#xA;   User prefers white socks&#xA;Else&#xA;   User prefers black socks&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is completely useless! If I just keep giving new identifiers to people, I'll assume every new costumer prefers black socks! (Except if one of the first 120 comes back).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While may be some advanced ways to deal with this problem to use the fact the user may repeat and that's relevant (it usually is), I wouldn't recommend it. 17k instances isn't that much to try for a more complex model, you could be overfitting. Only go towards some other strategy (like frequent pattern mining, or turning the problem in a time series of sort) if and only if removing the Id column is not good enough.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-10-30T15:00:51.523" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24198" PostTypeId="2" ParentId="24194" CreationDate="2017-10-30T15:26:02.207" Score="2" Body="&lt;p&gt;It depends on the context which you consider. For example, suppose there is a situation that all possible states can be covered by 10K different texts. It is trivial, if these texts are all annotated, then for 1000 tests, at least we can classify 500 of them truly (as we have two classes, and probability of wrong annotation for each text is at most 0.5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose 1K of 10K texts are annotated. Then, as annotations are exact, we can classify 1/10 of 1000 texts truly (because we have no idea about the other 9K possible states).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, in this situation quantity is more important than quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, we can consider these cases, when the possible states are 1K. It could be straightforward to show that in this case (if the power of annotators is same as the former case) quality can be more important than quantity. However, in the most cases this number is not realistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, as in the most cases the variety of texts are more than the power of annotators, we prefer quantity to quality, as we can cover more text space and machine can learn more. Although the accuracy can be less, but for two classes classification is negligible.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-10-30T15:26:02.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24199" PostTypeId="2" ParentId="15882" CreationDate="2017-10-30T15:27:55.750" Score="1" Body="&lt;p&gt;Perhaps the blog below provides an answer to your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without go through code in much detail, probably, your problem can be described as followed (from the blog):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the case that the quantile value q  is relatively far apart from the observed values within the partition, then because of the Gradient and Hessian both being constant for large difference x_i-q, the score stays zero and no split occurs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then the following solution is suggested:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An interesting solution is to force a split by adding randomization to the Gradient. When the differences between the observations x_i and the old quantile estimates q within partition are large, this randomization will force a random split of this volume.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="41202" LastActivityDate="2017-10-30T15:27:55.750" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24200" PostTypeId="2" ParentId="8213" CreationDate="2017-10-30T16:29:52.690" Score="3" Body="&lt;p&gt;As a complement to Jérémie Clos' and AN6U5's answers, there are at least two methods helping to cope with a large number of classes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use a hierarchy, like &lt;em&gt;hierarchical softmax&lt;/em&gt;. Instead of having a flat list of categories, one builds a tree of them, then on each node predicts if the correct category is on the left or on the right branch.&lt;/li&gt;&#xA;&lt;li&gt;do not classify directly, but first learn an &lt;em&gt;embedding&lt;/em&gt; into a lower-dimensional space, where instances of the same class should have close representations. A famous example for this is FaceNet (the use case is face recognition) : they embed the image of a face into a 128 dimensional byte vector. The algorithm to learn this embedding is triplet loss (I've heard of magnet loss as well). Then when presented with a new request, compute its representation (a small vector), and look for the closest vectors in the trainset. This is similar to kNN, or to &lt;a href=&quot;https://arxiv.org/abs/1607.05691&quot; rel=&quot;nofollow noreferrer&quot;&gt;label embedding&lt;/a&gt; if each instance belongs to several classes: &quot;Our method consists in embedding high-dimensional sparse labels onto a lower-dimensional dense sphere of unit-normed vectors, and treating the classification problem as a cosine proximity regression problem on this sphere.&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The work of &lt;a href=&quot;http://manikvarma.org/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://manikvarma.org/index.html&lt;/a&gt; might be relevant to your problem as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alos of potential use: &lt;a href=&quot;http://jmlr.org/papers/volume15/gupta14a/gupta14a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://jmlr.org/papers/volume15/gupta14a/gupta14a.pdf&lt;/a&gt; .&lt;/p&gt;&#xA;" OwnerUserId="3410" LastEditorUserId="3410" LastEditDate="2018-09-25T09:26:36.703" LastActivityDate="2018-09-25T09:26:36.703" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24202" PostTypeId="1" CreationDate="2017-10-30T16:50:00.693" Score="9" ViewCount="753" Body="&lt;p&gt;In the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (&lt;a href=&quot;https://arxiv.org/pdf/1502.03167.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), before explaining the process of batch normalization, the paper tries to explain the issues related with &lt;strong&gt;(I am not getting what the exact issue addressed here is)&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Excerpt from section 2, para 2:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We could consider whitening activations at every training&#xA;step or at some interval, either by modifying the&#xA;network directly or by changing the parameters of the&#xA;optimization algorithm to depend on the network activation&#xA;values (Wiesler et al., 2014; Raiko et al., 2012;&#xA;Povey et al., 2014; Desjardins &amp;amp; Kavukcuoglu). However,&#xA;if these modifications are interspersed with the optimization&#xA;steps, then the gradient descent step may attempt&#xA;to update the parameters in a way that requires&#xA;the normalization to be updated, which reduces the effect&#xA;of the gradient step. For example, consider a layer&#xA;with the input u that adds the learned bias &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;, and normalizes&#xA;the result by subtracting the mean of the activation&#xA;computed over the training data: &lt;span class=&quot;math-container&quot;&gt;$\hat x= x − E[x]$&lt;/span&gt; where&#xA;&lt;span class=&quot;math-container&quot;&gt;$x = u + b, X = {x_{1...N}}$&lt;/span&gt; is the set of values of &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; over&#xA;the training set, and &lt;span class=&quot;math-container&quot;&gt;$E[x] = \frac 1 N(\sum_{i=1}^nx_i)$&lt;/span&gt;&#xA;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;If a gradient&#xA;descent step ignores the dependence of E[x] on b, then it&#xA;will update &lt;span class=&quot;math-container&quot;&gt;$b ← b + ∆b$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$∆b ∝ −\partial l/\partial\hat x$&lt;/span&gt;. Then&#xA;&lt;span class=&quot;math-container&quot;&gt;$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Thus, the combination of the update to &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; and subsequent&#xA;change in normalization led to no change in the output&#xA;of the layer nor, consequently, the loss. As the training&#xA;continues, b will grow indefinitely while the loss remains&#xA;fixed. This problem can get worse if the normalization not&#xA;only centers but also scales the activations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Here is my understanding of the literature:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We have a batch of size &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; (One training batch).&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Let there be two arbitrary hidden layers connected to each other (&lt;span class=&quot;math-container&quot;&gt;$L_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$L_2$&lt;/span&gt;) connected by parameters &lt;span class=&quot;math-container&quot;&gt;$W$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Output coming out of &lt;span class=&quot;math-container&quot;&gt;$L_1$&lt;/span&gt; is &lt;span class=&quot;math-container&quot;&gt;$x_1$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$u = x_1 \cdot W$&lt;/span&gt; (this is where the literature above starts. dimension of &lt;span class=&quot;math-container&quot;&gt;$u$&lt;/span&gt; is &lt;span class=&quot;math-container&quot;&gt;$M \times N$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; is the number of units in &lt;span class=&quot;math-container&quot;&gt;$L_2$&lt;/span&gt;)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x = u+b$&lt;/span&gt; (dimension &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; = dimension &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; = dimension &lt;span class=&quot;math-container&quot;&gt;$u$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$M \times N$&lt;/span&gt;)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now before feeding &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; into &lt;span class=&quot;math-container&quot;&gt;$L_2$&lt;/span&gt; we center it by subtracting the mean of &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; from each entry in &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; (&lt;span class=&quot;math-container&quot;&gt;$\hat x= x − E[x]$&lt;/span&gt;)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We compute the loss, backpropagate the gradient, and update just this layer's &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; in order to give it a sanity test.&#xA;New &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$b + \Delta b$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We run it again on the same batch with updated &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat steps 3 and 4.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x_{new} = u + b + \Delta b$&lt;/span&gt; (dimension &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$\Delta b$&lt;/span&gt; = dimension &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; = dimension &lt;span class=&quot;math-container&quot;&gt;$u$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$M \times N$&lt;/span&gt;)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now before feeding &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; into &lt;span class=&quot;math-container&quot;&gt;$L_2$&lt;/span&gt; we center it by subtracting the mean of &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; from each entry in &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; (&lt;span class=&quot;math-container&quot;&gt;$\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$&lt;/span&gt;).&#xA;which is the same as what was calculated before updating &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; and hence updating &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; had to effect on the training.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;My question is with this part of the excerpt:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;If a gradient descent step ignores the dependence of E[x] on b, then it will update &lt;span class=&quot;math-container&quot;&gt;$b ← b + ∆b$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$∆b \propto −\partial l/\partial\hat x$&lt;/span&gt;. &lt;strong&gt;Then&lt;/strong&gt; &lt;span class=&quot;math-container&quot;&gt;$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Why is&#xA;&lt;span class=&quot;math-container&quot;&gt;$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$&lt;/span&gt; dependent upon what comes before it? What is even the point of that bit? Please also note the usage of the word &amp;quot;Then&amp;quot; (made bold) implying the statement necessarily draws causality from what comes before.&lt;/p&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="-1" LastEditDate="2023-08-25T04:15:11.350" LastActivityDate="2023-08-25T04:15:11.350" Title="Understanding batch normalization" Tags="|neural-network|deep-learning|batch-normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24203" PostTypeId="1" AcceptedAnswerId="24257" CreationDate="2017-10-30T17:02:43.040" Score="4" ViewCount="266" Body="&lt;p&gt;I'm using a KMeans to get the profile of several users according to several columns (I'm working with RStudio).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To analyze my clusters, I decided to realize a radar chart, so I decided to use feature scaling : &lt;code&gt;x-min(x)/diff(range(x))&lt;/code&gt;, to have my values in [0,1] (to get a quite good idea of my data per cluster). &#xA;However, since there are multiple choice for normalization, I was wondering if doing my analysis with another choice for normalization - for instance : &lt;code&gt;x-mean(x)/sd(x)&lt;/code&gt; - would give me the same results (in a general way at least)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or am I completly wrong for considering my scaled data and should I use my unscaled data in my radar chart ?&lt;/p&gt;&#xA;" OwnerUserId="39664" LastActivityDate="2017-11-01T16:10:19.137" Title="Does the choice of normalization change dramatically the result of a KMeans" Tags="|clustering|k-means|normalization|rstudio|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24204" PostTypeId="2" ParentId="24167" CreationDate="2017-10-30T17:55:34.870" Score="6" Body="&lt;p&gt;Yes. It is also a function, but not an affine transformation of the input but a relatively complex sum of products of indicator functions of the input. Usually, this function is represented by the fitted tree and not as a formula.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So e.g. if you learn a tree of depth one and the split is at age 40 with mean response of 80 if age &amp;lt; 40 and mean response of 100 if age $\ge$ 40, then the function could look like&#xA;$$&#xA;  \hat f(\text{age}, \text{income}) = 80 \cdot {\mathbf 1}\{\text{age} &amp;lt; 40\} + 100 \cdot {\mathbf 1}\{\text{age} \ge 40\}&#xA;$$&#xA;You can maybe imagine how long the formula is if the depth is 7...&lt;/p&gt;&#xA;" OwnerUserId="36977" LastEditorUserId="36977" LastEditDate="2017-10-30T21:05:27.790" LastActivityDate="2017-10-30T21:05:27.790" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24207" PostTypeId="2" ParentId="19976" CreationDate="2017-10-30T20:07:08.720" Score="1" Body="&lt;p&gt;This may be too simple for what you had in mind, but perhaps it will help:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Example data&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse); library(magrittr) # load R libraries&#xA;sample_data = data.frame(Customer = 1:100, A = 0, B = rep(c(10,NA), c(25,75)), C = rep(c(NA,25), c(25,75))) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Bar chart&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample_data %&amp;gt;% gather(&quot;Product&quot;, &quot;Time&quot;, -Customer) %&amp;gt;% ggplot()+geom_bar(aes(x=Time, fill=Product))+geom_label(aes(x=Time, label=Product), y=-Inf, vjust=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tJe64.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tJe64.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Point plot&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The use of this is that you can map the customers (since the y-axis is not the count but the Customer ID), i.e., you can add lines connecting them but they would be too many, so I haven't (they can be grouped though, for A-&gt;B, B-&gt;C, A-&gt;C, etc)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample_data %&amp;gt;% gather(&quot;Product&quot;, &quot;Time&quot;, -Customer) %&amp;gt;% ggplot()+geom_point(aes(x=Time, y=Customer, color=Product))+geom_label(aes(x=Time, label=Product), y=-Inf, vjust=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MdXKS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MdXKS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40767" LastActivityDate="2017-10-30T20:07:08.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24208" PostTypeId="2" ParentId="24179" CreationDate="2017-10-30T20:26:32.173" Score="1" Body="&lt;p&gt;&lt;strong&gt;Increasing the number of data points and using KernelPCA.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Increasing the number of points for the PCA model increased also the scores of that model, something I overlooked. Furthermore, going to a KernelPCA component analysis also looked promising here (using 'rbf' kernel and n_dimensions=20 for the kernel model). All of the following graphs are produced using the raw data (so no sorting, editing... happened).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D9IQd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D9IQd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optimizing the KernelPCA transformer with gridsearchCV yields that a Pipeline with KernelPCA model with parameters {'kernelpca__kernel': 'cosine', 'kernelpca__n_components': 21, 'svc__C': 2154.4346900318847} gives best results (where C could possibly be tweaked a bit further). However, this results in an accuracy of over 98% which is very satisfying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final model looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kernelmodel = make_pipeline(KernelPCA(21, kernel='cosine'), &#xA;       Imputer(strategy='median'), svm.SVC(C=2000))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where the imputer is added to account for som NaNs that were created (?) within the KernelPCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And results in the following graph&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mR1UJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mR1UJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other recommendations are welcome!&lt;/p&gt;&#xA;" OwnerUserId="41155" LastActivityDate="2017-10-30T20:26:32.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24213" PostTypeId="1" CreationDate="2017-10-31T09:04:52.247" Score="1" ViewCount="1914" Body="&lt;p&gt;In the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification_report&lt;/a&gt; provided by &lt;code&gt;sklearn&lt;/code&gt;, which score should I look at to make the best determination of the accuracy of my model? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             precision    recall  f1-score   support&#xA;&#xA;          0       0.70      0.68      0.69      5007&#xA;          1       0.65      0.54      0.59      2270&#xA;          2       0.37      0.22      0.28       614&#xA;          3       0.74      0.30      0.42       252&#xA;          4       0.59      0.42      0.49       262&#xA;          5       0.35      0.11      0.17       455&#xA;          6       0.34      0.23      0.27       248&#xA;          7       0.09      0.05      0.06       133&#xA;          8       0.38      0.15      0.21       395&#xA;          9       0.43      0.31      0.36       182&#xA;         10       0.23      0.12      0.16       230&#xA;&#xA;avg / total       0.60      0.52      0.55     10048&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As far as I understand it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Precision tells us the amount of samples the classifier has &lt;em&gt;correctly&lt;/em&gt;&#xA;marked as true positive out of all positive results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recall tells us about the number of samples the classifier was able&#xA;to get correct out of &lt;em&gt;all&lt;/em&gt; samples in the set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;F1-score is the harmonic mean of precision and recall.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Maybe I'm misinterpreting the &lt;code&gt;classification_report&lt;/code&gt;, but wouldn't the &lt;code&gt;f1-score&lt;/code&gt; give the best view on the performance of the classifier?&lt;/p&gt;&#xA;" OwnerUserId="39014" LastActivityDate="2019-02-22T20:15:52.610" Title="In sklearn's classification report, is f1 the best accuracy measure?" Tags="|classification|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24214" PostTypeId="1" AcceptedAnswerId="24218" CreationDate="2017-10-31T09:39:50.023" Score="15" ViewCount="22705" Body="&lt;p&gt;I'm new to data science and Neural Networks in general.&#xA;Looking around, many people say it is better to normalize the data before doing anything with the NN. I understand how normalizing the &lt;strong&gt;input&lt;/strong&gt; data can be useful.&lt;/p&gt;&#xA;&lt;p&gt;However, I really don't see how normalizing the &lt;strong&gt;output&lt;/strong&gt; data can help.&lt;/p&gt;&#xA;&lt;p&gt;I've also tried both cases with an easy dataset, and I achieved the same results. The only difference is that in some weird problems, it is really hard to then re-convert the output back.&lt;/p&gt;&#xA;&lt;p&gt;Can you give me some intuition on why we should also normalize the output?&lt;/p&gt;&#xA;&lt;p&gt;Or maybe why it is indifferent?&lt;/p&gt;&#xA;" OwnerUserId="31316" LastEditorUserId="156891" LastEditDate="2023-12-19T09:28:24.493" LastActivityDate="2023-12-19T09:28:24.493" Title="Why should I normalize also the output data?" Tags="|neural-network|deep-learning|regression|data-cleaning|normalization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="24215" PostTypeId="2" ParentId="24213" CreationDate="2017-10-31T09:45:28.823" Score="3" Body="&lt;p&gt;You got &lt;code&gt;recall&lt;/code&gt; incorrectly. It does not mean &quot;correct out of all the samples&quot;.&#xA;$\hspace{50mm}$&lt;img src=&quot;https://s3-eu-west-1.amazonaws.com/ppreviews-plos-725668748/783561/preview.jpg&quot; width=&quot;400&quot; height=&quot;400&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the image,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hspace{50mm}$ &lt;code&gt;Recall&lt;/code&gt; = $\frac{TP}{TP+FN}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is explained &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the above documentation, you can also see that you can modify your &lt;code&gt;F-beta score&lt;/code&gt; to suite which to weigh more, precision or recall. Therefore, &lt;code&gt;f-score&lt;/code&gt; would give you an overall of how good your classifier is, but if you need to know at which samples your classifier fails, you need &lt;code&gt;precision&lt;/code&gt; and &lt;code&gt;recall&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39234" LastActivityDate="2017-10-31T09:45:28.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24217" PostTypeId="2" ParentId="23596" CreationDate="2017-10-31T10:15:04.173" Score="0" Body="&lt;p&gt;For the &lt;code&gt;colors&lt;/code&gt;  argument you need to assign a vector with the same length as your data points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't explain how colors must be assigned to values of your output columms . In any regard, this would assign 4 different colors randomly: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cols = sample(1:4, nrow(output), replace=TRUE) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;plot it with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scatterplot3d(output$X2,output$X6 , output$X7 , color=cols, pch=19, cex=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You could create a new column in your dataframe, e.g. &lt;code&gt;rgbcolor&lt;/code&gt; and assign that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scatterplot3d(output$X2,output$X6 , output$X7 , color=output$rgbcolor, pch=19, cex=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pch&lt;/code&gt; is the plotting character id, usually 1, 19 or 20 (circle or larger dot or smaller dot. (see for yourself: &lt;code&gt;plot(1:25, 1:25, pch=1:25)&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tune the &lt;code&gt;cex&lt;/code&gt; param to make it larger or smaller (1 means size = 100%)&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-10-31T10:15:04.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24218" PostTypeId="2" ParentId="24214" CreationDate="2017-10-31T10:56:00.497" Score="21" Body="&lt;p&gt;That &lt;a href=&quot;http://www.faqs.org/faqs/ai-faq/neural-nets/part2/&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; gives a nice answer, where i quoted from. Search for &lt;code&gt;Should I standardize the target variables (column vectors)?&lt;/code&gt; in that page.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the usual least (mean) squares error function, then the variability of each target relative to the others can effect how well the net learns that target. If one target has a range of 0 to 1, while another target has a range of 0 to 1,000,000, the net will expend most of its effort learning the second target to the possible exclusion of the first. So it is essential to rescale the targets so that their variability reflects their importance, or at least is not in inverse relation to their importance. If the targets are of equal importance, they should typically be standardized to the same range or the same standard deviation. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="39234" LastEditorUserId="53032" LastEditDate="2018-06-01T16:03:00.547" LastActivityDate="2018-06-01T16:03:00.547" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24219" PostTypeId="2" ParentId="22643" CreationDate="2017-10-31T10:56:08.687" Score="0" Body="&lt;p&gt;I think no-one can solve this analytically,  and I cannot model this easily with ML algorithms. Furthermore I think  your problem specification is not clear enough. For instance, there are not enough constraints, and you don't say how many types of  agents there are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any regard, this seems to be a job for the &lt;a href=&quot;http://ccl.northwestern.edu/netlogo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Netlogo&lt;/a&gt; agent-based-modeling software. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer part 1: You could reframe your question, assign it a &quot;NetLogo&quot; tag, ask again on stackoverflow.com. I happen to know that the Netlogo developers and other experts hang out there. You could get a much better answer than I could provide here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe someone has developed a NetLogo Model that is similar  to your problem, and you can adapt and study it. (Netlogo has an internal grid-search tool called &quot;BehaviourSpace&quot;), you can export data, and there are APIs for connecting Netlogo with other tools such as R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer part 2: In any case, this reminds me of the &quot;El Farol&quot; model included in the &lt;a href=&quot;http://ccl.northwestern.edu/netlogo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Netlogo&lt;/a&gt; agent-based-modeling software. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;El Farol is a bar in Santa Fe, New Mexico. The bar is popular —&#xA;  especially on Thursday nights when they offer Irish music — but&#xA;  sometimes becomes overcrowded and unpleasant. In fact, if the patrons&#xA;  of the bar think it will be overcrowded they stay home; otherwise they&#xA;  go enjoy themselves at El Farol. This model explores what happens to&#xA;  the overall attendance at the bar on these popular Thursday evenings,&#xA;  as the patrons use different strategies for determining how crowded&#xA;  they think the bar will be. ... El Farol was originally put forth by Brian Arthur (1994) as an example of how one might model economic systems of boundedly rational agents who use inductive reasoning.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have included a screeenshot so you'll get an impression what I'm talking about.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the screenshot, replace&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The Blue square (The bar) with &quot;Your System&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;Crowded&quot; with &quot;Capacity Utilization: high&quot;, &lt;/li&gt;&#xA;&lt;li&gt;&quot;Bar Attendance&quot; with &quot;General Performance&quot; and&lt;/li&gt;&#xA;&lt;li&gt;&quot;number-strategies&quot; with &quot;agent-types&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CP79C.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CP79C.png&quot; alt=&quot;netlogo-screenshot&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-10-31T10:56:08.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24220" PostTypeId="1" CreationDate="2017-10-31T11:49:56.533" Score="1" ViewCount="53" Body="&lt;p&gt;I am using Keras LSTM for time series forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am facing an issue, with transforming my data correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a big dataset with values as follows&#xA;[[x1,x2,x3,x4,x5,x6,x7,x8],&#xA;[x1,x2,x3,x4,x5,x6,x7,x8],&#xA;[x1,x2,x3,x4,x5,x6,x7,x8]....]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict x8.&#xA;x1-x7 are values that are available before x8 is found, and x8 is dependant upon them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, as the input into the LSTM, I'd have say 10 rows of all variables and then 5 rows of x1-x7, with the 5 rows of x8 as output, but I am having troubles getting such a vector into it. How should I proceed?&lt;/p&gt;&#xA;" OwnerUserId="41238" LastActivityDate="2017-10-31T11:49:56.533" Title="Input data from time series, keeping part of the steps to predict" Tags="|keras|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24221" PostTypeId="2" ParentId="24189" CreationDate="2017-10-31T12:18:38.760" Score="3" Body="&lt;p&gt;Balancing is using which of the samples to consider in the data set (adding/reducing rows).&#xA;Features selection and feature engineering is removing and adding information about each sample (adding/removing columns).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, you have the liberty about how to combine them.&#xA;Many times, the order doesn't matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are normalising features, they will be normalised with respect to the data set and you might get different results before and after balancing.&#xA;Feature might change it's importance after balancing, influencing the feature selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if you are modeling on a data set, it is better to adapt the feature engineering to it so balance first and do feature engineering later later unless you have a reason to do otherwise.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-10-31T12:18:38.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24222" PostTypeId="2" ParentId="16872" CreationDate="2017-10-31T13:24:53.073" Score="1" Body="&lt;p&gt;You don't need a prediction model for this. Maybe if you have had users' data. But without anything else, then you just need labeled data. Historical data that you know if it was a real accident or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have your labeled data, then you can follow a process like this, which is still heavily dependent on the kind of your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start iterating on your labeled dataset and calculate the accuracy of a real accident's report for different combinations (5, 10, 15, 20, 25, 30 ... mins) and (1, 2, 3, 4, 5, 6, 7, etc users).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have a 2D matrix. I guess, acting fast on an accident is important in your case. Set an acceptable accuracy and choose the combination with the smallest interval, above that accuracy.&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-10-31T13:24:53.073" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24223" PostTypeId="1" CreationDate="2017-10-31T13:31:57.640" Score="0" ViewCount="191" Body="&lt;p&gt;I'm working on a unusual issue (for me) and I need some advice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to have a recommendation algorithm that propose tags for a document, based on all the previously tagged documents. For example, if I upload each month an invoice from &quot;stackoverflow&quot;, and I apply each time the same tags (&quot;stackoverflow&quot; and &quot;invoice&quot; for instance), the algorithm should recognize the document as similar to previous stackoverflow invoices and suggest the same tags to the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about extracting some features of the document ( text, sender's email...), compare the similarity with previous documents and suggest the tags of the most similar documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tags are user-defined, and thus not a finite set, so the algorithm should learn continuously. This is my issue, because I will have to store all the documents in a database and evaluating the similarity between a new document and all the previous ones will eventually be very expensive as the number of documents in my database grow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other side, incremental learning algorithms can learn continuously without a massive database, but don't support an non-finite set of output classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as I see it, I have 2 options :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;massive database with all documents, and evaluating similarity&#xA;between a new document and each document in the database : not very&#xA;scalable, possibly time-expensive&lt;/li&gt;&#xA;&lt;li&gt;re-train from scratch a classification algorithm every time a new document is uploaded : I will still have a massive database, but training can be done in the background. This is the best of the 2 options, but I'm wondering if there could be a better way.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Any advice about more appropriate algorithm would be greatly appreciated. If it matters, I work with Python.&lt;/p&gt;&#xA;" OwnerUserId="23679" LastActivityDate="2017-12-01T12:13:59.453" Title="Recommend tags for documents" Tags="|classification|recommender-system|multilabel-classification|online-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24227" PostTypeId="1" AcceptedAnswerId="24229" CreationDate="2017-10-31T15:26:11.990" Score="3" ViewCount="250" Body="&lt;p&gt;Some observations are far too voluminous in their raw state to be modeled by predictive modeling algorithms directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Common examples include image, audio, and textual data, but could just as easily include tabular data with millions of attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature extraction is a process of automatically reducing the dimensionality of these types of observations into a much smaller set that can be modelled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For tabular data, this might include projection methods like Principal Component Analysis and unsupervised clustering methods. For image data, this might include line or edge detection. Depending on the domain, image, video and audio observations lend themselves to many of the same types of DSP methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about generating new features with higher predictive values from the raw data and concatenate them to the raw data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I have data about student wealth, health, family status and I want somehow generate a new feature I can call Social status whic is generated from the raw data and with high predictive value? Is this possible? linear regression can be a good way I need to discover?&lt;/p&gt;&#xA;" OwnerUserId="40426" LastEditorUserId="29575" LastEditDate="2017-10-31T20:58:39.547" LastActivityDate="2017-10-31T20:58:39.547" Title="The automatic construction of new features from raw data" Tags="|machine-learning|deep-learning|feature-selection|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24228" PostTypeId="2" ParentId="17370" CreationDate="2017-10-31T15:51:44.347" Score="2" Body="&lt;p&gt;One way to look at this data is by drawing trend lines.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Line graph&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse) # for data transformations&#xA;library(magrittr) # for piping&#xA;library(forcats) # for factor manipulations&#xA;data %&amp;gt;% &#xA;separate(&quot;type&quot;, c(&quot;price&quot;, &quot;size&quot;), sep=-2, remove = F) %&amp;gt;% # optional, separate 'type' into 'price' and 'size' bands&#xA;gather(&quot;product&quot;, &quot;value&quot;, -c(price, size, type)) %&amp;gt;% # 'melt' products&#xA;mutate(type = fct_reorder(as.factor(type), value),&#xA;product = fct_reorder(as.factor(product), value)) %&amp;gt;% # optional, re-order 'type'/'product' factors by median 'value' numbers&#xA;ggplot() + geom_line(aes(x=type, y=value, color=product, group=product))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WrRDs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WrRDs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you would like a heatmap, there is another way to use that too. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Heat map&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data %&amp;gt;% &#xA;separate(&quot;type&quot;, c(&quot;price&quot;, &quot;size&quot;), sep=-2, remove = F) %&amp;gt;% &#xA;gather(&quot;product&quot;, &quot;value&quot;, -c(price, size, type)) %&amp;gt;% &#xA;mutate(type = fct_reorder(as.factor(type), value),&#xA;product = fct_reorder(as.factor(product), value)) %&amp;gt;% &#xA;ggplot() + geom_tile(aes(x=price, y=size, fill=value)) + facet_grid(product~.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a2wue.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a2wue.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Explanation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Notice that in the line graph the 'type' factor is re-ordered by the median 'value' numbers. The plot shows that A1 tends to have lower value, whereas C2 tends to have higher value.&lt;br&gt;&#xA;Similarly, in the heat map the 'product' factor is re-ordered by the median 'value' numbers. Looking at the plot you can tell that p4 has the highest median values, whereas p5 has the lowest median values.&lt;br&gt;&#xA;You can play with re-ordering and faceting to identify different trends.&lt;/p&gt;&#xA;" OwnerUserId="40767" LastActivityDate="2017-10-31T15:51:44.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24229" PostTypeId="2" ParentId="24227" CreationDate="2017-10-31T16:14:57.020" Score="2" Body="&lt;p&gt;'high predictive value' is only defined if you a target which you are trying to predict upon. It seems that you don't, and your goal is to &lt;em&gt;cluster&lt;/em&gt; data points according to some scale defined by a variety of factors. These can undoubtedly be used to cluster data points, and I'd advise you to look into the various methods available: some that may be interesting for you are Agglomerative and Hierarchical clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to answer the question, you can surely generate new features from the ones present in your dataset that may or may not help you achieve your goal. You can: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Bin your data: define some categories such as 'rich', 'average', 'poor' with specified ranges and create a new feature that maps a numerical value (wealth) to a bin&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;One-hot-encode&lt;/a&gt; categorical variables&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After these pre-processing steps are done, you could go ahead an apply the clustering methods I mentioned to group data together into their respective 'Social Status'. Of course, lots of tweaking and experimentation will be needed. As far as what I've encountered, there are not really automagic ways of generating new features, and mostly the available methods will depend greatly on the type of data and problem you are working on. &lt;/p&gt;&#xA;" OwnerUserId="23495" LastActivityDate="2017-10-31T16:14:57.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24231" PostTypeId="2" ParentId="24202" CreationDate="2017-10-31T18:17:35.417" Score="6" Body="&lt;p&gt;Let's assume that you are trying to minimize the following loss for a given task, $$\ell(y, \hat{y}) = \frac{1}{2}\| y - \hat{y}\|^2, $$ where $\|&#xA;\cdot\|$ is the Eucledian distance, and the predicted output is $\hat{y} =\hat{x}$ for simplicity. The gradients then can be computed as follow: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial b}, \hspace{20pt} \Delta \omega = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial \omega}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the gradient of $\hat{x}$ with respect to the bias $b$ is $$\frac{\partial\hat{x}}{\partial b} = \frac{\partial}{\partial b}(x-E[x]) = \frac{\partial}{\partial b}\left((u+b)-E[(u+b)]\right) = 1 - \frac{\partial}{\partial b}E[(u+b)]$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ignoring the fact $E(x)$ is dependant on $b$ makes the above gradient equals to 1, and hence keeps updating the bias as follow: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[x] = 1 -0 &#xA; = 1$$ &#xA;and then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (1), \hspace{20pt} b \leftarrow b + \Delta b $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, if you wanna consider this dependency, the gradient becomes 0, and hence no update as follow: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[u+b] =  1 -  \left(\frac{\partial}{\partial b}E[u]+\frac{\partial}{\partial b}E[b]\right) &#xA; = 1 - (0+1) = 0$$ &#xA;and then &#xA; $$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (0), \hspace{20pt} b \leftarrow b + 0 $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases, regardless bias update, the loss function will remain fixed,  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$u+(b+\Delta b)−E[u+(b+∆b)]=u+b−E[u+b], $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;however, in the first case, the bias will grow indefinitely. &lt;/p&gt;&#xA;" OwnerUserId="41240" LastActivityDate="2017-10-31T18:17:35.417" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24232" PostTypeId="1" CreationDate="2017-10-31T18:37:20.237" Score="1" ViewCount="76" Body="&lt;p&gt;I have multiple features and I want to predict three outcome scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Features:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Length in cm smallest is 40cm biggest is 209cm&lt;/li&gt;&#xA;&lt;li&gt;Kilo: 39 till 302&lt;/li&gt;&#xA;&lt;li&gt;Age: 19 till 111&lt;/li&gt;&#xA;&lt;li&gt;Gender: Male, female, transgender&lt;/li&gt;&#xA;&lt;li&gt;Diagnoses: numbers of different diseases&lt;/li&gt;&#xA;&lt;li&gt;Medicines: numbers of different medicines&lt;/li&gt;&#xA;&lt;li&gt;Urine level: 0 till 5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scores:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Scale of Happy 1 till 7&lt;/li&gt;&#xA;&lt;li&gt;Scale of depression 1 till 7&lt;/li&gt;&#xA;&lt;li&gt;Scale of health 1 till 7&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I know I can use supervised learning and create models where I predict the scores individually. I already pre-trained three different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have some feedback? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Happiness, depression and health are  related. Are there algorithms that can deal with this? Based on historical data (the features and predictions) and the trained model I want to predict the scores for new patients based on their features and historical scores.&lt;/p&gt;&#xA;" OwnerUserId="31990" LastEditorUserId="23305" LastEditDate="2017-10-31T20:10:53.677" LastActivityDate="2018-05-15T16:12:42.103" Title="Predicting three related scores with one joint model" Tags="|machine-learning|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24233" PostTypeId="2" ParentId="22556" CreationDate="2017-10-31T19:55:16.177" Score="2" Body="&lt;p&gt;Please clarify what you are looking for in the presence of cycles.  I'd assume the cycles aren't standalone- any node in a cycle could also have inputs from further upstream, and could also send output further downstream. Even in the unweighted case, it seems to me that some nodes in a cycle can be more &quot;upstream&quot; than others.  For instance, most of the nodes in the cycle might be downstream of a set of source nodes.  One node in that cycle might be the only path out of the cycle to a sink.  Surely that node is more &quot;downstream&quot; than the others in the cycle?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry the above isn't just a comment - I'm too new to have high enough reputation in this group for that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to give an answer as well.  Take the &quot;flow network&quot; analogy to an extreme.  Imagine each source upstream of the target node puts out a water flow, with dye added at the first instant only. Treat the weights as time delays.  For each node, calculate the first time the dye reaches a given node.  Among all the sinks downstream of your target node, pick either the longest time or the mean time (or itself, if the node is a sink).  The target node's position is then just its early arrival time divided by the sinks' mean or longest arrival time.  Computationally, this earliest dye arrival time is done by some modification of breadth first traversal of the directed graph.  Start at the sources with arrival times of 0, and then each node arrival time is just the minimum over all its predecessors.  (As in any graph traversal, visited and completed nodes have to be remembered so cycles aren't followed more than once. This would also be called shortest distance from a set of nodes to a given set of nodes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that cycles aren't given any particular special treatment.   If you really want cycles to matter, you could assume that output flows are split equally, and do mixing calculations, so that the dye concentration would decay gradually over time.  But that would seem ridiculously complex, and take time, for probably no benefit.  But in either case, the nodes in cycles won't all have the same value - some will be more &quot;upstream&quot; than others.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want all the nodes in a cycle to have the same value, you might need a separate pre-processing step to find the directed cycles, and aggregate each cycle into a single node (one for each cycle).  That adds complexity, although it simplifies the shortest distance calculations since then the graph will be acyclic and a trivial breadth first traversal will work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You probably don't really want to require a global analysis of a graph to get your feature value, although that's what you had to do in your initial analysis too.  If this is something you would run on a regular basis, you could probably just do local updating when changes occur, with limited propagation of earliest arrival times, assuming you could tolerate some &quot;error&quot; in the somewhat arbitrarily-chosen metric anyway.  &lt;/p&gt;&#xA;" OwnerUserId="40457" LastEditorUserId="40457" LastEditDate="2017-10-31T20:54:06.947" LastActivityDate="2017-10-31T20:54:06.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24236" PostTypeId="2" ParentId="24232" CreationDate="2017-10-31T22:28:03.433" Score="1" Body="&lt;p&gt;Do you have a labeled set? Any algorithm can do it in a supervised manner, as long as you have enough labeled data to train it on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A tree-based model (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forests Classifier&lt;/a&gt; for instance) could do really well on this task, especially since you have categorical data. All you need to do is encode the data as numbers (for instance 0: male, 1: female, 2: transgender) and feed it into your model (applying random splits and cross-validation and all that). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you don't, then the best you can do &lt;em&gt;without&lt;/em&gt; collecting new data is to find clusters of related patients in your dataset. The same rules for encoding variables apply, although you may have to pay some attention to how you scale your variables since clustering algorithms are sensitive to relative scaling of features. This happens because most clustering algorithms cluster data using distance metrics which lose meaning when you are dealing with categories. See &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;this&lt;/a&gt; for more info on clustering categorical data: &lt;/p&gt;&#xA;" OwnerUserId="23495" LastActivityDate="2017-10-31T22:28:03.433" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24238" PostTypeId="1" CreationDate="2017-10-31T23:37:02.427" Score="3" ViewCount="1343" Body="&lt;p&gt;I have a large sparse data matrix (bag of words, over large number of entries). I can easily treat it as a sparse matrix in &lt;code&gt;sklearn&lt;/code&gt; models such as &lt;code&gt;RandomForest&lt;/code&gt;. But, if I want to use &lt;code&gt;Catboost&lt;/code&gt;, I need to turn it into a dense matrix. I was wondering if there is any efficient method to work with Catboost that doesn't cause this? For example, any internal built-in feature such as &lt;code&gt;TFRecords&lt;/code&gt; of &lt;code&gt;Tensorflow&lt;/code&gt;, to load bacthes.&lt;/p&gt;&#xA;" OwnerUserId="31464" LastEditorUserId="29575" LastEditDate="2017-11-01T00:31:30.127" LastActivityDate="2024-01-04T11:00:55.087" Title="Large sparse dataset in Catboost" Tags="|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24239" PostTypeId="1" CreationDate="2017-10-31T23:43:09.730" Score="5" ViewCount="1010" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/pdf/1502.03167.pdf&quot; rel=&quot;noreferrer&quot;&gt;batch  normalization&lt;/a&gt; the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i - \mu_B)^2$&lt;br&gt;&#xA;where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and during the test time we calculate the population statistics for the same as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[x] = E_B[\mu_B]$ and $Var[x] = \frac{m}{m-1}E_B[\sigma_B^2]$ (as you can see the unbiased estimate of population variance is calculated during the test time as we are building the model to predict the population distribution)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it not seem more logical that bias correction be done during training phase itself and subsequently its expected value can be used without the $\frac {m}{m-1}$ correction factor. i.e&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During training:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_B^2 = \frac 1{m-1} \sum_{i=1}^{m} (x_i - \mu_B)^2$&lt;br&gt;&#xA;where $\mu_B = \frac 1m \sum_{i = 1}^{m}(x_i)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and during test time:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[x] = E_B[\mu_B]$ and $Var[x] = E_B[\sigma_B^2]$&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2017-11-03T12:05:17.367" Title="Batch normalization variance calculation" Tags="|deep-learning|batch-normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24240" PostTypeId="2" ParentId="24102" CreationDate="2017-11-01T02:00:12.113" Score="1" Body="&lt;p&gt;The sigmoid function is usually used for classification tasks. Basically, it converts a continuous value into a binary variable (0 or 1) based on a certain threshold. An example is predicting breast tumor as malignant (1) or benign (0) based on whether or not the output is greater than the threshold value (typically, 0.5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use ANN for regression, first you'll have to change your cost function into something more appropriate, like mean squared error (MSE). Then adapt the network to provide continuous output, by using a linear activation function, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://www.quora.com/Can-Deep-Learning-and-Neural-Networks-be-useful-for-regression-problems-where-the-output-variable-has-an-unknown-or-varying-in-real-time-upper-bound&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/Can-Deep-Learning-and-Neural-Networks-be-useful-for-regression-problems-where-the-output-variable-has-an-unknown-or-varying-in-real-time-upper-bound&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41254" LastEditorUserId="41254" LastEditDate="2017-11-01T06:14:16.417" LastActivityDate="2017-11-01T06:14:16.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24241" PostTypeId="1" AcceptedAnswerId="24250" CreationDate="2017-11-01T06:02:36.407" Score="-1" ViewCount="1672" Body="&lt;p&gt;Recently, I want to understand word2vec.&#xA;I know there are two algorithm behind word2vec.&#xA;One is CBOW another is Skip grams model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is question, is CBOW also have windows size like skip-gram model and will do iteration of corpus?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, &quot;I am eating pizza now&quot;, let say windows size is 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In CBOW, feature will be &quot;I, am, pizza, now&quot;, label will be &quot;eating&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it will also do &quot;I&quot; as label, &quot;am, eating&quot; as feature and so on in iteration?&lt;/p&gt;&#xA;" OwnerUserId="41260" LastActivityDate="2017-11-01T09:37:18.233" Title="Word2vec continuous bag of words and skip grams model" Tags="|machine-learning|neural-network|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24242" PostTypeId="1" CreationDate="2017-11-01T06:20:10.010" Score="1" ViewCount="36" Body="&lt;p&gt;df.features is like a list of features in each row.&#xA;First I set all those encoded to 0, and then did this:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for index, row in df.features.iteritems():&#xA;if type(row) == str:&#xA;    if 'Academia' in row:&#xA;        df.academia.iloc[index] = 1&#xA;    if 'Churrasqueira' in row:&#xA;        df.churrasqueira.iloc[index] = 1&#xA;    if 'Cinema' in row:&#xA;        df.cinema.iloc[index] = 1&#xA;    if 'Espaço gourmet' in row:&#xA;        df.espaco_gourmet.iloc[index] = 1&#xA;    if 'Espaço verde / Parque' in row:&#xA;        df.parque.iloc[index] = 1&#xA;    if 'Gramado' in row:&#xA;        df.gramado.iloc[index] = 1&#xA;    if 'Jardim' in row:&#xA;        df.jardim.iloc[index] = 1&#xA;    if 'Piscina' in row:&#xA;        df.piscina.iloc[index] = 1&#xA;    if 'Playground' in row:&#xA;        df.playground.iloc[index] = 1&#xA;    if 'Quadra de tênis' in row:&#xA;        df.quadra_tenis.iloc[index] = 1&#xA;    if 'Quadra poliesportiva' in row:&#xA;        df.quadra_poli.iloc[index] = 1&#xA;    if 'Quintal' in row:&#xA;        df.quintal.iloc[index] = 1&#xA;    if 'Salão de festas' in row:&#xA;        df.salao_festas.iloc[index] = 1&#xA;    if 'Salão de jogos' in row:&#xA;        df.salao_jogos.iloc[index] = 1&#xA;    if 'Aquecimento' in row:&#xA;        df.aquecimento.iloc[index] = 1&#xA;    if 'Ar condicionado' in row:&#xA;        df.ar_condicionado.iloc[index] = 1&#xA;    if 'Conexão à internet' in row:&#xA;        df.internet.iloc[index] = 1&#xA;    if 'Depósito' in row:&#xA;        df.deposito.iloc[index] = 1&#xA;    if 'Elevador' in row:&#xA;        df.elevador.iloc[index] = 1&#xA;    if 'Garagem' in row:&#xA;        df.garagem.iloc[index] = 1&#xA;    if 'Lareira' in row:&#xA;        df.lareira.iloc[index] = 1&#xA;    if 'Lavanderia' in row:&#xA;        df.lavanderia.iloc[index] = 1&#xA;    if 'Massagem' in row:&#xA;        df.massagem.iloc[index] = 1&#xA;    if 'Mobiliado' in row:&#xA;        df.mobiliado.iloc[index] = 1&#xA;    if 'Recepção' in row:&#xA;        df.recepcao.iloc[index] = 1&#xA;    if 'Sauna' in row:&#xA;        df.sauna.iloc[index] = 1&#xA;    if 'Spa' in row:&#xA;        df.spa.iloc[index] = 1&#xA;    if 'TV a cabo' in row:&#xA;        df.tv_cabo.iloc[index] = 1&#xA;    if 'Circuito de segurança' in row:&#xA;        df.circuito_seguranca.iloc[index] = 1&#xA;    if 'Condomínio fechado' in row:&#xA;        df.condominio_fechado.iloc[index] = 1&#xA;    if 'Interfone' in row:&#xA;        df.interfone.iloc[index] = 1&#xA;    if 'Segurança 24h' in row:&#xA;        df.seguranca_24h.iloc[index] = 1&#xA;    if 'Sistema de alarme' in row:&#xA;        df.alarme.iloc[index] = 1&#xA;    if 'Vigia' in row:&#xA;        df.vigia.iloc[index] = 1&#xA;    if 'Área de serviço' in row:&#xA;        df.area_servico.iloc[index] = 1&#xA;    if 'Cozinha' in row:&#xA;        df.cozinha.iloc[index] = 1&#xA;    if 'Escritório' in row:&#xA;        df.escritorio.iloc[index] = 1&#xA;    if 'Sacada' in row:&#xA;        df.sacada.iloc[index] = 1&#xA;    if 'Varanda gourmet' in row:&#xA;        df.varanda_gourmet.iloc[index] = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2017-11-01T08:11:37.043" Title="How would you improve and optimize a manual encoding like this?" Tags="|machine-learning|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24243" PostTypeId="1" AcceptedAnswerId="24244" CreationDate="2017-11-01T07:41:44.723" Score="3" ViewCount="125" Body="&lt;p&gt;I have the following code written using the &lt;code&gt;pandas&lt;/code&gt; library. I would like to know if there are ways to optimize the code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for column in df:&#xA; for index, row in df[column].iteritems():&#xA;    if type(row) == str:&#xA;        if 'R$' in row:&#xA;            n = row.replace('R$', '')&#xA;            n = n.replace(' ', '')&#xA;            n = n.replace('.', '')&#xA;            df[column].iloc[index] = float(n)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Just want to remove unwanted string parts.&lt;/p&gt;&#xA;" OwnerUserId="9988" LastEditorUserId="37377" LastEditDate="2017-11-01T20:29:38.740" LastActivityDate="2017-11-01T20:29:38.740" Title="How would you optimize this code?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-02T16:13:50.393" ContentLicense="CC BY-SA 3.0" />
  <row Id="24244" PostTypeId="2" ParentId="24243" CreationDate="2017-11-01T07:46:38.170" Score="2" Body="&lt;p&gt;You can replace the symbol in dataframe without iterating yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df = df.replace({'R\$': ''}, regex=True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then change the type of columns that can be numeric. If you don't know which are those columns, use this that will automatically change the type to numeric and ignore those that cannot be changed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df = df.apply(pd.to_numeric, errors='ignore')&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you use &lt;code&gt;replace&lt;/code&gt; and in general many other pandas features, it doesn't update your dataframe. It creates a new, temporary dataframe. So, you either need to assign it back to your original dataframe or use &lt;code&gt;inplace=True&lt;/code&gt; wherever it is available, like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.replace({'R\$': ''}, regex=True, inplace = True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$&lt;/code&gt; is a special character in regex, so you need to escape it. That's why the backslash before it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;dic = {'feature1': 'R$ aaa bb', 'feature2': 1} &#xA;df = pd.DataFrame(dic, index=[0,1])&#xA;&#xA;print(df)&#xA;&#xA;&amp;gt;&amp;gt;  feature1  feature2&#xA;0  R$ aaa bb         1&#xA;1  R$ aaa bb         1&#xA;&#xA;df = df.replace({'R\$': ''}, regex=True)&#xA;&#xA;print(df)&#xA;&#xA;&amp;gt;&amp;gt;  feature1  feature2&#xA;0   aaa bb         1&#xA;1   aaa bb         1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2017-11-01T09:02:50.897" LastActivityDate="2017-11-01T09:02:50.897" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24245" PostTypeId="2" ParentId="24242" CreationDate="2017-11-01T08:11:37.043" Score="2" Body="&lt;p&gt;The easiest and most efficient way to do it, is to use the One Hot Encoding &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ScikitLearn&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.preprocessing import OneHotEncoder&#xA;&amp;gt;&amp;gt;&amp;gt; enc = OneHotEncoder()&#xA;&amp;gt;&amp;gt;&amp;gt; enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])  &#xA;OneHotEncoder(categorical_features='all', dtype=&amp;lt;... 'numpy.float64'&amp;gt;,&#xA;       handle_unknown='error', n_values='auto', sparse=True)&#xA;&amp;gt;&amp;gt;&amp;gt; enc.n_values_&#xA;array([2, 3, 4])&#xA;&amp;gt;&amp;gt;&amp;gt; enc.feature_indices_&#xA;array([0, 2, 5, 9])&#xA;&amp;gt;&amp;gt;&amp;gt; enc.transform([[0, 1, 1]]).toarray()&#xA;array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-01T08:11:37.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24249" PostTypeId="1" CreationDate="2017-11-01T09:31:01.957" Score="4" ViewCount="147" Body="&lt;p&gt;I have historical data with labels and features about medical data about treating cancer. The labels represents: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How happy is the patient after the treatment (other patients did a survey in the past). From scale 1 till 7.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There features are numerical: &#xA;Via facial recognition we extracted six types emotions on a scale from zero till one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is performed for patients who are not able to talk and are in electric wheelchair with a tablet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained a random Forrest Regressor. That predicts how happy will be patient on a scale from 1 till 7. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the patient is done with the treatment the patient fills in a survey behind a screen. Each patient has a unique user ID. How would you apply personalization via reinforcement learning in order to know happy the patient will be during the next treatment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The happiness of the patients it's face should be updated if this is not correct for this user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&#xA;I want the agent to update the model for this patient it's belief (the old model that we trained) Let's say we predicted 6, but actually it was 4 next time this same patient arrives, we want to estimate the patients happiness with the new updated model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model can be correct for patient A, and incorrect for patient B. Because for patient B some emotion was more or less important compared to patient A. How would you solve this problem and it should be scalable for 2000+ patients. Patients with cancer are coming back to the hospital for treatment (on average 10 or 20 treatments). And each patient has an Unique ID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is making the prediction. Based on the predictions the doctors &quot;receive&quot; an happiness estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model makes: the wrong prediction for patient B but the correct one for patient A. Because patient B indicated after the treatment that the estimation was incorrect based on the scale scores. But Patient A indicated that his prediction was correct.&lt;/p&gt;&#xA;" OwnerUserId="31990" LastEditorUserId="31990" LastEditDate="2017-11-02T21:15:05.530" LastActivityDate="2018-05-14T20:26:34.787" Title="Reinforcement Learning different patients" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="10" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24250" PostTypeId="2" ParentId="24241" CreationDate="2017-11-01T09:37:18.233" Score="2" Body="&lt;p&gt;In CBOW you are predicting a target word from source context words.&#xA;In Skip-gram it is the inverse, given a target word, it predicts source context words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CBOW also creates vector representations for all the words apart from just &quot;eating&quot; in your example by constructing tuples for training based on the window size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So considering the window size of 2, the data would look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;([I, eating], am), ([am, pizza], eating), ([eating, now], pizza) and so on and so forth. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-11-01T09:37:18.233" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24251" PostTypeId="2" ParentId="24223" CreationDate="2017-11-01T11:32:36.280" Score="1" Body="&lt;p&gt;This is where Word Embedding play an amazing role. If you want to solve the problem you have at hand with Natural Language Processing approach, what you can do is learn vector representations for these documents and store them with the labels i.e tags for that text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use the similarity of vector representations between documents, to predict the tags. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, your computation is not intensive at all and you can keep retraining as and when you get more data. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-11-01T11:32:36.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24252" PostTypeId="2" ParentId="24239" CreationDate="2017-11-01T12:33:30.523" Score="2" Body="&lt;p&gt;Well, the only reason, I would say here, is the numerical stability. For instance, if the Batch size $m =1$ during the training, then you may end up with $NaN$ variance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; I doubled check the BN paper again, they clearly mentioned the condition that $m&amp;gt;1$. However, I found an interesting &lt;a href=&quot;https://stats.stackexchange.com/questions/207760/when-is-a-biased-estimator-preferable-to-unbiased-one?newreg=98a08fa483724e469d4e3098fc789bf3&quot;&gt;post&lt;/a&gt; discussing the cases when biased estimator preferable than the unbiased one. This might explain why the authors used the biased one during the training and the unbiased one inference. &lt;/p&gt;&#xA;" OwnerUserId="41240" LastEditorUserId="41240" LastEditDate="2017-11-01T16:37:53.477" LastActivityDate="2017-11-01T16:37:53.477" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24253" PostTypeId="1" AcceptedAnswerId="24288" CreationDate="2017-11-01T13:48:08.957" Score="1" ViewCount="59" Body="&lt;p&gt;Bioengineering data comprises of 512 binary features and a single Boolean label: if the particular mixture is worth further research. There are about 1,200,000 results of previous experiments available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every &quot;physical&quot; test costs, so it is tempting to train a model to predict if a certain combination of features is worth real testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried a few models, all failed to render better accuracy than 52% on test data, which is very close to blind guess considering binary classification. Namely, a Feed-forward Multi-layer Perceptron, Random Forest, Naive Bayes models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a plain simple XOR example from ML starter courses, for two binary inputs, all 4 cases are required to train the network.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.2 mln samples are, perhaps, still a &lt;strong&gt;too small dataset&lt;/strong&gt; to train for 512 inputs, considering a 512-bit number of all combinations and complex cross-influence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thinking from the opposite, how to prove that it is &lt;strong&gt;impossible to train&lt;/strong&gt; any model given available training data?&lt;/p&gt;&#xA;" OwnerUserId="41274" LastEditorUserId="41274" LastEditDate="2017-11-01T13:54:23.573" LastActivityDate="2017-11-02T07:02:12.993" Title="How to evaluate data capability to train a model?" Tags="|data-mining|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24255" PostTypeId="1" CreationDate="2017-11-01T15:32:49.237" Score="1" ViewCount="37" Body="&lt;p&gt;I read about analysis on local properties of neural networks. Some of them study the impact of &quot;infinitesimal&quot; change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via Influence Functions. They have neat formalization with these properties but I don't really get the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How local is local? Or maybe how small is &quot;infinitesimal&quot;. I get the idea that this is related to some derivative and stuff. But these results are applied to some changes that are not that small from my perspective, like a change of 0.1 in pixel value ranging from 0 to 1. Why is this considered correct?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From my understanding, if the derivative does not change much, it can be applied to a larger area. But is this the case in neural networks? It would be great if I can get some references on this topic.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since I don't really have much background in ml theory and serious math stuff. Other pointers that may help me understand these topics are also appreciated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you for the help in advance.&lt;/p&gt;&#xA;" OwnerUserId="27944" LastActivityDate="2017-11-01T15:32:49.237" Title="Why study properties with infinitesimal change?" Tags="|deep-learning|theory|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24257" PostTypeId="2" ParentId="24203" CreationDate="2017-11-01T16:10:19.137" Score="4" Body="&lt;p&gt;Yes. Feature scaling can &lt;em&gt;completely&lt;/em&gt; change the clustering result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People usually scale data to [0:1] or to have a standard deviation of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, that is nothing but a heuristic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many cases, the need for scaling is nothing but a &lt;strong&gt;symptom, caused by inappropriate data for the method&lt;/strong&gt;. You can't just fix this by some naive scaling, but it's just a hack that often works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For statistically meaningful results, all axes should be scaled to reflect attribute relevance, such that a difference of 1 unit is of the same importance in each attribute.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-01T16:10:19.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24258" PostTypeId="1" CreationDate="2017-11-01T16:42:07.230" Score="1" ViewCount="47" Body="&lt;p&gt;I have a classifier that predicts a class given roughly 100 datapoints. The classifier that tends to perform best on my data is &lt;strong&gt;RandomForest&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of metrics such as revenue for each of the past months, number of web traffic coming from different sources etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question, if I should calculate derived metrics as input for the model (e.g. year over year growth %, % traffic from each source etc.) or just put the features into the model as it is?&lt;/p&gt;&#xA;" OwnerUserId="40740" LastEditorUserId="40853" LastEditDate="2017-11-27T15:16:42.473" LastActivityDate="2017-11-27T15:16:42.473" Title="Need to calculate derived metrics in a classifier" Tags="|feature-selection|classifier|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24259" PostTypeId="1" CreationDate="2017-11-01T16:58:31.637" Score="1" ViewCount="102" Body="&lt;p&gt;I have implemented a prediction model.&#xA;Now I am checking if I should split the model to two context-based models (men and women).&#xA;I have created a new binary feature that is relevant to only part of the women's group and it can affect the prediction accuracy for this group.&#xA;I don’t have a lot of data so I am not eager to split to context-based models.&#xA;I know that in this cases it is best to use an interaction terms features (womenxnewfeature) but this feature is only part of the women's’ group and I have heard that adding the feature as it is it’s not good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion on what I can do?&lt;/p&gt;&#xA;" OwnerUserId="27645" LastEditorUserId="27645" LastEditDate="2017-11-02T08:38:26.297" LastActivityDate="2017-11-06T20:41:47.177" Title="context-based model and interaction terms" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24260" PostTypeId="1" AcceptedAnswerId="24265" CreationDate="2017-11-01T17:14:28.707" Score="1" ViewCount="1026" Body="&lt;p&gt;Logistic regression cost function is cross-entropy. It is defined as below: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kkzTW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kkzTW.png&quot; alt=&quot;J(θ)=1m∑i=1m−(y(i)×log(hθ(x(i)))+(1−y(i))×log(1−hθ(x(i))))&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such  as : ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to analytically find the minimum? if yes, what can we say of the computing complexity?&lt;/p&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="29575" LastEditDate="2017-12-21T15:05:49.153" LastActivityDate="2017-12-21T15:05:49.153" Title="Logistic Regression : Solving the cross-entropy cost function analytically" Tags="|machine-learning|scikit-learn|logistic-regression|cost-function|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24262" PostTypeId="2" ParentId="23376" CreationDate="2017-11-01T17:32:51.347" Score="12" Body="&lt;p&gt;You can try another Python library called &lt;a href=&quot;http://pyphen.org/&quot; rel=&quot;noreferrer&quot;&gt;Pyphen&lt;/a&gt;. It's easy to use and supports a lot of languages.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pyphen&#xA;dic = pyphen.Pyphen(lang='en')&#xA;print dic.inserted('Rohit')&#xA;&amp;gt;&amp;gt;'Ro-hit'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-01T17:32:51.347" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24263" PostTypeId="1" CreationDate="2017-11-01T18:14:55.817" Score="1" ViewCount="731" Body="&lt;p&gt;I have a dataset of 2000 documents where avg doc size is 300 words. The vocab is dominated by domain-specific words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to find similar documents. For this, I tried LDA, LSI, Doc2Vec (topics=100) but results are not great. LSI is better than the others in my dataset. I also tried word2vec (size=100) &amp;amp; word movers distance but again no luck. I am thinking of trying POS tags &amp;amp; then building some ontology model but not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are my results poor because of a bad dataset? What are some other techniques that I can try?&lt;/p&gt;&#xA;" OwnerUserId="41284" LastActivityDate="2019-12-16T19:15:01.727" Title="What are key dataset requirements for topic models and word embeddings?" Tags="|nlp|dataset|word-embeddings|topic-model|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24264" PostTypeId="1" CreationDate="2017-11-01T18:34:53.373" Score="1" ViewCount="90" Body="&lt;p&gt;I'm trying to understand the results of a classifier I used to predict two possible classes. Here is what I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     Precision     Recall     F1-score     Support&#xA;&#xA; 0      0.97        1.00        0.98        341091&#xA; 1      0.60        0.18        0.27        12629&#xA;&#xA; Avg    0.96        0.97        0.96        353720&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I'm trying to predict Class 1, does this model have any value? Can I improve the results of Class 1 prediction?&lt;/p&gt;&#xA;" OwnerUserId="41290" LastActivityDate="2020-08-11T14:52:56.383" Title="Interpreting Machine Learning Classification Metrics" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24265" PostTypeId="2" ParentId="24260" CreationDate="2017-11-01T19:08:59.273" Score="0" Body="&lt;p&gt;If you use just one neuron (linear-&gt;sigmoid), you can find the minimum with too much small error or maybe reach to the minimum using approaches like gradient descent or other optimization algorithms. The reason for finding the minimum is that cross entropy is a non-convex shape, but if you use sigmoid function as the activation function of logistic regression, the cross entropy cost function becomes convex and it is easy to find the only global minimum.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-01T19:15:34.433" LastActivityDate="2017-11-01T19:15:34.433" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24266" PostTypeId="2" ParentId="21955" CreationDate="2017-11-01T19:19:52.647" Score="3" Body="&lt;p&gt;In addition to Neil Slater's answer above, using ReLU to activate your neurons will converge it faster (allows you to train on a shorter period of time).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def feedForward(x,w,b):&#xA;    Input = tf.add(tf.matmul(x,w), b)&#xA;    Output = tf.nn.relu(Input)           # this ReLU&#xA;    return Output&#xA;&#xA;MHat = tf.nn.relu(Out2)                  # and for the prediction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sigmoid is mostly used for values ranging 0.0 to 1.0 (see &lt;a href=&quot;https://stats.stackexchange.com/a/206247/183154&quot;&gt;this&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="41034" LastActivityDate="2017-11-01T19:19:52.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24269" PostTypeId="2" ParentId="24183" CreationDate="2017-11-01T20:08:18.270" Score="0" Body="&lt;p&gt;This is the best solution I have come up with, I simply found a larger color pallet. The largest I could find was:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://bl.ocks.org/aaizemberg/78bd3dade9593896a59d&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://bl.ocks.org/aaizemberg/78bd3dade9593896a59d&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I merged &lt;code&gt;d3 catagory 20&lt;/code&gt; with the first few from &lt;code&gt;d3 category20b&lt;/code&gt; which was inspired from this heatmap tools &lt;a href=&quot;https://software.broadinstitute.org/morpheus/&quot; rel=&quot;nofollow noreferrer&quot;&gt;heatmap tool's&lt;/a&gt; source &lt;a href=&quot;https://github.com/cmap/morpheus.js/blob/331b63444284f7145fb8a2d773b54864a093250b/src/matrix/vector_color_model.js&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dm4OX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dm4OX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30754" LastActivityDate="2017-11-01T20:08:18.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24271" PostTypeId="1" AcceptedAnswerId="24278" CreationDate="2017-11-01T20:13:54.007" Score="0" ViewCount="2665" Body="&lt;p&gt;I am working on recognizing object classes in images using neuronal nets so I could make classifiers for cats, dogs... using Imagenet and some Conv nets famous architectures but my problems is if I had an image that contain a human and a cat and a car how can I detect and recognize 3 of them?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-15T22:21:49.870" LastActivityDate="2021-02-15T22:21:49.870" Title="Multiple Object recognition in image using deep Learning" Tags="|machine-learning|deep-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-02T16:16:44.490" ContentLicense="CC BY-SA 4.0" />
  <row Id="24272" PostTypeId="2" ParentId="24264" CreationDate="2017-11-01T20:18:12.010" Score="1" Body="&lt;p&gt;I assume the results you show have been evaluated according to a train-validation-test split approach. With the information you have provided, it is possible to figure your confusion matrix. It has to be something like :&lt;/p&gt;&#xA;&lt;p&gt;label=0 and pred=0 :  330.000  &lt;br/&gt;&#xA;label=0 and pred=1 :  4800 &lt;br/&gt;&#xA;label=1 and pred=0 : 11.000&lt;br/&gt;&#xA;label=1 and pred=1 : 7200&lt;/p&gt;&#xA;&lt;p&gt;According to this, your model has some value but and is better that what random choices could give you. Thinking of a random choice between the two classes given the distribution of the classes, you would get something close to :&lt;/p&gt;&#xA;&lt;p&gt;label=0 and pred=0 :  328.000  &lt;br/&gt;&#xA;label=0 and pred=1 :  12.600 &lt;br/&gt;&#xA;label=1 and pred=0 : 12.600&lt;br/&gt;&#xA;label=1 and pred=1 : 484&lt;/p&gt;&#xA;&lt;p&gt;So, I think you model has some value. Although, I think you have to confront this performance with the use-case you are working on, the features and model you are using and obviously the level of performance you think acceptable.&lt;/p&gt;&#xA;&lt;p&gt;To increase your model performance, it exists many approaches you can try. Here are some of them : &lt;br/&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try other classification algorithms (Naive Bayes, Logistic regression, SVM, tree-based technics...).&lt;/li&gt;&#xA;&lt;li&gt;Clean your input variables if there are dirty and extract insights from it. This is called features engineering.&lt;/li&gt;&#xA;&lt;li&gt;Diagnose your results and search if you are in an underfitting or an overftitting situation. Adapt your algorithm if needed.&lt;/li&gt;&#xA;&lt;li&gt;Apply technics relative to unbalanced classes. See more &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="98307" LastEditDate="2020-08-11T14:52:56.383" LastActivityDate="2020-08-11T14:52:56.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24273" PostTypeId="2" ParentId="24263" CreationDate="2017-11-01T20:23:21.207" Score="1" Body="&lt;p&gt;Similarity of documents can be done with varied approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As your documents are based on domain based words, you could employ a tfidf representation for each document and compute similarity based on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Previously, I have used word2vec representations of words and constructed document vectors by taking an average of all the vectors ( as one of the approaches ) and did a cosine similarity between these vectors. Basically an n*n similarity computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you do topic modelling, even though the idea is to find documents which have similar topic distribution, there is a prior to the approach i.e identifying number of topics. Just using randomly 100 topics without looking at the distribution of the words might lead you on a wrong path. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A very interesting approach I have recently come across is combining topic modelling and word vectors, here is the link to a blog by stitchfix : &lt;a href=&quot;http://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-11-01T20:23:21.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24275" PostTypeId="1" CreationDate="2017-11-01T21:29:26.340" Score="2" ViewCount="8050" Body="&lt;p&gt;I am trying to understand the differences between Scikit MLPClassifier and Tensorflow DNNClassifier for classification task and hoping that some experts can share a light. as far as I understand, both supports backpropagation, activation functions (inc. relu), optmizer (sgd/adam). MLPClassifier can also have deep neural networks by specifying the # of hidden layers and nodes.  The only difference between two that I can see is DNNClassifier supports GPU training while MLPClassifer does not. Outside of GPU support, are there other differences between them? Why do one want to use Tensorflow's DNNClassifier for basic deep neural network training (I am talking basic feed-forard NN here not the CNN, RNN, LSTM, etc.) than Scikit's MLPClassifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24153" LastActivityDate="2017-11-07T04:45:30.153" Title="Scikit MLPClassifier vs. Tensorflow DNNClassifier" Tags="|neural-network|scikit-learn|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24276" PostTypeId="1" CreationDate="2017-11-01T22:36:31.450" Score="2" ViewCount="72" Body="&lt;p&gt;Say I'm building a neural network to fit some classifier of some sort. To make things concrete, let's take the example of predicting housing prices using features of houses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should I do if one or two of my features consist of many more numbers than the other features, or even all other features combined? For example, say I have a few housing features: size in sqft, age, median income of location. These are 3 numbers. And then I have another feature, height of the roof for each square foot of the house (it's a bit contrived for this example of course) for which I would have actually &quot;size in sqft&quot;-numbers for this feature. So now my feature vector looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X = [1500sqft, 34 years, $54,000, 10ft, 10.1ft, 10.3ft...1497 more numbers here...] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that if I just naively put this into a neural net that the first 3 features would essentially be ignored since they only account for 3/1503 features. But they might actually be important. One try might be to simply average the &quot;height of roof&quot; feature over all of its elements to get an &quot;average height of the roof&quot; feature. That makes sense for this example, but what if sometimes I don't want to take this average?&#xA;Are there any industry practices on what I might try if I ran into a problem like this? &lt;/p&gt;&#xA;" OwnerUserId="41306" LastActivityDate="2017-12-02T01:54:49.723" Title="How to deal with features that have widely different dimensionalities" Tags="|neural-network|regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24277" PostTypeId="2" ParentId="24276" CreationDate="2017-11-01T23:00:45.943" Score="1" Body="&lt;p&gt;One solution is using the same technique that is used in sequence learning. In that technique we use from Recurrent Neural Network and LSTM module. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;we can train recurrent neural networks to learn very specific outputs for an arbitrary sequence of inputs, which is very powerful.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To know more about this, you can follow this &lt;a href=&quot;https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-sequence-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-01T23:00:45.943" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24278" PostTypeId="2" ParentId="24271" CreationDate="2017-11-01T23:08:11.147" Score="1" Body="&lt;p&gt;It falls under the multi label object detection problem. There have been lots of advancements done in this field and you should definitely look at implementations like :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Yolonet: &lt;a href=&quot;https://pjreddie.com/darknet/yolo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pjreddie.com/darknet/yolo/&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;Squeezenet: &lt;a href=&quot;http://songhan.github.io/SqueezeNet-Deep-Compression/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://songhan.github.io/SqueezeNet-Deep-Compression/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-11-01T23:08:11.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24280" PostTypeId="1" CreationDate="2017-11-02T00:12:03.920" Score="1" ViewCount="1353" Body="&lt;p&gt;I am thinking a simple question which just came to my mind.&lt;/p&gt;&#xA;&lt;p&gt;There are many people using data augmentation on their image data to train deep CNN.&lt;/p&gt;&#xA;&lt;p&gt;When I learn from Andrew Ng's DL courses, he mentioned that to train a better model, you generally need to have your train/test data come from the same distribution. Like, if you train you CNN with a lot of images of cars and human, it's (usually) not a good idea to use it to classify a cat or a dog.&lt;/p&gt;&#xA;&lt;p&gt;After the augmentation, my training data is kind of boosted, then can I still say my train/test data set comes from the same distribution? Am I violating what he said? Why? Or, why not?&lt;/p&gt;&#xA;" OwnerUserId="26125" LastEditorUserId="85045" LastEditDate="2020-12-31T12:42:23.443" LastActivityDate="2020-12-31T12:42:23.443" Title="Is data augmentation changing the train/test sets distribution?" Tags="|machine-learning|deep-learning|training|data-augmentation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24281" PostTypeId="2" ParentId="24276" CreationDate="2017-11-02T00:39:27.537" Score="1" Body="&lt;p&gt;&lt;strong&gt;PCA - Principal Component Analysis:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the best option here would be to perform some dimensionality reduction using &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;principal component analysis (PCA)&lt;/a&gt; prior to training your model and prior to predicting once you have a model. PCA will allow you to preserve a percentage of the total variance in the input data while significantly reducing the total number of dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the toy example that you present, I would expect a high degree of linear correlation between the height per square foot features and a high degree of linear orthogonality between the other features. In such a case, PCA will naturally mash the correlated height terms together into 1 or 2 features while preserving 95-99% of the total original variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only downside here, is that you will loose the physical meaning of your input features. I suppose you could decompose the eigenvectors and try to understand them, but loosing the physical meaning is usually not particularly hindering to training a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other methods, like lasso-ridge regression (you mention classification, but then provide a regression example e.g. the continuum variable of home price) or directly tracking the feature importance in a linear regression, but I prefer how PCA is able to project out linear dependence in input features, which seems particularly topical to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Give it a try and let us know how it goes. Hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="9420" LastActivityDate="2017-11-02T00:39:27.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24283" PostTypeId="1" CreationDate="2017-11-02T01:53:50.253" Score="1" ViewCount="1099" Body="&lt;p&gt;I'm currently working on the &lt;a href=&quot;https://www.kaggle.com/c/titanic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Titanic&lt;/a&gt; dataset from Kaggle. The features consist of both numerical and categorical variables and I've also engineered a few categorical variables using original features. I want to know what might be the best practice in terms of encoding the categorical variables and scaling features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ENCODING&lt;/strong&gt;&#xA;&lt;br&gt;&#xA;I know there are a few ways to encode categorical variables including:&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Ordinal Encoding&lt;/li&gt;&#xA;&lt;li&gt;One Hot Encoding&lt;/li&gt;&#xA;&lt;li&gt;Binary Encoding&lt;/li&gt;&#xA;&lt;li&gt;Hash Encoding&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know that Ordinal Encoding is undesirable sometimes as it makes the assumes that there's an ordered relationship between feature classes. I haven't gotten around to trying Binary and Hash encoding yet, and have done most of my work using One Hot Encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I run into another issue while using OHE. I have a pipeline that handles data preparation, feature selection/extraction and prediction that I run grid search on. So what happens is for different folds, the number of features generated through OHE are different as in certain folds one or more features' classes are likely being missed. I wanted to ask how this impacts performance and is there any other way I can go about OHE to avoid this? One way I was thinking is to handle it before data is split into a train and test split- would this work? Since in preprocessing like imputation the fitting is to be done using only train data and then used to transform the validation and test data, I've constructed the pipeline as such (train test split -&gt; fit using train data -&gt; transform test/val data when predicting).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat_pipeline = Pipeline([&#xA;    ('Selector', DataFrameSelector(cat_attribs)),&#xA;    ('Imputer', DataFrameImputer()),&#xA;    ('CategoryEncoder', cat_enc),&#xA;])&#xA;&#xA;num_pipeline = Pipeline([&#xA;    ('Selector', DataFrameSelector(num_attribs)),&#xA;    ('Imputer', Imputer(strategy='median')),&#xA;    ('Log', log_apply()),&#xA;    ('MinMaxScaler', MinMaxScaler()),&#xA;&#xA;])&#xA;full_pipeline = FeatureUnion([&#xA;    ('NumPipeline', num_pipeline),&#xA;    ('CatPipeline', cat_pipeline),&#xA;])&#xA;&#xA;Prepare = Pipeline([&#xA;    ('AttributesAdder', CustomAttributesAdder()),&#xA;    ('Scaler', full_pipeline),&#xA;])&#xA;&#xA;PrepSelPred = Pipeline([&#xA;    (&quot;Prepare&quot;, Prepare),&#xA;    (&quot;TopFeatureSelector&quot;, KernelPCA(n_components=3)),&#xA;    (&quot;Classifier&quot;, RandomForestClassifier()),&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SCALING&lt;/strong&gt;&#xA;&lt;br&gt;&#xA;As you can see from above I use a MinMaxScaler and normalize my numerical attributes rather than standardize them. This is so that they match the scale of the OH features. I've read that standardization is preferable for ML algorithms but also that it is inadvisable on OH vectors. What would be the best approach here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;FEATURE SELECTION/EXTRACTION&lt;/strong&gt;&#xA;&lt;br&gt;&#xA;I've tried both Feature Selection (using SelectFromKBest() and SelectFromModel()) as well as dimensionality reduction using PCA, though not together. Is one preferable to another, and do they work well in succession?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PREDICTING&lt;/strong&gt;&#xA;&lt;br&gt;&#xA;Given the nature of the problem and the kind of features I am dealing with, are there any algorithms that are better suited to the task. If I stick with OHE for example, are there algorithms that perform better when the majority of features are sparse?&lt;/p&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2017-11-02T01:53:50.253" Title="Preparing, Scaling and Selecting from a combination of numerical and categorical features" Tags="|machine-learning|python|feature-selection|feature-extraction|feature-scaling|" AnswerCount="0" CommentCount="0" ClosedDate="2017-11-02T16:13:45.547" ContentLicense="CC BY-SA 3.0" />
  <row Id="24284" PostTypeId="2" ParentId="24275" CreationDate="2017-11-02T04:01:34.457" Score="3" Body="&lt;p&gt;Maybe you want to use a more complicated activation function - i.e. leakyReLU or you want to add batch normalization. From a brief look at the API there is no option for batchnorm in SKlearn. Also it appears that you can't use dropout with MLPClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also feel more comfortable with tensorflow features - i.e. tensorboard, or you might feel more comfortable with SKlearn. To each his own.&lt;/p&gt;&#xA;" OwnerUserId="30982" LastEditorUserId="30982" LastEditDate="2017-11-02T04:12:24.030" LastActivityDate="2017-11-02T04:12:24.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24288" PostTypeId="2" ParentId="24253" CreationDate="2017-11-02T07:02:12.993" Score="2" Body="&lt;p&gt;1.2M is about 2^16.&#xA;You have 512 features plus the concept so the number of possibilities is much larger.&#xA;Therefore, you can claim that the number of samples you have is too small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though that, machine learning is done almost always in a situation that you have less samples the all permutations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/VC_dimension&quot; rel=&quot;nofollow noreferrer&quot;&gt;VC dimension&lt;/a&gt; allow estimate how well a data set fits the underlying distribution. It applies to an hypothesis class (e.g., linear functions) whose size is typically much smaller than the number of permutations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that both the number of permutations and the VC dimension refer to worse case scenarios. It is possible that on of the 512 is identical to the concept. In this case you will be able to learn though the other 511 features.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-11-02T07:02:12.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24289" PostTypeId="2" ParentId="24186" CreationDate="2017-11-02T07:48:26.087" Score="2" Body="&lt;p&gt;I assume that you are solving a &lt;em&gt;supervised classification&lt;/em&gt; problem, that is, you train your model on a labeled sample. I can think of two approaches to this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I. Classify tags, use neighbor tags for the features&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each tag you can calculate features like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(x, y) distances to the closest image&lt;/li&gt;&#xA;&lt;li&gt;(x, y) distances to the closest image &lt;em&gt;above this tag&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;number of images near this tag&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Such features could be fed into any classifier like SVM or decision tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;II Classify pairs of tags&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each tag $i$, you can consider all other tags $j$ and predict probability that $i$ is a caption of image $j$. This prediction may be based on concatenation of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;features of $i$&lt;/li&gt;&#xA;&lt;li&gt;features of $j$&lt;/li&gt;&#xA;&lt;li&gt;joint features of $i$ and $j$, like vertical and horizontal distance from $i$ to $j$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After these probabilities are predicted, you can classify $i$ as a caption indeed, if any of these probabilities exceeds some threshold. &lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-02T07:48:26.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24290" PostTypeId="1" AcceptedAnswerId="24292" CreationDate="2017-11-02T07:57:11.797" Score="1" ViewCount="2527" Body="&lt;p&gt;I am dealing with problem where i have to increase the sales by product recommendation.I only have customer data and product that they have purchased.No ratings,reviews or feedback is present.What approach fit best for my problem.&lt;/p&gt;&#xA;" OwnerUserId="41318" LastActivityDate="2017-11-02T11:11:28.133" Title="Product Recommendation based on purchase history" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-02T16:16:37.943" ContentLicense="CC BY-SA 3.0" />
  <row Id="24292" PostTypeId="2" ParentId="24290" CreationDate="2017-11-02T11:11:28.133" Score="2" Body="&lt;p&gt;Even if you don't have ratings or reviews, you can use the customer purchases to help creating your model and selecting the most appropriate one. If the customer has bought the product, you can suppose he has liked it. If he hasn't, we can suppose he has disliked it. This is an approach which is mainly used by e-shops.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find more on recommendation system selection and validation below. It is discussed how to choose the best recommender with both offline and online approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/recombee-blog/evaluating-recommender-systems-choosing-the-best-one-for-your-business-c688ab781a35&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/recombee-blog/evaluating-recommender-systems-choosing-the-best-one-for-your-business-c688ab781a35&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-11-02T11:11:28.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24296" PostTypeId="1" AcceptedAnswerId="24311" CreationDate="2017-11-02T14:21:55.763" Score="0" ViewCount="616" Body="&lt;p&gt;Given a machine reaches a broken state, there are potentially fixes that can to be applied to get the machine to run again. We'd like to know if, for the problem further defined below, with millions of datapoints, hundreds of features, and tens of labels, should we be looking at using a single deep neural network with multiple outputs, or create an ensemble of binary networks with human-selected features presumed better for their relevance? Is there a standard approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The state of the machine is captured in potentially hundreds of features, a good mix of continuous values and categorical data. We have millions of documented cases of machine states, and can identify when a machine had broke and what fixes were applied to get it running again. There are less than 40 fixes we are interested in, and we are considering a bucket for &quot;other&quot; fixes, and a bucket for &quot;unfixable&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are treating this as a multi-label classification, because it may take multiple discrete fixes (Fix-1, Fix-2...Fix-N) to get the machine up and running again. Not all features would be relevant to each fix, so the question we're wondering is whether each fix should have its own binary classification (each outputing a single value representing Fix-i or not Fix-i) network with what we think are the relevant features, or should we create one giant deep neural network with multiple labels, sigmoid over each fix (Fix-1, Fix-2...Fix-N). With the latter approach, should combinations of fixes be represented as their own label (Fix-1, Fix-2, Fix-1&amp;amp;2, Fix-3, Fix-1&amp;amp;3, Fix-2&amp;amp;3, Fix-1&amp;amp;2&amp;amp;3 ... etc).&lt;/p&gt;&#xA;" OwnerUserId="37776" LastEditorUserId="37776" LastEditDate="2017-11-02T17:06:03.093" LastActivityDate="2017-11-03T11:01:54.237" Title="Single vs Multiple deep learning networks for multi-label classification?" Tags="|machine-learning|classification|deep-learning|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24298" PostTypeId="1" CreationDate="2017-11-02T15:16:41.850" Score="0" ViewCount="2041" Body="&lt;p&gt;I'm looking for a pre-trained net recognizing vehicles, something like the Inception network for images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is impossible to find which vehicle detection algorithm would you suggest, possibly implemented in tensorflow? &lt;/p&gt;&#xA;" OwnerUserId="27767" LastEditorUserId="27767" LastEditDate="2017-11-02T15:22:18.563" LastActivityDate="2017-11-02T15:47:01.933" Title="Pre trained vehicle detection network" Tags="|image-recognition|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24299" PostTypeId="2" ParentId="24298" CreationDate="2017-11-02T15:27:37.717" Score="2" Body="&lt;p&gt;You can take a look at the YoloNet which detects Objects based on Pascal VOC 2012 dataset. &#xA;Here is the link to it : &lt;a href=&quot;https://github.com/pjreddie/darknet/wiki/YOLO:-Real-Time-Object-Detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/pjreddie/darknet/wiki/YOLO:-Real-Time-Object-Detection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to detect vehicles with this pre-trained model.&lt;/p&gt;&#xA;" OwnerUserId="1029" LastEditorUserId="1029" LastEditDate="2017-11-02T15:47:01.933" LastActivityDate="2017-11-02T15:47:01.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24300" PostTypeId="2" ParentId="24280" CreationDate="2017-11-02T15:54:40.730" Score="0" Body="&lt;p&gt;I don't know how to put this but your understanding of distribution is wrong and confusing you. Data Augmentation is a technique which helps the model generalize well. It is also useful when you have a limited amount of data and you still want to train a deep neural net. Transfer learning is the best example of that. And yes train/test should come from the same distribution. For example, and it makes sense if you put this thing in the real world too. If you are trained to speak English, would it make sense to evaluate you on how good you speak German?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does augmentation change the distribution?&lt;/strong&gt; I will ask you a question on this and you should reason that for yourself. Here is my question: If you rotate the image of a dog by 30 degrees, does it change the fact that it's a dog?&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-11-02T15:54:40.730" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24301" PostTypeId="2" ParentId="4836" CreationDate="2017-11-02T17:10:10.117" Score="3" Body="&lt;p&gt;The CLARIN-D project has collected some good pointers to tutorials for topic modeling and LDA on the &lt;a href=&quot;https://fedora.clarin-d.uni-saarland.de/hub/kwsearch/topic%20model/0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Teaching ans Learning Materials Collection (TeLeMaCo)&lt;/a&gt; site hosted by the &lt;em&gt;Universität des Saarlandes&lt;/em&gt; CLARIN centre.&lt;/p&gt;&#xA;" OwnerDisplayName="user10169" LastActivityDate="2017-11-02T17:10:10.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24302" PostTypeId="2" ParentId="24109" CreationDate="2017-11-02T18:08:53.177" Score="0" Body="&lt;p&gt;Usually, when talking about decision trees, were are referring to the &lt;a href=&quot;https://en.wikipedia.org/wiki/C4.5_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;C4.5 algorithm&lt;/a&gt;. That is, we build a &lt;strong&gt;binary&lt;/strong&gt; decision tree, where at each node we need to make a decision where we split the data using the rule &lt;strong&gt;&quot;variable &amp;lt; value&quot;&lt;/strong&gt; (if the variable is categorical, you can have an &lt;strong&gt;=&lt;/strong&gt; sign).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;How do know what the best split is?&lt;/em&gt; Simple, we test all possible combinations of &lt;strong&gt;variable&lt;/strong&gt; and of &lt;strong&gt;value&lt;/strong&gt;, and then choose the split which splits the data in the most egualitarian fashion (closest to 50-50).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;How do we know if the data is being well split?&lt;/em&gt; We try to minimize metrics such as entropy or maximize metrics such as the Gini coefficient. You want to reduce the original entropy down to zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;When do you stop?&lt;/em&gt; You either stop when you have a single sample in a node or when all samples have the same value. Alternatively, you can define a maximum depth for the decision tree (this is known as pre-pruning) or try to reduce your decision tree afterwards (this is known as post-pruning).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Why are decision trees so fast to create if they test all combinations of variables and values?&lt;/em&gt; Well, I lied to you. They don't test all combinations. The metrics used (entropy or gini) can be computed incrementally, so what the algorithms do is to sort the values for each variable, and then incrementally see whether including one more sample improves or reduces the score. But this is a technicality you need not be aware of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Anything more?&lt;/em&gt; You should keep in mind that decision trees do one-ahead optimization. They do not find the global best decision tree. They are myopic. Therefore, if you think two variables are correlated, you should feature engineer a new variable that uses both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;My decision tree is highly variable?&lt;/em&gt; Decision trees are know to be very different if the same changes a little bit. This is why people build ensembles of decision trees like random forests by resampling the data to make them stronger. You do lose interpretability: you can no longer draw a decision tree, if you use many of them. See &lt;a href=&quot;https://datascience.stackexchange.com/questions/23789/why-do-we-need-xgboost-and-random-forest/23913#23913&quot;&gt;this question&lt;/a&gt; for more information on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I recommend these resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.r2d3.us/visual-intro-to-machine-learning-part-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Visual presentation/animation&lt;/a&gt; about decision trees&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit-learn decision trees&lt;/a&gt; documentation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2017-11-02T18:08:53.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24303" PostTypeId="1" CreationDate="2017-11-02T18:30:08.770" Score="2" ViewCount="4296" Body="&lt;p&gt;I have a dataset with following specifications:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training dataset with 52968 samples with 8562 positives&lt;/li&gt;&#xA;&lt;li&gt;Test Dataset with 13242 samples with 2135 positives&lt;/li&gt;&#xA;&lt;li&gt;There are 137 features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;imp = Imputer(strategy=&quot;most_frequent&quot;, axis=0)&#xA;var_thr = VarianceThreshold(threshold=1.7)&#xA;pca = RandomizedPCA(n_components=16)&#xA;clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)&#xA;&#xA;return Pipeline(steps=[('imp', imp),&#xA;                       ('var_thr', var_thr),&#xA;                       ('pca', pca),&#xA;                       ('clf', clf)&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also tried to increase training data with positive results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;series = y_train[y_train==1]&#xA;dupli = x_train.loc[series.index.tolist(), :]&#xA;for _ in range(5):&#xA;    x_train = x_train.append(dupli)&#xA;    y_train = y_train.append(series)&#xA;&#xA;return x_train, y_train&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After fitting my model, the score result for my test data is 0.9954, and the cross validation is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_val_score(clf, x_train, y_train, cv=5)&#xA;[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Classification report for training data is perfect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             precision    recall  f1-score   support&#xA;&#xA;          0       1.00      1.00      1.00     44406&#xA;          1       1.00      1.00      1.00     42810&#xA;&#xA;avg / total       1.00      1.00      1.00     87216&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The confusion matrix is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[44203   203]&#xA; [  190 42620]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but the test data is much worse:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             precision    recall  f1-score   support&#xA;&#xA;          0       0.85      0.85      0.85     11107&#xA;          1       0.21      0.21      0.21      2135&#xA;&#xA;avg / total       0.75      0.75      0.75     13242&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Confusion matrix is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[9428 1679]&#xA; [1687  448]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used GridSearchCV for the threshold, n_components, max_features and max_depth. &#xA;How can I improve my model and obtain better prediction? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT -------&gt;&#xA;I changed clf in pipeline. I used RandomForestClassifier.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = RandomForestClassifier(&#xA;    n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15, &#xA;    random_state=1&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now cross validation is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Classification report for training data with confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score train result: 0.8514148780040359&#xA;             precision    recall  f1-score   support&#xA;&#xA;          0       0.86      0.85      0.85     44406&#xA;          1       0.85      0.85      0.85     42810&#xA;&#xA;avg / total       0.85      0.85      0.85     87216&#xA;&#xA;[[37757  6649]&#xA; [ 6310 36500]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Classification report for test data with confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score test result: 0.7341791270200876&#xA;             precision    recall  f1-score   support&#xA;&#xA;          0       0.89      0.79      0.83     11107&#xA;          1       0.30      0.47      0.36      2135&#xA;&#xA;avg / total       0.79      0.73      0.76     13242&#xA;&#xA;[[8719 2388]&#xA; [1132 1003]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It looks better but I search model with + 0.90 recall for traning and test data sets.&lt;/p&gt;&#xA;" OwnerUserId="39590" LastEditorUserId="39590" LastEditDate="2017-11-10T11:36:59.407" LastActivityDate="2017-11-10T11:36:59.407" Title="Decision tree classifier: possible overfitting" Tags="|python|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24304" PostTypeId="1" CreationDate="2017-11-02T20:19:58.583" Score="2" ViewCount="67" Body="&lt;p&gt;Recently I got interested in the process of data cleansing and specifically in record linkage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some lesser degree also about machine learning methods. It struck me that the key part of all algorithm basically introduce a metric space. Through the metric space every two data points can be assigned a distance. The distance is then basically a measure of how close these two data points are related to another. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I do wonder, if there were not also different kinds of algorithms that do not work on this principle? &lt;/p&gt;&#xA;" OwnerUserId="41344" LastEditorUserId="41344" LastEditDate="2017-11-02T21:01:21.617" LastActivityDate="2020-12-29T14:15:10.983" Title="Are there deduplication algorithms that do not work on a metric space?" Tags="|dataset|bigdata|algorithms|data-cleaning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24310" PostTypeId="1" CreationDate="2017-11-03T09:06:26.267" Score="0" ViewCount="913" Body="&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using natural language processing engines such as IBM's Watson Conversation and Microsoft's LUIS, which take a sentence and classify its intent. For example, &quot;i want to buy food&quot; =&gt; &quot;Buy_Food&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to come up with the best set of utterances to train those engines to understand natural language for a given application. My goal is to improve the accuracy of the NLP engine predicting the correct intent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a few thousand utterances, each with a known intent. I need to come up with a set of just a few hundred best utterances that will be used to create a model in the NLP engine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I tried&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first I tried to take a few hundred sentences randomly, create an NLP engine model, and test its accuracy using a few hundred other sentences from my set. Then I would replace one sentence in the NLP engine model, and re-run the test again. At first, Microsoft's LUIS would guess the intent correctly around 90-95% of the time, and after a few dozen iterations the accuracy goes up to 91-96% - very slightly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also thinking about using part-of-speech tagging to not just replace one sentence randomly, but to strive for syntactical diversity of those sentences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reason behind doing this&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason why I am doing all of this is that it seems that dumping more utterance examples into the NLP engine doesn't seem to make it better; in fact, it seems to make it worse. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why I am trying to think of a way to create a limited set of utterances that will be used to train the NLP model and improve the rate of how often the engine guesses the correct intent.&lt;/p&gt;&#xA;" OwnerDisplayName="user41366" LastActivityDate="2018-04-26T15:23:59.217" Title="Creating the optimal set of utterances to train a natural language processing engine" Tags="|nlp|dataset|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24311" PostTypeId="2" ParentId="24296" CreationDate="2017-11-03T11:01:54.237" Score="1" Body="&lt;p&gt;This kind of setup is normally addressed with a single neural network that has an output sigmoid layer of $N$ units (where $N$ is the number of possible fixes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having combinations of fixes represented with their own label does not make much sense; instead, let the different labels take the expected value: if 2 fixes are fine, let both expected outputs be 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having $N$ networks each one generating a single fix-nofix is equivalent but does not benefit from the potential gains of the &lt;em&gt;multitask&lt;/em&gt; learning obtained when training everything together and obtaining a shared internal representation.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-11-03T11:01:54.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24312" PostTypeId="1" AcceptedAnswerId="24325" CreationDate="2017-11-03T11:03:34.100" Score="3" ViewCount="314" Body="&lt;p&gt;In many cases an activation function is notated as &lt;code&gt;g&lt;/code&gt; (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as sigmoid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, where does this convention come from? And for what reason did &lt;code&gt;g&lt;/code&gt; start to be used?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-11-06T18:54:18.673" Title="Why is an activation function notated as &quot;g&quot;?" Tags="|deep-learning|notation|activation-function|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24314" PostTypeId="2" ParentId="24239" CreationDate="2017-11-03T12:05:17.367" Score="2" Body="&lt;p&gt;@shimao has answered it &lt;a href=&quot;https://stats.stackexchange.com/questions/311074/batch-normalization-variance-calculation/311679#311679&quot;&gt;Here&lt;/a&gt; on stats stackexchange&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To quote shimao&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Yes, it would be more logical, although equivalent both ways. I was able to find a bit of discussion on this &lt;a href=&quot;https://github.com/pytorch/pytorch/issues/1410&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like the original authors later implied that both training and test time should use the unbiased variance. Also the actual implementation in many libraries is not completely consistent with what is described in the paper.&quot;&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2017-11-03T12:05:17.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24315" PostTypeId="1" AcceptedAnswerId="24361" CreationDate="2017-11-03T12:22:02.330" Score="1" ViewCount="1246" Body="&lt;p&gt;I have a classifier with a heavily imbalanced dataset (1000 of each negative label for each positive.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm running a GradientBoostingClassifier with moderate success (AUC .75) but the curve has this strange look:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8E0y9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8E0y9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any good ideas on what would cause the curve to have this behaviour?&lt;/p&gt;&#xA;" OwnerUserId="40740" LastActivityDate="2017-11-04T21:39:29.223" Title="ROC curve shows strange results for imbalanced dataset" Tags="|scikit-learn|classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24317" PostTypeId="5" CreationDate="2017-11-03T12:41:17.537" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-11-03T12:41:17.537" LastActivityDate="2017-11-03T12:41:17.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24318" PostTypeId="4" CreationDate="2017-11-03T12:41:17.537" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-11-03T12:41:17.537" LastActivityDate="2017-11-03T12:41:17.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24319" PostTypeId="1" AcceptedAnswerId="24321" CreationDate="2017-11-03T12:41:54.203" Score="41" ViewCount="53217" Body="&lt;p&gt;I want to train a deep model with a large amount of training data, but my desktop does not have that power to train such a deep model with these abundant data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to know whether there are any free cloud services that can be used for training machine learning and deep learning models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also would like to know if there is a cloud service, where I would be able to track the training results, and the training would continue even if I am not connected to the cloud.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-28T16:06:13.927" LastActivityDate="2020-09-15T13:33:19.023" Title="Are there free cloud services to train machine learning models?" Tags="|machine-learning|neural-network|deep-learning|cloud-computing|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24320" PostTypeId="1" CreationDate="2017-11-03T13:08:48.567" Score="1" ViewCount="553" Body="&lt;p&gt;I have time-series data of a few metrics. I know which metric is the response variable and the independent ones. I need to fit a model between them. The relationship could be linear, quadratic, logarithmic, piecewise linear, multiple linear, etc. Basically, it could be anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any technique / properties I could use to find the relationship between the metrics and fit a model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now, I have written a brute-force script in R. &lt;br&gt;&#xA;For example, I have response variable A which depends on X1, X2 and X2.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A = C1*f1(X1)+C2*f2(X2)+C3*f3(X3) is my model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My script tries all possible combinations of f1, f2 and f3.&lt;br&gt;&#xA;By combinations, I mean that I initialy start with all linear, then one of them is quadratic, then cubic, then logarithmic, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using lm().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then choose the model which has the least AIC as my final model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This obviously takes too long.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I definitely need an automated way of finding this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you suggest a better way to do this?&lt;/p&gt;&#xA;" OwnerUserId="40584" LastActivityDate="2017-11-04T04:50:50.287" Title="Find type of relation between variables" Tags="|machine-learning|r|data-mining|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24321" PostTypeId="2" ParentId="24319" CreationDate="2017-11-03T13:17:28.973" Score="25" Body="&lt;p&gt;There are no &lt;em&gt;unlimited&lt;/em&gt; free services*, but some have starting credit or free offers on initial signup. Here are some suggested to date:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;AWS: If specifically deep learning on a large data set, then probably AWS is out - their free offer does not cover machines with enough processing power to tackle deep learning projects. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Google Cloud might do, the starting credit offer is good enough to do a little deep learning (for maybe a couple of weeks), although they have signup and tax restrictions. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Azure have a free tier with limited processing and storage options.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Most free offerings appear to follow the &quot;Freemium&quot; model - give you limited service that you can learn to use and maybe like. However not enough to use heavily (for e.g. training an image recogniser or NLP model from scratch) unless you are willing to pay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This best advice is to shop around for a best starting offer and best price. A review of services is not suitable here, as it will get out of date quickly and not a good use of Stack Exchange. But you can find &lt;a href=&quot;https://www.quora.com/What-is-best-cloud-solution-for-deep-learning&quot; rel=&quot;noreferrer&quot;&gt;similar questions on Quora&lt;/a&gt; and other sites - your best bet is to do a web search for &quot;cloud compute services for deep learning&quot; or similar and expect to spend some time comparing notes. A few specialist deep learning services have popped up recently such as &lt;a href=&quot;https://www.nimbix.net/&quot; rel=&quot;noreferrer&quot;&gt;Nimbix&lt;/a&gt; or &lt;a href=&quot;https://www.floydhub.com/&quot; rel=&quot;noreferrer&quot;&gt;FloydHub&lt;/a&gt;, and there are also the big players such as Azure, AWS, Google Cloud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You won't find anything completely free and unencumbered, and if you want to do this routinely and have time to build and maintain hardware then it is cheaper to buy your own equipment in the long run - at least at a personal level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To decide whether to pay for cloud or build your own, then consider a typical price for a cloud machine suitable for performing deep learning at around \$1 per hour (prices do vary a lot though, and it is worth shopping around, if only to find a spec that matches your problem). There may be additional fees for storage and data transfer. Compare that to pre-built deep learning machines costing from \$2000, or &lt;a href=&quot;https://medium.com/towards-data-science/build-a-deep-learning-pc-for-1-000-cad-f3c5f26ba134&quot; rel=&quot;noreferrer&quot;&gt;building your own for \$1000&lt;/a&gt; - such machines might not be 100% comparable, but if you are working by yourself then the payback point is going to be after only a few months use. Although don't forget the electricity costs - a powerful machine can draw 0.5kW whilst being heavily used, so this adds up to more than you might expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantages of cloud computing are that someone else does the maintenance work and takes on the risk of hardware failure. These are valuable services, and priced accordingly. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* But see &lt;a href=&quot;https://datascience.stackexchange.com/a/27085&quot;&gt;Jay Speidall's answer&lt;/a&gt; about Google's colab service, which appears to be free to use, but may have some T&amp;amp;C limitations which may affect you (for instance I doubt they will be happy for you to run content production of Deep Dream or Style Transfer on it)&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-04-02T11:59:48.707" LastActivityDate="2018-04-02T11:59:48.707" CommentCount="6" CommunityOwnedDate="2018-04-02T11:59:48.707" ContentLicense="CC BY-SA 3.0" />
  <row Id="24322" PostTypeId="1" CreationDate="2017-11-03T14:36:08.770" Score="1" ViewCount="29" Body="&lt;p&gt;I am trying to understand how to apply the kernelize the logistic regression is there a step by step tutorial with simple math to do so &lt;/p&gt;&#xA;" OwnerUserId="41297" LastActivityDate="2017-11-03T14:36:08.770" Title="Kernal logestic regression" Tags="|machine-learning|logistic-regression|" AnswerCount="0" CommentCount="1" ClosedDate="2017-11-05T09:16:06.063" ContentLicense="CC BY-SA 3.0" />
  <row Id="24323" PostTypeId="1" CreationDate="2017-11-03T15:05:54.343" Score="3" ViewCount="82" Body="&lt;p&gt;I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?&lt;/p&gt;&#xA;" OwnerUserId="31948" LastActivityDate="2023-10-15T19:04:25.490" Title="Restrictions on my skewed validation data" Tags="|machine-learning|classification|cross-validation|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24324" PostTypeId="1" AcceptedAnswerId="24333" CreationDate="2017-11-03T15:50:50.490" Score="3" ViewCount="4870" Body="&lt;p&gt;l have a dataset of images with their labels.&#xA;l put them into a k-means algorithm (as a feature extractor).&#xA;Now, l would like to use this new representation of images (features extracted from k-means algorithm) as SVM classifier inputs.&#xA;How can l do that ? &#xA;Number of cluster k=400 and numbers of images=1000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, l just have the vectors of centroids (400 centroids)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;l need to get the representation for each image with respect to the centroids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;package update &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import mixture&#xA;gmm = mixture.GMM(n_components=6).fit(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now l would like run k-means with different k=range(50,500), how can l get the distances for each k ?&#xA;Is is correct to do the following : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;K=range(50,500)&#xA;KM=[KMeans(n_clusters=k).fit(X) for k in K]&#xA;distances = [np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in C.cluster_centers_]) for C in KM]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41385" LastEditorUserId="41385" LastEditDate="2017-11-09T10:24:40.477" LastActivityDate="2017-11-09T10:24:40.477" Title="How to use k-means outputs (extracted features) as SVM inputs?" Tags="|svm|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24325" PostTypeId="2" ParentId="24312" CreationDate="2017-11-03T16:03:04.660" Score="4" Body="&lt;p&gt;The addition of the activation layer creates a &lt;a href=&quot;https://en.wikipedia.org/wiki/Function_composition&quot; rel=&quot;nofollow noreferrer&quot;&gt;composition of two functions&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;A &lt;a href=&quot;https://en.wikipedia.org/wiki/Function_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;general function&lt;/a&gt;, to be defined for a particular context, is usually denoted by a single letter, most often the lower-case letters f, g, h.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it comes down to the reason that he uses the &lt;a href=&quot;http://cs229.stanford.edu/notes/cs229-notes1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;hypothesis representation&lt;/a&gt; h(x)=wX+b which is a function, and that is wrapped by an activation function denoted as g. The choice of g seems to be purely alphabetical.&lt;/p&gt;&#xA;" OwnerUserId="37776" LastEditorUserId="37776" LastEditDate="2017-11-06T18:54:18.673" LastActivityDate="2017-11-06T18:54:18.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24327" PostTypeId="1" AcceptedAnswerId="24357" CreationDate="2017-11-03T16:56:19.410" Score="3" ViewCount="442" Body="&lt;p&gt;I would like to implement a convolutional autoencoder in Tensorflow, but it is not clear how the decoder part should work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each layer of the encoding, is a convolutional layer with activation function and then a pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how will the decoding work? I know that I have to add padding in each layer, but what will be the reverse of the convolution? How will it reproduce the original data from much less variables and the padding?&lt;/p&gt;&#xA;" OwnerUserId="15024" LastActivityDate="2018-02-14T23:25:33.060" Title="How to implement a convolutional autoencoder?" Tags="|neural-network|tensorflow|convolution|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24328" PostTypeId="1" CreationDate="2017-11-03T17:11:27.323" Score="0" ViewCount="46" Body="&lt;p&gt;I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;User1: {&#xA;   interests: {&quot;golf&quot;: 3, &quot;baseball: 1&quot;, &quot;tenis: 2&quot;}&#xA;}&#xA; User2: {&#xA;   interests: {&quot;golf&quot;: 5, &quot;baseball&quot;: 1, &quot;football&quot;: 4}&#xA;}&#xA; User3: {&#xA;   interests: {&quot;golf&quot;: 1, &quot;baseball&quot;: 1, &quot;tenis&quot;: 1}&#xA;}&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Comparing &lt;code&gt;User1&lt;/code&gt; &amp;amp; &lt;code&gt;User2&lt;/code&gt;:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;They have in common 2 interests, but &lt;code&gt;User2&lt;/code&gt; seems to be a very fan of &lt;em&gt;golf&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Comparing &lt;code&gt;User1&lt;/code&gt; &amp;amp; &lt;code&gt;User2&lt;/code&gt;:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;They have in common 3 interests, while &lt;code&gt;User3&lt;/code&gt; seems to have a slightly interest in all common interests, not being a very fan of any interests listed.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;What I want do to is&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The relationship between User1 and User2 should be a higher number than the relationship between User1 and User3, knowing that User3 has only a superficial interest in the topics, and User2 is more likely to be a fan like User1 to &lt;code&gt;golf&lt;/code&gt;, for example, meaning that they may want to know each other and etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. &lt;code&gt;User1&amp;amp;User2  = 2.5; User1&amp;amp;User3 = 1.8&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use the mean of the intersection of all interests between User-k and User-y and it seemed a reasonable approach, but I think it can have a better way to do this. Suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="41388" LastActivityDate="2017-11-03T18:34:59.187" Title="How to measure a relationship between interests of topics between two users?" Tags="|regression|data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24330" PostTypeId="1" AcceptedAnswerId="24548" CreationDate="2017-11-03T17:53:56.377" Score="1" ViewCount="697" Body="&lt;p&gt;When we create an RNN in Keras, does it learn an initial hidden layer (ie. $h_0$, like a bias term)? I'm having trouble figuring this out, but I have looked &lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone highlight a part in the code that shows an initial hidden layer is being learned? Or is it just randomly initializing the hidden layer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: The code is &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; , I'm just having trouble identifying where the hidden layer is initialized and what it is initialized to.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2017-11-04T15:37:24.197" LastActivityDate="2017-11-10T02:02:55.897" Title="Initializing a recurrent neural network (RNN) in Keras" Tags="|keras|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24331" PostTypeId="2" ParentId="10844" CreationDate="2017-11-03T17:58:37.833" Score="2" Body="&lt;p&gt;I have a repository that could help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ArthurFortes/Datasets-for-Recommneder-Systems/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ArthurFortes/Datasets-for-Recommneder-Systems/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41389" LastActivityDate="2017-11-03T17:58:37.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24332" PostTypeId="2" ParentId="24328" CreationDate="2017-11-03T18:34:59.187" Score="1" Body="&lt;p&gt;You can create a sparse user-interest matrix, impute missing values with some reasonable value, and apply a similarity function. E.g., center all non-missing values on 0 and mutate each value to adjust for the user's and interest's mean. Then impute 0s for all missing data, and use a traditional similarity function like cosine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also use Bayesian methods. You could model the relationship between users and interests. In this way you wouldn't have to impute missing data and it would account for confidence in inference and prediction.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-03T18:34:59.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24333" PostTypeId="2" ParentId="24324" CreationDate="2017-11-03T21:02:34.753" Score="7" Body="&lt;p&gt;'Prediction' of k-means algorithm for each observation is just the corresponding centroid. So you can take vector of predicted centroids and use it as a categorical feature (maybe one-hot encoded).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it is just one feature. With little coding you can do better. For example, you can find for each sample its distance to each of $k$ cluster center, and so create $k$ new features. A Python example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_iris&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.svm import SVC&#xA;import numpy as np&#xA;iris =  load_iris()&#xA;X = iris['data']&#xA;y = iris['target']&#xA;kmeans = KMeans(n_clusters=6).fit(X)&#xA;distances = np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in kmeans.cluster_centers_])&#xA;svm = SVC().fit(distances, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another (and maybe simpler way) is to fit a gaussian mixture model (e.g. by &lt;a href=&quot;http://scikit-learn.org/stable/modules/mixture.html&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt;). It is similar to k-means, but for each observation produces a probability distribution over clusters, instead of a single cluster label. These vectors of predicted cluster probabilities may be used as features as well.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.mixture import GaussianMixture&#xA;gmm = GaussianMixture(n_components=6).fit(X)&#xA;proba = gmm.predict_proba(X)&#xA;svm2 = SVC().fit(proba, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-03T21:02:34.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24334" PostTypeId="2" ParentId="24323" CreationDate="2017-11-03T21:13:02.760" Score="1" Body="&lt;p&gt;I suggest using your whole validation set but providing class-specific metrics. E.g., AUC only considering validation rows with targets in a single class. That will help you see if you're underperforming for a specific class. You can then say &quot;This model earned 0.xxx AUC on the whole validation set and no worse than 0.yyy for any single class.&quot;&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-03T21:13:02.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24335" PostTypeId="1" AcceptedAnswerId="24341" CreationDate="2017-11-03T21:19:00.647" Score="8" ViewCount="7292" Body="&lt;p&gt;Can some one please explain me what is the difference between one class SVM and SVDD(support vector data description)&lt;/p&gt;&#xA;" OwnerUserId="41297" LastActivityDate="2022-06-23T11:15:07.400" Title="SVDD vs once Class SVM" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24336" PostTypeId="2" ParentId="24303" CreationDate="2017-11-03T21:20:28.130" Score="1" Body="&lt;p&gt;Perfect training AUC is the hallmark of overfitting. When searching parameters, I often use a combination of test set AUC and the difference between training AUC and test AUC. When just using test set AUC in the search loss function, I found the model to still be pretty sensitive to which random set of test rows it sampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also agree with the commenters that you should get better results with &lt;em&gt;much&lt;/em&gt; smaller max_depth, probably on the order of 1-10, and ensemble models. Random forests are an excellent place to start.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-03T21:20:28.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24337" PostTypeId="2" ParentId="24320" CreationDate="2017-11-03T21:26:09.120" Score="1" Body="&lt;p&gt;How about using neural networks instead of linear models? The neural net will have an easier time learning arbitrary non-linear relationships.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-03T21:26:09.120" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24338" PostTypeId="2" ParentId="24315" CreationDate="2017-11-03T21:34:44.387" Score="2" Body="&lt;p&gt;I think there's some predictor Q for one or a few of your positive examples that also applies to a lot of negative examples. Because you have so few positive examples, there's not much to separate the good from the mediocre predictors for them. When you got to the validation set, Q must have applied to a greater proportion of negative examples than it did in the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To mitigate this, try n-fold cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-03T21:34:44.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24339" PostTypeId="1" AcceptedAnswerId="24342" CreationDate="2017-11-03T21:45:09.203" Score="31" ViewCount="32757" Body="&lt;p&gt;I have two questions related to decision trees:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If we have a continuous attribute, how do we choose the splitting value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: Age=(20,29,50,40....)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Imagine that we have a continuous attribute $f$ that have values in $R$. How can I  write an algorithm that finds the split point $v$, in order that when we split $f$ by $v$, we have a minimum gain for $f&amp;gt;v$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41397" LastEditorUserId="29575" LastEditDate="2017-11-04T19:07:23.027" LastActivityDate="2021-12-24T15:49:10.383" Title="How is a splitting point chosen for continuous variables in decision trees?" Tags="|classification|data|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24341" PostTypeId="2" ParentId="24335" CreationDate="2017-11-03T22:07:29.840" Score="7" Body="&lt;p&gt;Support vector data description (SVDD) finds the smallest hypersphere that contains all samples, except for some outliers. One-class SVM (OC-SVM) separates the inliers from the outliers by finding a hyperplane of maximal distance from the origin. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the kernel function has the property that &lt;span class=&quot;math-container&quot;&gt;$k(\mathbf{x}, \mathbf{x}) = 1 \quad \forall \mathbf{x} \in \mathbb{R}^d$&lt;/span&gt;, SVDD and OC-SVM learn identical decision functions. Many common kernels have this property, such as RBF, Laplacian and &lt;span class=&quot;math-container&quot;&gt;$\chi^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVDD and OC-SVM are also equivalent in the case that all samples lie on a hypersphere centered at the origin, and are are linearly separable from it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;em&gt;Lampert, C. H. (2009). &lt;a href=&quot;http://pub.ist.ac.at/~chl/papers/lampert-fnt2009.pdf&quot; rel=&quot;noreferrer&quot;&gt;Kernel methods in computer vision&lt;/a&gt; (Chapter 5)&lt;/em&gt; for more detailed descriptions of these models.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2019-02-21T21:26:24.917" LastActivityDate="2019-02-21T21:26:24.917" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24342" PostTypeId="2" ParentId="24339" CreationDate="2017-11-03T22:18:45.813" Score="33" Body="&lt;p&gt;In order to come up with a split point, the values are sorted, and the mid-points between adjacent values are evaluated in terms of some metric, usually information gain or gini impurity. For your example, lets say we have four examples and the values of the age variable are $(20, 29, 40, 50)$. The midpoints between the values $(24.5, 34.5, 45)$ are evaluated, and whichever split gives the best information gain (or whatever metric you're using) on the training data is used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can save some computation time by only checking split points that lie between examples of different classes, because only these splits can be optimal for information gain.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-03T22:18:45.813" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24344" PostTypeId="2" ParentId="24320" CreationDate="2017-11-04T04:50:50.287" Score="1" Body="&lt;p&gt;I would first make sure that I have only independent features (you can easily double check the covariance matrix and spot the zero singular values). Then, I would go for linear/logistic regression/linear SVM. If it doesn't give the desired performance, I would try the kernel SVM. &lt;/p&gt;&#xA;" OwnerUserId="41240" LastActivityDate="2017-11-04T04:50:50.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24345" PostTypeId="2" ParentId="24000" CreationDate="2017-11-04T05:49:17.263" Score="2" Body="&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&#xA;Yes. You can (iiuc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer Version:&lt;/strong&gt;&#xA;In fact, this is what many popular algorithms like Word2Vec and AutoEncoders do. (With respect to hidden layer outputs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2Vec:&#xA;Given an input word ('chicken'), the model tries to predict the neighbouring word ('wings')&#xA;In the process of trying to predict the correct neighbour, the model learns a hidden layer representation of the word which helps it achieve its task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, we just remove the last layer and use the hidden layer representation of the word as its $N$ dimensional vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rFSLZ.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rFSLZ.jpg&quot; alt=&quot;the embedded representation is what we use&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically, we feature engineered the word vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For autoencoders&lt;/strong&gt;,&#xA;It takes in $X$ as the input and tries to predict $X$ again, in the process learning a latent representation of the input signal X. The input hidden representation in Layer $L2$ can be used in other tasks. (Note: here X and X hat are the same)&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Gckml.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Gckml.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(In fact, you can use features learned by a CNN and feed them into an SVM and get good results)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Relevant to you:&lt;/strong&gt;&#xA;You can train your model on the given data and finally chop off the last prediction layer and use the output of the intermediate layers as features. I believe this should work because it works for so many tasks which I explained above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://medium.com/towards-data-science/learn-word2vec-by-implementing-it-in-tensorflow-45641adaf2ac&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learn Word2Vec by implementing it in Word2Vec&lt;/a&gt; : an article by me explaining word2vec. (Shameless self-advertising here but I feel the article is good and relevant)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng's unsupervised feature learning website&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33738" LastActivityDate="2017-11-04T05:49:17.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24347" PostTypeId="2" ParentId="14984" CreationDate="2017-11-04T07:21:49.043" Score="8" Body="&lt;p&gt;The paper cited in the question &quot;FaceNet: A Unified Embedding for Face Recognition and Clustering&quot; is available at &lt;a href=&quot;https://arxiv.org/pdf/1503.03832.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1503.03832.pdf&lt;/a&gt; . Page no 5 of this paper lists the reference [16] for the Inception module architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper introducing and describing this &lt;strong&gt;Inception&lt;/strong&gt; architecture is - &quot;Going Deeper With Convolutions&quot;, it is accessible at &lt;a href=&quot;https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An overview of inception modules is given in the diagram on page 4, its included here - &#xA;&lt;a href=&quot;https://i.stack.imgur.com/iNy2U.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iNy2U.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key idea for devising this architecture is to deploy multiple convolutions with multiple filters and pooling layers simultaneously in parallel within the same layer (inception layer). For example, in the architecture shown in this diagram (figure a) employs convolution with 1x1 filters as well as 3x3 and 5x5 filters and a max pooling layer. Figure b demonstrates how the use of 1x1 convolution filters can achieve dimensionality reduction (since no. of channels is reduced).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intention is to let the neural network learn the best weights when training the network and automatically select the more useful features. Additionally, it intends to reduce the no. of dimensions so that the no. of units and layers can be increased at later stages. The side-effect of this is to increase the computational cost for training this layer. To address this, a number of solutions have been suggested in the paper such as to deploy parallel computations for this architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an illustration of these concepts, the table on page 5 shows an incarnation of this architecture termed as &quot;GoogLeNet&quot; which was used for the ILSVRC 2014 competition.&lt;/p&gt;&#xA;" OwnerUserId="18540" LastEditorUserId="18540" LastEditDate="2017-11-04T17:37:37.753" LastActivityDate="2017-11-04T17:37:37.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24348" PostTypeId="2" ParentId="18840" CreationDate="2017-11-04T10:11:52.493" Score="2" Body="&lt;p&gt;The brief answers are: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1, 2. No. They depend on different subsets of examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3, 4. Yes, but only if it separates classes linearly and you are extremely lucky. Otherwise no. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;5, 6. No, because SVM and LDA find only one solution, but perceptron can find many.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let me explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Decision boundaries of classicl SVM and LDA are calculated offline based on the whole training sample at once. Thus, SVM and LDA will indeed always find exactly one hyperplane, regardless of the order of learning examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But these hyperplanes of LDA and SVM are not generally guaranteed to be different. As its name suggests, SVM solution is based &lt;em&gt;only on the support observations&lt;/em&gt;, which usually constitute a small fraction of the learning sample. LDA solution, however,  is sensitive to all the learning examples, because it is explicitly based on their classwise means. Thus, you can shift one non-support point a little, and SVM solution will not change, but LDA will. And it means that you cannot expect that SVM and LDA will find the same boundary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perceptron, in contrast, is updated online, so its decision boundary depends on the order of learning examples. Moreover, its solutions depends on the initialization of coefficients. Namely, if $w_n$ is vector of coefficients after seeing $n$ learning examples, then:&#xA;$$&#xA;w_n = w_0 + \lambda \sum_{i=1}^{n} e_i x_i&#xA;$$&#xA;where $e_i$ equals 1 if $i$'th example was false negative, -1 for false positives, and 0 otherwise, and $\lambda$ is the learning rate. By varying $w_0$ and order of feeding $x_i$ into perceptron, you can achieve many different hyperplanes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if (by luck!) you set $w_0$ equal to solution of SVM or LDA, and this solution linearly separates your classes, then perceptron will never change this solution, so it will be equal to solution of SVM or LDA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if the solution of SVM or LDA does not fully separate the classes (for LDA, this is may be the case even if the classes &lt;em&gt;are&lt;/em&gt; separable), then perceptron will change it with the next misclassified example, so its solution will diverge from SVM or LDA.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-04T10:11:52.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24351" PostTypeId="1" AcceptedAnswerId="24360" CreationDate="2017-11-04T12:56:01.533" Score="1" ViewCount="366" Body="&lt;p&gt;I'm working on a homework problem but don't fully understand it. The problem and solution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/j2q51.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/j2q51.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand the definition of the threshold function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it mean to pick one feature and classify the point based on that one feature? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's the only way I can think of to explain the solution, $N$ ways to pick a feature, for each feature there are $m+1$ ways to select the threshold.&lt;/p&gt;&#xA;" OwnerUserId="41410" LastEditorUserId="29575" LastEditDate="2017-11-04T21:27:18.593" LastActivityDate="2017-11-04T21:27:18.593" Title="How to calculate growth function for a threshold function" Tags="|machine-learning|statistics|homework|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24352" PostTypeId="1" AcceptedAnswerId="24358" CreationDate="2017-11-04T13:49:05.097" Score="5" ViewCount="5783" Body="&lt;p&gt;What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I noticed in many data sciences courses, there is always a strong assumption of using a normal/Gaussian data. I have always wonder why this is so, and most people would say that due to central limit theorem, the data is always assumed to be normal.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;However, what if the data that I am dealing with is not normally distributed?&lt;/li&gt;&#xA;&lt;li&gt;Should I perform log/ exponential transformation on the data for the sake of getting normal distributed data?&lt;/li&gt;&#xA;&lt;li&gt;Why is Gaussian data always best suited?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="41414" LastEditorUserId="29575" LastEditDate="2017-11-04T14:18:45.213" LastActivityDate="2017-11-06T23:47:11.177" Title="Working with Data which is not Normal/Gaussian" Tags="|distribution|gaussian|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24353" PostTypeId="2" ParentId="19895" CreationDate="2017-11-04T14:10:18.483" Score="4" Body="&lt;p&gt;I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt; contributor, who works a lot with topic modelling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He has a couple of notebooks on his &lt;a href=&quot;https://github.com/bhargavvader&quot; rel=&quot;nofollow noreferrer&quot;&gt;github account&lt;/a&gt; which could interest you, especially &lt;a href=&quot;https://github.com/bhargavvader/personal/tree/master/notebooks/text_analysis_tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt; (should be pretty much your use case if I understand your problem correctly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aforementioned notebook evaluates Latent Semantic Indexing, the Hierarchical Dirichlet Process as well as the Latent Dirichlet Allocation for identifying topics.&lt;/p&gt;&#xA;" OwnerUserId="41416" LastEditorUserId="29575" LastEditDate="2017-11-05T09:29:02.897" LastActivityDate="2017-11-05T09:29:02.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24357" PostTypeId="2" ParentId="24327" CreationDate="2017-11-04T16:59:22.003" Score="2" Body="&lt;p&gt;Transposed Convolutions is what you are looking for, for more details take a look here:&#xA;&lt;a href=&quot;https://medium.com/towards-data-science/types-of-convolutions-in-deep-learning-717013397f4d&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/towards-data-science/types-of-convolutions-in-deep-learning-717013397f4d&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41421" LastActivityDate="2017-11-04T16:59:22.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24358" PostTypeId="2" ParentId="24352" CreationDate="2017-11-04T17:18:31.500" Score="8" Body="&lt;p&gt;There are models that do not make assumption that the underlying data distribution is a normal distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, support vector machine just cares about the boundaries of the separating hyperplane and do not assume the exact shape of the distributions. Decision tree models also do not make such assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gaussian distribution is popular and it has been analyzed frequently for its simplicity but there are other models as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know the distributions that you believed your data can follow, you can also build your own model, for example by maximizing the likelihood or posterior distributrion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are able to convert your data to normal distribution and use a model that you are familiar with, you can try it and see how does it perform. &lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2017-11-04T17:18:31.500" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24360" PostTypeId="2" ParentId="24351" CreationDate="2017-11-04T18:00:30.527" Score="1" Body="&lt;p&gt;Yes, your interpretation is correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each member of $H$ is one such function, they are parametrized by $i$, the feature selected and $\theta$, the chosen threshold. Different $\theta$ might corresponds to the same function but effectively there are only $m+1$ such function for each $i$. Hence, the set $H$ consists of at most $(m+1)N$ elements.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2017-11-04T18:00:30.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24361" PostTypeId="2" ParentId="24315" CreationDate="2017-11-04T18:03:48.527" Score="3" Body="&lt;p&gt;Davis and Goadrich have explained the relationship between ROC and PR Curves in their &lt;a href=&quot;http://machinelearning.wustl.edu/mlpapers/paper_files/icml2006_DavisG06.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. It is always recommended to use PR curve over the ROC curve in the presence of highly imbalanced data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to the behavior of your ROC curve, It seems that you don't have more threshold points! I would also agree with Dan and do K-fold CV. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Davis, J. and Goadrich, M., 2006, June. The relationship between&#xA;  Precision-Recall and ROC curves. In Proceedings of the 23rd&#xA;  international conference on Machine learning (pp. 233-240). ACM.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="41240" LastEditorUserId="41240" LastEditDate="2017-11-04T21:39:29.223" LastActivityDate="2017-11-04T21:39:29.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24362" PostTypeId="2" ParentId="22587" CreationDate="2017-11-04T19:41:59.813" Score="7" Body="&lt;p&gt;I haven't yet successfully solved my record linkage problem, but I wanted to share some of the stuff I've found in the process in case it's of use to anyone else. This is a work in progress based &lt;a href=&quot;https://github.com/ropeladder/record-linkage-resources&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; on GitHub.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Record Linkage Resources&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;(also known as deduplication, data matching, entity resolution)&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Background&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Documents&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Wikipedia pages on &lt;a href=&quot;https://en.wikipedia.org/wiki/Record_linkage&quot; rel=&quot;nofollow noreferrer&quot;&gt;Record Linkage&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_deduplication&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Deduplication&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Overview slideshttps://www.umiacs.umd.edu/~getoor/Tutorials/ER_VLDB2012.pdf&lt;/li&gt;&#xA;&lt;li&gt;[Dedupe]'s explanation of how their software works: &lt;a href=&quot;https://dedupe.io/developers/library/en/latest/How-it-works.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dedupe.io/developers/library/en/latest/How-it-works.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Talks&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Peter Christen, Record Linkage lectures at ADRC-Scotland: &lt;a href=&quot;https://www.youtube.com/watch?v=DyGonV7A_EY&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt; &lt;a href=&quot;https://www.youtube.com/watch?v=dcNTvYDdun0&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt; &lt;a href=&quot;https://www.youtube.com/watch?v=HAKW5tHVCmw&quot; rel=&quot;nofollow noreferrer&quot;&gt;3&lt;/a&gt; &lt;a href=&quot;https://www.youtube.com/watch?v=4Iv5axrAWqQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;4&lt;/a&gt; (2015)&lt;/li&gt;&#xA;&lt;li&gt;Mike Mull, The Art and Science of Data Matching: &lt;a href=&quot;https://www.youtube.com/watch?v=Y-nYEOgq3YE&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=Y-nYEOgq3YE&lt;/a&gt; (2015)&lt;/li&gt;&#xA;&lt;li&gt;Rhydwyn Mcguire, Join for real life: &lt;a href=&quot;https://www.youtube.com/watch?v=cEcVIjyHfiQ&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=cEcVIjyHfiQ&lt;/a&gt; (2013)&lt;/li&gt;&#xA;&lt;li&gt;Andrew Rowe, Big Data Deduplication and Data Matching using Python: &lt;a href=&quot;https://www.youtube.com/watch?v=Z6mlvrYEYnk&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=Z6mlvrYEYnk&lt;/a&gt; (2013)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Books&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Peter Christen, &lt;em&gt;Data Matching&lt;/em&gt;: &lt;a href=&quot;http://www.springer.com/us/book/9783642311635&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.springer.com/us/book/9783642311635&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Thomas N. Herzog, Fritz J. Scheuren and William E. Winkler, &lt;em&gt;Data Quality and Record Linkage Techniques&lt;/em&gt;: &lt;a href=&quot;http://www.springer.com/us/book/9780387695020&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.springer.com/us/book/9780387695020&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Free software&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;(last updated, github stars as of Nov 2017)&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Python&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dedupe: &lt;a href=&quot;https://github.com/dedupeio/dedupe&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dedupeio/dedupe&lt;/a&gt; (2017, 1,413)&lt;/li&gt;&#xA;&lt;li&gt;RLTK: &lt;a href=&quot;https://github.com/usc-isi-i2/rltk/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/usc-isi-i2/rltk/&lt;/a&gt; (2017, 13)&lt;/li&gt;&#xA;&lt;li&gt;Febrl: &lt;a href=&quot;https://sourceforge.net/projects/febrl/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://sourceforge.net/projects/febrl/&lt;/a&gt; (2013)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Java&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Duke: &lt;a href=&quot;https://github.com/larsga/Duke&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/larsga/Duke&lt;/a&gt; (2016, 469)&lt;/li&gt;&#xA;&lt;li&gt;JedAI: &lt;a href=&quot;https://github.com/scify/JedAIToolkit&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/scify/JedAIToolkit&lt;/a&gt; (2017, 12)&lt;/li&gt;&#xA;&lt;li&gt;LSHDB: &lt;a href=&quot;https://github.com/dimkar121/LSHDB&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dimkar121/LSHDB&lt;/a&gt; (2017, 9)&lt;/li&gt;&#xA;&lt;li&gt;FRIL: &lt;a href=&quot;http://fril.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://fril.sourceforge.net/&lt;/a&gt; (2011) (Windows binary available)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;R&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RecordLinkage: &lt;a href=&quot;https://r-forge.r-project.org/projects/recordlinkage/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://r-forge.r-project.org/projects/recordlinkage/&lt;/a&gt; &lt;a href=&quot;https://cran.r-project.org/web/packages/RecordLinkage/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/RecordLinkage/index.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;fastLink: &lt;a href=&quot;https://github.com/kosukeimai/fastLink&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/kosukeimai/fastLink&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Other&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;OpenRecLink: &lt;a href=&quot;http://reclink.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://reclink.sourceforge.net/&lt;/a&gt; (2016)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;C++ with GUI&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Registry Plus™ Link Plus: &lt;a href=&quot;https://www.cdc.gov/cancer/npcr/tools/registryplus/lp_tech_info.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cdc.gov/cancer/npcr/tools/registryplus/lp_tech_info.htm&lt;/a&gt; (2007)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Commercial software and solutions&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data Ladder DataMatch: &lt;a href=&quot;https://dataladder.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dataladder.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Dedupe: &lt;a href=&quot;https://dedupe.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dedupe.io/&lt;/a&gt; (freemium frontend for &lt;a href=&quot;https://github.com/dedupeio/dedupe&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dedupe&lt;/a&gt; Python library)&lt;/li&gt;&#xA;&lt;li&gt;LinkageWiz: &lt;a href=&quot;http://www.linkagewiz.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.linkagewiz.net/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;WinPure Clean and Match: &lt;a href=&quot;http://www.winpure.com/cleanmatch.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.winpure.com/cleanmatch.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Reifier: &lt;a href=&quot;http://nubetech.co/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://nubetech.co/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;For SAS&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(free but requires SAS) The Link King: &lt;a href=&quot;http://www.the-link-king.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.the-link-king.com/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Data Cleaning&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Name Parsers&lt;/h3&gt;&#xA;&#xA;Python&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;probablepeople: &lt;a href=&quot;https://github.com/datamade/probablepeople&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/datamade/probablepeople&lt;/a&gt; (2017, 204)&lt;/li&gt;&#xA;&lt;li&gt;Name Parser: &lt;a href=&quot;https://github.com/derek73/python-nameparser&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/derek73/python-nameparser&lt;/a&gt; (2017, 232)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;JavaScript&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;parse-full-name: &lt;a href=&quot;https://github.com/dschnelldavis/parse-full-name&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dschnelldavis/parse-full-name&lt;/a&gt; (2017, 18)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Papers&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;US Census Bureau Data Linkage research: &lt;a href=&quot;https://www.census.gov/srd/csrm/RecordLinkage.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.census.gov/srd/csrm/RecordLinkage.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.utexas.edu/%7Eml/papers/marlin-dissertation-06.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learnable Similarity Functions and their Application to Record Linkage and Clustering&lt;/a&gt; (&lt;a href=&quot;https://github.com/dedupeio/dedupe&quot; rel=&quot;nofollow noreferrer&quot;&gt;via&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;[&lt;a href=&quot;https://www.microsoft.com/en-us/research/publication/improving-entity-resolution-with-global-constraints/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.microsoft.com/en-us/research/publication/improving-entity-resolution-with-global-constraints/&lt;/a&gt; (&lt;a href=&quot;https://dedupe.io/developers/library/en/latest/Bibliography.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;via&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1312.4645&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1312.4645&lt;/a&gt; (&lt;a href=&quot;https://dedupe.io/developers/library/en/latest/Bibliography.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;via&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://people.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick09entity.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://people.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick09entity.pdf&lt;/a&gt; (&lt;a href=&quot;https://dedupe.io/developers/library/en/latest/Bibliography.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;via&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://homes.cs.washington.edu/~pedrod/papers/mrdm04.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://homes.cs.washington.edu/~pedrod/papers/mrdm04.pdf&lt;/a&gt; (&lt;a href=&quot;https://dedupe.io/developers/library/en/latest/Bibliography.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;via&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.researchgate.net/publication/318874102_Clink_-_A_Novel_Record_Linkage_Methodology_based_on_Graph_Interactions&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.researchgate.net/publication/318874102_Clink_-_A_Novel_Record_Linkage_Methodology_based_on_Graph_Interactions&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Organizations&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;US Census Burea - Center for Statistical Research and Methodolgy - Record Linkage: &lt;a href=&quot;https://www.census.gov/srd/csrm/RecordLinkage.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.census.gov/srd/csrm/RecordLinkage.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Stanford Entity Resolution Framework: &lt;a href=&quot;http://infolab.stanford.edu/serf/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://infolab.stanford.edu/serf/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;ANU Data Mining and Matching Group &lt;a href=&quot;https://dmm.anu.edu.au/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dmm.anu.edu.au/&lt;/a&gt; &lt;a href=&quot;https://web.archive.org/web/20160515215747/datamining.anu.edu.au/projects/linkage.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://web.archive.org/web/20160515215747/datamining.anu.edu.au/projects/linkage.html&lt;/a&gt; (Archive.org link)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Misc&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;DuDe (framework for comparing record linkage results): &lt;a href=&quot;https://hpi.de/naumann/projects/data-quality-and-cleansing/dude-duplicate-detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://hpi.de/naumann/projects/data-quality-and-cleansing/dude-duplicate-detection.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Datasets to use for evaluating deduplication software: &lt;a href=&quot;https://hpi.de/naumann/projects/repeatability/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://hpi.de/naumann/projects/repeatability/datasets.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www2.vrdc.cornell.edu/news/3/20050420-Record%20Linkage%20Software.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www2.vrdc.cornell.edu/news/3/20050420-Record%20Linkage%20Software.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="38472" LastEditorUserId="38472" LastEditDate="2018-10-17T01:15:14.063" LastActivityDate="2018-10-17T01:15:14.063" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24363" PostTypeId="1" CreationDate="2017-11-04T20:37:03.373" Score="1" ViewCount="400" Body="&lt;p&gt;I want to train a OneClassSVM() using sklearn, and I have a set of around 800 images in my training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using opencv to read the images and resize them to constant dimensions (960x540) and then adding them to a numpy-array. The images are RGB and thus have 3-dimensions. For that, I am reshaping the numpy array after reading all the images:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Assume X is my numpy array which contains all the images before reshaping&#xA;#Now I reshape X&#xA;n_samples = len(X)&#xA;X = X.reshape(n_samples, 950*540*3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the number of features is huge (1,539,000 to be exact).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I try to train my model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = OneClassSVM(kernel='rbf', gamma=0.001)&#xA;model.fit(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After running my code, it crashed due to &lt;code&gt;MemoryError&lt;/code&gt;. If I'm not mistaken this is obvious due the large number of features? So, is there a better way to pre-process the images before fitting them, or to decrease the number of features? &lt;/p&gt;&#xA;" OwnerUserId="40795" LastActivityDate="2024-02-10T13:03:20.423" Title="Pre-process data images before training OneClassSVM and decrease number of features" Tags="|machine-learning|python|scikit-learn|preprocessing|numpy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24365" PostTypeId="2" ParentId="23603" CreationDate="2017-11-05T04:34:58.040" Score="3" Body="&lt;p&gt;Your intuition about 'no effect' is true in some sense. But this replacement may be not the best use of the information you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of missing value treatment depends on your initial problem statement. In all the cases I assume that you have already somehow estimated conditional means $\mu_0$ and $\mu_1$ and the common variance matrix $S$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 1&lt;/strong&gt;. For your test observation, you know $y$, but know nothing about $x$, and want to estimate $x_n$. If this is the case, $\mu_{y, n}$ is indeed your best guess of $x_n$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 2&lt;/strong&gt;. For your test observation, you know $x_{-n}$ (that is, $x$ except $x_n$) and $y$, and want to predict $x_n$ as best as possible. If it is the case, you should use expectation conditional not only on $y$, but also on $x_{-n}$:&#xA;$$&#xA;\hat{x_n} = \mathbb{E}(x_n|y, x_{-n}) = \mu_{(y,n)} + S_{(n,-n)}S_{(-n)}^{-1}(x_{-n}-\mu_{(y,-n)})&#xA;$$ &#xA;where $S_{*}$, $\mu_{y,*}$ are corresponding parts of covariance matrix and conditional expectation vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 3&lt;/strong&gt;. You know $x_{-n}$ but don't know $y$, and you want to estimate $x_n$. In this case, conditional distribution of $x_n$ is a mixture of two normal distributions, $p(x_n|y=0, x_{-n})$ and $p(x_n|y=1, x_{-n})$, with weights, proportional to $p(y=0|x_{-n})$ and $p(y=1|x_{-n})$ respectively. The mean of this distribution is just the weighted average of two means conditional on $y$. But this distribution may be bimodal, and its mean may be not a very useful statistic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 4&lt;/strong&gt;. You know only $x_{-n}$, and your final goal is to estimate probability of $y=1$. In this case, you should just forget about $x_n$ (you don't know anything of it anyway), and use the distribution of $x_{-n}$ conditional on $y$ for whatever algorithm you apply.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-05T04:34:58.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24366" PostTypeId="2" ParentId="24363" CreationDate="2017-11-05T09:04:17.167" Score="0" Body="&lt;p&gt;You should try converting them to Principle Components using PCA. Please refer this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analytics Vidya PCA&lt;/a&gt;, this should give give you a good understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA converts n large vector into p Principle components. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;First principal component is a linear combination of original predictor variables which captures the maximum variance in the data set&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Second PC is also a linear combination of original predictor variables which captures remaining variance. All the succeeding ones follow the same concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way you can select top Principle components which explains good enough cumulative variance in your data&lt;/p&gt;&#xA;" OwnerUserId="24818" LastActivityDate="2017-11-05T09:04:17.167" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24368" PostTypeId="2" ParentId="20016" CreationDate="2017-11-05T12:56:47.303" Score="1" Body="&lt;p&gt;You can check Fashion10000 dataset released by Delft University of Technology &lt;a href=&quot;http://www.st.ewi.tudelft.nl/~bozzon/fashion10000dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="41446" LastEditorUserId="26562" LastEditDate="2017-11-05T21:33:02.867" LastActivityDate="2017-11-05T21:33:02.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24369" PostTypeId="1" AcceptedAnswerId="24394" CreationDate="2017-11-05T13:02:58.023" Score="1" ViewCount="86" Body="&lt;p&gt;Is their any existing Ensemble technique which uses subset of training data to predict which algorithm is better for predicting each instance of testing data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have N sized training set and K sized testing set in which a particular attribute needed to be predicted using the training set. But there are hundreds of algorithms and ways we can use. We can divide training set into two parts and train each model with first half and decide test on second half. Based on characteristics, we can decide which algorithm to use for real test cases (K sized set). As an example lets say dataset have an attribute named &quot;temperature&quot;. Particular algorithm may work well when temperature is higher than 100 Celsius. We can then classify all the 100 degree or above instances to particular class. Then final prediction will be done based on that by with that model class trained with all N sized data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am asking is that is their any existing method similar to that? &lt;/p&gt;&#xA;" OwnerUserId="26854" LastEditorUserId="26562" LastEditDate="2017-11-05T23:47:43.347" LastActivityDate="2018-02-04T09:38:25.167" Title="Learning Algorithm that decide which model gives better results for each testing instance" Tags="|machine-learning|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24370" PostTypeId="1" AcceptedAnswerId="24385" CreationDate="2017-11-05T14:02:42.483" Score="1" ViewCount="7523" Body="&lt;p&gt;I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I have the tf idf vectors for the query and a document. I want to compute the cosine similarity between both vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I compute the magnitude for the document vector, do I sum the squares of all the terms in the vector or just the terms in the query?   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example : we have user query &quot;cat food beef&quot; .&lt;br&gt;&#xA;Lets say its vector is (0,1,0,1,1).( assume there are only 5 directions  in the vector one for each unique word in the query and the document)&lt;br&gt;&#xA;We have a document &quot;Beef is delicious&quot;&lt;br&gt;&#xA; Its vector is (1,1,1,0,0). We want to find the cosine similarity between the query and the document vectors. &lt;/p&gt;&#xA;" OwnerUserId="41448" LastEditorUserId="41448" LastEditDate="2017-11-05T15:56:58.437" LastActivityDate="2017-11-05T23:59:08.087" Title="Cosine similarity between query and document confusion" Tags="|similarity|information-retrieval|ranking|cosine-distance|similar-documents|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24372" PostTypeId="1" CreationDate="2017-11-05T15:33:20.117" Score="2" ViewCount="1335" Body="&lt;p&gt;I have a dataset with 600 examples of target class yes and 300 of no.&#xA;After classification I get high accuracy(75%) but low kappa (0.4).&#xA;What can it mean? It seems that kappa is really low, am I right?&#xA;Other metrics like precission or AUR are about 0.7-0.8 what seems to be good.&lt;/p&gt;&#xA;" OwnerUserId="41451" LastActivityDate="2017-11-05T18:49:50.327" Title="Low Kappa score but high accuracy" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24374" PostTypeId="2" ParentId="24369" CreationDate="2017-11-05T16:52:31.010" Score="0" Body="&lt;p&gt;I do not know of a specific algorithm that does this for you however it would not be hard to build. If you have two prediction algorithms then you can use them both (on training data) to put it into two classes. Each class would represent which algorithm would perform better. You then use a third algorithm to do binary classification to see if you can predict which of the original algorithms you should use. If you can predict which algorithm to use then you are set. You will get much better results if your classification algorithm is similar to your two prediction algorithms. In that case they will handle all the features in a similar way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once trained you then classify your test set and run the algorithms suggested by the classification. &lt;/p&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2017-11-05T16:52:31.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24375" PostTypeId="2" ParentId="23650" CreationDate="2017-11-05T17:45:12.990" Score="0" Body="&lt;p&gt;I suppose you can continue training your model after collecting new observations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest strategy would be to put more weight to the most recent observations (which are affected by model outcome). &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-11-05T17:45:12.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24376" PostTypeId="1" AcceptedAnswerId="24399" CreationDate="2017-11-05T17:46:21.110" Score="6" ViewCount="17990" Body="&lt;p&gt;I have the dataframe which has two colums(Reviews and Label):  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Reviews                                     Label  &#xA;0 [cameron, diaz, woman, marri, judg, play]   1  &#xA;1 [turgid, dialogu, feebl, character, harvey] 0  &#xA;2 [misfortun, watch, rubbish, sky, cinema]    1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to apply the TfidfVectorizer on the DF.&lt;br&gt;&#xA;I have written the following code.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer  &#xA;df_x=train_df[&quot;Reviews&quot;]  &#xA;df_y=train_df[&quot;Label&quot;]  &#xA;cv = TfidfVectorizer()   &#xA;df_xcv = cv.fit_transform(df_x)  &#xA;a=df_xcv.toarray()  &#xA;cv.get_feature_names()  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is giving an error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;AttributeError: 'list' object has no attribute 'lower'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why is this throwing an error?&lt;/p&gt;&#xA;" OwnerUserId="41419" LastEditorUserId="29575" LastEditDate="2017-11-09T16:06:34.547" LastActivityDate="2018-03-25T13:10:28.580" Title="Use of TfidfVectorizer on dataframe" Tags="|machine-learning|scikit-learn|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24377" PostTypeId="2" ParentId="24372" CreationDate="2017-11-05T18:49:50.327" Score="1" Body="&lt;p&gt;I'm not sure that anything is wrong here. With a binary target, chance is 50%, and 75% is half way to perfect accuracy of 100%.  Kappa, which ranges from 0 at chance to 1 at perfect separation, is 0.4, which is again a little less than half way to perfect prediction.  So it seems to be corroborating the performance of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kappa is defined as:&#xA;(observed accuracy - expected accuracy)/(1 - expected accuracy)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your classes appear balanced, you have (0.75 - 0.5) / (1 - 0.5) = 0.5&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-11-05T18:49:50.327" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24379" PostTypeId="1" AcceptedAnswerId="24388" CreationDate="2017-11-05T22:02:09.263" Score="4" ViewCount="614" Body="&lt;p&gt;I have a data set of 5 groups and their associations to different interests. The data is structured as follows with &lt;strong&gt;2000+&lt;/strong&gt; interests and &lt;strong&gt;5&lt;/strong&gt; known groups:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Interest     Group1     Group2     Group3     Group4     Group5&#xA;01.   Sports         10%        40%        30%       80%         65%&#xA;02.   Music          35%        60%        55%       10%         15%&#xA;03.   Fashion        80%        10%        75%        5%         25%&#xA; .&#xA; .&#xA; .&#xA;1998.  Movies        70%        20%        45%       50%         65%&#xA;1999.  Books         50%        70%        35%       65%         95%&#xA;2000.  Art           90%        50%        25%       15%         30%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to gain an understanding of what would be the best way to determine which groups are closest to each other and which ones are the most different from each other in terms of interests. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am leaning towards cluster analysis, however the structure of my data is throwing me off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="41460" LastEditorUserId="41460" LastEditDate="2017-11-06T02:26:37.170" LastActivityDate="2017-11-09T10:09:52.303" Title="Identifying which known groups are the most similar or most dissimilar" Tags="|data-mining|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24380" PostTypeId="1" AcceptedAnswerId="24443" CreationDate="2017-11-05T22:42:07.717" Score="4" ViewCount="5410" Body="&lt;p&gt;I am implementing K-Means from scratch and that exercise raised a question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To update my centroids, for each centroid, I have to find the points for which that centroid is the closest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases, especially when the number of centroids is high and the number of instances is low (i.e. k=20 and 100 instances), I find centroids for which no point has them as their closest centroid. In other words, they become &quot;orphans&quot; as no instances are allocated to them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5897Z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5897Z.png&quot; alt=&quot;Xs are centroids and Os are instances&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the example above, the two centroids (X) at the top clearly both have points for which they are the closest centroid. But the centroid at the bottom is never the closest centroid for any instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I deal with this?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should the lone centroid remain unmoved?&lt;/li&gt;&#xA;&lt;li&gt;Should I move that centroid? If yes, how?&lt;/li&gt;&#xA;&lt;li&gt;Should I remove it?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way to deal with this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;------------ EDIT: --------------&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On Github, I found this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/klebenowm/cs540/blob/master/hw1/KMeans.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/klebenowm/cs540/blob/master/hw1/KMeans.java&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This suggests that if a centroid becomes an &quot;orphan&quot;, it should be assigned to the point that is the furthest from its centroid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems like a sound method, is there any paper or theory supporting this?&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-11-08T19:37:17.193" LastActivityDate="2019-07-03T03:56:21.567" Title="In K-means, what happens if a centroid is never the closest to any point?" Tags="|clustering|k-means|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24382" PostTypeId="2" ParentId="24379" CreationDate="2017-11-05T22:55:12.763" Score="4" Body="&lt;p&gt;You only have five groups, so full blown clustering is probably not a good idea here, but looking at similarity scores between the group vectors may be insightful. An easy one to try at first would be cosine similarity, which essentially measures the angle between each of your group vectors:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;similarity = $cos(\theta)$ = $\frac{\mathbf{A} \cdot \mathbf{B}}{||\mathbf{A}||_2 ||\mathbf{B}||_2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mathbf{A} \cdot \mathbf{B}$ gives the dot product of two vectors, and $||\mathbf{A}||_2$ is the magnitude of a vector $\mathbf{A}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of your vector values are positive, so the result will be in the range $(0, 1)$. The closer this value is to $1$, the more similar the vectors. $0$ means they are completely decorrelated (vectors are orthogonal). You can compare the cosine similarities of each pair of groups to work out which ones are most similar/dissimilar. &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-05T22:55:12.763" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24384" PostTypeId="2" ParentId="24380" CreationDate="2017-11-05T23:45:18.513" Score="2" Body="&lt;p&gt;You should leave the lone centroid unmoved. In the next iteration, it's possible that the cluster centers have moved in such a way that now there are some instances that are closest to the lone centroid, and it can get picked up. At the end of the k-means algorithm, you could remove clusters that have no instances associated with them, but it doesn't really matter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day, it's probably not a good idea to use k-means in the manner that you described, and we should not expect it necessarily to produce good results.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-05T23:45:18.513" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24385" PostTypeId="2" ParentId="24370" CreationDate="2017-11-05T23:59:08.087" Score="1" Body="&lt;p&gt;You want to use all of the terms in the vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is computed as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;similarity $= \frac{\mathbf{q} \cdot \mathbf{d}}{||\mathbf{q}||_2 ||\mathbf{d}||_2} = \frac{0\times1+1\times1+0\times1+1\times0+1\times0}{\sqrt{1^2+1^2+1^2} \times \sqrt{1^2+1^2+1^2}} = \frac{0+1+0+0+0}{\sqrt{3}\sqrt{3}} = \frac{1}{3}$ &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-05T23:59:08.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24388" PostTypeId="2" ParentId="24379" CreationDate="2017-11-06T02:52:45.250" Score="2" Body="&lt;p&gt;For tiny sample sizes (5), hierarchical clustering and &lt;strong&gt;dendrograms&lt;/strong&gt; work best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As similarity measure, I'd go with Manhattan distance.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-06T02:52:45.250" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24389" PostTypeId="2" ParentId="20275" CreationDate="2017-11-06T04:16:15.850" Score="0" Body="&lt;p&gt;GLOSH works with regard to &lt;em&gt;local&lt;/em&gt; outliers, thus if a region is very dense and then has a few sparse points near it GLOSH will rate those sparse points as more outlying than other noise points in sparser regions. This is because it is considered particularly odd for a point to be near, but not in a very dense region -- local to the region the point is a significant outlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regard to your question: if you care about local outliers (things that are odd relative to the density of the nearby region) then GLOSH is a good choice. This would be useful, for example, in detecting network intrusions since an attacker will be trying to behave like normal users, but will be slightly outside the dense normal user cluster. If, on the other hand, you are more interested in generally outlying points then merely selecting the noise points from the clustering will pick out points in sparser regions of the space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally note that selecting noise points gives you only a single threshold; in contract GLOSH provides a range of scores and you can tune the threshold to find the number of outliers you desire.&lt;/p&gt;&#xA;" OwnerUserId="18513" LastActivityDate="2017-11-06T04:16:15.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24392" PostTypeId="1" AcceptedAnswerId="24393" CreationDate="2017-11-06T06:15:29.570" Score="15" ViewCount="4404" Body="&lt;p&gt;I would like to know &lt;em&gt;why&lt;/em&gt; we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue - by up sampling or down sampling or by using SMOTE.&lt;/p&gt;&#xA;&lt;p&gt;For example, if I have a rare disease 1 percent out of 100, and lets say I decided to have a balanced data set for my training set which is:  50/50 sample&#xA;won't that make the machine think 50% of patients will have disease? even though the ratio is 1 of 100.&#xA;So&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Why do we need to deal with data imbalance?&lt;/li&gt;&#xA;&lt;li&gt;What is the recommended ratio to have balance set?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="32220" LastEditorUserId="29169" LastEditDate="2021-03-13T20:14:07.490" LastActivityDate="2021-03-13T20:14:07.490" Title="Why do we need to handle data imbalance?" Tags="|classification|dataset|sampling|class-imbalance|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24393" PostTypeId="2" ParentId="24392" CreationDate="2017-11-06T06:48:29.507" Score="8" Body="&lt;p&gt;You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy, get 99% accuracy and go home early.&#xA;The problem with such a model is that though it has high accuracy, it will probably not what you are looking for. Most of the time you want to find the people with the disease (giving high weigh to false negatives) much more than you are afraid to send an healthy person to unneeded test (low weight to false positives). In a real world health problem the ratio between the weight can easily be 1 to 1,000. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The imbalance in the distribution fails most algorithms from finding a proper solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are correct that just balancing the distribution isn't the optimal solution. Indeed, an algorithm that is trained on a balanced distribution is not fitted to the natural distribution on which it will be evaluated.&#xA;My favorite method is adapting it back, as you can see &lt;a href=&quot;https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set/answer/DaL-9&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;For a discussion, see &lt;a href=&quot;https://datascience.stackexchange.com/questions/810/should-i-go-for-a-balanced-dataset-or-a-representative-dataset/8628#8628&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just setting the ratio to some other problem won't work since you will have the same problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;noreferrer&quot;&gt;Smote&lt;/a&gt; is working in a different way, which didn't work as well when I tried it, but it might fit your problem.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2017-11-08T11:46:05.263" LastActivityDate="2017-11-08T11:46:05.263" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="24394" PostTypeId="2" ParentId="24369" CreationDate="2017-11-06T07:48:51.303" Score="2" Body="&lt;p&gt;Applying any non-linear model in a &lt;a href=&quot;http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/&quot; rel=&quot;nofollow noreferrer&quot;&gt;model stacking&lt;/a&gt; approach should do what you want. In brief the approach is to take predictions from other models as new features, plus the original data and labels, then use them to train a meta-model. Read the link, it offers practical advice on how to do this within a k-fold validation framework, which will give it a much better chance of doing well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-linear models that combine simpler units - e.g. neural networks and models that use multiple decision trees (e.g. xgboost) - already perform this kind of internal split during training. So if you are already using those, you might not gain such a big improvement over simpler ensemble techniques, such as taking a mean or weighted mean over models.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-11-06T07:48:51.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24396" PostTypeId="1" AcceptedAnswerId="24413" CreationDate="2017-11-06T08:16:36.950" Score="1" ViewCount="3070" Body="&lt;p&gt;I am trying to calculate coverage metrics for a recommender system that I have designed. &lt;a href=&quot;https://gab41.lab41.org/recommender-systems-its-not-all-about-the-accuracy-562c7dceeaff&quot; rel=&quot;nofollow noreferrer&quot;&gt;This &lt;/a&gt; blog post talks about how to do it. I had some difficulties in understanding the same. It says that &lt;em&gt;These metrics will be for user u of &lt;strong&gt;U&lt;/strong&gt; users over n items out of &lt;strong&gt;N&lt;/strong&gt; potential items in a recommendation list L. Each item has a content vector &lt;strong&gt;C&lt;/strong&gt; of length c ...&lt;/em&gt; My difficulty arises in understanding &lt;strong&gt;N&lt;/strong&gt; and &lt;strong&gt;U&lt;/strong&gt;. Is &lt;strong&gt;U&lt;/strong&gt; the total number of users? Secondly, what does &lt;strong&gt;N&lt;/strong&gt; potential items mean? Is it the list of all items that are present or is it a subset of all the items present?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the Python LightFM package to generate the recommendations&lt;/p&gt;&#xA;" OwnerUserId="14360" LastActivityDate="2017-11-06T17:43:14.447" Title="How to calculate coverage in recommender systems?" Tags="|python|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24397" PostTypeId="2" ParentId="16807" CreationDate="2017-11-06T08:24:09.693" Score="17" Body="&lt;p&gt;Memory is &lt;em&gt;not&lt;/em&gt; really the reason for doing this, because you &lt;em&gt;could&lt;/em&gt; just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reasons that SGD is used so widely are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Efficiency.  Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation.  Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Generalization.  It seems (see &lt;a href=&quot;https://cbmm.mit.edu/sites/default/files/publications/CBMM-Memo-067.pdf&quot; rel=&quot;noreferrer&quot;&gt;Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD&lt;/a&gt;) that SGD actually helps generalization by finding &quot;flat&quot; minima on the training set, which are more likely to also be minima on the test set.  Intuitively, we can think of SGD as a sort of &lt;a href=&quot;https://en.wikipedia.org/wiki/Bootstrap_aggregating&quot; rel=&quot;noreferrer&quot;&gt;Bagging&lt;/a&gt; - by computing our parameters based on many minibatches of the data, we reenforce rules that generalize across minibatches, and cancel rules that don't, thereby making us less prone to overfitting to the training set.&lt;/p&gt;&#xA;" OwnerUserId="41482" LastEditorUserId="41482" LastEditDate="2017-12-27T01:56:53.260" LastActivityDate="2017-12-27T01:56:53.260" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24399" PostTypeId="2" ParentId="24376" CreationDate="2017-11-06T09:46:01.553" Score="5" Body="&lt;p&gt;Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['Reviews']=[&quot; &quot;.join(review) for review in df['Reviews'].values]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then run the vectorizer again. That will fix the problem. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-11-06T09:46:01.553" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="24400" PostTypeId="1" CreationDate="2017-11-06T09:47:21.450" Score="1" ViewCount="83" Body="&lt;p&gt;Working on a binary classification problem that tries to predict customer churn, the data set is imbalanced with 2000 observations of non-churn cases vs 600 observations of churn cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On using GLM I see that when the majority class[Non-churn] is the reference level I get ~40 % error rate[confusion matrix] on both the levels [churn non-churn] but when the minority class is set as the reference level I get 100% error rate in predicting the minority class or in a way everything gets predicted as non-churn case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After balancing the data using SMOTE the same trend continues, &lt;em&gt;how should I interpret this behaviour.&lt;/em&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it in a way saying that the non-churn population has users who have similar behaviour as the churners and hence the high error rate, but at the same time the non-churn users have a subset which are quite different than the churners in their behaviour and hence lower error rate when the reference class is the majority or the non-churn class.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Outcome on test data when majority class is set as the reference class:&#xA;Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:&#xA;             0   1    Error      Rate&#xA;    0      268 419 0.609898  =419/687&#xA;    1       46 168 0.214953   =46/214&#xA;    Totals 314 587 0.516093  =465/901&#xA;&#xA;Outcome on test data minority class is set as the reference class:&#xA;Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:&#xA;                   1   0    Error      Rate&#xA;            1      3 211 0.985981  =211/214&#xA;            0      1 686 0.001456    =1/687&#xA;            Totals 4 897 0.235294  =212/901&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3360" LastEditorUserId="3360" LastEditDate="2017-11-07T04:52:26.063" LastActivityDate="2017-11-07T16:12:40.960" Title="Changing reference class in imbalanced data drastically affects the error rate" Tags="|logistic-regression|glm|class-imbalance|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24401" PostTypeId="1" CreationDate="2017-11-06T10:21:24.577" Score="2" ViewCount="99" Body="&lt;p&gt;Processing language data with deep learning models often involves a lookup of a pre-trained embedding model. In the model development phase, it's very annoying that every time the entire embeddings are loaded, as embeddings could be very big (e.g., &lt;code&gt;Glove&lt;/code&gt;) and would consume a lot of timing loading it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to build/find a smaller and workable embedding just for debugging?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now what I've done is shrink the dimensions of &lt;code&gt;Glove&lt;/code&gt;, for example, reduce the dimension of embeddings from &lt;code&gt;300d&lt;/code&gt; to &lt;code&gt;30d&lt;/code&gt; by taking the first 30 dims, but that might potentially induce a risk that some of words will share the same embedding. &lt;/p&gt;&#xA;" OwnerUserId="41487" LastEditorUserId="41487" LastEditDate="2017-11-06T23:12:43.227" LastActivityDate="2017-11-07T00:51:42.157" Title="How to develop small workable embeddings for debugging" Tags="|deep-learning|nlp|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24402" PostTypeId="1" CreationDate="2017-11-06T11:03:51.883" Score="5" ViewCount="3700" Body="&lt;p&gt;I'm trying to modify the Doc2vec &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-IMDB.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; to calculate cosine similarity and take &lt;code&gt;Pandas&lt;/code&gt; dataframes instead of &lt;code&gt;.txt&lt;/code&gt; documents. I want to find the most similar sentence to a new sentence I put in from my data. However, after training, even if I give almost the same sentence that's present in the dataset, I get low-accuracy results as the top result and none of them is the sentence I modified. For example I have the sentence &quot;This is a nice cat you have.&quot; in the dataset I train Doc2vec with, then I use the new sentence &quot;This cat you have is quite nice.&quot; as input, and it doesn't bring up the first sentence as similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data comes from an excel sheet, and has roughly the looks of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Description                  | Group        | Number&#xA;0 Sent: This is a sentence       Regular        NUM1234&#xA;1 Sent: Another sentence         Regular        NUM1243&#xA;2 Sent: Basically all the input  Other group    NUM1278&#xA;3 Sent: Creating a test case to validate the routing between applications.  No action needed at this moment &#xA;                               | Other group  | NUM1287&#xA;...etc...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have the following code (some code not needed for comprehension was trimmed):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_excel(&quot;my_data.xls&quot;)&#xA;&#xA;df[&quot;Description&quot;] = df[&quot;Description&quot;].apply(lambda x: removeGeneric(x)) #removeGeneric() just strips &quot;Sent:&quot; from the beginning of each sentence &#xA;for index, row in df.iterrows():&#xA;    row[&quot;Description&quot;] = row[&quot;Description&quot;].lower()&#xA;    row[&quot;Description&quot;] = normalize_text(row[&quot;Description&quot;]) #normalize_text() removes stopwords defined in the nltk package and words shorter than 2 characters&#xA;&#xA;SentimentDocument = namedtuple('SentimentDocument', 'words tags')&#xA;&#xA;alldocs = []  &#xA;for index, row in df.iterrows():&#xA;    words = gensim.utils.to_unicode(row[&quot;Description&quot;]).split()&#xA;    tags = [row[&quot;Number&quot;]]&#xA;    alldocs.append(SentimentDocument(words, tags))&#xA;&#xA;doc_list = alldocs[:]&#xA;cores = multiprocessing.cpu_count()&#xA;assert gensim.models.doc2vec.FAST_VERSION &amp;gt; -1, &quot;This will be painfully slow otherwise&quot;&#xA;&#xA;simple_models = [&#xA;    # PV-DM w/ concatenation - window=5 (both sides) approximates paper's 10-word total window size&#xA;    Doc2Vec(dm=1, dm_concat=1, size=100, window=5, negative=5, hs=0, min_count=2, workers=cores),&#xA;    # PV-DBOW &#xA;    Doc2Vec(dm=0, size=100, negative=5, hs=0, min_count=2, workers=cores),&#xA;    # PV-DM w/ average&#xA;    Doc2Vec(dm=1, dm_mean=1, size=100, window=10, negative=5, hs=0, min_count=2, workers=cores),&#xA;]&#xA;&#xA;# Speed up setup by sharing results of the 1st model's vocabulary scan&#xA;simple_models[0].build_vocab(alldocs)  # PV-DM w/ concat requires one special NULL word so it serves as template&#xA;print(simple_models[0])&#xA;for model in simple_models[1:]:&#xA;    model.reset_from(simple_models[0])&#xA;    print(model)&#xA;&#xA;models_by_name = OrderedDict((str(model), model) for model in simple_models)&#xA;&#xA;from random import shuffle&#xA;&#xA;alpha, min_alpha, passes = (0.025, 0.001, 20)&#xA;alpha_delta = (alpha - min_alpha) / passes&#xA;&#xA;print(&quot;START %s&quot; % datetime.datetime.now())&#xA;&#xA;for epoch in range(passes):&#xA;    shuffle(doc_list)&#xA;&#xA;    for name, train_model in models_by_name.items():&#xA;        # Train&#xA;        duration = 'na'&#xA;        train_model.alpha, train_model.min_alpha = alpha, alpha&#xA;        with elapsed_timer() as elapsed:&#xA;            train_model.train(doc_list, total_examples=len(doc_list), epochs=1)&#xA;&#xA;for model in simple_models:&#xA;    new_sentence = &quot;Test case creation to validation of routing between applications.  No action needed&quot; #Notice how I'm testing with a sentence very similar to one in the original dataset&#xA;    new_sentence = removeGeneric(new_sentence)&#xA;    new_sentence = normalize_text(new_sentence)&#xA;    print(model.docvecs.most_similar(positive=[model.infer_vector(new_sentence)],topn=2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this I get the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[('NUM1254', 0.3154909014701843), ('NUM5247', 0.2487245500087738)]&#xA;[('NUM3875', 0.20226456224918365), ('NUM3793', 0.1970052272081375)]&#xA;[('NUM3585', 0.13086965680122375), ('NUM3857', 0.1298370361328125)]&#xA;creating test case validate routing applications action needed moment&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All the recommendations are completely unrelated, sentences like &quot;site id plant address good owner electricity request approved number al district province&quot; show up; the sentence it's actually close to (the sentence &quot;Creating a test case to validate the routing between applications.  No action needed at this moment&quot; from the dataset) is not on the list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you see anything that I'm doing wrong? What could I do to improve accuracy? Has anyone else experienced this inaccuracy in doc2vec's cosine similarity prediction? If I hand-code the implementation (&lt;a href=&quot;https://gist.github.com/f00-/624751d91b98845184fcccf500bdfb33&quot; rel=&quot;nofollow noreferrer&quot;&gt;like this for example&lt;/a&gt;), it does give the correct answers, which are completely different than those from doc2vec (but actually accurate).&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="38887" LastEditDate="2019-12-07T11:16:48.083" LastActivityDate="2019-12-14T07:03:12.270" Title="Doc2vec to calculate cosine similarity - absolutely inaccurate" Tags="|python|nlp|similarity|text|gensim|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="24403" PostTypeId="1" AcceptedAnswerId="24404" CreationDate="2017-11-06T12:16:39.393" Score="17" ViewCount="17192" Body="&lt;p&gt;Is there a method to calculate the prediction interval (probability distribution) around a time series forecast from an LSTM (or other recurrent) neural network? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say, for example, I am predicting 10 samples into the future (t+1 to t+10), based on the last 10 observed samples (t-9 to t), I would expect the prediction at t+1 to be more accurate than the prediction at t+10. Typically, one might draw error bars around the prediction to show the interval. With an ARIMA model (under the assumption of normally distributed errors), I can calculate a prediction interval (e.g. 95%) around each predicted value. Can I calculate the same, (or something that relates to the prediction interval) from an LSTM model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm been working with LSTMs in Keras/Python, following lots of examples from &lt;a href=&quot;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;noreferrer&quot;&gt;machinelearningmastery.com&lt;/a&gt;, from which my example code (below) is based on. I'm considering reframing the problem as classification into discrete bins, as that produces a confidence per class, but that seems a poor solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of similar topics (such as the below), but nothing seems to directly address the issue of prediction intervals from LSTM (or indeed other) neural networks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/25055/how-to-calculate-the-confidence-interval-for-time-series-prediction&quot;&gt;https://stats.stackexchange.com/questions/25055/how-to-calculate-the-confidence-interval-for-time-series-prediction&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/12721/time-series-prediction-using-arima-vs-lstm&quot;&gt;Time series prediction using ARIMA vs LSTM&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import LSTM&#xA;from math import sin&#xA;from matplotlib import pyplot&#xA;import numpy as np&#xA;&#xA;# Build an LSTM network and train&#xA;def fit_lstm(X, y, batch_size, nb_epoch, neurons):&#xA;    X = X.reshape(X.shape[0], 1, X.shape[1]) # add in another dimension to the X data&#xA;    y = y.reshape(y.shape[0], y.shape[1])      # but don't add it to the y, as Dense has to be 1d?&#xA;    model = Sequential()&#xA;    model.add(LSTM(neurons, batch_input_shape=(batch_size, X.shape[1], X.shape[2]), stateful=True))&#xA;    model.add(Dense(y.shape[1]))&#xA;    model.compile(loss='mean_squared_error', optimizer='adam')&#xA;    for i in range(nb_epoch):&#xA;        model.fit(X, y, epochs=1, batch_size=batch_size, verbose=1, shuffle=False)&#xA;        model.reset_states()&#xA;    return model&#xA;&#xA;# Configuration&#xA;n = 5000    # total size of dataset&#xA;SLIDING_WINDOW_LENGTH = 30&#xA;SLIDING_WINDOW_STEP_SIZE = 1&#xA;batch_size = 10&#xA;test_size = 0.1 # fraction of dataset to hold back for testing&#xA;nb_epochs = 100 # for training&#xA;neurons = 8 # LSTM layer complexity&#xA;&#xA;# create dataset&#xA;#raw_values = [sin(i/2) for i in range(n)]  # simple sine wave&#xA;raw_values = [sin(i/2)+sin(i/6)+sin(i/36)+np.random.uniform(-1,1) for i in range(n)]  # double sine with noise&#xA;#raw_values = [(i%4) for i in range(n)] # saw tooth&#xA;&#xA;all_data = np.array(raw_values).reshape(-1,1) # make into array, add anothe dimension for sci-kit compatibility&#xA;&#xA;# data is segmented using a sliding window mechanism&#xA;all_data_windowed = [np.transpose(all_data[idx:idx+SLIDING_WINDOW_LENGTH]) for idx in np.arange(0,len(all_data)-SLIDING_WINDOW_LENGTH, SLIDING_WINDOW_STEP_SIZE)]&#xA;all_data_windowed = np.concatenate(all_data_windowed, axis=0).astype(np.float32)&#xA;&#xA;# split data into train and test-sets&#xA;# round datasets down to a multiple of the batch size&#xA;test_length = int(round((len(all_data_windowed) * test_size) / batch_size) * batch_size)&#xA;train, test = all_data_windowed[:-test_length,:], all_data_windowed[-test_length:,:]&#xA;train_length = int(np.floor(train.shape[0] / batch_size)*batch_size) &#xA;train = train[:train_length,...]&#xA;&#xA;half_size = int(SLIDING_WINDOW_LENGTH/2) # split the examples half-half, to forecast the second half&#xA;X_train, y_train = train[:,:half_size], train[:,half_size:]&#xA;X_test, y_test = test[:,:half_size], test[:,half_size:]&#xA;&#xA;# fit the model&#xA;lstm_model = fit_lstm(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epochs, neurons=neurons)&#xA;&#xA;# forecast the entire training dataset to build up state for forecasting&#xA;X_train_reshaped = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])&#xA;lstm_model.predict(X_train_reshaped, batch_size=batch_size)&#xA;&#xA;# predict from test dataset&#xA;X_test_reshaped = X_test.reshape(X_test.shape[0], 1, X_test.shape[1])&#xA;yhat = lstm_model.predict(X_test_reshaped, batch_size=batch_size)&#xA;&#xA;#%% Plot prediction vs actual&#xA;&#xA;x_axis_input = range(half_size)&#xA;x_axis_output = [x_axis_input[-1]] + list(half_size+np.array(range(half_size)))&#xA;&#xA;fig = pyplot.figure()&#xA;ax = fig.add_subplot(111)&#xA;line1, = ax.plot(x_axis_input,np.zeros_like(x_axis_input), 'r-')&#xA;line2, = ax.plot(x_axis_output,np.zeros_like(x_axis_output), 'o-')&#xA;line3, = ax.plot(x_axis_output,np.zeros_like(x_axis_output), 'g-')&#xA;ax.set_xlim(np.min(x_axis_input),np.max(x_axis_output))&#xA;ax.set_ylim(-4,4)&#xA;pyplot.legend(('Input','Actual','Predicted'),loc='upper left')&#xA;pyplot.show()&#xA;&#xA;# update plot in a loop&#xA;for idx in range(y_test.shape[0]):&#xA;&#xA;    sample_input = X_test[idx]&#xA;    sample_truth = [sample_input[-1]] + list(y_test[idx]) # join lists&#xA;    sample_predicted = [sample_input[-1]] + list(yhat[idx])&#xA;&#xA;    line1.set_ydata(sample_input)&#xA;    line2.set_ydata(sample_truth)&#xA;    line3.set_ydata(sample_predicted)&#xA;    fig.canvas.draw()&#xA;    fig.canvas.flush_events()&#xA;&#xA;    pyplot.pause(.25)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29109" LastActivityDate="2023-09-03T15:15:05.023" Title="Prediction interval around LSTM time series forecast" Tags="|machine-learning|deep-learning|time-series|prediction|lstm|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24404" PostTypeId="2" ParentId="24403" CreationDate="2017-11-06T12:47:08.810" Score="12" Body="&lt;p&gt;Directly, this is not possible. However, if you model it in a different way you can get out confidence intervals. You could instead of a normal regression approach it as estimating a continuous probability distribution. By doing this for every step you can plot your distribution. Ways to do this are Kernel Mixture Networks (&lt;a href=&quot;https://janvdvegt.github.io/2017/06/07/Kernel-Mixture-Networks.html&quot; rel=&quot;noreferrer&quot;&gt;https://janvdvegt.github.io/2017/06/07/Kernel-Mixture-Networks.html&lt;/a&gt;, disclosure, my blog) or Density Mixture Networks (&lt;a href=&quot;http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.7-MixDensityNetworks.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.7-MixDensityNetworks.pdf&lt;/a&gt;), the first uses kernels as base and estimates a mixture over these Kernels and the second one estimates a mixture of distributions, including the parameters of each of the distributions. You use the log likelihood for training the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option for modeling the uncertainty is to use dropout during training and then also during inference. You do this multiple times and every time you get a sample from your posterior. You don't get distributions, only samples, but it's the easiest to implement and it works very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you have to think about the way you generate t+2 up to t+10. Depending on your current setup you might have to sample from the previous time step and feed that for the next one. That doesn't work very well with the first approach, nor with the second. If you have 10 outputs per time step (t+1 up to t+10) then all of these approaches are more clean but a bit less intuitive.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-06T12:47:08.810" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24405" PostTypeId="1" CreationDate="2017-11-06T12:58:58.223" Score="13" ViewCount="57499" Body="&lt;p&gt;I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But f_regression does not do stepwise regression but only give F-score and pvalues corresponding to each of the regressors, which is only the first step in stepwise regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What to do after 1st regressors with the best f-score is chosen?&lt;/p&gt;&#xA;" OwnerUserId="41490" LastEditorUserId="29575" LastEditDate="2017-11-06T15:47:03.400" LastActivityDate="2017-11-07T15:48:35.280" Title="How to do stepwise regression using sklearn?" Tags="|machine-learning|scikit-learn|regression|feature-selection|linear-regression|" AnswerCount="1" CommentCount="1" ClosedDate="2017-11-18T12:06:34.513" ContentLicense="CC BY-SA 3.0" />
  <row Id="24408" PostTypeId="1" CreationDate="2017-11-06T14:36:45.360" Score="1" ViewCount="1831" Body="&lt;p&gt;I'm training a model (NN) that gets some data as input and outputs a single value in the range of $[0, 1]$. Right now, the average of the outputs in my dataset is around 0.5, but I know that future data will largely consist of 0.0s, and thus there will eventually be a strong data imbalance towards 0.0. I want the training procedure to be future-proof and scalable, so I'm trying to find a way to automatically rebalance the dataset. My library (Keras) supports sample weights in training, which seems like a straightforward way to do this without losing any information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I think what I'm looking for is a function $w(y_i) \rightarrow w_i$ that given a training example $y_i$ gives me a weight $w_i$, so that the weighted average of all training examples $Y$ with the weights $W$ is $0.5$. I know there are many configurations of weights that have this property, but of course the weights should be as close to 1 as possible and definitely $&amp;gt; 0$. I also realize that this is not possible for cases where e.g. all numbers are the same, or all are $&amp;lt; 0.5$. But let's assume my data is diverse enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm sure I'm not the first one to think of this, but I can't find any solutions / best practices for this case. I guess I could treat it as a little optimization problem of its own, but I hope there is something simpler.&lt;/p&gt;&#xA;" OwnerUserId="38744" LastActivityDate="2017-11-08T20:23:00.013" Title="How to find optimal sample weights in binary classification/regression problem" Tags="|regression|class-imbalance|normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24410" PostTypeId="2" ParentId="24401" CreationDate="2017-11-06T15:57:23.010" Score="1" Body="&lt;p&gt;The embeddings are floating point vectors, it's very unlikely that they are exactly the same. And even if they were, it will only minorly impact the performance of your model, which is not what you are after anyway, because you are debugging. That said, if you want to use it for model selection you have a few alternatives. The easiest one is to apply PCA one time to your whole embedding space and take the first &lt;strong&gt;n&lt;/strong&gt; dimensions. This way you reduce your dimensionality without throwing away too much information. A better but more difficult approach would be to train your own Glove vectors, either on a public corpus or a private one, and set a low dimensionality during the training procedure. It might be difficult to use your findings on these embeddings for model selection because more expressiveness could lead to much better performance with deeper models or could lead to more overfitting.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-06T15:57:23.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24411" PostTypeId="2" ParentId="24408" CreationDate="2017-11-06T17:04:12.380" Score="1" Body="&lt;p&gt;There are a few things which are unclear so I am going to have to make some assumptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You say it is binary classification/regression. Are you trying to find the probability of each binary class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say &quot;future data&quot;, do you mean future training data or future test data? If it is test data then you do not need to worry at all about his effect. If it is future training data then yes you may have a problem. You do not want to do model building on a training set that is different from the training set you would actually use in the productionized model. I would suggest you produce a number of validation plots for this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really are doing binary classification. Most models in python have a method (eg &lt;strong&gt;predict_proba()&lt;/strong&gt; ) that will produce a probability and will let you decide on class based on that probability. The default being 50%. For this reason I do not think that you want to have the weighted average be 0.5 but the &quot;probability&quot; represent the true probability. Please refer to this tutorial &lt;a href=&quot;http://scikit-learn.org/stable/modules/calibration.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/calibration.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to your original question. You want the weights to be $M/N_i$ where $N_i$ is the number of samples in each class $i$ and $M$ the total number of samples. This is a way to do some sort of pseudo-upsampleing and have the classes be weighted by the number of samples. &lt;/p&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2017-11-06T17:04:12.380" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24412" PostTypeId="1" CreationDate="2017-11-06T17:31:57.403" Score="1" ViewCount="2030" Body="&lt;p&gt;I'm wondering if there has been some work done about using autoencoder versus using word2vec to produce word embeddings. Autoencoder could learn to map contexts words with themselves while word2vec usually maps one word with its context or a context with its word. Did you see some work comparing these two approaches ?  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-11-06T17:31:57.403" Title="Autoencoders versus Word2Vec?" Tags="|word2vec|word-embeddings|autoencoder|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24413" PostTypeId="2" ParentId="24396" CreationDate="2017-11-06T17:43:14.447" Score="3" Body="&lt;p&gt;Right, the capital letters denote the total available. In that blog post: &lt;strong&gt;U&lt;/strong&gt; means all users, &lt;strong&gt;N&lt;/strong&gt; means all items but in other places is usually written &lt;strong&gt;I&lt;/strong&gt;, and &lt;strong&gt;L&lt;/strong&gt; means all top-n recommendation lists. &quot;Top-n&quot; means that the recommender system outputs a ranked list of n items, so if you had 1000 users all getting a Top-10 list, you'd have &lt;strong&gt;L&lt;/strong&gt; length of 1000*10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you read Ge, Mouzhi, Carla Delgado-Battenfeld, and Dietmar Jannach. &quot;Beyond accuracy: evaluating recommender systems by coverage and serendipity.&quot; Proceedings of the fourth ACM conference on Recommender systems. ACM, 2010, at &lt;a href=&quot;http://ls13-www.cs.tu-dortmund.de/homepage/publications/jannach/Conference_RECSYS10b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ls13-www.cs.tu-dortmund.de/homepage/publications/jannach/Conference_RECSYS10b.pdf&lt;/a&gt; instead. It's a more clearly written definition of metrics and uses more traditional recommender system notation.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-06T17:43:14.447" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24414" PostTypeId="1" CreationDate="2017-11-06T18:25:24.187" Score="3" ViewCount="544" Body="&lt;p&gt;I had the following idea of Reinforcement Learning:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;There is the RL agent and the environment. The agent can execute actions, sends them to the environment and receives an observation and a reward (a float).&lt;/li&gt;&#xA;&lt;li&gt;Accounting problem: The reward might be delayed.&lt;/li&gt;&#xA;&lt;li&gt;Learning algorithms: Q-learning is one example. It basically tries to learn the mapping between State x Action to reward and works &lt;a href=&quot;https://martin-thoma.com/probabilistische-planung/#q-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;like this&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can Reinforcement Learning work for a setting similar to Dutch auctions?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The agent can make 3 actions: Increase, keep or lower the price&lt;/li&gt;&#xA;&lt;li&gt;The agents goal is to sell the item for the highest price. So the agent is the auctioneer.&lt;/li&gt;&#xA;&lt;li&gt;The auction item does not change. The bidders are anonymous and thus don't give a hint about their behavior.&lt;/li&gt;&#xA;&lt;li&gt;This means the observations are the same, only the time to the end gets lower - independently of the action of the agent.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-06T20:29:36.193" LastActivityDate="2017-11-06T22:03:38.950" Title="Can Reinforcement Learning work for Dutch auctions?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24417" PostTypeId="1" AcceptedAnswerId="24475" CreationDate="2017-11-06T19:51:59.263" Score="5" ViewCount="6540" Body="&lt;p&gt;I was wondering if it was okay to use torch.cat within my forward function. I am doing so because I want the first two columns of my input to skip the middle hidden layers and go directly to the final layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code: you can see that I use torch.cat at the last moment to make xcat. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the gradient propagate back? or does the torch.cat cover up what happened to my hidden variables?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class LinearRegressionForce(nn.Module):&#xA;&#xA;    def __init__(self, focus_input_size, rest_input_size, hidden_size_1, hidden_size_2, output_size):&#xA;        super(LinearRegressionForce, self).__init__()&#xA;&#xA;        self.in1 = nn.Linear(rest_input_size, hidden_size_1) &#xA;        self.middle1 = nn.Linear(hidden_size_1,hidden_size_2)&#xA;        self.out4 = nn.Linear(focus_input_size + hidden_size_2,output_size)&#xA;&#xA;    def forward(self, inputs):&#xA;        focus_inputs = inputs[:,0:focus_input_size]&#xA;        rest_inputs = inputs[:,focus_input_size:(rest_input_size+focus_input_size)]&#xA;        x = self.in1(rest_inputs).clamp(min=0)&#xA;        x = self.middle1(x).clamp(min=0)&#xA;        xcat = torch.cat((focus_inputs,x),1)&#xA;        out = self.out4(xcat).clamp(min=0)&#xA;        return out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I call it like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rest_inputs = Variable(torch.from_numpy(rest_x_train))&#xA;focus_x_train_ones = np.concatenate((focus_x_train, np.ones((n,1))), axis=1)&#xA;focus_inputs =  Variable(torch.from_numpy(focus_x_train_ones)).float()&#xA;inputs = torch.cat((focus_inputs,rest_inputs),1)&#xA;&#xA;predicted = model(inputs).data.numpy()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41508" LastEditorUserId="29575" LastEditDate="2017-11-06T21:43:14.733" LastActivityDate="2017-11-08T06:53:25.917" Title="Does torch.cat work with backpropagation?" Tags="|pytorch|torch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24418" PostTypeId="2" ParentId="5443" CreationDate="2017-11-06T19:58:32.170" Score="7" Body="&lt;p&gt;I'm surprised how many people are attached to the coolness of the profession rather than the actual job to be done. Excel is excellent tool, with free Powerpivot, Powerquery, it can do so much. (these are not available on OS X). And if you know VBA, you can do some nice stuff. And then if you add on the top of that knowledge of python you can combine the very first steps of data extraction and manipulation with python and then use excel, especially if you are a visual person. With excel you can really inspect aggregated data before feeding into any further processes or visualizing. Its a must have tool.&lt;/p&gt;&#xA;" OwnerUserId="41509" LastActivityDate="2017-11-06T19:58:32.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24419" PostTypeId="2" ParentId="24259" CreationDate="2017-11-06T20:41:47.177" Score="1" Body="&lt;p&gt;If you have gender data, use it. You're right that it will probably be a binary dummy feature (i.e., male = 0 or 1, and so on). You can use gender directly as a predictive feature, and/or interact it with other variables. You don't need to split your dataset by gender. Adding &lt;em&gt;arbitrary information&lt;/em&gt; increases the noise in your set and decreases model quality. But you can learn whether gender is arbitrary or not! Add gender and compare to a model without gender to see if there's any change.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-06T20:41:47.177" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24420" PostTypeId="2" ParentId="24414" CreationDate="2017-11-06T21:09:32.643" Score="2" Body="&lt;p&gt;Whether or not a Dutch Auction can be framed as a reinforcement learning problem depends on whether there is meaningful state that can be changed by the auctioneer's actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the bidders are behaving in a simplified rational manner* - e.g. they are working to a simple mechanic of buying the item when it reaches a price they have decided to pay - then the behaviour of the auctioneer will not make much difference, it is very formulaic as it is already part of a price optimising strategy. There is no meaningful state change between actions - the bidder valuations and attention stay fixed no matter what the auctioneer does. In an extreme, automated case, the optimal action would be to set a very high initial price and then reduce it in many small increments so as to trigger the highest purchase possible from bidders. This is not done in real auctions because it is a waste of everyone's time (and it would be weird, socially). However, it shows you a problem with the framing of the problem. In a basic rational setting, with nothing to gain or lose apart from money and value of the item, there is little room for altering strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem becomes far more interesting when bidders are not purely rational optimisers themselves (or have some other unknown values related to the auction such as personal time they are willing to invest in the process, or rivalry with other bidders), and when some information is available about their past behaviour, or at least in general about &lt;em&gt;human&lt;/em&gt; behaviour. In that case the problem can modelled as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Partially Observable Markov Decision Process (POMDP)&lt;/a&gt;, and you can concern yourself with rewards that might vary due to your actions - e.g. risk of making bidders skittish with &quot;odd&quot; behaviour as the auctioneer. There are limits to this due to the anonymous bidders. If you kept bidder identity between items (if a single auction consisted if multiple item sales with the same bidders) then you might gain some information on each sale in order to optimise better your actions within that auction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible way to have stateful behaviour is if item ownership has value, such that market forces of supply/demand come into effect (once a bidder has one item maybe they value the next one less, or maybe more if multiple items have higher value to the bidder when kept together). Again this is only going to be relevant to the auctioneer if bidder identities remain constant for a multi-item auction, and even then only makes a difference if there is non-rational pricing behaviour in the bidders.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problem you will have with training RL auctioneer to work with non-rational bidders, is modelling those bidders. You could maybe bootstrap with data from real auctions with human auctioneers - this can still be done in a Q learning framework. Q-Learning can learn about optimal policy from observing non-optimal behaviour within limits, optimal actions still need to be observed, but don't have to all be in the same episodes. However, in general to improve from that model and apply reinforcement learning, you have to you put it in control. When it explored unusual actions in order to learn, that could make it a liability if real money was at stake - to defend against this you may need to keep action choice limited.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Staying with &quot;purely rational&quot; approaches, you could get more sophisticated. You could maybe model the bidders as trying to predict the auctioneer and other bidder's next action, and extract the best expected value, in which case you have framed the problem more like adversarial game theory. This could make the auctioneer's actions have more impact. The most interesting part of this is occurring in the bidder though, not the auctioneer, as the bidder would have to incorporate a meaningful value to owning the item and be smart enough to second-guess other bidder's valuations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The auctioneer's goal would then be to find the best price that the highest bidder was willing to pay, and behave so that the highest bidder predicted that a rival would bid on the next lower increment. This would be very similar to the non-rational approach in practice, but more tractable to do in simulation - you would need a good simulated model of the bidders before you could train an auctioneer, and the bidders would probably need to be RL or similar too, with both auctioneer and bidder models evolving together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, and assuming the game theory model was stable here, I would expect the auctioneer, with no other information to go on than the number of bidders, would have an optimal strategy where selling price decreased in increments inversely sized to some fractional power of the number of bidders, as more rivals means that bidders will predict rivals willing to pay values closer to their own evaluation. A partially stochastic policy might also be favourable to the auctioneer, as being unpredictable means that bidders cannot rely on guessing the next price reduction accurately (although they might counter that by choosing whether to bid stochastically).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-06T22:03:38.950" LastActivityDate="2017-11-06T22:03:38.950" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24421" PostTypeId="1" CreationDate="2017-11-06T21:34:00.590" Score="1" ViewCount="204" Body="&lt;p&gt;What sort of models do they use?  Presumably some flavor of neural net.  Do they do a lot of feature engineering?  Or do they throw in huge raw matrices of oscilloscope output?  Given that it must be supervised learning, what is their target?  A thumbs up?  How do they deal with heterogeneity in the population? We're not independent nor are we identically distributed.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like it works better lately.  Curious how it works.&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-11-06T21:43:12.187" Title="Does anyone know how google play music's recommendation engine works?" Tags="|recommender-system|google|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24422" PostTypeId="2" ParentId="24421" CreationDate="2017-11-06T21:43:12.187" Score="1" Body="&lt;p&gt;Deep neural networks, probably with tens of internal layers, distributed across many many machines. Each user probably receives scores from many deep networks for various applications, and each network is probably automatically scored to determine when its particular model is most helpful. This is the pattern used across the industry. Music companies have the additional complexity of trying to parse sound data. Spotify does this with their Release Radar recommender.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as targets go, they probably build models to predict all forms of engagement. You probably get scores for listens, scores for time listening, scores for thumbs up or down, scores for add-to-list, and so on. Ensemble methods work well in practice and Google probably gives you on the order of dozens to hundreds of different machine-learned predictions.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-06T21:43:12.187" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24425" PostTypeId="1" AcceptedAnswerId="24437" CreationDate="2017-11-06T22:19:55.077" Score="0" ViewCount="785" Body="&lt;p&gt;I recently learned about &lt;a href=&quot;https://en.wikipedia.org/wiki/Goodhart%27s_law&quot; rel=&quot;nofollow noreferrer&quot;&gt;Goodhart's Law&lt;/a&gt;. Simply put,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When a measure becomes a target, it ceases to be a good measure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, in Data Science, we really do aim at improving our performance by increasing or decreasing a metric, and improve our models based on that. For instance, in Kaggle competitions. Is Goodhart's Law applicable to Data Science? Why or why not?&lt;/p&gt;&#xA;" OwnerUserId="20589" LastEditorUserId="20589" LastEditDate="2017-11-06T22:34:54.357" LastActivityDate="2017-11-08T04:03:55.820" Title="Goodhart's law applied to data science" Tags="|machine-learning|optimization|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24426" PostTypeId="1" AcceptedAnswerId="24441" CreationDate="2017-11-06T22:26:31.383" Score="1" ViewCount="1307" Body="&lt;p&gt;&lt;strong&gt;PROBLEM:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose if I have a small dataset containing some words and their tags/labels.&#xA;The main task is to provide tags to other words(which are not in the dataset) based on their contextual relationship with the words already in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, for example, my custom dataset includes&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;              Soap --&amp;gt; label__(cleaning_agent)&#xA;              pencil--&amp;gt; label__(stationary_item)&#xA;              mobile--&amp;gt; label__(electronics)&#xA;              washingmachine--&amp;gt; label(electronics)&#xA;              and so on.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want my program to be able to correctly predict the label of an unknown word, e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         washing powder to its correct category&#xA;         label__(cleaning_agent)&#xA;         radio to label__(electronics) etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ACTIONS:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the main problem is to find the relation between two words based on context, but I am not able to decide what can be the parameters for finding that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried a naive approach using datamuse API and fastText library.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    Naive approach is as follows-&amp;gt;&#xA;        step 1-&amp;gt; find all the related words of the given word(let's say W) e.g. pencil using datamuse API.&#xA;        step 2-&amp;gt; combine them into a string(let's say S) with spaces in between them&#xA;        step 3-&amp;gt; use the label name, W, S as the training dataset for fastText.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;NOTE: fastText requires label name, word, sentence(can be from news articles, blogs, Wikipedia etc ) as a context for that word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;RESULTS:&lt;/strong&gt;&#xA;fastText is not providing any reliable results.&#xA;I am thinking of building a neural network kind of thing for this purpose.&#xA;But I am not able to decide what can be the input parameters for our data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main problem is about custom word tagging. Our program should be able to tag unknown(not in training dataset) words to their most probable classes based on some score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I am new to NLP, I want to know what can be the next move forward.&lt;/p&gt;&#xA;" OwnerUserId="41514" LastActivityDate="2017-11-07T07:27:43.290" Title="How can I find contextually related words and classify into custom tags/labels?" Tags="|machine-learning|neural-network|classification|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24427" PostTypeId="2" ParentId="24352" CreationDate="2017-11-06T23:47:11.177" Score="2" Body="&lt;p&gt;Gaussian models are often used (and maybe sometimes over-used) because of their mathematical convenience (many statistical models can be found as built-in functions, when based on the Gaussian distribution, in some libraries such as mixture models, hidden Markov models,...). Also, when one has no idea about what distribution could best model the data, given the relationship of the Gaussian distribution with the Central-Limit Theorem, it is often a reasonable assumption to make.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if say, your goal is to generate new data similar to some training data and that you know that the data you would like to generate should be strictly positive, then a Gaussian assumption may not be the best one. Indeed, generating data from a Gaussian-based model gives no guarantee that these data will be positive (the Gaussian has an infinite support). Then one could think of basing the model on some distribution that insure generated data to be positive such as the Beta or the Dirichlet distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is always a good start to take the Gaussian as an assumption, get some results and if they are not good enough, try other assumptions and compare.&#xA;This can enhance the accuracy but can also require a big amount of work as most classical machine learning algorithms are often not implemented in main libraries for assumptions other than the Gaussian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;However, what if the data that I am dealing with is not normally distributed?&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then you should see an improvement of your results when using more appropriate distributions.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Should I perform log/ exponential transformation on the data for the sake of getting normal distributed data?&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It can indeed be worth it (and little work) to transform the data and see the impact on the results.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Why is Gaussian data always best suited?&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is not. As a proof, you can look at several publications about mixture models that compare for different data sets how Dirichlet, Gaussian, Beta mixtures models (and other) behave.&lt;/p&gt;&#xA;" OwnerUserId="25157" LastActivityDate="2017-11-06T23:47:11.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24428" PostTypeId="2" ParentId="24401" CreationDate="2017-11-07T00:51:42.157" Score="0" Body="&lt;p&gt;GloVe has a wide selection of pre-trained vectorizers. You can simply choose one of the smaller datasets, e.g. the Wikipedia trained model from &lt;a href=&quot;https://github.com/stanfordnlp/GloVe/blob/master/README.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/stanfordnlp/GloVe/blob/master/README.md&lt;/a&gt;. For me, loading the 50 dimensional file with 400K vocab size happens in only a second or two. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2017-11-07T00:51:42.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24429" PostTypeId="1" CreationDate="2017-11-07T01:30:56.353" Score="4" ViewCount="3751" Body="&lt;p&gt;Is it possible to use &lt;code&gt;class_weights&lt;/code&gt; with a one-hot encoding?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried &lt;code&gt;sparse_categorical_crossentropy&lt;/code&gt; and, for some reason, it's significantly worse than my classic &lt;code&gt;categorical_crossentropy&lt;/code&gt; with one hot encoding model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I'm calculating the class_weights with sparse:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;unique_class_weights = np.unique(labels)&#xA;class_weights = class_weight.compute_class_weight('balanced', unique_class_weights, labels)&#xA;class_weights_dict = { unique_class_weights[i]: w for i,w in enumerate(class_weights) }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Training like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;full_model.compile(loss='sparse_categorical_crossentropy',&#xA;                   optimizer='rmsprop',&#xA;                   metrics=['accuracy'])&#xA;&#xA;&#xA;full_model.fit_generator(line_generator(data_train, labels_train),&#xA;                         validation_data=line_generator(data_test, labels_test),&#xA;                         validation_steps=1,&#xA;                         steps_per_epoch=len(data)/GENERATOR_BATCH_SIZE,&#xA;                         class_weight=class_weights_dict,&#xA;                         epochs=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31639" LastEditorUserId="26616" LastEditDate="2018-05-31T19:32:11.963" LastActivityDate="2018-05-31T19:32:11.963" Title="How to use class_weights with one-hot encoding in keras?" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24430" PostTypeId="2" ParentId="15524" CreationDate="2017-11-07T02:55:14.200" Score="5" Body="&lt;p&gt;Seq2Seq architecture can definitely be used for time series problem. The only twist is that you will need a linear layer on top of your decoder to project the outputs to the required size (for example, 1 for univariate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The stepwise forecast approach can be used for short sequences but because any biases are compounded using this approach, it is not good for longer sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have a sequence where the value is constant at each time step $x_{i+1} = x_i$, but you model learned to do $x_{i+1}=1.01x_i$ instead (which is highly likely with gradient descent algorithm). For $t=10$, the target value will be $1^{50} = 1$, however you model will predict $1.01^{50}=1.64$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus a 1% single step error results in a 64% difference in 50 steps.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="29575" LastEditDate="2017-11-07T03:08:07.430" LastActivityDate="2017-11-07T03:08:07.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24431" PostTypeId="2" ParentId="23400" CreationDate="2017-11-07T03:34:49.127" Score="1" Body="&lt;p&gt;You can frame the problem this way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider each car as a sample of your dataset and use $Car^{i}$ to denote the $i^{th}$ sample. Then for each sample, you have the following features: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Time series of cars' condition $C_0^{i}...C_n^{i}$&lt;/li&gt;&#xA;&lt;li&gt;Other time-invariant features like colour $A^{i}$ and engine size $E^{i}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(I am not sure how mileage fits here, do you have a time series on this as well? Intuitively mileage changes over the life of a car so it should be time series as well)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of your question is how to incorporate those time-invariant features into an LSTM model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of only passing the $C_{t-1}$ as LSTM cell input you pass it with a vector $[C_{t-1}^i, A^i, E^i]$. Here, using Tensorflow's terminology, you &lt;code&gt;input_size&lt;/code&gt; is 3 instead 1. $A^i$ and $E^i$ is always the same at each timestep but different across samples.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-07T03:34:49.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24432" PostTypeId="2" ParentId="15694" CreationDate="2017-11-07T03:45:07.263" Score="3" Body="&lt;p&gt;No, it is not possible with Vanilla LSTM. It's not a Kera problem, but the fundamental structure of Vanilla LSTM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you have a bit of misunderstanding of how LSTM works. You are assuming given $x_0$...$ x_{19}$ the LSTM is predicting $x_{20}$...$x_{39}$. This is NOT the case. In LSTM if your input sequence is $x_0$...$ x_{19}$ then your output sequence is $x_1$...$ x_{20}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To achieve what you want, you can use something like Seq2Seq model.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-07T03:45:07.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24433" PostTypeId="2" ParentId="12721" CreationDate="2017-11-07T04:12:51.250" Score="5" Body="&lt;p&gt;Adding to @AN6U5's respond.&lt;/p&gt;&#xA;&lt;p&gt;From a purely theoretical perspective, this &lt;a href=&quot;https://link.springer.com/chapter/10.1007/11840817_66&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; has show RNN are universal approximators. I haven't read the paper in details, so I don't know if the proof can be applied to LSTM as well, but I suspect so. The biggest problem with RNN in general (including LSTM) is that they are hard to train due to gradient exploration and gradient vanishing problem. The practical limit for LSTM seems to be around 200~ steps with standard gradient descent and random initialization. And as mentioned, in general for any deep learning model to work well you need a lot of data and heaps of tuning.&lt;/p&gt;&#xA;&lt;p&gt;ARIMA model is more restricted. If your underlying system is too complex then it is simply impossible to get a good fit. But on the other hand, if you underlying model is simple enough, it is much more efficient than deep learning approach.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2021-01-31T22:12:12.133" LastActivityDate="2021-01-31T22:12:12.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24434" PostTypeId="2" ParentId="24403" CreationDate="2017-11-07T04:30:05.850" Score="1" Body="&lt;p&gt;I am going to diverge a little bit and argue that calculation confidence interval is in practice is usually not a valuable thing to do. The reason is there is always a whole bunch of assumptions you need to make. Even for the simplest linear regression, you need to have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Linear relationship. &lt;/li&gt;&#xA;&lt;li&gt;Multivariate normality. &lt;/li&gt;&#xA;&lt;li&gt;No or little multicollinearity. &lt;/li&gt;&#xA;&lt;li&gt;No auto-correlation. &lt;/li&gt;&#xA;&lt;li&gt;Homoscedasticity.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A much more pragmatic approach is to do a Monte Carlo simulation. If you already know or willing to make of assumption around the distribution of your input variables, take a whole bunch of sample and feed it to you LSTM, now you can empirically calculate your &quot;confidence interval&quot;. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-07T04:30:05.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24435" PostTypeId="2" ParentId="24275" CreationDate="2017-11-07T04:45:30.153" Score="1" Body="&lt;p&gt;MLPClassifier and DNNClassifier are both implementations of the simplest feed-forward neural network. So in principle, they are the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow is a deep learning library. scikit-learn is a more traditional machine learning library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;scikit-learn have very limited coverage for deep learning, only MLPClassifier and MLPregressor, which are the basic of basics. The devs of scikit-learn focus on a more traditional area of machine learning and made a deliberate choice to not expand too much into the deep learning area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow, on the other hand, is dedicated to deep learning. You can compose much very complex deep learning model with it. And GPU support is a big big deal in deep learning, we really need that 1000x speed up to get any meaningful work done in deep learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can play with toy data using MLPClassifier but nothing more. That being said, the scikit-learn source code and documentation are so accessible. It is amazing for learning. I highly recommend you work through the MLPClassifier source code if you just started learning deep learning. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-07T04:45:30.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24436" PostTypeId="2" ParentId="20019" CreationDate="2017-11-07T05:13:49.907" Score="0" Body="&lt;p&gt;There is no need of two rdd values,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plotdf=df.select(_c3,_c4);&#xA;plotdf.plot( style=[])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you still face the error, please follow &lt;a href=&quot;https://stackoverflow.com/questions/43206554/typeerror-float-argument-must-be-a-string-or-a-number-not-period&quot;&gt;the link&lt;/a&gt; and set index.&lt;/p&gt;&#xA;" OwnerUserId="41529" LastEditorUserId="29575" LastEditDate="2017-11-07T10:00:30.333" LastActivityDate="2017-11-07T10:00:30.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24437" PostTypeId="2" ParentId="24425" CreationDate="2017-11-07T05:26:17.143" Score="1" Body="&lt;p&gt;The function problem which Goodhart's Law described is the changes of the underlying model from which the data is generated.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to look at this is that the data generating process in the past is bias. Say we our try model is a product of two variables $z = xy$. In the past, $y$ has always been constant at $a$, so you come to the conclusion $z$ must equal to $ax$. But this is only true when $y=a$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Goodhart's setting (economic policy and social science in general), it is usually the prediction value of $z$ that causing the changes in $y$. It is a dynamic system of which we only have a bias static slice of data. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-08T04:03:55.820" LastActivityDate="2017-11-08T04:03:55.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24438" PostTypeId="2" ParentId="17299" CreationDate="2017-11-07T07:06:47.933" Score="1" Body="&lt;p&gt;Neural Network became available to Orange recently. The version that you have does not have it, and you have to install a newer version (say, the most recent version 3.6.0). Neural Network can be used both for regression and classification, in typical Orange workflow such as that for accuracy estimation with cross-validation (shown below) or class prediction (using Predictions widget.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Xsxyd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xsxyd.png&quot; alt=&quot;neural network in orange&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39624" LastActivityDate="2017-11-07T07:06:47.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24439" PostTypeId="2" ParentId="23100" CreationDate="2017-11-07T07:14:17.270" Score="1" Body="&lt;p&gt;No. Orange is a stand-alone, desktop application and is not a tool for designing analysis pipelines that would on their own run on the servers. Instead, it fosters interactive visualizations and exploratory data analysis that all require human input.&lt;/p&gt;&#xA;" OwnerUserId="39624" LastActivityDate="2017-11-07T07:14:17.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24440" PostTypeId="1" AcceptedAnswerId="24450" CreationDate="2017-11-07T07:23:29.140" Score="0" ViewCount="608" Body="&lt;p&gt;I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it.&#xA;What is the right tech/algorithm/Python framework to approach this?&lt;/p&gt;&#xA;" OwnerUserId="41532" LastActivityDate="2017-11-07T14:09:26.057" Title="Finding similar articles in realtime" Tags="|python|algorithms|similarity|similar-documents|search-engine|" AnswerCount="1" CommentCount="3" ClosedDate="2017-11-10T16:30:07.693" ContentLicense="CC BY-SA 3.0" />
  <row Id="24441" PostTypeId="2" ParentId="24426" CreationDate="2017-11-07T07:27:43.290" Score="1" Body="&lt;p&gt;You are working with word classification so you really don't have any contextual information that you can leverage. So the best thing would be to use freely available contextual information called as word2vec. I would suggest using the pretrained GLOVE Word2Vec as they are much cleaner than the Google ones. I am going to suggest two approaches for doing this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Naive Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load the pretrained word vectors. &lt;/li&gt;&#xA;&lt;li&gt;For every new word calculate cosine similarity with every item in your training list. &lt;/li&gt;&#xA;&lt;li&gt;Predict the label for the word which is closest to your new word. &lt;/li&gt;&#xA;&lt;li&gt;Alternatively you can calculate cosine similarity with the labels and just predict the label which comes the closest. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Better Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load the pretrained word vectors. &lt;/li&gt;&#xA;&lt;li&gt;Extract the vectors for every item in the training set. &lt;/li&gt;&#xA;&lt;li&gt;One hot encode the labels.&lt;/li&gt;&#xA;&lt;li&gt;Train a neural net or any multiclass classification algorithm using the vectors as features and save the model.&lt;/li&gt;&#xA;&lt;li&gt;For every new word extract the vector and run the model on this vector.&lt;/li&gt;&#xA;&lt;li&gt;The predicted label is your output. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can use the first approach to run quick tests and check the efficacy of the approach and then move on to the second one if you found it fit to your use. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-11-07T07:27:43.290" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24442" PostTypeId="1" AcceptedAnswerId="24445" CreationDate="2017-11-07T08:34:45.400" Score="1" ViewCount="479" Body="&lt;p&gt;In my project I use 2 MLP ANNs: for classification and for prediction. Basic training dataset is stored in database (MS SQL Server). But we need to make future incremental learning possible. It means that ANN is used in industrial realtime system, operator marks observations in database as objects of a particular class and ANN must take it into account. How can I do it?&lt;/p&gt;&#xA;" OwnerUserId="41535" LastActivityDate="2017-11-07T09:16:56.740" Title="MLP incremental learning" Tags="|machine-learning|neural-network|classification|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-10T16:32:33.033" ContentLicense="CC BY-SA 3.0" />
  <row Id="24443" PostTypeId="2" ParentId="24380" CreationDate="2017-11-07T08:39:03.817" Score="1" Body="&lt;p&gt;It's usually indicating bad starting centroids.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it happens later in the process, it may indicate kmeans doesn't work well on this data, because a &lt;em&gt;stable&lt;/em&gt; clustering just be easy to find.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-07T08:39:03.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24444" PostTypeId="1" CreationDate="2017-11-07T08:41:53.797" Score="1" ViewCount="992" Body="&lt;p&gt;I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, perplexity is increasing with increased number of topics on test corpus. I am not sure whether it is natural, but i have read perplexity value should decrease as we increase the number of topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, i plotted perplexity on train corpus and it is decreasing as topic number is increased. I am not sure whether it represent over-fitting of my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me understand&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Perplexity on Train Data&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dbI88.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dbI88.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Perplexity on Test Data&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/AYTlj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AYTlj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41454" LastActivityDate="2017-11-07T08:41:53.797" Title="Perplexity increasing on Test DataSet in LDA (Topic Modelling)" Tags="|r|clustering|text-mining|topic-model|lda|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24445" PostTypeId="2" ParentId="24442" CreationDate="2017-11-07T09:16:56.740" Score="0" Body="&lt;p&gt;If your old data is representatively of the underlying population and there is no radical shift of the underlying true model over time. Than you don't need to update you model as new data comes in. You might want to retrain you model with the new dataset periodically, but you don't need to do it in real time on your production machine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the underlying true does change radically overtime and you don't have any existing data to captures those changes then doesn't matter what algorithm you use, it's not going to work. The information is simply not there. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-07T09:16:56.740" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24446" PostTypeId="1" CreationDate="2017-11-07T10:33:19.003" Score="4" ViewCount="1437" Body="&lt;p&gt;Suppose we have a training set of 3 classes of image: 1.Cats, 2.Dogs, 3.Neither cats nor dogs.&#xA;We're only really bothered about detecting whether an image is a cat/dog, or neither, but we don't care whether it's a cat or a dog, just that it's one of the two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it inherently &quot;better&quot; to train a binary class CNN by merging the Dog and Cat classes into a single &quot;positive&quot; class rather than train a 3-class CNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is, does it then follow that if we &lt;em&gt;did&lt;/em&gt; care about whether the image was a dog or cat, it would also be better to first run the binary classifier to decide if it was a dog/cat or not, then run a second binary classifier trained on just cats and dogs to decide which of the two it actually is?&lt;/p&gt;&#xA;" OwnerUserId="41538" LastEditorUserId="29169" LastEditDate="2021-02-15T22:21:29.293" LastActivityDate="2021-02-15T22:21:29.293" Title="CNN Multi-class vs Binary Class Image Classification" Tags="|cnn|image-classification|multiclass-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24447" PostTypeId="2" ParentId="24405" CreationDate="2017-11-07T10:45:39.193" Score="17" Body="&lt;p&gt;Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and scikit-learn deliberately avoids inferential approach to model learning (significance testing etc). Moreover, pure OLS is only one of numerous regression algorithms, and from the scikit-learn point of view it is neither very important, nor one of the best. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are, however, some pieces of advice for those who still need a good way for feature selection with linear models:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use inherently sparse models like &lt;code&gt;ElasticNet&lt;/code&gt; or &lt;code&gt;Lasso&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Normalize your features with &lt;code&gt;StandardScaler&lt;/code&gt;, and then order your features just by &lt;code&gt;model.coef_&lt;/code&gt;. For perfectly independent covariates it is equivalent to sorting by p-values. The class &lt;code&gt;sklearn.feature_selection.RFE&lt;/code&gt; will do it for you, and &lt;code&gt;RFECV&lt;/code&gt; will even evaluate the optimal number of features.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://planspace.org/20150423-forward_selection_with_statsmodels/&quot; rel=&quot;noreferrer&quot;&gt;an implementation&lt;/a&gt; of forward selection by adjusted $R^2$ that works with &lt;code&gt;statsmodels&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Do brute-force forward or backward selection to maximize your favorite metric on cross-validation (it could take approximately quadratic time in number of covariates). A scikit-learn compatible &lt;code&gt;mlxtend&lt;/code&gt; package &lt;a href=&quot;http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/&quot; rel=&quot;noreferrer&quot;&gt;supports&lt;/a&gt; this approach for any estimator and any metric.&lt;/li&gt;&#xA;&lt;li&gt;If you still want vanilla stepwise regression, it is easier to base it on &lt;code&gt;statsmodels&lt;/code&gt;, since this package calculates p-values for you. A basic forward-backward selection could look like this:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;```&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;import pandas as pd&#xA;import numpy as np&#xA;import statsmodels.api as sm&#xA;&#xA;data = load_boston()&#xA;X = pd.DataFrame(data.data, columns=data.feature_names)&#xA;y = data.target&#xA;&#xA;&#xA;def stepwise_selection(X, y, &#xA;                       initial_list=[], &#xA;                       threshold_in=0.01, &#xA;                       threshold_out = 0.05, &#xA;                       verbose=True):&#xA;    &quot;&quot;&quot; Perform a forward-backward feature selection &#xA;    based on p-value from statsmodels.api.OLS&#xA;    Arguments:&#xA;        X - pandas.DataFrame with candidate features&#xA;        y - list-like with the target&#xA;        initial_list - list of features to start with (column names of X)&#xA;        threshold_in - include a feature if its p-value &amp;lt; threshold_in&#xA;        threshold_out - exclude a feature if its p-value &amp;gt; threshold_out&#xA;        verbose - whether to print the sequence of inclusions and exclusions&#xA;    Returns: list of selected features &#xA;    Always set threshold_in &amp;lt; threshold_out to avoid infinite looping.&#xA;    See https://en.wikipedia.org/wiki/Stepwise_regression for the details&#xA;    &quot;&quot;&quot;&#xA;    included = list(initial_list)&#xA;    while True:&#xA;        changed=False&#xA;        # forward step&#xA;        excluded = list(set(X.columns)-set(included))&#xA;        new_pval = pd.Series(index=excluded)&#xA;        for new_column in excluded:&#xA;            model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()&#xA;            new_pval[new_column] = model.pvalues[new_column]&#xA;        best_pval = new_pval.min()&#xA;        if best_pval &amp;lt; threshold_in:&#xA;            best_feature = new_pval.argmin()&#xA;            included.append(best_feature)&#xA;            changed=True&#xA;            if verbose:&#xA;                print('Add  {:30} with p-value {:.6}'.format(best_feature, best_pval))&#xA;&#xA;        # backward step&#xA;        model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()&#xA;        # use all coefs except intercept&#xA;        pvalues = model.pvalues.iloc[1:]&#xA;        worst_pval = pvalues.max() # null if pvalues is empty&#xA;        if worst_pval &amp;gt; threshold_out:&#xA;            changed=True&#xA;            worst_feature = pvalues.argmax()&#xA;            included.remove(worst_feature)&#xA;            if verbose:&#xA;                print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))&#xA;        if not changed:&#xA;            break&#xA;    return included&#xA;&#xA;result = stepwise_selection(X, y)&#xA;&#xA;print('resulting features:')&#xA;print(result)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This example would print the following output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Add  LSTAT                          with p-value 5.0811e-88&#xA;Add  RM                             with p-value 3.47226e-27&#xA;Add  PTRATIO                        with p-value 1.64466e-14&#xA;Add  DIS                            with p-value 1.66847e-05&#xA;Add  NOX                            with p-value 5.48815e-08&#xA;Add  CHAS                           with p-value 0.000265473&#xA;Add  B                              with p-value 0.000771946&#xA;Add  ZN                             with p-value 0.00465162&#xA;resulting features:&#xA;['LSTAT', 'RM', 'PTRATIO', 'DIS', 'NOX', 'CHAS', 'B', 'ZN']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-11-07T15:48:35.280" LastActivityDate="2017-11-07T15:48:35.280" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24448" PostTypeId="2" ParentId="17299" CreationDate="2017-11-07T10:54:03.407" Score="3" Body="&lt;p&gt;An infrared data set &quot;Collagen spectroscopy&quot; is available in the &quot;Data Sets&quot; widget. There are some other but small data sets in the infrared data folder: the easiest way to find the folder is to search for &quot;peach_juice.dpt&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Orange Infrared can read Opus, Envi, .spa, .nea, Omnic maps, and some ASCII file formats.&lt;/p&gt;&#xA;" OwnerUserId="41539" LastActivityDate="2017-11-07T10:54:03.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24449" PostTypeId="1" CreationDate="2017-11-07T12:00:26.950" Score="6" ViewCount="8596" Body="&lt;p&gt;Let's say I have a set of n time-series with sequence length 8&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And let's define the input that LSTM expects as a tensor of shape (samples,sequence-length,features)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict the last value of each one of them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I cant either feed the network a sequence of the 7 first values with the 8th as the target:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[a,b,c,d,e,f,g] -&gt; [h] &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case the tensor would be (samples,7,1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or I can do the classic rolling window, with a window size of, say, 2.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[a,b],&#xA;  [b,c],&#xA;  [c,d],&#xA;  [d,e],&#xA;  [e,f],&#xA;  [f,g] -&gt; [h]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In effect this shortens the length of the sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the input tensor would be (samples,2,1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the trade-offs between performing rolling-windows or giving the &quot;crude&quot; time-series to the LSTM?&lt;/p&gt;&#xA;" OwnerUserId="28599" LastActivityDate="2022-12-06T00:44:47.277" Title="Is it necessary to perform rolling-window on LSTMs?" Tags="|time-series|keras|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24450" PostTypeId="2" ParentId="24440" CreationDate="2017-11-07T14:09:26.057" Score="3" Body="&lt;p&gt;Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL isn't particularly good at it. Moreover, Elasticsearch is quite user friendly, so it won't be an overkill to actually install it and use it. You might discover in the process that finding most similar articles isn't that easy and that Elasticsearch is of a great. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://elasticsearch-py.readthedocs.io/en/master/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the documentation for a Python client:  &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2017-11-07T14:09:26.057" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24452" PostTypeId="1" AcceptedAnswerId="24453" CreationDate="2017-11-07T14:37:41.993" Score="90" ViewCount="120304" Body="&lt;p&gt;I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, so it is logical to remove one of them. But I can not understand why this can worsen the model. &lt;/p&gt;&#xA;" OwnerUserId="33386" LastEditorUserId="38887" LastEditDate="2017-11-08T23:30:28.850" LastActivityDate="2020-01-08T09:19:53.347" Title="In supervised learning, why is it bad to have correlated features?" Tags="|machine-learning|correlation|" AnswerCount="7" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24453" PostTypeId="2" ParentId="24452" CreationDate="2017-11-07T15:21:15.303" Score="77" Body="&lt;p&gt;Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For linear models (e.g., linear regression or logistic regression), &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;noreferrer&quot;&gt;multicolinearity&lt;/a&gt; can yield &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity#Consequences_of_multicollinearity&quot; rel=&quot;noreferrer&quot;&gt;solutions that are wildly varying and possibly numerically unstable&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Random forests can be good at detecting interactions between different features, but highly correlated features can mask these interactions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;More generally, this can be viewed as a special case of &lt;a href=&quot;https://en.wikipedia.org/wiki/Occam%27s_razor&quot; rel=&quot;noreferrer&quot;&gt;Occam's razor&lt;/a&gt;. A simpler model is preferable, and, in some sense, a model with fewer features is simpler. The concept of &lt;a href=&quot;https://en.wikipedia.org/wiki/Minimum_description_length&quot; rel=&quot;noreferrer&quot;&gt;minimum description length&lt;/a&gt; makes this more precise.&lt;/p&gt;&#xA;" OwnerUserId="9518" LastActivityDate="2017-11-07T15:21:15.303" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24454" PostTypeId="2" ParentId="24452" CreationDate="2017-11-07T16:01:13.943" Score="8" Body="&lt;p&gt;In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent data after updating or editing tuples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;If we add so much correlated features to the model we may cause the model to consider unnecessary features and we may have curse of high dimensionality problem&lt;/strong&gt;, &lt;em&gt;I guess this is the reason for worsening the constructed model.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of machine learning we usually use &lt;code&gt;PCA&lt;/code&gt; to reduce the dimension of input patterns. This approach considers removing correlated features by someway (using &lt;code&gt;SVD&lt;/code&gt;) and is an unsupervised approach. This is done to achieve the following purposes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwju5tH86KzXAhXCDpoKHfW-BBAQFggsMAE&amp;amp;url=http%3A%2F%2Fwww.scielo.br%2Fscielo.php%3Fscript%3Dsci_arttext%26pid%3DS1679-45082012000200004&amp;amp;usg=AOvVaw0fHz2y5dPeaNeqyqtfbw4v&quot; rel=&quot;noreferrer&quot;&gt;Compression&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/268704/why-do-we-use-pca-to-speed-up-learning-algorithms-when-we-could-just-reduce-the&quot;&gt;Speeding up learning algorithms&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwi2y8G46qzXAhVGEJoKHesVAVcQFggrMAA&amp;amp;url=http%3A%2F%2Fnghiaho.com%2F%3Fpage_id%3D1030&amp;amp;usg=AOvVaw01gd1IN4lVYY7HxilQCtcn&quot; rel=&quot;noreferrer&quot;&gt;Visualizing data&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjd07nf6qzXAhWCE5oKHew2DAYQFggtMAE&amp;amp;url=http%3A%2F%2Fwww.cs.haifa.ac.il%2F~rita%2Fuml_course%2Fadd_mat%2FPCA.pdf&amp;amp;usg=AOvVaw3GdiMQK2c5GTbXDMUYW34Z&quot; rel=&quot;noreferrer&quot;&gt;Dealing with curse of high dimensionality&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Although this may not seem okay but I have seen people that use removing correlated features in order to avoid overfitting but I don't think it is a good practice. For more information I highly recommend you to see &lt;a href=&quot;https://stats.stackexchange.com/a/268719/179078&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another reason is that in deep learning models, like &lt;code&gt;MLPs&lt;/code&gt; if you add correlated features, you just add unnecessary information which adds more calculations and parameters to the model.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-08-01T17:08:38.183" LastActivityDate="2018-08-01T17:08:38.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24455" PostTypeId="2" ParentId="24400" CreationDate="2017-11-07T16:12:40.960" Score="1" Body="&lt;p&gt;Decision function of GLM itself does not depend on the choice of &quot;reference level&quot;. What can probably depend is your &lt;strong&gt;threshold&lt;/strong&gt;, and I guess it is chosen poorly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the problem of churn prediction, you probably shouldn't use error rate or confusion matrix at all. You can read &lt;a href=&quot;https://datascience.stackexchange.com/questions/806/advantages-of-auc-vs-standard-accuracy&quot;&gt;here&lt;/a&gt; why ROC AUC or other metrics can be preferred to accuracy. Or if you still use error rate, choose your threshold in a wiser way than letting an algorithm maximize F1 metric for a single class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need in the end is to decide for each client whether to treat her as ready-to-churn (it would certainly cost you $c_1$) or leave her alone (but if she churns, you lose $c_2$). If it is the case, you will profit from your anti-churn measures iff the &lt;em&gt;probability&lt;/em&gt; of churn is higher than $\frac{c_1}{c_2}$. This is the natural threshold for your classification problem, and you can use the corresponding cost function to measure your success. Or if you don't know exact losses $c_1$ and $c_2$ in advance, use ROC AUC, which averages all the possible thresholds. And yes, ROC AUC is not affected by class balance/imbalance.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-07T16:12:40.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24457" PostTypeId="1" CreationDate="2017-11-07T16:33:57.553" Score="2" ViewCount="542" Body="&lt;p&gt;I've been using the pre-trained Deep Learning models as feature extractors on a project involving chest x-ray images to detect TB. I extract the features from the layer just before the Softmax. I could repeatedly find that the performance of AlexNet is way better than ResNet-50. Why is it so?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastActivityDate="2017-11-07T16:33:57.553" Title="Can AlexNet outperform ResNet as a feature extractor?" Tags="|machine-learning|deep-learning|feature-extraction|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24458" PostTypeId="1" CreationDate="2017-11-07T17:12:14.060" Score="15" ViewCount="2722" Body="&lt;p&gt;Google recently included in tensorflow's nightly builds its &lt;strong&gt;Eager&lt;/strong&gt; mode, an imperative API to access tensorflow computation capabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do tensorflow eager compare to PyTorch?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some aspects that could affect the comparison could be: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Advantages and disadvantages of eager due to its static graph legacy (e.g. names in nodes).&lt;/li&gt;&#xA;&lt;li&gt;Intrinsic limitations of either of them that the other does not have.&lt;/li&gt;&#xA;&lt;li&gt;Areas in which one of them needs improvement (e.g. feature completeness, computational optimizations).&lt;/li&gt;&#xA;&lt;li&gt;Ecosystem differences (e.g. tensorboard?).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note1: Yaroslav Bulatov wrote &lt;a href=&quot;https://medium.com/@yaroslavvb/tensorflow-meets-pytorch-with-eager-mode-714cce161e6c&quot; rel=&quot;noreferrer&quot;&gt;a review about eager's nice features&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note2: In &lt;a href=&quot;https://datascience.stackexchange.com/questions/16835/pytorch-vs-tensorflow-fold&quot;&gt;a previous question&lt;/a&gt;, I requested a comparison between PyTorch and Tensorflow Fold. At that time, it seemed to me that Fold could face PyTorch thanks to Google backing it. I was very very wrong: in the end, Google itself &lt;a href=&quot;https://github.com/tensorflow/fold/issues/87&quot; rel=&quot;noreferrer&quot;&gt;abandoned Fold&lt;/a&gt; in favour of Eager. I understand that this was due to intrinsic limitations in the normal tensorflow API that led Fold not to be very friendly, which constrained its adoption.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-02-19T06:54:11.123" Title="PyTorch vs. Tensorflow eager" Tags="|deep-learning|tensorflow|pytorch|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24459" PostTypeId="1" AcceptedAnswerId="24465" CreationDate="2017-11-07T17:35:38.417" Score="2" ViewCount="4483" Body="&lt;p&gt;I am a noob in machine learning and trying to build a classifier using keras by following this tutorial &lt;a href=&quot;https://machinelearningmastery.com/object-recognition-convolutional-neural-networks-keras-deep-learning-library/&quot; rel=&quot;nofollow noreferrer&quot;&gt;machine learning mastery tutorial&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to build a classifier based on MLP like in &lt;a href=&quot;https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification of MNIST using MLP&lt;/a&gt; for CIFAR-10 data set.Like in MNIST data set the 28*28 images to a 784 vector is given as a input to the neural network.&#xA;Like wise how should I frame the data in CIFAR-10 should it be changed in the 36*36 pixels rgb values to an array? or what type of array should I make to give as a input to the MLP network?&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2017-11-07T21:15:59.547" Title="How to give CIFAR-10 as an input to MLP" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24461" PostTypeId="2" ParentId="24452" CreationDate="2017-11-07T20:14:08.017" Score="5" Body="&lt;p&gt;Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows and a sample of columns. If two columns are highly correlated, there's a chance that one of them won't be selected in a particular tree's column sample, and that tree will depend on the remaining column. Correlated features mean you can reduce overfitting (through column sampling) without giving up too much predictive quality.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-07T20:14:08.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24463" PostTypeId="1" AcceptedAnswerId="24464" CreationDate="2017-11-07T20:49:40.170" Score="0" ViewCount="1331" Body="&lt;p&gt;I need a regex in R to exclude 1 or 2-character words, but which does not treat hyphens as word boundaries.  Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;z &amp;lt;- c(&quot;regulatory protein SR-B1 na na na na&quot;, &quot;Dr Foo is na Editor-in-chief&quot;, &quot;na P3 protein is popular na na&quot;)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The intended '&lt;strong&gt;gsub&lt;/strong&gt;' would produce the result &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1] &quot;regulatory protein SR-B1&quot; &quot; Foo Editor-in-chief&quot;     &quot;P3 protein  popular&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not produced from the &lt;strong&gt;gsub&lt;/strong&gt; statements below. Instead:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A - 2-character segments unintentionally dropped from hyphenated words (and hyphens kept).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;B - 2-character segments unintentionally dropped within hyphenated words (hyphens also dropped):  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;C - Produces: Error: '-' is an unrecognized escape in character string starting &quot;&quot;&#xA;*\b[[-&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;D - Nothing dropped or changed:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gsub(&quot; *\\b[[:alpha:]]{1,2}\\b *&quot;, &quot; &quot;, z) # A&#xA;&#xA;gsub(&quot; *\\b[-[:alpha:]]{1,2}\\b *&quot;, &quot; &quot;, z)  # B&#xA;&#xA;gsub(&quot; *\\b[[\-][:alpha:]]{1,2}\\b *&quot;, &quot; &quot;, z)  # C&#xA;&#xA;gsub(&quot; *\\b[[\\-][:alpha:]]{1,2}\\b *&quot;, &quot; &quot;, z) # D  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Input would be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="41566" LastActivityDate="2017-11-07T21:08:12.033" Title="Regex remove 1-2 character sequences: hyphens literal, not word boundaries" Tags="|r|text-mining|regex|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-10T16:34:18.623" ContentLicense="CC BY-SA 3.0" />
  <row Id="24464" PostTypeId="2" ParentId="24463" CreationDate="2017-11-07T21:08:12.033" Score="-1" Body="&lt;p&gt;Try this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gsub(&quot; *\\b(?&amp;lt;!-)\\w{1,2}(?!-)\\b *&quot;, &quot; &quot;, z, perl=T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41571" LastActivityDate="2017-11-07T21:08:12.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24465" PostTypeId="2" ParentId="24459" CreationDate="2017-11-07T21:15:59.547" Score="4" Body="&lt;p&gt;MNIST has 28x28 pixel greyscale images, so there are $28\times28 = 784$ features per image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CIFAR10 contains colour images that are 32x32 pixels. At first glance it may seem like there are only $32\times32 = 1024$ features per image, but there are red, blue and green channels in the image, which means each image is actually $3\times32\times32 = 3072$ features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, these channels are arranged as three separate 32x32 pixel images, rather than a single 32x32 image with RGB pixels. In keras, you can load the CIFAR10 dataset like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.datasets import cifar10&#xA;&#xA;(x_train, y_train), (x_test, y_test) = cifar10.load_data()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, this will load the train and test dataset in the shape &lt;code&gt;(num_samples, 3, 32, 32)&lt;/code&gt;. In order to input these into an MLP, we need to flatten the channels and pixel arrays to form an array of shape &lt;code&gt;(num_samples, 3072)&lt;/code&gt;, just like with MNIST. We can do this in python like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_train = x_train.reshape(-1, 3072)&#xA;x_test = x_test.reshape(-1, 3072)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-07T21:15:59.547" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24466" PostTypeId="1" CreationDate="2017-11-07T22:40:29.373" Score="2" ViewCount="135" Body="&lt;p&gt;Can anybody tell me the formula how to find the number of false positives with respect to the first class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VtabE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VtabE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y$ is the truth / target and $a(x)$ is the prediction &lt;/p&gt;&#xA;" OwnerUserId="32599" LastEditorUserId="41525" LastEditDate="2017-11-08T09:06:11.427" LastActivityDate="2017-11-08T19:03:19.137" Title="Count the number of false positives with respect to the first class" Tags="|data|" AnswerCount="1" CommentCount="6" FavoriteCount="0" ClosedDate="2017-11-10T16:30:14.830" ContentLicense="CC BY-SA 3.0" />
  <row Id="24468" PostTypeId="2" ParentId="22037" CreationDate="2017-11-08T00:49:30.043" Score="1" Body="&lt;p&gt;Being faster or lower is a relative term and must be understood in the context of what it is comparing to. So, in order to understand this, we must first consider how gradient descent works with other types of the activation function.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Example Setup&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Consider an MLP with $n$ hidden layers of size one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_1 = W_1 x + b_1 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_1 = f(z_1)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_n = W_n a_{n-1} + b_n$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = f(z_n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $f$ is the activation function.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Tanh and Sigmoid - Vanishing Gradient&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Suppose $f$ is Tanh or Sigmoid activation function. The derivate of those functions are bounded between -1 to 1, or formally $f'(x) \in (-1, 1)$ for any $x$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This causes a very important problem in deep learning known as &quot;gradient vanishing problem&quot;. Let's consider the derivative of $y$ w.r.t $W_1$. By chain rule, we have &#xA;$$ \frac{df}{dW_1} = \frac{df}{dW_{n}} \frac{dW_{n}}{dW_{n-1}} ... \frac{dW_{2}}{dW_{1}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and for any $0 &amp;lt; i &amp;lt; n$, notice that &#xA;$$ \frac{dX_{i}}{dX_{i-1}} = f'(W_{i-1}a_{i-2} + b_{i-1}) \times a_{i-2} \in (-1, 1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The first term is between $(-1, 1)$ because $f'$ is bounded as discussed earlier and $a_{i-2}$ is also between $(-1, 1)$ as squash the input value.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $\frac{df}{dW_1}$ is basically a product of lots of terms each is between (0, 1). The larger the $n$ (deeper the network) is the more of that term we need to multiply and as a result of the $\frac{df}{dW_1}$ becomes exponentially smaller. &#xA;Because of this exponential relationship, the gradient quickly becomes so small we can effectively consider it as zero. The consequence of having zero gradients is no learning can happen at all because our update rule for gradient descent is based on that gradient. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;RELU and Dead Neuron&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Relu is invented to deal with the vanishing gradient problem because its derivative is always 1 when $a_i &amp;gt; 0$ so when $f$ is RELU we have:&#xA;$$\frac{dX_{i}}{dX_{i-1}} = a_{i-2}$$&#xA;$$\frac{df}{dW_1} = a_1 a_2 a_3 ... a_{n-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It all nice and well when $x &amp;gt; 0$ but things fall apart whenever $x &amp;lt; 0$, this time not only is the gradient very close to zero, it is a pure zero. Once a neuron reaches there, there is not chance to get back at all. That's why this is known as the &quot;Dead Neuron&quot; problem&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Leaky RELU and ELU&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Leaky RELU and ELU is the natural development after RELU. They are similar to RELU as such the derivative equal to 1 when $x &amp;gt; 0$ but avoided &quot;dead neuron&quot; by avoiding zero derivates when $x&amp;lt;1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I quote the &lt;a href=&quot;https://arxiv.org/abs/1511.07289&quot; rel=&quot;nofollow noreferrer&quot;&gt;original paper&lt;/a&gt; for the difference between the two.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;While LReLUs and PReLUs have negative values, too, they do not ensure&#xA;  a noise-robust deactivation state. ELUs saturate to a negative value&#xA;  with smaller inputs and thereby decrease the forward propagated&#xA;  variation and information.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The intuitive explanation goes like the following. In ELU, whenever x became small enough, the gradient became really small and saturated (in the same way it happens for Tanh and Sigmoid). The small gradient means that the learning algorithm can focus on the tuning of other weights without worrying about the interactivity with the saturated neurons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a polynomial of degree 2 which can be represented as a smooth surface in a 3-d space. To find the local minimum, a gradient descent algorithm will need to consider the steepness in both x and y-direction. If the gradient is both negative in the x-direction and y-direction, it is not clear which way is better. So it is sensible to choose a path somewhere in between. But what if we already know everything is flat (zero gradients) in the x-direction, then it becomes a no-brainer to go for the y-direction. Or in other word, you search space become much smaller. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;special note&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In deep learning, there is a lot of claims without enough empirical evidence or in-depth understanding to support it. In ELU's case, while it might be true that it results in faster convergence for some datasets, it could also be true that it makes the learning algorithm to stuck at the local maximum for a different dataset. We just don't know enough yet.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-08T00:55:00.893" LastActivityDate="2017-11-08T00:55:00.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24469" PostTypeId="1" CreationDate="2017-11-08T01:11:35.013" Score="2" ViewCount="217" Body="&lt;p&gt;I am trying to do stratified sampling in R to sample from my data and one of the parameters is group, which takes variable names to sample from keeping same initial distribution of the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it common to include the response as a grouping variable or not and what are the implications of including/excluding it?&lt;/p&gt;&#xA;" OwnerUserId="41580" LastEditorUserId="29575" LastEditDate="2017-11-08T04:25:08.420" LastActivityDate="2017-11-08T04:25:08.420" Title="Stratified Sampling Variable Choice" Tags="|bigdata|sampling|distribution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24470" PostTypeId="2" ParentId="22047" CreationDate="2017-11-08T01:26:57.220" Score="4" Body="&lt;h3&gt;Is this the correct way to forecast time series with LSTMs?&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You method could work in theory. But typically, for vanilla LSTM, the setup looks like $x = [x_0, x_1, ..., x_{n-1}]$ and $y = [x_1, x_2 ..., x_n]$. I suspect you will get much better result if you do it this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Highly recommend Colah's excellent &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;explaination&lt;/a&gt; if you haven't read it already. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not recommended because any error in prediction is compounded over time. please find my answer in the following post for a detailed explanation &#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/15524/is-time-series-multi-step-ahead-forecasting-a-sequence-to-sequence-problem/24430#24430&quot;&gt;Is time series multi-step ahead forecasting a sequence to sequence problem?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;How can this behaviour be explained?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;del&gt;If you are getting constant prediction value, the chance is you are having so-called &quot;gradient explosion problem&quot;, try to clip the gradient before backprop might be the solution. &lt;/del&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Correction. Had a bit better thinking about it.  it is not a gradient explosion problem here. More likely it is due to saturated neuron. Because LSTM uses sigmoid function, if the input is not properly scaled or weight is initialised inproperly, it could cause neuron saturation very quickly. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-08T12:24:33.607" LastActivityDate="2017-11-08T12:24:33.607" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24471" PostTypeId="2" ParentId="24449" CreationDate="2017-11-08T03:01:47.873" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I cant either feed the network a sequence of the 7 first values with the 8th as the target:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For typical LSTM, if your input sequence is [a,b,c,d,e,f,g] then your target sequence will be [b, c,d,e,f, g, h]. You will train your network with a long sequence. And after training, you can simply pass in [h] as input to get the estimate for [i]. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or I can do the classic rolling window, with a window size of, say, 2.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There is no point in using LSTM if your window size is 2. You will end with one input and one output. There is no longer term dependency to be learned in this setting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might use a fixed window approach if your individual sequence is very long. You can slice your series using the window approach. The benefit of doing this &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Reduce the length of the sequence. LSTM will still have problem learning dependency over very long steps due to gradient vanishing at the forget gate. &lt;/li&gt;&#xA;&lt;li&gt;Augment the dataset in the similar way you doing rotation and scaling for an image dataset. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Summary&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Say you have a single sequence [a,b,c,d,e,f,g,h,i]. Using a sliding window of size 6, your single squence will be transformed into 4 sequences&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;    [a,b,c,d,e,f],&#xA;    [b,c,d,e,f,g],&#xA;    [c,d,e,f,g,h],&#xA;    [d,e,f,g,h,i],&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And you input X will &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;    [a,b,c,d,e],&#xA;    [b,c,d,e,f],&#xA;    [c,d,e,f,g],&#xA;    [d,e,f,g,h],&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And your corresponding target Y will look like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;    [b,c,d,e,f],&#xA;    [c,d,e,f,g],&#xA;    [d,e,f,g,h],&#xA;    [e,f,g,h,i],&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-08T03:01:47.873" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24472" PostTypeId="2" ParentId="24469" CreationDate="2017-11-08T03:47:13.957" Score="1" Body="&lt;p&gt;Yes, it is very common and sometimes necessary to use the target variable for stratified sampling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the case of fraud detection, given a bunch of features about a person (e.g. income, gender, position etc) we want to predict the likelihood of that person has committed the crime (a boolean value indicating whether the person is a suspect). This dataset is likely to be very asymmetric with very few positive examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we want to use k-fold cross-validation, we must stratify the samples using the target variable. If we don't, we might end up with a fold without any positive example at all and no metrics can be calculated from that fold. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-08T03:47:13.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24474" PostTypeId="1" CreationDate="2017-11-08T06:10:42.277" Score="2" ViewCount="2078" Body="&lt;p&gt;I  trained Muti layer perceptrons using keras to classify cifar-10 dataset the results I got shows that there is something wrong in the code ,because all the epoch are identical here is the code :&#xA;In this I flatten the data from 32x32x3 to 3072 array.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   #nn layers &#xA;import numpy&#xA;from keras.datasets import cifar10&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import Dropout&#xA;from matplotlib import pyplot&#xA;from scipy.misc import toimage&#xA;&#xA;#load data&#xA;(Xt,Yt),(Xts,Yts)=cifar10.load_data()&#xA;&#xA;num_pix=Xt.shape[1]*Xt.shape[2]*Xt.shape[3]&#xA;X_tr=Xt.reshape(-1,3072).astype('float32')&#xA;X_ts=Xts.reshape(-1,3072).astype('float32')&#xA;#feature scaling&#xA;X_tr= X_tr / 255&#xA;X_ts = X_ts / 255&#xA;#one hot encoding&#xA;from keras.utils import np_utils&#xA;Ytr=Yt.reshape(-1,50000)&#xA;Yt=np_utils.to_categorical(Ytr)&#xA;Yts=Yts.reshape(-1,10000)&#xA;Yts=np_utils.to_categorical(Yts)&#xA;num_class=Yts.shape[1]&#xA;&#xA;&#xA;def base_model():&#xA;    model=Sequential()&#xA;    #build layers Dense(10, init=&quot;normal&quot;, activation=&quot;relu&quot;&#xA;    model.add(Dense(num_pix,input_dim=num_pix,activation='relu'))#input layer&#xA;    model.add(Dense(num_class,activation='softmax'))#output layer&#xA;    #compile model&#xA;    model.compile(loss='squared_hinge',optimizer='adam',metrics=['accuracy'])&#xA;    return model&#xA;    #build the model&#xA;model=base_model()&#xA;#Fit model&#xA;model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)&#xA;#Final assesment&#xA;score=model.evaluate(X_ts,Yts,verbose=0)&#xA;print(&quot;Baseline Error: %.2f%%&quot; % (100-score[1]*100))&#xA;&#xA;  Results :&#xA;&#xA; Train on 50000 samples, validate on 10000 samples&#xA;    Epoch 1/10&#xA;    134s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 2/10&#xA;    97s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 3/10&#xA;    99s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 4/10&#xA;    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 5/10&#xA;    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 6/10&#xA;    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 7/10&#xA;    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 8/10&#xA;    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 9/10&#xA;    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Epoch 10/10&#xA;    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000&#xA;    Baseline Error: 90.00%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help will be appreciated.can my model be overfitted?&lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="38506" LastEditDate="2017-11-08T09:28:28.543" LastActivityDate="2017-11-09T22:56:17.157" Title="Low accuracy using Multi Layer perceptron network on CIFAR10 dataset?" Tags="|machine-learning|neural-network|keras|image-classification|overfitting|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24475" PostTypeId="2" ParentId="24417" CreationDate="2017-11-08T06:53:25.917" Score="7" Body="&lt;p&gt;Yes, &lt;code&gt;torch.cat&lt;/code&gt; works with &lt;code&gt;backward&lt;/code&gt; operation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VHTtn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VHTtn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-08T06:53:25.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24477" PostTypeId="1" CreationDate="2017-11-08T09:08:26.453" Score="-1" ViewCount="98" Body="&lt;p&gt;I have a problem figuring out how to visualize two simple quantities using R. I want to compare their size by visualizing two proportional spatial figures next to each other and the size proportional to their volume, height or whatever. E.g. I have two numbers 45 and 15, after my code execution I would get two spatial figures, but first is three times bigger. &#xA;&lt;a href=&quot;https://visual.ly/blog/45-ways-to-communicate-two-quantities/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://visual.ly/blog/45-ways-to-communicate-two-quantities/&lt;/a&gt; &amp;lt;- I want to do something like in this article example no.26 &quot;Volumes&quot;. &#xA;Figures could be like cone, cube or 3d pie. Is is possible to realize that using ggplot2 or any other package?&lt;/p&gt;&#xA;" OwnerUserId="40898" LastEditorUserId="471" LastEditDate="2017-12-09T15:54:59.727" LastActivityDate="2017-12-09T15:54:59.727" Title="Two quantities visualization" Tags="|r|visualization|ggplot2|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24478" PostTypeId="2" ParentId="22828" CreationDate="2017-11-08T09:14:04.760" Score="3" Body="&lt;p&gt;I think the &lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType&quot; rel=&quot;nofollow noreferrer&quot;&gt;clustMixType&lt;/a&gt; package might give you better results/insights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By using &lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType/clustMixType.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this package&lt;/a&gt; you can use combination of Categorical and Numeric Data directly, it doesn’t need any kind of hot encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You just need to feed in the data and it automatically segregates into Categorical and Numeric Data, if you find any issues at the time of segregation you can use functions like &lt;code&gt;as.factor(to convert to a categorical)&lt;/code&gt; and &lt;code&gt;as.numeric(to convert to a Numeric field)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can calculate &lt;code&gt;Lambda(mean Distance value)&lt;/code&gt; before hand and fed in as an input to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don’t know the optimal number of clusters, you can use &lt;code&gt;WSS(within Sum of Squares)&lt;/code&gt;, &lt;code&gt;plot(elbow chart)&lt;/code&gt; to decide the optimal number of clusters.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="29575" LastEditDate="2017-11-08T23:30:10.447" LastActivityDate="2017-11-08T23:30:10.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24479" PostTypeId="1" CreationDate="2017-11-08T09:16:58.020" Score="3" ViewCount="129" Body="&lt;h3&gt;Problem:&lt;/h3&gt;&#xA;&lt;p&gt;I have a regression problem and I decided to useg Gradient Boosting Regression Trees to solve it. After all the preprocessing, I end up having around 130 attributes, 70K rows, and my cross-validated R-squared is 0.62.&lt;/p&gt;&#xA;&lt;h3&gt;Work So Far:&lt;/h3&gt;&#xA;&lt;p&gt;To increase my R-squared (and tighten my prediction intervals) I tried using a hint-feature that divides the target into linear groups (I manually decided for 4 groups and placed the target into 4 bins, - why 4? not too small not to big and not cleverly decided.).&lt;/p&gt;&#xA;&lt;p&gt;I am aware that this is not a usual way to go, but this is doable for my application since eventually, I will be collecting these hints from the user. (I will be asking for the target value for that user and suggest him/her another target value depending on his/her data.)&lt;/p&gt;&#xA;&lt;p&gt;So it would be possible for me to place each user into such groups since it seems like not all of the users behave the same and they need a separated modeling for the ones that are extremely high/low target values, plus the distribution is extremely right-skewed.&lt;/p&gt;&#xA;&lt;p&gt;After using this cheating-hint-feature, my cross-validated R-squared became 0.81. However, obviously, dividing the target into linear groups like that, brings the disadvantage of having a step-like model that has a clear separation between those groups while this is not necessarily always the case (a user that gives his target value most probably &lt;em&gt;but not necessarily&lt;/em&gt; should belong to that group), the boundary values for such binning should not be manually decided, and this model would definitely need to be smoothed.&lt;/p&gt;&#xA;&lt;h3&gt;Next Steps?&lt;/h3&gt;&#xA;&lt;p&gt;After seeing that such an approach can be helpful (and I could really make use of the user-given values), I am now thinking about smarter ways to separate between the users regarding these hints. I am thinking about applying hierarchical clustering and using the cluster information as a hint-feature but not sure if this is the best way to proceed.&lt;/p&gt;&#xA;&lt;p&gt;Would you maybe have suggestions for me for such method (and papers would be great) or statistical approach to figure out such boundary values (other than density-based methods)?&lt;/p&gt;&#xA;" OwnerUserId="40029" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-08T18:31:15.237" Title="Making Use of the Target Values for Regression" Tags="|clustering|regression|statistics|decision-trees|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24480" PostTypeId="1" CreationDate="2017-11-08T14:44:33.157" Score="1" ViewCount="78" Body="&lt;p&gt;I am designing an algorithm that can detect cheating in a game of chess.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a database of players who have been flagged by moderators as cheating, along with a number of game-level characteristics such as move times and move accuracy (as determined by a chess engine) for both black and white. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's specify a simple regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cheatflag ~ whiteMoveTime + blackMoveTime + whiteMoveAccuracy + blackMoveAccuracy&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's say cheatflag is a factor-level variable that is 0 if no cheater, 1 if white is the cheater, 2 if black is the cheater.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is that the predictive value of variables like whiteMoveTime and whiteMoveAccuracy depends crucially on who is doing the cheating.  If white is cheating, those variables should have different coefficients than if black is doing the cheating.  As far as I know, running most blackbox algorithms will derive coefficients that average across all the cases of cheating without understanding the hierarchical nature of the variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem has a Bayesian flavor with some kind of conditional assumption -- if black is cheating, then these variables matter; if white is cheating, these variables matter, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I deal with this issue?&lt;/p&gt;&#xA;" OwnerUserId="41597" LastEditorUserId="41597" LastEditDate="2017-11-08T14:59:35.080" LastActivityDate="2017-11-08T15:02:58.783" Title="ML algorithm where variable importance depends upon other variables - specifying conditionality" Tags="|machine-learning|r|bayesian|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24481" PostTypeId="2" ParentId="24480" CreationDate="2017-11-08T15:02:58.783" Score="1" Body="&lt;p&gt;A simple place to get started is logistic regression for white outcomes first and then for black outcomes. Set cheatflag to 0 if there's no cheating and 1 if white cheated, then run the regression. Do it again, but set cheatflag to 0 if there's no cheating and 1 if black cheated.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-08T15:02:58.783" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24482" PostTypeId="1" CreationDate="2017-11-08T15:37:01.970" Score="3" ViewCount="674" Body="&lt;h3&gt;Background:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am trying to do some analysis on our customer's around identifying early markers for debt delinquency.  We bill customers on a weekly basis and payment patterns will differ between accounts.  Good customers will normally pay Weekly, Fortnightly, 4 weekly or monthly on a 4-4-5 pattern.  Other customers will have an erratic payment pattern.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even where the payment pattern is consistent, they may be overpaying or not be paying the balance in full so there may be an underlying rising or falling debt.  &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Work So Far:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;To this end I have created two strings to represent:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Whether the weekly balance has decreased (v), increased (^) or not moved (-) (infer a payment has been made if balance is down or unchanged) and ...&lt;/li&gt;&#xA;&lt;li&gt;The level of indebtedness on the account based on the weekly charge ( - for in credit, . for balance cleared, 0-9 for balance less than 10 weeks, X for balance over 10 weeks)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So I now have a number of strings that look like this (good accounts)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;^v^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    231212-012-012-012-012-0120112-012-0-0120012-012-012-0120112-012-012-012-012-0123&#xA;^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    12-012-012-012-012-012-012-012-012-0-012-012-012-012-012-012-012-012-012-012-0123&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and like this for bad/erratic accounts&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;^v^^^vv^^v^^^v^^^v^v^v^^^v^^^v^^^v^v^^vv^^^v^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^v^    341234112312231234233412341234123423123312341233123301220122-011-011-011--00--001&#xA;v^v^v^^v^vv^vv^vv^v^^^^v^^^v^^^^v^v^^^v^^^v^^^^v^^^v^^^^v^^^v^^^v^^^^v^^^v^^^v^ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX9XXX9XXXX9XXX9XXXX9XXX89XX89XXX89XX789X78&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Next Steps:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;What I want to do now is identify accounts with specific payment patterns:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fortnightly = ^v^v&lt;/li&gt;&#xA;&lt;li&gt;4weekly = ^^^v^^^v^^^v&lt;/li&gt;&#xA;&lt;li&gt;Monthly = ^^^v^^^v^^^^v (note 4-4-5)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I also want to identify where those patterns have changed. Bearing in mind that on a 4 weekly or monthly pattern we may not be at the start of a pattern I also need to try and find the 'seed' week for each pattern&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will regard a pattern as consistent if I have seen it repeat at least 3 times so 12 weeks for the fortnightly pattern, 36 weeks for the 4 weekly and 36 weeks for the Monthly [monthly could be 445, 454 or 544 but must repeat] so I then want to flag up accounts that were consistent that have become erratic or changed their pattern.  I have data going back up to 5 years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the data in Microsoft SQL and currently the only analysis tools I have to hand are Microsoft SQL and Excel.  I'm not looking for code, just what the next practical steps should be in my strategy to analyse the data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking find the first down step in the pattern and remove this and the preceding up ticks (and associated characters from the multiplier) then run the remaining string through a number of REPLACE() functions but this doesn't feel very 'smart' &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better strategy to adopt as it feels like I am MUNGing the data?&lt;/p&gt;&#xA;" OwnerUserId="41602" LastEditorUserId="29575" LastEditDate="2017-11-08T16:21:02.227" LastActivityDate="2018-02-06T19:05:08.987" Title="How do I find a repeating pattern of unknown length and start within a string" Tags="|classification|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24483" PostTypeId="1" CreationDate="2017-11-08T15:54:08.460" Score="1" ViewCount="343" Body="&lt;p&gt;My question is if there is a DBMS that allows storing financial data (i.e. timestamp+value) together with small text fragments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These small fragments will not be very long, but it might help an algorithm to predict changes in the future. In summary, I wonder if you can suggest me what it is the best way to massively store timestamps+numeric value+(optional)text fragment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="13618" LastActivityDate="2017-11-08T16:19:23.957" Title="Best way to store financial data together with news" Tags="|time-series|finance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24484" PostTypeId="1" CreationDate="2017-11-08T16:16:26.013" Score="2" ViewCount="77" Body="&lt;p&gt;Input: $x \in \mathbb{R}^+$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Expected output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x) = \begin{cases}x &amp;amp;\text{for } x \leq 5\\10 -&#xA; x &amp;amp;\text{for } x &amp;gt; 5\end{cases}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically you have a neural network with one input neuron and one output neuron. Let's say the output has the identity as an activation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the simplest network with ReLU activations that fits this function?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-09T08:53:26.487" LastActivityDate="2017-11-10T08:45:42.067" Title="What is the simplest ReLU network that can learn an upside-down v pattern?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24485" PostTypeId="2" ParentId="24483" CreationDate="2017-11-08T16:19:23.957" Score="3" Body="&lt;p&gt;The short answer is that (almost) any DBMS will be able to store that information for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use Microsoft SQL and you can store the record with a specified or system calculated date-time, a column for the numeric value and you can store the story text in one of four ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Plain text as a NVARCHAR&lt;/li&gt;&#xA;&lt;li&gt;structured text as XML (or JSON if you are on SQL2016 or above)&lt;/li&gt;&#xA;&lt;li&gt;Filestream if the data format is binary (normally an image, audio or video)&lt;/li&gt;&#xA;&lt;li&gt;As a reference to a file on disk&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;MSSQL is an RDBMS (relational database management system) Others in this classification are Oracle, MySql, Postgress.  There is another school of database classified as NOSQL or Document Databases and these include Mongo, CouchDB, DocumentDB and RavenDB.  These differ from RDBMS in the way they store data and manage the relationships (e.g. find all the articles relating to company X in the time period 1st April to 6th June)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would suggest that if you don't know how you are going to want to process the data in the future, look at storing it in a NOSQL database as it provides much greater flexibility of record structure.  Most can be installed on-premise with a free / open source licence and most are also available on a hosted platform for a relatively small sum.  Many of the RDBMS (esp Oracle and MSSQL) are exceeding expensive for this sort of project.  That said, Azure SQL - the hosted version of SQL can be had for as little as $8 per month depending on your storage requirements.&lt;/p&gt;&#xA;" OwnerUserId="41602" LastActivityDate="2017-11-08T16:19:23.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24488" PostTypeId="2" ParentId="24482" CreationDate="2017-11-08T17:59:45.543" Score="2" Body="&lt;p&gt;From your description, it sounds like &lt;a href=&quot;https://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow noreferrer&quot;&gt;autocorrelation&lt;/a&gt; could be a good initial step to identify the frequency. Autocorrelation is basically the product of the sequence with itself shifted at 0, 1, 2, 3, and so forth. If a client &lt;em&gt;tends&lt;/em&gt; to pay monthly, you will see the first positive bump around 4, because that is where each payment aligns with the payment after it. Note that this will work irrespective of the start of the sequence, and is even resilient to some noise. Even &lt;a href=&quot;http://www.real-statistics.com/time-series-analysis/stochastic-processes/autocorrelation-function/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Excel seems to have an autocorrelation function&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you could refine this further with other steps, and/or try this on different windows (in case the patter changes over time).&lt;/p&gt;&#xA;" OwnerUserId="9518" LastActivityDate="2017-11-08T17:59:45.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24489" PostTypeId="2" ParentId="24479" CreationDate="2017-11-08T18:31:15.237" Score="1" Body="&lt;p&gt;Clustering is definitely something that can help. As you describe, the issue is that you can see that not all instances &quot;behave&quot; the same way. So if you can cluster them into groups that do behave more similarly, you would probably improve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any clustering technique would work in theory. Although, techniques like K-means for instance do favour clusters of even-size, and so if your problem isn't necessarily &quot;balanced&quot;, I would be cautious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could also do to improve your clustering is to look at which features are more discriminative (perhaps look at the information gain) and use a weighted distance to favour these features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way I feel could help you is if you used meta learning/stacking. By that I mean that you could use decision trees to split your data in a supervised way (as opposed to clustering), and at the leaves, use a more elaborate regressor than majority voting.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-11-08T18:31:15.237" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24490" PostTypeId="2" ParentId="24452" CreationDate="2017-11-08T18:56:26.947" Score="40" Body="&lt;p&gt;(Assuming you are talking about supervised learning)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correlated features will not always worsen your model, but they will not always improve it either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three main reasons why you would remove correlated features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Make the learning algorithm faster&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Due to the curse of dimensionality, less features usually mean high improvement in terms of speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If speed is not an issue, perhaps don't remove these features right away (see next point)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Decrease harmful bias&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The keyword being harmful. If you have correlated features but they are also correlated to the target, you want to keep them. You can view features as hints to make a good guess, if you have two hints that are essentially the same, but they are good hints, it may be wise to keep them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some algorithms like Naive Bayes actually directly benefit from &quot;positive&quot; correlated features. And others like random forest may indirectly benefit from them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine having 3 features A, B, and C. A and B are highly correlated to the target and to each other, and C isn't at all. If you sample out of the 3 features, you have 2/3 chance to get a &quot;good&quot; feature, whereas if you remove B for instance, this chance drops to 1/2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if the features that are correlated are not super informative in the first place, the algorithm may not suffer much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So moral of the story, removing these features might be necessary due to speed, but remember that you might make your algorithm worse in the process. Also, some algorithms like decision trees have feature selection embedded in them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way to deal with this is to use a wrapper method for feature selection. It will remove redundant features only if they do not contribute directly to the performance. If they are useful like in naive bayes, they will be kept. (Though remember that wrapper methods are expensive and may lead to overfitting)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Interpretability of your model&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If your model needs to be interpretable, you might be forced to make it simpler. Make sure to also remember Occam's razor. If your model is not &quot;that much&quot; worse with less features, then you should probably use less features.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-11-08T18:56:26.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24491" PostTypeId="2" ParentId="24466" CreationDate="2017-11-08T19:03:19.137" Score="0" Body="&lt;p&gt;I always find the notion of false positive and negative confusing, especially when it comes to multi-class problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good rule of thumb that I came up with is the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;True positive&lt;/strong&gt;: &quot;I predicted that you were a certain class, and I was right&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;False positive&lt;/strong&gt;: &quot;I predicted that you were a certain class but I was wrong&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;True negative&lt;/strong&gt;: &quot;I predicted that you weren't a certain class, and I was right&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;False negative&lt;/strong&gt;: &quot;I predicted that you weren't a certain class, but I was wrong&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in your case, there are &lt;strong&gt;31&lt;/strong&gt; true positives with regard to the first class since you predicted 31 times that something was class 1 but you were wrong.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-11-08T19:03:19.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24492" PostTypeId="2" ParentId="8286" CreationDate="2017-11-08T19:12:17.297" Score="8" Body="&lt;p&gt;&lt;a href=&quot;http://featuretools.com&quot; rel=&quot;noreferrer&quot;&gt;Featuretools&lt;/a&gt; is a recently released python library for automated feature engineering.  It's based on an algorithm called &lt;a href=&quot;http://dai.lids.mit.edu/DSAA_DSM_2015.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deep Feature Synthesis&lt;/a&gt; originally developed in 2015 MIT and tested on public data science competitions on Kaggle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how it fits into the common data science process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4uPUo.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4uPUo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The aim of the library is to not only help experts build better machine learning models faster, but to make the data science process less intimidating to people trying to learn. If you have event driven or relational data, I highly recommend you check it it out!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer: I am one of the developers on the project.&lt;/p&gt;&#xA;" OwnerUserId="41611" LastActivityDate="2017-11-08T19:12:17.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24493" PostTypeId="2" ParentId="24392" CreationDate="2017-11-08T19:26:28.093" Score="5" Body="&lt;ul&gt;&#xA;&lt;li&gt;Short answer:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;you need to deal with class imbalance &lt;strong&gt;if/because&lt;/strong&gt; it makes your model better (on unseen data). &quot;Better&quot; is something that you have to define yourself. It could be accuracy, it could be a cost, it could be the true positive rate etc.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Long answer:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is a subtle nuance that is important to grasp when talking about class imbalance. Namely, is your data imbalanced because:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;the distribution of the data is itself imbalanced&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In some cases, one class occurs much more than another. And it's okay. In this case, you have to look at whether certain mistakes are more costly than others. This is the typical example of detecting deadly diseases in patients, figuring out if someone is a terrorist etc. This goes back to the short answer. If some mistakes are more costly than others, you'll want to &quot;punish&quot; them by giving them a higher cost. Therefore, a better model will have a lower cost. If all mistakes are as bad, then there is no real reason why you should use cost sensitive models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also important to note that using cost-sensitive models is not specific to imbalanced datasets. You can use such models if your data is perfectly balanced as well.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;strong&gt;it doesn't represent the true distribution of the data&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sometimes your data is &quot;imbalanced&quot; because it doesn't represent the true distribution of the data. In this case, you have to be careful, because you have &quot;too many&quot; examples of one class and &quot;too few&quot; of the other, and therefore, you need to make sure that your model doesn't over-/underfit on one of these classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is different than using costs because it might not be the case that one mistake is worse than another. What would happen is that you would be biased and it wouldn't be beneficial for your model if the unseen data doesn't have the same distribution as the data you trained on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that I give you training data and your goal is to guess if something is red or blue. Whether you mistake blue for red or red for blue doesn't make much of a difference. Your training data has 90% red instances where in real life, they only happen 10% of the time. You would need to deal with that in order to make your model better.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-11-09T09:59:26.737" LastActivityDate="2017-11-09T09:59:26.737" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24494" PostTypeId="1" AcceptedAnswerId="24498" CreationDate="2017-11-08T19:52:30.487" Score="4" ViewCount="4119" Body="&lt;p&gt;Should we add bias to each entry of the convolution then sum, or add bias once at end of calculating the convolution in CNNs?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-27T19:46:10.440" Title="Should the bias value be added after convolution operation in CNNs?" Tags="|machine-learning|deep-learning|cnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24495" PostTypeId="2" ParentId="24379" CreationDate="2017-11-08T20:01:40.190" Score="0" Body="&lt;p&gt;Since you only have 5 groups, you should probably look at distances instead of clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the question is: which distance should you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course you could use Euclidean, Manhattan, or cosine distances and be done with it. In this case, I'd pick cosine distance because it reduces the impact of a single dimension on the overall distance and since you have 2000 features/interests, it might help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I'm guessing that the interest groups are somewhat correlated. In an extreme case, 1999 interests are very correlated, and 1 isn't. If this happens and you use regular distances, it means that two groups who only agree on the 1999 interests will be considered much closer than the ones who only agree on the 1 interest. Even though, you know that you're only dealing with two interests here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you might want to use some form of weighting to compute your distances using the correlation between the interest groups. Perhaps the more unique groups should matter more than the groups who are very similar to other interests. To do that, you could use dimensionality reduction techniques like PCA. PCA will &quot;remove&quot; redundant interests and &quot;group&quot; them into one. Once you have reduced the dimensionality of your data (let's say that now you are looking at 20 interests instead of 2000), you can compute your distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, distances are subjective and you have to define how much agreeing on specific interests matters. Perhaps agreeing on sports matters more than agreeing on books. If you have this prior knowledge, you'd have to manually input it into your weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've decided on how to compute your distances, you could compute multiple distance matrices by sampling subsets of the features (interests). That way, groups that are far away from each other will remain far away most of the time, same thing with groups that are close to each other. You could then look at the average distance to decide how far away the groups are from each other.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-11-09T10:09:52.303" LastActivityDate="2017-11-09T10:09:52.303" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24496" PostTypeId="2" ParentId="24408" CreationDate="2017-11-08T20:23:00.013" Score="1" Body="&lt;p&gt;This is an interesting problem because obviously, you want to train a model that performs well on unseen data, and therefore you'd like to train it on data resembling the one you'll encounter later on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you already know the true distribution of your data. I would use what Keith mentioned in the comment. Fit a curve on the histogram of that data, and use the inverse as weight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could also do if you know the true mean of your distribution is build some type of &quot;discriminator&quot;,  which guesses whether a data point will fall on the &quot;left&quot; or &quot;right&quot; of the mean. Since your data is skewed, you could also have a discriminator for the median, which might perform better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you can build models for each &quot;side&quot; of the mean. Since your data is heavily skewed, you will have a lot of data for one, and not so much for the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you could have a final model that takes in as input the discriminator(s) (how certain it is that the data point will fall left or right), and the output of both models. That final model should smooth out some of the mistakes done by the discriminator(s) and both models.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-11-08T20:23:00.013" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24497" PostTypeId="1" CreationDate="2017-11-08T20:43:41.197" Score="2" ViewCount="1335" Body="&lt;p&gt;I working on multivariate time series data. I have sensor data generated by a machine every time it is operated. Data set consists of machine_ID(machines of same model), hours_ operated, measurements from various sensors. The machine starts to degrade after operating for certain hours. I would like to find the hours after which there is step change after which the performance starts to degrade. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to do this using machine learning approach preferably and would like to plot the graph marking the deviation. Which ML techniques could be used for this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pkwAf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pkwAf.jpg&quot; alt=&quot;Here is how the plot of values &amp;#39;sensor 2&amp;#39; from run-to-failure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/P2U4I.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/P2U4I.jpg&quot; alt=&quot;Snapshot of data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have performed exploratory data analysis where I could find the point at which there is deviation occurrence. Now, I want to confirm this by running a model to detect the occurrence of step change. In the figure above, the decline starts somewhere at 100 and decline gradually. Now, is there any way I could find this pint through models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I greatly appreciate any links or suggestion to deal with this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="41612" LastEditorUserId="41612" LastEditDate="2017-11-10T16:40:26.613" LastActivityDate="2023-12-22T21:07:15.827" Title="Detect the time at which deviation occurs in time series data" Tags="|machine-learning|python|deep-learning|time-series|anomaly-detection|" AnswerCount="3" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="24498" PostTypeId="2" ParentId="24494" CreationDate="2017-11-08T21:34:59.313" Score="7" Body="&lt;p&gt;Short answer: the bias is added once after the convolution has been calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long answer: discrete convolution that you see in CNNs is a linear function applied to pixel values in a small region of an image. The output of this linear function is then jammed through some nonlinearity (like ReLU). For a region $\mathbf{x}$ of size $i \times j$ of an image and a convolutional filter $\mathbf{k}$, and no bias term, this linear function $f$ would be defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;f(\mathbf{x}, \mathbf{k}) = \mathbf{x}*\mathbf{k} = \sum_{i,j} k_{i,j} x_{i,j} &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without a bias term, this linear function $f$ must go through the origin. In other words, if $\mathbf{x}$ or $\mathbf{k}$ is all zeroes, the output of $f$ will be zero as well. This may not be desirable, so we add a bias term $b$. This gives the model more flexibility by providing a value that is always added to the output of the convolution, regardless of the values of $\mathbf{x}$ and $\mathbf{k}$ -- in other words, it's the intercept value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;f(\mathbf{x}, \mathbf{k}, b) = b + (\mathbf{x}*\mathbf{k}) = b + \sum_{i,j} k_{i,j} x_{i,j} &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this value was added to each entry of the convolution, it would not achieve its purpose as $f$ would still necessarily go through the origin.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-08T21:34:59.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24499" PostTypeId="2" ParentId="19241" CreationDate="2017-11-08T22:22:31.667" Score="0" Body="&lt;p&gt;The code shown here is a recursive implementation of dynamic programming used for time series analysis for similiarity, there is though a more optimal implementation named Fast Dynamic Time Warping which significantly reduces time and memory complexity to $O(n)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will see a significant improvement from original DTW although the measured distance will be directly impacted depending on the distance measurement used (usually is the ecludidean distance as shown here). But there are others that would provide better results depending on the time series nature.&lt;/p&gt;&#xA;" OwnerUserId="41618" LastEditorUserId="29575" LastEditDate="2017-11-08T23:29:46.927" LastActivityDate="2017-11-08T23:29:46.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24500" PostTypeId="1" AcceptedAnswerId="24545" CreationDate="2017-11-08T22:35:21.340" Score="1" ViewCount="200" Body="&lt;p&gt;I have logs of user activity on my system. This is a CMS system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The logs consist of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;User ID &lt;/li&gt;&#xA;&lt;li&gt;Action Performed (one of 110 possible actions - things like page edited, page read, login, etc.)&lt;/li&gt;&#xA;&lt;li&gt;Site action was performed on (not relevant for all possible actions)&lt;/li&gt;&#xA;&lt;li&gt;Object action was performed on (document name or document id - not relevant for all possible actions) &lt;/li&gt;&#xA;&lt;li&gt;Date and time that the action was performed&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to use this data to cluster users into groups with similar behavior, so that I can then survey the groups to find out who they are and then target them with training interventions, newsletters etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need help with converting the above list of things into clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What techniques can I use to generate a matrix from the above data that I can feed into a K-Means cluster algorithm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried counting action by user and using that to feed a cluster algorithm but the result was useless. I also tried treating the site and document ids as text, feature hashing that and feeding it into a K-Means cluster model, also got useless data out.&lt;/p&gt;&#xA;" OwnerUserId="41616" LastEditorUserId="29575" LastEditDate="2017-11-08T23:29:49.677" LastActivityDate="2017-11-09T22:43:43.397" Title="How can I turn user audit data into behavior clusters?" Tags="|clustering|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24501" PostTypeId="2" ParentId="18154" CreationDate="2017-11-08T22:43:27.327" Score="5" Body="&lt;p&gt;Based on the short example DataFrame you provided, this block of code will include all of the months. It is based on using the Series.reindex method and creating a new MultiIndex with the additional values for the months:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;# Load example data into DataFrame&#xA;df = pd.read_table(&quot;categorical_data.txt&quot;, delim_whitespace=True)&#xA;&#xA;# Transform to a count&#xA;count = df.groupby(['id', 'code', 'month']).month.count()&#xA;&#xA;# Re-create a new array of levels, now including all 12 months&#xA;levels = [count.index.levels[0].values, count.index.levels[1].values, range(12)]&#xA;new_index = pd.MultiIndex.from_product(levels, names=count.index.names)&#xA;&#xA;# Reindex the count and fill empty values with zero (NaN by default)&#xA;count = count.reindex(new_index, fill_value=0)&#xA;print(count)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Printing the result, I get something like this (only showing the first entry for sally/s_A):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id     code  month&#xA;sally  s_A   0        4&#xA;             1        0&#xA;             2        0&#xA;             3        0&#xA;             4        0&#xA;             5        0&#xA;             6        0&#xA;             7        0&#xA;             8        0&#xA;             9        0&#xA;             10       0&#xA;             11       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41617" LastEditorUserId="69968" LastEditDate="2019-03-22T01:53:00.910" LastActivityDate="2019-03-22T01:53:00.910" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24502" PostTypeId="1" CreationDate="2017-11-08T23:23:25.750" Score="1" ViewCount="38" Body="&lt;p&gt;Are there any ways to check if more data can help the quality of kmeans clustering? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closest research I could find was &lt;a href=&quot;https://www.microsoft.com/en-us/research/publication/the-learning-curve-method-applied-to-clustering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;for EM algorithms&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thing similar for evaluating kmeans ? &lt;/p&gt;&#xA;" OwnerUserId="41293" LastEditorUserId="29575" LastEditDate="2017-12-02T22:13:51.353" LastActivityDate="2017-12-02T22:13:51.353" Title="Kmeans quality vs data" Tags="|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24503" PostTypeId="1" AcceptedAnswerId="24505" CreationDate="2017-11-08T23:48:16.360" Score="1" ViewCount="1205" Body="&lt;p&gt;I'm working on an implementation of a paper and I haven't been able to find a description of average unpooling. I tried using max unpooling instead and my results were not great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="41620" LastEditorUserId="29169" LastEditDate="2021-02-15T22:21:18.583" LastActivityDate="2021-02-15T22:21:18.583" Title="What is average unpooling?" Tags="|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24504" PostTypeId="2" ParentId="14524" CreationDate="2017-11-09T00:01:38.143" Score="1" Body="&lt;p&gt;Following the publication of Dynamic Routing of Capsules (&lt;a href=&quot;https://arxiv.org/abs/1710.09829&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1710.09829&lt;/a&gt;), there have been open source implementations of this idea released. See &lt;a href=&quot;https://github.com/XifengGuo/CapsNet-Keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/XifengGuo/CapsNet-Keras&lt;/a&gt; and &lt;a href=&quot;https://github.com/soskek/dynamic_routing_between_capsules&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/soskek/dynamic_routing_between_capsules&lt;/a&gt; for two such examples. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2017-11-09T00:01:38.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24505" PostTypeId="2" ParentId="24503" CreationDate="2017-11-09T00:20:48.440" Score="1" Body="&lt;p&gt;When you go backward for max-pooling you keep track of the position of the maximum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ X = \begin{bmatrix}&#xA;1 &amp;amp;&amp;amp; 2 \\&#xA;3 &amp;amp;&amp;amp; 4&#xA;\end{bmatrix} \quad \rightarrow \quad dZ *\begin{bmatrix}&#xA;0 &amp;amp;&amp;amp; 0 \\&#xA;0 &amp;amp;&amp;amp; 1&#xA;\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gives you the position of the input value that ultimately influenced the cost/output. The gradient will then be &quot;propagated&quot; back to this value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For average pooling, all values influence the cost equally and therefore it should look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ dZ = z \quad \rightarrow  \quad \begin{bmatrix}&#xA;z/n &amp;amp;&amp;amp; z/n \\&#xA;z/n &amp;amp;&amp;amp; z/n&#xA;\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $n$ depends on the filter size, here $n=4$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gradient gets distributed evenly back to all input values.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-11-09T00:20:48.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24506" PostTypeId="1" CreationDate="2017-11-09T02:55:49.833" Score="8" ViewCount="6534" Body="&lt;p&gt;I have read this question: &lt;a href=&quot;https://datascience.stackexchange.com/questions/12318/how-do-i-interpret-the-output-of-xgboost-importance&quot;&gt;How do i interpret the output of XGBoost importance?&lt;/a&gt; about the three different types of feature importances: frequency (called &quot;weight&quot; in Python XGBoost), gain, and cover.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I have a feature, Gender, that has a very low importance based on the frequency metric, but is the most important feature by far based on both the gain, and cover metrics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know gender should be important for what I'm predicting. If I plot only gender vs. the target, there is a clear correlation. I am not surprised that it's the most important feature. I just want to know why it's not considered an important feature based on the frequency metric. &lt;/p&gt;&#xA;" OwnerUserId="39580" LastActivityDate="2018-01-27T20:53:09.240" Title="XGBoost Feature importance - Gain and Cover are high but Frequency is low" Tags="|machine-learning|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24507" PostTypeId="1" CreationDate="2017-11-09T04:05:19.887" Score="2" ViewCount="648" Body="&lt;p&gt;I have been working on a tic-tac-toe assignment for my Robot Learning class. We were asked to program a tic-tac-toe game and assign; +1 if X wins, -1 if O wins and 0 it the game results with a draw. In part 1, we were told to use Q table and in part 2 we were told to replace the Q table with a Neural network as the functional approximator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?&lt;/p&gt;&#xA;" OwnerUserId="41626" LastEditorUserId="41626" LastEditDate="2017-11-12T04:10:47.940" LastActivityDate="2017-11-12T08:55:00.107" Title="Q learning and Neural Network for Tic Tac Toe" Tags="|machine-learning|neural-network|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24509" PostTypeId="2" ParentId="24502" CreationDate="2017-11-09T06:42:30.697" Score="1" Body="&lt;p&gt;The estimation quality of the mean improves with sqrt(n).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So with more and more data, your mean will get more precise. But the improvements get slower and slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It won't help with all the other issues of k-means, such as problems with different cluster diameters and outliers. And more data probably means more outliers...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try a similar approach to that paper with k-means with some slight modifications. Try plotting the sum of squared deviations between the holdout set, and the nearest centroid each. As long as this improves, the clusters fit your holdout set better.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-11-09T06:48:36.610" LastActivityDate="2017-11-09T06:48:36.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24510" PostTypeId="1" AcceptedAnswerId="24518" CreationDate="2017-11-09T07:14:31.013" Score="0" ViewCount="1315" Body="&lt;p&gt;In &lt;a href=&quot;https://guillaumegenthial.github.io/sequence-tagging-with-tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog&lt;/a&gt;, it teaches us how to get a word embedding using bi-lstm in character level like the image below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cBxHh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cBxHh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering how to optimize the word vector from character-based approach. Is there any loss function or any strategy to update or optimize the word vector trained by a word's characters?&lt;/p&gt;&#xA;" OwnerUserId="41629" LastEditorUserId="41629" LastEditDate="2017-11-10T01:42:52.793" LastActivityDate="2017-11-10T01:42:52.793" Title="Character-based word representation using bi-lstm" Tags="|deep-learning|word-embeddings|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24511" PostTypeId="1" AcceptedAnswerId="24539" CreationDate="2017-11-09T07:42:15.517" Score="69" ViewCount="101173" Body="&lt;p&gt;In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find any valuable reason for shuffling data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read this post &lt;a href=&quot;https://stats.stackexchange.com/a/180847/179078&quot;&gt;here&lt;/a&gt; discussing when we need to shuffle data, but it is not obvious why we should shuffle the data. Furthermore, I have frequently seen in algorithms such as Adam or SGD where we need batch gradient descent (data should be separated to mini-batches and batch size has to be specified). It is vital according to this &lt;a href=&quot;https://stats.stackexchange.com/questions/245502/shuffling-data-in-the-mini-batch-training-of-neural-network&quot;&gt;post&lt;/a&gt; to shuffle data for each epoch to have different data for each batch. So, perhaps the data is shuffled and more importantly changed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do we do this?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="29169" LastEditDate="2019-05-28T05:56:53.067" LastActivityDate="2023-04-14T14:39:58.223" Title="Why should the data be shuffled for machine learning tasks" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="11" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="24513" PostTypeId="1" CreationDate="2017-11-09T08:14:57.000" Score="4" ViewCount="2396" Body="&lt;p&gt;I solved the &lt;a href=&quot;https://gym.openai.com/envs/CartPole-v0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CartPole-v0&lt;/a&gt; with a CEM agent pretty easily (&lt;a href=&quot;https://martin-thoma.com/rl-agents/&quot; rel=&quot;nofollow noreferrer&quot;&gt;experiments and code&lt;/a&gt;), but I struggle to find a setup which works with DQN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know which parameters should be adjusted so that the mean reward is about 200 for this problem?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;What I tried&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adjustments in the model: Deeper / less deep, neurons per layer&lt;/li&gt;&#xA;&lt;li&gt;Memory size (how many steps are stored for replay)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;What I'm unsure about&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should I choose the memory? Is higher always better? - Some quick experiments indicate that there might be a sweet-spot - not too high, but also not too low. I have no idea how to figure out the region of that sweet spot.&lt;/li&gt;&#xA;&lt;li&gt;Window size: Having a window size of 1 seems to work well in this case. Bigger window sizes seem to be worse. Is there any indicator when to increase the Window size?&lt;/li&gt;&#xA;&lt;li&gt;How to deal with delayed rewards: Suppose the CartPole did not start upright, but down. Then it would only get rewards late. Would this be a case for increasing the window size?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;My current code&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I use &lt;a href=&quot;https://github.com/matthiasplappert/keras-rl&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras-RL&lt;/a&gt; for the model and &lt;a href=&quot;https://github.com/openai/gym&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI gym&lt;/a&gt; for the environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;import numpy as np&#xA;import gym&#xA;&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Activation, Flatten&#xA;from keras.optimizers import Adam&#xA;&#xA;from rl.agents.dqn import DQNAgent&#xA;from rl.policy import LinearAnnealedPolicy, EpsGreedyQPolicy&#xA;from rl.memory import EpisodeParameterMemory&#xA;&#xA;&#xA;def main(env_name, nb_steps):&#xA;    # Get the environment and extract the number of actions.&#xA;    env = gym.make(env_name)&#xA;    np.random.seed(123)&#xA;    env.seed(123)&#xA;&#xA;    nb_actions = env.action_space.n&#xA;    input_shape = (1,) + env.observation_space.shape&#xA;    model = create_nn_model(input_shape, nb_actions)&#xA;&#xA;    # Finally, we configure and compile our agent.&#xA;    memory = EpisodeParameterMemory(limit=2000, window_length=1)&#xA;&#xA;    policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1.,&#xA;                                  value_min=.1, value_test=.05,&#xA;                                  nb_steps=1000000)&#xA;    agent = DQNAgent(model=model, nb_actions=nb_actions, policy=policy,&#xA;                     memory=memory, nb_steps_warmup=50000,&#xA;                     gamma=.99, target_model_update=10000,&#xA;                     train_interval=4, delta_clip=1.)&#xA;    agent.compile(Adam(lr=.00025), metrics=['mae'])&#xA;    agent.fit(env, nb_steps=nb_steps, visualize=False, verbose=2)&#xA;&#xA;    # After training is done, we save the best weights.&#xA;    agent.save_weights('dqn_{}_params.h5f'.format(env_name), overwrite=True)&#xA;&#xA;    # Finally, evaluate the agent&#xA;    history = agent.test(env, nb_episodes=100, visualize=False)&#xA;    rewards = np.array(history.history['episode_reward'])&#xA;    print((&quot;Test rewards (#episodes={}): mean={:&amp;gt;5.2f}, std={:&amp;gt;5.2f}, &quot;&#xA;           &quot;min={:&amp;gt;5.2f}, max={:&amp;gt;5.2f}&quot;)&#xA;          .format(len(rewards),&#xA;                  rewards.mean(),&#xA;                  rewards.std(),&#xA;                  rewards.min(),&#xA;                  rewards.max()))&#xA;&#xA;&#xA;def create_nn_model(input_shape, nb_actions):&#xA;    &quot;&quot;&quot;&#xA;    Create a neural network model which maps the input to actions.&#xA;&#xA;    Parameters&#xA;    ----------&#xA;    input_shape : tuple of int&#xA;    nb_actoins : int&#xA;&#xA;    Returns&#xA;    -------&#xA;    model : keras Model object&#xA;    &quot;&quot;&quot;&#xA;    model = Sequential()&#xA;    model.add(Flatten(input_shape=input_shape))&#xA;    model.add(Dense(32))&#xA;    model.add(Activation('relu'))&#xA;    model.add(Dense(64))&#xA;    model.add(Activation('relu'))&#xA;    model.add(Dense(64))&#xA;    model.add(Activation('relu'))&#xA;    model.add(Dense(512))&#xA;    model.add(Activation('relu'))&#xA;    model.add(Dense(nb_actions))&#xA;    model.add(Activation('linear'))&#xA;    print(model.summary())&#xA;    return model&#xA;&#xA;&#xA;def get_parser():&#xA;    &quot;&quot;&quot;Get parser object.&quot;&quot;&quot;&#xA;    from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter&#xA;    parser = ArgumentParser(description=__doc__,&#xA;                            formatter_class=ArgumentDefaultsHelpFormatter)&#xA;    parser.add_argument(&quot;--env&quot;,&#xA;                        dest=&quot;environment&quot;,&#xA;                        help=&quot;OpenAI Gym environment&quot;,&#xA;                        metavar=&quot;ENVIRONMENT&quot;,&#xA;                        default=&quot;CartPole-v0&quot;)&#xA;    parser.add_argument(&quot;--steps&quot;,&#xA;                        dest=&quot;steps&quot;,&#xA;                        default=10000,&#xA;                        type=int,&#xA;                        help=&quot;how many steps is the model trained?&quot;)&#xA;    return parser&#xA;&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;    args = get_parser().parse_args()&#xA;    main(args.environment, args.steps)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-09T08:50:47.417" LastActivityDate="2019-12-31T18:01:20.147" Title="What is a minimal setup to solve the CartPole-v0 with DQN?" Tags="|reinforcement-learning|keras-rl|openai-gym|dqn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24514" PostTypeId="5" CreationDate="2017-11-09T08:15:39.670" Score="0" Body="&lt;h2&gt;Code&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/matthiasplappert/keras-rl&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/matthiasplappert/keras-rl&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-09T16:07:10.997" LastActivityDate="2017-11-09T16:07:10.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24515" PostTypeId="4" CreationDate="2017-11-09T08:15:39.670" Score="0" Body="A framework for Reinforcement Learning with Keras" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-09T16:07:04.107" LastActivityDate="2017-11-09T16:07:04.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24517" PostTypeId="2" ParentId="24484" CreationDate="2017-11-09T08:32:27.210" Score="2" Body="&lt;p&gt;In order to find the network, I tried a couple of different small networks:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;One hidden layer with two neurons&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This would be the function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ASZLU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ASZLU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GIHrT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GIHrT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;2 hidden layers, 3 neurons in total&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is pretty close:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KkLFb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KkLFb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always include biases.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The solution&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    / o (ReLU) \&#xA;IN o            o (linear)&#xA;    \ o (ReLU) /&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-10T08:45:42.067" LastActivityDate="2017-11-10T08:45:42.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24518" PostTypeId="2" ParentId="24510" CreationDate="2017-11-09T10:06:51.697" Score="1" Body="&lt;p&gt;This would generally only be a part of a network, used for some task like classification, entity recognition or machine translation. The loss used for this task will guide the learning of the embedding space. This is different from approaches like word2vec or GLoVe where you use context windows to learn an embedding, which is an unsupervised (or maybe self-supervised) approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach that you mention here could still be used in these scenarios, by structuring it with a similar loss as word2vec or GLoVe. One advantage would be that you can leverage word similarity (words that end with -ly have a similar function). However these models are usually trained on incredibly large corpora where these character based similarities are not that relevant, and the computational cost increases quite a bit by incorporating bigger models like LSTMs. &lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-09T10:06:51.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24521" PostTypeId="2" ParentId="24507" CreationDate="2017-11-09T10:44:14.013" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, I would expect a neural network for Q Learning to find the optimal policy, provided it remains stable*. The value estimates might be slightly inaccurate, but the resulting policy should be completely optimal. That is because in tic tac toe, all the value estimates should be -1, 0 or +1, and the data is cleanly separated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to get a neural network to learn the optimal Q table from the first experiment using supervised learning. In fact that would be a good test of whether your NN has capacity to learn that table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;* Neural networks added naively to Q-learning agents are often not stable. In fact that is so common a problem in scaling up RL agents that it has a name: &quot;&lt;a href=&quot;https://davidsanwald.github.io/2016/12/11/Double-DQN-interfacing-OpenAi-Gym.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the deadly triad&lt;/a&gt;&quot;. This is generally not solved by elegant mathematical changes to the agent, but by some engineering tricks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Experience replay. Save observations (S, A, R, S') and sample from this memory table later to train in mini-batches.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Alternating networks. Use an old frozen copy of the neural network to calculate $\text{max}_{a'} Q(S',a')$ for the TD target $R + \gamma\text{max}_{a'} Q(S',a')$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-12T08:55:00.107" LastActivityDate="2017-11-12T08:55:00.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24522" PostTypeId="2" ParentId="24511" CreationDate="2017-11-09T11:54:47.063" Score="8" Body="&lt;p&gt;Suppose data is sorted in a specified order. For example a data set which is sorted base on their class. So, if you select data for training, validation, and test without considering this subject, you will select each class for different tasks, and it will fail the process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, to impede these kind of problems, a simple solution is shuffling the data to get different sets of training, validation, and test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the mini-batch, answers to &lt;a href=&quot;https://stats.stackexchange.com/q/245502/144441&quot;&gt;this post&lt;/a&gt; can be a solution to your question.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastEditorUserId="27665" LastEditDate="2017-11-09T12:04:41.790" LastActivityDate="2017-11-09T12:04:41.790" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24524" PostTypeId="2" ParentId="24511" CreationDate="2017-11-09T12:38:49.853" Score="47" Body="&lt;p&gt;Shuffling data serves the purpose of reducing variance and making sure that models remain general and overfit less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious case where you'd shuffle your data is if your data is sorted by their class/target. Here, you will want to shuffle to make sure that your training/test/validation sets are representative of the overall distribution of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For batch gradient descent, the same logic applies. The idea behind batch gradient descent is that by calculating the gradient on a single batch, you will usually get a fairly good estimate of the &quot;true&quot; gradient. That way, you save computation time by not having to calculate the &quot;true&quot; gradient over the entire dataset every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. Shuffling your data after each epoch ensures that you will not be &quot;stuck&quot; with too many bad batches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regular stochastic gradient descent, when each batch has size 1, you still want to shuffle your data after each epoch to keep your learning general. Indeed, if data point 17 is always used after data point 16, its own gradient will be biased with whatever updates data point 16 is making on the model. By shuffling your data, you ensure that each data point creates an &quot;independent&quot; change on the model, without being biased by the same points before them.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-11-09T19:13:50.227" LastActivityDate="2017-11-09T19:13:50.227" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24525" PostTypeId="2" ParentId="2395" CreationDate="2017-11-09T12:56:03.700" Score="0" Body="&lt;p&gt;Since your question and the nice answer from @Stephan Kolassa discuss ARIMA and neural networks in particular, i wanted to mention that you can give the &lt;code&gt;forecast&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt; a go - it has a &lt;code&gt;nnetar&lt;/code&gt; function that trains a simple feed forward neural network with 1 hidden layer and lagged inputs.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you could try something along the lines of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;extract many features for each of your observations like day of week, day of month, weekday/weekend etc. (only datetime is mentioned as a potential dependency in your question so that's why i included this - but you can include all possible things that you believe could be influencing your variable of interest).&lt;/li&gt;&#xA;&lt;li&gt;lagged values of your variable of interest as well as the datetime information (like the day of week etc.) would be your inputs. you can include the datetime variables as external regressors (&lt;code&gt;xreg&lt;/code&gt;) for example.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and predict the future values of your var of interest based on these inputs.&#xA;Additionally, you could also think of including the the observed mean and variance/deviation on each given day of the value you want to predict. This would mean that you would have to first forecast your expected mean and variance with e.g. ARIMA then add that as additional input to the approach mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;hth. &lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2017-11-09T12:56:03.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24526" PostTypeId="2" ParentId="24477" CreationDate="2017-11-09T13:01:43.870" Score="0" Body="&lt;p&gt;Having some example data to work with would make it easier for people to help you here, imo. Without that I can suggest looking at &lt;a href=&quot;https://cran.r-project.org/web/packages/plot3D/vignettes/plot3D.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;plot3D&lt;/a&gt; or &lt;a href=&quot;https://cran.r-project.org/web/packages/rgl/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;rgl&lt;/a&gt;. hth.&lt;/p&gt;&#xA;" OwnerUserId="12341" LastActivityDate="2017-11-09T13:01:43.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24527" PostTypeId="2" ParentId="17927" CreationDate="2017-11-09T13:14:39.093" Score="1" Body="&lt;p&gt;I would definitely checkout this question first: &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data&quot;&gt;K-Means clustering for mixed numeric and categorical data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case it doesn't help, here is my explanation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case where you have mixed data types (i.e. numerical and categorical), you have several options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;turn numerical data into categorical data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can do that by using &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_binning&quot; rel=&quot;nofollow noreferrer&quot;&gt;binning&lt;/a&gt;. If you want to use K-Means for categorical data, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Hamming_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;hamming distance&lt;/a&gt; instead of Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;turn categorical data into numerical&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Categorical data can be ordered or not. Let's say that you have 'one', 'two', and 'three' as categorical data. Of course, you could transpose them as 1, 2, and 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in most cases, categorical data cannot be ordered nicely. So you can transform into numerical data by using &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot encoding&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Combine both using K-prototypes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;K-prototypes computes the distance between instances by combining the Euclidean distance between the numerical features and the hamming distance between the categorical features.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2017-11-09T13:23:05.367" LastActivityDate="2017-11-09T13:23:05.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24529" PostTypeId="2" ParentId="6899" CreationDate="2017-11-09T15:09:12.577" Score="1" Body="&lt;p&gt;If I understood the question correctly, you have trained an algorithm that splits your data into $N$ disjoint clusters. Now you want to assign prediction $1$ to some subset of the clusters, and $0$ to the rest of them. And amont those subsets, you want to find the pareto-optimal ones, i.e. those who maximize true positive rate given fixed number of positive predictions (this is equivalent to fixing PPV). Is it correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sounds very much like &lt;a href=&quot;http://en.wikipedia.org/wiki/Knapsack_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;knapsack problem&lt;/a&gt;! Cluster sizes are &quot;weights&quot; and number of positive samples in a cluster are &quot;values&quot;, and you want to fill your knapsack of fixed capacity with as much value as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The knapsack problem has several algorihms for finding exact solutions (e.g. by dynamic programming). But a useful &lt;a href=&quot;http://en.wikipedia.org/wiki/Knapsack_problem#Greedy_approximation_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;greedy solution&lt;/a&gt; is to sort your clusters in decreasing order of $\frac{value}{weight}$ (that is, share of positive samples), and take the first $k$. If you take $k$ from $0$ to $N$, you can very cheaply sketch your ROC curve. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you assign $1$ to the first $k-1$ clusters and to the random fraction $p\in[0,1]$ of samples in the $k$th cluster, you get the &lt;a href=&quot;https://math.stackexchange.com/questions/1267331/0-1-knapsack-upper-bound&quot;&gt;upper bound&lt;/a&gt; to the knapsack problem. With this, you can draw the upper bound for your ROC curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here goes a python example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from itertools import combinations, chain&#xA;import matplotlib.pyplot as plt&#xA;np.random.seed(1)&#xA;n_obs = 1000&#xA;n = 10&#xA;&#xA;# generate clusters as indices of tree leaves&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import cross_val_predict&#xA;X, target = make_classification(n_samples=n_obs)&#xA;raw_clusters = DecisionTreeClassifier(max_leaf_nodes=n).fit(X, target).apply(X)&#xA;recoding = {x:i for i, x in enumerate(np.unique(raw_clusters))}&#xA;clusters = np.array([recoding[x] for x in raw_clusters])&#xA;&#xA;def powerset(xs):&#xA;    &quot;&quot;&quot; Get set of all subsets &quot;&quot;&quot;&#xA;    return chain.from_iterable(combinations(xs,n) for n in range(len(xs)+1))&#xA;&#xA;def subset_to_metrics(subset, clusters, target):&#xA;    &quot;&quot;&quot; Calculate TPR and FPR for a subset of clusters &quot;&quot;&quot;&#xA;    prediction = np.zeros(n_obs)&#xA;    prediction[np.isin(clusters, subset)] = 1&#xA;    tpr = sum(target*prediction) / sum(target) if sum(target) &amp;gt; 0 else 1&#xA;    fpr = sum((1-target)*prediction) / sum(1-target) if sum(1-target) &amp;gt; 0 else 1&#xA;    return fpr, tpr&#xA;&#xA;# evaluate all subsets&#xA;all_tpr = []&#xA;all_fpr = []&#xA;for subset in powerset(range(n)):&#xA;    tpr, fpr = subset_to_metrics(subset, clusters, target)&#xA;    all_tpr.append(tpr)&#xA;    all_fpr.append(fpr)&#xA;&#xA;# evaluate only the upper bound, using knapsack greedy solution&#xA;ratios = [target[clusters==i].mean() for i in range(n)]&#xA;order = np.argsort(ratios)[::-1]&#xA;new_tpr = []&#xA;new_fpr = []&#xA;for i in range(n):&#xA;    subset = order[0:(i+1)]&#xA;    tpr, fpr = subset_to_metrics(subset, clusters, target)&#xA;    new_tpr.append(tpr)&#xA;    new_fpr.append(fpr)&#xA;&#xA;plt.figure(figsize=(5,5))&#xA;plt.scatter(all_tpr, all_fpr, s=3)&#xA;plt.plot(new_tpr, new_fpr, c='red', lw=1)&#xA;plt.xlabel('TPR')&#xA;plt.ylabel('FPR')&#xA;plt.title('All and Pareto-optimal subsets')&#xA;plt.show();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code will draw a nice picture for you:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YrzAA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YrzAA.png&quot; alt=&quot;TPR, FPR, and optimal curve&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blue dots are (FPR, TPR) tuples for all $2^{10}$ subsets, and the red line connects (FPR, TPR) for the pareto-optimal subsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And now the bit of salt: &lt;em&gt;you did not have to bother about subsets at all&lt;/em&gt;! What I did is sorted tree leaves by the fraction of positive samples in each. But what I got is exactly the ROC curve for the probabilistic prediction of the tree. This means, you cannot outperform the tree by hand-picking its leaves based on the target frequencies in the training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can relax and keep using ordinary probabilistic prediction :) &lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-11-09T16:10:35.763" LastActivityDate="2017-11-09T16:10:35.763" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24530" PostTypeId="1" CreationDate="2017-11-09T15:26:51.883" Score="1" ViewCount="88" Body="&lt;p&gt;Say I have a model that I'm using to forecast demand for some product.&#xA;I can train it on some training data in order to get good predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, the predictions it outputs will only really match past demand. This seems to be a problem related to the data that we give the model because if the past demand data is only how much we sold, but that was our supply as well, then this wouldn't be able to help improve future predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I had in supply 10 units of product x, and I always sell 10 units of that product, then based on this training data, my model will always say that I should always order 10 units for every day. But, really, it might be better to order 25 units because if I'm always selling everything, then there might be more demand than supply. The problem is that I don't know what the true demand is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is, how would I get my model to account for that and actually improve predictions rather than trying to fit to historical data.&lt;/p&gt;&#xA;" OwnerUserId="37946" LastEditorUserId="37946" LastEditDate="2017-11-09T23:22:45.557" LastActivityDate="2017-11-09T23:22:45.557" Title="How can I get a forecasting model to improve its forecasts over time instead of fitting to training data?" Tags="|machine-learning|time-series|forecast|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24533" PostTypeId="2" ParentId="23437" CreationDate="2017-11-09T16:40:00.450" Score="5" Body="&lt;p&gt;A general solution to this does not exist, even if we add some assumptions about the distribution of &lt;em&gt;e.g.&lt;/em&gt; colours and shapes in the images or temporal coupling such as consecutive frames being similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $F_1,\dots,F_i$ be the $n$ original frames, each with $m$ pixels. Let $P$ be the permutation that is applied to the pixels of each frame before we get them. You can think of $P$ as the enemy's code-book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, as input we are receiving $P(F_1),\dots,P(F_n)$. The goal is to find the inverse permutation $Q$ to restore the images. Thus $QP=I$ is the identity map and for example $Q(P(F_1))=F_1$. Note that we do not know any of the correct frames $F_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $Q_1,...,Q_{m!}$ be the $m!$ possible permutation functions of the $m$ pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to select the unique $j\in\{1,\dots,m!\}$ so that $Q_jP=I$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No General Solution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under our statistical model this means selecting the $Q_j$ which maximises the likelihood that $Q_j(P(F_i))$ is drawn from the same distribution as the reference statistics for images and the temporal statistics between consecutive frames $Q_j(P(F_{i})$ and $Q_j(P(F_{i+1})$ which is our prior knowledge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a canonical &lt;strong&gt;counter-example&lt;/strong&gt; where the enemy gives you a scrambled movie with two frames where all the pixels are the same colour, so $n=2$, $F_1=F_2$ and $Q_j(F_1)=Q_j(F_2)=F1=F2$ for every $j$. Thus, for all $j$, the in-frame and inter-frame statistics are equiprobable for each $j$ and give us no information to select the maximum likelihood permutation $Q_j$ (except in the degenerate case where $m!=1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, we cannot guarantee uniqueness and the problem is unsolvable without further assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further Assumptions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is interesting to see if we can solve the problem by adding more constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we restrict the enemy to only sending us &quot;real&quot; movies and assuming there are enough different pixels and frames to so that a unique $Q_j$ with maximum likelihood exists, we would still have to calculate statistics for $O(m! \times n)$ permuted frames to find the maximum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is brute force code-breaking.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to benefit from neural networks, and back-propagation in particular, we would need a differentiable loss function with respect to the input (which is an encoding of $j$ or our permutation $Q_j$). The question then, would be to see if such a function can be found.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise the problem is more similar to cryptanalysis in the special case where we know that the enemy's code book is a permutation of the clear-text (or clear-image).&lt;/p&gt;&#xA;" OwnerUserId="1424" LastActivityDate="2017-11-09T16:40:00.450" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24534" PostTypeId="1" AcceptedAnswerId="24537" CreationDate="2017-11-09T16:41:20.940" Score="47" ViewCount="68267" Body="&lt;p&gt;I am wondering whether there is any scenario in which gradient descent does not converge to a minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am also aware that it might diverge from an optimum if, say, the step size is too big. However, it seems to me that, if it diverges from some optimum, then it will eventually go to another optimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, gradient descent would be guaranteed to converge to a local or global optimum. Is that right? If not, could you please provide a rough counterexample?&lt;/p&gt;&#xA;" OwnerUserId="41658" LastEditorUserId="28175" LastEditDate="2018-01-10T16:55:25.190" LastActivityDate="2021-03-30T10:05:40.617" Title="Does gradient descent always converge to an optimum?" Tags="|machine-learning|neural-network|deep-learning|optimization|gradient-descent|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24535" PostTypeId="2" ParentId="24530" CreationDate="2017-11-09T17:29:28.997" Score="1" Body="&lt;p&gt;Like you stated, you get good predictions based on the information in the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model conflates demand and supply in the case of under-supply, since it has no information about unfulfilled demand: how many products were requested but not sold due to lack of supply. Instead it tracks $sold = min(demand,supply)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extend the Model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious solution is to have a separate feature for the true demand rather than the conflating it with unit sold. However, this data may not be available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Learn from Correlations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to use richer model: if you have multiple products where most of them don't sell out (&lt;em&gt;i.e.&lt;/em&gt; where $sold$ reflects the true demand) and their sales are correlated, then it can predict that a particular product needs to be resupplied at a higher rate than the observed sales. This prediction would be based on the correlations with products that don't hit the $min(demand,supply)$-cap and .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reinforcement Learning&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, another approach is to look to Reinforcement Learning where the agent learns the best resupplying strategy by experimenting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a simple setup, your agent will select between two actions: one that orders according to its prediction in the normal case and another action that orders more than predicted. In the case where the prediction matches actual demand the probability of the over-ordering action will be quite low, but it will still be selected with some small probability $\epsilon$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reinforcement Learning allows the agent to learn the relative probability of the two, and based on the cases where the actual demand was higher than what you had in stock, it will slowly learn to order more with a higher probability until the prediction based on past sales increases to a point where there is no extra sales to gain from ordering more and the probability of taking the over-ordering action starts to fall again.&lt;/p&gt;&#xA;" OwnerUserId="1424" LastActivityDate="2017-11-09T17:29:28.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24536" PostTypeId="2" ParentId="24534" CreationDate="2017-11-09T17:36:02.693" Score="21" Body="&lt;p&gt;Asides from the points you mentioned (convergence to non-global minimums, and large step sizes possibly leading to non-convergent algorithms), &quot;inflection ranges&quot; might be a problem too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following &quot;recliner chair&quot; type of function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/moXD4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/moXD4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, this can be constructed so that there is a range in the middle where the gradient is the 0 vector. In this range, the algorithm can be stuck indefinitely. &lt;a href=&quot;http://mathworld.wolfram.com/InflectionPoint.html&quot; rel=&quot;noreferrer&quot;&gt;Inflection points&lt;/a&gt; are usually not considered local extrema.&lt;/p&gt;&#xA;" OwnerUserId="9518" LastActivityDate="2017-11-09T17:36:02.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24537" PostTypeId="2" ParentId="24534" CreationDate="2017-11-09T17:56:45.890" Score="46" Body="&lt;p&gt;Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much. The reason is that the step size might be too large that prompts it recede one optimal point and the probability that it oscillates is much more than convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About gradient descent there are two main perspectives, machine learning era and deep learning era. During machine learning era it was considered that gradient descent will find the local/global optimum but in deep learning era where the dimension of input features are too much it is shown in practice that the probability that all of the features be located in there optimal value at a single point is not too much and rather seeing to have optimal locations in cost functions, most of the time saddle points are observed. This is one of the reasons that training with lots of data and training epochs cause the deep learning models outperform other algorithms. So if you train your model, it will find a detour or will find its way to go downhill and do not stuck in saddle points, but you have to have appropriate step sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more intuitions I suggest you referring &lt;a href=&quot;https://datascience.stackexchange.com/q/18802/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/RFANA/the-problem-of-local-optima&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-09T18:07:08.823" LastActivityDate="2017-11-09T18:07:08.823" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24538" PostTypeId="2" ParentId="23437" CreationDate="2017-11-09T18:13:32.440" Score="4" Body="&lt;p&gt;This is a fascinating combinatorial problem. I would featuring each pixel using its full temporal trajectory, then embed them in a grid using the k nearest neighbors. The real goal is to maximize the likelihood of the video being a sequence of natural (real life) images, which you can test with a classifier, but you might be able to get away with just a smoothness cost; say, the sum of differences between adjacent pixels. Once you have started filling in the grid, smoothness constraints will reduce the search space (since a pixel will have to be close to multiple other pixels), thus speeding things up, assuming you are using an efficient data structure for querying the nearest neighbors; see for example &lt;a href=&quot;http://www.itu.dk/people/pagh/SSS/ann-benchmarks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.itu.dk/people/pagh/SSS/ann-benchmarks/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-11-09T18:27:43.000" LastActivityDate="2017-11-09T18:27:43.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24539" PostTypeId="2" ParentId="24511" CreationDate="2017-11-09T19:51:20.033" Score="49" Body="&lt;p&gt;Based on &lt;a href=&quot;https://datascience.meta.stackexchange.com/q/2258/41663&quot;&gt;What should we do when a question posted on DataScience is a duplicate of a question posted on CrossValidated?&lt;/a&gt;, I am reposting my answer to the same question asked on CrossValidated (&lt;a href=&quot;https://stats.stackexchange.com/a/311318/89653&quot;&gt;https://stats.stackexchange.com/a/311318/89653&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note: throughout this answer I refer to minimization of training loss and I do not discuss stopping criteria such as validation loss. The choice of stopping criteria does not affect the process/concepts described below.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process of training a neural network is to find the minimum value of a loss function $ℒ_X(W)$, where $W$ represents a matrix (or several matrices) of weights between neurons and $X$ represents the training dataset. I use a subscript for $X$ to indicate that our minimization of $ℒ$ occurs only over the weights $W$ (that is, we are looking for $W$ such that $ℒ$ is minimized) while $X$ is fixed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if we assume that we have $P$ elements in $W$ (that is, there are $P$ weights in the network), $ℒ$ is a surface in a $P+1$-dimensional space. To give a visual analogue, imagine that we have only two neuron weights ($P=2$). Then $ℒ$ has an easy geometric interpretation: it is a surface in a 3-dimensional space. This arises from the fact that for any given matrices of weights $W$, the loss function can be evaluated on $X$ and that value becomes the elevation of the surface.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is the problem of non-convexity; the surface I described will have numerous local minima, and therefore gradient descent algorithms are susceptible to becoming &quot;stuck&quot; in those minima while a deeper/lower/better solution may lie nearby. This is likely to occur if $X$ is unchanged over all training iterations, because the surface is fixed for a given $X$; all its features are static, including its various minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A solution to this is mini-batch training combined with shuffling. By shuffling the rows and training on only a subset of them during a given iteration, $X$ changes with &lt;em&gt;every&lt;/em&gt; iteration, and it is actually quite possible that no two iterations over the entire sequence of training iterations and epochs will be performed on the exact same $X$. The effect is that the solver can easily &quot;bounce&quot; out of a local minimum. Imagine that the solver is stuck in a local minimum at iteration $i$ with training mini-batch $X_i$. This local minimum corresponds to $ℒ$ evaluated at a particular value of weights; we'll call it $ℒ_{X_i}(W_i)$. On the next iteration the shape of our loss surface actually changes because we are using $X_{i+1}$, that is, $ℒ_{X_{i+1}}(W_i)$ may take on a very different value from $ℒ_{X_i}(W_i)$ and it is quite possible that it does not correspond to a local minimum! We can now compute a gradient update and continue with training. To be clear: the shape of $ℒ_{X_{i+1}}$ will -- in general -- be different from that of $ℒ_{X_{i}}$. Note that here I am referring to the loss function $ℒ$ evaluated on a training set $X$; it is a complete surface defined over all possible values of $W$, rather than the evaluation of that loss (which is just a scalar) for a specific value of $W$. Note also that if mini-batches are used without shuffling there is still a degree of &quot;diversification&quot; of loss surfaces, but there will be a finite (and relatively small) number of unique error surfaces seen by the solver (specifically, it will see the same exact set of mini-batches -- and therefore loss surfaces -- during each epoch).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing I deliberately avoided was a discussion of mini-batch sizes, because there are a million opinions on this and it has significant practical implications (greater parallelization can be achieved with larger batches). However, I believe the following is worth mentioning. Because $ℒ$ is evaluated by computing a value for each row of $X$ (and summing or taking the average; i.e., a commutative operator) for a given set of weight matrices $W$, the arrangement of the rows of $X$ &lt;em&gt;has no effect&lt;/em&gt; when using full-batch gradient descent (that is, when each batch is the full $X$, and iterations and epochs are the same thing).&lt;/p&gt;&#xA;" OwnerUserId="41663" LastActivityDate="2017-11-09T19:51:20.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24543" PostTypeId="1" AcceptedAnswerId="24581" CreationDate="2017-11-09T21:39:15.143" Score="5" ViewCount="865" Body="&lt;p&gt;I have a set of about ~100,000 training examples. Ratio of positive to negative example is roughly 1:2. The true ratio is more like 1:100 so this represents a major downsampling of the negative class. It is also a very noisy dataset - the examples were automatically generated through distant supervision. Each example represents a set of sentences and has 700 columns. Number of rows may vary from 10 to 100 (maybe even more).&lt;/p&gt;&#xA;&lt;p&gt;I used a Convolution Neural Network in Tensorflow to train my model (model architecture similar to the one described &lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)  with only 2 epochs and stored the loss, f-score , precision and recall every 10 steps. I evaluated the model on a validation set (which too was generated automatically through distant supervision with negative class downsampling resulting in pos:neg ratio of ~1:2) every 100 steps. Here are the hyperparameters:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;batch size: 60 for train, 100 for validation &#xA;epochs: 2&#xA;convolution filter sizes: 700x5, 700x6, 700x7, 700x8, 700x9, 700x10 &#xA;number of convolution filters per filter size: 125 (so total of 750 filters)&#xA;dropout: 0.5&#xA;l2reg: 0.001&#xA;lr: 0.001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I'm seeing some strange behavior with the model and I don't understand why.&lt;/p&gt;&#xA;&lt;p&gt;My training precision, recall and f-score go over 0.95 in about a 100 steps (6000 examples) and then plateaus. The loss falls down from 0.8 to 0.2 in about 200 steps and then fluctuates between 0.1 and 0.4.&lt;/p&gt;&#xA;&lt;p&gt;On the validation set my precision, recall and f-score are over 0.95 starting from the first time I evaluate it on the 100th step. Loss fall slightly from 0.3 to 0.2.&lt;/p&gt;&#xA;&lt;p&gt;When I evaluated on a real-world test set (without downsampling negative class so it has the true ratio of pos:neg), the actual precision and recall were 0.37 and 0.85.&lt;/p&gt;&#xA;&lt;p&gt;My results are not making any sense to me. I use tensorflow metrics for calculating training precision, recall and fscore and scikit-learn metrics for calculation validation precision, recall and fscore. I can't find anything wrong in the code but I don't understand why I should have such results unless there is a bug. I would have understood having low precision and recall all through - the class imbalance favors the negative class and my set is noisy. However, I am very confused about why I'm having such misleadingly high scores all through..&lt;/p&gt;&#xA;&lt;p&gt;Given that my dev dataset is also noisy and generated in the same manner as the train set, the dev results might just be useless and it is possible that the model is overfitting the noisy set. But I still don't understand why the scores are so high so soon. Also, if overfitting is the issue, do you think I should make the dropout even higher?&lt;/p&gt;&#xA;&lt;p&gt;I've attached a screenshot of the graphs and would really appreciate your thoughts on this. Blue is train and red is dev. Thanks a lot!&#xA;&lt;a href=&quot;https://i.stack.imgur.com/PGmND.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PGmND.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41672" LastEditorUserId="29169" LastEditDate="2021-02-15T22:21:10.077" LastActivityDate="2021-02-15T22:21:10.077" Title="Convolution Neural Network Loss and performance" Tags="|machine-learning|deep-learning|neural-network|tensorflow|convolutional-neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="24544" PostTypeId="2" ParentId="12883" CreationDate="2017-11-09T22:37:35.533" Score="1" Body="&lt;p&gt;as already answered &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data/24#24&quot;&gt;here&lt;/a&gt;, k-means in its original way won't be very effective, as Euclidean distances won't do the job with categorical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some extensions exist (e.g. k-modes) or modifications with other distances (e.g. Gower). The discussion is expanded &lt;a href=&quot;https://www.kaggle.com/general/19741&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have a straight answer, but I suggest to look to the different possibilities already implemented and consider pros and cons of each one.&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/unsupervised_learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here you can find a lot of different methodologies&lt;/a&gt; that maybe can be adapted to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps&lt;/p&gt;&#xA;" OwnerUserId="21406" LastActivityDate="2017-11-09T22:37:35.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24545" PostTypeId="2" ParentId="24500" CreationDate="2017-11-09T22:43:43.397" Score="1" Body="&lt;h3&gt;Scaling&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Standardize you data by $(x - mean(x)) / std(x)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most K-mean implementation by default uses Euclidean distance which assumes the equal importance of all features. This requires proper scaling to prevent one action dominates the others. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Dimensionality&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;K-mean is not robust against the curse of dimensionality (see this &lt;a href=&quot;https://stats.stackexchange.com/questions/93488/usefulness-of-k-means-clustering-on-high-dimensional-data&quot;&gt;post&lt;/a&gt;). So as always it good to reduce you dimensionality before feed it to the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do some feature engineering first. For example, login and page load can be group together as a measure of passive engagement, page edit and page created could be considered as a single activate engagement feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you can try using some standard dimensionality reduction algorithm like &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-09T22:43:43.397" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24546" PostTypeId="2" ParentId="18042" CreationDate="2017-11-09T22:54:14.610" Score="0" Body="&lt;p&gt;Here is what I would do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;idw_output &amp;lt;- as.data.frame(idw)[, 1:3]&#xA;&#xA;# Adding the IDW raster&#xA;# The first step to incorporate the IDW results into the interactive map is to turn it into a spatial object&#xA;idw_raster &amp;lt;- rasterFromXYZ(idw_output)&#xA;&#xA;# We only want to show the interpolation results within the area of which we have data points&#xA;# To do this, we will clip the raster to data region by creating convex hull, buffer, and mask (clip) raster&#xA;# Perform convex hull and buffer operations on projected UTM data&#xA;xyz_hull &amp;lt;- gConvexHull(xyz)&#xA;xyz_buff &amp;lt;- gBuffer(xyz_hull, width = 25000) # arbitrary 1km buffer&#xA;&#xA;# Mask IDW raster to confine to regions that we have information for&#xA;idw_raster_crop &amp;lt;- mask(idw_raster, xyz)&#xA;&#xA;spplot(idw_raster_crop)&#xA;&#xA;# Create contours of the IDW&#xA;idw_contour &amp;lt;- rasterToContour(idw_raster_crop, nlevels = 15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you use your real data, you can use mapview, ie &lt;code&gt;mapview(idw_raster_crop)&lt;/code&gt; to see the interactive map. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cite: &lt;a href=&quot;https://rstudio-pubs-static.s3.amazonaws.com/212793_f130ecc723da4ed98680d6d3d5c4aff9.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://rstudio-pubs-static.s3.amazonaws.com/212793_f130ecc723da4ed98680d6d3d5c4aff9.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41675" LastEditorUserId="41675" LastEditDate="2017-11-10T16:18:22.840" LastActivityDate="2017-11-10T16:18:22.840" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24547" PostTypeId="2" ParentId="24474" CreationDate="2017-11-09T22:56:17.157" Score="2" Body="&lt;p&gt;CIFAR10 has 10 class label. So by random guessing, you should achieve an accuracy of 10%. And this is what you are getting. This means your algorithm is not learning at all. The most common problem causes this is your learning rate. Reduce your learning rate by replacing your line,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optimizer = keras.optimizers.Adam(lr=1e-4)&#xA;model.compile(loss='squared_hinge',optimizer=optimizer,metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I just did a test run, it is learning slowly. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-09T22:56:17.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24548" PostTypeId="2" ParentId="24330" CreationDate="2017-11-10T02:02:55.897" Score="0" Body="&lt;p&gt;In line 809, you can find that &lt;code&gt;SimpleRNNCell&lt;/code&gt; takes a &lt;code&gt;bias_initializer&lt;/code&gt; as an argument to the constructor function and it defaults to &lt;code&gt;zeros&lt;/code&gt;.   &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-10T02:02:55.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24549" PostTypeId="1" AcceptedAnswerId="24553" CreationDate="2017-11-10T04:04:27.133" Score="1" ViewCount="1494" Body="&lt;p&gt;I'm doing a Naive Bayes prediction model where I've 666 features to select from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried the SelectKBest chi2 test but it's a features-label test whereas what I'm trying to do is to find out the independency and also correlation feature-features. As the assumption to use Naive Bayes is that variables are all independent to each other, I have to remove features that are dependent/correlated to each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the other independence tests that I should look into apart from $chi^2$? (I have continuous features) &lt;/p&gt;&#xA;" OwnerUserId="41220" LastEditorUserId="29575" LastEditDate="2017-11-10T04:19:19.450" LastActivityDate="2017-11-10T09:29:14.253" Title="Independence/correlation test between features (not feautre-label)" Tags="|machine-learning|feature-selection|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24550" PostTypeId="1" CreationDate="2017-11-10T04:09:28.927" Score="1" ViewCount="30" Body="&lt;p&gt;I have a set of 1000 images of the dorsal side of the hand. I need to identify the different joints in the fingers and measure the distance between them. I have already cropped the images to same dimensions, centered the hand, and labeled the required points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone point me to the next steps and a general direction I should be looking at for training the machine I need? Which algorithms would be the closest for my need?&lt;/p&gt;&#xA;" OwnerUserId="41684" LastEditorUserId="29575" LastEditDate="2017-11-10T11:33:20.693" LastActivityDate="2017-11-10T11:33:20.693" Title="Training machine to identify hot spots" Tags="|machine-learning|deep-learning|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24551" PostTypeId="2" ParentId="24550" CreationDate="2017-11-10T04:22:01.937" Score="1" Body="&lt;p&gt;Convolution Neural Network (CNN) is the state of art algorithm for your problem. The standard terminology for this kind of problem is &quot;feature detection&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a good resource for the basics. &lt;a href=&quot;http://cs231n.stanford.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.stanford.edu/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your biggest problem probably is going to be the size of your training images. 1,000 is most likely not going to be enough. You should try to augment your data by rotating and translation etc.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-10T04:28:39.043" LastActivityDate="2017-11-10T04:28:39.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24552" PostTypeId="1" AcceptedAnswerId="24563" CreationDate="2017-11-10T04:32:27.397" Score="3" ViewCount="302" Body="&lt;p&gt;I've been learning about &lt;a href=&quot;https://en.wikipedia.org/wiki/Scale-invariant_feature_transform&quot; rel=&quot;nofollow noreferrer&quot;&gt;SIFT&lt;/a&gt; and all the ways its descriptors can be used to do different tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am particularly interested in the way SIFT can be used for image classification. (e.g. A 2006 paper by Niester &amp;amp; Stewenius relies on SIFT descriptors to build a vocabulary tree).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as of 2017, 11 years later, Deep Learning has been replacing the classical approaches in many ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some alternatives to find feature descriptors for images that, as of 2017, have shown more promising results?&lt;/p&gt;&#xA;" OwnerUserId="40075" LastEditorUserId="29575" LastEditDate="2017-11-10T07:06:51.947" LastActivityDate="2021-03-10T17:39:43.807" Title="Contemporary alternatives to SIFT for image feature extraction?" Tags="|classification|image-classification|image-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24553" PostTypeId="2" ParentId="24549" CreationDate="2017-11-10T05:30:14.197" Score="1" Body="&lt;p&gt;This &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;package&lt;/a&gt; is very useful, this does the feature engg. for you, gives you the important variables. sample code would be something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    set.seed(123)&#xA;    boruta.input &amp;lt;- Boruta(target variable~., data = training_data, doTrace = 2)&#xA;    print(boruta.input)&#xA;&#xA;    #plot a graph for better understanding&#xA;    plot(boruta.input, xlab = &quot;&quot;, xaxt = &quot;n&quot;)&#xA;    lz&amp;lt;-lapply(1:ncol(boruta.input$ImpHistory),function(i)&#xA;    boruta.input$ImpHistory[is.finite(boruta.input$ImpHistory[,i]),i])&#xA;    names(lz) &amp;lt;- colnames(boruta.input$ImpHistory)&#xA;    Labels &amp;lt;- sort(sapply(lz,median))&#xA;    axis(side = 1,las=2,labels = names(Labels),&#xA;    at = 1:ncol(boruta.input$ImpHistory), cex.axis = 0.7)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By using this you can get all the important features but the downfall is it takes time if you have more data&#xA;In my case my data consists 40 features and 200,000 records it took almost 2 hours but the results were good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For better understanding you can go through this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;--&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I think you can use this &lt;a href=&quot;https://cran.r-project.org/web/packages/IndependenceTests/IndependenceTests.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;test too&lt;/a&gt;, in the description it tells that it can be applied on numerical data too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have data which is Nominal data, you can use &lt;a href=&quot;https://rcompanion.org/rcompanion/b_06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;G-test&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attaching one more link along with this, which consists of tests for different kinds of features. Might be helpful in the future &lt;a href=&quot;https://stats.idre.ucla.edu/other/mult-pkg/whatstat/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a question, why aren't you applying PCA(or any DR Techniques) on the data-set? (assuming that you din't try any DR Techniques)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you apply DR Techniques, I think you might get better Components which would aid you in getting better results/better understanding of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you mentioned that you were using NB Classifier, why did you choose that, is there any specific reason?&#xA;As you know that there are many other Classifiers which might outrun NB Classifier.&#xA;In my case I used NB Classifier for twitter sentiment mining, as those are short sentences and NB Classifier best suited for my analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;NB: Naive Bayes&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;DR: Dimensionality Reduction&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-10T09:29:14.253" LastActivityDate="2017-11-10T09:29:14.253" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24557" PostTypeId="1" CreationDate="2017-11-10T08:37:45.310" Score="-1" ViewCount="2364" Body="&lt;p&gt;I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to the document. I need an approach to get to this, for relevancy I will also like to consider the indirect occurrence of entities Eg - In the document after first occurrence of the entity (by its name) it may be referred using terms like it, this, that ..etc. I tried frequency based approaches but not getting proper results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following is the example &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input text&lt;/strong&gt; &quot;In 2009, Jack Sparrow worked on NLP Api to process the written word, with all of its quirks and nuances, and got immediate traction. That first month, the company's eponymous language-analysis API processed 500,000 transactions. “Context is super-important,” he adds. “'I'm dying' is a lot different than 'I'm dying to buy the new iPhone.'” “As we move into new markets, we're going to be making some new hires,&quot; Jack says. &quot;We knocked down some walls and added 2,000 square feet to our office.” Clients include Walmart, PR Newswire and numerous publishers and advertising networks. “This allows a news organization to detect what a person likes to read about,” says Sparrow of publishers and advertisers..&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Representational Entities and their scores&lt;/strong&gt; – &#xA;Jack Sparrow   -   0.906712, &#xA;PR Newswire - 0.292422, &#xA;iPhone - 0.189069&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated.. &lt;/p&gt;&#xA;" OwnerUserId="41688" LastEditorUserId="41688" LastEditDate="2017-11-10T14:42:54.540" LastActivityDate="2018-04-09T22:34:51.340" Title="How to get relevancy score of a term with respect to text/document" Tags="|nlp|similar-documents|tfidf|score|" AnswerCount="1" CommentCount="6" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24559" PostTypeId="2" ParentId="24552" CreationDate="2017-11-10T09:20:26.480" Score="1" Body="&lt;p&gt;Depending on what you need, you should look at:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Fully convolutional networks (FCN) to obtain the mask of the input image&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ResNets for very deep networks(101 layers)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;MaskRCNN for instance segmentation&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="29169" LastEditDate="2021-03-10T17:39:43.807" LastActivityDate="2021-03-10T17:39:43.807" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24560" PostTypeId="1" CreationDate="2017-11-10T10:07:25.933" Score="5" ViewCount="10638" Body="&lt;p&gt;&lt;code&gt;unsup_df&lt;/code&gt; is a &lt;code&gt;DataFrame&lt;/code&gt; which has only one column: &lt;em&gt;review&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to form 2 clusters of the &lt;em&gt;reviews&lt;/em&gt;. One &lt;strong&gt;positive&lt;/strong&gt; and one &lt;strong&gt;negative&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import KMeans&#xA;&#xA;tfidf_vectorizer = TfidfVectorizer()  &#xA;tfidf_matrix = tfidf_vectorizer.fit_transform(unsup_df)  &#xA;num_clusters = 2  &#xA;km = KMeans(n_clusters=num_clusters)  &#xA;km.fit(tfidf_matrix)   &#xA;clusters = km.labels_.tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above piece of code is throwing an error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: n_samples=1 should be &gt;= n_clusters=2&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;on the line &lt;code&gt;km.fit(tfidf_matrix)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="41419" LastEditorUserId="54395" LastEditDate="2018-08-08T11:23:38.457" LastActivityDate="2022-04-10T13:22:28.833" Title="K-means clustering on the data frame having only one column" Tags="|machine-learning|scikit-learn|pandas|k-means|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24562" PostTypeId="2" ParentId="24560" CreationDate="2017-11-10T10:49:53.913" Score="0" Body="&lt;p&gt;Your &lt;code&gt;unsup_df&lt;/code&gt; must be in the wrong shape. Otherwise, it should work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FkLJf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FkLJf.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-10T10:49:53.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24563" PostTypeId="2" ParentId="24552" CreationDate="2017-11-10T11:39:26.543" Score="3" Body="&lt;p&gt;The &lt;strong&gt;Capsule Networks&lt;/strong&gt; by Geoff Hinton &lt;em&gt;et al.&lt;/em&gt; are based on a similar idea to SIFT: in fact, if you look at the 2011 paper, &lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-642-21735-7_6&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Transforming Auto-Encoders&quot;&gt;Transforming Auto-Encoders&lt;/a&gt; by Hinton &lt;em&gt;et al.&lt;/em&gt; they explicitly cite SIFT as an inspiration for Capsules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main idea is that they construct a network with a new type of unit called a Capsule which outputs a vector (rather than neurons outputting scalars, as we know in traditional neural networks).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of these vectors as an analogue to the SIFT keypoints and feature descriptors, but with the benefit that they can be learned via back-propagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a recent article, &lt;a href=&quot;https://arxiv.org/abs/1710.09829&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Dynamic Routing Between Capsules&quot;&gt;Dynamic Routing Between Capsules by Sabour, Frosst and Hinton&lt;/a&gt; elaborating on this which shows very promising results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have written a short summary of Capsule Networks on the AI Stack Exchange, see the question &lt;a href=&quot;https://ai.stackexchange.com/questions/4414/whats-the-main-concept-behind-capsule-networks&quot;&gt;What's the main concept behind Capsule Networks?&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1424" LastActivityDate="2017-11-10T11:39:26.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24564" PostTypeId="1" CreationDate="2017-11-10T13:05:26.780" Score="1" ViewCount="126" Body="&lt;p&gt;For my current use case, I have a high number of (noisy) samples that I do binary classification on. I use a neural network to approximate this with a sigmoid layer as output and log loss as loss function. My question is about the loss function, I understand that we use log loss to penalize probabilities that are far from the truth because of the log likelihood. The problem (or maybe it is not a problem) is that I want to aggregate my results to get the expected amount of positive samples in big groups. If we look at it from that perspective, looking at it as a mean as opposed to a probability also makes sense. In that case, having 2x 0.4 and 1x 0.7 is the same mistake as 1x 0. and 2x 0.75. Does it still make sense to use the log loss or might it be better to use L2 or L1 regression instead?&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-10T13:05:26.780" Title="Log loss and expected aggregates" Tags="|classification|regression|loss-function|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24565" PostTypeId="1" AcceptedAnswerId="24566" CreationDate="2017-11-10T13:48:46.220" Score="-1" ViewCount="37" Body="&lt;p&gt;I am starting to learn about machine learning as a whole and have found a big interest in Neural Networks. My question today is, &#xA;Given a &lt;strong&gt;simple&lt;/strong&gt; cost function $J$ with weights $w1, w2$ and inputs $m1, m2$ and a target prediction/error $b$ so that $J = w1*m1 + w2*m2 + b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say we want to develop an algorithm that can detect by the heights and length of the flower pedal which color the flower was. To save time I have already seen clustering in the graph and the red flowers which are close to 1 while the blue are close to 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most cost function outputs go from 0 to 1 (I think due to the sigmoid function), 1 being yes or some desired output = red and 0 being no or another desired output = blue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, what if instead of looking at two different colors (red and blue) what if our Machine wants to analyze 10 colors given the heights and length of the pedals? Would we say that if the output of the cost function was 2.1 (2 being green) it would be green for example or if its 5.2 (5 being yellow) its yellow? &#xA;Not sure how to analyze more than two things if we can only go from 0 to 1 due to the sigmoid function. &lt;/p&gt;&#xA;" OwnerUserId="41036" LastEditorUserId="29575" LastEditDate="2017-11-14T08:14:14.263" LastActivityDate="2017-11-14T08:14:14.263" Title="Neural network predicted value" Tags="|machine-learning|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24566" PostTypeId="2" ParentId="24565" CreationDate="2017-11-10T13:54:16.150" Score="2" Body="&lt;p&gt;With binary classification (your first example) you use a sigmoid function at the end, where the value indicates the probability P(Red), and 1-value equals the probability of P(Blue). Because there are only two classes we can use one value because together they sum to one. By using binary crossentropy loss we can fit our model. When you have to deal with more than two classes having one value is not enough anymore. What you do then is to have &lt;strong&gt;n&lt;/strong&gt; output values that are mapped to a probability distribution using the softmax. This comes down to taking the e-power of all of them and then normalizing them to sum up to 1. By using the multiclass crossentropy loss which is the generalized version of the binary crossentropy loss we can still estimate this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Where with binary classification we map using the sigmoid:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Red) = \frac{1}{1+e^{-Wx}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have a probability for all colors individually:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(Color = y) = \frac{e^{W_yx}}{\sum_{c\in C}e^{W_cx}}$$&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-11-10T14:07:28.793" LastActivityDate="2017-11-10T14:07:28.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24567" PostTypeId="1" CreationDate="2017-11-10T13:58:48.190" Score="-1" ViewCount="123" Body="&lt;p&gt;I am new to the Data Science. The problem I want to solve is relatively simple in terms of the problem statement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a Numbered document (usually a pdf) with/without an index, I need to extract all the sections based on their numerical value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a document may have something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. Title1&#xA;  1.1 Sub Title1&#xA;    1.1.1 Sub Sub Title 1&#xA;       Lorem Ipsum (a big paragraph)&#xA;    1.1.2 Sub Title 2&#xA;       Some table and graphs&#xA;  1.2 Sub Title 2&#xA;2. Title 2&#xA;......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think you get the point. What I have given above is a very reduced set of a bigger problem I am trying to solve. Is there any tool or technique that is available out there that can do the extraction by sections out of the box. Meaning when I query for Title1 -- it should give all of the Title1 in a tree format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are no such tools or techniques that exist, can someone point me in the right direction/approach to attack this problem?&lt;/p&gt;&#xA;" OwnerUserId="40867" LastActivityDate="2017-12-11T02:10:55.057" Title="Extracting paragraph from a document based on Numerals" Tags="|machine-learning|text-mining|data|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24570" PostTypeId="1" AcceptedAnswerId="24572" CreationDate="2017-11-10T14:42:34.260" Score="1" ViewCount="2077" Body="&lt;p&gt;I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At least I understand that one iteration of mini-batch gradient descent should be faster than one iteration of batch gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if I understand it correctly, since the mini-batch gradient descent must update the weights by the number of the batch size in one epoch, the training would be slower than the batch gradient descent, which computes and updates the weights only once in one epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? In that case, is it worth worrying about the loss of the overall training time?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-11-10T15:23:53.103" Title="Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?" Tags="|gradient-descent|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24571" PostTypeId="1" AcceptedAnswerId="24574" CreationDate="2017-11-10T15:00:07.620" Score="1" ViewCount="392" Body="&lt;p&gt;I was curious to see if one can use a cost function on a set of data points to find the &quot;optimial minimum&quot; solution for any given set of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know for a regular set of data that is clustered symmetrically following a straight regression line is easy to find the proper cost function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tqPIE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tqPIE.png&quot; alt=&quot;Linear model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but what if the data makes a funky shape like that of x^2 or 3*x^3 or etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see that the the second graph is a lot more noisy than the first. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zXSqZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zXSqZ.png&quot; alt=&quot;quadratic model x^2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then what do you do? is it the same process or is it different?&#xA;Can you find an optimal Minimum for any set of data points? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the cost function is using the residuals to form its best fit but I was just curious if it can do with any set of data. &lt;/p&gt;&#xA;" OwnerUserId="41036" LastActivityDate="2017-11-10T15:31:10.730" Title="Cost Function on non linear noisy data" Tags="|machine-learning|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24572" PostTypeId="2" ParentId="24570" CreationDate="2017-11-10T15:23:53.103" Score="3" Body="&lt;p&gt;You are correct, there is more overhead to process the same amount of data because you do more weight updates and maybe preprocessing the batches in your generator will take more time as well. However, since you are doing more updates and if your batch size is big enough, the gradients of your mini-batches will approximate the gradients of your full batch fairly well and you will have gotten much closer to the optimum of your full dataset by splitting your full batch into mini batches than you would have gotten by doing one non-stochastic weight update.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-10T15:23:53.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24573" PostTypeId="1" CreationDate="2017-11-10T15:25:31.407" Score="0" ViewCount="1633" Body="&lt;p&gt;I am trying to do simple calculations in R when no raw data but grouped data with frequencies is available only. This is the case when I have a large amount of records in a database, say a large SQL table, and then for given reasons GROUP BY and COUNT to aggregate instead of downloading the original table for analysis in R. As I understand, one could say in R that I'm talking about data in a table format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give a simple example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; original=c(1,2,2,3)&#xA;&amp;gt; aggregate=table(original)&#xA;original&#xA;1 2 3 &#xA;1 2 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the case I describe, only data structured like &lt;code&gt;aggregate&lt;/code&gt; would be available as the result of my database query, usually in a data.frame similar to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|value|frequency|&#xA;|1    |1        |&#xA;|2    |2        |&#xA;|3    |1        |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What are standard approaches in R to work with this kind of data, particulary when it's the aim to run analysis on &lt;code&gt;value&lt;/code&gt;? E.g., I may want to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Calculate Mean, Median, Quantiles&lt;/li&gt;&#xA;&lt;li&gt;Plot a boxplot which should give the same results as when I would run &lt;code&gt;boxplot(original)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I feel I'm just missing a basic concept of aggregated data processing in R, cannot imagine it's not something being build in a native way in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only idea I could find googling this was basically an approach to let R create &lt;code&gt;original&lt;/code&gt; based on the frequency information.&lt;/p&gt;&#xA;" OwnerUserId="41701" LastEditorUserId="29169" LastEditDate="2021-02-01T20:11:49.930" LastActivityDate="2021-02-01T20:11:49.930" Title="R: Calculations based on frequencies / grouped / aggregate data" Tags="|r|aggregation|data-table|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24574" PostTypeId="2" ParentId="24571" CreationDate="2017-11-10T15:31:10.730" Score="1" Body="&lt;p&gt;I don't think the second picture necessarily has more noise, it's just less linear than the first one. If you are doing regression (which is the case in both your problems) you can use the same loss function to optimize your family of functions that you are using to fit your data. The only difference is that with certain function families gradient descent might not get to a global optimum but will get stuck in a local one. The loss function only determines how to penalize certain errors (based on the residuals).&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-10T15:31:10.730" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24575" PostTypeId="2" ParentId="24557" CreationDate="2017-11-10T15:47:01.160" Score="0" Body="&lt;p&gt;Okay, so from what I understand you need a scoring scheme rather than the scores themselves. I would suggest you to look into the TextRank algorithm &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;source=web&amp;amp;rct=j&amp;amp;url=https://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf&amp;amp;ved=0ahUKEwiBxqWOq7TXAhVEpI8KHZ-YDwEQFgglMAA&amp;amp;usg=AOvVaw0BUSxH01zoqL1GTj3uu0k5&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper here&lt;/a&gt;. It is mainly used for keyword extraction and works like pagerank, however the transition probabilities are defined on a lot of metrics, including synonyms which would solve your problem of counting indirect occurence. It would return you a ranked list of words along with the scores, you can use the scores of the words that you have already extracted. Normalize the extracted scores so that you can get better relative importances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another simple approach is to replace all synonyms with one common token and then use the TF-IDF scores as the needed scores. You can use wordnet to extract synonyms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, understand that these approaches are frequency based and no other information other than synonym information is being used. So they will not be pefect for your application but they will do. If you need a better system then you need human scored data and then running some kind of regression on a set of extractef features for words, as to what makes a word relevant and what doesnt. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-11-10T15:47:01.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24577" PostTypeId="1" CreationDate="2017-11-10T16:18:59.343" Score="2" ViewCount="299" Body="&lt;p&gt;LSTM can be used to generate text, can they be used to fix corrupted text files?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say that my original was: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Alice was beginning to get very tired of sitting by her sister on the&#xA;bank, and of having nothing to do&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now it has become:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Alice0004 XGFT was beginning to get 900io ALPHA very tired 00New York 333 of siing XXXYYY by her siter on THIS IS 898 the&#xA;bank, and of &amp;amp;*&amp;amp; HHH GTY DOG AND CAT having nothing to do.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, a lot of crap was added (some meaningless words but also meaningful words, and some letters may have been erased).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way LSTM would be able to fix this? (Maybe by cleaning some beforehand and then let LSTM do the rest)&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2018-08-08T06:00:40.077" Title="Using LSTM to clear up corrupted text files" Tags="|neural-network|nlp|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24579" PostTypeId="1" AcceptedAnswerId="24584" CreationDate="2017-11-05T03:50:52.970" Score="1" ViewCount="4116" Body="&lt;p&gt;I have FINALLY been able to implement backpropagation, but there are still some bugs I need to fix.  The main is issue the following: &lt;strong&gt;My ReLU activation function produces really big dJdW values (derivative of error function wrt weights).&lt;/strong&gt;  When this gets subtracted from the weights, my output becomes a matrix of -int or inf.  How do I stop this?  As of now, the only solution I have is to make my learning rate &lt;em&gt;scalar&lt;/em&gt; variable REALLY small.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;&#xA;class Neural_Network(object):&#xA;    def __init__(self, input_, hidden_, output_, numHiddenLayer_, numExamples_):&#xA;        # Define Hyperparameters&#xA;        self.inputLayerSize = input_&#xA;        self.outputLayerSize = output_&#xA;        self.hiddenLayerSize = hidden_&#xA;        self.numHiddenLayer = numHiddenLayer_&#xA;        self.numExamples = numExamples_&#xA;        self.scalar = 0.0000000001 # LEARNING RATE: Why does ReLU produce such large dJdW values?&#xA;        # in -&amp;gt; out&#xA;        self.weights = [] # stores matrices of each layer of weights&#xA;        self.z = [] # stores matrices of each layer of weighted sums&#xA;        self.a = [] # stores matrices of each layer of activity &#xA;        self.biases = [] # stores all biases&#xA;&#xA;        # Biases are matrices that are added to activity matrix&#xA;        # Dimensions -&amp;gt; numExamples_*hiddenLayerSize or numExamples_*outputLayerSize&#xA;        for i in range(self.numHiddenLayer):&#xA;            # Biases for hidden layer&#xA;            b = [np.random.random() for x in range(self.hiddenLayerSize)];&#xA;            B = [b for x in range(self.numExamples)];&#xA;            self.biases.append(np.mat(B))&#xA;        # Biases for output layer&#xA;        b = [np.random.random() for x in range(self.outputLayerSize)]&#xA;        B = [b for x in range(self.numExamples)];&#xA;        self.biases.append(np.mat(B))&#xA;&#xA;&#xA;        # Weights (Parameters)&#xA;        # Weight matrix between input and first layer&#xA;        W = np.random.rand(self.inputLayerSize, self.hiddenLayerSize)&#xA;        self.weights.append(W)&#xA;&#xA;        for i in range(self.numHiddenLayer-1):&#xA;            # Weight matrices between hidden layers&#xA;            W = np.random.rand(self.hiddenLayerSize, self.hiddenLayerSize)&#xA;            self.weights.append(W)&#xA;        # Weight matric between hiddenlayer and outputlayer&#xA;        self.weights.append(np.random.rand(self.hiddenLayerSize, self.outputLayerSize))&#xA;&#xA;    def setBatchSize(self, numExamples):&#xA;        # Changes the number of rows (examples) for biases&#xA;        if (self.numExamples &amp;gt; numExamples):&#xA;            self.biases = [b[:numExamples] for b in self.biases]&#xA;&#xA;    def sigmoid(self, z):&#xA;        # Apply sigmoid activation function&#xA;        return 1/(1+np.exp(-z))&#xA;&#xA;    def sigmoidPrime(self, z):&#xA;        # Derivative of sigmoid function&#xA;        return 1-self.sigmoid(z)&#xA;&#xA;    def ReLU(self, z):&#xA;        # Apply activation function&#xA;        for (i, j), item in np.ndenumerate(z):&#xA;            if (item &amp;lt; 0):&#xA;                item *= 0.01&#xA;            else:&#xA;                item = item&#xA;        return z        &#xA;&#xA;&#xA;    def ReLUPrime(self, z):&#xA;        # Derivative of ReLU activation function&#xA;        for (i, j), item in np.ndenumerate(z):&#xA;            if (item &amp;lt; 0):&#xA;                item = 0.01&#xA;            else:&#xA;                item = 1&#xA;&#xA;        return z&#xA;&#xA;    def forward(self, X):&#xA;        # Propagate outputs through network&#xA;&#xA;        self.z.append(np.dot(X, self.weights[0]) + self.biases[0])&#xA;        self.a.append(self.ReLU(self.z[0]))&#xA;&#xA;        for i in range(1, self.numHiddenLayer):&#xA;            self.z.append(np.dot(self.a[-1], self.weights[i]) + self.biases[i])&#xA;            self.a.append(self.ReLU(self.z[-1]))&#xA;&#xA;        self.z.append(np.dot(self.z[-1], self.weights[-1]) + self.biases[-1])&#xA;        self.a.append(self.ReLU(self.z[-1]))&#xA;        yHat = self.ReLU(self.z[-1])&#xA;        return yHat&#xA;&#xA;    def backProp(self, X, y):&#xA;        # Compute derivative wrt W&#xA;        # out -&amp;gt; in&#xA;        dJdW = [] # stores matrices of each dJdW (equal in size to self.weights[])&#xA;        delta = [] # stores matrices of each backpropagating error&#xA;&#xA;        self.yHat = self.forward(X)&#xA;        delta.insert(0,np.multiply(-(y-self.yHat), self.ReLUPrime(self.z[-1]))) # delta = (y-yHat)(sigmoidPrime(final layer unactivated))&#xA;        dJdW.insert(0, np.dot(self.a[-2].T, delta[0])) # dJdW&#xA;        for i in range(len(self.weights)-1, 1, -1):&#xA;            # Iterate from self.weights[-1] -&amp;gt; self.weights[1]&#xA;            delta.insert(0, np.multiply(np.dot(delta[0], self.weights[i].T), self.ReLUPrime(self.z[i-1])))&#xA;            dJdW.insert(0, np.dot(self.a[i-2].T, delta[0]))&#xA;&#xA;        delta.insert(0, np.multiply(np.dot(delta[0], self.weights[1].T), self.ReLUPrime(self.z[0])))&#xA;        dJdW.insert(0, np.dot(X.T, delta[0]))&#xA;&#xA;&#xA;        return dJdW&#xA;&#xA;    def train(self, X, y):&#xA;        for t in range(60000):&#xA;            dJdW = self.backProp(X, y)&#xA;            for i in range(len(dJdW)):&#xA;                self.weights[i] -= self.scalar*dJdW[i]&#xA;&#xA;# Instantiating Neural Network&#xA;inputs = [int(np.random.randint(0,100)) for x in range(100)]&#xA;x = np.mat([x for x in inputs]).reshape(100,1)&#xA;y = np.mat([x+1 for x in inputs]).reshape(100,1)&#xA;NN = Neural_Network(1,3,1,1,100)&#xA;&#xA;&#xA;# Training&#xA;print(&quot;INPUT: &quot;, end = '\n')&#xA;print(x, end = '\n\n')&#xA;&#xA;print(&quot;BEFORE TRAINING&quot;, NN.forward(x), sep = '\n', end = '\n\n')&#xA;NN.train(x,y)&#xA;print(&quot;AFTER TRAINING&quot;, NN.forward(x), sep = '\n', end = '\n\n')&#xA;&#xA;# Testing&#xA;test = np.mat([int(np.random.randint(0,100)) for x in range(100)]).reshape(100,1)&#xA;print(&quot;TEST INPUT:&quot;, test, sep = '\n', end = '\n\n')&#xA;print(NN.forward(test), end = '\n\n')&#xA;&#xA;&#xA;NN.setBatchSize(1) # changing settings to receive one input at a time&#xA;&#xA;while True:&#xA;    # Give numbers between 0-100 (I need to fix overfitting) and it will get next value&#xA;    inputs = input()&#xA;    x = np.mat([int(i) for i in inputs.split(&quot; &quot;)])&#xA;    print(NN.forward(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I first made the ANN using sigmoid but Leaky ReLU is faster.&#xA;The code is a bit much so here is a summary:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Neural Network Class&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;define hyperparameter and stuff (include really small learning rate scalar)&lt;/li&gt;&#xA;&lt;li&gt;activation functions and their derivatives (&lt;em&gt;ReLU&lt;/em&gt; and sigmoid)&lt;/li&gt;&#xA;&lt;li&gt;Member functions: forward propagation, backpropagation, setBatchSize etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Instantiating ANN&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;setting hyperparameters (topology of ANN)&lt;/li&gt;&#xA;&lt;li&gt;creating data (one array has values x and the output array has values x+1)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Training&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using inputs generated in step 2 to train ANN&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Testing&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Testing using randomly generated inputs&lt;/li&gt;&#xA;&lt;li&gt;User can give inputs&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope that helps you help me.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="41755" OwnerDisplayName="Aniekan Umoren" LastActivityDate="2017-11-10T22:34:34.517" Title="ReLU activation function outputs HUGE numbers" Tags="|machine-learning|backpropagation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24580" PostTypeId="1" CreationDate="2017-11-10T17:32:32.087" Score="5" ViewCount="135" Body="&lt;p&gt;&lt;strong&gt;The problem:&lt;/strong&gt; We have several people that do a lot of ML work in our lab and have more people asking to use our equipment. We have several machines that have Ubuntu on them that people share. This causes a problem for resource contention. PersonA had all of their code/libraries on MachineA or a person is waiting on MachineB which has 4GPUs instead of 1GPU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Looking for:&lt;/strong&gt;&#xA;What I’m looking for is a way of creating a cluster with these machines that will allow people to utilize and share the resources. This would allow people to scale out and make better use of the resources that we already have (e.g. waiting for a specific machine that is already used). This could be used by interactively running code in their IDE or submitting jobs to the cluster like some type of scheduler (slurm, pbs). At this point the end user could look at this as a &quot;pool&quot; of resources. (e.g. we have 100GPUs, personA requests 4GPUs, we have 96GPUs available for someone else to request). Would also like to avoid the user from having to anything about the cluster. For example UserA simply requests 4GPUs, they don't know and don't care which machines it gets launched on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They use a mix of Tenserflow and mxnet with keras and some theano.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tensorflow/mxnet seem to have clustering option but don’t think this would handle the scheduling problem? or would it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have successfully deployed some type of machine learning cluster with GPU nodes, I would really be interested in seeing your architect, tools, and software you used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is out of scope for this forum please feel free to move this to another channel.&lt;/p&gt;&#xA;" OwnerUserId="14829" LastActivityDate="2018-06-09T08:50:46.260" Title="Architecture for Machine Learning resource management" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24581" PostTypeId="2" ParentId="24543" CreationDate="2017-11-10T17:43:50.850" Score="5" Body="&lt;h3&gt;Precision&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Your step change in precision looks to be almost entirely explained by the change in positive class frequency. It is reasonable to expect the proportion of false positives to increase when increasing the proportion of negative examples. Even if you assume your cv results were perfect, then you would see some increase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, assume you have cv results representative of test results &#xA; - which means same distribution before random under-sampling, and no over-fit to the cv set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you measured precision at 0.97 with a t:f ratio of 1:2, and for the sake of simplicity that this was due to the following confusion table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Predicted:  T    F&#xA;Real T           97    3&#xA;Real F            3  197&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What precision should you expect when going to the real distribution? That is the same as multiplying the bottom row of the confusion table by 50. Precision is $\frac{TP}{TP+FP}$, so your expected precision would be $\frac{97}{97+150} \approx 0.39$&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Recall&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The same effect does not impact recall, because it is about the ratio between true positive and false negative. So when you change the ratio of positive to negative classes, in theory recall should be unaffected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, recall has been affected, but a lot less than precision. that is promising. A drop from 0.95 to 0.85 between cv and test is not great perhaps, but it doesn't point to a really major problem, just room for improvement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few possible causes. The ones that I can think of are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Your test set might be too small, so estimates of precision and recall have large error. So in fact there is no problem . . .&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Test distribution might be different to train and cv set. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train/CV set split might allow some data leakage (e.g. they share some common features such as data about the same person, and should be split by that common feature). In which case CV estimates could be too high.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Your mechanism for under-sampling the negative class may be biased.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;What to do?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;First of all, these results are unlikely to be anything directly do with faults in the model, and are not informed much by the training curves. They are also not that bad out of context (i.e. they are much better than simply guessing which items are in the positive class) - the question is more whether you could improve on them, and what the costs are to you for the different types of error. It might be worth you actually assigning real-world comparable costs to each type of error, to help decide whether your model is successful/useful and to pick the best model later on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing from the training curves is that your cv and training loss look pretty close. It implies you are not over-fitting to the training data (or you should check to a train/cv data leak). You may have room to add more parameters and improve the model in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible you could make the model even better with different hyper-parameter choices, feature engineering etc, and that would improve the scores. There is no general advice for that though, it depends on what you have available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be worth experimenting with training on the unbalanced training set (take the raw data without undersampling) and instead weighting the loss function, so that costs are larger for inaccurate classification of positive class. This is not guaranteed to fix your problem, but will increase the amount of data you use for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you should investigate whether any of the possible causes listed above is likely and try to apply fixes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, in this situation, it is not unheard of to have a four-way data split:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A ratio-adjusted set split two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training data&lt;/li&gt;&#xA;&lt;li&gt;CV or &quot;Dev&quot; set A&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A same as production set split two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;CV or &quot;Dev&quot; set B&lt;/li&gt;&#xA;&lt;li&gt;Test set&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;CV set A is used to perform early stopping and low-level model selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CV set B is used to perform high-level model selection against production metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test set is used to assess the chosen &quot;best&quot; model without bias.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-10T17:57:27.817" LastActivityDate="2017-11-10T17:57:27.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24582" PostTypeId="2" ParentId="13461" CreationDate="2017-11-10T18:21:27.847" Score="1" Body="&lt;pre&gt;&lt;code&gt;self.result = self.model.predict(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where X is numpy array. That is all I did and it worked.&lt;/p&gt;&#xA;" OwnerUserId="41705" LastActivityDate="2017-11-10T18:21:27.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24583" PostTypeId="1" AcceptedAnswerId="24692" CreationDate="2017-11-10T20:48:08.787" Score="-1" ViewCount="927" Body="&lt;p&gt;I am working on a binary classification problem with imbalanced data. The dataset consists of 85% 'false' labels and 15% 'true' labels. I have about 1500 data points and am training with Sklearn ensemble tree methods and XGBoost. I know tree methods tend to work better with more data points but I'm not sure what sampling methods would help. I'm trying to optimize for more 'true' points to be in the top decile. I tried to oversample the 'true' class but since I'm using a top decile conversion, the top decile get artificially populated with a lot of 'true' points. This gave inconsistent conversion scores when I used stratified k-fold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I split the datasets by class and apply sample techniques? What are some relevant techniques that I can try? I'm working with Python and any relevant libraries would be helpful. Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="41710" LastActivityDate="2017-11-13T21:42:46.303" Title="Imbalanced classification data with a top decile conversion metric" Tags="|python|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24584" PostTypeId="2" ParentId="24579" CreationDate="2017-11-10T22:34:34.517" Score="1" Body="&lt;p&gt;I suspect the problem is the fact that your input data values are very high. You're trying to map the input variable $x \in (0,100)$ to $y = x+1$ in your code, but neural networks work best when the data has much lower values. A good strategy is to normalise the data before training so that each feature has zero mean and unit variance. Try scaling your data down like so (I've also changed the code that originally generates the inputs to make it more efficient in numpy):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Instantiating Neural Network&#xA;x = np.random.randint(0, 100, size=100).reshape(100,1)&#xA;y = x + 1&#xA;&#xA;# normalize data to have zero mean and unit variance&#xA;x_normalized = (x - x.mean()) / x.std()&#xA;y_normalized = (y - y.mean()) / y.std()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Train your network with &lt;code&gt;x_normalized&lt;/code&gt; and &lt;code&gt;y_normalized&lt;/code&gt; instead of &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;. Then, during testing, you normalize your input data like above, and you can scale your predictions back up to the original scale by rearranging the above formula.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate test data &amp;amp; normalize using train data mean&amp;amp;std&#xA;test = np.random.randint(0, 100, size=100).reshape(100,1)&#xA;test_x_normalized = (test - x.mean()) / x.std()&#xA;&#xA;# input to network to get normalized outputs&#xA;test_y_normalized = NN.forward(test_x_normalized)&#xA;&#xA;# rescale normalized outputs to original 0-100 scale&#xA;test_y = (test_y_normalized * y.std()) + y.mean()&#xA;&#xA;print(&quot;TEST INPUT:&quot;, test, sep = '\n', end = '\n\n')&#xA;print(test_y, end = '\n\n')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2017-11-10T22:34:34.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24585" PostTypeId="2" ParentId="24573" CreationDate="2017-11-10T22:45:31.423" Score="1" Body="&lt;p&gt;I think working with &quot;collapsed&quot; or &quot;summarized&quot; data as opposed to the data itself directly inside the SQL database is counter-intuitive. Also, trying to download the original table for analysis in R is a very bad route if the data is too big to fit in the memory, and given that you already have a tool like SQL, which is designed for that specific purpose. You can calculate the mean, median, quantiles, etc. in SQL; and then pull that into R (even into Excel) and draw box plots.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though overkill, here is what you can do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the mean, median, quantiles from the frequency chart by coding basic math in R. For example mean of a series is the sum over frequency(i)*value(i)/sum(frequency(i)), where i=1,...,n, and n is the number of distinct values in the frequency chart. Median is the value which is in the mid point of series 1,...,sum(frequency(i)). &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can recover the original series from summary statistics, but memory is again of concern. Let's say your original series was this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;mySeries &amp;lt;- c(5,9,4,4,2,5,8,9,9,5,5)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But obviously it sits in the database, and this is what you get from SQL as you described:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fromSQL &amp;lt;- data.frame(table(mySeries))&#xA;&amp;gt; fromSQL&#xA;  mySeries Freq&#xA;1        2    1&#xA;2        4    2&#xA;3        5    4&#xA;4        8    1&#xA;5        9    3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can just use the &lt;strong&gt;rep&lt;/strong&gt; function to recover the series like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; recoveredSeries &amp;lt;- as.integer(as.character(rep(fromSQL$mySeries, fromSQL$Freq)))&#xA;&amp;gt; recoveredSeries&#xA; [1] 2 4 4 5 5 5 5 8 9 9 9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now define a function that returns the summary of this series and gives you what you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; multiSummary &amp;lt;- function(x) {&#xA;+   c(mean = mean(x), median = median(x))&#xA;+ }&#xA;&amp;gt; multiSummary(recoveredSeries)&#xA;    mean   median &#xA;5.909091 5.000000 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As I said, in my opinion, all of this is overkill. Different tools are tailored for different needs, and trying to do something with a tool that is not really the best for your need will give you less-than-optimal results and cost you a lot of time. My recommendation would be performing SQL operations, or going for MapReduce framework for this kind of purposes, depending on the size of your data.&lt;/p&gt;&#xA;" OwnerUserId="20572" LastEditorUserId="20572" LastEditDate="2017-11-12T16:49:25.427" LastActivityDate="2017-11-12T16:49:25.427" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24586" PostTypeId="2" ParentId="24580" CreationDate="2017-11-10T23:22:43.253" Score="1" Body="&lt;p&gt;It depends on how the cluster (i.e., compute resources) are being managed. For example, &lt;a href=&quot;http://mesos.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Mesos&lt;/a&gt; is an open-source cluster management tool. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One established option is &lt;a href=&quot;https://github.com/uber/horovod&quot; rel=&quot;nofollow noreferrer&quot;&gt;Horovod&lt;/a&gt;. It is designed to manage workloads for &#xA;TensorFlow and Keras across multiple cores in a single machine or across multiple machines.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-10T23:22:43.253" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24587" PostTypeId="2" ParentId="24577" CreationDate="2017-11-10T23:36:09.773" Score="0" Body="&lt;p&gt;Yes, you can definitely try that. I suggest you try Character level LSTM as it helps in fixing certain spelling mistakes. Even if not that, you can try a simple trie. LSTM would probably need a lot of data so I suggest maybe using some basic rules first as there are a lot of unknown words. Maybe it is worth using a keyword based rule to eliminate the meaningless words. Maybe use some Wordnet and get words in English dictionary to get existing words. Then, check if a word is very close to a dictionary word using Levenshtein distance or another metric.&lt;/p&gt;&#xA;" OwnerUserId="21024" LastActivityDate="2017-11-10T23:36:09.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24588" PostTypeId="2" ParentId="24560" CreationDate="2017-11-10T23:51:19.827" Score="2" Body="&lt;p&gt;Here is how to fit k-means to a single-dimensional text data in Pandas:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;&#xA;df = pd.DataFrame({&amp;quot;corpus&amp;quot;: [&amp;quot;I am Sam. Sam-I-am&amp;quot;,&#xA;                              &amp;quot;That Sam-I-am! That Sam-I-am! I do not like that Sam-I-am&amp;quot;,&#xA;                              &amp;quot;Do you like green eggs and ham?&amp;quot;,&#xA;                              &amp;quot;I do not like them, Sam-I-am. I do not like green eggs and ham&amp;quot;]})&#xA;&#xA;X = TfidfVectorizer().fit_transform(df.corpus)&#xA;km = KMeans(n_clusters=2).fit(X)&#xA;km.labels_.tolist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2022-04-10T13:22:28.833" LastActivityDate="2022-04-10T13:22:28.833" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24589" PostTypeId="2" ParentId="24567" CreationDate="2017-11-11T00:14:52.857" Score="0" Body="&lt;p&gt;Assuming your file can be converted to &quot;plain text&quot; from pdf format, you could write a &lt;a href=&quot;https://en.wikipedia.org/wiki/Regular_expression&quot; rel=&quot;nofollow noreferrer&quot;&gt;regular expression&lt;/a&gt; rule to find the start of sections. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would require finding all lines that start with a digit followed by a literal period, ignoring tabs. The regular expression is something like &lt;code&gt;(^\t*\d\.)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-11T00:14:52.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24590" PostTypeId="2" ParentId="19191" CreationDate="2017-11-11T01:03:55.457" Score="2" Body="&lt;p&gt;This is called &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bilevel_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;bilevel optimization&lt;/a&gt;&quot; where one problem is embedded (nested) within another. If there are more than one objectives at one or both levels, it is then called &quot;multi-objective bilevel optimization&quot;. There is a rich literature on this problem, including the book &lt;a href=&quot;http://www.springer.com/us/book/9780792346937&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multilevel Optimization: Algorithms and Applications&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finding solutions to minimize the objectives often does not require machine learning, just &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0305054893900139&quot; rel=&quot;nofollow noreferrer&quot;&gt;linear programming&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-11T01:03:55.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24591" PostTypeId="2" ParentId="24497" CreationDate="2017-11-11T01:27:30.110" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://github.com/twitter/AnomalyDetection&quot; rel=&quot;nofollow noreferrer&quot;&gt;AnomalyDetection&lt;/a&gt; is an open-source R package to detect anomalies&#xA;  which is robust, from a statistical standpoint, in the presence of&#xA;  seasonality and an underlying trend.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A blogpost that introduces the package can be found &lt;a href=&quot;https://blog.twitter.com/engineering/en_us/a/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and more formal paper can be found &lt;a href=&quot;https://arxiv.org/abs/1704.07706&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-11T01:27:30.110" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24592" PostTypeId="1" AcceptedAnswerId="24594" CreationDate="2017-11-11T01:28:55.530" Score="5" ViewCount="354" Body="&lt;p&gt;Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This visualization confused me:&#xA;&lt;a href=&quot;http://scs.ryerson.ca/~aharley/vis/conv/flat.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scs.ryerson.ca/~aharley/vis/conv/flat.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Related side question:&lt;/em&gt; If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?&lt;/p&gt;&#xA;" OwnerUserId="39588" LastEditorUserId="29169" LastEditDate="2021-02-15T22:20:58.040" LastActivityDate="2021-02-15T22:20:58.040" Title="Do Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels?" Tags="|machine-learning|deep-learning|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24593" PostTypeId="1" CreationDate="2017-11-11T08:05:05.023" Score="2" ViewCount="470" Body="&lt;p&gt;I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-series based problem could be better solved using Hidden Markov Models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset consists of a time-stamp label, around 30 features collected from IoT sensors and then there is one target class which is a continuous variable. The problem is how do I determine the structure of the HMM given these features and number of hidden states to represent the high dimensional data and once the above is done, how to predict the continuous target value using the set of transition probabilities and model parameters that get generated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have gone through references where the structure of the HMM is specified beforehand which makes it easy to work with, but how do we go about predicting a continuous value with a HMM?&lt;/p&gt;&#xA;" OwnerUserId="41720" LastActivityDate="2023-12-18T15:09:06.047" Title="How to use HMMs for continuous value prediction" Tags="|machine-learning|predictive-modeling|time-series|markov-process|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24594" PostTypeId="2" ParentId="24592" CreationDate="2017-11-11T08:57:09.290" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. The usual convention in a CNN is that each kernel is always the same depth as the input, so you can also think of this as a &quot;stack&quot; of 2D kernels that are associated with the input channels and summed to make one output channel - because under the convention that $N_{in\_channels} = N_{kernel\_depth}$ this is mathematically the same. Expressing as a 3D convolution allows for simpler notation and code.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The diagram is non-standard in that respect, although it seems to show pooling and fully-connected layers as normal. It might be a mistake in the diagram, or something unconventional about that specific CNN.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Correct. You can see this in the &lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;visualised filters for AlexNet&lt;/a&gt; (do note that for computational reasons, AlexNet specialised one half of its filters to work in greyscale, and had other clever optimisations that we don't use nowadays because available GPU power is high enough to not need them). Most implementations will also treat a greyscale image as a 1-channel 3D shape for consistency.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-11T09:18:51.317" LastActivityDate="2017-11-11T09:18:51.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24595" PostTypeId="1" CreationDate="2017-11-11T10:18:25.693" Score="-3" ViewCount="480" Body="&lt;p&gt;My question is the same as here:&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/18258/how-to-force-weights-to-be-non-negative-in-linear-regression&quot;&gt;How to force weights to be non-negative in Linear regression&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Except that I can only use Numpy (I cannot use Scipy or Scikit Learn).&#xA;Indeed, I am running my Python script on a server which doesn't include these modules.&lt;/p&gt;&#xA;&lt;p&gt;Is there any solution ?&lt;/p&gt;&#xA;" OwnerUserId="41726" LastEditorUserId="43000" LastEditDate="2021-04-08T16:24:19.787" LastActivityDate="2021-04-08T16:24:19.787" Title="How to force weights to be non-negative in Linear regression only with Numpy" Tags="|python|regression|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24597" PostTypeId="2" ParentId="15032" CreationDate="2017-11-11T13:31:47.033" Score="6" Body="&lt;p&gt;Another option worth checking out is &lt;code&gt;[tf.einsum][1]&lt;/code&gt; - it's essentially a simplified version of &lt;a href=&quot;https://en.wikipedia.org/wiki/Einstein_notation&quot; rel=&quot;noreferrer&quot;&gt;Einstein Notation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following along with Neil and dumkar's examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;a = tf.placeholder(tf.float32, shape=(None, 3))&#xA;b = tf.placeholder(tf.float32, shape=(None, 3))&#xA;&#xA;c = tf.einsum('ij,ij-&amp;gt;i', a, b)&#xA;&#xA;with tf.Session() as session:&#xA;    print( c.eval(&#xA;        feed_dict={ a: [[1,2,3],[4,5,6]], b: [[2,3,4],[5,6,7]] }&#xA;    ) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first argument to &lt;code&gt;einsum&lt;/code&gt; is an equation representing the axes to be multiplied and summed over. The basic rules for an equation are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Input-tensors are described by a comma-separated string of dimension-labels&lt;/li&gt;&#xA;&lt;li&gt;Repeated labels indicate that the corresponding dimensions will be multiplied&lt;/li&gt;&#xA;&lt;li&gt;The output-tensor is described by another string of dimension-labels representing corresponding inputs (or products)&lt;/li&gt;&#xA;&lt;li&gt;Labels that are missing from the output string are summed over&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In our case, &lt;code&gt;ij,ij-&amp;gt;i&lt;/code&gt; means that our inputs will be 2 matrices of equal shape &lt;code&gt;(i,j)&lt;/code&gt;, and our output will be a vector of shape &lt;code&gt;(i,)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get the hang of it, you'll find that &lt;code&gt;einsum&lt;/code&gt; generalizes a huge number of other operations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = [[1, 2]]&#xA;Y = [[3, 4], [5, 6]]&#xA;&#xA;einsum('ab-&amp;gt;ba', X) == [[1],[2]]   # transpose&#xA;einsum('ab-&amp;gt;a',  X) ==  [3]        # sum over last dimension&#xA;einsum('ab-&amp;gt;',   X) ==   3         # sum over both dimensions&#xA;&#xA;einsum('ab,bc-&amp;gt;ac',  X, Y) == [[13,16]]          # matrix multiply&#xA;einsum('ab,bc-&amp;gt;abc', X, Y) == [[[3,4],[10,12]]]  # multiply and broadcast&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, &lt;code&gt;einsum&lt;/code&gt; takes a pretty hefty performance hit when compared to a manual multiply+reduce. Where performance is critical, I'd definitely recommend sticking with Neil's solution.&lt;/p&gt;&#xA;" OwnerUserId="41733" LastActivityDate="2017-11-11T13:31:47.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24598" PostTypeId="1" AcceptedAnswerId="24600" CreationDate="2017-11-11T15:12:32.687" Score="3" ViewCount="355" Body="&lt;p&gt;I'm implementing DQN algorithm from scratch on MountainCar simulation.&#xA;I'm using a setup of $reward = 1.0$ when car hits the flag, and $0$ otherwise. &#xA;Reward decay factor is set to $\gamma=0.99$. Algorithm starts with exploration factor of $\epsilon = 1.0$ and decreases that over time to $\epsilon = 0.1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understood correctly, $Q$ function for some state and action pair is defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q(s_t, a_t) = r_t + \gamma \times \operatorname{arg\,max}_a Q(s_{t+1}, a_{t+1})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, $Q_{max}$ would satisfy the condition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{max} = r_{max} + \gamma \times Q_{max}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q_{max} = \frac{r_{max}}{1 - \gamma}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, since my network only approximates the $Q$ function it is possible for it to sometimes produce value greater than $Q_{max}$. When that happens, further training causes the values to start growing exponentially, and the entire thing blows up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I clamp the error of expected value vs. current predicted value to some small number, it still causes the blowup just a bit slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only solution I can think of is to clamp the predicted value $Q(s_{t+1}, a{t+1})$ to $Q_{max}$ and forcing it to never go above that.&#xA;I have done that, and got OK results with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense? Is this a situation that happens in DQN? Or maybe I missed something and my implementation is a bit buggy?&lt;/p&gt;&#xA;" OwnerUserId="41736" LastEditorUserId="41736" LastEditDate="2017-11-11T16:10:32.810" LastActivityDate="2017-11-12T22:30:21.483" Title="Clamping Q function to it's theoretical maximum, yes or no?" Tags="|machine-learning|neural-network|algorithms|q-learning|dqn|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24599" PostTypeId="1" AcceptedAnswerId="24628" CreationDate="2017-11-11T15:36:06.917" Score="1" ViewCount="1509" Body="&lt;p&gt;I implemented a normal classification tree (that uses the Gini index to look for a split). I am using it to predict the age of people. My input data was a series of points on 1 axis (Only X coordinate), and output was an age (9 years old, 10 years old....).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p1 = 1.3, p2 = 3.4, p3 = 2.1 ........ =&gt; Output = 9 years old&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p1 = 1.4, p2 = 2.4, p3 = 2.6 ........ =&gt; Output = 10 years old&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My tree looks something like (just an example):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if p1 &amp;gt; 1&#xA;    if p2 &amp;lt; 3&#xA;        output = 10 years old&#xA;    else &#xA;        output = 9 years old&#xA;else&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is working fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to modify it, to work with 3D points. I mean points that have X, Y, Z and not just X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my input data would be like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;p1 = [1.2, 1.5, 4.3], p2 = [4.2, 1.3, 5.2] .... =&gt; output = 9 years old&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I approach this? should my tree use each coordinate as a seperate input (p1x, p1y, p1z, p2x, p2y....) or is there an approach that can be taken for my case?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am using classification trees but if you think there's another way of predicting in my case, I would love to know about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I'm using python code, but I'm looking for the logic not the programming part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="41737" LastActivityDate="2018-01-11T10:34:34.393" Title="Using a decision tree with 3 dimensional input points" Tags="|classification|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24600" PostTypeId="2" ParentId="24598" CreationDate="2017-11-11T16:10:18.607" Score="2" Body="&lt;p&gt;As MountainCar is often solved with $\gamma = 1$ and a negative reward per timestep, you would immediately hit a problem with your ability to calculate a maximum action value in that case. However, I don't think that is your problem here, the discounted return with positive reward at the end should still encourage desired behaviour for the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is likely that you are experiencing known problems with RL, and the &quot;deadly triad&quot;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Function approximation (neural network)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;on a bootstrap method (Q-Learning or any TD-learning approach)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;off policy (learning optimal policy from non-optimal behaviour*, which is a feature of Q-learning)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This combination is often unstable and difficult to train. Your Q value clamping is one way to help stabilise values. Some features of DQN approach are also designed to deal with this issue:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Experience replay. Agent does not learn online, but instead puts each sample (S, A, R, S') into a memory table and trains the neural network on mini-batches sampled from this memory. Commonly this minibatch update is run on every step (after enough experience collected) and might be e.g. size 32. So learning updates happen faster than experience is collected.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Frozen bootstrap target network. The network used to calculate $\operatorname{max}_{a'} Q(s', a')$ when setting the target values to learn (in td target $R + \gamma \operatorname{max}_{a'} Q(s', a')$) is kept stable, and refreshed to use a copy of the current weights after a certain number of steps (e.g. 100 or 1000, or once every 10 episodes).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Technically off-policy learning is learning the cost function for any policy $\pi$ from a different behaviour policy $b$, but the common case is for control systems attempting to find an optimal policy from an exploratory policy.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-12T22:30:21.483" LastActivityDate="2017-11-12T22:30:21.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24604" PostTypeId="1" CreationDate="2017-11-11T17:38:13.573" Score="1" ViewCount="124" Body="&lt;p&gt;I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?&lt;/p&gt;&#xA;" OwnerUserId="41741" LastActivityDate="2017-11-11T17:38:13.573" Title="How to decrease the learning rate only when cost function is stagnant?" Tags="|tensorflow|cost-function|learning-rate|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24605" PostTypeId="2" ParentId="13032" CreationDate="2017-11-11T17:50:46.640" Score="2" Body="&lt;p&gt;One thing to remember when we use Random Forest is when you use a categorical feature for training it shouldn't have more than 53 categories. Sometimes RandomForest takes numerical data as categorical. To overcome that make sure to convert all categorical as factors using this command.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['col_name'] = df['col_name'].astype('category')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Lastly, make sure that all the levels in a categorical feature are presnent in both train and test data or else the model cannot hadle such situations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you might get the model running if you can use any server(as the model can use the servers processing capacity to its fullest) and deploy on that and let it run before you leave for the day, before deploying make sure that you have tested the code on sample data. So, that it won't abend with some error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you need any additional information.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-11T17:50:46.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24607" PostTypeId="1" CreationDate="2017-11-11T20:46:44.770" Score="0" ViewCount="786" Body="&lt;p&gt;I have chosen the topic of &quot;Predicting future airfare using past data&quot; for my project and would love to get inputs on the best models to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Set&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data set consists of 6 months of time series data for every onward and return journey dates spaced by 8 days for a particular route. So, for the trip of 1 Oct-9 Oct, 2017, I have the flight price checked every day from 1 Apr to 30 Sep. I have this data for every trip date from 1 July, 2017 to 15 Oct, 2017.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Format&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Fake Data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| requestDate        | price     | tripStartDeparture | tripDestinationDeparture | flightCarrier |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 14APR2017:00:00:00 | 725.32    | 16SEP2017:10:50:02 | 23SEP2017:21:55:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 15APR2017:00:00:00 | 966.32    | 16SEP2017:13:20:02 | 23SEP2017:19:00:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 16APR2017:00:00:00 | 915.32    | 16SEP2017:13:20:02 | 23SEP2017:21:55:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 16APR2017:00:00:00 | 825.32    | 16SEP2017:10:50:02 | 23SEP2017:21:55:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 16APR2017:00:00:00 | 969.32    | 11SEP2017:13:20:02 | 18SEP2017:19:00:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;| 16APR2017:00:00:00 | 918.32    | 06SEP2017:13:20:02 | 13SEP2017:21:55:04       | XA            |&#xA;+--------------------+-----------+--------------------+--------------------------+---------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data Exploration&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Variation in price for different start dates. &#xA;X-axis: (Observation date - start date).days, &#xA;Y-axis: Price in USD&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VSlza.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VSlza.png&quot; alt=&quot;Variation in price for different start dates. X-axis: (Observation date - start date).days, Y-axis: Price in USD&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Goal&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given trip dates, the goal is to predict the price in the future, given the price today.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite the data being for 6 months for each trip date, airfares are mostly constant in until about 120 days before departure. It becomes quite noisy in those 120 days. Most probably, because 120 data points are little for time series forecasting, ARIMA did not perform very well. However, I think it is possible to create a way better model utilizing the data for all the data for all trips from 1 July, 2017 to 15 Oct, 2017.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would love to hear about possible approaches. Thank you for reading this wall of text.&lt;/p&gt;&#xA;" OwnerUserId="41747" LastEditorUserId="41747" LastEditDate="2017-11-18T18:05:00.740" LastActivityDate="2017-12-04T09:32:46.903" Title="Predicting future airfare using past data" Tags="|machine-learning|deep-learning|time-series|regression|forecast|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24608" PostTypeId="1" CreationDate="2017-11-11T22:15:15.017" Score="4" ViewCount="3012" Body="&lt;p&gt;I am currently using keras as a deep learning library on top of tensorflow. I just want to know is there any other library which is more efficient/easier than keras.&lt;/p&gt;&#xA;" OwnerUserId="40891" LastActivityDate="2023-03-06T22:02:57.477" Title="Is there any good alternative for Keras library?" Tags="|deep-learning|keras|" AnswerCount="3" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24609" PostTypeId="1" CreationDate="2017-11-11T23:18:12.817" Score="1" ViewCount="67" Body="&lt;p&gt;I have an excel spreadsheet that has over 1 million rows and about 120 columns. I frequently need to find a value or a part of a value (value is name or number) and return the row where the value is found. I only need values from about 20 columns so I could drop out the extra columns but that does not solve the problem of excel being very slow in finding the value and it is very difficult to read information in the row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would you recommend to use for this task? Ideally I would like to enter the value and get the row in some sort of a report but not necessary as long as it is well readable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of using Power BI but got kind of stuck and in the end felt that it wasn't the right tool for the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am open for all suggestions!&lt;/p&gt;&#xA;" OwnerUserId="33375" LastEditorUserId="9935" LastEditDate="2017-11-13T12:21:35.560" LastActivityDate="2017-11-13T20:32:54.427" Title="Find value and return row in excel spreadsheet with over 1 mil rows" Tags="|excel|" AnswerCount="2" CommentCount="0" ClosedDate="2017-11-15T06:37:55.803" ContentLicense="CC BY-SA 3.0" />
  <row Id="24610" PostTypeId="1" CreationDate="2017-11-11T23:20:19.680" Score="6" ViewCount="34751" Body="&lt;p&gt;I have read that the SMOTE package is implemented for binary classification. In the case of n classes, it creates additional examples for the smallest class. Can I balance all the classes by running the algorithm n-1 times?&lt;/p&gt;&#xA;" OwnerUserId="41750" LastEditorUserId="25849" LastEditDate="2020-10-02T16:10:42.177" LastActivityDate="2024-03-22T19:24:41.620" Title="SMOTE and multi class oversampling" Tags="|python|class-imbalance|sampling|smote|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24611" PostTypeId="1" AcceptedAnswerId="24612" CreationDate="2017-11-11T23:44:58.883" Score="0" ViewCount="141" Body="&lt;p&gt;I have a dataset C of 50,000 (binary) samples each of 128 features. The class label is also binary either 1 or -1. For instance, a sample would look like this [1,0,0,0,1,0, .... , 0,1] [-1]. My goal is to classify the samples based on the binary classes( i.e., 1 or -1). I thought to try using Recurrent NN (LSTM) to generate a good model for classification. For that, my question is: Is it possible to use Recurrent NN (LSTM) for such a classification problem?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tr_C, ts_C, tr_r, ts_r = train_test_split(C, r, train_size=.8)&#xA;batch_size = 1000&#xA;&#xA;print('&amp;gt;&amp;gt;&amp;gt; Build STATEFUL model... (%s, %s)' % (tr_C.shape[1], tr_C.shape[2]))&#xA;model = Sequential()&#xA;model.add(LSTM(129, batch_input_shape=(batch_size, C.shape[1], C.shape[2]), return_sequences=False, stateful=True))&#xA;model.add(Dense(1, activation='relu'))&#xA;&#xA;print('&amp;gt;&amp;gt;&amp;gt; Training...')&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;model.fit(tr_C, tr_r,&#xA;          batch_size=batch_size, epochs=1, shuffle=True,&#xA;          validation_data=(ts_C, ts_r))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="41668" LastEditorUserId="41668" LastEditDate="2017-11-12T00:32:07.410" LastActivityDate="2017-11-12T00:32:07.410" Title="Is it possible to use Recurrent NN (LSTM) for classification?" Tags="|machine-learning|neural-network|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24612" PostTypeId="2" ParentId="24611" CreationDate="2017-11-12T00:13:02.237" Score="2" Body="&lt;p&gt;Yes, it is possible. You will first use a LSTM layer and then use a Dense/Linear layer on top of it to get the final output. Make sure that your linear layer has number of outputs = number of classes.&lt;/p&gt;&#xA;" OwnerUserId="41752" LastActivityDate="2017-11-12T00:13:02.237" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24614" PostTypeId="1" CreationDate="2017-11-12T02:37:04.290" Score="0" ViewCount="1961" Body="&lt;p&gt;I am using Keras LSTM (Tensorflow backend) to fit a time series model. Here is my model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(LSTM(LSTM_nums, input_shape=(timestep,data_dim)))&#xA;model.add(Dense(2))&#xA;model.compile(loss='mean_squared_error', optimizer='adam')&#xA;model.fit(trainX, trainY, epochs=100, batch_size=1, verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am wondering is there a constraint or rule of thumb (or best practice) for setting the LSTM_nums value with a given timestep? I.e. should there be a relationship between &lt;code&gt;LSTM_nums ~ timestep&lt;/code&gt; ? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2018-05-04T06:46:35.453" Title="Keras: LSTM unit nums vs timesteps" Tags="|python|keras|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24615" PostTypeId="1" AcceptedAnswerId="24627" CreationDate="2017-11-12T03:11:08.080" Score="0" ViewCount="83" Body="&lt;p&gt;In my dataset i have 3 classes-&gt; 0,1,2. &#xA;0(72k),1(13k)and 2(13K) in brackets are there count.&#xA;So whenever i try to predict them with any algorithm ,i observed that almost all the &quot;2&quot;'s are predicted as &quot;0&quot;.&#xA;On little exploration i found that there are some rows where attributes of 0 and 2 are exactly same.&#xA;Any technique to tackle this issue?&lt;/p&gt;&#xA;" OwnerUserId="41491" LastEditorUserId="41491" LastEditDate="2017-11-12T03:21:33.840" LastActivityDate="2017-11-13T10:17:20.197" Title="Imbalanced class with same rows?" Tags="|python|data|class-imbalance|" AnswerCount="2" CommentCount="3" ClosedDate="2017-11-16T13:14:17.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="24616" PostTypeId="2" ParentId="24610" CreationDate="2017-11-12T03:20:11.627" Score="1" Body="&lt;p&gt;&lt;br&gt;&#xA;I am pretty sure that the SMOTE package in python can also be used for multi-class as well.&lt;br&gt;&#xA;Just you can check its implementation for iris dataset-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/plot_ratio_usage.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/plot_ratio_usage.html&lt;/a&gt;&lt;br&gt;&#xA;-Please correct me if I am wrong.&lt;/p&gt;&#xA;" OwnerUserId="41491" LastEditorUserId="29169" LastEditDate="2019-06-07T21:50:10.907" LastActivityDate="2019-06-07T21:50:10.907" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24619" PostTypeId="2" ParentId="24614" CreationDate="2017-11-12T05:04:48.767" Score="1" Body="&lt;p&gt;The n_neurons (LSTM_nums) on the first input layer should match those of n_features or in your case data_dim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further reading: &lt;a href=&quot;http://www.jakob-aungiers.com/articles/a/LSTM-Neural-Network-for-Time-Series-Prediction&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM Neural Network for Time Series Prediction&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27962" LastActivityDate="2017-11-12T05:04:48.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24620" PostTypeId="1" CreationDate="2017-11-12T05:10:35.503" Score="-2" ViewCount="526" Body="&lt;p&gt;I want to increase proportional increase weightage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have weights&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w1 = 0.4&#xA;w2 = 0.3&#xA;w3 = 0.2&#xA;w4 = 0.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with a constraint that the total sum of the weights should always be 1 i.e &lt;code&gt;w1 + w2 + w3 + w4 = 1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, w4 has been removed. I want to proportionally increase w1, w2 and w3. so that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w1 -&amp;gt; 0.44&#xA;w2 -&amp;gt; 0.33&#xA;w3 -&amp;gt; 0.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, the total sum of the weights should always be 1 i.e &lt;code&gt;w1 + w2 + w3 = 1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to do it mathematically?&lt;/p&gt;&#xA;" OwnerUserId="41761" LastActivityDate="2017-11-12T10:38:48.897" Title="How to redistribute weightage proportionally?" Tags="|feature-scaling|weighted-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24622" PostTypeId="2" ParentId="24607" CreationDate="2017-11-12T05:58:56.167" Score="1" Body="&lt;p&gt;I liked the way you put across your question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think we cannot cannot say in specific will work well with data, it is most likely trial &amp;amp; error method, If ARIMA is not performing well and assuming that there is no trend in data then you can use &lt;strong&gt;AR, Exponential Smoothening&lt;/strong&gt;. These are basic techniques but as you know in many scenarios basic models can explain better than complex models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These both works well in such scenarios. Give a try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be nice if you can share some graphs which can explain us a bit more about the how the data is, noise etc.&#xA;I mean if you have anything from your exploratory analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="41242" LastEditDate="2017-12-04T09:32:46.903" LastActivityDate="2017-12-04T09:32:46.903" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24623" PostTypeId="2" ParentId="989" CreationDate="2017-11-12T06:02:43.687" Score="2" Body="&lt;p&gt;Try normalising the data to [-1,1]. I faced a similar problem and upon normalisation everything worked fine. You can normalise data easily using:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import preprocessing&#xA;X_train = preprocessing.scale(X_train)&#xA;X_test = preprocessing.scale(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41764" LastEditorUserId="29169" LastEditDate="2021-02-07T20:23:17.633" LastActivityDate="2021-02-07T20:23:17.633" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24624" PostTypeId="2" ParentId="24620" CreationDate="2017-11-12T07:15:40.307" Score="3" Body="&lt;p&gt;Let $v_i$ be the updated $w_i$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then $$v_i = \frac{w_i}{\sum_{j\neq 4}w_j}=\frac{w_i}{1-w_4}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if $w_j$ is removed, then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_i =\frac{w_i}{1-w_j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that if $w_j=1$, then all the remaining $w_i=0$, $i \neq j$.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2017-11-12T10:38:48.897" LastActivityDate="2017-11-12T10:38:48.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24626" PostTypeId="2" ParentId="22968" CreationDate="2017-11-12T09:38:55.237" Score="0" Body="&lt;p&gt;The answer depends on what exactly outlier detector you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;EllipticEnvelope&lt;/code&gt; and &lt;code&gt;IsolationForest&lt;/code&gt; authomatically adjust to the scale and correlations between variables, so you cannot easily change relative weights of features. With &lt;code&gt;IsolationForest&lt;/code&gt; you can increase importance of B by just replicating this column several times and so icreasing its relative frequency in the tree splits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other algorithms, such as &lt;code&gt;LocalOutlierFactor&lt;/code&gt; and kernel-based &lt;code&gt;OneClassSVM&lt;/code&gt; are explicitly based on the Euclidean distance between the points. And distance depends on feature scaling. It means that you can multiply B by some factor after normalization, and its importance will increase by this factor.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-12T09:38:55.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24627" PostTypeId="2" ParentId="24615" CreationDate="2017-11-12T09:45:45.830" Score="1" Body="&lt;p&gt;The further exploration of your data would help. Are there, for example, some clusters where relative frequencies of your classes are much different from average?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I see two ways to increase sensitivity of your algorithm toward class 2:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use probabilistic prediction. Maybe what you really need is to estimate the probability that the current example belongs to class 2. If identical observations are sometimes marked as 0 and sometimes as 2, you cannot do much better then say &quot;with probability X this is the second class&quot;.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;class_weights&lt;/code&gt; and increase them for class 2. This will lead to more predictions of 2 and less predictions of 0 and 1. With imbalansed datasets it sometimes helps.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-12T09:45:45.830" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24628" PostTypeId="2" ParentId="24599" CreationDate="2017-11-12T09:51:21.930" Score="1" Body="&lt;p&gt;To use a single vector (p1x, p1y, p1z, p2x, p2y....) per person would be the most easy and obvious way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe, your prediction would also improve by including some other transformations of these features. But usually trees are eager to use all the information you feed them efficiently. So just a 9-dimensional vector would do.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-12T09:51:21.930" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24630" PostTypeId="2" ParentId="24595" CreationDate="2017-11-12T11:58:32.417" Score="0" Body="&lt;p&gt;The sklearn implementation of Lasso that can force non-negative weights (as in &lt;a href=&quot;https://datascience.stackexchange.com/a/19791/24162&quot;&gt;this answer&lt;/a&gt;) is based on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Coordinate_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;coordinate descent&lt;/a&gt; algorithm. You can reimplement it, using for example coordinate-wise &lt;a href=&quot;https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Newton method&lt;/a&gt;. For simplicity, I did not inclide intercept into the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;# generate the data&#xA;np.random.seed(1)&#xA;n = 1000&#xA;X = np.random.normal(size=(n, 5))&#xA;y = np.dot(X, [0,0,1,2,3]) + np.random.normal(size=n, scale=3)&#xA;# initial solution (with some negative weights):&#xA;beta = np.dot(np.linalg.inv(np.dot(X.transpose(),X)), np.dot(X.transpose(), y))&#xA;print(beta)&#xA;# clip the solution from below with zero&#xA;prev_beta = beta.copy()&#xA;beta = np.maximum(beta, 1)&#xA;# improve the solution by restricted coordinate-wise Newton descent&#xA;hessian = np.dot(X.transpose(), X)&#xA;while not (prev_beta == beta).all():&#xA;    prev_beta = beta.copy()&#xA;    for i in range(len(beta)):&#xA;        grad = np.dot(np.dot(X,beta)-y, X)&#xA;        beta[i] = np.maximum(0, beta[i] - grad[i] / hessian[i,i])&#xA;print(beta)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code will output initial and final beta's:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[-0.01404546 -0.02633036  1.06028543  1.99696564  2.93511618]&#xA;[ 0.          0.          1.05919989  1.99673774  2.93442334]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see that OLS beta differ from your optimal beta not only in the first two coefficients (that have been negative), but the rest of coefficients were also adjusted.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-12T11:58:32.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24631" PostTypeId="1" CreationDate="2017-11-12T12:59:14.783" Score="1" ViewCount="972" Body="&lt;p&gt;When I read several statistical papers, they mention &quot;oracle&quot; property or &quot;oracle&quot; estimator.&#xA;What do they mean by &quot;oracle&quot;?&#xA;I understand this oracle is not a company name, but have no idea what this means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example,&#xA;Candes and Tao 2007 is about &quot;oracle&quot; inequality in a context of large dimensional estimation of linear model (p&gt;&gt;n).&#xA;&lt;a href=&quot;https://projecteuclid.org/euclid.aos/1201012958&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://projecteuclid.org/euclid.aos/1201012958&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Zou 2006 discusses &quot;oracle&quot; property of Lasso. &#xA;(&lt;a href=&quot;http://www.tandfonline.com/doi/abs/10.1198/016214506000000735&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.tandfonline.com/doi/abs/10.1198/016214506000000735&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not familiar with this field but I have to read one paper which discusses high dimensional test in statistics.&lt;/p&gt;&#xA;" OwnerUserId="41775" LastEditorUserId="29575" LastEditDate="2017-11-13T08:00:43.777" LastActivityDate="2017-11-13T08:00:43.777" Title="What is &quot;oracle&quot; in statistics?" Tags="|statistics|parameter-estimation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24633" PostTypeId="1" AcceptedAnswerId="24654" CreationDate="2017-11-12T15:21:38.383" Score="-1" ViewCount="584" Body="&lt;p&gt;I'm trying to predict the winner of a race, when given 2 sets of features. The data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[[ 1.        ],&#xA;        [ 1.        ],&#xA;        [ 0.        ],&#xA;        [ 0.10447761],&#xA;        [ 0.18892762],&#xA;        [ 0.76190472],&#xA;        [ 1.        ],&#xA;        [ 0.0338983 ],&#xA;        [ 1.        ],&#xA;        [ 0.41789967]],&#xA;&#xA;       [[ 1.        ],&#xA;        [ 1.        ],&#xA;        [ 0.        ],&#xA;        [ 0.10447761],&#xA;        [ 0.18892762],&#xA;        [ 0.        ],&#xA;        [ 0.74712646],&#xA;        [ 0.        ],&#xA;        [ 0.44444448],&#xA;        [ 0.53915548]],&#xA;&#xA; ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first 5 entries are features of racer #1 and entries 6-10 are the same features for racer #2. In addition, I have the results of the output neuron (0 or 1) which tells me which winner is expected to win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest the following model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A 1D convolutional neural network with a kernel size of 5 and s stride size of 5. This is then followed by some flattening and fully connected layers. My results are very bad and I would like to know if there is conceptionally something wrong or if it's just the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts behind this is that it's like a convolution on an image, with a window size equivalent to the amount of features of each racer. On each of the two convolution, a filter set is then applied, which is equivalent for both of the two. The fully connected layer will then determine if the first or the second feature set is the one to be selected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any views if a convolutional layer can be used in that way are appreciated. Here the code snipped:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv1D(filters=64,&#xA;                 kernel_size=5,&#xA;                 strides=5,&#xA;                 input_shape=input_shape,&#xA;                 activation='relu'))&#xA;model.add(Dropout(0.2))&#xA;model.add(Flatten())&#xA;model.add(Dense(256, activation='relu', kernel_constraint=maxnorm(3)))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(128, activation='relu', kernel_constraint=maxnorm(3)))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(1, activation='softmax'))&#xA;&#xA;model.compile(loss=sparse_categorical_crossentropy,&#xA;              optimizer=Adam(),&#xA;              metrics=['accuracy'])&#xA;&#xA;model.fit(self.trainX,&#xA;          self.trainY,&#xA;          epochs=epochs,&#xA;          batch_size=1,&#xA;          verbose=1,&#xA;          callbacks=[tbCallBack, early_stop])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32860" LastEditorUserId="29169" LastEditDate="2021-02-15T22:20:45.590" LastActivityDate="2021-02-15T22:20:45.590" Title="Conv1D to predict winner of two feature sets" Tags="|neural-network|keras|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24635" PostTypeId="2" ParentId="24609" CreationDate="2017-11-12T17:06:57.103" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;the problem of excel being very slow in finding the value and it is&#xA;  very difficult to read information in the row&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What is &quot;slow&quot; for you (in seconds) and how fast would you like the search to be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a lot of cases, the performance can be considerably improved by &lt;a href=&quot;https://www.google.com/search?q=INDEX%20and%20MATCH%20Excel&amp;amp;oq=INDEX%20and%20MATCH%20Excel&amp;amp;aqs=chrome..69i57.7520j0j1&amp;amp;sourceid=chrome&amp;amp;ie=UTF-8&quot; rel=&quot;nofollow noreferrer&quot;&gt;using &lt;code&gt;INDEX&lt;/code&gt; and &lt;code&gt;MATCH&lt;/code&gt;&lt;/a&gt; to search for values in Excel, instead of using &lt;code&gt;VLOOKUP&lt;/code&gt; (which is known to be much slower in comparison).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could &lt;a href=&quot;https://www.microsoft.com/en-us/sql-server/sql-server-editions-express&quot; rel=&quot;nofollow noreferrer&quot;&gt;install SQL Server Express&lt;/a&gt; on your machine, import the Excel file into a table, and use some SQL statements to find the values you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Power BI is a visualization tool, and doesn't seem appropriate for what you're describing which is &lt;em&gt;preparing the data&lt;/em&gt;. You could use Power BI after, of course, to plot the final results.&lt;/p&gt;&#xA;" OwnerUserId="9935" LastEditorUserId="9935" LastEditDate="2017-11-13T20:32:54.427" LastActivityDate="2017-11-13T20:32:54.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24636" PostTypeId="1" CreationDate="2017-11-12T17:54:46.380" Score="1" ViewCount="106" Body="&lt;p&gt;I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the Gensim api, the method returns an array of log probabilities, with values ranging from zero to negative infinity per document&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to sort test documents of different lengths based on their probability to the model. I️ am new to log probabilities, and wasn’t sure if I️ need to sum, average, or grab the max value for my purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I make use of the log probabilities?&lt;/p&gt;&#xA;" OwnerUserId="41785" LastEditorUserId="29575" LastEditDate="2017-11-13T16:32:58.423" LastActivityDate="2017-11-13T16:32:58.423" Title="What do I️ do with an array of log probabilities? Doc2vec" Tags="|machine-learning|python|neural-network|gensim|similar-documents|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24637" PostTypeId="1" AcceptedAnswerId="24649" CreationDate="2017-11-12T19:41:55.017" Score="1" ViewCount="2899" Body="&lt;p&gt;I have a time series data about daily usage of a computer program, here is an example&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2017-11-10: &lt;strong&gt;0&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-09: &lt;strong&gt;14&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-08: &lt;strong&gt;0&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-07: &lt;strong&gt;6&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-06: &lt;strong&gt;102&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-05: &lt;strong&gt;0&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;2017-11-04: &lt;strong&gt;0&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you can see 11-06 has a spike at 102. Due to our way of gathering this data, we know that data is probably erroneous and we are sure that 102 is not correct according other values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we need to clean these dirty values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a mathematical way to do this?&#xA;Is there a python lib to help us?&lt;/p&gt;&#xA;" OwnerUserId="28917" LastActivityDate="2019-05-06T14:24:43.990" Title="Cleaning time series data" Tags="|data-cleaning|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24638" PostTypeId="2" ParentId="24637" CreationDate="2017-11-12T20:39:34.840" Score="1" Body="&lt;p&gt;One solution is using &lt;code&gt;mean&lt;/code&gt; and &lt;code&gt;variance&lt;/code&gt; to detect outlires in your time-series. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; data=np.array([0,0,102,6,0,14,0])&#xA;&amp;gt;&amp;gt; c = 1&#xA;&amp;gt;&amp;gt; abs(data - np.mean(data)) &amp;lt; c * np.std(data)&#xA;Output: array([ True,  True, False,  True,  True,  True,  True], dtype=bool)&#xA;&amp;gt;&amp;gt; clean_data= data[abs(data - np.mean(data)) &amp;lt; c * np.std(data)]&#xA;Output: array([ 0,  0,  6,  0, 14,  0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can play with &lt;code&gt;c&lt;/code&gt; based on your requirement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, instead of using mean and variance of all the data, you can use this method for each section of your time-series separately (e.g. every 30 days). Because there might be different behavior in different time-intervals.&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-11-12T20:39:34.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24639" PostTypeId="2" ParentId="24637" CreationDate="2017-11-12T21:17:38.580" Score="1" Body="&lt;p&gt;Here is what I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.cluster import MeanShift, estimate_bandwidth&#xA;&#xA;x = [0,14,0,6,102,0,0]&#xA;&#xA;X = list(zip(x,np.zeros(len(x))))&#xA;bandwidth = estimate_bandwidth(X, quantile=0.2)&#xA;ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)&#xA;ms.fit(X)&#xA;labels = ms.labels_&#xA;cluster_centers = ms.cluster_centers_&#xA;&#xA;labels_unique = np.unique(labels)&#xA;n_clusters_ = len(labels_unique)&#xA;&#xA;X = np.array(X)&#xA;for k in range(n_clusters_):&#xA;    my_members = labels == k&#xA;    print(k, X[my_members, 0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_mean_shift.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/cluster/plot_mean_shift.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28917" LastEditorUserId="28917" LastEditDate="2017-11-12T21:48:10.230" LastActivityDate="2017-11-12T21:48:10.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24640" PostTypeId="2" ParentId="24637" CreationDate="2017-11-12T21:23:49.177" Score="1" Body="&lt;p&gt;I would use the Interquartile range (&lt;span class=&quot;math-container&quot;&gt;$IQR$&lt;/span&gt;), where the outliers are the values larger than &lt;span class=&quot;math-container&quot;&gt;$Q3+1.5 \times IQR$&lt;/span&gt;, and the values less than &lt;span class=&quot;math-container&quot;&gt;$Q1-1.5 \times IQR$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$Q1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Q3$&lt;/span&gt; are the first and third quartiles, respectively. &lt;a href=&quot;https://www.khanacademy.org/math/statistics-probability/summarizing-quantitative-data/box-whisker-plots/a/identifying-outliers-iqr-rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a good example. &lt;/p&gt;&#xA;" OwnerUserId="41240" LastEditorUserId="25180" LastEditDate="2019-05-06T12:49:19.453" LastActivityDate="2019-05-06T12:49:19.453" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24643" PostTypeId="1" CreationDate="2017-11-12T22:49:34.447" Score="-1" ViewCount="82" Body="&lt;p&gt;I am modeling a binary classification and my loss function is the gini function (normalized area under the curve). Here's my implementation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split the data with k-folds&lt;/li&gt;&#xA;&lt;li&gt;Train k classifiers&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now I have k classifiers, but I need one classifier. So the naive approach is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$prediction_i = \frac{prediction_{i1} + prediction_{i2} + ... + prediction_{ik} }{k}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may be possible problems with this combining technique. For example, gini is scale invariant. I could take prediction1 and scale it with exp(p_i) and then scale prediction2 with sqrt(p_i). These would have zero affect on scoring the individually but it would mess up my combining step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the most appropriate combining function?&lt;/p&gt;&#xA;" OwnerUserId="35930" LastActivityDate="2017-11-13T04:32:28.303" Title="How to average classifiers with AUC metric?" Tags="|predictive-modeling|model-selection|ensemble-modeling|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24644" PostTypeId="1" AcceptedAnswerId="24645" CreationDate="2017-11-12T23:09:20.443" Score="6" ViewCount="341" Body="&lt;p&gt;I'm currently working for a health company in Brazil, and I want to create a Predictive model that would find users that are one step close to book an appointment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We currently have 1 million visits per month, and 60% of this visits are not booking anything (exam, consult, checkups, etc). I believe that with ML, we could find which users would book if we make a call or send an email.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been a professional web development for the past 7 years, but I don't have the skills to create a Predictive model, so I'm looking for directions, a way to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We do have Google Analytics, but I'm thinking to create a simple software that would track:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Age&#xA;Gender&#xA;Which pages the user visited&#xA;How many time the user expended in each page&#xA;Name&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I could segregate the users in two groups: the ones that booked, and the users that didn't book, and then... I'm stuck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone give a direction? Should I create a software to track this information or use GA? I want create a demo to get financing, so I don't have much time :/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerDisplayName="user41790" LastActivityDate="2017-11-13T02:43:04.870" Title="Create a Predictive model to find users that would book an exam" Tags="|machine-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24645" PostTypeId="2" ParentId="24644" CreationDate="2017-11-12T23:49:04.417" Score="4" Body="&lt;p&gt;Your problem is a kind of Binary Classification which predicts the probability of a target variable to be Yes(1) or No(0).&#xA;But, first of all, you need some ground-truth data, because most of the state-of-the-art solutions to this problem are supervised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're the admin of this system or have access to their server, you can collect these data by logging everything you need from every user and label their traces with 1 and 0 when they book or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After a while, you have a big labeled dataset and you can normalize and shape your inputs in a way that it fits for example to a deep neural network for prediction.&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-11-12T23:49:04.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24646" PostTypeId="2" ParentId="24631" CreationDate="2017-11-13T02:15:18.437" Score="3" Body="&lt;p&gt;&quot;Oracle&quot; refers to something that has access to the ground truth. It has the perfect information of which in practice, we rarely have some luxury. In such paper, you can typically see that it reflects a comparison with a model with perfect information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From Candes and Terry Tao's paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;Even though $n$ may be much smaller than $p$, our estimator achieves a loss within a logarithmic factor of the &lt;strong&gt;ideal&lt;/strong&gt; mean squared error one would achieve with an &lt;strong&gt;oracle&lt;/strong&gt; which would supply &lt;strong&gt;perfect information&lt;/strong&gt; about which coordinates are nonzero, and which were above the noise level.&lt;/li&gt;&#xA;&lt;li&gt;To see why this is true, suppose one had available an &lt;strong&gt;oracle letting us know in advance&lt;/strong&gt; the location of the S nonzero entries of the parameter vector&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From Hai Zhou's paper:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;We show that the adaptive lasso enjoys the &lt;strong&gt;oracle properties&lt;/strong&gt;; namely, it performs as well as if the &lt;strong&gt;true underlying model were given in advance&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Notice that oracle procedure/properties is also defined in Hai Zhou's paper reflecting the ability to rediscover the ground truth. &lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2017-11-13T02:15:18.437" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24647" PostTypeId="2" ParentId="24644" CreationDate="2017-11-13T02:22:33.420" Score="2" Body="&lt;p&gt;In addition to Moh's answer, which is impossible to disagree with, I understand you have limited time to prototype a binary classification model. Given that you don't have much experience coding machine learning models, right after you collect your data and label your dependent variable with 1 and 0, you should go for quick prototyping tools such as Orange, Weka, RapidMiner, SAS Enterprise Miner etc. These are point-and-click platforms that require little to no experience in machine learning. Very easy learning curve, you can build a proof-of-concept very easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later on, depending on your needs and limitations, you can stick with these easy-to-use tools, or move to R and Python, and code your classification model.&lt;/p&gt;&#xA;" OwnerUserId="20572" LastActivityDate="2017-11-13T02:22:33.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24648" PostTypeId="2" ParentId="24644" CreationDate="2017-11-13T02:43:04.870" Score="2" Body="&lt;p&gt;It's hard to answer your question because it is too general. It's like asking how to build a website without even knowing how to do programming. But I will be constructive &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your background and assuming you are familiar with Python, I recommend you using &lt;a href=&quot;https://automl.github.io/auto-sklearn/stable/#&quot; rel=&quot;nofollow noreferrer&quot;&gt;auto-sklearn&lt;/a&gt;. &lt;code&gt;auto-sklearn&lt;/code&gt; is a library based on the popular &lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; which automates a lot of steps involved in general machine learning, like feature selection and hyperparameter-tuning. It also tries a bunch of algorithms and ensemble them to produce the best result. It is hard to beat its performs as a machine learning beginner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While &lt;code&gt;auto-sklearn&lt;/code&gt; is very good for a beginner to get started, I must warn you that, without fully understand the how and why it is extremely risky for you rely on it blindly. One thing of particular importance is you need to understand how to evaluate the performance of a machine learning algorithm. Please make sure you understand the &lt;a href=&quot;https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set&quot;&gt;difference between train, validation and test dataset&lt;/a&gt;, before you make any claim on how good/accurate you prediction is. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-13T02:43:04.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24649" PostTypeId="2" ParentId="24637" CreationDate="2017-11-13T04:15:00.247" Score="3" Body="&lt;p&gt;I think you have a few options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have a pre-set rule to exclude outliers, such as a hard-threshold at 100 which you know the data shouldn't exceed, then something as simple as &lt;code&gt;x = [e for e in x if e &amp;lt; 100]&lt;/code&gt; will do.&lt;/li&gt;&#xA;&lt;li&gt;If you have a parametric belief, such as any observation that falls beyond so many standard deviations from mean, or quartiles, are outliers; then you can implement the other answers that have been mentioned.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Else, you can go for a clustering approach. Here I believe your first shot should be a k-means clustering. This is super easy to build and interpret. See my code below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x = [0,14,0,6,102,0,0]&#xA;from sklearn.cluster import KMeans&#xA;kmeans = KMeans(n_clusters=2).fit(np.array(x).reshape(-1, 1))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;#First cluster:&#xA;np.array(x)[np.where(kmeans.labels_ == 0)]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;#Second cluster (outliers):&#xA;np.array(x)[np.where(kmeans.labels_ == 1)]&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;K-means is known to be sensitive to outliers, hence a more robust method such as MeanShift, which you tried, is a good rival to k-means. I would run both, and stick with the result that makes better sense to me.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="20572" LastActivityDate="2017-11-13T04:15:00.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24650" PostTypeId="2" ParentId="24643" CreationDate="2017-11-13T04:32:28.303" Score="1" Body="&lt;h1&gt;Answer&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In ensemble methods, the predictions are typically made by majority voting. Using python and Numpy.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;pred1= [.5, .5 , .0]&#xA;pred2= [.3, .2 , .5]&#xA;pred3= [.5, .1 , .4]&#xA;&#xA;pred1 = (pred1 == np.max(pred1)).astype(np.int)&#xA;pred2 = (pred2 == np.max(pred2)).astype(np.int)&#xA;pred3 = (pred3 == np.max(pred3)).astype(np.int)&#xA;votes = pred1 + pred2 + pred3 # == array([2, 1, 1])&#xA;pred = np.argmax(votes) # pred == 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Side Note&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Also, K-Fold is generally used for hyper-parameter tuning. After you made a decision on the hyper-parameter you would refit the model on the entire dataset. If you want to use ensemble methods like random forest or gradient boosted tree, you would apply the algorithm to the entire dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear to me whether your method of ensembling will have any benefit over the standard approach. In your approach, each classifier is trained on the smaller set of data on all features available, and I can't think of any reason for us to do this. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-13T04:32:28.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24651" PostTypeId="1" AcceptedAnswerId="24655" CreationDate="2017-11-13T04:41:23.223" Score="0" ViewCount="2455" Body="&lt;p&gt;I am building a document classifier using Naive Bayes. There are 10 classes. My question is that :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Should each class contain the same number of documents for training?&#xA;  What if the number of training example in each class is different?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Does the number of classes and classification algorithm have any relation? say is there any thump rule like if there are 100 classes shall algorithm 'X' has better performance than 'Y'&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-11-13T09:35:27.623" Title="Bias in Naive Bayes classifier" Tags="|python|classification|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24654" PostTypeId="2" ParentId="24633" CreationDate="2017-11-13T06:24:04.983" Score="2" Body="&lt;p&gt;This methodology seems a bit strange and potentially overkill for the problem. I would try having your input just be a 5 dimensional vector that is the difference between racer 1 and racer 2 and have the output just be the result for racer 1 (or what every racer's features are being subtracted from the other racer's features.). From here you can just use a few fully connected layers (which is essentially what you are doing be having kernel's of size 5 anyway. I would also try some more traditional machine learning algorithms with this type of 1D input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: since signs are important for the difference between your two racer features, do not use a relu as your activation function for this approach.&lt;/p&gt;&#xA;" OwnerUserId="33922" LastActivityDate="2017-11-13T06:24:04.983" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24655" PostTypeId="2" ParentId="24651" CreationDate="2017-11-13T07:30:24.233" Score="3" Body="&lt;p&gt;&lt;strong&gt;Unbalanced class distributions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, unbalanced datasets will cause your model to have a bias towards the over-represented classes. If the distribution of the classes is not very drastic then this should not cause a significant problem with any algorithm you will employ. However, as the difference between the class distribution becomes more severe you should expect to get higher false negatives for that class. Consider this, you are trying to have the model adequately identify what it means for a specific example to belong to a class. If you do not provide sufficient examples, then the model will not be able to understand the extent of the variation which exists among the examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the class distribution is very different, then I would suggest anomaly detection techniques. These techniques allow you to learn the distribution of a single classification and then identify when novel examples fall within this distribution or not.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choosing an algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More classes will result in a higher dimensional output, thus contributing to the complexity of your model. For example, if you have a model which discriminates between 2 classes with a set dataset size. Then further discrimination (increasing the number of output classes) will cause the model to have higher bias. You should thus expect to see greater test error if you do not increase the size of your dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a set dataset X. Then you need to find the correct balance between bias and model complexity to get the optimal results. For example, a neural network based technique (highly complex) is not a good algorithm to use for a limited dataset with many output classes. However, Naive Baye's or Random Forest would be. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-11-13T07:30:24.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24656" PostTypeId="1" CreationDate="2017-11-13T07:45:19.650" Score="0" ViewCount="262" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pig_Latin&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pig-Latin&lt;/a&gt;, and other similair &lt;a href=&quot;https://en.wikipedia.org/wiki/Argot&quot; rel=&quot;nofollow noreferrer&quot;&gt;argots&lt;/a&gt;, were quite popular when i was young (but since i grew up in Sweden the biggest was &lt;a href=&quot;https://en.wikipedia.org/wiki/R%C3%B6varspr%C3%A5ket&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rövarspråket&lt;/a&gt;). These language games almost always consist of a given set of rules from which letters and words are translated into something else. For example in Pig-Latin the rules are the following (&lt;a href=&quot;https://en.wikipedia.org/wiki/Pig_Latin#Rules&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;): &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For words that begin with consonant sounds, all letters before the&#xA;  initial vowel are placed at the end of the word sequence. Then, &quot;ay&quot;&#xA;  is added, as in the following examples:[10]&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&quot;pig&quot; = &quot;igpay&quot;&lt;/li&gt;&#xA;  &lt;li&gt;&quot;latin&quot; = &quot;atinlay&quot;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;When words begin with consonant clusters (multiple consonants that&#xA;  form one sound), the whole sound is added to the end when speaking or&#xA;  writing.[11]&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&quot;cheers&quot; = &quot;eerschay&quot;&lt;/li&gt;&#xA;  &lt;li&gt;&quot;shesh&quot; = &quot;eshshay&quot;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;For words that begin with vowel sounds, one just adds &quot;way&quot; or &quot;yay&quot;&#xA;  to the end (or just &quot;ay&quot;). Examples are:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&quot;eat&quot; = &quot;eatway&quot; or &quot;eatay&quot;&lt;/li&gt;&#xA;  &lt;li&gt;&quot;omelet&quot; = &quot;omeletway&quot; or &quot;omeletay&quot;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;An alternative convention for words beginning with vowel sounds, one&#xA;  removes the initial vowel(s) along with the first consonant or&#xA;  consonant cluster. This usually only works for words with more than&#xA;  one syllable and offers a more unique variant of the words in keeping&#xA;  with the mysterious, unrecognizable sounds of the converted words.&#xA;  Examples are:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&quot;every&quot; = &quot;eryevay&quot;&lt;/li&gt;&#xA;  &lt;li&gt;&quot;omelet&quot; = &quot;eletomay&quot;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My question is if there are any Machine Learening algorithms that is suitable to be able to learn to translate a sentence to Pig-Latin by training on a large set of &lt;code&gt;&quot;english&quot;=&quot;pig_latin&quot;&lt;/code&gt;-pairs without explicity knowing anything about the rules? Or if anybody can motivate and point me to the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to be able to input an arbitrary word, which might not even be a real word, and get the &quot;Pig-Latin&quot;-translated word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial idea was to set up a Neural Network (mainly because thats what I know best) where the input neurons should take the ASCII-code for each letter and the output should be the ASCII-code for each letter (always capital letters), i.e. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input (for PIG):&#xA;080 073 071&#xA;&#xA;Output (for IGPAY)):&#xA;073 071 080 065 089&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A neural network can be trained to map for example 080 to 073, but how do i for example handle the aribtrary length of the input and output (if Neural Networks ar applicable to this problem).&lt;/p&gt;&#xA;" OwnerUserId="33334" LastEditorUserId="33334" LastEditDate="2017-11-13T08:06:19.177" LastActivityDate="2017-11-13T08:45:17.667" Title="Machine Learning algorhtm that can be trained to learn Pig-Latin" Tags="|neural-network|algorithms|data|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24657" PostTypeId="1" CreationDate="2017-11-13T07:48:04.283" Score="1" ViewCount="4950" Body="&lt;p&gt;How can we apply K-fold cross validation on say linear regression? Regression contains weight update and iterations, and so do we integrate K-fold to it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If done, is it like we take the first K-1 fold to train using linear regression and perform iterations to obtain weight, then take the next fold and proceed the same and so on until all folds are done and pick the best &lt;strong&gt;weight&lt;/strong&gt; or &lt;strong&gt;fold&lt;/strong&gt; from it?&lt;/p&gt;&#xA;" OwnerUserId="41801" LastEditorUserId="13748" LastEditDate="2017-11-13T11:38:30.263" LastActivityDate="2019-05-08T20:45:07.790" Title="Apply cross validation on regression algorithms" Tags="|linear-regression|cross-validation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24658" PostTypeId="1" CreationDate="2017-11-13T07:49:43.700" Score="0" ViewCount="607" Body="&lt;p&gt;I am currently working on a question categorization problem where I automatically want to assign a category to the question. The question set I have is unlabelled. The categories for the problem are well defined e.g. Business, Technology, Sports etc. I also have a specific set of words which are important for each of the category (which have been taken from the dataset) and which define the respective categories. Using these information, which learning algorithm can I use wherein I can specify these categories and the important words for that category in context of the unlabelled data that I have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Repsonses are highly valued. &#xA;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="26009" LastActivityDate="2018-06-11T14:50:30.183" Title="Automatic question categorization when we know important words in each category" Tags="|nlp|text-mining|unsupervised-learning|topic-model|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24659" PostTypeId="2" ParentId="24656" CreationDate="2017-11-13T08:38:11.647" Score="1" Body="&lt;p&gt;Look at Neural Machine Translation models, you could either 'cheat' and tokenize both the English and the Pig-Latin words or you could make it character based which I think is more interesting and it allows to generalize for words out of dictionary. These models are called sequence to sequence models and there are plenty of implementations that can take pairs of sequences to translate. I think this task can be solved with accuracy close to 100%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an implementation in Keras: &lt;a href=&quot;https://github.com/farizrahman4u/seq2seq&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/farizrahman4u/seq2seq&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an official TensorFlow tutorial: &lt;a href=&quot;https://github.com/tensorflow/nmt&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/nmt&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-11-13T08:45:17.667" LastActivityDate="2017-11-13T08:45:17.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24660" PostTypeId="1" AcceptedAnswerId="24671" CreationDate="2017-11-13T09:03:12.827" Score="3" ViewCount="7109" Body="&lt;p&gt;I am new to deep learning. Can anybody help me with the online learning implimentation for deep learning models. As per my understanding, i can save a keras/tensorflow model after training and when new data comes in, i can reload the network back and retrain the network using the new data. I haven’t seen anywhere documenting this method. Is my understanding incorrect? If yes, let me know what can be done so that the model keeps on getting retrained when new data comes in? &lt;/p&gt;&#xA;" OwnerUserId="30693" LastEditorUserId="30693" LastEditDate="2017-11-13T09:53:41.873" LastActivityDate="2018-06-12T01:59:20.133" Title="How to retrain the neural network when new data comes in?" Tags="|neural-network|deep-learning|online-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24662" PostTypeId="2" ParentId="24651" CreationDate="2017-11-13T09:35:27.623" Score="1" Body="&lt;p&gt;If the data is unbalanced then the classifier would give more weight-age that set of group which has higher number of records. So it is obvious that the outcome would be biased towards that group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To overcome such issues we balance the data before hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can use Naive Bayes Classifier, it works based on the probability.&#xA;Since your problem is document classification, Naive Bayes might give you good result, as you know in most of the scenarios simple models gives best results in complex scenarios. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other classifiers which you can try are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Random Forest&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decision Trees&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;SVM.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Most likely either of them would give you the desired result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understand there is no thumb as such because an algorithm can be used in many different ways. To put in bluntly, it is trail and error, very subjective to the business problem. So, please be careful when you finalize your model.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-13T09:35:27.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24664" PostTypeId="2" ParentId="24610" CreationDate="2017-11-13T10:10:58.040" Score="1" Body="&lt;p&gt;Yes that is what SMOTE does, even if you do manually also you get the same result or if you run an algorithm to do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are couple of other techniques which can be used for balancing multiclass feature.&#xA;Attaching those 2 links for your reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/applications/plot_multi_class_under_sampling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link 1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/11404/python-handling-imbalance-classes-in-python-machine-learning&quot;&gt;Link 2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link 3 is having implementation of couple of oversampling techniques:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn/tree/master/examples/over-sampling&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link 3&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ROSE also can be used for Oversampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful.&#xA;Do let me know of you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-16T09:39:48.400" LastActivityDate="2017-11-16T09:39:48.400" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24665" PostTypeId="2" ParentId="24615" CreationDate="2017-11-13T10:17:20.197" Score="1" Body="&lt;p&gt;Here you need to do more exploration on data, so that you find some unique feature which helps us in understanding why that record has been classified into that particular cluster/segment(on training data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to balance the data so that the model can understand and can predict accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/24162/david-dale&quot;&gt;@David&lt;/a&gt; has mentioned above such situation can be played around by giving weight-age to the classes and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-13T10:17:20.197" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24667" PostTypeId="1" AcceptedAnswerId="24716" CreationDate="2017-11-13T11:39:02.557" Score="0" ViewCount="1180" Body="&lt;p&gt;I have a set of data, that gives the length of a species of abalone, and its corresponding type (male, M, female, F, or infant, I). (&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/abalone&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://archive.ics.uci.edu/ml/datasets/abalone&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have constructed a logistic regression to create a model that will determine whether the abalone is M/F or I, given the length. (M and F are classed as the same.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I write the following in R to generate and test the model on data points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g &amp;lt;- glm(sex ~ length, family=&quot;binomial&quot;)&#xA;pairs &amp;lt;- paste(round(predict(g, type=&quot;response&quot;)), sex)&#xA;table(pairs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output table is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pairs&#xA;0 F  0 I  0 M  1 F  1 I  1 M &#xA;218    6  210 1089 1336 1318&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I correctly interpret this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the options I have come up with:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;1089 females correctly identified, 218 females incorrectly identified; 1318 males correctly identified, 210 males incorrectly identified; 1336 infants correctly identified, 6 incorrectly identified.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;218 females correctly identified, 6 infants and 210 males incorrectly identified; 1089 females incorrectly identified, 1336 infants and 1318 males incorrectly identified.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40736" LastActivityDate="2017-11-14T09:53:08.520" Title="How to correctly interpret the table function to determine logistic regression accuracy?" Tags="|r|regression|logistic-regression|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24668" PostTypeId="2" ParentId="24657" CreationDate="2017-11-13T11:43:37.170" Score="1" Body="&lt;p&gt;How to use cross-validation on regression (assuming 10-fold for example purposes): separate your dataset in 10% and 90%, train on 90%, test your metric (squared error or anything you're modeling) on the remaining 10%. Do that 10 times using different 10% groups. Now you have 10 of your metrics, you can analyze its mean and range to see if the model is robust against over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you pick a model: you train on the whole dataset. Cross-validation is for testing/validating purposes, you don't use the models generated by cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-11-13T11:43:37.170" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24669" PostTypeId="2" ParentId="24660" CreationDate="2017-11-13T11:44:10.380" Score="1" Body="&lt;p&gt;Yes, there is some methods to do this based on the library. The proper keyword for this functionality is &lt;strong&gt;Checkpoint&lt;/strong&gt;. Hence, you can check functions to save or restore the checkpoints from hard disk to continue learning or prediction in a new session. For example you can see the following in Tensorflow:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The tf.train.Saver class provides methods for saving and restoring models. The tf.train.Saver constructor adds save and restore ops to the graph for all, or a specified list, of the variables in the graph. The Saver object provides methods to run these ops, specifying paths for the &lt;strong&gt;checkpoint&lt;/strong&gt; files to write to or read from.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For more details about this in Tensorflow, you can follow this &lt;a href=&quot;https://www.tensorflow.org/programmers_guide/saved_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To complete the case you can follow this subject in another library such as CNTK in &lt;a href=&quot;https://docs.microsoft.com/en-us/cognitive-toolkit/How-do-I-Adapt-Models-in-BrainScript#continue-training-from-a-previously-saved-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastEditorUserId="27665" LastEditDate="2017-11-13T11:49:20.183" LastActivityDate="2017-11-13T11:49:20.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24670" PostTypeId="1" AcceptedAnswerId="24673" CreationDate="2017-11-13T11:44:13.823" Score="0" ViewCount="12738" Body="&lt;p&gt;I am trying to scrape &lt;a href=&quot;http://www.the-numbers.com/movie/budgets/all&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.the-numbers.com/movie/budgets/all&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but when I write the table into a csv file, there is an additional line with the counter index written in between each movie row... how can I get rid of this? I dont understand how that counter line is being written to the csv...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import csv,os&#xA;from bs4 import BeautifulSoup&#xA;from urllib.request import Request, urlopen, URLError&#xA;from selenium import webdriver&#xA;&#xA;counter = 0&#xA;currentDir=os.getcwd()&#xA;filename = currentDir + &quot;\\theNumbersScraper.csv&quot;&#xA;&#xA;pagecount = 1&#xA;headers=['ID',&#xA; 'Release Date',&#xA; 'Movie',&#xA; 'Production Budget',&#xA; 'Domestic Gross',&#xA; 'Worldwide Gross']&#xA;&#xA;with open(filename, 'w' ,newline='\n',encoding='utf-8') as csvfile:&#xA;        #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers&#xA;        #writer.writeheader()    &#xA;        #csvfile = open(filename, 'w', newline='',encoding='utf-8')&#xA;    writer = csv.writer(csvfile,delimiter='|')&#xA;    writer.writerow(headers)&#xA;&#xA;#with open(filename, 'a', newline='',encoding='utf-8') as csvfile:&#xA;    #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers&#xA;    #writer.writeheader()    &#xA;    #csvfile = open(filename, 'w', newline='',encoding='utf-8')&#xA;#    writer = csv.writer(csvfile,delimiter='|')&#xA;&#xA;    while pagecount &amp;lt;5401:&#xA;&#xA;        &quot;&quot;&quot;&#xA;        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1&#xA;        #to http://www.the-numbers.com/movie/budgets/all/5401&#xA;        #so there are 5400 entries&#xA;        &quot;&quot;&quot;&#xA;        request = Request(&quot;http://www.the-numbers.com/movie/budgets/all/&quot;+str(pagecount))&#xA;        request.add_header('User-agent', 'wswp')&#xA;        website = urlopen(request).read().strip()&#xA;&#xA;        soup = BeautifulSoup(website,'lxml')&#xA;&#xA;        &quot;&quot;&quot;#obsolete&#xA;        headertags = soup.find(&quot;table&quot;).find_next(&quot;tr&quot;).find_all(&quot;th&quot;)&#xA;&#xA;        headers= []&#xA;        for line in headertags:&#xA;            headers.append(line.string)&#xA;&#xA;        headers[0] = 'ID'&#xA;&#xA;        &quot;&quot;&quot;&#xA;&#xA;        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1&#xA;        #to http://www.the-numbers.com/movie/budgets/all/5401&#xA;        #so there are 5400 entries&#xA;&#xA;        all_tr = soup.find_all(&quot;tr&quot;)&#xA;        for movie in range(1, len(all_tr)):&#xA;            row=[]&#xA;            counter+=1&#xA;            row.append(counter)&#xA;            td = all_tr[movie].find_all(&quot;td&quot;)            &#xA;            for colIndex in range(1, len(td)):                &#xA;                row.append(td[colIndex].string)&#xA;            writer.writerow(row)&#xA;&#xA;&#xA;        &quot;&quot;&quot;&#xA;        for tr in all_tr:&#xA;            row = []&#xA;            td = tr.find_all(&quot;td&quot;)&#xA;            for i in range(1, 4):&#xA;                row.append(str(counter))&#xA;                row.append(td[i].string)&#xA;            writer.writerow(row)&#xA;            counter+=1&#xA;        &quot;&quot;&quot;&#xA;        pagecount +=100&#xA;&#xA;&#xA;csvfile.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The part of interest is actually this block only:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;all_tr = soup.find_all(&quot;tr&quot;)&#xA;for movie in range(1, len(all_tr)):&#xA;    row=[]&#xA;    counter+=1&#xA;    row.append(counter)&#xA;    td = all_tr[movie].find_all(&quot;td&quot;)            &#xA;    for colIndex in range(1, len(td)):                &#xA;        row.append(td[colIndex].string)&#xA;    writer.writerow(row)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID|Release Date|Movie|Production Budget|Domestic Gross|Worldwide Gross&#xA;1|12/18/2009|Avatar|$425,000,000|$760,507,625|$2,783,918,982&#xA;2&#xA;3|5/20/2011|Pirates of the Caribbean: On Stranger      Tides|$410,600,000|$241,063,875|$1,045,663,875&#xA;4&#xA;5|5/1/2015|Avengers: Age of Ultron|$330,600,000|$459,005,868|$1,408,218,722&#xA;6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on, with additional counter lines between the output that I don't want, how can I get rid of that? (and where does it come from?)&lt;/p&gt;&#xA;" OwnerUserId="26761" LastActivityDate="2017-11-13T16:46:32.890" Title="Python inserts newline by writing to csv" Tags="|python|csv|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24671" PostTypeId="2" ParentId="24660" CreationDate="2017-11-13T11:44:27.740" Score="1" Body="&lt;p&gt;Its extremely simple. There are a lot of ways of doing it. I am assuming you are familiar with Stochastic Gradient Descent.  I am going to tell one naive way of doing it.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Reload the model into RAM. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Write a SGD function like SGD(X,y). It will take the new sample and label and run one step of SGD on it and save the updated model. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As you can see this will be highly inefficient, a better way is to save a number of samples and then run a step of stochastic batch gradient descent on it. So that you dont have to reload the updated model every time you give it a new sample. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I hope this gives you a rough idea of how the implementation can be done. You can easily find much more efficient and scalable ways of doing this. If you are not familiar waith algorithms like SGD, I would recommend to get familiar with them because online learning is just a one sample mini batch gradient descent algorithm. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2017-11-13T11:44:27.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24672" PostTypeId="1" CreationDate="2017-11-13T12:00:01.883" Score="0" ViewCount="325" Body="&lt;p&gt;So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first remove any 0 variance features (~1M -&gt; 2k feats.), then using &lt;code&gt;sklearn.decomposition.TruncatedSVD&lt;/code&gt; I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I run &lt;code&gt;imblearn.under_sampling.RandomUnderSampler&lt;/code&gt; (100k samples for majority classes) and &lt;code&gt;imblearn.over_sampling.SMOTE&lt;/code&gt; (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this I train several models and then compare different metrics to get a better idea of which is the best choice:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;sklearn.linear_model.SGDClassifier(loss='log', alpha=1e-4,tol=1e-4,penalty='elasticnet',max_iter=1000,shuffle=True)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sklearn.linear_model.Perceptron(alpha=1e-3, penalty=None,tol=1e-4,max_iter=1000,shuffle=True)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sklearn.linear_model.PassiveAggressiveClassifier(C=1.0,tol=1e-4,max_iter=1000,shuffle=True)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sklearn.neural_network.MLPClassifier(solver='adam',learning_rate='adaptive',activation='tanh',validation_fraction=0.05,alpha=1e-5,learning_rate_init=1e-4,max_iter=20,hidden_layer_sizes=(100,))&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sklearn.naive_bayes.BernoulliNB(alpha=1.0e-5, fit_prior=True)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around, &lt;code&gt;AUC&lt;/code&gt; and &lt;code&gt;cohen_kappa&lt;/code&gt; are the more realiable metrics for multiclass and imbalanced data sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SVfLD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SVfLD.png&quot; alt=&quot;Metric score overview&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am concerned that the AUC metric show that all models are doing well, but the others, in particular &lt;code&gt;cohen_kappa&lt;/code&gt; which is significantly lower (I know though that it is hard to directly compare the metrics). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores? &lt;/p&gt;&#xA;" OwnerUserId="31948" LastActivityDate="2017-12-14T08:11:32.203" Title="Metrics show badly performing model for multiclass" Tags="|machine-learning|python|classification|multiclass-classification|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24673" PostTypeId="2" ParentId="24670" CreationDate="2017-11-13T12:06:16.157" Score="0" Body="&lt;p&gt;Check the source code of &lt;a href=&quot;http://www.the-numbers.com/movie/budgets/all/5401&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.the-numbers.com/movie/budgets/all/5401&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that table, every second row is an empty &lt;code&gt;&amp;lt;tr&amp;gt;&lt;/code&gt; tag.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-13T12:06:16.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24676" PostTypeId="1" CreationDate="2017-11-13T14:18:49.563" Score="0" ViewCount="133" Body="&lt;p&gt;I've managed to scrape the data from the &lt;a href=&quot;http://www.cinemetrics.lv/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cinemetrics&lt;/a&gt; website, but as it is user-submitted, some of the data needed cleaning. I've been running the titles through the search function of the &lt;a href=&quot;http://www.omdbapi.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;OMDb API&lt;/a&gt;, however this struggles with certain results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the film &lt;em&gt;Meghe Dhaka Tara&lt;/em&gt;. When searching with this as the query on the OMDb API I get no results. However a quick Google search, or even IMDb search, will find the film with the English title &lt;em&gt;The Cloud-Capped Star&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if anybody else has ever run into a similar problem and how they go about working around it?&lt;/p&gt;&#xA;" OwnerUserId="41814" LastActivityDate="2018-01-13T02:56:25.983" Title="Dealing with foreign film titles on IMDb" Tags="|scraping|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24677" PostTypeId="2" ParentId="23489" CreationDate="2017-11-13T14:51:20.883" Score="3" Body="&lt;p&gt;Data intensity is a critical factor, but that factor alone is not sufficient to choose a programming language. Please go through this article from your use case perspective and you can more easily evaluate Python with Java: &lt;a href=&quot;https://togglebrains.wordpress.com/2017/11/05/select-programming-language-for-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://togglebrains.wordpress.com/2017/11/05/select-programming-language-for-machine-learning/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41816" LastActivityDate="2017-11-13T14:51:20.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24678" PostTypeId="1" CreationDate="2017-11-13T14:52:08.360" Score="2" ViewCount="3661" Body="&lt;p&gt;I've been using R and the &lt;code&gt;caret&lt;/code&gt; package since a while. The &lt;code&gt;caret&lt;/code&gt; package provides a function to reduce the number of variables - &lt;code&gt;nearZeroVar()&lt;/code&gt; wherein variables that have close to zero variance are returned and one can remove them from the &lt;code&gt;dataframe&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Caret&lt;/code&gt; also provides another function &lt;code&gt;dummyVars&lt;/code&gt; that converts factors to dummy variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is - what should the order of using the two be? Should the features with near zero variance be thrown out and then converted to dummies or should it be the other way around? I have a feeling that the order matters because in some cases dummy variables might have a lot of zeros but still be important.&lt;/p&gt;&#xA;" OwnerUserId="20749" LastEditorUserId="27665" LastEditDate="2017-11-14T04:23:29.483" LastActivityDate="2017-11-17T09:27:01.537" Title="Should columns with close to zero variance be removed before or after one hot encoding?" Tags="|r|feature-selection|feature-engineering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24679" PostTypeId="1" AcceptedAnswerId="24705" CreationDate="2017-11-13T15:07:37.140" Score="1" ViewCount="249" Body="&lt;p&gt;I recently tried to run Regularized Greedy Forest algorithm (rgf Classifier) from this package &lt;a href=&quot;https://pypi.python.org/pypi/rgf_python&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pypi.python.org/pypi/rgf_python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did run pip install rgf_python, but when I tried to run the example this error pops out in spyder:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  File &quot;C:\Users\me\Anaconda3\lib\site-packages\rgf\sklearn.py&quot;, line 111, in &amp;lt;module&amp;gt;&#xA;    &quot;config flag 'exe_location' to RGF execution file.&quot;.format(_EXE_PATH))&#xA;&#xA;Exception: C:\Users\me\rgf.exe is not executable file. Please set config flag 'exe_location' to RGF execution file.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What should I change in order for config the flag 'exe_location'? May sound beginner but can't find it anywhere.&lt;/p&gt;&#xA;" OwnerUserId="16416" LastActivityDate="2017-11-14T04:55:21.727" Title="Error when running RGFClassifier" Tags="|python|" AnswerCount="1" CommentCount="1" ClosedDate="2017-11-16T01:45:52.637" ContentLicense="CC BY-SA 3.0" />
  <row Id="24681" PostTypeId="2" ParentId="24609" CreationDate="2017-11-13T15:30:03.637" Score="0" Body="&lt;p&gt;You can also use a pandas Dataframe in python (it has excel read in capability).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.read_excel(excel_name)&#xA;df = df[ df.column_name_you_search_in.str.contains(pattern)]&#xA;# print the output&#xA;print(df)&#xA;# save the output as csv&#xA;df.to_csv(&quot;temp_file.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As a more general comment I think the excel file is not sustainable as you are already almost at its maximum (&lt;a href=&quot;https://en.wikipedia.org/wiki/65536_(number)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/65536_(number)&lt;/a&gt;) &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-11-13T15:30:03.637" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24682" PostTypeId="2" ParentId="24670" CreationDate="2017-11-13T16:34:03.880" Score="0" Body="&lt;p&gt;ok so every second line was an empty  tag, I adapted my code accordingly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    all_tr = soup.find_all(&quot;tr&quot;)&#xA;    for movie in range(1, len(all_tr)):&#xA;        if movie % 2 == 0:&#xA;            continue&#xA;        else:&#xA;            row=[]&#xA;            counter+=1&#xA;            row.append(counter)&#xA;            td = all_tr[movie].find_all(&quot;td&quot;)            &#xA;            for colIndex in range(1, len(td)):                &#xA;                row.append(td[colIndex].string)&#xA;            writer.writerow(row)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26761" LastEditorUserId="26761" LastEditDate="2017-11-13T16:46:32.890" LastActivityDate="2017-11-13T16:46:32.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24683" PostTypeId="1" CreationDate="2017-11-13T16:39:38.480" Score="1" ViewCount="137" Body="&lt;p&gt;I'm looking for any papers/articles on how the word embedding dimensions affect the performance in NLP tasks. Specifically, has anyone ever used low dimensional(&amp;lt;10) word embeddings and achieved good results?&lt;/p&gt;&#xA;" OwnerUserId="41823" LastActivityDate="2017-11-13T16:39:38.480" Title="Is there any literature on selection of dimensions for word embeddings?" Tags="|nlp|word-embeddings|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24685" PostTypeId="1" AcceptedAnswerId="24693" CreationDate="2017-11-13T17:14:40.903" Score="2" ViewCount="1149" Body="&lt;p&gt;In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example if we have two documents with features vectors  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d1 = [1,1,1,0,0]  &#xA;d2 = [0,1,1,1,0]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can see that the two documents have the second feature so if we want to search for the second feature with query vector: q = [0,1,0,0,0] then the cosine similarity between q and d1,d2 will be $1/√3$, and not 1 because that we penalize the other features that we have not mention in the query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this discussion I don't understand why penalize it is a good Idea.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is penalizing unseen features good?  &lt;/li&gt;&#xA;&lt;li&gt;Is there another similarity measure that does not penalize them?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41825" LastEditorUserId="41825" LastEditDate="2017-11-13T17:34:30.933" LastActivityDate="2017-11-14T01:47:45.440" Title="Confusion with cosine similarity" Tags="|information-retrieval|cosine-distance|vector-space-models|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24689" PostTypeId="1" AcceptedAnswerId="24714" CreationDate="2017-11-13T18:21:20.023" Score="-3" ViewCount="245" Body="&lt;p&gt;Searched a lot of research papers, blogs and videos but couldnt find an acceptable answer for choosing the weights and biases in neural network. Few people have mentioned to use weights randomly but if that is the case isnt that is uncertain (fuzzy)?&lt;/p&gt;&#xA;" OwnerUserId="41828" LastActivityDate="2017-11-14T09:09:23.567" Title="What is the rationale to use weights and biases in a neural network?" Tags="|neural-network|deep-learning|fuzzy-logic|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24690" PostTypeId="1" AcceptedAnswerId="24691" CreationDate="2017-11-13T19:42:06.007" Score="-1" ViewCount="112" Body="&lt;p&gt;I am currently learning about Nueral Networks in Machine Learning and think it very interesting but I don't really know any other Applications NN have other than in Machine Learning and AI. I have tried researching these things but have come up short. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone tell me if you could use it in &quot;Inventory management&quot; or &quot;Finance&quot;? If you can would you have any examples? &lt;/p&gt;&#xA;" OwnerUserId="41036" LastEditorUserId="27665" LastEditDate="2017-11-14T08:13:52.720" LastActivityDate="2017-11-15T14:56:15.803" Title="Applications of Neural networks?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="1" ClosedDate="2017-11-15T16:12:52.323" ContentLicense="CC BY-SA 3.0" />
  <row Id="24691" PostTypeId="2" ParentId="24690" CreationDate="2017-11-13T20:28:22.697" Score="1" Body="&lt;p&gt;Yes, is applied in most of the fields:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A simple search in google scholar will provide a list what include: fraud detection, stock market trading decision, inventory classification and many more, including the link below&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;as_sdt=0,11&amp;amp;q=neural%20network%20inventory%20management&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural network inventory management&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As for Finance we can look at: credit scoring, bankruptcy forecast, financial forecasting and many more. Link from google scholar included below:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;as_sdt=0%2C11&amp;amp;q=neural%20network%20finance&amp;amp;btnG=&quot; rel=&quot;nofollow noreferrer&quot;&gt;neural network finance&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Including also working example for simple credit scoring with NN: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.r-bloggers.com/using-neural-networks-for-credit-scoring-a-simple-example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Using neural networks for credit scoring: a simple example&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8752" LastActivityDate="2017-11-13T20:28:22.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24692" PostTypeId="2" ParentId="24583" CreationDate="2017-11-13T21:42:46.303" Score="1" Body="&lt;p&gt;Tree-based classification models are generally insensitive to oversampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's because when you upsample the 'true' class, it affects leaf impurity metrics (such as Gini or entropy) only marginally. Imagine a leaf with $a$ 'true' and $b$ 'false' examples - the Gini would be $2\frac{ab}{a+b}$. When you upsample 'true' class by factor of $\lambda$, the Gini becomes $2\lambda\frac{ab}{a\lambda+b}$. If $a&amp;lt;&amp;lt;b$ (which is your case), then this change is approximately just multiplying all the Ginis by $\lambda$, which does not affect the choice of a next tree split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus my advice would be to give up upsampling attempts, and concentrate on feature engineering, parameter tuning, and maybe getting more real observations. If you still want resampling, then you can just set unequal &lt;code&gt;class_weights&lt;/code&gt; when fitting your trees - it will affect trainging, but keep cross-validation metrics intact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in top 10% conversion, you may well approximate this metric by ROC AUC and optimize the latter. AUC is not sensitive at all to over- and undersampling. If your model is precise enough, then most of the 15% 'true' examples will be somewhere around this top 10%. Thus, &lt;em&gt;ranking&lt;/em&gt; performance measured by AUC and &lt;em&gt;selection&lt;/em&gt; performance measured by 10% conversion would strongly correlate, and you can select your best model by AUC.  &lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-13T21:42:46.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24693" PostTypeId="2" ParentId="24685" CreationDate="2017-11-13T21:44:01.447" Score="0" Body="&lt;p&gt;Of course cosine similarity is not proper for searching a specific features in documents! To do this, you can exactly using dot product, as it will ignore zero features in query vector from documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity, in the current context, can be used to finding similarity between two documents. So, all features can be important to finding similarity. It means, if there is not a feature in query vector, but there is in  a document, or vice versa, these two are 100 percent similar. Hence, it makes sense.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-13T21:44:01.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24694" PostTypeId="1" AcceptedAnswerId="24695" CreationDate="2017-11-13T22:26:03.270" Score="8" ViewCount="4118" Body="&lt;p&gt;As far as I understand it, the Neural Style Transfer uses a content image and a style image, and generate a new image based on the two images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It tries to find a set of pixel values such that the cost function &lt;code&gt;J(C, S)&lt;/code&gt; is minimized. It does not have any labels associated in advance, but it has an output (generated image) that should be the target of the learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not sure if this is considered supervised or unsupervised learning. Which does it belong to?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2018-06-21T06:58:24.940" Title="Is Neural Style Transfer considered supervised or unsupervised learning?" Tags="|neural-style-transfer|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24695" PostTypeId="2" ParentId="24694" CreationDate="2017-11-13T23:25:09.203" Score="14" Body="&lt;p&gt;Neural style transfer is not really machine learning, but an interesting side effect/output of machine learning on image tasks. When performing neural style transfer using a pre-trained model, then a significant amount of supervised machine learning has already occurred to enable it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The style transfer algorithm &lt;em&gt;is&lt;/em&gt; still an example of gradient-based cost function optimisation, which it shares with many supervised and unsupervised learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output of style transfer is partially a probe into what a network has learned about different levels of structure in the problem domain it has been trained on. However, its main use has been to generate images with altered/mixed aesthetics for art and entertainment.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-11-13T23:25:09.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24696" PostTypeId="2" ParentId="20296" CreationDate="2017-11-13T23:25:52.813" Score="15" Body="&lt;p&gt;The answer from Neil is correct. However I think its important to point out that while the &lt;em&gt;loss&lt;/em&gt; does not depend on the distribution between the incorrect classes (only the distribution between the correct class and the rest), the &lt;em&gt;gradient&lt;/em&gt; of this loss function does effect the incorrect classes differently depending on how wrong they are. So when you use cross-ent in machine learning you will change weights differently for [0.1 0.5 0.1 0.1 0.2] and [0.1 0.6 0.1 0.1 0.1].  This is because the score of the correct class is normalized by the scores of all the other classes to turn it into a probability. &lt;/p&gt;&#xA;" OwnerUserId="41843" LastActivityDate="2017-11-13T23:25:52.813" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24697" PostTypeId="1" AcceptedAnswerId="24704" CreationDate="2017-11-13T23:49:28.583" Score="0" ViewCount="80" Body="&lt;h3&gt;Dataset Features&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Insurance underwriting dataset for 8 years. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;age&lt;/li&gt;&#xA;&lt;li&gt;location&lt;/li&gt;&#xA;&lt;li&gt;amount insured &lt;/li&gt;&#xA;&lt;li&gt;some other features...(medical evidence)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not all feature will be available to all applicants.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Target Variable&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Decision on whether the applicant can be insured&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Question&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;What techniques can be used and which ones would work best? &#xA;Outline a high-level overview - I do not think I have to go into too much detail as I do not have any data.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Things I have considered&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I am thinking to first slice the data and analyse it in parts and see can I find  a pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regression analysis could be carried out. Possible Logistic regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could take a sample and perform hypothesis tests?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that this is an ideal machine learning situation. I don't have any experience in this field and I think I am better to stick with methods I have some knowledge of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is very ambiguous, but a little nod in the right direction and I would be very appreciative. &lt;/p&gt;&#xA;" OwnerUserId="41844" LastEditorUserId="41525" LastEditDate="2017-11-14T08:09:49.237" LastActivityDate="2017-11-14T08:09:49.237" Title="New to data Science. Which techniques best to use Large data set in insurance company?" Tags="|machine-learning|r|data-mining|regression|logistic-regression|" AnswerCount="1" CommentCount="1" ClosedDate="2017-11-16T13:16:34.320" ContentLicense="CC BY-SA 3.0" />
  <row Id="24699" PostTypeId="2" ParentId="24685" CreationDate="2017-11-14T01:37:00.147" Score="0" Body="&lt;p&gt;The point of your example seems to be more one of length normalization. Long story short: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;rare terms are more informative than frequent words and and we do not want to weight all terms equally, &lt;/li&gt;&#xA;&lt;li&gt;the more frequent a term, the more likely a document carries information about a term. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.youtube.com/watch?v=eqpzVedX1kY&amp;amp;index=57&amp;amp;list=PL0174E49C0E0DD5C8&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; provides examples demonstrating why considering all the words in the text are important, including examples particular to the point you make in your text&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get an intuitive feel for why your search might be a problem, consider the fact that under your approach, all queries will be extremely similar to an English dictionary&lt;/p&gt;&#xA;" OwnerUserId="41847" LastEditorUserId="29575" LastEditDate="2017-11-14T01:47:45.440" LastActivityDate="2017-11-14T01:47:45.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24700" PostTypeId="2" ParentId="24676" CreationDate="2017-11-14T01:45:54.747" Score="1" Body="&lt;p&gt;&quot;Meghe Dhaka Tara&quot; is a noun phrase. It makes sense for an API looking for movies to not translate the phrase as that is not its purpose. I suspect, that at most it will have some basic spell checks in place (for instance &quot;did you mean `Megh Dhaka Tara'&quot;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &quot;The Cloud-Capped Star&quot; is indeed an english dub for the movie, then it suggests that the service provider has not linked movies to its translations. &lt;/p&gt;&#xA;" OwnerUserId="41847" LastActivityDate="2017-11-14T01:45:54.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24702" PostTypeId="1" CreationDate="2017-11-14T02:14:42.247" Score="8" ViewCount="1627" Body="&lt;p&gt;My question on A/B testing is about doing post test segmentation analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I run an A/B test on my website to track bounce rate.  On the&#xA;  treatment group, i put a video to explain my company. On the control&#xA;  group i put just plain text. I pick a segment of users who are first&#xA;  time users from USA to be split 50/50 into the 2 groups.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Metric that i am tracking is average bounce rate (assume 20%). &#xA;Power effect (0.8)&#xA;effect size i expect to see(10% so bounce rate should fall to (20% - 0.10 * 20% = 18%))&#xA;Calculated sample size required is say 1000 for each group. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Say i run the test for the correct amount of time. At the end of the test, i get a &lt;strong&gt;p-value of 0.06&lt;/strong&gt;. i do not reject the null hypothesis. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However, when i do post test segmentation analysis, for example, i saw&#xA;  that users who signed up for a free trial, 44% of them played the&#xA;  video.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case, how do i calculate if the 44% was significant? (while taking into account the multiple comparison problem?)&#xA;Like in the Airbnb experiment, they did post segmentation analysis on the browser type and was able to calculate the p-value. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/E3Lj9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/E3Lj9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that for every segment that i want to analyze, i need to have at least 1000 samples? Also how would i recalculate the p-value given that the p-value of this A/B test was already generated above as 0.06?&lt;/p&gt;&#xA;" OwnerUserId="41849" LastEditorUserId="100269" LastEditDate="2020-07-26T22:14:31.390" LastActivityDate="2020-12-09T07:25:03.580" Title="A/B testing: How to calculate p-value on post test segments?" Tags="|statistics|ab-test|experiments|hypothesis-testing|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24704" PostTypeId="2" ParentId="24697" CreationDate="2017-11-14T04:23:08.253" Score="1" Body="&lt;p&gt;I think you need to do couple of tests to see what all variables are important with respect to your Target Variable(client can be insured: Yes/No) - this kind of test is called Predictor Importance test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you you have mentioned this is sector is new for you, I would suggest you to take all the variable you think are useful. Convert categorical variables to factors using &lt;code&gt;as.factor()&lt;/code&gt; numeric variable to numeric as &lt;code&gt;as.numeric()&lt;/code&gt;. The reason for explicit transformation is sometimes algorithms cannot understand like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test1 &amp;lt;- c(1,2,2,4,4,1,2)&#xA;summary(test1)&#xA;   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;  1.000   1.500   2.000   2.286   3.000   4.000 &#xA;test1 &amp;lt;- as.factor(test1)&#xA;summary(test1)&#xA;  1 2 4 &#xA;  2 3 2 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now once the data is ready, you can give them as an input to &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt;, by using this you get the predictor importance graph. &#xA;For better understanding you can go through this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&#xA;or else if you want to learn why, how and when you can go through this &lt;a href=&quot;https://stats.idre.ucla.edu/other/mult-pkg/whatstat/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; for different tests for different variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boruta does all the above by itself and the outcome is set of important features, With Respect to that you can feed those respective features to your model for getting better results/accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As your problem is Binary Classification, you can use the following Algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://trevorstephens.com/kaggle-titanic-tutorial/r-part-3-decision-trees/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision Trees&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://trevorstephens.com/kaggle-titanic-tutorial/r-part-5-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/bayesian-network-in-r-introduction/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Network&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/machine-learning-using-support-vector-machines/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Support Vector Machines(SVM)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://beckmw.wordpress.com/2013/03/04/visualizing-neural-networks-from-the-nnet-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Network&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For all the above algorithms I've attached a link for your reference in which you can find at-least one example for each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful, mark as answer if you got what you needed.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-14T04:23:08.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24705" PostTypeId="2" ParentId="24679" CreationDate="2017-11-14T04:55:21.727" Score="2" Body="&lt;p&gt;I had the same problem, and I'm not proud of the very &quot;hacky&quot; solution, but the bottom line is it's working now. I'm working on a Mac, so I'm sure your solution will be a bit different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The normal pip install wasn't working for me, so to install I did...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;git clone https://github.com/fukatani/rgf_python.git&#xA;cd rgf_python&#xA;python setup.py install&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was then able to find the executable file your error message is complaining about in the new rgf_python directory, specifically...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rgf_python/include/rgf/bin/rgf&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From there, I tried to change the exe_location in my .rgfrc file to point to the above file, but it didn't work so I went to the /site-packages/rgf/sklearn.py file mentioned at the top of your error message and changed a line in the &quot;_get_paths&quot; function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me it was line 64/66. I suspect yours may be 54/56 if you're on Windows. I commented out the old declaration of &quot;exe&quot; and changed it to point to the rgf executable file I found earlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize how messy this is, so I hope someone can find a better solution, but in the meantime, this did the trick for me. Good luck, and let me know if you have any questions.&lt;/p&gt;&#xA;" OwnerUserId="41854" LastActivityDate="2017-11-14T04:55:21.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24706" PostTypeId="2" ParentId="24429" CreationDate="2017-11-14T05:18:46.077" Score="3" Body="&lt;p&gt;I'm not allowed to comment, but have you tried using the numpy array that you get from class_weight.compute_class_weight(), rather than converting it to a dict? I've always skipped that part, and in your case I would say class_weight=class_weights. Sorry if I'm suggesting something you already ruled out. Good luck.&lt;/p&gt;&#xA;" OwnerUserId="41854" LastActivityDate="2017-11-14T05:18:46.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24707" PostTypeId="2" ParentId="24672" CreationDate="2017-11-14T05:31:30.780" Score="1" Body="&lt;p&gt;I think you need to consider these metrics also i.e., &lt;strong&gt;Precision and Recall&lt;/strong&gt; when you decide on some model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can in the above outcome that there are Precision and Recall values, you din't explain anything about the problem's business perspective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Consider a data set with 100,000 observations. A data set consist of candidates who applied for Internship in Harvard. Apparently, harvard is well-known for its extremely low acceptance rate. The dependent variable represents if a candidate has been shortlisted (1) or not shortlisted (0). It was found ~ 98% did not get shortlisted and only ~ 2% got lucky. &#xA;In this scenario you need to understand that precision and recall plays a vital role at the time of selecting model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same way if your business problem is inline with the above scenario then you need to select a model with &lt;strong&gt;high&lt;/strong&gt; Precision and Recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above screenshot which you have shared, it think you have good performance scores too(Assuming that your model is not over fitting). So, you can select which ever model suits your business problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-14T08:11:32.203" LastActivityDate="2017-12-14T08:11:32.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24710" PostTypeId="1" AcceptedAnswerId="24711" CreationDate="2017-11-14T07:28:17.127" Score="5" ViewCount="1112" Body="&lt;p&gt;I am using a CNN based model to do sequence classification. Since training an entire dataset is very expensive, and I have a large set of features needed to try, its impossible for me to select features by full trainings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually a good sanity check before training, is try to let model overfit a small set of training samples, to make sure the model is at least capable of remember a small sample size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Borrowed from this idea, my question is, can I train a small subset of training data, and use its loss curve as a metric, to select best features? Each training is to test how quick would the training loss converges given a subset of selected features.&lt;/p&gt;&#xA;" OwnerUserId="41857" LastEditorUserId="29169" LastEditDate="2021-02-13T19:21:47.017" LastActivityDate="2021-02-13T19:21:47.017" Title="Feature selection by overfitting a small sample size" Tags="|machine-learning|deep-learning|feature-selection|convolutional-neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24711" PostTypeId="2" ParentId="24710" CreationDate="2017-11-14T07:44:00.237" Score="5" Body="&lt;p&gt;Congratulations! You have suggested independently the &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_selection#Wrapper_method&quot; rel=&quot;noreferrer&quot;&gt;Wrapper method&lt;/a&gt; for feature selection. Yes, you can use this method. &#xA;However, consider that the wrapper method is slow since you have to train a model for each iteration. More than that, feature selection is a NP-complete problem so don't expect the optimal subset.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2017-11-14T07:44:00.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24712" PostTypeId="2" ParentId="24678" CreationDate="2017-11-14T08:28:52.607" Score="1" Body="&lt;p&gt;I think 1st you need to remove the columns with near to zero variance(assuming that they are not nominal variables), only if you have more variables as the computation time can be reduced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us consider a scenario where there are 2 columns col1, col2 and you checked for variance then col1 had 0.003 variance and col2 had 0.5 var. once you apply the function &lt;code&gt;nearZeroVar()&lt;/code&gt;, it removes col1, you need to hot-encode only 1 variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By that you can save time and your computer performance won't be effected.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-14T08:28:52.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24714" PostTypeId="2" ParentId="24689" CreationDate="2017-11-14T09:09:23.567" Score="1" Body="&lt;p&gt;&lt;strong&gt;Weight Initialization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weights can be initialized by either setting them all to zero. Or by setting them randomly using a Gaussian distribution centered at 0 with a variance similar to the values of your input features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is every initialization going to give you the same results?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;No. Neural networks (NNs) converge to a local minimum. Thus, different initialization of the weights will cause the loss function to be minimized to a different value. If you are unhappy with your results you can always do random restarts with different weight initialization. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2017-11-14T09:09:23.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24716" PostTypeId="2" ParentId="24667" CreationDate="2017-11-14T09:37:47.550" Score="2" Body="&lt;p&gt;The outcome of a multi-nomial or binomial is confusion matrix (2*2 for binomial, n*n for multinomial),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Interpretation of the confusion matrix is for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zU0WF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zU0WF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Accuracy&lt;/strong&gt; of the model : &lt;code&gt;((TP + TN) / (TP + FN + FP + TN) ) * 100&lt;/code&gt;&#xA;It means you model could exactly pick this much percent of data was classified correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Precision&lt;/strong&gt;: &lt;code&gt;(TP / (TP + FP))*100&lt;/code&gt;&#xA;That measure of correctness achieved in positive prediction i.e. of observations labeled as positive, how many are actually labeled positive&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Recall&lt;/strong&gt;: &lt;code&gt;(TP / (TP + FN)) * 100&lt;/code&gt;&#xA;It is a measure of actual observations which are labeled (predicted) correctly i.e. how many observations of positive class are labeled correctly. It is also known as ‘Sensitivity’.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;F-Measure&lt;/strong&gt;: &lt;code&gt;((1 + β)² × Recall × Precision) / ( β² × Recall + Precision )&lt;/code&gt;&#xA;It combines precision and recall as a measure of effectiveness of classification in terms of ratio of weighted importance on either recall or precision as determined by β coefficient.&#xA;generally β is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want read them in just normal words in you example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        Predicted  &#xA;          M   F&#xA;       M 100 150&#xA;actual F 100 300&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It means your model could predict 100+300 correctly out of 100+150+100+300&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actual number of males where 250 but you could classify 100 correctly and 50 wrongly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, Actual number of females where 400 but you could classify 300 correctly and 100 wrongly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you any need more detail let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go through this &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;, you will get better idea.&#xA;This &lt;a href=&quot;https://upload.wikimedia.org/wikipedia/commons/0/0e/DiagnosticTesting_Diagram.svg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; is an interactive chart, which will give you better understanding.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-14T09:53:08.520" LastActivityDate="2017-11-14T09:53:08.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24717" PostTypeId="2" ParentId="24702" CreationDate="2017-11-14T11:00:57.457" Score="0" Body="&lt;p&gt;Well if you want to answer the question if a single segment reaches the same level and you ignore all other segments behaviors then this should be the required number (given that initial performance of the segments was the same). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a warning when you use to many segments this: &lt;a href=&quot;https://xkcd.com/882/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://xkcd.com/882/&lt;/a&gt;&#xA;can happen.&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2017-11-14T11:00:57.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24719" PostTypeId="1" AcceptedAnswerId="25848" CreationDate="2017-11-14T11:51:53.663" Score="1" ViewCount="380" Body="&lt;p&gt;I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where,  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$y$ is the outcome produced by a random process&lt;/li&gt;&#xA;&lt;li&gt;$x$ is the feature influencing the outcome $y$&lt;/li&gt;&#xA;&lt;li&gt;$L_\widetilde{p}(p)$ is the log likelihood&lt;/li&gt;&#xA;&lt;li&gt;$\widetilde{p}$ is the empirical distribution of training data&lt;/li&gt;&#xA;&lt;li&gt;$p(y|x)$ is the model&lt;/li&gt;&#xA;&lt;li&gt;$\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I went through this &lt;a href=&quot;http://www.cs.cmu.edu/afs/cs/user/aberger/www/html/tutorial/node8.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;reference tutorial on max entropy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38351" LastEditorUserId="38351" LastEditDate="2017-11-17T05:28:39.677" LastActivityDate="2017-12-20T15:00:57.450" Title="Maximum Entropy modelling - likelihood equation" Tags="|machine-learning|nlp|statistics|probability|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24720" PostTypeId="1" AcceptedAnswerId="26487" CreationDate="2017-11-14T12:15:19.640" Score="2" ViewCount="1883" Body="&lt;p&gt;The &lt;code&gt;xgboost&lt;/code&gt; package &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/input_format.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;has two files&lt;/a&gt; that must be used for ranking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;train.txt&lt;/code&gt; with the data&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;train.txt.group&lt;/code&gt; with the group of each observation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don't understand two things in each file:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What should I use as positive/negative classes? In ranking, there is no such thing as a positive/negative class...&lt;/li&gt;&#xA;&lt;li&gt;What should I use as group? Say that I have observations x1 &gt; x2 &gt; x3. How do I express this lexicographical order using groups?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="18375" LastEditDate="2017-12-11T15:46:32.623" LastActivityDate="2018-01-10T15:50:10.107" Title="XGBoost ranking file format" Tags="|xgboost|ranking|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24722" PostTypeId="1" AcceptedAnswerId="24736" CreationDate="2017-11-14T13:23:47.300" Score="2" ViewCount="1858" Body="&lt;p&gt;I'm trying to classify a binary sample with Keras and I would like to classify as many correctly as possible, while ignore the ones where the model is not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fully connected Nerual network currencly achieves around 65% but I would like to get a higher result of correctly classified ones, while ignoring the ones where the model is uncertain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to tell Keras to simply ignore the ones where the model is uncertain and achieve a higher accuracy that way? Or is there a network design that could achieve this, for example feeding the result of the network striaght into a second part of it which then decides whether the prediction is likely accurate or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I was thinking of achieving this is by building a second neural network on top of it that decides based on the result of the first network and all the input data of it, whether the classification will be correct or not. Would that work, and if yes, is there no more elegant way of achieving this in one go, such as directly having the results feeding into a second part of the network that then decides if the prediction is likley accurate or not?&lt;/p&gt;&#xA;" OwnerUserId="32860" LastActivityDate="2023-09-03T15:12:39.047" Title="Classifying with certainty" Tags="|classification|tensorflow|keras|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24723" PostTypeId="1" CreationDate="2017-11-14T14:52:06.040" Score="1" ViewCount="149" Body="&lt;p&gt;Say I have a spreadsheet of 1000 chat messages that are labeled as happy, sad or funny.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way of detecting if the labels are indeed correct or if they need improvement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I go about performing an analysis on the messages? How can I build a predictive model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of feature and model selection should be used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python is my preferred language and I wish to load them as a CSV.&lt;/p&gt;&#xA;" OwnerUserId="41881" LastActivityDate="2017-11-15T07:32:51.257" Title="Detecting and confirming sentiments in chat messages" Tags="|machine-learning|python|predictive-modeling|dataset|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24725" PostTypeId="1" AcceptedAnswerId="24815" CreationDate="2017-11-14T15:11:06.687" Score="1" ViewCount="3684" Body="&lt;p&gt;I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am worried if I decide for this parameter with experimenting on my current dataset, it will differ by time as I would be receiving new data points very soon. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I know that this parameter better than any other, just by experimenting it with outlier removal? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="40029" LastEditorUserId="40853" LastEditDate="2017-11-15T08:50:58.923" LastActivityDate="2017-11-16T13:54:10.627" Title="How to decide for the contamination value (proportion of the outliers) in my dataset?" Tags="|data-cleaning|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24727" PostTypeId="1" CreationDate="2017-11-14T16:49:43.590" Score="1" ViewCount="243" Body="&lt;p&gt;I have  customer buying data with each row specifying an item bought by customer. The problem is that even if at the same time customer buys five items then there are five different rows for it and as a result the total number of rows in data have gone too much to train. what can i do to reduce the size of data so that i can train it effectively.Just to give the context of the problem, i want to recommend products to the customers based on their buying data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset size: (7981262, 16)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data description:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Variable                                           Description&#xA;customerID                                    unique customer ID&#xA;DOB                                         date of birth of customer&#xA;Gender                                               gender&#xA;State                                          customer's state&#xA;PinCode                                pincode of area where customer lives&#xA;transactionDate                                  date of transaction&#xA;store_code                                 unique code of store&#xA;store_description                             description of store&#xA;till_no                                       counter no. in the store&#xA;transaction_number_by_till                 unique transaction number by counter, &#xA;transactionDate, store_code&#xA;promo_code                      if promotional code (offer) used in the transaction&#xA;promo_description                            description of the offer&#xA;product_code                               unique code of the product purchased&#xA;product_description                       description of the product purchased&#xA;sale_price_after_promo                 sale price of the product after applying &#xA;promotion&#xA;discountUsed                  after promo, customer used this discount(s) on transaction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40904" LastEditorUserId="18375" LastEditDate="2017-12-11T20:33:22.743" LastActivityDate="2024-03-04T10:01:25.190" Title="Reduce data length to train effectively" Tags="|machine-learning|data-mining|pandas|recommender-system|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24728" PostTypeId="1" CreationDate="2017-11-14T17:20:29.267" Score="6" ViewCount="6653" Body="&lt;p&gt;Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x) = ax + \mathcal{N}(b, c)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with $a, b, c \in \mathbb{R}^+$, e.g. something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;feature  | target&#xA;-----------------&#xA;0             0.0&#xA;0             1.0&#xA;0             1.5&#xA;0            -1.2&#xA;0            -0.9&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that neural networks can deal pretty well with labeling errors in classification problems. Meaning if you have a large dataset and a couple of examples have the wrong label, they get basically ignored.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for this kind of problem I'm not too sure. A first experiment indicates that they do smooth values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there choices in architecture / training which help the smoothing / averaging / removal of noise?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;What I tried&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I created a network which can solve this kind of regression problem without noise. It gets a MSE of about &lt;code&gt;0.0005&lt;/code&gt;. When I add a bit of noise to the training set only, I get an MSE of &lt;code&gt;0.001&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python&#xA;&#xA;# core modules&#xA;import random&#xA;&#xA;# 3rd party modules&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from sklearn.model_selection import train_test_split&#xA;import numpy as np&#xA;&#xA;&#xA;def main(add_noise=True):&#xA;    # Get data&#xA;    xs, ys = create_data_points(10000)&#xA;    x_train, x_test, y_train, y_test = train_test_split(xs, ys, test_size=0.20)&#xA;&#xA;    # Add noise to training data&#xA;    if add_noise:&#xA;        noise = np.random.normal(0, 0.1, len(x_train))&#xA;        x_train = x_train + noise&#xA;&#xA;    # Create model&#xA;    model = create_model()&#xA;    model.compile(optimizer='rmsprop',&#xA;                  loss='mse',&#xA;                  metrics=['mse'])&#xA;&#xA;    # Fit model to data.&#xA;    model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)&#xA;&#xA;    # Evaluate&#xA;    y_pred = model.predict(x_test, batch_size=100).flatten()&#xA;    print(&quot;MSE on test set:&quot;)&#xA;    print(((y_pred - y_test)**2).sum() / len(y_test))&#xA;&#xA;&#xA;def create_data_points(nb_points):&#xA;    xs = []&#xA;    ys = []&#xA;    for i in range(nb_points):&#xA;        x = random.random()&#xA;        xs.append(x)&#xA;        ys.append(2 * x)&#xA;    return np.array(xs), np.array(ys)&#xA;&#xA;&#xA;def create_model(input_dim=1, output_dim=1):&#xA;    model = Sequential()&#xA;    model.add(Dense(200, input_dim=input_dim, activation='relu'))&#xA;    model.add(Dense(200, input_dim=input_dim, activation='relu'))&#xA;    model.add(Dense(output_dim, activation='linear'))&#xA;    return model&#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Outliers&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In an earlier version of this question I wrote &quot;outlier&quot; when I meant &quot;label noise&quot;. For outliers, there is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.scialert.net/fulltext/?doi=jas.2005.1394.1398&amp;amp;org=11&quot; rel=&quot;noreferrer&quot;&gt;The Effects of Outliers Data on Neural Network Performance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2017-11-15T07:08:08.523" LastActivityDate="2017-11-15T09:10:55.147" Title="Can Neural Networks be trained to smooth values / output the average?" Tags="|neural-network|regression|noise|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24729" PostTypeId="1" AcceptedAnswerId="24731" CreationDate="2017-11-14T17:20:58.253" Score="18" ViewCount="29058" Body="&lt;p&gt;I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?&lt;/p&gt;&#xA;&lt;p&gt;I found this interesting &lt;a href=&quot;http://amunategui.github.io/feature-hashing/#sourcecode&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;But they are converting to class/object which I don't want. I want my final output as a data frame so that I can test with different machine learning models? Or, is there any way I can use the generated matrix to train the other machine learning models other than Logistic regression or XGBoost?&lt;/p&gt;&#xA;&lt;p&gt;Is there anyway I can implement it?&lt;/p&gt;&#xA;" OwnerUserId="40879" LastEditorUserId="85045" LastEditDate="2020-12-02T16:38:02.680" LastActivityDate="2021-03-13T20:12:29.053" Title="One hot encoding alternatives for large categorical values" Tags="|machine-learning|dataset|dataframe|dimensionality-reduction|encoding|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="24730" PostTypeId="2" ParentId="24727" CreationDate="2017-11-14T17:31:39.123" Score="0" Body="&lt;p&gt;If your total dataset size is too much, take a random sample of n%. I suggest taking a sample of n% of users, then using all rows for each of those sampled users. If you have timestamps, I further suggest making your train/test split based on time. I.e., for a user with 10 purchased items, train on her first 8 items and try to predict her last 2 items.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-14T17:31:39.123" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24731" PostTypeId="2" ParentId="24729" CreationDate="2017-11-14T17:33:46.247" Score="14" Body="&lt;p&gt;One option is to map rare values to 'other'.  This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian.  See e.g. the paper Deep Knowledge Tracing, which says this approach is motivated by the idea of compressed sensing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BARANIUK, R. Compressive sensing. IEEE signal processing magazine 24, 4 (2007).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, they map each vector of length N to a shorter vector of length log2(N).  I have not done this myself but I think it would be worth trying.  &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-14T17:33:46.247" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24732" PostTypeId="2" ParentId="24728" CreationDate="2017-11-14T17:41:56.520" Score="7" Body="&lt;p&gt;In general simpler models are more robust to noise in the input.  The strength of neural networks is also their biggest 'gotcha' - they are extremely expressive. This means they easily overfit and can be sensitive to noise in inputs.  The strategy of simplifying a model to make it more robust to noise is called regularization.  There are many types:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use smaller hidden layers (i.e. 20 instead of 200 nodes)&lt;/li&gt;&#xA;&lt;li&gt;use dropout, where during training inputs are randomly set to 0 during training (makes the network more robust to noise overall)&lt;/li&gt;&#xA;&lt;li&gt;stop training earlier - 'early stopping'&lt;/li&gt;&#xA;&lt;li&gt;use L1 or L2 regularization, which imposes a cost on the weights&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All of these can be done from with Keras.  You want to make your network more robust to noise without decreasing your validation quality.  To do this, I would try the above suggestions in order.  You can measure overfitting by looking at the difference between predictive accuracy on your training data and validation data.  If they are very different - your model has learned structure in your training data that is not in your validation that is NOT what you want.  Try to fiddle regularization nobs until (1) your train-validation AUC or accuracy is very similar, (2) your validation AUC/accuracy is still sufficiently high.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-14T17:41:56.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24734" PostTypeId="1" CreationDate="2017-11-14T18:09:26.207" Score="0" ViewCount="286" Body="&lt;p&gt;I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose that for a given customer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).&lt;/p&gt;&#xA;" OwnerUserId="41888" LastEditorUserId="41888" LastEditDate="2017-11-14T20:54:20.220" LastActivityDate="2017-11-15T02:14:20.153" Title="recommender system: how to compare different scores when calculated individually?" Tags="|recommender-system|model-evaluations|score|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24735" PostTypeId="2" ParentId="24734" CreationDate="2017-11-14T18:57:06.890" Score="1" Body="&lt;p&gt;You can continue to use machine learning for this model-combination step too. You're describing a form of ensembling. How about:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;set aside some purchases as your validation set V, and keep the remaining as training set TRAIN&lt;/li&gt;&#xA;&lt;li&gt;for TRAIN, use k-fold cross-validation to create your separate models&lt;/li&gt;&#xA;&lt;li&gt;use TRAIN again with k-fold cross-validation to ensemble the models together&lt;/li&gt;&#xA;&lt;li&gt;report performance of the ensembled models on V&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Ensemble techniques will take into account the predictive quality of each individual model.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-14T18:57:06.890" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24736" PostTypeId="2" ParentId="24722" CreationDate="2017-11-14T18:59:31.280" Score="2" Body="&lt;p&gt;Softmax output in neural networks can be misleading - often the confidence provided is higher than is intuitive.  See e.g. here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Baseline for Detecting Misclassified and&#xA;Out-of-Distribution Examples in Neural Networks&#xA;&lt;a href=&quot;https://pdfs.semanticscholar.org/c26e/1beaeaa55acae7336882de5df48716afb8bb.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pdfs.semanticscholar.org/c26e/1beaeaa55acae7336882de5df48716afb8bb.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which suggests that in practice, softmax is not helpfully interpretable as a probability but should instead be used for ranking among class options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to have an accurate probability estimate, you might consider using a Bayesian approach in which you explicitly model your estimate of each of the input variances, the output variance, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Failing that, having a second phase neural network that takes the input and predict correct or incorrect classification by the first network is an interesting idea - where incorrect classification is a proxy for 'low confidence' classification.  If you try it I'd be curious to know how it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: As @Emre said the input to the softmax would be more informative than the softmax itself because it's pre-scaled (i.e. not forced to sum to 1). So it should reflect confidence better, with values further away from 0 indicating higher confidence.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-14T18:59:31.280" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24738" PostTypeId="1" CreationDate="2017-11-14T20:20:30.630" Score="1" ViewCount="4423" Body="&lt;p&gt;Are there any heuristics for deciding on where to start with the number of layers for a neural network? I built one with with 7 layers that has 40 input features and this took around 8 hours to train completely, but now I want to add a bunch of polynomial features which will increase the input size of my network to a few thousand features. Since there are so many features will my network benefit from having more layers? If so, what is a good starting point for the number of features? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is some information about my problem: suppose I have a set of usage data from a website. So if a user logs into the website one day their usage statistics during that day is calculated. I am using a few statistical tools to summarize the data and I want to make a prediction about whether or not the user will use the website in the next 60 days. My plan is to try and engineer all the possible monomial quadratic features from this summary and feed this information into a DNN. Without the feature engineering I have roughly 60 different features, but with it I will have at least an addition 3600 features.&lt;/p&gt;&#xA;" OwnerUserId="41894" LastEditorUserId="41894" LastEditDate="2017-11-14T20:42:10.823" LastActivityDate="2017-11-15T03:26:46.820" Title="How deep should my neural network be?" Tags="|neural-network|deep-learning|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24739" PostTypeId="1" CreationDate="2017-11-14T21:34:11.063" Score="2" ViewCount="585" Body="&lt;p&gt;I want to model a process that is a function of time, $X(t)$. I have a data set which corresponds to a coarse sampling of the function values $X$ at different time points, $t_1, t_2, ...$. I have a analytic-derived model for the derivative $\frac{dX}{dt}$ = $F(X(t),G(t))$ which has a couple tunable parameters in the (known) functions $F$ and $G$. I've been able to tune these models with the following process:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Chose a set of parameters for the functions $F$ and $G$.&lt;/li&gt;&#xA;&lt;li&gt;Do a forward-Euler method from the earliest time in my data set to the ending point in my data set, generating the $X(t)$ which corresponds to the above parameters in $F$ and $G$.&lt;/li&gt;&#xA;&lt;li&gt;Use this $X(t)$, along with my measured data points to generate a model error.&lt;/li&gt;&#xA;&lt;li&gt;Use the above modeled error to go back to (1), iterating and doing an optimization over the parameters in the expression for $\frac{dX}{dt}$ to minimize the error.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note that in the above I have to deal with the function itself and not the derivative because my measured data is on a coarser time grid than the derivative needs to be computed. For example, my derivative changes meaningfully on a scale of $\Delta t$, but my data set might only have points every $5\Delta t$ to $10 \Delta t$. I cannot take the function values themselves from the data set and create a derivative myself because of this discrepancy in the time scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this process works pretty well, and I'm able to achieve a pretty good error. However, I'm wondering if there's a better way to model the derivative $\frac{dX}{dt}$ than this analytic model. For example, would a neural network or a regression tree be able to do a better job?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how would I do this in a computationally efficient way? Let's take a neural network as an example. I'm used to having inputs and their associated outputs when training a NN with standard packages. But the problem is, if I want to model the derivative, I &lt;em&gt;don't&lt;/em&gt; have the derivative as an output to train on. I only know the function values and have only been able to compute a error to minimize on through the forward Euler integration. Of course I could treat the NN parameters as I did with the parameters above, and do a brute force minimization on them, but I obviously lose a lot of the efficiency of backprop in doing so (or, if I was doing this with a decision tree, would lose the logic of how regression trees are normally optimized).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better way for me to do this? Are there methods that exist to handle this problem? If anyone could point me in a fruitful direction it would be much appreciated. &lt;/p&gt;&#xA;" OwnerUserId="41897" LastActivityDate="2018-09-22T17:01:16.803" Title="Using standard ML models for modeling a derivative when the data set only contains function values" Tags="|machine-learning|time-series|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24740" PostTypeId="1" CreationDate="2017-11-14T21:44:36.780" Score="0" ViewCount="1091" Body="&lt;p&gt;The goal is to assess similarity and dissimilarity between 6 known groups. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original data began with the &lt;strong&gt;6&lt;/strong&gt; known groups and &lt;strong&gt;2,700+&lt;/strong&gt; variables all on a scale of 0 to 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have performed PCA to reduce the 2700+ variables into 5 principal components using the dudi.pca function from the ade4 package in R. Here are the Eigenvalues for the components:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      eigenvalue variance.percent cumulative.variance.percent&#xA;Dim.1   998.3274        36.635867                    36.63587&#xA;Dim.2   670.1278        24.591848                    61.22771&#xA;Dim.3   482.2372        17.696776                    78.92449&#xA;Dim.4   352.2806        12.927728                    91.85222&#xA;Dim.5   222.0270         8.147781                   100.00000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would now like to assess the distances between the 6 known groups. Is this done as simply as generating a distance matrix using each group's coordinates for each of the principal components? If so, I am leaning towards using Manhattan distance to get the absolute distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the coordinates of each group:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           Dim.1       Dim.2       Dim.3        Dim.4        Dim.5&#xA;Group 1    69.019038    7.940190    0.4985599  - 6.847178     0.3964117&#xA;Group 2   -16.302322  -25.965373  -29.3084201  -23.013430     9.9183010&#xA;Group 3   -26.313850   50.159662    6.9486408  -10.713924     5.2883152&#xA;Group 4   -12.800767  -26.211432   39.5067264  - 8.775551   - 8.8840592&#xA;Group 5   - 9.228404    2.648632  -20.4297314   16.685426   -26.8559444&#xA;Group 6   - 4.373694   -8.571679    2.7842244   32.664657    20.1369757&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If not, what would be the appropriate way to assess individual similarity/ dissimilarity post PCA?&lt;/p&gt;&#xA;" OwnerUserId="41460" LastActivityDate="2023-12-14T05:02:35.357" Title="Assessing Group Similarities and Dissimilarities Post PCA" Tags="|r|similarity|pca|distance|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24741" PostTypeId="2" ParentId="24738" CreationDate="2017-11-14T21:53:27.157" Score="3" Body="&lt;h3&gt;Answer&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any heuristics for deciding on where to start with the number of layers for a neural network? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No there isn't one in general. The architecture of the network (including depth) is considered to the new &quot;feature engineering&quot;. It's more of an art than science at the moment. Anything above 3 layers worth considering. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to add a bunch of polynomial features &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You don't need to do this kind of feature engineering like you would in traditional machine learning. If you have a network that is sufficiently wide and deep, your network should be able to &quot;engineer&quot; those features itself. Just think about how the first hidden layer is a linear combination of input features. And the second hidden layers is a linear combination of the first hidden layer. It should be obvious to see how a neural network is capable of structuring any polynomial features itself.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Suggestion&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Try using the original 60 features directly and start with 3 layers. Given the size of the network, the training should take less than 10 minutes on a GPU-enabled machine. Compare your result with your benchmark model (you should always have a benchmark model). Play around with the hidden size and layers, rinse and repeat. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-14T21:53:27.157" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="24742" PostTypeId="2" ParentId="24739" CreationDate="2017-11-15T00:22:55.597" Score="1" Body="&lt;p&gt;This is actually really easy to implement in any deep learning framework with automatic differentiation capability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need two neural networks. One to approximate the $X$, another to approximate the $dX/dt$.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first step is easy. You have samples of input and target for $X$.&#xA;Just follow the normal process to train you network to learn $X$&lt;/li&gt;&#xA;&lt;li&gt;For the second step is that you don't have the target values (i.e. gradient) directly. The trick is that you already got an approximator for $X$ from the first step, so use that to get your target for the second network. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am most familiar with &lt;code&gt;pytorch&lt;/code&gt;, so I will use it as an example. &lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import torch&#xA;from torch.nn import Sequential, Linear, ReLU&#xA;from torch.autograd import Variable&#xA;from torch.optim import Adam&#xA;import numpy as np&#xA;&#xA;x = np.linspace(0, 10, 101).reshape(-1, 1)&#xA;y = np.sin(x) # Using sin as an example&#xA;&#xA;x = Variable(torch.Tensor(x), requires_grad=True) &#xA;y = Variable(torch.Tensor(y), requires_grad=False)&#xA;&#xA;f = Sequential(....) # Build f approximator&#xA;f_optimizer = Adam(f.parameters())&#xA;&#xA;for epoch in range(1000):&#xA;    loss= f(x) - y&#xA;    optimizer.zero_grad()&#xA;    loss.backward()&#xA;    optimizer.step()&#xA;&#xA;# Build grad approximator&#xA;f_grad = Sequential(....) &#xA;&#xA;# need a new optimizer to train f_grad&#xA;f_optimizer = Adam(f_grad.parameters()) &#xA;&#xA;for epoch in range(1000):&#xA;    # Directly backprop from the output to the gradient of the f&#xA;    model(x).backward() &#xA;    loss= f_grad(x) - x.grad # x.grad is the grad of f w.r.t x&#xA;    f_optimizer.zero_grad()&#xA;    loss.backward()&#xA;    f_optimizer.step()  &#xA;&#xA;f_grad # this is the gradient function you are looking for&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I don't have a machine with &lt;code&gt;pytorch&lt;/code&gt; install at the moment, so don't expect the code to run with some debugging)&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-15T00:22:55.597" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="24743" PostTypeId="2" ParentId="24728" CreationDate="2017-11-15T00:41:34.073" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I know that neural networks can deal pretty well with outliers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not necessarily. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on your loss function and sample size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In linear regression, the loss function is MSE. MSE is not robust against outliers, so linear regression is not robust against outliers. Having larger sample will mitigate the impact of a outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same principles applies to neural network.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there choices in architecture / training which help the smoothing / averaging / removal of noise?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's just the problem of over fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@tom has covered the most commonly used techniques. The only thing I will add is  using larger sample size. Sometime this can be achieve cheaply by using data augmentation techniques (i.e. image rotation etc).  &lt;/p&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-15T09:10:55.147" LastActivityDate="2017-11-15T09:10:55.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24744" PostTypeId="2" ParentId="24738" CreationDate="2017-11-15T01:51:35.717" Score="1" Body="&lt;p&gt;Yes, I would also totally agree with &lt;a href=&quot;https://datascience.stackexchange.com/users/41525/louis-t&quot;&gt;Louis T&lt;/a&gt;, as he has suggested it is better if you start with 3 hidden layers and see how the model is performing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are couple of things which I would like to add to his answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you to do feature engg. on data so that you can make better sense out of it, rather then model taking over that task, one of the major reason for that is sometimes machine cannot give some meaningful features with respect to the business and the other important thing would be &lt;strong&gt;Cost Function&lt;/strong&gt;, as the model increases its &lt;strong&gt;Complexity&lt;/strong&gt; the more the cost function would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how do you decide which model would give you better results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, there are 2 networks(assuming using NN for Prediction) :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Net-1: hidden layers = 3 and RMSE = 3.4500&lt;/li&gt;&#xA;&lt;li&gt;Net-2: hidden layers = 4 and RMSE = 3.4200&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you we look closely RMSE for Net-2 is less but considering the scenario WRT to hidden layers, I would go with Net-1 because we are considering the Complexity and Cost Function for deciding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope my answer is helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NN : Neural Network&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RMSE : Root Mean Square Error&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-15T01:51:35.717" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24745" PostTypeId="2" ParentId="24734" CreationDate="2017-11-15T02:14:20.153" Score="1" Body="&lt;p&gt;It would be great if you can find any trends which are similar, as you know a generalized model gives you better result when compared to the model which is built for a specific purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1st question is how many data points do you have?, assuming that you have good amount of data to train for each model in each product then it is fine(you can go ahead with your current process) but in a scenario where you have less data points then, you can find if there is a trend similar is followed in 2 different products(will be outcome of your Exploratory Analysis), then you can combine such data together, by this you are generalizing the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/29451/dan-jarratt&quot;&gt;Dan Jarratt&lt;/a&gt; has said, I would also agree with him. Ensemble Model would  help you in getting better results and with good Accuracy, for better understanding you can go through this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/02/introduction-to-ensembling-along-with-implementation-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; on Ensemble modelling in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-15T02:14:20.153" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="24746" PostTypeId="2" ParentId="18848" CreationDate="2017-11-15T03:19:48.630" Score="6" Body="&lt;p&gt;You're on the right track. Look at calculating a few more features, both in time and frequency domain. As long as number of samples &gt;&gt; number of features, you aren't likely to overfit. Is there any literature on a similar problem? If so, that always provides a great starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try a boosted tree classifier, like xgboost or LightGBM. They tend to be easier to tune hyperparameters, and provide good results with default parameters. Both Random Forest and boosted tree classifiers can return feature importance, so you can see which features are relevant to the problem. You can also try removing features to check for any covariance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most importantly though, if your results are unexpectedly poor, ensure your problem is properly defined. Manually check through your results to make sure there aren't any bugs in your pipeline.&lt;/p&gt;&#xA;" OwnerUserId="39845" LastActivityDate="2017-11-15T03:19:48.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24747" PostTypeId="2" ParentId="24738" CreationDate="2017-11-15T03:26:46.820" Score="0" Body="&lt;p&gt;Don't worry about engineering features for a neural network. The point of them is that they learn the features well themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell, first, make sure you have a representative validation set that you check regularly. Then, build a network large enough to overfit, and one small enough to underfit. The sweet spot is somewhere in the middle; find that based on minimisation of validation loss. &lt;/p&gt;&#xA;" OwnerUserId="39845" LastActivityDate="2017-11-15T03:26:46.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24749" PostTypeId="2" ParentId="24727" CreationDate="2017-11-15T04:04:47.753" Score="0" Body="&lt;p&gt;I think you might have heard something called as &lt;a href=&quot;http://r-statistics.co/Association-Mining-With-R.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Association Mining&lt;/a&gt;, even if you don't know please go through the link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the reason for me to suggest this is, before giving the raw transactions to the model. We need to convert them into a baskets i.e., if a customer bought 3 items then all of them will fall under the same basket using Customer ID(any unique identifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be done by a command, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;transDat &amp;lt;- as (myDataFrame, &quot;transactions&quot;) # convert to 'transactions' class&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By this you will get the baskets that is nothing but your desired outcome. Once that is done you can directly apply Association Mining to get the desired recommendations and select the &lt;strong&gt;Best Recommendations&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have anymore doubts.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-15T05:57:28.813" LastActivityDate="2017-11-15T05:57:28.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24750" PostTypeId="1" AcceptedAnswerId="24761" CreationDate="2017-11-15T04:07:00.537" Score="0" ViewCount="1180" Body="&lt;p&gt;This is my first time implementing a Machine Learning Algorithm in Python. I tried implementing K-Means using Python and Sklearn for this &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/seeds&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import KMeans&#xA;import numpy as np&#xA;import pandas as pd&#xA;from matplotlib import pyplot as plt&#xA;&#xA;# Importing the dataset&#xA;data = pd.read_csv('dataset.csv')&#xA;print(&amp;quot;Input Data and Shape&amp;quot;)&#xA;print(data.shape)&#xA;data.head()&#xA;&#xA;&#xA;# Getting the values and plotting it&#xA;f1 = data['Area'].values&#xA;f2 = data['perimeter'].values&#xA;f3 = data['Compactness'].values&#xA;f4 = data['length_kernel'].values&#xA;f5 = data['width_kernel'].values&#xA;f6 = data['asymmetry'].values&#xA;f7 = data['length_kernel_groove'].values&#xA;&#xA;&#xA;&#xA;X = np.array(list(zip(f1,f2,f3,f4,f5,f6,f7)))&#xA;# Number of clusters&#xA;kmeans = KMeans(n_clusters=7)&#xA;kmeans = kmeans.fit(X)&#xA;# Getting the cluster labels&#xA;labels = kmeans.predict(X)&#xA;# Centroid values&#xA;centroids = kmeans.cluster_centers_&#xA;&#xA;plt.scatter(X[:,0], X[:,1],cmap='rainbow')&#xA;plt.scatter(centroids[:,0], centroids[:1], color=&amp;quot;black&amp;quot;, marker='*')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The graph doesn't seem to plot the data correctly. How can I debug this issue?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://prnt.sc/ham828&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2oMVC.png&quot; alt=&quot;Image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;" OwnerUserId="41913" LastEditorUserId="25180" LastEditDate="2023-01-17T17:28:41.763" LastActivityDate="2023-01-17T17:28:41.763" Title="Wrong Graph Plot using K-Means in Python" Tags="|python|scikit-learn|clustering|k-means|plotting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24751" PostTypeId="2" ParentId="24725" CreationDate="2017-11-15T05:37:46.587" Score="1" Body="&lt;p&gt;If you want to experiment something like that I think you can add some outliers into your data for that respective field and see what are its effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you find that these are prone to outliers(most of of the features are), then you can handle situation by removing such outliers before appending to the existing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your target variable?&#xA;Did you do any kind of Correlation analysis/ Predictor Importance?&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-15T05:37:46.587" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24752" PostTypeId="1" AcceptedAnswerId="24773" CreationDate="2017-11-15T05:43:35.410" Score="2" ViewCount="676" Body="&lt;p&gt;1 Is there a way to handle data imbalance? ie if data in each class for training is not balanced, say some classes have 50 documents some other have 200 documents. How to handle this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 How to handle the classification problem with a large number of classes? I have around 50+ classes (may increase once more data is available) to learn. I am trying out different algorithms and features. Is there any way to handle classification problem with a large number of classes?&lt;/p&gt;&#xA;" OwnerUserId="5091" LastActivityDate="2017-11-16T09:42:30.670" Title="Handling data imbalance and class number for classification" Tags="|machine-learning|classification|data-mining|nlp|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24753" PostTypeId="2" ParentId="24740" CreationDate="2017-11-15T05:49:11.240" Score="0" Body="&lt;p&gt;What does the distance between groups &lt;em&gt;mean&lt;/em&gt; for your problem? Answering that problem would help you pick a distance metric.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you pick e.g. Euclidean distance, one simple way to find inter-group distance would be to first calculate the group centroid, i.e. average location of each group by averaging the locations of each group together.  You can do this by just averaging all the 5x1 vectors belonging to a group, and repeat for each group.  Then calculate the Euclidean distance between the resulting 6 centroids, giving you a 6x6 distance matrix.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-15T05:49:11.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24754" PostTypeId="2" ParentId="24723" CreationDate="2017-11-15T07:32:51.257" Score="1" Body="&lt;p&gt;You need to define best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If best means &lt;strong&gt;being exactly sure of your labeling&lt;/strong&gt;, then you'd have to manually go through all messages, use crowdsourcing and what not. This is assuming that the chat messages are in a language that you can understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If best means &lt;strong&gt;automating the process&lt;/strong&gt;, you could use an existing sentiment analysis model to predict the sentiment of each message and see if it lines up with the existing labeling. Of course, perfect alignment is unlikely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If best means &lt;strong&gt;doing it from scratch&lt;/strong&gt;, then I would try to cluster the sentences into 3 groups and then look at the composition of the groups. Of course, this is highly dependent on the clustering and how you define the distances between the messages&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2017-11-15T07:32:51.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24756" PostTypeId="2" ParentId="23591" CreationDate="2017-11-15T09:43:36.307" Score="0" Body="&lt;p&gt;&lt;strong&gt;Procedure-1&lt;/strong&gt; :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it would be better if you can try combining some geners, it is most likely that some follow similar trend, once you identify them try combining them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use some Dimensionality reduction, then you can make more sense out of the data, as of now even if you give directly also it might take time for the model to understand and give some useful results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get the outcome of Dimensionality reduction you can directly apply multi class classification algorithms like &lt;strong&gt;SVM&lt;/strong&gt;, &lt;strong&gt;RF&lt;/strong&gt; and many more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Procedure-2&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing which you can try is, You can concatenate all the features(Gener's) into 1 single feature and try try understanding and see if does makes any sense/ get some good insights(Exploratory Analysis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVM : Support Vector Machine&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RF : Random Forest&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-15T09:43:36.307" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24758" PostTypeId="1" CreationDate="2017-11-15T11:00:26.433" Score="2" ViewCount="90" Body="&lt;p&gt;I am using a three node system:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Master node: &lt;span class=&quot;math-container&quot;&gt;$64$&lt;/span&gt; GB RAM&lt;/li&gt;&#xA;&lt;li&gt;2 slave node: &lt;span class=&quot;math-container&quot;&gt;$32$&lt;/span&gt; GB RAM&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function of Apache spark then &lt;span class=&quot;math-container&quot;&gt;$64$&lt;/span&gt; GB memory of master node got totally consumed and start to use swap memory which makes execution too slow and eventually lead to heap memory full.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If matrix size is &lt;span class=&quot;math-container&quot;&gt;$3000 \times 3000$&lt;/span&gt;   then no issue but when we consider size greater than &lt;span class=&quot;math-container&quot;&gt;$3000$&lt;/span&gt; (like &lt;span class=&quot;math-container&quot;&gt;$3500$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$5000$&lt;/span&gt;) then above-mentioned issue arises.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: &lt;strong&gt;Even my MATLAB is able to compute the inverse of &lt;span class=&quot;math-container&quot;&gt;$10K \times 10K$&lt;/span&gt; matrix on 32 GB RAM but Apache Spark got Memory full if I used inbuilt SVD function on above system.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Code is as below:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.linalg.{Vectors,Vector,Matrix,SingularValueDecomposition,DenseMatrix,DenseVector}&#xA;import org.apache.spark.mllib.linalg.distributed.RowMatrix&#xA;&#xA;def computeInverse(X: RowMatrix): DenseMatrix = {&#xA;  val nCoef = X.numCols.toInt&#xA;  val svd = X.computeSVD(nCoef, computeU = true)&#xA;  if (svd.s.size &amp;lt; nCoef) {&#xA;    sys.error(s&quot;RowMatrix.computeInverse called on singular matrix.&quot;)&#xA;  }&#xA;&#xA;  // Create the inv diagonal matrix from S &#xA;  val invS = DenseMatrix.diag(new DenseVector(svd.s.toArray.map(x =&amp;gt; math.pow(x,-1))))&#xA;&#xA;  // U cannot be a RowMatrix&#xA;  val U = new DenseMatrix(svd.U.numRows().toInt,svd.U.numCols().toInt,svd.U.rows.collect.flatMap(x =&amp;gt; x.toArray))&#xA;&#xA;  // If you could make V distributed, then this may be better. However its alreadly local...so maybe this is fine.&#xA;  val V = svd.V&#xA;  // inv(X) = V*inv(S)*transpose(U)  --- the U is already transposed.&#xA;  (V.multiply(invS)).multiply(U)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18818" LastEditorUserId="30472" LastEditDate="2019-02-22T21:38:21.570" LastActivityDate="2019-02-22T21:38:21.570" Title="Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?" Tags="|bigdata|apache-spark|apache-hadoop|scala|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="24759" PostTypeId="1" AcceptedAnswerId="24763" CreationDate="2017-11-15T11:02:30.677" Score="1" ViewCount="6145" Body="&lt;p&gt;I am a complete beginner in coding and machine learning, and I've been tasked with learning what's under the hood of logistic regression (so I have pieced together the python code below) but I've been asked to figure out how to add bias into this code. I'm completely stuck on at what point it would need to be added into, I think at the point I am defining the hypothesis function - but if anyone would be able to point me in the right direction to figure this out I would really appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it helps, this logistic regression is being used to classify if a tumour is benign of malignant with the wisconsin breast cancer dataset (&lt;a href=&quot;https://www.kaggle.com/uciml/breast-cancer-wisconsin-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/uciml/breast-cancer-wisconsin-data&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)&#xA;&#xA;X = data[&quot;diagnosis&quot;].map(lambda x: float(x))&#xA;&#xA;X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]&#xA;X = np.array(X)&#xA;X = min_max_scaler.fit_transform(X)&#xA;Y = data[&quot;diagnosis&quot;].map(lambda x: float(x))&#xA;Y = np.array(Y)&#xA;&#xA;&#xA;def Sigmoid(z):&#xA;    if z &amp;lt; 0:&#xA;        return 1 - 1/(1 + math.exp(z))&#xA;    else:&#xA;        return 1/(1 + math.exp(-z))&#xA;def Hypothesis(theta, x):&#xA;    z = 0&#xA;    for i in range(len(theta)):&#xA;    z += x[i]*theta[i]&#xA;return Sigmoid(z)enter preformatted text here&#xA;&#xA;def Cost_Function(X,Y,theta,m):&#xA;sumOfErrors = 0&#xA;for i in range(m):&#xA;    xi = X[i]&#xA;    hi = Hypothesis(theta,xi)&#xA;    error = Y[i] * math.log(hi if  hi &amp;gt;0 else 1)&#xA;    if Y[i] == 1:&#xA;        error = Y[i] * math.log(hi if  hi &amp;gt;0 else 1)&#xA;    elif Y[i] == 0:&#xA;        error = (1-Y[i]) * math.log(1-hi  if  1-hi &amp;gt;0 else 1)&#xA;    sumOfErrors += error&#xA;&#xA;const = -1/m&#xA;J = const * sumOfErrors&#xA;print ('cost is: ', J ) &#xA;return J&#xA;&#xA;def Cost_Function_Derivative(X,Y,theta,j,m,alpha):&#xA;sumErrors = 0&#xA;for i in range(m):&#xA;    xi = X[i]&#xA;    xij = xi[j]&#xA;    hi = Hypothesis(theta,X[i])&#xA;    error = (hi - Y[i])*xij&#xA;    sumErrors += error&#xA;m = len(Y)&#xA;constant = float(alpha)/float(m)&#xA;J = constant * sumErrors&#xA;return J&#xA;&#xA;def Gradient_Descent(X,Y,theta,m,alpha):&#xA;new_theta = []&#xA;constant = alpha/m&#xA;for j in range(len(theta)):&#xA;    CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)&#xA;    new_theta_value = theta[j] - CFDerivative&#xA;    new_theta.append(new_theta_value)&#xA;return new_theta&#xA;&#xA;initial_theta = [0,1]  &#xA;alpha = 0.01&#xA;iterations = 1000&#xA;Logistic_Regression(X,Y,alpha,initial_theta,iterations)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41922" LastActivityDate="2017-11-15T12:57:56.323" Title="How to add bias consideration into logistic regression code?" Tags="|machine-learning|python|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24760" PostTypeId="1" CreationDate="2017-11-15T11:17:32.773" Score="13" ViewCount="8774" Body="&lt;p&gt;I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used  techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example papers which define the task would be welcome.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2021-12-04T06:42:06.030" Title="What is the difference between outlier detection and anomaly detection?" Tags="|algorithms|anomaly-detection|outlier|terminology|definitions|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24761" PostTypeId="2" ParentId="24750" CreationDate="2017-11-15T11:31:39.863" Score="1" Body="&lt;p&gt;Well, there are some issues:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Dimension vs K: Before talking about visualization I would like to address some clustering concept. Your data is in 7 dimensions but it does not mean that you have 7 clusters! Be careful here. For instance I have two features of people let's say salary and number of years they have working experience. Here I have two features but does it mean that there necessary two categories inside the data? sure not!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Visualization: Your data is in 7 dimension which is not visualizable. So you decided to reduce this to two which is a correct approach but you did a wrong thing for this correct approach. You can not take the first two features to visualize 7 dimensions, you need to REDUCE it to two features using Dimensionality Reduction algorithms like PCA, NMF, etc. What you did is actually IGNORING 5 dimensions of the points which are extremely informative for placing them in a 7-dimensional space.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Everything is right. Just add a PCA to your code like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;From sklearn.decomposition import PCA&#xA;Model = PCA(n_components=2)&#xA;X_new = Model.fit_transform(X)&#xA;... Use X_new instead of X for K-means procedure&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note that I wrote this relying on my memory so better to check the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; if I had a typo or smth. In case you have more question you can comment here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-11-15T11:31:39.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24763" PostTypeId="2" ParentId="24759" CreationDate="2017-11-15T12:57:56.323" Score="5" Body="&lt;p&gt;If I understood correctly, by &lt;em&gt;bias&lt;/em&gt; you mean the intercept term in your model, that is, $\alpha$ in the equation&#xA;$$&#xA;p(y=1|x) = \frac{1}{1+e^{-(\alpha + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is the case, you can easily incorporate intercept by adding a colum of ones into your &lt;code&gt;X&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.hstack([np.ones([X.shape[0],1]), X])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the first colum of $X$ is always one. Then, your model will look like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;p(y=1|x) = \frac{1}{1+e^{-(\beta_0 x_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}}&#xA;$$&#xA;where $\beta_0$ is your bias term and $x_0$ always equals $1$. Thus, your model would be equivalent to the one with explicit intercept.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-15T12:57:56.323" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24764" PostTypeId="1" CreationDate="2017-11-15T12:58:37.357" Score="1" ViewCount="58" Body="&lt;p&gt;I've been using GIST, HOG and SURF descriptors for extracting features from different collections of Chest-X-rays and measuring performance using accuracy and area under the curve. These collections are obtained using different machinery, from different medical institutions, and, with different pixel resolutions. I could repeatedly see that one descriptor performs better than the other and the performance is not the same across these collections though all are frontal chest X-rays. What attributes to these differences in performance across the collections though they are from the same imaging modality?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="41557" LastEditDate="2017-11-16T13:18:27.293" LastActivityDate="2017-11-16T13:18:27.293" Title="Why is there a difference in performance across the feature descriptors for the same imaging modality?" Tags="|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24765" PostTypeId="1" AcceptedAnswerId="24769" CreationDate="2017-11-15T13:06:14.567" Score="0" ViewCount="1253" Body="&lt;p&gt;I have serious doubts concerning the features standardization done before the learning process of a multilayer perceptron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using python-3 and the scikit-learn package for the learning process and for the features normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As suggested from the scikit-learn wiki &lt;a href=&quot;http://scikit-learn.org/stable/modules/neural_networks_supervised.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;(Tips on pratical use)&lt;/a&gt;, I'm doing a features standardization with the Preprocessing module, which means that all my features will appear as standard normal specifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that in the showed example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import StandardScaler  &#xA;scaler = StandardScaler()  &#xA;# Don't cheat - fit only on training data&#xA;scaler.fit(X_train)  &#xA;X_train = scaler.transform(X_train)  &#xA;# apply same transformation to test data&#xA;X_test = scaler.transform(X_test)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The standardization is done first on the training set, and then, after that, it's applied on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which means, that the distributions parameters (mean, st.deviation) will be estimated only from the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, because I need to save my multiple datasets in a database, on which I will perform the hold-out sampling for the training-test folds, I have done a preprocessing on all my instances at once, because I don't want to do that at run time execution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's clear that in this way the distributions parameters will change and the results will be different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, Someone knows the differences between the two approaches? Are there issues doing the preprocessing at once on all the dataset?&lt;/p&gt;&#xA;" OwnerUserId="15883" LastEditorUserId="15883" LastEditDate="2017-11-15T13:19:56.083" LastActivityDate="2017-11-15T14:16:57.897" Title="Features standardization - Multilayer perceptron" Tags="|machine-learning|python|scikit-learn|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24766" PostTypeId="1" AcceptedAnswerId="24767" CreationDate="2017-11-15T13:48:26.123" Score="1" ViewCount="252" Body="&lt;p&gt;I am a complete beginner in machine learning and coding in python. I have been tasked with coding logistic regression from scratch in comparison with using sklearn. My question is, with my code below I believe I have set the number of thetas I want with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]&quot;  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I am unsure how to prove this is true with my code below and its definition of theta, if I added more parameters (e.g. all 31 variables of this dataset [&lt;a href=&quot;https://www.kaggle.com/uciml/breast-cancer-wisconsin-data]&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/uciml/breast-cancer-wisconsin-data]&lt;/a&gt; which is for classifying tumours) would I just need to add them into this list above? Any help pointing me towards the right direction just to understand this better would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)&#xA;&#xA;X = data[&quot;diagnosis&quot;].map(lambda x: float(x))&#xA;&#xA;X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]&#xA;X = np.array(X)&#xA;X = min_max_scaler.fit_transform(X)&#xA;Y = data[&quot;diagnosis&quot;].map(lambda x: float(x))&#xA;Y = np.array(Y)&#xA;&#xA;&#xA;def Sigmoid(z):&#xA;    if z &amp;lt; 0:&#xA;        return 1 - 1/(1 + math.exp(z))&#xA;    else:&#xA;        return 1/(1 + math.exp(-z))&#xA;def Hypothesis(theta, x):&#xA;    z = 0&#xA;    for i in range(len(theta)):&#xA;    z += x[i]*theta[i]&#xA;return Sigmoid(z)enter preformatted text here&#xA;&#xA;def Cost_Function(X,Y,theta,m):&#xA;    sumOfErrors = 0&#xA;    for i in range(m):&#xA;        xi = X[i]&#xA;        hi = Hypothesis(theta,xi)&#xA;        error = Y[i] * math.log(hi if  hi &amp;gt;0 else 1)&#xA;        if Y[i] == 1:&#xA;            error = Y[i] * math.log(hi if  hi &amp;gt;0 else 1)&#xA;        elif Y[i] == 0:&#xA;            error = (1-Y[i]) * math.log(1-hi  if  1-hi &amp;gt;0 else 1)&#xA;        sumOfErrors += error&#xA;&#xA;    const = -1/m&#xA;    J = const * sumOfErrors&#xA;    print ('cost is: ', J ) &#xA;    return J&#xA;&#xA;def Cost_Function_Derivative(X,Y,theta,j,m,alpha):&#xA;    sumErrors = 0&#xA;    for i in range(m):&#xA;        xi = X[i]&#xA;        xij = xi[j]&#xA;        hi = Hypothesis(theta,X[i])&#xA;        error = (hi - Y[i])*xij&#xA;        sumErrors += error&#xA;    m = len(Y)&#xA;    constant = float(alpha)/float(m)&#xA;    J = constant * sumErrors&#xA;    return J&#xA;&#xA;def Gradient_Descent(X,Y,theta,m,alpha):&#xA;    new_theta = []&#xA;    constant = alpha/m&#xA;    for j in range(len(theta)):&#xA;        CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)&#xA;        new_theta_value = theta[j] - CFDerivative&#xA;        new_theta.append(new_theta_value)&#xA;    return new_theta&#xA;&#xA;def Logistic_Regression(X,Y,alpha,theta,num_iters):&#xA;    m = len(Y)&#xA;    for x in range(num_iters):&#xA;        new_theta = Gradient_Descent(X,Y,theta,m,alpha)&#xA;        theta = new_theta&#xA;        if x % 100 == 0:&#xA;            Cost_Function(X,Y,theta,m)&#xA;            print ('theta: ', theta)    &#xA;            print ('cost is: ', Cost_Function(X,Y,theta,m))&#xA;&#xA;initial_theta = [0,1]  &#xA;alpha = 0.01&#xA;iterations = 1000&#xA;Logistic_Regression(X,Y,alpha,initial_theta,iterations)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41922" LastActivityDate="2017-11-15T14:21:30.130" Title="How to add more theta parameters into my logistic regression?" Tags="|machine-learning|python|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24767" PostTypeId="2" ParentId="24766" CreationDate="2017-11-15T13:58:16.417" Score="0" Body="&lt;p&gt;Your pass your &lt;code&gt;initial_theta&lt;/code&gt; into &lt;code&gt;Logistic_Regression&lt;/code&gt; where it defines how the cost function and its derivative are evaluated. Just make &lt;code&gt;initial_theta&lt;/code&gt; the same width as &lt;code&gt;X&lt;/code&gt;. If you want your code to be fool-proof, check variable sizes within your function like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def Logistic_Regression(X,Y,alpha,theta,num_iters):&#xA;    assert len(theta) == X.shape[1], 'theta should have one coefficient per each column of X'&#xA;    ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A more convenient solution would be to define &lt;code&gt;initial_theta&lt;/code&gt; &lt;em&gt;inside&lt;/em&gt; the function, based on &lt;code&gt;X&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def Logistic_Regression(X,Y,alpha,num_iters):&#xA;    theta = np.zeros(X.shape[1])&#xA;    ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, you can be sure that the shape of initial $\theta$ is correct.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-11-15T14:21:30.130" LastActivityDate="2017-11-15T14:21:30.130" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24769" PostTypeId="2" ParentId="24765" CreationDate="2017-11-15T14:16:57.897" Score="1" Body="&lt;p&gt;Generally, preprocessing parameters are fit only on the training subset, because otherwise you could overfit your data and overestimate quality of your model on the test subset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With feature standardization, however, overfitting is not so dangerous, so I assume you can preprocess all your dataset at once safely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best practice, however, is to do all the processing in one &lt;a href=&quot;http://scikit-learn.org/stable/modules/pipeline.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pipeline&lt;/a&gt; and apply cross-validation to the whole pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.pipeline import make_pipeline&#xA;from sklearn.neural_network import MLPClassifier&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.model_selection import cross_val_score, GridSearchCV&#xA;scaler = StandardScaler()  &#xA;model = MLPClassifier()&#xA;pipeline = make_pipeline(scaler, model)&#xA;scores = cross_val_score(pipeline, X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you use scikit-learn cross-validation, incorporating pipelines would be simple.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-15T14:16:57.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24771" PostTypeId="2" ParentId="24690" CreationDate="2017-11-15T14:56:15.803" Score="0" Body="&lt;p&gt;The question &quot;how NN can be applied if not in machine learining&quot; is incorrect. It's like &quot;how cats can be applied if not in animals&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cats are animals. And &lt;strong&gt;neural networks &lt;em&gt;are&lt;/em&gt; a kind of machine learning&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a meaningful question is &quot;where machine learning can be applied?&quot;, or &quot;where statistics can be applied?&quot;, or &quot;where math can be applied?&quot;. And the answer is: everywhere.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-15T14:56:15.803" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24773" PostTypeId="2" ParentId="24752" CreationDate="2017-11-15T15:30:59.040" Score="6" Body="&lt;p&gt;People talk a lot about data imbalance, but in general I think you don't need to worry about it unless your data is &lt;em&gt;really&lt;/em&gt; imbalanced (like &amp;lt;1% of one label).  50/200 is fine. If you build a logistic regression model on that dataset, the model will be biased towards the majority class - but if you gave me no information about an input to classify, the prior probability is that the new input is a member of the majority class anyway. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question you want to be able to answer is whether you are &lt;em&gt;differentiating&lt;/em&gt; classes fine - so if you do have a minority class, do NOT use 'accuracy' as a metric. Use something like area under the ROC curve (commonly called AUC) instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data is really super imbalanced, you can either over-sample the minority class or use something called 'SMOTE', for &quot;Synthetic Minority Over-Sampling Technique&quot;, which is a more advanced version of the same thing.  Some algorithms also let you set higher weights on minority classes, which essentially incentivizes the model to pay attention to the minority class by making minority-class errors cost more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To learn to differentiate between lots of classes, I think (a) you will need to have a ton of examples to learn from and (b) a model that's expressive enough to capture class differences (like deep neural network, or boosted decision tree), and (c) use softmax output. If those still don't work, you might try a 'model-free' approach like K-nearest-neighbors, which matches each input to the most similar labeled data.  For kNN to work however, you need to have a very reasonable distance metric.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-15T15:30:59.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24775" PostTypeId="2" ParentId="20245" CreationDate="2017-11-15T17:52:26.350" Score="1" Body="&lt;p&gt;Before jumping into machine learning solutions, it would be good to think more about the problem you're solving. If there are only 20 games and some are unavailable at any given time, then a well laid-out menu with good navigation is superior to a recommender system. Recommender systems are only appropriate when people cannot adequately parse all of the available options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want personalized recommendations, you don't even have to start with machine learning models. You can simply recommend that players keep playing the same games or the most popular games.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if it turns out that machine learned models are best, I suggest looking at association rule mining based on unary data (which gives you shopping-basket recommendations: people who played games A, B, and C also played games D and E) or some variety of collaborative filtering based on ratings data (which gives you a user-item preference space). That totally depends on what sort of feedback you get from users about their individual game experiences.&lt;/p&gt;&#xA;" OwnerUserId="29451" LastActivityDate="2017-11-15T17:52:26.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24776" PostTypeId="2" ParentId="24764" CreationDate="2017-11-15T17:54:41.940" Score="1" Body="&lt;p&gt;Feature-extraction mechanisms like GIST, HOG, etc are built and optimized to improve performance on given datasets.  Because of this, they don't perform as well across datasets.  It's kind of like putting specialized fuel in a vehicle that isn't built to utilize it - it might even do harm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hand-engineered features are, as a rule, brittle.  I once heard it said that the dirty secret of machine learning is just knowing how to transform your domain-specific information into meaningful features - after that, you can use an extremely simple classifier and it may do surprisingly well.  The drawback is that the rules you built are very specific to your domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep neural networks, and convolutional neural networks in particular, are an advancement in that they &lt;em&gt;learn&lt;/em&gt; what features are useful about raw data - for CNNs, these are the raw pixel or time-series values.  Instead of hand-building feature extraction mechanisms, these architectures &lt;em&gt;automatically&lt;/em&gt; build them.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One benefit of this is that if you use a CNN to identify images in general, you can re-use the top few feature extraction layers of the CNN on a different image recognition dataset, and re-train the bottom few layers to make the network specific to recognizing e.g. dog breeds.  You can transfer what you learned about the statistical structure of natural images in general to other, more specific questions (general -&gt; specific).  In your case, the 'top few layers' are analogous to your GIST/HOG methods - and they wouldn't be expected to perform well when the task changes, because they were constructed for a specific task (specific -&gt; other specific).&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-15T17:54:41.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24777" PostTypeId="1" CreationDate="2017-11-15T18:01:46.697" Score="3" ViewCount="201" Body="&lt;p&gt;I have the impression the former is used in ML whereas the latter is used in econometrics. They both carry the idea that information from the target is &quot;leaking&quot; in explanatory variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any difference between those two notions?&lt;/p&gt;&#xA;" OwnerUserId="41888" LastActivityDate="2021-08-20T21:56:53.867" Title="What is the difference between data leakage and endogeneity?" Tags="|data-leakage|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24780" PostTypeId="1" AcceptedAnswerId="24793" CreationDate="2017-11-15T19:30:56.623" Score="0" ViewCount="3052" Body="&lt;p&gt;I am just starting off with TensorFlow and trying to implement an RNN for a dataset which consists of 11 Features (all numeric). These features will be used to predict the output of another column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently lost on where to start and tho I am able to understand how a RNN functions all the tutorials I could find were mainly related to image and text datasets. &lt;/p&gt;&#xA;" OwnerUserId="41913" LastActivityDate="2017-11-16T03:20:21.590" Title="Reading a CSV in TensorFlow RNN" Tags="|python|tensorflow|rnn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24781" PostTypeId="1" AcceptedAnswerId="85436" CreationDate="2017-11-15T19:56:17.757" Score="4" ViewCount="2509" Body="&lt;p&gt;I would like to use &lt;a href=&quot;https://docs.orange.biolab.si/3/visual-programming/widgets/data/featureconstructor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Feature Constructor&lt;/a&gt; to extract numbers from a column of strings.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Input          Output&#xA;S.c 3224   ---&amp;gt; 3224&#xA;South      ---&amp;gt; 0&#xA;322523     ---&amp;gt; 322523&#xA;S-1c 4     ---&amp;gt; 14&#xA;-1.37      ---&amp;gt; 137&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Internal Orange 3.7.0 Documentation says &lt;em&gt;For continuous variables you only have to construct an expression in Python.&lt;/em&gt;  Great!   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found information about allowed functions in &lt;a href=&quot;https://github.com/biolab/orange3/blob/master/Orange/widgets/data/owfeatureconstructor.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;the feature constructor code&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Only expressions with no list,set,dict,generator comprehensions&#xA;#are accepted.&#xA;&amp;lt;...&amp;gt;&#xA;__ALLOWED = [&#xA;    &quot;Ellipsis&quot;, &quot;False&quot;, &quot;None&quot;, &quot;True&quot;, &quot;abs&quot;, &quot;all&quot;, &quot;any&quot;, &quot;acsii&quot;,&#xA;    &quot;bin&quot;, &quot;bool&quot;, &quot;bytearray&quot;, &quot;bytes&quot;, &quot;chr&quot;, &quot;complex&quot;, &quot;dict&quot;,&#xA;    &quot;divmod&quot;, &quot;enumerate&quot;, &quot;filter&quot;, &quot;float&quot;, &quot;format&quot;, &quot;frozenset&quot;,&#xA;    &quot;getattr&quot;, &quot;hasattr&quot;, &quot;hash&quot;, &quot;hex&quot;, &quot;id&quot;, &quot;int&quot;, &quot;iter&quot;, &quot;len&quot;,&#xA;    &quot;list&quot;, &quot;map&quot;, &quot;memoryview&quot;, &quot;next&quot;, &quot;object&quot;,&#xA;    &quot;oct&quot;, &quot;ord&quot;, &quot;pow&quot;, &quot;range&quot;, &quot;repr&quot;, &quot;reversed&quot;, &quot;round&quot;,&#xA;    &quot;set&quot;, &quot;slice&quot;, &quot;sorted&quot;, &quot;str&quot;, &quot;tuple&quot;, &quot;type&quot;,&#xA;    &quot;zip&quot;&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Based on the list, I was hoping to use an expression like this:&lt;br&gt;&#xA;&lt;code&gt;int(filter(str.isdigit, inputStringColumn) or 0)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...but that is an &quot;Invalid Expression&quot; (no further feedback)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I extract a number from a string?&lt;/p&gt;&#xA;" OwnerUserId="41478" LastActivityDate="2020-11-15T02:30:14.160" Title="Use Feature Constructor in Orange to extract number from string?" Tags="|orange|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24784" PostTypeId="2" ParentId="24760" CreationDate="2017-11-15T22:00:57.500" Score="7" Body="&lt;p&gt;Fundamentally there is no difference. Say you have data and you want to build a model of it.  As the name suggests, modeling is about finding a model, that is, a simplified representation of your data.  In turn, we can view the model as an underlying process that generated your data in the first place, plus some noise.  From that point of view, the data you see was generated by the model - and we can say that some of the points you see are less likely to have been generated by your model than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you build a linear regression model, points far away from the regression line are less likely to have been generated by the model.  That's what people mean when they talk about 'residuals' in normal statistical parlance.  It's also called the likelihood of the data.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data points that have low likelihood, according to the model you've created, are anomalies or outliers.  From a model-building point of view, they are the same thing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Colloquially, people use the term 'outlier' to mean &quot;something I should remove from the dataset so that it doesn't skew my model I'm building&quot;, usually because they have a hunch that there is something wrong with that data and the model they want to build shouldn't need to account for it.  An outlier is often considered to being a hinderance to building a model that describes the data overall - simply because the model will ALSO try to explain the outlier, which is not what the practitioner wants.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, you can use the fact that a model also assigns a likelihood to each data point to your advantage - might build a model that describes a simpler trend in the data, and then actively &lt;em&gt;look&lt;/em&gt; for existing or new values that have very low likelihood.  These are what people mean when they say 'anomalies'.  If your goal is to detect anomalies, especially in new data, this is a great thing. One person's outlier is another person's anomaly!&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-15T22:00:57.500" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24785" PostTypeId="2" ParentId="23814" CreationDate="2017-11-15T22:25:27.850" Score="3" Body="&lt;p&gt;OK, here are my attempts with &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; &amp;amp; &lt;a href=&quot;http://ggplot2.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ggplot2&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;1 Simple stacked histogram&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MkIE9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MkIE9.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;2 Dodged stacked histogram ~ bacteria&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7iyPX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7iyPX.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;3 Dodged histogram ~ Culture&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pd5Sb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pd5Sb.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;4 Dodged histogram ~ change&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zEuxX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zEuxX.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;5 Grouped by number of unique 'patterns' Change~Bacteria&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/olWfE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/olWfE.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;6 Grouped by number of unique 'patterns' Change~Bacteria, jittered&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yrPYA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yrPYA.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;7 Grouped by number of unique 'patterns' Change~Bacteria, jittered (const width)&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oeFs2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oeFs2.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;8 Grouped by number of unique 'patterns' Change~Bacteria, jittered (const width)&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WDwPU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WDwPU.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;9 Table with all patterns &amp;amp; its counts&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GuiYS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GuiYS.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Remark: the charts showing the number of &amp;quot;patterns&amp;quot; do not take into account the fact that the number of &amp;quot;negative&amp;quot; observations is significantly higher than the number of &amp;quot;positive&amp;quot; observations (146 vs 57, see the table). So in some sense the comparison is not fair, but this can be fixed easily.&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-16T21:39:13.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24786" PostTypeId="1" AcceptedAnswerId="24791" CreationDate="2017-11-15T22:48:01.450" Score="1" ViewCount="530" Body="&lt;p&gt;Here is the Sample data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Values    Attribute1    Attribute2    Attribute3    Attribute4&#xA;  2.5       1980            A            1.5           C&#xA;  1.8       2000            A            0.4           B&#xA;  12.4      2017            S            18.5          D&#xA;  0.4       1987            R            10            P&#xA;......&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my real data, I have more than 60 attributes. What I need to do is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Category Values into different groups&lt;/li&gt;&#xA;&lt;li&gt;classify these groups using Attribute1 - Attribute4. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some of the difficulties are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Values mean something in my application. For example, Values&gt;10 will be put into one group. So, it might be not ideal to use a clustering algorithm such as density based to divide Values into groups. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When I use the groups based on Values and all the attributes to run a classification, I might need to choose some important attributes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I want is to give Values(meaningful data) to a clustering algorithm. So, that I can get best results/Insights from that Clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here are my Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to cluster?&lt;/li&gt;&#xA;&lt;li&gt;How to choose attributes?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16911" LastEditorUserId="40853" LastEditDate="2017-11-16T13:12:11.633" LastActivityDate="2018-02-01T15:02:58.343" Title="Classification based on a Clustering Result" Tags="|classification|clustering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24787" PostTypeId="1" CreationDate="2017-11-15T23:01:08.807" Score="0" ViewCount="1241" Body="&lt;p&gt;I have a dataset containing thousands of text posts. I am building a binary classifier that will classify posts as &lt;strong&gt;safe (0)&lt;/strong&gt; or  &lt;strong&gt;risky (1)&lt;/strong&gt;. I randomly picked some of them and manually labeled. Label 1 is minority.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imbalanced data results in skewed performance. To overcome this, I undersample to obtain a 50%/50% distribution for both train and test set.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If I use stratified strategy (in scikit-learn) for ZeroR, it acts like &lt;strong&gt;random guesser&lt;/strong&gt;. &lt;strong&gt;Is this a good baseline&lt;/strong&gt;?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If I use &lt;strong&gt;&quot;guess 1&quot;&lt;/strong&gt; all the time, then its &lt;strong&gt;recall is 100%&lt;/strong&gt; all the time. However &lt;strong&gt;recall is the most important metric&lt;/strong&gt; for me. So other algorithms' predictions look unsuccessful. Should I ignore recall when using ZeroR and only compare accuracy?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I also would like to apply 10-fold cross validation, should I apply undersampling before splitting as well?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Or instead should I go with F1-Score without undersampling? This time ZeroR has high F1 score due to 100% Recall.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="41948" LastEditorUserId="23305" LastEditDate="2017-11-16T00:18:52.607" LastActivityDate="2017-11-16T00:18:52.607" Title="ZeroR as baseline for balanced / imbalanced datasets" Tags="|classification|cross-validation|class-imbalance|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24788" PostTypeId="2" ParentId="24787" CreationDate="2017-11-16T00:08:38.027" Score="1" Body="&lt;p&gt;You should only undersample the data that is used for training. Test data should represent the true distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have at least two possible baselines. One is ZeroR (your second point), which always predicts the majority class. Given a true distribution of (0) 10% and (1) 90%, this would give you a recall of 100% for (1) and 0% for (0) and 90% accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If rows represent the true labels and the columns the predictions, the confusion matrix looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{bmatrix}90 &amp;amp; 0\\10 &amp;amp; 0\end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other would be to flip a coin with probability 90% for class (1) (your first point but without undersampling), which gives you a recall of 90% for (1) and 10% for (0), and 82% accuracy, which is a little more balanced:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\begin{bmatrix}81 &amp;amp; 9\\9 &amp;amp; 1\end{bmatrix}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your classes are strongly imbalanced then recall and accuracy have the same problems as you noted above. F1 might be a better choice, but the best metric depends on your application.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-11-16T00:08:38.027" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24790" PostTypeId="1" CreationDate="2017-11-16T02:42:04.143" Score="0" ViewCount="82" Body="&lt;p&gt;I'm looking at a collection of problems where I need to forecast the probability of a continuous variable. My dependent variables are a combination of categorical and continuous. It looks like there are four primary candidate areas I should be reviewing and/or testing. Can anyone here shed any light on the relative strengths and risks associated with each of RNADE, Mixture Density Networks, Bayesian Additive Regression Trees, and standard Bayesian Linear Regression?&lt;/p&gt;&#xA;" OwnerUserId="41957" LastActivityDate="2017-11-16T02:42:04.143" Title="RNADE vs MDR vs BART vs Bayesian Linear Regression" Tags="|neural-network|regression|linear-regression|bayesian|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24791" PostTypeId="2" ParentId="24786" CreationDate="2017-11-16T02:43:55.753" Score="3" Body="&lt;p&gt;I think you need to do some Feature Engineering, i.e., as you explained in the question, those values mean something to your application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example : 1-3 : Bad, 4-6 : Average, 7-10 : Good&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;V1 new_V1&#xA;3   Bad&#xA;5   Average&#xA;7   Good&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Something like this, so that clustering algorithm can make sense out of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you don't have any constraint in using R.&#xA;If you have both Numeric and Categorical variable(as above) then you can usethis package : &lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType/clustMixType.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ClustMixType&lt;/a&gt;, it can understand and do the clustering based on the data fed. As traditional K Means algorithm is not applicable here as it works only for Numeric Data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is any discrepancy in data you can convert them explicitly by &#xA;&lt;code&gt;new_V1 &amp;lt;- as.factor(new_V1)&lt;/code&gt;; &#xA;if it is a numeric variable &lt;code&gt;V2 &amp;lt;- as.numeric(V2)&lt;/code&gt; before feeding the data to Clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get it, then you can use this package: &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bruota&lt;/a&gt;, use this package to get Predictor Importance(what all variables are important). This can be done only when you know the &lt;strong&gt;Target Variable&lt;/strong&gt;, Most likely your target variable would be the Clustering outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implementation is over here, please follow this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most likely you would end up getting your desired result, Do let me know if you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-16T02:43:55.753" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24792" PostTypeId="2" ParentId="11962" CreationDate="2017-11-16T03:07:49.203" Score="1" Body="&lt;p&gt;Actually you don’t need a bias if you have back propagation with at least 1 hidden layer.  For example, if your input is zero, your forward propagation will result in 0.5 (for sigmoid) but your back propagation will adjust its weight where you finally get the right answer.&lt;/p&gt;&#xA;" OwnerUserId="41960" LastActivityDate="2017-11-16T03:07:49.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24793" PostTypeId="2" ParentId="24780" CreationDate="2017-11-16T03:20:21.590" Score="2" Body="&lt;p&gt;There is no fundamental difference between using RNNs for text vs. for numeric values. In fact, text is more difficult because you have to preprocess the text to convert into numeric values first.  In any case you might have a look at this tutorial:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find this site to be pretty useful in general for examples of working with Keras.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-16T03:20:21.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24794" PostTypeId="1" AcceptedAnswerId="24795" CreationDate="2017-11-16T03:40:52.137" Score="1" ViewCount="642" Body="&lt;p&gt;Say we have a pandas series with the following values &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[np.nan, np.nan, 1, np.nan, 2, np.nan]&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the most efficient way fill the nan value with 0 in the middle. so we have  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[np.nan, np.nan, 1, 0, 2, np.nan]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other word, how to we do interpolation with a fixed value, or a &lt;code&gt;.fillna&lt;/code&gt; operation but ignore the &lt;code&gt;nan&lt;/code&gt; at the beginning and end of the array.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-16T21:06:32.697" Title="Python Pandas - Interpolation with fixed value" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24795" PostTypeId="2" ParentId="24794" CreationDate="2017-11-16T04:06:16.230" Score="4" Body="&lt;p&gt;Current solution, I am using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def interpolate_with_fixed(s, value=0):&#xA;    i = s.first_valid_index()&#xA;    j = s.last_valid_index()&#xA;    s.loc[i:j].fillna(value, inplace=True)&#xA;    return s&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41525" LastEditorUserId="41525" LastEditDate="2017-11-16T21:06:32.697" LastActivityDate="2017-11-16T21:06:32.697" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24797" PostTypeId="1" CreationDate="2017-11-16T04:40:55.043" Score="1" ViewCount="132" Body="&lt;p&gt;I am testing an LSA classifier with a randomised set of test-cases, picked from a larger set using under-sampling (to balance the classes). I am performing parametric sweeps to examine the behaviour of the classifier under different hyperparameters. I noticed early on that the results have a margin of error, making the performance fine-tuning difficult. So I implemented an averaging technique which simply runs the same parametric scenario N times with different random test-cases and averages the results - using arithmetic mean. (I believe this qualifies as cross-validation.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works well and certainly smooths out the performance charts, so I can see the parameter change effects more clearly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I recently learned about the harmonic mean, and now I suspect I should be using it for this purpose - because I am averaging a rate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it probably doesn't matter for looking at one parameter sweep chart (it's relative to itself), but it probably will matter when I come to compare different classifiers (absolute comparison of scores).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am not talking about using harmonic mean to produce F1 scores from TPR, FPR etc. I am only talking about using it to average these performance metrics over multiple repeats of the entire test process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These charts show &quot;confusion matrix&quot; scores when sweeping a param.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first chart shows a single run. It looks like there might be a performance sweet-spot at k=70 but I'm pretty sure this is just noise due to the random selection of the test-cases (and the limited number I have).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NP3aT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NP3aT.png&quot; alt=&quot;Performance metrics - single run&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the same test run 10 times on different test-cases. (You can see the apparent minimum has gone, the results are smoother and even the ACd trend line is flat. This shows this range of k param isn't affecting performance.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is averaged by arithmetic mean, I'm wondering if harmonic mean is more correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Pmrsy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Pmrsy.png&quot; alt=&quot;Performance metrics - 10 repeat mean&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32940" LastEditorUserId="32940" LastEditDate="2017-11-17T01:20:39.010" LastActivityDate="2017-11-17T01:20:39.010" Title="Should I use harmonic mean for averaging metrics in repeat runs of classifier evaluation?" Tags="|machine-learning|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24799" PostTypeId="2" ParentId="16318" CreationDate="2017-11-16T07:07:31.740" Score="13" Body="&lt;p&gt;In researching the benefits of splitting into multiple files, the only &lt;a href=&quot;https://github.com/tensorflow/tensor2tensor/issues/190#issuecomment-319565596&quot; rel=&quot;noreferrer&quot;&gt;reasonable answer&lt;/a&gt; came from one of the Google folks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They said performance gains are negligible, but I agree that splitting files can help, especially if you want to transfer the dataset to another location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that now you don't need to shuffle before saving, because (currently) recommended method to read TFRecords uses &lt;code&gt;tf.data.TFRecordDataset&lt;/code&gt; which implements very useful &lt;code&gt;.shuffle()&lt;/code&gt; method.&lt;/p&gt;&#xA;" OwnerUserId="41965" LastEditorUserId="29575" LastEditDate="2017-11-17T14:21:53.663" LastActivityDate="2017-11-17T14:21:53.663" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24800" PostTypeId="1" AcceptedAnswerId="24838" CreationDate="2017-11-16T07:57:54.843" Score="36" ViewCount="15948" Body="&lt;p&gt;In this &lt;a href=&quot;https://people.duke.edu/~rnau/411diff.htm&quot; rel=&quot;noreferrer&quot;&gt;link on Stationarity and differencing&lt;/a&gt;, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, autocorrelation etc are constant over time. Since RNNs have a better capacity to learn non-linear relationships (&lt;a href=&quot;https://machinelearningmastery.com/promise-recurrent-neural-networks-time-series-forecasting/&quot; rel=&quot;noreferrer&quot;&gt;as per given here: The Promise of Recurrent Neural Networks for Time Series Forecasting&lt;/a&gt;) and perform better than traditional time series models when the data is large, it is essential to understand how stationarized data would affect its results. The questions I need to know the answer of are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In case of traditional time series forecasting models, stationarity in time series data makes it easier to predict, why and how? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;While building a time series prediction model using &lt;strong&gt;LSTMs&lt;/strong&gt;, is it important to make the time series data stationary? If so, then why?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23644" LastEditorUserId="29575" LastEditDate="2018-09-16T15:06:15.103" LastActivityDate="2019-12-16T16:21:02.203" Title="Time Series prediction using LSTMs: Importance of making time series stationary" Tags="|deep-learning|predictive-modeling|time-series|forecast|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24801" PostTypeId="2" ParentId="19988" CreationDate="2017-11-16T08:03:03.877" Score="1" Body="&lt;p&gt;About your first question, you are looking for Boolean indexing in Python Pandas.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;build_times = df[['Date','site','project','Duration','Status']]&#xA;build_times.loc[build_times['Status'] == 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;About the second question, you can filter again the dataframe and draw multiple lines on the same graph like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;import matplotlib.pyplot as plt&#xA;&#xA;sns.kdeplot(build_group.vals[(build_group.site == &quot;A&quot;) &amp;amp;&amp;amp; (build_group.project == &quot;example&quot;)],label='Project example on site A');&#xA;sns.kdeplot(build_group.vals[(build_group.site == &quot;B&quot;) &amp;amp;&amp;amp; (build_group.project == &quot;example&quot;)],label='Project example on site B');&#xA;&#xA;plt.xlabel('Date')&#xA;plt.ylabel('Average build time')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-16T08:03:03.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24802" PostTypeId="2" ParentId="19774" CreationDate="2017-11-16T08:09:56.840" Score="2" Body="&lt;p&gt;If two variables are independent, then their correlation will be zero. However, you cannot say the opposite. Zero correlation doesn't necessarily imply independence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's hard to answer your question without any plot or the raw data, but you need to seek information for non-linear dependency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a great example from &lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page&lt;/a&gt; to understand that Pearson cannot prove dependency&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vfwOl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vfwOl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-16T08:09:56.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24803" PostTypeId="2" ParentId="24232" CreationDate="2017-11-16T08:49:24.240" Score="1" Body="&lt;p&gt;You can try the &lt;em&gt;cascade&lt;/em&gt; model, e.g. you start by fitting three individual models&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;S1=F(x1,...,x7)&lt;/p&gt;&#xA;&lt;p&gt;S2=F(x1,...,x7)&lt;/p&gt;&#xA;&lt;p&gt;S3=F(x1,...,x7)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Next, you use the outcomes of these individual models to fit the second level models, e.g.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;M1=F(x1,...,x7,S1,S2)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;M2=F(x1,...,x7,S1,S3)&#xA;...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The structure and order of fitting the models depends on your data.&lt;/p&gt;&#xA;&lt;p&gt;You can either test the performance of some combinations or just set the structure based on expert judgement.&lt;/p&gt;&#xA;&lt;p&gt;Of course, there is no guarantee that the performance of these stacked models will be better the performance of your individual models.&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-16T08:49:24.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24804" PostTypeId="1" CreationDate="2017-11-16T08:58:23.750" Score="2" ViewCount="1372" Body="&lt;p&gt;I set up a forecasting model that predicts call data. The forecast model uses a random forest regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&#xA;I have call data about every day in 15 minutes intervals of a year since 2013.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of the accumulated values over months:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pg0Z6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pg0Z6.png&quot; alt=&quot;Calldata&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be clearly seen that call data has almost doubled in 2017 over 2016. This trend should also be observable for the next few years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, the format of my data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; DATE                     CALL&#xA; ....&#xA; 2017-10-23 10:15:00.000    259&#xA; 2017-10-23 10:30:00.000    292&#xA; 2017-10-23 10:45:00.000    309&#xA; ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From this I extracted the following features:&#xA;I have extracted the following features to predict my target variable Y (call data):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-Weekday&#xA;-Month&#xA;-Holiday (yes / no)&#xA;-Interval of the day&#xA;&#xA;So I ask my model:&#xA;&#xA;What is the call volume of a day and interval with the following features?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have used the years 2015-2016-2017 to train the model. However, the model does not give the desired prognosis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;He even predicts the days for 2017 wrong. Although I gave him the data as training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- Should I work on my features?&#xA;&#xA;- How do I show my forecasting model that the data will double year by year &#xA;  as observable since 2016?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41968" LastEditorUserId="41968" LastEditDate="2017-11-16T09:23:05.337" LastActivityDate="2018-08-04T19:28:43.930" Title="Forecast Model recognize future trend" Tags="|machine-learning|predictive-modeling|regression|feature-selection|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24805" PostTypeId="2" ParentId="23850" CreationDate="2017-11-16T09:25:42.657" Score="1" Body="&lt;p&gt;The easiest solution that comes to my mind is to create date indices to count the distinct dates per ID, and then perform a group by over those indices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First let's make your question reproducible:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame&#xA;&#xA;df = DataFrame({'ID':[1,1,1,2,2,3,3,4],\&#xA;                'date':[201610,201610,201611,201505,201505,201609,201702,\&#xA;                        201709],\&#xA;                'values': [10,20,30,40,50,60,70,80]})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hence df is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   ID    date  values&#xA;0   1  201610      10&#xA;1   1  201610      20&#xA;2   1  201611      30&#xA;3   2  201505      40&#xA;4   2  201505      50&#xA;5   3  201609      60&#xA;6   3  201702      70&#xA;7   4  201709      80&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now your groupby returns this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [137]: df.groupby(['ID', 'date']).sum()&#xA;Out[137]: &#xA;           values&#xA;ID date          &#xA;1  201610      30&#xA;   201611      30&#xA;2  201505      90&#xA;3  201609      60&#xA;   201702      70&#xA;4  201709      80&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And your question is how you can sum up the values at the 1st, 2nd, ... lines per ID. Your desired output is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;first_date: 30+90+60+80 = 260&lt;/p&gt;&#xA;&#xA;&lt;p&gt;second_date: 30+70 = 100&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OK. You don't have to do your groupby. You can just create indices of dates per ID, and groupby over them. Those indices will start from 1 for each ID, and will increase only when there is a new date for the same ID. Let's create and append them back to the df:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date = 201610&#xA;ID = 1&#xA;date_index = 1&#xA;date_indices = []&#xA;for i in range(df.shape[0]):&#xA;    if date != df['date'][i] and ID == df['ID'][i]:&#xA;        date_index += 1&#xA;    else:&#xA;        date_index = 1&#xA;    date_indices.append(date_index)&#xA;    date = df['date'][i]&#xA;    ID = df['ID'][i]&#xA;&#xA;df['date_indices'] = date_indices&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives us the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [140]: df&#xA;Out[140]: &#xA;   ID    date  values  date_indices&#xA;0   1  201610      10             1&#xA;1   1  201610      20             1&#xA;2   1  201611      30             2&#xA;3   2  201505      40             1&#xA;4   2  201505      50             1&#xA;5   3  201609      60             1&#xA;6   3  201702      70             2&#xA;7   4  201709      80             1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Groupby gives us the desired result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [141]: df.groupby(['date_indices']).sum()[['values']]&#xA;Out[141]: &#xA;              values&#xA;date_indices        &#xA;1                260&#xA;2                100&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please let me know if you need any further clarification/modification.&lt;/p&gt;&#xA;" OwnerUserId="20572" LastActivityDate="2017-11-16T09:25:42.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24806" PostTypeId="1" CreationDate="2017-11-16T09:31:31.020" Score="2" ViewCount="301" Body="&lt;p&gt;Suppose I have some data, given by $\boldsymbol{x}$ and $\boldsymbol{y}$ pairs, for instance$\{(\boldsymbol{x_1}, \boldsymbol{y_1}), (\boldsymbol{x_2},\boldsymbol{y_2}),...,(\boldsymbol{x_n},\boldsymbol{y_n})\}$ for $n\in\mathbb{N}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume that the data comes from some function $f(\boldsymbol{x})$ with some noise, I don't know the function, so we don't know whether it is periodic (very likely, it is NOT periodic).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means $\boldsymbol{y_i} = f(\boldsymbol{x_i}) + \boldsymbol{\epsilon_i}$ where $\boldsymbol{\epsilon_i} \sim \textbf{N}(\textbf{0}, \boldsymbol{\sigma}^2))$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I know that the Fourier Series is defined only for periodic functions. However (&lt;a href=&quot;https://math.stackexchange.com/questions/942740/fourier-series-for-a-non-periodic-function&quot;&gt;here&lt;/a&gt;) it is possible to &quot;extend&quot; a non-periodic function defined on an interval, to a periodic function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The amount of data that I have is finite, so will lie in an interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does it make sense (is it even possible) to try and fit a Fourier Series to the data, by considering the data (which is multi-dimensional, both x and y are vectors) in a multi-dimensional interval , by finding the coefficients that minimize some loss function? (for instance Mean Squared Error)&#xA;  How would someone define this fourier series for a multi-dimensional, non-periodic, noisy function?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(by using only a finite number of fourier terms!)&lt;/p&gt;&#xA;" OwnerUserId="31316" LastActivityDate="2017-11-16T09:31:31.020" Title="Fitting data with Fourier Series coefficients" Tags="|data-mining|statistics|loss-function|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24807" PostTypeId="2" ParentId="24804" CreationDate="2017-11-16T09:32:46.000" Score="2" Body="&lt;p&gt;Before going into modelling, I guess you can do a bit more of exploratory analysis(month by month, year by year). If you find any trend or seasonality and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why did you go to RF directly without using basing Techniques like &lt;strong&gt;ARIMA, ARMA, Exponential Smoothening&lt;/strong&gt; and &lt;strong&gt;AR&lt;/strong&gt; so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometime RF might not give you as good results as Base models, I think you don't have trend, this is from your graph(but not certain). If you can try doing some research and see if there are some external factors which are effecting your demand. Why did that happen and what is the root case for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To your model to understand, it needs some feature which explains its spikes somehow, it can achieved by doing feature engg/ research&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-16T09:32:46.000" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24809" PostTypeId="1" CreationDate="2017-11-16T10:29:36.310" Score="1" ViewCount="175" Body="&lt;p&gt;For a project at my university I have to develop a simple chatbot. Since I am new to machine learning, it should be really simple and work like a customer support. The chatbot has to recognize about 10 intents related to some really simple topic. For example I thought that it could give information about a party, so the questions would be something like &quot;Where is the party?, When does the party start? How much does the entry cost?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chatbot should classify the question, and out of the classification it should give back a prepared answer. If it cannot assign the question to the one of the 10 intents, it should do nothing or just answer that it does not understand the question. The same may happen if the question has spelling mistakes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it should be really simple, but be programmed with the deep learning approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, how much data should I have, to train this bot to assign questions to the intents? Would I be able to generate the data by my own, if I reformulate each question on, lets say 20 ways? Or do I have to  feed the algorithm with a very large amount of data, even for such simple task?&lt;/p&gt;&#xA;" OwnerUserId="41975" LastEditorUserId="836" LastEditDate="2017-11-16T12:04:52.767" LastActivityDate="2018-10-13T21:00:18.123" Title="Data amount for a very simple chatbot" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24810" PostTypeId="2" ParentId="24804" CreationDate="2017-11-16T10:36:02.937" Score="2" Body="&lt;p&gt;Random forest, and tree-based models in general, do not handle trends well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is simple: inside any decision tree, there are discrete rules such as:&#xA;$$&#xA;y = \begin{cases} y_1, &amp;amp; \text{if } x &amp;gt; c \\ y_2, &amp;amp; \text{if } x &amp;lt;= c \end{cases}&#xA;$$&#xA;This is a tree of depth 1 (so-called &quot;stem&quot;), but deeper trees obey the same logic. The variable $x$ and constants, $y_1$, $y_2$, $c$ are fit to the train data. And this is the problem: if in the training data $y$ was never higher than $y_1$, your tree will never predict $y&amp;gt;y_1$, even if $y$ is clearly increasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, linear models (such as XARIMA and its special cases) catch trends very well. But they are poor with non-linearities and feature interplay in your data. In my own experience, the following stacking approach works best:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fit a simple time-based linear model to your data.&lt;/li&gt;&#xA;&lt;li&gt;Fit a tree-based model (random forest or boosting) to the residuals of your linear model.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the linear model is specified correctly, it will catch and remove the non-stationarities in the data. Thus, the tree-based model will be predicting stationary residuals and find finer dependencies that the linear model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This Python example illustrates the issue:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.datasets import make_friedman2&#xA;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn.linear_model import HuberRegressor&#xA;from sklearn.metrics import r2_score&#xA;from sklearn.model_selection import cross_val_predict&#xA;&#xA;# make a difficult dataset with a linear trend&#xA;X, y = make_friedman2(n_samples=1000, random_state=1, noise=10)&#xA;time = np.arange(1000)&#xA;y += time * 1.5&#xA;X = np.hstack([X, time[:, np.newaxis]])&#xA;X_train, y_train = X[:700], y[:700]&#xA;X_test, y_test = X[700:], y[700:]&#xA;&#xA;# build a pure Random Forest&#xA;rf = RandomForestRegressor(random_state=1, n_jobs=-1).fit(X_train, y_train)&#xA;y_rf = rf.predict(X_test)&#xA;&#xA;# build a pure linear model&#xA;linear = HuberRegressor().fit(X_train, y_train)&#xA;y_lin = linear.predict(X_test)&#xA;&#xA;# build a stack of two models&#xA;lin_resid = y_train - cross_val_predict(linear, X_train, y_train)&#xA;rf.fit(X_train, lin_resid)&#xA;y_stack = y_lin + rf.predict(X_test)&#xA;&#xA;print(r2_score(y_test, y_rf))    # R2 on test data is only 0.34&#xA;print(r2_score(y_test, y_lin))   # R2 due to time trend is 0.86 &#xA;print(r2_score(y_test, y_stack)) # R2 of combined model is 0.95&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-11-16T10:50:58.567" LastActivityDate="2017-11-16T10:50:58.567" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24814" PostTypeId="1" CreationDate="2017-11-16T13:00:37.367" Score="1" ViewCount="711" Body="&lt;p&gt;In Keras' &lt;a href=&quot;https://keras.io/applications/#inceptionresnetv2&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation for the InceptionResNetV2 model&lt;/a&gt;, it says the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;input_shape: (...). It should have exactly 3 inputs channels, and width and height should be no smaller than 139.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm trying to figure out the reason behind the minimum size of 139px for this model, because I'm looking to train this model on smaller images (128px for starters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't find any clues in the &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/inception_resnet_v2.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;model's source code&lt;/a&gt; or in &lt;a href=&quot;https://arxiv.org/pdf/1602.07261.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the paper which describes the model&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming this number is not arbitrary, what is the reason behind this minimum size of 139px?&lt;/p&gt;&#xA;" OwnerUserId="41979" LastEditorUserId="29169" LastEditDate="2021-02-13T19:21:36.173" LastActivityDate="2024-01-23T09:00:45.070" Title="What is the reason behind the minimum image size in the Keras InceptionResNetV2 model?" Tags="|keras|convolutional-neural-network|inception|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24815" PostTypeId="2" ParentId="24725" CreationDate="2017-11-16T13:54:10.627" Score="1" Body="&lt;p&gt;Okay, so probably this was obvious but here is what I did. &#xA;I used the &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;scores = decision_function(x)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;instead of just predicting the anomalies, and obtained the anomaly scores for each instance. Afterwards, I could draw the anomaly distribution of the dataset and set a better outlier proportion regarding the distribution.&lt;/p&gt;&#xA;" OwnerUserId="40029" LastActivityDate="2017-11-16T13:54:10.627" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24817" PostTypeId="1" AcceptedAnswerId="24832" CreationDate="2017-11-16T16:42:15.470" Score="2" ViewCount="1062" Body="&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;:  I am working on a linear programming problem, i.e. a linear objective function to minimize:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{c}\cdot\mathbf{x}$,&lt;br&gt;&#xA;where $\mathbf{c},\mathbf{x}\in\mathbb{R}^{N}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Subject to constraints: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathbf{x}\geq 0$ and $\mathbf{A}\mathbf{x} \leq \mathbf{b}$&#xA;where $\mathbf{A}\in\mathbb{R}^{MxN}$ and $\mathbf{b}\in\mathbb{R}^{M}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;a href=&quot;https://github.com/coin-or/CyLP&quot; rel=&quot;nofollow noreferrer&quot;&gt;CyLP&lt;/a&gt; and the &lt;a href=&quot;https://www.coin-or.org/Clp/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;COIN-OR simplex solver&lt;/a&gt; to perform the optimization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: A feature I would like to be able to build into&#xA;the problem is the ability to handle some atypical constraints: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $x_{i}$ is a component of $\mathbf{x}$. I want to apply the following constraints:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_{i} = 0$ or $\mathrm{lb} \leq x_{i} \leq \mathrm{ub}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In geometric terms, I want $x_{i}\in [0]\cup [\mathrm{lb}, \mathrm{ub}]$.&#xA;Unlike the original problem, where all $x_{i}$ are bounded within convex sets (i.e. non-piecewise intervals).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good way to modify an LP for the above atypical constraint? &lt;/p&gt;&#xA;" OwnerUserId="41982" LastActivityDate="2017-11-17T00:40:18.723" Title="Non-convex feasible region in a linear programming problem" Tags="|python|optimization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24818" PostTypeId="1" CreationDate="2017-11-16T17:13:28.983" Score="2" ViewCount="203" Body="&lt;p&gt;I'm attempting to use a neural network as a kind of interpolator for a high-dimensional function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We're doing this to circumvent the need for a physical model that calculates this function exactly, but is expensive to run. We have a dataset of past runs of this physical model that we can now use as the basis for interpolation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input is a set of 28 parameter values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output we're trying to estimate consists of the shape of 432 periodic functions (i.e. value of 432 different functions along the circumference of a circle). We know that the output is uniquely determined by the 28 input values; there's no noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here is one of these periodic functions. For this example, it's only sampled with 12 points (though in production we will be sampling it more finely, perhaps 36 points around a circle). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6Ai6J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6Ai6J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current neural network Keras implementation is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input layer: 28 units (one per input parameter)&lt;/li&gt;&#xA;&lt;li&gt;One hidden layer: fully connected, 1000 units&lt;/li&gt;&#xA;&lt;li&gt;Output layer: fully connected, 432 periodic functions × 12 samples = 5184 units&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As illustrated above, this works okay (even before optimizing hyperparameters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I'm concerned that with these fully-connected layers, I'm treating the 5184 outputs as unrelated numbers, completely disjointed from one another. In reality, there's some spatial relationship between the 12 samples describing each of the 432 underlying functions in the output. But I'm just throwing this spatial information away. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there another kind of layer which could take advantage of this spatial relationship, and thus lend itself better to this kind of regression?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were going the other way round, i.e. from spatial information to distinct values, I would consider convolution layers (perhaps with a periodic boundary). But what about this reverse case?&lt;/p&gt;&#xA;" OwnerUserId="41639" LastActivityDate="2017-11-16T17:28:15.353" Title="Neural net layer that preserves spatial information" Tags="|deep-learning|regression|interpolation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24819" PostTypeId="2" ParentId="24818" CreationDate="2017-11-16T17:28:15.353" Score="1" Body="&lt;p&gt;Rather than trying to predict each value explicitly, you could pick a parametric function that you think might be expressive enough to describe your target function (for example, a polynomial of order 6), and get your network to learn the parameters of that polynomial that give you a minimum loss. This is basically polynomial regression.  Since your function is periodic you might use a more cycle-based basis set, like linear combinations of sine waves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Input data to the network&#xA;2) output a set of values that you treat as parameters to your function&#xA;3) for each target point, compare it to the predicted value from your function&#xA;4) calculate a loss (like mean squared error) based on the difference between the function predictions and the targets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a function will (1) restrict the space of possible outputs, and (2) make nearby outputs be self-similar.  &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-16T17:28:15.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24820" PostTypeId="2" ParentId="24794" CreationDate="2017-11-16T17:39:44.660" Score="1" Body="&lt;p&gt;I think your solution is quite idiomatic.&#xA;Here is an alternative solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [355]: s.loc[s.notnull().idxmax() : s[::-1].notnull().idxmax()].fillna(0, inplace=True)&#xA;&#xA;In [356]: s&#xA;Out[356]:&#xA;0    NaN&#xA;1    NaN&#xA;2    1.0&#xA;3    0.0&#xA;4    2.0&#xA;5    NaN&#xA;dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-11-16T17:39:44.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24821" PostTypeId="2" ParentId="22853" CreationDate="2017-11-16T18:37:56.357" Score="14" Body="&lt;p&gt;This is simply trying to convey my intuition, i.e. no rigor. The thing with saddle points is that they are a type of optimum which combines a combination of minima and maxima. Because the number of dimensions are so large with deep learning, the probability that an optimum only consists of a combination of minima is very low. This means 'getting stuck' in a local minimum is rare. At the risk of oversimplifying, it's harder to 'get stuck' in a saddle point because you can 'slide down one of the dimensions'. I think the Andrew Ng video you refer to comes from the Coursera course on Deep Learning by him.&lt;/p&gt;&#xA;" OwnerUserId="41985" LastActivityDate="2017-11-16T18:37:56.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24823" PostTypeId="2" ParentId="937" CreationDate="2017-11-16T19:35:53.737" Score="19" Body="&lt;p&gt;Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and scikit-learn deliberately avoids inferential approach to model learning (significance testing etc). Moreover, pure OLS is only one of numerous regression algorithms, and from the scikit-learn point of view it is neither very important, nor one of the best.&lt;/p&gt;&#xA;&lt;p&gt;There are, however, some pieces of advice for those who still need a good way for feature selection with linear models:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use inherently sparse models like &lt;code&gt;ElasticNet&lt;/code&gt; or &lt;code&gt;Lasso&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Normalize your features with &lt;code&gt;StandardScaler&lt;/code&gt;, and then order your features just by &lt;code&gt;model.coef_&lt;/code&gt;. For perfectly independent covariates it is equivalent to sorting by p-values. The class &lt;code&gt;sklearn.feature_selection.RFE&lt;/code&gt; will do it for you, and &lt;code&gt;RFECV&lt;/code&gt; will even evaluate the optimal number of features.&lt;/li&gt;&#xA;&lt;li&gt;Use [an implementation][1] of forward selection by adjusted &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; that works with &lt;code&gt;statsmodels&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Do brute-force forward or backward selection to maximize your favorite metric on cross-validation (it could take approximately quadratic time in number of covariates). A scikit-learn compatible &lt;code&gt;mlxtend&lt;/code&gt; package [supports][2] this approach for any estimator and any metric.&lt;/li&gt;&#xA;&lt;li&gt;If you still want vanilla stepwise regression, it is easier to base it on &lt;code&gt;statsmodels&lt;/code&gt;, since this package calculates p-values for you. A basic forward-backward selection could look like this:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;    from sklearn.datasets import load_boston&#xA;    import pandas as pd&#xA;    import numpy as np&#xA;    import statsmodels.api as sm&#xA;&#xA;    data = load_boston()&#xA;    X = pd.DataFrame(data.data, columns=data.feature_names)&#xA;    y = data.target&#xA;&#xA;&#xA;    def stepwise_selection(X, y, &#xA;                           initial_list=[], &#xA;                           threshold_in=0.01, &#xA;                           threshold_out = 0.05, &#xA;                           verbose=True):&#xA;        &amp;quot;&amp;quot;&amp;quot; Perform a forward-backward feature selection &#xA;        based on p-value from statsmodels.api.OLS&#xA;        Arguments:&#xA;            X - pandas.DataFrame with candidate features&#xA;            y - list-like with the target&#xA;            initial_list - list of features to start with (column names of X)&#xA;            threshold_in - include a feature if its p-value &amp;lt; threshold_in&#xA;            threshold_out - exclude a feature if its p-value &amp;gt; threshold_out&#xA;            verbose - whether to print the sequence of inclusions and exclusions&#xA;        Returns: list of selected features &#xA;        Always set threshold_in &amp;lt; threshold_out to avoid infinite looping.&#xA;        See https://en.wikipedia.org/wiki/Stepwise_regression for the details&#xA;        &amp;quot;&amp;quot;&amp;quot;&#xA;        included = list(initial_list)&#xA;        while True:&#xA;            changed=False&#xA;            # forward step&#xA;            excluded = list(set(X.columns)-set(included))&#xA;            new_pval = pd.Series(index=excluded)&#xA;            for new_column in excluded:&#xA;                model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()&#xA;                new_pval[new_column] = model.pvalues[new_column]&#xA;            best_pval = new_pval.min()&#xA;            if best_pval &amp;lt; threshold_in:&#xA;                best_feature = new_pval.idxmin()&#xA;                included.append(best_feature)&#xA;                changed=True&#xA;                if verbose:&#xA;                    print('Add  {:30} with p-value {:.6}'.format(best_feature, best_pval))&#xA;&#xA;            # backward step&#xA;            model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()&#xA;            # use all coefs except intercept&#xA;            pvalues = model.pvalues.iloc[1:]&#xA;            worst_pval = pvalues.max() # null if pvalues is empty&#xA;            if worst_pval &amp;gt; threshold_out:&#xA;                changed=True&#xA;                worst_feature = pvalues.idxmax()&#xA;                included.remove(worst_feature)&#xA;                if verbose:&#xA;                    print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))&#xA;            if not changed:&#xA;                break&#xA;        return included&#xA;&#xA;    result = stepwise_selection(X, y)&#xA;&#xA;    print('resulting features:')&#xA;    print(result)&#xA;&#xA;&#xA;This example would print the following output:&#xA;&#xA;    Add  LSTAT                          with p-value 5.0811e-88&#xA;    Add  RM                             with p-value 3.47226e-27&#xA;    Add  PTRATIO                        with p-value 1.64466e-14&#xA;    Add  DIS                            with p-value 1.66847e-05&#xA;    Add  NOX                            with p-value 5.48815e-08&#xA;    Add  CHAS                           with p-value 0.000265473&#xA;    Add  B                              with p-value 0.000771946&#xA;    Add  ZN                             with p-value 0.00465162&#xA;    resulting features:&#xA;    ['LSTAT', 'RM', 'PTRATIO', 'DIS', 'NOX', 'CHAS', 'B', 'ZN']&#xA;&#xA;&#xA;  [1]: http://planspace.org/20150423-forward_selection_with_statsmodels/&#xA;  [2]: http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2021-01-26T13:58:42.843" LastActivityDate="2021-01-26T13:58:42.843" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="24826" PostTypeId="1" CreationDate="2017-11-16T20:34:00.003" Score="2" ViewCount="449" Body="&lt;p&gt;I'm trying to do feature selection in the bayesian framework with a Laplace prior with the following code in Python;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#nb_predictors = len(df.columns) - 1 # we remove the target variable&#xA;nb_predictors = 7&#xA;beta = list()&#xA;&#xA;with pm.Model() as model:&#xA;    # Define priors&#xA;    intercept = pm.Normal('Intercept', mu=0, sd=1/25)&#xA;&#xA;    for cpt in range(1, nb_predictors + 1):&#xA;        beta.append(pm.Laplace('beta_' + str(cpt), mu=0, b=np.sqrt(2)))&#xA;&#xA;    # Define Likelihood    &#xA;    logit = intercept + beta[0] * df['satisfaction_level'] + beta[1] * df['last_evaluation'] \&#xA;                      + beta[2] * df['number_project'] + beta[3] * df['average_montly_hours'] \&#xA;                      + beta[4] * df['time_spend_company'] + beta[5] * df['Work_accident'] \&#xA;                      + beta[6] * df['promotion_last_5years']&#xA;&#xA;&#xA;&#xA;    likelihood = pm.Bernoulli('left', pm.math.sigmoid(logit), observed=df['left'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I was wondering is what happen if I add a new categorical variable (by one hot encoding), like this sales variable. Can I still use a laplace prior and observe if the density is close to 0 (so it is probably not related with the target variable) or it makes no sense with categorical variable ? and it only works with continuous variable ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Define Likelihood    &#xA;logit = intercept + beta[0] * df['satisfaction_level'] + beta[1] * df['last_evaluation'] \&#xA;                  + beta[2] * df['number_project'] + beta[3] * df['average_montly_hours'] \&#xA;                  + beta[4] * df['time_spend_company'] + beta[5] * df['Work_accident'] \&#xA;                  + beta[6] * df['promotion_last_5years'] + beta[7] * df['sales_low'] + beta[8] * df['sales_high']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40557" LastActivityDate="2017-11-16T20:34:00.003" Title="Bayesian linear regression / categorical variable / Laplace prior" Tags="|python|statistics|bayesian|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24827" PostTypeId="1" AcceptedAnswerId="24833" CreationDate="2017-11-16T20:35:26.330" Score="0" ViewCount="156" Body="&lt;p&gt;I'm fiddling around with some data that represent grocery store transactions. The data are in the following form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row represents a final transaction by a customer, with a column for user ID, timestamp of purchase, and basket contents. For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;| ID | Timestamp  |              Basket           |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;| 12 | 2016-04-02 | ['Celery', 'Beets', 'Cheese'] |&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question I'm trying to answer is &quot;How do I suggest an ideal basket to an individual customer?&quot;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problems I'm dealing with:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I can't seem to think past just suggesting the most frequently ordered items to a customer. Is there some technique that I'm missing here?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Since we have a customer ID, we can have multiple baskets for a customer over time. Because of this, I can't use apriori or eclat as those assume each transaction is independent.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I take into account seasonality with whatever technique I choose?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd really appreciate some general direction here, I'm having some serious analysis paralysis.&lt;/p&gt;&#xA;" OwnerUserId="41990" LastActivityDate="2017-11-17T01:47:39.877" Title="Modeling Grocery Store Transactions" Tags="|data-mining|time-series|recommender-system|association-rules|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24828" PostTypeId="1" CreationDate="2017-11-16T20:47:25.280" Score="1" ViewCount="28" Body="&lt;p&gt;I'm trying to understand if the latest ML methods associated with GANs can estimate the pdf of complex joint distributions from data (as opposed to the generator model).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've found from &lt;a href=&quot;https://arxiv.org/pdf/1701.00160.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ian Goodfellow's paper, pg 29&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;For GANs, the generator is $P_{model}$, while&#xA;for NCE and MLE, $P_{model}$ is part of the discriminator. Beyond this, the differences&#xA;between the methods lie in the update strategy. GANs learn both players with gradient&#xA;descent. MLE learns the discriminator using gradient descent, but has a heuristic&#xA;update rule for the generator. Specifically, after each discriminator update step, MLE&#xA;copies the density model learned inside the discriminator and converts it into a sampler&#xA;to be used as the generator. NCE never updates the generator; it is just a fixed&#xA;source of noise.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know further details of this process he refers to?  The closest thing I've found is &lt;a href=&quot;https://content.iospress.com/articles/intelligent-data-analysis/ida00694&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, which uses the well-known discriminator relationship&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$D(x)=\frac{P_{data}}{P_{data} + P_{model}}$ and inverts to get $P'_{data}=P_{model}\frac{D(x)}{1-D(x)}$ where $P'_{data}$ is the updated  $P_{model}$ for the next iteration.  Then the generator on the next iteration is $P'_{data}$ via rejection sampling.  By starting with a known pdf this has reliably converged in my experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem here is that the model continues to grow each iteration.  It's not especially scale-able.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know if Ian refers to a method that is trained more similarly to the GANs, with a &quot;static&quot; sized model in which the pdf can be extracted from the discriminator at the end of training, and the generator can be more like what we see in GANs (as opposed to rejection sampling)?&lt;/p&gt;&#xA;" OwnerUserId="41993" LastActivityDate="2017-11-16T20:47:25.280" Title="Extract density estimator from discriminator (MLE method) in adversarial training" Tags="|machine-learning|neural-network|probability|gan|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24829" PostTypeId="1" CreationDate="2017-11-16T21:05:39.813" Score="2" ViewCount="171" Body="&lt;p&gt;I have just read the paper from Ian Goodfellow et al. titled &lt;a href=&quot;https://arxiv.org/pdf/1302.4389.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Maxout Networks&quot;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, function that is used in many state-of-the-art models. Also, Maxout, in theory, should work better with dropout than other activation functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am therefore wondering, what is the intuition for using Relu instead of Maxout.&lt;/p&gt;&#xA;" OwnerUserId="41995" LastActivityDate="2017-11-16T21:05:39.813" Title="Why isn't Maxout used in the state of the art models?" Tags="|machine-learning|neural-network|deep-learning|dropout|activation-function|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="24831" PostTypeId="1" AcceptedAnswerId="24834" CreationDate="2017-11-16T23:51:33.523" Score="2" ViewCount="2376" Body="&lt;p&gt;I am interested in finding out how decision trees chose the order in which they split. I understand that splitting is based in information gain. The attribute with the lowest information gain is chosen as the root node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had a data set with columns:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;credit standing&lt;/li&gt;&#xA;&lt;li&gt;age&lt;/li&gt;&#xA;&lt;li&gt;income,&lt;/li&gt;&#xA;&lt;li&gt;marriage status&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and I was interested in finding out what determines a person to have a good or bad credit status, Am I correct in saying, I calculate entropy and information gain for each of these categorical attributes against the independent attribute i.e what I am investigating (credit standing), and that the calculation with the lowest information gain is chosen as the root node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if this root node (first split) was Age, Is the entropy and information gain of Age (the new independent attribute) against the remaining attributes (marriage status and income) calculated, and the calculation with the lowest information gain is then chosen as the second split node, and so on?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;information gain:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;credit standing vs age    = 0.01&#xA;credit standing vs status = 0.1&#xA;credit standing vs income = 0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Age is chosen as root (first split) node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then, information gain:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;age vs status = 0.2&#xA;age vs income = 0.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;income is chosen as second split node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I understanding this correctly?&lt;/p&gt;&#xA;" OwnerUserId="42000" LastEditorUserId="29575" LastEditDate="2017-11-17T00:27:36.790" LastActivityDate="2024-03-28T16:31:49.237" Title="Decision tree ordering" Tags="|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24832" PostTypeId="2" ParentId="24817" CreationDate="2017-11-17T00:31:37.543" Score="1" Body="&lt;p&gt;Sorry, this cannot be done with a continuous LP solver. As you observed, this construct introduces a non-convexity. However, it can be handled by a Mixed Integer Programming (MIP) solver.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x=0$ or $\ell\le x\le u$ is what is known as a &lt;strong&gt;semi-continuous variable&lt;/strong&gt;. Many advanced MIP solvers support this variable type directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a MIP solver does not support this, we can introduce a binary variable $\delta \in \{0,1\}$ and use the constraint:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \ell \cdot \delta \le x \le u \cdot \delta  $$   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;to simulate a semi-continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;COIN-OR has a capable MIP solver called CBC (I believe it supports semi-continuous variables directly).&lt;/p&gt;&#xA;" OwnerUserId="42002" LastEditorUserId="42002" LastEditDate="2017-11-17T00:40:18.723" LastActivityDate="2017-11-17T00:40:18.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24833" PostTypeId="2" ParentId="24827" CreationDate="2017-11-17T01:47:39.877" Score="0" Body="&lt;p&gt;I think that you can approach this problem in a better way. To do so you need some &lt;strong&gt;Customer Demographics&lt;/strong&gt;. Even if you try doing some recommendation using customer ID it won't give you ideal Basket/Recommendation, because from Customer ID alone you can't decide anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any possibility to get Transaction Id, so that you can unique baskets to apply Aprori Algo/ any association mining techniques by which you can give good recommendation using &lt;strong&gt;Lift&lt;/strong&gt; and &lt;strong&gt;Support&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the time stamp do you have time? As your sample record doesn't show time in it(just for confirmation), Let us consider a scenario you have it. Now ['Celery', 'Beets', 'Cheese'] - transform as Basket-1, ['Celery', 'Bread', 'Cheese'] as Basket-2 and so on... Now you see which Baskets are sold the most WRT timestamp. This is to see if there is any seasonality in the data and you can even find some trends, if there are any.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is just one way of looking at the problem but this is not the concrete solution.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-17T01:47:39.877" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24834" PostTypeId="2" ParentId="24831" CreationDate="2017-11-17T01:56:50.453" Score="1" Body="&lt;p&gt;Here in your Scenario you need to select the one with more &lt;strong&gt;Information Gain&lt;/strong&gt; rather than the least one and the process goes on till you reach the last feature/last node.&lt;/p&gt;&#xA;&lt;p&gt;Go through &lt;a href=&quot;https://web.cecs.pdx.edu/%7Emperkows/CLASS_ROBOTICS/FEBR26-2004/ROBOT-DECISION-TREE/robot-navigation-using-decision-trees-1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;These&lt;/a&gt; &lt;a href=&quot;http://www.ee.pdx.edu/%7Emperkows/CLASS_ROBOTICS/FEBR26-2004/ROBOT-DECISION-TREE/ics320Part3.DecisionTreeLearning.ppt&quot; rel=&quot;nofollow noreferrer&quot;&gt;Links&lt;/a&gt;. I think your doing it vice versa, I agree with &lt;a href=&quot;https://datascience.stackexchange.com/users/381/emre&quot;&gt;Emre&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In the &lt;a href=&quot;http://www.ee.pdx.edu/%7Emperkows/CLASS_ROBOTICS/FEBR26-2004/ROBOT-DECISION-TREE/ics320Part3.DecisionTreeLearning.ppt&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; it was explained with an example to decide whether to play tennis or not.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="29575" LastEditDate="2024-03-28T16:31:49.237" LastActivityDate="2024-03-28T16:31:49.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24835" PostTypeId="1" AcceptedAnswerId="24836" CreationDate="2017-11-17T03:30:42.573" Score="0" ViewCount="247" Body="&lt;p&gt;The question is:&#xA;what algorithms (and libraries) should i use if i want to build a recommender system with the following data in mind representation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;USER_ID  ZIP  Movie1  Movie2  Movie3&#xA;1  2483  5  0  3&#xA;2  2483  4  1  5&#xA;3  2345  3  1  5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically i want to factor in user data into a recommendation of a movie.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(zips can be transformed to long/lat but thats another question and out of scope now)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i was searching the internet for hours with no success. So i will be grateful if someone can point me in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="42008" LastActivityDate="2017-11-17T05:28:57.633" Title="User based recommendation factoring in user data" Tags="|recommender-system|" AnswerCount="1" CommentCount="4" ClosedDate="2017-11-18T12:03:35.790" ContentLicense="CC BY-SA 3.0" />
  <row Id="24836" PostTypeId="2" ParentId="24835" CreationDate="2017-11-17T05:28:57.633" Score="0" Body="&lt;p&gt;&lt;strong&gt;Collaborative Filtering:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Match users to people with similar tastes –recommend what they like&lt;/li&gt;&#xA;&lt;li&gt;Commonly used in e-retail&lt;/li&gt;&#xA;&lt;li&gt;Avoids the issue of users only being recommended more of what they already like (allows serendipity)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Example:&#xA;Your Scenario.&lt;/p&gt;&#xA;&lt;p&gt;Methods: Euclidean Distance, Cosine Distance, Pearson Correlation Coefficient (most common).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Content-Based Recommendation&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Match users directly to products and content&lt;/li&gt;&#xA;&lt;li&gt;Recommend based on what you have bought or viewed in the past&lt;/li&gt;&#xA;&lt;li&gt;Commonly used for document recommendation: webpages, news articles,&#xA;blogs etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Examples:&#xA;Which Movie(s) might our user like?&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Option1&lt;/em&gt;: The user selects preferences for the various features using pull-down menus etc. We match against the movies using Vector-Space methods (described next)&lt;/p&gt;&#xA;&lt;p&gt;Methods: Vector-Space Method, K Nearest Neighbour(kNN)&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Option2&lt;/em&gt;: The user rates a sample of the movies (explicitly or implicitly) as like/dislike. we then build a user profile model for that user using machine learning.&lt;/p&gt;&#xA;&lt;p&gt;Method: Decision Trees&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-17T05:28:57.633" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24838" PostTypeId="2" ParentId="24800" CreationDate="2017-11-17T05:49:05.887" Score="49" Body="&lt;p&gt;In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training set to still be appropriate for your test set.  That's the important underlying concept regarding stationarity.  Time series have the additional complexity that there may be &lt;em&gt;long term&lt;/em&gt; structure in your data that your model may not be sophisticated enough to learn.  For example, when using an autoregressive lag of N, we can't learn dependencies over intervals longer than N. Hence, when using simple models like ARIMA, we want data to also be &lt;em&gt;locally&lt;/em&gt; stationary.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;As you said, stationary just means the model's statistics don't change over time ('locally' stationary).  ARIMA models are essentially regression models where you use the past N values as input to linear regression to prediction the N+1st value. (At least, that's what the AR part does).  When you learn the model you're learning the regression coefficients.  If you have a time series where you learn the relationship between the past N points and the next point, and then you apply that to a different set of N points to predict the next value, you are implicitly assuming that the same relationship holds between the N predictor points and the following N+1st point you're trying to predict.  That's stationarity.  If you separated your training set into two intervals and trained on them separately, and got two very different models - what would you conclude from that? Do you think you would feel confident applying those models to predict &lt;em&gt;new&lt;/em&gt; data? Which one would you use? These issues arise if the data is 'non-stationary'.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;My take on RNNs is this - you are still learning a pattern from one segment of a time series, and you still want to apply it to another part of the time series to get predictions.  The model learns a simplified representation of the time series - and if that representation applies on the training set but not in the test set, it won't perform well.  However, unlike ARIMA, RNNs are capable of learning nonlinearities, and specialized nodes like LSTM nodes are even better at this.  In particular, LSTMs and GRUs are very good at learning long-term dependencies.  See for example &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;noreferrer&quot;&gt;this blog post&lt;/a&gt;.  Effectively this means that what is meant by  'stationarity' is less brittle with RNNs, so it's somewhat less of a concern. To be able to learn long term dependencies, however, you need LOTS of data to train on.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Ultimately the proof is in the pudding. That is, do model validation like you would with any other machine learning project.  If your model predicts well for hold-out data, you can feel somewhat confident in using it.  But like any other ML project - if your test data is ever significantly different than your training data, your model will not perform well.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastEditorUserId="41886" LastEditDate="2019-12-16T16:21:02.203" LastActivityDate="2019-12-16T16:21:02.203" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="24839" PostTypeId="1" CreationDate="2017-11-17T06:01:31.243" Score="1" ViewCount="609" Body="&lt;p&gt;In order to train a recurrent neural network, you have to unfold it say 50 times and treat it like a chain of RNN cells.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you are using this RNN to train in realtime, then it seems you may as well just have a chain of 50 cells. Maybe fix their weights to all be the same as in a convolutional neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So are RNN's any use in learning in realtime for temporal data?&lt;/p&gt;&#xA;" OwnerUserId="41463" LastActivityDate="2017-11-17T10:02:17.577" Title="How do you get an RNN to learn in real time?" Tags="|machine-learning|rnn|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-18T12:07:30.383" ContentLicense="CC BY-SA 3.0" />
  <row Id="24840" PostTypeId="2" ParentId="24678" CreationDate="2017-11-17T06:07:13.487" Score="1" Body="&lt;p&gt;I'd plot your data and see if the proportion of the label you are trying to predict is similar or different among the labels for your feature in question.  If similar, you can probably safely remove the column using the &lt;code&gt;nearZeroVar()&lt;/code&gt; function. If different - then the feature is predictive! Keep it, and convert the column into dummy variables.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, imagine I was trying to use a diamond's color (a categorical variable) to determine whether it was expensive (a categorical variable I invent below...)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(dplyr)&#xA;require(ggplot2)&#xA;data('diamonds')&#xA;&#xA;diamonds %&amp;gt;% &#xA;  mutate(expensive = price &amp;gt; 5000) %&amp;gt;%&#xA;  group_by(color,expensive) %&amp;gt;%&#xA;  summarise(n = n()) %&amp;gt;%&#xA;  mutate(freq = n/sum(n)) %&amp;gt;% &#xA;  filter(expensive) %&amp;gt;%&#xA;  ggplot() + &#xA;  geom_bar(aes(color,freq),stat='identity') + &#xA;  labs(x='color',y='proportion expensive')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And you get a plot like this - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A6PDf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A6PDf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where there is a lot of variance in the target label proportion among your feature labels (even if one is pretty rare), I'd keep it as a predictor.  If instead the proportion is very similar - you might as well drop it.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-17T06:07:13.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24841" PostTypeId="1" AcceptedAnswerId="24842" CreationDate="2017-11-17T06:10:12.810" Score="0" ViewCount="3544" Body="&lt;pre&gt;&lt;code&gt;nrow(df1$v1) = 63849&#xA;    nrow(df2$v2) = 3244&#xA;ifelse(df1$v2 == df$v1, 1, 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know this is an easy question but I tried different procedures but none of them are useful, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:nrow(df2)){&#xA;for(j in 1:nrow(df1)){&#xA;    if(df2$v1[i] == df1$v2[j]){&#xA;      df1$v2&amp;lt;- 1&#xA;    }&#xA;    df1$v2 &amp;lt;- 0&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This does the job but it take quiet sometime to get the job done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other Methods:&#xA;Method 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1$v2 &amp;lt;- ifelse(df2$v1 %in% df1$v1, 1,0 )&#xA;Error in `$&amp;lt;-.data.frame`(`*tmp*`, v2 , value = c(1, 1, 1, 1, 1, 1,  : &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;replacement has 3244 rows, data has 63849&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1$v2 &amp;lt;- ifelse(do.call(paste0, df2$v1) %in% &#xA;do.call(paste0, df1$v1), 1,0 )&#xA;summary(df1$v2)&#xA;Min. 1st Qu.  Median    Mean 3rd Qu.    Max. &#xA;      0       0       0       0       0       0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do suggest me if you have any better solution.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-17T06:22:58.793" Title="Comparing 2 data frames, if value is present replace with 1 or else 0" Tags="|r|feature-extraction|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24842" PostTypeId="2" ParentId="24841" CreationDate="2017-11-17T06:22:58.793" Score="0" Body="&lt;p&gt;Here is a way using dplyr:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(dplyr)&#xA;X = data_frame(A=c(1,2,3,4))&#xA;Y = data_frame(B=c(5,4,6,7))&#xA;&#xA;X &amp;lt;- X %&amp;gt;%&#xA;  mutate(A = A %in% Y$B,&#xA;         A = A*1.0) #converts boolean to numeric&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or using base R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X$A &amp;lt;- 1.0*(X$A %in% Y$B)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-17T06:22:58.793" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24843" PostTypeId="1" CreationDate="2017-11-17T07:13:01.973" Score="1" ViewCount="3225" Body="&lt;p&gt;When we fit any model into a data set for prediction, what exactly happens behind the scenes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am learning Regression and I am a little confused about how exactly it fits the classifier or regression into a dataset. &lt;/p&gt;&#xA;" OwnerUserId="42009" LastEditorUserId="29575" LastEditDate="2017-11-18T12:03:20.860" LastActivityDate="2017-11-18T21:47:45.177" Title="In layman terms, what is the meaning of fitting a model into a data set?" Tags="|regression|prediction|classifier|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24844" PostTypeId="2" ParentId="24843" CreationDate="2017-11-17T07:34:48.353" Score="1" Body="&lt;p&gt;You need to decide which algorithm you use based on your &lt;code&gt;Target Variable(Output, Dependent Variable)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your Target Variable is &lt;strong&gt;Categorical&lt;/strong&gt; you need to go for &lt;strong&gt;Classifier&lt;/strong&gt;(Binary/Multi), if your Target Variable is &lt;strong&gt;Continuous&lt;/strong&gt; you need to go for &lt;strong&gt;Prediction Algorithms&lt;/strong&gt; (ie: Regression, Decision Tree, etc...), if the data is Time dependent you need to use &lt;strong&gt;Forecasting Techniques&lt;/strong&gt; (Time Series, Croston, etc...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if you are looking at &lt;strong&gt;Linear Regression&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its basic idea is to fit the data in the form of a line. As we remember in our schooling they taught us about Linear Algebra, where you need to find a solution for a line by substituting points in that, if it satisfies, then it is solvable and many other cases. Same way, here if your target variable is Continuous then you use your features as variables. It attempts to model the relationship between two variables by fitting a linear equation to observed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Equation of Linear Regression is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Y = a + bX$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X$ is the Explanatory(Features) variable and $Y$ is the Dependent(Target) variable&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not get confused with Logistic Regression(LR) and other Regression Techniques, LR is not a Regression technique it is a Classifier, people generally get confused with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding &lt;strong&gt;Classifiers&lt;/strong&gt;, if the data is &lt;strong&gt;Linearly Separable&lt;/strong&gt;(Yes/No, 1/0) then we can use LR, if they are not Linearly Separable you need to use techniques like SVM.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="29575" LastEditDate="2017-11-17T07:45:46.237" LastActivityDate="2017-11-17T07:45:46.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24845" PostTypeId="1" CreationDate="2017-11-17T08:42:22.520" Score="1" ViewCount="130" Body="&lt;p&gt;I want to analyse the Price situation for flat/appartements in my City.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I process the Text from flat ads (like in newspapers or online with its typical structure!) to get Features like size or number of rooms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it simple to take a Package in R or Python (what is suitable?) and to Train the typical structure?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastActivityDate="2017-12-18T03:26:40.017" Title="NLP for appartement ads" Tags="|machine-learning|python|classification|r|deep-learning|" AnswerCount="3" CommentCount="2" ClosedDate="2017-12-18T05:14:37.380" ContentLicense="CC BY-SA 3.0" />
  <row Id="24848" PostTypeId="2" ParentId="24843" CreationDate="2017-11-17T09:25:43.000" Score="0" Body="&lt;p&gt;There is a hole a the wall you want to cover.Let's think about a series questions you would like to ask?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. What is the shape of the hole?&lt;/strong&gt;&#xA;Is it a circle, a square, a ellipse or maybe a triangle? Notice some of those shape is more general than the other, like a square is just a special rectangular.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. How do you describe the exact size of that hole?&lt;/strong&gt;&#xA;what's the diameter of the circle? How long is the size of the square?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;What does this has to do with fitting the model?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The first question about the shape of model is an analogy to the problem of model selection. There are lot of different model you can use to fit a model. There is ordinary linear regression where the model can be represented as a straight line. There is also more complex model like polynomials. And like how square is a special rectangular, linear model is just a very special polynomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once we have a particular type of model we have in mind, we need to find a set a numbers that completely describes that model. This the process of &lt;strong&gt;fitting&lt;/strong&gt; your model. In the case of linear regression, the model is a straight line, and the fitting the model is to find which straight line.&lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-17T09:25:43.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24849" PostTypeId="2" ParentId="24678" CreationDate="2017-11-17T09:27:01.537" Score="1" Body="&lt;p&gt;Your feeling is right, in &lt;strong&gt;some&lt;/strong&gt; cases the order might matter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying nzv after dummyVars means that you lump together very rare levels of nominal predictors. As Tom suggested, you have to make this decision depending on the properties of your particular dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some examples: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You have a classification task and want to identify fraudulent sellers (very imbalanced dataset!). If you apply nzv after dummyVars to the predictor variable „accepted payment type“ you’d lump the level „accept only western union transfers“ (which might be a very good indicator that such sellers are not trustworthy) with some other rare levels. This will significantly worsen the performance of your classifier. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In case of regression tasks, especially when your target function is smooth and nice, it is often less critical if you lump together very rare levels. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You also might want to experiment with different parameters of nzv, such as cutoff frequency. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-11-17T09:27:01.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24850" PostTypeId="2" ParentId="24839" CreationDate="2017-11-17T10:02:17.577" Score="0" Body="&lt;p&gt;I think you misunderstood the fundamental of RNN. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;fix their weights to all be the same&quot;. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes that's exact what RNN is suppose to be. In any type of RNN, you always only have one real RNN cell. The unrolled RNN cells are merely representations of that RNN cell overtime. In a single training step, the weights of those unrolled RNN cells are always the same. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Sp are RNN's any use in learning in real-time for temporal data?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, you can train an RNN in real-time with a catch. Although in theory the sequence is arbitrary long, but in practice we can only backprop a certain step back due to  &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-17T10:02:17.577" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24851" PostTypeId="2" ParentId="24845" CreationDate="2017-11-17T10:21:38.240" Score="0" Body="&lt;p&gt;Look for entity recognition. The entities in your case would be price, number of bedrooms, size etcetera. What I would do is label a number of training examples with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;From character 12 to character 22 indicates the number of bedrooms&lt;/li&gt;&#xA;&lt;li&gt;From character 39 to character 41 indicates the price&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Etcetera.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this you can train a model to classify entities. Sometimes the number of bedrooms will be in text instead of numbers so then you would need to write something that translates it to numerical values but you could likely just use some dictionary to do this for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Entity recognition models are plenty, but I think a character based LSTM approach could work well if your texts are not too long.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-17T10:21:38.240" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24852" PostTypeId="2" ParentId="24809" CreationDate="2017-11-17T10:46:26.130" Score="0" Body="&lt;p&gt;Your question says about classification of questions in 10 different intents/classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion it depends whether which algorithm you are using to classify the data. The one thing that matters is uniform distribution of data in every intent will definitely be helpful. &#xA;You can try following Python libarary which might help you to with classification of intent (Even it supports entity extraction). &#xA;&lt;a href=&quot;https://github.com/RasaHQ/rasa_nlu&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/RasaHQ/rasa_nlu&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27613" LastActivityDate="2017-11-17T10:46:26.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24853" PostTypeId="1" AcceptedAnswerId="24857" CreationDate="2017-11-17T12:04:23.677" Score="6" ViewCount="1630" Body="&lt;p&gt;I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is heavily distributed on only 3-6 topics and all the rest are close to zero ($K$~$\mathcal{O}(100)$). What I mean by this, is that the 3-6 highest contributing topics for all documents is orders of magnitude (about 6 orders) greater than the rest of the topic contributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence&quot; rel=&quot;noreferrer&quot;&gt;Jensen-Shannon distance&lt;/a&gt; to compute the similarity between documents, I need to store all values of the topic distribution as non-zero, even the very small values of the non contributing topics, because Jensen-Shannon divides by each discrete value in the distribution. This requires a lot of storage and is inefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If, however, I store the topic distributions of each document as a sparse matrix (the 3-6 highest contributing topics are non-zero and the rest are zero) where each row is a unique document and each column is a topic, then this uses far less space. But I can no longer use the Jensen-Shannon metric, because we would be dividing by 0. In this case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can I use the euclidean distance between documents topic distributions to compare similarity between documents?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the euclidean distance would require far less storage and is extremely fast to compute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate that Jensen-Shannon is one of the &quot;correct&quot; metrics to compare discrete probability distributions, as well as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bhattacharyya_distance&quot; rel=&quot;noreferrer&quot;&gt;Bhattacharyya distance&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Hellinger_distance&quot; rel=&quot;noreferrer&quot;&gt;Hellinger distance&lt;/a&gt;. But ultimately, the output of LDA is a discrete topic distribution for each doucment - &lt;em&gt;each document is a vector (or point) in a $K$ dimensional space&lt;/em&gt;. By this argument, is it valid to use the euclidean distance to calcualte documents similarities? Is there something blatantly wrong with this method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tested the euclidean distance to compare documents, and yielded good results, which works well for my industrial application. But I want to know the academics behind such a method. Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2017-11-17T13:46:56.820" Title="Can I use euclidean distance for Latent Dirichlet Allocation document similarity?" Tags="|nlp|lda|distance|similar-documents|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24854" PostTypeId="1" CreationDate="2017-11-17T12:11:05.573" Score="0" ViewCount="74" Body="&lt;p&gt;For regression module evaluation, I think only the &lt;code&gt;MAE&lt;/code&gt; (Mean absolute error) value is not objective or practical.&lt;br&gt;&#xA; Consider following situations:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A&lt;br&gt;&#xA;MAE=1 while target value follows the uniform distribution on the interval [1,100]&lt;/li&gt;&#xA;&lt;li&gt;B&lt;br&gt;&#xA;MAE=1 while target value follows the uniform distribution on the interval [1,10]  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously &lt;strong&gt;A model&lt;/strong&gt; is better.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how to get an intuitive value for regression module evaluation, &lt;strong&gt;regardless the data set's target value's scope&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="39820" LastEditorUserId="39820" LastEditDate="2017-11-20T02:29:36.113" LastActivityDate="2018-04-19T07:08:19.440" Title="How to get an intuitive value for regression module evaluation?" Tags="|machine-learning|regression|model-evaluations|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="24855" PostTypeId="1" AcceptedAnswerId="24869" CreationDate="2017-11-17T12:49:44.847" Score="6" ViewCount="6434" Body="&lt;p&gt;I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;noreferrer&quot;&gt;TfIDF&lt;/a&gt;) of each word in each document, multiply each words Tf by its corresponding word vector (this is the weighted part), and sum each of these weighted vectors together element wise to obtain a single vector for a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this method valid&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example, to make it clearer. Take the document:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&quot;The car drives on the road&quot;&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Tf for each word would be &lt;code&gt;{'the':0.3333, 'car':0.1666, 'drives':0.1666, 'on':0.1666, 'road':0.16666}&lt;/code&gt; (obtained by taking the word count of a word and dividing by the total number of words in the document). If we have a trained word2vec model, we can do the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$0.333*\begin{bmatrix}&#xA;    the_0 \\&#xA;    the_1 \\&#xA;    \vdots \\&#xA;    the_n &#xA;\end{bmatrix}&#xA;+ 0.1666*\begin{bmatrix}&#xA;    car_0 \\&#xA;    car_1 \\&#xA;    \vdots \\&#xA;    car_n&#xA;\end{bmatrix}&#xA;+ ...$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where each of the column vectors is the word vector for that word. The final result is an $n\times 1$ dimensional vector representing the document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{bmatrix}&#xA;    0.333*the_0 + 0.1666*car_0 + \dots \\&#xA;    0.333*the_1 + 0.1666*car_1 + \dots \\&#xA;    \vdots \\&#xA;    0.333*the_n + 0.1666*car_n + \dots &#xA;\end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate there are other methods such as &lt;a href=&quot;https://arxiv.org/pdf/1405.4053v2.pdf&quot; rel=&quot;noreferrer&quot;&gt;doc2vec&lt;/a&gt; that aim to do very much the same, but in a much more sophisticated way. But is my method valid / is there anything blatantly wrong here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tested this method and even used some document similarity metrics (yes, after normalizing the vectors of course), and yielded some good results for my industrial application. But I want to know more about the academics of this method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nice thing about this way, is that by using the word2vec vectors, similarity queries between documents yield very good results due to semantic similarities (euclidean closeness) between word vectors, even if different words are used across documents; this is something TfIDF cannot do as each word is treated differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2020-07-15T12:22:44.887" Title="Weighted sum of word vectors for document similarity" Tags="|nlp|word2vec|tfidf|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24857" PostTypeId="2" ParentId="24853" CreationDate="2017-11-17T13:46:56.820" Score="2" Body="&lt;p&gt;Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is a valid measure to use in comparing the topics represented within two documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you're doing by applying this method is quantifying a topic frequency difference within this newly defined space, and so &lt;em&gt;interpretation&lt;/em&gt; of these quanta will require analysis of the space. For example, what euclidean distance indicates that documents are relatively similar? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In distiction, the normalized result of something like the hellinger distance provides an easily interperable framework by which to evaluate the results- a score of 0 indicates no overlap in the distribution over the topics in question of the two documents, and a 1, perfect overlap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the efficiency concerns, it's not clear to me why you couldn't truncate your topics considered to the crucial topics and then calculate any of the metrics on the distributions over ony those topics, rather than the entire universe of considered topics.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2017-11-17T13:46:56.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24858" PostTypeId="1" AcceptedAnswerId="24861" CreationDate="2017-11-17T14:26:03.863" Score="-1" ViewCount="308" Body="&lt;p&gt;Suppose that we have a dataset of 2 samples :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; [{1,2,0}, {2,0,0}, {3,1,1}, {4,0,1}, {5,1,1}] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(the last element of each row is the class variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we want to reduce the variety of the column &quot;1&quot; we can put all values that are less or equal to &quot;2&quot; to a new value &quot;1&quot; and values that are greater than &quot;2&quot; to the value 2, so we would have a new dataset like : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[{1,2,0}, {1,0,0}, {2,1,1}, {2,0,1}, {2,1,1}] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example we can easly deduct that there is a information gain after this transfromation, but what if we have million of sample, is there any way to measure the information gain / loss after such a transformation.&lt;/p&gt;&#xA;" OwnerUserId="42029" LastEditorUserId="42029" LastEditDate="2017-11-17T16:01:03.423" LastActivityDate="2017-11-17T16:57:12.117" Title="Measure information gain / loss after a data Transformation operation" Tags="|information-theory|tranformation|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-18T12:07:13.130" ContentLicense="CC BY-SA 3.0" />
  <row Id="24860" PostTypeId="1" AcceptedAnswerId="25801" CreationDate="2017-11-17T15:23:37.800" Score="6" ViewCount="215" Body="&lt;p&gt;Is it possible to stack two networks on top of each other that operate on different resolutions of input data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here's my usecase: &lt;a href=&quot;https://arxiv.org/pdf/1704.03549.pdf&quot; rel=&quot;noreferrer&quot;&gt;like Google&lt;/a&gt;, I want to recognize text in images. Unlike Google, I have access to very limited computational resources. My solution for this is that instead of using a sequence-to-sequence network on the whole scene image, I first run object detection using &lt;a href=&quot;https://pjreddie.com/darknet/yolo/&quot; rel=&quot;noreferrer&quot;&gt;YOLO&lt;/a&gt; and then pass the crop to a sequence-to-sequence model, &lt;a href=&quot;https://github.com/da03/Attention-OCR&quot; rel=&quot;noreferrer&quot;&gt;Attention-OCR&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To further reduce processing time, I run the object detection on low resolution, and crop the detection result in the higher resolution input image, so I still have access to the high resolution input when using sequence recognition on my crop. To detect that there is text, the low resolution is adequate, but to read the text, the network needs the higher resolution input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This all works fine, but I suspect I would get better performance if I could train the whole system end-to-end, so the crop gets directly optimized to result in the best reading of the text. I could stack the sequence-to-sequence model on top of the object detection model to do this, but then the text reading operates on the same low resolution input that is used for the text detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have an idea how to solve this, or can anyone point me to research that is related to this?&lt;/p&gt;&#xA;" OwnerUserId="42031" LastEditorUserId="42031" LastEditDate="2017-11-17T15:53:28.997" LastActivityDate="2020-08-15T09:12:25.707" Title="Can I train two stacked models end-to-end on different resolutions?" Tags="|machine-learning|neural-network|rnn|ocr|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24861" PostTypeId="2" ParentId="24858" CreationDate="2017-11-17T15:50:05.583" Score="1" Body="&lt;p&gt;It will help to define what you mean by information.  The usual definition is 'Shannon information', which is equivalent to the amount of uncertainty an observation resolves.  If you mean something else, nothing I am about to write will apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps what you are looking for is the idea of &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow noreferrer&quot;&gt;mutual information&lt;/a&gt;.  It is a way to measure the shared information between two variables -&#xA; literally how much your uncertainty about one variable is reduced by knowing the value of another variable.  Closely related &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_entropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;conditional entropy&lt;/a&gt;, which measures the amount of uncertainty we still have about a variable given that we know the value of a second variable.  Those references might be worth reading to get a sense of how information between two variables is defined in a technical sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, your operation actually does &lt;em&gt;not&lt;/em&gt; increase the information in your dataset.  Your logical operation does not bring in any outside observation to the system, but is merely a rule-baesd transformation.  It is therefore subject to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Data_processing_inequality&quot; rel=&quot;nofollow noreferrer&quot;&gt;data processing inequality&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Re-reading your question, I think you are asking about the difference between I(X;Y) and I(X';Y), where I is the mutual information function, X is your input data, and Y is your target label.  So basically the answer to your question is Info gain = I(X';Y) - I(X;Y).  You can look up mutual information approximation methods - there are packages for most languages.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastEditorUserId="41886" LastEditDate="2017-11-17T16:57:12.117" LastActivityDate="2017-11-17T16:57:12.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24862" PostTypeId="2" ParentId="8457" CreationDate="2017-11-17T17:35:58.083" Score="2" Body="&lt;p&gt;There is a &lt;a href=&quot;https://www.datadoghq.com/blog/engineering/piecewise-regression/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; with a recursive implementation of piecewise regression. That solution fits discontinuous regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are unsatisfied with discontinuous model and want continuous seting, I would propose to look for your curve in a basis of &lt;code&gt;k&lt;/code&gt; L-shaped curves, using Lasso for sparsity:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.linear_model import Lasso&#xA;# generate data&#xA;np.random.seed(42)&#xA;x = np.sort(np.random.normal(size=100))&#xA;y_expected = 3 + 0.5 * x + 1.25 * x * (x&amp;gt;0)&#xA;y = y_expected + np.random.normal(size=x.size, scale=0.5)&#xA;# prepare a basis&#xA;k = 10&#xA;thresholds = np.percentile(x, np.linspace(0, 1, k+2)[1:-1]*100)&#xA;basis = np.hstack([x[:, np.newaxis],  np.maximum(0,  np.column_stack([x]*k)-thresholds)]) &#xA;# fit a model&#xA;model = Lasso(0.03).fit(basis, y)&#xA;print(model.intercept_)&#xA;print(model.coef_.round(3))&#xA;plt.scatter(x, y)&#xA;plt.plot(x, y_expected, color = 'b')&#xA;plt.plot(x, model.predict(basis), color='k')&#xA;plt.legend(['true', 'predicted'])&#xA;plt.xlabel('x')&#xA;plt.ylabel('y')&#xA;plt.title('fitting segmented regression')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code will return a vector of estimated coefficients to you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.57   0.     0.     0.     0.     0.825  0.     0.     0.     0.     0.   ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Due to Lasso approach, it is sparse: the model found exactly one breakpoint among 10 possible. Numbers 0.57 and 0.825 correspond to 0.5 and 1.25 in the true DGP. Although they are not very close, the fitted curves are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/swltL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/swltL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach does not allow you to estimate the breakpoint exactly. But if your dataset is large enough, you can play with different &lt;code&gt;k&lt;/code&gt; (maybe tune it by cross-validation) and estimate the breakpoint precisely enough.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2017-11-17T17:35:58.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24863" PostTypeId="1" CreationDate="2017-11-17T17:52:56.730" Score="6" ViewCount="230" Body="&lt;p&gt;I'm trying to build a neural network with an unconventional architecture and a having trouble figuring out how.  Usually we have connections like so, where $X=$ input, $H=$ hidden layer, $Y=$ output layer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X_t \rightarrow H_t \rightarrow Y_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H_t \rightarrow H_{t+1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normal Tensorflow or Keras nodes build above, where we end up with weight matrices connecting each component, i.e. $W_{xh}, W_{hh}, W_{hy}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also like to introduce a connection:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y_t \rightarrow H_{t+1}$, defined by a new weight matrix $W_{yh}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at the Tensorflow and Keras implementations and there is a &lt;code&gt;self.kernel&lt;/code&gt; ($W_{xh}$) and &lt;code&gt;self.recurrent_kernel&lt;/code&gt; ($W_{hh}$), where the second is defined in terms of the previous hidden node's output &lt;code&gt;prev_output = states[0]&lt;/code&gt;, defined &lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py#L893&quot; rel=&quot;noreferrer&quot;&gt;here.&lt;/a&gt; Does anyone have a sense of how to access the output of the output layer in the subsequent step?  Is it &lt;code&gt;states[1]&lt;/code&gt;?  How are &lt;code&gt;states&lt;/code&gt; defined, and where is that documented?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insight you can give! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Before anyone asks yes, I recognize that such a connection is not strictly necessary. I am trying to understand information flow through network nodes.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-17T17:52:56.730" Title="Connect output node to next hidden node in RNN" Tags="|tensorflow|keras|rnn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24864" PostTypeId="1" CreationDate="2017-11-17T18:01:18.467" Score="-1" ViewCount="270" Body="&lt;p&gt;Suppose I have a supervised machine learning regression problem to predict a value that's supposed to improve upon an already-existing algorithm. I am trying to predict the distance that a small rocket will travel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We currently have an algorithm that uses ballistic trajectory equations that produces a value called &lt;code&gt;physics_distance_in_km&lt;/code&gt;, and we measure RMSE against the &lt;code&gt;true_distance&lt;/code&gt; label in a test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We wanted to create a more accurate estimate for how far the small rocket will travel, so we formed a regression problem, added more features, and built models using OLS regression, random forest regression, neural network, etc. We are using &lt;code&gt;physics_distance_in_km&lt;/code&gt; as a feature along with &lt;code&gt;wind_speed&lt;/code&gt;, &lt;code&gt;air_density&lt;/code&gt;, &lt;code&gt;propellant_type&lt;/code&gt;, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We found that while our machine learning regression algorithm improves our RMSE slightly on average, sometimes it is quite far off from the true distance and &lt;code&gt;physics_distance_in_km&lt;/code&gt;. As one rare example, the rocket went 15 km, &lt;code&gt;physics_distance_in_km&lt;/code&gt; predicted 12 km, but our regression output predicted 29 km, which is possible due to the regression not having a bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of using &lt;code&gt;physics_distance_in_km&lt;/code&gt; as a feature in our model, should we instead leave it out? Instead of the distance as a label, we could use &lt;code&gt;delta_distance&lt;/code&gt; as the label computed as &lt;code&gt;true_distance&lt;/code&gt; - &lt;code&gt;physics_distance_in_km&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="40240" LastActivityDate="2017-11-17T18:52:14.920" Title="Regression problem as predicting a delta from another algorithm's output" Tags="|machine-learning|neural-network|regression|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24865" PostTypeId="2" ParentId="23376" CreationDate="2017-11-17T18:41:20.017" Score="6" Body="&lt;p&gt;I was facing the exact same issue, this is what I did:&lt;br /&gt;&#xA;Catch the key error you get when the word is not found in cmu's dictionary as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import cmudict&#xA;d = cmudict.dict()&#xA;&#xA;def nsyl(word):&#xA;    try:&#xA;        return [len(list(y for y in x if y[-1].isdigit())) for x in d[word.lower()]]&#xA;    except KeyError:&#xA;        #if word not found in cmudict&#xA;        return syllables(word)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Call the below syllables function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def syllables(word):&#xA;    #referred from stackoverflow.com/questions/14541303/count-the-number-of-syllables-in-a-word&#xA;    count = 0&#xA;    vowels = 'aeiouy'&#xA;    word = word.lower()&#xA;    if word[0] in vowels:&#xA;        count +=1&#xA;    for index in range(1,len(word)):&#xA;        if word[index] in vowels and word[index-1] not in vowels:&#xA;            count +=1&#xA;    if word.endswith('e'):&#xA;        count -= 1&#xA;    if word.endswith('le'):&#xA;        count += 1&#xA;    if count == 0:&#xA;        count += 1&#xA;    return count&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42042" LastEditorUserId="89115" LastEditDate="2020-01-29T23:54:47.507" LastActivityDate="2020-01-29T23:54:47.507" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24866" PostTypeId="2" ParentId="24864" CreationDate="2017-11-17T18:52:14.920" Score="1" Body="&lt;p&gt;I think predicting model &lt;em&gt;error&lt;/em&gt; (&lt;code&gt;delta_distance&lt;/code&gt;) makes a lot of sense and is worth trying. Since it's reasonable to expect that the error is a function of the overall rocket distance, I would keep &lt;code&gt;physics_distance_in_km&lt;/code&gt; as an input to the error-prediction model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding outliers...I suspect that's a function of your model type. It's difficult to say without more details of what you did.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-17T18:52:14.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24867" PostTypeId="1" CreationDate="2017-11-17T19:57:43.057" Score="3" ViewCount="1211" Body="&lt;p&gt;I'm training a 4-class classifier. I load pre-trained weights from vgg16, and only set the last layer to be trainable. This gives me a loss (categorical crossentropy) of 0.9 and a training accuracy of 70%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I want features for my images - the penultimate activations is one choice for getting features for every image. Hence I set the last and the penultimate layers to both be trainable, but then my loss jumps to 9.8 and training accuracy drops to 39%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this happen? Here's the code I'm using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;vgg.model.pop()&#xA;for layer in vgg.model.layers:&#xA;    layer.trainable = False&#xA;vgg.model.add(Dense(trn_batches.nb_class, activation='softmax'))&#xA;opt = Adam(lr=0.001)&#xA;&#xA;#only the last layer is trainable&#xA;vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])&#xA;vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1, &#xA;    validation_data=val_batches, nb_val_samples=val_batches.nb_sample)&#xA;&#xA;#make last 2 layers trainable (starting from -3 because there is a dropout layer)&#xA;for layer in vgg.model.layers[-3:]:&#xA;    layer.trainable = True&#xA;vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])&#xA;vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,   validation_data=val_batches, nb_val_samples=val_batches.nb_sample)&#xA;&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="6488" LastActivityDate="2018-01-01T00:57:25.423" Title="Accuracy drops if more layers trainable - weird" Tags="|neural-network|deep-learning|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24868" PostTypeId="1" AcceptedAnswerId="24870" CreationDate="2017-11-17T20:23:07.440" Score="1" ViewCount="2051" Body="&lt;p&gt;I am trying to extract various Named Entities from a Spanish language text file. I tried using nltk but to no success. I am using Python 2 with nltk 3.x.&lt;/p&gt;&#xA;" OwnerUserId="26444" LastEditorUserId="85045" LastEditDate="2020-12-06T22:06:22.513" LastActivityDate="2020-12-06T22:06:22.513" Title="Extracting NER from a Spanish language text file" Tags="|nlp|information-retrieval|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24869" PostTypeId="2" ParentId="24855" CreationDate="2017-11-17T20:36:35.443" Score="7" Body="&lt;p&gt;Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: &lt;a href=&quot;http://white.ucc.asn.au/publications/White2015SentVecMeaning.pdf&quot; rel=&quot;noreferrer&quot;&gt;How Well Sentence Embeddings Capture Meaning&lt;/a&gt;. It discusses your method and give some pointers to other papers that also discuss the validity of the method.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-17T20:36:35.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24870" PostTypeId="2" ParentId="24868" CreationDate="2017-11-17T20:41:40.160" Score="1" Body="&lt;p&gt;I suggest you take a look at the Python library &lt;a href=&quot;https://spacy.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spacy&lt;/a&gt; it has a model for &lt;a href=&quot;https://spacy.io/models/es&quot; rel=&quot;nofollow noreferrer&quot;&gt;spanish language&lt;/a&gt; that includes NER.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-17T20:41:40.160" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24871" PostTypeId="2" ParentId="20303" CreationDate="2017-11-17T21:28:36.427" Score="1" Body="&lt;p&gt;Do you just want to map your models on a scale of 1-10? If you are thinking of error linearly, then we can just find a linear transformation with your MAPE (or RMSE) scores. We &quot;map&quot; 0.449 to 10 and 1.840 to 1. So basically mapping &quot;MAPE&quot; to &quot;Rating&quot; by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rating = 12.9051 - 6.47017*MAPE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do a similar thing with RMSE. I'm not sure how meaningful this will be though.&lt;/p&gt;&#xA;" OwnerUserId="42044" LastActivityDate="2017-11-17T21:28:36.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24872" PostTypeId="1" CreationDate="2017-11-17T22:41:14.620" Score="3" ViewCount="4927" Body="&lt;p&gt;All the samples and articles i have seen are all having outputs of 1 or less. Is there a hidden reason why no one is using NN to produce higher value integers?? My situation is that I want NN to predict the exact outcome value rather than a -1&#xA;&#xA;&lt;p&gt;If I have a NN with say 4 inputs and 1 output with &#xA;Round 1 Input: 51,22,35,43 Output: 847&#xA;Round 2 Input: 25,31,46,29 Output: 761&#xA;I want &#xA;Round n Input: 51,33,19,27 Output: ??? (expected value)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read that GA is better in training NN but few examples have been provided to guide beginners like me. Am yet to see a detailed article on how to get this done. Using same example above, if i get the (near) best weights from first evolution using Round 1 error value, do i add the weight as parents in the next generation for Round 2?? How do i ensure that the best chromosome from previous rounds don't die after that generation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Links to existing articles and detailed explanation will be very much appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="42046" LastEditorUserId="42046" LastEditDate="2017-11-18T05:59:34.967" LastActivityDate="2017-11-22T21:46:33.747" Title="Can I have a neural network output values &gt;100?" Tags="|neural-network|algorithms|java|genetic-algorithms|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24874" PostTypeId="2" ParentId="24872" CreationDate="2017-11-17T23:43:35.220" Score="2" Body="&lt;p&gt;Is sounds like you want to use Neural Networks to do a regression problem instead of classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; gives an example of using a neural network to do regression.&lt;/p&gt;&#xA;" OwnerUserId="13335" LastActivityDate="2017-11-17T23:43:35.220" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24875" PostTypeId="1" AcceptedAnswerId="24879" CreationDate="2017-11-17T23:46:23.007" Score="1" ViewCount="430" Body="&lt;p&gt;I created an ANN in Python 3.  My backpropagation algorithm seems to work up to a point where the gradient becomes very small. I am familiar with the vanishing gradient problem, but I found that it only applies to really deep network; my simple test network is no such network.  It consists of an input layer (1 input node and bias), no hidden layers, and an output layer (1 output node).  &lt;strong&gt;How do I stop the gradient from vanishing?&lt;/strong&gt; Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from random import random&#xA;&#xA;class Neural_Network(object):&#xA;    def __init__(self):&#xA;        # Create a simple deterministic network for testing&#xA;&#xA;        # Define Hyperparameters&#xA;        self.inputLayerSize = 1&#xA;        self.outputLayerSize = 1&#xA;        self.hiddenLayerSize = 0&#xA;        self.numHiddenLayer = 0&#xA;        self.numExamples = 20&#xA;        self.learningRate = 0.07 # LEARNING RATE&#xA;        self.weightDecay = 0&#xA;        # in -&amp;gt; out&#xA;        self.weights = [] # stores matrices of each layer of weights&#xA;        self.z = [] # stores matrices of each layer of weighted sums&#xA;        self.a = [] # stores matrices of each layer of activity &#xA;        self.biases = [] # stores all biases&#xA;        self.biasNodes = []&#xA;&#xA;        # Biases are matrices that are added to activity matrix&#xA;        # Dimensions -&amp;gt; numExamples_*hiddenLayerSize or numExamples_*outputLayerSize&#xA;&#xA;        # Biases for output layer&#xA;        b = [0.5 for x in range(self.outputLayerSize)]&#xA;        B = [b for x in range(self.numExamples)];&#xA;        self.biases.append(np.mat(B))&#xA;&#xA;        # Bias nodes&#xA;        b= [1 for x in range(self.numExamples)]&#xA;        for i in range(self.numHiddenLayer+1):&#xA;            self.biasNodes.append(np.mat(b).reshape([self.numExamples,1]))&#xA;&#xA;        # Weights (Parameters)&#xA;        # Weight matrix between input and output layer&#xA;        W = np.matrix(&quot;0.5&quot;);&#xA;        self.weights.append(W)&#xA;&#xA;&#xA;&#xA;    def setBatchSize(self, numExamples):&#xA;        # Changes the number of rows (examples) for biases&#xA;        if (self.numExamples &amp;gt; numExamples):&#xA;            self.biases = [b[:numExamples] for b in self.biases]&#xA;&#xA;    def hypTan(self, z):&#xA;        # Apply hyperbolic tangent function&#xA;        return (np.exp(z) - np.exp(-z)) / (np.exp(z) + np.exp(-z))&#xA;&#xA;    def hypTanPrime(self, z):&#xA;        # Apply derivative hyperbolic tangent function&#xA;        return 4/np.multiply((np.exp(z) + np.exp(-z)), (np.exp(z) + np.exp(-z)))&#xA;&#xA;&#xA;    def forward(self, X):&#xA;        # Propagate outputs through network&#xA;        self.z = []&#xA;        self.a = []&#xA;&#xA;        self.z.append(np.dot(X, self.weights[0]) + self.biases[0])&#xA;        self.a.append(self.hypTan(self.z[0]))&#xA;&#xA;        yHat = self.a[-1]&#xA;        return yHat&#xA;&#xA;    def backProp(self, X, y):&#xA;        # Compute derivative wrt W&#xA;        # out -&amp;gt; in&#xA;        dJdWb = [] # stores matrices of each dJdWb value &#xA;        dJdW = [] # stores matrices of each dJdW (equal in size to self.weights[])&#xA;        delta = [] # stores matrices of each backpropagating error&#xA;        result = () # stores dJdW and dJdWb&#xA;        self.yHat = self.forward(X)&#xA;&#xA;        # Quantifying Error&#xA;        print(np.linalg.norm(y-self.yHat)/np.linalg.norm(y+self.yHat))&#xA;&#xA;&#xA;        delta.insert(0,np.multiply(-(y-self.yHat), self.hypTanPrime(self.z[-1]))) # delta = (y-yHat)(sigmoidPrime(final layer unactivated))&#xA;        dJdW.insert(0, np.dot(X.T, delta[0]) + (self.weightDecay*self.weights[-1]))&#xA;        dJdWb.insert(0, np.dot(self.biasNodes[-1].T, delta[0]) + (self.weightDecay*self.biases[-1])) # you need to backpropagate to bias nodes&#xA;&#xA;&#xA;        result = (dJdW, dJdWb)&#xA;        return result&#xA;&#xA;    def train(self, X, y):&#xA;        for t in range(10000):&#xA;            dJ = self.backProp(X, y)&#xA;            dJdW = dJ[0]&#xA;            dJdWb = dJ[1]&#xA;            for i in range(len(dJdW)):&#xA;                print(&quot;dJdW:&quot;, dJdW[i], sep = &quot; &quot;, end = &quot;\n&quot;)&#xA;                print(&quot;dJdWb:&quot;, dJdWb[i], sep = &quot; &quot;, end = &quot;\n\n&quot;)&#xA;                #print(&quot;Weights:&quot;, self.weights[i]);&#xA;                self.weights[i] -= self.learningRate*dJdW[i]&#xA;                self.biases[i] -= self.learningRate*dJdWb[i]&#xA;&#xA;&#xA;# Instantiating Neural Network&#xA;&#xA;&#xA;# Instantiating Neural Network&#xA;&#xA;NN = Neural_Network() # create a deterministic NN for testing&#xA;x = np.matrix(&quot;0.025; 0.05; 0.075; 0.1; 0.125; 0.15; 0.175; 0.2; 0.225; 0.25; 0.275; 0.3; 0.325; 0.35; 0.375; 0.4; 0.425; 0.45; 0.475; 0.5&quot;)&#xA;y = np.matrix(&quot;0.05; 0.1; 0.15; 0.2; 0.25; 0.3; 0.35; 0.4; 0.45; 0.5; 0.55; 0.6; 0.65; 0.7; 0.75; 0.8; 0.85; 0.9; 0.95; 1.0&quot;)&#xA;&#xA;# Training&#xA;print(&quot;INPUT: &quot;, end = '\n')&#xA;print(x, end = '\n\n')&#xA;&#xA;print(&quot;BEFORE TRAINING&quot;, NN.forward(x), sep = '\n', end = '\n\n')&#xA;print(&quot;ERROR: &quot;)&#xA;NN.train(x,y)&#xA;print(&quot;\nAFTER TRAINING&quot;, NN.forward(x), sep = '\n', end = '\n\n')&#xA;&#xA;NN.setBatchSize(1) # changing settings to receive one input at a time&#xA;&#xA;while True:&#xA;    inputs = input()&#xA;    x = np.mat([float(i) for i in inputs.split(&quot; &quot;)])&#xA;    print(NN.forward(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you run the program it will show the dJdW value (gradient values w.r.t weights) and dJWb values (gradient values w.r.t bias weights).  Then it will test the inputs on the newly trained network and print the outputs.  After that, you can give the network your own inputs (between 0 and 0.5 since i trained the network to multiply inputs by 2) and it will return outputs in console.  Please note that this is a highly simplified version of my real network.  I want to fix the problem here before adressing it in the full version.&lt;/p&gt;&#xA;" OwnerUserId="41755" LastActivityDate="2017-11-18T09:10:21.050" Title="Vanishing Gradient in a shallow network" Tags="|machine-learning|neural-network|algorithms|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24876" PostTypeId="1" CreationDate="2017-11-18T01:19:20.370" Score="4" ViewCount="14505" Body="&lt;p&gt;Decision trees handle only discrete values, but the continuous values we need to transform to discrete.  My question is HOW?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the steps which are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sort the value A in increasing order.&lt;/li&gt;&#xA;&lt;li&gt;Find the midpoint between the values of $a_i$ and $a_{i+1}$.&lt;/li&gt;&#xA;&lt;li&gt;Find entropy for each value.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have this example can someone explain how we calculated the entropy values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gtOWm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gtOWm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42049" LastEditorUserId="29169" LastEditDate="2021-02-13T19:21:26.120" LastActivityDate="2021-02-13T19:21:26.120" Title="How Can I Compute Information-Gain for Continuous- Valued Attributes" Tags="|dataset|data-mining|predictive-modeling|decision-trees|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24878" PostTypeId="1" AcceptedAnswerId="24883" CreationDate="2017-11-18T08:31:41.403" Score="17" ViewCount="12707" Body="&lt;p&gt;Are GANs (generative adversarial networks) good just for images or can they be used for text as well?&lt;/p&gt;&#xA;&lt;p&gt;Like training a network to generate meaningful text from a summary.&lt;/p&gt;&#xA;&lt;p&gt;UPD - quotes from the GAN inventor Ian Goodfellow.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;GANs have not been applied to NLP because GANs are only defined for&#xA;real-valued data. (&lt;a href=&quot;https://www.quora.com/Is-text-generation-using-GANs-a-fundamentally-flawed-idea&quot; rel=&quot;nofollow noreferrer&quot;&gt;2016&lt;/a&gt;) source&lt;/p&gt;&#xA;&lt;p&gt;It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) &lt;a href=&quot;https://www.quora.com/Is-text-generation-using-GANs-a-fundamentally-flawed-idea&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33183" LastEditorUserId="29169" LastEditDate="2021-03-11T20:13:21.423" LastActivityDate="2021-03-11T20:13:21.423" Title="GANs (generative adversarial networks) possible for text as well?" Tags="|gan|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="24879" PostTypeId="2" ParentId="24875" CreationDate="2017-11-18T09:10:21.050" Score="0" Body="&lt;p&gt;This is not a &quot;vanishing gradient&quot; problem, it is just your network converging as designed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is normal for gradients to become low as you approach convergence. In a really simple problem, like your linear regression, it is relatively easy to get a gradient of zero (within combined rounding errors). That is because, near a stationary point, gradients do approach zero. This can also be a weakness of gradient descent in general - learning will slow or halt near any stationary point, hence concerns about finding local minima as opposed to global minima.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way to check your gradients are correct is to test them against small weight deltas. Pick a set of weight parameters for the network. Calculate the gradients using your code. Then to test, take each weight in turn, change it by +/- $\epsilon$ and use both variants to generate cost values, use them to estimate the gradient for that weight $\frac{J_{+\epsilon} - J_{-\epsilon}}{2\epsilon}$. Use this alternative (and much slower) gradient calculation to build up a second opinion of what &lt;code&gt;dJdW&lt;/code&gt; should be. Then compare the two values. This process is often called &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;gradient checking&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-11-18T09:10:21.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24880" PostTypeId="1" CreationDate="2017-11-18T09:13:50.947" Score="-1" ViewCount="144" Body="&lt;p&gt;I have a &lt;strong&gt;Multiclass Dataset&lt;/strong&gt; and I am getting probabilities of classes from RandomForest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I want to divide the dataset for each class as examples of either the case belongs to that class or not(Binary Classification). I want to know , when I get the prediction of those models for a yes or no. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I ensemble them in such a way that &lt;strong&gt;Overall&lt;/strong&gt; Sum of probabilities as one?&lt;/p&gt;&#xA;" OwnerUserId="17041" LastEditorUserId="40853" LastEditDate="2017-11-18T10:55:03.273" LastActivityDate="2017-11-18T10:55:03.273" Title="Ensemble Probabilities of the different models" Tags="|random-forest|probability|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24881" PostTypeId="2" ParentId="24880" CreationDate="2017-11-18T09:36:33.320" Score="3" Body="&lt;p&gt;I think it can be done by using this command at the time of prediction, giving example in R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#To predict with probabilities&#xA;testSet$pred_rf_prob&amp;lt;-`predict(object,model_rf,testSet[,predictors],type='prob')`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To take average of the predictions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;testSet$pred_avg&amp;lt;-(testSet$pred_rf_prob$Y+testSet$pred_knn_prob$Y+testSet$pred_lr_prob$Y)/3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/02/introduction-to-ensembling-along-with-implementation-in-r/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; , might be helpful. Do have a look.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-18T09:36:33.320" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24882" PostTypeId="1" AcceptedAnswerId="24884" CreationDate="2017-11-18T09:46:18.343" Score="2" ViewCount="386" Body="&lt;p&gt;Kmodes is for categorical Data. But sometimes I read it somewhere that it is applicable on Mixed Data i.e., Numeric and categorical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So whats right?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastEditorUserId="40853" LastEditDate="2017-11-18T12:03:23.837" LastActivityDate="2017-11-19T15:15:07.470" Title="Kmodes for Mixed Data" Tags="|machine-learning|classification|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24883" PostTypeId="2" ParentId="24878" CreationDate="2017-11-18T09:52:33.070" Score="19" Body="&lt;p&gt;Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;GANs work by propagating gradients through the composition of Generator and Discriminator.&lt;/li&gt;&#xA;&lt;li&gt;Text is normally generated by having a final softmax layer over the token space, that is, the output of the network is normally the probabilities of generating each token (i.e. a discrete stochastic unit). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the &lt;a href=&quot;http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf&quot; rel=&quot;noreferrer&quot;&gt;REINFORCE algorithm&lt;/a&gt; and the &lt;a href=&quot;https://arxiv.org/abs/1611.01144&quot; rel=&quot;noreferrer&quot;&gt;Gumbel-Softmax reparameterization&lt;/a&gt; (also known as the &lt;a href=&quot;https://arxiv.org/abs/1611.00712&quot; rel=&quot;noreferrer&quot;&gt;Concrete distribution&lt;/a&gt;). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example of REINFORCE for textual GANs you can check the &lt;a href=&quot;https://arxiv.org/abs/1609.05473&quot; rel=&quot;noreferrer&quot;&gt;SeqGAN article&lt;/a&gt;. An example of Gumbel-Softmax you can check &lt;a href=&quot;https://arxiv.org/abs/1611.04051&quot; rel=&quot;noreferrer&quot;&gt;this article&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2017-11-23T08:50:12.710" LastActivityDate="2017-11-23T08:50:12.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24884" PostTypeId="2" ParentId="24882" CreationDate="2017-11-18T10:44:20.920" Score="2" Body="&lt;p&gt;As far as I remember Kmodes is used for Categorical data, even in the documentation I couldn't find anything related to Mixed Data Type, if you have some reference do share.&lt;/p&gt;&#xA;&lt;p&gt;I've used &lt;strong&gt;Kproto&lt;/strong&gt; for Mixed Data types as you have mentioned and the package is &lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType/clustMixType.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ClustMixType&lt;/a&gt;, it combination of &lt;strong&gt;Kmeans and Kmodes&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Do have a look and let me know if have any questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-18T10:44:20.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24885" PostTypeId="2" ParentId="24876" CreationDate="2017-11-18T11:25:50.627" Score="4" Body="&lt;p&gt;In decision trees, the (Shannon) entropy is not calculated on the actual attributes, but on the class label. If you wanted to find the entropy of a continuous variable, you could use &lt;a href=&quot;https://en.wikipedia.org/wiki/Differential_entropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Differential entropy&lt;/a&gt; metrics such as KL divergence, but that's not the point about decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When finding the entropy for a splitting decision in a decision tree, you find a threshold (such as midpoint or anything you come up with), and count the amount of each class label on each size of the threshold. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Var1 | Class&#xA;0.75 | 1&#xA;0.87 | 0&#xA;0.89 | 1&#xA;0.96 | 0&#xA;1.02 | 1&#xA;1.05 | 1&#xA;1.14 | 1&#xA;1.25 | 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's assume a threshold of 0.99 because I want two equal-sized bins and that's the midpoint between 0.96 and 1.02. The classes on the left are [1, 0, 1, 0] (since it's half/half, entropy is 1.0), and on the right [1, 1, 1, 1] (since it's all equal, entropy is 0.0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a table like the one you show here, you need to define N threshold points (2.0, 3.5, ..., 7.5, in your case), which will produce N+1 bins of data, you take the class of each bin and calculate the entropy. A more correct visualization to your table should be changing the &quot;Split point&quot; values to &quot;2.0 or lower&quot;, &quot;Between 2.0 and 3.5&quot;, &quot;Between 3.5 and 4.5&quot;, ..., &quot;7.5 or higher&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note this all applies to classification (decision) trees, and not regression trees, where the &quot;class&quot; is actually a continuous variable.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-11-18T11:25:50.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24886" PostTypeId="5" CreationDate="2017-11-18T13:37:50.723" Score="0" Body="&lt;p&gt;GAN refers to &lt;a href=&quot;https://arxiv.org/abs/1406.2661&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Networks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Such networks is made of two networks that compete against each other:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The first network generates new samples, based on random input samples to create variability&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The second network discriminates between generated samples and true samples. It usually entails running the first network to generate a new sample and the discriminator network must tag it as artificial. At the same time, a true sample is passed in the discriminator to and this one must be tagged as non artificial.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;By optimizing the first network in alternance with the second one, more and more realistic samples will be created by the generative network which can then be used without the discriminator once trained.&lt;/p&gt;&#xA;" OwnerUserId="61418" LastEditorUserId="61418" LastEditDate="2018-12-20T00:32:36.783" LastActivityDate="2018-12-20T00:32:36.783" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24887" PostTypeId="4" CreationDate="2017-11-18T13:37:50.723" Score="0" Body="GAN refers to Generative Adversarial Networks.&#xD;&#xA;&#xD;&#xA;Such networks is made of two networks that compete against each other. The first one generates new samples and the second one discriminates between generated samples and true samples." OwnerUserId="61418" LastEditorUserId="61418" LastEditDate="2018-12-20T00:32:44.713" LastActivityDate="2018-12-20T00:32:44.713" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24889" PostTypeId="1" CreationDate="2017-11-18T14:08:00.540" Score="2" ViewCount="151" Body="&lt;p&gt;I am working on an exercise for using PCA for compression of images and I don't quite understand how to use it on the test data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 300 images of hand drawn sixes, represented by 28x28 matrices, in the train data, and I have used PCA to find an appropriate low dimensional representation of these images (26 dimensions yields me the sought after 90% threshold), giving me a 300x26 Matrix, that I can use to project my images into that space&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have to test this with my test data of 10 similar images - so I have to project them into the same space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because I can't just use the train space (dimensions don't agree), if I understood correctly, I should run another PCA to find the Principle Components for these new images, but project them into the 26 dimensions as identified by my train PCA (PCA on just the test suggests that 6 dimensions would suffice, but I want the more accurate 26 from my wide range of training data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here is where I'm struggling: how do I centre the test data? To centralise the data before using PCA or the dual PCA, I deduct the mean from the data. When building my test PCA should I deduct the mean of the train data, or the mean of the test data?&lt;/p&gt;&#xA;" OwnerUserId="42058" LastEditorUserId="42058" LastEditDate="2017-11-18T14:46:03.520" LastActivityDate="2017-12-18T20:25:53.087" Title="How to test trained PCA used for compression?" Tags="|machine-learning|dimensionality-reduction|pca|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24890" PostTypeId="2" ParentId="24889" CreationDate="2017-11-18T14:28:26.820" Score="-2" Body="&lt;p&gt;You need to combine the datasets first, run the PCA and then split the datasets afterward. Once this is done, you can train your model and then test it. The PCA needs to be done on the whole dataset in order to have the same components in both sets.&lt;/p&gt;&#xA;" OwnerUserId="17959" LastActivityDate="2017-11-18T14:28:26.820" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24891" PostTypeId="2" ParentId="24889" CreationDate="2017-11-18T14:39:35.947" Score="0" Body="&lt;p&gt;When you run PCA on train data, you will get a vector space in lower dimension (After choosing the biggest components). As you know, this vector space contains the eigen vectors of the biggest eigen values. Now, using a proper function, project the test data into the vector space. So, you do not need to get PCA over test data anymore.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-18T14:39:35.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24893" PostTypeId="1" AcceptedAnswerId="24895" CreationDate="2017-11-18T16:51:37.650" Score="0" ViewCount="4386" Body="&lt;p&gt;Linear Regression is used to find the relation between dependent variable and independent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic Regression is used when dependent variable is categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this doesn't exactly suffice, what use cases you can use either of these, or can we use a combination of them?&lt;/p&gt;&#xA;" OwnerUserId="3563" LastEditorUserId="29575" LastEditDate="2017-11-18T17:14:28.277" LastActivityDate="2017-11-19T12:45:27.107" Title="When to use Linear Regression and When to use Logistic regression - use cases" Tags="|machine-learning|logistic-regression|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24894" PostTypeId="2" ParentId="24497" CreationDate="2017-11-18T17:08:22.920" Score="0" Body="&lt;p&gt;You could model the process as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Weibull_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Weibull distribution&lt;/a&gt; which is common for survival analysis and reliability engineering. There has been work on monitoring the &quot;health&quot; of systems using it, examples are &lt;a href=&quot;http://ieeexplore.ieee.org/abstract/document/5413608/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.researchgate.net/publication/220456625_Health_monitoring_of_electronic_products_based_on_Mahalanobis_distance_and_Weibull_decision_metrics&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-18T17:08:22.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24895" PostTypeId="2" ParentId="24893" CreationDate="2017-11-18T17:09:16.247" Score="2" Body="&lt;p&gt;&lt;strong&gt;Logistic Regression&lt;/strong&gt; is used when you know that the data is lineraly seperable/classifiable and the outcome is Binary or Dichotomous but it can extended when the dependent has more than 2 categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Linear Regression&lt;/strong&gt; is used to find the relation and based on the relation between them you can predict the outcome, the dependent variable should be numeric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of usecases are you expecting? give an example so that we can extend it further.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-18T17:09:16.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24896" PostTypeId="1" AcceptedAnswerId="24897" CreationDate="2017-11-18T17:14:25.377" Score="2" ViewCount="75" Body="&lt;p&gt;I have a column called &lt;strong&gt;item_colour&lt;/strong&gt; which describes the colour of products in my dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 85 colours and some of the colours are unique or they represent a small part of the data ( I have 100,000 observations). For example, there is only one &quot;almond&quot; colour or only two &quot;sky blue&quot;. I want to recombine rare values and put them together into a group called &quot;other colour&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I find the threshold to recombine them? For example, combine the colours together which have values less 50 or so. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.I am working with R&lt;/p&gt;&#xA;" OwnerUserId="41812" LastEditorUserId="40853" LastEditDate="2017-11-18T17:18:55.720" LastActivityDate="2017-11-18T18:17:26.867" Title="Recombining rare values" Tags="|r|dataset|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24897" PostTypeId="2" ParentId="24896" CreationDate="2017-11-18T17:38:07.530" Score="1" Body="&lt;p&gt;This is going to be a situation where there will be no fixed rule. One important factor is how meaningful colour differences are to the other parts of your problem. If colour has low correlation/impact in a supervised learning/prediction scenario for example, and the dataset is noisy, then you will want to merge more colours (at a higher fraction of total number) to reduce sampling bias effects that might otherwise assign importance to the colour and increase error rates in test and production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The safest approach is to treat the colour combination threshold as a hyper-parameter to the model building process, and test to see what differences it makes. If there is little or no impact to model effectiveness, then a higher threshold could be useful purely to reduce number of parameters in the model - decreasing resources used to train and run it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that seems time-consuming, then picking something by feel (e.g. your idea of picking count less than 50 in the dataset) is not usually too bad, at least to start with. You can go back and re-evaluate your choice if you have problems with the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One other possibility for feature engineering is to use the rareness of the specific colour as an additional feature. So in addition to categories for the popular colours and an &quot;other colour&quot; category, add a real value &quot;colour frequency&quot; = the observed ratio of that colour in the training set. Whether or not this is useful will depend on the problem, but it may help address some of the lost information when merging categories with a wider range of rareness values, assuming that unusual colours indicate anything at all (they may not).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-18T17:47:04.923" LastActivityDate="2017-11-18T17:47:04.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24900" PostTypeId="2" ParentId="24889" CreationDate="2017-11-18T18:35:08.143" Score="2" Body="&lt;p&gt;Found the answer in a different stackexchange question here: &lt;a href=&quot;https://stats.stackexchange.com/questions/142216/zero-centering-the-testing-set-after-pca-on-the-training-set&quot;&gt;https://stats.stackexchange.com/questions/142216/zero-centering-the-testing-set-after-pca-on-the-training-set&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answer: Yes, zero center the test data with the mean from the train data&lt;/p&gt;&#xA;" OwnerUserId="42058" LastActivityDate="2017-11-18T18:35:08.143" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24901" PostTypeId="2" ParentId="24607" CreationDate="2017-11-18T19:53:38.367" Score="0" Body="&lt;p&gt;Maybe you could try to turn this seemingly time series problem into a classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would then create a set of features, e.g:&#xA;number of days between requestDate and tripStartDeparture&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;binned departure and arrival times, e.g. 0-2, 3-5, .. 23&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;travel time  (tripDestinationDeparture - tripStartDeparture)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;day of the week, categorical features for both departure and arrival&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;categorical feature for month&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;flight carrier: maybe you can categorize them from low-cost to high-end airline&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;etc..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Actually, a bunch of new features can be created from your initial dataset if you are a little creative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then try out a series of classifiers independently then stack them to see model acccuracy you can get.&lt;/p&gt;&#xA;" OwnerUserId="2968" LastActivityDate="2017-11-18T19:53:38.367" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24902" PostTypeId="2" ParentId="24843" CreationDate="2017-11-18T21:47:45.177" Score="2" Body="&lt;p&gt;Your asking for a layman's explanation, and the phrasing of your question, suggests to me that you are getting tangled up in the semantics, and I totally get that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let's let go of the usual terms as much as we can, and speak in concepts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a bunch of observations -- or maybe even only one observation. This is your data. Maybe you are one of the particularly inquisitive ancients, and you notice some sort of relationship between lunar patterns and tides. While tracking this data, you start getting a sense for what sort of forces might be at work that would explain the apparent cause-and-effect relationship. The set of rules that you eventually develop becomes your model. The more correct your rules, the better you will be able to predict the tide tomorrow, next month, and next year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This concept you've developed -- probably something to do with gravity -- may be even more generally useful. If the general concept is re-used with different numbers, it can also help explain Earth's annual cycle. And why we see the Morning Star (Venus) with certain patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the cases where we are re-using the concept, we are &quot;fitting the model to the data&quot; -- we are testing whether the set of rules we've developed -- the model -- can explain the apparent forces at work in the observations -- the data -- that we have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the usual applications of regression (or any statistical method), we're not trying to formulate the laws of physics, we're trying to find rules that sufficiently govern complex systems -- often human behaviour -- and so we're unlikely to explain observations perfectly. But a model is useful to the extent that it successfully predicts future observations. Rather than waiting for the future, though, we typically perform analysis on part of a data set, and test for predictive power with the remainder of the data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A specific instance of regression analysis might be in trying to explain housing prices. There are many many possible factors -- square meterage (footage), number of floors, swimming pool or not, size of property, shape of property, age, likelihood of flooding, distance to neighbours, energy rating, economic conditions, quality of the local school, etc, etc, etc. Which factors are most important? Do one, two, three of the factors explain 95% of the variation in housing prices? Can we come up with an equation that will predict a house's selling price reasonably well, rather than relying on a real estate agent's recommendation -- and having to second-guess what his/her agenda is -- how they might be playing us?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if you will humour a little editorializing:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are countless examples of reliance on experts proving to be a mistake (Nate Silver's &quot;The Signal and the Noise&quot;, Daniel Kahneman's &quot;Thinking Fast, Thinking Slow&quot;, Philip Tetlock's &quot;Superforecasting&quot;) -- Data Science is about finding ways to derive insight from the data, rather than experts' opinions. But many data scientists get caught up in the mechanics, and lose sight of the purpose. It's not just about razzle-dazzle infographics. As Hamming puts it: &quot;the purpose of computing is insight, not numbers&quot;. So I applaud your effort to seek a layman's explanation, rather than technical speak, to understand regression.&lt;/p&gt;&#xA;" OwnerUserId="42074" LastActivityDate="2017-11-18T21:47:45.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24905" PostTypeId="1" CreationDate="2017-11-18T22:59:13.807" Score="2" ViewCount="1391" Body="&lt;p&gt;I have a data set where I need to detect fraud. 99% are not fraud and 1% are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What methods can be used to solve problems where classes are imbalanced?&lt;/p&gt;&#xA;" OwnerUserId="39331" LastEditorUserId="29575" LastEditDate="2017-11-20T01:39:56.090" LastActivityDate="2019-02-27T12:48:34.503" Title="Best methods to solve class imbalance problem and why?" Tags="|machine-learning|python|statistics|class-imbalance|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24906" PostTypeId="1" CreationDate="2017-11-18T23:00:00.840" Score="-2" ViewCount="374" Body="&lt;p&gt;I am trying to figure out what would be the best way to learn patterns with a data-set that has temporal dependency between its features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I want to predict whether a patient will suffer from a heart attack in the following minute by looking at his/her heart-rate, blood pressure and oxygen levels on each minute for the past five minutes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now each combination of the three features stated above has no predictive value on its own, but I assume that a temporal analysis might reveal some interesting predictive ability. For example, having a slow heart rate two minutes after heaving fast heart rate and low oxygen levels might point to an increase possibility of a heart attack. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best approach to tackle this kind of problem? Would a regular ML classification work in this case if I use each combination of index*time as a feature? (resulting in 3(index)*5(time-points) features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions/reading materials would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="42076" LastActivityDate="2017-11-19T03:44:44.433" Title="Best ML practice for temporal dependency between features" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24907" PostTypeId="1" AcceptedAnswerId="24919" CreationDate="2017-11-18T23:17:06.520" Score="0" ViewCount="3012" Body="&lt;p&gt;Gradient descent is one of the well-known optimisation algorithms. However, are the regression algorithms in scikit-learn implemented with gradient descent or some other techniques?&lt;/p&gt;&#xA;" OwnerUserId="32230" LastActivityDate="2017-11-19T15:51:05.490" Title="What optimization algorithms are used in scikit-learn?" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24908" PostTypeId="2" ParentId="24905" CreationDate="2017-11-19T02:53:43.583" Score="1" Body="&lt;p&gt;There are several techniques&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random up-sampling&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Disadvantage-&gt;create duplicated and/or artificial instances which may      introduce bias and /or noise to the original data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Random Down-Sampling&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Disadvantage -&gt; not all data points are used. Potentially removing useful information. Better choice for data with very high class imbalance.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Repeated Sampling&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;in this process, you identify few definite negatives from your data, along with definite positives.Train your model and classify all your sample data based upon this model. &lt;/li&gt;&#xA;&lt;li&gt;Repeat this process, and perform voting at the end to get the class labels for highly imbalanced datasets.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="40750" LastActivityDate="2017-11-19T02:53:43.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24909" PostTypeId="1" AcceptedAnswerId="25114" CreationDate="2017-11-19T03:08:02.867" Score="6" ViewCount="6725" Body="&lt;p&gt;In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UX1FN.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UX1FN.jpg&quot; alt=&quot;pic&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the picture above only has one unit, so I am wondering if it has more than one unit, for example memory unit=2, what will this model look like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SJLVD.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SJLVD.jpg&quot; alt=&quot;pic&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rnFCb.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rnFCb.jpg&quot; alt=&quot;pic&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or can someone point out the wrong part, or give a sample of visualize architecture of LSTM model with multiple units, thanks!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;perhaps my description was not clear enough, so i try to give a more clear description and graph of the question:&#xA;I try to illustrate the question as the picture below, and my question is: is the architect correct when LSTM model has 2_units,2_layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3Lh2p.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3Lh2p.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41629" LastEditorUserId="41629" LastEditDate="2017-11-21T03:38:46.470" LastActivityDate="2017-11-27T16:26:05.397" Title="The model of LSTM with more than one unit" Tags="|deep-learning|tensorflow|keras|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24911" PostTypeId="2" ParentId="24906" CreationDate="2017-11-19T03:44:44.433" Score="2" Body="&lt;p&gt;You might want to create new features from your data for different time instances.&#xA;eq, heart rate at hour 1, heart rate at hour 2 ...&#xA;These features once fed into your algorithm should answer your question.&lt;/p&gt;&#xA;" OwnerUserId="40750" LastActivityDate="2017-11-19T03:44:44.433" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24912" PostTypeId="2" ParentId="24905" CreationDate="2017-11-19T03:48:51.557" Score="2" Body="&lt;p&gt;There are 2 Techniques:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Oversampling&lt;/strong&gt;: There are many techniques under this, &lt;strong&gt;ROSE&lt;/strong&gt; and &lt;strong&gt;SMOTE&lt;/strong&gt; are the most famous techniques used for oversampling. In ROSE it just increases the minority classes. In SMOTE it synthetically generates more number of rare minority classes for balancing. Most of the Scenarios SMOTE gives better results than ROSE but you should try both. Other than that there is just another techniques which just duplicates the records to the make it equal n number. This &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;, is for implemenation of SMOTE in Python.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;UnderSampling&lt;/strong&gt;: There are many techniques under this too, but this &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/applications/plot_multi_class_under_sampling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;, &lt;a href=&quot;https://datascience.stackexchange.com/questions/11404/python-handling-imbalance-classes-in-python-machine-learning&quot;&gt;Link-2&lt;/a&gt; gives you better idea about undersampling. Generally I don't prefer Undersampling, as you would loose some infomation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The reason why you need to use these techniques are, if we don't use then the model accuracy will be very high it will be able to predict with 99% accuracy correctly the cases which are not fraud, which we don't want. if it can predict with same accuracy to find out fraud, then that is a great insight.&#xA;This can be achived only by using either of the above techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do have a look and Let me know if you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-19T04:24:49.437" LastActivityDate="2017-11-19T04:24:49.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24913" PostTypeId="2" ParentId="24867" CreationDate="2017-11-19T03:52:04.507" Score="2" Body="&lt;p&gt;Accuracy is not a function of number of features. You might have heard about feature selection processes, and that is the reason we use feature selection or dimensionality reduction.&#xA;There is huge theory behind this, but in very generic terms, we need to find out the best features, to make our model least complex, still keeping all relevant features in the model.&#xA;One example, can be adding a new feature reduces accuracy, which is highly biased towards one out come and hence it creates inaccuracy in your results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may also happen, if all your features are not normalised, and hence the newly added features is biasing your results and hence introducing inaccuracy.&lt;/p&gt;&#xA;" OwnerUserId="40750" LastActivityDate="2017-11-19T03:52:04.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24915" PostTypeId="1" CreationDate="2017-11-19T06:05:42.633" Score="-2" ViewCount="59" Body="&lt;p&gt;Let's say that I want to generate a cryptographic key based on my hand (I'm using hands just as an example). What I could do is I scan my hand (so that the friction ridges are apparent) and then use a key derivation algorithm. Problem is that if I want to get the same hash, I have to either store the scan somewhere, or be able to produce the &lt;em&gt;exact same scan&lt;/em&gt;, which is not very feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What if instead, we use a machine learning algorithm that takes a scan of a human hand and produces some data, subject to the following constraints:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the same human hand is scanned twice, produce the same data&lt;/li&gt;&#xA;&lt;li&gt;If different human hands are scanned, produce different data&lt;/li&gt;&#xA;&lt;li&gt;Additionally, it should be hard to predict valid outputs of the neural net, other than brute force (so an adversary would either need to be able to generate human hand scan images, or just feed in random data).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;One way I thought of doing this was to scan a bunch of hands multiple times, and then train a neural network in such a way that it is punished if it gives different values for hands from the same person, or if two humans get the same hand. Also, you could have another neural network trying to predict whether something is an output of the first one, and the first network is punished if the second one can do so well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, as this problem been studied before (not necessarily with hands, but producing the same output if two inputs satisfy some equivalence relation).&lt;/p&gt;&#xA;" OwnerUserId="42081" LastActivityDate="2017-11-19T06:24:06.257" Title="Can you use machine learning to extract the entropy from a hand print reliably?" Tags="|machine-learning|neural-network|reference-request|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24917" PostTypeId="2" ParentId="24915" CreationDate="2017-11-19T06:24:06.257" Score="0" Body="&lt;p&gt;You are describing a classification problem, where each 'class' is a person. You are trying to learn the hand-&gt;person mapping.  Having two inputs map to the same output if they are similar enough is exactly what it means to 'classify' an input.  The way you're framing your question matches it perfectly with a basic task of machine learning, multi-class classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea if hand recognition software actually works this way - it would need to learn your hand from a single example or a very few, so I expect there is some &lt;a href=&quot;https://en.wikipedia.org/wiki/One-shot_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;1-shot learning&lt;/a&gt; sorcery in the mix when used 'in the wild'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: The adversarial part is particularly interesting. There are techniques for training neural networks to be robust to attacks, for example you might read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_adversarial_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Networks&lt;/a&gt;, where a second network feeds inputs into the first to try to trick it. &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-19T06:24:06.257" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24918" PostTypeId="2" ParentId="24893" CreationDate="2017-11-19T12:45:27.107" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;Linear Regression,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;no class definition, the response variable is a continues value. Model the relationship between explanatory variables and the response variable.&#xA;Suppose you want to estimate the house price based upon the area of the land, you can use linear regression. It is an equation of a straight line, where your response can take any value between -Infinity to + infinity.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Logistic Regression,&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now suppose you want to fit a model for two class problem like ham or spam,&#xA;or malign or benign tumour(ie response variable can only take 0 or 1). here, you got only two values for the response variable. Now, if you want to fit this with a linear model (which as we know can take any value between - inf to + inf) how would be interpret response value between 0 and 1 and how would you interpret values more than 1 an double than 1 and ten times of 1...(you are getting my point?)&#xA;hence, in such cases, we tend to use Logit function, which can only take two values.&#xA;In this case, we have to decide about the threshold for the decision boundary but once done, you would know your response values as 0 r 1.&lt;/p&gt;&#xA;" OwnerUserId="40750" LastActivityDate="2017-11-19T12:45:27.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24919" PostTypeId="2" ParentId="24907" CreationDate="2017-11-19T13:36:47.857" Score="5" Body="&lt;p&gt;There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients&lt;/li&gt;&#xA;&lt;li&gt;Lasso and ElasticNet use &lt;a href=&quot;https://en.wikipedia.org/wiki/Coordinate_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;coordinate descent&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients&lt;/li&gt;&#xA;&lt;li&gt;ARDRegression and BayesianRidge use something like EM algorithm&lt;/li&gt;&#xA;&lt;li&gt;SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.&lt;/li&gt;&#xA;&lt;li&gt;HuberRegressor uses BFGS (a second-order optimization method)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-11-19T15:51:05.490" LastActivityDate="2017-11-19T15:51:05.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24921" PostTypeId="1" AcceptedAnswerId="24924" CreationDate="2017-11-19T15:25:01.673" Score="7" ViewCount="4093" Body="&lt;p&gt;I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, &quot;replay&quot; them through our neural network. This makes sense because we are trying to build out our rewards matrix and see if our episode ended in reward, scale that back through our matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think the sequence of actions that led to the reward state is what is important to capture - this sequence of actions (and not the actions independently) are what led us to our reward state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf&quot; rel=&quot;noreferrer&quot;&gt;Atari-DQN paper by Mnih&lt;/a&gt; and many tutorials since we see the practice of random sampling from the memory array and training. So if we have a memory of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(action\,a, state\,1) \rightarrow (action\,b, state\,2) \rightarrow (action\,c, state\,3) \rightarrow (action\,d, state\,4) \rightarrow reward!$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We may train a mini-batch of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[(action c state 3), (action b, state 2), reward!]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason given is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Second, learning directly from consecutive samples is inefficient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or from this pytorch tutorial:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;By sampling from it randomly, the transitions that build up a batch are decorrelated. It has been shown that this greatly stabilizes and improves the DQN training procedure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My intuition would tell me the sequence is what is most important in reinforcement learning. Most episodes have a delayed reward so most action/states do not have a reward (and are not &quot;reinforced&quot;). The only way to bring a portion of the reward to these previous states is to retroactively break the reward out across the sequence (through the &lt;code&gt;future_reward&lt;/code&gt; in the Q algorithm of &lt;code&gt;reward + reward * learning_rate(future_reward))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A random sampling of the memory bank breaks our sequence, how does that help when you are trying to back-fill a Q (reward) matrix?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps this is more similar to a Markov model where every state should be considered independent? Where is the error in my intuition?&lt;/p&gt;&#xA;" OwnerUserId="42094" LastEditorUserId="29575" LastEditDate="2017-11-20T01:39:34.693" LastActivityDate="2022-11-24T15:18:10.403" Title="Why random sample from replay for DQN?" Tags="|neural-network|deep-learning|reinforcement-learning|q-learning|dqn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24923" PostTypeId="1" AcceptedAnswerId="24933" CreationDate="2017-11-19T16:37:13.063" Score="-1" ViewCount="457" Body="&lt;p&gt;I have 90000 records for customer segmentation consisting of numeric nominal and ordinal variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*houselhold_ID* ID for household&#xA;&#xA;*quantity*: integer, number of products bought&#xA;&#xA;*price*: values 1 to 3 with 1 for 0-5, 2 for 5-8 and 3 for 8-10&#xA;&#xA;*product name*: nominal values 1 to 9 representing 9 different products like different kind of cheese...&#xA;&#xA;*duration*: days from the current buy to the next buy.&#xA;&#xA;*age*: catgeorical age groups: 1: 0-24 years, 2: 25-39 years, 3: 40-59 etc.&#xA;&#xA;*salary*: 1: to 1499 2 : 1500 bis 2499 3 : 2500 bis 3499 4 : 3500 or more &#xA;&#xA;*number_of_persons_household*: integer &#xA;&#xA;*price_con*: price conscious: 1 to 4 for not price conscious &#xA;to very price conscious&#xA;&#xA;*education*: 1 to 3 for lower education to high school/university.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I already did outliers detection and removed them having small frequencies. Most household have 40 buys.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to cluster analysis in R for to find patterns in this data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I currently run kmodes and and want to test kproto too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is there any possibility to things do like Hopkin statistics for clusterness like with numeric values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there also a kind of visualization if I have more than 2 classes? So that one can visualize the data points in to groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe someone has experience with such kind of data. I am open for hints and tips to analyse this data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the elbow plot for kmodes with 15 iterations and 20000 records from the full 90000:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oRzDt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oRzDt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So 2 or 3 clusters are good?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did now a Multi Factor Analysis in R. Additionally I did some engineering for the duration variable. I categorized the variable in three groups splitted according to the mean and quantiles in low, mid, high duration.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(&quot;FactoMineR&quot;)&#xA;library(&quot;factoextra&quot;)&#xA;    res.mfa &amp;lt;- MFA(dat[1:20000,], &#xA;                   group = c(1, 1, 1, 1,1,1,1,1,1), &#xA;                   type = c(&quot;c&quot;, &quot;n&quot;, &quot;n&quot;, &quot;n&quot;, &quot;n&quot;,&quot;c&quot;,&quot;n&quot;,&quot;n&quot;,&quot;n&quot;),&#xA;                   name.group = c(&quot;quantity&quot;,&quot;price&quot;,&quot;product&quot;,&#xA;                                  &quot;age&quot;,&quot;salary&quot;,&quot;persn&quot;,&quot;price_con&quot;,&quot;edu&quot;, &quot;duration&quot;),&#xA;&#xA;                   graph = FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used a group for each variable.&#xA;The result seem poor. Only 30% of variance can be explained with 6 dimensions?! So bad? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RbHal.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RbHal.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any Ideas?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastEditorUserId="32963" LastEditDate="2017-11-19T21:15:26.840" LastActivityDate="2017-11-20T14:31:40.850" Title="Clustering mixed data" Tags="|machine-learning|r|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24924" PostTypeId="2" ParentId="24921" CreationDate="2017-11-19T17:23:10.477" Score="6" Body="&lt;p&gt;The de-correlation effect is more important than following sequence of trajectories in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Single step Q-learning does not rely on trajectories to learn. It is slightly less efficient to do this in TD learning - a Q($\lambda$) algorithm which averages over multiple trajectory lengths would maybe work better if it were not for the instability of using function approximators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, the DQN-based learning bootstraps across single steps (State, Action, Reward, Next State). It doesn't need longer trajectories. And in fact due to bias caused by correlation, the neural network might suffer for it if you tried. Even with experience replay, the bootstrapping - using one set of estimates to refine another - can be unstable. So other stabilising influences are beneficial too, such as using a frozen copy of the network to estimate the TD target $R + \text{max}_{a'} Q(S', a')$ - sometimes written $R + \text{max}_{a'} \hat{q}(S', a', \theta^{\bar{ }})$ where $\theta$ are the learnable parameters for $\hat{q}$ function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might still be possible to use longer trajectories, sampled randomly, to get a TD target estimate based on more steps. This can be beneficial for reducing bias from bootstrapping, at the expense of adding variance due to sampling from larger space of possible trajectories (and &quot;losing&quot; parts of trajectories or altering predicted reward because of exploratory actions). However, the single-step method presented by DQN has shown success, and it is not clear which problems would benefit from longer trajectories. You might like to experiment with options though . . . it is not an open-and-shut case, and since the DQN paper, various other refinements have been published.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-19T19:05:40.697" LastActivityDate="2017-11-19T19:05:40.697" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="24925" PostTypeId="1" CreationDate="2017-11-19T18:42:22.913" Score="-2" ViewCount="47" Body="&lt;p&gt;Is it possible for a supervised random forest to obtain the cluster centers like in kmeans? I mean, if I train the Rf, I want the characteristics for every cluster ie. the mean values for every feature specifing each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.G: I use a random forest to rtain which features are important (and what mean values they have) to buy a specific product (the class lables)&lt;/p&gt;&#xA;" OwnerUserId="32963" LastActivityDate="2017-12-03T10:23:27.963" Title="Random Forest in R getting classification rules/centers" Tags="|r|clustering|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24928" PostTypeId="1" AcceptedAnswerId="24930" CreationDate="2017-11-19T20:06:24.170" Score="2" ViewCount="1799" Body="&lt;p&gt;I'm now learning YOLO but I don't understand how the number of grid cells is determined. Some article &lt;a href=&quot;http://machinethink.net/blog/object-detection-with-yolo/&quot; rel=&quot;nofollow noreferrer&quot;&gt;like this one&lt;/a&gt; uses 13 x 13 while others use 19 x 19 (e.g. Andrew Ng's course in Coursera). It seems that the height and width should always be the same, but how is it determined?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any general guideline regarding how many grid cells should be picked up over the others?&lt;/p&gt;&#xA;" OwnerUserId="8432" LastActivityDate="2017-11-19T20:51:31.670" Title="How is the number of grid cells in YOLO determined?" Tags="|neural-network|yolo|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24929" PostTypeId="1" AcceptedAnswerId="24947" CreationDate="2017-11-19T20:21:24.917" Score="0" ViewCount="280" Body="&lt;p&gt;KDD and CRISP-DM are both processes to structure your Data Mining procedure. Is data labeling not also a important part of Data Mining? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data labeling is for example in unsupervised learning the target of the Data Mining process. So if I want to classify a data set that was labelled by me before, do I just do the process twice? In my opinion sometimes the labeling is quite trivial, so that doing the process twice would be quite unnecessary? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to include the labeling into the data exploring or preprocessing phase? E.g. in CRISP-DM Preprocessing there is something like generating a new parameter. Can this Parameter be also a new target/label? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this question is quite process orientated and in Data Mining you are quite free but just assume that in this case you have to follow the process. &lt;/p&gt;&#xA;" OwnerUserId="37808" LastActivityDate="2017-11-20T09:26:13.063" Title="Which process step in KDD or CRISP-DM includes labeling of the data?" Tags="|data-mining|labels|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24930" PostTypeId="2" ParentId="24928" CreationDate="2017-11-19T20:51:31.670" Score="1" Body="&lt;p&gt;As far as I can tell, there is no specific rule. It will depend in part on how crowded your scene will become with items that you want to detect and locate separately. Creating a high granularity grid increases computational cost for training, and there is no reason to do so if it would only cover additional cases that are much rarer than the detection accuracy that the base algorithm achieves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice can be driven by the ground truth data. The ground truth for YOLO needs to be expressed in the form of grid locations with classes and bounding rectangle sizes. If you don't find any training examples where you want to label two items with their centre inside the same grid square, then this is a good indication that your grid size is fine-grained enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if there are one or two examples with a clash like this, you may be able justify labelling just one item in the ground truth and be OK with a resulting model that is not able to cope with close overlap between two separate objects. And even with smaller grid squares, YOLO may not be able to learn to separate the objects, if such an overlap only occurs rarely. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect a simple rule of diminishing returns applies. As datasets grow larger, and object detection can be trained on more powerful computers, we may see state of the art models still using YOLO but with more grid points.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-11-19T20:51:31.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24933" PostTypeId="2" ParentId="24923" CreationDate="2017-11-19T23:14:13.383" Score="1" Body="&lt;p&gt;Try HAC with Gower's similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a very heuristic approach - there is nothing going to save you from weighting variables - but worth a try.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-19T23:14:13.383" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="24935" PostTypeId="2" ParentId="24380" CreationDate="2017-11-20T01:38:23.533" Score="-2" Body="&lt;p&gt;You don't have to worry about it, as we know, in k-means clustering, you only have to choose the initial centroids. This would create first iteration of clusters. In the next iteration, the centroids would move to the center of the newly created clusters. this whole process will continue till you get convergence.&#xA;Now, the number of centroids selected at the first place are good or not, is a separate question.&#xA;Also, we also know, the result of a k-means is dependent upon the initial selected centroids and we have various methods of selecting intital centroids, but just answer your question here, run the full algorithm and you wont have such lone centroids at the end of the algo. &lt;/p&gt;&#xA;" OwnerUserId="40750" LastEditorUserId="40750" LastEditDate="2019-07-03T03:56:21.567" LastActivityDate="2019-07-03T03:56:21.567" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="24936" PostTypeId="2" ParentId="24854" CreationDate="2017-11-20T02:30:22.247" Score="1" Body="&lt;p&gt;To get an intuitive contrast, I came up with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer to &#xA;$$R^2=1-\frac{\sum ( y_i - \hat y_i)^2} {\sum (y_i - \bar y)^2}$$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My method is:&#xA;$$My Value=\frac {\sum |y_i - \bar y|} {\sum | y_i - \hat y_i|} $$&#xA;Indicating how many times the model better than other model, always predicts mean value.&lt;/p&gt;&#xA;" OwnerUserId="39820" LastEditorUserId="40853" LastEditDate="2017-11-20T02:57:31.743" LastActivityDate="2017-11-20T02:57:31.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24937" PostTypeId="1" CreationDate="2017-11-20T02:53:57.960" Score="1" ViewCount="3907" Body="&lt;p&gt;I have data census(name, sex, age, capital_gain) and I want to plot all possible views (in a histogram) and find the most interesting view based on the distinction to other views. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have to compute the similarity between two histogram plots, for example:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Average Capital Gain vs Sex (which computed by aggregate Capital gain to Sex)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rSArD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rSArD.png&quot; alt=&quot;Capital gain vs Sex&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Average age vs Sex (which computed by aggregate average age to sex)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uhCrx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uhCrx.png&quot; alt=&quot;Age vs Sex&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions is, what kind of measurement that can be used to check the similarity between those two histogram plots? I consider the features that want to be compared are (x-axis, y-axis, and pattern of the histogram)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really not sure about distance function that can be used for this case. I know Jaccard distance, but not sure it can be used in this situation or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you, &lt;/p&gt;&#xA;" OwnerUserId="42111" LastActivityDate="2017-11-20T14:18:13.003" Title="Finding similarity between two histogram plots" Tags="|python|scikit-learn|similarity|distance|plotting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24938" PostTypeId="2" ParentId="24937" CreationDate="2017-11-20T03:02:45.800" Score="2" Body="&lt;p&gt;I'm curious what other people will say, but one option is to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Definition&quot; rel=&quot;nofollow noreferrer&quot;&gt;KL-divergence&lt;/a&gt;.  If your two histograms have the same x-axis, you can divide every column by the total count to convert counts to proportions. Then you could treat the histograms as probability mass, and use the KL-divergence.  KL-divergence is really a measure of the distance between two probability distributions, but histograms are an approximation of a discrete distribution. You'd pick one of your distributions to be P and one to be Q, then calculate $-\sum_x P(X)\log{\frac{P(X)}{Q(X)}}$ where $x$ is each bin on the x-axis of the histogram.  Be aware that this distance metric is &lt;em&gt;directional&lt;/em&gt;, meaning you'll get a different answer if you swap P and Q.  The reason is that the KL-divergence is a measure of information transfer (like, the information you gain by moving from Q to P), but I think that's irrelevant to your question.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-20T03:02:45.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24939" PostTypeId="1" AcceptedAnswerId="25045" CreationDate="2017-11-20T05:18:01.260" Score="0" ViewCount="343" Body="&lt;p&gt;I want to try the IBM speech to text API. I created an IBM cloud account and went to &lt;a href=&quot;https://console.bluemix.net/catalog/services/speech-to-text&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://console.bluemix.net/catalog/services/speech-to-text&lt;/a&gt;. I see the error message:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Please create a space in the selected org or choose another org.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where can I create such a space? I wasn't able to find it yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tEIls.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tEIls.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2017-11-23T09:29:20.513" Title="How can I create a space in IBM Cloud?" Tags="|ibm-watson|speech-to-text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24941" PostTypeId="1" CreationDate="2017-11-20T06:01:04.567" Score="2" ViewCount="57" Body="&lt;p&gt;I'm trying to create an RNN with a one character memory. Here is my model base on an M-GRU (Minimal Gated Recurrent Unit) :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f_t = \sigma (W_f[h_t,x_t]+b_f)$$&#xA;$$\tilde{h_t} = \tanh(W_h[h_t,x_t]+b_h)$$&#xA;$$h_{t+1} = (1-f_t)*\tilde{h_t} + f_t*h_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x_t$ is the input and $h_t$ is the hidden state of the RNN. $f$ is the update gate for the state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I have the first and second elements of the vector $h_t$ have a special meaning to be the strength of setting or getting the memory. $s=h_t[0]$, $g=h_t[1]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This adds the input to the memory with strength $s$ and then softmaxed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_{t+1} = softmax( m_{t} + s x_t )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the candidate output is (it seems to generalise less well without the tanh but converges nicer):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$c = tanh(W_c h_{t+1}+b_c) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the real output is given by adding the memory with strength g:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$o_{t+1} = softmax( c + g m_{t+1} )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then it passes $h_{t+1}$ and $m_{t+1}$ onto the next time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I tried it on some data &quot;21(22)52(33)5421(44)32112(99)1232(33)132...&quot;&#xA;where basically it is random numbers except inside the brackets.&#xA;Then I try give it an starting string &quot;(x&quot; which if it successfully works it should output &quot;x)&quot; even though it never saw the character &quot;x&quot; before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It almost works but is not really converging well.  Here's a typical output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;(xx)7456725(11)9756x89994x548(33)383351051)(44)9887))1039(11))5407(77)953201(99)x1x85)53)(99))3846&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better model you know of that does this better (but simpler than a Neural Turing Machine)?&lt;/p&gt;&#xA;" OwnerUserId="41463" LastEditorUserId="41463" LastEditDate="2017-11-23T22:47:15.123" LastActivityDate="2017-11-23T22:47:15.123" Title="Trying to create an RNN with one character memory" Tags="|neural-network|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24942" PostTypeId="2" ParentId="11404" CreationDate="2017-11-20T06:18:34.493" Score="0" Body="&lt;p&gt;There are already some good answers here. I just thought I would add one more technique since you look to be using ensembles of trees. In many cases you are looking to optimize the Lift curve or the AUC for the ROC. For this I would recommend &lt;a href=&quot;https://www3.nd.edu/~nchawla/papers/DMKD11.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hellinger distance criterion&lt;/a&gt; for splitting the branches in your trees. At the time of writing this it is not in the &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;imbalanced-learn&lt;/a&gt; package but it looks like there is a &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/9947&quot; rel=&quot;nofollow noreferrer&quot;&gt;plan&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2017-11-20T06:18:34.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24943" PostTypeId="2" ParentId="11314" CreationDate="2017-11-20T07:34:23.700" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://conservancy.umn.edu/bitstream/handle/11299/92985/?sequence=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a pretty extensive guide on change/anomaly detection in time series: &quot;Anomaly Detection of Time Series&quot;, a masters thesis by Deepthi Cheboli.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can start from the simple parametric approaches that have been mentioned before, but I want to let you know that there is an actual methodology devoted to this exact problem, which is very easy to build and use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The name of your problem is &lt;strong&gt;novelty detection&lt;/strong&gt;, and the perfect tool to handle it without any burn-in data, is called &lt;strong&gt;One-Class Support Vector Machines&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What it does is it learns from the single class, and checks any test cases against the learned single group to see if it is different than what it is used to seeing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess this is what you are trying to do. You are avoiding sacrificing any time or data points after the change happens. If you were fine with burning in some data after the change, and build a &quot;classification&quot; (with an abuse of the term) approach, then you didn't have to go the one-class SVM way. If you believe the time-series that you currently have has a good behavior, and anything that doesn't look like it should be immediately detected, then one-class SVM it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This model is implemented in R, Python, Matlab, and I am sure many other platforms, so depending on which program you use, I'd recommend giving it a try. &lt;/p&gt;&#xA;" OwnerUserId="20572" LastEditorUserId="20572" LastEditDate="2017-11-20T15:49:22.413" LastActivityDate="2017-11-20T15:49:22.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24944" PostTypeId="1" CreationDate="2017-11-20T07:35:07.007" Score="-1" ViewCount="314" Body="&lt;p&gt;What are the drawbacks of V-measure clustering evaluation method? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For evaluating what clustering algorithms, is the V-measure evaluation method suitable?&lt;/p&gt;&#xA;" OwnerUserId="42121" LastActivityDate="2017-11-20T20:55:09.137" Title="What are the drawbacks of V-measure clustering evaluation method?" Tags="|machine-learning|clustering|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24945" PostTypeId="1" CreationDate="2017-11-20T07:46:50.167" Score="1" ViewCount="664" Body="&lt;p&gt;I'm new to orange and struggling to find p-values, using different regression tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems the outputs are only coefficients. What am I missing?&lt;/p&gt;&#xA;" OwnerUserId="42123" LastEditorUserId="40853" LastEditDate="2017-11-20T16:09:24.797" LastActivityDate="2018-01-22T12:07:16.177" Title="P-values in Orange" Tags="|regression|orange|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24947" PostTypeId="2" ParentId="24929" CreationDate="2017-11-20T09:26:13.063" Score="0" Body="&lt;p&gt;&lt;strong&gt;Data Labeling&lt;/strong&gt; is a very trivial process as you have mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, it falls under &lt;strong&gt;Data Understanding&lt;/strong&gt;(Exploration Analysis). When you don't know anything about data then you do exploratory analysis to understand and derive some insights. If you don't know the target variable then the problem falls under &lt;strong&gt;Unsupervised Learning&lt;/strong&gt;, as you have mentioned in the question that your problem is unsupervised. So, you don't know your Target Variable, you are trying to make new feature/dimension to get some good insight irrespective of the factor whether you derive your Target Variable or any other feature it falls under &lt;strong&gt;Data Preparation&lt;/strong&gt;(any new derived variables), which ever we think are important for our Analysis&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-20T09:26:13.063" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24949" PostTypeId="1" AcceptedAnswerId="24981" CreationDate="2017-11-20T10:27:04.330" Score="-1" ViewCount="7906" Body="&lt;pre&gt;&lt;code&gt;f1 &amp;lt;- function(a,b,c,d,e,f) { &#xA;&#xA;  -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = matrix(c(1:66), byrow = T, ncol = 6)&#xA;f1(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;An error of the following occurs:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in f1(x) : argument &quot;b&quot; is missing, with no default&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29967" LastEditorUserId="2961" LastEditDate="2017-11-20T16:09:29.487" LastActivityDate="2017-11-21T10:41:32.103" Title="Error in f1(x) : argument &quot;b&quot; is missing, with no default" Tags="|r|data-mining|dataset|data|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24950" PostTypeId="1" AcceptedAnswerId="24964" CreationDate="2017-11-20T11:36:38.240" Score="0" ViewCount="586" Body="&lt;p&gt;There are several &quot;classical&quot; ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These metrics however do not take &quot;costs&quot; into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="42128" LastEditorUserId="41886" LastEditDate="2017-11-20T18:22:01.907" LastActivityDate="2017-11-20T18:22:01.907" Title="Cost Function for evaluating a Regression Model" Tags="|machine-learning|regression|loss-function|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24951" PostTypeId="2" ParentId="17191" CreationDate="2017-11-20T11:56:03.317" Score="1" Body="&lt;p&gt;This was due to the fact that I should normalize my data before passing it into the algorithm. For example,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = StandardScaler()&#xA;scaler.fit(X_train)&#xA;X_train = scaler.transform(X_train)&#xA;X_test = scaler.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or, the &quot;normalize&quot; parameter of Lasso should be set as True. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Lasso(alpha=float(alpha), normalize=True, random_state=42)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2017-11-20T14:50:04.617" LastActivityDate="2017-11-20T14:50:04.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24952" PostTypeId="1" CreationDate="2017-11-20T12:01:11.113" Score="-1" ViewCount="118" Body="&lt;p&gt;I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I test based on the sample size whether this split is no longer random? Is it a test of proportions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if there is a flaw in the randomisation in this instance, does that invalidate the results of the test?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any and all help!&lt;/p&gt;&#xA;" OwnerUserId="42133" LastActivityDate="2017-11-20T21:44:30.640" Title="Testing whether observed control/test split invalidates my assumption of 50/50 randomised traffic" Tags="|ab-test|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24955" PostTypeId="2" ParentId="24937" CreationDate="2017-11-20T13:05:13.107" Score="0" Body="&lt;p&gt;Those plots you're showing are not histograms, they're just bar charts. In a histogram, the height of the bars is a count of how many data points fall in the bucket represented by that value on the X axis (e.g. how many people of that particular age or age group).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could do is plot a histogram of the &lt;em&gt;men's&lt;/em&gt; ages (how many men of each age or age group) and compare it with a histogram of the &lt;em&gt;women's&lt;/em&gt; ages. It's not clear what it would mean to compare a histogram of the men's ages with, say, a histogram of their wealth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the similarity between two histograms (or distributions) of the same variable can be done by adapting Jaccard similarity (see &lt;a href=&quot;https://pdfs.semanticscholar.org/9ce4/16e3eb2fcffa40b226fece01437c61afba84.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; for an example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find a high level of similarity (say 0.9) for age distribution but a low similarity for wealth. The interesting ones are perhaps those where the histograms are dissimilar (things are very different for men and women).&lt;/p&gt;&#xA;" OwnerUserId="9114" LastEditorUserId="9114" LastEditDate="2017-11-20T14:18:13.003" LastActivityDate="2017-11-20T14:18:13.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24956" PostTypeId="2" ParentId="24719" CreationDate="2017-11-20T13:10:51.993" Score="1" Body="&lt;p&gt;I'd merely add a comment if I could ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is your interpretation of p(y|x)? The usual expression is &quot;the probability of y given x&quot;, i.e. the dependent variable corresponding to the independent variable in the probability distribution. This is the function you want to maximize, not the inverse function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side note, there are two ways that the entropy form of the equation on the right can be used. This can be added to a cost function as a regularization kernel, with a modulating coefficient; or it can be the primary term in the cost function -- the entropy to be maximized -- with constraints from the physical system introduced with Lagrange multipliers.&lt;/p&gt;&#xA;" OwnerUserId="42074" LastActivityDate="2017-11-20T13:10:51.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24957" PostTypeId="2" ParentId="8869" CreationDate="2017-11-20T13:12:10.947" Score="1" Body="&lt;p&gt;From your clarification -&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;By database, lets just say that there is a huge list of the ngram&#xA;  model that represents the document&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You would do well to do something a bit more structured and put the data into a relational database. This would allow you to do much more detailed analysis more easily and quickly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess when you say &quot;ngram&quot; you mean &quot;1gram&quot;. You could extend the analysis to include 2grams, 3grams etc, if you wanted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would have a table structure that looks something like this -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1Grams&lt;/strong&gt;&lt;br&gt;&#xA;ID&lt;br&gt;&#xA;Value   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Docs&lt;/strong&gt;&lt;br&gt;&#xA;ID&lt;br&gt;&#xA;DocTitle&lt;br&gt;&#xA;DocAuthor&lt;br&gt;&#xA;etc.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Docs1Grams&lt;/strong&gt;&lt;br&gt;&#xA;1GramID&lt;br&gt;&#xA;DocID&lt;br&gt;&#xA;1GramCount   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in the record in the &lt;strong&gt;Docs1Grams&lt;/strong&gt; table when 1GramID points to the 1gram &quot;the&quot; and the DocID points to the document &quot;War and Peace&quot; then 1GramCount will hold the number of times the 1gram &quot;the&quot; appears in War and Peace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the DocID for 'War and Peace&quot; is 1 and the DocId for &quot;Lord of the Rings&quot; is 2 then to calculate the 1gram similarity score for these two documents you would this query -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Select count(*) from Docs1Grams D1, Docs1Grams D2   &#xA;where D1.DocID = 1 and   &#xA;D2.DocID = 2 and   &#xA;D1.1GramID = D2.1GramID and   &#xA;D1.1GramCount &amp;gt; 0 and   &#xA;D2.1GramCount &amp;gt; 0   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By generalizing and expanding the query this could be easily changed to automatically pick the highest such score / count comparing your chosen document with all the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By modifying / expanding the &lt;code&gt;D1.1GramCount &amp;gt; 0 and D2.1GramCount &amp;gt; 0&lt;/code&gt; part of the query you could easily make the comparison more sophisticated by, for instance, adding 2Grams, 3Grams, etc. or modifying the simple match to score according to the percentage match per ngram. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if your subject document has 0.0009% of the 1grams being &quot;the&quot;, document 1 has 0.001% and document 2 has 0.0015% then document 1 would score higher on &quot;the&quot; because the modulus of the difference (or whatever other measure you chose to use) is smaller.&lt;/p&gt;&#xA;" OwnerUserId="42134" LastEditorUserId="24000" LastEditDate="2017-11-20T15:05:23.137" LastActivityDate="2017-11-20T15:05:23.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24959" PostTypeId="2" ParentId="12554" CreationDate="2017-11-20T13:32:32.960" Score="4" Body="&lt;p&gt;A remark on Sandeep's answer:&#xA;Assuming 2 of your features are highly colinear (say equal 99% of time)&#xA;Indeed only 1 feature is selected at each split, but for the next split, the xgb can select the other feature. Therefore, the xgb feature ranking will probably rank the 2 colinear features equally. Without some prior knowledge or other feature processing, you have almost no means from this provided ranking to detect that the 2 features are colinear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, as for the relative importance that outputs the xgboost, it should be very similar (or maybe exactly similar) to the sklearn gradient boostined tree ranking. See &lt;a href=&quot;https://stats.stackexchange.com/questions/162162/relative-variable-importance-for-boosting&quot;&gt;here&lt;/a&gt; for explainations.&lt;/p&gt;&#xA;" OwnerUserId="42139" LastActivityDate="2017-11-20T13:32:32.960" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24964" PostTypeId="2" ParentId="24950" CreationDate="2017-11-20T17:16:28.070" Score="1" Body="&lt;p&gt;A loss function and cost function are the same thing.  As you intuit, classical regression treats loss/cost as symmetric, which is not always what you want. In classification tasks, you can make an asymmetric &lt;em&gt;loss matrix&lt;/em&gt;.  You can do a similar thing with regression if you solve it with gradient descent, but the ordinary least squares has symmetric loss baked in.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would consider either (1) using a numeric optimization library like sklearn or tensorflow to explicitly define the regression parameters you want to estimate, write your own custom loss function, and then do parameter estimation via gradient descent, or (2) finding a software package that allows for asymmetric loss, for example see &lt;a href=&quot;https://stats.stackexchange.com/questions/37955/how-to-design-and-implement-an-asymmetric-loss-function-for-regression&quot;&gt;this discussion&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-20T17:16:28.070" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24966" PostTypeId="1" CreationDate="2017-11-20T18:26:42.833" Score="2" ViewCount="970" Body="&lt;p&gt;When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good. &lt;/p&gt;&#xA;" OwnerUserId="15394" LastActivityDate="2019-04-19T06:02:05.370" Title="How much data is needed for a GBM to be more reliable than logistic regression for binary classification?" Tags="|logistic-regression|cross-validation|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24970" PostTypeId="2" ParentId="24944" CreationDate="2017-11-20T20:55:09.137" Score="0" Body="&lt;p&gt;It is one of many heuristics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't use it for evaluating clustering results. Use it as a guide for which result (or parameter, e.g., k) to explore &lt;em&gt;first&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there is no guarantee that the 'better' result by V measure is &lt;em&gt;actually&lt;/em&gt; better.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-20T20:55:09.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24971" PostTypeId="1" AcceptedAnswerId="24980" CreationDate="2017-11-20T21:02:09.890" Score="1" ViewCount="205" Body="&lt;p&gt;What do you say about this plot to find the number of cluster for kmean or kproto for mixed data. Where is the elbow to identify? I would say 5? I have 11 feautures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f0Dv2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f0Dv2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32963" LastEditorUserId="924" LastEditDate="2017-11-21T07:53:09.047" LastActivityDate="2017-11-21T07:53:54.870" Title="Clustering mixed data: Find optimal number of clusters" Tags="|machine-learning|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24972" PostTypeId="2" ParentId="24952" CreationDate="2017-11-20T21:44:30.640" Score="0" Body="&lt;p&gt;Randomness does not implies a 50/50 probability. You can have randomness as long as the probability is not 100/0 or 0/100. However, I believe it is not the randomness you are really asking here. In stead what you really asking is how to test whether the random process is bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Formally, let's use $Z \in (0, 1)$ to denote the outcome of an experiment of assigning a user. Then $X$ follows a Berniulli distribution with some probability $p$. The sum of the a series of Berniulli variables $X = Z_1 + Z_2 + ...+Z_n$ is a Binomal variable. We have also obvsered samples ($n = 304,641$) drawn from the experiment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your question is essentially a hypothesis test with &#xA;$H_0: p=.5, H_a:p\neq.5$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the null hypothesis, the mean and variance of X can be calculated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mu_x = np = 304,641 \times 0.5 = 152,320.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma_x^2 = np(1-p) = 76,160.25$ ($\sigma_x = 275.97$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because our $n$ is large, we can approximate the binomal distirbution by a normal distribution $N(\mu_x, \sigma_x&#xA;)$. It follows, assuming the null hypothesis, then $P(x &amp;gt;= 154,490) \approx 0.0$ . So we can reject the null hypothesis and conclude that the random process is not a 50/50 process. &lt;/p&gt;&#xA;" OwnerUserId="41525" LastActivityDate="2017-11-20T21:44:30.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24973" PostTypeId="2" ParentId="24971" CreationDate="2017-11-21T00:49:25.707" Score="1" Body="&lt;p&gt;You should select 9 as you can see from plot that for the WSS value there is a dip. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It doesn’t matter if you have 2 features or 9 features or n features. Clustering is on Data present in those features(it might depend on the amount of data).&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-21T07:53:54.870" LastActivityDate="2017-11-21T07:53:54.870" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24975" PostTypeId="1" AcceptedAnswerId="24976" CreationDate="2017-11-21T02:32:42.517" Score="-1" ViewCount="1187" Body="&lt;p&gt;I'm trying to learn from example on the internet. I choose the &lt;strong&gt;Iris&lt;/strong&gt; dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I checked for correlation, I saw that there was a correlation between '&lt;em&gt;petal length&lt;/em&gt;' and '&lt;em&gt;petal width&lt;/em&gt;'. But when I run RFE to select only 3 features, it took both these 2 features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that normal? &#xA;Doesn't correlation data contribute much to your model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: correlation value is 0.96&lt;/p&gt;&#xA;" OwnerUserId="42170" LastEditorUserId="42170" LastEditDate="2017-11-21T02:44:29.047" LastActivityDate="2017-11-21T02:52:53.053" Title="Correlation and feature selection" Tags="|feature-selection|correlation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24976" PostTypeId="2" ParentId="24975" CreationDate="2017-11-21T02:52:53.053" Score="2" Body="&lt;p&gt;As we were discussing above regarding the correlation, yes it is very important factor which would play an important role in selecting the features which are useful in explaining the &lt;strong&gt;Target Variable&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The RFE works like this, it would take all the features which ever are significant is explaining the Target Variable, if you take these 2 features separately and do the check then obviously they both are shown as significant features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before going to that step as you have done Correlation analysis and you should have eliminated one of the variable before giving to the model. The outcome of RFE is also correct as it was treating all the features independently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you need to remove the features which are strongly correlated because if two features explain the same thing then the model would be biased towards those features and you might end-up getting &lt;strong&gt;Wrong Insights&lt;/strong&gt;. So, strongly correlated features are highly recommended to be eliminated before doing RFE.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-21T02:52:53.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24978" PostTypeId="1" CreationDate="2017-11-21T05:17:15.060" Score="0" ViewCount="103" Body="&lt;p&gt;I am new to R programming and data analytics. This is probably a trivial question but I could not find an answer. I'm also limited with my ontology of data science and data manipulation. So, I'm not sure if I conveyed my question right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have huge irregular time series data close to 1 Million records. They are all in the form of time stamps with different discrete attributes. I want to rearrange the data as shown below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An Example of what I am trying to achieve FROM log entry of every car sold and the nature of the sale.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date         Vehicle        Sale Type       Registration    Profit&#xA;1-1-2017    Toyota  Camry   Cash Sale          CA           2500&#xA;1-1-2017    Toyota Rav4     Lease              CA           8500&#xA;1-1-2017    Toyota Camry    Installment        AZ           250&#xA;2-1-2017    Toyota Corolla  Installment        NV           250&#xA;4-1-2017    Toyota Etios    Cash Sale          AZ           2500&#xA;4-1-2017    Toyota Camry    Lease              CA           8500&#xA;4-1-2017    Toyota Prius    Installment        CA           250&#xA;13-1-2017   Toyota Prius    Installment        VA           250&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;TO with rows having car names and columns containing cash sale, lease, installment&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Vehicle Type    Cash Sale   Lease   Installment&#xA;Toyota Camry       1           1           1&#xA;Toyota Rav4                    1    &#xA;Toyota Corolla                             1&#xA;Toyota Etios       1        &#xA;Toyota Prius                               2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42173" LastEditorUserId="40853" LastEditDate="2017-11-21T05:30:24.437" LastActivityDate="2017-11-25T23:12:21.280" Title="R Programming rearranging rows and colums from timeline data" Tags="|r|data-cleaning|data-wrangling|" AnswerCount="2" CommentCount="1" ClosedDate="2017-11-21T20:40:31.657" ContentLicense="CC BY-SA 3.0" />
  <row Id="24979" PostTypeId="2" ParentId="24978" CreationDate="2017-11-21T06:54:45.730" Score="1" Body="&lt;p&gt;This would do the job:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#install packages&#xA;library(qdapTools)&#xA;library(dplyr)&#xA;#set working directory&#xA;setwd(&quot;C:/Users&quot;)&#xA;&#xA;#read csv&#xA;test &amp;lt;- read.csv(&quot;test.csv&quot;)&#xA;&#xA;#create dummy variable&#xA;#test.new &amp;lt;- dummy.data.frame(test$Sale.Type, sep = &quot;.&quot;)&#xA;test.new &amp;lt;- cbind(test, mtabulate(test$Sale.Type))&#xA;&#xA;#rename column&#xA;names(test.new)[names(test.new) == 'Cash Sale'] &amp;lt;- 'cashsale'&#xA;&#xA;#making new columns&#xA;test.new1 &amp;lt;- test.new %&amp;gt;%&#xA;                group_by(Vehicle) &#xA;&#xA;test.new2 &amp;lt;-  test.new1 %&amp;gt;%&#xA;                  mutate(sum_Cashsale = sum(cashsale)) %&amp;gt;%&#xA;                  mutate(sum_Ins = sum(Installment)) %&amp;gt;%&#xA;                  mutate(sum_Lease = sum(Lease))&#xA;&#xA;#select few necessary colums&#xA;test.new3 &amp;lt;- subset(test.new2, select=c(&quot;Vehicle&quot;, &quot;sum_Cashsale&quot;, &quot;sum_Ins&quot;,&quot;sum_Lease&quot;))&#xA;&#xA;#select the distict rows&#xA;test.new4 &amp;lt;- dplyr::distinct(test.new3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Just copy paste the code. &#xA;saved your sample data to test.csv&#xA;Give your working directory details, do let me know if you have any issues.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-21T07:21:11.977" LastActivityDate="2017-11-21T07:21:11.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24980" PostTypeId="2" ParentId="24971" CreationDate="2017-11-21T07:51:14.397" Score="1" Body="&lt;p&gt;The bumps at 8 and 11 are likely just due to random initialization, and if you rerun with a different random seed, then they will be at a different k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The elbow argument would probably suggest 3, but it is all but clear. I don't think there is a clear cut, but the values only drop as they would on uniform data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So most likely, a) your distance function is not good enough, 2) the algorithm does not work on this data, and/or c) this evaluation does not work on this data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-21T07:51:14.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24981" PostTypeId="2" ParentId="24949" CreationDate="2017-11-21T09:39:08.167" Score="0" Body="&lt;p&gt;This should give you the answer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f1 &amp;lt;- function(a,b,c,d,e,f) { &#xA;&#xA;  -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f&#xA;}&#xA;x = matrix(c(1:66), byrow = T, ncol = 6)&#xA;&#xA;for(i in 1:nrow(x)){&#xA;  new_out[i] &amp;lt;- f1(x[,1][i],x[,2][i],x[,3][i],x[,4][i],x[,5][i],x[,6][i])&#xA;}&#xA;&#xA;new_out &amp;lt;- as.data.frame(new_out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or instead of that for loop you can use this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_out2 &amp;lt;- as.data.frame(f1(x[,1],x[,2],x[,3],x[,4],x[,5],x[,6]))&#xA;colnames(new_out2) &amp;lt;- &quot;new_out&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Reason for the error is, it was taking x as 1 element. So, I called this function using its name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you any issues.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-21T10:41:32.103" LastActivityDate="2017-11-21T10:41:32.103" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24982" PostTypeId="2" ParentId="24978" CreationDate="2017-11-21T09:43:30.540" Score="1" Body="&lt;p&gt;You can do it also this way, using the &lt;a href=&quot;http://varianceexplained.org/r/teach-tidyverse/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tidyverse&lt;/a&gt; packages &lt;code&gt;tidyr&lt;/code&gt; and &lt;code&gt;dplyr&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# read in the data&#xA;&#xA;dat &amp;lt;- structure(list(Date = c(&#xA;  &quot;1-1-2017&quot;, &quot;1-1-2017&quot;, &quot;1-1-2017&quot;, &quot;4-1-2017&quot;,&#xA;  &quot;4-1-2017&quot;, &quot;4-1-2017&quot;&#xA;  ), Vehicle = c(&#xA;  &quot;Toyota Camry&quot;, &quot;Toyota Rav4&quot;,&#xA;  &quot;Toyota Camry&quot;, &quot;Toyota Etios&quot;, &quot;Toyota Camry&quot;, &quot;Toyota Prius&quot;&#xA;  ), SaleType = c(&#xA;  &quot;Cash Sale&quot;, &quot;Lease&quot;, &quot;Installment&quot;, &quot;Cash Sale&quot;,&#xA;  &quot;Lease&quot;, &quot;Installment&quot;&#xA;  ), Registration = c(&#xA;  &quot;CA&quot;, &quot;CA&quot;, &quot;AZ&quot;, &quot;AZ&quot;,&#xA;  &quot;CA&quot;, &quot;CA&quot;&#xA;  ), Profit = c(2500L, 8500L, 250L, 2500L, 8500L, 250L)), .Names = c(&#xA;  &quot;Date&quot;, &quot;Vehicle&quot;, &quot;SaleType&quot;, &quot;Registration&quot;,&#xA;  &quot;Profit&quot;&#xA;), class = &quot;data.frame&quot;, row.names = c(NA, -6L))&#xA;&#xA;&#xA;library(tidyr)&#xA;library(dplyr)&#xA;&#xA;# tidyr::gather() converts to &quot;long&quot; key-value format&#xA;dat %&amp;gt;% &#xA;            gather(k,v, -Registration, -Vehicle, -Date, -Profit) %&amp;gt;%   # -minus: keep as categorical variables&#xA;            group_by(Vehicle, v) %&amp;gt;%                                   # to add drill down info, add one of the &quot;-&quot; variables from gather()&#xA;            summarise(cnt=n()) %&amp;gt;% &#xA;            ungroup() %&amp;gt;%                                              # remove grouping metadata from data frame &#xA;            spread(key=v, value = cnt)                                 # create crosstab&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# A tibble: 4 x 4&#xA;       Vehicle `Cash Sale` Installment Lease&#xA;*        &amp;lt;chr&amp;gt;       &amp;lt;int&amp;gt;       &amp;lt;int&amp;gt; &amp;lt;int&amp;gt;&#xA;1 Toyota Camry           1           1     1&#xA;2 Toyota Etios           1          NA    NA&#xA;3 Toyota Prius          NA           1    NA&#xA;4  Toyota Rav4          NA          NA     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2017-11-25T23:12:21.280" LastActivityDate="2017-11-25T23:12:21.280" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="24983" PostTypeId="1" AcceptedAnswerId="24985" CreationDate="2017-11-21T11:19:40.607" Score="-1" ViewCount="621" Body="&lt;p&gt;I have an 2D array like the one shown below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;35046,  0.17    &#xA;1963,   0.34    &#xA;1135,   0.51    &#xA;405,    0.68    &#xA;99, 0.85    &#xA;1,  0.85    &#xA;15, 1.02    &#xA;2,  1.02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried using accumarray function in matlab on this data to get the results like this below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;35046,  0.17    &#xA;1963,   0.34    &#xA;1135,   0.51    &#xA;405,    0.68    &#xA;100,    0.85    &#xA;17, 1.02&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following is the code snippet for the same.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[num_dist, txt_dist]=xlsread(fname);&#xA;num_dist_final(:,1) = accumarray(num_dist(:,2),num_dist(:,1));&#xA;num_dist_final(:,2) = unique(num_dist(:,2));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But accumarray throws an error 'Error using accumarray: First input SUBS must contain positive integer subscripts.'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there way where I can use accumarray for float subscripts like the one in this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any other alternate methods are also welcome.&lt;/p&gt;&#xA;" OwnerUserId="26009" LastEditorUserId="29575" LastEditDate="2017-11-21T20:38:49.447" LastActivityDate="2017-11-21T20:38:49.447" Title="accumarray function not working in matlab" Tags="|matlab|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24985" PostTypeId="2" ParentId="24983" CreationDate="2017-11-21T14:20:28.917" Score="0" Body="&lt;p&gt;You can use the third output of &lt;a href=&quot;https://uk.mathworks.com/help/matlab/ref/unique.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;unique&lt;/code&gt;&lt;/a&gt; to convert your floats to a format suitable for &lt;code&gt;accumarray&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[C,~,ib] = unique(num_dist(:,2))&#xA;num_dist_final(:,1) = accumarray(ib,num_dist(:,1))&#xA;num_dist_final(:,2) = C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2532" LastEditorUserId="29575" LastEditDate="2017-11-21T16:49:32.450" LastActivityDate="2017-11-21T16:49:32.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24986" PostTypeId="1" AcceptedAnswerId="25102" CreationDate="2017-11-21T15:33:00.287" Score="20" ViewCount="8687" Body="&lt;p&gt;Let's say I have a top-down picture of an arrow, and I want to predict the angle this arrow makes. This would be between $0$ and $360$ degrees, or between $0$ and $2\pi$. The problem is that this target is circular, $0$ and $360$ degrees are exactly the same which is an invariance I would like to incorporate in my target, which should help generalization significantly (this is my assumption). The problem is that I don't see a clean way of solving this, are there any papers that try to tackle this problem (or similar ones)? I do have some ideas with their potential downsides:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a sigmoid or tanh activation, scale it to the ($0, 2\pi)$ range and incorporate the circular property in the loss function. I think this will fail fairly hard, because if it's on the border (worst prediction) only a tiny bit of noise will push the weights to go one way or the other. Also, values closer to the border of $0$ and $2\pi$ will be more difficult to reach because the absolute pre-activation value will need to be close to infinite.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Regress to two values, a $x$ and $y$ value and calculate the loss based on the angle these two values make. I think this one has more potential but the norm of this vector is unbounded, which could lead to numeric instability and could lead to blow ups or going to 0 during training. This could potentially be solved by using some weird regularizer to prevent this norm from going too far away from 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other options would be doing something with sine and cosine functions but I feel like the fact that multiple pre-activations map to the same output will also make optimization and generalizations very difficult.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2023-08-22T12:14:41.903" Title="Parameterization regression of rotation angle" Tags="|neural-network|deep-learning|loss-function|parameter-estimation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24987" PostTypeId="1" AcceptedAnswerId="25018" CreationDate="2017-11-21T17:11:49.800" Score="3" ViewCount="1691" Body="&lt;p&gt;So i'm very new to this, forgive my silly questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have .csv files containing data for every seconds over a year period. &#xA;The data consists simlpy of  timestamp and valueX.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to analyse these values to see if values X goes below a certain value, and if it does I have various calculations to carry out. Value A is proportional to X and just instantaneous, then value B is a counter proportional to A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you suggest a method/ language / software that would be the best and most accessible to do this? I don't have much experience in data analytics/big data apart from using excel and a bit of matlab and python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks in advance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &#xA;Thanks for the replies, my data is like this...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a text file for January values. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;01/02/2016 00:00:00,49.972332&#xA;01/02/2016 00:00:01,49.9690056&#xA;01/02/2016 00:00:02,49.9600029&#xA;01/02/2016 00:00:03,49.9490013&#xA;01/02/2016 00:00:04,49.9430046&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Different text files for each month, some files have text or notes at the top which I'd need to clean up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some context the second value is grid frequency, If it goes below a certain threshold, it constitutes an grid fault. I need to know how long these faults occur for, how often, how far the f value falls etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 2:&lt;/strong&gt;&#xA;Thanks for the tips folks, I've been able to read in my data, clean it up and combine it into one dataframe. &#xA;I'm now try to add some extra column to the dataframe that are dependent of the col 1 value but having some trouble.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &quot;Read in month csv files and create dataframes&quot; &#xA;df_jan_short = pd.read_csv('2016_01_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True) &#xA;df_feb_short = pd.read_csv('2016_02_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)&#xA;&#xA;    &quot;Combine dataframes into one dataframe for year&quot; months_short = [df_jan_short, df_feb_short] year_short = pd.concat(months_short)&#xA;&#xA;    &quot;Repalce NaN values in df with 50.00 Hz&quot; &#xA;year_short.fillna(50.00,inplace = True)&#xA;&#xA;    &quot;Check what range freq is in&quot;&#xA;&#xA;year_short['Case'] = 'No Case'&#xA;year_short['Case'](year_short['Freq'] &amp;gt; Fc &amp;amp; year_short['Freq'] &amp;lt; Ff ) = 'A' year_short['Case'](year_short['Freq'] &amp;gt; Fa &amp;amp; year_short['Freq'] &amp;lt; Fc ) = 'B'&#xA;year_short['Case'](year_short['Freq'] &amp;lt; Fa ) = 'C' &#xA;year_short['Case'](year_short['Freq'] &amp;gt; Ff &amp;amp; year_short['Freq'] &amp;lt; Fh ) = 'G'&#xA;year_short['Case'](year_short['Freq'] &amp;gt; Fh ) = 'H'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I get the following error...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')&#xA;Traceback (most recent call last):&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py&quot;, line 2862, in run_code&#xA;    exec(code_obj, self.user_global_ns, self.user_ns)&#xA;&#xA;  File &quot;&amp;lt;ipython-input-74-3a0fbc1f23f6&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py&quot;, line 710, in runfile&#xA;    execfile(filename, namespace)&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py&quot;, line 101, in execfile&#xA;    exec(compile(f.read(), filename, 'exec'), namespace)&#xA;&#xA;  File &quot;C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py&quot;, line 48&#xA;    year_short['Case'](year_short['Freq'] &amp;gt; Fc &amp;amp; year_short['Freq'] &amp;lt; Ff ) = 'A'&#xA;                                                                                ^&#xA;SyntaxError: can't assign to function call&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42200" LastEditorUserId="42200" LastEditDate="2017-11-24T16:01:44.080" LastActivityDate="2017-11-24T16:01:44.080" Title="How to deal with large data sets" Tags="|dataset|data|csv|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24988" PostTypeId="2" ParentId="24987" CreationDate="2017-11-21T17:16:55.703" Score="5" Body="&lt;p&gt;Welcome! I think you asked in the right place.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have familiarity with Python, you might look at the numpy and Pandas libraries.  Numpy implements fast array and matrix manipulations, and Pandas arranges numpy objects into tables.  Along with scipy, they make the basis of the Python numerical computing stack.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without more detail it's difficult to share a code example that'll do what you want, but if you run into any issues feel free to ask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: if your dataset is REALLY big and won't fit in memory, there are ways around that too, just specify.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-21T17:16:55.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24989" PostTypeId="1" CreationDate="2017-11-21T18:07:03.063" Score="-1" ViewCount="1250" Body="&lt;p&gt;I have a dataset of around 10 million rows and around 10 columns. I have missing data that occurs sporadically across 4 of these variables. What technique would you recommend I use in sklearn to fill these gaps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see most of the techniques in sklearn are for imputing 1 specific variable that you have missing data in. In this case there are 4 variables that are missing, sometimes individually, sometimes a number of them at the same time.&lt;/p&gt;&#xA;" OwnerUserId="42202" LastEditorUserId="29575" LastEditDate="2017-12-24T16:21:41.273" LastActivityDate="2018-04-23T20:54:53.807" Title="Imputing for multiple missing variables using sklearn" Tags="|scikit-learn|data-imputation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="24990" PostTypeId="1" AcceptedAnswerId="24992" CreationDate="2017-11-21T18:44:09.630" Score="14" ViewCount="5364" Body="&lt;p&gt;I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting the curve. Is the curve below abnormal? If yes, why and how can I correct it considering scikit learn automatically sorts the true and predicted labels. If the plot is OK, how best do I explain this behaviour?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4wQQy.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4wQQy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42204" LastActivityDate="2017-11-21T20:20:54.283" Title="Irregular Precision-Recall Curve" Tags="|scikit-learn|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="24992" PostTypeId="2" ParentId="24990" CreationDate="2017-11-21T20:20:54.283" Score="12" Body="&lt;p&gt;This is definitely possible. When you are reducing the threshold, you will never decrease the recall (you can only flag more of the positive examples as positive). Precision is looking at all the examples that you flag positively, and of those the fraction that are truly positive. This means when you are reducing the threshold, you might not add any true positives but only false positives, thereby lowering your precision. Let's take a look at a fabricated example, where P is positive and N is negative. The samples are ranked by score/probability. Everything before the threshold is flagged as positive:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PPNPNNPNNN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we put the threshold between items 2 and 3, we get a precision of $1$ and a recall of $0.5$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PP - NPNNPNNN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When we put it between items 3 and 4, we keep the same recall but our precision drops to $0.6667$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;PPN - PNNPNNN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-21T20:20:54.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24995" PostTypeId="1" CreationDate="2017-11-22T01:12:42.647" Score="3" ViewCount="579" Body="&lt;p&gt;What ML techniques can be used to determine relevance or context between sentences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I am going to the event at 9pm. Can you meet me there?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, there is context between the two sentences - the person is indicating they will be at the event at 9pm, and is asking the recipient to meet them there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conversely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I am going to the event at 9pm. Can you meet me at school tomorrow?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this text alone, it appears as if the sentences represent two disjoint thoughts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, although the examples above are of adjacent sentences, the context may exist across multiple sentences, or sentences that are not adjacent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I am going to the event at 9pm. I've heard it is fabulous. Can you meet me there?&quot;&lt;/p&gt;&#xA;" OwnerUserId="42218" LastActivityDate="2017-12-30T13:07:51.163" Title="What ML techniques can be used to determine context across sentences?" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="24997" PostTypeId="2" ParentId="24989" CreationDate="2017-11-22T01:32:49.150" Score="2" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand your question correctly you mean to say that you are facing issue with replacing the missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, we cannot use the &lt;strong&gt;Same Technique&lt;/strong&gt; to replace missing values for all the features. It depends on the &lt;strong&gt;Feature&lt;/strong&gt; and &lt;strong&gt;Business Understanding&lt;/strong&gt;. Let us consider a scenario, where you are having missing vales in your Income feature. Now to replace all the missing values with mean but that might work in your scenario or not. It should be your &lt;strong&gt;Understanding on Data&lt;/strong&gt; and if it works for your business. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another scenario, here this is a categorical variable. The field is room category in reservation data. Here if the data is missing then you can replace them with others/other rooms so on. All these depend on your business.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below links would give you a better idea about different methods and how to choose the one suits your business: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://measuringu.com/handle-missing-data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://liberalarts.utexas.edu/prc/_files/cs/Missing-Data.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-3&lt;/a&gt;,(in this link go for second segment, under the table of content)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do let me if you have any issues.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-22T07:12:52.463" LastActivityDate="2017-12-22T07:12:52.463" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="24998" PostTypeId="1" AcceptedAnswerId="25003" CreationDate="2017-11-22T02:00:04.717" Score="-1" ViewCount="549" Body="&lt;p&gt;I am a newbie in the field of AI/ML. I am trying to implement predictive analytics model on the data generated every minute from a device with sensors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are various ML algorithms I can use to predict the number of people in a room given temperature, humidity, luminosity, and motion { 0 | 1 }?&lt;/li&gt;&#xA;&lt;li&gt;What other things can we predict using the above data from the sensor that is deployed in a closed room?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Context:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The device sends temperature, humidity, luminosity, and motion(yes/no) in real-time. I deployed this device in a closed room and started collecting data. Now I want to use this data to predict the number of people in the room using the data collected. I believe a multiple (linear/poly) regression model will help me in achieving this but, wanted to know if there are any other algorithms or any other use cases I can look into.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sensor Specifications:&lt;/strong&gt;&#xA;&lt;a href=&quot;https://www.elsys.se/en/ers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;click for more details&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LoRa Alliance Certified&lt;/li&gt;&#xA;&lt;li&gt;Temperature (Accuracy: ± 0.5°C, Resolution: 0.1°C)&lt;/li&gt;&#xA;&lt;li&gt;Humidity (Accuracy: ±2%rh, Resolution: 0.1%rh)&lt;/li&gt;&#xA;&lt;li&gt;Light&lt;/li&gt;&#xA;&lt;li&gt;Motion (PIR)&lt;/li&gt;&#xA;&lt;li&gt;NFC for easy configuration&lt;/li&gt;&#xA;&lt;li&gt;Size : 86x86x26mm&lt;/li&gt;&#xA;&lt;li&gt;US902-928, EU863-870, AS923, AU915-928, KR920-923&lt;/li&gt;&#xA;&lt;li&gt;2 x 3.6V AA lithium battery&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42210" LastEditorUserId="42210" LastEditDate="2017-11-22T16:16:59.907" LastActivityDate="2017-11-22T16:16:59.907" Title="What are suitable predictive analytics models for data from multiple sensors?" Tags="|machine-learning|scikit-learn|regression|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="25000" PostTypeId="1" AcceptedAnswerId="25001" CreationDate="2017-11-22T03:19:49.220" Score="0" ViewCount="8158" Body="&lt;p&gt;I have a matrix with size m×n that is built from n number of individuals for person identification. So, n is the number of person and m is the number of feature's value for the person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It makes me confused about observation and variables. What will I call n and m? Which one represents observation and which one represents variable? My confusion will be cleared to you if you visit the following link: &lt;a href=&quot;https://datascience.stackexchange.com/questions/1159/how-to-do-svd-and-pca-with-big-data&quot;&gt;How to do SVD and PCA with big data?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42220" LastActivityDate="2020-12-06T21:36:35.893" Title="What is the difference between observation and variable?" Tags="|deep-learning|data-mining|bigdata|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25001" PostTypeId="2" ParentId="25000" CreationDate="2017-11-22T04:09:35.053" Score="1" Body="&lt;p&gt;In your case, 'n' is the number of observations, and 'm' is the number of variables.  Think of a table in a database - each row is a 'record', and each record has various properties.  A record corresponds to an observation.  Each property is a variable or property of that observation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some contexts people make a distinction between data (i.e. observations, things observed) and variables (parameters to learn - things that vary in a mathematical sense).  But in the context of the link you sent, a variable is a feature.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-22T04:09:35.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25002" PostTypeId="1" CreationDate="2017-11-22T04:28:10.480" Score="-1" ViewCount="527" Body="&lt;p&gt;I was using Dicekrigging in order to Bayesian optimization in R. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While finding the acquisition function I got the specified error. I got stuck for many hours. Any kind of help is appreciated. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#install packages&#xA;library(DiceKrigging)&#xA;library(DiceOptim)&#xA;library(lhs)&#xA;&#xA;nsteps &amp;lt;- 10&#xA;&#xA;d=6&#xA;&#xA;lower &amp;lt;- c(-2,-2,-2,0,0,0)&#xA;&#xA;upper &amp;lt;- c(5,5,5,1,10,10)&#xA;X=lhs(n,d)&#xA;&#xA;colnames(X)=c(&quot;x1&quot;,&quot;x2&quot;,&quot;x3&quot;,&quot;x4&quot;,&quot;x4&quot;,&quot;x5&quot;,&quot;x6&quot;)&#xA;&#xA;Temp=c(1:n)&#xA;&#xA;&#xA;mf2 &amp;lt;- km(~ 1+x1+sin(x2) +x3+I(x4^-1)+x5+x6,  design = X, &#xA;response=Temp,covtype=&quot;matern5_2&quot;,&#xA;          nugget=epsilon,multistart = 5,control=list(maxit=500,trace=T,popsize=50))&#xA;&#xA;&#xA;model &amp;lt;- mf2&#xA;&#xA;f1 &amp;lt;- function(X) {  x1 &amp;lt;- x[1]&#xA;x2 &amp;lt;- x[2]&#xA;x3 &amp;lt;- x[3]&#xA;x4 &amp;lt;- x[4]&#xA;x5 &amp;lt;- x[5]&#xA;x6 &amp;lt;- x[6]&#xA;&#xA;L=-111.563*x1-29.81*sin(x2)+35.202*x3-.00012485/x4-1.045997*x5+11.6344*x6+13.1035&#xA;return(L)&#xA;}&#xA;&#xA;&#xA;&#xA;oEGO &amp;lt;- EGO.nsteps(model = mf2, fun = f1, nsteps = nsteps,&#xA;                       + lower, upper, control = list(pop.size = 20, BFGSburnin = 2))&#xA;&#xA;&#xA;----------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I run the last code it is showing:&#xA;Error in y.predict.trend + y.predict.complement : non-conformable arrays&lt;/p&gt;&#xA;" OwnerUserId="29967" LastEditorUserId="40853" LastEditDate="2017-11-23T18:27:04.113" LastActivityDate="2017-11-23T18:27:04.113" Title="Error in y.predict.trend + y.predict.complement : non-conformable arrays" Tags="|machine-learning|r|data-mining|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25003" PostTypeId="2" ParentId="24998" CreationDate="2017-11-22T08:17:24.033" Score="2" Body="&lt;p&gt;First, to answer your questions directly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) What M/L algorithms to try: Since number of people can be treated as a continuous variable, if you are using python, you can try everything from &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;multiple linear regression &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LinearRegression&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;polynominal regression, first create polynomial features using sklearn.preprocessing library, then fit the model using linear regression again&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import PolynomialFeatures&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;support vector machine&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.svm import SVR&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;decision tree regressor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import DecisionTreeRegressor&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;random forest regressor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import RandomForestRegressor&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;AdaBoost regressor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import AdaBoostRegressor&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;XGBoost, this one needs to be installed separately, following the instructions on this link: &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/build.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/build.html#&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from xgboost import XGBClassifier&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Deep neural network, I wouldn't recommend trying this in the beginning since it can be constructed into very flexible and complex models with way too many choices on the network layout. But if you feel adventurous, this one can be quite fun. Simplest way to get started on this is probably using the keras wrapper for TensorFlow. Here're just some of the most commonly used models and layers&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import Dropout&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Recurrent neural network, since you do have time series sensor data, you can potentially use RNN to predict future occupancy based on the past sensor data. You would combine the above keras models, then add LSTM layers to get the 'memory' effect&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import LSTM&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;2) This one is very difficult to answer as it depends on the configuration of the sensors, setup of the room, how the sensor is placed, is the motion detector a PIR, if PIR, is it dual or quad sensor, what is the shape of the PIR lens, is there an ultrasound sensor, how many channels on the luminosity sensor module, does it read RGB ... etc. This one would require experience, tons of experimentation and ingenuity. You can maybe consult with someone with physics, electronics or electrical engineering background. Some wild ideas: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;if you know how many people are in the room (through your prediction), and you know the room setup, size and thermal characteristics, using temperature and humidity sensor, you can probably estimate the average mass of the individuals in the room. This one is interesting in that it combines a physics model and a machine learning model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the luminosity sensor can probably tell you if the room has natural light or artificial light (looking at rate of illuminance change, does it follow diurnal patterns)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;motion sensor can probably tell you (other than yes or no motion), whether the motion is large, small, fast or slow, depending on the sensor setup&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Finally, some thoughts on training this model. Before starting to play with all the ML algorithms, it's probably good to spend a good week or two figuring out the 'preprocessing' of the sensor time series. The model accuracy is highly dependent on the preprocessing steps and feature selection / engineering, probably even more so than the choice of the algorithms. If you feed most algorithms with enough data and do a good job on cross-validation, you can get similar accuracy results (at least for simple regression-type applications). Some preprocessing issues to consider&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What is your window frame size? Will you simply take moving average, or combine some other statistics within each data window?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How do you remove or reduce the noise from the sensors? e.g., how do you know the signal you are looking at results from actual physical phenomenon v.s. electronic noise?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does it help to do fourier transform and filtering? Is it better to use frequency-domain features or time-domain features? Or wavelets?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What about dimensionality reduction techniques like PCA, LDA?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Lots of things to consider, which is also why machine learning is both challenging and really fun.&lt;/p&gt;&#xA;" OwnerUserId="32296" LastActivityDate="2017-11-22T08:17:24.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25004" PostTypeId="1" CreationDate="2017-11-22T08:47:23.657" Score="2" ViewCount="3245" Body="&lt;p&gt;I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself.  I've pre-processed the content column in such a way that the subject and associated metadata have been completely removed.  My thinking, at this point, is that I should train the classifier on two separate bag-of-words, one for the subject column and one for the content column.  Is this the right approach?  Is it there a simple way to accomplish this using NLTK or a similar library?&lt;/p&gt;&#xA;" OwnerUserId="42224" LastActivityDate="2021-06-07T19:00:02.920" Title="Text Classifier with multiple bag-of-words" Tags="|classification|text-mining|nltk|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25005" PostTypeId="1" AcceptedAnswerId="25014" CreationDate="2017-11-22T08:52:53.410" Score="0" ViewCount="3386" Body="&lt;p&gt;I'm studying about Sentiment analysis. What is the purpose of using Document Level, Sentence Level and Aspect Level? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read this definition in Web Data Mining book by Bing liu. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this book it was stated that, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sentence Level sentiment analysis is to classify a sentence to negative, positive, neutral class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document level sentiment analysis to classify a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aspect Level sentiment analysis is to classify each aspect of entity mentioned in a review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="42225" LastEditorUserId="40853" LastEditDate="2017-11-22T10:08:39.990" LastActivityDate="2017-11-22T15:13:53.127" Title="Difference usage of document level, sentence level and aspect level in sentiment analysis" Tags="|machine-learning|data-mining|nlp|sentiment-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25006" PostTypeId="1" AcceptedAnswerId="25028" CreationDate="2017-11-22T09:38:36.057" Score="1" ViewCount="132" Body="&lt;p&gt;I am currently trying to learn about deep learning. I asked myself where deep learning outperforms SVM and Random Forest on regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any dataset for regression where deep learning outperforms SVM and Random Forest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main objective for asking this question, is to see on which kind of datasets deep learning outperforms SVM and Random Forests. Since I do not know what kind of datasets these are, I do not want to be too restrictive when asking about these datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;:&#xA;From the answer given by tom, it gets clear to me, what kind of dataset I am looking for. So to be more specific: If you know a dataset with spatial/temporal structure at different levels of granularity which is known that for example convolutional deep networks outperform SVM and Random Forest on this dataset, then please share this dataset or a link to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't know why the question is still marked as on hold for being to broad, as I updated the question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerDisplayName="user42229" LastEditorDisplayName="user42229" LastEditDate="2017-11-24T10:34:47.640" LastActivityDate="2017-11-24T10:34:47.640" Title="Do you know a dataset for regression where deep learning outperforms svm and random forests?" Tags="|deep-learning|dataset|regression|random-forest|svm|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-22T23:51:43.780" ContentLicense="CC BY-SA 3.0" />
  <row Id="25008" PostTypeId="2" ParentId="17079" CreationDate="2017-11-22T11:31:48.893" Score="8" Body="&lt;p&gt;You can apply a technique I described in &lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf#page=62&quot; rel=&quot;nofollow noreferrer&quot;&gt;my masters thesis (page 48ff)&lt;/a&gt; and called Confusion Matrix Ordering (CMO):&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Order the columns/rows in such a way, that most errors are along the diagonal.&lt;/li&gt;&#xA;&lt;li&gt;Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because there are to few data points.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Nice side effect: This method also automatically clusters similar classes together. Figure 5.12 of my masters thesis shows that:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dNw1T.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dNw1T.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;You can apply confusion matrix ordering with &lt;a href=&quot;https://github.com/MartinThoma/clana/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;clana&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2020-07-14T08:36:34.707" LastActivityDate="2020-07-14T08:36:34.707" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25010" PostTypeId="2" ParentId="25004" CreationDate="2017-11-22T13:19:50.243" Score="1" Body="&lt;p&gt;I cannot comprehend the question completely but let me see if I got this right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to train a classifier. You have different columns for subject and body. You have now pre-processed it in a manner such that subject and other meta information have been removed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I shall try to answer your question. Keep in mind that, this is how I would approach the problem (based on experience with similar task in the past) but there could be better ways or approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would not remove the subject column in the first place. I would also not train the classifier on two separate bag-of-words. I would append both, the subject and the content, into one single document and then classify them accordingly with one unit of bag-of-words instead of two. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would I do this? There could be times when the subject could have keywords or something that could lead to better classification than the body of the email. The vice versa could also be true. Hence, without having seen the entire dataset, and to be on the safer side, I would also give this approach a shot. Compare both of your approaches and then see where you have better scores of accuracy, precision, recall and other metrics that aids you in your conclusions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ways to accomplish this? Yes, NLTK is one way to get the bag of words. However, I would go ahead with scikit-learn package to train my classifier. I would also run LSA/LDA/HDP (Topic Modeling Algorithms) to see what constitutes my data. To do this, I would use Gensim. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://radimrehurek.com/gensim/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional tutorials to help you get started with text classification if you are new to it (that is what it seems like, pardon me if I am wrong). These will also help you in achieving better results to begin with:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://zacstewart.com/2015/04/28/document-classification-with-scikit-learn.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://zacstewart.com/2015/04/28/document-classification-with-scikit-learn.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although both the tutorials I mentioned primarily concentrate on binary classification, you can also achieve fine-grained (multi-class) classification in similar manner. &lt;/p&gt;&#xA;" OwnerUserId="16401" LastEditorUserId="25180" LastEditDate="2019-05-12T21:15:00.693" LastActivityDate="2019-05-12T21:15:00.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25011" PostTypeId="1" AcceptedAnswerId="25023" CreationDate="2017-11-22T13:24:23.247" Score="1" ViewCount="14923" Body="&lt;p&gt;Sample input dataset is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KBbgY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KBbgY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample output dataset what i want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0   2017-11-17 10:23:28.691 788&#xA;0   2017-11-17 10:23:28.691 756&#xA;0   2017-11-17 10:23:28.691 789&#xA;0   2017-11-17 10:23:28.691 780&#xA;1   2017-11-17 10:23:29.731 788&#xA;1   2017-11-17 10:23:29.731 783&#xA;2   2017-11-17 10:23:30.655 747&#xA;3   2017-11-17 10:23:31.627 766&#xA;4   2017-11-17 10:23:32.606 807&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I do this by pandas? or is there any other technique to do this?&lt;/p&gt;&#xA;" OwnerUserId="29824" LastEditorUserId="381" LastEditDate="2017-11-22T22:58:52.343" LastActivityDate="2019-10-05T21:45:03.623" Title="How to transpose rows into single column based on time-stamp index in python?" Tags="|python|pandas|ipython|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25012" PostTypeId="2" ParentId="16174" CreationDate="2017-11-22T14:09:06.920" Score="3" Body="&lt;p&gt;This is a good question - one that is of interest (or should be of interest) to those working in applied social sciences and operations or basically in any situation where the objective function is on panel data. There is a paper by Bai, Zhong and Gao on this topic: &lt;a href=&quot;https://arxiv.org/abs/1708.00185&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1708.00185&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42239" LastActivityDate="2017-11-22T14:09:06.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25013" PostTypeId="1" CreationDate="2017-11-22T15:02:30.417" Score="2" ViewCount="249" Body="&lt;p&gt;I've read about tangent distance used in machine learning but I don't really know how it works. Concretely, consider two very different manifolds &lt;strong&gt;M1&lt;/strong&gt; and &lt;strong&gt;M2&lt;/strong&gt; and two points &lt;strong&gt;x1&lt;/strong&gt;, &lt;strong&gt;x2&lt;/strong&gt; on that two manifolds respectively. &lt;strong&gt;M1&lt;/strong&gt; is very far from &lt;strong&gt;M2&lt;/strong&gt;. What if the tangent plane &lt;strong&gt;p1&lt;/strong&gt; of &lt;strong&gt;M1&lt;/strong&gt; at &lt;strong&gt;x1&lt;/strong&gt; isn't parallel to the tangent plane &lt;strong&gt;p2&lt;/strong&gt; of &lt;strong&gt;M2&lt;/strong&gt; at &lt;strong&gt;x2&lt;/strong&gt; ? In fact, &lt;strong&gt;p1&lt;/strong&gt; will intersect &lt;strong&gt;p2&lt;/strong&gt; in this case, so distance(&lt;strong&gt;x1&lt;/strong&gt;, &lt;strong&gt;x2&lt;/strong&gt;) is zero. This means that &lt;strong&gt;x1&lt;/strong&gt; and &lt;strong&gt;x2&lt;/strong&gt; will be classied into the same class regardless to the fact that &lt;strong&gt;M1&lt;/strong&gt; is very far from &lt;strong&gt;M2&lt;/strong&gt;. Can anybody explain this paradox for me ? Thanks alot for your helps&lt;/p&gt;&#xA;" OwnerUserId="42242" LastEditorUserId="42242" LastEditDate="2017-11-24T13:55:56.057" LastActivityDate="2017-11-24T13:55:56.057" Title="How tangent distance works in three-dimentional space" Tags="|machine-learning|deep-learning|distance|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25014" PostTypeId="2" ParentId="25005" CreationDate="2017-11-22T15:13:53.127" Score="0" Body="&lt;p&gt;I am building on Toros91's answer. He has said: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Use case of all these 3 different levels are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sentence Level: Used to classify a sentences for instance, whether a tweet is positive or negative or neutral.&#xA;Document Level: Used to classify the whole document level to say whether that document sentiment is positive or negative or neutral.&#xA;Aspect Level: Used to select an aspect and give the outcome of that respective Aspect, for instance let us consider Iphone X as an&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;aspect and mine tweets(this is an example you can use documents and so&#xA;  on) and carry out Sentiment Analysis.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, to answer your question further about how it can be beneficial, just take an example of some review. I am just forming my own review on something as an example, I do not know the specifics, so please do not quote me here. This has some business logic implied as well.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I feel the latest laptop from Mac is really good overall. It has&#xA;  amazing resolution. The computer is really very sleek and can slide&#xA;  into bags easily. However, I feel the weight is a let down. The price&#xA;  is a bit expensive given the configurations. I expect an SSD storage.&#xA;  However, the processor seems really good.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;At a document level, I can say the entire paragraph looks more positive than negative. So, the review, if has to be labeled as one thing, is positive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At a sentence level, I can break the document into 6 sentences and classify each of them as positive, negative or neutral. I get a bit more specific here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, at an aspect level, I get information about price, storage, processor and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given N customer reviews, you can know how well different aspects of a particular product are received. Say, 10 customers feel the price is fine, but 4 feel it is expensive. 13 of your 14 customers really love the resolution while 1 feel it is okayish. A lot of products in day to day life have multiple things to be considered. And this is where fine grained labeling help analysts or scientists. &lt;/p&gt;&#xA;" OwnerUserId="16401" LastActivityDate="2017-11-22T15:13:53.127" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25015" PostTypeId="1" AcceptedAnswerId="25016" CreationDate="2017-11-22T15:48:04.790" Score="4" ViewCount="1330" Body="&lt;p&gt;I need to extract relevant key phrases from a single document. Since I don't have a lot of documents, TF-IDF doesn't really work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I'm using TextRank. It produces okay-ish result - some really good phrases along with a lot of garbage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a better algorithm to use for this? Can anyone give me a rundown of available options?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Real-world use case&lt;/strong&gt;: I'm developing a help desk app that comes with Knowledge Base (a bunch of articles, think of it as FAQ). When a user writes a new support ticket I want to extract key phrases and find the most relevant KB articles. Overall there is not enough data to train a model. I need to compare sets of key phrases I think.&lt;/p&gt;&#xA;" OwnerUserId="42187" LastActivityDate="2019-03-18T14:54:32.140" Title="Extract key phrases from a single document" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25016" PostTypeId="2" ParentId="25015" CreationDate="2017-11-22T16:03:44.940" Score="3" Body="&lt;p&gt;A related keyword to your case can be &lt;strong&gt;Single Document Keyword Extraction&lt;/strong&gt;. A good paper about this is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We present a new keyword extraction algorithm that&#xA;  applies to a single document without using a corpus.&#xA;  Frequent terms are extracted first, then a set of cooccurrence&#xA;  between each term and the frequent terms,&#xA;  i.e., occurrences in the same sentences, is generated.&#xA;  &lt;strong&gt;Co-occurrence distribution shows importance of a term&#xA;  in the document as follows. If probability distribution&lt;/strong&gt; of&#xA;  co-occurrence between term a and the frequent terms is&#xA;  biased to a particular subset of frequent terms, then term&#xA;  a is likely to be a keyword. The degree of biases of distribution&#xA;  is measured by the $\chi^2$-measure. Our algorithm&#xA;  shows comparable performance to tfidf without using a&#xA;  corpus.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can find the paper &lt;a href=&quot;https://ocs.aaai.org/Papers/FLAIRS/2003/Flairs03-076.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, this paper gives a rank on keywords based on the defined $\chi^2$-measure.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-22T16:03:44.940" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25018" PostTypeId="2" ParentId="24987" CreationDate="2017-11-22T18:09:23.477" Score="4" Body="&lt;p&gt;As mentioned python's pandas library is good start. They have a lot of time series functionality, see e.g. the documentation &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/timeseries.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. You can load your data like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;s = pd.read_csv(&quot;your_file.csv&quot;,header=None,index_col=0,names=['timestamp','value'],squeeze=True,parse_dates=[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives you a timeseries with the timestamps as index:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s&#xA;Out[1]: &#xA;timestamp&#xA;2016-01-02 00:00:00    49.972332&#xA;2016-01-02 00:00:01    49.969006&#xA;2016-01-02 00:00:02    49.960003&#xA;2016-01-02 00:00:03    49.949001&#xA;2016-01-02 00:00:04    49.943005&#xA;Name: value, dtype: float64&#xA;&#xA;s&amp;lt;49.95&#xA;Out[2]: &#xA;timestamp&#xA;2016-01-02 00:00:00    False&#xA;2016-01-02 00:00:01    False&#xA;2016-01-02 00:00:02    False&#xA;2016-01-02 00:00:03     True&#xA;2016-01-02 00:00:04     True&#xA;Name: value, dtype: bool&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If your data is too big to process as a whole, you can iterate over chunks using the &lt;code&gt;chunksize&lt;/code&gt; option in &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;read_csv&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2017-11-22T18:09:23.477" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25021" PostTypeId="2" ParentId="21676" CreationDate="2017-11-22T19:20:41.870" Score="0" Body="&lt;p&gt;You can use clustering algorithm to cluster closer dates together. But since you've mentioned the number of dates to be clustered won't be more than 20, seems like you can just create a simple logic to group them together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pick a base date which can be anything and find the num of days/weeks/months from the base date to each date in your dataset. you'll get a bunch of numbers now. You can now bucket them together according to a threshold you like. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although clustering algorithm too would do the same. Just the thresholding would be be taken care of automatically based on optimal cutoff. Try the simplest (read: easy to understand) clustering algorithm: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;K-Means&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2017-11-22T19:20:41.870" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25023" PostTypeId="2" ParentId="25011" CreationDate="2017-11-22T20:04:32.173" Score="2" Body="&lt;p&gt;Assuming you have the following source DF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [21]: df&#xA;Out[21]:&#xA;         Time  val1  val2  val3&#xA;0  2017-11-17    11    12    13&#xA;1  2017-11-18    24    25     0&#xA;2  2017-11-19    37     0     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [22]: (df.replace(0, np.nan)&#xA;            .set_index('Time', append=True)&#xA;            .stack()&#xA;            .reset_index()&#xA;            .rename(columns={0:'val'})&#xA;            .drop('level_2',1))&#xA;Out[22]:&#xA;   level_0        Time   val&#xA;0        0  2017-11-17  11.0&#xA;1        0  2017-11-17  12.0&#xA;2        0  2017-11-17  13.0&#xA;3        1  2017-11-18  24.0&#xA;4        1  2017-11-18  25.0&#xA;5        2  2017-11-19  37.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-11-22T20:04:32.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25024" PostTypeId="1" AcceptedAnswerId="25027" CreationDate="2017-11-22T21:03:02.647" Score="16" ViewCount="15283" Body="&lt;p&gt;I'm trying to train a single perceptron (1000 input units, 1 output, no hidden layers) on 64 randomly generated data points. I'm using Pytorch using the Adam optimizer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import torch&#xA;from torch.autograd import Variable&#xA;&#xA;torch.manual_seed(545345)&#xA;N, D_in, D_out = 64, 1000, 1&#xA;&#xA;x = Variable(torch.randn(N, D_in))&#xA;y = Variable(torch.randn(N, D_out))&#xA;&#xA;model = torch.nn.Linear(D_in, D_out)&#xA;loss_fn = torch.nn.MSELoss(size_average=False)&#xA;&#xA;optimizer = torch.optim.Adam(model.parameters())&#xA;for t in xrange(5000):&#xA;  y_pred = model(x)&#xA;  loss = loss_fn(y_pred, y)&#xA;&#xA;  print(t, loss.data[0])&#xA;&#xA;  optimizer.zero_grad()&#xA;  loss.backward()&#xA;  optimizer.step()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Initially, the loss quickly decreases, as expected:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(0, 91.74887084960938)&#xA;(1, 76.85824584960938)&#xA;(2, 63.434078216552734)&#xA;(3, 51.46927261352539)&#xA;(4, 40.942893981933594)&#xA;(5, 31.819372177124023)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Around 300 iterations, the error reaches near zero:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(300, 2.1734419819452455e-12)&#xA;(301, 1.90354676465887e-12)&#xA;(302, 2.3347573874232808e-12)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This goes on for a few thousand iterations. However, after training for too long, the error starts to increase again:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(4997, 0.002102422062307596)&#xA;(4998, 0.0020302983466535807)&#xA;(4999, 0.0017039275262504816)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why is this happening?&lt;/p&gt;&#xA;" OwnerUserId="42255" LastEditorUserId="42255" LastEditDate="2017-11-22T21:23:07.770" LastActivityDate="2018-04-29T20:57:14.823" Title="Strange behavior with Adam optimizer when training for too long" Tags="|perceptron|pytorch|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25026" PostTypeId="2" ParentId="24872" CreationDate="2017-11-22T21:46:33.747" Score="3" Body="&lt;p&gt;Many neural network examples you see in the literature are doing classification problems, E.g. LeNet, AlexNet, Inception, etc are all image classification problems. In this domain, it's useful for the neural network to give outputs between 0 and 1 because an output between 0 and 1 can be interpreted, in some sense, as a probability. The reason these networks output numbers between 0 and 1 is in the layer activations of the network. The last layer in these networks is usually a softmax layer (or, if you're doing just binary classification, a sigmoid layer). Softmax and sigmoid functions have the nice property that they give outputs between 0 and 1 (softmax has the added nice property that it gives outputs which sum to 1). If you want your neural net to be able to output numbers that aren't between 0 and 1, simply change your activation function. Instead of a softmax last layer, you could use a linear one. In this case, it also makes sense to change the loss function you are using to perhaps something like Mean Squared Error (binary cross entropy, for example, won't work too well on negative numbers). There's nothing stopping you from using a deep neural network to perform regression rather than classification.  &lt;/p&gt;&#xA;" OwnerUserId="41306" LastActivityDate="2017-11-22T21:46:33.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25027" PostTypeId="2" ParentId="25024" CreationDate="2017-11-22T21:54:33.013" Score="31" Body="&lt;p&gt;This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the current gradient to decide the current step. However, when your gradient becomes and stays very close to zero, this will make the squares of the gradient become so low that they either have large rounding errors or are effectively zero, which can introduce instability (for instance a long-term stable gradient in one dimension makes a relatively small step from $10^{-10}$ to $10^{-5}$ due to changes in &lt;em&gt;other&lt;/em&gt; params), and the step size will start to jump around, before settling again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This actually makes Adam less stable and worse for your problem than more basic gradient descent, assuming you want to get as numerically close to zero loss as calculations allow for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice on deep learning problems, you don't get this close to convergence (and for some regularisation techniques such as early stopping, you don't want to anyway), so it is usually not a practical concern on the types of problem that Adam was designed for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can actually see this occurring for RMSProp in a comparison of different optimisers (RMSProp is the black line - watch the very last steps just as it reaches the target):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qAx2i.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qAx2i.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can make Adam more stable and able to get closer to true convergence by reducing the learning rate. E.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It will take longer to optimise. Using &lt;code&gt;lr=1e-5&lt;/code&gt; you need to train for 20,000+ iterations before you see the instability and the instability is less dramatic, values hover around $10^{-7}$. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-22T23:01:17.913" LastActivityDate="2017-11-22T23:01:17.913" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25028" PostTypeId="2" ParentId="25006" CreationDate="2017-11-22T22:15:10.370" Score="1" Body="&lt;p&gt;As a general rule, convolutional deep networks will perform better than SVM and Random Forest models on data that has hierarchical spatial/temporal structure, so for example sound, images, video.  Deep networks are very good at learning this spatial/temporal structure at different levels of granularity.  For classic datasets like Iris uses for teaching ML, however, I seriously doubt deep networks would perform better than SVM or Random Forest.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-22T22:15:10.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25029" PostTypeId="1" AcceptedAnswerId="28045" CreationDate="2017-11-22T22:24:28.950" Score="17" ViewCount="38551" Body="&lt;p&gt;I'm looking for a way to create a loss function that looks like this:&#xA;The function should then maximize for the reward. Is this possible to achieve in Keras?&#xA;Any suggestions how this can be achieved are highly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def special_loss_function(y_true, y_pred, reward_if_correct, punishment_if_false):&#xA;    loss = if binary classification is correct apply reward for that training item in accordance with the weight&#xA;           if binary classification is wrong, apply punishment for that training item in accordance with the weight&#xA;                    )&#xA;    return K.mean(loss, axis=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The approach I've been looking at for my example is to pass in the weights along with y_true and then cut the tensor into two, separating out the weights and the y_true as seen below. Would an approach like this be possible at all or would this interfere with the normalization process etc?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def decompose_y_true(y_true_and_weights):&#xA;    y_true = y_true_and_weights[:,1]&#xA;    weights= y_true_and_weights[:,1:]&#xA;    return y_true, weights&#xA;&#xA;&#xA;def custom_loss(y_true_and_weights, y_pred):&#xA;    y_true, y_weights = decompose_y_true(y_true_and_weights)&#xA;     loss = # some loss operation&#xA;    return K.mean(loss, axis=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Much more elegant would be if I could pass in my weights over the sample_weights parameter in the fit function, but it seems there are some limits what shape those weights can have, and also there's no way to retrieve them within the loss function as far as I can tell. Or is there any way to pass this into the loss function somehow so I can operate on them from there?&lt;/p&gt;&#xA;" OwnerUserId="32860" LastEditorUserId="32860" LastEditDate="2017-11-26T16:08:19.987" LastActivityDate="2018-02-20T15:51:40.433" Title="Custom loss function with additional parameter in Keras" Tags="|tensorflow|keras|loss-function|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25030" PostTypeId="2" ParentId="11134" CreationDate="2017-11-22T22:36:40.240" Score="4" Body="&lt;p&gt;You could use region embeddings. Rather than converting individual &amp;quot;tokens&amp;quot; to vectors you could use a strategy to convert regions of text to vectors. This approach is used &lt;a href=&quot;https://arxiv.org/abs/1504.01255&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;If you're not limited to CNN, you could use a hierarchical attention models such as &lt;a href=&quot;https://www.cs.cmu.edu/%7Ediyiy/docs/naacl16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt; where you have a pipeline of this kind:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;word vectors (combined into) sentence vectors (combined into) final document vector&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Note that, with this method, you will still have to convert all the word vectors to embeddings, but not all at once.&lt;/p&gt;&#xA;&lt;p&gt;To handle documents of different lengths, padding/cutting is the only solution so far.&lt;/p&gt;&#xA;&lt;p&gt;Finally, to increase speed, you could try to reduce the dimension of the text by only including important sections (maybe only the beginning of the document is sufficient to have good classification accuracy)&lt;/p&gt;&#xA;" OwnerUserId="13603" LastEditorUserId="98307" LastEditDate="2020-08-06T11:04:12.520" LastActivityDate="2020-08-06T11:04:12.520" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25031" PostTypeId="2" ParentId="24966" CreationDate="2017-11-22T22:40:08.047" Score="1" Body="&lt;p&gt;In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;correlated variables&lt;/li&gt;&#xA;&lt;li&gt;nonlinearity&lt;/li&gt;&#xA;&lt;li&gt;interactions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're worried that you've overfit your model to the test set, you'll need to find a new test set!&lt;/p&gt;&#xA;" OwnerUserId="10975" LastActivityDate="2017-11-22T22:40:08.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25032" PostTypeId="1" AcceptedAnswerId="25033" CreationDate="2017-11-23T02:20:39.670" Score="0" ViewCount="47" Body="&lt;p&gt;from &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/model.html#the-structure-score&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost tutorial&lt;/a&gt;, it described:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In this equation &lt;span class=&quot;math-container&quot;&gt;$w_j$&lt;/span&gt; are independent with respect to each other, the form &lt;span class=&quot;math-container&quot;&gt;$G_j w_j + \frac{1}{2}(H_j+λ)w_j^2$&lt;/span&gt; is quadratic and the best &lt;span class=&quot;math-container&quot;&gt;$w_j$&lt;/span&gt; for a given structure &lt;span class=&quot;math-container&quot;&gt;$q(x)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;and the best objective reduction we can get is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w^∗_j = \frac{−G_j}{H_j+λ}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$obj^∗=\frac{−1}{2}\sum_{j=1}^T\frac{G^2_j}{H_j+λ} + γT$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So my confusion is:&lt;/p&gt;&#xA;&lt;p&gt;if we would like to minimize the &lt;span class=&quot;math-container&quot;&gt;$obj$&lt;/span&gt; function, clearly, the best way is to set:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$G_j w_j + \frac{1}{2}(H_j + λ) w_j^2 = 0$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;thus we would have the best &lt;span class=&quot;math-container&quot;&gt;$w_j^*$&lt;/span&gt; is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w_j^* = \frac{-2G_j}{H_j + λ}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;which is different with the official explanation.&lt;/p&gt;&#xA;&lt;p&gt;anyone please help to point out where am I wrong?&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-23T02:28:27.540" Title="The best w_j confusion in xgboost" Tags="|machine-learning|xgboost|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25033" PostTypeId="2" ParentId="25032" CreationDate="2017-11-23T02:28:27.540" Score="1" Body="&lt;p&gt;Usually to find an optimum you set the &lt;em&gt;derivative&lt;/em&gt; of the function equal to 0. In your case that gives&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \frac{dG_j}{dw_J} = G_j + (H_j+\lambda)w_j = 0 $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;leading to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ w_j = \frac{-G_j}{H_j+\lambda}$$&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-23T02:28:27.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25034" PostTypeId="2" ParentId="25002" CreationDate="2017-11-23T02:51:23.877" Score="0" Body="&lt;p&gt;I've modified the code, have a look:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#install packages&#xA;library(DiceKriging)&#xA;library(DiceOptim)&#xA;library(lhs)&#xA;&#xA;nsteps &amp;lt;- 10&#xA;n &amp;lt;- 10&#xA;d &amp;lt;- 6&#xA;lower &amp;lt;- c(-2,-2,-2,0,0,0)&#xA;upper &amp;lt;- c(5,5,5,1,10,10)&#xA;x &amp;lt;- randomLHS(n,d)&#xA;&#xA;colnames(x)=c(&quot;x1&quot;,&quot;x2&quot;,&quot;x3&quot;,&quot;x4&quot;,&quot;x5&quot;,&quot;x6&quot;)&#xA;&#xA;Temp=c(1:n)&#xA;&#xA;&#xA;mf2 &amp;lt;- km(~ 1+x1+sin(x2)+x3+I(x4^-1)+x5+x6,  &#xA;          design = X, &#xA;          response=Temp,&#xA;          covtype=&quot;matern5_2&quot;,&#xA;          nugget = NULL, nugget.estim=FALSE,&#xA;          multistart = 5,&#xA;          control=list(maxit=500,trace=T,popsize=50))&#xA;&#xA;f1 &amp;lt;- function(x) {  &#xA;  x1 &amp;lt;- x[1]&#xA;  x2 &amp;lt;- x[2]&#xA;  x3 &amp;lt;- x[3]&#xA;  x4 &amp;lt;- x[4]&#xA;  x5 &amp;lt;- x[5]&#xA;  x6 &amp;lt;- x[6]&#xA;  L=-111.563*x1-29.81*sin(x2)+35.202*x3-.00012485/x4-1.045997*x5+11.6344*x6+13.1035&#xA;  return(L)&#xA;}&#xA;&#xA;oEGO &amp;lt;- EGO.nsteps(mf2,f1,nsteps,lower,upper)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you are getting some result but I'm not sure if this is what you are looking for. Let me know if you need any clarifications. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-23T02:51:23.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25035" PostTypeId="2" ParentId="25004" CreationDate="2017-11-23T04:43:09.143" Score="0" Body="&lt;p&gt;As I cannot comment @trollster's good response: for appending two bag-of-words or custom features extracted from the text you can use &lt;code&gt;FeatureUnion&lt;/code&gt; in scikit-learn. They provide a good example in their doc : &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html#sphx-glr-auto-examples-hetero-feature-union-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html#sphx-glr-auto-examples-hetero-feature-union-py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, their example treat a quite similar problem to yours (20 newsgroup dataset which consists in blog posts). As @trollster said, I would not separate subject and content as they might be highly related and contribute equally to the classification.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-11-23T04:43:09.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25036" PostTypeId="1" AcceptedAnswerId="25040" CreationDate="2017-11-23T06:10:38.783" Score="-1" ViewCount="416" Body="&lt;p&gt;What are some standard process models which people or companies use to solve their problems utilizing data? I'm familiar with current process models such as CRISP-DM and SEMMA, but I wondered if there are any other approaches or guidelines?&lt;/p&gt;&#xA;" OwnerUserId="6656" LastEditorUserId="6656" LastEditDate="2021-10-09T12:56:43.547" LastActivityDate="2021-10-09T12:56:43.547" Title="What are some popular process models for solving problems utilizing?" Tags="|data-mining|data-science-model|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-24T12:56:50.400" ContentLicense="CC BY-SA 4.0" />
  <row Id="25037" PostTypeId="1" AcceptedAnswerId="25039" CreationDate="2017-11-23T06:12:08.190" Score="0" ViewCount="516" Body="&lt;pre&gt;&lt;code&gt;pmv1=function(H,pa,Ta,fcl,Tr,Iclo,M){&#xA;&#xA;Tcl = (35.7 - 0.0275*H + 0.155*Iclo* (H - 0.31*(57.4 - 0.07*H - pa) - &#xA;      0.42*(H - 58) - 0.0017*M*(58.7 - pa) - 0.0014*M*(34 - Ta)))&#xA;&#xA;hc=(2.4*(Tcl - Ta))&#xA;&#xA;dT= (Tr - 22)   &#xA;&#xA;PMV= (4 + (0.303 *exp(-.036*H) + 0.0275)* (6.57 + 0.46*H +.31*pa + 0.0017*H*pa + &#xA;      0.0014*H*Ta - 4.13 *fcl *(1 + 0.01*dT)*(Tcl - Tr)-hc*fcl *(Tcl - Ta)))&#xA;&#xA;&#xA;  return((PMV-min(PMV))*3/(max(PMV)-min(PMV))-0)&#xA;&#xA;}&#xA;&#xA;&#xA;----------&#xA;d=7&#xA;&#xA;n &amp;lt;- 250&#xA;&#xA;set.seed(0)&#xA;&#xA;X1 &amp;lt;- optimumLHS(n, d)&#xA;&#xA;X1 &amp;lt;- data.frame(X1)&#xA;&#xA;colnames(X1) &amp;lt;- c(&quot;H&quot;,&quot;pa&quot;,&quot;Ta&quot;,&quot;fcl&quot;,&quot;Tr&quot;,&quot;Iclo&quot;,&quot;M&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;apply(X1, 1,pmv1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error in FUN(newX[, i], ...) :    argument &quot;Iclo&quot; is missing, with no&#xA;  default&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29967" LastEditorUserId="40853" LastEditDate="2017-11-24T01:03:05.587" LastActivityDate="2018-01-23T02:09:00.200" Title="Error in FUN(newX[, i], ...) : argument &quot;Iclo&quot; is missing, with no default" Tags="|r|data-mining|data|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25038" PostTypeId="1" CreationDate="2017-11-23T06:37:31.493" Score="2" ViewCount="1368" Body="&lt;p&gt;I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Included python code to reproduce the result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import spacy&#xA;nlp = spacy.load('en')&#xA;&#xA;vocab = nlp.vocab&#xA;words = np.array([x.orth_.encode('utf8') for x in vocab])&#xA;&#xA;hist1 = []&#xA;n = 1000&#xA;num_words = 1&#xA;for _ in range(n):&#xA;    x,y = choice(words, size=(2,num_words))&#xA;    x = nlp(&quot; &quot;.join([u.decode('utf8') for u in x]))&#xA;    y = nlp(&quot; &quot;.join([u.decode('utf8') for u in y]))&#xA;    s = x.similarity(y)&#xA;    hist1.append(s)&#xA;&#xA;hist10 = []&#xA;n = 1000&#xA;num_words = 10&#xA;for _ in range(n):&#xA;    x,y = choice(words, size=(2,num_words))&#xA;    x = nlp(&quot; &quot;.join([u.decode('utf8') for u in x]))&#xA;    y = nlp(&quot; &quot;.join([u.decode('utf8') for u in y]))&#xA;    s = x.similarity(y,)&#xA;    hist10.append(s)&#xA;&#xA;plt.hist([hist1,hist10], label=[1,10])&#xA;plt.legend()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DuUKb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DuUKb.png&quot; alt=&quot;Distribution of cosine similarity&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="42270" LastEditorUserId="381" LastEditDate="2017-11-23T08:18:32.680" LastActivityDate="2018-03-23T10:51:10.513" Title="Cosine similarity of averaged random word vectors" Tags="|machine-learning|word-embeddings|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25039" PostTypeId="2" ParentId="25037" CreationDate="2017-11-23T06:45:16.297" Score="0" Body="&lt;p&gt;This is the answer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pmv1 = function(H,pa,Ta,fcl,Tr,Iclo,M){&#xA;  Tcl = (35.7 - 0.0275*H + 0.155*Iclo * &#xA;           (H - 0.31*(57.4 - 0.07*H - pa) - &#xA;              0.42*(H - 58) - 0.0017*M*(58.7 - pa) - &#xA;              0.0014*M*(34 - Ta)))&#xA;&#xA;  hc=(2.4*(Tcl - Ta))&#xA;  dT= (Tr - 22)&#xA;&#xA;  PMV= (4 + (0.303 *exp(-.036*H) + 0.0275)* &#xA;          (6.57 + 0.46*H +.31*pa + 0.0017*H*pa + 0.0014*H*Ta - 4.13 *fcl &#xA;           *((1 + 0.01*dT) * (Tcl - Tr))-fcl *(Tcl - Ta)))&#xA;  return((PMV-min(PMV))*3/(max(PMV)-min(PMV))-0)&#xA;}&#xA;&#xA;d &amp;lt;- 7&#xA;&#xA;n &amp;lt;- 250&#xA;&#xA;set.seed(0)&#xA;&#xA;X1 &amp;lt;- optimumLHS(n, d)&#xA;X1 &amp;lt;- data.frame(X1)&#xA;&#xA;#colnames(X1) &amp;lt;- c(&quot;H&quot;,&quot;pa&quot;,&quot;Ta&quot;,&quot;fcl&quot;,&quot;Tr&quot;,&quot;Iclo&quot;,&quot;M&quot;)&#xA;new &amp;lt;- pmv1(X1[,1],X1[,2],X1[,3],X1[,4],X1[,5],X1[,6],X1[,7])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This would do the job.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-23T06:45:16.297" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25040" PostTypeId="2" ParentId="25036" CreationDate="2017-11-23T07:01:52.723" Score="0" Body="&lt;p&gt;As you know about CRIPS-DM and SEMMA, you can loo into this too &lt;a href=&quot;http://www2.cs.uregina.ca/~dbd/cs831/notes/kdd/1_kdd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;KDD&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9LiJ8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9LiJ8.png&quot; alt=&quot;KDD&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-23T07:01:52.723" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25043" PostTypeId="2" ParentId="24878" CreationDate="2017-11-23T09:04:28.073" Score="3" Body="&lt;p&gt;There is even more specific research on this topic:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;The trained generator is capable of producing sentences with certain level of grammar and logic.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Xuerong Xiao, &quot;Text Generation using Generative Adversarial Training&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question relates to this one: &#xA;&lt;a href=&quot;https://linguistics.stackexchange.com/questions/26448/how-to-translate-pelevins-creative-unit-idea-to-a-scientific-problem&quot;&gt;https://linguistics.stackexchange.com/questions/26448/how-to-translate-pelevins-creative-unit-idea-to-a-scientific-problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33183" LastEditorUserId="33183" LastEditDate="2017-11-23T09:21:15.520" LastActivityDate="2017-11-23T09:21:15.520" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25044" PostTypeId="2" ParentId="25038" CreationDate="2017-11-23T09:09:54.627" Score="1" Body="&lt;p&gt;Nothing too surprising here. As you sample more and more words, the sample mean is a better and better estimator of the population mean. This is called the law of large numbers.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastActivityDate="2017-11-23T09:09:54.627" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25045" PostTypeId="2" ParentId="24939" CreationDate="2017-11-23T09:29:20.513" Score="1" Body="&lt;p&gt;It seems that Watson doesn't allow some accounts to access some regions. Just choose another region, like &quot;US South&quot;, when creating the resource, and the problem will be gone.&lt;/p&gt;&#xA;" OwnerDisplayName="user41366" LastActivityDate="2017-11-23T09:29:20.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25046" PostTypeId="1" AcceptedAnswerId="25047" CreationDate="2017-11-23T10:16:59.363" Score="4" ViewCount="5948" Body="&lt;p&gt;I'm wondering whether there is a difference between Linear SVM and SVM with a linear kernel. Or is a linear SVM just a SVM with a linear kernel?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, what is the difference between the two variables &lt;code&gt;linear_svm&lt;/code&gt; and &lt;code&gt;linear_kernel&lt;/code&gt; in the following code. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;linear_svm = svm.LinearSVC(C=1).fit(X_train, y_train)&#xA;linear_kernel_svm=svm.SVC(kernel='linear', C=1).fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42275" LastEditorUserId="29169" LastEditDate="2019-05-28T23:19:35.533" LastActivityDate="2019-05-28T23:19:35.533" Title="What is the difference between Linear SVM and SVM with linear kernel?" Tags="|python|scikit-learn|svm|libsvm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25047" PostTypeId="2" ParentId="25046" CreationDate="2017-11-23T10:22:39.483" Score="3" Body="&lt;p&gt;As you can read in the documentation (&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&lt;/a&gt;) it's conceptually similar, but implemented in another way to add flexibility:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So if you only want to use linear support vectors use this one because it scales better and you get more freedom with your loss functions, but if you want to more easily try different kernels in a grid search use the more generic one.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="75002" LastEditDate="2019-05-28T20:37:29.277" LastActivityDate="2019-05-28T20:37:29.277" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="25050" PostTypeId="1" CreationDate="2017-11-23T11:43:06.533" Score="4" ViewCount="1602" Body="&lt;p&gt;I have a very interesting question. It's one month ago Geoffrey Hinton release his CapsNet paper. It's a complete new way for visual computing in the neural network stack. But my question is. Is it possible to use the CapsNet as a GAN.&#xA;I only understand the concept of the CapsNet but not technical limitations. So could this be possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sounds for me for a nice field to research:)  &lt;/p&gt;&#xA;" OwnerUserId="29064" LastActivityDate="2018-03-30T23:49:46.400" Title="CapsNet Generative adversarial network" Tags="|gan|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25051" PostTypeId="1" CreationDate="2017-11-23T14:17:54.573" Score="2" ViewCount="801" Body="&lt;p&gt;I am searching for &lt;strong&gt;&lt;em&gt;scientific work on skip-connections&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everybody is talking about improving gradient flow through the Network, which feels like it makes sense. But &lt;strong&gt;&lt;em&gt;I would love to understand and read what that means when it comes down to the math&lt;/em&gt;&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than the resnet &lt;a href=&quot;https://arxiv.org/pdf/1512.03385.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; I haven't found anything in depth about those magical skip-connections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for any hints, papers or equations.&lt;/p&gt;&#xA;" OwnerUserId="40552" LastEditorUserId="29575" LastEditDate="2017-11-25T13:30:48.573" LastActivityDate="2017-12-25T14:28:21.617" Title="Scientific explanation of Skip Connections" Tags="|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25052" PostTypeId="1" CreationDate="2017-11-23T14:20:19.310" Score="1" ViewCount="35" Body="&lt;p&gt;I want to perform &quot;reliable rule learning&quot;, i.e. mining a set of rules with a very low number of false negatives. I recently read the paper &quot;Reliable agnostic learning&quot; by Kalai et al. (&lt;a href=&quot;https://doi.org/10.1016/j.jcss.2011.12.026&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://doi.org/10.1016/j.jcss.2011.12.026&lt;/a&gt;) and they basically describe what I want: Rules are determined to reliably classify data points, and the reliability is partly reached by allowing &quot;I don't know&quot; as an additional answer. Sadly, their paper is purely theoretical and I could not find a corresponding implementation. Is there an implementation of reliable rule learning or a similar algorithm? At best, open source and Java-based?&lt;/p&gt;&#xA;" OwnerUserId="42291" LastActivityDate="2020-10-23T14:07:27.397" Title="Implementation of reliable rule learning" Tags="|machine-learning|data-mining|algorithms|software-recommendation|error-handling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25053" PostTypeId="1" CreationDate="2017-11-23T14:40:25.603" Score="33" ViewCount="60882" Body="&lt;p&gt;I have two sentences, S1 and S2, both which have a word count (usually) below 15.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the most practically useful and successful (machine learning) algorithms, which are possibly easy to implement (neural network is ok, unless the architecture is as complicated as Google Inception etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an algorithm that will work fine without putting too much time into it. Are there any algorithms you've found successful and easy to use? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can, but does not have to fall into the category of clustering. My background is from machine learning, so any suggestions are welcome :)&lt;/p&gt;&#xA;" OwnerUserId="31810" LastActivityDate="2021-08-05T19:44:18.673" Title="Best practical algorithm for sentence similarity" Tags="|nlp|clustering|word2vec|similarity|" AnswerCount="8" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25054" PostTypeId="2" ParentId="25053" CreationDate="2017-11-23T15:09:37.893" Score="18" Body="&lt;p&gt;&lt;a href=&quot;http://ws4jdemo.appspot.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cosine Similarity for Vector Space&lt;/a&gt; could be you answer.&lt;/p&gt;&#xA;&lt;p&gt;Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity?&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;This is a tree&amp;quot;,&#xA;&amp;quot;This is not a tree&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;If you want to check the semantic meaning of the sentence you will need a wordvector dataset. With the wordvector dataset you will able to check the relationship between words. Example: (King - Man + woman = Queen)&lt;/p&gt;&#xA;&lt;p&gt;Siraj Raval has a good &lt;a href=&quot;https://github.com/llSourcell/word_vectors_game_of_thrones-LIVE&quot; rel=&quot;nofollow noreferrer&quot;&gt;python notebook&lt;/a&gt; for creating wordvector datasets.&lt;/p&gt;&#xA;" OwnerUserId="29064" LastEditorUserId="98307" LastEditDate="2020-08-06T13:36:53.207" LastActivityDate="2020-08-06T13:36:53.207" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25055" PostTypeId="2" ParentId="25053" CreationDate="2017-11-23T15:15:12.147" Score="9" Body="&lt;p&gt;One approach you could try is averaging word vectors generated by word embedding algorithms (word2vec, glove, etc). These algorithms create a vector for each word and the cosine similarity among them represents semantic similarity among the words. In the case of the average vectors among the sentences. A good starting point for knowing more about these methods is this paper: &lt;a href=&quot;http://white.ucc.asn.au/publications/White2015SentVecMeaning.pdf&quot; rel=&quot;noreferrer&quot;&gt;How Well Sentence Embeddings Capture Meaning&lt;/a&gt;. It discusses some sentence embedding methods. I also suggest you look into &lt;a href=&quot;https://arxiv.org/pdf/1703.02507.pdf&quot; rel=&quot;noreferrer&quot;&gt;Unsupervised Learning of Sentence Embeddings&#xA;using Compositional n-Gram Features&lt;/a&gt; the authors claim their approach beat state of the art methods. Also they provide the code and some usage instructions in this &lt;a href=&quot;https://github.com/epfml/sent2vec&quot; rel=&quot;noreferrer&quot;&gt;github repo&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-23T15:15:12.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25056" PostTypeId="1" CreationDate="2017-11-23T15:20:33.023" Score="0" ViewCount="67" Body="&lt;p&gt;For combining the results of different measurements, I want to calculate their arithmetic mean. Imagine the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id;attribute;time&#xA;A;AB;1&#xA;B;AB;2&#xA;C;AB;3&#xA;B;AB;4&#xA;B;AB;5&#xA;A;AB;6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to combine all results with id &lt;code&gt;A&lt;/code&gt; and get their mean. The expected dataset looks like the followind. Be aware, that the value of the in between attribute, is not important, I've chosen the first entry.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id attribute     time&#xA;A  AB            3.500000&#xA;B  AB            3.666667&#xA;C  AB            3.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The code below does the right thing, but I'm wondering if there is a better solution for this kind of problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Read in the data&#xA;data &amp;lt;- data.frame(read_delim(&quot;~/...&quot;, &quot;;&quot;, escape_double = FALSE, trim_ws = TRUE))&#xA;# Visualize head of the data&#xA;head(data)    &#xA;&#xA;# Define function to calculate the mean&#xA;calculate_mean = function(data){&#xA;&#xA;  # Prepare empty result data set&#xA;  result = data.frame(id=character(), &#xA;                      attribute=character(),&#xA;                      time=integer())&#xA;&#xA;  # Iterate over specified id &amp;lt;-- Change this one to the one you want to iterate over&#xA;  iterator = unique(data$id)&#xA;&#xA;  # Fill result dataset with mean&#xA;  for(i in iterator){&#xA;    subset = subset(data, data$id == i)&#xA;    arithmetic_mean = mean(subset$time)&#xA;    result = rbind(result, data.frame(subset[1,]$id, &#xA;                                      subset[1,]$attribute, &#xA;                                      arithmetic_mean))&#xA;  }&#xA;&#xA;  # Set colnames properly&#xA;  colnames(result) = colnames(data)&#xA;&#xA;  return(result)&#xA;}    &#xA;&#xA;# Call function&#xA;result = calculate_mean(data)    &#xA;&#xA;# Display the result&#xA;result&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do you have an idea? Thanks for your support.&lt;/p&gt;&#xA;" OwnerUserId="15946" LastEditorUserId="15946" LastEditDate="2017-11-23T15:26:32.850" LastActivityDate="2017-11-25T08:36:24.380" Title="Combining results via arithmetic mean" Tags="|r|dataframe|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25057" PostTypeId="2" ParentId="25056" CreationDate="2017-11-23T15:28:06.027" Score="1" Body="&lt;p&gt;What your trying to do is call a group-by you want to group by the id column and the find the mean of column time in each group. I suggest you look into the &lt;a href=&quot;http://stat.ethz.ch/R-manual/R-patched/library/stats/html/aggregate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;aggregate function&lt;/a&gt;. This function does the following: Splits the data into subsets, computes summary statistics for each, and returns the result in a convenient form.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-23T15:28:06.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25060" PostTypeId="2" ParentId="14805" CreationDate="2017-11-23T18:39:56.160" Score="3" Body="&lt;p&gt;I you want some kind of data-sets like Google spell checking data I suggest you look into the &lt;a href=&quot;http://romang.home.amu.edu.pl/wiked/wiked.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The WikEd Error Corpus&lt;/a&gt; dataset. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types, this edits include: spelling error corrections, grammatical error corrections, stylistic changes. All these from the Wikipedia correction history. The owners (authors) of the data-set describe the data mining process in this &lt;a href=&quot;https://pdfs.semanticscholar.org/51e5/8ac760a466dda48dfe470b44688277fe2d89.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. Also check this question in &lt;a href=&quot;https://www.quora.com/Where-can-I-find-data-sets-for-learning-autocorrection-or-spelling-correction&quot; rel=&quot;nofollow noreferrer&quot;&gt;quora&lt;/a&gt; it contains links to various data-sets with spelling errors. Finally this &lt;a href=&quot;http://www.dcs.bbk.ac.uk/~ROGER/corpora.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;page&lt;/a&gt; can also be useful.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-23T18:39:56.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25061" PostTypeId="1" CreationDate="2017-11-23T18:54:28.107" Score="1" ViewCount="14825" Body="&lt;p&gt;I've been using Python for quite some time now, scripting command line tools, etc.  I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files.  Particularly, determining appropriate specs for a system, threading/multiprocessing, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm finding that it's taking an excessive amount of time to handle basic tasks; I've worked with python reading and processing large files (i.e. Log files), and it seems to run a lot faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example, I'm downloaded a json file from catalog.data.gov for traffic violations.  The file is 758Mb in size and it takes a long time to do something very simple.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import ijson&#xA;&amp;gt;&amp;gt;&amp;gt; &#xA;&amp;gt;&amp;gt;&amp;gt; filename = 'rows.json'&#xA;&amp;gt;&amp;gt;&amp;gt; with open(filename, 'r') as f:&#xA;...     objects = ijson.items(f, 'meta.view.columns.item')&#xA;...     columns = list(objects)&#xA;...     print columns[0]&#xA;...     f.close()&#xA;... &#xA;&#xA;&#xA;{u'name': u'sid', u'format': {}, u'dataTypeName': u'meta_data', u'fieldName': u':sid', u'renderTypeName': u'meta_data', u'position': 0, u'id': -1, u'flags': [u'hidden']}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've worked with json before, but this seems to be taking a long time for something so simple.  I'm did this on a laptop with 4GB of RAM (~2G free).&lt;/p&gt;&#xA;" OwnerUserId="42299" LastActivityDate="2019-07-31T05:48:13.650" Title="How to Process Large JSON Files with Python" Tags="|python|data|json|" AnswerCount="0" CommentCount="3" ClosedDate="2020-03-09T19:19:20.353" ContentLicense="CC BY-SA 3.0" />
  <row Id="25062" PostTypeId="1" AcceptedAnswerId="25064" CreationDate="2017-11-23T19:35:22.613" Score="3" ViewCount="189" Body="&lt;p&gt;I have trained a deep learning model for a face recognition application. The model has been trained on the existing subjects (or call it classes) in the gallery. Now consider the real world problem, where the gallery size is dynamic as new subjects are to be enrolled, how can deep learning solve those type of problems? I have tried transfer learning, are there any other techniques?&lt;/p&gt;&#xA;" OwnerUserId="42248" LastActivityDate="2017-11-24T08:20:35.077" Title="How can we scale up the number of classes for deep learning after training a model?" Tags="|machine-learning|neural-network|classification|deep-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25063" PostTypeId="1" CreationDate="2017-11-23T21:05:07.837" Score="2" ViewCount="290" Body="&lt;p&gt;I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the &lt;code&gt;fit_generator()&lt;/code&gt; method in Keras, but these are mostly tailored to image classification and involves really minimal preprocessing. I've also looked into using the newer &lt;code&gt;HDF5Matrix&lt;/code&gt; class in Keras, and have continued to run out of memory with batch sizes larger than 20. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other useful data reading/streaming tools in TensorFlow that could be used to solve this problem? Are there examples for how to do standard NLP preprocessing in such a pipeline in a memory-efficient way before feeding to a model? Any references would be appreciated. Thanks for reading.&lt;/p&gt;&#xA;" OwnerUserId="4897" LastEditorUserId="4897" LastEditDate="2017-11-23T21:10:49.183" LastActivityDate="2017-11-23T21:10:49.183" Title="Keras or TensorFlow Examples for Working with Large Text Datasets (~10M Sentences)" Tags="|machine-learning|deep-learning|dataset|tensorflow|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25064" PostTypeId="2" ParentId="25062" CreationDate="2017-11-23T21:52:11.723" Score="4" Body="&lt;p&gt;Face recognition from images is still an open area of research. Many different techniques have been tried.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is not to directly learn the identities of the people as classes, but to learn a multi-dimensional embedding for images of faces (e.g. 128 dimension vector) and train the network to make this embedding value closer for two different images of the same person than for two images of different people*. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the network to generate feature vectors for each person who is registered, and store those values. Later when you want to identify someone, find the closest one (according to norm of the difference) and if the distance is less than a certain cutoff you can consider it to be a match.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To perform the training for the embeddings with this goal in mind, you can use a loss function called triplet loss, which is trained on 3 forward passes at a time. The network calculates feature vectors for an anchor image, a positive match (i.e. another image of the same face, so you need at least 2 for each person, and they should be taken from different contexts) and a negative match. Then it measures whether the distance between features from same person is less than distance between different people. The loss function to express that looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{max}(0, |A - P|^2 - |A - N|^2 + \alpha)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . . where $A$ is feature vector for the anchor image, $P$ for positive match image, and $N$ for negative match image. The parameter $\alpha$ is a hyper-parameter which creates a &quot;margin&quot; that you reward the model for maintaining between different identities. The idea being that if an image of different person is clearly further away in the feature space than a cutoff, the loss on that training example is zero. The model does not get rewarded for very large differences, just having &quot;consistently enough distance&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tricky thing with triplet loss is picking which samples to use for training. Purely random samples could make the network's task too easy. You need it to focus a bit on the more difficult cases to prevent mistaken identity. One way of doing this is to pick preferentially from closer faces, e.g. using k-means clusters and picking negative cases within own cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model can be trained on large numbers of faces, not necessarily the ones you will later want to classify. The point is to learn a feature vector that emphasises important differences between faces (and ideally ignores other variations such as lighting, background, pose). It's possible to start with any image classifier and adapt it for this task, including your original face classifier (just remove the final classifier layer and work with one of the later fully-connected layers as your output).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How can we scale up the number of classes for deep learning after training a model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This approach may not work well in all cases covered by your question title. I have recommended it for face recognition based on a &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coursera course on CNNs&lt;/a&gt; where a few lectures and coding assignment are about face recognition specifically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some cases you may be better off re-building and re-training the model with new training set data, augmented by examples of your new class - you might start by fine-tuning only the last few layers.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* In fact this will be the case in many CNN classifiers, but is more likely to be reliably linked to a person's identity if you have trained the network explicitly for distinguishing between identities in general (as opposed to just classify specific identities). The triplet loss approach is one way to train for useful representations, and makes the goal of disambiguating identity explicit.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-24T08:20:35.077" LastActivityDate="2017-11-24T08:20:35.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25067" PostTypeId="1" CreationDate="2017-11-24T06:10:53.370" Score="1" ViewCount="69" Body="&lt;p&gt;We are trying to rank the products on the basis of score, score is calculated by analyzing the good or bad feedback about the product. After a lot of research and analysis we came up with this formula to compare and rank the products&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;product score = Internal Probability + Positive Contribution - exp(Negative Contribution) - Sentiment score&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Internal Probability = ( # of Positive for product / Total feedback of a product )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Positive Contribution = (# of Positive for product / # of Positive for all Products )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negative Contribution = (# of Negative for product / # of Negative for all Products )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sentiment score = Average(Negative Sentiment for product)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Range of Sentiment is from [0-1] where 1 shows very negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For large amount of data the average sentiment of all the product converges to the mean value, so it doesn't adds any value in the analysis. &#xA;Any suggestions about we could improve this formula or use sentiment score to be more meaningful? &lt;/p&gt;&#xA;" OwnerUserId="42317" LastActivityDate="2017-11-24T06:10:53.370" Title="Feedback Analysis" Tags="|machine-learning|data-mining|nlp|bigdata|sentiment-analysis|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25068" PostTypeId="1" CreationDate="2017-11-24T08:15:52.753" Score="2" ViewCount="846" Body="&lt;p&gt;I am currently building a (simple sequential) fully convolutional network for an object recognition task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Designing the architecture essentially amounts to choosing kernel size, stride and unit number for each layer.&#xA;While doing this, I have to keep track of several properties of my layers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;parameter count&lt;/li&gt;&#xA;&lt;li&gt;field of view of each kernel, wrt to the input/data&lt;/li&gt;&#xA;&lt;li&gt;a margin in the last layer that I need to account for when creating the label images&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've written a small script that computes these quantities for different CNN configurations. It's not very complicated, but…&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm under the impression that everyone using CNNs has to do this, so there should be a (graphical) tool for this. It could also be used to draw the fancy architecture diagrams. Because it works on the same input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there such a tool? How do you compute these quantities for your models?&lt;/p&gt;&#xA;" OwnerUserId="42322" LastEditorUserId="42322" LastEditDate="2017-11-24T08:26:26.193" LastActivityDate="2017-11-25T18:39:43.280" Title="Tool for designing CNN architectures" Tags="|tools|cnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25069" PostTypeId="2" ParentId="25056" CreationDate="2017-11-24T08:29:44.797" Score="2" Body="&lt;p&gt;Try it the &lt;a href=&quot;https://www.tidyverse.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;R tidyverse&lt;/a&gt; way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(dplyr)&#xA;dfr &amp;lt;- read.table(text = &quot;id;attribute;time&#xA;A;AB;1&#xA;B;AB;2&#xA;C;AB;3&#xA;B;AB;4&#xA;B;AB;5&#xA;A;AB;6&quot;, header=TRUE, sep=&quot;;&quot;)&#xA;&#xA;dfr %&amp;gt;% group_by(id, attribute) %&amp;gt;% summarize(mean=mean(time))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# A tibble: 3 x 3&#xA;# Groups:   id [?]&#xA;      id attribute  mean&#xA;  &amp;lt;fctr&amp;gt;    &amp;lt;fctr&amp;gt; &amp;lt;dbl&amp;gt;&#xA;1      A        AB 3.500&#xA;2      B        AB 3.667&#xA;3      C        AB 3.000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2017-11-24T08:29:44.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25071" PostTypeId="1" CreationDate="2017-11-24T08:49:58.677" Score="1" ViewCount="389" Body="&lt;p&gt;I have a fairly large dataset (about 40k rows, 40 columns) with many NA's (up to 40% of each variable). All of my work has been done in R so far, but I know R (which I'm told runs only on a single core) won't be able to impute the data with any type of respectable turn around time, especially on my laptop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I'm looking into more powerful cloud computing to help, I'm wondering if Python could offer better imputation times (presumably it can be run on more than one core).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also still deciding on the algorithm -- likely Cart, KNN, or Forest. It's survey data. &lt;/p&gt;&#xA;" OwnerUserId="42323" LastActivityDate="2020-08-11T14:52:51.297" Title="Which is faster for imputing: R or Python?" Tags="|python|r|data-imputation|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-28T01:50:10.533" ContentLicense="CC BY-SA 3.0" />
  <row Id="25072" PostTypeId="1" CreationDate="2017-11-24T08:52:04.300" Score="8" ViewCount="9381" Body="&lt;p&gt;I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training an LSTM feeding 3 steps back data points to predict the next 3 steps in the future. &#xA;The input data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = [[[A0, B0, C0],[A1, B1, C1],[A2, B2, C2]],[[ ...]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with dimensions: &lt;code&gt;(1000, 3, 3)&lt;/code&gt;. The output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = [[C3, C4, C5],[C4, C5, C6],...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with dimensions: &lt;code&gt;(1000, 3)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using a simple LSTM with 1 hidden layer (50 neurons).&#xA;I set up an LSTM with keras as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_features = 3&#xA;neurons = 50&#xA;ahead = 3&#xA;model = Sequential()&#xA;model.add(LSTM(input_dim=n_features, output_dim=neurons))&#xA;model.add(Dropout(.2))&#xA;model.add(Dense(input_dim=neurons, output_dim=ahead))&#xA;model.add(Activation('linear'))&#xA;model.compile(loss='mae', optimizer='adam')&#xA;model.fit(X, y, epochs=50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This model works fine. Now, I'd like to predict the values of B as well (using the same input). So I tried to reshape the output in a similar way as I did for the training that has multiple features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; y = [[[B3, C3],[B4, C4],[B5, C5]],[[ ...]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so that it has dimensions: &lt;code&gt;(1000, 3, 2)&lt;/code&gt;. However, this gives me an error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error when checking target: expected activation_5 to have 2 dimensions, &#xA;but got array with shape (1000, 3, 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess the structure of the network needs to change. I tried to modify &lt;code&gt;model.add(Dense(input_dim=neurons, output_dim=ahead))&lt;/code&gt; with no success. Should I reshape the &lt;code&gt;y&lt;/code&gt; differently? Is the structure of the network wrong?&lt;/p&gt;&#xA;" OwnerUserId="42325" LastEditorUserId="29575" LastEditDate="2018-10-31T02:44:48.543" LastActivityDate="2018-12-31T02:01:30.303" Title="Multiple output for multi step ahead prediction using LSTM with keras" Tags="|python|time-series|keras|lstm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="25073" PostTypeId="1" AcceptedAnswerId="25077" CreationDate="2017-11-24T10:15:36.180" Score="5" ViewCount="3741" Body="&lt;p&gt;l have a dataset of 200 examples with 10 classes. l would like to split the dataset into training set 50% and test set 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for each class, l have 20 examples. Hence, l would like to get for each class : 10 training examples and 10 test examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my classes :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classes=['BenchPress', 'ApplyLipstick', 'BabyCrawling', 'BandMarching', 'Archery', 'Basketball', 'ApplyEyeMakeup', 'BalanceBeam', 'BaseballPitch', 'BasketballDunk']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;l tried the following :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However it returns a 50% training set and 50 % test set, without respecting the proportion for each class (l would like to get 10 examples in test set and 10 examples in training set for each class).&#xA;Here is the resulted splitting : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1W20m.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1W20m.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42275" LastActivityDate="2018-12-17T10:19:38.147" Title="How can l get 50 % examples in training set and 50% in test set for each class when splitting data?" Tags="|python|scikit-learn|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25074" PostTypeId="1" CreationDate="2017-11-24T11:23:16.053" Score="3" ViewCount="338" Body="&lt;p&gt;I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read &lt;a href=&quot;http://bwlewis.github.io/covar/missing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; that pairwise deletion may not be the best way of dealing with this situation, but if I use listwise deletion I am afraid I'll end up with too little data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if I could estimate the missing values using least squares, and calculate the correlation using the now complete data set.&lt;/p&gt;&#xA;" OwnerUserId="42298" LastActivityDate="2018-03-30T21:35:24.177" Title="Correlation with missing values. Is least squares an acceptable option?" Tags="|linear-regression|correlation|missing-data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25075" PostTypeId="2" ParentId="25071" CreationDate="2017-11-24T12:00:46.083" Score="1" Body="&lt;p&gt;I recommend you to use Python. With sklearn, you have many ways to perform algorithm multicore on your data.&#xA;Another reason to use Python: If you want to perform distributed computing you have many modules like DASK which are very easy to use and schedule automatically Tasks on different Machines without spending hours in coding multithreading algorithm.&lt;/p&gt;&#xA;&lt;p&gt;40k rows, 40 columns are not a big problem for python, BUT I think &amp;quot;R&amp;quot; should be able to handle this, it's a very popular language for statistical algorithms. Maybe some settings are false...&lt;/p&gt;&#xA;&lt;p&gt;Short:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Install Python (or better Anaconda)&lt;/li&gt;&#xA;&lt;li&gt;on Sklearn-site you will find some &lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html#random-forests&quot; rel=&quot;nofollow noreferrer&quot;&gt;demo code&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Try it with your Dataset&lt;/li&gt;&#xA;&lt;li&gt;Maybe you could benchmark it :)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29064" LastEditorUserId="98307" LastEditDate="2020-08-11T14:52:51.297" LastActivityDate="2020-08-11T14:52:51.297" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25077" PostTypeId="2" ParentId="25073" CreationDate="2017-11-24T12:50:31.883" Score="5" Body="&lt;p&gt;sklearn version 0.17 onwards the &lt;code&gt;train_test_split&lt;/code&gt; should give you stratified results by using the stratify parameter.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42, stratify=true_label)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From the documentation about the parameter stratify:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;stratify: array-like or None (default is None) If not None, data is split in a stratified fashion, using this as the labels array. New in version 0.17: stratify splitting&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="8465" LastActivityDate="2017-11-24T12:50:31.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25080" PostTypeId="1" CreationDate="2017-11-24T15:59:50.337" Score="0" ViewCount="74" Body="&lt;p&gt;I want to evaluate my recommender system with the Top-N recommendation method and I have a problem. In some situations, e.g. N = 5, I don't have 5 items for listing and I cannot do that for evaluating. now, you suppose, I want to increase N from 5 to 10 or 15, I think I will have some problems with some active users.&#xA;Would you please give me a suggestion?!!!&lt;/p&gt;&#xA;" OwnerUserId="42335" LastEditorUserId="83275" LastEditDate="2021-03-15T06:00:20.263" LastActivityDate="2021-03-15T06:00:20.263" Title="How to develop Top-N recommendation for evaluating my system" Tags="|recommender-system|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25081" PostTypeId="1" AcceptedAnswerId="25085" CreationDate="2017-11-24T17:57:00.097" Score="2" ViewCount="2956" Body="&lt;p&gt;How do we find the gradient and the back propagation error if we had a bias which just added a scalar value after the convolution calculation?&lt;/p&gt;&#xA;" OwnerUserId="42341" LastActivityDate="2017-11-24T22:12:49.473" Title="How to update bias in CNN?" Tags="|machine-learning|deep-learning|backpropagation|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25082" PostTypeId="1" CreationDate="2017-11-24T19:03:06.123" Score="3" ViewCount="196" Body="&lt;p&gt;I am currently implementing a CNN to recognise the identity of people given a portrait picture of the person.&#xA;The objective is to maximise the clf.score function in sklearn, the database is composed of 144 images of 70 different individuals. The program will be tested on a database of images of the same 70 individuals, which I do not have access to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For most of the pictures, my facial recognition algorithm (openCV) will detect and crop out the background, leaving only the face, but in certain cases, it does not find a face.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What solution will return the best classification score?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Options I've considered are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Leaving the picture as is (Useless background info isn't removed)&lt;/li&gt;&#xA;&lt;li&gt;Cropping a predefined area (Could risk cropping out parts of the face)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42345" LastEditorUserId="29169" LastEditDate="2021-02-13T19:21:16.140" LastActivityDate="2021-02-13T19:21:16.140" Title="What to do when facial recognition fails to find a face?" Tags="|machine-learning|convolutional-neural-network|image-recognition|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25083" PostTypeId="1" CreationDate="2017-11-24T20:00:41.430" Score="6" ViewCount="609" Body="&lt;p&gt;I am incorporating Adversarial Training for Semantic Segmentation from &lt;a href=&quot;https://openreview.net/pdf?id=SJQO7UJCW&quot; rel=&quot;nofollow noreferrer&quot;&gt;Adversarial Learning for Semi-Supervised Semantic Segmentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is like this: &#xA;The discriminator takes as input a probability map (21x321x321) over 21 classes (PASCAL VOC dataset) and produces a confidence map of size 2x321x321. (a real/fake decision for each pixel). For an input image (3x321x321), the segmentation network (generator) produces 'fake' probability map (21x321x321). The 'real' probability map comes from the ground truth segmentation labels (21x321x321 using one-hot encoding). The Loss for both Generator and Discriminator are easier to understand through the code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With my current implementation, the discriminator loss quickly goes to 0, which is a failure more for GAN training(mentioned &lt;a href=&quot;https://github.com/soumith/ganhacks&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;).  I am new to pytorch (as well as to adversarial training). So, I am not sure if there's an issue with my network architecture, hyperparameters or simply my training scheme. I would really appreciate some pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how my training looks:  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1][0] LD: 1.402573823928833 LG: 3.0447137355804443&lt;br&gt;&#xA;  [1][1] LD: 0.8725658655166626 LG: 2.544170618057251&lt;br&gt;&#xA;  [1][2] LD: 0.6969347596168518 LG: 2.1177046298980713&lt;br&gt;&#xA;  [1][3] LD: 0.611475944519043 LG: 1.7778557538986206&lt;br&gt;&#xA;  [1][4] LD: 0.49319764971733093 LG: 2.1366050243377686&lt;br&gt;&#xA;  [1][5] LD: 0.30195319652557373 LG: 1.7873120307922363&lt;br&gt;&#xA;  [1][6] LD: 0.14412544667720795 LG: 1.045764684677124&lt;br&gt;&#xA;  [1][7] LD: 0.04816107824444771 LG: 1.5864180326461792&lt;br&gt;&#xA;  [1][8] LD: 0.012304163537919521 LG: 1.370680332183838&lt;br&gt;&#xA;  [1][9] LD: 0.0035684951581060886 LG: 1.3428194522857666&lt;br&gt;&#xA;  [1][10] LD: 0.0011156484251841903 LG: 1.145486831665039&lt;br&gt;&#xA;  [1][11] LD: 0.00045744137605652213 LG: 1.371126651763916&lt;br&gt;&#xA;  [1][12] LD: 0.0001588731538504362 LG: 1.378540277481079&lt;br&gt;&#xA;  [1][13] LD: 4.844377326662652e-05 LG: 1.504058837890625&lt;br&gt;&#xA;  [1][14] LD: 3.028669743798673e-05 LG: 1.5484118461608887&lt;br&gt;&#xA;  [1][15] LD: 1.5183023606368806e-05 LG: 1.584553837776184&lt;br&gt;&#xA;  [1][16] LD: 2.0302868506405503e-05 LG: 1.4818311929702759&lt;br&gt;&#xA;  [1][17] LD: 1.0679158549464773e-05 LG: 1.2976796627044678&lt;br&gt;&#xA;  [1][18] LD: 1.5313835319830105e-06 LG: 1.2631664276123047&lt;br&gt;&#xA;  [1][19] LD: 4.273606009519426e-06 LG: 1.770961046218872&lt;br&gt;&#xA;  [1][20] LD: 1.1575384633033536e-06 LG: 1.5112217664718628&lt;br&gt;&#xA;  [1][21] LD: 2.138318961897312e-07 LG: 1.2034248113632202&lt;br&gt;&#xA;  [1][22] LD: 1.0100056897499599e-06 LG: 1.581740140914917&lt;br&gt;&#xA;  [1][23] LD: 5.6876764631397236e-08 LG: 1.0763123035430908&lt;br&gt;&#xA;  [1][24] LD: 1.475878548262699e-07 LG: 1.6125952005386353&lt;br&gt;&#xA;  [1][25] LD: 6.919402721905499e-07 LG: 1.6719598770141602&lt;br&gt;&#xA;  [1][26] LD: 1.3498377526843797e-08 LG: 1.1914349794387817&lt;br&gt;&#xA;  [1][27] LD: 1.3576584301233652e-08 LG: 1.1994632482528687&lt;br&gt;&#xA;  [1][28] LD: 3.087819067104647e-08 LG: 1.2909866571426392&lt;br&gt;&#xA;  [1][29] LD: 3.416153049329296e-07 LG: 2.143049478530884&lt;br&gt;&#xA;  [1][30] LD: 4.477038118011478e-08 LG: 1.7709745168685913&lt;br&gt;&#xA;  [1][31] LD: 2.1782324832742006e-09 LG: 1.2023413181304932&lt;br&gt;&#xA;  [1][32] LD: 1.0589346999267946e-07 LG: 1.4242452383041382  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My training code is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;generator = deeplabv2.Res_Deeplab()&#xA;optimizer_G = optim.SGD(filter(lambda p: p.requires_grad, \&#xA;        generator.parameters()),lr=0.00025,momentum=0.9,\&#xA;        weight_decay=0.0001,nesterov=True)&#xA; discriminator = Dis(in_channels=21)&#xA;optimizer_D = optim.Adam(filter(lambda p: p.requires_grad, \&#xA;            discriminator.parameters()),lr=0.0001,weight_decay=0.0001)&#xA; for epoch in range(args.start_epoch,args.max_epoch+1):&#xA;&#xA;     for batch_id, (img,mask,ohmask) in enumerate(trainloader):&#xA;          img,mask,ohmask = Variable(img.cuda()),Variable(mask.cuda(),requires_grad=False),\&#xA;                                Variable(ohmask.cuda(),requires_grad=False)&#xA;        # ohmask : mask (HxW) converted to one-hot encoded probability map for each class( 21xHxW )&#xA;         out_img_map = generator(img)&#xA;         out_img_map = nn.LogSoftmax()(out_img_map)&#xA;        #######################&#xA;        # Adverarial Training#&#xA;        #######################&#xA;        if args.mode == 'adv':&#xA;&#xA;                N = out_img_map.size()[0]&#xA;                H = out_img_map.size()[2]&#xA;                W = out_img_map.size()[3]&#xA;&#xA;                # Generate the Real and Fake Labels&#xA;                target_fake = Variable(torch.zeros((N,H,W)).long().cuda(),requires_grad=False)&#xA;                target_real = Variable(torch.ones((N,H,W)).long().cuda(),requires_grad=False)&#xA;&#xA;                #########################&#xA;                # Discriminator Training#&#xA;                #########################&#xA;&#xA;                # Train on Real&#xA;                conf_map_real = nn.LogSoftmax()(discriminator(ohmask.float()))&#xA;&#xA;                optimizer_D.zero_grad()&#xA;&#xA;                LD_real = nn.NLLLoss2d()(conf_map_real,target_real)&#xA;                LD_real.backward()&#xA;&#xA;                # Train on Fake&#xA;                conf_map_fake = nn.LogSoftmax()(discriminator(Variable(out_img_map.data)))&#xA;                LD_fake = nn.NLLLoss2d()(conf_map_fake,target_fake)&#xA;                LD_fake.backward()&#xA;&#xA;                # Update Discriminator weights&#xA;&#xA;                optimizer_D.step()&#xA;&#xA;                ######################&#xA;                # Generator Training #&#xA;                #####################&#xA;                conf_map_fake = nn.LogSoftmax()(discriminator(out_img_map))&#xA;                LG_ce = nn.NLLLoss2d()(out_img_map,mask)&#xA;                LG_adv = args.lam_adv * nn.NLLLoss2d()(conf_map_fake,target_real)&#xA;&#xA;                LG_seg = LG_ce + args.lam_adv * LG_adv&#xA;                optimizer_G.zero_grad()&#xA;                LG_ce.backward(retain_variables=True)&#xA;                LG_adv.backward()&#xA;                optimizer_G.step()&#xA;                print(&quot;[{}][{}] LD: {} LG: {}&quot;.format(epoch,i,(LD_real + LD_fake).data[0],LG_seg.data[0]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using Resnet-101 as segmentation network (which is my generator) and my discriminator is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; class Dis(nn.Module):&#xA;    &quot;&quot;&quot;&#xA;        Discriminator Network for the Adversarial Training.&#xA;    &quot;&quot;&quot;&#xA;    def __init__(self,in_channels,negative_slope = 0.2):&#xA;        super(Dis, self).__init__()&#xA;        self._in_channels = in_channels&#xA;        self._negative_slope = negative_slope&#xA;&#xA;        self.conv1 = nn.Conv2d(in_channels=self._in_channels,out_channels=64,kernel_size=4,stride=2,padding=2)&#xA;        self.relu1 = nn.LeakyReLU(self._negative_slope,inplace=True)&#xA;        self.conv2 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=4,stride=2,padding=2)&#xA;        self.relu2 = nn.LeakyReLU(self._negative_slope,inplace=True)&#xA;        self.conv3 = nn.Conv2d(in_channels=128,out_channels=256,kernel_size=4,stride=2,padding=2)&#xA;        self.relu3 = nn.LeakyReLU(self._negative_slope,inplace=True)&#xA;        self.conv4 = nn.Conv2d(in_channels=256,out_channels=512,kernel_size=4,stride=2,padding=2)&#xA;        self.relu4 = nn.LeakyReLU(self._negative_slope,inplace=True)&#xA;        self.conv5 = nn.Conv2d(in_channels=512,out_channels=2,kernel_size=4,stride=2,padding=2)&#xA;&#xA;    def forward(self,x):&#xA;        x= self.conv1(x) # -,-,161,161&#xA;        x = self.relu1(x)&#xA;        x= self.conv2(x) # -,-,81,81&#xA;        x = self.relu2(x)&#xA;        x= self.conv3(x) # -,-,41,41&#xA;        x = self.relu3(x)&#xA;        x= self.conv4(x) # -,-,21,21&#xA;        x = self.relu4(x)&#xA;        x = self.conv5(x) # -,-,11,11&#xA;        # upsample&#xA;        x = F.upsample_bilinear(x,scale_factor=2)&#xA;        x = x[:,:,:-1,:-1] # -,-, 21,21&#xA;&#xA;        x = F.upsample_bilinear(x,scale_factor=2)&#xA;        x = x[:,:,:-1,:-1] # -,-,41,41&#xA;&#xA;        x = F.upsample_bilinear(x,scale_factor=2)&#xA;        x = x[:,:,:-1,:-1] #-,-,81,81&#xA;&#xA;        x = F.upsample_bilinear(x,scale_factor=2)&#xA;        x = x[:,:,:-1,:-1] #-,-,161,161&#xA;&#xA;        x = F.upsample_bilinear(x,scale_factor=2)&#xA;        x = x[:,:,:-1,:-1] # -,-,321,321&#xA;&#xA;        return x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="42349" LastEditorUserId="83275" LastEditDate="2019-11-17T15:40:38.997" LastActivityDate="2019-11-17T15:40:38.997" Title="Adversarial Learning for Semantic Segmentation" Tags="|pytorch|gan|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25085" PostTypeId="2" ParentId="25081" CreationDate="2017-11-24T20:51:03.997" Score="1" Body="&lt;p&gt;Like the update rule for bias terms in dense layers, in convolutional nets the bias gradient is calculated using the sum of derivatives of Z terms:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ dJ / db = \sum_h \sum_w dZ_{hw} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which J is the cost function, w is the width of the activation after convolution and h is the height of the activation after convolution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;db is computed by summing  dZs. It means you are summing over all the gradients of the conv output (Z) with respect to the cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculating the error of the net depends on the cost function that you have used. Depending on using cross entropy or mean-squared-error or other justified cost functions, you may have different update rule for other parameters of your net. But if you use cross entropy which is common for variants of classification tasks, the above update rule is used for updating bias terms.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-11-24T22:12:49.473" LastActivityDate="2017-11-24T22:12:49.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25086" PostTypeId="2" ParentId="21758" CreationDate="2017-11-24T21:14:34.610" Score="0" Body="&lt;p&gt;I personally go with Association Rules. Since orders are aggregated by transactions (customers), does buying Coke displace buying Pepsi.. Buying IOS displace buying Android? will end with a nice heat map ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Along with ideas around and practices, try your own art around benefiting the the business (the beauty here).&lt;/p&gt;&#xA;" OwnerUserId="42354" LastEditorUserId="40853" LastEditDate="2017-11-25T12:59:59.303" LastActivityDate="2017-11-25T12:59:59.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25087" PostTypeId="2" ParentId="9815" CreationDate="2017-11-24T21:47:59.053" Score="0" Body="&lt;p&gt;You have two areas to satisfy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your business need (how many cluster you want to have).&lt;/li&gt;&#xA;&lt;li&gt;The nature of the data (how dividable it is).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the second point, you can notice when using random seeds that being away from the optimal c number, will lead to &quot;more&quot; differences in the results for each run.&lt;/p&gt;&#xA;" OwnerUserId="42354" LastEditorUserId="29575" LastEditDate="2017-11-24T22:15:59.247" LastActivityDate="2017-11-24T22:15:59.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25088" PostTypeId="2" ParentId="10328" CreationDate="2017-11-24T22:15:18.250" Score="1" Body="&lt;p&gt;If you take aerial images for example, you might need to identify overlaps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say your goal is to segment water, vegetation and rock areas. More to be assigned to the closest centroid, there are hidden memberships to other centroids that can be identified by fuzzy c-means image segmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On this thesis at &lt;a href=&quot;https://zone.biblio.laurentian.ca/handle/10219/2616&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://zone.biblio.laurentian.ca/handle/10219/2616&lt;/a&gt;, at section (4.5 Analysis - Task 1) you can find detailed analysis using both (k-means &amp;amp; fuzzy c-means).&lt;/p&gt;&#xA;" OwnerUserId="42354" LastActivityDate="2017-11-24T22:15:18.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25089" PostTypeId="2" ParentId="25051" CreationDate="2017-11-24T22:28:40.877" Score="-1" Body="&lt;p&gt;I recently read &lt;a href=&quot;https://arxiv.org/pdf/1706.00388.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; which proposed a new technique without using skip-connections.It is proposed that the same accuracy can be achieved by increasing the width of the network rather than depth.&lt;/p&gt;&#xA;" OwnerUserId="42353" LastEditorUserId="42353" LastEditDate="2017-11-24T22:39:51.580" LastActivityDate="2017-11-24T22:39:51.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25090" PostTypeId="1" AcceptedAnswerId="26991" CreationDate="2017-11-24T23:28:22.233" Score="0" ViewCount="1400" Body="&lt;p&gt;Given a social network, I want to perform community detection and compare the result to known node metadata, such as gender, age, etc. to see if certain communities are largely composed of &quot;similar&quot; people. I have seen this done before in visualizations like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TRGsj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TRGsj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(image from &lt;a href=&quot;https://arxiv.org/pdf/0809.0690.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/0809.0690.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where each circle represents a community and the coloring of the circle shows the breakdown of some attribute (e.g. nationality) within that community. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know what tool can be used to create such a visualization? I have seen it in multiple papers, so there must be something...&lt;/p&gt;&#xA;" OwnerUserId="42358" LastActivityDate="2019-10-18T06:08:21.777" Title="Visualizing community composition using network of pie charts" Tags="|visualization|social-network-analysis|metadata|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25091" PostTypeId="2" ParentId="22725" CreationDate="2017-11-24T23:28:50.580" Score="12" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hellinger_distance&quot; rel=&quot;noreferrer&quot;&gt;Hellinger distance&lt;/a&gt; is a metric to measure the difference between two probability distributions. It is the probabilistic analog of &lt;a href=&quot;https://en.wikipedia.org/wiki/Euclidean_distance&quot; rel=&quot;noreferrer&quot;&gt;Euclidean distance&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is useful when quantifying the difference between two probability distributions. For example, if you estimate a distribution for users and non-users of a service. If the Hellinger distance is small between those groups for some features, then those features are not statistically useful for segmentation. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="26562" LastEditDate="2018-04-16T03:20:42.863" LastActivityDate="2018-04-16T03:20:42.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25092" PostTypeId="1" CreationDate="2017-11-25T00:30:22.040" Score="3" ViewCount="5200" Body="&lt;p&gt;When I evaluate the model I seem to be getting a decent RMSE score but when I try to actually see the predictions when I call the model all my values are the same.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; xdata = xgboost.DMatrix(X_train, y_train, feature_names=all_vars)&#xA; xdata_val = xgboost.DMatrix(X_valid, y_valid, feature_names=all_vars)&#xA; xgb_parms['seed'] = random.randint(0,1e9)&#xA; model = xgboost.train(xgb_parms, xdata)&#xA; model.eval(xdata_val)&#xA; ypred = model.predict(xdata_val)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I believe the error is on my last step, what am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="42359" LastActivityDate="2021-01-24T21:22:54.167" Title="XGBoost Predictions all the same" Tags="|xgboost|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25093" PostTypeId="1" AcceptedAnswerId="25107" CreationDate="2017-11-25T04:12:11.200" Score="0" ViewCount="511" Body="&lt;p&gt;I am working on a face recognition application using deep learning. To plot the ROC curves and do performance evaluation, I extracted the features from the last layer of the deep neural network and I have tried Euclidian and Manhattan distances between high dimensional feature representation, but this won't give me accurate measure in terms of comparison as faces have intra-personal and interpersonal differences and other challenges such as pose, lighting condition, and expression. I came across techniques such as &lt;a href=&quot;https://s3.amazonaws.com/academia.edu.documents/31414608/JointBayesian.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A&amp;amp;Expires=1511586467&amp;amp;Signature=nrjKQKdDBA%2Bbd71DtGBpIkNO8bA%3D&amp;amp;response-content-disposition=inline%3B%20filename%3DBayesian_Face_Revisited_A_Joint_Formulat.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;joint bayesian&lt;/a&gt;, &lt;a href=&quot;https://www.researchgate.net/publication/261236790_Blessing_of_Dimensionality_High-Dimensional_Feature_and_Its_Efficient_Compression_for_Face_Verification&quot; rel=&quot;nofollow noreferrer&quot;&gt;sparse projection&lt;/a&gt;, &lt;a href=&quot;http://openaccess.thecvf.com/content_iccv_2013/papers/Cao_A_Practical_Transfer_2013_ICCV_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;KL-divergence based&lt;/a&gt; regularizer/prior(EM algorithm) etc. Are there any other approaches that I'm missing here? What would be the efficient approach to generate similarity scores? &lt;/p&gt;&#xA;" OwnerUserId="42248" LastEditorUserId="42248" LastEditDate="2017-11-26T01:58:56.703" LastActivityDate="2017-11-26T01:58:56.703" Title="What is the efficient way to generate a similarity score when comparing two face images?" Tags="|machine-learning|classification|deep-learning|similarity|distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25095" PostTypeId="2" ParentId="25090" CreationDate="2017-11-25T07:20:08.373" Score="0" Body="&lt;p&gt;I think you can try using &lt;a href=&quot;https://gephi.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Gephi&lt;/strong&gt;&lt;/a&gt;, you might get that visualization and you can apply couple of Community Detection algorithms. Best part is, it is an open source tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But one drawback of using it is, you cannot handle huge datasets. If you can extract exact sample from population. Then it shouldn't be an issue. If you have a small dataset then you can directly import to Gephi.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try using it, the graphs(outcome) over there are animated too. Let me know if you have any issues.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-01-24T09:45:55.497" LastActivityDate="2018-01-24T09:45:55.497" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="25096" PostTypeId="2" ParentId="25056" CreationDate="2017-11-25T08:36:24.380" Score="0" Body="&lt;pre&gt;&lt;code&gt;aggregate( time~ id, FUN=mean, data=yourdataframe)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user42229" LastActivityDate="2017-11-25T08:36:24.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25097" PostTypeId="1" AcceptedAnswerId="25098" CreationDate="2017-11-25T10:44:59.617" Score="3" ViewCount="901" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aVcq2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aVcq2.png&quot; alt=&quot;VGG16 Architecture&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would we use two convolution layers in a row with same spec? E.g. in VGG16 first two layers, they used 3 x 3 filter with 64 depth twice. What is the difference if we use 3 x 3 filter with 128 depth once.&lt;/p&gt;&#xA;" OwnerUserId="42375" LastEditorUserId="29169" LastEditDate="2021-02-13T19:21:04.553" LastActivityDate="2021-02-13T19:21:04.553" Title="What is the difference between 8 filters twice and one 16 filters in convolutional layers at CNN?" Tags="|cnn|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25098" PostTypeId="2" ParentId="25097" CreationDate="2017-11-25T11:34:59.433" Score="5" Body="&lt;p&gt;Each layer has a limited amount that it can transform the layer below it. There is one linear component (weighted sum of output of layer beneath it), and one non-linear component (typically ReLU).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is in theory possible to approximate &lt;em&gt;any&lt;/em&gt; function with a large enough single layer in a fully-connected network. However, a stack of similar smaller layers is more expressive using less resources. That means for the same number of &lt;em&gt;parameters&lt;/em&gt; you have access to a more flexible function approximator. At some level of complexity for your target function, the cost (in terms of CPU time, data required and effort in training) of making a single layer wider is higher than the cost of stacking more, similar layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, for a CNN, you have to worry about receptive field. Any feature map can only express values that the filter can &quot;see&quot; due to width of the kernel. As you add more layers, each kernel applied extends the width and height of the base image that the features in the last layer effectively calculate over. If you also have a fully-connected layer after the convolutional layer, then you can in theory compensate for a poor receptive field with a very large fully-connected layer - but then you are back to the first problem of wide network with more parameters than strictly necessary to learn the function.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-11-25T11:34:59.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25099" PostTypeId="1" CreationDate="2017-11-25T12:18:59.940" Score="2" ViewCount="187" Body="&lt;p&gt;In practice, a large data set leads to a large K, and storing K may become a problem. One way to deal with this is to perform clustering on the dataset, and populate the kernel with the means of those clusters for speeding up and reducing storage.&#xA;How can I calculate the kernel using clustering?&lt;/p&gt;&#xA;" OwnerUserId="42220" LastEditorUserId="42220" LastEditDate="2017-11-27T04:24:43.657" LastActivityDate="2018-11-23T06:00:50.140" Title="How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis?" Tags="|machine-learning|deep-learning|data-mining|bigdata|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25102" PostTypeId="2" ParentId="24986" CreationDate="2017-11-25T15:30:29.490" Score="19" Body="&lt;p&gt;The second way, predicting &lt;span class=&quot;math-container&quot;&gt;$x=cos(\alpha)$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y=sin(\alpha)$&lt;/span&gt; is totally okay.&lt;/p&gt;&#xA;&lt;p&gt;Yes, the norm of the predicted &lt;span class=&quot;math-container&quot;&gt;$(x, y)$&lt;/span&gt; vector is not guaranteed to be near &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;. But it is not likely to blow up, especially if you use sigmoid activation functions (which are bounded by they nature) and/or regularize your model well. Why should your model predict a large value, if all the training samples were in &lt;span class=&quot;math-container&quot;&gt;$[-1, 1]$&lt;/span&gt;?&lt;/p&gt;&#xA;&lt;p&gt;Another side is vector &lt;span class=&quot;math-container&quot;&gt;$(x,y)$&lt;/span&gt; too close to &lt;span class=&quot;math-container&quot;&gt;$(0,0)$&lt;/span&gt;. This may sometimes happen, and could indeed result in predicting wrong angles. But it may be seen as a benefit of your model - you can consider norm of &lt;span class=&quot;math-container&quot;&gt;$(x,y)$&lt;/span&gt; as a measure of &lt;em&gt;confidence&lt;/em&gt; of your model. Indeed, a norm close to 0 means that your model is not sure where the right direction is.&lt;/p&gt;&#xA;&lt;p&gt;Here is a small example in Python which shows that it is better to predict sin and cos, that to predict the angle directly:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-python prettyprint-override&quot;&gt;&lt;code&gt;# predicting the angle (in radians)&#xA;import numpy as np&#xA;from sklearn.neural_network import MLPRegressor&#xA;from sklearn.model_selection import cross_val_predict&#xA;from sklearn.metrics import r2_score&#xA;# generate toy data&#xA;np.random.seed(1)&#xA;X = np.random.normal(size=(100, 2))&#xA;y = np.arctan2(np.dot(X, [1,2]), np.dot(X, [3,0.4]))&#xA;# simple prediction&#xA;model = MLPRegressor(random_state=42, activation='tanh', max_iter=10000)&#xA;y_simple_pred = cross_val_predict(model, X, y)&#xA;# transformed prediction&#xA;joint = cross_val_predict(model, X, np.column_stack([np.sin(y), np.cos(y)]))&#xA;y_trig_pred = np.arctan2(joint[:,0], joint[:,1])&#xA;# compare&#xA;def align(y_true, y_pred):&#xA;    &amp;quot;&amp;quot;&amp;quot; Add or remove 2*pi to predicted angle to minimize difference from GT&amp;quot;&amp;quot;&amp;quot;&#xA;    y_pred = y_pred.copy()&#xA;    y_pred[y_true-y_pred &amp;gt;  np.pi] += np.pi*2&#xA;    y_pred[y_true-y_pred &amp;lt; -np.pi] -= np.pi*2&#xA;    return y_pred&#xA;print(r2_score(y, align(y, y_simple_pred))) # R^2 about 0.57&#xA;print(r2_score(y, align(y, y_trig_pred)))   # R^2 about 0.99&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;You can go on and plot the predictions, to see that predictions of the sine-cosine model are nearly correct, although may need some further calibration:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-python prettyprint-override&quot;&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;plt.figure(figsize=(12, 3))&#xA;plt.subplot(1,4,1)&#xA;plt.scatter(X[:,0], X[:,1], c=y)&#xA;plt.title('Data (y=color)'); plt.xlabel('x1'); plt.ylabel('x2')&#xA;plt.subplot(1,4,2)&#xA;plt.scatter(y_simple_pred, y)&#xA;plt.title('Direct model'); plt.xlabel('prediction'); plt.ylabel('actual')&#xA;plt.subplot(1,4,3)&#xA;plt.scatter(y_trig_pred, y)&#xA;plt.title('Sine-cosine model'); plt.xlabel('prediction'); plt.ylabel('actual')&#xA;plt.subplot(1,4,4)&#xA;plt.scatter(joint[:,0], joint[:,1], s=5)&#xA;plt.title('Predicted sin and cos'); plt.xlabel('cos'); plt.ylabel('sin')&#xA;plt.tight_layout()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/T7v4J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/T7v4J.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;. A navigation engineer noticed that such a model would be most accurate when the angle is close to &lt;span class=&quot;math-container&quot;&gt;$\frac{\pi N}{2}$&lt;/span&gt;. Indeed, near 0° and 180° the angle &lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is almost linear in &lt;span class=&quot;math-container&quot;&gt;$\cos(\alpha)$&lt;/span&gt;, and near 90° and 270° it is almost linear in &lt;span class=&quot;math-container&quot;&gt;$\sin(\alpha)$&lt;/span&gt;. Thus, it could be beneficial to add &lt;em&gt;two more&lt;/em&gt; outputs, like &lt;span class=&quot;math-container&quot;&gt;$z=\sin(\alpha+\frac{\pi}{4})$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$w=\cos(\alpha+\frac{\pi}{4})$&lt;/span&gt;, to make model almost-linear near 45° and 135° respectively. In this case, however, restoring the original angle is not so obvious.&lt;/p&gt;&#xA;&lt;p&gt;The best solution may be to extract coordinates &lt;span class=&quot;math-container&quot;&gt;$(x,y)$&lt;/span&gt; from both representations (in the second one, we need to rotate &lt;span class=&quot;math-container&quot;&gt;$(z,w)$&lt;/span&gt; to get &lt;span class=&quot;math-container&quot;&gt;$(x,y)$&lt;/span&gt;), average them, and only then calculate &lt;code&gt;arctan2&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="53602" LastEditDate="2022-03-30T10:48:53.887" LastActivityDate="2022-03-30T10:48:53.887" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="25104" PostTypeId="1" CreationDate="2017-11-25T17:36:29.253" Score="0" ViewCount="83" Body="&lt;p&gt;I have some 14 years of work experience in operations management and education areas as Instructor of statistics and operations. I have little better command over statistical learnings. I do have a keen interest in data science learnings. Basically, want to perform a role of analyst on big data.  Will it right to make venture in Data Science field with this experience? There is touch on coding is with but do not project experiences. &lt;/p&gt;&#xA;" OwnerUserId="42384" LastEditorUserId="1330" LastEditDate="2017-11-25T18:51:00.280" LastActivityDate="2017-11-27T07:42:10.727" Title="Switching To Data Science Field" Tags="|career|" AnswerCount="1" CommentCount="2" ClosedDate="2017-11-27T16:28:49.320" ContentLicense="CC BY-SA 3.0" />
  <row Id="25105" PostTypeId="2" ParentId="23754" CreationDate="2017-11-25T18:31:07.403" Score="2" Body="&lt;p&gt;I am also using &lt;a href=&quot;https://github.com/lanpa/tensorboard-pytorch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorboard-PyTorch&lt;/a&gt; (TensorboardX). Over all I am quite happy with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But don't try to visualize graphs. At least none with a bit of complexity (e.g. a resnet50 won't work). There are some issues about it on &lt;a href=&quot;https://github.com/lanpa/tensorboard-pytorch/issues/7&quot; rel=&quot;nofollow noreferrer&quot;&gt;their github page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But better check out the Pytorch forum frequently. Pytorch seems to move quite fast. And a direct tensorboard implementation &lt;a href=&quot;https://discuss.pytorch.org/t/print-autograd-graph/692/2?u=xwgeng&quot; rel=&quot;nofollow noreferrer&quot;&gt;is planned&lt;/a&gt;, as well.&lt;/p&gt;&#xA;" OwnerUserId="42385" LastActivityDate="2017-11-25T18:31:07.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25106" PostTypeId="2" ParentId="25068" CreationDate="2017-11-25T18:39:43.280" Score="1" Body="&lt;p&gt;Given the number of parameters and the number of options for each parameter, a graphical tool (aka, graphical user interface or GUI) is not very practical for CNNs. Most practitioners store and manipulate parameters in text form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two graphical projects for TensorFlow. &lt;a href=&quot;http://playground.tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Playground&lt;/a&gt; which is an in-browser network that can be modified and run on sample datasets. &lt;a href=&quot;https://www.tensorflow.org/get_started/graph_viz&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorBoard&lt;/a&gt; which visualizes the computational graph and helps with debugging models.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-25T18:39:43.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25107" PostTypeId="2" ParentId="25093" CreationDate="2017-11-25T19:07:41.543" Score="2" Body="&lt;p&gt;A common method is defining &quot;landmarks&quot;, a set of specific points that exist on every face . For example, the outside edge of each eye or the bottom of the chin are landmarks on most faces. Train the deep learning network to learn the relationship between these landmarks which will form the embedding space. Then define a &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;distance metric&lt;/a&gt; in this embedding space. Finally, create a threshold for similarity between sets of points in this embedding space. That threshold can be learned through &quot;one-class&quot; classification techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One gotcha is &lt;a href=&quot;https://en.wikipedia.org/wiki/Pose_(computer_vision)&quot; rel=&quot;nofollow noreferrer&quot;&gt;pose&lt;/a&gt;. Faces turned different directions look totally different to a computer. Assuming you have isolated the faces in an image, it is best practice to warp each face so that the eyes and lips are always in the place in the image. This (and other normalization steps) should be done early in the processing pipeline to make subsequent analysis much easier.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-25T19:07:41.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25108" PostTypeId="1" AcceptedAnswerId="25115" CreationDate="2017-11-25T22:38:29.163" Score="6" ViewCount="509" Body="&lt;p&gt;Using adjacency matrix to represent a connection between node i and node j, 1 means connected and 0 means not connected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using eigenvector to represent the centrality means the more nodes with high centrality values a node is connected to, the higher that node's centrality value is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eigenvector itself is just a vector after applying a transformation, the result vector has the same direction or totally opposite direction as the original one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really can't see the relationship between those two properties.&lt;/p&gt;&#xA;" OwnerUserId="41792" LastEditorUserId="26243" LastEditDate="2017-12-23T13:00:31.603" LastActivityDate="2017-12-23T13:00:31.603" Title="Why an eigenvector might be reasonable notion of centrality" Tags="|social-network-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25109" PostTypeId="2" ParentId="25108" CreationDate="2017-11-25T23:42:49.020" Score="3" Body="&lt;p&gt;Let's define the centrality of a vertex as proportional to the sum of its neighbors' centralities. If you write it out and incorporate the adjacency matrix, the eigendecomposition emerges immediately with the proportional constant as the reciprocal of the eigenvalue. The relevance of the eigenvector is that the centrality is defined through it: the score of a vertex is the corresponding entry on the first eigenvector. We had to choose the first eigenvector because the adjacency matrix is non-negative and we want the centralities to be so too, due to the Perron-Frobenius theorem (&lt;a href=&quot;http://www.math.harvard.edu/library/sternberg/slides/1180912pf.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;see these lectures notes for details&lt;/a&gt;). So if our centralities are intrinsically related to the transition matrix' eigenvectors, how do we find them? By using the power method, which relies on their fixed point nature! If you transform an eigenvector, you get a collinear vector (the property you wanted to relate), so why not do this with a random estimate of it until convergence, normalizing as we go along?  Furthermore, if we slightly reformulate the problem to use stochastic matrices, the scores are directly interpretable as probabilities of random walks terminating at the corresponding vertex!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're really curious, there's a monograph: &lt;a href=&quot;https://press.princeton.edu/titles/8216.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google's PageRank and Beyond&lt;/a&gt;. Welcome to the site.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2017-11-26T16:29:54.363" LastActivityDate="2017-11-26T16:29:54.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25110" PostTypeId="2" ParentId="25080" CreationDate="2017-11-26T00:17:07.307" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Discounted_cumulative_gain&quot; rel=&quot;nofollow noreferrer&quot;&gt;Discounted cumulative gain (DCG)&lt;/a&gt; is one of the most common metrics for ranking quality. The gain is accumulated from the top of the result list to the bottom, with the gain of each subsequent result logarithmic discounted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The DCG accumulated at a particular rank position n is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qyxhF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qyxhF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since search result lists vary in length, &lt;a href=&quot;https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG&quot; rel=&quot;nofollow noreferrer&quot;&gt;normalized DCG&lt;/a&gt; is used. It is normalized by the ideal ranking, sorting all relevant documents in the corpus by their relative relevance and producing the maximum possible DCG:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OeOov.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OeOov.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-26T00:17:07.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25112" PostTypeId="2" ParentId="6030" CreationDate="2017-11-26T03:07:00.963" Score="1" Body="&lt;p&gt;John Langford's documentation on GitHub could help. You can find something on the &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki/Learning-to-Search-Sub-System&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning to Search Sub System page&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2017-11-26T03:07:00.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25113" PostTypeId="2" ParentId="25099" CreationDate="2017-11-26T09:27:10.663" Score="1" Body="&lt;p&gt;Foremost, we must understand what is &lt;code&gt;clustering&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is an unsupervised algorithm, applicable to scenario's where the target class is unknown. So essentially it's a data preprocessing algorithm. Continuing further, &lt;code&gt;cluster analysis&lt;/code&gt;, helps in detecting &lt;code&gt;patterns&lt;/code&gt; in data. Now, to detect patterns in data, one must ensure the &lt;code&gt;sample&lt;/code&gt; consist of &lt;code&gt;statistically significant&lt;/code&gt; variables. To find such significant variables, we need to perform various data preprocessing tasks like &lt;code&gt;outlier&lt;/code&gt;, &lt;code&gt;missing value('s)&lt;/code&gt;, &lt;code&gt;correlation&lt;/code&gt; and &lt;code&gt;dimensionality reduction&lt;/code&gt; detection and treatment. Post, performing these steps, the &lt;code&gt;noise&lt;/code&gt; from the data would be removed and one would be able to reveal the &lt;code&gt;signal&lt;/code&gt;. Otherwise, one can perform the &lt;code&gt;principal component analysis (pca)&lt;/code&gt; to detect the signal. This signal('s), can then aid in determining the true patterns or &lt;code&gt;clusters&lt;/code&gt; in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, coming to the second part of the question. You can use the &lt;code&gt;kernlab&lt;/code&gt; package in &lt;code&gt;R&lt;/code&gt; to calculate the kernel. Perhaps these posts can help you further, &lt;a href=&quot;https://rpubs.com/sandipan/205018&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/11621642/assign-new-data-point-to-cluster-in-kernel-k-means-kernlab-package-in-r&quot;&gt;2&lt;/a&gt;, &lt;a href=&quot;https://cran.r-project.org/web/packages/kernlab/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;3&lt;/a&gt;, &lt;a href=&quot;https://www.r-bloggers.com/learning-kernels-svm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;4&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11061" LastEditorUserId="11061" LastEditDate="2017-11-26T09:36:09.910" LastActivityDate="2017-11-26T09:36:09.910" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25114" PostTypeId="2" ParentId="24909" CreationDate="2017-11-26T11:03:04.927" Score="4" Body="&lt;p&gt;The &lt;a href=&quot;https://i.stack.imgur.com/rnFCb.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;last image&lt;/a&gt; that you draw, where (at a given timestep) each cell of the second layer receives input from all cells in the first layer, is the right one.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of recurrent layers as fully connected layers which receive sequences of input and apply their transformations at each timestep. They just receive other inputs depending on their previous inputs but the connectivity between layer is exactly the same as a fully connected : each unit of a given layer performs a weighted average of the activations of all the units in the previous layer (or of the inputs if it's the first layer).  &lt;/p&gt;&#xA;" OwnerUserId="42399" LastEditorUserId="29575" LastEditDate="2017-11-27T16:26:05.397" LastActivityDate="2017-11-27T16:26:05.397" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25115" PostTypeId="2" ParentId="25108" CreationDate="2017-11-26T11:39:52.960" Score="6" Body="&lt;p&gt;Let the adjacency matrix of our network be $A∈\{0,1\}^{n×n}$ with an empty diagonal ($A_{ii} = 0 ∀i$).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Direct approach&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let’s start with the approach that a node’s centrality ($C_i$) shall be proportional to the sum of the centralities of its neighbours with a proportionality constant $\frac{1}{λ}$ (chosen thusly with some foresight):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ C_i = \frac{1}{λ} \sum_{j=1}^{n} A_{ij} C_j.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is nothing but a line-wise formulation of the matrix–vector multiplication:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ λ \vec{C} = A·\vec{C},$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is exactly the definition of an eigenvector. Now, to see why the largest eigenvector is chosen, we can turn to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Perron%E2%80%93Frobenius_theorem&quot; rel=&quot;noreferrer&quot;&gt;Perron–Frobenius theorem&lt;/a&gt;, which tells us that for this eigenvalue (and in case of a connected network only for this eigenvalue), we can find an eigenvector all of whose components, i.e., the eigenvector centralities are positive.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Iterative approach&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Alternatively, we can interpret the above ansatz iteratively:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign random positive values to the components of $\vec{C}$.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Update these values according to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\vec{C} ← \frac{A·\vec{C}}{\left| A·\vec{C} \right|}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that each component is updated according to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$C_i ← \frac{1}{\left| A·\vec{C} \right|} \sum_{j=1}^{n} A_{ij} C_j,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e., you say the new centrality of a node is the sum of the centralities of its neighbours – times some normalisation to avoid values getting very big.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Repeat Step 2 until the centralities converge. The idea is that if this converges to a unique result, this result does not only fulfil $C_i \propto \sum_{j=1}^{n} A_{ij} C_j$ but is also robust in that respect.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For almost all initial choices of $\vec{C}$, this will converge to the positive, length-1 eigenvector to the largest eigenvalue (which exists and is unique for a connected network, see above). The reason for this is that components along the eigenvector to the largest eigenvalue will be most enlarged by the multiplication and thus dominate the others over the iterations. (Having no components along the eigenvector to the largest eigenvalue does not happen in reality and is the reason why it’s &lt;em&gt;almost all&lt;/em&gt; above.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that such an iteration can also be used to numerically determine the largest eigenvalue.&lt;/p&gt;&#xA;" OwnerUserId="13952" LastActivityDate="2017-11-26T11:39:52.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25118" PostTypeId="1" AcceptedAnswerId="25120" CreationDate="2017-11-26T19:08:43.097" Score="0" ViewCount="257" Body="&lt;p&gt;I am working on a multi-class classification task for 24 classes using XGBoost. I am training the model as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param = {'max_depth': max_depth, 'eta': learning_rate, 'silent': 1, 'objective': 'multi:softmax', 'num_class': 24}&#xA;bst = xgb.train(param, dtrain, num_round)&#xA;bst = xgb.Booster({'nthread': 4})&#xA;predictions = bst.predict(dval)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the predictions as this array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.5  0.5  0.5 ...,  0.5  0.5  0.5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know why it's predicting all classes as 0.5 instead of one of the indices in 0 to 23. Do I need to use any other objective parameter?&lt;/p&gt;&#xA;" OwnerUserId="42419" LastActivityDate="2017-11-26T19:47:48.197" Title="XGBoost Predictions" Tags="|machine-learning|classification|xgboost|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25119" PostTypeId="1" AcceptedAnswerId="25228" CreationDate="2017-11-26T19:32:57.543" Score="41" ViewCount="52358" Body="&lt;p&gt;How to calculate the mAP (mean Average Precision) for the detection task for the Pascal VOC leaderboards?&lt;/p&gt;&#xA;&lt;p&gt;There said - at &lt;strong&gt;&lt;a href=&quot;http://homepages.inf.ed.ac.uk/ckiw/postscript/ijcv_voc09.pdf&quot; rel=&quot;noreferrer&quot;&gt;page 11&lt;/a&gt;&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Average Precision (AP). For the VOC2007 challenge, the interpolated&#xA;average precision (Salton and Mcgill 1986) was used to evaluate both&#xA;classification and detection. For a given task and class, the&#xA;precision/recall curve is computed from a method’s ranked output.&#xA;Recall is defined as the proportion of all positive examples ranked&#xA;above a given rank. Precision is the proportion of all examples above&#xA;that rank which are from the positive class. The AP summarises the&#xA;shape of the precision/recall curve, and is defined as the mean&#xA;precision at a set of eleven equally spaced recall levels&#xA;[0,0.1,...,1]:&#xA;&lt;code&gt;AP = 1/11 ∑ r∈{0,0.1,...,1} pinterp(r)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;The precision at each recall level r is interpolated by taking the&#xA;maximum precision measured for a method for which the corresponding&#xA;recall exceeds r: &lt;code&gt;pinterp(r) = max p(r˜)&lt;/code&gt;, where p(r˜) is the measured&#xA;precision at recall ˜r&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;About &lt;a href=&quot;http://0agr.ru/wiki/index.php/Precision_and_Recall#Average_Precision&quot; rel=&quot;noreferrer&quot;&gt;mAP&lt;/a&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;So does it mean that:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We &lt;strong&gt;calculate Precision and Recall&lt;/strong&gt;:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A) For &lt;strong&gt;many different &lt;code&gt;IoU&lt;/code&gt;&lt;/strong&gt; &lt;code&gt;&amp;gt; {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}&lt;/code&gt; we calculate True/False Positive/Negative values&lt;/p&gt;&#xA;&lt;p&gt;Where &lt;code&gt;True positive = Number_of_detection with IoU &amp;gt; {0, 0.1,..., 1}&lt;/code&gt;, as said &lt;a href=&quot;https://datascience.stackexchange.com/a/16813/98307&quot;&gt;here&lt;/a&gt; and then we calculate:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Precision = True positive / (True positive + False positive)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Recall = True positive / (True positive + False negative)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;B) Or for &lt;strong&gt;many different thresholds&lt;/strong&gt; of detection algorithms we calculate:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Precision = True positive / (True positive + False positive)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Recall = True positive / (True positive + False negative)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;Where &lt;code&gt;True positive = Number_of_detection with IoU &amp;gt; 0.5&lt;/code&gt; as said &lt;a href=&quot;https://stackoverflow.com/a/43168882/9214357&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;C) Or for &lt;strong&gt;many different thresholds&lt;/strong&gt; of detection algorithms we calculate:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Precision = Intersect / Detected_box&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Recall = Intersect / Object&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;As shown &lt;a href=&quot;https://en.wikipedia.org/wiki/Precision_and_recall&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/JlHnn.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JlHnn.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;hr /&gt;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Then we &lt;strong&gt;build Precision-Recall curve&lt;/strong&gt;, as shown here:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Br0IM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Br0IM.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;hr /&gt;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Then we calculate AP (average precision) as &lt;strong&gt;average of 11 values of &lt;code&gt;Precision&lt;/code&gt;&lt;/strong&gt; at the points where &lt;code&gt;Recall = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}&lt;/code&gt;, i.e. &lt;code&gt;AP = 1/11 ∑ recall∈{0,0.1,...,1} Precision(Recall)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;(In general for each point, for example 0.3, we get MAX of Precision for Recall &amp;lt;= 0.3, instead of value of Precision at this point Recall=0.3)&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;ol start=&quot;4&quot;&gt;&#xA;&lt;li&gt;And when we calculate AP only for 1 something object class on all images - then we get &lt;strong&gt;AP (average precision)&lt;/strong&gt; for this class, for example, only for &lt;code&gt;air&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;So AP is a  integral (&lt;a href=&quot;https://stats.stackexchange.com/a/157019&quot;&gt;area under the curve&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;But when we calculate AP for all object classes on all images - then we get &lt;strong&gt;mAP (mean average precision)&lt;/strong&gt; for all images dataset.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is it right, and if it isn't, then how to calculate mAP for Pascal VOC Challenge?&lt;/li&gt;&#xA;&lt;li&gt;And which of the 3 formulas (A, B or C) is correct for calculating Precision and Recall, in paragraph 1?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;mAP = AVG(AP for each object class)&lt;/li&gt;&#xA;&lt;li&gt;AP = AVG(Precision for each of 11 Recalls {precision = 0, 0.1, ..., 1})&lt;/li&gt;&#xA;&lt;li&gt;PR-curve = Precision and Recall (for each Threshold that is in the Predictions bound-boxes)&lt;/li&gt;&#xA;&lt;li&gt;Precision = TP / (TP + FP)&lt;/li&gt;&#xA;&lt;li&gt;Recall = TP / (TP + FN)&lt;/li&gt;&#xA;&lt;li&gt;TP = number of detections with IoU&amp;gt;0.5&lt;/li&gt;&#xA;&lt;li&gt;FP = number of detections with IoU&amp;lt;=0.5 or detected more than once&lt;/li&gt;&#xA;&lt;li&gt;FN = number of objects that not detected or detected with IoU&amp;lt;=0.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37736" LastEditorUserId="98307" LastEditDate="2020-08-20T18:43:15.400" LastActivityDate="2024-02-18T08:39:02.907" Title="How to calculate mAP for detection task for the PASCAL VOC Challenge?" Tags="|machine-learning|neural-network|svm|computer-vision|object-recognition|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25120" PostTypeId="2" ParentId="25118" CreationDate="2017-11-26T19:47:48.197" Score="2" Body="&lt;p&gt;I do not know python API, but I guess with the following line you overwrite the trained object with a freshly created booster object (0.5 is a default prediction)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bst = xgb.Booster({'nthread': 4})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-11-26T19:47:48.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25121" PostTypeId="1" CreationDate="2017-11-26T21:52:04.710" Score="0" ViewCount="256" Body="&lt;p&gt;Let's say a User was Interested in football, so we were recommending him posts about football after few days or weeks he starts watching Baseball.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How'd we go about building a recommender system that also takes recency into consideration&#xA;Or&#xA;Let's say that it knows when a user was interested in what&lt;/p&gt;&#xA;" OwnerUserId="42422" LastActivityDate="2017-11-27T08:39:53.227" Title="How to build recommendation system that also takes time as a feature" Tags="|machine-learning|bigdata|recommender-system|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25122" PostTypeId="1" CreationDate="2017-11-26T22:56:05.017" Score="2" ViewCount="5344" Body="&lt;p&gt;I'm trying to classify images. I want to run each image through a pretrained CNN to apply convolution and pooling and end up with a smaller picture/matrix where the value of each pixel is a feature. Then I want to pass that to an SVM for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone provide code to get started with feature extraction with CNN and SVM ?&lt;/p&gt;&#xA;" OwnerUserId="42424" LastEditorUserId="95719" LastEditDate="2020-04-26T13:03:45.957" LastActivityDate="2020-04-26T13:03:45.957" Title="Pre-trained CNN for feature extraction" Tags="|machine-learning|neural-network|deep-learning|tensorflow|image-classification|" AnswerCount="2" CommentCount="0" ClosedDate="2017-11-28T01:46:59.630" ContentLicense="CC BY-SA 4.0" />
  <row Id="25123" PostTypeId="2" ParentId="25122" CreationDate="2017-11-26T23:46:08.280" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can anyone suggest code to use or a guide for getting started with&#xA;  CNNs in Python?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are a noob to CNN, Keras is the recommended starting point. The code is self-explanatory and easy to understand. Here is a link to a &lt;a href=&quot;http://www.codesofinterest.com/2016/11/lenet-model-for-face-recognition.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;simple classification related task&lt;/a&gt; on facial images tutorial using Keras. You can also start with &lt;a href=&quot;https://elitedatascience.com/keras-tutorial-deep-learning-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;MNIST classification&lt;/a&gt; and understand the core concepts of CNN. Once you understand the fundamentals, you can try and experiment with other libraries like Tensorflow, Theano, Pytorch...etc. &lt;/p&gt;&#xA;" OwnerUserId="42248" LastActivityDate="2017-11-26T23:46:08.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25124" PostTypeId="1" CreationDate="2017-11-27T04:15:36.477" Score="2" ViewCount="2296" Body="&lt;p&gt;&quot;I want to go swimming next Tuesday&quot;&#xA;I want to machine to learn the date I want to go swimming. Is there any approaches or libraries that can &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Extract &quot;next Tuesday&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And calculate the exact date?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2018-08-24T21:02:53.753" Title="Resolving time in NLP" Tags="|nlp|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25125" PostTypeId="2" ParentId="25124" CreationDate="2017-11-27T04:40:15.650" Score="0" Body="&lt;p&gt;A good solution as a service can be using &lt;a href=&quot;https://www.luis.ai/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Microsoft LUIS&lt;/a&gt;. In LUIS, you can use a prebuilt datetimeV2 entity which recognizes dates, times, date ranges and time durations. To know more about this prebuilt entity see &lt;a href=&quot;http://recognizes%20dates,%20times,%20date%20ranges%20and%20time%20durations.&quot; rel=&quot;nofollow noreferrer&quot;&gt;the documentation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-27T04:40:15.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25126" PostTypeId="1" AcceptedAnswerId="25128" CreationDate="2017-11-27T04:55:52.807" Score="2" ViewCount="101" Body="&lt;p&gt;In our organization, there are many people who are into analytics and data science who are OK in their work with in the sense that they know what packages in R/Python etc to use, what algorithms to call for a particular type of problem etc. The problem is that they have very little knowledge of underlying mathematics and are not able to think beyond calling ready made functions i.e. they can never build a custom solution to specific problem. Let us call them as Function Callers as opposed a genuine Data Scientist who has a fair knowledge of the underlying mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to conduction a training course to teach the mathematical foundations of machine learning to the Function Callers and help them become Data Scientists. For this we have shortlisted some ten topics namely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;• Probability&#xA;• Various Probability distributions and data&#xA;• Maximum likelihood methods&#xA;• Linear Algebra (advanced)&#xA;• Single variable calculus, Multi variable calculus&#xA;• Vector calculus&#xA;• Graphical models&#xA;• Bayesian networks&#xA;• Optimization techniques&#xA;• Statistical models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is going to be an &lt;strong&gt;80 hour&lt;/strong&gt; course so it is not possible to cover everything in detail like a college/university course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: Assume that the course will have enough time to teach and implement only &lt;strong&gt;3 or 4&lt;/strong&gt; things in each of the topics mentioned above. So given this background, what are the best 3 or 4 things that can be covered under each topic?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: The target audience have either science or college level mathematics background but they do not have deep knowledge about the mathematics that is used in machine learning.&lt;/p&gt;&#xA;" OwnerUserId="42439" LastEditorUserId="42439" LastEditDate="2017-11-27T05:13:58.243" LastActivityDate="2017-11-27T08:29:14.890" Title="Function Callers Vs Data Scientists" Tags="|machine-learning|statistics|" AnswerCount="2" CommentCount="0" ClosedDate="2017-11-27T16:27:37.767" ContentLicense="CC BY-SA 3.0" />
  <row Id="25127" PostTypeId="1" AcceptedAnswerId="25145" CreationDate="2017-11-27T05:02:36.747" Score="1" ViewCount="1479" Body="&lt;p&gt;When we use a function to approximate Q values or policy to state-action pairs we hear the above two terms &lt;strong&gt;fully and partially observed features&lt;/strong&gt;. what is the difference . Normally in Atari Games they generate Q values by taking the current pixels distribution of the frame .  &lt;/p&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2017-11-27T13:46:25.047" Title="What is the difference between fully observed and partially observed state features in Reinforcement learning ?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25128" PostTypeId="2" ParentId="25126" CreationDate="2017-11-27T05:20:52.203" Score="1" Body="&lt;p&gt;I've done my Masters in Analytics from National University of Singapore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer which I'm going to give is based on my experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ofcourse all the topics are important in the above mentioned list but if you need to prioritize from the topics which you have mentioned above I think the following are very important for any person who is working in a Data Science field. :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Various Probability and Data Distribution&lt;/strong&gt;: Here you can give basic intro on Probability. You don't need a separate Probability class. A bit more its applications.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Single variable calculus/Multi variable calculus&lt;/strong&gt;: There are very important, when you try to forecast something this plays an important role.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Graphical Models&lt;/strong&gt;: Yes, these are helpful in implementing all kinds of community detection algorithms, social networking and many more.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bayesian Network&lt;/strong&gt;: This is one of the basic techniques which is important and should be known by all statisticians.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Statistical Models&lt;/strong&gt;: They need to be exposed to different statistical models as you know there are many models which can perform better in different scenarios, It comes only by practice. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Rest are not so important at the start, you can conduct another session if necessary based on the outcome of the 1st session. Where you can cover all the topics like Linear Algebra (advanced),  Vector calculus, Optimization techniques, Maximum likelihood methods. As you know these techniques would come into play after getting the basic models ready and you would use these if you want to improve the accuracy of the model or tweaking the model WRT to your Business Problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this answer may help you.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-27T05:20:52.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25129" PostTypeId="2" ParentId="25104" CreationDate="2017-11-27T07:42:10.727" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you have mentioned that you are an Instructor of Statistics I think you can directly start with different methodologies/techniques available. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @Emre has suggested it would be better if you can work on different project available on &lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;, &lt;a href=&quot;https://www.analyticsvidhya.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analytics Vidya&lt;/a&gt;. You would be exposed to most of the necessary things in Data Science field. Following &lt;a href=&quot;https://www.r-bloggers.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R-Bloggers&lt;/a&gt;, will expose you to the latest packages. Ofcourse following Data Science Exchange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can doing small projects on different datasets on &lt;a href=&quot;http://archive.ics.uci.edu/ml/index.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI&lt;/a&gt;. For experimenting using different techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are just couple of things which would help you to be updated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Follow them for atlest next 2-3 months then you can access yourself.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-27T07:42:10.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25130" PostTypeId="1" CreationDate="2017-11-27T07:43:23.200" Score="1" ViewCount="1550" Body="&lt;p&gt;I want to extract the &lt;strong&gt;skills&lt;/strong&gt; from a resume using Natural languge processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume? &lt;/p&gt;&#xA;" OwnerUserId="42023" LastEditorUserId="40853" LastEditDate="2017-11-27T08:27:39.167" LastActivityDate="2017-11-27T08:33:23.950" Title="How to find the datasets for skill test (like java, python, c c++, etc.)" Tags="|data-mining|nlp|dataset|data|" AnswerCount="1" CommentCount="3" ClosedDate="2017-11-28T01:50:31.667" ContentLicense="CC BY-SA 3.0" />
  <row Id="25131" PostTypeId="2" ParentId="25130" CreationDate="2017-11-27T07:52:39.570" Score="2" Body="&lt;p&gt;a) You may try to find an anonymized resume dataset in the openbase of &lt;a href=&quot;https://www.kaggle.com/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) You may do some web scraping on professional social networks like LinkedIn (take the description of a profile as resume, and the LinkedIn skills as supervised training set) to build your own sample database (do not forget to anonymize it).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="42401" LastActivityDate="2017-11-27T07:52:39.570" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25133" PostTypeId="2" ParentId="25126" CreationDate="2017-11-27T08:29:14.890" Score="0" Body="&lt;p&gt;I totally understand your concern and I appreciate the fact that you want to teach the underlying math. From my experience I think a Better approach would be to not go straight to the math but teach math with problems you are already solving i.e hackers way of learning math. As programmers it's always a positive affirmation that we can program what we learn. Here is a very good repository that has content on these lines : &lt;a href=&quot;https://github.com/amitkaps/hackermath&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/amitkaps/hackermath&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2017-11-27T08:29:14.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25136" PostTypeId="2" ParentId="25121" CreationDate="2017-11-27T08:39:53.227" Score="1" Body="&lt;p&gt;Use a diminishing sample weight where the newest sample = 1 and older samples get lower weights. This way if a user changes his interests over time, your recommendations will change accordingly.&lt;/p&gt;&#xA;" OwnerUserId="42450" LastActivityDate="2017-11-27T08:39:53.227" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25137" PostTypeId="2" ParentId="25082" CreationDate="2017-11-27T08:51:19.373" Score="1" Body="&lt;p&gt;I guess you have a learning problem that has low number of training data. I recommend you a two-step solution.&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First you need to do error analysis. Find the images that your classifier does not recognize and try to compare with those which your classifier correctly classified them. find the features that your classifier can not do well on them and try to add features that your classifier can discriminate those, I mean try to find yourself what prompts your classifier to make mistake on those images and why you don't make mistake during recognizing them. Try to add those features that you care but your classifier does not and add them as input. &lt;/li&gt;&#xA;&lt;li&gt;Try to augment your data using data augmentation for both you mistake and those which you don't. There is an important fact here. don't change the distribution of your data. I mean if one class has two times bigger population than the other class, after augmentation try to keep this ratio. The reason is that maybe after equalizing the number of population of classes you reduce the Bayes error but at borders of classes the answers would be random.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-11-27T08:51:19.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25140" PostTypeId="2" ParentId="25124" CreationDate="2017-11-27T11:30:56.003" Score="0" Body="&lt;p&gt;I will list possible ways to approach you problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A- Naïve approach&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Save current date&lt;/li&gt;&#xA;&lt;li&gt;Extract day from string using regex and extract its rank in the week (example 0 for Monday 1 for Tuesday and/or so)&lt;/li&gt;&#xA;&lt;li&gt;Find preceding word to output 3&lt;/li&gt;&#xA;&lt;li&gt;If output 3 is &quot;Next&quot; --&gt; exact date = date for the output from 2 in the next calendar week&lt;/li&gt;&#xA;&lt;li&gt;If output 3 is &quot;on&quot;/&quot;this&quot; --&gt; exact date = date for the output from 2 in the this calendar week.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you are using python, you will need &lt;code&gt;re&lt;/code&gt; and &lt;code&gt;calendar&lt;/code&gt; packages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pros:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fast in developing&lt;/li&gt;&#xA;&lt;li&gt;Sufficient for most of the cases&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Slow in implementation, depending on how long is the length of text&lt;/li&gt;&#xA;&lt;li&gt;You have to account for all possible scenarios in output 4 above &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;B- Classification&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a ground truth of strings with positive and negative examples and feed it into a classifier to learn from those examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You would need to parse the text first as a pre-processing step (but this would be the case for most approaches, isn't it?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pros:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fast in implementation &lt;/li&gt;&#xA;&lt;li&gt;Less rule-based &lt;/li&gt;&#xA;&lt;li&gt;Accuracy is expected to be&#xA;high depending on the classifier&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cons:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Creating ground truth is a time consuming (could be minimised by combining this approach with the Naïve one or with some &lt;code&gt;nlp&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;LWill need time to train (but this is shouldn't be too long)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;C- Do it the Apple way&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at some information on the &lt;a href=&quot;http://www.miramontes.com/writing/add-cacm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apple Data Detectors&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27537" LastActivityDate="2017-11-27T11:30:56.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25141" PostTypeId="1" CreationDate="2017-11-27T12:11:38.633" Score="2" ViewCount="6238" Body="&lt;p&gt;I have training samples of the following shape: (1000,2). These are numeric sequences, each of length = 1000, dimensions = 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to build a Convolutional Neural Network to output Predictions/Sequences of the same shape (1000, 2). From what I understand, after applying convolution and pooling, the height and width of the input is reduced. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I then set up the fully connected layer(s) and an output layer in my CNN, so that the output dimensions match the input dimensions? Or in general, how should I set up my CNN architecture to achieve this? &lt;/p&gt;&#xA;" OwnerUserId="42457" LastEditorUserId="29169" LastEditDate="2021-02-13T19:20:44.543" LastActivityDate="2021-02-13T19:20:44.543" Title="How to make output dimensions match input dimensions in CNN?" Tags="|deep-learning|neural-network|cnn|pytorch|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25142" PostTypeId="2" ParentId="25141" CreationDate="2017-11-27T12:27:13.347" Score="3" Body="&lt;p&gt;You have (at least) two options. You can either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a bottleneck architecture where you use pooling layers to reduce the dimensions of your data and then upcast it again using deconvolutional/upsampling layers to go back to the original dimensions. The advantage of this approach is that you use a bigger receptive field for the final output, which means it can look at a bigger part of your input sequence. You could combine this with a highway connection like structure. A very similar concept is used in 2d with segmentation, here the spatial dimensions are preserved at the output. One of the best models is called a U-Net.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use no pooling and only padding-same convolutional layers with non-linearities. You can increase the depth but you will keep the (1000, 2) dimensions everywhere. The advantage is that it is very easy to implement, and if applicable to your problem only uses relatively local information.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-27T12:27:13.347" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25143" PostTypeId="2" ParentId="4980" CreationDate="2017-11-27T13:00:07.393" Score="1" Body="&lt;p&gt;I found this tutorial quite helpful: &lt;a href=&quot;http://nlpforhackers.io/named-entity-extraction/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Complete guide to build your own Named Entity Recognizer with Python&lt;/a&gt;&#xA;He uses the Groningen Meaning Bank (GMB) corpus to train his NER chunk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that you can check this tutorial from the same person: &lt;a href=&quot;http://nlpforhackers.io/training-ner-large-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Training a NER System Using a Large Dataset&lt;/a&gt; Where he uses scikit learn to improve the performance of his system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally some really useful tutorials can be found here: &lt;a href=&quot;https://pythonprogramming.net/tokenizing-words-sentences-nltk-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK tutorial&lt;/a&gt;&#xA;This guy has a youtube channel with a lot of tutorial in many subjects (ML,NLP, Python...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="42460" LastActivityDate="2017-11-27T13:00:07.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25144" PostTypeId="2" ParentId="25122" CreationDate="2017-11-27T13:01:52.883" Score="3" Body="&lt;p&gt;Here's some starter code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;from keras.applications.vgg19 import VGG19&#xA;from keras.preprocessing import image&#xA;from keras.applications.vgg19 import preprocess_input&#xA;from keras.models import Model&#xA;import numpy as np&#xA;&#xA;# define the CNN network&#xA;# Here we are using 19 layer CNN -VGG19 and initialising it&#xA;# with pretrained imagenet weights&#xA;base_model = VGG19(weights='imagenet')&#xA;&#xA;# Extract features from an arbitrary intermediate layer&#xA;# like the block4 pooling layer in VGG19&#xA;model = Model(inputs=base_model.input, outputs=base_model.get_layer('block4_pool').output)&#xA;&#xA;# load an image and preprocess it&#xA;img_path = 'elephant.jpg'&#xA;img = image.load_img(img_path, target_size=(224, 224))&#xA;x = image.img_to_array(img)&#xA;x = np.expand_dims(x, axis=0)&#xA;x = preprocess_input(x)&#xA;&#xA;# get the features &#xA;block4_pool_features = model.predict(x)&#xA;&lt;/code&gt;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then use these features for passing to a SVM classfier.&#xA;Here are some additional links for reference:&#xA;&lt;br&gt;&#xA;&lt;a href=&quot;https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding CNNS&lt;/a&gt;&#xA;&lt;br&gt;&#xA;&lt;a href=&quot;https://keras.io/applications/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras pretrained networks&lt;/a&gt;&#xA;&lt;br&gt;&#xA;&lt;a href=&quot;https://elitedatascience.com/keras-tutorial-deep-learning-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coding a convolutional neural network in keras&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39208" LastActivityDate="2017-11-27T13:01:52.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25145" PostTypeId="2" ParentId="25127" CreationDate="2017-11-27T13:46:25.047" Score="2" Body="&lt;p&gt;A fully observed state means that there is no hidden information. Clear examples of this are chess and Go because both players have all the information. The fact that both these games are deterministic doesn't matter. A game where the state changes are stochastic can still be fully observable. Games like poker, where both players can observe their own hand but not their opponents' are called partially observable. Other examples of this can be real time strategy games like Starcraft where you can only see in the line of sight of your units.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-27T13:46:25.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25146" PostTypeId="1" CreationDate="2017-11-27T14:12:13.907" Score="2" ViewCount="79" Body="&lt;p&gt;Given a line plot image (bitmap, png, etc) form, I'm looking to extract the set of original datapoints without any user input, as shown in the below image: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xc9c4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xc9c4.png&quot; alt=&quot;example line plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outcome should be something like below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;`-  A,B,C&#xA;x1: 1,2,3&#xA;x2: 2,4,9&#xA;x3: 3,8,27&#xA;x4: 4,16,23&#xA;x5: 5,13,18&#xA;x6: 6,18,21&#xA;x7: 7,19,23`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Solutions like &lt;a href=&quot;https://automeris.io/WebPlotDigitizer/&quot; rel=&quot;nofollow noreferrer&quot;&gt;webplotdigitizer&lt;/a&gt; require user input which isn't what I'm looking for.  &lt;a href=&quot;https://github.com/bloomberg/scatteract&quot; rel=&quot;nofollow noreferrer&quot;&gt;scatteract&lt;/a&gt; is very close but I don't think it works on using line plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody point me to papers or open source projects on this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="42463" LastEditorUserId="8878" LastEditDate="2017-11-28T16:18:33.897" LastActivityDate="2017-11-28T16:18:33.897" Title="Automatically extract original dataset from a line plot image" Tags="|machine-learning|computer-vision|image-recognition|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25147" PostTypeId="2" ParentId="25029" CreationDate="2017-11-27T15:24:52.623" Score="19" Body="&lt;p&gt;You can write a function that returns another function, as is done here on &lt;a href=&quot;https://github.com/fchollet/keras/issues/2121#issuecomment-214551349&quot; rel=&quot;noreferrer&quot;&gt;GitHub&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def penalized_loss(noise):&#xA;    def loss(y_true, y_pred):&#xA;        return K.mean(K.square(y_pred - y_true) - K.square(y_true - noise), axis=-1)&#xA;    return loss&#xA;&#xA;input1 = Input(batch_shape=(batch_size, timesteps, features))&#xA;lstm =  LSTM(features, stateful=True, return_sequences=True)(input1)&#xA;output1 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)&#xA;output2 = TimeDistributed(Dense(features, activation='sigmoid'))(lstm)&#xA;model = Model(input=[input1], output=[output1, output2])&#xA;model.compile(loss=[penalized_loss(noise=output2), &#xA;penalized_loss(noise=output1)], optimizer='rmsprop')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the wrapper function you can pass scalars or keras tensors like additional inputs. In your case you would add the weights in the top wrapper function and reference them in your inward function.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-27T15:24:52.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25148" PostTypeId="1" CreationDate="2017-11-27T15:25:44.513" Score="0" ViewCount="64" Body="&lt;p&gt;I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric in my problem. Is there any better way of doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance :)&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2017-11-27T21:43:55.050" Title="Identify important less frequent words" Tags="|data-mining|nlp|text-mining|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25150" PostTypeId="1" AcceptedAnswerId="25181" CreationDate="2017-11-27T16:13:36.960" Score="2" ViewCount="313" Body="&lt;p&gt;I have 400 time series of length 50.&#xA;200 of them have values between 1-10 and are considered of type A.&#xA;The rest 200 have values 1-10 with the exception that 3 from the total of 50 data points have value 20, and are considered of type B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use a RNN network to make it learn these differences using tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use GRU cell with input size is 50 (the whole time series), internal state is 100, 3 layers, feature length is 1 and dropout 0.8.&#xA;Batch size is 1 (1 whole time series of 50 values)&#xA;80% of the series are used for training and 20% for test, evenly distributed between type A and type B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use softmax as activation function and for gradient descent optimization i use RMSPropOptimizer. Also i am not changing the data at all before feeding it into the neural network, i tried normalizing them using (x-min()/max()-min()) but it didn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When i am trying to train it, i always get 50% error. Maximum epochs i tried are 10 and it still was 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same network can classify correctly if i use it on data that have way more differences than just on these 3 points, even on small dataset of about 50 runs of each type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it impossible for a RNN to learn these small differences on just those 3 data points out of the 50?&#xA;What would you suggest i can change in my configuration?&#xA;I also tried with 2000 time series and still the error is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that these 3 data points exist at the beginning of the time series. (in case this has something to do with my issue).&lt;/p&gt;&#xA;" OwnerUserId="40182" LastActivityDate="2017-11-28T15:53:11.560" Title="RNN unable to classify time series" Tags="|classification|time-series|tensorflow|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25151" PostTypeId="1" CreationDate="2017-11-27T17:59:31.387" Score="3" ViewCount="2875" Body="&lt;p&gt;I am new to neural networks. I tried coding the &lt;code&gt;backpropogation alogrithm&lt;/code&gt; and tried running it on a test set which gave wrong results. &#xA;I have used the following knowledge to code it,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the forward pass,&#xA;$$z^l = w^la^{l-1} + b^l$$&#xA;$$a^l = g^l (z^l)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the backward pass, (Here $\circ\text{ - Element wise Product}$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the last layer,&#xA;$$\delta ^L = (a^l - y) \circ g'^L(z^L)$$&#xA;$$\frac{\partial L}{\partial w^L} = \delta^L (a^{L-1})^T$$&#xA;$$\frac{\partial L}{\partial b^L} = \delta^L$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the other layers,&#xA;$$\delta ^l = (w^{l+1})^T(\delta^{l+1}) \circ g'^l(z^l)$$&#xA;$$\frac{\partial L}{\partial w^l} = \delta^l (a^{l-1})^T$$&#xA;$$\frac{\partial L}{\partial b^l} = \delta^l$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Update,&#xA;$$W : = W - \alpha \frac{\partial L}{\partial w^l}  $$&#xA;$$b : = b - \alpha\frac{\partial L}{\partial b^l} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is the code which I have written,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    X = load('iris.csv');    &#xA;    W1 = rand(3,4);  &#xA;    W2 = rand(1,3);&#xA;    b1 = rand(3,1);&#xA;    b2 = rand(1,1);&#xA;    y = [ones(1,50) 2*ones(1,50) 3*ones(1,50)]';&#xA;&#xA;    for j = 1:100    %for epoch&#xA;    for i = 1:150    %for iteration through dataset&#xA;    x1 = X(i,:)';&#xA;    z1 = W1*x1 + b1;&#xA;    a1 = tanh(z1);&#xA;    z2 = W2*a1 + b2;&#xA;    a2 = relu(z2);&#xA;&#xA;    dz2 = (a2 - y(i)).*reluGradient(a2);  %this is delta L&#xA;    dw2 = dz2*transpose(a1);              %The derivative term of Lth&#xA;    db2 = dz2;                            %for the bias&#xA;&#xA;    g1 = 1 - a1.^2;                       %Derivative &#xA;    dz1 = transpose(W2)*dz2 .* g1;        %FOr delta l&#xA;    dw1 = dz1*transpose(x1);              %derivative term for lth layer&#xA;    db1 = dz1;                            %bias update&#xA;&#xA;&#xA;    W1 = W1 - 0.1*dw1;&#xA;    b1 = b1 - 0.1*db1;&#xA;&#xA;    W2 = W2 - 0.1*dw2;&#xA;    b2 = b2 - 0.1*db2;&#xA;&#xA;    end&#xA;    end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am trying to train the net for the iris data set (150 X 4 - dataset Size).&#xA;I have considered 4 input units, 1 hidden layer with 3 hidden units and 1 output unit. Hence the dimension of the weight matrix for first layer is 3 X 4 and for the last layer is 1 X 3. When I try to test the network I always get the input classified to class 3. I tried changing the hyper parameters, but it seems there is something wrong with the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the code , I first load the CSV file, and then initialize the weight matrices accordingly. I have run two for loops one for the epoch and other for the iteration. I do a forward pass first using the above equations then a backward pass. I have made functions for the RELU and RELU_GRADIENT.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For relu:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function g = relu(z)&#xA;g = max(0,z);&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For relu gradient:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function z = reluGradient(z)&#xA;z(z&amp;gt;=0) = 1;&#xA;z(z&amp;lt;0) = 0;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would be obliged if someone can direct to me to the solution to this problem&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About IRIS.csv:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has four input features and 150 data samples.&#xA;50 of each class. Hence there are 3 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updates:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost function here I have used is the quadratic cost function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$C = \Sigma (a^L - y)^2 $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence all the derivatives have been calculated with respect to that cost function. For example for the last layer,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial C}{\partial z} = (a^L - y) \circ g'^L(z^L)$$ which is $\delta^L$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried using the sigmoidal transfer function instead of the relu transfer function with all the proper changes at all derivatives,weight and output vectors. (The output vector was then of (3,1) dim)&#xA;As 3 classes hence - $[1,0,0]^T$ , $[0,1,0]^T$ and $[0,0,1]^T$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="40815" LastEditorUserId="40815" LastEditDate="2017-11-29T03:07:39.507" LastActivityDate="2017-11-29T16:20:29.860" Title="Back Propagation Using MATLAB" Tags="|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25152" PostTypeId="1" CreationDate="2017-11-27T18:53:42.667" Score="0" ViewCount="282" Body="&lt;p&gt;I have a dataset with a binary classification, on which I train a kNN algorithm and a SVM algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the tests set I get approximately 75% error when using kNN, but only approximately a 20% error using an SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does this tell me about the dataset?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-12-27T19:43:12.993" Title="Why SVM classifier error is much smaller than KNN classifier error?" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25153" PostTypeId="2" ParentId="25152" CreationDate="2017-11-27T18:53:42.667" Score="1" Body="&lt;p&gt;Given that we have not applied any preprocessing steps then the first thing that comes to mind is that SVM works better in higher dimensionalities while kNN suffer from the curse of dimensionality and therefore our dataset should have a large dimensionality especially in comparison to the number of instances. Therefore we have a very sparse space where the points can be separated by a hyperplane while the k-nearest neighbors give little information of the actual class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that we should take under consideration that we have no knowledge if the test size follows the rule of thumb (10%-20%) and we have not repeat the experiment with multiple train-test splits to get a range of possible errors. We might have a very unlucky test set..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally we should take into account that we have little knowledge if the classes are imbalanced and how much.&#xA;A highly imbalanced class could cause most of classifications to belong to the majority class while SVM would be worse because it would try, in vain, to find a hyperplane that would separate the classes and thus have a larger error.&#xA;However here we have the opposite scenario. Therefore we should assume the case where the classes are more or less balanced but they are separable only in one (or a few) dimensions while pretty much mixed in the rest of the dimensions. Since knn does not learn any weights for each dimension, it assumes that the ones closest are the correct points while svm has taken a clear cut decision that anything beyond this line should be classified as such.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/msnP6.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/msnP6.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24885" LastActivityDate="2017-11-27T18:53:42.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25156" PostTypeId="2" ParentId="25148" CreationDate="2017-11-27T21:43:55.050" Score="1" Body="&lt;p&gt;You could try looking just at the IDF scores. Things like names/entities will score high, this may be desirable or undesirable depending on what you want.&lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2017-11-27T21:43:55.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25158" PostTypeId="1" CreationDate="2017-11-28T02:09:03.533" Score="0" ViewCount="55" Body="&lt;p&gt;I believe that in most cases, the positive case is arbitrarily defined to be the case with a lower frequency. If I instead define the negative case as the one with the lower frequency, what differences would this entail for the model? &lt;/p&gt;&#xA;" OwnerUserId="39580" LastEditorUserId="29575" LastEditDate="2017-12-31T18:02:20.290" LastActivityDate="2017-12-31T18:02:20.290" Title="Does switching the positive and negative case make a difference?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25159" PostTypeId="2" ParentId="25082" CreationDate="2017-11-28T02:52:07.677" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;For most of the pictures, my facial recognition algorithm (openCV) will detect and crop out the background, leaving only the face, but in certain cases, it does not find a face.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fzLsb.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fzLsb.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One alternate solution to approach your problem is to use D-lib facial landmark points instead of OpenCV haar cascades. D-lib facial landmarks provide 68 distinct points on the face, so as per the option that you have considered you can crop the region of interest (face) accurately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://dlib.net/face_detector.py.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is the link to sample code.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42248" LastActivityDate="2017-11-28T02:52:07.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25160" PostTypeId="1" CreationDate="2017-11-28T03:36:11.600" Score="1" ViewCount="987" Body="&lt;p&gt;I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is, how does one determine that a set of parameters are statistically significantly better than another?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?&lt;/p&gt;&#xA;" OwnerUserId="42485" LastActivityDate="2021-01-11T15:07:30.777" Title="How to determine best parameters after grid search?" Tags="|python|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25162" PostTypeId="1" CreationDate="2017-11-28T06:18:54.857" Score="2" ViewCount="476" Body="&lt;p&gt;In the &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.5709&quot; rel=&quot;nofollow noreferrer&quot;&gt;2000 paper&lt;/a&gt; that originally described the modern LSTM (Gers et al.), the training algorithm is described as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Output units use BPTT; output gates use slightly, modified, truncated BPTT. Weights to cells, input gates, and the novel forget gates, however, use a truncated version of RTRL.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However this choice is not justified in the text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an intuitive explanation for the decision of training output gates with BPTT and the other gates with RTRL?&lt;/p&gt;&#xA;" OwnerUserId="42491" LastEditorUserId="29575" LastEditDate="2017-11-28T16:22:00.547" LastActivityDate="2017-11-28T16:22:00.547" Title="LSTMs: BPTT vs RTRL in training" Tags="|backpropagation|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25163" PostTypeId="2" ParentId="25158" CreationDate="2017-11-28T07:16:25.493" Score="1" Body="&lt;p&gt;For all of the algorithms/procedures that I know, it makes no difference between a positive or negative interpretation of a class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that one category has a lower frequency (by that I am assuming that you mean a low probability) is insignificant. It is just a matter of communication, by maintaining the general designations in a specific field so that others will be able to read your work more easily. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, in anomaly detection, one designates the cases with lower frequency as anomaly, but even so, technically it does not matter how you label your cases, they are interchangeable. &lt;/p&gt;&#xA;" OwnerUserId="108" LastEditorUserId="29575" LastEditDate="2017-12-31T18:01:11.677" LastActivityDate="2017-12-31T18:01:11.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25166" PostTypeId="1" CreationDate="2017-11-28T09:21:40.790" Score="0" ViewCount="313" Body="&lt;p&gt;I am working with &lt;strong&gt;Time-Series&lt;/strong&gt; Data that has to be classified into two classes (Blue and Red) or at least Classify the data as one class (Red), I'm unable to come up with features that distinctly separate the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please advise as to how do I need to approach this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GCk8b.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GCk8b.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42495" LastEditorUserId="40853" LastEditDate="2017-11-28T10:04:40.817" LastActivityDate="2017-11-28T10:34:57.380" Title="Classifying time series data that overlap" Tags="|machine-learning|classification|time-series|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25168" PostTypeId="2" ParentId="25166" CreationDate="2017-11-28T10:34:57.380" Score="1" Body="&lt;p&gt;You could calculate the distance between time series using &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dynamic Time Wrapping DTW&lt;/a&gt; and then you could cluster them using K-means or so. &lt;a href=&quot;https://pypi.python.org/pypi/fastdtw&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a python implementation of the DTW or use the &lt;code&gt;dtw&lt;/code&gt; package in R.&lt;/p&gt;&#xA;" OwnerUserId="27537" LastActivityDate="2017-11-28T10:34:57.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25173" PostTypeId="1" AcceptedAnswerId="27327" CreationDate="2017-11-28T12:27:51.053" Score="5" ViewCount="684" Body="&lt;p&gt;I am collecting a big number of generated numeric features for the task of unsupervised anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can assume that all training data is considered normal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect some of the generated features to be characterised with low standard deviation, for example, some features might be always 0 in the train examples. In contrast, I expect that some of these features will deviate in anomaly instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I have a lot of feature, I strive to perform feature reduction / selection. However, using simple feature selection methods, will completely remove the non deviating features, affecting the upcoming detection for the worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about using stacked auto-encoders for the sake of feature reduction, so that whenever a feature deviates a lot from the stdv it will affect all resulted features - causing a noticeable anomaly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will this technique work? if not, why? and what other technique could work for that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if it does, and I am planning of using deep auto-encoders for the sake of anomaly detection as well, is the first step of feature reduction redundent?&lt;/p&gt;&#xA;" OwnerUserId="42500" LastActivityDate="2018-02-01T07:23:57.150" Title="Unsupervised feature reduction for anomaly detection with autoencoders" Tags="|feature-selection|unsupervised-learning|anomaly-detection|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25174" PostTypeId="1" AcceptedAnswerId="25179" CreationDate="2017-11-28T12:34:29.640" Score="1" ViewCount="947" Body="&lt;p&gt;I would like to ask if there is a metric in R and Python that serves not only for binary classification. I've found Matthews Correlation Coefficient works well in Python, but in R is only a binary version. Of course, I'm talking about &quot;real&quot; not just binary metrics - not something like &quot;It's binary, but you can use one vs one or one vs all.&quot;&lt;/p&gt;&#xA;" OwnerUserId="41750" LastActivityDate="2017-11-28T17:03:58.383" Title="Multi-class classification metrics in R and Python" Tags="|python|r|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25175" PostTypeId="2" ParentId="25174" CreationDate="2017-11-28T13:19:03.567" Score="1" Body="&lt;p&gt;How about the &lt;code&gt;MLmetrics&lt;/code&gt; package in R? It has multi class log loss function that you can use to assess the performance of your model.&lt;/p&gt;&#xA;" OwnerUserId="27537" LastActivityDate="2017-11-28T13:19:03.567" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25176" PostTypeId="1" CreationDate="2017-11-28T14:39:56.580" Score="1" ViewCount="91" Body="&lt;p&gt;I am currently working on a project where my job is to intent analysis of an article. Suppose i am given with the article and i want to classify that what kind of article it is?. Any suggestions, Which kind of model would be best to apply?&lt;/p&gt;&#xA;" OwnerUserId="41491" LastEditorUserId="41491" LastEditDate="2017-11-28T14:51:14.080" LastActivityDate="2018-01-28T05:39:08.623" Title="Algorithms to get intent of an article?" Tags="|neural-network|deep-learning|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25178" PostTypeId="2" ParentId="25176" CreationDate="2017-11-28T15:07:21.947" Score="2" Body="&lt;p&gt;I think the easiest way would be to cluster them with wordvectors.&#xA;The &quot;20 Newspaper&quot; dataset would be a good test for the algorithm&#xA;&lt;a href=&quot;http://qwone.com/~jason/20Newsgroups/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://qwone.com/~jason/20Newsgroups/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the Sklearn Website, you will find some easy example for using wordvectors and classification:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/datasets/twenty_newsgroups.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/datasets/twenty_newsgroups.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you go deeper into this space you could start using LSTM networks and neural networks, but this is a little bit harder to develop. So first, try to use the problem statistically. If you want to analyze the text semantically you will need LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I could help you. &lt;/p&gt;&#xA;" OwnerUserId="29064" LastActivityDate="2017-11-28T15:07:21.947" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25179" PostTypeId="2" ParentId="25174" CreationDate="2017-11-28T15:35:53.620" Score="1" Body="&lt;p&gt;You mean this? &lt;a href=&quot;https://mlr-org.github.io/mlr-tutorial/devel/html/performance/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLR's performance measures&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performance measure suitable for the iris classification task&#xA;listMeasures(iris.task)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1] &quot;kappa&quot;            &quot;multiclass.brier&quot; &quot;multiclass.aunp&quot; &#xA;   [4] &quot;multiclass.aunu&quot;  &quot;qsr&quot;              &quot;ber&quot;&lt;br&gt;&#xA;   [7] &quot;logloss&quot;          &quot;wkappa&quot;           &quot;timeboth&quot;&lt;br&gt;&#xA;  [10] &quot;timepredict&quot;      &quot;acc&quot;              &quot;lsr&quot;&lt;br&gt;&#xA;  [13] &quot;featperc&quot;         &quot;multiclass.au1p&quot;  &quot;multiclass.au1u&quot; &#xA;  [16] &quot;ssr&quot;              &quot;timetrain&quot;        &quot;mmce&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://topepo.github.io/caret/measuring-performance.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caret does too but fewer&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="30507" LastEditorUserId="30507" LastEditDate="2017-11-28T17:03:58.383" LastActivityDate="2017-11-28T17:03:58.383" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25180" PostTypeId="1" CreationDate="2017-11-28T15:46:13.243" Score="1" ViewCount="5392" Body="&lt;p&gt;I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any problem with this methodology?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the &lt;code&gt;max_depth&lt;/code&gt; parameter I get a really high value of 500 and that seems too high. Any advice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is:&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;from __future__ import print_function  &#xA;import sklearn  &#xA;import sklearn.ensemble  &#xA;import sklearn.metrics  &#xA;from sklearn.datasets import fetch_20newsgroups  &#xA;from sklearn.grid_search import GridSearchCV&#xA;&#xA;categories = ['sci.med', 'soc.religion.christian']  &#xA;newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, &#xA;remove=('headers', 'footers', 'quotes'))  &#xA;newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, &#xA;remove=('headers', 'footers', 'quotes'))  &#xA;class_names = ['medicine', 'christian']  &#xA;&#xA;vectorizer = &#xA;sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)  &#xA;train_vectors = vectorizer.fit_transform(newsgroups_train.data)  &#xA;test_vectors = vectorizer.transform(newsgroups_test.data)  &#xA;&#xA;rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')  &#xA;&#xA;param_grid = {  &#xA;           &quot;n_estimators&quot; : [10, 100, 1000],  &#xA;           &quot;max_depth&quot; : [5, 100, 500],  &#xA;           &quot;min_samples_leaf&quot; : [1, 20, 40]}  &#xA;&#xA;CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)  &#xA;CV_rf.fit(train_vectors, newsgroups_train.target)  &#xA;print(CV_rf.best_params_)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42510" LastEditorUserId="52196" LastEditDate="2019-07-03T13:53:47.240" LastActivityDate="2021-07-03T09:45:17.920" Title="Parameter Tuning by Cross Validation for Random Forest" Tags="|scikit-learn|text-mining|random-forest|cross-validation|hyperparameter|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25181" PostTypeId="2" ParentId="25150" CreationDate="2017-11-28T15:53:11.560" Score="2" Body="&lt;p&gt;I let it run for more epochs, as suggested, and i noticed that after 23 epochs, rnn did get 0% error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then tried changing activation function of GRUCell from the default tanh to relu and it greatly speed up training. It took 1-4 epochs to get 0% error.&lt;/p&gt;&#xA;" OwnerUserId="40182" LastActivityDate="2017-11-28T15:53:11.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25182" PostTypeId="2" ParentId="25180" CreationDate="2017-11-28T16:03:13.723" Score="2" Body="&lt;p&gt;Are you looking at the accuracy on your validation set, rather than your training set? (you should be). Are you making sure your gap between training and validation accuracy is low? (you should be!) Is there enough data to warrant 3-fold cross-validation, or should you do 10-fold (and use more data for training)?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, random search (where you sample randomly from the parameter space) will get you a good result faster than grid search.  It has the added benefit that you can specify how many models you want to build, as each model's parameters are sampled independently and not constrained to cover an entire space like grid search.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-28T16:03:13.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25183" PostTypeId="2" ParentId="25180" CreationDate="2017-11-28T16:47:06.017" Score="0" Body="&lt;p&gt;max_depth of 500 cannot be right, if I do not mistake it implies about 2^500 leafs, which is not feasible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You probably have some bugs in your code. Post your code if possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, are you sure you don’t mix up max_depth and number of trees?&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-11-28T16:47:06.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25184" PostTypeId="1" AcceptedAnswerId="25200" CreationDate="2017-11-28T17:01:09.673" Score="0" ViewCount="252" Body="&lt;pre&gt;&lt;code&gt;In [1]: df.head()&#xA;Out[1]: &#xA;&#xA;   uf    city&#xA;0  mg    araguari&#xA;1  mg    uberlandia&#xA;2  sp    sao-paulo&#xA;3  go    goiania&#xA;4  mg    belo-horizonte&#xA;&#xA;In [2]: len(df)&#xA;Out[2]: &#xA;&#xA;982&#xA;&#xA;In [3]: indicators.head()&#xA;Out[3]: &#xA;&#xA;     city               uf         population&#xA; 0   abadia-goias       go         6876&#xA; 1   abadia-dourados    mg         6704&#xA; 2   araguari           mg         19000&#xA; 3   abaete             mg         22690&#xA; 4   uberlandia         mg         700000&#xA;&#xA;&#xA;In [4]: len(indicators)&#xA;Out[4]: &#xA;&#xA;5554&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now here I'm trying to include 'population' in df, using the corresponding uf and city. But this next code is too slow.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['population'] = np.nan&#xA;&#xA;for city, uf, index in zip(df.city.values, \&#xA;    df.uf.values, df.index):&#xA;&#xA;    df['population'].iloc[index] = \&#xA;    indicators.population[indicators.uf == uf] \&#xA;    [indicators.city == city].values[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9988" LastEditorUserId="9988" LastEditDate="2017-11-30T07:40:54.643" LastActivityDate="2017-11-30T08:55:14.487" Title="How would you optimize this python/pandas code?" Tags="|python|pandas|" AnswerCount="1" CommentCount="4" ClosedDate="2017-11-29T16:20:44.633" ContentLicense="CC BY-SA 3.0" />
  <row Id="25185" PostTypeId="1" CreationDate="2017-11-28T17:29:37.160" Score="1" ViewCount="76" Body="&lt;p&gt;I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the Gradient Descent. I see two different formals to update weight by the Gradient Descent between NG and Nielson.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from Nielson:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7VmWi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7VmWi.png&quot; alt=&quot;the weight update formula&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the chapter Two, section The backpropagation algorithm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from NG:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/SkbFi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SkbFi.png&quot; alt=&quot;the weight update formal&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;both of them are used to update weights after forwarding pass in the backpropagation algorithm. in Neil version, it seems there is a learning rating -η in NG version, it does not have any learning rating, and it more like the learning rating is +1. I am very confused on this. is there anyone can help me to understand it?&lt;/p&gt;&#xA;" OwnerUserId="42501" LastActivityDate="2017-11-28T17:29:37.160" Title="The Gradient descent different between in Ng coursera and Michael A. Nielsen book" Tags="|neural-network|gradient-descent|backpropagation|learning-rate|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25187" PostTypeId="2" ParentId="25180" CreationDate="2017-11-28T18:11:37.663" Score="1" Body="&lt;p&gt;It may be that your coarse scale is too large (though it is a good idea to begin). If for example 150 depth would be the best solution but 500 is better than 100 then it will give you 500 as a parameter result.&#xA;Have you tried with shorter intervals between values for this parameter ?&#xA;What does it give if you try differently?&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-11-28T18:11:37.663" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25188" PostTypeId="1" AcceptedAnswerId="25196" CreationDate="2017-11-28T18:21:33.233" Score="0" ViewCount="777" Body="&lt;p&gt;I have recently been learning about the various clustering methods, and I decided to apply Furthest Point clustering in Weka with varying seed values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The seed value dictates the initial choice of points in the data set. It can't be totally random, since repeatedly doing the clustering with, say, seed = 3 gives the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does it select data points based on the seed?&lt;/p&gt;&#xA;" OwnerUserId="40736" LastActivityDate="2017-11-29T01:25:26.677" Title="How does the seed value work in Weka for clustering?" Tags="|clustering|weka|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25189" PostTypeId="2" ParentId="25151" CreationDate="2017-11-28T18:36:26.087" Score="3" Body="&lt;p&gt;Your problem is a classification problem with 3 classes. The appropriate function for it is softmax function, whereas you are using Relu. Another thing is that you need a cost function. The function that you optimize and the one you do calculate derivatives of with respect to parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cost function and last layer activation function must match each other. So first you choose appropriate &quot;last&quot; activation funtion for your problem and then cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For iris you can use softmax function with cross entropy function. The derivatives of these you can easily find. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing are the dimensions. The result of softmax function is the vector of the size (number of classes, 1) for each observation whereas you have (1,3) dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the backpropagation algorithm for the other layers it is looks ok, but the last layer equation is wrong and should be like the one below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4WVkS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4WVkS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where C is the cost function and we calculate derivative of C with respect to a (activation of last layer) and multiply element-wise by derivative of a (here it should be softmax function with respect to z). dw and db for last layers have correct equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this hints will help you come up with final neural network implementation for iris classification problem.&lt;/p&gt;&#xA;" OwnerUserId="42518" LastEditorUserId="29575" LastEditDate="2017-11-29T16:20:29.860" LastActivityDate="2017-11-29T16:20:29.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25190" PostTypeId="1" CreationDate="2017-11-28T19:32:26.367" Score="2" ViewCount="157" Body="&lt;p&gt;I'm working on a &lt;a href=&quot;https://github.com/Sergio0694/NeuralNetwork.NET&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;CNN library&lt;/strong&gt;&lt;/a&gt; for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Please note that the whole thing was built from scratch following the&#xA;  &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;neuralnetworksanddeeplearning&lt;/a&gt; book and other resources, so I'm aware there are ready-to-use libraries out there, as well as more optimized algorithms to perform certain operations (like the FFT convolution algorithm), but the point of this thing was to understand how a network works and to implement everything without using 3rd party libraries.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm quite sure the fully connected layers work fine, as I've tried training a network with the MNIST dataset and it quickly gets to over 96% in less than 60 epochs, just like the one in the book mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I went on and implemented the convolutional layers, and they seem to be working fine too, in fact if I try to have a network like this one (sigmoid activation on all the layers):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Convolution: 28*28*1 inputs, 20 5*5*1 kernels&lt;/li&gt;&#xA;&lt;li&gt;Fully Connected: 24*24*20 inputs, 100 outputs&lt;/li&gt;&#xA;&lt;li&gt;Fully Connected: 100 inputs, 10 outputs, cross-entropy cost&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And here's what I got: &lt;a href=&quot;https://i.stack.imgur.com/enp7h.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/enp7h.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the network quickly reaches over 96% accuracy in just 3 epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But&lt;/strong&gt;, as soon as I try to put a 2*2 pooling layer between the convolutional and fully connected layer (just like Michael Nielsen did &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap6.html#convolutional_neural_networks_in_practice&quot; rel=&quot;nofollow noreferrer&quot;&gt;in the 6th chapter of his book&lt;/a&gt;), here's what I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Kg5Gl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Kg5Gl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network reaches 30% and then just gets stuck there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've checked and everything seems to be working fine (but of course I can be wrong, and I probably am, since this thing is not working as it should), the forward pooling layer takes the maximum value in each 2*2 window across all the input images (doing so for each depth layer), and during the backpropagation through the pooling layer I just upscale the input delta using the previous outputs from the convolutional layer, so that each delta goes to the pixel that had the maximum value in the convolutional output, while all the other pixels get 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where should I start looking for the error? I mean, clearly something happens with the pooling layer, but the code looks fine to me, both in theory and with the actual Unit tests I've added to the project. What's the right approach to take here to investigate the issue? Are there probable explanations for this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code I'm using, if anyone has .NET Core 2.x installed and wants to try the library out (The code to download/parse the MNIST dataset is already in the library):&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;((float[,] X, float[,] Y) training, (float[,] X, float[,] Y) test) = DataParser.LoadDatasets();&#xA;INeuralNetwork network = NetworkTrainer.NewNetwork(&#xA;    NetworkLayers.Convolutional((28, 28, 1), (5, 5), 20, ActivationFunctionType.Identity),&#xA;    NetworkLayers.Pooling((24, 24, 20), ActivationFunctionType.Sigmoid),&#xA;    NetworkLayers.FullyConnected(12 * 12 * 20, 100, ActivationFunctionType.Sigmoid),&#xA;    NetworkLayers.FullyConnected(100, 10, ActivationFunctionType.Sigmoid, CostFunctionType.CrossEntropy));&#xA;await NetworkTrainer.TrainNetworkAsync(network, (training.X, training.Y), 60, 10, null,&#xA;    new TestParameters(test, new Progress&amp;lt;BackpropagationProgressEventArgs&amp;gt;(p =&amp;gt;&#xA;    {&#xA;        Printf($&quot;Epoch {p.Iteration}, cost: {p.Cost}, accuracy: {p.Accuracy}&quot;);&#xA;    })));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="42521" LastActivityDate="2017-11-28T19:32:26.367" Title="SGD learning gets stuck when using a max pooling layer (but it works fine with just conv + fc)" Tags="|neural-network|gradient-descent|backpropagation|convolution|mnist|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25191" PostTypeId="1" AcceptedAnswerId="25192" CreationDate="2017-11-28T19:53:46.040" Score="0" ViewCount="269" Body="&lt;p&gt;Consider the following problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that the normalisation solution is something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand the logic behind this normalisation. Could you explain please?&lt;/p&gt;&#xA;" OwnerUserId="40736" LastEditorUserId="27665" LastEditDate="2017-11-29T16:20:03.930" LastActivityDate="2017-12-29T18:01:02.827" Title="In a recommender system, how can you normalise the similarity between two arbitrary users?" Tags="|recommender-system|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25192" PostTypeId="2" ParentId="25191" CreationDate="2017-11-28T20:21:54.453" Score="1" Body="&lt;p&gt;This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, it can be confident factor which you apply into the similarity measure.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-11-28T20:21:54.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25195" PostTypeId="1" CreationDate="2017-11-29T00:50:04.150" Score="1" ViewCount="1330" Body="&lt;p&gt;I have set of newspaper articles and I use TextRank algorithms to identify their keywords to perform a classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the important informative keywords, I am also receiving garbage keywords as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;viewed&quot;, &quot;flutter&quot;, &quot;function&quot;, &quot;k&quot;, &quot;neighbour&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to post process and remove these keywords?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2017-11-29T11:25:16.113" Title="Text post-processing" Tags="|data-mining|nlp|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25196" PostTypeId="2" ParentId="25188" CreationDate="2017-11-29T01:25:26.677" Score="0" Body="&lt;p&gt;Pseudo-random number generators are initialized with a seed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is little real randomness available. Most just looks random to us, because the pattern is too complex. Read up some examples on pseudo random number generation such ad Mersenne Twisters.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-11-29T01:25:26.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25197" PostTypeId="2" ParentId="25195" CreationDate="2017-11-29T01:41:22.703" Score="3" Body="&lt;p&gt;Yes, you can do that by add them to the existing NLTK Stop-word dictionary for all such words/Creating a Custom Stop-word dictionary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for Custom Stop-word dictionary you need to include all the key words in the dictionary before processing, make sure to check if the words are present in the stop-word dictionary. If yes, remove them from the text else nothing needs to be done.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;set of code is for &lt;strong&gt;updating&lt;/strong&gt; new words to the &lt;strong&gt;NLTK Stop-words dictionary&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;#Custom function to remove the stop-words&#xA;    def removeStopWords(str):&#xA;    #select english stopwords from NLTK&#xA;    cachedStopWords = set(stopwords.words(&quot;english&quot;))&#xA;    #add custom words, mentioned above&#xA;    cachedStopWords.update((&quot;viewed&quot;, &quot;flutter&quot;, &quot;function&quot;, &quot;k&quot;, &quot;neighbour&quot;))&#xA;    #remove stop words&#xA;    new_str = ' '.join([word for word in str.split() if word not in cachedStopWords]) &#xA;    return new_str&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;set of code is for making a &lt;strong&gt;Custom stop-word dictionary&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;stopwordList = [&quot;viewed&quot;, &quot;flutter&quot;, &quot;function&quot;, &quot;k&quot;, &quot;neighbour&quot;]&#xA;StopWordsRemover(inputCol=&quot;words&quot;, outputCol=&quot;filtered&quot; ,stopWords=stopwordList)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-29T01:41:22.703" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25198" PostTypeId="1" AcceptedAnswerId="25199" CreationDate="2017-11-29T04:36:23.040" Score="0" ViewCount="40" Body="&lt;p&gt;I'm a newbie on deep learning and I have a simple question.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm reading some article about the neural network. It says that people created a simple neural network named perceptron. And this network can't solve a very simple question: exclusive or.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it's because I don't know how to train a neural network or how to set the weights for a neural network.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you describe the process of training for perceptron with the example AND: &lt;code&gt;0 &amp;amp; 0 = 0  1 &amp;amp; 0 = 0  0 &amp;amp; 1 = 0  1 &amp;amp; 1 = 1&lt;/code&gt;&lt;br&gt;&#xA;With such a training, how do the weights in perceptron look like?&lt;/p&gt;&#xA;" OwnerUserId="42535" LastActivityDate="2017-11-29T09:33:10.907" Title="What is training" Tags="|neural-network|deep-learning|training|" AnswerCount="1" CommentCount="0" ClosedDate="2017-11-29T16:20:50.653" ContentLicense="CC BY-SA 3.0" />
  <row Id="25199" PostTypeId="2" ParentId="25198" CreationDate="2017-11-29T05:15:40.560" Score="2" Body="&lt;p&gt;Welcome to the Site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that simple neural network means &lt;strong&gt;Single layer Perceptron&lt;/strong&gt;.&#xA;I think you need to understand this, a simple neural network with a single hidden layer cannot solve an XOR problem. As they are not &lt;strong&gt;Linearly Separable&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To overcome this, we use &lt;strong&gt;Multi-Layer Perceptron&lt;/strong&gt;, you can go through this &lt;a href=&quot;http://lab.fs.uni-lj.si/lasin/wp/IMIT_files/neural/nn04_mlp_xor/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; for better understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your question with respect weights in Neural Network for solving XOR, please go through this &lt;a href=&quot;http://www.cs.stir.ac.uk/courses/ITNP4B/lectures/kms/4-MLP.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;lecture&lt;/a&gt;, this &lt;a href=&quot;https://www.youtube.com/watch?v=apPiZd-qnZ8&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;, in the video they explain you how the weights are adjusted in every iteration before getting the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are stuck somewhere do let me know, will try to help you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other links:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.philbrierley.com/main.html?code/bpproof.html&amp;amp;code/codeleft.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another link suggested by @Yves : &lt;a href=&quot;http://toritris.weebly.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-29T09:33:10.907" LastActivityDate="2017-11-29T09:33:10.907" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25200" PostTypeId="2" ParentId="25184" CreationDate="2017-11-29T05:20:30.040" Score="1" Body="&lt;p&gt;It seems you just need to &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;merge&lt;/a&gt; two DFs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.merge(indicadores[['estado','cidade','population']],&#xA;              leftt_on=['address_locality','address_region'],&#xA;              right_on=['estado','cidade','city'],&#xA;              how='left')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;NOTE: usually you don't need to specify &lt;code&gt;left_on&lt;/code&gt; and &lt;code&gt;right_on&lt;/code&gt; if the &quot;joining&quot; columns have the same names in both DFs (Pandas will use all common (with same names) columns &#xA; from both DFs for joining/merging). But as i've seen from comments and from your question before you've edited it - it's not your case. So you would need to specify correct column names for &lt;code&gt;left_on&lt;/code&gt; and &lt;code&gt;right_on&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; demo (using updated sample data sets):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [7]: df&#xA;Out[7]:&#xA;   uf            city&#xA;0  mg        araguari&#xA;1  mg      uberlandia&#xA;2  sp       sao-paulo&#xA;3  go         goiania&#xA;4  mg  belo-horizonte&#xA;&#xA;In [8]: indicators&#xA;Out[8]:&#xA;              city  uf  population&#xA;0     abadia-goias  go        6876&#xA;1  abadia-dourados  mg        6704&#xA;2         araguari  mg       19000&#xA;3           abaete  mg       22690&#xA;4       uberlandia  mg      700000&#xA;&#xA;In [9]: df = df.merge(indicators,&#xA;   ...:               left_on=['city','uf'],&#xA;   ...:               right_on=['city','uf'],&#xA;   ...:               how='left')&#xA;   ...:&#xA;&#xA;In [10]: df&#xA;Out[10]:&#xA;   uf            city  population&#xA;0  mg        araguari     19000.0&#xA;1  mg      uberlandia    700000.0&#xA;2  sp       sao-paulo         NaN&#xA;3  go         goiania         NaN&#xA;4  mg  belo-horizonte         NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2017-11-30T08:55:14.487" LastActivityDate="2017-11-30T08:55:14.487" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25201" PostTypeId="1" AcceptedAnswerId="25204" CreationDate="2017-11-29T07:11:40.293" Score="0" ViewCount="215" Body="&lt;p&gt;&lt;a href=&quot;http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Taken from Policy Gradient lecture notes, slide 16 onward:&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N1xlA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N1xlA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here in the David Silver Lecture series I perfectly understand how he took the expectation value in the theorem part by combining the following equation . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cjhCw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cjhCw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how he removed the expectation part when talking stochastic gradient decent ? &lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="836" LastEditDate="2017-11-29T09:38:44.570" LastActivityDate="2017-11-29T09:49:15.823" Title="How to get rid of the expectation in Monte Carlo Policy Gradient method?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25202" PostTypeId="2" ParentId="25195" CreationDate="2017-11-29T07:17:16.703" Score="3" Body="&lt;p&gt;Identifying uninformative words is not an easy task and is domain-dependent. For example, stop words or punctuation often are discriminative a lot for sentiment analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to test the keyness of a words for classification, you may use selection scores as chi2, information gain or mutual information. chi2 score basically assess how good is a score at discriminating two classes based on occurences count.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus you could remove the lowest scored words with this technique.&#xA;Post-processing can be done as explained by @Toros91, adding that if for instance you consider characters to be garbage you can easily prior remove them based on string length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finding the &quot;garbageness&quot; of words is I think not an easy problem and is often over-simplified in text classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't hesitate if you have further questioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: adding code sample&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using scikit-learn :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_iris&#xA;from sklearn.feature_selection import SelectPercentile&#xA;from sklearn.feature_selection import chi2&#xA;import numpy&#xA;&#xA;iris = load_iris()&#xA;X, y = iris.data, iris.target&#xA;selector = SelectPercentile(score_func=chi2, percentile=100)&#xA;X_reduced = selector.fit_transform(X, y)&#xA;indices = numpy.argsort(selector.scores_)&#xA;# array containing the indices of features according to ascendant chi2 score&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then use the indices to remove k worst feature according to chi2 square in your data.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastEditorUserId="41728" LastEditDate="2017-11-29T11:25:16.113" LastActivityDate="2017-11-29T11:25:16.113" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="25204" PostTypeId="2" ParentId="25201" CreationDate="2017-11-29T08:21:38.870" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But how he removed the expectation part when talking stochastic gradient decent ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A result that shows what happens in expectation can be estimated by sampling it, and that is precisely what stochastic gradient descent (or ascent in this case) methods do - they operate on individual samples on the assumption that this will &lt;em&gt;on average&lt;/em&gt; produce reasonable direction for optimising parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So there is no need to &quot;get rid of&quot; the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-29T09:49:15.823" LastActivityDate="2017-11-29T09:49:15.823" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25206" PostTypeId="1" AcceptedAnswerId="25208" CreationDate="2017-11-29T09:06:26.693" Score="1" ViewCount="149" Body="&lt;p&gt;I decided to apply the logistic regression method to my categorical and quantitative data.&#xA;So, I followed these steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Eliminating the bad and inconsistent data.&lt;/li&gt;&#xA;&lt;li&gt;Preparing the target variable (categorical variable).&lt;/li&gt;&#xA;&lt;li&gt;Testing the dependencies between categorical  variables and the&#xA;target variable using the Ki-2 test for selecting the variables that&#xA;are well linked with the target variable.&lt;/li&gt;&#xA;&lt;li&gt;Testing the correlation between quantitative variables to avoid the&#xA;choice of two correlated variables at a time.&lt;/li&gt;&#xA;&lt;li&gt;Crossing some variables  to improve their significance.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After all this I do not find any significant variables in my logistic regression model knowing that the base is well coherent and well cleaned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work with the R language and I used the glm function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;glm (formula, family = familytype (link = linkfunction), data =)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see my data base. Note that Achat_client is target variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &amp;gt; M=data.frame(Type_peau,PEAU_CORPS,SENSIBILITE,IMPERFECTIONS,BRILLANCE ,GRAIN_PEAU,RIDES_VISAGE,ALLERGIES,MAINS,PEAU_CORPS,&#xA;+              INTERET_ALIM_NATURELLE,INTERET_ORIGINE_GEO,Crois_Prior1_Milieu,Crois_Profil_Prio,Crois_ALL_AGE,&#xA;+              INTERET_VACANCES,INTERET_COMPOSITION, PEAU_CORPS,Nbre_gift,w,Achat_client)&#xA;&amp;gt; str(M)&#xA;'data.frame':   836 obs. of  21 variables:&#xA; $ Type_peau             : Factor w/ 5 levels &quot;&quot;,&quot;Grasse&quot;,&quot;Mixte&quot;,..: 3 4 5 3 4 3 3 3 2 3 ...&#xA; $ PEAU_CORPS            : Factor w/ 4 levels &quot;&quot;,&quot;Normale&quot;,&quot;Sèche&quot;,..: 2 3 3 2 2 2 3 2 3 2 ...&#xA; $ SENSIBILITE           : Factor w/ 4 levels &quot;&quot;,&quot;Aucune&quot;,&quot;Fréquente&quot;,..: 4 4 4 2 4 3 4 2 4 4 ...&#xA; $ IMPERFECTIONS         : Factor w/ 4 levels &quot;&quot;,&quot;Fréquente&quot;,..: 3 4 3 4 3 2 3 4 3 3 ...&#xA; $ BRILLANCE             : Factor w/ 4 levels &quot;&quot;,&quot;Aucune&quot;,&quot;Partout&quot;,..: 4 2 2 4 4 4 4 4 3 4 ...&#xA; $ GRAIN_PEAU            : Factor w/ 4 levels &quot;&quot;,&quot;Dilaté&quot;,&quot;Fin&quot;,..: 4 4 4 2 4 2 4 4 2 4 ...&#xA; $ RIDES_VISAGE          : Factor w/ 4 levels &quot;&quot;,&quot;Aucune&quot;,&quot;Très visibles&quot;,..: 2 2 2 4 4 2 4 2 4 2 ...&#xA; $ ALLERGIES             : Factor w/ 4 levels &quot;&quot;,&quot;Non&quot;,&quot;Oui&quot;,..: 2 2 2 2 2 2 2 2 2 2 ...&#xA; $ MAINS                 : Factor w/ 4 levels &quot;&quot;,&quot;Moites&quot;,&quot;Normales&quot;,..: 3 4 4 3 3 3 3 4 4 4 ...&#xA; $ PEAU_CORPS.1          : Factor w/ 4 levels &quot;&quot;,&quot;Normale&quot;,&quot;Sèche&quot;,..: 2 3 3 2 2 2 3 2 3 2 ...&#xA; $ INTERET_ALIM_NATURELLE: Factor w/ 4 levels &quot;&quot;,&quot;Beaucoup&quot;,..: 2 4 4 4 2 2 2 4 4 2 ...&#xA; $ INTERET_ORIGINE_GEO   : Factor w/ 5 levels &quot;&quot;,&quot;Beaucoup&quot;,..: 2 4 2 5 2 2 2 2 2 2 ...&#xA; $ Crois_Prior1_Milieu   : Factor w/ 14 levels &quot;Per_nature_éclatante&quot;,..: 11 13 8 12 8 6 8 9 13 11 ...&#xA; $ Crois_Profil_Prio     : Factor w/ 294 levels &quot;Eclatante / Hydratée_éclatante&quot;,..: 141 227 221 74 56 184 13 86 227 68 ...&#xA; $ Crois_ALL_AGE         : Factor w/ 6 levels &quot;jeune_Avec_ALL&quot;,..: 2 4 2 4 2 2 4 4 6 2 ...&#xA; $ INTERET_VACANCES      : Factor w/ 6 levels &quot;&quot;,&quot;À la mer&quot;,..: 3 4 2 2 3 2 3 2 3 2 ...&#xA; $ INTERET_COMPOSITION   : Factor w/ 4 levels &quot;&quot;,&quot;Beaucoup&quot;,..: 2 2 2 4 2 2 2 2 4 2 ...&#xA; $ PEAU_CORPS.2          : Factor w/ 4 levels &quot;&quot;,&quot;Normale&quot;,&quot;Sèche&quot;,..: 2 3 3 2 2 2 3 2 3 2 ...&#xA; $ Nbre_gift             : int  1 4 1 1 2 1 1 1 1 1 ...&#xA; $ w                     : num  0.25 0.25 0.5 0.25 0.5 0 0 0 0 0.75 ...&#xA; $ Achat_client          : num  0 0 0 0 0 0 1 0 0 0 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I split my data base into tow parts: 70% of data for modeling and 30% for testing then the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;split = sample.split(M$Achat_client, SplitRatio = 0.70)&#xA;&#xA;&#xA;final.train = subset(M, split == TRUE)&#xA;final.test = subset(M, split == FALSE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then for application of logistic regression:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;final.log.model &amp;lt;- glm(formula=Achat_client ~ .-1,family=binomial(link=&quot;logit&quot;),data = final.train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; summary(final.log.model)&#xA;&#xA;Call:&#xA;glm(formula = Achat_client ~ . - 1, family = binomial(link = &quot;logit&quot;), &#xA;    data = final.train)&#xA;&#xA;Deviance Residuals: &#xA;     Min        1Q    Median        3Q       Max  &#xA;-1.60306  -0.26636  -0.04545  -0.00003   2.83714  &#xA;&#xA;Coefficients: (5 not defined because of singularities)&#xA;                                                  Estimate Std. Error z value Pr(&amp;gt;|z|)    &#xA;Type_peauGrasse                                 -7.530e+00  2.811e+00  -2.679 0.007383 ** &#xA;Type_peauMixte                                  -8.278e+00  2.535e+00  -3.265 0.001095 ** &#xA;Type_peauNormale                                -7.902e+00  2.568e+00  -3.077 0.002087 ** &#xA;Type_peauSèche                                  -9.710e+00  2.583e+00  -3.759 0.000171 ***&#xA;PEAU_CORPSSèche                                  8.639e-01  5.966e-01   1.448 0.147612    &#xA;PEAU_CORPSTrès sèche                             1.660e+00  8.959e-01   1.853 0.063843 .  &#xA;SENSIBILITEFréquente                            -1.978e-02  9.583e-01  -0.021 0.983531    &#xA;SENSIBILITEOccasionnelle                        -1.123e-01  8.905e-01  -0.126 0.899640    &#xA;IMPERFECTIONSOccasionnelle                      -2.825e-01  5.859e-01  -0.482 0.629712    &#xA;IMPERFECTIONSRares                              -1.234e+00  9.116e-01  -1.353 0.175956    &#xA;BRILLANCEPartout                                 1.720e+00  1.351e+00   1.273 0.202993    &#xA;BRILLANCEZone T                                  2.942e-01  7.643e-01   0.385 0.700299    &#xA;GRAIN_PEAUFin                                    1.901e+00  9.640e-01   1.972 0.048609 *  &#xA;GRAIN_PEAUMoyen                                  6.999e-01  6.976e-01   1.003 0.315696    &#xA;RIDES_VISAGETrès visibles                        2.350e+00  1.101e+00   2.134 0.032843 *  &#xA;RIDES_VISAGEVisibles                            -1.224e-02  5.696e-01  -0.021 0.982857    &#xA;ALLERGIESOui                                    -1.770e+01  4.714e+03  -0.004 0.997004    &#xA;MAINSNormales                                    1.767e+00  1.055e+00   1.675 0.093963 .  &#xA;MAINSSèches                                      1.679e+00  1.073e+00   1.565 0.117595    &#xA;PEAU_CORPS.1Sèche                                       NA         NA      NA       NA    &#xA;PEAU_CORPS.1Très sèche                                  NA         NA      NA       NA    &#xA;INTERET_ALIM_NATURELLEPas du tout               -6.929e-01  1.490e+00  -0.465 0.641833    &#xA;INTERET_ALIM_NATURELLEUn peu                    -1.801e+00  7.381e-01  -2.441 0.014666 *  &#xA;INTERET_ORIGINE_GEOPas du tout                   1.025e+00  1.293e+00   0.793 0.427673    &#xA;INTERET_ORIGINE_GEOUn peu                       -4.041e-01  6.494e-01  -0.622 0.533728    &#xA;Crois_Prior1_MilieuPer_nature_hydratée          -7.592e-01  2.015e+00  -0.377 0.706322    &#xA;Crois_Prior1_MilieuPer_nature_lisse             -3.454e-01  1.756e+00  -0.197 0.844073    &#xA;Crois_Prior1_MilieuPer_nature_matifiée          -1.509e-01  2.495e+00  -0.060 0.951761    &#xA;Crois_Prior1_MilieuPer_nature_nourrie           -1.860e+01  6.608e+03  -0.003 0.997754    &#xA;Crois_Prior1_MilieuPer_nature_purifiée          -3.315e-01  1.611e+00  -0.206 0.837001    &#xA;Crois_Prior1_MilieuPer_nature_reposée            1.284e+00  1.883e+00   0.682 0.495542    &#xA;Crois_Prior1_MilieuPer_urbain_éclatante         -1.819e-02  1.534e+00  -0.012 0.990538    &#xA;Crois_Prior1_MilieuPer_urbain_hydratée           3.449e-01  1.356e+00   0.254 0.799243    &#xA;Crois_Prior1_MilieuPer_urbain_lisse              1.080e+00  1.334e+00   0.810 0.418107    &#xA;Crois_Prior1_MilieuPer_urbain_matifiée           1.051e+00  1.398e+00   0.752 0.452290    &#xA;Crois_Prior1_MilieuPer_urbain_nourrie            3.070e+00  1.627e+00   1.887 0.059226 .  &#xA;Crois_Prior1_MilieuPer_urbain_purifiée          -1.456e+00  1.430e+00  -1.018 0.308601    &#xA;Crois_Prior1_MilieuPer_urbain_reposée            2.520e+00  1.493e+00   1.688 0.091498 .  &#xA;Crois_Profil_PrioLisse / Eclatante_éclatante     2.472e+00  1.486e+00   1.663 0.096222 .  &#xA;Crois_Profil_PrioMatifiée / Eclatante_éclatante  2.845e+00  1.631e+00   1.744 0.081110 .  &#xA;Crois_Profil_PrioNourrie / Eclatante_éclatante  -1.685e+01  7.795e+03  -0.002 0.998276    &#xA;Crois_Profil_PrioPurifiée / Eclatante_éclatante  1.182e+00  1.523e+00   0.776 0.437707    &#xA;Crois_Profil_PrioReposée / Eclatante_éclatante  -1.895e+01  5.224e+03  -0.004 0.997106    &#xA;Crois_Profil_PrioEclatante / Hydratée_hydratée  -1.763e+01  2.457e+03  -0.007 0.994276    &#xA;Crois_Profil_PrioLisse / Hydratée_hydratée      -1.409e-02  1.625e+00  -0.009 0.993081    &#xA;Crois_Profil_PrioMatifiée / Hydratée_hydratée   -1.623e+01  4.667e+03  -0.003 0.997225    &#xA;Crois_Profil_PrioNourrie / Hydratée_hydratée     1.917e+00  1.854e+00   1.034 0.301226    &#xA;Crois_Profil_PrioPurifiée / Hydratée_hydratée    1.841e+00  1.417e+00   1.299 0.193915    &#xA;Crois_Profil_PrioReposée / Hydratée_hydratée    -1.581e+01  7.573e+03  -0.002 0.998334    &#xA;Crois_Profil_PrioEclatante / Lisse_lisse         1.464e+00  1.427e+00   1.026 0.305064    &#xA;Crois_Profil_PrioHydratée / Lisse_lisse         -1.757e+01  3.171e+03  -0.006 0.995578    &#xA;Crois_Profil_PrioMatifiée / Lisse_lisse          3.308e+00  1.879e+00   1.760 0.078343 .  &#xA;Crois_Profil_PrioNourrie / Lisse_lisse           1.318e+00  1.829e+00   0.721 0.471155    &#xA;Crois_Profil_PrioPurifiée / Lisse_lisse         -1.844e+01  3.417e+03  -0.005 0.995695    &#xA;Crois_Profil_PrioReposée / Lisse_lisse          -1.627e+01  6.889e+03  -0.002 0.998115    &#xA;Crois_Profil_PrioEclatante / Matifiée_matifiée  -1.651e+01  4.405e+03  -0.004 0.997010    &#xA;Crois_Profil_PrioHydratée / Matifiée_matifiée    3.058e+00  1.669e+00   1.832 0.066970 .  &#xA;Crois_Profil_PrioLisse / Matifiée_matifiée       2.699e+00  1.761e+00   1.533 0.125319    &#xA;Crois_Profil_PrioNourrie / Matifiée_matifiée     3.910e+00  2.316e+00   1.688 0.091412 .  &#xA;Crois_Profil_PrioPurifiée / Matifiée_matifiée   -1.614e+01  3.076e+03  -0.005 0.995814    &#xA;Crois_Profil_PrioReposée / Matifiée_matifiée    -1.751e+01  1.235e+04  -0.001 0.998868    &#xA;Crois_Profil_PrioEclatante / Nourrie_nourrie     2.304e+00  1.620e+00   1.422 0.154909    &#xA;Crois_Profil_PrioHydratée / Nourrie_nourrie     -1.622e+01  3.909e+03  -0.004 0.996688    &#xA;Crois_Profil_PrioLisse / Nourrie_nourrie         4.018e+00  2.082e+00   1.931 0.053542 .  &#xA;Crois_Profil_PrioMatifiée / Nourrie_nourrie     -1.683e+01  9.424e+03  -0.002 0.998575    &#xA;Crois_Profil_PrioPurifiée / Nourrie_nourrie      2.889e+00  1.891e+00   1.528 0.126629    &#xA;Crois_Profil_PrioEclatante / Purifiée_purifiée   1.983e-01  1.773e+00   0.112 0.910932    &#xA;Crois_Profil_PrioHydratée / Purifiée_purifiée   -1.700e+00  2.025e+00  -0.840 0.401098    &#xA;Crois_Profil_PrioLisse / Purifiée_purifiée      -1.741e+01  5.381e+03  -0.003 0.997419    &#xA;Crois_Profil_PrioMatifiée / Purifiée_purifiée   -1.840e+01  4.515e+03  -0.004 0.996749    &#xA;Crois_Profil_PrioReposée / Purifiée_purifiée    -1.771e+01  1.039e+04  -0.002 0.998640    &#xA;Crois_Profil_PrioEclatante / Reposée_reposée     2.290e+00  1.796e+00   1.275 0.202363    &#xA;Crois_Profil_PrioHydratée / Reposée_reposée     -1.609e+01  3.890e+03  -0.004 0.996699    &#xA;Crois_Profil_PrioLisse / Reposée_reposée         2.916e+00  1.552e+00   1.879 0.060212 .  &#xA;Crois_Profil_PrioMatifiée / Reposée_reposée     -1.625e+01  9.461e+03  -0.002 0.998629    &#xA;Crois_Profil_PrioNourrie / Reposée_reposée      -1.724e+01  8.232e+03  -0.002 0.998329    &#xA;Crois_Profil_PrioPurifiée / Reposée_reposée      6.497e-01  1.724e+00   0.377 0.706274    &#xA;Crois_ALL_AGEjeune_Sans_ALL                     -7.512e-01  7.326e-01  -1.025 0.305170    &#xA;Crois_ALL_AGEsenior_Avec_ALL                     1.664e+01  4.714e+03   0.004 0.997183    &#xA;Crois_ALL_AGEsenior_Sans_ALL                     3.611e-01  6.555e-01   0.551 0.581703    &#xA;Crois_ALL_AGEvieux_Avec_ALL                      1.930e+01  4.714e+03   0.004 0.996733    &#xA;Crois_ALL_AGEvieux_Sans_ALL                             NA         NA      NA       NA    &#xA;INTERET_VACANCESÀ la montagne                    1.208e+00  5.537e-01   2.182 0.029090 *  &#xA;INTERET_VACANCESEn ville                         2.452e+00  9.143e-01   2.682 0.007317 ** &#xA;INTERET_COMPOSITIONPas du tout                  -1.763e+01  4.579e+03  -0.004 0.996929    &#xA;INTERET_COMPOSITIONUn peu                        4.350e-01  8.082e-01   0.538 0.590461    &#xA;PEAU_CORPS.2Sèche                                       NA         NA      NA       NA    &#xA;PEAU_CORPS.2Très sèche                                  NA         NA      NA       NA    &#xA;Nbre_gift                                        2.144e-01  1.520e-01   1.411 0.158265    &#xA;w                                                1.796e+00  1.027e+00   1.749 0.080317 .  &#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;(Dispersion parameter for binomial family taken to be 1)&#xA;&#xA;    Null deviance: 812.37  on 586  degrees of freedom&#xA;Residual deviance: 170.68  on 501  degrees of freedom&#xA;AIC: 340.68&#xA;&#xA;Number of Fisher Scoring iterations: 19&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Besides, I tested all types of family:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;*Family Default Link Function&#xA;binomial (link = &quot;logit&quot;)&#xA;gaussian (link = &quot;identity&quot;)&#xA;Gamma (link = &quot;inverse&quot;)&#xA;inverse.gaussian (link = &quot;1 / mu ^ 2&quot;)&#xA;fish (link = &quot;log&quot;)&#xA;quasi (link = &quot;identity&quot;, variance = &quot;constant&quot;)&#xA;quasibinomial (link = &quot;logit&quot;)&#xA;quasipoisson (link = &quot;log&quot;)*&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why then I can not find any significant variable in my model ???&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&lt;/p&gt;&#xA;" OwnerUserId="41072" LastEditorUserId="41072" LastEditDate="2017-11-29T09:44:36.043" LastActivityDate="2017-11-29T10:09:31.580" Title="Why I didn't get any significant variable in my logistic model?" Tags="|machine-learning|r|data-mining|logistic-regression|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="25207" PostTypeId="1" CreationDate="2017-11-29T09:13:10.777" Score="1" ViewCount="1445" Body="&lt;p&gt;I'm experiencing a weird issue when trying to finetune a model with PyTorch. I've adapted a script similar to &lt;code&gt;imagenet.py&lt;/code&gt; training script from PyTorch repository. Each time I stop the training, and trying to resume from a checkpoint, I'm seeing a sharp drop in accuracy. After one &quot;save cycle&quot; (mini-epoch?) the accuracy seems to recover, and sometimes is even doing better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the dataset I'm running on is big, I've made changes to log and save on shorter cycles, and so my training loop is a bit different from the original imagenet.py script. This is probably the cause of this bug, but I can't figure out what this might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WBdkh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WBdkh.png&quot; alt=&quot;accuracy drops when resuming training&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;    import os &#xA;    import shutil &#xA;    import time&#xA;&#xA;    import torch from tensorboard_logger &#xA;    import log_value&#xA;&#xA;    def train(train_dataset, train_loader, model, criterion, optimizer, val_loader, best_prec1, best_train_prec1, samples, checkpoint_directory, args, scheduler):&#xA;        &quot;&quot;&quot;Train for one epoch on the training set&quot;&quot;&quot;&#xA;        batch_time = AverageMeter()&#xA;        losses = AverageMeter()&#xA;        top1 = AverageMeter()&#xA;&#xA;        # switch to train mode&#xA;        model.train()&#xA;&#xA;        end = time.time()&#xA;        for i, (input, target) in enumerate(train_loader):&#xA;            if scheduler is not None:&#xA;                scheduler.batch_step()&#xA;            target = target.cuda(async=True)&#xA;            input = input.cuda()&#xA;            input_var = torch.autograd.Variable(input)&#xA;            target_var = torch.autograd.Variable(target)&#xA;&#xA;            # compute output&#xA;            output = model(input_var)&#xA;            loss = criterion(output, target_var)&#xA;&#xA;            # measure accuracy and record loss&#xA;            prec1 = accuracy(output.data, target, topk=(1,))[0]&#xA;            losses.update(loss.data[0], input.size(0))&#xA;            top1.update(prec1[0], input.size(0))&#xA;            samples += input.size(0)&#xA;&#xA;            # compute gradient and do SGD step&#xA;            loss.backward()&#xA;            if i % args.accum == 0:&#xA;                optimizer.step()&#xA;                optimizer.zero_grad()&#xA;&#xA;            # measure elapsed time&#xA;            batch_time.update(time.time() - end)&#xA;            end = time.time()&#xA;&#xA;            if i % args.print_freq == 0:&#xA;                print('Epoch: {0:.4f}\t'&#xA;                      'Step: {1}/{2}\t'&#xA;                      'Samples: [{samples}]\t'&#xA;                      'LR: {lr}\t'&#xA;                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'&#xA;                      'Samples/s {samples_per_sec:.0f}\t'&#xA;                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'&#xA;                      'Prec@1 {top1.val:.3f} ({top1.avg:.3f})'.format(&#xA;                    samples / len(train_dataset), i, len(train_loader), samples=samples, batch_time=batch_time,&#xA;                    samples_per_sec=input.size(0)/batch_time.avg,&#xA;                    lr=get_learning_rate(optimizer)[0],# *iter_accum ???&#xA;                loss=losses, top1=top1))&#xA;&#xA;            if i % args.save_steps_freq == 0:&#xA;                if i&amp;gt;0:&#xA;                    # evaluate on validation set&#xA;                    prec1 = validate(val_loader, model, criterion, samples, args)&#xA;&#xA;                    # remember best prec@1 and save checkpoint&#xA;                    print('Checkpoint')&#xA;                    is_best = prec1 &amp;gt; best_prec1&#xA;                    best_prec1 = max(prec1, best_prec1)&#xA;                    is_best_train = top1.avg &amp;gt; best_train_prec1&#xA;                    best_train_prec1 = max(top1.avg, best_train_prec1)&#xA;                    save_checkpoint({&#xA;                        'samples': samples,&#xA;                        'state_dict': model.state_dict(),&#xA;                        'best_prec1': best_prec1,&#xA;                        'best_train_prec1': best_train_prec1,&#xA;                        'train_prec1': top1.avg,&#xA;                    }, is_best, is_best_train,&#xA;                        directory=checkpoint_directory&#xA;                    )&#xA;&#xA;                    # log to TensorBoard&#xA;                    log_value('train_loss', losses.avg, samples)&#xA;                    log_value('train_acc', top1.avg, samples)&#xA;                    log_value('learning_rate', get_learning_rate(optimizer)[0], samples)&#xA;                    log_value('batch_size', input.size(0), samples)&#xA;                    log_value('effective_batch_size', input.size(0)*args.accum, samples)&#xA;                    log_value('accum', args.accum, samples)&#xA;&#xA;                batch_time.reset()&#xA;                losses.reset()&#xA;                top1.reset()&#xA;        return best_prec1, best_train_prec1, samples&#xA;&#xA;&#xA;    def validate(val_loader, model, criterion, samples, args):&#xA;        &quot;&quot;&quot;Perform validation on the validation set&quot;&quot;&quot;&#xA;        batch_time = AverageMeter()&#xA;        losses = AverageMeter()&#xA;        top1 = AverageMeter()&#xA;&#xA;        # switch to evaluate mode&#xA;        model.eval()&#xA;&#xA;        end = time.time()&#xA;        for i, (input, target) in enumerate(val_loader):&#xA;            # print(&quot;input={}&quot;, input.size())&#xA;            target = target.cuda(async=True)&#xA;            input = input.cuda()&#xA;            input_var = torch.autograd.Variable(input, volatile=True)&#xA;            target_var = torch.autograd.Variable(target, volatile=True)&#xA;&#xA;            # compute output&#xA;            output = model(input_var)&#xA;            # print(&quot;validate vars input={} target={} output={}&quot;.format(input_var.size(), target_var.size(), output.size()))&#xA;            loss = criterion(output, target_var)&#xA;&#xA;            # measure accuracy and record loss&#xA;            prec1 = accuracy(output.data, target, topk=(1,))[0]&#xA;            losses.update(loss.data[0], input.size(0))&#xA;            top1.update(prec1[0], input.size(0))&#xA;&#xA;            # measure elapsed time&#xA;            batch_time.update(time.time() - end)&#xA;            end = time.time()&#xA;&#xA;            if i % args.print_freq == 0:&#xA;                print('Test: [{0}/{1}]\t'&#xA;                      'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'&#xA;                      'Loss {loss.val:.4f} ({loss.avg:.4f})\t'&#xA;                      'Prec@1 {top1.val:.3f} ({top1.avg:.3f})'.format(&#xA;                    i, len(val_loader), batch_time=batch_time, loss=losses,&#xA;                    top1=top1))&#xA;&#xA;        print(' * Prec@1 {top1.avg:.3f}'.format(top1=top1))&#xA;&#xA;        # log to TensorBoard&#xA;        log_value('val_loss', losses.avg, samples)&#xA;        log_value('val_acc', top1.avg, samples)&#xA;&#xA;        return top1.avg&#xA;&#xA;&#xA;    def get_learning_rate(optimizer):&#xA;        if optimizer is None:&#xA;            return [0.0]&#xA;        lr=[]&#xA;        for param_group in optimizer.param_groups:&#xA;            lr +=[ param_group['lr'] ]&#xA;        return lr&#xA;&#xA;&#xA;    def save_checkpoint(state, is_best, is_best_train, directory, filename='checkpoint.pth.tar'):&#xA;        &quot;&quot;&quot;Saves checkpoint to disk&quot;&quot;&quot;&#xA;        if not os.path.exists(directory):&#xA;            os.makedirs(directory)&#xA;        filename = directory + filename&#xA;        torch.save(state, filename)&#xA;        if is_best:&#xA;            shutil.copyfile(filename, directory + 'model_best.pth.tar')&#xA;        if is_best_train:&#xA;            shutil.copyfile(filename, directory + 'model_best_train.pth.tar')&#xA;&#xA;&#xA;    class AverageMeter(object):&#xA;        &quot;&quot;&quot;Computes and stores the average and current value&quot;&quot;&quot;&#xA;&#xA;        def __init__(self):&#xA;            self.reset()&#xA;&#xA;        def reset(self):&#xA;            self.val = 0&#xA;            self.avg = 0&#xA;            self.sum = 0&#xA;            self.count = 0&#xA;&#xA;        def update(self, val, n=1):&#xA;            self.val = val&#xA;            self.sum += val * n&#xA;            self.count += n&#xA;            self.avg = self.sum / self.count&#xA;&#xA;&#xA;    def adjust_learning_rate(optimizer, epoch, lr):&#xA;        &quot;&quot;&quot;Sets the learning rate to the initial LR decayed by 10 after 150 and 225 epochs&quot;&quot;&quot;&#xA;        lr = lr * (0.1 ** (epoch // 150)) * (0.1 ** (epoch // 225))&#xA;        # log to TensorBoard&#xA;        log_value('learning_rate', lr, epoch)&#xA;        for param_group in optimizer.param_groups:&#xA;            param_group['lr'] = lr&#xA;&#xA;&#xA;    def accuracy(output, target, topk=(1,)):&#xA;        &quot;&quot;&quot;Computes the precision@k for the specified values of k&quot;&quot;&quot;&#xA;        maxk = max(topk)&#xA;        batch_size = target.size(0)&#xA;&#xA;        _, pred = output.topk(maxk, 1, True, True)&#xA;        pred = pred.t()&#xA;        correct = pred.eq(target.view(1, -1).expand_as(pred))&#xA;&#xA;        res = []&#xA;        for k in topk:&#xA;            correct_k = correct[:k].view(-1).float().sum(0)&#xA;            res.append(correct_k.mul_(100.0 / batch_size))&#xA;        return res&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26871" LastEditorUserId="1330" LastEditDate="2019-05-15T14:53:41.983" LastActivityDate="2019-05-15T14:53:41.983" Title="Resuming from checkpoint, accuracy drops for one cycle" Tags="|pytorch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25208" PostTypeId="2" ParentId="25206" CreationDate="2017-11-29T09:59:47.517" Score="0" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far what you have done is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before going to modelling you need to take care of couple of things(Exploratory Analysis), Like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Removing all the unnecessary variables based on business knowledge&lt;/li&gt;&#xA;&lt;li&gt;Imputing Missing values&lt;/li&gt;&#xA;&lt;li&gt;Removing Outliers&lt;/li&gt;&#xA;&lt;li&gt;removing unimportant variables&lt;/li&gt;&#xA;&lt;li&gt;Correlation analysis between variables&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;once these are achieved then look into Modelling(you have done most of the above steps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now coming to your problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are trying to apply Logistic Regression on data to predict then you need to take care of couple of things:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Remove all the factors with single level&lt;/li&gt;&#xA;&lt;li&gt;Then you need to make sure that data which is partitioned(Spliting between Test and Train) have &lt;strong&gt;Same Levels&lt;/strong&gt; before you predict. For now you din't see the issue but once you predict then you will get this error.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Important thing to remember is, Logistic regression works by converting Categorical variable to dummies before applying model, because of this it is unable to return you anything.Since you have more Categorical variables in your data I would suggest you not use Logistic Regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get Predictor Importance(Important Independent Variables), you can use a Package in R called &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt;, this link has the implementation in R.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get important variables then you can apply Random Forest, Decision Trees(Rpart) and see how they perform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you have any issues.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-11-29T10:09:31.580" LastActivityDate="2017-11-29T10:09:31.580" CommentCount="24" ContentLicense="CC BY-SA 3.0" />
  <row Id="25209" PostTypeId="1" AcceptedAnswerId="25212" CreationDate="2017-11-29T10:47:33.030" Score="8" ViewCount="7084" Body="&lt;p&gt;In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in policy iteration we also have to output a softmax vector related to each action. So I don't understand how this can be used to work with continuous action space.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are policy gradient methods preferred over value function approximation in continuous action domains?&lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorDisplayName="user10640" LastEditDate="2018-12-10T00:31:02.690" LastActivityDate="2018-12-10T00:31:02.690" Title="Why are policy gradient methods preferred over value function approximation in continuous action domains?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25211" PostTypeId="1" CreationDate="2017-11-29T11:08:37.617" Score="4" ViewCount="4059" Body="&lt;p&gt;I have the information about the behaviour of 400 users across period of 1 months (30 days).&#xA;Across those 30 days I measure 4 different information (let's call it A,B,C and D), hence I have a total of 4 time series.&#xA;My goal is to predict, starting from day 5, the future values of A, by knowing the past values of A, B, C and D.&#xA;So:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A(5) = f(A(1,2,3,4), B(1,2,3,4), C(1,2,3,4), D(1,2,3,4))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Ideally, I'd like to estimate the whole future values, from day 5 to day 30.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use LSTM by using as input the past series, to predict the future trends.&#xA;However, I am not getting great results, and I am hoping I can use a simpler approach.&#xA;I came across with the concept of Kalman Filter, where the value at time step k can be estimated by using the past values.&#xA;&lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/using-bayesian-kalman-filter-to-predict-positions-of-moving&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; there are some nice examples , however I have no idea where to start from.&#xA;Would it fit with my problem, if yes, how could I fit my past values into the equation for the estimation of A(k)?&#xA;Any suggestion is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="42327" LastEditorUserId="83275" LastEditDate="2022-06-19T06:13:19.893" LastActivityDate="2022-06-19T06:13:19.893" Title="Kalman filter for time series prediction" Tags="|time-series|lstm|prediction|forecast|kalman-filter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25212" PostTypeId="2" ParentId="25209" CreationDate="2017-11-29T11:21:30.613" Score="10" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But in policy iteration also we are have to output a softmax vector related to each actions &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So I don't understand how this can use to work with continuous action space ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;With policy gradient methods, the policy can be &lt;em&gt;any&lt;/em&gt; function of your parameters $\theta$ which:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Outputs a probability distribution&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can be differentiated with respect to $\theta$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So for instance your policy function can be &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\pi_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Why are policy gradient methods preferred over value function approximation in continuous action domains?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whilst it is still possible to estimate the &lt;em&gt;value&lt;/em&gt; of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-29T11:48:43.473" LastActivityDate="2017-11-29T11:48:43.473" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25216" PostTypeId="1" CreationDate="2017-11-29T13:36:30.530" Score="1" ViewCount="462" Body="&lt;p&gt;I have a small dataset which has timestamp and temperature values for 6 months(I.e. one temperature value per day). I would like to forecast 2-3 months of temperature.&#xA;I would like to know, what kind of models would be suitable for such purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used LSTM but the forecasts degrades as I go further  and the data size is pretty small for LSTM to be trained.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastActivityDate="2017-11-30T07:55:37.013" Title="Long term time series forecasts with small dataset" Tags="|time-series|forecasting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25217" PostTypeId="1" CreationDate="2017-11-29T15:10:35.877" Score="2" ViewCount="134" Body="&lt;p&gt;Working on a binary classification task to identify duplicate documents with 350,000 labeled samples, 70 features. There is about a 10:1 class imbalance. My best performing model so far has been a random forest, but I would like to see if I can use a deep feed forward neural network for better results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been playing with the number of layers and hidden units and verify performance using 5-fold cross validation. Here is an example of one architecture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for train_validation, test_validation in sfk.split(X_train,y_train):   &#xA;    model = Sequential()&#xA;    model.add(Dense(150, input_dim=X_train.shape[1], activation='relu')) &#xA;    model.add(Dense(125, activation='relu'))&#xA;    model.add(Dense(100, activation='relu'))&#xA;    model.add(Dense(75, activation='relu'))&#xA;    model.add(Dense(50, activation='relu'))&#xA;    model.add(Dense(25, activation='relu'))&#xA;    model.add(Dense(1, activation='sigmoid'))&#xA;    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])&#xA;&#xA;    model.fit(X_train.iloc[train_validation,:].values,&#xA;              y_train.iloc[train_validation].values.reshape([-1,1]),&#xA;              epochs=13,&#xA;              batch_size=64,&#xA;              verbose=2,&#xA;              validation_data=(X_train.iloc[test_validation ,:].values,y_train.iloc[val_test].values.reshape([-1,1])))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I run that, get the mean/std of the accuracy and error. Then try adding a layer, removing a layer, adding some units, removing some. There is movement in accuracy and error (in the +/- 0.00x range). This approach is unscientific, what is a better way?&lt;/p&gt;&#xA;" OwnerUserId="37776" LastActivityDate="2017-11-29T15:30:01.337" Title="Methodical approach to improve deep neural network performance?" Tags="|neural-network|classification|keras|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25219" PostTypeId="1" AcceptedAnswerId="25225" CreationDate="2017-11-29T16:43:12.107" Score="0" ViewCount="583" Body="&lt;p&gt;I am designing a classifier for an Imbalanced Data set. I have a queries regarding choosing the threshold for a classifier, currently I am using mean of the predicted probabilities as the threshold and I am maximizing the recall on positive classes. Is this a correct way of choosing a threshold (I.e. mean of predicted probabilities) or I should try doing something &lt;a href=&quot;http://appliedpredictivemodeling.com/blog/2014/2/1/lw6har9oewknvus176q4o41alqw2ow&quot; rel=&quot;nofollow noreferrer&quot;&gt;like this book&lt;/a&gt; by using recall as a metric on KFold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion would be highly appreciable. Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="26329" LastActivityDate="2017-11-29T19:29:12.690" Title="Classifier Threshold" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25220" PostTypeId="1" CreationDate="2017-11-29T17:38:05.263" Score="0" ViewCount="3656" Body="&lt;p&gt;Input file is :&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;time    event   earthAcceleration.x earthAcceleration.y earthAcceleration.z             &#xA;2017-11-22T09:20:13.944 motion  -0.006379993    -0.001029071 -0.010781396   &#xA;2017-11-22T09:20:13.954 motion  0.014739769 0.008146121 -0.001068155                &#xA;2017-11-22T09:20:13.964 motion  0.008520941 -0.008222823    0.022573506             &#xA;2017-11-22T09:20:13.974 motion  0.000573836 0.016701656 -0.016170829                &#xA;2017-11-22T09:20:13.984 motion  0.016282924 0.003181049 0.006969224&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I need to remove the date and hours and just need the milliseconds from the start. Example: &lt;code&gt;Time2 = time2-time1&lt;/code&gt; i.e. I need relative time in milliseconds such that I can plot a graph by time.&lt;/p&gt;&#xA;&lt;p&gt;Output:&#xA;(the time column should change into)&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;time&#xA;0.00&#xA;0.15&#xA;1.12&#xA;1.34&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42578" LastEditorUserId="83275" LastEditDate="2021-03-15T06:01:36.387" LastActivityDate="2021-03-15T06:01:36.387" Title="I have a csv file with time as the datetime format and want to edit the same column into miliseconds, how do I change that?" Tags="|python|csv|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25221" PostTypeId="1" CreationDate="2017-11-29T18:35:40.430" Score="2" ViewCount="237" Body="&lt;p&gt;Can you please explain when (under what circumstances) the dimensional reduction and/or feature selection techniques are not effective to mitigate the curse of dimensionality?&lt;/p&gt;&#xA;" OwnerUserId="41626" LastActivityDate="2018-04-01T03:43:30.213" Title="Why is the dimensionality reduction and/or feature selection techniques are considered to be not always effective?" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25222" PostTypeId="1" CreationDate="2017-11-29T18:40:13.493" Score="2" ViewCount="46" Body="&lt;p&gt;I have a time series with the percentage of the subscriber based on the total users in my platform. Let's say I did an event that could, potentially, affected the figures. I could run a &lt;a href=&quot;https://google.github.io/CausalImpact/CausalImpact.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CausalImpact&lt;/a&gt; from Google Github and check the exact effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my question is. Should I apply the algorithm to the raw data of the time series? Or it would be better if I do some kind of decomposition before?&lt;/p&gt;&#xA;" OwnerUserId="201" LastActivityDate="2017-11-29T18:40:13.493" Title="How to properly implement a CausalImpact on a time series" Tags="|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25223" PostTypeId="2" ParentId="25220" CreationDate="2017-11-29T18:59:28.627" Score="1" Body="&lt;p&gt;Familiarize yourself with the pandas library, which allows you to do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import read_csv&#xA;from io import StringIO&#xA;csv_file = StringIO(&quot;&quot;&quot;time event earthAcceleration.x earthAcceleration.y earthAcceleration.z&#xA;2017-11-22T09:20:13.944 motion -0.006379993 -0.001029071 -0.010781396&#xA;2017-11-22T09:20:13.954 motion 0.014739769 0.008146121 -0.001068155&#xA;2017-11-22T09:20:13.964 motion 0.008520941 -0.008222823 0.022573506&#xA;2017-11-22T09:20:13.974 motion 0.000573836 0.016701656 -0.016170829&#xA;2017-11-22T09:20:13.984 motion 0.016282924 0.003181049 0.006969224&quot;&quot;&quot;)&#xA;df = read_csv(csv_file, sep=' ', parse_dates=['time'])&#xA;df.assign(delta_t = df.time-df.time.ix[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I believe your output is wrong; the difference between each line is exactly 0.01s. Welcome to the site!&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2017-11-29T18:59:28.627" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="25224" PostTypeId="1" CreationDate="2017-11-29T19:07:23.703" Score="0" ViewCount="661" Body="&lt;p&gt;I understand how doc2vec works, but I am unclear the best practice on feeding in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a document with multiple sentences&lt;/p&gt;&#xA;&#xA;&lt;p&gt;['I really love football. Peyton Manning was a great player']&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we feed this into the algorithm as is, the window for 'Peyton' could include ['love','football','Manning','was']&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This doesn't make intuitive sense to me, however, since the words come from different sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="42586" LastActivityDate="2020-04-22T19:40:09.327" Title="Doc2Vec Input from Paragraphs" Tags="|nlp|gensim|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25225" PostTypeId="2" ParentId="25219" CreationDate="2017-11-29T19:29:12.690" Score="3" Body="&lt;p&gt;The threshold you choose depends on the specifics of the problem you are trying to solve.  More specifically, it should be based on how you weigh false positives vs. false negatives, i.e. how bad each of these are relative to each other. You mention that you are trying to maximize recall on the positive class, but if that were true you would should just classify everything as a positive class, and get a recall of 1.0.  Based on the domain you are working in, you should decide how much a false positive 'costs' vs. how much a false negative 'costs'.  Once you decide this, you can find the threshold that minimizes the function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;total cost = false negative count x FN cost + false positive count x FP cost&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-11-29T19:29:12.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25226" PostTypeId="2" ParentId="6676" CreationDate="2017-11-29T21:09:03.410" Score="1" Body="&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;: You could specify a grid of &lt;em&gt;alpha&lt;/em&gt; and &lt;em&gt;n_iter&lt;/em&gt;(or &lt;em&gt;max_iter&lt;/em&gt;) and use &lt;a href=&quot;https://github.com/jmcarpenter2/parfit&quot; rel=&quot;nofollow noreferrer&quot;&gt;parfit&lt;/a&gt; for hyper-optimization on SGDClassifier&lt;/p&gt;&#xA;&lt;p&gt;My colleague, Vinay Patlolla, wrote an excellent blog post on &lt;a href=&quot;https://medium.com/@vinnsvinay/how-to-make-sgd-classifier-perform-as-well-as-logistic-regression-using-parfit-cc10bca2d3c4&quot; rel=&quot;nofollow noreferrer&quot;&gt;How to make SGD Classifier perform as well as Logistic Regression using parfit&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/jmcarpenter2/parfit&quot; rel=&quot;nofollow noreferrer&quot;&gt;Parfit&lt;/a&gt; is a hyper-parameter optimization package that he utilized to find the appropriate combination of parameters which served to optimize SGDClassifier to perform as well as Logistic Regression on his example data set in much less time.&lt;/p&gt;&#xA;&lt;p&gt;In summary, the two key parameters for SGDClassifier are &lt;em&gt;alpha&lt;/em&gt; and &lt;em&gt;n_iter&lt;/em&gt;. To quote Vinay directly:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;n_iter in sklearn is None by default. We are setting it here to a sufficiently large amount(1000). An alternative parameter to n_iter, which has been recently added, is max_iter. The same advice should apply for max_iter.&lt;/p&gt;&#xA;&lt;p&gt;The alpha hyper-parameter serves a dual purpose. It is both a regularisation parameter and the initial learning rate under the default schedule. This means that, in addition to regularising the Logistic Regression coefficients, the output of the model is dependent on an interaction between alpha and the number of epochs (n_iter) that the fitting routine performs. Specifically, as alpha becomes very small, n_iter must be increased to compensate for the slow learning rate. This is why it is safer (but slower) to specify n_iter sufficiently large, e.g. 1000, when searching over a wide range of alphas.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="42593" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-29T21:09:03.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25227" PostTypeId="1" CreationDate="2017-11-29T21:11:39.200" Score="7" ViewCount="1712" Body="&lt;p&gt;I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from Inception and used them to train a fully connected layer. This FC gave me around 86% training accuracy. I then 'fine tuned' the model by sticking the trained FC layers on top of a topless Inception V3 and retrained along with the top two convolutional blocks with SGD, a low learning rate and high momentum. I also used Image Augmentation on the training images, but only during fine tuning stage. Instead of accuracy improving, this model hardly gives me 60% accuracy and performs worse than the bare transfer learning model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.Why is this happening? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Optional: 2.What can I do to improve my accuracy in 'general' on this model? The images I am trying to classify are from the BreakHis breast tumor database and is quite a difficult classification task. Is Transfer learning even appropriate for this task? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code for Transfer learning with bottleneck features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras.models import Sequential&#xA;from keras.layers import Dropout, Dense, GlobalAveragePooling2D&#xA;from keras.applications.inception_v3 import InceptionV3,preprocess_input&#xA;from keras.utils.np_utils import to_categorical&#xA;from keras.callbacks import ModelCheckpoint, EarlyStopping&#xA;import math&#xA;&#xA;img_width, img_height = (299, 299)&#xA;weight_path = 'bottleneck_fc_model.h5'&#xA;train_dir = 'Cancer_Data/Train'&#xA;validation_dir = 'Cancer_Data/Validate'&#xA;epochs = 150&#xA;batch_size = 128&#xA;&#xA;&#xA;&#xA;def train_top_model():&#xA;    datagen_top = ImageDataGenerator(preprocessing_function=preprocess_input)&#xA;    generator_top = datagen_top.flow_from_directory(&#xA;    train_dir,&#xA;    target_size=(img_width, img_height),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical',&#xA;    shuffle=False)&#xA;&#xA;nb_classes = len(generator_top.class_indices)&#xA;np.save('class_indices.npy', generator_top.class_indices)&#xA;&#xA;train_data = np.load('bottleneck_feature_train.npy')&#xA;train_labels = to_categorical(generator_top.classes, num_classes=nb_classes)&#xA;&#xA;generator_top2 = datagen_top.flow_from_directory(&#xA;    validation_dir,&#xA;    target_size=(img_width, img_height),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical',&#xA;    shuffle=False)&#xA;&#xA;validation_data = np.load('bottleneck_feature_validation.npy')&#xA;validation_labels = to_categorical(generator_top2.classes, num_classes=nb_classes)&#xA;&#xA;model = Sequential()&#xA;model.add(GlobalAveragePooling2D(input_shape=train_data.shape[1:]))&#xA;model.add(Dense(1024, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(nb_classes, activation='softmax'))&#xA;&#xA;model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&#xA;callbacks_list=[ModelCheckpoint(weight_path, monitor='val_acc', verbose=1, save_best_only=True),EarlyStopping(monitor='val_acc',patience=10,verbose=0)]&#xA;&#xA;&#xA;&#xA;model.fit(train_data, train_labels, epochs=epochs, batch_size=batch_size,&#xA;          validation_data=(validation_data, validation_labels), callbacks=callbacks_list)&#xA;&#xA;(eval_loss, eval_accuracy) = model.evaluate(&#xA;    validation_data, validation_labels, batch_size=batch_size, verbose=1)&#xA;&#xA;print(&quot;[INFO] accuracy: {:.2f}%&quot;.format(eval_accuracy * 100))&#xA;print(&quot;[INFO] Loss: {}&quot;.format(eval_loss))&#xA;&#xA;&#xA;train_top_model()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code for Fine Tuning:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.applications.inception_v3 import InceptionV3, preprocess_input&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras import optimizers&#xA;from keras.regularizers import l1_l2&#xA;from keras.models import Sequential, Model, model_from_json&#xA;from keras.layers import Dropout, GlobalAveragePooling2D, Dense&#xA;from keras.callbacks import ModelCheckpoint, EarlyStopping&#xA;import PIL&#xA;import math&#xA;&#xA;&#xA;weight_path = 'fine_tuned_weights.h5'&#xA;top_model_weight_path = 'top_model.h5'&#xA;&#xA;img_width, img_height = (229, 229)&#xA;&#xA;train_dir = 'Cancer_Data/Train'&#xA;validation_dir = 'Cancer_Data/Validate'&#xA;&#xA;epochs = 150&#xA;&#xA;batch_size = 128&#xA;&#xA;nb_train_samples = 6454&#xA;nb_validation_samples = 1464&#xA;&#xA;base_model =InceptionV3(weights= 'imagenet', include_top= False, input_shape=(229,229,3))&#xA;print &quot;Model Loaded.&quot;&#xA;&#xA;top_model= Sequential()&#xA;top_model.add(GlobalAveragePooling2D(input_shape=base_model.output_shape[1:]))&#xA;top_model.add(Dense(1024, activation='relu'))&#xA;top_model.add(Dense(8, activation= 'softmax'))&#xA;&#xA;top_model.load_weights(top_model_weight_path)&#xA;&#xA;model= Model(inputs= base_model.input, outputs= top_model(base_model.output))&#xA;&#xA;for layer in model.layers[:172]:&#xA;    layer.trainable=False&#xA;for layer in model.layers[172:]:&#xA;    layer.trainable=True&#xA;&#xA;model.compile(optimizer=optimizers.SGD(lr=0.0001, momentum=0.9),&#xA;          loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&#xA;train_datagen= ImageDataGenerator(&#xA;    preprocessing_function=preprocess_input,&#xA;    rotation_range=20,&#xA;    shear_range=0.3,&#xA;    zoom_range=0.3,&#xA;    horizontal_flip=True,&#xA;    vertical_flip=True)&#xA;&#xA;test_datagen= ImageDataGenerator(preprocessing_function=preprocess_input)&#xA;&#xA;train_generator= train_datagen.flow_from_directory(&#xA;    train_dir,&#xA;    target_size=(img_height,img_width),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical')&#xA;&#xA;validation_generator= test_datagen.flow_from_directory(&#xA;    validation_dir,&#xA;    target_size=(img_height,img_width),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical')&#xA;&#xA;callbacks_list=[ModelCheckpoint(weight_path, monitor='val_acc',verbose=1,save_best_only=True),&#xA;            EarlyStopping(monitor='val_acc',patience=10,verbose=0)]&#xA;&#xA;model_json = model.to_json()&#xA;with open(&quot;fine_tuned_model.json&quot;, &quot;w&quot;) as json_file:&#xA;    json_file.write(model_json)&#xA;&#xA;model.fit_generator(&#xA;    train_generator,&#xA;    steps_per_epoch = int(math.ceil(nb_train_samples / batch_size)),&#xA;    epochs=epochs,&#xA;    validation_data=validation_generator,&#xA;    validation_steps = int(math.ceil(nb_validation_samples / batch_size)), callbacks=callbacks_list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42473" LastEditorUserId="1330" LastEditDate="2021-11-29T14:27:32.540" LastActivityDate="2021-11-29T14:27:32.540" Title="Fine tuning accuracy lower than Raw Transfer Learning Accuracy" Tags="|python|deep-learning|neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25228" PostTypeId="2" ParentId="25119" CreationDate="2017-11-29T21:42:49.513" Score="27" Body="&lt;p&gt;To answer your questions:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Yes your approach is right&lt;/li&gt;&#xA;&lt;li&gt;Of A, B and C the right answer is B.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The explanation is the following: In order to calculate Mean Average Precision (mAP) in the context of Object Detection you must compute the Average Precision (AP) for each class, and then compute the mean across all classes. The key here is to compute the AP for each class, in general for computing Precision (P) and Recall (R) you must define what are: True Positives (TP), False Positives (FP), True Negative (TN) and False Negative (FN). In the setting of Object Detection of the Pascal VOC Challenge are the following:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;TP: are the Bounding Boxes (BB) that the intersection over union (IoU) with the ground truth (GT) is above 0.5 and are classified correctly&lt;/li&gt;&#xA;&lt;li&gt;FP: three cases (a) BB that the IoU with GT is below 0.5 (b) the BB that have IoU with a GT that has already been detected (c) BB with wrong classification label.&lt;/li&gt;&#xA;&lt;li&gt;TN: there are not true negative, the image are expected to contain at least one object&lt;/li&gt;&#xA;&lt;li&gt;FN: those ground truthes for which the method failed to produce a BB&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Now each predicted BB have a confidence value for the given class. So the scoring method sort the predictions for decreasing order of confidence and compute the P = TP / (TP + FP) and R = TP / (TP + FN) for each possible rank k = 1 up to the number of predictions. So now you have a (P, R) for each rank those P and R are the &amp;quot;raw&amp;quot; Precision-Recall curve. To compute the interpolated P-R curve foreach value of R you select the maximum P that has a corresponding R' &amp;gt;= R.&lt;/p&gt;&#xA;&lt;p&gt;There are two different ways to sample P-R curve points according to &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#SECTION00044000000000000000&quot; rel=&quot;nofollow noreferrer&quot;&gt;voc devkit doc&lt;/a&gt;.&#xA;For VOC Challenge before 2010, we select the maximum P obtained for any R' &amp;gt;= R, which R belongs to 0, 0.1, ..., 1 (eleven points). The AP is then the average precision at each of the Recall thresholds. For VOC Challenge 2010 and after, we still select the maximum P for any R' &amp;gt;= R, while R belongs to all unique recall values (include 0 and 1). The AP is then the area size under P-R curve. Notice that in the case that you don't have a value of P with Recall above some of the thresholds the Precision value is 0.&lt;/p&gt;&#xA;&lt;p&gt;For instance consider the following output of a method given the class &amp;quot;Aeroplane&amp;quot;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;BB  | confidence | GT&#xA;----------------------&#xA;BB1 |  0.9       | 1&#xA;----------------------&#xA;BB2 |  0.9       | 1&#xA;----------------------&#xA;BB3 |  0.7       | 0&#xA;----------------------&#xA;BB4 |  0.7       | 0&#xA;----------------------&#xA;BB5 |  0.7       | 1&#xA;----------------------&#xA;BB6 |  0.7       | 0&#xA;----------------------&#xA;BB7 |  0.7       | 0&#xA;----------------------&#xA;BB8 |  0.7       | 1&#xA;----------------------&#xA;BB9 |  0.7       | 1&#xA;----------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Besides it not detected bounding boxes in two images, so we have FN = 2. The previous table is the ordered rank by confidence value of the predictions of the method GT = 1 means is a TP and GT = 0 FP. So TP=5 (BB1, BB2, BB5, BB8 and BB9), FP=5. For the case of rank=3 the precision drops because BB1 was already detected, so even if the object is indeed present it counts as a FP. .&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;rank=1  precision=1.00 and recall=0.14&#xA;----------&#xA;rank=2  precision=1.00 and recall=0.29&#xA;----------&#xA;rank=3  precision=0.66 and recall=0.29&#xA;----------&#xA;rank=4  precision=0.50 and recall=0.29&#xA;----------&#xA;rank=5  precision=0.40 and recall=0.29&#xA;----------&#xA;rank=6  precision=0.50 and recall=0.43&#xA;----------&#xA;rank=7  precision=0.43 and recall=0.43&#xA;----------&#xA;rank=8  precision=0.38 and recall=0.43&#xA;----------&#xA;rank=9  precision=0.44 and recall=0.57&#xA;----------&#xA;rank=10 precision=0.50 and recall=0.71&#xA;----------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Given the previous results:&#xA;If we used the way before voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0, 0, 0. Then AP = 5.5 / 11 = 0.5 for the class of &amp;quot;Aeroplanes&amp;quot;.&#xA;Else if we used the way since voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0 for seven unique recalls that are 0, 0.14, 0.29, 0.43, 0.57, 0.71, 1.Then AP = (0.14-0)*1 + (0.29-0.14)*1 + (0.43-0.29)*0.5 + (0.57-0.43)*0.5 + (0.71-0.57)*0.5 + (1-0.71)*0 = 0.5 for the class of &amp;quot;Aeroplanes&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;Repeat for each class and then you have the (mAP).&lt;/p&gt;&#xA;&lt;p&gt;More information can be found in the following links &lt;a href=&quot;https://stats.stackexchange.com/questions/260430/average-precision-in-object-detection&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/36274638/map-metric-in-object-detection-and-computer-vision&quot;&gt;2&lt;/a&gt;. Also you should check the paper: &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/pubs/everingham15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The PASCAL Visual Object Classes Challenge: A Retrospective&lt;/a&gt; for a more detailed explanation.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="65916" LastEditDate="2024-02-18T08:39:02.907" LastActivityDate="2024-02-18T08:39:02.907" CommentCount="12" ContentLicense="CC BY-SA 4.0" />
  <row Id="25229" PostTypeId="2" ParentId="16890" CreationDate="2017-11-29T22:05:04.377" Score="4" Body="&lt;p&gt;You can implement this very simply in Python.&lt;br&gt;&#xA;Your X will be the collection of training x,y,z coordinates.&lt;br&gt;&#xA;Your Y will be the collection of testing x,y,z coordinates.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cross_validation                     &#xA;from sklearn.neural_network import MLPRegressor   &#xA;&#xA;model = MLPRegressor(solver='lbfgs',alpha=0.001,hidden_layer_sizes=(150,))&#xA;cross_validation.cross_val_score(model, X, Y,scoring='mean_squared_error')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42595" LastEditorUserId="60292" LastEditDate="2019-03-16T15:58:23.967" LastActivityDate="2019-03-16T15:58:23.967" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25231" PostTypeId="1" CreationDate="2017-11-30T00:04:45.180" Score="5" ViewCount="12107" Body="&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/HyAs5.png&quot; alt=&quot;Hierarchical Models Of Perception and Reasoning by Yann LeCun&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above image is from &quot;&lt;a href=&quot;https://cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf&quot; rel=&quot;noreferrer&quot;&gt;Deep Learning Tutorial&lt;/a&gt;&quot; by Yann LeCun and Marc'Aurelio Ranzato (see pages 73 and 81).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I don't understand why 64 kernels (from the input to layer 1) produce 64 feature maps while 4096 kernels (layer 2 to layer 3) give 256 feature maps.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="42599" LastActivityDate="2019-02-05T02:02:16.477" Title="the relationship between the number of filters/kernels and the number of feature maps" Tags="|neural-network|deep-learning|cnn|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25232" PostTypeId="1" CreationDate="2017-11-30T01:14:26.087" Score="0" ViewCount="79" Body="&lt;p&gt;My problem: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have a set of items.&lt;/li&gt;&#xA;&lt;li&gt;Two judges will subjectively rate each item giving a grade from 0 to 10.&lt;/li&gt;&#xA;&lt;li&gt;I have to select the &lt;em&gt;best&lt;/em&gt; 10 items. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I rank the items so that the judges are equally important?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose one of them is really rigorous, and grade the items with a lower value, with a median of 5 and a standard deviation of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other median is 8, but his grades are more spread, with a standard deviation of 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are examples of the grades distributions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MNL77.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MNL77.png&quot; alt=&quot;Judge One&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vfXmo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vfXmo.png&quot; alt=&quot;Judge Two&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second judge has more spread grades, so if I just calculate the mean and rank, a lower grade of Judge Two would eliminate one item. Judge One grades would just order the best items of Judge Two. A favorite of Judge One would hardly get into the favorite rank. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to rank the items of each judge would make it more fair, but wouldn't take the grade difference between the items. &lt;/p&gt;&#xA;" OwnerUserId="31819" LastEditorUserId="31819" LastEditDate="2017-12-01T22:25:26.570" LastActivityDate="2017-12-01T22:25:26.570" Title="How to fairly rank the best items having two set of grades?" Tags="|statistics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25233" PostTypeId="2" ParentId="25232" CreationDate="2017-11-30T01:58:44.427" Score="1" Body="&lt;p&gt;&lt;strong&gt;Procedure-1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convert them into percentage and then take average of outcome of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so for example for person A, judge-1 rated 5 out of 7 and judge-2 rated 7 out of 10,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;converting judge-1 score to percentage i.e., $5/7 *100$ = 71%(rounded 71.45)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;converting judge-2 score to percentage i.e., $7/10 *100$ = 70% &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now take average of both i.e., $71+70 / 2$ =  70.5%, now you can convert it back on to a scale from $1-10$ :  This person A's overall rating is 7(rounded 7.05 ) out of 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Procedure-2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can normalize the rating of both the judges by using this formula, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x=(x_1,...,x_n)$ and $z_i$ is now your $i^{th}$ normalized data.  As a proof of concept (although you did not ask for it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Procedure-3:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can convert the 0-7 rating to 0-10 and then take average of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you get the outcome then you can rank them accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any doubts.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-30T01:58:44.427" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25234" PostTypeId="2" ParentId="25224" CreationDate="2017-11-30T03:54:25.080" Score="1" Body="&lt;p&gt;The aim of Doc2Vec is to produce document level embeddings, thus even if words are sentence-separated if you include them in the same document it has to be considered part of the same semantic source for word similarities. If you don't want such a behavior, you might want to separate your documents in a sentence-level and maybe aggregate the embeddings later in different groups to form your initial paragraphs (depending on your plans)?&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-11-30T03:54:25.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25235" PostTypeId="1" CreationDate="2017-11-30T06:05:09.607" Score="8" ViewCount="5430" Body="&lt;p&gt;I'm participating in the kaggle &lt;a href=&quot;https://www.kaggle.com/c/statoil-iceberg-classifier-challenge&quot; rel=&quot;noreferrer&quot;&gt;Iceberg Classifier Challenge&lt;/a&gt;, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement stochastic gradient descent to get a better idea for how mini batch training works, since the data set won't fit in memory after I perform data augmentation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SKLearn has the following two libraries for implementing logistic regression, &lt;code&gt;sklearn.LogisticRegression&lt;/code&gt; and &lt;code&gt;sklearn.SGDClassifier&lt;/code&gt;. After experimenting for a while, I've noticed that the SGD classifier never performs better than the logistic regression classifier, and that there are some other behaviors in the results of the SGD classifier that make me question whether I've implemented it properly or whether it's innately buggy (I'm sure it's the former).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following plot summarizes the performance of the &lt;code&gt;SGDClassifier&lt;/code&gt; model versus &lt;code&gt;LogisticRegression&lt;/code&gt; on the iceberg data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bjCnz.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bjCnz.png&quot; alt=&quot;sgd-vs-logreg&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The color curves are &lt;code&gt;SGDClassifier&lt;/code&gt; training runs with different numbers of mini-batches ranging from 50 to 1604 (the total number of training samples in the data set). Each was trained for 50 passes using &lt;code&gt;SGDClassifier.partial_fit()&lt;/code&gt;. The straight horizontal lines represent the base accuracy when only 1s are predicted, the logistic regression score, and the SGD score when all 1604 training samples are fit over a single pass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having a hard time understanding the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Why does &lt;code&gt;LogisticRegression&lt;/code&gt; so significantly outperform &lt;code&gt;SGDClassifier&lt;/code&gt; even though both should be training in the same way on the same data?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why does the performance so greatly decreases when passing over the data 50 times instead of just once? The latter makes sense to me if it's&#xA;simply a matter of overfitting to the data set when making multiple&#xA;runs over the data, though it would be good to understand how to&#xA;prevent this with mini-batch training.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why do the results so wildly vary with the mini-batch size in a&#xA;non-monotonic way, even after 50 passes through the data? Since the&#xA;data is shuffled in between runs I would expect any noisiness in the&#xA;fits to be smoothed over after such a large number of iterations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here's the code used to generate the results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Load training date&#xA;df_train = pd.read_csv('train_data.csv', sep = ',', header = 0, index_col = 0)&#xA;&#xA;&#xA;# Set input names; 'band_i_j' is pixel j from input image i&#xA;img_size = 75*75&#xA;inputs = ['inc_angle'] + ['band_1_' + str(i) for i in range(img_size)] + ['band_2_' + str(i) for i in range(img_size)]&#xA;&#xA;# Set output name&#xA;output = 'is_iceberg'&#xA;&#xA;# Set na values to mean of column&#xA;inc_angle_mean = df_train['inc_angle'].mean()&#xA;df_train['inc_angle'].fillna(inc_angle_mean, inplace = True)&#xA;&#xA;# Get num training samples&#xA;N_train = len(df_train)&#xA;&#xA;# Calculate mean, max, min of each column; standardize inputs&#xA;train_means = df_train[inputs].mean()&#xA;train_maxes = df_train[inputs].max()&#xA;train_mins = df_train[inputs].min()&#xA;df_train[inputs] = (df_train[inputs] - train_means)/(train_maxes - train_mins)&#xA;&#xA;# Set 80% of data to belong to training set; reserve the rest for validation set&#xA;train_indices = list(range(int(0.8*N_train)))&#xA;valid_indices = list(range(int(0.8*N_train), N_train))&#xA;&#xA;# Base score&#xA;print('base score', len(df_train[df_train[output]==1])/N_train)&#xA;&#xA;# SGD full-batch&#xA;model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, max_iter = 1000, shuffle = False, random_state = 0)&#xA;model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])&#xA;print('SGD full-batch', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))&#xA;&#xA;# Log reg&#xA;model = sklearn.linear_model.LogisticRegression(C = 1, tol = 0.00001, random_state = 0)&#xA;model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])&#xA;print('Log reg', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))&#xA;&#xA;&#xA;# SGD mini-batch&#xA;num_passes = 50&#xA;batch_size = N_train&#xA;model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, shuffle = False, random_state = 0)&#xA;&#xA;sgd_minibatch_scores = []&#xA;for i in range(num_passes):&#xA;    np.random.shuffle(train_indices)&#xA;    for j in range(int(len(train_indices)/batch_size + 1)):&#xA;&#xA;        if j == int(len(train_indices)/batch_size+1):&#xA;            batch_train_indices = train_indices[j*batch_size:]&#xA;        else:&#xA;            batch_train_indices = train_indices[j*batch_size:(j+1)*batch_size]&#xA;&#xA;        model.partial_fit(df_train[inputs].iloc[batch_train_indices], df_train[output].iloc[batch_train_indices], classes = [0,1])&#xA;&#xA;    sgd_minibatch_scores.append(model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42515" LastEditorUserId="42515" LastEditDate="2017-11-30T17:02:11.617" LastActivityDate="2019-12-25T15:01:10.180" Title="sklearn: SGDClassifier yields lower accuracy than LogisticRegression" Tags="|python|scikit-learn|logistic-regression|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25236" PostTypeId="2" ParentId="25221" CreationDate="2017-11-30T06:10:51.533" Score="1" Body="&lt;p&gt;This is helpful when the data is not understandable and you don't have any data dictionary or you have too many columns which doesn't make any sense even after through investigation, then it is wise to go for &lt;strong&gt;Dimensionality Reduction&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is very good chance that you loose information at granular level, for example you gave 100 columns and you got 10 PC's(Applied PCA) which can explains &lt;strong&gt;most&lt;/strong&gt; of the Data. As you can only get that much out of that technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Selection&lt;/strong&gt;: If you miss even a single feature which is significant WRT target variable but you overlooked that feature then your model might not be able explain the most, you are even satisfied with the result but still there is window of improvement. You should be very careful and you need check as many times(many iterations) as possible to build a good/decent model and get the best results out of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this may help you.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-11-30T06:10:51.533" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25237" PostTypeId="2" ParentId="25209" CreationDate="2017-11-30T06:22:06.060" Score="3" Body="&lt;p&gt;In Value function methods (or Critic methods) we usually choose one of the following options to select our actions after we have estimated the relevant value function:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Boltzman function&lt;/strong&gt; with a (inverse) temperature parameter that controls how much we 'listen' the maximum of the value function to select our actions.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;$\epsilon$-greedy:&lt;/strong&gt; selects the action with the maximum value function with probability (1-$\epsilon$) and a random one with probability $\epsilon$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In Policy Gradient (or Actor Methods) we have two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Stochastic Policy Gradients (SPG):&lt;/strong&gt; Output is a probability over actions. For your algorithm, the output would be the parameters of a pre-specified distribution (usually Gausian) as Neil described. Then you sample that distribution in a similar way you sample the Boltzman distribution.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Deterministic Policy Gradients (DPG):&lt;/strong&gt; Output is the value of an action (e.g. speed, height etc).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;From the above you can clearly see that when continuous action space is involved, PG offers much more plausible solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information I suggest you the Master thesis &lt;a href=&quot;https://riashatislam.files.wordpress.com/2016/11/riashat-islam-report.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;PG Methods: SGD-DPG&lt;/a&gt; of Riashat. He worked closely with one of the co-authors of Silver's DPG paper and the thesis is very well structured with MATLAB code available.&#xA;Policy Gradients methods are much harder to understand because of the math involved (and many times the different notation used). However I would suggest you to start from &lt;a href=&quot;http://www.scholarpedia.org/article/Policy_gradient_methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jan Peters&lt;/a&gt; and then get back to Silver's slides (my opinion :) ).&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-11-30T06:22:06.060" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="25239" PostTypeId="2" ParentId="25216" CreationDate="2017-11-30T07:55:37.013" Score="1" Body="&lt;p&gt;You can use reinforcement learning, in instance &lt;a href=&quot;https://gitlab.tubit.tu-berlin.de/shoefer/mbrrl2/tree/master&quot; rel=&quot;nofollow noreferrer&quot;&gt;mbrrl2&lt;/a&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As temperature is very stational dependent variable. You should have at least 1 year, better 2 years, to make some predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other way to solve this, is having a collection of historical temperatures, clustering them, and matching your dataset with the closest cluster.&lt;/p&gt;&#xA;" OwnerUserId="29959" LastActivityDate="2017-11-30T07:55:37.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25240" PostTypeId="2" ParentId="25231" CreationDate="2017-11-30T08:14:10.740" Score="2" Body="&lt;p&gt;In the linked tutorial, each kernel is 2-dimensional, and applied to a single channel/feature map from its input. To construct a feature map, the output from $N_{input\_channels}$ convolutions are added together - this is important as it allows to build feature maps that have interactions between feature maps in the previous layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the diagram, the first input layer has 1 channel (a greyscale image), so each kernel in layer 1 will generate a feature map. However, once you have 64 channels in layer 2, then to produce each feature map in layer 3 will require 64 kernels added together. If you want 256 feature maps in layer 3, and you expect all 64 inputs to affect each one, then you usually need 64 * 256 = 16384 kernels. The value 4096 is coming from some other aspect of the architecture not shown in the diagram, such as dividing the feature map into groups so that each output layer only processes a fraction of the input layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few notation and presentation differences between presentations and tutorials on convolutional networks, depending on the source. This is one of them. Other sources may arrange the kernels into a 4D structure: $N_{input\_channels} \times N_{output\_channels} \times K_{width} \times K_{height}$ to make the relationship between kernels and input/output more explicit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to show the same thing is to treat the kernels as 3D, and ensure that the kernel depth $K_{depth}$ is the same as the number of input channels $N_{input\_channels}$ - this is same thing mathematically as summing up multiple separate convolutions, and dimensions might be $N_{output\_channels} \times K_{width} \times K_{height} \times K_{depth}$ &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-11-30T10:04:03.663" LastActivityDate="2017-11-30T10:04:03.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25241" PostTypeId="1" AcceptedAnswerId="25249" CreationDate="2017-11-30T09:18:10.407" Score="1" ViewCount="8881" Body="&lt;p&gt;In storing floating point values both overflow and underflow problems cause loss of data. In machine learning tasks underflow is a common problem. I wanted to know if &lt;code&gt;double&lt;/code&gt; is better than &lt;code&gt;float64&lt;/code&gt; in TensorFlow or not and if there is any difference between them?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="13748" LastEditDate="2017-11-30T11:55:56.743" LastActivityDate="2017-11-30T12:05:17.717" Title="What is the difference between float64 and double in TensorFlow?" Tags="|machine-learning|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25242" PostTypeId="1" CreationDate="2017-11-30T10:11:41.627" Score="0" ViewCount="349" Body="&lt;p&gt;I have gone through few of the paper for AUC calculation, following paper has mentioned AUC can be calculated as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AUC = (Sensitivity +Specificity)/2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0306457309000259&quot; rel=&quot;nofollow noreferrer&quot;&gt;A systematic analysis of performance measures for classification tasks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know that whether it is correct or not.&lt;/p&gt;&#xA;" OwnerUserId="18818" LastActivityDate="2017-11-30T11:55:49.590" Title="Whether AUC can be calculated as average of sensitivity and specificity or not?" Tags="|machine-learning|neural-network|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25245" PostTypeId="1" CreationDate="2017-11-30T11:23:41.243" Score="1" ViewCount="596" Body="&lt;p&gt;I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are there and values are numerical). I have already made a prediction model and I am considering the values with high prediction errors are outliers/ anomalies. I have done the sensitivity analysis for individual input values but in the dataset values are correlated with some other input values, e.g. value 1 is correlated with value 3,4,7; value 2 is correlated with 5,10,18 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For outlier explanation, first I am checking if input values also contain any outlying inputs, if there are some then using sensitivity check I want to find if the values are more sensitive to the output value. Because the values are correlated with other inputs so individual sensitivity analysis does not make much sense, but in the end I want to find the most influential group of input values that makes the outlying value to normal. After that I will verify if it is valid for similar outliers and then I would provide the group as an explanation for the outlier. So my confusion is how I can check sensitivity of a group using in this case? If this approach does not sound logical, please let me know where does it sound confusing? &lt;/p&gt;&#xA;" OwnerUserId="37977" LastActivityDate="2024-01-06T13:06:05.907" Title="Sensitivity analysis in outlier explanation" Tags="|python|data-mining|bigdata|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25246" PostTypeId="5" CreationDate="2017-11-30T11:39:53.400" Score="0" Body="&lt;p&gt;Optical Character Recognition (OCR) is a computer vision task where the input is an image containing characters and/or digits, and being able to recognize these characters, transforming into a machine representation that allows easier manipulation.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastEditorUserId="13748" LastEditDate="2017-12-02T22:14:35.340" LastActivityDate="2017-12-02T22:14:35.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25247" PostTypeId="4" CreationDate="2017-11-30T11:39:53.400" Score="0" Body="Optical character recognition, usually abbreviated to OCR, is the mechanical or electronic translation of scanned images of handwritten, typewritten or printed text into machine-encoded text." OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-08T17:28:10.530" LastActivityDate="2019-04-08T17:28:10.530" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25248" PostTypeId="2" ParentId="25242" CreationDate="2017-11-30T11:55:49.590" Score="1" Body="&lt;p&gt;The article you mentioned is referring to the AUC of Receiver Operating Characteristic Curve (ROC-Curve), that is different from the AUC of the Precision-Recall Curve (PR-Curve). See this &lt;a href=&quot;http://www.chioka.in/differences-between-roc-auc-and-pr-auc/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; to know about the differences between the PR-Curve and the ROC-Curve. If what you want is that same value for the PR-Curve you should look at the F1-Score, see this wikipedia &lt;a href=&quot;https://en.wikipedia.org/wiki/Youden%27s_J_statistic&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; for an explanation of the relation between the AUC of the ROC-Curve, the Youden Index and the F1-Score. Also a good indicator of the AUC of the PR-Curve is the Average Precision (AP) as explained &lt;a href=&quot;https://stats.stackexchange.com/questions/157012/area-under-precision-recall-curve-auc-of-pr-curve-and-average-precision-ap/157019#157019&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-11-30T11:55:49.590" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25249" PostTypeId="2" ParentId="25241" CreationDate="2017-11-30T12:05:17.717" Score="4" Body="&lt;p&gt;Taking a look at tensorflow's &lt;code&gt;dtypes.py&lt;/code&gt;, there's this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;double = float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So &lt;code&gt;double&lt;/code&gt; is exactly the same at &lt;code&gt;float64&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-11-30T12:05:17.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25250" PostTypeId="1" AcceptedAnswerId="25430" CreationDate="2017-11-30T12:48:56.513" Score="12" ViewCount="3716" Body="&lt;p&gt;I'm trying to create a regression model that predicts the duration of a task. The training data I have consists of roughly 40 thousand completed tasks with these variables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Who performed the task (~250 different people)&lt;/li&gt;&#xA;&lt;li&gt;What part (subproject) of the project the task was performed on (~20 different parts)&lt;/li&gt;&#xA;&lt;li&gt;The type of task&lt;/li&gt;&#xA;&lt;li&gt;The start date of the task (10 years worth of data)&lt;/li&gt;&#xA;&lt;li&gt;How long the person who has to do the task estimates it will take&lt;/li&gt;&#xA;&lt;li&gt;The actual duration this task took to finish&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The duration can vary between half an hour to a couple of hundreds of hours, but is heavily right skewed (most tasks are completed within 10 hour). On log scale the distribution is still slightly right skewed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction doesn't have to be perfect, but I'm trying to improve the people's estimations. One question to ask is &quot;What measure can we use the define &lt;em&gt;beter&lt;/em&gt;?&quot; I think the best measure would be the Mean Squared Error (MSE) since it weighs large errors much worse than small errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before I turned to machine learning I tried some simple approaches such as adjusting the estimation by the average or median error, adjusting it by the average/median error grouped by person, grouped by subproject but each of these happened to perform worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With machine learning, one of the first problem I encountered was the number of categorical variables since for most models these have to be encoded someway (e.g. one-hot). Anyway, I tried to apply some linear models, for example with Stochastic Gradient Descent my approach would be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One-hot encode the categorical features&lt;/li&gt;&#xA;&lt;li&gt;The converted the date to unix timestamps&lt;/li&gt;&#xA;&lt;li&gt;Normalize all the features that are not already between 0 and 1&lt;/li&gt;&#xA;&lt;li&gt;Split the data in 80/20 learn and test sets.&lt;/li&gt;&#xA;&lt;li&gt;With Grid Search cross validation and the learn set try to find the best hyper parameters and fit the model.&lt;/li&gt;&#xA;&lt;li&gt;Predict with the test set&lt;/li&gt;&#xA;&lt;li&gt;Calculate the error/score&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now one thing I noticed was that the results varied quite a bit: On one run the MSE was close to double of another run (150 and 280). Another thing is that the MSE of the people's estimate is about 80, so my model performs a bit worse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During my efforts to improve the performance I stumbled across &lt;a href=&quot;https://datascience.stackexchange.com/questions/15472/which-algorithm-to-use-to-predict-the-duration-of-some-task&quot;&gt;this question&lt;/a&gt; where someone suggests to use survival models. Now I'm unfamilliar with these kinds of models but it sounded promising but during my initial tests with this it turns out to be way too slow for my purposes (too large of a dataset).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same Datascience answer that suggested to use the survival models (and &lt;a href=&quot;https://en.wikipedia.org/wiki/Proportional_hazards_model#Relationship_to_Poisson_models&quot; rel=&quot;noreferrer&quot;&gt;the Wikipedia page&lt;/a&gt;) they also mentioned Poisson regression, but I'm not sure how I would apply this to my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So a long story short: I have just two questions: 1. Was my approach of using SGD 'correct' and do you think I can improve the results with that? 2. Are other models better suited for this kind of prediction and if so, can you explain a bit how I would use them?&lt;/p&gt;&#xA;" OwnerUserId="42622" LastActivityDate="2023-09-09T14:11:42.860" Title="Predict task duration" Tags="|regression|survival-analysis|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25252" PostTypeId="1" CreationDate="2017-11-30T13:43:50.157" Score="0" ViewCount="100" Body="&lt;p&gt;I wanna Search how many times &quot;bad&quot;  and &quot;good&quot; words are repeated in the data frame and visualize  this with histogram. &lt;/p&gt;&#xA;" OwnerUserId="41419" LastActivityDate="2017-12-30T17:02:41.243" Title="Search the Number of occurrences of the particular words in data using Pandas." Tags="|machine-learning|python|pandas|nltk|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="25253" PostTypeId="1" AcceptedAnswerId="25257" CreationDate="2017-11-30T14:08:58.723" Score="1" ViewCount="303" Body="&lt;p&gt;I have a collection of statements which I need to classify into 5 classes. Each statement have meta data in different columns:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Author|Editor| date of release| statement | Class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can one use the meta data to improve the text classification task?&lt;/p&gt;&#xA;" OwnerUserId="40515" LastActivityDate="2017-11-30T14:46:57.303" Title="How to add incorporate meta data into text classification?" Tags="|machine-learning|classification|text|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-02T22:15:09.753" ContentLicense="CC BY-SA 3.0" />
  <row Id="25254" PostTypeId="1" CreationDate="2017-11-30T14:20:38.720" Score="1" ViewCount="62" Body="&lt;p&gt;I am currently looking into some papers discussing &lt;code&gt;Incident duration prediction&lt;/code&gt;. The dataset consists of ca. 250 monitored events. For all of those incidents are multiple features available and the main goal is the prediction of the &lt;code&gt;incident duration&lt;/code&gt;. These durations are right-skewed and log-normal distributed. ANOVA test (and KS) identified 13 binary features to be independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To predict the durations the researchers compared &lt;code&gt;Multiple linear regression&lt;/code&gt;, &lt;code&gt;DT&lt;/code&gt;, &lt;code&gt;ANN&lt;/code&gt;, &lt;code&gt;SVM / RVM&lt;/code&gt; and &lt;code&gt;KNN&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I am looking into a way to transfer the knowledge from this case to another task. Predicting the duration of incidents in an &lt;strong&gt;online system&lt;/strong&gt; (online in the sense of continuous flow of information) and also accounting for &lt;strong&gt;trends&lt;/strong&gt; (e.g. increasing duration of incidents because it is getting darker or colder at the same time throughout different seasons). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, there is such a huge amount of methods besides the mentioned methods (non-parametric, hazard models, reliability models, etc) - it seems like a jungle&lt;/p&gt;&#xA;" OwnerUserId="41919" LastActivityDate="2017-11-30T14:20:38.720" Title="Incident duration prediction but online with trend" Tags="|machine-learning|neural-network|regression|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25255" PostTypeId="1" CreationDate="2017-11-30T14:25:55.480" Score="3" ViewCount="567" Body="&lt;p&gt;I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What stopping metric can be used to optimize the sensitivity score on validation set?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current set-up uses AUC as stopping metric.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- as.h2o(df)&#xA;split &amp;lt;- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%&#xA;train &amp;lt;- h2o.assign(split[[1]], &quot;train.hex&quot;) # 60%&#xA;valid &amp;lt;- h2o.assign(split[[2]], &quot;valid.hex&quot;) # 20%&#xA;test &amp;lt;- h2o.assign(split[[3]], &quot;test.hex&quot;) # 20%&#xA;&#xA;x &amp;lt;- setdiff(names(df), &quot;disease&quot;)&#xA;y &amp;lt;- &quot;disease&quot;&#xA;&#xA;gbm &amp;lt;- h2o.gbm(&#xA;  x = x, &#xA;  y = y, &#xA;  training_frame = train, &#xA;  validation_frame = valid,&#xA;  ntrees = 10000,                                                            &#xA;  learn_rate=0.01,     &#xA;  # Stopping parameters                                                    &#xA;  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = &quot;AUC&quot;, &#xA;  sample_rate = 0.8,                                                       &#xA;  col_sample_rate = 0.8,                                                   &#xA;  seed = 1234,                                                             &#xA;  nfolds = 50,&#xA;  score_tree_interval = 10                                                 &#xA;)&#xA;h2o.auc(h2o.performance(gbm, valid = TRUE)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42631" LastActivityDate="2020-01-24T13:30:01.287" Title="What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?" Tags="|machine-learning|classification|r|cross-validation|gbm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25256" PostTypeId="2" ParentId="25252" CreationDate="2017-11-30T14:34:35.250" Score="0" Body="&lt;p&gt;Without any knowledge how you would define 'good' or 'bad' this would be the general approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def count_good(text):&#xA;    # TODO: how do you define good?&#xA;    pass&#xA;&#xA;&#xA;def count_bad(text):&#xA;    # TODO: how do you define bad?&#xA;    pass&#xA;&#xA;def analyze(df)&#xA;    good_count = df.words.apply(count_good).sum()&#xA;    bad_count = df.words.apply(count_bad).sum()&#xA;&#xA;    plt.bar([0, 1], [good_count, bad_count])&#xA;    plt.xticks([0, 1], ['Good', 'Bad'])&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But you really need to improve your question if you want a serious answer.&lt;/p&gt;&#xA;" OwnerUserId="42622" LastActivityDate="2017-11-30T14:34:35.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25257" PostTypeId="2" ParentId="25253" CreationDate="2017-11-30T14:46:57.303" Score="2" Body="&lt;p&gt;Some models cannot really handle this, while others lend themselves for it easily. I'll explain two approaches that you could use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Naive bayes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Naive Bayes you can use other categorical values as well as your normal n-grams or sparse bag of words vectors. Just add them one-hot encoded to your features and it is also incorporated. With numerical features you would need to use something like Gaussian Naive Bayes, to fit a distribution to your features per target class, then you can use the likelihoods of these features per class to compute the probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Neural network&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use a neural network approach like CNNs or RNNs, you can add any type of feature representation network and concatenate it somewhere in your original network. In your case you would have a softmax at the end of your RNN. Before this, concatenate the output of your 'normal-feature' neural network, add some dense layers and feed this to your softmax output layer. This way you can train your model end-to-end and it will learn important interactions as well.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-11-30T14:46:57.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25258" PostTypeId="1" CreationDate="2017-11-30T15:29:35.163" Score="1" ViewCount="3671" Body="&lt;p&gt;I have a Cassandra cluster of 2 nodes and a keyspace &quot;test&quot; with a replication factor of 2.  The consistency level is &lt;strong&gt;All&lt;/strong&gt;. I am running a spark job which dumps data in Cassandra database. &#xA;So, the thing is in one spark job I dump the data multiple times in Cassandra database. Sometimes it shows me the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;not enough replicas available for query at consistency all (2 required but only 1 alive)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem is that if I run the same job again then it might or might not give this error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Need help to understand why this error might be coming and why such an inconsistency in the error i.e. it is coming sometimes only.&lt;/p&gt;&#xA;" OwnerUserId="24157" LastEditorUserId="11097" LastEditDate="2017-11-30T16:05:50.650" LastActivityDate="2017-11-30T17:46:21.200" Title="Not enough replicas available for query at consistency all (2 required but only 1 alive)" Tags="|bigdata|apache-spark|databases|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25259" PostTypeId="1" AcceptedAnswerId="25322" CreationDate="2017-11-30T15:42:04.850" Score="8" ViewCount="3498" Body="&lt;p&gt;I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fRoy9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fRoy9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are these two conditions? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the logic behind the first derivative equality? Is it just that we assume these two derivatives should be equal since there should be &lt;strong&gt;some kind of connection with direction of value function approximation gradient and our policy like-hood&lt;/strong&gt;?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then what is the epsilon value? What is this mean square value? Q value with w parameters means the Q values function new approximation. And the other Q value is the value got by following our policy right? So is this OFF policy value approximation or ON policy? I think this is on policy since normally in the Q value update we take the max.  &lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="29169" LastEditDate="2019-06-07T23:42:18.793" LastActivityDate="2019-06-07T23:42:18.793" Title="What is Compatible Function Approximation theorem in reinforcement learning?" Tags="|deep-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25260" PostTypeId="5" CreationDate="2017-11-30T15:52:52.957" Score="0" Body="&lt;p&gt;Everything related to recommender systems&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Types of Recommender Systems&lt;/strong&gt;&#xA;Some types of recommender systems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Collaborative Filtering&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Based on user-interaction data such as &quot;user A watches movie Z&quot;&lt;/li&gt;&#xA;&lt;li&gt;Common methods: Matrix Factorization, Neighborhood Models&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Content Based Recommender Systems &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Based on user and item content such as user demographics, movie genres, actors in the movie, etc&lt;/li&gt;&#xA;&lt;li&gt;Common methods: Classification, Regression&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44089" LastEditorUserId="11097" LastEditDate="2018-03-27T12:16:09.710" LastActivityDate="2018-03-27T12:16:09.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25261" PostTypeId="4" CreationDate="2017-11-30T15:52:52.957" Score="0" Body="Everything related to recommender systems&#xD;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-09T10:21:41.380" LastActivityDate="2019-04-09T10:21:41.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25262" PostTypeId="1" CreationDate="2017-11-30T16:41:43.627" Score="2" ViewCount="176" Body="&lt;p&gt;I'm trying to predict additional recipients of a message given the content of the message (like subject and body) and the current recipients of the message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for ex: I have 4 users in the system U1, U2, U3 and U4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following messages between them&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;message-id  message-content sender  recipients&#xA;     m1          text        U1      U2,U3&#xA;     m2          text        U1      U2&#xA;     m3          text        U1      U3&#xA;     m4          text        U2      U3,U4&#xA;     m5          text        U3      U4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see U1 only communicates (Send/Receive) with U2 and U3. Similarly, U2 communicates with U1, U3 and U4 while U3 communicates with U1, U2 and U4. Lastly U4 communicates with U2 and U3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is when a user is composing a message and already entered some content and recipients on the To line, I want to suggest other recipients the user should include in the To line. The suggestion should be relevant to the message's content and also only contain users with whom the user has communicated with previously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for U1 we should never recommend U4 as there is no communication between them before. Similarly, for U4 we would never suggest U1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently thinking of solving this as a multi-label classification problem where I generate a personalized model per user using Binary Relevance method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For U1, the data looks as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;message-id  message-content U2  U3&#xA;   m1          text         1   1&#xA;   m2          text         1   0&#xA;   m3          text         0   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I can train a OneVsRest classifier (from scikit-learn) on this data using svm.LinearSVC (or some other classifier) to predict multi-recipients given a message. I'll use content and recipients of the message as features. This works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is this needs a personalized model per user. Given I have 10's of millions of users and each user sends/receives 1000's of messages per month and communicates with 1000's of other users overall, how can I create a single global model with personalized features for prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to avoid creating millions of personal model per user if possible and want to train a single global model using data for all users and then do personalized predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is multi-label classification not suitable for learning a global model? Should I instead use a ranking model approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest how can I do this?&lt;/p&gt;&#xA;" OwnerUserId="40751" LastActivityDate="2017-11-30T16:41:43.627" Title="How to create a global model with personalized features for multi-label classification problem" Tags="|python|scikit-learn|svm|linear-regression|multilabel-classification|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25263" PostTypeId="2" ParentId="25252" CreationDate="2017-11-30T16:49:14.377" Score="1" Body="&lt;p&gt;Jurgy's answer should work. However, based on comments above, you can simply do like this,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['yourcolumn_name_good_bad'].value_counts().plot(kind='bar')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Otherwise, if you want to use &lt;code&gt;seaborn&lt;/code&gt; library and create a plot follow &lt;a href=&quot;https://stackoverflow.com/questions/31460146/plotting-value-counts-in-seaborn-barplot&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="34028" LastEditorUserId="34028" LastEditDate="2017-11-30T16:55:56.707" LastActivityDate="2017-11-30T16:55:56.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25264" PostTypeId="2" ParentId="6508" CreationDate="2017-11-30T16:51:13.633" Score="7" Body="&lt;p&gt;Finding the elbow can be made more easier by computing the angles between the consecutive segments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replace your:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kIdx = 10-1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seg_threshold = 0.95 #Set this to your desired target&#xA;&#xA;#The angle between three points&#xA;def segments_gain(p1, v, p2):&#xA;    vp1 = np.linalg.norm(p1 - v)&#xA;    vp2 = np.linalg.norm(p2 - v)&#xA;    p1p2 = np.linalg.norm(p1 - p2)&#xA;    return np.arccos((vp1**2 + vp2**2 - p1p2**2) / (2 * vp1 * vp2)) / np.pi&#xA;&#xA;#Normalize the data&#xA;criterion = np.array(avgWithinSS)&#xA;criterion = (criterion - criterion.min()) / (criterion.max() - criterion.min())&#xA;&#xA;#Compute the angles&#xA;seg_gains = np.array([0, ] + [segments_gain(*&#xA;        [np.array([K[j], criterion[j]]) for j in range(i-1, i+2)]&#xA;    ) for i in range(len(K) - 2)] + [np.nan, ])&#xA;&#xA;#Get the first index satisfying the threshold&#xA;kIdx = np.argmax(seg_gains &amp;gt; seg_threshold)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and you will see something like:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/o81mG.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o81mG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you visualize the seg_gains, you will see something like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/UyV6R.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UyV6R.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope you can find the tricky elbow now :)&lt;/p&gt;&#xA;" OwnerUserId="42637" LastEditorUserId="42637" LastEditDate="2017-12-01T17:10:04.803" LastActivityDate="2017-12-01T17:10:04.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25266" PostTypeId="1" AcceptedAnswerId="25272" CreationDate="2017-11-30T18:28:09.573" Score="0" ViewCount="506" Body="&lt;p&gt;So, it seems that OneHotEncoder won't work with the np.int64 datatype (only np.int32)! Here's a sample of code:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;import numpy as np&lt;/p&gt;&#xA;&lt;p&gt;import pandas as pd&lt;/p&gt;&#xA;&lt;p&gt;from sklearn.preprocessing import OneHotEncoder&lt;/p&gt;&#xA;&lt;p&gt;a = np.array([[56748683,8511896545,51001984320],[18643548615,28614357465,56748683],[8511896545,51001984320,40084357915]])&lt;/p&gt;&#xA;&lt;p&gt;b = pd.DataFrame(a, dtype=np.int64)&lt;/p&gt;&#xA;&lt;p&gt;ohe = OneHotEncoder()&lt;/p&gt;&#xA;&lt;p&gt;c = ohe.fit_transform(b).toarray()&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;When I run this I get the following error: &amp;quot;ValueError: X needs to contain only non-negative integers.&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;As you can see, X DOES contain only non-negative integers! When I trim a few of the digits and change the datatype to int32 it works fine:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;a = np.array([[56748,8511896,51001984],[18643548,28614357,56748],[8511896,51001984,40084357]])&lt;/p&gt;&#xA;&lt;p&gt;b = pd.DataFrame(a, dtype=np.int32)&lt;/p&gt;&#xA;&lt;p&gt;ohe = OneHotEncoder()&lt;/p&gt;&#xA;&lt;p&gt;c = ohe.fit_transform(b).toarray()&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Unfortunately, the data I need to encode has 11 digits (which can't be represented by int32). So, any suggestions would be helpful...&lt;/p&gt;&#xA;&lt;p&gt;Also, I should mention, I don't necessarily need a one hot encoding, just need to create dummy variables. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="42643" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-11-30T22:03:46.633" Title="Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing" Tags="|python|scikit-learn|preprocessing|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25267" PostTypeId="1" CreationDate="2017-11-30T19:33:23.220" Score="39" ViewCount="105139" Body="&lt;p&gt;What is the difference between &lt;code&gt;val_loss&lt;/code&gt; and &lt;code&gt;loss&lt;/code&gt; during training in Keras?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/20&#xA;1000/1000 [==============================] - 1s - loss: 0.1760, val_loss: 0.2032  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On some sites I read that on validation, dropout was not working.&lt;/p&gt;&#xA;" OwnerUserId="39706" LastEditorUserId="836" LastEditDate="2017-12-31T13:15:33.193" LastActivityDate="2023-03-19T14:36:46.480" Title="Keras difference beetween val_loss and loss during training" Tags="|machine-learning|deep-learning|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25268" PostTypeId="1" CreationDate="2017-11-30T19:57:04.113" Score="6" ViewCount="2826" Body="&lt;p&gt;I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to plot a correlation matrix or a correlation graph between two of the categories and selected columns?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, 19 columns on x and 12 columns on y. Or summing 12 and 19 columns and having a correlation between only 31 columns instead of 150. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Python. Which packages could help me?&lt;/p&gt;&#xA;" OwnerUserId="39661" LastEditorUserId="1330" LastEditDate="2017-12-01T21:41:41.623" LastActivityDate="2017-12-01T21:41:41.623" Title="Correlation between specific columns of a data set" Tags="|machine-learning|python|visualization|pandas|matplotlib|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25269" PostTypeId="2" ParentId="25267" CreationDate="2017-11-30T20:03:29.963" Score="27" Body="&lt;p&gt;&lt;code&gt;val_loss&lt;/code&gt; is the value of cost function for your cross-validation data and loss is the value of cost function for your training data. On validation data, neurons using drop out do not drop random neurons. The reason is that during training we use drop out in order to add some noise for avoiding over-fitting. During calculating cross-validation, we are in the recall phase and not in the training phase. We use all the capabilities of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to one of our dear friends, I quote and explain the contents from &lt;a href=&quot;https://keras.io/models/sequential/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; which are useful.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;validation_split&lt;/code&gt;: Float between 0 and 1. The fraction of the training&#xA;  data to be used as validation data. The model will set apart this&#xA;  fraction of the training data, will not train on it, and will evaluate&#xA;  the loss and any model metrics on this data at the end of each epoch.&#xA;  The validation data is selected from the last samples in the &lt;em&gt;x&lt;/em&gt; and&#xA;  &lt;em&gt;y&lt;/em&gt; data provided, before shuffling.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;code&gt;validation_data&lt;/code&gt;: tuple (x_val, y_val) or tuple (x_val, y_val,&#xA;  val_sample_weights) on which to evaluate the loss and any model&#xA;  metrics at the end of each epoch. The model will not be trained on&#xA;  this data. This will override  validation_split.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you can see&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0, steps_per_epoch=None, validation_steps=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;fit&lt;/code&gt; method used in &lt;code&gt;Keras&lt;/code&gt; has a parameter named validation_split, which specifies the percentage of data used for evaluating the model which is created after each epoch. After evaluating the model using this amount of data, that will be reported by &lt;code&gt;val_loss&lt;/code&gt; if you've set verbose to &lt;code&gt;1&lt;/code&gt;; moreover, as the documentation clearly specifies, you can use either &lt;code&gt;validation_data&lt;/code&gt; or &lt;code&gt;validation_split&lt;/code&gt;. Cross-validation data is used to investigate whether your model over-fits the data or does not. This is what we can understand whether our model has generalization capability or not.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="25849" LastEditDate="2020-01-21T13:17:40.270" LastActivityDate="2020-01-21T13:17:40.270" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="25271" PostTypeId="2" ParentId="25268" CreationDate="2017-11-30T21:04:54.237" Score="3" Body="&lt;p&gt;I recommend you to use the following example and try to manipulate the arguments and adjust them for your work:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import cm&#xA;cmap = cm.get_cmap('gnuplot')&#xA;scatter = pd.scatter_matrix(YOUR_TRAINING_DATA, c = YOUR_LABELS_OF_TRAINING, marker = 'o', s = 40, hist_kwds = {'bins':15}, figsize = (12, 12), cmap = cmap)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1fAcf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1fAcf.png&quot; alt=&quot;Code and image given from coursera data science course&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-11-30T21:04:54.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25272" PostTypeId="2" ParentId="25266" CreationDate="2017-11-30T22:03:46.633" Score="1" Body="&lt;p&gt;Try the following approach:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [51]: pd.get_dummies(b.astype(str), prefix_sep='')&#xA;Out[51]:&#xA;   018643548  056748  08511896  128614357  151001984  18511896  240084357  251001984  256748&#xA;0          0       1         0          0          0         1          0          1       0&#xA;1          1       0         0          1          0         0          0          0       1&#xA;2          0       0         1          0          1         0          1          0       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-11-30T22:03:46.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25274" PostTypeId="2" ParentId="17624" CreationDate="2017-12-01T00:58:48.083" Score="1" Body="&lt;p&gt;Twitter has rules that limit sharing of complete datasets, instead limiting sharing to the tweet ids (see this &lt;a href=&quot;https://twittercommunity.com/t/twitter-and-open-data-in-academia/51934&quot; rel=&quot;nofollow noreferrer&quot;&gt;discussion&lt;/a&gt; and the &lt;a href=&quot;https://developer.twitter.com/en/developer-terms/agreement-and-policy&quot; rel=&quot;nofollow noreferrer&quot;&gt;developer agreement&lt;/a&gt;).  Tools such as &lt;a href=&quot;https://github.com/DocNow/twarc&quot; rel=&quot;nofollow noreferrer&quot;&gt;twarc&lt;/a&gt; can be used to &quot;rehydrate&quot; the tweet metadata by calling the Twitter api and retrieving the info.  The Twitter API has rate limiting which can make this a somewhat slow process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a package that works in R, see &lt;a href=&quot;https://cran.r-project.org/web/packages/RTextTools/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RTextTools&lt;/a&gt; or check out &lt;a href=&quot;https://cran.r-project.org/web/packages/RWeka/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RWeka&lt;/a&gt; which bridges the gap between R and Java to use Weka.  If using Python, you can also use scikit-learn's svm implementation.&lt;/p&gt;&#xA;" OwnerUserId="42657" LastActivityDate="2017-12-01T00:58:48.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25276" PostTypeId="1" AcceptedAnswerId="25286" CreationDate="2017-12-01T03:25:30.507" Score="5" ViewCount="4462" Body="&lt;p&gt;I'm reading a wonderful &lt;a href=&quot;http://toritris.weebly.com/perceptron-5-xor-how--why-neurons-work-together.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; about neural network. This is the best tutorial I've ever seen but I can't understand one thing as below:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the link above, it is talking about how the neural work solves the XOR problem.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It says that we need two lines to separate the four points. But I don't know the second table.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XOR:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input1    input2    output&#xA;  0         0         0&#xA;  0         1         1&#xA;  1         0         1&#xA;  1         1         0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First table:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input1    input2    output&#xA;  0         0         0&#xA;  0         1         1&#xA;  1         0         1&#xA;  1         1         1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In my opinion, the first table is OK because it includes the XOR, which means that what the second table need to do is to remove the forth input. So I think the second table should be as below:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input1    input2    output&#xA;  0         0         1&#xA;  0         1         1&#xA;  1         0         1&#xA;  1         1         0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How in the link it says the second table is like this:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input1    input2    output&#xA;  0         0         0&#xA;  0         1         0&#xA;  1         0         0&#xA;  1         1         1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In a word, I can understand why the single layer neural network can't solve the XOR problem but I can't understand how the two layers neural network work to solve it.&lt;/p&gt;&#xA;" OwnerUserId="42535" LastEditorUserId="42535" LastEditDate="2017-12-04T00:44:32.630" LastActivityDate="2017-12-04T00:44:32.630" Title="How does neural network solve XOR problem" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25279" PostTypeId="2" ParentId="10179" CreationDate="2017-12-01T05:21:52.697" Score="6" Body="&lt;p&gt;&lt;code&gt;set_group&lt;/code&gt; is very important to ranking, because only the scores in one group are comparable.&#xA;You can sort data according to their scores in their own group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For easy ranking, you can use my &lt;a href=&quot;https://github.com/bigdong89/xgboostExtension&quot; rel=&quot;noreferrer&quot;&gt;xgboostExtension&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="42668" LastEditorUserId="29575" LastEditDate="2017-12-01T16:08:34.733" LastActivityDate="2017-12-01T16:08:34.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25280" PostTypeId="1" CreationDate="2017-12-01T07:20:53.670" Score="2" ViewCount="65" Body="&lt;p&gt;I have a data set and few human rules. I want to learn the machine learning system. One of the paper I came across with is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.petuum.com/pdf/Hu_etal_ACL16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Harnessing Neural Network&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Above paper trained some teacher network to learn the system in order to improve the accuracy but they still require enough data.&#xA;But my requirement is different. It may be possible that we have very few data points corresponding to a rule. How can I design a system which can utilize the both learning from data set and human rules.&lt;/p&gt;&#xA;" OwnerUserId="32764" LastActivityDate="2021-05-06T17:03:41.897" Title="Machine learning system which can learn from data and human rules" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25281" PostTypeId="1" AcceptedAnswerId="25285" CreationDate="2017-12-01T08:31:10.297" Score="-1" ViewCount="124" Body="&lt;p&gt;I have a list.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;list = ['It is a delightfully naive and entertaining movie',&#xA;        'The songs are boring and dated in 2009',&#xA;        'was a great movie from genre director Luc Besson']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and I want a result like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;list_result = [ It is a delightfully naive and entertaining movie The songs are boring and dated in 2009 was a great movie from genre director Luc Besson]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how I can do this? (list_result can also be a corpus.)&lt;/p&gt;&#xA;" OwnerUserId="41419" LastEditorUserId="20492" LastEditDate="2017-12-01T09:59:12.383" LastActivityDate="2017-12-01T09:59:12.383" Title="concatenating the content of list in python" Tags="|python|pandas|" AnswerCount="3" CommentCount="0" ClosedDate="2017-12-01T14:27:01.877" ContentLicense="CC BY-SA 3.0" />
  <row Id="25282" PostTypeId="1" CreationDate="2017-12-01T08:36:59.640" Score="1" ViewCount="1232" Body="&lt;p&gt;I have a homework question that ask me to derive the update rule from the distortion metric:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Swv8M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Swv8M.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It then says: &quot;The update rule is derived by computing the gradient for each element of the k-th mean and solving for the value where the gradient is zero.&quot; Can somebody tell me what this question means? I already understand how k-means clustering works, but I need some clarification about what it means by &quot;k-th mean&quot; and &quot;gradient&quot;. &lt;/p&gt;&#xA;" OwnerUserId="42674" LastActivityDate="2017-12-01T08:45:11.347" Title="Deriving the update rule for mean in k-means clustering" Tags="|machine-learning|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25283" PostTypeId="2" ParentId="25282" CreationDate="2017-12-01T08:45:11.347" Score="1" Body="&lt;p&gt;The k-th mean just means the mean of the k-th cluster. We want to minimize the sum of squares of each of the clusters, which we do by minimizing this so-called distortion metric. The gradient is a multivariate generalization of derivatives and the gradient of this distortion metric is the zero-vector when this is optimized, and after this you update your clusters.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-12-01T08:45:11.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25284" PostTypeId="2" ParentId="25276" CreationDate="2017-12-01T08:50:56.053" Score="1" Body="&lt;p&gt;The table that you are referring to is doing OR operation. whenever you have just a neuron in your net you are able to have one line to separate your data. but for xor data you have to have two line separators. it is common for solving this problem to have two neurons in the first layers which both do OR operation and in the second layer to have one neuron to do and operation. if you stack these two layers, you will get the result. I suggest you to see the shape of &lt;a href=&quot;https://datascience.stackexchange.com/questions/21896/can-the-linearly-non-separable-data-be-learned-using-polynomial-features-with-lo&quot;&gt;this&lt;/a&gt; to see the nonlinear decision boundary. Also the second table is doing AND operation.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-01T08:50:56.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25285" PostTypeId="2" ParentId="25281" CreationDate="2017-12-01T09:02:50.977" Score="1" Body="&lt;pre&gt;&lt;code&gt;result = &quot;&quot;&#xA;for sentence in list:&#xA;    result += sentence&#xA;    result += &quot; &quot;&#xA;list_result = [result]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Go over list comprehension if you want a more pythonic way to do it, here is the most understandable to begin.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-12-01T09:02:50.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25286" PostTypeId="2" ParentId="25276" CreationDate="2017-12-01T09:25:15.243" Score="4" Body="&lt;p&gt;Notice that the first table (orange line) is performing an OR operation and the second table (blue line) is performing an AND operation. XOR can be defined as &lt;code&gt;(x OR y) AND NOT (x AND y)&lt;/code&gt; or $(x \lor y) \land \lnot (x \land y)$, so in other words: orange should fire and blue shouldn't fire. If we now look at this figure from your tutorial:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/32gkA.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/32gkA.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we can see that the weight of blue in the second layer is negative and small enough (more negative) such that the output can never fire if blue fires, i.e. the output can't fire if both inputs are firing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 + 0 \ngtr 1 : \emptyset $ shouldn't fire  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-2 + 0 \ngtr 1 : (x \land y) = T$ shouldn't fire  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 + 1.1 \gt 1 : (x \lor y) \land \lnot (x \land y) = T$ should fire  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-2 + 1.1 \ngtr 1 : (x \land y) \land (x \lor y) = T$ shouldn't fire&lt;/p&gt;&#xA;" OwnerUserId="42622" LastEditorUserId="42622" LastEditDate="2017-12-01T09:35:31.333" LastActivityDate="2017-12-01T09:35:31.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25287" PostTypeId="2" ParentId="25281" CreationDate="2017-12-01T09:43:24.257" Score="1" Body="&lt;p&gt;Firstly, this question isn't suitable for Datascience, go over to &lt;a href=&quot;https://stackoverflow.com&quot;&gt;Stackoverflow&lt;/a&gt; for these kind of questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, I advice you to look up some introductory tutorials for Python. The Python wiki has a whole list of these &lt;a href=&quot;https://wiki.python.org/moin/BeginnersGuide/Programmers&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, as to answer your question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l = ['AB', 'CD', 'EF']&#xA;o = ' '.join(l)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; o&#xA;'AB CD EF'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42622" LastActivityDate="2017-12-01T09:43:24.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25288" PostTypeId="2" ParentId="25281" CreationDate="2017-12-01T09:52:21.210" Score="1" Body="&lt;p&gt;Suppose you have the following DF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [11]: df&#xA;Out[11]:&#xA;                                                 txt&#xA;0  It is a delightfully naive and entertaining movie&#xA;1             The songs are boring and dated in 2009&#xA;2   was a great movie from genre director Luc Besson&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.cat.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Series.str.cat&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [12]: df['txt'].str.cat(sep=' ')&#xA;Out[12]: 'It is a delightfully naive and entertaining movie The songs are boring and dated in 2009 was a great movie from genre director Luc Besson'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2017-12-01T09:52:21.210" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25289" PostTypeId="1" CreationDate="2017-12-01T10:37:27.800" Score="1" ViewCount="375" Body="&lt;p&gt;I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the &lt;a href=&quot;http://cocodataset.org/#home&quot; rel=&quot;nofollow noreferrer&quot;&gt;coco&lt;/a&gt; dataset which has multiple labels per image. I'd like to evaluate the performance of the model but I'm not sure what type of metric should be used here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it was just a single label per image, I'd go for something like mean average precision (given a query image of a dog, check how many dog images the system retrieved, evaluate MAP). But this obviously cannot be used on the multi label task (given a query image of three classes, system retrieved image with just one of them, it's not totally correct but it's not incorrect either). So are there any commonly used metrics to evaluate this kind of tasks? If so, please refer me to them. Or do I have to come up with something own (maybe some kind of a weighted MAP)?&lt;/p&gt;&#xA;" OwnerUserId="40265" LastActivityDate="2017-12-01T12:06:00.140" Title="How to evaluate multi label image retrieval model" Tags="|image-classification|information-retrieval|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25290" PostTypeId="2" ParentId="5737" CreationDate="2017-12-01T11:14:41.023" Score="0" Body="&lt;p&gt;One thing I cannot see here is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cleaning, pre-processing and structuring&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Although this does not sound too exciting, it is probably the most important thing when it comes to working with &quot;real&quot; data. Real data is never clean. If you receive an Email data dump you'll find all kinds of garbage. Forwarded messages, different kinds of quotation styles, different languages (or mixes), bullet point lists etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes you'll see messages with 99% garbage and only one line with actual information embedded in a stream of forwarded messages etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning how to clean this data and extracting the relevant information is actually the hard part. It's not that fancy, everybody likes to work with clean data, but it's nevertheless another thing that can be done with an Email data set that comes from the real world.&lt;/p&gt;&#xA;" OwnerUserId="25289" LastActivityDate="2017-12-01T11:14:41.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25291" PostTypeId="1" CreationDate="2017-12-01T11:35:06.777" Score="3" ViewCount="3991" Body="&lt;p&gt;I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DeprecationWarning: The StanfordTokenizer will be deprecated in version &#xA;3.2.5. Please use nltk.tag.corenlp.CoreNLPPOSTagger or &#xA;nltk.tag.corenlp.CoreNLPNERTagger instead.&#xA;super(StanfordNERTagger, self).__init__(*args, **kwargs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My code looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk import word_tokenize, pos_tag, ne_chunk&#xA;from nltk.tag import StanfordNERTagger&#xA;&#xA;sentence = &quot;Today George went to school and met his friend Peter.&quot;&#xA;&#xA;# stanford's NER tagger 3 entity classification&#xA;st = StanfordNERTagger('/home/hercules/Desktop/PhD/Tools/stanford-ner-&#xA;     2017-06-09/classifiers/english.all.3class.distsim.crf.ser.gz',&#xA;     '/home/hercules/Desktop/PhD/Tools/stanford-ner-2017-06-09/stanford-&#xA;     ner.jar',&#xA;     encoding='utf-8')&#xA;&#xA;tokenized_text = word_tokenize(sentence)&#xA;classified_text = st.tag(tokenized_text)&#xA;&#xA;print(&quot;Stanford NER tagger:&quot;)&#xA;print(classified_text)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to use CoreNLPNERTagger but I could not find any examples or documentation. &#xA;I only found this &lt;a href=&quot;http://www.nltk.org/_modules/nltk/tag/stanford.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link:&lt;/a&gt;&#xA;where it gives something like an example in the comments of the &#xA;class CoreNLPNERTagger(CoreNLPTagger)                                      (I found it by searching the keyword &quot;CoreNLPNERTagge&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to follow that example with no use.&#xA;I think I should start (if that is the correct term) the coreNLP server first but if is that the case I don't know how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone got any idea or advice I would be grateful.&lt;/p&gt;&#xA;" OwnerUserId="42460" LastActivityDate="2018-07-30T19:53:40.067" Title="StanfordTokenizer will be deprecated in version 3.2.5 Warning" Tags="|python|nltk|named-entity-recognition|stanford-nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25292" PostTypeId="2" ParentId="25289" CreationDate="2017-12-01T12:06:00.140" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;First consider analyzing the results per class (with normal accuracy/F-score/whatever) - this way you can have a good idea of where the algorithm is doing well and where it may be improved.&lt;/li&gt;&#xA;&lt;li&gt;Consider only the exact match. It's a &quot;hit&quot; if the algorithm got the exact labels that it should have. You can then use the hit count to calculate precision, recall and F-score.&lt;/li&gt;&#xA;&lt;li&gt;Hamming Loss: this is a favorite of researchers in my experience, it represents the how many of the total labels were misclassified&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-label_classification#Statistics_and_evaluation_metrics&quot; rel=&quot;nofollow noreferrer&quot;&gt;relevant Wikipedia article&lt;/a&gt; seems to agree with me and adds the Jaccard index to the mix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper that presents the famous Classifier Chains method (READ J. et al, Classifier Chains for Multi-label Classification, 2009) uses four different evaluation methods: a accuracy variation that closely resembles Jaccard distance, a similarly changed F1-score, and a log-loss function. The fourth method they use to evaluate is the area under the Precision-Recall curve, but that's one I'd argue should not be used (see the work on Precision-Recall-Gain curves by Peter Flach).&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-12-01T12:06:00.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25293" PostTypeId="1" CreationDate="2017-12-01T13:48:34.893" Score="1" ViewCount="152" Body="&lt;p&gt;The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that because of Jensen's inequality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect of the averaging is basically just to shrink the weights by some factor. Why average at all? Has anyone tried simply not averaging, and controlling the magnitude of the parameters with e.g. L2 regularization?  One would still get the benefits of preventing codependence, but one would need to regularize some other way.&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-12-01T14:27:19.380" Title="Dropout without the averaging" Tags="|neural-network|regularization|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25295" PostTypeId="2" ParentId="25293" CreationDate="2017-12-01T14:19:08.297" Score="1" Body="&lt;p&gt;With dropout, the sum of activations feeding each neuron during training and testing needs to be roughly the same. The problem is that during training some neurons are masked out, and this does not happen during testing. Therefore there &lt;em&gt;has&lt;/em&gt; to be a scaling difference between train and test phases using dropout, otherwise each neuron would be receiving larger magnitude inputs and the network as a whole could behave very differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is &lt;em&gt;not&lt;/em&gt; mathematically the same as averaging the weights - although the effect is compared intuitively to bagging many related neural networks and averaging their output. Instead, it is usually explained that dropout scales the neuron &lt;em&gt;activations&lt;/em&gt; to have same expected total magnitude when dropout is active (during training) vs inactive (during testing/prediction). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying L2 or any other kind of parameter-controlling regularisation technique will not help with this, because these do not change any of the activations between train and test phases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You either treat training activation values as canonical and adjust activations to be lower at test time (multiply by keep probability $p$), which is &quot;classic dropout&quot; or you treat test activation values as canonical and multiply activations by $\frac{1}{p}$ during training, which is called &quot;inverse dropout&quot;. The latter approach is nowadays more common, as it is possible to completely ignore dropout layers during prediction that way.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-12-01T14:27:19.380" LastActivityDate="2017-12-01T14:27:19.380" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25297" PostTypeId="2" ParentId="19241" CreationDate="2017-12-01T14:47:38.320" Score="1" Body="&lt;p&gt;Could the difference be caused by the step pattern you are using? Dtw in R defaults to the &lt;code&gt;symmetric2&lt;/code&gt; step pattern. Explanation of the step pattern from &lt;a href=&quot;https://cran.r-project.org/web/packages/dtw/dtw.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Toni Giorgino's paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;symmetric2 is normalizable, symmetric, with no local slope constraints. Since one diagonal step costs as much as the two equivalent steps along the sides, it can be normalized dividing by $N+M$ (query+reference lengths).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I thought this might be the problem after reading &lt;a href=&quot;http://wiktorski.github.io/blog/comparing-dtw-r-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Comparing Dynamic Time Warping in R and Python&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your problem is the same try adding &lt;code&gt;step=symmetric1&lt;/code&gt; to your &lt;code&gt;R.dtw&lt;/code&gt; arguements.&lt;/p&gt;&#xA;" OwnerUserId="42688" LastEditorUserId="29575" LastEditDate="2017-12-01T17:16:18.127" LastActivityDate="2017-12-01T17:16:18.127" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25298" PostTypeId="1" AcceptedAnswerId="25344" CreationDate="2017-12-01T15:39:36.230" Score="0" ViewCount="166" Body="&lt;p&gt;Facing this dataset: &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/wine+quality&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://archive.ics.uci.edu/ml/datasets/wine+quality&lt;/a&gt; I treated it both as a classification task and as a regression task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the fact that classifying the quality of a wine as 8 while it is a 3 is a huge difference while classifying a 5 as 6 is not that bad&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has two subquestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there an easy way to change the weight of the classification in a (scikit-learn) classifier?&lt;/li&gt;&#xA;&lt;li&gt;Following the regression approach and having the output being rounded to the nearest valid wine quality value (converting the outputs to a classifier-like outputs) hides any pitfalls?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="23305" LastEditDate="2017-12-01T20:00:31.923" LastActivityDate="2017-12-11T15:47:49.763" Title="How to know when to treat a problem as a classification task or a regression task?" Tags="|classification|regression|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25300" PostTypeId="1" CreationDate="2017-12-01T17:56:01.750" Score="3" ViewCount="2623" Body="&lt;p&gt;Eigengap heuristic Method suggest number of clusters k is usually given by the value of k that maximizes the eigengap (difference between consecutive eigenvalues). &#xA;I plotted the Eigenvalue distribution for my Data:&#xA;[&lt;img src=&quot;https://i.stack.imgur.com/lideD.jpg&quot; alt=&quot;Ranked Eigenvalue]&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eigengap heuristic would suggest cluster number: 1. &#xA;How should I choose number of clusters in this case ? &lt;/p&gt;&#xA;" OwnerUserId="42694" LastActivityDate="2020-10-15T12:50:04.623" Title="How do I choose number of clusters when Eigengap heuristic suggest 1 for spectral clustering?" Tags="|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25301" PostTypeId="2" ParentId="25298" CreationDate="2017-12-01T18:09:25.667" Score="3" Body="&lt;p&gt;In effect it is ordinal regression/classification. I suggest you to go for Mean Absolute Error to take into account the missclassification that play a bigger role when far from the ground truth :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$MAE = \frac{\sum_{i=1}^n|h(x_i) - y_i|}{n}$, creating your own scoring function in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the number of different outputs you have, I recommend you to go for regression. Anyway, rounding any hypothesis output would only play on the score, it's up to you to decide if it makes more sense to see wine quality scores as classes or not.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-12-01T18:09:25.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25302" PostTypeId="2" ParentId="25235" CreationDate="2017-12-01T19:08:51.670" Score="2" Body="&lt;p&gt;I found a related post here that suggests that a larger number of iterations are needed for convergence with &lt;code&gt;sklearn.SGDClassifier()&lt;/code&gt;. After 3000 passes with &lt;code&gt;sklearn.SGDClassifier()&lt;/code&gt; I was able to achieve around the same accuracy as &lt;code&gt;sklearn.LogisticRegression()&lt;/code&gt;. I still find it strange that &lt;code&gt;SGDLearn.fit()&lt;/code&gt; and &lt;code&gt;LogisticRegression.fit()&lt;/code&gt; are not equivalent when training on the exact same samples and with the same arguments, but they must fundamentally train differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kjbNK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kjbNK.png&quot; alt=&quot;sgd-vs-logreg&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42515" LastActivityDate="2017-12-01T19:08:51.670" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25303" PostTypeId="2" ParentId="9344" CreationDate="2017-12-01T19:10:32.653" Score="2" Body="&lt;p&gt;The way R displays the string (using print command) is different from the actual value of this string. Even though you see 2 backslashes for your output, the actual string contains a single backslash. This is because when you type the name of your variable it shows the value within the quotes and within quotes it should use 2 backslashes. When you display the value of the sting using message() or other similar function, when it just prints the value without quotes, then you will see a single back slash only:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#create a vector of names (words)&#xA;words &amp;lt;- c(&quot;word1&quot;,&quot;word2&quot;,&quot;word3&quot;)&#xA;&#xA;#add prefix and suffix part to each word&#xA;words_vec &amp;lt;- paste0(&quot;\\'&quot;,words,&quot;\\'&quot;)&#xA;words_vec&#xA;&quot;\\'word1\\'&quot; &quot;\\'word2\\'&quot; &quot;\\'word3\\'&quot;&#xA;&#xA;#Combine the vector into the string&#xA;line &amp;lt;-paste(words_vec,collapse='')&#xA;line&#xA;&quot;\\'word1\\'\\'word2\\'\\'word3\\'&quot;&#xA;&#xA;#Check how the string looks:&#xA;message(line)&#xA;\'word1\'\'word2\'\'word3\'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By the way, the above (vectorized) method is more efficient than the for-loop method you are using to put all names together.&lt;/p&gt;&#xA;" OwnerUserId="42696" LastActivityDate="2017-12-01T19:10:32.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25305" PostTypeId="2" ParentId="25221" CreationDate="2017-12-01T22:03:16.567" Score="1" Body="&lt;p&gt;Both approaches are 'lossy'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For PCA, assuming you retain fewer components than variables, you necessarily throw away information. If you don't have a few components that capture most of the variance, you could throw a lot away. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For feature selection, (I assume you mean an automated approach like lasso), you again throw away information, hopefully only incremental so. But, depending on how you structure the selection routine, you let an algorithm make some level of design decisions which result in less data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether this is bad or not depends on intent. Fire pure classification it's less an issue. For interpretability it can be essential. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-12-01T22:03:16.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25306" PostTypeId="2" ParentId="25224" CreationDate="2017-12-01T22:23:06.160" Score="0" Body="&lt;p&gt;With Doc2Vec, each string will be treated as a separate document regardless of any formatting such as sentences or paragraphs. So for an analysis of a book from project Gutenberg, you could have each chapter of the book treated as a document or you could treat each paragraph as a document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for a two sentence paragraph like your example  ['I really love football. Peyton Manning was a great player'] and use a sentence tokenizer such as the NLTK package in Python's nltk.tokenize.sent_tokenizer() function which will divide your string into separate sentences.  You would then run the tokenized sentences through Doc2Vec and have each sentence as a different document.  See &lt;a href=&quot;http://textminingonline.com/dive-into-nltk-part-ii-sentence-tokenize-and-word-tokenize&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt; for an example with a sentence tokenizer in Python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of which level of text to treat as a document should be based on what your research design is and what unit of analysis you want to interpret for the end product.  You tagged the question with gensim, so I assume you are trying to do a topic model. You have to weigh whether you are interested on change in topics from sentence to sentence, in paragraph chunks, or some other hierarchy within the text.&lt;/p&gt;&#xA;" OwnerUserId="42657" LastActivityDate="2017-12-01T22:23:06.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25307" PostTypeId="1" AcceptedAnswerId="25404" CreationDate="2017-12-01T23:23:17.920" Score="0" ViewCount="594" Body="&lt;p&gt;I'm trying to understand the &lt;a href=&quot;https://www.rdocumentation.org/packages/quanteda/versions/0.99.22/topics/textstat_collocations&quot; rel=&quot;nofollow noreferrer&quot;&gt;lambda collocation scoring metric&lt;/a&gt; from the quanteda package in R, but the explanation in the documentation is incredibly difficult to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone explain it better?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having an example worked out on a toy set of text would help too, like something with 5-10 words.  This line (and the one right before it) is especially confusing:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18481" LastEditorUserId="29575" LastEditDate="2017-12-02T00:04:02.580" LastActivityDate="2017-12-05T10:59:21.117" Title="Can someone explain the lambda collocation metric?" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25308" PostTypeId="2" ParentId="21638" CreationDate="2017-12-01T23:56:46.443" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features-&gt;decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I do not understand what you wanted to say with this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, the most straightforward way to consider the &lt;em&gt;certainty&lt;/em&gt; is to use it as a &lt;em&gt;weight&lt;/em&gt; (or &lt;em&gt;exposure&lt;/em&gt;) and use the methods that support it, e.g. GLMs or GBMs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the concept of &lt;em&gt;weights&lt;/em&gt; is explained in &lt;a href=&quot;http://bc.bojanorama.pl/2015/09/linear-models-with-weighted-observations/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2018-03-07T14:40:30.530" LastActivityDate="2018-03-07T14:40:30.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25309" PostTypeId="1" CreationDate="2017-12-02T00:47:30.953" Score="-2" ViewCount="712" Body="&lt;h2&gt;Introduction&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a large CSV file with about &lt;code&gt;400 rows&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each row contains &lt;code&gt;1000 x values&lt;/code&gt; and &lt;code&gt;400 y values&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, my current model is taking in a row of 1000 x values and predicting a row of 400 y values:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(input) &lt;code&gt;1000 x values&lt;/code&gt; ----&gt; (neural network) ----&gt; &lt;code&gt;400 y values&lt;/code&gt; (output prediction)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenge is the &lt;code&gt;400 y values&lt;/code&gt; are not one hot vectors.E.g, they are not all zero's with 1 y having a value:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0,0,0,0,0,...,1,0,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;400 y values&lt;/code&gt; actually contain multiple discrete values, like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0,13,0,23,5,...,1,85,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Keras model:&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Intialize the artificial neural network&#xA;model = Sequential()&#xA;&#xA;# Input layer and hidden layer &#xA;model.add(Dense(activation=&quot;relu&quot;, input_dim=1000, units=700, kernel_initializer=&quot;uniform&quot;))&#xA;&#xA;# Dropout to aid overfitting&#xA;model.add(Dropout(0.285))&#xA;&#xA;# Output layer&#xA;model.add(Dense(activation=&quot;linear&quot;, input_dim=700, units=400, kernel_initializer=&quot;uniform&quot;))&#xA;&#xA;# Compile the ANN&#xA;model.compile(optimizer=&quot;adagrad&quot;, loss=&quot;mean_squared_error&quot;, metrics=[&quot;accuracy&quot;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I fit my data to the training set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Fit the ANN to the training set&#xA;model = baseline_model()&#xA;result = model.fit(X_train, y_train, batch_size=10, nb_epoch=200, validation_data=(X_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Is it okay to use my model for this machine learning problem? Or is my neural network in Keras incorrect?&lt;/p&gt;&#xA;" OwnerUserId="42705" LastActivityDate="2017-12-02T12:55:54.653" Title="Can I use a regression machine learning model for predicting a vector with multiple values?" Tags="|machine-learning|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25310" PostTypeId="2" ParentId="25309" CreationDate="2017-12-02T01:32:44.127" Score="0" Body="&lt;p&gt;There is a bit too little information on the actual inputs, outputs and use-case to answer this question properly. However, I will try to give you some intuition for thoughts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are essentially doing is reducing the dimensions of the input. This is legit as you seem to be intending to do that: for each row you provide a vector of [1000x1] and want to get [400x1] that represents your original input the best way possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Let's look at your architecture:&lt;/strong&gt;&#xA;In the hidden layer you are using a ReLu activation function, so your outputs from that layer will be sparse: zeroes and ones. Then you apply a linear layer, so your transformation on the output layer is of the format b*v+bias where b is the vector of the hidden layer and v the weight vector between hidden and output layer. This implies that v does not influence the zero elements of b, but solely the bias determines their values after the transformation. It's hard to tell whether that's good or bad. Anyway, you may want to think about trying out continuous functions at the hidden layer, such as a Sigmoid. Furthermore, using a uniform distribution for weight initialization of your network is not quite ideal in most cases. A truncated normal distribution is a common choice of initialization, however, again, it depends on your input and output data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you would like to provide more information on the data for us to help you better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your goal is to get a more low-dimensional representation of your data and not care about the exact target values, you may want to look into &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AutoEncoders&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31582" LastEditorUserId="31582" LastEditDate="2017-12-02T12:55:54.653" LastActivityDate="2017-12-02T12:55:54.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25311" PostTypeId="2" ParentId="24276" CreationDate="2017-12-02T01:54:49.723" Score="0" Body="&lt;p&gt;&lt;strong&gt;Let's sum up what you are doing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You intend to apply a Neural Network on a dataset of X samples with 1,503 features to get the most accurate prediction of price (i.e. continuous variable). You are worrying that your engineered features may only contribute marginally to the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Your concern may be unnecessary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your overall goal is to get the most accurate prediction, I believe dimensionality reduction such as PCA (as mentioned by OP) is not the way to go here. With every feature you create you may explain a fraction of the variance, even if they are highly correlated. Worst case is, that it is useless. If your network is complex enough, i.e. your architecture can ensure that relationships between the variables can be accounted for completely, you will catch everything that's there and your chosen algorithm and setup is capable of. Any variable that's not important will not play a role in your prediction as weights will simply be (close to) zero. However, if you follow this way, ensure that you have a robust method to measure whether you are overfitting or not, such as K-fold cross validation. You may also want to look into regularization techniques suchas L1, L2 and DropOut (neural networks).&lt;/p&gt;&#xA;" OwnerUserId="31582" LastActivityDate="2017-12-02T01:54:49.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25314" PostTypeId="2" ParentId="24446" CreationDate="2017-12-02T03:22:47.913" Score="2" Body="&lt;p&gt;I would train one classifier with 3 classes: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat, dog, neither&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use cross entropy as your cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many examples of this online. If you Google MNIST neural network you'll find many simple examples. Or you can take one of Google's trained networks and retrain the last layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a convolutional being trained on the &lt;a href=&quot;https://github.com/timestocome/DeepLearning/blob/master/Convolutional_tf.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cifar set of images&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using a pre-trained network to &lt;a href=&quot;https://github.com/timestocome/RaspberryPi-Robot/tree/master/catID&quot; rel=&quot;nofollow noreferrer&quot;&gt;tell two house cats apart&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="42712" LastEditorUserId="29575" LastEditDate="2017-12-02T22:13:35.173" LastActivityDate="2017-12-02T22:13:35.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25315" PostTypeId="1" AcceptedAnswerId="25409" CreationDate="2017-12-02T10:03:03.470" Score="8" ViewCount="17487" Body="&lt;p&gt;In the last layer of &lt;code&gt;CNNs&lt;/code&gt; and &lt;code&gt;MLPs&lt;/code&gt; it is common to use softmax layer or units with sigmoid activation functions for multi-class classification. I have seen somewhere, I don't remember where, that softmax is used whenever the classes are mutually exclusive and the layer with units containing sigmoid activation function are used in tasks with multiple labels, e.g. recognizing animals in an image which can contain numerous animals. Am I right? Is there any other deduction for distinguishing between them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen &lt;a href=&quot;https://stackoverflow.com/questions/41409248/softmax-and-sigmoid-function-for-the-output-layer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/a/218589/179078&quot;&gt;here&lt;/a&gt; but they do not contain what I want.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-04T14:18:56.087" LastActivityDate="2019-01-10T16:48:02.953" Title="Why should softmax be used in CNN" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25317" PostTypeId="1" CreationDate="2017-12-02T10:46:23.000" Score="22" ViewCount="10400" Body="&lt;p&gt;What are the pros and cons of &lt;a href=&quot;https://github.com/keras-team/keras&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt; and &lt;a href=&quot;https://github.com/tflearn/tflearn&quot; rel=&quot;noreferrer&quot;&gt;TFlearn&lt;/a&gt;? When is one library preferred over the other?&lt;/p&gt;&#xA;" OwnerUserId="42069" LastEditorDisplayName="user10640" LastEditDate="2018-07-11T07:19:27.720" LastActivityDate="2018-07-12T00:11:11.150" Title="What are the pros and cons of Keras and TFLearn?" Tags="|machine-learning|keras|tflearn|" AnswerCount="1" CommentCount="0" CommunityOwnedDate="2018-07-12T00:12:39.810" ContentLicense="CC BY-SA 4.0" />
  <row Id="25318" PostTypeId="2" ParentId="25317" CreationDate="2017-12-02T11:21:52.190" Score="17" Body="&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://tflearn.org/&quot; rel=&quot;noreferrer&quot;&gt;TFlearn&lt;/a&gt; is a modular and transparent deep learning library built on top of Tensorflow. It was designed to provide a higher-level API to TensorFlow in order to facilitate and speed-up experimentations, while remaining fully transparent and compatible with it&lt;/em&gt;. Even with TensorFlow, however, we face a choice of which “front-end” framework to use. Should we use straight TensorFlow, or TF Learn, or Keras, or the new TF-Slim library that Google released within TensorFlow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;&lt;a href=&quot;https://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt; is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Straight &lt;code&gt;TensorFlow&lt;/code&gt; is really verbose while &lt;code&gt;Keras&lt;/code&gt; and &lt;code&gt;TfLearn&lt;/code&gt; both seem solid, but the &lt;code&gt;TfLearn&lt;/code&gt; syntax seems a little cleaner. One drawback to Tflearn is the lack of easily integrated pre-trained models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually there are so many answers for your question &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/50eokb/which_one_should_i_choose_keras_tensorlayer/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.quora.com/session/Fran%C3%A7ois-Chollet/1&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and I quote some of them here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;TensorFlow is currently the mainstream of deep learning framework, they are all the wrapper of TF. Whereas, Keras was released at the age of Theano, and therefore having a good support from Theano’s users. While TensorLayer and TFLearn are both released after TensorFlow. A good reason to choose Keras is that you could use TensorFlow backend without actually learning it. Plus Keras tends to wrap up the model deeply, so you don't necessarily need to consider the backend to be Theano or TF, which is a big advantage of Keras.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;It depend on what do you want to do, fast prototyping or something else?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Keras : Many people are using it, easy to find examples on github. Suitable for beginner. Capable of running on top of either TensorFlow or Theano. Tflearn : Why no one discuss it? It is also a famous library, transparent over TensorFlow. High running speed. TensorLayer: Just release (Sep 2016), transparent over TensorFlow. High running speed. Easy to extend, suitable for professional, its tutorial include all modularized implementation of Google TensorFlow Deep Learning tutorial. TF-Silm: Just release (Aug 2016) similar with Tflearn, but no RNN layer at the moment (Sep 2016).&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The best deep learning framework is the one you know best.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-02T11:33:46.510" LastActivityDate="2017-12-02T11:33:46.510" CommentCount="0" CommunityOwnedDate="2018-07-12T00:12:39.810" ContentLicense="CC BY-SA 3.0" />
  <row Id="25320" PostTypeId="1" CreationDate="2017-12-02T12:09:14.453" Score="1" ViewCount="53" Body="&lt;p&gt;I am trying to predict the appearance count of a particular item using Neural Network.My data has severe class imbalance i.e 90% of data contains zero as output, remaining values ranges from 1 to 5. I am training a neural network with item features and week number of year.Whenever I run the neural network and predict for the &lt;strong&gt;training data the sum of all predicted counts are equal to the sum of observed counts but the individual counts are not even getting close.I tried with both RMSE and MAE as error metrics&lt;/strong&gt;.I am not sure about this is correct,&#xA; any help would be great &lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-12-02T12:09:14.453" Title="Abnormal behavior while predicting with training data" Tags="|machine-learning|neural-network|predictive-modeling|statistics|supervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25322" PostTypeId="2" ParentId="25259" CreationDate="2017-12-02T17:27:22.510" Score="6" Body="&lt;p&gt;There are lots of questions but I will try to answer in a way that might clear things up for you and also give you some guidance. Please note that the proofs for your questions involve lots of math operations so instead I will provide you with references.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your main reference is the paper from Sutton &lt;a href=&quot;https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf&quot; rel=&quot;noreferrer&quot;&gt;PG Methods with Function Approximation&lt;/a&gt;. I highly recommend you to read the paper a couple of times (or even more!) and do some search in the relevant literature when you will be familiar with the main objectives, notation and math around the general approach of the methods. PG Methods are not easy to get a grasp of them mainly because of their sampling nature, notation and discrete/continuous math involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PG Methods satisfy (or at least should) the PG theorem (eq. 2 from the paper). An interesting approach would be to substitute the true $Q^\pi (s,a)$ by some approximate function ($f_w$ in the paper, $Q_w$ in your question). Now, we are wondering what conditions should be satisfied by that proposed approximation in order to satisfy the PG Theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing you notice is that a natural choice for updating the parameters $w$ is to update them towards the direction that minimizes the mean squared error of the exact $Q^\pi (s,a)$ with the function approximation. In your question this is the $\epsilon$. In such a scenario the exact $Q^\pi (s,a)$ is estimated using unbiased samples such as $r_t$.&#xA;This is explained in detail in Part 2 of the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the PG theorem to hold (proof consists of the 3 lines before Part 3) the grad of your approximate function should satisfy the compatibility condition. To sum up we started from PG theorem and we found a suitable family of function approximators for our action-value function that the PG theorem holds. In Part 3 you can see an example of a compatible function. From this of course you can use even non-linear approximators such as NNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A clarification on on/off-policy: The David Silver's slide that you posted here has to do with theoretical guarantees and has nothing to do with an actual RL algorithm. By the way Q-learning algorithm in which you use the $max_{a'}{Q(s',a')}$ is OFF-policy as you you don't actually use for updates the ongoing policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2017-12-03T02:19:31.143" LastActivityDate="2017-12-03T02:19:31.143" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25323" PostTypeId="1" CreationDate="2017-12-02T17:34:44.223" Score="1" ViewCount="55" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/abs/1606.04474&quot; rel=&quot;nofollow noreferrer&quot;&gt;the paper &quot;Learning to learn by gradient descent by gradient descent&quot;&lt;/a&gt; they describe an RNN which learns gradient transformation to learn an optimizer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimizer network directly interacts with the environment to take actions,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1} = \theta_t + g_t(∇f(\theta_t), \phi).$ (Equation 1 from the paper)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and hence feels like a reinforcement learning problem in continuous action space. The formulation of optimization equation looks like what one would  typical do in a supervised learning problem,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(\phi) = E_f[f(\theta^*(f,\phi))]$ (Equation 2 from the paper)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this an indirect formulation of policy gradient?   &lt;/p&gt;&#xA;" OwnerUserId="42737" LastEditorUserId="42737" LastEditDate="2018-03-06T18:08:26.427" LastActivityDate="2018-03-06T18:08:26.427" Title="Isn't the optimizer network in deepminds learning to learn a DRQN?" Tags="|neural-network|deep-learning|q-learning|meta-learning|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="25324" PostTypeId="1" CreationDate="2017-12-02T18:52:42.470" Score="0" ViewCount="562" Body="&lt;p&gt;I want to do the following project and I think the best way is using tensorflow/keras: I have photos as images from a list with prices of products and I need classify these text objects from the images such that i identify the product name and the price. &#xA;So I think it is different from normal Image classification since my objects are text objects? &#xA;First, I thought it is like object detection in images but I think i need more to take these texts to another neural network?&lt;/p&gt;&#xA;" OwnerUserId="32963" LastActivityDate="2017-12-02T19:06:18.077" Title="Text Mining from Images" Tags="|machine-learning|tensorflow|text-mining|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25325" PostTypeId="2" ParentId="25324" CreationDate="2017-12-02T19:06:18.077" Score="1" Body="&lt;p&gt;I think you are referring to a king of image auto captioning, and why not sentence alignement ! I suggest you to go over the paper &lt;em&gt;Deep Visual-Semantic Alignments for Generating Image Descriptions&lt;/em&gt; which is one of the references today (by M. Karpathy).&#xA;The first step is about recognizing objects in an image and align sentences to them (done with R-CNN and BRNN) and the second is about generating new captions (CNN + RNN)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though it may be quite difficult to deeply go through, there is some code available on GitHub and it has been remade by a a lot of people with various implementations. &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/deepimagesent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the summary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck !&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-12-02T19:06:18.077" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25330" PostTypeId="1" AcceptedAnswerId="25333" CreationDate="2017-12-03T06:39:49.883" Score="2" ViewCount="166" Body="&lt;p&gt;I'm trying to restore this function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ F(x) = x*sin(\alpha x)+b; \space\space \alpha,b \in (-20,20) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My NN model(with Keras) is:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 layer: GRU, 9 neurons, selu activation&lt;br&gt;&#xA;2 layer: GRU, 3 neurons, selu activation&lt;br&gt;&#xA;1 layer: GRU, 7 neurons, selu activation&lt;br&gt;&#xA;1 layer: Dense, 1 neuron, linear activation&lt;br&gt;&#xA;&lt;em&gt;kernel init he_normal for all layers&lt;/em&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the training dataset I've generated function values $f(x), x \in -5..5$ (e.g. &lt;code&gt;linspace(-5,5,500)&lt;/code&gt; and random &lt;strong&gt;integer&lt;/strong&gt; values for $\alpha$ and $b$, 250 times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I've selected $f[i-1]$ and $f[i]$ (previous steps) and for this &quot;xs&quot; the output is $f[i+1]$. So the training dataset is like:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;X&lt;/strong&gt;:&lt;br&gt;&#xA;   first row: previous previous $f$, $\alpha$, $b$&lt;br&gt;&#xA;   second row: previous $f$, $\alpha$, $b$  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Y&lt;/strong&gt;: &#xA;   current $f$ value  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After training over 200 epochs, mae on validation data was 0.1851.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if I try to predict new data it seems normal, but when I try to predict new values using previously predicted by model points it falls down and it doesn't looks like sine function at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="42730" LastEditorUserId="29575" LastEditDate="2017-12-11T17:00:49.777" LastActivityDate="2017-12-11T17:00:49.777" Title="Regression and Neural networks" Tags="|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25332" PostTypeId="1" AcceptedAnswerId="25334" CreationDate="2017-12-03T08:06:27.603" Score="1" ViewCount="34" Body="&lt;p&gt;Would you say, you could design,  tune and/or train any DNN for any application, or do their designs inherently postulate some specialization?&#xA;Is there such a review?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, are CNNs better for the spatial domain (e.g image analysis), and RNN for temporal (e.g. time series data, or speech), or it does not matter?&lt;/p&gt;&#xA;" OwnerUserId="33183" LastActivityDate="2017-12-03T09:02:01.290" Title="Classes of neural nets and their applications" Tags="|reference-request|methodology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25333" PostTypeId="2" ParentId="25330" CreationDate="2017-12-03T08:19:22.263" Score="2" Body="&lt;p&gt;The problem is that your error is accumulating and diverging. In other words, a small error in the first prediction is leading to a larger error in your second prediction, which is leading to an even larger error in your third prediction, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that said, &lt;a href=&quot;https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction&quot; rel=&quot;nofollow noreferrer&quot;&gt;this LSTM example&lt;/a&gt; seems to do well predicting multiple future values for a sine wave.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2017-12-03T08:19:22.263" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25334" PostTypeId="2" ParentId="25332" CreationDate="2017-12-03T09:02:01.290" Score="2" Body="&lt;p&gt;Convolutional Neural Networks have consistently outperformed other methods for image recognition and related tasks. In fact, beyond a certain image size it is not practical to train a fully-connected network on raw pixels due to the enormous number of parameters that would be required. The number of parameters in a &lt;code&gt;CNN&lt;/code&gt; however grows much more slowly as you increase the input image size, since they usually do not require larger filters, and the number of parameters per filter is independent of input size.&lt;/p&gt;&#xA;&lt;p&gt;With that said, there is a much more remarkable demonstration of the power of the architecture of a &lt;code&gt;CNN&lt;/code&gt; for image processing in the recent paper &lt;a href=&quot;https://dmitryulyanov.github.io/deep_image_prior&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Image Prior&lt;/a&gt;. The abstract reads:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a &lt;strong&gt;randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting&lt;/strong&gt;. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.&lt;/p&gt;&#xA;&lt;p&gt;Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So they were able to demonstrate that the raw network &lt;em&gt;prior to any training&lt;/em&gt; was powerful enough for sophisticated tasks, which is the strongest and most direct illustration of the power of the specific architecture itself that I have seen.&lt;/p&gt;&#xA;&lt;p&gt;The answer is not so clear for time series prediction, since &lt;code&gt;CNNs&lt;/code&gt; with various tricks have performed on par with &lt;code&gt;RNNs&lt;/code&gt; for some datasets, but in general &lt;code&gt;RNNs&lt;/code&gt; seem much more suited to the problem, since they can handle inputs of arbitrary length, and they can store state.&lt;/p&gt;&#xA;&lt;p&gt;However, either method will significantly outperform previous methods such as passing in a sliding window of fixed size into a fully-connected network. Even when a fully-connected network is combined with some state-preserving model such as &lt;code&gt;ARIMA&lt;/code&gt; it still cannot match the performance of &lt;code&gt;CNNs&lt;/code&gt; or &lt;code&gt;RNNs&lt;/code&gt;. See &lt;a href=&quot;https://arxiv.org/abs/1701.01887&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning for Time-Series Analysis&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-03T09:02:01.290" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25336" PostTypeId="1" CreationDate="2017-12-03T10:17:17.820" Score="1" ViewCount="168" Body="&lt;p&gt;Planning a classifier to recognize specific documents based on text. Categories are mutually exclusive and include: job application resumes, paychecks, quarterly financial reports etc, and data is text extracted from doc, pdf, xls etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;At the feature engineering stage, with questions in mind:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One Multiclass classifier &lt;strong&gt;or&lt;/strong&gt; a number of One vs. Rest (one per category)?&lt;/li&gt;&#xA;&lt;li&gt;Word embedding (Word2Vec/GloVe) - one embedding for all categories &lt;strong&gt;or&lt;/strong&gt; a separate embedding for each category?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My tendency is towards a number of one vs. rest classifiers (to allow for dynamic use of just some of the categories), and one embedding for all categories (to save prediction time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Project architecture&lt;/strong&gt;: initially basic LogReg, but will soon develop to CNN / CNN+RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Project scale&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training: a few hundreds per category&lt;/li&gt;&#xA;&lt;li&gt;Prediction: millions of files&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please feel free to share any insights or references, thank you&lt;/p&gt;&#xA;" OwnerUserId="38377" LastActivityDate="2018-02-23T02:54:30.777" Title="Document classification - optimal classifier &amp; embedding" Tags="|machine-learning|classification|nlp|word2vec|word-embeddings|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25337" PostTypeId="2" ParentId="24925" CreationDate="2017-12-03T10:23:27.963" Score="0" Body="&lt;p&gt;Why would you put a classifier in this process? It can only increase the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just compute the centers of each class in your training data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-03T10:23:27.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25342" PostTypeId="2" ParentId="22037" CreationDate="2017-12-03T13:35:32.860" Score="0" Body="&lt;p&gt;I have an intuitive notion of why smooth functions are faster to optimize but no mathematical proof or anything. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient descent computes the derivative of the activation function to determine the change in weights. When the activation function has a hard cut (e.g. at z=0 for ReLu) the activation of the unit can changes radically (i.e. always zero or linear) for specific data points when changing the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other weights need to adept to this radically different behaviour of a specific unit for specific data points. If, however, the behaviour of the unit change radically again during the next epoch the network keeps on adepting towards changes in the previous epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a smooth function there are no such radical changes. And thus the network can more gradually stabilise. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2017-12-03T13:35:32.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25343" PostTypeId="2" ParentId="21984" CreationDate="2017-12-03T14:31:08.107" Score="4" Body="&lt;p&gt;I have had a couple of talks about this once and I think you should know about three different concepts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Different attributes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Attributes can be divided into three different categories. &lt;em&gt;Hard identifiers&lt;/em&gt; refer to a specific person, e.g. full name or passport number. A combination of &lt;em&gt;soft identifiers&lt;/em&gt; can also be used to identify a single person, because the combination is unique. &lt;em&gt;Sensitive&lt;/em&gt; attributes are the ones that infringe privacy and that should be protected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Measuring privacy&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several measures for how anonymous your dataset is. These measures help you to focus on the attributes that need to be altered. In order of increasingly stronger requirements you have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A dataset is &lt;a href=&quot;https://en.wikipedia.org/wiki/K-anonymity&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;$k$-anonymous&lt;/em&gt;&lt;/a&gt; if every combination of soft identifiers occurs at least $k$ times. &lt;/li&gt;&#xA;&lt;li&gt;A dataset is &lt;a href=&quot;https://en.wikipedia.org/wiki/L-diversity&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;$i$-diverse&lt;/em&gt;&lt;/a&gt; if for every combination of soft identifiers there are at least $i$ different values of the sensitive attributes. &lt;/li&gt;&#xA;&lt;li&gt;A dataset is &lt;a href=&quot;https://en.wikipedia.org/wiki/T-closeness&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;$t$-closed&lt;/em&gt;&lt;/a&gt; if for every combination of soft identifiers, the distribution of sensitive attributes is not more than $t$ (i.e. KL-divergence) different from the global distribution of sensitive attributes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Privacy protecting modifications&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hard identifiers can be treated with masking or pseudomization. &lt;em&gt;Masking&lt;/em&gt; refers to deleting the hard identifier. A less drastic approach is  &lt;em&gt;pseudomization&lt;/em&gt; where each records hard identifiers gets a pseudonym (e.g. hash) that is only known by the dataset creator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Soft identifiers can be treated with suppression, generalisation or randomisation. &lt;em&gt;Suppressing&lt;/em&gt; values of soft-identifiers refers to replacing it by a default value (e.g. '*' or the mode). Another strategy, called &lt;em&gt;generalisation&lt;/em&gt;, is to create groups of variables that are alike (e.g. 'age 10-20' instead of age 19). A third way is to add randomness to quantitative variables by adding zero-mean noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sensitive attributes can be treated with permutation or randomisation. &lt;em&gt;Permutation&lt;/em&gt; refers to permuting the sensitive attributes within every combination of soft identifiers (within a dataset that is at least $2$-anonymous). Like soft identifiers, quantitative sensitive attributes can also be &lt;em&gt;randomised&lt;/em&gt; by adding noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ps. as an R-package you could consider &lt;code&gt;sdcMicro&lt;/code&gt;, and this &lt;a href=&quot;http://www2.uaem.mx/r-mirror/web/packages/sdcMicro/vignettes/guidelines.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;guideline&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="20372" LastEditorUserId="20372" LastEditDate="2017-12-03T14:36:20.933" LastActivityDate="2017-12-03T14:36:20.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25344" PostTypeId="2" ParentId="25298" CreationDate="2017-12-03T15:02:09.323" Score="1" Body="&lt;p&gt;This is ordinal classification. A quick approach to handle this with any classifier that outputs probabilities is explained thoroughly &lt;a href=&quot;https://pdfs.semanticscholar.org/df47/a60a50f4ddde9a4fe3b560091ba925cfd495.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In a few words: if you have, for example, three ordinal classes &lt;span class=&quot;math-container&quot;&gt;$Cool &amp;lt; Mild &amp;lt; Hot$&lt;/span&gt;, then you create binary classifiers.&lt;/p&gt;&#xA;&lt;p&gt;For the first and last class you calculate the probabilities immediately like so:&lt;/p&gt;&#xA;&lt;h3&gt;Is temperature larger than &lt;span class=&quot;math-container&quot;&gt;$cool$&lt;/span&gt;?&lt;/h3&gt;&#xA;&lt;p&gt;This is given by &lt;span class=&quot;math-container&quot;&gt;$P(temp &amp;gt; cool)$&lt;/span&gt;, where, &lt;span class=&quot;math-container&quot;&gt;$P(temp = cool) = 1 - P(temp &amp;gt; cool)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h3&gt;Is temperature larger than &lt;span class=&quot;math-container&quot;&gt;$mild$&lt;/span&gt;?&lt;/h3&gt;&#xA;&lt;p&gt;You calculate (neither mild or cool) as: &lt;span class=&quot;math-container&quot;&gt;$P(temp &amp;gt; mild)$&lt;/span&gt;, therefore &lt;span class=&quot;math-container&quot;&gt;$P(temp = hot) = P(temp &amp;gt; mild)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h3&gt;Is the temperature &lt;span class=&quot;math-container&quot;&gt;$mild$&lt;/span&gt;?&lt;/h3&gt;&#xA;&lt;p&gt;All the intermediate classifications (here we have only one) result from subtractions. We have all the cases larger than &lt;span class=&quot;math-container&quot;&gt;$cool$&lt;/span&gt; (here &lt;span class=&quot;math-container&quot;&gt;$mild$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$hot$&lt;/span&gt;)&#xA;but we also remove the probability of being larger than &lt;span class=&quot;math-container&quot;&gt;$mild$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This leaves us with the probability of being just &lt;span class=&quot;math-container&quot;&gt;$mild$&lt;/span&gt;:&lt;br /&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$P(temp = mild) = P(temp &amp;gt; cool) - P(temp &amp;gt; mild)$$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-11T15:47:49.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25346" PostTypeId="1" CreationDate="2017-12-03T15:40:28.990" Score="0" ViewCount="912" Body="&lt;p&gt;So I need to code an SVM from bottom up in Python, and I cannot use stuff like libSVM or scikit-learn, for reasons of my own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Referring to Andrew Ng's &lt;a href=&quot;http://cs229.stanford.edu/notes/cs229-notes3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;excellent notes on Support Vector Machines&lt;/a&gt;, I need to specifically implement the second equation on page 20, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\max_\alpha W(\alpha) = \sum_{i=1}^m\alpha_i - \frac 1 2 \sum_{i,j=1}^m(y^{(i)} \times y^{(j)} \times \alpha_i \times \alpha_j \times \langle x_i, x_j\rangle)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Subject to the given constraints, $\sum_{i=1}^m\alpha_i y_i = 0$, and $0 \le \alpha_i \le C$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does someone have the code for this in any language, or can explain the intuition behind coding this? Most SVM implementations I could find either stick to scikit-learn or neglect kernel functions altogether. While I am working in Python, I would try to understand code in any language.&lt;/p&gt;&#xA;" OwnerUserId="39637" LastEditorUserId="38892" LastEditDate="2017-12-04T19:26:15.177" LastActivityDate="2017-12-04T19:26:15.177" Title="How to code an SVM's equation including kernels?" Tags="|python|svm|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25348" PostTypeId="1" CreationDate="2017-12-03T18:36:23.733" Score="9" ViewCount="8594" Body="&lt;p&gt;What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more elegant solutions?&lt;/p&gt;&#xA;" OwnerUserId="26202" LastEditorUserId="26202" LastEditDate="2022-02-14T23:12:09.983" LastActivityDate="2022-02-14T23:12:09.983" Title="How to train a xgboost model on data that is too big for the memory?" Tags="|python|xgboost|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25350" PostTypeId="1" CreationDate="2017-12-03T19:05:43.657" Score="4" ViewCount="586" Body="&lt;p&gt;From the second paragraph of 3.1 &lt;em&gt;Factorization into smaller convolution&lt;/em&gt; in the paper &lt;a href=&quot;https://arxiv.org/abs/1512.00567&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rethinking the inception architecture for computer vision&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This setup clearly reduces the parameter count by shar- ing the weights between adjacent tiles. To analyze the expected computational cost savings,&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;This way, we end up with a net 9+9 × reduction of computation, resulting&#xA;  25 in a relative gain of 28% by this factorization&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Apprantly this design decreases the number of parameters. But I can't understand why it decreases the computational cost? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the case of using 2 3*3 convnets to replace 5*5, I think it increases (3*3*9+3*3)/5*5 = 3.6 times of computational cost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do I miss here?&lt;/p&gt;&#xA;" OwnerUserId="12731" LastEditorUserId="29169" LastEditDate="2021-02-13T19:20:33.313" LastActivityDate="2021-02-13T19:20:33.313" Title="How does inception decrease the computational cost?" Tags="|convolutional-neural-network|inception|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25351" PostTypeId="1" CreationDate="2017-12-03T21:45:15.070" Score="1" ViewCount="45" Body="&lt;p&gt;I think of implementing training and predicting in an app (both Android and iOS), but existing packages I found not seem to be very mobile-friendly (scikit-learn, xgboost, lightgbm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random forest will also work fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you guys could advise an easier way so it does not require a lot of porting? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would appreciate any help, Thanks!&lt;/p&gt;&#xA;" OwnerUserId="35461" LastActivityDate="2017-12-03T21:45:15.070" Title="What gradient boosting library is the easiest to implement on mobile?" Tags="|random-forest|xgboost|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25352" PostTypeId="1" AcceptedAnswerId="25368" CreationDate="2017-12-03T22:03:48.730" Score="0" ViewCount="380" Body="&lt;p&gt;I am looking at data from the London Data Store based on social characteristics between London boroughs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since there are only about 30 London boroughs, the data sets I am looking at are naturally very small. For example, I might be fitting regression/correlations to a plot of about 30 points. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are appropriate ways to conduct classification on such small data sets, and why? 'Why' is important.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I was thinking of something like SVM, or Naive Bayes. Or regression if the data is continuous.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;What are very inappropriate ways to conduct classification here?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40736" LastEditorUserId="40736" LastEditDate="2017-12-03T22:11:28.680" LastActivityDate="2018-02-02T10:17:02.500" Title="How should classification be done for a very small data set?" Tags="|classification|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25353" PostTypeId="1" CreationDate="2017-12-03T23:42:06.057" Score="1" ViewCount="1027" Body="&lt;p&gt;I'm using a random forest classifier (scikit learn). When I do a 5-fold cross validation, the average accuracy score is around 70%. However, when I look at precision_score and recall_score, they are both 1. Is that possible? If it's wrong, what could be the problem? Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = RandomForestClassifier(n_estimators=100) &#xA;clf.fit(X, Y.values.ravel())&#xA;scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring=&quot;accuracy&quot;)&#xA;Y_pred = clf.predict(X)&#xA;precision = precision_score(Y, Y_pred)&#xA;recall = recall_score(Y, Y_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17916" LastEditorUserId="381" LastEditDate="2017-12-04T04:22:48.843" LastActivityDate="2019-12-29T11:02:15.280" Title="high precision and recall but low cross validated accuracy" Tags="|scikit-learn|random-forest|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25354" PostTypeId="2" ParentId="25346" CreationDate="2017-12-04T01:03:17.077" Score="2" Body="&lt;p&gt;The SVM problem is a quadratic programming problem. It depends on whether you are willing to call a quadratic programming packages to solve your problem, if you are willing to do so, you just have to reformat the question in the input format of a programming language. &lt;a href=&quot;https://scaron.info/blog/quadratic-programming-in-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a resource for how to call quadratic programing solver in Python and &lt;a href=&quot;https://www.mathworks.com/help/optim/ug/quadprog.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is how to do it in matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sum_{i=1}^m \alpha_i-\frac12\sum_{i,j=1}^my^{(i)}y^{(j)}\alpha_i\alpha_jK( x_i, x_j)=e^T\alpha-\frac12\alpha^TA\alpha$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $A_{ij}=y^{(i)}y^{(j)}K(x_i,x_j)$ , $K$ is the kernel function, and $e$ is the all one vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose you are not willing to call a quadratic programming solver. If you read on the document that you attached, you will reach a section called sequential minimal optimization. &lt;a href=&quot;https://en.wikipedia.org/wiki/Sequential_minimal_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the wikipedia page of the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea of the algorithm is that we can't update a single $\alpha_i$ at a time due to the linearity constraint, hence we do the update in pairs of $\alpha_i$ and $\alpha_j$. You might want to refer to Platt's &lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; for details including heuristic on how to choose the pair for update.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2017-12-04T01:03:17.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25355" PostTypeId="1" AcceptedAnswerId="25356" CreationDate="2017-12-04T01:10:37.220" Score="1" ViewCount="79" Body="&lt;p&gt;I have a database like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Site    X    Y&#xA;S1      1    1.5&#xA;S1      1    1.3&#xA;S1      2    1.7&#xA;S1      1    1.1&#xA;S1      4    5.9&#xA;S2      3    4.0&#xA;S2      2    2.5&#xA;S2      4    9.1&#xA;S2      4    9.2&#xA;S2      1    2.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to find if $Y$ increases as $X$ increases for every site. In other words, Bigger $X$ corresponds to bigger $Y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know linear regression might suit this problem. But please take a look at the following graph:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GM9XW.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GM9XW.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure 1 is not what I want because small $X$ corresponds to big $Y$. However, Figure 2 is what I want. When I use a linear regression model and RMSE as a measure, it cannot tell the difference between Figure 1 and Figure 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing about my database is: $X$ are like levels, which are the same to all the sites. But the $Y\text{'s}$ of every sites are different. For example, for one site, $X=1$ and $Y=20$ means $20$ is a small value because it corresponds the lowest level of $X$. But for another site, $Y=15$ and $X=6$ means $15$ is the highest value because $X$ is the highest level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my problem is: for every site, I need to use a linear model or any other algorithms to judge if $Y$ increases as $X$ does. Then, I need to use a measure to select some sites.&lt;/p&gt;&#xA;" OwnerUserId="16911" LastEditorUserId="29575" LastEditDate="2017-12-10T10:21:12.593" LastActivityDate="2017-12-10T10:21:12.593" Title="Can linear regression or any other models be used to judge if Y increases as X increases?" Tags="|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25356" PostTypeId="2" ParentId="25355" CreationDate="2017-12-04T03:03:20.160" Score="1" Body="&lt;p&gt;Linear Regression will help you decide whether &lt;code&gt;Y&lt;/code&gt; &lt;em&gt;tends&lt;/em&gt; to increase with &lt;code&gt;X&lt;/code&gt;, but it is not a good tool to &lt;em&gt;prove&lt;/em&gt; that &lt;code&gt;Y&lt;/code&gt; &lt;em&gt;always&lt;/em&gt; increases with &lt;code&gt;X&lt;/code&gt;. For this you need to design an algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To prove that &lt;code&gt;Y&lt;/code&gt; always increases with &lt;code&gt;X&lt;/code&gt; at each site, first create a table of unique &lt;code&gt;X&lt;/code&gt; in ascending order with the corresponding min and max &lt;code&gt;Y&lt;/code&gt; for each site:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S1:&#xA;X    min(Y)    max(Y)&#xA;1    1.1       1.5&#xA;2    1.7       1.7    &#xA;4    5.9       5.9&#xA;&#xA;S2:&#xA;X    min(Y)    max(Y)&#xA;1    2.3       2.3&#xA;2    2.5       2.5    &#xA;3    4.0       4.0&#xA;4    9.1       9.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now for each site verify the following: For each &lt;code&gt;X&lt;/code&gt; check that max(&lt;code&gt;Y&lt;/code&gt;) is less than min(&lt;code&gt;Y&lt;/code&gt;) for &lt;code&gt;X+1&lt;/code&gt;. If this condition ever fails then you have shown that &lt;code&gt;Y&lt;/code&gt; does not always increase with &lt;code&gt;X&lt;/code&gt; at every site, otherwise you can say that it does.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-04T22:02:14.240" LastActivityDate="2017-12-04T22:02:14.240" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25358" PostTypeId="2" ParentId="25353" CreationDate="2017-12-04T03:44:45.077" Score="1" Body="&lt;p&gt;First of all, you should use &lt;code&gt;cross_val_predict&lt;/code&gt; to get you predictions vector, so that you followed approximately the same validation scheme to get them :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y_pred = cross_val_predict(clf, Y, cv=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, it is totally possible to have 1 for recall or precision given that these are the scores for the considered positive class for scikit. In fact scikit assigns by default one of your class as the 'positive' one and to metrics inherent to classes are computed according to it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as you said you can't have 1 for both and 70% accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact is that your clf.predict gave you 100% accuracy while during cross validation it was not the case because in your second scheme:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;clf.fit(X, Y.values.ravel())&lt;/code&gt; and &lt;code&gt;Y_pred = clf.predict(X)&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;you train and predict on the same data ! So you obtain 100% accuracy which is common, while during cross validation :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring=&quot;accuracy&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you trained and tested on different folds which logically will lead to worse results (but closer to real condition experiment).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, try to use &lt;code&gt;cross_val_predict&lt;/code&gt; and then &lt;code&gt;precision_recall_fscore_support(Y, Y_pred)&lt;/code&gt; in order to have a detailed report of metrics for each class.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastEditorUserId="41728" LastEditDate="2017-12-04T05:34:48.683" LastActivityDate="2017-12-04T05:34:48.683" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25359" PostTypeId="1" CreationDate="2017-12-04T06:30:36.457" Score="2" ViewCount="67" Body="&lt;p&gt;I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the variables are made up of discrete categories and so these end up being encoded into many dimensions of binary variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a subset of the data where observations are made in two separate locations instead of just one. As a result, many of the dimensions are repeated for this subset. Most of the repeated data results in extra dimensions of binary class variables which would just be zero for the compliment set (i.e. only one location) of observations. However, one dimension is a salary where there are real values for the 2-location subset but Nans for the remaining data. To illustrate:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Obs | pay_1 | loc_1_A | loc_1_B| ... | job_1_A | job_1_B | ... | pay_2 | loc_2_A | ...&#xA;======================================================================================&#xA;1   | 50k   |   1     |   0    |     |   1     |   0     |     |   40k |   1     |&#xA;2   | 40k   |   0     |   1    |     |   1     |   0     |     |   45k |   1     |&#xA;3   | 70k   |   1     |   0    |     |   0     |   1     |     |  NaN  |   0     |&#xA;4   | 50k   |   1     |   0    |     |   1     !   0     |     |  NaN  |   0     |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;How to deal with data that has this systematically disjoint set of dimensions in which it lives?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe for many of the variables in the higher-dimensional subset it is no problem, since they will simply equal zero for any of the data points not in this subset. But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data ($pay_2$) to have values similar to a standard gaussian. Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a couple of approaches, one being to have a kind of non-linear &quot;switch&quot; variable that turns the higher dimensions on and off:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = A * pay_1 + B * \text{loc_1_A} + ... + S * ( M * pay_2 + N * \text{loc_2_A} + ... )$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;though I don't see how I can implement this in a lot of standard methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option could be to just copy the $pay_1$ values into the $pay_2$ column for those rows where there is no existing $pay_2$ value. This would seem to me like it is imposing a lot of structure onto the data that is difficult to justify. Maybe if I add another binary &quot;switch&quot; variable on top of this, it can separate the classes in my data space?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this is probably a very problem/domain-specific issue but I would like to know if there are any standard treatments for data which has this kind of structure, as I'm sure it is a pretty common scenario. Any suggestions of ways to think about this or similar problems in other fields to look at?&lt;/p&gt;&#xA;" OwnerUserId="26853" LastEditorUserId="29575" LastEditDate="2017-12-10T20:51:45.940" LastActivityDate="2017-12-11T22:01:10.897" Title="Dealing with a dataset where a subset of points live in a higher dimensional space" Tags="|classification|data-cleaning|feature-engineering|preprocessing|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25360" PostTypeId="1" AcceptedAnswerId="25364" CreationDate="2017-12-04T07:00:30.733" Score="10" ViewCount="8352" Body="&lt;p&gt;I need to create this type of chart, where you can see a quantity is split into a hierarchy of classes or taxonomy. In the example below the quantity is a household monthly income, and the classes are different monthly expenditures, classified broadly to the left, and progressively split into more detailed sub-classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't know what it is called and thus cannot find which tool to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NH670.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NH670.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42819" LastEditorUserId="836" LastEditDate="2017-12-04T07:58:59.997" LastActivityDate="2017-12-09T17:17:39.790" Title="What is the name for this chart which splits a quantity by allocation to a class hierarchy?" Tags="|infographics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25361" PostTypeId="1" CreationDate="2017-12-04T07:37:12.080" Score="1" ViewCount="57" Body="&lt;p&gt;For example, I consider an NN with n number of input units. What should the construction of NN be changed if the number of input units increases to 10*n?&#xA;I think the number of hidden units and hidden layer should be increased. And, we should have a higher number of samples in the dataset (e.g., using data augmentation).&lt;/p&gt;&#xA;&lt;p&gt;I also think dimensionality reduction could be used (like word embedding).&lt;/p&gt;&#xA;&lt;p&gt;What do you think about this problem?&lt;/p&gt;&#xA;" OwnerUserId="42821" LastEditorUserId="85045" LastEditDate="2021-02-08T17:16:19.093" LastActivityDate="2021-02-08T17:16:19.093" Title="What should we do with the construction of a classifier (e.g., NN) if we have more number of input features?" Tags="|neural-network|dimensionality-reduction|" AnswerCount="0" CommentCount="0" ClosedDate="2021-02-09T17:01:46.260" ContentLicense="CC BY-SA 4.0" />
  <row Id="25362" PostTypeId="1" AcceptedAnswerId="25389" CreationDate="2017-12-04T08:23:34.820" Score="1" ViewCount="144" Body="&lt;p&gt;From &lt;a href=&quot;https://github.com/cinqs/RL-An-Introduction-Solutions/blob/master/Reinforcement%20Learning.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton and Barto, Reinforcement Learning: An Introduction (second edition draft)&lt;/a&gt;, in equation 3.4 of page 38.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The probabilities given by the four-argument function p completely characterize the dynamics of a&#xA;  finite MDP. From it, one can compute anything else one might want to know about the environment,&#xA;  such as the state-transition probabilities (which we denote, with a slight abuse of notation, as a threeargument function &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$p(s^{'} | s, a) \dot{=}Pr\{S_t=s^{'} | S_{t-1} = s, A_{t-1}=a\} = &#xA;\sum_{r\in{R}}{p(s^{'},r|s,a)}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The author mentioned, &lt;em&gt;with a slight abuse of notation&lt;/em&gt;. &#xA;where is the abuse in the notation please? I didn't see anything that is not proper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="26856" LastEditorUserId="836" LastEditDate="2017-12-04T08:34:21.660" LastActivityDate="2017-12-04T23:15:48.537" Title="Cannot see what the &quot;notation abuse&quot; is, mentioned by author of book" Tags="|reinforcement-learning|notation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25364" PostTypeId="2" ParentId="25360" CreationDate="2017-12-04T08:49:33.133" Score="16" Body="&lt;p&gt;This is called a Sankey-Diagram:&#xA;&quot;Sankey diagrams are a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity.&quot; (&lt;a href=&quot;https://en.wikipedia.org/wiki/Sankey_diagram&quot; rel=&quot;noreferrer&quot;&gt;wiki&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easy package to make these graphs would be &quot;d3Network&quot; in R. But many other options are available. &lt;/p&gt;&#xA;" OwnerUserId="42825" LastActivityDate="2017-12-04T08:49:33.133" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25368" PostTypeId="2" ParentId="25352" CreationDate="2017-12-04T09:54:53.880" Score="0" Body="&lt;p&gt;I don't think you need some classification algorithm, you can use your basic understanding on data/ Business Knowledge to do the classification. As the number of data points are too low, the model cannot give you good/generalised results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you try applying some complex algorithm like &lt;strong&gt;SVM/NN&lt;/strong&gt;, it is of no use as the data is too low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still want to apply some machine learning algorithm and then you can apply &lt;strong&gt;Naive Bayes, Decision Tree&lt;/strong&gt; as these are the basic algorithms, can do the job.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-04T09:54:53.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25369" PostTypeId="1" CreationDate="2017-12-04T10:47:22.790" Score="1" ViewCount="28" Body="&lt;p&gt;What is the state of the art for DNNs to infer that they need more neurons than in the initial setup? Can they be acquired dynamically from the underlying &quot;resource substrate&quot;?&lt;/p&gt;&#xA;" OwnerUserId="33183" LastActivityDate="2017-12-04T10:47:22.790" Title="Possibility of self-growing neural nets" Tags="|reference-request|research|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25372" PostTypeId="1" AcceptedAnswerId="25374" CreationDate="2017-12-04T11:31:35.917" Score="1" ViewCount="805" Body="&lt;p&gt;In the pre-implemented models in keras (VGG16 ect) it is specified that we can change shape of the inputs of the models and still load the pre-trained imagenet weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am confused about is then what happens to the first layer weights? If the input tensor  has a different shape, then the number of weights will be different than for the pre-trained models isn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/fchollet/keras/blob/master/keras/applications/vgg16.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the implementation of the Keras VGG16 model for reference.&lt;/p&gt;&#xA;" OwnerUserId="42832" LastEditorUserId="42832" LastEditDate="2017-12-04T15:25:14.373" LastActivityDate="2017-12-04T15:25:14.373" Title="First layer weights for transfer learning with new input tensor in keras.applications models?" Tags="|machine-learning|keras|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25373" PostTypeId="1" AcceptedAnswerId="25375" CreationDate="2017-12-04T12:02:47.673" Score="3" ViewCount="8281" Body="&lt;p&gt;I am trying to build a neural network (3 layers, 1 hidden) in Python on the classic &lt;a href=&quot;https://www.kaggle.com/c/titanic/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;Titanic dataset&lt;/a&gt;.&#xA;I want to include a bias term following &lt;a href=&quot;https://github.com/llSourcell/make_a_neural_net_live_demo/blob/master/demo.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Siraj's examples&lt;/a&gt;, and the &lt;a href=&quot;https://www.youtube.com/watch?v=tIeHLnjs5U8&amp;amp;t=176s&quot; rel=&quot;nofollow noreferrer&quot;&gt;3Blue1Brown tutorials&lt;/a&gt; to update the bias by backpropagation, but I know my dimensionality is wrong. (I feel I am updating the biases incorrectly which is causing the incorrect dimensionality)&lt;/p&gt;&#xA;&lt;p&gt;The while loop in the code below works for a training dataset, where the node products and biases have the same dimension, but once I pass in a test example into the &lt;code&gt;predict&lt;/code&gt; function, the dimensions do not match up and get an error. I have commented my code with the dimensions of the calculations of dot products between nodes and inputs.&lt;/p&gt;&#xA;&lt;p&gt;Can someone help me understand what the dimensionality of the &lt;strong&gt;bias term&lt;/strong&gt; should be, both in this particular case and in general, and how it should be added (row-wise, column-wise)?&lt;/p&gt;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;def sigmoid(x, deriv=False):&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    Activation function&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    if(deriv==True):&#xA;        return (x*(1-x))&#xA;    return 1/(1+np.exp(-x))&#xA;&#xA;# learning rate, hidden layer dimension, error threshold, dropout rate&#xA;alpha, hidden_size, threshold, drop_rate = (0.035,32,0.1,0.5)&#xA;&#xA;# x_train and y_train are the training dataset and corresponding classes&#xA;&#xA;# syn0 and syn1 are the synapses, weight matrices between layers (3 layers, 2 synpases)&#xA;syn0 = 2*np.random.random((x_train.shape[1],hidden_size)) - 1 # NxH&#xA;syn1 = 2*np.random.random((hidden_size,1)) - 1 # Hx1&#xA;b1 = np.random.random((x_train.shape[0],hidden_size)) # MxH&#xA;b2 = np.random.random((x_train.shape[0],1)) # Mx1&#xA;&#xA;layer_2_error = 100*np.abs(np.random.random((y_train.shape[0],1))) - 1 # Mx1&#xA;avg_err = []&#xA;count = 0&#xA;&#xA;while np.mean(np.abs(layer_2_error)) &amp;gt; threshold:&#xA;&#xA;    # Forward&#xA;    layer_0 = x_train # training dataset&#xA;    A = np.dot(layer_0,syn0) + b1 # MxN X NxH + MxH ~ MxH&#xA;    layer_1 = sigmoid(A)&#xA;    # drop out to reduce overfitting&#xA;    layer_1 *= np.random.binomial([np.ones((len(x_train),hidden_size))],1-drop_rate)[0] * (1/(1-drop_rate))&#xA;&#xA;    B = np.dot(layer_1,syn1) + b2 # MxH X Hx1 + Mx1 ~ Mx1&#xA;    layer_2 = sigmoid(B)&#xA;&#xA;    # Backprop&#xA;    layer_2_error = layer_2 - y_train # Mx1&#xA;    layer_2_delta = layer_2_error * sigmoid(layer_2,deriv=True) # Mx1 * Mx1 ~ Mx1&#xA;&#xA;    layer_1_error = np.dot(layer_2_delta,syn1.T) # Mx1 X 1xH ~ MxH&#xA;    layer_1_delta = layer_1_error * sigmoid(layer_1,deriv=True) # MxH * MxH ~ MxH&#xA;&#xA;    # update weights&#xA;    syn1 -= alpha*np.dot(layer_1.T,layer_2_delta) # HxM X Mx1 ~ Hx1&#xA;    syn0 -= alpha*np.dot(layer_0.T,layer_1_delta) # NxM X MxH ~ NxH&#xA;&#xA;    # update biases&#xA;    b2 -= alpha*layer_2_delta # Mx1&#xA;    b1 -= alpha*layer_1_delta # MxH&#xA;&#xA;    avg_err.append(np.mean(np.abs(layer_2_error)))&#xA;    if count % 500 == 0:&#xA;        print(&amp;quot;Error after&amp;quot;,count,&amp;quot;iterations:&amp;quot;,np.mean(np.abs(layer_2_error)))&#xA;&#xA;    count += 1&#xA;&#xA;&#xA;def predict(x, w0, w1, b1, b2):&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    Function to predict an output given a data x, weight matrices w1 &amp;amp; w1 and biases b1 &amp;amp; b2&#xA;    &amp;quot;&amp;quot;&amp;quot;&#xA;    A = np.dot(x,w0) + b1 # mXN X NxH (+ MxH) ~ mxH&#xA;    layer_1 = sigmoid(A)&#xA;    B = np.dot(layer_1,w1) + b2 # mxH X Hx1 (+ Mx1) ~ mx1 (preds)&#xA;    layer_2 = B&#xA;    return (sigmoid(layer_2) &amp;gt; 0.5).astype(int)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41929" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2022-04-23T06:59:39.173" Title="What is the dimensionality of the bias term in neural networks?" Tags="|python|neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25374" PostTypeId="2" ParentId="25372" CreationDate="2017-12-04T12:04:43.367" Score="3" Body="&lt;p&gt;The first layers are convolution and pooling ones:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For the convolutional layers, the only weights are the kernels and the biases, and they have fixed size (e.g. 3x3x3, 5x5x3) and do not depend on the input tensor shape.&lt;/li&gt;&#xA;&lt;li&gt;The pooling layers do not have weights at all.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That's why you can reuse the weights independently from the input tensor shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With dense layers (i.e. the final layers), you need shapes to match, so you cannot reuse them if they do not.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2017-12-04T12:04:43.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25375" PostTypeId="2" ParentId="25373" CreationDate="2017-12-04T12:27:18.757" Score="6" Body="&lt;p&gt;As per the general case, the bias vector must have the same dimensions as the output vector.&lt;/p&gt;&#xA;&lt;p&gt;Please, have a look at this excellent &lt;a href=&quot;https://docs.google.com/presentation/d/1TVixw6ItiZ8igjp6U17tcgoFrLSaHWQmMOwjlgQY9co/pub?slide=id.g110257a6da_0_476&quot; rel=&quot;nofollow noreferrer&quot;&gt;presentation&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/co9zG.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/co9zG.gif&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In this example by M.Görner, there are 10 classes, so is bias dimension.  Once inputs are multiplied by weights, the bias is added pointwise (it is 'broadcasted'). And that's pretty much it.&lt;/p&gt;&#xA;&lt;p&gt;For those wondering the origin of 100 and 784.  There are 100 training examples (images) and 784 features (total pixels/image).&lt;/p&gt;&#xA;" OwnerUserId="2968" LastEditorUserId="134425" LastEditDate="2022-04-23T06:59:39.173" LastActivityDate="2022-04-23T06:59:39.173" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25376" PostTypeId="1" AcceptedAnswerId="25381" CreationDate="2017-12-04T12:56:11.733" Score="0" ViewCount="3291" Body="&lt;p&gt;I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? How should i approach for initial tweet pre-processing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw many tutorials but all of them used plain texts. I am a newbie in data science. &lt;/p&gt;&#xA;" OwnerUserId="42836" LastActivityDate="2017-12-04T16:49:01.440" Title="Better input for Doc2Vec" Tags="|text-mining|data-cleaning|word2vec|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25377" PostTypeId="1" CreationDate="2017-12-04T15:12:58.917" Score="2" ViewCount="2048" Body="&lt;p&gt;I am trying to implement simple recommender system and I am trying to understand different approaches to achieve my goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset consists of users and items that they bought. I have information about what items user bought and descriptions of these items in form of titles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first I though I could use user based collaborative filtering approach but I am stuck at this. I am not quite sure how to calculate similarity for boolean data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I have data like this for example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   1  2  3  4&#xA;A  0  1  0  1&#xA;B  0  1  0  1&#xA;C  1  0  1  1&#xA;D  0  1  0  0&#xA;E  0  0  1  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I want to recommend items for user &lt;strong&gt;E&lt;/strong&gt;, so how should I calculate similarity in this case? I chosen for example cosine similarity from scikit learn module in python. But I am not quite sure what should be considered as input. From what I read it should be only vectors of items that two users for which similarity is calculated have in common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example if I wanted to compute similarity between user &lt;strong&gt;E&lt;/strong&gt; and &lt;strong&gt;C&lt;/strong&gt; what should be my input? Because if I input only values that they have in common it does not make sense right? Beacuse input will be [1, 1] and [1, 1] and for that similarity is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I tried to input the whole vector like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import cosine_similarity&#xA;from numpy import array, reshape&#xA;&#xA;c = array([1, 0, 1, 1])&#xA;e = array([0, 0, 1, 1])&#xA;&#xA;result = cosine_similarity(c.reshape(1, -1), e.reshape(1, -1))&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; result is 0.81649658&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this approach I think makes more sense but I am not sure if it is acceptable based on what I learned about this type of recommendation.&lt;/p&gt;&#xA;" OwnerUserId="42840" LastEditorUserId="42840" LastEditDate="2017-12-05T10:03:14.020" LastActivityDate="2019-07-09T13:51:20.840" Title="Calculate similarity on boolean data" Tags="|recommender-system|similarity|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25378" PostTypeId="2" ParentId="25377" CreationDate="2017-12-04T15:56:30.963" Score="5" Body="&lt;p&gt;You should look at the &lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jaccard Index&lt;/a&gt;, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, 1 means the item is present, 0 otherwise. For example: for an universe of items banana, orange and apple. the set banana, orange will be represented by (1, 1, 0). The Jaccard Index is the intersection of the sets over union the sets so for a set (1, 1) it's value is 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity is for real-valued vectors, but whether cosine similarity is better than the Jaccard index and vice versa depends on the application. You should do a test on your data and verify which is better, for a discussion see this &lt;a href=&quot;https://datascience.stackexchange.com/questions/5121/applications-and-differences-for-jaccard-similarity-and-cosine-similarity&quot;&gt;question&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-05T10:16:19.897" LastActivityDate="2017-12-05T10:16:19.897" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25380" PostTypeId="1" CreationDate="2017-12-04T16:41:09.930" Score="0" ViewCount="615" Body="&lt;p&gt;It has been a long I am confused on understanding some of the AlexNet architecture : &lt;a href=&quot;https://i.stack.imgur.com/W7ahH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W7ahH.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The output of the first conv layer is 55x55x48 (96 considering the division between GPUs but let's stick to 1 GPU so depth 48). Then max pooling is applied and there come my problem.&lt;/p&gt;&#xA;&lt;p&gt;When applying max pooling, the result is 27x27x48 right ? If so, how is applied the next convolution over this result (with 5x5x48 filters) to output 27x27x128 ? I finally don't see how and when to apply max-pooling in between convolutions. I must miss something here...&lt;/p&gt;&#xA;" OwnerUserId="41728" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-04T20:11:17.880" Title="AlexNet second layer understanding" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25381" PostTypeId="2" ParentId="25376" CreationDate="2017-12-04T16:49:01.440" Score="1" Body="&lt;p&gt;There are plenty of different approaches you can use, and none is the &lt;em&gt;universal best solution&lt;/em&gt;. However, in general, preprocessing in twitter data, especially for Doc2Vec, follow:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;tokeninzing (nltk tokenizer, custom regex tokenizer) to identify words. Depending on the application, you can work on special cases such as english contractions, negations (sentiment analysis especially) and others&lt;/li&gt;&#xA;&lt;li&gt;normalizing urls and/or mentions, transforming each of them to the same version to reduce voacbulary size (&lt;a href=&quot;http://someurl&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://someurl&lt;/a&gt;, @mention) &lt;strong&gt;or&lt;/strong&gt; totally removing them.&lt;/li&gt;&#xA;&lt;li&gt;removing of numbers, punctuation or others. Note that this is highly domain dependent, for example &quot;!&quot; has been shown to express a lot in sentiment analysis. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In fact, doc2vec doesn't expect anything, it uses the vocabulary you give. So you decide what you want to keep. A good practice is to experiment different tokenizers and training phases if you have time to see what works best. Also use what has already been done in your field of application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget that the input of Doc2Vec is an iterator over a list of TaggedDocument. See this &lt;a href=&quot;https://towardsdatascience.com/a-gentle-introduction-to-doc2vec-db3e8c0cce5e&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; for more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck !&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-12-04T16:49:01.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25382" PostTypeId="1" AcceptedAnswerId="25391" CreationDate="2017-12-04T18:06:33.343" Score="0" ViewCount="393" Body="&lt;p&gt;l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance&lt;/p&gt;&#xA;&#xA;&lt;p&gt;v_1=[0,4,0,0,1] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 at index 1 means we have four values of 1 and 1 at index 4 it means we have one value of 4.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose of normalizing these histograms is to get a distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a sample of my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Vectors=[[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What l tried ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vectors=(vectors-np.mean(vectors,axis=0)) / np.std(vectors,axis=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and l tried :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vectors=sklearn.preprocessing.scale(vectors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A)Is it correct what l have tried ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;B)l have question to normalization by subtracting means and dividing by standard deviation:&#xA;through all the vectors, through each dimension, means( of a dimension) is computed. Than each data point of this dimension is subtracted from its mean. Is it correct what l'm saying ? &#xA;in other way for each dimension, we have it's own mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if we have dim(vectors)=(400,2000) such that 400 is the number of examples.&lt;br&gt;&#xA;we compute 2000 means and 2000 std since we have 2000 dimensions. And then each data point is subtracted from its means and divided by its std (mean and std of that dimension)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for correcting me.&lt;/p&gt;&#xA;" OwnerUserId="42275" LastEditorUserId="1330" LastEditDate="2022-09-07T12:45:44.943" LastActivityDate="2022-09-07T12:45:44.943" Title="What is the best way to normalize histogram vectors to get distribution?" Tags="|machine-learning|python|scikit-learn|preprocessing|histogram|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25383" PostTypeId="1" CreationDate="2017-12-04T18:52:19.273" Score="1" ViewCount="190" Body="&lt;p&gt;I am currently working as a registered nurse, but I have decided that I can no longer do this as a career. I was essentially forced by my parents to go into the health field. After doing research on many career fields, I believe that a career as a data scientist is a career that I would enjoy and be satisfied in. The only problem is that I don’t know how to get started. I have a bachelor’s degree in biology apart from my nursing degree. I’m tnis degree program, I had to take college level calculus, prob&amp;amp;stats, algebra, etc.  To get to my point, should I go back to school to obtain a degree in statistics or CS, or should I train on my own to learn the skill necessary for the job?  I am very disciplined when it comes to education, so self- teaching would not be an issue on my end. Any advice would be very helpful. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="42854" LastActivityDate="2017-12-05T06:05:49.603" Title="How to get started?" Tags="|beginner|" AnswerCount="1" CommentCount="3" ClosedDate="2017-12-05T16:01:58.640" ContentLicense="CC BY-SA 3.0" />
  <row Id="25384" PostTypeId="2" ParentId="25380" CreationDate="2017-12-04T19:29:48.063" Score="2" Body="&lt;p&gt;Okay, I got it. If anyone interested, they use 5x5 filter but with padding 2 and striding 1 so that with bias it doesn't change the 2D dimension of the output when applied on the result of max-pooling. The info on padding isn't present on the original paper...&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2017-12-04T19:29:48.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25385" PostTypeId="1" CreationDate="2017-12-04T20:31:04.167" Score="2" ViewCount="240" Body="&lt;p&gt;We have been tasked with building an in-house sentiment tool and we are going to use it on a multitude of data sources; survey responses, reviews, social listening etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may be obvious to some but I would like to hear some thoughts. So my question is, would you train separate models for each data source since the documents from each source would be quite different? We would then apply the appropriate model depending on the data source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it best practice to train one model using a sample of all the data sources and apply that model to everything?&lt;/p&gt;&#xA;" OwnerUserId="41460" LastEditorUserId="41460" LastEditDate="2017-12-04T22:32:07.497" LastActivityDate="2017-12-09T18:27:53.497" Title="Sentiment Analysis: Train Separate Models or Use One for All" Tags="|machine-learning|training|sentiment-analysis|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25387" PostTypeId="2" ParentId="25250" CreationDate="2017-12-04T22:46:16.923" Score="1" Body="&lt;p&gt;Most likely,  SGD is not a limiting factor for you. But, have you considered taking a classification rather than regression approach? (It looks like you're predicting real values as opposed to classes). Since you  state that the prediction doesn't have to be perfect, why not try grouping your outcome variable into bins, then predict the bins?  You'll have a far less granular solution, but you might find it works. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-12-04T22:46:16.923" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25389" PostTypeId="2" ParentId="25362" CreationDate="2017-12-04T23:15:48.537" Score="2" Body="&lt;p&gt;The mathematical expression is completely legit. The abuse is in the fact that the function $p$, which is defined first time in equation 3.2, which:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The function $p: S$ x $R$ x $S$ x $A \rightarrow [0,1]$. is an ordinary&#xA;  deterministic function of four arguments...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;is re-defined slightly differently just two lines after this definition (equation 3.4), as a three-argument function $p: S$ x $S$ x $A \rightarrow [0,1]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they used $p$ to represent the regular probability measure, there would be no abuse. In the authors' notation, $p$ is a deterministic function, while the regular probability function is denoted as $Pr$; and keeping the same name for slightly different functions, is where the &quot;innocent&quot; notation abuse comes from.&lt;/p&gt;&#xA;" OwnerUserId="20572" LastActivityDate="2017-12-04T23:15:48.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25390" PostTypeId="2" ParentId="25385" CreationDate="2017-12-04T23:39:20.703" Score="1" Body="&lt;p&gt;I would suggest that you train a single model. The features correlated with positive or negative sentiment in a review are likely the same as those in a survey response. That is, “good” means good in both, “bad” means bad in both. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general you should combine models and datasets when you think that the features in the different situations have the same meanings/interpretations. If they have contradictory or different meanings/interpretations, then you should consider training separate models. &lt;/p&gt;&#xA;" OwnerUserId="1037" LastActivityDate="2017-12-04T23:39:20.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25391" PostTypeId="2" ParentId="25382" CreationDate="2017-12-05T00:10:44.380" Score="2" Body="&lt;p&gt;It's okay, what you are saying, except the &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;each data point is subtracted from its means&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;part because it's in the opposite way, you have to subtract the means from their data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess your problem is coming from that you put your vectors into the matrix as row vectors, but &lt;code&gt;np.mean(..., axis = 0)&lt;/code&gt; calculates the column means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use the above formula for normalizing then you should transpose your matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vectors = np.array([[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]])&#xA;vectors.T&#xA;array([[0, 5, 1, 0],&#xA;       [4, 0, 0, 0],&#xA;       [0, 0, 0, 6],&#xA;       [0, 3, 0, 0],&#xA;       [1, 1, 0, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And now the &lt;code&gt;np.mean(vectors.T, axis=0)&lt;/code&gt; gives the correct means for your vectors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([ 1. ,  1.8,  0.2,  1.2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Substract the means from the transposed matrix like &lt;code&gt;vectors.T - np.mean(vectors.T, axis=0)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[-1. ,  3.2,  0.8, -1.2],&#xA;       [ 3. , -1.8, -0.2, -1.2],&#xA;       [-1. , -1.8, -0.2,  4.8],&#xA;       [-1. ,  1.2, -0.2, -1.2],&#xA;       [ 0. , -0.8, -0.2, -1.2]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And finally divide it by the standard deviation of the transposed matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;t_vectors = (vectors.T-np.mean(vectors.T,axis=0)) / np.std(vectors.T,axis=0)&#xA;t_vectors&#xA;array([[-0.64549722,  1.65027399,  2.        , -0.5       ],&#xA;       [ 1.93649167, -0.92827912, -0.5       , -0.5       ],&#xA;       [-0.64549722, -0.92827912, -0.5       ,  2.        ],&#xA;       [-0.64549722,  0.61885275, -0.5       , -0.5       ],&#xA;       [ 0.        , -0.4125685 , -0.5       , -0.5       ]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sklearn gives the same &lt;code&gt;sklearn.preprocessing.scale(vectors.T)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14202" LastEditorUserId="14202" LastEditDate="2017-12-05T09:46:51.150" LastActivityDate="2017-12-05T09:46:51.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25392" PostTypeId="1" AcceptedAnswerId="25402" CreationDate="2017-12-05T00:19:48.393" Score="1" ViewCount="4223" Body="&lt;p&gt;I'm trying to evaluate &lt;em&gt;classifiers comparison&lt;/em&gt; by running the sample script that can be found &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;What I noticed is that in some cases the classifier is not reset.&#xA;In fact, duplicating some of those (with no parameter change) the score and the countour change between the two.&lt;/p&gt;&#xA;&lt;p&gt;This can be seen simply replacing &lt;code&gt;AdaBoostClassifier()&lt;/code&gt; in the classifier list with another &lt;code&gt;MLPClassifier(alpha=1)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;I guess that at every cicle of the for loop the classifier should be reset in order to make a fair comparison among the different models, and this case should behave the same I think.&lt;/p&gt;&#xA;&lt;p&gt;In particular, differences are noticed duplicating the MPL (Neural Net) and the Random Forest, while there is no change duplicating KNN or RBF SVM.&lt;/p&gt;&#xA;&lt;p&gt;I also tried to &lt;code&gt;clone&lt;/code&gt; the classifier, and even &lt;code&gt;del clf&lt;/code&gt; in the loop, but the behaviour stays the same.&lt;/p&gt;&#xA;&lt;p&gt;How can I make the evaluation replicable and not influenced by the previous run? I want to be sure that when I use the same model, and only change the parameters the result is correct, and this will be possible only if two identical model yield the same result.&lt;/p&gt;&#xA;" OwnerUserId="42863" LastEditorUserId="98307" LastEditDate="2020-08-11T13:42:43.917" LastActivityDate="2020-08-11T13:42:46.743" Title="scikit-learn classifier reset in loop" Tags="|machine-learning|python|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25394" PostTypeId="1" CreationDate="2017-12-05T02:37:47.330" Score="0" ViewCount="81" Body="&lt;p&gt;I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm training it with the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X = [[-3], [-2], [3], [2]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y = [[0], [0], [1.], [1.]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I change Y to [&lt;a href=&quot;https://github.com/demilp/NeuralNet/blob/master/NeuralNet.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;https://github.com/demilp/NeuralNet/blob/master/NeuralNet.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my gradrient function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    def gradients(self, X, Y):&#xA;    dws = []&#xA;    dbs = []&#xA;    h, activations, zs = self.forward(X)&#xA;    da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput&#xA;    for d in np.arange(1, len(self.weights))[::-1]:&#xA;        a = activations[d] #output for layer d&#xA;        z = zs[d] #linear for layer d&#xA;        w = self.weights[d] #weights for layer w&#xA;&#xA;        dz = da * a * (1-a)&#xA;        m = w.shape[1]&#xA;        dw = 1./m * dz.dot(activations[d].T)&#xA;        db = 1./m * np.sum(dz, axis=1, keepdims=True)&#xA;        da = w.T.dot(dz)&#xA;        dws.insert(0, dw)&#xA;        dbs.insert(0, db)&#xA;    dws.insert(0, None)&#xA;    dbs.insert(0, None)&#xA;    return dws, dbs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the whole code:&#xA;&lt;a href=&quot;https://github.com/demilp/NeuralNet/blob/master/NeuralNet.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42865" LastActivityDate="2017-12-05T03:07:17.277" Title="Why can it be that my neural network is predicting the contrary?" Tags="|neural-network|deep-learning|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25395" PostTypeId="2" ParentId="25385" CreationDate="2017-12-05T02:41:48.217" Score="1" Body="&lt;p&gt;I think what @Kevin H has said is right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are couple of more thing which I would suggest you to spend more time on&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Spend more time on &lt;strong&gt;understanding data&lt;/strong&gt;, what does each word in each scenario means.&lt;/li&gt;&#xA;&lt;li&gt;Generate a &lt;strong&gt;Stopword Dictionary&lt;/strong&gt; to remove all the unnecessary words, &lt;strong&gt;Lemmatization&lt;/strong&gt;  of words.&lt;/li&gt;&#xA;&lt;li&gt;Based on your &lt;strong&gt;Business Understanding&lt;/strong&gt; make sure that all the important details are capture.&lt;/li&gt;&#xA;&lt;li&gt;Using a &lt;strong&gt;Single Model&lt;/strong&gt; helps in &lt;strong&gt;generalizing&lt;/strong&gt; the results but before deciding that as the final model, make different models for different business problems and do ensemble of them and validate to know which model give better results(Ensemble or Single Model).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Main reason for suggesting Ensemble is Navie Bayes classifier works best with short sentence but for rest that is not the best model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding Sampling, I think to do that you need to understand data clearly as to extract sample you need to make sure that every dimension of the population is covered(&lt;strong&gt;Sample should exactly represent the population&lt;/strong&gt;). So, you should be very careful when implementing Sampling techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choose the best model, you need to try all the methods apply them in all applicable conditions and based on your Business Application and results achieved, you need to decide but generally people implement Ensemble to achieve best results(in most of the cases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more on Ensemble or any other technique let me know, will help you!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-05T02:41:48.217" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25396" PostTypeId="2" ParentId="25394" CreationDate="2017-12-05T03:07:17.277" Score="1" Body="&lt;p&gt;After being all day trying to figure this out I had to post this question to find clarity.&#xA;The problem is here&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dw = 1./m * dz.dot(activations[d].T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it should be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dw = 1./m * dz.dot(activations[d-1].T)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42865" LastActivityDate="2017-12-05T03:07:17.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25397" PostTypeId="2" ParentId="14581" CreationDate="2017-12-05T04:12:26.690" Score="-2" Body="&lt;p&gt;GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance.  &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2017-12-05T04:12:26.690" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25399" PostTypeId="1" CreationDate="2017-12-05T06:05:38.170" Score="1" ViewCount="61" Body="&lt;p&gt;When tuning the Deep Q net parameters we use the immediate rewards . Specially in action replay and regressing .  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in the run time we don't care about the rewards at all. Because our neural network will take states as the input . I am explaining this relevant to the deep mind Atari game play paper . Any one thing it is wastage of input ? Is there a mechanism we can use something like Score even in the run time . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;p.s - I understand how they update the Neural Net parameters with regression . Sometimes with TD(λ) . &lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="33338" LastEditDate="2017-12-05T07:06:38.750" LastActivityDate="2017-12-08T03:36:22.817" Title="How can we use the current rewards as a system input in the RUN time when working with Deep Q learning?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25400" PostTypeId="2" ParentId="25383" CreationDate="2017-12-05T06:05:49.603" Score="2" Body="&lt;p&gt;Congrats on your career move, but please know that data science is one of the more difficult fields to enter, being a combination of statistics, programming, computer science, mathematics, etc. But, for someone motivated enough, with sharp acumen and intuitive data skills, it's a great field. A commenter recommended Andrew Ng's (pronounced 'ing' ) online course through coursera, and I also agree this is a great foundation.  It is dense, but you'll need to master it if you hope to be a quality data scientist. He uses Octave, but I advise that you come up to speed quickly in R or Python. If you can translate Andy's code to either of those platforms, you'll be in good shape to start your practice. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-12-05T06:05:49.603" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25401" PostTypeId="2" ParentId="25399" CreationDate="2017-12-05T07:24:23.777" Score="1" Body="&lt;p&gt;My friend - Thushan Gave me this answer . So I will post it . I think it will describe it . Sometimes in the run time it's not desirable to work with immediate rewards . It can mess up the agent. I will quote it here . &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Your argument makes sense, but not always. And this is a design decision &#xA;  you make in your problem formulation. However to highlight why this&#xA;  won't work in some situations, I'll give an example.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Say you're playing a shooting game with a RL agent. And at some&#xA;  specific location you get killed by getting shot from an enemy.&#xA;  However if you include the reward with the state, this might suggest&#xA;  to the algorithm, that this location is always bad and will try to&#xA;  avoid it (which is wrong). We should't be avoiding locations, but&#xA;  enemies.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2017-12-05T07:24:23.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25402" PostTypeId="2" ParentId="25392" CreationDate="2017-12-05T09:49:32.590" Score="1" Body="&lt;p&gt;The behaviour you are seeing is not related to not properly resetting models but the stochastic nature of most of these algorithms. By setting a random seed the same random numbers will be generated every time. See:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/q/40750394/9214357&quot;&gt;How to seed the random number generator for scikit-learn?&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;However, while this will lead to a reproducible sample, this might still not be fair. If one model randomly gets a good seed and another one a bad one you will unfairly always favor the first one. What you could do is run the models multiple times with the same hyperparameters but with different seed and look for the average performance. This way you get a more fair comparison and the reproducibility. Pick the seeds up front, then you can loop over them. Something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;seeds = (1, 2, 3, 4, 5)&#xA;performances = []&#xA;for seed in seeds:&#xA;    performances.append(score(Model(param1=1, param2=2, random_state=seed)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="98307" LastEditDate="2020-08-11T13:42:46.743" LastActivityDate="2020-08-11T13:42:46.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25403" PostTypeId="1" CreationDate="2017-12-05T10:23:18.150" Score="0" ViewCount="1466" Body="&lt;p&gt;I have a set of documents where I need to extract important keywords in the document and then rank those keywords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ranking should be done based on relevance and/or other metrics. Are there any popular algorithms/methods/libraries that I can use for this task?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="1330" LastEditDate="2017-12-05T20:42:54.520" LastActivityDate="2018-02-23T00:52:04.260" Title="Rank terms in a bag -of-words model" Tags="|data-mining|nlp|text-mining|learning-to-rank|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25404" PostTypeId="2" ParentId="25307" CreationDate="2017-12-05T10:59:21.117" Score="1" Body="&lt;p&gt;I will try to illustrate the lambda collocation metric, first we have to define a function c(x, z) that receives two K-word expressions where x is the K-word expression you want to score and z corresponds to all possible values of a K-word expression in the corpus. c(x, z) returns a binary vector (j1, ..., jk) such that ji = 1 if xi == zi, 0 otherwise. Now a binary vector can represent a decimal number up to  2^K = M, so c(x, z) = ci means that the binary vector returned was equal to the binary representation of the decimal number i. Now as it says on the documentation ni corresponds to the number of times the c(x, z) function was equal to ci through the corpus. Now an example, consider the following sentence:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;this sentence is an example for this sentence&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The target expression is &quot;this sentence&quot; so K=2. You have the following possible values for z:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;this sentence&#xA;sentence is&#xA;an example&#xA;example for&#xA;for this&#xA;this in&#xA;in this&#xA;this sentence&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Given that K=2, the possible values for c(x, z) are 00 = 0, 01 = 1, 10 = 2 and 11 = 3. Then considering the target expression and all the values of z above you have the following, (for x = &quot;this sentence&quot;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c(x, this sentence) = 11 = c3&#xA;c(x, sentence is) = 00 = c0&#xA;c(x, an example) = 00 = c0&#xA;c(x, example for) = 00 = c0&#xA;c(x, for this) = 00 = c0&#xA;c(x, this in) = 10 = c2&#xA;c(x, in this) = 00 = c0&#xA;c(x, this sentence) = 11 = c3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we have 5 times c0 which means n0 = 5, 2 times c3 means n3 = 2 and n2 = 1. Finally bi is the number of bits equals to 1 in c1 which b0 = 0, b1 = 1, b2 = 1, b3 = 2. This completes all the values you need for computing the lambda collocation scoring metric. Also take into account you could add smoothing to ni to avoid 0 counts. Now with respect to &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The definition of a contingency table for the array data is simply the number of times a given value appears (i.e. its frequency). See &lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/contingency-tables-in-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and the usage of the &lt;a href=&quot;https://www.rdocumentation.org/packages/base/versions/3.4.1/topics/table&quot; rel=&quot;nofollow noreferrer&quot;&gt;table command&lt;/a&gt; for more information. For the example above we have the following array data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;c3 c0 c0 c0 c0 c2 c0 c3&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the corresponding contingency table is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c0 | c1 | c2 | c3&#xA;-----------------&#xA; 5 |  0 |  1 |  2 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-05T10:59:21.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25405" PostTypeId="1" CreationDate="2017-12-05T11:29:19.657" Score="0" ViewCount="730" Body="&lt;p&gt;I am trying to predict risk labels of late deliveries for the upcoming 1-2 months based on 3 years of historic delivery data. &#xA;The target variable is a categorical output based on how many days late a delivery was (on time, 1-2 days late, one week late, more than one week late).&#xA;Input variables include things like material information, suppliers location &amp;amp; distance to warehouse, # of times delivery was marked as delayed and suppliers monthly quality ratings as well as information on known quality issues registered with supplier over time etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So looking at the information above and that I have records per year of around 500.000 delivieries from 300 suppliers I believe random forests would be a good choice to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I´m thinking it would not capture important time sensitive trends, such as a supplier all of the sudden delivering late while he was reliable the last 3 years.&#xA;So what is a good way to capture time sensitive trends in RF or should I choose a different algorithm form the time-series domain?&#xA;Or is there a way to combine multiple approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for you input! &lt;/p&gt;&#xA;" OwnerUserId="42889" LastEditorUserId="42889" LastEditDate="2017-12-05T16:24:49.153" LastActivityDate="2018-04-12T21:13:30.967" Title="Which algorithm to use for predicting late deliveries at warehouse?" Tags="|data-mining|random-forest|supervised-learning|prediction|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25406" PostTypeId="1" AcceptedAnswerId="25492" CreationDate="2017-12-05T11:39:15.193" Score="3" ViewCount="1313" Body="&lt;p&gt;In order to have an actor critic RL model there are two things to be satisfied . &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Value approximation function should converge to a local minimum&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - f_w(s,a)]\frac{\partial f_w(s,a)}{\partial w} = 0$$&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;The following condition should be satisfied with the parameterization &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial f_w(s,a)}{\partial w} = \frac{\partial \pi(s,a)}{\partial \theta} \frac{1}{ \pi(s,a)} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So specifically how can we design a model to meet the second condition?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here I want to highlight the value function approximation in actor-critic methods . We need to optimize the critic also as we did for the Q learning but following the on policy which is taking the TD error according to the actor. Here I will put the final equation of actor critic. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N87zk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N87zk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here simply we can take the critic neural net's output as the state value function or the f above. So how to satisfy the condition 2 ? &lt;/p&gt;&#xA;" OwnerUserId="33338" LastEditorUserId="33338" LastEditDate="2017-12-09T17:42:44.203" LastActivityDate="2017-12-11T18:31:38.680" Title="How to design two different neural nets for actor and critic RL?" Tags="|reinforcement-learning|actor-critic|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25407" PostTypeId="1" AcceptedAnswerId="25413" CreationDate="2017-12-05T12:09:52.173" Score="11" ViewCount="1248" Body="&lt;p&gt;I have many times analysed a dataset on which I could not really do any sort of classification. To see whether I can get a classifier I have usually used the following steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate box plots of label against numerical values.&lt;/li&gt;&#xA;&lt;li&gt;Reduce the dimensionality to 2 or 3 to see if classes are separable, also tried LDA sometimes.&lt;/li&gt;&#xA;&lt;li&gt;Forcefully try to fit SVMs and Random Forests and look at feature-importance to see if the features make any sense or not.&lt;/li&gt;&#xA;&lt;li&gt;Try to change the balance of classes and techniques like under-sampling and over-sampling to check if class imbalance might be an issue.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are many other approaches I can think of, but have not tried. Sometimes I know that these features are not good and not at all related to the label we are trying to predict. I then use that business intuition to end the exercise, concluding that we need better features or totally different labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how does a Data Scientist report that the classification can not be done with these features. Is there any statistical way to report this or fitting the data in different algorithms first and looking at validation metric is the best option?&lt;/p&gt;&#xA;" OwnerUserId="16368" LastEditorUserId="28175" LastEditDate="2017-12-28T12:03:16.050" LastActivityDate="2018-01-17T15:33:14.337" Title="When do we say that the dataset is not classifiable?" Tags="|machine-learning|classification|deep-learning|class-imbalance|bayes-error|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25408" PostTypeId="2" ParentId="25385" CreationDate="2017-12-05T12:10:57.643" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;since the documents from each source would be quite different&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Well if this is true I would suggest to train different models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not that the opinions of the others are wrong, but if you have way different&#xA;documents, for example tweets and newspapers articles, the accuracy of a model trained in both tweets and articles could not match the accuracy of models that are trained in only one kind of document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tweets have a lot of orthographic mistakes, lots of time the names are not capitalized, they also have emoji and slang. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand articles have most of the times a more &quot;neutral&quot; language regarding sentiments, they are way bigger in size than tweets and most of the times are well written.&lt;/p&gt;&#xA;" OwnerUserId="42460" LastEditorUserId="29575" LastEditDate="2017-12-09T18:27:53.497" LastActivityDate="2017-12-09T18:27:53.497" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25409" PostTypeId="2" ParentId="25315" CreationDate="2017-12-05T12:32:48.407" Score="5" Body="&lt;p&gt;Yes, you are right. The soft-max layer outputs a probability distribution, i.e. the values of the output sum to 1. The sigmoid function outputs marginal probabilities and therefore can be used for multiple-class classification, when the classes are not mutually exclusive. Additionally the soft-max layer is soft version of the max-output layer so it is differentiable and also resilient to &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function#Softmax_normalization&quot; rel=&quot;nofollow noreferrer&quot;&gt;outliers&lt;/a&gt;. A problem with sigmoids is that as you reach saturation (values get close to 1 or 0), the gradients vanish. This is detrimental to optimization speed and soft-max doesn't have this &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/32iyt9/question_comparison_between_softmax_and_sigmoid/&quot; rel=&quot;nofollow noreferrer&quot;&gt;problem&lt;/a&gt;. Another interpretation is soft-max as a &lt;a href=&quot;https://stats.stackexchange.com/a/254071&quot;&gt;generalization&lt;/a&gt; of sigmoid, actually when there are two classes they are the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wrapping up you should use soft-max when the classes are mutually exclusive and sigmoid when the classes are independent. This can be summarized in the following table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Soft-Max                            |                 Sigmoid&#xA;-------------------------------------------------------------------------&#xA;Used for multi-classification       |  Used for binary classification in &#xA;in logistic regression model.       |  logistic regression model.&#xA;-------------------------------------------------------------------------&#xA;The probabilities sum will be 1     | The probabilities sum need not be 1&#xA;-------------------------------------------------------------------------&#xA;Used in the different layers of     | Used as activation function while &#xA;neural networks.                    | building neural networks&#xA;-------------------------------------------------------------------------&#xA;The high value will have the higher | The high value will have the high &#xA;probability than other values.      | probability but not the higher&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more information on soft-max look at the following links: &lt;a href=&quot;http://cs231n.github.io/linear-classify/#softmax&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;https://www.quora.com/Artificial-Neural-Networks-Why-do-we-use-softmax-function-for-output-layer&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt; and &lt;a href=&quot;https://deepnotes.io/softmax-crossentropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;3&lt;/a&gt;. For a step-by-step guide, including usage in Python see this reference &lt;a href=&quot;https://www.pyimagesearch.com/2016/09/12/softmax-classifiers-explained/&quot; rel=&quot;nofollow noreferrer&quot;&gt;4&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more on soft-max vs sigmoid check this: &lt;a href=&quot;https://www.quora.com/Why-is-it-better-to-use-Softmax-function-than-sigmoid-function&quot; rel=&quot;nofollow noreferrer&quot;&gt;5&lt;/a&gt;, &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/32iyt9/question_comparison_between_softmax_and_sigmoid/&quot; rel=&quot;nofollow noreferrer&quot;&gt;6&lt;/a&gt; and this &lt;a href=&quot;http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/&quot; rel=&quot;nofollow noreferrer&quot;&gt;7&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a more reliable source on why to use soft-max regression for mutually exclusive classes you can look &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression#Softmax_Regression_vs._k_Binary_Classifiers&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. This page is part of Unsupervised Feature Learning and Deep Learning Tutorial of Stanford University, it contains material contributed by Andrew Ng and others. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-06T11:00:24.793" LastActivityDate="2017-12-06T11:00:24.793" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25410" PostTypeId="1" AcceptedAnswerId="25412" CreationDate="2017-12-05T13:35:11.837" Score="2" ViewCount="88" Body="&lt;p&gt;I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists &quot;&amp;lt;1&quot; ppm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a good way to visualize this in a graph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I pick an arbitrary decimal less than one, so that it can be shown on the graph? I feel setting it to zero would not be right either, as they went to the extent of listing it, so it must be significant.&lt;/p&gt;&#xA;" OwnerUserId="42895" LastEditorUserId="29575" LastEditDate="2017-12-05T14:53:50.650" LastActivityDate="2023-06-08T01:57:57.450" Title="When visualizing data that has &lt;1 or &lt;5 ppm how do you display this?" Tags="|visualization|data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25411" PostTypeId="1" AcceptedAnswerId="25415" CreationDate="2017-12-05T13:52:29.013" Score="3" ViewCount="720" Body="&lt;p&gt;I am reviewing various autoencoder setups for MNIST reconstruction, Seq2Seq translation and others.&#xA;My naive understanding of data flow is as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input -&gt; [Encoder] -&gt; Hidden Representation -&gt; [Decoder] -&gt; Output.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zzzp7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zzzp7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in case of Seq2Seq translation task similar to &lt;a href=&quot;https://arxiv.org/pdf/1409.3215.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutskever et. al.&lt;/a&gt; decoder input is combined from hidden state + input sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uqg0V.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uqg0V.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder how is the input of decoder dependent on the target task? Why do we need to put input sequence besides the hidden state? Any high-level explanation is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="42896" LastActivityDate="2018-01-29T12:49:16.493" Title="What exactly is the input of decoder in autoencoder setup" Tags="|machine-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25412" PostTypeId="2" ParentId="25410" CreationDate="2017-12-05T15:10:17.143" Score="1" Body="&lt;p&gt;Is it a univariate plot?  If so, just bin the data and toss it into the lowest bin.&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-12-05T15:10:17.143" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25413" PostTypeId="2" ParentId="25407" CreationDate="2017-12-05T15:17:24.840" Score="4" Body="&lt;p&gt;It depends on your data. There is something called human level error. Suppose tasks like reading of printed books, humans do not struggle to read and it might not happen to make a mistake unless because of bad quality of printing. In cases like reading hand-written manuscripts, it may happen a lot not to understand all words if the font of the writer is odd to reader. In the first situation the human level error is too low and the learning algorithms can have the same performance but the second example illustrates the fact that in some situations the human level error is so much high and in a usual manner (if you use the same features as humans) your learning algorithm will have so much error ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In statistical learning, there is something called &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwj0p_jek_PXAhWJp6QKHXlVC-sQFggnMAA&amp;amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FBayes_error_rate&amp;amp;usg=AOvVaw1n3f-z6MZ5un6DnflDtZ_d&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Bayes Error&lt;/code&gt;&lt;/a&gt;, whenever the distribution of classes overlap, the ratio of error is large. without changing the features, the Bayes error of the current distributions is the best performance and can not be reduced at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also suggest you reading &lt;a href=&quot;https://stats.stackexchange.com/questions/302900/what-is-bayes-error-in-machine-learning&quot;&gt;here&lt;/a&gt;. Problems with a large amount of Bayes error with appointed features are considered &lt;em&gt;not classifiable&lt;/em&gt; with in the space of those features. As another example you can suppose you want to classify cars with lights on. If you try to do that in the morning, you yourself may have lots of errors and if you use same images for training the learning algorithm, that may have too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I recommend you not to change the distribution of your classes. In such cases, the result of classifier near the boundary would be completely random. The distribution of data for training your machine learning algorithm should not be changed and should be as it is in the real condition.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T15:33:14.337" LastActivityDate="2018-01-17T15:33:14.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25414" PostTypeId="2" ParentId="25380" CreationDate="2017-12-05T15:36:36.373" Score="1" Body="&lt;p&gt;It uses &lt;strong&gt;same&lt;/strong&gt; padding which means the output of max-pooling is padded with zeros in a way that the output of next layer preserves the width and height. for information take a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/MmYe2/classic-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-05T16:17:53.530" LastActivityDate="2017-12-05T16:17:53.530" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25415" PostTypeId="2" ParentId="25411" CreationDate="2017-12-05T15:37:22.577" Score="2" Body="&lt;p&gt;Your are mixing two different beasts. Despite both having encoder and decoder parts, the way in which a normal feedforward image transformation network (i.e. the autoencoder) and an autoregressive model (i.e. the seq2seq) are actually used is very different:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In the type of image transformation done in a vanilla autoencoder, you input an image and get another image at the output. That's it, both at training and at inference time.&lt;/li&gt;&#xA;&lt;li&gt;In an autoregressive model like seq2seq, you feed the network with the input &lt;strong&gt;and the initial tokens of the expected output sequence&lt;/strong&gt; (i.e. the prefix), and you get the next token as output. During training, those initial tokens are normally from the &lt;em&gt;gold data&lt;/em&gt; (&lt;em&gt;aka&lt;/em&gt; teacher forcing), that is, you feed as prefix real tokens from the training data. In inference, those tokens are generated by the model itself: you generate the first token, then you generate the second token given the first one you previously generated, then you generate the third token given the previous 2 generated tokens, and so on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-01-29T12:49:16.493" LastActivityDate="2018-01-29T12:49:16.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25416" PostTypeId="1" AcceptedAnswerId="25456" CreationDate="2017-12-05T16:09:36.860" Score="1" ViewCount="3755" Body="&lt;p&gt;I am trying to apply word2vec/doc2vec to find similar sentences. First consider word2vec for word similarity. What I understand is, CBOW can be used to find most suitable word given a context, whereas Skip-gram is used to find the context given some word, so in both cases, I am getting words that co-occur frequently. But how does it work to find similar words? My intuition is, since similar words tend to occur in similar contexts, the words similarity is actually measured from the similarity among contextual/co-occuring words. In the neural net, when the vector representation for some word at the hidden layer is passed through to the output layer, it outputs probabilities of co-occuring words. So, the co-occuring words influence the vectors of some words, and since similar words have similar set of co-occuring words, their vector representations are also similar. To find the similarity, we need to extract the hidden layer weights (or vectors) for each word and measure their similarities. Do I understand it correctly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, what is a good way to find tweet text (full sentence) similarity using word2vec/doc2vec?&lt;/p&gt;&#xA;" OwnerUserId="31180" LastActivityDate="2018-12-10T05:07:08.193" Title="How word embedding work for word similarity?" Tags="|nlp|word2vec|word-embeddings|gensim|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25418" PostTypeId="1" CreationDate="2017-12-05T18:11:37.113" Score="1" ViewCount="39" Body="&lt;p&gt;Complete self taught novice here.&lt;/p&gt;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&lt;p&gt;Are there any popular standards, or commonly used formats for representing/saving/sharing/serializing a network?&lt;/p&gt;&#xA;&lt;h1&gt;Background&lt;/h1&gt;&#xA;&lt;p&gt;I built a simple but very large (5.5m nodes, 16.7m edges) network matrix using my own python code. But if I wanted to use this matrix with other data science tools what format should I use?&lt;/p&gt;&#xA;&lt;p&gt;Currently I have it in a giant csv file with headers and &amp;quot;1&amp;quot; representing edges. Which I think is called a binary matrix.&lt;/p&gt;&#xA;&lt;h1&gt;Goal&lt;/h1&gt;&#xA;&lt;p&gt;I want to make use of any benefits present in using binary files, or formatted flat files that would help me learn other tools.&lt;/p&gt;&#xA;" OwnerUserId="28542" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-10T15:42:27.287" Title="On Creating an Interoperable Network Matrix" Tags="|social-network-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25419" PostTypeId="1" AcceptedAnswerId="25428" CreationDate="2017-12-05T18:18:31.563" Score="1" ViewCount="1253" Body="&lt;p&gt;Has anyone here tried to predict a commodity's price by using other commodities prices as features in a machine learning algorithm? What techniques have been successful? &lt;/p&gt;&#xA;" OwnerUserId="17047" LastEditorUserId="1330" LastEditDate="2017-12-05T20:43:54.987" LastActivityDate="2018-01-05T02:05:06.220" Title="Using machine learning technique to predict commodity prices" Tags="|machine-learning|predictive-modeling|regression|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25420" PostTypeId="1" CreationDate="2017-12-05T18:23:08.927" Score="2" ViewCount="1538" Body="&lt;p&gt;l have a dataset of dim=(200,2000) 200 examples and 2000 features. l have 10 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;l used sklearn  for both cases :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svm.svc(kernel=linear)&#xA;LinearSVC()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However LinearSVC() performs drastically better than svm with linear kernel. 60% against 23%.&#xA;l'm supposed to get the same or comparable results since they are fed with same parameters and data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's wrong ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="42275" LastActivityDate="2017-12-06T15:58:10.427" Title="Why Liblinear performs drastically better than libsvm linear kernel?" Tags="|machine-learning|scikit-learn|svm|libsvm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25421" PostTypeId="2" ParentId="25403" CreationDate="2017-12-05T19:02:17.513" Score="2" Body="&lt;p&gt;It would help if you clarified what &quot;relevance&quot; and &quot;important&quot; mean, but you should take a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Term Frequency-Inverse Document Frequency&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;tf-idf weights words by the frequency of appearance within a document but penalizes words that appear across many documents. The concept is that words appearing many times in a single document are likely noteworthy, but they could also just be common words that appear frequently in all documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;tf-idf has implementations in most popular langauges including Python and R. &#xA;The Python Implementation is in Sci-kit learn and is called &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt;. It takes a list of filenames or a single document as input. You can also specifiy the stop words to apply to the documents and allows for tokenization. You probably also would want to apply a stemmer to your documents which can be done using Sci-kit learn. &#xA;If you prefer to work in R, you would want to use the &lt;a href=&quot;https://cran.r-project.org/web/packages/tm/tm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tm package&lt;/a&gt;. This package has numerous functions for text mining. &lt;/p&gt;&#xA;" OwnerUserId="42908" LastActivityDate="2017-12-05T19:02:17.513" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25423" PostTypeId="2" ParentId="23650" CreationDate="2017-12-05T19:11:54.353" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Because of this, more and more patients who would have been classified&#xA;  as high risk are actually not catching the disease and therefore&#xA;  decreasing the accuracy of your model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'd rather challenge the assumption and say the algorithm didn't miss in this case: it correctly identified a high risk case, which was its original purpose. You're not building a model to identify people who will catch the disease, but to prevent the disease of actually happening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Always think hard about which metrics make sense and what they represent. If you build a confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Predicted \ Actual | Caught the disease | Did not catch the disease&#xA;High risk          | A                  | B&#xA;Low risk           | C                  | D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, when you use accuracy, you're saying A and D are good, B and C are bad, all with the same weights. But is B really bad? Isn't C way worse than anything else? Or depending on the cost to prevent the disease, maybe B is worse? I don't know, it requires domain knowledge. But point being, using accuracy mindlessly is not a good way of dealing with this situation: &lt;strong&gt;adjust your metrics to your problem&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13748" LastActivityDate="2017-12-05T19:11:54.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25424" PostTypeId="2" ParentId="761" CreationDate="2017-12-05T20:47:03.757" Score="10" Body="&lt;p&gt;GPS coordinates can be directly converted to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Geohash&quot; rel=&quot;noreferrer&quot;&gt;geohash&lt;/a&gt;. Geohash divides the Earth into &quot;buckets&quot; of different size based on the number of digits (short Geohash codes create big areas and longer codes for smaller areas). Geohash is an adjustable precision clustering method.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-02-02T05:39:58.110" LastActivityDate="2018-02-02T05:39:58.110" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25425" PostTypeId="1" CreationDate="2017-12-05T21:16:20.347" Score="1" ViewCount="285" Body="&lt;p&gt;I am using a Kaggle dataset to learn more about using sound with Deep Learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have extracted the mfcc features using this cool library - librosa. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I want to try feature extraction using CNN's and then train on LSTM. &#xA;(Any advice here would be good, if I am on the right track)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I just put the raw wav file into CNN or should I do some data augmentation before that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Link to the &lt;a href=&quot;https://www.kaggle.com/rtatman/speech-accent-archive/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42917" LastActivityDate="2017-12-05T21:16:20.347" Title="speech accent recognition data augmentation and training" Tags="|deep-learning|lstm|audio-recognition|cnn|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25426" PostTypeId="1" AcceptedAnswerId="25457" CreationDate="2017-12-06T00:44:20.280" Score="1" ViewCount="1882" Body="&lt;p&gt;I am looking for a definition of Data Scout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what is the difference between Data Mining and Data Scout?&lt;/p&gt;&#xA;" OwnerUserId="42921" LastEditorUserId="40853" LastEditDate="2017-12-06T10:36:27.383" LastActivityDate="2017-12-06T19:57:19.740" Title="What is the definition of Data Scout?" Tags="|machine-learning|r|data-mining|data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25427" PostTypeId="1" CreationDate="2017-12-06T01:33:34.040" Score="1" ViewCount="173" Body="&lt;p&gt;I am trying to recreate a model based on its description. The model is described as a &quot;10 layers of size 500 ... 10&quot;. From my understanding, the size refers to the number of hidden units, which I could customize using model.add(dense)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential() &#xA;model.add(Conv2D(32, (3,3), activation='relu', input_shape=input_shape)&#xA;model.add(Dense(500, activation = 'relu')) &#xA;... &#xA;model.add(Dense(10, activation = 'softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it is yielding a very low accuracy rate (about 20% lower than what I was supposed to get). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I doing something wrong? &lt;/p&gt;&#xA;" OwnerUserId="42923" LastEditorUserId="29575" LastEditDate="2018-01-05T20:08:01.603" LastActivityDate="2018-01-05T20:08:01.603" Title="How to create a multiple layer perceptron with layers of specific sizes in keras?" Tags="|machine-learning|deep-learning|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25428" PostTypeId="2" ParentId="25419" CreationDate="2017-12-06T01:45:08.253" Score="3" Body="&lt;p&gt;Based on your question there are couple of things which I would assume to answer your question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As you need to predict the commodity price the data which is collected is time series data.&lt;/li&gt;&#xA;&lt;li&gt;Since you want to use other commodity to predict, it means that you don't have any past data of the product which you want to predict.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The answer could be derived by performing some exploratory analysis on the existing data. i.e., based on your business understanding you need to decide which product is similar to the new product. This kind of techniques is used to understand the sale of the new product/how is it going to perform after the launch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Techniques which can be used over here are Time Series Analysis like &lt;strong&gt;ARIMA&lt;/strong&gt;, if seasonality is present then &lt;strong&gt;SARIMA&lt;/strong&gt;, if no trend then &lt;strong&gt;Exponential Smoothing&lt;/strong&gt;(too many spikes),  there are other models like &lt;strong&gt;Auto Regression&lt;/strong&gt;, &lt;strong&gt;Moving Average&lt;/strong&gt;, &lt;strong&gt;Croston&lt;/strong&gt; if there are 0's etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one way of looking at your problem.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-06T01:45:08.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25429" PostTypeId="1" CreationDate="2017-12-06T02:40:17.500" Score="0" ViewCount="81" Body="&lt;p&gt;I have the following keywords retrieved from a text document.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;natural language processing&#xA;sequential labeling &#xA;programmable&#xA;spell checking&#xA;techniques&#xA;forensics &#xA;important issue &#xA;categorial grammar&#xA;girls&#xA;applications&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to remove unwanted words such as &lt;code&gt;programmable, techniques, important issue, girls, applications&lt;/code&gt; from this keyword list. Is there a way to automate this? Can I consider some pos-tag pattern to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me!&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2017-12-14T21:24:15.370" Title="Filter unwanted terms" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25430" PostTypeId="2" ParentId="25250" CreationDate="2017-12-06T02:51:00.467" Score="7" Body="&lt;p&gt;I think the analysis which you have done was good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the Survival Analysis procedure, I think using it in your scenario is good enough. Even it might take time but the results from that are good and very insightful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you have applied survival analysis on the data, you need to make sure that these assumptions are met:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There are several different ways to estimate a survival function or&#xA;a survival curve. There are a number of popular parametric methods&#xA;that are used to model survival data, and they differ in terms of&#xA;the assumptions that are made about the distribution of survival&#xA;times in the population. Some popular distributions include the&#xA;exponential, Weibull, Gompertz and log-normal distributions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Perhaps the most popular is the exponential distribution, which&#xA;    assumes that a participant's likelihood of suffering the event of&#xA;    interest is independent of how long that person has been event-free.&#xA;    Other distributions make different assumptions about the probability&#xA;    of an individual developing an event (i.e., it may increase,&#xA;    decrease or change over time). More details on parametric methods&#xA;    for survival analysis can be found in Hosmer and Lemeshow and Lee&#xA;    and Wang1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Here on two nonparametric methods, which make no assumptions about&#xA;how the probability that a person develops the event changes over&#xA;time. Using nonparametric methods, we estimate and plot the survival&#xA;distribution or the survival curve. Survival curves are often&#xA;plotted as step functions, as shown in the figure below. Time is&#xA;shown on the X-axis and survival (proportion of people at risk) is&#xA;shown on the Y-axis. Note that the percentage of participants&#xA;surviving does not always represent the percentage who are alive&#xA;(which assumes that the outcome of interest is death). &quot;Survival&quot;&#xA;can also refer to the proportion who are free of another outcome&#xA;event (e.g., percentage free of MI or cardiovascular disease), or it&#xA;can also represent the percentage who do not experience a healthy&#xA;outcome (e.g., cancer remission).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can go through this &lt;a href=&quot;http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Survival/BS704_Survival_print.html&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt; for better understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding &lt;strong&gt;Poisson Distribution&lt;/strong&gt;, Did you plot and check whether the data is following Poisson Distribution like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NGbkx.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NGbkx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Poisson distribution is an appropriate model if the following assumptions are true.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;k is the number of times an event occurs in an interval and k can&#xA;take values 0, 1, 2, ….&lt;/li&gt;&#xA;&lt;li&gt;The occurrence of one event does not affect the probability that a&#xA;second event will occur. That is, events occur independently.&lt;/li&gt;&#xA;&lt;li&gt;The rate at which events occur is constant. The rate cannot be higher&#xA;in some intervals and lower in other intervals.&lt;/li&gt;&#xA;&lt;li&gt;Two events cannot occur at exactly the same instant; instead, at each&#xA;very small sub-interval exactly one event either occurs or does not&#xA;occur.&lt;/li&gt;&#xA;&lt;li&gt;The probability of an event in a small sub-interval is proportional&#xA;to the length of the sub-interval.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The actual probability distribution is given by a binomial&#xA;distribution and the number of trials is sufficiently bigger than the&#xA;number of successes one is asking about&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If these conditions are met then you can use Poisson Model, go through this &lt;a href=&quot;https://en.wikipedia.org/wiki/Poisson_distribution&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implementation of this in &lt;a href=&quot;http://www.r-tutor.com/elementary-statistics/probability-distributions/poisson-distribution&quot; rel=&quot;noreferrer&quot;&gt;R&lt;/a&gt;, &lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.random.poisson.html&quot; rel=&quot;noreferrer&quot;&gt;Python&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, to address your 2 questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Your approach is Correct there is no problem with using that method, to improve your results you need to work on feature engg(deriving new variables).Since it you are considering the duration as a continuous variable(did you perform that log transformation, which you have stated in the beginning?) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In your scenario I think Survival and Poisson gives you better result, if you think these take more time then try getting sample of data and get your work done. If you are considering the out-come as a continuous variable then you can use Random Forest,XGBoost , all methods which are used for predicting a continuous variable(but if were you I would spend more time in fitting Survival and Poisson and then shift to other prediction techniques)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let me know if you have any issues!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-06T02:51:00.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25433" PostTypeId="2" ParentId="749" CreationDate="2017-12-06T05:10:57.983" Score="5" Body="&lt;p&gt;Another example, consider the case of users to movie rating matrix like the Netflix setup. This will be a huge sparse matrix which is difficult to process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that each user will have a specific preference like sci-fi movies or romance movies etc. So, instead of storing all the movie ratings we could store a single latent feature like the movie category which belongs to different &lt;strong&gt;Genre's&lt;/strong&gt; for example: sci-fi or romance, whichever quantifies his taste for each category. These are called &lt;strong&gt;Latent Features&lt;/strong&gt;, which captures the essence of his taste rather than storing the entire movie list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this will be an approximation, but on the flip-side, you have a very little to store.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is usually done using matrix decomposition techniques, like SVD which breaks an $N*N$ user to item recommendation matrix to $N*1$ user preference matrix and $1*N$ item preference matrix, added Advantage is that instead of storing $N^2$ number we effectively store $2N$.&lt;/p&gt;&#xA;" OwnerUserId="42928" LastEditorUserId="40853" LastEditDate="2017-12-07T06:12:02.333" LastActivityDate="2017-12-07T06:12:02.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25435" PostTypeId="2" ParentId="25407" CreationDate="2017-12-06T06:34:18.817" Score="5" Body="&lt;p&gt;Take a sample element from one class and a sample element from the other class. Is it possible for these two elements to have the exact same feature vector? If that can &lt;em&gt;ever&lt;/em&gt; happen then the two classes are not completely seperable using your current feature vectors (since the classification decision is based entirely in the feature vector for a given element).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if *every&quot; element in one class has a corresponding element in the other class such that the two elements have the same feature vectors, then the two classes are indistinguishable using your current feature vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, if that condition holds for only some of your elements and not others, then you are somewhere in between, and you can use that as a basis to measure how well you can hope a classifier to perform using your current feature set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these evaluations can be used to argue to varying degrees that you need to extract more features.&lt;/p&gt;&#xA;" OwnerDisplayName="user17471" LastActivityDate="2017-12-06T06:34:18.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25436" PostTypeId="2" ParentId="25427" CreationDate="2017-12-06T07:06:20.160" Score="1" Body="&lt;p&gt;There may be quite a lot of reasons for this. As I understand you try to repeat someone's results, and all you know is the architecture. Your keras architecture seems correct, but you should also take into account:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Data - did the authors used data augmentation? &lt;/li&gt;&#xA;&lt;li&gt;Regularizations, such as dropout, L1, L2, batch normalization - that can significantly influence your results.&lt;/li&gt;&#xA;&lt;li&gt;Other hyper-parameters, like optimizer, batch size etc. Did you do a grid/random search of those? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you took all this into account, provide us with more complete training code, so we don't have to guess.&lt;/p&gt;&#xA;" OwnerUserId="42903" LastActivityDate="2017-12-06T07:06:20.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25437" PostTypeId="1" CreationDate="2017-12-06T07:47:23.147" Score="0" ViewCount="216" Body="&lt;p&gt;Whether any machine learning model can dynamically predict different types of stages it would go through for completing a process.&lt;/p&gt;&#xA;" OwnerUserId="42933" LastEditorUserId="42933" LastEditDate="2017-12-07T05:28:21.927" LastActivityDate="2017-12-07T06:23:17.840" Title="How to predict a group of events using machine learning?" Tags="|machine-learning|azure-ml|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-07T22:56:33.163" ContentLicense="CC BY-SA 3.0" />
  <row Id="25439" PostTypeId="1" CreationDate="2017-12-06T08:58:46.353" Score="4" ViewCount="217" Body="&lt;p&gt;My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on how to monitor if it is still doing well:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Suppose Amazon would use Reinforcement Learning for optimizing the order of a search:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;An episode starts when a user starts a search&lt;/li&gt;&#xA;  &lt;li&gt;An episode ends after a threshold or when the user buys things.&lt;/li&gt;&#xA;  &lt;li&gt;The observation the agent gets is the search terms as well as 20 products for which he has to give an ordering.&lt;/li&gt;&#xA;  &lt;li&gt;He gets a reward if one of those 20 things is bought.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Of course, one can (should?) let the agent learn all the time as products change and probably search terms / language changes. But when do I know that the agent learned something weird / that I should stop it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can imagine the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Case-based single examples&lt;/li&gt;&#xA;&lt;li&gt;Having a ground-truth for some trivial searches + products and checking if the model does those right (in non-exploratory mode)&lt;/li&gt;&#xA;&lt;li&gt;Letting the agent learn in batch-mode (e.g. update the model only once a week) and doing A/B tests for the current model / new model.&lt;/li&gt;&#xA;&lt;li&gt;Measuring mean reward and setting a threshold. If the mean reward of the agent drops below the threshold, reset the agent to a past &quot;save&quot; state.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there literature about it? Blog posts?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know at least one example where RL went wrong / monitoring didn't quite work:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist&quot; rel=&quot;noreferrer&quot;&gt;Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day&lt;/a&gt; (&lt;a href=&quot;https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter&quot; rel=&quot;noreferrer&quot;&gt;The Guardian&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2017-12-10T17:15:16.000" Title="How can RL agents be monitored?" Tags="|reinforcement-learning|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25440" PostTypeId="2" ParentId="25437" CreationDate="2017-12-06T09:01:34.147" Score="2" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assumptions before answering your question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The Target variable is a categorical(each stage is considered as a separate category).&lt;/li&gt;&#xA;&lt;li&gt;You have the past data of the customers whose application got rejected(if hasn't reached the last stage)/successful(reach last stage) at the respective level. For Example, customer A applied and got his application rejected at stage-2, No Legal). &lt;/li&gt;&#xA;&lt;li&gt;You have some demographics of the customer to predict the stage at which the application might get rejected. &lt;/li&gt;&#xA;&lt;li&gt;You have good data(no outliers, well engineered features, good understanding on data etc).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now if the above assumptions are met, then you can apply Machine Learning algorithms which are used for doing multi-class classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applicable Algorithms:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Neural Networks&lt;/li&gt;&#xA;&lt;li&gt;Random Forest&lt;/li&gt;&#xA;&lt;li&gt;K nearest Neighbor&lt;/li&gt;&#xA;&lt;li&gt;Decision Tress&lt;/li&gt;&#xA;&lt;li&gt;Support Vector Machines etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finally, you can apply Ensemble model by combining results of the 2 or more models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For measuring the accuracy you can use Confusion Matrix(By diving the data into Test: 30% of data and Train: 70% of data).&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-07T06:23:17.840" LastActivityDate="2017-12-07T06:23:17.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25443" PostTypeId="1" CreationDate="2017-12-06T11:17:19.067" Score="7" ViewCount="21035" Body="&lt;p&gt;I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this &lt;a href=&quot;https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/&quot; rel=&quot;noreferrer&quot;&gt;blog&lt;/a&gt;, and not really understanding how the quantile parameters affect the model (I am using the suggested parameter values on the blog). &#xA;When I apply this code to my data, I obtain nonsense results, such as negative predictions for my target values while my target values are always over 10K. &#xA;I don't understand how should this code vary according to my data and would really appreciate any help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differences in my data to the data that is used on the blog are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;My distribution is Poisson like. &lt;/li&gt;&#xA;&lt;li&gt;I have over 100 features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note: I tried tuning the delta, threshold and var parameters, but they don't seem to have a controllable effect on the results and predictions remains nonsense. &lt;/p&gt;&#xA;" OwnerUserId="40029" LastActivityDate="2022-10-05T08:18:22.230" Title="Prediction Intervals Using XGBoost" Tags="|regression|xgboost|prediction|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25444" PostTypeId="1" CreationDate="2017-12-06T11:43:02.803" Score="6" ViewCount="1149" Body="&lt;p&gt;What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do they perform better than non-monotonic ones?&lt;/li&gt;&#xA;&lt;li&gt;Is this mathematically proven?&lt;/li&gt;&#xA;&lt;li&gt;Are there any papers/references that are related to this?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42945" LastEditorUserId="42945" LastEditDate="2017-12-07T00:15:23.710" LastActivityDate="2017-12-07T00:15:23.710" Title="Advantages of monotonic activation functions over non-monotonic functions in neural networks?" Tags="|machine-learning|neural-network|deep-learning|activation-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25446" PostTypeId="2" ParentId="25444" CreationDate="2017-12-06T13:16:34.850" Score="4" Body="&lt;p&gt;I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian kernel, symmetric around $x=0$ but slides off towards $f(x)=0$ if x strays away from 0 on either side. If we have a sample that we feed into our network that performs poorly when our activation is high, we want to change the input of our node to give a lower activation. In case of a non-monotonic activation, whether we want to decrease or increase the input depends on whether the input was positive or negative, and is mostly dependent on our weight initialization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes learning more difficult, because if another sample also  needs it to be lower but was on the other side of the top, backpropogation will attempt to map the input to the other side. Most of the time the best solution will be to put everything on one side of the top, making it monotonic again. Another way of looking at it is that monotonic are somewhat one-to-one (not entirely true, for example ReLU). This means that two very different inputs don't map to the same output unless everything in between also maps there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here was a similar question with some links: &lt;a href=&quot;https://datascience.stackexchange.com/questions/9233/why-do-activation-functions-have-to-be-monotonic?rq=1&quot;&gt;(Why) do activation functions have to be monotonic?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-12-06T13:16:34.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25447" PostTypeId="2" ParentId="25444" CreationDate="2017-12-06T13:26:32.343" Score="1" Body="&lt;p&gt;In addition to computational reasons, you can read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Biological_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;biological neural networks&lt;/a&gt; to know this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In neuroscience, a biological neural network is a series of interconnected neurons whose activation defines a recognizable linear pathway.The interface through which neurons interact with their neighbors usually consists of several axon terminals connected via synapses to dendrites on other neurons. &lt;strong&gt;If the sum of the input signals into one neuron surpasses a certain threshold, the neuron sends an action potential (AP) at the axon hillock and transmits this electrical signal along the axon.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As activation of a neuron depends on the sum of inputs, biologically makes sense that activation function be an increasing function.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-12-06T13:26:32.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25448" PostTypeId="2" ParentId="25426" CreationDate="2017-12-06T13:47:38.507" Score="1" Body="&lt;p&gt;Copying from the abstract I found &lt;a href=&quot;https://arxiv.org/abs/1708.06925&quot; rel=&quot;nofollow noreferrer&quot;&gt;here - read the paper for more insights&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In the year 2011, the CMS collaboration introduced the novel concept&#xA;of data scouting, &lt;strong&gt;allowing to take data that otherwise would be&#xA;rejected by the usual trigger filters&lt;/strong&gt;. This special data flow, based&#xA;on event-size reduction, was created to maintain sensitivity to new&#xA;light resonances decaying to jets or muons, with very small online and&#xA;offline resources allocated to it. The challenges implied by this new&#xA;workflow and the solutions developed within the CMS experiment are&#xA;shown. This technique is now a standard ingredient for CMS data-taking&#xA;strategy. The present status of data scouting in CMS is presented.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I do not come from a Physics background, and hence I could not completely understand the context while reading the paper in the larger scheme of things. But atleast this gives you a heads up about what it is.&lt;/p&gt;&#xA;&lt;p&gt;Coming to Data Mining, there are plenty of definitions you can find. My definition would be using raw data to find patterns or insights from large datasets. This could be done with the use of many algorithms from math, computer science or statistics.&lt;/p&gt;&#xA;&lt;p&gt;Some data mining algorithms are k-nearestneighbor, k-means, naive bayes, and so on.&lt;/p&gt;&#xA;&lt;p&gt;So what is the difference between data scouting and mining? In mining, you consider raw datasets and perform some operations. In scouting, if I am not wrong, you take the data discarded by some filters (do not know what these filters are) while performing some operation, and then considering that data to perform additional operations.&lt;/p&gt;&#xA;" OwnerUserId="16401" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-06T13:47:38.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25450" PostTypeId="1" CreationDate="2017-12-06T13:59:48.030" Score="2" ViewCount="1351" Body="&lt;p&gt;I want to freeze the layers, except the first three layers, in the &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/research/inception&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inception v3 model&lt;/a&gt; with TensorFlow in Python 3, and modify the weights of these three layers to be able to re-initialize and re-train only the three first layers of the network.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this can't be done within the inception model, is there any other network (in TensorFlow) with which this could be done?  &lt;/p&gt;&#xA;" OwnerUserId="42951" LastEditorUserId="18375" LastEditDate="2017-12-11T17:00:18.400" LastActivityDate="2018-01-10T17:19:19.173" Title="Transfer learning within Tensorflow's inception model" Tags="|machine-learning|python|tensorflow|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25452" PostTypeId="2" ParentId="25420" CreationDate="2017-12-06T15:58:10.427" Score="2" Body="&lt;p&gt;Here is just a guess, but according to me, the linearSVC might perfoms better than SVM with linear kernel because of regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because linearSVC is based on liblinear rather than libsvm, it has more flexibility and it gives you the possibility to use regularization with your SVM (default is L2-Ridge regularization).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because you have more features than observations, it exists multiple solutions to your classification model. Some of them are more &quot;robust&quot; than other ones. L2 regularization will help you reduce the ampltitude of your model coefficients and maybe lead to a more stable solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More on LinearSVC in the documentation : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More on Ridge regression benefits when working with a lot of features (and so multicollinearity) : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago&quot;&gt;https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-06T15:58:10.427" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25453" PostTypeId="2" ParentId="25443" CreationDate="2017-12-06T17:06:46.660" Score="0" Body="&lt;p&gt;Try the following code. It must work fine. It might take a lot of time (more than 100 features).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change max_depth to 6 if you want more accuracy. (because of 100 features.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can change learning_rate between 0 and 1, to improve the efficiency.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xgb&#xA;model_xgb = xgb.XGBRegressor(colsample_bytree=0.4603, gamma=0.0468, &#xA;                             learning_rate=0.05, max_depth=3, &#xA;                             min_child_weight=1.7817, n_estimators=4200,&#xA;                             reg_alpha=0.4640, reg_lambda=0.8571,&#xA;                             subsample=0.5213, silent=1,&#xA;                             nthread=-1)&#xA;&#xA;X_train, X_test, Y_train, Y_test= train_test_split(X, Y, random_state= 0)&#xA;def model_score_error(model):&#xA;    prepared_model=model.fit(X_train, Y_train)&#xA;    x=prepared_model.score(X_test,Y_test)&#xA;    print('Score: ',x)&#xA;    Target_predicted=prepared_model.predict(X_test) &#xA;    MSE=mean_squared_error(Y_test,Target_predicted) &#xA;    print('mean square error', MSE)&#xA;&#xA;model_score_error(model_xgb)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42488" LastEditorUserId="42488" LastEditDate="2017-12-06T17:41:49.127" LastActivityDate="2017-12-06T17:41:49.127" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25455" PostTypeId="1" CreationDate="2017-12-06T17:25:21.957" Score="1" ViewCount="292" Body="&lt;p&gt;I have a dataframe, df, that is of the format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dimension, 1, 2, ..., 55&#xA;&lt;br&gt;&#xA;Dim1, 0.22, 0.152, ..., 0.10&#xA;&lt;br&gt;&#xA;...&#xA;&lt;br&gt;&#xA;Dim20, 0.52, 0.89, ..., 0.09&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I have a training set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;participant_1 :{Dimension: Dim12, response: {1,5,24,6,12}}&#xA;participant_2 :{Dimension: Dim5, response: {1,5,24,30,32}}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find weights w_1, w_2, ..., w_55 such that I can weight the columns of df corresponding to a participants response est, add the columns together and the row with the highest resulting value should be the participants dimension value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best method to find the set of weights in Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also looking for advice on how to format questions here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!!&lt;/p&gt;&#xA;" OwnerUserId="42958" LastActivityDate="2017-12-06T17:25:21.957" Title="How Do I Perform Gradient Descent for Discrete Predictions" Tags="|machine-learning|python|pandas|linear-regression|gradient-descent|" AnswerCount="0" CommentCount="1" ClosedDate="2017-12-08T03:28:39.660" ContentLicense="CC BY-SA 3.0" />
  <row Id="25456" PostTypeId="2" ParentId="25416" CreationDate="2017-12-06T19:12:52.303" Score="2" Body="&lt;p&gt;I think you have it mostly correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word embeddings can be summed up by: &lt;em&gt;A word is known by the company it keeps&lt;/em&gt;. You either predict the word given the context or vice versa. In either case similarity of word vectors is similarity in terms of replaceability. i.e. if two words are similar one could replace the other in the same context. Note that this means that &quot;hot&quot; and &quot;cold&quot; are (or might be) similar within this context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use word embeddings for a similarity measure of tweets there are a couple approaches you can take. One is to compute paragraph vectors (AKA doc2vec) on the corpus, treating each tweet as a separate document. (There are good examples of running doc2vec on Gensim on the web.) An alternate approach is to AVERAGE the individual word vectors from within each tweet, thus representing each document as an average of its word2vec vectors. There are a number of other issues involved in optimizing similarity on tweet text (normalizing text, etc) but that is a different topic.&lt;/p&gt;&#xA;" OwnerUserId="14752" LastActivityDate="2017-12-06T19:12:52.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25457" PostTypeId="2" ParentId="25426" CreationDate="2017-12-06T19:57:19.740" Score="1" Body="&lt;p&gt;In business, there is little time to look through the data that has been eliminated before a specific analysis project so I put that data in an elim_bin (project code#, project date) I ALWAYS attach data source &amp;amp; date pulled for each eliminated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later, I can &quot;scout&quot; that data with an algorithm to see if something could be useful/learned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I use &quot;scout&quot; data. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="42961" LastActivityDate="2017-12-06T19:57:19.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25459" PostTypeId="2" ParentId="23328" CreationDate="2017-12-06T21:39:36.147" Score="-1" Body="&lt;p&gt;If I am getting it correctly, you are facing a problem in training an LSTM for a multivariate timeseries dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem is called as the multivariate time-series problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/time-series-forecasting-supervised-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is a tutorial to learn it quickly&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42940" LastEditorUserId="52089" LastEditDate="2018-06-28T12:38:18.377" LastActivityDate="2018-06-28T12:38:18.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25461" PostTypeId="1" CreationDate="2017-12-06T23:14:36.077" Score="1" ViewCount="68" Body="&lt;p&gt;For example&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Shoes priced at \$10, 30 of 1000 customers buy for revenue of \$300&lt;/li&gt;&#xA;&lt;li&gt;Shoes priced at \$20, 25 of 1000 customers buy for revenue of \$500&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And then now I have the problem of determining how certain I am the second pricing is yields higher returns&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: How can I find out X so that I'm X% confident that the \$20 price yields higher returns than the \$10?&lt;/p&gt;&#xA;" OwnerUserId="42969" LastActivityDate="2017-12-07T04:01:29.783" Title="Hill climbing with noisy data" Tags="|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25462" PostTypeId="2" ParentId="19160" CreationDate="2017-12-07T00:08:39.757" Score="1" Body="&lt;p&gt;My guess is that the dataset is very small. Word2Vec won't be able to capture word relationship in the embedding space with limited information. Try to train word2vec on a very large corpus to get a very good word vector before training your classifier might help.   &lt;/p&gt;&#xA;" OwnerUserId="25531" LastActivityDate="2017-12-07T00:08:39.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25463" PostTypeId="1" CreationDate="2017-12-07T02:01:44.953" Score="10" ViewCount="9507" Body="&lt;p&gt;I'm trying to learn how &lt;code&gt;LSTM&lt;/code&gt; networks work, and even if I get the basics, the details of the internal structure is not clear for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On this blog &lt;a href=&quot;https://jasdeep06.github.io/posts/Understanding-LSTM-in-Tensorflow-MNIST/&quot; rel=&quot;noreferrer&quot;&gt;link&lt;/a&gt;, I found this scheme of a &lt;code&gt;LSTM&lt;/code&gt; architecture&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://github.com/jasdeep06/jasdeep06.github.io/blob/master/posts/Understanding-LSTM-in-Tensorflow-MNIST/images/num_units.png?raw=True&quot; alt=&quot;LSTM&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where apparently, every circle should correspond to an individual &lt;code&gt;LSTM&lt;/code&gt; unit like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://github.com/jasdeep06/jasdeep06.github.io/blob/master/posts/Understanding-LSTM-in-Tensorflow-MNIST/images/lstm_unit.png?raw=True&quot; alt=&quot;LSTM&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is this correct?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is each unit in the cell independent from the others? Or do they share information?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Imagine I have the following configuration:&#xA;Number of samples = 1000&#xA;Number of time-steps = 10&#xA;Number of features = 5&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In this case, each unit in a cell will take as input a vector of size 5 right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what will be the size of the output for one unit? 1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="42974" LastEditorUserId="28175" LastEditDate="2018-03-08T07:26:13.107" LastActivityDate="2018-03-08T12:16:29.890" Title="Questions about LSTM cells, units and inputs" Tags="|machine-learning|neural-network|deep-learning|lstm|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25464" PostTypeId="2" ParentId="19160" CreationDate="2017-12-07T02:43:37.273" Score="1" Body="&lt;p&gt;I assume that you have knowledge about how things work in complex models(RandomForestClassifier, ExtraTreesClassifier), not talking much about how they work. I would explain it with reference to project which I worked on &#xA;Twitter Sentiment Mining using Real-Time data&#xA;&lt;em&gt;Topic&lt;/em&gt;: Samsung Galaxy S8&#xA;&lt;em&gt;Outcome&lt;/em&gt;: How much percent of tweets are Positive and Negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;possible reasons&lt;/em&gt; for the &lt;strong&gt;model to perform bad&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/25531/unaki&quot;&gt;unaki&lt;/a&gt; has suggested you need to try increasing the corpus&#xA;size, I want to add one more point to it i.e., for any model to&#xA;perform well you need to have a &lt;strong&gt;huge, quality corpus&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my scenario for predicting the sentiment of tweets(real-time tweets) I used 1.5 GB of Tweets which where segregated manually as Positive or Negative. I used that for training my model and I could get an accuracy of 90%(out of 10 tweets 9 were segregated correctly). It was time consuming process but it helped me to get good outcome.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;As you have mentioned that you have some data which is not in English, make sure that these are handled properly before giving to model i.e., if you don't want such words get rid of them or train your model to understand those other language words.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In my Scenario, I had to remove all the Non-English as my model wasn't trained to handle such words. by which my model performed well. There was an increase of 2-3% of accuracy, which was significant for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To &lt;em&gt;Improve&lt;/em&gt; these &lt;strong&gt;Models Accuracy&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, to improve my model I made an new dictionary for some words like &lt;em&gt;not great&lt;/em&gt; - which falls under a &lt;strong&gt;Neutral Sentiment&lt;/strong&gt;(but I don't have one), If these words are found then that tweet is pushed to negative. By this you can expect better segregation of sentiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the same way to improve you need to look for things which are not classified properly(wrongly classified by models- False Positive or False Negative these terms are with respect to Confusion Matrix) and find a pattern which is being missed by the model and use that for maximizing your models capacity.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-07T02:43:37.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25466" PostTypeId="2" ParentId="25461" CreationDate="2017-12-07T04:01:29.783" Score="2" Body="&lt;p&gt;When in doubt, bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Make a list with 970 0s (the non-sales at \$10) and 30 10s (the sales at \$10) - &lt;code&gt;customers_10&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Make a list with 975 0s (the non-sales at \$20) and 25 20s (the sales at \$20) - &lt;code&gt;customers_20&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Repeat lots of times (maybe N = 100,000 times - increase this until the the result settles down)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Randomly take 1000 samples (with replacement) from both lists - &lt;code&gt;samples_10&lt;/code&gt;, &lt;code&gt;samples_20&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Sum the samples for each list - &lt;code&gt;sum_s10&lt;/code&gt;, &lt;code&gt;sum_s20&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Record which one generated more revenue (i.e. whether &lt;code&gt;sum_20 &amp;gt; sum_10&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;The percentage of times sum_s20 was greater than sum_s10 is your X&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(Another approach, which would be equivalent in the limit of N large, would be to model this as a binomial process. But processing is cheap, bootstrapping is robust and easy to implement - getting the binomial model right would be easier to mess up).&lt;/p&gt;&#xA;" OwnerUserId="42975" LastActivityDate="2017-12-07T04:01:29.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25467" PostTypeId="1" CreationDate="2017-12-07T06:46:01.720" Score="9" ViewCount="12874" Body="&lt;p&gt;I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I compare the results of a GBM model (using python xgboost and light GBM) and I found that it doesn't matter what are the hyper-parameters of the model the 'thinner' version leads to better results (AUC) even that all 20 original variables are included in the wider version.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when I compare the same using Lasso model - the wider version is better (~1% higher) as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess it can be related to to the randomness in the GBM but I was surprised to see that GBM doesn't fix it along the way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any explanation of the phenomena will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="40176" LastEditorUserId="303" LastEditDate="2020-03-14T08:54:14.180" LastActivityDate="2020-03-14T08:54:14.180" Title="Adding feature leads to worse results" Tags="|xgboost|feature-engineering|model-selection|gbm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25468" PostTypeId="2" ParentId="25467" CreationDate="2017-12-07T07:25:30.803" Score="3" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand your question correctly you want to know why a model would perform worse when a new feature is added?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So every time you do feature engineering (add new columns, derive columns, standardize the data, normalize the data, etc) there is always a flip side of the coin. If you add some features and if those features explain something about the target variable then it would aid in increasing the accuracy; on the other hand, if the feature doesn't have much relation to the target variable then it doesn't aid in increasing the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now before going to modeling you can go through a couple of things like (assuming that you have done all these things):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Eliminating Unnecessary Features using Business Understanding&lt;/li&gt;&#xA;&lt;li&gt;Removing Outliers&lt;/li&gt;&#xA;&lt;li&gt;Imputing Missing Data (XGBoost is not prone to missing values)&lt;/li&gt;&#xA;&lt;li&gt;Standardizing data&lt;/li&gt;&#xA;&lt;li&gt;Correlation analysis with the target variable and within the variables too, eliminate if there is high correlation between variables (as you need independent variables), eliminate the variables which are not much related to the target variable.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Once the above steps are done you can get a model which explains the data well. To improve the accuracy you need to do more feature engineering and try to find if there are some external factors which might effect your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many reasons why a model doesn't work well. Some of the above stated might be the reason why your model is not performing well in your scenario. Do have a closer look on the data -- this might give you a better idea. This is a top level view.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-08T00:56:19.673" LastActivityDate="2017-12-08T00:56:19.673" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="25469" PostTypeId="2" ParentId="13669" CreationDate="2017-12-07T07:58:23.640" Score="1" Body="&lt;p&gt;I am not sure if it is possible to export only one cell in &lt;em&gt;Jupyter&lt;/em&gt; notebooks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I do know that you could download as a python file and simply copy paste the particular cell you want in your new &lt;em&gt;Jupyter&lt;/em&gt; notebook. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you need any more help!&lt;/p&gt;&#xA;" OwnerUserId="42985" LastEditorUserId="40853" LastEditDate="2017-12-07T08:30:24.680" LastActivityDate="2017-12-07T08:30:24.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25470" PostTypeId="1" CreationDate="2017-12-07T08:30:49.110" Score="0" ViewCount="1900" Body="&lt;p&gt;In many ptb/mini shakespeare  LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: &lt;a href=&quot;https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html&lt;/a&gt;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;During training, these tutorials manually feed back the hidden state of LSTM, like codes below. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = 32&#xA;hidden_state_in = cell.zero_state(batch_size, tf.float32) &#xA;output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)&#xA;...&#xA;#For loop used in training: ...&#xA;    output, hidden_state = sess.run([output, hidden_state_out],&#xA;                            feed_dict={hidden_state_in:hidden_state})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):&#xA;    with tf.Session() as sess:&#xA;        sess.run(tf.initialize_all_variables())&#xA;        g['saver'].restore(sess, checkpoint)&#xA;        state = None&#xA;        current_char = vocab_to_idx[prompt] &#xA;        chars = [current_char]&#xA;&#xA;        for i in range(num_chars):&#xA;             if state is not None:&#xA;                feed_dict={g['x']: [[current_char]], g['init_state']: state}&#xA;                #shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1 &#xA;            else:&#xA;                feed_dict={g['x']: [[current_char]]}&#xA;            preds, state = sess.run([g['preds'],g['final_state']], feed_dict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When using dynamic_rnn with LSTM, the hidden state has shape of  [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?&lt;/p&gt;&#xA;" OwnerUserId="25515" LastActivityDate="2018-05-03T08:30:49.173" Title="Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow)" Tags="|tensorflow|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25476" PostTypeId="2" ParentId="16623" CreationDate="2017-12-07T10:42:24.773" Score="1" Body="&lt;p&gt;Why not just keep training your old model?&#xA;just pass the previous xgb model (path + filename) as an argument in the train function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bst = xgb.train(param0, dtrain, num_round, evals=[(dtrain, &quot;training&quot;)], xgb_model=prev_xgb_model)&#xA;&#xA;xgb_model=prev_xgb_model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42613" LastActivityDate="2017-12-07T10:42:24.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25477" PostTypeId="1" AcceptedAnswerId="25719" CreationDate="2017-12-07T11:00:25.460" Score="0" ViewCount="160" Body="&lt;p&gt;when I was read the paper about SMO　for SVM.&#xA;&lt;a href=&quot;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;John C. Platt paper&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;choosing the alpha which violated KKT condition is very confusing me. I do not very understand the code as below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if ((r2 &amp;lt; -tol &amp;amp;&amp;amp; alph2 &amp;lt; C) || (r2 &amp;gt; tol &amp;amp;&amp;amp; alph2 &amp;gt; 0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;why does it choose r2 &amp;lt; -tol when alph2 &amp;lt; C, not r2 &amp;lt; tol?&lt;/p&gt;&#xA;" OwnerUserId="42501" LastActivityDate="2017-12-16T08:40:49.183" Title="svm KKT condition violation that alpha choice in SMO" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25478" PostTypeId="2" ParentId="11678" CreationDate="2017-12-07T12:23:27.213" Score="1" Body="&lt;p&gt;Spark will automatically un-persist/clean the RDD or Dataframe if the RDD is not used any longer. &#xA;To check if a RDD is cached, please check into the Spark UI and check the Storage tab and look into the Memory details. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the terminal, you can use &lt;code&gt;rdd.unpersist()&lt;/code&gt; or &lt;code&gt;sqlContext.uncacheTable(&quot;sparktable&quot;)&lt;/code&gt; to &#xA;remove the RDD or tables from Memory. Spark made for Lazy Evaluation, unless and until you say any action, &#xA;it does not load or process any data into the RDD or DataFrame.&lt;/p&gt;&#xA;" OwnerUserId="3089" LastEditorUserId="24000" LastEditDate="2017-12-08T03:22:26.990" LastActivityDate="2017-12-08T03:22:26.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25479" PostTypeId="1" CreationDate="2017-12-07T13:37:45.230" Score="2" ViewCount="2870" Body="&lt;p&gt;I build a classification model on keras (Vanilla MLP),It went quite well by using a tutorial.But what are the default weights of layers because I didn't mention any in the code.It has a feature of intializing weights, but what if we don't initialize.&lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="18375" LastEditDate="2017-12-09T10:27:22.263" LastActivityDate="2017-12-09T10:27:22.263" Title="What are default keras layer weights" Tags="|classification|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25480" PostTypeId="2" ParentId="25479" CreationDate="2017-12-07T14:34:31.770" Score="2" Body="&lt;p&gt;It uses &lt;a href=&quot;http://deepdish.io/2015/02/24/network-initialization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Xavier initialization&lt;/a&gt;. If you don't initialize it, by default it uses this method for initialization. If you want to know how to perform that you can use &lt;a href=&quot;https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. In the first link at the end, link of the paper is also available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;This Xavier initialization (after Glorot’s first name) is a neat trick that works well in practice. However, along came rectified linear units (ReLU), a non-linearity that is scale-invariant around 0 and does not saturate at large input values. This seemingly solved both of the problems the sigmoid function had.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-07T16:35:30.347" LastActivityDate="2017-12-07T16:35:30.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25481" PostTypeId="1" CreationDate="2017-12-07T14:59:40.197" Score="2" ViewCount="45" Body="&lt;p&gt;I am a beginner and need further help to elaborate following problem. I want to &amp;quot;read&amp;quot; text from images. In my problem setting, the text on all the images has the same structure. I found this &lt;a href=&quot;https://www.tensorflow.org/mobile/mobile_intro#optical_character_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;solution&lt;/a&gt; by googling about &amp;quot;optical character recognition&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;The link mentions that I need an object localization algorithm. How can I implement this in tensorflow?&lt;/p&gt;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="32963" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-08T14:58:58.533" Title="Text recognition within images" Tags="|deep-learning|tensorflow|beginner|" AnswerCount="0" CommentCount="0" ClosedDate="2017-12-08T01:01:32.253" ContentLicense="CC BY-SA 3.0" />
  <row Id="25482" PostTypeId="1" AcceptedAnswerId="25528" CreationDate="2017-12-07T15:19:21.847" Score="2" ViewCount="177" Body="&lt;p&gt;I have a collection of 1000s of bottles of wine. I want to understand what could / likely to be driving the price either up or down. Below is an example of the data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Wine      Country       Area        Grape        Class    Year    Price&#xA;A         France        Burgundy    Pinot Noir   Grand    2014    +6%&#xA;B         France        Burgundy    Chardonnay   1er      2014    -1%&#xA;C         France        Burgundy    Pinot Noir   1er      2014    +4%&#xA;D         USA           California  Pinot Noir            2013    +1%&#xA;E         USA           California  Blend                 2014    +0.5%&#xA;F         USA           California  Chardonnay            2014    -4%&#xA;G         USA           California  Chardonnay            2013    -5%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From these fictitious numbers i would assume the reason bottle A is up is because pinot is up but more importantly France is up V USA wine. To the same extent Chardonnay is doing poorly but USA again is doing worse than France. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here this is rather easy but if this was extended by 1000s of rows with more categories added, the exercise becomes less easy to see the trend. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original thought would be to use the importance of a feature from a decision tree. But is there a better way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks &lt;/p&gt;&#xA;" OwnerUserId="24259" LastActivityDate="2017-12-10T07:04:42.083" Title="Understanding what is going on" Tags="|machine-learning|data-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25483" PostTypeId="2" ParentId="25463" CreationDate="2017-12-07T16:00:17.137" Score="1" Body="&lt;p&gt;I think your image is mislabeled.  I think each blue box is an LSTM layer, composed of multiple cells/units, each of which accepts a vector input x_t.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that, the answers to your questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Yes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Yes, they are independent (at a single time step - they share information with each other between time steps)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Yes, each unit cell will take an input of size 5. I &lt;em&gt;think&lt;/em&gt; the output size is always 1, similar to neural network nodes (like sigmoidal units) that combine and then activate.  &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-12-07T16:00:17.137" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25484" PostTypeId="2" ParentId="25482" CreationDate="2017-12-07T16:08:27.957" Score="1" Body="&lt;p&gt;Looks like all of your features are categorical. Just for a quick look. Try box blots of price for each variable by each category-class. If for some variable you start seeing a clear differences in the boxes (Hight of the box and the whiskers). That will help you claim that feature's importance analytically. Else doing a decision-tree and looking at feature importance seems like a good solution.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2017-12-07T16:08:27.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25485" PostTypeId="1" AcceptedAnswerId="25486" CreationDate="2017-12-07T21:21:13.643" Score="7" ViewCount="5476" Body="&lt;p&gt;All the techniques/models that I have learnt so far for deep learning start with some sort of normalization to the features, for example gaussian method, minmax scaling, robust scaling, batch normalization, instance normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any techniques to run neural networks without normalization so that the network can see (in absolute values) the magnitude of the value and respond according to that instead of normalized values? Will there be exploding/vanishing gradient issues if I don't normalize my data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I am training a custom LSTM network for multivariate time series data, the input dimension for a feature vector &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; is all the values from &lt;span class=&quot;math-container&quot;&gt;$t-n$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; is the number of time steps and the output vector is the value at &lt;span class=&quot;math-container&quot;&gt;$t+1$&lt;/span&gt;. Is there any need for normalization in this case?&lt;/p&gt;&#xA;" OwnerUserId="40454" LastEditorDisplayName="user10640" LastEditDate="2020-01-07T18:41:54.650" LastActivityDate="2020-01-07T18:41:54.650" Title="Are neural networks able to deal with non-normalised inputs?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25486" PostTypeId="2" ParentId="25485" CreationDate="2017-12-07T21:59:39.897" Score="6" Body="&lt;p&gt;Normalization helps to eliminate scale factors that might exist between variables in your data. Take, for example, the classic problem of predicting home prices. If you represent the square footage of your home in square millimeters, a large change in this value will have a relatively small effect on home price, implying a small gradient on this variable. If you represent that value in square kilometers, a small numerical change will have a large impact on price, implying a large gradient. Normalization isn't necessarily required, but can help to balance the problem by making all variables have &quot;equal weight&quot; in your model. If you were to include both the square millimeter and square kilometer variables in your training data, the neural network would likely spend a lot of effort optimizing on the square kilometer variable, since it is numerically more important. You can still do training with un-normalized data, but it will likely take longer, and possibly have worse output if your important variables are numerically smallest.&lt;/p&gt;&#xA;" OwnerUserId="37184" LastActivityDate="2017-12-07T21:59:39.897" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25490" PostTypeId="2" ParentId="25399" CreationDate="2017-12-08T03:36:22.817" Score="0" Body="&lt;p&gt;I am not very sure what do you mean as input. In Supervised Learning the learning signal comes from the difference between true response and model's response (&quot;teacher's supervision&quot;). In Reinforcement Learning the learning signal comes from the reward which might come delayed, sometimes not at all in that particular trial (but in another yes) etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep-Q learning which basically is Q-learning with function approximation is a Model-free RL. It means that at the end you want your system to learn a mapping between stimulus and response. Think of it as a reflex elicited by a stimulus. The reward cannot be the input to your system as it is your learning signal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are referring to the experience replay, as I mentioned to you, the reward sometimes doesn't come &quot;on time&quot;. So we need to decorrelate states,actions and sequences and that's why we don't update the network at every single time step. Instead we prefer to build a buffer with experience and sample from that. So as you stated, you want to learn to avoid enemies not locations and for this if you sample experience from the buffer the network's training will be more &quot;intuitive&quot;.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2017-12-08T03:36:22.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25491" PostTypeId="1" CreationDate="2017-12-08T03:47:58.000" Score="3" ViewCount="143" Body="&lt;p&gt;In this &lt;a href=&quot;https://arxiv.org/pdf/1605.06065.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Deep-Mind on one shot learning they have published an architecture explaining how the system works with an external meory. I understand the mechanism perfectly. But what I don't understand is how they feed the data in to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S9UrS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S9UrS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In  this part of the paper they explain as follows ,&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mmKXu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mmKXu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this part I actually don't understand is do do the concatenate inputs with labels ? And what is label shuffling ? &lt;/p&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2017-12-08T03:47:58.000" Title="How to feed the input to a Memory Augmented Neural Network (MAAN) to do one shot learning?" Tags="|deep-learning|meta-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25492" PostTypeId="2" ParentId="25406" CreationDate="2017-12-08T03:55:00.750" Score="3" Body="&lt;p&gt;There are many papers out there that deal with neural networks and RL. This blog will give a very good insight on a Policy Gradient Network: &lt;a href=&quot;http://karpathy.github.io/2016/05/31/rl/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep RL with PG&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for your question. You really need to be familiar on how we train a neural network. A simple one for classification. If you check the derivations and how the weights are getting updated things will be very clear to you on how you can implement the above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will describe it to you as simple as possible so you get the link. A Neural Network in a very broad sense consists of nested functions. The function that contains all the others is the one at your output layer. In case of Stochastic Policy Gradients this is your Boltzmann function. So your output layer will take all the previous layer's outputs and will pass them through the Boltzmann function. In the case of NN the parametrization comes from all previous layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blog link I sent you describes a very nice and simple example of a vanilla Policy Gradient with NN (REINFORCE algorithm). By using the code (plus your understanding on feedforward networks) you will see that the gradients are multiplied by the reward. It is a very good exercise!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Actor-Critic, you need in general a network performing PG (stochastic or deterministic) -- you Actor -- and a network that will give you the reward signal (like the simple case in the blog. However, for various reasons, instead of the actual reward we use another network that estimates the reward by performing Q-learning as in Deep-Q learning (minimizing square error between estimated reward and true reward).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2017-12-11T18:31:38.680" LastActivityDate="2017-12-11T18:31:38.680" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25494" PostTypeId="1" CreationDate="2017-12-08T06:17:33.810" Score="1" ViewCount="68" Body="&lt;p&gt;In this paper &lt;a href=&quot;http://sites.skoltech.ru/app/data/uploads/sites/25/2017/11/deep_image_prior.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep_image_prior&lt;/a&gt; :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To demonstrate the power of this parametrization, we consider inverse&#xA;  tasks such as denoising, super-resolution and inpainting. These can be&#xA;  expressed as energy minimization problems of the type&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YC4Ze.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YC4Ze.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;where E(x; x0) is a task-dependent data term, x0 the&#xA;  noisy/low-resolution/occluded image, and R(x) a regularizer. The&#xA;  choice of data term E(x; x0) is dictated by the application and will&#xA;  be discussed later. The choice of regularizer,  which usually captures&#xA;  a generic prior on natural images, is more difficult and is the&#xA;  subject of much research. As a simple example, R(x) may be the Total&#xA;  Variation (TV) of the image, which encourages solutions to contain&#xA;  uniform regions. In this work, we replace the regularizer R(x) with&#xA;  the implicit prior captured by the neural network, as follows:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/17yO4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/17yO4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then I see :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A parametrization with high noise impedance. One may wonder why a&#xA;  &lt;strong&gt;&lt;code&gt;high-capacity network fθ&lt;/code&gt;&lt;/strong&gt; can be used as a prior at all.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here, I wonder why they call fθ &lt;code&gt;high-capacity network&lt;/code&gt; ? What is the definition? Google just lead me to many non-relevant pages. &lt;/p&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="18375" LastEditDate="2017-12-09T10:27:16.820" LastActivityDate="2017-12-09T10:27:16.820" Title="What network is called high-capacity network? Why?" Tags="|neural-network|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25495" PostTypeId="1" CreationDate="2017-12-08T07:02:54.423" Score="1" ViewCount="820" Body="&lt;p&gt;I have been given different error rates for k=1,2,3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to calculate the average value of error,and determine whether it is a good error rate or not?&lt;/p&gt;&#xA;" OwnerUserId="43017" LastEditorUserId="40853" LastEditDate="2017-12-08T07:05:17.237" LastActivityDate="2017-12-09T05:08:52.880" Title="Calculate Average Error Rate in KNN" Tags="|machine-learning|data-mining|" AnswerCount="1" CommentCount="11" ClosedDate="2017-12-08T16:07:22.170" ContentLicense="CC BY-SA 3.0" />
  <row Id="25496" PostTypeId="2" ParentId="25495" CreationDate="2017-12-08T08:30:04.890" Score="1" Body="&lt;p&gt;Based on the explanation you gave I think you are talking about &lt;strong&gt;k-fold Cross validation&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming it as k-fold cross validation then you need to take mean of all the error rates. And the mean outcome would be your final error rate.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-09T05:08:52.880" LastActivityDate="2017-12-09T05:08:52.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25497" PostTypeId="1" CreationDate="2017-12-08T09:21:33.717" Score="1" ViewCount="147" Body="&lt;p&gt;I Would like to know or get a clear picture of Question answering system. How to find which is the best model to implement. I don't need it to be any domain specific.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, I find all models looks same to me, it may be IR-based question answering system, Knowledge-Based Question answering system, Ontology-based system or few other systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please guide me which one is best and what all approaches to be followed to implement those models.&lt;/p&gt;&#xA;" OwnerUserId="14929" LastActivityDate="2017-12-08T09:21:33.717" Title="Which is best Model to implement Question Answering System" Tags="|machine-learning|nlp|" AnswerCount="0" CommentCount="0" ClosedDate="2017-12-08T16:16:46.197" ContentLicense="CC BY-SA 3.0" />
  <row Id="25500" PostTypeId="1" CreationDate="2017-12-08T13:11:58.363" Score="1" ViewCount="53" Body="&lt;p&gt;I want to create a neural network in Matlab.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 16 import matrices in each iteration,  my import data changes in every iteration and there are more than 100000 iterations. The target is a matrix like [x y z] and is sum of imported data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have any suggestion for creating this kind of neural network?&#xA;any idea where should I start?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="43027" LastEditorUserId="43027" LastEditDate="2017-12-11T10:48:42.020" LastActivityDate="2017-12-11T10:48:42.020" Title="How to create a neural network with many different values in Matlab?" Tags="|machine-learning|neural-network|dataset|matlab|" AnswerCount="0" CommentCount="3" ClosedDate="2017-12-08T16:07:27.830" ContentLicense="CC BY-SA 3.0" />
  <row Id="25502" PostTypeId="1" AcceptedAnswerId="25518" CreationDate="2017-12-08T13:54:01.250" Score="2" ViewCount="3021" Body="&lt;p&gt;I have large number of csv files and each of them are timeseries based csv files sampled at Avery 5 seconds for 2-3 mins. I have 20k such files with 200-300 variables in each file.&#xA;I am aggregating the data by mean over the entire 2-3 mins window are using it for binary classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am using mean of each column in the .CSV file to represent that file, so basically I am summarizing the csv's using one scalar value per column.&#xA; so each file is one sample represented by its respective mean value.&#xA;Could anyone suggest me some better ways to summarize the timeseries data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastEditorUserId="29781" LastEditDate="2017-12-11T08:36:47.637" LastActivityDate="2019-06-09T15:30:47.140" Title="What are the best ways to use a time series data for binary classification" Tags="|time-series|feature-extraction|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25503" PostTypeId="2" ParentId="25180" CreationDate="2017-12-08T14:57:51.113" Score="0" Body="&lt;p&gt;500 can be right. It does &lt;strong&gt;not&lt;/strong&gt; imply 2^500 leafs, some leafs can stop earlier. How many observations do you have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In random forest you could use the out-of-bag predictions for tuning. That would make your tuning algorithm faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Max_depth = 500 does not have to be too much. The default of random forest in R is to have the maximum depth of the trees, so that is ok. You should validate your final parameter settings via cross-validation (you then have a nested cross-validation), then you could see if there was some problem in the tuning process. &lt;/p&gt;&#xA;" OwnerUserId="26692" LastActivityDate="2017-12-08T14:57:51.113" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25504" PostTypeId="2" ParentId="23637" CreationDate="2017-12-08T15:05:44.760" Score="3" Body="&lt;p&gt;Seems like you already tuned your algorithm based on the mean_test_score. You also could tune the AUC, which is the area under the ROC curve. Tuning this can even provide better mean_test_score as I show in my blog post: &lt;a href=&quot;http://philipppro.github.io/Tuning_random_forest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://philipppro.github.io/Tuning_random_forest/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26692" LastActivityDate="2017-12-08T15:05:44.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25505" PostTypeId="1" AcceptedAnswerId="25508" CreationDate="2017-12-08T19:01:58.110" Score="0" ViewCount="674" Body="&lt;p&gt;So I have a database of web-scraped cars, and I want to find the similarity between cars based on the km driven (e.g 69000), the model year (e.g 2012), and the trim of the car which will be one of three categories: base, mid, top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What similarity measure can I use that will give me a decently accurate similarity ratio&lt;/p&gt;&#xA;" OwnerUserId="43040" LastEditorUserId="18375" LastEditDate="2017-12-09T10:27:24.787" LastActivityDate="2017-12-09T10:27:24.787" Title="How to find a similarity value between cars" Tags="|categorical-data|similarity|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25506" PostTypeId="2" ParentId="25467" CreationDate="2017-12-08T21:23:04.020" Score="7" Body="&lt;p&gt;To put it shortly, xgboost tries to &lt;em&gt;fix it&lt;/em&gt; and although it is very good in getting rid of overfitting, it is not perfect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the  increased complexity overweight the added value from extra features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand you‘ve tested quite wide range of hyper-parameters and enough combinations. If you apply regularisation via &lt;em&gt;colsample_bytree&lt;/em&gt; and/or &lt;em&gt;colsample_bylevel&lt;/em&gt; it might happen that at some stage the randomly chosen columns (features) are less informative than your original features and the algorithm is forced to use these features for further splits. Does it make sense to you?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of added features might be crucial, if it is too high relative to original 20 the new features become just too dominant. For example, this might happen if one adds some nominal features with high cardinality, which are then dummy-encoded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to improve the results with wider data you might want to play with the parameters controlling the early stopping and stop the fitting even earlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rationale for my last suggestion:&#xA;I assumed that worsened performance on wider dataset is due to overfitting, i.e. significantly worse performance on test dataset than on training data. Early stopping should prevent / control the overfitting, but it seems it didn’t work so good in your case and thus should be improved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can and should test different combination of new features, but trustworthy quality metric is crucial for the model choice. If you significantly overfit, the performance on training data (or even cross-validation) will not help you to choose the model (or feature combination) with good performance on test data set. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2017-12-10T16:06:07.700" LastActivityDate="2017-12-10T16:06:07.700" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25507" PostTypeId="1" CreationDate="2017-12-08T21:42:40.720" Score="-1" ViewCount="48" Body="&lt;p&gt;The data looks like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Vehicle                   MilesPerGallon      DriverID    Date&#xA;Mazda 6 2016                8.6               10014       11/14/2017&#xA;Toyota Camry 2014           12.4              10051       11/4/2017&#xA;Nissan Altima 2012          7.1               10014       11/15/2017&#xA;Nissan Altima 2012          9.3               10065       11/15/2017&#xA;Mazda 6 2016                8.1               10058       11/15/2017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My objective is to find which vehicles give me good MilesPerGallon or in other words, what is the relation between Vehicle and MilesPerGallon&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The project is about creating a scoring algorithm for drivers based on their performance(with the goal on how to improve MPG). The first part is to calculate score based on driver behavior. I have done that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I should not penalize the drivers because they are driving old vehicle or vehicle that are not efficient. To take the vehicle out the equation, I want to normalize the score based on what vehicle they are driving.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I did for that was, ran correlation between Vehicle and fuel economy and if they value is -ve, I added that value to score. if the correlation is +ve, I subtracted that value from the final score. I know this is not the correct method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to find the relation and normalization value?&lt;/p&gt;&#xA;" OwnerUserId="43045" LastEditorUserId="43045" LastEditDate="2017-12-12T15:18:22.153" LastActivityDate="2017-12-14T01:18:11.813" Title="Conceptual Question about finding relation between one categorical variable and one numeric variable" Tags="|machine-learning|python|data-mining|clustering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25508" PostTypeId="2" ParentId="25505" CreationDate="2017-12-08T22:06:25.937" Score="2" Body="&lt;p&gt;I suggest you use a weighted per-attribute similarity, for instance let a and b a pair of tuples representing the attributes of some car A and another car B. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = (69000, 2012, base)&#xA;b = (71000, 2013, base)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the km driven I would use Skm = 1 / (1 + |km(a) - km(b)|), so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Skm = 1 / (1 + |69000 - 70000|) = 1 / 1001 = 0.0009&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the case of the year the same function could be use, then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Syear = 1 / (1 + |2012 - 2013|) = 1 / 2 = 0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Last, for the trim of the car I would use a function that returns 0 if the categories were different 1 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Strim = (base == base) = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S = Wkm * Skm + Wyear * Syear + Wtrim * Strim&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where Wkm + Wyear + Wtrim = 1.0 and I would adjust the values according to what you think is decent accurate similarity ratio. Setting the values to be Wkm = 0.6, Wyear = 0.2, Wtrim = 0.2  the similarity S(a, b) = 0.30054. This similarity has the nice property that tuples with the same attributes yield a value of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good practice is to normalize the values of each attribute so you don't assign low values to any attribute for instance you could divide the columns of the km and year attributes for the maximum value, therefore for the pair of tuples in the example you will have the following transformed tuples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a' = (0.985, 0.99, base)&#xA;b' = (1, 1, base)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the similarity using the same weights is S(a, b) = 0.9. Alternatively you could use a similarity matrix for the trim of the car, this is base could be more similar to mid thant to top. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cat  | base | mid | top |&#xA;-------------------------&#xA;base |  1   | 0.5 | 0.2 |&#xA;-------------------------&#xA;mid  |  0.5 |  1  | 0.7 |&#xA;-------------------------&#xA;top  |  0.2 | 0.7 |  1  |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The values of the matrix have to be adjusted to fit your needs. For a more in depth comparison between similarity functions between categorical attributes (trim) see &lt;a href=&quot;http://epubs.siam.org/doi/abs/10.1137/1.9781611972788.22&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for a comparison of different similarity measures for continuous attributes (year, km) see &lt;a href=&quot;http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144059&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again to see which similarity is the best will depend on the data and on your perception, I would select a random set of pairs of cars and manually assign a similarity. Then split this set into a train set and test set. Use the train set to adjust the parameters of the different possible similarities, in the case of my proposal Wkm, Wyear and Wtrim. Then once your satisfied evaluate on the test set. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-08T22:06:25.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25509" PostTypeId="2" ParentId="25348" CreationDate="2017-12-08T22:43:29.040" Score="6" Body="&lt;p&gt;You can train xgboost, calculate the output (margin) and then continue the training, see example in &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R&quot; rel=&quot;nofollow noreferrer&quot;&gt;boost from prediction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I‘ve not tried it myself, but maybe you could train on the first subset of your data (say 10%) and then continue on another subset, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Step by step procedure&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split the data into N manageable subsets, set n=1&lt;/li&gt;&#xA;&lt;li&gt;Train xgboost on n-th subset &lt;/li&gt;&#xA;&lt;li&gt;Calculate the prediction (margin) for n+1 subset using the model obtained from previous&lt;/li&gt;&#xA;&lt;li&gt;Add the margin into the n+1 subset via setinfo&lt;/li&gt;&#xA;&lt;li&gt;Increment n&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Steps 2-5 to be repeated N times. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2018-10-26T13:43:37.067" LastActivityDate="2018-10-26T13:43:37.067" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25510" PostTypeId="2" ParentId="25507" CreationDate="2017-12-08T23:05:11.827" Score="2" Body="&lt;p&gt;You can just calculate average miles per gallon for each given vehicle, e.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E(Miles per gallon | vehicle = Mazda 6 2016) =(8.6+8.1)/2&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2017-12-09T13:08:54.473" LastActivityDate="2017-12-09T13:08:54.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25511" PostTypeId="2" ParentId="25355" CreationDate="2017-12-08T23:17:03.783" Score="3" Body="&lt;p&gt;I don’t think linear model is a good idea for your problem, because it can capture only linear pattern. Furthermore, if I understand you correctly the results for different sites will not be comparable due to different scales/ slopes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead I suggest using the &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spearman rank correlation coefficient&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-12-08T23:17:03.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25513" PostTypeId="2" ParentId="25359" CreationDate="2017-12-08T23:34:19.867" Score="1" Body="&lt;p&gt;Why don’t you just put the data with pay_2 40k and 45k into two additional rows? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you could add an indicator variable loc_1_2 and set it to 1 or 2 depending on the location. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, dummy encoding might be dangerous. Instead you could convert your addresses or locations into physical coordinates (latitude and longitude) or just distance from, let’s say, centre of the city or center of the country (depending on your data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok, I see now that putting the data with pay_2 into additional rows will not work. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data (pay2) to have values similar to a standard gaussian. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For your information: not all classification algorithms need standardised data, e.g. you may want to consider random forest or GBMs. If you stick to SVMs, regularised logistic regression or similar then I agree, it’d make sense to standardise. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The standard approach you asked about, would depend on the properties of your classifier. I’m aware of the following options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If you classifier can handle NaNs (NAs), e.g. R version of xgboost, then you can just use the data as you show in your table above. &lt;/li&gt;&#xA;&lt;li&gt;If you have to standardise the data, then you can use mean-imputation. The average and standard deviation that are used for the standardisation should be estimated on non-NaN data, of course, (pay_2 column excl NaNs). In general, in order to prevent data leakage you should not use the data from holdout data subset (or “test”, “validation” or whatever you call it) for calculating the mean and standard deviation. &lt;/li&gt;&#xA;&lt;li&gt;If your classifier doesn’t require standardisation, you can put some numerical sentinel values instead of NaNs, e.g. -1, 0, or -9999. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you pick option 2 or 3, you’ll need to add a binary predictor indicating whether pay2 column was modified or not (NaN or not). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Re dummy encoding: &#xA;You haven’t asked about it explicitly and it’s difficult to suggest anything concrete without knowing the cardinality of your nominal variables, so I just put here the link to my answer about &lt;a href=&quot;https://datascience.stackexchange.com/a/24153/29781&quot;&gt;impact coding&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2017-12-11T22:01:10.897" LastActivityDate="2017-12-11T22:01:10.897" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25514" PostTypeId="1" AcceptedAnswerId="25517" CreationDate="2017-12-09T05:20:46.957" Score="0" ViewCount="115" Body="&lt;p&gt;How the neural networks are overfitted for regression.Either it tries&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to equal individual observation values or equals to the sum of all observations&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2017-12-09T12:21:37.350" Title="Neural Networks overfitting" Tags="|machine-learning|neural-network|deep-learning|regression|supervised-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-11T05:20:22.780" ContentLicense="CC BY-SA 3.0" />
  <row Id="25515" PostTypeId="2" ParentId="23637" CreationDate="2017-12-09T07:38:59.657" Score="1" Body="&lt;p&gt;I assume you are running classification, and have a binary target variable.  If that's the case, it does not make sense to show component ROC curves, because your separation may be based on on combinations of 2, 3, or more predictors that individual ROC curves will not reflect. I would show your overall ROC curve, along with perhaps variable importance measures.  If you have a handful of predictors that are clear winners, you could re-run your model including only those, and then show that ROC. Otherwise, I don't see what it buys you.&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-12-09T07:38:59.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25516" PostTypeId="2" ParentId="16225" CreationDate="2017-12-09T08:52:26.253" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;How is your experience using feature normalization with boosted trees does it in general improve our models?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My rather limited experience with scaling of features suggests that it has virtually no impact on xgboost results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose by normalisation you mean subtracting the mean and then dividing by standard deviation. If you calculated the statistics based on entire dataset (including holdout) you would get data leakage, which might indeed, at least theoretically, degrade the performance on holdout. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to my understanding of xgboost, &lt;em&gt;the correctly performed&lt;/em&gt; scaling should have no impact on the performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you double check your implementation or provide more details on how you do it, preferably with including a reproducible example. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-12-09T08:52:26.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25517" PostTypeId="2" ParentId="25514" CreationDate="2017-12-09T12:21:37.350" Score="2" Body="&lt;p&gt;Neural Networks basically act as a high memory-based machine learning algorithm. So for a given dataset the chance of it perfectly aligned with all the data at a given instance is high, as it most likely just ends up remembering every data point you give. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overfitting occurs precisely because of this, when a new expansive data set is introduced there is no way it can adjust its fit to the new data, graphically it ends up missing more of the values than its supposed to fit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, it does not work well in the case where the scoring population is significantly different compared to training sample. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-12-09T12:21:37.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25518" PostTypeId="2" ParentId="25502" CreationDate="2017-12-09T13:03:49.470" Score="5" Body="&lt;p&gt;From your comment, I understand that you are trying to solve the binary classification problem using your aggregated data and you are getting very poor results when you simply use the mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on specifics of your data and the shape of your time series, there are several alternatives that you could try. Note, that you might need (significantly) more than just a single number per time series to solve your problem. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;In addition to the mean, you could use the quantiles or some other summary statistic, like standard deviation, min, or max. &lt;/li&gt;&#xA;&lt;li&gt;You could try to sample the data, i.e. instead of taking the entire time series, pick only the values that are minutes, hours or days a part. Or pick only mid-day values. The frequency of the sampling depends on your data. &lt;/li&gt;&#xA;&lt;li&gt;Or just pre-aggregate by calculating averages for every hour, day, month, etc. &lt;/li&gt;&#xA;&lt;li&gt;Additionally, you could calculate the periodicity of your time series and use it as a new feature. &lt;/li&gt;&#xA;&lt;li&gt;Or calculate some trends. &lt;/li&gt;&#xA;&lt;li&gt;Try to fit some standard time series models to your data, e.g. ARIMA and use the coefficients as informative features.&lt;/li&gt;&#xA;&lt;li&gt;Last but not least, use the domain knowledge re what could be relevant feature for your classification problem: the biggest jump (max first order difference), change of regime, etc. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&#xA;I’d pick at least 10-20 features per time series generated as described above and apply logistic regression with LASSO or even xgboost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After selecting 10-20 features per time series you also could try PCA to reduce the dimension. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29169" LastEditDate="2019-06-09T15:30:47.140" LastActivityDate="2019-06-09T15:30:47.140" CommentCount="12" ContentLicense="CC BY-SA 4.0" />
  <row Id="25519" PostTypeId="2" ParentId="24477" CreationDate="2017-12-09T13:47:21.490" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is is possible to realize that using ggplot2 or any other package?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, it is possible, although, in general, not advisable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I may suggest, instead of cubes, cones and 3D pie charts it’d be better to use simple dot plots. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/if7Qj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/if7Qj.jpg&quot; alt=&quot;dot plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2017-12-09T14:08:22.030" LastActivityDate="2017-12-09T14:08:22.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25520" PostTypeId="1" CreationDate="2017-12-09T13:52:25.563" Score="6" ViewCount="490" Body="&lt;p&gt;In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while training. That is, SGD iteratively chooses one sample to perform forward pass followed by backpropagation to adjust the weights, as oppose to GD where the backpropagation starts only after the entire samples have been calculated in the forward pass).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: When the Gradient Descent (or even mini-batch Gradient Descent) is the chosen approach, how do we represent the error from a single forward pass? Assuming that my network has only a single output neuron, is the error represented by averaging all the individual errors from each sample or by summing all of them? To me this sounds like an implementation-dependent manner but I want to know if there is such a conventional way of that.&lt;/p&gt;&#xA;" OwnerUserId="43060" LastEditorUserId="43060" LastEditDate="2017-12-09T14:01:16.747" LastActivityDate="2021-07-09T12:10:58.220" Title="How backpropagation through gradient descent represents the error after each forward pass" Tags="|machine-learning|neural-network|scikit-learn|gradient-descent|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25521" PostTypeId="2" ParentId="25439" CreationDate="2017-12-09T15:39:27.893" Score="2" Body="&lt;p&gt;This is an open issue in Reinforcement Learning (and all of machine learning). Google has published a paper titled &lt;a href=&quot;https://research.google.com/pubs/pub43146.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Machine Learning: The High Interest Credit Card of Technical Debt&quot;&lt;/a&gt; which addresses the many ways these systems can degrade over time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way is to follow fundamental production engineering techniques (e.g., test coverage and graceful exit). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way is to monitor the data, then retrain the agent if new data is statistically out-of-sample.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2017-12-10T17:15:16.000" LastActivityDate="2017-12-10T17:15:16.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25524" PostTypeId="1" CreationDate="2017-12-10T02:06:24.947" Score="3" ViewCount="11961" Body="&lt;p&gt;I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2019-11-23T07:02:10.267" Title="How does Phrases in Gensim work?" Tags="|nlp|word2vec|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25525" PostTypeId="1" AcceptedAnswerId="25584" CreationDate="2017-12-10T03:41:59.203" Score="1" ViewCount="316" Body="&lt;p&gt;I have implemented LSTM in c++ which steadily decreases in error, but slows down at the certain error value. It also seems to predict most of the characters, but gets stuck and not able to correct some mistakes (or will correct them very slowly), even after 5000 backprop iterations. For example, asking it to predict character one by one might result in&lt;/p&gt;&#xA;&#xA;&lt;p&gt;abcdefg&lt;strong&gt;jff&lt;/strong&gt;klmnopqrstu&lt;strong&gt;x&lt;/strong&gt;wxyz &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or something similar. Notice, the network almost gets things right. Also, it never 'gets lost' after making a mistake, for instance in the above example, after &lt;strong&gt;jff&lt;/strong&gt; it produces &lt;strong&gt;k&lt;/strong&gt; and gets back on track as if it had never made that mistake. The results are always different - sometimes network learns all the letters. However, the error still plateaus at the same value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error starts from around 7.0 and decreases down to 2.35 after which it slows down with every iteration, which seems like it's hitting a plateau.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my alphabet consists only from &lt;em&gt;a,b&lt;/em&gt;, then network almost instantly realises it should be producing &lt;em&gt;abababababab&lt;/em&gt;, however the error starts at 0.8 and now always plateas at around 0.2 to 0.28&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If with &lt;em&gt;a,b&lt;/em&gt;, we set 4 timesteps, network learns to produce &lt;em&gt;abab&lt;/em&gt;, but after 50 000 back-props (being well-stuck even after 25 000) it predicts 'a' only with 85%-ish certainty, even though I would expect it to be 99.999%;  Similar value when 'b' has to be predicted. Once it gets stuck it maintains values similar to these. So it could keep guessing the same value over and over again when working with the &lt;em&gt;a,b&lt;/em&gt; dataaset; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Strangelly, when working with that &lt;em&gt;a,b&lt;/em&gt; dataset most of the time, I observe the final learnt probabilities to be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a=[0.68, 0.31] b=[0.15, 0.85] &lt;/li&gt;&#xA;&lt;li&gt;and sometimes, after re-initializing the network it learns the final probabilities as a=[0.8205, 0.1794]  and b=[0.1795, 0.8205]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Disabling momentum (previous frame's grads times zero) still has same effect on &lt;em&gt;a,b&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network doesn't explode, but its gradients seem to vanish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it usual to get stuck at those values? Back-propping after 26 timesteps, done 200 000 times, and by that time the changes get turtle slow. The error sits at around 2.35 and is not worth the wait for another 0.000001 change in error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experimenting with a smaller learning rate (0.0004) allows the error to get down to 2.28 -that's the best I've got. Also, using a momentum coefficient with 0.2; It's applied to the previous frame's gradient. I don't increase momentum while the program executes, but keep it at a constant 0.2;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newgradient = newgradientMatrix + prevFrameGradientMatrix*0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am not using any form of dropout etc. &lt;strong&gt;I just want the network to overfit, but it's getting stuck at 2.35 for 26-char aphabet.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am only getting 0 error when an entire alphabet consists of a single character. In that case, the NN will predict aaaaaaaaaaaaaaaaa and error will be 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Forward prop:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All is done on a single thread, in CPU. Using 'float' for the components of vectors.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tanh on dataGate and after the 'cell' is constructed (before the 'cell' is multiplied by an output gate)&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Sigmoid on Input, Forget and Output gate&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each gate has Matrix of weights where each column is weights from neuron in current LSTM unit to all neurons in previous LSTM unit. Last column is ignored because nothing should feed into our bias. Also, bias value (but not the weights from it!) is set manually to 1.0 just to be sure.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Each gate has a separate NxN matrix with recurrent weights (U-matrix) operating on the results of the current LSTM unit at [time-1]&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Both W and U keep the last row, so they &lt;strong&gt;both&lt;/strong&gt; sample bias of lower-LSTM. This shouldn't create issues, granted that both of biases are back-propagated properly. In fact, last row was removed from U-Matrix altogether - just to be sure, but the error still plateaus at the same 2.35 quantity regardless.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Weight initialization:&#xA;&lt;a href=&quot;http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Xavier-Benjio&lt;/a&gt; with a uniform distribution page 253, bottom right.&#xA;Boundaries of the uniform distrib are defined as mentioned &lt;a href=&quot;https://stackoverflow.com/a/33711310/9007125&quot;&gt;here&lt;/a&gt;, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;low = -4*np.sqrt(6.0/(fan_in + fan_out)); // use |4| for sigmoid gates, |1| for tanh gates &#xA;high = 4*np.sqrt(6.0/(fan_in + fan_out));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cost function&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result of LSTM unit are softmaxed (bias is ignored), then passed through cross-entropy function to get a single float value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross entropy is the correct one, for multi&#xA;class classification. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;float cost = 0;&#xA;for(each vector component){&#xA;    if(predictedVal == 0){  continue;  }&#xA;    cost += -(targetVec[i])*naturalLog(predictedVec[i]);&#xA;}&#xA;return cost;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The cost is then summed up across all timesteps and its average is returned right before we do the backprop. This is where I am getting plateaus at 2.3 for 26-character alphabet &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the Cell (aka c) and Result (aka h) are cached after the last (26th timestep). After back-propagation they are used by timestep0. This could be (and was for some time) disabled, but the results are similar.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Backpropagation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will list a couple of important gotchas &amp;amp; keypoints that I took care of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;de_dh&lt;/em&gt; is simply (targetVec - predictedVec), in that order. That's because the gathered gradient will be &lt;em&gt;subtracted&lt;/em&gt; from the W and U matrices. This is due to derivatives cancelling out nicely when the softmax &amp;amp; crossEntropy are used together during forward prop.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To &lt;em&gt;de_dh&lt;/em&gt; an extra gradient is added from t+1. That added quantity is sum from all 4 gates. To explain better, recall that one of such gates was forward-propping as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;dataGate = tanh(W * incomingVal + &lt;strong&gt;U * lstmResultPrevT&lt;/strong&gt; + bias_ComesFrom_W_and_U); //one of the 4 gates&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above formula, the bold represents the quantity from where the gradient is taken of one of such four gates at [t+1]. Such a gradient is then summed up across those 4 gates and added to &lt;em&gt;de_dh&lt;/em&gt;, as stated originally. It's necessary to be done because during forward prop, 'H' of [t] has affected all 4 gates of [t+1] &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When computing the gradient for Cell at [t], the cell's gradient from [t+1]is added. Afterwards, we compute a gradient for C of [t-1], to be able to repeat this process when we arrive to the earlier timestep.   &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The gradient for the U-weights leading to bias at [t-1] is computed with remembering that the bias's original value was 1.0; Also, it's double-checked to ensure gradient &lt;strong&gt;doesn't&lt;/strong&gt; flow from our bias at [t] to the neurons at [t-1]. That's because nothing fed into our bias originally. As follows, the entire last column of U-gradient matrix is always 0.0; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar thing is done for such a bias-column of the W matrix too - that entire column is zero.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, the gradient is computed for each H of [t-1] for each of the four gates. This is done so that the '2. key-point' is possible (adding the 4-grads to &lt;em&gt;de_dh&lt;/em&gt;), when we get to the earlier timestep in this back-prop.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Unit Tests &amp;amp; debugging:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;after 20 000 backprops (done every 26 timesteps) a file is collected.&#xA;it was observed that gradients were very small on all 4 gates, especially after being passed through the activation function at each gate. &#xA;This is one of the reasons why Xavier init (above) was introduced, to prevent weights from being too large (shrinks grad after pushing-back through activation) or being too small (shrinks grad after pushing-back through weights).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A significant improvement was observed after &lt;a href=&quot;http://nmarkou.blogspot.co.uk/2017/07/deep-learning-why-you-should-use.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;'norm clipping'&lt;/a&gt; was used, allowing my LSTM to seemingly learn a correct sequence even when 56 unique characters were used and backprop was done after 56 timesteps. Similar to the original example (with the 26 chars) only a couple of characters were predicted incorrectly. However the error still always plateaus, at a higher value (around 4.5)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once again, is this traditional behavior, and I just have to rely on the things like dropout and averaging the results of multiple networks? However it seems that my network isn't even capable of overfitting...&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've discovered one thing - the result of LSTM is vector, whose components cannot be less than -1 or greater than 1 (curtesy of tanh and sigmoid) As a result, &lt;span class=&quot;math-container&quot;&gt;$e^x$&lt;/span&gt; cannot be smaller than ~0.36 or greater than ~2.71&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the probabilities always have some 'precipitation' dangling, and network always 'worries' that it can't reach 100% confidence? Tried to get clarification on that &lt;a href=&quot;https://datascience.stackexchange.com/q/25531/43077&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2020-01-15T21:17:10.663" LastActivityDate="2020-01-15T21:17:10.663" Title="My LSTM can't reduce error down to zero, when overfitting" Tags="|backpropagation|lstm|overfitting|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25526" PostTypeId="1" CreationDate="2017-12-10T03:47:57.483" Score="2" ViewCount="304" Body="&lt;p&gt;How to create domain rules from raw unstructured text using NLP and deep learning techniques ?&#xA;For example for the below text on symptoms of Dengue, all three look pretty similar but if you want to make sure a person is having Dengue you want to exact definite common minimum rules out of these raw text , so as to confirm a person is having Dengue. Can someone give reference to some research or blogs where similar problem has been solved ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) Symptoms of dengue fever include severe joint and muscle pain, swollen lymph nodes, headache, fever, exhaustion, and rash. &#xA;&#xA;2) High fever and at least two of the following:&#xA;Severe headache&#xA;Severe eye pain (behind eyes)&#xA;Joint pain&#xA;Muscle and/or bone pain&#xA;Rash&#xA;Mild bleeding manifestation (e.g., nose or gum bleed, petechiae, or easy bruising)&#xA;Low white cell count&#xA;&#xA;3) Aching muscles and joints&#xA;Body rash that can disappear and then reappear&#xA;high fever&#xA;intense headache&#xA;pain behind the eyes&#xA;vomiting and feeling nauseous&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the above three extracts commons rule would look like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) Fever&#xA;2) Joint and muscle pain&#xA;3) Headache&#xA;4) Rash&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18313" LastActivityDate="2022-01-12T00:04:00.520" Title="How to create domain rules from raw unstructured text using NLP and deep learning?" Tags="|machine-learning|deep-learning|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25527" PostTypeId="2" ParentId="25520" CreationDate="2017-12-10T04:14:12.110" Score="3" Body="&lt;p&gt;To get total error before back propagating - it is common to take an average of all the forward-pass errors. This is what's done in RNN such as LSTM.&lt;/p&gt;&#xA;&lt;p&gt;In the case of linear regression and logistic regression, The traditional &lt;a href=&quot;http://cens.ioc.ee/local/man/matlab/toolbox/nnet/mse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mean Squared Error Function&lt;/a&gt; can produce such a value.&lt;/p&gt;&#xA;&lt;p&gt;In essence, this value is represented by an average of errors: &lt;span class=&quot;math-container&quot;&gt;$Y(w) = 1/n{\sum_{i=1}^n Y_i(w)}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also, as a reminder, speaking of an actual backpropagation - &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;from wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;When used to minimize the above function, a standard (or &amp;quot;batch&amp;quot;) gradient descent method would perform the following iterations:&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$w:=w - {\alpha}\nabla Y(w) $$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;which is basically&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$w:= w - \alpha{\sum_{i=1}^n} \nabla Y_i(w)/n $$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;notice the &lt;span class=&quot;math-container&quot;&gt;$/n$&lt;/span&gt; When used with the &lt;span class=&quot;math-container&quot;&gt;${\sum_{i=1}^n}$&lt;/span&gt; it results in the average of all gradients&lt;/p&gt;&#xA;&lt;p&gt;:= means 'becomes qual to'&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\alpha$&lt;/span&gt; is the learning rate&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2021-07-09T12:10:58.220" LastActivityDate="2021-07-09T12:10:58.220" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25528" PostTypeId="2" ParentId="25482" CreationDate="2017-12-10T07:04:42.083" Score="3" Body="&lt;p&gt;What you are looking for is &lt;strong&gt;feature contributions&lt;/strong&gt; to the final score of an observation having a positive or negative price. Feature importances from a decision tree or random forest is not going to help you because a feature's importance is fixed across all observations (wines). They tell a story about the overall model, but nothing about the individual observations. So if you want to know what caused a wine having a negative or positive price, you should look at feature contributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's elaborate a little bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's forget about decision trees, and go for ensemble of them, say random forest, for all the good reasons. (I highly recommend you to read why ensemble models uniformly beat the base learner, and how they help with the high bias - high variance problem base learners might have.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first step would be to label your dependent variable, price, as $1$ and $0$, depending on the price being positive and negative ($y = 1$ iff $price &amp;gt; 0$). After the typical exploratory analysis (where you can look at boxplots and stuff as a beginning study) and feature engineering, you can fit your random forest to predict a wine having a positive or negative price (i.e. $y$ being $1$ or $0$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you do that, you will have an estimated probability for each and every wine, representing the classification model's estimation of that wine's price being positive or negative. Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Wine Prob&#xA;A    0.4&#xA;B    0.7&#xA;C    0.9&#xA;D    0.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These probabilities actually consist of the summation of a bias term and the individual contributions of all features &lt;strong&gt;to that observation only&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(y_i = 1) = bias + \sum_{i=1}^{m}(contributionOfFeature_i),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $m$ is the total number of features. This is how you can analyze each and every feature's contribution on the final probability of a particulat wine's price being positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding how to code this, if you are using scikit, you can use this very easy and convenient tool:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/andosa/treeinterpreter&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/andosa/treeinterpreter&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20572" LastActivityDate="2017-12-10T07:04:42.083" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25530" PostTypeId="1" CreationDate="2017-12-10T10:00:32.127" Score="1" ViewCount="718" Body="&lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;: I am trying to train different sub-samples of MNIST dataset(1 model for 10,000 , another model for 20,000 samples, and so on until a model for all 60,000 training samples) using Caffe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: How do I sub-split the training dataset of 60,000 samples &lt;strong&gt;keeping the format of the training files same(intact) as originally published&lt;/strong&gt; on &lt;a href=&quot;http://yann.lecun.com/exdb/mnist/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://yann.lecun.com/exdb/mnist/&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wDxYS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wDxYS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43088" LastActivityDate="2017-12-10T10:33:09.277" Title="How to divide MNIST training dataset into subparts?" Tags="|machine-learning|deep-learning|caffe|mnist|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-14T01:00:01.047" ContentLicense="CC BY-SA 3.0" />
  <row Id="25531" PostTypeId="1" AcceptedAnswerId="25540" CreationDate="2017-12-10T10:27:01.697" Score="3" ViewCount="2327" Body="&lt;p&gt;When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;notice, LSTM's result is a $h=tanh(c) \circ \sigma(p)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where c is the cell state passed through a tanh as well, and $\circ$ is the component-wise product of two vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall that tanh never goes beyond -1 and never goes above 1; The $\sigma$ never goes below 0 and never above 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that if we have a 4-neurons on the output - the best guess a network can make will be 1, -1, -1, -1?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When softmaxed, this will produce $\frac{2.72}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}$ which are these probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.7, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.1, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.1, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we can never get ~100% certainty, (&lt;strong&gt;Edit:&lt;/strong&gt; or in this example even above 70%) no matter the learning rate? &#xA;Is there a way to combat it without destroying its derivative working nicely with Cross-entropy? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a solution, I've tried to determine how close to $e^{−1}$ and $e^1$ the $e^x$ actually lies. Basically an 'inverse lerp', and it indeed puts values to 0 and 1 range. However, I am not sure if it affects derivatives &amp;amp; if it no longer will work out of the box when using Cross Entropy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the usual way to compute softmaxed-vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;float totalSum = 0;&#xA;Vec softmaxedVec;&#xA;for(int i=0; i&amp;lt; tanhVec.Length; ++i){&#xA;   softmaxedVec[i] =  exp( tanhVec[i]);&#xA;   totalSum += softmaxedVec[i];&#xA;}&#xA;&#xA;for(int i=0; i&amp;lt;softmaxedVec.Length; ++i){&#xA;   softmaxedVec[i] = softmaxedVec[i] / totalSum;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here is what I am now trying to do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;float totalSum = 0;&#xA;Vec softmaxedVec;&#xA;for(int i=0; i&amp;lt; tanhVec.Length; ++i){&#xA;   softmaxedVec[i] =  inverse_lerp(0.3678,  2.71828,  exp( tanhVec[i]));&#xA;   totalSum += softmaxedVec[i];&#xA;}&#xA;&#xA;for(int i=0; i&amp;lt;softmaxedVec.Length; ++i){&#xA;   softmaxedVec[i] = softmaxedVec[i] / totalSum;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//returns percentage: 0 the value sits on min, 1.0 the value sits on max&#xA;float inverse_lerp(float min, float max, float currValue){&#xA;    return (currValue - min) / (max - min);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, with this I am afraid that I might have lost that gorgeous $\frac{\partial E}{\partial W} = expected - target $  (curtesy of softmax &amp;amp; cross entropy working together) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;..or if I basically ruined the benefit of softmax&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2017-12-10T13:41:05.410" LastActivityDate="2017-12-10T15:45:38.480" Title="Softmax classifier never allows for 100% probability in LSTM?" Tags="|probability|lstm|cost-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25532" PostTypeId="2" ParentId="25530" CreationDate="2017-12-10T10:33:09.277" Score="1" Body="&lt;p&gt;I think you can just read each line of the file which gives you one data point and write it into an output file. Run a loop for the number of data points you want.&lt;/p&gt;&#xA;" OwnerUserId="43091" LastActivityDate="2017-12-10T10:33:09.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25533" PostTypeId="2" ParentId="25531" CreationDate="2017-12-10T11:47:28.257" Score="2" Body="&lt;p&gt;If want you mean by 100% certainty is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;0,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ e^{h_i} \neq 0 $$ $$ \forall h_i \in \mathbb{R} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See this &lt;a href=&quot;https://www.quora.com/Why-cant-e-x-0&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; for better explanation. Note that $ e^x $ can be close to 0 but it will never be 0. &#xA;The kind of output that you want is the one that the max function would output, sadly the max function is not differentiable so a soft version is needed hence softmax.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that the problem is that you are using softmax after tanh, and this so you must eliminate one of the &quot;squashing&quot; functions, see this &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/42lle4/does_it_make_sense_to_use_softmax_right_after/&quot; rel=&quot;nofollow noreferrer&quot;&gt;discussion&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason to use softmax is if the classes are mutually exclusives, also one of the advantages of using softmax is that it reduces sensitivity to outliers by giving at least some probability to the other values. See this links for a more detailed explanation &lt;a href=&quot;https://www.quora.com/Artificial-Neural-Networks-Why-do-we-use-softmax-function-for-output-layer&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-10T13:53:41.130" LastActivityDate="2017-12-10T13:53:41.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25535" PostTypeId="5" CreationDate="2017-12-10T13:44:04.640" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-12-10T13:44:04.640" LastActivityDate="2017-12-10T13:44:04.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25536" PostTypeId="4" CreationDate="2017-12-10T13:44:04.640" Score="0" Body="LSTM stands for Long Short-Term Memory. When we use this term most of the time we refer to a recurrent neural network or a block (part) of a bigger network. " OwnerUserId="42460" LastEditorUserId="42460" LastEditDate="2017-12-14T00:57:09.447" LastActivityDate="2017-12-14T00:57:09.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25537" PostTypeId="5" CreationDate="2017-12-10T15:02:42.347" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-12-10T15:02:42.347" LastActivityDate="2017-12-10T15:02:42.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25538" PostTypeId="4" CreationDate="2017-12-10T15:02:42.347" Score="0" Body="MNIST is a database of handwritten digits collected by Yann Lecun, a famous computer scientist, when he was working at AT&amp;T-Bell Labs on the problem of automation of check readings for banks. The data set is a benchmark widely used in machine learning research. Due to its small size it is also widely used for educational purposes.s based on C++ but provides APIs for Python and Matlab." OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-12-14T00:57:14.620" LastActivityDate="2017-12-14T00:57:14.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25540" PostTypeId="2" ParentId="25531" CreationDate="2017-12-10T15:45:38.480" Score="2" Body="&lt;p&gt;If you feed the output of the LSTM directly into a softmax, you probably won't get good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use a softmax layer after a tanh layer, bad stuff  happens.  As you  say, the confidence will never get near 100%.  For instance, if there are two classes, you can never get above about 88% confidence.  If there are $k$ classes, you can never get confidence above $e/(e + (k-1)/e)) = e^2/(e^2 + k-1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, rather than directly feeding the output of the LSTM directly into softmax, you can instead use the output of the LSTM as the input to one or more (fully-connected) layers of neural network.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2017-12-10T15:45:38.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25542" PostTypeId="1" CreationDate="2017-12-10T18:21:56.273" Score="0" ViewCount="862" Body="&lt;p&gt;I'm trying to predict a UPDRS score (regression) from the &lt;a href=&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/telemonitoring/&quot; rel=&quot;nofollow noreferrer&quot;&gt;parkinsons dataset&lt;/a&gt;. I wanted to start with a fairly basic structure and the add in more layers/techniques to improve performance but no matter what I do the network constantly predicts 1 for every input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been banging my head against this problem for a couple of days now with no success. Originally I was using RELU and thought it was a case of dead neurons but using leaky RELU and monitoring the outputs I can see them firing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Changing the network structure or training steps always results with the same horrible MSE of 896.39105. This leads me to believe the mistake is somewhere in the normalization but when I check the intermediate values they all seem fine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me where I've messed up?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&#xA;import tensorflow as tf&#xA;import pandas as pd&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn.preprocessing import normalize&#xA;from sklearn.preprocessing import StandardScaler&#xA;&#xA;&#xA;def leaky_relu(x, alpha):&#xA;  return tf.nn.relu(x) - alpha * tf.nn.relu(-x)&#xA;&#xA;raw = pd.read_csv('parkinsons_updrs.data')&#xA;print(&quot;CSV loaded successfully&quot;)&#xA;&#xA;#Split into features and targets&#xA;features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1).as_matrix()&#xA;targets = raw['total_UPDRS'].get_values()&#xA;&#xA;scale = StandardScaler(with_mean=0, with_std=1)&#xA;scale.fit(features, targets)&#xA;scaled_feats = scale.transform(features)&#xA;&#xA;print(&quot;pre-processing complete&quot;)&#xA;&#xA;#Split data into test/train&#xA;data_train, data_test, targets_train, targets_test = train_test_split(scaled_feats,targets,test_size = 0.2,random_state=123)&#xA;&#xA;print(&quot;Train/Test split complete&quot;)&#xA;&#xA;&#xA;#Need to reshape to match placeholder shape&#xA;targets_train = targets_train.reshape(targets_train.shape[0],-1)&#xA;targets_test = targets_test.reshape(targets_test.shape[0],-1)&#xA;&#xA;&#xA;&#xA;#Network Structure&#xA;X = tf.placeholder(tf.float32, [None,features.shape[1]])&#xA;y = tf.placeholder(tf.float32, [None,1])&#xA;prob = tf.placeholder(tf.float32)&#xA;&#xA;w1 = tf.Variable(tf.random_normal([features.shape[1],100]))&#xA;w2 = tf.Variable(tf.random_normal([100,1]))&#xA;&#xA;b1 = tf.Variable(tf.random_normal([100]))&#xA;b2 = tf.Variable(tf.random_normal([1]))&#xA;&#xA;&#xA;layer1 = tf.nn.dropout(leaky_relu(tf.add(tf.matmul(X,w1),b1),0.2),prob)&#xA;out = tf.nn.dropout(tf.nn.tanh(tf.add(tf.matmul(layer1,w2),b2)),prob)&#xA;&#xA;loss = tf.reduce_mean(tf.square(out - y))&#xA;train_step = tf.train.AdamOptimizer(0.001).minimize(loss)&#xA;&#xA;sess = tf.Session()&#xA;sess.run(tf.global_variables_initializer())&#xA;&#xA;print(&quot;Training...&quot;)&#xA;&#xA;&#xA;minibatch = False&#xA;batchsize = 512&#xA;&#xA;&#xA;for i in range(1000):&#xA;    if minibatch:&#xA;        for x in range(0,targets_train.shape[0],batchsize):&#xA;            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})&#xA;    else:&#xA;        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})&#xA;    if i%25==0:&#xA;        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train,prob:1})&#xA;        print (&quot;Training accuracy: %.5f&quot; % (acc))&#xA;        #print(sess.run(layer1,feed_dict={X: data_train, y: targets_train,prob:1})[0])&#xA;&#xA;&#xA;&#xA;test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test,prob:1})&#xA;print (&quot;Test accuracy: %.5f&quot; % (test_acc))&#xA;&#xA;#Check if still predicting 1&#xA;output = sess.run(out,feed_dict={X: data_test, y: targets_test,prob:1})&#xA;print(sum(output)==len(output))&#xA;&#xA;&#xA;&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="43107" LastEditorUserId="43107" LastEditDate="2017-12-10T18:27:13.977" LastActivityDate="2020-03-13T14:08:08.273" Title="Tensorflow regression predicting 1 for all inputs" Tags="|machine-learning|neural-network|deep-learning|regression|tensorflow|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25543" PostTypeId="1" AcceptedAnswerId="25594" CreationDate="2017-12-10T18:40:33.817" Score="5" ViewCount="1534" Body="&lt;p&gt;since some time I have a question to which I have not found the proper answer yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My doubt concerns the interpretation of the results of a &lt;code&gt;clustering algorithm&lt;/code&gt; which was run on features to which a &lt;code&gt;log-transformation&lt;/code&gt; was applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, let's assume we want to run a &lt;code&gt;k-means algorithm&lt;/code&gt; on 3 interval variables. Unfortunately, these three interval variables are extremely bad distributed and the k-means gives the worst result we have ever seen.&#xA;However, let's imagine that by applying a &lt;code&gt;log transformation&lt;/code&gt; to each variable, we obtain three incredibly perfect &lt;code&gt;normal distribution&lt;/code&gt;.&lt;br&gt;&#xA;Then, we run again the &lt;code&gt;k-means&lt;/code&gt; and we obtain perfect &lt;code&gt;clusters&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, my doubt concerns the interpretation of this cluster obtained by running a &lt;code&gt;k-means&lt;/code&gt; on three &lt;code&gt;log-transformed&lt;/code&gt; variables: it is not clear whether our interpretation of the clusters obtained should be made on the original variables or it should be made on the &lt;code&gt;log-transformed&lt;/code&gt; variables?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, my example is related to log-transformation but we can talk about &lt;code&gt;z-score&lt;/code&gt; or &lt;code&gt;min-max normalization&lt;/code&gt; or any other kind of &lt;code&gt;transformation&lt;/code&gt; that we apply in order to improve the quality &lt;code&gt;distribution&lt;/code&gt; before running the clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify, what I mean by interpretation is the &lt;code&gt;profiling&lt;/code&gt; of the cluster, which means try to describe which are the characteristics common to the individuals belonging to that cluster.&lt;/p&gt;&#xA;" OwnerUserId="43110" LastActivityDate="2019-04-18T10:56:27.160" Title="Interpret clustering results after variable transformation" Tags="|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25545" PostTypeId="2" ParentId="25502" CreationDate="2017-12-10T20:36:06.560" Score="2" Body="&lt;p&gt;You can try taking Fourier transform of the series if your domain suggests that frequency elements might have some meaning or relevance for classification. I once took top 10 coefficients of the transform along usual statistical features as suggested by aivanov. It helped me classify my data. Before taking the transform, you might also benefit from passing the series through a high-pass/low-pass or a band filter.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2017-12-10T20:36:06.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25546" PostTypeId="2" ParentId="19541" CreationDate="2017-12-11T01:00:09.737" Score="0" Body="&lt;p&gt;It seems overkill, but I gonna tell you how to get strict math dependency from your data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;same sets of inputs should be converted to single set and rate instead of bool, where true increments rate and false decrements the rate for same sets of values.&lt;/li&gt;&#xA;&lt;li&gt;input vars set should be converted to system of equations: $v_1=a, v_2=b ... $ where $a,b,c,d,e$ are particular inputs set. And one more var for rate.&lt;/li&gt;&#xA;&lt;li&gt;simultaneous equations may be converted to single equation formed by summing it squares: $(v1-a)^2+(v2-b)^2+(v3-c)^2...=0$&lt;/li&gt;&#xA;&lt;li&gt;the equation may be used if you insert new set of vars values in equation and see which rate will outcome.&lt;/li&gt;&#xA;&lt;li&gt;find the equation that gives best fit to your set of data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you have aggressive calculation abilities then you may want to proceed to next level of precision:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do steps 1-3 of the previous sequence.&lt;/li&gt;&#xA;&lt;li&gt;Different sets are non-simultaneous equation of the system you need. Non-simulteneous system may be converted to single equation using multiplication. You'l have big product of all your data like $((v1-a1)^2+(v2-b1)^2+(v3-c1)^2...)((v1-a2)^2+(v2-b2)^2+(v3-c2)^2...)((v1-a3)^2+(v2-b3)^2+(v3-c3)^2...)...=0$&lt;/li&gt;&#xA;&lt;li&gt;Use the system and OpenCL or IBM XL C++ compiler OpenMP extension offload to Nvidia GPUs feature to simulteneously calculate members of the product that have all information of flight results. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Similar approach is used in &lt;a href=&quot;https://github.com/ohhmm/openmind/blob/expression/omnn/extrapolator/test/test_extrapolator.cpp&quot; rel=&quot;nofollow noreferrer&quot;&gt;this extrapolator implementation&lt;/a&gt; and its math framework.&lt;/p&gt;&#xA;" OwnerUserId="43112" LastEditorUserId="40853" LastEditDate="2017-12-11T11:37:26.033" LastActivityDate="2017-12-11T11:37:26.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25547" PostTypeId="2" ParentId="25542" CreationDate="2017-12-11T03:12:21.440" Score="2" Body="&lt;p&gt;Ok, I've managed to get a decent improvement. Currently producing an MSE of 20 for training and 30 for test. This is after converting all layers to relu and increasing their size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem appeared the activation function on the output. I was under the impression that a sigmoid output layer was common practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was also normalizing before the test/train split and leaking information into the test set! (though this likely wasn't the cause of my problem)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my 'improved' code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&#xA;import pandas as pd&#xA;import numpy as np&#xA;from scipy import stats&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn import preprocessing&#xA;&#xA;&#xA;raw = pd.read_csv('parkinsons_updrs.data')&#xA;print(&quot;CSV loaded successfully&quot;)&#xA;&#xA;events = raw.shape[0]&#xA;#drop events where any feature has a zscore &gt; 3 for that feature&#xA;raw = raw[(np.abs(stats.zscore(raw)) &amp;lt 3).all(axis=1)]&#xA;print(&quot;Removed &quot; + str(events-raw.shape[0]) + &quot; outlier events&quot;)&#xA;&#xA;features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1)&#xA;targets = raw['total_UPDRS'].get_values()&#xA;print(&quot;pre-processing complete&quot;)&#xA;&#xA;#Split data into test/train&#xA;data_train, data_test, targets_train, targets_test = train_test_split(features,targets,test_size = 0.33,random_state=123)&#xA;print(&quot;Train/Test split complete&quot;)&#xA;&#xA;&#xA;#Normalize data to mean 0 and std 1&#xA;#We normalize after the split to avoid leaking information to the training set&#xA;data_train = (data_train-data_train.mean())/data_train.std()&#xA;data_test = (data_test-data_test.mean())/data_test.std()&#xA;&#xA;targets_train = targets_train.reshape(targets_train.shape[0],-1)&#xA;targets_test = targets_test.reshape(targets_test.shape[0],-1)&#xA;&#xA;&#xA;&#xA;#Network Structure&#xA;X = tf.placeholder(tf.float32, [None,18])&#xA;y = tf.placeholder(tf.float32, [None,1])&#xA;prob = tf.placeholder_with_default(1.0, shape=())&#xA;&#xA;w1 = tf.Variable(tf.random_normal([18,72]))&#xA;w2 = tf.Variable(tf.random_normal([72,36]))&#xA;w3 = tf.Variable(tf.random_normal([36,1]))&#xA;&#xA;b1 = tf.Variable(tf.random_normal([72]))&#xA;b2 = tf.Variable(tf.random_normal([36]))&#xA;b3 = tf.Variable(tf.random_normal([1]))&#xA;&#xA;&#xA;&#xA;layer1 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(X,w1),b1)),prob)&#xA;layer2 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(layer1,w2),b2)),prob)&#xA;out = tf.nn.relu(tf.add(tf.matmul(layer2,w3),b3))&#xA;&#xA;loss = tf.losses.mean_squared_error(y, out)&#xA;&#xA;train_step = tf.train.AdamOptimizer(0.005).minimize(loss)&#xA;&#xA;sess = tf.Session()&#xA;sess.run(tf.global_variables_initializer())&#xA;&#xA;print(&quot;Training...&quot;)&#xA;&#xA;&#xA;minibatch = True&#xA;batchsize = 512&#xA;&#xA;&#xA;for i in range(10000):&#xA;    if minibatch:&#xA;        for x in range(0,targets_train.shape[0],batchsize):&#xA;            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})&#xA;    else:&#xA;        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})&#xA;    if i%25==0:&#xA;        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train})&#xA;        print (&quot;Training accuracy: %.5f&quot; % (acc))&#xA;&#xA;&#xA;&#xA;test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test})&#xA;print (&quot;Test accuracy: %.5f&quot; % (test_acc))&#xA;&#xA;output = sess.run(out,feed_dict={X: data_test, y: targets_test})&#xA;&#xA;error = abs(output-targets_test)&#xA;&#xA;print(max(error))&#xA;print(sum(error)/len(error))&#xA;print(min(error))&#xA; &lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'd be very interested to know why this happened. Both tanh and sigmoid get 'stuck' in the 900s whereas relu sorts itself out and carries on.&lt;/p&gt;&#xA;" OwnerUserId="43107" LastEditorUserId="43107" LastEditDate="2017-12-11T03:19:51.640" LastActivityDate="2017-12-11T03:19:51.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25549" PostTypeId="1" AcceptedAnswerId="25551" CreationDate="2017-12-11T06:26:43.883" Score="3" ViewCount="1906" Body="&lt;p&gt;I am trying to implement a 1 channel CNN by slightly changing this article: &lt;a href=&quot;https://arxiv.org/pdf/1408.5882v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;.&#xA;The problem is that I am new to keras and deep learning and I don't know this far why I am getting this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Negative dimension size caused by subtracting 100 from 1 for 'conv2d_1/convolution' (op: 'Conv2D') with input shapes: [?,1,70,100], [100,100,100,64]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously, it's a mismatch in the dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using this code code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Embedding&#xA;from keras.layers import Conv2D&#xA;from keras.models import Sequential&#xA;from keras.layers import MaxPooling2D&#xA;from keras.layers import Reshape&#xA;import pdb&#xA;Vocab_Size=11123&#xA;MAX_SEQUENCE_LENGTH=70&#xA;EMBED_DIM=100&#xA;model = Sequential()&#xA;embed1=Embedding(Vocab_Size+1,EMBED_DIM,input_length=MAX_SEQUENCE_LENGTH,input_shape=(MAX_SEQUENCE_LENGTH,EMBED_DIM,1))&#xA;nb_labels=6&#xA;model = Sequential()&#xA;model.add(embed1)&#xA;model.add(Reshape((1,MAX_SEQUENCE_LENGTH, EMBED_DIM)))&#xA;model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation=&quot;relu&quot;, padding='valid'))&#xA;model.add(MaxPooling2D((MAX_SEQUENCE_LENGTH-5+1,1)))&#xA;model.add(Flatten())&#xA;model.add(Dense(256, activation=&quot;relu&quot;))&#xA;model.add(Dropout(0.3))&#xA;model.add(Dense(len(nb_labels), activation=&quot;softmax&quot;))&#xA;&#xA;model.compile(loss='categorical_crossentropy',&#xA;                optimizer='rmsprop',&#xA;                metrics=['acc'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit1:&#xA;I updated as mentionned by Media padding to same.&#xA;Now I have another problem for the next layer:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Negative dimension size caused by subtracting 66 from 1 for 'max_pooling2d_1/MaxPool' (op: 'MaxPool') with input shapes: [?,1,14,64].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40515" LastEditorUserId="28175" LastEditDate="2017-12-11T15:58:23.343" LastActivityDate="2017-12-11T15:58:23.343" Title="Implementing a CNN with one convolution layer" Tags="|machine-learning|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25550" PostTypeId="2" ParentId="25526" CreationDate="2017-12-11T07:41:33.437" Score="1" Body="&lt;p&gt;I think your question can be solved using &lt;strong&gt;Case Based Reasoning&lt;/strong&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basic principle how it works is, you need to train the model using whole lot of different cases which you have. Based on the symptoms which you give the outcome is predicted(which disease).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/O0QcB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/O0QcB.png&quot; alt=&quot;Process Flow&quot;&gt;&lt;/a&gt;&#xA;Please refer to the links below for deep diving into the topic, appended couple of links with respect to health care industry:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://ml.informatik.uni-freiburg.de/former/_media/teaching/ss10/cbr1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -1&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-540-85502-6_1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.skateboardingalice.com/papers/2010_Begum.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -3&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Case-based_reasoning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -4&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.scribd.com/document/279763300/A-Tutorial-on-Case-Based-Reasoning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -5&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://home.etf.rs/~vm/os/dmsw/Case-Based%20Reasoning.ppt&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -6&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://ibug.doc.ic.ac.uk/media/uploads/documents/courses/syllabus-CBR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link -7&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you need any additional information.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-11T08:00:25.343" LastActivityDate="2017-12-11T08:00:25.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25551" PostTypeId="2" ParentId="25549" CreationDate="2017-12-11T07:44:46.023" Score="2" Body="&lt;p&gt;I guess you should change the following line to solve the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation=&quot;relu&quot;, padding='valid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead use this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation=&quot;relu&quot;, padding='same'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this will keep the dimension of your input. If it does not work, let me know to help you more.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-11T07:44:46.023" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25554" PostTypeId="1" CreationDate="2017-12-11T09:33:50.547" Score="2" ViewCount="1225" Body="&lt;p&gt;I wonder how to evaluate variable long sequence-to-sequence predictions?&#xA;Let us say I have the following $Y$ and $\hat{Y}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y = [[&quot;1&quot;, &quot;2&quot;, &quot;2&quot;], [&quot;3&quot;, &quot;2&quot;, &quot;2&quot;], [&quot;1&quot;, &quot;3&quot;, &quot;2&quot;, &quot;2&quot;]]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{Y} = [[&quot;1&quot;, &quot;3&quot;, &quot;2&quot;], [&quot;3&quot;, &quot;3&quot;], [&quot;1&quot;, &quot;3&quot;, &quot;2&quot;]]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shall I use a binary comparison where any mismatch counts as zero and any full match as one? Or shall calculate conventional accuracy by character-wise comparison?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My concern here is that on one hand, if this is a numeric predcition then any digit mismatch spoils the whole number so doesn't really matter where is the mismatch; on the other hand it would be nice which digits tend to miscalculated in order to find ways to improve the training sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addition: the task is a numeric OCR so -- in contrast with a machine translation job where minor mistranslations are tolerable -- any digit mismatch could result significant business problem (different invoice sums for example). Moreover I'd like to know which individual digits tend to be misread more often so I need a way to get a statistic this way also.&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21560" LastEditDate="2017-12-12T09:03:35.673" LastActivityDate="2017-12-12T19:35:37.350" Title="How to evaluate sequence to sequence models?" Tags="|machine-learning|classification|model-evaluations|sequence|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25555" PostTypeId="5" CreationDate="2017-12-11T09:36:55.757" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2017-12-11T09:36:55.757" LastActivityDate="2017-12-11T09:36:55.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25556" PostTypeId="4" CreationDate="2017-12-11T09:36:55.757" Score="0" Body="the process of using domain knowledge of the data to create features that improve machine learning algorithms" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2019-04-03T08:26:28.407" LastActivityDate="2019-04-03T08:26:28.407" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25557" PostTypeId="1" CreationDate="2017-12-11T09:40:49.993" Score="1" ViewCount="25" Body="&lt;p&gt;I'm trying to create the most basic and simple neural network to simulate a situation where the random initialization of the variables matters a lot.&#xA;So if I run the network and the initialization over and over again the network will not converge well, unless the random initialization hit the sweet spot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically trying to come up with the most simple example of a non-convex cost function to simulate a situation of non-convergence.&lt;/p&gt;&#xA;" OwnerUserId="32042" LastActivityDate="2017-12-11T09:40:49.993" Title="Create the most simple/basic deep network where variable initialization does matter a lot" Tags="|neural-network|tensorflow|keras|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="25561" PostTypeId="2" ParentId="12851" CreationDate="2017-12-11T12:33:37.177" Score="13" Body="&lt;p&gt;Here is yet another way - &lt;a href=&quot;https://github.com/martisak/dotnets&quot; rel=&quot;noreferrer&quot;&gt;dotnets&lt;/a&gt;, using &lt;a href=&quot;http://www.graphviz.org/&quot; rel=&quot;noreferrer&quot;&gt;Graphviz&lt;/a&gt;, heavily inspired by &lt;a href=&quot;https://tgmstat.wordpress.com/2013/06/12/draw-neural-network-diagrams-graphviz/&quot; rel=&quot;noreferrer&quot;&gt;this post&lt;/a&gt; by Thiago G. Martins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://github.com/martisak/dotnets/raw/master/test.png&quot; alt=&quot;dotnets example&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="43144" LastActivityDate="2017-12-11T12:33:37.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25562" PostTypeId="2" ParentId="25450" CreationDate="2017-12-11T13:30:37.007" Score="2" Body="&lt;p&gt;To freeze the lower layers during training, the simplest solution is to give the optimizer the list of variables to train, excluding the variables from the lower layers:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,&#xA;                               scope=&quot;hidden[34]|outputs&quot;)&#xA;training_op = optimizer.minimize(loss, var_list=train_vars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first line gets the list of all trainable variables in hidden layers 3 and 4 and in the output layer. This leaves out the variables in the hidden layers 1 and 2.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2017-12-11T13:30:37.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25571" PostTypeId="2" ParentId="25554" CreationDate="2017-12-11T17:01:33.330" Score="5" Body="&lt;p&gt;Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values &lt;strong&gt;correlate with a good or bad performance&lt;/strong&gt; of the system in the &lt;strong&gt;real world problem&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though I don't know exactly what your system is expected to do [written before question edit, see below for addition], maybe you could take as reference how performance is measured in speech and text recognition tasks. There you have a reference and a predicted sequence composed of characters and these characters form words. The length of the reference and the prediction is not necessarily the same. The performance is measured both at character and at word level (character error rate (CER) and word error rate (WER)). When measured at word level, even if only one character is wrong, the whole word is wrong. In both cases the main idea is to compute the &lt;strong&gt;levenshtein/edit distance&lt;/strong&gt; (either at word or character level) between the reference and the prediction and then divide by the length of the reference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y=[[&quot;1&quot;,&quot;2&quot;,&quot;2&quot;],[&quot;3&quot;,&quot;2&quot;,&quot;2&quot;],[&quot;1&quot;,&quot;3&quot;,&quot;2&quot;,&quot;2&quot;]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ŷ=[[&quot;1&quot;,&quot;3&quot;,&quot;2&quot;],[&quot;3&quot;,&quot;3&quot;],[&quot;1&quot;,&quot;3&quot;,&quot;2&quot;]]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At inner bracket level the edit distances for the three inner brackets would be 1+2+1=4, since in the first a &quot;3&quot; is substituted by &quot;2&quot; (one edit) in the second a &quot;3&quot; is substituted by a &quot;2&quot; and a &quot;2&quot; is inserted (two edits) and in the last case again a &quot;2&quot; is inserted (one edit). The total length of the reference is 3+2+3=8. So the final performance measure is 4/8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if each of the inner brackets in your case are different samples (i.e., Ŷ is the whole dataset) or Ŷ is a single sample and you have something similar to characters and words. But anyway, if you do the edit distance between Ŷ and Y and divide by the length of Ŷ then you have 3/3, since all three brackets are different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional response to question edit: If the task is number OCR, then you would compute character error rate. Computing an error value for each of the 10 digits is also possible with the edit distance. It would be by counting how many edit operations correspond to errors in each of the digits. For this in python you can use the editops function from the python-Levenshtein package (&lt;a href=&quot;https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-editops&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-editops&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="42135" LastEditorUserId="42135" LastEditDate="2017-12-12T19:35:37.350" LastActivityDate="2017-12-12T19:35:37.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25572" PostTypeId="2" ParentId="12872" CreationDate="2017-12-11T18:47:18.217" Score="0" Body="&lt;p&gt;In a context free grammar, I think it is really kind of impossible to determine the closeness of words. What you can do is use lexicon vectors and then if a word is close in values between two lexicons then the value should be close.&lt;/p&gt;&#xA;" OwnerUserId="43157" LastEditorUserId="29575" LastEditDate="2017-12-11T21:08:43.523" LastActivityDate="2017-12-11T21:08:43.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25573" PostTypeId="1" AcceptedAnswerId="25575" CreationDate="2017-12-11T19:45:00.323" Score="4" ViewCount="141" Body="&lt;p&gt;Is there a practical strategy that can learn to price a product optimally? Right now I have the following arbitrary hill-climbing algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Run an experiment at starting price &lt;code&gt;P&lt;/code&gt; and gather 500 data points (e.x. 20 buy and 480 not buy). &lt;/li&gt;&#xA;&lt;li&gt;Run a t-test on what confidence level &lt;code&gt;P&lt;/code&gt; yields higher revenue per visitor than &lt;code&gt;P * 1.1&lt;/code&gt; and &lt;code&gt;P * 0.9&lt;/code&gt;. Then do a 3-way weighted coin-flip and the winner gets to run the next experiment.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There's many problems with this approach. For example, if price is at optimal, it can't price a product at a more optimal pricing e.x. &lt;code&gt;P * 1.03&lt;/code&gt;. Another is that if at some price point &lt;code&gt;P = K&lt;/code&gt; we happen to get really unlucky and get 1 buy of 500 data points, the algorithm won't converge fast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem gets easy if we take lots of data points but that would reduce long term revenue. Is there a fast algorithm that can converge to the optimal price and then not do anymore exploration?&lt;/p&gt;&#xA;" OwnerUserId="42969" LastEditorUserId="42969" LastEditDate="2017-12-11T19:52:11.927" LastActivityDate="2017-12-12T15:05:31.560" Title="Exploration vs exploitation tradeoff to find a price that maximizes revenue" Tags="|machine-learning|regression|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25574" PostTypeId="2" ParentId="25443" CreationDate="2017-12-11T19:52:22.187" Score="3" Body="&lt;p&gt;To produce confidence intervals for xgboost model you should train several models (you can use bagging for this). Each model will produce a response for test sample - all responses will form a distribution from which you can easily compute confidence intervals using basic statistics. You should produce response distribution for each test sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this answer is provided here:&#xA;&lt;a href=&quot;https://stackoverflow.com/questions/37418938/how-to-obtain-a-confidence-interval-or-a-measure-of-prediction-dispersion-when-u&quot;&gt;https://stackoverflow.com/questions/37418938/how-to-obtain-a-confidence-interval-or-a-measure-of-prediction-dispersion-when-u&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42488" LastActivityDate="2017-12-11T19:52:22.187" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25575" PostTypeId="2" ParentId="25573" CreationDate="2017-12-11T19:59:00.817" Score="7" Body="&lt;p&gt;Without making any underlying assumptions you will not get anywhere. That said, there are multi-arm bandit strategies that try to optimize the rewards, there is a ton of research on this field. It comes down to sampling from a distribution of your options (in your case two) and adapting this distribution based on the rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-armed_bandit&quot; rel=&quot;noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Multi-armed_bandit&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you know that the reward distribution from each bandit comes from a specific distribution, you can deduce optimal sampling strategies. Once you have at least some prior information you can do fairly well although not always optimal. Regardless, most strategies will do better than normal A/B testing if the strategy is not super greedy.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-12-11T19:59:00.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25576" PostTypeId="1" AcceptedAnswerId="27679" CreationDate="2017-12-11T20:38:44.600" Score="5" ViewCount="2994" Body="&lt;p&gt;I'm trying to solve a simple regression problem using TensorFlow and Pandas to see what's the expected conversion rate for a given product given how much I'm willing to pay for each visit (cost per click or &lt;em&gt;cpc&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each data point represents a single day and its given acquisition cost and conversion rate. I expect conversion rate to decrease exponentially as visits increase because my audience gets wider and not as focused anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data seems to agree with this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ohel7.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ohel7.png&quot; alt=&quot;acquisition cost X conversion rate&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that for the majority of data points, the conversion rate is &lt;em&gt;zero&lt;/em&gt; (i.e. no products were sold) so any model is likely to perform well by merely guessing zero sales. For this particular subset (one of the best selling products) the ratio of zero-to-non-zero is roughly 4:1 but in the general data it's about 300:1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two best ways that I can think of dealing with this are &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) summarizing data by time periods large enough to contain at least one conversion in them (problem with this is that I lose data by having to settle for the average cost instead of the more detailed daily data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b) splitting the problem into a classification problem (&lt;em&gt;will I sell at least one unit&lt;/em&gt;) and a regression problem (&lt;em&gt;given that I'm going to sell at least one unit, how many units can I expect to sell&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do either of those sound like a smart idea? Are there any tried-and-true methods to solve this kind of problem?&lt;/p&gt;&#xA;" OwnerUserId="43159" LastActivityDate="2021-03-24T07:20:28.100" Title="Best method to deal with too many zeroes in regression problem?" Tags="|machine-learning|regression|pandas|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25577" PostTypeId="1" CreationDate="2017-12-11T20:49:43.623" Score="0" ViewCount="219" Body="&lt;pre&gt;&lt;code&gt;    Word ID  X            Y   Page IDFont Style  Color  size    bold    an uppercse all uppercase   single digit    multiple digit  special character   email   hperlink    calender month&#xA;0   234378  247.400000  27.058  p1  nimbussanl  #000000 14.0    0   True    True    False   False   False   False   False   False&#xA;1   234378  293.539231  27.058  p1  nimbussanl  #000000 14.0    0   True    True    False   False   False   False   False   False&#xA;2   71195   376.700000  54.470  p1  nimbussanl  #000000 10.0    0   True    False   False   False   False   False   False   False&#xA;3   201816  395.973846  54.470  p1  nimbussanl  #000000 10.0    0   False   False   False   False   False   False   False   False&#xA;4   166593  406.110769  54.470  p1  nimbussanl  #000000 10.0    0   True    True    False   False   False   False   False   False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2056r.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2056r.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word ID : I have used NLTK’s english word corpus to convert the words to unique ID’s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data set that is given above contains data types that have classes(Page ID,Font), Boolean and Numeric values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have successfully pre-processed (Normalized X and Y;One hot encoding other columns) all the columns except Word ID. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word ID is the ID given to each word in the dataset. Can somebody help me to preprocess the Word ID column that contains value as high as 234378 and as low as 230 in the original dataset. I've this doubt because &lt;strong&gt;Word ID&lt;/strong&gt;(numerical) as feature &lt;strong&gt;must not numerically influence&lt;/strong&gt; the final answer.&lt;/p&gt;&#xA;" OwnerUserId="37502" LastEditorUserId="40853" LastEditDate="2017-12-12T18:13:40.970" LastActivityDate="2017-12-16T02:31:55.413" Title="How should I preprocess WORD ID?" Tags="|machine-learning|scikit-learn|dataset|data-cleaning|pandas|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25579" PostTypeId="1" CreationDate="2017-12-11T21:01:58.860" Score="1" ViewCount="158" Body="&lt;p&gt;What happens if we flip the arrows in a Naive Bayes classifier? To clarify - from what I have found naive Bayes is defined for the following network structure:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vl5RQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vl5RQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested to understand what happens if instead from y-&gt;x the child will be y. As shown in the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/huqvw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/huqvw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure if we get the same results as a regular naive Bayes classifier (intuitively we don't), and if not how to estimate both the likelihood of the network and the parameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further clarification: lets say there is a training set of size n; each data point consisting of (x,y) where x is a vector of size m of binary values and y is the class and is also binary. E.g. (0,1,0,1,1) where the last index is the class. I'm trying to figure out the likelihood of the network (where y is the child) and how to estimate the parameters in the case of y being the child.&lt;/p&gt;&#xA;" OwnerUserId="43162" LastActivityDate="2017-12-11T22:37:15.383" Title="Reversed Naive Bayes - likelihood and parameter estimation" Tags="|classification|naive-bayes-classifier|bayesian-networks|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25580" PostTypeId="2" ParentId="25579" CreationDate="2017-12-11T22:37:15.383" Score="1" Body="&lt;p&gt;To elaborate on my comment, the first image depicts conditional dependence of $X_i | Y$ for $i \in 1,...,m$. &#xA;Therefore, your likelihood for this model would be  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{aligned}&#xA;P(Y = y) \times P(X_1 | Y = y) \times ... \times P(X_m | Y = y)&#xA;\end{aligned}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second image (bottom) depicts $Y | X_i$, therefore the likelihood would be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;P(X_1) \times P(X_2) \times ... \times P(Y | X_1, X_2, ..., X_m) \\&#xA;\ = P(X_1) \times P(X_2) \times ... \times P(Y | X_1) \times ... \times P(Y | X_m)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that both scenarios do not have conditional independence, the two are not equivalent!&lt;/p&gt;&#xA;" OwnerUserId="16220" LastActivityDate="2017-12-11T22:37:15.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25581" PostTypeId="1" AcceptedAnswerId="25582" CreationDate="2017-12-11T22:51:57.513" Score="20" ViewCount="42727" Body="&lt;p&gt;We can use CountVectorizer to count the number of times a word occurs in a corpus:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Tokenizing text&#xA;from sklearn.feature_extraction.text import CountVectorizer&#xA;count_vect = CountVectorizer()&#xA;X_train_counts = count_vect.fit_transform(twenty_train.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we convert this to a data frame, we can see what the tokens look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eB4Uh.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eB4Uh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the 35,780th word of the 3rd document occurs twice. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency and not the inverse) as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfTransformer&#xA;tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)&#xA;X_train_tf = tf_transformer.transform(X_train_counts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Converting this to a data frame, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3VnIe.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3VnIe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?&lt;/p&gt;&#xA;" OwnerUserId="30359" LastEditorUserId="85045" LastEditDate="2019-12-12T17:16:02.233" LastActivityDate="2020-12-10T11:41:17.753" Title="What is the difference between CountVectorizer token counts and TfidfTransformer with use_idf set to False?" Tags="|machine-learning|nlp|scikit-learn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25582" PostTypeId="2" ParentId="25581" CreationDate="2017-12-11T23:02:33.297" Score="21" Body="&lt;p&gt;Actually, the documentation was pretty clear. I'll keep it posted in case someone else searches before reading:&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;TfidfTransformer&lt;/code&gt; &lt;strong&gt;transforms a count matrix&lt;/strong&gt; to a &lt;strong&gt;normalized&lt;/strong&gt; tf or tf-idf representation. So although both the &lt;code&gt;CountVectorizer&lt;/code&gt; and &lt;code&gt;TfidfTransformer&lt;/code&gt; (with &lt;code&gt;use_idf=False&lt;/code&gt;) produce term frequencies, &lt;code&gt;TfidfTransformer&lt;/code&gt; is normalizing the count.&lt;/p&gt;&#xA;" OwnerUserId="30359" LastEditorUserId="14869" LastEditDate="2020-12-10T11:41:17.753" LastActivityDate="2020-12-10T11:41:17.753" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25583" PostTypeId="1" AcceptedAnswerId="25593" CreationDate="2017-12-11T23:19:13.800" Score="1" ViewCount="53" Body="&lt;p&gt;I am working with a corpus that has 5 datasets in product reviews (A, B, C, D and E), mine is a text classification problem and I need to find the best 5 top models in terms of classification performance (F1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started with collection A: the mp3 reviews,  Because it has the largest numbers of documents (900: yes, 750: No). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I trained the data using 10-fCv using different algorithms and pre-processing tasks, got the weighted results for all experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose the top 5 models and I want to apply them to the rest of the corpus: B, C, D and E (other products' reviews).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My plan is to run 10-fCv and get the results for all the collections and compute the Micro-average for precision, recall and F1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right way to choose a model for a large collection?&lt;/p&gt;&#xA;" OwnerUserId="43166" LastEditorUserId="40853" LastEditDate="2017-12-12T16:20:53.227" LastActivityDate="2017-12-12T16:20:53.227" Title="After choosing top models in classification? Can I apply it on the rest of my dataset" Tags="|classification|text-mining|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25584" PostTypeId="2" ParentId="25525" CreationDate="2017-12-12T02:12:34.160" Score="2" Body="&lt;p&gt;As stated in the last edit of my question, the issue indeed was to do with the softmax function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As clarified &lt;a href=&quot;https://datascience.stackexchange.com/questions/25531/softmax-classifier-never-allows-for-100-probability-in-lstm&quot;&gt;here&lt;/a&gt; We shouldn't apply softmax directly to the result of the last LSTM. Notice, LSTM will produce a vector of values, each of which is bounded between -1 and 1 (due to the tanh squashing function that's applied to the Cell).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I've created a traditional fully-connected layer (just additional weight matrix), and feed result of LSTM to that layer. This &quot;output&quot; layer isn't activated - it feeds into a softmax function, which actually serves as an activation instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I modified  the back-prop algorithm to supply the gradient generated by the softmax to the Output layer. Of course, if you used a cross entropy Cost function originally, then such a gradient will remain $(predicted - expected)$. It's then pushed through the weights of that Output Layer, to get the gradient w.r.t. LSTM. After this the backprop is applied as usual and the network finally converges. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; There is slight improvement to momentum.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Also, using a momentum coefficient with 0.2; It's applied to the previous frame's gradient. I don't increase momentum while the program executes, but keep it at a constant 0.2;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newgradient = newgradientMatrix + prevFrameGradientMatrix*0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That's fine, but changing the momentum-coefficient will require us to also re-adjust learning rate. A cleaner version will be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newgradient = newgradientMatrix*(1-0.9) + prevFrameGradientMatrix*0.9&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is an exponential moving average, that remembers roughly $\frac{1}{0.1} = 10$ days. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the 10th day, the coefficinet will be $$\frac{(1-\epsilon)^{\frac{1}{\epsilon}}}{0.9} = \frac{(1-0.1)^\frac{1}{0.1}}{0.9} = \frac{0.9^{10}}{0.9} \approx 0.387 \approx \frac{1}{e}$$ of the peak; Because 9 newer days have larger coefficients (larger than 0.387), their avearge really makes 10th day and older be negligible&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any older days will have even less contribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, don't forget about the bias correction, - which helps get a better estimate when we are &quot;just starting&quot; to compute the average. Without the bias correction it would start very low, and will take some time to catch-up with the expected &quot;exponential moving average&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;newval = curval*(1-0.9) + prevVal*0.9&#xA;newval /= 1-(0.9)^t  //where t is timestep&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in practice, it will be fine after approximately 10 timesteps - and so there is no real need for the bias correction in Momentum, but it should be done if we are using combination of Adam (a combination of Momentum &amp;amp; RMSProp)&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2017-12-28T15:25:10.363" LastActivityDate="2017-12-28T15:25:10.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25588" PostTypeId="1" AcceptedAnswerId="25630" CreationDate="2017-12-12T08:23:39.347" Score="1" ViewCount="7337" Body="&lt;p&gt;I would appreciate if you could let me know how to select features based on feature importance using SelectFromModel. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# data&#xA;X = np.array(pd.read_csv('who_X_1.csv',header=None).values)&#xA;y = np.array(pd.read_csv('who_Y_1.csv',header=None).values.ravel())&#xA;indices = np.arange(y.shape[0])&#xA;&#xA;# # Divide Data into Train and Test&#xA;X_train, X_test, y_train, y_test,idx_train,idx_test = train_test_split(X, yy,indices,stratify=yy,test_size=0.3, random_state=42)&#xA;&#xA;&#xA;scaler = StandardScaler()&#xA;&#xA;# # Compute Cohen's Kappa or Auc as scoring criterion due to imbalanced data set&#xA;kappa_scorer = make_scorer(cohen_kappa_score)&#xA;auc_scorer=make_scorer(roc_auc_score)&#xA;F_measure_scorer = make_scorer(f1_score)&#xA;&#xA;##hyperparameter&#xA;&#xA;param_grid = {&#xA;    'clf__colsample_bytree': [i/10.0 for i in range(7,10)],&#xA;    #&quot;clf__subsample&quot;  : [i/10.0 for i in range(5,10)],&#xA;    #'clf__max_depth':range(5,15,1),&#xA;    #'clf__gamma':[i/10.0 for i in range(0,5)],&#xA;    #'clf__reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]&#xA;&#xA;            }&#xA;&#xA;##Classifier&#xA;xg=XGBClassifier(max_depth=3,&#xA;                 learning_rate=0.05,&#xA;                 n_estimators=350,&#xA;                 objective=&quot;binary:logistic&quot;,&#xA;                 booster=&quot;gbtree&quot;,&#xA;                 gamma=0,&#xA;                 min_child_weight=0.8,&#xA;                 subsample=1,&#xA;                 colsample_bylevel=1,&#xA;                 colsample_bytree=0.6,&#xA;                 reg_alpha=0.001,&#xA;                 reg_lambda=1,&#xA;                 scale_pos_weight=22,&#xA;                 random_state=4,n_jobs=-1)&#xA;pipe=Pipeline(steps=[('pre',scaler),&#xA;                    ('clf',xg)])&#xA;&#xA;rg_cv = GridSearchCV(pipe, param_grid, cv=5, scoring = 'f1')&#xA;rg_cv.fit(X_train, y_train)&#xA;print(&quot;Tuned rf best params: {}&quot;.format(rg_cv.best_params_))&#xA;&#xA;# Use SelectFromModel&#xA;thresholds = np.sort(rg_cv.best_estimator_.named_steps[&quot;clf&quot;].feature_importances_)&#xA;for thresh in thresholds:&#xA;    # select features using threshold&#xA;    selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)&#xA;    select_X_train = selection.transform(X_train)&#xA;&#xA;    # train model&#xA;    selection_model = rg_cv&#xA;    selection_model.fit(select_X_train, y_train)&#xA;&#xA;    # eval model&#xA;    select_X_test = selection.transform(X_test)&#xA;    y_pred = selection_model.predict(select_X_test)&#xA;    predictions = [round(value) for value in y_pred]&#xA;    accuracy = accuracy_score(y_test, predictions)&#xA;    print(&quot;Thresh=%.3f, n=%d, Accuracy: %.2f%%&quot; % (thresh, select_X_train.shape[1], accuracy * 100.0))&#xA;    print(confusion_matrix(y_test, predictions))&#xA;    print(classification_report(y_test, predictions))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the following error occurred:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: The underlying estimator GridSearchCV has no `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to SelectFromModel or call fit before calling transform.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="134" LastEditDate="2018-12-27T14:56:40.940" LastActivityDate="2018-12-27T14:56:40.940" Title="How to select features based on feature importance using SelectFromModel?" Tags="|python|scikit-learn|feature-selection|xgboost|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25590" PostTypeId="1" CreationDate="2017-12-12T10:47:58.210" Score="2" ViewCount="428" Body="&lt;h3&gt;Background:&lt;/h3&gt;&#xA;&lt;p&gt;I've built a neural network for predicting categorical outcomes and wanted to test variable importance with the variable importance method proposed by &lt;a href=&quot;http://www.massey.ac.nz/%7Emkjoy/pdf/Olden,Joy&amp;amp;DeathEM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Olden et al. 2004&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h3&gt;Problem:&lt;/h3&gt;&#xA;&lt;p&gt;However, I've noticed that the more 'important' variables, i.e., variables with greater absolute value on the output from Olden's algorithm, tend to have far fewer samples in the training set than the 'less important' variables. For example, a variable &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; with variable importance value 5000 might only have 5 observations in the training set that have &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; encoded as true (I've used one-hot encoding to encode observations here) whilst variable &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; with importance value 72 might have 750 observations in the training set that have &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; encoded as true.&lt;/p&gt;&#xA;&lt;h3&gt;Questions&lt;/h3&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Has anybody come across this issue?&lt;/li&gt;&#xA;&lt;li&gt;How would you resolve this?&lt;/li&gt;&#xA;&lt;li&gt;Is it reasonable to say that variable &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; has more variable importance than variable &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt;, even though it has a much smaller sample size?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43191" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-12T12:04:55.627" Title="Variable Importance for NN's with Olden's Method" Tags="|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25593" PostTypeId="2" ParentId="25583" CreationDate="2017-12-12T12:27:41.793" Score="1" Body="&lt;p&gt;This is an interesting question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general the split of data is about the underlying distribution. That means you split a dataset into train-test sets in a way that random split of train-test does not dramatically affect the distribution. &lt;strong&gt;But Splitting based on topics is not random!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specially in your case you are talking about text in which the distribution is super sensitive to the domain i.e. if you collect the commentary of 1000 football games and the narrations of 1000 documentary movies about wild life, you will see that they are literally two different things. The conceptual difference between products most likely affects the distribution of words/terms/phrases therefore the model trained on reviews of mp3s &lt;strong&gt;MUST NOT&lt;/strong&gt; be validated on reviews of football shoes!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, I would say the train-test split (CV folds) should be done on whole data together so that you maintain the original topology of word distribution (topology here is not a Math term but I simply mean the shape of distribution). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case if you do &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Topic Modeling&lt;/a&gt; on the whole training data you simply see 5 different product topics. Or if you use &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; or &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt; you hopefully see 5 different clusters. Then you can run your models in this setting for evaluation. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;NOTE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the size of classes are very different you need to come up with some solution for small classes. If it was the case, just drop me a line in comments and we can discuss solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-12-12T12:27:41.793" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25594" PostTypeId="2" ParentId="25543" CreationDate="2017-12-12T12:55:06.177" Score="4" Body="&lt;p&gt;Very interesting!&#xA;What you did to your data is simply a feature mapping/transformation. So how this affects the clustering results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is not a clearly defined problem but at least we know something about it: It's about internal similarities (patterns) so these similarities should be maintained through the feature transformation. In your example if you found clusters in transformed space, it shows that you have had clusters in original space as well. You just couldn't see them according to the algorithm you used in that space!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance if you use Kernelized versions of algorithms you easily find that what they do is nothing but what you did as transformation. They first use a kernel to map the data into new space and then use the algorithm in that space (of course with a bit of theoretical differences/constraints).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize, no transformation produces a &lt;strong&gt;fake&lt;/strong&gt; pattern in the data. In worst case it vanishes the original pattern and in the best case it reveals the pattern which was not visible originally (which is your case).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I mentioned Fake Pattern above so let me say a bit more on it. I think there is a fundamental issue concerning your question: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You assume that there is a Right clustering that you got after transformation. &lt;strong&gt;Actually there is no right clustering!&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We do not have fake pattern! If there is a pattern in a feature space, then that is true! i.e. you found an interesting representation of your data. If it does not match with the labels then either the data is very noisy or wrong features have been chosen to represent classes (maybe more reasons. just these two came to my mind now). If there is no label (your case) be sure there is a correlation between features of those cluster members.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2017-12-12T12:55:06.177" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25596" PostTypeId="1" CreationDate="2017-12-12T13:04:59.383" Score="18" ViewCount="141544" Body="&lt;p&gt;I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns with different mean values.&lt;/p&gt;&#xA;&lt;p&gt;I merged both data frames into a &lt;code&gt;total_year&lt;/code&gt; data frame.&lt;/p&gt;&#xA;&lt;p&gt;The output of &lt;code&gt;total_year&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BAs6x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BAs6x.png&quot; alt=&quot;Totle year Output&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Now, I would like to plot &lt;code&gt;total_year&lt;/code&gt; on a line graph in which the X axis should contain the year column and the Y axis should contain both the action and the comedy columns.&lt;/p&gt;&#xA;&lt;p&gt;I can plot only 1 column at a time on Y axis using following code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;total_year[-15:].plot(x='year', y='action' ,figsize=(10,5), grid=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How can I plot both columns on the Y axis?&lt;/p&gt;&#xA;&lt;p&gt;I took this photo from google just to let you know guys I want to draw graph in this way:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/h5luU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/h5luU.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43200" LastEditorUserId="29169" LastEditDate="2021-03-13T20:07:32.307" LastActivityDate="2021-03-13T20:07:45.597" Title="How to plot two columns of single DataFrame on Y axis" Tags="|python|pandas|visualization|dataframe|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25597" PostTypeId="2" ParentId="25576" CreationDate="2017-12-12T13:08:30.357" Score="0" Body="&lt;p&gt;You asked about „tried-and-true“ method, thus I‘m suggesting the logistic regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I‘m not familiar with TF yet, so not sure whether this approach is easily doable there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the link to the answer explaining how to do it with R‘s glm: to put it shortly, besides specifying the &lt;strong&gt;response&lt;/strong&gt; column (rates between 0 and 1) you have to provide the &lt;strong&gt;weight&lt;/strong&gt; column representing the total number of trials. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/26779&quot;&gt;how to do logistic regression for fractional outcome&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-12-12T13:08:30.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25599" PostTypeId="1" CreationDate="2017-12-12T13:36:51.800" Score="1" ViewCount="426" Body="&lt;p&gt;I have seen &lt;strong&gt;Dataframe&lt;/strong&gt; as new API on Spark2 instead of &lt;strong&gt;RDD&lt;/strong&gt;.So I have following few question about the utility of Spark in terms of time series data.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is forecasting still limited to memory available in one computer&#xA;instead of utilizing distributed fashion of Spark?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Has storing time series data in form of Dataframe rather than RDD&#xA;eliminated the necessity of creating separate &lt;strong&gt;TimeseriesRDD&lt;/strong&gt;&#xA;which some projects did such as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/sryza/spark-timeseries&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sparkts library&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/twosigma/flint&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flint library&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How is processing of TBs of data utilized in forecasting and&#xA;deciding models such as ARIMA, ARMA if the first question is true?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43203" LastEditorUserId="40853" LastEditDate="2017-12-13T16:10:06.487" LastActivityDate="2017-12-13T16:10:06.487" Title="Spark fitting into Data Science Paradigm for timeseries data" Tags="|machine-learning|time-series|bigdata|apache-spark|forecasting|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25601" PostTypeId="1" CreationDate="2017-12-12T15:12:59.767" Score="1" ViewCount="1695" Body="&lt;p&gt;I wrote a simple CNN with a maxpool, a dense layer and a drop layer. Unfortunately there is two messages why it doesn't work. &#xA;In a normal session codes complaines about a misshape between y_conv and y_&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nsorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1400] vs. [100]&#xA; [[Node: eval/Equal = Equal[T=DT_INT64, _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](eval/ArgMax, eval/ArgMax_1)]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I use the invoke stepper from the tensorflow debugger I get the message &quot;Attempting to use uniniatialized variables&quot;. I am very confused. Does anyone have a clue?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = tf.placeholder(tf.float32, [None, 784], name=&quot;entry&quot;)&#xA;&#xA;#layer conv 1&#xA;&#xA;x_image = tf.reshape(x, [-1, 28, 28, 1])&#xA;&#xA;W_conv1 = weight_variable([5, 5, 1, 32], &quot;Wconv1&quot;) # 5 x 5 &#xA;b_conv1 = bias_variable([32], &quot;bconv1&quot;)&#xA;&#xA;h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)&#xA;&#xA;&#xA;#layer pool1&#xA;h_pool1 = max_pool_2x2(h_conv1)&#xA;h_pool1 = tf.Print(h_pool1, [tf.shape(h_pool1)], 'shape=')&#xA;&#xA;# Dense Layer&#xA;&#xA;W_fc1 = weight_variable([ 7 * 64, 1024], &quot;Wconv2&quot;) # 7 x 7 &#xA;b_fc1 = bias_variable([1024], &quot;bconv2&quot;)&#xA;&#xA;h_pool2_flat = tf.reshape(h_pool1, [-1, 7*64])&#xA;h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)&#xA;&#xA;#drop out&#xA;&#xA;keep_prob = tf.placeholder(tf.float32)&#xA;h_fc1_drop = tf.nn.dropout (h_fc1, keep_prob)&#xA;&#xA;W_fc2 = weight_variable([1024, 10], &quot;WconvD&quot;)&#xA;b_fc2 = bias_variable([10], &quot;bconv2&quot;)&#xA;&#xA;# softmax&#xA;&#xA;y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2, name=&quot;y_conv&quot;)&#xA;&#xA;&#xA;with tf.name_scope('train'):&#xA;    y_ = tf.placeholder(tf.float32, shape=[None, 10], name=&quot;y_&quot;)&#xA;    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))&#xA;    train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)&#xA;&#xA;with tf.name_scope('eval'):&#xA;    correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))&#xA;    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These are the definitions of some functions:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def weight_variable(shape, name):&#xA;    initial = tf.truncated_normal(shape, stddev=0.1, name=name) #normalverteilung&#xA;    return tf.Variable(initial)&#xA;&#xA;def bias_variable(shape, name):&#xA;    initial = tf.constant(0.1, shape=shape, name=name)&#xA;    return tf.Variable(initial)&#xA;&#xA;def conv2d(x, W):&#xA;    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')&#xA;&#xA;def max_pool_2x2(x):&#xA;    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name=&quot;maxpool&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43141" LastEditorUserId="43141" LastEditDate="2017-12-12T17:25:10.430" LastActivityDate="2017-12-12T17:25:10.430" Title="How do I fix the misshape in a CNN?" Tags="|neural-network|tensorflow|convolution|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25602" PostTypeId="1" AcceptedAnswerId="25637" CreationDate="2017-12-11T14:33:27.067" Score="0" ViewCount="77" Body="&lt;p&gt;Is there a name for a machine learning algorithm, that learns 'clustering approach' from examples of clustered datapoints (with different numbers of clusters each time)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine the training set as an ensemble of sets with their clusters assigned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone please encountered a similar problem in any literature? My goal is to read something more about this problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example #1: &#xA;Underlying rule: &quot;cluster words to the same group if they do exist on the same page of an imagined storybook&quot;&#xA;Training dataset:&#xA;1) Source values: [banana,car,rock,orange] -&amp;gt; Target groups: [banana and orange],[car],[rock]&#xA;2) Source values: [kiwi,audi,folswagen] -&amp;gt; [kiwi,audi,folswagen]&#xA;...) .. and so on...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;... And that we do not know the original story, but only instances of (datapoints -&gt; their groupping). And also we do not know the rule :)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Example #2: &#xA;Underlying rule: &quot;be evil - cluster like kmeans, but if there seems to be a cloud of points looking like a smiley, do not cluster at all!&quot;&#xA;Training dataset: &#xA;1) Source values: [bunch of points in a shape of two circles] -&amp;gt; [each circle is a cluster]&#xA;2) Source values: [bunch of points and a cloud of points in a shape of smiley] -&amp;gt; []&#xA;...) and so on ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Yes it can be solved by detecting if there is the smiley, but we do not know the rule, we want the algorithm to learn it from the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;maybe the word clustering is not a wise choice from me. The quality of output will not be evaluated by user, but on a big validation set to see if the algorithm did learn from samples it has seen. There also supposedly does exist one best solution, from the nature of our problem (the underlying rule is always present, although not known).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Original, unclear, description before editing:&#xA;&lt;em&gt;The problem is as follows: &#xA;Each row of data is a shuffled list/set of datapoints (source variable) and their assignment to groups (target variable).&#xA;It would seem as a classification task, BUT the groups make sense only to each specific datarow (and their number also varies).&#xA;It would seems as a clustering task, BUT the underlying principle needs to be applied again and again to each different row.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="43384" OwnerDisplayName="Khaj" LastEditorUserId="43384" LastEditDate="2017-12-15T16:24:07.173" LastActivityDate="2017-12-15T16:24:07.173" Title="Name of ML discipline for trainable clusterer - supervised clustering, or group ID assignment" Tags="|machine-learning|neural-network|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25603" PostTypeId="1" CreationDate="2017-12-12T15:41:28.937" Score="0" ViewCount="349" Body="&lt;p&gt;I'm using DBSCAN clustering on a set of documents. The documents' content was converted to TF-IDF matrix, and &lt;strong&gt;I'd like to find consistent ways to evaluate the clusters when no added information is given (labels etc.)&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Metrics comparing clusters&lt;/strong&gt; - a score for each cluster - the goal here is to figure out which clusters are 'better' than others (more dense and more unique). Thought of two metrics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Intra-cluster similarity - measure the similarity of documents inside each cluster - I used the cosine distances between documents in the TF-IDF representation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Inter-cluster similarity - measure the similarity between clusters (to highlight the more unique clusters) - similarly, planning to use the cosine distances between clusters' kernels&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Metrics comparing clustering algorithms&lt;/strong&gt; - a score for the clustering technique (such as Silhouette score). Will be used for hyper-parameter tuning and choosing an algorithm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="38377" LastActivityDate="2017-12-12T23:03:17.830" Title="Clustering documents - how to evaluate results?" Tags="|machine-learning|clustering|unsupervised-learning|model-evaluations|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25604" PostTypeId="1" AcceptedAnswerId="25633" CreationDate="2017-12-12T16:10:19.710" Score="4" ViewCount="3568" Body="&lt;p&gt;Let's say we have $n$ two-dimensional vectors: $$\mathbf{x}_1,\dots,\mathbf{x}_i,\dots,\mathbf{x}_n=(x_{1_1},x_{1_2})^T,\dots,(x_{i_1},x_{i_2})^T,\dots,(x_{n_1},x_{n_2})^T$$ How do you set $\sigma$ for the Gaussian similarity kernel:&#xA;$$s(\mathbf{x}_i,\mathbf{x}_j)=\exp\left(-\frac{||\mathbf{x}_i-\mathbf{x}_j||^2}{2\sigma^2}\right)$$&lt;/p&gt;&#xA;" OwnerUserId="43210" LastEditorUserId="43210" LastEditDate="2017-12-12T16:20:49.030" LastActivityDate="2018-10-04T14:12:50.617" Title="How do you set sigma for the Gaussian similarity kernel?" Tags="|similarity|graphs|variance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25605" PostTypeId="1" CreationDate="2017-12-12T17:44:13.777" Score="4" ViewCount="657" Body="&lt;p&gt;When you build a classification or regression model, you typically split the data into a train data set and a test data set. The test data is a randomly selected subset of the overall data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you are done with the training, you discard the test data, and apply the model you built to new unknown data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in the case of time series, this wouldn't work: You can't just randomly choose any subset of the data for your test data set, since the data is ordered. The test data set has to be composed of later observations of the data, while the train data set is made up of earlier observations of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say for example that you have data for 11 months of the year, and you want to forecast the values for December. You would train your model with the data from January through September, and then test it on the data from October and November. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But after you've successfully trained your model, what do you do with the test data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On one hand, it doesn't make sense to forecast the values for December using a model built with Jan-Sep data. Such a model would miss any important trends that occurred in Oct and Nov. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other if you bring the data from Oct and Nov back into the model, the parameters of the model will change, and so we're no longer sure that we are going to get the same accuracy that we got when we trained the model with just the Jan-Sep data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There seems to be a dilemma here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how is this problem approached when using machine learning models, especially non parametric models like Neural Networks for time series forecasting? Do they incorporate the test data into the model or do they discard it? &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2020-12-17T04:26:11.837" Title="When forecasting time series, how does one incorporate the test data back into the model after training?" Tags="|time-series|training|forecasting|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25606" PostTypeId="1" CreationDate="2017-12-12T18:18:22.963" Score="5" ViewCount="3649" Body="&lt;p&gt;I need help in understanding the gradient flow through a concatenation operation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm implementing a network (mostly a CNN) which has a concatenation operation (in pytorch). The network is defined such that the responses of passing two different images through a CNN are concatenated and passed through another CNN and the training is done end to end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the first CNN is shared between both of the inputs to the concatenation, I was wondering how the gradients should be distributed through the concatenation operation during backprop? I'm not an expert on backprop and this is the first time I'm tinkering with a custom backward implementation so any pointers would be helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can provide more details if you guys need it.&lt;/p&gt;&#xA;" OwnerUserId="43212" LastEditorUserId="29169" LastEditDate="2021-02-13T19:20:22.053" LastActivityDate="2023-11-30T18:01:00.130" Title="Gradient flow through concatenation operation" Tags="|deep-learning|computer-vision|convolutional-neural-network|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25607" PostTypeId="2" ParentId="25605" CreationDate="2017-12-12T18:43:06.127" Score="3" Body="&lt;p&gt;I agree with what Emre has commented on your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have enough data, I would try cross-validating your model by training it on different time sections of your data. For example, train your model on Jan-March data and then make predictions on April's data. Then you could train the model on Feb-April data, and predict May's data. Essentially creating a few training and testing sets to see how well your model generalizes. Once you feel you are getting good results, you could train your final &quot;production&quot; model on all of your available data and then keep an eye on how it does when real data comes in and make adjustments on what you see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used a similar approach in the past on accelerometer data to classify exercises from a specific workout, and it worked well. Maybe something similar will work for you.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-12T18:43:06.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25608" PostTypeId="1" AcceptedAnswerId="25611" CreationDate="2017-12-12T19:40:07.250" Score="6" ViewCount="5106" Body="&lt;p&gt;I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(&lt;a href=&quot;https://www.archives.gov/veterans/locate-service-members.html&quot; rel=&quot;noreferrer&quot;&gt;Link&lt;/a&gt;) and Canada Industrial Data (&lt;a href=&quot;https://www.ic.gc.ca/app/ccc/sld/cmpny.do?tag=010&amp;amp;profileId=2161&amp;amp;lang=eng&quot; rel=&quot;noreferrer&quot;&gt;Link&lt;/a&gt;) and not sure if I should do. I personally don't hesitate to scrape govt. data but sometimes it may take the whole company in trouble specially in US. I need some valuable suggestions how to classify any website whether we should scrape or not.&lt;/p&gt;&#xA;" OwnerUserId="42648" LastActivityDate="2018-07-09T20:25:06.540" Title="Can I scrape data from government websites if there is no mention about commercial usage?" Tags="|data|scraping|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25610" PostTypeId="2" ParentId="25603" CreationDate="2017-12-12T23:03:17.830" Score="2" Body="&lt;p&gt;None of the internal evaluation metrics will work well on &lt;em&gt;text&lt;/em&gt; in my experience. Probably because of the curse of dimensionality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, DBSCAN does not cluster everything, but can also produce noise points. Few evaluation methods (and even fewer  implementations...) handle it well that &lt;strong&gt;noise is &lt;em&gt;not&lt;/em&gt; a cluster&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-12T23:03:17.830" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25611" PostTypeId="2" ParentId="25608" CreationDate="2017-12-13T00:33:11.700" Score="4" Body="&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; It all depends on how you plan to use the data you scraped. If you plan to upload it to your website, that is generally not OK since content can be copy write protected and you could get sued. If you plan to just use the data for personal use offline you should generally not have a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you take a look at these links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.archives.gov/global-pages/privacy.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.archives.gov/global-pages/privacy.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.ic.gc.ca/eic/site/icgc.nsf/eng/h_07033.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.ic.gc.ca/eic/site/icgc.nsf/eng/h_07033.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those are the privacy pages found at the bottom of the links you provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will find all the information you need there, including email addresses to which you can write to if you have any questions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though to help you out a bit, both of the sites you provided state that most of the content or all of it &lt;strong&gt;is&lt;/strong&gt; copy write protected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In conclusion, ask permission if you are going to re post the data somewhere on the internet. Websites that allow you to do so clearly state it in their privacy policies or terms of use.&lt;/p&gt;&#xA;" OwnerUserId="43228" LastEditorUserId="43228" LastEditDate="2017-12-13T00:40:41.233" LastActivityDate="2017-12-13T00:40:41.233" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25613" PostTypeId="2" ParentId="25577" CreationDate="2017-12-13T02:58:40.690" Score="1" Body="&lt;p&gt;I would agree with what @Berry said.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think using a &lt;strong&gt;Unique Identifier&lt;/strong&gt; is not suggestible as it is might be &lt;strong&gt;Over-fitting/ Over-train&lt;/strong&gt; your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before deciding to go forward and use a Unique Identifier I would suggest you to check the model by taking some random data by giving some random identifier and see what are its results. If my guess is right, it would worsen the model. BY that you can prove yourself that it is not right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The possible reasons for that to happen are:&#xA;you can go through the &lt;a href=&quot;https://stats.stackexchange.com/questions/224565/overfitting-due-to-a-unique-identifier-among-features&quot;&gt;link&lt;/a&gt;, it is specified clearly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, even if you try to normalize and use an unique identifier it is going to worsen the model. Unless you have a method which says that the Unique Identifier is assigned based on some value(explained in 1st para by Ryan in the link attached).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit:&lt;/em&gt;&#xA;If the target variable is skewed towards one feature then we need to balance the training set and train the model with the new balance data. using the test data you can predict. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To over come the imbalance of data, we can use SMOTE,ROSE packages. Generally SMOTE is rated better over ROSE but suggestible to try both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implementation of &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/over-sampling/plot_smote.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt;, a &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; to go through different type of over and under-sampling techniques&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go through this &lt;a href=&quot;https://stats.stackexchange.com/questions/166458/rose-and-smote-oversampling-methods&quot;&gt;link&lt;/a&gt; for better understanding on ROSE and SMOTE&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any additional questions!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-16T02:31:55.413" LastActivityDate="2017-12-16T02:31:55.413" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="25614" PostTypeId="1" CreationDate="2017-12-13T03:09:00.640" Score="1" ViewCount="608" Body="&lt;p&gt;I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor lookup. Each point cloud has roughly 500 points each. I am looking at using a cached KD tree but I am unsure about how to turn a KD tree into a cached KD tree. Is there an implementation in Scipy? SVD in python seems fast enough for my needs at the moment, but I would also be open to any known faster SVD solvers. My current code for the ICP is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;tree=spatial.KDTree(TimeStep1)&#xA;&amp;gt;qHat=TimeStep2&#xA;&amp;gt; for i in range(10):&#xA;&amp;gt;         Phat=tree.query(qHat,eps=.15,distance_upper_bound=2)&#xA;&amp;gt;         H = np.dot(qHat.T, TimeStep1[Phat[1]-1,:])&#xA;&amp;gt;         U, S, Vt = np.linalg.svd(H)&#xA;&amp;gt;         R = np.dot(Vt.T, U.T)&#xA;&amp;gt;         if np.linalg.det(R) &amp;lt; 0:&#xA;&amp;gt;             Vt[2,:] *= -1&#xA;&amp;gt;             R = np.dot(Vt.T, U.T)&#xA;&amp;gt;             Yes.append(1)&#xA;&amp;gt;         Eu=rotationMatrixToEulerAngles(R)&#xA;&amp;gt;         T=TimeStep1[Phat[1]-1,:].T-np.dot(R,qHat.T)&#xA;&amp;gt;         T=TimeStep1[Phat[1]-1,:]-qHat&#xA;&amp;gt;         T=T.mean(axis=0)&#xA;&amp;gt;         qHat=np.dot(R,TimeStep2.T).T+T&#xA;&amp;gt;         qHat=TimeStep2+T.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32936" LastActivityDate="2018-09-10T17:44:13.430" Title="Is there a quick way to speed up ICP in python using a cached KD-tree" Tags="|python|optimization|linear-algebra|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25616" PostTypeId="1" CreationDate="2017-12-13T06:19:04.850" Score="5" ViewCount="1101" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/plw7X.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/plw7X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the confusion matrix I got when I was playing with Forest Type Cover Kaggle dataset : &lt;a href=&quot;https://www.kaggle.com/c/forest-cover-type-prediction/data&quot; rel=&quot;noreferrer&quot;&gt;Link&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the matrix, light color and higher numbers represent higher error rates, so as you can see, lots of mis-classification happened between class 1 and 0 . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder what kind of methods I can use to reduce these two error rates though improvements have been made through combing two classifiers, Random Forest and Extra Tree. Will stacking help in this case? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data can be found on &#xA;&lt;a href=&quot;https://www.kaggle.com/c/forest-cover-type-prediction/data&quot; rel=&quot;noreferrer&quot;&gt;https://www.kaggle.com/c/forest-cover-type-prediction/data&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43238" LastEditorUserId="43238" LastEditDate="2017-12-13T07:59:30.550" LastActivityDate="2019-04-08T18:02:20.433" Title="Dealing with unbalanced error rate in confusion matrix" Tags="|machine-learning|classification|confusion-matrix|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25617" PostTypeId="2" ParentId="25616" CreationDate="2017-12-13T06:37:56.157" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think &lt;strong&gt;Ensemble Method&lt;/strong&gt; is very &lt;strong&gt;tricky&lt;/strong&gt;. when one of the model doesn't work well then the accuracy of the Ensemble also goes down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance let us consider that you are using RandomForest(RF) and Rpart for classification and RF accuracy is 90% and Rpart accuracy is 60%. &lt;em&gt;If you ensemble these 2 models then the Ensemble accuracy goes down&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to your scenario, you need to be very careful at the time of stacking, you need to select the models that are performing moderately and then stack them to improve the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is the distribution of 0/1's, if they are &lt;strong&gt;imbalanced&lt;/strong&gt; then you need to balance to improve the accuracy of the model. To handle imbalance data situation we use packages like &lt;strong&gt;SMOTE,ROSE&lt;/strong&gt; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Feature Engineering&lt;/strong&gt; like adding external factors or adding new features, might help you to improve your models &lt;strong&gt;accuracy&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any additional questions. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-13T06:37:56.157" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="25619" PostTypeId="1" AcceptedAnswerId="25663" CreationDate="2017-12-13T08:29:42.043" Score="1" ViewCount="126" Body="&lt;p&gt;I have a problem that doesn't seem to fall into a common machine learning category, and I was wondering if this still could potentially be solved with ML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem: I have two signals recorded from two sensors, and would like to determine whether they are correlated (i.e. record the same physical event) or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The catch&lt;/strong&gt;: I don't have access to the full signal time series of both sensors, but only one at a time - I can only exchange a small descriptor on the order of 32 bits to see if the signals match or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our current approach is to calculate a bunch of numerical signal features such as mean, derivative, zero-crossings, FFT etc. and see which ones provide the best correlation - but that seems to be a lot of guesswork and doesn't work very well in any case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I had the following idea:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Start with a neural network which takes a fixed window out of the signal (+ possibly the FFT of that window) as an input, and produces a 32-bit output&lt;/li&gt;&#xA;&lt;li&gt;Pick two random correlated samples out of the pool of examples, and run the network &lt;em&gt;&lt;strong&gt;twice&lt;/strong&gt;&lt;/em&gt;, once with each sample (and its FFT)&lt;/li&gt;&#xA;&lt;li&gt;Take the &lt;strong&gt;&lt;em&gt;difference&lt;/em&gt;&lt;/strong&gt; between the two output values as error measure and perform backpropagation as usual&lt;/li&gt;&#xA;&lt;li&gt;Repeat from 2. until the difference for all examples is below a threshold&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here are my questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this approach seem feasible at all?&lt;/li&gt;&#xA;&lt;li&gt;As someone relatively new to machine learning, how would I implement this?&lt;/li&gt;&#xA;&lt;li&gt;I've had a look at Keras - would this be a suitable starting point?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance, and best regards, Florian&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addendum: I've found this somewhat related post (&lt;a href=&quot;https://datascience.stackexchange.com/questions/14503/is-it-possible-using-tensorflow-to-create-a-neural-network-that-maps-a-certain-i&quot;&gt;Is it possible using tensorflow to create a neural network that maps a certain input to a certain output?&lt;/a&gt;), but I don't think that this is the same problem, as I don't actually care what the output looks like, just that it is as unique as possible for each matching pair of samples.&lt;/p&gt;&#xA;" OwnerUserId="43243" LastEditorUserId="43243" LastEditDate="2017-12-13T09:02:16.593" LastActivityDate="2017-12-14T21:14:56.427" Title="Using ML to create unique descriptors?" Tags="|machine-learning|neural-network|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25621" PostTypeId="1" CreationDate="2017-12-13T08:49:19.670" Score="4" ViewCount="37781" Body="&lt;p&gt;I've trained a model with Keras, saved it and when I'm trying to apply it on new data, I'm encountering an error :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the code for training and saving the trained model :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Import necessary modules&#xA;import numpy as np  # numpy is just used for reading the data&#xA;import keras&#xA;from keras.layers import Dense&#xA;from keras.models import Sequential&#xA;from keras.models import load_model # To save and load model&#xA;&#xA;&#xA;# fix random seed for reproducibility&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;&#xA;# load the dataset&#xA;dataset = np.loadtxt(&quot;modiftrain.csv&quot;, delimiter=&quot;;&quot;)&#xA;&#xA;# split into input (X) and output (Y) variables&#xA;X = dataset[:,0:5]&#xA;Y = dataset[:,5]&#xA;&#xA;# create model&#xA;model = Sequential()&#xA;&#xA;# Add the first layer&#xA;# input_dim=   has to be the number of input variables. &#xA;# It represent the number of inputs in the first layer,one per column &#xA;model.add(Dense(12, input_dim=5, activation='relu'))&#xA;&#xA;# Add the second layer&#xA;model.add(Dense(8, activation='relu'))&#xA;&#xA;# Add the output layer&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;# Compile model&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;# Fit the model&#xA;model.fit(X, Y, epochs=150, batch_size=10)&#xA;&#xA;# Evaluate the model&#xA;scores = model.evaluate(X, Y)&#xA;print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&#xA;# Save the model&#xA;model.save('model_file.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the code for predicting :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np &#xA;from keras.models import load_model # To save and load model&#xA;&#xA;# Load the model&#xA;my_model = load_model('model_file.h5')&#xA;&#xA;# Load the test data file and make predictions on it&#xA;predictions = my_model.predict(np.loadtxt(&quot;modiftest.csv&quot;, delimiter=&quot;;&quot;))&#xA;&#xA;print(predictions.shape)&#xA;&#xA;my_predictions=my_model.predict(predictions)&#xA;&#xA;print(my_predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here comes the error :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;predict01.py&quot;, line 14, in &amp;lt;module&amp;gt;&#xA;    my_predictions=my_model.predict(X_predictions)&#xA;  File &quot;C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\models.py&quot;, line 913, in predict&#xA;    return self.model.predict(x, batch_size=batch_size, verbose=verbose)&#xA;  File &quot;C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py&quot;, line 1695, in predict&#xA;    check_batch_axis=False)&#xA;  File &quot;C:\Users\Philippe\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\training.py&quot;, line 144, in _standardize_input_data&#xA;    str(array.shape))&#xA;ValueError: Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help.&lt;/p&gt;&#xA;" OwnerUserId="43248" LastEditorUserId="43248" LastEditDate="2017-12-14T06:07:44.723" LastActivityDate="2020-02-29T06:23:09.277" Title="Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)" Tags="|keras|" AnswerCount="3" CommentCount="2" ClosedDate="2020-03-02T16:42:40.040" ContentLicense="CC BY-SA 3.0" />
  <row Id="25622" PostTypeId="1" CreationDate="2017-12-13T09:11:36.180" Score="4" ViewCount="67" Body="&lt;p&gt;Let's say I start with a standard conv net architecture capable of 99% accuracy on MNIST such as &lt;a href=&quot;https://www.tensorflow.org/get_started/mnist/pros&quot; rel=&quot;noreferrer&quot;&gt;this one&lt;/a&gt;, but let's say I merge in an &quot;easy&quot; feature to the fully-connecter layer such as a vector of length 10 that encodes the correct output digit 95% of the time and a random digit otherwise.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Will my network ever reach 99% accuracy&lt;/li&gt;&#xA;&lt;li&gt;Will it take longer to get there?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think the answer the question 1 is yes, because we will should always be able to find a downhill in the error before the conv net path has reached its full potential. In fact I think it may surpass the accuracy of the original architecture since we are leaking information about the correct output labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am not so sure about question 2. Does this make the shape of error function unfavorable in any way? I can't decide whether reaching the same accuracy of the original architecture will be faster, slower, or exactly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully there is an easy answer, otherwise I will run an experiment and report back!&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-13T09:53:47.567" LastActivityDate="2017-12-13T12:53:32.767" Title="Will my neural network get lazy if I give it an &quot;easy&quot; feature?" Tags="|neural-network|training|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25624" PostTypeId="2" ParentId="25606" CreationDate="2017-12-13T11:41:10.517" Score="1" Body="&lt;p&gt;For concatenation, the gradient values during back propagation split to their respective source layers. There is no direct interaction between gradients in either of the source layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The layer immediately after the concatenated layer does interact with both networks, and it will have some weight parameters that multiply outputs from network A and some that multiply outputs from network B. There will not be any parameters that multiply outputs from both layers (unless you are forcing them to be the same through weight sharing, but that won't be the case if for example you are stacking features from both starting networks).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only issue you might have is clearly identifying which parameters link to each original network. That is an implementation detail, so you would need to share your code so far in order to debug that if it goes wrong.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-12-13T11:41:10.517" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25625" PostTypeId="2" ParentId="14064" CreationDate="2017-12-13T12:44:10.680" Score="1" Body="&lt;p&gt;Implementing this with a recurrent neural network is not that difficult. My suggestion is close to what you already suggested but instead of sampling different points in time I would just broadcast your target over all time steps. If there are a total of N time steps, and we want to predict on steps t=1 to t=N-1, our X is a matrix of shape [N-1, k] with k the number of features. Y is a sequence of shape [N-1] with all the values being equal to EventN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is similar to what you actually want to achieve, namely at every time step you want to predict the final value. Your loss function will be calculated over all time steps and backpropagated through time. It's possible that you might want to weigh your function as a function of the time step, maybe later predictions are more or less important, you'll need to do some analysis on this. Another recommendation would be to add some feature of time in your feature explicitly, that might help the network.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-12-13T12:44:10.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25626" PostTypeId="2" ParentId="25622" CreationDate="2017-12-13T12:49:52.513" Score="2" Body="&lt;p&gt;Interesting theoretical question. While I cannot answer this with 100% certainty, my own intuition and experience say:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I would be very surprised if this is not the case. If we look at different problems, where we have one very important feature that holds most of the information but some other features with some signal in there the model generally still improves by adding these lower value features. That is exactly what you are doing here, you have the MNIST pixels that individually hold little value but combined tell you quite a bit. The value of these pixels goes down quite a bit by adding such a strong feature, but the value is absolutely not reduced to 0. I think in theory this should lead to a strictly better model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I think it will take around the same time to converge on average. Fitting to the one-hot encoded feature will be very fast which means most of the time will be spent on fitting to the features, which is a similar problem as without the feature. However, instead of going from 10% to 99% accuracy, we are now kind of going from 95% accuracy to 99.7% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm less certain about the second question but I think this is what you would find if you do the (interesting) experiment. Let us know if you do implement it!&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2017-12-13T12:49:52.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25627" PostTypeId="2" ParentId="25622" CreationDate="2017-12-13T12:53:32.767" Score="2" Body="&lt;p&gt;This should also depend on the network architecture:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the cost-function causes a high penalty for the remaining 5%, then they should be learned as well.&lt;/li&gt;&#xA;&lt;li&gt;If the architecture uses drop-out, then this will force the NN to not rely on the one easy feature.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="34130" LastActivityDate="2017-12-13T12:53:32.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25628" PostTypeId="1" CreationDate="2017-12-13T13:01:29.127" Score="-1" ViewCount="43" Body="&lt;p&gt;I have some variables $X_1,X_2,X_3,X_4,X_5,X_6$ (may be correlated with each other) and my dependent variable is $Y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know the contribution of two groups $X_1,X_2,X_3$ and $X_4,X_5,X_6$ to $Y$ separately, i.e. given only values of $X_1,X_2,X_3$, what will be the predicted $Y$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this same as calculating importance matrix? Or what other approach should I use? This is not a linear regression problem, but if it was, I could have used regression coefficients directly, right?&lt;/p&gt;&#xA;" OwnerUserId="43262" LastEditorUserId="40853" LastEditDate="2017-12-14T00:55:52.387" LastActivityDate="2017-12-14T00:55:52.387" Title="How to predict output due to some variables out of all variables which affect it?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25629" PostTypeId="2" ParentId="25628" CreationDate="2017-12-13T14:01:46.667" Score="0" Body="&lt;p&gt;You can use &lt;a href=&quot;http://www.sthda.com/english/wiki/one-way-anova-test-in-r&quot; rel=&quot;nofollow noreferrer&quot;&gt;ANOVA&lt;/a&gt; in this case to get the relation between different features(categorical variables) within them and with respect to the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to check the feature importance with respect to target variable then you can this package &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt; in R, implementable in python too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know more, which test when can you use it, go through this &lt;a href=&quot;https://stats.idre.ucla.edu/spss/whatstat/what-statistical-analysis-should-i-usestatistical-analyses-using-spss/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any doubts!&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2017-12-13T14:01:46.667" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25630" PostTypeId="2" ParentId="25588" CreationDate="2017-12-13T14:03:47.950" Score="1" Body="&lt;p&gt;The error is related to this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It should be modified as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(&quot;Tuned rf best params: {}&quot;.format(rg_cv.best_params_))&#xA;be=rg_cv.best_estimator_&#xA;best=be.fit(X_train,y_train)&#xA;thresholds = np.sort(rg_cv.best_estimator_.named_steps[&quot;clf&quot;].feature_importances_)&#xA;for thresh in thresholds:&#xA;    # select features using threshold&#xA;    selection = SelectFromModel(be.steps [1][1], threshold=thresh, prefit=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastActivityDate="2017-12-13T14:03:47.950" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25631" PostTypeId="1" CreationDate="2017-12-13T14:28:36.460" Score="1" ViewCount="986" Body="&lt;p&gt;I recently used a prediction algorithm to try to predict how much an user is willing to interact with an item using ALS (Matrix Factorisation) but I had to &quot;condense&quot; multiple attributes into one, deciding some arbitrary weights for the values to be able to calculate the rating to use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works (sort of) but I think it is definitely wrong because weighting the attributes manually is static and do not adapt to the data themselves dynamically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking to switch to a Distributed Random Forest algorithm but even if it would use all the attributes (randomly) they would not be used all together and this makes it not fit for my purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During my researches I found out that the Recurrent neural networks may be fit for what I am trying to achieve but before focusing on them, because I am not really an expert in the field, I would like to have suggestions :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Potentially I could take advantage of the time-series feature of the RNNs but for now I prefer to focus on supporting the set of attributes I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently the &quot;rating&quot; for the ALS is calculated weighting some attributes like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;how many times the user viewed the item page&lt;/li&gt;&#xA;&lt;li&gt;how many times the user viewed the category page&lt;/li&gt;&#xA;&lt;li&gt;how many times the user viewed the sub category page&lt;/li&gt;&#xA;&lt;li&gt;how many times the user bought something from the category&lt;/li&gt;&#xA;&lt;li&gt;how many times the user bought something from the sub category&lt;/li&gt;&#xA;&lt;li&gt;how much time ago the user bought the item&lt;/li&gt;&#xA;&lt;li&gt;how much time ago the user bought something from the same category&lt;/li&gt;&#xA;&lt;li&gt;how much time ago the user bought something from the same sub category&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;(and similar)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it worth trying with an RNN? If yes, should I try to go with tensorflow or caffe?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="43265" LastActivityDate="2017-12-13T16:33:43.863" Title="Prediction with multiple features (attributes)" Tags="|machine-learning|rnn|cnn|matrix-factorisation|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-14T00:58:27.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="25632" PostTypeId="1" AcceptedAnswerId="25642" CreationDate="2017-12-13T15:26:42.083" Score="-2" ViewCount="58" Body="&lt;p&gt;I'm building a custom Convolutional Neural Network for image recognition.  I'm running into the issue of only having around 100 images or so to train and test on.  From my research and model results, this is not enough.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone who of a service for hire to have someone search web/research and build up images based on certain criteria?  &lt;/p&gt;&#xA;" OwnerUserId="32754" LastEditorUserId="28175" LastEditDate="2017-12-14T00:57:21.433" LastActivityDate="2017-12-14T00:57:21.433" Title="Building CNN, Need More Images" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ClosedDate="2017-12-14T01:00:22.097" ContentLicense="CC BY-SA 3.0" />
  <row Id="25633" PostTypeId="2" ParentId="25604" CreationDate="2017-12-13T15:31:45.140" Score="3" Body="&lt;h2&gt;Updated Answer&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;According to a reference paper in Spectral Clustering (&lt;a href=&quot;https://www.cs.cmu.edu/~aarti/Class/10701/readings/Luxburg06_TR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;von Luxburg&lt;/a&gt;) the $\sigma$ is simply set to 1. A further tuning can be applied with some visualization inspection but I did not find any discussion regarding setting this parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using code snippet below you see the effect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from scipy.spatial.distance import pdist&#xA;&#xA;def gausker(x1,x2,sigma):&#xA;    dist = np.linalg.norm(x1-x2)&#xA;    return np.exp(-dist**2/(2.*(sigma**2.)))&#xA;&#xA;data = np.array([[0,0],[1,1],[1,0],[0,1],[10,10],[10,9],[9,10],[9,9]])&#xA;plt.figure()&#xA;plt.plot(data[:,0],data[:,1],'o',ms=20)&#xA;plt.show()&#xA;s = np.std(pdist(data))&#xA;for sigma in [1, s, 10, 100]:&#xA;    gaus = np.zeros((8,8))&#xA;    for ii in range(8):&#xA;        for jj in range(8):&#xA;            gaus[ii,jj] = gausker(data[ii,:],data[jj,:],sigma)&#xA;    plt.figure()&#xA;    plt.imshow(gaus,extent=[0, 1, 0, 1])&#xA;    plt.colorbar()&#xA;    plt.title(str(sigma))&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For Machine Learning algorithms is better to have more distinction. The Gaussian similarity kernel cares about local similarities. The image show the kernel for $\sigma=1$ &lt;a href=&quot;https://i.stack.imgur.com/JDHUt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JDHUt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conceptually is similar to a k-nearest neighbors graph, since it considers local neighborhood and almost disregards the relationship between two nodes far apart.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Dp0la.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Dp0la.png&quot; alt=&quot;Data&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xg9o2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xg9o2.png&quot; alt=&quot;$\sigma 1$&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dG36n.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dG36n.png&quot; alt=&quot;stddev&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oMgVV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oMgVV.png&quot; alt=&quot;$\sigma 10$&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/79iTi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/79iTi.png&quot; alt=&quot;$\sigma 100$&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="43210" LastEditDate="2017-12-19T00:38:53.873" LastActivityDate="2017-12-19T00:38:53.873" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25634" PostTypeId="2" ParentId="18103" CreationDate="2017-12-13T15:37:57.170" Score="3" Body="&lt;p&gt;I have the exact same issue with a DNN that I am currently building. Taking my data set and synthesizing new data with a GAN seems like a great idea. But the GAN itself will only learn to output images with the same image variance and standard deviations as was learned in the training set. So your newly generated data will simply represent more permutations of the same sample distribution. This will help your NN train better on the same distribution, therefore it may lead to greater over training. &lt;/p&gt;&#xA;" OwnerUserId="43270" LastActivityDate="2017-12-13T15:37:57.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25635" PostTypeId="1" CreationDate="2017-12-13T15:40:01.427" Score="5" ViewCount="995" Body="&lt;p&gt;I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like &lt;code&gt;org.java.somepackage.validateLogin&lt;/code&gt;). What I want to do is to give a &lt;strong&gt;representative label&lt;/strong&gt; for each cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the sake of simplicity, let's assume I'm able to clean the data (i.e. break 'validateLogin' to 'validate' and 'login')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done a little research on the subjects of &lt;em&gt;topics modeling&lt;/em&gt; and &lt;em&gt;cluster labeling&lt;/em&gt; and encountered a few algorithms, such as: LDA, NMF and TF-IDF (not quite an algorithm on itself)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, many algorithms are documents oriented which are reach of words and not short-text/labels oriented. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is worth mentioning:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;We can use the fact that different clusters might have different labels, so maybe a proper label for a cluster could be unique words in the overall bag of words (that can be done with TF-IDF I guess) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Labels can be one word but can also imply a hierarchy (i.e. &lt;code&gt;packageA.packageB.packageC.funcName&lt;/code&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'd be glad if you could give me your insights on this problem and what approach could fit here.&lt;/p&gt;&#xA;" OwnerUserId="43268" LastEditorUserId="29575" LastEditDate="2017-12-13T17:10:47.540" LastActivityDate="2018-01-09T13:58:06.827" Title="Topic modeling for short length sentences" Tags="|machine-learning|nlp|clustering|text-mining|lda|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25636" PostTypeId="2" ParentId="25631" CreationDate="2017-12-13T16:33:43.863" Score="1" Body="&lt;p&gt;There is nothing &quot;wrong&quot; about weighting different actions to get a measure of preference between a user and an item, and then using ALS to compute a latent feature space. This is quite common in my experience. However, the consequence of ALS is that it can detect categories and sub-categories using &lt;em&gt;individual&lt;/em&gt; user/item interactions. So, I don't see why you need to be adding higher level preference data into your user/item matrix. The weighting should be done using only behavior that indicates preference between a specific user and specific item, not user/category or user/sub-category interactions - ALS will detect if a user likes a category on its own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to weight the preference behavior is somewhat tricky, but you can use heuristics or information from your data. For example, a page view is not as strong as a product purchase and should be weighted less. You might look at the distribution of page views vs purchases to get an idea how common one is vs the other, to develop a weighting scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure that you could also use an RNN in a classification setting, to predict the next item that a user will purchase given an input sequence of the last n items they've purchased or interacted with. The framework you use depends on your preference, for Python based Keras, MXNet, PyTorch, and Tensorflow are all worthy. For easy Spark integration and JVM-based, DL4J is a good choice.&lt;/p&gt;&#xA;" OwnerUserId="43273" LastActivityDate="2017-12-13T16:33:43.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25637" PostTypeId="2" ParentId="25602" CreationDate="2017-12-13T16:47:57.193" Score="1" Body="&lt;p&gt;Perhaps you are looking for some combination of &lt;strong&gt;Meta clustering&lt;/strong&gt; and &lt;strong&gt;Ensemble clustering&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.cs.cornell.edu/~caruana/ICDM06.metaclust.caruana.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Meta clustering&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A sampling-based approach that searches for distance&#xA;  metrics that yield the clusterings most useful to the user&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Generate many good, yet qualitatively different, baselevel&#xA;  clusterings of the same data.&lt;/li&gt;&#xA;  &lt;li&gt;Measure the similarity between the base-level clusterings&#xA;  generated in the first step so that similar clusterings&#xA;  can be grouped together.&lt;/li&gt;&#xA;  &lt;li&gt;Organize the base-level clusterings at a meta level (either&#xA;  by clustering or by low-dimension projection) and&#xA;  present them to users.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensemble Clustering&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43276" LastActivityDate="2017-12-13T16:47:57.193" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25638" PostTypeId="2" ParentId="24660" CreationDate="2017-12-13T16:54:53.893" Score="0" Body="&lt;p&gt;Having more data may not mean that you have online learning.&#xA;Above answers are correct if you assume that the coming new data samples will not have any concept drift. Meaning the true data generating distribution may change in a way that your current ( most recent model) is in a point that is very far away from the optimal minimum for the new data distribution. &#xA;I recommend you to read more about “concept drift” and understand if your data will have any concept drift.&#xA;Try to search for how NN deals with concept drifts. Unfortunately I am not aware of any technique or method can make NN adapt to concept drift.&#xA;If you do not expect any concept drift, then saving the check point and continue training will be good solution if you are using stochastic gradient descent &lt;/p&gt;&#xA;" OwnerUserId="17696" LastActivityDate="2017-12-13T16:54:53.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25639" PostTypeId="1" CreationDate="2017-12-13T17:13:20.400" Score="4" ViewCount="1303" Body="&lt;p&gt;I was recently reading the &lt;code&gt;Knowledge Distillation&lt;/code&gt; paper, and encountered the term &lt;code&gt;smooth probabilities&lt;/code&gt;. The term was used to denote when the logits were divided a temperature. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/EYRVX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EYRVX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Neural networks typically produce class probabilities by using a&#xA;  &lt;code&gt;softmax&lt;/code&gt; output layer that converts the logit, &lt;code&gt;zi&lt;/code&gt;, computed for&#xA;  each class into a probability, &lt;code&gt;qi&lt;/code&gt;, by comparing &lt;code&gt;zi&lt;/code&gt; with the other&#xA;  logits where &lt;code&gt;T&lt;/code&gt; is a temperature that is normally set to 1. Using a&#xA;  higher value for &lt;code&gt;T&lt;/code&gt; produces a softer probability distribution over&#xA;  classes.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does that mean intuitively?&lt;/p&gt;&#xA;" OwnerUserId="16338" LastActivityDate="2018-01-13T13:03:52.687" Title="what does smooth/soft probablity mean?" Tags="|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25642" PostTypeId="2" ParentId="25632" CreationDate="2017-12-13T18:31:34.210" Score="1" Body="&lt;p&gt;I recommend you using &lt;code&gt;Keras&lt;/code&gt; and employing its pre-trained models. Because of low number of data-set, you should use &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwiPoa-KzofYAhUDaVAKHXhbDkAQFggpMAE&amp;amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTransfer_learning&amp;amp;usg=AOvVaw2HhdpOpHnaIlHbqcB21o43&quot; rel=&quot;nofollow noreferrer&quot;&gt;transfer learning&lt;/a&gt;. There are lots of researches about that like &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=6&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwiPoa-KzofYAhUDaVAKHXhbDkAQFghJMAU&amp;amp;url=http%3A%2F%2Fcs231n.github.io%2Ftransfer-learning%2F&amp;amp;usg=AOvVaw3meugAV_bCpOZhjgZv3eRW&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Based on the data that you have, you should choose a model which is appropriate for your task and have been trained for tasks which are like yours, and then use it. I guess &lt;em&gt;ResNet&lt;/em&gt; and &lt;em&gt;GoogleNet&lt;/em&gt; already have been trained on &lt;em&gt;ImageNet&lt;/em&gt; data set and are in the &lt;code&gt;Keras&lt;/code&gt;. You have to freeze the weights of the convolution and dense layers. You should change the soft-max layer with your own. In this kind of learning, the pre-trained model has already the ability to find features. You are just supposed to let it learn how to classify your data. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-13T20:59:33.693" LastActivityDate="2017-12-13T20:59:33.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25643" PostTypeId="2" ParentId="25632" CreationDate="2017-12-13T20:10:45.777" Score="2" Body="&lt;p&gt;You should apply data augmentation to generate more data. Check &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://machinelearningmastery.com/image-augmentation-deep-learning-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2017-12-13T20:10:45.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25650" PostTypeId="1" AcceptedAnswerId="25657" CreationDate="2017-12-14T01:53:22.020" Score="21" ViewCount="55522" Body="&lt;p&gt;I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we use these deep learning architectures in unsupervised problems?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="29575" LastEditDate="2017-12-28T15:48:12.493" LastActivityDate="2018-12-07T10:05:28.033" Title="What is LSTM, BiLSTM and when to use them?" Tags="|machine-learning|deep-learning|rnn|lstm|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25651" PostTypeId="1" CreationDate="2017-12-14T02:30:41.810" Score="1" ViewCount="522" Body="&lt;p&gt;As is clear from the figure, the blue points, which don't follow the trend, are anomalous points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering about the best non-parametric method to detect those points. I have tested some outlier detection methods such as standard deviation, etc. but they don't provide good results while it is clear from the figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nEHDI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nEHDI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="29575" LastEditDate="2017-12-17T20:07:04.593" LastActivityDate="2018-03-10T13:38:09.760" Title="How to detect anomalous points" Tags="|statistics|visualization|anomaly-detection|outlier|anomaly|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25652" PostTypeId="2" ParentId="25651" CreationDate="2017-12-14T02:42:43.350" Score="0" Body="&lt;p&gt;A solution can be using DBSCAN algorithm to cluster data. Then, if you set a proper radius for the DBSCAN algorithm, you will get three clusters. Therefore, you can detect some clusters as anomaly that number of their members is less than a threshold. &lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-12-14T02:42:43.350" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25653" PostTypeId="2" ParentId="25651" CreationDate="2017-12-14T03:23:14.087" Score="1" Body="&lt;p&gt;In your example what differentiates the clusters is not the raw value but rapid departure from previous points.  I might look into change-point detection. Nonparametic, but still requires some fiddling with tuning parameters. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2017-12-14T03:23:14.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25655" PostTypeId="1" CreationDate="2017-12-14T03:58:38.120" Score="2" ViewCount="10265" Body="&lt;p&gt;I'm writing code that captures statistics from Spark runs and stores the results in XML files. I'd like to capture, ideally with the REST API, information about the number of executors and the number and size of the Data Frames that were created. Is there an easy way to do this? What are the API calls?&lt;/p&gt;&#xA;" OwnerUserId="13672" LastActivityDate="2018-09-16T03:00:42.850" Title="API to find out how many executors are running my Spark jobs?" Tags="|apache-spark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25657" PostTypeId="2" ParentId="25650" CreationDate="2017-12-14T06:06:07.563" Score="15" Body="&lt;p&gt;&lt;code&gt;RNN&lt;/code&gt; architectures like &lt;code&gt;LSTM&lt;/code&gt; and &lt;code&gt;BiLSTM&lt;/code&gt; are used in occasions where the learning problem is sequential, e.g. you have a video and you want to know what is that all about or you want an agent to read a line of document for you which is an image of text and is not in text format. I highly encourage you take a look at &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;LSTMs&lt;/code&gt; and their bidirectional variants are popular because they have tried to learn how and when to forget and when not to using gates in their architecture. In previous &lt;code&gt;RNN&lt;/code&gt; architectures, vanishing gradients was a big problem and caused those nets not to learn so much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Bidirectional &lt;code&gt;LSTMs&lt;/code&gt;, you feed the learning algorithm with the original data once from beginning to the end and once from end to beginning. There are debates here but it usually learns faster than one-directional approach although it depends on the task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can use them in unsupervised learning too depending on your task. take a look at &lt;a href=&quot;http://deeplearning.stanford.edu/tutorial/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/selftaughtlearning/SelfTaughtLearning/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-14T09:29:41.017" LastActivityDate="2017-12-14T09:29:41.017" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25659" PostTypeId="2" ParentId="25651" CreationDate="2017-12-14T09:43:43.640" Score="1" Body="&lt;p&gt;I would recommend a rolling average, it can be quite robust and is not upset by slow changes over time. You then can use your existing data to determine at which level of deviation you want an alarm. This optimization depends on weither you want rather to catch most of the deviations or minimize the number of false positive alarms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT1: In the end you will always need to tweak parameters. There is no real ground truth. An anomaly is always a subjective thing. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2017-12-15T10:04:35.030" LastActivityDate="2017-12-15T10:04:35.030" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25661" PostTypeId="2" ParentId="25639" CreationDate="2017-12-14T11:15:27.000" Score="7" Body="&lt;p&gt;When $T$ gets larger, i.e. $$T \rightarrow \infty $$ the probability distribution resembles the uniform distribution, so $q_i = \frac{1}{n}$. You can find the proof &lt;a href=&quot;https://math.stackexchange.com/questions/57434/boltzmann-soft-max-distribution&quot;&gt;here&lt;/a&gt;, as well as some other interesting properties. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the previously mentioned fact, I guess he is referring to &lt;a href=&quot;https://en.wikipedia.org/wiki/Smoothing&quot; rel=&quot;noreferrer&quot;&gt;smoothing &lt;/a&gt; in the sense that the noise if left out and what remains are the important patterns of the data, that is the distribution is made less 'spiky' as @Emre mentioned.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-14T11:15:27.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25663" PostTypeId="2" ParentId="25619" CreationDate="2017-12-14T11:37:05.353" Score="1" Body="&lt;p&gt;If I understood correctly your question, you want a function that takes a signal (a fixed window) and outputs a 32-bit representation in a way that the correlation between it and any other signal is preserved. Mathematically speaking given signals $s_1$ and $s_2$ in $S$ and a correlation function $corr(s_1, s_2)$ you want some function $f : S \rightarrow B$ (where $B$ is the space of 32-bit binary numbers) that you could use another correlation function for instance $corr_f(b_1, b_2) \approx corr(s_1, s_2)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is what you want you should at hashing techniques in particular &lt;a href=&quot;https://cs.nju.edu.cn/lwj/L2H.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;learning to hash&lt;/a&gt;. Essentially in hashing want you do is you represent your input by binary numbers in a way that the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hamming_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;hamming distance&lt;/a&gt; between the binary numbers preserves some target similarity (distance) function such as the correlation. In particular for cross-correlation (inner product) you have some methods based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Locality-sensitive_hashing#Random_projection&quot; rel=&quot;nofollow noreferrer&quot;&gt;random projections&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So once you have learned (designed) your hashing function $f$, what I would do is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;b1 = f(s1)&#xA;send b1&#xA;receive b1&#xA;b2 = f(s2)&#xA;return h(b2, b1) # this value is going to tell you if the signals are correlated&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-14T11:37:05.353" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25664" PostTypeId="1" CreationDate="2017-12-14T11:53:22.367" Score="1" ViewCount="49" Body="&lt;p&gt;I have a dataset with 19 features (columns). I normalized them using &lt;code&gt;sklearn.preprocessing.normalize&lt;/code&gt; then I used PCA to reduce them to 2 components for plotting purposes following the steps in this link:&lt;br&gt;&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I saw is that my data points form a perfect ellipse shape as in the following picture:&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/02uTy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/02uTy.png&quot; alt=&quot;K-means clustering with PCA reduction&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if this has any special meaning for my data. &lt;/p&gt;&#xA;" OwnerUserId="43317" LastActivityDate="2017-12-14T11:53:22.367" Title="PCA Reduction resulted in an elliptical form" Tags="|python|scikit-learn|clustering|pca|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25666" PostTypeId="1" CreationDate="2017-12-14T12:26:06.487" Score="4" ViewCount="12246" Body="&lt;p&gt;The major limitations of decision tree approaches to data analysis that I know of are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Provide less information on the relationship between the predictors and the   response.&lt;/li&gt;&#xA;&lt;li&gt;Biased toward predictors with more variance or levels.&lt;/li&gt;&#xA;&lt;li&gt;Can have issues with highly collinear predictors.&lt;/li&gt;&#xA;&lt;li&gt;Can have poor prediction accuracy for responses with low sample sizes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Are there any others? Are they robust to traditional statistical assumptions such as homogeneity, normality, independence? &lt;/p&gt;&#xA;" OwnerUserId="43318" LastEditorUserId="29575" LastEditDate="2017-12-17T20:08:46.987" LastActivityDate="2017-12-25T19:26:41.730" Title="What are limitations of decision tree approaches to data analysis?" Tags="|machine-learning|classification|random-forest|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25668" PostTypeId="2" ParentId="23828" CreationDate="2017-12-14T13:28:06.213" Score="2" Body="&lt;p&gt;Support Vector Machines attempt to find a hyerplane that divides two classes with the largest margin.  Therefore this is an optimization problem.  Support vectors are the points that lie along the supporting hyerplane.  One key factor that plays into the complexity of the runtime for a support vector is the slack parameter C.   The slack parameter allows for a soft-margin and better generalization.  The number of support vectors varies depending on how much slack we allow and how the data is distributed.  The less slack we give the SVM the fewer support vectors we get and converserly the more slack we give it the more support vectors we receive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There also exist a relationship between the complexity of the model. The more complex the models tend to require more support vectors.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, it doesn't really matter how many features you feed it, the SVM computational complexity is linear in the number of support vectors.   &lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2017-12-14T15:13:06.273" LastActivityDate="2017-12-14T15:13:06.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25671" PostTypeId="2" ParentId="25621" CreationDate="2017-12-14T13:54:47.393" Score="1" Body="&lt;p&gt;The way you defined your model it expects an input of shape (None, 5) and will return an output of shape (m, 1) where m is the number of examples, i.e. the number of lines in the array you load.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for your data set you get an output of shape (200, 1) stored in your variable predictions. I don't know why you want to call the predict method on your predictions, but when you do it has to result in an error.&lt;/p&gt;&#xA;" OwnerUserId="41234" LastActivityDate="2017-12-14T13:54:47.393" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25672" PostTypeId="1" CreationDate="2017-12-14T14:50:31.600" Score="0" ViewCount="714" Body="&lt;p&gt;I need to predict age or age range of a user according to his/her browsing history from a news website. The only data I am able to get is USER-ID, URLs-visited, TAGs from those URLs, &quot;which category news a person reads most&quot; etc. My only option seems like creating this dataset somehow. But I do not know how do I decide the age range of a person. The problem is if a person is reading a politics news he could belong to an age range of 50 and above but then what if this is a high trending news and everybody is reading it. So basically how do I decide which age range people read which kind of news most?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I am able to explain what my problem is. If somebody has worked on such a problem I would like to hear how you proceeded with it.&#xA;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="43325" LastActivityDate="2017-12-15T10:19:15.470" Title="Age range prediction according to browsing history" Tags="|machine-learning|dataset|" AnswerCount="1" CommentCount="4" ClosedDate="2017-12-16T16:42:20.263" ContentLicense="CC BY-SA 3.0" />
  <row Id="25673" PostTypeId="1" AcceptedAnswerId="25676" CreationDate="2017-12-14T15:07:36.643" Score="4" ViewCount="74967" Body="&lt;p&gt;I have trained and saved a model :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;# load the dataset&#xA;dataset = np.loadtxt(&quot;modiftrain.csv&quot;, delimiter=&quot;;&quot;)&#xA;&#xA;# split into input (X) and output (Y) variables&#xA;X_train = dataset[:,0:5]&#xA;Y_train = dataset[:,5]&#xA;&#xA;from sklearn.naive_bayes import GaussianNB&#xA;&#xA;# create Gaussian Naive Bayes model object and train it with the data&#xA;nb_model = GaussianNB()&#xA;&#xA;nb_model.fit(X_train, Y_train.ravel())&#xA;&#xA;# predict values using the training data&#xA;nb_predict_train = nb_model.predict(X_train)&#xA;&#xA;# import the performance metrics library&#xA;from sklearn import metrics&#xA;&#xA;# Accuracy&#xA;print(&quot;Accuracy: {0:.4f}&quot;.format(metrics.accuracy_score(Y_train, nb_predict_train)))&#xA;print()&#xA;&#xA;# import the lib to load / Save the model&#xA;from sklearn.externals import joblib  &#xA;&#xA;# Save the model&#xA;joblib.dump(nb_predict_train, &quot;trained-model.pkl&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, i'm loading the model and try to make predictions on a new dataset :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# import the lib to load / Save the model&#xA;from sklearn.externals import joblib  &#xA;&#xA;import numpy as np &#xA;&#xA;# Load the model&#xA;nb_predict_train = joblib.load(&quot;trained-model.pkl&quot;)&#xA;&#xA;# load the test dataset&#xA;df_predict = np.loadtxt(&quot;modiftest.csv&quot;, delimiter=&quot;;&quot;)&#xA;&#xA;X_train = df_predict&#xA;&#xA;nb_predict_train.predict(X_train)&#xA;&#xA;print(X_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here comes the error :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  File &quot;predict01.py&quot;, line 14, in &amp;lt;module&amp;gt;&#xA;    nb_predict_train.predict(X_train)&#xA;AttributeError: 'numpy.ndarray' object has no attribute 'predict'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43248" LastActivityDate="2017-12-14T15:38:18.797" Title="AttributeError: 'numpy.ndarray' object has no attribute 'predict'" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25674" PostTypeId="2" ParentId="25650" CreationDate="2017-12-14T15:09:51.787" Score="4" Body="&lt;p&gt;Humans don’t start their thinking from scratch every second. As you read this essay, you understand each word based on your understanding of previous words. You don’t throw everything away and start thinking from scratch again. Your thoughts have persistence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Traditional neural networks can’t do this, and it seems like a major shortcoming. For example, imagine you want to classify what kind of event is happening at every point in a movie. It’s unclear how a traditional neural network could use its reasoning about previous events in the film to inform later ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For further reading this go to&lt;/strong&gt; &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cohen's Blog&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32762" LastActivityDate="2017-12-14T15:09:51.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25675" PostTypeId="1" AcceptedAnswerId="25682" CreationDate="2017-12-14T15:28:12.777" Score="1" ViewCount="121" Body="&lt;p&gt;I am new to machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having a task of predicting whether a user will churn in March, given February feature data and the churn result. However, March data is leaked and now I'm assigned to predict for April. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My strategy is to train the model with February data and perform cross validation with March data. Then I will try to predict April data using model trained with February data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is my strategy good?&lt;/li&gt;&#xA;&lt;li&gt;Or should I append both February data and March data to predict April&#xA;data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="43330" LastEditorUserId="40853" LastEditDate="2017-12-15T07:25:19.497" LastActivityDate="2017-12-15T07:25:19.497" Title="About training and cross validation on a time series problem" Tags="|time-series|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25676" PostTypeId="2" ParentId="25673" CreationDate="2017-12-14T15:38:18.797" Score="6" Body="&lt;p&gt;You don't want to pickle the predictions but rather the fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change &lt;code&gt;joblib.dump(nb_predict_train, &quot;trained-model.pkl&quot;)&lt;/code&gt; to &lt;code&gt;joblib.dump(nb_model, &quot;trained-model.pkl&quot;)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-14T15:38:18.797" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25677" PostTypeId="1" AcceptedAnswerId="25684" CreationDate="2017-12-14T15:51:59.513" Score="0" ViewCount="40" Body="&lt;p&gt;I have a dataset with monthly revenue per customer. I want to build a model that can try to predict if the customer will exceed $10,000 3 months out (yes/no).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While this seems like a traditional ML problem I have an important questions&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Should I build my dataset with one row per customer id and let the label be the revenue 3 months out&lt;/li&gt;&#xA;&lt;li&gt;Should I instead have one row per month per customer and let the label by the revenue 3 months out&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="40740" LastActivityDate="2017-12-14T18:57:40.787" Title="Classify future performance of customer" Tags="|classification|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25678" PostTypeId="1" AcceptedAnswerId="25681" CreationDate="2017-12-14T16:23:35.070" Score="2" ViewCount="732" Body="&lt;p&gt;I'm trying to implement a value-iteration algorithm to solve a grid-world problem (I'm new to the field). The usual formula that I encounter about the value function V(s) is:&#xA;$$V(s) = R(s) + max_{a \in A} \sum_{s' \in S} T(s, a, s') V(s')$$&#xA;where $S$ is the set of states, $A$ the set of actions, $T$ the transition model &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$T(s, a, s') = P(s_{t+1} = s' | s_t = s, a_t = a)$$&#xA;and $R$ the reward function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I'm working on a model-based problem, $T$ and $R$ should be known; the problem is that I don't know how to define (or compute knowing the details of the problem) $R$. If I'm in a state $s$ and take an action $a$ that would make the agent hit a boundary of the grid world, the new state $s'$ will be $s$, and the reward, as defined by the problem, is -5. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if I had arrived in state $s$ through another state, the reward will be -1. So basically, $R(s)$ depends on the previous state and the previous action. How do I represent that in the $V(s)$ formula?&lt;/p&gt;&#xA;" OwnerUserId="43334" LastEditorUserId="29575" LastEditDate="2017-12-17T09:45:32.860" LastActivityDate="2017-12-17T09:45:32.860" Title="Reinforcement Learning - What's the formula for the value function" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25679" PostTypeId="2" ParentId="25672" CreationDate="2017-12-14T16:45:32.727" Score="1" Body="&lt;p&gt;This problem is quite broad and I would like to add some of my suggestions. Refer to good source of Age wise browsing data, age-wise social behaviors and their preferences publicly available. Here are some datasource that you may refer and you will have to do extensive research on lot of factors to predict age based on browsing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Datasource 1: &lt;a href=&quot;https://www.google.com/publicdata/directory&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.google.com/publicdata/directory&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Datasource 2: &lt;a href=&quot;https://www.ciser.cornell.edu/ASPs/datasource.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.ciser.cornell.edu/ASPs/datasource.asp&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer to this paper : &lt;a href=&quot;https://link.springer.com/chapter/10.1007/3-540-44934-5_3&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://link.springer.com/chapter/10.1007/3-540-44934-5_3&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would definitely say that this problem doesn't seem to have any perfect solution, only thing we may do is to try our level best by various sources and implementations. Learning algorithms on public datasets on behavior, survey,  social and mental age specific activities etc can help as well. I have also attached a link to a paper on similar problem. I hope this helps to some extent. &lt;/p&gt;&#xA;" OwnerUserId="42648" LastActivityDate="2017-12-14T16:45:32.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25680" PostTypeId="1" AcceptedAnswerId="25692" CreationDate="2017-12-14T17:26:16.113" Score="3" ViewCount="724" Body="&lt;p&gt;I have &quot;millions&quot; of items each having N binary features. When a feature is &quot;0&quot; it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to have a probability of the &quot;0&quot; features being &quot;1&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking this can be a Neural network with all features as input and same as output. But then I don't know how the training would work. I don't have ground truth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like some help expressing my problem and hopefully not reinvent the wheel. Is this is a classical problem in ML, and what approach can be applied?&lt;/p&gt;&#xA;" OwnerUserId="15402" LastEditorUserId="29575" LastEditDate="2017-12-17T20:07:03.230" LastActivityDate="2017-12-17T20:07:03.230" Title="Predicting Missing Features" Tags="|machine-learning|neural-network|missing-data|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25681" PostTypeId="2" ParentId="25678" CreationDate="2017-12-14T17:37:54.590" Score="1" Body="&lt;p&gt;The formula you have quoted is a bit unwieldy, precisely because the R function as defined needs to &quot;look ahead&quot; to all possible outcomes and their probabilities, but how to do that is not included explicitly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are actually a few variants of the Bellman equation that express more  or less detail. A good place to start for a truly generic version is Sutton &amp;amp; Barto (2nd Edition):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v^*(s) = \text{max}_{a \in \mathcal{A}}[\sum_{r,s'}p(r,s'|s,a)(r + v^*(s')]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\sum_{r,s'}$ is over all possible reward and next state pairs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above equation changes your transition function that only handles next state, to a similar function that handles successor state and reward:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(r,s'|s,a) = Pr\{ R_{t+1} = r, S_{t+1} = s' | S_{t} = s, A_{t} = a \}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually this does not increase the number of items to sum over, or add much complexity, because reward will most often associated with the transition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The benefit is that this approach removes the need for a reward function that works with &lt;em&gt;expected&lt;/em&gt; reward, and just works with specific rewards. Other variants are possible too, such as an expected reward function based on $(s,a)$ or $(s,a,s')$ - the difference is just a little bit of juggling with the expression so that it remains effectively the same given the subtle differences in definition for $r$.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-12-14T17:44:27.463" LastActivityDate="2017-12-14T17:44:27.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25682" PostTypeId="2" ParentId="25675" CreationDate="2017-12-14T18:16:35.380" Score="3" Body="&lt;p&gt;You might want to check out my answer to a related question &lt;a href=&quot;https://datascience.stackexchange.com/questions/25605/when-forecasting-time-series-how-does-one-incorporate-the-test-data-back-into-t/25607#25607&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to cross-validate time series data, I would suggest creating some sort of sliding window on which you train your model and predict the next month following that window. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you could train your model on February data and then predict the data of the first week of March. Then slide your window to include data from the second week of February through the first week of March and then predict the data of the second week in March. The length of the windows that you use to train and test on are parameters that you will need to play around with yourself to see what gets you the best results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you feel like you are getting good results from your cross-validation, I would try training a new model on all of the data you have and see how it performs on the new data coming in. Depending on the data and model you are using, this may provide better results having trained on more data. Of course, this depends on the problem you are trying to solve, the nature of your data and your choice of model.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-14T18:16:35.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25684" PostTypeId="2" ParentId="25677" CreationDate="2017-12-14T18:57:40.787" Score="1" Body="&lt;p&gt;I'm not sure it will matter very much whether you have one row per user or if you have one row for per month per user. The important part is that the data you have is accurate for that user for a particular month. You might construct your training data like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;--------|-----------------|--------------------|&#xA;cust. id| time on website | profit over 10,000?|&#xA;--------|-----------------|--------------------|&#xA; 3         30               0&#xA;--------|-----------------|--------------------|&#xA; 3         80               1&#xA;--------|-----------------|--------------------|&#xA; 5         100              1&#xA;--------|-----------------|--------------------|&#xA; 7         5                0&#xA;--------|-----------------|--------------------|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The important thing to notice is that even though customer 3 is in the dataset more than once, he/she has different values for their data features on which to predict, and different from how they were represented in the previous month. This is assuming that you are aggregating the data by month per customer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.yhat.com/posts/predicting-customer-churn-with-sklearn.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This blog&lt;/a&gt; predicts customer churn, but you might be able to use the general strategy for your problem.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-14T18:57:40.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25686" PostTypeId="2" ParentId="24729" CreationDate="2017-12-14T20:41:41.220" Score="2" Body="&lt;p&gt;You can read the data and first get a list of all the unique values of your categorical variables. Then you can fit a one hot encoder object (like the sklearn.preprocessing.CategoricalEncoder) on your list of unique values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method can also help in a train test framework or when you are reading your data in chunks. I have created a python module that does all this on its own. You can find it in this GitHub repository - &lt;a href=&quot;https://github.com/yashu-seth/dummyPy&quot; rel=&quot;nofollow noreferrer&quot;&gt;dummyPy&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A short a tutorial on this - &lt;a href=&quot;https://yashuseth.wordpress.com/2017/12/14/how-to-one-hot-encode-categorical-variables-of-a-large-dataset-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;How to One Hot Encode Categorical Variables in Python?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43342" LastEditorUserId="43342" LastEditDate="2017-12-14T22:53:46.923" LastActivityDate="2017-12-14T22:53:46.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25687" PostTypeId="2" ParentId="25619" CreationDate="2017-12-14T21:14:56.427" Score="1" Body="&lt;p&gt;For the record, the paper referred to from the comments above (&quot;DPSH: Feature learning based deep supervised hashing with pairwise labels&quot;) is available on arXiv: &lt;a href=&quot;https://arxiv.org/abs/1511.03855&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1511.03855&lt;/a&gt;&#xA;The paper applies the same idea to images, but the principle is more or less exactly identical to my original proposal.&lt;/p&gt;&#xA;" OwnerUserId="43243" LastActivityDate="2017-12-14T21:14:56.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25688" PostTypeId="2" ParentId="25429" CreationDate="2017-12-14T21:24:15.370" Score="1" Body="&lt;p&gt;Not sure what you are exactly looking for. There are many ways to do this. One simple way is like this, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word_list = ['natural language processing', 'sequential labeling', 'programmable', 'spell checking', 'techniques', 'forensics', 'important issue', 'categorial grammar', 'girls', 'applications']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since you said, a list of key words,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stopwords = ['programmable','techniques', 'important issue', 'girls', 'applications']&#xA;&#xA;resultwords  = [word for word in word_list if word.lower() not in stopwords]&#xA;result = ' '.join(resultwords)&#xA;&#xA;print (result)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will yield,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; natural language processing sequential labeling spell checking forensics categorial grammar&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Otherwise, if you just have a string of text, use &lt;code&gt;split()&lt;/code&gt; method to put each word into a list splitting by space. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;querywords = word_list.split()&#xA;&#xA;resultwords  = [word for word in querywords  if word.lower() not in stopwords]&#xA;result = ' '.join(resultwords)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can always warp them into a function and automate this. As I said there are many ways to do it, without knowing what have you tried, this is a simple way to do without using &lt;code&gt;nltk&lt;/code&gt; library.&lt;/p&gt;&#xA;" OwnerUserId="34028" LastActivityDate="2017-12-14T21:24:15.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25689" PostTypeId="1" AcceptedAnswerId="25691" CreationDate="2017-12-14T22:28:57.167" Score="1" ViewCount="364" Body="&lt;p&gt;I am trying to create a kNN search from scratch for a project. I think I have the concept of it and how it works, but I can't understand what exactly is a class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a matrix of $X*Y$ where $X$ is the number of elements and $Y$ their vector. So if the values are completely random, is every element/row a different class, or do I specify the classes some other way?&lt;/p&gt;&#xA;" OwnerUserId="43349" LastEditorUserId="29575" LastEditDate="2017-12-14T22:45:04.810" LastActivityDate="2017-12-15T09:40:00.157" Title="What exactly is a class in kNN search?" Tags="|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25691" PostTypeId="2" ParentId="25689" CreationDate="2017-12-15T00:02:44.900" Score="2" Body="&lt;p&gt;Given the context of your question I assume your are referring to the k-NN classifier. The idea of the classification method is you have a set of feature vectors $f_i$, in general, $f_i \in \mathbb{R}^d$  where $d$ is the dimension of the vectors. Additionally you have a class $c_i$ for each $f_i$. To classify an unseen feature vector $f_j$ you select the $k$ nearest neighbors (under some distance, usually the euclidean distance) and the most common class among the neighbors is your prediction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, summarizing, the input of the algorithm are the feature vectors $f_i$ and the classes $c_i$. For more on the algorithm see this wikipedia &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image (from wikipedia) below illustrates a typical example, in this example the classes are: &lt;em&gt;red triangle&lt;/em&gt;,&lt;em&gt;blue square&lt;/em&gt; and the point to predict is the &lt;em&gt;green circle&lt;/em&gt;. Each feature vector is a two dimensional point, i.e. $f_i, f_j \in \mathbb{R}^2$. In this situation the predicted class will be &lt;em&gt;red triangle&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;&lt;a href=&quot;https://i.stack.imgur.com/OPS0f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OPS0f.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For an explanation using both Python and R see &lt;a href=&quot;https://kevinzakka.github.io/2016/07/13/k-nearest-neighbor/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-15T00:57:00.217" LastActivityDate="2017-12-15T00:57:00.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25692" PostTypeId="2" ParentId="25680" CreationDate="2017-12-15T00:37:34.197" Score="6" Body="&lt;p&gt;A simple approach could be the following: suppose $i \in \{0,1\}^d$ is the vector you want to predict which of the $0$ entries could be $1$ and $j \in J$ the rest of the feature vectors. Take the $k$ nearest neighbors, under some suitable distance (&lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index&quot; rel=&quot;noreferrer&quot;&gt;Jaccard&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Hamming_distance&quot; rel=&quot;noreferrer&quot;&gt;Hamming&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Taxicab_geometry&quot; rel=&quot;noreferrer&quot;&gt;Manhattan distance&lt;/a&gt;). For each $0$ entry the probabilities could be the percentage of the $k$ nearest neighbors that have $1$ in the corresponding entry. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This problem has been extensively study in the &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;noreferrer&quot;&gt;collaborative filtering&lt;/a&gt; community. The best known example being the &lt;a href=&quot;https://www.netflixprize.com/&quot; rel=&quot;noreferrer&quot;&gt;Netflix Prize&lt;/a&gt;. This &lt;a href=&quot;https://medium.com/@victorkohler/item-item-collaborative-filtering-with-binary-or-unary-data-e8f0b465b2c3&quot; rel=&quot;noreferrer&quot;&gt;blog post&lt;/a&gt; provides a nice explanation of this approarch for binary data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another, more involved, approach is &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_completion&quot; rel=&quot;noreferrer&quot;&gt;matrix completion&lt;/a&gt;, in particular check this &lt;a href=&quot;https://papers.nips.cc/paper/5005-probabilistic-low-rank-matrix-completion-with-adaptive-spectral-regularization-algorithms.pdf&quot; rel=&quot;noreferrer&quot;&gt;reference&lt;/a&gt;. If you are into deep learning check &lt;a href=&quot;https://dl.acm.org/citation.cfm?id=3141950&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-15T00:37:34.197" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25693" PostTypeId="1" CreationDate="2017-12-15T01:08:13.223" Score="6" ViewCount="1538" Body="&lt;p&gt;In &lt;a href=&quot;https://keras.io/getting-started/faq/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page&lt;/a&gt; it is mentioned that when &lt;code&gt;trainable=false&lt;/code&gt;, the weight won't be updated and is used for optimization, too. But I still do not understand how it can be useful? (e.g., I want to find out the best number of neurons or the best &lt;code&gt;drop out&lt;/code&gt; rate, can it help?)&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="29575" LastEditDate="2017-12-28T15:49:29.100" LastActivityDate="2017-12-28T15:49:29.100" Title="What is the purpose of untrainable weights in Keras" Tags="|machine-learning|deep-learning|keras|transfer-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25694" PostTypeId="2" ParentId="15630" CreationDate="2017-12-15T01:39:27.377" Score="3" Body="&lt;p&gt;Per your last question:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Then I am wondering this way, I won't be able to perform n-fold cross&#xA;validation, right? Because my data is so small (especially for the&#xA;minority class)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This is not true. You can try upsampling if your data is really small (but how small is it?)&lt;/p&gt;&#xA;" OwnerUserId="43353" LastEditorUserId="29169" LastEditDate="2021-07-09T21:25:35.343" LastActivityDate="2021-07-09T21:25:35.343" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25696" PostTypeId="1" AcceptedAnswerId="25697" CreationDate="2017-10-30T13:03:14.823" Score="0" ViewCount="49" Body="&lt;p&gt;I have a question regarding annotating text data for classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).&lt;/li&gt;&#xA;&lt;li&gt;Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thoughts,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.&lt;/li&gt;&#xA;&lt;li&gt;will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="21257" OwnerDisplayName="Isbister" LastActivityDate="2017-12-15T04:11:45.477" Title="Text annotating process, quality vs quantity?" Tags="|machine-learning|nlp|classification|data-mining|text-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-16T16:41:19.660" ContentLicense="CC BY-SA 3.0" />
  <row Id="25697" PostTypeId="2" ParentId="25696" CreationDate="2017-10-31T10:50:18.160" Score="2" Body="&lt;p&gt;Both metods are against the nature of machine/statistical learning. Lack of data will not let you generalize well in any case. It will have different effect though. Generally speaking:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You will be able to classify more test samples, but sometimes will do&#xA;it wrong due to errors in data. &lt;/li&gt;&#xA;&lt;li&gt;You will have precise classification of some test samples, but will have no idea how to classify the rest of them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, it's up to your's classification score/quality estimation function which case is better for you.&lt;/p&gt;&#xA;" OwnerUserId="43374" OwnerDisplayName="CrazyElf" LastActivityDate="2017-10-31T10:50:18.160" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25698" PostTypeId="2" ParentId="25693" CreationDate="2017-12-15T04:44:42.187" Score="7" Body="&lt;p&gt;One common application is to freeze an embedding layer. Freezing this layer will prevent the embedding from updating its weight which can be a desirable thing, especially for a text embedding layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There also exist designs where updating a weight during a certain batch is not wanted.  For example some GAN implementations only want to train a model during the combined model phase and therefore freezes the Generator and Discriminator layers for batches.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also see this used for stacked autoencoder where someone may train one layer at a time.  Here is a &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders&quot; rel=&quot;nofollow noreferrer&quot;&gt;quick link&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="29575" LastEditDate="2017-12-17T11:34:35.067" LastActivityDate="2017-12-17T11:34:35.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25699" PostTypeId="2" ParentId="25693" CreationDate="2017-12-15T05:11:08.200" Score="6" Body="&lt;p&gt;We use freezing to employ &lt;a href=&quot;https://en.wikipedia.org/wiki/Transfer_learning&quot; rel=&quot;noreferrer&quot;&gt;transfer learning&lt;/a&gt;. Deep learning has a great hunger for data. In some tasks you may not have so much data, but there may already be a pre-trained network that can be helpful. In such cases you use the model and its weights and by replacing the soft-max layer, in situations where you have small amount of data, you try to customize the network for you specific task. If you have more data, the more number of layers can be trainable. Take a look at &lt;a href=&quot;http://ruder.io/transfer-learning/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-15T05:11:08.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25700" PostTypeId="1" CreationDate="2017-12-15T06:09:44.720" Score="6" ViewCount="1022" Body="&lt;p&gt;I cannot understand the way how algorithm &lt;strong&gt;Differential Semi-gradient Sarsa&lt;/strong&gt; updates its estimated average reward $\bar{R}$. The algorithm I am looking at is from Sutton's text book &lt;a href=&quot;http://incompleteideas.net/book/bookdraft2017nov5.pdf&quot; rel=&quot;noreferrer&quot;&gt;Reinforcement Learning:An Introduction&lt;/a&gt; , section 10.3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does not update $\bar{R}$ using the reward $R$ got at current step like $\bar{R} = (1-\beta)\bar{R}+\beta*R$ ? Since based on definition, $\bar{R}$ is the estimated average reward. I cannot understand the reason why the updating is like this: $\bar{R} = \bar{R}+\beta*\delta$, where $\delta$ is just the TD error. Why using TD error to update the average reward?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following figure shows the algorithm.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/W38yU.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W38yU.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43358" LastActivityDate="2018-07-02T18:43:51.900" Title="How differential semi-gradient Sarsa updates estimated average reward?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25702" PostTypeId="2" ParentId="25689" CreationDate="2017-12-15T09:40:00.157" Score="1" Body="&lt;p&gt;Additional to feynman410's answer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The class is your attribute that the classifier wants to predict.&#xA;For example say you want to predict the color of an object and it turns out the object can be red, blue, purple, green and yellow.&#xA;Then each group of objects that have the same color form a class.&#xA;Now in order to predict the class (in this case the color) of the object your algorithm needs some objects where it already knows their attributes and the class (color).&#xA;Hence also the word &quot;classifier&quot;, which means predicting the class of something.&lt;/p&gt;&#xA;" OwnerUserId="43367" LastActivityDate="2017-12-15T09:40:00.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25704" PostTypeId="1" CreationDate="2017-12-15T10:37:27.210" Score="1" ViewCount="3857" Body="&lt;p&gt;I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)&#xA;x = Activation('relu')(x)&#xA;x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)&#xA;x = Activation('relu')(x)&#xA;x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;x = Dropout(0.25)(x)&#xA;x = Flatten()(x)&#xA;conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)&#xA;&#xA;lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]&#xA;&#xA;sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)&#xA;model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At human check I see that as a result, all predicted labels are a constant sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-02-13T18:10:29.370" Title="High, constant training loss with CNN" Tags="|machine-learning|neural-network|keras|cnn|ocr|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25705" PostTypeId="1" CreationDate="2017-12-15T11:52:52.660" Score="4" ViewCount="7232" Body="&lt;p&gt;I recently read &lt;a href=&quot;https://arxiv.org/pdf/1609.05158.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network&lt;/a&gt; by Wenzhe Shi et al. I cannot understand the difference between deconvolution (mentioned in Section 2.1) and the Efficient sub-pixel convolution layer (ESCL for short) (Section 2.2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Section 2.2 defines the ESCL as performing convolution with a fractional stride. Isn't this what deconvolution is? Why is a distinction being made?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="41977" LastEditorUserId="29169" LastEditDate="2021-02-13T19:20:12.880" LastActivityDate="2021-02-13T19:20:12.880" Title="Deconvolution vs Sub-pixel Convolution" Tags="|neural-network|convolutional-neural-network|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25706" PostTypeId="2" ParentId="22314" CreationDate="2017-12-15T11:56:34.250" Score="1" Body="&lt;p&gt;There are various ways to handle string inputs to neural networks, but since you are trying to predict spelling difficulty, I suggests representing your words as a sequence of characters. This will preserve information about the particular spelling of the words including the order of the letters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To represent a sequence of characters you can one-hot encode each character, so each word will be represented as a sequence of one-hot length 26 vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To handle this kind of input I suggest either a 1D Convolutional Neural Network or some flavor of a Recurrent Neural Network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you choose a 1D CNN you will have to feed it fixed sized inputs. To do this choose a max word length, &lt;code&gt;k&lt;/code&gt;, and either cut off or pad with zeros each input word to fit into a &lt;code&gt;k*26&lt;/code&gt; input matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of a CNN architecture you might use in Keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu', input_shape=(k, 26))&#xA;model.add(MaxPooling1D())&#xA;model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu')&#xA;model.add(MaxPooling1D())&#xA;model.add(Dropout(0.25))&#xA;model.add(Flatten())&#xA;model.add(Dense(64, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(3, activation='softmax'))&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-15T17:59:41.887" LastActivityDate="2017-12-15T17:59:41.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25708" PostTypeId="1" CreationDate="2017-12-15T14:27:11.183" Score="2" ViewCount="83" Body="&lt;p&gt;Take the following case of a hyperparameter and prediction error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3ILj3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3ILj3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- too high and you're underfit and too low and you're overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I keep getting nonconvex plots like the one above when doing cross-validation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess this just points to a lot of noise during training -- I've got a lot of variables for a modest sample size, and I need to regularize heavily to get a good model.  But still I'm a little bit unsure whether this sort of thing might point to a bug in some aspect of my implementation.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone come across this sort of thing before?  And did you just shrug off the nonconvexity and go with the model that minimized the prediction error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, that begs the question: why not just compute a prediction error at each update during training, saving any set of weights that minimizes prediction error -- even if the model is nowhere near converged.  Basically letting the noise work in your favor.  This seems appealing, because sometimes I get &lt;em&gt;really low&lt;/em&gt; prediction errors early on, only to have them evaporate as the loss function declines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems horribly unprincipled, but I ask myself &quot;why should I care if it is&quot;?  And &quot;is it unprincipled anyway?&quot;&lt;/p&gt;&#xA;" OwnerUserId="30335" LastActivityDate="2017-12-15T15:58:16.550" Title="How to think about prediction error that is not convex in hyperparameter, or over the course of training" Tags="|neural-network|deep-learning|hyperparameter|regularization|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25709" PostTypeId="2" ParentId="25708" CreationDate="2017-12-15T15:21:26.213" Score="3" Body="&lt;p&gt;The underlying true performance is likely convex or at least likely only has one minimum, but you don't know the true underlying performance, you only get a stochastic sample with a tiny sample size because you apply cross validation. The performance is a random variable where you are interested in the expected value, but you only get &lt;strong&gt;k&lt;/strong&gt; samples where &lt;strong&gt;k&lt;/strong&gt; is the number of folds. I think if you would increas &lt;strong&gt;k&lt;/strong&gt; your curve would look more smooth. That said, even if your hypothesis counts in general, for this specific set there might be some weird data points that make random settings perform worse than expected still.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with your suggested approach is that you are treating this noise as signal. If you have a normally distributed random variable, you sample 1000 times from it and take the max value, this will be much higher than the true expected value. An alternative explanation could be that it is not noise, but that you are actually overfitting. Then your approach makes more sense, and if the samples surrounding it during training are also relatively low then this is actually a good strategy, it is called early stopping, a different form of regularization and commonly used in Neural Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I think you are right that I didn't explain that analogy very well, we are trying to optimize a function g(x) where x is our hyperparameters and g(x) is a random variable function, let's say it had a true underlying value f(x) and additive noise $\epsilon$. If $\epsilon$ is Gaussian distributed with a high variance compared to the underlying value, if we take the max without looking at anything else we could take one that had lucky noise. I think what we are seeing in your graph is relatively high variance due to low sample size of your data (which means small validation sets in your folds) whih will lead to non-convex behaviour. That said, your explanation of early on seeing lower validation losses does imply overfitting which early stopping can help with.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2017-12-15T15:58:16.550" LastActivityDate="2017-12-15T15:58:16.550" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25710" PostTypeId="2" ParentId="25704" CreationDate="2017-12-15T17:06:01.287" Score="1" Body="&lt;p&gt;I suspect you are facing a dying ReLU problem.  Check the gradients for each layer and see if they are starting to become 0.  Due to the way backpropagation works and a simple application of the chain rule, once a gradient is 0, it ceases to contribute to the model.  Therefore, if you're model is stuck then it's likely that a significant number of your neurons are now dead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try using a LeakyRelu or Maxout and see if that helps.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-15T17:06:01.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25712" PostTypeId="1" AcceptedAnswerId="25713" CreationDate="2017-12-15T18:08:11.380" Score="13" ViewCount="21903" Body="&lt;p&gt;Suppose I have a set of time-domain signals with absolutely &lt;strong&gt;no labels&lt;/strong&gt;. I want to cluster them in 2 or 3 classes. Autoencoders are unsupervised networks that learn to compress the inputs. So given an input $x^{(i)}$, weights $W_1$ and $W_2$, biases $b_1$ and $b_2$, and output $\hat{x}^{(i)}$, we can find the following relationships:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$z^{(i)} =W_1x^{(i)}+b_1$$&#xA;$$\hat{x}^{(i)} =W_2z^{(i)}+b_2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So $z^{(i)}$ would be a compressed form of $x^{(i)}$, and $\hat{x}^{(i)}$ the reconstruction of the latter. So far so good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is how this could be used for clustering (if there is any way to do it at all). For example, in the first figure of &lt;a href=&quot;https://ac.els-cdn.com/S1877050917318707/1-s2.0-S1877050917318707-main.pdf?_tid=14eb11a8-e1c1-11e7-8826-00000aab0f6c&amp;amp;acdnat=1513360685_cb6b43cd7a7a84c40288a6493dbddbc0&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;, there is a block diagram I'm not sure I understand. It uses the $z^{(i)}$ as the inputs to the feed-forward network, but there is no mention to how that network is trained. I don't know if there is something I'm ignoring or if the paper is incomplete. Also, &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/&quot; rel=&quot;noreferrer&quot;&gt;this tutorial&lt;/a&gt; at the end shows the weights learned by the autoencoder, and they seem to be kernels a CNN would learn to classify images. So... I guess the autoencoder's weights can be used somehow in a feed-forward network for classification, but I'm not sure how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My doubts are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If $x^{(i)}$ is a time-domain signal of length $N$ (i.e. $x^{(i)}\in\mathbb{R}^{1\times N}$), can $z^{(i)}$ only be a vector as well? In other words, would it make sense for $z^{(i)}$ to be a &lt;em&gt;matrix&lt;/em&gt; with one of its dimensions greater than $1$? I believe it would not, but I just want to check.&lt;/li&gt;&#xA;&lt;li&gt;Which of these quantities would be the input to a classifier? For example, if I want to use a classic MLP that has as many output units as classes I want to classify the signals in, what should I put at the input of this fully-connected network ($z^{(i)}$,$\hat{x}^{(i)}$, any other thing)?&lt;/li&gt;&#xA;&lt;li&gt;How can I use the learned weights and biases in this MLP? Remember that we assumed that &lt;strong&gt;absolutely no labels&lt;/strong&gt; are available, so it is impossible to train the network. I think the learned $W_i$ and $b_i$ should be useful somehow in the fully-connected network, but I don't see how to use them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Observation: note that I used an MLP as an example because it is the most basic architecture, but the question applies to any other neural network that could be used to classify time-domain signals.&lt;/p&gt;&#xA;" OwnerUserId="35479" LastActivityDate="2020-07-06T20:30:58.013" Title="How can autoencoders be used for clustering?" Tags="|neural-network|clustering|time-series|unsupervised-learning|autoencoder|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25713" PostTypeId="2" ParentId="25712" CreationDate="2017-12-15T20:18:19.340" Score="14" Body="&lt;p&gt;Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar.  Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space.  It doesn't do clustering per se - but it is a useful &lt;em&gt;preprocessing&lt;/em&gt; step for a secondary clustering step.  You would map each input vector $x_i$ to a vector $z_i$ (not a matrix...) with a smaller dimensionality, say 2 or 3.  You'd then use some other clustering algorithm on all the $z_i$ values.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe someone else can chime in on using auto-encoders for time series, because I have never done that. I would suspect that you would want one of the layers to be a 1D convolutional layer, but I am not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people use autoencoders as a data pre-processing step for classification too.  In this case, you would first use an autoencoder to calculate the $x$-to-$z$ mapping, and then throw away the $z$-to-$\hat{x}$ part and use the $x$-to-$z$ mapping as the first layer in the MLP.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-12-15T20:18:19.340" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25714" PostTypeId="1" AcceptedAnswerId="25715" CreationDate="2017-12-15T20:35:00.220" Score="10" ViewCount="10042" Body="&lt;p&gt;The value of a state $s$ under a certain policy $\pi$, $V^\pi(s)$, is defined as the &quot;expected return&quot; starting from state $s$. More precisely, it is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;V^\pi(s) = \mathbb{E}\left(R_t \mid s_t = s \right)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $R_t$ can be defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\sum_{k=0}^\infty \gamma^k r_{t+k+1}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is a sum of &quot;discounted&quot; rewards after time $t$, i.e. starting from time $t+1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$V^\pi(s)$ can also be interpreted, even more precisely, as the &lt;strong&gt;&lt;em&gt;expected cumulative future discounted reward&lt;/em&gt;&lt;/strong&gt;. This denotation contains all the words which refer to specific parts of the formula above, where&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&quot;Expected&quot; refers to the &quot;expected value&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;Cumulative&quot; refers to the summation&lt;/li&gt;&#xA;&lt;li&gt;&quot;Future&quot; refers to the fact that it's an expected value of a future quantity with respect to the present quantity, i.e. $s_t = s$.&lt;/li&gt;&#xA;&lt;li&gt;&quot;Discounted&quot; refers to the &quot;gamma&quot; factor, which is a way to adjust the importance of how much we value rewards at future time steps, i.e. starting from $t + 1$.&lt;/li&gt;&#xA;&lt;li&gt;&quot;Reward&quot; refers to the main quantity of interested, i.e. the reward received from the environment.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Meanwhile, I've heard the term &quot;expected reward&quot;, but I am not sure if it refers to the same concept or not, that is if &quot;expected reward&quot; or &quot;expected return&quot; are the same thing or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know there's also the concept of &quot;expected value of the next reward&quot;, often denoted as $\mathcal{R}^a_{ss'}$, and defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which, again, is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the &quot;expected reward&quot; actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?&lt;/p&gt;&#xA;" OwnerDisplayName="user10640" LastActivityDate="2017-12-15T21:27:55.097" Title="What is the difference between &quot;expected return&quot; and &quot;expected reward&quot; in the context of RL?" Tags="|reinforcement-learning|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25715" PostTypeId="2" ParentId="25714" CreationDate="2017-12-15T21:27:55.097" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is the &quot;expected reward&quot; actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In short, yes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although there is some context associated - $\mathcal{R}^a_{ss'}$ is in the context of specific action and state transition. You will also find $\mathcal{R}^a_{s}$ used for expected reward given only current state and action (which works fine, but moves around some terms in the Bellman equations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Return&quot; may also be called &quot;Utility&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RL suffers a bit from naming differences, however the meaning of &lt;em&gt;reward&lt;/em&gt; is not one of them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notation differences also abound, and in Sutton &amp;amp; Barto &lt;em&gt;Reinforcement Learning: An Introduction (2nd edition)&lt;/em&gt;, you will find:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$R_t$ is a placeholder for reward received at time $t$, a random variable.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$G_t$ is a placeholder for return received after time $t$, and you can express the value equation as $v_{\pi}(s) = \mathbb{E}[G_t|S_t=s] = \mathbb{E}[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}|S_t=s]$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$r$ is a specific reward value&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You won't see &quot;expected reward&quot; used directly in an equation from the book, as the notation in the revised book relies on summing over distribution of reward values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In some RL contexts, such as control in continuous problems with function approximation, it is more convenient to work with maximising average reward, than maximising expected return. But this is not quite the same as &quot;expected reward&quot;, due to differences in context (average reward includes averaging over the expected state distribution when following the policy)&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-12-15T21:27:55.097" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="25717" PostTypeId="1" CreationDate="2017-12-16T00:18:41.067" Score="6" ViewCount="1167" Body="&lt;p&gt;There's the concept of &quot;expected value of the next reward&quot;, often denoted as $\mathcal{R}^a_{ss'}$, and defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is &lt;em&gt;the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the italicized definition does not seem to be &quot;consistent&quot; with the Bellman equation for the value function, which is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;V^\pi(s) &amp;amp;= \sum_{a \in \mathcal{A}(s)} \pi(s, a) \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right) &#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do I say this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{R}_{ss'}^a$ depends on $s$, $a$ and $s'$, but in the Bellman equation above, specifically in the inner sum $\sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)$, we are essentially iterating over all possible next states, $s'$, of $s$. But, if that's the case, then what's the purpose of making $\mathcal{R}_{ss'}^a$ depend on the action $a$? That is, if we already have the next state $s'$, then the action is useless, given that we are already taking into consideration the possible next states, indeed. In other words, $a$ and $s'$ may not be compatible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose we are in a certain state $s$, and we take action $a$ from there, and we end up in a state, say, $x$, which must be different to all next states $s'$ (in the inner summation), apart at most from one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope you can see my point and my doubts, and I hope you can clarify them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, given this doubt, I am unable to implement a reward function. Also, I have seen an implementation of the reward function, that is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def reward(state, next_state, action):&#xA;    if GRID[state] != GOAL and GRID[next_state] == GOAL:&#xA;        return 10.0&#xA;    else:&#xA;        return 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which actually does not use the &lt;code&gt;action&lt;/code&gt;, but I don't understand why. Why should we use &lt;code&gt;next_state&lt;/code&gt; and not the next state of &lt;code&gt;state&lt;/code&gt; according to &lt;code&gt;action&lt;/code&gt; (or vice-versa)? How could (and why would) we use both &lt;code&gt;next_state&lt;/code&gt; and &lt;code&gt;action&lt;/code&gt;, if they may be &quot;incompatible&quot; in the sense I described above?&lt;/p&gt;&#xA;" OwnerDisplayName="user10640" LastEditorDisplayName="user10640" LastEditDate="2017-12-16T00:30:40.147" LastActivityDate="2021-10-12T16:19:45.097" Title="How is that possible that a reward function depends both on the next state and an action from current state?" Tags="|python|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25719" PostTypeId="2" ParentId="25477" CreationDate="2017-12-16T08:29:57.270" Score="0" Body="&lt;p&gt;Equation $12$ of the paper states the $KKT$ condition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\alpha_i = 0 &amp;amp;\iff y_iu_i\geq1 \\&#xA;0&amp;lt;\alpha_i &amp;lt; C &amp;amp;\iff y_iu_i = 1 \\&#xA;\alpha_i = C &amp;amp;\iff y_iu_i \le 1&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence if $\alpha_i &amp;lt; C$, we are suppose to have $y_iu_i \geq 1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To violate $y_iu_i \geq 1$ means $y_iu_i &amp;lt; 1$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_iu_i &amp;lt; 1 = y_i^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;is equivalent to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y_i(u_i-y_i ) &amp;lt; 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the notation of the paper, it is $r_2=y_i E_i &amp;lt; 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Numerically, we consider $y_iE_i$ to be negative if it is smaller than $-\epsilon$ for some small positive number $\epsilon$.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2017-12-16T08:40:49.183" LastActivityDate="2017-12-16T08:40:49.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25720" PostTypeId="2" ParentId="16580" CreationDate="2017-12-16T08:40:04.887" Score="1" Body="&lt;p&gt;In my view, Empirical Likelihood method is a very data-driven method but it has nothing to do with machine learning. Here is a link talking about the Empirical Likelihood method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Empirical_likelihood&quot; rel=&quot;nofollow noreferrer&quot;&gt;Empirical Likelihood Method&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43406" LastActivityDate="2017-12-16T08:40:04.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25721" PostTypeId="2" ParentId="25717" CreationDate="2017-12-16T09:04:13.680" Score="6" Body="&lt;p&gt;The equations you have given &lt;em&gt;are&lt;/em&gt; consistent with each other. The outer sum over $a$ provides the value of $a$. The &quot;link&quot; between $s, a$ and $s'$ that gives the correct associations is the transition function $\mathcal{P}^a_{ss'}$ which will be zero for any state/action/next-state triplet that does not make sense for the MDP. By &quot;not make sense&quot;, I mean not allowed by the dynamics of the problem - such as an action that would successfully move too many spaces or through a wall for a maze problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is normal for the reward to &lt;em&gt;not&lt;/em&gt; actually depend on $a$, but only on the successor state. This might happen in e.g. a dynamic or adversarial environment where the desired goal is to achieve a specific end state. However, it is also possible for the reward to depend on action taken in a given state, such as activating some object in a particular location. It depends on how you have set up the MDP. Another example of where $a$ might be important is if there is a cost associated with each action that varies, for example an energy requirement, so that the total reward might be separate contribution from $s'$ (for reaching a goal) and $a$ (for spending energy to reach any next state).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using an expected reward $\mathcal{R}_{ss'}^a$ that depends on all three terms $s, a, s'$ gives you a generic version of the Bellman equation that can be applied to the widest range of MDP designs. It doesn't matter if, in a specific case, the only thing you care about is $s'$ which is what the example reward function shows. Once you decide that the expected reward is dependent on $s'$, then the Bellman equation has to have that expected reward term inside the inner sum (the only place where $s'$ is defined), and there is no additional structure in the equation needed to have this term dependent on $s$ and $a$ as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could legitimately use a variant $\mathcal{R}_{s'}$ in a Bellman equation that is otherwise identical to the equation you give in the question, to describe the value function for a MDP where reward only depends on the state transitioned to. However, you will not find that in the literature because it is less generic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One common variant that you will see is $\mathcal{R}_{s}^{a}$ i.e. the expected reward for taking action $a$ in state $s$. You might use this for cost-based systems, where the goal is to complete a task efficiently and taking actions expends some resource such as time. Using your notation, the Bellman equation for that design looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;V^\pi(s) &amp;amp;= \sum_{a \in \mathcal{A}(s)} \pi(s, a)( \mathcal{R}_{s}^a + \gamma \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} V^\pi(s')) &#xA;\end{align}&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-12-17T17:40:17.960" LastActivityDate="2017-12-17T17:40:17.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25722" PostTypeId="1" AcceptedAnswerId="25728" CreationDate="2017-12-16T11:30:18.037" Score="5" ViewCount="16822" Body="&lt;p&gt;Can dropout be applied to convolution layers or just dense layers. If so, should it be used after pooling or before pooling and after applying activation?&lt;br /&gt;&#xA;Also I want to know whether batch normalization can be used in convolution layers or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/42nnpe/why_do_i_never_see_dropout_applied_in/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; but I couldn't find valuable answers because of lacking reference.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-17T16:58:57.473" LastActivityDate="2020-12-30T16:04:19.403" Title="Can dropout and batch normalization be applied to convolution layers" Tags="|machine-learning|deep-learning|cnn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25723" PostTypeId="2" ParentId="13762" CreationDate="2017-12-16T13:52:25.407" Score="2" Body="&lt;p&gt;I'd say there are pros and cons of using FeatureHasher for this purpose.&#xA;If you really striving to use it, then just instantiate it like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;In [1]:&#xA;from sklearn.feature_extraction import FeatureHasher&#xA;h = FeatureHasher(n_features=5, input_type='string')&#xA;f = h.transform(mail_id)&#xA;f.toarray()&#xA;&#xA;Out[1]:&#xA;array([[ 1.,  0.,  0.,  0.,  0.],&#xA;      [ 0., -1.,  0.,  0.,  0.],&#xA;      [ 1.,  0.,  0.,  0.,  0.],&#xA;      [ 0.,  0., -1.,  0.,  0.],&#xA;      [ 0., -1.,  0.,  0.,  0.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So, after you have instantiated it, just .transform each your upcoming mail_id and use results in upstream applications ( like online learning, for instance ).&#xA;Obviously n_features is some knob to tune. But this has its flip side: the cardinality of mail ids is apriori high, so unless you have very limited amount of users you will need enormous n_features to minimize collisions.&lt;/p&gt;&#xA;&lt;p&gt;The better would be to take logs, where your ids coappear, and learn &lt;a href=&quot;https://arxiv.org/pdf/1603.04259v2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;item2vec style model&lt;/a&gt;. This will deliver much denser (and meaningful) representation of mail_ids than FeatureHasher would do.&lt;/p&gt;&#xA;&lt;p&gt;Also, take a look at &lt;a href=&quot;http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="13787" LastEditorUserId="85045" LastEditDate="2020-12-25T10:07:44.783" LastActivityDate="2020-12-25T10:07:44.783" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25725" PostTypeId="1" AcceptedAnswerId="25732" CreationDate="2017-12-16T15:47:39.437" Score="1" ViewCount="1517" Body="&lt;p&gt;I'm working on keyword/phrase extraction from a single document. I started by doing term frequency analysis, but this returns words like &quot;new&quot; which aren't very helpful. So I want to penalize the common words and phrases, for which we normally use idf (inverse document frequency). But since it's for a single document, I'm not sure how to do idf analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use tf-idf method with pre-calculated idf values for (all?) words?&#xA;And are such values available somewhere?&lt;/p&gt;&#xA;" OwnerUserId="25345" LastEditorUserId="25345" LastEditDate="2017-12-17T04:20:30.850" LastActivityDate="2021-02-09T03:03:04.440" Title="Idf values of English words" Tags="|nlp|tfidf|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25727" PostTypeId="1" CreationDate="2017-12-16T17:54:39.503" Score="2" ViewCount="231" Body="&lt;p&gt;This phrase is from the &quot;Rethinking the Inception Architecture for Computer Vision&quot; paper. it says :   &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;Higher dimensional representations are easier to process locally within a network. Increasing the activations per tile in a&#xA;  convolutional network allows for more disentangled features. The&#xA;  resulting networks will train faster.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does the phrase &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;Higher dimensional representations are easier to process locally within a network&quot;&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;mean?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it simply mean, a higher representation is achieved more easily by having more feature-maps in a layer? again indicating, having lots of feature-maps per layer, (more width) is more beneficial?&lt;br&gt;&#xA;If so, does the next part indicate the same thing where it says :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Increasing the activations per tile in a convolutional network allows&#xA;  for more disentangled features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;what does disentagled features mean in that regard? &lt;/p&gt;&#xA;" OwnerUserId="16338" LastActivityDate="2017-12-18T09:28:36.750" Title="Some questions about GoogleNet paper" Tags="|deep-learning|inception|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25728" PostTypeId="2" ParentId="25722" CreationDate="2017-12-16T17:58:59.013" Score="5" Body="&lt;p&gt;In short, yes.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Batch Normalization&lt;/strong&gt; Batch Normalization layer can be used in between two convolution layers, or between two dense layers, or even between a convolution and a dense layer. The important question is &lt;code&gt;Does it help?&lt;/code&gt; Well, it is recommended to use BN layer as it shows improvement generally but the amount of improvement you will get is more problem dependent. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Dropout:&lt;/strong&gt; Convolution layers, in general, are not prone to overfitting but it doesn't mean that you shouldn't use dropout. You can, but again this is problem dependent. For example, I was trying to build a network where I used &lt;code&gt;Dropout&lt;/code&gt; in between conv blocks and my model got better with it. It is better if you apply dropout after pooling layer. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-12-16T17:58:59.013" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25729" PostTypeId="2" ParentId="25725" CreationDate="2017-12-16T19:32:56.593" Score="2" Body="&lt;p&gt;I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are looking to get rid of words such as &lt;em&gt;the&lt;/em&gt;, &lt;em&gt;as&lt;/em&gt;, and &lt;em&gt;it&lt;/em&gt; which don't carry much meaning, &lt;code&gt;nltk&lt;/code&gt; in Python has some useful tools to remove these &quot;stop words&quot; from your document and might help you. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a helpful &lt;a href=&quot;https://pythonspot.com/nltk-stop-words/&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-16T19:32:56.593" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25731" PostTypeId="1" AcceptedAnswerId="25748" CreationDate="2017-12-16T22:54:55.223" Score="0" ViewCount="67" Body="&lt;p&gt;In Practice with CNN what would be easier: Building a CNN from scratch or using a an existing architecture with some updates?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T19:20:03.090" LastActivityDate="2021-02-13T19:20:03.090" Title="What it Would be easier Building a Deep Net From Scratch or Using an existing Architecture?" Tags="|machine-learning|deep-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ClosedDate="2017-12-24T12:53:19.067" ContentLicense="CC BY-SA 4.0" />
  <row Id="25732" PostTypeId="2" ParentId="25725" CreationDate="2017-12-16T23:22:30.040" Score="2" Body="&lt;p&gt;The list of 20,000 most common words in English is avaiable &lt;a href=&quot;https://github.com/first20hours/google-10000-english/blob/master/20k.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;By using &lt;a href=&quot;http://mathworld.wolfram.com/ZipfsLaw.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zipf's law&lt;/a&gt;, we can obtain the probability of these words as below.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Zipf's Law&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the harmonic series diverges. Pierce's (1980, p. 87) statement that sumP(r)&amp;gt;1 for  r=8727 is incorrect. Goetz states the law as follows: The frequency of a word is inversely proportional to its statistical rank r such that&lt;/p&gt;&#xA;&lt;p&gt;P(r) = 1/(rln(1.78R)),&lt;/p&gt;&#xA;&lt;p&gt;where R is the number of different words.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;These probability values can be used as a substitute for idf.&lt;/p&gt;&#xA;" OwnerUserId="25345" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-16T23:22:30.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25737" PostTypeId="2" ParentId="23895" CreationDate="2017-12-17T05:04:47.427" Score="53" Body="&lt;p&gt;From the &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-run-a-keras-model-on-multiple-gpus&quot; rel=&quot;noreferrer&quot;&gt;Keras FAQs&lt;/a&gt;, below is copy-pasted code to enable 'data parallelism'. I.e. having each of your GPUs process a different subset of your data independently.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from keras.utils import multi_gpu_model&#xA;&#xA;# Replicates `model` on 8 GPUs.&#xA;# This assumes that your machine has 8 available GPUs.&#xA;parallel_model = multi_gpu_model(model, gpus=8)&#xA;parallel_model.compile(loss='categorical_crossentropy',&#xA;                       optimizer='rmsprop')&#xA;&#xA;# This `fit` call will be distributed on 8 GPUs.&#xA;# Since the batch size is 256, each GPU will process 32 samples.&#xA;parallel_model.fit(x, y, epochs=20, batch_size=256)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Note that this appears to be valid only for the Tensorflow backend at the time of writing.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Update (Feb 2018)&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;p&gt;Keras now accepts automatic gpu selection using multi_gpu_model, so you don't have to hardcode the number of gpus anymore. Details in this &lt;a href=&quot;https://github.com/keras-team/keras/pull/9226&quot; rel=&quot;noreferrer&quot;&gt;Pull Request&lt;/a&gt;. In other words, this enables code that looks like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;try:&#xA;    model = multi_gpu_model(model)&#xA;except:&#xA;    pass&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But to be more &lt;a href=&quot;https://www.python.org/dev/peps/pep-0020/&quot; rel=&quot;noreferrer&quot;&gt;explicit&lt;/a&gt;, you can stick with something like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;parallel_model = multi_gpu_model(model, gpus=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Bonus&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;p&gt;To check if you really are utilizing all of your GPUs, specifically NVIDIA ones, you can monitor your usage in the terminal using:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;watch -n0.5 nvidia-smi&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://keras.io/utils/#multi_gpu_model&quot; rel=&quot;noreferrer&quot;&gt;https://keras.io/utils/#multi_gpu_model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/8223811/top-command-for-gpus-using-cuda&quot;&gt;https://stackoverflow.com/questions/8223811/top-command-for-gpus-using-cuda&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42531" LastEditorUserId="85045" LastEditDate="2021-01-01T04:42:14.593" LastActivityDate="2021-01-01T04:42:14.593" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="25738" PostTypeId="1" CreationDate="2017-12-17T06:47:07.400" Score="1" ViewCount="204" Body="&lt;p&gt;I am very new to machine learning, I started to understand fundamental and using it in &lt;code&gt;azure studio&lt;/code&gt;, I see that there is an algorithm in &lt;code&gt;scikit&lt;/code&gt; for feature selection called &lt;code&gt;SelectKBest&lt;/code&gt;, then I searched azure studio and I coudn't find it by name, I was wondering if is it supported there?&lt;/p&gt;&#xA;" OwnerUserId="43398" LastActivityDate="2017-12-17T19:42:04.180" Title="How to apply SelectKBest in azure machine learning" Tags="|feature-selection|azure-ml|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25739" PostTypeId="1" CreationDate="2017-12-17T07:44:06.213" Score="4" ViewCount="829" Body="&lt;p&gt;I am just interested to know if we can use the generated word embedding vectors to extract keywords from a document or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, how?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastEditorUserId="8878" LastEditDate="2017-12-18T14:10:18.920" LastActivityDate="2018-10-09T10:00:47.007" Title="Word embedding vectors for keyphrase extraction" Tags="|nlp|text-mining|word2vec|word-embeddings|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25740" PostTypeId="1" AcceptedAnswerId="25751" CreationDate="2017-12-17T12:43:07.460" Score="2" ViewCount="83" Body="&lt;p&gt;I am currently working on a classification problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset (2000 features, 25 labels) I am using is seperable by using a 2-layered Multi-Layer-Perceptron (1 Input + 1 OutputLayer = 1 Weight Matrix) with achieving great accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The Data I am using are spectrograms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now what I am curious about is the following: Can I use a deep network (more than 2 layers) to even increase the accuracy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I think about: Increasing the network structure would make the network interpret the data in a more complex way and therefore should be able to even correctly classify little &quot;outbrakers&quot; and &quot;noisy elements&quot; of the dataset.&lt;/p&gt;&#xA;" OwnerUserId="33963" LastEditorUserId="33963" LastEditDate="2017-12-18T19:01:44.957" LastActivityDate="2017-12-18T19:01:44.957" Title="Solving classification task with deep network" Tags="|machine-learning|neural-network|classification|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25741" PostTypeId="2" ParentId="24845" CreationDate="2017-12-17T12:48:00.443" Score="0" Body="&lt;p&gt;You can use Apache OpenNLP for your purpose. First, you need to train your model with training data. In your case the training data may look like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;start:size&amp;gt; Big &amp;lt;end&amp;gt; Apartment at &amp;lt;start:address&amp;gt;Pavelle Road &amp;lt;end&amp;gt; on sale at just &amp;lt;start:price&amp;gt; $400k &amp;lt;end&amp;gt;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you can manage to have a large collection of annotated data like this, you can use the model to get data thereon. For more information look in Apache OpenNLP, its Open Source.&lt;/p&gt;&#xA;" OwnerUserId="37340" LastActivityDate="2017-12-17T12:48:00.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25742" PostTypeId="1" CreationDate="2017-12-17T12:56:58.843" Score="3" ViewCount="36" Body="&lt;p&gt;I have a dataset which contains multiple columns. On analysis, I found out that there were few columns which contain just a single value. There was 0 in all the rows of these columns. Does it hold any information or should I remove such columns from the dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few columns were also like that it contains 100k rows of type1 and 3-4 rows of type2. Should i also remove this columns or let them remain as such? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Target column is different than the columns described.&lt;/p&gt;&#xA;" OwnerUserId="43429" LastEditorUserId="836" LastEditDate="2017-12-17T13:02:11.233" LastActivityDate="2017-12-17T13:23:51.860" Title="Columns with no (or nearly no) differences between rows worth keeping?" Tags="|data|pandas|learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25743" PostTypeId="2" ParentId="25742" CreationDate="2017-12-17T13:16:06.790" Score="2" Body="&lt;p&gt;The features you described are known as low variance features and in general you should remove those features. The rationale behind this, is that low variance feature contain less information. See &lt;a href=&quot;https://chrisalbon.com/machine-learning/variance_thresholding_for_feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for a succinct explanation and some code in python. Be sure to &lt;a href=&quot;https://stats.stackexchange.com/questions/253920/should-we-normalize-before-using-variancethreshold-in-sklearn/&quot;&gt;normalize your features&lt;/a&gt;. One of the advantages is to speed the training phase. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-17T13:23:51.860" LastActivityDate="2017-12-17T13:23:51.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25744" PostTypeId="1" CreationDate="2017-12-17T13:18:04.917" Score="2" ViewCount="91" Body="&lt;p&gt;Imagine this example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZCev1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZCev1.png&quot; alt=&quot;Atributes table&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We see that the attributes &lt;code&gt;color&lt;/code&gt; and &lt;code&gt;points&lt;/code&gt; have the same value. What attribute should we choose for the construction of the tree?&lt;/p&gt;&#xA;" OwnerUserId="43431" LastEditorUserId="26562" LastEditDate="2017-12-17T20:07:58.937" LastActivityDate="2022-02-21T00:06:15.543" Title="Machine Learning - Same impurity values" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25745" PostTypeId="1" CreationDate="2017-12-17T13:49:18.790" Score="2" ViewCount="2163" Body="&lt;p&gt;What is the interpretation of output &lt;code&gt;0.5&lt;/code&gt; of a typical classifier?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made a prediction and the probability of that data point being from the True class is 0.5.&lt;/p&gt;&#xA;" OwnerUserId="32762" LastEditorUserId="28175" LastEditDate="2018-01-10T15:20:13.273" LastActivityDate="2018-01-11T11:26:28.797" Title="Explain Binary Classification with output 0.5 (True)" Tags="|machine-learning|neural-network|classification|deep-learning|" AnswerCount="2" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25746" PostTypeId="2" ParentId="25745" CreationDate="2017-12-17T14:56:06.323" Score="4" Body="&lt;p&gt;It sounds like your model is saying there is a 50% chance that this data point could be in either class. Since this is a special case, prediction algorithms need to pick which class to choose when there is an equal probability for each class. In this case, it looks like your model defaults to the True class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some models in &lt;code&gt;sklearn&lt;/code&gt; like &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM&lt;/a&gt; that have &lt;code&gt;predict_proba&lt;/code&gt; functions. You might find more information if you look there or check if the model you are using has documentation on how its probabilities are calculated.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-17T14:56:06.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25747" PostTypeId="2" ParentId="25745" CreationDate="2017-12-17T16:50:22.300" Score="3" Body="&lt;p&gt;Depending on your algorithm, it may have different interpretations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you are using &lt;strong&gt;SVM&lt;/strong&gt; &lt;code&gt;with kernels&lt;/code&gt;, it means your input data is exactly between two data points with different classes or if you are using &lt;code&gt;linear kernels&lt;/code&gt; in &lt;strong&gt;SVM&lt;/strong&gt; it means that &lt;strong&gt;your data&lt;/strong&gt; is &lt;strong&gt;on the separator line&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using &lt;strong&gt;Neural Networks&lt;/strong&gt;, this is the &lt;strong&gt;Probability&lt;/strong&gt; of each &lt;strong&gt;class&lt;/strong&gt;. In such cases there maybe different explanations for this phenomenon. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Suppose that you are trying to classify images of green and purple&#xA;cats! and suppose that you have two same cats with both colors, then&#xA;your classifier may output this result&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another interpretation may also exist. suppose that you have&#xA;unbalanced data-set for your two classes. suppose that there are&#xA;same feature vectors of both classes. in such cases, the&#xA;distribution of your samples overlap. A class which has more data samples would be the winner and will occupy the space much more than the class with less samples. In such cases, if you get 50 percent as the output, You can not tell that both classes have same expectations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to clarify this more, I would do that with this example.&#xA;Suppose that you have a &lt;strong&gt;car classifier&lt;/strong&gt; for distinguishing&#xA;between white and blue cars. during training you had &lt;strong&gt;100&lt;/strong&gt; images&#xA;of &lt;strong&gt;blue car&lt;/strong&gt; and &lt;strong&gt;20&lt;/strong&gt; images of &lt;strong&gt;white car&lt;/strong&gt;. During &lt;em&gt;recall&#xA;phase&lt;/em&gt;, if for an arbitrary image you have 50 percent for each&#xA;class, you can never say that the image has same probability to be&#xA;each one at all&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="40853" LastEditDate="2018-01-11T11:26:28.797" LastActivityDate="2018-01-11T11:26:28.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25748" PostTypeId="2" ParentId="25731" CreationDate="2017-12-17T18:53:55.820" Score="2" Body="&lt;p&gt;It depends on your task and the amount of data you have. If you have so much data but you can not find similar tasks to have appropriate architecture you should stack convolution and dense layers yourself. But if you have appropriate amount of data and there exist good architectures then you have to decide what you want and how is your situation. Suppose that you want to have recognition task, there are so many architectures that are applied to &lt;code&gt;ImageNet&lt;/code&gt; data-set. You can use transfer learning but there is a point here. Suppose that you want to fine tune &lt;code&gt;GoogleNet&lt;/code&gt;. This is a very large network and is capable for recognizing about a thousand distinct classes. If you have a recognition task with 5 classes and you have an agent that should be online, this is not logical to have such a &lt;strong&gt;big&lt;/strong&gt; network. You may have similar performance by stacking a few layers and get better time complexity. If you don't have so much data, freezing the layers and applying transfer learning to the last layer maybe a typical solution.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-17T18:53:55.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25749" PostTypeId="2" ParentId="155" CreationDate="2017-12-17T19:16:15.010" Score="2" Body="&lt;p&gt;This question already has so many answers but I guess it will be very useful for machine-learning practitioners to use so many &lt;code&gt;CSV&lt;/code&gt; files that are &lt;a href=&quot;http://iranopendata.org/en/datasets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-13T13:35:55.827" LastActivityDate="2018-02-13T13:35:55.827" CommentCount="0" CommunityOwnedDate="2017-12-17T19:16:15.010" ContentLicense="CC BY-SA 3.0" />
  <row Id="25750" PostTypeId="2" ParentId="25738" CreationDate="2017-12-17T19:42:04.180" Score="1" Body="&lt;p&gt;I have found there are other feature selection algorithms there&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see the below screen shots&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CaxvG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CaxvG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xlYOc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xlYOc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43398" LastActivityDate="2017-12-17T19:42:04.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25751" PostTypeId="2" ParentId="25740" CreationDate="2017-12-17T20:47:24.537" Score="2" Body="&lt;p&gt;If you are using two &lt;em&gt;hidden&lt;/em&gt; layers, it may mean that your data is not linearly separable. If you just use one unit in a single hidden layer then you can claim that your data is linearly separable, which in your case you may say that there exist a hyper-plane that separates your data linearly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the answer to your question, yes. It is possible to reduce the amount of error by adding layers or adding extra units to the current architecture but there are some points that are necessary to be considered. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Whenever you add more layers, there will be vanishing and exploding gradients which may cause your network not to learn, or learning may happen so slowly. To avoid, you should use &lt;code&gt;ReLU&lt;/code&gt; activation in order to avoid saturation of gradients. Moreover you have to use &lt;em&gt;He&lt;/em&gt; or &lt;em&gt;Xavier&lt;/em&gt; initialization techniques to avoid having bad random weights. There are other techniques for solving this problem which are called skip connections but at least I've never seen the use of them in &lt;code&gt;MLP&lt;/code&gt;s Although they are really helpful for solving the mentioned problem.&lt;/li&gt;&#xA;&lt;li&gt;Covariat shift is the problem of learning for deeper layers. As a solution you have to use &lt;em&gt;Batch Normalization&lt;/em&gt; to somehow normalize the activations of the deeper layers. &lt;/li&gt;&#xA;&lt;li&gt;Overfitting is a problem that happens for large architectures that are not fed with enough training data. You should use regularization techniques. There are so many papers about this but in your case which is using &lt;code&gt;MLP&lt;/code&gt; I highly recommend you using Dropout technique which is invented by the so called &lt;em&gt;God Father&lt;/em&gt; of deep learning. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-18T04:06:30.477" LastActivityDate="2017-12-18T04:06:30.477" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25752" PostTypeId="1" AcceptedAnswerId="67114" CreationDate="2017-12-17T21:02:54.017" Score="3" ViewCount="10185" Body="&lt;p&gt;I am using this code for a multilabel problem classification.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import print_function&#xA;&#xA;from keras.preprocessing import sequence&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Embedding&#xA;from keras.layers import LSTM&#xA;from keras.models import Model&#xA;from keras.datasets import imdb&#xA;&#xA;max_features = len(vocabDic)&#xA;maxlen = 500  # cut texts after this number of words (among top max_features most common words)&#xA;batch_size = 32&#xA;&#xA;train_set = sequence.pad_sequences(train_set, maxlen=maxlen)&#xA;test_set = sequence.pad_sequences(test_set, maxlen=maxlen)&#xA;&#xA;print('train_set shape:', train_set.shape)&#xA;print('test_set shape:', test_set.shape)&#xA;&#xA;print('Build model...')&#xA;model = Sequential()&#xA;model.add(Embedding(max_features, 128))&#xA;model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))&#xA;model.add(Dense(90, activation='sigmoid'))&#xA;&#xA;# try using different optimizers and different optimizer configs&#xA;model.compile(loss='binary_crossentropy',&#xA;              optimizer='adam',&#xA;              metrics=['accuracy'])&#xA;&#xA;print('Train...')&#xA;model.fit(train_set, train_labels,&#xA;          batch_size=batch_size,&#xA;          epochs=15,&#xA;          validation_data=(test_set, test_labels))&#xA;score, acc = model.evaluate(test_set, test_labels,&#xA;                            batch_size=batch_size)&#xA;print('Test score:', score)&#xA;print('Test accuracy:', acc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My problem is there are 90 classes and the accuracy is too high from the second epoch. I suspect keras is computing something incorrectly.&#xA;Any clues?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I calculated the total recall of the model. It is barely 5%( 5 times better than random). Is it a normal behavior for such a problem?&lt;/p&gt;&#xA;" OwnerUserId="40515" LastEditorUserId="1330" LastEditDate="2017-12-19T19:12:08.203" LastActivityDate="2020-08-18T10:33:33.280" Title="How does keras calculate accuracy for multi label classification?" Tags="|neural-network|text-mining|keras|lstm|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25753" PostTypeId="1" CreationDate="2017-12-17T21:51:21.947" Score="4" ViewCount="1011" Body="&lt;p&gt;I'm trying to find an optimal dithering pattern which can be used as a threshold on a greyscale image to generate a 1 bit black and white image. Ideally it would be optimal in the sense that a human would judge it perceptually most close to the source image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, dithering with white noise looks pretty bad:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3QsEz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3QsEz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While dithering with blue noise looks a lot better, despite having the same amount of error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/noMjF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/noMjF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(images from &lt;a href=&quot;https://blog.demofox.org/2017/10/31/animating-noise-for-integration-over-time/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.demofox.org/2017/10/31/animating-noise-for-integration-over-time/&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While white noise doesn't look good, it can be easily generated in real time, whereas blue noise cannot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of this, I'm trying to find a dithering pattern which is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;NxN in size&lt;/li&gt;&#xA;&lt;li&gt;Has an even distribution of values: 0/N, 1/N ... (N-1)/N&lt;/li&gt;&#xA;&lt;li&gt;Perceptually the best bang for the buck&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After I have the best pattern for a given N (or a reasonably good pattern), step 2 would be to try and figure out how to calculate it mathematically, but the first step is to find the pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any recommendations on a good way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm currently trying is using simulated annealing to find an optimal solution.  I'm using SSIM to calculate the score of how good a solution is, for use in in the simulated annealing logic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the simulated annealing, I initialize the pattern by doing this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Initialize the NxN pattern such that row M is set to M/N.&lt;/li&gt;&#xA;&lt;li&gt;Shuffle the pattern&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When calculating a possible move to make for the simulated annealing, I pick two values at random in the NxN pattern and swap them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't working very well for me, but I am unsure if this is not a good way to go about it, or if maybe I'm choosing bad parameters for the simulating annealing and/or the SSIM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simulated annealing has a starting temperature, and a specified number of iterations to take it from there down to temperature 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SSIM on the other hand has c1, c2, and c3 which I'm setting to zero.  I am also doing SSIM of every 8x8 block in the dithered image (ALL of them, even though they overlap) and averaging them together to get the SSIM score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of a 5x5 dither pattern this algorithm came up with, which really doesn't look good at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AcbDl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AcbDl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I manually derived a 5x5 by hand which looks much better in fact:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dg8Gd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dg8Gd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any recommendations on how I might search / solve this problem better?&lt;/p&gt;&#xA;" OwnerUserId="27729" LastActivityDate="2018-01-28T18:00:00.000" Title="How to search for an optimal dithering pattern?" Tags="|gradient-descent|meta-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25754" PostTypeId="1" AcceptedAnswerId="25759" CreationDate="2017-12-17T21:51:57.997" Score="17" ViewCount="23559" Body="&lt;p&gt;I am currently trying to understand the architecture of a CNN. I understand the convolution, the ReLU layer, pooling layer, and fully connected layer. However, I am still confused about the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a normal neural network, each neuron has its own weight. In the fully connected layer, each neuron would also have its own weight. But what I dont know is if each filter has its own weight. Do I just have to update the weights in the fully connected layer during back propagation? Or do the filters all have a separate weight that I need to update?&lt;/p&gt;&#xA;" OwnerUserId="43441" LastEditorUserId="26562" LastEditDate="2017-12-17T22:09:48.633" LastActivityDate="2017-12-18T01:06:31.580" Title="Updating the weights of the filters in a CNN" Tags="|machine-learning|neural-network|deep-learning|image-classification|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25755" PostTypeId="2" ParentId="25754" CreationDate="2017-12-17T21:56:22.603" Score="4" Body="&lt;p&gt;During back propagation both dense layers and convolution layers get updated but max-pooling layers do not have any weight to be updated. Dense layers are updated to help the net to classify. Convolution layers get updated to let the network learn the features itself. Because you have not asked in the question I just add a link for you if you want to know more. There is a very good explanation for back propagation &lt;a href=&quot;http://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which can be useful for you.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-17T22:10:00.810" LastActivityDate="2017-12-17T22:10:00.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25756" PostTypeId="1" AcceptedAnswerId="25757" CreationDate="2017-12-17T22:41:28.680" Score="1" ViewCount="46" Body="&lt;p&gt;I have a script from a lecture. Basically it says that based on the Voronoi partitioning we identify the corresponding (nearest) class $w_k$ to a vector $x$ where $\left| {{w_k} - x} \right| = \mathop {\min }\limits_i \left( {\left| {{w_i} - x} \right|} \right)$ given the classes $w$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The script uses the absolute value notation. This does not make much sense as we are using vectors. &lt;strong&gt;Which vector norms can or should be used?&lt;/strong&gt; The 1 or 2 norm?&#xA;The 1 norm is faster to compute but maybe there are drawbacks I cannot currently think of.&lt;/p&gt;&#xA;" OwnerUserId="28011" LastActivityDate="2017-12-18T00:00:53.157" Title="Use 1 or 2 norm for Voronoi vector quantization?" Tags="|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25757" PostTypeId="2" ParentId="25756" CreationDate="2017-12-18T00:00:53.157" Score="1" Body="&lt;p&gt;In general the quality of the quantization is measured using the mean squared error (MSE) between the input vector $x$ and the reproduction value $q(x)$ ($w_k$ in the notation you used in the question). For the MSE the best partition is the one defined by Voronoi regions under the euclidean distance ($l^2$ norm), i.e: &#xA;$$V_j = \{ x \in R^d : \|x - w_j \|^2 \leq \|x - w_i \|^2\ for\ all\ i   \}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find the proof in &lt;a href=&quot;http://gubner.ece.wisc.edu/ece729/vq.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Also &lt;a href=&quot;https://hal.archives-ouvertes.fr/file/index/docid/514462/filename/paper_hal.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.lx.it.pt/~mtf/Vector_quantization.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; the euclidean distance is used.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-18T00:00:53.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25758" PostTypeId="2" ParentId="25731" CreationDate="2017-12-18T00:59:46.793" Score="2" Body="&lt;p&gt;Generally, if you can find a network that has been trained on data remotely similar to yours, then it's a good idea to take advantage of this pre-trained network and use &lt;em&gt;fine-tuning / transfer learning&lt;/em&gt; to train the model on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transfer learning occurs when we use knowledge that was gained from solving one problem and apply it to a new, but related problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fine-tuning is a way of applying or utilizing transfer learning. Specifically, fine-tuning is a process that takes a model that has already been trained for a given task and then &lt;em&gt;tuning&lt;/em&gt; the model to make it perform a second task. Assuming the original task is at least loosely similar to the new task, using an artificial neural network that has already been created and trained allows us to take advantage of what the model has already learned without having to develop it from scratch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I explain the concept of fine-tuning in &lt;a href=&quot;https://youtu.be/5T-iXNNiwIs&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt;. At 4:10, it also links to videos showing how this is process is done in code using Keras.&lt;/p&gt;&#xA;" OwnerUserId="43444" LastEditorUserId="43444" LastEditDate="2017-12-18T01:24:15.463" LastActivityDate="2017-12-18T01:24:15.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25759" PostTypeId="2" ParentId="25754" CreationDate="2017-12-18T01:06:31.580" Score="15" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;In a normal neural network, each neuron has its own weight.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not correct. Every connection between neurons has its own weight. In a fully connected network each neuron will be associated with many different weights. If there are &lt;code&gt;n0&lt;/code&gt; inputs (i.e. &lt;code&gt;n0&lt;/code&gt; neurons in the previous layer) to a layer with &lt;code&gt;n1&lt;/code&gt; neurons in a fully connected network, that layer will have &lt;code&gt;n0*n1&lt;/code&gt; weights, not counting any bias term.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be able to see this clearly in this diagram of a fully connected network from &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;noreferrer&quot;&gt;CS231n&lt;/a&gt;. Every edge you see represents a different trainable weight:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UHrnq.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UHrnq.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Convolutional layers are different in that they have a fixed number of weights governed by the choice of filter size and number of filters, but independent of the input size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each filter has a separate weight in each position of its shape. So if you use two 3x3x3 filters then you will have 54 weights, again not counting bias. This is illustrated in a second diagram from &lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;noreferrer&quot;&gt;CS231n&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S5B1k.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S5B1k.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The filter weights absolutely must be updated in backpropagation, since this is how they learn to recognize features of the input.  If you read the section titled &quot;Visualizing Neural Networks&quot; &lt;a href=&quot;https://github.com/nehal96/Deep-Learning-ND-Exercises/blob/master/Convolutional%20Neural%20Networks/convolutional-neural-networks-notes.md&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; you will see how layers of a CNN learn more and more complex features of the input image as you got deeper in the network. These are all learned by adjusting the filter weights through backpropagation.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2017-12-18T01:06:31.580" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25761" PostTypeId="2" ParentId="24845" CreationDate="2017-12-18T03:09:30.810" Score="0" Body="&lt;p&gt;Contrary to what others are suggesting, trying to extract data on square footage and number of rooms from apartment ads is not a problem for machine learning - especially not when you don't have any training data. You will spend 10 times as long trying to build a machine learning system for this task than you would manually extracting the data yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want an automated system, consider building some simple regex rules based on ads you have seen. There are only so many ways people describe these features, and you should be able to get most of them when they are present in the ad. You won't get 100% accuracy, but you're not going to do any better another way, in my opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can start with some simple rules, and improve them as you look at more and more ads. In Python, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;&#xA;ad = '$3582 / 1br 1ba - 724ft2 - Breathtaking Seaport District'&#xA;&#xA;sqft_pattern = '([0-9]+)\s?(sq|ft)'&#xA;beds_pattern = '(([1-9](\.5)?)\s?(br|bd|beds?|bedrooms?))|(beds|bedrooms):?\s?[1-9]'&#xA;bath_pattern = '(([1-9](\.5)?)\s?(ba|baths?|bathrooms?))|(baths|bathrooms):?\s?[1-9]'&#xA;&#xA;m = re.search(sqft_pattern, ad)&#xA;sqft = m.group(1) if m else None&#xA;m = re.search(beds_pattern, ad)&#xA;beds = m.group(2) if m else None&#xA;m = re.search(bath_pattern, ad)&#xA;bath = m.group(2) if m else None&#xA;&#xA;print('square  feet: {}'.format(sqft))&#xA;print('bedrooms: {}'.format(beds))&#xA;print('bathrooms: {}'.format(bath))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;square feet: 724&#xA;bedrooms: 1&#xA;bathrooms: 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-18T03:26:40.017" LastActivityDate="2017-12-18T03:26:40.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25763" PostTypeId="1" AcceptedAnswerId="25764" CreationDate="2017-12-18T08:34:29.863" Score="1" ViewCount="8462" Body="&lt;p&gt;I have a data frame which has two columns, &quot;Title&quot; and &quot;Description&quot;. The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same test but, due to minor changes in the titles, titles are shown as unique.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]&#xA;&#xA;labels = ['title', 'description']&#xA;import pandas as pd&#xA;labtest = pd.DataFrame.from_records(values, columns = labels) # Create data frame&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is how the data frame looks like. [The actual dataset has many such titles, this is only for the purpose of this question]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Title                       Description&#xA;Complete blood test         AB&#xA;COMPLETE Blood test\        AB&#xA;Blood glucose               AB&#xA;Complete blood picture      AB&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is what I would the like the data frame to look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Title                       Description&#xA;Blood test                   AB&#xA;Blood test                   AB&#xA;Blood test                   AB&#xA;Blood test                   AB&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to search for the word &quot;blood&quot; in each title and if its true, then change the whole title with &quot;Blood test&quot;. Is there a way to do this?&lt;/p&gt;&#xA;" OwnerUserId="43458" LastEditorUserId="29575" LastEditDate="2017-12-18T16:15:56.507" LastActivityDate="2017-12-19T03:58:27.697" Title="Grouping of similar looking text" Tags="|python|pandas|text|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25764" PostTypeId="2" ParentId="25763" CreationDate="2017-12-18T09:10:28.590" Score="3" Body="&lt;p&gt;One possible solution is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;import pandas as pd&#xA;&#xA;pattern = re.compile('blood', re.IGNORECASE)&#xA;&#xA;def change(text):&#xA;    if pattern.findall(text):&#xA;        return 'Blood test'&#xA;    else:&#xA;        return text&#xA;&#xA;values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]&#xA;labels = ['title', 'description']&#xA;&#xA;# Create data frame&#xA;labtest = pd.DataFrame.from_records(values, columns=labels)&#xA;labtest['title'] = labtest['title'].apply(change)&#xA;&#xA;print labtest&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        title description&#xA;0  Blood test          AB&#xA;1  Blood test          AB&#xA;2  Blood test          AB&#xA;3  Blood test          AB&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The first line import the regex (regular expression) module of Python. The line: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pattern = re.compile('blood', re.IGNORECASE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;creates a regex that finds the word blood ignoring case. The function change, replace the input text with 'Blood test' in case the string 'blood' was found. Finally you used the apply method from pandas DataFrame to transform the column. Finally the apply method, as the name suggests, 'applies' the function change to every value in the 'title' column.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More info on regular expressions using Python and pandas apply method can be found &lt;a href=&quot;https://docs.python.org/2/library/re.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If you want to know more about text processing in Python I would recommend you take a look at the pointers in this &lt;a href=&quot;https://stackoverflow.com/questions/676253/python-and-text-manipulation&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-18T09:20:53.367" LastActivityDate="2017-12-18T09:20:53.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25765" PostTypeId="2" ParentId="25727" CreationDate="2017-12-18T09:28:36.750" Score="3" Body="&lt;p&gt;&quot;Higher dimensional representation&quot; refers to having more feature maps like you suggested (check the inception module proposed in figure 7).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to what disentangled features mean, I believe they mean decorrelated: the more features (with different filters) the inception module extracts, the more and the faster the network learns. Because the network will have access to more information leading to detecting which features are salient earlier in the training. &lt;/p&gt;&#xA;" OwnerUserId="43459" LastActivityDate="2017-12-18T09:28:36.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25766" PostTypeId="1" AcceptedAnswerId="25795" CreationDate="2017-12-18T10:30:40.363" Score="0" ViewCount="1974" Body="&lt;p&gt;I'm trying to plot two different size matrices using one graph (in R), but can't manage to do so.&#xA;I tried using &lt;code&gt;matplot&lt;/code&gt; and the regular &lt;code&gt;plot&lt;/code&gt;, but it didn't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to plot it?&lt;/p&gt;&#xA;" OwnerUserId="43463" LastEditorUserId="26562" LastEditDate="2017-12-18T10:37:59.620" LastActivityDate="2017-12-19T08:56:48.490" Title="Plot of two different matrices in R" Tags="|r|plotting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25767" PostTypeId="2" ParentId="25350" CreationDate="2017-12-18T10:31:30.737" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;However, in the overall scheme, sliding this network can be represented by two 3 x 3 convolutional layers which reuses the activation between adjacent tiles. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since the replacement (two 3x3 instead of one 5x5) share weights, we don't have to calculate them twice. That is where the gain comes from.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gain comes from the sliding: using the following formula (n-filtersize+1) x (n-filtersize+1) that calculates the output of a filter on an n x n input &lt;/p&gt;&#xA;&#xA;&lt;p&gt;a more detailed answer can be found here: &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/43382499/reducing-filter-size-in-convolutional-neural-network&quot;&gt;Reducing Filter Size in Convolutional Neural Network&lt;/a&gt; &#xA;Thank you @Thomas W. &lt;/p&gt;&#xA;" OwnerUserId="43459" LastEditorUserId="43459" LastEditDate="2017-12-20T07:49:15.563" LastActivityDate="2017-12-20T07:49:15.563" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25768" PostTypeId="1" CreationDate="2017-12-18T10:39:09.657" Score="1" ViewCount="290" Body="&lt;p&gt;&lt;strong&gt;When trying to overfit&lt;/strong&gt; the network, what is the &lt;strong&gt;practical&lt;/strong&gt; maximum depth of an LSTM Neural network before it will start to fall apart? If possible, what is the state-of-the-art depth in LSTMs produced by companies like Google?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that in real world number of layers is implementation-dependent &amp;amp; number of neurons can be estimated, &lt;a href=&quot;https://stats.stackexchange.com/a/1097/187816&quot;&gt;[1]&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/a/35522597/9007125&quot;&gt;[2]&lt;/a&gt; but curious if my results are typical:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've implemented a system in C++, using iRProp+ &lt;a href=&quot;https://pdfs.semanticscholar.org/df9c/6a3843d54a28138a596acc85a96367a064c2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(here is the paper, section 3.1)&lt;/a&gt;, however, I am struggling to get the error-propagation under control when going above &lt;strong&gt;10 layers&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am just trying to overfit the network, it successfully and very quickly converges with 4-6 layers (~100 iRProp iterations and the error is down to 0.00001, where it started at around 4.00000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network tries to predict the next character in the alphabet (made from 26 characters), so each layer has an LSTM that works with 27-dimensional vectors. The error-propagation happens after 25 timesteps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I crank-up to 10-14 layers, the error is really hesitant to even begin climbing down, and seems to simply oscillate around the 3.6 value. In incredibly rare cases if a weight initialization was lucky, with 10-14 layers the error will decrease, but usually it will just oscillate. Is that usual? I am using &lt;code&gt;float&lt;/code&gt; datatype, however, tested the double datatype and the oscilations still happen, so I doubt it's anything to do with precision. Adjusting the $n$ value (acceleration) doesn't seem to affect it either&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Been looking at examples &lt;a href=&quot;http://www.jakob-aungiers.com/articles/a/LSTM-Neural-Network-for-Time-Series-Prediction&quot; rel=&quot;nofollow noreferrer&quot;&gt;[3]&lt;/a&gt;, but so far getting impression people use a maximum of 2-4 layers...&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2017-12-19T15:23:34.043" LastActivityDate="2017-12-19T15:23:34.043" Title="A practical max-depth of LSTM layers, when trying to overfit?" Tags="|backpropagation|lstm|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25769" PostTypeId="2" ParentId="25763" CreationDate="2017-12-18T11:09:32.310" Score="1" Body="&lt;p&gt;Another solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_values=[]&#xA;for tup in values:&#xA;    if tup[0].lower().find('blood')&amp;gt;=0:&#xA;        new_values.append(['Blood test',tup[1]])&#xA;    else: new_values.append([ tup[0],tup[1] ])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This basically takes your &lt;strong&gt;values&lt;/strong&gt; list and creates an &lt;strong&gt;new_values&lt;/strong&gt; list with the replaced text. See below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values =     [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]&#xA;new_values = [['Blood test', 'AB'], ['Blood test', 'AB'], ['Blood test', 'AB'], ['Blood test', 'AB']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So now you can use &lt;strong&gt;new_values&lt;/strong&gt; in your dataframe (&lt;em&gt;pd.DataFrame.from_records(new_values, columns = labels)&lt;/em&gt;) or use it to replace &lt;strong&gt;values&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43447" LastActivityDate="2017-12-18T11:09:32.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25770" PostTypeId="2" ParentId="25763" CreationDate="2017-12-18T12:27:39.167" Score="2" Body="&lt;p&gt;Pandas can directly do that &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.contains.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;string compare&lt;/a&gt;, then use the compare result to lookup appropriate rows so that they can be set.  This can be done with a single expression as:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;How does this work?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;From the inside out we have:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Select the 'title' column as a string vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labtest['title'].str&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Convert the string vector to a boolean vector:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labtest['title'].str.contains('blood', case=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use the boolean vector to select specific rows in the title column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labtest['title'][labtest['title'].str.contains('blood', case=False)] &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assign those cells to the desired new value&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values = [&#xA;    ('Complete blood picture', 'AB'),&#xA;    ('Complete BLOOD test', 'AB'),&#xA;    ('blood glucose', 'AB'),&#xA;    ('COMplete blood Profile', 'AB'),&#xA;    ('bloud glucose', 'AB'),&#xA;]&#xA;labels = ['title', 'description']&#xA;&#xA;# Create data frame&#xA;labtest = pd.DataFrame.from_records(values, columns=labels)&#xA;&#xA;labtest['title'][labtest['title'].str.contains('blood', case=False)] = 'Blood test'&#xA;&#xA;print(labtest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           title description&#xA;0     Blood test          AB&#xA;1     Blood test          AB&#xA;2     Blood test          AB&#xA;3     Blood test          AB&#xA;4  bloud glucose          AB&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="29575" LastEditDate="2017-12-19T03:58:27.697" LastActivityDate="2017-12-19T03:58:27.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25771" PostTypeId="1" CreationDate="2017-12-18T12:34:01.187" Score="1" ViewCount="39" Body="&lt;p&gt;I understand that dropout is used to drop some weights during training, is there a way one can conventionally train a neural network with full weights and then drop some weights at the inference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that during testing, for instance using MNIST data-sets, the test accuracy is computed as: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_accuracy = accuracy.eval ({X: mnist.test.images, Y: mnist.test.labels})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I modify the code so that some weights are set to zero?&lt;/p&gt;&#xA;" OwnerUserId="43467" LastEditorUserId="29169" LastEditDate="2021-02-13T19:19:45.930" LastActivityDate="2021-02-13T19:19:45.930" Title="How to drop input channels or neurons at the inference phase" Tags="|machine-learning|neural-network|convolutional-neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25772" PostTypeId="1" AcceptedAnswerId="25798" CreationDate="2017-12-18T12:45:30.210" Score="0" ViewCount="3147" Body="&lt;p&gt;I would like to know which is the correct procedure for inferring vectors in &lt;em&gt;Gensim doc2vec&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataframe &lt;code&gt;df&lt;/code&gt; with a feature, called &lt;code&gt;name&lt;/code&gt;, and composed of two subsets &lt;code&gt;train&lt;/code&gt; and &lt;code&gt;test&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df = train + test&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim is to find the most similar &lt;code&gt;name&lt;/code&gt; in &lt;code&gt;train&lt;/code&gt; given a &lt;code&gt;name&lt;/code&gt; in &lt;code&gt;test&lt;/code&gt;.&#xA;For doing this I have to train the &lt;em&gt;doc2vec&lt;/em&gt; model, and I have two possible choices:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;train the model on the entire &lt;code&gt;df&lt;/code&gt; and then infer the most similar &lt;code&gt;name&lt;/code&gt; by &lt;code&gt;model.infer_vector()&lt;/code&gt; on &lt;code&gt;test&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;train the model on &lt;code&gt;train&lt;/code&gt;, &lt;strong&gt;letting out&lt;/strong&gt; &lt;code&gt;test&lt;/code&gt;, and then use &lt;code&gt;model.infer_vector()&lt;/code&gt; on &lt;code&gt;test&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I suppose that the correct procedure is first one, but I am not sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, so doing, there is the possibility that the most similar &lt;code&gt;name&lt;/code&gt;  given &lt;code&gt;test&lt;/code&gt; will be again in &lt;code&gt;test&lt;/code&gt; and not in &lt;code&gt;train&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2017-12-19T09:49:25.470" Title="How to correctly infer vectors in Gensim doc2vec?" Tags="|python|gensim|similar-documents|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25775" PostTypeId="1" CreationDate="2017-12-18T14:51:35.490" Score="1" ViewCount="111" Body="&lt;p&gt;I found a code for a classifier for a  6 class problem where the classes are integers from 0 to 5.&#xA;I found this code that I tested and it improved my accuracy highly.&#xA;Can anyone explain the spirit behind the idea of transforming the problem into a binary classification and how it improves the results?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Classifier(BaseEstimator):&#xA;    def __init__(self):&#xA;        self.clf1 = BernoulliNB()&#xA;        self.clf2 = BernoulliNB()&#xA;&#xA;    def fit(self, X, y):&#xA;        y1 = y.copy()&#xA;        y1[y==0] = 1; y1[y==2] = 1; y1[y==3] = 4; y1[y==5] = 4&#xA;        self.clf1.fit(X, y)&#xA;        y2 = y.copy()&#xA;        y2[y==1] = 0; y2[y==2] = 0; y2[y==3] = 0; y2[y==4] = 1; y2[y==5] = 1&#xA;        self.clf1.fit(X, y1)&#xA;        self.clf2.fit(X,y2)&#xA;&#xA;    def predict(self, X):&#xA;        self.clf1.predict(X)&#xA;        self.clf2.predict(X)&#xA;&#xA;    def predict_proba(self, X):&#xA;        A = np.array([[0,0],[1,0],[0,0],[0,0],[0,1],[0,0]])&#xA;        proba = self.clf1.predict_proba(X)&#xA;        B = self.clf2.predict_proba(X)&#xA;        #return proba_modified(np.array(list(map(lambda x: A@x, proba))),B)&#xA;        return np.array(list(map(lambda x: A.dot(x), proba)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40515" LastActivityDate="2017-12-18T14:51:35.490" Title="Transforming multiclass problem into binary problem to improve accuracy" Tags="|multiclass-classification|binary|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25776" PostTypeId="1" CreationDate="2017-12-18T14:59:20.990" Score="2" ViewCount="648" Body="&lt;p&gt;I need to use a customized linkage function for my project. For this function, on every step I should calculate the objective containing &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;similarity (distance) between subclusters to be merged &lt;/li&gt;&#xA;&lt;li&gt;distances between leaves and roots in terms of edges in the forest of subclusters.         &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Currently I'm using scikit-learn AgglomerativeClustering. I looked through a &lt;a href=&quot;https://datascience.stackexchange.com/questions/11304/hierarchical-clustering-customized-linkage-function&quot;&gt;question related to mine&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that the merging operation for max link and average link happens in &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/51a765acfa4c5d1ec05fc4b406968ad233c75162/sklearn/cluster/_hierarchical.pyx&quot; rel=&quot;nofollow noreferrer&quot;&gt;the script&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I cannot find where the the distances between all the subclusters are calculated and how to derive a number of edges between leaves and roots in subclusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this be done?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPD: I am trying to implement &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2915&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chakrabarti&lt;/a&gt; algorithms for smooth clustering over time. For agglomerative clustering, a choice for next clusters to be merged depends on two components: similarity between subclusters at current time stamp and a historical cost related to the previous time stamp. The latter has several variations. For example, one includes merge distance (not only that): if we would merge two subclusters S1 and S2, what would be the average distance (in terms of edges) between all leaves in S1 and S2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of questions about sklearn AgglomerativeClustering:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What lines in the code implement a choice of clusters to be merged at the current step?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is it possible to extract a distance between a leaf and a root of a subcluster at each merging step? I noticed there is an attribute 'children_' in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AgglomerativeClustering&lt;/a&gt; which I could use to create the resulting tree and find distances between any nodes. How to extract such an information for every merging step?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43473" LastEditorUserId="43473" LastEditDate="2017-12-19T10:23:43.920" LastActivityDate="2017-12-19T10:23:43.920" Title="Customized function for Agglomerative Clustering" Tags="|machine-learning|python|scikit-learn|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25777" PostTypeId="2" ParentId="25776" CreationDate="2017-12-18T15:31:08.213" Score="1" Body="&lt;p&gt;Most implementations will begin with a distance matrix (and not a data matrix) as input, as the algorithms are O(n²) anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second question, I don't understand how that is defined. If your linkage is not a Lance Williams update, you may need to reimplement hierarchical clustering completely, to make sure you have all the information you need. Lance Williams are much more elegant to compute.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-18T15:31:08.213" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25779" PostTypeId="1" AcceptedAnswerId="25784" CreationDate="2017-12-18T16:52:48.520" Score="4" ViewCount="631" Body="&lt;p&gt;I have built a machine learning classifier using Sklearn and pandas as my main tools. Now, one of the input features to the model is country (to letter country code such as US). I have fit a model using the &lt;code&gt;pd.get_dummies&lt;/code&gt; function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to run inference on the data but a few of the countries haven't appeared in my dataset over the past months, so the &lt;code&gt;pd.get_dummies&lt;/code&gt; function is misaligned with the fitted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can this dealt with?&lt;/p&gt;&#xA;" OwnerUserId="40740" LastEditorUserId="29575" LastEditDate="2017-12-19T03:42:43.367" LastActivityDate="2017-12-19T03:42:43.367" Title="Creating dummy variables to match fitted model at inference" Tags="|scikit-learn|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25780" PostTypeId="1" AcceptedAnswerId="25783" CreationDate="2017-12-18T17:07:22.010" Score="1" ViewCount="236" Body="&lt;p&gt;$$\mathcal{L}(w,b,\xi,\alpha,r) = \frac12w^Tw+C\sum_{i=1}^m \xi_i-\sum_{i=1}^m \alpha_i[y^{(i)}(x^Tw+b)-1+\xi_i]-\sum_{i=1}^mr_i\xi_i$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, the $\alpha_i$'s and $r_i$'s are our Lagrange multipliers (constrained to be $\ge 0$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To maximize the Lagrangian of soft margin SVM (see the formula above), we set the derivatives with respect to $w$, $\xi$ and $b$ to $0$ respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what if we set the derivatives w.r.t $r$ to zero first? Wouldn't that result in $\xi$ being all $0$s? Meaning that the optimal solution is reached only when all the relaxing terms $\xi$ are $0$? But that doesn't seem right, does it?&lt;/p&gt;&#xA;" OwnerUserId="40540" LastEditorUserId="38892" LastEditDate="2017-12-19T00:39:20.297" LastActivityDate="2017-12-19T00:39:20.297" Title="A math question about solving the Lagrangian of Support Vector Machine" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25781" PostTypeId="1" CreationDate="2017-12-18T17:14:07.607" Score="3" ViewCount="690" Body="&lt;p&gt;I'm trying to use autoencoders in keras to create a linear transformation similar to independent component analysis (ICA) (using this to denoise electroencephalographic data, time series of 64x100000 pts). While the autoencoder does a good job of re-creating the input using a smaller number of neurons in the hidden layers, there's no structure to the weights in the hidden layers, i.e., it doesn't seem to isolate structure in the data, it just mixes everything up in the compressed layers. For example, ICA is able to isolate things like heart-beats, eye-blinks, and brain activity, but the reconstructed time-series in the hidden layers are just linear mixtures of all of these different signals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to force the time series constructed in the hidden layers to be temporally independent? I'm thinking I'll have to create some kind of custom regularizer, but I have no idea how to go about this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my keras code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import scipy as scipy&#xA;from keras.models import Sequential&#xA;from keras.layers.core import Dense, Activation, Dropout&#xA;from keras.layers import BatchNormalization&#xA;import matplotlib.pyplot as plt &#xA;from keras import regularizers&#xA;from keras.constraints import max_norm, non_neg, unit_norm&#xA;from keras import backend as K&#xA;&#xA;raw = scipy.io.loadmat('raw.mat')&#xA;raw = raw['raw']; &#xA;&#xA;#acts = scipy.io.loadmat('acts.mat')&#xA;#acts = acts['acts'];&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(32,input_shape=(64,)))&#xA;model.add(Activation('linear'))&#xA;model.add(Dense(16))&#xA;model.add(Activation('linear'))&#xA;model.add(Dense(64))&#xA;model.add(Activation('linear'))&#xA;&#xA;model.summary()&#xA;&#xA;model.compile(optimizer='Adamax',loss='mean_absolute_error')&#xA;xtrain = raw[:,:].transpose()&#xA;#xref = acts[:,:].transpose() &#xA;model.fit(xtrain, xtrain, verbose=1, batch_size=200, epochs=20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43476" LastEditorUserId="26562" LastEditDate="2017-12-19T00:45:27.673" LastActivityDate="2017-12-19T00:45:27.673" Title="Using an autoencoder to mimic independent component analysis?" Tags="|machine-learning|deep-learning|keras|autoencoder|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25782" PostTypeId="1" AcceptedAnswerId="29628" CreationDate="2017-12-18T18:21:29.833" Score="0" ViewCount="1170" Body="&lt;p&gt;If you used the analyzeSentiment function from SentimentAnalysis in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tm)&#xA;library(SentimentAnalysis)&#xA;cor &amp;lt;- VCorpus(VectorSource(c('testing sentiment', 'this is great'))&#xA;sent &amp;lt;- analyzeSentiment(cor)&#xA;names(sent)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The names of the columns are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;WordCount&quot;, &quot;SentimentGI&quot;, &quot;NegativityGI&quot;, &quot;PositivityGI&quot;, &quot;SentimentHE&quot;, &quot;NegativityHE&quot;, &quot;PositivityHE&quot;, &quot;SentimentLM&quot;, &quot;NegativityLM&quot;, &quot;PositivityLM&quot;, &quot;RatioUncertaintyLM&quot; &quot;SentimentQDAP&quot;, &quot;NegativityQDAP&quot;, &quot;PositivityQDAP&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than wordcount, what do all these labels mean?  I didn't see any definitions in the &lt;a href=&quot;https://cran.r-project.org/web/packages/SentimentAnalysis/vignettes/SentimentAnalysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;vignette&lt;/a&gt; or &lt;a href=&quot;https://www.rdocumentation.org/packages/SentimentAnalysis/versions/1.3-0/topics/analyzeSentiment&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="18481" LastActivityDate="2023-10-02T13:32:22.773" Title="What do all the column names mean in the SentimentAnalysis package's analyzeSentiment() function in R?" Tags="|r|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25783" PostTypeId="2" ParentId="25780" CreationDate="2017-12-18T18:38:38.527" Score="3" Body="&lt;p&gt;Since you have an inequality constraint you need to meet the necessary &lt;a href=&quot;https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions#Necessary_conditions&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kuhn-Tucker Conditions&lt;/a&gt; which for non-negativity constraints are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$\xi \geq 0$&lt;/li&gt;&#xA;&lt;li&gt;$-r \cdot \xi = 0$ &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is not enough to set the derivative w.r.t. to $r$ to zero as is the case for equality constraints and &lt;a href=&quot;https://en.wikipedia.org/wiki/Lagrange_multiplier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lagrange multipliers&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, there are &lt;strong&gt;two&lt;/strong&gt; cases to distinguish:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Either the condition is tight ($\xi=0$) and you can have $r \neq 0$&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Or&lt;/strong&gt; $\xi &amp;gt; 0$ and $r = 0$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2017-12-18T21:26:12.340" LastActivityDate="2017-12-18T21:26:12.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25784" PostTypeId="2" ParentId="25779" CreationDate="2017-12-18T19:46:59.597" Score="3" Body="&lt;p&gt;I think &lt;a href=&quot;https://stackoverflow.com/questions/41335718/keep-same-dummy-variable-in-training-and-testing-data&quot;&gt;the answer to this question&lt;/a&gt; will solve your problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;train = pd.DataFrame(data = [['a', 123, 'ab'], ['b', 234, 'bc']],&#xA;                     columns=['col1', 'col2', 'col3'])&#xA;test = pd.DataFrame(data = [['c', 345, 'ab'], ['b', 456, 'ab']],&#xA;                     columns=['col1', 'col2', 'col3'])&#xA;&#xA;train_objs_num = len(train)&#xA;dataset = pd.concat(objs=[train, test], axis=0)&#xA;dataset_preprocessed = pd.get_dummies(dataset)&#xA;train_preprocessed = dataset_preprocessed[:train_objs_num]&#xA;test_preprocessed = dataset_preprocessed[train_objs_num:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If that doesn't help, a simple solution would be to add columns of all zeros for each country that the new dataset it missing. This will give you the correct shape of data that your classifier is expecting.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastEditorUserId="43215" LastEditDate="2017-12-18T19:54:19.440" LastActivityDate="2017-12-18T19:54:19.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25786" PostTypeId="1" CreationDate="2017-12-18T23:48:13.190" Score="0" ViewCount="3012" Body="&lt;p&gt;I am curious about the network architecture of &lt;a href=&quot;https://en.wikipedia.org/wiki/AlphaZero&quot; rel=&quot;nofollow noreferrer&quot;&gt;AlphaZero&lt;/a&gt;. Can anybody tell me details about it? Is it the same as in AlphaGo Zero?&lt;/p&gt;&#xA;" OwnerUserId="43488" LastEditorUserId="29575" LastEditDate="2017-12-18T23:55:13.727" LastActivityDate="2017-12-20T14:13:04.537" Title="Network architecture of AlphaZero" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2017-12-26T04:15:50.787" ContentLicense="CC BY-SA 3.0" />
  <row Id="25787" PostTypeId="2" ParentId="16797" CreationDate="2017-12-19T02:46:16.787" Score="4" Body="&lt;p&gt;AP is averaged over all categories. Traditionally, this is called &quot;mean average precision&quot; (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cocodataset.org/#detections-eval&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cocodataset.org/#detections-eval&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43495" LastActivityDate="2017-12-19T02:46:16.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25788" PostTypeId="1" CreationDate="2017-12-19T03:31:41.093" Score="0" ViewCount="46" Body="&lt;p&gt;I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works mainly on the applied/business side of things. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By advanced theoretical knowledge, I mean things: Like understanding the details of how the Kernel trick works in SVM, the &quot;no free lunch&quot; theorem and its relationship to machine learning, the details of how neural networks are universal approximators, the VC dimension of a classifier, etc... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By applied Data Scientist, I mean someone who solves business problems using existing algorithms as opposed to someone who develops new ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question: Are there situations in an applied Data Scientist's life where such theoretical knowledge is useful? Or is this type of knowledge useful only for people who work on developing new algorithms? &lt;/p&gt;&#xA;" OwnerUserId="8517" LastActivityDate="2017-12-19T06:36:28.093" Title="Situations where advanced theoretical knowledge of ML helped solve a real world problem?" Tags="|self-study|theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25789" PostTypeId="1" CreationDate="2017-12-19T06:08:02.173" Score="8" ViewCount="14984" Body="&lt;p&gt;From Jon Shlens's &lt;a href=&quot;https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;A Tutorial on Principal Component Analysis - version 1&lt;/em&gt;&lt;/a&gt;, page 7, section 4.5, II:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The formalism of sufficient statistics captures the notion that the&#xA;  mean and the variance entirely describe a probability distribution.&#xA;  The only zero-mean probability distribution that is fully described by&#xA;  the variance is the Gaussian distribution. In order for this&#xA;  assumption to hold, the probability distribution of $x_i$ must be&#xA;  Gaussian.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;($x_i$ denotes a random variable - the value of the $i^\text{th}$ original feature.&lt;br/&gt;i.e. the quote seems to claim that for the assumption to hold, each of the original features must be normally distributed.)&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why the Gaussian assumption and why might &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;noreferrer&quot;&gt;PCA&lt;/a&gt; fail if the data are not Gaussian distributed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: to give more info, in the &lt;a href=&quot;https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf&quot; rel=&quot;noreferrer&quot;&gt;tutorial&lt;/a&gt;, page 12, the author gave an example of non-Gaussian distributed data that causes PCA to fail. &lt;/p&gt;&#xA;" OwnerUserId="40540" LastEditorUserId="29575" LastEditDate="2018-08-18T14:09:33.310" LastActivityDate="2021-04-06T19:08:43.267" Title="Why does PCA assume Gaussian Distribution?" Tags="|pca|gaussian|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25790" PostTypeId="2" ParentId="25789" CreationDate="2017-12-19T06:27:41.783" Score="4" Body="&lt;p&gt;Someone correct me if I'm wrong, but the PCA process itself doesn't assume anything about the distribution of your data.  The PCA algorithm is simple - &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;find the direction of greatest variance in your data&lt;/li&gt;&#xA;&lt;li&gt;write down the direction of the vector pointing in that direction, and 'divide' the data along that direction by its variance in that direction, so the resulting variance in that direction is 1.  This provides you with an eigenvector (direction) and associated eigenvalue (scale).&lt;/li&gt;&#xA;&lt;li&gt;repeat steps 1-2, potentially as many times as you have dimensions, but with the constraint that the next vector must be orthogonal (aka at a right angle) to all previous. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The result will be an ordered list of orthogonal vectors (eigenvectors), and scales (eigenvalues).  This set of vectors/values can be viewed as a summary of your data, particularly if all you care about is your data's variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there is an implicit assumption that the &lt;em&gt;orthogonality&lt;/em&gt; implies &lt;em&gt;independence&lt;/em&gt; of the resulting vectors, and from what I understand that's true if the data is Gaussian but not necessarily true in general.  So I suppose whether your data can be modeled as Gaussian may or may not matter, depending on your use case.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-12-19T06:27:41.783" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25791" PostTypeId="2" ParentId="25788" CreationDate="2017-12-19T06:36:28.093" Score="2" Body="&lt;p&gt;At the end of the day, as an applied data scientist you have a bag of tools you can use to solve business problems.  It's helpful to know what your tools are capable of, where each is useful, and also what their limits are.  If you don't have at least &lt;em&gt;some&lt;/em&gt; theory, knowing which method to use in which circumstance will seem like a meaningless list of details, and when things break, you may not know why - is there an error in my code, or am I making a fundamental error in applying an algorithm in a scenario where it has fundamental limitations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can be pragmatic in applying tools as you've seen them applied before.  Theory will help you have an intuition about whether it's possible or reasonable to apply given tools to new situations that you haven't seen.  It can help you answer questions like: how will this algorithm scale with the size or dimensionality of data?  SVMs in particular suffer from poor scaling with data size, precisely because they rely on kernels.  That kind of basic knowledge is probably more useful than being able to write down the kernel trick.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2017-12-19T06:36:28.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25793" PostTypeId="1" CreationDate="2017-12-19T07:42:15.533" Score="4" ViewCount="153" Body="&lt;p&gt;Requirement is to optimally move passengers from one seat map to another which has a different configuration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Move should be based on many rules like -&#xA;&lt;br&gt;1) Families should be sitting together&#xA;&lt;br&gt;2) Those who were at windows seat should be moved to window seat if available.&#xA;&lt;br&gt;3) Kids should be seated at middle seat.&#xA;&lt;br&gt;4) and many more..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many seat map configurations available and at a time we need to move passengers from one to another as per the requirement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a reinforcement learning problem ? If yes, how it can be approached.&lt;/p&gt;&#xA;" OwnerUserId="42726" LastEditorUserId="15828" LastEditDate="2017-12-19T12:09:44.913" LastActivityDate="2018-01-15T04:07:23.337" Title="Is reseating passengers a reinforcement learning problem?" Tags="|machine-learning|optimization|reinforcement-learning|methodology|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25794" PostTypeId="1" AcceptedAnswerId="25805" CreationDate="2017-12-19T08:06:12.827" Score="2" ViewCount="85" Body="&lt;p&gt;I was wondering if it is possible (and if yes, how is it done) to incorporate a luck component in statistical models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let’s assume I’d perform a regression on the goal difference between two teams in order to model the outcome of a sports match. How could I also model a random luck component that would make the underdog draw or win in certain rare cases?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And what if I took it a step further and tried to model the match outcome using neural networks? Is is possible to include luck in this case?&lt;/p&gt;&#xA;" OwnerUserId="43474" LastEditorUserId="29575" LastEditDate="2017-12-19T14:16:41.693" LastActivityDate="2017-12-19T14:19:51.487" Title="Incorporate luck in statistical modelling" Tags="|neural-network|r|deep-learning|regression|prediction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25795" PostTypeId="2" ParentId="25766" CreationDate="2017-12-19T08:56:48.490" Score="1" Body="&lt;p&gt;(Assuming this is a similar question to what was posted on Cross Validated, but was closed): You can merge the data from both matrices while adding a variable that specifies the origin (group), then plot them together in ggplot2:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library( ggplot2 )&#xA;&#xA;N1 = 19&#xA;N2 = 17&#xA;M  = 10&#xA;&#xA;m1 = matrix( rnorm(N1*M,mean=0,sd=1), N1, M)&#xA;m2 = matrix( rnorm(N2*M,mean=0,sd=1), N2, M)&#xA;&#xA;y     = c( as.vector( t( m1 ) ),  as.vector( t( m2 ) )  )&#xA;x     = c( rep(1:10, each = N1 ), rep(1:10, each = N2 ) )&#xA;group = c( rep( '1', N1 * M ),    rep( '2', N2 * M )    )&#xA;&#xA;df = data.frame( state = x, value = y, group = group )&#xA;&#xA;ggplot( df, aes( x = state, y = value, colour = group ) ) + &#xA;    geom_point() +&#xA;    ggtitle( &quot;State values in group 1 and 2&quot; ) +&#xA;    labs( x = &quot;State&quot;, y = &quot;Value&quot; ) +&#xA;    scale_x_continuous( breaks = seq(10) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the result:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/15gze.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/15gze.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16693" LastActivityDate="2017-12-19T08:56:48.490" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25796" PostTypeId="2" ParentId="25793" CreationDate="2017-12-19T09:18:59.713" Score="2" Body="&lt;p&gt;Yes. It can be. For reinforcement learning you need States, Actions and Rewards to run a reinforcement algorithm on a situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;States&lt;/strong&gt; are seats, &lt;strong&gt;Actions&lt;/strong&gt; are moving a person from a seat to another seat, and &lt;strong&gt;Rewards&lt;/strong&gt; are born from specified rules. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, moving a baby to a windows seat could have $-\infty$ reward (or you can prevent to do this action and exclude it from the possible action).&lt;/p&gt;&#xA;" OwnerUserId="27665" LastEditorUserId="40853" LastEditDate="2018-01-15T04:07:23.337" LastActivityDate="2018-01-15T04:07:23.337" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25797" PostTypeId="1" CreationDate="2017-12-19T09:37:38.240" Score="4" ViewCount="1219" Body="&lt;p&gt;I do experiments with the following Keras architecture with multiple outputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_model(conv_kernels = 32, dense_nodes = 512):&#xA;    model_input=Input(shape=(img_channels, img_rows, img_cols))&#xA;    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)&#xA;    x = Activation('relu')(x)&#xA;    x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)&#xA;    x = Activation('relu')(x)&#xA;    x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;    x = Dropout(0.25)(x)&#xA;    x = Flatten()(x)&#xA;&#xA;    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)&#xA;&#xA;    x1 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x2 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x3 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x4 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;&#xA;    lst = [x1, x2, x3, x4]&#xA;&#xA;    model = Model(inputs=model_input, outputs=lst)&#xA;    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)&#xA;    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])&#xA;&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I try to apply grid search this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = KerasClassifier(build_fn=create_model1, epochs=nb_epoch, batch_size=batch_size, verbose=0)&#xA;param_grid = dict(conv_kernels = [16, 32, 64], dense_nodes = [128, 256, 512])&#xA;grid = GridSearchCV(estimator=model, cv=4, param_grid=param_grid, n_jobs=-1)&#xA;grid_result = grid.fit(X_train, Y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following error message:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Found input variables with inconsistent numbers of samples: [9416, 4]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;9416 is the number of training examples, 4 is the model output number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the problem here? Grid search is unavailable for multiple outputs? If so, what is the best way to apply parameter search (apart from pure manual method)?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="29575" LastEditDate="2017-12-19T14:08:38.323" LastActivityDate="2020-05-22T23:02:05.253" Title="Grid seach is unavailable for Keras in case of multiple outputs?" Tags="|machine-learning|keras|grid-search|hyperparameter-tuning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25798" PostTypeId="2" ParentId="25772" CreationDate="2017-12-19T09:49:25.470" Score="1" Body="&lt;p&gt;I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to avoid the problem you mentioned, you have to find the most similar vector to a vector in &lt;code&gt;test&lt;/code&gt; considering  only vectors in &lt;code&gt;train&lt;/code&gt;. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = [v1, v2, v3]&#xA;test = [v4, v5]&#xA;most_similar = {}&#xA;for vector in test:&#xA;most_similar[vector] = v1&#xA;  for vector2 in train:&#xA;   if similarity(vector2, vector) &amp;gt; similarity(most_similar[vector], vector):&#xA;     most_similar[vector] = vector2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the end in &lt;code&gt;most_similar&lt;/code&gt; you have the most similar vector in train for each vector in test.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-19T09:49:25.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25799" PostTypeId="2" ParentId="17855" CreationDate="2017-12-19T12:38:57.370" Score="1" Body="&lt;p&gt;The point is to learn &lt;em&gt;useful&lt;/em&gt; variations of data instead of just splitting by large categorial variable. Each new row after encoding becomes immediately related with the output, while original categorial variable may be related only in indirect, latent manner. Plus, the interactions between output and the original variable are represented too, by definition. Think of it as of adding this interaction explicitly to add more intuitive justification to this method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I think this method will be perfect not only for divide-and-conquer RF-style methods but also for plain LR as well.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if the encoding for test set is always 0.5, then what is the point of this encoding?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's okay, as soon as you know how this should be related to the output. &#xA;0.5 in this example is just a rough approximation of input, but given that you have learned model even this approximation is still meaningful, as it instructs model -- which path to output to choose. If your model of choice is RF then 0.5 will land at about same set of leaves as more granular distribution ( if you were able to know it )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Meanwhile much stronger approximations are possible, for instance, one can use KDE to estimate response variable distribution and then draw samples from it at test time. Adding uniform random noise is just a hint in this direction. Maybe you want to follow it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing only particular level of categorial variable is way less informative, than this 0.5, isn't it? &lt;/p&gt;&#xA;" OwnerUserId="13787" LastActivityDate="2017-12-19T12:38:57.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25800" PostTypeId="2" ParentId="14755" CreationDate="2017-12-19T13:13:26.123" Score="1" Body="&lt;p&gt;I see many facets of your question and in what follows will present my top 2 :-)&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Facet 1 -- assessing the uncertainty in estimated coefficients&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In logistic regression, assessing the uncertainty in the estimated coefficients is virtually the same as for least-squares regression .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both logistic regression and least-squares regression, the regression coefficient table will include a column for the regression coefficients followed by a column of standard errors, then by a column of test statistics, and finally a column of p-values. The table below shows the example coefficient table output for the some regression problem (where the   probability of buying some magazine for kids is estimated).&#xA;&lt;a href=&quot;https://i.stack.imgur.com/y6Q65.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y6Q65.png&quot; alt=&quot;example problem coefficients table&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the test statistics are labeled “z value” and the p-values are labeled “P(&gt;|t|)” in the table above.&#xA;The standard errors can be used to construct confidence intervals for the regression coefficients. Roughly speaking, going plus or minus 2 times the standard error from the regression coefficient gives approximately a 95% confidence interval for the coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for Residence Length, the regression coefficient is 0.024680. The next column gives the standard error of the regression coefficient which is 0.013800. Thus an approximate 95% confidence interval for the Residence Length regression coefficient is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;0.024680 \pm 2 \times 0.013800 = 0.024680 \pm 0.0276\&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that the regression coefficient for Residence Length could be anywhere from -0.00292 to 0.05228 (with 95% confidence).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As is well known, we often use the odds-ratio, which is the exponential of the regression coefficient (i.e., $\exp(\beta)$ ), to help to interpret the meaning of the regression coefficient. The odds-ratio for the Residence Length coefficient, as shown in the coefficient table, is 1.0250. This means that there is a 2.5% increase in the odds of buying the magazine associated with each additional year of residence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can also compute the odds-ratios corresponding to the ends of the confidence interval. These odds-ratios will give us an equivalent confidence intervals for the odds. So continuing the example using Residence Length, the odds ratios corresponding to the ends of the confidence interval are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA; \exp(-0.00292)=0.99708&#xA;$$&#xA; and&#xA;$$&#xA; \exp(0.05228)=1.05367.&#xA;$$&#xA;Thus, the interval [0.99708,1.05367] is an approximate 95% confidence interval for the odds ratio. This means that there could be anywhere from a 0.292% decrease to a 5.367% increase in the odds of buying the magazine associated with each additional year of residence.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Facet 2 -- modeling data with measurements uncertainty&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here again many options are known. Take a look at this to get started: &lt;a href=&quot;https://pdfs.semanticscholar.org/0a7a/a5e3d407b24e2f5c24287bfdda20e573bd05.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pdfs.semanticscholar.org/0a7a/a5e3d407b24e2f5c24287bfdda20e573bd05.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will just cite the source:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;in some datasets, the outcome of interest is measured with imperfect&#xA;  sensitivity and specificity. It is well known that the&#xA;  misclassification induced by such an imperfect diagnostic test will&#xA;  lead to biased estimates of the odds ratios and their variances.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In this paper, the authors show that when the sensitivity and specificity&#xA;  of a diagnostic test are known, it is straightforward to incorporate&#xA;  this information into the fitting of logistic regression models.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;An EM algorithm that produces unbiased estimates of the odds ratios and&#xA;  their variances is described. The resulting odds ratio estimates tend&#xA;  to be farther from the null but have greater variance than estimates&#xA;  found by ignoring the imperfections of the test. The method can be&#xA;  extended to the situation where the sensitivity and specificity differ&#xA;  for different study subjects, i.e., nondifferential misclassification.&#xA;  The method is useful even when the sensitivity and specificity are not&#xA;  known, as a way to see the degree to which various assumptions about&#xA;  sensitivity and specificity affect one's estimates. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="13787" LastActivityDate="2017-12-19T13:13:26.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25801" PostTypeId="2" ParentId="24860" CreationDate="2017-12-19T13:45:25.883" Score="2" Body="&lt;p&gt;I would suggest you to split your problem into two:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to train?&lt;/li&gt;&#xA;&lt;li&gt;How to make inference when resources are limited?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Very common pattern is to train model on large data (by renting AWS server, for instance) and then use its predictions to train much smaller network.&#xA;&lt;a href=&quot;https://arxiv.org/abs/1503.02531&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is fundamental paper on the subject from Hinton et al.&lt;/p&gt;&#xA;&lt;p&gt;This way you can stick to more conventional object detection method without stacking involved and then deploy 'squeezed' network to your device.&lt;/p&gt;&#xA;&lt;p&gt;But back to your question. If you are rigid in your approach for various reasons and can't do like I described above, then lots of end-to-end approaches are available in literature, take a look at &lt;a href=&quot;http://www.svcl.ucsd.edu/publications/conference/2016/mscnn/mscnn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; to get started.&lt;/p&gt;&#xA;&lt;p&gt;And even &lt;a href=&quot;https://github.com/zhaoweicai/mscnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I will just cite the paper:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;A unified deep neural network, denoted the multi-scale CNN (MS-CNN),&#xA;is proposed for fast multi-scale object detection. The MS- CNN&#xA;consists of a proposal sub-network and a detection sub-network.&lt;/p&gt;&#xA;&lt;p&gt;In the proposal sub-network, detection is performed at multiple output&#xA;layers, so that receptive fields match objects of different scales.&#xA;These complementary scale-specific detectors are combined to produce a&#xA;strong multi-scale object detector. The unified network is learned&#xA;end-to-end, by optimizing a multi-task loss.&lt;/p&gt;&#xA;&lt;p&gt;Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs.&#xA;State-of-the-art object&#xA;detection performance, at up to 15 fps, is reported on datasets, such&#xA;as KITTI and Caltech, containing a substantial number of small&#xA;objects.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;But nevertheless feel free to experiment with network compression (take a look at &lt;a href=&quot;https://arxiv.org/pdf/1709.04344.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;). It typically has high degree of redundacy, and by being creative in exploring this redundacy it's possible to have many gains&lt;/p&gt;&#xA;" OwnerUserId="13787" LastEditorUserId="98307" LastEditDate="2020-08-15T09:12:25.707" LastActivityDate="2020-08-15T09:12:25.707" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="25802" PostTypeId="1" CreationDate="2017-12-19T14:00:12.053" Score="1" ViewCount="222" Body="&lt;p&gt;I am trying to apply Probabilistic Neural Network (PNN) on MNIST dataset using MATLAB on 28000 samples. I tried to build the model but it gave an error saying matrix dimension exceeds memory showing it will require 5.08 GB (28000x28000) matrix.&lt;/p&gt;&#xA;&lt;p&gt;I want to know should I decrease the number of samples as input for training from 28000 to 200 samples as I read in &lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-540-37258-5_41&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Should I reduce the sample size or is there something else wrong?&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;is pnn only applicable on small data sets?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-19T14:29:39.910" Title="Classifying hand written digits dataset using Probabilistic neural network (PNN)?" Tags="|machine-learning|image-classification|matlab|mnist|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25803" PostTypeId="1" AcceptedAnswerId="25808" CreationDate="2017-12-19T14:01:28.170" Score="0" ViewCount="485" Body="&lt;p&gt;I have a Support Vector Machine in Scikit-learn (Python) that gets trained once in a while when enough new data has accumulated (user help train the model by submitting new data). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I store the model in .pkl format for persistence. However, the SVM needs scaled data and I'm wondering what would be a good way to persist the scaler, since it was fitted on the train data that changes over time (as users submit more data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a common solution for this? Or is there a value that I can store with the model that works as some sort of seed to recreate the scaler?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT (added implementation):&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a database with a dataset. users periodically add data points to this dataset. I also have an estimator database which keeps track of all the trained models and their .pkl file locations. When enough new data points are added the system will start to train a new estimator. It extracts all the features and scales the data, then trains a model. After that it compare the existing best model with the newly trained one and if the new one is better, it sets the new one to 'active'. There is only one active model a a time and that is the one that will be used for predictions on new data. (I hope this helps)&lt;/p&gt;&#xA;" OwnerUserId="43508" LastEditorUserId="43508" LastEditDate="2017-12-20T09:39:00.020" LastActivityDate="2017-12-20T09:39:00.020" Title="How to persist data scaler for predictions" Tags="|prediction|feature-scaling|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25804" PostTypeId="2" ParentId="22365" CreationDate="2017-12-19T14:07:09.323" Score="3" Body="&lt;p&gt;I would investigate a &lt;a href=&quot;https://en.wikipedia.org/wiki/Fuzzy_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;soft clustering&lt;/a&gt; approach using &lt;a href=&quot;http://hdbscan.readthedocs.io/en/latest/soft_clustering_explanation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;HDBSCAN clustering&lt;/a&gt;. &#xA;Some reading on text clustering &lt;a href=&quot;https://recast.ai/blog/introduction-text-clustering/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, which includes discussion on clustering with DBSCAN, HDBSCAN's non-hierarchical precursor &lt;/p&gt;&#xA;" OwnerUserId="43276" LastActivityDate="2017-12-19T14:07:09.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25805" PostTypeId="2" ParentId="25794" CreationDate="2017-12-19T14:19:51.487" Score="2" Body="&lt;p&gt;Use a probabilistic model. For example, if the probability of one team beating the other is $σ(w⋅(team1−team2))$, where $σ(⋅)$ is the logistic function, then there's always a probability that the weaker team will win. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to use a neural network you'd use it to replace the logistic function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;&lt;sub&gt;Answer copied from comments.&lt;/sub&gt;&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-12-19T14:19:51.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25806" PostTypeId="2" ParentId="25797" CreationDate="2017-12-19T14:20:57.267" Score="1" Body="&lt;p&gt;It does not appear possible  &lt;a href=&quot;https://github.com/keras-team/keras/issues/6451&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;as stated here&quot;&gt;as stated here&lt;/a&gt;.  The problem is basically how sklearn expects it's input.  Currently it only accepts a 2d array.  I think it'll be rather straight forward to go to sklearn wrapper and modify is so that it can accept multiple arrays.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-19T14:20:57.267" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25807" PostTypeId="2" ParentId="25802" CreationDate="2017-12-19T14:29:39.910" Score="1" Body="&lt;p&gt;Reducing the sample size is pretty much up to you.  PNN have the drawback that the size of the network after training is equal to the size of the training dataset.  Therefore it's a bit impractical to use for large datasets.  I also believe that there is evidence that suggest the performance may also decrease when the layers are big.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have access to the paper but I imagine a big reason why they reduced the sample was to avoid the memory error that you're encountering right now.  I also suspect they carefully selected a good representation for the sample.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-19T14:29:39.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25808" PostTypeId="2" ParentId="25803" CreationDate="2017-12-19T15:01:23.127" Score="1" Body="&lt;p&gt;So after looking some more I found this question: &lt;a href=&quot;https://stackoverflow.com/questions/41993565/save-scaler-model-in-sklearn&quot;&gt;save scaler model in sklearn&lt;/a&gt;&#xA;which says that one can picle the scaler aswell, which works.&lt;/p&gt;&#xA;" OwnerUserId="43508" LastActivityDate="2017-12-19T15:01:23.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25809" PostTypeId="1" CreationDate="2017-12-19T15:06:26.227" Score="2" ViewCount="217" Body="&lt;p&gt;I want to classify emails as Spam and Non-Spam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a labelled dataset of 20,000 emails in TXT format. The emails are in individual files and also in one combined file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example email looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;From: &quot;Sender Display Name&quot; &amp;lt;sender@abc.com&amp;gt;&#xA;&#xA;To: systudent &amp;lt;systudent@abc.com&amp;gt;, tystudent &amp;lt;tystudent@abc.com&amp;gt;, &#xA;btechstudent &amp;lt;btechstudent@abc.com&amp;gt;, mtech16 &amp;lt;mtech16@abc.com&amp;gt;, mtech17 &amp;lt;mtech17@abc.com&amp;gt;&#xA;&#xA;Subject: Register to the event&#xA;&#xA;Date: Tue, 21 Nov 2017 14:16:17 +0000&#xA;&#xA;X-Originating-IP: [13.90.24.116]&#xA;&#xA;&#xA;Body:&#xA;&#xA;Some spam text&#xA;&#xA;&amp;lt;https://somelink.com/abc&amp;gt;&#xA;&#xA;&#xA;EOM&#xA;Label: Spam&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The features that I want to use are: Sender Display Name, Sender Email address, Receivers, Subject, Date, IP, URL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I convert these to input feature vectors or how can I give these as input which are currently in TXT format, to Machine Learning Algorithms like Random Forest, Naive Bayes, etc. ?&lt;/p&gt;&#xA;" OwnerUserId="43511" LastEditorUserId="43511" LastEditDate="2017-12-19T17:11:59.087" LastActivityDate="2017-12-19T17:11:59.087" Title="Giving Emails as Input to Machine Learning Algorithms" Tags="|machine-learning|classification|feature-selection|feature-extraction|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25810" PostTypeId="2" ParentId="25793" CreationDate="2017-12-19T15:06:56.200" Score="7" Body="&lt;p&gt;Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using &lt;a href=&quot;https://en.wikipedia.org/wiki/Global_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Global Optimization&lt;/a&gt; would be a more direct approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially you want to design a cost function that describes how good a particular seating is and then use it to search the space of possible seatings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example to solve the problem with &lt;a href=&quot;https://en.wikipedia.org/wiki/Simulated_annealing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Simulated Annealing&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Design a cost function $e(s)$ that measures how good a seating arrangement is. Lower cost means better seating.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Design an acceptance probability function $P(e, e', T)$ that takes the costs of seating arrangements $s$ and $s'$, and a temperature $T$, and returns a probability $P$ with the properties (a) $P&amp;gt;0$ even if $s'$ is worse than $s$ (b) the better $s'$ is relative to $s$, the higher $P$ is and (b) the lower the temperature, $T$, the close $P$ is to $0$ when $s'$ is worse than $s$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Set some large initial $T$ and design an annealing schedule for decreasing $T$ with the number of iterations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Start with some seating arrangement $S$. This can be random or chosen according to a &lt;a href=&quot;https://en.wikipedia.org/wiki/Greedy_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;greedy&lt;/a&gt; strategy that tries to satisfy as many constraints as possible.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Repeat for $i$ in $k$ steps (for some sufficiently large $k$):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Consider a neighboring state $S'$ that can be reached by randomly swapping two seated people or moving someone to an empty seat. Set $S$ to $S'$ with probability $P(S, S', T)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decrease $T$ according to the annealing schedule at step $i$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The simulation starting at (4) can be run once or many times saving only the best result. At the end you should arrive at close to or exactly the best possible seating arrangement.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-19T18:43:07.697" LastActivityDate="2017-12-19T18:43:07.697" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25811" PostTypeId="1" CreationDate="2017-12-19T15:06:57.890" Score="2" ViewCount="2434" Body="&lt;p&gt;I have a question related to evaluating out-of-sample predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S095741740400096X&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.sciencedirect.com/science/article/pii/S095741740400096X&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?&lt;/p&gt;&#xA;" OwnerUserId="43513" LastActivityDate="2017-12-19T17:42:12.810" Title="Splitting hold-out sample and training sample only once?" Tags="|machine-learning|prediction|model-evaluations|grid-search|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25812" PostTypeId="2" ParentId="25811" CreationDate="2017-12-19T16:29:08.073" Score="2" Body="&lt;p&gt;In general, it's a good idea to split up your data into three sets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training Set (60-80% of your data)&lt;/li&gt;&#xA;&lt;li&gt;Cross-Validation Set (10-20% of your data)&lt;/li&gt;&#xA;&lt;li&gt;Test Set (10-20% of your data)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is usually an overly optimistic estimate of your model's generalization error. This is because you are essentially fitting your model to another parameter $d$, where $d$ is your test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to address this problem, you can train a model on your training set and then choose the model that performs best on your cross-validation set. Now, this model is no longer fit to the test set and you can safely estimate the generalization error on the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you are already following this approach in selecting the model by choosing which one performs best on a 10-fold cross-validation on your training set. In this case, the error on your test set should be a good estimate of generalization error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also mentioned that randomness might affect your training due to outliers in your training data:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In machine learning, it is often assumed that our data is iid (independent and identically distributed) from some unknown distribution. On average, we should get data points that are representative of this distribution when we split up our data. A common practice to better ensure this is to randomly shuffle the entire dataset before splitting up your data.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastEditorUserId="43215" LastEditDate="2017-12-19T17:42:12.810" LastActivityDate="2017-12-19T17:42:12.810" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25813" PostTypeId="2" ParentId="22441" CreationDate="2017-12-19T16:35:21.510" Score="1" Body="&lt;p&gt;t-SNE is not really designed that way.  Since t-SNE is non-parametric there isn't a function that maps data from an input space to the map.  The standard approach usually is to train a multivariate regression to predict the map location from input data.  You can read more about this in this paper &lt;a href=&quot;https://lvdmaaten.github.io/publications/papers/AISTATS_2009.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt;.  In the paper you should note that the author takes the approach to minimize the t-SNE loss directly.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-19T16:35:21.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25814" PostTypeId="1" AcceptedAnswerId="25815" CreationDate="2017-12-19T16:43:35.087" Score="12" ViewCount="19708" Body="&lt;p&gt;so given that the sigmoid function is defined as hθ(x) = g(θ^(T)x), how can I implement this funcion in Octave given that g = zeros(size(z)) ?&lt;/p&gt;&#xA;" OwnerUserId="43517" LastActivityDate="2017-12-19T17:00:44.990" Title="How do I implement the sigmoid function in Octave?" Tags="|regression|logistic-regression|octave|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-24T12:27:16.310" ContentLicense="CC BY-SA 3.0" />
  <row Id="25815" PostTypeId="2" ParentId="25814" CreationDate="2017-12-19T17:00:44.990" Score="11" Body="&lt;p&gt;This will compute the sigmoid of a scalar, vector or matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function g = sigmoid(z)&#xA;%   SIGMOID Compute sigmoid function&#xA;%   g = SIGMOID(z) computes the sigmoid of z.&#xA;&#xA;&#xA;% Compute the sigmoid of each value of z (z can be a matrix,&#xA;% vector or scalar).&#xA;&#xA;SIGMOID = @(z) 1./(1 + exp(-z));&#xA;&#xA;g = SIGMOID(z);&#xA;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-19T17:00:44.990" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25816" PostTypeId="2" ParentId="25752" CreationDate="2017-12-19T18:00:12.877" Score="3" Body="&lt;p&gt;Why are you using binary_crossentropy?  You should be using categorical_crossentropy.  However, if you insist on using binary_crossentropy change your metric to metrics=['binary_accuracy', 'categorical_accuracy'] (this will display both accuracies).  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-19T18:00:12.877" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25817" PostTypeId="1" CreationDate="2017-12-19T19:06:57.233" Score="2" ViewCount="205" Body="&lt;p&gt;I want to recognize these geometric shapes which are related to each other. For instance, looking at the image of the roof below, just by knowing the existence of the ridges in RED, I know that the ridge in BLUE should also exist (even if it is not visible in the image). If I have thousands of such labelled images, a ML model should be able to learn this as well. However, I can't figure out how to represent this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/a2iIi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/a2iIi.png&quot; alt=&quot;Original Image (Example 1)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qR49l.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qR49l.png&quot; alt=&quot;Label over original Image (Example 1)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Label(s) : C, Z&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/izxhq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/izxhq.png&quot; alt=&quot;Original Image (Example 2)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aiKK7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aiKK7.png&quot; alt=&quot;Label over original Image (Example 2)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Label(s) : D&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/azQIb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/azQIb.png&quot; alt=&quot;Original Image (Example 3)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rHePP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rHePP.png&quot; alt=&quot;Label over original Image (Example 3)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Label(s) : C, Z&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fVbQz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fVbQz.png&quot; alt=&quot;Original Image (Example 4)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QwnSm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QwnSm.png&quot; alt=&quot;Label over original Image (Example 4)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Label(s) : E, G&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's call these ridges as lines, as in the 1st example, we have X and Y lines being detected by simple edge detection but not Z because it's not visible. The same way line D is not detected but lines A, B, C are, from example 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want is that I formulate a ML model that learns from the X and Y that there should be a Z and subsequently D from A, B, C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set of such examples where ridges are labeled (the red and blue is just to distinguish, all ridges are labeled with same color). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some important things to keep in mind.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The brightness of the image could vary a lot.&lt;/li&gt;&#xA;&lt;li&gt;The ridge could have any scale or even orientation (within reasonable limits).&lt;/li&gt;&#xA;&lt;li&gt;The input image is almost always very noisy.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33040" LastActivityDate="2017-12-19T19:06:57.233" Title="Recognition of Geometric shapes for ridges" Tags="|machine-learning|deep-learning|computer-vision|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25818" PostTypeId="1" AcceptedAnswerId="25820" CreationDate="2017-12-19T19:22:10.020" Score="0" ViewCount="70" Body="&lt;p&gt;I don't have a maths / stats / data science background and need to evaluate which of the two evaluations below (numerical regression on Amazon Machine Learning) predict more accuracy. Both models use the same data set but it's looking at different time frames both on the independent and dependent variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I evaluate which one of the two models is more accurate? And is there a way to tell how accurate these two models are in general (e.g. 75%)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SFW6N.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SFW6N.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EL4Vp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EL4Vp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GCLZR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GCLZR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Vdu2R.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Vdu2R.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43502" LastActivityDate="2017-12-19T19:35:38.003" Title="Layman's comparison of RMSE" Tags="|regression|amazon-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25819" PostTypeId="2" ParentId="12905" CreationDate="2017-12-19T19:23:43.807" Score="1" Body="&lt;p&gt;The better question might be - Why you do want to mix deep neural networks (DNN) and a finite state machine (FSM)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is &lt;a href=&quot;https://papers.nips.cc/paper/757-fools-gold-extracting-finite-state-machines-from-recurrent-network-dynamics.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;research&lt;/a&gt; showing that a DNN can simulate any FSM. Since there are more frameworks for DNN and DNN can perform more tasks than FSM, it appears to be more useful just to forgo FSM altogether. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, &quot;&lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S1877050917323724&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural network for synthesizing deterministic finite automata&lt;/a&gt;&quot; shows how a relatively simple neural network (NN) can quickly and automatically learn the correct deterministic finite automaton (DFA).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a strong trend towards end-to-end DNN.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-01-21T17:51:45.623" LastActivityDate="2018-01-21T17:51:45.623" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25820" PostTypeId="2" ParentId="25818" CreationDate="2017-12-19T19:35:38.003" Score="1" Body="&lt;p&gt;Typically you want a smaller RMSE and without getting into detail it should be sufficient to just take the smaller one.  However, I am concerned because you state that the models were ran on the same dataset but at different timeframes.  Since RMSE scale depends on the dependent variable scale, it's entirely possible that these two timeframes are scaled different.  A somewhat contrive example would be energy consumption.  I would expect a model trained on daytime consumption to have a higher RMSE than for one trained between 1am and 3 am.  In that case, comparing the RMSE may be meaningless.  You can try to normalize your data and RMSE to help with this, but i'm unsure if AWS provides this ability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for your second questions, you really won't get a 75% accurate number for regression.  You can look at the deviations of the residual or do cross valiadtion and see how well the model performs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again this may not be possible in AWS.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit:  I juse realized that the histograms were residual plots.  Do three things  Increase bin size.  Check to see if the residuals are centered around 0 and then check if there is skewness in the data.  If the data is centered around 0 and symmetric then you can say the model error is basically random and does not favor over or under predicting.  If the data is not centered around 0 and there is skewness, then the errors can be systematic and then in that case considering adding more variables.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-19T19:35:38.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25821" PostTypeId="1" AcceptedAnswerId="25824" CreationDate="2017-12-19T20:36:04.713" Score="4" ViewCount="59018" Body="&lt;p&gt;I was trying to write a python code that can set some neural network channels or neurons to zero at the inference; and I wrote the code below. The code generates 10 different arrays for different percentage of the channels or neurons that are set to zero. My challenge is how to combine these arrays into a single array or a list so that they can be individually accessed, but all I was getting was a list containing arrays with zero elements. Please help me.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;def three_steps(n):&#xA;    return step(n, steps=[1, 0.5, 0.25])&#xA;&#xA;B = np.linspace(0.1, 1.0, 10)&#xA;print B   &#xA;B = np.array(three_steps(100))&#xA;print &quot;A = &quot;, A&#xA;&#xA;for i, value in enumerate(B):&#xA;    A[i:int(value*100)] = 0&#xA;    print A&#xA;list = [np.append(A, i) for k in range(i + 1)]&#xA;print list&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43491" LastEditorUserId="15412" LastEditDate="2017-12-21T05:36:15.910" LastActivityDate="2017-12-21T05:36:15.910" Title="How to create an array from the list of arrays in python" Tags="|python|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-25T19:29:50.810" ContentLicense="CC BY-SA 3.0" />
  <row Id="25822" PostTypeId="2" ParentId="16769" CreationDate="2017-12-19T20:50:12.013" Score="3" Body="&lt;p&gt;MLXtend library has been really useful for me. In its docummentation there is an Apriori implementation that outputs the frequent itemset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please check the first example available in  &lt;a href=&quot;http://rasbt.github.io/mlxtend/user_guide/frequent_patterns/apriori/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://rasbt.github.io/mlxtend/user_guide/frequent_patterns/apriori/&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="43525" LastActivityDate="2017-12-19T20:50:12.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25823" PostTypeId="1" CreationDate="2017-12-19T21:03:09.797" Score="2" ViewCount="143" Body="&lt;p&gt;Given text data, I'd like to extract numerical information based on the presence of other information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, &quot;There are 119 people in the gym right now. 100 people in the weight room, 19 on treadmills&quot;. If I was interested in &quot;weight room&quot; then I would extract &quot;100&quot; or if I was interested in &quot;treadmills&quot; then I would extract &quot;19&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can create dependency trees with Python's spaCy package. Is there a way to extract these dependencies? &lt;/p&gt;&#xA;" OwnerUserId="9618" LastEditorUserId="38887" LastEditDate="2020-05-08T10:43:12.317" LastActivityDate="2020-05-08T10:43:12.317" Title="How can I extract numerical information conditional on other information?" Tags="|python|nlp|spacy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25824" PostTypeId="2" ParentId="25821" CreationDate="2017-12-19T21:04:41.737" Score="6" Body="&lt;p&gt;This &lt;a href=&quot;https://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python&quot;&gt;post&lt;/a&gt; from stack overflow should give you what you want. The magic code boils down to the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;flat_list = [item for sublist in l for item in sublist]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also loop through the list of lists and merge them.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lists = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]&#xA;&#xA;merged_list = []&#xA;&#xA;for l in lists:&#xA;    merged_list += l&#xA;&#xA;print(merged_list)&#xA;[1, 2, 3, 4, 5, 6, 7, 8, 9]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-19T21:04:41.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25826" PostTypeId="2" ParentId="25786" CreationDate="2017-12-19T21:27:10.180" Score="1" Body="&lt;p&gt;The AlphaGo Zero paper tests a few different architectures:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Seperate 12-convolutional-block policy and value networks (as in AlphaGo)&lt;/li&gt;&#xA;&lt;li&gt;A single 12-convolutional-block network with dual policy and value outputs&lt;/li&gt;&#xA;&lt;li&gt;Separate 20-residual-block networks&lt;/li&gt;&#xA;&lt;li&gt;A single 20-residual-block network with dual policy and value outputs&lt;/li&gt;&#xA;&lt;li&gt;A single 40-residual-block network with dual policy and value outputs&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://arxiv.org/abs/1712.01815&quot; rel=&quot;nofollow noreferrer&quot;&gt;AlphaZero&lt;/a&gt; paper AlphaZero Go wins 60 games in a 100-game match against the 20-block single resnet AlphaGo Zero, but it is not explicitly stated which architecture AlphaZero is using. It is either the 40-block single resnet since this was the best, or it is the 20-block single resnet in order to make a fair comparison with AlphaGo Zero.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-20T14:13:04.537" LastActivityDate="2017-12-20T14:13:04.537" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25827" PostTypeId="2" ParentId="25655" CreationDate="2017-12-19T21:46:21.013" Score="3" Body="&lt;p&gt;Keep in mind that the number of executors is independent of the number of partitions of your dataframe. You set the number of executors when creating &lt;em&gt;SparkConf()&lt;/em&gt; object.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://spark.apache.org/docs/latest/submitting-applications.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://spark.apache.org/docs/latest/submitting-applications.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To retrieve the number of executors from code, it depends of the API you use. For python, this is not currently implemented. For Scala, you can. Here is a good stack post : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/38660907/how-to-get-the-number-of-workersexecutors-in-pyspark&quot;&gt;https://stackoverflow.com/questions/38660907/how-to-get-the-number-of-workersexecutors-in-pyspark&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of partitions refers to the number of parts your dataframe is composed of. With pyspark API, you can get it by transform your dataframe to a rdd and then use method &lt;em&gt;.getNumPartitions()&lt;/em&gt; :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_partitions = df.rdd.getNumPartitions()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-19T21:46:21.013" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25829" PostTypeId="1" CreationDate="2017-12-20T00:36:09.197" Score="4" ViewCount="918" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;The docs on sklearn.multioutput.MultiOutputRegressor&lt;/a&gt; state that it implements a strategy for extending regressors that do not natively support multi-target regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested to know: which ones do natively support multi-target regression ? (and/or how can I find out?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;based on &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/plot_multioutput_face_completion.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;, I assume:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ExtraTreesRegressor &lt;br&gt;&#xA;KNeighborsRegressor &lt;br&gt;&#xA;LinearRegression &lt;br&gt;&#xA;RidgeCV &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All support this? (unless I'm getting confused and a 'multi-output estimator'  is not the same thing as 'a regressor that natively supports multi-target regression'; I'm new to this!)&lt;/p&gt;&#xA;" OwnerUserId="43531" LastActivityDate="2021-07-13T10:06:50.707" Title="Scikit learn: which regressors natively support multi-target regression?" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25831" PostTypeId="1" AcceptedAnswerId="25847" CreationDate="2017-12-20T03:35:13.573" Score="0" ViewCount="358" Body="&lt;p&gt;I'm trying to understand feature hashing, specifically in the context of document classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm under the impression that it is useful because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it allows us to easily deal with 'new' words/features/predictors that we haven't seen before &lt;/li&gt;&#xA;&lt;li&gt;it is rather efficient because it allows us to exploit sparsity which is common in document/word data.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Say my word/feature set consists of 50,000 unique words and I use 16 hash bits $(2^{16}=65,536)$. In this setting, is it likely each word gets&#xA;mapped to one of the 65,536 indices?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How does this aid us in the situation where I build a model and try to make a prediction on a document that contains a word we haven't seen before?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Say my feature set is 100,000 words and I still use 16 hash bits. Does this effectively mean I have (on average?) $100,000-65,536 = 34,464$ words mapped randomly on to other words? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It seems I'm missing crucial point or idea here. Can anyone recommend some good introductory material?&lt;/p&gt;&#xA;" OwnerUserId="37475" LastActivityDate="2017-12-20T14:06:24.433" Title="Some questions about feature hashing in the context of document classification" Tags="|nlp|dimensionality-reduction|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25832" PostTypeId="1" AcceptedAnswerId="25851" CreationDate="2017-12-20T03:39:45.393" Score="14" ViewCount="15096" Body="&lt;p&gt;Let's assume a vanilla MLP for classification with a given activation function for hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid is the activation function and -0.5 and 0.5 if tanh is the activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about ReLu ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I normalise the network input between 0 and 1, -0.5 and 0.5, or -1 and 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any known best practices there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not talking about normalisation of the input of the ReLu like using Batch Normalisation just before or just after the ReLu : &lt;a href=&quot;https://arxiv.org/pdf/1508.00330&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1508.00330&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I am talking about normalising the input of the whole network.&lt;/p&gt;&#xA;" OwnerUserId="43532" LastEditorUserId="28175" LastEditDate="2017-12-20T20:05:05.743" LastActivityDate="2017-12-20T20:05:05.743" Title="Input normalization for ReLu?" Tags="|machine-learning|neural-network|deep-learning|activation-function|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25833" PostTypeId="2" ParentId="25621" CreationDate="2017-12-20T04:44:15.800" Score="2" Body="&lt;p&gt;I had a very similar issue when executing predict/predict_classes;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  ...&#xA;    classes = model.predict(XpredictInputData) &#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/models.py&quot;, line 1006, in predict&#xA;    return self.model.predict(x, batch_size=batch_size, verbose=verbose)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 1772, in predict&#xA;    check_batch_axis=False)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&quot;, line 153, in _standardize_input_data&#xA;    str(array.shape))&#xA;ValueError: Error when checking : expected dense_1_input to have shape (None, 100) but got array with shape (100, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem was occurring because the input array to keras predict was 1 dimensional (caused by a combination of a) numpy.genfromtxt producing a 1d array from a 1 row data file and b) a broadcasting failure). Keras predict appears to require a two dimensional array, even if there is only 1 prediction to be made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I solved the issue by adding the following check before executing predict;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if (XpredictInputData.ndim == 1):&#xA;    XpredictInputData = numpy.array([XpredictInputData])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43534" LastActivityDate="2017-12-20T04:44:15.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25834" PostTypeId="1" AcceptedAnswerId="25843" CreationDate="2017-12-20T04:44:23.207" Score="1" ViewCount="388" Body="&lt;p&gt;I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with two columns, &quot;Titles&quot; and &quot;Description&quot;. Titles column has names of clinical lab tests and description column has description about results of corresponding laboratory test( can be seen below). There many titles specific to a particular lab test.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Title                       Description&#xA;Complete blood test         RBC: Normocytic and Normochromic&#xA;COMPLETE Blood test\        Platelets: Adequate on the smear&#xA;Blood glucose               COLOUR - COLOURLESS&#xA;Complete blood picture      WBC: Total and Differential counts are within normal limits&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have only shared a small part of the data frame.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem 1: I have manually grouped the similar looking titles. For example, I have grouped the titles in the above data frame as &quot;Blood Test&quot;. Is there a way to use NLP technique to group similar looking titles ( As shown below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem 2: Based on the description, i have manually labeled a particular outcome for a lab test as normal or abnormal. Again i am looking for a way to do this without having to manually label the outcome (As shown below).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Title                       Description                           Outcome&#xA;Blood Test                   RBC: Normocytic and Normochromic       Normal&#xA;Blood Test                   Platelets: Adequate on the smear       Normal&#xA;Blood Test                   COLOUR - COLOURLESS                    Normal&#xA;Blood Test                   WBC: Total and Differential counts     Normal&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any suggestions or resources to help me get started would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="43458" LastEditorUserId="43458" LastEditDate="2017-12-20T05:19:00.737" LastActivityDate="2017-12-21T08:10:47.010" Title="Text classification problem using Python or R" Tags="|python|r|nlp|text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25835" PostTypeId="2" ParentId="19030" CreationDate="2017-12-20T07:31:30.220" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Test on train data&lt;/strong&gt; uses the whole data set for training and then for testing. This method practically always gives wrong results.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, yes, in this case, the model is learned and tested on the same data set. In most cases, the model learned this way will be overfitted and will perform poorly on unseen data (e.g., your testing dataset).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Test on test data:&lt;/strong&gt; the above methods use the data from Data signal only. To input another data set with testing examples (for instance from another file or some data selected in another widget), we select Separate Test Data signal in the communication channel and select Test on test data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, the answer to the second question is also &quot;yes:&quot; The model is trained on the data labeled &lt;code&gt;Training&lt;/code&gt; in you diagram and then tested on the &lt;code&gt;Testing&lt;/code&gt; data. The &quot;quality&quot; of the data and the way it was split in the training and testing set will influence the performance of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.orange.biolab.si/3/visual-programming/widgets/evaluation/testandscore.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Orange documentation on &quot;Test &amp;amp; Score&quot;&lt;/a&gt;:&lt;/p&gt;&#xA;" OwnerUserId="42142" LastActivityDate="2017-12-20T07:31:30.220" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25836" PostTypeId="1" AcceptedAnswerId="25839" CreationDate="2017-12-20T07:50:34.477" Score="0" ViewCount="262" Body="&lt;p&gt;If I get 95+ % accuracy in normal models, should I still consider Ensemble models? Why should I choose Ensemble models over normal models?&lt;/p&gt;&#xA;" OwnerUserId="43541" LastEditorUserId="15412" LastEditDate="2017-12-21T05:31:32.093" LastActivityDate="2021-05-13T01:31:07.847" Title="Ensemble Model vs Normal model" Tags="|machine-learning|ensemble-modeling|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25837" PostTypeId="2" ParentId="25666" CreationDate="2017-12-20T07:57:49.833" Score="3" Body="&lt;p&gt;1)&lt;strong&gt;Over Fitting&lt;/strong&gt; is one of the most practical difficulty for decision tree models. This problem gets solved by setting constraints on model parameters and pruning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)&lt;strong&gt;Not fit for continuous variables&lt;/strong&gt;: While working with continuous numerical variables, decision tree looses information when it categorizes variables in different categories.&lt;/p&gt;&#xA;" OwnerUserId="43541" LastActivityDate="2017-12-20T07:57:49.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25838" PostTypeId="1" AcceptedAnswerId="25917" CreationDate="2017-12-20T08:57:37.400" Score="0" ViewCount="155" Body="&lt;p&gt;I'm pretty new to ML and most of the supervised techniques I've discovered so far are along the lines of &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pass model training set inputs and get it to generate predictions&lt;/li&gt;&#xA;&lt;li&gt;evaluate predictions (predicted 'y') against real results (actual 'y') for the training set&lt;/li&gt;&#xA;&lt;li&gt;Perform some kind of gradient descent to tweak the model parameters to give outputs that are closer to the real results&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The precise method of training can vary, but the common theme in the examples I've seen so far is that the outputs of the model &lt;em&gt;are&lt;/em&gt; the actual things that we want to get out of the system. For example, we might be trying to predict house prices - in this case, an output of the model would actually be a predicted price, and it's that that I would care about as part of my cost function and as a user of the system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm imagining a scenario where we have a robot that's able to hold and use a palette and paintbrush, and we want it to copy existing paintings. (Strictly theoretical - I'm not trying to forge anything here!). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I imagine I'd want is to &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;present robot with example input (existing paintings)&lt;/li&gt;&#xA;&lt;li&gt;have the robot run a model where the output is some brush motions&lt;/li&gt;&#xA;&lt;li&gt;evaluate the painting &lt;em&gt;resulting from&lt;/em&gt; the brush strokes against the original painting&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The difference here is that the output of my model is 'brush strokes', but that output (from the model) &lt;em&gt;isn't&lt;/em&gt; ultimately what I care about - it's what's rendered by those brush strokes. I don't mind if the robot makes the same strokes that the original artist made, but I do mind about the similarity of what's rendered by those strokes to the original.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering what areas of ML I need to look into to find something that works this way, where our cost or fitness is evaluated based on some &lt;strong&gt;transform&lt;/strong&gt; of the model output, rather than the model output itself. I tried searching for examples of where a transform of the model output was incorporated into the cost function, but I couldn't see that it was a common technique.&lt;/p&gt;&#xA;" OwnerUserId="43531" LastEditorUserId="43531" LastEditDate="2017-12-22T21:47:36.410" LastActivityDate="2017-12-22T23:46:15.513" Title="What approaches could be used to teach a model to make appropriate brush strokes to reproduce a painting?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25839" PostTypeId="2" ParentId="25836" CreationDate="2017-12-20T09:46:21.817" Score="-1" Body="&lt;p&gt;Firstly, welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When do we use Ensemble model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;when there are 2 models which perform moderately then we combine their results to get a model which performs better. In your scenario you already have a model which gives you good results what is the point of implementing Ensemble Models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As @Tagoma said, it depends on your data and your goal. For example, you are trying to predict stock rates, every 0.01% matters. In such scenarios you need to use complex algorithms to maintain balance on that slim line i.e., not to over-fit, not to over-train and just predict. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Measures to check if your model is over-trained is by giving some random data and see how it performs, add some noise in the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more important thing to do is, to check for the Predictor Importance and see if there is any highly correlated feature with the target variable. For example, you are planning to predict age and if you have DOB as a feature in that yes of course you would predict with 99.99% accuracy but that is not what we use ML for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all of them are satisfied and achieving that accuracy then it means that your model's performance is good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, Implementation of Ensemble is dependent on your business problem and your understanding on business.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2017-12-21T05:52:46.037" LastActivityDate="2017-12-21T05:52:46.037" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="25840" PostTypeId="1" AcceptedAnswerId="25842" CreationDate="2017-12-20T09:51:17.320" Score="0" ViewCount="3907" Body="&lt;p&gt;I want to applicate the randomForest to my data for predicting target variable, but I have got a warnings message saying:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Warning message:&#xA;In randomForest.default(m, y, ...) :&#xA;  The response has five or fewer unique values.  Are you sure you want to do regression?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I didn't know what was going on?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(randomForest)&#xA;M=data.frame(Type_peau,PEAU_CORPS,SENSIBILITE,IMPERFECTIONS,BRILLANCE ,GRAIN_PEAU,RIDES_VISAGE,ALLERGIES,MAINS,&#xA;             INTERET_ALIM_NATURELLE,INTERET_ORIGINE_GEO,INTERET_VACANCES,INTERET_COMPOSITION,DataQuest1,Priorite2,&#xA;             Priorite1,DataQuest4,Age,Nbre_gift,w,Achat_client)&#xA;&#xA;factor_vars &amp;lt;- c(&quot;Type_peau&quot;,&quot;PEAU_CORPS&quot;,&quot;SENSIBILITE&quot;,&quot;IMPERFECTIONS&quot;,&quot;BRILLANCE&quot;,&quot;GRAIN_PEAU&quot;,&#xA;                 &quot;RIDES_VISAGE&quot;,&quot;ALLERGIES&quot;,&quot;MAINS&quot;,&quot;INTERET_ALIM_NATURELLE&quot;,&quot;INTERET_ORIGINE_GEO&quot;,&#xA;                 &quot;INTERET_VACANCES&quot;,&quot;INTERET_COMPOSITION&quot;,&quot;DataQuest1&quot;,&quot;Priorite2&quot;,&quot;Priorite1&quot;,&quot;DataQuest4&quot;)&#xA;&#xA;head(M)&#xA;str(M)&#xA;&#xA;sample.ind &amp;lt;- sample(2, nrow(M), replace = T,prob = c(0.6,0.4))&#xA;cross.sell.dev &amp;lt;- M[sample.ind==1,]&#xA;cross.sell.val &amp;lt;- M[sample.ind==2,]&#xA;&#xA;table(cross.sell.dev$Achat_client)/nrow(cross.sell.dev)&#xA;&#xA;table(cross.sell.val$Achat_client)/nrow(cross.sell.val)&#xA;&#xA;varNames &amp;lt;- names(M)&#xA;# Exclude ID or Response variable&#xA;varNames &amp;lt;- varNames[!varNames %in% c(&quot;Achat_client&quot;)]&#xA;&#xA;# add + sign between exploratory variables&#xA;varNames1 &amp;lt;- paste(varNames, collapse = &quot;+&quot;)&#xA;&#xA;# Add response variable and convert to a formula object&#xA;rf.form &amp;lt;- as.formula(paste(&quot;Achat_client&quot;, varNames1, sep = &quot; ~ &quot;))&#xA;&#xA;test$Variable1 &amp;lt;- factor(test$Variable1,levels=levels(train$Varialbe1))&#xA;&#xA;# Buiding &#xA;cross.sell.rf &amp;lt;- randomForest(rf.form,cross.sell.dev,ntree=500,importance=T)&#xA;plot(cross.sell.rf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help!&lt;/p&gt;&#xA;" OwnerUserId="41072" LastActivityDate="2022-01-08T02:55:09.977" Title="Warning message in randomForest" Tags="|machine-learning|r|data-mining|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25841" PostTypeId="1" CreationDate="2017-12-20T10:05:05.640" Score="6" ViewCount="27762" Body="&lt;p&gt;I have built the following model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_model(conv_kernels = 32, dense_nodes = 512):&#xA;&#xA;    model_input=Input(shape=(img_channels, img_rows, img_cols))&#xA;    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)&#xA;    x = Activation('relu')(x)&#xA;    x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)&#xA;    x = Activation('relu')(x)&#xA;    x = MaxPooling2D(pool_size=(2, 2))(x)&#xA;    x = Dropout(0.25)(x)&#xA;    x = Flatten()(x)&#xA;&#xA;    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)&#xA;&#xA;    x1 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x2 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x3 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;    x4 = Dense(nb_classes, activation='softmax')(conv_out)&#xA;&#xA;    lst = [x1, x2, x3, x4]&#xA;&#xA;    model = Model(inputs=model_input, outputs=lst)&#xA;    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)&#xA;    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])&#xA;&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I do a prediction:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it works properly. However, when I want to get prediction probability like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.predict_proba(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my model has no predict_proba function. Why not? Is it because of the multiple-output nature of the model?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="28175" LastEditDate="2017-12-20T20:05:11.493" LastActivityDate="2018-02-18T20:56:58.283" Title="Keras - no prediction probability for multiple output models?" Tags="|machine-learning|python|deep-learning|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25842" PostTypeId="2" ParentId="25840" CreationDate="2017-12-20T10:26:26.413" Score="1" Body="&lt;p&gt;In fact, the problem was in the type of target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be assumed as factor and not numeric !!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the solution is just to did like that after reading table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Achat_client=as.factor(Achat_client)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="41072" LastActivityDate="2017-12-20T10:26:26.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25843" PostTypeId="2" ParentId="25834" CreationDate="2017-12-20T10:54:27.000" Score="0" Body="&lt;p&gt;Take a supervised approach:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each group &quot;Blood test&quot;, &quot;Stool test&quot; etc,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Take a subset of your rows as a training set, say 200 rows.&lt;/li&gt;&#xA;&lt;li&gt;Create a new numeric or logical column, &quot;IsBloodTest&quot; or similar for this new subset of your dataset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For each row in these 200&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Classify them manually: if it is a blood test, assign 1, if not assign 0 to &quot;IsBloodTest&quot; &lt;/li&gt;&#xA;&lt;li&gt;Split the &quot;Description&quot; column into word vectors. This creates a Document-Term matrix with only 1 or 0 values in the cells (Word is present/not present)&lt;/li&gt;&#xA;&lt;li&gt;Classify the rows of the sparse new matrix (omitting your own newly manually created attribute) with the Multinomial NaiveBayes algorithm. This assigns &lt;em&gt;another&lt;/em&gt; column with &quot;0/1&quot; prediction, say &quot;PredictedBloodTest&quot;  to  your training set. &#xA;With these 0/1 values in the two new columns you can build a confusion matrix.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Perform Cross validation experiments to check your confusion matrix, or use an extra validation set. Label more rows manually if necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the classification result is good enough, use it for the remaining 1000s of rows from your data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This procedure is simple, but it might get you results faster than learning more sophisticated approaches first.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2017-12-20T11:40:00.067" LastActivityDate="2017-12-20T11:40:00.067" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25844" PostTypeId="2" ParentId="22750" CreationDate="2017-12-20T11:22:05.153" Score="1" Body="&lt;p&gt;Slightly related with the question (issues with Find Informative Projections), I found this post trying to solve this issue: the tool Find Informative Projections was disabled and Orange did not provide any hint about why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end the issue was on my side: I forgot to set any variable as Target when loading the data so it was useless to select it in the Scatter Plot points-colors section.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Double click on the File to load the data where you can choose if a variable is categorical or numerical, then double click on the variable I wanted to use as Target (that was marked as Feature) in order to select it as Target and the Find Informative Projections was enabled again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I have also noted that when making predictions, the file loaded should have all variables marked as Features. If you mark as before one of the variables as target, Orange shows a red circle with an exclamation mark on the Predictions widget, and a message: &lt;strong&gt;&quot;Data does not have the same target as predictors&quot;&lt;/strong&gt; (even though it is). I guess that the target in the prediction file is automatically set by checking which info is missing in the file.&lt;/p&gt;&#xA;" OwnerUserId="43548" LastActivityDate="2017-12-20T11:22:05.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25845" PostTypeId="2" ParentId="25350" CreationDate="2017-12-20T13:20:06.790" Score="0" Body="&lt;p&gt;Just to add on a bit.  I'm not sure where you got &quot;(3*3*9+3*3)/5*5&quot;.  A 3x3 filter has 9 parameters and a 5x5 has 25 paraemters.  Therefore, even if you stack two 3x3 filters, you still only have 18 compared to the 25.  That is in fact a 28% reduction in parameters that need to be calculated.  This simple reduction spread out across an entire network can dramatically improve speeds.  In fact, you can even take this principle and apply a nx1 filter that moves only moves across the horizon followed by a 1xn flter that moves vertically to see similar gains.  (Although this may introduce bottlenecks.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a good way to understand the computational cost is to actually calculate a simple filter by hand.  Keep track of all the operations you're doing and find out what is reusable.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, i'll also mention that stacking two 3x3 kernels gives you a receptive field of a 5x5 kernel.  Stacking three of 3x3 kernels gives you a receptive field of a 7x7 and so on and so forth.  Understanding the receptive field is critical for any computer vision task.  Here's some more info &lt;a href=&quot;https://arxiv.org/pdf/1701.04128.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;DCNN Receptive Field Info&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-20T13:20:06.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25847" PostTypeId="2" ParentId="25831" CreationDate="2017-12-20T14:06:24.433" Score="1" Body="&lt;p&gt;Throughout the hashing process those words may map to the same indices and for the example you gave, i'll say it's extremely likely you'll have collision.  How the hashing table chooses to handle that is up to the implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature hashing may purpose is when you have a large amount of data and that data tends to be sparse.  Words tend to be a good example because word matrix of counts tend to be rather sparse.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hashing trick is useful for large datasets because the more traditional ways of handling text data is essentially by making two passes on the dataset.  You basically have to go over the dataset once to create a dictionary and then make your transformation.  For large data this can be rather expensive process.  Feature hashing allows for a one time pass of the data.  This also enables the ability to do online learning, which brings me to this point.  I wouldn't say that feature hashing helps with new words rather that it basically handles it for you.   In traditional approaches you have to ignore the new word or create a new dictionary.  Feature hashing will simply just map it to an existing index or a new one if there's space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last point is that feature hashing rarely if ever improves performance of a model.  It also hurts the ability to be introspective.  You are going to lose information, and in fact, you can view feature hashing as a way to introduce noise.  However, research indicates that in large sparse data, the noise does not impact performance in major ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jmlr.org/papers/volume10/shi09a/shi09a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hashing Function&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/0902.2206.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Large Scale Hashing&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dzone.com/articles/feature-hashing-for-scalable-machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Short idea behind hashing&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-20T14:06:24.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25848" PostTypeId="2" ParentId="24719" CreationDate="2017-12-20T15:00:57.450" Score="1" Body="&lt;p&gt;Let $X_1, ..., X_n$ have a discrete join probability distribution $P(x \mid \theta)$ where $\vec{x} = (x_1, ... , x_n)$ is a vector in the sample and $\theta$ is a parameter from parameter space.  We will call $P(x \mid \theta)$ the likelihood function when it is a function of $\theta$.  By the ML principle we must maximize the likelihood function over some training set.  Assuming that the samples are i.i.d then ML is maximize by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$theta^{*} = argmax \sum_{i=1}^{m} log(P(x_i \mid \theta))$&lt;br&gt;&#xA;The above part should be review.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let  $\chi$ be the set $\{a_1, ... , a_n\}$ and let $P(a \mid \phi)$ be the probability of drawing $a_i$.   Finally let $x_1, ... , x_n$ be the sequence of the drawn and be i.i.d based on probability P.  $f(a)$ is the measurement of the draws i.e. $f(a) = | \{i : x_i = a\} |$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The empirical distribution is defined by $\hat{P}(a) = \frac{f(a)}{\sum_{a \in \chi}f(\alpha)} = \frac{f(a)}{m}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The joint probability $P(x_1, .. x_m | \theta) = \prod_{i=1}^{m}p(x_{i} | \phi)^{f(a)}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see here &lt;a href=&quot;http://www.cs.cmu.edu/afs/cs/user/aberger/www/html/tutorial/node4.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Training data&lt;/a&gt;, your tutorial and I have defined our emperical distribution similiary and obtained the same result.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;TLDR, it's just a matter of notation.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-20T15:00:57.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25849" PostTypeId="2" ParentId="25841" CreationDate="2017-12-20T15:08:07.333" Score="6" Body="&lt;p&gt;As you can see &lt;a href=&quot;https://keras.io/models/sequential/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; &lt;code&gt;Keras&lt;/code&gt; models contain predict method but they do not have the method &lt;code&gt;predict_proba()&lt;/code&gt; you have specified and they actually do not need it. The reason is that &lt;code&gt;predict&lt;/code&gt; method itself returns the probability of membership of the input to each class. If the last layer is &lt;code&gt;softmax&lt;/code&gt; then the probability which is used would be mutually exclusive membership. If all of the neurons in the last layer are &lt;code&gt;sigmoid&lt;/code&gt;, it means that the results may have different labels, e.g. existence of dog and cat in an image. For more information refer &lt;a href=&quot;https://datascience.stackexchange.com/q/25315/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information as stated &lt;a href=&quot;https://stackoverflow.com/questions/40747679/keras-what-is-the-difference-between-model-predict-and-model-predict-proba&quot;&gt;here&lt;/a&gt; &lt;em&gt;in the recent version of keras, predict and predict_proba are the same i.e. both give probabilities. To get the class labels use predict_classes. The documentation is not updated. (adapted from Avijit Dasgupta's comment)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-20T15:41:59.047" LastActivityDate="2017-12-20T15:41:59.047" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25850" PostTypeId="2" ParentId="25841" CreationDate="2017-12-20T15:18:37.060" Score="2" Body="&lt;p&gt;As you can see &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/models.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, keras predict_proba is basically the same as predict.  In fact, predict_proba, simply calls predict.  As for why it is not working, I have no idea, it seems like it should work.  However, it's mostly irrelevant since predict will give you the probabilities and predict_class will give you the labels.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2017-12-20T15:18:37.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25851" PostTypeId="2" ParentId="25832" CreationDate="2017-12-20T15:23:24.530" Score="10" Body="&lt;p&gt;You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping to other small intervals near to zero may also be fine but the latter case usually takes more time than the other. If you use &lt;code&gt;ReLU&lt;/code&gt;, again based on experience, you have to normalize your data and use standard initialization techniques for your weights, like &lt;a href=&quot;https://datascience.stackexchange.com/q/13061/28175&quot;&gt;He or Glorot&lt;/a&gt; methods. The reason is that your should avoid each activation to be so large, because your net would be so much dependent to that activation, and you may have overfitting problem. When you use &lt;code&gt;ReLU&lt;/code&gt; because there is no limit for its output, you have to normalize the input data and also use initialization techniques that avoid having large values for weights. For more information I encourage you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/23493/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/q/23287/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-20T18:46:00.567" LastActivityDate="2017-12-20T18:46:00.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25855" PostTypeId="1" CreationDate="2017-12-20T18:10:13.070" Score="1" ViewCount="74" Body="&lt;p&gt;For those of you who may implemented any machine learning application in your business context, can you share your experience and results obtained? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Amidst all the hype on AI I am interested to hear from actual applications. &lt;/p&gt;&#xA;" OwnerUserId="17047" LastActivityDate="2017-12-20T19:11:19.623" Title="Application of machine learning in your job" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2017-12-21T15:55:47.230" ContentLicense="CC BY-SA 3.0" />
  <row Id="25856" PostTypeId="2" ParentId="25855" CreationDate="2017-12-20T19:11:19.623" Score="0" Body="&lt;p&gt;Using machine-learning techniques, nowadays computers can automatically see and understand better than years ago. Tasks like recognition, localization, detection, semantic segmentation and related things have better performance today. This progress has promoted other fields too. Automatic cars, using autonomous driving benefit these advantages for pedestrian detection or car detection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recommender systems are so popular today. They suggest you exactly what you need based on your actions. On Facebook, you can find your old friends without even trying to. You can see a lot of movies which you may be sure that will like them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;During searching in search engines, you can see exactly what you are looking for without even knowing how the learning algorithm is ranking the results to show you the appropriate result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your email service finds spam and junk mails automatically and delete them for you and you can be confident that you won't loose any important thing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;OCR systems transform images of books and make them as text files. Using this, you can search in texts and find what you want without searching a lot of images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tasks which are difficult for finding algorithms, like walking of robots, are so better than before, using data and machine-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There have been designed agents which can defeat human brain in famous games, like chess. Even in AAA computer games, the agents use reinforcement learning to learn how to defeat the human rivals using experience of each specific player based on their behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning is EVERYWHERE.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-20T19:11:19.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25858" PostTypeId="1" CreationDate="2017-12-20T20:32:40.180" Score="4" ViewCount="2681" Body="&lt;p&gt;Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary information would be about the set S ∪ {v} --- although by now I no longer have access to the original members of S. I'd like the summary information to include the mean and variance of these sets, and as minimal additional information as needed. A natural idea for the additional information would be S's cardinality. But I'm willing to save more complicated information about S if needed. My main constraint is to minimize the size of the retained information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I only cared about the mean of the sets, then storing the mean plus cardinality of S would obviously be enough. I could update with a new value by just taking a weighted average of the old mean (times the old cardinality) and the new value. But I'd like to be able to keep track of the variance of the sets too. A good estimate is enough; I don't need to be able to reconstruct what the exact mean and variance of S ∪ {v} would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I expect that even asking this displays how naive I am about statistics, but I'd appreciate any help. I don't know where to look for answers.&lt;/p&gt;&#xA;" OwnerUserId="43569" LastActivityDate="2017-12-21T00:02:23.993" Title="How can I calculate mean and variance incrementally?" Tags="|aggregation|variance|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25859" PostTypeId="2" ParentId="25858" CreationDate="2017-12-20T21:01:24.920" Score="4" Body="&lt;p&gt;Following that link about moving variance in my comment, I came upon this: &lt;a href=&quot;https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Welford's online algorithm for calculating variance&lt;/a&gt;, which seems to supply what I'm looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the algorithm:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_count = old_count + 1&#xA;d1 = new_value - old_mean&#xA;new_mean = old_mean + d1/new_count&#xA;d2 = new_value - new_mean&#xA;new_sum_squares = old_sum_squares + d1*d2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From a saved &lt;code&gt;(count, mean, sum_squares)&lt;/code&gt;, the population variance can be computed as &lt;code&gt;sum_squares/count&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given an initial value &lt;code&gt;v&lt;/code&gt;, one can start with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;count = 1&#xA;mean = v&#xA;sum_squares = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want a weighted mean and variance, you can modify the algorithm like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_count = old_count + new_weight&#xA;d1 = (new_value - old_mean)*new_weight&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The other lines stay the same. (Here the mean of values &lt;code&gt;a,b&lt;/code&gt; with weights &lt;code&gt;x,y&lt;/code&gt;, respectively, is &lt;code&gt;(ax+by)/(x+y)&lt;/code&gt;; and the weighted sum_squares is &lt;code&gt;x(a-mean)^2 + y(b-mean)^2&lt;/code&gt;.)&lt;/p&gt;&#xA;" OwnerUserId="43569" LastEditorUserId="43569" LastEditDate="2017-12-21T00:02:23.993" LastActivityDate="2017-12-21T00:02:23.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25860" PostTypeId="1" CreationDate="2017-12-20T21:17:09.883" Score="1" ViewCount="1243" Body="&lt;p&gt;Hi I want to tune/search hyper-parameters of SVM in Orange tool. How can I do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to apply Exhaustive Grid Search to SVM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/grid_search.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43571" LastActivityDate="2017-12-20T21:17:09.883" Title="How to tune the hyper-parameters of an estimator in Orange Tool" Tags="|python|scikit-learn|svm|orange|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25861" PostTypeId="1" CreationDate="2017-12-20T22:24:56.750" Score="1" ViewCount="57" Body="&lt;p&gt;I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These reports are helpful summaries of information for the user to read later and gauge their progress. The data looks something like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user  |  hours until first report&#xA;------|--------------------------&#xA;3     | 20&#xA;------|--------------------------&#xA;4     | NaN&#xA;------|--------------------------&#xA;7     | 24&#xA;------|--------------------------&#xA;9     | 13&#xA;------|--------------------------&#xA;6     | 500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A &lt;code&gt;NaN&lt;/code&gt; value likely means that no report was ever written for this user because they unsubscribed before one could be written. I thought to fill in the blank values with the median, most common value, or the mean but this change in the data would show that unsubscribed users received written reports in the same time frame as the average user, which is incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of filling the missing value with the mean, I am considering adding a feature that tells whether a report was written at all and changing the &lt;code&gt;NaN&lt;/code&gt; values to -1 so that I can pass the data to a sklearn algorithm.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user  |  hours until first report  | report written&#xA;------|--------------------------------------------&#xA;3     | 20                         | 1&#xA;------|--------------------------------------------&#xA;4     | NaN                        | 0&#xA;------|--------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is this a valid way of solving this particular problem? What are the risks that I should be aware of?&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-20T22:24:56.750" Title="Fix missing data by adding another feature instead of using the mean?" Tags="|machine-learning|feature-engineering|missing-data|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25862" PostTypeId="1" CreationDate="2017-12-20T22:41:59.420" Score="3" ViewCount="962" Body="&lt;p&gt;I am not sure how to handle this error. This is from an RNN tutorial found &lt;a href=&quot;https://github.com/hunkim/word-rnn-tensorflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I vaguely understand that the variables need to be able to be reused, but I don't know how to implement this fix. Train.py ran for several hours and finished successfully; this particular error comes when running sample.py&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the long training time, I would very much like to get this working, rather than starting over.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Variable rnnlm/softmax_w already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:&#xA;  File &quot;C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py&quot;, line 1269, in __init__&#xA;    self._traceback = _extract_stack()&#xA;  File &quot;C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py&quot;, line 2506, in create_op&#xA;    original_op=self._default_original_op, op_def=op_def)&#xA;  File &quot;C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py&quot;, line 767, in apply_op&#xA;    op_def=op_def)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43575" LastEditorUserId="29575" LastEditDate="2017-12-20T22:48:11.157" LastActivityDate="2017-12-20T22:48:11.157" Title="Help with the following error: Variable already exists, disallowed. Did you mean to set reuse=True in VarScope?" Tags="|machine-learning|deep-learning|tensorflow|rnn|text-generation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25863" PostTypeId="2" ParentId="25858" CreationDate="2017-12-20T22:56:25.673" Score="4" Body="&lt;p&gt;This problem was discussed, with proof and some alternate methods over on &lt;a href=&quot;https://math.stackexchange.com/questions/102978/incremental-computation-of-standard-deviation&quot; title=&quot;math.stackexchange&quot;&gt;math.stackexchange&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43578" LastActivityDate="2017-12-20T22:56:25.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25865" PostTypeId="1" AcceptedAnswerId="25866" CreationDate="2017-12-21T02:29:41.053" Score="9" ViewCount="3068" Body="&lt;p&gt;Google Brain's &lt;a href=&quot;https://arxiv.org/abs/1706.03762&quot; rel=&quot;noreferrer&quot;&gt;Attention Is All You Need&lt;/a&gt; paper on sequence-to-sequence translation reports:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German&#xA;  translation task, improving over the existing best results, including&#xA;  ensembles, by over 2 BLEU&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but the &lt;a href=&quot;https://en.wikipedia.org/wiki/BLEU&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia entry on BLEU&lt;/a&gt; says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;BLEU’s output is always a number between 0 and 1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What definition of BLEU is the Google Brain paper using? I could not find a separate definition in the paper itself.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2019-03-06T10:27:57.500" Title="What is the BLEU score used in Google Brain's &quot;Attention Is All You Need&quot; paper?" Tags="|neural-network|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25866" PostTypeId="2" ParentId="25865" CreationDate="2017-12-21T04:57:06.957" Score="8" Body="&lt;p&gt;BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google uses the same definition of BLEU but multiplies the typical score by 100.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="68983" LastEditDate="2019-03-06T10:27:57.500" LastActivityDate="2019-03-06T10:27:57.500" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25868" PostTypeId="2" ParentId="17099" CreationDate="2017-12-21T06:26:04.137" Score="0" Body="&lt;p&gt;This is probably not the most efficient way, but the static variables could be repeated to timeseries length using &lt;code&gt;tf.tile()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43594" LastEditorUserId="29575" LastEditDate="2017-12-21T06:50:24.443" LastActivityDate="2017-12-21T06:50:24.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25869" PostTypeId="1" CreationDate="2017-12-21T06:50:56.707" Score="1" ViewCount="1323" Body="&lt;p&gt;What consideration do I need to take when choosing between One vs All or One vs One algorithm for Multiclass SVM?&lt;/p&gt;&#xA;" OwnerUserId="43592" LastActivityDate="2017-12-22T01:25:40.603" Title="Choosing One vs All and One vs One for Multiclass SVM" Tags="|svm|multiclass-classification|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-27T04:37:14.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="25870" PostTypeId="2" ParentId="25834" CreationDate="2017-12-21T08:10:47.010" Score="0" Body="&lt;p&gt;&lt;strong&gt;Problem 1:&lt;/strong&gt; &#xA;&lt;em&gt;Approach 1 :&lt;/em&gt; &#xA;Do you have with you the possible types of tests that can be conducted? &#xA;If yes, then as suggested above - assign each row to one of the test type. Then build the classifier model based on this. This model can be used to predict the test type for new data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Approach 1 :&lt;/em&gt; &#xA;If you do not have the information about what are the possible test types and you want the machine to find the pattern, then go for clustering approach , where similar titles will be grouped together, and you will be getting multiple such groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem 2:&lt;/strong&gt; &#xA;Clearly you have only two output classes , its a classification problem.You need labelled data initially to build the model. You will have to (again) initially label data based on your domain knowledge, and then feed it to the classification algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If using R - you can use the tm package , if python then nltk should help you.The choice of which algorithm to use depends on mulitple factors.Do let me know if you need more info.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="24575" LastActivityDate="2017-12-21T08:10:47.010" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25873" PostTypeId="2" ParentId="25836" CreationDate="2017-12-21T09:41:14.920" Score="0" Body="&lt;p&gt;It is a problem specific and data specific problem. For one problem, 95% accuracy may be good, for some other it might not be very good and there might still be room for improvements. Likewise, it might happen that the problem is well defined, your architecture is good but due to a lot of noise in the data, you can't do better than 95%. In that case, we will say the results are good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ensembles:&lt;/strong&gt; Ensembling is a general term for combining many classifiers by averaging or voting. It is a form of meta-learning in that it focuses on how to merge results of arbitrary underlying classifiers. Generally, ensembling improves the final results but again, not necessarily. If you are ensembling two highly correlated models, your ensembled model will give almost the same result as your stand-alone models.&lt;/p&gt;&#xA;" OwnerUserId="15412" LastActivityDate="2017-12-21T09:41:14.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25875" PostTypeId="1" CreationDate="2017-12-21T11:21:31.917" Score="3" ViewCount="679" Body="&lt;p&gt;I am using a stock auto-encoder anomaly detector from &lt;a href=&quot;https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/feedforward/anomalydetection/MNISTAnomalyExample.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deeplearning4j&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was getting unexpected results from my own variant of the auto-encoder, which looks for anomalies in my own (non-image) data, and to try and investigate, I added some additional images, 10 white and 10 black to the MNIST test data, to see the effect. See the output picture below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hFltK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hFltK.png&quot; alt=&quot;the low reconstruction and high reconstruction digits&quot;&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What was unexpected was that those black images (all zero input values) have zero reconstruction error (the left column in the &quot;best&quot; picture), despite the auto-encoder being trained only on stereotypical digits (no black images), while the white images(i.e. all 1 input values) have the highest reconstruction error (the left column on the &quot;worst&quot; image). Given the generally accepted definition of an anomaly detector, I would have expected both all-black and all-white to have high reconstruction error, as neither appeared in the training set, and hence should be anomalous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is consistent with the unexpected output from my own variant of the auto-encoder.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suspect there is a data science reason why this is the case with an auto-encoder, but can someone shed light, or point me to references why?&lt;/p&gt;&#xA;" OwnerUserId="43604" LastEditorUserId="29575" LastEditDate="2017-12-21T11:42:55.023" LastActivityDate="2017-12-21T11:42:55.023" Title="Autoencoder behavior with All White/Black MNIST" Tags="|deep-learning|anomaly-detection|autoencoder|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25876" PostTypeId="1" CreationDate="2017-12-21T11:43:25.610" Score="1" ViewCount="816" Body="&lt;p&gt;Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by calculating the cosine similarity between the documents' TfIDF vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I now compile a set of bigrams based on frequency, and replace instances of two words in each document with the concatenation of the bigram, &lt;strong&gt;what affect will this have on&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) Calculating TfIDF vectors?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) Calculating document similarity?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be explicit, lets take an example in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;doc1 = ['the', 'car', 'drove', 'from', 'new','york', 'to', 'washington'] # a single text document&#xA;top_n = {('new', 'york'), ('drove', 'from')...} # set of top n bigrams by frequency&#xA;new_doc = replace_ngrams(doc1) # function that replaces words with concatenated bigrams&#xA;print(new_doc)&#xA;&amp;gt;&amp;gt;&amp;gt; ['the', 'car', 'drovefrom', 'newyork', 'to', 'washington']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-03-21T17:41:52.527" Title="What affect will replacing words with bigrams have on TfIDF?" Tags="|python|nlp|tfidf|ngrams|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25877" PostTypeId="1" CreationDate="2017-12-21T13:33:10.953" Score="1" ViewCount="107" Body="&lt;p&gt;I have a data set obtained from the event log of a bunch of access points in a building. At the moment I am interested in forecasting, for each access point, the number of connected devices at each access point on a given point in time. The features in this data set are &lt;em&gt;timestamp&lt;/em&gt;, containing the datetime and &lt;em&gt;AP1&lt;/em&gt;, ... , &lt;em&gt;AP9&lt;/em&gt;, containing the (relative) number of connected devices for each access point. An example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; synced_logs[500:515,]&#xA;              timestamp 1 2  3  4  5  6  7  8  9&#xA;500 2017-09-25 12:55:00 1 0  1 -1 -1  0  1 -1  0&#xA;501 2017-09-25 12:56:00 1 0  1 -1 -1  0  1 -1  0&#xA;502 2017-09-25 12:57:00 1 0  1 -1 -1  0  1 -1  0&#xA;503 2017-09-25 12:58:00 1 0 -1 -1  1  0  1 -1  0&#xA;504 2017-09-25 12:59:00 1 0 -1  1  1  0 -1 -1  0&#xA;505 2017-09-25 13:00:00 1 0  1  1  1  0 -1 -1  0&#xA;506 2017-09-25 13:01:00 1 0  0  1 -1  1 -1 -1  0&#xA;507 2017-09-25 13:02:00 1 0  0  1 -1  0 -1 -1  0&#xA;508 2017-09-25 13:03:00 1 0 -2  1 -1 -1  3  3  0&#xA;509 2017-09-25 13:04:00 1 0  1  0 -1 -1  1 -2  0&#xA;510 2017-09-25 13:05:00 1 0  0  0 -1  1 -1  2  0&#xA;511 2017-09-25 13:06:00 1 0 -1  1 -1 -1  1 -2  1&#xA;512 2017-09-25 13:07:00 1 0 -2 -2 -1  3  1 -2  1&#xA;513 2017-09-25 13:08:00 1 0 -2 -2 -1  1 -1 -2 -1&#xA;514 2017-09-25 13:09:00 1 0  1 -2 -1  1  0 -2 -1&#xA;515 2017-09-25 13:10:00 1 0  1 -2 -1  1 -2 -2 -1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(the exact number of already connected devices at the first measurement is not known, so the counts are all relative). The data set spans a period of a week and contains 9803 observations (at the minute level). The data set was transformed from a asynchronous event log to a synchronous time series format. I plan on adding lagged variables to this data set if needed (based on partial auto- and cross-correlation statistics).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to try to model the counts using neural networks, but have a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do neural networks play nice with non-negative, discrete data?&lt;/li&gt;&#xA;&lt;li&gt;Are certain network topologies preferred over others in a case like this? E.g. LSTMs?&lt;/li&gt;&#xA;&lt;li&gt;Is it better to model a multivariate outcome (i.e. a vector a counts) or is it better to fit a model for each access point?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43608" LastActivityDate="2017-12-21T13:33:10.953" Title="Multivariate non-negative, discrete time series forecasting with neural networks" Tags="|neural-network|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25878" PostTypeId="2" ParentId="13730" CreationDate="2017-12-21T15:20:53.063" Score="1" Body="&lt;p&gt;Let's start with your problem definition: &quot;a good strategy make the relationships first and then count the occurrences&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, roughly, the basic strategy that market basket analysis algorithms use. However, algorithms like &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apriori&lt;/a&gt; or FPGrowth are specially designed to analyze such datasets (at scale) and infer the inherent association rules between items across all baskets. My recommendation would be to use one of these to gather the relationships between items purchased, instead of reinventing them; especially because you'll face a lot of the hard problems these algorithms already solve (namely the large search space when generating combinations of basket items).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use any of several libraries or languages to do this, namely R, Python, etc. Doing this in Spark would be pretty simple using MLLib, your workflow would be something like: 1) choose an algorithm, e.g. FPGrowth; 2) prepare your data to fit the format required by FPGrowth (each transaction should be an array of basket items); 3) run FPGrowth and output its frequent itemsets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a good example of this at Spark's &lt;a href=&quot;https://spark.apache.org/docs/2.1.2/mllib-frequent-pattern-mining.html#fp-growth&quot; rel=&quot;nofollow noreferrer&quot;&gt;website&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.fpm.FPGrowth&#xA;import org.apache.spark.rdd.RDD&#xA;&#xA;val data = sc.textFile(&quot;data/mllib/sample_fpgrowth.txt&quot;)&#xA;&#xA;//prepare the data to use with FPGrowth&#xA;val transactions: RDD[Array[String]] = data.map(s =&amp;gt; s.trim.split(' '))&#xA;&#xA;//create and run the model&#xA;val fpg = new FPGrowth()&#xA;  .setMinSupport(0.2)&#xA;  .setNumPartitions(10)&#xA;val model = fpg.run(transactions)&#xA;&#xA;//output the frequent itemsets (items frequently bought together)&#xA;model.freqItemsets.collect().foreach { itemset =&amp;gt;&#xA;  println(itemset.items.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;) + &quot;, &quot; + itemset.freq)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43615" LastActivityDate="2017-12-21T15:20:53.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25880" PostTypeId="2" ParentId="25876" CreationDate="2017-12-21T15:42:48.633" Score="1" Body="&lt;p&gt;TFIDF decreases as term frequency will  be decreased linearly and idf increases log linearly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2017-12-21T15:42:48.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25881" PostTypeId="1" AcceptedAnswerId="25885" CreationDate="2017-12-21T16:03:56.423" Score="2" ViewCount="2557" Body="&lt;p&gt;I am working on using random forest to predict 1 or 0. I have about 20 variables available for modeling. I realized that if I put different variables will have different accuracy/sensitivity/specificity. I am wondering if there is a test or method can tell me which variables combination has the highest accuracy? Or which variable combination has the highest sensitivity and specificity respectively? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="28860" LastActivityDate="2017-12-21T19:04:50.940" Title="Features selection/combination for random forest" Tags="|machine-learning|classification|feature-selection|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25882" PostTypeId="1" CreationDate="2017-12-21T16:05:29.863" Score="2" ViewCount="215" Body="&lt;p&gt;I'm working on extending a model of human immediate serial recall task performance, originally described in &lt;a href=&quot;http://www.cnbc.cmu.edu/~plaut/papers/abstracts/BotvinickPlaut06PR.serialOrder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;. This model takes a sequence of items, such as digits or phonemes, stores them as they are presented, and then repeats the sequence back in order. Their model uses one-hot encoding and decoding to describe items to be remembered and recalled (such as digits or phonemes) as atomic units in memory. I'd like to adapt this model to describe items as collections of features instead. For example, rather than encoding the phoneme /b/ as a unique element in a one-hot vector, I'd like to model it as a combination of phoneme features (e.g. voiced = true, manner = stop, place = labial), with each input and output corresponding to one feature.  Similarly, rather than modeling the digit &quot;one&quot; as a unique element in a one-hot vector, I'd like to model it as an input sequence of the phonemes (e.g. [w,^,n]).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This formulation of the problem naturally leads to a set of possible model outputs that is larger than the set of items that occur in the training set. However, the models I've trained so far do not appear to have learned this structure in the data set. When the model produces an incorrect phoneme, it produces phoneme features combinations that do not occur in the training set at about a chance rate.  Similarly, the model does not seem to learn the transition probabilities between phonemes within a word, and instead its incorrect responses seem unrelated across sequential phonemes.  Restated, the model doesn't seem to learn to exploit the set of phonemes or words that are in the training data, and instead seem to &quot;brute-force&quot; the recall task by remembering every input as a sequence of phoneme features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, my question is if there are specific model structures or optimizers designed to exploit this cross-output and cross-time information to learn that some output patterns are &quot;valid&quot; (occur within the training set) or &quot;invalid&quot; (never occur within the training set)?  I don't know if this is a problem that has been previously addressed, or whether I'm using the correct terminology, so any guidance would be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To date, I've tried a vanilla recurrent layer with a set of dense one-hot output layers to represent the output features.  I've tried adding a softmax-activation dense layer before and after the recurrent layer, but this seemed to impair the model ability to store sequences. For optimizers, I have tried SGD, Adagrad, and Adam optimizers, with Adam generally giving the best results.  Replacing the vanilla recurrent layer with LSTM or GRU units has improved the model ability to recall item sequences, but doesn't address the &quot;invalid&quot; response rate. I've been using Keras to implement the model.&lt;/p&gt;&#xA;" OwnerUserId="43614" LastActivityDate="2017-12-21T16:05:29.863" Title="Train a multi-output neural network to learn subset of &quot;valid&quot; response combinations" Tags="|neural-network|multiclass-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25883" PostTypeId="1" AcceptedAnswerId="25888" CreationDate="2017-12-21T17:02:47.573" Score="3" ViewCount="1749" Body="&lt;p&gt;I am using Random Forests, XGBoost and SVMs to classify whether the home team wins or the away team wins their bowl game (in college football). I trained the models on all the games during the season.&lt;/p&gt;&#xA;&lt;p&gt;I've come across something that is a bit weird and can't explain. I calculated a prediction confidence by subtracting the class probabilities. The XGBoost confidence values are consistency higher than both Random Forests and SVM's. I've attached the image below.&lt;/p&gt;&#xA;&lt;p&gt;I did some hyper-parameter tuning for all of my models and used the best parameters based on testing accuracy.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Random Forest:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;700 trees&lt;/li&gt;&#xA;&lt;li&gt;15 variables randomly sampled (mtries)&lt;/li&gt;&#xA;&lt;li&gt;minimum split criteria of 5 rows.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;XGBoost:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;0.5, Learn rate&lt;/li&gt;&#xA;&lt;li&gt;gbtree as my booster&lt;/li&gt;&#xA;&lt;li&gt;max depth of 6&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;SVM:&#xA;&lt;ul&gt;&#xA;&lt;li&gt;RBF kernel&lt;/li&gt;&#xA;&lt;li&gt;C (slack) of 1&lt;/li&gt;&#xA;&lt;li&gt;0.01, Sigma&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;I wasn't clear with my question: Why exactly does XGBoost prefer one class greatly to the other? In comparison to these other methods. I'm trying to figure out why my prediction confidences of a class are so high for XGboost.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WTCo5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WTCo5.png&quot; alt=&quot;First half&quot; /&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aDyqz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aDyqz.png&quot; alt=&quot;Second half&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29601" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-02T12:20:19.720" Title="XGBoost Classification Probabilities higher than RF or SVM?" Tags="|r|random-forest|svm|xgboost|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25884" PostTypeId="1" CreationDate="2017-12-21T18:12:13.463" Score="3" ViewCount="300" Body="&lt;p&gt;I would like to know where I could find useful articles/papers about the basic concepts of &lt;em&gt;RPN&lt;/em&gt; and &lt;em&gt;R-CNN&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastEditorUserId="83275" LastEditDate="2023-04-15T17:25:25.723" LastActivityDate="2023-04-15T17:25:25.723" Title="How does region proposal network (RPN) and R-CNN works?" Tags="|deep-learning|cnn|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25885" PostTypeId="2" ParentId="25881" CreationDate="2017-12-21T19:04:50.940" Score="2" Body="&lt;p&gt;The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt; model in sklearn has a &lt;code&gt;feature_importances_&lt;/code&gt; attribute to tell you which features are most important. Here is a &lt;a href=&quot;http://blog.datadive.net/selecting-good-features-part-iii-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;helpful example&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few other algorithms for selecting the best features that generalize to other models such as &lt;em&gt;sequential backward selection&lt;/em&gt; and &lt;em&gt;sequential forward selection&lt;/em&gt;. In the case of sequential forward selection, you begin by finding the single feature that provides you with the best accuracy. Then, you find the next feature in combination with the first that gives you the best accuracy. This pattern continues until you find $k$ features, where $k$ is the number of features you want to use. Sequential backward selection is just the opposite, where you start with all of the features and remove those which inhibit your accuracy the most. You can find more information on these algorithms &lt;a href=&quot;https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/#example-2-toggling-between-sfs-sbs-sffs-and-sbfs&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43215" LastActivityDate="2017-12-21T19:04:50.940" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25886" PostTypeId="1" CreationDate="2017-12-21T20:31:56.173" Score="2" ViewCount="481" Body="&lt;p&gt;I am getting the following error whenever I train my model in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- keras_model_sequential()&#xA;&#xA;model %&amp;gt;%&#xA;  layer_lstm(units=data_dim, input_shape = c(timesteps, data_dim),return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %&amp;gt;%&#xA;  layer_activation(&quot;linear&quot;) %&amp;gt;%&#xA;  layer_lstm(units=data_dim,return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %&amp;gt;%&#xA;  layer_activation(&quot;linear&quot;) %&amp;gt;%&#xA;  layer_lstm(units=5*data_dim,return_sequences=TRUE, use_bias=TRUE,dropout=0.35,recurrent_dropout=0.35) %&amp;gt;%&#xA;  layer_activation(&quot;linear&quot;) %&amp;gt;%&#xA;  layer_dense(units=out_dim,activation=&quot;linear&quot;)&#xA;&#xA;optimizer &amp;lt;- optimizer_rmsprop(lr = 0.01)&#xA;&#xA;model %&amp;gt;% compile(&#xA;  loss = &quot;mean_squared_error&quot;, &#xA;  optimizer = optimizer&#xA;)&#xA;&#xA;model %&amp;gt;% fit(&#xA;    train_in, train_out,&#xA;    batch_size = batch_size,&#xA;    epochs = 1000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;UserWarning: Method on_batch_end() is slow compared to the batch update (1.533262). Check your callbacks.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can somebody please explain what's going on?&lt;/p&gt;&#xA;" OwnerUserId="43474" LastEditorUserId="29575" LastEditDate="2017-12-22T05:49:50.567" LastActivityDate="2017-12-22T05:49:50.567" Title="Getting the following error with my LSTM in R" Tags="|machine-learning|neural-network|r|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25887" PostTypeId="2" ParentId="25883" CreationDate="2017-12-21T20:51:59.817" Score="1" Body="&lt;p&gt;As a side note, you have not mentioned any regularisation parameters of xgboost, so I understand you don’t use any. In general, it is not good and might lead to overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your question, my hypothesis is that in your case xgboost classifier is simply more powerful than other two approaches and thus is more confident, which is indicated by higher probabilities assigned to particular classes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe xgboost is even overconfident, i.e. overfits, but one cannot be sure without thorough testing on unseen test data. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-12-21T20:51:59.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25888" PostTypeId="2" ParentId="25883" CreationDate="2017-12-21T21:02:52.633" Score="2" Body="&lt;p&gt;Rather than answering why XGBoost give very confident predictions, I will answer why random forest and SVM give not-so-confident predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random forest probability estimates are given by the percentage of the forest that predicted a particular class. For example, if you have $100$ trees in your forest and $81$ of them predict some class for some example, the probability estimate for that example belonging to that class is calculated to be $\frac{81}{100} = 0.81$. Because of the random nature of the ensemble members, it's very unlikely that each individual tree will end up with the correct prediction, even if the majority do. This makes probability estimates from random forests shy away from the extreme ends of the scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVM is a slightly different case, because they are unable to produce probability estimates directly. Typically, Platt scaling (essentially logistic regression) is used to scale the SVM output to a probability estimate. This has the added benefit of calibrating the probability estimates, meaning the predicted probability is quite accurate - in other words, if a probability of $0.8$ is given for a prediction, it actually has approximately an $80\%$ chance of being correct. For a problem like this where there's a lot of noise (underdog teams &lt;em&gt;do&lt;/em&gt; win sometimes, and there's a lot of evenly matched games that are hard to predict), these predictions will tend not to be overconfident.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have a good reason as to why XGBoost is possibly overconfident, but it has been observed in the past that additive boosting models tend to provide distorted probability estimates without applying post-training calibration e.g. &lt;a href=&quot;http://avesbiodiv.mncn.csic.es/estadistica/bt4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, &lt;a href=&quot;https://projecteuclid.org/euclid.aos/1024691352&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; &amp;amp; &lt;a href=&quot;https://projecteuclid.org/euclid.aos/1016218223&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-04-02T12:20:19.720" LastActivityDate="2018-04-02T12:20:19.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25889" PostTypeId="2" ParentId="24506" CreationDate="2017-12-21T22:14:32.183" Score="9" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I just want to know why it's not considered an important feature based on the frequency metric.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Most likely, the variable &lt;em&gt;gender&lt;/em&gt; has much smaller number of possible values (often only two: male/female or 0/1, depending on the representation) compared to other predictors in your dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;em&gt;gender&lt;/em&gt; is just binary in your case, it means that it can be used at most once in each tree, while, let say, &lt;em&gt;age&lt;/em&gt; might appear much more often on different levels of the trees. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2017-12-21T22:14:32.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25890" PostTypeId="1" CreationDate="2017-12-21T22:24:35.860" Score="2" ViewCount="108" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a way to reduce the noisiness and stochasticity of &lt;a href=&quot;https://pdfs.semanticscholar.org/916c/eefae4b11dadc3ee754ce590381c568c90de.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rprop&lt;/a&gt; (and for that matter the &lt;a href=&quot;https://pdfs.semanticscholar.org/df9c/6a3843d54a28138a596acc85a96367a064c2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;iRprop+&lt;/a&gt;)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Specifically, in deep networks (with 8+ layers) this effect starts to become apparent, as the earliest layers are adjusted. &#xA;This has a massive effect on the outcome, and the error jumps around. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The noise also happens in a network with just 2-3 layers. However, it is only apparent when the error reached 0.00, and the iRprop+ keeps running. In some cases it will suddenly cause a very abrupt change, and the cross-entropy cost function will produce error larger than 1 000 000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've built a custom LSTM in c++ and experiencing this noise during debugging of overfitting&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps the learning rates from each layer should be initialized with a different magnitude, depending on the layer's  depth? &lt;a href=&quot;https://datascience.stackexchange.com/q/25874/43077&quot;&gt;[link]&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastActivityDate="2017-12-21T22:24:35.860" Title="Rprop is too noisy" Tags="|backpropagation|noise|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25891" PostTypeId="1" AcceptedAnswerId="25901" CreationDate="2017-12-21T22:56:47.560" Score="0" ViewCount="563" Body="&lt;p&gt;I have an assignment to train a model to classify text data, the brief for the assignment mentions that any for any learning model used I have to provide a reasoning for the similarity or distance metric used. What does this refer to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Minkowski) or L2 (Euclidean). Would this be correct?&lt;/p&gt;&#xA;" OwnerUserId="43634" LastEditorUserId="43634" LastEditDate="2017-12-22T15:20:19.213" LastActivityDate="2017-12-22T15:20:19.213" Title="What are similarity and distance metrics in classification?" Tags="|machine-learning|distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25892" PostTypeId="1" CreationDate="2017-12-21T23:50:10.587" Score="0" ViewCount="63" Body="&lt;p&gt;I have the following table with three columns: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Type    Category    Actual&#xA;A       1           47&#xA;A       1           90&#xA;A       1           74&#xA;A       2           64&#xA;A       2           25&#xA;A       3           55&#xA;A       3           55&#xA;A       3           19&#xA;B       6           80&#xA;B       6           28&#xA;B       2           60&#xA;B       2           49&#xA;B       2           7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, I'm given a static table that provides predictions by Type and Category.  I will be using the &quot;Prediction&quot; column values as the actual predictions.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Type    Category    Prediction&#xA;A       1           83&#xA;A       2           48&#xA;B       6           54&#xA;B       2           57&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So once I combine the two tables by Type and Category, I have the following table:    &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Type    Category    Actual  Prediction&#xA;A       1           47      83&#xA;A       1           90      83&#xA;A       1           74      83&#xA;A       2           64      48&#xA;A       2           25      48&#xA;B       6           80      54&#xA;B       6           28      54&#xA;B       2           60      57&#xA;B       2           49      57&#xA;B       2           7       57&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to evaluate my data by distinct Type and Category combinations.  So my output table could be something like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Type    Category    Avg_Actual  Avg_Prediction&#xA;A       1           value       value &#xA;A       2           value       value&#xA;B       6           value       value&#xA;B       2           value       value &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: What are different metrics I can use to evaluate how predictive the &quot;Prediction&quot; column is against the &quot;Actual&quot; values given my data set.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please be specific with your metrics and the more you can provide (with details on how to derive them) the better.  So far, the only one I came up with was calculating RMSE.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like all the metrics at a distinct Type-Category level.  Additionally, is there a way to incorporate confidence intervals into this analysis?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is greatly appreciated.  &lt;/p&gt;&#xA;" OwnerUserId="43635" LastActivityDate="2017-12-22T02:10:21.940" Title="What Are Stats Metrics To Analyze How Well One Column Predicts Another Column" Tags="|regression|statistics|prediction|model-evaluations|accuracy|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-26T04:18:18.883" ContentLicense="CC BY-SA 3.0" />
  <row Id="25893" PostTypeId="1" AcceptedAnswerId="26005" CreationDate="2017-12-21T23:52:07.440" Score="1" ViewCount="391" Body="&lt;p&gt;I have a problem. I have a data set with some users and their ratings in several movies. The movies are separated into &lt;strong&gt;19 genres&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to cluster the users by their preferences(ratings in the movies). The problem is, that I want to find a $threshold ( θ )$ to do the clustering, but I do not know how to do this, because the data are discrete and I cannot use the statistics methods that I know.&#xA;The threshold is the maximum distance that two users can have to be in the same cluster, like 2 users that likes the same genre movies or have little differences in their tastes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried to find a threshold using simple statistics. For example, for a user sum all of his ratings in a genre and divide the result via the number of ratings and find some means in some genres, but I didn't got an answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I must use BSAS&lt;/p&gt;&#xA;" OwnerUserId="43637" LastEditorUserId="43637" LastEditDate="2017-12-25T00:26:59.243" LastActivityDate="2017-12-26T01:43:16.307" Title="Find threshold in large dataset" Tags="|machine-learning|python|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25896" PostTypeId="2" ParentId="25869" CreationDate="2017-12-22T01:25:40.603" Score="1" Body="&lt;p&gt;The main consideration is the number of classes, assume you have $N$ different classes: &quot;one vs all&quot; will train one classifier per class, so $N$ classifiers in total. For a given class $c_i$ the classifier assumes samples with $c_i$ as positives and the rest as negatives. Obviously, this leads to imbalanced datasets, moreover, in &quot;one vs all&quot; &lt;a href=&quot;https://stats.stackexchange.com/questions/143152/why-do-one-versus-all-multi-class-svms-need-to-be-calibrated/143563&quot;&gt;the outputs must be calibrated to produce comparable results&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &quot;one vs one&quot; you train a separate classifier for each different pair of classes, so $\frac{N(N - 1)}{2}$, obviously this is computationally expensive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details see this: &lt;a href=&quot;https://stats.stackexchange.com/questions/91091/one-vs-all-and-one-vs-one-in-svm&quot;&gt;1&lt;/a&gt; and &lt;a href=&quot;https://www.quora.com/In-multi-class-classification-what-are-pros-and-cons-of-One-to-Rest-and-One-to-One&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-22T01:25:40.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25897" PostTypeId="1" CreationDate="2017-12-22T01:27:14.717" Score="1" ViewCount="1847" Body="&lt;p&gt;I have data coming from different sources having similar information like the below example where different sources want to specify the age criteria. Is there a NLP or Deep learning based approach which I can use to extract the age rule as shown below from raw unstructured text. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A Criteria&#xA;Applicants should be above 21 years of age and up to 65 years or less at the time of maturity.&#xA;&#xA;B Criteria&#xA;You are between 25-58 years of age.&#xA;&#xA;C Criteria&#xA;Age of the all applicants should be between 21 and 65 .&#xA;&#xA;&#xA;A Rule&#xA;age &amp;gt;=21 AND age &amp;lt;= 65&#xA;&#xA;B Rule&#xA;age &amp;gt;=25 AND age &amp;lt;= 58&#xA;&#xA;C Rule&#xA;age &amp;gt;=21 AND age &amp;lt;= 65&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18313" LastActivityDate="2018-09-04T09:14:35.733" Title="How to extract specific information from raw , unstructured text using NLP and Deep Learning?" Tags="|deep-learning|nlp|named-entity-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25898" PostTypeId="1" CreationDate="2017-12-22T02:05:40.973" Score="3" ViewCount="772" Body="&lt;p&gt;I am trying to create a custom loss function for a Keras regression task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am predicting the points scored per minute in a game, and training on &quot;matches&quot; of variable lengths, in minutes. In an attempt to help the model learn, I would like to include the number of minutes the group played as part of the loss function, so we can properly &quot;punish&quot; for missing on groups that play for longer durations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I need to use Keras Backend/tensor operations, but I'm stuck on how to prepare the minutes/how to multiply the minutes tensor. Here's some basic code what I have so far (pseudo code/skipped where not important):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def penalized_loss(minutes):&#xA;&#xA;    # minutes is np.array - do we need to convert to tensor?&#xA;    #    Is this useful/how to do it?&#xA;    minutes_tensor = K.constants(minutes) &#xA;&#xA;    def loss(y_true, y_pred):&#xA;&#xA;        # Normal mean_squared_error - this makes sense to me&#xA;        lost = K.mean(K.square(y_pred - y_true), axis=-1)&#xA;&#xA;        # Would like something like this where minutes are included, but as a valid tensor operation&#xA;        not_valid_result = K.mean(K.square(y_pred - y_true) * minutes, axis=-1)&#xA;&#xA;        # Is this closer to what we need to do?&#xA;        maybe_closer_result = K.mean(K.dot(K.square(y_pred - y_true), minutes),  axis=-1)&#xA;&#xA;        return lost&#xA;&#xA;    return loss&#xA;&#xA;# minutes is np.array, shape (12024,)&#xA;# input/predictions also have shape 12024&#xA;&#xA;def baseline_model():&#xA;    model = Sequential()&#xA;    # Create the model....&#xA;&#xA;    # Compile model&#xA;    model.compile(loss=[penalized_loss(minutes)], optimizer='adam')&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The closure to pass in minutes works as I expect, I'm just stuck on how to do the &lt;code&gt;* minutes&lt;/code&gt; portion of the loss calculation. I've looked through the docs and I feel like I'm missing something easy/obvious. Can anyone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="43641" LastEditorUserId="43641" LastEditDate="2017-12-22T02:25:50.403" LastActivityDate="2017-12-22T02:25:50.403" Title="Keras custom loss - operation on additional data" Tags="|neural-network|regression|keras|loss-function|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25899" PostTypeId="2" ParentId="25892" CreationDate="2017-12-22T02:10:21.940" Score="1" Body="&lt;p&gt;Other possible evaluation measures are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Mean Absolute Error (MAE): $\frac{\sum_{i=1}^n{\| y_i - x_i\|}}{n}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;R^2: also known as coefficient of determination, the formula is rather lengthy, but a detailed definition, as well as a motivation, can be found &lt;a href=&quot;https://en.wikipedia.org/wiki/Coefficient_of_determination&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Root Mean Squared Logarithmic Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(x_i + 1))^2}}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Explained Variance Score: $1 - \frac{Var(y - x)}{Var \ y}$ more &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Explained_variation&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mean absolute percentage error (MAPE): see the &lt;a href=&quot;https://en.wikipedia.org/wiki/Mean_absolute_percentage_error&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt; for the formula and major drawbacks of this measure.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Median Absolute Error (MAE2): $Median(| y_1 - x_1 | \dots | y_n - x_n |)$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These measures, as well as others, can be found &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; together with some Python code. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-22T02:10:21.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25901" PostTypeId="2" ParentId="25891" CreationDate="2017-12-22T05:14:38.243" Score="0" Body="&lt;p&gt;The choice of the metric is somewhat dependent on how you are representing the text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common metric for vector space models would be cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity: K(X, Y) = &amp;lt; X, Y &gt; / (||X||*||Y||) &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Blockquote My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Manhattan) or L2 (Euclidean). Would this be correct?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In vector space models for text representation, the vectors are from a high dimension and sparse. When using L2 or L1 norm, there will be a term for every dimension which has a term in either vector. However, for consine similarity, since it is a dot product, there will only be a term when both elements in either vector is non-zero. The sparsity renders the consine similarity a better choice for such comparisons.&lt;/p&gt;&#xA;" OwnerUserId="37809" LastActivityDate="2017-12-22T05:14:38.243" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25902" PostTypeId="1" AcceptedAnswerId="25904" CreationDate="2017-12-22T07:40:41.697" Score="7" ViewCount="1106" Body="&lt;p&gt;I was looking at keras source &lt;a href=&quot;https://github.com/keras-team/keras/blob/85fe6427a50f6def7777413a4bc8b286f7ab8f8d/keras/backend/tensorflow_backend.py#L2753&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which calculates cross entropy loss using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output /= tf.reduce_sum(output,&#xA;                        reduction_indices=len(output.get_shape()) - 1,&#xA;                        keep_dims=True)&#xA;# manual computation of crossentropy&#xA;epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)&#xA;output = tf.clip_by_value(output, epsilon, 1. - epsilon)&#xA;return - tf.reduce_sum(target * tf.log(output),&#xA;                       reduction_indices=len(output.get_shape()) - 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;target&lt;/code&gt; is the truth data, which is 0 or 1, and output is the output of the neural net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it looks like the loss is of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J_{y'} (y) = - \sum_{i} y_{i}' \log (y_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y_i$ is the model output for class $i$, and $y_i'$ is the truth data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean the errors for $y_i' = 0$ do not contribute to the loss?  Why isn't the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;used?&lt;/p&gt;&#xA;" OwnerUserId="43645" LastEditorUserId="29575" LastEditDate="2017-12-22T15:31:51.403" LastActivityDate="2017-12-22T15:31:51.403" Title="Does keras categorical_cross_entropy loss take incorrect classification into account" Tags="|keras|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25903" PostTypeId="2" ParentId="10039" CreationDate="2017-12-22T07:49:09.433" Score="1" Body="&lt;p&gt;SVD is operation of decomposing a matrix $M$ into the matrix product $M=U\Lambda V$, where $U$ and $V$ are unitary matrices ($U^TU=UU^T=I$, etc.), and $\Lambda$ is diagonal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the formal answer is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;No, there are no assumptions about distribution of $M$. It does not to be &lt;em&gt;random&lt;/em&gt; at all, it is just a matrix of any nature.&lt;/li&gt;&#xA;&lt;li&gt;Condition $n&amp;gt;m$, where $(n, m)$ is shape of $M$, is not necessary. Indeed, if you have $n&amp;gt;m$, you can transpose the equation  $M=U\Lambda V$ to get  $M^T=V^T\Lambda U^T$, and it will be also a valid SVD decomposition.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now the less formal part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In data mining SVD has two main applications: for computations (like matrix inversion, least squares, etc.), and for dimensionality reduction (e.g. to compress the user-item matrix in recommender systems). For computations, only algebraic properties of SVD (shown above) matter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For dimensionality reduction, truncated SVD is used: the small elements of $\Lambda$ are discarded, and only $k$ largest are kept. This operation is equivalent to finding a $k$-dimensional hyperplane such that projection of the data ($M$) on this hyperplane is closest (in Euclidian distance) to the original data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the last case, the analyst would like the decomposition to generalize well to unseen data, and this problem can be formulated well in probabilistic language. If we state that $M$ consist of $n$ IID $m$-dimensional random variables, then it appears that SVD works best if they are &lt;strong&gt;joint normal&lt;/strong&gt;. That's because:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. multivariate normal is indeed distributed nearly a hyperplane, and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. for multivariate normal, Euclidean distance is tightly connected with probability density. Therefore truncated SVD (or PCA, which is mathematically identical) can be viewed as maximum likelihood estiomation of multivariate normal distribution with $k$ independent components. For more details, see the &lt;a href=&quot;http://www.di.ens.fr/~fbach/courses/fall2010/Bishop_Tipping_1999_Probabilistic_PCA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article by Bishop&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2017-12-23T10:06:19.833" LastActivityDate="2017-12-23T10:06:19.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25904" PostTypeId="2" ParentId="25902" CreationDate="2017-12-22T09:13:09.057" Score="7" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;Does this mean the errors for &lt;span class=&quot;math-container&quot;&gt;$y_i=0$&lt;/span&gt; do not contribute to the loss?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;That is correct.&lt;/p&gt;&#xA;&lt;p&gt;However, the respective weights that connect to wrong neurons will still have gradients due to the error, and those gradients will be influenced by the size of each incorrect classification. That is due to how softmax works:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\hat{y}_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;(where &lt;span class=&quot;math-container&quot;&gt;$z_i$&lt;/span&gt; is the pre-softmax value of each neuron, a.k.a. the &lt;em&gt;logit&lt;/em&gt;) . . . weights that affect one neuron's pre-transform value affect the post-transform value of all neurons. So those weights will still be adjusted to produce a lower &lt;span class=&quot;math-container&quot;&gt;$z_j$&lt;/span&gt; value for the incorrect neurons during weight updates.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Why isn't the formula&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;used?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;It is not clear why when selecting a single class, that you would care how probability estimates were distributed amongst incorrect classes, or what the benefit would be to drive the incorrect values to be equal. For instance if &lt;span class=&quot;math-container&quot;&gt;$y' = [1, 0, 0, 0]$&lt;/span&gt; then using the suggested formula for &lt;span class=&quot;math-container&quot;&gt;$J_{y'}(y)$&lt;/span&gt; gives ~ 0.67 for &lt;span class=&quot;math-container&quot;&gt;$y = [0.7, 0.1, 0.1, 0.1]$&lt;/span&gt; and ~0.72 for  &lt;span class=&quot;math-container&quot;&gt;$y = [0.73, 0.26, 0.05, 0.05]$&lt;/span&gt;, yet arguably the second result is better.&lt;/p&gt;&#xA;&lt;p&gt;However, you &lt;em&gt;would&lt;/em&gt; use this loss when dealing with non-exclusive classes (where the outputs would use sigmoid as opposed to softmax activation).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-22T09:37:25.230" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="25905" PostTypeId="2" ParentId="10810" CreationDate="2017-12-22T09:28:55.213" Score="4" Body="&lt;p&gt;I've written a generic probabalistic fuzzy matcher in Python which will do a reasonable job of matching any type of data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/robinl/fuzzymatcher&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/robinl/fuzzymatcher&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's in memory, so you probably don't want to use it to match datasets which are above about 100k rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also written a similar project specific to UK addresses, but this assumes you have access to Addressbase Premium.  This one isn't in memory, so has been used against the 100m or so UK addresses. See here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/RobinL/AddressMatcher&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/RobinL/AddressMatcher&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get this going quickly I'd reccommend using &lt;a href=&quot;https://github.com/openvenues/libpostal&quot; rel=&quot;nofollow noreferrer&quot;&gt;libpostal&lt;/a&gt; to normalise your addresses and then feed them into my generic fuzzymatcher (&lt;code&gt;pip install fuzzymatcher&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find usage examples &lt;a href=&quot;https://mybinder.org/v2/gh/RobinL/fuzzymatcher/master?filepath=examples.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.  &lt;/p&gt;&#xA;" OwnerUserId="43649" LastActivityDate="2017-12-22T09:28:55.213" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25907" PostTypeId="1" AcceptedAnswerId="25966" CreationDate="2017-12-22T11:15:52.843" Score="7" ViewCount="3970" Body="&lt;p&gt;I have been looking at feature agglomeration in Python's scikit-learn. According to the user guide, feature agglomeration &quot;applies Hierarchical clustering to group together features that behave similarly&quot;. Does this mean it is testing for interactions between features and groups features that do interact together? What does &quot;behave similarly&quot; mean in this context?&lt;/p&gt;&#xA;" OwnerUserId="38734" LastActivityDate="2018-09-21T06:00:09.817" Title="Feature agglomeration: Is it testing interactions?" Tags="|python|scikit-learn|clustering|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25908" PostTypeId="1" CreationDate="2017-12-22T11:30:45.283" Score="1" ViewCount="1032" Body="&lt;p&gt;I try to construct the following neural network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;img_rows = 63&#xA;img_cols= 494&#xA;img_channels = 1&#xA;&#xA;nb_classes = 12&#xA;nb_positions = 13&#xA;&#xA;conv_filters = 16&#xA;kernel_size = (3, 3)&#xA;pool_size = 2&#xA;time_dense_size = 32&#xA;rnn_size = 512&#xA;minibatch_size = 32&#xA;act = 'relu'&#xA;&#xA;input_shape = (img_channels, img_rows, img_cols)&#xA;&#xA;input_data = Input(name='the_input', shape=input_shape, dtype='float32')&#xA;inner = Conv2D(conv_filters, kernel_size, padding='same', activation=act, kernel_initializer='he_normal', name='conv1', data_format=&quot;channels_first&quot;)(input_data)&#xA;inner = MaxPooling2D(pool_size=(pool_size, pool_size), name='max1', data_format=&quot;channels_first&quot;)(inner)&#xA;inner = Conv2D(conv_filters, kernel_size, padding='same', activation=act, kernel_initializer='he_normal', name='conv2', data_format=&quot;channels_first&quot;)(inner)&#xA;inner = MaxPooling2D(pool_size=(pool_size, pool_size), name='max2', data_format=&quot;channels_first&quot;)(inner)&#xA;&#xA;#&quot;inner&quot; shape here: TensorShape([Dimension(None), Dimension(16),Dimension(15), Dimension(123)])&#xA;&#xA;conv_to_rnn_dims = (img_rows // (pool_size ** 2), (img_cols // (pool_size ** 2)) * conv_filters)&#xA;inner = Reshape(target_shape=conv_to_rnn_dims, name='reshape')(inner)&#xA;inner = Dense(time_dense_size, activation=act, name='dense1')(inner)&#xA;&#xA;gru_1 = GRU(rnn_size, return_sequences=True, kernel_initializer='he_normal', name='gru1')(inner)&#xA;gru_1b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer='he_normal', name='gru1_b')(inner)&#xA;gru1_merged = add([gru_1, gru_1b])&#xA;gru_2 = GRU(rnn_size, return_sequences=True, kernel_initializer='he_normal', name='gru2')(gru1_merged)&#xA;gru_2b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer='he_normal', name='gru2_b')(gru1_merged)&#xA;&#xA;inner = Dense(nb_classes+1, kernel_initializer='he_normal', name='dense2')(concatenate([gru_2, gru_2b]))&#xA;y_pred = Activation('softmax', name='softmax')(inner)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, in case of the two Dense layers I got error messages like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Dimensions must be equal, but are 15 and 32 for 'dense1_12/add' (op: 'Add') with input shapes: [?,15,32], [1,32,1].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I cannot interpret this message; isn't Dense layers' node numbers are independent of the previous layers and free to define? What is not passing here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Addition: the whole error message is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-110-844e7f2bb4a0&amp;gt; in &amp;lt;module&amp;gt;()&#xA;----&amp;gt; 1 inner = Dense(time_dense_size, activation=act, name='dense1')(inner)&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in __call__(self, inputs, **kwargs)&#xA;    601 &#xA;    602             # Actually call the layer, collecting output(s), mask(s), and shape(s).&#xA;--&amp;gt; 603             output = self.call(inputs, **kwargs)&#xA;    604             output_mask = self.compute_mask(inputs, previous_mask)&#xA;    605 &#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/keras/layers/core.pyc in call(self, inputs)&#xA;    843         output = K.dot(inputs, self.kernel)&#xA;    844         if self.use_bias:&#xA;--&amp;gt; 845             output = K.bias_add(output, self.bias)&#xA;    846         if self.activation is not None:&#xA;    847             output = self.activation(output)&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in bias_add(x, bias, data_format)&#xA;   3578         if data_format == 'channels_first':&#xA;   3579             if len(bias_shape) == 1:&#xA;-&amp;gt; 3580                 x += reshape(bias, (1, bias_shape[0], 1))&#xA;   3581             else:&#xA;   3582                 x += reshape(bias, (1, bias_shape[1], bias_shape[0]))&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.pyc in binary_op_wrapper(x, y)&#xA;    819       if not isinstance(y, sparse_tensor.SparseTensor):&#xA;    820         y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name=&quot;y&quot;)&#xA;--&amp;gt; 821       return func(x, y, name=name)&#xA;    822 &#xA;    823   def binary_op_wrapper_sparse(sp_x, y):&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.pyc in add(x, y, name)&#xA;     71     A `Tensor`. Has the same type as `x`.&#xA;     72   &quot;&quot;&quot;&#xA;---&amp;gt; 73   result = _op_def_lib.apply_op(&quot;Add&quot;, x=x, y=y, name=name)&#xA;     74   return result&#xA;     75 &#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)&#xA;    766         op = g.create_op(op_type_name, inputs, output_types, name=scope,&#xA;    767                          input_types=input_types, attrs=attr_protos,&#xA;--&amp;gt; 768                          op_def=op_def)&#xA;    769         if output_structure:&#xA;    770           outputs = op.outputs&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)&#xA;   2336                     original_op=self._default_original_op, op_def=op_def)&#xA;   2337     if compute_shapes:&#xA;-&amp;gt; 2338       set_shapes_for_outputs(ret)&#xA;   2339     self._add_op(ret)&#xA;   2340     self._record_op_seen_by_control_dependencies(ret)&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in set_shapes_for_outputs(op)&#xA;   1717       shape_func = _call_cpp_shape_fn_and_require_op&#xA;   1718 &#xA;-&amp;gt; 1719   shapes = shape_func(op)&#xA;   1720   if shapes is None:&#xA;   1721     raise RuntimeError(&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in call_with_requiring(op)&#xA;   1667 &#xA;   1668   def call_with_requiring(op):&#xA;-&amp;gt; 1669     return call_cpp_shape_fn(op, require_shape_fn=True)&#xA;   1670 &#xA;   1671   _call_cpp_shape_fn_and_require_op = call_with_requiring&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.pyc in call_cpp_shape_fn(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)&#xA;    608     res = _call_cpp_shape_fn_impl(op, input_tensors_needed,&#xA;    609                                   input_tensors_as_shapes_needed,&#xA;--&amp;gt; 610                                   debug_python_shape_fn, require_shape_fn)&#xA;    611     if not isinstance(res, dict):&#xA;    612       # Handles the case where _call_cpp_shape_fn_impl calls unknown_shape(op).&#xA;&#xA;/home/dsshow/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.pyc in _call_cpp_shape_fn_impl(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)&#xA;    674       missing_shape_fn = True&#xA;    675     else:&#xA;--&amp;gt; 676       raise ValueError(err.message)&#xA;    677 &#xA;    678   if missing_shape_fn:&#xA;&#xA;ValueError: Dimensions must be equal, but are 15 and 32 for 'dense1_17/add' (op: 'Add') with input shapes: [?,15,32], [1,32,1].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="21560" LastEditDate="2017-12-27T08:30:14.227" LastActivityDate="2017-12-27T08:30:14.227" Title="Keras dense layer input shape mismatch" Tags="|neural-network|keras|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25910" PostTypeId="1" CreationDate="2017-12-22T13:27:50.460" Score="2" ViewCount="188" Body="&lt;p&gt;I am working on the KDD dataset given in this &lt;a href=&quot;https://www.kaggle.com/c/kddcup2012-track1#Description&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset is related to a typical recommendation systems dataset. So you find an item and information about the item. One of the information given about the Item is its category. Item-Category is a string &lt;code&gt;a.b.c.d&lt;/code&gt;, where the character delimits the categories in the hierarchy &lt;code&gt;.&lt;/code&gt;, ordered in a top-down fashion (i.e., category &lt;code&gt;a&lt;/code&gt; is a parent category of &lt;code&gt;b&lt;/code&gt;, and category &lt;code&gt;b&lt;/code&gt; is a parent category of &lt;code&gt;c&lt;/code&gt;, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to use this information correctly in my feature engineering. For example, the simplest information I can derive is for each Item I can estimate the topmost category to which it belongs. However, to go beyond this and use the subcategory information, how can I model this feature for a linear regression? &lt;/p&gt;&#xA;" OwnerUserId="43660" LastEditorUserId="29575" LastEditDate="2017-12-22T15:33:45.357" LastActivityDate="2017-12-22T15:33:45.357" Title="Feature engineering for hierarchical data" Tags="|feature-extraction|feature-engineering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25911" PostTypeId="1" CreationDate="2017-12-22T14:08:11.507" Score="1" ViewCount="34" Body="&lt;p&gt;I am working on a dataset which contains more than 80 features, and thousands of instances. Among those features there are some nominal ones, such as IP SOURCE, IP DESTINATION, Flow ID, which don't have any signification for my machine learning model. My question is, should I remove those features manually, or do I have to replace their values with numerical ones?&lt;/p&gt;&#xA;" OwnerUserId="33349" LastEditorUserId="29575" LastEditDate="2017-12-22T19:41:52.737" LastActivityDate="2017-12-22T19:43:11.110" Title="Should I modify dataset?" Tags="|dataset|feature-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25913" PostTypeId="1" CreationDate="2017-12-22T18:46:45.913" Score="2" ViewCount="23" Body="&lt;p&gt;For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the metrics, it looks like there would be a better precision/recall balance for my needs with a higher score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I use a split of 0.6 score, which results in a 0.8 precision and 0.9 recall, how do I communicate the overall confidence of future sample scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if a sample returns a predicted probability of 1 vs one that returns a 0.75. Based on the threshold chosen, both are going to be considered as part of the positive class. However, it's estimated that only 80% of the time that will be true. On top of that, the 1 and 0.75 scores seem to indicate another level of confidence for the model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This feels like Bayes should be used here to summarize the scores for interpretation. But, I am not sure. So, given the model has an estimated performance based on a score threshold, and given some scores, how do you give an overall percentage of the samples actually being in the positive class? And how would one communicate this to non-technical people?&lt;/p&gt;&#xA;" OwnerUserId="37776" LastActivityDate="2017-12-22T18:46:45.913" Title="Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?" Tags="|classification|probability|model-evaluations|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25914" PostTypeId="2" ParentId="25911" CreationDate="2017-12-22T19:30:43.687" Score="1" Body="&lt;p&gt;You don't need to remove your features from your data set you may just drop the features which you don't want considered as features or lables.  Dropping a feature or label works by excluding the columns while fitting the data into the training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/z4SYs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/z4SYs.png&quot; alt=&quot;i used it in my project shown above&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43438" LastEditorUserId="29575" LastEditDate="2017-12-22T19:43:11.110" LastActivityDate="2017-12-22T19:43:11.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25915" PostTypeId="1" CreationDate="2017-12-22T19:40:10.270" Score="1" ViewCount="1057" Body="&lt;p&gt;I am trying the below code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array(df.drop(['label'], 1))&#xA;X = preprocessing.scale(X)&#xA;X = X[:-forecast_out]&#xA;df.dropna(inplace=True)&#xA;y = np.array(df['label'])&#xA;X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)&#xA;X = X.reshape(X.shape[1:])&#xA;X = X.tranpose()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and getting error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Found input variables with inconsistent numbers of samples: [3292, 3326]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43668" LastEditorUserId="29575" LastEditDate="2017-12-22T19:49:07.173" LastActivityDate="2017-12-22T19:49:07.173" Title="Python 3: Please help.. Found input variables with inconsistent numbers of samples: [3292, 3326]" Tags="|linear-regression|model-selection|" AnswerCount="0" CommentCount="3" ClosedDate="2017-12-27T04:39:30.510" ContentLicense="CC BY-SA 3.0" />
  <row Id="25916" PostTypeId="1" AcceptedAnswerId="25968" CreationDate="2017-12-22T22:26:49.267" Score="1" ViewCount="469" Body="&lt;p&gt;I am working on a credit risk binary classification problem. The classes are GoodPayers and BadPayers.  The training set has variables/features that contains: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;DemoGraphics Data  such as - Age, Education, Loan Amount, Interest Rate&lt;/li&gt;&#xA;&lt;li&gt;Behavioral Data such as - Payment in Month1, Payment in Month2, Payment in Month3, Payment delay in Month1, Payment delay in Month2. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The 10-fold cross-validation has 0.82 AUC on this set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the unseen data just contains the 'Demographics Data' and does not have 'Behavioral Data' of Payment.  How do we deploy/test the model based on DemoGraphics Dataset only? &lt;/p&gt;&#xA;" OwnerUserId="4886" LastEditorUserId="4886" LastEditDate="2017-12-23T17:46:45.093" LastActivityDate="2017-12-24T18:06:11.913" Title="Evaluating machine learning model with missing features" Tags="|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25917" PostTypeId="2" ParentId="25838" CreationDate="2017-12-22T23:07:48.517" Score="2" Body="&lt;p&gt;&lt;strong&gt;Main Idea:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main idea is the you could measure how good is the output and the input of your model simply representing it by some mathematical object, for example a vector in a high dimensional space and then using some error function to measure the fitness (how good your model is performing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A possible formalization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The scenario your describing fits in a more general optimization function, the most general optimization scenario is;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ min_{x \in X} f(x) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $X$ is the domain. In the case of image processing and computer vision $x$ is a matrix of pixels, this matrix of pixels can be represented by a vector $x \in \mathbb{R}^{m \times n}$. In the case you describe you could also represent the input (painting) as a vector $y\in \mathbb{R}^{m \times n}$. &#xA;Then $f$ coul be &lt;a href=&quot;https://en.wikipedia.org/wiki/Root-mean-square_deviation&quot; rel=&quot;nofollow noreferrer&quot;&gt;RSME&lt;/a&gt; between some $x$ and $y$, i.e. $f(x) = {\| y - x \|}_2$. Notice that $y$ is given and it does not change. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the key question here is how to represent a stroke, or more formally a function the values of $x$. A very basic idea can be represent the stroke by the following tuple $(o, e, t, c)$ where $o$ and $e$ are two dimensional points, the origin and the end of the stroke; $t$ is the thickness of the stroke and $c$ is the color of the stroke. Let $S$ be the space of all the strokes, assuming the thickness and colors are finite, S is finite. The possible pairs of $o$ and $e$ are finite namely all the pairs of possible points in $m \times n$. Now let $S^k$ be the composition of $S$ with itself $k$ times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally the problem can be represented the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\min_{s \in S^k}{\| y - t(s, x) \|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $t$ is the function that applies $s$, that is a number of different strokes to an empty canvas $x$, by empty canvas I mean a matrix full of zeros. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summarizing the ideas expose above are a possible formalization of your problem, of course is not the only one. The main issue is that the robot does not &quot;learn&quot; is simply computes the optimal set of strokes to apply.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other ideas&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another possible approach is to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Reinforcement_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;reinforcement learning&lt;/a&gt;, this learning paradigm does not needs labeled training samples, it relies in a reward function $R$ to guide the learning process. Also if defines a set of states and actions the agent can be or do. The states could be the strokes and the actions add another stroke, the reward function can be a variation of the error function. Also I am far from being an expert in reinforcement learning I guess is worth looking into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A proven approach&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem you describe has been demonstrated in real life. See this &lt;a href=&quot;https://pdfs.semanticscholar.org/1592/8cb593865a647586719a6539b29692fe2e4c.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;: Feedback-guided Stroke Placement for a Painting Machine. The paper demonstrates a system that describes your scenario, also see this &lt;a href=&quot;http://graphics.uni-konstanz.de/publikationen/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;site&lt;/a&gt; for more work of the authors or &lt;a href=&quot;http://graphics.uni-konstanz.de/eDavid/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for the site of the project. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-22T23:46:15.513" LastActivityDate="2017-12-22T23:46:15.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25918" PostTypeId="2" ParentId="16347" CreationDate="2017-12-23T00:58:58.427" Score="2" Body="&lt;p&gt;Have you seen this &lt;a href=&quot;https://www.google.ca/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwi7-L7h9p7YAhVC5GMKHdxLBpoQFggpMAA&amp;amp;url=http%3A%2F%2Fwww.cs.uccs.edu%2F~jkalita%2Fwork%2FStudentResearch%2FRajagopalSureshMSProject2016.pdf&amp;amp;usg=AOvVaw3UB2HaOPohzGBqrT-Ewfvg&quot; rel=&quot;nofollow noreferrer&quot;&gt;thesis&lt;/a&gt;? The author uses DTW to identify a collection of chart patterns. &lt;/p&gt;&#xA;" OwnerUserId="43675" LastActivityDate="2017-12-23T00:58:58.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25921" PostTypeId="1" AcceptedAnswerId="25931" CreationDate="2017-12-23T06:42:19.303" Score="1" ViewCount="1915" Body="&lt;p&gt;I'm implementing prediction system for young cricketers in ODI format using Naive Bayes classifier. The output of the system is to predict whether the young player is rising star or not. I have collected data from statsguru API of espncricinfo but I'm getting only about 300 records of players from ODI. Is it very small for training dataset?&lt;/p&gt;&#xA;" OwnerUserId="43679" LastActivityDate="2017-12-24T07:13:24.927" Title="Small amount of training data set for naive Bayes classifier for binary classification" Tags="|machine-learning|classification|supervised-learning|naive-bayes-classifier|bayesian-networks|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25922" PostTypeId="1" CreationDate="2017-12-23T07:20:56.230" Score="0" ViewCount="85" Body="&lt;p&gt;My data set contains the pickup latitude, pickup longitude of the customer and drivers current location latitude and longitude.&#xA;My aim is to distribute a set of  M or M-x (in the case where shortage of drivers)&#xA;for N customers. The driver has to be chosen with less distance and less ETA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any solution in data science with python or R?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;customer_id  pickuplocationaddress  pickuplatd  pickuplong  driverlatd  driverlong  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43680" LastEditorUserId="43680" LastEditDate="2018-01-09T10:29:17.493" LastActivityDate="2018-01-09T10:29:17.493" Title="allocation to customers" Tags="|python|r|algorithms|" AnswerCount="1" CommentCount="3" ClosedDate="2017-12-27T04:39:53.483" ContentLicense="CC BY-SA 3.0" />
  <row Id="25924" PostTypeId="1" AcceptedAnswerId="25935" CreationDate="2017-12-23T08:03:03.610" Score="11" ViewCount="34535" Body="&lt;p&gt;Since interpolate and fillna method does the same work of filling na values. What is the basic difference between the two. What is the significance of having these two different methods?? Can anyone explain me in layman terms. I already visited through the official documentation and wanted to know the difference&lt;/p&gt;&#xA;" OwnerUserId="43681" LastEditorUserId="43681" LastEditDate="2017-12-24T14:20:53.273" LastActivityDate="2018-01-23T15:24:23.313" Title="Difference between interpolate() and fillna() in pandas" Tags="|python|bigdata|pandas|jupyter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25925" PostTypeId="1" AcceptedAnswerId="25930" CreationDate="2017-12-23T08:09:38.067" Score="3" ViewCount="389" Body="&lt;p&gt;I have been trying to apply a simple neural network using keras to predict a sequence of numbers and the rule is if the input integer is odd it should be 4 and if its even it should be 2. Yet the neural network gets stuck at a 60% accuracy rate. Anyone know a solution to this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import Dropout&#xA;&#xA;from keras.layers.normalization import BatchNormalization&#xA;&#xA;from sklearn.model_selection import cross_val_score&#xA;from keras.wrappers.scikit_learn import KerasClassifier&#xA;&#xA;import numpy as np&#xA;&#xA;&#xA;def gen(x):&#xA;    if (x%2==0):&#xA;        return 2;&#xA;    else:&#xA;        return 4;&#xA;&#xA;&#xA;a = []           &#xA;for i in range(1,100001):&#xA;    a.append([i,gen(i)])&#xA;&#xA;a = np.array(a)&#xA;&#xA;&#xA;x = a[:,0:1]&#xA;y = a[:,1:2]&#xA;&#xA;&#xA;def MakeClassifier():&#xA;    network_classifier = Sequential()&#xA;    network_classifier.add(Dense(units=2,kernel_initializer=&quot;uniform&quot;,activation=&quot;relu&quot;,input_dim=1)) #Hidden Layer1 taking into account number of inputs(independant variables(x)&#xA;    network_classifier.add(BatchNormalization())&#xA;    network_classifier.add(Dense(units=1,kernel_initializer=&quot;uniform&quot;,activation=&quot;sigmoid&quot;))#OutPutLayer&#xA;&#xA;    network_classifier.compile(optimizer=&quot;adam&quot;,loss=&quot;binary_crossentropy&quot;,metrics=[&quot;accuracy&quot;])#If multicategorical then categorical_crossentropy&#xA;    return network_classifier&#xA;&#xA;classifier = KerasClassifier(build_fn= MakeClassifier , batch_size = 10 , epochs = 1000)&#xA;&#xA;classifier.fit(x,y,epochs=100,batch_size=1000)&#xA;&#xA;print(classifier.predict([[6],[7]])) #Should Predict 2 and 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43677" LastEditorUserId="28175" LastEditDate="2018-01-17T16:27:31.830" LastActivityDate="2018-01-17T16:27:31.830" Title="ANN on Pattern Recognition" Tags="|machine-learning|neural-network|classification|deep-learning|keras|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25927" PostTypeId="2" ParentId="11485" CreationDate="2017-12-23T10:45:39.030" Score="0" Body="&lt;p&gt;Simply put, nstart will create multiple configurations and will show the best one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, nstart = 25 , will create 25 initial configurations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use nstart argument to try multiple random initial values.&lt;/p&gt;&#xA;" OwnerUserId="37657" LastActivityDate="2017-12-23T10:45:39.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25928" PostTypeId="2" ParentId="22441" CreationDate="2017-12-23T12:36:23.120" Score="2" Body="&lt;p&gt;Here's an approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Get the lower dimensional embedding of the training data using t-SNE model. &lt;/li&gt;&#xA;&lt;li&gt;Train a neural network or any other non-linear method, for predicting the t-SNE embedding of a data point. This will essentially be a regression problem.&lt;/li&gt;&#xA;&lt;li&gt;Use the model trained in step 2 to first predict the t-SNE embedding of a test data point and then assign it to a class using kNN. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This ensures that there is no data leakage between your training and test set. I believe this approach is a hacky way of bringing t-SNE into the binary classification picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do note that t-SNE was mainly intended for visualization of high dimensional data points and not to extract good features for a classification model. The fact that you could observe a clear separation between classes using the t-SNE visualisation implies that the data can be easily modeled as a binary classification task using a non-linear classification algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were you, I would consider using ELMs, SVMs with non-linear kernels or good old Logistic Regression with regularisation.&lt;/p&gt;&#xA;" OwnerUserId="43689" LastEditorUserId="29575" LastEditDate="2017-12-23T16:05:05.717" LastActivityDate="2017-12-23T16:05:05.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25929" PostTypeId="1" CreationDate="2017-12-23T12:40:52.847" Score="1" ViewCount="1395" Body="&lt;p&gt;Suppose that I have an input image with 224 * 224 * 3 dimensions. I pass it through a convolution layer with 64 filters and same convolution operation. Suppose that I want to reconstruct the original image using the output of the convolution layer which has the size 224 * 224 * 64. I have the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Should I use deconvolution? If so, how is the arrangement of deconvolution layer (number of filters and the value of weights. Also when should the activation be applied)?&lt;/li&gt;&#xA;&lt;li&gt;Are the number of filters and weights in forward pass equal to the backward pass?&lt;/li&gt;&#xA;&lt;li&gt;Is there any technique other than deconvolution?&lt;/li&gt;&#xA;&lt;li&gt;Is there any available &lt;code&gt;Keras&lt;/code&gt; code for my need?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've seen &lt;a href=&quot;https://datascience.stackexchange.com/q/6107/28175&quot;&gt;here&lt;/a&gt; and also &lt;a href=&quot;https://github.com/vdumoulin/conv_arithmetic&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; but they represent a high level abstraction and don't contain appropriate detailed answer.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-23T16:29:39.647" Title="Using deconvolution in practice" Tags="|machine-learning|deep-learning|keras|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25930" PostTypeId="2" ParentId="25925" CreationDate="2017-12-23T13:20:52.310" Score="2" Body="&lt;p&gt;There are two possible reasons for this result:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Low number of training examples&lt;/li&gt;&#xA;&lt;li&gt;Using dense layers without batch normalization&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You have a relatively deep network and your training set size is small. In such cases if you run your model too many times you will certainly overfit the training data. The reason is that whenever you have a powerful model that can learn complicated functions and if you provide low number of training examples, it has capability to fit the data and not learn it. Suppose that you have a fitting problem in calculus and you have 4 points and you also have polynomials with degree of four. In such cases you can fit the data exactly without any error but the point is that you are fitting it, not learning it. In your case your model is powerful and it tries to fit the data, not to learn it. The cure is to provide enough training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason that your model can not learn is that you are stacking dense layers without batch normalization. each time you update the weights, the output of the deep layers change and this cause the covariat shift. To avoid this problem use batch normalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a solution for you I recommend you to putting just two hidden layers and manipulate the number of units in those layers and provide more training examples. Your task can easily be learned by a two hidden-layer-network.&#xA;&lt;hr /&gt;&#xA;After struggling for about one day finally I want to express my opinion about the problem. Depending on the problems, they may be solved using machine-learning or other techniques. Machine learning problems are those which you can not define an appropriate function to map the input to output or it may be so much hard to do so. Whenever you have the function, it can be used as the hypothesis, the final goal of machine learning algorithms. I tried hard and put so much time on this code and I get the same result. Nothing is going to be learned at all. I have been trying for about one day and still no progress has been seen. To explain the reason, the data is so much hard to be learned! for imagining how difficult it is, I recommend you to write numbers from one to ten in a straight line and put a line between consecutive numbers. The numbers are endless, so you will have no generalization because the boundaries that are going to be found will work just for the two neighbor numbers. This means that if you use the current features, you can not separate, learn, your data. I tried to do, somehow, feature engineering and used the following code to solve the problem: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;import numpy as np&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import Dropout&#xA;from keras.layers.normalization import BatchNormalization &#xA;from keras import regularizers &#xA;from keras.optimizers import Adam&#xA;&#xA;def gen(x):&#xA;    if (x % 2 == 0):&#xA;        return 0; # represents 2&#xA;    else:&#xA;        return 1; # represents 4&#xA;&#xA;&#xA;a = []           &#xA;for i in range(1,100001):&#xA;    temp = np.random.randint(0, 10000000)&#xA;    a.append([temp, temp ** 2, temp ** 3, temp ** 4, gen(temp)])&#xA;&#xA;a = np.array(a)&#xA;&#xA;&#xA;x = a[:, 0: a.shape[1] - 1]&#xA;y = a[:, a.shape[1] - 1:]&#xA;&#xA;mean_of_x = np.mean(x, axis = 0, keepdims = True)&#xA;std_of_x = np.std(np.float64(x), axis = 0, keepdims = True)&#xA;x = (x - mean_of_x) / std_of_x&#xA;&#xA;n_classes = 2&#xA;y = keras.utils.to_categorical(y, 2)&#xA;&#xA;percentage = 95 / 100&#xA;limit = int(percentage * x.shape[0])&#xA;&#xA;x_train = x[: limit, :]&#xA;y_train = y[: limit, :]&#xA;&#xA;x_test = x[limit: , :]&#xA;y_test = y[limit: , :]&#xA;&#xA;x_train.shape&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(1000, activation='relu', input_shape=(x_train.shape[1],)))&#xA;model.add(BatchNormalization())&#xA;# model.add(Dropout(0.5))&#xA;model.add(Dense(1000, activation='relu'))&#xA;model.add(BatchNormalization())&#xA;# model.add(Dropout(0.5))&#xA;&#xA;model.add(Dense(n_classes, activation='softmax'))&#xA;&#xA;model.summary()&#xA;&#xA;model.compile(loss = 'categorical_crossentropy', optimizer = keras.optimizers.Adam(lr = 0.0001, decay = 1.5), metrics=['accuracy'])&#xA;&#xA;model.fit(x_train, y_train, batch_size = 128, epochs = 2000, verbose = 1, validation_data=(x_test, y_test), shuffle = True,&#xA;          class_weight = {0: 10, 1: 1})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the above code uses high order polynomial. Surprisingly, no progress was seen here too. this model has the mentioned problem for the previous feature in higher dimensions. That's why the learning does not happen here too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;the point here is that although you can not learn using the current feature, number itself, or its high order polynomials, you already have a solution for the problem. Instead of passing the number itself to the learning problem, pass the modulus two of the current number. This feature is so much easy to be learned. you may need just one unit. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-26T16:23:41.773" LastActivityDate="2017-12-26T16:23:41.773" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25931" PostTypeId="2" ParentId="25921" CreationDate="2017-12-23T13:36:30.867" Score="1" Body="&lt;p&gt;Actually in machine learning more data equals more accuracy but as you mentioned in the question you had 300 sample dataset.So, the classifier has the little room to decide whom it should select but if you have less number of classes and features you may get better results.When I'm doing my project based on sensor data I used just about 100 sample records from a sensor and it actually predicted very well and got the accuracy of 77 and it has grown once I increased the dataset.but don't train it with the noise like outliers and unwanted features they affect your accuracy a lot&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2017-12-23T13:36:30.867" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25932" PostTypeId="1" AcceptedAnswerId="25934" CreationDate="2017-12-23T14:10:59.707" Score="4" ViewCount="769" Body="&lt;p&gt;Let’s suppose I wanted to classify some input as one of three categories using a simple neural network.&lt;/p&gt;&#xA;&lt;p&gt;The output of my network are three columns (one for each possible category I assume) with values between 0 and 1. Moreover, the single rows add up to precisely one when adding the three columns together. Is is possible to interpret the output as the probability of my input belonging to each single category?&lt;/p&gt;&#xA;" OwnerUserId="43474" LastEditorUserId="85045" LastEditDate="2021-02-26T06:29:59.667" LastActivityDate="2021-02-26T06:29:59.667" Title="Multiclass classification with Neural Networks" Tags="|machine-learning|neural-network|deep-learning|r|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25933" PostTypeId="1" AcceptedAnswerId="26010" CreationDate="2017-12-23T14:24:58.980" Score="0" ViewCount="8825" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aWlvb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aWlvb.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm getting the above error while I'm extracting the sensor data from serial port and giving it for prediction.Tried reshape and string functions but every trial throws back me same error.i'm using mlp neuralnetwork in my model.Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2017-12-26T07:08:16.980" Title="TypeError: Cannot cast array data from dtype('float64') to dtype('S32') according to the rule 'safe'" Tags="|classification|scikit-learn|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25934" PostTypeId="2" ParentId="25932" CreationDate="2017-12-23T14:32:23.320" Score="6" Body="&lt;p&gt;Indeed, this is the standard interpretation of continuous classifier outputs, not only for neural networks, but for the more general case called &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/&quot; rel=&quot;noreferrer&quot;&gt;Softmax Regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, provided that you have used softmax activation on the final layer (in order, among other things, to ensure that your outputs indeed sum up to 1), you can interpret the continuous outputs as the respective probabilities of a particular data sample belonging to each one of your classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also the discussion in this (rather unfortunately titled) discussion at SO: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/1523420/how-to-convert-the-output-of-an-artificial-neural-network-into-probabilities&quot;&gt;How to convert the output of an artificial neural network into probabilities?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43000" LastActivityDate="2017-12-23T14:32:23.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25935" PostTypeId="2" ParentId="25924" CreationDate="2017-12-23T15:24:27.303" Score="15" Body="&lt;p&gt;&lt;code&gt;fillna&lt;/code&gt; fills the &lt;code&gt;NaN&lt;/code&gt; values with a given number with which you want to substitute. It gives you an option to fill according to the index of rows of a &lt;code&gt;pd.DataFrame&lt;/code&gt; or on the name of the columns in the form of a python &lt;code&gt;dict&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But &lt;code&gt;interpolate&lt;/code&gt; is a god in filling. It gives you the flexibility to fill the missing values with many kinds of interpolations between the values like linear (which &lt;code&gt;fillna&lt;/code&gt; does not provide) in the example provided below and many more interpolations possible. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; import pandas as pd, numpy as np&#xA;&amp;gt;&amp;gt; df = pd.Series([1, np.nan, np.nan, 3])&#xA;&amp;gt;&amp;gt; df.interpolate()&#xA;0    1.000000&#xA;1    1.666667&#xA;2    2.333333&#xA;3    3.000000&#xA;dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pandas documentation on &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html&quot; rel=&quot;noreferrer&quot;&gt;fillna&lt;/a&gt; and &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.16.2/generated/pandas.DataFrame.interpolate.html&quot; rel=&quot;noreferrer&quot;&gt;interpolate&lt;/a&gt; is very clear on this.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2017-12-23T15:24:27.303" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25937" PostTypeId="2" ParentId="25929" CreationDate="2017-12-23T16:29:39.647" Score="1" Body="&lt;p&gt;Since what you are describing sounds like some sort of auto-encoder, you can use this blog on how to use &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;convolution layers in Keras&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the logic is straightforward: If you wish to reconstruct your input, you need to your output to have the same spatial dimensions and the same number of channels as your input. As in most neural network architectures, you will gradually decrease the spatial dimensions of your input and increase the number of filters, and if you wish to reconstruct your input you will use the same concept of layers, but in reverse order, using deconvolution layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to whether you need to use deconvolution or what is the optimal number of channels, layers and symmetry is entirely up to what you are trying to accomplish and the specific details of your problem.&lt;/p&gt;&#xA;" OwnerUserId="43526" LastActivityDate="2017-12-23T16:29:39.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25938" PostTypeId="1" AcceptedAnswerId="25942" CreationDate="2017-12-23T17:08:59.973" Score="4" ViewCount="1018" Body="&lt;p&gt;I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'? &lt;/p&gt;&#xA;" OwnerUserId="42945" LastEditorUserId="28175" LastEditDate="2017-12-23T20:12:12.920" LastActivityDate="2017-12-23T20:12:12.920" Title="Is there a way to set a different activation function for each hidden unit in one layer in keras?" Tags="|machine-learning|neural-network|deep-learning|keras|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25941" PostTypeId="1" AcceptedAnswerId="25951" CreationDate="2017-12-23T18:19:15.633" Score="2" ViewCount="187" Body="&lt;p&gt;Powerful techniques are sometimes rediscovered by various disciplines at different points in time, because the particular scientific fields do not overlap or interact.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A couple of ML researchers have pointed out, that the back-propagation (&lt;a href=&quot;https://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rumelhart et al, 1986&lt;/a&gt;) algorithm (a solution to credit-assignment) has been rediscovered by many fields.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In what other fields is backprop used? Under which names and in which context?&lt;/p&gt;&#xA;" OwnerUserId="14457" LastEditorUserId="14457" LastEditDate="2017-12-23T18:24:36.180" LastActivityDate="2017-12-24T00:42:54.313" Title="Backpropagation in other fields" Tags="|backpropagation|history|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-27T04:36:50.967" ContentLicense="CC BY-SA 3.0" />
  <row Id="25942" PostTypeId="2" ParentId="25938" CreationDate="2017-12-23T18:46:57.753" Score="2" Body="&lt;p&gt;If I get the point, you can use a similar code like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import merge, Convolution2D, MaxPooling2D, Input&#xA;&#xA;input = Input(shape=(256, 256, 3))&#xA;&#xA;seq1 = Dense(1, activation = 'relu')(input)&#xA;seq2 = Dense(1, activation = 'sigmoid')(input)&#xA;seq3 = Dense(1, activation = 'tanh')(input)&#xA;&#xA;acum = merge([seq1, seq2, seq3], mode='concat', concat_axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Depending on your task, specify &lt;code&gt;concat_axis&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-23T18:46:57.753" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="25943" PostTypeId="1" AcceptedAnswerId="25950" CreationDate="2017-12-23T18:57:57.937" Score="3" ViewCount="2728" Body="&lt;p&gt;I want to use a pre-trained &lt;a href=&quot;https://keras.io/applications/#vgg16&quot; rel=&quot;nofollow noreferrer&quot;&gt;VGG16&lt;/a&gt; in keras. My question is simple. Should I normalize the input image before predicting its label?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-24T03:56:59.390" Title="Should the input data be normalized using keras pre-trained models" Tags="|machine-learning|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25944" PostTypeId="1" CreationDate="2017-12-23T20:54:52.300" Score="3" ViewCount="154" Body="&lt;p&gt;I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. &#xA;For example: I have a dataset which contains information about a product. This can be category, brand_name etc. Let's assume there are information missing of brand name for a specific product(4261 rows / 10 000). How can this information be predicted based on the information I have in my dataset?&lt;/p&gt;&#xA;" OwnerUserId="43704" LastEditorUserId="6433" LastEditDate="2017-12-25T11:47:36.147" LastActivityDate="2017-12-25T11:47:36.147" Title="Define a Model for predicting missing values in Data Set" Tags="|classification|r|data-imputation|" AnswerCount="1" CommentCount="0" ClosedDate="2017-12-27T04:38:58.247" ContentLicense="CC BY-SA 3.0" />
  <row Id="25946" PostTypeId="1" AcceptedAnswerId="25947" CreationDate="2017-12-23T21:53:29.453" Score="0" ViewCount="2113" Body="&lt;p&gt;I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I troubleshoot these kind of CUDA driver issues?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:&#xA;name: GeForce GTX 1080 Ti&#xA;major: 6 minor: 1 memoryClockRate (GHz) 1.6575&#xA;pciBusID 0000:01:00.0&#xA;Total memory: 11.00GiB&#xA;Free memory: 10.71GiB&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0&#xA;2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y&#xA;2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -&gt; (device&#xA;: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro&#xA;m device: CUDA_ERROR_OUT_OF_MEMORY&#xA;2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from&#xA;device: CUDA_ERROR_OUT_OF_MEMORY&#xA;2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf&#xA;low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from&#xA;device: CUDA_ERROR_OUT_OF_MEMORY&#xA;built graph&lt;/p&gt;&#xA;" OwnerUserId="43705" LastActivityDate="2017-12-23T22:11:43.727" Title="CUDA_ERROR_OUT_OF_MEMORY" Tags="|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25947" PostTypeId="2" ParentId="25946" CreationDate="2017-12-23T22:11:43.727" Score="0" Body="&lt;p&gt;Did you try to start your training with a smaller data set and that worked?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about using generators to gradually input your data? For me that worked quite well (in Keras : model.fit_generator(...))&lt;/p&gt;&#xA;" OwnerUserId="37808" LastActivityDate="2017-12-23T22:11:43.727" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25948" PostTypeId="1" CreationDate="2017-12-23T23:51:32.320" Score="3" ViewCount="1481" Body="&lt;p&gt;From other posts (see &lt;a href=&quot;https://datascience.stackexchange.com/questions/16342/unbalanced-multiclass-data-with-xgboost/18823#18823&quot;&gt;Unbalanced multiclass data with XGBoost&lt;/a&gt;) and the documentation, &lt;code&gt;scale_pos_weight&lt;/code&gt; in XGBoost appears to balance positive and negative cases, which seems to apply only to classification. However, it also appears to be an option in XGBRegressor (see  &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;). Before I dive into the source code, can someone explain what this does for regression? &lt;/p&gt;&#xA;" OwnerUserId="43706" LastActivityDate="2020-02-21T19:00:50.090" Title="What does xgb's scale_pos_weight parameter do for regression?" Tags="|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25950" PostTypeId="2" ParentId="25943" CreationDate="2017-12-24T00:15:38.550" Score="5" Body="&lt;p&gt;According to &lt;a href=&quot;https://www.arxiv-vanity.com/papers/1409.1556/&quot; rel=&quot;noreferrer&quot;&gt;Very Deep Convolutional Networks for Large-Scale Image Recognition&lt;/a&gt;, which is the paper that first presented VGG: &amp;quot;&lt;em&gt;...The only pre-processing we do is subtracting the mean RGB value, computed on the training set, from each pixel.&lt;/em&gt;&amp;quot;- (Karen Simonyan, Andrew Zisserman).&lt;/p&gt;&#xA;&lt;p&gt;I am certain that Keras provides a preprocess function based on the above mentioned principle.&lt;/p&gt;&#xA;&lt;p&gt;Checkout: &lt;code&gt;tensorflow.keras.applications.vgg16.preprocess_input&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;If it is in the TF version, its certainly in the other version.&lt;/p&gt;&#xA;" OwnerUserId="43707" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-24T00:15:38.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25951" PostTypeId="2" ParentId="25941" CreationDate="2017-12-24T00:42:54.313" Score="3" Body="&lt;p&gt;Beyond its use in deep learning, backpropagation has been used in many other areas, ranging from weather forecasting to analyzing numerical stability. In fact, the algorithm has been reinvented several times in different fields. The general, application independent, name is &quot;reverse-mode differentiation&quot; &lt;a href=&quot;http://colah.github.io/posts/2015-08-Backprop/&quot; rel=&quot;nofollow noreferrer&quot;&gt;[3]&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The modern version of backpropagation, also known as automatic differentiation, was first published by Seppo Linnainmaa &lt;a href=&quot;http://people.idsia.ch/~juergen/who-invented-backpropagation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1]&lt;/a&gt; in 1970. He used it as a tool for estimating the effects of arithmetic rounding errors on the results of complex expressions &lt;a href=&quot;https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gerardi Ostrowski discovered and used it some five years earlier in the context of certain process models in chemical engineering &lt;a href=&quot;https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the sixties Hachtel et al. considered the optimization of electronic&#xA;circuits using the costate equation of initial value problems and its discretizations to compute gradients in the reverse mode for explicitly time-dependent problems &lt;a href=&quot;https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Others researchers that discovered it include Bernt Speelpenning. He arrived at the reverse mode via compiler optimization when asked to automatically generate efficient codes for Jacobians of stiff ODEs &lt;a href=&quot;https://www.math.uni-bielefeld.de/documenta/vol-ismp/52_griewank-andreas-b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More info can be found in: &lt;a href=&quot;https://arxiv.org/pdf/1404.7828.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning in Neural Networks: An Overview &lt;/a&gt; and in &lt;a href=&quot;http://people.idsia.ch/~juergen/who-invented-backpropagation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Who invented backpropagation?&lt;/a&gt; and references therein. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Who invented backpropagation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Who Invented the Reverse Mode of Differentiation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] Calculus on Computational Graphs: Backpropagation&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-24T00:42:54.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25952" PostTypeId="2" ParentId="25922" CreationDate="2017-12-24T01:11:10.360" Score="1" Body="&lt;p&gt;This problem is an instance of the more general &lt;a href=&quot;https://en.wikipedia.org/wiki/Assignment_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;assignment problem&lt;/a&gt;. In the assignment problem you have:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The assigment problem has a number of agents and a number of tasks. Any agent can be assigned to perform any task, incurring some cost that may vary depending on the agent-task assignment. It is required to perform all tasks by assigning exactly one agent to each task and exactly one task to each agent in such a way that the total cost of the assignment is minimized.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In your description the agents are drivers and the tasks are &quot;driving to the pickup location&quot;, the cost can be either the eta or the distance of the driver's location to the pickup location. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of algorithms for solving this problem, one of the most famous is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hungarian_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hungarian algorithm&lt;/a&gt; also known as Kuhn-Munkres algorithm.  There is an implementation of the algorithm in the &lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy library&lt;/a&gt; (this is in Python), under the name of linear_sum_assignment, the documentation is &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.linear_sum_assignment.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-24T01:11:10.360" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25953" PostTypeId="2" ParentId="25944" CreationDate="2017-12-24T01:19:54.410" Score="0" Body="&lt;p&gt;You could try a classification approach like random forests for categorical data.&#xA;For example in your brand name assumption, Take your existing data and treat the brand name as the response variable. So when you give the values of the other features in the entry where the brand name is missing it should try to predict what the brand name is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I am suggesting random forests is that it uses an ensemble approach. Please explore the working of Random Forests if you don't know the working behind it.  &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-12-24T01:19:54.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25955" PostTypeId="2" ParentId="25921" CreationDate="2017-12-24T07:13:24.927" Score="1" Body="&lt;p&gt;I think 300 is a good enough size. Try doing split validation and see what kind of results you are getting. I implemented a naive bayes classifier with a dataset of 100 rows and the results were not too bad. My application was text classification but try for your data and let's see how the accuracy is. Of course you will need training data, so you may have to create some data rows first by classifying them yourself.&lt;/p&gt;&#xA;" OwnerUserId="17047" LastActivityDate="2017-12-24T07:13:24.927" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="25957" PostTypeId="1" CreationDate="2017-12-24T09:41:33.190" Score="0" ViewCount="164" Body="&lt;p&gt;I am stuck in the &lt;a href=&quot;http://tflearn.org/tutorials/quickstart.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tflearn quickstart guide&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried tit and it works perfectly with accuracy around 78%, but the problem is I don't understand why the &quot;labels&quot; consist of two &quot;survived&quot; columns that are the opposite of each other. I've trained the same data but used only single &quot;survived&quot; column in the labels or the output, but with the same code I just get around 36% with 0 loss or even NaN loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why is there two columns output/labels? And why can't I use only a single column?&lt;/p&gt;&#xA;" OwnerUserId="43714" LastEditorUserId="29575" LastEditDate="2017-12-24T16:45:22.540" LastActivityDate="2018-07-11T00:19:13.257" Title="Why there is two output in Titanic case in tflearn quickstart?" Tags="|python|tensorflow|kaggle|tflearn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25959" PostTypeId="1" CreationDate="2017-12-24T10:23:40.597" Score="1" ViewCount="221" Body="&lt;p&gt;I have a word 2 vector model which works great but let's say I typed Eminem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And it gives me:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;rap, music, revival and recovery.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How do I make another layer that shows revival first?&lt;/p&gt;&#xA;" OwnerUserId="42422" LastEditorUserId="29575" LastEditDate="2017-12-27T02:55:21.627" LastActivityDate="2017-12-27T02:55:21.627" Title="How do I use recency/time as a feature after I've a word2vec model" Tags="|machine-learning|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25961" PostTypeId="1" AcceptedAnswerId="26007" CreationDate="2017-12-24T12:47:41.707" Score="0" ViewCount="6175" Body="&lt;p&gt;I have a training dataset with 8000 rows, and I am trying to train a Keras Neural Network on it, using 100 epochs. However, the training process gets stuck around epoch 6 every time, as shown below. I'm not sure if it's because of my computer (8GB RAM Macbook Pro) or because of some inappropriately set parameters for my model. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import keras&#xA;    from keras.models import Sequential&#xA;    from keras.layers import Dense&#xA;&#xA;    classifier = Sequential()&#xA;&#xA;    classifier.add(Dense(activation=&quot;relu&quot;, input_dim=11, units=6, kernel_initializer=&quot;uniform&quot;))&#xA;&#xA;    classifier.add(Dense(activation=&quot;relu&quot;, units=6, kernel_initializer=&quot;uniform&quot;))&#xA;&#xA;    classifier.add(Dense(activation=&quot;sigmoid&quot;, units=1, kernel_initializer=&quot;uniform&quot;))&#xA;&#xA;    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])&#xA;&#xA;    classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 100)&#xA;&#xA;8000/8000 [==============================] - 3s 348us/step - loss: 0.3890 - acc: 0.8367&#xA;Epoch 2/100&#xA;8000/8000 [==============================] - 3s 329us/step - loss: 0.3894 - acc: 0.8376&#xA;Epoch 3/100&#xA;8000/8000 [==============================] - 3s 314us/step - loss: 0.3887 - acc: 0.8370&#xA;Epoch 4/100&#xA;8000/8000 [==============================] - 3s 346us/step - loss: 0.3895 - acc: 0.8379&#xA;Epoch 5/100&#xA;8000/8000 [==============================] - 3s 320us/step - loss: 0.3887 - acc: 0.8370 - ETA: 2s - loss: 0.4087 - acc: 0.8077&#xA;Epoch 6/100&#xA;7650/8000 [===========================&amp;gt;..] - ETA: 0s - loss: 0.3881 - acc: 0.8363 &#xA;//STUCK HERE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I do get the following message on my terminal window though (even though I'm using jupyter notebook): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;2017-12-24 20:45:30.464660: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="35386" LastActivityDate="2017-12-26T05:17:24.827" Title="Keras Neural Network training is stuck (gets stuck around epoch 6)" Tags="|python|neural-network|deep-learning|keras|jupyter|" AnswerCount="1" CommentCount="5" ClosedDate="2017-12-27T04:37:34.597" ContentLicense="CC BY-SA 3.0" />
  <row Id="25962" PostTypeId="1" AcceptedAnswerId="25971" CreationDate="2017-12-24T14:10:35.280" Score="3" ViewCount="17874" Body="&lt;p&gt;While performing a simple fitting operation on the Titanic dataset. The following is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = pd.read_csv(r'.\Desktop\DS\Titanic\train.csv')&#xA;sex_train = data['Sex'].map({'male':0,'female':1})&#xA;survived_train = data['Survived']&#xA;sex_survivor_tree = GaussianNB()&#xA;sex_survivor_tree.fit(sex_train,survived_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43718" LastActivityDate="2017-12-27T03:04:14.777" Title="Error 'Expected 2D array, got 1D array instead:'" Tags="|machine-learning|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25964" PostTypeId="1" CreationDate="2017-12-24T14:27:32.607" Score="1" ViewCount="35" Body="&lt;h1&gt;Short question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I want to learn how to construct data science packages on top of core packages. Is there a list of excellent data science packages I can learn from?&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Long question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I recently came across an &lt;a href=&quot;https://github.com/joelgrus/joelnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;excellent video&lt;/a&gt; where Joel Grus live codes a neural network library in Python. As an inexperienced data scientist without a software engineering background, this was the first time I saw the construction of a &quot;complete&quot; data science package from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data analysis code up to this point has been very straightforward, only splitting into functions here and there, and never splitting into modules even where it may have been beneficial. I have since tried to find more resources to teach myself this skill. &lt;a href=&quot;http://docs.python-guide.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Hitchhiker's Guide to Python&lt;/a&gt; has been useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I gather that it is important to learn by reading excellent packages. The Hitchhiker's guide does suggest some Python packages, but none that I could find were within data science (like Joel Grus's neural network library).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not asking for a list of the core data science packages of a language, e.g., numpy, scipy, sklearn, pandas, etc. Rather, a list of packages constructed on top of those.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does such a list already exist? If not, can (should) we create it here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: In hindsight, sklearn is possibly the perfect example of what I'm looking for, since its built on top of numpy and scipy.&lt;/p&gt;&#xA;" OwnerUserId="41869" LastEditorUserId="41869" LastEditDate="2017-12-24T14:33:41.530" LastActivityDate="2017-12-24T14:33:41.530" Title="I want to learn how to construct data science packages on top of core packages. Is there a list of excellent data science packages I can learn from?" Tags="|python|programming|software-development|" AnswerCount="0" CommentCount="0" ClosedDate="2017-12-27T04:38:49.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="25966" PostTypeId="2" ParentId="25907" CreationDate="2017-12-24T15:53:42.407" Score="8" Body="&lt;p&gt;From the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.FeatureAgglomeration.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Similar to AgglomerativeClustering, but recursively merges features instead of samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In standard agglomerative clustering you receive a matrix $M^{n \times m}$ representing $n$ samples of dimension $m$ that you want to cluster. In feature agglomeration the algorithm clusters the transpose of the matrix, i.e. $M^T$ so it clusters $m$ samples of dimension $n$, these samples represent the features.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The default distance used to cluster the features (samples in the transpose matrix) is the euclidean distance, but you can also use l1, cosine and others. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example suppose you have 3 samples of dimension 3 (a matrix 3x3 matrix)::&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;----------------------------------&#xA;| feature1 | feature2 | feature3 |&#xA;----------------------------------&#xA;|     1    |   1.05   |    10    |&#xA;----------------------------------&#xA;|     1    |   1.05   |    10    |&#xA;----------------------------------&#xA;|     2    |   2.05   |    20    |&#xA;----------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to reduce the dimension of your dataset to 2 dimensions, the algorithm clusters together feature1 and feature2, and leaves feature3 unchanged, the new matrix is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;----------------------------------&#xA;| feature1 - feature2 | feature3 |&#xA;----------------------------------&#xA;|         1.025       |    10    |&#xA;----------------------------------&#xA;|         1.025       |    10    |&#xA;----------------------------------&#xA;|         2.025       |    20    |&#xA;----------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting feature is determined by the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.FeatureAgglomeration.html#sklearn.cluster.FeatureAgglomeration.pooling_func&quot; rel=&quot;noreferrer&quot;&gt;pooling function&lt;/a&gt; in the case above it computes the arithmetic mean. See &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_digits_agglomeration.html&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; for a more in depth usage example.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastEditorUserId="26243" LastEditDate="2017-12-24T16:05:22.863" LastActivityDate="2017-12-24T16:05:22.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25968" PostTypeId="2" ParentId="25916" CreationDate="2017-12-24T18:06:11.913" Score="0" Body="&lt;p&gt;If less than 20-25% of behavioral data is missing, maybe you could try to impute missing data  using one of the following solutions : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Impute missing behavioral data using some business rule or by training a machine learning model with demographics data as input and behavioral data as output variable.&lt;/li&gt;&#xA;&lt;li&gt;Impute missing data with feature mean/median.&lt;/li&gt;&#xA;&lt;li&gt;Impute missing data by picking up random value in the feature distribution (hot-deck).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In case you have more than 20-25% missing data, it will be really hard to impute values. I think in this situation you should consider creating a new model such as : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The new model doesn't use behavioral data anymore.&lt;/li&gt;&#xA;&lt;li&gt;The new model is based on a different train-val-test split in order to have behavioral data in each dataset.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you can't create a new model neither impute missing data, I guess hot-deck would be the best option you have to avoid bad performance on unseen data.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-24T18:06:11.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25969" PostTypeId="1" CreationDate="2017-12-24T18:10:23.917" Score="1" ViewCount="488" Body="&lt;p&gt;I am trying to build ARIMA model, I have 144 terms in my standardized time series, which represent residuals form original time series. This residuals, on which I would like to build ARIMA model, are obtained when I subtracted linear trend and periodical component from original time series, so residuals are stochastic component. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of that subtraction I modeled residuals like stationary series (d=0), so model is ARIMA(p,d,q)=ARIMA(?,0,?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ACF and PACF functions of my residuals are not very clear as cases in literature for identification ARIMA models, and when I choose parameters p and q according to criteria that they are last values outside of confidence interval, I got values p=109, q=97. Matlab gave me error for this case:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error using arima/estimate (line 386)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Input response series has an insufficient number of observations.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;On the other side, when I am looking only to N/4 length of time series for identifying p and q parameters, I got p=36, q=34. Matlab gave me error for this case &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Warning: Nonlinear inequality constraints are active; standard errors may be inaccurate. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;In arima.estimate at 1113 &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Error using arima/validateModel (line 1306)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The non-seasonal autoregressive polynomial is unstable.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Error in arima/setLagOp (line 391)&#xA;     Mdl = validateModel(Mdl);&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Error in arima/estimate (line 1181)&#xA;    Mdl = setLagOp(Mdl, 'AR' , LagOp([1 -coefficients(iAR)' ], 'Lags', [0 LagsAR ]));&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How do I need to correct identify p and q parameters and what is wrong here? And wwhat does it mean in this partial autocorrelation diagram, why are last values so big?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vK5xi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vK5xi.png&quot; alt=&quot;ACF of time series&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/1dMbr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1dMbr.png&quot; alt=&quot;PACF of time series&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27460" LastActivityDate="2017-12-24T18:10:23.917" Title="How to identify ARIMA model parameters" Tags="|time-series|matlab|model-selection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25970" PostTypeId="2" ParentId="17047" CreationDate="2017-12-24T19:20:11.937" Score="1" Body="&lt;p&gt;suppose $w_1 = w_2 =... w_n = w$ then $y = w^n \times x $. In this sense is $y$ a linear function of $x$ and a non-linear function of $w$. &lt;/p&gt;&#xA;" OwnerUserId="42945" LastActivityDate="2017-12-24T19:20:11.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25971" PostTypeId="2" ParentId="25962" CreationDate="2017-12-24T22:23:33.413" Score="4" Body="&lt;p&gt;This is a bit tricky. Using pandas data, sklearn only accepts input variables (features) with type &lt;code&gt;pandas.Dataframe&lt;/code&gt;. In your code variable &lt;code&gt;sex_train&lt;/code&gt; in &lt;code&gt;pandas.Series&lt;/code&gt; type. Try the following code :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sex_train = data['Sex'].map({'male':0,'female':1}).to_frame()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="29575" LastEditDate="2017-12-27T03:04:14.777" LastActivityDate="2017-12-27T03:04:14.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25972" PostTypeId="2" ParentId="19813" CreationDate="2017-12-24T22:40:03.573" Score="1" Body="&lt;p&gt;You have instantiated a sparkContext object with &quot;local&quot; mode configuration. It means you have allocated ressources for a single multi-core Java Virtual Machine on your computer. In this configuration, you can't reach better performs than with python. Because : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With local mode, you have less ressources (cpu, memory...) with Spark than with python (you can't have a JVM with more ressources than your computer).&lt;/li&gt;&#xA;&lt;li&gt;PySpark code is converted into Scala code before execution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Spark benefits arise when you use Spark on multiple nodes&lt;/strong&gt;. In this configuration, spark master is yarn, mesos or standalone. In that case, a spark job would be separated in multiple tasks and each node would be dedicated to different tasks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have 3 nodes and 6 tasks, each node can handle 2 tasks. If you have 6 nodes and 6 tasks, each node will handle a task. With simple python, think that you have only 1 node for &quot;6 tasks&quot;. So, for large tasks (big enough datasets), Spark will results in better latencies than simple python. &lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-24T22:40:03.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25973" PostTypeId="1" CreationDate="2017-12-24T23:38:40.183" Score="2" ViewCount="3854" Body="&lt;p&gt;I am new to both data science and python. &#xA;I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW) is a useful method to find alignments between two time series which may vary in time or speed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found &lt;code&gt;dtw_std&lt;/code&gt; in &lt;code&gt;mlpy&lt;/code&gt; library and &lt;code&gt;scipy.cluster.hierarchy&lt;/code&gt; in &lt;code&gt;SciPy&lt;/code&gt; in order to cluster my data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html#scipy.cluster.hierarchy.linkage&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy docs&lt;/a&gt;, I find that I could use my custom distance function:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;metric : str or function, optional The distance metric to use in the&#xA;  case that y is a collection of observation vectors; ignored otherwise.&#xA;  See the pdist function for a list of valid distance metrics. A custom&#xA;  distance function can also be used.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But I am stuck matching this information to implement clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset is in the format of &lt;code&gt;dataframe&lt;/code&gt; which each row corresponds to a sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- How can I provide distance matrics for the linkage function? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- How to set my custom distance function?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import scipy.cluster.hierarchy as hac&#xA;import mlpy&#xA;&#xA;dataset = pd.read_csv ( &quot;dataset.csv&quot;,encoding='utf-8' )&#xA;X # distance matrics&#xA;Z = hac.linkage(X, metrics=mlpy.dtw_std, method='average')&#xA;cluster = hac.fcluster(Z, t=100, criterion='maxclust')&#xA;leader = hac.leaders(Z, t=100, criterion='maxclust')&#xA;&#xA;&#xA;fig = plt.figure(figsize=(25, 10))&#xA;dn = dendrogram(Z)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how I compute distance matrix, then I pass it to linkage:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# computing distance matrix&#xA;dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))&#xA;z = hac.linkage(dm, method='average')&#xA;cluster = hac.fcluster(z, t=100, criterion='maxclust')&#xA;leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='maxclust')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42904" LastEditorUserId="42904" LastEditDate="2017-12-27T12:46:04.323" LastActivityDate="2024-02-05T15:01:15.230" Title="Agglomerative Hierarchial Clustering in python using DTW distance" Tags="|machine-learning|python|clustering|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25974" PostTypeId="1" CreationDate="2017-12-25T01:45:46.540" Score="1" ViewCount="239" Body="&lt;p&gt;I have been trying to convert an implementation of Yolo written in dark-net &lt;a href=&quot;https://github.com/hollance/Forge/blob/master/Examples/YOLO/YOLO/YOLO.swift&quot; rel=&quot;nofollow noreferrer&quot;&gt;found here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having issues converting the original source code as the layers implemented in an alternative library seem to have an implementation gap as tiny_dnn requires more parameters to be set...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the link above, line 44:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;let output = input&#xA;    --&amp;gt; Resize(width: YOLO.inputWidth, height: YOLO.inputHeight)&#xA;    --&amp;gt; Convolution(kernel: (3, 3), channels: 16, activation: leaky, name: &quot;conv1&quot;)&#xA;    --&amp;gt; MaxPooling(kernel: (2, 2), stride: (2, 2))&#xA;    --&amp;gt; Convolution(kernel: (3, 3), channels: 32, activation: leaky, name: &quot;conv2&quot;)&#xA;--&amp;gt; MaxPooling(kernel: (2, 2), stride: (2, 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm trying to translate the above into c++ but I am missing parameters, as the swift code above must infer a number of variables... surely... as it seems to not require stating the following variables:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Window size, &lt;/li&gt;&#xA;&lt;li&gt;bias, &lt;/li&gt;&#xA;&lt;li&gt;wstride, &lt;/li&gt;&#xA;&lt;li&gt;hstride, &lt;/li&gt;&#xA;&lt;li&gt;padtype,&lt;/li&gt;&#xA;&lt;li&gt;pooling size&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here is the code I'v tried in c++:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;int w=416;&#xA;int h=416;&#xA;net &#xA;&amp;lt;&amp;lt; convolutional_layer(w, h,3,16,16)&#xA;&amp;lt;&amp;lt; leaky_relu_layer(0.1F)&#xA;&amp;lt;&amp;lt; max_pooling_layer(shape3d(w , h , 3),2,2, core::default_engine());&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;however that results in an error between the leaky_relu_layer and the max_pooling as it says the output of the leaky relu layer does not match the input of the max pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The parameters of the c++ are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  convolutional_layer(size_t in_width,&#xA;                      size_t in_height,&#xA;                      size_t window_size,&#xA;                      size_t in_channels,&#xA;                      size_t out_channels,&#xA;                      padding pad_type             = padding::valid,&#xA;                      bool has_bias                = true,&#xA;                      size_t w_stride              = 1,&#xA;                      size_t h_stride              = 1,&#xA;                      core::backend_t backend_type = core::default_engine())&#xA;&#xA;&#xA;max_pooling_layer(const shape3d &amp;amp;in_shape,&#xA;                    size_t pooling_size,&#xA;                    size_t stride,&#xA;                    core::backend_t backend_type = core::default_engine())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could anyone possibly explain why the output of my convolutional/leaky relu results in a different sized output than the input of the max pooling layer?&lt;/p&gt;&#xA;" OwnerUserId="31075" LastEditorUserId="29169" LastEditDate="2021-02-13T19:19:36.613" LastActivityDate="2021-02-13T19:19:36.613" Title="Convoluational layer is larger than max pooling layer" Tags="|neural-network|cnn|convolutional-neural-network|yolo|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="25975" PostTypeId="1" AcceptedAnswerId="25976" CreationDate="2017-12-25T06:42:55.643" Score="0" ViewCount="109" Body="&lt;p&gt;I have searched online, but am still not satisfied with answers like &lt;a href=&quot;https://stats.stackexchange.com/q/212344&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://www.quora.com/What-is-the-purpose-of-using-more-than-1-fully-connected-layer-in-a-convolutional-neural-network&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is that fully connected layers are completely linear. That means no matter how many FC layers are used, the expressiveness is always limited to linear combinations of previous layer. But mathematically, one FC layer should already be able to learn the weights to produce the exactly the same behavior. Then why do we need more? Did I miss something here?&lt;/p&gt;&#xA;" OwnerUserId="38370" LastActivityDate="2017-12-25T07:19:56.113" Title="Is there an intuitive explanation why some neural networks have more than one fully connected layers?" Tags="|neural-network|deep-learning|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25976" PostTypeId="2" ParentId="25975" CreationDate="2017-12-25T07:19:56.113" Score="2" Body="&lt;p&gt;There is a nonlinear activation function inbetween these fully connected layers. Thus the resulting function is not simply a linear combination of the nodes in the previous layer.&lt;/p&gt;&#xA;" OwnerUserId="36977" LastActivityDate="2017-12-25T07:19:56.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25977" PostTypeId="1" CreationDate="2017-12-25T08:07:12.960" Score="2" ViewCount="3005" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EKvbi.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EKvbi.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &#xA;I want to know the effect of applying different C values on SVM Classifier margin&lt;/p&gt;&#xA;" OwnerUserId="43734" LastActivityDate="2018-01-24T10:32:26.213" Title="Using SVM Classifier with C=0 and C=infinity ,what would be the effect on classifying this data?" Tags="|machine-learning|svm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="25978" PostTypeId="2" ParentId="25977" CreationDate="2017-12-25T08:42:47.247" Score="2" Body="&lt;p&gt;C value is like lambda, the L2/L1 regularization hyper parameter, but in reverse manner. Whenever C is large, means there is high probability that your model overfit the data in hand. Whenever it is small, your model endure some errors in order to avoid producing small margin and consequently you will have a model less prone to overfitting. The first one is called hard SVM and the latter one is soft SVM. In your case, I guess there will be no difference because your model is linearly separable. If &lt;em&gt;X4&lt;/em&gt; was between &lt;em&gt;X2&lt;/em&gt; and &lt;em&gt;X3&lt;/em&gt; you would have different result for each hard and soft. Soft and hard differ in cases which the data is not linearly separable or maybe they are separable but the margin would be narrow or wide if SVM discards some samples. &lt;em&gt;Soft&lt;/em&gt; tries to make the separator space, margin, as large as possible by discarding those minority data which hinder a large margin to be crated but &lt;em&gt;hard&lt;/em&gt; tries to separate the data to reduce the error as much as possible. In your case, you will get same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vIGFw.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vIGFw.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For illustrating more, look at the left figure. It tries to discard the sample which hinders the large margin to be created but large margin, considers all the samples.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-25T08:42:47.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25979" PostTypeId="2" ParentId="25959" CreationDate="2017-12-25T09:11:28.963" Score="1" Body="&lt;p&gt;The purpose of the Word2vec is it will itself learn a hidden structure in your text data. If &quot;recovery&quot; comes after rap and music is because Eminem is more frequent with these context words than with recovery. You can't simply change that. You should avoid making a manual intervention on your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, here are 2 ideas you could try if you really want &quot;recovery&quot; to be closer of Eminem word :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try an other Word2Vec algorithm architecture. There are CBOW and skip-gram you can try. In my opinion CBOW could perform better because the CBOW model trains each word against its context. It asks &quot;given this set of context words, what missing word is likely to also appear at the same time?&quot;. Skip-gram trains each the context against the word. It asks &quot;given this single word, what are the other words that are likely to appear near it at the same time?&quot;&lt;/li&gt;&#xA;&lt;li&gt;Add sentences relative to recovery album in your data. Because words recovery and Eminem should dominate in these sentences, It could reconcile the 2 words. Same, you can replicate sentences containing both Eminem and recovery word and randomly change the order of words in the sentence.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-25T09:11:28.963" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25982" PostTypeId="1" CreationDate="2017-12-25T09:48:15.933" Score="1" ViewCount="968" Body="&lt;p&gt;Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, scaling this to $10$ million data points requires a creatively efficient implementation since DBSCAN requires $O(n^2)$ to calculate the distance matrix and crushes my memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There must be some efficient sampling-based method to overcome this, ideally something similar to minHash - but I'm not sure how to approach this, and if there exists a  solution that can work on existing sklearn DBSCAN algorithm. Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="38377" LastEditorUserId="27665" LastEditDate="2017-12-25T16:12:02.360" LastActivityDate="2018-03-26T13:18:44.377" Title="Scaling DBSCAN clustering - minHash?" Tags="|machine-learning|clustering|scalability|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25985" PostTypeId="1" CreationDate="2017-12-25T10:37:11.347" Score="3" ViewCount="301" Body="&lt;p&gt;I need to predict technical aggregate condition using vibration monitoring data. We consider this data to be nonstationary i.e. distribution parameters and descriptive statistics are not constant. I found that one of the best algorithms for such tasks in Learn++.NSE and we us it with MLP as a base classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I know, it's necessary no normalize data for operations with ANN. We decided to normalize using mean, stdev and sigmoidal function. We train networks of ensemble with sets with different values distribution parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my questions are the following&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to normalize new trainig set during previous networks evaluation? Problem is in description statistics change&lt;/li&gt;&#xA;&lt;li&gt;How to normalize input data while ensemble usage? Current statistics differ from the previous ones.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41535" LastActivityDate="2024-01-21T15:04:50.633" Title="Data normalization in nonstationary data classification with Learn++.NSE based on MLP" Tags="|neural-network|time-series|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25986" PostTypeId="1" CreationDate="2017-12-25T11:33:03.113" Score="1" ViewCount="40" Body="&lt;pre&gt;&lt;code&gt;library(ggplot2) &#xA;library(readr) &#xA;library(stringr)&#xA;library(neuralnet)&#xA;&#xA;train &amp;lt;- read.delim(&quot;../input/train.tsv&quot;)&#xA;&#xA;test &amp;lt;- read.delim(&quot;../input/test.tsv&quot;)&#xA;&#xA;category &amp;lt;- str_split_fixed(train$category_name,&quot;/&quot;,3)&#xA;&#xA;category_test &amp;lt;- str_split_fixed(test$category_name,&quot;/&quot;,3)&#xA;&#xA;train$category_a &amp;lt;- category[,1]&#xA;&#xA;test$category_a &amp;lt;- category_test[,1]&#xA;&#xA;train$category_b &amp;lt;- category[,2]&#xA;&#xA;test$category_b &amp;lt;- category_test[,2]&#xA;&#xA;train$category_c &amp;lt;- category[,3]&#xA;&#xA;test$category_c &amp;lt;- category_test[,3]&#xA;&#xA;&#xA;train = train[,!(names(train) %in% c(&quot;category_name&quot;))]&#xA;&#xA;train = train[,!(names(train) %in% c(&quot;item_description&quot;))]&#xA;&#xA;train = train[,!(names(train) %in% c(&quot;name&quot;))]&#xA;&#xA;test = test[,!(names(test) %in% c(&quot;category_name&quot;))]&#xA;test = test[,!(names(test) %in% c(&quot;item_description&quot;))]&#xA;test = test[,!(names(test) %in% c(&quot;name&quot;))]&#xA;&#xA;train$brand_name &amp;lt;- as.numeric(train$brand_name)&#xA;train$category_a &amp;lt;- as.numeric(factor(train$category_a))&#xA;train$category_b &amp;lt;- as.numeric(factor(train$category_b))&#xA;train$category_c &amp;lt;- as.numeric(factor(train$category_c))&#xA;&#xA;test$brand_name &amp;lt;- as.numeric(test$brand_name)&#xA;&#xA;test$category_a &amp;lt;- as.numeric(factor(test$category_a))&#xA;&#xA;test$category_b &amp;lt;- as.numeric(factor(test$category_b))&#xA;&#xA;test$category_c &amp;lt;- as.numeric(factor(test$category_c))&#xA;&#xA;&#xA;max = apply(train , 2 , max)&#xA;&#xA;min = apply(train, 2 , min)&#xA;&#xA;max_test = apply(test , 2 , max)&#xA;&#xA;min_test = apply(test, 2 , min)&#xA;&#xA;scaled = as.data.frame(scale(train, center = min, scale = max - min))&#xA;&#xA;scaled_test = as.data.frame(scale(test, center = min_test, scale = max_test - min_test))&#xA;&#xA;set.seed(2)&#xA;results = neuralnet(price ~ item_condition_id + shipping + category_a + category_b , scaled, hidden = 3,linear.output = T)&#xA;#results &amp;lt;- lm(item_condition_id + price ~ shipping + category_a + category_b , data=train)&#xA;&#xA;pre &amp;lt;- compute(results,scaled_test)&#xA;&#xA;solution &amp;lt;- data.frame(test_id = test$test_id, price = pre)&#xA;&#xA;write.csv(solution, file = 'sample_submission.csv', row.names = F)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43742" LastEditorUserId="836" LastEditDate="2017-12-25T14:20:36.237" LastActivityDate="2017-12-25T14:20:36.237" Title="Neural network algorithm doesn't work" Tags="|r|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25987" PostTypeId="1" AcceptedAnswerId="27142" CreationDate="2017-12-25T13:29:48.573" Score="0" ViewCount="10256" Body="&lt;p&gt;I used &lt;code&gt;BalancedBaggingClassifier&lt;/code&gt; from &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/generated/imblearn.ensemble.BalancedBaggingClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;imblearn&lt;/a&gt; library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction with feature names especially when the &lt;code&gt;max_features&lt;/code&gt; are less than the total number of features? For example, in the following code total number of features equal to 20 but &lt;code&gt;max_features&lt;/code&gt; are 8.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import Counter&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import train_test_split&#xA;from imblearn.ensemble import BalancedBaggingClassifier&#xA;from xgboost.sklearn import XGBClassifier&#xA;&#xA;X, y = make_classification(n_classes=2, class_sep=2,weights=[0.1, 0.9], n_informative=3, n_redundant=1, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)&#xA;print('Original dataset shape {}'.format(Counter(y)))&#xA;&#xA;ln = X.shape&#xA;names = [&quot;x%s&quot; % i for i in range(1, ln[1] + 1)]&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)&#xA;bbc = BalancedBaggingClassifier(base_estimator=XGBClassifier(n_estimators=10,min_child_weight=5,max_depth=3,learning_rate=0.02,colsample_bytree=1,subsample=1,scale_pos_weight=.26), ratio='all',random_state=0, max_features=8)&#xA;bbc.fit(X_train,y_train)&#xA;for estimator in bbc.estimators_:&#xA;    print(sorted(zip(map(lambda x: round(x, 4), estimator.steps[1][1].feature_importances_),names), reverse=True))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think there is a problem with the above code because always printed features are named x1 to x8 while for example, feature x19 may be among the most important features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2021-04-30T09:36:11.423" LastActivityDate="2021-04-30T09:36:11.423" Title="How to print feature names in conjunction with feature Importance using Imbalanced-learn library?" Tags="|python|classification|scikit-learn|xgboost|imbalanced-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25988" PostTypeId="1" CreationDate="2017-12-25T13:37:31.387" Score="2" ViewCount="7044" Body="&lt;p&gt;I am working on Street view house numbers dataset using CNN in Keras on tensorflow backend. I have queries regarding why loss of network is not decreasing, I have doubt whether I am using correct loss function or not. First I preprocess dataset so  my train and test dataset shapes are:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('Training set and labels', (26721,32,32,1), (26721, 6) )&#xA;('Validation set and labels ', (6680,32,32,1), (6680,6) )&#xA;('Test set and labels', (13068,32,32,1), (13068,6) )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Reason why labels array has 6 columns, because maximum digits in one image is 6. For suppose, if image has 2 digits &quot;1 and 2&quot;, then labels array is considered as [1,2,10,10,10,10] , where 10 represents no digits. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now this is my model definition: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def modelCNN(input_shape, num_classes):&#xA;&#xA;  model = Sequential()&#xA;&#xA;  model.add(Convolution2D(16, 7, 4, border_mode='same',&#xA;                    input_shape=input_shape))&#xA;  model.add(PReLU())&#xA;  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))&#xA;  model.add(BatchNormalization())&#xA;&#xA;  model.add(Convolution2D(32, 5, 3, border_mode='same'))&#xA;  model.add(PReLU())&#xA;  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))&#xA;  model.add(BatchNormalization())&#xA;&#xA;  model.add(Convolution2D(64, 3, 3, border_mode='same'))&#xA;  model.add(PReLU())&#xA;  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), border_mode='same'))&#xA;&#xA;  model.add(Flatten())&#xA;  model.add(Dense(512))&#xA;  model.add(PReLU())&#xA;  model.add(Dropout(0.5))&#xA;  model.add(Dense(num_classes))&#xA;  model.add(Activation('softmax'))&#xA;&#xA;return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried with different customs model, but nothing seems working. I do compilation, loss and optimization like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(optimizer='adam', loss='categorical_crossentropy',&#xA;                                    metrics=['accuracy'])&#xA;&#xA;print(model.summary())&#xA;csv_logger = CSVLogger('training.log')&#xA;early_stop = EarlyStopping('val_acc', patience=200, verbose=1)&#xA;model_checkpoint = ModelCheckpoint(model_save_path,&#xA;                                    'val_acc', verbose=0,&#xA;                                    save_best_only=True)&#xA;&#xA;model_callbacks = [early_stop, model_checkpoint, csv_logger]&#xA;K.get_session().run(tf.global_variables_initializer())&#xA;&#xA;model.fit_generator(train,&#xA;          samples_per_epoch=np.ceil(len(train_dataset)/batch_size),&#xA;          epochs=num_epochs,&#xA;          verbose=1,&#xA;          validation_data=valid,&#xA;      validation_steps=batch_size,&#xA;          callbacks=model_callbacks)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, What I think that &lt;strong&gt;categorical_crossentropy&lt;/strong&gt; loss needs one hot vector representation, but I am not sure how should i convert my labels array. Because unlike Mnist dataset, I have multiple labels in each image.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any loss function that I can use? or Any suggestion to make things work? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am using CNN just for classification only, not for detection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks very much in advance&lt;/p&gt;&#xA;" OwnerUserId="43744" LastActivityDate="2019-11-26T18:38:25.120" Title="Losses of keras CNN model is not decreasing" Tags="|machine-learning|deep-learning|keras|loss-function|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25989" PostTypeId="1" CreationDate="2017-12-25T15:32:16.547" Score="1" ViewCount="1552" Body="&lt;p&gt;I want to visualize my word2vec vector space (with zoom in and zoom out). I found a really interesting GitHub project named &lt;a href=&quot;https://github.com/dominiek/word2vec-explorer&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec explorer&lt;/a&gt;. However, it does not seem to work in windows and mac.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I am interested in knowing if there are any other off-the-shelf project I can use for this?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2018-01-01T17:37:23.597" Title="Simple way to visualise word2vec vector space" Tags="|python|visualization|word2vec|word-embeddings|tsne|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25990" PostTypeId="1" CreationDate="2017-12-25T15:51:21.500" Score="1" ViewCount="933" Body="&lt;p&gt;Im studying the perceptron algorithm. &#xA;I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o4vlo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o4vlo.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In every web page i read up for a detailed explanation, it's generally said that the perceptron algorithm will find the &lt;strong&gt;optimal&lt;/strong&gt; weights w. But optimal in which sense ?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="28175" LastEditDate="2017-12-25T17:02:45.387" LastActivityDate="2018-05-25T19:25:29.057" Title="Meaning of Perceptron optimal weights" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25991" PostTypeId="2" ParentId="25988" CreationDate="2017-12-25T15:54:05.707" Score="2" Body="&lt;p&gt;you can try to remove some of the irrelevant features that may be the reason or you could try different solvers and activation methods.I previously answered similar question earlier in the following link&lt;a href=&quot;https://datascience.stackexchange.com/a/25958/43438&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2017-12-25T15:54:05.707" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="25993" PostTypeId="2" ParentId="25990" CreationDate="2017-12-25T16:40:01.457" Score="-1" Body="&lt;p&gt;Using &lt;code&gt;perceptron&lt;/code&gt;, you specify a cost function, &lt;code&gt;Mean Squared Error&lt;/code&gt; for regression tasks or maybe &lt;code&gt;Cross Entropy&lt;/code&gt; for classification tasks. The input data are the constants and the weights are the parameters of your learning problem. When you specify the cost function, if you have error, the cost would be non-zero. You use algorithms like gradient descent to decrease the cost value. This is an optimization problem which you try to decrease the value of error. When we say &lt;code&gt;Perceptron&lt;/code&gt; finds the optimal point, the reason is that the shape of cost function, e.g. &lt;code&gt;MSE&lt;/code&gt; is convex and there is just one optimal point which gradient is zero there and the cost has the least possible value there. If you use neural networks, the cost with respect to its parameters, weights, is not convex and you usually can not find the optimal point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you looking &lt;a href=&quot;https://datascience.stackexchange.com/q/18802/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/24537/28175&quot;&gt;here&lt;/a&gt; for understanding more neural nets optimality.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-26T14:51:59.897" LastActivityDate="2017-12-26T14:51:59.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25994" PostTypeId="1" CreationDate="2017-12-25T16:57:29.313" Score="1" ViewCount="201" Body="&lt;p&gt;I have a social network and I want to identify the most social people in the graph. In a typical graph experiments this could have done using different network features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I recently came across graph embeddings and interested in knowing if my problem can be solved using graph embeddings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, please letme know how I can do that?&lt;/p&gt;&#xA;" OwnerUserId="37986" LastActivityDate="2018-05-24T01:16:04.397" Title="Identify social users using graph embeddings" Tags="|word2vec|graphs|word-embeddings|social-network-analysis|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25995" PostTypeId="1" CreationDate="2017-12-25T17:31:24.240" Score="4" ViewCount="620" Body="&lt;p&gt;I have a multi-label classification problem wherein each example can belong to one of the pre-defined classes (or can belong to none of them). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if I can somehow apply multi-task learning (MTL) to this problem. Essentially, treat each class as a separate class and use a NN based model with common feature extractor layer, and on top of it class specific layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My doubt is that generally there is a correlation between labels in a multi-label setting. In the MTL model, after feature extraction each class is handled separately without sharing information. Does a MTL kind of model for multi-label makes sense, given this information? &lt;/p&gt;&#xA;" OwnerUserId="13518" LastActivityDate="2020-07-11T18:03:00.290" Title="Multi-task learning for Multi-label classification?" Tags="|neural-network|classification|deep-learning|multilabel-classification|multitask-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="25996" PostTypeId="1" CreationDate="2017-12-25T18:06:54.720" Score="1" ViewCount="18" Body="&lt;p&gt;Let's say I am trying to predict whether a cat will be adopted, and I have found the ratio &lt;code&gt;purred_when_petted / total_times_petted&lt;/code&gt; aka &lt;code&gt;p / n&lt;/code&gt; to have predictive power. However, for some cats I have much more data than others. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cat A:&#xA;    p: 1&#xA;    n: 1&#xA;&#xA;Cat B:&#xA;    p: 9&#xA;    n: 10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like my neural network to consider that there is more data for Cat B and possible consider that it is more likely to be adopted. How should I tell it about both the ratio and the sample size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some ideas I have so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use ratio as input feature and don't worry about sample size&lt;/li&gt;&#xA;&lt;li&gt;Use ratio and sample size as separate input features&lt;/li&gt;&#xA;&lt;li&gt;Use $p$ and $n$ as separate input features, and let the NN do what it wants with them.&lt;/li&gt;&#xA;&lt;li&gt;Use an average of the cat statistic and the average population statistic over all $M$ cats weighted by number of samples:&#xA;$$&#xA;x_i = \beta  * (\frac{p_i}{n_i}) + (1-\beta) * (\frac{1}{M}\sum_{j=1}^{M}\frac{p_j}{n_j})&#xA;$$&#xA;where $\beta \in [0,1]$ is an increasing function of sample size $n_i$.&lt;/li&gt;&#xA;&lt;li&gt;Use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval&quot; rel=&quot;nofollow noreferrer&quot;&gt;Binomial proportion confidence interval&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there a standard way to do this? I am most interested in neural networks but also open to advice relevant to any other type of model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that I also have other categorical and numeric features such as &lt;code&gt;color&lt;/code&gt; and &lt;code&gt;age&lt;/code&gt; that also need to be fed in to my model.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2017-12-25T18:33:01.120" LastActivityDate="2017-12-25T18:33:01.120" Title="How do I tell my model about the sample size of input statistics?" Tags="|neural-network|feature-selection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25997" PostTypeId="2" ParentId="25666" CreationDate="2017-12-25T19:26:41.730" Score="5" Body="&lt;p&gt;Simple decision trees have some limitations listed below. Fortunately, some of these can be fixed used ensemble learning techniques (think bagging, boosting...).&lt;/p&gt;&#xA;&lt;p&gt;Concerning limitations :&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Trees tend to overfit quickly at the bottom. If you have few observations in last nodes, poor decision can be taken. In this situation, consider reducing the number of levels of your tree or using pruning.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Trees can be unstable because small variations in the data might result in a completely different tree being generated.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Decision trees perform greedy search of best splits at each node. This is particularly true for CART based implementation which tests all possible splits. For a continuous variable, this represents 2^(n-1) - 1 possible splits with n the number of observations in current node.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For classification, if some classes dominate, it can create biased trees. It is therefore recommended to balance the dataset prior to fitting.&lt;/p&gt;&#xA;&lt;p&gt;Also, Some distributions can be hard to learn for a decision tree. An example below (XOR) :&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dUeZP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dUeZP.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-25T19:26:41.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25998" PostTypeId="1" AcceptedAnswerId="25999" CreationDate="2017-12-25T20:06:28.810" Score="1" ViewCount="82" Body="&lt;p&gt;I'm building a logicsic regression classifier for binary classification. I have trained it and going to choose a cut-off value using ROC curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what set to use for it: training or validation?&lt;/p&gt;&#xA;" OwnerUserId="41535" LastActivityDate="2017-12-25T20:18:19.140" Title="Set for building ROC curve and and choosing logistic regression cut-off" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="25999" PostTypeId="2" ParentId="25998" CreationDate="2017-12-25T20:18:19.140" Score="0" Body="&lt;p&gt;Usually you need to generate the ROC curve and choose the threshold within the training data. Then, with the selected threshold, you have the possibility to report accuracy, sensitivity, recall results you reach on your validation set.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-25T20:18:19.140" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26001" PostTypeId="1" CreationDate="2017-12-25T21:35:25.167" Score="3" ViewCount="1230" Body="&lt;p&gt;I have a deep learning model that can be trained in one GPU, however, is very slow. Is there a way to accelerate the training by parallelizing it across several GPUs? How would be the training process?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can use any framework (pytorch, tensorflow, etc). And I know they accept multiple GPU and how to do it.  I am more interested in how to spread, conceptually, a single model between GPUs to gain performance. I do mostly NLP models with RNN but I am also interested in CNN. &lt;/p&gt;&#xA;" OwnerUserId="23492" LastEditorUserId="11097" LastEditDate="2018-02-02T05:43:14.797" LastActivityDate="2018-02-02T05:43:14.797" Title="Accelerate deep learning model training on several GPUs" Tags="|deep-learning|training|gpu|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26002" PostTypeId="2" ParentId="26001" CreationDate="2017-12-25T23:06:27.743" Score="3" Body="&lt;p&gt;What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-run-a-keras-model-on-multiple-gpus&quot; rel=&quot;nofollow noreferrer&quot;&gt;How can i run a keras model on multi gpus&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please provide more informations to your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with tf.device('/gpu:1'):&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually?&#xA;If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all I would try to train your Model on the CPUs before doing anything else.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/23895/multi-gpu-in-keras&quot;&gt;This&lt;/a&gt; post gives a quite good overview how you can use multi GPU. &lt;/p&gt;&#xA;" OwnerUserId="37808" LastEditorUserId="37808" LastEditDate="2017-12-26T11:20:24.137" LastActivityDate="2017-12-26T11:20:24.137" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26004" PostTypeId="2" ParentId="23858" CreationDate="2017-12-26T01:28:56.503" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://snap.stanford.edu/class/cs224w-2016/projects/cs224w-2-final.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Improving Community Detection in in Wikipedia Articles using Semantic Features&quot;&lt;/a&gt;&#xA;This paper talks about various methods of community detection and might be helpful.&lt;/p&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2017-12-26T01:28:56.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26005" PostTypeId="2" ParentId="25893" CreationDate="2017-12-26T01:43:16.307" Score="1" Body="&lt;p&gt;Somehow you have to come up with some sort of numerical classification system for your movie genres.&lt;br&gt;&#xA;I would start by creating a relationship tree between genres.  For example action movies and then action movies with comedy and then action movies with comedy with animation etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could develop a whole Forest of trees that relate movie genres to one another.  You can then test the genres paths of individuals to compare.&lt;/p&gt;&#xA;" OwnerUserId="24409" LastActivityDate="2017-12-26T01:43:16.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26006" PostTypeId="1" CreationDate="2017-12-26T04:35:57.240" Score="1" ViewCount="451" Body="&lt;p&gt;As we all know, XGBoost constructs trees based on gradient.&#xA;I wonder how does XGBoost define gradient of MAE loss, as MAE itself is not differentiable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After some digging of the source code, I found the implementation of MSE loss &lt;a href=&quot;https://github.com/dmlc/xgboost/blob/7c6673cb9ef7f68f92075123dc78e9661ce3d4e5/src/objective/regression_obj.cc#L23&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. But I can't find any implementation of MAE loss. The original &lt;a href=&quot;https://arxiv.org/abs/1603.02754&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; didn't discuss MAE loss either.&lt;/p&gt;&#xA;" OwnerUserId="28628" LastActivityDate="2021-07-16T15:22:49.080" Title="How does XGBoost implement MAE loss?" Tags="|xgboost|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26007" PostTypeId="2" ParentId="25961" CreationDate="2017-12-26T05:17:24.827" Score="1" Body="&lt;p&gt;I had a similar problem. I guess the reason is incompatibility of &lt;code&gt;Keras&lt;/code&gt; with your current anaconda. Just update it. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-26T05:17:24.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26009" PostTypeId="1" CreationDate="2017-12-26T06:29:27.790" Score="6" ViewCount="1757" Body="&lt;p&gt;I was reading the &lt;a href=&quot;http://www.deeplearningbook.org/contents/ml.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning book&lt;/a&gt; and came across the following para (page 109, second para):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted &lt;span class=&quot;math-container&quot;&gt;$p_{\text{data}}$&lt;/span&gt;. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can somebody please explain to me the meaning of this paragraph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On page 122 the last paragraph, it also gives an example &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a set of samples &lt;span class=&quot;math-container&quot;&gt;$\{x(1), \dots, x(m) \}$&lt;/span&gt; that are independently and identically distributed according to a Bernoulli distribution with mean &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this mean?&lt;/p&gt;&#xA;" OwnerUserId="43769" LastEditorDisplayName="user10640" LastEditDate="2019-11-05T06:59:46.287" LastActivityDate="2019-11-05T06:59:46.287" Title="What does it mean for the training data to be generated by a probability distribution over datasets" Tags="|deep-learning|dataset|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26010" PostTypeId="2" ParentId="25933" CreationDate="2017-12-26T07:08:16.980" Score="0" Body="&lt;p&gt;Spent some time with error and found that serial data consists a newline character&quot;\n&quot; in it and it caused error pitch data is a float and due added character roll value is a string.So, changed code as below a little and it worked &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/j71YH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/j71YH.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2017-12-26T07:08:16.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26011" PostTypeId="1" AcceptedAnswerId="26018" CreationDate="2017-12-26T07:11:46.703" Score="1" ViewCount="715" Body="&lt;p&gt;I have a question regarding CNN.&#xA;I do understand how they work on the surface. Very simply put, they are D-NN for images.&lt;/p&gt;&#xA;&lt;p&gt;I'll use this &lt;a href=&quot;https://www.tensorflow.org/versions/master/tutorials/layers&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; as a reference for this question. In the example, they are not initializing weights and biases anywhere. They are using &lt;code&gt;tf.layers.conv2d()&lt;/code&gt; function in the example.&lt;/p&gt;&#xA;&lt;p&gt;Let's focus only on 2 of the function's arguments:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;filters&lt;/code&gt;: specify the number of filters to apply&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;kernel_size&lt;/code&gt;: specifies the size of each filter&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3&gt;Questions:&lt;/h3&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We are not defining these filters. Does tensorflow define them for us or in general if I were to use this for a different set of images how does this work?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does this filter include both weights and biases? If not, then what exactly does the &lt;code&gt;GradientDescentOptimizer()&lt;/code&gt; defined in the above example update after each training step?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I understood the code and understand how the entire process works. I also understand convolution and how it works. But, I'm trying to apply these concepts and implement my own code using tensorflow and CNN and I'm kind of stuck here.&lt;/p&gt;&#xA;" OwnerDisplayName="user43771" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2017-12-26T19:34:40.803" Title="Questions about CNN: weights and biases" Tags="|machine-learning|deep-learning|tensorflow|cnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26013" PostTypeId="1" AcceptedAnswerId="26020" CreationDate="2017-12-26T09:53:26.060" Score="3" ViewCount="1953" Body="&lt;p&gt;I'm using sklearn Decision Tree Classifier with some continuous features.&#xA;When I run export_graphviz I see the same features in more than one nodes and with different values. Example:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LqZ0L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LqZ0L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to take the top important ones and want to use feature_importances_ for that.&#xA;The problem is that feature_importances_ is array without reference to the tree nodes. I have the original features but as each one can be more than one time in the tree I'm not sure how to relate importance to node.&lt;/p&gt;&#xA;" OwnerUserId="43777" LastActivityDate="2018-01-26T00:57:41.123" Title="python sklearn decision tree classifier feature_importances_ with feature names when using continuous values" Tags="|python|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-28T21:42:13.323" ContentLicense="CC BY-SA 3.0" />
  <row Id="26014" PostTypeId="2" ParentId="23895" CreationDate="2017-12-26T11:16:32.067" Score="2" Body="&lt;p&gt;Basically, you can take example of the following example. All you need is specifying cpu and gpu consumption values after importing keras.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;&#xA;config = tf.ConfigProto( device_count = {'GPU': 1 , 'CPU': 56} )&#xA;sess = tf.Session(config=config) &#xA;keras.backend.set_session(sess)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After then, you would fit the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(x_train, y_train, epochs=epochs, validation_data=(x_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, you can decrease the consumption values not the work on upper limits.&lt;/p&gt;&#xA;" OwnerUserId="43779" LastActivityDate="2017-12-26T11:16:32.067" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26015" PostTypeId="2" ParentId="25982" CreationDate="2017-12-26T12:19:24.910" Score="1" Body="&lt;p&gt;DBSCAN is O(n) times the cost of a neighbor search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use an index like LSH that could answer neighborhood search in O(1) (assuming a very even data distribution, with O(1) neighbors per point) then DBSCAN can run in O(n) plus the time needed to build such an index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, minHash indexes could be used if they are appropriate for your data and distance.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-26T12:19:24.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26016" PostTypeId="1" AcceptedAnswerId="26136" CreationDate="2017-12-26T13:23:28.223" Score="11" ViewCount="627" Body="&lt;p&gt;Input Data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X$ -&gt; features of t shirt (colour,logo,etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Y$ -&gt; profit margin&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have trained a random forest on the above $X$ and $Y$ and have achieved reasonable accuracy on a test data.&#xA;So, I have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y|X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I would like to find $P(X|Y)$ i.e probability distribution of $X$ features given I am expecting this much profit margin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do that with a random forest(or any other discriminative model)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One suggestion for me could be to start with a generative model rather than a discriminative model. But, my understanding is generative model generally require a lot of data to train unless that make some very restrictive assumptions such as conditional independence of $X$'s in case of Naive Bayes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other suggestion could be to just switch $X$ and $Y$ and train a discriminative model. Now $X$ will be profit margin and $Y$ will be features of a t shirt. $P(Y|X)$ will directly give me the probability distribution of t shirt features, given a target profit margin. But this approach doesn't seem right to me, as I have always though of $X$ as casual variables and $Y$ to be effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, from what I have heard, similar question has been posed for drug discovery and algorithms have been designed which come up with candidate new drugs that have high degree of success. Can someone point me to research literature in this domain?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come across &lt;a href=&quot;https://blogs.nvidia.com/blog/2017/02/08/ai-drug-discovery/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/pdf/1610.02415.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; which talks about GANs being used for drug discovery. Generative adversial networks seem like a good fit for my problem statement so I have been reading about them. But one thing I understood is GAN generate samples in an unsupervised way. They try to produce sample which is like first capturing the underlying distribution of X and then sampling from that distribution. But I am interested in X|Y. X and Y are defined above. Should I explore something other than GANs? Any pointers please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Follow up Question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I have a GAN trained that has learned how to make t shirts(output sample Xs). How can I get the top 5 shirts for given Y?&lt;/p&gt;&#xA;" OwnerUserId="27108" LastEditorUserId="11097" LastEditDate="2018-01-05T09:35:08.027" LastActivityDate="2018-01-05T09:35:08.027" Title="Find optimal P(X|Y) given I have a model that has good performance when trained on P(Y|X)" Tags="|machine-learning|random-forest|generative-models|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26017" PostTypeId="1" AcceptedAnswerId="26024" CreationDate="2017-12-26T13:42:26.920" Score="3" ViewCount="324" Body="&lt;p&gt;If this is the dataframe, in pandas&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+--------+-------+&#xA;|  Col1  |  Col2  |  Sex  |&#xA;+--------+--------+-------+&#xA;| Value  | Value  | F     |&#xA;| Value  | Value  | M     |&#xA;| Value  | Value  | M     |&#xA;| Value  | Value  | Other |&#xA;| Value  | Value  | F     |&#xA;| Value  | Value  | M     |&#xA;+--------+--------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Should it converted to &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+--------+-------+-------+&#xA;|  Col1  |  Col2  | Sex_M | Sex_F |&#xA;+--------+--------+-------+-------+&#xA;| Value  | Value  |     0 |     1 |&#xA;| Value  | Value  |     1 |     0 |&#xA;| Value  | Value  |     1 |     0 |&#xA;| Value  | Value  |     0 |     0 |&#xA;| Value  | Value  |     0 |     1 |&#xA;| Value  | Value  |     1 |     0 |&#xA;+--------+--------+-------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+--------+-------+&#xA;|  Col1  |  Col2  |  Sex  |&#xA;+--------+--------+-------+&#xA;| Value  | Value  |    1  |&#xA;| Value  | Value  |    0  |&#xA;| Value  | Value  |    0  |&#xA;| Value  | Value  |    2  |&#xA;| Value  | Value  |    1  |&#xA;| Value  | Value  |    0  |&#xA;+--------+--------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43783" LastActivityDate="2017-12-26T16:02:29.300" Title="When using numerical duplicates for categorical data, new columns should be added or values be converted?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26018" PostTypeId="2" ParentId="26011" CreationDate="2017-12-26T13:51:14.730" Score="0" Body="&lt;p&gt;If you do not specify them, as it is clear in the signature of the functions you are referring to, the function will use the default value for them. For instance, you can see that the default value for stride is (2, 2) which means if you don't define the value of stride the method will use the mentioned value as the default value for the stride. Consequently, if you don't specify it, it does not mean there isn't such thing. In programming this approach helps programmers not to define many different overloads of a typical function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the response for the second question, again, if you don't specify the initial weights, &lt;code&gt;TensorFlow&lt;/code&gt; itself will use &lt;code&gt;Glorot&lt;/code&gt; method for initialization. So, definitely the filters will have values called weights in order to operations like convolution, actually cross correlation, be applicable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a recommendation, I highly suggest you taking a look at &lt;a href=&quot;http://yosinski.com/deepvis&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://gist.github.com/akiross/754c7b87a2af8603da78b46cdaaa5598&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-26T13:51:14.730" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26020" PostTypeId="2" ParentId="26013" CreationDate="2017-12-26T14:26:49.363" Score="4" Body="&lt;p&gt;I think you are mixing two different things here.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The feature_importance_ - this is an array which reflects how much each of the model's original features contributes to overall classification quality.  &lt;/li&gt;&#xA;&lt;li&gt;The features positions in the tree - this is a mere representation of the decision rules made in each step in the tree. A feature position(s) in the tree in terms of importance is not so trivial.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are some potential heuristics for understanding the relation between the two. If a feature dosn't appear in the tree it has 0 importance and generally the higher the feature is in the tree the more important it is (assuming its being compared to another feature on the same branch).&lt;/p&gt;&#xA;" OwnerUserId="27983" LastEditorUserId="29575" LastEditDate="2018-01-26T00:57:41.123" LastActivityDate="2018-01-26T00:57:41.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26021" PostTypeId="1" AcceptedAnswerId="26029" CreationDate="2017-12-26T14:37:21.760" Score="2" ViewCount="2859" Body="&lt;p&gt;I have a question regarding appropriate activation functions with environments that have both positive and negative rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In reinforcement learning, our output, I believe, should be the expected reward for all possible actions. Since some options have a negative reward, we would want an output range that includes negative numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So two questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want to have both negative and positive outputs, are you limited to just tanh and linear?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?&lt;/p&gt;&#xA;" OwnerUserId="42094" LastActivityDate="2017-12-26T18:48:51.047" Title="Negative Rewards and Activation Functions" Tags="|machine-learning|reinforcement-learning|q-learning|activation-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26024" PostTypeId="2" ParentId="26017" CreationDate="2017-12-26T16:02:29.300" Score="2" Body="&lt;p&gt;It depends of the algorithm you are using. For a linear model (linear / logistic regression, SVM...), you need to create dummy variables meaning features &quot;Sex_M&quot; and &quot;Sex_F&quot; as you noticed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you are using tree-based technics, create an integer typed column with Sex in [0, 1, 2] should be sufficient for these algorithms. Reason is, contrary to linear model, tree-based technics are non-linear and will evaluate all possible splits to partition your observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the way you map your categorical variale into integers can lead to different trees structures. Below is an example. Suppose you want to predict y variable using indexed feature &quot;Sex&quot;. On left chart, there is a slight difference between category 2 and categories 0 and 1 because categories are not oriented. It will results into two consecutive splits and should happen late in the tree building. However, on the right chart, the means difference between categories is larger that on left chart. So the split should happen earlier in the tree building. Moreover, you will only need 1 split to make the difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To conclude, I think simple indexing can be used with tree-based technics. I would also recommend you to sort your categorical features in an orderly way to easy tree learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZBwt9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZBwt9.png&quot; alt=&quot;Tree learning with indexed categorical variable&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2017-12-26T16:02:29.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26025" PostTypeId="1" CreationDate="2017-12-26T16:30:16.060" Score="2" ViewCount="991" Body="&lt;p&gt;I am implementing LSTM to a time series prediction which requires predicting both the expected value (i.e. the mean of the prediction), and the standard deviation (i.e. the interval that the future value should fall into).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have considered that one-hot encoding can return the possibility of the prediction of each category, which can then be used to both find the mean and the S.D. of the prediction, for example if the output of the LSTM is&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Category  | Possibility   &lt;/li&gt;&#xA;&lt;li&gt;[0,5)     |    0.1   &lt;/li&gt;&#xA;&lt;li&gt;[5,10)    |    0.2&lt;/li&gt;&#xA;&lt;li&gt;[10,15)   |    0.4   &lt;/li&gt;&#xA;&lt;li&gt;[15,20)   |    0.3&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I then conclude that the expected value of the prediction is&#xA;(2.5x0.1 + 7.5x0.2 + 12.5x0.4 + 17.5x0.3 = 12)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the S.D. is find in the similar manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is this method a valid method to find the mean and the S.D. of the prediction, why / why not?&lt;/p&gt;&#xA;" OwnerUserId="43793" LastActivityDate="2018-02-26T07:36:45.477" Title="LSTM predicting expected value and standard deviation with one-hot encoding" Tags="|predictive-modeling|prediction|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26027" PostTypeId="1" CreationDate="2017-12-26T17:33:36.667" Score="1" ViewCount="873" Body="&lt;p&gt;I'm using the adult dataset from &lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/Adult&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI&lt;/a&gt; and trying to predict the native country of a person, based on workclass, education.num, race, gender and income. I'm using sklearn and &lt;code&gt;KNeighborsClassifier&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this, predicting the test split I get 0.917252 accuracy,  with K = 15. But, when I use the test dataset (given in the UCI link), that accuracy goes to less than 1%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where am I going wrong?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = pd.read_csv('datasets/adult_data/adult.csv')&#xA;test = pd.read_csv('datasets/adult_data/adulttest.csv')&#xA;&#xA;# Cleaning null values&#xA;train = train[train[&quot;workclass&quot;] != &quot; ?&quot;]&#xA;train = train[train[&quot;occupation&quot;] != &quot; ?&quot;]&#xA;train = train[train[&quot;native.country&quot;] != &quot; ?&quot;]&#xA;test = test[test[&quot;workclass&quot;] != &quot; ?&quot;]&#xA;test = test[test[&quot;occupation&quot;] != &quot; ?&quot;]&#xA;test = test[test[&quot;native.country&quot;] != &quot; ?&quot;]&#xA;&#xA;category_col =['workclass', 'race', 'education','marital.status', 'occupation',&#xA;               'relationship', 'gender', 'native.country', 'income']&#xA;for col in category_col:&#xA;    b, c = np.unique(train[col], return_inverse=True) &#xA;    train[col] = c&#xA;&#xA;for col in category_col:&#xA;    b, c = np.unique(test[col], return_inverse=True) &#xA;    test[col] = c&#xA;&#xA;features = train.drop('native.country',axis=1)&#xA;target = train[&quot;native.country&quot;]&#xA;features_test = test.drop('native.country', axis=1)&#xA;features_target = test[&quot;native.country&quot;]&#xA;&#xA;features_name = ['workclass', 'education.num', 'race', 'gender', 'income']&#xA;&#xA;features = features[features_name]&#xA;features_test = features_test[features_name]&#xA;&#xA;#spliting data into train and test data&#xA;X_train,X_test,y_train,y_test = train_test_split(features,target,random_state = 12)&#xA;&#xA;k_values = np.arange(1,26)&#xA;scores = []&#xA;&#xA;for i in k_values:&#xA;    clf = KNeighborsClassifier(n_neighbors=i)&#xA;    clf.fit(X_train,y_train)&#xA;&#xA;    y_predict = clf.predict(features_test)&#xA;    # y_predict = clf.predict(X_train)&#xA;&#xA;    scores.append(metrics.accuracy_score(features_target, y_predict))&#xA;    # scores.append(metrics.accuracy_score(y_train, y_predict))&#xA;&#xA;print(&quot;Accuracy for {} is {}&quot;.format(np.argmax(scores),max(scores)))&#xA;&#xA;plt.plot(np.arange(1,26),scores)&#xA;plt.title('Variation of accuracy with K value, with all features')&#xA;plt.xlabel('K values')&#xA;plt.ylabel('Accuracy')&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43791" LastEditorUserId="29575" LastEditDate="2017-12-26T22:33:19.790" LastActivityDate="2017-12-27T05:40:30.423" Title="Good accuracy on train dataset with cross validation, but low accuracy on test dataset" Tags="|scikit-learn|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26028" PostTypeId="1" AcceptedAnswerId="26030" CreationDate="2017-12-26T18:42:37.020" Score="1" ViewCount="49" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Hadoop divides the input to a MapReduce job into fixed-size pieces&#xA;  called input splits, or just splits. Hadoop creates one map task for&#xA;  each split, which runs the user-defined map function for each record&#xA;  in the split. Having many splits means the time taken to process each&#xA;  split is small compared to the time to process the whole input. So if&#xA;  we are processing the splits in parallel, &lt;strong&gt;the processing is better&#xA;  load balanced when the splits are small&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why ?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastActivityDate="2017-12-26T19:01:31.483" Title="Hadoop and input informations divided in splits" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ClosedDate="2018-08-05T17:52:09.793" ContentLicense="CC BY-SA 3.0" />
  <row Id="26029" PostTypeId="2" ParentId="26021" CreationDate="2017-12-26T18:48:51.047" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally you want a linear output, unless you can guarantee scaling total possible reward to within a limited range such as $[-1,1]$ for $\text{tanh}$. Reminder this is not for estimating individual rewards but for total expected reward when following the policy you want to predict (typically the optimal policy eventually, but you will want the function to be able to estimate returns for other policies visited during optimisation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check carefully in the papers you mention, whether the activation function is applied in the output. If all rewards are positive, there should be no problem using ReLU for regression, and it may in fact help stabilise the network in one direction if the output is capped at a realistic minimum. However, you should not find in the literature a network with ReLU on output layer that needs to predict a negative return.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you do want to have both negative and positive outputs, are you limited to just tanh and linear?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are likely others, but linear will be far the most common.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It may sometimes be worth considering scaling rewards by a factor, or normalising them, to limit gradients, so that learning is stable. This was used in the Atari-games-playing DQN network to help the same algorithm tackle multiple games with different ranges of scoring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In continuous problems, the absolute value of reward is usually flexible, you are generally interested in getting the best mean reward per time step. So in that case you could scale so that minimum reward is 0, and use ReLU or other range limited transform in output - as above that &lt;em&gt;might&lt;/em&gt; help with numeric stability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In episodic problems without a fixed length, you typically don't have a such a free choice, because the agent is encouraged to end the episode quickly when rewards are negative. This is something you might want for instance if the goal is to complete a task as quickly or energy-efficiently as possible. A good example of this is &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Mountain_car_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mountain Car&lt;/a&gt;&quot; - granting only positive rewards in that scenario would be counter-productive, although you might still get acceptable results with positive reward only at the end and discounting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general case is that rewards can be arbitrarily scaled and centred for continuous problems without changing the agent's goal meaningfully, but only arbitrarily scaled for episodic problems.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2017-12-26T18:48:51.047" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26030" PostTypeId="2" ParentId="26028" CreationDate="2017-12-26T19:01:31.483" Score="1" Body="&lt;p&gt;All bigdata eco system works on something called parallel processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have to process 100gigs of file. If we didnt split the file, then all the 100 gigs should be processed by single JVM(single map).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we split the file into 1000 parts each of 100mb, then we can process each part with different JVM and apply the map function in less time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MPP: Massively parallel Processing&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="41529" LastActivityDate="2017-12-26T19:01:31.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26031" PostTypeId="1" CreationDate="2017-12-26T19:29:07.710" Score="3" ViewCount="2907" Body="&lt;p&gt;I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The query took 8-15 seconds in SAS, 20-30 seconds in Python, and 50-70 seconds in R. Does anyone know why? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;R Packages Used:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I used the RODBC package in R to query to the Oracle database. Then I tried the ROracle package, but both packages were much slower than SAS. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python Packages Used:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Python, I used Oracle's cx_Oracle package for the query. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot,&#xA;Sean&lt;/p&gt;&#xA;" OwnerUserId="43801" LastEditorUserId="43801" LastEditDate="2018-01-04T15:25:44.453" LastActivityDate="2018-01-04T15:25:44.453" Title="How to make R or Python as fast as SAS for ODBC Oracle queries?" Tags="|python|r|bigdata|sas|etl|" AnswerCount="1" CommentCount="1" ClosedDate="2017-12-27T04:36:38.147" ContentLicense="CC BY-SA 3.0" />
  <row Id="26033" PostTypeId="1" AcceptedAnswerId="26034" CreationDate="2017-12-26T19:48:12.840" Score="3" ViewCount="427" Body="&lt;p&gt;When doing entity recognition using NLTK, one gets as a result a &lt;code&gt;Tree&lt;/code&gt; with a bunch of words mapped to tags (eg. &lt;code&gt;Mark -&amp;gt; NNP&lt;/code&gt;, &lt;code&gt;first -&amp;gt; JJ&lt;/code&gt;, ...). It's not at all clear what all the tags stand for at first glance and I was unable to find any documentation about these tags in the &lt;code&gt;NLTK&lt;/code&gt; docs. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from nltk import word_tokenize, pos_tag, ne_chunk&#xA;&amp;gt;&amp;gt;&amp;gt; sentence = &quot;Mark and John are the first to work at Google from one years old in 39 years.&quot;&#xA;&amp;gt;&amp;gt;&amp;gt; print ne_chunk(pos_tag(word_tokenize(sentence)))&#xA;(S&#xA;  (PERSON Mark/NNP)&#xA;  and/CC&#xA;  (PERSON John/NNP)&#xA;  are/VBP&#xA;  the/DT&#xA;  first/JJ&#xA;  to/TO&#xA;  work/VB&#xA;  at/IN&#xA;  (ORGANIZATION Google/NNP)&#xA;  from/IN&#xA;  one/CD&#xA;  years/NNS&#xA;  old/JJ&#xA;  in/IN&#xA;  39/CD&#xA;  years/NNS&#xA;  ./.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I ended up looking into the source code to get the mapping. Posting in case anyone else runs into the same problem.&lt;/p&gt;&#xA;" OwnerUserId="43803" LastEditorUserId="43803" LastEditDate="2017-12-30T22:08:08.303" LastActivityDate="2018-01-05T05:43:17.513" Title="What is the tag mapping for entity recognition in nltk?" Tags="|nltk|named-entity-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26034" PostTypeId="2" ParentId="26033" CreationDate="2017-12-26T19:48:12.840" Score="3" Body="&lt;p&gt;Tag mapping according to &lt;a href=&quot;https://github.com/nltk/nltk/blob/develop/nltk/app/chunkparser_app.py#L52&quot; rel=&quot;nofollow noreferrer&quot;&gt;nltk source&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  'CC':   'Coordinating conjunction',   'PRP$': 'Possessive pronoun',&#xA;  'CD':   'Cardinal number',            'RB':   'Adverb',&#xA;  'DT':   'Determiner',                 'RBR':  'Adverb, comparative',&#xA;  'EX':   'Existential there',          'RBS':  'Adverb, superlative',&#xA;  'FW':   'Foreign word',               'RP':   'Particle',&#xA;  'JJ':   'Adjective',                  'TO':   'to',&#xA;  'JJR':  'Adjective, comparative',     'UH':   'Interjection',&#xA;  'JJS':  'Adjective, superlative',     'VB':   'Verb, base form',&#xA;  'LS':   'List item marker',           'VBD':  'Verb, past tense',&#xA;  'MD':   'Modal',                      'NNS':  'Noun, plural',&#xA;  'NN':   'Noun, singular or masps',    'VBN':  'Verb, past participle',&#xA;  'VBZ':  'Verb,3rd ps. sing. present', 'NNP':  'Proper noun, singular',&#xA;  'NNPS': 'Proper noun plural',         'WDT':  'wh-determiner',&#xA;  'PDT':  'Predeterminer',              'WP':   'wh-pronoun',&#xA;  'POS':  'Possessive ending',          'WP$':  'Possessive wh-pronoun',&#xA;  'PRP':  'Personal pronoun',           'WRB':  'wh-adverb',&#xA;  '(':    'open parenthesis',           ')':    'close parenthesis',&#xA;  '``':   'open quote',                 ',':    'comma',&#xA;  &quot;''&quot;:   'close quote',                '.':    'period',&#xA;  '#':    'pound sign (currency marker)',&#xA;  '$':    'dollar sign (currency marker)',&#xA;  'IN':   'Preposition/subord. conjunction',&#xA;  'SYM':  'Symbol (mathematical or scientific)',&#xA;  'VBG':  'Verb, gerund/present participle',&#xA;  'VBP':  'Verb, non-3rd ps. sing. present',&#xA;  ':':    'colon',&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43803" LastActivityDate="2017-12-26T19:48:12.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26035" PostTypeId="2" ParentId="26033" CreationDate="2017-12-26T19:59:14.077" Score="1" Body="&lt;p&gt;To extend on the answer of @Kevin S Lin. &lt;a href=&quot;https://en.wikipedia.org/wiki/Part-of-speech_tagging&quot; rel=&quot;nofollow noreferrer&quot;&gt;POS tagging&lt;/a&gt; or Part-of Speech tagging refers to:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the process of marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition and its context—i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph. A simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;More information on each particular tag can be found using the:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nltk.help.upenn_tagset()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although the &lt;code&gt;tagsets&lt;/code&gt; dataset needs to be downloaded. More on these matter can be found &lt;a href=&quot;https://stackoverflow.com/questions/15388831/what-are-all-possible-pos-tags-of-nltk&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-26T19:59:14.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26036" PostTypeId="2" ParentId="26031" CreationDate="2017-12-27T00:16:01.420" Score="3" Body="&lt;p&gt;This answer some of your questions from a Python perspective:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is Python any faster?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This question is a bit tricky to answer, it will depend on your usage of Python, but Python is not a fast language per se. However, the &lt;a href=&quot;https://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas library&lt;/a&gt; in Python have been reported to handle tables of 33M-100M rows, see &lt;a href=&quot;https://www.quora.com/Can-Python-Pandas-handle-10-million-rows-What-are-some-useful-techniques-to-work-with-the-large-data-frames&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. I myself have used to handle around 10M rows from a Postgres table. For a detailed experimentation using pandas, see &lt;a href=&quot;https://blog.treasuredata.com/blog/2016/07/26/testing-big-data-in-pandas-with-aws-ec2-instance-r3-8xlarge/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. In the link they apply some operations on datasets of 88M rows and 74 columns.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do we need Hadoop or parallel processing or something else to make R/Python as fast as SAS?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Before trying using Hadoop or Spark, I recommend you to follow some optimization tricks (tips): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1]&lt;/a&gt; A Beginner’s Guide to Optimizing Pandas Code for Speed&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.dataquest.io/blog/pandas-big-data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt; Using pandas with large data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; (Don't use Hadoop - your data isn't that big) can be useful too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also for a comparison of SAS and pandas you can read the pandas documentation on such &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/comparison_with_sas.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;comparison&lt;/a&gt; or &lt;a href=&quot;http://www.richardafolabi.com/blog/python-pandas-vs-sas-head-to-head-data-analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it because R and Python are matrix-based languages and SAS is more&#xA;  traditional-database-oriented?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Python is not a matrix-based language, as far as I know the language does not offer &lt;strong&gt;by default&lt;/strong&gt; any capabilities for handling matrices. I think your referring to the &lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;numpy/scipy&lt;/a&gt; stack, this is a separated library.   &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-27T00:16:01.420" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26037" PostTypeId="2" ParentId="12645" CreationDate="2017-12-27T02:45:11.077" Score="4" Body="&lt;p&gt;Or, you could simply make use of the info method for dataframe objects:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.info()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which provides counts of non-null values for each column.&lt;/p&gt;&#xA;" OwnerUserId="43808" LastActivityDate="2017-12-27T02:45:11.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26038" PostTypeId="1" CreationDate="2017-12-27T03:09:02.637" Score="1" ViewCount="2437" Body="&lt;p&gt;Suppose I have a data like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iYMMM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iYMMM.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;and I want to change the value of name and gender into an integer, and I have a dictionary like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TFhbZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TFhbZ.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I understand that I can use replace function in pandas, but I don't know how to use an excel dictionary to replace the value&lt;/p&gt;&#xA;&lt;p&gt;thank you&lt;/p&gt;&#xA;" OwnerUserId="43714" LastEditorUserId="98307" LastEditDate="2020-08-11T13:42:56.447" LastActivityDate="2020-08-11T13:42:56.447" Title="How to Replace an object in Pandas array using replace with dictionary from excel file?" Tags="|python|pandas|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26040" PostTypeId="2" ParentId="26027" CreationDate="2017-12-27T05:40:30.423" Score="1" Body="&lt;p&gt;I think you are using commented lines in the loop to calculate the cross validation accuracy.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# y_predict = clf.predict(X_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; y_predict = clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then you should check accuracy score with 'y_test' not 'y_train'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are predicting on the data KNN has learnt on. And since KNN algorithms kind of learns the data. You are getting good accuracy.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2017-12-27T05:40:30.423" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26041" PostTypeId="1" CreationDate="2017-12-27T06:23:06.710" Score="1" ViewCount="332" Body="&lt;p&gt;I am using Orange Data mining 3.8 to classify a dataset using &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Leave-one-out_cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;leave one out cross validation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the results ($AUC$, $CA$, $F1$) are averaged; How can I get the results ($AUC$, $CA$, $F1$) of every fold?&lt;/p&gt;&#xA;" OwnerUserId="43812" LastEditorUserId="29575" LastEditDate="2017-12-27T14:47:28.457" LastActivityDate="2018-06-28T06:16:11.833" Title="Getting results of folds in orange canvas" Tags="|classification|cross-validation|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26042" PostTypeId="1" AcceptedAnswerId="26056" CreationDate="2017-12-27T09:34:07.903" Score="0" ViewCount="334" Body="&lt;p&gt;Before facing &lt;a href=&quot;https://datascience.stackexchange.com/q/25925/28175&quot;&gt;this&lt;/a&gt; question, I always thought non-learnable problems are those which the provided data for the problem has high amount of outliers, those which don't have sufficient features or those for which the Bayes error is large because of having same features with different labels. As you can see, it seems that the data is fine because the learning should be comparable with human level inference. A human can distinguish between even or odd numbers by just looking at them. I know that we as human begins, do modulus two operation in our mind to decide whether a number is even or odd, the feature extraction part, but we are doing that with just the number itself. It is clear that we can not find a decision boundary to be able to generalize because the inputs have alternative behavior. 1 is even 2 is odd, 3 is even 4 is odd and all the other numbers in this manner. I want to know this kind of problem ,which does not have the mentioned problems which may cause an algorithm not to learn, has any special name?&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-11T21:31:48.477" LastActivityDate="2018-02-11T21:31:48.477" Title="Non trainable problems" Tags="|machine-learning|neural-network|deep-learning|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26045" PostTypeId="2" ParentId="26009" CreationDate="2017-12-27T11:32:02.593" Score="1" Body="&lt;p&gt;The paragraph you mentioned explains a the parametric procedure of creating training data and testing data from a given data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us take an example let us consider that the distribution of a certain dataset follows Normal distribution (Gaussian) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7Mkgw.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7Mkgw.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that 68% of the data lies near the mean of the dataset. Also since the dataset has been identified as gaussian we also know the expected probability function (pdf) of the data set assuming we know the mean and variance of the given dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(x) = \frac{1}{\sqrt{2 \pi \sigma ^2}} e^{\frac{-(x-\mu)^2}{2 \sigma ^2}}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we have the formula we can use random variate generation techniques on this formula to create training and test data separately which can be used for the model to learn and test its efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To learn more about random variate generation I'd direct you to this resource &lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/0131446797&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It has a great chapter which can help you with understanding the statistical technique behind it.  &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2017-12-27T11:32:02.593" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26046" PostTypeId="2" ParentId="25973" CreationDate="2017-12-27T11:53:41.000" Score="0" Body="&lt;p&gt;Use a precomputed distance matrix, and &lt;code&gt;distance=&quot;precomputed&quot;&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HAC will compute a distance matrix anyway.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-27T11:53:41.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26048" PostTypeId="1" AcceptedAnswerId="26049" CreationDate="2017-12-27T16:00:00.310" Score="0" ViewCount="473" Body="&lt;p&gt;I am doing basic data analysis on an csv file in jupyter notebook&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def answer_two():&#xA;    return (df['Gold']-df['Gold.1']).argmax()&#xA;answer_two&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above code snippet is to subtract two columns of the dataframe.I am expecting an answer in the form of a country name but I am getting the following output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;function __main__.answer_two&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am unable to figure out why this is happening .Occasionally ,the required output in the form of country name is coming but not running it everytime.&lt;/p&gt;&#xA;" OwnerUserId="43833" LastActivityDate="2017-12-27T16:04:21.573" Title="Unable to print in Jupyter Notebook using Pandas" Tags="|data-cleaning|pandas|jupyter|ipython|data-wrangling|" AnswerCount="1" CommentCount="1" ClosedDate="2018-01-04T23:09:54.947" ContentLicense="CC BY-SA 3.0" />
  <row Id="26049" PostTypeId="2" ParentId="26048" CreationDate="2017-12-27T16:04:21.573" Score="1" Body="&lt;p&gt;Instead of calling &lt;code&gt;answer_two&lt;/code&gt;, call &lt;code&gt;answer_two()&lt;/code&gt;. you are referring to the function object now. You have to call the function.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-27T16:04:21.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26050" PostTypeId="2" ParentId="26025" CreationDate="2017-12-27T16:31:28.457" Score="2" Body="&lt;p&gt;The standard deviation and mean of a categorical variable is not meaningful. It looks like the original data are from a range of [0, 20), and the space has been discretized. Now, instead of ranges, you have ordered categories 1 through 4. The individual numbers within each category have lost their meaning. For more details, see the accepted answer &lt;a href=&quot;https://stats.stackexchange.com/questions/51248/how-can-i-find-the-standard-deviation-in-categorical-distribution&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2017-12-27T16:31:28.457" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26051" PostTypeId="1" CreationDate="2017-12-27T16:54:53.177" Score="1" ViewCount="44" Body="&lt;p&gt;Google Trip is now performing automated clustering of travel items into trip bundles. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://support.google.com/inbox/answer/6228360?hl=en-GB&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://support.google.com/inbox/answer/6228360?hl=en-GB&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if in your mail you have a flight confirmation mail and a hotel confirmation mail related to the same trip, Google will cluster them automatically into a trip bundle.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;They scan all your e mail&lt;/li&gt;&#xA;&lt;li&gt;They find the ones related to trips&lt;/li&gt;&#xA;&lt;li&gt;They cluster those for the same trip into a bundle.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question regards point 3. Which kind of algorithm is needed to perform such task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider that the task is unsupervised, because the user can not interact with the system while creating the bundles..&lt;/p&gt;&#xA;" OwnerUserId="23482" LastActivityDate="2017-12-27T18:59:38.933" Title="How Google Trip Bundle works?" Tags="|clustering|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26053" PostTypeId="1" AcceptedAnswerId="26059" CreationDate="2017-12-27T18:37:09.750" Score="1" ViewCount="172" Body="&lt;p&gt;If I have fitted training data to a probability distribution, e.g. a poisson distribution, how can I test this fit on some test data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To fit the poisson distribution I am using R's &lt;code&gt;fitdistrplus&lt;/code&gt; package that using MLE for determining the optimal coefficients of a given distribution. Therefore, I have the estimated $\lambda$ for a poisson distribution based on my training data but I am not sure how to test this on some unseen test data.&lt;/p&gt;&#xA;" OwnerUserId="38503" LastActivityDate="2017-12-27T21:18:10.907" Title="Testing fit of probability distribution" Tags="|probability|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26054" PostTypeId="1" CreationDate="2017-12-27T18:44:11.313" Score="1" ViewCount="120" Body="&lt;p&gt;I'm looking for a paper that does some comparisons between neural networks (deep learning) and traditional methods in order to prove that DL usually performs better with enough data.&#xA;I know this is &quot;known&quot; fact but I'm struggling to find a good paper doing some research in this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="18501" LastActivityDate="2018-07-17T20:08:07.200" Title="Paper proving neural networks with more data are better in NLP than traditional methods" Tags="|neural-network|deep-learning|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26055" PostTypeId="2" ParentId="26051" CreationDate="2017-12-27T18:59:38.933" Score="1" Body="&lt;p&gt;Company secrets - Google does not disclose what they actually use. Any answer to this is pure speculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It probably is not even just &quot;one&quot; algorithm, but many. But it may be as simple as finding overlapping dates and locations, assuming that people cannot be in two places at once. You could test this: book a flight, and a hotel on a different destination during the same time interval. I wouldn't be surprised if they nevertheless show up as one &quot;trip&quot; (and it doesn't really degrade user experience much).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will very likely be a &lt;em&gt;supervised&lt;/em&gt; approach. Which does not mean &quot;interactive&quot;. On the contrary, unsupervised approaches will likely need interactions to choose parameters. Using some training data (e.g., from employees, or user interactions) to improve matching quality looks promising, though.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-27T18:59:38.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26056" PostTypeId="2" ParentId="26042" CreationDate="2017-12-27T19:26:11.683" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/161189/train-a-neural-network-to-distinguish-between-even-and-odd-numbers&quot;&gt;In this discussion&lt;/a&gt;, it is described how a neural network that distinguishes odd and even numbers can be constructed.&#xA;Your question can be rephrased more generally: Is there a function that cannot be learned by a machine learning algorithm. This question was discussed &lt;a href=&quot;https://stackoverflow.com/questions/25609347/can-neural-networks-approximate-any-function-given-enough-hidden-neurons&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://towardsdatascience.com/can-neural-networks-really-learn-any-function-65e106617fc6&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Both discussions refer to the Universal Approximation Theorem that basically states that any computable function on a given finite range can be approximated by a neural network. So, what is left is &lt;a href=&quot;https://en.wikipedia.org/wiki/Computable_function#Uncomputable_functions_and_unsolvable_problems&quot; rel=&quot;nofollow noreferrer&quot;&gt;uncomputable functions or undecidable problems&lt;/a&gt;. These cannot be learned by a machine-learning algorithm.&lt;/p&gt;&#xA;" OwnerUserId="42142" LastActivityDate="2017-12-27T19:26:11.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26057" PostTypeId="1" CreationDate="2017-12-27T20:26:21.423" Score="1" ViewCount="30" Body="&lt;p&gt;In a scientific paper (Neural Networks: A Systematic Introduction, page 86) about the Perceptron Algorithm I found:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A good initial heuristic is to start with the average of the positive input vectors minus the average of the negative input vectors. In many cases this yields an initial vector near the solution region.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can you show me geometrically why this is true?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="29575" LastEditDate="2017-12-28T00:17:40.363" LastActivityDate="2017-12-28T00:17:40.363" Title="Perceptron algorithm initial vector" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26058" PostTypeId="1" CreationDate="2017-12-27T21:12:23.413" Score="1" ViewCount="171" Body="&lt;p&gt;What would be the best Sequence Mining Algorithm to use for Hand Written digit Recognition system (other then Deep learning)?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="1330" LastEditDate="2021-07-16T17:22:43.457" LastActivityDate="2021-08-24T08:00:55.857" Title="What is the best non-deep learning hand written digits recognition system?" Tags="|machine-learning|data-mining|sequential-pattern-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26059" PostTypeId="2" ParentId="26053" CreationDate="2017-12-27T21:18:10.907" Score="4" Body="&lt;p&gt;Use chi-square test to check the goodness of fit to a specific distribution&#xA;&lt;a href=&quot;http://courses.wcupa.edu/rbove/Berenson/10th%20ed%20CD-ROM%20topics/section12_5.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://courses.wcupa.edu/rbove/Berenson/10th%20ed%20CD-ROM%20topics/section12_5.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43844" LastActivityDate="2017-12-27T21:18:10.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26060" PostTypeId="1" CreationDate="2017-12-27T21:30:24.617" Score="2" ViewCount="450" Body="&lt;p&gt;In order to identify the similarity between images (products) I want to use a neural network approach similar to &lt;a href=&quot;https://github.com/paucarre/tiefvision&quot; rel=&quot;nofollow noreferrer&quot;&gt;TiefVision&lt;/a&gt;. This pre-trained neural network is basically translating the images into a feature vectors and then creating a similarity measure between the images using a distance measure between the vectors. To make it more tangible have a look at a &lt;a href=&quot;https://i.stack.imgur.com/PkKBA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;2D visual representation&lt;/a&gt; below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to take it one step further: When a single user &quot;likes&quot; multiple images, I want to average their feature vectors. This will result in a new vector, which I want to use to calculate recommendations. My assumption is that images close to the combined feature vector will possess similar features towards all &quot;liked&quot; images together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I wonder: Is my thinking flawed - maybe because averaging the vectors will simply lead to entirely different images, or will it actually produce images with similar features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PkKBA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PkKBA.png&quot; alt=&quot;Image&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://indico.io/blog/wp-content/uploads/2016/01/clothing-similarity-tsne.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;indico.io&lt;/a&gt;)&lt;/sub&gt;  &lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="25180" LastEditDate="2019-02-18T09:40:04.510" LastActivityDate="2024-02-06T02:08:30.913" Title="Image similarity: Similarity of mixed vector" Tags="|neural-network|similarity|image-recognition|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="26061" PostTypeId="1" AcceptedAnswerId="26081" CreationDate="2017-12-27T22:21:32.287" Score="3" ViewCount="3344" Body="&lt;p&gt;Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our weights. This completes processing 1 minibatch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read these posts &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/245799/lstm-how-to-feed-the-network-with-a-mini-batch-when-to-reset-the-lstm-state&quot;&gt;[1]&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://groups.google.com/forum/#!topic/theano-users/chsxlScGJIk&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;,&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/a/245951/187816&quot;&gt;about padding entries in a minibatch so they have same length&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;and &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/4egr3v/in_rnnlstm_why_passing_the_states_across/&quot; rel=&quot;nofollow noreferrer&quot;&gt;about preserving the cell state&lt;/a&gt; but the following is still unclear to me:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question part a:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How a minibatch entity would look like for LSTM? &#xA;Say, I want it to reproduce Shakespeare, letter by letter (30 characters to choose from). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I launch LSTM, let it predict for 200 characters of a poem, then perform back propagation. (hence, my LSTM works with 200 timesteps). Does this mean my minibatch consist of 1 example whose length is 200?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question part b:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted to launch 63 other minibatches in parallel, would I just pick 63 extra poems?  (&lt;strong&gt;Edit:&lt;/strong&gt; Original answer doesn't mention this explicitly, but we don't train minibatches in parallel. We train on 1 minibatch, but train its &lt;em&gt;examples&lt;/em&gt; in parallel)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question part C:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I wanted each minibatch to consist of 10 different examples, what would such examples be, and how would they be different from 'what I perceive as a minibatch'?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-05-16T11:06:40.123" LastActivityDate="2019-05-16T11:06:40.123" Title="how does minibatch for LSTM look like?" Tags="|lstm|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26062" PostTypeId="2" ParentId="22537" CreationDate="2017-12-27T23:24:15.540" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is does LSTM by nature, also take previous output values besides feature vector as new feature element for the next output values?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You didn't specify which framework you are using, so I will explain for a custom (c++ etc) implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each timestep LSTM will:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take a new input from you (for example, as a &lt;a href=&quot;https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot encoded vector&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Internally&lt;/strong&gt; fetch a separate vector which is the LSTM's output from Timestep [t-1]&lt;/li&gt;&#xA;&lt;li&gt;These two vectors are like a 'fuel' and will be used to shape out a Cell at [t]&lt;/li&gt;&#xA;&lt;li&gt;The result gets produced for this Timestep [t]&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Back Propagation creates gradients for the weights within the LSTM, but also gradients for these two vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because there are multiple timesteps, you get a Gradient during each Timestep, which has to be &lt;a href=&quot;http://arunmallya.github.io/writeups/nn/lstm/index.html#/8&quot; rel=&quot;nofollow noreferrer&quot;&gt;added&lt;/a&gt; (component-wise) before being used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, this last url link has a blue arrow at the bottom right of the screen, so you can click to see the next slides.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-02-27T18:08:32.307" LastActivityDate="2019-02-27T18:08:32.307" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26063" PostTypeId="1" AcceptedAnswerId="26074" CreationDate="2017-12-28T01:14:46.563" Score="1" ViewCount="167" Body="&lt;p&gt;Which classification algorithms can handle 24000 features? What are their pros and cons?&lt;/p&gt;&#xA;" OwnerUserId="43398" LastActivityDate="2018-01-02T08:34:12.890" Title="Which classification algorithms can handle 24000 features" Tags="|machine-learning|classification|algorithms|" AnswerCount="2" CommentCount="2" ClosedDate="2018-01-03T21:34:04.397" ContentLicense="CC BY-SA 3.0" />
  <row Id="26064" PostTypeId="1" CreationDate="2017-12-28T01:39:07.190" Score="1" ViewCount="33" Body="&lt;p&gt;In &lt;em&gt;Keras&lt;/em&gt;, the area outside of the input is normally padded with zeros. Could we put other values instead of that? Because the characteristic of zeros to border is something we don't care about. &lt;/p&gt;&#xA;" OwnerUserId="43848" LastEditorUserId="29575" LastEditDate="2017-12-28T15:51:08.240" LastActivityDate="2017-12-28T15:51:08.240" Title="How to pad the input of convolution layer with other values instead of zero" Tags="|machine-learning|deep-learning|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26065" PostTypeId="1" CreationDate="2017-12-28T03:53:45.333" Score="1" ViewCount="237" Body="&lt;p&gt;I have 667 features and I want to find features that have a significant boundary between a binary class label before I apply a classification model (e.g Naive Bayes/ SVM) to improve classification model learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I know is, if the features' values between the two classes are overlapping, this will cause poor classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I have done a 2 samples t-test to calculate the statistical significance of features between binary class label.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy import stats&#xA;p=[]&#xA;failure = [1]&#xA;&#xA;#separate out the non-failure and failure group data into 2 dataframes to calculate the &#xA;t-test between each feature.&#xA;&#xA;df_failure= df.loc[df['label'].isin(failure)]&#xA;df_nonfailure= df.loc[~df['label'].isin(failure)]&#xA;&#xA;for x in listofname:&#xA;&#xA;     p.append((stats.ttest_ind(df_failure[x],df_nonfailure[x], equal_var=False)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is, is this a good features selection approach beside recursive feature elimination/wrapper method? Is there any similar method out there?  &lt;/p&gt;&#xA;" OwnerUserId="41220" LastActivityDate="2024-01-11T06:04:05.420" Title="statistical significance test between binary label features" Tags="|python|statistics|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26066" PostTypeId="1" CreationDate="2017-12-28T06:27:05.853" Score="1" ViewCount="125" Body="&lt;p&gt;I'm a .NET developer trying to learn more about AI/Machine Learning.&lt;/p&gt;&#xA;&lt;p&gt;I have a dataset with few variables and known results. I'm looking for a language/framework/tool/approach to analyze the data and identify which variable(s) contributed the most towards the results and eventually predict what the result will be given only the variables.&lt;/p&gt;&#xA;&lt;p&gt;I live in a Microsoft eco-system. Could someone suggest a starting point to research? Does this sound like a common problem?&lt;/p&gt;&#xA;" OwnerUserId="43856" LastEditorUserId="85045" LastEditDate="2020-11-15T18:00:16.170" LastActivityDate="2020-11-15T18:00:16.170" Title="Finding relationships in datasets and predicting data" Tags="|predictive-modeling|correlation|tools|.net|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="26068" PostTypeId="1" CreationDate="2017-12-28T08:26:22.350" Score="4" ViewCount="3929" Body="&lt;p&gt;I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?&lt;/p&gt;&#xA;" OwnerUserId="40625" LastEditorUserId="28175" LastEditDate="2017-12-28T10:17:54.243" LastActivityDate="2022-02-23T10:49:45.327" Title="Are there any differences between Recurrent Neural Networks and Residual Neural Networks?" Tags="|machine-learning|neural-network|deep-learning|lstm|cnn|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26069" PostTypeId="1" CreationDate="2017-12-28T09:21:33.593" Score="1" ViewCount="116" Body="&lt;p&gt;I have been working on a project and we were trying to convert a PSD (Adobe Photoshop) file to a HTML for web applications as well as a Layout XML for android. We worked our way to generate basic skeletal html/xml but hit a wall for complex scenarios such as identifying separate divs and components.&lt;/p&gt;&#xA;&lt;p&gt;Our initial approach was to standardize the PSD and get metadata about each component from PSD but due to it's limitations we could only add metadata as a prefix to the name of each layer or component for example : If PSD has a component which is supposed to be button we would name it btn_submit and convert this component into button in HTML/XML based on btn suffix. But this was not scalable and seems more like Expert Systems. So our team decided to move to Machine Learning.&lt;/p&gt;&#xA;&lt;p&gt;We are considering supervised learning as our first approach so I would like to know which algorithms can guide me to achieve this and is it better to use Expert System instead of Machine Learning in this case?&lt;/p&gt;&#xA;" OwnerUserId="43860" LastEditorUserId="1330" LastEditDate="2021-07-30T18:15:53.973" LastActivityDate="2021-07-30T18:15:53.973" Title="Algorithm to parse PSD into html/XML?" Tags="|machine-learning|machine-translation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26070" PostTypeId="2" ParentId="26068" CreationDate="2017-12-28T09:41:04.557" Score="5" Body="&lt;p&gt;No, they are not. Residual networks are deep networks with building blocks characterised by having skip connections across layers to facilitate backwards flow of gradients (or approximation of identity functions). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent networks are a different beast altogether, with cyclic connections, variable size inputs, ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For residual nets, see &lt;a href=&quot;https://arxiv.org/abs/1512.03385&quot; rel=&quot;noreferrer&quot;&gt;the paper&lt;/a&gt;.&#xA;For recurrent nets, see e.g. &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;this wonderful post&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31718" LastActivityDate="2017-12-28T09:41:04.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26071" PostTypeId="1" CreationDate="2017-12-28T09:46:54.970" Score="1" ViewCount="25" Body="&lt;p&gt;I have a list of elements in a column. Example:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;UID.................Flow&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2............................qpwoei, alskdj, zmxncb, qmwneb, zpxoci, glafhk&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3............................uinbwe, cvbnfg, sfjvlr, sdfvll, sfvlas, fogjwl&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UID and Flow are the column names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for each UID, I want to create a graph which shows the flow like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - qwerty -&gt; asdfgh -&gt; zxcvbn -&gt; poiuyt -&gt; lkjhgf -&gt; mnbvcx&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember, if any element is repeated, it should point back to it, unlike just putting arrows in between all the elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So ultimately, for every UID, I should be able to see this graph which shows me the flow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone help me build the code for this? Or is there any package which we can use to create graphical viz like these? Thanks in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-- Anirudh&lt;/p&gt;&#xA;" OwnerUserId="43862" LastEditorUserId="43862" LastEditDate="2017-12-28T09:59:20.430" LastActivityDate="2017-12-28T09:59:20.430" Title="Create a graphical viz of list of elements residing in a column in desired order" Tags="|machine-learning|neural-network|nlp|markov-process|bayesian-networks|" AnswerCount="0" CommentCount="0" ClosedDate="2018-01-05T05:44:32.090" ContentLicense="CC BY-SA 3.0" />
  <row Id="26072" PostTypeId="2" ParentId="26068" CreationDate="2017-12-28T09:56:24.367" Score="8" Body="&lt;p&gt;Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of &lt;code&gt;ResNet&lt;/code&gt; paper had seen that by stacking many layers of convolution and dense layers, the learning did not increase although they used &lt;code&gt;ReLU&lt;/code&gt; activation and batch normalization. They used a concept named skip connection which helped the nets to &lt;em&gt;learn&lt;/em&gt; whether the input to a typical layer should be preserved or it should be transformed by that layer. Using this concept allowed them to increase the number of layers without hesitating whether they would have vanishing/exploding gradients. The concept of residual nets was originally this. The paper uses this concept for spatial data but recently I've seen people debating using them in temporal cases too __time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent nets are used in temporal domains. Tasks like sequence classification are examples of their usage. In this domain the net should know the information of previous seen data. Well known examples of these nets are &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;LSTMS&lt;/a&gt;. Early recurrent nets had vanishing/exploding gradient problem too. but after years &lt;code&gt;LSTM&lt;/code&gt;s get popular amongst deep-learning practitioners. They defined a concept named gates which could learn when to forget and when to keep the previous data. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2017-12-28T09:56:24.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26073" PostTypeId="1" CreationDate="2017-12-28T10:05:23.133" Score="1" ViewCount="21" Body="&lt;p&gt;All is in the question:&#xA;Consider the case we specify an output size of 64 but with current convolution layer parameters we can get 128 features.&#xA;How is this handled in general and using keras in particular?&lt;/p&gt;&#xA;" OwnerUserId="40515" LastActivityDate="2017-12-28T10:05:23.133" Title="How does keras handle convolution in case the size of the output is less than the possible number of extracted features using convolution?" Tags="|keras|convolution|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26074" PostTypeId="2" ParentId="26063" CreationDate="2017-12-28T10:57:40.797" Score="2" Body="&lt;p&gt;Deep Learning Algorithms and Graphical model algorithms can handle that scale of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example a typical parsing algorithm using &lt;a href=&quot;https://taku910.github.io/crfpp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CRF++&lt;/a&gt; computes millions of features. &#xA;In case of Deep Learning, A typical image of 256*256*3 has to deal with 196608 number of features where each pixel in image is a feature.&lt;/p&gt;&#xA;" OwnerUserId="43369" LastActivityDate="2017-12-28T10:57:40.797" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26075" PostTypeId="1" CreationDate="2017-12-28T11:15:12.643" Score="1" ViewCount="210" Body="&lt;p&gt;Suppose in the multi-arm bandit problem I know my rewards are distributed as $0$ or $1$ i.e according to a Bernoulli distribution rather than the condition that they lie in the range $[0,1]$. Does anyone know if we can do better with our confidence bounds with this restricted condition? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular how does the upper confidence bound algorithm change and what is the corresponding upper bound on the expected regret? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone provide links to a paper or a set of notes on this particular problem?&lt;/p&gt;&#xA;" OwnerUserId="43865" LastEditorUserId="29575" LastEditDate="2017-12-28T15:54:23.660" LastActivityDate="2017-12-28T15:54:23.660" Title="Multi-arm bandit problem for bernoulli reward distribution" Tags="|machine-learning|reinforcement-learning|probability|randomized-algorithms|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26076" PostTypeId="1" AcceptedAnswerId="26092" CreationDate="2017-12-28T11:15:27.280" Score="1" ViewCount="450" Body="&lt;p&gt;My question may sound like a duplicate of, for example, &lt;a href=&quot;https://datascience.stackexchange.com/questions/25717/how-is-that-possible-that-a-reward-function-depends-both-on-the-next-state-and-a&quot;&gt;How is that possible that a reward function depends both on the next state and an action from current state?&lt;/a&gt;&#xA;but I still feel confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In neural network approximation of the Q function, I follow the experience replay routine. Papers and manuals suggest storing state, action, received reward, and next state information in the experience replay. However when one is about to calculate a maximum of Q value based on the next action, which state do they use: state or next state? If the next state is used to calculate the max Q, then what is the whole purpose of storing the previous state and action information?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example (2D world):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An agent is at cell A1 (state). His goal is to get to C3 to get a positive reward. He then moves to B2 (action); a received reward is, let's say, 0. Next state is B2 (after the action was done). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?&lt;/p&gt;&#xA;" OwnerUserId="42953" LastEditorUserId="42953" LastEditDate="2017-12-28T12:03:39.313" LastActivityDate="2017-12-28T20:40:59.250" Title="state-action-reward-new state: confusion of terms" Tags="|deep-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26079" PostTypeId="1" AcceptedAnswerId="26080" CreationDate="2017-12-28T14:03:28.563" Score="5" ViewCount="4139" Body="&lt;p&gt;Is it legal, at least as long as I am not &quot;selling&quot; the video under my name, to scrape YouTube videos to train a neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is not, is there a procedure to get permission for the above? I am in academia and require a massive amount of video data that YouTube would be the perfect source in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I couldn't figure out where to post this questions, as it is not &lt;em&gt;technically&lt;/em&gt; technical. I am posting it here only because I believe someone here would know a bit or two. Would appreciate any suggestions regarding community this question might have a better chance in. What about &lt;a href=&quot;https://law.stackexchange.com/&quot;&gt;Law&lt;/a&gt;?&lt;/p&gt;&#xA;" OwnerUserId="43875" LastEditorUserId="43875" LastEditDate="2017-12-28T14:11:21.187" LastActivityDate="2017-12-29T07:35:17.753" Title="Is it legal to scrape YouTube videos for training data?" Tags="|machine-learning|neural-network|data-mining|dataset|" AnswerCount="1" CommentCount="1" ClosedDate="2018-01-03T15:38:00.680" ContentLicense="CC BY-SA 3.0" />
  <row Id="26080" PostTypeId="2" ParentId="26079" CreationDate="2017-12-28T14:13:55.707" Score="6" Body="&lt;p&gt;It will depend on the rights of the videos themselves, although probably the terms of service of youtube wouldn't agree with it anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you have the &lt;a href=&quot;https://research.google.com/youtube8m/&quot; rel=&quot;nofollow noreferrer&quot;&gt;YouTube 8M dataset&lt;/a&gt;, released by Google for research purposes.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;YouTube-8M is a large-scale labeled video dataset that consists of millions of YouTube video IDs and associated labels from a diverse vocabulary of 4700+ visual entities. It comes with precomputed state-of-the-art audio-visual features from billions of frames and audio segments, designed to fit on a single hard disk. This makes it possible to get started on this dataset by training a baseline video model in less than a day on a single machine! At the same time, the dataset's scale and diversity can enable deep exploration of complex audio-visual models that can take weeks to train even in a distributed fashion. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: And Facebook and MIT just released &lt;a href=&quot;http://slac.csail.mit.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SLAC&lt;/a&gt; (dataset to be released soon, as well as pretrained models for transfer learning).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This project presents a novel video dataset, named SLAC (Sparsely Labeled ACtions), for action recognition and localization. It consists of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="31718" LastEditorUserId="31718" LastEditDate="2017-12-29T07:35:17.753" LastActivityDate="2017-12-29T07:35:17.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26081" PostTypeId="2" ParentId="26061" CreationDate="2017-12-28T14:16:08.290" Score="3" Body="&lt;p&gt;I think you need to distinguish between training and execution of the model. During training, you can use batches, which in your case will be different fragments from Shakespeare. So, a batch will be a list of fragments, and the language model will start from the first character on each element of the batch and do the backward and forward pass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you execute the model, once it is trained, you would like to see one single example, in which case you can set the batch size to one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe this answers your three questions.&lt;/p&gt;&#xA;" OwnerUserId="23492" LastActivityDate="2017-12-28T14:16:08.290" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="26082" PostTypeId="2" ParentId="12055" CreationDate="2017-12-28T14:18:24.827" Score="0" Body="&lt;p&gt;A recurrent neural network can take unstructured data, such as video or raw text, to make a prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the model can use as input the text from a large set of emails, and from there try to forecast something.&lt;/p&gt;&#xA;" OwnerUserId="23492" LastActivityDate="2017-12-28T14:18:24.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26083" PostTypeId="2" ParentId="26038" CreationDate="2017-12-28T15:19:15.440" Score="1" Body="&lt;p&gt;Assuming you have a pair of csv files: &quot;replace.csv&quot; representing the first table and &quot;table.csv&quot; representing the second table i.e. &quot;dict&quot;. Use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;frame = pd.read_csv('replace.csv')&#xA;table = pd.read_csv('table.csv', header=None)&#xA;&#xA;mapping = dict([(k, v) for k, v in table.values])&#xA;&#xA;frame.replace(to_replace={&quot;name&quot;: mapping, &quot;gender&quot;: mapping}, inplace=True)&#xA;print frame&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The line &lt;code&gt;mapping = dict([(k, v) for k, v in table.values])&lt;/code&gt; is the one that transforms your dataframe into a dict. For more on transforming a dataframe into a dictionary see the &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_dict.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt;, also &lt;a href=&quot;https://stackoverflow.com/questions/18695605/python-pandas-dataframe-to-dictionary&quot;&gt;this question&lt;/a&gt; provides different ways of transforming a dataframe into a dictionary. For reading excel files, instead of csv files, see &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2017-12-28T15:19:15.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26084" PostTypeId="2" ParentId="12055" CreationDate="2017-12-28T16:01:51.503" Score="1" Body="&lt;p&gt;How you use data depends mostly on the domain problem you're trying to address.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case you mention, it would depend on what exactly you're trying to forecast. For example, if you're trying to forecast sales or renewalls, there are some immediate use cases where you can leverage &quot;unstructured&quot; data to increase the number and/or the quality of the signals (features) that you are feeding into your model: 1) process audio data from contact centre interactions to determine the overall sentiment of a customer (i.e. &quot;is he up for a sales call?&quot;); 2) process text from customer reviews and use sentiment/keywords as a preditor for a churn model; etc!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;it seems that they always end up reducing the unstructured data to&#xA;  some structured format first, then feed it to whatever forecasting&#xA;  method they are using.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Probably because their forecasting algorithm or method requires that data is inputted in a specific format, like most forecasting or classification algorithms do; different algorithms will have different tolerances to data issues like missing values or outliers. So you process a piece of data that is unstructured with the goal of using it, usually by adding it to a data model or inputting it as data for a report or as a feature for a predictive model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I'm curious if anyone knows a case where you just throw unstructured data at a predictive algorithm (e.g. a neural network) and get something meaningful or useful out of it.&lt;/p&gt;&#xA;" OwnerUserId="43615" LastEditorUserId="43615" LastEditDate="2017-12-28T23:20:20.890" LastActivityDate="2017-12-28T23:20:20.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26087" PostTypeId="1" AcceptedAnswerId="29050" CreationDate="2017-12-28T16:14:43.173" Score="3" ViewCount="7936" Body="&lt;p&gt;Why pruning is not currently supported in scikit-learn? How can we tune the decision trees to make a workaround?&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2019-09-01T13:04:40.013" Title="sklearn : missing pruning for decision trees" Tags="|python|scikit-learn|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26089" PostTypeId="5" CreationDate="2017-12-28T16:42:09.950" Score="0" Body="&lt;p&gt;Transfer learning is the process of learning a set of characteristics from one data and applying this &amp;quot;knowledge&amp;quot; to another similar dataset (i.e. using the same model across datasets).&lt;/p&gt;&#xA;" OwnerUserId="41637" LastEditorUserId="41637" LastEditDate="2021-03-16T23:23:40.183" LastActivityDate="2021-03-16T23:23:40.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26090" PostTypeId="4" CreationDate="2017-12-28T16:42:09.950" Score="0" Body="Transfer learning is the process of learning a set of characteristics from one data and applying this &quot;knowledge&quot; to another similar dataset (i.e. using the same model across datasets). " OwnerUserId="41637" LastEditorUserId="41637" LastEditDate="2021-03-16T23:26:34.837" LastActivityDate="2021-03-16T23:26:34.837" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26092" PostTypeId="2" ParentId="26076" CreationDate="2017-12-28T19:11:48.610" Score="1" Body="&lt;p&gt;The &lt;em&gt;TD Target&lt;/em&gt; (for learning update) for using $\hat{q}(s,a)$ neural network in Q-learning is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$r + \text{max}_{a'} \hat{q}(s',a')$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to calculate this, you need a starting state $s$, the action taken form that state $a$, and the resulting reward $r$ and state $s'$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need the $s, a$ to generate the &lt;em&gt;input&lt;/em&gt; to the neural network (what you might call &lt;code&gt;train_X&lt;/code&gt; for supervised learning). You need $r, s'$ to generate the TD Target shown above as an &lt;em&gt;output&lt;/em&gt; to learn for regression (in supervised learning, that would be &lt;code&gt;train_y&lt;/code&gt;). And you need to work through all possible $a'$ based on $s'$ in order to find the maximum value for the TD Target equation used in Q learning - other RL algorithms may use variations of this for calculating the TD Target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means your suggestions are all &lt;em&gt;close&lt;/em&gt; but not quite right.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sort of. The B2 state is $s'$ from the equation, so is responsible for calculating the TD target for state-action value $q(s,a)$.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You still need to know $s$ is A1, because the representation of A1 (and whichever action is taken) will be the &lt;em&gt;input&lt;/em&gt; to your network.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Iterate over actions from B2 using your neural network to pick the highest estimate.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm not sure I fully understand this, but it does not seem quite right. One thing that might be confusing you is having your actions as &quot;move to state&quot;. Whilst this is quite normal in many deterministic environments, especially board games, most RL formula and tutorials are written using separate state/action pairs, so whilst getting this straight in your mind, best to find another way to represent actions - e.g. move piece from X to Y, or place new piece at (I,J) . . . you can return to &quot;change state from A1 to B2&quot; as the representation later. This is actually more efficient and is called the &lt;em&gt;after-state&lt;/em&gt; representation, but most of the literature will show &lt;em&gt;state-action&lt;/em&gt; value functions.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2017-12-28T20:40:59.250" LastActivityDate="2017-12-28T20:40:59.250" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26093" PostTypeId="1" AcceptedAnswerId="26297" CreationDate="2017-12-28T20:34:47.873" Score="0" ViewCount="1618" Body="&lt;p&gt;I have several nominal variables which I've encoded using the &lt;code&gt;LabelEncoder()&lt;/code&gt; function. Now I want to replace the encoded values in the place of the raw datas of the features in the dataframe. I tried &lt;code&gt;df.replace()&lt;/code&gt; but no luck. Below is the code snippet.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nominal_values = [&#xA;    'HouseStyle','ExterQual','BsmtQual','BsmtExposure','BsmtFinType1',&#xA;    'KitchenQual','GarageType','GarageFinish','GarageQual','GarageCond']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One of the column &quot;KitchenQual&quot; has values as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{kitchenqual :['Ex','Gd','TA','Fa','Po']} representing &quot;Excellent&quot;,&quot;Good&quot;,&quot;Typical/Average&quot;,&quot;Fair&quot;,&quot;Poor&quot; respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another Feature &quot;BsmtFinType1&quot; has values as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;{BsmtFinType1: ['GLQ','ALQ','BLQ','Rec','LwQ','Unf','NA']} represeting&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Good Living Quarters&quot;,&quot;Average Liv. Quat.&quot;,&quot;Below Avg.Liv.Quat,&quot;,&quot;Avg. rec room&quot;, &quot;Low Quality&quot;, &quot;Unfinished&quot;,&quot;No Basement&quot; respectively.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import LabelEncoder&#xA;&#xA;lbe = LabelEncoder()&#xA;for noms in nominal_values:&#xA;    encode = lbe.fit_transform(cat_var[noms])&#xA;    cat_var.replace(to_replace=cat_var[noms],&#xA;                    value=pd.Series(encode),inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used &lt;code&gt;pd.Series(encode)&lt;/code&gt; since the replace function will support only one of the following data structures&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scalar&lt;/li&gt;&#xA;&lt;li&gt;dict&lt;/li&gt;&#xA;&lt;li&gt;Series&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="43890" LastEditorUserId="43890" LastEditDate="2017-12-31T11:33:25.417" LastActivityDate="2018-01-04T18:23:10.637" Title="Replacing the feature variables with encoded variables" Tags="|machine-learning|python|scikit-learn|data-cleaning|pandas|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26094" PostTypeId="2" ParentId="492" CreationDate="2017-12-28T21:29:23.787" Score="2" Body="&lt;p&gt;Here are some ideas about how to use word vectors for NER, which adopts a mostly unsupervised word2vec-centric approach.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Given a set of word vectors (either something you trained, or something off-the-shelf like GoogleNews-vectors-negative300.bin), discover clusters in the vector space. These clusters are basically your definitions for various unnamed concepts.&lt;/li&gt;&#xA;&lt;li&gt;With minimal supervision, you can map/convert the unnamed clusters to match human knowledge, thus creating named concepts grounded on top of the known word vectors and unnamed concepts. For example, a method &lt;code&gt;findCluster(['joy', 'surprise', 'disgust', 'trust', 'fear', 'sadness', 'anger', 'anticipation'])&lt;/code&gt; might return a list containing hundreds of words mostly related to emotion. If you name this list 'emotion', then there you have a named concept 'emotion' defined based on the vector space.&lt;/li&gt;&#xA;&lt;li&gt;You can also do vector math to find the intermediate concept between two given ones. For example, the vector math might tell you that when given two words 'surprise' and 'disgust', the following are found between the two: dismay, astonishment, disappointment, amazement, bewilderment, exasperation, disbelief, shock, etc. This allows you to build up relationship between concepts.&lt;/li&gt;&#xA;&lt;li&gt;You can repeat the above to build up various types of named concpts, like: weekdays, all emotions, happy emotions, vehicles, etc.&lt;/li&gt;&#xA;&lt;li&gt;Once you have built up layers of named concepts, you can then train a RNN on a text corpus that has been augmented with the named concepts, so 'brown fox jumps' is also '{color} {animal} {action}' etc. This way the RNN should be able to learn some rudimentary grammar in an unsupervised manner.&lt;/li&gt;&#xA;&lt;li&gt;If you have built up a sufficiently powerful grammar from the above, then you should be able to apply it toward some of your NER tasks.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43896" LastEditorUserId="29575" LastEditDate="2017-12-28T22:43:41.953" LastActivityDate="2017-12-28T22:43:41.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26095" PostTypeId="1" CreationDate="2017-12-28T22:45:40.603" Score="1" ViewCount="67" Body="&lt;p&gt;As a beginner in ML field, should I in beginning implement various algorithms from scratch (using only NumPy for example) and then start using ML libraries like TensorFlow, Keras or vice-versa? For example, I implemented multilayer feed-forward neural network in Python using only NumPy but it took me few weeks to understand NNs at the level that I can implement them from scratch, would it be better that I spent that time to cover theory less thoroughly and use libraries to make more advanced models easily?&lt;/p&gt;&#xA;" OwnerUserId="43720" LastActivityDate="2017-12-29T05:55:38.487" Title="Should a beginner in ML use libraries like TF and Keras?" Tags="|machine-learning|neural-network|tensorflow|keras|" AnswerCount="2" CommentCount="2" ClosedDate="2018-01-01T18:57:52.650" ContentLicense="CC BY-SA 3.0" />
  <row Id="26096" PostTypeId="2" ParentId="13619" CreationDate="2017-12-28T23:34:34.403" Score="0" Body="&lt;p&gt;Algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the association rules between items across all baskets. Each of those will require that data is input in a specific format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for FPGrowth in Spark it's an RDD of [Array[String]], where each Array represents a market basket (a transaction/purchase), Strings being the transaction's items' names/identifiers. So, you have to transform the data into that format to input it to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's a good example of this at Spark's &lt;a href=&quot;https://spark.apache.org/docs/2.1.2/mllib-frequent-pattern-mining.html#fp-growth&quot; rel=&quot;nofollow noreferrer&quot;&gt;website&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import org.apache.spark.mllib.fpm.FPGrowth&#xA;import org.apache.spark.rdd.RDD&#xA;&#xA;val data = sc.textFile(&quot;data/mllib/sample_fpgrowth.txt&quot;)&#xA;&#xA;//prepare the data to use with FPGrowth&#xA;val transactions: RDD[Array[String]] = data.map(s =&amp;gt; s.trim.split(' '))&#xA;&#xA;//create and run the model&#xA;val fpg = new FPGrowth()&#xA;  .setMinSupport(0.2)&#xA;  .setNumPartitions(10)&#xA;val model = fpg.run(transactions)&#xA;&#xA;//output the frequent itemsets (items frequently bought together)&#xA;model.freqItemsets.collect().foreach { itemset =&amp;gt;&#xA;  println(itemset.items.mkString(&quot;[&quot;, &quot;,&quot;, &quot;]&quot;) + &quot;, &quot; + itemset.freq)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43615" LastEditorUserId="43615" LastEditDate="2017-12-28T23:58:12.850" LastActivityDate="2017-12-28T23:58:12.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26097" PostTypeId="2" ParentId="26095" CreationDate="2017-12-29T02:56:35.987" Score="1" Body="&lt;p&gt;Start of by reading up on math/theory. For neural networks I can really recommend Michael Jensens free online book. All along he gives good python examples using only numpy. &#xA;For DNNs, TF and Keras are IMO the easiest approach. For linear models, scikit learn would probably cover most. &lt;/p&gt;&#xA;" OwnerUserId="41700" LastActivityDate="2017-12-29T02:56:35.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26098" PostTypeId="1" CreationDate="2017-12-29T04:07:31.747" Score="0" ViewCount="43" Body="&lt;p&gt;I went through basic concepts of ML algorithms but still had few queries on their advantages over each other. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;When should we use linear regression vs Neural Networks for regression?&lt;/li&gt;&#xA;&lt;li&gt;When should be use Logistic Regression vs SVM vs Neural Networks classification vs Decision Trees?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="39331" LastEditorUserId="29575" LastEditDate="2017-12-29T15:41:39.270" LastActivityDate="2017-12-29T15:41:39.270" Title="Using appropriate algorithm" Tags="|machine-learning|neural-network|svm|logistic-regression|linear-regression|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-05T05:44:02.207" ContentLicense="CC BY-SA 3.0" />
  <row Id="26099" PostTypeId="1" CreationDate="2017-12-29T05:55:08.317" Score="0" ViewCount="213" Body="&lt;p&gt;I am reading this &lt;a href=&quot;http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;amp;arnumber=990477&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;, but don't really understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do the words &quot;part-based&quot; or &quot;local&quot; for non-negative matrix factorization (NMF) mean that the algorithm aims to factorize some specific parts more than other unimportant parts? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like the algorithm tries to seek some max signals in matrix and focus on that part? Is there a simple explanation about local-NMF?&lt;/p&gt;&#xA;" OwnerUserId="37718" LastEditorUserId="29575" LastEditDate="2017-12-29T15:52:33.680" LastActivityDate="2018-05-11T16:39:55.023" Title="What is local-NMF? How is it better than original NMF?" Tags="|algorithms|recommender-system|image-classification|dimensionality-reduction|matrix-factorisation|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26100" PostTypeId="2" ParentId="26095" CreationDate="2017-12-29T05:55:38.487" Score="1" Body="&lt;p&gt;I would recommend using the ML libraries because you will get stuff done. Ultimately something will go wrong and you will need to understand how the algorithms actually work. I highly recommend Andrew Ng's Machine Learning class on Coursera to help you fix any issues and obtain a deeper understanding of the algorithm. The class is worth the fee.&lt;/p&gt;&#xA;" OwnerUserId="43914" LastActivityDate="2017-12-29T05:55:38.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26101" PostTypeId="2" ParentId="26098" CreationDate="2017-12-29T05:59:23.320" Score="2" Body="&lt;p&gt;If you need something interpretable but accuracy is less important, use the methods on the left side of the question (linear regression, logistic regression). If accuracy is more important, use the methods on the right side of the question (neural nets). SVM is a somewhere in the middle.&lt;/p&gt;&#xA;" OwnerUserId="43914" LastActivityDate="2017-12-29T05:59:23.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26102" PostTypeId="1" CreationDate="2017-12-29T06:11:43.827" Score="1" ViewCount="66" Body="&lt;p&gt;I am trying to classify customer support sessions using supervised machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each customer support session I have 3 bags of information.&#xA;1. The title of the customer's complaint &#xA;2. Information about the device the customer was using&#xA;3. Text of the chat session with the customer support agent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each customer support session, there are 6 different classes. Is it better to:&#xA;1. Train a classifier on each bag of information and have them vote on which class the session belongs to?&#xA;2. Put all of the information into a single set of features, and train a single classifier to determine which class the session belongs to?&#xA;3. Other?&lt;/p&gt;&#xA;" OwnerUserId="43914" LastActivityDate="2022-09-08T10:02:55.327" Title="Supervised learning on sources of information with different importance" Tags="|machine-learning|nlp|feature-selection|aggregation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26103" PostTypeId="1" AcceptedAnswerId="26119" CreationDate="2017-12-29T08:12:48.523" Score="46" ViewCount="74427" Body="&lt;p&gt;I am trying to merge two Keras models into a single model and I am unable to accomplish this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use this as input to the layer $B1$ (of dimension 8 again) in Model $B$ and then combine both Model $A$ and Model $B$ as a single model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the functional module to create Model $A$ and Model $B$ independently.  How can I accomplish this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: $A1$ is the input layer to model $A$ and $B1$ is the input layer to model $B$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Chgpo.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Chgpo.jpg&quot; alt=&quot;See Picture&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43921" LastEditorUserId="61023" LastEditDate="2019-01-29T20:45:58.467" LastActivityDate="2022-06-12T12:26:59.970" Title="Merging two different models in Keras" Tags="|machine-learning|python|deep-learning|keras|tensorflow|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26105" PostTypeId="1" CreationDate="2017-12-29T09:49:37.147" Score="1" ViewCount="100" Body="&lt;p&gt;i have given a submission dataSet with fewer rows how to extrapolate it so that model could perform better&lt;/p&gt;&#xA;" OwnerUserId="43356" LastActivityDate="2017-12-29T09:49:37.147" Title="How to extrapolate given dataset in machine learning" Tags="|machine-learning|" AnswerCount="0" CommentCount="0" ClosedDate="2018-01-03T14:32:25.350" ContentLicense="CC BY-SA 3.0" />
  <row Id="26106" PostTypeId="2" ParentId="26016" CreationDate="2017-12-29T10:04:09.597" Score="0" Body="&lt;p&gt;One way to move forward could be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Create a feedforward neural network that, given Y (probably you want to normalise it) predicts the X. So the output of the model (the last layer) would be a set of softmax neurons for each feature. So if the feature 1 (e.g. colour) has 4 options, you will apply the softmax over four neurons, and do the same over each feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then your loss function could be the sum (or a linear combination if you prefer) of the cross entropy for each feature.&lt;/p&gt;&#xA;" OwnerUserId="23492" LastActivityDate="2017-12-29T10:04:09.597" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26107" PostTypeId="2" ParentId="26102" CreationDate="2017-12-29T10:33:48.200" Score="-1" Body="&lt;p&gt;Its better to do supervised classification then unsupervised clustering since you have the right labels for the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in supervised classification approach 1 is better than approach 2 because in approach 2 you will the reducing the already available information by combining features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Approach 3 is advised where you will create more features from the given 3 features by use of NLP processing, sentiments, tags, chunking, time taken, text length etc and then selecting the features those provide more information gain using feature selection approaches.&lt;/p&gt;&#xA;" OwnerUserId="43369" LastActivityDate="2017-12-29T10:33:48.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26108" PostTypeId="2" ParentId="21849" CreationDate="2017-12-29T11:25:14.550" Score="3" Body="&lt;p&gt;I've done this (using anaconda) with the following libraries. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from mlxtend.frequent_patterns import apriori&#xA;from mlxtend.frequent_patterns import association_rules&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Have a look &lt;a href=&quot;https://rasbt.github.io/mlxtend/user_guide/frequent_patterns/association_rules/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="31457" LastEditorUserId="29575" LastEditDate="2017-12-29T15:38:36.423" LastActivityDate="2017-12-29T15:38:36.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26111" PostTypeId="1" CreationDate="2017-12-29T14:04:30.003" Score="1" ViewCount="108" Body="&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In a Feed Forward network, assume we have a Mini-batch of 64 examples.&lt;/li&gt;&#xA;&lt;li&gt;Our layer $l$ contains 20 neurons&lt;/li&gt;&#xA;&lt;li&gt;Because every neuron in layer will require Mean and Variance from the whole minibatch, we must  to run all 64 'examples' in parallel (is this true?). We must wait until each such 'example' will have the appropriate sum (for that particular neuron!) available across all these minibatches. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Wouldn't this break parallelism? Previously, each of our example-threads could have computed everything, up to the activation values for the entire layer $l$. However, now we have to: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Wait for all example-threads to finish computing the sums for our layer $l$ (hence, our speed slows down to match the slowest example-thread)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pause multithreading (!) and compute mean &amp;amp; variance for the layer $l$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply it to the sums, resume example-threads, so each thread completes the job, by activating its corresponding adjusted sums&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do I miss a part of the puzzle, or it really has to be done like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$$$&#xA;&lt;strong&gt;my understanding of Batch Norm:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Batch norm &lt;a href=&quot;http://proceedings.mlr.press/v37/ioffe15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(link to original paper)&lt;/a&gt; makes the search space more even &amp;amp; simpler to navigate during forward &amp;amp; backward passes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is usually applied to the sum $Z$ going into activation function (although there is a lot of debate &lt;a href=&quot;https://github.com/cvjena/cnn-models/issues/3&quot; rel=&quot;nofollow noreferrer&quot;&gt;(1)&lt;/a&gt; &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/67gonq/d_batch_normalization_before_or_after_relu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2)&lt;/a&gt;). This is done for every neuron in a layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in usual settings, the neuron's output on layer $l$ is $$H_{i} = \phi(W\cdot X +b)$$  where W is a vector of weights leading into our $i$'th unit, $X$ is a vector of outputs from previous layer $l_{-1}$ and $b$ is a bias&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With Batch norm, we still have our usual non-activated sum for neuron $i$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z_i = W\cdot X$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but also have our mean for that neuron across all minibatches (Just for that neuron!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\mu_i = \frac{1}{m} \sum_{p}^m Z_{ip} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar, for our Variance as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sigma^2_i = \frac{1}{m}\sum_{p}^m (Z_{ip} - \mu_i)^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our non-activated normalized sum for neuron $i$ (here, $\epsilon$ is a tiny number, in case if variance is too small, to avoid division by zero):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z_{norm} = \frac{Z_i - \mu_i}{\sqrt{\sigma^2_i + \epsilon}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We give a network a chance to learn the actual distribution it needs, so it's different from default distrib. We introduce 2 learnable parameters, $\gamma$ and $\beta$, and network can use it to 'squash and shift' the distributions as it needs (even canceling-out the batch-norm if required).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Z_{adjusted} = {\gamma}Z_{norm} + \beta $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally neuron's output is computed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_i =  \phi (Z_{adjusted}) $$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bias isn't included, because we already have $\beta$ for the whole layer $l$.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-01-13T21:03:44.440" LastActivityDate="2018-01-13T21:03:44.440" Title="Batch Normalization will disrupt multi-threading?" Tags="|lstm|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26112" PostTypeId="1" CreationDate="2017-12-29T14:32:33.657" Score="7" ViewCount="12463" Body="&lt;p&gt;I'm currently training a CNN with &lt;code&gt;Keras&lt;/code&gt; and I'm using the Adam optimizer. My plan is to gradually reduce the learning rate after each epoch. That's what I thought the decay parameter was for.  For me, the documentation does not clearly explain how it works:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;decay: float &gt;= 0. Learning rate decay over each update.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, when looking at the used learning rate in tensorboard, it stays the same as the initial learning rate. So, how does this decay parameter actually work?&lt;/p&gt;&#xA;" OwnerUserId="27237" LastEditorUserId="28175" LastEditDate="2018-02-13T15:57:10.450" LastActivityDate="2018-06-13T17:41:03.283" Title="Decay Parameter in Keras Optimizers" Tags="|machine-learning|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26114" PostTypeId="2" ParentId="26112" CreationDate="2017-12-29T15:00:08.563" Score="3" Body="&lt;p&gt;Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each epoch, &lt;code&gt;TensorFlow&lt;/code&gt; uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient &lt;em&gt;descent&lt;/em&gt; which implies moving toward low-level places. &#xA;&lt;hr /&gt;&#xA;Recently I was looking the code of optimizers in &lt;code&gt;Keras&lt;/code&gt; and I found that as the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if self.initial_decay &amp;gt; 0:&#xA;       lr *= (1. / (1. + self.decay * K.cast(self.iterations,&#xA;                                             K.dtype(self.decay))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;learning changes each iteration and not each epoch.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-13T15:56:30.993" LastActivityDate="2018-02-13T15:56:30.993" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26115" PostTypeId="2" ParentId="21788" CreationDate="2017-12-29T15:29:19.523" Score="0" Body="&lt;p&gt;I suggest you using the pre-trained model and freezing all the convolution layers. You should just train the weights of your dense layers in this situation. So use &lt;code&gt;transfer learning&lt;/code&gt; and freeze the convolution layers and replace the dense layer with your desired one and also the last layer which is &lt;code&gt;softmax&lt;/code&gt;. The reason is that the image-net already can find the features of a basketball play very well. You just need to classify each game. If you want to see an explanation, read &lt;a href=&quot;https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2812140&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-29T15:38:58.963" LastActivityDate="2017-12-29T15:38:58.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26116" PostTypeId="1" AcceptedAnswerId="26300" CreationDate="2017-12-29T15:42:16.210" Score="1" ViewCount="52" Body="&lt;p&gt;I've been statistically validating the performance of different Deep Learning models in classifying parasitized and normal cells. In the process, I could find that there is no statistically significant difference in performance between the pre-trained CNN models. In the absence of such significance, which model am I supposed to choose?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="26562" LastEditDate="2018-01-04T22:31:22.933" LastActivityDate="2018-01-04T22:31:22.933" Title="How to select a model based on statistical analyses?" Tags="|deep-learning|statistics|performance|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26118" PostTypeId="2" ParentId="26103" CreationDate="2017-12-29T17:31:28.763" Score="14" Body="&lt;p&gt;In Keras there is a helpful way to define a model: using the &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;noreferrer&quot;&gt;functional API&lt;/a&gt;.  With functional API you can define a directed acyclic graphs of layers, which lets you build completely arbitrary architectures. Considering your example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#A_data = np.zeros((1,30))&#xA;#A_labels = np.zeros((1,30))&#xA;#B_labels =np.zeros((1,30))&#xA;&#xA;A1 = layers.Input(shape=(30,), name='A_input')&#xA;A2 = layers.Dense(8, activation='???')(A1)&#xA;A3 = layers.Dense(30, activation='???', name='A_output')(A2)&#xA;&#xA;&#xA;B2 = layers.Dense(40, activation='???')(A2)&#xA;B3 = layers.Dense(30, activation='???', name='B_output')(B2)&#xA;&#xA;## define A&#xA;A = models.Model(inputs=A1, outputs=A3)&#xA;&#xA;## define B&#xA;B = models.Model(inputs=A1, outputs=B3) &#xA;&#xA;B.compile(optimizer='??',&#xA;          loss={'B_output': '??'}&#xA;          )&#xA;&#xA;B.fit({'A_input': A_data},&#xA;  {'B_output': B_labels},&#xA;  epochs=??, batch_size=??)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, that's it! You can see the result by: &lt;code&gt;B.summary()&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Layer (type)                 Output Shape              Param    &#xA;A_input (InputLayer)         (None, 30)                0         &#xA;_________________________________________________________________&#xA;dense_8 (Dense)              (None, 8)                 248     &#xA;______________________________________________________________&#xA;dense_9 (Dense)              (None, 40)                360       &#xA;_________________________________________________________________&#xA;B_output (Dense)             (None, 30)                1230      &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38657" LastEditorUserId="38657" LastEditDate="2018-01-04T13:48:54.837" LastActivityDate="2018-01-04T13:48:54.837" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26119" PostTypeId="2" ParentId="26103" CreationDate="2017-12-29T19:11:32.073" Score="42" Body="&lt;p&gt;I figured out the answer to my question and here is the code that builds on the above answer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Input, Dense&#xA;from keras.models import Model&#xA;from keras.utils import plot_model&#xA;&#xA;A1 = Input(shape=(30,),name='A1')&#xA;A2 = Dense(8, activation='relu',name='A2')(A1)&#xA;A3 = Dense(30, activation='relu',name='A3')(A2)&#xA;&#xA;B2 = Dense(40, activation='relu',name='B2')(A2)&#xA;B3 = Dense(30, activation='relu',name='B3')(B2)&#xA;&#xA;merged = Model(inputs=[A1],outputs=[A3,B3])&#xA;plot_model(merged,to_file='demo.png',show_shapes=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and here is the output structure that I wanted:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2xIdb.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2xIdb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43921" LastActivityDate="2017-12-29T19:11:32.073" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26120" PostTypeId="2" ParentId="26041" CreationDate="2017-12-29T19:37:39.627" Score="1" Body="&lt;p&gt;For AUC, you can not. Leave-one-out uses only one data instance in the training set. Computation of AUC requires more data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For general cross-validation (say, 10-fold cross-validation), there are several ways to examine what happened in each fold. Test &amp;amp; Score widget that performs cross-validation outputs a data table with predictions for each of the data instances in the test set. Together with predicted probabilities, predicted class, original class, the data also include information about the fold. You can then select the results by the fold and analyze it further, say, in a box plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nrfjz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nrfjz.png&quot; alt=&quot;test and score with fold selection&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To compute evaluation statistics for each of the folds, use Data Sampler, chose &quot;Cross Validation&quot; for sampling, select the desired fold, and wire the connection between Data Sampler and Test &amp;amp; Score widget as shown below. Notice that in this setting the Data Sampler outputs the fold data as a sample (to be used as test data), and out-of-fold data as remaining data (to be used as training data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YsjX3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YsjX3.png&quot; alt=&quot;cross-valiadation-based data sampling&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39624" LastActivityDate="2017-12-29T19:37:39.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26124" PostTypeId="1" AcceptedAnswerId="26144" CreationDate="2017-12-30T03:16:54.410" Score="0" ViewCount="473" Body="&lt;p&gt;I have a &lt;strong&gt;SampleData&lt;/strong&gt; (2653 observation, 11 features) of bank transaction done in the 1-month timeframe. &lt;a href=&quot;https://drive.google.com/uc?export=download&amp;amp;id=10nxd8MqjykF9ZkwCZLr8kYuhUrq4GU98&quot; rel=&quot;nofollow noreferrer&quot;&gt;Download Dataset&lt;/a&gt; &lt;em&gt;Size=250KB&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MvxPk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MvxPk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to come up with algorithms (Single or Combine) that can segment users into different categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UX7A6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UX7A6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since there are mostly categorical features involved except &lt;code&gt;tx_amount&lt;/code&gt;, Which algorithm is best suited here, or how should I approach this problem to create user segments?&lt;/p&gt;&#xA;" OwnerUserId="30686" LastEditorUserId="30686" LastEditDate="2017-12-31T02:57:30.377" LastActivityDate="2017-12-31T02:57:30.377" Title="Best algorithm to create segment in case of categorical variables" Tags="|machine-learning|r|clustering|categorical-data|" AnswerCount="2" CommentCount="0" ClosedDate="2018-01-04T23:07:49.547" ContentLicense="CC BY-SA 3.0" />
  <row Id="26126" PostTypeId="2" ParentId="10103" CreationDate="2017-12-30T05:16:52.593" Score="8" Body="&lt;p&gt;Check out FFT-accelerated Interpolation-based t-SNE (&lt;a href=&quot;https://arxiv.org/abs/1712.09005&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;, &lt;a href=&quot;https://github.com/KlugerLab/FIt-SNE&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt;, and &lt;a href=&quot;https://pypi.python.org/pypi/fitsne/0.1.10&quot; rel=&quot;noreferrer&quot;&gt;Python package&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the abstract: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We present Fast&#xA;  Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE),&#xA;  which dramatically accelerates the computation of t-SNE. The most&#xA;  time-consuming step of t-SNE is a convolution that we accelerate by&#xA;  interpolating onto an equispaced grid and subsequently using the fast&#xA;  Fourier transform to perform the convolution. We also optimize the&#xA;  computation of input similarities in high dimensions using&#xA;  multi-threaded approximate nearest neighbors. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The paper also includes an example of a dataset with a million points and 100 dimensions (similar to OP's setting), and it seems to take ~1 hour. &lt;/p&gt;&#xA;" OwnerUserId="43889" LastEditorUserId="43889" LastEditDate="2018-01-10T17:13:52.200" LastActivityDate="2018-01-10T17:13:52.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26127" PostTypeId="1" AcceptedAnswerId="88094" CreationDate="2017-12-30T07:18:25.337" Score="1" ViewCount="121" Body="&lt;p&gt;I need to compare the price of the local drug-list of my country with prices of drug-lists of other countries (CSV format), I have a manually-matched table whic is my target and I use python via Jupyter on windows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;In the following, there are two records of two different drug-lists&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;one drug in list 'A'&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name:   40MG ABSEAMED*&#xA;&#xA;Form:   injectable solution&#xA;&#xA;Dosage: 0.9%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;matched drug in list 'B'&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name:   ABSEAMED SOLUTION FOR INJECTION 40MG/0.4ML&#xA;&#xA;Form:   SOLUTION FOR INJECTION&#xA;&#xA;Dosage: 25mg/ml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My simple process is to clean data from undesired symbols and spaces, then to count occurances of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problems:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using simple distribution function &lt;code&gt;(intersected words in both) / (words of list 'A')&lt;/code&gt; the name of drug (ABSEAMED) is considered as priority as other features ('FOR', 'SOLUTION', ...)&lt;/li&gt;&#xA;&lt;li&gt;By removing the &lt;code&gt;Form&lt;/code&gt; words from &lt;code&gt;Name&lt;/code&gt; words, some names becomes null&lt;/li&gt;&#xA;&lt;li&gt;after cleaning data, I get noisy features (25mg/ml ===&gt; 25 mg ml) and this dropped the accuracy of matching from &lt;code&gt;56%&lt;/code&gt; to &lt;code&gt;24%&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are the basic recommended techniques/tools (Automates, POS_tagger, ...) of Natural Language Processing that may give a high-accuracy results ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I thought about creating my own POS tag to match tags before contents&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(e.x. the drug name '0.225% W/V SODIUM CHLORIDE (1/4 NORMAL SALINE)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can be tagged as &lt;code&gt;\dosage \symbol \name \quote&lt;/code&gt;*&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but I'm not sure enough that it can improve my result&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Any help is appreciated.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="28495" LastActivityDate="2021-01-17T17:32:49.227" Title="(Pattern-Matching) what is the best measurement can be used to match drug names from different countries" Tags="|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26128" PostTypeId="2" ParentId="26124" CreationDate="2017-12-30T08:13:40.723" Score="1" Body="&lt;p&gt;Do not approach this by using whatever algorithm you can get to run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no use in just doing something that you don't understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, &lt;em&gt;you&lt;/em&gt; will first need to identify what is a good way to group data that &lt;em&gt;solves your problem&lt;/em&gt;. Then you can begin looking for an algorithm that helps solving &lt;em&gt;your&lt;/em&gt; problem (and not some useless other problem).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2017-12-30T08:13:40.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26129" PostTypeId="1" CreationDate="2017-12-30T08:36:54.653" Score="1" ViewCount="1899" Body="&lt;p&gt;I am referring &lt;a href=&quot;http://www.nltk.org/book/ch06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nltk.org/book/ch06.html&lt;/a&gt; for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set.&#xA;I am trying to build a classifier that considers only the verbs and assesses if the movie review is positive or negative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;Please explain if this approach is better, if yes how can it be improved else what other parts of speech tags need to be included to improve the feature set. &#xA;&lt;br&gt;Please refer to the following code&#xA;&lt;br&gt;1)Import nltk and corpus&#xA;&lt;br&gt;2)Understand the categories&#xA;&lt;br&gt;3)Create a list of &quot;documents&quot; where each document consists of words that are not in stopwords.words() and filtered from list of &quot;punctuations&quot; as well.&#xA;&lt;br&gt;4)Shuffle the documents and generate a list called &quot;all_words&quot; that contains all the words appearing in movie_reviews minus stopwords and punctuations.&#xA;&lt;br&gt;5)Create a Freq Dist of &quot;all_words&quot; &#xA;&lt;br&gt;6)Create a list of &quot;verbs&quot; by looking at the pos_tag of each word appearing in &quot;all_words&quot;&#xA;&lt;br&gt;7)Create a feature dictionary for each document in the &quot;documents&quot; list created in step 3, such that the dictionary will contain keys corresponding to words in &quot;verb&quot; list and a boolean to show if that verb appeared in the document under consideration. This is handled by documentFeature().&#xA;&lt;br&gt;8)Create a naive bayes classifier instance and train and compute test accuracy.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.corpus import movie_reviews&#xA;from nltk.corpus import stopwords&#xA;from nltk.tokenize import RegexpTokenizer&#xA;import nltk&#xA;movie_reviews.categories()&#xA;#['pos','neg']&#xA;#the regextokenier is used to tokenize the words    &#xA;tokenizer=RegexpTokenizer(r'\w+')&#xA;#creating documents based on filtration of stopwords for each review and running a tokenizer on each document&#xA;documents=[(tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words(fileid))-set(stopwords.words()))),category) &#xA;           for category in movie_reviews.categories()&#xA;           for fileid in movie_reviews.fileids() &#xA;          ]&#xA;import random&#xA;random.shuffle(documents)&#xA;#each document contains words that are not in stopwords and punctuations.&#xA;for i in documents[:5]:&#xA;    temp=nltk.FreqDist([j.lower() for j in i[0]])&#xA;    print(temp.most_common(5),i[1]) &#xA;#output of 5 documents&#xA;#[('vampires', 1), ('clever', 1), ('interesting', 1), ('sunlight', 1), ('partners', 1)] neg&#xA;#[('family', 1), ('nino', 1), ('friends', 1), ('acting', 1), ('higher', 1)] pos&#xA;#[('inconsistent', 1), ('eye', 1), ('yes', 1), ('interesting', 1), ('praise', 1)] neg&#xA;#[('acting', 1), ('science', 1), ('bucks', 1), ('huge', 1), ('terrific', 1)] pos&#xA;#[('acting', 1), ('shielded', 1), ('somewhere', 1), ('think', 1), ('touched', 1)] neg&#xA;&#xA;#generate a list called 'all_words' that contains all the set of words that have appeared so far&#xA;all_words=tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words())-set(stopwords.words())))&#xA;freqdist=nltk.FreqDist(all_words)&#xA;&#xA;#create a list of all verbs for each word appearing in 'all_words'&#xA;verb=[]&#xA;pos_=nltk.pos_tag(all_words)&#xA;#print([i[1] for i in pos_])&#xA;for i in pos_:&#xA;    if i[1] in ['VB','VBG','VBN','VBZ','VBD','VBP']:&#xA;        verb.append(i[0])&#xA;&#xA;#document - feature set, build a dictionary of verbs for each document&#xA;def documentFeature(document):&#xA;    feature={}&#xA;    for i in verb:&#xA;        feature['contains({0})'.format(i)]=(i in document)&#xA;    return feature    &#xA;#build a naive bayes classifier&#xA;featureSet=[(documentFeature(d),c) for d,c in documents]&#xA;trainSet,testSet=featureSet[100:], featureSet[:100]&#xA;classifier=nltk.NaiveBayesClassifier.train(trainSet)&#xA;&#xA;print(nltk.classify.accuracy(classifier, testSet))&#xA;#0.03 a very poor accuracy on the testset&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Currently I am getting 0.03 accuracy, please help me with improving the accuracy.&lt;/p&gt;&#xA;" OwnerUserId="40801" LastEditorUserId="40801" LastEditDate="2017-12-30T08:42:12.023" LastActivityDate="2020-08-26T13:04:40.483" Title="Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews" Tags="|python|nlp|feature-engineering|nltk|feature-construction|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26130" PostTypeId="2" ParentId="26111" CreationDate="2017-12-30T09:57:47.190" Score="1" Body="&lt;p&gt;My experience with batch and layer normalisation is that it is around 3 to 5 times slower per step than without it. Although the model may converge in less number of steps (which is what the authors report in the article) it takes much longer to calculate each step, and hence the whole training takes longer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has been reported in a few places like: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/4ufmxy/layer_normalization_implemented_in_tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.reddit.com/r/MachineLearning/comments/4ufmxy/layer_normalization_implemented_in_tensorflow/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/cooijmanstim/recurrent-batch-normalization/issues/2&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/cooijmanstim/recurrent-batch-normalization/issues/2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/keras-team/keras/issues/1309&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/issues/1309&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="23492" LastActivityDate="2017-12-30T09:57:47.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26131" PostTypeId="1" CreationDate="2017-12-30T11:22:01.817" Score="5" ViewCount="1291" Body="&lt;p&gt;The &lt;a href=&quot;http://www.nltk.org/book&quot; rel=&quot;noreferrer&quot;&gt;Natural Language Processing with Python&lt;/a&gt; book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and provides a method to perform sentence segmentation on unseen corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NLTK provides &lt;a href=&quot;http://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;word_tokenize&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;http://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;sent_tokenize&lt;/code&gt;&lt;/a&gt;. Creating our own tokenizers can help us understand how one works, but in a production environment why would we want a custom tokenizer?  And if I built a custom tokenizer, how could I measure if it was better that NLTK's tokenizer?&lt;/p&gt;&#xA;" OwnerUserId="40801" LastEditorUserId="29575" LastEditDate="2018-01-01T00:52:54.410" LastActivityDate="2018-08-29T03:01:56.907" Title="Accuracy of word and sent tokenize versus custom tokenizers in nltk" Tags="|python|nlp|nltk|tokenization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26132" PostTypeId="1" AcceptedAnswerId="26141" CreationDate="2017-12-30T11:47:11.090" Score="0" ViewCount="140" Body="&lt;p&gt;I'm trying to classify same kind of text labels in to one category. For example, if I have labels like &lt;code&gt;qty, quantity, qty_no&lt;/code&gt; all of them should direct to Quantity. Since I'm new to data science, what's the best way to start this kind of thing?&lt;/p&gt;&#xA;" OwnerUserId="43958" LastActivityDate="2017-12-30T15:03:20.743" Title="Classify text labels in to a similar category" Tags="|python|classification|text|" AnswerCount="2" CommentCount="0" ClosedDate="2018-01-04T23:08:07.810" ContentLicense="CC BY-SA 3.0" />
  <row Id="26133" PostTypeId="2" ParentId="26129" CreationDate="2017-12-30T11:47:52.083" Score="1" Body="&lt;p&gt;Without reading your code (sorry!) I can suggest that you drop the verbs-only approach and use a word embedding to account for word similarity. Your feature dictionary is bound to find very few &quot;hits&quot; in the documents, unless you have a humongous dataset, and even then YMMV because you are just using verbs. Also: what's the justifications for using verbs? &quot;I liked&quot; and &quot;I didn't like&quot; both contain the verb &quot;to like&quot; and they mean opposite things. Same goes for hate (because you only have binary classification: no nuances) and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A popular python implementation of word2vec is &lt;a href=&quot;https://radimrehurek.com/gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt;, but you could use that of &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt; or some other embedding like the (allegedly superior) &lt;a href=&quot;https://blog.conceptnet.io/2016/05/25/conceptnet-numberbatch-a-new-name-for-the-best-word-embeddings-you-can-download/&quot; rel=&quot;nofollow noreferrer&quot;&gt;conceptnet numberbatch&lt;/a&gt;. If you want to summarize whole documents into numbers you can try doc2vec (aka paragraph2vec, &lt;a href=&quot;https://arxiv.org/abs/1405.4053&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper here&lt;/a&gt;), also available in gensim, tensorflow, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need a big dataset. Once you have this up and running you can try other classifiers.&lt;/p&gt;&#xA;" OwnerUserId="31718" LastActivityDate="2017-12-30T11:47:52.083" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26134" PostTypeId="1" AcceptedAnswerId="26139" CreationDate="2017-12-30T11:56:10.490" Score="12" ViewCount="23944" Body="&lt;p&gt;I'm new to datascience so please just don't blast me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a text book i found:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now, the reason we don't initialize the weights to zero is that the&#xA;  learning rate (eta) only has an effect on the classification outcome&#xA;  if the weights are initialized to non-zero values. If all the weights&#xA;  are initialized to zero, the learning rate parameter eta affects only&#xA;  the scale of the weight vector, not the direction.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now, why? Are you able to explain me? &lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="28175" LastEditDate="2018-01-25T16:40:39.747" LastActivityDate="2020-04-24T13:41:17.127" Title="Initialize perceptron weights with zero" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26136" PostTypeId="2" ParentId="26016" CreationDate="2017-12-30T12:43:01.230" Score="11" Body="&lt;p&gt;This response has been significantly modified from its original form. The flaws of my original response will be discussed below, but if you would like to see roughly what this response looked like before I made the big edit, take a look at the following notebook: &lt;a href=&quot;https://nbviewer.jupyter.org/github/dmarx/data_generation_demo/blob/54be78fb5b68218971d2568f1680b4f783c0a79a/demo.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nbviewer.jupyter.org/github/dmarx/data_generation_demo/blob/54be78fb5b68218971d2568f1680b4f783c0a79a/demo.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TL;DR: Use a KDE (or the procedure of your choice) to approximate $P(X)$, then use MCMC to draw samples from $P(X|Y) \propto P(Y|X)P(X)$, where $P(Y|X)$ is given by your model. From these samples, you can estimate the &quot;optimal&quot; $X$ by fitting a second KDE to the samples you generated and selecting the observation that maximizes the KDE as your maximum a posteriori (MAP) estimate.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Maximum Likelihood Estimation&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;... and why it doesn't work here&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;In my original response, the technique I suggested was to use MCMC to perform maximum likelihood estimation. Generally, MLE is a good approach to finding the &quot;optimal&quot; solutions to conditional probabilities, but we have a problem here: because we're using a discriminative model (a random forest in this case) our probabilities are being calculated relative to decision boundaries. It doesn't actually make sense to talk about an &quot;optimal&quot; solution to a model like this because once we get far enough away from the class boundary, the model will just predict ones for everything. If we have enough classes some of them might be completely &quot;surrounded&quot; in which case this won't be a problem, but classes on the boundary of our data will be &quot;maximized&quot; by values that aren't necessarily feasible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To demonstrate, I'm going to leverage some convenience code you can find &lt;a href=&quot;https://github.com/dmarx/data_generation_demo/blob/master/generative_sampler.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, which provides the &lt;code&gt;GenerativeSampler&lt;/code&gt; class which wraps code from my original response, some additional code for this better solution, and some additional features I was playing around with (some which work, some which don't) which I probably won't get into here.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.random.seed(123)&#xA;sampler = GenerativeSampler(model=RFC, X=X, y=y, &#xA;                            target_class=2, &#xA;                            prior=None, &#xA;                            class_err_prob=0.05, # &amp;lt;-- the score we use for candidates that aren't predicted as the target class&#xA;                            rw_std=.05,          # &amp;lt;-- controls the step size of the random walk proposal&#xA;                            verbose=True, &#xA;                            use_empirical=False)&#xA;samples, _ = sampler.run_chain(n=5000)&#xA;&#xA;burn = 1000&#xA;thin = 20&#xA;X_s = pca.transform(samples[burn::thin,:])&#xA;&#xA;# Plot the iris data&#xA;col=['r','b','g']&#xA;for i in range(3):&#xA;    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')&#xA;plt.plot(*X_s.T, 'k')&#xA;plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]))&#xA;plt.colorbar()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2kTTV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2kTTV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this visualization, the x's are the real data, and the class we're interested in is green. The line-connected dots are the samples we drew, and their color corresponds to the order in which they were sampled, with their &quot;thinned&quot; sequence position given by the color bar label on the right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the sampler diverged from the data fairly quickly and then just basically hangs out pretty far away from values of the feature space that correspond to any real observations. Clearly this is a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way we can cheat is to change our proposal function to only allow features to take values that we actually observed in the data. Let's try that and see how that changes the behavior of our result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.random.seed(123)&#xA;sampler = GenerativeSampler(model=RFC, X=X, y=y, &#xA;                            target_class=2, &#xA;                            prior=None, &#xA;                            class_err_prob=0.05, &#xA;                            verbose=True, &#xA;                            use_empirical=True) # &amp;lt;-- magic happening under the hood&#xA;samples, _ = sampler.run_chain(n=5000)&#xA;&#xA;X_s = pca.transform(samples[burn::thin,:])&#xA;&#xA;# Constrain attention to just the target class this time&#xA;i=2&#xA;plt.scatter(*X_r[y==i,:].T, c='k', marker='x')&#xA;plt.scatter(*X_s.T, c='g', alpha=0.3)&#xA;#plt.colorbar()&#xA;plt.show()&#xA;&#xA;&#xA;sns.kdeplot(X_s, cmap=sns.dark_palette('green', as_cmap=True))&#xA;plt.scatter(*X_r[y==i,:].T, c='k', marker='x')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/brU62.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/brU62.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U0ycE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U0ycE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is definitely a significant improvement and the mode of our distribution corresponds roughly to what we're looking for, but it's clear we're still generating a lot of observations that don't correspond to feasible values of $X$ so we shouldn't really trust this distribution either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The obvious solution here is to incorporate $P(X)$ somehow to anchor our sampling process to regions of the feature space that the data is actually likely to take. So let's instead sample from the joint probability of the likelihood given by the model, $P(Y|X)$, and a numerical estimate for $P(X)$ given by a KDE fit on the entire dataset. So now we're... sampling from... $P(Y|X)P(X)$....&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Enter Bayes Rule&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;After you hounded me to be less hand-wavey with the math here, I played around with this a fair amount (hence me building the &lt;code&gt;GenerativeSampler&lt;/code&gt; thing), and I encountered the problems I laid out above. I felt really, really stupid when I made this realization, but obviously what you are asking for calls for an application of Bayes rule and I apologize for being dismissive earlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're not familiar with bayes rule, it looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many applications the denominator is a constant which acts as a scaling term to ensure that the numerator integrates to 1, so the rule is often restated thusly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(B|A) \propto P(A|B)P(B)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or in plain English: &quot;the posterior is proportional to the prior times the likelihood&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look familiar? How about now:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(X|Y) \propto P(Y|X)P(X)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yeah, this is exactly what we worked up to earlier by constructing an estimate for the MLE that is anchored to the observed distribution of the data. I've never thought about Bayes rule this way, but it makes sense so thank you for giving me the opportunity to discover this new perspective. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To backtrack a tiny bit, MCMC is one of those applications of bayes rule where we can ignore the denominator. When we calculate the acceptanc ratio, $P(Y)$ will take the same value in both the numerator and denominator, canceling out, and allowing us to draw samples from unnormalized probability distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, having made this insight that we need to incorporate a prior for the data, let's do that by fitting a standard KDE and see how that changes our result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.random.seed(123)&#xA;sampler = GenerativeSampler(model=RFC, X=X, y=y, &#xA;                            target_class=2, &#xA;                            prior='kde',         # &amp;lt;-- the new hotness&#xA;                            class_err_prob=0.05,&#xA;                            rw_std=.05,          # &amp;lt;-- back to the random walk proposal&#xA;                            verbose=True, &#xA;                            use_empirical=False)&#xA;samples, _ = sampler.run_chain(n=5000)&#xA;&#xA;burn = 1000&#xA;thin = 20&#xA;X_s = pca.transform(samples[burn::thin,:])&#xA;&#xA;# Plot the iris data&#xA;col=['r','b','g']&#xA;for i in range(3):&#xA;    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')&#xA;plt.plot(*X_s.T, 'k--')&#xA;plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]), alpha=0.2)&#xA;plt.colorbar()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mzcNV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mzcNV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Much better! Now, we can estimate your &quot;optimal&quot; $X$ value using what's called the &quot;maximum a posteriori&quot; estimate, which is a fancy way of saying we fit a second KDE -- but to our samples this time -- and find the value that maximizes the KDE, i.e. the value corresponding to the mode of $P(X|Y)$. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# MAP estimation&#xA;&#xA;from sklearn.neighbors import KernelDensity&#xA;from sklearn.model_selection import GridSearchCV&#xA;from scipy.optimize import minimize&#xA;&#xA;grid = GridSearchCV(KernelDensity(), {'bandwidth': np.linspace(0.1, 1.0, 30)}, cv=10, refit=True)&#xA;kde = grid.fit(samples[burn::thin,:]).best_estimator_&#xA;&#xA;def map_objective(x):&#xA;    try:&#xA;        score = kde.score_samples(x)&#xA;    except ValueError:&#xA;        score = kde.score_samples(x.reshape(1,-1))&#xA;    return -score&#xA;&#xA;x_map = minimize(map_objective, samples[-1,:].reshape(1,-1)).x&#xA;&#xA;print(x_map)&#xA;&#xA;x_map_r = pca.transform(x_map.reshape(1,-1))[0]&#xA;col=['r','b','g']&#xA;for i in range(3):&#xA;    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')&#xA;sns.kdeplot(*X_s.T, cmap=sns.dark_palette('green', as_cmap=True))&#xA;plt.scatter(x_map_r[0], x_map_r[1], c='k', marker='x', s=150)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FhDoA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FhDoA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there you have it: the large black 'X' is our MAP estimate (those contours are the KDE of the posterior).&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-05T03:38:19.353" LastActivityDate="2018-01-05T03:38:19.353" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="26137" PostTypeId="2" ParentId="24995" CreationDate="2017-12-30T13:07:51.163" Score="1" Body="&lt;p&gt;What you want to do is to extract the relation within the sentence and in this case beyond - so basically a cross-sentence relation extraction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ambiguity and the context of words is a big deal when doing this kind of task. There might be multiple ways to get there, but it also depends on your goal (production environment, research or just doing it for fun to get into NLP). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you looking into Graph LSTMs (e.g. this &lt;a href=&quot;https://arxiv.org/abs/1708.03743&quot; rel=&quot;nofollow noreferrer&quot;&gt;Paper&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="31846" LastActivityDate="2017-12-30T13:07:51.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26138" PostTypeId="2" ParentId="26132" CreationDate="2017-12-30T13:19:34.377" Score="0" Body="&lt;p&gt;It would have been good to see some example of your trainings data in order to get a better understanding of the question you want to get answered. :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyway, as far as I understood it, you got several labels which some of them refer to the exact same category. In your case called &lt;em&gt;quantity&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my current point of view that is purely related to data preparation, rename the labels (in your traings data file e.g. CSV) to what you want them to be and then you are good to go. &lt;/p&gt;&#xA;" OwnerUserId="31846" LastActivityDate="2017-12-30T13:19:34.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26139" PostTypeId="2" ParentId="26134" CreationDate="2017-12-30T14:26:54.277" Score="9" Body="&lt;p&gt;If you initialize all weights with zeros then every hidden unit will get zero independent of the input. So, when all the hidden neurons start with the zero weights, then all of them will follow the same gradient and for this reason &quot;it affects only the scale of the weight vector, not the direction&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, having zero ( or equal) weights to start with will prevent the network from learning. The errors backpropagated through the network is proportional to the value of the weights. If all the weights are the same, then the backpropagated errors will be the same, and consequently, all of the weights will be updated by the same amount. To avoid this symmetry problem, the initial weights to the network should be unequal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at these links in more detail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;a href=&quot;https://www.quora.com/Why-does-it-work-to-initialize-weights-of-a-deep-Neural-Network-to-zero-plus-some-noise-N-0-epsilon-and-not-anything-else&quot; rel=&quot;noreferrer&quot;&gt;https://www.quora.com/Why-does-it-work-to-initialize-weights-of-a-deep-Neural-Network-to-zero-plus-some-noise-N-0-epsilon-and-not-anything-else&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;a href=&quot;http://staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html&quot; rel=&quot;noreferrer&quot;&gt;http://staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) &lt;a href=&quot;https://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers&quot;&gt;https://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) &lt;a href=&quot;https://stats.stackexchange.com/questions/27112/danger-of-setting-all-initial-weights-to-zero-in-backpropagation&quot;&gt;https://stats.stackexchange.com/questions/27112/danger-of-setting-all-initial-weights-to-zero-in-backpropagation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2017-12-30T14:26:54.277" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26140" PostTypeId="1" CreationDate="2017-12-30T14:59:26.090" Score="5" ViewCount="5897" Body="&lt;p&gt;I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set that will be around 100k units, how to I determine the best predictive or factor model to use when approaching this small'ish data set? I want to avoid overfitting etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any common pitfalls that I should avoid and/or tricks I should use to get more out of this data?&lt;/p&gt;&#xA;" OwnerUserId="43962" LastEditorUserId="43962" LastEditDate="2017-12-30T15:32:15.553" LastActivityDate="2017-12-30T21:49:11.210" Title="Small data set in machine learning" Tags="|machine-learning|data|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-03T21:33:39.680" ContentLicense="CC BY-SA 3.0" />
  <row Id="26141" PostTypeId="2" ParentId="26132" CreationDate="2017-12-30T15:03:20.743" Score="2" Body="&lt;p&gt;One way to do it could be with fuzzy string search. &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance&lt;/a&gt; algorithm is what you may use for it.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other. &lt;br&gt;&lt;br&gt; &lt;em&gt;From wikipedia&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;More on Levenshtein distance -&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/5859823/5406448&quot;&gt;https://stackoverflow.com/a/5859823/5406448&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43783" LastActivityDate="2017-12-30T15:03:20.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26142" PostTypeId="2" ParentId="17557" CreationDate="2017-12-30T17:35:34.460" Score="5" Body="&lt;p&gt;You've brought up some very good points. Let's walk through all of this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A word embedding is a mathematical representation of a word.&lt;/strong&gt; &#xA;This is needed since we cant work with text as plain input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to get these word embeddings, there a different ways, methods and settings on how to calculate them. cbow, skip-gram and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different pretrained word embeddings out there e.g.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md&quot; rel=&quot;noreferrer&quot;&gt;Fasttext&lt;/a&gt; 2017, trained on Wikipedia; &lt;a href=&quot;https://arxiv.org/abs/1607.04606&quot; rel=&quot;noreferrer&quot;&gt;cudos to Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://code.google.com/archive/p/word2vec/&quot; rel=&quot;noreferrer&quot;&gt;Word2Vec&lt;/a&gt; 2013, trained Google News dataset; &lt;a href=&quot;https://arxiv.org/abs/1310.4546&quot; rel=&quot;noreferrer&quot;&gt;cudos to Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;noreferrer&quot;&gt;Glove&lt;/a&gt; 2013, Wikipedia 2014 + Gigaword 5; &lt;a href=&quot;https://nlp.stanford.edu/pubs/glove.pdf&quot; rel=&quot;noreferrer&quot;&gt;cudos to Jeffrey Pennington, Richard Socher, Christopher D. Manning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is just an excerpt of the most well-known ones. As you can see, they used different data sets - thus different word vocabulary and word embeddings respectively.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't understand what is the real difference between these&#xA;  approaches regarding the desired outcome? Maybe the internal-only&#xA;  solution is not a semantic representation? What is the point of&#xA;  applying embedding layer to an external matrix of which the rows&#xA;  already have fix length?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Keras is an awesome toolbox and the embedding layer is a very good possibility to get things up and running pretty fast. Convert the text into one-hot/count matrix, use it as the input into the word embedding layer and you are set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand if you use pre-trained word vectors then you convert each word into a vector and use that as the input for your neural network. This approach would give you more flexibility when it comes to feature engineering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned above, pre-trained word vectors were given mostly general text data sets. You might run into the point where you have some special kind of data (e.g. Tweets) where people write or behave differently. So you might look into training your own embeddings, on your own dataset - at the end of the day it depends on your task/problem and the metrics that you are tuning towards.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you said correctly, it is to retrain the weights of the embeddings with the data set you use.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;One of the most important things in NLP is feature engineering. It is the same as you sitting in school and learning a language, what needs to be considered, vocabulary, it's grammar and rules. Things that makes it easier for you as a human being to understand the language. The same is applied here. You can see it as one part of feature engineering, it all sums up to the bigger picture. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And finally: how could a model predict the translation of a word which is not represented in the training set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Either you convert the word that couldn't be found to an &lt;code&gt;&amp;lt;UNK&amp;gt;&lt;/code&gt; token (unknown word) which then represents its group. However it requires having the &lt;code&gt;&amp;lt;UNK&amp;gt;&lt;/code&gt; token in the trained word embeddings. Or you could use fasttext binary file, which calculates word vectors on the fly for unknown words.&lt;/p&gt;&#xA;" OwnerUserId="31846" LastActivityDate="2017-12-30T17:35:34.460" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26144" PostTypeId="2" ParentId="26124" CreationDate="2017-12-30T20:14:04.663" Score="1" Body="&lt;p&gt;Before focusing on a specific algorithm or analytic tool, first identify one or more specific research hypotheses you would like to test with this data. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What are the reasons why you want to segment these customers? &lt;/li&gt;&#xA;&lt;li&gt;Based on what you already know about the business, what information might be required to meaningfully distinguish between customers? Does this information already exist in a useable format? If not, what steps would I need to take to gather the necessary information? &lt;/li&gt;&#xA;&lt;li&gt;What would be my null hypothesis / hypotheses?&lt;/li&gt;&#xA;&lt;li&gt;For each null hypothesis, what would be my alternate hypothesis? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you're trying to group customers into market segments, you'll need more information than what is provided in this data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a number of the variables in the data set referenced in the original post don't change across transactions within an individual customer (e.g. a business account never does individual account transactions), and therefore likely contribute little valuable information to the analysis (i.e. you already know that minors behave differently than adults).  Starting with a set of research hypotheses will allow you to determine the data you need to collect in order to conduct a useful analysis of the data. &lt;/p&gt;&#xA;" OwnerUserId="43256" LastEditorUserId="43256" LastEditDate="2017-12-30T21:03:56.213" LastActivityDate="2017-12-30T21:03:56.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26145" PostTypeId="1" AcceptedAnswerId="26147" CreationDate="2017-12-30T20:54:38.130" Score="1" ViewCount="193" Body="&lt;p&gt;Have you ever seen a neural network without matrices?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm asking, because I'm currently building one for educational purposes.&lt;/p&gt;&#xA;" OwnerUserId="43966" LastEditorUserId="29575" LastEditDate="2017-12-30T21:58:35.777" LastActivityDate="2017-12-30T21:58:35.777" Title="Neural network without matrices" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26146" PostTypeId="2" ParentId="26140" CreationDate="2017-12-30T20:56:16.350" Score="4" Body="&lt;p&gt;In general, Machine Learning algorithms handle volumes data. This doesn't mean that you cannot extract information from &quot;small&quot; data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Keep in mind:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Overfitting. With only a few data, the risk to overfit your model is far higher.&lt;/li&gt;&#xA;&lt;li&gt;Outliers. Those become nastier. If you have millions of data, a couple of outliers will not be a problem. But with only a few, they will definitely skew your results.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Models to use&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid overfitting, you need to avoid complexity. As a result, you need a model that has as less of parameters as you can. For example, linear/logistic regression could be what you need. Naive Bayes and SVM might work well, but this is not a generalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Cleaning up your data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Together with the outliers, you need to be careful with the rest of the cleaning process. Noise can skew a lot your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a nice blog post about &quot;small data&quot; and what you can do when you are working with them. Highly recommended.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/rants-on-machine-learning/what-to-do-with-small-data-d253254d1a89&quot; rel=&quot;nofollow noreferrer&quot;&gt;What to do with “small” data? (7 min read) &lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;About Outliers&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend the answer from this &lt;a href=&quot;https://datascience.stackexchange.com/questions/19849/is-it-good-to-remove-outliers-from-the-dataset&quot;&gt;Data Science Stack Exchange question&lt;/a&gt;. Short story. You need to examine the outliers one by one. There isn't any rule written in stone. But you need to identify if those outliers are just wrong input data or data that makes sense to be there, even as an outlier. For example, if you have the sales of an eshop per day and in your data you have a specific day where the sales are 10x more, you need to understand if that was an error or something happened that day to increase the sales (maybe a discount campaign or email advertisement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, it depends on what kind of problem you have. If your problem is an anomaly detection, then you don't want to remove the outliers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another suggestion (I don't have any source to link) is to update the values of those outliers with the mean, mode or another metric from your data. If you have only a few rows of them, removing a dozen will make it more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, this should be part of your model process. You try once with outliers, a few times without or with updated values and try to understand if it is underperformed or overfitted.&lt;/p&gt;&#xA;" OwnerUserId="201" LastEditorUserId="201" LastEditDate="2017-12-30T21:49:11.210" LastActivityDate="2017-12-30T21:49:11.210" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26147" PostTypeId="2" ParentId="26145" CreationDate="2017-12-30T21:00:22.367" Score="2" Body="&lt;p&gt;Matrix multiplication is just a simplified notation for a particular set of addition and multiplication operations. You can absolutely represent a neural network without invoking matrix notation, it'll just be really tedious (and it will run slower).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start with a single perceptron and build your way up from there. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2017-12-30T21:00:22.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26148" PostTypeId="1" CreationDate="2017-12-30T21:39:50.770" Score="1" ViewCount="197" Body="&lt;p&gt;I would like some help with the maths and to check I have understood the algorithm correctly. So I’ve been learning off of a &lt;a href=&quot;https://youtu.be/r1in0YNetG8&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; and I tried my own example. At the end of it I got dodgy results; I think they are “dodgy” because my outputs are not between 0 and 1. If I went with the higher output then yes it has correctly classified my new data but I’m just not sure since the numbers are completely different. I tried by doing the math by hand with a calculator instead of with python but I still got a weird result, so it makes me think that my method is wrong.Here’s what I did, please have a look and tell me what you think: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# data &#xA;&#xA;x = [[65,50],&#xA;&#xA;[62,49],&#xA;&#xA;[130,78],&#xA;&#xA;[124,79],&#xA;&#xA;[54,47],            # the heights/weights of a child&#xA;&#xA;[169,76],              (0), or an adult (1)&#xA;&#xA;[59,58],&#xA;&#xA;[169,79],&#xA;&#xA;[52,52]]&#xA;&#xA;&#xA;&#xA;y = [0,0,1,1,0,1,0,1,0]&#xA;&#xA;&#xA;&#xA;new = [58, 54]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Okay so here’s what I think I should do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Work out the prior probability (in this case 5/9 children, 4/9 adults)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Work out the means and variances for each feature and each class:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;x̄(C,H) = 58.4        x̄(C,W) = 51.2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;x̄(A,H) = 148          x̄(A,W) = 78&lt;/p&gt;&#xA;&#xA;&lt;p&gt;σ²(C,H) = 23.44    σ²(C,W) = 14.16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;σ²(A,H) = 445.5    σ²(A,W) = 1.5&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the Gaussians f(x,x̄, σ²):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;f(58, 58.4, 23.44) = 0.08212001659 (child height)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;f(54, 51.2, 14.16) = 0.08038037412 (child weight)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;f(58, 148, 445.5) = 2.129877168 x10(-6) (adult height)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;f(54, 78, 1.5) = 1.343766812 x10(-84) (adult weight)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multiply the Gaussians for the adult and child classes:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;0.08212001659 x 0.08038037412 = 6.600837656 x10(-3) (child Gauss)&#xA;2.129877168 x10(-6) x 1.343766812 x10(-84) = 2.862058252 x10(-90) (adult Gauss)&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;Calculate the probability using bayes theorem:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(child gauss x child prob)/(child gauss x child prob x adult gauss x adult prob)&#xA;(adult gauss x adult prob)/(child gauss x child prob x adult gauss x adult prob)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(6.600837656 x10(-3) x 5/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 9.841633571 x10(89)&#xA;(2.862058252 x10(-90) x 4/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 341.3788376&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...so those were the final probabilities. If i went with the higher one which you are supposed to then the algorithm would've correctly classed the data as a child, but the probabilities were not between 0 and 1 which makes me think this method may not work for other data. I also ran the data through Scikit learn which classified the new data as a child also. Please let me know what you think and if i have been unclear about something, let me know in the comments.&lt;/p&gt;&#xA;" OwnerUserId="37363" LastActivityDate="2017-12-30T23:44:31.383" Title="How do i use the Gaussian function with a Naive Bayes Classifier?" Tags="|python|classification|probability|naive-bayes-classifier|gaussian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26150" PostTypeId="2" ParentId="26148" CreationDate="2017-12-30T23:44:31.383" Score="2" Body="&lt;p&gt;You got the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayes%27_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayes theorem&lt;/a&gt; wrong. There should be a plus in the denominator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(child gauss x child prob)/(child gauss x child prob + adult gauss x adult prob) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(adult gauss x adult prob)/(child gauss x child prob + adult gauss x adult prob)&lt;/p&gt;&#xA;" OwnerUserId="43975" LastActivityDate="2017-12-30T23:44:31.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26151" PostTypeId="1" AcceptedAnswerId="26152" CreationDate="2017-12-31T02:04:49.840" Score="1" ViewCount="1233" Body="&lt;p&gt;I have a dataset which has information about food recipes (in german), that looks like this: &lt;a href=&quot;https://i.stack.imgur.com/YScVx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YScVx.png&quot; alt=&quot;reciped_data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link to a small .csv file (first 1000 rows of my data)&#xA;&lt;a href=&quot;https://drive.google.com/file/d/1C7thFlOnDn-oTc6AaDWA3CXXcX8m9NRu&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://drive.google.com/file/d/1C7thFlOnDn-oTc6AaDWA3CXXcX8m9NRu&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to cluster the recipe names into n-categories so that afterwards I can assign every recipe to a category. Of note, there are tags and ingredients for every recipe, maybe this information helps to refine the clusters?&#xA;For example the algorithm (maybe a semantic analysis?) should output: &lt;em&gt;categorised recipes into 200 biggest found categories: (hamburger, soup, pizza, ...)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I have for every recipe min. 1 image. The idea is to label my images with n-categories, afterwards to train a convolutional neural network with my data.&#xA;The input would be an food image, the output would be a category.&lt;/p&gt;&#xA;" OwnerUserId="43977" LastActivityDate="2017-12-31T22:16:53.610" Title="Cluster titles or ingredients of food into n-categories" Tags="|clustering|dataset|text-mining|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26152" PostTypeId="2" ParentId="26151" CreationDate="2017-12-31T02:12:56.073" Score="1" Body="&lt;p&gt;This sounds like a job for latent dirichlet alocation (i.e. topic modeling).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://radimrehurek.com/gensim/wiki.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://radimrehurek.com/gensim/wiki.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2017-12-31T02:12:56.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26153" PostTypeId="1" CreationDate="2017-12-31T02:47:48.060" Score="0" ViewCount="2260" Body="&lt;p&gt;I have a dataset with features that most of them are nominal categorical features, I have converted my model to indicator values, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Original&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F1,F2,L&#xA;1 ,1 ,50&#xA;2 ,3 ,30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After indicator values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F1-1,F1-2,F2-1,F2-3,L&#xA;1   ,0   ,1   ,0   ,50&#xA;0   ,1   ,0   ,1   ,30&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used different regression algorithm (Poisson, Bayesian, Decision Tree reg, Decision Forest reg, Boosted decision tree reg, linear regression, neural network), but all of then have low performance (r2 ~ 20-30)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I was thinking how regression can find values, then I found something interesting : relation of data with label&#xA;They are like below picture&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dZWAo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dZWAo.png&quot; alt=&quot;regression categorical features&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/bDqAq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bDqAq.png&quot; alt=&quot;regression nominal data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in most of the books and examples and samples suitable data for regression are like below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zq6bq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zq6bq.png&quot; alt=&quot;regression&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the point I got confused!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is how regression (or which algorithms) are suitable for predicting values in high categorical data&lt;/p&gt;&#xA;" OwnerUserId="43398" LastActivityDate="2017-12-31T06:13:00.850" Title="How regression algorithm works on categorical features" Tags="|machine-learning|regression|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26154" PostTypeId="2" ParentId="26153" CreationDate="2017-12-31T04:18:05.120" Score="1" Body="&lt;p&gt;TL;DR: It is a convention to convert categorical features to numeric features before you can use them in regression, or any other machine learning algorithm for that matter. &lt;em&gt;Technically&lt;/em&gt;, there is nothing that is stopping ML algorithms from working with categorical features but their software implementation would be prohibitively expensive, hence this convention in ML practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Turning to your problem, if you are trying to convert categorical label values into numerical values, there are various methods but it appears the best one for your data would be One-Hot encoder. Both &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit-Learn&lt;/a&gt; and &lt;a href=&quot;https://spark.apache.org/docs/2.1.0/ml-features.html#onehotencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;PySpark&lt;/a&gt;, and most other libraries provide handy functions for it, as it is very common operation. For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import OneHotEncoder&#xA;one_hot = OneHotEncoder()&#xA;one_hot.fit(...your_columns...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once you have all data in numeric form, you can use pretty much any algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the last figure, it is scatter plot between x and y, and it is not obvious what x and y are. Weirdly enough, it is &lt;em&gt;not&lt;/em&gt; one-on-one relationship because a given value of x seems to produce multiple value of y?!!! And I am not sure what you are plotting in earlier figures that is giving you straight lines!&lt;/p&gt;&#xA;" OwnerUserId="43918" LastEditorUserId="43918" LastEditDate="2017-12-31T04:30:12.463" LastActivityDate="2017-12-31T04:30:12.463" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26155" PostTypeId="1" CreationDate="2017-12-31T05:59:59.547" Score="3" ViewCount="897" Body="&lt;p&gt;I want to train my neural networks for the task of programming in a different language. So, I want to know whether is it possible or not to train a neural net for programming?&lt;/p&gt;&#xA;" OwnerUserId="26065" LastEditorUserId="29575" LastEditDate="2017-12-31T16:58:34.990" LastActivityDate="2018-02-01T00:05:35.023" Title="Is it possible to train the neural networks for programming" Tags="|machine-learning|neural-network|deep-learning|data-mining|bigdata|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26156" PostTypeId="2" ParentId="26153" CreationDate="2017-12-31T06:13:00.850" Score="0" Body="&lt;p&gt;The relation that your graph is showing for you categorical variable versus the numerical label does not look to be very strong as for every value of SGPriorityValue, you have all sorts of values of y it seems. For a categorical variable the relationship might not look as smooth  and continuous (Like the last graph you have) but there has to be pattern for any model to fit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;usually if a categorical variable is related to a numerical value, you can expect a graph like below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B9lVl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B9lVl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2017-12-31T06:13:00.850" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26157" PostTypeId="1" AcceptedAnswerId="26176" CreationDate="2017-12-31T07:18:18.557" Score="2" ViewCount="294" Body="&lt;p&gt;I am trying to substitute a sequence of words with some symbols from a long string appearing in multiple documents. As an example, suppose I want to remove:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Decision and analysis and comments&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;from a long string. Let the string be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s = Management's decision and analysis and comments is to be removed.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to remove &lt;code&gt;Decision and analysis and comments&lt;/code&gt; from &lt;code&gt;s&lt;/code&gt;. The catch is, between &lt;code&gt;Decision&lt;/code&gt;, &lt;code&gt;and&lt;/code&gt;, &lt;code&gt;analysis&lt;/code&gt;, &lt;code&gt;and&lt;/code&gt;, &lt;code&gt;comments&lt;/code&gt;, in &lt;code&gt;s&lt;/code&gt; there could be 0, 1 or multiple spaces and newline characters &lt;code&gt;(\n)&lt;/code&gt; showing up with no pattern in different documents, for example, one document shows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Management's decision  \n \n and analysis\n and \n comments is to be removed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while another has a different pattern. How do I account for this and still remove it from the string?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried the following, of course unsuccessfully:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;st = 'Management's decision  \n \n and analysis\n and  \n comments is to be removed'    &#xA;re.sub(r'Decision[\s\n]and[\s\n]analysis[\s\n]and[\s\n]comments','',s)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="29575" LastEditDate="2017-12-31T23:28:36.473" LastActivityDate="2017-12-31T23:28:36.473" Title="Facing a difficult regular expression issue in cleaning text data" Tags="|python|text-mining|data-cleaning|regex|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26159" PostTypeId="1" CreationDate="2017-12-31T08:54:25.113" Score="3" ViewCount="1551" Body="&lt;p&gt;I'm studying &lt;code&gt;Perceptron&lt;/code&gt; algorithm. Some books use this step function &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1 if x&gt;=0 else -1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where x is a dot product between the weights w and a sample x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other books use:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1 if x&gt;=0 else 0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What are the practical differences between these step functions?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="28175" LastEditDate="2018-01-22T01:20:36.827" LastActivityDate="2024-01-26T04:07:50.883" Title="Perceptron - Which step function to choose" Tags="|machine-learning|neural-network|deep-learning|perceptron|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26160" PostTypeId="1" CreationDate="2017-12-31T09:17:21.043" Score="6" ViewCount="2002" Body="&lt;p&gt;I am working on a regression problem, where I want to modify the loss function in xgboost library such that my predictions should never be lesser than the actual value. I have written this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def custom_loss(preds, dtrain):&#xA;    labels = dtrain.get_label()&#xA;    df = preds - labels&#xA;    df = pd.DataFrame(df, columns=['val'])&#xA;    df['valg'] = df['val'].apply(lambda x: 10*abs(x) if x&amp;lt;0 else x)&#xA;    grad = df['valg'].as_matrix()&#xA;    return preds-labels, grad&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This essentially means that I want to penalize those predictions which are lesser than my actual value more. However, this isn't working and there is no improvement in my predictions. Can anyone help me figure out where I am going wrong? Thanks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whole Python script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = {&quot;booster&quot; : &quot;gbtree&quot;,&#xA;  &quot;eta&quot;: 0.20,&#xA;  &quot;max_depth&quot;: 4,&#xA;  &quot;subsample&quot;: 0.75,&#xA;  &quot;colsample_bytree&quot;: 0.65,&#xA;  &quot;silent&quot;: 1,&#xA;  &quot;eval_metric&quot;: &quot;rmse&quot;,&#xA;  }&#xA;num_round = 400&#xA;&#xA;def custom_loss(preds, dtrain):&#xA;    labels = dtrain.get_label()&#xA;    df = preds - labels&#xA;    df = pd.DataFrame(df, columns=['val'])&#xA;    df['valg'] = df['val'].apply(lambda x: 5*abs(x) if x&amp;lt;0 else x)&#xA;    grad = df['valg'].as_matrix()&#xA;    return preds-labels, grad&#xA;&#xA;&#xA;dtrain = xgb.DMatrix(X_train.drop('price_act', axis=1), &#xA;label=X_train['price_act'])&#xA;dtest = xgb.DMatrix(X_test.drop('price_act',axis=1), &#xA;label=X_test['price_act'])&#xA;&#xA;watchlist = [(dtrain,'train'), (dtest,'eval')]&#xA;&#xA;bst = xgb.train(params, dtrain, num_round, watchlist, custom_loss)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38308" LastActivityDate="2020-02-08T20:28:33.917" Title="Custom objective function in xgboost for Regression" Tags="|machine-learning|python|regression|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26161" PostTypeId="2" ParentId="26159" CreationDate="2017-12-31T09:21:00.610" Score="0" Body="&lt;p&gt;They have the same meaning in this context although during training using &lt;code&gt;Rosenblatt&lt;/code&gt; update rule, the former may have great changes during each update. &lt;code&gt;Perceptron&lt;/code&gt; is used for binary classification which means there are two possible classes to classify. If the result of inner product, here dot product, is greater than or equal to zero, the class of inputs will be the first class and if it's smaller than zero the class of inputs would be the other class. &lt;code&gt;Perceptron&lt;/code&gt; just has one neuron. It's a simple linear classifier. The value of threshold is just important. Means that if the product is greater than zero, the input belongs to e.g. positive class and if is negative it belongs to negative class. The step functions and &lt;code&gt;Rosenblatt&lt;/code&gt; update rule are not used any more. They have so much oscillation. Today networks learn using gradient descending algorithms which uses the concept of derivative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you progress, you will see that neural nets which use other activation functions like &lt;code&gt;Sigmoid&lt;/code&gt; or &lt;code&gt;Tanh&lt;/code&gt; are different. the former has &lt;code&gt;0.5&lt;/code&gt; expected value and the latter has &lt;code&gt;0&lt;/code&gt; expected value which causes the second learn so much faster. Although now a days &lt;code&gt;ReLU&lt;/code&gt; is more popular among other activation functions.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2017-12-31T09:46:37.823" LastActivityDate="2017-12-31T09:46:37.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26162" PostTypeId="1" CreationDate="2017-12-31T10:47:03.243" Score="1" ViewCount="213" Body="&lt;p&gt;Given either, different decision trees each based on a particular feature value (like separate models for each male and female) or a single decision tree, should both give the same result?&lt;/p&gt;&#xA;" OwnerUserId="43983" LastEditorUserId="29575" LastEditDate="2017-12-31T17:01:55.907" LastActivityDate="2024-03-16T02:07:30.910" Title="separate decision tree for categorical feature values" Tags="|classification|random-forest|decision-trees|xgboost|" AnswerCount="2" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26163" PostTypeId="1" CreationDate="2017-12-31T11:39:33.533" Score="2" ViewCount="23" Body="&lt;p&gt;I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file &lt;code&gt;yarn-site.xml&lt;/code&gt;, I had to insert:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;name&amp;gt;yarn.nodemanager.aux-services&amp;lt;/name&amp;gt;&#xA;&amp;lt;value&amp;gt;mapreduce_shuffle&amp;lt;/value&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the &quot;mapreduce_shuffle&quot; option?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="29575" LastEditDate="2017-12-31T17:54:45.697" LastActivityDate="2017-12-31T17:54:55.190" Title="Yarn service parameter for pseudodistributed" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-01T18:57:39.550" ContentLicense="CC BY-SA 3.0" />
  <row Id="26164" PostTypeId="1" AcceptedAnswerId="26395" CreationDate="2017-12-31T13:31:02.697" Score="1" ViewCount="423" Body="&lt;p&gt;I've implemented Nesterov Accelerated Gradient (NAG) (&lt;a href=&quot;http://ruder.io/optimizing-gradient-descent/index.html#adamax&quot; rel=&quot;nofollow noreferrer&quot;&gt;link, section: &quot;Nesterov Accelerated Gradient&quot;&lt;/a&gt;), however not sure it's viable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_t = \gamma v_{t-1} +\eta\nabla_\theta J(\theta - \gamma v_{t-1})$$&#xA;$$\theta = \theta - v_t$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I attempted the following (consider a minibatch consisting of merely 1 training case):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;run a forward pass, and get the error&lt;/li&gt;&#xA;&lt;li&gt;Apply the scaled previous momentum, (&lt;code&gt;momentum_tMinOne * coeff&lt;/code&gt;) to the weights&lt;/li&gt;&#xA;&lt;li&gt;run a backward pass using the error of each pass.&lt;/li&gt;&#xA;&lt;li&gt;set &lt;code&gt;momentum_tMinOne = currentGradient&lt;/code&gt;, but don't apply it to the weights yet&lt;/li&gt;&#xA;&lt;li&gt;repeat forward pass, from 1.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do I require to remember something like &lt;code&gt;momentum_tMinTwo&lt;/code&gt;,  or just keeping hold of the previous momentum will suffice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct that this way we don't need to keep two matrices for momentums, but only need &lt;code&gt;momentum_tMinOne&lt;/code&gt; matrix?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2017-12-31T20:29:45.250" LastActivityDate="2019-10-01T12:15:11.327" Title="Nesterov Accelerated gradient is in correct order?" Tags="|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26165" PostTypeId="1" AcceptedAnswerId="28038" CreationDate="2017-12-31T14:46:10.767" Score="11" ViewCount="1510" Body="&lt;p&gt;In the paper &lt;a href=&quot;https://arxiv.org/abs/1609.04802&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network&lt;/em&gt;&lt;/a&gt; by Christian Ledig et al., the distance between images (used in the loss function) is calculated from feature maps extracted from the VGG19 network. The two used in the article are (a bit confusingly) called VGG22 and VGG54.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are these feature maps? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do the designations &quot;22&quot; and &quot;54&quot; mean?    &lt;/p&gt;&#xA;" OwnerUserId="43472" LastEditorUserId="29575" LastEditDate="2017-12-31T17:03:56.130" LastActivityDate="2018-02-20T15:33:15.863" Title="What are &quot;VGG54&quot; and &quot;VGG22&quot; derived from the VGG19 CNN?" Tags="|deep-learning|cnn|gan|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26168" PostTypeId="1" CreationDate="2017-12-31T17:10:35.307" Score="0" ViewCount="1178" Body="&lt;p&gt;I see the authors of this &lt;a href=&quot;https://arxiv.org/pdf/1703.07464.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I have looked at the &lt;a href=&quot;https://github.com/rksltnl/Deep-Metric-Learning-CVPR16/tree/master/code/evaluation&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt;, I am not sure about the actual algorithm.&lt;/p&gt;&#xA;" OwnerUserId="33569" LastEditorUserId="33569" LastEditDate="2017-12-31T17:17:40.020" LastActivityDate="2024-02-08T11:02:35.390" Title="How to measure F1 score and NMI for clustering task?" Tags="|clustering|k-means|model-evaluations|mutual-information|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26169" PostTypeId="2" ParentId="26163" CreationDate="2017-12-31T17:54:55.190" Score="1" Body="&lt;p&gt;The &lt;code&gt;mapreduce_shuffle&lt;/code&gt; in this config file is part of &lt;a href=&quot;https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plugable Shuffle and Sort&lt;/a&gt;.  Shuffle and Sort are what connect the mappers to the reducers. A nice graphical representation of this is the following (shuffle is called &quot;copy&quot; in this figure):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aIGRQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aIGRQ.png&quot; alt=&quot;Pipeline&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Hadoop arcitecture alllows these steps to be customized:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;mapreduce_shuffle&lt;/code&gt; parameter is mentioned in the &lt;a href=&quot;https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html#NodeManager_Configuration_properties.2C_yarn-site.xml_in_all_nodes:&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hadoop Docs&lt;/a&gt;, and is the default value for this configuration.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-12-31T17:54:55.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26170" PostTypeId="2" ParentId="26168" CreationDate="2017-12-31T19:47:15.343" Score="0" Body="&lt;p&gt;Use the original papers, and books, on F1 and NMI.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is little reason to only use a 2016 arxiv (non-reviewed) paper that &lt;em&gt;uses&lt;/em&gt; them, because they have been introduced long before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't see F1 mentioned anywhere in the paper,and it would be an uncommon choice for clustering (see literature for details).&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-12-31T19:52:20.880" LastActivityDate="2017-12-31T19:52:20.880" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26171" PostTypeId="2" ParentId="26155" CreationDate="2017-12-31T20:06:00.457" Score="0" Body="&lt;p&gt;The closest thing I can think of in the literature is pix2code, where the authors trained a GAN to generate simple DSL code for a UI based on an input image of the intended UI layout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1705.07962.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1705.07962.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2017-12-31T20:06:00.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26172" PostTypeId="1" CreationDate="2017-12-31T20:22:05.330" Score="1" ViewCount="1917" Body="&lt;p&gt;When we fit an LSTM model, each LSTM has a cell state which contains the information we want. I am wondering what's the dimension in the cell state (i.e. cell state itself should be a vector, then how large is the vector?)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BuVnQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BuVnQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;(from &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&lt;/a&gt; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i.e. In the above architecture, how do we determine the dimension of the matrix &lt;code&gt;Wc&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am using Keras package, how do I find the Cell state dimensions of each LSTM unit? Is it something that one can adjust? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="17310" LastEditDate="2017-12-31T20:36:10.977" LastActivityDate="2020-05-16T18:05:49.760" Title="LSTM unit: cell state dimension" Tags="|keras|rnn|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26173" PostTypeId="2" ParentId="26151" CreationDate="2017-12-31T20:54:19.563" Score="1" Body="&lt;p&gt;I would suggest looking into &lt;em&gt;frequent itemset mining&lt;/em&gt; such as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;APRIORI algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequent itemsets will then correspond to &lt;strong&gt;frequent combinations of ingredients&lt;/strong&gt;. It is easy to imagine this to yield interesting results, such as particular genres of cuisine, but also obvious patterns such as milk, eggs, butter and flour.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2017-12-31T22:16:53.610" LastActivityDate="2017-12-31T22:16:53.610" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26174" PostTypeId="1" AcceptedAnswerId="26177" CreationDate="2017-12-31T21:27:40.880" Score="6" ViewCount="10278" Body="&lt;p&gt;For classification, it is obvious how a decision tree is used to make a prediction.You just have to find the final leaf. However for regression problems, how can you find the prediction considering the continous aspect of the variable to predict?&lt;/p&gt;&#xA;" OwnerUserId="40515" LastActivityDate="2018-01-03T09:38:19.970" Title="How does decision tree for regression makes a prediction?" Tags="|regression|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26175" PostTypeId="1" CreationDate="2017-12-31T23:10:56.523" Score="7" ViewCount="1709" Body="&lt;p&gt;I am trying to understand how the weight matrix in an LSTM cell is used. An LSTM unit has several weight matrix: &lt;code&gt;Wf, Wi, Wc, Wo&lt;/code&gt; like below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/p7P2U.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/p7P2U.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OzaVr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OzaVr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6fcea.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6fcea.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;( from &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&lt;/a&gt; )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the same time, I am playing with the Keras LSTM and studying its source code:&#xA;&lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L1871&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L1871&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the source code, there is only one kernel mentioned. I am wondering is it referring to &lt;code&gt;Wc&lt;/code&gt; only? Then where are the other weight matrix &lt;code&gt;Wf, Wi, Wo&lt;/code&gt; initialized and used? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2020-01-05T04:36:19.243" Title="Kerns LSTM kernel" Tags="|keras|rnn|lstm|kernel|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26176" PostTypeId="2" ParentId="26157" CreationDate="2017-12-31T23:25:19.367" Score="2" Body="&lt;p&gt;To remove multiple white space matches, you will need &lt;code&gt;[\s\n]+&lt;/code&gt;, note the inclusion of the &lt;code&gt;+&lt;/code&gt; (match one or more).&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here is a function which will build the regex automatically from a text snippet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def remove_words(to_clean, words, flags=re.IGNORECASE):&#xA;    regex = r'[\s\n]+'.join([''] + words.split() + [''])&#xA;    return re.sub(regex, ' ', to_clean, flags)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;st = &quot;Management's decision  \n \n and analysis\n &quot; \&#xA;     &quot;and  \n comments is to be removed&quot;&#xA;print(remove_words(st, 'decision and analysis and comments'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Management's is to be removed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2017-12-31T23:25:19.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26177" PostTypeId="2" ParentId="26174" CreationDate="2017-12-31T23:52:28.540" Score="2" Body="&lt;p&gt;Depends on the implementation but commonly used is a certain cursive partitioning method called CART.  The algorithm works as following:  It searches for every distinct values for your predictors and chooses to the split based on what minimize the SSE for two groups of dependent variables.  The difference is within the SSE is usually the difference between the actual value and the average of the sample or the difference between the actual value and the output of a linear regression.  For each group, the method will recursively split the predictor values within the groups.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, the method stops when a certain sample size threshold is met.  The node with the lowest SSE becomes the root node.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: You can read more about tree based regressions here: &lt;a href=&quot;http://www.dcc.fc.up.pt/~ltorgo/PhD/th3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tree-based regressions&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-02T13:14:09.117" LastActivityDate="2018-01-02T13:14:09.117" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26178" PostTypeId="2" ParentId="24867" CreationDate="2018-01-01T00:51:57.277" Score="0" Body="&lt;p&gt;You probably need to get more training data as you increase your features to justify the increase. Insufficient data can cause the accuracy to be reduced.&lt;/p&gt;&#xA;" OwnerUserId="43996" LastEditorUserId="29575" LastEditDate="2018-01-01T00:57:25.423" LastActivityDate="2018-01-01T00:57:25.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26179" PostTypeId="2" ParentId="26131" CreationDate="2018-01-01T00:52:58.783" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why would we want a custom tokenizer?  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer.  Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, is useful and important.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How could I measure if it was better that NLTK's tokenizer?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Anytime you are trying to quantify &lt;em&gt;performance&lt;/em&gt; (ie: better) you will need to first define what is meant by better.  Once this is done, then typically you would perform the using the various methods under measurement, and then compare the results against your definition of better.  A couple of links which discuss these topics:  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://alexpnt.github.io/2017/08/27/nlp-performance/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Performance of Different NLP Toolkits in Formal and Social Text&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.cornell.edu/home/cardie/papers/hlt-emnlp05-art.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Optimizing to Arbitrary NLP Metrics using Ensemble Selection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2018-01-01T00:52:58.783" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26180" PostTypeId="1" CreationDate="2018-01-01T01:58:56.993" Score="31" ViewCount="69917" Body="&lt;p&gt;I see some literature consider L2 loss (least squared error) and mean squared error loss are two different kinds of loss functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it seems to me these two loss functions essentially compute the same thing (with a 1/n factor difference).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering if I have missed anything? Is there any scenario that one should use one of the two loss functions?&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="26019" LastEditDate="2020-05-19T03:14:31.187" LastActivityDate="2023-06-24T21:44:58.000" Title="L2 loss vs. mean squared loss" Tags="|loss-function|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26181" PostTypeId="1" CreationDate="2018-01-01T06:34:38.540" Score="2" ViewCount="61" Body="&lt;p&gt;I came across this statement from &lt;a href=&quot;https://cntk.ai/pythondocs/CNTK_103A_MNIST_DataLoader.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNTK 103A&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;gzfname, h = urlretrieve(src, './delete.me')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I understand all the rest but that &lt;code&gt;gzfname, h = ......&lt;/code&gt;&lt;/p&gt;&#xA;&lt;p&gt;What exactly is the purpose of &lt;code&gt;h&lt;/code&gt;, is it a filehead thing?&lt;/p&gt;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;def load_or_download_mnist_files(filename, num_samples, local_data_dir):&#xA;    gzfname, h = urlretrieve(src, './delete.me')&#xA;    if (local_data_dir):&#xA;        local_path = os.path.join(local_data_dir, filename)&#xA;    else:&#xA;        local_path = os.path.join(os.getcwd(), filename)&#xA;&#xA;    if os.path.exists(local_path):&#xA;        gzfname = local_path&#xA;    else:&#xA;        local_data_dir = os.path.dirname(local_path)&#xA;        if not os.path.exists(local_data_dir):&#xA;            os.makedirs(local_data_dir)&#xA;        filename = &amp;quot;http://yann.lecun.com/exdb/mnist/&amp;quot; + filename&#xA;        print (&amp;quot;Downloading from&amp;quot; + filename, end=&amp;quot; &amp;quot;)&#xA;        gzfname, h = urlretrieve(filename, local_path)&#xA;        print (&amp;quot;[Done]&amp;quot;)&#xA;&#xA;    return gzfname&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43996" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-31T20:17:32.127" Title="Unknown return value in CNTK 103A" Tags="|python|dataset|image-classification|mnist|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26182" PostTypeId="1" AcceptedAnswerId="26204" CreationDate="2018-01-01T11:44:55.487" Score="4" ViewCount="2325" Body="&lt;p&gt;I have this formula for the density estimation.&#xA;$$p_n(x) = \frac{k_n / n}{V_n}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been told that with a Parzen window approach you can specify $V_n$ as a function of $n$. So if $V$ decreased when $n$ increased it is clear that it is a fixed volume.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also been told that with a knn approach you specify $k_n$ as a function of $n$. So if you increase as $n$ is raised, it is clear that the volume is dependent of the volume.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QitqN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QitqN.png&quot; alt=&quot;parzen and knn&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can anyone explains me the above statements. I think it is somewhat clear to me how knn and Parzen works. (knn count the $k$ nearest neighbor, at the new sample is assigned to the class which has most votes. In Parzen the volume is fixed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also do not understand the two formulas in the figure. The figure illustrated two methods for estimating the density at a point $x$ at the center of&#xA;each square. The top knn, bottom Parzen&lt;/p&gt;&#xA;" OwnerUserId="44004" LastEditorUserId="29575" LastEditDate="2018-01-01T14:53:14.330" LastActivityDate="2018-01-02T13:48:16.877" Title="Parzen and k nearest neighbor" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26183" PostTypeId="1" AcceptedAnswerId="26193" CreationDate="2018-01-01T12:18:06.063" Score="4" ViewCount="7828" Body="&lt;p&gt;I am new to data science, I have clustered some data using Scipy agglomerative clustering. how can I fit new data into the learned clusters?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))&#xA;z = hac.linkage(dm, method='average')&#xA;cluster = hac.fcluster(z, t=100, criterion='distance')&#xA;leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='distance')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to cluster new data into the same clusters, How can I do it?&lt;/p&gt;&#xA;" OwnerUserId="42904" LastEditorUserId="42904" LastEditDate="2018-01-02T10:33:40.833" LastActivityDate="2018-01-02T18:46:36.113" Title="Clustering Data to learned cluster" Tags="|python|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26184" PostTypeId="2" ParentId="15903" CreationDate="2018-01-01T12:28:20.033" Score="1" Body="&lt;p&gt;Building on &lt;a href=&quot;https://datascience.stackexchange.com/users/14675/ncasas&quot;&gt;ncasas&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/users/27250/horacet&quot;&gt;horaceT&lt;/a&gt; answers, ConvNets are very efficient because:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;They are invavriant to geometrical transformations and learn features that get increasingly complicated and detailed, hence being powerful hierarchical &lt;strong&gt;feature extractors&lt;/strong&gt; thanks to the convolutional layers.&lt;/li&gt;&#xA;&lt;li&gt;They combine the extracted features and aggregate them in a non linear fashion to predict the output and therefore being &lt;strong&gt;robust classifiers&lt;/strong&gt; thanks to the fully connected layers.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you want to learn more about convnets and the different blocks building them as well as the intuitions that underly, here's a &lt;a href=&quot;https://ahmedbesbes.com/understanding-deep-convolutional-neural-networks-with-a-practical-use-case-in-tensorflow-and-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; I recently wrote on my personal blog that goes through the details.&lt;/p&gt;&#xA;" OwnerUserId="44006" LastEditorUserId="44006" LastEditDate="2018-01-01T12:44:46.513" LastActivityDate="2018-01-01T12:44:46.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26185" PostTypeId="1" CreationDate="2018-01-01T14:54:21.630" Score="0" ViewCount="38" Body="&lt;p&gt;If i have a lot of data points describing the price of a used car. How would I find the market value of the car (assuming that the price points in the data set are the only determinant used, and the basis of determination will be the frequency [higher the frequency the better] price data point occurence for that particular car).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A count of absolute value recurrences will not work, as I want to bucket numbers that are similar (less than 5% difference of total price) into one category.&lt;/p&gt;&#xA;" OwnerUserId="43040" LastActivityDate="2018-01-02T05:23:57.320" Title="Finding an appropriate price using thousands of data points" Tags="|metadata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26186" PostTypeId="1" CreationDate="2018-01-01T16:17:23.110" Score="0" ViewCount="72" Body="&lt;p&gt;I have a collection of articles and for each articles I have the number of clicks and views, so I can measure the Click-Through Rate (CTR) as the number of clicks over the number of views.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the CTR is not a very informative quantity because, for instance, two pages with CTR 10% can have very different values for the clicks and the views. For example, page A can have 10 views and 1 clicks and page B 10 million views and 1 million clicks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was wondering if there was an algorithm that can rank pages based on their CTR that also account for the absolute value of clicks and views because in the previous example page B may have a better ranking than A.&lt;/p&gt;&#xA;" OwnerUserId="13680" LastActivityDate="2018-05-01T19:13:50.527" Title="How to rank pages based on CTR" Tags="|algorithms|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26187" PostTypeId="2" ParentId="26183" CreationDate="2018-01-01T16:49:35.500" Score="1" Body="&lt;p&gt;Clustering is not predictive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e., the models do &lt;em&gt;not&lt;/em&gt; generalize to new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually your best approach is to do a 1 nearest neighbor &lt;em&gt;classification&lt;/em&gt;, but you could train any other classifier on your data set.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-01T16:49:35.500" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26188" PostTypeId="2" ParentId="26186" CreationDate="2018-01-01T16:55:06.617" Score="1" Body="&lt;p&gt;Just CTR is not important. Also, cost-per-click should be applied. Hence, to rank sites in an ads auction, using $CPC\times CTR$. &lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-01T16:55:06.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26189" PostTypeId="2" ParentId="14329" CreationDate="2018-01-01T17:07:44.193" Score="1" Body="&lt;p&gt;The nInputPlane is the depth or the number of the layers of the input image. In case of RGB images, this should be 3 which corresponds to the first number in the &lt;code&gt;input_shape=(3, 256, 256)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nOutputPlane is the the number of the layers of the volume that the convolution step will produce which is also the number of filters/kernels applied to the input. By convention, there is an output layer for each filter. This corresponds to the first argument of the &lt;code&gt;Convolution2D&lt;/code&gt; function.&lt;/p&gt;&#xA;" OwnerUserId="30311" LastActivityDate="2018-01-01T17:07:44.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26190" PostTypeId="2" ParentId="25989" CreationDate="2018-01-01T17:29:29.510" Score="3" Body="&lt;p&gt;You can easily visualize word2vec vectors using &lt;a href=&quot;https://www.tensorflow.org/get_started/summaries_and_tensorboard&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorBoard&lt;/a&gt;, which is very easy to understand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;I have made a &lt;a href=&quot;https://www.youtube.com/watch?v=BkeQzJt0f5A&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; about using TensorBoard.&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="44011" LastEditorUserId="29575" LastEditDate="2018-01-01T17:37:23.597" LastActivityDate="2018-01-01T17:37:23.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26191" PostTypeId="2" ParentId="13461" CreationDate="2018-01-01T17:54:59.143" Score="5" Body="&lt;p&gt;This would be how to predict for one element, this time number 17.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.predict_classes(X_test[17:18])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44012" LastEditorUserId="29575" LastEditDate="2018-01-01T18:18:25.173" LastActivityDate="2018-01-01T18:18:25.173" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26192" PostTypeId="2" ParentId="26181" CreationDate="2018-01-01T18:32:40.633" Score="1" Body="&lt;p&gt;&lt;code&gt;gzfname, h&lt;/code&gt; is doing tuple unpacking from the return of &lt;code&gt;urlretrieve&lt;/code&gt;.  From the &lt;a href=&quot;https://docs.python.org/3/library/urllib.request.html#urllib.request.urlretrieve&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;, &lt;code&gt;urlretrieve&lt;/code&gt; returns:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Return a tuple (filename, headers) where filename is the local file name under which the object can be found, and headers is whatever the info() method of the object returned by urlopen() returned (for a remote object). Exceptions are the same as for urlopen().&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the &lt;code&gt;info()&lt;/code&gt; method of the &lt;a href=&quot;https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;urlopen&lt;/code&gt;&lt;/a&gt; object returns:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;info()&lt;/code&gt; — return the meta-information of the page, such as headers, in the form of an email.message_from_string() instance (see Quick Reference to HTTP Headers)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, &lt;code&gt;h&lt;/code&gt; is meta information about the retrieved page.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2018-01-01T18:32:40.633" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26193" PostTypeId="2" ParentId="26183" CreationDate="2018-01-01T21:52:41.597" Score="5" Body="&lt;p&gt;The word &quot;prediction&quot; does not &lt;em&gt;belong&lt;/em&gt; to any specific type of machine learning. There is nothing wrong with &quot;predicting&quot; new data to the cluster it belongs to; (e.g. there are many applications that place new customers into pre-discovered market segments). A conditional probability, like that used in classification, is not &quot;stronger&quot; than an unsupervised approach, as it rests its assumption on properly labelled classes; something that is &lt;strong&gt;not&lt;/strong&gt; guaranteed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why there are packages that provide a &lt;strong&gt;predict&lt;/strong&gt; function to clustering algorithms. &lt;a href=&quot;https://stackoverflow.com/questions/20621250/simple-approach-to-assigning-clusters-for-new-data-after-k-means-clustering&quot;&gt;Here&lt;/a&gt; is an example using the flexclust package with the kcaa function. That being said, the prediction step is usually handled by a supervised classifier, so the approach would be to sit a classifier on top of your learned clusters (treating cluster assignments as &quot;labels&quot;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You just have to reason about your &lt;em&gt;weaknesses&lt;/em&gt;. As stated above, the weakness in classification is the assumption that labelled data is tagged correctly, whereas the weakness in clustering is that your discovered clusters are assumed to be valid. Unsupervised approached cannot be validated the same way it is done with classification. Clustering requires a variety of &lt;em&gt;cluster validity techniques&lt;/em&gt; along with &lt;em&gt;domain experience&lt;/em&gt; (e.g. show campaign managers your market segments to validate customer types).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, you are just matching an incoming vector (new data) to the cluster most similar. For example, in k-means this could be accomplished by finding the smallest distance between the incoming vector and all the centroids of your clusters. This kind of pattern matching depends on the data you are using. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This works best for clustering techniques that have well-defined cluster objects with exemplars in the center, like k-means. Using hierarchical techniques means you would need to &lt;strong&gt;cut&lt;/strong&gt; the tree to obtain flat clusters, then use the &quot;label&quot; assignment to run a classifier on top. This comes baked with a lot of assumptions, so you need to make sure you understand your data very well, and validate any clusters with non-technical users that have deep domain experience. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;POSSIBLE APPROACH&#xA;If you're bent on using hierarchical clustering, then here is the general approach. Note I am not suggesting this is the best way. Every approach comes baked with a number of assumptions. You will need to work to understand your data, attempt many models, validate with stakeholders, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Readers can use the &lt;a href=&quot;https://joernhees.de/blog/2015/08/26/scipy-hierarchical-clustering-and-dendrogram-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; by Jörn Hees to get started in hierarchical clustering if needed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Create some example data:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import pyplot as plt&#xA;from scipy.cluster.hierarchy import dendrogram, linkage&#xA;import numpy as np&#xA;&#xA;np.random.seed(42)  &#xA;a = np.random.multivariate_normal([10, 0], [[3, 1], [1, 4]], size=[100,])&#xA;b = np.random.multivariate_normal([0, 20], [[3, 1], [1, 4]], size=[50,])&#xA;X = np.concatenate((a, b),)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Confirm clusters exist in synthetic data:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(X[:,0], X[:,1])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/umdkg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/umdkg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Generate the linkage matrix using the Ward variance minimization algorithm&lt;/strong&gt;:&#xA;(This assumes your data should be be clustered to minimize the overall intra-cluster variance in euclidean space. If not, try Manhattan, cosine or hamming. You can also try different linking options).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Z = linkage(X, 'ward')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Check the Cophenetic Correlation Coefficient to assess quality of clusters:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.cluster.hierarchy import cophenet&#xA;from scipy.spatial.distance import pdist&#xA;&#xA;c, coph_dists = cophenet(Z, pdist(X))&#xA;&#xA;0.98001483875742679&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Calculate full dendrogram:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(25, 10))&#xA;plt.title('Hierarchical Clustering Dendrogram')&#xA;plt.xlabel('sample index')&#xA;plt.ylabel('distance')&#xA;dendrogram(&#xA;    Z,&#xA;    leaf_rotation=90.,  # rotates the x axis labels&#xA;    leaf_font_size=8.,  # font size for the x axis labels&#xA;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pNIHP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pNIHP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Determine the number of clusters&lt;/strong&gt; (e.g. can be done manually by looking for any large jumps in the dendrogram...see Jörn's blog for plotting function): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vdhRL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vdhRL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Retrieve clusters:&lt;/strong&gt; (using our max distance determined from reading dendrogram)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.cluster.hierarchy import fcluster&#xA;&#xA;max_d = 50&#xA;clusters = fcluster(Z, max_d, criterion='distance')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Map cluster assignments back to original frame:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;def add_clusters_to_frame(or_data, clusters):&#xA;    or_frame = pd.DataFrame(data=or_data)&#xA;    or_frame_labelled = pd.concat([or_frame, pd.DataFrame(clusters)], axis=1)&#xA;    return(or_frame_labelled)&#xA;&#xA;df = add_clusters_to_frame(X, clusters)&#xA;df.columns = ['A', 'B', 'cluster']&#xA;&#xA;df.head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1cG0Z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1cG0Z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Build a classifier using this &quot;labelled&quot; data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, I'll just use the original data and the assigned clusters along with a knn classifier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.random.seed(42)&#xA;indices = np.random.permutation(len(X))&#xA;X_train = X[indices[:-10]]&#xA;y_train = clusters[indices[:-10]]&#xA;X_test  = X[indices[-10:]]&#xA;y_test  = clusters[indices[-10:]]&#xA;&#xA;# Create and fit a nearest-neighbor classifier&#xA;from sklearn.neighbors import KNeighborsClassifier&#xA;knn = KNeighborsClassifier()&#xA;knn.fit(X_train, y_train) &#xA;&#xA;res = knn.predict(X_test)&#xA;print(res)&#xA;print(y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;predicted labels&lt;/strong&gt;: [2 2 1 1 2 2 1 2 2 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;test labels&lt;/strong&gt;: [2 2 1 1 2 2 1 2 2 1] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As with any classifier, your incoming data needs to be in the same representation as your training data. As new data arrives you run it against the predict function provided by your classifier (here we use sci-kit learn's knn.predict). This effectively assign new data to the cluster it belongs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ongoing cluster validation would be required in the model monitoring step of the machine learning workflow. New data can change the distribution and results of your approach. BUT, this isn't unique to unsupervised as all machine learning approaches will suffer from this (all models eventually go stale). As argued by Jörn in the reference above, manual inspection typically trumps automated approaches, so regular visual/manual inspection of the flat clusters is recommended.  &lt;/p&gt;&#xA;" OwnerUserId="30359" LastEditorUserId="30359" LastEditDate="2018-01-02T18:46:36.113" LastActivityDate="2018-01-02T18:46:36.113" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26194" PostTypeId="2" ParentId="19747" CreationDate="2018-01-01T22:30:26.007" Score="1" Body="&lt;p&gt;I had the same issue, also trying to read a .csv  file. Although the answer from @minerals would be technically correct, it assumes that your .csv file is not formatted correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, the csv file had the correct number of indices, and was validated as not corrupted (using the freeware software 'CSVed'). I was able to load my dataframe by saving the csv as a .xlsx (Microsoft Excel). It loaded fine. Interestingly, exporting back to .csv recreated the error!&lt;/p&gt;&#xA;" OwnerUserId="26748" LastActivityDate="2018-01-01T22:30:26.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26196" PostTypeId="2" ParentId="26185" CreationDate="2018-01-02T05:23:57.320" Score="1" Body="&lt;p&gt;It is not clear from your question if you are trying to predict value for many used cars as a function of different vehicle characteristics (like age of car, model, mileage) or predicting price of a single car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming the former (prediction using variables), you can set it up as a regression problem, where price is dependent variable and (age, model, mileage) are independent variables. Google &quot;Boston housing price prediction&quot; which is a similar application in different domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have only prices data (ignoring vehicle characteristics), you are looking for summary statistics. You can start with the mean (average) or median price. You can then it little more complex by modelling the price as some reasonable statistical distribution (normal or Poisson) and then take typical summary statistics on the distribution as estimate (say inter-quantile range).&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2018-01-02T05:23:57.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26199" PostTypeId="2" ParentId="26063" CreationDate="2018-01-02T08:34:12.890" Score="2" Body="&lt;p&gt;Linear models like Logistic Regression and Support Vector Machines can also handle such feature dimensionalities. Often in text mining problems like text classification the dimensianality of the feature space equals the vocabulary size which is high.  &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-02T08:34:12.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26203" PostTypeId="1" CreationDate="2018-01-02T12:00:45.243" Score="2" ViewCount="782" Body="&lt;p&gt;Suppose I want to test a multi-arm bandit algorithm in the contextual setting on a set of historical data. For simplicity, let's assume there are only two arms A and B and suppose the rewards are binary. Furthermore, suppose I have a data set where users were shown one of the two arms and I have a record of the rewards. What would be the best approach to simulating the scenario of running the algorithm online? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of doing the following: If the algorithm outputs A then I want to record a reward so the algorithm can learn, to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to know if this approach is suitable and if anyone knows of any better ways to simulate online learning algorithms with historical data, in particular the multi-arm bandit problem?&lt;/p&gt;&#xA;" OwnerUserId="43865" LastEditorUserId="43865" LastEditDate="2018-01-03T14:07:09.647" LastActivityDate="2018-01-05T08:07:26.150" Title="Testing Multi-Arm Bandits on Historical Data" Tags="|machine-learning|reinforcement-learning|online-learning|simulation|randomized-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26204" PostTypeId="2" ParentId="26182" CreationDate="2018-01-02T13:48:16.877" Score="2" Body="&lt;p&gt;The probability that a vector $x$ is drawn from $p(x)$ in some region $R$ of a sample space is given by $ P = \int_{R} p(x')dx'$.  Given a set of N vectors drawn from the distribution; it should be obvious that the probability k of these N vectors fall in $R$ is given by $P(k) = \binom{N}{k} p^{k} (1-p)^{N-k}$.  From the properties of a binomial p.m.f the mean and variance of the ratio $\frac{k}{N}$ are ${E}[\frac{k}{N}] = P$ and ${var}[\frac{k}{N}] = \frac{P(1-P)}{N}$.  Therefore, as $N  \rightarrow \infty$ the distribution becomes more defined and the variance smaller.  Hence, we can expect a decent estimate of the probability P to be obtained from the mean fraction of points that fall within the region $R$.  Hence $P \cong \frac{k}{N}$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider if the region $R$ is small such that $p(x)$ does not vary considerably within it, then $\int_{R} p(x')dx' \cong p(x)V $.  Combining this result with the one above.  We see that $p(x) \cong \frac{k}{NV}$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's where the formula you found basically comes from.  Therefore if we want to improve $p(x)$ we should let V approach 0.  However, then $R$ would become so small that we would find no examples.  Thus we really only have two choices in practice.  We have to let V be large enough to find examples in $R$ or small enough such that p(x) is constant within $R$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The basic approaches include using KDE (parzen window) or kNN.  The KDE fixes V while kNN fixes k.  Either way, it can be shown that both methods converge to the true probability density as N increases providing that V shrinks with N and that k grows with N.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formulas used in the picture are just arbitrary examples that fulfill this requirement.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-02T13:48:16.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26206" PostTypeId="1" AcceptedAnswerId="26207" CreationDate="2018-01-02T15:21:36.827" Score="7" ViewCount="34835" Body="&lt;p&gt;I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;token_dict=&quot;wrist. overlapping. direction. receptacles. comprising. portion. adjacent. side. hand. receive. adapted. finger. comprising. thumb. ...............................&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By using k-means clustering, I clustered this data by using k=3. Now, I want to calculate the distance between each data point in a cluster to its respective cluster centroid. I have tried to calculate euclidean distance between each data point and centroid but somehow I am failed at it. My code is as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.decomposition import PCA&#xA;import matplotlib.pyplot as plt&#xA;from matplotlib import style&#xA;import numpy as np&#xA;style.use('ggplot')&#xA;token_dict = []&#xA;import glob   &#xA;path = 'E:\\Project\\*.txt'   &#xA;files=glob.glob(path)   &#xA;for file in files:     &#xA;    f=open(file, 'r')&#xA;    text = f.read()&#xA;    token_dict.append(text)&#xA;vectorizer = TfidfVectorizer(max_df=0.8, max_features=10000, min_df=2, use_idf=True)&#xA;&#xA;#print(X)&#xA;&#xA;km = KMeans(n_clusters=3)&#xA;#labels = km.fit_predict(vectorizer)&#xA;#print(labels)&#xA;X = vectorizer.fit_transform(token_dict).todense()&#xA;km.fit(X)&#xA;pca = PCA(n_components=2).fit(X)&#xA;data2D = pca.transform(X)&#xA;# =============================================================================&#xA;# cluster_0=np.where(X==0)&#xA;# print(cluster_0)&#xA;# &#xA;# X_cluster_0 = data2D[cluster_0]&#xA;# print (X_cluster_0)&#xA;# =============================================================================&#xA;&#xA;# =============================================================================&#xA;# def euclidean(X1, X2):&#xA;#     return(X1-X2)&#xA;# =============================================================================&#xA;&#xA;# =============================================================================&#xA;# distance = euclidean(X_cluster_0[0], km.cluster_centers_[0])&#xA;# print(distance)&#xA;# =============================================================================&#xA;# =============================================================================&#xA;# &#xA;# km.predict()&#xA;# =============================================================================&#xA;order_centroids = km.cluster_centers_&#xA;centers2D = pca.transform(order_centroids)&#xA;labels = km.labels_&#xA;colors = [&quot;y.&quot;, &quot;b.&quot;,&quot;g.&quot;]&#xA;for i in range(len(X)):&#xA;    plt.plot(data2D[i][0], data2D[i][1], colors[labels[i]], markersize=10)&#xA;&#xA;plt.scatter(centers2D[:, 0], centers2D[:, 1], marker='x', s=200, linewidths=3, c='r')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone see where I went wrong?&lt;/p&gt;&#xA;" OwnerUserId="43954" LastEditorUserId="29575" LastEditDate="2018-01-02T15:34:38.720" LastActivityDate="2018-11-14T09:55:38.310" Title="calculate distance between each data point of a cluster to their respective cluster centroids" Tags="|machine-learning|python|k-means|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26207" PostTypeId="2" ParentId="26206" CreationDate="2018-01-02T16:18:28.030" Score="6" Body="&lt;pre&gt;&lt;code&gt;def k_mean_distance(data, cx, cy, i_centroid, cluster_labels):&#xA;        distances = [np.sqrt((x-cx)**2+(y-cy)**2) for (x, y) in data[cluster_labels == i_centroid]]&#xA;        return distances&#xA;clusters=km.fit_predict(data2D)&#xA;centroids = km.cluster_centers_&#xA;&#xA;distances = []&#xA;for i, (cx, cy) in enumerate(centroids):&#xA;    mean_distance = k_mean_distance(data2D, cx, cy, i, clusters)&#xA;    distances.append(mean_distance)&#xA;&#xA;print(distances)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this function I have solved my problem&lt;/p&gt;&#xA;" OwnerUserId="43954" LastActivityDate="2018-01-02T16:18:28.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26209" PostTypeId="1" AcceptedAnswerId="26210" CreationDate="2018-01-02T10:58:46.863" Score="12" ViewCount="8171" Body="&lt;h1&gt;Details:&lt;/h1&gt;&#xA;&lt;p&gt;&lt;strong&gt;GPU&lt;/strong&gt;: GTX 1080&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Training&lt;/strong&gt;: ~1.1 Million images belonging to 10 classes&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Validation&lt;/strong&gt;: ~150 Thousand images belonging to 10 classes&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Time per Epoch&lt;/strong&gt;: ~10 hours&lt;/p&gt;&#xA;&lt;p&gt;I've setup CUDA, cuDNN and Tensorflow( Tensorflow GPU as well).&lt;/p&gt;&#xA;&lt;p&gt;I don't think my model is that complicated that is takes 10 hours per epoch. I even checked if my GPU was the problem but it wasn't.&lt;/p&gt;&#xA;&lt;p&gt;Is the training time due to the Fully connected layers?&lt;/p&gt;&#xA;&lt;h1&gt;My model:&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add()&#xA;model.add(Conv2D(64, (3, 3), padding=&amp;quot;same&amp;quot;, strides=2))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.25))&#xA;model.add(Conv2D(64, (3, 3), padding=&amp;quot;same&amp;quot;, strides=2))&#xA;model.add(Activation('relu'))&#xA;model.add(Dropout(0.25))&#xA;model.add(Conv2D(32, (3, 3)))&#xA;model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(3, 3), strides=2))&#xA;model.add(Flatten())&#xA;model.add(Dense(256))&#xA;model.add(Activation('relu'))&#xA;model.add(Dense(4096))&#xA;model.add(Activation('relu'))&#xA;model.add(Dense(10))&#xA;model.add(Activation('softmax'))&#xA;&#xA;model.summary()&#xA;&#xA;opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)&#xA;&#xA;model.compile(loss='categorical_crossentropy',&#xA;          optimizer=opt,&#xA;          metrics=['accuracy']&#xA;          )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Because there is a lot of data I used the ImageDataGenerator.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;gen = ImageDataGenerator(&#xA; horizontal_flip=True&#xA;)&#xA;&#xA;train_gen = gen.flow_from_directory(&#xA;        'train/',&#xA;        target_size=(512, 512),&#xA;        batch_size=5,&#xA;        class_mode=&amp;quot;categorical&amp;quot;&#xA;)&#xA;&#xA;valid_gen = gen.flow_from_directory(&#xA;        'validation/',&#xA;        target_size=(512, 512),&#xA;        batch_size=5,&#xA;        class_mode=&amp;quot;categorical&amp;quot;&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44057" OwnerDisplayName="Pradyumna Rahul" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-02T22:57:20.707" Title="Why is training take so long on my GPU?" Tags="|machine-learning|python|keras|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26210" PostTypeId="2" ParentId="26209" CreationDate="2018-01-02T14:17:34.207" Score="8" Body="&lt;p&gt;That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.&lt;/p&gt;&#xA;" OwnerUserId="44120" OwnerDisplayName="shimao" LastActivityDate="2018-01-02T14:53:12.937" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26211" PostTypeId="1" AcceptedAnswerId="26215" CreationDate="2018-01-02T18:53:43.053" Score="0" ViewCount="315" Body="&lt;p&gt;I am building a deep neural network to predict values 0 and 1 .My training data contains class labels 0 and 1 .Currently i am getting a cross validation loss around .25 . How good is the model performing?&lt;/p&gt;&#xA;" OwnerUserId="13515" LastActivityDate="2018-01-02T20:46:51.413" Title="MSE to predict Values 0 and 1" Tags="|neural-network|deep-learning|keras|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26212" PostTypeId="2" ParentId="26211" CreationDate="2018-01-02T19:00:13.803" Score="0" Body="&lt;p&gt;Why are you using MSE?  It's not completely impossible but the usual framework is done against the probabilities and usually for forecasts. You can read more about it by researching Brier scores.  However, it doesn't appear you're doing this and instead just minimize against the labels.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should use a more appropriate measure like cross entropy.   &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-02T19:00:13.803" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26213" PostTypeId="1" CreationDate="2018-01-02T19:52:49.527" Score="1" ViewCount="1351" Body="&lt;pre&gt;&lt;code&gt;feature   Definition    Key&#xA;survival    Survival    0 = No, 1 = Yes&#xA;pclass   Ticket class   1 = 1st, 2 = 2nd, 3 = 3rd&#xA;sex       Sex           M/F&#xA;Age     Age in years    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;feature notes given below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pclass: A proxy for socio-economic status (SES)&#xA;1st = Upper&#xA;2nd = Middle&#xA;3rd = Lower&#xA;&#xA;age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Suppose I have dataset same as mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to handle a case where I can have a unseen value for a class. For example : In our dataset we have 3 different value for pclass. But how to handle a case where pclass value may have 4rth class say &quot;elite&quot; which did not appear in training dataset but appeared in the test dataset&lt;/p&gt;&#xA;" OwnerUserId="37502" LastActivityDate="2018-03-05T14:39:40.917" Title="How to handle unseen class feature in test dataset" Tags="|machine-learning|neural-network|scikit-learn|dataset|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26214" PostTypeId="2" ParentId="26213" CreationDate="2018-01-02T20:11:51.613" Score="1" Body="&lt;p&gt;Depends on the model.  I would say most of the time the new level is dropped or mapped to a NA level.  If possible, you should try to do a stratified sample for your training set to ensure that you get every level you care about.   &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-02T20:11:51.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26215" PostTypeId="2" ParentId="26211" CreationDate="2018-01-02T20:38:32.343" Score="1" Body="&lt;p&gt;Predicting $0.5$ for all items in your case would also give you MSE of $0.25$. That is because independently of whether the true label is $0$ or $1$, the squared error for each example will be $0.5^2 = 0.25$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your model is performing badly under an MSE measure, when such a simple model that does not take input data into account can get the same score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is debatable whether MSE gives you a useful metric here.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-02T20:46:51.413" LastActivityDate="2018-01-02T20:46:51.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26216" PostTypeId="1" CreationDate="2018-01-02T20:56:27.400" Score="1" ViewCount="179" Body="&lt;p&gt;In experimental science there is the 'scientific method', in mathematics there is the 'proof.' What are the standards for knowledge production in data science? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've gathered that there a few different philosophies one can take; some of them are similar to the scientific method (introduce a falsifiable model and see if the data fits), others seem more hodge-podge (introduce some kind of learning machine and hope that the regularization prevents over fitting, and that cross validation catches it if not).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(There are some loose connections between those two, like how falsifiability fits into VC entropy and learning rates; this is explained in Vapnik's book on the Foundations of Statistical Learning.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, supposing that I have some data and I want to extract knowledge from it. What are the community standards for doing so? Where can I learn about the epistemological philosophy behind these standards, and their mathematical expression?&lt;/p&gt;&#xA;" OwnerUserId="44063" LastEditorUserId="13727" LastEditDate="2018-01-04T09:29:33.063" LastActivityDate="2018-01-04T09:29:33.063" Title="Is there a standard data science epistemology?" Tags="|machine-learning|data-mining|methodology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26217" PostTypeId="1" CreationDate="2018-01-02T21:14:02.690" Score="2" ViewCount="69" Body="&lt;p&gt;I feel like a lot of the resources on ML have been focused on algorithms/modeling whereas in practice (industry applications) what's arguably the harder part, IMO, is what's before and after modeling (collecting unbiased training data, error diagnosis, conduct appropriate A/B testing, explore/exploit, etc) What are some of the resources to learn these practical issues in ML/DS ? Is experimental design the right keyword?&lt;/p&gt;&#xA;" OwnerUserId="44064" LastActivityDate="2018-01-02T21:14:02.690" Title="What are some of the resources to learn practical issues in machine learning and data science?" Tags="|machine-learning|statistics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26218" PostTypeId="2" ParentId="26209" CreationDate="2018-01-02T22:57:20.707" Score="3" Body="&lt;p&gt;As shimao said, that's about what you'd expect. Despite not having many layers, an input size of 512x512 is a large image to be convolving over. The large computation time is likely more due to convolving 64 filters over the large image, instead of the fully connected layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network you've put together has a funny information bottleneck in it though. You start off with 64 filters on the original sized image, only decreasing as your image size reduces. As the image passes through your network, the features you're learning become more and more abstracted and complex. Your Conv2D(32, (3, 3)) layer essentially limits the network to learning a 128x128 map of 32 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most network architectures double the number of features every time they pool, and most recent imagenet architectures actually ditch the fully connected layers in favour of an average pool over the final feature map, and basically perform logistic regression on the output of that pool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try starting off with fewer filters, say 16 in your first convolution layer, doubling each time you stride or pool. Do this a few more times than you are, to increase receptive field and decrease feature map size. Do this down to 64x64 or 32x32, which would be 128 or 256 filters. You can use Keras' Global Avg or Max pooling to eliminate fully connected layers as well. That should about double the speed of the network, and I would expect an increase in accuracy at the same time.&lt;/p&gt;&#xA;" OwnerUserId="39845" LastActivityDate="2018-01-02T22:57:20.707" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26219" PostTypeId="1" CreationDate="2018-01-03T02:41:12.997" Score="1" ViewCount="1018" Body="&lt;p&gt;I am using ubuntu 16.04.&#xA;after anacoda installation sypder not open.&#xA;Anaconda version: 4.4.6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Anaconda environment&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_root                  /home/shiva/.conda/envs/my_root&#xA;py27                     /home/shiva/.conda/envs/py27&#xA;py35                     /home/shiva/.conda/envs/py35&#xA;base                  *  /home/shiva/anaconda&#xA;myenv                    /home/shiva/anaconda/envs/myenv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I get the following error&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/spyder.py&quot;, line 2331, in main&#xA;    mainwindow = run_spyder(app, options, args)&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/spyder.py&quot;, line 2215, in run_spyder&#xA;    main.setup()&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/spyder.py&quot;, line 752, in setup&#xA;    from spyderlib.plugins.inspector import ObjectInspector&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/plugins/inspector.py&quot;, line 41, in &amp;lt;module&amp;gt;&#xA;    from spyderlib.widgets.ipython import IPythonControlWidget&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/widgets/ipython.py&quot;, line 28, in &amp;lt;module&amp;gt;&#xA;    pygments_patch.apply()&#xA;  File &quot;/usr/lib/python3/dist-packages/spyderlib/pygments_patch.py&quot;, line 22, in apply&#xA;    from pygments.lexer import _TokenType, Text, Error&#xA;  File &quot;/usr/lib/python3/dist-packages/pygments/lexer.py&quot;, line 19, in &amp;lt;module&amp;gt;&#xA;    from pygments.filters import get_filter_by_name&#xA;  File &quot;/usr/lib/python3/dist-packages/pygments/filters/__init__.py&quot;, line 20, in &amp;lt;module&amp;gt;&#xA;    from pygments.plugin import find_plugin_filters&#xA;  File &quot;/usr/lib/python3/dist-packages/pygments/plugin.py&quot;, line 39, in &amp;lt;module&amp;gt;&#xA;    import pkg_resources&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2927, in &amp;lt;module&amp;gt;&#xA;    @_call_aside&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2913, in _call_aside&#xA;    f(*args, **kwargs)&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2952, in _initialize_master_working_set&#xA;    add_activation_listener(lambda dist: dist.activate())&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 956, in subscribe&#xA;    callback(dist)&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2952, in &amp;lt;lambda&amp;gt;&#xA;    add_activation_listener(lambda dist: dist.activate())&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2515, in activate&#xA;    declare_namespace(pkg)&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2097, in declare_namespace&#xA;    _handle_ns(packageName, path_item)&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2047, in _handle_ns&#xA;    _rebuild_mod_path(path, packageName, module)&#xA;  File &quot;/usr/lib/python3/dist-packages/pkg_resources/__init__.py&quot;, line 2066, in _rebuild_mod_path&#xA;    orig_path.sort(key=position_in_sys_path)&#xA;AttributeError: '_NamespacePath' object has no attribute 'sort'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how can I solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="40102" LastEditorUserId="29575" LastEditDate="2018-01-03T02:49:54.980" LastActivityDate="2018-01-03T02:52:29.630" Title="After anaconda installation spyder not working" Tags="|python|" AnswerCount="1" CommentCount="1" ClosedDate="2018-01-05T05:43:46.097" ContentLicense="CC BY-SA 3.0" />
  <row Id="26220" PostTypeId="2" ParentId="26219" CreationDate="2018-01-03T02:52:29.630" Score="2" Body="&lt;p&gt;According to this &lt;a href=&quot;https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2990&quot; rel=&quot;nofollow noreferrer&quot;&gt;bug report&lt;/a&gt;, you can solve this with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install --upgrade pip setuptools&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2018-01-03T02:52:29.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26222" PostTypeId="1" AcceptedAnswerId="28678" CreationDate="2018-01-03T05:24:12.223" Score="2" ViewCount="785" Body="&lt;p&gt;Number of labels: 1000, Dataset size: 200000 images&#xA;Final probability for 1000 labels is in the range of 0.3 to 0.34.&#xA;I was expecting large variation in probabilities. Can someone tell me what I am doing wrong. I am following &lt;a href=&quot;https://towardsdatascience.com/multi-label-image-classification-with-inception-net-cbb2ee538e30&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; tutorial&lt;/p&gt;&#xA;" OwnerUserId="37760" LastEditorUserId="37760" LastEditDate="2018-01-03T08:12:14.347" LastActivityDate="2018-03-06T02:40:57.800" Title="Transfer learning (on pre-trained inception net model) for multi label classification is giving similar probability for all labels" Tags="|deep-learning|multilabel-classification|inception|transfer-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26223" PostTypeId="1" AcceptedAnswerId="26344" CreationDate="2018-01-03T06:46:58.643" Score="2" ViewCount="4461" Body="&lt;p&gt;I am implementing the linear regression model from scratch.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#!/usr/bin/env python2&#xA;# -*- coding: utf-8 -*-&#xA;&quot;&quot;&quot;&#xA;Created on Thu Nov 16 14:40:53 2017&#xA;&#xA;@author: user&#xA;&quot;&quot;&quot;&#xA;&#xA;import os&#xA;import random&#xA;os.chdir('/home/user/Desktop/andrewng/machine-learning-ex1/ex1')&#xA;&#xA;import pandas as pd&#xA;data = pd.read_csv('/home/user/Desktop/andrewng/machine-learning-ex1/ex1/ex1data1.txt',header=None)&#xA;&#xA;theta_0 = random.random() &#xA;theta_1 = random.random() &#xA;alpha = 0.001&#xA;&#xA;print(len(data))&#xA;&#xA;&#xA;hist = -90&#xA;cost = 0&#xA;print('theta_0    + theta_1    ')&#xA;while(cost-hist&amp;gt;0.001):&#xA;    hist = cost&#xA;    cost = 0&#xA;    a = 0&#xA;    b = 0&#xA;    for i in range(len(data)):&#xA;        k = data.iloc[i]&#xA;        a = a +  theta_0 + theta_1*k[0] - k[1]&#xA;        b = b + (theta_0 + theta_1*k[0] - k[1])*k[0]&#xA;    theta_0 = theta_0 - alpha*a/len(data)&#xA;    theta_1 = theta_1 - alpha*b/len(data)&#xA;    #print(str(theta_0)+'    '+str(theta_1))&#xA;    for j in range(len(data)):&#xA;        k = data.iloc[i]&#xA;        cost  = cost + (theta_0 + theta_1*k[0] - k[1])**2&#xA;    cost = cost/(2*len(data))&#xA;    print(cost)&#xA;    #if(cost&amp;gt;hist):&#xA;    #    print(str(theta_0)+'    '+str(theta_1))&#xA;    #    break&#xA;print(str(theta_0)+'    '+str(theta_1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to the theory the cost should decrease at each iteration but for me the cost keeps increasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://gist.github.com/anonymous/160a8934ec0ce6fa1170569d208c0269&quot; rel=&quot;nofollow noreferrer&quot;&gt;DATA&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="13330" LastEditorUserId="29575" LastEditDate="2018-01-03T14:19:37.380" LastActivityDate="2020-03-28T16:26:48.247" Title="Why is the cost increasing in the linear regression method?" Tags="|machine-learning|python|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26225" PostTypeId="2" ParentId="20131" CreationDate="2018-01-03T07:43:14.203" Score="1" Body="&lt;h3&gt;In training set:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You would better use some methods to &quot;presume&quot; certain values...average / pro-rata......which you could find in various learning sources&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Leaving them as blank (not sure of Azure but in a low level programming platform) would cause either inaccuracy or problems.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;For testing set:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You are not changing the Learned Parameters, but still that would probably causing some inaccurate measurement but Azure ML may have done something internally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference from &lt;a href=&quot;https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/prepare-data&quot; rel=&quot;nofollow noreferrer&quot;&gt;MSDN&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h3&gt;How to deal with missing values?&lt;/h3&gt;&#xA;  &#xA;  &lt;p&gt;To deal with missing values, it is best to first identify the reason for the missing values to better handle the problem. Typical missing value handling methods are:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Deletion: Remove records with missing values&lt;/li&gt;&#xA;  &lt;li&gt;Dummy substitution: Replace missing values with a dummy value: e.g, unknown for categorical or 0 for numerical values.&lt;/li&gt;&#xA;  &lt;li&gt;Mean substitution: If the missing data is numerical, replace the missing values with the mean.&lt;/li&gt;&#xA;  &lt;li&gt;Frequent substitution: If the missing data is categorical, replace the missing values with the most frequent item&lt;/li&gt;&#xA;  &lt;li&gt;Regression substitution: Use a regression method to replace missing values with regressed values.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43996" LastEditorUserId="29575" LastEditDate="2018-01-03T14:05:27.763" LastActivityDate="2018-01-03T14:05:27.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26228" PostTypeId="2" ParentId="26174" CreationDate="2018-01-03T09:38:19.970" Score="1" Body="&lt;p&gt;First and foremost you need to know the difference between the type of data you are trying to predict. The two general categories are discrete and continuous. Most people tend to miss out that classification is at its core a discrete &quot;regression&quot;. The values are predicted for discrete variables by considering a decision-based approach which ultimately leads to finding or predicting the variable in question. You can call this classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For continuous variables, however, since the values can have a certain range it most closely mirrors a curve fit with the addition of an error boundary. I think this should clarify what you're essentially trying to ask. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2018-01-03T09:38:19.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26229" PostTypeId="1" CreationDate="2018-01-03T10:28:22.260" Score="1" ViewCount="386" Body="&lt;p&gt;I have some questions about Custom Writable objects in Hadoop. First of all, the private instance variables of the Writable object have to be Writable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does Hadoop handle &lt;code&gt;TreeSet&amp;lt;IntWritable&amp;gt;&lt;/code&gt; differently from &lt;code&gt;ArrayWritable&amp;lt;IntWritable&amp;gt;&lt;/code&gt; in &lt;code&gt;org.apache.giraph.utils&lt;/code&gt; (Class ArrayWritable&amp;lt;T extends org.apache.hadoop.io.Writable)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following variables: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;private TreeSet&amp;lt;IntWritable&amp;gt; vertexList; &#xA;private IntWritable shortestPathCounter;&#xA;private IntWritable totalShortestPathCounter;&#xA;private DoubleWritable bCentrality;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the following method for serializing. Are write and readFields method correctly handled this way?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public void readFields(DataInput in) throws IOException {&#xA;    for(IntWritable i: vertexList)&#xA;        i.readFields(in);&#xA;    shortestPathCounter.readFields(in);&#xA;    totalShortestPathCounter.readFields(in);&#xA;    bCentrality.readFields(in);&#xA;}&#xA;&#xA;public void write(DataOutput out) throws IOException {&#xA;    for(IntWritable i: vertexList)&#xA;        i.write(out);&#xA;    shortestPathCounter.write(out);&#xA;    totalShortestPathCounter.write(out);&#xA;    bCentrality.write(out);&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44088" LastEditorUserId="29575" LastEditDate="2018-01-03T14:30:07.510" LastActivityDate="2018-01-03T14:30:07.510" Title="Custom Writable Serialization in Hadoop" Tags="|apache-hadoop|java|" AnswerCount="0" CommentCount="0" ClosedDate="2018-08-05T17:51:04.750" ContentLicense="CC BY-SA 3.0" />
  <row Id="26230" PostTypeId="1" CreationDate="2018-01-03T10:31:59.400" Score="1" ViewCount="91" Body="&lt;p&gt;I have a text-classification problem with a lot of training data. Running cross-validation takes a lot of time - several days or even weeks. In order to make the system more responsive, I am thinking of the following scheme:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Train a network on instances 1,...,100.&lt;/li&gt;&#xA;&lt;li&gt;Test it on instances 101,...200; output the accuracy.&lt;/li&gt;&#xA;&lt;li&gt;Train the existing network on instances 101,...200; &lt;/li&gt;&#xA;&lt;li&gt;Test it on instances 201,...300; output the accuracy.&lt;/li&gt;&#xA;&lt;li&gt;And so on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ideally, I would like that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training on each additional 100 instances will take a constant time (i.e, I will not have to re-train the network on all previous instances);&lt;/li&gt;&#xA;&lt;li&gt;The trained network will have the combined &quot;wisdom&quot; of all previous instances (so its accuracy will tend to improve with time).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is this possible to do with standard deep-learning tools (e.g. dynet)?&lt;/p&gt;&#xA;" OwnerUserId="33051" LastActivityDate="2018-02-02T15:23:05.900" Title="A neural network that keeps learning as new training data becomes available" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26232" PostTypeId="2" ParentId="26216" CreationDate="2018-01-03T12:26:41.343" Score="1" Body="&lt;p&gt;Interesting question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I see it, the justification comes from both mathematics and experimental science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you suggest a new algorithm, you should state your assumptions and provide a proof, just like in math. However, you should also demonstrate it on real data, showing that your assumptions are reasonable and that your algorithm can deliver value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you suggest a model, you should justify it based on its performance on the observations, just like in experimental sciences. This is true no matter how did you derive the model. However, examining the model logic itself and the way it was derived will help to gain confidence in it and trace where it needs modification.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2018-01-03T12:26:41.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26233" PostTypeId="2" ParentId="26230" CreationDate="2018-01-03T12:35:29.573" Score="1" Body="&lt;p&gt;You can achieve that by training the network using batches of 100 examples. In this case, you train on a batch and update the neural network parameters, you predict on the second batch to estimate the accuracy and then you use the examples of this second batch to continue training the network and so on and so forth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Batch training (a form of online learning) is implemented on every framework, see for instances &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-use-keras-with-datasets-that-dont-fit-in-memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for Keras.&lt;/p&gt;&#xA;" OwnerUserId="24215" LastEditorUserId="29575" LastEditDate="2018-01-03T15:01:15.473" LastActivityDate="2018-01-03T15:01:15.473" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26235" PostTypeId="1" CreationDate="2018-01-03T15:01:33.540" Score="2" ViewCount="413" Body="&lt;p&gt;Often clustering algorithms only output a bunch of class labels, and do not provide any sort of interpretation of the classes formed by the algorithm. It seems to me not entirely unreasonable to attempt to get some sort of interpretation by using the class labels provided by the clustering algorithm as the target in a supervised classification problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a concrete example, say you cluster using k-means, and use a single decision tree classifier to predict the cluster based off of other features. The decision tree then should be able to give you some way of interpreting the clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I can't find any literature (or blog posts...) mentioning this as a technique for interpreting the results of a clustering algorithm, which leads me to believe it is problematic. So in short:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;b&gt;Question:&lt;/b&gt; Can supervised classification algorithms be used to interpret the reuslts of unsupervised clustering algorithms? If not, why?&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="44104" LastActivityDate="2020-10-10T12:34:59.247" Title="Using Classification Algorithms to Interpret Clustering Reuslts" Tags="|machine-learning|classification|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26236" PostTypeId="1" AcceptedAnswerId="26260" CreationDate="2018-01-03T15:20:29.057" Score="0" ViewCount="317" Body="&lt;p&gt;I want to apply unsupervised clustering on a set of short texts, which I need to divide into 2 clusters. &#xA;Also I know that one of my clusters is likely to contain some words (non-exhaustive list) and I would like to use them for driving my machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not really find any simple way to do it (from any algorithm, library, tool...). Do you know one or have any solution to help me doing this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about applying some kind of k-means on my DTM matrix, using a specific metric e.g the euclidian distance between 2 &#xA;texts which I would &quot;artificially&quot; reduce when the 2 texts words match my primary list of words which are likely to appear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not test it yet and I was wondering if this solution could be relevant or not.&lt;/p&gt;&#xA;" OwnerUserId="43308" LastEditorUserId="29575" LastEditDate="2018-01-03T17:31:16.503" LastActivityDate="2018-01-04T10:15:54.477" Title="Unsupervised text clustering using a driving list" Tags="|clustering|text-mining|unsupervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26237" PostTypeId="2" ParentId="26203" CreationDate="2018-01-03T15:29:15.677" Score="2" Body="&lt;p&gt;The problem you have is that the users were originally shown A or B under a different policy to that which the optimiser is learning. You probably don't know the probabilities for A or B from that policy (if you do, that would allow for a bit more subtlety). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to reject samples that do not match what the optimiser does at any point. This may leave you with far less historical data to mine, and the possibility of getting a biased result.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I understand this correctly, you are planning to: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pick an event (including user data) for input context&lt;/li&gt;&#xA;&lt;li&gt;Generate the choice of A or B using your learning agent&lt;/li&gt;&#xA;&lt;li&gt;Find another user that was presented the same output in the historical data, by sampling randomly from &quot;all users shown A&quot; or &quot;all users shown B&quot;&lt;/li&gt;&#xA;&lt;li&gt;Count the reward generated in the sampled historical record&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sadly this will not work, because in step 3 you divorce the context from the agent's decision. So you will not be measuring the agent's likely reward, but the hybrid reward of an agent that uses a different user population and gains rewards according to the historic policies (which you may not even know).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A less biased approach might be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pick an event (including user data) randomly from historic data&lt;/li&gt;&#xA;&lt;li&gt;Generate the choice of A or B using your learning agent&lt;/li&gt;&#xA;&lt;li&gt;If the agent's choice and historic choice for that event match, then count the reward, else discard the event from the test&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You may get sampling bias issues here if the agent's policy is different to historic policy for large number of users. The problem is that you are systematically removing users where old policies said one thing and new policies another, due to lack of data about the reward. So unless the original policy was completely random, the samples you actually train/evaluate on will not be chosen with correct ratios across the user population as it varies. Importance sampling could help mitigate that, but only if you know the probabilities of making A|B selection in both historic and learning agent policies.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Another possibility is to train a supervised model that predicts reward given user data and action from your historic data, and use that to drive a &lt;em&gt;simulator&lt;/em&gt; of the online environment. If you are also &lt;em&gt;training&lt;/em&gt; your agent during this test, you may need to simulate variance - e.g. turn probability of a click-through event into either a click or no click event by sampling $x &amp;lt; p(click)$ - because that is the data you need your online algorithm to learn from. This is clearly limited by the accuracy of the supervised model from the training data, but could reduce variance, because you will get some kind of indication of reward for events that did not happen historically (with some bounds on accuracy that you can estimate). That is due to function approximation in the supervised model - it is pretty similar idea to bootstrapping the Critic evaluation in an Actor-Critic reinforcement learning model.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-05T08:07:26.150" LastActivityDate="2018-01-05T08:07:26.150" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="26239" PostTypeId="1" AcceptedAnswerId="26246" CreationDate="2018-01-03T18:36:15.723" Score="3" ViewCount="1068" Body="&lt;p&gt;I recently read the LIPO blog post on the dlib blog: &#xA;&lt;a href=&quot;http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It mentions that it can be used for optimizing hyperparameters of eg metaheuristic algorithmsike simulated annealing or genetic algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked for info on how optimizing hyperparameters work in general and the Wikipedia page is the most informative I found but it doesn't answer my basic questions:&#xA;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Hyperparameter_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Hyperparameter_optimization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is just: what is the basic idea for optimizing hyperparameters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have some problem I'm trying to solve with simulated annealing, I know that the starting temperature and the cooldown rate are important in determining how well the algorithm does at finding a solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that I could completely run the algorithm with one set of parameters, modify one of the parameters, completely run it again, then reset the parameters and modify the other parameter and run it again. This could give me a numerical gradient that I could use to modify the parameters via gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However... At this point I had to run the whole algorithm 3 times just to get a single modification of the hyperparameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like I must be missing something obvious because optimizing the hyperparameters would take many many hundreds or thousands of times or more the cost of running the whole thing once, which doesn't seem useful at all. Can someone clue me in?&lt;/p&gt;&#xA;" OwnerUserId="27729" LastActivityDate="2018-01-03T21:32:18.993" Title="Basic method of optimizing hyperparameters" Tags="|hyperparameter|hyperparameter-tuning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26243" PostTypeId="2" ParentId="26235" CreationDate="2018-01-03T20:37:34.990" Score="2" Body="&lt;p&gt;There's nothing wrong with this idea and although I don't have literature on hand, I'm fairly confident I've seen this sort of thing done. I disagree that clustering algorithms often don't provide interpretation though. There are definitely plenty that don't, but I'm not sure k-means is one of them. The centroids of your clusters should provide you with the interpretability you're looking for. Passing the results of k-means into a decision tree is probably just going to exchange the centroid for left and right bounds for your features (although it might actually be interesting if the tree ignores particular dimensions in the decision process). Generative models like GMM and LDA also give a lot of useful information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding literature, although I don't think I've seen this specifically applied to clustering, there's definitely a fair amount of on-going research into techniques for adding interpretability to &quot;black-box models&quot;. Consider for example this article: &lt;a href=&quot;https://arxiv.org/abs/1707.01154&quot; rel=&quot;nofollow noreferrer&quot;&gt;Interpretable &amp;amp; Explorable Approximations of Black Box Models&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We propose Black Box Explanations through Transparent Approximations (BETA), a novel model agnostic framework for explaining the behavior of any black-box classifier by simultaneously optimizing for fidelity to the original model and interpretability of the explanation. To this end, we develop a novel objective function which allows us to learn (with optimality guarantees), a small number of compact decision sets each of which explains the behavior of the black box model in unambiguous, well-defined regions of feature space. Furthermore, our framework also is capable of accepting user input when generating these approximations, thus allowing users to interactively explore how the black-box model behaves in different subspaces that are of interest to the user. To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences. Experimental evaluation with real-world datasets and user studies demonstrates that our approach can generate highly compact, easy-to-understand, yet accurate approximations of various kinds of predictive models compared to state-of-the-art baselines.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Poke around for &quot;black box model interpretability&quot; and you'll find a ton more recent research.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-03T20:37:34.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26244" PostTypeId="1" CreationDate="2018-01-03T20:50:30.813" Score="0" ViewCount="30" Body="&lt;p&gt;Is it possible to utilize/create a larger data set with like data in order to be able to improve the Ml model optimization and/or ML model feature options that you can utilize on your subset of data? If so what should one take into account when building this data? i.e. when sourcing additional data is there mistakes to avoid and/or how many multiples of your subset data should you ideally add?&lt;/p&gt;&#xA;" OwnerUserId="43962" LastActivityDate="2018-02-02T23:23:44.303" Title="Will a larger data set in order to support your subset?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26245" PostTypeId="1" AcceptedAnswerId="26249" CreationDate="2018-01-03T21:04:04.997" Score="1" ViewCount="377" Body="&lt;p&gt;My data is a list of sentences, where each sentence contains between 1 and 4 words. These sentences are typed in manually so some of them contain typos and some additional words such as &lt;code&gt;GmbH&lt;/code&gt;, &lt;code&gt;GER&lt;/code&gt; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I &lt;em&gt;do know&lt;/em&gt; the set of valid sentences. As an example we assume this valid set is given by &lt;code&gt;{Hello human, Horse, Hello bird}&lt;/code&gt; and the data (where some sentences contain typos and extra words) is given by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Hello human&#xA;Horse&#xA;Hello human GmbH&#xA;Hello human GmbH, GER&#xA;Horse GmbH&#xA;Horse&#xA;Hello humn&#xA;Hell humn&#xA;Hello human&#xA;Hello bird&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to give each sentence above an ID 1, 2 or 3 where 1 is for &lt;code&gt;Hello world&lt;/code&gt;, 2 is &lt;code&gt;Horse&lt;/code&gt; and 3 is &lt;code&gt;Hello bird&lt;/code&gt;. But due to the typos and extra words such as &lt;code&gt;GmbH&lt;/code&gt;, &lt;code&gt;GER&lt;/code&gt; I cannot make a simple comparison between strings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a numerical technique within NLP or a related field that I can use to achieve this task?&lt;/p&gt;&#xA;" OwnerUserId="44117" LastEditorUserId="44117" LastEditDate="2018-01-03T21:13:45.443" LastActivityDate="2018-01-03T23:19:20.590" Title="Classify sentences containing typos into groups" Tags="|python|r|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26246" PostTypeId="2" ParentId="26239" CreationDate="2018-01-03T21:07:42.580" Score="2" Body="&lt;p&gt;Hyperparameter optimization follows the same rules as model selection. Each set of hyperparameters effectively represents a different model you are considering, so the data you use to fit the model with some set of hyperparameters needs to be different from the data you use to evaluate which set of hyperparameters you want to ultimately use. A common approach for evaluating hyperparameters is nested cross-validation. This basically means that you need to treat hyperparameter selection as part of your model training process, and when you evaluate your model you evaluate the entire process front to back, i.e. treating hyperparameter tuning as a component of model training with respect to cross-validating over your training process. There's an excellent discussion of this in &lt;a href=&quot;https://web.stanford.edu/~hastie/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;section 7.10.2 (&quot;The Wrong and Right Way&#xA;to Do Cross-validation&quot;) of Elements of Statistical Learning&lt;/a&gt;, which you can read online and/or download for free. The general idea is that if you're not careful, you can actually &lt;a href=&quot;http://www.andrewng.org/portfolio/preventing-overfitting-of-cross-validation-data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;overfit to your evaluation data&lt;/a&gt;. Play with &lt;a href=&quot;https://web.stanford.edu/class/stats202/content/lab11&quot; rel=&quot;nofollow noreferrer&quot;&gt;this demo&lt;/a&gt; to see for yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But yes, your intuition is correct. Hyperparameter tuning is often very computationally expensive. One way people sometimes try to minimize this cost is by limiting the search space of feasible parameters to a small discrete set, e.g. &lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;grid search&lt;/a&gt;. Another approach is to use guassian processes or KDEs to &lt;a href=&quot;https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;approximate the cost surface&lt;/a&gt; in parameter space. You can even use a &lt;a href=&quot;https://arxiv.org/abs/1603.06560&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-armed bandit&lt;/a&gt; approach. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frankly, you can really use pretty much any non-linear optimization technique for hyperparameter tuning, as long as you follow the appropriate cross-validation rules. The trick is figuring out bang-for-your-buck in terms of how much time/effort/compute you are willing to expend exploring parameters vs. the potential improvement to your model. Additionally, there are concerns about &lt;a href=&quot;https://research.googleblog.com/2015/08/the-reusable-holdout-preserving.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the limits of repeated cross-validation/holdout evaluations&lt;/a&gt;, but that's a whole other rabbit hole.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-03T21:32:18.993" LastActivityDate="2018-01-03T21:32:18.993" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26247" PostTypeId="1" CreationDate="2018-01-03T21:28:51.467" Score="0" ViewCount="3780" Body="&lt;p&gt;I have a binary classification problem. I get the following results: a &lt;code&gt;val_loss&lt;/code&gt; (far) lower than the &lt;code&gt;train_loss&lt;/code&gt;, but the &lt;code&gt;accuracy&lt;/code&gt; is also lower for the validation compared to the training set. How is that possible ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/10&#xA;10708/10708 [=======] - loss: 0.6356 - acc: 0.8289 - val_loss: 0.4981 - val_acc: 0.7760&#xA;Epoch 2/10&#xA;10708/10708 [=======] - loss: 0.6243 - acc: 0.8248 - val_loss: 0.5075 - val_acc: 0.7609&#xA;Epoch 3/10&#xA;10708/10708 [=======] - loss: 0.6204 - acc: 0.8302 - val_loss: 0.5152 - val_acc: 0.7694&#xA;Epoch 4/10&#xA;10708/10708 [=======] - loss: 0.6215 - acc: 0.8307 - val_loss: 0.4981 - val_acc: 0.7824&#xA;Epoch 5/10&#xA;10708/10708 [=======] - loss: 0.6180 - acc: 0.8318 - val_loss: 0.4942 - val_acc: 0.7848&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the Keras model I'm using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(LSTM(16, input_shape=(1, 1)))&#xA;model.add(Dense(2, activation='softmax'))&#xA;&#xA;adam = Adam(lr=0.001)&#xA;model.compile(loss='categorical_crossentropy',&#xA;          optimizer=adam,&#xA;          metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42409" LastActivityDate="2018-04-04T13:06:37.307" Title="The validation loss &lt; training loss and validation accuracy &lt; training accuracy" Tags="|keras|accuracy|loss-function|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26248" PostTypeId="2" ParentId="26244" CreationDate="2018-01-03T21:30:31.607" Score="1" Body="&lt;p&gt;This is a common approach to address class-imbalance. You'll generally see it referred to as &quot;upsampling&quot; or &quot;over-sampling&quot;. In addition to simply reusing existing data, you can &quot;augment&quot; your data by transforming it or adding noise, or generate synthetic data. A good demonstration of &lt;a href=&quot;http://benanne.github.io/2014/04/05/galaxy-zoo.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;data augmentation&lt;/a&gt; is the winning solution to the kaggle Galaxy Zoo contest. A popular technique for generating new data is &lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt;. Keep in mind: if you change the class proportions in your dataset, your effectively changing the prior for those classes. Depending on your choice of algorithm and what you are trying to accomplish, you may want/need to apply a &lt;a href=&quot;http://www.data-mining-blog.com/tips-and-tutorials/overrepresentation-oversampling/&quot; rel=&quot;nofollow noreferrer&quot;&gt;probability adjustment&lt;/a&gt; to the model's output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another technique you can use is to just pick a different decision threshold. For example, if you're performing a logistic regression and don't care about performing inference on parameters, duplicating the data for one class won't actually change the slope of the decision boundary, it will just shift its location. You could equivalently use the decision boundary learned on the unmodified data and then pick a cutoff other than $P(X=1) \ge 0.5$. A good heuristic is to use the threshold which maximizes &lt;a href=&quot;https://en.wikipedia.org/wiki/Youden%27s_J_statistic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Youden's J&lt;/a&gt;, which can be trivially calculated from a ROC curve. If you have a specific model evaluation metric you are concerned with, choose your threshold relative to that.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-03T21:30:31.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26249" PostTypeId="2" ParentId="26245" CreationDate="2018-01-03T21:48:08.523" Score="2" Body="&lt;p&gt;If I understand correctly, you're looking for string similarity. There are several techniques available, the most simple is &quot;edit distance&quot; (aka levenshtein distance), which is the count of the minimum insertion/deletion/substitution/transposition operations needed to get from one string to the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your particular task, I suspect &quot;jaro-winkler similarity&quot; would be better. JW is similar to ED, but was specifically designed for &quot;entity resolution&quot; (i.e. &quot;record linkage&quot;), which it looks like is what you're trying to accomplish. You can see a short demonstration of how this would work &lt;a href=&quot;https://stackoverflow.com/a/16840661/819544&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="29575" LastEditDate="2018-01-03T23:19:20.590" LastActivityDate="2018-01-03T23:19:20.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26251" PostTypeId="2" ParentId="23549" CreationDate="2018-01-03T22:41:52.523" Score="2" Body="&lt;p&gt;I can't find any literature discussing setting a fixed learning rate differently for the bias specifically than for the other weights, but if you asked the researcher responsible for that model their answer would probably be something like:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;The network wasn't training properly and I noticed the error for the bias term wasn't converging as fast as I'd like. When I tried doubling the learning rate for just that term it seemed to fix the problem. I dunno man, it just worked.&quot; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As far as I can tell, training neural networks is still more art than science and the amount of skill, patience, and care taken in training a neural network (depending on the architecture) can have as much impact on the model's performance as the network topology. Consider issues like mode collapse in GANs, vanishing gradient with ReLUs, or just getting stuck in local optima. Neural networks often need to be &lt;a href=&quot;http://cs231n.github.io/neural-networks-3/#baby&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;babysat&quot;&lt;/a&gt; during training, and consequently trial and error may contribute significantly to certain training process decisions. I'm sorry if that's not a satisfactory answer, but I have a strong suspicion that's what's going on here. If you post some specific examples, it might be easier to figure out concretely what's going on, but otherwise my guess is that this was just the result of trial and error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding using parameter-specific learning rates generally, &lt;a href=&quot;http://cs231n.github.io/neural-networks-3/#ada&quot; rel=&quot;nofollow noreferrer&quot;&gt;most modern optimizers do this&lt;/a&gt;, including AdaGrad, RMSProp and Adam.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-03T22:41:52.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26252" PostTypeId="2" ParentId="26247" CreationDate="2018-01-03T22:53:47.783" Score="2" Body="&lt;p&gt;How did you split your training and test sets? My guess is that the minority class is under-represented in your test set. Consider the case where the minority class has frequencies of 25% and 5% in the training and test sets, respectively. If my model predicts everything is the majority class, the recall for the minority class is 0% on both sets but we get 75% training accuracy and 95% validation accuracy. Now let's say we train the model a bit more and  achieve 50% recall while still correctly labeling all of the majority class samples: now we're up to 87% training and 97% accuracy. Because the training and test sets exhibit significantly different class frequencies, the test set will basically always have a higher accuracy than the training set (up to the point where you overfit, at which point the validation accuracy could be be lower than training accuracy but not necessarily). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution here is to either use stratified sampling to make sure your classes have equal representation between your different datasets, or to monitor other evaluation metrics like precision and recall. Plotting the taining and validation accuracies could also be helpful, since it should be visually pretty obvious if your model starts to overfit, regardless of class imbalance between your training and test sets.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-03T22:53:47.783" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26253" PostTypeId="1" CreationDate="2018-01-03T23:14:08.250" Score="2" ViewCount="180" Body="&lt;p&gt;I am looking for a dataset containing a large number of NLP research papers and abstracts. Are there any open access datasets like that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, can you please share the details?&lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2024-03-28T11:21:09.720" Title="Datasets in NLP research papers" Tags="|data-mining|nlp|dataset|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26254" PostTypeId="1" AcceptedAnswerId="26257" CreationDate="2018-01-03T23:52:32.980" Score="1" ViewCount="58" Body="&lt;p&gt;When having a &lt;strong&gt;single vector&lt;/strong&gt; you use an &lt;strong&gt;MLP&lt;/strong&gt; neural network&lt;br&gt;&#xA;When having a &lt;strong&gt;2D&lt;/strong&gt; structure you use a &lt;strong&gt;CNN&lt;/strong&gt; neural network&lt;br&gt;&#xA;When having a &lt;strong&gt;sequence&lt;/strong&gt; you use a &lt;strong&gt;RNN&lt;/strong&gt; neural network&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you have preprocessed an instance and the result is a &lt;strong&gt;tree structure&lt;/strong&gt;.&lt;br&gt;&#xA;Let's say for simplicity that the tree structure is always the same tree; only the node values differ among instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of &lt;strong&gt;neural network architecture&lt;/strong&gt; would be required to consume the information of a tree structure but also leveraging the connections between the tree nodes?&lt;/p&gt;&#xA;" OwnerUserId="24885" LastEditorUserId="24885" LastEditDate="2018-01-04T14:01:58.143" LastActivityDate="2018-01-04T14:01:58.143" Title="How to treat input that inherently has a tree structure?" Tags="|neural-network|hierarchical-data-format|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26256" PostTypeId="1" CreationDate="2018-01-04T03:49:53.097" Score="1" ViewCount="83" Body="&lt;p&gt;When people start to figure the self driving cars will replace some vehicles on the road in the very near future, it somehow means the learner in the robot software could reach a very low empirical error to ensue the safety of passengers. The great deal of driving data collected and the deep reinforcement learning algorithms contribute mostly to the success of the self driving practice as we see it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to many cases of our own practice, we yet cannot build up a predictive model to such a high accuracy to bring the huge difference to our own business. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, when I saw the machine learning techniques can make revolution on the car driving due to its successful algorithms to implement behavioral cloning tasks well, what is the foremost reason for this success, the big training data set or the deep reinforcement learning techniques or any particular reason in self driving problems?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, can this success be copied in most of the practical problems we solve in machine learning? In other words, if today's machine learning techniques can help a car drive by itself, how can I reach the same success when developing my own predictive models to boost my business? If we cannot, what are the constraints, not enough data in the particular case or not a smart model yet? The reason the self driving thrives can help answer this question. &lt;/p&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2018-01-04T05:59:38.743" Title="Does self driving technology gain more from the data or the state-of-the-art algorithm?" Tags="|machine-learning|deep-learning|predictive-modeling|reinforcement-learning|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26257" PostTypeId="2" ParentId="26254" CreationDate="2018-01-04T04:24:14.033" Score="3" Body="&lt;p&gt;As @Emre mentioned, RNN is a good option. It's worth noting that if the number of possible nodes in each tree is the same or at least has the same upper bound, you could use literally any architecture you want and just pass in the adjacency matrix. Alternatively, you could build an intermediate model to convert your graph into a &lt;a href=&quot;https://arxiv.org/pdf/1705.02801.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph embedding&lt;/a&gt; and then once again, you can do basically whatever you want with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A pretty big piece of potentially important information here is what you are trying to accomplish, which could have significant consequences for how you want to represent your inputs.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-04T04:24:14.033" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26258" PostTypeId="2" ParentId="26256" CreationDate="2018-01-04T04:56:38.410" Score="1" Body="&lt;p&gt;I'll give a high level answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Our Bias on Effort&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although self-driving technology is now on the verge of becoming mainstream, incredible effort has gone into its development. The DARPA Grand Challenge, which helped in this technology's advancement, started in 2004. Imagine the total effort that went into this technology over 13-14 years. Many labs were setup with sole aim of competing in this challenge. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So when you say that we can't replicate the success in our businesses directly, it is better to ask if the comparison is fair. In your business, have you worked on a single problem to such an extent that &lt;em&gt;many&lt;/em&gt; bright minds have worked on it for &lt;em&gt;many years&lt;/em&gt;? If not, the comparison is not fair (to your business!).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Self-driving cars are &lt;em&gt;not overnight success due to a single algorithm&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No Single Secret Sauce&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Media reporting makes us believe that there's one algorithm that rules the technology, reality is more complex. &lt;a href=&quot;http://robots.stanford.edu/papers/thrun.stanley05.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Overview paper by Stanley team on DARPA challenge&quot;&gt;Check this review paper on the complexity of tasks&lt;/a&gt;. There are many modules involved (sensors, perception, planner and control module, feedback loops). Yes, probability and machine learning plays a large part in all modules, but there's no single &lt;em&gt;oracle&lt;/em&gt; algorithm. Old workhorse algorithms like Kalman Filters also shoulder heavy workload, but wouldn't get mentioned in the deep learning hype.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes large data helps, reinforcement learning helps. But so do rules inferred from manual interventions and control theory and older algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many areas from engineering (classic control theory, reinforcement learning, machine learning) form the basis of self driving cars. We can't copy over the success to other domain easily: there's no single secret sauce but tons of hard work and solid engineering over &lt;em&gt;long&lt;/em&gt; time.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastEditorUserId="23204" LastEditDate="2018-01-04T05:59:38.743" LastActivityDate="2018-01-04T05:59:38.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26260" PostTypeId="2" ParentId="26236" CreationDate="2018-01-04T06:07:26.187" Score="0" Body="&lt;p&gt;Your direction of using k-means on term-document matrix is correct. To go further, you can&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Use Weighted k-means&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try artificially boosting the weight for certain columns/variables (corresponding to your pre-specified list of words). This will force the similarity calculation to be dominated by those words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Use Constrained k-means&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read up more about constrained k-means algorithm: you can force certain points to be in the same cluster. In other words, if there are two documents containing words from the pre-specified list, they are forced to be in same cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Alternative: Classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know that there are only two clusters to be formed, why stick to only unsupervised learning? Depending on the application and availability of labelled data, you may think of it as a binary classification task using nearest neighbor or similar algorithm.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2018-01-04T06:07:26.187" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26261" PostTypeId="1" CreationDate="2018-01-04T06:10:11.240" Score="1" ViewCount="646" Body="&lt;p&gt;I want to cluster my document vectors (doc2vec) using affinity propagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am just confused if I should use cosine similarity or cosine distance to cluster my document vectors. Currently, I am using cosine similarity for my affinity propagation clustering. Thus, my first question is;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it correct to use cosine similarity to cluster my doc2vec document vectors?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, I would like to visualize my cluster results using t-sne. However, I saw that t-sne requires distance matrix as the input. Hence, my second question is;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is it correct to use distance matrix (cosine distance) for t-sne, while I use cosine similarity for clustering?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my code is required I can post it too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me.&lt;/p&gt;&#xA;" OwnerUserId="38395" LastActivityDate="2018-07-03T14:28:45.490" Title="Cluster doc2vec using Affinity Propagation" Tags="|scikit-learn|clustering|word2vec|gensim|tsne|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26264" PostTypeId="1" CreationDate="2018-01-04T08:10:52.187" Score="6" ViewCount="8777" Body="&lt;p&gt;I often stumble upon papers saying that the NN was trained using maximum likelihood principle. I think I generally understand what maximum likelihood is. But I have trouble understanding how to apply it to neural network training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How are neural network trained using maximum likelihood principle?&lt;/p&gt;&#xA;" OwnerUserId="21586" LastEditorUserId="29575" LastEditDate="2018-01-04T14:04:10.560" LastActivityDate="2018-11-05T15:57:03.140" Title="How to train neural network using maximum likelihood principle" Tags="|neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26265" PostTypeId="1" AcceptedAnswerId="26267" CreationDate="2018-01-04T08:27:24.110" Score="0" ViewCount="402" Body="&lt;p&gt;Logistic function is well known to be a good binary classificator, as it can be easily shown with this image (let x be the dot product of [x1,x2] and [w1, w2]): enter image description here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IWk7u.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IWk7u.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently learning how a multiclass logistic classifier works, and this time with x1 and x2 as axes, three classes can be represented as follows (from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_multinomial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt;): &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Aa44I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Aa44I.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to several sites, both the &quot;one vs the rest&quot; and the multinomial logistic classifiers, would have one logistic regression for each of these 3 classes, with the difference of training them &quot;together&quot; or separately. If this is true, and we assume the next figure to be related to the previous example, I can't figure out how these 3 regressions (as in the first example) would be plotted in there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kX3OJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kX3OJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As much I could figure, in the &quot;one vs the rest&quot; case, how to draw the logistic function for blue and yellow categories against the rest, but how would it be for the red category against the rest?&lt;/p&gt;&#xA;" OwnerUserId="43309" LastActivityDate="2018-01-04T09:52:27.120" Title="How would be plotted the logistic functions associated to a multiclass logistic classifier in a X-Y plot?" Tags="|visualization|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26266" PostTypeId="2" ParentId="26264" CreationDate="2018-01-04T08:34:15.127" Score="1" Body="&lt;p&gt;If you have a probabilistic cost function (e.g log-loss), I'm pretty sure backprop is an estimator for the MLE. Consider the derivation here: &lt;a href=&quot;http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture3_flat.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture3_flat.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have articles on hand, I'd be interested to see an example of an experiment where they discussed fitting a network via MLE but explicitly were not using backprop. Otherwise, I think you can assume that the authors were describing backprop when they said &quot;MLE&quot;. Maybe they thought &quot;MLE&quot; sounded more academic?&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-04T08:34:15.127" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26267" PostTypeId="2" ParentId="26265" CreationDate="2018-01-04T08:47:28.193" Score="0" Body="&lt;p&gt;If you are asking how to reproduce those plots, the following technique works to visualize the decision boundaries for any model:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate a dense grid of coordinates that fill your plotting area&lt;/li&gt;&#xA;&lt;li&gt;Score each point on your grid&lt;/li&gt;&#xA;&lt;li&gt;Plot your results, coloring observations based on your model's predictions. This serves as the background.&lt;/li&gt;&#xA;&lt;li&gt;Overlay your test data as a scatter plot&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In your link, they generate the grid using np.meshgrid and constrtuct the background as a contour plot. The lines representing the decision boundaries of each respective 1-vs-all classifier is plotted using a closed form solution for logistic decision boundaries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: To answer the question in your comment, you don't have a single $X$ dimension, and consequently your model output doesn't correspond to a curve as simple as this: it's a 3D surface. The simplest solution would be to just apply the strategy I suggested earlier (and which is also described in your link) but instead of calling &lt;code&gt;.predict()&lt;/code&gt; to construct the background coloration, you'd call &lt;code&gt;.predict_proba()&lt;/code&gt; and use a sequential color map (e.g. the default veridis) so color intensity corresponds to your class likelihood, giving you something which should look like &lt;a href=&quot;https://matplotlib.org/examples/images_contours_and_fields/pcolormesh_levels.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. Alternatively, you could plot &lt;a href=&quot;https://matplotlib.org/2.0.0/examples/pylab_examples/contour_demo.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;surface curves&lt;/a&gt;. Both of these solutions are projections of the $Y$ axis, $P(Y|x1,x2)$, onto the $X_1$-$X_2$ plane. If that isn't satisfactory, you could pass scored results through PCA to combine your $X$ dimensions into a single $X$ feature -- call this $PC_1$ -- and then plot $PC_1$ vs. $P(Y|x1,x2)$. Personally I think this is significantly less informative, but it would give you an x vs. y plot. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-04T09:52:27.120" LastActivityDate="2018-01-04T09:52:27.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26268" PostTypeId="2" ParentId="26261" CreationDate="2018-01-04T08:56:29.153" Score="1" Body="&lt;p&gt;Both of them convert the distances back to similarities, albeit using different methods. They will, if I recall correctly, also &lt;em&gt;square&lt;/em&gt; the distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may be problematic with the most common variant of cosine distance, which already is a squared distance. So it may be a good idea to modify the methods to be able to directly work with the similarities. But you probably need to modify the source code for this (and understand the methods!)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-04T08:56:29.153" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26271" PostTypeId="1" CreationDate="2018-01-04T10:05:10.240" Score="1" ViewCount="954" Body="&lt;p&gt;In the following example, the prediction obtained for $x=[0,0]$ after fitting the model is $1$, but expected is $0$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array([&#xA;    [0, 0],&#xA;    [1, 0],&#xA;    [0, 1]&#xA;])&#xA;&#xA;Y = np.array([0,1,1])&#xA;&#xA;model = LogisticRegression()&#xA;model.fit(X,Y)&#xA;&#xA;x = [0,0]&#xA;y = model.predict([x])[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It only works if I duplicate the training row of ($x=[0,0], y=[0]$)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But why?&lt;/p&gt;&#xA;" OwnerUserId="43309" LastEditorUserId="29575" LastEditDate="2018-01-04T14:10:54.117" LastActivityDate="2018-01-04T14:10:54.117" Title="Simple logistic regression wrong predictions" Tags="|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26272" PostTypeId="2" ParentId="26236" CreationDate="2018-01-04T10:15:54.477" Score="0" Body="&lt;p&gt;K-means doesn't work reliable with such distance hacks. It can prevent the algorithm from converging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual approach would be to rather use this information to guide cluster extraction from a hierarchical clustering. I.e., you would merge clusters if they agree on the desired words, and stop merging if they do not agree, rather than cutting at a single height.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-04T10:15:54.477" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26273" PostTypeId="1" AcceptedAnswerId="26277" CreationDate="2018-01-04T11:03:41.687" Score="3" ViewCount="1944" Body="&lt;p&gt;I'm trying to build a toy model which can identify a constant difference between two variables: &lt;code&gt;(if variable1- variable2&amp;gt;10 then 1 else 0)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should be a quite simple task for any regression model, but I want to solve it with NN. However all simple NN I built can not give me more than 51% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something that I do not understand?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seed = 7&#xA;np.random.seed(seed)&#xA;&#xA;x1 = np.arange(50000)&#xA;x2 = x1+10+(0.5-np.random.rand(len(x1)))&#xA;X = np.column_stack((x1,x2))&#xA;&#xA;Y = (x2-x1)&amp;gt;10&#xA;&#xA;encoder = LabelEncoder()&#xA;encoder.fit(Y)&#xA;encoded_Y = encoder.transform(Y)&#xA;&#xA;train_X, test_X, train_y, test_y = train_test_split(&#xA;    X, Y, train_size=0.9, random_state=0)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(2, input_dim=2,activation='relu'))&#xA;model.add(Dense(16, activation='relu'))&#xA;model.add(Dense(1,  activation='sigmoid'))&#xA;&#xA;model.compile(optimizer=RMSprop(lr=0.001), &#xA;  loss=binary_crossentropy, metrics=[binary_accuracy])&#xA;&#xA;history = model.fit(train_X, train_y, epochs=1000, batch_size=10,&#xA;  validation_data=(test_X, test_y), verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44136" LastEditorUserId="836" LastEditDate="2018-01-04T12:16:35.337" LastActivityDate="2018-01-04T12:33:06.487" Title="Binary classification toy problem" Tags="|classification|keras|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26274" PostTypeId="1" AcceptedAnswerId="26301" CreationDate="2018-01-04T11:37:28.733" Score="2" ViewCount="77" Body="&lt;p&gt;I'm studying the gradient descent algorithm for single hidden layer neural networks.&#xA;Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why mathematically do the normalized features carry the same information of the initial features?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="26562" LastEditDate="2018-01-05T00:10:32.370" LastActivityDate="2018-01-05T23:53:26.290" Title="After a Feature Scaling do i have the same initial information?" Tags="|machine-learning|deep-learning|information-theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26275" PostTypeId="2" ParentId="26247" CreationDate="2018-01-04T11:45:48.110" Score="0" Body="&lt;p&gt;This is a common thing with neural networks and different batch sizes.  The training loss is the average of losses for the minibatch.  Naturally for the first few batches you'll have a higher loss and as it goes through the data the loss gets smaller.  Mean while the loss for the validation set is calculated against the entire dataset. &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-04T11:45:48.110" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26276" PostTypeId="2" ParentId="26271" CreationDate="2018-01-04T11:56:58.190" Score="1" Body="&lt;p&gt;Because you have a class imbalance and very little data. Your model is essentially working off of a prior probability that disproprtionately favors the positive class, and you didn't give the model enough data to escape this prior in the learning process. Consider the following detailed output from your model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.score(X,Y) # 0.6666&#xA;model.coef_      # array([[ 0.36566712,  0.36566712]])&#xA;model.intercept_ # array([ 0.18517658])&#xA;&#xA;model.predict_proba(X)&#xA;# array([[ 0.45383769,  0.54616231],&#xA;#        [ 0.36566869,  0.63433131],&#xA;#        [ 0.36566869,  0.63433131]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your model correctly identifies that [0,0] is more likely to be negative than either of the other observations, but the model isn't able to escape its bias towards the positive class. If you duplicate your dataset a couple of times, the model will eventually have enough &quot;evidence&quot; to escape this bias. Geometrically, two things will happen when we give the model more data: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The discrimination boundary will shift towards the negative class. This will manifest as an increasingly negative intercept term.&lt;/li&gt;&#xA;&lt;li&gt;The slope of the logistic curve will steepen, having the effect that a unit change in the inputs will have a larger impact on the outcome. If we give the model enough perfectly separable data, the curve will approach a step function and the model will output probabilities of 0 and 1. This manifests in the magnitude of the coefficients. With just three observations, the curve is fairly flat and changing the values of the inputs doesn't have much effect on the outcome probability, so the resulting class assignment is essentially determined by the intercept alone.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Contrast the results from your original model with the what happens when we replicate your dataset 100 times:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model2 = LogisticRegression()&#xA;X2 = np.matlib.repmat(X,100,1)&#xA;Y2 = np.matlib.repmat(Y,100,1).ravel()&#xA;model2.fit(X2, Y2)&#xA;&#xA;model2.score(X,Y) # 1.0&#xA;model2.coef_      # array([[ 4.95489068,  4.95489068]])&#xA;model2.intercept_ # array([-2.00091433])&#xA;model2.predict_proba(X)&#xA;&#xA;# array([[ 0.88089304,  0.11910696],&#xA;#        [ 0.04954891,  0.95045109],&#xA;#        [ 0.04954891,  0.95045109]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-04T11:56:58.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26277" PostTypeId="2" ParentId="26273" CreationDate="2018-01-04T12:13:22.390" Score="5" Body="&lt;p&gt;Your problem is that neural networks work poorly when the input is not scaled to a simple range. A usual choice is to scale and offset each column so that it has mean 0 and standard deviation 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, &lt;code&gt;x1&lt;/code&gt; and &lt;code&gt;x2&lt;/code&gt; vary from 0 to 49999 and roughly 10 to 50009. This range for inputs will causes lots of numeric issues. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a balanced dataset as you have, 51% accuracy is basically just guessing (within experimental error), so the network has learned nothing. Try again with scaling - e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 = (x1 - 25000) / 14433&#xA;x2 = (x2 - 25000) / 14433&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tested your code with this addition, and it gains 100% validation accuracy within the first epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to assess other values in testing later you will need to scale them in the same way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note your predictions may be off in testing when &lt;code&gt;x1&lt;/code&gt; and &lt;code&gt;x2&lt;/code&gt; are not close to 10 apart, because you have only trained with examples which are close to exactly 10 apart. How the network behaves when this is not the case - e.g. for inputs of x1 = 100 and x2 = 1000, or x1 = 90 and x2 = 15, may not generalise well compared to the original comparison function.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-04T12:33:06.487" LastActivityDate="2018-01-04T12:33:06.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26278" PostTypeId="2" ParentId="26271" CreationDate="2018-01-04T12:13:37.833" Score="1" Body="&lt;p&gt;This is because you have an imbalanced dataset towards class 0. I have taken a look on the logistic regression coefficient you get. On the below chart 1, I have plotted the decision boundary you get with your logistic regression. On the second chart, I have plotted what you have expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why this difference and why does your logistic regression yield chart 1? The reason is even if it is a machine learning algorithm, it is not really smart. The logistic regression algorithm  wants to minimize its cost fucntion (cross-entropy). Cross-entropy can be defined in a really simple way as the distance between your points and the decision boundary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, in your training case, you have two class-1 observations for only one class_0 observations. Thus the machine learning algorithm will see that in order to reach the lowest cost function as possible, the best choice is to promote the two class-1 observations than the only one class-0 observation. The cost function value is larger on chart 2 than on chart 1, this is why it has yielded chart 1 result. Majority wins !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To avoid this problem, here are three ideas you can do : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Undersample / oversample your dataset in order to have a similar number of observations in each class of your training set.&lt;/li&gt;&#xA;&lt;li&gt;Change your observations class weight.&lt;/li&gt;&#xA;&lt;li&gt;Use an other cost function like Hinge loss that is the cost function used in SVM and where the goal is to maximize the margin between classes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0fdvi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0fdvi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-01-04T12:13:37.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26280" PostTypeId="1" AcceptedAnswerId="26281" CreationDate="2018-01-04T12:19:07.550" Score="1" ViewCount="1703" Body="&lt;p&gt;my array is looking like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;a=np.array([[ 25,  29,  19,  93],&#xA;       [ 27,  59,  23,  345],&#xA;       [ 24,  426,  15,  593],&#xA;       [ 24,  87,  50.2, 139],&#xA;       [ 13,  86,  12.4, 139],&#xA;       [ 13,  25,  85, 142],&#xA;       [ 62,  62,  68.2, 182],&#xA;       [ 27,  25,  20, 150],&#xA;       [ 25,  53,  71, 1850],&#xA;       [ 64,  67,  21.1, 1570],&#xA;       [ 64,  57,  73, 1502]])&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i want to return the lowest value of column 2 based on the unique value of column 0. column 0 should contain unique values.&#xA;I tries the following code, but was not giving me the exact result.&#xA;Can some one help me to sort out this? thanks&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;sidx = np.lexsort(a[:,[2,0]].T)&#xA;dx = np.append(np.flatnonzero(a[2:,0] &gt;a[:-2,0]), a.shape[0]-1) &#xA;result = a[sidx[idx]]&#xA;print result&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to get result like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;[25...&#xA; 27&#xA; 24&#xA; 13&#xA; 62&#xA; 64...]&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;pre&gt;&#xA;a=[[196512 28978 Decimal('12.7805170314276')]&#xA; [196512 34591 Decimal('12.8994111000000')]&#xA; [196512 13078 Decimal('12.9135746000000')]&#xA; [196641 114569 Decimal('12.9267705000000')]&#xA; [196641 118910 Decimal('12.8983353775637')]&#xA; [196641 100688 Decimal('12.9505091000000')]]this is a big list&#xA;i used,&#xA;df = pd.DataFrame(a)&#xA;df.columns = ['a','b','c']&#xA;df.index = df.a.astype(str) &#xA;dd=df.groupby('a').min()['c']&#xA;&#xA;but i am getting,&#xA;&#xA;195556    12.7805170314276&#xA;195937    12.7805170314276&#xA;196149    12.7805170314276&#xA;196152    12.7805170314276&#xA;196155    12.7805170314276&#xA;196262    12.7805170314276&#xA;&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="43680" LastEditorUserId="43680" LastEditDate="2018-01-04T13:35:18.667" LastActivityDate="2018-01-04T13:37:33.313" Title="unique column value in python numpy" Tags="|python|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26281" PostTypeId="2" ParentId="26280" CreationDate="2018-01-04T12:41:49.577" Score="1" Body="&lt;p&gt;Here's an easy solution. The sort order changes, but that shouldn't be difficult to address if you really care:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame(a)&#xA;df.columns = ['a','b','c','d']&#xA;df.index = df.a.astype(str) # to preserve correspondence&#xA;df.groupby('a').min()['b']&#xA;&#xA;a&#xA;13.0    25.0&#xA;24.0    87.0&#xA;25.0    29.0&#xA;27.0    25.0&#xA;62.0    62.0&#xA;64.0    57.0&#xA;Name: b, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Edit: I think you meant to name your array &lt;code&gt;y&lt;/code&gt; instead of &lt;code&gt;a&lt;/code&gt;. This works for me:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from decimal import Decimal&#xA;&#xA; y=np.array([[196512, 28978, Decimal('12.7805170314276')], &#xA;    [196512, 34591, Decimal('12.8994111000000')] ,&#xA;    [196512, 13078, Decimal('12.9135746000000')] ,&#xA;    [196641, 114569, Decimal('12.9267705000000')] ,&#xA;    [196641, 118910, Decimal('12.8983353775637')] ,&#xA;    [196641, 100688, Decimal('12.9505091000000')]])&#xA;&#xA;&#xA; df = pd.DataFrame(y) &#xA; df.columns = ['a','b','c'] &#xA; df.index = df.a.astype(str) &#xA; dd=df.groupby('a').min()['c'] &#xA;&#xA;In [210]: dd&#xA;Out[210]:&#xA;a&#xA;196512    12.7805170314276&#xA;196641    12.8983353775637&#xA;Name: c, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-04T13:37:33.313" LastActivityDate="2018-01-04T13:37:33.313" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26282" PostTypeId="1" CreationDate="2018-01-04T12:56:29.543" Score="2" ViewCount="127" Body="&lt;p&gt;I have three questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can we assess (or measure) the performance of the ranking algorithms? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are there any specific measures, or performance metrics, for this?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;More specifically, how can we compare the performance of AHP-based ranking and probabilistic ranking algorithms?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26784" LastEditorUserId="29575" LastEditDate="2018-01-04T14:00:18.787" LastActivityDate="2018-01-04T16:55:19.713" Title="Peformance evaluation of ranking algorithms" Tags="|information-retrieval|ranking|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26283" PostTypeId="1" AcceptedAnswerId="26284" CreationDate="2018-01-04T13:03:28.490" Score="14" ViewCount="86560" Body="&lt;p&gt;I need to find the accuracy of a training dataset by applying Random Forest Algorithm. But my the type of my data set are both categorical and numeric. When I tried to fit those data, I get an error.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;'Input contains NaN, infinity or a value too large for dtype('float32')'.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;May be the problem is for object data types. How can I fit categorical data without transforming for applying RF?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my code.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V6sHp.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V6sHp.png&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/k7nOE.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k7nOE.png&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3NXtO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3NXtO.png&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="55122" LastEditDate="2020-04-27T21:26:23.093" LastActivityDate="2020-06-16T21:50:26.260" Title="How can I fit categorical data types for random forest classification?" Tags="|python|scikit-learn|data-mining|random-forest|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26284" PostTypeId="2" ParentId="26283" CreationDate="2018-01-04T13:09:42.183" Score="21" Body="&lt;p&gt;You need to convert the categorical features into numeric attributes. A common approach is to use one-hot encoding, but that's definitely not the only option. If you have a variable with a high number of categorical levels, you should consider combining levels or using the hashing trick. Sklearn comes equipped with several approaches (check the &quot;see also&quot; section): &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder&quot; rel=&quot;noreferrer&quot;&gt;One Hot Encoder&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html&quot; rel=&quot;noreferrer&quot;&gt;Hashing Trick&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're not committed to sklearn, the &lt;a href=&quot;http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/drf.html&quot; rel=&quot;noreferrer&quot;&gt;h2o random forest&lt;/a&gt; implementation handles categorical features directly.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="29575" LastEditDate="2018-08-16T02:09:42.100" LastActivityDate="2018-08-16T02:09:42.100" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26285" PostTypeId="2" ParentId="26271" CreationDate="2018-01-04T13:13:36.020" Score="0" Body="&lt;p&gt;In the simplest language, your model &quot;sees&quot; more number of class 1 examples than class 0 examples (because of very little data), so the probability of having class 1 as output is higher. As others have pointed out, this is a class imbalance problem. So, even if you give &lt;code&gt;[0,0]&lt;/code&gt; in prediction, it will return &lt;strong&gt;class 1&lt;/strong&gt; as output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current ratio of classes 1 and 0 in training set is 2:1. When you duplicate the data, ratio becomes 1:1, so the model now correctly predicts your example.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-01-04T13:13:36.020" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26286" PostTypeId="2" ParentId="26213" CreationDate="2018-01-04T13:32:14.810" Score="0" Body="&lt;p&gt;I don't think having new class value would be a problem, unless you decide to convert the class values into features. If you don't want to run into errors, either drop the new feature in test phase, or ensure that it comes during training phase.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-01-04T13:32:14.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26287" PostTypeId="2" ParentId="26264" CreationDate="2018-01-04T14:02:54.863" Score="7" Body="&lt;p&gt;This is pretty common. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last step for a simple deep neural network would be to determine a way to choose values of &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt; that maximize the likelihood of your training data. So, what do you do? Typically, you write a log-likelihood function, and then find the values that maximize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, it's relatively trivial to prove that minimizing the loss is equivalent to maximizing the likelihood of the input. Typically, in academic papers, you may see people prefer maximizing likelihood simply for the mere fact that some proofs are easier that way.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Consider the &lt;em&gt;Binary Cross Entropy&lt;/em&gt; formula &lt;span class=&quot;math-container&quot;&gt;$$CE(y,x,\theta) = - \sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$&lt;/span&gt;  Naturally, you would optimize this with respect to &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can instead choose to take a probabilistic approach and maximize the likelihood of the data under some probabilistic model.  A natural choice would be to use a Bernoulli distribution, such that &lt;span class=&quot;math-container&quot;&gt;$$p(y \mid \pi) = \prod_{i=1}^{y_{i}} \pi_{i}^{y_i}(1-\pi_{i})^{1-y_{i}}.$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the task here is to train a neural network to estimate &lt;span class=&quot;math-container&quot;&gt;$\pi$&lt;/span&gt;. The likelihood function is simply &lt;span class=&quot;math-container&quot;&gt;$$p(y \mid x, \theta) = \prod_{i=1}^{y_{i}} p_{\theta}(y \mid x_{i})^{y_i} (1-p_{\theta}(y \mid x_{i}))^{1-y_{i}}.$$&lt;/span&gt;  We want to maximize the function w.r.t &lt;span class=&quot;math-container&quot;&gt;$\theta$&lt;/span&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you take the log, and we find &lt;span class=&quot;math-container&quot;&gt;$$\sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$&lt;/span&gt;  Thus there is literally no difference in minimizing a cross entropy or maximizing the log likelihood.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the only difference is the approach. At the end of the day, you're doing the same thing regardless what you choose to call it.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorDisplayName="user10640" LastEditDate="2018-11-05T15:57:03.140" LastActivityDate="2018-11-05T15:57:03.140" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26288" PostTypeId="1" CreationDate="2018-01-04T14:43:00.370" Score="2" ViewCount="813" Body="&lt;p&gt;I have recently started Andrew NG's Machine Learning course on Coursera and I came across this cost function which is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fp2i0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fp2i0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does the error in the cost function need to be squared? If its purpose is to eliminate the negative sign in the error then why don't we simply use the absolute function?&lt;/p&gt;&#xA;" OwnerUserId="44143" LastEditorUserId="29575" LastEditDate="2018-01-04T14:47:45.830" LastActivityDate="2018-01-04T16:12:04.240" Title="Machine Learning: Why do the error in cost function need to be squared?" Tags="|machine-learning|predictive-modeling|statistics|" AnswerCount="2" CommentCount="0" ClosedDate="2018-01-05T05:45:52.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="26289" PostTypeId="2" ParentId="26288" CreationDate="2018-01-04T14:52:12.570" Score="0" Body="&lt;p&gt;The simple answer is that it's conviences rather than necessities.  You're more than welcome to take the absolute value and in many cases it may be better to do so.  Squaring a function makes the math a bit happier and easier and for proofs has desirable properties.  While you can basically do the same proofs with an absolute value function, you may have to handle certain edge cases and just amounts to more writing.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-04T14:52:12.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26290" PostTypeId="1" CreationDate="2018-01-04T15:31:49.880" Score="3" ViewCount="3617" Body="&lt;p&gt;I understand &lt;code&gt;Conv2D&lt;/code&gt; filters. I think I understand &lt;code&gt;Conv1D&lt;/code&gt; filters as well but have not seen any examples of the filters like what we have in 2D filters, e.g. &lt;code&gt;Sobel&lt;/code&gt; as shown in the following image. Are there any standard one dimensional filters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sobel 2D Filter example&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RBFN4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RBFN4.png&quot; alt=&quot;Sobel Filter&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41108" LastEditorUserId="26562" LastEditDate="2018-01-06T01:39:20.460" LastActivityDate="2018-02-05T01:44:11.443" Title="Example of 1D ConvNet filter" Tags="|machine-learning|neural-network|deep-learning|convolution|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26291" PostTypeId="2" ParentId="15903" CreationDate="2018-01-04T16:04:20.020" Score="28" Body="&lt;p&gt;Actually I guess the question is a bit broad! Anyway. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Understanding Convolution Nets&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;What is learned in &lt;code&gt;ConvNets&lt;/code&gt; tries to minimize the cost function to categorize the inputs correctly in classification tasks. All parameter changing and learned filters are in order to achieve the mentioned goal. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Learned Features in Different Layers&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;They try to reduce the cost by learning low level, sometimes meaningless, features like horizontal and vertical lines in their first layers and then stacking them to make abstract shapes, which often have meaning, in their last layers. For illustrating this fig. 1, which has been used from &lt;a href=&quot;http://yosinski.com/deepvis#toolbox&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, can be considered. The input is the bus and the gird shows the activations after passing the input through different filters in the first layer. As it can be seen the red frame which is the activation of a filter, which its parameters have been learned, has been activated for relatively horizontal edges. The blue frame has been activated for relatively vertical edges. It is possible that &lt;code&gt;ConvNets&lt;/code&gt; learn unknown filters that are useful and we, as e.g. computer vision practitioners, have not discovered that they may be useful. The best part of these nets is that they try to find appropriate filters by their own and don't use our limited discovered filters. They learn filters to reduce the amount of cost function. As mentioned these filters are not necessarily known. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/j7QHf.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/j7QHf.png&quot; alt=&quot;**Figure 1.** *Low level activations*&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In deeper layers, the features learned in previous layers come together and make shapes which often have meaning. In &lt;a href=&quot;http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; it has been discussed that these layers may have activations which are meaningful to us or the concepts which have meaning to us, as human beings, may be distributed among other activations. In fig. 2 the green frame shows the activatins of a filter in the fifth layer of a &lt;code&gt;ConvNet&lt;/code&gt;. This filter cares about the faces. Suppose that the red one cares about hair. These have meaning. As it can be seen there are other activations that have been activated right in the position of typical faces in the input, the green frame is one of them; The blue frame is another example of these. Accordingly, abstraction of shapes can be learned by a filter or numerous filters. In other words, each concept, like face and its components, can be distributed among the filters. In cases where the concepts are distributed among different layers, if someone look at each of them, they may be sophisticated. The information is distributed among them and for understanding that information all of those filters and their activations have to be considered although they may seem so much complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3cGMQ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3cGMQ.png&quot; alt=&quot;**Figure 2.** *High level activations*&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;CNNs&lt;/code&gt; should not be considered as black boxes at all. &lt;em&gt;Zeiler et all&lt;/em&gt; in &lt;a href=&quot;http://arxiv.org/pdf/1311.2901v3.pdf&quot; rel=&quot;noreferrer&quot;&gt;this amazing paper&lt;/a&gt; have discussed the &lt;em&gt;development of better models is reduced to trial and error&lt;/em&gt; if you don't have understanding of what is done inside these nets. This paper tries to visualize the feature maps in &lt;code&gt;ConvNets&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Capability to Handle Different Transformations to Generalize&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ConvNets&lt;/code&gt; use &lt;code&gt;pooling&lt;/code&gt; layers not only to reduce the number of parameters but also to have the capability to be insensitive to the exact position of each feature. Also the use of them enables the layers to learn different features which means first layers learn simple low level features like edges or arcs, and deeper layers learn more complicated features like eyes or eyebrows. &lt;code&gt;Max Pooling&lt;/code&gt; e.g. tries to investigate whether a special feature exists in a special region or not. The idea of &lt;code&gt;pooling&lt;/code&gt; layers is so useful but it is just capable to handle transition among other transformations. Although filters in different layers try to find different patterns, e.g. a rotated face is learned using different layers than a usual face, &lt;code&gt;CNNs&lt;/code&gt; by there own do not have any layer to handle other transformations. To illustrate this suppose that you want to learn simple faces without any rotation with a minimal net. In this case your model may do that perfectly. suppose that you are asked to learn all kind of faces with arbitrary face rotation. In this case your model has to be much more bigger than the previous learned net. The reason is that there have to be filters to learn these rotations in the input. Unfortunately these are not all transformations. Your input may also be distorted too. These cases made &lt;em&gt;Max Jaderberg et all&lt;/em&gt; angry. They composed &lt;a href=&quot;https://arxiv.org/abs/1506.02025&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; paper to deal with these problems in order to settle down our anger as theirs.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Convolutional Neural Networks Do Work&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Finally after referring to these points, they work because they try to find patterns in the input data. They stack them to make abstract concepts by there convolution layers. They try to find out whether the input data has each of these concepts or not in there dense layers to figure out which class the input data belongs to. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I add some links which are helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/vdumoulin/conv_arithmetic&quot; rel=&quot;noreferrer&quot;&gt;Understanding convolution operation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://gist.github.com/akiross/754c7b87a2af8603da78b46cdaaa5598&quot; rel=&quot;noreferrer&quot;&gt;Understanding arithmetic behind &lt;code&gt;ConvNets&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/&quot; rel=&quot;noreferrer&quot;&gt;Useful toolboxes for tracking what happens in these nets&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-04T16:18:18.483" LastActivityDate="2018-01-04T16:18:18.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26292" PostTypeId="2" ParentId="26288" CreationDate="2018-01-04T16:12:04.240" Score="0" Body="&lt;p&gt;We use cost function to have the amount of error for a specified set of weights. We should find the weights which minimize the cost function. The used approach for minimizing the cost function are based on gradients. Means that you should move toward directions which minimize the error. To do so, the cost function have to have derivatives. Absolute function does not have derivatives in some places. Quadratic functions like square, has derivative. Although there are other reasons that we have become to this squared function, the reason it's not absolute was what I referred to.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-04T16:12:04.240" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26293" PostTypeId="2" ParentId="26282" CreationDate="2018-01-04T16:55:19.713" Score="1" Body="&lt;p&gt;Kaggle's famous competition &lt;a href=&quot;https://www.kaggle.com/c/chess&quot; rel=&quot;nofollow noreferrer&quot;&gt;Chess ratings - Elo versus the Rest of the World&lt;/a&gt;, that aimed &quot;to discover whether other approaches can predict the outcome of chess games more accurately than the workhorse Elo rating system&quot;, used this structure&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A similar structure - starting from a complete dataset, using first part &#xA; for training and last part to check the outcome - could be useful to &lt;em&gt;measure the performance of the ranking algorithms&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27334" LastActivityDate="2018-01-04T16:55:19.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26294" PostTypeId="1" AcceptedAnswerId="26316" CreationDate="2018-01-04T17:38:58.427" Score="2" ViewCount="766" Body="&lt;p&gt;I am working on titanic dataset. I have one feature &lt;strong&gt;Pclass&lt;/strong&gt; which I understand is an ordinal variable having values 1,2 and 3. I have created a new feature &lt;strong&gt;cabin_int&lt;/strong&gt; from feature &lt;em&gt;Cabin&lt;/em&gt;, which is essentially the number of cabins alloted to a passenger. So, it has values like 0,1,2,3 and 4. Now this new feature is not ordinal, it is just a continuous variable taking only integer values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how does Machine Learning algorithm understands the difference between the two, if I pass these two columns (&lt;strong&gt;Pclass&lt;/strong&gt; and &lt;strong&gt;cabin_int&lt;/strong&gt;) as they are during training of my model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If some more feature engineering needs to be done, please tell.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-06T12:41:32.003" Title="Ordinal Integer variable vs Continuous Integer variable" Tags="|machine-learning|feature-engineering|feature-construction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26296" PostTypeId="1" CreationDate="2018-01-04T18:18:28.390" Score="2" ViewCount="36" Body="&lt;p&gt;A colleague told me that there are terms for two different methods of representing repeated text in the training set for a classifier, but he could not recall them. What are the terms for the options described below?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I want to train a classifier, such as the &lt;code&gt;scikit-learn&lt;/code&gt; Naive Bayes classifier on sentences extracted from text documents. Any sentence may be appear any number of times in the documents. I have determined the number of instances of each distinct sentence and assigned labels to the ones I will use for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 1: One Instance per Distinct Sentence&lt;/strong&gt;&#xA;I could include in my training set one instance of each distinct sentence. This method gives equal weight to different sentences, regardless of their frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 2: Multiple Instances by Relative Frequency&lt;/strong&gt;&#xA;I could include in my training set multiple instances of each distinct sentence based on the number of times the sentence appears in the corpus. If the corpus contains 100 instances of Sentence 1 and 25 instances of Sentence 2, then my training set would include four instances of Sentence 1 and one instance of Sentence 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method gives greater weight to sentences that appear more frequently in the corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Other Options:&lt;/strong&gt; Are there related strategies that also have standard names?&lt;/p&gt;&#xA;" OwnerUserId="41093" LastActivityDate="2018-01-04T18:18:28.390" Title="Term for Methods of Representing Repeated Text in Classifier" Tags="|classification|scikit-learn|training|terminology|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26297" PostTypeId="2" ParentId="26093" CreationDate="2018-01-04T18:23:10.637" Score="0" Body="&lt;p&gt;I don't know what is wrong with your code but this worked fine for me-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;cat_var.replace(to_replace=cat_var[noms].tolist(),value=encode,inplace=True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of converting encoded values to series, convert series to list using &lt;code&gt;tolist()&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-01-04T18:23:10.637" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26298" PostTypeId="2" ParentId="17382" CreationDate="2018-01-04T19:32:55.083" Score="2" Body="&lt;p&gt;BRAT was not designed with classification tasks in mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like Valentin mentions, Prodigy is a more modern annotation management systems that fits well in a classification task workflow. We've been testing it at work - It is promising, but also has its limits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You mention the potential need for crowdsourcing - I'm sure you know about AWS Mechanical Turk. You also talk about data/annotation integrity validation - Your best bet might very well be to develop your own annotation management system to cater to these needs. We have very similar requirements, and this is the route we've chosen to take.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We evaluated Prodigy before its 1.0 release - It had quite a few quirks, plus not all features were available at the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to be very good at quick prototyping and building MVPs, but less adapted at serious large-scale annotation tasks. It's great to be able to answer quickly YES/NO to machine-generated annotations, but in our experience in many tasks those are not accurate often enough and it would be more efficient to manually annotate. It seems we can now load our own models into it, so there would be a possibility to improve that if we would invest dev time into Prodigy's framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another very important aspect missing is any kind of distributed annotator management system - Multi-users, conflicts resolutions, etc. Add to that no support for integrity/validation checks. Prodigy team says it's all in the works though:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We're also working on an extension library, the Prodigy Annotation Manager, which will integrate with Prodigy and will let you set up complex annotation projects, manage multiple annotators, enforce quality control and keep track of the progress via an admin console. You can sign up to our mailing list to be notified about the private beta.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Edit: Just stumbled on &lt;a href=&quot;https://webanno.github.io/webanno/&quot; rel=&quot;nofollow noreferrer&quot;&gt;WebAnno&lt;/a&gt;, which might be worth a look.&lt;/p&gt;&#xA;" OwnerUserId="44160" LastEditorUserId="44160" LastEditDate="2018-01-05T15:11:57.793" LastActivityDate="2018-01-05T15:11:57.793" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26300" PostTypeId="2" ParentId="26116" CreationDate="2018-01-04T20:55:31.900" Score="0" Body="&lt;p&gt;This is a very general question, since you are not providing any statistics, that one can comment on. But in general: yes, this can be the case. However, there is an interesting paper on comparing classification models with statistical tests written by &lt;a href=&quot;http://www.jmlr.org/papers/v7/demsar06a.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Demšar (2006)&lt;/a&gt;. He shows a way to vizualize the results of different models that do not significantly differ in quality. &lt;/p&gt;&#xA;" OwnerUserId="25217" LastActivityDate="2018-01-04T20:55:31.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26301" PostTypeId="2" ParentId="26274" CreationDate="2018-01-05T00:04:26.290" Score="0" Body="&lt;p&gt;I assume by &lt;em&gt;mean normalization&lt;/em&gt;, you mean scaling each feature by subtracting the mean and dividing by standard deviation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;x_{\text{scaled}} = \frac{x - \bar{x}}{\sigma}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x$ is a feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though you are changing all of the values of $x$, they are each being scaled by the same amount - initially, a translation by a constant (subtracting the mean), and then scaling by a constant (dividing by standard deviation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a two-dimensional, randomly generated dataset generated with scikit-learn's &lt;code&gt;make_blobs&lt;/code&gt; function (left) and the scaled version using the above equation for the $x$- and $y$-coordinates (right):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mYTDV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mYTDV.png&quot; alt=&quot;Original data and mean normalised data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The $x$- and $y$-values for each point have changed, but they are still all in the same place &lt;em&gt;relative to each other&lt;/em&gt;. If you look closely, you can see that the structure of the data is identical, even though it has been scaled a bit, and you could go back to something that looks more like the original by simply 'zooming in' on the data. Because the structure of the data is the same, we say no information was lost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider a transformation where we only take the $x$-value, and set all $y$-values to $0$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kk3Xk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kk3Xk.png&quot; alt=&quot;Original data and transformed by only keeping x-coordinate&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The structure of the data has changed, and there is no way to return to the original data by scaling or stretching space uniformly, so we say information was lost here. This is an extreme example, but hopefully it illustrates the point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to think about it is to think if it is more or less difficult for a classifier to distinguish between the classes after the transformation. In the first case, we can draw a line that perfectly separates the two clusters just as easily with the original data or the normalised data, but in the second case, there is no such line that separates the transformed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, if you normalise each example rather than each feature (as asked in your comment), for this data, you end up with something that looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Oxa8p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Oxa8p.png&quot; alt=&quot;Data normalised by example rather than feature&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where all points land on either $(-1,1)$ or $(1,-1)$. This makes sense, because normalisation makes the range of the values span from $-1$ to $1$. When there are only two dimensions, one of them has to become $-1$ and the other has to become $1$. Hopefully it's fairly obvious that information is lost here, and it's generally not a good idea to do this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is quite a hand-wavy explanation and doesn't really cover any actual information theory concepts, but hopefully it gives you some intuition for this. If you want to dive deeper into the mathematical side of things, have a look at the Wikipedia article for &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_theory&quot; rel=&quot;nofollow noreferrer&quot;&gt;information theory&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-01-05T23:53:26.290" LastActivityDate="2018-01-05T23:53:26.290" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26302" PostTypeId="1" CreationDate="2018-01-05T00:14:14.103" Score="1" ViewCount="83" Body="&lt;p&gt;Suppose I have examples of two categories, Products, and Aisles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each product has variables like price, weight, moisture rating etc and then Aisles has variables like max weight it can handle, moisture near it, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to run some algorithm which classifies which products should go in which aisles based on product and aisles variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;examples if product A has weight 10kg, moisture it can handle x, it should go to Aisle which can handle that weight and has that moisture rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided I have same variables across product and aisles, how can I solve my problem?&lt;/p&gt;&#xA;" OwnerUserId="44167" LastEditorUserId="29575" LastEditDate="2018-01-05T13:54:24.017" LastActivityDate="2018-01-10T12:20:52.390" Title="Which algorithm to use to match two categories with n dimensions" Tags="|machine-learning|classification|data-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26303" PostTypeId="1" CreationDate="2018-01-05T01:19:50.887" Score="2" ViewCount="950" Body="&lt;p&gt;I want to use the result of my PCA as an input for my LSTM model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I began by Applying the MinMaxScaler and then did the PCA, (then I reshaped my data of course) : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sc = MinMaxScaler(feature_range=(0, 1))&#xA;data = sc.fit_transform(data)&#xA;pca = PCA()&#xA;data = pca.fit_transform(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is, this give me a data between -1,23 and 1,33.&#xA;I know that the input of an LSTM must be between range 0 and 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what can I do ? Should I apply one more time a MinMaxScaler on &quot;data&quot; ?&lt;/p&gt;&#xA;" OwnerUserId="44169" LastActivityDate="2018-01-05T01:19:50.887" Title="Re-scale data after PCA for an LSTM?" Tags="|python|deep-learning|rnn|pca|lstm|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26304" PostTypeId="1" AcceptedAnswerId="26306" CreationDate="2018-01-05T01:27:04.647" Score="1" ViewCount="779" Body="&lt;p&gt;I have a cosine similarity matrix where I want to adjust it to inputto t-sne. I fond the following explanation in a FAQ. As mentioned there I have made the diagonals to zero. what does it mean by &lt;code&gt;symmetrize the pairwise similarity matrix, and normalize it to sum up to one&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can I use a pairwise similarity matrix as input into t-SNE?&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Yes you can! For instance, we successfully applied t-SNE on a dataset&#xA;  of word association data. Download the Matlab implementation, make&#xA;  sure the diagonal of the pairwise similarity matrix contains only&#xA;  zeros, symmetrize the pairwise similarity matrix, and normalize it to&#xA;  sum up to one. You can now use the result as input into the tsne_p.m&#xA;  function.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="38395" LastEditorUserId="26562" LastEditDate="2018-01-05T03:04:46.093" LastActivityDate="2018-01-05T03:25:27.430" Title="Converting similarity matrix before inputting to t-sne" Tags="|machine-learning|text-mining|tsne|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26305" PostTypeId="1" CreationDate="2018-01-05T01:29:29.850" Score="2" ViewCount="376" Body="&lt;p&gt;Is there a way to check the result of A/B test to determine the better one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group A has 100 views and 10 clicks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Group B has 10 views and 2 clicks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to chose which one produces more clicks ?&lt;/p&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2020-02-08T03:03:15.503" Title="How to verify A/B test" Tags="|statistics|ab-test|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26306" PostTypeId="2" ParentId="26304" CreationDate="2018-01-05T03:06:49.050" Score="1" Body="&lt;p&gt;You need to scale the values by some constant factor so the sum of every entry in the matrix results in 1.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can achieve this by using &lt;code&gt;mat /= mat.sum()&lt;/code&gt;, where &lt;code&gt;mat&lt;/code&gt; is your matrix.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-01-05T03:25:27.430" LastActivityDate="2018-01-05T03:25:27.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26307" PostTypeId="1" CreationDate="2018-01-05T04:17:54.863" Score="1" ViewCount="139" Body="&lt;p&gt;Given a set of lines, is there a way to train to extract geometric shapes. For example, the picture on left has some blue lines (with red endpoints). How can I train to extract shapes like on the right side? The output must be different shapes with each having a set of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I searched for related research papers. I could not find any paper with this exact problem, though some of them came close to this with RNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uQd39.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uQd39.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44145" LastActivityDate="2018-01-05T11:08:08.853" Title="Shape extracting on 2D geometric data" Tags="|neural-network|rnn|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26308" PostTypeId="1" AcceptedAnswerId="26312" CreationDate="2018-01-05T04:27:19.680" Score="14" ViewCount="64764" Body="&lt;p&gt;The dataframe contains&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; df&#xA;        A          B            C&#xA;A&#xA;196512  196512    1325  12.9010511000000&#xA;196512  196512  114569  12.9267705000000&#xA;196512  196512  118910  12.8983353775637&#xA;196512  196512  100688  12.9505091000000&#xA;196795  196795   28978  12.7805170314276&#xA;196795  196795   34591  12.8994111000000&#xA;196795  196795   13078  12.9135746000000&#xA;196795  196795   24173  12.8769653100000&#xA;196341  196341  118910  12.8983353775637&#xA;196341  196341  100688  12.9505091000000&#xA;196641  196641   28972  12.7805170314276&#xA;196641  196641   34591  12.8994111000000&#xA;196346  196341  118910  12.8983353775637&#xA;196346  196341  100688  12.9505091000000&#xA;196646  196641   28980  12.7805170314276&#xA;196646  196641   34591  12.8994111000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to get minimum value for each group and display using the following code,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.columns = ['a','b','c']&#xA;df.index = df.a.astype(str)&#xA;dd=df.groupby('a').min()['c']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it gives the result&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;196512    12.7805170314276&#xA;196795    12.7805170314276&#xA;196341    12.7805170314276&#xA;196346    12.7805170314276&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but after grouping, I want to get the row with the minimum 'c' value, grouped by column 'a' and display that full matching row in result &#xA;like,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;196512    118910      12.8983353775637  &#xA;196795     28978      12.7805170314276&#xA;196341     28972      12.7805170314276&#xA;196346     28980      12.7805170314276&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43680" LastEditorUserId="29575" LastEditDate="2018-08-25T18:06:41.900" LastActivityDate="2019-02-26T19:45:43.353" Title="after grouping to minimum value in pandas, how to display the matching row result entirely along min() value" Tags="|python|pandas|dataframe|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26309" PostTypeId="2" ParentId="26307" CreationDate="2018-01-05T04:48:20.373" Score="0" Body="&lt;p&gt;I haven't attempted this, but I'm pretty sure it would work reasonably well. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Identify all points that are connected to more than two lines. We know these must be vertex points.&lt;/li&gt;&#xA;&lt;li&gt;For for all the points that are attached to only one line, identify their nearest neighbor and draw a line between them (this would connect that hole on the right wall of your house). All points should now be either labeled as a vertex or touching only two lines.&lt;/li&gt;&#xA;&lt;li&gt;For each point touching two lines, calculate the angle between the two lines. &lt;/li&gt;&#xA;&lt;li&gt;Find the min and max of these angles. &lt;/li&gt;&#xA;&lt;li&gt;Use these as prototypes to classify angles into two proximity clusters using agglomerative clustering. Alternatively, pick a fixed angle value as your threshold (maybe around 120 degrees) and use that as the decision boundary for this classification.&lt;/li&gt;&#xA;&lt;li&gt;The angles that clustered with the min correspond to the missing vertices.&lt;/li&gt;&#xA;&lt;li&gt;Infer the vertex-vertex connectivity graph through the non-vertex points.(i.e. find all paths between pairs of vertices that do not pass through a third vertex).&lt;/li&gt;&#xA;&lt;li&gt;Draw the graph. &lt;/li&gt;&#xA;&lt;li&gt;Find the cycle's in the graph and color each one separately.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Everything doesn't have to be a big fancy neural network :)&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-05T11:08:08.853" LastActivityDate="2018-01-05T11:08:08.853" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26311" PostTypeId="2" ParentId="26308" CreationDate="2018-01-05T07:40:09.857" Score="5" Body="&lt;p&gt;First check your data. You can't get value &lt;strong&gt;12.780517&lt;/strong&gt; for index &lt;strong&gt;196341&lt;/strong&gt;. This value will be for index &lt;strong&gt;196641&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, to get column &lt;strong&gt;'b'&lt;/strong&gt; in your result, use &lt;strong&gt;pd.merge&lt;/strong&gt;. Don't make column &lt;strong&gt;'a'&lt;/strong&gt; as index. Leave it as it is.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df = pd.DataFrame({'a':[196512, 196512, 196512, 196512, 196795, 196795, 196795, 196795, 196341, 196341, 196641, 196641, 196346, 196346, 196646, 196646],'b':[1325 , 114569 , 118910 , 100688 , 28978 ,34591 , 13078 ,  24173 , 118910 , 100688 , 28972 , 34591 , 118910 , 100688 , 28980 , 34591 ],'c':[12.9010511000000 ,12.9267705000000 ,12.8983353775637 ,12.9505091000000 ,12.7805170314276 ,12.8994111000000 ,12.9135746000000 ,12.8769653100000 ,12.8983353775637 ,12.9505091000000 ,12.7805170314276 ,12.8994111000000 ,12.8983353775637 ,12.9505091000000 ,12.7805170314276 ,12.8994111000000 ]})&#xA;&amp;gt;&amp;gt;&amp;gt; df1 = df.groupby(['a'])['c'].min()&#xA;&amp;gt;&amp;gt;&amp;gt; df1&#xA;a&#xA;196341    12.898335&#xA;196346    12.898335&#xA;196512    12.898335&#xA;196641    12.780517&#xA;196646    12.780517&#xA;196795    12.780517&#xA;Name: c, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;df1 is a series type object here. Convert it to dataframe.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df1 = pd.DataFrame(df1, columns = ['c'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, make index of df1 as column &lt;strong&gt;'a'&lt;/strong&gt; and change index.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df1['a'] = df1.index&#xA;&amp;gt;&amp;gt;&amp;gt; df1.index = range(df1.shape[0])&#xA;&amp;gt;&amp;gt;&amp;gt; df1&#xA;       c       a&#xA;0  12.898335  196341&#xA;1  12.898335  196346&#xA;2  12.898335  196512&#xA;3  12.780517  196641&#xA;4  12.780517  196646&#xA;5  12.780517  196795&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you have to join this &lt;strong&gt;df1&lt;/strong&gt; with &lt;strong&gt;df&lt;/strong&gt; on both columns &lt;strong&gt;'a'&lt;/strong&gt; and &lt;strong&gt;'c'&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; pd.merge(df,df1, on = ['a','c'])&#xA;    a       b          c&#xA;0  196512  118910  12.898335&#xA;1  196795   28978  12.780517&#xA;2  196341  118910  12.898335&#xA;3  196641   28972  12.780517&#xA;4  196346  118910  12.898335&#xA;5  196646   28980  12.780517 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And you have your output as desired!&lt;/p&gt;&#xA;" OwnerUserId="44139" LastEditorUserId="44139" LastEditDate="2018-01-05T08:40:39.053" LastActivityDate="2018-01-05T08:40:39.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26312" PostTypeId="2" ParentId="26308" CreationDate="2018-01-05T08:11:41.210" Score="11" Body="&lt;p&gt;You can do this. But I doubt the efficiency.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&gt;&gt; import pandas as pd&#xA;&gt;&gt; df = pd.DataFrame({'a':[1,1,3,3],'b':[4,5,6,3], 'c':[1,2,3,5]})&#xA;&gt;&gt; df&#xA;   a  b  c&#xA;0  1  4  1&#xA;1  1  5  2&#xA;2  3  6  3&#xA;3  3  3  5&#xA;&gt;&gt; df[df['c'].isin(df.groupby('a').min()['c'].values)]&#xA;   a  b  c&#xA;0  1  4  1&#xA;2  3  6  3&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-01-05T08:11:41.210" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26314" PostTypeId="1" CreationDate="2018-01-05T11:14:24.750" Score="2" ViewCount="2547" Body="&lt;p&gt;I'm new to using the keras framework. I have read some examples about how to construct deep learning models with the Sequential and Graph classes in keras. However, I see that, independently if I use Sequential or Graph, it is assumed that each hidden unit of a layer is fully connected with all the hidden units of the other layer, isn't it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to construct a deep feed forward network that it is not fully connected, for instance the first hidden unit of the second layer is not connected to the second unit of the third layer...etc, even I want add connections (skip connections) between hidden units that belong to non-consecutive layers, how can I implement this in keras?&lt;/p&gt;&#xA;" OwnerUserId="44189" LastEditorUserId="44189" LastEditDate="2018-01-05T18:47:02.333" LastActivityDate="2018-04-05T21:17:31.870" Title="How implement a feed forward network with arbitrary node connections in keras?" Tags="|deep-learning|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26315" PostTypeId="2" ParentId="26314" CreationDate="2018-01-05T11:46:00.930" Score="0" Body="&lt;p&gt;You probably want to use the functional API.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Functional API&lt;/a&gt; has a lot of examples you can follow and used whenever you need to build more complicated models.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-05T11:46:00.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26316" PostTypeId="2" ParentId="26294" CreationDate="2018-01-05T12:13:15.110" Score="1" Body="&lt;p&gt;There is a rule called &lt;em&gt;there is no free launch&lt;/em&gt;. It means that there isn't a learning algorithm that solves all the problems. You as a machine learning practitioner should decide when and how to use which algorithm. Suppose that you want to recognize faces. This problem is a learning problem which if you increase the number of training data, you will get better results. In these cases neural nets and deep nets are highly recommended. In this case it is not logical to use &lt;code&gt;non-linear SVM&lt;/code&gt; because it will be so costly and you may not even get good answers. the reason is that deep nets cares about local patterns but &lt;code&gt;SVM&lt;/code&gt; considers all the input pattern simultaneously. Actually in your case, I guess your data is categorical. For categorical data, people often use decision trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate an example, once I decided to train a simple MLP to distinguish whether an input pattern is in correct position, to solve 8-queen problem. I solve the game using &lt;code&gt;Genetic algorithm&lt;/code&gt; and made data for training the net. The data I brought to net was categorical in some extant. I used it and the net was so good for the trained data, but input features similar to training data which were a bit different had bad recall rate. I trained a decision tree, I get so much better result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm depends on your task and your input features. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-05T12:13:15.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26317" PostTypeId="2" ParentId="371" CreationDate="2018-01-05T12:25:33.010" Score="0" Body="&lt;p&gt;There are several approaches to creating a parser that would parse plain text message into SQL. For example, you can create a grammar-based parser and use an NLP algorithm to build the structured query. If you already have plenty of parsed messages from one domain (like e-commerce) - you can try a Machine Learning approach and use it for your further parsing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I think the best approach is to combine a grammar-based parser for text-to-SQL translation, and ML to complement the rule-based grammar by fixing the syntax, eradicating typos, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learn more about the different approaches &lt;a href=&quot;https://friendlydata.io/blog/machine-learning-vs-rule-based-systems&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44193" LastEditorUserId="29575" LastEditDate="2018-01-05T14:12:22.037" LastActivityDate="2018-01-05T14:12:22.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26318" PostTypeId="2" ParentId="9262" CreationDate="2018-01-05T12:50:52.027" Score="4" Body="&lt;p&gt;This trick avoids conditional code and may therefore provide better performance.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;def KL(P,Q):&#xA;&quot;&quot;&quot; Epsilon is used here to avoid conditional code for&#xA;checking that neither P nor Q is equal to 0. &quot;&quot;&quot;&#xA;     epsilon = 0.00001&#xA;&#xA;     # You may want to instead make copies to avoid changing the np arrays.&#xA;     P = P+epsilon&#xA;     Q = Q+epsilon&#xA;&#xA;     divergence = np.sum(P*np.log(P/Q))&#xA;     return divergence&#xA;&#xA;# Should be normalized though&#xA;values1 = np.asarray([1.346112,1.337432,1.246655])&#xA;values2 = np.asarray([1.033836,1.082015,1.117323])&#xA;&#xA;# Note slight difference in the final result compared to Dawny33&#xA;print KL(values1, values2) # 0.775278939433&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44192" LastActivityDate="2018-01-05T12:50:52.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26319" PostTypeId="1" CreationDate="2018-01-05T12:54:06.900" Score="1" ViewCount="300" Body="&lt;p&gt;In hadoop , if i want to read a file from client , the class DFSInputStream enters in action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About this, my textbook says &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The DFSInputStream also verifies checksums for the data transferred to it from the datanode.&#xA;If a corrupted block is found, the DFSInputStream attempts to read a replica of the block from another datanode; it also reports the corrupted block to the namenode.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are :&#xA;1) What's the meaning in saying that the data is trasferred to DFSInputStream ?&#xA; Am i wrong, or the client communicates directly with the datanode ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)If the checksum shows that the transfer of a block has going wrong, why does the DFSInputStream request this block from another datanode instead of trying again with the datanode that generated the corrupted block ?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastActivityDate="2018-01-05T12:54:06.900" Title="Hadoop - checksum while reading file from client" Tags="|bigdata|apache-hadoop|" AnswerCount="0" CommentCount="0" ClosedDate="2018-01-05T17:40:49.093" ContentLicense="CC BY-SA 3.0" />
  <row Id="26320" PostTypeId="1" CreationDate="2018-01-05T13:40:27.033" Score="2" ViewCount="41" Body="&lt;p&gt;The patch for the &lt;a href=&quot;https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Meltdown&lt;/a&gt; vulnerability disables speculative execution, which will impact all processing activities. The degree of impact is highly dependent on the type of processing being done. Is there hard data or experience of how machine learning and data processing will be impacted in measurable terms? &lt;/p&gt;&#xA;" OwnerUserId="27911" LastActivityDate="2018-01-05T14:30:30.453" Title="Meltdown patch impact on data processing speeds" Tags="|processing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26321" PostTypeId="2" ParentId="26290" CreationDate="2018-01-05T13:59:53.293" Score="1" Body="&lt;p&gt;Images are two dimensional signals. The use of &lt;code&gt;Conv1D&lt;/code&gt; is for one dimensional signals like voice and sound. CNNs are good for these signals too because of taking care of local input patterns. Definitely there are standard one dimensional filters which are most used in signal processing like &lt;em&gt;high pass filters&lt;/em&gt; and &lt;em&gt;low pass filters&lt;/em&gt; which are so much popular. In order to show you an example take a look at the following figure which illustrates the convolution of two signals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kTBiy.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kTBiy.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-05T13:59:53.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26322" PostTypeId="2" ParentId="25427" CreationDate="2018-01-05T14:17:28.873" Score="0" Body="&lt;p&gt;If I get the point right based on the title of the question, in your code you are not making multi layer perceptron. You have tried to make somehow a convolutional network. In MLPs you just have to stack dense layers. Do something like the following code snippet which stacks just dense layers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(keras.layers.Dense(10))&#xA;model.add(keras.layers.Dense(10))&#xA;model.add(keras.layers.Dense(10))&#xA;model.add(keras.layers.Dense(10, activation = 'softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-05T14:17:28.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26323" PostTypeId="1" AcceptedAnswerId="26867" CreationDate="2018-01-05T14:18:30.227" Score="1" ViewCount="139" Body="&lt;p&gt;Excuse me if this is a simple question (I'm a newbie to data science).&#xA;I am trying to implement a Naive Bayes classifier in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have implemented a standard Naive Bayes classifier (Bernoulli) and this went very well.&#xA;Suppose I have 50,000 samples where the first half is labelled '0' and the second half '1'.&#xA;Then the shape of the X_train matrix will be (#number_of_unique_words/features, 50000) and shape of y_train (50000,).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From how I understand Naive Bayes, I can now also join the first 25000 documents to represent one big document with label '0' and one big document with label '1'. However, this gives me very different results when I try to predict the test set.&#xA;Is my understanding of Naive Bayes correct or is my implementation just wrong?&lt;/p&gt;&#xA;" OwnerUserId="44198" LastEditorUserId="44198" LastEditDate="2018-01-05T14:45:12.133" LastActivityDate="2018-07-19T23:03:59.100" Title="Can you reduce many Naive Bayes training samples to a few big ones?" Tags="|scikit-learn|naive-bayes-classifier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26324" PostTypeId="2" ParentId="26320" CreationDate="2018-01-05T14:30:30.453" Score="2" Body="&lt;p&gt;It looks like the Tech Web Sites that reported &lt;a href=&quot;https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Meltdown&lt;/a&gt;, may have overblown the negative impacts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;https://security.googleblog.com/2018/01/more-details-about-mitigations-for-cpu_4.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google&lt;/a&gt;, in their cloud infrastructure, there seems to be little to worry about:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends on the rate of system calls made by an application. On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.techspot.com/article/1554-meltdown-flaw-cpu-performance-windows/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Techspot &lt;/a&gt; ran some benchmarks on Windows 10, and they concluded...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Desktop users have little to worry about in terms of performance loss, particularly gamers. We've yet to test older CPUs, but given the type of workloads we’re seeing impacted by the patch, I don’t think there’s going to be an issue with any desktop hardware, but we’ll certainly report back if there is.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2018-01-05T14:30:30.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26325" PostTypeId="1" CreationDate="2018-01-05T14:41:44.903" Score="3" ViewCount="7738" Body="&lt;p&gt;I'm dealing with an imbalanced dataset and as usual it's very easy to obtain a high accuracy, but the recall on the less frequent class is very low. I would like to improve on the false negative of the less frequent class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Focusing for concreteness on the sklearn Random Forest, one possible strategy is to set a class_weight penalizing the errors on the less frequent class and scoring with a sklearn scoring function as ROC.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid = GridSearchCV(pipe, cv=number_of_cross_validations, n_jobs=1, param_grid=param_grid , scoring = scorer)&#xA;classifier = grid.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;pipe&lt;/code&gt; is pipeline containing the RandomForestClassifier and &lt;code&gt;param_grid&lt;/code&gt; contains different &lt;code&gt;class_weight&lt;/code&gt; to try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is probably related to &lt;a href=&quot;https://datascience.stackexchange.com/questions/21961/custom-metrics-for-unbalanced-classes-problem-in-randomforest-or-svm?rq=1&quot;&gt;this question&lt;/a&gt;, indeed &lt;code&gt;class_weight&lt;/code&gt; alone seems to not be enough to lower significantly the false negative. As an extreme example, if I set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; class_weight = {0: 0.0000001, 1: 0.9999999} &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(where 1 is the class with less instances, with a ratio 1:50), I would expect a final classifier predicting nearly always 1, since every time it makes an error the cost to pay is very high. But that's not what happen, the fitted classifier continue to make a significant number of false negative. Why is it so?&lt;/p&gt;&#xA;" OwnerUserId="31603" LastEditorUserId="29575" LastEditDate="2018-01-05T15:00:04.583" LastActivityDate="2018-11-08T22:53:07.920" Title="Class weight ineffective in sklearn" Tags="|scikit-learn|cross-validation|grid-search|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26327" PostTypeId="1" AcceptedAnswerId="26330" CreationDate="2018-01-05T15:47:32.177" Score="1" ViewCount="302" Body="&lt;p&gt;I read a paper on Deep Neural Networks Compression (link: &lt;a href=&quot;https://openreview.net/forum?id=SkhQHMW0W&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://openreview.net/forum?id=SkhQHMW0W&lt;/a&gt;) and came across a term &quot;gradient exchange&quot;, I tried making sense out of it but couldn't exactly understand what it intuitively refers to. What does the term gradient exchange mean?&lt;/p&gt;&#xA;" OwnerUserId="43491" LastEditorUserId="43491" LastEditDate="2018-01-05T16:18:15.507" LastActivityDate="2018-01-05T17:16:26.400" Title="Gradient Exchange" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26328" PostTypeId="1" CreationDate="2018-01-05T16:26:58.993" Score="2" ViewCount="197" Body="&lt;p&gt;I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I'm unsure how they've used the Bernstein inequality on page 6 to conclude $\mu_{n}(\mathcal{H},1) \leq c^{-1} \sqrt{k \mathrm{ln}(m)/n}$. Could someone please elaborate on this as it seems Bernsteins inequality seems to measure whp the deviation of a sum of random variables from it's mean. Where as the regret bound $\mu_{n}(\mathcal{H},1)$ is defined as the expected regret from the empirically best policy from the absolute best policy. Could someone fill in the detail?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can we get any reasonable estimate for the constant $c$ if i was to try and implement this in practice?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd really appreciate any help, Thanks. &lt;/p&gt;&#xA;" OwnerUserId="43865" LastActivityDate="2018-01-05T16:26:58.993" Title="Epoch greedy algorithm for contextual bandits" Tags="|machine-learning|statistics|reinforcement-learning|probability|randomized-algorithms|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26330" PostTypeId="2" ParentId="26327" CreationDate="2018-01-05T17:16:26.400" Score="2" Body="&lt;p&gt;&lt;em&gt;Gradient exchange&lt;/em&gt; occurs in distributed learning systems that perform gradient descent, when one part of the distributed system need to use the gradient values from another part in order to complete a task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, you may distribute a large data set between multiple nodes, and want to calculate a gradient descent step as part of optimisation. One way to do so is calculate a subset of batch gradients on each node and collate them at a single node in order to alter parameters synchronously. This means it is necessary to fetch gradients from all nodes into a single node so that a combined gradient for some weight parameters can be calculated and the parameters updated consistently in the update step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Gradient exchange&lt;/em&gt; is just a term to describe that event - node A needs some gradients that node B has calculated, so they are requested (or pushed) and have to travel between the nodes. This is a relatively slow I/O process - it is necessary for distributed system to work, but for high performance you want to minimise time spent moving the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other data (such as the parameters) also needs to be shared between nodes. This particular piece of data regarding the gradients is singled out for the paper as the authors have discovered a way to compress it significantly without losing performance of the learning algorithm. This is partly because gradients can be treated approximately in the first place. Many learning algorithms further adjust or normalise gradients after they have been calculated, so using super-precise values is not as important as you might think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There may also be clever ways of splitting the update work so that each node only needs some of the gradients and only updates some of the parameters at each step. That will keep node CPU busy, possibly at the expense of more complicated communications. I do not know the details of any optimised distributed learning system in order to tell you the precise data exchanges and optimisations taking place. There are likely to be a few variations possible, depending on the framework and which algorithm is being implemented.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-01-05T17:16:26.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26332" PostTypeId="1" CreationDate="2018-01-05T18:23:42.300" Score="2" ViewCount="358" Body="&lt;p&gt;After training Logistic regression on training data, getting each test sample to algorithm computes weighted sum of sample features. If this is greater than zero, we know the sample is from negative class and otherwise, sample is positive. I want to know why we need to calculate the logistic function that map our weighted sum to range zero and one. One reason can be that we want to interpret the results as probability. are there any other reasons?  &lt;/p&gt;&#xA;" OwnerUserId="33245" LastEditorUserId="28175" LastEditDate="2018-02-04T22:30:09.090" LastActivityDate="2018-03-07T00:35:45.103" Title="What is the role of logistic function in logistic regression?" Tags="|machine-learning|classification|logistic-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26333" PostTypeId="1" CreationDate="2018-01-05T18:40:33.767" Score="77" ViewCount="217539" Body="&lt;p&gt;I am trying to convert a list of lists which looks like the following into a Pandas Dataframe&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[['New York Yankees ', '&quot;Acevedo Juan&quot;  ', 900000, ' Pitcher\n'], &#xA;['New York Yankees ', '&quot;Anderson Jason&quot;', 300000, ' Pitcher\n'], &#xA;['New York Yankees ', '&quot;Clemens Roger&quot; ', 10100000, ' Pitcher\n'], &#xA;['New York Yankees ', '&quot;Contreras Jose&quot;', 5500000, ' Pitcher\n']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am basically trying to convert each item in the array into a pandas data frame which has four columns. What would be the best approach to this as pd.Dataframe does not quite give me what I am looking for.&lt;/p&gt;&#xA;" OwnerUserId="44204" LastEditorUserId="381" LastEditDate="2018-06-05T18:51:20.273" LastActivityDate="2019-10-30T20:53:41.503" Title="Convert a list of lists into a Pandas Dataframe" Tags="|pandas|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26335" PostTypeId="2" ParentId="26333" CreationDate="2018-01-05T18:46:40.790" Score="89" Body="&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;data = [['New York Yankees', 'Acevedo Juan', 900000, 'Pitcher'], &#xA;        ['New York Yankees', 'Anderson Jason', 300000, 'Pitcher'], &#xA;        ['New York Yankees', 'Clemens Roger', 10100000, 'Pitcher'], &#xA;        ['New York Yankees', 'Contreras Jose', 5500000, 'Pitcher']]&#xA;&#xA;df = pd.DataFrame.from_records(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="381" LastEditorUserId="79761" LastEditDate="2019-10-06T21:09:22.053" LastActivityDate="2019-10-06T21:09:22.053" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="26336" PostTypeId="2" ParentId="26332" CreationDate="2018-01-05T18:56:01.190" Score="1" Body="&lt;p&gt;Logistic and other activation functions are used to add non linearity to the neural network models.&#xA;In logistic model which uses just one neuron it may have a nice interpretation. To interpret what it does other than probability of belonging to each class, its sign shows whether your input data is on the left of the decision boundary or on the right of that. Also, if it has great value, it shows more confidence of belonging to that class.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-05T18:56:01.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26337" PostTypeId="2" ParentId="26314" CreationDate="2018-01-05T19:17:18.457" Score="1" Body="&lt;p&gt;A possible solution can be found at &lt;a href=&quot;https://stackoverflow.com/questions/48111217/how-implement-a-feed-forward-network-with-arbitrary-node-connections-in-keras?noredirect=1#comment83198128_48111217&quot;&gt;https://stackoverflow.com/questions/48111217/how-implement-a-feed-forward-network-with-arbitrary-node-connections-in-keras?noredirect=1#comment83198128_48111217&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The suggestion is simply work with many little layers and make as many skip connections as you want. Many parallel layers with 1 node represent well a single layer with many nodes. So you can, for instance, create lots of Dense(1) layers and treat them as nodes. Then you connect them in any way you like.&quot;&lt;/p&gt;&#xA;" OwnerUserId="44189" LastActivityDate="2018-01-05T19:17:18.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26338" PostTypeId="1" AcceptedAnswerId="26339" CreationDate="2018-01-05T19:51:00.210" Score="1" ViewCount="153" Body="&lt;p&gt;For example: &#xA;Here is a textual input: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;ALL imagery SINCE 1952&#xA;20 MULE T&#xA;aerial BOOSTER&#xA;&amp;amp; Multi-Purpose Neutralizer.MAY BE â€œAHMFLL |-E SWALL &quot;&quot;&quot;&quot;&quot;&quot; -5 NETWT4LBS1 DZ (65 OZ) 1.84 kg&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&#xA;Output desired:&#xA;4 LBS&#xA;65 OZ&#xA;1.84 kg&#xA;&quot;&lt;/p&gt;&#xA;" OwnerUserId="44207" LastEditorUserId="44207" LastEditDate="2018-01-12T00:21:40.883" LastActivityDate="2018-01-12T00:21:40.883" Title="How to extract specific parts of text from a string?" Tags="|nlp|text-mining|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ClosedDate="2018-01-07T01:38:27.173" ContentLicense="CC BY-SA 3.0" />
  <row Id="26339" PostTypeId="2" ParentId="26338" CreationDate="2018-01-05T19:58:04.750" Score="0" Body="&lt;p&gt;It is a specific pattern which can be detected by numbers and after that a unit name with two or three letters.&#xA;In the other words, detect numbers first, and then find numbers which after of them there is two or three letters.&#xA;Moreover, you can check these letters are in a predefined list of units or not.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-05T19:58:04.750" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26340" PostTypeId="1" CreationDate="2018-01-05T21:02:46.697" Score="0" ViewCount="859" Body="&lt;p&gt;Currently I have a data-set with roughly 7 types of nominal data; these are things like &quot;workclass&quot; or &quot;marital status&quot;. It is my understanding it is best to convert nominal data like this into numeric data. So, what I've done so far, as a simple experiment, is make up a dictionary for each nominal data type, so in our example of &quot;marital status&quot;, it would be &quot;Divorced&quot;:0, &quot;Married&quot;:1, &quot;Widowed&quot;:3, &quot;Separated&quot;:4. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as I understand, there are some problems with this approach, as it implies that Divorced-&gt;Married are more related than Divorced-&gt;Widowed. I tried to arrange this data into a logical pattern, but for a lot of it its impossible. So here are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is there a good approach to do this without running into the above problem, specifically in Python? I'm using Pandas DF's at the moment, and I saw an example online of something called Binary Vectorizers, but I'm not sure if that won't introduce too many attributes/columns to my data set. &lt;/li&gt;&#xA;&lt;li&gt;When I convert a nominal value to an interval one, do I replace the  original (For example Workclass) with the Workclass_Number data? Do I only enter the number data into my models?&lt;/li&gt;&#xA;&lt;li&gt;Are there any guidelines on the numbers that should be used? For example I used simple intervals of 1 for all of my data, starting at 0. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44211" LastEditorUserId="29575" LastEditDate="2018-01-05T22:17:46.203" LastActivityDate="2018-01-06T01:11:22.123" Title="Converting nominal data to numeric - is using dictionaries the right approach?" Tags="|python|scikit-learn|pandas|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26341" PostTypeId="2" ParentId="12909" CreationDate="2018-01-05T21:30:19.063" Score="8" Body="&lt;p&gt;From article on &lt;a href=&quot;https://docs.aws.amazon.com/machine-learning/latest/dg/training-ml-models.html&quot; rel=&quot;noreferrer&quot;&gt;Amazon Machine Learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process of training an ML model involves providing an ML algorithm (that is, the learning algorithm) with training data to learn from. &lt;code&gt;The term ML model refers to the model artifact that is created by the training process.&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="44215" LastActivityDate="2018-01-05T21:30:19.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26342" PostTypeId="1" CreationDate="2018-01-05T21:47:01.173" Score="2" ViewCount="289" Body="&lt;p&gt;In TD(0) learning where the value function is given by $V(s) = w^T\phi(s)$ where $w$ is a weight vector and $\phi$ is a feature map, the weight update is given by $w_{t+1} = w_{t} + \eta\delta_{t+1}\phi(s_{t})$, where $\eta$ is the learning rate and $\delta$ is the temporal difference error. The temporal difference error is given by $\delta_{t+1} = r_{t}+\gamma V_{t+1} -V_{t}$, where $r$ is the reward and $\gamma$ is the discount factor.  Note that the weight update is proportional to the old state features. This update rule can be thought of as trying to make $V_{t}$ closer to $r_t +\gamma V_{t+1}$ to make the value function more self consistent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is also possible to make the value of $\gamma V_{t+1}$ closer to $V_{t}-r_{t}$ by making the weight update rule $w_{t+1} = w_{t} - \frac{\eta}{\gamma}\delta_{t+1}\phi(s_{t+1})$. It is also possible to do something in the middle like $w_{t+1} = w_{t} + \frac{\eta}{2}\delta_{t+1}(\phi(s_{t})-\frac{1}{\gamma}\phi(s_{t+1}))$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these viable weight update rules? If not, why? What properties do the learning algorithms corresponding to these update rules have?&lt;/p&gt;&#xA;" OwnerUserId="44214" LastActivityDate="2018-11-06T06:00:35.343" Title="Reasoning for temporal difference update rule" Tags="|machine-learning|reinforcement-learning|online-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26343" PostTypeId="2" ParentId="12909" CreationDate="2018-01-05T21:48:31.887" Score="8" Body="&lt;p&gt;I like the Machine Learning definition given by &lt;strong&gt;Tom Mitchell&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;A computer program is said to learn from &lt;strong&gt;experience&lt;/strong&gt; &lt;em&gt;E&lt;/em&gt; with&#xA;respect to some class of &lt;strong&gt;tasks&lt;/strong&gt; &lt;em&gt;T&lt;/em&gt; and &lt;strong&gt;performance&lt;/strong&gt; measure &lt;em&gt;P&lt;/em&gt;&#xA;if its performance at tasks in T, as measured by P, improves with&#xA;experience E.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So, given this definition, I should say that a model is the acquired experience after doing some task T.&lt;/p&gt;&#xA;" OwnerUserId="31009" LastEditorUserId="31009" LastEditDate="2021-01-24T01:38:32.657" LastActivityDate="2021-01-24T01:38:32.657" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="26344" PostTypeId="2" ParentId="26223" CreationDate="2018-01-05T21:55:24.330" Score="7" Body="&lt;p&gt;The reason is your learning rate alpha is too large for this optimization problem. Start with a really small value (&amp;lt; 0.000001) and you will observe a decrease in your cost function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that when the learning rate is too large, the gradient descent algorithm will miss the global minimum (global because MSE cost function is convex) and will diverge. This is why you have observed still increasing cost function values with alpha = 0.001.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-01-05T21:55:24.330" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26346" PostTypeId="2" ParentId="26340" CreationDate="2018-01-06T01:11:22.123" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Usually, when dealing with nominal attributes, you want to use the binary vectoriser approach. Unless you have a very large number of nominal values, it typically doesn't matter that you're adding a few attributes. If you do have a huge amount of nominal values, you can try things like dimensionality reduction on the binary attributes (although this reduces interpretability) or joining some nominal values together.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the nominal attribute is discrete but ordered (called an &lt;em&gt;ordinal&lt;/em&gt; attribute) -- for example, the severity of a cancer (stage 1, 2, 3 or 4) --&#xA;then you can use whatever numbers you feel are appropriate. It really depends on your domain knowledge, and the choice of model. For some models like decision trees, it doesn't matter, but for others like logistic regression the choices of the numbers can be influential. I would recommend just continuing as you have been by using intervals of 1, unless your knowledge of the domain says that particular adjacent nominal values should be further or closer than the others. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-01-06T01:11:22.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26347" PostTypeId="1" CreationDate="2018-01-06T01:15:49.937" Score="0" ViewCount="5024" Body="&lt;p&gt;I need to calculate the compound annual growth rate for two periods of time: 2014 &amp;amp; 2009. I'm currently working with Excel, I have two columns with data for each year (One column with only data of 2009 and the other only with data of 2014), where each row contains info for 2009 &amp;amp; 2014 for certain category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been trying to calculate the CAGR between the data for each row, but I can't seem to find a function to do precisely that in &quot;R&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a package with a growth.rate function for time series, but it doesn't take my data in the order I need (it uses bayes method for velocity of growth, which I don´t need). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any package in &quot;R&quot; where I can get especially that rate? Any tips in how I can accommodate the data so that R reads row per row and calculates like that?&lt;/p&gt;&#xA;" OwnerUserId="44212" LastEditorUserId="29575" LastEditDate="2018-01-17T14:48:01.733" LastActivityDate="2021-07-30T03:52:49.060" Title="Compound Annual Growth Rate Function in R Statistics" Tags="|r|dataset|excel|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26348" PostTypeId="1" CreationDate="2018-01-06T04:42:41.403" Score="5" ViewCount="2561" Body="&lt;p&gt;In the textbook im reading about Deep Learning, i found :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When designing features or algorithms for learning features, our goal&#xA;  is usually to separate the factors of variation that explain the&#xA;  observed data [...]&#xA;  &lt;strong&gt;the factors are usually not combined by multiplication&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What's the meaning of the bold phrase ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ps: what's your definition for &quot;factors of variation&quot; ?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="43656" LastEditDate="2018-01-06T04:56:50.810" LastActivityDate="2022-03-26T13:12:40.030" Title="Factors of Variation in Deep Learning" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26349" PostTypeId="2" ParentId="26333" CreationDate="2018-01-06T05:43:30.043" Score="7" Body="&lt;p&gt;You can just directly define it as a data frame as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;data = [['New York Yankees', 'Acevedo Juan', 900000, 'Pitcher'], &#xA;        ['New York Yankees', 'Anderson Jason', 300000, 'Pitcher'], &#xA;        ['New York Yankees', 'Clemens Roger', 10100000, 'Pitcher'], &#xA;        ['New York Yankees', 'Contreras Jose', 5500000, 'Pitcher']]&#xA;&#xA;data = pd.DataFrame(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26163" LastActivityDate="2018-01-06T05:43:30.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26351" PostTypeId="1" CreationDate="2018-01-06T09:20:02.170" Score="4" ViewCount="7906" Body="&lt;p&gt;Suppose I want to use the Gradient Descent algorithm.&#xA;I have a training set and a test set and I want to do the feature scaling with mean normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I use the same mean and variance for equivalent features in the 2 sets? Why?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="28175" LastEditDate="2018-01-10T12:41:50.720" LastActivityDate="2020-12-30T16:28:51.403" Title="Feature Scaling of Training Set and Test Set" Tags="|machine-learning|classification|deep-learning|supervised-learning|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26352" PostTypeId="2" ParentId="26348" CreationDate="2018-01-06T11:59:07.853" Score="3" Body="&lt;p&gt;Factors of variation are some factors which determine varieties in observed data. If that factors change, the behavior of the data will change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bold phrase said these factors are usually independent and one factor does not change by changing the value of the others.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-06T11:59:07.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26353" PostTypeId="1" CreationDate="2018-01-06T12:12:31.853" Score="1" ViewCount="192" Body="&lt;p&gt;In python i have been using treeinterpreter on 2d data. It has worked well and enabled me to look into the blackbox to understand what is going on. I am interested in the why than the predicted output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, a new set of data is given to me daily. It is the same categorised data every day but some of it is dynamic. This however has added a new dimension of time. It is not as important as the features / categories but still has an effect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way of dealing with this so i can still look into the tree / svm / etc....i am not too fussed which one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may be i need to stack the models in order to get the insight - but i am unsure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;example of the data on one day below - the price change and rating can changes daily. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Wine      Country       Area        Grape        Class    Year    Price   Rating&#xA;A         France        Burgundy    Pinot Noir   Grand    2014    +6%      94&#xA;B         France        Burgundy    Chardonnay   1er      2014    -1%      91&#xA;C         France        Burgundy    Pinot Noir   1er      2014    +4%      96&#xA;D         USA           California  Pinot Noir            2013    +1%      92&#xA;E         USA           California  Blend                 2014    +0.5%    88&#xA;F         USA           California  Chardonnay            2014    -4%      81&#xA;G         USA           California  Chardonnay            2013    -5%      91&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24259" LastActivityDate="2018-01-06T12:12:31.853" Title="treeinterpreter / feature selection with multidimensional data" Tags="|machine-learning|python|scikit-learn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26354" PostTypeId="1" CreationDate="2018-01-06T12:17:45.160" Score="1" ViewCount="39" Body="&lt;p&gt;Im new to Big Data, so please just dont blast me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain me why HDFS works&#xA;best when the file blocks are evenly spread across the cluster ?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastActivityDate="2022-08-26T05:43:04.860" Title="Hdfs Data Balance on Cluster" Tags="|bigdata|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26356" PostTypeId="1" CreationDate="2018-01-06T12:58:10.217" Score="1" ViewCount="111" Body="&lt;p&gt;I am new to big data and &lt;a href=&quot;https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Hadoop YARN&lt;/a&gt;. This picture show how Yarn runs applications:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Usv1C.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Usv1C.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is made to run.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that every task of a Yarn Application are run is the same datanode?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="29575" LastEditDate="2018-01-07T00:52:19.817" LastActivityDate="2018-01-07T00:52:19.817" Title="Tasks of a Yarn Process in Hadoop" Tags="|bigdata|apache-hadoop|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26358" PostTypeId="1" CreationDate="2018-01-06T17:39:06.580" Score="1" ViewCount="211" Body="&lt;p&gt;I’ve a data set with many Timestamp features like: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Employee duty report time&lt;/li&gt;&#xA;&lt;li&gt;Employee duty release time&lt;/li&gt;&#xA;&lt;li&gt;Employee work start time&lt;/li&gt;&#xA;&lt;li&gt;Employee work end time&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We currently have a Java application that uses this data to tell if it is legal for an employee to work on a given day or not. The work is not continuous, some days employee work only 2 hours and other days they work straight 8 hours based on various shift assignment rules.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We feed 15 days of data to a Java Rule Engine to find out if an employee is legal to work or not. The Java Rule Engine uses many rules like if an employee works 8 hours in a day then the employee must take minimum 12 hours of rest before starting to work again. From the Java Rule Engine I collected all these data into two sets one with legal to work and other is not legal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to use logistic regression to feed the data to classify if an employee is legal or not. What is the best approach to select the various timestamp features? Can this data set can be considered as a “time series” model?&lt;/p&gt;&#xA;" OwnerUserId="44244" LastEditorUserId="29575" LastEditDate="2018-01-06T18:28:33.813" LastActivityDate="2018-01-06T18:28:33.813" Title="Feature selection with many Time stamp data and Model classification" Tags="|machine-learning|time-series|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26359" PostTypeId="2" ParentId="26351" CreationDate="2018-01-06T18:23:43.667" Score="1" Body="&lt;p&gt;You do feature scaling for accelerating learning process. Features may have different scales. One maybe from 1 to 10 and one may be from -100 to 1000. Using normalization, you make the scale of them the same as each other, helps accelerate the learning process. You should find the mean and variance for each feature separately on your training data. then during training and testing each feature should be reduced by the corresponding mean and be divided by the corresponding standard deviation. So yes, for each feature during testing and training you have to provide same values for mean and std which are obtained using training data. I suggest you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/2595/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-10T12:42:28.173" LastActivityDate="2018-01-10T12:42:28.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26360" PostTypeId="1" CreationDate="2018-01-06T18:36:27.480" Score="9" ViewCount="12236" Body="&lt;p&gt;In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks? &lt;/p&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2018-01-08T04:37:09.007" Title="How to get a confidence score for predictions?" Tags="|machine-learning|neural-network|regression|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26361" PostTypeId="1" CreationDate="2018-01-06T20:08:22.303" Score="1" ViewCount="355" Body="&lt;p&gt;I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; I'm sitting on roughly 50 at the moment. I've looked at using PCA to reduce this number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I can understand the things I've been reading, I should exclude my target variable from the analysis. But I'm not sure if I should perform PCA on the whole remaining data set (Including the values that were already numerical, like 'age') or just on the values I converted from Nominal to Numerical and then re-add those to the already numeric values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To clarify, I've already converted this data-set from nominal to binary, and I'm not sure if I should apply PCA to just the binary columns generated, or the entire thing.&lt;/p&gt;&#xA;" OwnerUserId="44248" LastEditorUserId="26562" LastEditDate="2018-01-06T23:06:59.200" LastActivityDate="2020-08-14T08:14:45.817" Title="Should I apply PCA on the entire dataset or just the nominal values?" Tags="|pandas|preprocessing|pca|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26363" PostTypeId="2" ParentId="26361" CreationDate="2018-01-06T20:49:36.020" Score="1" Body="&lt;p&gt;It is not advised for you to apply PCA on a dataset with nominal values. You can, but PCA translates variables in space. It is hard to find the relationship between nominal values in space. For example, how would one quantify the space between 'male' and 'female' or 'white' and 'red' or 'PC' and 'mobile phone'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some alternatives are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use a tree model, like random forest, which will easily handle nominal values.&lt;/li&gt;&#xA;&lt;li&gt;Use FactoMine in R.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43914" LastActivityDate="2018-01-06T20:49:36.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26364" PostTypeId="2" ParentId="26361" CreationDate="2018-01-06T22:17:46.993" Score="0" Body="&lt;p&gt;For categorical attributes, use correspondence anlaysis rather than PCA. Since you tagged this &amp;quot;pandas&amp;quot;, &lt;a href=&quot;https://pypi.python.org/pypi/mca/1.0.2&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;'s a python package.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="98307" LastEditDate="2020-08-14T08:14:45.817" LastActivityDate="2020-08-14T08:14:45.817" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26366" PostTypeId="1" CreationDate="2018-01-06T23:41:20.297" Score="124" ViewCount="114678" Body="&lt;p&gt;I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;noreferrer&quot;&gt;keras documentation&lt;/a&gt;, it says the input to an RNN layer must have shape &lt;code&gt;(batch_size, timesteps, input_dim)&lt;/code&gt;. This suggests that all the training examples have a fixed sequence length, namely &lt;code&gt;timesteps&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this is not especially typical, is it? I might want to have the RNN operate on sentences of varying lengths. When I train it on some corpus, I will feed it batches of sentences, all of different lengths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it. But then does that mean I can't make predictions at test time with input length greater than that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a question about Keras's particular implementation, I suppose, but I'm also asking for what people typically do when faced with this kind of a problem in general.&lt;/p&gt;&#xA;" OwnerUserId="44256" LastEditorUserId="33202" LastEditDate="2018-12-20T00:32:13.707" LastActivityDate="2019-11-11T13:30:34.737" Title="Training an RNN with examples of different lengths in Keras" Tags="|python|keras|rnn|training|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26367" PostTypeId="1" CreationDate="2018-01-07T01:10:30.030" Score="1" ViewCount="308" Body="&lt;p&gt;Looking at the Tensor-flow object detection API, and walking through the &quot;How to train your own Object Detector&quot; for raccoons, which is a handy guide to get you up and running.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is append the SSD_Mobile model to include my own images along with the old pre-trained data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any guide out there that shows how this can be done?&lt;/p&gt;&#xA;" OwnerUserId="27814" LastEditorUserId="29575" LastEditDate="2018-01-07T01:19:03.693" LastActivityDate="2018-03-25T20:18:33.647" Title="Tensorflow Object Detection API" Tags="|tensorflow|object-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26370" PostTypeId="2" ParentId="26360" CreationDate="2018-01-07T04:38:59.377" Score="9" Body="&lt;p&gt;No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are reported as expectations). Here's the general procedure:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.&lt;/li&gt;&#xA;&lt;li&gt;Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)&lt;/li&gt;&#xA;&lt;li&gt;For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$&lt;/li&gt;&#xA;&lt;li&gt;Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$&lt;/li&gt;&#xA;&lt;li&gt;Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 -&#xA; \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-08T04:37:09.007" LastActivityDate="2018-01-08T04:37:09.007" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26371" PostTypeId="1" CreationDate="2018-01-07T06:17:41.347" Score="1" ViewCount="284" Body="&lt;p&gt;I have data on various standardized tests that certain students took and whether or not they ended up passing a specific class in college. Using the pass/fail as the dependent variable and the different scores on the test for the independent variables, I would like to figure out a feature that best predicts the students success in that class; i.e., Student gets these scores on these tests hence the probability that they pass is so and so.&lt;/p&gt;&#xA;&lt;p&gt;Logistic regression is my first instinct but I was wondering if there are other, possibly better, ways to approach this problem.&lt;/p&gt;&#xA;" OwnerUserId="44263" LastEditorUserId="1330" LastEditDate="2021-07-21T14:49:37.940" LastActivityDate="2021-07-21T14:49:37.940" Title="Which machine learning technique to predict student passing based on standardized tests?" Tags="|python|scikit-learn|random-forest|logistic-regression|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26372" PostTypeId="2" ParentId="26371" CreationDate="2018-01-07T06:36:40.477" Score="-1" Body="&lt;p&gt;In general, it's never a bad idea to use logistic regression as a first stab at a classification problem. If it doesn't work great, it at least gives you a baseline. Random forest would probably work well here as well. Generally, random forests aren't considered very interpretable, but they're actually pretty interpretable if you're interested in &lt;a href=&quot;http://blog.datadive.net/interpreting-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;understanding the decision process for a single prediction&lt;/a&gt;, which it sounds like would be sufficient for your needs here (i.e., the model can tell you a student is likely to perform a certain way because of their behavior on specific tests). A random forest will probably also handle missing data better, which might be useful to you since I'm guessing not all students took the same tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option you can try here which handles null data extremely well and would also be simple for inference, is naive bayes. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-07T06:36:40.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26373" PostTypeId="1" AcceptedAnswerId="26390" CreationDate="2018-01-07T08:55:31.023" Score="0" ViewCount="270" Body="&lt;p&gt;In the &lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE paper&lt;/a&gt;, the authors present the logic of creating synthetic examples when all features are nominal (section 6.2, SMOTE-N):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To generate new minority class feature vectors, we can create new set&#xA;  feature values by taking the majority vote of the feature vector in&#xA;  consideration and its k nearest neighbors&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Along with this example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Let F1 = A B C D E be the feature vector under consideration and let&#xA;  its 2 nearest neighbors be &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;F2 = A F C G N &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;F3 = H B C D N &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;The&#xA;  application of SMOTE-N would create the following feature vector: FS =&#xA;  A B C D N&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How would FS be chosen in the case that &lt;code&gt;F3 = H B C I N&lt;/code&gt;? How does Value Difference Metric by Cost and Salzberg described in the paper assist in this case?&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="25849" LastEditDate="2020-10-03T08:16:09.420" LastActivityDate="2020-10-03T08:16:09.420" Title="How to perform SMOTE-N when there is no majority vote?" Tags="|feature-selection|categorical-data|class-imbalance|smote|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26374" PostTypeId="1" AcceptedAnswerId="28394" CreationDate="2018-01-07T09:54:03.007" Score="4" ViewCount="2227" Body="&lt;p&gt;In the SMOTE paper &lt;a href=&quot;https://jair.org/index.php/jair/article/view/10302&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, the authors present the logic for creating synthetic examples when some of the features are nominal and some are continuous (section 6.1, SMOTE-NC).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example is provided:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$F_1$&lt;/span&gt; = 1 2 3 A B C [Let this be the sample for which we are&#xA;  computing nearest neighbors]    &lt;span class=&quot;math-container&quot;&gt;$F_2$&lt;/span&gt; = 4 6 5 A D E    &lt;span class=&quot;math-container&quot;&gt;$F_3$&lt;/span&gt; = 3 5 6 A&#xA;  B K    So, Euclidean Distance between &lt;span class=&quot;math-container&quot;&gt;$F_2$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$F_1$&lt;/span&gt; would be:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Eucl$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$\sqrt{(4-1)^2 + (6-2)^2 + (5-3)^2 + Med^2 + Med^2}$&lt;/span&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Med is the median of the standard deviations of continuous features&#xA;  of  the minority class.    The median term is included twice for&#xA;  feature numbers &lt;span class=&quot;math-container&quot;&gt;$5: B→D$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$6:  C→E$&lt;/span&gt;, which differ for the two&#xA;  feature vectors: &lt;span class=&quot;math-container&quot;&gt;$F_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$F_2$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The paper lacks explanation about why the nominal features should be affected by the continuous ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone provide such explanation? Did I miss it in the paper?&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="25849" LastEditDate="2020-10-02T13:47:07.697" LastActivityDate="2020-10-02T13:47:07.697" Title="Logic behind SMOTE-NC?" Tags="|feature-selection|categorical-data|class-imbalance|smote|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26376" PostTypeId="2" ParentId="26253" CreationDate="2018-01-07T12:08:46.020" Score="1" Body="&lt;p&gt;I have come across such a &lt;a href=&quot;https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/master/data/arvix_abstracts.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt; in &lt;a href=&quot;https://github.com/chiphuyen/stanford-tensorflow-tutorials&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS 20SI GitHub repo&lt;/a&gt;: it's a collection of abstracts from 7200 research papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need even more, you can always write a simple crawler of &lt;a href=&quot;https://arxiv.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;arXiv web-site&lt;/a&gt;. The abstracts are outlined right on the webpage in a &lt;code&gt;&amp;lt;blockquote&amp;gt;&lt;/code&gt; tag, no need to download and parse a pdf.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-01-07T12:08:46.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26377" PostTypeId="1" AcceptedAnswerId="26379" CreationDate="2018-01-07T12:31:32.830" Score="0" ViewCount="70" Body="&lt;p&gt;My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. The tasks are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first job does analysis of a large quantity of text to look for ERROR messages (grep).&lt;/li&gt;&#xA;&lt;li&gt;The second job does machine learning &amp;amp; calculate models prediction on some data with an iterative way.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;My question is: Which one of the two jobs will benefit from SPARK the most?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SPARK relies on memory so I think that it is more suited to machine learning. The quantity of DATA isn't that large compared to the logs JOB. But I'm not sure. Can someone here help me if I neglected some piece of information?&lt;/p&gt;&#xA;" OwnerUserId="44274" LastEditorUserId="26562" LastEditDate="2018-01-07T20:15:29.730" LastActivityDate="2018-01-07T20:15:29.730" Title="Which one of these tasks will benefit the most from SPARK?" Tags="|apache-spark|apache-hadoop|map-reduce|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26379" PostTypeId="2" ParentId="26377" CreationDate="2018-01-07T14:03:48.800" Score="1" Body="&lt;p&gt;I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you have mentioned, spark is able to keep data in memory between two iterations while Hadoop MapReduce has to write and read data to file system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a good comparison of the two frameworks : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4RaD2.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4RaD2.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-01-07T14:03:48.800" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26380" PostTypeId="1" AcceptedAnswerId="26389" CreationDate="2018-01-07T15:17:04.047" Score="1" ViewCount="210" Body="&lt;p&gt;So, &lt;code&gt;sklearn&lt;/code&gt; doesn't support categorical data in its models. Is there a known alternative for categorical data modeling (such as random forests, etc.) for Python?&lt;/p&gt;&#xA;" OwnerUserId="43870" LastActivityDate="2019-07-14T14:12:34.363" Title="Scikit alternative for categorical data modeling?" Tags="|python|scikit-learn|categorical-data|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26382" PostTypeId="1" AcceptedAnswerId="26384" CreationDate="2018-01-07T15:47:08.357" Score="4" ViewCount="6970" Body="&lt;p&gt;I want to do a GAN with coloured pictures. This means I need a three dimensional input and therefore I like to use conv3d and conv3d_transpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately in the &lt;a href=&quot;https://www.tensorflow.org/versions/r0.12/api_docs/python/nn/convolution#conv3d_transpose&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow documentation&lt;/a&gt;, I can't find any formula for the output shape. Can anyone give me a hint on how to find the shape of the function's results?&lt;/p&gt;&#xA;" OwnerUserId="43141" LastEditorUserId="29169" LastEditDate="2021-02-13T19:19:23.820" LastActivityDate="2021-02-13T19:19:23.820" Title="What is the shape of conv3d and conv3d_transpose?" Tags="|deep-learning|tensorflow|cnn|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26384" PostTypeId="2" ParentId="26382" CreationDate="2018-01-07T16:01:43.183" Score="4" Body="&lt;p&gt;Although your input data is three dimensional, you have to use &lt;code&gt;Conv2D&lt;/code&gt; for your task. I guess &lt;code&gt;Conv3d&lt;/code&gt; is used for data with temporal characteristic, yours is just a simple picture. To illustrate why you should &lt;code&gt;Conv2D&lt;/code&gt;, suppose your input image is &lt;code&gt;224 * 224 * 3&lt;/code&gt; and you employ a &lt;code&gt;Conv2D&lt;/code&gt; layer with 10 filters. You have to specify stride and padding in order to specify the output shape. You have to specify dimensions to illustrate the height and width of you filters, also known as kernels, filter size will affect the output size if you assign padding to 'valid'. Here, there is a point. Suppose you have specified the filter size a &lt;code&gt;10 * 10&lt;/code&gt; filter, then if the input shape was &lt;code&gt;224 * 224 * 1&lt;/code&gt;, each filter would be of size &lt;code&gt;10 * 10 * 1&lt;/code&gt; to fit the input area. Now that the input is of size &lt;code&gt;224 * 224 * 3&lt;/code&gt; the size of each kernel is &lt;code&gt;10 * 10 * 3&lt;/code&gt; to fit the input &lt;strong&gt;volume&lt;/strong&gt;. Consider in all cases the output of each convolution operation, better to say cross correlation, is a scalar. For more information take a look at videos &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/home/week/1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and for your case I encourage you watching &lt;em&gt;Convolution Over Volume&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-07T17:51:40.730" LastActivityDate="2018-01-07T17:51:40.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26386" PostTypeId="1" CreationDate="2018-01-07T17:53:35.680" Score="1" ViewCount="903" Body="&lt;p&gt;I am building an LSTM to attempt to learn the historic trend of some time-series data set (e.g. the daily share price of a company). When training my network, I am taking batches of size 1, each consisting of 25 sequential daily closing prices, where it then makes a prediction for the price on the 26th day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say the first sequence in an epoch used to train begins at $t=0$ and ends at $t=24$. I am using a stateful LSTM and hence taking the output state of one batch and inputting it to the state of the next batch, does this mean that my 2nd batch must be $[t=25, t=49]$? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I instead &quot;slide&quot; each batch by 1 time step so that the 2nd batch is $[t=1, t=25]$, or does this defeat the purpose of passing the state between batches?&lt;/p&gt;&#xA;" OwnerUserId="37461" LastEditorUserId="29575" LastEditDate="2018-01-07T18:14:24.890" LastActivityDate="2021-01-16T05:00:45.080" Title="Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps" Tags="|neural-network|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26387" PostTypeId="2" ParentId="26382" CreationDate="2018-01-07T18:18:53.450" Score="2" Body="&lt;p&gt;By convention, in addition to the input feature map - which may be 1D for audio, 2D for a typical image, 3D for a sequence of video frames - for a convolutional network, there are two additional dimensions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The number of examples in a batch or mini-batch, counting even a single example as a mini-batch of size 1)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The number of feature maps or channels in the current layer, counting even a single grayscale image as an array of 1 channel, or RGB (or other colour space) image as 3 channels&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So in your case you will want a &lt;code&gt;conv2d&lt;/code&gt; layer to process the image. The precise order of dimensions varies a lot - check your library docs, and also note most libraries allow you to alter the arrangement for cross-compatibility with different toolchains. But a typical layer input or output might be arranged as $\text{BatchSize} \times \text{Height} &#xA; \times \text{Width} \times \text{Channels}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input and output dimensions are the same (but usually with different sizes), so that layers can be chained together naturally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The documentation you linked does in fact say this (sort of, it could explain the sameness in more detail, since data type and dimensionality are in fact the same and both could vary):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Returns:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;A Tensor with the same type as value.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To answer your question in the title:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the shape of conv3d and conv3d_transpose?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is an order 5 tensor, and the dimensions are: $\text{BatchSize} \times \text{Depth} \times \text{Height} \times \text{Width} \times \text{Channels}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could in theory use this for your GAN, but you would need to add (a probably useless) &lt;em&gt;depth&lt;/em&gt; dimension to the shape.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-07T18:26:32.073" LastActivityDate="2018-01-07T18:26:32.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26388" PostTypeId="1" CreationDate="2018-01-07T19:59:02.353" Score="7" ViewCount="637" Body="&lt;p&gt;I'm trying to come up with a good way to explain the 3 importance metrics &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/R-package/discoverYourData.html#build-the-feature-importance-data-table&quot; rel=&quot;noreferrer&quot;&gt;(Gain, Cover, Frequency)&lt;/a&gt; to a layman with only a basic understanding of XGBoost and trees in general.  How best would you frame this explanation?&lt;/p&gt;&#xA;" OwnerUserId="44290" LastEditorUserId="29575" LastEditDate="2018-01-07T20:08:36.167" LastActivityDate="2018-01-24T20:16:33.407" Title="Layman's Interpretation of XGBoost Importance" Tags="|python|r|xgboost|" AnswerCount="0" CommentCount="3" ClosedDate="2018-11-07T19:55:34.450" ContentLicense="CC BY-SA 3.0" />
  <row Id="26389" PostTypeId="2" ParentId="26380" CreationDate="2018-01-07T20:07:07.533" Score="2" Body="&lt;p&gt;There are definitely ways to process your data to make categorical data compatible with sklearn (e.g one-hot encoding). An alternative you can look into is h2o, which supports categorical features natively (although it doesn't offer the breadth of models of sklearn).&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-07T20:07:07.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26390" PostTypeId="2" ParentId="26373" CreationDate="2018-01-07T20:10:12.233" Score="1" Body="&lt;p&gt;I don't know what &quot;VDM&quot; stands for, but a simple solution for tie breaking is to randomly pick one of the tied options.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-07T20:10:12.233" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26391" PostTypeId="1" CreationDate="2018-01-07T21:29:53.810" Score="1" ViewCount="571" Body="&lt;p&gt;Which of the algorithms in the current literature for contextual bandits can be implemented for online learning and which ones can't? I'd really appreciate it if someone could provide a link to papers too! Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="43865" LastActivityDate="2021-05-11T00:01:25.163" Title="contextual bandits for online learning" Tags="|machine-learning|reinforcement-learning|online-learning|randomized-algorithms|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26392" PostTypeId="1" AcceptedAnswerId="40627" CreationDate="2018-01-07T22:07:49.193" Score="1" ViewCount="3753" Body="&lt;p&gt;I have started learning deep learning and using &lt;code&gt;keras&lt;/code&gt; library. But I am confused as to how to take a proper estimate of the value to use for &lt;code&gt;units&lt;/code&gt; parameter of the &lt;code&gt;dense&lt;/code&gt; method. I came across this tip that we can take it as the average of the number of input nodes and output nodes but everywhere it says that it comes from experience. I want to know if there are things to look out for to estimate it wisely or any other things I need to know. Thanks for the help.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier.add(Dense(units=6, kernel_initializer=&quot;uniform&quot;, activation=&quot;relu&quot;, input_dim=11))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43967" LastEditorUserId="43967" LastEditDate="2018-01-08T12:19:30.957" LastActivityDate="2018-11-02T15:21:27.297" Title="correct value of output_dim or units parameter of dense" Tags="|python|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26394" PostTypeId="1" AcceptedAnswerId="26406" CreationDate="2018-01-08T01:02:06.843" Score="1" ViewCount="3048" Body="&lt;p&gt;I have two datasets A and B. What I would like to do is for each observation in A, I would like to find 5 observations from B that are closest and match to A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I start? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your help!&lt;/p&gt;&#xA;" OwnerUserId="44297" LastActivityDate="2018-01-08T10:56:34.620" Title="Find similar observations in two datasets" Tags="|machine-learning|data-mining|predictive-modeling|statistics|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ClosedDate="2018-01-09T06:49:45.237" ContentLicense="CC BY-SA 3.0" />
  <row Id="26395" PostTypeId="2" ParentId="26164" CreationDate="2018-01-08T01:02:18.610" Score="3" Body="&lt;p&gt;The &lt;strong&gt;standard momentum&lt;/strong&gt; would have these steps:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;straight away, recompute the new momentum:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\mu_{t+1} = \mu_{t} \cdot (decayScalar) + (learnRate)\cdot \nabla$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;adjust the weights by this new momentum&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{t+1} := \theta_{t} - \mu_{t+1}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Nesterov momentum&lt;/strong&gt; has this:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Make a big jump: &lt;strong&gt;correct&lt;/strong&gt; the weights by any &lt;span class=&quot;math-container&quot;&gt;$\mu$&lt;/span&gt; we have so far in our posession:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{t+1} := \theta_{t} - \mu\cdot (decayScalar)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the gradient &lt;span class=&quot;math-container&quot;&gt;$\nabla$&lt;/span&gt; from the new weights &lt;span class=&quot;math-container&quot;&gt;$\theta_{t+1}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Correct&lt;/strong&gt; these weights by this gradient (right now without any momentum):&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{t+2} := \theta_{t+1} -  (learnRate)\cdot \nabla$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;At the very end, re-compute the momentum as follows:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\mu := \theta_{t+2} - \theta_{t}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;So, the momentum is updated at the very end. It becomes a vector from &amp;quot;weights before the big jump&amp;quot;, pointing towards the &amp;quot;weights after the correction by the fresh gradient&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;reference: &lt;a href=&quot;https://youtu.be/N18Km9YIIug?t=6m52s&quot; rel=&quot;nofollow noreferrer&quot;&gt;Geoffrey Hinton Lecture 6C Corsera&lt;/a&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Re-arranging:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;To avoid sticking the gradient computation in the middle of our optimizer function (steps 2 and 3), we can instead re-arrange things as follows:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;compute gradient for the weights we have thus far.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;correct such  weights by the gradient (right now without any momentum), as follows:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{t+1}:=\theta_t - (learnRate)\cdot \nabla$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;update the momentum:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\mu := \theta_{t+1} - \theta_{cached}$$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{cached} := \theta_{t+1}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;big jump&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\theta_{t+2} := \theta_{t+1} - \mu\cdot (decayScalar)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Notice, this way steps 2,3,4 are all inside of our optimizer. We can compute the gradient, &lt;em&gt;outside&lt;/em&gt; of our optimizer (during step 1) making our code much more readable :)&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    size_t _numApplyCalled = 0;&#xA;      &#xA;    //Nesterov Accelerated Gradient.&#xA;    //Placed at the end of a backprop, should be followed by a usual forward-propagation&#xA;    // https://datascience.stackexchange.com/a/26395/43077&#xA;    //&#xA;    void apply( float *toChange,  float *newGrad,  size_t count ){&#xA;&#xA;        float learnRate = get(OptimizerVar::LEARN_RATE);//scalar&#xA;        float momentumCoeff = get(OptimizerVar::MOMENTUM_1);//scalar&#xA;&#xA;        const bool isFirstEver_apply = _numApplyCalled == 0;&#xA;&#xA;        for (int i=0; i&amp;lt;_arraySize; ++i){&#xA;                //correction by gradient alone:&#xA;                toChange[i]  -= newGrad[i]*learnRate;&#xA;&#xA;                // determine momentum:&#xA;                if (isFirstEver_apply){//nothing was cached yet.&#xA;                    _momentumVals[i] = 0.0f;&#xA;                }&#xA;                else {&#xA;                    _momentumVals[i] = toChange[i] - _weightsCached[i];&#xA;                }&#xA;                //caching, AFTER momentum calc, but BEFORE the jump:&#xA;                _weightsCached[i] = toChange[i];&#xA;            &#xA;                //jump:&#xA;                toChange[i] -= _momentumVals[i] * momentumCoeff;&#xA;        }//end for&#xA;&#xA;        ++_numApplyCalled;//increments by 1&#xA;    }&#xA;&#xA;    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-10-01T12:15:11.327" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26396" PostTypeId="1" AcceptedAnswerId="37502" CreationDate="2018-01-08T02:54:53.180" Score="5" ViewCount="242" Body="&lt;p&gt;I'm trying to create a neural network that can learn how to write text character by character from the book &lt;a href=&quot;http://www.gutenberg.org/files/766/766-0.txt&quot; rel=&quot;noreferrer&quot;&gt;David Copperfield&lt;/a&gt; (via Project Gutenburg). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It starts great, then forgets punctuation around epoch 25 and devolves into nonsense at epoch 26. I've been trying to find a starting point where I can start attacking this problem. I've read research papers on the concept of clipping the gradient to stop it from vanishing or exploding, but I'm having a hard time finding a way to first visualize the gradient and what's going wrong and then how to clip it at the most appropriate value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've saved checkpoint models from all fifty epochs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using a clipping the gradient at 5 based on a research paper about gradient clipping in LSTMs and it didn't change anything. I don't have to budget to find the optimum value through experimentation, but if that's the only way to do it I'll make it work.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've been working on this a long time, but I'm self-taught and feel a a bit out of my depth here. A nudge in the right direction from a subject matter expert would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch 20:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;and that I saw Traddles the service in the house, person a sort of something and so great hand of the counter being a prospect of Mr. Omer, had replied to go and so moment as a case so refersing the house of his man as distracting him on the door and last hearing her after for the streng to a head, and Mr. Micawber, that many beer and when I was so consequent when the man. I can be so looked to keep me all the black desiring the constinging and enearly to his part of more arms were in the scound as my delight, that she asked the time and herself of what I had been with the windows years of such a green eye of the&#xA;  most night was a dear, but that there are you? I began to make dear, she was not that a present words, long a back hand with a better was many to a relief in search had one been what he was to say that the hand sitting weeks, and made with a present time, and my face, so he sat no more about the mount, and set of the time of petitution and the door was in a proceeding the spirit,&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Epoch 25: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sir, I am willing to the beautiful thought who was not her to the best flew for a son, when I said, said I. When she had to speak the strong spirits of this comprier of an and bill time of the resist to my street at the garden with it, and that he before my for the sense of the room in the old evening in the time with a real arms child would take it a little books to the send with a shining the care that I cant have seen it a good shoulder of the morning for a moment in a time of a good mist from the expect of laughing so like a deport for one of the carrier of that success of a spoke with the wild of the table just over and thing being pretending in the proposition that there took a far in my devoted towards me what he went to a sense of the stare to the breditol for the great contriout, and that I were done in the shadow and man of our fathers dream at once with me, as to say, but he said a man. I think it was one far that he had been over the desire that was not loved in the grief that could says more&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Epoch 26:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;spi  ,  i ed o eoa t  i    i.  nhsaae     s  t?h  h t    h    n&lt;br&gt;&#xA;  t  cha,p   t r   ieto  t wo&#xA;  a hw   ne s  uawpai  ,y  na  a    e  t dttte  t?atsh      oh h  i au a    a a   ddn e haf e  s t.rooe  wt   etdt s&lt;br&gt;&#xA;  t a   ,,  i t e a h e&#xA;     a, dt os   rhr   tis        m  elrii    e a ao   ty   otatp   ya  r t a  ty o  he,  ee , s s  i.&#xA;  hn    e  o  te  a o o se  s h u     te  senea    e tt        s d  ew , ie s  I ee     ihi hnts  a an r rv otso t   a    eshne     tta o  tt&lt;br&gt;&#xA;    e  o  m l   h arnt led n    sa a a    e  h  tww n  ee   t     h   ha  ,tdeh  t e  nntt  i atnr,e     wt eee     hb a     t  n  oea    a     e    ei t -.     de  e a e s e n   atehh   h e  a s   ef   e n  to  hr   d , eh  In.i o st watn t  htih e io    tt a x h   s s ,  e   s  r ohsmtldal  er e&lt;br&gt;&#xA;  n, t,dtthan,t   h s hdhe   oa  oh tbh t  ot isn  o  e tr      et ttnm an  a    ot  ng c a ds   tr .s&#xA;  t h    a t&#xA;  t    ewehso         n sr o e  se   i   e e   httst    b    i tt  .  t n I   he o   w so tt t,w sttt  nta t ai   o  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Code Follows: &#xA;(I started with Tensorflow, but switched to TFLearn for simplicity. I'm willing to learn any frameworks that have the tools to solve the problem, though. I'm a self-taught student, so learning is really the only objective here.) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import time &#xA;start_script_time = time.time()&#xA;&#xA;&#xA;import numpy as np&#xA;import tflearn&#xA;import random&#xA;import pickle&#xA;''' create data ''' &#xA;&#xA;log_file = 'dickens_log.txt'&#xA;def my_log(text, filename=log_file):&#xA;    text = str(text)&#xA;    print(text)&#xA;    with open(filename, 'a', newline='\n') as file:&#xA;        file.write(text + '\n')&#xA;&#xA;&#xA;try:&#xA;    book_name = 'as_loaded.txt'&#xA;    book = open(book_name, errors='ignore',&#xA;                encoding='ascii', newline='\n').read() &#xA;except:&#xA;    book_name = 'copperfield.txt'&#xA;    book = open(book_name, errors='ignore',&#xA;                encoding='utf-8', newline='\n').read()&#xA;    #book = book.replace('\r', '')&#xA;    #book = book.replace('\n', ' ')&#xA;    with open('as_loaded.txt', 'w', newline='\n') as file:&#xA;        file.write(book)&#xA;&#xA;# make smaller slice for quickly testing code on CPU &#xA;# book = book[0:1500]&#xA;# del(book_name)&#xA;&#xA;# length of strings in the training set&#xA;string_length = 30&#xA;&#xA;def process_book(book, string_length, redundant_step=3):&#xA;&#xA;    # Remember to pickle to dictionary as a binary. This is pretty critical for loading your model on a different machine than you trained on. &#xA;    try:&#xA;        pickle_ld = open('charDict.pi', 'rb')&#xA;        charDict = pickle.load(pickle_ld)&#xA;        pickle_ld.close()&#xA;    except:&#xA;        # dictionary of character-number pairs&#xA;        chars = sorted(list(set(book)))&#xA;        charDict = dict((c, i) for i, c in enumerate(chars))&#xA;        #charDict.pop('\r')&#xA;        pickle_sv = open('charDict.pi', 'wb')&#xA;        pickle.dump(charDict, pickle_sv)&#xA;        pickle_sv.close()&#xA;&#xA;    len_chars = len(charDict)&#xA;&#xA;    # train is a string input and target is the &#xA;    # expected next character     &#xA;    train = []&#xA;    target = []&#xA;    for i in range(0, len(book)-string_length, redundant_step):&#xA;        train.append(book[i:i+string_length])&#xA;        target.append(book[i+string_length])&#xA;&#xA;    # create containers for data with appropriate dimensions&#xA;    # 3D (n_samples, sample_size, n_categories)&#xA;    X = np.zeros((len(train), string_length, len_chars), dtype=np.bool)&#xA;    # 2D (n_samples, n_categories)&#xA;    y = np.zeros((len(train), len_chars), dtype=np.bool)&#xA;&#xA;    # fill arrays&#xA;    for i, string in enumerate(train):&#xA;        for j, char in enumerate(string):&#xA;            # X is a sparse 3D tensor where a 1 value signals &#xA;            # that a information is present in 3rd dimension index&#xA;            X[i, j, charDict[char]] = 1&#xA;        y[i, charDict[target[i]]] = 1&#xA;&#xA;    return charDict, X, y&#xA;&#xA;charDict, X, y = process_book(book, string_length)&#xA;&#xA;''' build the network ''' &#xA;&#xA;# number of hidden layers in each LSTM layer&#xA;lstm_hidden = 512&#xA;drop_rate = 0.5&#xA;&#xA;net = tflearn.input_data(shape=(None, string_length, len(charDict)))&#xA;&#xA;# input shape is the length of the strings by the number of characters&#xA;# leading None is necessary if no placeholders &#xA;net = tflearn.lstm(net, lstm_hidden, return_seq=True)&#xA;net = tflearn.dropout(net, drop_rate)&#xA;&#xA;# You have to use a separate dropout layer. There's a glitch where tflean&#xA;# will drop out all the time, not just during training, making prediction&#xA;# impossible. &#xA;net = tflearn.lstm(net, lstm_hidden, return_seq=True)&#xA;net = tflearn.dropout(net, drop_rate)&#xA;&#xA;net = tflearn.lstm(net, lstm_hidden, return_seq=False)&#xA;net = tflearn.dropout(net, drop_rate)&#xA;&#xA;net = tflearn.fully_connected(net, len(charDict), activation='softmax')&#xA;&#xA;net = tflearn.regression(net, optimizer='adam', &#xA;                         loss='categorical_crossentropy', &#xA;                         learning_rate=0.005)&#xA;&#xA;# https://www.quora.com/What-is-gradient-clipping-and-why-is-it-necessary &#xA;model = tflearn.SequenceGenerator(net, dictionary=charDict, &#xA;                                  seq_maxlen=string_length,&#xA;                                  clip_gradients=5,&#xA;                                  checkpoint_path='model_checkpoint_v3')&#xA;&#xA;&#xA;my_log('Character dictionary for ' + book_name)&#xA;my_log(charDict)&#xA;my_log('charDict length: ' + str(len(charDict)))&#xA;my_log('&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;&amp;amp;')&#xA;&#xA;def random_seed_test(book, temp=0.5, gen_length=300):&#xA;    my_log('#######################')    &#xA;    seed_no = random.randint(0, len(book) - string_length)&#xA;    seed = book[seed_no : seed_no + string_length]&#xA;    my_log('(temp ' + str(temp) + ') ' + 'Seed: &quot;' + seed + '&quot;')&#xA;    my_log('++++++++++++++++++++++')  &#xA;    my_log(model.generate(seq_length=gen_length, temperature=temp, &#xA;                              seq_seed=seed))&#xA;    my_log('#######################') &#xA;&#xA;&#xA;&#xA;# If you train one epoch at a time in a loop, you can get an idea &#xA;# of how the model progressed. With other ML problems, error rate and &#xA;# accuracy reveal a lot, but with this problem performance is subjective. &#xA;for epoch in range(50):&#xA;    start_epoch = time.time()&#xA;    my_log('======================================================')&#xA;    my_log('Begin epoch %d' % (epoch+1))&#xA;    model.fit(X, y, validation_set=0.1, batch_size=128, n_epoch=1)&#xA;    my_log('End epoch %d' % (epoch+1))&#xA;    epoch_time = time.time() - start_epoch&#xA;    my_log('This epoch took ' + str(epoch_time) + ' seconds.')&#xA;    random_seed_test(book, temp=0.5, gen_length=1000)&#xA;    random_seed_test(book, temp=0.75, gen_length=1000)&#xA;    random_seed_test(book, temp=1.0, gen_length=1000)&#xA;    my_log('End epoch %d' % (epoch+1))&#xA;    my_log('======================================================')&#xA;&#xA;&#xA;full_time = time.time() - start_script_time&#xA;my_log('This program took ' + str(full_time) + ' seconds.')&#xA;&#xA;model.save('dickens_compute_4.model')&#xA;&#xA;my_log('finished')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44299" LastEditorUserId="29575" LastEditDate="2018-01-08T03:25:10.060" LastActivityDate="2018-08-28T07:00:08.793" Title="Suspected Exploding Gradient in Character Generator LSTM" Tags="|neural-network|deep-learning|lstm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26397" PostTypeId="2" ParentId="24319" CreationDate="2018-01-08T03:36:24.027" Score="11" Body="&lt;p&gt;Yes, with limitations. Google Cloud Compute gives you 300 dollars worth of free credit signing up, and Microsoft Azure gives you 200 dollars (but their GPU time is a bit cheaper, so it's almost the same). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This gets you a lot of GPU time, and will get you started while you weigh your options. &lt;/p&gt;&#xA;" OwnerUserId="44299" LastEditorUserId="44299" LastEditDate="2018-04-02T02:33:53.860" LastActivityDate="2018-04-02T02:33:53.860" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26399" PostTypeId="1" CreationDate="2018-01-08T07:57:42.003" Score="1" ViewCount="624" Body="&lt;p&gt;I have studied the &lt;code&gt;Perceptron&lt;/code&gt; algorithm and now I'm trying to understand &lt;code&gt;logistic regression&lt;/code&gt;. I know what odd ratio and &lt;code&gt;Logit&lt;/code&gt; functions are but I don't understand why we create the following equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$logit(\frac{p}{1-p}) = \bar{\omega}^{T}\bar{x}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;w&lt;/code&gt; is the weight vector. What does it represent, and why? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone show me, how does that work?&lt;/p&gt;&#xA;" OwnerUserId="43656" LastEditorUserId="40853" LastEditDate="2018-01-08T10:06:28.167" LastActivityDate="2018-01-09T14:50:38.883" Title="Logit function in Logistic Regression" Tags="|machine-learning|neural-network|logistic-regression|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26401" PostTypeId="1" CreationDate="2018-01-08T08:55:19.083" Score="14" ViewCount="14538" Body="&lt;p&gt;I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently see two different kind of codes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type 1 is where no TimeDistributed applied like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=Sequential()&#xA;&#xA;model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1], border_mode=&quot;valid&quot;, input_shape=[1, 56,14]))&#xA;model.add(Activation(&quot;relu&quot;))&#xA;model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))&#xA;model.add(Activation(&quot;relu&quot;))&#xA;model.add(MaxPooling2D(pool_size=pool_size))&#xA;&#xA;model.add(Reshape((56*14,)))&#xA;model.add(Dropout(0.25))&#xA;model.add(LSTM(5))&#xA;model.add(Dense(50))&#xA;model.add(Dense(nb_classes))&#xA;model.add(Activation(&quot;softmax&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Type 2 is where TimeDistributed is applied like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(InputLayer(input_shape=(5, 224, 224, 3)))&#xA;model.add(TimeDistributed(Convolution2D(64, (3, 3))))&#xA;model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))&#xA;model.add(LSTM(10))&#xA;model.add(Dense(3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Is my assumption correct that Type 1 is the one-to-many kind and Type&#xA;2 is the many-to-many kind? Or &lt;code&gt;TimeDistributed&lt;/code&gt; has no relevance in&#xA;this aspect?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In either case of one-to-many or many-to-many is the last dense layer&#xA;supposed to be 1 node &quot;long&quot; (emitting only one value in turn) and&lt;br&gt;&#xA;the previous recurrent layer is responsible to determine how many&lt;br&gt;&#xA;1-long value to emit? Or the last dense layer is supposed to consist &#xA;of N nodes where &lt;code&gt;N=max sequence length&lt;/code&gt;? If so, what is the point of&lt;br&gt;&#xA;using RNN here when we could produce a similar input with multiple&lt;br&gt;&#xA;outputs with N parallel &quot;vanilla&quot; estimators?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to define the number of timesteps in RNNs? Is it somehow&lt;br&gt;&#xA;correlated with the output sequence length or is it just a&lt;br&gt;&#xA;hyperparameter to tune?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Inn case of my Type 1 example above what is the point of applying&lt;br&gt;&#xA;LSTM when the model emits only one class prediction (of possible&lt;br&gt;&#xA;&lt;code&gt;nb_classes&lt;/code&gt;)? What if one omits the LSTM layer?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-07-30T13:45:42.713" Title="How to implement &quot;one-to-many&quot; and &quot;many-to-many&quot; sequence prediction in Keras?" Tags="|keras|rnn|lstm|sequence|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26402" PostTypeId="2" ParentId="26399" CreationDate="2018-01-08T08:55:26.470" Score="1" Body="&lt;p&gt;&lt;code&gt;W&lt;/code&gt; represents the wight matrix. This is what you try to learn, in other words to fix. &lt;code&gt;X&lt;/code&gt; is the input data. During training, you have to provide &lt;code&gt;X&lt;/code&gt;, feature vector, in order to learn parameters &lt;code&gt;W&lt;/code&gt;. You learn this parameter matrix / vector,  in order to have a model which represents your data. This model will be able to classify your seen and unseen data. Suppose that you have the following data. The input feature vector &lt;code&gt;X&lt;/code&gt; is a two dimensional vector, horizontal and vertical axes represent the value for each dimension. The input features belong to either class blue or red. You try to find the separator line by learning. You learn how to specify parameter vector &lt;code&gt;W&lt;/code&gt; to have a line which separates the data to each class. Using this procedure, after learning, you will be able to classify unseen data based on position of the data. Consider each setting of parameters will result in a different separator line. You try to find the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IiDeV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IiDeV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-08T08:55:26.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26403" PostTypeId="1" CreationDate="2018-01-08T09:09:26.580" Score="9" ViewCount="4605" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fVbPv.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fVbPv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In YOLO algorithm how do these grids output a prediction if some grids only see a small black portion of the car if the model was trained on datasets with full images?&lt;/p&gt;&#xA;" OwnerUserId="41741" LastActivityDate="2019-10-04T05:39:51.690" Title="How does YOLO algorithm detect objects if the grid size is way smaller than the object in the test image?" Tags="|image-classification|image-recognition|object-recognition|yolo|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26404" PostTypeId="2" ParentId="26403" CreationDate="2018-01-08T10:01:07.340" Score="4" Body="&lt;p&gt;Each grid predictor in YOLO should only have a high score that an object is within it, if it detects the &lt;em&gt;centre&lt;/em&gt; of the bounding rectangle is inside itself. So a grid point that contains only the wing mirror of a car should decide it has a low probability of containing the centre of the car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The predicted bounding rectangles are not constrained in the same way - YOLO can (and often does) predict bounding box dimensions from the centre that are larger than the grid cell dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each grid point is independently able to predict whether or not it contains an object's centre, what the bounding box dimensions are for the object, and what the object class is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you trace the layer connectivity, you will see that the grid cells are effectively interconnected in lower output layers, so the network as a whole &quot;sees&quot; more of each object and can influence individual object predictions, suppressing some and encouraging others, when objects span multiple grid locations. The grid cells are not isolated into sections, or restricted to only using data from the area that they cover for the prediction. The concept of what part of each image a feature &quot;pixel&quot; can access from the base image in a CNN is called the &quot;receptive field&quot; of the network, and can be &lt;a href=&quot;https://medium.com/@nikasa1889/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807&quot; rel=&quot;nofollow noreferrer&quot;&gt;calculated based on the architecture as explained in this blog on Medium&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-08T17:41:52.727" LastActivityDate="2018-01-08T17:41:52.727" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26406" PostTypeId="2" ParentId="26394" CreationDate="2018-01-08T10:56:34.620" Score="4" Body="&lt;p&gt;Look at &quot;unsupervised nearest neighbor&quot; algorithm. This algorithm needs records to be first expressed as vectors so that &quot;distance&quot; between two point so that it makes sense to talk about distance between two points. For each point in data A, you can look for K nearest neighbors from data B, after expressing all observations in a common vector space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will have to handle categorical columns correctly (say using one-hot encoding), as there's no concept of (direct) distance between categorical data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python scikit-learn library has a good &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;sklearn nearest neighbors documentation&quot;&gt;implementation of this algorithm&lt;/a&gt;. Reading the API documentation is a good place to start.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2018-01-08T10:56:34.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26407" PostTypeId="1" CreationDate="2018-01-08T11:19:43.483" Score="2" ViewCount="143" Body="&lt;p&gt;I am trying to scope out an approach to learn to identify repetitive human body activity based on collected motion data and would like some advice on some challenges I identified. I determined a set of data, the approach, and a set of potential challenges using cosine waveforms as an analogy before I start on the actual work.&lt;/p&gt;&#xA;&lt;h3&gt;Data:&lt;/h3&gt;&#xA;&lt;p&gt;Self generate the data by creating a long series of cosine waves of varying frequencies and amplitudes.&lt;/p&gt;&#xA;&lt;h3&gt;Machine Learning approach:&lt;/h3&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sequence input with X dimensions&lt;/li&gt;&#xA;&lt;li&gt;LSTM with Y hidden units&lt;/li&gt;&#xA;&lt;li&gt;Z fully connected layer&lt;/li&gt;&#xA;&lt;li&gt;Softmax&lt;/li&gt;&#xA;&lt;li&gt;Classification Output with cross entropy&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3&gt;Challenge:&lt;/h3&gt;&#xA;&lt;p&gt;One particular challenge I will face is that the network is likely to only learn to classify cosine waveforms based on the frequencies and amplitudes that appears in the training set.&lt;/p&gt;&#xA;&lt;p&gt;How should I modify my approach to make it less sensitive to the frequency and amplitude and more on the signature cosine waveform that we see?&lt;/p&gt;&#xA;" OwnerUserId="44312" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-08T13:53:44.153" Title="Machine Learning: Repetitive Pattern recognition regardless of amplitude and frequency" Tags="|time-series|lstm|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26409" PostTypeId="1" AcceptedAnswerId="30535" CreationDate="2018-01-08T16:19:33.187" Score="5" ViewCount="1730" Body="&lt;p&gt;I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum message size: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    scala&amp;gt; val mod = pipeline.fit(df)&#xA;    mod: org.apache.spark.ml.PipelineModel = pipeline_936bcade4716&#xA;    scala&amp;gt; mod.write.overwrite().save(modelPath.concat(&quot;model&quot;))&#xA;    18/01/08 10:00:32 WARN TaskSetManager: Stage 8 contains a task of very large size &#xA;    (755610 KB). The maximum recommended task size is 100 KB.&#xA;    org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task &#xA;    2606:0 was 777523713 bytes, which exceeds max allowed: spark.rpc.message.maxSize &#xA;    (134217728 bytes). Consider increasing spark.rpc.message.maxSize &#xA;    or using broadcast variables for large values.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Making the following assumptions about the problem area:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It's not possible to decrease the size of the model  &lt;strong&gt;AND&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;It's not possible to increase the maximum message size to a point where the pipeline would fit in a single message.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Are there any methods that would allow me to save the pipeline successfully to HDFS?&lt;/p&gt;&#xA;" OwnerUserId="2932" LastEditorUserId="2932" LastEditDate="2018-03-26T13:02:16.450" LastActivityDate="2021-05-28T12:47:43.933" Title="Saving Large Spark ML Pipeline to HDFS" Tags="|apache-spark|apache-hadoop|scala|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26410" PostTypeId="2" ParentId="26399" CreationDate="2018-01-08T16:20:27.620" Score="2" Body="&lt;p&gt;W is just the coefficients that we estimate to determine the contribution of each variable to the model. We attempt to estimate the log odds ratio by multiplying coefficients by the input variables that we have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we didn't have W, by default it would just be the 1 matrix multiplied by X, or each variable output would be assumed to affect the response equally (ignoring the scale differences between predictors).&lt;/p&gt;&#xA;" OwnerUserId="43397" LastActivityDate="2018-01-08T16:20:27.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26411" PostTypeId="1" CreationDate="2018-01-08T16:28:12.343" Score="4" ViewCount="1037" Body="&lt;p&gt;How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?&lt;/p&gt;&#xA;" OwnerUserId="44319" LastEditorUserId="29169" LastEditDate="2019-06-07T23:42:50.927" LastActivityDate="2019-06-07T23:42:50.927" Title="Evaluating new features" Tags="|machine-learning|feature-engineering|model-evaluations|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="26413" PostTypeId="2" ParentId="26411" CreationDate="2018-01-08T19:29:45.590" Score="2" Body="&lt;p&gt;Actually in these cases I usually plot the data based on features &lt;a href=&quot;https://datascience.stackexchange.com/a/25271/28175&quot;&gt;two by two&lt;/a&gt;. This approach is so much similar to watching &lt;code&gt;covariance matrix&lt;/code&gt;. In this way you should see the new added features whether they have correlation, whether they change linearly with each of previous features. Suppose that you have already one feature for a classification task. Then you may want to add another feature. You have to plot data with each feature as one axis and investigate whether they have linear correlation or not. If they don't have any correlation or their correlation is near to zero, you should add the feature because that may help you, they provide a kind of knowledge that the previous features didn't provide. If a new feature has correlation with the previous feature, it means that adding that does not help you have a new knowledge or perception of that concept. Although there are debates &lt;a href=&quot;https://datascience.stackexchange.com/q/24452/28175&quot;&gt;here&lt;/a&gt; I prefer not to add correlated features because of computation complexity. That will be so time consuming. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For illustrating more, suppose you have a data-set A = {X1, X2, y} in which X1 and X2 are the feature and y is the label and all are binary values. Also suppose that the covariance matrix between these is as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2CJKw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2CJKw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;We may infer that we can ignore X1 in the process of classification without loss of accuracy, if e is equal to zero which means label y is independent of X1, since they are binary valued. We can ignore X1 in classification, regardless of other values. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We may infer that we can ignore either one of features in the process of classification without loss of accuracy, if |d| &gt;&gt; 0 which means there is high correlation between X1 and X2 and hence information redundancy. So we may ignore one of them without problem.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We may infer that we can not ignore neither X1 nor X2 in the process of classification, if d = 0 and none of e or f are zero. Then the feature are independent and both are influential on label. So we may not ignore any of them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-15T12:19:26.107" LastActivityDate="2018-01-15T12:19:26.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26414" PostTypeId="1" CreationDate="2018-01-08T19:36:59.913" Score="1" ViewCount="162" Body="&lt;p&gt;I am implementing an attention network model over a categorical CQA dataset.&#xA;Following is my code for the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; class QACNN(Layer):&#xA;&#xA;    def __init__(self, vocab_size=MAX_NB_WORDS, embedding_size=emb_dim,filter_sizes=[1,2,3], num_filters=400, dropout_keep_prob=1.0,paras=None,learning_rate=1e-2,embeddings=None,trainable=True, **kwargs):&#xA;        self.learning_rate=learning_rate&#xA;        self.paras=paras&#xA;        self.filter_sizes=filter_sizes&#xA;        self.num_filters=num_filters&#xA;        self.dropout_keep_prob = dropout_keep_prob&#xA;        self.embeddings=embeddings&#xA;&#xA;&#xA;        self.embedding_size=embedding_size&#xA;        self.model_type=&quot;base&quot;&#xA;        self.num_filters_total=self.num_filters * len(self.filter_sizes)        &#xA;&#xA;        # Embedding layer&#xA;        self.updated_paras=[]&#xA;        with tf.name_scope(&quot;embedding&quot;):&#xA;            if self.paras==None:&#xA;                if self.embeddings ==None:&#xA;                    print (&quot;random embedding&quot;)&#xA;                    self.Embedding_W = tf.Variable(&#xA;                        tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),&#xA;                        name=&quot;random_W&quot;)&#xA;                else:&#xA;                    self.Embedding_W = tf.Variable(np.array(self.embeddings),name=&quot;embedding_W&quot; ,dtype=&quot;float32&quot;,trainable=trainable)&#xA;            else:&#xA;                print (&quot;load embeddings&quot;)&#xA;                self.Embedding_W=tf.Variable(self.paras[0],trainable=trainable,name=&quot;embedding_W&quot;)&#xA;            self.updated_paras.append(self.Embedding_W)&#xA;        super(QACNN, self).__init__(** kwargs)&#xA;&#xA;&#xA;    def build(self, input_shape): &#xA;            self.kernels=[]  &#xA;            self.input_dim = input_shape[-1]  &#xA;            for i, filter_size in enumerate(self.filter_sizes):&#xA;                with tf.name_scope(&quot;conv-maxpool-%s&quot; % filter_size):&#xA;                    filter_shape = [filter_size, self.embedding_size, 1, self.num_filters]&#xA;                    if self.paras==None:&#xA;                        W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name=&quot;kernel_W&quot;)&#xA;                        b = tf.Variable(tf.constant(0.1, shape=[self.num_filters]), name=&quot;kernel_b&quot;)&#xA;                        self.kernels.append((W,b))&#xA;                    else:&#xA;                        _W,_b=self.paras[1][i]&#xA;                        W=tf.Variable(_W)                &#xA;                        b=tf.Variable(_b)&#xA;                        self.kernels.append((W,b))   &#xA;                    self.updated_paras.append(W)&#xA;                    self.updated_paras.append(b)&#xA;&#xA;    def compute_mask(self, inputs, mask=None):&#xA;        mask = super(QACNN, self).compute_mask(inputs, mask)&#xA;        return mask                &#xA;&#xA;    def call(self,sentence, mask=None):&#xA;        embedded_chars_1 = tf.nn.embedding_lookup(self.Embedding_W, sentence)&#xA;        print &quot;embedded::&quot;,embedded_chars_1&#xA;        embedded_chars_expanded_1 = tf.expand_dims(embedded_chars_1, -1)&#xA;        print &quot;expanded embedded::&quot;,embedded_chars_expanded_1&#xA;        output=[]&#xA;        for i, filter_size in enumerate(self.filter_sizes): &#xA;            conv = tf.nn.conv2d(&#xA;                embedded_chars_expanded_1,&#xA;                self.kernels[i][0],&#xA;                strides=[1, 1, 1, 1],&#xA;                padding='VALID',&#xA;                name=&quot;conv-1&quot;&#xA;            )&#xA;            print &quot;conv::&quot;,conv&#xA;            h = tf.nn.relu(tf.nn.bias_add(conv, self.kernels[i][1]), name=&quot;relu-1&quot;)&#xA;            print &quot;relu::&quot;,h&#xA;            output.append(h)&#xA;        tf_reshape = tf.reshape(conv,[-1,int(h.shape[1]), int(h.shape[3])])    &#xA;        self.output_dim = [int(tf_reshape.shape[1]), int(tf_reshape.shape[2])]&#xA;        return tf_reshape&#xA;&#xA;    def compute_output_shape(self, input_shape):&#xA;        newShape = list(input_shape)&#xA;        newShape[1] = self.output_dim[0]&#xA;        newShape.append(self.output_dim[1])&#xA;        return tuple(newShape)&#xA;&#xA;&#xA;sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')&#xA;sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')&#xA;&#xA;qns_cnn = QACNN()(sequence_1_input)&#xA;ans_cnn = QACNN()(sequence_2_input)&#xA;print qns_cnn, ans_cnn &#xA;&#xA;## calculating Similarity&#xA;qns_perm = Permute((2,1))(qns_cnn)&#xA;dens_qns = Dense(58, kernel_initializer='random_uniform')(qns_perm)&#xA;ans_perm = Permute((2,1))(ans_cnn)&#xA;qns_ans_dens = merge([dens_qns, ans_perm], mode='dot', dot_axes=1)&#xA;print qns_ans_dens&#xA;&#xA;## Column max pooling&#xA;col_max = Permute((2,1))(qns_ans_dens)&#xA;col_max = GlobalMaxPooling1D()(col_max)&#xA;print col_max&#xA;&#xA;## Row-max pooling&#xA;row_max = GlobalMaxPooling1D()(qns_ans_dens)&#xA;print row_max&#xA;&#xA;## Attention&#xA;soft_col = Activation('softmax')(col_max)&#xA;soft_row = Activation('softmax')(row_max)&#xA;&#xA;## Final Representation&#xA;qns_soft = merge([qns_cnn, soft_col], mode='dot', dot_axes=1)&#xA;ans_soft = merge([ans_cnn, soft_row], mode='dot', dot_axes=1)&#xA;&#xA;## Cosine similarity&#xA;qns_ans_tensors = merge([qns_soft, ans_soft], mode='cos', dot_axes=-1)&#xA;dist = Lambda(lambda x: 1-x)(qns_ans_tensors)&#xA;dist = Dense(3,activation = 'softmax')(dist)&#xA;&#xA;## Model Training&#xA;model = Model(inputs = [sequence_1_input, sequence_2_input], outputs = dist)    &#xA;print model.summary()&#xA;adagrad = Adagrad(lr = 0.1)&#xA;sgd = SGD()&#xA;adam = Adam(lr = 0.01)&#xA;model.compile(optimizer=adagrad, loss='categorical_crossentropy', metrics=['acc'])&#xA;&#xA;model.fit([data_topic_body,data_cmt], score_train,&#xA;                       validation_data=[[data_topic_body_dev,data_cmt_dev], score_val],&#xA;                       epochs=80, batch_size=64)    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was following &lt;a href=&quot;https://arxiv.org/pdf/1602.03609.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; for the implementation of the above model .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems I am missing something here as the accuracy of the model is very poor. Am I doing something wrong here?         &lt;/p&gt;&#xA;" OwnerUserId="44328" LastEditorUserId="29575" LastEditDate="2018-01-08T19:53:55.407" LastActivityDate="2018-01-08T19:53:55.407" Title="Attention network over text dataset" Tags="|deep-learning|tensorflow|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26415" PostTypeId="1" CreationDate="2018-01-08T20:51:03.470" Score="0" ViewCount="237" Body="&lt;p&gt;I know next to nothing about Data Science. Where should I start? Where do you recommend I should take courses (hopefully free)? Or even a book. &lt;/p&gt;&#xA;" OwnerUserId="44331" LastEditorUserId="44331" LastEditDate="2018-01-08T20:54:51.817" LastActivityDate="2018-01-08T21:43:10.787" Title="Where to start with Data Science" Tags="|machine-learning|r|data-mining|" AnswerCount="1" CommentCount="6" ClosedDate="2018-01-09T08:46:47.723" ContentLicense="CC BY-SA 3.0" />
  <row Id="26417" PostTypeId="2" ParentId="26415" CreationDate="2018-01-08T21:16:24.987" Score="1" Body="&lt;p&gt;Well, I started studying in &lt;a href=&quot;https://www.datacamp.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Datacamp&lt;/a&gt; and I really recommend this &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine learning&lt;/a&gt; course it's very complete for new learners.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Udacity has this &lt;a href=&quot;https://classroom.udacity.com/courses/ud730&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep-learning&lt;/a&gt; course and it's a complement for the Coursera &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine learning&lt;/a&gt; course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's very important to research a lot in other sources but there you can paint a path for your &lt;strong&gt;Data Science&lt;/strong&gt; skills through these ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working and studying in Machine learning and I store the test projects in &lt;a href=&quot;https://github.com/haroldjcastillo/ml-test&quot; rel=&quot;nofollow noreferrer&quot;&gt;this repository&lt;/a&gt;, but is very important to have a little bit of knowledge of &lt;a href=&quot;https://www.python.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;python&lt;/a&gt; and &lt;a href=&quot;https://www.gnu.org/software/octave/&quot; rel=&quot;nofollow noreferrer&quot;&gt;octave&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Already exist other languages and tools like &lt;a href=&quot;https://www.r-project.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt; for test your data sets&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="43760" LastEditorUserId="29575" LastEditDate="2018-01-08T21:43:10.787" LastActivityDate="2018-01-08T21:43:10.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26419" PostTypeId="1" AcceptedAnswerId="26433" CreationDate="2018-01-08T23:33:46.703" Score="3" ViewCount="2908" Body="&lt;p&gt;So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, &lt;a href=&quot;http://adventuresinmachinelearning.com/word2vec-tutorial-tensorflow/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word2Vec in Tensorflow&lt;/a&gt; or &lt;a href=&quot;http://adventuresinmachinelearning.com/gensim-word2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word2Vec trained with Gensim&lt;/a&gt; ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In what cases would implementing it through the more manual first approach be useful vs. the second one? If there is already an easier way to train a word2vec model using gensim, why is that not used always? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, what is the benefit in using a pre-trained model like the &lt;a href=&quot;http://mccormickml.com/2016/04/12/googles-pretrained-word2vec-model-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google News dataset&lt;/a&gt;? What happens when there are words that are not included in the news dataset? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if this question is basic, I just want to get a clearer grasp of the overall picture. &lt;/p&gt;&#xA;" OwnerUserId="35399" LastActivityDate="2018-01-09T10:42:53.577" Title="When to use different Word2Vec training approaches?" Tags="|python|tensorflow|word2vec|word-embeddings|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26420" PostTypeId="2" ParentId="23901" CreationDate="2018-01-09T03:21:51.487" Score="4" Body="&lt;p&gt;A concept class &lt;strong&gt;&lt;em&gt;C&lt;/em&gt;&lt;/strong&gt; is a set of true functions &lt;em&gt;f&lt;/em&gt;. Hypothesis class &lt;strong&gt;&lt;em&gt;H&lt;/em&gt;&lt;/strong&gt; is the set of candidates to formulate as the final output of a learning algorithm to well approximate the true function &lt;em&gt;f&lt;/em&gt;. Hypothesis class &lt;strong&gt;&lt;em&gt;H&lt;/em&gt;&lt;/strong&gt; is chosen before seeing the data (training process). &lt;strong&gt;&lt;em&gt;C&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;H&lt;/em&gt;&lt;/strong&gt; can be either same or not and we can treat them independently. &lt;/p&gt;&#xA;" OwnerUserId="26163" LastEditorUserId="26163" LastEditDate="2018-01-09T03:29:48.280" LastActivityDate="2018-01-09T03:29:48.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26421" PostTypeId="1" AcceptedAnswerId="26426" CreationDate="2018-01-09T03:36:11.183" Score="2" ViewCount="1091" Body="&lt;p&gt;As a beginner at machine learning, I wanted to work on a small project in which the dataset has only 80 rows and 5 columns. The dataset I am working with is related to a medical condition, with 4 columns as biomarkers, and the 5th column indicates whether the row (a patient) has the condition or not.&#xA; So far, I have fitted the following 5 models (with accuracy and MCC scores):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;KNN (Accuracy: 43.5%, MCC:-0.164)&lt;/li&gt;&#xA;&lt;li&gt;Logistic Regression (Accuracy: 65.2%, MCC: .312)&lt;/li&gt;&#xA;&lt;li&gt;SVM (Accuracy: 60.9%, MCC: .214&lt;/li&gt;&#xA;&lt;li&gt;Random Forest (Accuracy: 86.95%, MCC: .769)&lt;/li&gt;&#xA;&lt;li&gt;Decision trees (Accuracy: 65.2%, MCC: .312)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have used 5-fold cross validation to prevent overfitting, and yet most of my models are underperforming. I was also considering ensembling and bootstrapping, but with these lacking results, I am not sure how effective they would be. Do you have any tips concerning either:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Better algorithms for small datasets&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Improvements I could make on the algorithms I have so far&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another method (e.g. Regularization)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44339" LastEditorUserId="29575" LastEditDate="2018-01-09T03:41:25.913" LastActivityDate="2018-01-09T06:43:52.543" Title="Performing machine learning on small datasets" Tags="|machine-learning|classification|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26424" PostTypeId="1" CreationDate="2018-01-09T05:46:40.543" Score="1" ViewCount="139" Body="&lt;p&gt;I am using spark als.train() to build my &lt;strong&gt;user-items&lt;/strong&gt; recommendation system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is I want to cover more item feature. So, I need to input 7 days user action data. But the als train become slow than just input 1 day data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, is it possible that I just input 1 day data, and compare the similarities between other runs(every time just input 1 day)?&lt;/p&gt;&#xA;" OwnerUserId="44343" LastEditorUserId="40853" LastEditDate="2018-01-09T08:47:18.893" LastActivityDate="2024-02-13T22:01:15.110" Title="Is spark als item feature comparable between several runs" Tags="|machine-learning|apache-spark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26425" PostTypeId="1" CreationDate="2018-01-09T06:09:12.590" Score="1" ViewCount="390" Body="&lt;p&gt;My dataframe contains 7000 rows with the following library login details: login time, libraryid (userid), login date, fined/not.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This data is for a local library, I want to create a model which predicts a persons visit to library. Say, the chance that the member with &lt;code&gt;libraryid=1015&lt;/code&gt; will visit the library tomorrow. The library has had many regular visitors over the past 2-3 yrs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = RandomForestClassifier(n_estimators=30)&#xA;clf.fit(df_train[features], df_train['a'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since the date is involved it is not getting the required result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;       login_time             id    login_date  fined&#xA;0      2016-02-29 23:28:58    1015  2016-02-29  1&#xA;1      2016-03-01 00:19:27    4890  2016-03-01  1&#xA;2      2016-03-01 04:40:17    1020  2016-03-01  0&#xA;3      2016-03-01 04:41:05    9134  2016-03-01  1&#xA;4      2016-03-01 05:00:27    7798  2016-03-01  1&#xA;5      2016-03-01 05:01:21    1325  2016-03-01  1&#xA;6      2016-03-01 05:02:22    5017  2016-03-01  1&#xA;7      2016-03-01 05:05:47    2730  2016-03-01  1&#xA;8      2016-03-01 05:09:57    8125  2016-03-01  1&#xA;9      2016-03-01 05:10:47    8604  2016-03-01  1&#xA;10     2016-03-01 05:16:26    9033  2016-03-01  1&#xA;11     2016-03-01 05:17:23    7096  2016-03-01  1&#xA;12     2016-03-01 05:18:02    1022  2016-03-01  1&#xA;13     2016-03-01 05:19:33    1015  2016-03-01  1&#xA;14     2016-03-01 05:19:48    3087  2016-03-01  1&#xA;15     2016-03-01 05:21:40    5641  2016-03-01  0&#xA;16     2016-03-01 05:28:25    5548  2016-03-01  1&#xA;17     2016-03-01 05:31:54    8970  2016-03-01  0&#xA;18     2016-03-01 05:58:52    7781  2016-03-01  1&#xA;19     2016-03-01 06:09:12    5859  2016-03-01  1&#xA;20     2016-03-01 06:13:28    1017  2016-03-01  1&#xA;21     2016-03-01 06:14:35    8138  2016-03-01  1&#xA;22     2016-03-01 06:15:41    1018  2016-03-01  1&#xA;23     2016-03-01 06:17:40    9355  2016-03-01  1&#xA;24     2016-03-01 06:18:10   10583  2016-03-01  1&#xA;25     2016-03-01 06:20:44    2394  2016-03-01  0&#xA;26     2016-03-01 06:29:17   10168  2016-03-01  1&#xA;27     2016-03-01 06:31:11    4235  2016-03-01  0&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used ARIMA model, the model is taking too much time for running, any other solutions please.&lt;/p&gt;&#xA;" OwnerUserId="43680" LastEditorUserId="43680" LastEditDate="2018-01-09T12:51:41.810" LastActivityDate="2018-01-09T12:51:41.810" Title="Predict whether or not a user will visit the library tomorrow using historical data" Tags="|python|scikit-learn|predictive-modeling|pandas|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26426" PostTypeId="2" ParentId="26421" CreationDate="2018-01-09T06:43:52.543" Score="1" Body="&lt;p&gt;With 5-fold cross validation, in each fold you're reducing your training dataset to 64 observations and evaluating against 16 observations. Assuming your data is balanced and you're stratifying folds, you're only giving your models 32 observations from each class to learn from, and misclassification of a single test set observation results in a 6.25% point change in accuracy for that fold. Even if it's classified correctly in the other folds, that single misclassification will still have a 1.25% point change on the across-folds average. Yikes. It should be no surprise your models aren't performing well: not only do they not have much data to train on, but your evaluation methodology is extremely unforgiving.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yeah, your data is pretty darn small. Off the top of my head, I can think of a handful of general strategies you can potentially use to address this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Use more of your data for training&lt;/strong&gt;. Instead of 5-fold CV, try leave-one-out or bootstrap.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Generate fake data&lt;/strong&gt;. This probably won't get you very far, but it's at least an option. Check out the SMOTE algorithm.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Transfer learning&lt;/strong&gt;. Depending on what you're doing, it might be possible to leverage a pre-trained model and then tweak it slightly to suit your needs. If this is an option it can be extremely powerful, but chances are this won't be something you can reasonably pursue. Anyway, here's an article demonstrating this on another small data medical example: a pre-trained general purpose image classifier was trained to detect cancer from just 600 images! &lt;a href=&quot;https://arxiv.org/abs/1711.10752&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1711.10752&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Go Bayesian&lt;/strong&gt;. Bayesian methods allow you to incorporate outside information as a &quot;prior belief&quot;. If you have subject matter expertise (or better yet, citations) that you can use to set your expectations for values of the model parameters or hyperparameters, bayesian methods will allow you to incorporate that information explicitly, and this can in turn help your model find good parameters faster since it doesn't need to learn everything about your problem directly from the little data available to it. If you're not careful, this can result in you giving yourself the model you want to have, in which case your evaluation metrics may not be as informative as you think they are. Here there be dragons: this approach is powerful, but it can be hard to do correctly, and playing with the prior is basically an invitation for people to be skeptical of your methods even if what you're doing is sound.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Get more data&lt;/strong&gt;. I'm guessing this isn't an option or you would have done so already. But if there's a chance it's out there: go find it. Try poking around the literature associated with this condition, maybe you'll get lucky and find a public dataset. If you're feeling bold, you could try emailing other researchers and just ask them politely if you can use their data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-09T06:43:52.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26427" PostTypeId="1" CreationDate="2018-01-09T06:45:41.613" Score="6" ViewCount="11517" Body="&lt;p&gt;How to extract Only Question/s from document with NLTK ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we categorise this Question into Y/N and details type answerable ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: I am one week old in NLTK ;-)&lt;/p&gt;&#xA;" OwnerUserId="44350" LastActivityDate="2018-01-09T17:18:27.137" Title="How to extract Question/s from document with NLTK?" Tags="|python|nltk|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26428" PostTypeId="2" ParentId="26425" CreationDate="2018-01-09T07:28:54.020" Score="5" Body="&lt;p&gt;First and foremost, you need to reformat your data into what's called a &lt;strong&gt;balanced panel&lt;/strong&gt; structure. For each day in your training data, each user should have a record for that day associated with an indicator variable for whether or not they visited. If every record in your training data corresponds to a visit, you're not giving those classifiers much to work with. You're model needs to know what a user's status looks like on days when they don't visit to be able to discriminate between days when they will and won't visit. The indicator variable, denoting &quot;this user visited today&quot;, is the target of your prediction. Not the date or time column, if that's what you were trying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you're going to need to do some &lt;strong&gt;feature engineering&lt;/strong&gt;. This process comprises the bulk of time and creativity expended on most data science projects, and the amount of effort you will put in will make or break your project. For each record in the data set, you should calculate things like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the day of the week/month/year?&lt;/li&gt;&#xA;&lt;li&gt;how many times did the user visit on this day of the week in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;how many times did the user visit on weekdays/weekends in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;How many days has it been since the users last visit?&lt;/li&gt;&#xA;&lt;li&gt;How many times has the user visited the library in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;how many times has the user visited the library and paid a fine in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;How many users visited in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;Is the library open this day (e.g. it could be a national holiday)?&lt;/li&gt;&#xA;&lt;li&gt;Are the public schools currently in session? On holiday? In finals?&lt;/li&gt;&#xA;&lt;li&gt;How many days since the beginning of the most recent semester?&lt;/li&gt;&#xA;&lt;li&gt;How many days until the semester ends?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Better yet, if you're not limited to what you've shown us...&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How many books are currently in the users possession? How many are late?&lt;/li&gt;&#xA;&lt;li&gt;If the user visited today, how much would they owe in fines?&lt;/li&gt;&#xA;&lt;li&gt;how many books did they check out in their last visit?&lt;/li&gt;&#xA;&lt;li&gt;how many books have they checked out in the last k days?&lt;/li&gt;&#xA;&lt;li&gt;how long has this person been a member at the library?&lt;/li&gt;&#xA;&lt;li&gt;has their membership expired?&lt;/li&gt;&#xA;&lt;li&gt;in the last k years, have they allowed their membership to expire?&lt;/li&gt;&#xA;&lt;li&gt;have they recently updated their address?&lt;/li&gt;&#xA;&lt;li&gt;How much have they donated in the last k years?&lt;/li&gt;&#xA;&lt;li&gt;What fraction of books checked out in the last 5 years by this user have belonged to each dewey decimal class?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are just some ideas off the top of my head. Don't let yourself be limited by my suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really only have the data you've shown us, a project you can try that would be more likely to give you good results would be to predict just the number of people who will visit the library on a particular day, rather than which actual people. If this sounds like it might be useful to you, check out Poisson models.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-09T07:37:48.337" LastActivityDate="2018-01-09T07:37:48.337" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26429" PostTypeId="2" ParentId="26427" CreationDate="2018-01-09T08:23:40.520" Score="4" Body="&lt;p&gt;Check out chapter 6 section 2.2 of the &lt;a href=&quot;http://www.nltk.org/book/ch06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK book&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: since apparently the community wants me to copy/paste stuff, here ya go:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2.2   Identifying Dialogue Act Types&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as &quot;I forgive you&quot; or &quot;I bet you can't climb that hill.&quot; But greetings, questions, answers, assertions, and clarifications can all be thought of as types of speech-based actions. Recognizing the dialogue acts underlying the utterances in a dialogue can be an important first step in understanding the conversation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The NPS Chat Corpus, which was demonstrated in 1, consists of over 10,000 posts from instant messaging sessions. These posts have all been labeled with one of 15 dialogue act types, such as &quot;Statement,&quot; &quot;Emotion,&quot; &quot;ynQuestion&quot;, and &quot;Continuer.&quot; We can therefore use this data to build a classifier that can identify the dialogue act types for new instant messaging posts. The first step is to extract the basic messaging data. We will call  xml_posts() to get a data structure representing the XML annotation for each post:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; posts = nltk.corpus.nps_chat.xml_posts()[:10000]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next, we'll define a simple feature extractor that checks what words the post contains:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; def dialogue_act_features(post):&#xA;...     features = {}&#xA;...     for word in nltk.word_tokenize(post):&#xA;...         features['contains({})'.format(word.lower())] = True&#xA;...     return features&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, we construct the training and testing data by applying the feature extractor to each post (using  post.get('class') to get a post's dialogue act type), and create a new classifier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; featuresets = [(dialogue_act_features(post.text), post.get('class'))&#xA;...                for post in posts]&#xA;&amp;gt;&amp;gt;&amp;gt; size = int(len(featuresets) * 0.1)&#xA;&amp;gt;&amp;gt;&amp;gt; train_set, test_set = featuresets[size:], featuresets[:size]&#xA;&amp;gt;&amp;gt;&amp;gt; classifier = nltk.NaiveBayesClassifier.train(train_set)&#xA;&amp;gt;&amp;gt;&amp;gt; print(nltk.classify.accuracy(classifier, test_set))&#xA;0.67&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-09T17:18:27.137" LastActivityDate="2018-01-09T17:18:27.137" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26430" PostTypeId="1" CreationDate="2018-01-09T09:23:54.573" Score="1" ViewCount="282" Body="&lt;p&gt;I want to build a classifier from a dataset of vectors that include exclusively boolean values. Is there any chances that my classifier might perform better if, previously to the learning, I add features that consist in combinations (with logic operators « OR » and « AND ») of the original ones or would that be pointless ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I have the following dataset :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            feature A   feature B   feature C&#xA;vector 1    True        False       True&#xA;vector 2    True        True        False&#xA;vector 3    False       True        True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Imagine I have the feeling that the fact that a data has True for both feature C and feature D would typically make it go in one particular class. Before learning the datas to build the classifier, should I add to each vector en extra feature computed by the logical operation « feature B AND feature C » :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            feature A   feature B   feature C   feature D&#xA;vector 1    True        False       True        False   &#xA;vector 2    True        True        False       False&#xA;vector 3    False       True        True        True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or would the relevance of « feature B AND feature C » be taken into account by the classifier anyway ? Does it depend on the algorithm (svm, Knn, etc. ) ?&lt;/p&gt;&#xA;" OwnerUserId="43650" LastActivityDate="2019-11-11T02:02:29.387" Title="adding logic combinations of boolean features in classification" Tags="|classification|feature-selection|binary|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26431" PostTypeId="2" ParentId="26419" CreationDate="2018-01-09T09:31:58.957" Score="2" Body="&lt;p&gt;Tensorflow has implementations for a pool of machine learning algorithms, so it should be comfortable if your application needs to build something on top of word2vec. Gensim is mainly intended for topic modelling techniques, but pretty robust as its their main work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get a clear grasp of how the algorithm works, then implementing manually makes sense. Else, just go with one of the implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Google word2vec model is pretty good and covers most of the English words. Use it, if you do not have computational power or time to train a model. Manual training gives you the freedom of choosing domain-specific dataset, window size, cbow or sg, length of vector. If there are Out of vocabulary(OOV) words, it will throw an error.&lt;/p&gt;&#xA;" OwnerUserId="16024" LastActivityDate="2018-01-09T09:31:58.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26432" PostTypeId="2" ParentId="26430" CreationDate="2018-01-09T10:35:07.280" Score="1" Body="&lt;p&gt;Try it and see what happens. Neural networks don't have enough representational power to learn an XOR operation without at least one hidden layer, so there are definitely some interesting features you can construct with logical operations. The AND operation is equivalent to multiplication, which corresponds to interaction terms linear models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But yeah, it does depend on the model. For example, a decision tree can learn these kinds of features on its own (although it won't necessarily). For exame, an AND operation would correspond to two tests on the same branch.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-09T10:35:07.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26433" PostTypeId="2" ParentId="26419" CreationDate="2018-01-09T10:42:53.577" Score="3" Body="&lt;p&gt;The advantage of using pre-trained vectors is being able to inject knowledge from a larger corpus than you might have access to: &lt;a href=&quot;https://code.google.com/archive/p/word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; has a vocabulary of 3 million words and phrases trained on the google news dataset comprising ~100 billion tokens, and there's no cost to you in training time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, they are fast and easy to use, just load the embeddings and look them up. It's straightforward to substitute different sets of pre-trained vectors (&lt;a href=&quot;https://fasttext.cc/&quot; rel=&quot;nofollow noreferrer&quot;&gt;fastText&lt;/a&gt;, &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GloVe&lt;/a&gt; etc) as one might be more suited to a particular use case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when your vocabulary does not have an entry in word2vec, by default you'll end up with a null entry in your embedding layer (depending on how you handle it). You'll need to consider the scale/impact and how to address it (keep/discard/&lt;a href=&quot;http://rutumulkar.com/blog/2015/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;consider online training&lt;/a&gt;). As yazhi says a decision must be made about how to handle out of vocabulary words. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The advantage of learning word vectors from your own corpus is that they would be derived from your dataset, so if you have reason to believe that the composition of your data is significantly different from the corpus used for the pre-trained vectors then that may result in better downstream performance. However, that comes at a cost in time taken to train your own vector representations. &lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2018-01-09T10:42:53.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26435" PostTypeId="1" AcceptedAnswerId="27608" CreationDate="2018-01-09T12:52:22.783" Score="1" ViewCount="144" Body="&lt;p&gt;I'm curious as to whether the training data for a sentiment analysis tool needs to be specifically geared toward the domain it's being using in. For example, the IMDB movie review data makes sense if I want to gauge sentiment on movie reviews, but would it still be okay to use if I want to gauge sentiment on what the general public thinks about a company's stock?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to gauge sentiment on a company's stock, should I create my own training set that uses data from financial news/social media reaction to companies?&lt;/p&gt;&#xA;" OwnerUserId="38917" LastActivityDate="2018-02-08T15:10:40.510" Title="Sentiment analysis using sources other than the IMDB data" Tags="|machine-learning|nlp|text-mining|sentiment-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26436" PostTypeId="2" ParentId="25635" CreationDate="2018-01-09T13:20:47.247" Score="2" Body="&lt;p&gt;A possible approach might be to use the most predictable and predictive word(s) in each cluster as its name(s). The following is inspired by Fisher's &lt;a href=&quot;https://en.wikipedia.org/wiki/Category_utility&quot; rel=&quot;nofollow noreferrer&quot;&gt;category utility&lt;/a&gt; used by the &lt;a href=&quot;https://link.springer.com/article/10.1023%2FA%3A1022852608280&quot; rel=&quot;nofollow noreferrer&quot;&gt;COBWEB algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An attribute is highly &lt;strong&gt;predictable&lt;/strong&gt; in a cluster $C_l$ if most elements in that cluster have the same value $a$ for it, thus a word represented by attribute $A_i$ has a predictability of $P(A_i=a|C_l)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An attribute is highly &lt;strong&gt;predictive&lt;/strong&gt; for a cluster $C_l$ if knowing it's value $a$ implies you can say with high certainty to which cluster it belongs, expressed as $P(C_l|A_i=a)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume now you process each label &lt;code&gt;org.java.somepackage.validateLogin&lt;/code&gt; as a sentence: &lt;em&gt;&quot;org java somepackage validate Login&quot;&lt;/em&gt; and apply &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot encoding&lt;/a&gt; to all your sentences in the dataset. The occurrence of a word is now represented by a value of 1 for it's corresponding attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task of representing each cluster by a word can now be formulated as finding for each cluster $C_l$ the word represented by attribute $A_i$ that equals 1 and has the highest predictability and predictiveness, weighted by the total probability of this word being in a sentence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ P(A_i=1|C_l)\times P(C_l|A_i=1) \times P(A_i=1)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which can be calculated by counting occurrence of a word per cluster and cluster per word.&lt;/p&gt;&#xA;" OwnerUserId="44357" LastEditorUserId="44357" LastEditDate="2018-01-09T13:44:40.260" LastActivityDate="2018-01-09T13:44:40.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26437" PostTypeId="2" ParentId="25635" CreationDate="2018-01-09T13:58:06.827" Score="1" Body="&lt;p&gt;Your research has yielded results like LDA, NMF, etc. But since you want to keep your concentration to short texts and generate short labels, I would recommend you to check this algorithm called: &lt;strong&gt;Bi Term Topic Model for Short Texts.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the paper from the author: &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.402.4032&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.402.4032&amp;amp;rep=rep1&amp;amp;type=pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is the code from the author again: &lt;a href=&quot;https://github.com/xiaohuiyan/BTM&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/xiaohuiyan/BTM&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used this personally and can very vouch for this for its performance. &lt;/p&gt;&#xA;" OwnerUserId="16401" LastActivityDate="2018-01-09T13:58:06.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26438" PostTypeId="2" ParentId="26435" CreationDate="2018-01-09T14:12:43.780" Score="0" Body="&lt;p&gt;If I understand your concern correctly, you want to know if you can perform sentiment analysis on sources other than imdb data (domains other than movie reviews that is). Short answer: &lt;strong&gt;YES!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Researchers in the past (even today) have tried to perform sentiment analysis on various aspects of life such as:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Opinions of public on software/apps/products - &lt;a href=&quot;http://ieeexplore.ieee.org/document/6912257/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/6912257/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Extracting rationale&#xA;   from development artifacts such as issue trackers, chat messages,&#xA;   emails, etc. One can mine whether developers are positive or&#xA;   negative about a feature being introduced or the steps taken to&#xA;   solve them. - &lt;a href=&quot;http://ieeexplore.ieee.org/document/8048891/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/8048891/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Here is a paper which also tries to make predictions on&#xA;   stock markets based on sentiment analysis from twitter data:&#xA;   &lt;a href=&quot;http://ieeexplore.ieee.org/document/7955659/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/7955659/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The challenge here would lie in the fact that you have to get some data for yourself if it already does not exists. &lt;/p&gt;&#xA;" OwnerUserId="16401" LastActivityDate="2018-01-09T14:12:43.780" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26440" PostTypeId="1" CreationDate="2018-01-09T14:36:16.137" Score="2" ViewCount="1042" Body="&lt;p&gt;I am trying to retrain inception final layer on new set of images. I am using docker TensorFlow image on Windows environment. Below are the steps that I am following.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Install docker toolbox for windows.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pulling the tensorflow docker image.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;docker run -it gcr.io/tensorflow/tensorflow:latest-devel&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Linking tensorflow to the folder on host machine where new set of images are stored.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;docker run -it -v &quot;$(pwd)&quot;:/tf_files/samples gcr.io/tensorflow/tensorflow:latest-devel&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;tf_files/samples contains the set of images on which I want to retrain the model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;cd /tensorflow&#xA;&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;git pull&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I am running the below script.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;python tensorflow/examples/image_retraining/retrain.py \&#xA;--bottleneck_dir=/tf_files/bottlenecks \&#xA;--how_many_training_steps 500 \&#xA;--model_dir=/tf_files/inception \&#xA;--output_graph=/tf_files/retrained_graph.pb \&#xA;--output_labels=/tf_files/retrained_labels.txt \&#xA;--image_dir /tf_files/samples&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Inception model is being downloaded properly but after that I am getting an error which says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;tensorflow.python.framework.errors_impl.PermissionDeniedError: /tf_files/samples&#xA;  /AppData/Local/Application Data; Operation not permitted&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Below is the exact error that I am getting.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;root@54808ee651a1:/tensorflow# python tensorflow/examples/image_retraining/retra&#xA;in.py \&#xA;&amp;gt; --bottleneck_dir=/tf_files/bottlenecks \&#xA;&amp;gt; --how_many_training_steps 500 \&#xA;&amp;gt; --model_dir=/tf_files/inception \&#xA;&amp;gt; --output_graph=/tf_files/retrained_graph.pb \&#xA;&amp;gt; --output_labels=/tf_files/retrained_labels.txt \&#xA;&amp;gt; --image_dir /tf_files/samples&#xA;Traceback (most recent call last):&#xA;  File &quot;tensorflow/examples/image_retraining/retrain.py&quot;, line 1326, in &amp;lt;module&amp;gt;&#xA;    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py&quot;, line 48, in run&#xA;    _sys.exit(main(_sys.argv[:1] + flags_passthrough))&#xA;  File &quot;tensorflow/examples/image_retraining/retrain.py&quot;, line 988, in main&#xA;    FLAGS.validation_percentage)&#xA;  File &quot;tensorflow/examples/image_retraining/retrain.py&quot;, line 144, in create_image_lists&#xA;    sub_dirs = [x[0] for x in gfile.Walk(image_dir)]&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py&quot;, line 518, in walk&#xA;    for subitem in walk(os.path.join(top, subdir), in_order):&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py&quot;, line 518, in walk&#xA;    for subitem in walk(os.path.join(top, subdir), in_order):&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py&quot;, line 518, in walk&#xA;    for subitem in walk(os.path.join(top, subdir), in_order):&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py&quot;, line 499, in walk&#xA;    listing = list_directory(top)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py&quot;, line 478, in list_directory&#xA;    compat.as_bytes(dirname), status)&#xA;  File &quot;/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py&quot;, line 473, in __exit__&#xA;    c_api.TF_GetCode(self.status.status))&#xA;tensorflow.python.framework.errors_impl.PermissionDeniedError: /tf_files/samples&#xA;/AppData/Local/Application Data; Operation not permitted&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44366" LastEditorUserId="1330" LastEditDate="2021-07-26T22:52:04.097" LastActivityDate="2021-07-26T22:52:04.097" Title="python.framework.errors_impl.permissiondeniederror" Tags="|python|tensorflow|image-classification|inception|windows|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26441" PostTypeId="1" CreationDate="2018-01-09T14:37:13.080" Score="2" ViewCount="228" Body="&lt;p&gt;I am little bit confused on why the scoring function that is the $f(X,W)$ is chosen to be $W,X$ while they talk about Softmax and SVM loss in &lt;a href=&quot;http://cs231n.github.io/linear-classify/#softmax&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can't they take Softmax classifier or SVM classifier and then explain the losses? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Was there a particular need of taking the above mentioned scoring function?&lt;/p&gt;&#xA;" OwnerUserId="38368" LastEditorUserId="29575" LastEditDate="2018-01-09T15:00:31.660" LastActivityDate="2020-05-08T15:00:25.027" Title="In CS231n lecture, can't the linear classifier be softmax itself?" Tags="|machine-learning|deep-learning|information-theory|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26443" PostTypeId="2" ParentId="26399" CreationDate="2018-01-09T14:50:38.883" Score="1" Body="&lt;p&gt;You use the inverse logit $f(x) = 1 / (1 + exp(- \omega x))$ because you want f(x) to be bound between between 0 and 1 (like a probability). If f(x) &gt; .5 you would infer 1, 0 otherwise.&lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2018-01-09T14:50:38.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26444" PostTypeId="1" AcceptedAnswerId="26509" CreationDate="2018-01-09T15:06:46.763" Score="4" ViewCount="1991" Body="&lt;p&gt;I use &lt;a href=&quot;https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cramer's V&lt;/a&gt; to calculate correlation of features in a dataset made of only nominal features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider the following dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a  |  b&#xA;--------&#xA;0  |  0&#xA;0  |  1&#xA;0  |  0&#xA;1  |  2&#xA;1  |  2&#xA;1  |  3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Calculating Cramer's V for features &lt;code&gt;a&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; yields 0.707. Since it's symmetric, there's information loss in this case - as we can see, knowing the value of &lt;code&gt;b&lt;/code&gt; means we know for sure what is the value of &lt;code&gt;a&lt;/code&gt;, but this is no the case if we are given the value of &lt;code&gt;a&lt;/code&gt;; in this case, the number possible values of &lt;code&gt;b&lt;/code&gt; decreases, but it's still not known for sure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to find an asymmetric metric that will provide this information for nominal values - meaning, will give a different value when calculated &lt;code&gt;a&lt;/code&gt; -&gt; &lt;code&gt;b&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; -&gt; &lt;code&gt;a&lt;/code&gt;. Is there anything like this?&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="43870" LastEditDate="2018-01-09T15:29:56.993" LastActivityDate="2019-12-28T02:10:45.533" Title="Is there an asymmetric version of nominal correlation?" Tags="|categorical-data|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26446" PostTypeId="1" AcceptedAnswerId="26453" CreationDate="2018-01-09T16:05:11.893" Score="12" ViewCount="23112" Body="&lt;p&gt;Most discussions of KNN mention Euclidean,Manhattan and Hamming distances, but they dont mention cosine similarity metric. Is there a reason for this?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2020-12-23T05:59:27.433" Title="Can I use cosine similarity as a distance metric in a KNN algorithm" Tags="|classification|recommender-system|cosine-distance|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26447" PostTypeId="2" ParentId="26446" CreationDate="2018-01-09T16:26:47.540" Score="0" Body="&lt;p&gt;If there does exist a reason it probably has to do with the fact the Cosine distance is not a proper distance metric.  Nevertheless, it's still a useful little thing.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-09T16:26:47.540" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26448" PostTypeId="1" CreationDate="2018-01-09T18:14:26.963" Score="1" ViewCount="648" Body="&lt;p&gt;I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The format and structure of the documents are unknown ahead of time. However, we can assume that the name of the document type occurs explicitly somewhere within the first 50 words of each document. E.g. A pathology report will contain the words &quot;pathology report&quot; somewhere near the top.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The contents of the documents are otherwise irrelevant. I simply want to cluster documents together by document type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know the names or number of document types ahead of time. I would like the algorithm to automatically determine the best number of clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not very experienced in machine learning, but I am a competent programmer. What's the best way to attack this problem? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="44375" LastActivityDate="2018-01-11T06:09:33.430" Title="Unsupervised clustering of unstructured text by document type" Tags="|clustering|unsupervised-learning|text|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26449" PostTypeId="1" AcceptedAnswerId="26464" CreationDate="2018-01-09T18:28:08.657" Score="16" ViewCount="4245" Body="&lt;p&gt;I'm a Computer Science engineer with no background in statistics or advanced math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm studying the book &lt;a href=&quot;https://sebastianraschka.com/books.html#python-machine-learning-3rd-edition&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Machine Learning&lt;/a&gt; by Raschka and Mirjalili, but when I tried to understand the math of the Machine Learning, I wasn't able to understand the great book that a friend suggest me &lt;a href=&quot;https://web.stanford.edu/~hastie/ElemStatLearn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know any easier statistics and math books for Machine Learning? If you don't, how should I move?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="94454" LastEditDate="2020-04-24T17:44:28.170" LastActivityDate="2021-03-11T20:03:31.197" Title="Beginner math books for Machine Learning" Tags="|machine-learning|statistics|reference-request|mathematics|esl|" AnswerCount="5" CommentCount="1" CommunityOwnedDate="2018-01-10T03:33:16.997" ContentLicense="CC BY-SA 4.0" />
  <row Id="26450" PostTypeId="2" ParentId="26441" CreationDate="2018-01-09T18:50:16.767" Score="1" Body="&lt;p&gt;The notation they're using is a bit funny. $f$ is just the dot product between the input and weights, $w^Tx_i$. The loss function is what differentiates those classifiers. Each loss function takes that dot product as an input and uses it in different ways. All of those loss functions compare the output of $f$ in some way with an associated $y$ value: you are computing your loss relative to a target variable to understand how well your model is performing, and then you can compute the gradient of the loss function to improve those predictions. If you ignore the loss function, you're just left with linear regression predictions (and even linear regression needs a loss function, MSE, to fit the weights) which aren't being evaluated relative to anything.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-09T18:50:16.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26451" PostTypeId="1" AcceptedAnswerId="36847" CreationDate="2018-01-09T19:14:18.727" Score="15" ViewCount="41195" Body="&lt;p&gt;Currently I code a GAN to generate MNIST numbers but the generator doesnt want to work. First I choose z with shape 100 per Batch, put into a layer to get into the shape (7,7, 256). Then conv2d_transpose layer&#xA;to into 28, 28, 1. (which is basically a mnist pic) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions &#xA;1.) This code doesn't work for obvious. Do you have any clue, why?&#xA;2.) I am very aware how transpose convolution works but I can't find any resource to calculate the output size given input, strides and kernel size specific to Tensorflow. The useful information I found is &lt;a href=&quot;https://arxiv.org/pdf/1603.07285v1.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1603.07285v1.pdf&lt;/a&gt; but e.g. padding in Tensorflow works very different. Can you help me?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mb_size = 32 #Size of image batch to apply at each iteration.&#xA;X_dim = 784&#xA;z_dim = 100&#xA;h_dim = 7*7*256&#xA;dropoutRate = 0.7&#xA;alplr = 0.2 #leaky Relu&#xA;&#xA;&#xA;def generator(z, G_W1, G_b1, keepProb, first_shape):&#xA;&#xA;    G_W1 = tf.Variable(xavier_init([z_dim, h_dim]))&#xA;    G_b1 = tf.Variable(tf.zeros(shape=[h_dim]))    &#xA;&#xA;&#xA;    G_h1 = lrelu(tf.matmul(z, G_W1) + G_b1, alplr)&#xA;    G_h1Drop = tf.nn.dropout(G_h1, keepProb)  # drop out&#xA;&#xA;    X = tf.reshape(G_h1Drop, shape=first_shape)&#xA;    out = create_new_trans_conv_layer(X, 256, INPUT_CHANNEL, [3, 3], [2,2], &quot;transconv1&quot;, [-1, 28, 28, 1])    &#xA;    return out&#xA;&#xA;&#xA;&#xA;&#xA;# new transposed cnn&#xA;def create_new_trans_conv_layer(input_data, num_input_channels, num_output_channels, filter_shape, stripe, name, output_shape):&#xA;    # setup the filter input shape for tf.nn.conv_2d&#xA;    conv_filt_shape = [filter_shape[0], filter_shape[1], num_output_channels, num_input_channels]&#xA;&#xA;&#xA;    # initialise weights and bias for the filter&#xA;    weights = tf.Variable(tf.truncated_normal(conv_filt_shape, stddev=0.03),&#xA;                          name=name + '_W')&#xA;    bias = tf.Variable(tf.truncated_normal([num_input_channels]), name=name + '_b')&#xA;&#xA;    # setup the convolutional layer operation&#xA;    conv1 = tf.nn.conv2d_transpose(input_data, weights, output_shape, [1, stripe[0], stripe[1], 1], padding='SAME')&#xA;&#xA;    # add the bias&#xA;    conv1 += bias&#xA;&#xA;    # apply a ReLU non-linear activation&#xA;&#xA;    conv1 = lrelu(conv1, alplr)&#xA;&#xA;    return conv1&#xA;&#xA;&#xA;...&#xA;&#xA;&#xA;    _, G_loss_curr = sess.run(&#xA;        [G_solver, G_loss],&#xA;        feed_dict={z: sample_z(mb_size, z_dim), keepProb: 1.0} #training generator&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43141" LastEditorUserId="29169" LastEditDate="2021-02-13T16:47:32.677" LastActivityDate="2021-02-13T16:47:32.677" Title="How to calculate the output shape of conv2d_transpose?" Tags="|neural-network|tensorflow|convolutional-neural-network|gan|generative-models|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26452" PostTypeId="2" ParentId="26342" CreationDate="2018-01-09T19:26:35.797" Score="1" Body="&lt;p&gt;First of all the weights update is derived using gradient descent. So the proper form of update is the first one you have. This is derived using math and satisfies that the updates are towards minimizing the Mean Square Error between true value and approximated one. For the true value we use a biased sample of the true value at the next time step plus the reward obtained at the current timestep: $r_t + \gamma \hat{v}_{t+1}$ which is your TD target (what you try to approximate). So the tricky part in RL with FA is that you try to approximate something which is also an approximation of a true quantity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not very sure what do you mean by making the value of γVt+1 closer to Vt−rt. For linear approximation, as you stated at the beginning of your question, the update needs to have the following form in order to reduce the MSE of the true value function: $\Delta w=\eta(v_\pi-\hat{v}_w)\phi(s)$. As I mentioned above because in RL we don't have knowledge of the true value, but only reward signals, we substitute the target (true value) with the target form I described in the previous paragraph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first proposed update, first of all, eliminates the discount factor which ensures that in an infinite horizon scenario the sum of rewards converges into a real value and not to infinity. Even if you consider episodic tasks the sign change makes your weights updating into a direction that doesn't follow the gradient of the MSE. In your second update again you use a an arbitrary rule for updating and not derived through gradient descent. By the way cutting in half the learning rate does not help in any sense as again you can define a new learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I refer you to these documents in which you can clarify how the update rule is defined and why it has that particular form and what are the optimality criteria and the role of the discount factor in RL:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/FA.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;RL with FA&lt;/a&gt;, &#xA;&lt;a href=&quot;https://pdfs.semanticscholar.org/968b/ab782e52faf0f7957ca0f38b9e9078454afe.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;2.3 Optimality Criteria and Discounting&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-01-09T19:26:35.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26453" PostTypeId="2" ParentId="26446" CreationDate="2018-01-09T19:26:56.283" Score="16" Body="&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; Cosine distance is not the overall best performing distance metric out there&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Although similarity measures are often expressed using a &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_(mathematics)&quot; rel=&quot;noreferrer&quot;&gt;distance metric&lt;/a&gt;, it is in fact a more flexible measure as it is not required to be symmetric or fulfill the triangle inequality. Nevertheless, it is very common to use a proper &lt;em&gt;distance metric&lt;/em&gt; like the Euclidian or Manhattan distance when applying nearest neighbour methods due to their &lt;a href=&quot;https://www2.ia-engineers.org/iciae/index.php/iciae/iciae2015/paper/viewFile/572/423&quot; rel=&quot;noreferrer&quot;&gt;proven performance&lt;/a&gt; on real world datasets. They will therefore be often mentioned in discussions of KNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find &lt;a href=&quot;https://arxiv.org/pdf/1708.04321.pdf&quot; rel=&quot;noreferrer&quot;&gt;this review&lt;/a&gt; from 2017 informative, it attempts to answer the question &lt;em&gt;&quot;which distance measures to be used for the KNN classifier among a large number of distance and similarity measures?&quot;&lt;/em&gt; They also consider inner-product metrics like the cosine distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, they conclude that (no surprise) no optimal distance metric can be used for all types of datasets, as the results show that each dataset favors a specific distance metric, and this result complies with the no-free-lunch theorem. It is clear that, among the metrics tested, the cosine distance isn't the overall best performing metric and even performs among the worst (lowest precision) in most noise levels. It does however  outperform other tested distances in 3/28 datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So can I use cosine similarity as a distance metric in a KNN algorithm?&lt;/strong&gt; &lt;strong&gt;Yes&lt;/strong&gt;, and for some datasets, like &lt;a href=&quot;https://en.wikipedia.org/wiki/Iris_flower_data_set&quot; rel=&quot;noreferrer&quot;&gt;Iris&lt;/a&gt;, it should even yield &lt;a href=&quot;https://arxiv.org/pdf/1708.04321.pdf&quot; rel=&quot;noreferrer&quot;&gt;better performance&lt;/a&gt; (p.30) compared as to Euclidian.&lt;/p&gt;&#xA;" OwnerUserId="44357" LastActivityDate="2018-01-09T19:26:56.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26454" PostTypeId="1" AcceptedAnswerId="26465" CreationDate="2018-01-09T19:52:15.317" Score="3" ViewCount="2236" Body="&lt;p&gt;I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would like to pick a use case from my field (finance). What use case would be suitable for graph algirithms and databases?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2019-09-30T10:34:54.580" Title="Use cases for graph algorithms and graph data structures in finance and banking" Tags="|apache-spark|graphs|neo4j|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26455" PostTypeId="1" CreationDate="2018-01-09T20:25:37.097" Score="2" ViewCount="141" Body="&lt;p&gt;I have a data set I'm trying to create a predictor model for.  The 5 features and outcome are all categorical data.  One of the features contains 5,000 unique levels. While the other 4 are all under 100 levels.  The output variable in my dataset contains 400 unique levels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I handle this scenario for modeling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My approach was going to be:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Convert the 5,000 level feature to a sequential number list via a lookup table.&lt;/li&gt;&#xA;&lt;li&gt;Convert the remaining 4 features using One Hot Encoding.&lt;/li&gt;&#xA;&lt;li&gt;Convert the output variable to a sequential number list via a lookup table.&lt;/li&gt;&#xA;&lt;li&gt;Use xgboost algorithm.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don't think I can bin the 5,000 level feature because they are user id's.  So to me, there is no way to bin them.&lt;/p&gt;&#xA;" OwnerUserId="44371" LastEditorUserId="44371" LastEditDate="2018-01-13T13:57:36.607" LastActivityDate="2018-01-13T13:57:36.607" Title="What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?" Tags="|predictive-modeling|xgboost|categorical-data|encoding|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26456" PostTypeId="2" ParentId="26455" CreationDate="2018-01-09T20:35:04.730" Score="3" Body="&lt;p&gt;Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users which have good eyesight will play basketball even if it rains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So try to think like that. &lt;/p&gt;&#xA;" OwnerUserId="44385" LastEditorUserId="29575" LastEditDate="2018-01-09T21:15:03.990" LastActivityDate="2018-01-09T21:15:03.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26457" PostTypeId="1" CreationDate="2018-01-09T21:33:24.960" Score="0" ViewCount="775" Body="&lt;p&gt;I'm trying to do one hot encoding on a data set containing 4 categorical features in R. Unique levels per feature:  400, 60, 6, 5, respectively. I get the following error during the first call to the ohe function (acm.disjonctif):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Error in sort.list(y) : 'x' must be atomic for 'sort.list'&#xA;Have you called 'sort' on a list?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't do sorting on a per column basis because it will remove the data relationship between columns.  What am I missing in understanding?  I searched on the web but couldn't find anything helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code I'm running in RStudio:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ohe_feats = c('RSN_CD', 'RSN_CAT', 'ELIG_SYS_CD', 'PROD_CD')&#xA;for (f in ohe_feats) {&#xA;    df_all_dummy = acm.disjonctif(as.factor(nodups[f]))   # package = ade4&#xA;    nodups[f] = NULL&#xA;    nodups = cbind(nodups, df_all_dummy)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44371" LastEditorUserId="26562" LastEditDate="2018-01-10T00:39:23.903" LastActivityDate="2018-02-01T02:57:05.043" Title="One hot encoding error &quot;sort.list(y)...&quot;" Tags="|r|categorical-data|encoding|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26459" PostTypeId="1" CreationDate="2018-01-09T22:16:53.293" Score="1" ViewCount="702" Body="&lt;p&gt;I am fairly new to the Machine Learning field, but I am working on building a multi-task network for a project I am working on. Unfortunately, despite all my best googling and digging through documentation, blogs and research papers, I am so far unable to get my network training on more than a few of the tasks at a time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For context, the problem entails a set of images of metallic test samples, all taken under standard lighting and at a standard distance. For confidentiality reasons, I cannot share the dataset itself. The goal of the network is to be able to score 3 different surface defects in 3 distinct regions each, a total of 9 outputs, each with an integer rating score from 0 to 5 (6 possible outcomes for each output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been writing the NN in Python using Keras and visualising the metrics and graph with TensorBoard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that, as I attempt to train the network, it only (at best) seems to learn for up to 4 of the 9 tasks, even though each image has a rating for each task. To illustrate my point, here are two screenshots of the TensorBoard metrics for one of the more successful training runs along with the graph showing the network structure:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PuU7o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PuU7o.png&quot; alt=&quot;TensorBoard page 1&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ejUhw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ejUhw.png&quot; alt=&quot;TensorBoard page 2&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HQT5K.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HQT5K.png&quot; alt=&quot;TensorBoard Graph&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, the tasks WCP_3T and WCP_5T are training well, but nothing else really is (the high accuracy on the WCP_F task is due to heavily weighted data, and not much else I'm afraid). The plateau in accuracy for each of the other tasks seems to me to be the network quickly finding the rating that is most common for that task, and remaining there. Thus, the varying accuracies for each task is an artifact of the data for that category.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried deepening (and shallowing) the other two main branches (BS and BD), I have experimented with altering learning rate, batch size, optimizer function (currently using Adam) and activation functions (currently using relu for the WCP branch and LeakyReLU for the other two, but I have also tried ordinary relu for them with no benefit). I have altered the number of filters in the various convolutional layers, and I have increased and decreased the dropout on various layers also. Still, nothing I seem to try can bring the network to train on all the tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible that the size of the dataset is a problem, but I only have a small number of images available and I don't think it is possible to get many more. I am currently using a training set of 506 images and a batch size of 32, which only leaves about 170 images for each testing and validation. Data augmentation to provide more images may also be out of the question as that could conceivably alter the scores placed on the different defects, and I would not know how (they were graded by experts, and the data was then passed to me for this task).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone has had a similar problem before and could suggest something else I could try, or could at least point me towards somewhere that might help to solve my problem, I would be extremely grateful!&lt;/p&gt;&#xA;" OwnerUserId="44340" LastActivityDate="2018-01-16T04:46:33.840" Title="Multitask learning NN only trains on a few tasks" Tags="|machine-learning|tensorflow|keras|multitask-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26460" PostTypeId="1" CreationDate="2018-01-10T00:22:05.137" Score="2" ViewCount="39" Body="&lt;p&gt;The basis of my question is that a CNN that does great on MNIST is far smaller than a CNN that does great on ImageNet. Clearly, as the number of potential target classes increases, along with image complexity (background, illumination, etc.), the network needs to become deeper and wider to be able to sufficiently capture all of the variation in the dataset. However, the downside of larger networks is that they become far slower for both inference and backprop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume you wanted to build a network that runs on a security camera in front of your house. You are really interested in telling when it sees a person, or a car in your driveway, or a delivery truck, etc. Let's say you have a total of 20 classes that you care about (maybe you want to know minivan, pickup, and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You gather a dataset that has plenty of nice, clean data. It has footage from lots of times of the day, with lots of intra-class variation and great balance between all of the classes. Finally, assume that you want this network to run at the maximum possible framerate (I know that security cameras don't need to do this, but maybe you're running on a small processor or some other reason that you want to be executing at really high speed).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any advantage, computationally, to splitting your network into smaller networks that specialize? One possibility is having a morning, an afternoon/evening, and a night network and you run the one corresponding to the time of day. Each one can detect all 20 classes (although you could split even farther and make it so that there is a vehicle one, and a person one, and so on). Your other option is sharing base layers (similar to using VGGNet layers for transfer learning). Then, you have the output of those base layers fed into several small networks, each specialized like above. Finally, you could also have just one large network that runs in all conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question: Is there a way to know which of these would be faster other than building them?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my head, it feels like sharing base layers and then diverging will run as slow as the &quot;sub-network&quot; with the most additional parameters. Similar logic for the separate networks, except you save a lot of computation by sharing base layers. Overall, though, it seems like one network is probably ideal. Is there any research/experimentation along these lines?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I also asked this question on AI.SE &lt;a href=&quot;https://ai.stackexchange.com/questions/4907/is-one-big-network-faster-than-several-small-ones&quot;&gt;here&lt;/a&gt; to no response. If I get one here, I'll link to it over there and vice versa.&lt;/p&gt;&#xA;" OwnerUserId="43846" LastActivityDate="2018-01-10T00:22:05.137" Title="Is one big network faster than several small ones?" Tags="|neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26462" PostTypeId="2" ParentId="24720" CreationDate="2018-01-10T01:50:15.177" Score="1" Body="&lt;p&gt;In my understanding, ranking means &lt;strong&gt;which one is more likely to be a positive sample&lt;/strong&gt;. So it's using for sorting test data for information retrieval or recommend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, each group should have one positive sample (may be more? I'm not sure).In the file &lt;code&gt;train.txt&lt;/code&gt;, &lt;strong&gt;the first line&lt;/strong&gt; is the instance label (you can see this in the &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/input_format.html#basic-input-format&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the group file, each line is one group's size. So the train data should in order by group, not shuffled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to your description, maybe &lt;code&gt;reg:logistic&lt;/code&gt; is what you want——describe x1 &gt; x2 &gt; x3 in [0,1] at the label column, training model and predicting, then groupby data an&lt;/p&gt;&#xA;" OwnerUserId="44397" LastEditorUserId="29575" LastEditDate="2018-01-10T02:19:12.207" LastActivityDate="2018-01-10T02:19:12.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26463" PostTypeId="2" ParentId="26449" CreationDate="2018-01-10T02:43:57.050" Score="2" Body="&lt;p&gt;Before doing my master in Analytics, I was suggested by my seniors to go through these couple of books to know more about Machine Learning and Statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Namely:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://drive.google.com/open?id=1PaZk1K99pG5MwbQxazaHCYsROyT8J6cG&quot; rel=&quot;nofollow noreferrer&quot;&gt;Discovering statistics with SPSS/R - Andy Field&lt;/a&gt;  &lt;/li&gt;&#xA;&lt;li&gt;R Beginner and R for Everyone&lt;/li&gt;&#xA;&lt;li&gt;Predictive Analytics - The Power to Predict Who Will Click, Buy, Lie or Die&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://drive.google.com/open?id=1xOgNZYE6J5Bfs1FmDg0JbYZDaoJQ0hFP&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science for Business&lt;/a&gt; and many more&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you cannot find these books online, do let me know will share the link, I have them on my drive. These books helped me in understanding the basics of stats with examples explained in layman terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are looking for some online courses, let me know can suggest you couple of good courses(most of them are free).&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-01-22T14:42:01.313" LastActivityDate="2018-01-22T14:42:01.313" CommentCount="7" CommunityOwnedDate="2018-01-10T03:33:16.997" ContentLicense="CC BY-SA 3.0" />
  <row Id="26464" PostTypeId="2" ParentId="26449" CreationDate="2018-01-10T03:55:05.917" Score="10" Body="&lt;p&gt;Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/course/intro-to-statistics--st101&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learn Statistics - Intro to Statistics Course&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/course/intro-to-descriptive-statistics--ud827&quot; rel=&quot;nofollow noreferrer&quot;&gt;Intro to Descriptive Statistics&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.udacity.com/course/intro-to-inferential-statistics--ud201&quot; rel=&quot;nofollow noreferrer&quot;&gt;Inferential Statistics: Learn Statistical Analysis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I want to recommend a book, I would recommend the following book which is free under &lt;em&gt;CC license&lt;/em&gt;. It has nice examples and is so much practical; moreover, there are lots of codes in it which help you feel statistics in real world examples. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://greenteapress.com/thinkstats2/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Think Python by Allen B. Downey&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://jakevdp.github.io/PythonDataScienceHandbook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Data Science Handbook&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also the following link may help: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.blog.google/topics/machine-learning/learn-google-ai-making-ml-education-available-everyone/amp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;From Google Itself Good And Concise&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-03-02T18:46:31.780" LastActivityDate="2018-03-02T18:46:31.780" CommentCount="5" CommunityOwnedDate="2018-01-10T03:55:05.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="26465" PostTypeId="2" ParentId="26454" CreationDate="2018-01-10T04:01:07.650" Score="5" Body="&lt;p&gt;There are many use cases of graph theory in Finance industry and it is a very broad question. As Emre said can be used for Fraud Detection, Risk Modelling, Economic Networks etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These below links can give you better understanding of different application, please go through for better understanding:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://jonathankinlay.com/2016/09/applications-graph-theory-finance/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Applications of Graph Theory In Finance&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.turingfinance.com/computational-model-of-systemic-risk-for-the-banking-industry/&quot; rel=&quot;nofollow noreferrer&quot;&gt; Graph Theory for Systemic Risk Models &lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://quantdare.com/graph-theory-finance/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graph theory: connections in the market&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.researchgate.net/publication/308264777_Analysis_of_Equity_Markets_A_Graph_Theory_Approach&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analysis of Equity Markets: A Graph Theory Approach&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.wolfram.com/2012/06/01/graph-theory-and-finance-in-mathematica/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Portfolio Diversification&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.springer.com/cda/content/document/cda_downloaddocument/9783540684077-c1.pdf?SGWID=0-0-45-624108-p173821480&quot; rel=&quot;nofollow noreferrer&quot;&gt;From Graph Theory to Models of Economic Networks. A Tutorial&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you need any additional information.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="83043" LastEditDate="2019-09-30T10:34:54.580" LastActivityDate="2019-09-30T10:34:54.580" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="26467" PostTypeId="2" ParentId="22045" CreationDate="2018-01-10T07:59:21.603" Score="1" Body="&lt;p&gt;I have the same problem, I was told to avoid Excel as it did not conform to the ISO 8601 standard of 2016-12-27. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I transferred the data to Google Sheets as illustrated in Orange tutorial YouTube videos. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dates were altered to match the required format however the results remain exactly the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XOwl9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XOwl9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/X0nFO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0nFO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/35u5P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/35u5P.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44408" LastEditorUserId="40853" LastEditDate="2018-01-10T15:23:21.737" LastActivityDate="2018-01-10T15:23:21.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26468" PostTypeId="1" CreationDate="2018-01-10T08:36:03.927" Score="10" ViewCount="2571" Body="&lt;p&gt;I would like to check JPG files if they were manipulated to change the content.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I consider NOT photoshopped:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cropping&lt;/li&gt;&#xA;&lt;li&gt;Rotating&lt;/li&gt;&#xA;&lt;li&gt;(Scaling)&lt;/li&gt;&#xA;&lt;li&gt;Image resolution&lt;/li&gt;&#xA;&lt;li&gt;Automatic changes smartphones might make&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I consider photoshopping:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adding a new image on top of parts of the old image&lt;/li&gt;&#xA;&lt;li&gt;Changing text of a part of an image&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can this automatically be checked?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(And: Are there ready-to-use libraries for it?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-01-16T19:31:08.900" Title="How can I detect if an image was photoshopped?" Tags="|computer-vision|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26469" PostTypeId="2" ParentId="26468" CreationDate="2018-01-10T08:41:34.087" Score="3" Body="&lt;p&gt;Error Level Analysis as described Error Level Analysis, found at &lt;a href=&quot;https://github.com/afsalashyana/FakeImageDetection&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/afsalashyana/FakeImageDetection&lt;/a&gt; seems to be one seems to be one way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You exploit that local compression ratios might be different. And it seems to be possible to train neural networks on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't find a paper which says how well this works so far&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-01-10T08:47:15.277" LastActivityDate="2018-01-10T08:47:15.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26471" PostTypeId="1" AcceptedAnswerId="26474" CreationDate="2018-01-10T11:03:39.813" Score="3" ViewCount="4804" Body="&lt;p&gt;After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of on-policy) methods. Please correct me if I am misled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) With an on-policy algorithm we use the current policy (a regression model with weights &lt;strong&gt;&lt;em&gt;W&lt;/em&gt;&lt;/strong&gt;, and ε-greedy selection) to generate the next state's &lt;strong&gt;&lt;em&gt;Q&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's &lt;strong&gt;&lt;em&gt;Q&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since &lt;strong&gt;&lt;em&gt;Q&lt;/em&gt;&lt;/strong&gt; is derived using the same greedy policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eYGX5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eYGX5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://courses.engr.illinois.edu/cs440/fa2009/lectures/Lect24.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://courses.engr.illinois.edu/cs440/fa2009/lectures/Lect24.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One more reading: &lt;a href=&quot;http://mi.eng.cam.ac.uk/~mg436/LectureSlides/MLSALT7/L3.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://mi.eng.cam.ac.uk/~mg436/LectureSlides/MLSALT7/L3.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42953" LastActivityDate="2018-08-13T09:05:15.690" Title="Is my understanding of On-Policy and Off-Policy TD algorithms correct?" Tags="|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26472" PostTypeId="2" ParentId="26449" CreationDate="2018-01-10T12:08:59.953" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://math.mit.edu/%7Egs/linearalgebra/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Linear Algebra&lt;/a&gt;  is a good starting point. Make sure you are good with probability theory, linear algebra, and statistics. A very in depth knowledge may not be necessary, but having a good knowledge is required.&lt;/p&gt;&#xA;" OwnerUserId="43590" LastEditorUserId="29169" LastEditDate="2021-03-11T20:03:31.197" LastActivityDate="2021-03-11T20:03:31.197" CommentCount="1" CommunityOwnedDate="2018-01-10T12:08:59.953" ContentLicense="CC BY-SA 4.0" />
  <row Id="26473" PostTypeId="2" ParentId="26302" CreationDate="2018-01-10T12:20:52.390" Score="1" Body="&lt;p&gt;There is no generalized algorithm to match two categories of data where the dimension is different and perhaps the attribute could also be different. However you can transform the data into similar dimension. One method I would suggest you is to take PCA of both the data.  After taking PCA you can pick the first k principal component of both the data making them virtually identical data with similar attributes. Then you can use k-nearest neighbour on the transformed data of Aisles to train your model. The model could be easily used to test the transformed data of products. &lt;/p&gt;&#xA;" OwnerUserId="43590" LastActivityDate="2018-01-10T12:20:52.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26474" PostTypeId="2" ParentId="26471" CreationDate="2018-01-10T12:28:16.120" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;1) With an on-policy algorithm we use the current policy (a regression model with weights W, and ε-greedy selection) to generate the next state's Q.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. To avoid confusion, it may be better to use the terms &quot;behaviour policy&quot; for the the policy that controls current actions and &quot;target policy&quot; for the policy being evaluated and/or learned.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's Q.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sort of. The only &lt;em&gt;requirement&lt;/em&gt; for an algorithm to be &lt;em&gt;off-policy&lt;/em&gt; is that the target policy is different to the behaviour policy. The usual target policy in Q-learning is not necessarily a greedy version of the behaviour policy, but is the maximising policy over Q. However, if the behaviour policy is ε-greedy over Q, and adapting to updates in Q, then yes your statement holds.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since Q is derived using the same greedy policy.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is true when comparing SARSA with Q learning, but may not hold when looking at other algorithms. This greedy-only action selection would not be a very efficient learner in all environments.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Experience replay is not &lt;em&gt;directly&lt;/em&gt; related to on-policy vs off-policy learning. Technically though, yes when the experience is stored and used later, that makes it off-policy for SARSA if Q values have changed enough between the sample and current parameters of the learning agent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you will see experience replay used more often with off-policy methods, since off-policy learners that boot-strap (i.e. use Q value of next state/action to help estimate current Q value) are less stable when used with function approximators. Experience replay helps to address that problem.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="57586" LastEditDate="2018-08-13T09:05:15.690" LastActivityDate="2018-08-13T09:05:15.690" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="26475" PostTypeId="1" AcceptedAnswerId="26481" CreationDate="2018-01-10T13:07:47.997" Score="48" ViewCount="33003" Body="&lt;p&gt;Activation functions are used to introduce non-linearities in the linear output of the type &lt;code&gt;w * x + b&lt;/code&gt; in a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which I am able to understand intuitively for the activation functions like sigmoid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation. However, I am not able to understand why is ReLU used as an activation function if its output is linear?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doesn't the whole point of being the activation function get defeated if it won't introduce non-linearity?&lt;/p&gt;&#xA;" OwnerUserId="16807" LastEditorUserId="29575" LastEditDate="2018-01-10T14:20:34.830" LastActivityDate="2022-02-23T06:27:04.127" Title="Why is ReLU used as an activation function?" Tags="|machine-learning|neural-network|deep-learning|activation-function|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26476" PostTypeId="2" ParentId="26475" CreationDate="2018-01-10T13:32:27.317" Score="13" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not completely true. The neurons are not dead. If you use sigmoid-like activations, after some iterations the value of gradients saturate for most the neurons. The value of gradient will be so small and the process of learning happens so slowly. This is vanishing and exploding gradients that has been in sigmoid-like activation functions. Conversely, the dead neurons may happen if you use &lt;code&gt;ReLU&lt;/code&gt; non-linarity, which is called &lt;a href=&quot;https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks&quot;&gt;&lt;em&gt;dying ReLU&lt;/em&gt;&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I am not able to understand why is ReLU used as an activation function if its output is linear&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Definitely it is not linear. As a simple definition, linear function is a function which has same derivative for the inputs in its domain.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://www.columbia.edu/itc/sipa/math/linear.html&quot; rel=&quot;noreferrer&quot;&gt;The linear function&lt;/a&gt; is popular in economics. It is attractive because it is simple and easy to handle mathematically. It has many important applications. Linear functions are those whose graph is a straight line. A linear function has the following property:&lt;br /&gt;&lt;br /&gt;&#xA;  &lt;span class=&quot;math-container&quot;&gt;$f(ax + by) = af(x) + bf(y)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Why-is-ReLU-non-linear&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;ReLU&lt;/code&gt; is not linear&lt;/a&gt;. &lt;em&gt;The simple answer is that &lt;code&gt;ReLU&lt;/code&gt;'s output is not a straight line, it bends at the x-axis. The more interesting point is what’s the consequence of this non-linearity. In simple terms, linear functions allow you to dissect the feature plane using a straight line. But with the non-linearity of &lt;code&gt;ReLU&lt;/code&gt;s, you can build arbitrary shaped curves on the feature plane.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ReLU&lt;/code&gt; may have a disadvantage which is its expected value. There is no limitation for the output of the &lt;code&gt;Relu&lt;/code&gt; and its expected value is not zero. &lt;code&gt;Tanh&lt;/code&gt; was more popular than &lt;code&gt;sigmoid&lt;/code&gt; because its expected value is equal to zero and learning in deeper layers occurs more rapidly. Although &lt;code&gt;ReLU&lt;/code&gt; does not have this advantage &lt;code&gt;batch normalization&lt;/code&gt; solves &lt;a href=&quot;https://datascience.stackexchange.com/q/23493/28175&quot;&gt;this problem&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also refer &lt;a href=&quot;https://stats.stackexchange.com/q/299915/179078&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stats.stackexchange.com/q/141960/179078&quot;&gt;here&lt;/a&gt; for more information.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2020-04-16T18:32:05.383" LastActivityDate="2020-04-16T18:32:05.383" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26477" PostTypeId="1" AcceptedAnswerId="26500" CreationDate="2018-01-10T13:36:22.847" Score="1" ViewCount="2084" Body="&lt;p&gt;How do I get from a dataframe with multiple columns that have similar values and need to be merged:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1 = pd.DataFrame({'firstcolumn':['ab', 'ca', 'da', 'ta','la'],&#xA;'secondcolumn':['ab', 'ca', 'ta', 'da', 'sa'], 'index':[2011,2012,2011,2012,2012]})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To a crosstab that tells me for each year how many values were collected?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Index ab ca da ta sa la&#xA;2011 2  0  1  1  0  0&#xA;2012 0  2  1  1  1  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, how could then plot the table? &lt;/p&gt;&#xA;" OwnerUserId="44424" LastEditorUserId="11097" LastEditDate="2018-01-11T09:13:09.713" LastActivityDate="2018-01-13T16:10:02.657" Title="How to merge columns, value count them and then plot the results?" Tags="|python|pandas|plotting|descriptive-statistics|counts|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26478" PostTypeId="1" AcceptedAnswerId="26482" CreationDate="2018-01-10T13:37:01.603" Score="3" ViewCount="9720" Body="&lt;p&gt;I have some categorical data which also contains '?' as data in some rows. I need to filter those rows depending on '?', that which row contain that instances will be deleted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7zUVI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7zUVI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mct4B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mct4B.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to drop those rows by applying these command but I failed.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = train.drop[~train.str.contains('\?')]&#xA;train = train.drop[train['?']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How could I identify those rows which contain '?' instance and drop those row ?&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="28175" LastEditDate="2018-01-10T15:39:29.140" LastActivityDate="2018-01-10T15:59:52.263" Title="Filter row depending on specific object value and delete those instances" Tags="|python|data-mining|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26479" PostTypeId="1" AcceptedAnswerId="26510" CreationDate="2018-01-10T13:51:59.893" Score="2" ViewCount="7113" Body="&lt;p&gt;I m new in text analysis and need your advice to help medical students to write properly and correctly. The students describe sicknesses as they  observe them; however, they must use an &quot;official sickness description&quot;. I have the data collected from the students and the correct data. Let me explain:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a csv table1 containing 300K rows / one column : each row is the description of health condition written by students ( they can be redundant). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have an other table2 containing almost 200K rows and 3 columns&#xA;Column 1: the official name of the condition ( the correct one the student should use)&#xA;Column 2: a code ( number) &#xA;Column 3: the translation of &quot;the official name of the condition&quot; in another language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am confused:&#xA;Should I go for a recommendation engine or a search/rank algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the steps you can identify to accomplish this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: the final output might be an API where student start to write the description of the sickness and obtain a list of choices/recommendations where they can pick one. &#xA;I hope I was clear ! Thanks &lt;/p&gt;&#xA;" OwnerUserId="40426" LastEditorUserId="30472" LastEditDate="2018-01-10T19:37:56.553" LastActivityDate="2018-01-11T08:52:00.997" Title="Text mining for text matching" Tags="|machine-learning|python|data-mining|nlp|text-mining|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26480" PostTypeId="2" ParentId="9292" CreationDate="2018-01-10T13:55:33.260" Score="1" Body="&lt;p&gt;You can use time as an additional attribute of the data. If you are having monthly data then your time feature could have 12 different values from 0 to 11. If you are having quarterly data your time attribute could have 4 different value from 0 to 3. You can refer &lt;a href=&quot;http://otexts.org/fpp2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Forecasting: Principles and Practice&lt;/a&gt; for more clarification. It's a good book.&lt;/p&gt;&#xA;" OwnerUserId="43590" LastEditorUserId="29575" LastEditDate="2018-01-10T14:23:16.647" LastActivityDate="2018-01-10T14:23:16.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26481" PostTypeId="2" ParentId="26475" CreationDate="2018-01-10T14:15:18.940" Score="57" Body="&lt;p&gt;In mathematics (linear algebra) a function is considered linear whenever a function&lt;span class=&quot;math-container&quot;&gt;$f: A  \rightarrow B$&lt;/span&gt; if for every &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; in the domain &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; has the following property:  &lt;span class=&quot;math-container&quot;&gt;$f(x) + f(y) = f(x+y)$&lt;/span&gt;.  By definition the ReLU is &lt;span class=&quot;math-container&quot;&gt;$max(0,x)$&lt;/span&gt;.  Therefore, if we split the domain from &lt;span class=&quot;math-container&quot;&gt;$(-\infty, 0]$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$[0, \infty)$&lt;/span&gt; then the function is linear.  However, it's easy to see that &lt;span class=&quot;math-container&quot;&gt;$f(-1) + f(1) \neq f(0)$&lt;/span&gt;.  Hence by definition ReLU is not linear.&lt;/p&gt;&#xA;&lt;p&gt;Nevertheless, ReLU is so close to linear that this often confuses people and wonder how can it be used as a universal approximator.  In my experience, the best way to think about them is like Riemann sums.  You can approximate any continuous functions with lots of little rectangles.  ReLU activations can produced lots of little rectangles.  In fact, in practice, ReLU can make rather complicated shapes and approximate many complicated domains.&lt;/p&gt;&#xA;&lt;p&gt;I also feel like clarifying another point.  As pointed out by a previous answer, neurons do not die in Sigmoid, but rather vanish.  The reason for this is because at maximum the derivative of the sigmoid function is .25.  Hence, after so many layers you end up multiplying these gradients and the product of very small numbers less than 1 tend to  go to zero very quickly.&lt;/p&gt;&#xA;&lt;p&gt;Hence if you're building a deep learning network with a lot of layers, your sigmoid functions will essentially stagnant rather quickly and become more or less useless.&lt;/p&gt;&#xA;&lt;p&gt;The key take away is the vanishing comes from multiplying the gradients not the gradients themselves.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="85064" LastEditDate="2021-04-20T14:52:50.427" LastActivityDate="2021-04-20T14:52:50.427" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26482" PostTypeId="2" ParentId="26478" CreationDate="2018-01-10T14:27:42.163" Score="5" Body="&lt;p&gt;You can replace &lt;code&gt;?&lt;/code&gt; with &lt;code&gt;nan&lt;/code&gt; and use &lt;code&gt;dropna()&lt;/code&gt;. This will work if you don't already have rows with &lt;code&gt;nan&lt;/code&gt; entries that you want to keep.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = train.replace('?', np.nan).dropna()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another option is to filter rows where any value is &lt;code&gt;?&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = train[~(train == '?').any(axis=1)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After looking at your data I found the problem: Your csv file has spaces after the commas, so the rows containing &lt;code&gt;?&lt;/code&gt; have a leading space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use &lt;code&gt;train = pd.read_csv('adult.data', sep=', ', engine='python')&lt;/code&gt; to read your data then either of the above methods will work.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-01-10T15:59:52.263" LastActivityDate="2018-01-10T15:59:52.263" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26483" PostTypeId="2" ParentId="26478" CreationDate="2018-01-10T14:42:24.280" Score="2" Body="&lt;p&gt;The answer given by Imran is correct and more general. It will allow you to drop any row containing '?' in any column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Posting here only to explain why your two code attempts don't work. In general, for sub-setting rows, you need boolean labels for each row indicating whether to include it or not. So, you should write your expression such that it returns those boolean masks. Your attempt was almost correct, but you ran .str.contains on the entire data frame. Instead, you should run it on individual columns. For conditions on more than one columns, combine. I demonstrate in the code below.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create sample data containing missing values    &#xA;tmp =  pd.DataFrame({'a' : ['Craft-repair', 'Sales', '?'], 'b' : [40, '?', 60]})&#xA;print ~(tmp['a'].str.contains('\?')) # how boolean mask is generated&#xA;tmp = tmp[~(tmp['a'].str.contains('\?'))] # filter on single column&#xA;print tmp[~ ((tmp['a'].str.contains('\?')) | (tmp['b'].str.contains('\?')))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the '?' was present in the data while reading it, you can also replace it by np.nan at the time of reading, making it easier to drop those rows using dropna later.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastActivityDate="2018-01-10T14:42:24.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26484" PostTypeId="1" CreationDate="2018-01-10T14:49:34.660" Score="1" ViewCount="99" Body="&lt;p&gt;I recently read a few papers, especially&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://papers.nips.cc/paper/6859-elf-an-extensive-lightweight-and-flexible-research-platform-for-real-time-strategy-games.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ELF: An Extensive, Lightweight, and Flexible&#xA;Research Platform for Real-time Strategy Games&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1709.02878.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow Agents:  Efficient Batched Reinforcement Learning in TensorFlow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As far as I can understand, ELF uses two separate environments (C++ and Python), because they say&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Most existing RL environments (OpenAI Gym, ...) provide Python interfaces which wrap only single game instances.  As a result, parallelism needs to be built in Python when applying modern RL methods. However, thread-level parallelism in Python can only poorly utilize multi-core processors, due to the Global Interpreter Lock (GIL). Process-level parallelism will also introduce extra data exchange overhead between processes and increase complexity to framework design.  In contrast, our parallelism is achieved with C++ threads for better scaling on multi-core CPUs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The team from DeepMind (which developed the TensorFlow Agent lib) on the other hand explain it as such:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Our implementation is completely defined within the TensorFlow graph.&#xA;  First, we parallelize the forward pass of the neural network by vectorizing the&#xA;  agent computation to produce a batch of actions from a batch of observations. [...] Using a batch size during inference allows us to leverage the internal&#xA;  thread pool of the TensorFlow session, or hardware accelerators such as GPUs and&#xA;  TPUs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, how are those two different? Sure, ELF uses C++, because they say doing it in Python is tedious and not efficient. TF agents uses &quot;vectorization&quot; and runs everything in the TensorFlow graph. I know, TF uses C underneath and a lot of Protobuffers to communicate data between Python &amp;amp; the underlying C code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So did FB with their ELF just reinvent the wheel or did they do something else that is clever and the TF team didn't adopt?&lt;/p&gt;&#xA;" OwnerUserId="44429" LastEditorUserId="44429" LastEditDate="2018-01-14T10:49:40.823" LastActivityDate="2018-01-14T10:49:40.823" Title="What are the major differences between Facebook's ELF RF framework and TensorFlows &quot;Agents&quot;?" Tags="|python|tensorflow|reinforcement-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26485" PostTypeId="2" ParentId="26455" CreationDate="2018-01-10T14:59:30.823" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't think I can bin the 5,000 level feature because they are user id's.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Unless there are many training examples per user, using &lt;code&gt;user_id&lt;/code&gt; as a feature will most likely result in overfitting. This should be fairly easy to verify using a tree model that can handle categorical features of high cardinality, as you suggested.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach I have seen in the past is to only label the top &lt;code&gt;M&lt;/code&gt; most frequent users and label everyone else &lt;code&gt;other&lt;/code&gt;, so you only have &lt;code&gt;M+1&lt;/code&gt; possible labels for the category. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another is to cluster users into &lt;code&gt;N&lt;/code&gt; groups based on some useful similarity measure. You will probably have to apply some domain knowledge to come up with a good similarity metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you expect the top &lt;code&gt;M&lt;/code&gt; users to be present in a significant percentage of future examples then this might be useful, otherwise it would be better to cluster users based on other features. In the latter case you compute the user's cluster membership at prediction time and use that cluster label as an input feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should compare all models to one that doesn't use &lt;code&gt;user_id&lt;/code&gt; at all to gauge its usefulness.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Convert the 5,000 level feature to a sequential number list via a lookup table.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I understand this correctly you want to convert this feature to a single number like an integer from &lt;code&gt;1&lt;/code&gt; to &lt;code&gt;5000&lt;/code&gt; to reduce memory usage. This will not affect the results you get from a tree model, but most other models will now interpret the labels as having a natural ordering and an importance relative to the size of the number representing them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a few possible tricks for encoding high-cardinality categorical features as a single number (or a lower-dimensional vector).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Encode each &lt;code&gt;user_id&lt;/code&gt; as the number of times it appears in the training set&lt;/li&gt;&#xA;&lt;li&gt;Encode each &lt;code&gt;user_id&lt;/code&gt; as the mean of the target value in the training set. See &quot;Target-based encoding&quot; &lt;a href=&quot;http://www.saedsayad.com/encoding.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If the target is categorical you can use a vector of frequencies for each target category, eg `[0.4, 0.1, 0.3, 0.2].&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As mentioned, these and other methods will only be useful if each user tends to appear multiple times in the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also see the &lt;a href=&quot;https://github.com/scikit-learn-contrib/categorical-encoding&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical-encoding&lt;/a&gt; library for more ideas.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-01-10T19:16:38.297" LastActivityDate="2018-01-10T19:16:38.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26486" PostTypeId="2" ParentId="25418" CreationDate="2018-01-10T15:15:09.243" Score="2" Body="&lt;p&gt;There is no single universally accept standard, however, many packages use edgelist for sparse (un)directed (un)weighted graphs. Regarding storing and sharing, I usually see compressed text files (in ascii format) instead of binary ones, due to binary compatibility problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you should take a look at this draft: &lt;a href=&quot;https://arxiv.org/abs/1503.02781&quot; rel=&quot;nofollow noreferrer&quot;&gt;Unravelling Graph-Exchange File Formats&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44050" LastEditorUserId="29575" LastEditDate="2018-01-10T15:42:27.287" LastActivityDate="2018-01-10T15:42:27.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26487" PostTypeId="2" ParentId="24720" CreationDate="2018-01-10T15:50:10.107" Score="2" Body="&lt;p&gt;Since I asked this question, it seems &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/input_format.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the documentation was expanded&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to provide a &quot;.group&quot; file. For example, the file could be&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;2&#xA;3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;  &#xA;  &lt;p&gt;This means that, the data set contains 5 instances, and the first two&#xA;  instances are in a group and the other three are in another group. The&#xA;  numbers in the group file are actually indicating the number of&#xA;  instances in each group in the instance file in order.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2018-01-10T15:50:10.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26488" PostTypeId="1" AcceptedAnswerId="26492" CreationDate="2018-01-10T16:01:03.573" Score="3" ViewCount="2336" Body="&lt;p&gt;This is something I can't achieve with the reshape2 library for R. I have the following data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; zone       code        literal&#xA; 1: A         14           bicl&#xA; 2: B         14           bicl&#xA; 3: B         24          calso&#xA; 4: A         51           mara&#xA; 5: B         51           mara&#xA; 6: A         125           gan&#xA; 7: A         143          carc&#xA; 8: B         143          carc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i.e.: each zone has 4 codes with its corresponding literal. I would like to transform it to a dataset with one column for each of the four codes and one column for each of the four literals:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; zone  code1 literal1   code2 literal2   code3 literal3   code4 literal4    &#xA; 1: A    14     bicl      51     mara     125      gan      143   carc&#xA; 2: B    14     bicl      24    calso      51     mara      143   carc &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any easy way to achieve this in R? If not, I would also be comfortable with a solution in Python.&lt;/p&gt;&#xA;" OwnerUserId="2489" LastEditorUserId="29169" LastEditDate="2021-02-01T20:11:36.947" LastActivityDate="2021-02-01T20:11:36.947" Title="rows to columns in data.table R (or Python)" Tags="|python|r|dataset|reshape|data-table|" AnswerCount="4" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26489" PostTypeId="2" ParentId="26351" CreationDate="2018-01-10T16:02:03.580" Score="1" Body="&lt;p&gt;The main reason to use statistics computed on only the training set is to avoid leaking information from the test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is not a concern, then it is perfectly OK to use statistics from the entire data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://stats.stackexchange.com/questions/77350/perform-feature-normalization-before-or-within-model-validation&quot;&gt;here&lt;/a&gt; for further discussion.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-01-10T16:02:03.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26490" PostTypeId="2" ParentId="26488" CreationDate="2018-01-10T16:02:53.870" Score="1" Body="&lt;p&gt;Using &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.T.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas&lt;/a&gt; in python you can transpose the rows and columns with .T&lt;/p&gt;&#xA;" OwnerUserId="18869" LastEditorUserId="28175" LastEditDate="2018-01-10T16:56:38.080" LastActivityDate="2018-01-10T16:56:38.080" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26491" PostTypeId="1" CreationDate="2018-01-10T16:54:51.180" Score="2" ViewCount="1775" Body="&lt;p&gt;I have implemented kmeans clustering on iris dataset (inbuilt dataset) in R. The code is given below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X=as.matrix(iris[-5]);&#xA;&#xA;K=3;&#xA;&#xA;prevCentroids=matrix(0,K,dim(X)[2]);&#xA;centroids=X[sample(1:dim(X)[1],K),];&#xA;&#xA;dot=numeric(3);&#xA;C=numeric(150);&#xA;&#xA;while(!isTRUE(all.equal(centroids,prevCentroids)))&#xA;{&#xA;  for(i in 1:dim(X)[1])&#xA;   {&#xA;    for(j in 1:dim(centroids)[1])&#xA;     {&#xA;       dot[j]=(X[i,]-centroids[j,])%*%(X[i,]-centroids[j,]);&#xA;     }&#xA;     C[i]=which.min(dot);&#xA;   }&#xA;&#xA;  prevCentroids=centroids;&#xA;&#xA;  for(k in 1:K)&#xA;   {&#xA;     centroids[k,]=colMeans(X[which(C==k),]);&#xA;   }&#xA;}&#xA;&#xA;print(cbind(iris,C));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sometimes, with this code, I get 85% clustering correct. But sometimes, it is just 37% as correct, if I compare it with the already clustered inbuilt iris dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone please tell me where I am going wrong?&lt;/p&gt;&#xA;" OwnerUserId="44436" LastEditorUserId="40853" LastEditDate="2018-01-11T07:25:25.127" LastActivityDate="2019-03-12T15:01:30.303" Title="Implementation of kmeans clustering using R" Tags="|k-means|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26492" PostTypeId="2" ParentId="26488" CreationDate="2018-01-10T17:43:21.583" Score="1" Body="&lt;p&gt;Here is a python solution, given a dataframe (&lt;code&gt;df&lt;/code&gt;) containing the data you have above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from itertools import chain&#xA;&amp;gt;&amp;gt;&amp;gt; data = []&#xA;&amp;gt;&amp;gt;&amp;gt; for zone in df.zone.unique():&#xA;...    codetuples = [(row[2], row[3]) for row in df[df['zone']==zone].itertuples()]&#xA;...    data.append([zone] + list(chain.from_iterable(codetuples)))&#xA;...&#xA;&amp;gt;&amp;gt;&amp;gt; df = pandas.DataFrame(data, columns=['zone', 'code1', 'literal1', 'code2', 'literal2', 'code3', 'literal3', 'code4', 'literal4'])&#xA;&amp;gt;&amp;gt;&amp;gt; df&#xA;  zone   code1 literal1   code2 literal2 code3 literal3  code4 literal4&#xA;0    A      14     bicl      51     mara   125      gan    143     carc&#xA;1    B      14     bicl      24    calso    51     mara    143     carc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Explanation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.itertuples()&lt;/code&gt; returns an iterator through the rows of a dataframe as tuples. The first entry (0 indexed in the tuple) will be the index, so the 2nd and 3rd columns of the df will be the two you are interested in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no guarantee of order for code1 vs code2; I stored the data from the df in a variable &lt;code&gt;codetuples&lt;/code&gt; so you can sort or something. There is also no guarantee that you will have exactly 4 pairs of code and literal, so you could put error checking in there, if you needed to. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have an acceptable list of four tuples, &lt;code&gt;from_iterable()&lt;/code&gt; &lt;a href=&quot;https://stackoverflow.com/a/953097/5061485&quot;&gt;flattens&lt;/a&gt; this list. Then append the zone number to the front and store it as another dataframe.&lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-01-10T17:43:21.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26493" PostTypeId="2" ParentId="26479" CreationDate="2018-01-10T17:50:06.117" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/seatgeek/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;FuzzyWuzzy&lt;/a&gt; would be a good python library for this purpose. A code example from the GitHub readme:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; choices = [&quot;Atlanta Falcons&quot;, &quot;New York Jets&quot;, &quot;New York Giants&quot;, &quot;Dallas Cowboys&quot;]&#xA;&amp;gt;&amp;gt;&amp;gt; process.extract(&quot;new york jets&quot;, choices, limit=2)&#xA;    [('New York Jets', 100), ('New York Giants', 78)]&#xA;&amp;gt;&amp;gt;&amp;gt; process.extractOne(&quot;cowboys&quot;, choices)&#xA;    (&quot;Dallas Cowboys&quot;, 90)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I imagine you would like to use the &lt;code&gt;extract&lt;/code&gt; function with limit 5 with &lt;code&gt;choices&lt;/code&gt; being set equal to all the 'official names' from your table2 csv file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Details on installation and use are at the link above.&lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-01-10T17:50:06.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26494" PostTypeId="2" ParentId="26468" CreationDate="2018-01-10T18:24:20.880" Score="1" Body="&lt;p&gt;Actually I'm currently not working on this area but I remember something from past that may help you. JPG files use quantization, it is really difficult for forgery detection but I suggest you reading the following paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://ieeexplore.ieee.org/document/7453773/?reload=true&quot; rel=&quot;nofollow noreferrer&quot;&gt;Performance analysis of forgery detection of JPEG image compression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;The proposed forensic algorithm to discriminate between original and forged regions in JPEG images, under the hypothesis that the tampered image presents a double JPEG compression, either aligned (A-DJPG) or nonaligned (NA-DJPG). Unlike previous approaches, the proposed algorithm does not need to manually select a suspect region in order to test the presence or the absence of double compression artifacts. Based on an improved and unified statistical model characterizing the artifacts that appear in the presence of both A-DJPG or NA-DJPG, the proposed algorithm automatically computes a likelihood map indicating the probability for each 8 × 8 discrete cosine transform block of being doubly compressed. The validity of the proposed approach has been assessed by evaluating the performance of a detector based on thresholding the likelihood map, considering different forensic scenarios. The effectiveness of the proposed method is also confirmed by tests carried on realistic tampered images. An interesting property of the proposed Bayesian approach is that it can be easily extended to work with traces left by other kinds of processing.&lt;/em&gt;  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-16T19:31:08.900" LastActivityDate="2018-01-16T19:31:08.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26495" PostTypeId="2" ParentId="9074" CreationDate="2018-01-10T18:26:10.620" Score="2" Body="&lt;p&gt;After having read a lot more papers and having talked to many people about machine learning topics, this is how I would define the words:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;class&lt;/strong&gt; as an abstract concept which exists. Each class has properties and can have a lot of different labels. For example, the class &lt;em&gt;cat&lt;/em&gt; has the properties &quot;feet&quot; (with the value 4), the property &quot;Genus&quot; with the value &quot;Felis&quot;. There are many way members of the class can look like. Also many labels: cat, Katze, Felis silvestris, , .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;label&lt;/strong&gt; is just a sticker you put on the concept. A name. We need a word to be able to talk about the concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use &lt;strong&gt;labeling&lt;/strong&gt; for the manual process of defining which parts of the dataset belong to which class. And I use &lt;strong&gt;classification&lt;/strong&gt; for the process of the automatic classifier deciding which part of the data belongs to which class. So typically, labeling is done by a human and proceeds classification which is done by the machine.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-01-10T18:32:19.417" LastActivityDate="2018-01-10T18:32:19.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26496" PostTypeId="1" AcceptedAnswerId="26512" CreationDate="2018-01-10T19:28:59.477" Score="7" ViewCount="1422" Body="&lt;p&gt;Reposting here because someone correctly pointed out it is better suited for here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a bunch of titles of eBay listings. I want to extract product information from each title, so I can categorize the listings by product since these titles are coming from wide-ranging searches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The are the following Titles:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;5 PCS 3 Digits 0.56&quot; NUMERIC DISPLAY 7 seg segment Common cathode&lt;/li&gt;&#xA;&lt;li&gt;5 PCS LD-3361AS 3 Digits 0.36&quot; NUMERIC DISPLAY 7 SEG COMMON CATHODE digit&lt;/li&gt;&#xA;&lt;li&gt;6 x MAN71A Common Anode Red Led 7-Segment Display (6 pcs)&lt;/li&gt;&#xA;&lt;li&gt;Experiment Board Breadboard Circuit Board ZYJ - 60 White NEW&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The format in which I want to extract the data is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;1st&lt;/strong&gt; listing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;There are 5 pieces&#xA;It has 3 digits&#xA;It is of size .56&quot;&#xA;It is 7 segments&#xA;It is numeric&#xA;It is cathode&#xA;It is a display&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;2nd&lt;/strong&gt; listing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;There are 5 pieces&#xA;It is of size .36&quot; &#xA;It is model LD-3361AS&#xA;It has 3 Digits&#xA;It is Numeric&#xA;It has 7 segments&#xA;It is cathode&#xA;It is a display&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;3rd&lt;/strong&gt; listing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;There are 6 pieces&#xA;It is brand MAN71A&#xA;it has 7 segments&#xA;It is Anode&#xA;It is a display&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;4th&lt;/strong&gt; listing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;It is New&#xA;It is a Breadboard&#xA;It is White&#xA;It is model ZYJ - 60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The reason I want to learn machine learning to do this is that it would make it very easy to just take those specifics and store it for later use and because it can adapt to any search and not just in a specific category. It would also be able to adapt to different items or formats in the title. I tried this with scikitlearn's kmeans, but it gave me a lot of overlap and just clustered them, rather than extract the details from it. What I want to do is have the program look at the title, identify specifics as what they refer to (i.e., size, color, brand, etc.) and extract the data. Ideally, this would be unsupervised learning, but as I write this, I start to realize that may be impossible to do with this. The part I'm stuck on is what to use to achieve this. Should I use NLTK? One of the classifiers from scikitlearn? Clustering?&lt;/p&gt;&#xA;" OwnerUserId="44396" LastEditorUserId="40853" LastEditDate="2018-01-11T03:17:13.140" LastActivityDate="2019-10-18T05:40:30.257" Title="How to use machine learning to extract product info from the titles of eBay listings" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26497" PostTypeId="1" CreationDate="2018-01-10T20:28:56.413" Score="2" ViewCount="180" Body="&lt;p&gt;I read about fine tuning and transfer learning for CNNs and was wondering if we can do fine tuning after using transfer learning on the same CNN? If so, will this increase the performance of the model or decrease it?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T16:47:15.730" LastActivityDate="2024-04-03T00:05:40.840" Title="Doing a fine tuning after a transfer learning" Tags="|machine-learning|deep-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26498" PostTypeId="1" CreationDate="2018-01-10T21:09:38.533" Score="1" ViewCount="47" Body="&lt;p&gt;I'm reading the following paper&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1003.0146.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1003.0146.pdf&lt;/a&gt; on building a fair online simulator for contextual bandits. In particular i'm interested in the proof on page 665 of Theorem 1 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yBlFT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yBlFT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't believe this has been proved rigourously so i wanted to try and prove it myself. Is the following correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any given tuple $(x_{1},\ldots,x_{k},a,r_{a})$ the probability that the policy evaluator adds this tuple to the history $h_{t-1}$ is the same as the probability that the for loop terminates with a match on $(x_{1},\ldots,x_{k},a,r_{a})$. Since the stream is infinite this success can come at any time from $r=1,\ldots,\infty$. The probability that we have a success at time $r=p$ is equal to the probability of having $p-1$ failures and a sucess on the $p$th trial and thus is equal to &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$((k-1)/k)^{p-1}P((x_{1},\ldots,x_{k},a,r_{a})\times 1/k$ the total probability of obtaining event $(x_{1},\ldots,x_{k},a,r_{a})$ in the simulated environment at time $h_{t}$  is then &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sum_{p=1}^{\infty} ((k-1)/k)^{p-1}P((x_{1},\ldots,x_{k},a,r_{a})\times 1/k=P((x_{1},\ldots,x_{k},a,r_{a})$ and thus the policy evaluator is unbiased&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this correct? If not could someone provide a more detailed proof? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="43865" LastEditorUserId="43865" LastEditDate="2018-01-11T14:34:52.517" LastActivityDate="2018-01-11T14:34:52.517" Title="An unbiased simulator for policy simulation in reinforcement learning" Tags="|machine-learning|statistics|reinforcement-learning|probability|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26499" PostTypeId="1" AcceptedAnswerId="26513" CreationDate="2018-01-10T21:09:48.580" Score="3" ViewCount="230" Body="&lt;p&gt;I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many times I find myself need to perform &quot;joins&quot; across databases, for example a student ID that appears in multiple databases with data that I want to gather. The joins are usually done after data is streamed out of the database, since the data is not located in the same database, and this sometimes requires hours just for thousands of records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can services such as AWS redshift or Google BigQuery allow you to somehow &quot;import&quot; data from many data sources and then you can perform SQL queries to join them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about Hadoop and Hive? Where we steam data out from the database and place it as files in Hadoop, and let Hive Query the data?&lt;/p&gt;&#xA;" OwnerUserId="24119" LastActivityDate="2018-01-11T09:41:08.443" Title="Big Data - Data Warehouse Solutions?" Tags="|bigdata|databases|redshift|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26500" PostTypeId="2" ParentId="26477" CreationDate="2018-01-10T21:12:12.083" Score="1" Body="&lt;p&gt;It can be done like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;melted = pd.melt(df1, id_vars=[&quot;index&quot;], var_name=&quot;Var&quot;, value_name=&quot;Score&quot;).dropna()&#xA;table=pd.crosstab(index=melted['index'], columns=melted['Score'])&#xA;%matplotlib inline&#xA;table.plot.bar() #for simple axes subplots&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44424" LastEditorUserId="44424" LastEditDate="2018-01-13T16:10:02.657" LastActivityDate="2018-01-13T16:10:02.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26501" PostTypeId="1" AcceptedAnswerId="26535" CreationDate="2018-01-10T21:13:56.903" Score="4" ViewCount="3710" Body="&lt;p&gt;I'm having a stumbling block with SageMaker. How do I know what my job name is?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mnist_estimator = MXNet(entry_point='/home/ec2-user/sample-notebooks/sagemaker-python-sdk/mxnet_mnist/mnist.py',&#xA;                        role=role,&#xA;                        output_path=model_artifacts_location,&#xA;                        code_location=custom_code_upload_location,&#xA;                        train_instance_count=1,&#xA;                        train_instance_type='ml.m4.xlarge',&#xA;                        base_job_name=’foo’,&#xA;                        hyperparameters={'learning_rate': 0.1})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then when I call fit() it prints out:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;INFO: Creating training job with name: foo-2018-01-10-20-13-57-893&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and when I look in my S3 bucket I see:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2018-01-10 15:20:45     411784 artifacts/foo-2018-01-10-20-13-57-893/output/model.tar.gz&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So my job name is &quot;foo-2018-01-10-20-13-57-893&quot; and not &quot;foo&quot;, but how can I get this from python? I guess I'm looking for a way of extracting that from the Estimator itself, but I'm just blanking on it. I'm reading the python source but that's not helping me, probably because I'm just learning python also.&lt;/p&gt;&#xA;" OwnerUserId="40235" LastActivityDate="2020-09-04T02:40:28.827" Title="How do I get the name of Sagemaker Estimator's job" Tags="|aws|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26502" PostTypeId="1" CreationDate="2018-01-10T21:34:24.393" Score="1" ViewCount="55" Body="&lt;p&gt;&lt;strong&gt;Little backstory&lt;/strong&gt; : I've recently purchased the book Hands-On Machine Learning with Scikit-Learn and Tensorflow and I've started trying to apply what I've read so far in the book. Before this, I tested my knowledge on the &lt;a href=&quot;https://www.kaggle.com/c/titanic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Titanic&lt;/a&gt; &quot;competition&quot;. I now want to refactor my code to use some good practices I learned while reading the book.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At some point, the author talks about &lt;code&gt;sklearn.pipeline.Pipeline&lt;/code&gt;. I started writing my own &lt;code&gt;Transformers&lt;/code&gt; to populate my pipeline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I face is that, in some transformers, I'd use some &lt;code&gt;pandas&lt;/code&gt; functions, in other &lt;code&gt;numpy&lt;/code&gt; functions and I keep converting my dataset back and forth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some research and din't find anything useful, it would be of great help if someone can answer the following questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are there good practices concerning &lt;code&gt;Pipeline&lt;/code&gt;?&lt;/li&gt;&#xA;&lt;li&gt;Should I stick to &lt;code&gt;pandas&lt;/code&gt;, &lt;code&gt;numpy&lt;/code&gt;, is it a good practice to convert back and forth?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44446" LastEditorUserId="40853" LastEditDate="2018-01-11T19:10:48.150" LastActivityDate="2018-01-11T19:10:48.150" Title="Working back and forth with DataFrame and nparray in Pipeline transformers" Tags="|scikit-learn|pandas|numpy|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26503" PostTypeId="1" CreationDate="2018-01-11T04:00:04.267" Score="3" ViewCount="270" Body="&lt;p&gt;Video platforms like YouTube, Netflix, Amazon prime have an excellent search system - given a search string, find most relevant videos. Which Machine Learning /Deep Learning techniques used for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers will be of great help&lt;/p&gt;&#xA;" OwnerUserId="13457" LastEditorUserId="13457" LastEditDate="2018-01-11T08:45:27.177" LastActivityDate="2020-02-07T06:49:55.157" Title="What ML/DL techniques power Youtube/Netflix search systems?" Tags="|machine-learning|deep-learning|search-engine|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26504" PostTypeId="2" ParentId="26479" CreationDate="2018-01-11T05:09:40.880" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would recommend against using an ML classifier because you're likely to exclude some of the 200k official names if only 5 options are given back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many fuzzy text matching algorithms to match your rows to an official name. &lt;a href=&quot;https://pypi.python.org/pypi/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;FuzzyWuzzy&lt;/a&gt;'s and several other algorithms are based on the &lt;a href=&quot;https://people.cs.pitt.edu/~kirk/cs1501/Pruhs/Spring2006/assignments/editdistance/Levenshtein%20Distance.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein&lt;/a&gt; distance. You can use a for-loop to go through the 200k official names. Depending on how much text there is this might take a while. Sort the list and slice/pop values that have to low scores, and you will have the top 5 best matching official names for a row. I guess you want to rank the frequency of each official name and return the highest ranked as a first result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because the health descriptions given by the students can be redundant, you might want run a &lt;a href=&quot;https://arxiv.org/ftp/arxiv/papers/1705/1705.07890.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Zipf ranking algorithm&lt;/a&gt; to filter more common words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is answer is far from complete. Comments are welcome.&lt;/p&gt;&#xA;" OwnerUserId="44393" LastEditorUserId="40853" LastEditDate="2018-01-11T05:53:09.177" LastActivityDate="2018-01-11T05:53:09.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26505" PostTypeId="2" ParentId="26366" CreationDate="2018-01-11T05:56:22.253" Score="12" Body="&lt;p&gt;@kbrose seems to have a better solution&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is usually a good solution. Maybe try max length of sequence + 100. Use whatever works best for your application.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But then does that mean I can't make predictions at test time with input length greater than that?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not necessarily. The reason a fixed length is used in keras, is because it greatly improves performance by creating tensors of fixed shapes. But that's only for training. After training, you'll have learned the right weights for your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume, after training for hours, you realise your model's max length wasn't big/small enough and you now need to change the time steps, just extract the learned weights from the old model, build a new model with the new time steps and inject the learned weights into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can probably do this using something like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;new_model.set_weights(old_model.get_weights())&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I haven't tried it out myself. Please try it and post your results here for everyone's benefit. Here are some links:&#xA;&lt;a href=&quot;https://github.com/keras-team/keras/issues/5348&quot; rel=&quot;noreferrer&quot;&gt;one&lt;/a&gt; &lt;a href=&quot;https://github.com/keras-team/keras/issues/91&quot; rel=&quot;noreferrer&quot;&gt;two&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33738" LastEditorUserId="33738" LastEditDate="2018-02-18T04:37:35.603" LastActivityDate="2018-02-18T04:37:35.603" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26506" PostTypeId="2" ParentId="26448" CreationDate="2018-01-11T06:09:33.430" Score="1" Body="&lt;p&gt;&quot;pathology report, doctor visit notes, prescription orders&quot; these are classes, not clusters. Clustering may as well find &quot;people with headache&quot;, &quot;patients on Thursdays&quot;, ... you do not get to control this unsupervised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'd rather suggest to do a &lt;strong&gt;classification&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, put all documents that contain &quot;patient report&quot; in the first lines into a separate set. Inspect the remainder for another keyword. Add another &lt;strong&gt;rule&lt;/strong&gt;. If the data is as explained, 10 such rules may get the leftover documents down to a manageable scale, and they could eventually just be treated as 'others'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is nothing wrong with inspecting documents and adding rules if that solves your problem. You probably don't need to make this work on different document collections (say, law texts) ever.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-11T06:09:33.430" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26507" PostTypeId="1" CreationDate="2018-01-11T06:24:28.847" Score="3" ViewCount="179" Body="&lt;p&gt;I am trying to build a very naive version of Api.ai, now Google DailogueFlow. &#xA;I wanted to know two things.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How DF classifies sentences with entities in it that can be user created and/or things like date, place, time, currency or even random text (which they call 'system.any')&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How data is extracted from slots after user map the slots with the sentence. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What algorithm/libraries and techniques do they use to classify, extract and identify user queries. &lt;/p&gt;&#xA;" OwnerUserId="44453" LastEditorUserId="29575" LastEditDate="2018-01-11T06:25:47.757" LastActivityDate="2018-01-11T06:25:47.757" Title="How does Api.ai Google dialogueflow classifies &quot;intent&quot; and extracts data from slots" Tags="|nlp|text-mining|named-entity-recognition|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26508" PostTypeId="2" ParentId="26503" CreationDate="2018-01-11T06:53:17.080" Score="2" Body="&lt;p&gt;At &lt;strong&gt;Netflix&lt;/strong&gt;, the machine ​learning algorithms used are more complex since the data is really vast and increases day to day. Take a look at these:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Netflix Recommendations: Beyond the 5 stars &amp;#40;Part 1&amp;#41;&quot;&gt;Netflix Recommendations: Beyond the 5 stars (Part 1)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.udacity.com/2016/06/machine-learning-everywhere-netflix-personalized-medicine-fraud-prevention.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Machine Learning Is Everywhere: Netflix, Personalized Medicine, and Fraud Prevention | Udacity&quot;&gt;Machine Learning Is Everywhere: Netflix, Personalized Medicine, and Fraud Prevention | Udacity&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32762" LastActivityDate="2018-01-11T06:53:17.080" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26509" PostTypeId="2" ParentId="26444" CreationDate="2018-01-11T08:28:19.777" Score="11" Body="&lt;p&gt;I found what I was looking for - it's called &lt;a href=&quot;https://en.wikipedia.org/wiki/Uncertainty_coefficient&quot; rel=&quot;noreferrer&quot;&gt;Theil's U&lt;/a&gt;, or the Uncertainty Coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used it in this &lt;a href=&quot;https://www.kaggle.com/shakedzy/alone-in-the-woods-using-theil-s-u-for-survival&quot; rel=&quot;noreferrer&quot;&gt;Kaggle kernel&lt;/a&gt;, you can check it out for an example and code implementation in Python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I also have a &lt;a href=&quot;https://medium.com/@shakedzy/the-search-for-categorical-correlation-a1cf7f1888c9&quot; rel=&quot;noreferrer&quot;&gt;blogpost&lt;/a&gt; about it.&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="43870" LastEditDate="2019-05-10T12:51:50.507" LastActivityDate="2019-05-10T12:51:50.507" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26510" PostTypeId="2" ParentId="26479" CreationDate="2018-01-11T08:52:00.997" Score="0" Body="&lt;p&gt;Thank you all I will definitely test the FuzzyWuzzy Lib. Meanwhile please take a look on the code I have made yesterday ( it can be enhanced , Go ahead if you can make it better : –&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#define function to read the CSV files&#xA;def GetDocument(link):&#xA;    dataframe = pd.read_csv(link, encoding = &quot;ISO-8859-1&quot;, sep = ';')&#xA;    return dataframe&#xA;#define function convert text/CSV data to lists &#xA;def ColumnToList(column):&#xA;    return column.tolist()&#xA;#read document csv &#xA;document1 = GetDocument('C:/Users/smegrhi/Desktop/DOC1.csv')&#xA;document2 = GetDocument('C:/Users/smegrhi/Desktop/DOC2.csv')&#xA;#get columns into lists&#xA;list1 = ColumnToList(document1['DIA_LBL'])&#xA;list2 = ColumnToList(document2['ICD10_LBL_FR'])&#xA;&#xA;#Data pre-processing&#xA;&#xA;def ProcessText(sentence):&#xA;&#xA;    #Convert to lower case&#xA;    sentence = sentence.lower()&#xA;    #Convert www.* or https?://* to URL&#xA;    sentence = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','',sentence)&#xA;#Convert &#xA;    sentence = re.sub('@[^\s]+','',sentence)&#xA;#Remove additional white spaces&#xA;    sentence = re.sub('[\s]+', ' ', sentence)&#xA;#Replace &#xA;    sentence = re.sub(r'#([^\s]+)', r'\1', sentence)&#xA;    return sentence&#xA;&#xA;&#xA;#execute&#xA;for i,text in enumerate(list1):&#xA;    res = ProcessText(text)&#xA;    list1[i] = res&#xA;&#xA;#vectorize the data&#xA;WORD = re.compile(r'\w+')&#xA;#function vectorize data&#xA;def text_to_vector(text):&#xA;    words = WORD.findall(text)&#xA;    return Counter(words)&#xA;&#xA;#calculate similarity&#xA;def get_cosine(vec1, vec2):&#xA;    intersection = set(vec1.keys()) &amp;amp; set(vec2.keys())&#xA;    numerator = sum([vec1[x] * vec2[x] for x in intersection])&#xA;&#xA;    sum1 = sum([vec1[x]**2 for x in vec1.keys()])&#xA;    sum2 = sum([vec2[x]**2 for x in vec2.keys()])&#xA;    denominator = math.sqrt(sum1) * math.sqrt(sum2)&#xA;&#xA;    if not denominator:&#xA;        return 0.0&#xA;    else:&#xA;        return float(numerator) / denominator&#xA;&#xA;# compare sentence from list 1 with list 2 : Choose Indication from Data/DOC1 &#xA;&#xA;text1 = list1[1]&#xA;#text1 = shigellose à shigella dysenteriae&#xA;vector1 = text_to_vector(text1)&#xA;listCosine = []&#xA;for i,text in enumerate(list2):&#xA;    vector2 = text_to_vector(text)&#xA;    cosine = get_cosine(vector1, vector2)&#xA;    listCosine.append(cosine)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The results are not that bad neither good ...&#xA;Your suggestions ?&lt;/p&gt;&#xA;" OwnerUserId="40426" LastActivityDate="2018-01-11T08:52:00.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26511" PostTypeId="1" AcceptedAnswerId="28189" CreationDate="2018-01-11T08:56:20.023" Score="7" ViewCount="1078" Body="&lt;p&gt;I know it depends on the data and question asked but imagine a scenario that for a given dataset you could either go for a fairly complex nonlinear model (hard to interpret though) giving you a better prediction power perhaps because the model may see the nonlinearities present in the data, or have a simple model (perhaps a linear model or something) with less prediction power but easier to interpret. Here is a very good &lt;a href=&quot;https://www.oreilly.com/ideas/ideas-on-interpreting-machine-learning&quot; rel=&quot;noreferrer&quot;&gt;post&lt;/a&gt; discussing ideas on how to interpret machine learning models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Industries, while being very cautious, are slowly becoming more interested in adopting more complex models! Still they want to know the trade-off clearly? A data scientist perhaps is the one sitting between data team and decision-makers, and often need to be able to explain these stuffs in layman's terms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to brainstorm here to see what analogy you would come up with to describe such trade-off to a non-technical person?&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-02-22T19:05:10.533" Title="How would you describe the trade-off between model interpretability and model prediction power in layman's terms?" Tags="|machine-learning|predictive-modeling|model-selection|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26512" PostTypeId="2" ParentId="26496" CreationDate="2018-01-11T09:30:57.563" Score="3" Body="&lt;p&gt;Like Toros91 explained, it is a raw data format. You want to convert &lt;code&gt;5 PCS&lt;/code&gt; into &lt;code&gt;There are 5 pieces&lt;/code&gt;. How does you know &lt;em&gt;PCS&lt;/em&gt; means &lt;em&gt;pieces&lt;/em&gt;? You have a link somewhere in your brain that tells this. You have to do the same in programming. Create a dictionary which tells the machine that wherever it encounters PCS in line, &lt;code&gt;PCS&lt;/code&gt; means &lt;code&gt;pieces&lt;/code&gt; (it can be a model name also, you have to take care of that). The number just before PCS is the value. All this you have to code manually. There is no shortcut here. And of course you have to use NLP to achieve your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pre-Processing your raw data is the hard task, using ML algorithm on a formatted data is easy one.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-01-11T09:30:57.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26513" PostTypeId="2" ParentId="26499" CreationDate="2018-01-11T09:41:08.443" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can services such as AWS redshift or Google BigQuery allow you to&#xA;  somehow &quot;import&quot; data from many data sources and then you can perform&#xA;  SQL queries to join them?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It depends on your data and the type of joins you are performing. But, yeah, databases like Redshift can definitely perform better in your use case as they are column-based databases. Read &lt;a href=&quot;https://datascience.stackexchange.com/q/8244/11097&quot;&gt;this post&lt;/a&gt; and the associated answers for understanding how columnar data stores handle data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How about Hadoop and Hive?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Hadoop + Hive&lt;/code&gt; is mostly a DIY hosted/cloud version of what Redshift gives you on cloud.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2018-01-11T09:41:08.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26514" PostTypeId="1" CreationDate="2018-01-11T09:59:11.163" Score="1" ViewCount="46" Body="&lt;p&gt;I've been participating in a project where I had to cut most of the dataset due to mismeasurements/invalid values and we ended up with a small dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The dataset is regarding  the following:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tool Deflection : &lt;strong&gt;Metrology - Sensor Measured&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Tool Wear : &lt;strong&gt;Metrology - Sensor Measured&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Piece - Form Accuracy : &lt;strong&gt;Metrology - Sensor Measured&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Piece - Roughness : &lt;strong&gt;Metrology - Sensor Measured&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Machine Input - Parameters : &lt;strong&gt;System Inputs&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I thought of generating more data for each metrology dataset, using the Machine Inputs as reference, however, I feel like it could potentially bias my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, I'm quite confused with respect to this subject. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What should I expect if I move towards this solution? &lt;/li&gt;&#xA;&lt;li&gt;Is there any restriction I should be aware of?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44461" LastEditorUserId="40853" LastEditDate="2018-01-11T11:56:37.700" LastActivityDate="2018-04-11T13:54:08.150" Title="What should be the main concerns when generating extra data?" Tags="|machine-learning|data-mining|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26515" PostTypeId="2" ParentId="26514" CreationDate="2018-01-11T10:24:20.780" Score="1" Body="&lt;p&gt;It's as you said there is a chance of bias in the dataset. To avoid this you'll have to conform to an algorithm to generate data. Right off the bat, the solution I would propose is parametric methods. Find the statistical distribution of data and depending on that distribution fill in data accordingly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know a bit more about parametric methods you can look at my answer here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/26009/what-does-it-mean-for-the-training-data-to-be-generated-by-a-probability-distrib/26045#26045&quot;&gt;What does it mean for the training data to be generated by a probability distribution over datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some machine learning methods look at this answer I posted here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/23607/what-are-the-best-way-to-handle-missing-values/23609#23609&quot;&gt;What are the best way to handle missing values&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is never one single correct way to do what you're asking, but there are good ways and it depends on the nuances of your data. &lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2018-01-11T10:24:20.780" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26516" PostTypeId="1" AcceptedAnswerId="26529" CreationDate="2018-01-11T10:34:50.610" Score="5" ViewCount="8391" Body="&lt;p&gt;I have created a dataset which has rather large number of features for example-100,000. Is it too large for a decent computer to handle ( I have a 1080ti )?&lt;/p&gt;&#xA;" OwnerUserId="44464" LastEditorUserId="28175" LastEditDate="2018-01-11T18:40:53.433" LastActivityDate="2018-01-15T12:26:52.293" Title="Is there a maximum limit to the number of features in a Neural Network?" Tags="|machine-learning|neural-network|deep-learning|cnn|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26517" PostTypeId="1" CreationDate="2018-01-11T11:00:26.417" Score="2" ViewCount="1396" Body="&lt;p&gt;Many scikit-learn and pandas objects/functions use &lt;code&gt;random_state=None&lt;/code&gt; as a default parameter. How can it be overridden to &lt;code&gt;random_state=100&lt;/code&gt; by default for all objects without manually editing the &lt;code&gt;random_state&lt;/code&gt; for each object?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;make_blobs(n_samples=100, n_features=2, centers=3, cluster_std=1.0, &#xA;           center_box=(-10.0, 10.0), shuffle=True, random_state=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27403" LastEditorUserId="29575" LastEditDate="2018-01-14T01:58:51.243" LastActivityDate="2018-01-14T01:58:51.243" Title="Sklearn - Override random_state=None by default" Tags="|machine-learning|python|scikit-learn|pandas|parameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26518" PostTypeId="2" ParentId="26517" CreationDate="2018-01-11T11:41:42.573" Score="1" Body="&lt;p&gt;For scikit-learn can set &lt;code&gt;np.random.seed(1)&lt;/code&gt;, for example, and as long as nothing in your script is modifying the seed nondeterministically then you should get reproducible results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is described in the scikit-learn FAQ under &lt;a href=&quot;http://scikit-learn.org/stable/faq.html#how-do-i-set-a-random-state-for-an-entire-execution&quot; rel=&quot;nofollow noreferrer&quot;&gt;How do I set a &lt;code&gt;random_state&lt;/code&gt; for an entire execution?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't believe it is possible to do the same thing for pandas. See &lt;a href=&quot;https://github.com/pandas-dev/pandas/issues/13143&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for discussion.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-01-11T11:41:42.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26519" PostTypeId="1" AcceptedAnswerId="26522" CreationDate="2018-01-11T12:21:11.653" Score="3" ViewCount="120" Body="&lt;p&gt;If I have 10,000 training samples then what should I do: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bootstrapping and train 10 classifiers on it and then aggregating&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;randomly divide the dataset into 10 parts and train 10 classifiers on them and then aggregating. Which will be better? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will the 2nd method reduce variance and will it be better than 1st method&lt;/p&gt;&#xA;" OwnerUserId="27916" LastEditorUserId="27916" LastEditDate="2018-01-12T12:05:40.713" LastActivityDate="2018-01-12T12:05:40.713" Title="Bootstrapping or Randomly Dividing Dataset to reduce variance?" Tags="|random-forest|decision-trees|cross-validation|variance|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26520" PostTypeId="1" CreationDate="2018-01-11T13:59:57.087" Score="1" ViewCount="194" Body="&lt;p&gt;I ask from the practitioner point-of-view, and I hope the answer does not come down to nit-picking, but I would like to settle the matter once and for all in the work I do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the components of an Information Extraction (IE) pipeline is a relation(ship) extractor which picks out associations between entity mentions.  &lt;a href=&quot;https://nlp.stanford.edu/software/relationExtractor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CoreNLP&lt;/a&gt; talks about this extractor strictly using the term relation(s) extractor.  Not that it is authoritative, but &lt;a href=&quot;https://en.wikipedia.org/wiki/Relationship_extraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt; talks about relationship extraction.  The former seems more prevalent based upon both code and papers I have come across.  So is there an authoritative answer as to which is more proper, or are there legitimate cases to use both when talking about what we are trying to do?&lt;/p&gt;&#xA;" OwnerUserId="66" LastActivityDate="2018-01-11T13:59:57.087" Title="In practice, is relation extraction or relationship extraction the correct term?" Tags="|nlp|stanford-nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26521" PostTypeId="1" AcceptedAnswerId="26530" CreationDate="2018-01-11T14:20:06.120" Score="2" ViewCount="332" Body="&lt;p&gt;Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Text #1 : Comedy 10%, Drama 50%, Fiction 20%, Romance 1%, Mythology 5%, Adventure 10%&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Text #2 : Comedy 40%, Drama 3%, Fiction 2%, Romance 30%, Mythology 5%, Adventure 10%&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We have given keywords possibly ngrams in each class through which we make a comparison&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Class 1 Comedy : k11, k12, ..., k1m&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Class 2 Drama : k21, k22, ..., k2n&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Class 3 Fiction : k31, k32, ..., k3o&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Class 4 Romance : k41, k42, ..., k4p&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Class 5 Mythology : k51, k52, ..., k5q&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Class 6 Adventure: k61, k62, ..., k6r&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What can be the best probabilistic model that we can use for this task&lt;/p&gt;&#xA;" OwnerUserId="38492" LastActivityDate="2018-01-11T20:10:31.840" Title="How to estimate probabilities of different classes for a Text" Tags="|nlp|text-mining|probability|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="26522" PostTypeId="2" ParentId="26519" CreationDate="2018-01-11T14:50:17.297" Score="1" Body="&lt;p&gt;I think the second method will yield less correlated models than the first method. It is particularly true with decision trees which tend to quickly overfitting in the bottom nodes. It will help reduce the variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;However&lt;/strong&gt;, by using the second approach, you will end with 10 smaller datasets and so you risk to introduce a variance error due to the too small number of observations. Discussing about decision trees again, it would mean that your trees algorithms will tend to overfitting upper in the tree. And so you will increase your variance error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, for most of datasets, it is still better to use the first approach than the second. I think that very low correlated estimators won't bring a better improvement that the first method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can also observe that differences in the two approaches depend also of the number of observations, the number of features, the kind of estimators you are using.. A benchmark would be really interesting !&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-01-11T14:50:17.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26524" PostTypeId="1" CreationDate="2018-01-11T15:05:14.227" Score="1" ViewCount="209" Body="&lt;p&gt;I have a dataset that is growing at about 30 million rows per day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data schema is follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;day_hour_timestamp timestamp,&#xA;campaign_id        varchar,&#xA;creative_id        int,&#xA;uuid               varchar,&#xA;device             varchar,&#xA;country            varchar,&#xA;os                 varchar,&#xA;browser            varchar&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each record in the dataset is information about an advertisement impression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I visited a particular advertisement at a website on 10/11/2017 13:30. I will be one datapoint in the dataset for 10/11/2017 hour 13:00 along with my device, country, os, browser, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how do detect anomalies within this dataset? Anomalies such as data not normal to the columns, for example if the I receive an OS that is NULL or browser is 'Apple'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this dataset is pretty big would checking each data point be expensive?&lt;/p&gt;&#xA;" OwnerUserId="17663" LastActivityDate="2018-01-12T04:56:10.367" Title="How to detect Anomalies in raw level data?" Tags="|machine-learning|predictive-modeling|anomaly-detection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26526" PostTypeId="2" ParentId="25957" CreationDate="2018-01-11T17:31:31.423" Score="1" Body="&lt;p&gt;Since you have only one binary target to predict (survived vs not survived) you can use only one output (that provide the probability of surviving). The problem with this approach is that you can not longer use &lt;em&gt;softmax&lt;/em&gt; activation, since it will always be 1. Remember that softmax activation has the form&#xA;$$g(x_k)=\frac{e^{x_k}}{\sum_{i=1}^n e^{x_i}}$$&#xA;Since you have only one element to run the activation it will be normalized always as 1.&#xA;It is used mostly for categorical prediction (i.e. among several possibilities only one is correct).&#xA;You are getting $0$/$NaN$ loss because your model always return 1. Instead of using softmax activation you might try sigmoid activation, so you can interpret the result of the network evaluation as the probability of survive.&lt;/p&gt;&#xA;" OwnerUserId="44481" LastActivityDate="2018-01-11T17:31:31.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26528" PostTypeId="1" CreationDate="2018-01-11T18:00:19.167" Score="4" ViewCount="8863" Body="&lt;p&gt;I'm looking to reproduce the doc2vec, i.e. &lt;a href=&quot;https://cs.stanford.edu/%7Equocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paragraph vector approach&lt;/a&gt; by Le &amp;amp; Mikolov.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Is anyone aware of a full script using Tensorflow?&lt;/strong&gt; In particular, I'm looking for a solution where the paragraph vectors of PV-DM and PV-DBOW are concatenated.&lt;/p&gt;&#xA;&lt;p&gt;So far, I found:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/wangz10/tensorflow-playground/blob/master/doc2vec.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub #1&lt;/a&gt;&lt;/strong&gt;: Only PD-DM, also I'm not sure how to run this script?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/PacktPublishing/TensorFlow-Machine-Learning-Cookbook/blob/master/Chapter%2007/doc2vec.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub #2&lt;/a&gt;&lt;/strong&gt;: Unclear what it is (PV-DM?)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/JonathanRaiman/PVDM&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub #3&lt;/a&gt;&lt;/strong&gt;: Looks like PV-DM?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;There seems to be no solution that relates to the original IMDB data, and covers both PV-DM and PV-DBOW. I am aware of a few &lt;code&gt;gensim&lt;/code&gt; scripts, but would like to know whether there's a TF solution out there.&lt;/p&gt;&#xA;" OwnerUserId="44483" LastEditorUserId="26019" LastEditDate="2020-08-20T17:17:35.237" LastActivityDate="2020-08-20T17:17:35.237" Title="Full Doc2Vec Implementation/Repdoduction in TensorFlow?" Tags="|tensorflow|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26529" PostTypeId="2" ParentId="26516" CreationDate="2018-01-11T18:40:28.787" Score="5" Body="&lt;p&gt;It highly depends on your data. If it's image, I guess it is somehow logical but if not I recommend you constructing covariance matrix and tracking whether features have correlation or not. If you see many features are correlated, it is better to discard correlated features. You also can employ &lt;code&gt;PCA&lt;/code&gt; to do this. Correlated features cause larger number of parameters for neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I have to say that maybe you can reduce the number of parameters if your inputs are images by resizing them. In popular nets the length and height of input images are usually less than three hundred which makes the number of input features &lt;code&gt;90000&lt;/code&gt;. Also you can employ max-pooling after some convolution layers, if you are using convolutional nets, to reduce the number of parameters. Refer &lt;a href=&quot;https://datascience.stackexchange.com/a/26413/28175&quot;&gt;here&lt;/a&gt; which maybe helpful.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-15T12:26:52.293" LastActivityDate="2018-01-15T12:26:52.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26530" PostTypeId="2" ParentId="26521" CreationDate="2018-01-11T18:47:14.870" Score="1" Body="&lt;p&gt;If I understand correctly, we are interested in soft &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-label_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multilabel classification&lt;/a&gt;, where a single text can have multiple correct genres.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to your comment, we don't have any training data, just a list of keywords associated with each genre.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can try computing the similarity between each document and each keyword list:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Text_normalization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Normalize&lt;/a&gt; the document (convert to lowercase, remove punctuation, diacritics, non-alphanums, etc)&lt;/li&gt;&#xA;&lt;li&gt;Remove &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;nofollow noreferrer&quot;&gt;stopwords&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Convert the document to &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf&lt;/a&gt; vector over our genre keyword vocabulary: Each document gets an n-length vector where each entry is the frequency of the ith genre keyword in the document. Normalize this vector to magnitude 1.&lt;/li&gt;&#xA;&lt;li&gt;Convert each genre keyword list to a tf-idf vector in the same way (again over the keyword vocabulary for all genres).&lt;/li&gt;&#xA;&lt;li&gt;Compute the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;cosine similarity&lt;/a&gt; between the document vector and each genre vector.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For each document, this will give us a number in the range [0,1] for each genre. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         Comedy Drama Fiction Romance Mythology Adventure&#xA;Text #1: 0.15   0.11  0.03    0.00    0.00      0.07&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we were doing single label classification we could normalize each row to add up to 1 and we might have a working model. However there is no such trick for multilabel classification here. We don't have a good way to calibrate these values into probability estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this point the only solution I see is to build a small training set so we can fit our model to actual data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After gathering some training examples, we can run a multilabel regression with &lt;a href=&quot;https://stats.stackexchange.com/a/234578/1104&quot;&gt;sigmoid activation and binary crossentropy loss&lt;/a&gt; with the cosine similarities as input features to get a probability estimate for each class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using this method our list of genre keywords will at least save us having to build a large training set to solve the problem directly with &lt;a href=&quot;https://en.wikipedia.org/wiki/Bag-of-words_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;bag-of-words&lt;/a&gt; or similar approaches.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-01-11T20:10:31.840" LastActivityDate="2018-01-11T20:10:31.840" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26531" PostTypeId="1" CreationDate="2018-01-11T19:13:36.383" Score="4" ViewCount="314" Body="&lt;p&gt;Suppose that in a binary classification task, I have separate classifiers &lt;code&gt;A&lt;/code&gt;, &lt;code&gt;B&lt;/code&gt;, and &lt;code&gt;C&lt;/code&gt;. If I use &lt;code&gt;A&lt;/code&gt; alone, I will get a high precision, but low recall. In other words, the number of true positives are very high, but it also incorrectly tags the rest of the labels as &lt;code&gt;False&lt;/code&gt;. &lt;code&gt;B&lt;/code&gt;, and &lt;code&gt;C&lt;/code&gt; have much lower precision, but when used separately, they may (or may not) result in better recall. How can I define an ensemble classifier that gives precedence to classifier &lt;code&gt;A&lt;/code&gt; only in cases where it labels the data as &lt;code&gt;True&lt;/code&gt; and give more weight to the predictions of other classifiers when &lt;code&gt;A&lt;/code&gt; predicts the label as &lt;code&gt;False&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is, &lt;code&gt;A&lt;/code&gt; is already outperforming others in catching true positives and I only want to improve the recall without hurting precision.   &lt;/p&gt;&#xA;" OwnerUserId="44488" LastActivityDate="2020-07-10T14:06:15.587" Title="How to create an ensemble that gives precedence to a specific classifier" Tags="|classification|prediction|ensemble-modeling|binary|ensemble|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26532" PostTypeId="2" ParentId="26531" CreationDate="2018-01-11T20:06:04.723" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/0911.0460&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature-Weighted Linear Stacking&lt;/a&gt; might be what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;FWLS combines model predictions linearly using coefficients that are&#xA;  themselves linear functions of meta-features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In your example you can use the meta-feature &quot;Does &lt;code&gt;A&lt;/code&gt; label the example as &lt;code&gt;True&lt;/code&gt;?&quot;&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-01-11T20:06:04.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26533" PostTypeId="1" AcceptedAnswerId="30524" CreationDate="2018-01-11T20:07:56.107" Score="4" ViewCount="102" Body="&lt;p&gt;I'm currently working on web advertising topics and I'd like to know if it does make sense to, let's say, compare the resulting weights of two logits?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take as an example the case where input variables are the same across models. A variable gets a weight of 0.22 as the result of the first logit, and a weight of 0.08 as the result of the second logit (all the other variables remaining the same, and their weights may change obviously)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;May I conclude that this variable is of highest importance for explaining hidden patterns within the first dataset, or is such a comparison forbidden? (In fact, when thinking about the problem I couldn't find any obvious mistake...)&lt;/p&gt;&#xA;" OwnerUserId="42831" LastEditorUserId="11097" LastEditDate="2018-01-14T06:30:38.413" LastActivityDate="2018-04-19T14:29:01.560" Title="Is there a proper way of comparing weight coefficients of two logits?" Tags="|python|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26534" PostTypeId="1" CreationDate="2018-01-11T20:28:35.197" Score="0" ViewCount="256" Body="&lt;p&gt;How would I set up a NN to classify a cryptographic key for asymmetric cryptography, or generally a VERY large number/bit string?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could split this bit string into bytes/16bit/32bit/64bit numbers to use as NN inputs, but that does not seem like a good idea, since I am not looking for relationships and patterns between the bytes or blocks. I am looking for structure in the whole key.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this exact case I want to classify an almost 20000-bit string into 2 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the correct approach?&lt;/p&gt;&#xA;" OwnerUserId="44496" LastActivityDate="2018-01-11T20:28:35.197" Title="How to use a Neural Network to classify a very large number/bit string?" Tags="|neural-network|classification|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26535" PostTypeId="2" ParentId="26501" CreationDate="2018-01-11T20:55:54.567" Score="6" Body="&lt;p&gt;The _current_job_name contains the name of the job. So in the example from the question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(mnist_estimator._current_job_name)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;would print&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foo-2018-01-10-20-13-57-893&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40235" LastActivityDate="2018-01-11T20:55:54.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26538" PostTypeId="1" CreationDate="2018-01-11T21:46:06.037" Score="1" ViewCount="220" Body="&lt;p&gt;When fitting neural nets and getting close to the bottom, I consistently get a very distinct pattern in the loss function (and the mse).  See below:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/qQx3j.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qQx3j.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The lower right plots just give the last 100 iterations, and the x axis is always the iteration.)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once it gets close to the bottom, the net starts overshooting and then correcting itself -- eventually working its way to the bottom (or a bottom) of the loss surface).  This is annoying -- I'd like it to go faster and not dither.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Features of the net:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;N = about 7000, about 2000 input variables, 10 layers of 100 nodes each&lt;/li&gt;&#xA;&lt;li&gt;The loss is L2-penalized squared error.  This one is only very lightly penalized (and thus will almost certainly end up overfitting given the architecture)&lt;/li&gt;&#xA;&lt;li&gt;This particular net is using full-batch gradient descent, no dropout, and full-connections.  &lt;/li&gt;&#xA;&lt;li&gt;But I see the same thing when using minibatch, and either dropout or convolutional architectures.  &lt;/li&gt;&#xA;&lt;li&gt;The learning rate gets multiplied by a constant every time the loss goes down, and divided by the same constant squared when the loss goes up.&lt;/li&gt;&#xA;&lt;li&gt;Activation is the leaky ReLU&lt;/li&gt;&#xA;&lt;li&gt;No batch normalization&lt;/li&gt;&#xA;&lt;li&gt;RMSprop for adaptive learning rate&lt;/li&gt;&#xA;&lt;li&gt;Fit with &lt;a href=&quot;https://github.com/cranedroesch/panelNNET&quot; rel=&quot;nofollow noreferrer&quot;&gt;this package&lt;/a&gt; (I'm the author)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to correct this behavior, and I'd appreciate suggestions.  It would seem like the adaptive stepsize algorithm would need to know when it is going to hit a wall in a given direction, before it hits that wall.  Is that possible?  Would something like adagrad improve over RMSprop?  Alternatively, I haven't yet implemented batch normalization -- not sure how that would help, but, would it for some reason?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: here's an update, a few hundred iterations later.  Same behavior.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/eO5YL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eO5YL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30335" LastEditorUserId="30335" LastEditDate="2018-01-11T21:55:55.777" LastActivityDate="2018-01-11T21:55:55.777" Title="loss = function(iteration) gets super wobbly once it gets near the bottom" Tags="|machine-learning|neural-network|deep-learning|optimization|gradient-descent|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26539" PostTypeId="1" AcceptedAnswerId="26551" CreationDate="2018-01-11T21:54:19.183" Score="1" ViewCount="4820" Body="&lt;p&gt;How does back-propagation handle multiple different activation functions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example in a neural network of 3 hidden layers, each with a separate activation function such as tanh, sigmoid and ReLU, the derivatives of each of these functions would be different, so do we just compute the output error of each of these layers with the derivative of the activation function of that layer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also will the gradient of the cost function used to compute the output error of the last layer change at all, or is the activation function of that layer used to compute the gradient of the cost function?&lt;/p&gt;&#xA;" OwnerUserId="44502" LastEditorUserId="28175" LastEditDate="2018-01-12T05:16:58.973" LastActivityDate="2018-01-12T08:34:30.953" Title="Backpropagation with multiple different activation functions" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26540" PostTypeId="1" CreationDate="2018-01-11T22:29:34.293" Score="1" ViewCount="23" Body="&lt;p&gt;What is the standard (or just good) open data set to train a model for question answering? Ideally, not only in English.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="43432" LastActivityDate="2018-01-11T22:29:34.293" Title="Data set for Q &amp; A system training" Tags="|machine-learning|deep-learning|nlp|dataset|" AnswerCount="0" CommentCount="1" ClosedDate="2018-01-12T05:44:45.503" ContentLicense="CC BY-SA 3.0" />
  <row Id="26541" PostTypeId="2" ParentId="26524" CreationDate="2018-01-11T22:40:06.110" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would go with unsupervised or semi-supervised learning. Use 'normal' data as training data and then predict what is normal and what is abnormal. Depending on your needs you can fine-tune the cut-off.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One-class SVM should work. Or a combination of one-class SVM and SV Data-Description for possibly &lt;a href=&quot;https://arxiv.org/pdf/1710.09207.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;better results&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="44393" LastActivityDate="2018-01-11T22:40:06.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26544" PostTypeId="1" CreationDate="2018-01-12T00:43:43.247" Score="0" ViewCount="320" Body="&lt;p&gt;I have large 3 dimensional dataset that I need to classify in a particular way. I'm hoping for something like a k-means clustering method, but instead of inputting the number of clusters I would like to input the maximum radius of a cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Said another way, given a sphere of a defined size, I would like to find the minimum number of non-empty spheres that will cover all the data and classify the points accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to output a 2D array of the same height and width, with labels that represent the sphere each point in the 3D array was classified in. No two points with the same label should have a euclidian distance greater than the maximum radius, no sphere should be empty, no spheres should overlap and the algorithm should use the minimum number of spheres possible to accomplish this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I have little knowledge of the field, I have been looking into clustering algorithms for a little while and have not found one that accomplishes this.&lt;/p&gt;&#xA;" OwnerUserId="44511" LastActivityDate="2018-01-12T20:46:29.740" Title="Distance Based Classification in Python" Tags="|python|classification|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26546" PostTypeId="1" CreationDate="2018-01-12T02:33:31.240" Score="2" ViewCount="868" Body="&lt;p&gt;I am working on an LSTM autoencoder in keras. The aim here is to obtain a latent space representation for the time sequences which I intend to use for clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My input sequences (each feature) have very low variance among them.&#xA;The input before normalization looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uCUC9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uCUC9.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one of the sequences, it has 4 features(the columns) and variable length (in this case 11 the number of rows).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other sequences range from 11 to 200 in length. The number of features obviously remain constant. After normalization over the entire feature space (normalizing on each feature individually) these subtle differences in input sequences become even smaller. And I think the autoencoder is assuming this is noise and is not learning it (or rather behaving like a denoising autoencoder). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on how I can scale the data better? Should I make any changes to how I am treating the problem statement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There is no problem with the code as I was able to generate very good latent representation on a toy dataset whose features were more evenly spaced out.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I have tried standardization (z score- subtracting by mean and dividing by standard deviation) but problem still persists.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="32420" LastEditorUserId="32420" LastEditDate="2018-01-12T03:16:27.870" LastActivityDate="2018-01-12T03:16:27.870" Title="How to scale data for LSTM autoencoder?" Tags="|machine-learning|time-series|lstm|autoencoder|feature-scaling|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26547" PostTypeId="2" ParentId="26524" CreationDate="2018-01-12T04:56:10.367" Score="1" Body="&lt;p&gt;How I go about anomaly detection is having a domain knowledge. In your case there are multiple domains as columns so you can use Ontologies as a way to detect anomalies. There are multiple resources for that like WordNet, ConceptNet and GloVe which provide extensive data. You can get what classes the query belongs to and check if the class you want is present.&lt;/p&gt;&#xA;" OwnerUserId="44514" LastActivityDate="2018-01-12T04:56:10.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26550" PostTypeId="1" CreationDate="2018-01-12T07:05:55.410" Score="3" ViewCount="157" Body="&lt;p&gt;I am a starter in ML and I need some help...&lt;/p&gt;&#xA;&lt;h2&gt;The problem&lt;/h2&gt;&#xA;&lt;p&gt;Assume that I have a classifier which can classify left hand / right hand well.&#xA;I am curious whether it can decide whether there's a hand in the image?&lt;/p&gt;&#xA;&lt;p&gt;Because if it is not the case, I have to collect additional data that are labeled as &amp;quot;no-hand&amp;quot;. But there's so many things that are &amp;quot;no-hand&amp;quot; such like dogs, cats, flies... I don't think it is possible to collect all those possible &amp;quot;no-hand&amp;quot; data..&lt;/p&gt;&#xA;&lt;h2&gt;My intuition&lt;/h2&gt;&#xA;&lt;p&gt;Would it be reasonable that we use some activation functions like [sigmoid] as the final outputs for the classifier(neural network) and we consider the outputs for both left/right classes as the scores that classifier think how far the input is from the left/right hands classes.&lt;/p&gt;&#xA;&lt;p&gt;Then if the outputs are below some threshold for both left hand and right hand we can say there's no hand in this image input.&lt;/p&gt;&#xA;" OwnerUserId="44517" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-12T07:35:10.763" Title="Is classifier able to say there's no-such-case?" Tags="|machine-learning|neural-network|classification|image-classification|activation-function|" AnswerCount="2" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26551" PostTypeId="2" ParentId="26539" CreationDate="2018-01-12T07:26:40.357" Score="3" Body="&lt;p&gt;In short, all the activation functions in the backpropagation algorithm are evaluated independently through the chain rule, thus, you can mix and match to your hearts content.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h1&gt;What are we optimizing in backpropagation?&lt;/h1&gt;&#xA;&lt;p&gt;Backpropagation allows you to update your weights as a gradient function of the resulting loss. This will tend towards the optimal loss (the highest accuracy). After each forward pass of your training stage, you get an output at the last layer. You then calculate the resulting loss &lt;span class=&quot;math-container&quot;&gt;$E$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The consequence of each of your weights on your final loss is computed using its partial derivative. In other words, this is how much loss is attributed to each weight. How much error can be attributed to that value. The larger this value is the more the weight will change to correct itself (training).&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w^k_{i, j}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;How can we compute such a random partial derivative? Using the chain rule of derivatives, and putting together everything that led to our output during the forward pass. Let's look at what led to our output before getting into the backpropagation.&lt;/p&gt;&#xA;&lt;h1&gt;The forward pass&lt;/h1&gt;&#xA;&lt;p&gt;In the final layer of a 3-layer neural network (&lt;span class=&quot;math-container&quot;&gt;$k = 3$&lt;/span&gt;), the output (&lt;span class=&quot;math-container&quot;&gt;$o$&lt;/span&gt;), is a function (&lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt;) of the outputs of the previous layer (&lt;span class=&quot;math-container&quot;&gt;$o^2$&lt;/span&gt;) and the weights connecting the two layers (&lt;span class=&quot;math-container&quot;&gt;$w^2$&lt;/span&gt;).&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$y_0 = o^3_1 = \phi(a^3_1) = \phi(\sum_{l=1}^n w^2_{l,1}o^2_l)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The function &lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt; is the activation function for the current layer. Typically chosen to be something with an easy to calculate derivative.&lt;/p&gt;&#xA;&lt;p&gt;You can then see that the previous layers' outputs are calculated in the same way.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$o^2_1 = \phi(a^2_1) = \phi(\sum_{l=1}^n w^1_{l,1}o^1_l)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;So the outputs of the third layer can also be written as a function of the outputs of layer 1 by substituting the outputs of layer 2. This point becomes important for how the backpropagation propagates the error along the network.&lt;/p&gt;&#xA;&lt;h1&gt;Backpropagation&lt;/h1&gt;&#xA;&lt;p&gt;The partial derivative of the error in terms of the weights is broken down using the chain rule into&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w^k_{i, j}}$&lt;/span&gt; = &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial o^k_{j}} \frac{\partial o^k_{j}}{\partial a^k_{j}} \frac{\partial a^k_{j}}{\partial w^k_{i,j}}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Let us look at each of these terms separately.&lt;/p&gt;&#xA;&lt;h1&gt;1. &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial o^k_{j}}$&lt;/span&gt;&lt;/h1&gt;&#xA;&lt;p&gt;is the error caused by the output of the previous layer. For the last layer, using R2 loss, the error of the first output node is&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial o^3_{1}} = \frac{\partial E}{\partial y_{1}} = \frac{\partial }{\partial y_{1}} 1/2(\hat{y}_1-y_1)^2 = y_1 - \hat{y}_1$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;In words, this is how far our result, &lt;span class=&quot;math-container&quot;&gt;$y_1$&lt;/span&gt;, from the actual target &lt;span class=&quot;math-container&quot;&gt;$\hat{y}_1$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This is the same for all previous layers, where we need to substitute in the errors propagating through the network, this is written as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial o^k_{j}} = \sum_{l \in L} (\frac{\partial E}{\partial o^{k+1}_{l}} \frac{\partial o^{k+1}_{l}}{\partial a^{k+1}_{l}} w^{k}_{j,l}) $&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where L is the set of all neurons in the next layer &lt;span class=&quot;math-container&quot;&gt;$k+1$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h1&gt;2. &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial o^k_{j}}{\partial a^k_{j}}$&lt;/span&gt;&lt;/h1&gt;&#xA;&lt;p&gt;This is where the current layer's activation function will make a difference. Because we are taking the derivative of the output as a function of its input. And the output is related to the input through the activation function, &lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;So just take the derivative of the activation function. For logistic function this is easy and its&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}} = \phi(a_j)(1-\phi(a_j))$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h1&gt;3. &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial a^k_{j}}{\partial w^k_{i,j}}$&lt;/span&gt;&lt;/h1&gt;&#xA;&lt;p&gt;a is simply a linear combination of w and the subsequent layers outputs. Thus,&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial a^k_{j}}{\partial w^k_{i,j}} = o_i$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h1&gt;Finally&lt;/h1&gt;&#xA;&lt;p&gt;You can see that the activation functions of your layers are evaluated separately in the backpropagation algorithm. They will just be added onto your ever growing back-chain as independent terms within your chain rule.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-12T08:34:30.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26552" PostTypeId="2" ParentId="26550" CreationDate="2018-01-12T07:32:36.237" Score="4" Body="&lt;p&gt;Taking your questions one after the other:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Assume that I have a classifier which can classify left hand / right&#xA;  hand well. I am curious whether it can decide whether there's a hand&#xA;  in the image?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;. It cannot. A model which is trained on hand/no-hand can do that. Not this. At most, it shall predict one of the either (left/hand) with a very low confidence score. But, it definitely is not a nice idea to use it for a hand/no-hand identification.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But there's so many things that are &quot;no-hand&quot; such like dogs, cats,&#xA;  flies... I don't think it is possible to collect all those possible&#xA;  &quot;no-hand&quot; data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are trying to do build a binary hand.no-hand classification model, not a hand/cat/dog/... style multi-label classification model. So, you just need to have a bunch of images with a hand in it, and some more which doesn't have one.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Would it be reasonable that we use some activation functions like&#xA;  [sigmoid] as the final outputs for the classifier(neural network) and&#xA;  we consider the outputs for both left/right classes as the scores that&#xA;  classifier think how far the input is from the left/right hands&#xA;  classes.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Possible, but not advisable. &lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2018-01-12T07:32:36.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26553" PostTypeId="2" ParentId="26550" CreationDate="2018-01-12T07:35:10.763" Score="3" Body="&lt;p&gt;Many classifiers do not directly give you L or R. They will give you the option which has a higher decision metric. For Naive Bayes this would be the class with the higher probability between $p(L|x)$ and $p(R|x)$. This is true for most classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is set a hyper-parameter (if you have some anomalous instances) to determine how well contained within a class a certain novel example must be in order to be classified as pertaining to that class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if $p(L|x)=0.01$ and $p(R|x)=0.05$ then we can reject both hypotheses and say the novel example $x$ belongs to neither class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should look further into anomaly detection algorithms, you can also use p-value through the GLRT to test if your prediction falls enough within the class such that it can be classified as such. For example, set you p-value, $p = 0.05$, if the p-value of a given example falls below this threshold then do not attribute a class to it and flag it as anomalous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's A LOT of different ways this can be done, including using some clustering or NN techniques. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In short, you can wrap your classifier with an anomaly detection algorithm to catch hypotheses that are below an acceptable bound.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-12T07:35:10.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26555" PostTypeId="1" AcceptedAnswerId="27182" CreationDate="2018-01-12T08:21:16.300" Score="2" ViewCount="65376" Body="&lt;p&gt;I am running a multiple linear regression using backward elimination. Below is the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import statsmodels.formula.api as sm&#xA;X = np.append(arr = np.ones((50, 1)).astype(int), values = X, axis = &#xA;1)&#xA;X_opt = X[:,[0,1,2,3,4,5]]&#xA;regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()&#xA;regressor_OLS.summary()&#xA;X_opt = X[:,[0,1,3,4,5]]&#xA;regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()&#xA;regressor_OLS.summary()&#xA;X_opt = X[:,[0,3,4,5]]&#xA;regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()&#xA;regressor_OLS.summary()&#xA;X_opt = X[:,[0,3,5]]&#xA;regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()&#xA;regressor_OLS.summary()&#xA;X_opt = X[:,[0,3]]&#xA;regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()&#xA;regressor_OLS.summary()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I am predicting using the above regressor_OLS model,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_new = X[:, 3]&#xA;y_pred2 = regressor_OLS.predict(X_new)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the below error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    y_pred2 = regressor_OLS.predict(X_new)&#xA;    Traceback (most recent call last):&#xA;&#xA;  File &quot;&amp;lt;ipython-input-18-263dee38fc26&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    y_pred2 = regressor_OLS.predict(X_new)&#xA;&#xA;  File &quot;/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py&quot;, line 749, in predict&#xA;    return self.model.predict(self.params, exog, *args, **kwargs)&#xA;&#xA;  File &quot;/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py&quot;, line 359, in predict&#xA;    return np.dot(exog, params)&#xA;&#xA;ValueError: shapes (1,50) and (2,) not aligned: 50 (dim 1) != 2 (dim 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried &lt;code&gt;X_new = X_test[:,3]&lt;/code&gt; but still same error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;&#xA;  File &quot;&amp;lt;ipython-input-19-5020d55a4448&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    y_pred2 = regressor_OLS.predict(X_ne1)&#xA;&#xA;  File &quot;/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py&quot;, line 749, in predict&#xA;    return self.model.predict(self.params, exog, *args, **kwargs)&#xA;&#xA;  File &quot;/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py&quot;, line 359, in predict&#xA;    return np.dot(exog, params)&#xA;&#xA;ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44519" LastActivityDate="2023-02-04T10:34:57.987" Title="ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0)" Tags="|machine-learning|python|regression|linear-regression|" AnswerCount="5" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26556" PostTypeId="1" CreationDate="2018-01-12T08:33:39.070" Score="2" ViewCount="598" Body="&lt;p&gt;In the literature, I've come across statements like &lt;strong&gt;People with higher income and with long working hours are more likely to be diagnosed with chronic diseases such as stroke&lt;/strong&gt;. The &lt;a href=&quot;https://www.google.co.in/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=2&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwi6ncTBwdDYAhVGqI8KHSkbA6UQFgg2MAE&amp;amp;url=http%3A%2F%2Fwww.mdpi.com%2F2227-9032%2F5%2F1%2F8%2Fpdf&amp;amp;usg=AOvVaw2lQB57o12GGXn_-ruG1W6Z&quot; rel=&quot;nofollow noreferrer&quot;&gt;above-mentioned study&lt;/a&gt; (Page:8), explores the association between Behavioral Habits and Chronic Diseases using ANN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm new to ML, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I am unable to figure out how to make such conclusions with feature&#xA;study in neural networks or other machine learning techniques.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a way to quantify the likelihood in ANN similar to logistic&#xA;regression wherein regression coefficients give the change in the&#xA;log odds of the outcome for a one unit increase in the predictor&#xA;variable?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Currently using Azure ML studio&lt;/p&gt;&#xA;" OwnerUserId="44484" LastEditorUserId="40853" LastEditDate="2018-01-12T08:40:52.583" LastActivityDate="2018-03-14T20:54:24.040" Title="How does an individual feature affect predictions in neural network classification problem?" Tags="|machine-learning|neural-network|feature-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26557" PostTypeId="2" ParentId="24658" CreationDate="2018-01-12T11:04:07.490" Score="1" Body="&lt;p&gt;You said you have a bunch of keywords for each category extracted from the data. It is better to explain how did you get them as your data does not contain labels i.e. how do you know the word $w_i$ is a representative of category business? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might have assigned them based on semantics which is not bad but does not give you a level of confidence that they are really the representatives of category. The whole point about clustering is to find this representative words via the algorithm. An example is the famous concept of Stop Words. Everyone thinks that words such as $The$, $is$, $an$ and the rest of common lists are stop words but empirically you always get better classification result including them! That's because in NLP the only reference to indicate a stop word is the data itself. So be careful about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming your words are really representatives (however I would rely on LDA results), I would design a score for similarity of each question to each category representation list. It could be the category which has the maximum number of representatives in the sentence. It most probably falls in overfitting problem however you will not see that (your problem is unsupervised) but if you insist using them it could be a starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the direction of this idea, another one could be to extract the very extreme words for each category (e.g. &quot;Manchester united&quot; and &quot;Neymar&quot; are more probable to represent sport) then do the same clustering with only extreme words and based on found categories find more representatives and feed them to the algorithm again (iterative manner) and continue till some stopping criterion is satisfied. This looks like above but the difference is that the least input (first set of words) are fed manually and the rest will be found through algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also may have a look at embedding algorithm such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; or &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt; and use them with some similarity measure (cosine similarity for example).&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-12T11:04:07.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26559" PostTypeId="1" AcceptedAnswerId="26567" CreationDate="2018-01-12T11:55:52.810" Score="2" ViewCount="707" Body="&lt;p&gt;I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into reinforcement learning and specifically q learning with neural networks, to try this out and see if i could implement this, i used a q learning neural network for tic tac toe. However I am unsure as to why my program does not learn correctly. One idea I have is that i have implemented the input neurons incorrectly.I am using visual basic console application so just let me know if you need me to post any code&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My view on how to implement this:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?al networks, so they can play against each other and teach themselves  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This next potential problem i have is how I use my threshold and exploration values to choose whether to pick a random action or one the neural network chooses. I increase my threshold linearly from 0 to 1 throughout the iterations. while exploration is a random value between 0 and 1. is this correct? or is there a better way of doing it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback would be greatly appreciated, and if anyone has any problems with my question such as it being unclear, not making sense or anything else please let me know so i can fix it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you for all who take the time out to try and help.&lt;/p&gt;&#xA;" OwnerUserId="44530" LastEditorUserId="44530" LastEditDate="2018-01-12T14:20:20.577" LastActivityDate="2023-02-17T08:13:36.757" Title="Q Learning Neural network for tic tac toe Input implementation problem" Tags="|machine-learning|neural-network|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26562" PostTypeId="2" ParentId="19748" CreationDate="2018-01-12T13:43:50.190" Score="1" Body="&lt;p&gt;Wow, what a great question! This is a critical task for anyone who works with data in a marketing function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First&lt;/strong&gt;, I'll address your question about pairing the prediction to email. Your email is serving as a unique identifier. When you split your data set, you'll have to keep the email and suppress it from the prediction, and use it to match/compare to the validation data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If these are new predictions, you could create a new column in your data that contains the &quot;predicted&quot; value. Specifics on this process would be better suited for another community, probably StackOverflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second&lt;/strong&gt;, I'd like to talk about what you're trying to achieve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think there are two questions ways you could approach this with the work you've already done:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Identify who is &lt;em&gt;most likely to subscribe&lt;/em&gt;, and focus your efforts on creating nurturing campaigns to &quot;warm&quot; these prospects. The final goal is to get a subscription.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you're going to be using those &quot;events&quot; that you mentioned. What exactly do you have? in the past. I've just dumped web activity data and started working with that. You'll have to engineer a lot of features to capture information that your web activity won't report on (these could be # of logins, # of checkouts, time since last visit, this is where you need to get creative).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take this data and try to make some predicte who is likely to subscribe. In some sales industries, this is refereed to as a lead score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you're using R, you could try using &lt;code&gt;XGBoost&lt;/code&gt; -&lt;a href=&quot;https://cran.r-project.org/web/packages/xgboost/vignettes/xgboostPresentation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/xgboost/vignettes/xgboostPresentation.html&lt;/a&gt;. Since you're trying to predict a binary variable, you could use their decision tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is going to take a while, so don't get frustrated if you don't this to production in the first week (or month). Here's some reading:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datastories.com/gallery/predictive-user-scoring-example&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://datastories.com/gallery/predictive-user-scoring-example&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rdatascientist.wordpress.com/2015/01/26/predictive-lead-scoring-using-r-first-of-a-two-part-blog-series/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://rdatascientist.wordpress.com/2015/01/26/predictive-lead-scoring-using-r-first-of-a-two-part-blog-series/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Identify which &lt;em&gt;types&lt;/em&gt; of customers you want to acquire, and focus on building campaigns to engage these types of customers. The final goal here is to acquire quality customers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is more marketing and strategizing than it is machine learning. Don't get me wrong, there &lt;em&gt;is&lt;/em&gt; maching learning in identifying the clusters. Here's a great example I used: &lt;a href=&quot;http://www.kimberlycoffey.com/blog/2016/8/k-means-clustering-for-customer-segmentation&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.kimberlycoffey.com/blog/2016/8/k-means-clustering-for-customer-segmentation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you identify what the different segments and what behavior they exhibit, you have to team up with the other stakeholders and try to decide what kind of customers do you want? Whats the benefit is getting a customer who makes one large purchase a year vs. a customer who makes five smaller purchases a year? You'll have to take your time identifying customers/prospects who are the most valuable, and plan a strategy around that. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44537" LastActivityDate="2018-01-12T13:43:50.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26564" PostTypeId="1" CreationDate="2018-01-12T13:54:42.957" Score="1" ViewCount="63" Body="&lt;p&gt;Words like &lt;code&gt;myotonic&lt;/code&gt; are biomedical, but words like &lt;code&gt;new&lt;/code&gt; appear in regular English texts. I can imagine a few ways to classify words into biomedical and non-biomedical. Has this problem been tackled before? Does anybody know any ready-to-use solutions for this problem?&lt;/p&gt;&#xA;" OwnerUserId="44539" LastEditorUserId="44539" LastEditDate="2018-01-12T14:15:45.780" LastActivityDate="2018-03-13T18:10:34.113" Title="Classify phrases as biomedical or non-biomedical" Tags="|classification|nlp|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26565" PostTypeId="1" AcceptedAnswerId="26576" CreationDate="2018-01-12T14:37:32.890" Score="0" ViewCount="35525" Body="&lt;p&gt;I'm attempting to compute the class_weights for an highly imbalanced set of 9 classes based on the examples discussed in &lt;a href=&quot;https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras&quot;&gt;How to set class weights for imbalanced classes in Keras?&lt;/a&gt;. Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import math&#xA;&#xA;# labels_dict : {ind_label: count_label}&#xA;# mu : parameter to tune &#xA;&#xA;def create_class_weight(labels_dict,mu=0.15):&#xA;    total = np.sum(labels_dict.values())&#xA;    keys = labels_dict.keys()&#xA;    class_weight = dict()&#xA;&#xA;    for key in keys:&#xA;        score = math.log(mu*total/float(labels_dict[key]))&#xA;        class_weight[key] = score if score &amp;gt; 1.0 else 1.0&#xA;&#xA;    return class_weight&#xA;&#xA;# random labels_dict&#xA;labels_dict = {0: 3400, 1: 1700, 2: 4700, 3: 6800, 4: 3400, 5: 2300, 6: 8300, 7: 1000, 8:9600}&#xA;&#xA;create_class_weight(labels_dict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm getting an error log like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; File &quot;&amp;lt;ipython-input-34-7a9feda1053b&amp;gt;&quot;, line 20, in &amp;lt;module&amp;gt;&#xA;    create_class_weight(labels_dict)&#xA;&#xA;  File &quot;&amp;lt;ipython-input-34-7a9feda1053b&amp;gt;&quot;, line 11, in create_class_weight&#xA;    score = math.log(mu*total/float(labels_dict[key]))&#xA;&#xA;TypeError: unsupported operand type(s) for *: 'float' and 'dict_values'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm running the code with Python 3.6.3. What modifications am I supposed to make?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="15412" LastEditDate="2018-01-12T16:49:01.110" LastActivityDate="2020-03-18T11:04:33.033" Title="TypeError: unsupported operand type(s) for *: 'float' and 'dict_values'" Tags="|python|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26567" PostTypeId="2" ParentId="26559" CreationDate="2018-01-12T14:55:27.613" Score="1" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;One idea I have is that i have implemented the input neurons incorrectly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;There is no single correct way to implement the solution to this problem. There are less efficient and more efficient ways for specific problems.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;This should work. The first set of 9 neurons will receive a representation of the current game state. The second set of 9 neurons will receive a representation of the intended action.&lt;/p&gt;&#xA;&lt;p&gt;For predicting Q values from a state, action pair, this is a reasonable approach. Staying with the same architecture of 18 inputs, you could also have the second set of neurons be one-hot encoded to where the agent will to place the 'X' or 'O'.&lt;/p&gt;&#xA;&lt;p&gt;In the game Tic Tac Toe however, you can look for more compact representations if you like. You have already noticed that because the game is simple and deterministic, that you can represent the agent's action as simply &amp;quot;desired next state&amp;quot;. And in fact, the current state does not actually matter to a player, other than to enforce the rules of what are valid next states.&lt;/p&gt;&#xA;&lt;p&gt;You won't be implementing the rules of the game into the agent - they are part of the environment. Therefore, you can do away with the initial state altogether in your estimate, and work with the end state of each move - this is called the &lt;em&gt;afterstate&lt;/em&gt; in the RL literature, and you will find it used a fair bit in deterministic games, or even in non-deterministic games where the randomness happens before the action choice (e.g. in Backgammon). An afterstate representation is more efficient because it encodes the fact that you don't care what route a player took to get to certain board position, you just care about the value of that position as the game continues.&lt;/p&gt;&#xA;&lt;p&gt;Having said all that, if your goal is to learn basic RL, then you don't need to be looking for the most efficient solution, just one that works. Don't expect your NN-based learner with state-action logic to be the most efficient learner however.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Not as inputs to the network at the same time, no. In your (state, action [=next_state]) representation, your action representation should be the board state for the current player's move and &lt;em&gt;before&lt;/em&gt; the other player takes any action. The &lt;em&gt;resulting next state&lt;/em&gt; however, will be after the other player takes their action.&lt;/p&gt;&#xA;&lt;p&gt;If you want to train two separate bots against each other, then each would see the current state, then it would choose an action, then it would either get the reward for winning, or the opponent would take a turn. If the opponent won, then the first bot should receive the (negative) reward. If the opponent's move was not final, then the first bot should see the state &lt;em&gt;after&lt;/em&gt; the opponent's move and get to choose its next action.&lt;/p&gt;&#xA;&lt;p&gt;Again, this is inefficient. For a win/draw/lose game like Tic Tac Toe, you don't need two separate agents, each with their own learning algorithm, in order to train through self-play. You can instead alter Q-Learning slightly to work with the minimax algorithm. In brief this means alternating between the agent selecting actions that maximise the expected reward (for player 1) or minimise it (for player 2).&lt;/p&gt;&#xA;&lt;p&gt;However, like before, your 2 networks set up should be able to work, and is quite interesting dynamic - you could try different learning parameters, different NNs etc, and see which learns to win quickly (but don't forget starting player has an advantage for early random play, so you'd want to switch which learning algorithm was used for which player to get a fair assessment).&lt;/p&gt;&#xA;&lt;p&gt;The difference again is in terms of efficiency - a single network inside a modified RL with minimax will typically learn faster that two separate networks.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2023-02-17T08:13:36.757" LastActivityDate="2023-02-17T08:13:36.757" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="26568" PostTypeId="1" CreationDate="2018-01-12T15:18:33.850" Score="3" ViewCount="147" Body="&lt;p&gt;From left to right:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Maximum Q value for action selection (averaged) &lt;/li&gt;&#xA;&lt;li&gt;Train error (averaged) &lt;/li&gt;&#xA;&lt;li&gt;Reward from environment (averaged)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q23Sd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q23Sd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run double Q-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A behavioral policy is ε-greedy, ε constant decaying to 0.1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used learning rate decay as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;learning.rate = start_learn_rate / log(counter + 1), #decaying learning rate&#xA;&#xA;optimizer = 'sgd'&#xA;start_learn_rate = 0.001&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I observe degradation of the average reward after it hits its maximum, with a clear oscillation behaviour from high to low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this chart typical to TD (Q) learning routine?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Could you advise on what parameters could be changed to stabilize the result?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why the models behave like that? Don't they ought to expose a stable behaviour when Q is stabilized?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Environment, agent actions, and reward structure&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;My task is quite specific. I model sequences of actions applied to a process generating time series for the purpose of making something like predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ennMP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ennMP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Observe a sine function (timeseries). Agent gets information about lagged values (and some engineered features) of that process from it's PAST. Another part of environment consists of a series of PAST actions (i.e., past 10 actions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;buy (green)&lt;/li&gt;&#xA;&lt;li&gt;hold (black)&lt;/li&gt;&#xA;&lt;li&gt;sell (red)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A return (basis for the reward) is generated when conditions meet:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;buy -&gt; sell (close buy position and immediately enter sell)&lt;/li&gt;&#xA;&lt;li&gt;sell -&gt; buy (close sell position and immediately enter buy)&lt;/li&gt;&#xA;&lt;li&gt;buy -&gt; hold (close buy and wait for next action)&lt;/li&gt;&#xA;&lt;li&gt;sell -&gt; hold (close sell and wait for next action)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The return is maximum (equals 2) when the agent keeps the series of one-type actions starting at hill and down to valley, and vice versa for other sign positions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At this chart you could see that in the bottom plot the agent learned to guess the direction right in most cases, but the return is not maximized to the value of 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is not just to observe the leraning statistics, but to find an optimal policy to be applied to an out of sample data, with further exploration and policy refinement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuitive guess is that the learning rate is too high when theagent is close to optimality and is forced to escape the region:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rfnr8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rfnr8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get almost perfect solution of the optimal policy in my toy task, however, the neural network is not quite stable, and I have to manually choose the moment to stop learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how the system evolved during training:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BlmnY.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BlmnY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again the optimality seems to be reached at some point (with later fluctuations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning behaviour of the agent:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/60AW7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/60AW7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that after aprx. 30 000 iterations the anent starts to make clear sequences of actions leading to maximized reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At test (with greedy action selection) the behaviour is that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vpyTy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vpyTy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And test trades are also pretty clear (not perfect though):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3p1Ux.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3p1Ux.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 2 points I want to improve:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Stopping criteria for learning&lt;/li&gt;&#xA;&lt;li&gt;Stability of max-Q and averaged return (and sum of returns) during&#xA;test.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42953" LastEditorUserId="42953" LastEditDate="2018-01-22T17:17:23.527" LastActivityDate="2018-01-22T17:18:07.887" Title="Graphical results of Q-Learning: is improvement possible by parameter tweaking?" Tags="|reinforcement-learning|q-learning|" AnswerCount="0" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="26569" PostTypeId="1" CreationDate="2018-01-12T15:30:44.483" Score="1" ViewCount="70" Body="&lt;p&gt;I am reading a recent paper &quot;&lt;a href=&quot;https://arxiv.org/abs/1710.05468&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generalization in Deep Learning&lt;/a&gt;&quot; and I am unable to understand a step. In this step, they first take neural network as a direct acyclic graph(DAG) and described output of neural network as follows,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dKD8E.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dKD8E.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then they decomposed generalization gap as follows,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PoSEr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PoSEr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;without any explanation. Here z is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ixwkR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ixwkR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain this or point me towards the right direction.&lt;/p&gt;&#xA;" OwnerUserId="44543" LastActivityDate="2018-01-12T15:30:44.483" Title="How can we decompose generalization gap as done in the paper &quot;Generalization in Deep Learning&quot;?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26570" PostTypeId="1" CreationDate="2018-01-12T16:03:47.373" Score="0" ViewCount="804" Body="&lt;p&gt;I am using &lt;code&gt;GridSearchCV&lt;/code&gt; to train a dataset with several parameters.  Using the methods on the object I am able to get to the &quot;best&quot; parameter set (based on the scoring function).  I would like to be able to get to one (or more) of the other models.  Is there a way to do that?  Is that saved?  If so what is the way to get to those?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;from sklearn import datasets&#xA;from sklearn.linear_model import SGDClassifier&#xA;from sklearn.model_selection import GridSearchCV&#xA;&#xA;bc = datasets.load_breast_cancer()&#xA;X,Y = bc[&quot;data&quot;],bc[&quot;target&quot;]&#xA;&#xA;sgd = SGDClassifier(loss='log', penalty='l1', max_iter=1E6, tol=1E-6, shuffle=True, random_state=123456)&#xA;param_grid = [{'alpha':np.append(10.**-np.arange(3,6), 10.**-np.arange(3,6)/2)}]&#xA;&#xA;sgd_gs = GridSearchCV(sgd, param_grid, scoring='roc_auc', n_jobs=4, cv=4)&#xA;sgd_gs.fit(X, Y)&#xA;results_pd = pd.DataFrame(sgd_gs.cv_results_).iloc[:,2:7].sort_values(by='rank_test_score').reset_index(drop=True)&#xA;results_pd&#xA;&#xA;scores_train = sgd_gs.best_estimator_.predict_proba(X)[:,1]&#xA;print sum(scores_train)&#xA;&#xA;#**code below does not seem to work**&#xA;params = results_pd.loc[2, 'params']&#xA;scores_train = sgd_gs.best_estimator_.set_params(**params).predict_proba(X)[:,1]&#xA;print sum(scores_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32199" LastActivityDate="2018-01-12T16:54:56.300" Title="Getting to other parameter sets from GridSearchCV in sklearn" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26572" PostTypeId="2" ParentId="26564" CreationDate="2018-01-12T16:11:29.223" Score="1" Body="&lt;p&gt;Yes. There has been work going on in the healthcare domain which involves NLP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Resources like &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pubmed&lt;/a&gt; have been put together for the exact same purpose. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can make use of the freely available &lt;a href=&quot;http://bio.nlplab.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pubmed vectors&lt;/a&gt; for training your models like maybe a sentence tagging style model for identification. We(my team at work) use them for identification of medical terms like diseases, medicines, etc.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2018-01-12T16:11:29.223" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26573" PostTypeId="2" ParentId="22556" CreationDate="2018-01-12T16:31:48.957" Score="1" Body="&lt;p&gt;I believe what you are looking are graph &lt;a href=&quot;http://en.wikipedia.org/wiki/Centrality&quot; rel=&quot;nofollow noreferrer&quot;&gt;centrality&lt;/a&gt; measures. The &lt;em&gt;Betweenness centrality&lt;/em&gt; for each vertex reflects the ratio of shortest paths going through that vertex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$g(v)=\sum _{{s\neq v\neq t}}{\frac  {\sigma _{{st}}(v)}{\sigma _{{st}}}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\sigma_{st}$ is the total number of shortest paths from node $s$ to node $t$ and $\sigma_{st}(v)$ is the number of those paths that pass through $v$. Also check &lt;a href=&quot;http://en.wikipedia.org/wiki/Graph_drawing&quot; rel=&quot;nofollow noreferrer&quot;&gt;graph drawing&lt;/a&gt; algorithms. Some of them such as force based and spectral methods can give reasonable results for your purpose, but with no theoretical guaranties. &lt;/p&gt;&#xA;" OwnerUserId="10216" LastActivityDate="2018-01-12T16:31:48.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26574" PostTypeId="2" ParentId="26570" CreationDate="2018-01-12T16:54:56.300" Score="1" Body="&lt;p&gt;If the parameter &lt;code&gt;refit&lt;/code&gt; is set to &lt;code&gt;True&lt;/code&gt;, the &lt;code&gt;GridSearchCV&lt;/code&gt; object will have the attributes &lt;code&gt;best_estimator_&lt;/code&gt;, &lt;code&gt;best_score_&lt;/code&gt; etc. (&quot;Best&quot; measured in terms of the metric provided through the &lt;code&gt;scoring&lt;/code&gt; parameter.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For other models you can only access the cross-validation scores from &lt;code&gt;cv_results_&lt;/code&gt;. The models are not stored. If you want to use them to make predictions, you have the re-train the model where you explicitly set the parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use all the models to make predictions, it would be better to write your own code and simply loop over the parameter constellations without using &lt;code&gt;GridSearchCV&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-01-12T16:54:56.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26575" PostTypeId="2" ParentId="18848" CreationDate="2018-01-12T17:37:37.133" Score="3" Body="&lt;p&gt;You can add more features to your dataset as below.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can try nolds package if your data is from a highly non linear process.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;max, min, mean , skew, kurtosis, and if possible some rolling stats.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am working on something similar, and I asked a &lt;a href=&quot;https://datascience.stackexchange.com/q/25502/26329&quot;&gt;related question&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26329" LastEditorUserId="29575" LastEditDate="2018-01-12T20:23:05.697" LastActivityDate="2018-01-12T20:23:05.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26576" PostTypeId="2" ParentId="26565" CreationDate="2018-01-12T17:38:34.087" Score="2" Body="&lt;p&gt;You can't use numpy to sum the values of a dictionary. You have to use &lt;code&gt;sum&lt;/code&gt; function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;total = sum(labels_dict.values())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can check that &lt;code&gt;total&lt;/code&gt; is an integer :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(type(total))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;class 'int'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="27003" LastEditorUserId="27003" LastEditDate="2018-01-12T18:03:21.910" LastActivityDate="2018-01-12T18:03:21.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26577" PostTypeId="1" CreationDate="2018-01-12T18:24:46.100" Score="4" ViewCount="419" Body="&lt;p&gt;How do neural networks account for &lt;strong&gt;outliers&lt;/strong&gt; and &lt;strong&gt;overfitting&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="33020" LastEditorUserId="28175" LastEditDate="2018-02-11T21:23:38.537" LastActivityDate="2018-02-11T21:23:38.537" Title="How do neural networks account for outliers?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26578" PostTypeId="1" CreationDate="2018-01-12T19:04:18.553" Score="4" ViewCount="3308" Body="&lt;p&gt;I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, specs, ...) out of 4 possible brands, how do you encode the brand variable? My answer was &lt;code&gt;one hot encoding&lt;/code&gt;, it was accepted but then they asked me to do it explicitly and I sketched something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;brand A -&amp;gt; [1,0,0,0]&#xA;brand B -&amp;gt; [0,1,0,0]&#xA;brand C -&amp;gt; [0,0,1,0]&#xA;brand D -&amp;gt; [0,0,0,1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then, I was corrected under the reason that these columns were not independent. And that the solution should have been three binary columns instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later it hit me that I do not know why independence is required, and also that three binary variables are not independent. Two would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone provide some explanation to help with my confusion?&lt;/p&gt;&#xA;" OwnerUserId="39830" LastEditorUserId="29575" LastEditDate="2018-01-12T20:33:30.543" LastActivityDate="2018-01-13T08:03:00.157" Title="One hot encoding of target space" Tags="|data-cleaning|preprocessing|classifier|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26579" PostTypeId="2" ParentId="26577" CreationDate="2018-01-12T20:40:08.237" Score="2" Body="&lt;p&gt;Here's a math answer for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of tweaking parameters $\theta$ to make $f$ as close as possible to $p$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(\theta) \approx p(X, Y)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sup&gt;Side note: usually $f$ is considered to approximate the conditional $p(Y|X)$, but it can be viewed more generally.&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in this terms, &lt;em&gt;outliers&lt;/em&gt; are the values $(x, y)$ clearly outside of the distribution $p(X, Y)$. Hence, a neural network &lt;em&gt;can not&lt;/em&gt; account for them, this has to be done separately. One possibility is to gather more data to make these outliers look ordinary, i.e., from $p(X, Y)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for &lt;em&gt;overfitting&lt;/em&gt;, it's completely different term, which relates to inability to &lt;em&gt;generalize&lt;/em&gt;. This happens because in practice the true distribution $p(X, Y)$ is never fully known. Instead, the researches have the sample of it -- $(\hat{X}, \hat{Y})$, a.k.a. the &lt;em&gt;training data&lt;/em&gt;. If the dimensionality of $\theta$ is large, $f$ can approximate $p(\hat{X}, \hat{Y})$ so well that it actually learns the noise and fails to capture $p(X, Y)$. There are ways to deal with overfitting, most importantly &lt;a href=&quot;https://en.wikipedia.org/wiki/Regularization_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;regularization&lt;/em&gt;&lt;/a&gt;, which is equivalent to adding noise to $(\hat{X}, \hat{Y})$. So the answer to this question is: it can be extended to account for it, by using special techniques. But if the researcher does nothing, the neural network won't learn to deal with overfitting by itself.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-01-12T20:40:08.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26580" PostTypeId="2" ParentId="26544" CreationDate="2018-01-12T20:46:29.740" Score="1" Body="&lt;p&gt;Finding the &lt;em&gt;minimum&lt;/em&gt; number of spheres likely makes this problem NP-hard, and a variant of the &lt;strong&gt;set cover problem&lt;/strong&gt;. Good luck. It gets particularly difficult if you want to consider balls tha can be inbetween of data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two obvious clustering algorithms that will find an approximation: Leader and Complete Linkage Agglomerative Hierarchical Clustering. But they will &lt;em&gt;not&lt;/em&gt; find the maximum, only an approximation, and probably with no guarantees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, your assumptions are wrong: if you cover the data with balls of radius r, the maximum distance of two points will be 2*r.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-12T20:46:29.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26581" PostTypeId="1" AcceptedAnswerId="26582" CreationDate="2018-01-12T21:08:05.720" Score="3" ViewCount="4282" Body="&lt;p&gt;I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I achieved imputing missing values in independant variables. For numerical data by using the Imputer with the means strategy and for one categorical variable I used the Labelencoder and afterwards imputed with the mode strategy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now I face the issue that the dependant variable $y$ also contains missing values. Should I delete those lines or should I impute $y$ which is numerical. &lt;/p&gt;&#xA;" OwnerUserId="44556" LastEditorUserId="29575" LastEditDate="2018-01-12T21:45:21.993" LastActivityDate="2022-05-06T03:58:16.323" Title="Should I Impute target values?" Tags="|machine-learning|regression|data-cleaning|preprocessing|data-imputation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26582" PostTypeId="2" ParentId="26581" CreationDate="2018-01-12T21:53:18.330" Score="4" Body="&lt;p&gt;For the missing data problem, one thing to be aware of, is the missingness mechanism. Depending of the dataset, the NA's (Missing Values) you have could be a result of a condition of the phenomenon and you shouldn't impute then using mean but maybe. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides, for the dependent variable, if you want to train a model with the independent ones to predict this, let's say Y, you wouldn't train a model using this observation with NA on the dependent (target?). Then, you would drop this lines or maybe using another technique which takes into account the dependence of the other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think a good start is to give look at this:&#xA;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/missing.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Missing-data imputation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It shows the limitations of using some approaches like yours and defines the mechanisms of missing data.&lt;/p&gt;&#xA;" OwnerUserId="30448" LastEditorUserId="29575" LastEditDate="2018-01-12T22:28:53.260" LastActivityDate="2018-01-12T22:28:53.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26586" PostTypeId="2" ParentId="26578" CreationDate="2018-01-13T07:21:59.413" Score="1" Body="&lt;p&gt;For certain models, it can be important to make sure the inputs are linearly independent. I'm pretty sure this is never the case for the outputs. If we have more than two classes, it's extremely difficult if not impossible to output class-probabilities if we don't have a specific dimension for each class in the output vector. In other words, your suggestion has much more explanatory power than what was suggested by your interviewer, which is solving a problem that exists on the inputs for some models but not for the outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In multi-class problems, it is indeed the norm that each class has their own dimension in the output vector as you suggested. Either the interviewer was trying to look smart and confused input and output encoding (which still would only apply to certain families of models), or you aren't remembering the events of the interview properly. If you're confident that the interview went as you described, you should consider complaining to your point of contact that you were criticized for giving a correct answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feel free to cite the following as popular/canonical examples of multi-class classifiers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Returns: array of shape = [n_samples, n_classes]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://keras.io/losses/#categorical_crossentropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/losses/#categorical_crossentropy&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;when using the categorical_crossentropy loss, your targets should be in categorical format (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros expect for a 1 at the index corresponding to the class of the sample)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-13T08:03:00.157" LastActivityDate="2018-01-13T08:03:00.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26587" PostTypeId="1" CreationDate="2018-01-13T08:19:18.083" Score="0" ViewCount="1496" Body="&lt;p&gt;I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL table to dataframe.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_sql(&quot;select * from naivebayes_player&quot;,connection)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I'm facing problem in comparing dataframe column value with a string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a column named classification in MySQL table which has 2 fixed values 'RS' or 'NRS' stored in varchar(10) format. Since I've converted an entire table into dataframe whenever I calculate the count of 'RS' values in classification column in dataframe it always returns 0. But actually, there are 63 entries of 'RS'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;total_RS = df['classification'][df['classification']=='RS'].count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In above line of code I'm trying to find out all records where classification is 'RS' which should be 63 but I'm getting 0. What am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried above code when reading data from CSV instead of MySQL table and everything worked fine.&lt;/p&gt;&#xA;" OwnerUserId="43679" LastActivityDate="2023-12-24T15:00:45.073" Title="Comparing dataframe object with string value in django" Tags="|machine-learning|pandas|dataframe|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26588" PostTypeId="1" AcceptedAnswerId="26859" CreationDate="2018-01-13T08:27:47.840" Score="2" ViewCount="955" Body="&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt;: I've N classification models (independent), for each of these N models, I've different versions (eg: V&lt;sub&gt;0&lt;/sub&gt;, V&lt;sub&gt;1&lt;/sub&gt;, ..., V&lt;sub&gt;final_production&lt;/sub&gt;,V&lt;sub&gt;experimental&lt;/sub&gt;). I'm looking for a way to store my datasets efficiently on the cloud (for redudency).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: We're not talking about BigData here.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current Solution&lt;/strong&gt;: Created a private GitHub repo. Made N directories and inside, pushed different dataset versions as different files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there better solutions for this (because I feel VCS is an overkill for this problem)?&lt;/p&gt;&#xA;" OwnerUserId="9388" LastActivityDate="2018-01-22T08:15:48.957" Title="Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions?" Tags="|dataset|data|version-control|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26589" PostTypeId="2" ParentId="26587" CreationDate="2018-01-13T08:40:20.967" Score="0" Body="&lt;p&gt;I don't see anything glaringly wrong with your code, but you can try this instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['classification'].value_counts()['RS']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you omit the indexing at the end there, the call to &lt;code&gt;value_counts()&lt;/code&gt; will concretely show you what values appear with what frequency in the &lt;code&gt;classification&lt;/code&gt; column, which might help you debug what's going on if this still doesn't work.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-13T08:40:20.967" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26590" PostTypeId="2" ParentId="23608" CreationDate="2018-01-13T10:14:03.133" Score="0" Body="&lt;p&gt;I came across a similar problem. So I thought instead of finding a solution to convert the text into something understandable, we can process directly with character-based model. This link will help you a lot &lt;a href=&quot;https://offbit.github.io/how-to-read/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://offbit.github.io/how-to-read/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="44240" LastActivityDate="2018-01-13T10:14:03.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26592" PostTypeId="1" CreationDate="2018-01-13T12:50:38.070" Score="2" ViewCount="44" Body="&lt;p&gt;I'm researching on abstractive text summarization, and has come across many recent papers. They all seem to be focusing on Sequence to Sequence models based on RNNs. Apart from RNNs, what other approaches are there when it comes to abstractive text summarization? &#xA;Does ontology-based summarization revolve around the same seq2seq model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the material I have come across are research papers on this subject; what better sources are there to get an understanding of the underlining concepts of abstractive summarization?&lt;/p&gt;&#xA;" OwnerUserId="44573" LastActivityDate="2018-01-13T12:50:38.070" Title="What other approaches are there for abstractive summarization, other then seq2seq?" Tags="|nlp|rnn|automatic-summarization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26593" PostTypeId="1" CreationDate="2018-01-13T13:11:28.850" Score="1" ViewCount="1099" Body="&lt;p&gt;I was trying to apply SVM into a dataset to find its accuracy. I also applied bagging and boosting on the same dataset and they worked properly. But when I tried to fit SVM into a model its not working. I couldn't understand whats the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm providing the sample of my dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_new.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oCH6o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oCH6o.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# define X and y&#xA;feature_cols = ['age','workclass','fnlwgt','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# X is a matrix, hence we use [] to access the features we want in feature_cols&#xA;X = train_new[feature_cols]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# y is a vector, hence we use dot to access 'label'&#xA;y = train_new['label']&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;from sklearn import svm&#xA;model = svm.SVC(kernel='linear', C=1, gamma=1)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and while trying to fit with model it's almost minutes passed and it's just look like, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fzHNk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fzHNk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the actual problem and could I solve it ?&lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2018-02-14T09:10:54.103" Title="SVM is not fitted when tried to fit it into a model" Tags="|python|classification|scikit-learn|svm|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26595" PostTypeId="2" ParentId="26556" CreationDate="2018-01-13T15:01:28.643" Score="0" Body="&lt;p&gt;As you are new to ML, I will try to explain in my simplest way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. I am unable to figure out how to make such conclusions with feature study in neural networks or other machine learning techniques.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning has many applications, what you are talking about here comes under the term &lt;strong&gt;Inference&lt;/strong&gt;. It means to understand- how your output is affected as your input changes. I suggest you follow the book- &lt;strong&gt;An Introduction to Statistical Learning with Applications in R&lt;/strong&gt;. On page 19 of this book, it is given-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Inference&#xA;&#xA;We are often interested in understanding the way that Y is affected as X1,...,Xp change.&#xA;........&#xA;We instead want to understand the relationship between X and Y.&#xA;&#xA;- Which predictors are associated with the response? &#xA;- What is the relationship between the response and each predictor?&#xA;- Can the relationship between Y and each predictor be adequately summarized using a linear equation, or is the relationship more complicated?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have not posted the whole thing here, just some important points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here, instead of prediction, you just analyze your model. After analyzing you can make such conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Is there a way to quantify the likelihood in ANN similar to logistic regression wherein regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understand, ANN has multiple layers. It is not like logistic regression which just defines one coefficient for each predictor. In ANN, the coefficients are defined for each layer separately, and in each layer, for each node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-01-13T15:01:28.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26596" PostTypeId="1" AcceptedAnswerId="26629" CreationDate="2018-01-13T15:11:17.830" Score="1" ViewCount="189" Body="&lt;p&gt;I'm new to machine learning domain. I want to build a Classification tree for Binary classification of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a training data set of 269 records out of which 56 records belong to 'yes' class and 213 records to 'no' class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this data imbalanced for building CART model? Do I need to undersample 'no' class records? Also from Gini index, Chi-Square and Information Gain, which algorithm is best for node splitting? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.:- I can't increase the size of the dataset further.&lt;/p&gt;&#xA;" OwnerUserId="43679" LastEditorUserId="40853" LastEditDate="2018-01-15T02:48:35.180" LastActivityDate="2018-01-15T09:58:36.483" Title="Identifying whether training dataset is imbalanced in CART" Tags="|machine-learning|classification|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26597" PostTypeId="1" CreationDate="2018-01-13T15:26:31.233" Score="41" ViewCount="11040" Body="&lt;p&gt;I am a beginner to neural networks and have had trouble grasping two concepts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.&lt;/li&gt;&#xA;&lt;li&gt;How does one decide the number of neurons in each middle layer? Is it recommended having an equal number of neurons in each middle layer or does it vary with the application?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="39854" LastEditorUserId="28175" LastEditDate="2018-01-16T21:50:18.703" LastActivityDate="2020-08-20T18:36:36.680" Title="How to set the number of neurons and layers in neural networks" Tags="|machine-learning|neural-network|deep-learning|hyperparameter|hyperparameter-tuning|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26598" PostTypeId="1" AcceptedAnswerId="26601" CreationDate="2018-01-13T15:50:38.783" Score="2" ViewCount="441" Body="&lt;p&gt;I'm very new to machine learning and want to understand the general process by which it is carried out. I've worked through the famous 'iris' tutorial and want to ask if the principles in that tutorial are applicable to every future machine learning project I undertake.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I work in biological sciences, and am interested in applying machine learning algorithms to biological sequence data as away of categorizing classes or doing unsupervised clustering. From my understanding, each project starts with:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Me defining the 'scope' or aim of what I want to learn from the raw data.&lt;/li&gt;&#xA;&lt;li&gt;Generating features/attributes from my OWN code/algorithms that could potentially differentiate between the two classes&lt;/li&gt;&#xA;&lt;li&gt;This basically generates a huge matrix of x_features by y_entries&lt;/li&gt;&#xA;&lt;li&gt;Feed the matrix into a machine learning algorithm (I'm sure this is vastly oversimplified).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To give an example, say I have 10,000 protein sequences, and I believe 5000 are 'Class1' and 5000 are 'Class2', but I do not know how to differentiate them by eye. I need to generate x_features (in some informed way) of this sequence using my own custom algorithms, and feed the resulting 10000 entries into an algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right approach? I'd be eternally grateful if someone could direct me towards a beginners tutorial that revolves around analyzing biological sequence data.&lt;/p&gt;&#xA;" OwnerUserId="44581" LastActivityDate="2018-01-13T18:15:02.940" Title="General Machine Learning Workflow Question" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26600" PostTypeId="1" AcceptedAnswerId="26602" CreationDate="2018-01-13T16:25:10.343" Score="3" ViewCount="160" Body="&lt;p&gt;I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is to do with the threshold value. this is a neural network - q learning specific variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;basically the theshold is a value that is between 0 and 1, you then make a random number between 0 and 1, if this random number is larger than the threshold then you pick a completely random choice, otherwise the neural network chooses by finding the largest q value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is that with this threshold value, i am currently implementing it as starting at almost 0, then increasing linearly until it reaches 1 by the time the program has reached the final  iteration. Is this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason i suspect this is incorrect is that when plotting an error graph from training the neural network, the program doesnt not learn at all, but when the threshold reaches almost 1, it starts to learn very fast, and if you run more iterations after it reaches 1, the all the game sets in the replay memory become the same and the error is basically 0 from their on in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any feedback is greatly appreciated and if this question in unclear in anyway just let me know and i will try and fix it. Thank you to anyone who helps out.&lt;/p&gt;&#xA;" OwnerUserId="44530" LastActivityDate="2018-01-13T18:48:05.743" Title="Neural network q learning for tic tac toe - how to use the threshold" Tags="|machine-learning|neural-network|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26601" PostTypeId="2" ParentId="26598" CreationDate="2018-01-13T18:15:02.940" Score="3" Body="&lt;p&gt;I agree with most of the answer. However, I think you are missing some points including the cross-validation step. I try below to provide an overview of a common machine learning project. I assume a common project is a supervised machine learning problem (like iris dataset). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. You defining the 'scope' or aim of the project :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You have to define the purpose of the learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When working with business in a company, it is a good idea to correctly express the need, the value of the project and its goal. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;You have to define the evaluation metrics (accuracy, recall, F1 score, AUC...). You can also define a minimum result you want to reach (say 80% accuracy for example).&lt;/li&gt;&#xA;&lt;li&gt;You can also ask yourself about the level of interpretability you need (do you care about model explanation? If no, maybe you could try more blackbox algorithms such as boosting, neural networks...).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Explore your data :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Using statistics, visualization and intuition, try to learn your dataset and understand your features and labels.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can also search for missing data and outliers. Correct these observations refers as data cleaning process.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Understanding your input variables will greatly help you to create and select relevant features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Generating features/attributes from your own code/algorithm&lt;/strong&gt; :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;This phase refers as features engineering. It is about creating features relevant to the learning problem.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In this phase, you can clean your missing data and outliers in order to help the learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can derive new features from input variables relevant to your learning problem (handle categorical variables, rescale your features, apply transformations on input variables).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. cross-validation :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Cross-validation refers to your algorithm evaluation. In supervised machine learning, it is common, &lt;em&gt;at least&lt;/em&gt;, to split dataset into 3 datasets (train, validation and test). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train dataset (about 60% of data) aims to train the algorithm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Validation dataset (20% of data) helps to find the best hyperparameters of your model (max depth for a tree, regularization for a linear/logistic regression...).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, test set (20% of data) gives you the true result you get on unseen data. It is the final evaluation.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;5. Machine learning, feed the matrix into an algorithm :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In this part, you train machine learning algorithms with regards to the cross-validation process (part 4).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can test different models. Some yield different performance results. Interpretability is not the same neither.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To help the learning, you can diagnose your algorithm performs on both train and validation sets. This diagnostic is also called &lt;em&gt;learning curves&lt;/em&gt;. It can tell you how to improve your learning. The purpose of learning curve is to help handle the underfitting/overfitting tradeoff. Underfitting is when you have a large bias error meaning your algorithm is not complex enough while overfitting means your algorithm is too complex and learns perfectly but is not able anymore to generalize learning on new unseen observations.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can also look at residuals (errors between predictions and real values) to improve your algorithm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Make features selection may also improve your algorithm learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;6. Restitution&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Interpret the model and the performance you get.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Create restitutions to business? Run into production?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Improve your machine learning model and performance is mostly about improving the above introduced points. By making new exploration, create new features, try a more powerful algorithm and so on, you can reach best results. Machine learning is a whole pipeline you have to optimize. I also think machine learning projects managements are really suitable with agile approaches.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-01-13T18:15:02.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26602" PostTypeId="2" ParentId="26600" CreationDate="2018-01-13T18:48:05.743" Score="1" Body="&lt;p&gt;You are effectively implementing $\epsilon$-greedy action selection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The usual way to represent this in RL, at least that I am familiar with, is not as a &quot;threshold&quot; for probability of choosing the best estimated action, but as a small probability, $\epsilon$, of not choosing the best estimated action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For consistency with RL literature that I know, I will use the $\epsilon$-greedy form, so instead of considering what happens as your threshold rises from 0 to 1, I will consider what happens when $\epsilon$ drops from 1 to 0. It is the same thing. I hope you can either adjust to using $\epsilon$ or mentally convert the rest of this answer so it is about your threshold . . .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When monitoring Q-Learning, you have to be careful how you measure success. Monitoring the behaviour on the learning games will give you slightly off feedback. The agent will make exploratory moves (with probability $\epsilon$), and the results from a learning game might involve the agent losing even though it already has a policy good enough to not lose from the position where it started exploring. If you want to measure how well the agent has learned the game, you have to stop the training stage and play some games with $\epsilon$ set to $0$. I suspect this could be one problem - that you are measuring results from behaviour during training (note this would work with SARSA)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, choosing values that are too high or low for your problem will reduce the speed of learning. High values interfere with Q-learning because it has to reject some of data from exploratory moves, and the agent will rarely see a full game played using its preferred policy. Low values stifle learning because the agent does not explore different options enough, just repeating the same game play when there might be better moves that it has not tried. For Tic Tac Toe and Q-learning I would suggest picking a value of $\epsilon$ between $0.01$ and $0.2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, with Q-learning there is no need to change the value of $\epsilon$. You should be able to pick a value, say $0.1$, and stick with it. The agent will still learn an optimal policy, because Q-learning is an &lt;a href=&quot;https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning&quot;&gt;off-policy algorithm&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-01-13T18:48:05.743" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26603" PostTypeId="2" ParentId="26597" CreationDate="2018-01-13T22:45:43.703" Score="13" Body="&lt;p&gt;Short Answer: It is very related to the dimensions of your data and the type of the application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Choosing the right number of layers can only be achievable with practice. &lt;em&gt;There is no general answer to this question yet&lt;/em&gt;. By choosing a network architecture, you constrain your space of possibilities (hypothesis space) to a specific series of tensor operations, mapping input data to output data. In a DeepNN each layer can only access information present in the output of the previous layer. If one layer drops some information relevant to the problem at hand, this information can never be recovered by later layers. This is usually referred as &quot;&lt;strong&gt;Information Bottleneck&lt;/strong&gt;&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Information Bottleneck is a double-edged sword:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) If you use a few number of layers/neurones, then the model will just learn a few useful representations/features of your data and lose some important ones, because the capacity of middle layers are very limited (&lt;em&gt;underfitting&lt;/em&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) If you use a big number of layers/neurones, then the model will learn  too much representations/features that are specific to the training data and don’t generalize to data in the real-world and outside of your training set (&lt;em&gt;overfitting&lt;/em&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Useful links for examples and more finding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &lt;a href=&quot;https://livebook.manning.com#!/book/deep-learning-with-python/chapter-3/point-1130-232-232-0&quot; rel=&quot;noreferrer&quot;&gt;https://livebook.manning.com#!/book/deep-learning-with-python/chapter-3/point-1130-232-232-0&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] &lt;a href=&quot;https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/&quot; rel=&quot;noreferrer&quot;&gt;https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2018-01-13T22:45:43.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26604" PostTypeId="1" AcceptedAnswerId="26609" CreationDate="2018-01-13T23:04:34.337" Score="1" ViewCount="465" Body="&lt;p&gt;This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i am currently doing is when the program plays through the game, if the number of game sets recorded has reached the max amount, every time the program makes a move, it will pick a random game state from its memory and back propagate using that game state and reward. this will then continue every time the program makes a move as the replay memory will always be full from then on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The association between rewards and game state and action from history, is that when a game has been completed, and the rewards have been calculated for each step, meaning that the total reward per step has been calculated, the method i use to calculate the reward is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q(s,a) += reward * gamma^(inverse position in game state)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in this case, gamma is a value predetermined to reduce the amount that the reward is taken into account the further you go back, and the inverse position in game state means that if there have been 5 total moves in a game, then the inverse position in game state when changing the reward for the first move would be 5, then for the second, 4, third 3 and so on. this just allows the reward to be taken less into account the earlier the move is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should this allow the program to learn correctly?&lt;/p&gt;&#xA;" OwnerUserId="44530" LastEditorUserId="836" LastEditDate="2018-01-14T07:42:31.283" LastActivityDate="2018-01-14T16:08:18.573" Title="Q learning Neural network Tic tac toe - When to train net" Tags="|machine-learning|neural-network|q-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26606" PostTypeId="2" ParentId="20435" CreationDate="2018-01-14T04:05:47.680" Score="1" Body="&lt;p&gt;It is not important that the data is separable in 2-D. You can try a PCA + SVM combination to check at which point (number of dimensions) the data is best separated. You can do it with Grid Search on the parameters of PCA. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are just visualising the data, you can try T-SNE embedding as well.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2018-01-14T04:05:47.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26607" PostTypeId="1" CreationDate="2018-01-14T09:10:39.627" Score="1" ViewCount="4871" Body="&lt;p&gt;For classification, I was trying to convert categorical data into numeric by applying OneHotEncoder. But it shows error &lt;code&gt;could not convert string to float&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the sample of my categorical data set and code of One Hot Encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nKzXM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nKzXM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# TODO: create a OneHotEncoder object, and fit it to all of X&#xA;from sklearn.preprocessing import OneHotEncoder&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# 1. INSTANTIATE&#xA;enc = OneHotEncoder()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# 2. FIT&#xA;enc.fit(train_obj)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# 3. Transform&#xA;train_ = enc.transform(train_obj)&#xA;train_.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And error message:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9t2gk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9t2gk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't understand what's the problem &amp;amp; how could I solve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I apply &lt;code&gt;get_dummies()&lt;/code&gt; method, is it similar of OneHotEncoder ?&lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2018-01-14T13:06:24.237" Title="Faced problem while applying OneHotEncoder" Tags="|python|classification|data-mining|categorical-data|encoding|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26608" PostTypeId="1" CreationDate="2018-01-14T09:13:41.307" Score="3" ViewCount="279" Body="&lt;p&gt;I'm studying Logistic Regression, there is a thing that I'm not able to understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know what the Logistic Function is, I studied it. But I don't understand why the logistic function is equal to the dot product between $w$ and $x$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \log \left(\frac{p}{1-p}\right)=\bar{\omega }\cdot \bar{x} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me these are two completely different concepts, I don't understand why are related in this manner. Can you help me?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="29575" LastEditDate="2018-01-14T14:54:03.007" LastActivityDate="2018-01-14T14:54:03.007" Title="Question about Logistic Regression" Tags="|machine-learning|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26609" PostTypeId="2" ParentId="26604" CreationDate="2018-01-14T09:32:13.110" Score="2" Body="&lt;p&gt;This update scheme:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Q(s,a) += reward * gamma^(inverse position in game state)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;has a couple of problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;You are - apparently - incrementing Q values rather than training them to a reference target. As a result, the estimate for Q will likely diverge, predicting total rewards that are impossibly high or low. Although in your case with a zero sum game and initial random moves, it may just random walk around zero for a long time first.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ignoring the increment, the formula you are using is not from Q-learning, but effectively &lt;a href=&quot;http://www.incompleteideas.net/book/ebook/node54.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;on policy Monte Carlo control&lt;/a&gt;, because you use the end-of-game sum of rewards as the Q value estimate. In theory, with a few tweaks this can be made to work, but it is a different algorithm than you say you want to learn.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is worth clarifying a few related terms (you clearly know these  already, but I want to make sure you have them separated in your  understanding of the rest of the answer):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Reward&lt;/em&gt;. In RL, a reward (a real number) can be returned on every increment, after taking an action. The set of rewards is part of the problem definition. Often noted as $R$ or $r$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Return&lt;/em&gt; (aka &lt;em&gt;Utility&lt;/em&gt;). The sum of all - maybe discounted - rewards from a specific point. Often noted as $G$ or $U$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Value&lt;/em&gt;, as in state value or action value. This is usually the expected return from a specific state or state, action pair. $Q(S_t, A_t)$ is the expected return when in state $S_t$ and taking action $A_t$. Note that using $Q$ does not make your algorithm Q-learning. The $Q$ action value is the basis for several RL algorithms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your formula &lt;code&gt;reward * gamma^(inverse position in game state)&lt;/code&gt; gives you the &lt;em&gt;Return&lt;/em&gt;, $G$ seen in a sampled training game, $G_t = \gamma^{T-t} R_T$ where $T$ is the last time step in the game. That's provided the game only has a single non-zero reward at the end - in your case that is true. So you could use it as a training example, and train your network with input $S_t, A_t$ and desired output of $G_t$ calculated in this way. That should work. However, this will only find the optimal policy if you decay the exploration parameter $\epsilon$ and also remove older history from your experience table (because the older history will estimate returns based on imperfect play).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the usual way to use experience replay with Q learning:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When saving experience, store $S_t, A_t, R_{t+1}, S_{t+1}$ - note that means storing immediate &lt;em&gt;Reward&lt;/em&gt;, not the &lt;em&gt;Return&lt;/em&gt; (yes you will store a lot of zeroes). Also note you need to store the next state.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When you have enough experience to sample from, typically you do not learn from just one sample, but pick a minibatch size (e.g. 32) and train with that many each time. This helps with convergence.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For Q-learning, your TD target is $R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$, and you bootstrap from your current predictions for Q, which means:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For each sample in the minibatch, you need to calculate the predicted Q value of all allowed actions from the &lt;em&gt;next&lt;/em&gt; state $S_{t+1}$ - using the neutral network. Then use the maximum value from each state to calculate $\text{max}_{a'} Q(S_{t+1}, a')$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train your network on the minibatch for a single step of gradient descent, with NN inputs $[S_t, A_t]$ and training label of the TD target from each example.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes that means you use the same network to first predict and then learn from a formula based on those predictions. This can be a source of problems, so you may need to maintain two networks, one to predict and one that learns. Every few hundred updates, refresh the prediction network as a copy of the current learning network. This is quite common addition to experience replay (it is something that Deep Mind did for DQN), although may not be necessary in your case for a game as simple as Tic Tac Toe.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The TD target is a bootstrapped and biased estimate of expected $G$. The bias is a potential source of problems (you may read that using NNs with Q-learning is not stable, this is one of the reasons why). However, with the right precautions, such as experience replay, the bias will reduce as the system learns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In case you are wondering, it is the use of both $S_t$ (as NN input) and $S_{t+1}$ (to calculate TD target) in the Q-learning algorithm, which effectively distributes the end-of-game reward back to the start of the game's Q value.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In your case (and in many episodic games), it should be fine to use no discount, i.e. $\gamma = 1$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;From your previous question, you noted that you were training two competing agents. That does in fact cause a problem for experience replay. The trouble is that the next state you need to train against will be the state &lt;em&gt;after&lt;/em&gt; the opponent has made a move. So the opponent is technically viewed as being part of the environment for each agent. The agent learns to beat the current opponent. However, if the opponent is also learning an improved strategy, then its behaviour will change, meaning your stored experiences are no longer valid (in technical terms, the environment is non-stationary, meaning a policy that is optimal at one time may become suboptimal later). Therefore, you will want to discard older experience relatively frequently, even using Q-learning, if you have two self-modifying agents.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-14T16:08:18.573" LastActivityDate="2018-01-14T16:08:18.573" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26611" PostTypeId="2" ParentId="26608" CreationDate="2018-01-14T11:11:05.840" Score="3" Body="&lt;p&gt;It is just a matter of modelling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $w \cdot x = \sum_{i=1}^n w_i x_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example in linear regression (ignore intercept term for now, we can always insert the intercept term by including $1$ as a feature), we try to fit a model of the type of $y(x) = \sum_{i=1}^n w_ix_i$, however, the problem with this when the dependent variable is that $y$ can take values beyond $0$ and $1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, we consider another model $F(p) = \sum_{i=1}^n w_ix_i$ where $F(p)$ can take any real value and $F$ is a function which takes value from $(0,1)$ and be able to map it to $\mathbb{R}$. $F(p)$ can be chosen to be $\log \left( \frac{p}{1-p} \right)$, which is an increasing function which map $(0,1)$ to $\mathbb{R}$.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-01-14T11:11:05.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26613" PostTypeId="2" ParentId="26607" CreationDate="2018-01-14T11:33:47.667" Score="0" Body="&lt;p&gt;To get from a &lt;code&gt;string&lt;/code&gt; to &lt;strong&gt;one hot encoding&lt;/strong&gt; is basically a 2-step process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;sklearn&lt;/code&gt;'s &lt;code&gt;OneHotEncoder&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Encode[s] categorical &lt;strong&gt;integer&lt;/strong&gt; features using a one-hot aka one-of-K&#xA;  scheme. (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, one first has to convert &lt;code&gt;strings&lt;/code&gt; to &lt;code&gt;integers&lt;/code&gt;, which can be easily done using &lt;code&gt;sklearn&lt;/code&gt;'s &lt;code&gt;LabelEncoder&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Encode labels with value between 0 and n_classes-1. (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There several such examples on &lt;code&gt;StackOverflow&lt;/code&gt;, e.g. &lt;a href=&quot;https://stackoverflow.com/questions/43588679/issue-with-onehotencoder-for-categorical-features&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2968" LastActivityDate="2018-01-14T11:33:47.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26614" PostTypeId="2" ParentId="26608" CreationDate="2018-01-14T11:35:45.863" Score="2" Body="&lt;p&gt;Logistic regression is a statistical method for analyzing a dataset in which there are one or more independent variables that determine an outcome. The outcome is measured with a dichotomous variable &lt;a href=&quot;https://www.medcalc.org/manual/logistic_regression.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;in which there are only two possible outcomes&lt;/a&gt;. You can think like entropy in decision trees. If probability of situation is binary there is no relationship with outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function that is included to your post about theory. In an application you are going to use this term for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w \cdot x = \sum_{i=1}^n w_i x_i$ or $F(p) = \sum_{i=1}^n w_ix_i$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And it will give you a curve shape like half of Gaussian curve &lt;/p&gt;&#xA;" OwnerUserId="13696" LastEditorUserId="29575" LastEditDate="2018-01-14T14:52:46.747" LastActivityDate="2018-01-14T14:52:46.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26615" PostTypeId="1" AcceptedAnswerId="26762" CreationDate="2018-01-14T12:38:55.037" Score="2" ViewCount="6731" Body="&lt;p&gt;I am practicing to use sklearn for decision tree, and I am using the play tennis data set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Txb3U.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Txb3U.png&quot; alt=&quot;DataSet&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;play_ is the target column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;as per my pen and paper calculation of entropy and Information Gain, &lt;strong&gt;the root node should be outlook_ column&lt;/strong&gt; because it has the highest entropy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But somehow, my current decision tree has &lt;strong&gt;humidity as the root node&lt;/strong&gt;, and look likes this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IGS70.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IGS70.png&quot; alt=&quot;Decision Tree Current Scenario&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current code in python: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import train_test_split &#xA;from sklearn.tree import DecisionTreeClassifier &#xA;from sklearn.metrics import accuracy_score &#xA;from sklearn import tree &#xA;from sklearn.preprocessing import LabelEncoder&#xA;&#xA;import pandas as pd &#xA;import numpy as np &#xA;&#xA;df = pd.read_csv('playTennis.csv') &#xA;&#xA;lb = LabelEncoder() &#xA;df['outlook_'] = lb.fit_transform(df['outlook']) &#xA;df['temp_'] = lb.fit_transform(df['temp'] ) &#xA;df['humidity_'] = lb.fit_transform(df['humidity'] ) &#xA;df['windy_'] = lb.fit_transform(df['windy'] )   &#xA;df['play_'] = lb.fit_transform(df['play'] ) &#xA;X = df.iloc[:,5:9] &#xA;Y = df.iloc[:,9]&#xA;&#xA;X_train, X_test , y_train,y_test = train_test_split(X, Y, test_size = 0.3, random_state = 100) &#xA;&#xA;clf_entropy = DecisionTreeClassifier(criterion='entropy')&#xA;clf_entropy.fit(X_train.astype(int),y_train.astype(int)) &#xA;y_pred_en = clf_entropy.predict(X_test)&#xA;&#xA;print(&quot;Accuracy is :{0}&quot;.format(accuracy_score(y_test.astype(int),y_pred_en) * 100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44607" LastEditorUserId="29575" LastEditDate="2018-01-14T14:48:38.490" LastActivityDate="2018-01-17T20:54:12.460" Title="Decision Tree : PlayTennis Data Set" Tags="|python|scikit-learn|pandas|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26617" PostTypeId="2" ParentId="26607" CreationDate="2018-01-14T12:57:33.580" Score="1" Body="&lt;p&gt;Consider the following demo:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source DF:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [53]: df&#xA;Out[53]:&#xA;   A          B                   C&#xA;0  1  Bachelors  Married-civ-spouse&#xA;1  2    HS-grad            Divorsed&#xA;2  3       11th  Married-civ-spouse&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [54]: for c in df.columns[df.dtypes.eq('object')]:&#xA;    ...:     df = df.join(df[c].str.get_dummies())&#xA;    ...:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [55]: df&#xA;Out[55]:&#xA;   A          B                   C  11th  Bachelors  HS-grad  Divorsed  Married-civ-spouse&#xA;0  1  Bachelors  Married-civ-spouse     0          1        0         0                   1&#xA;1  2    HS-grad            Divorsed     0          0        1         1                   0&#xA;2  3       11th  Married-civ-spouse     1          0        0         0                   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [58]: df = df.join(pd.get_dummies(df.select_dtypes(['object'])))&#xA;&#xA;In [59]: df&#xA;Out[59]:&#xA;   A          B                   C  B_11th  B_Bachelors  B_HS-grad  C_Divorsed  C_Married-civ-spouse&#xA;0  1  Bachelors  Married-civ-spouse       0            1          0           0                     1&#xA;1  2    HS-grad            Divorsed       0            0          1           1                     0&#xA;2  3       11th  Married-civ-spouse       1            0          0           0                     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2018-01-14T13:06:24.237" LastActivityDate="2018-01-14T13:06:24.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26618" PostTypeId="1" AcceptedAnswerId="26621" CreationDate="2018-01-14T13:50:32.237" Score="5" ViewCount="4775" Body="&lt;p&gt;I've been struggling with Neural Networks for a while now.&#xA;I get the math behind backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still as reference I'm using the formulas from &lt;a href=&quot;https://www.youtube.com/watch?v=zpykfC4VnpM&amp;amp;list=WL&amp;amp;index=11&amp;amp;t=635s&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/giZA8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/giZA8.png&quot; alt=&quot;Screenshot of the formulas&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;The Network learns XOR:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Prediction After Training: [0.0003508415406266712] Expected: [0.0]&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;But basically doesn't learn anything on the MNIST dataset:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Error after n trainings examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; - 0      Total Net Error:  4.3739634316135225 &#xA; - 10000  Total Net Error:  0.4876292680858326&#xA; - 20000  Total Net Error:  0.39989816082272944&#xA; - 30000  Total Net Error:  0.49507443066631834&#xA; - 40000  Total Net Error:  0.5483594859079792&#xA; - 50000  Total Net Error:  0.5135921029479789&#xA; - 59000  Total Net Error:  0.4686434346776871&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[Prediction] [Expected]&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; - [0.047784337754445516]               [0]&#xA; - [0.09444684951344406]                [0]&#xA; - [0.0902378720783441]                 [0]&#xA; - [0.09704810171673675]                [0]&#xA; - [0.02940947956812051]                [0]&#xA; - [0.12494839048272757]                [1]&#xA; - [0.1512762065177885]                 [0]&#xA; - [0.055847446615593155]               [0]&#xA; - [0.22983410239796548]                [0]&#xA; - [0.09162426430286492]                [0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;The Old Code ( can be ignored )&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I've broken the network down as much as possible. With no matrix or vector multiplication. Here the code for the different classes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Main File:    &#xA;&#xA;    # Load Trainigs Data&#xA;    rawImages, rawLabels, numImagePixels = get_data_and_labels(&quot;C:\\Users\\Robin\\Documents\\MNIST\\Images\\train-images.idx3-ubyte&quot;, &quot;C:\\Users\\Robin\\Documents\\MNIST\\Labels\\train-labels.idx1-ubyte&quot;)&#xA;&#xA;    # Prepare Data&#xA;    print(&quot;Start Preparing Data&quot;)&#xA;    images = []&#xA;    labels = []&#xA;    for i in rawImages:&#xA;        insert = []&#xA;        for pixel in i:&#xA;            insert.append(map0to1(pixel, 255))&#xA;        images.append(insert)&#xA;    for l in rawLabels:&#xA;        y = [0] * 10&#xA;        y[l] = 1&#xA;        labels.append(y)&#xA;    print(&quot;Finished Preparing Data&quot;)&#xA;&#xA;&#xA;    # Create Network&#xA;    mnistNet = Network((numImagePixels, 16, 16, 10))&#xA;&#xA;    # Train&#xA;    print(&quot;Start Training&quot;)&#xA;    for index in range(len(images)):&#xA;        netError = mnistNet.train(images[index], labels[index])&#xA;        if index % 10000 == 0:&#xA;            print(index, &quot; Total Net Error: &quot;, netError)&#xA;&#xA;&#xA;    prediction = mnistNet.predict(images[0])&#xA;    print(&quot;After Training The Network Predicted:&quot;, prediction, &quot;Expected Was:&quot;, labels[0])&#xA;&#xA;&#xA;&#xA;class Network:&#xA;&#xA;    def __init__(self, topology):&#xA;        # Make Layer List&#xA;        self.layerList = []&#xA;        # Make Input Layer&#xA;        self.layerList.append(Layer(0, topology[0], 0))&#xA;        # Make All Other Layers&#xA;        for index in range(1, len(topology)):&#xA;            self.layerList.append(Layer(index, topology[index], topology[index-1]))&#xA;&#xA;    def predict(self, x):&#xA;        # Set x As Value Of Input Layer&#xA;        self.layerList[0].setInput(x)&#xA;        # Feed Through Network&#xA;        for index in range(1, len(self.layerList)):&#xA;            self.layerList[index].feedForward(self.layerList[index-1].getA())&#xA;        # Return The Output Of The Last Layer&#xA;        return self.layerList[-1].getA()&#xA;&#xA;&#xA;    def train(self, x, y):&#xA;        # Feed Through Network&#xA;        prediction = self.predict(x)&#xA;        # Container For The Calculated Layer Errors&#xA;        errorsPerLayer = []&#xA;        # Calculate Error Of The Output Layer&#xA;        errorOutputLayer = listSubtract(prediction,y)&#xA;        # Add The Error To The Container&#xA;        errorsPerLayer.append(errorOutputLayer)&#xA;        # Calculate The Total Error Of The Network&#xA;        totalError = calcTotalError(errorOutputLayer)&#xA;        # Calculate The Error Of The Hidden Layers&#xA;        for layerNum in range(len(self.layerList)-2, 0, -1):&#xA;            # Get The Error Of The Next Layer&#xA;            errorOfNextLayer = errorsPerLayer[0]&#xA;            # Forward The Calculation To The Next Layer, Which Returns The Weighted Error, By Giving It It's Error&#xA;            weightedError = self.layerList[layerNum+1].calculateWeightedError(self.layerList[layerNum].getNeuronNum(), errorOfNextLayer)&#xA;            # Forward The Calculation To The Current Layer, Which Returns The Error Of The Layer, By Giving It The Number Of Neurons In The Current Layer And The Weighted Error Of The Next Layer&#xA;            currentLayerError = self.layerList[layerNum].calculateError(weightedError)&#xA;            # Add The Just Calculated Error To The List&#xA;            errorsPerLayer.insert(0, currentLayerError)&#xA;        # Insert 0 As Error For The Input Layer, It's Not Important But That Way It's Size Matches The One Of The Layer List&#xA;        errorsPerLayer.insert(0, 0)&#xA;        # Update Weights And Biases&#xA;        for layerNum in range(1, len(self.layerList)):&#xA;            # Get The Output Of The Previous Layer&#xA;            aOfPrevLayer = self.layerList[layerNum-1].getA()&#xA;            # Forward The Error Of The Current Layer And The Output Of The Previous Layer To The Current Layer For Calculating Delta W&#xA;            self.layerList[layerNum].updateWeightsAndBiases(errorsPerLayer[layerNum], aOfPrevLayer)&#xA;&#xA;        #print(&quot;The Network Predicted: &quot;, prediction, &quot; Expected Was: &quot;, y, &quot; The Error Of The Output Layer Is: &quot;, errorOutputLayer)&#xA;        # Return The Total Error Of The Network For Usage Outisde This Class&#xA;        return totalError&#xA;&#xA;&#xA;&#xA;    def getNetworkInfo(self):&#xA;        for layer in self.layerList:&#xA;            print(layer.getLayerInfo())&#xA;&#xA;&#xA;----------&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;class Layer:&#xA;&#xA;    def __init__(self, layerNum, numNeurons, numNeuronsPrevLayer):&#xA;        self.neurons = []&#xA;        # Set The Number Of The Layer&#xA;        self.layerNum = layerNum&#xA;        # Create The Neurons&#xA;        for index in range(numNeurons):&#xA;            self.neurons.append(Neuron(numNeuronsPrevLayer))&#xA;        # Print Info&#xA;        print(&quot;Layer &quot;, layerNum, &quot; makes &quot;, numNeurons, &quot; Neurons&quot;, len(self.neurons))&#xA;&#xA;&#xA;    def feedForward(self, aPrevLayer):&#xA;        # Give It To The Neurono For Processing&#xA;        for neuron in self.neurons:&#xA;            neuron.feedForward(aPrevLayer)&#xA;&#xA;&#xA;    def calculateWeightedError(self, numNeuronsCurrentLayer, errorOfNextLayer):&#xA;        # The Container For The Weighted Error Of The Next Layer&#xA;        weightedError = []&#xA;        # The Calulation For Every Neuron Of The Current Layer One After Another&#xA;        for neuronNum in range(numNeuronsCurrentLayer):&#xA;            eSum = 0&#xA;            # The Error Of The Neuron With The Neuron For Later Calculation&#xA;            for e, n in zip(errorOfNextLayer, self.neurons):&#xA;                # Forward The Calculation To The Current Neuron, By Giving It It's Error And The Connecting Neuron Num&#xA;                eSum += n.weightError(e, neuronNum)&#xA;            # Add The Summed And Weighted Error&#xA;            weightedError.append(eSum)&#xA;&#xA;        # Return The Error Of The Current Layer&#xA;        return weightedError&#xA;&#xA;&#xA;    def calculateError(self, weightedError):&#xA;        # The Container For The Error Of The Current Layer&#xA;        errorOfCurrentLayer = []&#xA;        # The Weighted Error For The Neuron With The Neruon&#xA;        for wE, n in zip(weightedError, self.neurons):&#xA;            # Add The Product Of The Weighted Error With The Z Of The Current Neuron Run To Sigmoid Prime&#xA;            errorOfCurrentLayer.append(wE * sigmoidPrime(n.getZ()))&#xA;        # Return The Error Of The Current Layer&#xA;        return errorOfCurrentLayer&#xA;&#xA;&#xA;    def updateWeightsAndBiases(self, errorOfCurrentLayer, aOfPrevLayer):&#xA;        # The Error For The Neuron With The Neuron&#xA;        for e, n in zip(errorOfCurrentLayer, self.neurons):&#xA;            # Error Of Current Layer Is Equal To The Delta Of The Bias So Apply That&#xA;            n.updateBias(e)&#xA;            # Forward The Error And All The Activity Of The Previous Layer To The Current Neuron To Update It's Weights&#xA;            n.updateWeights(e, aOfPrevLayer)&#xA;&#xA;    def setInput(self, x):&#xA;        # Set It To Every Neuron&#xA;        for neuron, val in zip(self.neurons, x):&#xA;            neuron.setInput(val)&#xA;&#xA;    def getA(self):&#xA;        aOfLayer = []&#xA;        for neuron in self.neurons:&#xA;            aOfLayer.append(neuron.getA())&#xA;        return aOfLayer&#xA;    def getNeuronNum(self):&#xA;        return len(self.neurons)&#xA;&#xA;    def getLayerInfo(self):&#xA;        return &quot;Layer( %i ), has %i Neurons&quot; % (self.layerNum, len(self.neurons))&#xA;&#xA;&#xA;----------&#xA;&#xA;&#xA;&#xA;class Neuron:&#xA;&#xA;&#xA;    def __init__(self, numNeuronsPrevLayer):&#xA;        self.a = 0&#xA;        self.z = 0&#xA;        self.b = 0.5&#xA;        if numNeuronsPrevLayer != 0:&#xA;            self.w = np.random.uniform(low = 0, high = 0.5, size=(numNeuronsPrevLayer,))&#xA;&#xA;    def feedForward(self, aPrevLayer):&#xA;        # Reset Z&#xA;        self.z = 0&#xA;        # Calculate Z&#xA;        for w, a in zip(self.w, aPrevLayer):&#xA;            self.z += w*a&#xA;        # Add Bias&#xA;        self.z += self.b&#xA;        # Calculate A&#xA;        self.a = sigmoid(self.z)&#xA;&#xA;    def weightError(self, e, neuronNum):&#xA;        # Weight Error With The Connecting Weight&#xA;        return e * self.w[neuronNum]&#xA;&#xA;    def updateWeights(self, e, aOfPrevLayer):&#xA;        # The Weight With The Matching Activity Of The Previous Layer&#xA;        for index in range(len(self.w)):&#xA;            # The Delta Of The Weight Is The Error Of That Neuron Mutliplied With The Through The Weight Connected Activiy Of The Previous Layer&#xA;            self.w[index] -= e * aOfPrevLayer[index]&#xA;&#xA;    def updateBias(self, e):&#xA;        # E Is The Delta Of The Bias1&#xA;        self.b -= e&#xA;&#xA;    def setInput(self, x):&#xA;        self.z = x&#xA;        self.a = x&#xA;&#xA;    def getA(self):&#xA;        return self.a&#xA;    def getZ(self):&#xA;        return self.z&#xA;    def getB(self):&#xA;        return self.b&#xA;    def getW(self):&#xA;        return self.w&#xA;&#xA;&#xA;----------&#xA;&#xA;**helper functions**&#xA;&#xA;    def map0to1(val, valMax):&#xA;        return val/valMax&#xA;&#xA;&#xA;&#xA;    def calcTotalError(errorOutputLayer):&#xA;        totalError = 0&#xA;        for e in errorOutputLayer:&#xA;            totalError += e**2&#xA;        totalError *= 0.5&#xA;        return totalError&#xA;&#xA;&#xA;&#xA;    def listSubtract(list1, list2):&#xA;        subbed = []&#xA;        for l1, l2 in zip(list1, list2):&#xA;            subbed.append(l1-l2)&#xA;        return subbed&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;The New Code&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Main File&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import random&#xA;from network import *&#xA;from mnistreader import *&#xA;&#xA;&#xA;def map0to1(val, valMax):&#xA;    return val/valMax&#xA;&#xA;&#xA;# THIS WORKS&#xA;'''&#xA;trainX = [ [0.0,0.0], [1.0,0.0], [0.0,1.0], [1.0,1.0] ]&#xA;trainY = [ [ 0.0], [ 1.0 ], [ 1.0 ], [ 0.0 ] ]&#xA;&#xA;&#xA;# Create Network&#xA;xorNet = Network((2,2,1))&#xA;&#xA;# Train&#xA;for index in range(100000):&#xA;    randIndex = random.randint(0, 3)&#xA;    xorNet.train(trainX[randIndex], trainY[randIndex])&#xA;&#xA;print(&quot;Prediction After Training:&quot;, xorNet.predict(trainX[0]), &quot;Expected:&quot;, trainY[0])&#xA;print(&quot;Prediction After Training:&quot;, xorNet.predict(trainX[1]), &quot;Expected:&quot;, trainY[1])&#xA;print(&quot;Prediction After Training:&quot;, xorNet.predict(trainX[2]), &quot;Expected:&quot;, trainY[2])&#xA;print(&quot;Prediction After Training:&quot;, xorNet.predict(trainX[3]), &quot;Expected:&quot;, trainY[3])&#xA;'''&#xA;&#xA;mnistNet = Network((784, 30, 10))&#xA;&#xA;# Load Trainigs Data&#xA;rawImages, rawLabels, numImagePixels = get_data_and_labels(&quot;C:\\Users\\Robin\\Documents\\MNIST\\Images\\train-images.idx3-ubyte&quot;, &quot;C:\\Users\\Robin\\Documents\\MNIST\\Labels\\train-labels.idx1-ubyte&quot;)&#xA;&#xA;# Prepare Data&#xA;print(&quot;Start Preparing Data&quot;)&#xA;images = []&#xA;labels = []&#xA;for i in rawImages:&#xA;    insert = []&#xA;    for pixel in i:&#xA;        insert.append(map0to1(pixel, 255))&#xA;    images.append(insert)&#xA;for l in rawLabels:&#xA;    y = [0] * 10&#xA;    y[l] = 1&#xA;    labels.append(y)&#xA;print(&quot;Finished Preparing Data&quot;)&#xA;&#xA;&#xA;# Define Variables&#xA;learningRate = 0.0001&#xA;error = 10&#xA;&#xA;# Training&#xA;while error &amp;gt; 0.1:&#xA;    for tNum in range(len(images)):&#xA;        error = mnistNet.train(images[tNum], labels[tNum], learningRate)&#xA;    print(&quot;Error:&quot;, error, &quot;\n Prediction:\n&quot;, mnistNet.predict(images[1]), &quot;\nExpected:&quot;, rawLabels[1], &quot;\n\n&quot;)&#xA;&#xA;# Test Prediction&#xA;print(&quot;For&quot;, rawLabels[1], &quot;Predicted\n&quot;, mnistNet.predict(images[1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Network Class&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;from layer import *&#xA;from transferfunction import *&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;class Network:&#xA;&#xA;&#xA;&#xA;    def __init__(self, shape):&#xA;        # Save The Shape Of The Nework&#xA;        self.shape = shape&#xA;        # Create A List Of Layers&#xA;        self.layers = []&#xA;        # Create Input Layer&#xA;        self.layers.append(Layer((shape[0],), layerType = 'Input'))&#xA;        # Create Hidden Layers&#xA;        for numNeurons, numNeuronsPrevLayer in zip(shape[1:], shape[:-2]):&#xA;            self.layers.append(Layer((numNeurons, numNeuronsPrevLayer), layerType = 'Hidden'))&#xA;        # Create Output Layer&#xA;        self.layers.append(Layer((shape[-1], shape[-2]), layerType = 'Output'))&#xA;&#xA;&#xA;&#xA;&#xA;    def predict(self, x):&#xA;        # X Is A Row So Shape It To Be A Column&#xA;        x = np.array(x).reshape(-1, 1)&#xA;        # Set X To Be The Ouput Of The Input Layer&#xA;        self.layers[0].setOutput(x)&#xA;        # Feed Through Other Layers&#xA;        for layerNum in range(1,len(self.layers)):&#xA;            self.layers[layerNum].feedForward(self.layers[layerNum-1].getOutput())&#xA;        # Return The Output Of The Output Layer&#xA;        return self.layers[-1].getOutput()&#xA;&#xA;&#xA;    def train(self, x, y, learningRate):&#xA;        '''&#xA;        1. Feed Forward&#xA;        2. Calculate Error&#xA;        3. Calulate Deltas&#xA;        4. Apply Deltas&#xA;&#xA;        Error Output Layer = f'(z) * (prediction - target)&#xA;        Error Hidden Layer = f'(z) * ( transposed weights next layer DOT error next layer )&#xA;        Delta Bias         = learning rate * error&#xA;        Delta Weights      = learning rate * ( error DOT transposed activity previous layer )&#xA;&#xA;        '''&#xA;&#xA;&#xA;        # Feed Through Network&#xA;        prediction = self.predict(x)&#xA;        # Y Is A Row So Shape It To Be A Column&#xA;        y = np.array(y).reshape(-1, 1)&#xA;&#xA;        # Calculate Error&#xA;        error = prediction - y&#xA;        # Calculate Total Error&#xA;        totalError = 0.5 * np.sum(error**2)&#xA;&#xA;        # Create Container For The Deltas&#xA;        deltas = []&#xA;        # Calculate Delta For Output Layer&#xA;        deltas.append( np.multiply(sigmoidPrime(self.layers[-1].getZ()), error) )&#xA;        # Calculate Deltas For Every Hidden Layer&#xA;        for layerNum in range(len(self.layers)-2, 0, -1):&#xA;            # Compute The Weighted Error Of The Next Layer&#xA;            weightedErrorOfNextLayer = np.dot(self.layers[layerNum+1].getW().T, deltas[0])&#xA;            # Compute The Delta And Add It In Front&#xA;            deltas.insert(0, np.multiply(sigmoidPrime(self.layers[layerNum].getZ()), weightedErrorOfNextLayer) )&#xA;        # Insert Placeholder To Make Delta List As Big As The Layer List&#xA;        deltas.insert(0, 0)&#xA;&#xA;        # For Numerical Grandient Checking Do&#xA;        numericalGradients = self.performNumericalGradientChecking(x,y)&#xA;&#xA;        # Update Weights And Biases&#xA;        for layerNum in range(1, len(self.layers)):&#xA;            self.layers[layerNum].updateBias(deltas[layerNum], learningRate)&#xA;            self.layers[layerNum].updateWeight(deltas[layerNum], self.layers[layerNum-1].getOutput(), learningRate, numericalGradients[layerNum-1])&#xA;&#xA;&#xA;        # Show Information&#xA;        #print('Network Predicted: \n', prediction, '\nTarget:\n', y, '\nError: ', totalError)&#xA;        return totalError&#xA;&#xA;&#xA;    def performNumericalGradientChecking(self, x, y):&#xA;        # Container Saving All Current Weight Values&#xA;        weightSave = []&#xA;        # Save All The Weight Values&#xA;        for layerNum in range(1, len(self.layers)):&#xA;            weightSave.append(self.layers[layerNum].getW())&#xA;&#xA;        # Define Epsilon To Be A Small Number&#xA;        epsilon = 1e-4&#xA;&#xA;        # Gradient Container&#xA;        numericalGradients = []&#xA;&#xA;        #Perform The Check For Every Layer Therfore Every Set Of Weights&#xA;        for layerNum in range(1, len(self.layers)):&#xA;&#xA;            # Feed Forward With Changed Weights(+epsilon), And Compute Cost&#xA;            self.layers[layerNum].setW(weightSave[layerNum-1] + epsilon)&#xA;            prediction = self.predict(x)&#xA;            loss2 = 0.5 * np.sum ((prediction - y)**2)&#xA;&#xA;            # Feed Forward With Changed Weights(-epsilon), And Compute Cost&#xA;            self.layers[layerNum].setW(weightSave[layerNum-1] - epsilon)&#xA;            prediction = self.predict(x)&#xA;            loss1 = 0.5 * np.sum ((prediction - y)**2)&#xA;&#xA;            # Reset Weight&#xA;            self.layers[layerNum].setW(weightSave[layerNum-1])&#xA;&#xA;            # Calculate Numerical Loss&#xA;            numericalGradient = (loss2 - loss1) / (2 * epsilon)&#xA;&#xA;            # Add The Numerical Grandient&#xA;            numericalGradients.append(numericalGradient)&#xA;&#xA;        return numericalGradients&#xA;&#xA;    def __str__(self):&#xA;        strBuff = ''&#xA;        for layer in self.layers:&#xA;            strBuff += layer.getInfo()&#xA;        return strBuff&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Layer Class&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;from transferfunction import *&#xA;&#xA;&#xA;class Layer:&#xA;&#xA;&#xA;&#xA;    def __init__(self, neurons, layerType = 'Hidden'):&#xA;        ''' Neurons Is A Tuple Consisting Of [0]=NumNeurons And [1]=numNeuronsPrevLayer '''&#xA;&#xA;        # Remember The Type Of The Layer&#xA;        self.layerType = layerType&#xA;        # Remember How Many Neurons This Layer Has&#xA;        self.neuronCount = neurons[0]&#xA;        # Create Layer Based On Type&#xA;        if layerType.lower() == 'input':&#xA;            # Create Container For Input Data&#xA;            self.a = []&#xA;        elif layerType.lower() == 'hidden' or layerType.lower() == 'output':&#xA;            # Create Container For Activation&#xA;            self.z = []&#xA;            # Create Container For Neurons Input&#xA;            self.a = []&#xA;            # Create Weights&#xA;            self.w = np.random.uniform(low = 0.0, high = 0.4, size=(neurons[0], neurons[1]))&#xA;            # Create Prev Delta Weight For Momentum&#xA;            self.momentum = 0.3&#xA;            self.prevDelta = np.full((neurons[0], neurons[1]), 0)&#xA;            # Create Bias&#xA;            self.b = np.full((neurons[0],1), 0, dtype=float)&#xA;        else:&#xA;            print('Wrong Type Of Layer Specified')&#xA;&#xA;&#xA;    def feedForward(self, aPrevLayer):&#xA;        self.z = np.dot(self.w, aPrevLayer) + self.b&#xA;        self.a = sigmoid(self.z)&#xA;&#xA;    def updateBias(self, e, learningRate):&#xA;        self.b -= learningRate * e&#xA;    def updateWeight(self, e, aPrevLayer, learningRate, numericalGradient):&#xA;        # Calulate The Delta Of The Weights&#xA;        deltaW = np.dot(e, aPrevLayer.T)&#xA;&#xA;        # Compare DeltaW With The Numerical Grandient&#xA;        check = np.linalg.norm(deltaW - numericalGradient) / np.linalg.norm(deltaW + numericalGradient)&#xA;        # DEBUG&#xA;        print(check)&#xA;&#xA;        # The Weight Change Is The Delta With The Addition Of The Momentum&#xA;        self.w -= ( learningRate * deltaW ) + ( self.momentum * self.prevDelta)&#xA;        # Save The Current DeltaW&#xA;        self.prevDelta = deltaW&#xA;&#xA;&#xA;&#xA;&#xA;&#xA;    def getW(self):&#xA;        return self.w&#xA;    def getZ(self):&#xA;        return self.z&#xA;    def getOutput(self):&#xA;        return self.a&#xA;&#xA;&#xA;    def setOutput(self, x):&#xA;        self.a = x&#xA;    def setW(self, w):&#xA;        self.w = w&#xA;&#xA;&#xA;    def getInfo(self):&#xA;        if self.layerType.lower() == 'input':&#xA;            return 'Input Layer With ' + str(self.neuronCount) + ' Neurons\n'&#xA;        else:&#xA;            return self.layerType + ' Layer With ' + str(self.neuronCount) + ' Neurons And Weights Of Shape: ' + str(self.w.shape) + ' With Biases Of Shape: ' + str(self.b.shape) + '\n'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So my Question simply is: &quot;What's wrong?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problems:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Error gets stuck at 0.45&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;When using a hidden layer with 800 neurons I get the warning division by zero and all becomes NaN's from sigmoid prime&lt;/li&gt;&#xA;&lt;li&gt;Numerical gradient checking for the hidden Layer is: 1.0 and for the output layer: 0.995784895209. I know that is supposed to be a very small number. But on the second trainings example it creates a overflow error and becomes NaN's&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Major Edit&lt;/strong&gt;&#xA;I'm truly grateful for all suggestions so far, I've updated the question using a vectorized form so it's easier to get an overview of what I'm doing here. &#xA;I tried gradient checking now too, not sure if I implemented it right (used the tutorial by Welch Labs (&lt;a href=&quot;https://youtu.be/pHMzNW8Agq4&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://youtu.be/pHMzNW8Agq4&lt;/a&gt;))&#xA;I hope the Code is readable &lt;/p&gt;&#xA;" OwnerUserId="44610" LastEditorUserId="44610" LastEditDate="2018-01-26T13:50:22.230" LastActivityDate="2018-02-01T01:13:21.237" Title="Neural Network Performs Bad On MNIST" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="26619" PostTypeId="1" CreationDate="2018-01-14T18:01:12.293" Score="0" ViewCount="189" Body="&lt;p&gt;MNIST dataset is handwritten images which contains 60000 training and 10000 testing samples , can I use only a fraction of dataset for instance 10000 for training and 4000 for testing. I want to know that whether my results will be comparable to other claimed results. Like I used random forest classifier on 10k training and 4k testing and got accuracy of 93 %. would this result be equivalent when I use 60k training and 10k testing .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and in some published papers I read that they only used 40 images per class and evaluated their accuracy and performance using them does it conforms to the standards that use 60k as training and 10k as testing.&#xA;Or we can say that the choice of selection of training and testing data is dependent on the method used to classify&lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="38506" LastEditDate="2018-01-14T18:06:20.767" LastActivityDate="2018-01-15T03:56:31.550" Title="does accuracy and performance depends on whole dataset or fraction of that data mimics the whole?" Tags="|random-forest|image-classification|mnist|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26621" PostTypeId="2" ParentId="26618" CreationDate="2018-01-14T18:26:11.887" Score="10" Body="&lt;p&gt;Your network has 28 x 28 = 784 (normal MNIST size) inputs, 16 + 16 hidden nodes and 10 outputs. This is not enough for an enough accurate model as a result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://stackoverflow.com/questions/45447740/neural-network-mnist-backpropagation-is-correct-but-training-test-accuracy-ver/45448307#45448307&quot;&gt;question&lt;/a&gt; suggests to use 256 x 256 hidden nodes and &lt;a href=&quot;https://en.wikipedia.org/wiki/MNIST_database&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia page on MNIST&lt;/a&gt; gives for 2-layer reference the values: 784-800-10 meaning 800 x 10 nodes. Wikipedia gives error rate 0.016 for that 800 x 10 solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Sidenote: For 6-layer Deep Neural Network the numbers are : 784-2500-2000-1500-1000-500-10 so the new numbers I gave aren't that big. Of course the error rate on DNN is 0.0035 so it needs those layers.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do I have to change the learning rate which I now defined to be 0.3?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Answer in &lt;a href=&quot;https://stackoverflow.com/questions/45447740/neural-network-mnist-backpropagation-is-correct-but-training-test-accuracy-ver/45448307#45448307&quot;&gt;the referred question&lt;/a&gt; says 0.0001 is more appropriate value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit of question author&lt;/strong&gt;&#xA;I marked this as solved because now it is working. I implemented it again and after tweaking the hyperparameters it worked. So this is, in a way the solution to my problem. Because I asked why this problem occurs in the first place, my implementation error aside. &lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44610" LastEditDate="2018-02-01T01:13:21.237" LastActivityDate="2018-02-01T01:13:21.237" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="26622" PostTypeId="1" AcceptedAnswerId="26737" CreationDate="2018-01-14T19:15:45.697" Score="2" ViewCount="1064" Body="&lt;p&gt;I have collected the &lt;a href=&quot;https://github.com/tflearn/tflearn/blob/master/examples/images/dcgan.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;TFLearn DCGAN example&lt;/a&gt; code and put it into my local Jupyter environment. Furthermore, I have changed some comments and added &lt;code&gt;with tf.device('/gpu:0'):&lt;/code&gt; right before calling &lt;code&gt;gan.fit(...)&lt;/code&gt;, resulting in the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# coding: utf-8&#xA;&#xA;# In[1]:&#xA;&#xA;&#xA;get_ipython().magic('matplotlib inline')&#xA;from __future__ import division, print_function, absolute_import&#xA;&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;import tflearn&#xA;&#xA;&#xA;# In[2]:&#xA;&#xA;&#xA;# Data loading and preprocessing&#xA;import tflearn.datasets.mnist as mnist&#xA;X, Y, testX, testY = mnist.load_data()&#xA;X = np.reshape(X, newshape=[-1, 28, 28, 1])&#xA;&#xA;&#xA;# In[3]:&#xA;&#xA;&#xA;# Noise data input&#xA;z_dim = 200&#xA;total_samples = len(X)&#xA;&#xA;&#xA;# In[4]:&#xA;&#xA;&#xA;# Generator&#xA;def generator(x, reuse=False):&#xA;    with tf.variable_scope('Generator', reuse=reuse):&#xA;        x = tflearn.fully_connected(x, n_units=7 * 7 * 128)&#xA;        x = tflearn.batch_normalization(x)&#xA;        x = tf.nn.tanh(x)&#xA;        x = tf.reshape(x, shape=[-1, 7, 7, 128])&#xA;        x = tflearn.upsample_2d(x, 2)&#xA;        x = tflearn.conv_2d(x, 64, 5, activation='tanh')&#xA;        x = tflearn.upsample_2d(x, 2)&#xA;        x = tflearn.conv_2d(x, 1, 5, activation='sigmoid')&#xA;        return x&#xA;&#xA;&#xA;# In[5]:&#xA;&#xA;&#xA;# Discriminator&#xA;def discriminator(x, reuse=False):&#xA;    with tf.variable_scope('Discriminator', reuse=reuse):&#xA;        x = tflearn.conv_2d(x, 64, 5, activation='tanh')&#xA;        x = tflearn.avg_pool_2d(x, 2)&#xA;        x = tflearn.conv_2d(x, 128, 5, activation='tanh')&#xA;        x = tflearn.avg_pool_2d(x, 2)&#xA;        x = tflearn.fully_connected(x, 1028, activation='tanh')&#xA;        x = tflearn.fully_connected(x, 2)&#xA;        x = tf.nn.softmax(x)&#xA;        return x&#xA;&#xA;&#xA;# In[6]:&#xA;&#xA;&#xA;# Input data&#xA;gen_input = tflearn.input_data(shape=[None, z_dim], name='input_gen_noise')&#xA;input_disc_noise = tflearn.input_data(shape=[None, z_dim], name='input_disc_noise')&#xA;input_disc_real = tflearn.input_data(shape=[None, 28, 28, 1], name='input_disc_real')&#xA;&#xA;&#xA;# In[7]:&#xA;&#xA;&#xA;# Build discriminator&#xA;disc_fake = discriminator(generator(input_disc_noise))&#xA;disc_real = discriminator(input_disc_real, reuse=True)&#xA;disc_net = tf.concat([disc_fake, disc_real], axis=0)&#xA;&#xA;&#xA;# In[8]:&#xA;&#xA;&#xA;# Build stacked Generator/Discriminator&#xA;gen_net = generator(gen_input, reuse=True)&#xA;stacked_gan_net = discriminator(gen_net, reuse=True)&#xA;&#xA;&#xA;# In[9]:&#xA;&#xA;&#xA;# Build training ops for Discriminator&#xA;# Each network optimization should only update its own variable, thus we need&#xA;# to retrieve each network variable (with get_layer_variables_by_name)&#xA;disc_vars = tflearn.get_layer_variables_by_name('Discriminator')&#xA;# We need 2 target placeholders, for both the real and fake image target&#xA;disc_target = tflearn.multi_target_data(['target_disc_fake', 'target_disc_real'],&#xA;                                        shape=[None, 2])&#xA;disc_model = tflearn.regression(disc_net, optimizer='adam',&#xA;                                placeholder=disc_target,&#xA;                                loss='categorical_crossentropy',&#xA;                                trainable_vars=disc_vars,&#xA;                                batch_size=64, name='target_disc',&#xA;                                op_name='DISC')&#xA;&#xA;&#xA;# In[10]:&#xA;&#xA;&#xA;# Build training ops for Generator&#xA;gen_vars = tflearn.get_layer_variables_by_name('Generator')&#xA;gan_model = tflearn.regression(stacked_gan_net, optimizer='adam',&#xA;                               loss='categorical_crossentropy',&#xA;                               trainable_vars=gen_vars,&#xA;                               batch_size=64, name='target_gen',&#xA;                               op_name='GEN')&#xA;&#xA;&#xA;# In[11]:&#xA;&#xA;&#xA;# Define GAN model, that outputs the generated images&#xA;gan = tflearn.DNN(gan_model, tensorboard_verbose=3)&#xA;&#xA;&#xA;# In[12]:&#xA;&#xA;&#xA;# Training&#xA;# Prepare input data to feed to the discriminator&#xA;disc_noise = np.random.uniform(-1., 1., size=[total_samples, z_dim])&#xA;# Prepare target data to feed to the discriminator (0: fake image, 1: real image)&#xA;y_disc_fake = np.zeros(shape=[total_samples])&#xA;y_disc_real = np.ones(shape=[total_samples])&#xA;y_disc_fake = tflearn.data_utils.to_categorical(y_disc_fake, 2)&#xA;y_disc_real = tflearn.data_utils.to_categorical(y_disc_real, 2)&#xA;&#xA;&#xA;# In[13]:&#xA;&#xA;&#xA;# Prepare input data to feed to the stacked Generator/Discriminator&#xA;gen_noise = np.random.uniform(-1., 1., size=[total_samples, z_dim])&#xA;# Prepare target data to feed to the Discriminator&#xA;# The Generator tries to fool the Discriminator, thus target is 1 (real images)&#xA;y_gen = np.ones(shape=[total_samples])&#xA;y_gen = tflearn.data_utils.to_categorical(y_gen, 2)&#xA;&#xA;&#xA;# In[14]:&#xA;&#xA;&#xA;# Start training, feed both noise and real images&#xA;with tf.device('/gpu:0'):&#xA;    gan.fit(X_inputs={'input_gen_noise': gen_noise,&#xA;                      'input_disc_noise': disc_noise,&#xA;                      'input_disc_real': X},&#xA;            Y_targets={'target_gen': y_gen,&#xA;                       'target_disc_fake': y_disc_fake,&#xA;                       'target_disc_real': y_disc_real},&#xA;            n_epoch=10)&#xA;&#xA;&#xA;# In[15]:&#xA;&#xA;&#xA;# Create another model from the Generator graph to generate some samples&#xA;# for testing (re-using the same session to re-use the weights learnt)&#xA;gen = tflearn.DNN(gen_net, session=gan.session)&#xA;&#xA;&#xA;# In[16]:&#xA;&#xA;&#xA;f, a = plt.subplots(4, 10, figsize=(10, 4))&#xA;for i in range(10):&#xA;    # Noise input&#xA;    z = np.random.uniform(-1., 1., size=[4, z_dim])&#xA;    g = np.array(gen.predict({'input_gen_noise': z}))&#xA;    for j in range(4):&#xA;        # Generate image from noise. Extend to 3 channels for matplot figure.&#xA;        img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),&#xA;                         newshape=(28, 28, 3))&#xA;        a[j][i].imshow(img)&#xA;&#xA;f.show()&#xA;plt.draw()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to run this code on my NVIDIA GPU. I already have CUDA and cuDNN installed on my machine. Upon examining Windows Task Manager during training, I see that my CPU is stressed and my GPU lies dorment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RWZ1k.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RWZ1k.png&quot; alt=&quot;Windows Task Manager during training&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give advice on how to properly implement &lt;code&gt;with tf.device('/gpu:0'):&lt;/code&gt; as it's clear that the above code does not run on my NVIDIA GPU?&lt;/p&gt;&#xA;" OwnerUserId="44130" LastActivityDate="2018-01-17T09:42:46.727" Title="Why does TFLearn DCGAN not run on my GPU (on Windows)?" Tags="|machine-learning|python|tensorflow|gan|tflearn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26623" PostTypeId="1" AcceptedAnswerId="26626" CreationDate="2018-01-14T19:51:30.243" Score="4" ViewCount="1483" Body="&lt;p&gt;I would like to use the hour of the day (0-23) as a continuous feature, so the model will know that 12pm comes before 13:00, and that 8:00 is further from 21:00 than from 10:00. How do I engineer this feature so it will also understand that 0:00 comes after 23:00?&lt;/p&gt;&#xA;" OwnerUserId="43870" LastActivityDate="2018-01-19T08:13:44.563" Title="How to use hours of the day as a continuous feature?" Tags="|feature-selection|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26624" PostTypeId="1" AcceptedAnswerId="26636" CreationDate="2018-01-14T20:08:31.047" Score="4" ViewCount="645" Body="&lt;p&gt;I would like to compute parameters such as mean, variance, quantiles, etc. for a &lt;a href=&quot;https://en.wikipedia.org/wiki/Probability_density_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;PDF&lt;/a&gt; which is only given as a piece of code. That is, it can only be evaluated numerically at given points; no closed-form expression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, after using scikit-learn for kernel density estimation, I would like to compute parameters of the resulting PDF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel that popular libraries such as numpy/scipy, scikit-learn, or pandas probably provide such functionality. However, I couldn't find it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody please point me to it?&lt;/p&gt;&#xA;" OwnerUserId="29912" LastEditorUserId="24162" LastEditDate="2018-01-15T13:10:40.167" LastActivityDate="2018-01-15T13:10:40.167" Title="Compute parameters of a PDF (probability density function) for which no closed form expression is available" Tags="|scikit-learn|probability|scipy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26625" PostTypeId="1" AcceptedAnswerId="56106" CreationDate="2018-01-14T22:06:42.957" Score="3" ViewCount="15550" Body="&lt;p&gt;Is there a way I can access just the vocabulary list of pre-trained vectors for word2vec and GloVe? I do not need the entire n-dimensional embeddings.&lt;/p&gt;&#xA;" OwnerUserId="10799" LastActivityDate="2019-07-21T15:02:52.150" Title="Vocabulary List From word2vec and GloVe" Tags="|nlp|word2vec|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26626" PostTypeId="2" ParentId="26623" CreationDate="2018-01-14T22:43:17.447" Score="5" Body="&lt;p&gt;Great question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd say you should either use fourier transforms or sine / cosine transforms as they should get 0 to be right next to 23. The below post is talking about transforming day features, but should be easily applicable here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/126230/optimal-construction-of-day-feature-in-neural-networks&quot; title=&quot;Transforming day features&quot;&gt;Transforming day features&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/17759/encoding-features-like-month-and-hour-as-categorial-or-numeric&quot; title=&quot;Transforming hour features&quot;&gt;Transforming hour features&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21702" LastEditorUserId="21702" LastEditDate="2018-01-14T23:15:31.493" LastActivityDate="2018-01-14T23:15:31.493" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26627" PostTypeId="1" AcceptedAnswerId="26630" CreationDate="2018-01-15T01:09:38.067" Score="2" ViewCount="1992" Body="&lt;p&gt;I have a neural network model written in Keras with (8,5,5,5,32) neurons, as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Sequential&#xA;model = Sequential()&#xA;&#xA;# Neural network&#xA;model.add(Dense(5, input_dim=len(X[0]), activation='sigmoid' ))&#xA;model.add(Dense(5, activation='sigmoid' ))&#xA;model.add(Dense(5, activation='sigmoid' ))&#xA;model.add(Dense(len(y[0]), activation='sigmoid' ))&#xA;&#xA;&#xA;# Compile model&#xA;# sgd = optimizers.SGD(lr=0.01, decay=0.0, momentum=0.0, nesterov=False)&#xA;model.compile(loss='mean_squared_error', optimizer='adam', metrics=['acc'])&#xA;&#xA;# Fit model&#xA;history = model.fit(X, y, nb_epoch=200, validation_split=0.2, batch_size=30)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I access the weights after training, I get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.get_weights()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which returns a long array of arrays corresponding to the weights in between neurons of each layer. What I don't understand is a smaller array represents (same number of weights as number of neurons per layer) in between larger arrays (same number of sub-arrays as number of neurons per layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the biais that the neurons experience?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the results of 'model.get_weights()':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[array([[ 0.31680015,  0.22357693, -0.63079047, -0.04600599, -0.26949674],&#xA;        [ 0.0525099 , -0.41120723,  0.28259486, -0.37071031,  0.42028651],&#xA;        [ 0.35435981, -0.35501873, -0.05099263,  0.3633016 , -0.64845532],&#xA;        [ 0.60027206,  0.40594664,  0.29894602, -0.13255124,  0.52797431],&#xA;        [-0.32299024,  0.54219592,  0.34114835, -0.59672344, -0.47126439],&#xA;        [-0.51338726,  0.64451784, -0.35283062,  0.47248691, -0.31077194],&#xA;        [-0.59289241, -0.207461  ,  0.00371859, -0.52090681,  0.10946763],&#xA;        [-0.37216368, -0.23905358, -0.38580573,  0.0401655 , -0.34231418]], dtype=float32),&#xA; array([-0.03448975,  0.04991768,  0.11635038, -0.04274927,  0.0325128 ], dtype=float32),   # WHAT IS THIS ARRAY REPRESENTING?&#xA; array([[ 0.10943446, -0.24749899,  0.58269709,  0.54208171, -0.05888808],&#xA;        [ 0.02320727,  0.08465887, -0.79114383, -0.19608408,  0.55898732],&#xA;        [-0.81141329,  0.19124934, -0.69268334, -0.44021448,  0.72605485],&#xA;        [ 0.32895803,  0.08196118,  0.53820646,  0.6348688 , -0.06715827],&#xA;        [-0.0850288 ,  0.5077976 ,  0.36972848,  0.44874495,  0.36402631]], dtype=float32),&#xA; array([-0.30626452,  0.15301916, -0.17855364,  0.12410269,  0.2502442 ], dtype=float32),&#xA; array([[-0.31872895, -0.46534333, -0.4664084 , -0.23720025,  0.30465502],&#xA;        [-0.37690881, -0.00396255,  0.38115206,  1.20845091,  0.69348788],&#xA;        [ 0.15064301, -0.29923961,  0.13108611, -0.29579154, -0.34181508],&#xA;        [ 0.62893951,  0.49498206,  0.02549251,  0.6561147 , -0.52280194],&#xA;        [ 1.07029617,  0.66126752,  0.50944209,  0.58811921, -0.04030331]], dtype=float32),&#xA; array([ 0.57747394,  0.59574115,  0.68391681,  0.78335029,  0.26046163], dtype=float32),&#xA; array([[ -1.09961331e+00,  -7.06328213e-01,  -7.93772519e-01,&#xA;          -5.50112486e-01,  -5.05448937e-01,  -4.78618711e-01,&#xA;          -3.32313687e-01,  -5.46549559e-01,   3.88661295e-01,&#xA;           3.64094585e-01,  -1.93489313e-01,  -7.61669278e-02,&#xA;           1.23761639e-01,   4.93125141e-01,   4.78168607e-01,&#xA;          -7.07402304e-02,  -2.54306406e-01,  -2.37895012e-01,&#xA;          -1.36467636e-01,  -7.16407061e-01,   1.32701367e-01,&#xA;          -4.23079096e-02,  -4.71717492e-03,   2.56372184e-01,&#xA;           1.89603701e-01,   2.20276624e-01,  -2.55215704e-01,&#xA;          -1.04997739e-01,   2.81909049e-01,  -7.00806752e-02,&#xA;           2.99933195e-01,   3.84294897e-01],&#xA;        [ -9.60830688e-01,  -6.18087530e-01,  -8.46309483e-01,&#xA;          -6.05162561e-01,  -2.39057451e-01,   3.03133931e-02,&#xA;          -2.07459703e-01,  -6.84834659e-01,   3.47823203e-01,&#xA;          -6.44357502e-02,   2.55657077e-01,  -2.37801671e-01,&#xA;           2.57411227e-02,  -1.01771923e-02,  -6.76048512e-04,&#xA;           3.06296106e-02,   2.05646217e-01,   8.02281871e-02,&#xA;          -4.01538044e-01,  -5.49115181e-01,   3.00252885e-01,&#xA;           3.31445992e-01,  -1.75046876e-01,  -3.36513370e-01,&#xA;           1.65666446e-01,   1.74015135e-01,  -2.15066984e-01,&#xA;           3.79294544e-01,   1.67991996e-01,   2.39770293e-01,&#xA;          -5.49201332e-02,  -1.67401493e-01],&#xA;        [ -1.24525476e+00,  -5.97414970e-01,  -1.36500984e-01,&#xA;          -5.60880482e-01,  -4.26550537e-01,   1.46522000e-01,&#xA;          -6.26730978e-01,  -8.33723724e-01,  -2.20034972e-01,&#xA;           3.54697943e-01,   2.86612272e-01,   2.60758907e-01,&#xA;          -5.10771237e-02,   1.91444799e-01,   3.32518548e-01,&#xA;           1.51452944e-01,  -2.18744278e-01,  -2.07690187e-02,&#xA;           9.60563496e-02,  -2.26809219e-01,   8.80904198e-02,&#xA;           2.33646557e-01,   2.45599806e-01,   2.53560930e-01,&#xA;           1.55982673e-01,   6.49829209e-01,  -1.26019821e-01,&#xA;           5.47675073e-01,   3.28564644e-01,   8.67465809e-02,&#xA;          -1.40921310e-01,  -2.35581279e-01],&#xA;        [ -1.51756608e+00,  -1.11596704e+00,  -4.97624874e-01,&#xA;          -4.95555460e-01,  -4.83801186e-01,  -2.23367065e-01,&#xA;          -1.08115244e+00,  -9.68795598e-01,   4.32208836e-01,&#xA;           1.16957083e-01,  -1.02919623e-01,  -8.19303747e-03,&#xA;           4.21310306e-01,   1.09493546e-01,   1.54512182e-01,&#xA;          -1.46762207e-01,   1.58293337e-01,  -3.95552874e-01,&#xA;          -2.07770184e-01,  -1.90177709e-01,   2.07072627e-02,&#xA;           6.61122620e-01,   5.44478893e-01,  -1.46910429e-01,&#xA;           4.22070086e-01,   2.49319345e-01,   6.19665794e-02,&#xA;           9.74300727e-02,   3.37298632e-01,   2.90907085e-01,&#xA;           8.78930092e-02,   1.25872776e-01],&#xA;        [ -7.40014732e-01,  -7.02967405e-01,  -1.42469540e-01,&#xA;           1.66655079e-01,  -1.59682855e-01,  -2.07361296e-01,&#xA;          -9.04432237e-02,  -1.22986667e-01,  -3.28961462e-01,&#xA;           9.21192244e-02,  -2.13514805e-01,  -1.59033865e-01,&#xA;          -2.79709876e-01,  -1.64602488e-01,   1.96248814e-01,&#xA;          -1.98676869e-01,   2.80951142e-01,  -4.50290412e-01,&#xA;           2.34707281e-01,  -3.13370705e-01,  -2.24865358e-02,&#xA;           2.63352484e-01,   4.90205318e-01,   1.96813330e-01,&#xA;           4.13736820e-01,   7.11815134e-02,  -1.92510381e-01,&#xA;           5.77223562e-02,  -4.25750390e-02,  -3.79479416e-02,&#xA;           6.17611647e-01,   3.11740283e-02]], dtype=float32),&#xA; array([-1.66324592, -1.4556272 , -1.10267663, -0.52273899, -0.59953606,&#xA;        -0.52498704, -1.15079761, -1.19500864,  0.13858946, -0.01602219,&#xA;         0.02289196,  0.2923668 ,  0.14240141,  0.19787838,  0.15867165,&#xA;         0.24876554,  0.21616565, -0.35774839, -0.40798596, -0.71499836,&#xA;         0.36547631,  0.48771939,  0.40948448,  0.01851768,  0.56893039,&#xA;         0.93250728, -0.32663229,  0.34419572,  0.15536141,  0.21145843,&#xA;         0.53275597,  0.0364211 ], dtype=float32)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32511" LastActivityDate="2018-01-15T03:40:20.113" Title="How to define weights on Keras neural network" Tags="|python|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26629" PostTypeId="2" ParentId="26596" CreationDate="2018-01-15T03:00:07.153" Score="0" Body="&lt;p&gt;Yes, your data is not balanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before applying any kind of modeling it is suggestible to balance the data, you can use packages like &lt;strong&gt;SMOTE,ROSE&lt;/strong&gt; to overcome the imbalance in your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your next question, which index to use: It is subjective to your data. Please go through these links &lt;a href=&quot;https://www.quora.com/What-is-the-difference-between-information-gain-and-gini-index-When-should-I-apply-each-of-the-method&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;, &lt;a href=&quot;http://www.learnbymarketing.com/481/decision-tree-flavors-gini-info-gain/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-2&lt;/a&gt; for better understanding but suggestible to try both Entropy and Gini Impurity. There is a good chance to get similar trees and similar outcome. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gain Ratio works better to reduce bias towards highly branching features like time, salary etc. (Information about the features of the dataset isn’t mentioned)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any additional questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, how big is your test dataset?&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-01-15T09:58:36.483" LastActivityDate="2018-01-15T09:58:36.483" CommentCount="16" ContentLicense="CC BY-SA 3.0" />
  <row Id="26630" PostTypeId="2" ParentId="26627" CreationDate="2018-01-15T03:40:20.113" Score="1" Body="&lt;p&gt;You are correct! The array you see between those for your weights are the bias of the node in the starting layer. Notice how there are 5 elements in that array, this coincides with the 5 nodes in your first hidden layer. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-15T03:40:20.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26631" PostTypeId="1" CreationDate="2018-01-15T03:54:23.343" Score="1" ViewCount="271" Body="&lt;p&gt;From the paper &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Human-level control through deep reinforcement&#xA;  learning, Mnih et al. Nature 2015&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It says&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Reinforcement learning is known to be unstable or even to diverge&#xA;when a nonlinear function approximator such as a neural network is&#xA;used to represent the action-value (also known as Q) function 20 .&#xA;This instability has several causes: the correlations present in the&#xA;sequence of observations&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am unsure how to understand this and cannot create any hypothetical examples where this can happen. What are some hypothetical scenarios, or real examples where the correlations present in the sequence disrupt the use of a 'deep learning' approximator is used?&lt;/p&gt;&#xA;" OwnerUserId="34" LastActivityDate="2021-03-29T08:50:34.683" Title="Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness" Tags="|deep-learning|reinforcement-learning|correlation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26632" PostTypeId="2" ParentId="26619" CreationDate="2018-01-15T03:56:31.550" Score="1" Body="&lt;p&gt;This is actually a pretty complex question that has to do with many facets of how information entropy is carried in a dataset and how your learning algorithms will make use of the information. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Machine Learning is Optimization&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Machine learning at its core is just finding the ideal parameter values in order for a specific function to be represented. This means you will use the data in your dataset and some framework in order to tune these parameters. Learning is an iterative process, as you can see providing one instance to your algorithm will not result in very good results. However, there comes a point when you have provided sufficient data that the information contained in that subset is representative of the information contained in the entire dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see this in practice, when training your model you will notice that performance starts to converge to a specific value. If you had infinite data this value would not get any better. This is caused by limitations in your data and not your model (machine learning algorithm). This is the best the specific model you are using can do with that kind of data.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How to get better performance?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Assuming you still have more data but performance is no longer improving, you will want to consider a more complex model. This model will likely be able to extract information from your dataset that is too subtle for your simpler model. This is why deep learning outperforms most classical machine learning algorithms when big data is available. Deep learning makes use of very high complexity to approximate functions very accurately, however, the tradeoff is that a lot of data is needed. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Actually my dataset is quite limited, can I improve my performance?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You actually can! This goes back to the core of machine learning, the dataset you are using. The dataset you have constructed has a certain information entropy based on the features you selected. By increasing the information entropy in your dataset, the complexity of your model can be reduced and this will lead to better performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, if you are trying to classify dog/cat, but your dataset contains information about the weather the day the measurements were taken, this will reduce the entropy of the information for classifying dog/cat, thus obscuring your other features and reducing performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, sometimes you may overlook the importance of a certain feature, or overlooking the potential of performing transformations to your features to better suit your model. For example, if you are using linear regression to determine points on 2 circles (radius=1, radius=4). Performing linear regression on the Cartesian plane will lead to very poor results, essentially 50%. However, transforming your $x, y$ coordinates into polar coordinates will lead to 100% performance as they will be perfectly separable. This is a simple application of the kernel trick. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-15T03:56:31.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26633" PostTypeId="1" CreationDate="2018-01-15T04:08:52.587" Score="2" ViewCount="296" Body="&lt;p&gt;Can any one suggest a dataset that contains multi channel/multi dimensional feature values that can be classified using a deep feed forward neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please ask me if the question is not clear enough.&lt;/p&gt;&#xA;" OwnerUserId="44354" LastEditorUserId="28175" LastEditDate="2018-01-15T12:35:53.530" LastActivityDate="2018-05-15T17:12:42.157" Title="multi channel feature classification using deep feed forward neural network on tensorflow" Tags="|machine-learning|neural-network|classification|deep-learning|dataset|" AnswerCount="1" CommentCount="0" ClosedDate="2018-05-18T01:44:02.557" ContentLicense="CC BY-SA 3.0" />
  <row Id="26634" PostTypeId="2" ParentId="24497" CreationDate="2018-01-15T05:07:01.923" Score="0" Body="&lt;p&gt;I would try using Google's CausalImpact package.  Your use-case isn't causal inference exactly, but CausalImpact relies on bayesian structural time series models (using the bsts package) and has some good defaults that keep you from needing to dive into bsts immediately.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you fit a model to the first part of your data, then forecast the rest. You see where your model deviates from the forecast.  Using bayesian models means you can get error bounds - so you can have a degree of confidence in the deviation.  In your case, you'd set your 'intervention' point to wherever timestamp you want to separate your modeling data from your forecasting data. Then compare the forecast to the actual data (look up 'nowcasting').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://google.github.io/CausalImpact/CausalImpact.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is a tutorial to get you started&lt;/a&gt;, and &lt;a href=&quot;https://www.youtube.com/watch?v=GTgZfCltMm8&quot; rel=&quot;nofollow noreferrer&quot;&gt;here's an introductory video.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-01-15T05:07:01.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26636" PostTypeId="2" ParentId="26624" CreationDate="2018-01-15T06:06:04.430" Score="5" Body="&lt;p&gt;Your problem could be solved either by direct numeric integration or by MCMC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Numeric integration&lt;/strong&gt; can be performed most easily by scipy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import scipy.stats&#xA;import scipy.integrate&#xA;&#xA;def weird_density(x):&#xA;    &quot;&quot;&quot; The function I want to sample &quot;&quot;&quot;&#xA;    return scipy.stats.lognorm.pdf(x, 1.0)&#xA;&#xA;def quad_quantile(fun, q, precision=1e-10, minus_inf=-10e10, lower=-10e10, upper=10e10):&#xA;    &quot;&quot;&quot; Use bisection to evaluate a quantile &quot;&quot;&quot;&#xA;    while upper - lower &amp;gt; precision:&#xA;        med = (upper + lower) * 0.5&#xA;        cdf_med = scipy.integrate.quad(fun, minus_inf, med)[0]&#xA;        if cdf_med &amp;lt; q:&#xA;            lower = med&#xA;        else:&#xA;            upper = med&#xA;    return (upper + lower) * 0.5&#xA;&#xA;print('true mean  :', scipy.stats.lognorm(1).mean())&#xA;print('integrated mean:', scipy.integrate.quad(lambda x: weird_density(x) * x, 0, 100)[0])&#xA;print('true median  :', scipy.stats.lognorm(1).ppf(0.5))&#xA;print('integrated median:', quad_quantile(weird_density, 0.5, 0.001, 0, 0, 100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;true mean  : 1.6487212707&#xA;integrated mean: 1.6484641126903046&#xA;true median  : 1.0&#xA;integrated median: 0.9998321533203125&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if the distribution is multidimensional, direct integration may be intractable. In this case,  &lt;strong&gt;MCMC methods&lt;/strong&gt; (Markov chain Monte Carlo) can help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What they do is just make a correct (in some sense) sample from the distribution for which you know the PDF. When you have a sample, you can calculate all your parameters from it as classical sample statistics, just like from any observed data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MCMC algorithms may be implemented manually, like in the example below. Another option is to use the &lt;code&gt;PyMC3&lt;/code&gt; library or its analogs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the best known MCMC algorithms, &lt;a href=&quot;https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Metropolis-Hastings&lt;/a&gt;, works as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sample_next(x):&#xA;    &quot;&quot;&quot; Generate random next point&quot;&quot;&quot;&#xA;    return scipy.stats.norm(loc=x).rvs(1)[0]&#xA;&#xA;def metropolis_hastings(density, generator, n_samples, starting_point=1, random_state=None):&#xA;    &quot;&quot;&quot; Generate sample from the given density function &quot;&quot;&quot;&#xA;    np.random.seed(random_state)&#xA;    result = [starting_point]&#xA;    for i in range(n_samples):&#xA;        current_point = result[-1]&#xA;        next_candidate = generator(current_point)&#xA;        acceptance_ratio = density(next_candidate) / density(current_point)&#xA;        if np.random.uniform() &amp;lt;= acceptance_ratio:&#xA;            result.append(next_candidate)&#xA;        else:&#xA;            result.append(current_point)&#xA;    return np.array(result)&#xA;&#xA;sample = metropolis_hastings(weird_density, sample_next, 1000, random_state=1)&#xA;&#xA;print('sample mean:', np.mean(sample))&#xA;print('sample median:', np.median(sample))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code prints&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample mean: 1.2872857165&#xA;sample median: 0.932826883898&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You see that the results are not very close, but with longer sampling (1000 points is too few) they will converge. And the shape of distribution is already well matched:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;plt.hist(sample, bins=30, normed=True)&#xA;idx = np.linspace(0, sample.max())&#xA;plt.plot(idx, scipy.stats.lognorm(1.0).pdf(idx))&#xA;plt.legend(['true density', 'sample density'])&#xA;plt.title('convergence of MCMC sample')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dtCpX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dtCpX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24162" LastEditorUserId="24162" LastEditDate="2018-01-15T12:01:12.057" LastActivityDate="2018-01-15T12:01:12.057" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26637" PostTypeId="2" ParentId="26631" CreationDate="2018-01-15T07:57:09.353" Score="1" Body="&lt;p&gt;One reason can be the high correlation between training data can cause bias the deep learner. In this way, if the learner gets a new data which is not correlated with the training data, it will have large error.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-15T07:57:09.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26638" PostTypeId="2" ParentId="26593" CreationDate="2018-01-15T09:06:25.560" Score="1" Body="&lt;p&gt;15 attributes with 30161 instances are not much to cause this delay, I've trained with much more. Your system config also plays a major role in this. For instance if you are using anaconda on chrome and you have multiple processes going on then it would cause delay as distances are calculated on the fly which will be heavy on memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try &lt;a href=&quot;http://scikit-learn.org/stable/modules/sgd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SGD&lt;/a&gt; as you are going for linear. It is faster than SVM.&lt;/p&gt;&#xA;" OwnerUserId="44514" LastActivityDate="2018-01-15T09:06:25.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26639" PostTypeId="1" CreationDate="2018-01-15T09:47:51.457" Score="2" ViewCount="5100" Body="&lt;p&gt;I'm working on Timeseries sequence prediction using LSTM. &#xA;My goal is to use window of 25 past values in order to generate a prediction for the next 25 values. I'm doing that recursively:&#xA;I use 25 known values to predict the next value. Append that value as know value then shift the 25 values and predict the next one again until i have 25 new generated values (or more) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using &quot;Keras&quot; to implement the RNN&#xA;Architecture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;regressor = Sequential()&#xA;regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))&#xA;regressor.add(Dropout(0.1))&#xA;&#xA;regressor.add(LSTM(units = 50, return_sequences = True))&#xA;regressor.add(Dropout(0.1))&#xA;&#xA;regressor.add(LSTM(units = 50))&#xA;regressor.add(Dropout(0.1))&#xA;&#xA;regressor.add(Dense(units = 1))&#xA;&#xA;regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')&#xA;&#xA;regressor.fit(X_train, y_train, epochs = 10, batch_size = 32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem: &#xA;Recursive prediction always converge to the some value no matter what sequence comes before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XFqmO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XFqmO.png&quot; alt=&quot;example 1&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ca7fw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ca7fw.png&quot; alt=&quot;example 2&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/xj50h.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xj50h.png&quot; alt=&quot;example 3&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sure this is not what I want, I was expecting that the generated sequence will be different depending on what I have before and I'm wondering if someone have an idea about this behavior and how to avoid it. Maybe I'm doing something wrong ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried different epochs number and didn't help much, actually more epochs made it worse. Changing Batch Size, Number of Units , Number of Layers , and window size didn't help too in avoiding this issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using MinMaxScaler for the data.&lt;/p&gt;&#xA;" OwnerUserId="44637" LastEditorUserId="44637" LastEditDate="2018-01-16T22:24:42.847" LastActivityDate="2020-07-13T21:01:24.400" Title="LSTM Timeseries recursive prediction converge to same value" Tags="|time-series|regression|keras|rnn|lstm|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26640" PostTypeId="1" AcceptedAnswerId="26666" CreationDate="2018-01-15T10:35:20.277" Score="4" ViewCount="17303" Body="&lt;p&gt;I am using machine learning predictions for the sample iris dataset. For instance, I am using the support vector machines (SVMs) from &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn&lt;/a&gt; in order to predict the accuracy. However, it returns an accuracy of 1.0. Here is the code I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)&#xA;svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')&#xA;svm_model.fit(X_train,y_train)&#xA;predictions = svm_model.predict(X_test)&#xA;accuracy_score(predictions, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How to find out or to measure if this over-fitting or if the model is so good? I assume that its not over-fitting but what are the best ways to validate this?&lt;/p&gt;&#xA;" OwnerUserId="42238" LastEditorUserId="28175" LastEditDate="2018-01-15T12:48:22.937" LastActivityDate="2021-02-08T04:42:22.920" Title="How to check for overfitting with SVM and Iris Data?" Tags="|machine-learning|scikit-learn|overfitting|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26641" PostTypeId="1" CreationDate="2018-01-15T11:37:23.640" Score="2" ViewCount="975" Body="&lt;p&gt;I am new on Tensorflow. I am trying to build a neural network that is able to recognize letters and words from documents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have divided the task into 3 phases as what is mentioned in &lt;a href=&quot;http://rrc.cvc.uab.es/?ch=5&amp;amp;com=tasks&quot; rel=&quot;nofollow noreferrer&quot;&gt;ICDAR2017&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Text Localization&lt;/li&gt;&#xA;&lt;li&gt;Cropped Word Recognition&lt;/li&gt;&#xA;&lt;li&gt;End-to-End Recognition&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am having some problems in the 1st phase which is Text Localization. I used an architecture that is called &lt;a href=&quot;https://arxiv.org/abs/1704.03155&quot; rel=&quot;nofollow noreferrer&quot;&gt;EAST&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Architecture stages: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fully Convolutional Network&lt;/li&gt;&#xA;&lt;li&gt;NMS merging stage&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is the &lt;a href=&quot;https://github.com/argman/EAST/blob/master/model.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;model&lt;/a&gt; I started with. It works fine but I am facing some problems in detecting some characters and letters as in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nZfk4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nZfk4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model layers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def model(images, weight_decay=1e-5, is_training=True):&#xA;        '''&#xA;        define the model, we use slim's implemention of resnet&#xA;        '''&#xA;        images = mean_image_subtraction(images)&#xA;&#xA;        with slim.arg_scope(resnet_v1.resnet_arg_scope(weight_decay=weight_decay)):&#xA;            logits, end_points = resnet_v1.resnet_v1_50(images, is_training=is_training, scope='resnet_v1_50')&#xA;&#xA;        with tf.variable_scope('feature_fusion', values=[end_points.values]):&#xA;            batch_norm_params = {&#xA;            'decay': 0.997,&#xA;            'epsilon': 1e-5,&#xA;            'scale': True,&#xA;            'is_training': is_training&#xA;            }&#xA;            with slim.arg_scope([slim.conv2d],&#xA;                                activation_fn=tf.nn.relu,&#xA;                                normalizer_fn=slim.batch_norm,&#xA;                                normalizer_params=batch_norm_params,&#xA;                                weights_regularizer=slim.l2_regularizer(weight_decay)):&#xA;                f = [end_points['pool5'], end_points['pool4'],&#xA;                     end_points['pool3'], end_points['pool2']]&#xA;                for i in range(4):&#xA;                    print('Shape of f_{} {}'.format(i, f[i].shape))&#xA;                g = [None, None, None, None]&#xA;                h = [None, None, None, None]&#xA;                num_outputs = [None, 128, 64, 32]&#xA;                for i in range(4):&#xA;                    if i == 0:&#xA;                        h[i] = f[i]&#xA;                    else:&#xA;                        c1_1 = slim.conv2d(tf.concat([g[i-1], f[i]], axis=-1), num_outputs[i], 1)&#xA;                        h[i] = slim.conv2d(c1_1, num_outputs[i], 3)&#xA;                    if i &amp;lt;= 2:&#xA;                        g[i] = unpool(h[i])&#xA;                    else:&#xA;                        g[i] = slim.conv2d(h[i], num_outputs[i], 3)&#xA;                    print('Shape of h_{} {}, g_{} {}'.format(i, h[i].shape, i, g[i].shape))&#xA;&#xA;                # here we use a slightly different way for regression part,&#xA;                # we first use a sigmoid to limit the regression range, and also&#xA;                # this is do with the angle map&#xA;                F_score = slim.conv2d(g[3], 1, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None)&#xA;                # 4 channel of axis aligned bbox and 1 channel rotation angle&#xA;                geo_map = slim.conv2d(g[3], 4, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None) * FLAGS.text_scale&#xA;                angle_map = (slim.conv2d(g[3], 1, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None) - 0.5) * np.pi/2 # angle is between [-45, 45]&#xA;                F_geometry = tf.concat([geo_map, angle_map], axis=-1)&#xA;&#xA;        return F_score, F_geometry&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have been advised to use &lt;a href=&quot;https://arxiv.org/pdf/1612.03144.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pyramid Networks&lt;/a&gt;, I don't know who to integrate this Pyramid Networks with the EAST model in Tensorflow and it is worth it or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like also to know my options to increase the Text Localization accuracy in the table based data&lt;/p&gt;&#xA;" OwnerUserId="44641" LastEditorUserId="29575" LastEditDate="2018-01-15T15:58:06.573" LastActivityDate="2018-01-15T15:58:06.573" Title="Text Localization using Convolutional NN" Tags="|python|neural-network|tensorflow|cnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26642" PostTypeId="2" ParentId="26597" CreationDate="2018-01-15T11:59:37.477" Score="28" Body="&lt;p&gt;The consideration of the number of neurons for each layer and number of layers in &lt;strong&gt;fully connected networks&lt;/strong&gt; depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of &lt;a href=&quot;https://scholar.google.com/citations?user=HOauvegAAAAJ&amp;amp;hl=en&quot; rel=&quot;noreferrer&quot;&gt;a scientist&lt;/a&gt;. For understanding other nets like &lt;code&gt;CNN&lt;/code&gt; I recommend you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have just a single neuron, in this case after learning the parameters of the network you will have a linear decision boundary which can separate the space to two individual classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Yo1rq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Yo1rq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gYDvI.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gYDvI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that you are asked to separate the following data. You will need &lt;code&gt;d1&lt;/code&gt; which specifies the upper decision boundary and somehow is doing &lt;code&gt;AND&lt;/code&gt; operation to determine whether the input data is on the left side of it or on the right side. Line &lt;code&gt;d2&lt;/code&gt; is doing another &lt;code&gt;AND&lt;/code&gt; operation which investigates whether the input data is upper than &lt;code&gt;d2&lt;/code&gt; or not. In this case &lt;code&gt;d1&lt;/code&gt; is trying to understand whether the input is on the left side of line to classify the input as &lt;em&gt;circle&lt;/em&gt;, also &lt;code&gt;d2&lt;/code&gt; is trying to figure out whether the input is on the right side of the line to classify the input as &lt;em&gt;circle&lt;/em&gt;. Now we need another &lt;code&gt;AND&lt;/code&gt; operation to wrap up the results of the two lines which are constructed after training their parameters. If the input is on the left side of &lt;code&gt;d1&lt;/code&gt; and on the right side of &lt;code&gt;d2&lt;/code&gt;, it should be classified as &lt;em&gt;circle&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JOB2t.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JOB2t.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose that you have the following problem and you are asked to separate the classes. In this case the justification is exactly like the above's.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1kIMG.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1kIMG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the following data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6hLmG.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6hLmG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the decision boundary is not convex and is more complex than the previous boundaries. First you have to have a sub-net which finds the inner circles. Then you have to have another sub-net which finds the inner rectangular decision boundary which decides the inputs which are inside of the rectangle are not circle and if they are outside, they are circle. After these, you have to wrap up the results and say if the input data is inside the bigger rectangle and outside of the inner rectangle, it should be classified as &lt;em&gt;circle&lt;/em&gt;. You need another &lt;code&gt;AND&lt;/code&gt; operation for this purpose. The network would be like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A8FiS.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A8FiS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr /&gt;&#xA;&#xA;&lt;p&gt;Suppose that you are asked to find the following &lt;strong&gt;circled&lt;/strong&gt; decision boundary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XXFsf.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XXFsf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case your network would be like the following network which was referred to but with much more neurons in the first hidden layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UM1rH.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UM1rH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-15T11:59:37.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26644" PostTypeId="2" ParentId="4974" CreationDate="2018-01-15T12:17:37.263" Score="2" Body="&lt;p&gt;If you are using python, and have created a weighted graph using &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NetworkX&lt;/a&gt;, then you can use &lt;a href=&quot;https://github.com/taynaud/python-louvain&quot; rel=&quot;nofollow noreferrer&quot;&gt;python-louvain&lt;/a&gt; for clustering. Where G is a weighted graph:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import community &#xA;partition = community.best_partition(G, weight='weight')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-01-15T12:17:37.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26645" PostTypeId="2" ParentId="26633" CreationDate="2018-01-15T12:34:43.790" Score="1" Body="&lt;p&gt;You have not specified which deep net you are using. You can use any data set which is of the right format in &lt;code&gt;TensorFlow&lt;/code&gt;. If you are using convolutional networks, You may need data which locality is important. In this case:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;MNIST data set &lt;/li&gt;&#xA;&lt;li&gt;ImageNet data set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;may help you. If you are doing other tasks I highly recommend you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/155/28175&quot;&gt;this&lt;/a&gt; question which is so much popular and has great answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to quote, I quote the &lt;a href=&quot;https://datascience.stackexchange.com/a/4997/28175&quot;&gt;following answer&lt;/a&gt;. &lt;a href=&quot;http://networkrepository.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Repository&lt;/a&gt; an interactive data repository with over 600+ networks in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks my &lt;strong&gt;help&lt;/strong&gt; you.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h2&gt;Data Sets&lt;/h2&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://academictorrents.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Academic Torrents&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://hadoopilluminated.com/hadoop_illuminated/Public_Bigdata_Sets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;hadoopilluminated.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://catalog.data.gov/dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;data.gov&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.quandl.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quandl&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;https://www.freebase.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;freebase.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://usgovxml.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;usgovxml.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://public.enigma.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;enigma.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://datahub.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;datahub.io&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://aws.amazon.com/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;aws.amazon.com/datasets&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://databib.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;databib.org&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.datacite.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;datacite.org&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;https://www.quandl.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;quandl.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://figshare.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;figshare.com&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://dev.maxmind.com/geoip/legacy/geolite/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GeoLite Legacy Downloadable Databases&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora's Big Datasets Answer&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://hadoopilluminated.com/hadoop_illuminated/Public_Bigdata_Sets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Public Big Data Sets&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://data.ohouston.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Houston Data Portal&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;https://www.kaggle.com/wiki/DataSources&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Data Sources&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.1000genomes.org/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Deep Catalog of Human Genetic Variation&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;https://www.freebase.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;A community-curated database of well-known people, places, and things&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.google.com/publicdata/directory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Public Data&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://data.worldbank.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;World Bank Data&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://nyctaxi.herokuapp.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NYC Taxi data&lt;/a&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://www.opendataphilly.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open Data Philly&lt;/a&gt; Connecting people with data for Philadelphia&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://networkrepository.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Repository&lt;/a&gt; An interactive data repository with over 600+ networks  in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks, etc.&lt;/li&gt;&#xA;  &lt;li&gt;&lt;a href=&quot;http://ahmetkurnaz.net/en/statistical-data-sources/&quot; rel=&quot;nofollow noreferrer&quot;&gt;A list of useful sources&lt;/a&gt; A blog post includes many data set databases&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/okulbilisim/awesome-datascience#data-sets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Sets&lt;/a&gt; From &lt;a href=&quot;https://github.com/okulbilisim/awesome-datascience&quot; rel=&quot;nofollow noreferrer&quot;&gt;awesome-datascience&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-15T13:02:17.303" LastActivityDate="2018-01-15T13:02:17.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26646" PostTypeId="2" ParentId="26640" CreationDate="2018-01-15T12:51:09.863" Score="0" Body="&lt;p&gt;Based on &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, use &lt;code&gt;sklearn.model_selection.train_test_split(*arrays, **options)&lt;/code&gt; in order to split your data into train and test. Train your model on train-split and use the &lt;code&gt;predict&lt;/code&gt; method to see the performance on the test data. As an example take a look at the following code which splits the data to two separate groups.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.model_selection import train_test_split&#xA;X, y = np.arange(10).reshape((5, 2)), range(5)&#xA;X&#xA;&#xA;array([[0, 1],&#xA;       [2, 3],&#xA;       [4, 5],&#xA;       [6, 7],&#xA;       [8, 9]])&#xA;&#xA;list(y)&#xA;[0, 1, 2, 3, 4]&#xA;&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(&#xA;...     X, y, test_size=0.33, random_state=42)&#xA;...&#xA;X_train&#xA;array([[4, 5],&#xA;   [0, 1],&#xA;   [6, 7]])&#xA;y_train&#xA;[2, 0, 3]&#xA;X_test&#xA;array([[2, 3],&#xA;   [8, 9]])&#xA;y_test&#xA;[1, 4]&#xA;&#xA;train_test_split(y, shuffle=False)&#xA;[[0, 1, 2], [3, 4]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-15T12:51:09.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26647" PostTypeId="2" ParentId="5589" CreationDate="2018-01-15T12:57:27.603" Score="4" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Launch an EC2 instance with enough storage  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;ssh to the instance &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Obtain the curl command corresponding to the download from your local machine. You can use the developer options in Google chrome -&gt; network tab -&gt; copy -&gt; copy as curl &#xA;(this step is necessary for some websites requiring authentication such as kaggle) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;From the instance terminal, run the &lt;code&gt;curl&lt;/code&gt; command (append &lt;code&gt;-o output_file&lt;/code&gt; to the command). This will download and save the file &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Configure aws credentials to connect the instance to s3 (one way is to use the command &lt;code&gt;aws config&lt;/code&gt;, provide AWS access key Id and secret), &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use this command to upload the file to s3: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;aws s3 cp path-to-file s3://bucket-name/&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44643" LastEditorUserId="29575" LastEditDate="2018-01-15T15:41:34.147" LastActivityDate="2018-01-15T15:41:34.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26648" PostTypeId="1" AcceptedAnswerId="26649" CreationDate="2018-01-15T13:22:44.650" Score="13" ViewCount="36404" Body="&lt;p&gt;I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value &lt;code&gt;0.8660254&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[ 1.         0.8660254]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[ 0.8660254  1.       ]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import cosine_similarity&#xA;vec1 = [1,1,0,1,1]&#xA;vec2 = [0,1,0,1,1]&#xA;print(cosine_similarity([vec1, vec2]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44645" LastEditorUserId="28175" LastEditDate="2018-02-11T21:18:07.140" LastActivityDate="2021-05-04T13:14:39.783" Title="cosine_similarity returns matrix instead of single value" Tags="|machine-learning|python|scikit-learn|cosine-distance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26649" PostTypeId="2" ParentId="26648" CreationDate="2018-01-15T13:26:22.680" Score="15" Body="&lt;p&gt;Based on the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt; &lt;code&gt;cosine_similarity(X, Y=None, dense_output=True)&lt;/code&gt; returns &lt;em&gt;an array with shape (n_samples_X, n_samples_Y)&lt;/em&gt;. Your mistake is that you are passing [vec1, vec2] as the first input to the method. Also your vectors should be &lt;code&gt;numpy&lt;/code&gt; arrays:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics.pairwise import cosine_similarity&#xA;import numpy as np&#xA;vec1 = np.array([[1,1,0,1,1]])&#xA;vec2 = np.array([[0,1,0,1,1]])&#xA;#print(cosine_similarity([vec1, vec2]))&#xA;print(cosine_similarity(vec1, vec2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;X : ndarray or sparse array, shape: (n_samples_X, n_features)&#xA;      Input data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So you have to specify the dimension. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.array([1, 2]).shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;has funny shape:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(2, )&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-15T13:42:19.007" LastActivityDate="2018-01-15T13:42:19.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26650" PostTypeId="2" ParentId="26615" CreationDate="2018-01-15T13:38:31.857" Score="2" Body="&lt;p&gt;In the code, you have done a split of the data into train/test. If you have used all samples to &quot;train&quot; manually your decision tree you have more samples to do the calculations than the sklearn algorithm, so the results may change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can look which samples has been chosen to train the tree and do the calculations with those samples.&lt;/p&gt;&#xA;" OwnerUserId="44647" LastActivityDate="2018-01-15T13:38:31.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26651" PostTypeId="1" CreationDate="2018-01-15T14:22:56.887" Score="1" ViewCount="6173" Body="&lt;p&gt;I have data estimating what kind of food is used on a daily basis at school to feed kids.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Pizza   Apples  Lemonade  Lemon   Banana  Milk  Soup&#xA; 12        3       10       1      9      10     6&#xA; 14        5       8        0      13     11     7&#xA; ....................................................&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to create a new variable &lt;code&gt;Fruits_total&lt;/code&gt; which is a sum of variables &lt;code&gt;Apples&lt;/code&gt;, &lt;code&gt;Lemon&lt;/code&gt;, and &lt;code&gt;Banana&lt;/code&gt;, and add it to an existing data frame.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  df['Fruits_total'] = df.Apples + df.Lemon + df.Banana&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it doesn't work. Where is my mistake?&lt;/p&gt;&#xA;" OwnerUserId="44652" LastEditorUserId="29575" LastEditDate="2018-01-15T15:45:15.877" LastActivityDate="2018-01-22T14:35:59.653" Title="How do I create a new column to an existing data frame which is the product of other columns?" Tags="|python|pandas|" AnswerCount="1" CommentCount="1" ClosedDate="2018-01-28T21:49:58.563" ContentLicense="CC BY-SA 3.0" />
  <row Id="26652" PostTypeId="1" CreationDate="2018-01-15T14:44:23.153" Score="1" ViewCount="10909" Body="&lt;p&gt;I'm new to ML and trying out &lt;a href=&quot;https://github.com/yashhy/sklearn-kick-start&quot; rel=&quot;nofollow noreferrer&quot;&gt;basic samples&lt;/a&gt; using sklearn. I have achieved converting text (single dimension) to numbers using TF-IDF and got the &lt;a href=&quot;https://github.com/yashhy/sklearn-kick-start/tree/master/spam-or-not/test-ur-sentence&quot; rel=&quot;nofollow noreferrer&quot;&gt;predictions correct&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have a different &lt;a href=&quot;https://raw.githubusercontent.com/yashhy/sklearn-kick-start/master/tfidf-2d-matrix/adult.data&quot; rel=&quot;nofollow noreferrer&quot;&gt;use-case&lt;/a&gt; to predict &lt;strong&gt;&lt;em&gt;Work Hours&lt;/em&gt;&lt;/strong&gt; based on &lt;strong&gt;&lt;em&gt;Marital-Status&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;Relationship Status&lt;/em&gt;&lt;/strong&gt; so my training data is now two dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have &lt;code&gt;DataFrame&lt;/code&gt; total of &lt;code&gt;24420 rows * 2 columns&lt;/code&gt; of shape --&gt; &lt;code&gt;(24420, 2)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this &lt;code&gt;DataFrame&lt;/code&gt; is been passed to &lt;code&gt;TfidfVectorizer.fit_transform()&lt;/code&gt; function it turns to shape --&gt; &lt;code&gt;(2, 3)&lt;/code&gt; Not sure why!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And while training &lt;code&gt;mnb.fit(x_train_tf, y_train)&lt;/code&gt; it returns an error&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Found input variables with inconsistent numbers of samples: [2, 24420]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You could see the initial &lt;code&gt;DataFrame&lt;/code&gt; shape of &lt;code&gt;24420*2&lt;/code&gt; it transposed to &lt;code&gt;2*24420&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All code is up &lt;a href=&quot;https://github.com/yashhy/sklearn-kick-start/blob/f3e5c5f1e1fcabe6ac80a2dbb21b84b93143c4d0/tfidf-2d-matrix/tfidf_vectorize_2d.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Why after &lt;code&gt;fit_transform()&lt;/code&gt; the shape changes to &lt;code&gt;(2,3)&lt;/code&gt; what happened to &lt;code&gt;(24420,2)&lt;/code&gt; data? What does this denotes?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why does the &lt;code&gt;ValueError: Found input variables with inconsistent numbers of samples [2, 24420]&lt;/code&gt; occurs?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why initial &lt;code&gt;DataFrame&lt;/code&gt; shape changed to &lt;code&gt;2*24420&lt;/code&gt;?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What is wrong with the implementation?&lt;/p&gt;&#xA;" OwnerUserId="44649" LastEditorUserId="29575" LastEditDate="2018-01-15T15:48:16.417" LastActivityDate="2019-02-03T15:42:34.173" Title="Sklearn tfidf vectorize returns different shape after fit_transform()" Tags="|python|scikit-learn|feature-extraction|tfidf|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26653" PostTypeId="1" AcceptedAnswerId="26657" CreationDate="2018-01-15T15:28:09.020" Score="1" ViewCount="176" Body="&lt;p&gt;I have a Recommender system which uses a Collaborative bayesian approach using pSDAE for recommending scientific articles from the Citeulike Dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The text information (as input to pSDAE) is in the file mult.dat and the rating matrix (as input for the MF part) is in the file cf-train-1-users.dat and is loaded using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_mult():&#xA;    X = read_mult('mult.dat',8000).astype(np.float32)&#xA;    return X&#xA;&#xA;def read_user(f_in='cf-train-1-users.dat',num_u=5551,num_v=16980):&#xA;    fp = open(f_in)&#xA;    R = np.mat(np.zeros((num_u,num_v)))&#xA;    for i,line in enumerate(fp):&#xA;        segs = line.strip().split(' ')[1:]&#xA;        for seg in segs:&#xA;            R[i,int(seg)] = 1&#xA;    return R&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The raw data is in proper Excel format with citations as doc-id, title, citeulike-id, raw-title, raw-abstract.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The mult.dat file containing hte text information looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ieTrU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1 &lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trainusers.dat file looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jzIPL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the link to th ipynb for the whole Recommender system:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/js05212/MXNet-for-CDL/blob/master/collaborative-dl.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/js05212/MXNet-for-CDL/blob/master/collaborative-dl.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40277" LastEditorUserId="29575" LastEditDate="2018-01-15T15:50:53.287" LastActivityDate="2018-01-15T16:54:18.853" Title="Can't interpret the text information and ratings matrix imported to NN" Tags="|machine-learning|neural-network|deep-learning|dataset|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26654" PostTypeId="1" AcceptedAnswerId="26673" CreationDate="2018-01-15T15:42:13.390" Score="1" ViewCount="855" Body="&lt;p&gt;In the &lt;a href=&quot;https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tkdd11.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;original publication&lt;/a&gt; of the Isolation Forest algorithm, the authors mention a height limit parameter to control the granularity of the algorithm. I did not find that explicit parameter on the Sklearn implementation of the algorithm, and I was wondering whether it is possible to control granularity in some other way?&lt;/p&gt;&#xA;" OwnerUserId="44658" LastEditorUserId="29575" LastEditDate="2018-01-15T15:52:58.393" LastActivityDate="2022-05-25T19:36:03.250" Title="Isolation Forest height limit absent in SkLearn implementation" Tags="|scikit-learn|decision-trees|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26655" PostTypeId="1" CreationDate="2018-01-15T15:44:15.837" Score="1" ViewCount="52" Body="&lt;p&gt;I am out of ideas in regards to plotting different sets of data .... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 10 sets of data ... Each set contains Actual Speed of vehicle and predicted Speed of Vehicle..  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can i combine all sets of data and show in a meaningful manner ?  I don't think I can simply average as they are absolute values (In one run , i was going 25mph and second run , i might be going 30 mph)//...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any idea on how i can represent 10 sets of data in a meaningful manner like variance  ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas are appreciated &lt;/p&gt;&#xA;" OwnerUserId="44660" LastActivityDate="2018-04-25T22:01:19.140" Title="Representing different sets of data" Tags="|visualization|plotting|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26656" PostTypeId="1" CreationDate="2018-01-15T16:22:34.080" Score="1" ViewCount="160" Body="&lt;p&gt;What techniques are currently available for e.g. locating the ZIP code on the envelope and cutting it to single digits prior to feeding them to digit recognition algorithm (trained on MNIST dataset)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or imagine a task of automatic recognition if a phone call recording contains a sound of a train passing a track switch (just yes or no). If present at all, it can be gentle or loud, depending if a window was open. It can be slow or fast, depending how quick the train was going through the switch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found so far Spatial Transformations Networks mentioned &lt;a href=&quot;https://datascience.stackexchange.com/questions/22086/using-neural-networks-to-recognize-digits-in-a-scene&quot;&gt;in this comment&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44654" LastActivityDate="2018-01-15T16:22:34.080" Title="MNIST digits... And what technique to find ZIP code on the envelope?" Tags="|machine-learning|beginner|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26657" PostTypeId="2" ParentId="26653" CreationDate="2018-01-15T16:46:32.730" Score="1" Body="&lt;p&gt;&lt;a href=&quot;http://www.wanghao.in/&quot; rel=&quot;nofollow noreferrer&quot;&gt;I am the author of the CDL paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the mult.data file, in&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;63&lt;/code&gt; is the number of words for this documents, &lt;code&gt;1:2&lt;/code&gt; means word 1 appears twice in the document, &lt;code&gt;1666:1&lt;/code&gt; means word 1666 appears once in the document, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the trainuser.dat, in&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;10&lt;/code&gt; is the number of positive samples for this user, the rest is a list of 10 items that are related to (liked by) this user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check the README file in the &lt;a href=&quot;http://www.wanghao.in/data/ctrsr_datasets.rar&quot; rel=&quot;nofollow noreferrer&quot;&gt;Datasets collection&lt;/a&gt; for more details on the datasets.&lt;/p&gt;&#xA;" OwnerUserId="44664" LastEditorUserId="29575" LastEditDate="2018-01-15T16:54:18.853" LastActivityDate="2018-01-15T16:54:18.853" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26659" PostTypeId="2" ParentId="26640" CreationDate="2018-01-15T19:36:36.613" Score="1" Body="&lt;p&gt;It might be a problem of over-fitting, or that by just doing a single train / test split isn't giving a reliable estimate of the generalizable error of your SVM.&lt;/p&gt;&#xA;&lt;p&gt;I'd recommend using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;KFold&quot;&gt;KFold&lt;/a&gt; validation to check.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import KFold&#xA;import numpy as np&#xA;acc_score = []&#xA;&#xA;kf = KFold(n_splits=5)&#xA;&#xA;for train_index, test_index in kf.split(X):&#xA;&#xA;    X_train, X_test = X[train_index], X[test_index]&#xA;    y_train, y_test = y[train_index], y[test_index]&#xA;&#xA;    svm_model.fit(X_train,y_train)&#xA;    predictions = svm_model.predict(X_test)&#xA;    acc_score.append(accuracy_score(predictions, y_test))&#xA;&#xA;np.mean(acc_score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;If the average is still 1.0 then you've done good, but my gut feeling is that your high score is dependent on the cut of the data you're looking at.&lt;/p&gt;&#xA;" OwnerUserId="21702" LastEditorUserId="106250" LastEditDate="2021-02-08T04:42:22.920" LastActivityDate="2021-02-08T04:42:22.920" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26661" PostTypeId="1" AcceptedAnswerId="26662" CreationDate="2018-01-15T20:19:20.643" Score="1" ViewCount="525" Body="&lt;p&gt;In sklearn, there is a method for cross validation called cross_val_score. One of the parameters of this method is 'cv'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand in cross validation, there is no splitting the data into training and validation (70-30 split). Instead, you split the data into 'k' subsamples, then train it on the K-1 subsamples and validate using the kth sample.&#xA;And repeat it for each of the 'k' subsamples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is this cv = k, i.e the number of subsamples in which you split the training data?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2018-01-15T20:37:52.743" Title="What do you pass for the cv parameter in the sklearn method cross_val_score" Tags="|python|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26662" PostTypeId="2" ParentId="26661" CreationDate="2018-01-15T20:37:52.743" Score="2" Body="&lt;p&gt;It determines the splitting strategy used by sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The default (“none”) is 3-fold CV. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SXzCz.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SXzCz.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30359" LastActivityDate="2018-01-15T20:37:52.743" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26663" PostTypeId="1" CreationDate="2018-01-15T20:51:20.910" Score="13" ViewCount="22726" Body="&lt;p&gt;I'm trying to use LSTM on time-series data in order to generate future sequences that looks like the original sequences in term of values and progression direction. &#xA;My approach is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;train RNN to predict a value based on 25 past values then use the model to recursively generate future predictions by appending the predicted values to the original sequence and shift the old values ... &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Playing with LSTM cells, I found that the model is not able to learn to generate sequences that looks like the original data. It only predict next value then start converging to an 'equilibrium' or static value which is the same whatever the input sequence is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm wondering if Stateful LSTM would help to learn better from past values and try to predict something close to what it have already? My goal here is to generate sequences that looks like something that the model have seen already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know if I'm missing something or if you had a similar situation and you found the best approach to generate timeseries sequences that looks like what the model learned in the past.&lt;/p&gt;&#xA;" OwnerUserId="44637" LastEditorUserId="88351" LastEditDate="2020-01-19T02:15:44.360" LastActivityDate="2020-01-19T02:15:44.360" Title="When to use Stateful LSTM?" Tags="|time-series|rnn|lstm|generative-models|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="26665" PostTypeId="1" CreationDate="2018-01-15T21:52:53.153" Score="1" ViewCount="124" Body="&lt;p&gt;Consider a scenario where I have a lot of aggregate data as well as individual data that makes up that aggregated data. For example, Rotten Tomatoes, where aggregate data would be movie tags and overall critic and audience rankings, while the individual data would be each review. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to treat this as input to a neural network? Do I just repeat the corresponding aggregated data when feeding in each individual review to be classified? Is there a better way to do this? &lt;/p&gt;&#xA;" OwnerUserId="44678" LastActivityDate="2018-01-15T21:52:53.153" Title="Handling aggregate and individual data input to neural networks?" Tags="|machine-learning|neural-network|classification|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26666" PostTypeId="2" ParentId="26640" CreationDate="2018-01-15T21:55:28.883" Score="6" Body="&lt;p&gt;You check for hints of &lt;strong&gt;overfitting&lt;/strong&gt; by using a &lt;strong&gt;training set&lt;/strong&gt; and a &lt;strong&gt;test set&lt;/strong&gt; (or a training, validation and test set). As others have mentioned, you can either &lt;strong&gt;split&lt;/strong&gt; the data into training and test sets, or use &lt;strong&gt;cross-fold validation&lt;/strong&gt; to get a more accurate assessment of your classifier's performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since your dataset is small, splitting your data into training and test sets isn't recommended. Use cross validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done using either the &lt;strong&gt;cross_validate&lt;/strong&gt; or &lt;strong&gt;cross_val_score&lt;/strong&gt; function; the latter providing &lt;em&gt;multiple&lt;/em&gt; metrics for evaluation. In addition to &lt;strong&gt;test scores&lt;/strong&gt; the latter also provides &lt;strong&gt;fit times&lt;/strong&gt; and &lt;strong&gt;score times&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using your example; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;from sklearn.datasets import load_iris&#xA;from sklearn import svm&#xA;from sklearn.metrics import accuracy_score&#xA;&#xA;iris = load_iris()&#xA;X = iris.data[:, :5]  # we only take the first two features.&#xA;y = iris.target&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)&#xA;svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')&#xA;svm_model.fit(X_train,y_train)&#xA;predictions = svm_model.predict(X_test)&#xA;accuracy_score(predictions, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;raw accuracy: 0.96666666666666667&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using the &lt;strong&gt;cross_val_score&lt;/strong&gt; function, and printing the &lt;strong&gt;mean score&lt;/strong&gt; and &lt;strong&gt;95% confidence interval&lt;/strong&gt; of the score estimate: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import cross_val_score&#xA;&#xA;scores = cross_val_score(svm_model, iris.data, iris.target, cv=5)&#xA;print(&quot;Accuracy: %0.2f (+/- %0.2f)&quot; % (scores.mean(), scores.std() * 2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Accuracy: 0.98 (+/- 0.03)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the iris dataset is a &lt;em&gt;toy example&lt;/em&gt;. On larger real-world datasets you are likely to see your test error be higher than your training error, with cross-validation providing a lower accuracy than the raw number. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I wouldn't use the iris dataset to showcase overfitting. Choose a larger, messier dataset, and then you can start working towards reducing the &lt;em&gt;bias&lt;/em&gt; and &lt;em&gt;variance&lt;/em&gt; of the model (the &quot;causes&quot; of overfitting). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can start &lt;em&gt;exploring&lt;/em&gt; tell-tale &lt;strong&gt;signs&lt;/strong&gt; of whether it's a bias problem or a variance problem. See here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/How-many-training-samples-are-needed-to-get-a-reliable-model-in-ML/answer/Sean-McClure-3?srid=zGgv&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-many-training-samples-are-needed-to-get-a-reliable-model-in-ML/answer/Sean-McClure-3?srid=zGgv&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eXCNE.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eXCNE.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30359" LastEditorUserId="30359" LastEditDate="2018-01-16T15:08:30.487" LastActivityDate="2018-01-16T15:08:30.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26669" PostTypeId="1" AcceptedAnswerId="26670" CreationDate="2018-01-15T22:23:52.037" Score="1" ViewCount="1684" Body="&lt;p&gt;You split the data in k subsamples. Train it on k-1 subsamples, test it on kth subsample, record the performance with some error merric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do it k times for each of the k subsamples, record the error each time.&#xA;Then choose the model with the lowest error?&#xA;Is it the same as ensemble technique?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2018-01-15T22:41:09.443" Title="How does k fold cross validation work?" Tags="|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26670" PostTypeId="2" ParentId="26669" CreationDate="2018-01-15T22:41:09.443" Score="3" Body="&lt;p&gt;Imagine you have 1500 labeled data points, and you want to estimate how well some classifier will work on new data. One (naive) method would be to train a model with all 1500 of your data points, and then check how many of the 1500 data points were classified correctly. This is not likely to give a good estimate of the performance on new data, because new data was not used to test the model. Some models like decision trees and neural networks will often be able to get 100% accuracy on the training data, but perform much worse on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you think to yourself that you will split the data into two sets - a training set which you will build a model with, and a testing set that you will use to evaluate the model. Lets say you decided to train the model with 1000 of your examples, and evaluate with 500. This should give a reasonable estimate of how well your model will perform on new data, but it seems a bit limited; after all, one third of your data has not been used for training at all! We only have predictions for the 500 test samples - if these ones randomly happened to be easier to classify correctly on average, then our performance estimate is overly optimistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross validation is a way to address this. Lets set $k=3$, so the data is split into three sets of 500 points (A, B and C). Use A &amp;amp; B to train a model, and get predictions for C with this model. Use B &amp;amp; C to train a model, to get predictions for A. Finally, use A &amp;amp; C to train a model, and get predictions for B. Now we have a prediction for every point in our labeled data that came from a model trained on different data. By averaging the performance of each of these models, we can end up with a better estimate of how well the model will perform on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you should then re-train your model using all 1500 labeled points if you want to apply it to new data. Cross validation is only for estimating the performance of this new model. Also if your data is large enough, cross validation is probably unnecessary and you could just make a single train/test or train/valid/test split. &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-01-15T22:41:09.443" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26671" PostTypeId="1" CreationDate="2018-01-15T23:12:51.163" Score="4" ViewCount="13345" Body="&lt;p&gt;I'm trying to draw the keras model with the plotmodel.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Setup:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I installed graphviz binaries with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;choco install graphviz&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;added path to the bin folder, and then I did:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install pydotplus&#xA;pip install graphviz&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Getting this error when I try to execute:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.utils.vis_utils import plot_model&#xA;model = Sequential()&#xA;model.add(Dense(2, input_dim=1, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Error:&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ImportError: Failed to import pydot. You must install pydot and graphviz for &lt;code&gt;pydotprint&lt;/code&gt; to work.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest some alternative to draw the keras model? I need to implement this for a CNN.&lt;/p&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="29575" LastEditDate="2018-01-15T23:16:46.970" LastActivityDate="2018-08-15T05:13:45.930" Title="How to use plot model in keras?" Tags="|python|keras|plotting|jupyter|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26673" PostTypeId="2" ParentId="26654" CreationDate="2018-01-16T01:26:30.750" Score="1" Body="&lt;p&gt;Unfortunately, it seems like there is no &lt;code&gt;hlim&lt;/code&gt; parameter incorporated into &lt;code&gt;sklearn.ensemble.IsolationForest&lt;/code&gt;. &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/iforest.py#L281&quot; rel=&quot;nofollow noreferrer&quot;&gt;Calculation&lt;/a&gt; of anomaly score is just based on the depth each point settles to and by the average path length. The only way to tune in a bit is by using contamination which calculates the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/iforest.py#L203&quot; rel=&quot;nofollow noreferrer&quot;&gt;threshold&lt;/a&gt; needed to set for anomaly score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To achieve the granularity that was given in the original paper using &lt;code&gt;hlim=6&lt;/code&gt; to detect a cluster of small points, using a lot of estimators may solve the problem (still depends heavily on how you sample data from smaller cluster into a lot of estimators). But if that small cluster of data is very less in number, I don't think this idea works and there is nothing much we can do from the current implementation in &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/iforest.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-01-16T01:26:30.750" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26674" PostTypeId="1" CreationDate="2018-01-16T03:54:32.490" Score="3" ViewCount="2465" Body="&lt;p&gt;I am working on network analysis project at one of our Nation's Service Academies, and I need a little help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a starting point, we are looking at a citation network that we build by using the keyword, &quot;Network Analysis&quot;, for peer reviewed articles only in Web of Science. We took the top 50 results, and looked at the articles to which those cited and built a citation network.  It outputs the information in Bibtex, Tab-Delimited, HTML, Plain Text, End Note, and a few others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the best way to create a citation network?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;We have these formats, but it is not usable yet.  Ideally, we would have a matrix with article names on the x and y, and binary numbers for the&#xA;    connections.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/rdCpp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rdCpp.png&quot; alt=&quot;sample data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44683" LastEditorUserId="44683" LastEditDate="2018-01-16T13:55:26.830" LastActivityDate="2018-01-17T09:09:43.643" Title="Building a Citation Network to Analyze in R" Tags="|r|social-network-analysis|data-formats|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26675" PostTypeId="1" CreationDate="2018-01-16T04:14:14.197" Score="0" ViewCount="170" Body="&lt;p&gt;I am looking at a problem where I have two documents, both are textual documents. The first one is a long (few pages long) textual document, while the second one contains about 10 short texts, each being a one liner. The problem is to automatically understand the 1st document and &quot;map&quot; key ideas from here to one or more short texts in the second document. I am not being able to get a good grip on the problem as to how to proceed. I tried the following approach: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;first collected as much similar documents as possible which are relevant; I have obtained 136 such documents, each few pages long. Then build a document level embedding using each sentence as a document (doc2vec). Then for each short sentence in the second document, inferred an embedding based on the document model I built. Then tried to find the most similar sentences for each of these short texts from the 2nd document. However the results are not good. Only about 30% match is what I am getting and the  corresponding sentences are not much related to the target sentence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if this is the right approach or are there any other approach available which I am not aware of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your advice would be greatly appreciated. It may be the case that my explanation above is not adequate. Please let me know and I will try to improve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your input would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="3314" LastEditDate="2018-01-16T18:35:55.733" LastActivityDate="2018-01-16T18:35:55.733" Title="Text understanding and mapping" Tags="|nlp|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26676" PostTypeId="2" ParentId="26459" CreationDate="2018-01-16T04:46:33.840" Score="1" Body="&lt;p&gt;So, it turns out that my problem had a lot more to do with network structure than anything else. I was trying to share too many lower levels, and then still not leaving enough room for the later branches to learn individual features/locations. In the end, I re-structured my network so that there was only 1-2 shared convolutional layers, then each primary branch shared 1 further convolutional layer before again each branching into 3 sub-branches (one for each output), each with their own convolutional, dense, flatten and then softmax layers. I also possibly had too many filters on each convolutional layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This enabled 'training' across each of the 9 outputs, but has left me with some serious overfitting to try to overcome, due to the depth of the network and the small data set. I am currently trialing methods of correcting for this.&lt;/p&gt;&#xA;" OwnerUserId="44340" LastActivityDate="2018-01-16T04:46:33.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26677" PostTypeId="2" ParentId="26674" CreationDate="2018-01-16T05:55:24.957" Score="1" Body="&lt;p&gt;Parse an edgelist out of your data, load it into a dataframe with two columns (source, target), then feed that to &lt;a href=&quot;http://igraph.org/r/doc/graph_from_data_frame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;igraph::graph_from_data_frame&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-16T05:55:24.957" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26678" PostTypeId="1" CreationDate="2018-01-16T05:57:38.527" Score="1" ViewCount="4490" Body="&lt;p&gt;I have an excel database of 50 rows and some 28,000 columns. The database provides information of water parameters such as pressure, roughness, pipe length, diameter, valve, pump, flow and so on. I want to apply k-means code on this data to find outliers.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import read_excel&#xA;import numpy as np&#xA;from numpy import arange&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.decomposition import PCA&#xA;import plotly.graph_objs as go&#xA;from plotly.offline import init_notebook_mode, iplot&#xA;init_notebook_mode&#xA;&#xA;&#xA;dataset = read_excel('D:\M.tech\Dataset\Testing database1 New.xlsx')&#xA;#print(dataset.tail())&#xA;dataset.fillna(0)&#xA;cols=dataset.columns[1:]&#xA;#print(cols)&#xA;np.isnan(dataset).any()&#xA;dataset = dataset.fillna(method='ffill')&#xA;&#xA;clusters=KMeans(n_clusters=4)&#xA;dataset[&quot;clusters&quot;]=clusters.fit_predict(dataset[cols])&#xA;dataset.head(20)&#xA;&#xA;#Principal component separation to create 2 dim picture&#xA;pca=PCA(n_components=2)&#xA;dataset['x']=pca.fit_transform(dataset[cols])[:,0]&#xA;dataset['y']=pca.fit_transform(dataset[cols])[:,1]&#xA;dataset=dataset.reset_index()&#xA;print(dataset.tail())&#xA;&#xA;&#xA;trace0= go.Scatter(x=dataset[dataset.clusters == 0]['x'],&#xA;                   y=dataset[dataset.clusters == 0]['y'],&#xA;                   name=&quot;Cluster1&quot;,&#xA;                   mode =&quot;markers&quot;,&#xA;                   marker =dict(size=10,color=&quot;rgba(15,152,152,0.5)&quot;,line=dict(width=1,color=&quot;rgb(0,0,0)&quot;)))&#xA;&#xA;trace1= go.Scatter(x=dataset[dataset.clusters == 1]['x'],&#xA;                   y=dataset[dataset.clusters == 1]['y'],&#xA;                   name=&quot;Cluster2&quot;,&#xA;                   mode =&quot;markers&quot;,&#xA;                   marker =dict(size=10,color=&quot;rgba(180,18,180,0.5)&quot;,line=dict(width=1,color=&quot;rgb(0,0,0)&quot;)))&#xA;&#xA;trace2= go.Scatter(x=dataset[dataset.clusters == 2]['x'],&#xA;                   y=dataset[dataset.clusters == 2]['y'],&#xA;                   name=&quot;Cluster3&quot;,&#xA;                   mode =&quot;markers&quot;,&#xA;                   marker =dict(size=10,color=&quot;rgba(132,132,132,0.8)&quot;,line=dict(width=1,color=&quot;rgb(0,0,0)&quot;)))&#xA;&#xA;trace3= go.Scatter(x=dataset[dataset.clusters == 3]['x'],&#xA;                   y=dataset[dataset.clusters == 3]['y'],&#xA;                   name=&quot;Cluster4&quot;,&#xA;                   mode =&quot;markers&quot;,&#xA;                   marker =dict(size=10,color=&quot;rgba(230,20,30,0.5)&quot;,line=dict(width=1,color=&quot;rgb(0,0,0)&quot;)))&#xA;&#xA;data =[trace0,trace1,trace2,trace3]&#xA;&#xA;iplot(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Above code displays no error but when I run the code in ipython console, it does not display any figure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this?&lt;/p&gt;&#xA;" OwnerUserId="44688" LastEditorUserId="29575" LastEditDate="2018-01-18T15:17:28.877" LastActivityDate="2018-01-26T01:13:15.600" Title="K-means program in python for large excel database" Tags="|machine-learning|python|k-means|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26680" PostTypeId="2" ParentId="26675" CreationDate="2018-01-16T08:34:56.363" Score="1" Body="&lt;p&gt;Instead of using embeddings and doc2vec, maybe you should start by an easier and more straightforward information retrieval approach. Start by building a bag-of-words representation of your long document. Given its vocabulary $V$ of size $|V|$, you represent it as well as the short texts for the second document in a space $\mathbb{R}^{|V|}$. This will give a vector for the first document and several vectors, one for each of the lines, for the second document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Proceed by calculating the similarity (euclidean distance) of the vectors of the lines of the second document with the vector of the first document. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many ways in this line of though that would enable you to use word embeddings if you want, but I would suggest starting with the simplest approach first.  &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-16T08:34:56.363" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26681" PostTypeId="1" CreationDate="2018-01-16T08:38:00.593" Score="1" ViewCount="450" Body="&lt;p&gt;I require to develop an machine learning algorithm to predict that if a secondary car battery should be connected to load at any given time of day based on the automobile usage profile of the user. (the output is either 1 or 0 , ON or OFF)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some Background : My project car is equipped with a secondary battery which has to run a car Computer, LTE modem etc. So it is important that the battery is not connected to these loads all the time so as to extend the run time of the battery when the car is not in use( primary car battery charges the secondary battery when the car is running.) . These electronic parts are used for IOT operation, running alexa voice interaction etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is necessary to predict at what times user is using his car and switch the circuit ON before the driver actually gets in the car, So that it can start alexa voice interaction  with user as soon as he enters the car without any delay( if it starts after user gets in , it will take some time before all the internet connections are up and running.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on which existing machine learning algorithm I can use here? For the start I’m planning to predict the state based on the day, time and location only.  For example if the time is 7.am, weekday, and the location is home,  the car is usually ON to travel to office., if the time is 10.00am and the location is Office , the car is usually OFF.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help me with any  ideas/ suggestions that can also  improve the concept.&lt;/p&gt;&#xA;" OwnerUserId="44691" LastActivityDate="2020-03-22T13:28:34.490" Title="ML algorithm to predict the state of state of a device (ON/ OFF) based on day, time and location" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26682" PostTypeId="2" ParentId="26681" CreationDate="2018-01-16T09:03:20.117" Score="1" Body="&lt;p&gt;You need to predict 2 classes (on/off). The first question is: Based on what data (exactly) this happens?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem statement&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You do have some training data, so this is a &lt;em&gt;supervised learning&lt;/em&gt; task. The algorithm choice really depends on the question what data exactly to need and what kind of patterns you expect to detect:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is the prediction based on a single user (data will be scarce then) or on multiple users (enough to determine similar users for predictions)&lt;/li&gt;&#xA;&lt;li&gt;does it really suffice to consider Daytime + Week/Weekend? What about public holidays or seasons, where the usage behavior likely changes&lt;/li&gt;&#xA;&lt;li&gt;Will vacation times impact the usage behaviour beyond date/time?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Picking an algorithm&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you now based on which data you will recognise what kind of pattern you think about algorithms that fit well. For example: If you really only need time + weekday/weekend then very simple algorithms might do. Decisions trees for example (with simple rules as if weekday + 7.a.m. then turn on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If rules need to be a lot more complex (and I think this is the case here) you might need a more complex model. I have very little experience with time based predictions but things like &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;recurring neural networks&lt;/a&gt;&lt;/strong&gt; come to mind which are heavily used for time based predictions (e.g. web analytics in order to find anomalies in usage patterns for any given point in time). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Concerns&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem sounds like you need a very accurate prediction (minute-wise accuracy) while the actual user behaviour will be a bit more random (some times earlier, some times later). Many algorithms try to reduce an error which often leads to in-between values if the actual time of an event fluctuates around a mean value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When using neural networks you can influence how the error is calculated and you can train it in order to favour one kind of error (being early) over another. This makes it more complex, and requires that being &quot;too early&quot; is not an issue (or that the user is always very reliable).&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-01-16T09:56:08.730" LastActivityDate="2018-01-16T09:56:08.730" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26683" PostTypeId="1" CreationDate="2018-01-16T10:03:13.640" Score="8" ViewCount="7747" Body="&lt;p&gt;I have a Dataset of around 200 features. Most of them are categorical and only a few are numerical. It seems that an artificial neural network with an &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Autoencoder&quot;&gt;Autoencoder&lt;/a&gt; has some problems with that kind and amount of features. Therefore, I thought to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;PCA&quot;&gt;PCA&lt;/a&gt; to reduce the dimensions and then apply the Autoencoder afterwards. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the combination of a PCA before neural nets makes sense as the neural nets also reduces the information in internal layers? Does anybody has experiences with such a combination?&lt;/p&gt;&#xA;&#xA;&lt;hr/&gt;&#xA;&#xA;&lt;p&gt;Edit: There is also an interesting answer in the &lt;a href=&quot;https://stats.stackexchange.com/a/55059/179078&quot;&gt;stats forum&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="42238" LastEditorUserId="28175" LastEditDate="2020-02-26T18:20:56.107" LastActivityDate="2020-02-26T18:20:56.107" Title="Does it make sense to combine PCA with an artificial neural network?" Tags="|machine-learning|neural-network|deep-learning|predictive-modeling|pca|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="26685" PostTypeId="2" ParentId="26683" CreationDate="2018-01-16T10:53:11.390" Score="5" Body="&lt;p&gt;Neural networks are actually extremely effective at performing dimensionality reduction. A great example is &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;noreferrer&quot;&gt;word2vec&lt;/a&gt;, which applies a shallow neural network to reduce inputs on the order of several million features (i.e. unnormalized text) to a 30-150 dimensional vector via a process that is mathematically analogous to &lt;a href=&quot;https://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization&quot; rel=&quot;noreferrer&quot;&gt;matrix factorization&lt;/a&gt; (which is the class of techniques PCA belongs to). Autoencoders function very similarly to word2vec: if you're planning on using an autoencoder to learn an embedding of your data, I wouldn't expect you'd gain anything from applying PCA first rather than using an autoencoder (or something similar) to learn a better embedding for your data that isn't constrained by the assumptions of PCA.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-16T10:53:11.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26686" PostTypeId="1" AcceptedAnswerId="26689" CreationDate="2018-01-16T10:54:34.930" Score="0" ViewCount="139" Body="&lt;p&gt;I want to determine the similarity between images based on different features. The images show the same type of object (e.g. cars). I want to order images based on their similarity (e.g. through a feature vector). There are ways to solve this, for example a convolutional neural network. However, the images may be taken from slightly different perspectives: Sometimes directly from the front, sometimes with a slight ankle. What I DONT want is images ordered by their perspective rather than by the actual object. My questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What can I do to avoid primary order by perspective&lt;/li&gt;&#xA;&lt;li&gt;Do I have to expect the ankle of the picture to be a primary factor in sorting order or will it more likely be negligible?&lt;/li&gt;&#xA;&lt;li&gt;Is there a way to extract/visualise the features the network uses (unsupervised) in order to manipulate the sorting order (e.g. by saying I want to increase the weight of THIS visual feature when ordering)?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-01-16T11:39:19.073" Title="Image similarity without perspective" Tags="|image-recognition|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26687" PostTypeId="2" ParentId="26652" CreationDate="2018-01-16T10:56:13.650" Score="3" Body="&lt;p&gt;What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; x_train_tf.toarray()&#xA;array([[ 0.70710678,  0.        ,  0.70710678],&#xA;       [ 0.        ,  1.        ,  0.        ]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you check what features you are getting, you will see -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; tfidf.get_feature_names()&#xA;[u'marital', u'relationship', u'status']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What's happening is, while passing dataframe, the &lt;strong&gt;TfidfVectorizer&lt;/strong&gt; is only taking the column names and converting them into numeric form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think you need to use &lt;strong&gt;tfidf&lt;/strong&gt; here. As far as I understand, your data is categorical text, so use &lt;code&gt;pandas.get_dummies()&lt;/code&gt; instead of tfidf. This will convert your categorical data to numeric form which you can use for modelling.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; pd.get_dummies(x_train.iloc[:10,])&#xA;       marital-status_Divorced  marital-status_Married-civ-spouse  marital-status_Never-married  marital-status_Widowed  relationship_Husband  relationship_Not-in-family  relationship_Own-child  relationship_Unmarried &#xA;1521                   1                          0                              0                        0                       0                            1                         0                        0 &#xA;2274                   0                          1                              0                        0                       1                            0                         0                        0 &#xA;20209                  0                          1                              0                        0                       1                            0                         0                        0 &#xA;5529                   0                          1                              0                        0                       1                            0                         0                        0 &#xA;27639                  0                          1                              0                        0                       1                            0                         0                        0 &#xA;26670                  0                          0                              1                        0                       0                            1                         0                        0 &#xA;16635                  0                          0                              0                        1                       0                            0                         0                        1 &#xA;30824                  0                          0                              1                        0                       0                            1                         0                        0 &#xA;15181                  0                          0                              1                        0                       0                            0                         1                        0 &#xA;9119                   1                          0                              0                        0                       0                            0                         1                        0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, for demo purpose, I have taken only 10 rows.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastEditorUserId="44139" LastEditDate="2018-01-16T11:12:50.717" LastActivityDate="2018-01-16T11:12:50.717" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26688" PostTypeId="1" AcceptedAnswerId="26740" CreationDate="2018-01-16T11:00:58.210" Score="0" ViewCount="1205" Body="&lt;p&gt;I've been looking for ways to wrangle my data which contains both text and numerical attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are of course several algorithms for numerical data, but I am looking for suggestions regarding how to deal with textual data, for instance: for sorting based on &lt;code&gt;K-means&lt;/code&gt; clustering and predicting missing data using &lt;code&gt;kNN&lt;/code&gt;. I would really appreciate any thoughts regarding that. I am using &lt;code&gt;scikit-learn&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44701" LastEditorUserId="29575" LastEditDate="2018-01-16T15:06:02.640" LastActivityDate="2018-01-29T15:41:00.553" Title="Ways to convert textual data to numerical data" Tags="|scikit-learn|clustering|text-mining|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26689" PostTypeId="2" ParentId="26686" CreationDate="2018-01-16T11:39:19.073" Score="0" Body="&lt;p&gt;Generative Adverserial Networks can be tolerant of this type of perturbation. In fact, if you look at figure 8 of the &lt;a href=&quot;https://arxiv.org/pdf/1511.06434.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;DCGAN paper&lt;/a&gt;, the authors demonstrate how the GAN learned to encode perspective directly such that the authors are able to use the GAN to rotate the camera on an image by applying a &quot;turn vector&quot;. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-16T11:39:19.073" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26690" PostTypeId="2" ParentId="26678" CreationDate="2018-01-16T11:42:43.560" Score="1" Body="&lt;p&gt;First, you can read your Excel File with python to a pandas dataframe as described here: &lt;a href=&quot;https://stackoverflow.com/questions/3239207/how-can-i-open-an-excel-file-in-python&quot; title=&quot;how-can-i-open-an-excel-file-in-python&quot;&gt;how-can-i-open-an-excel-file-in-python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, you can use scikit-learn for the k-means clustering on your imported dataframe as described here:&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;KMeans&quot;&gt;KMeans&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42238" LastActivityDate="2018-01-16T11:42:43.560" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26692" PostTypeId="2" ParentId="26688" CreationDate="2018-01-16T12:45:32.703" Score="0" Body="&lt;p&gt;A renown method is using two layers Neural Network which transform each word to a vector. you can use word2vec library also to do this. To know more about this you can find &lt;a href=&quot;http://www.lifestyletrading101.com/word2vec-deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this online article&lt;/a&gt; useful.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-16T12:45:32.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26693" PostTypeId="1" AcceptedAnswerId="26704" CreationDate="2018-01-16T12:45:46.863" Score="3" ViewCount="3199" Body="&lt;p&gt;Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Text #1 : Comedy 10%, Horror 50%, Romance 1%&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Text #2 : Comedy 40%, Horror 3%, Romance 30%&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We have given keywords in each class through which we make a comparison. Below is the code that explains this scenario better&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;from sklearn.metrics.pairwise import cosine_similarity&#xA;import numpy as np&#xA;&#xA;# Comedy&#xA;keywords_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']&#xA;&#xA;# Horror&#xA;keywords_2 = ['horror', 'fear', 'shock', 'panic', 'scream']&#xA;&#xA;# Romance&#xA;keywords_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']&#xA;&#xA;text = ('funny hilarious fear passionate')&#xA;&#xA;cv1 = CountVectorizer(vocabulary = keywords_1) &#xA;data = cv1.fit_transform([text]).toarray()&#xA;vec1 = np.array(data)&#xA;vec2 = np.array([[1, 1, 1, 1, 1]])&#xA;print(cosine_similarity(vec1, vec2))&#xA;&#xA;cv2 = CountVectorizer(vocabulary = keywords_2) &#xA;data = cv2.fit_transform([text]).toarray()&#xA;vec1 = np.array(data)&#xA;vec2 = np.array([[1, 1, 1, 1, 1]])&#xA;print(cosine_similarity(vec1, vec2))&#xA;&#xA;cv3 = CountVectorizer(vocabulary = keywords_3) &#xA;data = cv3.fit_transform([text]).toarray()&#xA;vec1 = np.array(data)&#xA;vec2 = np.array([[1, 1, 1, 1, 1]])&#xA;print(cosine_similarity(vec1, vec2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem with this approach is that &lt;code&gt;vocabulary&lt;/code&gt; in &lt;code&gt;CountVectorizer()&lt;/code&gt; doesn't consider different word classes (Nouns, Verbs, Adjectives, Adverbs, plurals, etc.) of a word in a text. For example, let's say we have keywords list as below&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;keywords_1 =  [(...),&#xA;  ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),&#xA;  (...),&#xA;  ('hilarious', 'hilariously')&#xA;  (...)]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and we want to compute similarity as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cv1 = CountVectorizer(vocabulary = keywords_1) &#xA;data = cv1.fit_transform([text]).toarray()&#xA;vec1 = np.array(data) # [[f1, f2, f3, f4, f5]]) # fi is the count of number of keywords matched in a sublist&#xA;vec2 = np.array([[n1, n2, n3, n4, n5]]) # ni is the size of sublist&#xA;print(cosine_similarity(vec1, vec2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can we modify the above code to capture this scenario. Any advice is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="38492" LastActivityDate="2018-01-18T10:01:37.420" Title="How to use different classes of words in CountVectorizer()" Tags="|python|nlp|scikit-learn|text-mining|probability|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26695" PostTypeId="1" CreationDate="2018-01-16T13:54:32.110" Score="1" ViewCount="71" Body="&lt;p&gt;I am using an ANN for predicting high value customers based on their first 0, 5, 10 days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've made 3 variables: &lt;code&gt;PurchasesOnDay0&lt;/code&gt;, &lt;code&gt;PurchasesOnDay0to5&lt;/code&gt;, &lt;code&gt;PurchasesOnDay0to10&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;PurchasesOnDay0to5&lt;/code&gt; will include &lt;code&gt;PurchasesOnDay0&lt;/code&gt; and &lt;code&gt;PurchasesOnDay0to10&lt;/code&gt; will include &lt;code&gt;PurchasesOnDay0to10&lt;/code&gt; and &lt;code&gt;PurchasesOnDay0to5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, will the ANN pick up the correlation and adjust accordingly or will this be a problem when creating my model?&lt;/p&gt;&#xA;" OwnerUserId="44712" LastEditorUserId="29575" LastEditDate="2018-01-16T15:01:41.143" LastActivityDate="2018-01-16T15:21:33.300" Title="ANN Variable Correlation" Tags="|machine-learning|neural-network|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26696" PostTypeId="2" ParentId="26695" CreationDate="2018-01-16T14:03:13.163" Score="1" Body="&lt;p&gt;Generically speaking a neural network can automatically learn the needed feature transformation for a model to perform well.  What tends to matter more is how the features relate to your output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're ever bored, you can run your original matrix, a transformed matrix that has been de-correlated via a principle axis transformation, and the original matrix combined with correlation information.  I suspect performance using the same architecture will be relatively similar.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-16T15:21:33.300" LastActivityDate="2018-01-16T15:21:33.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26697" PostTypeId="2" ParentId="26683" CreationDate="2018-01-16T15:33:50.730" Score="5" Body="&lt;p&gt;&lt;code&gt;PCA&lt;/code&gt; is used to abandon having redundant features. It expands directions which your data is highly distributed in those directions. During this process, it does not care about the labels of your data. In those directions your data will be highly distributed and none of features have correlation. &lt;code&gt;PCA&lt;/code&gt; just does the preceding things. In that feature space your data may be easily separable or not. But using this before neural nets is a way to reduce the redundant features which may cause your net have too many parameters. It is a kind of pre-processing for just reducing the correlated features, although there are different reasons that one can apply &lt;code&gt;PCA&lt;/code&gt;, like data visualization or understanding data, or even reporting data based on e.g. three main components. I recommend you using that because you may find that in the new space you would classify your data with a more smaller net.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does the combination of a PCA before neural nets makes sense&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, you can do that as a pre-processing stage.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... as the neural nets also reduces the information in internal layers? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;neural nets do not necessarily reduce the information in internal layers. In convolutional nets, max-pooling layer somehow reduces the unnecessary information but other usual nets such as convolutional layers or dense layers, try to change the space of the inputs of the layers or equivalently with another interpretation they try to find other features and patterns that the data can be separated in those spaces. &lt;code&gt;PCA&lt;/code&gt; actually reduces the correlated features. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does anybody has experiences with such a combination?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. It all depends on your data. In some applications it worked for me, which I had so many correlated features. But it has happened that it does not work very well in both cases of existing or not existing of correlated features. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-16T15:44:15.857" LastActivityDate="2018-01-16T15:44:15.857" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26698" PostTypeId="1" CreationDate="2018-01-16T15:38:04.190" Score="3" ViewCount="159" Body="&lt;p&gt;I want to be able to create a model that would be able to classify an image that has been split into 9 parts and jumbled around.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did see a paper on it but it is quite old (7-8 years old). Could anyone point me towards any resources? Is building a CNN the best approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="44717" LastEditorUserId="11097" LastEditDate="2018-01-16T15:45:24.833" LastActivityDate="2020-03-31T15:02:00.353" Title="Classification of jumbled images" Tags="|image-classification|computer-vision|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26699" PostTypeId="1" AcceptedAnswerId="26950" CreationDate="2018-01-16T17:04:37.950" Score="22" ViewCount="18761" Body="&lt;p&gt;&lt;strong&gt;Issue 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am confused by the &lt;a href=&quot;https://github.com/Microsoft/LightGBM/blob/master/docs/Features.rst#references&quot; rel=&quot;noreferrer&quot;&gt;description of LightGBM&lt;/a&gt; regarding the way the tree is expanded. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;They state:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Most decision tree learning algorithms grow tree by level&#xA;  (depth)-wise, like the following image:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/e1FWe.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e1FWe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions 1&lt;/strong&gt;: Which &quot;most&quot; algorithms are implemented this way? &#xA;As far as I know C4.5 and CART use DFS. XGBoost uses BFS. Which other algorithms or packages use BFS for decision trees?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Issue 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LightGBM states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;LightGBM grows tree by leaf-wise (best-first).It will choose the leaf with max delta loss to grow. When growing same&#xA;  leaf, leaf-wise algorithm can reduce more loss than level-wise algorithm.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YOE9y.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YOE9y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2&lt;/strong&gt;: Is it correct to say that level-wise growth trees will have equal depth for all leaves? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions 3:&lt;/strong&gt; If Question 2 is not correct, then the trees from level-wise and leaf-wise growth will look the same at the end of the traversal (without pruning etc). Is it a correct statement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions 4:&lt;/strong&gt; If question 3 is correct, how can &quot;leaf-wise algorithm can reduce more loss than level-wise algorithm&quot;? Does it have to do with the post-pruning algorithm?&lt;/p&gt;&#xA;" OwnerUserId="32230" LastEditorUserId="32230" LastEditDate="2018-01-17T10:19:39.440" LastActivityDate="2022-06-23T17:13:59.767" Title="Decision trees: leaf-wise (best-first) and level-wise tree traverse" Tags="|decision-trees|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26700" PostTypeId="2" ParentId="26693" CreationDate="2018-01-16T17:59:13.557" Score="0" Body="&lt;p&gt;If you want to use for vocabulary sets like your example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keywords_1 = [&#xA;    (...), &#xA;    ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),&#xA;    (...), &#xA;    ('hilarious', 'hilariously'),&#xA;    (...)&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I advise to you use &lt;a href=&quot;http://www.nltk.org/api/nltk.stem.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;stemmer&lt;/a&gt; and delete &lt;a href=&quot;https://www.geeksforgeeks.org/removing-stop-words-nltk-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;stopword&lt;/a&gt; before it.&lt;/p&gt;&#xA;" OwnerUserId="13696" LastEditorUserId="29575" LastEditDate="2018-01-17T03:41:07.670" LastActivityDate="2018-01-17T03:41:07.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26701" PostTypeId="2" ParentId="26688" CreationDate="2018-01-16T18:03:59.393" Score="0" Body="&lt;p&gt;In basic term you actualy studying in &quot;Natural Language Processing&quot;(NLP)&#xA;I advice to you &quot;nltk&quot; library for python. And that library have a good count vectorization tool for describe a word as number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can easily delete &quot;stopwords&quot; and can easily manage &quot;stemmer&quot; function.&lt;/p&gt;&#xA;" OwnerUserId="13696" LastActivityDate="2018-01-16T18:03:59.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26704" PostTypeId="2" ParentId="26693" CreationDate="2018-01-16T18:30:02.083" Score="5" Body="&lt;p&gt;First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as &lt;a href=&quot;https://spacy.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spacy&lt;/a&gt; or &lt;a href=&quot;http://www.nltk.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NLTK&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other point to mention here is that despite the other answer, what libraries has as Stop Words list is not actually stop word! &lt;strong&gt;Do no remove them&lt;/strong&gt;! In NLP stop words should be extracted based on working corpus not based on a predefined list. In practice removing this kind of stop words usually reduces the performance on specific domain corpuses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Third point is that depending on the classifier and loss function you use, &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF&lt;/a&gt; might be better than Count Vectorizer. I suppose it works better specially if Log Loss is the cost function but I am not sure. Just give it a try.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-01-18T10:01:37.420" LastActivityDate="2018-01-18T10:01:37.420" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26705" PostTypeId="1" CreationDate="2018-01-16T18:35:45.340" Score="1" ViewCount="657" Body="&lt;p&gt;I am trying to build a RNN model to classify time series.&#xA;My time series data consist of numbers with length 200.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using tensorflow, and when i create the placeholder for the data it is like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = tf.placeholder(tf.float32, [None, 200, 1], name='inputs')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;200 is the sequence length and 1 is the features, which in my case is 1 feature per time step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And when i create the RNN using GRU cell i use this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.contrib.rnn.GRUCell(400)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I use 400 as cell size of GRU cell because as far as i know the internal state + the inputs will be used as the new input for the next GRU cell, so i suppose the cell must be input length x 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this logic of mine correct?&#xA;Is the initialization of inputs correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BATCH SIZE = None&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SEQUENCE LENGTH = 200&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FEATURES = 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And is the GRU cell parameters correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CELL SIZE = 2 x SEQUENCE LENGTH?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or have i understood it incorrectly?&lt;/p&gt;&#xA;" OwnerUserId="40182" LastActivityDate="2018-01-16T18:35:45.340" Title="RNN sequence length and cell size" Tags="|tensorflow|rnn|sequence|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26706" PostTypeId="1" CreationDate="2018-01-16T18:36:38.690" Score="1" ViewCount="108" Body="&lt;p&gt;Suppose that we perform density estimation in m-dimensional space: we estimate the value $p(a)$ for some point $a$ given observations $\{x_1, \dots, x_n \}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is known that if region $A \subset \mathbb{R}^m$ is &quot;small&quot; enough to consider density being constant on points from $A$ then we can make the following estimate:&#xA;$$ p(a) \approx \frac{k / n}{|A|} $$&#xA;where $k$ is the number of observations that lie in $A$ and $|A|$ is Lebesgue measure of $A$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let parameter $h$ be small enough to consider density as constant inside hypercube centered at $a$ with side length equal to $h$. The volume of this hypercube is equal to $h^m$ and point $x$ lies inside this hypercube iff $K(\frac{x-a}{h}) = 1$ where&#xA;$$K(u) =\cases {&#xA;      1\text{, if $|\frac{u^k - a^k}{h}| \leq \frac{1}{2}, k = 1,\dots,m$}\cr&#xA;      0\text{, otherwise}&#xA;}$$&#xA;It's easy to see that the number of observations inside this hypercube equals to&#xA;$$k = \sum_{i = 1}^{n} K(\frac{x-a}{h})$$ &#xA;and so the estimation described above gets the following form:&#xA;$$p(a) \approx \frac{1}{n h^m} \sum_{i = 1}^{n} K(\frac{x-a}{h}) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can interpret $K$ as &quot;weight&quot; given to particular observations and one of the drawbacks of hypercubic approach is that all observations lying inside hypercube have equal weights despite having different distances from $a$. Yet another drawback is that the resulting estimate is not continuous. That's what i understand to be the main reason of using non-hypercubic kernels such as gaussian kernel which give more weight to points close to $a$ and yields continuous estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But i have troubles with interpreting the usage of such kernels. The sum $\sum_{i = 1}^{n} K(\frac{x-a}{h})$ is no longer equal to $k$ so we can't justify the usage of these kernels by formula $p(a) \approx \frac{k / n}{|A|} $. Finally here are my questions: how do we justify the usage of smooth kernels? how can one interpret this usage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for any ideas.&lt;/p&gt;&#xA;" OwnerUserId="38802" LastActivityDate="2024-01-08T05:04:15.483" Title="Intuition behind using non-hypercubic kernels in density estimation" Tags="|probability|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26707" PostTypeId="2" ParentId="26706" CreationDate="2018-01-16T19:28:28.777" Score="0" Body="&lt;p&gt;Histograms and methods based on binning have a number of &lt;a href=&quot;https://stats.stackexchange.com/a/51753/127417&quot;&gt;well-known problems&lt;/a&gt;. Different anchor points etc. can introduce artificial patterns that make interpretation unreliable. Smooth kernels don't use a grid and thus smooth out the noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also has the advantage that it makes it easier to get a single overall picture of the data because it takes into account neighboring points and smooths the data into areas where no data is observed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smooth kernels can also be justified by their favorable statistical properties. Popular methods like &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0167947316300408?via%3Dihub&quot; rel=&quot;nofollow noreferrer&quot;&gt;fastKDE&lt;/a&gt; use the fact that one can find &quot;an empirical kernel that is optimal in the sense that the integrated, squared difference between the resulting KDE and the true PDF is minimized.&quot;&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-01-16T19:28:28.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26708" PostTypeId="2" ParentId="16535" CreationDate="2018-01-16T19:29:28.910" Score="1" Body="&lt;p&gt;It's not free, but it is the best option that I've found... MyFusionHelper will let you auto export any saved report in IFS to Google Sheets with one of its features. Then when you connect Data Studio the Google Sheet that is automatically updated... Boom... you get your reports. You can set update frequency and a lot more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only thing I've ever found that did this right other than programming an API connection yourself. It's not free but it's relatively inexpensive.&lt;/p&gt;&#xA;" OwnerUserId="44724" LastEditorUserId="29575" LastEditDate="2018-01-16T19:33:13.893" LastActivityDate="2018-01-16T19:33:13.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26710" PostTypeId="2" ParentId="3699" CreationDate="2018-01-16T19:58:35.087" Score="6" Body="&lt;p&gt;So you ask &lt;strong&gt;how does class imbalance affect classifier performance under different losses?&lt;/strong&gt; &#xA;You can make a numeric experiment. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do binary classification by logistic regression. However, the intuition extends on the broader class of models, in particular, neural networks. I measure performance by cross-validated ROC AUC, because it is insensitive to class imbalance. I use an inner loop of cross validation to find the optimal penalties for L1 and L2 regularization on each dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.linear_model import LogisticRegression, LogisticRegressionCV&#xA;import matplotlib.pyplot as plt&#xA;&#xA;cvs_no_reg = []&#xA;cvs_lasso = []&#xA;cvs_ridge = []&#xA;imb = [0.5, 0.4, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001]&#xA;Cs = [1e-5, 3e-5, 1e-4, 3e-4, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1]&#xA;for w in imb:&#xA;    X, y = make_classification(random_state=1, weights=[w, 1-w], n_samples=10000)&#xA;    cvs_no_reg.append(cross_val_score(LogisticRegression(C=1e10), X, y, scoring='roc_auc').mean())&#xA;    cvs_ridge.append(cross_val_score(LogisticRegressionCV(Cs=Cs, penalty='l2'), X, y, scoring='roc_auc').mean())&#xA;    cvs_lasso.append(cross_val_score(LogisticRegressionCV(Cs=Cs, solver='liblinear', penalty='l1'), X, y, scoring='roc_auc').mean())&#xA;&#xA;plt.plot(imb, cvs_no_reg)&#xA;plt.plot(imb, cvs_ridge)&#xA;plt.plot(imb, cvs_lasso)&#xA;plt.xscale('log')&#xA;plt.xlabel('fraction of the rare class')&#xA;plt.ylabel('cross-validated ROC AUC')&#xA;plt.legend(['no penalty', 'ridge', 'lasso'])&#xA;plt.title('Sensitivity to imbalance under different penalties')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/be35G.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/be35G.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that under high imbalance (left-hand side of the picture) &lt;strong&gt;L1 regularization performs better&lt;/strong&gt; than L2, and both better than no regularization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if the imbalance is not so serious (the smallest class share is 0.03 and higher), all the 3 models perform equally well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second question, &lt;strong&gt;what is a good loss function for imbalanced datasets&lt;/strong&gt;, I will answer that &lt;strong&gt;log loss is good enough&lt;/strong&gt;. Its useful property is that it doesn't make your model turn the probability of a rare class to zero, even if it is &lt;em&gt;very very&lt;/em&gt; rare.&lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2018-01-16T19:58:35.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26711" PostTypeId="2" ParentId="26706" CreationDate="2018-01-16T20:09:59.907" Score="0" Body="&lt;p&gt;If we're estimating a continious distribution's density, perhaps we should introduce an integral in here right?  A kernel estimate should be such that $\int_{-\infty}^{\infty}K(x)dx = 1$.  Therefore, it should be relatively easy to see that an estimate for $f(x)$ called $\hat{f}(x)$ should have the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\int_{-\infty}^{\infty}\hat{f}(x)dx = \frac{1}{n}\sum_{j=1}^{n}\frac{1}{h}K(\frac{x-a}{h}) $&#xA;$= \frac{1}{n}\sum_{j=1}^{n}1 = 1$.  Naturally since, the kernal and the estimate for the pdf are greater than 1, then our hat function is also a probability density function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for a bit more detail:  $\hat{f}(x)$ is usually derived from a definition of the derivative of the emperical CDF.  So instead of justifying it via the way you would a parzen window, you instead just justify it from what it means to be a pdf and what you want a good estimate for that pdf to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit:  With regards to knn and your estimator.  I think it's also important to realize that the for any fixed point the nearest neighhor estiamte is the kernel estimate.  However, it is different estimate for each point.  The kernel still remains an estimate because each individual estimate is a density so overall the kernel is a linear combination of densities.  Furthermore the coefficients for the k estimates will sum up to 1.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-16T20:24:18.473" LastActivityDate="2018-01-16T20:24:18.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26712" PostTypeId="1" CreationDate="2018-01-16T20:10:13.967" Score="1" ViewCount="379" Body="&lt;p&gt;df.head()= &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;day SBP     exercise&#xA;0   146.0   False&#xA;1   149.0   False&#xA;2   146.0   True&#xA;3   148.0   False&#xA;4   149.0   False&#xA;5   149.0   False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i am looking to plot &lt;code&gt;day&lt;/code&gt; vs &lt;code&gt;sbp&lt;/code&gt; with each data point coloured as per &lt;code&gt;exercise&lt;/code&gt;.&#xA;i used following command but couldn't get each point colored as per exercise:&#xA;&lt;code&gt;plt.plot('user','SBP', color='green', linestyle='dashed', marker='o',data=df)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can anyone help how i should go about this..?&lt;/p&gt;&#xA;" OwnerUserId="44725" LastActivityDate="2018-01-17T13:48:17.937" Title="Create continous plot using matplotlib/seaborn with specific hue" Tags="|plotting|seaborn|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-18T03:30:50.637" ContentLicense="CC BY-SA 3.0" />
  <row Id="26713" PostTypeId="1" AcceptedAnswerId="26717" CreationDate="2018-01-16T20:10:37.313" Score="3" ViewCount="20486" Body="&lt;p&gt;For a K nearest neighbors algorithm using a Euclidean distance metric, how does the algorithm compute euclidean distances when one(or all) of the features are categorical? Or does it just go by the most commonly occurring value among the neighbors?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So e.g. if the 2 features of 3 neighbors are age and gender with values:&#xA;age,gender=[ [20,M], [31,F], [23,M] ], and we need to pick the 2 nearest neighbors for a new observation [20,F], how do we convert the gender to a numeric scale to compute euclidean distances?&lt;/p&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2018-01-16T21:10:32.187" Title="How does KNN handle categorical features" Tags="|classification|scikit-learn|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26714" PostTypeId="1" AcceptedAnswerId="26716" CreationDate="2018-01-16T20:42:49.727" Score="14" ViewCount="11525" Body="&lt;p&gt;I've understood that principal component analysis is a dimensionality reduction technique i.e. given 10 input features, it will produce a smaller number of independent features that are orthogonal and linear transformation of original features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is &lt;code&gt;PCA&lt;/code&gt; by itself considered as a learning algorithm or it is a data pre-processing step.&lt;/p&gt;&#xA;" OwnerUserId="10522" LastEditorUserId="28175" LastEditDate="2018-01-16T21:43:50.073" LastActivityDate="2018-01-22T17:28:10.920" Title="Is PCA considered a machine learning algorithm" Tags="|machine-learning|pca|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26715" PostTypeId="1" CreationDate="2018-01-16T20:44:23.317" Score="2" ViewCount="3118" Body="&lt;p&gt;I'm trying to understand what's going on under the hood in Keras' LSTM using the approach taken by &lt;a href=&quot;https://github.com/greydanus/mr_london/blob/master/app/model/textgen.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/greydanus/mr_london/blob/master/app/model/textgen.py&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I have a sequential model with two layers of LSTM followed by a dense layer and the output layer. I can access weights for each layer from Keras model and am trying to use those weights to replicate the same model prediction using numpy. However, whenever I set &amp;quot;units&amp;quot; value in the code greater than 1, I get very different output compared to Keras' &amp;quot;model.predict()&amp;quot;. Also, the shape of outputs from Keras' &amp;quot;model.summary()&amp;quot; is different than what I am getting. My code is as below.&lt;/p&gt;&#xA;&lt;h1&gt;Import libraries&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Model   &#xA;from keras.layers import Input, Dense, LSTM  &#xA;import numpy as np  &#xA;import keras.backend as K  &#xA;from __future__ import division  &#xA;from numpy import array  &#xA;from keras.models import Sequential  &#xA;np.random.seed(42)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Bunch of basic functions&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def tanh(x):&#xA;    return np.tanh(x)  &#xA;&#xA;def sigmoid(x):  &#xA;    return 1 / (1 + np.exp(-x))&#xA;&#xA;def hard_sigmoid(x):&#xA;    slope = 0.2&#xA;    shift = 0.5&#xA;    x = (x * slope) + shift&#xA;    x = np.clip(x, 0, 1)&#xA;    return x&#xA;&#xA;def softmax_2D(X):&#xA;    maxes = np.amax(w, axis=1)&#xA;    maxes = maxes.reshape(maxes.shape[0], 1)&#xA;    e = np.exp(w - maxes)&#xA;    dist = e / np.sum(e, axis=1, keepdims=True)&#xA;    dist = np.float64(dist)&#xA;    return dist&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Helper function for LSTM layer&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def nsteps(xi, xf, xo, xc, hprev, Cprev, U_i, U_f, U_o, U_c):&#xA;    nsteps = xi.shape[0] # should be n long&#xA;    output = np.zeros_like(xi) # [n,1,m]&#xA;    memory = np.zeros_like(xi) # [n,1,m]&#xA;&#xA;    for t in range(nsteps):&#xA;        xi_t = xi[:,t,:] ; xf_t = xf[:,t,:] ; xc_t = xc[:,t,:] ; xo_t = xo[:,t,:] # [1,m] for all&#xA;    &#xA;        i_t = hard_sigmoid(xi_t + np.dot(hprev, U_i)) #[1,m] + [m]*[m,m] -&amp;gt; [1,m]&#xA;        f_t = hard_sigmoid(xf_t + np.dot(hprev, U_f)) #[1,m] + [m]*[m,m] -&amp;gt; [1,m]&#xA;        o_t = hard_sigmoid(xo_t + np.dot(hprev, U_o)) #[1,m] + [m]*[m,m] -&amp;gt; [1,m]&#xA;        c_t = f_t*Cprev + i_t * np.tanh(xc_t + np.dot(hprev, U_c)) #[1,m]*[m] + [1,m] * [1,m] -&amp;gt; [1,m]&#xA;        h_t = o_t * np.tanh(c_t) #[1,m]*[1,m] (elementwise)&#xA;&#xA;        output[t,:,:] = h_t ; memory[t,:,:] = c_t&#xA;        hprev = h_t # [1,m]&#xA;        Cprev = c_t # [1,m]&#xA;    return [output, memory]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Main LSTM layer&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def lstm_layer(X, units, i=0, seq=False):&#xA;    kernel_weights = model.layers[i].get_weights()[0]&#xA;    recurrent_kernel_weights = model.layers[i].get_weights()[1]&#xA;    bias = model.layers[i].get_weights()[2]&#xA;&#xA;    #Get the weights&#xA;&#xA;    # kernels&#xA;    kernel_i = kernel_weights[:, :units]&#xA;    kernel_f = kernel_weights[:, units: units * 2]&#xA;    kernel_c = kernel_weights[:, units * 2: units * 3]&#xA;    kernel_o = kernel_weights[:, units * 3:]&#xA;&#xA;    # recurrent kernel&#xA;    h_i = recurrent_kernel_weights[:, :units]&#xA;    h_f = recurrent_kernel_weights[:, units: units * 2]&#xA;    h_c = recurrent_kernel_weights[:, units * 2: units* 3]&#xA;    h_o = recurrent_kernel_weights[:, units * 3:]&#xA;&#xA;    # bias&#xA;    bias_i = bias[:units]&#xA;    bias_f = bias[units: units * 2]&#xA;    bias_c = bias[units * 2: units * 3]&#xA;    bias_o = bias[units * 3:]&#xA;&#xA;    x_i = np.dot(X,kernel_i) + bias_i&#xA;    x_f = np.dot(X,kernel_f) + bias_f&#xA;    x_c = np.dot(X,kernel_c) + bias_c&#xA;    x_o = np.dot(X,kernel_o) + bias_o&#xA;&#xA;    # Set cell &amp;amp; memory states to zeros&#xA;    h_tm1 = np.zeros((1, len(bias_i)))&#xA;    c_tm1 = np.zeros((1, len(bias_i)))&#xA;&#xA;    h_tm1_i = h_tm1&#xA;    h_tm1_f = h_tm1&#xA;    h_tm1_c = h_tm1&#xA;    h_tm1_o = h_tm1&#xA;&#xA;    [output, memory] = nsteps(x_i, x_f, x_o, x_c, h_tm1, c_tm1, h_i, h_f, h_o, h_c)&#xA;&#xA;    if seq:&#xA;        return output&#xA;    else:&#xA;         output = output[:,timesteps-1,:]&#xA;         output = output.reshape(1,units)&#xA;    return output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Now a dense layer&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def dense(X, i=0):&#xA;    W = model.layers[i].get_weights()[0]&#xA;    b = model.layers[i].get_weights()[1]&#xA;    output = np.dot(X, W) + b&#xA;    return output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Create the model now&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;data_dim = 28  &#xA;timesteps = 32  &#xA;num_classes = 2  &#xA;units = 5  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Expected input data shape: (batch_size, timesteps, data_dim)&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()  &#xA;model.add(LSTM(units, return_sequences=True, input_shape=(timesteps, &#xA;data_dim)))  &#xA;model.add(LSTM(units))  &#xA;model.add(Dense(2, activation='softmax'))  &#xA;model.compile(loss='categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Generate dummy training data&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;x_train = np.random.random((1, timesteps, data_dim))  &#xA;x_train = np.where(x_train &amp;gt; 0.5, 1.0, 0.0)  &#xA;y_train = np.random.random((1, num_classes))  &#xA;y_train = np.where(y_train &amp;gt; 0.5, 1.0, 0.0)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Generate dummy validation data&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;x_val = np.random.random((1, timesteps, data_dim))  &#xA;x_val = np.where(x_val &amp;gt; 0.5, 1.0, 0.0)  &#xA;y_val = np.random.random((1, num_classes))  &#xA;y_val = np.where(y_val &amp;gt; 0.5, 1.0, 0.0)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Fit the model&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(x_train, y_train,batch_size=1, epochs=1,validation_data=(x_val, y_val))  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Print model summary&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;print(model.summary())  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Numpy prediction model&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;def predict_i(X, units):&#xA;    print ('Shape of input to LSTM layer 1:', X.shape)&#xA;    h = lstm_layer(X, units=units, i=0, seq=True) ; X = h&#xA;    print('Shape of output from LSTM layer 1:', X.shape)&#xA;    h = lstm_layer(X, units = units, i = 1, seq=False) ; X = h&#xA;    print ('Shape of output from LSTM layer 2:', X.shape)&#xA;    h = dense(X, i = 2) ; X = h&#xA;    print ('Shape of output from dense layer:', X.shape)&#xA;    h = softmax_2D(X) ; X = h[0] &#xA;    return h&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Get prediction from numpy model&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;predict_i(x_train,units)    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;Get prediction from Keras model&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;model.predict(x_train)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Prediction from Keras' model does not match with numpy model. Any help/feedback is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="44726" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-09-11T14:03:08.627" Title="Keras LSTM: use weights from Keras model to replicate predictions using numpy" Tags="|keras|rnn|lstm|numpy|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26716" PostTypeId="2" ParentId="26714" CreationDate="2018-01-16T20:58:47.097" Score="10" Body="&lt;p&gt;It's not uncommon for someone to label it as an unsupervised technique.  You can do some analysis on the eigenvectors and that help explain behavior of the data. Naturally if your transformation still has a lot of features, then this process can be pretty hard.  Nevertheless it's possible thus I consider it machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my answer was selected (no idea why) I figured i'll add more detals.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA does two things which are equivalent.  First, and what is commonly referred, it maximizes the variances.  Secondly, it minimizes the reconstruction error by looking at pair-wised distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By looking at the eigenvectors and eigenvalues, it becomes rather simple to deduce which variables and features are contributing to the variance and also how different variables move in conjunction with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, it really depends on how you define &quot;learning&quot;.  PCA learns a new feature space that captures the characteristics of the original space.  I tend to think that can be meaningful.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it complex?  No, not really, but does that diminish it as an algorithm?  No I don't think so.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-22T17:28:10.920" LastActivityDate="2018-01-22T17:28:10.920" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26717" PostTypeId="2" ParentId="26713" CreationDate="2018-01-16T21:10:32.187" Score="4" Body="&lt;p&gt;It doesn't handle categorical features. This is a fundamental weakness of kNN.  kNN doesn't work great in general when features are on different scales. This is especially true when one of the 'scales' is a category label.  You have to decide how to convert categorical features to a numeric scale, and somehow assign inter-category distances in a way that makes sense with other features (like, age-age distances...but what is an age-category distance?).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all features are categorical, and inter-category distances are all treated as somehow equal, your job actually gets a little easier since you aren't converting between categorical and scalar scales.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-01-16T21:10:32.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26718" PostTypeId="2" ParentId="26714" CreationDate="2018-01-16T21:23:06.073" Score="6" Body="&lt;p&gt;Absolutely, it is not a learning algorithm, as you do not learn anything in PCA. &#xA;However, it can be used in different learning algorithms to reach a better performance in real, likes the most of the other dimension reduction methods.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-01-16T21:23:06.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26719" PostTypeId="2" ParentId="26714" CreationDate="2018-01-16T21:39:10.963" Score="4" Body="&lt;p&gt;PCA is used to eliminate redundant features. It finds directions which data is highly distributed in. It does not care about the labels of the data, because it is a projections which &lt;strong&gt;represents&lt;/strong&gt; data in least-square sense. Multiple Discriminant Analysis, &lt;code&gt;MDA&lt;/code&gt; try to find projections which best &lt;strong&gt;separates&lt;/strong&gt; the data. The latter considers the label and finds directions that data can be separated the best, although it has some details about the kind of decision that finds. To wrap up, &lt;code&gt;PCA&lt;/code&gt; is not a learning algorithm. It just tries to find directions which data are highly distributed in order to eliminate correlated features. Similar approaches like &lt;code&gt;MDA&lt;/code&gt; try to find directions in order to classify the data. Although &lt;code&gt;MDA&lt;/code&gt; is so much like &lt;code&gt;PCA&lt;/code&gt;, but the former is used for classification, it considers the labels, but the latter is not directly used for classification. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T05:29:45.310" LastActivityDate="2018-01-17T05:29:45.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26720" PostTypeId="2" ParentId="8546" CreationDate="2018-01-16T22:02:32.507" Score="1" Body="&lt;p&gt;If you have unordered factors with 30 levels then when your decision tree is looking to make a split then it will look through all combinations to decide how to make a split. Let's calculate that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(30 choose 1) + (30 choose 2) + ... + (30 choose 29) + (30 choose 30)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I.e. a huge number. This is computed at &lt;strong&gt;each split&lt;/strong&gt;. I'm not an expert on this but it seems like unordered factor variables need to be avoided. &lt;/p&gt;&#xA;" OwnerUserId="16916" LastEditorUserId="29575" LastEditDate="2018-01-16T22:24:00.327" LastActivityDate="2018-01-16T22:24:00.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26721" PostTypeId="1" CreationDate="2018-01-16T22:10:04.250" Score="1" ViewCount="1947" Body="&lt;p&gt;I am working on a NLP related task. I have about 150 documents, each few pages long (5/6 pages long on average). After removing stopwords and other unnecessary symbols and digits, I have about 104,000 unique words. The task at hand probably require some kind of word embedding (such as word2vec) as simple bag-of-words type approach aren't working properly. However, I am concerned about the size of the data I have. I have looked at pre-trained word embedding (GloVec), however, due to the narrow focus of the domain (manufacturing) of our texts, I am hesitating to use these pre-trained vectors. That leaves me with training our own. However, the size of our data set concerns me. Hence I am just throwing this question out there: What should be the lower bound on the size of the vocabulary that we need in order to train a word embedding model (word2vec) that will be reasonable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any response would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="3314" LastEditorUserId="3314" LastEditDate="2018-01-16T22:29:34.870" LastActivityDate="2018-05-17T01:18:46.257" Title="What is a lower bound on the vocabulary size for generating word/sentence embedding using word2vec or skip thought vectors?" Tags="|nlp|word2vec|word-embeddings|stanford-nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26722" PostTypeId="2" ParentId="26721" CreationDate="2018-01-16T22:45:51.033" Score="1" Body="&lt;p&gt;It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is  more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can set N to for example 10 or 20. This is common practice as you need a certain number of repetitions of the same word to have some meaningful results. &lt;/p&gt;&#xA;" OwnerUserId="23492" LastActivityDate="2018-01-16T22:45:51.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26723" PostTypeId="1" CreationDate="2018-01-17T01:07:50.157" Score="1" ViewCount="2184" Body="&lt;p&gt;Developing a application where sometimes when I make a query I get millions or 100 of thousands of records in the results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is , when I get these huge result sets, how do I visualize (in charts such as bar, radar, line graphs, etc.) this data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I just take a sample of the data and visualize that? Do I just go and plot million records? What is the best way to go about this ? &lt;/p&gt;&#xA;" OwnerUserId="44739" LastEditorUserId="8878" LastEditDate="2018-01-17T14:13:14.980" LastActivityDate="2018-01-18T15:45:11.787" Title="How would you visualize data that comes in the millions of records?" Tags="|visualization|plotting|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26726" PostTypeId="2" ParentId="26674" CreationDate="2018-01-17T03:06:57.587" Score="1" Body="&lt;p&gt;If I understand it correctly, the matrix which was shown(snapshot) was actually generated manually. I'm I correct?, If yes, then the process which you have done is right. For generating the Graphs/Social Network you need to transform the data into &lt;strong&gt;Source and Edges&lt;/strong&gt;(Weights too if possible but not mandatory). I think you already have it ready in that format, by which I mean you where saying that both &lt;code&gt;x and y&lt;/code&gt; to be &lt;strong&gt;Source&lt;/strong&gt;/words and &lt;strong&gt;Edges&lt;/strong&gt; as connections (0 or 1).&lt;/p&gt;&#xA;&lt;p&gt;If considering that the data is ready, then next question is&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Are you looking just for Visualizing and deriving insights?&lt;/p&gt;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you want to do some analysis on the data to find communities and see&#xA;a how are they distributed/spread within themselves or determine which is a key player etc.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Now answer with respect to question-1, you have tool named Gephi, which can give you amusing visualizations. For example you can see the &lt;a href=&quot;https://www.youtube.com/watch?v=FLiv3xnEepw&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;. Can use Tableau, have done something similar to this using Tableau and R.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/okpSj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/okpSj.jpg&quot; alt=&quot;Airport Connections&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;With respect to question-2, you can use different algorithms under &lt;a href=&quot;https://www.r-bloggers.com/summary-of-community-detection-algorithms-in-igraph-0-6/&quot; rel=&quot;nofollow noreferrer&quot;&gt;igraph&lt;/a&gt; package in R and get some output. Use different visualization tool for getting insights from the outcome. The link attached gives you idea about different algorithms available in R for performing community detection.&lt;/p&gt;&#xA;&lt;p&gt;Finally to answer the question for making the data into the sample shown by you, should be done manually. Just to let you know, generally in the life-cycle of any analytics project, It is very likely to spend most of the time in Data Preparation phase(30-50% of time) just to make sure that the data is ready, ignore this if you know this before. I mean, there is no shortcut/easy path for for preparing the data.&lt;/p&gt;&#xA;&lt;p&gt;Please go through this &lt;a href=&quot;https://datascience.stackexchange.com/questions/26454/use-cases-for-graph-algorithms-and-graph-data-structures-in-finance-and-banking/26465#26465&quot;&gt;Link&lt;/a&gt;, to know how network analysis would help us in getting good insights but it is respective to finance industry. It might help you to derive similar insights.&lt;/p&gt;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.linkedin.com/pulse/key-player-community-detection-using-graph-theory-imdb-joshua-johnson/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;, would help you to understand the scope of Network Analysis. This analysis was done by one of my friend during our course work. I was really amused by the insight which he could derive from the analysis. Probably you could also perform something similar. Just sharing this link for your reference.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-17T07:57:57.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26727" PostTypeId="1" CreationDate="2018-01-17T04:02:59.067" Score="1" ViewCount="36" Body="&lt;p&gt;For eg. I've been studying a paper on Recommender systems using collaborative deep learning and I've just started learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper revolves around the NN representation as shown below&#xA;&lt;a href=&quot;https://i.stack.imgur.com/tjJLD.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tjJLD.jpg&quot; alt=&quot;Graphical model of cdl&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JiNjn.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JiNjn.jpg&quot; alt=&quot;A two layer sdae&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hierarchical models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IgC4J.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IgC4J.jpg&quot; alt=&quot;collaborative deep learning &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5zKA1.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5zKA1.jpg&quot; alt=&quot;generalized bayesian sdae&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having the main optimization problem&#xA;&lt;a href=&quot;https://i.stack.imgur.com/psxRP.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/psxRP.jpg&quot; alt=&quot;optimization problem &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Need help interpreting it. &#xA;Here is a link to the original paper:&#xA;&lt;a href=&quot;http://www.wanghao.in/paper/KDD15_CDL.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wanghao.in/paper/KDD15_CDL.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40277" LastActivityDate="2018-01-17T04:02:59.067" Title="How to interpret long equations in Deep Learning papers" Tags="|machine-learning|neural-network|deep-learning|autoencoder|bayesian|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26728" PostTypeId="2" ParentId="26723" CreationDate="2018-01-17T05:12:15.280" Score="0" Body="&lt;p&gt;I recommend you using &lt;code&gt;PCA&lt;/code&gt;. It finds directions which data is highly distributed in. Using this procedure, the components __ new features __ will be in descending order for the eigenvalues. Each eigenvalue that has greater value than the next eigenvalues, will have much information than them. After using &lt;code&gt;PCA&lt;/code&gt; you can use its first three principal components for plotting. Each of the new features is a linear combination of the previous features. Using e.g. first three principal components will have so much information which will be representative of your data. In cases that data is not correlated, the preceding statement may not be always true but in your case that you have so many features, based on experience, you definitely have so many correlated features. For more information take a look at &lt;a href=&quot;https://support.minitab.com/en-us/minitab/18/help-and-how-to/modeling-statistics/multivariate/how-to/principal-components/interpret-the-results/key-results/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat505/node/54&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which may help you.  &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T05:27:26.643" LastActivityDate="2018-01-17T05:27:26.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26730" PostTypeId="2" ParentId="3699" CreationDate="2018-01-17T05:54:17.420" Score="2" Body="&lt;p&gt;If you have unbalanced data, at first I recommend you try to have real data. I mean do not replicate your data by hand if you don't have balanced data. You should never ever change the distribution of your data. Although it may seem that you reduce the Bayse error, your classifier won't do well in your specified application. I have two suggestions for imbalanced data:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use class weights to improve your cost function. For the rare class use a much larger value than the dominant class.&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;code&gt;F1&lt;/code&gt; score to evaluate your classifier&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For an imbalanced set of data is it better to choose an L1 or L2 regularization&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;These are for dealing with over-fitting problem. First of all you have to learn the training data to solve high bias problem. The latter is more common in usual tasks. They are just fine for imbalanced data set but consider the point that first you have to deal with high bias problem, learning the data, then deal with high variance problem, avoiding over-fitting.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T07:07:52.777" LastActivityDate="2018-01-17T07:07:52.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26731" PostTypeId="2" ParentId="26714" CreationDate="2018-01-17T07:15:31.977" Score="19" Body="&lt;p&gt;PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The basis is &lt;a href=&quot;https://en.wikipedia.org/wiki/Orthonormality&quot; rel=&quot;noreferrer&quot;&gt;orthonormal&lt;/a&gt;. This is incredibly useful if your features exhibit &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;noreferrer&quot;&gt;multicolinearity&lt;/a&gt; (two or more features are linearly dependent): applying PCA is guaranteed to give you a basis where this is no longer a problem. This procedure is known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_regression&quot; rel=&quot;noreferrer&quot;&gt;principal component regression&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The basis vectors are meaningful with respect to the spread of the data: they are the &lt;a href=&quot;https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors&quot; rel=&quot;noreferrer&quot;&gt;eigenvectors&lt;/a&gt; of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Covariance&quot; rel=&quot;noreferrer&quot;&gt;covariance matrix&lt;/a&gt;. This second property gives rise to PCA's famous utility as a dimensionality reduction technique: after rotating the data, &lt;a href=&quot;https://en.wikipedia.org/wiki/Projection_(linear_algebra)&quot; rel=&quot;noreferrer&quot;&gt;projecting&lt;/a&gt; the data onto a subset of the basis vectors associated with a significant portion of the total variance yields a lower dimensional representation that (often) retains (most of) the (interesting) structural properties of the data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So: &lt;strong&gt;is it a learning algorithm?&lt;/strong&gt; This is sort of a philosophical question. What makes something a learning algorithm? Certainly PCA isn't a &quot;supervised&quot; learning algorithm since we can do it with or without a target variable, and we generally associate &quot;unsupervised&quot; techniques with clustering. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, PCA is a preprocessing procedure. But before you write it off completely as not &quot;learning&quot; something, I'd like you to consider the following: PCA &lt;em&gt;can&lt;/em&gt; be calculated by literally taking the eigenvectors of the covariance matrix, but this is not how it's generally done in practice. A numerically equivalent and more computationally efficient procedure is to just take the &lt;a href=&quot;https://en.wikipedia.org/wiki/Singular-value_decomposition&quot; rel=&quot;noreferrer&quot;&gt;SVD&lt;/a&gt; of the data. Therefore, PCA is just a specific application of SVD, so asking if PCA is a learning algorithm is really asking if SVD is a learning algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, although you may feel comfortable writing off PCA as not a learning algorithm, here's why you should be less comfortable doing the same with SVD: it is a surprisingly powerful method for &lt;a href=&quot;https://en.wikipedia.org/wiki/Topic_model&quot; rel=&quot;noreferrer&quot;&gt;topic modeling&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;noreferrer&quot;&gt;collaborative filtering&lt;/a&gt;. The properties of SVD that make it useful for these applications are exactly the same properties that make it useful for dimensionality reduction (i.e. PCA). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SVD is a generalization of the eigendecomposition, and that too is extremely powerful even as a constrained version of SVD. You can perform &lt;a href=&quot;https://arxiv.org/abs/physics/0605087&quot; rel=&quot;noreferrer&quot;&gt;community detection&lt;/a&gt; on a graph by looking at the eigenvectors of the adjacency matrix, or determine the steady-state probabilities of a &lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_matrix&quot; rel=&quot;noreferrer&quot;&gt;markov model&lt;/a&gt; by looking at the eigenvectors of the transition matrix, which coincidentally is also essentially how &lt;a href=&quot;https://en.wikipedia.org/wiki/PageRank&quot; rel=&quot;noreferrer&quot;&gt;PageRank&lt;/a&gt; is calculated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under the hood, PCA is performing a simple linear algebra operation. But, this is exactly the same operation that underlies a lot of applications that most people wouldn't question applying the label &quot;machine learning&quot; to. This class of algorithms is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Matrix_decomposition&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;Matrix Factorization&lt;/strong&gt;&lt;/a&gt;, and even extends to sophisticated techniques like &lt;a href=&quot;https://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization&quot; rel=&quot;noreferrer&quot;&gt;word2vec&lt;/a&gt;: indeed, you can actually get word2vec-like results by literally just &lt;a href=&quot;https://ronan.collobert.com/pub/matos/2014_hellinger_eacl.pdf&quot; rel=&quot;noreferrer&quot;&gt;applying PCA to a word co-occrrence matrix&lt;/a&gt;. Generalizing again, another word for the results of PCA is an &lt;a href=&quot;https://www.tensorflow.org/programmers_guide/embedding&quot; rel=&quot;noreferrer&quot;&gt;&lt;strong&gt;embedding&lt;/strong&gt;&lt;/a&gt;. Word2vec is probably the most famous example of an embedding, but constructing embeddings (as intermediaries) is also an important component of the encoder-decoder architecture used in &lt;a href=&quot;https://machinelearningmastery.com/encoder-decoder-long-short-term-memory-networks/&quot; rel=&quot;noreferrer&quot;&gt;RNNs&lt;/a&gt; and &lt;a href=&quot;https://phillipi.github.io/pix2pix/&quot; rel=&quot;noreferrer&quot;&gt;GANs&lt;/a&gt;, which are the bleeding edge of ML research right now.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;So back to your question: is PCA a &quot;machine learning algorithm?&quot; Well, if it's not, you should be prepared to say the same about collaborative filtering, topic modeling, community detection, network centrality, and embedding models as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just because it's simple linear algebra doesn't mean it isn't also magic.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-17T16:17:25.423" LastActivityDate="2018-01-17T16:17:25.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26732" PostTypeId="2" ParentId="26723" CreationDate="2018-01-17T08:13:32.013" Score="0" Body="&lt;p&gt;Sampling is a totally good option, especially if the size of your data is bogging down the tool you're using to plot it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's not the problem, a common issue is that plotting opaque markers will show you where data is located, but will disguise density information. For example, imagine a situation where every pixel of your plotting area is associated with at least ome observation (i.e. you have a uniformly colored plot) but one pixel is actually associated with 99% of your data. A good technique for situations like this is to try to visualize the density of the data. A simple approach is to add transparency to your markers (often by adjusting the &quot;alpha&quot; parameter), or you can model the density more directly with binning (e.g. a histogram or hexgrid) or with a kernel density estimate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have discrete data, overplotting will likely be an issue but density might give you weird results. A good way to address this is to &quot;jitter&quot; your data by adding noise to one or more plotting dimensions to force your data to spread out more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have time series data, you can resample to a coarser resolution: e.g., if you have a data point for every millisecond, your data will probably be easier to visualize if you aggregate by hour, day, or week.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, you can summarize the data by plotting a model. Plot $X$ vs $E[Y|X]$ instead of $X$ vs $Y$and throw in some error bands for good measure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that said: just try plotting it first and see what happens. Your visualization tool might do some stuff under the hood to render at least some of this manual effort unnecessary. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-17T08:13:32.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26733" PostTypeId="1" AcceptedAnswerId="26741" CreationDate="2018-01-17T08:30:33.510" Score="2" ViewCount="619" Body="&lt;p&gt;I am trying to use &lt;code&gt;CountVectorizer()&lt;/code&gt; in a loop, But I am getting an unexpected result. On the other hand, if I use it outside the loop then it works fine. I believe there is some small problem with the logic.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;&#xA;keys_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']&#xA;keys_2 = ['horror', 'fear', 'shock', 'panic', 'scream']&#xA;keys_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']&#xA;&#xA;text = ('funny amusing fear passionate')&#xA;&#xA;for i in range(3):&#xA;    keys = 'keys_' + str(i+1)&#xA;    cv = CountVectorizer(vocabulary = keys) &#xA;    data = cv.fit_transform([text]).toarray()&#xA;    print(data)&#xA;&#xA;cv1 = CountVectorizer(vocabulary = keys_1) &#xA;data = cv1.fit_transform([text]).toarray()&#xA;print(data)&#xA;&#xA;cv2 = CountVectorizer(vocabulary = keys_2) &#xA;data = cv2.fit_transform([text]).toarray()&#xA;print(data)&#xA;&#xA;cv3 = CountVectorizer(vocabulary = keys_3) &#xA;data = cv3.fit_transform([text]).toarray()&#xA;print(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[0 0 0 0 0 0]]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[[0 0 0 0 0 0]]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[[0 0 0 0 0 0]]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[[1 1 0 0 0]]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[[0 1 0 0 0]]&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;[[0 0 1 0 0]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="38492" LastActivityDate="2018-01-17T11:30:27.577" Title="Getting unexpected result while using CountVectorizer()" Tags="|python|nlp|scikit-learn|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26734" PostTypeId="2" ParentId="26674" CreationDate="2018-01-17T08:58:51.883" Score="2" Body="&lt;p&gt;I don't understand if you want to build a network of authors who cite each other, or a network of papers who cite other papers (which would be a much sparser network because the coauthorship relationships won't show up as edges).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would follow a strategy similar to this one (off the top of my head):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assign ids to your papers&lt;/li&gt;&#xA;&lt;li&gt;Build 2 csv files : papers.csv, citations.csv&lt;/li&gt;&#xA;&lt;li&gt;read in 2-col csv file &quot;paper.csv&quot; as a two-column data frame:  col1: paper_id, col2: title &lt;/li&gt;&#xA;&lt;li&gt;read in 2-col csv file &quot;citations.csv&quot; as a two-column data frame:  col1: paper_id, col2: cites_id, &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With R's &lt;a href=&quot;https://github.com/igraph/igraph/&quot; rel=&quot;nofollow noreferrer&quot;&gt;igraph&lt;/a&gt; package, you can construct a  network pg (for papers_graph) &#xA;with (pseudocode)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pg &amp;lt;- igraph::graph_from_data_frame(citations)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you assign &quot;vertex attributes&quot; to the nodes in the network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pg &amp;lt;- set_vertex_attribute(pg, &quot;title&quot;, value= papers) &#xA;# same as:  V(pg)$title &amp;lt;- papers$title&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(and possibly many other attributes)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can use igraph's many functions (~200) to analyze the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For visualizations, you plot the ids, and use a diagram type which gives you the title when you move the mouse over / click on the node symbol (which is labeled only with the id to save screen space). You can use other design elements such as coloring the nodes by release year for instance).&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="14588" LastEditDate="2018-01-17T09:09:43.643" LastActivityDate="2018-01-17T09:09:43.643" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26735" PostTypeId="1" AcceptedAnswerId="26739" CreationDate="2018-01-17T09:41:17.877" Score="1" ViewCount="337" Body="&lt;p&gt;Hi all: I have a very fundamental question on how CNN works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand fully the training process as to take a bunch of images, start with random filters, convolve, activate, calculate loss, back propagate and learn weights. Fully understood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But once the training is done, the last convolution layer has the most complex and complete features like faces, ears, wheels and such filters that can get activated by full features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is so, during testing, do we need to pass our images through all the layers again? Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated? And pass that on to the fully connected layer for classification?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand there might me inconsistencies in the layers and the inputs but except that anything more important?&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-01-17T10:17:43.707" Title="Convolutional Neural networks" Tags="|neural-network|feature-extraction|convolution|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26736" PostTypeId="1" AcceptedAnswerId="43884" CreationDate="2018-01-17T09:42:39.867" Score="4" ViewCount="86" Body="&lt;p&gt;Is there an alternative to the vgam Package to do a zero truncated negativ Binomial GLM in R? &lt;/p&gt;&#xA;" OwnerUserId="40928" LastActivityDate="2020-10-10T00:03:55.427" Title="Alternative to VGAM for Zero Truncated Negativ Binomial GLM in R" Tags="|r|glm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26737" PostTypeId="2" ParentId="26622" CreationDate="2018-01-17T09:42:46.727" Score="3" Body="&lt;p&gt;The parts of your code that need to be inside the &lt;code&gt;with tf.device('/gpu:0'):&lt;/code&gt; context is the actual computation graph, where the neural network lives. All the parameters that should be updated on the GPU should be defined within there. I don't know tflearn very well, but I assume if you wrap the code from &lt;code&gt;IN 6&lt;/code&gt; to &lt;code&gt;IN 12&lt;/code&gt; inside the context manager, it should work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens is that when you define the variables (weights) and the operations in your graph is that you create them but also have to place them and save the reference. Normally this would happen inside your RAM and the variable itself is an adress to the RAM. In the case of TensorFlow, two things are different. First of all, sometimes you want to place some things or all the things on your GPU, due to some inherent advantages that it has for the types of computations that happen in TensorFlow. By using these context managers you show where you want to place them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;fit&lt;/code&gt; that you had inside your context manager will only call the training operations that are already defined on your CPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems slightly verbose to do it this way, but you get a lot of flexibility with this. You can spread your graph over multiple GPUs, or do part of the compute on the CPU because it's faster there, or you don't have enough GPU RAM for the full graph.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-01-17T09:42:46.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26739" PostTypeId="2" ParentId="26735" CreationDate="2018-01-17T10:17:43.707" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The answer is that all the layers are fully dependent on the exact features of the previous layer. The last layer is simply not capable of taking a raw image as input and outputting meaningful values. Most likely it is not even going to be able to fit to the shape of the output, it will require dozens of channels, where a colour image only has 3. If you somehow forced it to fit, then the colour image data is meaningless to it. It might still be made to output &lt;em&gt;something&lt;/em&gt;, but it would essentially be gibberish, and unrelated to the desired outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each layer is a function of the previous layer. Ignoring the details of convolution, a neural network is essentially a composition of multiple functions (let's call them $f, g, h, i, j$ for example) so that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y = j(i(h(g(f(x))))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are essentially asking here, can you just do $y = j(x)$ instead of running all those functions in sequence. And the answer is no.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-01-17T10:17:43.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26740" PostTypeId="2" ParentId="26688" CreationDate="2018-01-17T10:19:11.100" Score="0" Body="&lt;p&gt;Do you really mean textual attribute or just Categorical attribute? e.g. if an attribute has three values $a$,$b$ and $c$ it does not mean that you need to work with text but just categories. Here I assume you have really textual attribute e.g. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;| attr1: Age    | attr2: msg                            | &#xA;| ------------- |---------------------------------------|&#xA;| 45            | I do NLP                              |&#xA;| 21            | I do math                             |&#xA;| 34            | I am a mathematician who does NLP     |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case you have 2 options; either going for classic ways like Bag of Words representation of text data and counting frequency of terms (words/bi-grams/tri-grams/etc.) and see how it works, or try to shortcut the way if you have a specific corpus with specific info to be extracted e.g. in example above a &quot;Keyword Extraction&quot; after a Stemming step will give you a vocabulary of fields in each of which a person can get a value ($0,1$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For K-means, one should have in mind that it works with numerical data. So if you want to use it, first you need to embedd your text into a n-dimensional space using text embedding algorithms (word2vec, doc2vec, etc) or even using frequency terms (as they are numbers) but the problem with text embedding is that despite you want to train your own Neural Network (which needs descent amount of data) you have to use pre-trained NNs which might not work well if your corpus is about a very specific kind of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other point is that KNN algorithm is supervised and k-means is unsupervised. So be careful that you understood the underlying concepts properly otherwise you will end up with improper results.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-17T10:19:11.100" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26741" PostTypeId="2" ParentId="26733" CreationDate="2018-01-17T11:30:27.577" Score="1" Body="&lt;p&gt;The problem lies in the line: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keys = 'keys_' + str(i+1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, keys becomes a string variable while I guess you would expect it to take the value of the list you defined in the first lines..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try instead with a dictionary: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_keys= {&#xA;&quot;keys_1&quot; : ['funny', 'amusing', 'humorous', 'hilarious', 'jolly'],&#xA;&quot;keys_2&quot; : ['horror', 'fear', 'shock', 'panic', 'scream'],&#xA;&quot;keys_3&quot; : ['romantic', 'intimate', 'passionate', 'love', 'fond']&#xA;}&#xA;text = ('funny amusing fear passionate')&#xA;&#xA;for i in range(3):&#xA;    keys = my_keys['keys_' + str(i+1)]&#xA;    cv = CountVectorizer(vocabulary = keys) &#xA;    data = cv.fit_transform([text]).toarray()&#xA;    print(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-17T11:30:27.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26742" PostTypeId="1" CreationDate="2018-01-17T11:32:53.587" Score="3" ViewCount="189" Body="&lt;p&gt;I was going through some material and the term &quot;localized filters&quot; was used, can someone please explain what it means and what are the advantages of having a localized filter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT-1 &#xA;My question is in the context of the word used in NIPS 2017 tutorial on &lt;a href=&quot;https://www.youtube.com/watch?v=LvmjbXZyoP0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Geometric Deep Learning on Graphs and Manifolds&lt;/a&gt; at 7:23 it is said that &quot;In many successful instances of CNN models, these filters are localized in space. And the intuition why they have to be localized can be understood from the geometric point of view, that if I apply my filters to the inputs that are slightly deformed then many of the filter responses are going to be stable.&quot; I am not able to understand the last line.&#xA;For this purpose, I have read stuff related to localization of rings and related things in Algebra but can't relate from there as well.&lt;/p&gt;&#xA;" OwnerUserId="44755" LastEditorUserId="44755" LastEditDate="2018-01-19T09:24:48.417" LastActivityDate="2018-01-22T20:08:39.723" Title="Localised filters for CNNs" Tags="|machine-learning|deep-learning|convolution|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26743" PostTypeId="1" AcceptedAnswerId="28110" CreationDate="2018-01-17T13:00:03.880" Score="0" ViewCount="260" Body="&lt;p&gt;I need to build a live prediction engine (either using Ensemble methods/ RNN/ Keras Classifier) that could learn from historical data what kinds of users are likely to transact and, in real-time, generate transaction probabilities for current users on a DAILY basis. I do have their demographics, transactional and app activities data. There is an amount limit that can be transacted by the users in a month. How do I prepare the historical data to predict the next day if he is going to transact or not? Thanks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demographics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CustomerID | CustomerAge | Gender | Nationality | Joining Date | Loyalty Program Opt-in | Address&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1222 | 25 | M | British |  21-Apr-2016 | Yes | Old kent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Transactions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beneficiary | CustomerID| TransactionDateTime | TransactionAmount&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4534534 | 545544 | 19/02/2016 16:02:56 | 1888.85&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4353222 | 545544 | 23/02/2016 08:49:49 | 3242.43&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2453453 | 535455 | 26/02/2016 10:10:56 | 4776.57&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5353532 | 435353 | 11/02/2016 12:43:38 | 4862.16&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4532522 | 345353 | 23/02/2016 17:42:22 | 4923.54&lt;/p&gt;&#xA;" OwnerUserId="38143" LastEditorUserId="38143" LastEditDate="2018-01-17T17:15:05.343" LastActivityDate="2018-02-21T14:58:46.010" Title="Preparing data for a live prediction engine" Tags="|classification|data-mining|feature-engineering|preprocessing|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26744" PostTypeId="2" ParentId="26712" CreationDate="2018-01-17T13:48:17.937" Score="1" Body="&lt;p&gt;You mean something like this plot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for doing it is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;&#xA;day=[0,1,2,3,4,5]&#xA;SBP=[146.0, 149.0, 146.0, 148.0, 149.0, 149.0]&#xA;exercise=[False, False, True, False, False, False]&#xA;&#xA;plt.scatter(day, SBP, c=exercise, s=50)&#xA;plt.xlabel(&quot;Day&quot;)&#xA;plt.ylabel(&quot;SBP&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The color of each point is already specified in the array &quot;exercise&quot;, so you have only to let matplotlib point to it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Wgbb5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Wgbb5.png&quot; alt=&quot;Resulting plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-17T13:48:17.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26746" PostTypeId="2" ParentId="26347" CreationDate="2018-01-17T14:18:08.863" Score="4" Body="&lt;p&gt;Instead of depending on a library, you can create your own vectorized function:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tibble); library(dplyr)&#xA;&#xA;&#xA;example_data &amp;lt;- tribble(&#xA;  ~Values_2009, ~Values_2014,&#xA;  10000, 19500,&#xA;  10500, 15000,&#xA;  25000, 35500&#xA;)&#xA;&#xA;&#xA;CAGR_formula &amp;lt;- function(FV, PV, yrs = 4) {&#xA;values &amp;lt;- ((FV/PV)^(1/yrs)-1)&#xA;return(values)&#xA;}&#xA;&#xA;formula is (FV/PV)^1/n -1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;example_data %&amp;gt;% &#xA;  mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)&#xA;#&amp;gt; # A tibble: 3 x 3&#xA;#&amp;gt;   Values_2009 Values_2014      CAGR&#xA;#&amp;gt;         &amp;lt;dbl&amp;gt;       &amp;lt;dbl&amp;gt;     &amp;lt;dbl&amp;gt;&#xA;#&amp;gt; 1       10000       19500 18.170386&#xA;#&amp;gt; 2       10500       15000  9.326511&#xA;#&amp;gt; 3       25000       35500  9.162151&#xA;&#xA;# If Percentage is needed use the below mutate instead&#xA;#mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32426" LastEditorUserId="-1" LastEditDate="2018-06-14T03:54:54.690" LastActivityDate="2018-06-14T03:54:54.690" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26747" PostTypeId="1" AcceptedAnswerId="26749" CreationDate="2018-01-17T14:22:07.327" Score="2" ViewCount="8266" Body="&lt;p&gt;I've read in sklearn's documentation that we have to take special care in balancing the input for a decision tree, but it doesn't tell you what function to use. However, I've found the parameter &lt;code&gt;class_weight&lt;/code&gt;. If I use &lt;code&gt;class_weight: balanced&lt;/code&gt; as a parameter, will that mean that I can omit balancing the input by hand?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2018-01-17T14:42:19.473" Title="Does class_weight solve unbalanced input for Decision Tree?" Tags="|scikit-learn|decision-trees|class-imbalance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26748" PostTypeId="2" ParentId="16182" CreationDate="2018-01-17T14:26:56.633" Score="0" Body="&lt;p&gt;You have label of classes which are not mutually exclusive which means as the label of each sample, your data won't be in one-hot-encoding format. Each output vector may have multiple ones. In such occasions you shouldn't use soft-max as the output layer. You have to use &lt;code&gt;Sigmoid&lt;/code&gt; activation function for each neuron in the last layer. Suppose you have ten labels and for a typical movie each of them may be activated. So, in the last layer use a dense layer with ten &lt;code&gt;Sigmoid&lt;/code&gt; activation function. You can see &lt;a href=&quot;https://datascience.stackexchange.com/q/25315/28175&quot;&gt;here&lt;/a&gt; which may help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And as a side answer, each cost function that you are going to use should be categorical, because you have different categories. I respect so much for the other answer and I thank him/her for the answer, but I guess you should use categorical cost function otherwise the code won't work because your output matrix actually is a &lt;em&gt;matrix&lt;/em&gt; of samples and not a vector of samples. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T14:46:25.190" LastActivityDate="2018-01-17T14:46:25.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26749" PostTypeId="2" ParentId="26747" CreationDate="2018-01-17T14:31:23.733" Score="4" Body="&lt;p&gt;Yes you don't need to balance your train data by hand. But your test data can still be (truly) unbalanced. &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2018-01-17T14:31:23.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26750" PostTypeId="2" ParentId="26747" CreationDate="2018-01-17T14:42:19.473" Score="4" Body="&lt;p&gt;For classification problems, not just decision trees, it isn't uncommon for unbalanced classes to give overly optimistic accuracy scores.  There's a few common ways to handle this.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Resamble your data.  You can oversample the minority class or undersample the majority class.  The end goal is to balance out the data more or less.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Give your model a prior to help inform frequency.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Similar you can pass a weight argument.  The weight will penalize the classification function for misclassification of a rare positive cases.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Lastly you can modify the accuracy measurement.  A common measurement would be to use F1 statistic instead of just accuracy.  Maximize the F1 statistic by cross validation and see if it's stable during testing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I also advise you to combine these techniques. For example, give your model a prior and use the F1 score.  I think you'll find good results going this route.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-17T14:42:19.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26751" PostTypeId="1" CreationDate="2018-01-17T14:54:19.740" Score="10" ViewCount="12262" Body="&lt;p&gt;Machine learning can be divided into several areas: supervised learning, unsupervised learning, semi-supervised learning, learning to rank, recommendation systems, etc, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One such area is &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;noreferrer&quot;&gt;PU Learning&lt;/a&gt;, where only Positive and Unlabeled instances are available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many publications about this, usually involving a lot of mathematics...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When looking at the literature, I was expecting to see methods similar to &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0925231217309608&quot; rel=&quot;noreferrer&quot;&gt;self-training&lt;/a&gt; (from semi-supervised learning), where labels are adjusted gradually according to the classifier margins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think these is what practitioners from the area do, and I was unable to navigate the mathematics or to find a survey on PU learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone from the area perhaps clarify what said practitioners do? Why can they not just use a binary classifier where the negative class=unlabeled? Can negative labels exist among the unlabeled data? What is the goal and what metrics exist to evaluate said goal?&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="28175" LastEditDate="2018-01-17T15:22:11.670" LastActivityDate="2018-10-11T14:37:27.587" Title="Why positive-unlabeled learning?" Tags="|machine-learning|classification|semi-supervised-learning|" AnswerCount="4" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26752" PostTypeId="2" ParentId="10977" CreationDate="2018-01-17T15:18:36.827" Score="2" Body="&lt;p&gt;From your results I see that the network is working good: the loss function is reducing, that means that the calculated results are approaching the expected ones.&#xA;But in facts the accuracy is not, so there should be some error in the way you calculate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fact that the loss function start to oscillate at some point (~after 20000 iterations) is expected, and that is the reason why usually one lower the learning rate after some epochs, or use some other technique of annealing.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-17T15:18:36.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26753" PostTypeId="2" ParentId="26751" CreationDate="2018-01-17T15:20:36.850" Score="1" Body="&lt;p&gt;Whenever you have skewed data-set, it means that you know a typical class better than the others. In such cases it means that the data is your &lt;em&gt;knowledge&lt;/em&gt; and is not in a way that finds the minimum of the Bayes error because you don't know the distribution of other available classes and consequently you won't be able to find out whether the distribution of different classes overlap in the current feature space. and there are learning algorithms for such occasions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider an important fact here. Suppose that you have feature vectors of conditions of a nuclear company and they describe whether the company is in danger of nuclear radiation or not. In such cases it is clear that it does not happen a lot that you have infected companies so almost, all of you data has label of healthy condition. You have so much knowledge about the healthy class but you don't know much about the infected class because you don't have much data; consequently, you don't know its distribution and you can not estimate it well. Whenever your data is skewed, it means that e.g. you have 1 million feature vectors of negative class and 5 feature vectors of positive class if there is any.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I quote from &lt;a href=&quot;https://datascience.stackexchange.com/a/25413/28175&quot;&gt;here&lt;/a&gt; that &lt;em&gt;in statistical learning there is something called Bayes Error, whenever the distribution of classes overlap, the ratio of error is large. without changing the features, the Bayes error of the current distributions is the best performance and can not be reduced at all&lt;/em&gt; If the number of samples of each class is equal. In anomaly problems, this is not possible. You can not find balanced samples for each class.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-17T15:34:57.760" LastActivityDate="2018-01-17T15:34:57.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26754" PostTypeId="2" ParentId="10977" CreationDate="2018-01-17T15:26:00.573" Score="1" Body="&lt;p&gt;It seems that after some epochs your training oscillates. I guess the reason is that the learning rate is high. Try to set the decay parameter of &lt;code&gt;Adam&lt;/code&gt; optimizer to a number more than one and resume training. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-17T15:26:00.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26755" PostTypeId="1" AcceptedAnswerId="26761" CreationDate="2018-01-17T15:45:06.247" Score="13" ViewCount="12301" Body="&lt;p&gt;Consider a Convolutional Neural Network (CNN) for image classification. In order to detect local features, weight-sharing is used among units in the same convolutional layer. &#xA;In such a network, the kernel weights are updated via the backpropagation algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An update for the kernel weight $h_j$ in layer $l$ would be as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h_j^l = h_j^l - \eta \cdot \frac{\delta R}{\delta h_j^l} = h_j^l - \eta \cdot \frac{\delta R}{\delta x_j^{L}} \cdot \frac{\delta x_j^{L}}{\delta x_j^{L - 1}} \cdot ... \cdot \frac{\delta x_j^{l}}{\delta h_j^l}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can the kernel weights be updated and still be the same (=shared)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 2 possible explanations: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Weights of the same layer, which are initialized to the same value, will stay the same (independently of the input). This would imply that the expression $\frac{\delta R}{\delta h_j^l}$ is the same for all of these weights $h_1^l$ to $h_J^l$. This does not make sense, since $x_j^l$ is different for different j's. Or am I missing something here?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;There is a trick, e.g. after the back-propagation update, the shared weights are set to their mean.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;The confusion I had was that I didn't consider that if a weight is shared, its parameter $h_j^l$ appears several times in the loss function. When differentiating for $h_j^l$, several terms (considering the according inputs) will &quot;survive&quot;. Therefore the updates will be the same.&lt;/p&gt;&#xA;" OwnerUserId="44768" LastEditorUserId="44768" LastEditDate="2018-01-18T10:30:38.743" LastActivityDate="2018-04-27T08:46:30.163" Title="CNN - How does backpropagation with weight-sharing work exactly?" Tags="|neural-network|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26756" PostTypeId="2" ParentId="23669" CreationDate="2018-01-17T15:48:57.693" Score="2" Body="&lt;p&gt;What I see is that there is a big variance in your data. This means  that without a sufficient amount of samples (bigger than 315) it will be simply impossible to describe the dataset for any model you can think.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if the variance is not so big:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can try a data augmentation technique.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Have you shuffled the data before separating it in training/validation set? Maybe is a banal operation, but if in the validation set you have samples which are too much different from the training set, they will be never correctly predicted.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Are the classes quite unbalanced? If so, maybe you have too few samples of one of them, and in that case you can try some upsamplig technique.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="29575" LastEditDate="2018-01-17T16:46:29.560" LastActivityDate="2018-01-17T16:46:29.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26757" PostTypeId="1" AcceptedAnswerId="26964" CreationDate="2018-01-17T15:53:38.397" Score="5" ViewCount="600" Body="&lt;p&gt;When i look at the available datasets in &lt;a href=&quot;https://www.openml.org&quot; rel=&quot;noreferrer&quot;&gt;https://www.openml.org&lt;/a&gt; i often see a BNG dataset with no further information about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explane what BNG means in this context?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am especially interested in this dataset: &lt;a href=&quot;https://www.openml.org/d/1389&quot; rel=&quot;noreferrer&quot;&gt;https://www.openml.org/d/1389&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone more information about where this data set comes from?&lt;/p&gt;&#xA;" OwnerUserId="44771" LastActivityDate="2018-01-23T16:37:29.837" Title="What does BNG stand for" Tags="|dataset|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26758" PostTypeId="2" ParentId="26751" CreationDate="2018-01-17T16:00:15.337" Score="3" Body="&lt;p&gt;A couple of points I have since found myself:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I was right in suspecting that self-training could be used for PU learning. In fact, I found &lt;a href=&quot;https://www.cs.uic.edu/~liub/NSF/PSC-IIS-0307239.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the original paper on PU Learning&lt;/a&gt;, and indeed the paper is a variation on self-training. (Oddly enough, the original authors had Positive, Unlabeled and Negative examples!)&lt;/li&gt;&#xA;&lt;li&gt;The authors &lt;a href=&quot;https://www.eecis.udel.edu/~vijay/fall13/snlp/lit-survey/PositiveLearning.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;of this survey&lt;/a&gt; identify three families of methods: (i) two-step strategy (identify reliable negative examples in the unlabeled data and then use supervised learning), (ii) weight the positive and unlabeled&#xA;examples, and estimate the conditional probability of positive label given an example (I believe this is akin to semi-supervised &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0925231217309608&quot; rel=&quot;nofollow noreferrer&quot;&gt;self-training&lt;/a&gt;), and (iii) just treat the unlabeled data as highly noisy negative data.&lt;/li&gt;&#xA;&lt;li&gt;There are also some interesting loss functions to be used with neural networks (and I imagine could be adapted for gradient boosting) described in &lt;a href=&quot;https://arxiv.org/abs/1703.00593&quot; rel=&quot;nofollow noreferrer&quot;&gt;Table 1 of this paper&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="16853" LastActivityDate="2018-01-17T16:00:15.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26759" PostTypeId="2" ParentId="23669" CreationDate="2018-01-17T16:57:10.540" Score="2" Body="&lt;p&gt;Try a random forest, or a regularized model like lasso instead. They're less susceptible to overwriting. RF has the added benefit that you can use the OOB estimate to evaluate performance, so you don't have to slice up your tiny dataset into train and test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boosting generally isn't a good choice for small data and tends to overfit as you've observed. If you insist on boosting, &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.949.552%26rep%3Drep1%26type%3Dpdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;robust boosting&lt;/a&gt; is a modified variant that is specifically designed to work better on small datasets.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-17T17:02:45.473" LastActivityDate="2018-01-17T17:02:45.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26760" PostTypeId="2" ParentId="23906" CreationDate="2018-01-17T17:07:40.397" Score="1" Body="&lt;p&gt;The easiest way (depending on the scale we're talking about) is to set &lt;code&gt;n_jobs=-1&lt;/code&gt; for algorithms that support parallelization (e.g. random forest,  cross validation, grid search). This will take advantage of all the cores on your machine. If that's not good enough, you should probably move to spark. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-17T17:07:40.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26761" PostTypeId="2" ParentId="26755" CreationDate="2018-01-17T18:07:26.753" Score="13" Body="&lt;p&gt;I think you're misunderstanding what &quot;weight sharing&quot; means here. A convolutional layer is generally comprised of many &quot;filters&quot;, which are usually 2x2 or 3x3. These filters are applied in a &quot;sliding window&quot; across the entire layer's input. The &quot;weight sharing&quot; is using fixed weights for this filter across the entire input. It does not mean that all of the filters are equivalent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be concrete, let's imagine a 2x2 filter $F$ striding a 3x3 input $X$ with padding, so the filter gets applied 4 times. Let's denote the unrolled filter $\beta$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X = \begin{bmatrix}&#xA;    x_{11}       &amp;amp; x_{12} &amp;amp; x_{13} \\&#xA;    x_{21}       &amp;amp; x_{22} &amp;amp; x_{23} \\&#xA;    x_{31}       &amp;amp; x_{32} &amp;amp; x_{33} &#xA;\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F = \begin{bmatrix}&#xA;    w_{11}       &amp;amp; w_{12} \\&#xA;    w_{21}       &amp;amp; w_{22} &#xA;\end{bmatrix}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta= [w_{11}, w_{12}, w_{21}, w_{22} ]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$F*X = \begin{bmatrix}&#xA;\beta \cdot [x_{11}, x_{12}, x_{21}, x_{22}] &amp;amp; \beta \cdot [x_{12}, x_{13}, x_{22}, x_{23}] \\&#xA;\beta \cdot [x_{21}, x_{22}, x_{31}, x_{32}] &amp;amp; \beta \cdot [x_{22}, x_{23}, x_{32}, x_{33}]&#xA;\end{bmatrix}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Weight sharing&quot; means when we apply this 2x2 filter to our 3x3 input, we reuse the same four weights given by the filter across the entire input. The alternative would be each filter application having its own set of inputs (which would really be a separate filter for each region of the image), giving a total of 16 weights, or a dense layer with 4 nodes giving 36 weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sharing weights in this way significantly reduces the number of weights we have to learn, making it easier to learn very deep architectures, and additionally allows us to learn features that are agnostic to what region of the input is being considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: To further motivate this, here's an animation of a 3x3 filter applied to a 5x5 input&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cWe7D.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cWe7D.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-17T21:46:39.780" LastActivityDate="2018-01-17T21:46:39.780" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26762" PostTypeId="2" ParentId="26615" CreationDate="2018-01-17T20:54:12.460" Score="3" Body="&lt;p&gt;If you would skip this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test , y_train,y_test = train_test_split(X, Y, test_size = 0.3, random_state = 100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and change the method parameters to train your model like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf_entropy.fit(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it should work as expected.&lt;/p&gt;&#xA;" OwnerUserId="44372" LastActivityDate="2018-01-17T20:54:12.460" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26763" PostTypeId="1" AcceptedAnswerId="26768" CreationDate="2018-01-17T22:26:54.913" Score="3" ViewCount="8172" Body="&lt;p&gt;I'm trying to tackle a classification problem with a neural net tensor using flow.&#xA;I have some continuous variable features and some categorical features. &#xA;The continuous features are normalized using sklearn's StandardScaler.&#xA;For the categorical features I am using a series of embedding features that I'm concatenating together with my continuous features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The embedding features are created like so :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; airline = tf.feature_column.categorical_column_with_hash_bucket(&#xA;        'AIRLINE', hash_bucket_size=10)  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; tf.feature_column.embedding_column(airline, 8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I am having trouble choosing my &lt;strong&gt;&lt;em&gt;embedding feature output size&lt;/em&gt;&lt;/strong&gt;. I understand this transforms my sparse one hot encode &quot;airline&quot; feature into a float vector of size 8. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there a heuristic I can use to choose an embedding feature size ?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My neural net's accuracy remains stuck at 31%. It doesn't seem to be learning even after a 100 epochs. Could the size of the embedding features be a cause for such a behaviour ? &lt;/p&gt;&#xA;" OwnerUserId="37236" LastActivityDate="2018-01-18T03:33:08.853" Title="Choosing an embedding feature dimension" Tags="|neural-network|embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26766" PostTypeId="2" ParentId="26723" CreationDate="2018-01-17T23:21:21.590" Score="0" Body="&lt;p&gt;If you must plot raw values, use a random sampling strategy or some form of decimation (every nth value). Otherwise, computing and plotting summary statistics will be orders of magnitude faster. While lossy, careful attention to variability around the metrics will help you understand the form of the raw data&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-01-17T23:21:21.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26767" PostTypeId="1" CreationDate="2018-01-17T23:21:49.123" Score="2" ViewCount="65" Body="&lt;p&gt;I have a multiclass(7 labels) classification problem implemented in MLP. I am trying to classify 7 cancers based on some data. The overall accuracy is quite low, around 58%. However, some of the cancers is classified with accuracy around 90% for different parameters. Below cancer 1,2,3 etc means different types of cancer, For example 1= breast cancer, 2 = lung cancer etc. Now, for different parameter settings I get different classification accuracy. For example,&lt;/p&gt;&#xA;&lt;h1&gt;1. hyper parameters&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;    learning_rate = 0.001&#xA;    training_epochs = 10&#xA;    batch_size = 100&#xA;    hidden_size = 256&#xA;    #overall accuracy 53%, cancer 2 accuracy 91%, cancer 5 accuracy 88%, &#xA;     #cancer 6 accuracy 89% &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;h1&gt;2. hyper parameters&lt;/h1&gt;&#xA;&lt;pre&gt;&lt;code&gt;    learning_rate = 0.01&#xA;    training_epochs = 30&#xA;    batch_size = 100&#xA;    hidden_size = 128&#xA;    #overall accuracy 56%, cancer 2 accuracy 86%, cancer 5 accuracy 93%, &#xA;    #caner 6 accuracy 75%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As you can see, for different parameter settings I am getting totally different results. Cancer 1,3,4,7 have very low accuracy, so I excluded them. But cancer 2, 5,6 have comparatively better results. But, for cancer 6, the results vary by great number depending on the parameter settings.&lt;/p&gt;&#xA;&lt;p&gt;An important note is, here overall accuracy is not important but if I can classify 2-3 cancers with more than 90% accuracy that is more important. So my question is, how do I interpret the results? In my paper how should I show the results? which parameter settings should I show/use? Or should I show different parameter settings for different cancer types? So basically, how to handle this type of situations?&lt;/p&gt;&#xA;" OwnerUserId="35812" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2022-04-20T06:01:45.147" Title="Deep learning: parameter selection and result interpretation" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26768" PostTypeId="2" ParentId="26763" CreationDate="2018-01-18T01:16:03.780" Score="4" Body="&lt;p&gt;I think post below is a good resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, all categorical variable is initially converted to one-hot encoding, then layer size defined by &lt;code&gt;dimension&lt;/code&gt; argument is stacked on top of one-hot encoding; thus learning optimal representation of categorical variable based on specified dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is general rule in the blog post to take the 4th root of the number of categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to perform MDS to inspect your categorical variables to decide dimensions.&lt;/p&gt;&#xA;" OwnerUserId="2869" LastEditorUserId="2869" LastEditDate="2018-01-18T03:33:08.853" LastActivityDate="2018-01-18T03:33:08.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26769" PostTypeId="1" CreationDate="2018-01-18T02:22:39.100" Score="1" ViewCount="149" Body="&lt;p&gt;This is to seek career advice for a data scientist. What pertains within the role of data scientist and what does not regarding ElasticSearch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does backend development for ElasticSearch using languages/frameworks such as Node.js, Django etc fall under the role of data scientist, or does it come under software engineering/hard code backend development? I believe there are many data scientists in this forum who work on ElasticSearch, so I thought of checking if anyone has also expanded their role as a backend developer for ElasticSearch.&lt;/p&gt;&#xA;" OwnerUserId="13100" LastEditorUserId="26562" LastEditDate="2018-01-18T04:24:15.180" LastActivityDate="2018-01-18T04:24:15.180" Title="ElasticSearch for data scientists" Tags="|beginner|information-retrieval|reference-request|career|self-study|" AnswerCount="0" CommentCount="0" ClosedDate="2018-01-28T21:49:37.393" ContentLicense="CC BY-SA 3.0" />
  <row Id="26772" PostTypeId="1" CreationDate="2018-01-18T06:20:20.717" Score="12" ViewCount="8243" Body="&lt;p&gt;I know that topic modeling and clustering are related, but not similar techniques. Can anyone suggest what are the main differences?&lt;/p&gt;&#xA;" OwnerUserId="32220" LastActivityDate="2018-01-18T08:20:22.270" Title="What is the difference between topic modeling and clustering?" Tags="|clustering|unsupervised-learning|topic-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26773" PostTypeId="2" ParentId="26772" CreationDate="2018-01-18T08:20:22.270" Score="10" Body="&lt;p&gt;The purpose of topic modeling methods is to &lt;em&gt;discover&lt;/em&gt; the latent themes (topics) assumed to have generated the documents of a corpus. Topic modeling methods are built on the distributional hypothesis, suggesting that similar words occur in similar contexts. To this end, they assume a generative process (a sequence of steps), which is  a set of assumptions that describe how the documents are generated. &lt;em&gt;Given&lt;/em&gt; the assumptions of the generative process, inference is done, which results in learning the latent variables of the model. For instance, for Latent Dirichlet Allocation, this is the per-topic document distributions and the per-word document distributions. In this sense, a document can be represented by its per-topic distribution (doc$_1$ = 0.3$\times$&lt;em&gt;Sports&lt;/em&gt; + 0.7$\times$&lt;em&gt;Cinema&lt;/em&gt;). This later can be seen as a soft clustering approach, i.e., doc$_1$ belongs 30% in cluster &lt;em&gt;Sports&lt;/em&gt; and 70% in &lt;em&gt;Cinema&lt;/em&gt;. But topic models are not solely clustering methods, as can also been used for understanding, exploring, visualizing a collection. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, clustering methods aim at &lt;em&gt;partitioning&lt;/em&gt; data into coherent groups. Of course, what is coherent and how the partitioning is performed differs between the various clustering algorithms. The distance between the data instances is central for clustering methods and for this the instances can be represented in various ways: for documents this can be term frequencies (tf), tf-idf, and even with the per-document topic distributions learned by topic models.  &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-18T08:20:22.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26774" PostTypeId="1" CreationDate="2018-01-18T09:35:56.097" Score="4" ViewCount="196" Body="&lt;p&gt;If I wanted to determine the parameters of a deterministic algorithm that optimise some performance measure, there are any number of well-known techniques I could try. My problem concerns an algorithm whose behaviour is stochastic. Mathematically, the aim is to maximise $\mathbb{E}f(\mathbf{x})$, where for any $\mathbf{x}$ a random value of $f(\mathbf{x})$ is calculable but $\mathbb{E}f(\mathbf{x})$ is not (except by averaging empirical values of $f(\mathbf{x})$). My efforts with gradient descent and an EA have failed, and I think it's because $f$ is too noisy. (Sampling $f$ many times at the same $\mathbf{x}$ would give a less noisy average, but is too computationally expensive.) Are there any algorithms designed for this situation?&lt;/p&gt;&#xA;" OwnerUserId="44804" LastEditorUserId="44804" LastEditDate="2018-01-18T09:51:43.200" LastActivityDate="2022-08-22T16:07:40.777" Title="Optimizing random function" Tags="|machine-learning|optimization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26775" PostTypeId="1" CreationDate="2018-01-18T10:02:49.040" Score="7" ViewCount="27496" Body="&lt;p&gt;In SKLearn's documentation on Decision Trees, they say we should pay special attention not to overfit the tree. How can we do this? I am aware that using random forests may prevent it, but how do I generally tell if it's overfitting? Can you tell by an accuracy score? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is, for example, a 0.99 accuracy score an indicator of overfitting? Would in this case 0.95 mean not overfitting? What are some best practices in addition to a balanced input to avoid overfitting (especially with using SKLearn)?&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="11097" LastEditDate="2018-01-18T10:11:29.113" LastActivityDate="2018-01-18T10:22:20.580" Title="How to prevent/tell if Decision Tree is overfitting?" Tags="|scikit-learn|decision-trees|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26776" PostTypeId="1" CreationDate="2018-01-18T10:05:28.857" Score="1" ViewCount="34755" Body="&lt;p&gt;What would be a good way to go around finding the best depth for a DecisionTree (in SKLearn)? How can I tell if I've gone too deep and am overfitting? I know I can find the best parameters with f.e. GridSearchCV, but the best score might not mean I get the best classifier as I may overfit the tree to the data.&lt;/p&gt;&#xA;" OwnerUserId="21254" LastActivityDate="2018-01-18T16:15:16.667" Title="How to calculate ideal Decision Tree depth without overfitting?" Tags="|scikit-learn|decision-trees|parameter-estimation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26777" PostTypeId="2" ParentId="26775" CreationDate="2018-01-18T10:10:08.437" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can you tell by an accuracy score?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A general notion of gauging an overfit or an underfit is via &lt;a href=&quot;http://scikit-learn.org/stable/modules/learning_curve.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;validation curves&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How can we do this?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not just a decision tree, (almost) every ML algorithm is prone to overfitting. One needs to pay special attention to the parameters of the algorithms in sklearn(or any ML library) to understand how each of them could contribute to overfitting, like in case of decision trees it can be the depth, the number of leaves, etc.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I am aware that using random forests may prevent it&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Just a pointer that the concept of ensembling helps with better generalization of the model (which helps keep overfit in check).&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2018-01-18T10:10:08.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26778" PostTypeId="2" ParentId="26775" CreationDate="2018-01-18T10:15:09.947" Score="5" Body="&lt;p&gt;Overfitting meaning your model is learning the noise from the data and its ability to generalize the results is very low. In this case you have a small training error but very large validation error. If you inspect (e.g. by plotting) the evolution of training and validation errors, you see that training error is always going down but validation error is goes up at some point. That is the point you need to stop training to avoid overfitting. I strongly recommend you to read &lt;a href=&quot;https://datascience.stackexchange.com/questions/9758/overfitting-underfitting-with-data-set-size/9760#9760&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the 0.98 and 0.95 accuracy that you mentioned could be overfitting and could not! The point is that you also need to check the validation accuracy beside them. If validation accuracy is falling down then you are on overfitting zone!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are some best practices in addition to a balanced input to avoid&#xA;  overfitting?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is called &lt;em&gt;Prunning&lt;/em&gt;. Beside general ML strategies to avoid overfitting, for decision trees you can follow pruning idea which is described (more theoretically) &lt;a href=&quot;http://www.uni-weimar.de/medien/webis/teaching/lecturenotes/machine-learning/unit-en-decision-trees-pruning.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; and (more practically) &lt;a href=&quot;https://www.ismll.uni-hildesheim.de/lehre/ml-08w/skript/decision_trees2.pdf&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. In SciKit-Learn, you need to take care of parameters like depth of the tree or maximum number of leafs.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-01-18T10:22:20.580" LastActivityDate="2018-01-18T10:22:20.580" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26779" PostTypeId="1" CreationDate="2018-01-18T10:22:11.903" Score="0" ViewCount="49" Body="&lt;p&gt;I have been doing ML for sometimes and have explored and implemented Deep Learning as for some projects. ML algorithms assume a certain pattern (Geometric/Bayesian/Distribution) in the data .This can usually be referred as Hypothesis Space in the literature, then we design a solution to best fit that hypothesis to our data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to know are there some patterns (Hypothesis spaces), which do not yet have a reliable algorithm to be mined? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like Decision Trees used the Information Theory to assume that Information Gain from a variable can help us better classify the data. This was a new approach than SVM which assumes a geometrical space to where a plane can classify the data. I hope there must me some new theories which are being worked on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if I can improve this question.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2018-03-20T00:36:47.610" Title="Is there any ongoing research on known patterns ML algorithms can not mine?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26781" PostTypeId="1" CreationDate="2018-01-18T10:46:06.630" Score="0" ViewCount="364" Body="&lt;p&gt;I have data that has the shape of a step function, which is obstructed by noise. I would like to identify the sections with constant slope in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The noise does not necessarily have 0 mean, and the steps are not always abrupt. Also, the points on the x-axis are not equidistantly spaced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about trying one-sided differentiators (like &lt;a href=&quot;http://www.holoborodko.com/pavel/numerical-methods/numerical-derivative/smooth-low-noise-differentiators/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) but not sure if they would work with non-equidistantly spaced data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like some sort of a global approach based on optimization would be an elegant way to solve that but so far I couldn't come up with one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Y9q3p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y9q3p.png&quot; alt=&quot;Example function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m5i5P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m5i5P.png&quot; alt=&quot;Zoomed in&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29912" LastEditorUserId="29912" LastEditDate="2018-01-18T13:01:13.907" LastActivityDate="2018-01-19T10:01:23.927" Title="Detect constant (zero-slope) sections in a noisy step function" Tags="|time-series|algorithms|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26783" PostTypeId="1" AcceptedAnswerId="26786" CreationDate="2018-01-18T11:30:18.233" Score="5" ViewCount="28079" Body="&lt;p&gt;There are total 50 students(john, Roy..) and used some action to do a job. My dataSet something like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    John    Roy    Micheal   Ron  ....... Smith&#xA;     A       B        B       A            C&#xA;     A       A        C       B            B&#xA;     C       A        A       B            B&#xA;     .       .        .       .            .&#xA;     .       .        .       .            .&#xA;     .       .        .       .            .&#xA;     F       G        E       A            G&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here A,B,C...G are strings. Final data is like this...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;         A   B   C   D   E   F   G&#xA; John    3  34  23  34   4   3   1&#xA; Roy     5  23  12  3    5   39  46&#xA; ...................................&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This means John used &quot;A&quot; 3 times whereas Roy used &quot;A&quot; 5 times.&#xA;So based on their action I want to cluster them (i.e. the student used the same number of activities will be in the same group). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which clustering method can be used and how ?&lt;/p&gt;&#xA;" OwnerUserId="44808" LastActivityDate="2018-01-18T12:14:45.580" Title="Clustering for multiple variable" Tags="|python|r|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26784" PostTypeId="1" AcceptedAnswerId="26831" CreationDate="2018-01-18T11:35:58.380" Score="1" ViewCount="9630" Body="&lt;p&gt;&lt;strong&gt;Using TF backend&lt;/strong&gt;, I need to construct a similarity matrices of two 3D vectors, both with shape (batch_size, N, M), being N and M natural numbers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function &lt;code&gt;tf.losses.cosine_distance&lt;/code&gt; is only between 1D tensors. I need to build a Tensor matrix batch_sizexNxM such that matrix[k][i][j] will be the cosine similarity of the Tensor1[k][i] and Tensor2[k][j].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this?&lt;/p&gt;&#xA;" OwnerUserId="44785" LastEditorUserId="44785" LastEditDate="2018-01-23T19:24:44.027" LastActivityDate="2018-01-23T19:24:44.027" Title="Keras - Computing cosine similarity matrix of two 3D tensors" Tags="|machine-learning|python|deep-learning|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26785" PostTypeId="2" ParentId="26776" CreationDate="2018-01-18T11:36:55.380" Score="5" Body="&lt;p&gt;No! the best score on validation set means you are not in overfitting zone. As explained in my previous answer to your question, overfitting is about high score on training data but low score on validation. So to avoid overfitting you need to check your score on Validation Set and then you are fine. There is no theoretical calculation of the best depth of a decision tree &lt;strong&gt;to the best of my knowledge&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So here is what you do: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Choose a number of tree depths to start a for loop (try to cover whole area so try small ones and very big ones as well)&lt;/li&gt;&#xA;&lt;li&gt;Inside a for loop divide your dataset to train/validation (e.g. 70%/30%)&lt;/li&gt;&#xA;&lt;li&gt;Each time train your decision tree with that depth on training data and test it on the validation set, then keep the validation error (you can also keep the training error)&lt;/li&gt;&#xA;&lt;li&gt;Plot the validation error (you can combine it with evolution of training error to have a prettier plot for understanding!) &lt;/li&gt;&#xA;&lt;li&gt;Find the global minimum of validation error.&lt;/li&gt;&#xA;&lt;li&gt;Then you can narrow your search in a new for loop according to the value you found to reach a more precise value&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-18T11:36:55.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26786" PostTypeId="2" ParentId="26783" CreationDate="2018-01-18T12:14:45.580" Score="12" Body="&lt;h2&gt;K-means&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Your data has $7$ dimensions so k-means is worth to try. See the &lt;a href=&quot;http://sebastianraschka.com/Articles/2014_pca_step_by_step.html&quot; rel=&quot;noreferrer&quot;&gt;PCA&lt;/a&gt; of your data and check if any cluster is visible there as K-means will have a tough time if clusters are not Gaussian. the setup is: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each person is a point in $7D$ space (a $50\times7$ matrix)&lt;/li&gt;&#xA;&lt;li&gt;Apply PCA and inspect it.&lt;/li&gt;&#xA;&lt;li&gt;If different clusters visible then you will have a result&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Fuzzy C-Means&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I would suggest soft clustering algorithms. Soft clustering means that output is not binary (each sample belong only to one cluster and does not belong to others) but it assigns a membership score for belongness of each sample to each cluster. It minimizes the same objective function as K-means but with a &lt;em&gt;weight&lt;/em&gt; which is calculated in each iteration and can be found &lt;a href=&quot;https://en.wikipedia.org/wiki/Fuzzy_clustering#Fuzzy_C-means_clustering&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. Libraries usually have this algorithm under the name &lt;strong&gt;FCM&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;K-means Through GMM&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Another soft version of K-means is called Gaussian Mixture Models in which you try to estimate Gaussian kernels whose superposition describes the data (as you see again the Gaussian distribution of data is crucial here). The setup is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Choose a $k$ as the initial number of clusters and produce $k$ arbitrary Gaussian kernels (i.e. with arbitrary $\mu$ and $\sigma$)&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://people.csail.mit.edu/dsontag/courses/ml12/slides/lecture21.pdf&quot; rel=&quot;noreferrer&quot;&gt;Expectation Maximization algorithm&lt;/a&gt; for updating the new clustering structure at each step.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Spectral Clustering&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Define a &lt;a href=&quot;https://en.wikipedia.org/wiki/Similarity_measure&quot; rel=&quot;noreferrer&quot;&gt;Similarity Matrix&lt;/a&gt; from the data by any means. For example calculate the distances between points in $7D$ space and reverse that. Or apply a &lt;a href=&quot;https://en.wikipedia.org/wiki/Radial_basis_function_kernel&quot; rel=&quot;noreferrer&quot;&gt;RBF kernel&lt;/a&gt; for determining the similarity between them. Then calculate the second eigenvalue-eigenvector pair according to sorted eigenvalues. Use K-means to cluster those elements of eigenvectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have more detailed questions just drop a comment here. Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-18T12:14:45.580" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26787" PostTypeId="1" AcceptedAnswerId="26799" CreationDate="2018-01-18T13:02:07.920" Score="1" ViewCount="55" Body="&lt;p&gt;I have a data stream that I would like to share with some data scientists. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a regularly captured time series with some fields that are simple scalars, booleans. Each sample has a UTC time and fractional seconds since start of capture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also captured is a 3d spline that varies in size. These splines are also regularly sampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, there are some other multi dimensional fields as well ([x,y,z,pitch,yaw,roll]).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simple data sets I would normally use CSV. However due to the nature of the more complex data I need a more appropriate format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are my options for formats that would allow easy loading into Matlab or other common data science tools?&lt;/p&gt;&#xA;" OwnerUserId="44815" LastActivityDate="2018-01-18T19:00:38.607" Title="Format for time series data with non-trivial sampled data" Tags="|data-formats|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26788" PostTypeId="2" ParentId="26723" CreationDate="2018-01-18T14:05:51.067" Score="1" Body="&lt;p&gt;Holoviews visual library can handle very large data &lt;a href=&quot;http://holoviews.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://holoviews.org/&lt;/a&gt; &lt;a href=&quot;http://holoviews.org/user_guide/Large_Data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://holoviews.org/user_guide/Large_Data.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43374" LastActivityDate="2018-01-18T14:05:51.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26789" PostTypeId="1" AcceptedAnswerId="26985" CreationDate="2018-01-18T14:22:34.167" Score="3" ViewCount="10143" Body="&lt;p&gt;I am thinking of two use cases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You submit a resume in PDF format to a web site and it extracts your contact information, job titles, etc.&lt;/li&gt;&#xA;&lt;li&gt;You receive an email from a friend that says, &quot;let's have lunch next Tuesday&quot; and your email program detects it and asks if you want to save a new calendar entry for &quot;lunch on Tuesday&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I can see how this can be done with both ML and with (very sophisticated) regex. What approaches are typically used for these scenarios? I'm assuming that the same approach can be applied to both of the scenarios above?&lt;/p&gt;&#xA;" OwnerUserId="23240" LastEditorUserId="-1" LastEditDate="2018-02-01T17:19:57.290" LastActivityDate="2019-03-05T08:44:54.890" Title="Should I use regex or machine learning?" Tags="|machine-learning|nlp|automatic-summarization|" AnswerCount="7" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26790" PostTypeId="2" ParentId="26789" CreationDate="2018-01-18T14:33:24.863" Score="7" Body="&lt;p&gt;I would say depends on the requirements and how much effort you want to give the task. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Using regex is definitely easier but in the some time you will not be able to cover everything specially if the text you have is not structured as you expect all the time. So in this case you will be updating you regex patterns each time you miss something ...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you go for machine learning approach then you will need data to train your models on it ( a lot of this data). Time for training, and enhancing the quality of your model. Hopefully you will get something good.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a conclusion, I think, if you can cover the requirement with regex go for it. If regex will not be a good solution then start thinking in machine learning solutions. &lt;/p&gt;&#xA;" OwnerUserId="44637" LastActivityDate="2018-01-18T14:33:24.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26792" PostTypeId="1" AcceptedAnswerId="39224" CreationDate="2018-01-18T15:21:12.467" Score="11" ViewCount="18169" Body="&lt;p&gt;According to this scintillating &lt;a href=&quot;http://ruder.io/optimizing-gradient-descent/&quot; rel=&quot;noreferrer&quot;&gt;blogpost&lt;/a&gt; Adam is very similar to RMSProp with momentum. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/train/RMSPropOptimizer&quot; rel=&quot;noreferrer&quot;&gt;tensorflow documentation&lt;/a&gt; we see that &lt;code&gt;tf.train.RMSPropOptimizer&lt;/code&gt; has following parameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;__init__(&#xA;    learning_rate,&#xA;    decay=0.9,&#xA;    momentum=0.0,&#xA;    epsilon=1e-10,&#xA;    use_locking=False,&#xA;    centered=False,&#xA;    name='RMSProp'&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while &lt;code&gt;tf.train.AdamOptimizer&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;__init__(&#xA;    learning_rate=0.001,&#xA;    beta1=0.9,&#xA;    beta2=0.999,&#xA;    epsilon=1e-08,&#xA;    use_locking=False,&#xA;    name='Adam'&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the conceptual difference if we put &lt;code&gt;beta1 = decay&lt;/code&gt; and &lt;code&gt;beta2 = momentum&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="42903" LastActivityDate="2022-02-05T15:53:48.903" Title="Difference between RMSProp with momentum and Adam Optimizers" Tags="|tensorflow|optimization|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26793" PostTypeId="2" ParentId="26723" CreationDate="2018-01-18T15:45:11.787" Score="1" Body="&lt;p&gt;Plotting millions of entries through histograms, pie charts, doughnut charts, tree maps, area charts, bar charts, choropleths (and so on - and on and on) does not pose any challenge. You can only find it very slow and annoying if you were to use scatter plots/violin plots, or visualise as a very large graph.&lt;/p&gt;&#xA;" OwnerUserId="27768" LastActivityDate="2018-01-18T15:45:11.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26794" PostTypeId="2" ParentId="26789" CreationDate="2018-01-18T15:53:28.087" Score="4" Body="&lt;p&gt;If you foresee to build something for a big public, definitely you cannot use regular expressions. There is no way you can write a regular expression that can span the variance that a class of documents (email or PDF) can have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if you are happy with a regex that can handle efficiently a (small) percentage of the possible documents, and then amend it from time to time, it will take so much more time with respect to find the training data and train a ML algorithm to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only if you have to parse some kind of standardized document (PDF or email), you can think of using some regex parser.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's the reason why, in general, you need to use some ML technique to reach your goal.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="29134" LastEditDate="2018-01-19T16:02:28.500" LastActivityDate="2018-01-19T16:02:28.500" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26795" PostTypeId="2" ParentId="26776" CreationDate="2018-01-18T16:15:16.667" Score="2" Body="&lt;p&gt;Actually there is the possibility of overfitting the validation set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This because the validation set is the one where your parameters (the depth in your case) perform at best, but this does not means that your model will generalize well on unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's the reason why usually you split your data into three set: train, validation and &lt;em&gt;test&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You train on the training set, tune the parameters on the validation set, and finally, when you are happy with the parameters, you test your model as a whole with the test set. And usually, the error on the test set will be higher than the error on the validation test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this difference is small, you accept the model. But if it's big, you need to act:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) reorganizing the three sets, because maybe you have a variance problem between the sets;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) adding some penalty on your model, acting on the regularization parameters; or lowering the depth of the trees, if you are interested only on it&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-18T16:15:16.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26796" PostTypeId="2" ParentId="15921" CreationDate="2018-01-18T16:33:25.320" Score="5" Body="&lt;p&gt;This question gets at some very important qualities of RNNs and DNNs in general. I'll answer each of your sub-questions, though not in the same order (I'll try to highlight where I do)&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Parameter Sharing&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;First, the most important quality of RNNs is called &lt;em&gt;parameter sharing&lt;/em&gt;. Sequential data is typically input into separate layers. For a length 20 input, an RNN network would have 20 layers. The same internal parameters are used for each layer, so all 20 layers use the same weights $W$ and bias $b$. Compare this to a multilayer perceptron which would have 20 separate weights and biases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parameter sharing has several benefits: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;We now have many fewer parameters. There is 1 repeated block instead of 20 separate layers. A 20x reduction! &lt;/li&gt;&#xA;&lt;li&gt;This effectively multiplies the training data. The recurrent layer gets to learn from every word in a single example sentence whereas each layer in an MLP learns from a single word per sentence.&lt;/li&gt;&#xA;&lt;li&gt;Our network is now much more flexible. We can train on sentences up to 20 words and then generalize to a 25 word sentence by adding more steps or by using a &lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;dynamic RNN&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Network architecture&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You ask about tanh and sigmoid activations. To answer this we have to talk about specific RNN architectures. The simple RNN discussed above has a single activation. Simple RNNs tend to create the problem of &lt;a href=&quot;https://stats.stackexchange.com/questions/185639/how-does-lstm-prevent-the-vanishing-gradient-problem&quot;&gt;vanishing&lt;/a&gt; gradients (or exploding!) due to the repeated application of the same weights and activation function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gated RNN blocks (like GRUs and LSTMs) use gating mechanisms to pass activations into and out of memory states and to combine memory states with input to generate the output of each block. Because of this, gates can stop the gradient from propagating backward. &lt;em&gt;Sigmoid&lt;/em&gt; is a common activation function for gates because it squashes activations to (0,1)---0 completely stops the activation and 1 lets it pass through. Any decent activation function with a similar squashing profile works, though. Anecdotally, &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/keras/backend/hard_sigmoid&quot; rel=&quot;nofollow noreferrer&quot;&gt;hard sigmoid&lt;/a&gt; is quite common these days.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to the gates, gated RNN blocks have an internal state for which the activation varies quite a bit. Because gating limits gradient backprop, we have a lot of flexibility on this activation. It need not be squashing for instance, and this is where the rectifying activations (relu, elu, islu, etc.) are often seen. Tanh is a perfectly sensible choice as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding biases and weights, each activation in an RNN cell typically has its own weight and bias. So a GRU has 3 activations (hidden, update, and reset) and each has its own weight and bias. Though, recall that as an RNN, each of these are re-used for every timestep.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Backward Pass&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;That covers the forward pass pretty well but you also ask an important question about how error propagates backward. There are two methods for approaching this.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Teacher Forcing&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For RNNs that output a prediction at each time step (such as predicting the outcome of following steps, translation, or phoneme recognition), teacher forcing is a method for isolating each step of the RNN. By removing these dependencies, Teacher Forcing allows RNN to use conventional backprop with the chain rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But how does it work? Teacher forcing networks have separate train and test architectures. For training, at each timestep $t$, the input $x_t$ is concatenated with the previous target, $y_{t-1}$. Imagine this for a network tasked with predicting the following character. The network has just tried to predict the character for the previous timestep. But we instead use the &lt;em&gt;observed&lt;/em&gt; character at that timestep (we know this because we're in the training phase). Thus the error at timestep $t$ depends only on the observed value at $t-1$ and the input at $t$. We've thus removed any connections through time from the network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At test, we don't know the true value at each timestep so we replace $y_{t-1}$ with the output of the previous layer $o_{t-1}$. In this case, temporal connections have returned but only for the test phase.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Back Propagation Through Time&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;But we don't &lt;em&gt;have&lt;/em&gt; to resort to Teacher Forcing. Back propogation through time allows us to apply the backprop algorithm to RNNs. Consider the network with $n$ timesteps and an input $x$, hidden state $h$, output $o$, and observed value $y$ for each timestep. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;BPTT works in the following steps.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute the gradient $\nabla_{o_t}$ for each $o_t, y_t$ pair. (This may be done all at once.)&lt;/li&gt;&#xA;&lt;li&gt;Compute the gradient $\nabla_{h_t}$ for each timestep, beginning with the last timestep and iteratively working backward. (This must be done one at a time.) &lt;/li&gt;&#xA;&lt;li&gt;This gives us $n$ &lt;em&gt;edges&lt;/em&gt; for each internal parameter of our RNN. The trick to updating the parameters is finding the gradient contribution for each timestep (eg $\nabla_{W_t}$ even though we only have one $W$) and then summing those gradients to update the internal parameters. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h1&gt;Further reading&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I highly recommend chapter 10 of Goodfellow, Bengio, and Courville's &lt;a href=&quot;http://www.deeplearningbook.org/contents/rnn.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt; for more information on RNNs. Additionally Graves' &lt;a href=&quot;https://www.cs.toronto.edu/~graves/preprint.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;RNN book&lt;/a&gt; is fantstic for higher-level detail &lt;/p&gt;&#xA;" OwnerUserId="38379" LastEditorUserId="38379" LastEditDate="2018-02-05T11:54:19.163" LastActivityDate="2018-02-05T11:54:19.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26797" PostTypeId="1" AcceptedAnswerId="53767" CreationDate="2018-01-18T17:12:16.317" Score="1" ViewCount="1450" Body="&lt;p&gt;I think the question is self-explanatory. But let's say you have a data with a few features with categorical data, and when building a model for example XGBoost you one-hot encode categorical features. Now you want to do prediction based on test data using the saved model. Obviously the test data needs to be one-hot encoded and need have similar features as training set. The question is whether it is possible to find a way not one-hot encode the test data and directly use it for prediction? Would this be somehow possible? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me it appears that whatever comes in to my saved model need to be as it was used during training i.e. one-hot encoded features! But this is not neat, especially when building widgets and dashboards!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any comments/hints are appreciated. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2020-04-09T09:40:22.773" Title="Is it possible to use the saved xgboost model (with one-hot encoding features) on unseen data (without one-hot encoding) for prediction?" Tags="|machine-learning|xgboost|prediction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26798" PostTypeId="2" ParentId="26797" CreationDate="2018-01-18T17:47:05.510" Score="3" Body="&lt;p&gt;A model is built on a specific set of features, which may include categorical features encoded using one-hot encoding.  If you have new data with additional categories, your model has no idea how to interpret the significance of those categories.  You should either map the new value to &lt;em&gt;none&lt;/em&gt; of the 1-hot values identified in training, or to an 'other' value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say you trained on data that had color=[blue,green].  Your one-hot fields would have color_blue and color_green.  You could also have a field called color=other, that you might use to encode very infrequent values.  That's a data preparation choice. So for 'red', you could encode that as either:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;color_green = 0&lt;/li&gt;&#xA;&lt;li&gt;color_blue = 0&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;color_green = 0&lt;/li&gt;&#xA;&lt;li&gt;color_blue = 0&lt;/li&gt;&#xA;&lt;li&gt;color_other = 1&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Using either of these techniques will work with xgboost, but as xgboost only accepts numeric inputs, you will have to choose one of these methods as a data pre-processing step.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-01-18T17:47:05.510" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26799" PostTypeId="2" ParentId="26787" CreationDate="2018-01-18T19:00:38.607" Score="1" Body="&lt;p&gt;Based on &lt;a href=&quot;https://arxiv.org/pdf/1707.05364&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; article about Data Formats in Data Science, the short answer to your question is: use &lt;a href=&quot;https://en.m.wikipedia.org/wiki/JSON&quot; rel=&quot;nofollow noreferrer&quot;&gt;JSON&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This may seem natural next step after csv and no more, but it is actually quite interesting:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;JSON is widely supported in many programming languages, it is a valid MIME type in Internet standards (&lt;code&gt;application/json&lt;/code&gt;, data stream mentioned in your question) and it has several standard and non standard extensions (&lt;a href=&quot;http://geojson.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;geoJSON&lt;/a&gt;, &lt;a href=&quot;https://en.m.wikipedia.org/wiki/BSON&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary json&lt;/a&gt;). If there is not a standard structure, JSON makes it easy to invent your own, because it is easily modifiable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the article JSON works as value in HADOOP. For HADOOP it is suitable, because basically JSON is plain text when you think it as a data item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also article mentions JSON more efficient than CSV.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-01-18T19:00:38.607" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26800" PostTypeId="2" ParentId="22984" CreationDate="2018-01-18T19:18:01.313" Score="2" Body="&lt;p&gt;An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: &quot;Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class.&quot;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-18T19:18:01.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26802" PostTypeId="1" AcceptedAnswerId="26805" CreationDate="2018-01-18T19:48:25.913" Score="1" ViewCount="102" Body="&lt;p&gt;I have a list of files, an I use the KNN algorithm to classify these files.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataset = pd.read_csv(file)&#xA;training_samples = get_sample_number(dataset)&#xA;X_train = dataset.iloc[:training_samples, 5:9]&#xA;y_train = dataset.iloc[:training_samples, 9]&#xA;X_test = dataset.iloc[training_samples:, 5:9]&#xA;&#xA;# Feature Scaling&#xA;sc = StandardScaler()&#xA;X_train = sc.fit_transform(X_train)&#xA;X_test = sc.fit_transform(X_test)&#xA;&#xA;# Fitting classifier to the training set&#xA;classifier = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2)&#xA;classifier.fit(X_train, y_train)&#xA;&#xA;y_pred = classifier.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I have my categories in my y_pred array. But I want to save the result in the file where I read the dataset. How can I link a prediction to the right row in the file (or dataset)?&lt;/p&gt;&#xA;" OwnerUserId="44135" LastActivityDate="2018-01-19T08:46:23.303" Title="Link prediction with input data" Tags="|pandas|numpy|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26803" PostTypeId="2" ParentId="8967" CreationDate="2018-01-18T21:15:46.457" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/vegas-viz/Vegas&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vegas&lt;/a&gt; is a library that strives to substitute for matplotlib in Zeppelin/Spark environments. It works on Spark DataFrames.&lt;br /&gt;&#xA;The project is no longer maintain.&lt;/p&gt;&#xA;" OwnerUserId="44838" LastEditorUserId="52527" LastEditDate="2022-03-05T04:19:35.897" LastActivityDate="2022-03-05T04:19:35.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26804" PostTypeId="2" ParentId="26781" CreationDate="2018-01-18T21:18:01.830" Score="3" Body="&lt;p&gt;For what you're trying to do, first-order differences without interpolation should work just fine. Once you've done that, the problem reduces to a simple anomaly detection task. Here's a demonstration using &lt;a href=&quot;http://dmarx.github.io/Topological-Anomaly-Detection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;topoligical anomaly detection&lt;/a&gt;. Briefly:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The technique is essentially a density based outlier detection algorithm that, instead of calculating local densities, constructs a graph of the data using nearest-neighbors. The algorithm is different from other kNN outlier detection algorithms in that instead of setting ‘k’ as a parameter, you instead set a maximal inter-observation distance (called the graph “resolution” by &lt;a href=&quot;http://www.cis.rit.edu/~mxgpci/pubs/gartley-7334-1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gartley and Basener&lt;/a&gt;). If the distance between two points is less than the graph resolution, add an edge between those two observations to the graph. Once the full graph is constructed, determine which connected components comprise the “background” of the data by setting some threshold percentage of observations ‘p’: any components with fewer than ‘p’ observations is considered an anomalous component, and all the observations (nodes) in this component are outliers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;First, let's generate some data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_obs = 100&#xA;spike_perc = .1&#xA;&#xA;###################&#xA;&#xA;set.seed(123)&#xA;&#xA;n_spikes = floor(spike_perc * n_obs)&#xA;inter_arrival = rexp(n_obs)&#xA;xv = cumsum(inter_arrival)&#xA;&#xA;mu_spike = 5&#xA;sd_spike = sqrt(mu_spike)&#xA;sd_noise = mu_spike/75&#xA;spike_events = sample(n_obs, n_spikes)&#xA;&#xA;flat_noise = rnorm(n_obs, 0, sd_noise)&#xA;spike_ampl = abs(rnorm(n_spikes, mu_spike, sd_spike))&#xA;&#xA;yv = rep(0, n_obs)&#xA;yv[spike_events] = yv[spike_events]  + spike_ampl&#xA;yv = cumsum(yv)&#xA;yv = yv + flat_noise&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's what our data looks like (spike events in red):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/b6UG9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/b6UG9.png&quot; alt=&quot;Generated data (spike events in red)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate first-order differences (appending 0 to preserve index mapping):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dat = c(0,diff(yv))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Find us some outliers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(igraph)&#xA;&#xA;# Helper function. Equivalently, use mefa::stack&#xA;row_col_from_condensed_index = function(n, ix){&#xA;  nr=ceiling(n-(1+sqrt(1+4*(n^2-n-2*ix)))/2)&#xA;  nc=n-(2*n-nr+1)*nr/2+ix+nr&#xA;  cbind(nr,nc)&#xA;}&#xA;&#xA;# Algorithm parameters, may require tuning&#xA;rq = .2 # distance quantile for thresholding (higher-&amp;gt;fewer outliers)&#xA;p = .1  # percent of population necessary for a component to be flagged as an inlier&#xA;method='euclidean'&#xA;&#xA;# calculate inter-observation distances and threshold to define adjacency matrix&#xA;n=NROW(dat)&#xA;d = dist(dat, method=method)&#xA;r = quantile(d, rq)&#xA;edges = t(sapply(which(d&amp;lt;=r), function(x) row_col_from_condensed_index(n,x)))&#xA;&#xA;# build adjacency graph&#xA;g = graph.empty(n, directed=FALSE)&#xA;g[from=edges[,1], to=edges[,2]]=1&#xA;V(g)$name = 1:n&#xA;&#xA;# Flag outliers based on component size&#xA;components = clusters(g, mode=&quot;strong&quot;)&#xA;csize = components$csize[components$membership]&#xA;outliers = V(g)[csize&amp;lt;=p*n]$name&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here's how we did (flagged outliers in red):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cQKxf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cQKxf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you go this route, you may need to tweak the &lt;code&gt;rq&lt;/code&gt; and &lt;code&gt;p&lt;/code&gt; parameters a bit. If you'd rather not deal with the quantile thresholding parameter (rq), an alternative approach is to use a nearest neighbor graph, in which case you'll have to tinker with the 'k' parameter (of KNN). That approach would replace the distance calculations and graph instantiation above with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(cccd)&#xA;&#xA;g = nng(data.frame(dat), k=5, use.fnn=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-19T10:01:23.927" LastActivityDate="2018-01-19T10:01:23.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26805" PostTypeId="2" ParentId="26802" CreationDate="2018-01-18T21:40:34.733" Score="1" Body="&lt;p&gt;First as &quot;timleathart&quot; mentioned, you need to fix your code by changing this line : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test = sc.fit_transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_test = sc.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For your question :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;you have already the number of samples (training_samples) used for training. so all you need is to iterate over the y_pred and save the values in new column in the dataset starting from &quot;training_samples&quot; as row index. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="44637" LastActivityDate="2018-01-18T21:40:34.733" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26806" PostTypeId="2" ParentId="26802" CreationDate="2018-01-18T21:40:43.767" Score="1" Body="&lt;p&gt;First and foremost: I would &lt;em&gt;strongly&lt;/em&gt; advise against modifying your original data file. It introduces risk that your workflow will become unrepeatable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To persist your results, I recommend doing something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;in_fname = '{}.csv'.format(filename)&#xA;out_fname = '{}_SCORED.csv'.format(filename)&#xA;dataset = pd.read_csv(in_fname)&#xA;&#xA;... do stuff&#xA;&#xA;dataset.loc[training_samples:, 'scores'] = y_pred&#xA;dataset.to_csv(out_fname, header=True, index=False)    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you really want to overwrite your original data, just set &lt;code&gt;in_fname=out_fname&lt;/code&gt;. But I'd advise against it.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-19T08:46:23.303" LastActivityDate="2018-01-19T08:46:23.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26807" PostTypeId="1" AcceptedAnswerId="26812" CreationDate="2018-01-18T22:13:37.790" Score="1" ViewCount="376" Body="&lt;p&gt;I am creating my own implementation of a Naïve Bayes classifier. While it’s behaviour and functionalities are clear to me, my concerns are on the nature of the training and testing data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I acquired several sets of product reviews from Amazon. The first thing I do is parsing them, that is, taking the rating (1 to 5 stars) and the text, which I parse with a regex to only contain alphabetical lowercase characters and spaces. Next, I convert ratings to polar values, so 1 and 2 stars become “-“ and 4 and 5 stars become “+”. I’m intentionally skipping reviews with 3 stars; could this be an issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here come my real concerns. When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews (such as 7 positive and 7 negative reviews for training and 3 positive and 3 negative reviews for testing)? Right now I’m acquiring as many positive as negative reviews from the chosen set, but I’m wondering if that should be the case. For instance, if a set contains 7 positive reviews and 4 negative ones, I discard 3 positive reviews to equate them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I observed that negative reviews tend to contain longer texts on average. So, if I’m using an equal number of positive and negative reviews, but they differ on average text length, would this have an impact on the way my classifier tries to predict?&lt;/p&gt;&#xA;" OwnerUserId="44844" LastActivityDate="2018-01-19T00:19:49.393" Title="Naïve Bayes and Training Data" Tags="|naive-bayes-classifier|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26808" PostTypeId="2" ParentId="26779" CreationDate="2018-01-18T22:15:56.790" Score="1" Body="&lt;p&gt;The closest I can think of is research into &quot;adversarial&quot; transformations of data (usually images) that cause classifiers (usually CNNs) to fail. Here's a &lt;a href=&quot;https://blog.openai.com/adversarial-example-research/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; which discusses a bit about how these transformations are constructed, some approaches that have been tried to address them, and what some of the consequences of their existence might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also be interested to read about the &lt;a href=&quot;https://en.wikipedia.org/wiki/No_free_lunch_theorem&quot; rel=&quot;nofollow noreferrer&quot;&gt;No Free Lunch Theorem&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-18T22:15:56.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26811" PostTypeId="1" CreationDate="2018-01-18T23:48:57.410" Score="4" ViewCount="16595" Body="&lt;p&gt;Suppose I have data with &lt;strong&gt;X_train, X_test, y_train, y_test&lt;/strong&gt; given. As it is a classification problem I want to use &lt;strong&gt;XGBoost&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is that there are more than 300 features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found online that there are ways to find features which are important. But as I have lot of features it's causing an issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current code is below. How can I modify it to say select top n ( n = 20) features and use them for training the model. I tried sorting the features based on importance but it doesn't work. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   import xgboost as xgb&#xA;    gbm = xgb.XGBClassifier(max_depth=3, n_estimators=300, learning_rate=0.05).fit(X_train,y_train)&#xA;    predictions = gbm.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2019-05-02T02:57:25.110" Title="How to find and use the top features for XGBoost?" Tags="|machine-learning|python|random-forest|xgboost|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26812" PostTypeId="2" ParentId="26807" CreationDate="2018-01-19T00:19:49.393" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I’m intentionally skipping reviews with 3 stars; could this be an issue?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's not a &lt;em&gt;problem&lt;/em&gt;, it's just your definition of your question. It assumes rate 3 as neutral and blow/above that as negative/positive. One may say I assume every rate is positive so each number is a level of positiveness then rate 3 works for him. The main point is that you get the answer for the question you define so be careful about what you want from your classifier to set up your question correctly.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Nice question. Before specifically taking Naive Bayes into account, it is a general machine learning problem when the population of classes are imbalanced. If this is the case then better to balance them, not only in train/test split but also during train itself as the dominating class will bias your result. If the classes are not that imbalanced then you can split things randomly and it's fine. Regarding Bayes Classifier itself the balance inside training set should be more important as NB learns from the statistics of your training set. On test set, it just uses the already learned statistics so portion of classes should not impact.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I observed that negative reviews tend to contain longer texts on average. would this have an impact on the way my classifier tries to predict?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Again nice question. Yes it does but that is not necessarily that bad. Longer text means that the probability of having larger number for term frequencies are higher. This is a feature itself! (Just think again that you already found a &lt;strong&gt;difference&lt;/strong&gt; between classes. Well ... you want NB to do the same right?!!) The only thing is that if the difference in lengths of comments is very huge, then it might affect the value of features in a way that you need normalization (e.g. instead of counting terms as features, you can use TF-IDF which is a bounded score)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps. Good Luck!  &lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-19T00:19:49.393" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26813" PostTypeId="1" AcceptedAnswerId="26814" CreationDate="2018-01-19T04:05:24.083" Score="6" ViewCount="870" Body="&lt;p&gt;I have been working with the &lt;a href=&quot;https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition&quot; rel=&quot;noreferrer&quot;&gt;cats vs dogs dataset from kaggle&lt;/a&gt; which consist on 25000 images of cats and dogs labelled accordingly (btw, great dataset, totally recommended!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the things I did was to augment the data by simply fliping the images 180 degrees, so an image like this&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mmlbL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mmlbL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;will became this&#xA;&lt;a href=&quot;https://i.stack.imgur.com/1rltT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1rltT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far so good, now I have 50000 images, but the question is: should I make SURE that an image that has been augmented remains totally into the training set? I mean, with the previous two images, what would happen if I have one of them in the training set and another one in the test (or cross validation) set? does it mean that I am leaking training data into test data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that technically they are DIFFERENT images, but still my intuitions seems to resist such idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correct to assume that augmented images should not leak to the test set?&lt;/p&gt;&#xA;" OwnerUserId="38339" LastActivityDate="2018-01-19T04:13:47.080" Title="Should I prevent augmented data to leak to the test/cross validation sets" Tags="|dataset|image-classification|overfitting|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26814" PostTypeId="2" ParentId="26813" CreationDate="2018-01-19T04:13:47.080" Score="4" Body="&lt;p&gt;Yes! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy of your algorithm should be tested on images that you expect to receive. This would not include data that has been rotated, shifted, blurred, etc. However, augmenting your data in such ways can and usually will lead to better results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Split your data first. Then augment only the training data while staying away from the validation data. Only use your testing data when you are confident that your solution leads to good results on your validation set without overfitting it. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-19T04:13:47.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26815" PostTypeId="1" CreationDate="2018-01-19T06:02:16.237" Score="3" ViewCount="5436" Body="&lt;p&gt;I have implemented a custom layer in keras which takes in multiple input and also results to multiple output shape. My code goes as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Attention(Layer):&#xA;def __init__(self, max_input_left=MAX_SEQUENCE_LENGTH,max_input_right=MAX_SEQUENCE_LENGTH,&#xA;             filter_sizes=[1,2,3], num_filters=400, **kwargs):&#xA;    self.init = initializers.get('normal')&#xA;    self.max_input_left = max_input_left&#xA;    self.max_input_right = max_input_right&#xA;    self.filter_sizes = filter_sizes&#xA;    self.num_filters = num_filters&#xA;    if self.num_filters!=0:&#xA;        self.num_filters_total=self.num_filters * len(self.filter_sizes)   &#xA;    super(Attention, self).__init__(** kwargs)&#xA;&#xA;def build(self, input_shape):&#xA;    input_dim=input_shape[1][-1]*len(self.filter_sizes)&#xA;    self.U = self.init((input_dim,input_dim))&#xA;    super(Attention, self).build(input_shape)&#xA;&#xA;def compute_mask(self, inputs, mask=None):&#xA;    mask = super(Attention, self).compute_mask(inputs, mask)&#xA;    return mask&#xA;&#xA;def call(self,inputs, mask=None):&#xA;    input_left = inputs[0]&#xA;    input_right = inputs[1]&#xA;    Q = tf.reshape(input_left,[-1,self.max_input_left,len(self.filter_sizes) * self.num_filters],name = 'Q')&#xA;    A = tf.reshape(input_right,[-1,self.max_input_right,len(self.filter_sizes) * self.num_filters],name = 'A')&#xA;        # G = tf.tanh(tf.matmul(tf.matmul(Q,self.U),\&#xA;        # A,transpose_b = True),name = 'G')&#xA;    print Q&#xA;    print tf.reshape(Q,[-1,len(self.filter_sizes) * self.num_filters])&#xA;    first = tf.matmul(tf.reshape(Q,[-1,len(self.filter_sizes) * self.num_filters]),self.U)&#xA;&#xA;    second_step = tf.reshape(first,[-1,self.max_input_left,len(self.filter_sizes) * self.num_filters])&#xA;    result = tf.matmul(second_step,tf.transpose(A,perm = [0,2,1]))&#xA;        # print 'result',result&#xA;    G = tf.tanh(result)&#xA;        # G = result&#xA;        # column-wise pooling ,row-wise pooling&#xA;    row_pooling = tf.reduce_max(G,1,True,name = 'row_pooling')&#xA;    col_pooling = tf.reduce_max(G,2,True,name = 'col_pooling')&#xA;&#xA;    self.attention_q = tf.nn.softmax(col_pooling,1,name = 'attention_q')&#xA;    print self.attention_q&#xA;    self.attention_a = tf.nn.softmax(row_pooling,name = 'attention_a')&#xA;    R_q = tf.reshape(tf.matmul(Q,self.attention_q,transpose_a = 1),[-1,self.num_filters * len(self.filter_sizes)],name = 'R_q')&#xA;    R_a = tf.reshape(tf.matmul(self.attention_a,A),[-1,self.num_filters * len(self.filter_sizes)],name = 'R_a')&#xA;    self.output_dim1  = R_q&#xA;    self.output_dim2 = R_a&#xA;    return R_q,R_a  &#xA;&#xA;def compute_output_shape(self, input_shapes):&#xA;&#xA;    print (tuple(K.int_shape(self.output_dim1)),tuple(K.int_shape(self.output_dim2)))&#xA;    return (tuple(K.int_shape(self.output_dim1)),tuple(K.int_shape(self.output_dim2)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am calling this layer as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  attn1, attn2 = Attention()([qns_cnn, ans_cnn])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives me the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;File &quot;workspace-2/SemEval/test2QA.py&quot;, line 253, in &amp;lt;module&amp;gt;&#xA;attn1 = Attention()([qns_cnn, ans_cnn])&#xA;File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py&quot;, line 619, in __call__&#xA;arguments=user_kwargs)&#xA;File &quot;/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py&quot;, line 681, in _add_inbound_node&#xA;output_tensors[i]._keras_shape = output_shapes[i]&#xA;AttributeError: 'tuple' object has no attribute '_keras_shape'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest where is the issue?&lt;/p&gt;&#xA;" OwnerUserId="44328" LastActivityDate="2018-01-19T06:02:16.237" Title="Custom layer in keras with multiple input and multiple output" Tags="|deep-learning|tensorflow|keras|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26816" PostTypeId="1" CreationDate="2018-01-19T06:54:04.610" Score="1" ViewCount="37" Body="&lt;p&gt;I've been studying a paper on Recommender systems using collaborative deep learning where in a hierarchical bayesian model called collaborative deep learning is developed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stacked denoising autoencoders are used for the feature learning of the input which is the data from citeulike.&#xA;And CF is used for learning the content information and collaborative filtering for the ratings(feedback) matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link to the paper :&lt;a href=&quot;http://www.wanghao.in/paper/KDD15_CDL.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wanghao.in/paper/KDD15_CDL.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40277" LastEditorUserId="40277" LastEditDate="2018-01-19T09:25:23.707" LastActivityDate="2018-01-19T09:25:23.707" Title="How does a feature learning component work" Tags="|machine-learning|neural-network|recommender-system|" AnswerCount="0" CommentCount="0" ClosedDate="2018-01-22T20:10:24.957" ContentLicense="CC BY-SA 3.0" />
  <row Id="26817" PostTypeId="2" ParentId="26623" CreationDate="2018-01-19T08:13:44.563" Score="1" Body="&lt;p&gt;Notice that if you want to establish an order relation that is cyclic you end up with a contradiction. In fact, you get: &#xA;0:00 &amp;lt; 23:59 &amp;lt; 0:00.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want to compute distances and you give up with continuity you might proceed as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, you transform the domain hh:mm in minutes and then linearly map into an integer number between 0 and $2^k-1$. The transformation function might be $$f(h,m)=\mathrm{round}\left(\frac{60h+m}{60\cdot23+59}\times (2^k-1)\right)$$&#xA;You can choose $k=10$. It may be better to use less bits than those required to exactly represent time (exact time requires 11 bit if seconds are not used, but not all $2^{11}$ configurations are used).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you encode $f(h,m)$ with a &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Gray_code&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gray code&lt;/a&gt;. Gray code assure that consecutive numbers are encoded in a way that at most one bit is changed; also, the code is cyclic in the sense that the encoding of $0$ and the encoding of $2^k-1$ differ of one bit only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can evaluate the distance between two time instants through bitwise Hamming distance, which counts the number of different bit between two codes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want more precision, you might use seconds accordingly.&lt;/p&gt;&#xA;" OwnerUserId="28713" LastActivityDate="2018-01-19T08:13:44.563" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26818" PostTypeId="1" CreationDate="2018-01-19T08:57:34.627" Score="1" ViewCount="217" Body="&lt;p&gt;I'm trying to find the patterns in the crime records I have in a database.&#xA;I thought clustering would be a way to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my (cooked up) dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;age,nationality,country_of_birth,place_of_birth,no_of_checkedinbaggage,noofcabinbaggage,no_of_co_passengers,watchlist&#xA;34,GBR,GBR,London,2,1,0,Drug Trafficker&#xA;32,IND,IND,Delhi,2,1,0,Human Trafficker&#xA;31,USA,USA,Tampa,2,1,0,Arms Dealer&#xA;.....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, I'd want to identify the clusters of watchlists and see if they are having a pattern. Based on the cluster, I'd want to predict future data as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is clustering (K-Means) the correct choice? And also, do all the variables have to be numeric? If so, I'm not sure how I can encode them to numerics. Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="44861" LastEditorUserId="29575" LastEditDate="2018-01-19T13:28:55.553" LastActivityDate="2018-01-19T13:28:55.553" Title="KMeans for crime patterns" Tags="|machine-learning|data-mining|apache-spark|k-means|pyspark|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26819" PostTypeId="2" ParentId="26818" CreationDate="2018-01-19T09:04:29.273" Score="0" Body="&lt;p&gt;Yes for clustering K-Means algorithm is good choice. The only thing is that you should select the number of cluster that you want as result.&#xA;exp: k = 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As i see you have categorical variables in your data, you can use &quot;One-Hot-Encoding&quot; to convert them to numerical feature. I would suggest you scale your data after that too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;keywords here: &quot;encode categorical feature&quot;, &quot;One Hot Encoder&quot;, &quot;feature Scaling&quot;. i Suggest you read more about those. &lt;/p&gt;&#xA;" OwnerUserId="44637" LastActivityDate="2018-01-19T09:04:29.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26820" PostTypeId="2" ParentId="26818" CreationDate="2018-01-19T09:18:08.660" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you know KMeans is an Unsupervised learning and it helps you to find out if there are any patterns in the data. Yes, the procedure which you are following to find some commonality/patterns in the data. But this is not generally used for prediction. FYI you can use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;K-means&lt;/a&gt; for predicting too, recently I came across that but I don't know whether it would yield the desired results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the data is categorical you need to apply &lt;a href=&quot;https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f&quot; rel=&quot;nofollow noreferrer&quot;&gt;One-Hot Encoding&lt;/a&gt;, which converts the categorical data into numeric, you can go through the link for better understanding. If not so you cannot apply K-Means algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cannot convert the categorical data into numeric data then you can use this package &lt;a href=&quot;https://cran.r-project.org/web/packages/clustMixType/clustMixType.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ClustMixType&lt;/a&gt; this is in R and you can use KMODES in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have your target variable ready, as you haven't mentioned the data target variable, if the target variable is Numeric you can use &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Neural Network &lt;/li&gt;&#xA;&lt;li&gt;Regression&lt;/li&gt;&#xA;&lt;li&gt;SVM&lt;/li&gt;&#xA;&lt;li&gt;Random Forest and many more&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;if the target variable is Binary(is he a criminal yes/no)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Neural Network&lt;/li&gt;&#xA;&lt;li&gt;SVM&lt;/li&gt;&#xA;&lt;li&gt;Logistic Regression&lt;/li&gt;&#xA;&lt;li&gt;Random Forest&lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes Classifier&lt;/li&gt;&#xA;&lt;li&gt;KNN and many more.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please go through this &lt;a href=&quot;https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data/24050#24050&quot;&gt;Link&lt;/a&gt;, for better understanding on Mixed Data type Clustering &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you need any help.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-01-19T09:18:08.660" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26821" PostTypeId="1" AcceptedAnswerId="26823" CreationDate="2018-01-19T09:35:55.657" Score="1" ViewCount="6673" Body="&lt;p&gt;I understand how to build and train a neural network like shown below, as well as those low-level features/filters. I wonder what are those &lt;strong&gt;high-level features&lt;/strong&gt;: how exactly do you obtain them from a trained neural network? (Are those like the &quot;&lt;a href=&quot;https://en.wikipedia.org/wiki/Eigenface&quot; rel=&quot;nofollow noreferrer&quot;&gt;eigenfaces&lt;/a&gt;&quot;?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wRiiU.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wRiiU.jpg&quot; alt=&quot;DNN&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: the image is by NVIDIA, and I don't know the specifics of the classification problem here. If needed, suppose the network is trained to distinguish human from cat. &lt;/p&gt;&#xA;" OwnerUserId="33728" LastActivityDate="2018-01-19T10:04:55.037" Title="High-level features of a neural network" Tags="|machine-learning|neural-network|classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26822" PostTypeId="2" ParentId="26821" CreationDate="2018-01-19T09:44:19.927" Score="3" Body="&lt;p&gt;Its about how the neural net learn inside. Usually in deep neural network you have multiple layers, the first layers will learn the low level feature then the more you approach the output layer the more the layers will learn the high level feature. &#xA;Here its identifying faces task, that's why you see that in the first layers small patterns are learned (mainly edges ...) then faces components ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to obtain them : you should watch what makes neuron activated in each layer depending on the input. As you know each neuron will be activated (once the DNN is trained) for specific input combinations. &#xA;By visualizing that you can get an idea about what exactly each layer has learned in term of high-low level features. &lt;/p&gt;&#xA;" OwnerUserId="44637" LastActivityDate="2018-01-19T09:44:19.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26823" PostTypeId="2" ParentId="26821" CreationDate="2018-01-19T09:58:31.210" Score="1" Body="&lt;p&gt;The image, and variants of it that are commonly used are for illustrative purposes only. They generally do not represent data that has been extracted from real CNNs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first &quot;Low-level features&quot; part of the diagram is possibly from a real network (I am not sure in this case, it looks more like a constructed filter, e.g. Sobel, to me). That is because it is feasible and relatively easy to interpret the first layer's filter weights directly as images, and the filters do indeed look like the components that they detect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;Mid-level features&quot; and &quot;High-level features&quot; in your specific diagram have probably been constructed without using a neural network. They are likely to be an artists impression of what the high level features might be. They may have been sampled from real datasets, then just cropped and arranged into the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caveat: I cannot find absolute evidence for the specific image being constructed for illustration only, just I suspect this to be the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to extract visualisations of features detected by deeper layers. The two common ways to do this are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Dataset matching. Finding examples in the dataset which trigger a specific neuron to output a high value. This can be isolated to a crop of the original image, because you know the combined sizes of all the filters and pools that occur before layer you are interested in.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Optimising the input image. Using gradient ascent, but instead of changing the weights, make a cost function that scores the neuron you want to visualise and keep adjusting the input until&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can get more information from resources such as &lt;a href=&quot;https://distill.pub/2017/feature-visualization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article on feature visualisation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-19T10:04:55.037" LastActivityDate="2018-01-19T10:04:55.037" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26824" PostTypeId="1" AcceptedAnswerId="26828" CreationDate="2018-01-19T10:30:49.460" Score="2" ViewCount="41" Body="&lt;p&gt;When you have multiple methods to accomplish a task, how do you choose which one to implement?&lt;/p&gt;&#xA;" OwnerUserId="44857" LastEditorUserId="11097" LastEditDate="2018-01-19T10:35:04.603" LastActivityDate="2018-01-19T14:21:57.977" Title="How do you avoid 'analysis paralysis' when choosing a method to implement?" Tags="|management|project-planning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26825" PostTypeId="2" ParentId="26175" CreationDate="2018-01-19T12:16:28.170" Score="0" Body="&lt;p&gt;They use this variable to save all the weight matrices by concatenating them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the &lt;code&gt;call&lt;/code&gt; function of the &lt;code&gt;LSTMCell&lt;/code&gt; you can see, how they are unpacked:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.kernel_i = self.kernel[:, :self.units]&#xA;self.kernel_f = self.kernel[:, self.units: self.units * 2]&#xA;self.kernel_c = self.kernel[:, self.units * 2: self.units * 3]&#xA;self.kernel_o = self.kernel[:, self.units * 3:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="21846" LastEditorUserId="29575" LastEditDate="2018-01-19T13:30:47.817" LastActivityDate="2018-01-19T13:30:47.817" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26826" PostTypeId="1" CreationDate="2018-01-19T13:06:15.137" Score="4" ViewCount="147" Body="&lt;p&gt;I want to detect tweet text agreement. Suppose someone posts some subjective opinion in twitter. Other users will post reply either agreeing or opposing the original tweet. I want to estimate the amount of agreement. Is there any algorithm/library in any language to do that or any labeled dataset? &lt;/p&gt;&#xA;" OwnerUserId="31180" LastEditorUserId="31180" LastEditDate="2018-01-19T14:04:49.170" LastActivityDate="2019-08-15T00:02:20.260" Title="How to detect if one tweet is agreeing with another" Tags="|nlp|sentiment-analysis|twitter|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26827" PostTypeId="2" ParentId="26826" CreationDate="2018-01-19T13:39:28.983" Score="1" Body="&lt;p&gt;Not sure there is anything for that, you could check&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is it a verbatim retweet&lt;/li&gt;&#xA;&lt;li&gt;does it have the same sentiment&lt;/li&gt;&#xA;&lt;li&gt;is the edit distance low&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Or you can train your own model, where you label (agree) by hand and then build features.  &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2018-01-19T13:39:28.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26828" PostTypeId="2" ParentId="26824" CreationDate="2018-01-19T13:42:45.973" Score="1" Body="&lt;p&gt;This is a very broad question but in general you can define some criteria that you want the method to meet (eg low error, efficiency, scalability) and then you score the methods, which one solves your problem best. &lt;/p&gt;&#xA;" OwnerUserId="11080" LastActivityDate="2018-01-19T13:42:45.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26829" PostTypeId="2" ParentId="26824" CreationDate="2018-01-19T14:21:57.977" Score="1" Body="&lt;p&gt;Adding to Dirk’s answer, a structured approach is to create what is known as an &lt;strong&gt;Algorithm/Model Test Harness&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;See here&lt;/strong&gt;: &lt;a href=&quot;https://machinelearningmastery.com/create-algorithm-test-harness-scratch-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/create-algorithm-test-harness-scratch-python/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This allows you to pass in a variety of learning algorithms and quickly evaluate them against your set of quality metrics. These metrics can be determined upfront and based on both statistical &lt;em&gt;and&lt;/em&gt; domain-specific notions of quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally, you create a &lt;strong&gt;json&lt;/strong&gt; structure containing learning algorithms, initial hyper-parameters, constraints, above mentioned qualifiers, etc. and pass it into your test harness. Everything gets evaluated within the harness and the output is a set it accuracies (and other qualifiers). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously you need to consider &lt;strong&gt;runtimes&lt;/strong&gt; if you decide to construct a test harness that does many combinations of your json contents. But the idea is to &lt;em&gt;balance&lt;/em&gt; your upfront ideas about what is best with the more data-driven approach achieved using a test harness. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So start with smaller samples of your data and construct a model test harness that evaluates a decent number of combinations. This will help you narrow your choices, at which point you can start thinking more deeply with less upfront assumptions. &lt;/p&gt;&#xA;" OwnerUserId="30359" LastActivityDate="2018-01-19T14:21:57.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26830" PostTypeId="2" ParentId="26826" CreationDate="2018-01-19T15:17:55.570" Score="1" Body="&lt;p&gt;Sure there can be more complicated approach but if you are dealing with raw tweets, I think problem is twofolds;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Topic Discovery&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You first need to find out what the tweet is talking about. It will be much easier task if you can skip this, given that you are looking at tweets with specific tags; that you know what the tweet is pertaining to. Otherwise, you can use LDA or &lt;code&gt;gensim&lt;/code&gt; library in Python.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Sentiment Analysis &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is much easier task. For each topic, for all tweets, associate tweets with probability of positive / negative and also, you could scale this by certainty. This could be using out of box solution such as from &lt;code&gt;nltk&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This github repo seems to be doing what you are intending to do, and could get some inspiration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/nagarmayank/twitter_sentiment_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/nagarmayank/twitter_sentiment_analysis&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2869" LastActivityDate="2018-01-19T15:17:55.570" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26831" PostTypeId="2" ParentId="26784" CreationDate="2018-01-19T15:18:04.753" Score="7" Body="&lt;p&gt;I know of no pairwise distance operations in Keras or tensorflow. But the matrix math can be implemented in TF/Keras backend code and then placed in a Keras layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the matrix representation of the cosine similarity of two vectors:&#xA;$$&#xA;cos(\theta) = \frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|_2 \|\mathbf{B}\|_2}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll show the code and a test that confirms that it works. First, generate non-trivial test data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import keras&#xA;import keras.backend as K&#xA;&#xA;# set up test data&#xA;n_batch = 100&#xA;n = 400 # number of points in the first set&#xA;m = 500 # number of points in the second set&#xA;d = 200 # number of dimensions&#xA;&#xA;A = np.random.rand(n_batch, n, d)&#xA;B = np.random.rand(n_batch, m, d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Define pairwise cosine similarity function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# convenience l2_norm function&#xA;def l2_norm(x, axis=None):&#xA;    &quot;&quot;&quot;&#xA;    takes an input tensor and returns the l2 norm along specified axis&#xA;    &quot;&quot;&quot;&#xA;&#xA;    square_sum = K.sum(K.square(x), axis=axis, keepdims=True)&#xA;    norm = K.sqrt(K.maximum(square_sum, K.epsilon()))&#xA;&#xA;    return norm&#xA;&#xA;def pairwise_cosine_sim(A_B):&#xA;    &quot;&quot;&quot;&#xA;    A [batch x n x d] tensor of n rows with d dimensions&#xA;    B [batch x m x d] tensor of n rows with d dimensions&#xA;&#xA;    returns:&#xA;    D [batch x n x m] tensor of cosine similarity scores between each point i&amp;lt;n, j&amp;lt;m&#xA;    &quot;&quot;&quot;&#xA;&#xA;    A, B = A_B&#xA;    A_mag = l2_norm(A, axis=2)&#xA;    B_mag = l2_norm(B, axis=2)&#xA;    num = K.batch_dot(A_tensor, K.permute_dimensions(B_tensor, (0,2,1)))&#xA;    den = (A_mag * K.permute_dimensions(B_mag, (0,2,1)))&#xA;    dist_mat =  num / den&#xA;&#xA;    return dist_mat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Build a shallow Keras model around the function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# build dummy model&#xA;A_tensor = K.constant(A)&#xA;B_tensor = K.constant(B)&#xA;A_input = keras.Input(tensor=A_tensor)&#xA;B_input = keras.Input(tensor=B_tensor)&#xA;dist_output = keras.layers.Lambda(pairwise_cosine_sim)([A_input, B_input])&#xA;dist_model = keras.Model(inputs=[A_input, B_input], outputs=dist_output)&#xA;dist_model.compile(&quot;sgd&quot;, &quot;mse&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Compare to sklearn implementation&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sk_dist = np.zeros( (n_batch, n, m) )&#xA;for i in range(n_batch):&#xA;    sk_dist[i,...] = cosine_similarity(A[i,...], B[i,...])&#xA;&#xA;keras_dist = dist_model.predict(None, steps=1)&#xA;np.allclose(sk_dist, keras_dist)&#xA;&amp;gt; True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38379" LastEditorUserId="38379" LastEditDate="2018-01-23T18:25:45.200" LastActivityDate="2018-01-23T18:25:45.200" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="26833" PostTypeId="1" CreationDate="2018-01-19T15:53:48.463" Score="28" ViewCount="18073" Body="&lt;p&gt;When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure) instead of validation loss?&lt;/p&gt;&#xA;&lt;p&gt;All the examples I have seen so far are similar to this one:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42945" LastEditorUserId="29169" LastEditDate="2021-03-11T19:19:10.693" LastActivityDate="2021-03-11T19:19:10.693" Title="Is there away to change the metric used by the Early Stopping callback in Keras?" Tags="|machine-learning|neural-network|deep-learning|keras|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26835" PostTypeId="2" ParentId="26833" CreationDate="2018-01-19T16:54:38.243" Score="10" Body="&lt;p&gt;Of course, just create your own!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class EarlyStopByF1(keras.callbacks.Callback):&#xA;    def __init__(self, value = 0, verbose = 0):&#xA;        super(keras.callbacks.Callback, self).__init__()&#xA;        self.value = value&#xA;        self.verbose = verbose&#xA;&#xA;&#xA;    def on_epoch_end(self, epoch, logs={}):&#xA;         predict = np.asarray(self.model.predict(self.validation_data[0]))&#xA;         target = self.validation_data[1]&#xA;         score = f1_score(target, prediction)&#xA;         if score &amp;gt; self.value:&#xA;            if self.verbose &amp;gt;0:&#xA;                print(&quot;Epoch %05d: early stopping Threshold&quot; % epoch)&#xA;            self.model.stop_training = True&#xA;&#xA;&#xA;callbacks = [EarlyStopByF1(value = .90, verbose =1)]&#xA;model.fit(X, y, batch_size = 32, nb_epoch=nb_epoch, verbose = 1, &#xA;validation_data(X_val,y_val), callbacks=callbacks)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I haven't tested this out but that should be the general flavor for how you go about it.  If it doesn't work let me know and i'll retry over the weekend.  I'm also assuming you have your own f1 score implemented already.  If not just import for sklearn.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-19T16:54:38.243" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26836" PostTypeId="1" CreationDate="2018-01-19T17:00:37.540" Score="2" ViewCount="1620" Body="&lt;p&gt;I see a lot of people varying the width of each layer in a deep neural network. ie. Input-&gt;50-&gt;100-&gt;150-&gt;Output. I'm curious what, if any, are the advantages of this structure over static layer widths ie. Input-&gt;50-&gt;50-&gt;50-&gt;Output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the following three model structures what would be the real-world implications and differences between them and how they interpret data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example 1: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Input -&gt; 100 -&gt; 100 -&gt; 100 -&gt; Output&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Example 2: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Input -&gt; 50 -&gt; 100 -&gt; 150 -&gt; Output&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Example 3: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Input -&gt; 150 -&gt; 100 -&gt; 50 -&gt; Output&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="44271" LastEditorUserId="28175" LastEditDate="2018-01-20T14:46:23.770" LastActivityDate="2018-05-20T17:47:01.923" Title="Should the different layers of deep learning models have same size or they should be changed based on a rule" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26837" PostTypeId="2" ParentId="26811" CreationDate="2018-01-19T17:01:24.493" Score="5" Body="&lt;p&gt;SKLearn is friendly on this. Simply with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_selection import SelectFromModel&#xA;selection = SelectFromModel(gbm, threshold=0.03, prefit=True)&#xA;selected_dataset = selection.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you will get a dataset with only the features of which the importance pass the threshold, as Numpy array. Point that the threshold is relative to the total importance, so it goes from 0 to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to visualize the importance, maybe to manually select the features you want, you can do like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgb.plot_importance(booster=gbm ); plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-19T17:01:24.493" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26838" PostTypeId="1" AcceptedAnswerId="26857" CreationDate="2018-01-19T17:09:44.113" Score="0" ViewCount="1017" Body="&lt;p&gt;I have a very fundamental question on what CNN'S actually are.&#xA;I understand fully the training process as to take a bunch of images, start with random filters, convolve, activate, calculate loss, back propagate and learn weights. Fully understood....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But recently I came across this line on &lt;code&gt;Slack&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;CNN'S can act as a frequency filter as well&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example, a blur is a low-pass filter and it can be implemented as convolution with fixed weights;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please Explain? (Can't understand this at all)&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="28175" LastEditDate="2018-01-20T12:08:39.017" LastActivityDate="2018-02-02T12:26:27.003" Title="CNN'S are what?" Tags="|machine-learning|neural-network|deep-learning|cnn|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26839" PostTypeId="2" ParentId="26838" CreationDate="2018-01-19T17:28:51.213" Score="4" Body="&lt;p&gt;I suspect this is referring to kernels that are used in image processing for a variety of tasks. Without seeing the conversation, this is my interpretation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea of a kernel in image processing is that you take a grid (maybe 3x3) and apply it to all of the locations in an image. The way you apply it is by placing it at a pixel location. Then, the 9 pixels covered by the kernel are multiplied by its weights and summed. That value is placed at that pixel location in the new image. The picture below shows it nicely. &lt;a href=&quot;https://developer.apple.com/library/content/documentation/Performance/Conceptual/vImage/Art/kernel_convolution.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yOWsr.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yOWsr.jpg&quot; alt=&quot;Example of how a kernel is applied &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see how this is very much like how a convolutional filter in a CNN is applied. A set of example kernels is listed on &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_(image_processing)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;. These kernels perform actions such as blurring and edge detection, so if you had a filter in your CNN with those weights, it would perform the same action.&lt;/p&gt;&#xA;" OwnerUserId="43846" LastActivityDate="2018-01-19T17:28:51.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26840" PostTypeId="1" CreationDate="2018-01-19T17:46:00.033" Score="1" ViewCount="68" Body="&lt;p&gt;I have been learning about segmentation models recently and was reading this paper today: &lt;a href=&quot;https://arxiv.org/pdf/1705.08881.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dense Transformer Networks&lt;/a&gt;. On page 2, it describes segmentation as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Given the success of deep learning methods on image-related applications, numerous recent attempts&#xA;  have been made to solve dense prediction problems using CNNs. A central idea of these methods is&#xA;  to extract a square patch centered on each pixel and apply CNNs on each of them to compute the label&#xA;  of the center pixel. The efficiency of these approaches can be improved by using fully convolutional&#xA;  or encoder-decoder networks. Specifically, fully convolutional networks [23] replace fully connected&#xA;  layers with convolutional layers, thereby allowing inputs of arbitrary size during both training and&#xA;  test. In contrast, deconvolution networks [25] employ an encoder-decoder architecture. The encoder&#xA;  path extracts high-level representations using convolutional and pooling layers. The decoder path&#xA;  uses deconvolutional and up-pooling layers to recovering the original spatial resolution. In order to&#xA;  transmit information directly from encoder to decoder, the U-Net [28] adds skip connections [14]&#xA;  between the corresponding encoder and decoder layers. A common property of all these methods is&#xA;  that the label of any pixel is determined by a regular (usually square) patch centered on that pixel.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, I know that the models used take a single image as an input and output a segmentation mask for the entire image as an output. Are the authors here writing that the underlying operation being performed by the models is to extract patches and classify them, while this is all done in parallel? I would assume these patches correspond to the receptive field of the network, if that is what they are saying.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My overall question is just hoping for an explanation of how this statement is actually what is being performed in a segmentation model.&lt;/p&gt;&#xA;" OwnerUserId="43846" LastEditorUserId="29169" LastEditDate="2021-02-13T16:46:42.770" LastActivityDate="2021-02-13T16:46:42.770" Title="What principle is behind semantic segmenation with CNNs?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26841" PostTypeId="2" ParentId="26836" CreationDate="2018-01-19T17:46:52.057" Score="0" Body="&lt;p&gt;It's really dependent on the problem you are solving. Neural network layers attempt to get more and more abstract representations of your input, tailored towards your output. So the factors that influence your output need to be able to be represented well at each abstraction level. Each abstraction level represents more and more non-linear interactions between your inputs and hopefully less and less noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take convolutional neural networks as an example, because it's slightly easier to interpret what is happening. The first levels are looking for small patterns while the later levels are looking for combinations of these smaller patterns that are relevant. If there are a lot of different small patterns that are relevant for your task but the later layers are only looking for a few different combinations of these filters your 3rd example would be appropriate. On the other hand, if you have some edge detectors and then some basic shapes in the first few abstraction layers, but you are trying to distinguish between 1000 classes that are all built up from these basic shapes, then you would need a more extreme variant of the second example. The first example is somewhere in between.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-01-19T17:46:52.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26843" PostTypeId="1" AcceptedAnswerId="26862" CreationDate="2018-01-19T18:41:57.753" Score="1" ViewCount="1182" Body="&lt;p&gt;For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If label ordering is &lt;code&gt;[apple, pear, big, yellow]&lt;/code&gt; then a “&lt;strong&gt;big yellow apple&lt;/strong&gt;” picture should have the label &lt;code&gt;[1,0,1,1]&lt;/code&gt;. First two part is mutually exclusive, one-hot, however last two are not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, what loss function should I use for this problem?&lt;/p&gt;&#xA;" OwnerUserId="41156" LastEditorUserId="28175" LastEditDate="2018-01-21T04:20:03.060" LastActivityDate="2018-01-21T04:20:03.060" Title="What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?" Tags="|machine-learning|classification|deep-learning|multiclass-classification|multilabel-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26844" PostTypeId="1" CreationDate="2018-01-19T18:58:45.890" Score="1" ViewCount="1118" Body="&lt;p&gt;Currently, I have a balanced dataset (that I artificially over-sampled to make it balanced). My classes are binary (0 or 1). I'm wondering if &quot;accuracy&quot; is the &quot;best&quot; metric to use in the situation when a dataset has roughly equal class balance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More broadly though, is there a good &quot;rule-of-thumb&quot; or &quot;best practice&quot; regarding which scoring metric (accuracy, precision, recall, AUC, ...) to use based on the dataset's class label &quot;distribution&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Common scenarios I can think of are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binary Classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;high imbalance of 0 class&lt;/li&gt;&#xA;&lt;li&gt;high imbalance of 1 class&lt;/li&gt;&#xA;&lt;li&gt;roughly equal class quantities&lt;/li&gt;&#xA;&lt;li&gt;last, imagine a 75/25 ratio for either&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multiclass Classification&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Majority of samples are dominated by one class (very common in my experience)&lt;/li&gt;&#xA;&lt;li&gt;Classes have roughly equally the same amount&lt;/li&gt;&#xA;&lt;li&gt;Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm aware of techniques like under-sampling and over-sampling to deal with imbalance by altering the dataset's number of samples, but let's say you didn't in this scenario. Which metrics are best for the above scenarios?&lt;/p&gt;&#xA;" OwnerUserId="39547" LastEditorUserId="28175" LastEditDate="2018-01-20T13:59:20.743" LastActivityDate="2018-01-20T13:59:20.743" Title="Which Classification Metrics Are Appropriate For Each Class Distribution Scenario?" Tags="|classification|deep-learning|multiclass-classification|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26845" PostTypeId="1" AcceptedAnswerId="26864" CreationDate="2018-01-19T20:13:37.830" Score="3" ViewCount="498" Body="&lt;p&gt;I have to use a neural network to classify whether some reviews of hotels are deceptive or truthful. I also have to use pre-trained word embeddings to fed the neural networks. So I can use Word2vec to get the word vectors from a way larger dataset of hotels reviews. However, Word2vec gives the possibility to use continuous bag-of-words and continuous skip-gram models for this task. Which one would be generally better for this specific task?&lt;/p&gt;&#xA;" OwnerUserId="44884" LastActivityDate="2018-02-19T16:15:44.073" Title="What's the proper Word2vec model to get pre-trained word embedding for a classification task?" Tags="|neural-network|word2vec|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26846" PostTypeId="1" CreationDate="2018-01-20T01:29:01.677" Score="1" ViewCount="55" Body="&lt;p&gt;This might make you feel like I am looking for a recommender engine, but I am not. A recommender engine works well if accuracy isn't an issue, but in my case, it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have proposed is to associate tags with certain question's answers, and map those tags to a particular video(s) and discern based on the mapping, whether the person should see it. Personalization. What do you think?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the topic at hand to put this into context. Each video deals with a particular workout and is to be served depending on the answers in the onboarding screen. It then forms part of a program. &lt;/p&gt;&#xA;" OwnerUserId="44898" LastEditorUserId="29575" LastEditDate="2018-01-20T02:02:58.643" LastActivityDate="2018-01-20T04:02:33.027" Title="I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?" Tags="|data-mining|recommender-system|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26848" PostTypeId="2" ParentId="26846" CreationDate="2018-01-20T03:13:38.610" Score="3" Body="&lt;p&gt;Frame this as a classification problem and learn a decision tree to map question responses to video selections.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Fleshing this out a bit more:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Collect appropriate data&lt;/strong&gt;. Get members of your target population to complete the survey and also indicate which videos they think would be appropriate to them, or alternatively have subject matter experts associate surveys with appropriate videos. However you do it, you need to acquire a dataset pairing surveys with &quot;ground truth&quot; video suggestions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;For each video, train a classifier&lt;/strong&gt; with the survey responses as input to predict the binary target variable of whether or not that video was suggested for that survey.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;To construct a suggested set for a new survey, use all your classifiers to score it. Any classifier that outputs a 1 is a video you should send in response to that survey.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-20T04:02:33.027" LastActivityDate="2018-01-20T04:02:33.027" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="26849" PostTypeId="1" CreationDate="2018-01-20T08:44:56.260" Score="1" ViewCount="186" Body="&lt;p&gt;The &lt;a href=&quot;http://www.cs.sfu.ca/~jpei/publications/sigmod00.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;FP-Growth Mining Algorithm&lt;/a&gt; finds frequent itemsets i.e. it does not search for patterns by item. If I want to find ALL frequent itemsets for an item, I can use the nodelink property to traverse all paths in the fp-tree for that item, which means I have to traverse all paths for all children of that item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the worst case this effectively means I have to run the mining algorithm on the entire fp-tree (and then filter the results for the specific item I'm looking for). This seems like an expensive operation to perform repeatedly, as I'm only looking for a small subset of results. Is there any other mechanism that would allow me to mine patterns selectively?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;sub&gt;1) &lt;a href=&quot;http://www.cs.sfu.ca/~jpei/publications/sigmod00.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mining Frequent Patterns without Candidate Generation; Jiawei Han, Jian Pei, and Yiwen Yin&lt;/sub&gt;&#xA;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44901" LastEditorUserId="29575" LastEditDate="2018-01-21T17:02:56.197" LastActivityDate="2024-03-01T13:07:05.857" Title="FP-Growth - find ALL patterns containing a specific item(s) only" Tags="|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26850" PostTypeId="1" AcceptedAnswerId="26873" CreationDate="2018-01-20T09:31:11.183" Score="4" ViewCount="1608" Body="&lt;p&gt;I have designed NB classifier from scratch in python for binary classification problem. There are total 220 records out of which 85 records belongs to 'Yes' class and 135 to 'No' class. My classifier is giving 88% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, whenever I calculate the posterior probability of one sample that belongs to 'Yes' class it is very low in terms of numbers. For eg., I'm making a prediction that whether the batsman is rising star (i.e. the probability of sample belongs to 'Yes' class ). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here The posterior probability of being rising star i.e. P(RS) is very low in numbers something like 2.33E-8. But the posterior probability of being not rising star is also very low similar in the range of E-8 to E-16. Some features I'm using to calculate posterior probability are also small values in the range of 0.1 to 0.01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to represent posterior probability i.e. P(RS) in terms of percentage. Like P(RS)=90%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: I googled this problem and tried log method which returns a negative value.&lt;/p&gt;&#xA;" OwnerUserId="43679" LastEditorUserId="43679" LastEditDate="2018-01-21T04:44:24.470" LastActivityDate="2018-01-21T04:44:24.470" Title="Very low probability in naive Bayes classifier" Tags="|machine-learning|prediction|probability|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26851" PostTypeId="1" CreationDate="2018-01-20T09:54:35.570" Score="2" ViewCount="59" Body="&lt;p&gt;I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available would constitute a privacy leak. I am considering making a moving average available instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone recommend either some literature on this, or some alternative methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that this is a case by case question. However, I think there should be a general answer available along the following lines:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Privacy leaks occur because you can match up the time stamp to an individual, by using outside information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Therefore, you want to make it so that each window aggregates the data of several individuals. (The data is of a form where the mean is a meaningful quantity.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are of course ways to break this privacy, if one is sufficiently determined. I think in this case no one is. So I'm looking for literature that deals with some real world case studies, if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This situation is hypothetical. I do not have access to the data. I am 'the public' that wants the data, and I want to suggest a reasonable approach for aggregation.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also asked this question over on cross validated: &lt;a href=&quot;https://stats.stackexchange.com/questions/324204/privacy-through-moving-averages&quot;&gt;https://stats.stackexchange.com/questions/324204/privacy-through-moving-averages&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44063" LastEditorUserId="44063" LastEditDate="2018-01-21T03:03:47.713" LastActivityDate="2018-01-23T23:31:56.053" Title="Privacy through moving averages?" Tags="|time-series|aggregation|privacy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26852" PostTypeId="2" ParentId="26838" CreationDate="2018-01-20T09:59:05.513" Score="2" Body="&lt;p&gt;As you say, CNN convolves. In signal processing the convolution is with continuous data, here it is with discrete values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Are-convolutional-neural-networks-related-to-the-convolution-of-two-functions-in-mathematics&quot; rel=&quot;nofollow noreferrer&quot;&gt;This article in Quora&lt;/a&gt; explains, how the mathematic expression in neural network forms the same equation than that of two functions in mathematics.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mathematically, applying a Gaussian blur to an image is the same as convolving the image with a Gaussian function.&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Gaussian_blur&quot; rel=&quot;nofollow noreferrer&quot;&gt;BLUR in wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Gaussian_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian function&lt;/a&gt; is higher on lower values so it kinda is a low pass filter when applied with convolution. &lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-01-20T09:59:05.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26853" PostTypeId="2" ParentId="26850" CreationDate="2018-01-20T10:23:49.123" Score="1" Body="&lt;p&gt;The question is a bit fuzzy so if i did not get the point please comment me.&#xA;You have a binary classification problem and you want to solve it using NB. Well, then you go through Bayes formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(class|data)=P(data|class)P(class)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the representation of each class GIVEN a sample point i.e. this representation varies according to feature values. The simple percentage of each class does not need NB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the probability of predictions for one class on validation set is low it means that they were not from that class! For more detailed answer please say how did you set up your train/test split and what were these small numbers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If some numbers are strange then you may need to attach your code as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is how to represent posterior probability i.e. P(RS) in terms of percentage. Like P(RS)=90%.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When you get probabilities out then 100 times of it is percentage. So let's focus on probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above formulation is simplified for classification. The complete formulation is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(class|data)=\frac{P(data|class)P(class)}{P(data)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the denumerator is a probability, so it is between 0 and 1, it increases the whole fraction but it's not used in classification task as it's practically constant! Therefore as you want to see the probability (percentage) of your posterior, maybe including it in your formula solves the your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-01-20T20:38:21.600" LastActivityDate="2018-01-20T20:38:21.600" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26854" PostTypeId="1" AcceptedAnswerId="26855" CreationDate="2018-01-20T10:25:00.973" Score="6" ViewCount="7650" Body="&lt;p&gt;I am using CNN in order to predict codes after analyzing text. As an example, I will write &quot;I am crazy&quot; .. the model will predict some code &quot; X321&quot;.&#xA; All this based on CNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to evaluate my model. I used Fscore (recall and precision). Can you advice me more metrics?&lt;/p&gt;&#xA;" OwnerUserId="40426" LastEditorUserId="28175" LastEditDate="2018-01-21T07:53:44.773" LastActivityDate="2020-03-31T23:47:25.397" Title="What are useful evaluation metrics used in machine learning" Tags="|machine-learning|neural-network|deep-learning|nlp|text-mining|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26855" PostTypeId="2" ParentId="26854" CreationDate="2018-01-20T11:40:21.353" Score="4" Body="&lt;p&gt;For model evaluation there are different metrics based on your model:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Confusion matrix&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Classification accuracy&lt;/strong&gt;:&lt;br /&gt;&#xA;(TP + TN) / (TP + TN + FP + FN)&lt;br /&gt;&#xA;&lt;strong&gt;Error rate&lt;/strong&gt;: &lt;br /&gt;&#xA;(FP + FN) / (TP + TN + FP + FN)&lt;br /&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Paired criteria&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Precision&lt;/strong&gt;: (or Positive predictive value)&lt;br /&gt;&#xA;proportion of predicted positives which are actual positive&lt;br /&gt;&#xA;TP / (TP + FP)&lt;br /&gt;&#xA;&lt;strong&gt;Recall&lt;/strong&gt;: proportion of actual positives which are predicted positive&lt;br /&gt;&#xA;TP / (TP + FN)&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Sensitivity&lt;/strong&gt;: proportion of actual positives which are predicted positive&lt;br /&gt;&#xA;TP / (TP + FN)&lt;br /&gt;&#xA;&lt;strong&gt;Specificity&lt;/strong&gt;: proportion of actual negative which are predicted negative&lt;br /&gt;&#xA;TN / (TN + FP)&lt;br /&gt;&#xA;&lt;strong&gt;True positive rate&lt;/strong&gt;: proportion of actual positives which are predicted&#xA;positive &lt;br /&gt;&#xA;TP / (TP + FN)&lt;br /&gt;&#xA;&lt;strong&gt;True negative rate&lt;/strong&gt;: proportion of actual negative which are predicted&#xA;negative&lt;br /&gt;&#xA;TN / (TN + FP)&lt;br /&gt;&#xA;&lt;strong&gt;Positive likelihood&lt;/strong&gt;: likelihood that a predicted positive is an actual positive &lt;br /&gt;&#xA;sensitivity / (1 - specificity)&lt;br /&gt;&#xA;&lt;strong&gt;Negative likelihood&lt;/strong&gt;: likelihood that a predicted negative is an actual&#xA;negative&lt;br /&gt;&#xA;(1 - sensitivity) / specificity&lt;br /&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Combined criteria&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BCR&lt;/strong&gt;: Balanced Classification Rate&lt;br /&gt;&#xA;½ (TP / (TP + FN) + TN / (TN + FP))&lt;br /&gt;&#xA;&lt;strong&gt;BER&lt;/strong&gt;: Balanced Error Rate, or HTER&lt;br /&gt;&#xA;&lt;strong&gt;Half Total Error Rate&lt;/strong&gt;: 1 - BCR&lt;br /&gt;&#xA;F-measure harmonic mean between precision and recall &lt;br /&gt;&#xA;2 (precision . recall) / (precision + recall)&lt;br /&gt;&#xA;&lt;strong&gt;Fβ-measure&lt;/strong&gt; weighted harmonic mean between precision and recall&lt;br /&gt;&#xA;(1+β)2 TP / ((1+β)2 TP + β2 FN + FP)&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The harmonic mean between specificity&#xA;  and sensitivity is also often used and&#xA;  sometimes referred to as F-measure.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Youden's index: arithmetic mean&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;between sensitivity and specificity&lt;/strong&gt;&lt;br /&gt;&#xA;sensitivity - (1 - specificity)&lt;br /&gt;&#xA;&lt;strong&gt;Matthews correlation&lt;/strong&gt;: correlation between the actual and predicted&lt;br /&gt; &#xA;(TP . TN – FP . FN) / ((TP+FP) (TP+FN) (TP + FP) (TN+FN)) ^ (1/2) &lt;br /&gt;&#xA;comprised between -1 and 1 Discriminant power normalized likelihood index&#xA;sqrt(3) / π .&#xA;(log (sensitivity / (1 – specificity)) +&#xA;log (specificity / (1 - sensitivity)))&#xA;&amp;lt;1 = poor, &gt;3 = good, fair otherwise&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can find much more &lt;a href=&quot;http://www.damienfrancois.be/blog/files/modelperfcheatsheet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Also there are some explanations &lt;a href=&quot;https://www.dezyre.com/data-science-in-python-tutorial/performance-metrics-for-machine-learning-algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and you can find useful code snippet from &lt;a href=&quot;https://machinelearningmastery.com/metrics-evaluate-machine-learning-algorithms-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which are implemented. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-20T11:40:21.353" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26857" PostTypeId="2" ParentId="26838" CreationDate="2018-01-20T12:06:07.983" Score="2" Body="&lt;p&gt;If I've got the point of your question, it means that convolutional nets can learn different filters. You may train a net with random initialization and after training you can find learned-filters which are familiar to you or not. You may see that your model has learned &lt;code&gt;Sobel&lt;/code&gt; filter which is a high pass filter or maybe you may see that the weights which are learned are exactly the same as a mean filter which is a low pass filter. In all cases network tries to learn filters that help it find appropriate features of the inputs. The learned filters can be anything. They may be known to you or not. So, yes. They can learn all kind of features like low pass and high pass filters. Also for visualization purposes to fully figure out what is going on inside them I recommend you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-20T12:06:07.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26859" PostTypeId="2" ParentId="26588" CreationDate="2018-01-20T13:14:00.880" Score="0" Body="&lt;p&gt;I used my existing directory structure with &lt;a href=&quot;https://github.com/git-lfs/git-lfs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Git LFS&lt;/a&gt; and it works well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the advantages I got by using Git LFS other than the alternative solutions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;It saves space&lt;/strong&gt;: Since Git LFS manages my dataset files (100MB to 1GB) with pointers and keeping only the minimum in the local memory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;It works seamlessly with a git enabled repository&lt;/strong&gt;: I was planning to use some storage solution (like Amazon S3) specific for my datasets. But I would have to make extra efforts to manage multiple version and keep my dataset files in sync.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9388" LastEditorUserId="9388" LastEditDate="2018-01-22T08:15:48.957" LastActivityDate="2018-01-22T08:15:48.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26861" PostTypeId="2" ParentId="26844" CreationDate="2018-01-20T13:55:44.893" Score="2" Body="&lt;p&gt;First of all don't change the distribution of your data. Your classifier won't perform good at test time if the real data is not balanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have approximately equal amount of data for each class and they have same importance, accuracy is a good metric which can help you understand how much your classifier has performed well without regarding to the number of labels, binary or multi-class classificaitons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Binary classification&lt;/strong&gt;&lt;br /&gt;&#xA;There is no difference between 0 or 1 because you yourself choose to assign each class to a label but it is common to set the rare and maybe, in some occasions, the desired output to 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In cases which data is distributed badly and it is so much skewed it recommended to use &lt;code&gt;F1&lt;/code&gt; score. For &lt;em&gt;last, imagine a 75/25 ratio for either&lt;/em&gt; I personally prefer to use &lt;code&gt;F1&lt;/code&gt; because it is unbalanced.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in cases where one class is much important than the other class and maybe the data is skewed, it is better to use &lt;code&gt;recall&lt;/code&gt;, although you have to change the cost function to emphasize the importance of one class over the other. As an example, suppose that you want to know whether a patient has a bad illness or not. In such cases you try to predict correctly all of those who has that illness because if you tell that someone has an illness but you're wrong, it will have less harm than saying a patient is healthy. So in these cases &lt;code&gt;recall&lt;/code&gt; is so much better than &lt;code&gt;precision&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Multi-Class classification&lt;/strong&gt;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Majority of samples are dominated by one class (very common in my experience)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case first of all, if the data is so much skewed, I suggest you using anomaly detection. But if you have more than &lt;em&gt;few&lt;/em&gt; data, you can again use &lt;code&gt;F1&lt;/code&gt; score. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Classes have roughly equally the same amount&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case you can use accuracy if the classes have the same importance. If they don't, again use accuracy but change the cost function to emphasize on the important class.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In this case I don't think you will have a good learning. Anyway, you have to use confusion matrix in this for tracking all classes. Although I guess your model will be over-fitted because of the amount of provided data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;Consider that in all cases you can use &lt;em&gt;confusion matrix&lt;/em&gt;.&#xA;&lt;hr /&gt;&#xA;Finally I suggest you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26855/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.damienfrancois.be/blog/files/modelperfcheatsheet.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-20T13:55:44.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26862" PostTypeId="2" ParentId="26843" CreationDate="2018-01-20T14:23:45.067" Score="1" Body="&lt;p&gt;The labels of data are not mutually exclusive so you can't say this is a one vs. all problem, because more than one entry may be one in the output vector. Moreover, if in the seen there should be an apple or pear this can be considered as an exhaustive problem which means one of them should happen for each input. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My opinion is that for this problem you don't have to make a new cost function. For the mutually exclusive part, as you have truly stated that, and for the second part of your vector, the well known &lt;code&gt;cross-entropy&lt;/code&gt; cost function will perform fine. I guess the problem is something else. For problems with mutually exclusive classes, we use &lt;a href=&quot;https://datascience.stackexchange.com/q/25315/28175&quot;&gt;soft max layer&lt;/a&gt; as the last layer for neural nets while for cases which classes are not mutually exclusive, you can use &lt;code&gt;sigmoid&lt;/code&gt; as the activation function. In your case that you have combination of them I suggest you an alternative approach:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change your mutually exclusive part to a binary output, means if the corresponding entry is less than half, you can understand that it is e.g. an apple otherwise, it is the other class. and for the rest, just keep the output vector as it is. Finally use &lt;code&gt;sigmoid&lt;/code&gt; activation function as the last layer if you are using neural nets.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-20T14:23:45.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26863" PostTypeId="2" ParentId="26836" CreationDate="2018-01-20T14:45:14.473" Score="1" Body="&lt;p&gt;Actually it is not completely clear which deep neural nets you are referring to but I guess you are referring to Dense, aka fully connected, networks. It depends on your data but for simplification, based on the answer &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt;, neurons of the first hidden layer try to find lines for separating the data, and the neurons of the subsequent layers try to consider them simultaneously, I suggest reading the hole answer. As you increase the number of neurons in the first layer, you try to increase the number of lines for separation, The last shape of the link, and as you try to increase the number of e.g. the second layer, the number of convex shapes will increase. If you increase them too much you will over-fit your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Increasing the number of filters, neurons or kernels, in convolutional nets has another meaning. If you increase the number of filters, it means that you try to find more features that can represent your data better. In conv layers you actually try to find features of data. Although you are somehow doing same procedure in a dense net but because the neurons are less connected and they get updated similarly in conv nets, the features learned by conv nets are completely different from dense layers. for more information I recommend you taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-20T14:45:14.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26864" PostTypeId="2" ParentId="26845" CreationDate="2018-01-20T16:11:20.673" Score="1" Body="&lt;p&gt;I think this article gives general idea of pros / cons between CBOW and Skip Gram,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-are-the-continuous-bag-of-words-and-skip-gram-architectures&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;According to Mikolov:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Skip-gram: works well with small amount of the training data,&#xA;  represents well even rare words or phrases.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;CBOW: several times faster&#xA;  to train than the skip-gram, slightly better accuracy for the frequent&#xA;  words.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So without looking at data, assuming that you have large amount of data, CBOW might be my bet. However, if you have labels for &quot;deceptive&quot; and &quot;truthful&quot;, it will be best to train both models, and use whichever outperforms via cross-validation.&lt;/p&gt;&#xA;" OwnerUserId="2869" LastActivityDate="2018-01-20T16:11:20.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26865" PostTypeId="1" CreationDate="2018-01-20T16:17:12.200" Score="0" ViewCount="66" Body="&lt;p&gt;In the below figure, we plotted some data of sensors in normal condition and under attack(outlier):&#xA;1.  Green points are normal samples in the training dataset.&#xA;2.  Cyan and red points are normal samples in the test dataset.&#xA;3.  Black and blue points are attack samples in the test dataset.&#xA;We wish to classify attack from normal samples in the test phase by using only the information we extract from the training dataset (the green points). I highly appreciate any suggestions for this purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/X6I8M.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X6I8M.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="8878" LastEditDate="2018-01-23T14:15:20.417" LastActivityDate="2018-01-23T14:44:53.170" Title="Differentiating between normal and attack clusters" Tags="|machine-learning|classification|clustering|visualization|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26867" PostTypeId="2" ParentId="26323" CreationDate="2018-01-20T17:43:23.677" Score="0" Body="&lt;p&gt;Your assumption seems to be correct that both approaches are actually equivalent. I can not see any difference here in the propabilities and conditional propabilities for either one way or the other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why the predictions should also be the same for both ways. It is more likely that there is something wrong with your implementation.&lt;/p&gt;&#xA;" OwnerUserId="44372" LastActivityDate="2018-01-20T17:43:23.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26868" PostTypeId="1" CreationDate="2018-01-20T17:46:04.680" Score="1" ViewCount="1879" Body="&lt;p&gt;I created a two layered fully connected neural network as a part of a recommendation engine (after I use embedding layers for products and users). I have been trying to tune the hyper-parameters for the past couple of weeks now. Additionally I am using 5 fold cross validation and 'mse' as the error metric. Below is the curve for training and cross-validation error that I obtain post training on a GPU for 10 epochs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9Jcvq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9Jcvq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure what really to make of this, fundamentally. The model doesn't overfit or underfit the data. What is even more perplexing is that the validation error is across five folds. Is this an inherent bias in the data and alludes to the fact that the data is structured such that a weak model does well on unseen data ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers or comments will be greatly helpful. Thanks !&lt;/p&gt;&#xA;" OwnerUserId="42182" LastActivityDate="2018-01-22T03:52:32.983" Title="Training error is oscillating, cross validation error is continuously decreasing" Tags="|python|neural-network|deep-learning|keras|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26869" PostTypeId="2" ParentId="26865" CreationDate="2018-01-20T17:55:19.887" Score="1" Body="&lt;p&gt;First of all please ignore my previous answer. I did not understand you properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difficulty of the your problem goes back to the distribution of normal points in train and test. The distribution of new normal points are so different than what your model can learn from only green points (training set) so I assume any model will have a relatively poor performance on it. Anyways, having a look at &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#svm-outlier-detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn documentation&lt;/a&gt; is good if you did not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About &lt;strong&gt;Clustering&lt;/strong&gt; as you asked I have an idea. An active are in &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graph&lt;/a&gt; Analysis and &lt;a href=&quot;http://barabasi.com/networksciencebook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Science&lt;/a&gt; is &lt;a href=&quot;http://barabasi.com/networksciencebook/chapter/9&quot; rel=&quot;nofollow noreferrer&quot;&gt;Community Detection&lt;/a&gt; (which is simply the clustering of graph nodes). The idea is that, assume all your points in this figure are nodes of a graph and based on their Euclidean distance in PC space, you put an edge between them (either binary or weighted). Then you can use &lt;a href=&quot;https://arxiv.org/abs/0803.0476&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modularity-based algorithms&lt;/a&gt; to find dense subgraphs. I assume if you apply it to your data it will work. Also a similar idea is &lt;a href=&quot;https://www.cs.cmu.edu/~aarti/Class/10701/slides/Lecture21_2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spectral Clustering&lt;/a&gt; which internally uses the idea of graph partitioning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the density of clusters, I would also propose to use &lt;a href=&quot;https://en.wikipedia.org/wiki/DBSCAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;density-based clustering&lt;/a&gt; approaches. They might be able to distinguish between those overlapping clusters &quot;a little bit&quot; (e.g. this &lt;a href=&quot;http://www.cs.sfu.ca/~ester/papers/SDM2006.DenStream.final.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;). &lt;a href=&quot;http://sites.psu.edu/mcnl/files/2017/03/9-2dhti48.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper&lt;/a&gt; has an approach to find the peaks of densities. Sounds compatible with your data.&#xA;Please note that those overlapping clusters are anyways difficult to distinguish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My apologies for the first misunderstanding and hope it helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-01-23T14:44:53.170" LastActivityDate="2018-01-23T14:44:53.170" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="26871" PostTypeId="1" AcceptedAnswerId="26876" CreationDate="2018-01-20T19:59:10.853" Score="1" ViewCount="2666" Body="&lt;p&gt;The model below reads in data from a csv file (date, open, high, low, close, volume), arranges the data and then builds a LSTM model trying to predict next day's close based on a number of previous days close values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, validation accuracy is about 53.8% no matter if i...&#xA;- change hyperparameters&#xA;- make it a deep model&#xA;- uses many more features than just close&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tRVvn.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tRVvn.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To test if I made a simple mistake I generated another data source that was a signal I created from adding sin, cosine and a little noise so that i KNEW a model should be able to be trained and it was. The model below got about 94% validation without any tuning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With that in mind. How come when I try to use it on actual data (eurusd 1-min data) it doesn't seem to work..?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone that sees an error or can point me in the right direction?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;fpath = 'Data/'&#xA;fname = 'EURUSD_M1_1'&#xA;df = pd.read_csv(fpath + fname + '_clean.csv')&#xA;# file contains date, open, high, low, close, volume&#xA;&#xA;# &quot;y&quot; is whether the next period's Close value is higher or lower than current Close value&#xA;outlook = 1&#xA;df['y'] = df['Close']&amp;lt;df['Close'].shift(-outlook)&#xA;&#xA;# Drop all NAN's&#xA;df.dropna(how=&quot;any&quot;,inplace=True)&#xA;&#xA;# Get X and y. To keep it simple, just use Close&#xA;X_df = df['Close']&#xA;&#xA;# &quot;y&quot; is whether the next period's Close value is higher or lower than current Close value&#xA;outlook = 1&#xA;y_df = df['Close']&amp;lt;df['Close'].shift(-outlook)&#xA;&#xA;# Train/test split&#xA;def train_test_split(X_df,y_df,train_perc):&#xA;    idx = int(train_perc/100*X_df.shape[0])&#xA;    X_train_df = X_df.iloc[0:idx]&#xA;    X_test_df = X_df.iloc[idx:]&#xA;    y_train_df = y_df.iloc[0:idx]&#xA;    y_test_df = y_df.iloc[idx:]&#xA;    return X_train_df.as_matrix(), X_test_df.as_matrix(), y_train_df.as_matrix(), y_test_df.as_matrix()&#xA;&#xA;X_train_df, X_test_df, y_train, y_test = train_test_split(X_df,y_df,90)&#xA;&#xA;# Scaling&#xA;def scale(X):&#xA;    Xmax = max(X)&#xA;    Xmin = min(X)&#xA;    return (X-Xmin)/(Xmax - Xmin)&#xA;&#xA;X_train_scaled = scale(X_train_df)&#xA;X_test_scaled = scale(X_test_df)&#xA;&#xA;# Build the model&#xA;import tensorflow as tf&#xA;from keras.models import Sequential&#xA;from keras.layers import LSTM, Dense&#xA;&#xA;# ### Constants&#xA;num_time_steps = 5   # Num of steps in batch (also used for prediction steps into the future)&#xA;num_features = 1   # Number of features&#xA;num_neurons = 97&#xA;num_outputs = 1 # Just one output (True/False), predicted time series&#xA;learning_rate = 0.0001 # learning rate, 0.0001 default, but you can play with this&#xA;nb_epochs = 10 # how many iterations to go through (training steps), you can play with this&#xA;batch_size = 32&#xA;&#xA;# Reshaping&#xA;X_train_scaled = np.reshape(X_train_scaled,[-1,1])&#xA;nb_samples_train = X_train_scaled.shape[0]  - num_time_steps&#xA;&#xA;X_train_scaled_reshaped = np.zeros((nb_samples_train, num_time_steps, num_features))&#xA;y_train_reshaped = np.zeros((nb_samples_train))&#xA;&#xA;for i in range(nb_samples_train):&#xA;    y_position = i + num_time_steps&#xA;    X_train_scaled_reshaped[i] = X_train_scaled[i:y_position]&#xA;    y_train_reshaped[i] = y_train[y_position]&#xA;&#xA;model = Sequential()&#xA;&#xA;stacked = False&#xA;if stacked == True:&#xA;    model.add(LSTM(num_neurons, return_sequences=True, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))&#xA;    model.add(LSTM(num_neurons, activation='relu', dropout=0.5))&#xA;else:&#xA;    model.add(LSTM(num_neurons, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))&#xA;&#xA;model.add(Dense(units=num_outputs, activation='sigmoid'))&#xA;&#xA;model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics=['accuracy'])&#xA;&#xA;history = model.fit(X_train_scaled_reshaped,&#xA;                    y_train_reshaped,&#xA;                    batch_size = batch_size,&#xA;                    epochs = nb_epochs,&#xA;                    validation_split=0.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38626" LastActivityDate="2019-01-01T22:17:36.417" Title="Can someone spot anything wrong with my LSTM forex model?" Tags="|lstm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26873" PostTypeId="2" ParentId="26850" CreationDate="2018-01-20T20:17:41.610" Score="3" Body="&lt;p&gt;I can't tell you for sure without you describing your calculation more or showing code, but my guess is you're not actually calculating the posterior probability here. I bet this is just the conditional likelihood, or at best the unnormalized posterior. Remember: the posterior calculation has a division component. Does yours? You're probably forgetting to divide by the &quot;evidence&quot;.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-20T20:17:41.610" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26874" PostTypeId="1" AcceptedAnswerId="26905" CreationDate="2018-01-20T20:51:40.460" Score="2" ViewCount="523" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ArQAy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ArQAy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the question which is mentioned in the above picture. It is trying to find the derivative of &lt;code&gt;f&lt;/code&gt; with respect to weight matrix &lt;code&gt;W1&lt;/code&gt;. Can any one help me how I can accomplish that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I solved in this way :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Nzffg.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Nzffg.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/X0GnB.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0GnB.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44477" LastEditorUserId="44477" LastEditDate="2018-01-23T05:51:50.943" LastActivityDate="2018-01-23T06:37:21.387" Title="Compute backpropagation" Tags="|neural-network|deep-learning|backpropagation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26876" PostTypeId="2" ParentId="26871" CreationDate="2018-01-21T01:01:25.947" Score="3" Body="&lt;p&gt;What you are up against is this fundamental property of most tradable, liquid financial price series, and that is, they are Brownian Motion. In discrete time, it's also known as random walk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most important property of Brownian Motion is that it's memoryless, whose mathematical expression is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[p_{t+1} | p_{-{\infty} : t}] = p_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recurrent neural net, particularly the LSTM flavor, is very powerful in capturing and modeling a long memory process. In fact, it was invented to deal with state that depends on itself many time steps ago (that famous LSTM paper was dated 20 yrs ago[1]). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you've demonstrated using RNN (assuming your code and training are free of bug) is precisely this property. Put it another way, there is no way to beat a fair coin in predicting tomorrow's FX price.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another perspective on your attempt. If such naive model could accurately predict FX time series, it would have been exploited by the whales in the hedge fund industry long ago and the opportunity would cease to exist.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] Hochreiter, S., &amp;amp; Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. &lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="27250" LastEditDate="2018-01-21T01:10:30.163" LastActivityDate="2018-01-21T01:10:30.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26877" PostTypeId="1" CreationDate="2018-01-21T03:53:33.837" Score="1" ViewCount="286" Body="&lt;p&gt;I wanted to know how to implement my handmade features in my Keras model.&#xA;Below I have some code where the embedding_matrix is the Stanford Glove NLP dataset. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def BidLstm(maxlen, max_features, embed_size, embedding_matrix):&#xA;   inp = Input(shape=(maxlen, ))&#xA;   x = Embedding(max_features, embed_size, weights=[embedding_matrix],&#xA;                 trainable=False)(inp)&#xA;   x = Bidirectional(LSTM(300, return_sequences=True, dropout=0.25,&#xA;                          recurrent_dropout=0.25))(x)&#xA;   x = Attention(maxlen)(x)&#xA;   x = Dense(256, activation=&quot;relu&quot;)(x)&#xA;   x = Dropout(0.25)(x)&#xA;   x = Dense(6, activation=&quot;sigmoid&quot;)(x)&#xA;   model = Model(inputs=inp, outputs=x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to implement some extra features (preferably numpy array of features) into my keras model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="44936" LastEditorUserId="29575" LastEditDate="2018-01-21T03:58:35.543" LastActivityDate="2018-01-21T12:33:57.237" Title="How to implement handmade features in a Keras Sequential model?" Tags="|python|keras|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26878" PostTypeId="2" ParentId="26868" CreationDate="2018-01-21T09:12:11.877" Score="1" Body="&lt;p&gt;By definition, the training error should always decrease with a proper learning rate until it reaches his plateau, then starts to oscillate because it's crossing forward and backward the minimum of the loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, what you expect from the validation error is that it decrease with the training error, and at some point it starts to increase, sign of over fitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And it is not unusual that at the begin of the training you found the validation error lower than the training error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your plot I would say:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Train and validation errors are simply swapped: the red line is actually the training error, and the blue one is the validation error;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Stop training just after the crossing point. This because the validation set is just a set where you tune your parameters, and not the set of the &quot;Revealed Truth&quot;, so it is also prone to over fitting.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you make a third dataset (&quot;test&quot; dataset), and you try to check your metric (accuracy?) on it, I bet that the model trained after two epochs performs better than the model trained with ten epochs.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="29575" LastEditDate="2018-01-22T03:52:32.983" LastActivityDate="2018-01-22T03:52:32.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26879" PostTypeId="5" CreationDate="2018-01-21T09:38:13.737" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2018-01-21T09:38:13.737" LastActivityDate="2018-01-21T09:38:13.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26880" PostTypeId="4" CreationDate="2018-01-21T09:38:13.737" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2018-01-21T09:38:13.737" LastActivityDate="2018-01-21T09:38:13.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26881" PostTypeId="1" AcceptedAnswerId="30706" CreationDate="2018-01-21T11:56:56.290" Score="11" ViewCount="9402" Body="&lt;p&gt;Let me present you with a toy example and a reasoning on image normalisation I had:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (black) to 255 (white).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class 0:&#xA;Images that contain a filled circle in the centre of the image. The pixel values within the circle are high, close to 255. Pixels outside the circle are low, close to 0. Below is an example of a class 0 picture.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sFbo4.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sFbo4.png&quot; alt=&quot;class 0 example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Class 1:&#xA;Images that are empty, containing pixels close to 0. An example is below, basically a dark image. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H3DXT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H3DXT.png&quot; alt=&quot;class 1 example&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: The images show basically binary values: white/black, however, assume that black values could be in the range of [0, 30] and white ones [230-255] with some random distribution.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to feeding the network with the images, we pre-process and normalize the data. This basically means (i) centring the data and (ii) scaling the data. A typical and simple approach could be to subtract 127.5 to all pixels and scale by 255 so as to have pixel values ranging [-1, 1]. Alternatively, another approach consists of computing the &quot;mean image&quot; and the &quot;dynamic-range image&quot; and centre and scale the images pixel-wise. Let us focus on the latter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Computing the &quot;mean image&quot; mainly means summing all images pixel-wise and taking the average pixel-wise. As a result, we obtain a NxN average image. This can sometimes shed some light on our task and help us understand the data we are working with. Likewise, the &quot;dynamic range image&quot; is obtained by computing the maximum and minimum values pixel-wise. The &quot;dynamic range image&quot; shows us which range of values each pixel can take.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hypothesis 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During training, the kernels of the different layers in the CNN will be trying to detect the circle. Since the task is rather simple, the kernels basically would have to learn that there is a strong pixel-value-transition from &quot;empty region&quot; to &quot;circle region&quot; when a circle is present.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hypothesis 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After normalising the images pixel-wise, we have that all pixels have the same range and same mean, namely [-1, 1] and 0, respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Final Hypothesis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a class 0 image (i.e. image has a circle). Before normalizing, pixel (i, j) has value 30 (dark), which happens to be the maximum that pixel (i, j) ever takes. Pixel (i, j+1) has a value 255 (white), also the maximum value that pixel (i, j+1) ever takes. So we have a strong transition that a kernel from first layers should be able to learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if we normalise the data with &quot;mean image&quot; and &quot;dynamic range image&quot;&#xA;isn't this strong transition going to be softened? In the particular, the transition would be from value 1 to value 1, since both were the maximums at their respective positions. From a clear and hard transition we have ended with a transitioned from similarly valued pixels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&#xA;Using this pixel-wise normalization we end up losing global pixel information, which might make the job of the kernels more difficult. I believe same conclusions might be drawn if &quot;variance image&quot; is used instead of &quot;dynamic range image&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;&#xA;Am I missing some point or is this normalisation no longer used? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bonus&lt;/strong&gt;&#xA;I also thought of Batch Normalisation, which sometimes is used at the first layer so that normalisation is done within the network. I thought that maybe the parameters &quot;beta&quot; and &quot;gamma&quot; used in BN might be able to fix this issue if it ever appears while training. &lt;/p&gt;&#xA;" OwnerUserId="17978" LastEditorUserId="17978" LastEditDate="2018-01-21T14:40:14.920" LastActivityDate="2018-04-23T18:29:12.120" Title="Data preprocessing: Should we normalise images pixel-wise?" Tags="|machine-learning|deep-learning|image-classification|preprocessing|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26883" PostTypeId="2" ParentId="26877" CreationDate="2018-01-21T12:33:57.237" Score="-2" Body="&lt;p&gt;Very interesting  question , I am  wondering if there  are nough skilled  data science  entusiast   that  could   answer this question , because I am  interested in simmilar stuff as well.&#xA;Congratulations  for this  question  man&lt;/p&gt;&#xA;" OwnerUserId="10094" LastActivityDate="2018-01-21T12:33:57.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26884" PostTypeId="2" ParentId="26851" CreationDate="2018-01-21T12:43:44.990" Score="1" Body="&lt;p&gt;Depending on what you're working with here, one approach that might work would be to &quot;jitter&quot; the data. In other words: add noise. Your concern seems to be about people getting recognized from specific timestamp values, so jitter those. depending on the data, you could end up with new timestamps that don't align with any individuals in the data at all while minimally affecting the public's ability to draw inferences from the data.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-21T12:43:44.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26885" PostTypeId="1" CreationDate="2018-01-21T13:03:26.267" Score="5" ViewCount="1437" Body="&lt;p&gt;I want to train a CNN for a multilabel image classification task using keras. However I am not sure how to prepare my tranining data. More specifically, I am wondering if I need training images that show a combination of two or more labels or if it is sufficient to train the network on single labels and it will then be able to detect multiple labels within an image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think my question becomes clearer with an &lt;strong&gt;example&lt;/strong&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I am using the dog vs cat classification dataset and I want to build a model that is able to classify images as either being a dog or a cat or seeing both animals in one image. In this case, do I need to train the model with images showing cats, dogs AND images that show both in one image or is it sufficient to only have training images that only display cats and dogs?&lt;/p&gt;&#xA;" OwnerUserId="44945" LastEditorUserId="28175" LastEditDate="2018-01-21T15:30:13.653" LastActivityDate="2018-01-21T15:30:13.653" Title="Multilabel image classification: is it necessary to have traning data for each combination of labels?" Tags="|machine-learning|neural-network|deep-learning|keras|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26886" PostTypeId="1" CreationDate="2018-01-21T14:17:59.397" Score="4" ViewCount="29476" Body="&lt;p&gt;I am trying to fit my data into my model which takes numpy as input, so I feed the model with the dataframe values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;stacked_averaged_models.fit(train.values, y_train1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the following error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-145-9ba69af8df05&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 X_traintrain = train.as_matrix().astype(np.float)&#xA;      2 from sklearn.metrics import r2_score&#xA;----&amp;gt; 3 stacked_averaged_models.fit(train.values, y_train1)&#xA;      4 stacked_train_pred = stacked_averaged_models.predict(train.values)&#xA;      5 stacked_pred = np.expm1(stacked_averaged_models.predict(test.values))&#xA;&#xA;&amp;lt;ipython-input-140-dfca4af6e9d1&amp;gt; in fit(self, X, y)&#xA;     18                 instance = clone(model)&#xA;     19                 self.base_models_[i].append(instance)&#xA;---&amp;gt; 20                 instance.fit(X[train_index], y[train_index])&#xA;     21                 y_pred = instance.predict(X[holdout_index])&#xA;     22                 out_of_fold_predictions[holdout_index, i] = y_pred&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params)&#xA;    248         Xt, fit_params = self._fit(X, y, **fit_params)&#xA;    249         if self._final_estimator is not None:&#xA;--&amp;gt; 250             self._final_estimator.fit(Xt, y, **fit_params)&#xA;    251         return self&#xA;    252 &#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\linear_model\coordinate_descent.py in fit(self, X, y, check_input)&#xA;    705                              order='F', dtype=[np.float64, np.float32],&#xA;    706                              copy=self.copy_X and self.fit_intercept,&#xA;--&amp;gt; 707                              multi_output=True, y_numeric=True)&#xA;    708             y = check_array(y, order='F', copy=False, dtype=X.dtype.type,&#xA;    709                             ensure_2d=False)&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)&#xA;    574     if multi_output:&#xA;    575         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,&#xA;--&amp;gt; 576                         dtype=None)&#xA;    577     else:&#xA;    578         y = column_or_1d(y, warn=True)&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)&#xA;    451                              % (array.ndim, estimator_name))&#xA;    452         if force_all_finite:&#xA;--&amp;gt; 453             _assert_all_finite(array)&#xA;    454 &#xA;    455     shape_repr = _shape_repr(array.shape)&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\sklearn\utils\validation.py in _assert_all_finite(X)&#xA;     42             and not np.isfinite(X).all()):&#xA;     43         raise ValueError(&quot;Input contains NaN, infinity&quot;&#xA;---&amp;gt; 44                          &quot; or a value too large for %r.&quot; % X.dtype)&#xA;     45 &#xA;     46 &#xA;&#xA;ValueError: Input contains NaN, infinity or a value too large for dtype('float64').&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I did a check on NaN and infinity, it did pass the test&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_traintrain = train.as_matrix().astype(np.float)&#xA;print(np.any(np.isnan(X_traintrain)))&#xA;print(np.all(np.isfinite(X_traintrain)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;False&#xA;True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How else can I solve, or at least, debug this?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X1      X2       X3     X4       X5    X6   X7     X8   Y1      Y2&#xA;0.64    784.00  343.00  220.50  3.50    5   0.00    0   10.56   16.67&#xA;0.62    808.50  367.50  220.50  3.50    2   0.00    0   8.60    12.07&#xA;0.62    808.50  367.50  220.50  3.50    5   0.00    0   8.50    12.04&#xA;0.98    514.50  294.00  110.25  7.00    2   0.10    1   24.58   26.47&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is few rows of my dataset&lt;/p&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="44680" LastEditDate="2018-01-23T00:56:54.307" LastActivityDate="2019-08-13T18:00:04.287" Title="ValueError: Input contains NaN, infinity or a value too large for dtype('float64')" Tags="|machine-learning|python|scikit-learn|pandas|numpy|" AnswerCount="2" CommentCount="6" ClosedDate="2019-12-13T15:40:03.890" ContentLicense="CC BY-SA 3.0" />
  <row Id="26887" PostTypeId="2" ParentId="26885" CreationDate="2018-01-21T14:21:29.227" Score="1" Body="&lt;p&gt;You need images from both labels, otherwise your CNN will predict any image as the label which you used for training. The model usually gets biased unless you use equal number of images for all labels. You can use data augmentation to generate more images so that all labels contain equal distribution.&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-01-21T14:21:29.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26888" PostTypeId="1" AcceptedAnswerId="26892" CreationDate="2018-01-21T14:38:00.590" Score="1" ViewCount="3243" Body="&lt;p&gt;Is there any detailed materials that can help explain how to set up HMM on R Studio?&lt;/p&gt;&#xA;" OwnerUserId="44949" LastEditorUserId="28175" LastEditDate="2018-01-21T15:28:33.423" LastActivityDate="2018-08-04T05:51:17.950" Title="Hidden Markov Model on R Studio" Tags="|machine-learning|r|markov-hidden-model|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26889" PostTypeId="2" ParentId="26885" CreationDate="2018-01-21T15:04:03.223" Score="2" Body="&lt;p&gt;Both cases will work. The point that you have to consider is that you shouldn't use &lt;code&gt;Softmax&lt;/code&gt; activation in the last layer because your classes have intersection and they are not mutually exclusive. You should employ &lt;code&gt;Sigmoid&lt;/code&gt; activation function and there should be two of them which each of them can be one or zero as the output vector. Consider that in this case you should not compare each activation to the other. Each one shows the probability of existence of the corresponding object, cat or dog.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your case I suggest you to provide images which don't have either and in some cases have both. The former is more important. Because if you don't provide negative labels, not existing, the net always try to make something out of nothing. Means the net always try to flag similar patters as cats or dogs. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-21T15:04:03.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26891" PostTypeId="1" AcceptedAnswerId="26893" CreationDate="2018-01-21T15:13:23.433" Score="1" ViewCount="1564" Body="&lt;p&gt;I'm new to &lt;em&gt;TensorFlow&lt;/em&gt; and currently I'm trying to implement an &lt;code&gt;LSTM&lt;/code&gt; using jupyter notebook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I run the following code segment, I got some errors and couldn't find any solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I work through this error?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lstmCell = tf.contrib.rnn.BasicLSTMCell(lstmUnits)&#xA;lstmCell = tf.contrib.rnn.DropoutWrapper(cell=lstmCell, output_keep_prob=0.75)&#xA;value, _ = tf.nn.dynamic_rnn(lstmCell, data, dtype=tf.float32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Error:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------------------------&#xA;AttributeError                            Traceback (most recent call last)&#xA;&amp;lt;ipython-input-29-db6a6fc2c55e&amp;gt; in &amp;lt;module&amp;gt;()&#xA;----&amp;gt; 1 lstmCell = tf.contrib.rnn.BasicLSTMCell(lstmUnits)&#xA;      2 lstmCell = tf.contrib.rnn.DropoutWrapper(cell=lstmCell, output_keep_prob=0.75)&#xA;      3 value, _ = tf.nn.dynamic_rnn(lstmCell, data, dtype=tf.float32)&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\python\util\lazy_loader.py in __getattr__(self, item)&#xA;     51 &#xA;     52   def __getattr__(self, item):&#xA;---&amp;gt; 53     module = self._load()&#xA;     54     return getattr(module, item)&#xA;     55 &#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\python\util\lazy_loader.py in _load(self)&#xA;     40   def _load(self):&#xA;     41     # Import the target module and insert it into the parent's namespace&#xA;---&amp;gt; 42     module = importlib.import_module(self.__name__)&#xA;     43     self._parent_module_globals[self._local_name] = module&#xA;     44 &#xA;&#xA;C:\Anaconda3\lib\importlib\__init__.py in import_module(name, package)&#xA;    124                 break&#xA;    125             level += 1&#xA;--&amp;gt; 126     return _bootstrap._gcd_import(name[level:], package, level)&#xA;    127 &#xA;    128 &#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap.py in _gcd_import(name, package, level)&#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap.py in _find_and_load(name, import_)&#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)&#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap.py in _load_unlocked(spec)&#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap_external.py in exec_module(self, module)&#xA;&#xA;C:\Anaconda3\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\__init__.py in &amp;lt;module&amp;gt;()&#xA;     29 from tensorflow.contrib import data&#xA;     30 from tensorflow.contrib import deprecated&#xA;---&amp;gt; 31 from tensorflow.contrib import distributions&#xA;     32 from tensorflow.contrib import estimator&#xA;     33 from tensorflow.contrib import factorization&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\distributions\__init__.py in &amp;lt;module&amp;gt;()&#xA;     31 from tensorflow.contrib.distributions.python.ops.distribution_util import matrix_diag_transform&#xA;     32 from tensorflow.contrib.distributions.python.ops.distribution_util import softplus_inverse&#xA;---&amp;gt; 33 from tensorflow.contrib.distributions.python.ops.estimator import *&#xA;     34 from tensorflow.contrib.distributions.python.ops.geometric import *&#xA;     35 from tensorflow.contrib.distributions.python.ops.independent import *&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\distributions\python\ops\estimator.py in &amp;lt;module&amp;gt;()&#xA;     19 from __future__ import print_function&#xA;     20 &#xA;---&amp;gt; 21 from tensorflow.contrib.learn.python.learn.estimators.head import _compute_weighted_loss&#xA;     22 from tensorflow.contrib.learn.python.learn.estimators.head import _RegressionHead&#xA;     23 from tensorflow.python.framework import ops&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\__init__.py in &amp;lt;module&amp;gt;()&#xA;     90 &#xA;     91 # pylint: disable=wildcard-import&#xA;---&amp;gt; 92 from tensorflow.contrib.learn.python.learn import *&#xA;     93 # pylint: enable=wildcard-import&#xA;     94 &#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\__init__.py in &amp;lt;module&amp;gt;()&#xA;     21 &#xA;     22 # pylint: disable=wildcard-import&#xA;---&amp;gt; 23 from tensorflow.contrib.learn.python.learn import *&#xA;     24 # pylint: enable=wildcard-import&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\__init__.py in &amp;lt;module&amp;gt;()&#xA;     23 from tensorflow.contrib.learn.python.learn import basic_session_run_hooks&#xA;     24 from tensorflow.contrib.learn.python.learn import datasets&#xA;---&amp;gt; 25 from tensorflow.contrib.learn.python.learn import estimators&#xA;     26 from tensorflow.contrib.learn.python.learn import graph_actions&#xA;     27 from tensorflow.contrib.learn.python.learn import learn_io as io&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\__init__.py in &amp;lt;module&amp;gt;()&#xA;    295 from tensorflow.contrib.learn.python.learn.estimators._sklearn import NotFittedError&#xA;    296 from tensorflow.contrib.learn.python.learn.estimators.constants import ProblemType&#xA;--&amp;gt; 297 from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNClassifier&#xA;    298 from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNEstimator&#xA;    299 from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNRegressor&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\dnn.py in &amp;lt;module&amp;gt;()&#xA;     28 from tensorflow.contrib.layers.python.layers import optimizers&#xA;     29 from tensorflow.contrib.learn.python.learn import metric_spec&#xA;---&amp;gt; 30 from tensorflow.contrib.learn.python.learn.estimators import dnn_linear_combined&#xA;     31 from tensorflow.contrib.learn.python.learn.estimators import estimator&#xA;     32 from tensorflow.contrib.learn.python.learn.estimators import head as head_lib&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\dnn_linear_combined.py in &amp;lt;module&amp;gt;()&#xA;     29 from tensorflow.contrib.layers.python.layers import optimizers&#xA;     30 from tensorflow.contrib.learn.python.learn import metric_spec&#xA;---&amp;gt; 31 from tensorflow.contrib.learn.python.learn.estimators import estimator&#xA;     32 from tensorflow.contrib.learn.python.learn.estimators import head as head_lib&#xA;     33 from tensorflow.contrib.learn.python.learn.estimators import model_fn&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py in &amp;lt;module&amp;gt;()&#xA;     47 from tensorflow.contrib.learn.python.learn.estimators import tensor_signature&#xA;     48 from tensorflow.contrib.learn.python.learn.estimators._sklearn import NotFittedError&#xA;---&amp;gt; 49 from tensorflow.contrib.learn.python.learn.learn_io import data_feeder&#xA;     50 from tensorflow.contrib.learn.python.learn.utils import export&#xA;     51 from tensorflow.contrib.learn.python.learn.utils import saved_model_export_utils&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\learn_io\__init__.py in &amp;lt;module&amp;gt;()&#xA;     19 from __future__ import print_function&#xA;     20 &#xA;---&amp;gt; 21 from tensorflow.contrib.learn.python.learn.learn_io.dask_io import extract_dask_data&#xA;     22 from tensorflow.contrib.learn.python.learn.learn_io.dask_io import extract_dask_labels&#xA;     23 from tensorflow.contrib.learn.python.learn.learn_io.dask_io import HAS_DASK&#xA;&#xA;C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\learn_io\dask_io.py in &amp;lt;module&amp;gt;()&#xA;     24 try:&#xA;     25   # pylint: disable=g-import-not-at-top&#xA;---&amp;gt; 26   import dask.dataframe as dd&#xA;     27   allowed_classes = (dd.Series, dd.DataFrame)&#xA;     28   HAS_DASK = True&#xA;&#xA;C:\Anaconda3\lib\site-packages\dask\dataframe\__init__.py in &amp;lt;module&amp;gt;()&#xA;      1 from __future__ import print_function, division, absolute_import&#xA;      2 &#xA;----&amp;gt; 3 from .core import (DataFrame, Series, Index, _Frame, map_partitions,&#xA;      4                    repartition, to_delayed, to_datetime, to_timedelta)&#xA;      5 from .groupby import Aggregation&#xA;&#xA;C:\Anaconda3\lib\site-packages\dask\dataframe\core.py in &amp;lt;module&amp;gt;()&#xA;     38 if PANDAS_VERSION &amp;gt;= '0.20.0':&#xA;     39     from pandas.util import cache_readonly&#xA;---&amp;gt; 40     pd.core.computation.expressions.set_use_numexpr(False)&#xA;     41 else:&#xA;     42     from pandas.util.decorators import cache_readonly&#xA;&#xA;AttributeError: module 'pandas.core.computation' has no attribute 'expressions'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tensorflow version - '1.4.0'&lt;/li&gt;&#xA;&lt;li&gt;Python version - 3.6.3:: Anaconda, Inc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="44950" LastEditorUserId="29575" LastEditDate="2018-01-21T16:05:39.163" LastActivityDate="2018-01-21T16:05:39.163" Title="LSTM Implementation using tensorflow (anaconda)" Tags="|deep-learning|tensorflow|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26892" PostTypeId="2" ParentId="26888" CreationDate="2018-01-21T15:20:18.763" Score="0" Body="&lt;p&gt;You can take a look at &lt;a href=&quot;http://hiddendomains.sourceforge.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. As you can read from there:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Download and unpack hiddenDomains&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;  &#xA;  &lt;p&gt;If you haven't already done this, download the latest version of hiddenDomains from the Sourceforge &lt;a href=&quot;https://sourceforge.net/projects/hiddendomains/&quot; rel=&quot;nofollow noreferrer&quot;&gt;website&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Now unpack it&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shell$ tar -xzvf hiddenDomains.VERSION.NUMBER.tar.gz&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Where VERSION and NUMBER represent the release that you downloaded.&lt;/p&gt;&#xA;  &#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;Install R and required R packages (if necessary)&#xA;  Download and install R if you don't already have it.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;  &#xA;  &lt;p&gt;If you don't already have them, you will need to install two hidden Markov model libraries:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;a href=&quot;http://cran.r-project.org/web/packages/depmixS4/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;depmixS4&lt;/a&gt;&lt;br /&gt;&#xA;  &lt;a href=&quot;http://cran.r-project.org/web/packages/HiddenMarkov/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;HiddenMarkov&lt;/a&gt;&lt;br /&gt;&#xA;  You can do this by starting R on the command line:&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shell$ R&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or, if you prefer RStudio, you can use that.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Regardless of how you started R, type the following commands to install the packages.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; install.packages(&quot;depmixS4&quot;)&#xA;&amp;gt; install.packages(&quot;HiddenMarkov&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/23678786/rstudio-hiddenmarkov-package-and-dthmm-function&quot;&gt;Here&lt;/a&gt; and &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=1&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjPv8q-q-nYAhXHLFAKHXU1BsQQFggnMAA&amp;amp;url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2FHMM%2FHMM.pdf&amp;amp;usg=AOvVaw1ywdsUCA92akpljCkPV2N0&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; may also be helpful for knowing how to use it.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-21T15:20:18.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26893" PostTypeId="2" ParentId="26891" CreationDate="2018-01-21T15:26:01.477" Score="1" Body="&lt;p&gt;Based on the solution &lt;a href=&quot;https://stackoverflow.com/questions/43833081/attributeerror-module-object-has-no-attribute-computation&quot;&gt;here&lt;/a&gt; you have different choices:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest solution is to revert back to &lt;code&gt;Pandas 0.19.2&lt;/code&gt;. For this purpose use the following command in your command line or terminal.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install pandas=0.19.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-21T15:26:01.477" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26894" PostTypeId="1" AcceptedAnswerId="26897" CreationDate="2018-01-21T17:32:55.203" Score="4" ViewCount="84" Body="&lt;p&gt;I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally &quot;done&quot; you evaluate it on your test set. Your test performance comes back much worse than your dev performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now you conclude, &quot;My dev set must be too small, causing me to overfit my hyperparameters.&quot; So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set &quot;too much&quot; in the above way (increasing the dev set size in a cycle again and again), it is still &quot;mostly&quot; unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think this is an accurate take?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Incidentally, I'm unsure if there's anything &lt;em&gt;else&lt;/em&gt; you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options? &lt;/p&gt;&#xA;" OwnerUserId="44394" LastActivityDate="2018-01-21T23:53:49.667" Title="To be useful, doesn't a test set often become a second dev set?" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26896" PostTypeId="2" ParentId="26894" CreationDate="2018-01-21T19:43:43.953" Score="1" Body="&lt;p&gt;actually you are right, you have to make use of the test set the less times you can if you want to be quite sure it will stay a test set, and not another dev set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Said that, when you see that the performances on the test set are much worse than the dev set, you have other options than touch the size of the dev set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) add more penalties in the model, like a form of regularization (L1, L2, etc.);&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) decrease the degrees of freedom of your model, like the number of layer/nodes in case of NN, or the number of trees in case of decision trees et similia;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) decrease the learning epochs;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) try some (more aggressive) data augmentation technique;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use data augmentation in the test set: for each sample of the test set you can use the same form of data augmentation you used on the training set, and finally taking as result the average of the various single results. Even if this technique do not resolve a very big performance difference from the dev set and the test set, it can make your model acceptable. An in this last case, you have of course to use it also on the real data when the model is deployed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in general if you are in need of accessing the test set several times, in order to have a less biased model you can reshuffle all the data, and then divide it again in train/dev/test. Of course, the rule of accessing the test set the less times possible stays true.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-21T19:43:43.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26897" PostTypeId="2" ParentId="26894" CreationDate="2018-01-21T23:53:49.667" Score="3" Body="&lt;p&gt;You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the &lt;a href=&quot;https://research.googleblog.com/2015/08/the-reusable-holdout-preserving.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reusable Holdout&lt;/a&gt;. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-21T23:53:49.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26898" PostTypeId="1" AcceptedAnswerId="26904" CreationDate="2018-01-22T00:49:55.897" Score="4" ViewCount="5539" Body="&lt;p&gt;Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas &lt;code&gt;dataframes&lt;/code&gt; in Python. Does anyone know of a way to do this, or perhaps a package that will allow me to do this? &lt;/p&gt;&#xA;" OwnerUserId="43801" LastEditorUserId="28175" LastEditDate="2018-01-22T01:08:51.683" LastActivityDate="2020-08-22T16:17:32.880" Title="Is there a package for using SQL to manipulate Pandas dataframes in Python?" Tags="|python|pandas|sql|dataframe|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26899" PostTypeId="1" AcceptedAnswerId="26901" CreationDate="2018-01-22T00:58:24.920" Score="0" ViewCount="1074" Body="&lt;p&gt;I know that popular linear regression models such as Lasso or Logistic Regression are widely used in practice because they perform reasonably good, are efficient and &lt;strong&gt;interpretable&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know, the only way for these models to learn non-linear relationships between $X$ and $y$ is by applying non-linear transformations to $X$ through basis functions $\phi(X)$ (as it is summarized in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/linear_model.html#polynomial-regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn documentation&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if this approach is actually used in real world problems, where usually we don't have any prior information about how to design $\phi$.&lt;/p&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="17484" LastEditDate="2018-01-23T11:27:19.313" LastActivityDate="2018-01-24T05:23:37.577" Title="Are linear regression models with non linear basis functions used in practice?" Tags="|linear-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26900" PostTypeId="2" ParentId="26898" CreationDate="2018-01-22T01:04:51.843" Score="1" Body="&lt;p&gt;Based on my experience you can almost do everything that can be done using pandas in your sql. I've not seen recent versions of pandas but I remember that sql is even better because using pandas you are restricted to the size of memory. If memory fills out you may crash, something that does not happen using sql commands. You can save your pandas data frame in a &lt;code&gt;csv&lt;/code&gt; file and manipulate that &lt;code&gt;csv&lt;/code&gt; file using your sql. &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;This link&lt;/a&gt; and also &lt;a href=&quot;https://stackoverflow.com/questions/16923281/pandas-writing-dataframe-to-csv-file&quot;&gt;here&lt;/a&gt; may help you. Also for importing your &lt;code&gt;csv&lt;/code&gt; file to your sql, you have not specified what sql you have but &lt;a href=&quot;https://stackoverflow.com/questions/15242757/import-csv-file-into-sql-server&quot;&gt;this link&lt;/a&gt; may help you. Other sqls also provide this behavior. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-22T01:04:51.843" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26901" PostTypeId="2" ParentId="26899" CreationDate="2018-01-22T01:06:24.153" Score="3" Body="&lt;p&gt;Yep, that's a thing. It's called a &quot;Generalized additive model (GAM)&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Generalized_additive_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Generalized_additive_model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also be interested in &quot;multivariate adaptive regression splines (MARS)&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Regarding demonstrating that these are actually used, I'm not sure what you're looking for. I've never seen a survey to try and gauge the popularity of specific models, and I'm not sure how useful such a thing would be. I could speak from my subjective experience, but I don't consider that particularly meaningful either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just want examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/?term=generalized%20additive%20model&amp;amp;sortorder=best&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hits for GAM models on PubMed&lt;/a&gt; - 2114 hits&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/?term=adaptive%20regression%20splines&amp;amp;sortorder=best&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hits for MARS models on Pubmed&lt;/a&gt; - 190 hits&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For context:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/?term=random%20forest&amp;amp;sortorder=best&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hits for Random Forest models on PubMed&lt;/a&gt; - 5604&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/?term=deep%20learning&amp;amp;sortorder=best&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hits for Deep Learning on PubMed&lt;/a&gt; - 4435 hits&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/?term=neural%20network&amp;amp;sortorder=best&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hits for Neural Network on PubMed&lt;/a&gt; - 57170 hits&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Take these counts with a solid dose of salt: in my experience random forests are in much wider use in practice than neural networks. But like I said earlier, you shouldn't really trust my subjective experience either since that's just anecdotal.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-24T05:23:37.577" LastActivityDate="2018-01-24T05:23:37.577" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26902" PostTypeId="1" AcceptedAnswerId="26903" CreationDate="2018-01-22T02:22:25.620" Score="2" ViewCount="939" Body="&lt;p&gt;We start with a matrix of user ratings for different movies with some elements unknow i.e the rating of a yet to be seen movie by an user. We need to fill in this gap. So &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How can you decompose or factorize a matrix where all elements are not known in advance.&lt;/li&gt;&#xA;&lt;li&gt;after you somehow decompose it, do you just multiply the matrices back to recreate the original one, but now with the missing item populated?&lt;/li&gt;&#xA;&lt;li&gt;How do you know which factorization method like non negative,singular value,eigen to choose,without going into too much math?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="10522" LastActivityDate="2018-01-22T05:29:07.477" Title="How exactly does matrix factorization help with collaborative filtering" Tags="|recommender-system|matrix-factorisation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26903" PostTypeId="2" ParentId="26902" CreationDate="2018-01-22T05:23:02.057" Score="2" Body="&lt;p&gt;&lt;strong&gt;Answers to your question&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You factorize the matrix in order to approximate original one as closely as possible. This is generally done by starting with randomized values, and updating based on error (between product of factors and original matrix). In other words, for a given matrix A, you are trying to find matrices C &amp;amp; D such that Error(A - (C x D)) is lowest. The algorithm is designed to find an approximation , which might result in original missing entries being replaced by new values (recommendations or ratings).&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you just multiply? Yes. That it is the essence of calculation. For every user and product, multiplication gives you ratings or score. Sorting by score, and picking the index you get recommendation for each user. It also allows you to now store much smaller matrices than the original one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Choice of factorization will be also dictated by application. If your application has only positive ratings, then it is better to use non-negative matrix factorization. You may start using matrix factorization methods without knowing the implementation to start with, as long as you are aware of the overall idea (and the pitfalls in using it).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Further Comments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a little perplexing that you start with a matrix with many missing entries (unseen items), approximate the matrix via factorization and expect to get non-missing entries (which help in doing predictions). If the task is to approximate original entries, then recommendations won't be good since you can have missing/zero entries in the approximation and still get lowest error on approximation task. The idea: regularization imposed in the algorithm (point 1 above) ensures that noise in the original data is filtered and only patterns are detected. But this idea has its fair share of critics. The introduction section of &lt;a href=&quot;https://arxiv.org/abs/1205.2618&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper by Stefen Rendle&lt;/a&gt; gives a readable introduction to what is happening in simple matrix factorization methods and what can go wrong. The paper also re-formulates the task with better optimization criteria.&#xA;You can also read &lt;a href=&quot;http://sifter.org/~simon/journal/20061211.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post by Simon Funk&lt;/a&gt; which explains the mechanics in a readable language (and code), for matrix factorization applied to recommendations.&lt;/p&gt;&#xA;" OwnerUserId="23204" LastEditorUserId="23204" LastEditDate="2018-01-22T05:29:07.477" LastActivityDate="2018-01-22T05:29:07.477" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26904" PostTypeId="2" ParentId="26898" CreationDate="2018-01-22T05:27:15.580" Score="5" Body="&lt;p&gt;I found a package called &lt;a href=&quot;https://pypi.python.org/pypi/pandasql&quot; rel=&quot;noreferrer&quot;&gt;pandasql&lt;/a&gt;, which is based on &lt;a href=&quot;https://www.rdocumentation.org/packages/sqldf/versions/0.3-5&quot; rel=&quot;noreferrer&quot;&gt;sqldf for R&lt;/a&gt;. It seems quite a bit slower than doing the transformations with the pandas package, but it gets the job done. Just put the SQL query into a string like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;query_string = &quot;&quot;&quot;&#xA;    select * from df&#xA;&quot;&quot;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then use the string in the pandasql.sqldf package, as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_dataframe = pandasql.sqldf(query_string, globals())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Choose globals() or locals(), depending on the scope you want for your variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I mentioned, it seems a bit slow, but I couldn't find anything else. I may use this from time to time until I become better at Pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sean&lt;/p&gt;&#xA;" OwnerUserId="43801" LastActivityDate="2018-01-22T05:27:15.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26905" PostTypeId="2" ParentId="26874" CreationDate="2018-01-22T05:41:03.350" Score="3" Body="&lt;p&gt;You are missing one little step here. At the output node the backpropagation algorithm starts with the loss function &lt;span class=&quot;math-container&quot;&gt;$L$&lt;/span&gt;. You want to find out how much the weight &lt;span class=&quot;math-container&quot;&gt;$w_1$&lt;/span&gt; impacts our loss function thus&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial L}{\partial w_1}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;We can break this down with the chain rule as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial \sigma} \frac{\partial \sigma}{\partial w_1}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h1&gt;Why do we do this?&lt;/h1&gt;&#xA;&lt;p&gt;We do this to simplify the math. Because the &lt;span class=&quot;math-container&quot;&gt;$L$&lt;/span&gt; is a function of &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; is a function of &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; is a function of the weights. But if we ignore the loss function for now.&lt;/p&gt;&#xA;&lt;h1&gt;How do we calculate &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial f}{\partial w^2_{i, j}}$&lt;/span&gt;?&lt;/h1&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial f}{\partial w^2_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial w^2_1}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$f = h_1^2 w_{1,1}^3 + h_2^2 w_{2,1}^3$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial f}{\partial h^2_i} = w_{i, 1}^3$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial h^2_i}{\partial w^2_1} = \sigma(W^2h^1) (1 - \sigma(W^2h^1))$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;h1&gt;Backpropagation&lt;/h1&gt;&#xA;&lt;p&gt;Based on this answer we can see that &lt;span class=&quot;math-container&quot;&gt;$h_i^2$&lt;/span&gt; is a function of the inputs from the previous layer and the activation at the neuron. Thus, we can substitute this function into our equation and continue taking the derivatives and we can see how the error propagates backwards through the network.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial f}{\partial w^1_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial h^1_i} \frac{\partial h^1_i}{\partial w^1_{i,j}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;(Update)&lt;/p&gt;&#xA;&lt;h1&gt;The answer&lt;/h1&gt;&#xA;&lt;p&gt;This may not be correct, the work needs to be checked.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$f = \begin{bmatrix} w_{11}^3 &amp;amp; w_{21}^3 \end{bmatrix} \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \textbf{W}^3\textbf{h}^2$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\textbf{h}^2 = \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^2 &amp;amp; w_{21}^2 \\ w_{21}^2 &amp;amp; w_{22}^2\end{bmatrix} \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix}) = \sigma(\textbf{W}^2\textbf{h}^1)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\textbf{h}^1 = \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^1 &amp;amp; w_{21}^1 \\ w_{21}^1 &amp;amp; w_{22}^1\end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}) = \sigma(\textbf{W}^1\textbf{X})$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Then using the chain rule&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial f}{\partial \textbf{W}^1} = \frac{\partial f}{\partial \textbf{h}^2} \frac{\partial \textbf{h}^2}{\partial \textbf{h}^1} \frac{\partial \textbf{h}^1}{\partial \textbf{W}^1} = \textbf{W}^3 * \sigma(\textbf{W}^2\textbf{h}^1)(1-\sigma(\textbf{W}^2\textbf{h}^1)) * \sigma(\textbf{W}^1\textbf{X})(1-\sigma(\textbf{W}^1\textbf{X}))$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-23T06:37:21.387" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26906" PostTypeId="1" CreationDate="2018-01-22T06:09:57.890" Score="2" ViewCount="791" Body="&lt;p&gt;If deep-learning learns features, aren't we saying it can learn association rules ?&lt;/p&gt;&#xA;" OwnerUserId="44978" LastEditorUserId="27665" LastEditDate="2018-01-22T09:46:56.600" LastActivityDate="2018-01-22T09:46:56.600" Title="If deep-learning learns features, aren't we saying it can learn association rules?" Tags="|machine-learning|deep-learning|association-rules|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26907" PostTypeId="1" CreationDate="2018-01-22T06:34:21.980" Score="3" ViewCount="70" Body="&lt;p&gt;I'm recently reading a paper about &lt;a href=&quot;http://www.lx.it.pt/~asmc/pub/talks/09-TA/ta_pres.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scoring Mechanisms for Bayesian Networks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it mean that the best network is always the empty network?&lt;/p&gt;&#xA;" OwnerUserId="32204" LastEditorUserId="29575" LastEditDate="2018-01-22T06:37:01.020" LastActivityDate="2024-01-06T00:05:15.660" Title="Is the maximum BDeu Bayesian Network always the empty network?" Tags="|machine-learning|bayesian-networks|scoring|pgm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26908" PostTypeId="2" ParentId="26906" CreationDate="2018-01-22T07:45:16.590" Score="2" Body="&lt;p&gt;Finding valuable association rules is defined base on their support and confidence. ANN is not proper for these kinds of specifications. &#xA;Hence, if we use ANN for learning association rules, we can't guarantee the specified support and confidence are satisfied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyhow, there are other machine learning methods to learn association rules such as decision tree. Although, that methods not such that proper to find rules with specified support and confidence. You can know more about this in &lt;a href=&quot;https://www.rocq.inria.fr/axis/COMPSTAT2010/TU_Berka-Rauch_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastEditorUserId="27665" LastEditDate="2018-01-22T07:57:30.350" LastActivityDate="2018-01-22T07:57:30.350" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26909" PostTypeId="1" AcceptedAnswerId="26912" CreationDate="2018-01-22T09:57:47.533" Score="1" ViewCount="1743" Body="&lt;p&gt;Logistic and Linear Regression have different cost functions. But I don't get how the gradient descent in logistic regression is the same as Linear Regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We get the Gradient Descent formula by deriving the Squared Error cost function. However in Logistic Regression we use a Logarithmic Cost function instead. I think I am lost here. &lt;/p&gt;&#xA;" OwnerUserId="44986" LastEditorUserId="40853" LastEditDate="2018-01-22T10:13:21.460" LastActivityDate="2018-01-22T10:50:43.740" Title="Gradient Descent in logistic regression" Tags="|logistic-regression|linear-regression|gradient-descent|cost-function|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26910" PostTypeId="1" CreationDate="2018-01-22T09:58:22.547" Score="11" ViewCount="5440" Body="&lt;p&gt;I'm new to ML and TensorFlow (I started about a few hours ago), and I'm trying to use it to predict the next few data points in a time series. I'm taking my input and doing this with it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/----------- x ------------\&#xA;.-------------------------------.&#xA;| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |&#xA;'-------------------------------'&#xA;     \----------- y ------------/&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I thought I was doing is using &lt;em&gt;x&lt;/em&gt; as the input data and &lt;em&gt;y&lt;/em&gt; as the desired output for that input, so that given 0-6 I could get 1-7 (the 7 in particular). However, when I run my graph with &lt;em&gt;x&lt;/em&gt; as the input, what I get is a prediction that looks more like &lt;em&gt;x&lt;/em&gt; than &lt;em&gt;y&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the code (based on &lt;a href=&quot;https://mapr.com/blog/deep-learning-tensorflow/&quot; rel=&quot;noreferrer&quot;&gt;this post&lt;/a&gt; and &lt;a href=&quot;http://technopreneurlife.com/deep-temporal-forecasting-with-recurrent-neural-networks-rnns/&quot; rel=&quot;noreferrer&quot;&gt;this post&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;import matplotlib.pyplot as plot&#xA;import pandas as pd&#xA;import csv&#xA;&#xA;def load_data_points(filename):&#xA;    print(&quot;Opening CSV file&quot;)&#xA;    with open(filename) as csvfile:&#xA;        print(&quot;Creating CSV reader&quot;)&#xA;        reader = csv.reader(csvfile)&#xA;        print(&quot;Reading CSV&quot;)&#xA;        return [[[float(p)] for p in row] for row in reader]&#xA;&#xA;flatten = lambda l: [item for sublist in l for item in sublist]&#xA;&#xA;data_points = load_data_points('dataset.csv')&#xA;&#xA;print(&quot;Loaded&quot;)&#xA;&#xA;prediction_size = 10&#xA;num_test_rows = 1&#xA;num_data_rows = len(data_points) - num_test_rows&#xA;row_size = len(data_points[0]) - prediction_size&#xA;&#xA;# Training data&#xA;data_rows = data_points[:-num_test_rows]&#xA;x_data_points = np.array([row[:-prediction_size] for row in data_rows]).reshape([-1, row_size, 1])&#xA;y_data_points = np.array([row[prediction_size:] for row in data_rows]).reshape([-1, row_size, 1])&#xA;&#xA;# Test data&#xA;test_rows = data_points[-num_test_rows:]&#xA;x_test_points = np.array([[data_points[0][:-prediction_size]]]).reshape([-1, row_size, 1])&#xA;y_test_points = np.array([[data_points[0][prediction_size:]]]).reshape([-1, row_size, 1])&#xA;&#xA;tf.reset_default_graph()&#xA;&#xA;num_hidden = 100&#xA;&#xA;x = tf.placeholder(tf.float32, [None, row_size, 1])&#xA;y = tf.placeholder(tf.float32, [None, row_size, 1])&#xA;&#xA;basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=num_hidden, activation=tf.nn.relu)&#xA;rnn_outputs, _ = tf.nn.dynamic_rnn(basic_cell, x, dtype=tf.float32)&#xA;&#xA;learning_rate = 0.001&#xA;&#xA;stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, num_hidden])&#xA;stacked_outputs = tf.layers.dense(stacked_rnn_outputs, 1)&#xA;outputs = tf.reshape(stacked_outputs, [-1, row_size, 1])&#xA;&#xA;loss = tf.reduce_sum(tf.square(outputs - y))&#xA;optimizer = tf.train.AdamOptimizer(learning_rate)&#xA;training_op = optimizer.minimize(loss)&#xA;&#xA;init = tf.global_variables_initializer()&#xA;&#xA;iterations = 1000&#xA;&#xA;with tf.Session() as sess:&#xA;    init.run()&#xA;    for ep in range(iterations):&#xA;        sess.run(training_op, feed_dict={x: x_data_points, y: y_data_points})&#xA;        if ep % 100 == 0:&#xA;            mse = loss.eval(feed_dict={x: x_data_points, y: y_data_points})&#xA;            print(ep, &quot;\tMSE:&quot;, mse)&#xA;&#xA;    y_pred = sess.run(stacked_outputs, feed_dict={x: x_test_points})&#xA;&#xA;    plot.rcParams[&quot;figure.figsize&quot;] = (20, 10)&#xA;&#xA;    plot.title(&quot;Actual vs Predicted&quot;)&#xA;    plot.plot(pd.Series(np.ravel(x_test_points)), 'g:', markersize=2, label=&quot;X&quot;)&#xA;    plot.plot(pd.Series(np.ravel(y_test_points)), 'b--', markersize=2, label=&quot;Y&quot;)&#xA;    plot.plot(pd.Series(np.ravel(y_pred)), 'r-', markersize=2, label=&quot;Predicted&quot;)&#xA;    plot.legend(loc='upper left')&#xA;    plot.xlabel(&quot;Time periods&quot;)&#xA;    plot.tick_params(&#xA;        axis='y',&#xA;        which='both',&#xA;        left='off',&#xA;        right='off',&#xA;        labelleft='off')&#xA;    plot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result shown in the graph below is a prediction that follows &lt;em&gt;x&lt;/em&gt;, rather than being shifted to the left (and including the predicted points on the right) as it should be to resemble &lt;em&gt;y&lt;/em&gt;. Obviously the desire is for the red line to be as close to the blue one as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/sJicm.png&quot; alt=&quot;graph&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea what I'm doing with all this, so please ELI5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, also, my data points are fairly small numbers (order of 0.0001). If I don't multiply them by, say, 1000000, the results are so small that the red line is almost flat at the bottom of the chart. Why? I'm guessing it's because of the squaring in the fitness function. Should data be normalized before use, and if so, to what? 0-1? If I use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;normalized_points = [(p - min_point) / (max_point - min_point) for p in data_points]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my prediction fluctuates more wildly as it progresses: &lt;img src=&quot;https://i.stack.imgur.com/pjyeu.png&quot; alt=&quot;fluctuating&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; I'm being dumb and only giving it one example to learn from, not 500, aren't I? So I should be giving it multiple 500-point samples, right?&lt;/p&gt;&#xA;" OwnerUserId="44985" LastEditorUserId="44985" LastEditDate="2018-01-22T23:22:43.383" LastActivityDate="2018-07-18T10:35:57.983" Title="Trying to use TensorFlow to predict financial time series data" Tags="|machine-learning|python|time-series|tensorflow|rnn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26911" PostTypeId="2" ParentId="12851" CreationDate="2018-01-22T10:48:27.040" Score="18" Body="&lt;p&gt;&lt;strong&gt;Keras&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://keras.io/visualization/&quot; rel=&quot;noreferrer&quot;&gt;keras.utils.vis_utils module&lt;/a&gt; provides utility functions to plot a Keras model (using graphviz)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following shows a network model that the first hidden layer has 50 neurons and expects 104 input variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot_model(model, to_file='model.png', show_shapes=True, show_layer_names=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o17GY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o17GY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44989" LastActivityDate="2018-01-22T10:48:27.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26912" PostTypeId="2" ParentId="26909" CreationDate="2018-01-22T10:50:43.740" Score="3" Body="&lt;p&gt;Gradient Descent is an universal method, you can us it with basically every loss function you can find in known ML algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you have only to derive the logarithmic cost function. You can find a detailed calculation at&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression&quot;&gt;https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-01-22T10:50:43.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26913" PostTypeId="1" CreationDate="2018-01-22T10:52:32.617" Score="1" ViewCount="347" Body="&lt;p&gt;I am clustering text based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can spread very randomly, I found &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html#sklearn.neighbors.KernelDensity&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kernel Density Estimators&lt;/a&gt; relevant.&lt;/p&gt;&#xA;&lt;p&gt;However, the scores of KDE are very sensitive to the choice of bandwidth hyper-parameter, which I could not pre-estimate. Most bandwidth values end up with either infinite score for points outside the cluster, of zero for the points in the cluster. I need a way to 'automatically' choose the bandwidth, so that my results will yield scores that make sense for the points in the cluster (larger values) and the points outside of it (smaller values).&lt;/p&gt;&#xA;&lt;p&gt;I tried:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Both Silverman and Scott factors methods to evaluate the bandwidth depending on #points and #features, both were far from relevant in my case&lt;/li&gt;&#xA;&lt;li&gt;GridSearchCV returns the minimal bandwidth in the grid&lt;/li&gt;&#xA;&lt;li&gt;Different kernel types (all the relevant ones are similarly sensitive)&lt;/li&gt;&#xA;&lt;li&gt;Reduce dimension, but as expected this severely hurt the KDE results without making the bandwidth that less sensitive&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.neighbors import KernelDensity&#xA;&#xA;# indexes of points in cluster3&#xA;docs = np.where(y_pred == 3)[0] &#xA;&#xA;kde = KernelDensity(kernel='gaussian',bandwidth=0.399).fit(X_tfidf[docs].todense())&#xA;&#xA;# evaluate scores on all points&#xA;scores = np.exp(kde.score_samples(X_tfidf.todense())) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Note that there are ~2200 features in the TF-IDF, a few dozen points (40-120) in each cluster the KDE was fitted to, and about 4000 points in total.&lt;/p&gt;&#xA;&lt;p&gt;Any ideas on anything (even beyond KDE) are welcome, thank you.&lt;/p&gt;&#xA;" OwnerUserId="38377" LastEditorUserId="29169" LastEditDate="2021-08-09T18:33:51.423" LastActivityDate="2024-01-03T11:09:26.080" Title="KDE on TF-IDF - sensitive bandwidth" Tags="|scikit-learn|clustering|tfidf|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="26914" PostTypeId="2" ParentId="26913" CreationDate="2018-01-22T11:07:18.517" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;GridSearchCV returns the minimal bandwidth in the grid&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then alter your grid to have a lower lower bound. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-22T11:07:18.517" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26915" PostTypeId="1" CreationDate="2018-01-22T11:49:57.430" Score="1" ViewCount="434" Body="&lt;p&gt;Apologies in advance, I am completely new to the world of machine learning. But I just wanted a bit of help in areas to explore or even just a blunt answer if the following was possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a chrome extension that gets the content of a webpage and am wanting to classify this text into what topic it is - ie. math, science, geography, etc. I know there are several APIs that I could send the data of too initially but was wanting to know if this thing would be possible to do within the extension using javascript.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I made a quick and dirty demo in which I trained &lt;a href=&quot;https://github.com/nemo/natural-synaptic&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/nemo/natural-synaptic&lt;/a&gt; with a small amount of data and exported it to JSON. I then imported this into the extension and used it to classify the text, it works okay. However, I would need more training data to classify it correctly and it also seems to have a small performance hit when loading an already small amount of trained data (4mb) so increasing this may affect performance even more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My overall questions are - How feasible is a text classifier using javascript all client side without affecting performance too much? And is there any frameworks, resource you could suggest I look into? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="44991" LastEditorUserId="1330" LastEditDate="2021-07-17T23:26:51.660" LastActivityDate="2021-07-17T23:26:51.660" Title="Using machine learning to classify text client-side in JavaScript?" Tags="|machine-learning|javascript|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="26916" PostTypeId="2" ParentId="12851" CreationDate="2018-01-22T12:00:34.043" Score="5" Body="&lt;p&gt;You can read the popular paper &lt;a href=&quot;http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf&quot; rel=&quot;noreferrer&quot;&gt;Understanding Neural Networks Through Deep Visualization&lt;/a&gt; which discusses visualization of convolutional nets. &lt;a href=&quot;https://github.com/yosinski/deep-visualization-toolbox&quot; rel=&quot;noreferrer&quot;&gt;Its implementation&lt;/a&gt; not only  displays each layer but also depicts the activations, weights, deconvolutions and many other things that are deeply discussed in the paper. It's code is in &lt;code&gt;caffe'&lt;/code&gt;. The interesting part is that you can replace the pre-trained model with your own.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-22T12:00:34.043" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26917" PostTypeId="1" CreationDate="2018-01-22T12:54:56.470" Score="1" ViewCount="335" Body="&lt;p&gt;My dataset has the following class distribution&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;CLASS      FREQUENCY&#xA;2          22696&#xA;4          2541&#xA;1          2093&#xA;5          1298&#xA;3          1116&#xA;0          960&#xA;6          14&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;definitely I would want to generate a new sample, for that I will use &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/introduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;python imblearn&lt;/a&gt; , I have three options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;oversample the minority classes&lt;/li&gt;&#xA;&lt;li&gt;undersample the majority classes&lt;/li&gt;&#xA;&lt;li&gt;choose a median class and apply both undersample the majority classes and oversamples the minority classes to equal the median class.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Later I will use the generated dataset to train three estimators RandomForest classifier, SVC and ensemble of both. I will choose the one with the best f-1 score. what would be the best option and which oversampling/undersampling algo should I use? &lt;/p&gt;&#xA;" OwnerUserId="44673" LastActivityDate="2021-07-18T13:23:57.873" Title="suggestion to implement undersample and oversample" Tags="|machine-learning|python|class-imbalance|smote|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26918" PostTypeId="1" CreationDate="2018-01-22T12:58:28.003" Score="2" ViewCount="2447" Body="&lt;p&gt;I'm trying to implement the validation curve based on &lt;a href=&quot;http://scikit-learn.org/0.15/auto_examples/plot_validation_curve.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; are shaped (266531, 23) and (266531,). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The curve looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WMNCA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WMNCA.png&quot; alt=&quot;The training and validation scores completely overlap&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn.datasets import load_digits&#xA;from sklearn.svm import SVC&#xA;from sklearn.learning_curve import validation_curve&#xA;import psutil&#xA;&#xA;np.random.seed(0)&#xA;&#xA;X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()&#xA;indices = np.arange(y.shape[0])&#xA;np.random.shuffle(indices)&#xA;X, y = X[indices], y[indices]&#xA;&#xA;param_range = np.arange(1, 41, 2)&#xA;train_scores, test_scores = validation_curve(&#xA;    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name=&quot;max_depth&quot;, cv=10, &#xA;    param_range=param_range,n_jobs=psutil.cpu_count(),&#xA;    scoring=&quot;accuracy&quot;)&#xA;&#xA;train_scores_mean = np.mean(train_scores, axis=1)&#xA;train_scores_std = np.std(train_scores, axis=1)&#xA;test_scores_mean = np.mean(test_scores, axis=1)&#xA;test_scores_std = np.std(test_scores, axis=1)&#xA;plt.title(&quot;Validation Curve with DecisionTree&quot;)&#xA;plt.xlabel(&quot;max_depth&quot;)&#xA;plt.ylabel(&quot;Score&quot;)&#xA;plt.ylim(0.0, 1.1)&#xA;plt.plot(param_range, train_scores_mean, label=&quot;Training score&quot;,&#xA;             color=&quot;r&quot;)&#xA;plt.plot(param_range, test_scores_mean, label=&quot;Cross-validation score&quot;,&#xA;             color=&quot;g&quot;)&#xA;&#xA;plt.legend(loc=&quot;best&quot;)&#xA;plt.xticks(param_range)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A comment suggested I made &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/8Q6La.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8Q6La.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="21254" LastEditDate="2018-02-07T09:43:47.703" LastActivityDate="2018-02-07T09:43:47.703" Title="Validation curve unlike SKLearn sample" Tags="|python|scikit-learn|cross-validation|plotting|matplotlib|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26919" PostTypeId="2" ParentId="12721" CreationDate="2018-01-22T13:28:49.300" Score="4" Body="&lt;p&gt;ARIMA models are linear and LSTM models are nonlinear. Some other parametric nonlinear time series models that statisticians have studied are Threshold Autoregressive Models (TAR) and Smooth Transition Autoregressive Models (STAR). The R package tsDyn implements these models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder how STAR models do vs. LSTM.&lt;/p&gt;&#xA;" OwnerUserId="44994" LastActivityDate="2018-01-22T13:28:49.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26920" PostTypeId="1" CreationDate="2018-01-22T13:32:32.010" Score="2" ViewCount="6959" Body="&lt;p&gt;I am a beginner in Apache Kafka. I am reading now some architecture documents and I find in a diagram 2 components: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Kafka Channel&lt;/li&gt;&#xA;&lt;li&gt;Kafka Broker&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What is the difference between these 2 components?&lt;/p&gt;&#xA;" OwnerUserId="44995" LastEditorUserId="29575" LastEditDate="2018-01-22T15:56:41.263" LastActivityDate="2018-01-22T15:56:41.263" Title="What is the difference between Kafka Channel and Kafka Broker?" Tags="|apache-kafka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26921" PostTypeId="2" ParentId="23323" CreationDate="2018-01-22T13:51:48.217" Score="6" Body="&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/therohk/india-headlines-news-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a massive dataset of news with categories which I created for exactly such a reason.&lt;/p&gt;&#xA;&lt;p&gt;Includes all the headlines published by Times of India from 2001-2019 with categories.&lt;/p&gt;&#xA;&lt;p&gt;Contains ~3 million entries.&lt;/p&gt;&#xA;" OwnerUserId="44998" LastEditorUserId="98307" LastEditDate="2020-08-17T01:00:15.710" LastActivityDate="2020-08-17T01:00:15.710" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="26923" PostTypeId="2" ParentId="26651" CreationDate="2018-01-22T14:35:59.653" Score="4" Body="&lt;p&gt;Use the below code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['Fruits_total'] = df[['Apples','Lemon','Banana']].sum(axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;axis = 1, shows that you are summing up the rows. This is what you missed in your code.&lt;/p&gt;&#xA;" OwnerUserId="38390" LastActivityDate="2018-01-22T14:35:59.653" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26924" PostTypeId="1" CreationDate="2018-01-22T14:39:34.960" Score="1" ViewCount="30" Body="&lt;p&gt;I am currently working on a product classification business case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have over 2 millions products from one source to classify in 23 categories. The input is the name of the product(ex:'Nike Air Max'). These products are already labeled. Whereas I can split my dataset into training and test set to compute the accuracy (which is 90%), I would like to predict categories of a whole new dataset, coming from an other source.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once it will be done, I need to check products one by one to know if my classifier works well on this new dataset because it is not labeled and I really need to know the accuracy of prediction for this new dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I know, the size of the sample coming from the new dataset (which has about 500k products) I need to check to be sure that my accuracy will have a small error margin?&lt;/p&gt;&#xA;" OwnerUserId="45000" LastEditorUserId="1330" LastEditDate="2021-07-17T23:21:27.190" LastActivityDate="2021-07-19T21:56:49.233" Title="How to compute the size of a significant sample for manual validation in classification?" Tags="|statistics|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26926" PostTypeId="1" CreationDate="2018-01-22T15:52:02.457" Score="1" ViewCount="1106" Body="&lt;h3&gt;Question:&lt;/h3&gt;&#xA;&lt;p&gt;Suggest me an approach to that kind of problem. Is it pure optimization problem?&#xA;What might Machine Learning techniques can be helpful here?&lt;/p&gt;&#xA;&lt;h3&gt;Case Study&lt;/h3&gt;&#xA;&lt;p&gt;My reduced case study is as follows:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;I have 3 types of ingredients (A, B, C). In each experiment, some&#xA;combination from the ingredients is chosen and applied on something.&#xA;Result can be &lt;strong&gt;1:poor&lt;/strong&gt;, &lt;strong&gt;2:middle&lt;/strong&gt;, &lt;strong&gt;3:high&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Example:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;| Input      | Output      |&#xA;|:----------:|:-----------:|&#xA;| A          |       2     | &#xA;| AB         |       1     |   &#xA;| ABC        |       2     | &#xA;| BC         |       3     | &#xA;| C          |       2     | &#xA;|...         |      ...    |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I need to find out what is the winning combination of ingredients according to the results.&lt;/p&gt;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="45005" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-23T18:55:08.220" Title="Find best (most efficient) combination of Ingredients" Tags="|machine-learning|algorithms|optimization|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="26928" PostTypeId="2" ParentId="26920" CreationDate="2018-01-22T15:55:04.860" Score="1" Body="&lt;h3&gt;What is a Kafka Broker?&lt;/h3&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://kafka.apache.org/intro&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kafka&lt;/a&gt; is a distributed streaming platform. It is run as a cluster on one or more nodes, and each node in the cluster is called a Kafka Broker.&lt;/p&gt;&#xA;&lt;h3&gt;What is a Kafka Channel?&lt;/h3&gt;&#xA;&lt;p&gt;A &lt;a href=&quot;http://flume.apache.org/FlumeUserGuide.html#kafka-channel&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kafka Channel&lt;/a&gt; is a type of &lt;a href=&quot;http://flume.apache.org/FlumeUserGuide.html#flume-channels&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flume Channel&lt;/a&gt;. &lt;a href=&quot;http://flume.apache.org/FlumeUserGuide.html#overview&quot; rel=&quot;nofollow noreferrer&quot;&gt;Flume&lt;/a&gt; is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.&lt;/p&gt;&#xA;&lt;p&gt;The use of Apache Flume is not only restricted to log data aggregation. Since data sources are customizable, Flume can be used to transport massive quantities of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source possible.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A &lt;a href=&quot;http://flume.apache.org/FlumeUserGuide.html#flume-channels&quot; rel=&quot;nofollow noreferrer&quot;&gt;Channel&lt;/a&gt; is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Channels are the repositories where the events are staged on a agent.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So hopefully, it is now clear that a Kafka Channel, is a Flume Channel that uses Kafka.&lt;/p&gt;&#xA;" OwnerUserId="29575" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-22T15:55:04.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26929" PostTypeId="1" CreationDate="2018-01-22T16:56:57.453" Score="-2" ViewCount="67" Body="&lt;p&gt;Some machine learning projects produce great results for the client. Some don't, for a variety of reasons. What makes the difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can you manage scope, schedule, and costs for a machine learning project?&lt;/p&gt;&#xA;" OwnerUserId="8417" LastEditorUserId="8417" LastEditDate="2018-01-24T07:34:35.107" LastActivityDate="2018-01-24T07:34:35.107" Title="What can possibly go wrong?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2018-01-23T01:51:34.360" ContentLicense="CC BY-SA 3.0" />
  <row Id="26930" PostTypeId="2" ParentId="26929" CreationDate="2018-01-22T17:29:51.570" Score="3" Body="&lt;p&gt;Very open question- here are my two cents: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is the problem solvable? &#xA;Some problems (like predicting long term stock markets or, for the UK police, finding the difference between pornographic imagery and dunes &lt;a href=&quot;https://gizmodo.com/british-cops-want-to-use-ai-to-spot-porn-but-it-keeps-m-1821384511&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;)&#xA;are just very difficult. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Quality of data. If your data is an unreliable pile of garbage this can make your life a living nightmare of data cleaning, imputation and tons of exceptions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How willing is the client to actually accept all data findings. I have seen this time over time. Some people are really good at cherry picking the one indicator in their preferred direction while the rest of them points in the other direction. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Usability of the result. If you produce a result that does not scale well (like a computational expensive predictor or similar) it might just not be feasible to use it. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="23305" LastEditDate="2018-01-22T18:22:00.920" LastActivityDate="2018-01-22T18:22:00.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26933" PostTypeId="2" ParentId="26742" CreationDate="2018-01-22T20:08:39.723" Score="1" Body="&lt;h2&gt;Attempt at a Conceptual Approach&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WaPcT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WaPcT.png&quot; alt=&quot;Graphical Representation of Conv. Filter&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will attempt to answer this question focusing primarily on the concept itself, rather than the mathematics which have little explicit relevance. Therefore, correct me if I am wrong, but I believe that when you state &lt;strong&gt;&lt;em&gt;&lt;code&gt;&quot;Localized Filter&quot;&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt; you are referring to  what is also, commonly known as a &lt;strong&gt;&lt;em&gt;&lt;code&gt;&quot;Convolution Layer&quot;&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt; or &lt;strong&gt;&lt;em&gt;&lt;code&gt;&quot;Neuron&quot;&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt; even occasionally a &lt;strong&gt;&lt;em&gt;&lt;code&gt;&quot;Kernel&quot;&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;. In a manner, of which I hope will be suitable to your application, I would have one visualize this as a sort of mask, who's depth &lt;strong&gt;MUST&lt;/strong&gt; equal that of the source image depth, that is to be masking the source input data, centered at some point (x,y,z), then sequentially shifted one position at a time in a manner such that, the area covered by the filter in the &lt;strong&gt;&lt;em&gt;second&lt;/em&gt;&lt;/strong&gt; position will be overlapping the &lt;strong&gt;&lt;em&gt;first&lt;/em&gt;&lt;/strong&gt; all but one column. To better understand this concept, imagine the way the data input to a CNN would be &lt;strong&gt;&lt;em&gt;&quot;seen&quot;&lt;/em&gt;&lt;/strong&gt; by the machine, the following images attached to this guide, show the general concept; although the picture above this paragraph is &lt;strong&gt;&lt;em&gt;AN ACTUAL GRAPHICAL REPRESENTATION OF A NEURON&lt;/em&gt;&lt;/strong&gt;, the same principle applies to both:  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Some array -&gt; &lt;strong&gt;&lt;em&gt;N&lt;sub&gt;1&lt;/strong&gt;&lt;/sub&gt;x&lt;strong&gt;N&lt;sub&gt;2&lt;/strong&gt;&lt;/sub&gt;x&lt;strong&gt;N&lt;sub&gt;3&lt;/strong&gt;&lt;/sub&gt;&lt;/em&gt; of numbers. &#xA;  &lt;a href=&quot;https://i.stack.imgur.com/VwLX0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VwLX0.png&quot; alt=&quot;CNN, how we see, how machine see&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;After successfully shifting, the algorithm will then perform an &lt;strong&gt;element wise&lt;/strong&gt; multiplication across the values in the original matrix(&lt;strong&gt;&lt;em&gt;X&lt;sub&gt;0&lt;/sub&gt;&lt;/em&gt;&lt;/strong&gt;) that the &lt;strong&gt;&lt;em&gt;&quot;conv layer&quot;&lt;/em&gt;&lt;/strong&gt; currently &lt;strong&gt;&lt;em&gt;Filters&lt;/em&gt;&lt;/strong&gt; and its own numeric values(&lt;strong&gt;&lt;em&gt;X&lt;sub&gt;1&lt;/sub&gt;&lt;/em&gt;&lt;/strong&gt;) then stored in some other array (&lt;strong&gt;&lt;em&gt;A&lt;/em&gt;&lt;/strong&gt;), referred to as the &lt;strong&gt;&lt;em&gt;&lt;code&gt;&quot;Activation Map&quot;&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;, that becomes some degree smaller than the original data in relation to the filter size: &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;np.multiply(&lt;strong&gt;X&lt;sub&gt;0&lt;/strong&gt;&lt;/sub&gt;,&lt;strong&gt;X&lt;sub&gt;1&lt;/strong&gt;&lt;/sub&gt;)&lt;/em&gt; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Of which the result is stored in some Matrix, of numerals, &lt;strong&gt;Pt&lt;/strong&gt;, position total. Upon completion of the sequence of the entire original input data, the algorithm &#xA;then computes the sum of all elements and stores them, so to gain the ability to pass to the next layer of the CNN:  &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Pt&lt;sup&gt;final&lt;/strong&gt;&lt;/sup&gt; = sum(&lt;strong&gt;Pt&lt;/strong&gt;)&lt;/em&gt;&#xA;  &lt;a href=&quot;https://i.stack.imgur.com/OcoNU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OcoNU.png&quot; alt=&quot;Activation map representation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The filter itself, being representative of some pattern, for example a circle, would then result in the outcome of a large number after completing the second step if there was another circle within its area as the pixels that have any data within them will result in an element of the original array having some positive number. As an example reference the below picture's and notice the difference in the resulting Integers for both instance's:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rH3io.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rH3io.png&quot; alt=&quot;Image of filter&quot;&gt;&lt;/a&gt;&#xA;  &lt;a href=&quot;https://i.stack.imgur.com/aIfvL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aIfvL.png&quot; alt=&quot;Example image set&quot;&gt;&lt;/a&gt;&#xA;  &lt;a href=&quot;https://i.stack.imgur.com/vb3cI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vb3cI.png&quot; alt=&quot;Matching values&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;h2&gt;&lt;a href=&quot;https://i.stack.imgur.com/n07ey.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/n07ey.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/h2&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where as the opposite is true, of an area's values of which &lt;strong&gt;do not&lt;/strong&gt; match those of the &lt;strong&gt;&lt;em&gt;&quot;conv layer&quot;&lt;/em&gt;&lt;/strong&gt;. This scenario will result in a very low number, relative to a confirmed match, or even a zero if the element happens to be empty of data other than zero. This method is what comprises the &quot;secret layer&quot; in a CNN, which generally alternates in a sequence between different variants, &lt;strong&gt;&lt;em&gt;&quot;Convolution&quot;&lt;/em&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;em&gt;&quot;ReLU&quot;&lt;/em&gt;&lt;/strong&gt; as well as &lt;strong&gt;&lt;em&gt;&quot;Pool&quot;&lt;/em&gt;&lt;/strong&gt; layers.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FPS3F.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FPS3F.png&quot; alt=&quot;Fully connected CNN&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I hope that this explained better what is going on in the &lt;em&gt;localized filters&lt;/em&gt; that you came upon in your research! If I was to ambiguous on any part of the explanation, please feel free to espouse and I will try to extrapolate.&lt;/p&gt;&#xA;" OwnerUserId="44814" LastActivityDate="2018-01-22T20:08:39.723" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26934" PostTypeId="2" ParentId="21779" CreationDate="2018-01-22T22:04:51.343" Score="1" Body="&lt;p&gt;An ARIMA model with order of differencing d equal to 1 can model a time series with a unit root, which is a form of long-range dependence. If d=0 one has an ARMA model, which cannot model long-range dependence unless the number or AR or MA parameters is very high. To model long range dependence in a parsimonious manner, look at fractionally integrated ARMA models, known as ARFIMA models.&lt;/p&gt;&#xA;" OwnerUserId="44994" LastActivityDate="2018-01-22T22:04:51.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26935" PostTypeId="2" ParentId="8660" CreationDate="2018-01-22T22:34:48.620" Score="4" Body="&lt;p&gt;I've put together some example tensorflow code to help explain (the full, working code is in this &lt;a href=&quot;https://gist.github.com/sophiaray/c436a6f97c4ad0e624691b4c4661e959&quot; rel=&quot;nofollow noreferrer&quot;&gt;gist&lt;/a&gt;). This code implements the capsule network from the first part of section 2 in the paper you linked:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;N_REC_UNITS = 10&#xA;N_GEN_UNITS = 20&#xA;N_CAPSULES = 30&#xA;&#xA;# input placeholders&#xA;img_input_flat = tf.placeholder(tf.float32, shape=(None, 784))&#xA;d_xy = tf.placeholder(tf.float32, shape=(None, 2))&#xA;&#xA;# translate the image according to d_xy&#xA;img_input = tf.reshape(img_input_flat, (-1, 28, 28, 1))&#xA;trans_img = image.translate(img_input, d_xy)&#xA;flat_img = tf.layers.flatten(trans_img)&#xA;&#xA;capsule_img_list = []&#xA;&#xA;# build several capsules and store the generated output in a list&#xA;for i in range(N_CAPSULES):&#xA;    # hidden recognition layer&#xA;    h_rec = tf.layers.dense(flat_img, N_REC_UNITS, activation=tf.nn.relu)&#xA;    # inferred xy values&#xA;    xy = tf.layers.dense(h_rec, 2) + d_xy&#xA;    # inferred probability of feature&#xA;    p = tf.layers.dense(h_rec, 1, activation=tf.nn.sigmoid)&#xA;    # hidden generative layer&#xA;    h_gen = tf.layers.dense(xy, N_GEN_UNITS, activation=tf.nn.relu)&#xA;    # the flattened generated image&#xA;    cap_img = p*tf.layers.dense(h_gen, 784, activation=tf.nn.relu)&#xA;&#xA;    capsule_img_list.append(cap_img)&#xA;&#xA;# combine the generated images&#xA;gen_img_stack = tf.stack(capsule_img_list, axis=1)&#xA;gen_img = tf.reduce_sum(gen_img_stack, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does anyone know how the mapping between input pixels to capsules should work?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This depends on the network structure. For the first experiment in that paper (and the code above), each capsule has a receptive field that includes the entire input image. That's the simplest arrangement. In that case, it's a fully-connected layer between the input image and the first hidden layer in each capsule.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, the capsule receptive fields can be arranged more like CNN kernels with strides, as in the later experiments in that paper. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What exactly should be happening in the recognition units?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The recognition units are an internal representation that each capsule has. Each capsule uses this internal representation to calculate &lt;code&gt;p&lt;/code&gt;, the probability that the capsule's feature is present, and &lt;code&gt;xy&lt;/code&gt;, the inferred translation values. Figure 2 in that paper is a check to make sure the network is learning to use &lt;code&gt;xy&lt;/code&gt; correctly (it is). &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How it should be trained? Is it just standard back prop between every connection?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Specifically, you should train it as an autoencoder, using a loss that enforces similarity between the generated output and the original. Mean square error works well here. Aside from that, yes, you'll need to propagate the gradient descent with backprop. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss = tf.losses.mean_squared_error(img_input_flat, gen_img)&#xA;train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38379" LastEditorUserId="38379" LastEditDate="2018-01-23T23:01:34.960" LastActivityDate="2018-01-23T23:01:34.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26936" PostTypeId="1" CreationDate="2018-01-22T22:46:54.830" Score="0" ViewCount="1301" Body="&lt;p&gt;I am working on a text Categorization problem, the objective is to classify related companies into their corresponding categories. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Preparation -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Broke the documents in the list of tokens/ words. &lt;/li&gt;&#xA;&lt;li&gt;Removed numbers, stop words, punctuations.&lt;/li&gt;&#xA;&lt;li&gt;Performed stemming. &lt;/li&gt;&#xA;&lt;li&gt;Convert corpus into DTM and,&lt;/li&gt;&#xA;&lt;li&gt;specify the word length in-between(3, 10)&lt;/li&gt;&#xA;&lt;li&gt;Transformed the documents into TF-IDF vectors&lt;/li&gt;&#xA;&lt;li&gt;convert it into a matrix and further into a data frame&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These will be used a feature list for the&#xA;classification algorithm. Used Random forest (&quot;ranger&quot; by using mlr). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 853 categorized documents with 20 categories, and I used 75:25 ration for training and test dataset. I used mlr package in R. The classification accuracy I have managed to get is 45% and I need it to be at least 80%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also applied xgboost but got only 38% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Looked like the below-mentioned image: - &#xA;&lt;a href=&quot;https://i.stack.imgur.com/UulHh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UulHh.png&quot; alt=&quot;Click to see the image of data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example code I used so far: - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;terms &amp;lt;-DocumentTermMatrix(clean_corp,&#xA;                                      control=list(wordLengths=c(3, 9), &#xA;                                                   weighting = weightTfIdf))&#xA;&#xA;m &amp;lt;- as.matrix(terms)&#xA;vertical_market &amp;lt;- process$label&#xA;termsDf &amp;lt;- as.data.frame(m)&#xA;termsDf$label &amp;lt;- as.factor(label)&#xA;names(termsDf) = lapply(names(termsDf), as.name)&#xA;&#xA;words &amp;lt;- colnames(termsDf)&#xA;&#xA;# taking words with length more than 4&#xA;data &amp;lt;- termsDf[, str_length(words) &amp;lt; 16 &amp;amp; str_length(words) &amp;gt; 3]&#xA;colnames(data) &amp;lt;- c(paste0(&quot;w&quot;, 1:(ncol(data)-1)), &quot;label&quot;)&#xA;&#xA;    task &amp;lt;- makeClassifTask(data = data, target = &quot;label&quot;)&#xA;    lrn &amp;lt;- makeLearner(&quot;classif.ranger&quot;, mtry=100,respect.unordered.factors = FALSE, &#xA;                       predict.type = &quot;response&quot;)&#xA;    n &amp;lt;- nrow(data)&#xA;    set.seed(123)&#xA;    train.set &amp;lt;- sample(n, size = 2/3*n)&#xA;    test.set &amp;lt;- setdiff(1:n, train.set)&#xA;    model &amp;lt;- mlr::train(lrn, task, subset = train.set)&#xA;    pred &amp;lt;- predict(model, task = task, subset = test.set)&#xA;    performance(pred, measures = list(mmce,acc))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.&lt;/p&gt;&#xA;" OwnerUserId="45018" LastEditorUserId="45018" LastEditDate="2018-01-23T12:14:59.643" LastActivityDate="2018-01-23T12:14:59.643" Title="How to improve the accuracy of Random Forest for Text Categorization" Tags="|machine-learning|classification|nlp|text-mining|random-forest|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26937" PostTypeId="1" AcceptedAnswerId="26939" CreationDate="2018-01-22T23:16:58.353" Score="0" ViewCount="437" Body="&lt;p&gt;So I am implementing Word2Vec for the first time, and I have a set of training data that I would like to train a word2vec model on. Predictably, the problem is the dataset is rather large, and I have more limited computational power than I would like. This is a very common problem of course, but are there any ways to minimize the input text without too horribly affecting the performance? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I had the example sentences: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. Mr. Dursley was the director of a firm called runnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I could do some thing like take out the stop words... but as I googled, that's a bad idea according to &lt;a href=&quot;https://www.quora.com/Is-it-compulsory-to-remove-stop-words-with-word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;one link&lt;/a&gt; and a &lt;a href=&quot;https://stackoverflow.com/a/34737150/4573703&quot;&gt;good idea&lt;/a&gt; according to another... so which is it? I also looked into &lt;a href=&quot;https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;text summarization&lt;/a&gt; but that seems &lt;a href=&quot;https://hackernoon.com/how-to-run-text-summarization-with-tensorflow-d4472587602d&quot; rel=&quot;nofollow noreferrer&quot;&gt;harder than implementing word2vec&lt;/a&gt;. Another idea I naively had was to randomly take out X% of sentences from the text. But that would obviously be a performance hit in the model, and I'm not sure how big. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, are there any general methods for trying to do this, and is stop word removal a way to do it? &lt;/p&gt;&#xA;" OwnerUserId="35399" LastEditorUserId="35399" LastEditDate="2018-01-23T00:17:47.940" LastActivityDate="2018-01-23T00:37:55.130" Title="Reducing text input size into word2vec without affecting performance too badly?" Tags="|nlp|word2vec|word-embeddings|automatic-summarization|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26938" PostTypeId="1" AcceptedAnswerId="26945" CreationDate="2018-01-22T23:18:32.323" Score="48" ViewCount="27148" Body="&lt;p&gt;Apparently, in reinforcement learning, temporal-difference (TD) method is a bootstrapping method. On the other hand, Monte Carlo methods are not bootstrapping methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What exactly is bootstrapping in RL? What is a bootstrapping method in RL?&lt;/p&gt;&#xA;" OwnerDisplayName="user10640" LastEditorDisplayName="user10640" LastEditDate="2018-06-15T12:14:47.627" LastActivityDate="2023-12-09T19:00:55.693" Title="What exactly is bootstrapping in reinforcement learning?" Tags="|reinforcement-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26939" PostTypeId="2" ParentId="26937" CreationDate="2018-01-23T00:37:55.130" Score="2" Body="&lt;p&gt;I don't think there really is a right or wrong answer to the &quot;removing stopwords&quot; question.  Some people will argue that throwing away information will reduce model performance, while others argue that it'll increase noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I personally follow a simple rule of thumb.  If my model depends on sentence structure, then I keep stopwords.  If i'm modeling topics and more interested in important phrases, then I remove them.  This seems to work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're looking for ways to reduce you matrix, then yes removing stopwords is a perfectly acceptable idea.  Another thing you can do is apply common feature reduction techniques like LSA or chi2 to find the most important words and reduce your input space to the most meaningful words.  However, doing this may dramatically effect the performance or your word2vec model.  But if it is your only choice, then why not give it a go.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-23T00:37:55.130" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26940" PostTypeId="2" ParentId="26910" CreationDate="2018-01-23T02:25:19.597" Score="3" Body="&lt;p&gt;Ok let's go part by part. There's quite a few parts here where you do not take into consideration the bias in your network.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Choosing your inputs and output&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If the vector 0-6 is determined there really is no need to output 1-7. The 1-6 is already known and adding additional outputs will only add complexity to your model. Unless you have substantial amounts of data you want to keep your model as simple as possible in order to get good performance. Thus, I would output a simple neuron with a continuous value. You can use RMSE as your loss function with a regression output from your neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you should supplement the samples you put into your input space with some additional information that you might think would contain information about the trend line. For example, if I had 2 different products, bitcoin and gold, and their input vector was the same, I might expect the gold to have very little fluctuation but the bitcoin to have very high fluctuation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your input features to your network contain all the information from which your network will learn. Thus, you want to make sure that you are supplying sufficient information to have a meaningful prediction. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Deep learning is data hungry&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You will need approximately 100,000+ instances. Each instance is a set of features. These should be drawn independently and such that they are identically distributed. In other words, you want to get multiple trendlines from a varied source of data that you wish to use your network with and then you will randomly draw 0-6 points, that is your features, and 7 which will be your label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the data distribution you are trying to learn. If you want your network to classify cats/dogs, you need to give a wide range of different looking cats and dogs such that the network can identify the variance which exists in both of these classes. If you restrict the data source too much it will have high bias and will not generalize to novel data that you will later feed into it. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Try these things and let us know what happens.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-23T02:25:19.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26942" PostTypeId="1" AcceptedAnswerId="26956" CreationDate="2018-01-23T03:40:58.220" Score="2" ViewCount="14397" Body="&lt;p&gt;I have a function I am trying to debug which is yielding the following error message:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: &quot;concat&quot; mode can only merge layers with matching output&#xA;  shapes except for the concat axis. Layer shapes: [(None, 128, 80,&#xA;  256), (None, 64, 80, 80)]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm running a kernel from a Kaggle competition called &lt;a href=&quot;https://www.kaggle.com/c/dstl-satellite-imagery-feature-detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dstl Satellite Imagery Feature Detection&lt;/a&gt; (&lt;a href=&quot;https://www.kaggle.com/drn01z3/end-to-end-baseline-with-u-net-keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernel available here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the function where I am having issues merging a list of tensors into a single tensor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; def get_unet():&#xA;    inputs = Input((8, ISZ, ISZ))&#xA;    conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(inputs)&#xA;    conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv1)&#xA;    pool1 = MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;)(conv1)&#xA;&#xA;    conv2 = Convolution2D(64, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(pool1)&#xA;    conv2 = Convolution2D(64, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv2)&#xA;    pool2 = MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;)(conv2)&#xA;&#xA;    conv3 = Convolution2D(128, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(pool2)&#xA;    conv3 = Convolution2D(128, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv3)&#xA;    pool3 = MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;)(conv3)&#xA;&#xA;    conv4 = Convolution2D(256, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(pool3)&#xA;    conv4 = Convolution2D(256, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv4)&#xA;    pool4 = MaxPooling2D(pool_size=(2, 2), dim_ordering=&quot;th&quot;)(conv4)&#xA;&#xA;    conv5 = Convolution2D(512, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(pool4)&#xA;    conv5 = Convolution2D(512, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv5)&#xA;&#xA;    up6 = merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv5), conv4], mode='concat', concat_axis=1)&#xA;    conv6 = Convolution2D(256, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(up6)&#xA;    conv6 = Convolution2D(256, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv6)&#xA;&#xA;    up7 = merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv6), conv3], mode='concat', concat_axis=1)&#xA;    conv7 = Convolution2D(128, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(up7)&#xA;    conv7 = Convolution2D(128, 3, 3, activation='relu', border_mode='same')(conv7)&#xA;&#xA;    up8 = merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv7), conv2], mode='concat', concat_axis=1)&#xA;    conv8 = Convolution2D(64, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(up8)&#xA;    conv8 = Convolution2D(64, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv8)&#xA;&#xA;    up9 = merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv8), conv1], mode='concat', concat_axis=1)&#xA;    conv9 = Convolution2D(32, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(up9)&#xA;    conv9 = Convolution2D(32, 3, 3, activation='relu', border_mode='same', dim_ordering=&quot;th&quot;)(conv9)&#xA;&#xA;    conv10 = Convolution2D(N_Cls, 1, 1, activation='sigmoid', dim_ordering=&quot;th&quot;)(conv9)&#xA;&#xA;    model = Model(input=inputs, output=conv10)&#xA;    model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=[jaccard_coef, jaccard_coef_int, 'accuracy'])&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm running &lt;code&gt;keras&lt;/code&gt; with a &lt;code&gt;TensorFlow&lt;/code&gt; backend. My thoughts are that there are some compatibility issues with versions of software (i.e. the original code is over a year old). Or, perhaps I need to reshape the data somehow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What might be causing this error?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the full error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;&#xA;  File &quot;&amp;lt;ipython-input-1-e8f13915ac9b&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    runfile('/Users/aaron/temp/tmp/kaggle_dstl_v3.py', wdir='/Users/aaron/temp/tmp')&#xA;&#xA;  File &quot;/Users/aaron/anaconda3/envs/kaggle-dstl-env/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py&quot;, line 705, in runfile&#xA;    execfile(filename, namespace)&#xA;&#xA;  File &quot;/Users/aaron/anaconda3/envs/kaggle-dstl-env/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py&quot;, line 102, in execfile&#xA;    exec(compile(f.read(), filename, 'exec'), namespace)&#xA;&#xA;  File &quot;/Users/aaron/temp/tmp/kaggle_dstl_v3.py&quot;, line 513, in &amp;lt;module&amp;gt;&#xA;    model = train_net()&#xA;&#xA;  File &quot;/Users/aaron/temp/tmp/kaggle_dstl_v3.py&quot;, line 416, in train_net&#xA;    model = get_unet()&#xA;&#xA;  File &quot;/Users/aaron/temp/tmp/kaggle_dstl_v3.py&quot;, line 294, in get_unet&#xA;    up8 = merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv7), conv2], mode='concat', concat_axis=1)&#xA;&#xA;  File &quot;/Users/aaron/anaconda3/envs/kaggle-dstl-env/lib/python3.6/site-packages/keras/legacy/layers.py&quot;, line 458, in merge&#xA;    name=name)&#xA;&#xA;  File &quot;/Users/aaron/anaconda3/envs/kaggle-dstl-env/lib/python3.6/site-packages/keras/legacy/layers.py&quot;, line 111, in __init__&#xA;    node_indices, tensor_indices)&#xA;&#xA;  File &quot;/Users/aaron/anaconda3/envs/kaggle-dstl-env/lib/python3.6/site-packages/keras/legacy/layers.py&quot;, line 191, in _arguments_validation&#xA;    'Layer shapes: %s' % (input_shapes))&#xA;&#xA;ValueError: &quot;concat&quot; mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 128, 80, 256), (None, 64, 80, 80)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17539" LastEditorUserId="29169" LastEditDate="2021-02-13T16:44:25.730" LastActivityDate="2021-02-13T16:44:25.730" Title="&quot;concat&quot; mode can only merge layers with matching output shapes except for the concat axis" Tags="|deep-learning|keras|tensorflow|convolutional-neural-network|kaggle|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26943" PostTypeId="1" CreationDate="2018-01-23T06:34:33.647" Score="3" ViewCount="10137" Body="&lt;p&gt;I am trying to use CoNLL-2003 NER (English) Dataset and I am trying to utilize pretrained embeddings for it. I am using SENNA pretrained embeddings. Now I have around 20k words in my vocabulary and out of this I have embedding available for only 9.5k words. &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current approach is to initialize an array of &lt;code&gt;20k X embedding_size&lt;/code&gt; with zeros and initialize the 9.5k words whose embeddings is known to me and make all the embeddings learn-able.&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is what is the best way to do this? Any reference to such research will be very helpful ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also asked a similar question later on Stackoverflow, &lt;a href=&quot;https://stackoverflow.com/q/48395570/3316461&quot;&gt;here&lt;/a&gt; is the link.&lt;/p&gt;&#xA;" OwnerUserId="33704" LastEditorUserId="33704" LastEditDate="2018-01-24T04:58:14.507" LastActivityDate="2018-01-24T04:58:14.507" Title="How to initialize word-embeddings for Out of Vocabulary Word?" Tags="|machine-learning|deep-learning|nlp|word-embeddings|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="26945" PostTypeId="2" ParentId="26938" CreationDate="2018-01-23T08:22:06.773" Score="47" Body="&lt;p&gt;Bootstrapping in RL can be read as &quot;using one or more estimated values in the update step &lt;em&gt;for the same kind&lt;/em&gt; of estimated value&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most TD update rules, you will see something like this SARSA(0) update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$Q(s,a) \leftarrow Q(s,a) + \alpha(R_{t+1} + \gamma Q(s',a') - Q(s,a))$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The value &lt;span class=&quot;math-container&quot;&gt;$R_{t+1} + \gamma Q(s',a')$&lt;/span&gt; is an estimate for the true value of &lt;span class=&quot;math-container&quot;&gt;$Q(s,a)$&lt;/span&gt;, and also called the TD target. It is a bootstrap method because we are in part using a Q value to update another Q value. There is a small amount of real observed data in the form of &lt;span class=&quot;math-container&quot;&gt;$R_{t+1}$&lt;/span&gt;, the immediate reward for the step, and also in the state transition &lt;span class=&quot;math-container&quot;&gt;$s \rightarrow s'$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contrast with Monte Carlo where the equivalent update rule might be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$Q(s,a) \leftarrow Q(s,a) + \alpha(G_{t} - Q(s,a))$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;span class=&quot;math-container&quot;&gt;$G_{t}$&lt;/span&gt; was the total discounted reward  at time &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt;, assuming in this update, that it started in state &lt;span class=&quot;math-container&quot;&gt;$s$&lt;/span&gt;, taking action &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt;, then followed the current policy until the end of the episode. Technically, &lt;span class=&quot;math-container&quot;&gt;$G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$T$&lt;/span&gt; is the time step for the terminal reward and state. Notably, this target value does not use any existing estimates (from other Q values) at all, it only uses a set of observations (i.e., rewards) from the environment. As such, it is guaranteed to be &lt;em&gt;unbiased&lt;/em&gt; estimate of the true value of &lt;span class=&quot;math-container&quot;&gt;$Q(s,a)$&lt;/span&gt;, as it is technically a &lt;em&gt;sample&lt;/em&gt; of &lt;span class=&quot;math-container&quot;&gt;$Q(s,a)$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main disadvantage of bootstrapping is that it is biased towards whatever your starting values of &lt;span class=&quot;math-container&quot;&gt;$Q(s',a')$&lt;/span&gt; (or &lt;span class=&quot;math-container&quot;&gt;$V(s')$&lt;/span&gt;) are. Those are are most likely wrong, and the update system can be unstable as a whole because of too much self-reference and not enough real data - this is a problem with off-policy learning (e.g. Q-learning) using neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without bootstrapping, using longer trajectories, there is often &lt;em&gt;high variance&lt;/em&gt; instead, which, in practice, means you need more samples before the estimates converge. So, despite the problems with bootstrapping, if it can be made to work, it may learn significantly faster, and is often preferred over Monte Carlo approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can compromise between Monte Carlo sample based methods and single-step TD methods that bootstrap by using a mix of results from different length trajectories. This is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Temporal_difference_learning#TD-Lambda&quot; rel=&quot;noreferrer&quot;&gt;TD(&lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt;) learning&lt;/a&gt;, and there are a variety of specific methods such as SARSA(&lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt;) or Q(&lt;span class=&quot;math-container&quot;&gt;$\lambda$&lt;/span&gt;).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorDisplayName="user10640" LastEditDate="2018-09-23T13:04:39.017" LastActivityDate="2018-09-23T13:04:39.017" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="26947" PostTypeId="1" CreationDate="2018-01-23T09:39:17.013" Score="15" ViewCount="11957" Body="&lt;p&gt;In the Sequence-to-Sequence models, we often see that the START (e.g. &lt;code&gt;&amp;lt;s&amp;gt;&lt;/code&gt;) and END (e.g. &lt;code&gt;&amp;lt;/s&amp;gt;&lt;/code&gt;) symbols are added to the inputs and outputs before training the model and before inference/decoding unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. &lt;a href=&quot;http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html&quot; rel=&quot;noreferrer&quot;&gt;http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SOS_token = 0&#xA;EOS_token = 1&#xA;&#xA;&#xA;class Lang:&#xA;    def __init__(self, name):&#xA;        self.name = name&#xA;        self.word2index = {}&#xA;        self.word2count = {}&#xA;        self.index2word = {0: &quot;SOS&quot;, 1: &quot;EOS&quot;}&#xA;        self.n_words = 2  # Count SOS and EOS&#xA;&#xA;    def addSentence(self, sentence):&#xA;        for word in sentence.split(' '):&#xA;            self.addWord(word)&#xA;&#xA;    def addWord(self, word):&#xA;        if word not in self.word2index:&#xA;            self.word2index[word] = self.n_words&#xA;            self.word2count[word] = 1&#xA;            self.index2word[self.n_words] = word&#xA;            self.n_words += 1&#xA;        else:&#xA;            self.word2count[word] += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Is there a technical definition or academic explanation of why that is necessary?&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Or does that need to add the END symbol only applies to Natural Language Processing task where the sentence generation needs to end?&lt;/strong&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;But what does the START symbol do? Other than to take the initial state where the trained network will start inferrring.&lt;/strong&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="122" LastEditorUserId="122" LastEditDate="2018-01-23T12:33:58.757" LastActivityDate="2018-02-22T16:31:49.267" Title="Why do we need to add START &lt;s&gt; + END &lt;/s&gt; symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?" Tags="|deep-learning|rnn|sequence-to-sequence|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26948" PostTypeId="1" CreationDate="2018-01-23T09:56:05.400" Score="1" ViewCount="149" Body="&lt;p&gt;I am looking for a public data-set of images that differ from each other only slightly, so that after applying &lt;code&gt;PCA&lt;/code&gt; they can be reconstructed with a small error from very few &lt;code&gt;PCA&lt;/code&gt; coefficients. It can be any type of images, the purpose is only to demonstrate an extreme example of &lt;code&gt;PCA&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45034" LastEditorUserId="28175" LastEditDate="2018-01-23T12:27:55.493" LastActivityDate="2024-02-14T06:02:45.300" Title="Dataset of extremely low-dimensional images for PCA" Tags="|machine-learning|dataset|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26949" PostTypeId="2" ParentId="26947" CreationDate="2018-01-23T10:25:21.703" Score="12" Body="&lt;p&gt;Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the embedding, and we don't want the encoder to have to guess as to whether or not the input sentence is complete or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The EOS token is important for the decoder as well: the explicit &quot;end&quot; token allows the decoder to emit arbitrary-length sequences. The decoder will tell us when it's done emitting tokens: without an &quot;end&quot; token, we would have no idea when the decoder is done talking to us and continuing to emit tokens will produce gibberish.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The start-of-sequence (SOS) token is more important for the decoder: the decoder will progress by taking the tokens it emits as inputs (along with the embedding and hidden state, or using the embedding to initialize the hidden state), so before it has emitted anything it needs a token of some kind to start with. Hence, the SOS token.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, if we're using a bidirectional RNN for the encoder, we're definitely going to want to use both SOS and EOS tokens since the SOS token will signal to the reversed-input layer when the input is complete (otherwise, how would it know?). &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-23T10:25:21.703" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26950" PostTypeId="2" ParentId="26699" CreationDate="2018-01-23T10:58:03.327" Score="23" Body="&lt;p&gt;If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the &lt;em&gt;order&lt;/em&gt; in which the tree is expanded. Since we don't normally grow trees to their full depth, order matters: application of early stopping criteria and pruning methods can result in very different trees. Because leaf-wise chooses splits based on their contribution to the global loss and not just the loss along a particular branch, it often (not always) will learn lower-error trees &amp;quot;faster&amp;quot; than level-wise. I.e. for a given number of leaves, leaf-wise will probably out-perform level-wise. As you add more nodes, without stopping or pruning they will converge to the same performance because they will literally build the same tree eventually.&lt;/p&gt;&#xA;&lt;p&gt;Reference:&lt;/p&gt;&#xA;&lt;p&gt;Shi, H. (2007). &lt;strong&gt;Best-first Decision Tree Learning&lt;/strong&gt; (Thesis, Master of Science (MSc)). The University of Waikato, Hamilton, New Zealand. Retrieved from &lt;a href=&quot;https://hdl.handle.net/10289/2317&quot; rel=&quot;noreferrer&quot;&gt;https://hdl.handle.net/10289/2317&lt;/a&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;EDIT: Regarding your first question, both C4.5 and CART are depth-first examples, not best-first. Here's some relevant content from the reference above:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h2&gt;1.2.1 Standard decision trees&lt;/h2&gt;&#xA;&lt;p&gt;Standard algorithms such as C4.5 (Quinlan, 1993) and CART (Breiman et al., 1984) for the top-down&#xA;induction of decision trees expand nodes in depth-first order in each&#xA;step using the divide-and-conquer strategy. Normally, at each node of&#xA;a decision tree, testing only involves a single attribute and the&#xA;attribute value is compared to a constant. The basic idea of standard&#xA;decision trees is that, first, select an attribute to place at the&#xA;root node and make some branches for this attribute based on some&#xA;criteria (e.g. information or Gini index). Then, split training&#xA;instances into subsets, one for each branch extending from the root&#xA;node. The number of subsets is the same as the number of branches.&#xA;Then, this step is repeated for a chosen branch, using only those&#xA;instances that actually reach it. A fixed order is used to expand&#xA;nodes (normally, left to right). If at any time all instances at a&#xA;node have the same class label, which is known as a pure node,&#xA;splitting stops and the node is made into a terminal node. This&#xA;construction process continues until all nodes are pure. It is then&#xA;followed by a pruning process to reduce overfittings (see Section&#xA;1.3).&lt;/p&gt;&#xA;&lt;h2&gt;1.2.2 Best-first decision trees&lt;/h2&gt;&#xA;&lt;p&gt;Another possibility, which so far appears to only have been evaluated in the context of boosting&#xA;algorithms (Friedman et al., 2000), is to expand nodes in best-first&#xA;order instead of a fixed order. This method adds the ”best” split node&#xA;to the tree in each step. The ”best” node is the node that maximally&#xA;reduces impurity among all nodes available for splitting (i.e. not&#xA;labelled as terminal nodes). Although this results in the same&#xA;fully-grown tree as standard depth-first expansion, it enables us to&#xA;investigate new tree pruning methods that use cross-validation to&#xA;select the number of expansions. Both pre-pruning and post-pruning can&#xA;be performed in this way, which enables a fair comparison between them&#xA;(see Section 1.3).&lt;/p&gt;&#xA;&lt;p&gt;Best-first decision trees are constructed in a divide-and-conquer&#xA;fashion similar to standard depth-first decision trees. The basic idea&#xA;of how a best-first tree is built is as follows. First, select an&#xA;attribute to place at the root node and make some branches for this&#xA;attribute based on some criteria. Then, split training instances into&#xA;subsets, one for each branch extending from the root node. In this&#xA;thesis only binary decision trees are considered and thus the number&#xA;of branches is exactly two. Then, this step is repeated for a chosen&#xA;branch, using only those instances that actually reach it. In each&#xA;step we choose the ”best” subset among all subsets that are available&#xA;for expansions. This constructing process continues until all nodes&#xA;are pure or a specific number of expansions is reached. Figure 1.1&#xA;shows the difference in split order between a hypothetical binary&#xA;best-first tree and a hypothetical binary depth-first tree. Note that&#xA;other orderings may be chosen for the best-first tree while the order&#xA;is always the same in the depth-first case.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="13348" LastEditDate="2022-06-23T17:13:59.767" LastActivityDate="2022-06-23T17:13:59.767" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="26951" PostTypeId="2" ParentId="26948" CreationDate="2018-01-23T11:17:33.767" Score="0" Body="&lt;p&gt;Actually in your case I guess the pure images are not that important. The features that you extract from them are important because if your feature space is constructed base on intensity of images at different picture elements, pixels, then you will need so many coefficients. As an easy solution, use &lt;code&gt;MNIST&lt;/code&gt; digits and use &lt;a href=&quot;https://www.researchgate.net/publication/40902683_A_Survey_of_Shape_Feature_Extraction_Techniques&quot; rel=&quot;nofollow noreferrer&quot;&gt;shape features&lt;/a&gt; to extract features from the images of numbers. You can use plausible number of features and then use &lt;code&gt;PCA&lt;/code&gt; for the data that is in the new feature space that you have just constructed. In this case smaller number of coefficients will be needed if the features are fine.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-23T11:22:51.823" LastActivityDate="2018-01-23T11:22:51.823" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26952" PostTypeId="2" ParentId="25160" CreationDate="2018-01-23T11:53:14.407" Score="1" Body="&lt;p&gt;I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like &lt;a href=&quot;https://datascience.stackexchange.com/users/381/emre&quot;&gt;Emre&lt;/a&gt; said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this &lt;a href=&quot;https://www.youtube.com/watch?v=Gol_qOgRqfA&amp;amp;list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&amp;amp;index=8&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44139" LastEditorUserId="98307" LastEditDate="2020-08-11T08:50:25.933" LastActivityDate="2020-08-11T08:50:25.933" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="26953" PostTypeId="2" ParentId="26899" CreationDate="2018-01-23T12:05:01.250" Score="1" Body="&lt;p&gt;In responses to the comment requesting real world application.  I have not found models like GAM used particularly often in the industry, but there were notable exceptions in my career.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Insurance companies, especially life insurance companies, tend to prefer an explainable model over one that just makes good predictions.  In this space, I have used GAM to model spatial distribution.  I can't go into particular details, but to give you a related example, imagine an experiment design regarding the survey of fish.  A properly designed survey will stratified an environment; however, within the environment there exist covariates.  The traditional approach would be at apply a GLM and add covariates into the model.  However, it's easy to see that some predictors may not be linear.  For example, fish are drawn to different temperature, so the relation for the environment of the water and the temperature of the wish should probably not be modeled in a linear fashion.  Thankfully we could use a cubic spline instead and produce a readily interpretable result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example, would be in credit scoring.  In practice, there's usually a myriad of mixed models working together to produce a credit score and there exist different types of models for different level of products and risk.  Generically speaking though, some of these models can be hard to understand and find what driving factors effect it.  MARS can be used as a good baseline piecewise model for credit scores.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, it's like any other model.  There's a use, time and place for it.  I've found that companies that rely heavily on more traditional statistics and process understanding are more inclined to use models like GLM or GAM, while newer tech heavy companies favor the more blackbox quick iteration push to production approach.  Neither is better than the other, just that each industry has its own standards and goals.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-23T20:38:54.263" LastActivityDate="2018-01-23T20:38:54.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26954" PostTypeId="2" ParentId="26926" CreationDate="2018-01-23T12:10:46.743" Score="2" Body="&lt;p&gt;This is an optimization problem and optimization is all what Machine Learning does in classic sense (mostly not unsupervised approaches).&lt;/p&gt;&#xA;&lt;p&gt;Your problem can be designed to be solved by Genetic Algorithms. For this you need to make a sequences of ingrediants which is called chromosome (genetic representation of solution) and an objective function which is to be minimized/maximized according to the chromosomes (it's usually called fitness function and is to be maximized then).&lt;/p&gt;&#xA;&lt;p&gt;You have genetic operators to be applied randomly on your chromosomes:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Reproduction&lt;/strong&gt;: 2 parent chromosomes result in two children.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Mutation&lt;/strong&gt;: One single bit (one of ingrediants) is randomly chosen to&#xA;be switched with another one (like in a genetically normal family, a&#xA;baby is born with a genetic phenomenon).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Natural Selection&lt;/strong&gt;: Uncle Darwin believed that nature &amp;quot;keeps the best&#xA;and  discards the rest&amp;quot;. But who is the best? Well, you already have a fitness function!&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Continue till you reach the optimum and select the best chromosome.&lt;/p&gt;&#xA;&lt;p&gt;Of course it was the intuitive explanation of this beautiful idea. See more &lt;a href=&quot;https://ocw.mit.edu/courses/institute-for-data-systems-and-society/ids-338j-multidisciplinary-system-design-optimization-spring-2010/lecture-notes/MITESD_77S10_lec11.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and if you needed implementation details please drop a comment here.&lt;/p&gt;&#xA;&lt;p&gt;Good Luck.&lt;/p&gt;&#xA;&lt;h1&gt;Update: More Details&lt;/h1&gt;&#xA;&lt;p&gt;I have an optimization problem which is too complex or large to be solved analytically so I need a heuristic method to approximate the optimal solution. For instance each subset of &lt;span class=&quot;math-container&quot;&gt;$28$&lt;/span&gt; ingredients outputs a result &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$y \in [0,100]$&lt;/span&gt; (it's much bigger than your sample problem just to use optimization properly).&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Initialization:&lt;/strong&gt; Randomly produce &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; combinations of material as initial population. Choose probabilities &lt;span class=&quot;math-container&quot;&gt;$P_{crossover}=0.25$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$P_{mutation}=0.01$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$n_{selection}=250$&lt;/span&gt; (values are just examples)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Evaluation of Fitness:&lt;/strong&gt; Evaluate the output for each of them&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;| input         | output &#xA;| ------------- |:-------------:|&#xA;| ABCDEFGH      | 23.5          |&#xA;| BDCGTRFJ      | 45.6          |&#xA;| ZHTRFGOIUPOIZ | 18.3          |&#xA;| ...           | ...           |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Choose top &lt;span class=&quot;math-container&quot;&gt;$n_{selection}$&lt;/span&gt; and ignore the rest. If you found an optimal solution (in your case you said output=3 is desired) stop, if not:&lt;/li&gt;&#xA;&lt;li&gt;According to &lt;span class=&quot;math-container&quot;&gt;$P_{crossover}$&lt;/span&gt; choose 25% of survived samples and reproduce children by combining them through a breaking point (image from &lt;a href=&quot;http://www.wikiwand.com/en/Crossover_(genetic_algorithm)&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yIWv4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yIWv4.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;For instance if &lt;span class=&quot;math-container&quot;&gt;$ABCDEFGH$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$BCDEFGHJ$&lt;/span&gt; are survived, then choose a breaking point and exchange their parts (the image is 2-point crossover. You may apply 1-point crossover. It's all about the variations of the same concept). Technically the crossover point should be also determined probabilistically but I just skipped it. Break them from wherever you want! Here we choose &lt;span class=&quot;math-container&quot;&gt;$3^{rd}$&lt;/span&gt; gene for this so &lt;span class=&quot;math-container&quot;&gt;$ABCDEFGH$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$BCDEFGHJ$&lt;/span&gt; will become &lt;span class=&quot;math-container&quot;&gt;$ABDEFGHJ$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$BCCDEFGH$&lt;/span&gt;. Two children out of two parents. Apply it on all chosen survived samples.&lt;/p&gt;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;According to &lt;span class=&quot;math-container&quot;&gt;$P_{mutation}$&lt;/span&gt; choose 1% of survived samples and randomly(again, technically with a predefined probability) change one material in it (technically the samples are usually binary so you flip between &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;. Here as you have nominal variables you need to change the variable). Let's say in survived sample &lt;span class=&quot;math-container&quot;&gt;$FVBGTZHU$&lt;/span&gt; I randomly choose &lt;span class=&quot;math-container&quot;&gt;$V$&lt;/span&gt; and exchange it with &lt;span class=&quot;math-container&quot;&gt;$P$&lt;/span&gt; so I get &lt;span class=&quot;math-container&quot;&gt;$FPBGTZHU$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Now the new generation is born (keep it as the same size of the original population for better structure). Go back to 2 and continue till you find your answer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;For different variations of GA (Elitism, Island Model, etc.) see the literature for &lt;a href=&quot;https://en.wikipedia.org/wiki/Evolutionary_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Evolutionary Algorithms&lt;/a&gt; e.g. &lt;a href=&quot;https://www.ri.cmu.edu/pub_files/pub2/baluja_shumeet_1995_1/baluja_shumeet_1995_1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-23T18:55:08.220" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26955" PostTypeId="1" AcceptedAnswerId="26957" CreationDate="2018-01-23T12:25:04.987" Score="2" ViewCount="3505" Body="&lt;p&gt;I'm trying to see how well a decision tree classifier performs on my input. For this I'm trying to use the &lt;a href=&quot;http://scikit-learn.org/stable/modules/learning_curve.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;validation and learning curves&lt;/a&gt; and SKLearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross-validation methods&lt;/a&gt;. However, they differ, and I don't know what to make of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The validation curve shows up as follows:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/MoZqH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MoZqH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on varying the maximum depth parameter, I'm getting worse and worse cross-val scores. However, when I try the &lt;code&gt;cross_val_score&lt;/code&gt;, I get ~72% accuracy reliably:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LjBDY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LjBDY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I was using the default tree depth for &lt;code&gt;clf&lt;/code&gt; here, it still puzzles me how the validation curve never reaches even 0.6, but the cross-val scores are all above 0.7. What does this mean? Why is there a discrepancy?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Code for reference below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the Validation curve:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;&#xA;from sklearn.datasets import load_digits&#xA;from sklearn.svm import SVC&#xA;from sklearn.model_selection import validation_curve&#xA;&#xA;X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()&#xA;&#xA;param_range = np.arange(1, 50, 5)&#xA;train_scores, test_scores = validation_curve(&#xA;    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name=&quot;max_depth&quot;, param_range=param_range,&#xA;    cv=None, scoring=&quot;accuracy&quot;, n_jobs=1)&#xA;train_scores_mean = np.mean(train_scores, axis=1)&#xA;train_scores_std = np.std(train_scores, axis=1)&#xA;test_scores_mean = np.mean(test_scores, axis=1)&#xA;test_scores_std = np.std(test_scores, axis=1)&#xA;&#xA;plt.title(&quot;Validation Curve with Decision Tree Classifier&quot;)&#xA;plt.xlabel(&quot;max_depth&quot;)&#xA;#plt.xticks(param_range)&#xA;plt.ylabel(&quot;Score&quot;)&#xA;plt.ylim(0.0, 1.1)&#xA;lw = 2&#xA;plt.plot(param_range, train_scores_mean, label=&quot;Training score&quot;,&#xA;             color=&quot;darkorange&quot;, lw=lw)&#xA;plt.fill_between(param_range, train_scores_mean - train_scores_std,&#xA;                 train_scores_mean + train_scores_std, alpha=0.2,&#xA;                 color=&quot;darkorange&quot;, lw=lw)&#xA;plt.plot(param_range, test_scores_mean, label=&quot;Cross-validation score&quot;,&#xA;             color=&quot;navy&quot;, lw=lw)&#xA;plt.fill_between(param_range, test_scores_mean - test_scores_std,&#xA;                 test_scores_mean + test_scores_std, alpha=0.2,&#xA;                 color=&quot;navy&quot;, lw=lw)&#xA;plt.legend(loc=&quot;best&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the cross-val scores:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = DecisionTreeClassifier(class_weight='balanced')&#xA;X_train, X_test, y_train, y_test = train_test_split(&#xA;    X, y, test_size=0.33, random_state=42)&#xA;clf.fit(X_train, y_train)&#xA;y_pred = clf.predict(X_test)&#xA;clf.score(X_test, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE&lt;/strong&gt;&#xA;A comment has been asked about shuffling. When I shuffle the data by &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()&#xA;indices = np.arange(y.shape[0])&#xA;np.random.shuffle(indices)&#xA;X, y = X[indices], y[indices]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zaAcd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zaAcd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which makes even less sense to me. What does this mean? &lt;/p&gt;&#xA;" OwnerUserId="21254" LastEditorUserId="28175" LastEditDate="2018-01-23T13:13:52.883" LastActivityDate="2020-05-19T13:16:33.147" Title="validation_curve differs from cross_val_score?" Tags="|machine-learning|python|scikit-learn|decision-trees|cross-validation|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="26956" PostTypeId="2" ParentId="26942" CreationDate="2018-01-23T12:34:23.610" Score="2" Body="&lt;p&gt;First of all, you are correct that your code is old as some functions being used are deprecated (e.g. Convolution2D is now Conv2D &lt;a href=&quot;https://github.com/keras-team/keras/wiki/Keras-2.0-release-notes#layers&quot; rel=&quot;nofollow noreferrer&quot;&gt;see here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the error clearly states that you are trying to concatenate two tensors that their &lt;strong&gt;dimensions do not match&lt;/strong&gt;. When concatenating two tensors along a specific axis, all other dimensions except the one being concatenated must be the &lt;strong&gt;same&lt;/strong&gt;. In your case, you are trying to concatenate along axis=1, but the last dimension is different (256 for the first tensor and 80 for the second). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend basing your U-Net code on a newer implementation (&lt;a href=&quot;https://github.com/zhixuhao/unet/blob/master/unet.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;link1&lt;/a&gt;, &lt;a href=&quot;https://github.com/zizhaozhang/unet-tensorflow-keras/blob/master/model.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;link2&lt;/a&gt;, &lt;a href=&quot;https://www.kaggle.com/cjansen/u-net-in-keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;link3&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These implementations replace your &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;merge([UpSampling2D(size=(2, 2), dim_ordering=&quot;th&quot;)(conv5), conv4], mode='concat', concat_axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;layers with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;up_conv5 = UpSampling2D(size=(2, 2), data_format=&quot;channels_last&quot;)(conv5)&#xA;ch, cw = get_crop_shape(conv4, up_conv5)&#xA;crop_conv4 = Cropping2D(cropping=(ch,cw), data_format=&quot;channels_last&quot;)(conv4)&#xA;up6   = concatenate([up_conv5, crop_conv4], axis=concat_axis)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which also crops the height and the width of the previous layer to perform the concatenation correctly (the shape of the crop is determined through an auxiliary function).  &lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-01-23T12:34:23.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26957" PostTypeId="2" ParentId="26955" CreationDate="2018-01-23T12:54:02.973" Score="0" Body="&lt;p&gt;First of all, you have to shuffle your data because it seems that the model has learned a special pattern in the training data which has not occurred in the test data so much. After that, suppose that you get a validation curve like the current one. As you can see, Increasing the value of depth, does not change the learning. The two lines are parallel. In cases which each of the lines may have intersection, the upper line has negative slope and the lower one has positive slope, in the future on seen levels, you may want to increase the number of levels, not in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having same error, means that you are not over-fitting. but as you can see the amount of learning is not too much which means that you are having high bias problem, which means you have not learned the problem so well. In this case means that your current feature space maybe has high Bayes error which means that there are samples which have same features and different labels. Actually the distributions of different classes overlap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is something to argue about decision trees. If you have numerical features which are continuous, you may not have exactly same input patterns but they have overlap in their range.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-23T13:23:33.393" LastActivityDate="2018-01-23T13:23:33.393" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="26958" PostTypeId="1" AcceptedAnswerId="26987" CreationDate="2018-01-23T13:35:15.817" Score="1" ViewCount="292" Body="&lt;p&gt;I recently started playing with SVMs for a one class classification, I was able to get some reasonable classifications from real data and but was trying to optimize the nu and gamma parameters when I came across this example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the code below, I train an SVM with an array of ones, then I present the same array of ones for classification and it classifies all ones as outliers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn import svm&#xA;import numpy as np&#xA;&#xA;nu = 0.01&#xA;gamma = 1&#xA;ones = pd.DataFrame(np.ones(100))&#xA;clf = svm.OneClassSVM(nu=nu, kernel=&quot;rbf&quot;, gamma=gamma)&#xA;clf.fit(ones)&#xA;ones[&quot;predicted&quot;] = clf.predict(ones)&#xA;#Returns -1 for all entries&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: why does this happen? I thought this data would be trivial for any parameter configuration.&lt;/p&gt;&#xA;" OwnerUserId="42781" LastEditorUserId="29169" LastEditDate="2019-06-07T06:23:44.010" LastActivityDate="2020-07-30T12:57:15.560" Title="Why won't my SVM learn a sequence of repeated elements" Tags="|python|scikit-learn|svm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="26959" PostTypeId="1" AcceptedAnswerId="26960" CreationDate="2018-01-23T13:40:51.730" Score="3" ViewCount="1978" Body="&lt;p&gt;I have a regression problem which I solved using &lt;code&gt;SVR&lt;/code&gt;. Accidentally, I normalized my output along with the inputs by removing the mean and dividing by standard deviation from each feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surprisingly, the Rsquare score increased by 10%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can one explain the impact of output normalization for svm regression? &lt;/p&gt;&#xA;" OwnerUserId="40515" LastEditorUserId="28175" LastEditDate="2018-01-23T14:52:17.020" LastActivityDate="2018-01-23T14:52:17.020" Title="Should the output of regression models, like SVR, be normalized?" Tags="|machine-learning|regression|svm|svr|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26960" PostTypeId="2" ParentId="26959" CreationDate="2018-01-23T13:57:09.917" Score="3" Body="&lt;p&gt;In regression problems it is customary to normalize the output too, because the scale of output and input features may differ. After getting the result of the &lt;code&gt;SVR&lt;/code&gt; model, you have to add the mean to the result and multiply that by the standard deviation, if you have done that during normalizing.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How can one explain the impact of output normalization for &lt;code&gt;svm&lt;/code&gt; regression?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you normalize your data, you will have a cost function which is well behaved. means that you can find local parts more easily. The reason is that you have to construct the output using the input features in regression problems. It is difficult to make large values with small normalized features but with small numbers, making a normalized output is easier and can be learned faster. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-23T13:57:09.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26961" PostTypeId="2" ParentId="26789" CreationDate="2018-01-23T15:59:40.013" Score="2" Body="&lt;p&gt;In scale, unless you are expecting to receive only a particular format, it is machine learning. For the first task, you should first parse the text and then scan it, probably with a Named Entity Recognition (NER) system to extract the information you are after. Having a NER system would work, as you can manually code different types of features that will greatly improve the performance. If you just want to perform candidate matching, then standard bag-of-word approaches would perform decently. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second case, things are similar. You can rely in some syntactic analysis of the sentence to obtain the invitation and the time/day of the proposal. This again can be coupled with NER systems. Lately, for both tasks neural networks yielded promising approaches. But, in any case, you need labeled data, which can be cumbersome to obtain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, regex can be great ways to go with, especially if you can predict/adapt to the variability of the incoming data. In any case, they can be used to create your first training data. &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-23T15:59:40.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26962" PostTypeId="2" ParentId="26943" CreationDate="2018-01-23T16:09:31.697" Score="10" Body="&lt;p&gt;You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have the time, as I imagine its performance might be much better. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;1 Use existing corpus to learn embeddings for new words&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You can do this as you describe, though I would initialize the unknown embeddings with random noise instead of zeros. This will probably do fine, but it suffers from the problem that the model has no way of knowing that the SENNA embeddings are to be trusted more than the randomly initialized ones. So, it will generally take more (time, examples, etc.) to train this one well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option is to try to capture the difference between known embeddings and randomly initialized ones. My suggestion would be to create your own embeddings [20k x emb_dim] and initialize it however you want. Then add a penalty to your model's loss for deviation from the SENNA embeddings. This is what's done in bilingual embedding models like &lt;a href=&quot;https://ai.stanford.edu/~wzou/emnlp2013_ZouSocherCerManning.pdf&quot; rel=&quot;noreferrer&quot;&gt;this one&lt;/a&gt;. This will push the known embeddings close to SENNA and allow the unknown ones to freely vary. You could also reduce the coefficient for this alignment penalty as training progresses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could either learn these embeddings on a dedicated embedding task (like CBOW) or as part of your primary task. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;2 Hybrid model for unknown embeddings&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;A second approach is to forgo attempting to learn embeddings for your unknown words at all. Typically, mosty word RNN models do this for low-frequency words through something like the &lt;code&gt;&amp;lt;unk&amp;gt;&lt;/code&gt; tag. For you, that would be half your vocabulary. Fortunately, there are a lot of tools for filling in these out-of-vocabulary (OOV) words. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;2a Subword embeddings&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As @Emre pointed out, subword embeddings have been used to great effect to solve out-of-vocabulary problems. Here's &lt;a href=&quot;https://arxiv.org/pdf/1604.00788.pdf&quot; rel=&quot;noreferrer&quot;&gt;an example&lt;/a&gt; of a model that uses subword embeddings to achieve an &quot;open vocabulary&quot; model. This means you would combine your pre-trained embeddings (which you would not need to make learnable) with a character-level model to learn when and how to produce OOV words.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;2b &lt;code&gt;&amp;lt;unk&amp;gt;&lt;/code&gt; replacement&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Another approach is to, as a post-processing step, replace the &lt;code&gt;&amp;lt;unk&amp;gt;&lt;/code&gt; symbols with words from your corpus. You can build your own language model (even a simple markov chain would be an improvement). Also, &lt;a href=&quot;https://arxiv.org/pdf/1410.8206.pdf&quot; rel=&quot;noreferrer&quot;&gt;some modern models&lt;/a&gt; have more sophisticated methods for replacement that work quite well.&lt;/p&gt;&#xA;" OwnerUserId="38379" LastActivityDate="2018-01-23T16:09:31.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26963" PostTypeId="1" CreationDate="2018-01-23T16:10:18.973" Score="5" ViewCount="1996" Body="&lt;p&gt;How can I get the feature vector of my dataset. I have a fine-tuned CNN model with my data. Now I want to feed the features of all my dataset extracted from the last layer of the CNN into a LSTM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cnn_model = load_model('weights/pre-trained_CNN.hdf5')&#xA;cnn_output = cnn_model.get_layer('fc7').output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know I will have to feed all my dataset into this model but I have no idea on how to &quot;save&quot; the features.&lt;/p&gt;&#xA;" OwnerUserId="40217" LastActivityDate="2019-04-06T17:57:09.943" Title="Extract feature vector of a CNN" Tags="|keras|feature-extraction|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26964" PostTypeId="2" ParentId="26757" CreationDate="2018-01-23T16:37:29.837" Score="6" Body="&lt;p&gt;The Bayesian Network Generated (BNG) datasets are a set of artificially generated datasets openly available on OpenML. These datasets were generated to fill the need for a large heterogeneous set of large datasets. This paper describes the BNG generator best:&#xA;&lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-319-11812-3_28&quot; rel=&quot;noreferrer&quot;&gt;Algorithm Selection on Data Streams&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Small quote from the paper about the BNG data generator:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The generator takes a dataset as input, and outputs a data stream containing&#xA;  a similar concept, with a predefined number of instances. The input dataset is&#xA;  preprocessed with the following operations: all missing values are first replaced by the majority value of that attribute, and numeric attributes are discretized using Weka’s binning algorithm. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A personal note: For general Machine Learning studies, I would refrain from using BNG (or any other kind of artificially generated) datasets, as the concept is generally simpler than the original dataset. Instead, it is recommendable to use a per-defined benchmark suite, such as the &lt;a href=&quot;https://arxiv.org/abs/1708.03731&quot; rel=&quot;noreferrer&quot;&gt;OpenML-100&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45049" LastActivityDate="2018-01-23T16:37:29.837" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26965" PostTypeId="2" ParentId="26918" CreationDate="2018-01-23T16:45:13.130" Score="3" Body="&lt;p&gt;This is exactly your code just with &lt;em&gt;digits&lt;/em&gt; data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn.datasets import load_digits&#xA;from sklearn.svm import SVC&#xA;from sklearn.learning_curve import validation_curve&#xA;import psutil&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;&#xA;np.random.seed(0)&#xA;&#xA;# X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()&#xA;&#xA;&#xA;digits = load_digits()&#xA;X, y = digits.data, digits.target&#xA;&#xA;indices = np.arange(y.shape[0])&#xA;np.random.shuffle(indices)&#xA;X, y = X[indices], y[indices]&#xA;&#xA;param_range = np.arange(1, 41, 2)&#xA;train_scores, test_scores = validation_curve(&#xA;    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name=&quot;max_depth&quot;, cv=10, &#xA;    param_range=param_range,n_jobs=psutil.cpu_count(),&#xA;    scoring=&quot;accuracy&quot;)&#xA;&#xA;train_scores_mean = np.mean(train_scores, axis=1)&#xA;train_scores_std = np.std(train_scores, axis=1)&#xA;test_scores_mean = np.mean(test_scores, axis=1)&#xA;test_scores_std = np.std(test_scores, axis=1)&#xA;plt.figure(figsize=(10,10))&#xA;plt.title(&quot;Validation Curve with DecisionTree&quot;)&#xA;plt.xlabel(&quot;max_depth&quot;)&#xA;plt.ylabel(&quot;Score&quot;)&#xA;plt.ylim(0.0, 1.1)&#xA;#plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color=&quot;r&quot;)&#xA;plt.plot(param_range, train_scores_mean, label=&quot;Training score&quot;,&#xA;             color=&quot;r&quot;)&#xA;#plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color=&quot;g&quot;)&#xA;plt.plot(param_range, test_scores_mean, label=&quot;Cross-validation score&quot;,&#xA;             color=&quot;g&quot;)&#xA;&#xA;plt.legend(loc=&quot;best&quot;)&#xA;plt.xticks(param_range)&#xA;plt.savefig('DSSE.jpg')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is the result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iwGp1.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iwGp1.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So whatever the problem is, it's in your data that you didn't explain how you created it. Either double check it or post it here so your question can be answered. I assume somewhere in data preparation you made X and Y identical by mistake.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-01-23T16:45:13.130" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="26966" PostTypeId="2" ParentId="12607" CreationDate="2018-01-23T16:46:13.270" Score="0" Body="&lt;p&gt;I'm curious. Which Algorithm did you use. On sklearn it is said that Naive Bayes Algorithm perform on word occurrence. They add that MultinomialNB for instance MAY work with ratio figure like TDIDF feature but it seems its not guaranteed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently working on a similar problem than yours and using OneVsRestClassfiier wrapping classifier like SGDClassifier or LinearSVC, MultinomialNB gives a predicition score  of 0  with TFIDF features! much worse than the other classifier.&lt;/p&gt;&#xA;" OwnerUserId="45052" LastEditorUserId="40853" LastEditDate="2018-01-24T01:14:12.573" LastActivityDate="2018-01-24T01:14:12.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26967" PostTypeId="1" CreationDate="2018-01-23T17:00:20.927" Score="3" ViewCount="6483" Body="&lt;p&gt;I have already created an online chatbot for my Android app using Dialogflow (formerly api.ai). It allows users to add a new item to their inventory. I now need to create an offline version of this chatbot. However, Dialogflow does not allow you to download the agent to work offline.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked into several options, including RiveScript and ALICE (Pandorabots). These seem to be a bit too low-level for my purposes and require quite a bit of customization. I was wondering if there are any options similar to Dialogflow that can be used offline.&lt;/p&gt;&#xA;" OwnerUserId="45048" LastActivityDate="2020-08-17T11:13:15.830" Title="What are some options for an offline chatbot on Android?" Tags="|machine-learning|nlp|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26969" PostTypeId="1" CreationDate="2018-01-23T18:18:31.363" Score="5" ViewCount="19160" Body="&lt;p&gt;I understand the dimensionality of convolutions, max pooling and dense as function of stride and kernel size. But I'm having trouble wrapping my head around how to use these layers to end up with my final prediction, which should be one continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I'm working with:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;X training: n 2-channel 3D grids of size 46x46x46, shape: (n, 46, 46, 46, 2)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Y training: n-element vector of continuous values, shape: (n,)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would imagine there will be some resizing and some concatenation involved. But there's no point doing that if I don't actually understand what it's doing.&lt;/p&gt;&#xA;" OwnerUserId="45060" LastEditorUserId="28175" LastEditDate="2018-01-26T19:30:54.767" LastActivityDate="2018-06-26T04:21:39.107" Title="How to make a CNN predict a continuous value?" Tags="|machine-learning|neural-network|deep-learning|convolution|cnn|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26972" PostTypeId="1" AcceptedAnswerId="27008" CreationDate="2018-01-23T18:37:20.193" Score="5" ViewCount="6301" Body="&lt;p&gt;I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{array}&#xA;\\&#xA;p &amp;amp;= \frac{1}{1+e^{-x}}&#xA;\\&#xA;\hat{y} &amp;amp;= min(max(p, 10^{-7}, 1-10^{-7})&#xA;\\&#xA;FN &amp;amp;= y \times log(\hat{y})&#xA;\\&#xA;FP &amp;amp;= (1-y) \times log(1-\hat{y})&#xA;\\&#xA;Loss &amp;amp;= \frac{-1}{N}\sum_i 5 \times FN + FP&#xA;\end{array}&#xA;$$&#xA;I have calculated the gradient and hessian for this loss function:&#xA;$$&#xA;\begin{array}&#xA;\\&#xA;\frac{dLoss}{dx} &amp;amp;= 4py + p - 5y&#xA;\\&#xA;\frac{d^2Loss}{dx^2} &amp;amp;= (4y + 1) * p (1.0 - p)&#xA;\end{array}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And my code:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def logistic_obj(y_hat, dtrain):&#xA;    y = dtrain.get_label()&#xA;    p = 1.0 / (1.0 + np.exp(-y_hat))&#xA;    grad = 4 * p * y + p - 5 * y&#xA;    hess = (4 * y + 1) * (p * (1.0 - p))&#xA;    return grad, hess&#xA;&#xA;def err_rate(y_hat, dtrain):&#xA;    y = dtrain.get_label()&#xA;    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)&#xA;    loss_fn = y*np.log(y_hat)&#xA;    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)&#xA;    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)&#xA;&#xA;xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic', &#xA;      'max_depth': 6, 'tree_method': 'hist', 'seed': 42}&#xA;&#xA;model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'), &#xA;            (d_val, 'vld')], obj=logistic_obj, feval=err_rate)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as &lt;code&gt;feval&lt;/code&gt;. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the &lt;code&gt;objective&lt;/code&gt; directive, it apparently defaults to RMSE which makes matters worse.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0] trn-error:0.065108  vld-error:0.056749  trn-error:0.782048  vld-error:0.755389&#xA;[1] trn-error:0.064876  vld-error:0.056645  trn-error:0.727871  vld-error:0.695685&#xA;[2] trn-error:0.064487  vld-error:0.05651   trn-error:0.699920  vld-error:0.662203&#xA;[3] trn-error:0.064573  vld-error:0.056553  trn-error:0.691798  vld-error:0.64864&#xA;[4] trn-error:0.064484  vld-error:0.056514  trn-error:0.698498  vld-error:0.649974&#xA;[5] trn-error:0.064483  vld-error:0.056514  trn-error:0.716450  vld-error:0.662659&#xA;[6] trn-error:0.064470  vld-error:0.056507  trn-error:0.742848  vld-error:0.683847&#xA;[7] trn-error:0.064466  vld-error:0.056506  trn-error:0.775665  vld-error:0.71153&#xA;[8] trn-error:0.064435  vld-error:0.056497  trn-error:0.813440  vld-error:0.744165&#xA;[9] trn-error:0.064164  vld-error:0.056393  trn-error:0.854973  vld-error:0.780628&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45058" LastEditorUserId="45058" LastEditDate="2018-01-23T18:47:50.210" LastActivityDate="2018-02-05T01:12:11.857" Title="Cost-sensitive Logloss for XGBoost" Tags="|xgboost|loss-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26973" PostTypeId="2" ParentId="26969" CreationDate="2018-01-23T19:27:29.307" Score="1" Body="&lt;p&gt;What you need to do:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Ensure that your output vector for training and test data is exactly what you need, continuous for each element of output vector. &lt;/li&gt;&#xA;&lt;li&gt;Use what you said and familiar for the layers before the last layer.&lt;/li&gt;&#xA;&lt;li&gt;For the last layer use a dense layer with n, number of outputs, outputs each having linear activation, y = x.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-24T05:16:06.267" LastActivityDate="2018-01-24T05:16:06.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26974" PostTypeId="2" ParentId="26969" CreationDate="2018-01-23T19:29:43.987" Score="2" Body="&lt;p&gt;A common way to do this is flatten your output after your last convolution layer, and pass it through a fully connected layer. &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-23T19:29:43.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26978" PostTypeId="1" CreationDate="2018-01-23T22:11:06.913" Score="0" ViewCount="160" Body="&lt;p&gt;I have a temperature dataset (data every 15 mins) to build a supervised classification/prediction algorithm, but only know one of the true classes (when data is nearly flatlining around 35deg)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, given the academic literature on the subject and the data visualization it looks like there are three &quot;true&quot; classes: regulated, sub-ideal regulation, and no regulation. Here is a graph with vertical lines delineating 3 potential categories. &lt;a href=&quot;https://i.stack.imgur.com/djYnT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/djYnT.png&quot; alt=&quot;Graph&quot;&gt;&lt;/a&gt;&#xA;What is the best practice to build this classification? First attempt binned data every 6hrs and built categories based off stdev, but that seemed a bit arbitrary and did not align well with the three hypothesized categories.&lt;/p&gt;&#xA;" OwnerUserId="45071" LastEditorUserId="45071" LastEditDate="2018-01-24T14:35:44.430" LastActivityDate="2018-01-25T04:20:15.997" Title="Creating categorical variable, without knowing true categories (through binning time series data)?" Tags="|machine-learning|python|classification|time-series|supervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26979" PostTypeId="2" ParentId="26851" CreationDate="2018-01-23T23:31:56.053" Score="2" Body="&lt;p&gt;The key question is how we can allow the public to make useful queries to a dataset without revealing private information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The field of &lt;strong&gt;Differential Privacy&lt;/strong&gt; deals with answering just that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key concept is to track a user's queries and the information revealed and add uncertainty to the answers to guarantee privacy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a broad query like &lt;em&gt;&quot;what is the rate of cancer in Europe?&quot;&lt;/em&gt; could have an answer of with &quot;1% +/- .1%&quot;, whereas &lt;em&gt;&quot;what is the rate of cancer for the population of men aged 72 living at my neighbours's address?&quot;&lt;/em&gt; would give an answer of &quot;50% +/- 50%&quot;: we do not want to reveal whether a specific individual has cancer or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differential privacy gives users an information budget and tracks the queries, results and information revealed. The more you ask, or the closer you get to personal information the less specific the answer. At one point the information budget is exhausted and can query no more. You end up with nothing more than &quot;your neighbour has cancer with probability p=0.5 +/- 0.5&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also guards against constructing a clever sequence of queries to pin down the answer like in the children's game of &quot;20 questions&quot;. For example, you might try to ask questions like &lt;em&gt;&quot;what is the rate of men of cancer for men of age 72 in this city?&quot;&lt;/em&gt;, &lt;em&gt;&quot;what is the rate of cancer for men of height 182 cm at age 72?&quot;&lt;/em&gt;, &lt;em&gt;&quot;what is the height distribution of men with my cancer in my street?&quot;&lt;/em&gt; &lt;em&gt;etc.&lt;/em&gt; until you can deduce whether your 182 cm tall, 72 year old, male neighbour has cancer or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differential Privacy guarantees that such strategies will never work (under certain assumptions).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Moving Averages&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With this background, your example of only publishing the moving average seems quite reasonable. Differential Privacy can be used to calculate how long or short the moving average needs to be on your dataset in order to provide the best possible information while preserving privacy.&lt;/p&gt;&#xA;" OwnerUserId="1424" LastActivityDate="2018-01-23T23:31:56.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26980" PostTypeId="1" CreationDate="2018-01-24T00:52:02.337" Score="18" ViewCount="1906" Body="&lt;p&gt;I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature in an observation by a very small amount can make the probability output jump from 0.5 to 0.99.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I barely see outputs in the 0.6-0.8 range. In all cases, the probability is less than 0.99 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of post training calibration methods such as Platt Scaling and Logistic Correction, but I was wondering if there is anything I can tweak in the XGBoost training process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I call XGBoost from different languages using FFI, so it would be nice if I can fix this issue without introducing other calibration libraries, e.g., changing eval metric from AUC to log loss.&lt;/p&gt;&#xA;" OwnerUserId="45076" LastEditorUserId="85045" LastEditDate="2019-11-14T21:15:15.377" LastActivityDate="2020-06-18T10:29:08.153" Title="XGBoost outputs tend towards the extremes" Tags="|machine-learning|classification|xgboost|probability|probability-calibration|" AnswerCount="4" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="26985" PostTypeId="2" ParentId="26789" CreationDate="2018-01-24T06:43:55.200" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;You receive an email from a friend that says, &quot;let's have lunch next Tuesday&quot; and your email program detects it and asks if you want to save a new calendar entry for &quot;lunch on Tuesday&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What you describe is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_extraction&quot; rel=&quot;nofollow noreferrer&quot;&gt;Information extraction&lt;/a&gt; and is a big field of NLP (Natural Language Processing). You are looking for &lt;em&gt;temporal expression identification&lt;/em&gt;. You can have a look at the &lt;a href=&quot;http://nlp.stanford.edu:8080/sutime/process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Temporal Tagger: SUTime&lt;/a&gt; to get a &quot;live&quot; demo. From what I see &lt;a href=&quot;https://nlp.stanford.edu/software/sutime.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; it is a regex-based rule system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an impression how powerful rule-based systems can be: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Weizenbaum’s own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing, “I had not realized… that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Source: &lt;a href=&quot;https://en.wikipedia.org/wiki/ELIZA&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia: Eliza&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;See also&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.miramontes.com/writing/add-cacm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apple Data Detectors&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.aaai.org/Papers/Symposia/Spring/2003/SS-03-07/SS03-07-005.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TimeML: Robust specification of event and temporal expressions in text&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-01-24T06:43:55.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26986" PostTypeId="2" ParentId="26978" CreationDate="2018-01-24T07:07:21.240" Score="2" Body="&lt;p&gt;So, if I understood you correctly, you want to classify your data based on the frequency and amplitude of the temperature change, right? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think your attempt with stdev is not so far away, but your interval is a little off, 6 hours might be to much. Think about Shannon, your sample rate must be at least twice the highest frequency in your dataset, if you want to reconstruct your signal completely. If your sample rate is 6 hours you might miss the big leaps at the end of the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation would be to use a sliding window and use Fast Fourier Transformation to get to the frequency spectrum. With those datapoints you can try to cluster them with SVM or kNN approaches. I think SVM is the way to go, since your class should be cleanly seperateble. And since you know how many classes you have there is no need for hypertuning this parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I certainly don't know if this is best practice, but I think this would be a feasible approach.&lt;/p&gt;&#xA;" OwnerUserId="45086" LastEditorUserId="29575" LastEditDate="2018-01-24T07:18:10.003" LastActivityDate="2018-01-24T07:18:10.003" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="26987" PostTypeId="2" ParentId="26958" CreationDate="2018-01-24T08:57:41.767" Score="4" Body="&lt;p&gt;What you are facing is a small but crucial definition difference:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;novelty detection:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;The training data is &lt;strong&gt;not polluted&lt;/strong&gt; by outliers, and we are interested in detecting anomalies in new observations.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;outlier detection:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;The training data &lt;em&gt;&lt;strong&gt;contains&lt;/strong&gt;&lt;/em&gt; outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.&lt;/p&gt;&#xA;&lt;p&gt;OneClassSVM is an Unsupervised Outlier Detection. Therefor your data needs to have outliers in order for the algortihm to detect them. My best guess, why its prediction every input as an outlier is, that if there are no real outliers, everything must be an outlier.&lt;/p&gt;&#xA;&lt;p&gt;Let me demonstrate this quickly. I adjustet the kernel to linear&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn import svm&#xA;import numpy as np&#xA;&#xA;nu = 0.5&#xA;gamma = 1.0&#xA;ones= pd.DataFrame(np.ones(100))&#xA;&#xA;clf = svm.OneClassSVM(nu=nu, kernel=&amp;quot;linear&amp;quot;, gamma=gamma)&#xA;clf.fit(ones)&#xA;clf.predict(-1) # -1&#xA;clf.predict(1) # -1&#xA;clf.predict(1.00001) # 1&#xA;clf.predict(2) # 1&#xA;clf.predict(10) # 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45086" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-01-24T08:57:41.767" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26989" PostTypeId="2" ParentId="23297" CreationDate="2018-01-24T09:53:34.407" Score="2" Body="&lt;p&gt;The deconv layers are probably to blame. Check out this &lt;a href=&quot;https://distill.pub/2016/deconv-checkerboard/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;distill&lt;/em&gt; article&lt;/a&gt; for a fairly in depth discussion about how deconv layers create checkerboard artifacts. The gist is that deconv striding creates interference patterns which can cancel out if you're careful, but are more likely to be worsened as you add more deconv layers. The authors suggest using nearest neighbors for upsampling followed by a convolutional layer, which they call &lt;em&gt;resize-convolution&lt;/em&gt; upsampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/45NtS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/45NtS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-24T09:53:34.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26991" PostTypeId="2" ParentId="25090" CreationDate="2018-01-24T10:07:05.757" Score="1" Body="&lt;p&gt;Layout algorithms in tools like networkx, igraph, and gephi will associate coordinates with your nodes which you should be able to access fairly easily. Once you have those coordinates, you just need to plot your pie-charts on top of the relevant node location. Alternative, these tools also support using external images as node markers, so instead of building the plots in the same script you could build the pie charts separately, save them to disk, and then associate them with nodes when you draw the graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've never seen an &quot;out-of-the-box&quot; solution for this specific kind of graphic, but it shouldn't be too hard to do this yourself. You just need to figure out how to access the layout coordinates. If you clarify what your preferred analytic environment is and/or graph analysis tool, I can give you more specific advice.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT: I managed to find the code that was used to build the chart in that paper. I searched the paper for &quot;we used&quot; and found this in the acknowledgements:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We are especially indebted to Aaron Clauset and James Fowler for thorough readings of a draft of this manuscript &lt;strong&gt;and to Christina Frost for developing some of the graph visualizations we used.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Searching &quot;Christina Frost UNC&quot; led me to &lt;a href=&quot;http://netwiki.amath.unc.edu/VisComms/VisComms&quot; rel=&quot;nofollow noreferrer&quot;&gt;this page&lt;/a&gt; which contains a collection of graph visualization tools for matlab. The one you are looking for is at the bottom: &lt;a href=&quot;http://netwiki.amath.unc.edu/uploads/VisComms/drawForceCPie.m&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;drawForceCPie.m&lt;/code&gt;&lt;/a&gt;. The site is super slow, but it eventually shared the code with me. Here it is for posterity in case the site crashes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function drawForceCPie(A,XY,scores,gn)&#xA;&#xA;gnu=unique(gn);&#xA;CAM=commAdjMatrixSparse(gn,A);&#xA;map=colormap;&#xA;if min(scores)==0&#xA;    map=[.7 .7 .7; map];&#xA;end&#xA;colorsu=unique(scores);&#xA;% colorsu(2:end)=colorsu(randperm(length(colorsu)-1)+1);&#xA;% scores2=zeros(size(scores));&#xA;% for i=1:length(colorsu);&#xA;%      scores2(find(scores==colorsu(i)))=i;&#xA;% end&#xA;scores2=scores;&#xA;nodes=length(scores2);&#xA;C=length(map);&#xA;colorsu=unique(scores2);&#xA;R=colorsu-min(min(colorsu))+1e-10;&#xA;Rcolor=C*R/max(max(R));&#xA;idcolors = map(ceil(Rcolor),:);&#xA;edges=find(CAM);&#xA;We=[CAM(edges),edges];&#xA;sortWe=sortrows(We);&#xA;&#xA;hold on&#xA;&#xA;alpha=2;&#xA;x=XY(:,1);&#xA;y=XY(:,2);&#xA;str=(CAM/max(max(CAM))).^alpha;&#xA;N=length(CAM);&#xA;&#xA;% for i=1:length(colorsu),&#xA;%     if colorsu(i)==1&#xA;%         h=plot(XY(1,1),XY(1,2),'o','markersize',10);&#xA;%         set(h,'Color','k');&#xA;%     else&#xA;%         h=plot(XY(1,1),XY(1,2),'.','markersize',25);&#xA;%         set(h,'Color',idcolors(i,:));&#xA;%     end&#xA;% end&#xA;&#xA;for ie=sortWe(:,2)',&#xA;    i=mod(ie-1,N)+1;&#xA;    j=floor((ie-1)/N)+1;&#xA;    if (j&amp;gt;i)&#xA;        h=plot(x([i,j]),y([i,j]),'k-'); &#xA;%        set(h,'linewidth',str(i,j))&#xA;        set(h,'color',[.5 .5 .5]*(1-str(i,j)));&#xA;    end&#xA;end&#xA;for i=1:length(gnu)&#xA;     nodes_percom = length(find(gnu(i)==gn));&#xA;     idx=find(gnu(i)==gn);&#xA;     radius=15*(((nodes_percom)^(.25))*(pi/sqrt(nodes*length(gnu))));&#xA;    comcolors=scores2(idx);&#xA;    comcolorsu=unique(comcolors);&#xA;     for j=1:length(colorsu)&#xA;         percents(j)=length(find(comcolors==colorsu(j)))/length(idx);&#xA;     end&#xA;     drawpie(percents,XY(i,:),radius,idcolors);&#xA;end&#xA;&#xA;hold off&#xA;&#xA;end&#xA;function drawpie(percents,pos,radius,colors)&#xA;&#xA;&#xA;&#xA;points = 40;&#xA;x = pos(1);&#xA;y = pos(2);&#xA;last_t = 0;&#xA;if (length(find(percents))&amp;gt;1)&#xA;    for i = 1:length(percents)&#xA;        end_t = last_t + percents(i)*points;&#xA;        tlist = [last_t ceil(last_t):floor(end_t) end_t];&#xA;        xlist = [0 (radius*cos(tlist*2*pi/points)) 0] + x;&#xA;        ylist = [0 (radius*sin(tlist*2*pi/points)) 0] + y;&#xA;        patch(xlist,ylist,colors(i,:))&#xA;        last_t = end_t;&#xA;    end&#xA;else&#xA;    i=find(percents);&#xA;    tlist = [0:points];&#xA;    xlist = x+radius*cos(tlist*2*pi/points);&#xA;    ylist = y+radius*sin(tlist*2*pi/points);&#xA;    patch(xlist,ylist,colors(i,:))&#xA;end&#xA;end&#xA;&#xA;function mat = commAdjMatrixSparse(groups, A)&#xA;% Creates a community adjacency matrix using&#xA;% groups from the output for reccurrcommsNew2Sparse, A is the adjacency matrix&#xA;% 0's on the diagonal, other elements consist of the total number of&#xA;% connections between the two communities&#xA;%&#xA;% Last Modified by ALT 20 June 2007&#xA;&#xA;&#xA;h=sort(groups);&#xA;g=unique(h);&#xA;d=diff(g);&#xA;f=sort(d);&#xA;z=unique(f);&#xA;cuts=size(z,2);&#xA;&#xA;[communities cut]=findcommunitiesatcut(groups,cuts);&#xA;rows = max(communities);&#xA;mat=spalloc(rows,rows,2*rows);&#xA;&#xA;for i = 1:rows&#xA;    for j = 1:rows&#xA;        if(i ~= j)&#xA;            comm1 = find(communities==i);&#xA;            comm2 = find(communities==j);&#xA;            %comm1=comm1(find(comm1));&#xA;            %comm2=comm2(find(comm2));&#xA;            mat(j, i) = sum(sum(A(comm1, comm2)));&#xA;        end&#xA;    end&#xA;end&#xA;end&#xA;&#xA;function [communities cut] = findcommunitiesatcut(groups,cut)&#xA;%[communities cut]=findcommunitiesatcut(groups,cut)&#xA;%&#xA;% Gives the community numbers at a requested cut or level in the groups vector,&#xA;% if the cut number is not valid the program changes it to a valid one.&#xA;% Uses a groups vector and a scalar cut number, gives communities and the cut number,&#xA;% which is needed when cut is changed.&#xA;%&#xA;%&#xA;%Last modified by ALT, 20 June 2007&#xA;&#xA;%Error checking&#xA;n=unique(groups);&#xA;f=diff(n);&#xA;z=unique(f);&#xA;cutmax=length(z);&#xA;if(cut&amp;gt;cutmax)&#xA;    disp(['That is too many cuts! I have changed the cut number.']);&#xA;    cut=cutmax;&#xA;elseif(cut&amp;lt;0)&#xA;    cut=cutmax;&#xA;    disp(['Negative numbers dont work, I have changed the cut number to the max'])&#xA;end&#xA;&#xA;%Identify distinct group values and number of cut levels in dendrogram:&#xA;groupnumbers=unique(groups);&#xA;differences=diff(groupnumbers);&#xA;diffnumbers=unique(differences);&#xA;cuts=length(diffnumbers);&#xA;&#xA;if cut==0,&#xA;    communities=ones(size(groups));&#xA;else&#xA;    cutdiff=diffnumbers(cuts+1-cut); %NOTE THERE IS NO ERROR CHECKING HERE, ASSUMED VALID CUT NUMBER&#xA;&#xA;    commnumbers=cumsum([1,diff(groupnumbers)&amp;gt;=cutdiff]);&#xA;&#xA;    %Define communities by replacing the groupnumbers values in groups with the&#xA;    %corresponding commnumbers values, component by component.&#xA;    %Is there a more efficient way to specify this in MATLAB?&#xA;    communities=groups;&#xA;    for ig=1:length(groupnumbers),&#xA;        indx=find(groups==groupnumbers(ig));&#xA;        communities(indx)=commnumbers(ig);&#xA;    end&#xA;end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you use this code for research, I believe this is the citation you should reference (in addition to the UNC webpage that hosted the code):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Visualization of communities in networks,&quot; Amanda L. Traud, Christina Frost, Peter J. Mucha, and Mason A. Porter, Chaos 19, 041104 (2009).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-01-24T10:27:50.863" LastActivityDate="2018-01-24T10:27:50.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26992" PostTypeId="1" CreationDate="2018-01-24T10:14:42.490" Score="2" ViewCount="1168" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/viyQ1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/viyQ1.png&quot; alt=&quot;Sample FFT Data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As shown in the picture, I have some samples of FFT data, out of which some are from the system which are working good, and some from systems which has faults. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried calculating PeakValue, RMS and tried to train the network using DNN as well as Linear methods, but in vain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, Do we have any mechanishm so that we can feed the raw data (without pre processing) and train the system, if not, what pre-processing steps should be taken care and what model of neural network should I deploy ?&lt;/p&gt;&#xA;" OwnerUserId="45096" LastEditorUserId="8878" LastEditDate="2018-01-24T12:42:57.477" LastActivityDate="2018-01-24T12:42:57.477" Title="How do I feed a FFT data as input to a CNN/DNN?" Tags="|machine-learning|neural-network|deep-learning|tensorflow|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="26996" PostTypeId="1" CreationDate="2018-01-24T12:42:29.620" Score="1" ViewCount="715" Body="&lt;p&gt;I am trying to build an Autoencoder with LSTMs. My input data has been one hot encoded, which results in around 13.000 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem at this point is, that I get the following error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;File &quot;/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py&quot;, line 926, in internal_convert_to_tensor&#xA;ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)&#xA;File &quot;/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py&quot;, line 229, in _constant_tensor_conversion_function&#xA;return constant(v, dtype=dtype, name=name)&#xA;File &quot;/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py&quot;, line 208, in constant&#xA;value, dtype=dtype, shape=shape, verify_shape=verify_shape))&#xA;File &quot;/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py&quot;, line 447, in make_tensor_proto&#xA;&quot;Cannot create a tensor proto whose content is larger than 2GB.&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;ValueError: Cannot create a tensor proto whose content is larger than 2GB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I reduce the size of the tensor without secraficing the deepth of the model respectively without rewriting the code in tensor myself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using tesnorflow as my backend. My model looks as following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# current values for timestamps and features: 1 and 12113&#xA;def build_model(timestamps, features):&#xA;    model = Sequential()&#xA;    layers = {'hidden1': 64, 'hidden2': 16, 'output': features}&#xA;&#xA;    model.add(LSTM(&#xA;            batch_input_shape=(batch_size, timestamps, features), #batch_size is set to 500&#xA;            units=layers['hidden1'],&#xA;            return_sequences=True,&#xA;            stateful=True))&#xA;&#xA;    model.add(LSTM(&#xA;            units=layers['hidden2'],&#xA;            return_sequences=True,&#xA;            stateful=True))&#xA;&#xA;    model.add(LSTM(&#xA;            units=layers['output'],&#xA;            return_sequences=False,&#xA;            stateful=True))&#xA;&#xA;    model.compile(loss=&quot;binary_crossentropy&quot;,&#xA;                    optimizer=&quot;rmsprop&quot;, # is recommended vor RNNs)&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45086" LastActivityDate="2018-01-24T12:42:29.620" Title="Using keras LSTM implementation with sparse data" Tags="|machine-learning|tensorflow|keras|rnn|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="26997" PostTypeId="1" CreationDate="2018-01-24T13:04:39.890" Score="0" ViewCount="274" Body="&lt;p&gt;I have some unlabeled data with 15 attributes and 30161 instances. I took 70% data for clustering and cluster those data into 2 cluster. I also added one column after clustering named 'cluster' and the column '&amp;lt;=50k' was label column. Now I need to plot those data except label column, How can I plot it ? &#xA;Here's the sample of my code and data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Zq1r0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Zq1r0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;cols_x=train.columns[0:14]&#xA;cols_y = train.columns[14]&#xA;X = train[cols_x]&#xA;y = train[cols_y]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;kmeans = KMeans(n_clusters=2) &#xA;train[&quot;cluster&quot;] = kmeans.fit_predict(train[X_train.columns[0:14]])&#xA;train.head(10)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/g0fIi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/g0fIi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="43455" LastEditDate="2018-01-29T13:17:49.183" LastActivityDate="2018-02-28T13:51:34.213" Title="How can I plot data after cluster it into two cluster?" Tags="|machine-learning|python|scikit-learn|clustering|plotting|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26998" PostTypeId="1" CreationDate="2018-01-24T13:46:47.283" Score="2" ViewCount="1470" Body="&lt;p&gt;I am attempting to solve quadratic equations using machine learning. I tried to write &lt;a href=&quot;https://stackoverflow.com/questions/48401060/mxnet-regression-divide-by-zero-error&quot;&gt;code using mxnet&lt;/a&gt; but fails because it is using linear regression while my problem is non-linear (as far as I know).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to predict the solution for arbitrary equations provided the at least one real root exists?&lt;/p&gt;&#xA;" OwnerUserId="45107" LastActivityDate="2019-03-18T18:38:48.643" Title="solving quadratic equations using machine learning?" Tags="|machine-learning|regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="26999" PostTypeId="2" ParentId="26997" CreationDate="2018-01-24T13:52:04.840" Score="0" Body="&lt;p&gt;maybe the first step could be to try to figure out how you could distinguish visually your two grouped elements (using colors maybe ?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully you'll find some help here :&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/31083/how-to-produce-a-pretty-plot-of-the-results-of-k-means-cluster-analysis&quot;&gt;https://stats.stackexchange.com/questions/31083/how-to-produce-a-pretty-plot-of-the-results-of-k-means-cluster-analysis&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;good luck !&lt;/p&gt;&#xA;" OwnerUserId="42831" LastActivityDate="2018-01-24T13:52:04.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27000" PostTypeId="2" ParentId="17540" CreationDate="2018-01-24T14:07:53.377" Score="0" Body="&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(20,15))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;plt&lt;/code&gt; is not always defined, I can use seaborn without &lt;code&gt;plt&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use the above line you need to also import plt like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import plt&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45112" LastEditorUserId="29575" LastEditDate="2018-01-24T14:34:05.777" LastActivityDate="2018-01-24T14:34:05.777" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27003" PostTypeId="1" AcceptedAnswerId="65123" CreationDate="2018-01-24T15:50:51.917" Score="12" ViewCount="7472" Body="&lt;p&gt;I am working with a dataset with large number of categorical features (&gt;80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical features. And learned that one-hot encoding I have been using in past is really bad idea especially when it comes to lots of categorical features with many levels (read these &lt;a href=&quot;https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/&quot; rel=&quot;noreferrer&quot;&gt;post&lt;/a&gt;, and &lt;a href=&quot;https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-66041f734512&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I've come across methods like target-based encoding (smoothing) of categorical features often based on mean of target values for each feature perhaps this &lt;a href=&quot;https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features&quot; rel=&quot;noreferrer&quot;&gt;post/kernel&lt;/a&gt; in Kaggle. Still I am struggling to find a more concrete way till I found &lt;a href=&quot;https://tech.yandex.com/catboost/&quot; rel=&quot;noreferrer&quot;&gt;CatBoost&lt;/a&gt; an open-source gradient boosting on decision trees released last year by Yandex group. They seem to offer extra statistical counting options for categorical features likely much more efficient than simple one-hot encoding or smoothing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt; is the documentation is not helpful how to set &lt;strong&gt;&lt;em&gt;CTR settings&lt;/em&gt;&lt;/strong&gt;. I have tried different ways but it just does not work. The doc says &lt;em&gt;CTR setting&lt;/em&gt; as &lt;strong&gt;&lt;em&gt;simple_ctr&lt;/em&gt;&lt;/strong&gt;, to be given as (&lt;a href=&quot;https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_parameters-list-docpage/&quot; rel=&quot;noreferrer&quot;&gt;CTR setting section&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',&#xA; 'CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',&#xA;  ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a super simple example, the data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import catboost&#xA;data = [{'profit': '342','country': 'holland','account': 'Jones LLC', 'saving': 150, 'debt': -60, 'age': 28},&#xA;         {'profit': '875','country': 'germany','account': 'Alpha Co',  'saving': 200, 'debt': -10, 'age': 42},&#xA;         {'profit': '127','country': 'italy','account': 'Blue Inc',  'saving': 50,  'debt': -300,  'age': 38 }]&#xA;df = pd.DataFrame(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a simple Catboost Regressor:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train = df.drop(['profit'],axis=1)&#xA;Y_train = df['profit']&#xA;categorical_features_indices = [0,2]&#xA;&#xA;train_pool = catboost.Pool(X_train, Y_train, cat_features=categorical_features_indices)&#xA;&#xA;model = catboost.CatBoostRegressor(&#xA;    depth=3,&#xA;    iterations=5,&#xA;    eval_metric='RMSE',&#xA;    simple_ctr=None)&#xA;&#xA;model.fit(train_pool);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;simple_ctr&lt;/strong&gt;, one of CTR settings, is the problem!! It is pity because it looks like it the package offers various methods, so far no way to access them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE Aug. 9th, 2018:&lt;/strong&gt; A few days ago I raised this problem to Catboost developer, see &lt;a href=&quot;https://github.com/catboost/catboost/issues/436&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, and they opened a ticket for it to provide a tutorial.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-08-09T21:18:31.150" LastActivityDate="2020-01-20T10:01:40.180" Title="Catboost Categorical Features Handling Options (CTR settings)?" Tags="|machine-learning|decision-trees|categorical-data|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27004" PostTypeId="2" ParentId="17540" CreationDate="2018-01-24T16:05:37.257" Score="0" Body="&lt;p&gt;The basic idea is to increase the default figure size in your plotting tool. You need to import matplotlib and set either default figure size or just the current figure size to a bigger one. Also, seaborn is built on top of matplotlib. You need to install and import matplitlib to make the best use of seaborn library. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-01-24T16:05:37.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27005" PostTypeId="2" ParentId="26367" CreationDate="2018-01-24T17:09:39.623" Score="2" Body="&lt;p&gt;To build a model to detect the old categories and your new ones, you need to re-train the model with your own dataset and the dataset used to pre-train the model. Fortunately, this dataset is available to the public for downloading. The dataset used to train the model is written in the name of model you choose. It's probably the &lt;a href=&quot;http://cocodataset.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;COCO dataset&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the model as been pre-trained to detect the old categories, I think you don't need to download all the dataset. The data is just necessary to avoid the model to &quot;forget&quot; what it has learned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope you understood me, and that I have answered well your question.&lt;/p&gt;&#xA;" OwnerUserId="45123" LastActivityDate="2018-01-24T17:09:39.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27006" PostTypeId="2" ParentId="11129" CreationDate="2018-01-24T17:37:25.540" Score="1" Body="&lt;p&gt;I would seriously consider using the bsts package (in R), with 'logistic' as the model family.  That will give you a forecast of the probability of 1's and 0's, based on past trends or periodicities (depending on how you construct the model). You can also add covariates, i.e. other things that change in a way related to your 1/0 observations, to act as predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See Figures 9 and 10 on this page:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.unofficialgoogledatascience.com/2017/07/fitting-bayesian-structural-time-series.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.unofficialgoogledatascience.com/2017/07/fitting-bayesian-structural-time-series.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-01-24T17:37:25.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27007" PostTypeId="2" ParentId="10188" CreationDate="2018-01-24T18:57:16.770" Score="5" Body="&lt;p&gt;In addition to the key points made by others, using squared error puts a greater emphasis on larger error (what happens to 1/2 when you square it vs 3/2?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having an algorithm that moves the fractional errors, that would likely result in correct classification or very small difference between estimate and ground truth, if left alone close to zero, while leaving the large errors as large errors or misclassifications, is not a desirable characteristic of an algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using squared error uses the error as an implied importance weight for adjusting prediction.&lt;/p&gt;&#xA;" OwnerUserId="45129" LastEditorUserId="29575" LastEditDate="2018-01-24T19:31:56.977" LastActivityDate="2018-01-24T19:31:56.977" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27008" PostTypeId="2" ParentId="26972" CreationDate="2018-01-24T19:27:17.850" Score="3" Body="&lt;p&gt;I'm answering my question. Well, I've ditched XGBoost for LightGBM (the only Microsoft product I seem to enjoy by far) but since the interface is quite similar, the same should apply to XGBoost. Apparently I don't need to apply Sigmoid to predictions. I don't know why the examples suggest otherwise.&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def logistic_obj(y_hat, dtrain):&#xA;    y = dtrain.get_label()&#xA;    p = y_hat # p = 1. / (1. + np.exp(-y_hat))&#xA;    grad = p - y&#xA;    hess = p * (1. - p)&#xA;    grad = 4 * p * y + p - 5 * y&#xA;    hess = (4 * y + 1) * (p * (1.0 - p))&#xA;    return grad, hess&#xA;&#xA;def err_rate(y_hat, dtrain):&#xA;    y = dtrain.get_label()&#xA;    # y_hat = 1.0 / (1.0 + np.exp(-y_hat))&#xA;    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)&#xA;    loss_fn = y*np.log(y_hat)&#xA;    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)&#xA;    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y), False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45058" LastActivityDate="2018-01-24T19:27:17.850" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27009" PostTypeId="1" AcceptedAnswerId="27011" CreationDate="2018-01-24T20:01:23.673" Score="7" ViewCount="267" Body="&lt;p&gt;For a task on sentiment analysis, suppose we have some classes represented by $c$ and features $i$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can represent the conditional probability of each class as: $$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)}$$&#xA;where $w_i$ represents each feature and $c$ is the class we have. Then empirically, we can represent $$P(w_i|c) = \frac{n_{ci}}{n_c}$$ &#xA;$$P(w_i) = \frac{n_{i}}{n}$$&#xA;Our priors for each classes are then given by: $$P(c) = \frac{n_c}{n}$$ where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n$ is the total number of features in all classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_{ci}$ represents the number of counts of that feature $i$ in class $c$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_c$ is the total number of features for the class, and &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$n_i$ is the total number of features for all classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my understanding of the above correct? So given these $P(c|w_i)$ probabilities for each word, I'm the naive bayes assumption is that the words are independent, so I simply multiply each word in a document for a certain class, i.e. to compute $\prod P(c|w_i), i \in N$ where $N$ is the number of words in the document. Is this correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To actually compute the conditional probability numerically, would it suffice to do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)} = \frac{n_{ci}}{n_c} \cdot \frac{n_c}{n}\cdot \frac{n}{n_i} = \frac{n_{ci}}{n_i}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability. &lt;/p&gt;&#xA;" OwnerUserId="45133" LastEditorUserId="23305" LastEditDate="2018-10-23T18:48:03.260" LastActivityDate="2018-10-23T18:48:03.260" Title="Understanding of naive bayes: computing the conditional probabilities" Tags="|machine-learning|naive-bayes-classifier|sentiment-analysis|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27011" PostTypeId="2" ParentId="27009" CreationDate="2018-01-24T21:07:52.803" Score="4" Body="&lt;p&gt;Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the second equation holds because of the naïve Bayes assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For classification purposes you can ignore $P(w_1,\ldots,w_N)$ because it is constant (given the data). The formula is still simple (&quot;naïve&quot;) but doesn't simplify quite as much. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that while Naïve Bayes is a decent classifier for many applications, the generated probabilities are usually not very representative.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-01-24T21:26:14.380" LastActivityDate="2018-01-24T21:26:14.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27012" PostTypeId="1" AcceptedAnswerId="27014" CreationDate="2018-01-24T21:45:27.283" Score="4" ViewCount="3907" Body="&lt;p&gt;What is the advantage of converting a series like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df&#xA;   Color&#xA;0    Red&#xA;1   Blue&#xA;2  Green&#xA;3    Red&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To a multiple series like the below? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; pd.get_dummies(df)&#xA;   Color_Blue  Color_Green  Color_Red&#xA;0           0            0          1&#xA;1           1            0          0&#xA;2           0            1          0&#xA;3           0            0          1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;One could as well have a hot encoded values for the Color Column as below?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; labels=list(set(df.Color))&#xA;&amp;gt;&amp;gt;&amp;gt; pd.DataFrame(df.Color.map({x:labels.index(x) for x in labels}).rename('Color_Code'))&#xA;   Color_Code&#xA;0           1&#xA;1           2&#xA;2           0&#xA;3           1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know syntactically &lt;code&gt;pd.get_dummies&lt;/code&gt; looks much simpler, but somehow I want to lean towards a lesser number of features than more number of features...&lt;/p&gt;&#xA;" OwnerUserId="45136" LastActivityDate="2018-01-24T23:54:56.043" Title="Cleaning input data with pd.get_dummies()" Tags="|feature-selection|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27014" PostTypeId="2" ParentId="27012" CreationDate="2018-01-24T22:12:34.537" Score="4" Body="&lt;p&gt;Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this &lt;a href=&quot;https://blog.myyellowroad.com/using-categorical-data-in-machine-learning-with-python-from-dummy-variables-to-deep-category-66041f734512&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; and this &lt;a href=&quot;https://www.kdnuggets.com/2015/12/beyond-one-hot-exploration-categorical-variables.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;one&lt;/a&gt; to learn what are the advantages and disadvantages of choosing each. The first method you showed is called one-hot encoding that you can get easily by pd.get_dummies as you mentioned, and frequently used by people in algorithms like XGBoost. One of the biggest downfall of the first method OHE, is increasing often unnecessarily the dimensionality of your feature space. Other cons could be not being a good representative of feature space as well as problem of missing features in test data. The second method is LabelEncoding, usually used in target variables or when you have very few categorical features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem surfaces when you start having a large number of categorical features (&gt;50 % of feature space) with each feature having many sub levels. Here you need more clever way of feature encoding like Target-based mean encoding (smoothing, you can find some kernels in Kaggle like this). If you are Python user, I have found a fairly new &lt;a href=&quot;https://github.com/scikit-learn-contrib/categorical-encoding&quot; rel=&quot;nofollow noreferrer&quot;&gt;package&lt;/a&gt; under scikit-learn-contrib offering a wide range of Categorical Encoding Methods. This &lt;a href=&quot;https://www.kdnuggets.com/2015/12/beyond-one-hot-exploration-categorical-variables.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kdnuggets post&lt;/a&gt; also compares some of the aspect of using different methods. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="29575" LastEditDate="2018-01-24T23:54:56.043" LastActivityDate="2018-01-24T23:54:56.043" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27016" PostTypeId="1" CreationDate="2018-01-25T01:27:00.413" Score="11" ViewCount="8265" Body="&lt;p&gt;While I was reading the paper &amp;quot;&lt;a href=&quot;https://www.cs.utexas.edu/%7Epstone/Papers/bib2html/b2hd-AAAI17-Hanna.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;Grounded Action Transformation for Robot Learning in Simulation&lt;/em&gt;&lt;/a&gt;&amp;quot;, I came across the term &amp;quot;policy&amp;quot;. Could someone explain to me what that actually is (in general and in the particular context of the paper)?&lt;/p&gt;&#xA;" OwnerUserId="45147" LastEditorUserId="70066" LastEditDate="2020-08-05T02:19:42.973" LastActivityDate="2020-08-05T02:19:42.973" Title="What is a policy in machine learning?" Tags="|machine-learning|reinforcement-learning|beginner|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27018" PostTypeId="2" ParentId="27016" CreationDate="2018-01-25T02:46:02.867" Score="9" Body="&lt;p&gt;A policy is a state-action mapping.  A 'state' is a formalism used in AI that represents the state of the world, i.e. what the agent's idea of the world is.  The action is, naturally, what action it should take in that state.  A policy just maps states to actions.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the basic problems in AI is how to maximize reward over time at some task. &#xA; One strategy for the agent is to try to understand the system and predict the results of their actions, and the reward that would follow.  Another strategy is for the agent to try lots of things and record the results.  Either of these (eventually) allow the agent to calculate a good policy - and once it's calculated, the difficult computational work is done and the agent just has to 'look up' what action to take in each state.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-01-25T02:46:02.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27019" PostTypeId="1" CreationDate="2018-01-25T02:52:58.807" Score="0" ViewCount="81" Body="&lt;p&gt;I am a recent college graduate with a bachelors in computer engineering. I am very interested in deep learning however someone had mentioned to me that I should learn machine learning before I learn deep learning. Is that true ? And if so which online course would you recommend I start at ? &lt;/p&gt;&#xA;" OwnerUserId="45151" LastActivityDate="2018-01-25T06:57:18.573" Title="Learning AI, Machine Learning, Deep Learning" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ClosedDate="2018-01-25T07:12:22.710" ContentLicense="CC BY-SA 3.0" />
  <row Id="27020" PostTypeId="2" ParentId="27016" CreationDate="2018-01-25T03:01:17.657" Score="6" Body="&lt;p&gt;It's not so much a machine learning term as it is a &lt;em&gt;control theory&lt;/em&gt; term. A &quot;control policy&quot; is a heuristic that suggests a particular set of actions in response to the current state of the agent (in your case, a robot) and the environment. In the case of reinforcement learning, a policy is parameterized by the network weights. Changing the weights changes the policy, so the distribution of weights comprises a distribution over policies, hence why fitting models in this context is often referred to as &quot;policy search&quot;. It's not uncommon to use an ensemble for these kinds of problems, in which case each component of the ensemble comprises a different policy which recommends some action, and then the ensemblification mechanism selects an action from one of these distinct policies (e.g. by vote or highest score) or combines their recommendations into an action (e.g. by taking an average). The whole ensemble can also be described as representing a policy. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-01-25T03:01:17.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27021" PostTypeId="2" ParentId="26978" CreationDate="2018-01-25T03:22:40.623" Score="0" Body="&lt;p&gt;Depending on the sampling rate of the data, I'd create a new variable with stdev calculated using a trailing window, and create k-means clusters using stdev. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on a visual assessment, you could go with three clusters but try with 2 or 4 too and select whichever results in the smallest &lt;strong&gt;WSS&lt;/strong&gt;(within cluster sum of squares).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've found these you can assign each cluster a class. However, the problem with this strategy is it will only work with future data that has &lt;strong&gt;Similar&lt;/strong&gt; stdev.&lt;/p&gt;&#xA;" OwnerUserId="45148" LastEditorUserId="40853" LastEditDate="2018-01-25T04:20:15.997" LastActivityDate="2018-01-25T04:20:15.997" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27023" PostTypeId="2" ParentId="27019" CreationDate="2018-01-25T06:42:24.000" Score="1" Body="&lt;p&gt;Bascially this is true. Deep Learning is kind of like stacking your machine learning algorithms on top of each other. Have a look &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/7qru88/d_differences_between_ml_ds_and_ai_my_attempt_at/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and the associated picture. &lt;/p&gt;&#xA;" OwnerUserId="45086" LastEditorUserId="29575" LastEditDate="2018-01-25T06:57:18.573" LastActivityDate="2018-01-25T06:57:18.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27024" PostTypeId="2" ParentId="5228" CreationDate="2018-01-25T07:43:29.207" Score="0" Body="&lt;p&gt;There is no single answer. Part of the reason why there are different clustering algorithms is that there are different criteria for a cluster. &#xA;One is the number of triangles within a cluster, compared with the number that cross the boundary - but this is useless in a bipartite graph. Infomap has a subtle one that sometimes gives good results. &#xA;One criterion compares the  number of edges within the cluster (divided by the size of the cluster) to the number of edges that leave the cluster (divided by the size of the rest of the graph).&#xA;Cuts, as suggested by the previous answer, are very appropriate if the edge weight can reasonably be thought of as the capacity for a flow of something between nodes, e.g. information. In that case, the min cut spanning tree is an reasonable summary of the graph. Strong links near leaves of the tree and weak links near the centre would indicate modularity.&lt;/p&gt;&#xA;" OwnerUserId="8417" LastActivityDate="2018-01-25T07:43:29.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27025" PostTypeId="1" CreationDate="2018-01-25T09:33:23.283" Score="10" ViewCount="3061" Body="&lt;p&gt;How is the embedding layer trained in Keras Embedding layer? &#xA;(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we do not use a pretrained embedding.&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="38887" LastEditDate="2020-05-08T10:42:07.920" LastActivityDate="2020-08-09T09:55:13.887" Title="How the embedding layer is trained in Keras Embedding layer" Tags="|python|keras|word-embeddings|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27026" PostTypeId="2" ParentId="22565" CreationDate="2018-01-25T09:47:48.807" Score="0" Body="&lt;p&gt;The baseline of my answer is certainly, try the &lt;strong&gt;simplest&lt;/strong&gt; approach first, then go on. Machine Learning is not about heavy lifting, it is about smart usage of your tools. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of your algorithm depends mostly on your data and the types of anomalies you would expect. Do you have timeseries data with sesonality effects and trends, maybe the twitter anomaly detection package would be a good start. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance. If your CPU is hot you would expect it to run with high load. If the load is low maybe it is an anomaly or it has suffered from a high load. So one of your questions would be, can you spot anomalies or identify normal behaviour by just looking at one sample?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore the, &quot;No-free-Lunch&quot;, theorem applies again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, SVM and Isolation Forest are a good start. Even if you have a time series problem, where you want to track short term dependencies. Just include &quot;old&quot; rows to your current set:&lt;a href=&quot;https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can even try simpler approaches like Gaussian distribution of your values, calculating the probabilities of each permutation (if you features are small enough) or building a markov modell.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unsatisfying results? Dive deeper, maybe Auto Encoder do the trick, maybe LSTMs, maybe a combination of both.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A short summary of unsupervised and semi-supervised ML algorithms I recently used, sorted by complexity (kind of):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Median Absolute Deviation (Twitters anomaly detection library)&lt;/li&gt;&#xA;&lt;li&gt;Local Outloer Factor&lt;/li&gt;&#xA;&lt;li&gt;Isolation Forest&lt;/li&gt;&#xA;&lt;li&gt;Elliptic Envelope&lt;/li&gt;&#xA;&lt;li&gt;Autoregressive integrated moving average&lt;/li&gt;&#xA;&lt;li&gt;Auto Encoder&lt;/li&gt;&#xA;&lt;li&gt;Time Series Discordes&lt;/li&gt;&#xA;&lt;li&gt;LSTMs for forcasting &lt;/li&gt;&#xA;&lt;li&gt;Replicator Neuronal Networks&lt;/li&gt;&#xA;&lt;li&gt;Bayesian networks&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are certainly supervised approaches but I am not familiar with those. &lt;/p&gt;&#xA;" OwnerUserId="45086" LastActivityDate="2018-01-25T09:47:48.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27028" PostTypeId="1" AcceptedAnswerId="27029" CreationDate="2018-01-25T10:07:14.350" Score="4" ViewCount="1000" Body="&lt;p&gt;I have a difficulty where to start the implementation of incremental stochastic gradient descent algorithm and its respective implementation in logistic regression. I don't quite understand this algorithm and there are few sources to explain it with crystal-clear interpretation and possible demo code. I am quite new with ML algorithm and I have no idea which would be efficient workaround to solve this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, the problem I am studying is to implement hogWild! algorithm for logistic regression, which asks me to program incremental SGD algorithm with a sequential order. Can anyone give me a general idea or possible pipeline to make this happen in python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;logistic loss function and gradient&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/coO0z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/coO0z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my implementation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import scipy as sp&#xA;import sklearn as sl&#xA;from scipy import special as ss&#xA;from  sklearn import datasets&#xA;&#xA;X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')&#xA;X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt', &#xA;                                          n_features=X_train.shape[1])&#xA;&#xA;class ISGD:&#xA;def lossFunc(X,y,w):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y* np.dot(X,w,))))&#xA;    rslt=np.float(lossFnc)&#xA;    return rslt&#xA;&#xA;def gradFnc(X,y,w):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    gradF1=-y*np.nan_to_num(ss.expm1(-y))&#xA;    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))&#xA;    gradF3=gradF2.resize(gradF2.shape[0],)&#xA;    return gradF3&#xA;&#xA;def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):&#xA;    self.w=None&#xA;    self.learnRate=learnRate&#xA;    self.verbose=verbose&#xA;    self.num_iter=num_iter&#xA;&#xA;&#xA;def fitt(self, X,y):&#xA;    n,d=X.shape&#xA;    self.w=np.zeros(shape=(d,))&#xA;&#xA;    for i in range(self.num_iter):&#xA;        print (&quot;\n:&quot;, &quot;Iteration:&quot;, i)&#xA;&#xA;        grd=gradFnc(self.w, X,y)&#xA;        grd.resize((grd.shape[0],1))&#xA;        self.w=self.w-grd&#xA;        print &quot;Loss:&quot;, lossFunc(self.w,X,y)&#xA;&#xA;    return self&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Seems my above implementation has some problems. Can anyone help me how to correct that? Plus, I don't have a solid idea how to implement incremental SGD sequentially. How can I make this happen? Any idea?&lt;/p&gt;&#xA;" OwnerUserId="45011" LastEditorUserId="45011" LastEditDate="2018-01-25T16:05:46.227" LastActivityDate="2022-10-27T20:07:17.913" Title="How to understand incremental stochastic gradient algorithm and its implementation in logistic regression [updated]?" Tags="|machine-learning|python|algorithms|gradient-descent|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27029" PostTypeId="2" ParentId="27028" CreationDate="2018-01-25T10:36:38.133" Score="3" Body="&lt;h1&gt;Gradient Descent&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The idea of gradient descent is to traverse a function $f_{LR}(\textbf{w})$ and find a local maximum or minimum for a set a values $\textbf{w}$. Gradient descent is an iterative process. At each iteration you will evaluate the function given your current set of values. Then you will take the derivative of that function with respect to those values to see how much each value contributes to the slope of the function. We can then change the values in a proportionate way to move towards this optimal point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The gradient descent equation is described as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\textbf{w}^{(k+1)} = \textbf{w}^{(k)} - \rho \frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\rho$ is the learning rate, usually small number. A constant which determines the speed at which we want to change the values we are optimizing. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Initializing the values&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There's many ways to initialize these values. We can either set them all to zero, or set them randomly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can take a random instance in your dataset $x_i$ and $y_i$ and compute the derivative &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}} = \textbf{x}_i (-y_i \frac{e^{-y_i \textbf{x}_i \textbf{w}}}{1 + e^{-y_i \textbf{x}_i \textbf{w}}})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once the compute this you can put the result into the gradient descent equation and update all your weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You then continue to go through this process until your weights $\textbf{w}$ converge to a value, or some other condition is met. You might want to limit your algorithm with some iteration counter to avoid infinite loops caused by weights that do not converge. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Gradient Descent Code&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Here is an example using gradient descent to train weights along the logistic regression loss function. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;def update_weights(x_i, y_i, w):&#xA;   p = 0.8                  # The learning rate&#xA;   yhat = predict(x_i, w)   # The predicted target&#xA;   error = y_i - yhat       &#xA;   return w + p * (y_i - yhat) * x_i  # Update the weights&#xA;&#xA;def predict(x_i, w):&#xA;   return 1/(1+np.exp(-1 * np.dot(w.T, x_i)))&#xA;&#xA;def train_weights(x, y, verbose = 0):&#xA;   w = np.zeros((x.shape[1],))&#xA;   w_temp = np.zeros((x.shape[1],)) &#xA;&#xA;   epoch = 0&#xA;   while epoch &amp;lt;= 1000:&#xA;      for i, x_i in enumerate(x):&#xA;         w = update_weights(x_i, y[i], w)&#xA;&#xA;      pred = 1/(1+np.exp(-1 * np.dot(x, w)))&#xA;      error = np.sum(pred - y)&#xA;&#xA;      pred[pred &amp;lt; 0.5] = 0&#xA;      pred[pred &amp;gt;= 0.5] = 1&#xA;      epoch += 1&#xA;&#xA;      if verbose == 1:&#xA;         print('------------------------------------------------')&#xA;         print('Targets: ', y)&#xA;         print('Predictions: ', pred)&#xA;         print('Predictions: ', pred)&#xA;         print('------------------------------------------------')&#xA;&#xA;      # Check if we have reach convergence&#xA;      if np.sum(np.abs(w_temp - w)) &amp;lt; 0.001:&#xA;         print(epoch)&#xA;         print(error)&#xA;         return w&#xA;      else: &#xA;         w_temp = w&#xA;  return w&#xA;&#xA;# Create some artificial database&#xA;x = np.zeros((4,3))&#xA;y = np.zeros((4,)) &#xA;x[0] = [1, 1, 1]&#xA;x[1] = [1, 2, 2]&#xA;x[2] = [1, 10, 10]&#xA;x[3] = [1, 11, 11]&#xA;y[0] = 0&#xA;y[1] = 0&#xA;y[2] = 1&#xA;y[3] = 1&#xA;&#xA;print(x)&#xA;print(y)&#xA;&#xA;# Train the weights&#xA;w = train_weights(x, y, 0)&#xA;print('Trained weights: ', w)&#xA;&#xA;# Get predicitons&#xA;pred = 1/(1+np.exp(-1 * np.dot(x, w)))&#xA;error = np.sum(pred - y)   &#xA;pred[pred &amp;lt; 0.5] = 0&#xA;pred[pred &amp;gt;= 0.5] = 1&#xA;print('Predictions: ', pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-01-26T12:31:30.580" LastActivityDate="2018-01-26T12:31:30.580" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27030" PostTypeId="1" CreationDate="2018-01-25T10:41:25.267" Score="4" ViewCount="896" Body="&lt;p&gt;I have an understanding of the technical details of word2vec. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is why semantically similar words should have high cosine similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings&lt;/p&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="43271" LastEditDate="2018-01-25T13:07:51.930" LastActivityDate="2018-01-25T13:07:51.930" Title="Why do good word embeddings come out of maximizing cosine similarity?" Tags="|neural-network|nlp|word2vec|word-embeddings|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27031" PostTypeId="1" AcceptedAnswerId="27033" CreationDate="2018-01-25T11:09:14.883" Score="5" ViewCount="19952" Body="&lt;p&gt;I have pile of vectors where the values could be plotted like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dfiU8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dfiU8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to extract the &quot;spike values&quot; (over a certain threshold say 15,000). In this case there is fifteen. How could this be done with Python? (There is no predefined number of spikes but the threshold is a reliable filter value.)&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2018-11-23T23:17:45.230" Title="How to get spike values from a value sequence?" Tags="|python|feature-extraction|preprocessing|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27032" PostTypeId="1" CreationDate="2018-01-25T11:47:23.453" Score="1" ViewCount="2220" Body="&lt;p&gt;I want to read an excel file using &lt;strong&gt;pandas&lt;/strong&gt; &lt;strong&gt;read_excel&lt;/strong&gt; method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it shows an out of range error for the date value . &#xA;How can I fix this?. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_excel('/home/destiny2/Documents/elejio_data/Raipur/4 Star Hotels.xls')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return epoch + datetime.timedelta(days, seconds, 0, milliseconds)&#xA;&#xA;OverflowError: date value out of range&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40102" LastEditorUserId="43271" LastEditDate="2018-01-25T11:55:34.863" LastActivityDate="2018-01-25T11:55:34.863" Title="Date value out of range (pandas)" Tags="|python|pandas|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27033" PostTypeId="2" ParentId="27031" CreationDate="2018-01-25T11:52:27.723" Score="9" Body="&lt;p&gt;This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_df[data_df &amp;gt; 15000]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this data is sitting in a particular column, you can use this instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_df[data_df['column_name'] &amp;gt; 15000]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These will return the peak values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Updated Answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want local extreme points (e.g. maximum or minimum ) around each peak, check &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.signal.argrelextrema.html&quot; rel=&quot;noreferrer&quot;&gt;scipy.signal.argrelextrema&lt;/a&gt; in Scipy. A concrete example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's make a artificial random data with random spikes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np                    &#xA;import matplotlib.pyplot as plt &#xA;&#xA;random_number1 =np.random.randint(0,200,20)&#xA;random_number2=np.random.randint(0,20,100)&#xA;random_number=np.concatenate((random_number1,random_number2))&#xA;np.random.shuffle(random_number)&#xA;plt.plot(random_number)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/I5InT.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/I5InT.jpg&quot; alt=&quot;original data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now using &lt;em&gt;argrelextrema&lt;/em&gt; function you will find the index of relative extreme values (either minimum or maximum)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c_max_index = argrelextrema(random_number, np.greater, order=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please make sure you understand the &quot;order&quot; option. It basically look around 5 neighbouring points, and it returns the maximum in this case. And you can see how it works by pinpointing the found points on the actual graph like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.plot(random_number)&#xA;plt.scatter(c_max_index[0],random_number[c_max_index[0]],linewidth=0.3, s=50, c='r')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XQvuv.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XQvuv.jpg&quot; alt=&quot;original data with peak points in red&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that you can retrieve peak points via random_number[c_max_index[0]], and c_max_index are just indexes of the extreme points.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-01-25T14:21:29.780" LastActivityDate="2018-01-25T14:21:29.780" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27034" PostTypeId="2" ParentId="27030" CreationDate="2018-01-25T13:07:08.007" Score="3" Body="&lt;p&gt;Training word embeddings does not rely on optimizing the cosine similarity of words. It usually relies on prediction problems. Take for instance the skipgram model: you are predicting the context of a word, given the word. Such models, project words in geometric spaces (for example the commonly used ~300 dimensions). In other words, a word is associated with 300-dimensional dense vectors. Due to the way that these vectors are learned, they capture the semantics of the words and hence similar words are close to the induced space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cosine similarity (or dot product) captures exactly this semantic closeness. Intuitively, we want similar words to be close, &lt;em&gt;because&lt;/em&gt; they are similar and we hope that the space models some of the word properties and meaning. &lt;/p&gt;&#xA;" OwnerUserId="24215" LastActivityDate="2018-01-25T13:07:08.007" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27035" PostTypeId="1" AcceptedAnswerId="27076" CreationDate="2018-01-25T13:13:54.110" Score="3" ViewCount="1097" Body="&lt;p&gt;I'm trying to learn a simple linear softmax model on some data. The LogisticRegression in scikit-learn seems to work fine, and now I am trying to port the code to TensorFlow, but I'm not getting the same performance, but quite a bit worse. I understand that the results will not be exactly equal (scikit learn has regularization params etc), but it's too far off.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;total = pd.read_feather('testfile.feather')&#xA;&#xA;labels = total['labels']&#xA;features = total[['f1', 'f2']]&#xA;&#xA;print(labels.shape)&#xA;print(features.shape)&#xA;&#xA;classifier = linear_model.LogisticRegression(C=1e5, solver='newton-cg', multi_class='multinomial')&#xA;classifier.fit(features, labels)&#xA;pred_labels = classifier.predict(features)&#xA;&#xA;print(&quot;SCI-KITLEARN RESULTS: &quot;)&#xA;print('\tAccuracy:', classifier.score(features, labels)) &#xA;print('\tPrecision:', precision_score(labels, pred_labels, average='macro'))&#xA;print('\tRecall:', recall_score(labels, pred_labels, average='macro'))&#xA;print('\tF1:', f1_score(labels, pred_labels, average='macro'))&#xA;&#xA;# now try softmax regression with tensorflow &#xA;print(&quot;\n\nTENSORFLOW RESULTS: &quot;)&#xA;&#xA;## By default, the OneHotEncoder class will return a more efficient sparse encoding. &#xA;## This may not be suitable for some applications, such as use with the Keras deep learning library. &#xA;## In this case, we disabled the sparse return type by setting the sparse=False argument.&#xA;enc = OneHotEncoder(sparse=False)&#xA;enc.fit(labels.values.reshape(len(labels), 1)) # Reshape is required as &#xA;Encoder expect 2D data as input&#xA;labels_one_hot = enc.transform(labels.values.reshape(len(labels), 1))&#xA;&#xA;# tf Graph Input&#xA;x = tf.placeholder(tf.float32, [None, 2]) # 2 input features&#xA;y = tf.placeholder(tf.float32, [None, 5]) # 5 output classes&#xA;&#xA;W = tf.Variable(tf.zeros([2, 5]))&#xA;b = tf.Variable(tf.zeros([5]))&#xA;&#xA;# Construct model&#xA;pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax&#xA;&#xA;clas = tf.argmax(pred, axis=1)&#xA;&#xA;# Minimize error using cross entropy&#xA;cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))&#xA;# Gradient Descent&#xA;optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)&#xA;&#xA;# Initialize the variables (i.e. assign their default value)&#xA;init = tf.global_variables_initializer()&#xA;&#xA;# Start training&#xA;with tf.Session() as sess:&#xA;&#xA;   # Run the initializer&#xA;   sess.run(init)&#xA;&#xA;   # Training cycle&#xA;   for epoch in range(1000):&#xA;      # Run optimization op (backprop) and cost op (to get loss value)&#xA;    _, c = sess.run([optimizer, cost], feed_dict={x: features, y: labels_one_hot})&#xA;&#xA;# Test model&#xA;correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))&#xA;class_out = clas.eval({x: features})&#xA;&#xA;# Calculate accuracy&#xA;accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))&#xA;print(&quot;\tAccuracy:&quot;, accuracy.eval({x: features, y: labels_one_hot}))&#xA;print('\tPrecision:', precision_score(labels, class_out, average='macro'))&#xA;print('\tRecall:', recall_score(labels, class_out, average='macro'))&#xA;print('\tF1:', f1_score(labels, class_out, average='macro'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output of this code is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1681,)&#xA;(1681, 2)&#xA;SCI-KITLEARN RESULTS:&#xA;    Accuracy: 0.822129684711&#xA;    Precision: 0.837883361162&#xA;    Recall: 0.784522522208&#xA;    F1: 0.806251963817&#xA;&#xA;&#xA;TENSORFLOW RESULTS:&#xA;    Accuracy: 0.694825&#xA;    Precision: 0.735883666192&#xA;    Recall: 0.649145125846&#xA;    F1: 0.678045562185&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I inspected the result of the one-hot-encoding, and the data, but I have no idea why the result in TensorFlow is much worse.&lt;/p&gt;&#xA;" OwnerUserId="42681" LastEditorUserId="28175" LastEditDate="2018-01-26T10:28:04.580" LastActivityDate="2018-01-26T10:28:04.580" Title="Softmax: Different output scikit-learn and TensorFlow" Tags="|machine-learning|neural-network|deep-learning|scikit-learn|tensorflow|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27036" PostTypeId="2" ParentId="27031" CreationDate="2018-01-25T13:25:46.360" Score="0" Body="&lt;p&gt;You could implement some sort of sliding window algorithm with with a window size of ca. 50 (according to your diagram) and then pick only the highest value within the window.&lt;/p&gt;&#xA;" OwnerUserId="43367" LastActivityDate="2018-01-25T13:25:46.360" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27038" PostTypeId="1" AcceptedAnswerId="27044" CreationDate="2018-01-25T13:36:24.913" Score="3" ViewCount="6216" Body="&lt;p&gt;I am trying to use an autoencoder (as described here &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.keras.io/building-autoencoders-in-keras.html#&lt;/a&gt;) for anomaly detection. I am using a ~1700 feature vector (rather than images, which were used in the example) with each vector describing a different protein interaction. I have a &quot;normal&quot; category of interactions on which I train the AE, then I feed it new vectors and use reconstruction error to detect anomalous interactions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Adjusting my threshold so I get a true positive rate of 0.95, I get a false positive rate of 0.15, which is rather high. When I trained xgboost on the normal and anomalous vectors (using both types of interactions in training and testing) I was able to get precision of 0.98 **.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that mean that my model (or indeed my approach of using an AE) is ineffective, or maybe this is the best I could hope for when training an anomaly detector rather than a 2 category classifier (that is, xgboost in my case)?&#xA;How should I proceed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;** Of course, this is merely a sanity check, and cannot be used as the solution.  I need the model to detect anomalies that can be very different from those I currently have - thus I need to train it on the normal interaction set, and leave anomalies for testing alone.&lt;/p&gt;&#xA;" OwnerUserId="43472" LastEditorUserId="43472" LastEditDate="2018-01-25T14:06:00.250" LastActivityDate="2018-01-29T09:31:30.983" Title="Autoencoder for anomaly detection from feature vectors" Tags="|keras|anomaly-detection|autoencoder|bioinformatics|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27039" PostTypeId="2" ParentId="26618" CreationDate="2018-01-25T14:58:18.510" Score="2" Body="&lt;p&gt;I guess you are doing something wrong in your code. I guess its better to use &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/htA0l/gradient-checking&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;gradient checking&lt;/code&gt;&lt;/a&gt; approach for figuring out whether the whole code has any problem or not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the comments, if I want to show you exactly what happens, first take a look at &lt;a href=&quot;https://www.coursera.org/learn/neural-networks/lecture/zO1Is/a-simple-example-of-learning-6-min&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which professor Hinton himself explains that what &lt;code&gt;MLPs&lt;/code&gt; learn is like learning masks instead of learning the features of the inputs. For illustrating more you can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt; which shows that MLPs can learn what and I hope that you can expand it to higher dimensions, for your case 784 dimensions. If you use &lt;code&gt;MLPs&lt;/code&gt; alone, you will face to the problem of something which is like masking. What &lt;code&gt;CNNs&lt;/code&gt; try to find is feature. They try to find features which can be better explained by finding hierarchical features. You can also take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt; which explains that using &lt;code&gt;CNNs&lt;/code&gt; you try to find features, and after convolution layers, you use dense layers, &lt;code&gt;MLPs&lt;/code&gt; which they try to classify the features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;MLPs&lt;/code&gt; are good for classifying &lt;code&gt;MNIST&lt;/code&gt; data set but they are weak for generalizing unseen data. If your code works fine after using &lt;em&gt;gradient checking&lt;/em&gt; try to use different hyper parameters which I guess our friend has explained so much well for you.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-25T14:58:18.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27041" PostTypeId="1" CreationDate="2018-01-25T15:28:51.923" Score="2" ViewCount="56" Body="&lt;p&gt;I am reading a text book that basically says the following:&#xA;Given a matrix &lt;strong&gt;A&lt;/strong&gt; where A is USERS x ITEMS we can use SVD to decompose the matrix into:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$A = U \times \Sigma \times V^T$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we can take the first $n$ columns of these matrices to get:&#xA;$$A \approx U_n \times \Sigma_n \times V_n^T$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then say we have a USER vector, $u$, that has ratings for items, we find out where $u$ is position in the $n$-dimensional space:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$u_{nd} = u \times U_n \times\Sigma_{n}^{-1}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we can use the position of $u_{nd}$ to discover what items and users are similar to $u_{nd}$ by using a method like cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So this all makes sense to me (I think). Pretty straight forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if I go to &lt;a href=&quot;https://cambridgespark.com/content/tutorials/implementing-your-own-recommender-systems-in-Python/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt;, and go to the SVD section, she writes that you can provide predictions by just getting a low-rank approximation of a ratings matrix? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, if you have:&#xA;$$A = U \times \Sigma \times V^T$$&#xA;We can simply make predictions by:&#xA;$$A \approx U_n \times \Sigma_n \times V_n^T$$&#xA;where $n$ is the $n$ lower-rank approximation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do both of these methods work? What is the point of the first method if I can just use the second method to make predictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FURTHER EDIT: Why does the first method use the inverse of matrix $\Sigma$ whereas the second does not?&lt;/p&gt;&#xA;" OwnerUserId="45172" LastEditorUserId="45172" LastEditDate="2018-01-25T18:00:26.897" LastActivityDate="2018-01-25T18:00:26.897" Title="How does SVD actually provide the recommendations? I seem to get conflicting answers" Tags="|predictive-modeling|recommender-system|matrix-factorisation|linear-algebra|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27042" PostTypeId="1" AcceptedAnswerId="27043" CreationDate="2018-01-25T16:10:12.147" Score="3" ViewCount="797" Body="&lt;p&gt;In the paper &lt;a href=&quot;https://nlp.stanford.edu/pubs/glove.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;GloVe: Global Vectors for Word Representation&lt;/em&gt;&lt;/a&gt;, there is this part (bottom of third page) I don't understand:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tQCzJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tQCzJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand what groups and homomorphisms are. What I don't understand is what requiring $ F $ to be a homomorphism between $ (\mathbb{R},+) $ and $ (\mathbb{R}_{&amp;gt;0},\times) $ has to do with making $ F $ symmetrical in $ w $ and $ \tilde{w}_k $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I misunderstanding something? We want $ F $ to be unchanged if we either interchange $ w_i $ and $ \tilde{w}_k $ OR interchange $ w_j $ and $ \tilde{w}_k $, right? Is this the only way to achieve the symmetry between  $ w $ and $ \tilde{w}_k $?&lt;/p&gt;&#xA;" OwnerUserId="45178" LastEditorUserId="29575" LastEditDate="2018-01-25T16:50:35.670" LastActivityDate="2018-01-25T16:55:10.050" Title="GloVe vector representation homomorphism question" Tags="|word-embeddings|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27043" PostTypeId="2" ParentId="27042" CreationDate="2018-01-25T16:29:49.470" Score="2" Body="&lt;p&gt;If you're asking if the group homomorphism makes the the process symmetric then no it doesn't directly.  However, they use the fact that they require a group homomorphism to show that $w_{i}^{T} \tilde{w}_k = log(P_{ik})=log(X_{ik}) - log(X_{i})$  This nearly gives us symmetry.  Finally by adding $\tilde{b}_{k}$ into the equation you restore symmetry.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in short $w_{i}^{T} \tilde{w}_k + b_{i} + \tilde{b}_{k} = log(X_{ik})$ is what ensures symmetry, and the group homomorphism is a tool to get there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some more details&#xA;Essentially, what we want is the ability to peform a label switch.  Group homomorphism helps with this process because it perseves a mapping between the $(R, +)$ and $(R, x)$.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$F((w_{i}^{T} - w_{j}^{T})w_{k}^{'})=F(w_{i}^{T}w_{k}^{'}+( - w_{j}^{T}w_{k}^{'})) = F(w_{i}^{T}w_{k}^{'}) \times F(-w_{j}^{T}w_{k}^{'} )= F(w_{i}^{T}) \times F(w_{j}^{T}w_{k}^{'})^{-1} = \frac{F(w_{i}^{T}w_{k}^{'})}{F(w_{j}^{T}w_{k}^{'})}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The group homomorphism here allows for that to occur.   Therefore we can see that by setting $F(w_{i}^{T}w_{k}^{i}) = \frac{X_{ik}}{X_{i}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now finally we can say that $w_{i}^{T} {w}_k^{'} = log(P_{ik})=log(X_{ik}) - log(X_{i}).$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as far as your comment, it is the most sensible chocie for their method and of which they buld the core mathematicals to GloVE.  Changing it, I imagine wouldn't be a trivial thing.  I imagine if you did, much of what is derived, including the loss function would change.  But with that said, I imagine there are otherwise to achieve label switching.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-01-25T16:55:10.050" LastActivityDate="2018-01-25T16:55:10.050" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27044" PostTypeId="2" ParentId="27038" CreationDate="2018-01-25T16:44:43.407" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Does that mean that my model (or indeed my approach of using an AE) is ineffective&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How should I proceed?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, the MNIST database contains 50,000 image. And you need enough variance in order to not overfit your training data. Tree based approaches are, at least in my experience, easier to train. If you like to stick at the anomaly detection part, which I recommend since you don't know what anomalies you will face, try the Isolation Forest Algorithm. But for a solid recommendation I would need to know how your data looks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Btw, A good metric to use in such a case is  the ROC score, which basically tells you how likely it is that your model will classify new data points correctly. Check out the link for an visual explanation&lt;a href=&quot;https://www.balabit.com/blog/novel-performance-metrics-anomaly-detection-algorithms&quot; rel=&quot;nofollow noreferrer&quot;&gt;ROC explained&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So Baseline is, try less complex approaches until you a certain that they are not sufficient enough. &lt;/p&gt;&#xA;" OwnerUserId="45086" LastActivityDate="2018-01-25T16:44:43.407" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27046" PostTypeId="1" AcceptedAnswerId="27052" CreationDate="2018-01-25T17:07:28.273" Score="0" ViewCount="506" Body="&lt;p&gt;I have a dataset of shape edges, that I am trying to make a model for with sklearn. I'm new to the machine learning world, so I am struggling to create a good model. Using SVM, I was able to get a supposed 81% precision, but when I feed it an image outside the training or test set, it consistently returns the wrong prediction, almost every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SbfuI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SbfuI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OPANz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OPANz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wXiEH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wXiEH.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question: Is there is a better way of doing this than using SVM? Or are these shapes too similar? I have 90 images in the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://github.com/teslaworksumn/guild-master/blob/Machine_Learning/Machine_Learning/SVM_model_creator.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; to my ML code.&lt;/p&gt;&#xA;" OwnerUserId="45181" LastEditorUserId="29575" LastEditDate="2018-01-25T17:20:59.667" LastActivityDate="2018-01-25T20:37:24.353" Title="Shape classification algorithm" Tags="|classification|scikit-learn|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27047" PostTypeId="1" CreationDate="2018-01-25T17:12:41.587" Score="9" ViewCount="1302" Body="&lt;p&gt;I'm working on using an LSTM to predict the direction of the market for the next day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$ represents a vector of features for day $i$, i.e $x_i \in \mathbb{R}^D$. The target variable $y_i$ is the direction of the market for the next day $$ y_i = I_{P_{i+2} - P_{i+1} &amp;gt; 0} $$ where $P_i$ is the opening price of the asset for day $i$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering now how to configure the input to fit in an LSTM framework. The LSTM requires a sequence of length $T$ and uses this together with target $y_T$. One approach is a rolling window. Take input as input the sequences $I_1 = x_1, \ldots, x_T$, $I_2 = x_2, \ldots, x_{T+1} \ldots $ and use $y_T, y_{T+1} \ldots$ as the target variables. The problem with this is that $I_1, I_2$ are very similar, they differ only on two points yet the target variable may be $1$ for the first series and $0$ for the other, making it impossible for the LSTM to learn in my opinion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if anybody has any idea of how to approach this problem. The rolling window approach above is considering the inputs $I_k$ as independent, similar to if we would have input $I_k$ to be a sentence which should be classified as french or english. I want the LSTM to take into consideration that the sequences its fed are all parts of the same long sequence if that makes sense.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of papers use recurrent neural networks for this problem, but never really specify on how they structured the input.&lt;/p&gt;&#xA;" OwnerUserId="45183" LastEditorUserId="29575" LastEditDate="2018-01-25T17:29:27.997" LastActivityDate="2022-02-19T10:04:05.590" Title="Input for LSTM for financial time series directional prediction" Tags="|machine-learning|time-series|lstm|rnn|sequence-to-sequence|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27048" PostTypeId="1" CreationDate="2018-01-25T17:30:33.267" Score="5" ViewCount="2564" Body="&lt;p&gt;I am working on sentiment analysis using tweets text. I am able to build a word vector using Keras &lt;code&gt;text_to_sequence()&lt;/code&gt; method with pretrained GloVe embeddings in Python. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is how to add extra features beside the created GloVe embeddings and shape it in right way to feed to the embedding layer and the algorithm in general? For example, assuming the embedding's length is 100 and the extra features are 3, will appending the 3 features to 100 vector resulting in 103 vector length is the right approach? If so how would you implement it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;corpusAsList&lt;/code&gt; is the corpus as text in a list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;maxlen&lt;/code&gt; is the maximum number of columns to pad the word sequence with, so that all vectors have the same length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;maxVocabulary&lt;/code&gt; is the maximum amount of words to be used in the sequence and ignore the rest of the words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And &lt;code&gt;tokenizer&lt;/code&gt; is used to represent a word as a digit in the vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if we called &lt;code&gt;texts_to_sequences()&lt;/code&gt; like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   # Create tokenized corpuss&#xA;    listTokenized = tokenizer.texts_to_sequences(corpusAsList)&#xA;&#xA;    # Sequence dataset&#xA;    sequenceMatrixPrediction = sequence.pad_sequences(listTokenized, maxlen=maxlen)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then trained a Bidirectional LSTM model like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def getLSTMModel(maxlen, maxVocabulary, embedSizeLSTM=100, weights=None):&#xA;&#xA;    inp = Input(shape=(maxlen,))&#xA;    x = Embedding(maxVocabulary, embedSizeLSTM,&#xA;                 weights=[weights], trainable=False)(inp)&#xA;    x = Bidirectional(LSTM(80, return_sequences=True))(x)&#xA;    x = GlobalMaxPool1D()(x)&#xA;    x = Dropout(0.25)(x)&#xA;    x = Dense(70, activation=&quot;relu&quot;)(x)&#xA;    x = Dropout(0.25)(x)&#xA;    x = Dense(1, activation=&quot;sigmoid&quot;)(x)&#xA;    model = Model(inputs=inp, outputs=x)&#xA;    model.compile(loss='binary_crossentropy',&#xA;                  optimizer='adam',&#xA;                  metrics=['accuracy'])&#xA;&#xA;&#xA;return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How could we add three extra features to this model?&lt;/p&gt;&#xA;" OwnerUserId="45184" LastEditorUserId="13023" LastEditDate="2018-05-02T17:43:13.087" LastActivityDate="2018-05-02T18:15:38.397" Title="How to add more features in addition to a 100D word vector" Tags="|python|nlp|keras|word-embeddings|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27050" PostTypeId="2" ParentId="17134" CreationDate="2018-01-25T17:52:38.853" Score="1" Body="&lt;p&gt;The recurrent part of a network allows you, generally speaking, to model long and short term dependencies. So your model can have some sense of state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is typically advantageous if you're using timeseries. For example, if you have data from a heart rate monitor and like to like to classify between resting, stress and recovering. If your datapoint says your heart rate is at 130, it depends on whether you are recovering from high loads or something else.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &#xA;I forgotten your second question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I could think off a few possible answers. By convoluting the recurrent part you kind of filtering it. So you get a cleaner signal and errors won't stack as much. Vanilla rnn suffer from exploding vanishing gradients, so this could be his approach to overcome it.&#xA;Furthermore, you are embedding your features within the rcnn, which can lead, as he stated, to more paths to exploit. Which makes it less prone to overfitting, thus more generalizable.&lt;/p&gt;&#xA;" OwnerUserId="45086" LastEditorUserId="45086" LastEditDate="2018-01-25T21:37:16.333" LastActivityDate="2018-01-25T21:37:16.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27051" PostTypeId="1" CreationDate="2018-01-25T18:40:54.317" Score="2" ViewCount="548" Body="&lt;p&gt;I have ~4000 3D images with 0,1,2 labels. 70% of the labels are zeros.&#xA;The dimension of each image is [40, 30, 30]&#xA;The values usually range between -50 to 50 and about 10% of each image has non-zero values(bigger than 10^-4).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The below is my 3D CNN model with 3 pooling layers and 3 fully connected layers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.conv1 = conv3d(self.X, 10, 1, 32, name='conv1')&#xA;self.norm1 = tf.contrib.layers.batch_norm(self.conv1, center=True, scale=True, scope='bn1')&#xA;self.pool1 = max_pool(self.norm1, 3, 2, name='pool1')&#xA;self.conv2 = conv3d(self.pool1,8,1,64, name='conv2')&#xA;self.norm2 = tf.contrib.layers.batch_norm(self.conv2, center=True, scale=True, scope='bn2')&#xA;self.pool2 = max_pool(self.norm2, 3, 2, name='pool2') &#xA;self.conv3 = conv3d(self.pool2,8,1,64, name='conv3')&#xA;self.pool3 = max_pool(self.conv3, 3, 2, name='pool3') &#xA;proddim = np.prod(self.pool3.get_shape()[1:], dtype=np.uint)&#xA;&#xA;self.flat = tf.reshape(self.pool3, [-1,proddim ])&#xA;&#xA;self.fc4 = fc(self.flat, num_output=1024, name='fc4')&#xA;self.dropout4 = tf.layers.dropout(inputs=self.fc4, rate=0.2, training=self.dropout_train, name='dropout4')&#xA;self.fc5 = fc(self.dropout4, num_output=384, name='fc5')&#xA;self.dropout5 = tf.layers.dropout(inputs=self.fc5, rate=0.2, training=self.dropout_train, name='dropout5')  &#xA;self.fc6 = fc(self.dropout5, num_output=3, name='fc6')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I double, triple checked that the inputs are correct but my model always predict everything as zero.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it because my tensor or labels mostly have zero values? Or do I have too little data?&lt;/p&gt;&#xA;" OwnerUserId="45185" LastEditorUserId="29575" LastEditDate="2018-01-25T19:10:27.067" LastActivityDate="2018-01-26T10:33:13.217" Title="My CNN predicts everything as zero" Tags="|tensorflow|cnn|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27052" PostTypeId="2" ParentId="27046" CreationDate="2018-01-25T20:37:24.353" Score="0" Body="&lt;p&gt;That sounds like you're suffering from overfitting, probable duo the &quot;curse of dimensionality&quot;, which means 2 things. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;When trying to measure distance between two points in high dimensional space and trying to interpret this data, the difference between the longest and the shortest distance get's less meaningful.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Trying to train an algorithm with way less samples then features, like in your case, will cause that the algorithm will memorize your samples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I see 3 options. Getting more data, reduce your feature set or try greater values for C. C will regularize the decision boundary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More to read: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality&quot;&gt;Curse of dimensionality&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/10423/number-of-features-vs-number-of-observations&quot;&gt;n &gt;&gt; p&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45086" LastActivityDate="2018-01-25T20:37:24.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27055" PostTypeId="1" AcceptedAnswerId="27074" CreationDate="2018-01-25T22:52:10.020" Score="2" ViewCount="111" Body="&lt;p&gt;I have a collection of graph objects of variable size (input) which are each paired to another graph of variable size (output). The task is, given an input graph, produce the most likely output graph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been looking at 'structured output' techniques such as SSVM but as far as I can tell, they all act on outputs of fixed size (or at least a size that matches the input size, i.e. sequence tagging). Are there any tools that can map an input to a structured object of arbitrary size?&lt;/p&gt;&#xA;" OwnerUserId="45197" LastActivityDate="2018-01-26T20:03:11.370" Title="What algorithm can predict structured outputs of arbitrary size?" Tags="|svm|graphs|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27056" PostTypeId="1" CreationDate="2018-01-26T00:41:47.610" Score="1" ViewCount="3503" Body="&lt;p&gt;VGG16 struct is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Img-&amp;gt;&#xA;Conv1(3)-&amp;gt;Conv1(3)-&amp;gt;Pool1(2) ==&amp;gt;&#xA;Conv2(3)-&amp;gt;Conv2(3)-&amp;gt;Pool2(2) ==&amp;gt;&#xA;Conv3(3)-&amp;gt;Conv3(3)-&amp;gt;Conv3(3)-&amp;gt;Pool3(2) ==&amp;gt;&#xA;Conv4(3)-&amp;gt;Conv4(3)-&amp;gt;Conv4(3)-&amp;gt;Pool4(2) ==&amp;gt;&#xA;Conv5(3)-&amp;gt;Conv5(3)-&amp;gt;Conv5(3) ====&amp;gt; FC&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The flow :&#xA;&lt;a href=&quot;http://ethereon.github.io/netscope/#/gist/dc5003de6943ea5a6b8b&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ethereon.github.io/netscope/#/gist/dc5003de6943ea5a6b8b&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In keras code &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/applications/vgg16.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/blob/master/keras/applications/vgg16.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Block 1&#xA;x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(img_input)&#xA;x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)&#xA;x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)&#xA;&#xA;&#xA;....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It looks like just pass &lt;code&gt;Conv1_1&lt;/code&gt; result to &lt;code&gt;Conv1_2&lt;/code&gt;, and the rest too.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I can understand this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Img-&amp;gt;&#xA;Conv1(3)-&amp;gt;Pool1(2) ==&amp;gt;&#xA;Conv2(3)-&amp;gt;Pool2(2) ==&amp;gt;&#xA;Conv3(3)-&amp;gt;Pool3(2) ==&amp;gt;&#xA;Conv4(3)-&amp;gt;Pool4(2) ==&amp;gt;&#xA;Conv5(3) ====&amp;gt; FC&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But don't understand why Conv1_1 can connect to a same layer - Conv1_2 :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Img: 224x224x3&lt;/li&gt;&#xA;&lt;li&gt;kernel:3x3&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;depth:3 (channel)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Conv1_1: use 64 kenels. So there are 64 &lt;code&gt;3x3x3 kernel&lt;/code&gt; used to scan 224x224x3(with padding 1) ,  then get &lt;code&gt;Conv1_1_feature_map&lt;/code&gt; 224x224x64 .&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here, does it mean:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Conv1_2: use 64 &lt;code&gt;3x3x64 kernel&lt;/code&gt; to scan &lt;code&gt;Conv1_1_feature_map&lt;/code&gt; 224x224x64 , then get &lt;code&gt;Conv1_2_feature_map&lt;/code&gt; 224x224x64 ??&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If it is, could you explain what is the benifit ? I can't understand the meaning of this flow clearly. It is not clear that why &lt;code&gt;Conv1(3)-&amp;gt;Conv1(3)-&amp;gt;Pool1(2)&lt;/code&gt;  is better than &lt;code&gt;Conv1(3)-&amp;gt;Pool1&lt;/code&gt; . On my feeling, another &lt;strong&gt;same size Conv&lt;/strong&gt; looks like just make previous layer output more blurred/ambiguous, doesn't like Pooling layer concentrate previous Layer's output feature. &lt;/p&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="29169" LastEditDate="2021-02-13T16:44:10.167" LastActivityDate="2021-02-13T16:44:10.167" Title="How to understand conv layer to another same conv layer in VGG16?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|vgg16|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27057" PostTypeId="1" CreationDate="2018-01-26T00:51:26.563" Score="4" ViewCount="1210" Body="&lt;p&gt;Are there any advanced packages that allow automated hyperparameter tuning for neural networks and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, random search etc. that could allow for faster discovery of the optimal parameters)? I have heard of hyperopt, but it seems there are some &lt;a href=&quot;https://johnflux.com/2017/02/10/python-hyperopt-finding-the-optimal-hyper-parameters/&quot; rel=&quot;nofollow noreferrer&quot;&gt;problems&lt;/a&gt;, and I am not sure it can train traditional machine learning algorithms?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="29169" LastEditDate="2021-08-12T20:55:30.803" LastActivityDate="2021-08-12T20:56:55.597" Title="Automated Hyperparameter tuning" Tags="|hyperparameter|hyperparameter-tuning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27058" PostTypeId="1" CreationDate="2018-01-26T02:36:43.137" Score="1" ViewCount="538" Body="&lt;p&gt;I have files like the following: &lt;a href=&quot;https://pastebin.com/5mkXY1aU&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pastebin.com/5mkXY1aU&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are created by filling predefined forms so there are thousands of them that would match a pattern.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will try to give more generic example for the sake of simplicity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have thousands of documents created using different patterns like `&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;- &quot;sjkghkjfs &amp;lt;data&amp;gt; skjfs &amp;lt;data&amp;gt; kjskdfjsfkjs &amp;lt;data&amp;gt; sahkj&quot;;&#xA;- &quot;tretyer erytewr fskjdf &amp;lt;data&amp;gt; trjk&quot;;&#xA;- &quot;sdhfjsdhj &amp;lt;data&amp;gt; &amp;lt;data&amp;gt; &amp;lt;data&amp;gt; dsjadh&quot;;&#xA;- &quot;&amp;lt;data&amp;gt; djfhsdk&quot;;` and so on.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;lt;data&amp;gt;&lt;/code&gt; could be just any string.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what these patterns are beforehand. I want to find them out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can easily come up with some bruteforce solution but it would obviously not work on such amount of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds to me like a problem that might have already been solved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;I wondered if there is some readily available software tool I could utilize directly to solve this or a library atleast ?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;In case there isn't one how should I proceed to solve it in the most painless way ?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;It is a one time task to find these patterns out, so I am not into the idea of spending days on implementing a solution from the scratch.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="45206" LastEditorUserId="40853" LastEditDate="2018-01-26T02:44:23.353" LastActivityDate="2018-01-26T07:56:30.987" Title="Finding repeating string patterns in thousands of files" Tags="|classification|nlp|text-mining|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27061" PostTypeId="1" AcceptedAnswerId="27065" CreationDate="2018-01-26T04:08:16.267" Score="1" ViewCount="102" Body="&lt;p&gt;I have applied K-means and Hierarchical Agglomerative clustering method on some data and clustered them into 5 groups. For validation (agreement) purpose I used this formula:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c=(# of cluster shared/ Total cluster)*100 %&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I feel its a wrong validation technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use entropy, Rand index, Dunn number etc to measure the validation?&lt;/p&gt;&#xA;" OwnerUserId="44808" LastActivityDate="2018-03-28T00:33:50.023" Title="How to validate different method of clustering" Tags="|clustering|similarity|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27062" PostTypeId="1" AcceptedAnswerId="27560" CreationDate="2018-01-26T04:09:04.700" Score="1" ViewCount="1235" Body="&lt;p&gt;I used the the feature selection method: RFE to select feature for features. Now I want to select 20 features(there are about 50 features or more) with LGBM model as following code: But I found that the features(sel.ranking_) are different when the os or os version, or computer is changed. I don't know what caused the changing. More, how to solve it that features selection is fixed. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gbm = lgb.LGBMClassifier(&#xA;    boosting_type='gbdt',&#xA;    objective='binary',&#xA;    learning_rate=0.01,&#xA;    colsample_bytree=0.9,&#xA;    subsample=0.8,&#xA;    random_state=21,&#xA;    n_estimators=200,&#xA;    num_leaves=18)&#xA;sel = RFE(gbm, step=1, n_features_to_select=20, verbose=1)&#xA;sel.fit(X_train, y_train)&#xA;print (sel.ranking_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35597" LastActivityDate="2018-06-07T13:52:06.803" Title="why the RFE to select features is changing when the os or computer is changed" Tags="|machine-learning|python|scikit-learn|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27065" PostTypeId="2" ParentId="27061" CreationDate="2018-01-26T05:34:25.887" Score="0" Body="&lt;p&gt;Could always try a &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Silhouette_(clustering)&quot; rel=&quot;nofollow noreferrer&quot;&gt;silhouette score&lt;/a&gt;&lt;/strong&gt;, although it may not perform well on high dimensional data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Silhouette scores compare a datapoint's placement relative to its own cluster to its placement relative to other cluster/clusters.  The values returned will range between -1 (indicating a poor clustering) to +1 (indicating &quot;good&quot; clustering).  Using Wikipedia's example, use 2 clusters and assume there is an observation &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt; that has been placed in Cluster 1.  The Silhouette score uses distance measures (Euclidean, for example, but others can also be used) to determine the average distance of &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt;' to every other point in Cluster 1.  Then it determines the average distance of &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt; to every point in Cluster 2.  The silhouette formula for point &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt; would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;silhouette&lt;/em&gt;&lt;/strong&gt;(i) = $\frac{b(i)-a(i)}{Max\left \{ a(i), b(i) \right \}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remembering for this example that &lt;strong&gt;&lt;em&gt;i&lt;/em&gt;&lt;/strong&gt; is in Cluster 1, then:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If an object is closer, on average, to the points in Cluster 1 than&#xA;Cluster 2, then a(i) &amp;lt; b(i) and silhouette score is positive for that&#xA;point.&lt;/li&gt;&#xA;&lt;li&gt;If an object is closer, on average, to the points in Cluster 2 than&#xA;Cluster 1, then a(i) &gt; b(i) and silhouette score is negative for that&#xA;point.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Run this comparison across all points and all clusters, then average, and you will obtain a global silhouette score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will see most people capture silhouette score in a for loop where they vary the number of clusters; this allows quick comparison of a range of clusters.  &lt;/p&gt;&#xA;" OwnerUserId="36991" LastEditorUserId="36991" LastEditDate="2018-01-26T15:02:58.023" LastActivityDate="2018-01-26T15:02:58.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27066" PostTypeId="2" ParentId="27056" CreationDate="2018-01-26T05:50:37.803" Score="1" Body="&lt;p&gt;If I've got the meaning of question, first convolution accepts inputs of size &lt;code&gt;224*224*3&lt;/code&gt; means that height and width are both &lt;code&gt;224&lt;/code&gt; and the size of depth or number of activation maps, channels here, is equal to three. The output of this layer will be activation maps with equal height and width as the input, &lt;code&gt;224&lt;/code&gt;, because it is &lt;em&gt;same&lt;/em&gt; convolution. The number of activation maps for this layer which is going to be passed to the next layer is equal to &lt;code&gt;64&lt;/code&gt; because you have &lt;code&gt;64&lt;/code&gt; filters in this layer. The point is that each filter is of size &lt;code&gt;3*3*3&lt;/code&gt; to fit to the input. The output of each filter is an activation map of size 224*224*1. The output of filters come together and construct output of size &lt;code&gt;224*224*64&lt;/code&gt; which means the input of the next layer will have &lt;code&gt;64&lt;/code&gt; channels, actually depth here. Consequently, the filters of the second convolution layer is of size &lt;code&gt;3*3*224&lt;/code&gt; to match the entire input of the previous layer. In other words, you will have 64 filters of size &lt;code&gt;3*3*64&lt;/code&gt; and each will have an output of size &lt;code&gt;224*224*1&lt;/code&gt;. Take a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/MmYe2/classic-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which definitely can help you.&#xA;&lt;hr /&gt;&#xA;The purpose of writers of &lt;code&gt;VGG&lt;/code&gt; net was to make a network which was just deep enough to perform well on &lt;code&gt;ImageNet&lt;/code&gt; data-set. &lt;code&gt;CNNs&lt;/code&gt; have lots of hyper parameters which setting them is not based on well behaved math stuff. I mean there is no proof to show which hyper parameter is better than the others. They are found based on experience. They are gained practically. Writers of this paper tried to show that if you use same hyper parameters for convolution layers and just make the network deep and deeper, it was much deeper than &lt;code&gt;AlexNet&lt;/code&gt;, you will gain good performance without caring about different set of hyper parameters. For understanding what &lt;code&gt;ConvNets&lt;/code&gt; do, there is already an answer &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt; which may help you, it contains the interpretation of different layers in &lt;code&gt;CNNs&lt;/code&gt;. They have reached to this setting of convolution layers and architecture by experience and there idea was too use just a deep net without complicated hyper parameters.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-26T09:00:41.270" LastActivityDate="2018-01-26T09:00:41.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27067" PostTypeId="2" ParentId="10615" CreationDate="2018-01-26T07:08:03.570" Score="30" Body="&lt;p&gt;Following previous answers,&#xA;The number of parameters of LSTM, taking input vectors of size $m$ and giving output vectors of size $n$ is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$4(nm+n^2)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However in case your LSTM includes bias vectors, (&lt;a href=&quot;https://keras.io/layers/recurrent/#lstm&quot; rel=&quot;noreferrer&quot;&gt;this is the default in keras for example&lt;/a&gt;), the number becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$4(nm+n^2 + n)$$&lt;/p&gt;&#xA;" OwnerUserId="45217" LastEditorUserId="29575" LastEditDate="2018-01-26T14:34:28.137" LastActivityDate="2018-01-26T14:34:28.137" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27070" PostTypeId="2" ParentId="27058" CreationDate="2018-01-26T07:56:30.987" Score="1" Body="&lt;p&gt;It looks like you want new word discover ? &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Because thousand is not a big deal&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Just build ngrams of filenames , count them would be fine .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Trie&quot; rel=&quot;nofollow noreferrer&quot;&gt;Trie&lt;/a&gt; to store string counts,  reduce memory cost , I can provide a dict way(in python):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import defaultdict, Counter&#xA;&#xA;# for memory effcient, you would need trie here&#xA;# t = Trie()&#xA;t = defaultdict(int)&#xA;&#xA;&#xA;&#xA;filenames = [&quot;sjkghkjfs &amp;lt;data&amp;gt; skjfs &amp;lt;data&amp;gt; kjskdfjsfkjs &amp;lt;data&amp;gt; sahkj&quot;, &#xA;            &quot;tretyer erytewr fskjdf &amp;lt;data&amp;gt; trjk&quot;,&#xA;            &quot;....&quot;]&#xA;&#xA;# some preprocess, tokenize to  sentence , to  words , filter useless one&#xA;&#xA;def ngrams(s, start, end):&#xA;    for i in range(start, end+1):&#xA;        if len(s) &amp;gt; i:&#xA;            for j in range(i, len(s)):&#xA;                yield s[j:j+i]&#xA;&#xA;# Suppose you need same string with length 4 ~ 6&#xA;for s in filenames:&#xA;    for word in ngrams(s, 4, 7):&#xA;        t[word]+=1&#xA;&#xA;# Suppose you need most_common 5&#xA;print(Counter(t).most_common(5))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Further more&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;you can &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;tokenize the filenames at first. &lt;/li&gt;&#xA;&lt;li&gt;Calculate tf-idf or word entropy to remove the useless words&lt;/li&gt;&#xA;&lt;li&gt;then do something like above .&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h2&gt;And&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If you have a group of traget strings , &#xA;word2vec may be a good tool, you can use it to search the strings in similar domain.&lt;/p&gt;&#xA;" OwnerUserId="15325" LastActivityDate="2018-01-26T07:56:30.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27071" PostTypeId="1" AcceptedAnswerId="27073" CreationDate="2018-01-26T08:07:09.543" Score="0" ViewCount="12047" Body="&lt;p&gt;I have loaded a dataset and converted into data frame while I am using linear regression I am receiving the following value error as shown in my code below. I am at the moment doing a tutorial and could not figure out how come the arrays are 1-D as the error shows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;boston_dataset=load_boston()&#xA;&#xA;#create a pandas dataframe and store the data&#xA;df_boston=pd.DataFrame(boston_dataset.data)&#xA;df_boston.columns=boston_dataset.feature_names&#xA;&#xA;#append price, target as a new column in the dataset&#xA;df_boston['Price']=boston_dataset.target&#xA;&#xA;#print first five observations&#xA;df_boston.head()&#xA;&#xA;CRIM    ZN  INDUS   CHAS    NOX RM  AGE DIS RAD TAX PTRATIO B   LSTAT   Price&#xA;0   0.00632 18.0    2.31    0.0 0.538   6.575   65.2    4.0900  1.0 296.0   15.3    396.90  4.98    24.0&#xA;1   0.02731 0.0 7.07    0.0 0.469   6.421   78.9    4.9671  2.0 242.0   17.8    396.90  9.14    21.6&#xA;2   0.02729 0.0 7.07    0.0 0.469   7.185   61.1    4.9671  2.0 242.0   17.8    392.83  4.03    34.7&#xA;3   0.03237 0.0 2.18    0.0 0.458   6.998   45.8    6.0622  3.0 222.0   18.7    394.63  2.94    33.4&#xA;4   0.06905 0.0 2.18    0.0 0.458   7.147   54.2    6.0622  3.0 222.0   18.7    396.90  5.33    36.2&#xA;&#xA;&#xA;#assign features on x-axis&#xA;X_features=boston_dataset.data&#xA;&#xA;#assign target on y-axis&#xA;Y_target=boston_dataset.target&#xA;&#xA;#import linear model-the estimator&#xA;from sklearn.linear_model import LinearRegression&#xA;lineReg=LinearRegression()&#xA;&#xA;#fit data into the estimator&#xA;lineReg.fit(X_features,Y_target)&#xA;&#xA;ValueError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-16-4b5b068e587b&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 #fit data into the estimator&#xA;----&amp;gt; 2 lineReg.fit(X_features,Y_target)&#xA;&#xA;/usr/local/lib/python3.4/dist-packages/sklearn/linear_model/base.py in fit(self, X, y, sample_weight)&#xA;    480         n_jobs_ = self.n_jobs&#xA;    481         X, y = check_X_y(X, y, accept_sparse=['csr', 'csc', 'coo'],&#xA;--&amp;gt; 482                          y_numeric=True, multi_output=True)&#xA;    483 &#xA;    484         if sample_weight is not None and np.atleast_1d(sample_weight).ndim &amp;gt; 1:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41504" LastActivityDate="2022-01-31T00:56:41.957" Title="ValueError while using linear regression" Tags="|machine-learning|scikit-learn|linear-regression|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27072" PostTypeId="2" ParentId="21794" CreationDate="2018-01-26T08:33:07.337" Score="4" Body="&lt;p&gt;Very good and analytical answers are provided &lt;a href=&quot;https://stackoverflow.com/questions/45398737/is-there-any-way-to-get-samples-under-each-leaf-of-a-decision-tree&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://stackoverflow.com/questions/32506951/how-to-explore-a-decision-tree-built-using-scikit-learn&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me, the easiest way would be to find the leaves where each sample belongs and then split the dataframe into clusters using &lt;code&gt;iloc&lt;/code&gt;, to further analyse the clusters. This can be done using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.apply(iris.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.predict(iris.data) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45221" LastEditorUserId="29575" LastEditDate="2018-01-26T14:27:49.300" LastActivityDate="2018-01-26T14:27:49.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27073" PostTypeId="2" ParentId="27071" CreationDate="2018-01-26T08:38:44.423" Score="1" Body="&lt;p&gt;It seems in your code you build a pandas dataframe but you do not use it. I recreated your code line by line and was unable to get the same error. Try the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;from sklearn.linear_model import LinearRegression&#xA;boston = load_boston()&#xA;X = boston.data&#xA;Y = boston.target&#xA;&#xA;lineReg = LinearRegression()&#xA;lineReg.fit(X, Y)&#xA;lineReg.score(X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This results in an error of $0.7406$. Of course, this result is kind of meaningless because you should split your data into a training and testing set in order to accurately test your results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should do the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;from sklearn.linear_model import LinearRegression&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;boston = load_boston()&#xA;X = boston.data&#xA;Y = boston.target&#xA;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)&#xA;&#xA;lineReg = LinearRegression()&#xA;lineReg.fit(X_train, y_train)&#xA;lineReg.score(X_test, y_test )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will always give a different score because the data is shuffled before being split into your two sets. You MUST separate your training data are the start of your algorithm and do not pollute your training set with the testing set otherwise you risk biasing your model! This is very important. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-26T08:38:44.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27074" PostTypeId="2" ParentId="27055" CreationDate="2018-01-26T09:23:37.850" Score="1" Body="&lt;p&gt;You better try graph embedding and for that I propose going through &lt;a href=&quot;https://scholar.google.com/citations?user=Lksqw0EAAAAJ&quot; rel=&quot;nofollow noreferrer&quot;&gt;Horst Bunke from University of Bern&lt;/a&gt; who has been doing it for years. Just search him in google scholar and go through his publication network (&lt;a href=&quot;https://scholar.google.com/citations?user=7jAfLSkAAAAJ&amp;amp;hl=en&quot; rel=&quot;nofollow noreferrer&quot;&gt;co-authors&lt;/a&gt;,cited papers, citing papers,etc) for example &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/0167865583900338&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0167865597001797&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; as more classic papers or &lt;a href=&quot;http://www.worldscientific.com/doi/abs/10.1142/9789814304726_0006&quot; rel=&quot;nofollow noreferrer&quot;&gt;this as an exact answer to your question&lt;/a&gt;. I am pretty confident you will find your solution as it's exactly his research direction. I am on mobile so referencing is difficult now but I can update my answer later.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The other point that I would like to mention is that graph embedding has two kinds:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Embedding the vertices (nodes) of &lt;strong&gt;one&lt;/strong&gt; graph into a n-dimensional space i.e. each vector will be a node of the graph. (see &lt;a href=&quot;https://arxiv.org/abs/1705.02801&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Embedding a dataset of graphs into a n-dimensional space in which each vector will be an entire graph.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You need the second one.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The other thing worth to mention is looking at the problem from &lt;a href=&quot;http://barabasi.com/networksciencebook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Science&lt;/a&gt; point of view in which the statistical and structural of a graph is widely studied. There are plenty of graph/network measures which are less or more tolerante to the number of nodes e.g. &lt;a href=&quot;https://en.wikipedia.org/wiki/Clustering_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Clustering Coefficient&lt;/a&gt;. Using them as features can be fed to a classifier to see how it works.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The last but not the least is what you actually asked! What you are looking for is called &lt;strong&gt;Graph Kernel&lt;/strong&gt;. It is nothing but the definition of the ML-known concept Kernel but on graphs. So kernel methods can get benefit of that by skipping the feature extraction step (like what I suggested above) and directly work with structural inputs. For this purpose I would suggest that you equip yourself with the graph kernels e.g. &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume11/vishwanathan10a/vishwanathan10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-01-26T20:03:11.370" LastActivityDate="2018-01-26T20:03:11.370" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27075" PostTypeId="1" CreationDate="2018-01-26T09:56:44.590" Score="2" ViewCount="260" Body="&lt;p&gt;I am developing a neural network to determine if comments posted on a blog-type website are appropriate or not (to reject spam, poorly written comments, etc). I use Keras with Tensorflow to achieve this by defining a number of scalar features such as message length, number of words, fraction of words that are in English dictionary... All are normalized by appropriate quantities. These numbers are the input to a neural network defined as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;inp = Input(shape=(n_inputs, ))&#xA;hidden_1 = Dense(hidden_size, activation='relu')(inp)&#xA;hidden_2 = Dense(hidden_size, activation='relu')(hidden_1)&#xA;out = Dense(num_classes, activation='softmax')(hidden_2)&#xA;model = Model(inputs=inp, outputs=out)&#xA;&#xA;model.compile(loss='categorical_crossentropy',&#xA;              optimizer='adam',&#xA;              metrics=[f1_score])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I would like to add some inputs that do not always exist. For example, a comment may be posted after another comment from the same user, and in that case the delay between the comment and the previous one would be a relevant quantity to consider as a user may be spamming by sending quickly several comments in a row.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I use this quantity that does not always exist?&lt;/li&gt;&#xA;&lt;li&gt;Would it be good to use two quantities: a binary indicating if the&#xA;post is not the first one from that user, and a second number&#xA;indicating the delay from the previous comment, but filled with an&#xA;arbitrary value (0, 1, 0.5, the mean delay on all my dataset?) if&#xA;the post is the first one?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I think this is related to the problem of missing data, see &lt;a href=&quot;https://www.researchgate.net/post/Is_it_possible_to_train_a_neural_network_with_missing_data2&quot; rel=&quot;nofollow noreferrer&quot;&gt;this thread&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45222" LastEditorUserId="98307" LastEditDate="2020-08-12T12:51:01.323" LastActivityDate="2020-08-12T12:51:01.323" Title="Scalar input to neural network whose existence is conditional" Tags="|neural-network|tensorflow|keras|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27076" PostTypeId="2" ParentId="27035" CreationDate="2018-01-26T09:58:36.057" Score="2" Body="&lt;p&gt;The problem turned out to be silly, I just needed more epochs, a smaller learning rate (and for efficiency I turned to AdamOptimizer, results are now equal.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1681,)&#xA;(1681, 2)&#xA;SCI-KITLEARN RESULTS:&#xA;    Accuracy: 0.822129684711&#xA;    Precision: 0.837883361162&#xA;    Recall: 0.784522522208&#xA;    F1: 0.806251963817&#xA;&#xA;TENSORFLOW RESULTS:&#xA;    Accuracy: 0.82213&#xA;    Precision: 0.837883361162&#xA;    Recall: 0.784522522208&#xA;    F1: 0.806251963817&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42681" LastActivityDate="2018-01-26T09:58:36.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27077" PostTypeId="2" ParentId="27035" CreationDate="2018-01-26T09:59:12.213" Score="2" Body="&lt;p&gt;You are initializing your weights to zero. This will cause symmetric problem in updating weights. Use &lt;code&gt;Xavier&lt;/code&gt; or &lt;code&gt;He&lt;/code&gt; technique for initialization and train for some epochs.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-26T09:59:12.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27079" PostTypeId="2" ParentId="27057" CreationDate="2018-01-26T11:48:06.887" Score="3" Body="&lt;p&gt;There are a number of methods to automate the optimization of your hyper-parameters, such as GridSearch and RandomSearch which the article you linked discusses briefly.&lt;/p&gt;&#xA;&lt;p&gt;The main reason to choose one over the other is if you want the best possible parameters, and do not care how long it takes to get them: go for &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;GridSearch&lt;/a&gt;. On the other hand, if you do not want the optimization to take a long time, but still want some good parameters then go for &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomSearch&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;These two implementations in scikit-learn are not exactly &amp;quot;advanced packages&amp;quot;, but they will get the job done for any model in sckit-learn (Random Forest, MLPClassifiers, etc). Emre's comment also has some pretty cool advanced packages which are scikit-learn or TensorFlow compatible.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastEditorUserId="29169" LastEditDate="2021-08-12T20:56:55.597" LastActivityDate="2021-08-12T20:56:55.597" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27081" PostTypeId="1" AcceptedAnswerId="27091" CreationDate="2018-01-26T14:20:47.687" Score="3" ViewCount="582" Body="&lt;p&gt;I am working with a multi-modality classification problem (with &lt;code&gt;Keras&lt;/code&gt;). I have 1000, 5000 and 10000 samples for three different classes. I would like to do a five fold cross-validation to select the best pre-trained deep learning model for deployment. I'm including class weights during model training to give more weightage for less-pronounced classes. For a given fold, I would be validating with 200, 1000 and 2000 samples from these three classes. Is accuracy a good prediction measure to be used in this case? Or do I have to measure the F1-score and Matthews correlation coefficients as well? Am I doing it right? &lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="28175" LastEditDate="2018-01-26T15:53:12.660" LastActivityDate="2018-01-26T19:49:41.583" Title="How to cross-validate a deep learning model for highly imbalanced datasets?" Tags="|machine-learning|neural-network|deep-learning|keras|multiclass-classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27082" PostTypeId="1" CreationDate="2018-01-26T14:23:22.223" Score="3" ViewCount="74" Body="&lt;p&gt;I have been working on data science project where I am trying to build a metric for how inbred a source is in a network. We hope to apply this to intelligence reporting, in which documents tend to cite other sources without the knowledge of an inherent bias.  Think WBD Iraq, where a small footnote in a single intelligence report led to second and third order effects that allowed the piece of information to gain more ground then it should—it created an echo chamber. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before applying this to intelligence reporting, we decided to use a citation network to demonstrate capability.  We took the first 200 sources, sorted by relevance, using the keyword “network science.”  Then we built two edge list: First, we built a source edge list that has an “ID” column and a column with the source name.  Second, we built a citation edge list that has an “ID” column and a column with each sources citations.  So, we have list of 200 sources and about 5,000 citations.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our code merges the data set so that each source has each of its citations with the correct ID, and we plot it.  However, we are trying to get it to plot as a DAG.  In our pseudocode, we have tried this twice unsuccessful.  We plan to do transitive reduction after we can make a DAG to get at our point of a metric of the echo effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Two Questions:&lt;/strong&gt;&#xA;1)  Did we merge the data correctly or should we do more on the front end?&lt;br&gt;&#xA;2)  We need to work on how to plot a DAG and how we can perform transitive reduction.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Reading the Data into R&#xA;cit.citations &amp;lt;- read.csv(file.choose())&#xA;cit.sources &amp;lt;- read.csv(file.choose())&#xA;&#xA;# Merge the Data Frames&#xA;cit.total &amp;lt;- merge(cit.sources, cit.citations, by=&quot;ID&quot;)&#xA;&#xA;# List all the sources&#xA;sources.vertices &amp;lt;- cit.total$Sources&#xA;sources &amp;lt;- (as.character(sources.vertices))&#xA;&#xA;citation.edges &amp;lt;- cit.total$Citations&#xA;citations &amp;lt;- (as.character(citation.edges))&#xA;&#xA;total &amp;lt;- rbind(sources,citations)&#xA;&#xA;# Redefining the Data Frame into a Network Object&#xA;net.cit.obj &amp;lt;- graph_from_data_frame(d = cit.total, directed=TRUE)&#xA;&#xA;# Network - using dplyr package&#xA;cit.test &amp;lt;- dplyr::inner_join(cit.sources,cit.citations, by = &quot;ID&quot;)[,-2]&#xA;cit.test.network &amp;lt;- as.network(cit.test, directed = TRUE, loops = FALSE)&#xA;as.sociomatrix(cit.test.network)&#xA;as.edgelist(cit.test.network)&#xA;plot(cit.test.network)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kHSmk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kHSmk.png&quot; alt=&quot;Network with merged data&quot;&gt;&lt;/a&gt;&#xA;  Network with merged data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TNr5g.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TNr5g.png&quot; alt=&quot;This is what it looks like when we merge&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44683" LastActivityDate="2018-01-26T14:23:22.223" Title="Echo-Effect-Metric Network" Tags="|r|graphs|social-network-analysis|plotting|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27083" PostTypeId="1" CreationDate="2018-01-26T15:32:25.917" Score="1" ViewCount="67" Body="&lt;p&gt;This is probably best explained via an analogy: let’s assume we’re baking cookies because cookies are awesome! Let’s also assume we can only control three variables in our cooking process: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;time (x1)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;temperature (x2)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;humidity (x3)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There’s an obvious interaction between time and temperature, but let’s also assume there are additional interactions among the other variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our outcome variable (&lt;code&gt;y&lt;/code&gt;) is “cookie goodness” – for the sake of argument, let’s force it to be binary – we have “good cookies” and “bad cookies”.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are plenty of methods and models for classifying and understanding how to determine and predict good cookies vs. bad cookies, but let’s assume we want the best “settings” for time, temperature, and humidity. Since these are settings, we can assume they must be continuous “ranges” for their settings (for example, one possible rule for settings would be: keep time between 15-25 minutes, temperature between 350 and 375, and humidity above 50…there is no other cross section of variables that will yield more “good cookies”).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there methods or techniques for approaching a classification model with these constraints? Initially we may think tree-based models, but a decision tree will likely return several interactions and since I need one ideal “setting” for each variable, how do we know we're not missing some subset of “settings” down a different branch of the tree? &lt;/li&gt;&#xA;&lt;li&gt;Is this just an exercise of optimizing/pruning the tree parameters and selecting the best branch? &lt;/li&gt;&#xA;&lt;li&gt;Is there some method or way to force a decision tree to only include one set of continuous ranges for each variable or to collapse/summarize the output of the tree to continuous rulesets for each variable? &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Aside: This almost feels like a &lt;a href=&quot;http://www.tandfonline.com/doi/abs/10.1080/0954412022000002009&quot; rel=&quot;nofollow noreferrer&quot;&gt;reverse Design of Experiments (DOE)&lt;/a&gt;?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Practical Considerations&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then there’s the problem when the tree suggests too “tight” of a range (e.g. keep temperature between 400 and 401 – my oven’s not that accurate)? I suppose we could check branches with &lt;code&gt;high support&lt;/code&gt; and &lt;code&gt;high confidence&lt;/code&gt; and then see if they have &lt;em&gt;implementable&lt;/em&gt; ranges. I suppose &quot;best/optimal&quot; is subjective, so let's say we need to have at least 10% support (50 trials) and 75% confidence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case anyone wants to explain with some fake data, here is some &lt;code&gt;R&lt;/code&gt; code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse)&#xA;&#xA;n &amp;lt;- 500&#xA;time &amp;lt;- runif(n, 15, 40)&#xA;temp &amp;lt;- runif(n, 300, 450)&#xA;humidity &amp;lt;- runif(n, 35, 60)&#xA;&#xA;cookie_goodness &amp;lt;- (2*time + 3*time*temp + 4*humidity*temp  + 5*humidity + rnorm(n)) / 10000&#xA;&#xA;good_cookie_lower_limit &amp;lt;- quantile(cookie_goodness, 0.25)&#xA;good_cookie_upper_limit &amp;lt;- quantile(cookie_goodness, 0.75)&#xA;&#xA;df &amp;lt;- data_frame(&#xA;  trial = 1:n,&#xA;  time = time,&#xA;  temp = temp,&#xA;  humidity = humidity,&#xA;  good_cookie = as.factor(&#xA;    if_else(between(cookie_goodness, good_cookie_lower_limit, good_cookie_upper_limit),&#xA;            1, 0)&#xA;    )&#xA;)&#xA;&#xA;# # A tibble: 500 x 5&#xA;#    trial  time  temp humidity good_cookie&#xA;#    &amp;lt;int&amp;gt; &amp;lt;dbl&amp;gt; &amp;lt;dbl&amp;gt;    &amp;lt;dbl&amp;gt; &amp;lt;fctr&amp;gt;     &#xA;# 1      1  40.0   377     54.5 0          &#xA;# 2      2  39.0   384     53.4 0          &#xA;# 3      3  30.9   325     38.4 0          &#xA;# 4      4  25.5   447     35.1 1          &#xA;# 5      5  19.0   348     44.7 0          &#xA;# 6      6  37.1   431     42.7 0          &#xA;# 7      7  34.8   400     43.5 1          &#xA;# 8      8  26.2   346     36.9 0          &#xA;# 9      9  19.2   409     38.0 0          &#xA;# 10    10  38.0   408     46.9 0          &#xA;# # ... with 490 more rows&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="6458" LastActivityDate="2018-01-26T15:32:25.917" Title="Baking perfect cookies: methods or techniques for determining optimal ranges of x variables in classification models" Tags="|machine-learning|classification|r|predictive-modeling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27084" PostTypeId="1" CreationDate="2018-01-26T15:38:54.597" Score="3" ViewCount="5022" Body="&lt;p&gt;how can we find the accuracy autoencoders for classification of images? because we will get the reconstruction of the image and when we will plug in the test data it will spit out the image but how would we able to calculate the accuracy? or we should take difference between the prediction and input of the image .﻿&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2019-09-10T21:39:01.737" Title="how to find classification accuracy in autoencoders?" Tags="|machine-learning|image-classification|autoencoder|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27085" PostTypeId="2" ParentId="24319" CreationDate="2018-01-26T15:38:59.877" Score="23" Body="&lt;p&gt;I want to add one more resource, &lt;a href=&quot;https://colab.research.google.com/&quot; rel=&quot;noreferrer&quot;&gt;Google Colaboratory&lt;/a&gt;. It's a free cloud iPython notebook and gives you free usage of a GPU . I'm not sure of the exact limitations just yet, but it appears you get 12 hours of GPU time per instance and can do this multiple times per month. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This looks like a great resource for students and other non-professionals, especially for smaller jobs that you can run in half a day. It essentially saves you up to $10 per training session, which is a pretty significant resource for machine learning research in my opinion. I seriously hope it doesn't get abused. &lt;/p&gt;&#xA;" OwnerUserId="44299" LastEditorUserId="44299" LastEditDate="2019-04-27T17:50:56.077" LastActivityDate="2019-04-27T17:50:56.077" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="27086" PostTypeId="2" ParentId="27081" CreationDate="2018-01-26T15:46:55.330" Score="1" Body="&lt;p&gt;You have imbalanced data set, so you should use &lt;code&gt;F1&lt;/code&gt; score. Also you can use weight for rare classes, so that your cost function will be formed in a way that it cares about rare classes so much and tries to classify them correctly. You also can use confusion matrix for the details, but &lt;code&gt;F1&lt;/code&gt; will suffice. And yes, use &lt;code&gt;F1&lt;/code&gt; instead of precision or recall. You can also take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26730/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-26T19:00:03.837" LastActivityDate="2018-01-26T19:00:03.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27087" PostTypeId="2" ParentId="26789" CreationDate="2018-01-26T16:35:42.387" Score="2" Body="&lt;p&gt;If in case 1 there would be one template for organizing information on CVs then you can go for regex, but in order to have really helpful tool for real world CV you have to train ML solution. Just type in google &quot;cv example&quot; and you will see that people use different words and have different order of information and different formatting (formatting could be eliminated from problem scope). You can use regex but it will be enormous and I doubt that you will outperform some NER solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case 2 it's common chat bot case so it's question of whether on not use ML to implement chat bot (spoiler - use ML)&lt;/p&gt;&#xA;" OwnerUserId="45177" LastEditorUserId="29575" LastEditDate="2018-01-26T17:38:32.967" LastActivityDate="2018-01-26T17:38:32.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27088" PostTypeId="1" CreationDate="2018-01-26T18:41:47.137" Score="0" ViewCount="145" Body="&lt;p&gt;I'm studying Logistic Regression for the first time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My book explains how to fit the weights $w$ using the likelihood function $L(w)$, assuming that &lt;strong&gt;the individual samples in our dataset are&#xA;independent of one another&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know what independent events are. But I don't understand the meaning of &lt;strong&gt;the individual samples in our dataset are&#xA;independent of one another&lt;/strong&gt; in the Logistic Regression case. What does this mean? And why?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="29575" LastEditDate="2018-01-26T20:07:13.030" LastActivityDate="2018-02-27T21:01:57.103" Title="Logistic Regression Independent Samples" Tags="|machine-learning|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27089" PostTypeId="2" ParentId="27088" CreationDate="2018-01-26T19:25:51.030" Score="2" Body="&lt;p&gt;Context / big picture:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two events are independent if they have no influence on each other's outcomes. For example, if event A is &quot;I go get coffee&quot; and event B is &quot;it's raining outside&quot;, then events A and B are independent, because I'm a coffee fiend and I don't care whether it's raining - I'm getting that coffee anyway!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic regression example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have blood pressure samples from 50 different individuals at a hospital. You want to classify these as high or low blood pressure. In this example, your X (input variable) is the blood pressure from the 50 individuals, and your Y (binary response) is yes/no to the question 'is this blood pressure high?'. Each person's blood pressure is independent (if we assume they are unrelated strangers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were to sample the same person's blood pressure twice, like before and after physical exercise, then you would no longer have independent samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need independent samples in logistic regression because otherwise the degrees of freedom of the model are not what we expect them to be, and this affects further calculations. For example, in your dataset, you have 50 independent samples. You use 2 to estimate your intercept and slope, which leaves you with 48 degrees of freedom to work with. If you didn't have independent samples, then you would not have 48 degrees of freedom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: a bit more detail on why independent samples are important.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assumption of independence ensures that each sample contributes the same amount of information to the experiment. With independent samples, this is the case. A blood pressure measurement from one individual is not dependent on the blood pressure of another individual - they're independent. However, if you measure the same person's blood pressure before and after exercise, a different amount of information is added. You can no longer ask the same questions as you could with independent samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's pick a simpler model to describe what's going on. We will follow standard hypothesis testing principles to illustrate the practical difference between independent and dependent samples. I will round to two decimal places for ease of reading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have two groups of people: those with blood type A and those with blood type O. (Aside: notice people cannot have both blood types at once). We want to investigate whether those with blood type A have a higher resting heart rate than those with blood type O. We have 12 people with blood type A and 10 people with blood type O. To answer this question, we'll conduct a two-sample hypothesis test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Null hypothesis: The mean resting heart rate of those with blood type A is the same as the mean resting heart rate of those with blood type O &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternate hypothesis: The mean resting heart rate of those with blood type A is different from the mean resting heart rate of those with blood type O.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We measure each person's resting heart rate, and we get the following results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type A: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Type O: 61, 64, 70, 72, 59, 62, 66, 78, 69, 75&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We then calculate the sample mean and sample standard deviation of each group:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;mean(Type A) = 69.67; standard deviation(Type A) = 8.35 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;mean(Type B) = 67.6; standard deviation(type B) = 6.28 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can then calculate a test statistic, which will follow a t-distribution. We may decide to use either Welch's method or a pooled variance test - in this case, I'll show Welch's method, which doesn't require the population variances to be equal (unlike the pooled variance test).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test statistic is calculated via:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;standard error: sqrt((8.35^2/12) + (6.28^2/12)) = 3.02&lt;/p&gt;&#xA;&#xA;&lt;p&gt;test statistic = (69.67 - 67.6)/3.02 = 0.69&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The degrees of freedom here are difficult to calculate, but a generally accepted approximation is &quot;the lesser of the degrees of freedom in the two groups&quot;, that is min(df_1, df_2). In this case the degrees of freedom of the first group is 12 - 1 = 11 and the degrees of freedom of the second group is 10 - 1 = 9. We subtract 1 here from each sample size, because we have estimated one parameter for each group (the sample mean). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we wanted to continue this example, we could look at a t-distribution on 9 degrees of freedom and decide based on the test statistic whether to reject or not reject the null hypothesis. In this case, we're more interested in the degrees of freedom than the result, so we'll move on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dependent samples&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we have a that same group of people who all have blood type A. Instead of measuring their resting heart rate just once, we’ll measure each Type A person’s resting heart rate twice. This introduces dependence between the samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Null hypothesis: the mean resting heart rate of people with blood type A is the same as the mean resting heart rate of all blood types&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternative hypothesis: the mean resting heart rate of people with blood type A is different from the mean resting heart rate of all blood types&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We measure each person's resting heart rate before exercise, and we get the following results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we measure their resting heart rate after exercise, and we get the following results (notice that person 1 in the previous list of heart rates is the same as person 1 in this list):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93, 84, 85&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have two observations for each individual. We can’t ask the same question, unless we want to throw out half of our data! Instead, we’ll ask whether there is a difference in the resting heart rate before and after exercise. We’ll use a paired-difference t-test. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To conduct this test, we have to take both observations from each individual into account. We calculate the differences between each person’s before and after measurement, and treat this set of differences as our actual data set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;60-90, 65-85,70-70, 62-72, …&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then we can calculate a sample mean, a sample standard deviation, etc. Our degrees of freedom would be 12 -1 = 11. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What would have happened if we had assumed that those dependent samples were independent?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that when we had dependent samples, our degrees of freedom was 11; when we had independent samples, our degrees of freedom was 9. If we had assumed that the dependent samples were in fact two independent samples, then we could have used the wrong degrees of freedom. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;** What if we had had only 10 measurements instead of 12 in the ‘dependent samples’ section? Wouldn’t we have had the same degrees of freedom? **&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, but it still wouldn’t have meant the same thing. We would have had:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we would have had to ‘throw away’ the last two ‘Before exercises’ measurements and use only the first ten, as the last two had no partner measurement. The meaning has still changed, though. With dependent samples, you cannot ask the same kinds of questions as with independent samples.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2018-01-28T20:43:25.360" LastActivityDate="2018-01-28T20:43:25.360" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27090" PostTypeId="1" CreationDate="2018-01-26T19:34:25.207" Score="1" ViewCount="58" Body="&lt;p&gt;Are there any available resources that contain data on LaTeX symbols usage metrics. I want to know what the most used symbols are to determine which I should account for in a project I'm working on.&lt;/p&gt;&#xA;" OwnerUserId="43048" LastActivityDate="2018-01-26T19:34:25.207" Title="LaTex Symbol Usage Statistics" Tags="|data|" AnswerCount="0" CommentCount="2" ClosedDate="2020-12-07T09:57:53.923" ContentLicense="CC BY-SA 3.0" />
  <row Id="27091" PostTypeId="2" ParentId="27081" CreationDate="2018-01-26T19:49:41.583" Score="3" Body="&lt;p&gt;Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include &lt;code&gt;roc_auc_score&lt;/code&gt; (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) and &lt;code&gt;average_precision_score&lt;/code&gt; (see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;), both defined through scikit-learn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using Keras, I would recommend using &lt;code&gt;class_weights&lt;/code&gt; (note this will not work well if you have a multi-label problem, although there are some workarounds, for example &lt;a href=&quot;https://stackoverflow.com/questions/48254832/keras-class-weight-in-multi-label-binary-classification&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-01-26T19:49:41.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27092" PostTypeId="1" AcceptedAnswerId="27097" CreationDate="2018-01-26T21:40:05.100" Score="3" ViewCount="331" Body="&lt;p&gt;In multiple linear regression there is an &lt;code&gt;F-test&lt;/code&gt; which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the &lt;code&gt;F-statistic&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="41894" LastEditorUserId="28175" LastEditDate="2018-01-27T04:45:33.207" LastActivityDate="2018-01-27T15:59:28.057" Title="What are some methodologies for performing feature selection for simple feed-forward neural networks?" Tags="|machine-learning|neural-network|deep-learning|feature-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27093" PostTypeId="2" ParentId="26969" CreationDate="2018-01-26T22:08:19.007" Score="1" Body="&lt;p&gt;You need to use linear regressor instead of logistic regressor. The difference between the two is, the latter is used to &quot;classify&quot; things whereas the former provides a continuous output as you require.&#xA;In order to get &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;...&quot;one continuous variable&quot;...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;you need to use for example KerasRegressor if you are using Keras. Similar functions exist in other systems like scikit-learn, caffe etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;great example&lt;/a&gt; (written in Keras), that describes how to predict house prices (a continuous value) using Keras machine learning library.&#xA;Here is it for &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;, and here is a &lt;a href=&quot;https://caffe2.ai/docs/tutorials.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial on linear regression using caffe&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this &lt;a href=&quot;https://stackoverflow.com/questions/12146914/what-is-the-difference-between-linear-regression-and-logistic-regression&quot;&gt;StackOverflow post&lt;/a&gt; for in-depth analysis of the difference between linear and logistic regression.&lt;/p&gt;&#xA;" OwnerUserId="10383" LastActivityDate="2018-01-26T22:08:19.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27094" PostTypeId="2" ParentId="27061" CreationDate="2018-01-26T22:21:06.667" Score="0" Body="&lt;p&gt;Define &quot;shared&quot;. Don't assume cluster 1 in A is cluster 1 in B. The numbers are meaningless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So yes, you should use standard techniques like adjusted Rand index.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-26T22:21:06.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27095" PostTypeId="2" ParentId="27084" CreationDate="2018-01-26T22:29:54.810" Score="3" Body="&lt;p&gt;Are you using the autoencoder for classification or reconstruction?&#xA;if you are pre-training the autoencoder for classification then you use the usual logloss to determine the accuracy of your classifier. If on the otherhand, if you are using the autoencoder for reconstruction, then there is no classification, and you can use something like KL divergence (&lt;a href=&quot;http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/&quot; rel=&quot;nofollow noreferrer&quot;&gt;described in detail here&lt;/a&gt;) to measure and track your reconstruction and compression performance.&lt;/p&gt;&#xA;" OwnerUserId="10383" LastActivityDate="2018-01-26T22:29:54.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27096" PostTypeId="2" ParentId="27056" CreationDate="2018-01-26T22:47:16.097" Score="2" Body="&lt;p&gt;to address your question about:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It is not clear that why Conv1(3)-&gt;Conv1(3)-&gt;Pool1(2) is better than Conv1(3)-&gt;Pool1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;..lot of neural network design is a trial and error process and more art than science. So one experiments with your suggestion of Conv1(3)-&gt;Pool1 and fails to get the accuracy needed and experiments some more to end up with Conv1(3)-&gt;Conv1(3)-&gt;Pool1(2) that gives them the accuracy that they need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is seldom a real scientific explanation of why Conv1(3)-&gt;Conv1(3)-&gt;Pool1(2) is better than Conv1(3)-&gt;Pool1, but generally speaking, we need as many weights as possible to capture all the variability in data as possible, but not more (otherwise we overfit.)&lt;/p&gt;&#xA;" OwnerUserId="10383" LastActivityDate="2018-01-26T22:47:16.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27097" PostTypeId="2" ParentId="27092" CreationDate="2018-01-27T04:44:57.487" Score="2" Body="&lt;p&gt;I usually try to do the following process, I don't know whether it has name or not:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try to find features which an expert can say what the label is without hesitating. &lt;/li&gt;&#xA;&lt;li&gt;Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply &lt;code&gt;PCA&lt;/code&gt;. Consider the point that &lt;code&gt;PCA&lt;/code&gt; alone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-27T04:44:57.487" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27100" PostTypeId="1" CreationDate="2018-01-27T10:42:18.000" Score="-1" ViewCount="494" Body="&lt;p&gt;&lt;strong&gt;The context of the question:&lt;/strong&gt;&#xA;I have a pandas dataframe where one column has text values and others have categorical values. I trained a &lt;code&gt;word2vec&lt;/code&gt; model with tensorflow with some sample data. And I convert my text column into the vector representation. But I want to feed these data to &lt;a href=&quot;https://github.com/catboost/catboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;Catboost&lt;/a&gt; for regression task. But I can't feed those data to catboost because of catboost only accept the &lt;code&gt;Categorical&lt;/code&gt; variable. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I found a tutorial on Catboost &lt;a href=&quot;https://github.com/catboost/catboost/blob/master/catboost/tutorials/quora_catboost_w2v.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github page&lt;/a&gt; for this. But this&#xA;  example is for the classification task. It finds &lt;code&gt;cosine&lt;/code&gt; and other&#xA;  types of relationship between two vector representation of text. But&#xA;  in my case, I have only one text field. So how can calculate &lt;code&gt;cosine&lt;/code&gt;&#xA;  or other types relationship?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So my question is how to extract categorical features from the vector representation of text data?&lt;/p&gt;&#xA;" OwnerUserId="45263" LastActivityDate="2018-06-27T09:08:59.513" Title="How to find categorical features from a vector representation of text?" Tags="|machine-learning|nlp|feature-extraction|word2vec|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27102" PostTypeId="1" CreationDate="2018-01-27T10:53:25.670" Score="1" ViewCount="183" Body="&lt;p&gt;I have a daily set of data that I need to detect if there are any anomalies in it. These are variables related to the stock market so I'd just give an example (totally fictitious) here:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date     Type  Notional Underlying Expiry Market_Value DV01 X    Y    Z&#xA;1/1/2018 Bonds 100,000  USTreasury Mar18  100,000      2.5  2.5  5.0 15.0&#xA;1/2/2018 Bonds 150,000  USTreasury Mar18  149,000      4.0  4.0  8.0 12.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I'm trying to say is that there're some unknown correlations a,b,c between DV01 and parameters X, Y and Z. However I'm not entirely sure what the relation is, for simplicity sake I'm just letting a=1, b=2, c=3. For other type of instruments such as Options, the correlations will be entirely different. What I hope to achieve here is to be able to pick out values which do not conform to these relations. If the following data comes in the next day, the 50.0 in Z should be flagged.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date     Type  Notional Underlying Expiry Market_Value DV01 X    Y    Z&#xA;1/3/2018 Bonds 200,000  USTreasury Mar18  200,500      5.5  5.5  11.0 50.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately a,b,c are not constants; they change with time. The model should be able to learn from the past, say 10 days, and establish what type of relation it is and check the next day if there're any anomalies. What would be a good way to start?&lt;/p&gt;&#xA;" OwnerUserId="45261" LastActivityDate="2018-01-27T13:44:29.797" Title="Detecting anomalies from time series correlated variables" Tags="|machine-learning|python|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27104" PostTypeId="1" CreationDate="2018-01-27T12:31:58.763" Score="1" ViewCount="127" Body="&lt;p&gt;I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ie multiple inputs to multiple outputs (continuous values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some experience using mllib for single value predictions in Scala but I can't see that multiple outputs are supported. It doesn't even look to me that mllib has continuous value output support as I can't see how to get a layer without an activation function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know of an ml library that can both support multidimensional regression (please correct me if I have the terminology wrong) and integrate with Scala?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going forward I'd also love to try an RNN as our data has a time sequence element to it, but I believe this is even rarer in ml libraries.&lt;/p&gt;&#xA;" OwnerUserId="45267" LastEditorUserId="29575" LastEditDate="2018-01-27T13:43:24.380" LastActivityDate="2023-12-17T11:03:40.517" Title="Multidimensional regression in Scala" Tags="|machine-learning|neural-network|scala|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27105" PostTypeId="1" AcceptedAnswerId="27134" CreationDate="2018-01-27T12:49:50.810" Score="2" ViewCount="626" Body="&lt;p&gt;I'm experimenting with using an A3C network to learn to play old Atari video games. My network outputs a set of probabilities for each possible action (e.g. left, right, shoot), and I use this information to determine which action to take in the game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I have gotten to thinking how would one go about playing a game with non-binary actions. For example, steering a car left or right with a wheel instead of keyboard keys. I thought about simply translating probabilities into intensities (e.g. if I have values of 1.0/0.0 for left right, then make the hardest left turn possible, but make a much more gradual turn if my values are 0.6/0.4) but I'm not sure if this makes sense mathematically, or even in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a standard approach to doing this?&lt;/p&gt;&#xA;" OwnerUserId="45269" LastActivityDate="2018-01-28T18:16:13.273" Title="A3C - Turning action probabilities into intensities" Tags="|machine-learning|reinforcement-learning|probability|actor-critic|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27107" PostTypeId="2" ParentId="27105" CreationDate="2018-01-27T15:12:39.993" Score="1" Body="&lt;p&gt;The standard approach with policy gradients for continuous action spaces is to output a vector of parameters to a probability distribution. To resolve the policy into an action for the agent, you then sample from the distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In policy gradients with discrete action spaces, this is actually already the case - the softmax layer is providing a discrete distribution for you, that you must sample from to choose the action.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general rule is that your probability distribution function needs to be differentiable. A common choice is the Normal distribution, and for the output vector to be the mean and standard deviation. This adds an extra &quot;interpretation layer&quot; to the agent's model in addition to the NN, which needs to be included in the gradient calculation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your idea:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;e.g. if I have values of 1.0/0.0 for left right, then make the hardest left turn possible, but make a much more gradual turn if my values are 0.6/0.4&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;. . . is almost there. However, you need to interpret the output values &lt;em&gt;stochastically&lt;/em&gt;, not deterministically, in order to use policy gradients. A deterministic output based on your parameters has no gradient wrt to improvements to the policy, so the policy cannot be adjusted*. Another way to think of this is that policy gradient methods must have exploration built into the policy function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be quite difficult to turn the left/right outputs you have into a PDF that could be made progressively tighter around the optimal value as the agent homed in on the best actions, so I would instead suggest the common mean, standard deviation split for this, and have the environment cut off the actions at min/max steer if the sampled action ended up as e.g. hard left times 1.7&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Actually this is incorrect as pointed out in Constantinos' answer. There are &lt;a href=&quot;https://deepmind.com/research/publications/deterministic-policy-gradient-algorithms/&quot; rel=&quot;nofollow noreferrer&quot;&gt;deterministic policy gradient solvers&lt;/a&gt;, and sometimes they are better. They work by learning off-policy. Your network could simply output a steering direction -1.0 to 1.0, but you would &lt;em&gt;also&lt;/em&gt; need a behaviour policy which added some randomness to this output in order to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also &lt;em&gt;think&lt;/em&gt; you would need to switch from A3C to A2C in order to take advantage of deterministic policy gradient solvers.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-28T08:46:10.470" LastActivityDate="2018-01-28T08:46:10.470" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27108" PostTypeId="2" ParentId="27092" CreationDate="2018-01-27T15:59:28.057" Score="1" Body="&lt;p&gt;In addition to what was suggested by @Media, you may consider adding a softmax layer to your model right above your input layer. This is a way of visualizing which features are strongly associated with each response. Of course as you say, an F-test will not really be appropriate. This is an active area of research as far as I know.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-01-27T15:59:28.057" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27109" PostTypeId="2" ParentId="27100" CreationDate="2018-01-27T16:43:40.430" Score="0" Body="&lt;p&gt;You could try a clustering algorithm on the vector representation of your text data to see what clusters exist. Then once you have discovered the unique clusters, assign a cluster identity (for example, a single digit value indicating cluster identity (1, 2, 3, 4 etc)). Then you can use Keras's &lt;code&gt;to_categorical&lt;/code&gt; function to convert the cluster identity to categorical values.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-01-27T16:43:40.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27111" PostTypeId="2" ParentId="26789" CreationDate="2018-01-27T17:24:55.007" Score="2" Body="&lt;p&gt;Not every Tuesday is lunch day even if you are talking about some lunch on some Tuesday. It should be a NLP system using NER to provide the best prediction. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-01-27T17:24:55.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27112" PostTypeId="2" ParentId="26618" CreationDate="2018-01-27T17:41:50.607" Score="2" Body="&lt;p&gt;You need to increase no of hidden units. The more the number, more extract feature your network builds. Be wise with it, as too many features will make it overfit and it won't generalize on new data.&lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-01-27T17:41:50.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27113" PostTypeId="1" AcceptedAnswerId="27115" CreationDate="2018-01-27T17:47:21.037" Score="2" ViewCount="2715" Body="&lt;p&gt;Seeking clarity on single class object detection model using ML. I have prepared a custom database for this purpose up to 400 images which is split in 80%-20% as training and testing data-set. These are top view only images. The data collection followed the basic guidelines provided at &lt;a href=&quot;https://stackoverflow.com/questions/36144993/data-augmentation-techniques-for-small-image-datasets&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zxVY3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zxVY3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective now is to detect the zebra crossing in below contexts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/W7kwf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W7kwf.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model is failing in terms of accuracy. Although it identifies the class but fails to localize correctly in some contexts(greater extent).The desirable result is in blue whereas the model throws up red.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/wTF5W.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wTF5W.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What changes are required to training dataset to rectify this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT1:-&lt;/strong&gt;&#xA;Tensorflow object detection API is used for this task. Detection accuracy attained is above 90%. Looking forward for suggestions to fix Localisation issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT2:-&lt;/strong&gt;&#xA;The illustrations here are only for outlining the issue.All my images are real world pics.&lt;/p&gt;&#xA;" OwnerUserId="35503" LastEditorUserId="35503" LastEditDate="2018-01-29T02:27:20.367" LastActivityDate="2018-02-28T02:46:16.007" Title="Preparing custom dataset for object detection using ML" Tags="|machine-learning|neural-network|deep-learning|tensorflow|object-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27114" PostTypeId="1" CreationDate="2018-01-27T18:28:05.983" Score="1" ViewCount="28" Body="&lt;p&gt;I want to upgrade my GTX 650ti 1GB for something better suited for deep learning. I was offered a Titan Black 6 GB for US 270. I know there are several better cards today, but my budget limit is US$ 300.&#xA;Any recommendations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gonzalo&lt;/p&gt;&#xA;" OwnerUserId="45278" LastActivityDate="2018-01-27T18:28:05.983" Title="Titan Black 6GB for US$ 270 - good deal for DL?" Tags="|deep-learning|" AnswerCount="0" CommentCount="1" ClosedDate="2018-01-28T21:41:48.223" ContentLicense="CC BY-SA 3.0" />
  <row Id="27115" PostTypeId="2" ParentId="27113" CreationDate="2018-01-27T18:33:23.770" Score="2" Body="&lt;p&gt;I have the following suggestion:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The size of your data-set is small, you should increase the sample size. You also have to use data augmentation methods for improving the performance, translation, orientation and blurring are highly recommended. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For your task I guess the most popular approach is what which is discussed in &lt;code&gt;YOLO&lt;/code&gt; paper, and it is highly recommended using that if you want to have object detection not just localization and recognition for good performance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I also suggest you providing real data for your task. The images of first row are artificially made. You have to provide real data which your classifier is going to face. The distribution of your test time should be like validation and train time if you want to have good performance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-27T18:33:23.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27116" PostTypeId="1" CreationDate="2018-01-27T19:12:30.593" Score="5" ViewCount="45567" Body="&lt;p&gt;I am a Python-Newbie and want to plot a list of values between -0.2 and 0.2. The list looks like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[...-0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01501152092971969,&#xA;  -0.01489985147131656,&#xA;  -0.015833709930856088,&#xA;  -0.015833709930856088,&#xA;  -0.015833709930856088,&#xA;  -0.015833709930856088,&#xA;  -0.015833709930856088...and so on].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In statistics I've learned to group my data into classes to get a useful plot for a histogram, which depends on such large data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I add classes in python to my plot? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.hist(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and histogram looks like&#xA;&lt;a href=&quot;https://i.stack.imgur.com/GKhrP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GKhrP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But it should look like &#xA;&lt;a href=&quot;https://i.stack.imgur.com/k5rwR.png&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Just an example - scales are different!&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k5rwR.png&quot; alt=&quot;enter image description here&quot; title=&quot;Just an example - scales are different!&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45282" LastEditorUserId="1330" LastEditDate="2022-09-07T16:25:42.347" LastActivityDate="2022-09-07T16:25:42.347" Title="Histogram plot with plt.hist()" Tags="|python|histogram|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27117" PostTypeId="2" ParentId="27116" CreationDate="2018-01-27T19:28:01.770" Score="2" Body="&lt;p&gt;You have to specify the bin size, if I've figured out the question. As stated &lt;a href=&quot;https://stackoverflow.com/a/12176344/5120235&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;You can give a list with the bin boundaries.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.hist(data, bins=[0, 10, 20, 30, 40, 50, 100])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you just want them equally distributed, you can simply use range:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.hist(data, bins=range(min(data), max(data) + binwidth, binwidth))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also take a look at &lt;a href=&quot;https://matplotlib.org/devdocs/api/_as_gen/matplotlib.pyplot.hist.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://matplotlib.org/examples/statistics/histogram_demo_histtypes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-27T19:28:01.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27119" PostTypeId="2" ParentId="26980" CreationDate="2018-01-27T20:37:56.017" Score="0" Body="&lt;p&gt;Yes, check the log-loss distribution as the number of iterations increases. If it start's shooting up before your final boosting iteration then it's over-fitting.&lt;/p&gt;&#xA;" OwnerUserId="45182" LastActivityDate="2018-01-27T20:37:56.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27120" PostTypeId="2" ParentId="25092" CreationDate="2018-01-27T20:50:48.940" Score="1" Body="&lt;p&gt;Make sure to pass the model an &amp;quot;objective&amp;quot; parameter and also use &amp;quot;rmse&amp;quot; for the &amp;quot;eval_metric&amp;quot; parameter.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45182" LastEditorUserId="45182" LastEditDate="2021-01-24T21:22:54.167" LastActivityDate="2021-01-24T21:22:54.167" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27122" PostTypeId="1" CreationDate="2018-01-27T23:44:36.963" Score="-1" ViewCount="132" Body="&lt;p&gt;The Prototype Widget &lt;a href=&quot;https://github.com/biolab/orange3-prototypes/blob/master/orangecontrib/prototypes/widgets/owsignificantgroups.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Significant Groups&quot;&lt;/a&gt; would be a great help in hypothesis testing. &lt;strong&gt;Has anyone gotten it running???&lt;/strong&gt; I have tried just about everything. Whenever I try to &quot;Compute&quot;, I get the pictured error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iQYz4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iQYz4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/v39S8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/v39S8.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45290" LastEditorUserId="29575" LastEditDate="2018-01-27T23:48:53.727" LastActivityDate="2018-04-30T02:08:49.933" Title="Has anyone gotten the Orange3 Prototype Widget &quot;Significant Groups&quot; to work?" Tags="|orange|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27123" PostTypeId="2" ParentId="22045" CreationDate="2018-01-28T00:10:24.843" Score="0" Body="&lt;p&gt;I have run into this issue when attempting to view temporal data on the scatterplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps Excel is outputting a &lt;a href=&quot;https://support.office.com/en-us/article/DATEVALUE-function-df8b07d4-7761-4a93-bc33-b7471bbff252&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;DATEVALUE&lt;/code&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried some Excel magic to create columns with Text conforming to the ISO 8601 standard -&gt; This looked promising, but data in these columns could not be assigned as a datetime variable in orange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end I got &lt;a href=&quot;https://support.office.com/en-us/article/YEAR-function-c64f017a-1354-490d-981f-578e8ec8d3b9&quot; rel=&quot;nofollow noreferrer&quot;&gt;around the issue&lt;/a&gt; by adding columns using the &quot;year&quot; and &quot;month&quot; functions in Excel.&lt;/p&gt;&#xA;" OwnerUserId="45292" LastEditorUserId="29575" LastEditDate="2018-01-28T00:13:33.800" LastActivityDate="2018-01-28T00:13:33.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27124" PostTypeId="1" CreationDate="2018-01-28T00:24:08.590" Score="7" ViewCount="1858" Body="&lt;p&gt;I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to the NN, but I'm not sure how to structure the NN itself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have made a NN with convolutions that can do a pretty decent job with a single series, but I'd like to exploit cross-series correlations. What topology works to let the NN notice correlations between the time series?&lt;/p&gt;&#xA;" OwnerUserId="39081" LastActivityDate="2019-04-30T03:02:50.520" Title="which neural network topology to learn correlations between time series?" Tags="|machine-learning|neural-network|time-series|correlation|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27125" PostTypeId="2" ParentId="27124" CreationDate="2018-01-28T00:55:09.407" Score="2" Body="&lt;p&gt;It depends a little on what kind of correlations you're looking for. Are you expecting a correlation that is present at each time step/window, or a different level of correlation per time step/window? Are you doing a classification or a regression task? Sometimes predicting the next value involves classification, but I'll assume you're looking for regression for now.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a starting point, try feeding each of those sequences separately as input to a recurrent neural network (start with a basic LSTM and pare it down if it's overkill). I like your suggestion of overlapping windows. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example (pseudocode-ish):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;series_1 = 1, 2, 3, ..., 100&#xA;&#xA;series_2 = 5, 6, 7, ..., 200&#xA;&#xA;input_1 = Input(series_1,       window_size)&#xA;&#xA;input_2 = Input(series_2, window_size)&#xA;&#xA;layer_1 = LSTM(input_1, input_2)&#xA;&#xA;final_layer = fully_connected(layer_1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="35644" LastEditDate="2018-03-29T13:06:16.670" LastActivityDate="2018-03-29T13:06:16.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27126" PostTypeId="1" CreationDate="2018-01-28T02:19:24.000" Score="1" ViewCount="1494" Body="&lt;p&gt;Are there any differences between &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;get_dummies&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;labelbinarizer&lt;/a&gt; in terms of what they want to achieve? It seems to be both will somehow do a one-hot encoding.&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="7812" LastEditDate="2018-01-28T02:57:45.203" LastActivityDate="2018-06-13T05:34:53.220" Title="Differences of get_dummies and labelbinarizer?" Tags="|feature-engineering|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27129" PostTypeId="1" AcceptedAnswerId="27131" CreationDate="2018-01-28T02:59:28.860" Score="4" ViewCount="2344" Body="&lt;p&gt;I'm using &lt;code&gt;Scikit Learn&lt;/code&gt; to guess the tag of Stack Overflow posts given the title and body. I represent the title and body as two 300-dimensional vectors of floats. The documentation for Scikit learn's &lt;code&gt;SGDClassifier&lt;/code&gt; states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For best results using the default learning rate schedule, the data should have zero mean and unit variance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;With one-dimensional data, I understand what mean and variance are but when we go into higher dimensions how would one calculate the mean and variance for a dataset?&lt;/p&gt;&#xA;" OwnerUserId="45289" LastEditorUserId="28175" LastEditDate="2018-01-28T05:54:15.987" LastActivityDate="2018-01-28T05:54:15.987" Title="What do mean and variance mean for high dimensional data?" Tags="|machine-learning|neural-network|deep-learning|scikit-learn|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27130" PostTypeId="1" CreationDate="2018-01-28T03:00:32.660" Score="4" ViewCount="275" Body="&lt;p&gt;There is one behavior of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;labelbinarizer&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import preprocessing&#xA;lb = preprocessing.LabelBinarizer()&#xA;lb.fit(np.array([[0, 1, 1], [1, 0, 0]]))&#xA;lb.classes_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output is &lt;code&gt;array([0, 1, 2])&lt;/code&gt;. Why there is a 2 there?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="7812" LastEditDate="2018-01-28T03:06:56.993" LastActivityDate="2018-12-01T23:40:27.253" Title="2D matrix for labelbinarizer" Tags="|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27131" PostTypeId="2" ParentId="27129" CreationDate="2018-01-28T04:49:59.457" Score="2" Body="&lt;p&gt;It's just the expansion of one dimensional mean and standard deviation. Suppose that you are trying to estimate the weight of a person and you have two inputs, salary and height. For finding the weight, you have two inputs which are of different scales, so you try to find the mean and variance of each feature, salary and height, separately using the data samples. Suppose you have two data samples in a tuple like (salary, height). They are as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(10000, 180)&lt;br /&gt;&#xA;  (5000, 175)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Although the first guy may seem so rich, the real point about the first and second data samples is that their scale is not the same. The approach is like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Find the mean of first feature among data samples, salary here. &lt;code&gt;(10000 + 5000) / 2 = 7500&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Find the mean of second feature among data samples, height here. &lt;code&gt;(180 + 175) / 2 = 177.5&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Find the standard deviation of first feature, among data samples, which is &lt;code&gt;2500&lt;/code&gt; for salary.&lt;/li&gt;&#xA;&lt;li&gt;Find the standard deviation of second feature, among data samples, which is &lt;code&gt;2.5&lt;/code&gt; for height. &lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reduce the amount of mean from corresponding feature for each data sample and divide each with the standard deviation of corresponding feature.&lt;br /&gt;&#xA;&lt;code&gt;(10000 - 7500) / 2500 = 1&lt;/code&gt;&lt;br /&gt;&#xA;&lt;code&gt;(5000 - 7500) / 2500 = -1&lt;/code&gt;&lt;br /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;(180 - 177.5) / 2.5 = 1&lt;/code&gt;&lt;br /&gt;&#xA;&lt;code&gt;(175 - 177.5) / 2.5 = -1&lt;/code&gt;&lt;br /&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Data samples which have been normalized now are like as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(1, 1)&lt;br /&gt;&#xA;  (-1, -1)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Whenever you normalize your data, your cost function would be so easier to learn. The weights don't have to struggle to reach to high values in situations where your features are not in same range.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-28T05:44:34.647" LastActivityDate="2018-01-28T05:44:34.647" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27132" PostTypeId="1" AcceptedAnswerId="27137" CreationDate="2018-01-28T05:12:10.113" Score="5" ViewCount="3665" Body="&lt;p&gt;I am creating decision trees modeling data that looks like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pelvic_radius   degree_spondylolisthesis    class   &#xA;82.45603817    41.6854736      Abnormal    &#xA;114.365845     -0.421010392    Normal&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When finished, I run my test data through my tree and compare the outputs from the run to the ones I am given. This will allow me to check my accuracy, precision, and recall values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TP = 0; % True Positives&#xA;TN = 0; % True Negatives&#xA;FP = 0; % False Positives&#xA;FN = 0; % False Negatives&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then once those values are calculated, I can calculate the following.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;precision = TP/(TP+FP);&#xA;accuracy = (TP+TN)/(TP+TN+FP+FN);&#xA;recall = TP/(TP+FN);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, this can be done in two ways. One considering the 'Normal' class as Positive and one Considering the 'Abnormal' class as positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the sudo code to further explain what I mean.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for k=1:length(resultsOfTestSet)&#xA;    if(strcmp(resultsOfTestSet{k},'Normal'))&#xA;        if (strcmp(testSet{k}, 'Normal'))&#xA;            % TRUE POSITIVE&#xA;            TP = TP + 1;&#xA;        else&#xA;            % FALSE POSITIVE&#xA;            FP = FP + 1;&#xA;        end&#xA;    elseif(strcmp(resultsOfTestSet{k},'Abnormal'))&#xA;        if(strcmp(testSet{k},'Abnormal'))&#xA;            % TRUE NEGATIVE&#xA;            TN = TN + 1;&#xA;        else&#xA;            % FALSE NEGATIVE&#xA;            FN = FN + 1;&#xA;        end&#xA;    end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The above case assumes Normal as the 'Positive' resultant class. However by just flipping the compare statements, I can get alternate values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for k=1:length(resultsOfTestSet)&#xA;    if(strcmp(resultsOfTestSet{k},'Abnormal'))&#xA;        if (strcmp(testSet{k}, 'Abnormal'))&#xA;            % TRUE POSITIVE&#xA;            TP = TP + 1;&#xA;        else&#xA;            % FALSE POSITIVE&#xA;            FP = FP + 1;&#xA;        end&#xA;    elseif(strcmp(resultsOfTestSet{k},'Normal'))&#xA;        if(strcmp(testSet{k},'Normal'))&#xA;            % TRUE NEGATIVE&#xA;            TN = TN + 1;&#xA;        else&#xA;            % FALSE NEGATIVE&#xA;            FN = FN + 1;&#xA;        end&#xA;    end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So after running it for both cases, I get the following values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Abnormal being my Positive case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;precision = 96.5517&#xA;accur = 95&#xA;recall = 87.5000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For Normal = Positive case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;precision =  94.3662&#xA;accur = 95&#xA;recall = 98.5294&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So how do I calculate the combined result, prec, accu, and recall? Or, am I just missing the point and you just calculate it for one class at a time, like the one you are focusing on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I am asking is because now lets say I have a set with multiple class outcomes in my decision tree. This is where I realized I have to pick a class to determine as my positive, or just look at classes individually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a similar set with 3 class possibilities. Again, how do I calculate for the whole data set? Or is an individual class thing? Or do you calculate the individuals and then come together with a total for the whole decision tree.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pelvic_radius   degree_spondylolisthesis    class   &#xA;82.45603817    41.6854736      Abnormal    &#xA;114.365845     -0.421010392    Normal&#xA;95             25              Perfect&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45299" LastEditorUserId="29575" LastEditDate="2018-01-28T05:54:25.623" LastActivityDate="2018-01-28T08:59:21.697" Title="Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question" Tags="|deep-learning|data-mining|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27133" PostTypeId="2" ParentId="25176" CreationDate="2018-01-28T05:39:08.623" Score="2" Body="&lt;p&gt;It sounds like you are trying to classify each article based on a set of tags that are inherently generated from each of the articles. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an alternative to Christian's solution (which I think does serve the purpose that you had described), you could consider using an N-Gram model and Parts-of-Speech (POS) tagging in order to do the following (in this order):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate the frequency of recurring, common phrases within a document / corpus&lt;/li&gt;&#xA;&lt;li&gt;Apply POS tagging to classify nouns (proper and common), verbs, adjectives... etc.&lt;/li&gt;&#xA;&lt;li&gt;(Optional) Apply some sort of rule to only select the top &lt;em&gt;Y&lt;/em&gt; most frequent terms&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Using a standard language model, you can compute the relative probabilities of those words occurring within each document to then weight which article is most likely to mention a certain tag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can learn about &lt;a href=&quot;https://stackoverflow.com/questions/21852066/counting-word-frequency-and-making-a-dictionary-from-it&quot;&gt;creating frequency tables&lt;/a&gt;, &lt;a href=&quot;http://www.nltk.org/book/ch05.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;POS tagging&lt;/a&gt;, calculating &lt;a href=&quot;https://stackoverflow.com/questions/29179074/relative-frequency-in-python&quot;&gt;relative frequencies&lt;/a&gt; with the attached links (in python).&lt;/p&gt;&#xA;" OwnerUserId="8251" LastActivityDate="2018-01-28T05:39:08.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27134" PostTypeId="2" ParentId="27105" CreationDate="2018-01-28T08:01:57.220" Score="2" Body="&lt;p&gt;In general, with Policy Gradients (PG) you can have two 'flavors' of them: &lt;strong&gt;Stochastic&lt;/strong&gt; and &lt;strong&gt;Deterministic&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you can build a NN that outputs continuous actions or one that approximates the sufficient statistics of a probability distribution that you will sample the actions from. The main references are: &lt;a href=&quot;http://proceedings.mlr.press/v32/silver14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theoretical Proofs&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/pdf/1509.02971&quot; rel=&quot;nofollow noreferrer&quot;&gt;NNs with DPG&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are no rules on which one to use. There are many examples in the papers I cite that DPG outperforms SPG but also the opposite. Sometimes you really want the output to be deterministic. For example, in pricing: If a client comes in a store and you use SGD for assigning prices, the client will encounter every single time different prices (small differences but still differences).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my personal experience I find it a bit hard to stabilize the DPG with NNs and you need also to come up with a good exploration strategy. However, once the network is stabilized you get the continuous action values that could control your vehicle. This is a detailed &lt;a href=&quot;https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; of controlling a car with DPG using tensorflow and Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For general theoretical exploration about the two frameworks I suggest you   &lt;a href=&quot;https://riashatislam.files.wordpress.com/2016/11/riashat-islam-report.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; Master Thesis. It provides a concise overview of the two methods (no NN implementations). You can look up also the answer I gave &lt;a href=&quot;https://datascience.stackexchange.com/questions/25209/why-are-policy-gradient-methods-preferred-over-value-function-approximation-in-c/25237?noredirect=1#comment29714_25237&quot;&gt;here&lt;/a&gt; if you decide to get on board with PGs.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2018-01-28T18:16:13.273" LastActivityDate="2018-01-28T18:16:13.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27135" PostTypeId="1" CreationDate="2018-01-28T08:36:24.230" Score="8" ViewCount="7690" Body="&lt;p&gt;Should there be a flat layer in between the conv layers and dense layer in YOLO?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's something not specified in the paper, but I see most implementations of YOLO on github do this. In my implementation, I do not flatten the 7*7*1024 feature map and directly add a Dense(4096) layer after it (I'm using keras with tensorflow backend).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code in question:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = Conv2D(filters=1024, kernel_size=(3,3), padding='same', activation=leaky_relu)(x)&#xA;# x = Flatten()(x) # not using this right now&#xA;x = Dense(4096, activation=leaky_relu)(x)&#xA;x = Dense(30)(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My output of model.summary is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conv2d_294 (Conv2D)          (None, 14, 14, 1024)      9438208   &#xA;_________________________________________________________________&#xA;conv2d_295 (Conv2D)          (None, 7, 7, 1024)        9438208   &#xA;_________________________________________________________________&#xA;conv2d_296 (Conv2D)          (None, 7, 7, 1024)        9438208   &#xA;_________________________________________________________________&#xA;conv2d_297 (Conv2D)          (None, 7, 7, 1024)        9438208   &#xA;_________________________________________________________________&#xA;dense_14 (Dense)             (None, 7, 7, 4096)        4198400   &#xA;_________________________________________________________________&#xA;dense_15 (Dense)             (None, 7, 7, 30)          122910    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I feel that this implementation is more correct than the flat one (unless otherwise specified by the author) because on flattening the output we lose the spatial information needed to predict bounding boxes and confidence scores for each element in the grid.&lt;/p&gt;&#xA;" OwnerUserId="20327" LastEditorUserId="29169" LastEditDate="2021-02-13T16:43:57.393" LastActivityDate="2021-02-13T16:43:57.393" Title="Should there be a flat layer in between the conv layers and dense layer in YOLO?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|yolo|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27136" PostTypeId="1" AcceptedAnswerId="27246" CreationDate="2018-01-28T08:38:13.387" Score="1" ViewCount="2421" Body="&lt;p&gt;&lt;em&gt;I am asking this question here after it went unanswered in Stack Overflow.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;I'm trying to adapt the Keras example for &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;VAE&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I have modified the code to use noisy mnist images as the input of the autoencoder and the original, noiseless mnist images as the output.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from scipy.stats import norm&#xA;&#xA;from keras.layers import Input, Dense, Lambda, Layer&#xA;from keras.models import Model&#xA;from keras import backend as K&#xA;from keras import metrics&#xA;from keras.datasets import mnist&#xA;&#xA;batch_size = 100&#xA;original_dim = 784&#xA;latent_dim = 2&#xA;intermediate_dim = 256&#xA;epochs = 1&#xA;epsilon_std = 1.0&#xA;&#xA;&#xA;&#xA;x = Input(shape=(original_dim,))&#xA;h = Dense(intermediate_dim, activation='relu')(x)&#xA;z_mean = Dense(latent_dim)(h)&#xA;z_log_var = Dense(latent_dim)(h)&#xA;&#xA;def sampling(args):&#xA;    z_mean, z_log_var = args&#xA;    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0.,&#xA;                              stddev=epsilon_std)&#xA;    return z_mean + K.exp(z_log_var / 2) * epsilon&#xA;&#xA;&#xA;z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])&#xA;&#xA;# we instantiate these layers separately so as to reuse them later&#xA;decoder_h = Dense(intermediate_dim, activation='relu')&#xA;decoder_mean = Dense(original_dim, activation='sigmoid')&#xA;h_decoded = decoder_h(z)&#xA;x_decoded_mean = decoder_mean(h_decoded)&#xA;&#xA;&#xA;# Custom loss layer&#xA;class CustomVariationalLayer(Layer):&#xA;    def __init__(self, **kwargs):&#xA;        self.is_placeholder = True&#xA;        super(CustomVariationalLayer, self).__init__(**kwargs)&#xA;&#xA;    def vae_loss(self, x, x_decoded_mean):&#xA;        xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)&#xA;        kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)&#xA;        return K.mean(xent_loss + kl_loss)&#xA;&#xA;    def call(self, inputs):&#xA;        x = inputs[0]&#xA;        x_decoded_mean = inputs[1]&#xA;        loss = self.vae_loss(x, x_decoded_mean)&#xA;        self.add_loss(loss, inputs=inputs)&#xA;        # We won't actually use the output.&#xA;        return x&#xA;&#xA;y = CustomVariationalLayer()([x, x_decoded_mean])&#xA;vae = Model(x, y)&#xA;vae.compile(optimizer='rmsprop', loss=None)&#xA;&#xA;&#xA;# train the VAE on MNIST digits&#xA;(x_train, y_train), (x_test, y_test) = mnist.load_data()&#xA;&#xA;x_train = x_train.astype('float32') / 255.&#xA;x_test = x_test.astype('float32') / 255.&#xA;x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))&#xA;x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))&#xA;&#xA;noise_factor = 0.5&#xA;x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape) &#xA;x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape) &#xA;&#xA;x_train_noisy = np.clip(x_train_noisy, 0., 1.)&#xA;x_test_noisy = np.clip(x_test_noisy, 0., 1.)&#xA;&#xA;&#xA;&#xA;vae.fit(x_train_noisy, x_train,&#xA;        shuffle=True,&#xA;        epochs=epochs,&#xA;        batch_size=batch_size,&#xA;        validation_data=( x_test_noisy,x_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But I am getting the following error message:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;File &amp;quot;ask_vae.py&amp;quot;, line 86, in &amp;lt;module&amp;gt;&#xA;    validation_data=( x_test_noisy,x_test))&#xA;  File &amp;quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&amp;quot;, line 1574, in fit&#xA;    batch_size=batch_size)&#xA;  File &amp;quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&amp;quot;, line 1411, in _standardize_user_data&#xA;    exception_prefix='target')&#xA;  File &amp;quot;/usr/local/lib/python2.7/dist-packages/keras/engine/training.py&amp;quot;, line 58, in _standardize_input_data&#xA;    'expected no data, but got:', data)&#xA;ValueError: ('Error when checking model target: expected no data, but got:', array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       ...,&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.],&#xA;       [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;It seems that the model is not capable of receiving an output ; it works when I change the output to None, like so:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;vae.fit(x_train_noisy, None,&#xA;        shuffle=True,&#xA;        epochs=epochs,&#xA;        batch_size=batch_size,&#xA;        validation_data=( x_test_noisy,None))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Is that because of the way the Custom Loss Layer is defined? How should I proceed?&lt;/p&gt;&#xA;&lt;p&gt;Thanks :)&lt;/p&gt;&#xA;" OwnerUserId="43472" LastEditorUserId="98307" LastEditDate="2020-08-11T13:42:49.637" LastActivityDate="2020-08-11T13:42:49.637" Title="Adapting the Keras variational autoencoder for denoising images" Tags="|python|keras|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27137" PostTypeId="2" ParentId="27132" CreationDate="2018-01-28T08:59:21.697" Score="3" Body="&lt;p&gt;The metrics you calculate are of two types, metrics that depict the entire prediction model you have built like &lt;code&gt;accuracy&lt;/code&gt; which will be same in both the cases of your pseudo code. While the others like &lt;code&gt;precision&lt;/code&gt; says &lt;em&gt;how precise are you in explaining particular class of interest&lt;/em&gt; (&lt;code&gt;accuracy&lt;/code&gt; can also be expressed this way in multi-class classification, see the diagram). This score depends on which class you had selected as a positive one. If you put positive class as face of your model, then it is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;ppv or precision&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;npv&lt;/code&gt;&lt;/a&gt;, if vice versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to the multi-class classification, the core definition holds the same. Now the matrix will be &lt;code&gt;n x n&lt;/code&gt;(&lt;code&gt;n&lt;/code&gt; being number of classes). The sample matrix looks likes this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jYlBa.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jYlBa.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The diagonal elements explains the number of &lt;code&gt;1 class&lt;/code&gt;'s predicted as &lt;code&gt;1&lt;/code&gt;. Now there are &lt;code&gt;n&lt;/code&gt; precision values for each class. Precision for class &lt;code&gt;1&lt;/code&gt; is how many values were truly predicted as &lt;code&gt;1&lt;/code&gt; divided by how many are predicted as &lt;code&gt;1&lt;/code&gt; (FP's included), which is the sum of the first column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not but not the least if you adamantly want a precision like metrics for the entire model, you got &lt;a href=&quot;http://rushdishams.blogspot.in/2011/08/micro-and-macro-average-of-precision.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;micro &amp;amp; macro averaging&lt;/code&gt;&lt;/a&gt; methods, which are helpful in giving a combined metric. This blog post explains it pretty well. Hope this clears something.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-01-28T08:59:21.697" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27138" PostTypeId="1" AcceptedAnswerId="27154" CreationDate="2018-01-28T09:56:24.757" Score="3" ViewCount="6638" Body="&lt;p&gt;I have a training set composed of images having different width and height, I need to resize them to a fix dimension &lt;em&gt;nxn&lt;/em&gt; or &lt;em&gt;nxm&lt;/em&gt; before passing them as the input of a &lt;code&gt;CNN&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know which parameters I need to take into account for correctly choose the new scaled dimensions &lt;em&gt;n&lt;/em&gt; and &lt;em&gt;m&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastEditorUserId="28175" LastEditDate="2018-01-28T17:44:37.040" LastActivityDate="2021-03-10T17:37:29.983" Title="How to correctly resize input images in a CNN?" Tags="|machine-learning|neural-network|deep-learning|image-classification|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27141" PostTypeId="1" CreationDate="2018-01-28T10:59:45.270" Score="0" ViewCount="58" Body="&lt;p&gt;Set of financial instruments represent the set of vertices of the graph. For any pair of vertices $i$ and $j$, an edge connecting them is added to the graph if the corresponding correlation coefficient  $C_{ij}$ based on the price  fluctuations of instruments $i$ and $j$ is greater than or equal to a specified threshold $\theta, (\theta ∈ [ − 1; 1])$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0167947304000258&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Finding correlation&lt;/li&gt;&#xA;&lt;li&gt;Degree of distribution &lt;/li&gt;&#xA;&lt;li&gt;Analysis of cliques and independent sets in the market graph&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am wondering how to achieve the above? I am willing to understand the internal structure of the market.&lt;/p&gt;&#xA;" OwnerUserId="45313" LastEditorUserId="29575" LastEditDate="2018-01-28T18:00:47.310" LastActivityDate="2018-05-29T05:52:11.453" Title="Constructing graph of crypto financial instruments 2016-2017" Tags="|classification|graphs|distribution|finance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27142" PostTypeId="2" ParentId="25987" CreationDate="2018-01-28T11:26:41.297" Score="0" Body="&lt;p&gt;The issue is that you are not introspecting properly the feature importances.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for x in bbc.estimators_:&#xA;    print(x.named_steps['classifier'].feature_importances_)&#xA;&#xA;[ 0.  0.  0.  0.  1.  0.  0.  0.]&#xA;[ 0.  0.  0.  0.  0.  1.  0.  0.]&#xA;[ 0.  1.  0.  0.  0.  0.  0.  0.]&#xA;[ 0.  0.  1.  0.  0.  0.  0.  0.]&#xA;[ 1.  0.  0.  0.  0.  0.  0.  0.]&#xA;[ 0.   0.   0.   0.5  0.   0.   0.   0.5]&#xA;[ 0.  0.  1.  0.  0.  0.  0.  0.]&#xA;[ 0.  0.  0.  1.  0.  0.  0.  0.]&#xA;[ 0.  0.  0.  0.  1.  0.  0.  0.]&#xA;[ 0.  0.  1.  0.  0.  0.  0.  0.]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, the XGBoostClassifier reports the feature importances only on 8 features as defined in the parameters list of the BalancedBaggingClassifier. Those 8 features presented to each XGBoostClassifer are in fact randomly selected for each estimator of the ensemble. So you have to first find which features were subsampled and given to each XGBoostClassifier and then indirectly find the real feature importances.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;names = np.array(names)&#xA;for x, feat_sel in zip(bbc.estimators_, bbc.estimators_features_):&#xA;    feat_imp =  np.nonzero(x.named_steps['classifier'].feature_importances_)&#xA;    print(names[feat_sel[feat_imp]])&#xA;&#xA;['x11']&#xA;['x11']&#xA;['x7']&#xA;['x11']&#xA;['x11']&#xA;['x16' 'x3']&#xA;['x7']&#xA;['x6']&#xA;['x7']&#xA;['x7']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20867" LastActivityDate="2018-01-28T11:26:41.297" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27144" PostTypeId="2" ParentId="25050" CreationDate="2018-01-28T12:41:23.303" Score="2" Body="&lt;p&gt;It is definitely possible to use them in the &lt;em&gt;Encoder&lt;/em&gt; (Discriminator) part of a &lt;strong&gt;GAN&lt;/strong&gt;, since according to Hinton's original paper it's used for classification which means it fits in Discriminators as well even though the proper loss function is a subject to research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose a topic of Capsule Nets in Discriminators of GANs for my Master thesis and I'm &lt;a href=&quot;https://github.com/gusgad/capsule-GAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;working on the model right now&lt;/a&gt;. However, I simplified the routing algorithm by not vectorizing the &lt;em&gt;DigitCaps&lt;/em&gt; values (otherwise Keras code gets complicated, although I'm not sure if what I've done is 100% correct).&lt;/p&gt;&#xA;" OwnerUserId="45317" LastEditorUserId="29575" LastEditDate="2018-01-28T15:44:04.237" LastActivityDate="2018-01-28T15:44:04.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27146" PostTypeId="1" AcceptedAnswerId="27735" CreationDate="2018-01-28T14:44:32.207" Score="2" ViewCount="1188" Body="&lt;p&gt;I tried to implement a function &lt;code&gt;propagate()&lt;/code&gt; that computes the cost function and its gradient knowing that :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forward Propagation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You get X&lt;/li&gt;&#xA;&lt;li&gt;You compute $A = \sigma(w^T X + b) = (a^{(0)}, a^{(1)}, ..., a^{(m-1)}, a^{(m)})$&lt;/li&gt;&#xA;&lt;li&gt;You calculate the cost function: $J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are the two formulas I was using: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$&#xA;$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which I coded as :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def propagate(w, b, X, Y):&#xA;    &quot;&quot;&quot;&#xA;    Implement the cost function and its gradient for the propagation explained above&#xA;&#xA;    Arguments:&#xA;    w -- weights, a numpy array of size (num_px * num_px * 3, 1)&#xA;    b -- bias, a scalar&#xA;    X -- data of size (num_px * num_px * 3, number of examples)&#xA;    Y -- true &quot;label&quot; vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)&#xA;&#xA;    Return:&#xA;    cost -- negative log-likelihood cost for logistic regression&#xA;    dw -- gradient of the loss with respect to w, thus same shape as w&#xA;    db -- gradient of the loss with respect to b, thus same shape as b&#xA;&#xA;    Tips:&#xA;    - Write your code step by step for the propagation. np.log(), np.dot()&#xA;    &quot;&quot;&quot;&#xA;&#xA;    m = X.shape[1]&#xA;&#xA;    # FORWARD PROPAGATION (FROM X TO COST)&#xA;    ### START CODE HERE ### (≈ 2 lines of code)&#xA;    A = sigmoid(np.dot(w.T,X)+b)                                    # compute activation&#xA;    cost = -1/m * np.sum(np.dot(Y.T,np.log(A)) + np.dot((1-Y).T,np.log(1-A)))                                # compute cost&#xA;    ### END CODE HERE ###&#xA;&#xA;    # BACKWARD PROPAGATION (TO FIND GRAD)&#xA;    ### START CODE HERE ### (≈ 2 lines of code)&#xA;    dw = 1/m*np.dot(X,(A-Y).T)&#xA;    db = 1/m*np.sum(A-Y)&#xA;    ### END CODE HERE ###&#xA;&#xA;    assert(dw.shape == w.shape)&#xA;    assert(db.dtype == float)&#xA;    cost = np.squeeze(cost)&#xA;    assert(cost.shape == ())&#xA;&#xA;    grads = {&quot;dw&quot;: dw,&#xA;             &quot;db&quot;: db}&#xA;&#xA;    return grads, cost&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With the following example :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])&#xA;grads, cost = propagate(w, b, X, Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My output was :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dw = [[ 0.99845601]&#xA; [ 2.39507239]]&#xA;db = 0.00145557813678&#xA;cost = 10.6046359582&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Whereas the expected output was&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pQr7p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pQr7p.png&quot; alt=&quot;Expected output, cost =5.801545...&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41692" LastEditorUserId="2584" LastEditDate="2018-03-06T22:38:28.440" LastActivityDate="2018-03-06T22:38:28.440" Title="Why does my cost function return double the expected value?" Tags="|logistic-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27148" PostTypeId="2" ParentId="24722" CreationDate="2018-01-28T14:52:38.803" Score="0" Body="&lt;p&gt;I got a &lt;a href=&quot;https://datascience.stackexchange.com/questions/27147/tuning-a-binary-classifier-for-desired-tradeoff-bewteen-type-i-type-ii-errors&quot;&gt;similar&lt;/a&gt; task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can tune your trade-off by designing an appropriate &lt;a href=&quot;https://datascience.stackexchange.com/questions/9850/neural-networks-which-cost-function-to-use&quot;&gt;cost function&lt;/a&gt;. Or, see &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;F Score&lt;/a&gt; on wikipedia.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reinforcement learning could also have tools you are looking for, namely, means of rewarding 'acting only if sure'. See e.g. the intro in &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton, Barto&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, try searching so called &lt;a href=&quot;https://scottlocklin.wordpress.com/2016/12/05/predicting-with-confidence-the-best-machine-learning-idea-you-never-heard-of/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Conformal&lt;/a&gt; &lt;a href=&quot;http://onlineprediction.net/?n=Main.ConformalPrediction&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prediction&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://osteoporosis.frederick.ac.cy/NNet-VP.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Venn Prediction&lt;/a&gt; is an extension of the original Conformal Prediction framework...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;... for producing well-calibrated probabilistic predictions. It provides well-calibrated lower and upper bounds for the conditional probability of an example belonging to each possible class of the problem at hand. This paper proposes five VP methods based on Neural Networks&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Keep us updated...&lt;/p&gt;&#xA;" OwnerUserId="44654" LastEditorUserId="44654" LastEditDate="2018-01-31T21:15:58.670" LastActivityDate="2018-01-31T21:15:58.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27149" PostTypeId="2" ParentId="27138" CreationDate="2018-01-28T14:55:41.043" Score="2" Body="&lt;p&gt;Two solutions:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Crop images to a fixed size&lt;/li&gt;&#xA;&lt;li&gt;Pad images to a fixed size with 0s.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Keep in mind if you are doing segmentation, gt masks have to be cropped/padded too.&lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="29169" LastEditDate="2021-03-10T17:37:29.983" LastActivityDate="2021-03-10T17:37:29.983" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="27151" PostTypeId="2" ParentId="27141" CreationDate="2018-01-28T17:00:48.453" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;Finding correlation - It has been described in the provided reference&lt;/li&gt;&#xA;&lt;li&gt;Degree of distribution - Every decent package in which you build the network will have a function that will calculate the empirical distribution of the vertex degrees.&lt;/li&gt;&#xA;&lt;li&gt;Analysis of cliques and independent sets in the market graph - This is a problem of Graph Clustering. There are many methods available. Start by reading &lt;a href=&quot;http://www.leonidzhukov.net/hse/2015/networks/papers/GraphClustering_Schaeffer07.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45077" LastEditorUserId="29575" LastEditDate="2018-01-28T18:01:21.590" LastActivityDate="2018-01-28T18:01:21.590" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27152" PostTypeId="1" CreationDate="2018-01-28T17:11:44.620" Score="2" ViewCount="89" Body="&lt;p&gt;The typical steps for solving a machine learning/pattern recognition problem: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Data Analysis and splitting the data into test and train sets.&lt;/li&gt;&#xA;&lt;li&gt;Choosing a model.&lt;/li&gt;&#xA;&lt;li&gt;Training the model, and testing the model against the test set.&lt;/li&gt;&#xA;&lt;li&gt;If the accuracy of the model is not acceptable, start over with a new model. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Step (2) can be automated to some extent using a grid search, but are there ways of automating the whole process? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking in particular of large scale data applications (for example when dealing with retail data, or customer analytics on a site like Netflix, ) where there are millions of instances of similar but distinct machine learning problems that each need to be trained and validated separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In such a situation it is impossible for a team of analysts or data scientists to perform the above steps and some sort of automated model development framework must be used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the frameworks that allow for this? &lt;/p&gt;&#xA;" OwnerUserId="8517" LastEditorUserId="8517" LastEditDate="2018-01-28T20:42:51.623" LastActivityDate="2019-08-22T15:30:29.923" Title="Are there any frameworks available that allow for automated large scale supervised machine learning?" Tags="|machine-learning|training|automation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27153" PostTypeId="1" AcceptedAnswerId="27756" CreationDate="2018-01-28T17:20:14.870" Score="9" ViewCount="30983" Body="&lt;p&gt;I have built a convolutional neural network which is needed to classify the test data into either 0 or 1. I am training the CNN with labels either 0 or 1 but while running the below code I am getting the following result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predictions = classifier.predict(x_test)&#xA;&#xA;print(predictions)&#xA;&#xA;[0.0128037 ]&#xA; [0.01182843]&#xA; [0.01042355]&#xA; [0.00906552]&#xA; [0.00820154]&#xA; [0.00726516]......&#xA;&#xA;logloss_score = log_loss(y_test, predictions)&#xA;print(logloss_score)&#xA;0.047878393431377855&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I get the results between 0 and 1? What do I need to modify in the above code?&lt;/p&gt;&#xA;" OwnerUserId="45329" LastEditorUserId="29169" LastEditDate="2021-02-13T16:43:43.977" LastActivityDate="2021-02-13T16:43:43.977" Title="How to get predicted class labels in convolution neural network?" Tags="|machine-learning|deep-learning|neural-network|cnn|convolutional-neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27154" PostTypeId="2" ParentId="27138" CreationDate="2018-01-28T17:26:27.973" Score="1" Body="&lt;p&gt;I have a suggestion for you. Maybe not complete enough to be a complete solution but that is exactly what I've experienced. Suppose that you have writings in a paper and the paper is a part of a scene, or you have boys with nice mustaches which are beside a wide scene. In such cases which you have special information which are significant for classifying, the first one may help the classifier recognize book and written things and the second one can help the classifier find the gender of people in the scene, try to resize the images in a way that such significant things be recognizable to the human. If humans can understand them, you can hope that your classifier will be able too. Consider that you shouldn't resize the images to small pixels to avoid your net being with lots parameters. If you resize the images to too small ones, you may loose important information.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-28T17:26:27.973" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27155" PostTypeId="2" ParentId="25753" CreationDate="2018-01-28T18:00:00.000" Score="3" Body="&lt;p&gt;I think you're on the right track. If you can produce an objective scoring algorithm for the output which more closely resembles your subjective perception than 1:1 error you could definitely use simulated annealing to derive from that an optimal matrix at a given size N.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your current scoring calculation for error seems to focus too much on high-frequency information (details), and not enough on low frequency information (larger areas) - notice the results you deem subjectively superior, if blurred (low-pass-filter) more closely resemble the input image (if also blurred) compared to the results you have deemed inferior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it were me, my next step might be to try something hierarchical in order to avoid bias toward any given frequencies, maybe a quadtree where at each level 1x1, 2x2, 4x4, etc. I average the values of the node's children, then scoring uses the error between trees.&lt;/p&gt;&#xA;" OwnerUserId="6585" LastActivityDate="2018-01-28T18:00:00.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27157" PostTypeId="2" ParentId="27130" CreationDate="2018-01-28T18:35:49.733" Score="3" Body="&lt;p&gt;I think the documentation is kind of self-explanatory here. Fit takes in array of size &lt;code&gt;n_samples&lt;/code&gt; in which each element is the class of the datum or if the data point belongs to multiple classes, the input would be obviously of size &lt;code&gt;n_samples x n_classes&lt;/code&gt;. That is what you gave in as input in your example. Each point can belong to any of the three classes. That is why you have &lt;code&gt;[0, 1, 2]&lt;/code&gt; as number of classes. So as mentioned in the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html#sklearn.preprocessing.LabelBinarizer.fit&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; if you try&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; lb.transform([0, 1, 2, 0])&#xA;[[1 0 0]&#xA; [0 1 0]&#xA; [0 0 1]&#xA; [1 0 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and if you try a class that is non-existent after fit like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; lb.transform([0, 1, 2, 1000])&#xA;[[1 0 0]&#xA; [0 1 0]&#xA; [0 0 1]&#xA; [0 0 0]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;No class named &lt;code&gt;1000&lt;/code&gt; exists, so multi-targeted conversion for &lt;code&gt;1000&lt;/code&gt; class case is plainly &lt;code&gt;[0, 0, 0]&lt;/code&gt;. Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-01-28T18:35:49.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27158" PostTypeId="1" CreationDate="2018-01-28T18:38:47.470" Score="1" ViewCount="317" Body="&lt;p&gt;Why explaining away concept is not applicable in restricted Boltzmann machines? Their hidden units form a V structure from which probabilistic influence can flow given the observed visible variable. Why is this a problem in deep belief nets?&lt;/p&gt;&#xA;" OwnerUserId="45333" LastEditorUserId="29575" LastEditDate="2018-01-28T18:54:09.597" LastActivityDate="2023-12-29T01:03:20.437" Title="How hidden units are conditionally independent in restricted Boltzmann machines" Tags="|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27160" PostTypeId="1" CreationDate="2018-01-28T20:21:03.760" Score="1" ViewCount="51" Body="&lt;p&gt;We planned to create an Android app which predicts the present student in class through face detection by camera in the class, my job is to write algorithm for face detection and the problem is I can do it for octave and Python but never did for Java or other programming language.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I use the octave or Python script for Android app, is there any possible ways to connect the dots between Android app and Python/octave script?&lt;/p&gt;&#xA;" OwnerUserId="45336" LastEditorUserId="29575" LastEditDate="2018-01-28T20:27:31.677" LastActivityDate="2018-01-28T20:27:31.677" Title="is it possible to write machine learning algorithm in octave or python for application created in android" Tags="|python|neural-network|java|" AnswerCount="0" CommentCount="1" ClosedDate="2018-01-28T21:42:18.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="27164" PostTypeId="1" CreationDate="2018-01-29T00:19:52.100" Score="1" ViewCount="644" Body="&lt;p&gt;I'm working on a free text classification problem with over 100 classes in the training data. There is huge variation in the sample sizes of the classes: ranging from 1 to around 6000. I am using a custom-made algorithm which builds a training model of each class's most significant keyword or phrase (up to 3 consecutive words). The significance is a value between 0 and 1 and is the difference in the frequency that the term occurs in the class and the frequency that it occurs among all data not in the class. I'm then using this model to test whether whether new, manually class-labelled records have the correct class by assigning PASS if the new record's class has a significance level &gt; .9 AND the test record's free text contains the keyword/phrase. If its &gt; .9 and lacks the keyword, I'm giving it a FAIL. And if the class significance is &amp;lt;= .9, then I'm giving it a NOT CHECKED, regardless of if the test text contains the keyword or not because the strength of the top keyword is too weakly associated with the class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I developed this custom-made algorithm after finding that Naive Bayes (a slightly more complicated probabilistic method taking into account all words) wasn't doing that well...I think because of the nature of my data: the text is often short and there are some classes where 100% should contain a specific keyword. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results so far are promising...far fewer false positives than Naive Bayes. But I've found that classes with tiny sample sizes in the training data had a LOT more false positives than with large sample sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: what would the best way to account for % confidence based on the sample size?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best idea I have right now is to use the Sigmoid function, where input x = sample size and output y = something like % confidence, then multiple this Sigmoid output by the significance level described above and only doing the incoming record class check if this modified significance level is &gt; .9. Are there better options?&lt;/p&gt;&#xA;" OwnerUserId="45341" LastActivityDate="2018-01-29T00:19:52.100" Title="Multiclass classification with many classes and wide range of sample sizes" Tags="|multiclass-classification|text|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27165" PostTypeId="2" ParentId="26997" CreationDate="2018-01-29T00:21:30.350" Score="1" Body="&lt;p&gt;KMeans on such data is meaningless if used carelessly. It looks as if your result is based on whether the ID is larger than 200000 or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rather than copy and pasting example code from the internet now also to visualize, I think you should spend more time thinking about what you are actually computing. Because solving the wrong problem doesn't help.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-29T00:21:30.350" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27166" PostTypeId="1" AcceptedAnswerId="27174" CreationDate="2018-01-29T03:02:05.057" Score="3" ViewCount="848" Body="&lt;p&gt;I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything else makes sence. The q-learning algorithm is an &lt;a href=&quot;https://studywolf.wordpress.com/2013/07/01/reinforcement-learning-sarsa-vs-q-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;off-policy algorithm&lt;/a&gt;, unlike &lt;a href=&quot;https://stackoverflow.com/questions/32846262/q-learning-vs-sarsa-with-greedy-select#&quot;&gt;SARSA&lt;/a&gt;. The Bellman equation describes q-learning as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) - Q(s_t, a_t) \right] $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The q value for action $a$ taken in state $s$ at time $t$ becomes equal to: that same q-value &lt;strong&gt;plus&lt;/strong&gt; small amount of: &lt;em&gt;currently received reward (reward could be zero) with some amount $\gamma$ of the best Q-value available in the new state, &lt;strong&gt;minus&lt;/strong&gt; our current value&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To me, this $-Q(s_t, a_t)$ term at the very end is redundant. ...If we set gamma to $0.8$, the future rewards will decay anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, if instead we now set $\gamma=0$, then the $-Q(s_t, a_t)$ term will drag our value down - is there a case when it would be useful and what would the result be?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Q-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipeda uses a slighly different form&lt;/a&gt; of the bellman equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) \right] $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's the same equation as the one above, because we can multiply $Q(s_t, a_t)$ with an $\alpha$ and then factor it out, obtaining the first equation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This representation makes me kind-off understand that we are linearly interpolating from current to newer Q-value, but I can't tie it to the original representation ...In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-01-29T03:13:47.313" LastActivityDate="2018-01-29T10:49:52.467" Title="Q-learning why do we subtract the Q(s, a) term during update?" Tags="|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27167" PostTypeId="1" AcceptedAnswerId="27168" CreationDate="2018-01-29T03:22:25.480" Score="3" ViewCount="458" Body="&lt;p&gt;I have a large database of images that are only &lt;strong&gt;partially labeled&lt;/strong&gt; for &lt;strong&gt;multiple&lt;/strong&gt;, &lt;strong&gt;non-exclusive&lt;/strong&gt; characteristics or objects present on them. For instance, an underwater scene might feature the labels &lt;code&gt;water&lt;/code&gt;, &lt;code&gt;swimsuit&lt;/code&gt; and &lt;code&gt;fish&lt;/code&gt; on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem is it's only partially labeled, meaning that the fact the label &lt;code&gt;cat&lt;/code&gt; is absent doesn't mean there's no cat on the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What'd be the best strategy to train a CNN able to tell the missing labels on the dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image base has 230 thousand images on it, but given that multiple labels are possible, it's hard to tell the ratio of images that are fully labeled. I'm able to code python and managed to use keras to train on mnist dataset on gpu.&lt;/p&gt;&#xA;" OwnerUserId="45344" LastActivityDate="2018-02-28T03:46:17.747" Title="What's the best strategy to train a CNN with images that only have labels for positive characteristics?" Tags="|machine-learning|neural-network|convolution|labels|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27168" PostTypeId="2" ParentId="27167" CreationDate="2018-01-29T03:37:29.803" Score="2" Body="&lt;p&gt;I think your best bet would be transfer learning. Start with a model that has already been trained with a wider dataset such as the ones presented &lt;a href=&quot;https://www.tensorflow.org/tutorials/image_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. From there you can train the model with your specific dataset. You can then use output nodes for the labels which you have available to you, and you can get the predictions for the other images from the pre-trained model which are usually trained for thousands of different classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can train a model with all the output classes you have in your label set and another output for &quot;other&quot;. Then when an output node is selected you can pass that same input to the pre-trained model which was trained with your data as well. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-29T03:37:29.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27169" PostTypeId="1" CreationDate="2018-01-29T04:19:13.520" Score="2" ViewCount="1108" Body="&lt;p&gt;I'm fitting a neural network using a very small data set, &#xA;so try splitting the data into training and validation sets.&#xA;(there is a separate test set)&#xA;If I split training/validation randomly multiple times, &#xA;construct a neural network for each training/validation split,&#xA;and take average of predicted values of the neural networks on the test set,&#xA;can it be called a ensemble model?&#xA;Or is there a specific name for such a technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I just found that a similar technique is called 'repeated random sub-sampling validation,' but the &lt;code&gt;RRSSV&lt;/code&gt; splits the data into training and test set (though it is called 'validation data' according to Wikipedia, it's actually test data). My method splits the given data into training and validation set, and use separate test data.&#xA;I think my method can also be called &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Repeated_random_sub-sampling_validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;RRSSV&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="8478" LastEditorUserId="28175" LastEditDate="2018-01-29T12:25:15.140" LastActivityDate="2018-01-29T12:25:15.140" Title="Taking average of multiple neural networks?" Tags="|machine-learning|neural-network|deep-learning|cross-validation|ensemble-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27171" PostTypeId="1" AcceptedAnswerId="27175" CreationDate="2018-01-29T07:18:01.433" Score="1" ViewCount="1662" Body="&lt;p&gt;I am new to machine learning techniques. I was going through few supervised machine learning model examples and i have doubt in predicting future values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have daily time series dataset from database where my target variable is complete noise signal like this:&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/MXThe.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MXThe.jpg&quot; alt=&quot;img&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train and predict the models, we divide the dataset into train set, validation set and test set to check if the model is efficient or not. I have two independent variable and one target variable. I am using linear regression,Keras LSTM and other models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My basic question is how do i predict for future values (for next one week, one month) when i don't have independent variables for next week? what am i supposed to give as predictors in that case? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any information is much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="45351" LastActivityDate="2018-01-29T08:41:59.543" Title="What to give as predictors to predict future values?" Tags="|machine-learning|python|r|predictive-modeling|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27173" PostTypeId="2" ParentId="27169" CreationDate="2018-01-29T07:43:57.873" Score="6" Body="&lt;p&gt;I think even this method is also called Ensemble Method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I conclude that?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You might have heard about this algorithm named &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_forest&quot; rel=&quot;noreferrer&quot;&gt;Random Forest&lt;/a&gt;,&#xA;what does it do? It take data randomly at row level and column level&#xA;builds different trees and takes an average of it. It is also&#xA;considered as one of the best algorithm for Prediction and&#xA;Classification. Can go through this &lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/random-forests-explained-intuitively&quot; rel=&quot;noreferrer&quot;&gt;explanation&lt;/a&gt; for better&#xA;understanding. Random Forest is called Ensemble model(of trees).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;A suggestion, as you have mentioned in the question that you have very less data, at that time the models cannot generalize well and you cannot achieve good results. If you have any way to increase the dataset size by collecting more data, it could aid you in achieving better accuracy. This is also explained the link attached(explanation). Do go through and let me know if you have any additional questions. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-01-29T07:53:32.343" LastActivityDate="2018-01-29T07:53:32.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27174" PostTypeId="2" ParentId="27166" CreationDate="2018-01-29T08:13:48.400" Score="4" Body="&lt;p&gt;The wikipedia formulation does indeed show you a better view of how the update rule for action values is constructed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')) \right] $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;. . . here you can see that you are taking a weighted average between $Q(s_t, a_t)$ and $ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')$. An aside: In both cases you have written $\text{argmax}$, where the actual term is $\text{max}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuitive reason for the update is because the second term contains a new estimate of the true value of Q from the environment - the immediate reward $r_t$, and the actual state transition that occurred $s_{t+1}$ are observed data that are part of the new estimate. The term  $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ is usually called the &lt;em&gt;Temporal Difference Target&lt;/em&gt; or just &lt;em&gt;TD Target&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The $\gamma$ term is the discount rate, and nothing to do with convergence of Q-learning or SARSA. It is a term used to control preference for more immediate rewards (low values) or longer-term rewards (high values), and as such is part of the problem definition. It is not a learning parameter like $\alpha$, the learning rate. And in fact $\gamma = 0$ is possible (rarely used, but meaning that only next reward is important); $\gamma = 1$ is also possible and quite a common choice for episodic problems. Clearly when using $\gamma = 1$, there is no &quot;decay of future rewards&quot; happening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is actually happening is that the TD Target is a stochastic estimate for the correct Q action value (in fact with all TD learning such as SARSA and Q-learning, this is a biased estimate, although the bias should reduce as the system approaches convergence). If you rename $Q(s,a)$ as $Q_{so\_far}$ and $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ as $Q_{latest\_estimate}$, you get the idea of the update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q_{updated} \leftarrow (1-\alpha)Q_{so\_far} + \alpha Q_{latest\_estimate}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is the same as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q_{updated} \leftarrow Q_{so\_far} + \alpha(Q_{latest\_estimate} - Q_{so\_far})$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might ask &quot;Why is the TD Target an estimate of $Q(s_t, a_t)$?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer can be derived from the definition of Q, and is called the Bellman equation - in this case the Bellman equation for action value under the optimal policy $\pi^*$ (anther aside: what you have called the Bellman equation is not, although it is related). I won't give the full derivation here, but in essence the definition of Q is &quot;the expected total reward when starting with state $s_t$ and action $a_t$ then following the optimal policy form then on&quot;, or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$q_{*}(s, a) = \mathbb{E}_{\pi^*}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1} | S_t = s, A_t = a]$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and from this definition, you can extract the first reward term and write out $q_*(s_t, a_t)$ in terms of $q_*(s_{t+1}, a_{t+1})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$q_{*}(s, a) = \sum_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}_{a'} q_{*}(s', a'))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $p(r,s'|s,a)$ is the transition probability of getting reward $r$ and ending up in state $s'$ when starting in state $s$ and taking action $a$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note this is an equality, not an update rule. Q-learning and SARSA turn it into an update rule by sampling the right hand side, and using that sample to update the current best estimate of the left-hand term. Note that by taking a sample, you don't need to know $p(r,s'|s,a)$ . . . that is what is meant by &quot;model free&quot; learning, as if you knew $p(r,s'|s,a)$ then you would have a model of the underlying Markov Decision Process (MDP).&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-01-29T10:49:52.467" LastActivityDate="2018-01-29T10:49:52.467" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27175" PostTypeId="2" ParentId="27171" CreationDate="2018-01-29T08:41:59.543" Score="0" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, when we use any kind of predicting algorithm, then you need to have the future values of the independent variables(features for explaining the target variable), for Prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, If you don't have such values then you cannot use predicting algorithms, we have forecasting algorithms to do such task. To apply this you need to have data which is time series data. From the sample and graph I think your data is time series data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYI, you can also use additional features even in time series. For &lt;a href=&quot;https://machinelearningmastery.com/basic-feature-engineering-time-series-data-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Engineering&lt;/a&gt; on Time Series data you can go through the link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need any additional information let me know.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-01-29T08:41:59.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27177" PostTypeId="2" ParentId="25291" CreationDate="2018-01-29T09:06:49.050" Score="2" Body="&lt;p&gt;@imoutidi, I also encountered the same deprecation warning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After digging around a bit, it looks like the new/replacement package can be imported with the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;from nltk.tag.stanford import CoreNLPNERTagger&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when trying to run the tag() method I end up getting an unexpected HTTP connection refused error. I haven't figured out if this is an environment configuration issue (missing .jar path) or an issue with the NLTK codebase. I tried temporarily turning off my local firewall and that didn't help, so I'm guessing it is one of the other two possibilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope the python import statement is at least some help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yz7EH.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yz7EH.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45360" LastActivityDate="2018-01-29T09:06:49.050" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27178" PostTypeId="2" ParentId="27038" CreationDate="2018-01-29T09:31:30.983" Score="0" Body="&lt;p&gt;Adding StandardScaler from sklearn.preprocessing improved the results somewhat, as did (in this case) making the net deeper.&lt;/p&gt;&#xA;" OwnerUserId="43472" LastActivityDate="2018-01-29T09:31:30.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27179" PostTypeId="1" AcceptedAnswerId="27201" CreationDate="2018-01-29T09:40:46.870" Score="2" ViewCount="180" Body="&lt;p&gt;I am trying to make a CNN for 3D image recognition but everything is predicted to only one class out of three. And the prediction even quickly converges during the first epoch.   I have been working on this for an week and totally lost.&lt;br&gt;&#xA;I have my own several guess why it always converges to one class.   &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;My 3D tensor is huge as 40x35x30 and there are a lot of white spaces because I had to put my objects in cubic box. Would this be problematic?   &lt;/li&gt;&#xA;&lt;li&gt;I have only ~5000 samples for training and ~500 for test sets. Do I have too little samples?&lt;/li&gt;&#xA;&lt;li&gt;My labels have 3 classes and ~75% of the whole data belong to the class number 1. During the first epoch,  my model quickly converges to predict 99% of data as this class. Would this unbalanced data cause the problem?&lt;/li&gt;&#xA;&lt;li&gt;My model is designed as&lt;br&gt;&#xA;conv1(kernel_size=10,stride=1,filter=32)-batch_norm1-maxpool1-conv2(8,1,64)-batch_norm2-maxpool2-conv3(8,1,64)-maxpool3-fc4(1024)-batchnorm4-dropout4(20%)-fc5(384)-batchnorm5-dropout5(20%)-fc6(3)&lt;br&gt;&#xA;I standardized the input between -1 and 1 I use leaky-relu activation for conv layers. I use Adam optimizer with decay rate of 0.99.&lt;br&gt;&#xA;First, I am not sure if it is okay to perform batch normalization at every layer. Do I miss any important concept for designing CNN model here?   &lt;/li&gt;&#xA;&lt;li&gt;or maybe my data is just bad.. I kind of suspect that my data are not significantly different from each other and there is no pattern at all.  In this case, is there any statistical method/model to check if my data have meaningful differences? (The 3D images are some chemical/physical data in 3D space that I converted as numpy matrix)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But I think it is more like vanishing problem because when I initialize the variable with Xavier's way, the convergence to the class #1 is slower. Please someone help me :(&lt;/p&gt;&#xA;" OwnerUserId="45185" LastEditorUserId="45185" LastEditDate="2018-01-29T17:02:19.860" LastActivityDate="2018-01-29T17:51:42.743" Title="Several fundamental questions about CNN" Tags="|deep-learning|image-classification|convolution|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27180" PostTypeId="1" CreationDate="2018-01-29T10:13:08.977" Score="1" ViewCount="94" Body="&lt;p&gt;I have a problem statement wherein the following input&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The&lt;/em&gt; &lt;strong&gt;James&lt;/strong&gt; &lt;em&gt;world&lt;/em&gt; &lt;strong&gt;Bond&lt;/strong&gt; &lt;em&gt;shall&lt;/em&gt; &lt;strong&gt;can't&lt;/strong&gt; &lt;em&gt;end&lt;/em&gt; &lt;strong&gt;save us&lt;/strong&gt; &lt;em&gt;tomorrow.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;must be converted into this - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The world shall end tomorrow&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;amp; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;James Bond can't save us&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially my inputs are a set of sentences interleaved with each other, and I need to separate them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training data contains slight variations of these interleaving sentences, I have been trying to work with a 1024 cell 2 Layered LSTM network, currently achieving about 65% accuracy when trying to predict the next word given the current set of words. I use the prediction from this model to chain words together into distinct sentences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions that I have are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is LSTM RNN the best way to achieve something like this?&lt;/li&gt;&#xA;&lt;li&gt;What should be a good LSTM Cell size? (I have a vocab of about 22K words.)&lt;/li&gt;&#xA;&lt;li&gt;Are 2 layers sufficient for modeling this problem statement?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45367" LastActivityDate="2018-01-29T10:13:08.977" Title="How do I separate Interleaved sentences using LSTM networks?" Tags="|tensorflow|word2vec|lstm|sequence|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27182" PostTypeId="2" ParentId="26555" CreationDate="2018-01-29T11:13:40.477" Score="3" Body="&lt;pre&gt;&lt;code&gt;X_ne1 = X_test[:,3]&#xA;y_pred2 = regressor_OLS.predict(X_ne1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44519" LastActivityDate="2018-01-29T11:13:40.477" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27183" PostTypeId="1" CreationDate="2018-01-29T11:14:10.953" Score="1" ViewCount="246" Body="&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from numpy import exp, array, random, dot&#xA;&#xA;R = np.matrix([[-1, -1, -1, -1,1, -1],        # for correct action the &#xA;reward is 1 and for wrong action it's -1                     &#xA;           [-1, -1, -1, 1, -1, 1],&#xA;           [-1, -1, -1, 1, -1, -1],&#xA;           [-1, 1, 1, -1, 1, -1],&#xA;           [-1, 1, 1, -1, -1, 1],&#xA;           [-1, 1, -1, -1, 1, 1]])&#xA;&#xA;Q = np.matrix(np.zeros([6, 6]))  # Q matrix&#xA;&#xA;gamma = 0.99  # Gamma (learning parameter).&#xA;lr = 0.1  # learning rate&#xA;initial_state = 1  # Initial state. &#xA;w = 2*np.random.random((6,1)) - 1  # random weights for each action&#xA;&#xA;def sigmoid(x):&#xA;    return 1 / (1 + exp(-x))&#xA;&#xA;def sigmoid_derivative(x):&#xA;    return x * (1 - x)&#xA;&#xA;def available_actions(state):  # This function returns all available actions &#xA;in the state given as an argument&#xA;    current_state_row = R[state,]&#xA;    return current_state_row&#xA;&#xA;def sample_next_action(available_actions_range):  # This function chooses at &#xA;random which action to be performed within the rangeof all the available &#xA;actions.&#xA;    next_action = int(np.random.choice(available_actions_range, 1))&#xA;    print(&quot;next_action: &quot;, next_action)&#xA;    return next_action&#xA;&#xA;def update(action, gamma,predict):  # This function updates the Q matrix &#xA;according to the path selected &#xA;&#xA;    Q[current_state, action] = action + gamma * predict&#xA;    return Q[current_state, action]&#xA;&#xA;for i in range(1000): ## Training&#xA;    current_state = np.random.randint(0, int(Q.shape[0]))&#xA;    available_act = np.array(available_actions(current_state))&#xA;    action = np.max(available_act)&#xA;&#xA;    Qout = np.multiply(available_act,w) # forward propagation # multiplying &#xA;    the selected action with the weights)&#xA;    predict = sigmoid(np.max(Qout))&#xA;    Qtarget = update(action, gamma, predict)  # target q values(rewards)&#xA;    loss = np.sum(Qtarget - Qout) # backward prop # calculating the error &#xA;     between predicted and target q values&#xA;    adjustment = np.multiply(action ,loss , Qout)&#xA;    w = w + adjustment # adjusting weights&#xA;    Q += Qout&#xA;&#xA;print(&quot;Trained network:&quot;)&#xA;print(&quot;--------&quot;)&#xA;print(&quot;weights : &quot;, w)&#xA;np.set_printoptions(precision=2,suppress=True)&#xA;print(Q / np.max(Q) * 100)  # Normalize the &quot;trained&quot; Q matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am a newbie into neural networks and reinforcement learning. I am trying to create a neural network (for learning purpose) which contains a single neuron with 6 inputs and 1 output. The problems that I am facing are as follows -&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I am not sure if the way I am trying to predict the Q value through my Q network is correct ? and I am unsure as the calculation that I am doing to find out the target Q value is correct.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When I run the code the weights become 'Nan' after a few iterations and this leads the Q matrix to have all the Nan values.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45356" LastEditorUserId="1330" LastEditDate="2021-07-18T13:41:58.697" LastActivityDate="2021-07-18T13:41:58.697" Title="Simple Q-learning neural network using numpy" Tags="|neural-network|q-learning|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27184" PostTypeId="2" ParentId="19207" CreationDate="2018-01-29T11:17:54.133" Score="1" Body="&lt;p&gt;From experience, I've found that &lt;code&gt;ranger&lt;/code&gt; provides a quick implementation of &lt;code&gt;randomForest&lt;/code&gt; (probably since it is designed specifically to run on high-dimensional data sets). It's also more flexible than &lt;code&gt;randomForest&lt;/code&gt; while providing comparable accuracy.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-01-29T11:17:54.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27185" PostTypeId="2" ParentId="27169" CreationDate="2018-01-29T12:06:31.590" Score="1" Body="&lt;p&gt;You might try extending your approach to include adding random noise to your training data, sometimes called noise injection. By doing this you can theoretically extend the amount of training data you have almost infinitely and avoid over-fitting of a small training sample. An internet search will turn up several papers on the subject, e.g. &lt;a href=&quot;https://arxiv.org/pdf/1612.01490.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Whiteout:  Gaussian Adaptive Noise Regularization&#xA;in FeedForward Neural Networks&lt;/a&gt;  &lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2018-01-29T12:06:31.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27188" PostTypeId="2" ParentId="23661" CreationDate="2018-01-29T13:50:25.947" Score="3" Body="&lt;p&gt;I'll think you'll find what you're looking for in &lt;a href=&quot;https://opendata.stackexchange.com/q/4126&quot;&gt;this question&lt;/a&gt; on Open Data. I've checked the &lt;a href=&quot;http://www.pewresearch.org/data/download-datasets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pew Research Center data&lt;/a&gt;, and found &lt;a href=&quot;http://www.pewinternet.org/dataset/june-2016-cybersecurity-knowledge/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this poll&lt;/a&gt; about cyber-security that seems to be something like what you look for.&lt;/p&gt;&#xA;" OwnerUserId="43870" LastActivityDate="2018-01-29T13:50:25.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27191" PostTypeId="1" CreationDate="2018-01-29T15:19:15.547" Score="1" ViewCount="2167" Body="&lt;p&gt;I'm using xgboost package on python 3.5 for time base predication&lt;br&gt;&#xA;The result of the f-score Partial dependence on the order of the columns in data frame.&lt;br&gt;&#xA;The rmse of the predication is a same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example: &lt;br&gt;&#xA;In case of that columns order: &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;columns_order = ['MONTH', 'MDAY', 'WEEK', 'WDAY', 'YEAR', 'DYEAR', 'START_TIME_NORMALIZE', '1_last_x', '2_last_x',&#xA;                 '3_last_x', '4_last_x', '5_last_x', 'EPISODE_NUMBER', 'SEASON_NUMBER', 'pre_1_last_x', 'pre_2_last_x',&#xA;                 'pre_3_last_x', 'pre_4_last_x', 'pre_5_last_x']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fscore result is:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fscore = [('START_TIME_NORMALIZE', 7910), ('DYEAR', 7518), ('1_last_x', 7108), ('MDAY', 6977), ('WDAY', 6797),&#xA;          ('WEEK', 6319), ('2_last_x', 5490), ('pre_1_last_x', 5269), ('5_last_x', 5202), ('3_last_x', 5096),&#xA;          ('EPISODE_NUMBER', 5057), ('4_last_x', 4723), ('SEASON_NUMBER', 4442), ('MONTH', 4097),&#xA;          ('pre_2_last_x', 3502), ('pre_5_last_x', 3401), ('pre_3_last_x', 3221), ('YEAR', 2956),&#xA;          ('pre_4_last_x', 2827)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And in case of that columns order: &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;columns_order = ['1_last_x', '2_last_x', '3_last_x', '4_last_x', '5_last_x', 'MDAY', 'WEEK', 'WDAY', 'MONTH', 'YEAR',&#xA;                 'DYEAR', 'START_TIME_NORMALIZE', 'EPISODE_NUMBER', 'SEASON_NUMBER', 'pre_1_last_x', 'pre_2_last_x',&#xA;                 'pre_3_last_x', 'pre_4_last_x', 'pre_5_last_x']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The fscore result is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fscore = [('1_last_x', 9909), ('START_TIME_NORMALIZE', 9407), ('DYEAR', 9005), ('WEEK', 8624), ('MDAY', 8202),&#xA;          ('WDAY', 7726), ('2_last_x', 6916), ('3_last_x', 6533), ('EPISODE_NUMBER', 6023), ('pre_1_last_x', 5825),&#xA;          ('5_last_x', 5703), ('4_last_x', 5232), ('SEASON_NUMBER', 5093), ('pre_2_last_x', 4477),&#xA;          ('pre_3_last_x', 3672), ('pre_4_last_x', 3564), ('pre_5_last_x', 3382), ('YEAR', 2892), ('MONTH', 1060)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The average of the &lt;code&gt;n_last_x&lt;/code&gt; variances is &lt;code&gt;0.00636&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="26327" LastActivityDate="2018-01-29T15:19:15.547" Title="xgboost f-score" Tags="|python|decision-trees|xgboost|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27192" PostTypeId="2" ParentId="26688" CreationDate="2018-01-29T15:41:00.553" Score="0" Body="&lt;p&gt;It's not enough to just somehow encode everything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is easy: just encode everything as 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you need to consider are &lt;em&gt;properties&lt;/em&gt; of this encoding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, KMeans is only meaningful when &lt;em&gt;least squared errors&lt;/em&gt; on the encoding imply a better result in your original data. I don't know any encoding of text that has this property.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't just do something because you don't know anything else to do. First understand your date, then the problem, then the solution that solves the &lt;em&gt;right&lt;/em&gt; problem. Spell out the mathematical objective for your problem, and why it is important to solve.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-01-29T15:41:00.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27193" PostTypeId="1" AcceptedAnswerId="27198" CreationDate="2018-01-29T15:41:12.317" Score="1" ViewCount="95" Body="&lt;p&gt;I have a data-set with 2 target classes.&#xA;In training dataset, the ratio of the 2 classes are 1:93&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With my neural network, the current accuracy is 63%.&#xA;I tried undersampling, oversampling, equal sampling but not improvements.&lt;/p&gt;&#xA;" OwnerUserId="45385" LastEditorUserId="28175" LastEditDate="2018-01-29T17:44:23.543" LastActivityDate="2018-01-29T17:44:23.543" Title="Handle Unbalanced data" Tags="|machine-learning|neural-network|deep-learning|class-imbalance|" AnswerCount="1" CommentCount="3" ClosedDate="2018-02-01T18:24:55.610" ContentLicense="CC BY-SA 3.0" />
  <row Id="27194" PostTypeId="1" AcceptedAnswerId="27204" CreationDate="2018-01-29T16:03:00.513" Score="0" ViewCount="161" Body="&lt;p&gt;I have trained an XGBoost model for prediction. The algorithm is able to calculate variable importances.  I was asked why I have not analyzed these variable importances and I did not because as I understood XGBoost is rather a predictive than a descriptive model. I am not sure what extent theses importances can be used to capture real importance (I understand that it may be dependent on the context). Am I right that XGBoost is rather a predictive model?&lt;/p&gt;&#xA;" OwnerUserId="33095" LastActivityDate="2018-01-29T18:48:28.730" Title="XGBoost: predictive, descriptive (or both) model?" Tags="|r|predictive-modeling|xgboost|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27196" PostTypeId="1" CreationDate="2018-01-29T16:24:02.347" Score="6" ViewCount="1455" Body="&lt;p&gt;I have read &lt;a href=&quot;https://arxiv.org/pdf/1505.04597.pdf&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; paper about &lt;em&gt;U-Net&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This kind of network is quite similar to an autoencoder, in addition it has concatenations between the encoder and the decoder parts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know the meaning of this concatenations, and why the feature maps are cropped before this concatenation.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2020-08-15T09:12:33.797" Title="How does u-net work?" Tags="|cnn|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27197" PostTypeId="1" CreationDate="2018-01-29T16:46:30.830" Score="3" ViewCount="496" Body="&lt;p&gt;I have a 3D tensor on which I apply 2D convolutions. Sometimes, this 3D is padded both in width and height to have a fixed size. How could I apply masking (like with RNNs) so that the gradients associated with the weights of these padded cells don't get updates while backpropagating the error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="44785" LastEditorUserId="29169" LastEditDate="2021-02-13T16:43:29.730" LastActivityDate="2021-02-13T16:43:29.730" Title="Keras - Masking CNNs" Tags="|machine-learning|deep-learning|keras|tensorflow|convolutional-neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27198" PostTypeId="2" ParentId="27193" CreationDate="2018-01-29T17:43:30.063" Score="0" Body="&lt;p&gt;You have not specified that what neural network you are using but as comments, you should try to fit your data first. You have to try to find a model that learns your training data. For this purpose you don't have to increase the number of data, at least not at this stage. You should try to find a good model which suits your data. For this purpose you have to change the hyper-parameters of your neural network, e.g. number of layers or number of neurons in layers. You can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt; which the former can help you and the latter helps you understand the features learned by &lt;code&gt;CNNs&lt;/code&gt; in case you are using them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For using &lt;code&gt;F1&lt;/code&gt; score in &lt;code&gt;Keras&lt;/code&gt; I've not seen but you can implement it and pass it to compile method, take a look at &lt;a href=&quot;https://stackoverflow.com/q/45411902/5120235&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-29T17:43:30.063" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27199" PostTypeId="2" ParentId="27179" CreationDate="2018-01-29T17:44:55.357" Score="1" Body="&lt;p&gt;Here's my take on each of your points.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You have very sparse data. Are you storing these data optimally in a sparse object, for example a csr sparse matrix if you're using Keras? This is more likely to affect training time than accuracy, but is something to think about.&lt;/li&gt;&#xA;&lt;li&gt;5000 training examples and 500 testing examples sounds alright to me, except you may have too few data to fit the type of model you have (you may be severely underfitting). Try a simpler model and see if you can improve results with that (try something stupidly simple like an MLP with one layer and see what happens).&lt;/li&gt;&#xA;&lt;li&gt;This seems the most likely problem. My first guess is this has to do with the unbalanced classes. If you're using a metric like accuracy to evaluate training at each epoch, I'd recommend instead using something like &lt;code&gt;average_precision&lt;/code&gt; or &lt;code&gt;roc_auc_score&lt;/code&gt; or &lt;code&gt;f1&lt;/code&gt;, all available through scikit-learn. If you're  using Keras, try using &lt;code&gt;class_weights&lt;/code&gt; as well, which will weight underrepresented classes higher in the loss function, essentially biasing the algorithm to consider underrepresented classes on an equal playing field. If you're not using Keras, try implementing some similar class weighting scheme.&lt;/li&gt;&#xA;&lt;li&gt;Batch normalization is less popular now than it used to be. It's worth trying the architecture with and without batch normalization to see if it provides a clear benefit.&lt;/li&gt;&#xA;&lt;li&gt;Yes, if your data are purely random then you will not detect a signal at all. This goes back to point number 2 (try a simpler model and see if you can detect a pattern). You can also try visualizing some of your samples to see if you can visually see a difference between the classes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With regard to Xavier initialization, I don't think that's the likely cause of the effect you're seeing. The type of initialization can affect results, of course, but from what you're describing, I strongly suspect this is due to too little signal from the data or the unbalanced classes problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps!&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-01-29T17:44:55.357" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27201" PostTypeId="2" ParentId="27179" CreationDate="2018-01-29T17:51:42.743" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;You more than likely do not have enough training data for a neural network.  &lt;/li&gt;&#xA;&lt;li&gt;Your class imbalance problem is probably an issue.  Instead of using accuracy as a measurement trying some type of F-score.&lt;/li&gt;&#xA;&lt;li&gt;Batch normalization should be applied between the convolution layer and the activation function.  &lt;/li&gt;&#xA;&lt;li&gt;If you think you have a vanishing or dying activation problem, plot the gradients or the sum of gradients.  It'll give you an idea if you're right or not.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-01-29T17:51:42.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27202" PostTypeId="1" CreationDate="2018-01-29T18:02:02.633" Score="3" ViewCount="2038" Body="&lt;p&gt;I have a trained model. For single prediction I restore the last checkpoint and pass a single image for prediction but the result is the same for every row. Does anyone have a clue of what might be wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the code for prediction: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nV906.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nV906.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is the output:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jyVLA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jyVLA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45389" LastEditorUserId="29575" LastEditDate="2018-01-29T18:34:54.800" LastActivityDate="2018-01-29T18:49:27.093" Title="Tensorflow predicting same value for every row" Tags="|deep-learning|tensorflow|prediction|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27204" PostTypeId="2" ParentId="27194" CreationDate="2018-01-29T18:48:28.730" Score="1" Body="&lt;p&gt;You're right that it is a predictive model, but analysing feature importance, partial dependence and other metrics/plots are intended to allow you to gain an understanding of the classification process of models which are often very black-box in nature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can use this to gain some understanding of the underlying data, but I'd be more inclined to say that the primary reason for looking at feature importance is to understand your model rather than to understand your data.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2018-01-29T18:48:28.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27205" PostTypeId="2" ParentId="27202" CreationDate="2018-01-29T18:49:27.093" Score="1" Body="&lt;p&gt;I think I found the problem. I need my input to have shape of (1, 256, 256, 1), thats why I performed image = image[...,0][...,None]/255 but doing so changed all the values to 0&lt;/p&gt;&#xA;" OwnerUserId="45389" LastActivityDate="2018-01-29T18:49:27.093" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27206" PostTypeId="2" ParentId="27196" CreationDate="2018-01-29T18:56:03.630" Score="3" Body="&lt;p&gt;Excerpts from the very same source tell the answers:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;In order to localize, high resolution features from the contracting path are combined with the upsampled output.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;A successive convolution layer can then learn to assemble a more precise output based on this information.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;--&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The cropping is necessary due to the loss of border pixels in every convolution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So, the combined data adds precision of U-net (reduces error) and cropping filters blurry borders out of that combined data.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;For the pixels on the border of image matrix, some elements of the kernel might stands out of the image matrix and therefore does not have any corresponding element from the image matrix.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;(kernel is the matrix with which the convolution of image is done to produce the output)&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://machinelearninguru.com/computer_vision/basics/convolution/image_convolution_1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source of edit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="98307" LastEditDate="2020-08-15T09:12:33.797" LastActivityDate="2020-08-15T09:12:33.797" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27207" PostTypeId="1" CreationDate="2018-01-29T18:56:25.670" Score="2" ViewCount="171" Body="&lt;p&gt;I am stuck in a problem wherein I have hierarchical data, say $A-&amp;gt;B-&amp;gt;C$(smaller to biggest), and the smallest unit of data is a block(A consists of multiple blocks, B consists of multiple A's, and C consists of multiple B's), and I want to classify blocks into labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the block labels for each group of A is independent of block labels for another group of A, however the &quot;trends or patterns&quot; followed by data could be similar and that is what is to be learnt. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complexity I am facing is variable input sizes. I cannot possibly train single neural networks for groups of A, since its a large number. So, I am thinking in terms of groups at level B, but how could I create a scheme which could handle these variable input sizes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each block is represented by a one dimensional array of the total number of labels in the group of A it belongs to.Also, I have the information for hierarchy for every block(smallest unit) possible. Any help would be appreciated. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="45391" LastEditorUserId="40853" LastEditDate="2018-01-30T01:04:16.970" LastActivityDate="2018-01-30T01:04:16.970" Title="Using neural networks for classification in Hierarchical data" Tags="|machine-learning|neural-network|keras|hierarchical-data-format|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27208" PostTypeId="1" CreationDate="2018-01-29T19:45:28.290" Score="2" ViewCount="98" Body="&lt;p&gt;I want to classify product images into 8 discrete classes. For several reasons the number of input images need to be as small as possible. Related to this I have 2 questions: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What strategies can I employ to reduce the number or required images&lt;/li&gt;&#xA;&lt;li&gt;What is the magnitude of images per classes needed (A) in a simple scenario where the image is always isolated, see below or (B) in a complex scenario where the image is a scenery (also below). I know this is hard to tell in general but a rough number would be fine.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Additional information&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Architecture&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently a CNN seems to be most fitting for the problem. I am also thinking about a pre-trained network (removing the deeper layers of a powerful network such as AlexNet).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine I am trying to classify dinner tables by the type of legs they have. In the simple scenario images from 3 different classes could look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TRqBK.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TRqBK.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Class: regular&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eqqaW.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eqqaW.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Class: outward &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wY25H.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wY25H.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Class: Wire&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a complex scenario they might look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7gkvY.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7gkvY.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Techniques to reduce input data needed&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a couple of ideas I have to reduce the need for data (please add your own):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use a pre-trained network (good idea or not?)&lt;/li&gt;&#xA;&lt;li&gt;Pre-process images (how? fiddle with image orientation?)&lt;/li&gt;&#xA;&lt;li&gt;Does it make sense to change the images more deeply (such as chopping off the top 50% since I only care for the legs?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am looking forward for your thoughts!&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-01-30T13:06:30.920" LastActivityDate="2018-05-01T09:11:53.700" Title="Image classification: Strategies for minimal input count" Tags="|neural-network|image-classification|multiclass-classification|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27210" PostTypeId="2" ParentId="27152" CreationDate="2018-01-29T19:56:12.030" Score="1" Body="&lt;p&gt;I think, Rapidminer can be used for that. However, the free version can only load 10,000 rows of data.&lt;/p&gt;&#xA;" OwnerUserId="33095" LastActivityDate="2018-01-29T19:56:12.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27212" PostTypeId="1" AcceptedAnswerId="27220" CreationDate="2018-01-29T21:00:43.117" Score="3" ViewCount="27609" Body="&lt;p&gt;Working with &lt;code&gt;Sklearn&lt;/code&gt; stratified kfold split, and when I attempt to split using multi-class, I received on error (see below). When I tried and split using binary, it works no problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_classes = len(np.unique(y_train))&#xA;y_train_categorical = keras.utils.to_categorical(y_train, num_classes)&#xA;kf=StratifiedKFold(n_splits=5, shuffle=True, random_state=999)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;splitting data into different folds&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i, (train_index, val_index) in enumerate(kf.split(x_train, y_train_categorical)):&#xA;   x_train_kf, x_val_kf = x_train[train_index], x_train[val_index]&#xA;   y_train_kf, y_val_kf = y_train[train_index], y_train[val_index]&#xA;&#xA;ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way I can used KFold with multi-class?&lt;/p&gt;&#xA;" OwnerUserId="32754" LastEditorUserId="28175" LastEditDate="2018-01-30T02:56:03.420" LastActivityDate="2018-08-11T00:24:43.640" Title="StratifiedKFold: ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead" Tags="|machine-learning|neural-network|scikit-learn|cross-validation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27213" PostTypeId="2" ParentId="27122" CreationDate="2018-01-29T21:03:00.893" Score="0" Body="&lt;p&gt;I guess I wasn't logged in when I first asked this question. I am now using the most current version of orange canvas available when installed via the 64 bit Miniconda installer. Prior to installing the Miniconda version, I had tried updating all relevant packages via pip, and was using Orange Canvas 3.8. The resulting error remains. If anyone can get this Prototype to work in Orange Canvas (i.e. using one of the add-on datasets) please share an image of what the pipeline and results look like, and state what version of Orange Canvas is being used. Many thanks in advance and best regards!&lt;/p&gt;&#xA;" OwnerUserId="45292" LastEditorUserId="45292" LastEditDate="2018-01-30T00:06:25.600" LastActivityDate="2018-01-30T00:06:25.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27217" PostTypeId="1" AcceptedAnswerId="50936" CreationDate="2018-01-30T00:35:51.420" Score="11" ViewCount="1759" Body="&lt;p&gt;Attention mechanisms in RNNs are reasonably common to sequence to sequence models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output vectors from the encoder network. This is used to produce a new input vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is that the learned weight vectors $\alpha$ must be a fixed size vector because it's treated as learned weights, but it's applied to a variable length sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If someone could help me understand this particular mechanism I'd appreciate it.&lt;/p&gt;&#xA;" OwnerUserId="24792" LastEditorUserId="67328" LastEditDate="2019-04-26T22:28:42.180" LastActivityDate="2022-07-23T14:36:24.010" Title="How do attention mechanisms in RNNs learn weights for a variable length input" Tags="|neural-network|rnn|sequence-to-sequence|attention-mechanism|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27218" PostTypeId="1" AcceptedAnswerId="27223" CreationDate="2018-01-30T00:36:19.327" Score="0" ViewCount="91" Body="&lt;p&gt;Even after all these years of data science from 2010 to 2018, why is there no general framework for putting a predictive model into production?&lt;/p&gt;&#xA;" OwnerUserId="45297" LastActivityDate="2018-01-30T04:09:24.700" Title="Putting a predictive model into production" Tags="|data-product|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-17T19:01:24.253" ContentLicense="CC BY-SA 3.0" />
  <row Id="27219" PostTypeId="2" ParentId="19207" CreationDate="2018-01-30T02:22:23.907" Score="2" Body="&lt;p&gt;Firstly, you need to know how RPART works, it is a decision tree which works by building a tree based on the features available i.e, Trees (also called decision trees, recursive partitioning) are a simple yet powerful tool in predictive statistics. The idea is to split the co-variable space into many partitions and to fit a constant model of the response variable in each partition. &#xA;In case of regression, the mean of the response variable in one node would be assigned to this node. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The structure is similar to a real tree (from the bottom up): there is a root, where the first split happens. After each split, two new nodes are created (assuming we only make binary splits). Each node only contains a subset of the observations. The partitions of the data, which are not split any more, are called terminal nodes or leafs. This simple mechanism makes the interpretation of the model pretty easy. to explain it with an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now coming to your scenario, as you have mentioned it has 400+ features, now it start with a target variable and then divides into 2 parts, and goes to feature 1 and divides based on the value present in it like 0 or 1(for example 0 on the left node and 1 on the right node) and then it goes on till the last feature(sometimes it might not go till last feature) like that literally 400+ levels are generated for each and every record(but not 400 everytime because some records are similar, so those records gets generalized). So now you imagine for 400+ features, 7000 records, how big the tree might become and how hard it would be to generalize the results. So RPART couldn't give any result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, how did by using RF we achieved the results?&#xA;RF works as Ensemble of trees i.e., it splits the data at row level and column level by which the tree generated are small and it takes ensemble of all the trees and gives an average(other ways too like voting etc) of all the small trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go through these links for better understanding of &lt;a href=&quot;https://cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;RPART&lt;/a&gt;, &lt;a href=&quot;https://www.r-bloggers.com/classification-trees-using-the-rpart-function/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go through these links for better understanding of &lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/random-forests-explained-intuitively&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomForest&lt;/a&gt;, &lt;a href=&quot;https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-2&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion to get better results using Rpart is possible by reducing the number of features by applying PCA or any Dimensionality Reduction algorithms. &#xA;Since you have obtained desired results using different algorithms so the above is for your understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-01-30T03:48:47.563" LastActivityDate="2018-01-30T03:48:47.563" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27220" PostTypeId="2" ParentId="27212" CreationDate="2018-01-30T02:47:54.947" Score="2" Body="&lt;p&gt;There is an easier way instead of using loops. &lt;code&gt;Scikit&lt;/code&gt; provides &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.cross_val_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross_val_score&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import KFold, cross_val_score&#xA;k_fold = KFold(len(y), n_folds=10, shuffle=True, random_state=0)&#xA;clf = &amp;lt;any classifier&amp;gt;&#xA;print cross_val_score(clf, X, y, cv=k_fold, n_jobs=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The topic also has been discussed &lt;a href=&quot;https://stackoverflow.com/questions/16379313/how-to-use-the-a-k-fold-cross-validation-in-scikit-with-naive-bayes-classifier-a/38711253#38711253&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which has a code snippet which may help you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import KFold&#xA;X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])&#xA;y = np.array([1, 2, 3, 4])&#xA;kf = KFold(n_splits=2)&#xA;kf.get_n_splits(X)&#xA;&#xA;print(kf)  &#xA;&#xA;for train_index, test_index in kf.split(X):&#xA;   print(&quot;TRAIN:&quot;, train_index, &quot;TEST:&quot;, test_index)&#xA;   X_train, X_test = X[train_index], X[test_index]&#xA;   y_train, y_test = y[train_index], y[test_index]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which &lt;em&gt;The first n_samples % n_splits folds have size n_samples // n_splits + 1, other folds have size n_samples // n_splits, where n_samples is the number of samples.&lt;/em&gt;  &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-30T03:01:39.387" LastActivityDate="2018-01-30T03:01:39.387" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27221" PostTypeId="1" AcceptedAnswerId="27474" CreationDate="2018-01-30T02:50:37.800" Score="0" ViewCount="1052" Body="&lt;p&gt;How to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM&lt;/a&gt; in Orange for sentiments analysis? We will generate sentiments using sentiments analysis widget but now how to use SVM and how to teach SVM in Orange? i am use xlsx file format then it work good below example ... but i want to use text file (csv) format in which generate error target variable set how to remove this error&#xA;&lt;a href=&quot;https://i.stack.imgur.com/AxTFu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AxTFu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41909" LastEditorUserId="41909" LastEditDate="2018-02-06T14:41:17.480" LastActivityDate="2022-11-03T21:20:50.370" Title="How to use SVM in Orange" Tags="|machine-learning|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27223" PostTypeId="2" ParentId="27218" CreationDate="2018-01-30T04:09:24.700" Score="1" Body="&lt;p&gt;Depending on what exactly you mean by &lt;strong&gt;framework&lt;/strong&gt;, I would argue that there is. Using a REST interface to serve a production model at inference time is pretty close to a general serving framework at this point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are a potentially infinite number of use cases that would diverge from this standard even if one categorically existed. Some mobile apps using image recognition models on personal data that the user doesn't want transmitted to the cloud are pushing the envelope of ways to compress formerly huge neural network architectures into ones that easily fit into mobile-sized memory, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're talking software framework, it shouldn't be surprising that a field of programming practice (or, rather, plethora of fields) that find needs for a variety of &lt;em&gt;languages&lt;/em&gt; and even &lt;em&gt;ecosystems within languages&lt;/em&gt; would have difficulty forming a consensus around a single aporoach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's likely that deployment frameworks have yet to build in enough benefit to justify a reduction in general flexibility to justify their use.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2018-01-30T04:09:24.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27225" PostTypeId="1" CreationDate="2018-01-30T06:30:54.517" Score="5" ViewCount="10164" Body="&lt;p&gt;In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What values should we initialize them with? How should we initialize them?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the &lt;a href=&quot;http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html&quot; rel=&quot;noreferrer&quot;&gt;PyTorch tutorial&lt;/a&gt;, it simply initializes zeros to the hidden states. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is initializing zero the usual way of initializing hidden states in RNN seq2seq networks?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about &lt;a href=&quot;https://jamesmccaffrey.wordpress.com/2017/06/21/neural-network-glorot-initialization/&quot; rel=&quot;noreferrer&quot;&gt;glorot initialization&lt;/a&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a single-layer vanilla RNN wouldn't the fan-in and fan-out be equals to $(1 + 1)$ which gives a variance of $1$ and the gaussian distribution with $mean=0$ gives us a uniform distribution of $0$s.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for-each input-hidden weight&#xA;  variance = 2.0 / (fan-in +fan-out)&#xA;  stddev = sqrt(variance)&#xA;  weight = gaussian(mean=0.0, stddev)&#xA;end-for&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For single layer encoder-decoder architecture with attention, if we use glorot, we'll get a very very small variance when initializing the decoder hidden state since the fan-in would include the attention which is mapped to all possible vocabulary from the encoder output. So we result in a gaussian mean of ~= 0 too since stdev is really really small. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What other initialization methods are there, esp. for the use on RNN seq2seq models?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="122" LastEditorUserId="122" LastEditDate="2018-01-30T06:46:23.120" LastActivityDate="2019-08-23T04:04:35.460" Title="How/What to initialize the hidden states in RNN sequence-to-sequence models?" Tags="|pytorch|rnn|sequence-to-sequence|glorot-initialization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27226" PostTypeId="2" ParentId="10108" CreationDate="2018-01-30T06:31:21.030" Score="2" Body="&lt;p&gt;Most programming languages and markup languages have a relatively simple syntax, so it is not usually necessary to use machine learning techniques to parse these languages. In machine learning, the process of automatically learning a formal grammar is also known as &lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;as_sdt=0%2C47&amp;amp;q=%22grammar+induction%22+parser&amp;amp;btnG=&quot; rel=&quot;nofollow noreferrer&quot;&gt;grammar induction&lt;/a&gt;. Several &lt;a href=&quot;https://scholar.google.com/scholar?hl=en&amp;amp;as_sdt=0%2C47&amp;amp;q=%22adaptive+parser%22+%22%28machine+OR+grammar%29+learning%22&amp;amp;btnG=&quot; rel=&quot;nofollow noreferrer&quot;&gt;adaptive parsers&lt;/a&gt; have been designed for this purpose.&lt;/p&gt;&#xA;" OwnerUserId="45414" LastEditorUserId="45414" LastEditDate="2018-01-30T06:56:42.813" LastActivityDate="2018-01-30T06:56:42.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27227" PostTypeId="1" CreationDate="2018-01-30T07:22:05.493" Score="1" ViewCount="70" Body="&lt;p&gt;I am building a project for fun - reverse astrology!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am making a dataset like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;planet_name, life_impact&#xA;&#xA;Saturn, impact1&#xA;&#xA;Saturn, impact2&#xA;&#xA;Sun, impact2&#xA;&#xA;Sun, impact3&#xA;&#xA;Sun, impact4&#xA;&#xA;Mars, impact 5&#xA;&#xA;Mars, impact 6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc…&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have trained a bayesian network over this^. I want the network to learn metadata based learnings also.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ie.&#xA;Right now for positive evidence “impact 1” network will contribute probability 1/2 to Saturn and 1/3 to Sun.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I have metadata(duration) for feature: life_impact&#xA;duration for impact2 for Saturn is 1/5&#xA;duration for impact2 for Sun is 5/5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should change the probability contribution to sun/saturn: Now contribution should change as per the value of duration also.&#xA;eg. &#xA;impact 1 with duration 1 should contribute more to Saturn and less to Sun, similarly impact 1 with duration 4 or 5 should contribute more to Sun.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that metadata duration might not be there for all impacts, something else might be present like “relevance”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I design my features to accommodate this?&lt;/p&gt;&#xA;" OwnerUserId="45371" LastActivityDate="2018-01-30T07:22:05.493" Title="Training with feature metadata - bayesian network (naive bayes)" Tags="|machine-learning|python|naive-bayes-classifier|bayesian-networks|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-01T18:19:15.917" ContentLicense="CC BY-SA 3.0" />
  <row Id="27228" PostTypeId="2" ParentId="21889" CreationDate="2018-01-30T08:47:18.860" Score="1" Body="&lt;p&gt;To add to @Andrey, you should be able to achieve an accuracy of 98+% accuracy on the training with a simple 2 layer nn with 250-350 hidden nodes, sigmoid activation, learning rate of 0.1. I ran my test 1,000,000 iterations via stochastic gradient descent. However, I'm very sure it converged far earlier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example showing how I structured the experiment and results: &lt;a href=&quot;https://github.com/kennycason/neural_network_kotlin/blob/master/src/test/kotlin/com/kennycason/nn/MNISTBackErrorPropagationAccuracyTest.kt&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45416" LastEditorUserId="45416" LastEditDate="2018-01-30T10:14:28.157" LastActivityDate="2018-01-30T10:14:28.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27229" PostTypeId="1" CreationDate="2018-01-30T09:03:13.013" Score="1" ViewCount="125" Body="&lt;p&gt;Consider a platform for content recommendation based on the user history. The contents are books and articles and by history I mean what the user has read, what he has shared and so on. I know that there are many studies on how to train a recommendation system, however I don't know what to do to &quot;start&quot; (the platform has not been launched yet and the dataset is empty). Are there techniques that consider an initialization phase where no data is available?&lt;/p&gt;&#xA;" OwnerUserId="45420" LastActivityDate="2024-01-04T14:06:30.707" Title="Initialize a recommender system with no dataset" Tags="|dataset|recommender-system|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27230" PostTypeId="1" AcceptedAnswerId="27240" CreationDate="2018-01-30T09:03:20.220" Score="1" ViewCount="2530" Body="&lt;p&gt;I want to create a data model out of a &lt;code&gt;.csv&lt;/code&gt; file using Python. I mean to create dependencies, for example the primary key and stuff such that I can check if the new &lt;code&gt;.csv&lt;/code&gt; complies with the given data model. I would appreciate some suggestions regarding how to do that, the libraries, frameworks etc&lt;/p&gt;&#xA;" OwnerUserId="44701" LastActivityDate="2018-02-01T17:42:25.337" Title="Creating data model out of .csv file using Python" Tags="|python|dataset|csv|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27231" PostTypeId="2" ParentId="11356" CreationDate="2018-01-30T09:16:45.463" Score="17" Body="&lt;p&gt;Sometime, when the dataframes to combine do not have the same order of columns, it is better to &lt;code&gt;df2.select(df1.columns)&lt;/code&gt; in order to ensure both df have the same column order before the union.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import functools &#xA;&#xA;def unionAll(dfs):&#xA;    return functools.reduce(lambda df1,df2: df1.union(df2.select(df1.columns)), dfs) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df1 = spark.createDataFrame([[1,1],[2,2]],['a','b'])&#xA;# different column order. &#xA;df2 = spark.createDataFrame([[3,333],[4,444]],['b','a']) &#xA;df3 = spark.createDataFrame([555,5],[666,6]],['b','a']) &#xA;&#xA;unioned_df = unionAll([df1, df2, df3])&#xA;unioned_df.show() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/F22Mq.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F22Mq.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;else it would generate the below result instead.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from functools import reduce  # For Python 3.x&#xA;from pyspark.sql import DataFrame&#xA;&#xA;def unionAll(*dfs):&#xA;    return reduce(DataFrame.unionAll, dfs) &#xA;&#xA;unionAll(*[df1, df2, df3]).show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VJ4a3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VJ4a3.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45421" LastEditorUserId="29169" LastEditDate="2021-02-09T04:26:26.343" LastActivityDate="2021-02-09T04:26:26.343" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27232" PostTypeId="1" CreationDate="2018-01-30T09:53:38.497" Score="3" ViewCount="139" Body="&lt;p&gt;This is the equation that is given in &lt;a href=&quot;http://cvxopt.org/examples/tutorial/lp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the example&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DQRWC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DQRWC.png&quot; alt=&quot;The equation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the code to replicate it in python is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from cvxopt import matrix, solvers&#xA;&amp;gt;&amp;gt;&amp;gt; A = matrix([ [-1.0, -1.0, 0.0, 1.0], [1.0, -1.0, -1.0, -2.0] ])&#xA;&amp;gt;&amp;gt;&amp;gt; b = matrix([ 1.0, -2.0, 0.0, 4.0 ])&#xA;&amp;gt;&amp;gt;&amp;gt; c = matrix([ 2.0, 1.0 ])&#xA;&amp;gt;&amp;gt;&amp;gt; sol=solvers.lp(c,A,b)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone please explain how are the the data given to variables &lt;code&gt;a, b, c&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="45423" LastEditorUserId="29575" LastEditDate="2018-01-30T13:52:04.023" LastActivityDate="2019-04-29T04:04:55.857" Title="need explanation on how an equation is being converted to cvxopt logic in solver.lq" Tags="|python|convergence|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27233" PostTypeId="2" ParentId="27232" CreationDate="2018-01-30T10:14:51.960" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Media has mentioned values of A,b,c are passed through those matrices.&#xA;To understand it better I'm naming the constraints from top to bottom i.e., 1st constraint as constraints-1,......constraint-4. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, lets talk about matrix b which consists of all the values on the right hand side of the constraints i.e., 1,2,0,4. He converted all the constraints to $⩽$ by multiplying with -1 on both sides. So he multiplied with -1 for constraints-2 and 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the constraints-2 has changed from $x_1 + x_2 ⩾ 2$ to $-x_1 - x_2 ⩽ -2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Values of Matrix b would be 1,-2,0(-0),4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now lets go through the constraints i.e., &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;$-x_1 + x_2 ⩽ 1 $ for the coefficient of $x_1,x_2$ are -1 and 1. The values of Matrix A row 1 values are -1,1 &lt;/li&gt;&#xA;&lt;li&gt;Similarly for row 2 to row 4 as you have 4 contraints.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finally our objective function coefficients, are the values of c i.e., 2 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you have any more doubts. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-01-30T10:14:51.960" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27234" PostTypeId="1" CreationDate="2018-01-30T10:42:54.193" Score="1" ViewCount="585" Body="&lt;p&gt;The question is strictly related to&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/5990/what-is-a-good-way-to-transform-cyclic-ordinal-attributes/6335#6335&quot;&gt;What is a good way to transform Cyclic Ordinal attributes?&lt;/a&gt;&#xA;and&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/13567/ways-to-deal-with-longitude-latitude-feature?rq=1&quot;&gt;Ways to deal with longitude/latitude feature&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They presented a very clear answer about the approach to normalise hour variable and latlong variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The addition of this question is about how to approach normalization having both &lt;code&gt;weekday&lt;/code&gt; and &lt;code&gt;hour&lt;/code&gt; features. I wonder this because they are different granular level of the same information, hence, &lt;strong&gt;strictly related.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to run a k-means algorithm combining the normalised weekday and hour features with other traditional numeric features.&lt;/p&gt;&#xA;" OwnerUserId="43110" LastActivityDate="2018-02-09T14:20:03.030" Title="Clustering combining numeric features and weekday &amp; hour cyclic features" Tags="|machine-learning|feature-extraction|feature-scaling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27235" PostTypeId="2" ParentId="27230" CreationDate="2018-01-30T10:53:27.700" Score="2" Body="&lt;p&gt;In my opinion this is probably achieved fastest with a rule set. You can easily load and manipulate data via pandas. Then you just have to determine what are rules. But without any details its difficult to go more into details. Regular expressions might be a powerful tool to use but that is speculation given the sparse info you provide. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastEditorUserId="29217" LastEditDate="2018-01-30T12:09:15.743" LastActivityDate="2018-01-30T12:09:15.743" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27236" PostTypeId="1" CreationDate="2018-01-30T10:56:12.100" Score="5" ViewCount="2616" Body="&lt;p&gt;What is actually the best neural network architecture for the classic &lt;code&gt;MNIST&lt;/code&gt; digit classifying task? I couldn't find any that would claim to be the winner...&lt;/p&gt;&#xA;" OwnerUserId="21560" LastEditorUserId="29169" LastEditDate="2021-02-13T16:43:18.483" LastActivityDate="2021-02-13T16:43:18.483" Title="What is the state-of-the art ANN architecture for MNIST?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|mnist|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27237" PostTypeId="2" ParentId="27236" CreationDate="2018-01-30T11:24:06.483" Score="5" Body="&lt;p&gt;Actually, what I'm going to discuss is not an architecture and is like a module used in networks. It performs really well on character based data-sets like &lt;code&gt;MNIST&lt;/code&gt; although there is other functionality for them. There is a paper called &lt;em&gt;Spatial Transformer Networks&lt;/em&gt; written by &lt;em&gt;Max Jaderberg&lt;/em&gt; et al. It tries to introduce an alternative for pooling layers. What it does is tries to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional networks to &lt;em&gt;really&lt;/em&gt; be spatial invariant. The work is amazing and the paper is easy to be read. Based on experience, using these modules may reduce the number of neurons and layers, because networks employing them won't try to learn extra stuff of the inputs, like applied transformations to the inputs. The reason is that the net will try to learn the &lt;strong&gt;canonical&lt;/strong&gt; shape of inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/b6KQh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/b6KQh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(a) shows the arbitrary input to the network (b) shows what spatial transformer has done and finally (c) is the output of the spatial transformer which can be used through other layers of the networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the significant achievements of this module is that it tries to enhance distorted inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hqj67.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hqj67.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its performance can be seen &lt;a href=&quot;https://drive.google.com/file/d/0B1nQa_sA3W2iN3RQLXVFRkNXN0k/view&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="29169" LastEditDate="2019-06-07T06:19:08.700" LastActivityDate="2019-06-07T06:19:08.700" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="27239" PostTypeId="2" ParentId="5427" CreationDate="2018-01-30T11:47:52.163" Score="1" Body="&lt;p&gt;There is a very common approach to deal with imbalanced datasets, called SMOTE, which generates synthetic samples from the minority class. It works by perturbing minority samples using the differences with its neighbors (multiplied by some random number between 0 and 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a quote from thew original paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Synthetic samples&#xA;  are generated in the following way: Take the difference between the feature vector (sample)&#xA;  under consideration and its nearest neighbor. Multiply this difference by a random number&#xA;  between 0 and 1, and add it to the feature vector under consideration.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can find more information &lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="39289" LastEditorUserId="29575" LastEditDate="2018-01-30T13:32:04.743" LastActivityDate="2018-01-30T13:32:04.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27240" PostTypeId="2" ParentId="27230" CreationDate="2018-01-30T12:28:40.460" Score="2" Body="&lt;p&gt;Adding on top of @El Burro's  answer, most of the training/testing proof of concepts of model making happens on manipulating &lt;code&gt;DataFrame&lt;/code&gt; which provides easy functionalities like chaining operations, broadcasting, filling missing values etc and &lt;code&gt;pandas&lt;/code&gt; is one such library. It has datatype inferring too and it uses python stack &lt;code&gt;numpy&lt;/code&gt; which is fast manipulating arrays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that, as you asked if you want to check format of incoming data that is being passed to the model. You can use &lt;code&gt;pandas&lt;/code&gt; here like (this is just a demo that you can achieve all kinds of stuff, there might be a better way to pull this off)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt; df1 = pd.DataFrame({'a': [1, 2, 3], 'b': ['a', 'b', 'c']})&#xA;&amp;gt;&amp;gt; df1&#xA;   a  b&#xA;0  1  a&#xA;1  2  b&#xA;2  3  c&#xA;&amp;gt;&amp;gt; df2 = pd.DataFrame({'a': [1, 2, 3], 'b': [1.0, 2.0, 3.0]})&#xA;&amp;gt;&amp;gt; df2&#xA;   a  b&#xA;0  1  1.0&#xA;1  2  2.0&#xA;2  3  3.0&#xA;&amp;gt;&amp;gt; df1.dtypes&#xA;a     int64&#xA;b    object&#xA;dtype: object&#xA;&amp;gt;&amp;gt; df2.dtypes&#xA;a      int64&#xA;b    float64&#xA;dtype: object&#xA;&amp;gt;&amp;gt; df1.dtypes.to_dict() == df2.dtypes.to_dict()&#xA;False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-01-30T12:28:40.460" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27241" PostTypeId="1" AcceptedAnswerId="27394" CreationDate="2018-01-30T13:30:44.470" Score="4" ViewCount="112" Body="&lt;p&gt;I'll assume most of us know about cross-validation and are accustomed to using Training and Test sets. However, I've also seen references to a third set that you use to test a completed model. It's the test to see how your model does against a dataset that it hasn't seen before; helps determine if your model has been overly-fit to the training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen this dataset referred to everything from &quot;the third dataset&quot;, &quot;the wild dataset&quot;, &quot;the real-world dataset&quot;. So what is the appropriate name for this third dataset? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastEditorUserId="23240" LastEditDate="2018-01-30T13:42:00.170" LastActivityDate="2018-02-03T08:40:13.223" Title="What is the appropriate name for this dataset?" Tags="|dataset|data|cross-validation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27244" PostTypeId="1" CreationDate="2018-01-30T14:56:06.260" Score="1" ViewCount="437" Body="&lt;p&gt;I am trying to program &lt;code&gt;incremental stochastic gradient descent (ISGD) algorithm&lt;/code&gt; in logistic regression. Initially, I coded respective &lt;code&gt;logistic regression' loss function&lt;/code&gt; and its &lt;code&gt;gradient&lt;/code&gt;, also got some idea to proceed rest of workflow. But, I have no idea how to apply sequential operation in incremental stochastic gradient descent algorithm which can be used in the respective logistic regression. How can I implement the sequential operation in incremental SGD? Any way to make this happen in Python? How can I do that? Any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Initial implementation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import scipy as sp&#xA;import sklearn as sl&#xA;from scipy import special as ss&#xA;from  sklearn import datasets&#xA;&#xA;## load input dataset&#xA;X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')&#xA;X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt', &#xA;                                          n_features=X_train.shape[1])&#xA;&#xA;# logistic loss function&#xA;def lossFunc(x_i,y_i,w):&#xA;    w.resize((w.shape[0],1))&#xA;    y_i.resize((y_i.shape[0],1))&#xA;&#xA;    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y_i* np.dot(x_i,w,))))&#xA;    rslt=np.float(lossFnc)&#xA;    return rslt&#xA;&#xA;#gradient function&#xA;&#xA;def gradFnc(X,y,w):&#xA;    w.resize((w.shape[0],1))&#xA;    y.resize((y.shape[0],1))&#xA;&#xA;    gradF1=-y*np.nan_to_num(ss.expm1(-y))&#xA;    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))&#xA;    gradF3=gradF2.resize(gradF2.shape[0],)&#xA;    return gradF3&#xA;&#xA;class ISGD:&#xA;    def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):&#xA;        self.w=None&#xA;        self.learnRate=learnRate&#xA;        self.verbose=verbose&#xA;        self.num_iter=num_iter&#xA;&#xA;&#xA;    def fitt(self, X,y):&#xA;        n,d=X.shape&#xA;        self.w=np.zeros(shape=(d,))&#xA;&#xA;        for i in range(self.num_iter):&#xA;            print (&quot;\n:&quot;, &quot;Iteration:&quot;, i)&#xA;&#xA;            grd=gradFnc(self.w, X,y)&#xA;            grd.resize((grd.shape[0],1))&#xA;            self.w=self.w-grd&#xA;            print &quot;Loss:&quot;, lossFunc(self.w,X,y)&#xA;&#xA;        return self&#xA;&#xA;def predict(x_i, w):&#xA;        y_hat=w[0]&#xA;        for idx in range(len(x_i)-1):&#xA;            y_hat+=w[i+1]*x_i[idx]&#xA;        return 1.0/(1.0+np.nan_to_num(ss.expm1(-y_hat)))&#xA;&#xA;def update_weights(x_i, y_i,w):&#xA;    lr=0.8&#xA;    yhat=predict(x_i, w)&#xA;    error=y_i-yhat&#xA;    return w+lr*(y_i-yhat)*x_i&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How to proceed rest of workflow?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is blog about &lt;a href=&quot;https://medium.com/@krishna_srd/parallel-machine-learning-with-hogwild-f945ad7e48a4&quot; rel=&quot;nofollow noreferrer&quot;&gt;HogWild! for parallel machine learning&lt;/a&gt;. The particular interpretation of incremental SGD can be found here: &lt;a href=&quot;http://papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-gradient-descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;hogwild! algorithm for logistic regression&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I have no idea how to apply sequential operation in incremental SGD which can be used in the respective logistic regression. How can I make this happen? Is there any efficient workaround to implement sequential incremental SGD algorithm for logistic regression? What is the efficient programming pipeline to accomplish the task that I stated above? Any more thoughts?&lt;/p&gt;&#xA;" OwnerUserId="45011" LastEditorUserId="1330" LastEditDate="2021-07-18T13:41:43.627" LastActivityDate="2021-07-18T13:41:43.627" Title="Any way to implement sequential incremental stochastic gradient descent algorithm in logistic regression?" Tags="|machine-learning|python|algorithms|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27245" PostTypeId="1" CreationDate="2018-01-30T14:58:10.897" Score="1" ViewCount="56" Body="&lt;p&gt;In NEAT (neuroevolution through augmenting topologies) algorithm description, an innovation number, e.g. id, is assigned to each gene so that genomes can be crossed over meaningfully: genes having corresponding innovation numbers are aligned and then crossover takes place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this innovation number global only within distinct genemoes ? Meaning that each genome has its own counter or is it the case that there is a unique counter for all genomes ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Description of algorithm: &lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume21/stanley04a-html/node3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume21/stanley04a-html/node3.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="34202" LastActivityDate="2018-03-01T19:51:30.373" Title="Gene innovation numbers in NEAT implementatoins" Tags="|machine-learning|neural-network|genetic-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27246" PostTypeId="2" ParentId="27136" CreationDate="2018-01-30T15:05:44.597" Score="1" Body="&lt;p&gt;&lt;em&gt;Since I asked this question here as well, I am pasting my answer to it here.&lt;/em&gt;&#xA;I used a different way to define the VAE loss, as demonstrated in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/keras-team/keras/blob/keras-2/examples/variational_autoencoder.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/blob/keras-2/examples/variational_autoencoder.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I changed it to allow for denoising of the data. It works now, but I'll have to play around with the hyperparameters to allow it to correctly reconstruct the original images.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import time&#xA;import sys&#xA;import os&#xA;&#xA;&#xA;from scipy.stats import norm&#xA;&#xA;from keras.layers import Input, Dense, Lambda&#xA;from keras.models import Model&#xA;from keras import backend as K&#xA;from keras import metrics&#xA;from keras.datasets import mnist&#xA;&#xA;from keras.callbacks import ModelCheckpoint&#xA;&#xA;filepath_for_w='denoise_by_VAE_weights_1.h5'&#xA;&#xA;&#xA;&#xA;###########&#xA;##########&#xA;experiment_dir= 'exp_'+str(int(time.time()))&#xA;os.mkdir(experiment_dir)&#xA;this_script=sys.argv[0]&#xA;from shutil import copyfile&#xA;copyfile(this_script, experiment_dir+'/'+this_script)&#xA;##########&#xA;###########&#xA;&#xA;&#xA;batch_size = 100&#xA;original_dim = 784&#xA;latent_dim = 2&#xA;intermediate_dim = 256&#xA;epochs = 10&#xA;epsilon_std = 1.0&#xA;&#xA;x = Input(batch_shape=(batch_size, original_dim))&#xA;h = Dense(intermediate_dim, activation='relu')(x)&#xA;z_mean = Dense(latent_dim)(h)&#xA;z_log_var = Dense(latent_dim)(h)&#xA;&#xA;&#xA;def sampling(args):&#xA;    z_mean, z_log_var = args&#xA;    epsilon = K.random_normal(shape=(batch_size, latent_dim), mean=0.,&#xA;                              stddev=epsilon_std)&#xA;    return z_mean + K.exp(z_log_var / 2) * epsilon&#xA;&#xA;# note that &quot;output_shape&quot; isn't necessary with the TensorFlow backend&#xA;z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])&#xA;&#xA;# we instantiate these layers separately so as to reuse them later&#xA;decoder_h = Dense(intermediate_dim, activation='relu')&#xA;decoder_mean = Dense(original_dim, activation='sigmoid')&#xA;h_decoded = decoder_h(z)&#xA;x_decoded_mean = decoder_mean(h_decoded)&#xA;&#xA;&#xA;def vae_loss(x, x_decoded_mean):&#xA;    xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)&#xA;    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)&#xA;    return xent_loss + kl_loss&#xA;&#xA;vae = Model(x, x_decoded_mean)&#xA;vae.compile(optimizer='rmsprop', loss=vae_loss)&#xA;&#xA;&#xA;&#xA;&#xA;# train the VAE on MNIST digits&#xA;(x_train, y_train), (x_test, y_test) = mnist.load_data()&#xA;&#xA;&#xA;#after loading the data, change to the new experiment dir&#xA;os.chdir(experiment_dir) #&#xA;##########################&#xA;&#xA;x_train = x_train.astype('float32') / 255.&#xA;x_test = x_test.astype('float32') / 255.&#xA;x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))&#xA;x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))&#xA;&#xA;&#xA;noise_factor = 0.5&#xA;&#xA;x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape) &#xA;x_test_noisy = np.clip(x_test_noisy, 0., 1.)&#xA;&#xA;&#xA;for i in range (10):&#xA;&#xA;    x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape) &#xA;    x_train_noisy = np.clip(x_train_noisy, 0., 1.)&#xA;&#xA;    checkpointer=ModelCheckpoint(filepath_for_w, monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='auto', period=1)&#xA;    vae.fit(x_train_noisy, x_train,&#xA;            shuffle=True,&#xA;            epochs=epochs,&#xA;            batch_size=batch_size,&#xA;            validation_data=(x_test_noisy, x_test),&#xA;            callbacks=[checkpointer])&#xA;    vae.load_weights(filepath_for_w) &#xA;&#xA;    #print (x_train.shape)&#xA;    #print (x_test.shape)&#xA;&#xA;    decoded_imgs = vae.predict(x_test,batch_size=batch_size)&#xA;    np.save('decoded'+str(i)+'.npy',decoded_imgs)&#xA;&#xA;&#xA;np.save('tested.npy',x_test_noisy)&#xA;#np.save ('true_catagories.npy',y_test)&#xA;np.save('original.npy',x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43472" LastActivityDate="2018-01-30T15:05:44.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27247" PostTypeId="1" CreationDate="2018-01-30T15:36:41.393" Score="7" ViewCount="3680" Body="&lt;p&gt;In my class I often need to work with color map images. I would show the image and try to make inferences/observations about different subjects. Often times I need to actually quantify some aspects, but it is always very approximate and somehow vague because the images are provided &amp;quot;as is&amp;quot; and I do not necessarily know their content a priori.&lt;/p&gt;&#xA;&lt;p&gt;Let's imagine I'm working with two images (*). Is it possible to indicate the computer &amp;quot;learn&amp;quot; the color scale bar so I can click at any point in the map and get the value of altitude at that point? Image 1 is a discrete color scale bar, while Image 2 has a continuous color mapping. I included both examples because maybe there are differences in the approach of how to do this.&lt;/p&gt;&#xA;&lt;p&gt;So, is there a software/way to do this? Preferably open source. I tried ImageJ and couldn't come up with a decent solution.&lt;/p&gt;&#xA;&lt;p&gt;Image 1:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://upload.wikimedia.org/wikipedia/commons/2/24/AYool_topography_15min.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://upload.wikimedia.org/wikipedia/commons/2/24/AYool_topography_15min.png&quot; alt=&quot;Image 1&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Image 2:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/knHLL.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/knHLL.jpg&quot; alt=&quot;Image 2&quot; /&gt;&lt;/a&gt;&lt;br /&gt;&#xA;&lt;sub&gt;(source: &lt;a href=&quot;https://topocreator.com/ned-jpg/city_a/600/co.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;topocreator.com&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;&lt;p&gt;As you can see, the color scale is part of the image in all cases.&lt;/p&gt;&#xA;&lt;p&gt;(*) I do not own any of these images, just examples I found online to illustrate my point.&lt;/p&gt;&#xA;" OwnerUserId="45441" LastEditorUserId="25180" LastEditDate="2020-12-21T04:37:30.493" LastActivityDate="2020-12-21T04:37:30.493" Title="Can I get numeric data from a color map?" Tags="|image-recognition|software-recommendation|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="27248" PostTypeId="1" AcceptedAnswerId="27252" CreationDate="2018-01-30T16:03:30.863" Score="3" ViewCount="2510" Body="&lt;p&gt;When reading about convolutional neural networks (CNNs), I often come across a special notation used in the community and in scientific papers, describing the architecture of the network in terms of layers. However, I was not able to find a paper or resource describing this notation in detail.&lt;br&gt;&#xA;Could someone explain to me the details or point to where it is described or &quot;standardized&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;input−100C3−MP2−200C2−MP2−300C2−MP2−400C2−MP2−500C2−output&lt;/code&gt;&lt;br&gt;&#xA;(&lt;a href=&quot;https://arxiv.org/abs/1409.6070&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;input−(300nC2−300nC2−MP2)_5−C2−C1−output&lt;/code&gt;&lt;br&gt;&#xA;(&lt;a href=&quot;https://arxiv.org/abs/1412.6071&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A good guess seems that &lt;em&gt;x&lt;/em&gt;&lt;code&gt;C&lt;/code&gt;&lt;em&gt;y&lt;/em&gt; are convolution layers (&lt;em&gt;x&lt;/em&gt; is number of filters? &lt;em&gt;y&lt;/em&gt; is one side of square kernel?). &lt;code&gt;MP&lt;/code&gt;&lt;em&gt;z&lt;/em&gt; is max-pooling layer (pool size &lt;em&gt;z&lt;/em&gt;×&lt;em&gt;z&lt;/em&gt;?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But instead of guessing, I would love to have a reference (which I could possibly also reference in a paper).&lt;/p&gt;&#xA;" OwnerUserId="41615" LastEditorUserId="29169" LastEditDate="2021-02-13T16:43:07.190" LastActivityDate="2021-02-13T16:43:07.190" Title="Layer notation for convolutional neural networks" Tags="|convolutional-neural-network|notation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27249" PostTypeId="1" CreationDate="2018-01-30T16:31:04.117" Score="0" ViewCount="54" Body="&lt;p&gt;I am trying to print a accuracy score and get this message:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;lt;function accuracy_score at 0x0000018A76046840&amp;gt;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I am wondering if I am missing a package or is something installed incorrect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;from sklearn.metrics import accuracy_score&#xA;from sklearn import svm&#xA;iris = datasets.load_iris()&#xA;X = iris.data&#xA;y= iris.target&#xA;#print (iris.keys)&#xA;print('**********')&#xA;#print (iris['feature names'])&#xA;print('**********')&#xA;#print (iris)&#xA;#print (iris['data'].shape,'legth: ',len(iris['target']))&#xA;clf = svm.SVC(gamma=0.001, C=100)&#xA;clf.fit(X,y)&#xA;y_hat=clf.predict(X)&#xA;accuracy_score(y, y_hat)&#xA;score = accuracy_score&#xA;print (score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45448" LastEditorUserId="16401" LastEditDate="2018-01-30T17:04:32.307" LastActivityDate="2018-01-30T17:04:32.307" Title="running scikit-learn on Sypder with Python 3.5" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27250" PostTypeId="2" ParentId="27249" CreationDate="2018-01-30T16:37:02.083" Score="1" Body="&lt;p&gt;The reason is that you are printing the object of function. Instead call the function. Suppose the name of your function is &lt;code&gt;func&lt;/code&gt;. You are doing something like &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(func)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead try to call it by adding parenthesis. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(func())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-01-30T16:37:02.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27251" PostTypeId="2" ParentId="27249" CreationDate="2018-01-30T16:37:20.620" Score="1" Body="&lt;p&gt;This is how it must be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score = accuracy_score(y, y_hat)&#xA;print (score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You assigned empty function to score. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your entire code should now be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets&#xA;from sklearn.metrics import accuracy_score&#xA;from sklearn import svm&#xA;iris = datasets.load_iris()&#xA;X = iris.data&#xA;y= iris.target&#xA;#print (iris.keys)&#xA;print('**********')&#xA;#print (iris['feature names'])&#xA;print('**********')&#xA;#print (iris)&#xA;#print (iris['data'].shape,'legth: ',len(iris['target']))&#xA;clf = svm.SVC(gamma=0.001, C=100)&#xA;clf.fit(X,y)&#xA;y_hat=clf.predict(X)&#xA;&#xA;score = accuracy_score(y, y_hat)&#xA;print (score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16401" LastActivityDate="2018-01-30T16:37:20.620" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27252" PostTypeId="2" ParentId="27248" CreationDate="2018-01-30T16:59:01.757" Score="3" Body="&lt;p&gt;One paper referenced by the first paper you linked to is &lt;a href=&quot;https://arxiv.org/abs/1202.2745&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It explains in section 3 (experiments) the following notation:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2x48x48-100C5-MP2-100C5-MP2-100C4-MP2-300N-100N-6N represents a net&#xA;  with:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;2 input images of size 48x48&lt;/li&gt;&#xA;  &lt;li&gt;a convolutional layer with 100 maps and 5x5 filters&lt;/li&gt;&#xA;  &lt;li&gt;a max-pooling layer over non-overlapping regions of size 2x2&lt;/li&gt;&#xA;  &lt;li&gt;a convolutional layer with 100 maps and 4x4 filters&lt;/li&gt;&#xA;  &lt;li&gt;a max-pooling layer over non overlapping regions of size 2x2&lt;/li&gt;&#xA;  &lt;li&gt;a fully connected layer with 300 hidden units,&lt;/li&gt;&#xA;  &lt;li&gt;a fully connected layer with 100 hidden units&lt;/li&gt;&#xA;  &lt;li&gt;a fully connected layer with 6 neurons (one per class)&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From this, the answer to your question is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;100C3 means a convolutional layer with 100 maps and 3x3 filters&lt;/li&gt;&#xA;&lt;li&gt;MP2 means a max-pooling layer with non overlapping regions of size 2x2&lt;/li&gt;&#xA;&lt;li&gt;200C2 means a convolutional layer with 200 maps and 2x2 filters&lt;/li&gt;&#xA;&lt;li&gt;etc with the &quot;C&quot; layers (means convolutional, preceding integer is the number of features maps; final integer is the filter size)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;According to the second paper you linked, the subscript &lt;code&gt;_5&lt;/code&gt; indicates five pairs of &lt;code&gt;300nC2−300nC2−MP2&lt;/code&gt; connected layers (&lt;a href=&quot;https://arxiv.org/abs/1412.6071&quot; rel=&quot;nofollow noreferrer&quot;&gt;see section 3&lt;/a&gt;), and the &lt;code&gt;n&lt;/code&gt; indicates &quot;the number of filters in the nth convolutional layer is [300]n&quot;.  According to the accompanying model diagram (figure 3 in the linked paper), the &lt;code&gt;C2&lt;/code&gt; and &lt;code&gt;C1&lt;/code&gt; layers produce 1x1 output, meaning a scalar value. This would mean &lt;code&gt;C2&lt;/code&gt; is a convolutional layer with 1 map and a 2x2 filter and &lt;code&gt;C1&lt;/code&gt; is a convolutional layer with 1 map and 1x1 filter (though I don't fully understand what this adds).&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="41615" LastEditDate="2018-01-31T14:24:45.783" LastActivityDate="2018-01-31T14:24:45.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27254" PostTypeId="2" ParentId="26488" CreationDate="2018-01-30T17:47:38.507" Score="0" Body="&lt;p&gt;Below is one way to achieve this in R using Tidyverse : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data %&amp;gt;% &#xA;mutate(group = rep(1:4, each = 2)) %&amp;gt;% # in this example such is the rule # data-dependent step&#xA;gather(&quot;key&quot;, &quot;value&quot;, c(&quot;code&quot;, &quot;literal&quot;)) %&amp;gt;% &#xA;mutate(key = paste0(key, group)) %&amp;gt;% &#xA;dplyr::select(-group) %&amp;gt;% &#xA;spread(key, value) %&amp;gt;% &#xA;dplyr::select(zone, ends_with(&quot;1&quot;), ends_with(&quot;2&quot;), ends_with(&quot;3&quot;), ends_with(&quot;4&quot;)) # to arrange the column names, because otherwise all the 'code' columns are together and 'literal' columns come after # data-dependent step&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40767" LastActivityDate="2018-01-30T17:47:38.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27256" PostTypeId="2" ParentId="26488" CreationDate="2018-01-30T19:33:07.103" Score="2" Body="&lt;p&gt;Just two lines in plain R&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X &amp;lt;- read.table(header = TRUE, text = &quot;&#xA;zone       code        literal&#xA; A         14           bicl&#xA; B         14           bicl&#xA; B         24          calso&#xA; A         51           mara&#xA; B         51           mara&#xA; A         125           gan&#xA; A         143          carc&#xA; B         143          carc&quot;)&#xA;&#xA;X$time &amp;lt;- ave(X$code, X$zone, FUN = seq_along)&#xA;reshape(X, direction = &quot;wide&quot;, timevar = &quot;time&quot;, idvar = &quot;zone&quot;, sep = &quot;&quot;)&#xA;&#xA;# output&#xA;  zone code1 literal1 code2 literal2 code3 literal3 code4 literal4&#xA;1    A    14     bicl    51     mara   125      gan   143     carc&#xA;2    B    14     bicl    24    calso    51     mara   143     carc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="36977" LastEditorUserId="36977" LastEditDate="2018-01-30T20:29:04.923" LastActivityDate="2018-01-30T20:29:04.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27258" PostTypeId="1" AcceptedAnswerId="27260" CreationDate="2018-01-30T20:58:57.793" Score="1" ViewCount="750" Body="&lt;p&gt;I &lt;a href=&quot;https://sebastianraschka.com/faq/docs/regularized-logistic-regression-performance.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;read that&lt;/a&gt; L2 regularization in logistic regression creates a sort of sphere that limits the choice of $w$ weight, but why does this happen? &lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="381" LastEditDate="2018-01-31T00:00:51.190" LastActivityDate="2018-01-31T00:00:51.190" Title="Math behind L2 Regularization for Logistic Regression" Tags="|machine-learning|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27259" PostTypeId="2" ParentId="14102" CreationDate="2018-01-30T21:27:37.267" Score="1" Body="&lt;p&gt;First of all, it seems that your data is discrete, and therefore I would advise using Multinomial Naive Bayes (scikit-learn also provides an implementation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ran your code and the reason your code breaks is not because of the implementation, but because of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fitting works, but if you run &lt;code&gt;model.sigma_&lt;/code&gt;, you will see that the variance of your features equals 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will cause a runtime error since Gaussian Naive Bayes models class probabilities as follows: &lt;a href=&quot;https://i.stack.imgur.com/EVxvg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EVxvg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since sigma equals zero here, your code will crash. If you simply change your code to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.naive_bayes import GaussianNB&#xA;import numpy as np&#xA;&#xA;&quot;&quot;&quot;&#xA;    Labels : Lost, Draw, Won [-1,0,1]&#xA;    Features&#xA;    ==========&#xA;        Toss(Lost,Won) = [-1,1]&#xA;        Bat(First, Second) = [-1,1]&#xA;&quot;&quot;&quot;&#xA;#Based on Existing Data Features are:&#xA;features = np.array([[-1, -1],[-1, 1]])&#xA;labels = np.array([0,1])&#xA;# Create a Gaussian Classifier&#xA;model = GaussianNB()&#xA;&#xA;# Train the model using the training sets&#xA;model.fit(features, labels)&#xA;&#xA;# Predict Output&#xA;predicted = model.predict([[1,0]])&#xA;print(predicted)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where all you do is change your features to &lt;code&gt;features = np.array([[-1, -1],[-1, 1]])&lt;/code&gt;, your code will now run.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-01-30T21:27:37.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27260" PostTypeId="2" ParentId="27258" CreationDate="2018-01-30T21:29:49.217" Score="5" Body="&lt;p&gt;Your question is really about &lt;a href=&quot;https://en.wikipedia.org/wiki/Lagrange_multiplier&quot; rel=&quot;noreferrer&quot;&gt;the method of Lagrange multipliers&lt;/a&gt; in constrained optimization, not logistic regression per se. The gist of it is that a constrained optimization problem can be recast as an unconstrained optimization problem by adding a term, called the &lt;em&gt;regularizer&lt;/em&gt;, and vice versa. The sphere comes from recasting the unconstrained problem into a constrained one; recall that a constant $L_2$ norm defines a hypersphere.&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2018-01-30T21:29:49.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27261" PostTypeId="2" ParentId="9087" CreationDate="2018-01-30T21:40:08.843" Score="1" Body="&lt;p&gt;Yes, it will affect the performance of Naive Bayes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is called Naive because it assumes an independence between the features, which in practice is rarely the case. However, it's shown to be fairly robust to this and to be able to perform well on real-world problems. So having correlation will go against the Naive assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, correlation is not necessarily a bad or a good thing for the performance of your model. Correlation between features in Naive Bayes simply means that if one feature &quot;says&quot; it's class A, then the other feature(s) will often say the same. Therefore, if your correlated features happen to be good predictors, your model will actually benefit from it, if they happen to be bad predictors, your model will be worse off.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-01-30T21:40:08.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27262" PostTypeId="1" AcceptedAnswerId="27275" CreationDate="2018-01-30T21:41:26.287" Score="4" ViewCount="8397" Body="&lt;p&gt;I want to apply a &lt;code&gt;ConvNet&lt;/code&gt; on my one dimensional data retrieved from 13 sensors. So, each of my samples consists of 13 channels (of 51 values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using 'conv1d' to apply a ConvNet on my data. The network works nicely, but I wonder how 'conv1d' determines the number of channels for it's filters... To my knowledge, a filter should have an equal number of channels as its input data, which makes it a &lt;span class=&quot;math-container&quot;&gt;$5x13$&lt;/span&gt; filter. I set the filter to have a width of 5, but don't need to set the number of channels anywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: how does layer 'conv1' determine it's number of channels?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a portion of my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# We have 13 1D channels of 51 points each&#xA;&#xA;# Note that we've indicated -1 for batch size, which specifies that this dimension should be dynamically computed &#xA;# based on the number of input values in features[&quot;x&quot;], holding the size of all other dimensions constant.&#xA;&#xA;input_layer = tf.reshape(features[&quot;x&quot;], [-1, 51, 13])&#xA;&#xA;# Convolutional Layer #1&#xA;# Shouldn't this filter also need to number of channels? &#xA;This should match the input number of channels&#xA;conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding=&quot;same&quot;, activation=tf.nn.relu)&#xA;&#xA;# Pooling Layer #1&#xA;pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42681" LastEditorUserId="29169" LastEditDate="2019-06-07T17:26:13.130" LastActivityDate="2019-06-07T17:26:13.130" Title="TensorFlow: number of channels of conv1d filter" Tags="|machine-learning|python|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27263" PostTypeId="2" ParentId="16817" CreationDate="2018-01-30T21:58:12.273" Score="4" Body="&lt;p&gt;Let me try to answer your questions point by point. Perhaps you already solved your problem, but your questions are interesting and so perhaps other people can benefit from this discussion.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Naive Bayes has shown to perform well on document classification, but that doesn't mean that it cannot overfit data. There is a difference between the task, document classification, and the data. Overfitting can happen even if Naive Bayes is implemented properly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It's good that you're keeping the class distribution the same in your training and test set. Are you using cross-validation? Perhaps try it because, even though it's rare, it might happen that you just get unlucky with your splitting due to some seed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;You're correct, if it was an issue, it would show in the training set as well.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This doesn't seem to be the issue then. You could perhaps reduce that percentage by using stemming or lemmatization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Laplace smoothing is useful especially when you do not have a lot of data and you need to account for some uncertainty. For this dataset, this doesn't seem to be an issue, as shown by the similar results you've experienced.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Are the documents of the same lengths? Because perhaps label 16 simply contains documents that are larger and therefore have bigger word frequencies. They might also contain very common words. It would be interesting to look at a histogram of the words in each class. This could be very informative in understanding whether label 16 is very different from the others.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-01-30T21:58:12.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27264" PostTypeId="2" ParentId="8015" CreationDate="2018-01-30T22:14:23.747" Score="6" Body="&lt;p&gt;First off, yes there are different Naive Bayes algorithms. But they are all based on the same principle, namely Bayes Theorem where the features are assumed to be independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a short guide on when to use which for spam detection (or document classification in general for that matter):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Bernoulli Naive Bayes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters as much as if it isn't. For instance, the fact that &quot;viagra&quot; is in the e-mail might mean it's &quot;spam&quot; and the fact it isn't might mean it's &quot;not spam&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multinomial Naive Bayes with boolean features&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters more than if it isn't. For instance, if &quot;grandma&quot; is in the e-mail, it might mean that it's &quot;not spam&quot;, but if it isn't in the e-mail, it doesn't necessarily mean that it's &quot;spam&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Multinomial Naive Bayes with term frequency&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Each e-mail is represented by how many times each word occurs.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Gaussian Naive Bayes&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This deals with continuous values, so this isn't applicable here.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2018-02-04T13:07:03.410" LastActivityDate="2018-02-04T13:07:03.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27265" PostTypeId="2" ParentId="16036" CreationDate="2018-01-30T22:26:57.980" Score="1" Body="&lt;p&gt;Perhaps you could use word embeddings to better represent the distance between certain skills. For instance, &quot;Python&quot; and &quot;R&quot; should be closer together than &quot;Python&quot; and &quot;Time management&quot; since they are both programming languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The whole idea is that words that appear in the same context should be closer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have these embeddings, you would have a set of skills for the candidate, and sets of skills of various size for the jobs. You could then use &lt;a href=&quot;https://en.wikipedia.org/wiki/Earth_mover%27s_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Earth Mover's Distance&lt;/a&gt; to calculate the distance between the sets. This distance measure is rather slow (quadratic time) so it might not scale well if you have many jobs to go through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To deal with the scalability issue, you could perhaps rank the jobs based on how many skills the candidate has in common in the first place, and favor these jobs.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-01-30T22:26:57.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27266" PostTypeId="2" ParentId="27258" CreationDate="2018-01-30T22:42:03.687" Score="1" Body="&lt;p&gt;A simple way to think about this is to appreciate that you are minimizing an objective function. L2 regularization alters the output of the objective function such that smaller values are favored. So you have this constant 'pressure' on the parameters aiming towards 0. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-01-30T22:42:03.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27267" PostTypeId="1" AcceptedAnswerId="27269" CreationDate="2018-01-30T22:56:11.130" Score="0" ViewCount="240" Body="&lt;p&gt;I am reading the book &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://neuralnetworksanddeeplearning.com/chap3.html&lt;/a&gt; by Michael Nielsen.&#xA;So this is a question mostly for the people familiar with the book and understanding the material.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the equations (71-75) we are trying to find a cost-function $C$ satisfying:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial{C}}{\partial{\omega_j}}=x_j (a-y),$$&#xA;$$\frac{\partial{C}}{\partial{b}}=(a-y),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w_j$ and $b$ are weights and bias of a neuron, $a$ is the output of the sigmoid function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$a = \sigma(\sum{\omega_{j} x_j + b)}=\sigma(z).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We apply the chain rule (equation 73):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\partial{C}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\frac{\partial{a}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\sigma\prime(z).$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in the next line the author writes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Using&lt;/em&gt; $\sigma\prime(z)=\sigma(z)(1-\sigma(z))=a(1-a)$ &lt;em&gt;the last equation becomes...&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where does this expression come from? $\sigma(z)(1-\sigma(z))$&lt;/p&gt;&#xA;" OwnerUserId="18219" LastEditorUserId="18219" LastEditDate="2018-01-31T09:27:36.360" LastActivityDate="2018-01-31T09:27:36.360" Title="Derivation of the cross-entropy equation in Michael Nielsen's book" Tags="|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27268" PostTypeId="1" CreationDate="2018-01-31T00:07:01.387" Score="3" ViewCount="1270" Body="&lt;p&gt;I am wondering which ML algoirthms (supervised) are commonly used for TS analysis. &#xA;Which ones have used and found to be succesful for your projects? &lt;/p&gt;&#xA;" OwnerUserId="44668" LastActivityDate="2018-01-31T12:20:11.757" Title="Machine learning algorithms for time series analysis" Tags="|machine-learning|scikit-learn|time-series|supervised-learning|" AnswerCount="1" CommentCount="4" ClosedDate="2018-02-01T18:12:42.060" ContentLicense="CC BY-SA 3.0" />
  <row Id="27269" PostTypeId="2" ParentId="27267" CreationDate="2018-01-31T00:08:07.840" Score="5" Body="&lt;p&gt;The author has taken &lt;code&gt;activation function&lt;/code&gt; as &lt;code&gt;sigmoid&lt;/code&gt; in this case. The derivative of this function can be re-written like&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\sigma(z) = \frac{1}{1+e^{-x}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;chap1&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2018-01-31T03:12:35.123" LastActivityDate="2018-01-31T03:12:35.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27270" PostTypeId="1" CreationDate="2018-01-31T00:57:03.973" Score="1" ViewCount="2325" Body="&lt;p&gt;I have 'practice' data set which I can split into training, validation, and test set and I will play with data to make a machine learning model. But in real situation, I will be given a very small data set which I will split into training and validation sets. There is no enough observations to make a separate test set. Then how can I estimate the 'fair' performance of the model on the real data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only idea I have is to find a relation between performance on the validation set and test set on the practice data (a regression line etc) and then apply the formula to the performance on the validation set of the real data to find that on the test set of the real data (which actually does not exist.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other good idea?&lt;/p&gt;&#xA;" OwnerUserId="8478" LastActivityDate="2018-01-31T09:26:19.030" Title="how should I measure performance if there is no test data?" Tags="|cross-validation|training|performance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27271" PostTypeId="2" ParentId="27270" CreationDate="2018-01-31T04:20:26.620" Score="3" Body="&lt;p&gt;If you are working with only enough data for training and validation, consider using &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;K-Fold Cross Validation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Assuming your &quot;practice&quot; data is drawn from the same distribution as your &quot;real&quot; data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example you might get results like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;practice data:&#xA;  training accuracy: 90%&#xA;  test accuracy: 88%&#xA;&#xA;real data:&#xA;  training accuracy: 89%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about &lt;code&gt;2%&lt;/code&gt; worse than training accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;practice data:&#xA;  training accuracy: 95%&#xA;  test accuracy: 75%&#xA;&#xA;real data:&#xA;  training accuracy: 85%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict &lt;code&gt;65%&lt;/code&gt; test accuracy, and in the second case we might predict &lt;code&gt;75%&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-01-31T09:26:19.030" LastActivityDate="2018-01-31T09:26:19.030" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27272" PostTypeId="1" CreationDate="2018-01-31T05:14:37.723" Score="3" ViewCount="5224" Body="&lt;p&gt;I got the following warning:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;94: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 1200012120 elements. This may consume a large amount of memory.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from wordbatch.extractors import WordSeq&#xA;import wordbatch&#xA;from keras.layers import Input,Embedding&#xA;...&#xA;wb = wordbatch.WordBatch(normalize_text, extractor=(WordSeq, {&quot;seq_maxlen&quot;: MAX_NAME_SEQ}), procs=NUM_PROCESSOR)&#xA;wb.dictionary_freeze = True&#xA;full_df[&quot;ws_name&quot;] = wb.fit_transform(full_df[&quot;name&quot;])&#xA;...&#xA;name = Input(shape=[full_df[&quot;name&quot;].shape[1]], name=&quot;name&quot;)&#xA;emb_name = Embedding(MAX_TEXT, 50)(name)&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That is I make use of WordSeq (from WordBatch) output from the Embedding layer of a GRU network. How should I modify the code to make it work without converting to dense tensor?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastEditorUserId="7812" LastEditDate="2018-01-31T05:32:53.127" LastActivityDate="2018-12-21T17:43:01.127" Title="Converting sparse IndexedSlices to a dense Tensor" Tags="|tensorflow|keras|word-embeddings|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27273" PostTypeId="1" CreationDate="2018-01-31T05:28:24.743" Score="4" ViewCount="5728" Body="&lt;p&gt;I'm new to machine learning and deep learning. I've wanted to solve time series problem, which has data every single second. Plus, I've been doing research on word2vector and time series data lately. And one day, I've come up with an idea transforming sequence data like date time into one-hot-encoding?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    time&#xA;2017-11-01 00:00:01&#xA;2017-11-01 00:00:02&#xA;2017-11-01 00:00:03&#xA;2017-11-01 00:00:04&#xA;.&#xA;.&#xA;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My idea has some limitations like below,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;too high dimensions for learning (1day = 60* 60 * 24 = 86400(s))&lt;/li&gt;&#xA;&lt;li&gt;unlimited time - time will be generated every moment even right now as well&lt;/li&gt;&#xA;&lt;li&gt;difference between seconds is too small to learn&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I want you to determine what I'm saying above about limitations. Plus, I'd like you to give me some idea to develop time series data into one-hot-vector for machine learning and deep learning? + what do you think about this idea?&lt;/p&gt;&#xA;" OwnerUserId="45485" LastActivityDate="2018-01-31T06:16:17.077" Title="Does it make sense that datetime encodes one-hot-vector like one-hot-encoding or something else like" Tags="|machine-learning|deep-learning|time-series|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27274" PostTypeId="2" ParentId="27273" CreationDate="2018-01-31T05:41:39.833" Score="6" Body="&lt;p&gt;If you are trying to predict future values then it doesn't make sense to treat them as categorical features. There is nothing you will learn that can predict future data, since you won't see those times again. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same holds even if you are trying to predict unseen past data but each time appears only once or a small number of times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead the time values should determine the order of your training data. This way you can avoid leaking future data, and models with state can learn features from the sequential nature of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in some cases you may want to extract additional features from the time values. Here are some examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Time since last interesting event&lt;/li&gt;&#xA;&lt;li&gt;Number of interesting events in last time window of size n&lt;/li&gt;&#xA;&lt;li&gt;Time of day (morning / afternoon / etc)&lt;/li&gt;&#xA;&lt;li&gt;Day of week&lt;/li&gt;&#xA;&lt;li&gt;Holiday&lt;/li&gt;&#xA;&lt;li&gt;Season&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Let's make up an example. Here is a dataset of times that users visited a website:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time               user&#xA;2017-11-01 00:00   Alice&#xA;2017-11-01 00:00   Bob&#xA;2017-11-02 00:00   Chris&#xA;2017-11-03 00:00   Alice&#xA;2017-11-04 00:00   Alice&#xA;2017-11-04 00:00   Bob&#xA;2017-11-07 00:00   Chris&#xA;2017-11-10 00:00   Alice&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here is the same dataset with additional features we have added:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time               user    last_visit   weekend?   time_of_day&#xA;2017-11-01 16:22   Alice   N/A          No         afternoon&#xA;2017-11-01 11:13   Bob     N/A          No         morning&#xA;2017-11-02 20:35   Chris   N/A          No         evening&#xA;2017-11-03 16:07   Alice   2 days       No         afternoon&#xA;2017-11-04 17:20   Alice   1 day        Yes        afternoon&#xA;2017-11-04 10:44   Bob     3 days       Yes        morning&#xA;2017-11-07 08:06   Chris   5 days       No         morning&#xA;2017-11-10 17:11   Alice   6 days       No         afternoon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If we are trying to predict when a certain user might visit next, then these features might help us a lot. For example we might decide that Alice is more likely to visit in the afternoon, or Bob is unlikely to visit two days in a row.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-01-31T06:16:17.077" LastActivityDate="2018-01-31T06:16:17.077" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27275" PostTypeId="2" ParentId="27262" CreationDate="2018-01-31T06:41:30.437" Score="2" Body="&lt;p&gt;In &lt;code&gt;TensorFlow&lt;/code&gt; there are different convolution layers. &lt;code&gt;Conv1d&lt;/code&gt;, &lt;code&gt;Conv2d&lt;/code&gt; and &lt;code&gt;Conv3d&lt;/code&gt;. the first one is used for one dimensional signals like sounds, the second one is used for images, gray-scale or &lt;code&gt;RGB&lt;/code&gt; images and both cases are considered to be two dimensional signals. The last one is used for three dimensional signals like video frames, images as two dimensional signals vary during time. In your case &lt;code&gt;Conv1d&lt;/code&gt; is used as one dimensional signal and you can specify the number of filters in the arguments of the method. You can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26321/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/26384/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-01-31T06:49:51.467" LastActivityDate="2018-01-31T06:49:51.467" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27276" PostTypeId="1" CreationDate="2018-01-31T07:05:23.973" Score="2" ViewCount="238" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/abs/1711.09784&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Distilling a Neural Network Into a Soft Decision Tree&lt;/em&gt;&lt;/a&gt; the authors mention &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;exponentially decaying running average of the actual probabilities with a time window&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but provide no formula for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does it mean speaking formal?&lt;/p&gt;&#xA;" OwnerUserId="45486" LastActivityDate="2020-12-19T09:48:36.723" Title="Details on soft decision trees" Tags="|neural-network|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27277" PostTypeId="1" CreationDate="2018-01-31T07:12:32.497" Score="8" ViewCount="5358" Body="&lt;p&gt;I am trying to understand the whole Faster-RCNN, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From  &lt;a href=&quot;https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work&quot; rel=&quot;noreferrer&quot;&gt;https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Then a sliding window is run spatially on these feature maps. The size of sliding window is n×n (here 3×3). For each sliding window, a set of 9 anchors are generated which all have the same center (xa,ya)(xa,ya) but with 3 different aspect ratios and 3 different scales as shown below. Note that all these coordinates are computed with respect to the original image.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PWd2d.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PWd2d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is much more clear than other articles for my opion, but still hard to understand how the feature map generate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw another flow pics :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ci0Vw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ci0Vw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/gpj2I.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gpj2I.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/veyVL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/veyVL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem, I write below steps for example:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If input is 600x1000x3 pic&lt;/li&gt;&#xA;&lt;li&gt;Through VGG16 convnet , layer 13 output feature map is 40x60x512&lt;/li&gt;&#xA;&lt;li&gt;Use a 3x3 sliding window, generate 1x1x512 feature map ???&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here, how 3x3 sliding window use a set of 9 anchors ??? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, I am really new to object detection and image proccessing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I only have a little understand about the steps, I known &lt;strong&gt;9 anchor shapes&lt;/strong&gt;(not the real anchor) are used to generate a lot of anchors(2400*9 in this case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can only imagine that use 9 anchor shape to slide the &lt;strong&gt;original image&lt;/strong&gt; to get the all IoU . I don't understand how to use 3x3 sliding window in conv feature map here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how anchors be selected,  2400*9 -&gt; ignore cross-boundary -&gt; 6000 -&gt; apply NMS -&gt; 2000 ,  in each minibatch, it randomly choose 512 anchors from 2000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can't understand is 3x3 slide with 9 anchor shape . Because from original paper, anchors with is 16, height from 11 to 273 . I don't think it use the 13 layer conv output feature map to calculate IoU . Anchor must be apply in original image,  so what is the 3x3 sliding window doing ??&lt;/p&gt;&#xA;" OwnerUserId="15325" LastEditorUserId="29169" LastEditDate="2021-02-13T16:42:56.327" LastActivityDate="2021-02-13T16:42:56.327" Title="Faster-RCNN how anchor work with slider in RPN layer?" Tags="|image-classification|computer-vision|convolutional-neural-network|object-detection|faster-rcnn|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27278" PostTypeId="1" CreationDate="2018-01-31T07:30:17.320" Score="1" ViewCount="7343" Body="&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn import datasets, linear_model&#xA;house_price = [245, 312, 279, 308, 199, 219, 405, 324, 319, 255]&#xA;size = [1400, 1600, 1700, 1875, 1100, 1550, 2350, 2450, 1425, 1700]&#xA;size2 = np.array(size).reshape((-1, 1))&#xA;#fitting into the model&#xA;regr = linear_model.LinearRegression()&#xA;regr.fit(size2, house_price)&#xA;print(&quot;Coefficients: \n&quot;, regr.coef_)&#xA;print(&quot;intercept: \n&quot;, regr.intercept_)&#xA;#############################&#xA;#formula obtained for the trained model&#xA;def graph(formula, x_range):&#xA;   x = np.array(x_range)&#xA;   y = eval(formula)&#xA;   plt.plot(x, y)&#xA;#plotting the prediction line &#xA;graph('regr.coef_*x + regr.intercept_', range(1000, 2700))&#xA;print(regr.score(size2, house_price))&#xA;plt.scatter (size,house_price, color='black')&#xA;plt.ylabel('house price')&#xA;plt.xlabel('size of house')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Error&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;**Error Line:print regr.predict([2000])**&#xA;Error: File &quot;&amp;lt;ipython-input-4-9afa91ca7f9e&amp;gt;&quot;, line 1&#xA;    print regr.predict([2000])&#xA;             ^&#xA;SyntaxError: invalid syntax&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="45488" LastEditorUserId="29575" LastEditDate="2018-01-31T14:23:49.270" LastActivityDate="2018-01-31T14:23:49.270" Title="Please help me out with this Python error - 'invalid syntax'" Tags="|machine-learning|python|linear-regression|" AnswerCount="1" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="27279" PostTypeId="2" ParentId="27208" CreationDate="2018-01-31T07:38:21.223" Score="1" Body="&lt;p&gt;After running a few tests and collecting some training data I got a rough impression of how much data is needed. The question is still not answered entirely but I figured these insights might help others:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how many images did I need?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Architecture: &#xA;- Pre-trained Inception V3 model with Imagenet data&#xA;- Removed last layer &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data / Images per category:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;101 images  &lt;/li&gt;&#xA;&lt;li&gt;258 images&lt;/li&gt;&#xA;&lt;li&gt;45 images&lt;/li&gt;&#xA;&lt;li&gt;1064 images&lt;/li&gt;&#xA;&lt;li&gt;259 images&lt;/li&gt;&#xA;&lt;li&gt;607 images&lt;/li&gt;&#xA;&lt;li&gt;161 images&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Complexity:&#xA;Mixed complex sceneries with simple images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Result:&#xA;The quality depends a lot on how difficult the task is. In my scenario the images to be classified where rather similar to the ones in the training data. In these situations all categories with 160 images and above are recognised rather reliably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will do some more testing and update my answer.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-01-31T07:38:21.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27281" PostTypeId="2" ParentId="27278" CreationDate="2018-01-31T07:54:39.823" Score="3" Body="&lt;p&gt;In your current line &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print regr.predict([2000])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will not work. The first error is the lack of brackets around the contents of your print statement which is required in Python 3. Change this first to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(regr.predict([2000]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, you will see that this does not work either. I suspect you are attempting to evaluate the price for a new $size = 2000$. You will need to reshape the input to your regression for this to work. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_size = np.array([2000]).reshape((-1, 1))&#xA;print(regr.predict(new_size))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 317.78380528]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-01-31T07:54:39.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27282" PostTypeId="2" ParentId="27277" CreationDate="2018-01-31T08:46:09.193" Score="1" Body="&lt;p&gt;For each anchor you find an IoU with the object in the picture and set 1 if IoUexceeds the threshold and 0 if it is below a lower tjreshold(e.g. 0.3). If it's a hit a bbox offset is calculated, distance between prediction and true bbox. Hence there are two loss fumctions: object/bg and bbox regression&lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2018-01-31T08:46:09.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27283" PostTypeId="2" ParentId="13759" CreationDate="2018-01-31T08:47:24.947" Score="0" Body="&lt;p&gt;In case someone bumps into the page looking for a solution. This one worked for me: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;install.packages('devtools')&#xA;library(devtools)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now install an earlier version of slam in case you are using an older version of R. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;slam_old &amp;lt;- &quot;https://cran.r-project.org/src/contrib/Archive/slam/slam_0.1-37.tar.gz&quot;&#xA;install_url(slam_old)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="16401" LastActivityDate="2018-01-31T08:47:24.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27284" PostTypeId="2" ParentId="27025" CreationDate="2018-01-31T09:34:07.747" Score="0" Body="&lt;p&gt;As far as  I understand, it is a simple autoencoder, meaning that all it does is trying to map the input into another space, so no fancy training, just some plain feed-forward and backprop. This is why it's rather fast to train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use pre-trained embeddings, &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;you can do it that way&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-01-31T09:34:07.747" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27285" PostTypeId="1" CreationDate="2018-01-31T10:00:21.177" Score="3" ViewCount="75" Body="&lt;p&gt;I read a few days ago about multi-scale CNN &lt;em&gt;(&lt;code&gt;OverFeat&lt;/code&gt; method)&lt;/em&gt; which you can access to presentation via this &lt;a href=&quot;http://vision.stanford.edu/teaching/cs231b_spring1415/slides/overfeat_eric.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;. They performed CNN on different scales of an image and then combine all output maps. They said inside of that presentation:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Classification    performed   at  6   scales  at  test time,  but only    1 scale at  run time    .&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So my question is: &lt;strong&gt;If we use 6 different scales of CNN architecture, then we have different convolution layers in every scale (I guess so). So how in &lt;code&gt;OverFeat&lt;/code&gt;, they use 1 scale in run time? If we use a specific scale, then how can we access other feature extractors of different scales?&lt;/strong&gt; And I see in the article, they combine feature maps of different scales but I can't figure out how this process performed.&lt;/p&gt;&#xA;" OwnerUserId="45494" LastEditorUserId="85045" LastEditDate="2020-12-02T16:38:17.200" LastActivityDate="2020-12-02T16:38:17.200" Title="How multi-scale CNN selects final output map" Tags="|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27286" PostTypeId="1" CreationDate="2018-01-31T10:26:32.707" Score="4" ViewCount="1903" Body="&lt;p&gt;I have been going through this paper: &lt;a href=&quot;https://inst.eecs.berkeley.edu/~cs194-26/fa16/hw/proj4-seamcarving/imret.pdf&quot; rel=&quot;noreferrer&quot;&gt;Seam Carving for Content-Aware Image Resizing&lt;/a&gt; which talks about resizing images by seam carving depending on the image energy or the energy function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some related quotes from the paper are:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A seam is a connected path of low energy pixels in an image&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A seam is an optimal 8-connected path of pixels on a single image from&#xA;  top to bottom, or left to right, where optimality is defined by an&#xA;  image energy function&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, &lt;strong&gt;what does the 'energy' in an image mean, in this particular context?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;PS: I have gone through &lt;a href=&quot;https://stackoverflow.com/q/4562801/4993513&quot;&gt;this question&lt;/a&gt;, but the answers were very broad and vague because of &lt;a href=&quot;https://stackoverflow.com/questions/4562801/what-is-energy-in-image-processing#comment5004572_4562801&quot;&gt;this reason&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2019-04-08T11:01:50.277" Title="What does 'energy' in image processing mean?" Tags="|computer-vision|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27287" PostTypeId="2" ParentId="27286" CreationDate="2018-01-31T10:38:33.380" Score="2" Body="&lt;p&gt;I quote from &lt;a href=&quot;https://en.wikipedia.org/wiki/Energy_(signal_processing)&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which our friend has provided.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Energy in this context is not, strictly speaking, the same as the conventional notion of energy in physics and the other sciences. The two concepts are, however, closely related, and it is possible to convert from one to the other:&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KV0IX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KV0IX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;where Z represents the magnitude, in appropriate units of measure, of the load driven by the signal.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also for figuring out the interpretation of energy in this context you can see &lt;a href=&quot;https://electronics.stackexchange.com/a/77679/111225&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;After a while, I found a bit better explanation for energy as you can read the answers from &lt;a href=&quot;https://stackoverflow.com/q/4562801/5120235&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Energy is used to describe a measure of &quot;information&quot; when formulating an operation under a probability framework such as MAP (maximum a priori) estimation in conjunction with Markov Random Fields. Sometimes the energy can be a negative measure to be minimised and sometimes it is a positive measure to be maximized.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;or you can read:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Energy is a fairly loose term used to describe any user defined function (in the image domain).&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-17T17:45:03.237" LastActivityDate="2018-02-17T17:45:03.237" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27288" PostTypeId="1" AcceptedAnswerId="30632" CreationDate="2018-01-31T11:23:54.903" Score="2" ViewCount="3826" Body="&lt;p&gt;I want to retrieve the list of trainable variables/weights in my model (wrapped in a &lt;code&gt;tf.Estimator&lt;/code&gt;). However, &lt;code&gt;tf.trainable_variables&lt;/code&gt; always returns an empty list, what am I doing wrong?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;import sys&#xA;import globals&#xA;import pkg_resources&#xA;import numpy as np&#xA;&#xA;import tensorflow as tf&#xA;&#xA;&#xA;def cnn_model_fn(features, labels, mode):&#xA;&#xA;    input_layer = tf.reshape(features[&quot;x&quot;], [-1, 51, 13])&#xA;    input_layer = tf.cast(input_layer, tf.float32)&#xA;&#xA;    # Convolutional Layer #1&#xA;    conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding=&quot;same&quot;, activation=tf.nn.relu)&#xA;&#xA;    # Pooling Layer #1&#xA;    pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)&#xA;&#xA;    # Convolutional Layer #2 and Pooling Layer #2&#xA;    conv2 = tf.layers.conv1d(inputs=pool1, filters=64, kernel_size=5, padding=&quot;same&quot;, activation=tf.nn.relu)&#xA;&#xA;    # Pooling layer #2&#xA;    pool2 = tf.layers.max_pooling1d(inputs=conv2, pool_size=2, strides=2)&#xA;&#xA;    # flatten the feature map&#xA;    pool2_flat = tf.reshape(pool2, [-1, 12 * 64])&#xA;&#xA;    # Dense Layer&#xA;    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)&#xA;    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)&#xA;&#xA;    # Logits Layer&#xA;    logits = tf.layers.dense(inputs=dropout, units=5)&#xA;&#xA;    predictions = {&#xA;        # Generate predictions (for PREDICT and EVAL mode)&#xA;        &quot;classes&quot;: tf.argmax(input=logits, axis=1),&#xA;        # Add `softmax_tensor` to the graph. It is used for PREDICT and by the `logging_hook`.&#xA;        &quot;probabilities&quot;: tf.nn.softmax(logits, name=&quot;softmax_tensor&quot;)&#xA;    }&#xA;&#xA;    if mode == tf.estimator.ModeKeys.PREDICT:&#xA;        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)&#xA;&#xA;    # Calculate Loss (for both TRAIN and EVAL modes)&#xA;    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)&#xA;&#xA;    # Configure the Training Op (for TRAIN mode)&#xA;    if mode == tf.estimator.ModeKeys.TRAIN:&#xA;        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)&#xA;        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())&#xA;        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)&#xA;&#xA;    # Add evaluation metrics (for EVAL mode)&#xA;    eval_metric_ops = {&quot;accuracy&quot;: tf.metrics.accuracy(labels=labels, predictions=predictions[&quot;classes&quot;])}&#xA;&#xA;    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)&#xA;&#xA;def main(unused_argv):&#xA;&#xA;    total = pd.read_feather('testfile.feather')&#xA;    labels = total['labels']&#xA;    features = total.iloc[:, 16:679]&#xA;&#xA;    mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, model_dir=&quot;/tmp/gait_convnet_model&quot;)&#xA;&#xA;    # Log the values in the &quot;Softmax&quot; tensor with label &quot;probabilities&quot;&#xA;    tensors_to_log = {&quot;probabilities&quot;: &quot;softmax_tensor&quot;}&#xA;    logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=50)    &#xA;&#xA;    # Train the model&#xA;    train_input_fn = tf.estimator.inputs.numpy_input_fn(x={&quot;x&quot;: np.array(features)}, y=np.array(labels), batch_size=100, num_epochs=None, shuffle=True)&#xA;    mnist_classifier.train(input_fn=train_input_fn, steps=1, hooks=[logging_hook])&#xA;&#xA;    temp_list = tf.trainable_variables()&#xA;    print(temp_list)&#xA;&#xA;if __name__ == '__main__':&#xA;    tf.app.run()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42681" LastActivityDate="2018-04-22T14:03:24.280" Title="TensorFlow: trainable_variables() is empty" Tags="|python|neural-network|tensorflow|convolution|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27290" PostTypeId="2" ParentId="18842" CreationDate="2018-01-31T11:25:03.757" Score="1" Body="&lt;p&gt;Both approaches seem to do the same:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;safe_sparse_dot(X, self.coef_.T, dense_output=True) + self.intercept_ &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;seems to compute a dot product of the coefficients (you called these $w_1, w_2$, and combined them to $\theta$), and the point ($x_1, x_2$, combined to $\mathbf{X}$). The intercept $b$ is added. So the formulas should do exactly the same, once in mathematical, once in Python notation: they compute the distance to the separating hyperplane, called the &lt;em&gt;margin&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also wanted to know&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;how it works in the case of SGD Classifier&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As that also fits a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn-linear-model-sgdclassifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;linear classifier&lt;/a&gt;, the computation should be the same (even though the hyperplane might have different parameters).&lt;/p&gt;&#xA;" OwnerUserId="34009" LastActivityDate="2018-01-31T11:25:03.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27291" PostTypeId="2" ParentId="27268" CreationDate="2018-01-31T12:20:11.757" Score="0" Body="&lt;p&gt;It mainly depends on the task you are looking to accomplish with your time series data. Is it a regression or a classification task. In general, Random Forests are known to be easy to use and pretty useful. So I think it's a good way to start. Of course, dimensionality reduction techniques and feature extractors like PCA, kernel PCA, t-SNE and many others are very useful if you want to get better results. Neural networks are also useful, depending on the task you can try both recurrent (LSTM) or convolutional NN.  &lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2018-01-31T12:20:11.757" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27293" PostTypeId="1" AcceptedAnswerId="27316" CreationDate="2018-01-31T13:01:54.077" Score="0" ViewCount="898" Body="&lt;p&gt;This example is taken from the book Deep Learning With Python from Jason Brownlee. It  applies a fully connected neural model with one hidden layer  (13, 13, 1) using Keras library and the Tensorflow backend.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Import the packages&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.wrappers.scikit_learn import KerasRegressor&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.model_selection import KFold&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.datasets import load_boston&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;2 - Load the dataset&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;boston = load_boston()&#xA;X = boston.data[:,0:13]&#xA;Y = boston.target&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;3 - Define base model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def baseline_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Dense(13, input_dim=13, \&#xA;                    kernel_initializer='normal', activation='relu'))&#xA;    model.add(Dense(1, kernel_initializer='normal'))&#xA;    # Compile model&#xA;    model.compile(loss='mean_squared_error', optimizer='adam')&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;4 - Fix random seed for reproducibility&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seed = 7&#xA;numpy.random.seed(seed)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;5 - Fit &amp;amp; evaluate model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimator = KerasRegressor(build_fn=baseline_model, epochs=10, \ &#xA;                           batch_size=5, verbose=0)&#xA;kfold = KFold(n_splits=10, random_state=seed)&#xA;results = cross_val_score(estimator, X, Y, cv=kfold)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;6 - Value of &lt;code&gt;results&lt;/code&gt; parameter&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Out[63]: &#xA;array([-13.89524042, -14.2215869 ,  -6.21156782, -42.65242339,&#xA;       -26.58890147, -56.30860755, -28.6575911 , -89.67339525,&#xA;       -27.7172946 , -22.67604859])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which returns a mean negative value&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(&quot;Baseline: %.2f (%.2f) MSE&quot; % (results.mean(), results.std()))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It returns about -30 and it should return about +30 according to the book. I've tried both Theano and Tensorflow with no success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also tested this code both on Windows and Linux, having obtained the same bad result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem seems to be in the cross-validation part, because if I don't run cross validation, I get more sensible results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7 - Without cross-validation&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(13, input_dim=13, kernel_initializer='normal', \ &#xA;                activation='relu'))&#xA;model.add(Dense(1, kernel_initializer='normal')) &#xA;model.compile(loss='mean_squared_error', optimizer='adam')&#xA;model.fit(X, Y, epochs=150, batch_size=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if I evaluate the model I get a more sensible value&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [68]: model.evaluate(X, Y)&#xA;506/506 [==============================] - 0s 58us/step&#xA;Out[68]: 27.778296670423664&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What can be happening here? Why is the cross-validation procedure returning negative values?&lt;/p&gt;&#xA;" OwnerUserId="15199" LastActivityDate="2019-09-03T18:15:06.020" Title="ANN applied to Boston Housing dataset returns negative value" Tags="|python|neural-network|regression|keras|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27294" PostTypeId="1" CreationDate="2018-01-31T13:14:08.327" Score="1" ViewCount="37" Body="&lt;p&gt;I’m working with segmented cells from thin blood smear images and using deep learning models to classify parasitic and uninfected cells. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I obtained the following values: Accuracy: 0.986, AUC: 0.99, Sensitivity: 0.981, Specificity: 0.992, F1-score: 0.987, MCC: 0.972, TP: 1329 , FN: 26, FP: 11, TN: 1364. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In one of my submissions, I have got a question from a reviewer like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given a system (with p sensitivity, r specificity), how many RBCs need to be analyzed to confidently decide that a person is infected by malaria (given he has q% parasitemia). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the above reasoning/study, report how many RBCs you need to analyze to confidently decide (statistically significant) a person (with q% parasitemia) is infected or not by the best system that you have developed. Also report by the same system, what is the lower level of parasitemia you can diagnose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest me an answer (assuming the degree of parasitemia) since we don't have that measure.&lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="1330" LastEditDate="2021-07-16T12:05:24.197" LastActivityDate="2021-07-16T12:05:24.197" Title="How many samples are needed to find a positive case in classification?" Tags="|classification|deep-learning|performance|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27295" PostTypeId="1" AcceptedAnswerId="27342" CreationDate="2018-01-31T13:28:51.617" Score="1" ViewCount="91779" Body="&lt;p&gt;I am trying to perform two sample t test. My data set consists of 744 rows and 186 columns for which I have calculated total sum and mean. I need to perform two sample t test. My csv looks like this from which I have to calculate ttest and rank sum test for each row as individual row denotes separate ID and have the corresponding values :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SRA ID  ERR169499            ERR169498           ERR169497&#xA;Label   1                    0                   1&#xA;TaxID   PRJEB3251_ERR169499  PRJEB3251_ERR169499 PRJEB3251_ERR169499&#xA;333046  0.05                 0.99                99.61&#xA;1049    0.03                 2.34                34.33&#xA;337090  0.01                 9.78                23.22&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The labels 0 and 1 are for case and control respectively. So far I have done this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from scipy.stats import ttest_ind&#xA;from scipy.stats import ranksums&#xA;&#xA;def transposer(filename):&#xA;file = open(filename, 'rt')&#xA;pd.read_csv(file).T.to_csv(str(filename).split(&quot;/&quot;)&#xA;[-1].split(&quot;.&quot;)[0]+'_transposed.csv',header=False)&#xA;&#xA;&#xA;pd.read_csv('project.csv').T.to_csv('transposed.csv', header=False)&#xA;&#xA;file = open('transposed.csv', 'rt')&#xA;out = open('final_out.csv', 'w')&#xA;meta = open('Meta3251.csv', 'rt')&#xA;contents = {}&#xA;for ids in meta:&#xA;    contents[ids.split(',')[1]]=ids.split(',')[-1]&#xA;count = 0&#xA;for row in file:&#xA;    if count == 0:&#xA;    out.write('SraID, Label,'+row)&#xA;    count=1&#xA;else:&#xA;&#xA;    try:&#xA;        pid = row.split(',')[0].split('_')[1]&#xA;out.write(pid.replace('\n','')+','+contents[pid].replace('\n','')&#xA;+','+str(row))&#xA;        out.flush()&#xA;    except:&#xA;        print(pid)&#xA;        pass&#xA;file.close()&#xA;out.close()&#xA;transposer('final_out.csv')&#xA;file1 = open('final_out_transposed.csv','rt')&#xA;label = []&#xA;data = {}&#xA;&#xA;x = open('final_out_transposed.csv','rt')&#xA;for r in x:&#xA;    datas = r.split(',')&#xA;    if datas[0] == ' Label':&#xA;        label.append(r.split(&quot;,&quot;)[1:])&#xA;label = label[0]&#xA;label[-1] = label[-1].replace('\n','')&#xA;counter = len(label)&#xA;for row in file1:&#xA;    content = row.split(',')&#xA;if content[0]=='SraID' or content[0]== 'TaxID' or content[0]==' Label':&#xA;    pass&#xA;else:&#xA;    dt = row.split(',')&#xA;    dt[-1] = dt[-1].replace('\n','')&#xA;&#xA;    data[dt[0]]=dt[1:]&#xA;keys = list(data)&#xA;sum_file = open('sum.csv','w')&#xA;sum_file.write('TaxId,sum_case,sum_ctrl,case_count,&#xA;ctrl_count,case_mean,ctrl_mean,\n')&#xA;for key in keys:&#xA;    sum_case = 0&#xA;    sum_ctrl = 0&#xA;    count_case = 0&#xA;    count_ctrl = 0&#xA;    mean_case = 0&#xA;    mean_ctrl = 0&#xA;for i in range(counter):&#xA;    if label[i] == '0':&#xA;        sum_case=np.float64(sum_case)+np.float64(data[key][i])&#xA;        count_case = count_case+1&#xA;        mean_case = sum_case/count_case&#xA;    else:&#xA;        sum_ctrl = np.float64(sum_ctrl)+np.float64(data[key][i])&#xA;        count_ctrl = count_ctrl+1&#xA;        mean_ctrl = sum_ctrl/count_ctrl&#xA;sum_file.write(key+','+str(np.float64((sum_case)))+','&#xA;&#xA;+str(np.float64((sum_ctrl)))+','+str(np.float64((count_case)))        &#xA;+','+str(np.float64((count_ctrl)))+','+str(np.float64((mean_case)))&#xA;+','+str(np.float64((mean_ctrl)))+'\n')&#xA;sum_file.flush()&#xA;sum_file.close()&#xA;&#xA;df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])&#xA;case = df.xs('0', axis=1, level=0).dropna()&#xA;ctrl = df.xs('1', axis=1, level=0).dropna()&#xA;(tt_val, p_ttest) = ttest_ind(case, ctrl, equal_var=False)&#xA;print (tt_val)&#xA;print (p_ttest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting the error: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: operands could not be broadcast together with shapes (92,) (95,)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I handle this error. I cannot change my data.&lt;/p&gt;&#xA;" OwnerUserId="45433" LastEditorUserId="29575" LastEditDate="2018-02-01T01:36:22.817" LastActivityDate="2018-02-01T14:31:17.480" Title="ValueError: operands could not be broadcast together with shapes while using two sample independent t test" Tags="|python|statistics|scipy|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="27296" PostTypeId="1" AcceptedAnswerId="27304" CreationDate="2018-01-31T13:36:06.113" Score="1" ViewCount="118" Body="&lt;p&gt;I'm developing machine learning algorithms to aid in the diagnosis and prognosis of various cancers for my PhD. My lab is an Nvidia teaching center (CUDA). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My supervisor thinks that I need to also optimize ML by parallelizing it in CUDA. However, as I see it, a model is trained once and there is no need to train again. Testing a model is also not time consuming. My interests are in ML, not Parallel Processing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Should I spend a large chunk of my time parallelizing with CUDA?&lt;br&gt;&#xA;2) Is CUDA still a viable framework for research? &lt;br&gt;&#xA;3) In the world outside of research, will this make it easier to get a ML job?&lt;/p&gt;&#xA;" OwnerUserId="45506" LastActivityDate="2018-01-31T16:30:30.060" Title="Does it make sense to parallelize machine learning algorithms as part of PhD research?" Tags="|machine-learning|research|nvidia|" AnswerCount="1" CommentCount="2" ClosedDate="2018-02-01T03:48:06.937" ContentLicense="CC BY-SA 3.0" />
  <row Id="27297" PostTypeId="1" CreationDate="2018-01-31T14:02:52.823" Score="1" ViewCount="566" Body="&lt;p&gt;Doing some data science on ~4 million samples, with lots of columns being categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One column has ~1000 categories and my boss insists on including it in the analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My output is also predicting classes (I'll use &lt;code&gt;gnb.predict_proba()&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I'm taking a random subset of my data for partial fitting, and repeating.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# train = ~3 million rows of data as a dataframe&#xA;gnb = naive_bayes.GaussianNB()&#xA;for i in range(10):&#xA;    dds = train.sample(n=10**4)&#xA;    (dfX,dfY) = makeXY(dds) #gets one-hot- encoded X and Y dataframes&#xA;    gnb.partial_fit(dfX,[getClass(x) for x in dfY.values],classes=np.unique([getClass(x) for x in dfY.values]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I ensure I get all the possible classes AND that they are in the same order every time?&lt;/p&gt;&#xA;" OwnerUserId="41807" LastEditorUserId="29575" LastEditDate="2018-01-31T14:28:22.040" LastActivityDate="2024-03-02T07:04:00.507" Title="partial fitting, how to ensure one hot captures all features consistently" Tags="|python|pandas|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27298" PostTypeId="1" AcceptedAnswerId="27332" CreationDate="2018-01-31T14:51:31.867" Score="4" ViewCount="13101" Body="&lt;p&gt;I am currently reading a white paper relating to Expectation-Maximisation (EM) and would like to encode a formula so I can play with it in order to help my understanding. The formula in question is a sum over values and shown below;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WMHsq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WMHsq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering what the advice on the best way to acheive this would be. I am currently thinking about using a for loop to create some kind of cumulative value as I iterate from t = 1 to m.&lt;/p&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2018-02-01T11:44:50.350" Title="Recreating the sum symbol using python" Tags="|python|mathematics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27303" PostTypeId="2" ParentId="5443" CreationDate="2018-01-31T15:48:36.133" Score="1" Body="&lt;p&gt;Excel can be an excellent tool. Sure, depending on what you do, it might not fit the bill but if it does, it would be almost foolish to dismiss it. While it takes a while to set up your pipeline, in Excel you can hit the ground pretty much running: built-in UI, easy extendibility via VBA even with Python (e.g. &lt;a href=&quot;https://www.xlwings.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.xlwings.org&lt;/a&gt;). It might not be ideal when it comes to stuff like version control but there are ways to make it work with Git (e.g. &lt;a href=&quot;https://www.xltrail.com/blog/auto-export-vba-commit-hook&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.xltrail.com/blog/auto-export-vba-commit-hook&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="16108" LastActivityDate="2018-01-31T15:48:36.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27304" PostTypeId="2" ParentId="27296" CreationDate="2018-01-31T16:30:30.060" Score="1" Body="&lt;p&gt;From practise point of view, just sharing some thoughts. I don't have any research (PhD) type of experience, and your supervisor may communicate similar contents below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And, paralleling calculation here, personally, I defined as single PC that utils graphical card GPU cores for calculation acceleration as oppose to clustering calculation.  &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Some Theoretical Thoughts&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Sometimes, it worth building multiple models, or even, same model multiple times on the same data in ML world.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example would be like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;using cross validation (same method multiple times) to have robust model output or parameters. Or&lt;/li&gt;&#xA;&lt;li&gt;model ensemble with lots of weak learners (multiple models) to have better accuracy.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Those ML process are time consuming, and paralleling can help reducing the time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, from the info provided, I assume your ML project is image recognition with GPU acceleration. But I've no idea what's the main purpose of the project.  It could be developing/improving new ML methods, or comparing known ML methods to form a academy review.  Whatever the case, I assume, the result should still achieve some certain accuracy level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, it would make sense to consider some efficient method (e.g. paralleling calculation) to accelerate the modelling process.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Some Practical Considerations&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In practise, efficiency is very import.  When you have a theoretically accurate model, but takes a long time to build models, it won't be acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can step back to say let's come back to some simple quick model with less accuracy. But what's the point of ML when traditional methods have similar or better accuracy?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Personal Answer to Questions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;1) Should I spend a large chunk of my time parallelizing with CUDA?&lt;/em&gt;&#xA;Paralleling is useful to make the ML training quicker.  And CUDA is a cool technique to learn and apply to ML.  Just balance the time between main purpose of the project and time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is CUDA still a viable framework for research? &#xA;This meant to be the first chapter of your report / essay / dissertation.  At least, more and more ML uses deep learning, which can be much quicker with GPU involved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) In the world outside of research, will this make it easier to get a ML job?&#xA;I cannot definitely say knowing CUDA/paralleling would make you a top player (as other aspects would also be considered). But when other candidates have similar background, one with paralleling calculation experience would stand out.&lt;/p&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-01-31T16:30:30.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27305" PostTypeId="2" ParentId="26134" CreationDate="2018-01-31T16:43:22.873" Score="14" Body="&lt;p&gt;&lt;strong&gt;If you choose zero initial weights, then the perceptron algorithm's learning rate &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; has no influence on a neuron's predicted class label.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because the decision function used in the perceptron algorithm depends only on the &lt;em&gt;sign&lt;/em&gt; of &lt;span class=&quot;math-container&quot;&gt;$z$&lt;/span&gt;: &#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\phi(z) = \begin{cases} 1 &amp;amp;\text{ if } z \ge 0\\ -1 &amp;amp;\text{ otherwise.} \end{cases}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider a perceptron algorithm with a single neuron.&#xA;(I will write &lt;span class=&quot;math-container&quot;&gt;${\bf x} \cdot {\bf y}$&lt;/span&gt; for the vector product &lt;span class=&quot;math-container&quot;&gt;${\bf x}^T {\bf y}$&lt;/span&gt; to avoid ugly double superscripts).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Pick an initial weight vector &lt;span class=&quot;math-container&quot;&gt;${\bf w}^{(0)}.$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;Plug in the first input vector &lt;span class=&quot;math-container&quot;&gt;${\bf x^{(1)}}$&lt;/span&gt; and predict the class label&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\hat{y}^{(1)}  = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}).&#xA;$$&lt;/span&gt;&#xA;This gives the weight update&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)},&#xA;$$&lt;/span&gt;&#xA;where &lt;span class=&quot;math-container&quot;&gt;$\eta \in (0, 1)$&lt;/span&gt; is the learning rate and &lt;span class=&quot;math-container&quot;&gt;$y^{(1)}$&lt;/span&gt; is the true class label.&#xA;The new weights are &lt;span class=&quot;math-container&quot;&gt;${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$&lt;/span&gt; &lt;/li&gt;&#xA;&lt;li&gt;Similarly, plug in our the input vector &lt;span class=&quot;math-container&quot;&gt;${{\bf x}^{(2)}}$&lt;/span&gt; and predict the class label &#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}).&#xA;$$&lt;/span&gt; &#xA;This gives the weight update&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)},&#xA;$$&lt;/span&gt;&#xA;where &lt;span class=&quot;math-container&quot;&gt;$y^{(2)}$&lt;/span&gt; is the true class label.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Notice that &lt;span class=&quot;math-container&quot;&gt;$\Delta{\bf w}^{(2)}$&lt;/span&gt; implicitly depends on &lt;span class=&quot;math-container&quot;&gt;$\Delta{\bf w}^{(1)}$&lt;/span&gt;, which in turn implicitly depends on &lt;span class=&quot;math-container&quot;&gt;${\bf w}^{(0)}$&lt;/span&gt;.&#xA;Let us unravel these dependencies by plugging in:&#xA;&lt;span class=&quot;math-container&quot;&gt;\begin{align*}&#xA;\Delta {\bf w}^{(2)} &amp;amp;= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\&#xA;&amp;amp;= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\&#xA;&amp;amp;= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\&#xA;&amp;amp;=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\&#xA;&amp;amp;=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.&#xA;\end{align*}&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose we have initialized with zero weights: &lt;span class=&quot;math-container&quot;&gt;${\bf w}^{(0)} = {\bf 0}.$&lt;/span&gt;&#xA;Since &lt;span class=&quot;math-container&quot;&gt;${\bf 0} \cdot {\bf x}^{(1)} = 0$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$\phi(0) = 1,$&lt;/span&gt; the last line in this calculation simplifies to&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us zoom in here and look at &lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt; and its argument:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big).&#xA;$$&lt;/span&gt;&#xA;Since &lt;span class=&quot;math-container&quot;&gt;$\eta &amp;gt; 0$&lt;/span&gt;, it does not change the sign of &lt;span class=&quot;math-container&quot;&gt;$(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$&lt;/span&gt; But the sign is all that matters to the function &lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt;.&#xA;So we can simply remove &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; from the function argument without changing the result:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same arguments hold for &lt;span class=&quot;math-container&quot;&gt;$\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It follows that&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\Delta {\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$&lt;/span&gt;&#xA;for every &lt;span class=&quot;math-container&quot;&gt;$i = 1, 2, \dots$&lt;/span&gt;&#xA;This is what your quote means by saying that the parameter &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; &quot;affects only the scale of the weight vector, not the direction.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For nonzero initial weights, this is different. Recall the last line of the long calculation &lt;span class=&quot;math-container&quot;&gt;$\Delta {\bf w}^{(2)} = \dots$&lt;/span&gt; above and zoom in again on the function &lt;span class=&quot;math-container&quot;&gt;$\phi$&lt;/span&gt; and its argument:&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big).&#xA;$$&lt;/span&gt;&#xA;If &lt;span class=&quot;math-container&quot;&gt;${\bf w}^{(0)}$&lt;/span&gt; is nonzero, then &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; may affect the sign of the function argument, and therefore the predicted class label.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2020-03-05T09:12:31.037" LastActivityDate="2020-03-05T09:12:31.037" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27306" PostTypeId="1" CreationDate="2018-01-31T17:15:29.013" Score="1" ViewCount="1567" Body="&lt;p&gt;I am working on implementing an autoencoder for unsupervised learning, and I have some questions about the overall process. From what I was reading &lt;a href=&quot;https://github.com/keras-team/keras/issues/3464&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, @rjpg suggests the following general approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Slice the NN in half on the last encoder (before the decode starts)&lt;/li&gt;&#xA;&lt;li&gt;Freeze the the weights of the encoders&lt;/li&gt;&#xA;&lt;li&gt;Add or concat a new NN in front of the last encoding layer&lt;/li&gt;&#xA;&lt;li&gt;(post) Train this concat NN with a softmax output layer to classify the digit&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I haven't been able to find an example of this using Keras.  I have seen examples in R or H20, but the code is so encapsulated, it is difficult for me to implement a similar solution in Keras.  I have three questions related to how to implement this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this strategy seem reasonable?&lt;/li&gt;&#xA;&lt;li&gt;How to freeze the weights of the encoders in Keras?&lt;/li&gt;&#xA;&lt;li&gt;What is meant by adding or concatting a new NN in front of the last encoding layer, or why is this needed?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="45397" LastActivityDate="2018-01-31T17:15:29.013" Title="General unsupervised learning strategy when using convolutional autoencoder (CAE)" Tags="|keras|unsupervised-learning|convolution|autoencoder|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27307" PostTypeId="1" AcceptedAnswerId="27452" CreationDate="2018-01-31T17:20:04.507" Score="2" ViewCount="1147" Body="&lt;p&gt;I'm trying to code my own logistic regression algorithm using Andrew NG's machine learning using Octave. lectures. So what I did was make a csv file, the first row being some parameter and the second one being the result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;121,1&#xA;124,0&#xA;97,0&#xA;104,0&#xA;110,0&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Overall there are only 24 examples, but I've chosen points such that some &lt;a href=&quot;https://i.stack.imgur.com/M1Ute.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;pattern can be followed&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = load('data.dat');&#xA;x = data(:, 1);&#xA;&#xA;y = data(:, 2)&#xA;m = length(y);&#xA;&#xA;#plot(x, y, 'rx', 'MarkerSize', 10);&#xA;#xlabel('IQ');&#xA;#ylabel('Pass/Fail');&#xA;#title('Logistic Regression');&#xA;&#xA;x = [ones(size(x, 1), 1) x];&#xA;alpha = 0.00001;&#xA;i = 15000;&#xA;&#xA;g = inline(&quot;1 ./ (1 + exp(-z))&quot;)&#xA;&#xA;theta = zeros(size(x(1, :)))';&#xA;j = zeros(i, 1);&#xA;&#xA;for num = 1:i&#xA;  z = x * theta;&#xA;  h = g(z);&#xA;  j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))&#xA;  grad = 1./m * x' * (h - y);&#xA;  theta = theta - alpha * grad;&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However &lt;a href=&quot;https://i.stack.imgur.com/KHk85.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;the output&lt;/a&gt; of the sigmoid function shows every value below 0.5... surely this has to be wrong. I've also tried with different learning rates and iterations, but to no avail. What is wrong with the code, or data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="45515" LastEditorUserId="45515" LastEditDate="2018-02-04T15:47:41.827" LastActivityDate="2018-02-05T08:55:58.433" Title="logistic regression algorithm fails to work" Tags="|machine-learning|logistic-regression|gradient-descent|cost-function|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27309" PostTypeId="1" CreationDate="2018-01-31T18:38:34.280" Score="1" ViewCount="30" Body="&lt;p&gt;I have a large dataset that has session length records per user basis. And I am trying to predict the purchase behaviour based on the session length. But this data has multiple zero in the session length column. How should I deal with these zeros? Should I impute the zeros with other values? Kindly suggest.&lt;/p&gt;&#xA;" OwnerUserId="45519" LastActivityDate="2018-01-31T22:04:43.120" Title="Re: Missing Value" Tags="|data-mining|predictive-modeling|model-selection|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27311" PostTypeId="1" CreationDate="2018-01-31T20:28:47.240" Score="3" ViewCount="3943" Body="&lt;p&gt;My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is too costly for us to emulate the agent (executing the action and assessing the reward) meaning our only option is to learn the optimal policy on our dataset using an off-policy algorithm. However I still do not know how to do that, Q-learning, while off-policy, still require an emulator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you please provide me some guideline on how to do that? Which type of algorithm should I use? &lt;/p&gt;&#xA;" OwnerUserId="45527" LastEditorUserId="29575" LastEditDate="2018-01-31T22:57:13.413" LastActivityDate="2020-12-14T21:10:56.220" Title="Reinforcement Learning on data only (NO emulators)" Tags="|machine-learning|dataset|algorithms|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27312" PostTypeId="2" ParentId="27311" CreationDate="2018-01-31T20:54:31.097" Score="4" Body="&lt;p&gt;Q-learning does not in fact need to be online or need an emulator, it can learn exclusively from &lt;em&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/20535/understanding-experience-replay-in-reinforcement-learning&quot;&gt;experience replay&lt;/a&gt;&lt;/em&gt;. If you put all your history into a table or &lt;em&gt;state, action, reward, next state&lt;/em&gt; and then sample from it, it should be possible to train your agent that way.&lt;/p&gt;&#xA;&lt;p&gt;To do this, you will need to skip the algorithm steps that take actions and store results. The algorithm will then learn from the data you have. It will just not be possible to collect more. Depending on the problem you are trying to solve, this could be OK, or it may inhibit learning.&lt;/p&gt;&#xA;&lt;p&gt;RL algorithms learning optimal control in complex environments benefit from sampling near to their current policy, so it is possible in your case that your agent will reach a limit on what it can learn from historic data. It may end up still quite far from optimal behaviour, although it should stand a reasonable chance of improving on the best that the historic data shows.&lt;/p&gt;&#xA;&lt;p&gt;If you need to use function approximation (e.g. a neural network) due to size of the state, action space, then take extra care, because it will be hard to detect whether the action values have converged correctly. This is because you are learning the &lt;em&gt;optimal&lt;/em&gt; Q values, and you will have no test data that demonstrates what those should be (to collect that data, you need to follow the optimal policy and measure the total reward).&lt;/p&gt;&#xA;&lt;p&gt;Here is roughly what an experience-replay-only Q-learning algorithm would look like:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Input: History &lt;span class=&quot;math-container&quot;&gt;$H$&lt;/span&gt;, consisting of rows of &lt;span class=&quot;math-container&quot;&gt;$S,A,R,S'$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Initialise the NN for calculating &lt;span class=&quot;math-container&quot;&gt;$\hat{q}(s,a)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Repeat until NN converges:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\qquad$&lt;/span&gt; Sample &lt;span class=&quot;math-container&quot;&gt;$S,A,R,S'$&lt;/span&gt; from &lt;span class=&quot;math-container&quot;&gt;$H$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\qquad$&lt;/span&gt; &lt;span class=&quot;math-container&quot;&gt;$tdtarget \leftarrow R + \gamma \text{max}_{a'}[\hat{q}(S',a')]$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\qquad$&lt;/span&gt; Train NN single step, &lt;span class=&quot;math-container&quot;&gt;$\hat{q}(S,A) \rightarrow tdtarget$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;You can make use of mini-batch processing to generate multiple &lt;span class=&quot;math-container&quot;&gt;$tdtarget$&lt;/span&gt; values at once and train on them. A worthwhile improvement for stability is to use a frozen copy of the neural network when calculating the &lt;span class=&quot;math-container&quot;&gt;$tdtarget$&lt;/span&gt; value, and update it only every N steps to be a copy of most recent network, with N maybe 1000 steps.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-01T12:40:00.013" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27313" PostTypeId="2" ParentId="27309" CreationDate="2018-01-31T22:04:43.120" Score="1" Body="&lt;p&gt;The question you have to ask yourself is whether or not having a 0 is really a missing value, or actually a session length. On it's own, it doesn't seem like a problem. Some users could have a really short session. I'm assuming it's on a website or something and they could have loaded the page than immediately left. It also depends on how the data came to you, there might have been some processing beforehand you don't know about. What if some of the values actually are 0, and other values are missing, but have been filled in with 0? You likely won't be able to tell the difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, if it is truly representing missing, from a statistical point of view, it's more conservative to throw out those records, even if you're potentially losing information in other columns. It's only valid to impute values if the data is &lt;a href=&quot;https://www.theanalysisfactor.com/missing-data-mechanism/&quot; rel=&quot;nofollow noreferrer&quot;&gt;missing at random&lt;/a&gt;. If it is, then there are several options for imputing, the simplest being using the mean/median value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a machine learning point of view, it doesn't matter what the missingness mechanism is as much, you just need to impute it with a value to be able to model at all. And since it's already 0, it's ready to go. But if you were to impute it, it would typically be done during variable preprocessing, such as if you want to center the column. After doing that, impute the mean of the centered variables. And there you have several options for centering, normalizing the column, min max scaling, max absolute value scaling, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some situations, even if you impute values, it may be useful to create an additional variable indicating whether or not the variable was missing to begin with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no simple answer, it's probably worth experimenting with a few options and seeing what works best for the data.&lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-01-31T22:04:43.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27314" PostTypeId="2" ParentId="27297" CreationDate="2018-01-31T22:32:21.023" Score="1" Body="&lt;p&gt;If you're using sklearn, this is a great use of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CountVectorizer&lt;/a&gt; as a workaround since you can specify a vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start, get a list of all the 1000 categories and set that as the vocabulary in the transformer. Then convert the column to a string data type and apply this transformer to each batch:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import CountVectorizer&#xA;&#xA;cv = CountVectorizer(vocabulary = pre_made_category_list)&#xA;endoded_variable_matrix = cv.fit_transform(dfX[categorical_column])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Even though it's technically a count vectorizor, since there is only one word in the string, the counts for each row will be 1 for the category it is and 0 for everything else, so its effectively one hot encoding the variable. The order of the columns will be the order of the vocabulary, so the matrix will be consistent between folds.&lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-01-31T22:32:21.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27315" PostTypeId="2" ParentId="26155" CreationDate="2018-01-31T22:35:19.370" Score="1" Body="&lt;p&gt;It depends on what kind of functionality you would want to achieve. In essence, a trained neural network is a mapping function, which takes some input and produces some output, which then may be variously interpreted (for example using a certain threshold for classification).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, if the desired functionality of your program would be coming up with a computational function, you could treat your trained neural network as the program itself. Multiple machine learning libraries allow that easily, producing a class instance which contains the trained neural network inside and allows you to call it with given parameters just like any other function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I assume that you are looking for a way to output complex computer code with control structures, etc. - that unfortunately is very difficult to achieve, since you'd need an effective way to interpret the neural network outputs as code - not just the commands, but their structure and ordering as well. Such interpretations would be so complex that they'd make it very difficult for any Neural Network training method to converge to anything sensible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To mitigate this problem, you can try machine learning techniques which manipulate and optimize the code &quot;directly&quot;. The most famous of such is Genetic Programming - in which a computer program in encoded as a set of &quot;genes&quot; which are then improved upon through genetic algorithm techniques (crossovers which recombine large parts of code and mutations which randomly change small parts of the code). This has been successfully applied to some programming problems such as the Towers of Hanoi (see &lt;a href=&quot;http://www.oakgp.org/towers-of-hanoi&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.oakgp.org/towers-of-hanoi&lt;/a&gt; ).&lt;/p&gt;&#xA;" OwnerUserId="45521" LastActivityDate="2018-01-31T22:35:19.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27316" PostTypeId="2" ParentId="27293" CreationDate="2018-01-31T23:21:15.260" Score="1" Body="&lt;p&gt;&quot;This is to be expected. sklearn has changed their API to invert their cost functions. Nothing to be concerned about.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got this answer from Dr. Jason Brownlee.&lt;/p&gt;&#xA;" OwnerUserId="15199" LastActivityDate="2018-01-31T23:21:15.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27317" PostTypeId="2" ParentId="26155" CreationDate="2018-02-01T00:05:35.023" Score="1" Body="&lt;p&gt;François Chollet addresses this in his book, &lt;a href=&quot;https://www.manning.com/books/deep-learning-with-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning with Python&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;9.2 The limitations of deep learning&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;The space of applications that can be&#xA;implemented with deep learning is nearly infinite. And yet, many&#xA;applications are completely out of reach for current deep-learning&#xA;techniques—even given vast amounts of human-annotated data. Say, for&#xA;instance, that you could assemble a dataset of hundreds of&#xA;thousands—even millions—of English-language descriptions of the&#xA;features of a software product, written by a product manager, as well&#xA;as the corresponding source code developed by a team of engineers to&#xA;meet these requirements. Even with this data, you could &lt;em&gt;not&lt;/em&gt; train a&#xA;deep-learning model to read a product description and generate the&#xA;appropriate codebase. That’s just one example among many. In general,&#xA;anything that requires reasoning—like programming or applying the&#xA;scientific method—long-term planning, and algorithmic data&#xA;manipulation is out of reach for deep-learning models, no matter how&#xA;much data you throw at them. Even learning a sorting algorithm with a&#xA;deep neural network is tremendously difficult.&lt;/p&gt;&#xA;&lt;p&gt;This is because a&#xA;deep-learning model is just a &lt;em&gt;chain of simple, continuous geometric&#xA;transformations&lt;/em&gt; mapping one vector space into another. All it can do&#xA;is map one data manifold X into another manifold Y , assuming the&#xA;existence of a learnable continuous transform from X to Y . A&#xA;deep-learning model can be interpreted as a kind of program; but,&#xA;inversely, &lt;em&gt;most programs can’t be expressed as deep-learning&#xA;models&lt;/em&gt;—for most tasks, either there exists no corresponding&#xA;deep-neural network that solves the task or, even if one exists, it&#xA;may not be &lt;em&gt;learnable&lt;/em&gt;: the corresponding geometric transform may be far&#xA;too complex, or there may not be appropriate data available to learn&#xA;it.&lt;/p&gt;&#xA;&lt;p&gt;Scaling up current deep-learning techniques by stacking more&#xA;layers and using more training data can only superficially palliate&#xA;some of these issues. It won’t solve the more fundamental problems&#xA;that deep-learning models are limited in what they can represent and&#xA;that most of the programs you may wish to learn can’t be expressed as&#xA;a continuous geometric morphing of a data manifold.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-01T00:05:35.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27320" PostTypeId="2" ParentId="26457" CreationDate="2018-02-01T02:57:05.043" Score="2" Body="&lt;p&gt;I may be missing the point, but you don't need to do this one variable at a time in a loop and then bind the columns together - that's not the R way.  Here's an alternative:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create some simulated data&#xA;n &amp;lt;- 10000&#xA;X &amp;lt;- data.frame(f_a_ = factor(sample(1:400, n, replace = TRUE)),&#xA;                f_b_ = factor(sample(1:60, n, replace = TRUE)),&#xA;                f_c_ = factor(sample(1:6, n, replace = TRUE)),&#xA;                f_d_ = factor(sample(1:5, n, replace = TRUE)),&#xA;                y = rep(1, n))&#xA;&#xA;# use R's inbuilt understanding of formulas to do the &quot;one hot encoding&quot; for you&#xA;x_onehot &amp;lt;- model.matrix(y ~ ., data = X)                &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15775" LastActivityDate="2018-02-01T02:57:05.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27321" PostTypeId="2" ParentId="26972" CreationDate="2018-02-01T03:33:30.547" Score="2" Body="&lt;p&gt;For those of you interested in penalising &lt;s&gt;&lt;em&gt;false negatives&lt;/em&gt;&lt;/s&gt; &lt;em&gt;false positives&lt;/em&gt;, the derivation is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let the loss for the $i$th instance be $L$. As usual, $p(x)$ is the sigmoid function. Finally, we will put a weight $\beta &amp;gt;1$ on the false positives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$L = -y\ln p - \beta(1-y) \ln(1-p)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{grad} = \frac{\partial L}{\partial x} = \frac{\partial L}{\partial p} \frac{\partial p}{\partial x} = p(\beta + y -\beta y) -y$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\text{hess} = \frac{\partial^2 L}{\partial x^2} = p(1-p)(\beta + y -\beta y)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def weighted_logloss(y_hat, dtrain):&#xA;    y = dtrain.get_label()&#xA;    p = y_hat&#xA;    beta = 5&#xA;    grad = p * (beta + y - beta*y) - y&#xA;    hess = p * (1 - p) * (beta + y - beta*y)&#xA;    return grad, hess&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39286" LastEditorUserId="45058" LastEditDate="2018-02-05T01:12:11.857" LastActivityDate="2018-02-05T01:12:11.857" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27322" PostTypeId="1" AcceptedAnswerId="27323" CreationDate="2018-02-01T05:28:00.247" Score="1" ViewCount="613" Body="&lt;p&gt;I am having unbalanced dataset(1:93) and want to use kappa's metric.&#xA;However, for that I need to capture how many correct predictions are made for each class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried understanding from &lt;a href=&quot;https://machinelearningmastery.com/custom-metrics-deep-learning-keras-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and other google links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to capture class wise #correct predictions made?&lt;/p&gt;&#xA;" OwnerUserId="45385" LastActivityDate="2018-02-01T05:38:44.447" Title="Get number of correct predictions for each class in Keras" Tags="|machine-learning|deep-learning|keras|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27323" PostTypeId="2" ParentId="27322" CreationDate="2018-02-01T05:38:44.447" Score="1" Body="&lt;p&gt;That's true. For understanding how many correct decision your classifier has made, confusion matrix can be used. The main diagonal illustrates that. It depicts how many data samples are correctly classified and how many are mislabeled to which class. you can take a look at &lt;a href=&quot;http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-01T05:38:44.447" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27326" PostTypeId="2" ParentId="14435" CreationDate="2018-02-01T06:33:46.223" Score="0" Body="&lt;p&gt;You can find a probability that a datapoint $d_i$ will be clustered into a particular cluster $k_j$, $P(k_j|d_i)$, by running k-means hundreds of times and counting how many times datapoint $d_i$ was assigned to cluster $k_j$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since cluster id's don't mean anything in real life, you can identify clusters across k-means iterations by utilizing the value of the centroids. I.e., after each k-means converges remap the cluster id's based on a list of id's indexed by centroid values.  &lt;/p&gt;&#xA;" OwnerUserId="25583" LastActivityDate="2018-02-01T06:33:46.223" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27327" PostTypeId="2" ParentId="25173" CreationDate="2018-02-01T07:23:57.150" Score="5" Body="&lt;p&gt;I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When outliers show up these values are different and the network was unable to construct these particular data points back and the error scores for these values should be typically higher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used this with Deeplearning4j with SparkDl4jMultiLayer with KLDivergence loss functions on all layers except output layer with MSE loss functions. with Sigmoid activation function.&lt;/p&gt;&#xA;" OwnerUserId="45550" LastActivityDate="2018-02-01T07:23:57.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27328" PostTypeId="1" AcceptedAnswerId="27790" CreationDate="2018-02-01T07:30:07.920" Score="5" ViewCount="848" Body="&lt;p&gt;I am a newbie in neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saw this article Object detection with deep learning and OpenCV.&#xA;These three type of neural network are shortlisted in the article&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1506.01497&quot; rel=&quot;noreferrer&quot;&gt;Faster R-CNNs&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1506.02640&quot; rel=&quot;noreferrer&quot;&gt;You Only Look Once (YOLO)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/abs/1512.02325&quot; rel=&quot;noreferrer&quot;&gt;Single Shot Detectors (SSDs)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Found a lot of online resources that help in understanding how the neural network actual works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As building a neural network from scratch is time consuming and not entirely foolproof to get the desired efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I came across these articles &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/06/transfer-learning-the-art-of-fine-tuning-a-pre-trained-model/&quot; rel=&quot;noreferrer&quot;&gt;Transfer learning &amp;amp; The art of using Pre-trained Models in Deep Learning&lt;/a&gt;,&lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/4THzO/transfer-learning&quot; rel=&quot;noreferrer&quot;&gt;Transfer Learning&lt;/a&gt; and &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;noreferrer&quot;&gt;Transfer Learning&lt;/a&gt;, where we transfer the learning from an existing pre trained network to train the object which we would desire the network to detect.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of these pre trained network have been trained on a data from either COCO or ImageNet or PASCAL VOC which contain different categories images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train one of these to count the number of bananas here in this image&lt;a href=&quot;https://i.stack.imgur.com/SbwB2.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SbwB2.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should my training set of banana images be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need a fixed resolution of the image I feed into the network, so can I even feed a half banana like this for the training? I don't have to remove the background for training. Please correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mlfZz.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mlfZz.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45545" LastActivityDate="2018-02-14T11:49:06.097" Title="Type of images used to train a neural network" Tags="|deep-learning|training|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27329" PostTypeId="1" CreationDate="2018-02-01T08:53:42.473" Score="3" ViewCount="1114" Body="&lt;p&gt;I have a historical data of orders from different customers of my company and delivery details. I wanted to develop a model which predicts the probability of customers order getting delivered before or on the &quot;Requested Delivery Date&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my data-set, i've details about &#xA;customer&#xA;delivery location&#xA;product&#xA;delivery type&#xA;product classification (Can be exported or not) etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was not able to start this model as I am not sure how to tackle this problem. It would be of great help, if you could suggest some reading material on the same or few link directing the same.  &lt;/p&gt;&#xA;" OwnerUserId="45553" LastActivityDate="2018-05-06T17:01:04.710" Title="Predicting probability of order arriving on or before customer Requested Delivery Date(RDD)" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27330" PostTypeId="1" CreationDate="2018-02-01T09:02:41.477" Score="9" ViewCount="2675" Body="&lt;p&gt;I'm trying to use OpenCV via Python to find multiple objects in a train image and match it with the key points detected from a query image. For my case, I'm trying to detect the tennis courts in the image provided below. I looked at the online tutorials and could only figure that it can only detect only one object. I thought of inserting a loop in for it to find multiple objects but I failed to do so. Any idea on how to do it?&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;I used SIFT as ORB does not work that well for my case.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here's the code and a sample set of images:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import cv2&#xA;from matplotlib import pyplot as plt&#xA;&#xA;MIN_MATCH_COUNT = 10&#xA;img1 = cv2.imread('Image 11.jpg',0)          # queryImage&#xA;img2 = cv2.imread('Image 5.jpg',0) # trainImage&#xA;&#xA;# Initiate SIFT detector&#xA;sift = cv2.xfeatures2d.SIFT_create()&#xA;&#xA;# find the keypoints and descriptors with SIFT&#xA;kp1, des1 = sift.detectAndCompute(img1,None)&#xA;kp2, des2 = sift.detectAndCompute(img2,None)&#xA;FLANN_INDEX_KDTREE = 1&#xA;index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)&#xA;search_params = dict(checks = 50)&#xA;flann = cv2.FlannBasedMatcher(index_params, search_params)&#xA;matches = flann.knnMatch(des1,des2,k=2)&#xA;&#xA;# store all the good matches as per Lowe's ratio test.&#xA;good = []&#xA;for m,n in matches:&#xA;    if m.distance &amp;lt; 0.7*n.distance:&#xA;        good.append(m)&#xA;if len(good)&amp;gt;MIN_MATCH_COUNT:&#xA;    src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)&#xA;    dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)&#xA;    M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)&#xA;    matchesMask = mask.ravel().tolist()&#xA;    h,w = img1.shape&#xA;    pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)&#xA;    dst = cv2.perspectiveTransform(pts,M)&#xA;    img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)&#xA;else:&#xA;    print( &amp;quot;Not enough matches are found - {}/{}&amp;quot;.format(len(good), MIN_MATCH_COUNT) )&#xA;    matchesMask = None&#xA;draw_params = dict(matchColor = (0,255,0), # draw matches in green color&#xA;                   singlePointColor = None,&#xA;                   matchesMask = matchesMask, # draw only inliers&#xA;                   flags = 2)&#xA;img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)&#xA;plt.imshow(img3, 'gray'),plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Train Image :&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/noqE8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/noqE8.jpg&quot; alt=&quot;Train Image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Query Image :&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0OYpz.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0OYpz.jpg&quot; alt=&quot;Query Image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I have already posted this question in Stack Overflow but was recommended by @sophros to try posting it here.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45554" LastEditorUserId="1330" LastEditDate="2020-11-29T19:27:18.473" LastActivityDate="2021-03-10T12:52:03.927" Title="Python : Feature Matching + Homography to find Multiple Objects" Tags="|python|computer-vision|opencv|object-recognition|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27332" PostTypeId="2" ParentId="27298" CreationDate="2018-02-01T11:44:50.350" Score="3" Body="&lt;p&gt;In most cases, I would go for NumPy.&#xA;Implement a Python function &lt;code&gt;f(t)&lt;/code&gt; that calculates the $t$-th summand.&#xA;Then run&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;result = np.array([f(t) for t in range(1,m+1)]).sum()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will be very fast, unless $m$ is so large that &lt;code&gt;[f(t) for t in range(1,m+1)]&lt;/code&gt; does not fit into memory. In this case, I would follow your approach and use a for-loop:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;result = 0&#xA;for t in range(1, m+1):&#xA;    result += f(t)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-01T11:44:50.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27333" PostTypeId="1" AcceptedAnswerId="27335" CreationDate="2018-02-01T12:05:02.440" Score="3" ViewCount="312" Body="&lt;p&gt;I am working on a project which aims at determining whether a patient has cervical issues or not, based on a certain movement (for instance, turning the head from left to right and so on).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each patient, I have one or more sets of coordinates acquired with a VR headset. One problem is, all the patients are assumed healthy and I cannot compare their data with patients who actually have cervical issues. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working with two coordinates at a time, not the three, and I am considering two approaches: the first is to use approximation (splines...), the second is to use concave hulls. I am a bit more inclined to use hulls and I thought that I could calculate the distance between two hulls A and B as follows: area(A\B)+area(B\A). Note that the curves are parametric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two questions:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is it possible to &quot;classify&quot; patients using a data set that only consists of healthy patients? Or to find a &quot;descriptor&quot; for them ?  &lt;/li&gt;&#xA;&lt;li&gt;If so, or assuming I can get data on unhealthy patients, what tools can I use to classify the curves? I did not find anybody who worked on a similar problem.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45560" LastEditorUserId="45560" LastEditDate="2018-02-01T17:24:54.313" LastActivityDate="2018-02-01T17:24:54.313" Title="How to classify parametric curves?" Tags="|machine-learning|classification|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27335" PostTypeId="2" ParentId="27333" CreationDate="2018-02-01T12:19:07.200" Score="2" Body="&lt;p&gt;Whenever you have skewed dataset, it means that you know some classes better than some others. In such cases it means that the data is your knowledge and there are learning algorithms for such occasions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider an important fact here. Suppose that you have feature vectors of conditions of a nuclear company and they describe whether the company is in danger of nuclear radiation or not. In such case it is clear that it does not happen a lot that you have infected companies so most of your data have label of healthy condition. You have so much knowledge about the healthy class but you don't know much about the infected class because you don't have much data consequently you don't know its distribution and you can not estimate it well. Whenever your data is skewed, it means that e.g. you have 1 million feature vectors of negative class and 5 feature vectors of positive class. Now suppose that you change the feature vectors. In such cases you have imbalanced data-set or you just have the data samples of some classes without some other, you can use &lt;a href=&quot;https://en.wikipedia.org/wiki/Anomaly_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;anomaly detection&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In data mining, anomaly detection (also outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset. Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-01T12:19:07.200" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="27336" PostTypeId="2" ParentId="23560" CreationDate="2018-02-01T12:30:55.317" Score="1" Body="&lt;p&gt;It might actually be called &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/p9gxz/intersection-over-union&quot; rel=&quot;nofollow noreferrer&quot;&gt;Intersection Over Union&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So how do you tell if your object detection algorithm is working well? In this video, you'll learn about a function called, &quot;Intersection Over Union&quot;. And as we use both for evaluating your object detection algorithm, as well as in the next video, using it to add another component to your object detection algorithm, to make it work even better. Let's get started. In the object detection task, you expected to localize the object as well. So if that's the ground-truth bounding box, and if your algorithm outputs this ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="45545" LastEditorUserId="29575" LastEditDate="2018-02-01T15:56:58.900" LastActivityDate="2018-02-01T15:56:58.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27337" PostTypeId="1" CreationDate="2018-02-01T12:40:10.053" Score="1" ViewCount="510" Body="&lt;p&gt;I was thinking of adding another column called &quot;Test / Train&quot; in which a 0 would indicate test and a 1 would indicate train data, and then as I'm collecting data end the train data collection phase and start entering each row with 1 in that column.  Then using select rows widget to select all rows with &quot;Test / Train&quot; = 0 or 1.  However, under PCA, the original columns go away so this wouldn't work!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How then, can I just select all rows with row id &gt; k?  I need this because the past data is supposed to predict a smaller set of future data, so I want to be sure that it works and not just select the test set randomly from the training data (which is an option in Test &amp;amp; Score).&lt;/p&gt;&#xA;" OwnerUserId="45450" LastActivityDate="2018-02-01T14:16:16.870" Title="In orange, how do you split a given table at a certain row id?" Tags="|predictive-modeling|prediction|orange|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27338" PostTypeId="2" ParentId="27230" CreationDate="2018-02-01T12:50:05.870" Score="1" Body="&lt;p&gt;A loop might give you more control.&#xA;The &lt;code&gt;csv&lt;/code&gt; package lets you read a CSV file line by line.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import csv&#xA;&#xA;# Open the file in read mode. If encoding is an issue, specify the correct encoding here.&#xA;with open('my_csv_file.csv', mode='r', encoding='UTF8') as file:&#xA;&#xA;    # Read CSV file. Specify the correct delimiter and how to handle quotation marks.&#xA;    lines = csv.reader(file, delimiter=',', quoting=csv.QUOTE_NONE)&#xA;&#xA;    # Exctract header.&#xA;    header = next(data)&#xA;    ... # Check if the header complies to your data model.&#xA;&#xA;    # Loop through body.&#xA;    for line in lines:&#xA;        ... # Check if the line complies to your data model.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the actual checking, have a look at the &lt;code&gt;assert&lt;/code&gt; command. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = 1.0&#xA;try:&#xA;    assert(isinstance, x, int)    # Check if x is an integer.&#xA;except AssertionError:&#xA;    x = int(x)    # Handle the error.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-01T17:42:25.337" LastActivityDate="2018-02-01T17:42:25.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27340" PostTypeId="1" AcceptedAnswerId="41020" CreationDate="2018-02-01T13:09:30.333" Score="0" ViewCount="161" Body="&lt;p&gt;Looking at the literature, there are 2 distinct approaches to LSTM &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Some people use recurrent weights with &lt;strong&gt;Input, Forget, Output&lt;/strong&gt; - notice, their equations don't even mention dataGate, they start from describing the &lt;span class=&quot;math-container&quot;&gt;$f$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$i$&lt;/span&gt; gate &lt;a href=&quot;http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/&quot; rel=&quot;nofollow noreferrer&quot;&gt;(1)&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;: &lt;a href=&quot;https://wikimedia.org/api/rest_v1/media/math/render/svg/2db2cba6a0d878e13932fa27ce6f3fb71ad99cf1&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lke this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://wikimedia.org/api/rest_v1/media/math/render/svg/2db2cba6a0d878e13932fa27ce6f3fb71ad99cf1&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://wikimedia.org/api/rest_v1/media/math/render/svg/2db2cba6a0d878e13932fa27ce6f3fb71ad99cf1&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Other people use recurrent weights with &lt;strong&gt;dataGate (z), Input, Forget, Output&lt;/strong&gt; &lt;a href=&quot;https://saurabhmathur96.github.io/deep-learning-notes/LSTM%20A%20Search%20Space%20Odyssey&quot; rel=&quot;nofollow noreferrer&quot;&gt;(1)&lt;/a&gt; &lt;a href=&quot;https://www.semanticscholar.org/paper/Occam-s-Gates-Raiman-Sidor/733ab0a2c81a55a41734e5171f1ac16882ecca6c/figure/0&quot; rel=&quot;nofollow noreferrer&quot;&gt;(2)&lt;/a&gt;  &lt;a href=&quot;https://datascience.stackexchange.com/a/17412/43077&quot;&gt;(3)&lt;/a&gt;, &lt;a href=&quot;http://nlpapps.nl/article1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;(4)&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IHVnl.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IHVnl.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having tried the first approach (&lt;strong&gt;I still used standard weights on dataGate but no recurrent weights in that place&lt;/strong&gt;) seems fine but a little odd - network converges well, but very rarely jumps over local mimima, as if too high learning rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally I like the second approach (recurrent weights on all 4 entry points), which one do I use?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to make things worse, the &lt;a href=&quot;http://www.jmlr.org/papers/volume3/gers02a/gers02a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;LSTM Peephole paper&quot;, page 121&lt;/a&gt; describes 3 connections (no peephole for dataGate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This destroys the equal-sizes of my matrices, because now 3 gates use the Usual, Recurrent and Peephole weights, but the very first gate only uses the Usual weights&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-11-27T01:09:53.883" LastActivityDate="2019-11-27T01:09:53.883" Title="recurrent weights on LSTM Data-gate? (the one with tanh)" Tags="|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27341" PostTypeId="2" ParentId="27337" CreationDate="2018-02-01T14:16:16.870" Score="1" Body="&lt;p&gt;Actually, I can add in a string column and should still be able to do the row selection trick even with PCA.  In the file widget you set that column as text / meta and it shows up in the forward row selection widget.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this will work.&lt;/p&gt;&#xA;" OwnerUserId="45450" LastActivityDate="2018-02-01T14:16:16.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27342" PostTypeId="2" ParentId="27295" CreationDate="2018-02-01T14:31:17.480" Score="2" Body="&lt;p&gt;The answer to this question would be :&#xA;The objects created by the &lt;code&gt;xs&lt;/code&gt; method of the &lt;code&gt;Pandas DataFrame&lt;/code&gt; look like &lt;code&gt;two-dimensional arrays&lt;/code&gt;. These must be &lt;code&gt;flattened&lt;/code&gt; to look like &lt;code&gt;one-dimensional arrays&lt;/code&gt; when passed to &lt;code&gt;ttest_ind&lt;/code&gt;. The values attribute of the Pandas objects gives a &lt;code&gt;numpy array&lt;/code&gt;, and the &lt;code&gt;ravel()&lt;/code&gt; method flattens the array to one-dimension. &#xA;It would go like : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])&#xA;case = df.xs('0', axis=1, level=0).dropna()&#xA;ctrl = df.xs('1', axis=1, level=0).dropna()&#xA;(tt_val,p_ttest ) = ttest_ind(case.values.ravel(), ctrl.values.ravel(), &#xA;equal_var=False)&#xA;print (tt_val)&#xA;print(p_ttest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45433" LastActivityDate="2018-02-01T14:31:17.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27343" PostTypeId="1" CreationDate="2018-02-01T14:54:01.003" Score="3" ViewCount="4055" Body="&lt;p&gt;If I am training on a GRU model, is there a way I can output the learnt parameters so that when I train next time with more data, I can initialize with those learnt parameters as a starting point?&lt;/p&gt;&#xA;" OwnerUserId="7812" LastActivityDate="2018-02-01T22:53:37.347" Title="Output trained parameters of Keras model" Tags="|tensorflow|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27344" PostTypeId="2" ParentId="24786" CreationDate="2018-02-01T15:02:58.343" Score="2" Body="&lt;p&gt;&lt;strong&gt;1. About clustering of Values&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Values has a specific meaning in that problem, you might want to select the categories arbitrarily setting ranges yourself. (using the same example as the above answer, this would look like: 1-3 : Bad, 4-6 : Average, 7-10 : Good). In this case, if the range of answers is not pre-defined (so it's not like high-school grades, but more like people's weight), make sure to use ranges of same size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to use a distance based algorithm for this purpose (e.g. kmeans), so that the ranges of the clusters won't be stable but automatically selected to minimize distances. It is similar as the t-shirts example in &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/93VPG/k-means-algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt; (starting 1.35) but with one variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both cases though you will have to set the number of clusters yourself. In the second case this can be done easier by visualizing the data and see if the clusters make sense. Since this is your output variable (and this procedure will be performed only once) I think you should not do this important step automatically but use your judgement before selecting the final grouping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. About feature selection:&lt;/strong&gt;&#xA;There are plenty feature selection algorithms you can choose from. &lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.295.8115&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; is a review paper that can give you some ideas. SVM-RFE (Support Vector Machines with Recursive Feature Elimination) is a very strong combination of feature selection technique with classification that usually works well. You can deal with categorical attributes by turning them into dummy variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Decision Tree Classifier could also do the job. They do the classification based on automatically generated rules, based to some criterion (e.g. entropy). You can see the importance of each attribute for the classification (GINI importance)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are very good implementation for both suggested approaches in python's scikit learn package:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SVM&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RFE&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision Tree Classifier&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Food for thought:&lt;/strong&gt; I don't know what is the exact problem you are trying to solve, but have you thought of using a Decision Tree Regressor directly to the data? You can use Values as the target-variable and the rest of attributes as your dataset. The end-leaves of the created Decision Tree will be the &quot;Values clusters&quot; and each one of the samples will be classified to one of them. (Also available in scikit learn toolbox &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-02-01T15:02:58.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27345" PostTypeId="1" AcceptedAnswerId="27350" CreationDate="2018-02-01T15:05:11.813" Score="0" ViewCount="794" Body="&lt;p&gt;I am currently about to do clustering analysis regarding Steam users activity. So I have a thousands of CSV’s, each representing a Steam user and his/her purchased games (with ID and genre). I am planning to use k-modes clustering because my data is categorical, and I have found a python library &lt;a href=&quot;https://pypi.python.org/pypi/kmodes/&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-modes&lt;/a&gt;. This is what a CSV file looks like, representing a random steam user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PTM3k.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PTM3k.png&quot; alt=&quot;CSV file image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have researched a lot online, but it seems that many people have all their data in one CSV file, just like this example of stock exchange data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1ZawT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1ZawT.png&quot; alt=&quot;Stock Data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I read each file and represent it as one unique data then do clustering on all the data among the thousands of CSV files?&lt;/p&gt;&#xA;" OwnerUserId="45570" LastEditorUserId="29575" LastEditDate="2018-02-01T15:59:34.737" LastActivityDate="2018-02-01T20:38:54.110" Title="Clustering: How to cluster multiple CSV files that each represent a Steam user" Tags="|clustering|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27347" PostTypeId="1" AcceptedAnswerId="27409" CreationDate="2018-02-01T15:58:12.597" Score="5" ViewCount="711" Body="&lt;p&gt;I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to train against static data rather than another neural network. &#xA;This lead to me following the guidelines from this website - &lt;a href=&quot;http://outlace.com/rlpart3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://outlace.com/rlpart3.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;however after programming this, the simple version works half the time, this is the version without experience replay. on some runs of the program the game will be played correctly, others it just moves back and forth wehn doing test runs.&#xA;When trying to implement experience replay to complete the harder version. the program will just constantly get itself into a loop of going back and forth when testing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i have a limit of 100 batches in which a batch is what the neural network is trained on. I am wondering whether this is an appropriate amount, or if there could be any common problems with implementing experience replay that i may have made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current perspective of what experience replay is:&#xA;1. run the program&#xA;2. after each turn, the data you used to train the network on, gets saved into a batch &#xA;3. when you have reaches x(100) amount of batches, pick one out and train on it.&#xA;4. overwrite the oldest batch with the new batches that come in. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone could let me know where I have gone wrong, or if there is any feedback about the experience replay or the quality of the question please let me know and i would be very grateful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself,m or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.&lt;/p&gt;&#xA;" OwnerUserId="44530" LastEditorUserId="43077" LastEditDate="2020-03-22T16:31:07.543" LastActivityDate="2023-02-20T17:00:28.053" Title="Q learning - how to use experience replay, when playing against other agent?" Tags="|machine-learning|neural-network|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27348" PostTypeId="1" CreationDate="2018-02-01T16:03:43.370" Score="3" ViewCount="1834" Body="&lt;p&gt;I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've come across &lt;a href=&quot;https://github.com/joewandy/hlda/blob/master/notebooks/bbc_test.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; implementation of Hierarchical LDA, but I'm having a hard time implementing it (no community support). And I don't think gensim's hdpModel is what I want, given &lt;a href=&quot;https://datascience.stackexchange.com/questions/128/latent-dirichlet-allocation-vs-hierarchical-dirichlet-process&quot;&gt;this&lt;/a&gt; discussion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently doing this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) train LDA model on all records to get general topics&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) use this LDA model to assign each record a primary topic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) for each topic, retrieve only the records that were assigned that topic&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) train a new LDA model only on the filtered records (for example, where topic ID == 3) to generate sub topics from filtered set&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Assign each record a general topic ID and a sub topic ID&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a valid way to get topics and sub topics? Should I be weary of this approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any insight.&lt;/p&gt;&#xA;" OwnerUserId="44886" LastActivityDate="2018-09-12T21:00:02.807" Title="Sub topics with Latent Dirichlet Allocation" Tags="|python|nlp|lda|gensim|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27349" PostTypeId="1" AcceptedAnswerId="27376" CreationDate="2018-02-01T17:04:02.030" Score="2" ViewCount="3057" Body="&lt;p&gt;I was surprised to see the results of my feature importance table from my xgboost model. Based on the tutorials that I've seen online, gain/cover/frequency seems to be somewhat similar (as I would expect because if a variable improves accuracy, shouldn't it increase in frequency as well?) but my numbers are drastically different. Am I perhaps doing something wrong or is my intuition wrong? Thank you in advance!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Feature       Gain      Cover   Frequency&#xA;&#xA;1:    Var1 0.21943765 0.02821822 0.009433962&#xA;2:    Var2 0.18207910 0.05509272 0.066037736&#xA;3:    Var3 0.10746529 0.22117710 0.216981132&#xA;4:    Var4 0.10294292 0.05267401 0.018867925&#xA;5:    Var5 0.06928732 0.10185434 0.141509434&#xA;6:    Var6 0.05745753 0.05482397 0.047169811&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45578" LastActivityDate="2018-02-02T10:17:05.773" Title="Xgboost interpretation: shouldn't cover, frequency, and gain be similar?" Tags="|xgboost|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27350" PostTypeId="2" ParentId="27345" CreationDate="2018-02-01T17:11:12.057" Score="2" Body="&lt;p&gt;You're likely going to have to do a little data wrangling to get the data in a better format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm assuming each file has a varying amount of rows, one for each game the user purchased? So, you could create one big matrix with the rows representing users and the columns representing games, and create an indicator matrix for to map purchases to games. Then potentially add additional columns per user including additional insight, like:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the number/percent of games purchased for each category(action, RPG, ...)&lt;/li&gt;&#xA;&lt;li&gt;total purchases&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Having data in that format may lend to more useful application of clustering algorithms.&lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-01T17:11:12.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27351" PostTypeId="2" ParentId="27328" CreationDate="2018-02-01T18:08:01.657" Score="1" Body="&lt;p&gt;You should read &lt;a href=&quot;https://www.pyimagesearch.com/2017/12/04/how-to-create-a-deep-learning-dataset-using-google-images/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt; from the same author. It is about how to build a learning dataset with Google's search, some javascript and some python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea there is to collect as many pictures of the object as possible (object = banana) and then let the deep learning network to compare and learn to differentiate them from bulk random images from some free database.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose background or partial image variations are all ok. You cannot know which part of the object is visible when the whole bunch is evaluated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data needs to be fit to same size, at least Adobe &lt;a href=&quot;https://forums.adobe.com/thread/86003&quot; rel=&quot;nofollow noreferrer&quot;&gt;has some tool&lt;/a&gt; for that. &lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-02-01T18:08:01.657" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27352" PostTypeId="1" CreationDate="2018-02-01T17:45:05.740" Score="2" ViewCount="9713" Body="&lt;p&gt;I want to run some machine learning algorithms such as PCA and KNN with a relatively large dataset of images (&gt;2000 rgb images) in order to classify these images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My source code is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;import numpy as np&#xA;import os&#xA;from glob import glob&#xA;from sklearn.decomposition import PCA&#xA;from sklearn import neighbors&#xA;from sklearn import preprocessing&#xA;&#xA;&#xA;data = []&#xA;&#xA;# Read images from file&#xA;for filename in glob('Images/*.jpg'):&#xA;&#xA;    img = cv2.imread(filename)&#xA;    height, width = img.shape[:2]&#xA;    img = np.array(img)&#xA;&#xA;    # Check that all my images are of the same resolution&#xA;    if height == 529 and width == 940:&#xA;&#xA;        # Reshape each image so that it is stored in one line&#xA;        img = np.concatenate(img, axis=0)&#xA;        img = np.concatenate(img, axis=0)&#xA;        data.append(img)&#xA;&#xA;# Normalise data&#xA;data = np.array(data)&#xA;Norm = preprocessing.Normalizer()&#xA;Norm.fit(data)&#xA;data = Norm.transform(data)&#xA;&#xA;# PCA model&#xA;pca = PCA(0.95)&#xA;pca.fit(data)&#xA;data = pca.transform(data)&#xA;&#xA;# K-Nearest neighbours&#xA;knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)&#xA;distances, indices = knn.kneighbors(data)&#xA;&#xA;print(indices)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, my laptop is not sufficient for this task as it needs many hours in order to process more than 700 rgb images. So I need to use the computational resources of an online platform (e.g. like the ones provided by GCP).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I simply make a call from Pycharm to Compute Engine API (after a I have created a virtual machine in it) to run my python script?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is it possible either to install PyCharm in the virtual machine and run the python script in it or to write my source code in a docker container?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In all, how can I simply run a python script on GCP Compute Engine without wasting time in needless things?&lt;/p&gt;&#xA;" OwnerUserId="45610" OwnerDisplayName="Universalis" LastActivityDate="2018-02-02T09:55:10.770" Title="How to run a python script on GCP Compute Engine?" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27353" PostTypeId="2" ParentId="27352" CreationDate="2018-02-01T17:54:55.467" Score="1" Body="&lt;p&gt;First, you will need to install Cloud SDK: &lt;a href=&quot;https://cloud.google.com/sdk/downloads#apt-get&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cloud.google.com/sdk/downloads#apt-get&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the simplest way is to run your script through your Terminal (mac and I presume the instructions also work on Linux):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Configure your project: gcloud config set project insert_your_project_name&lt;/li&gt;&#xA;&lt;li&gt;Set up SSH keys: gcloud compute config-ssh&lt;/li&gt;&#xA;&lt;li&gt;Connect to the VM: gcloud beta compute ssh vm_name --internal-ip&lt;/li&gt;&#xA;&lt;li&gt;Run script: python your_script.py&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can also connect PyCharm directly to GCP and run everything on your VM but you will need PyCharm Pro, otherwise the deployment option is not available. Let me know if this works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if you want to use the interactive version of setting your project then in step 1 do this instead: gcloud init&lt;/p&gt;&#xA;" OwnerUserId="33688" OwnerDisplayName="francium87d" LastActivityDate="2018-02-01T17:54:55.467" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27354" PostTypeId="1" CreationDate="2018-02-01T19:01:49.837" Score="1" ViewCount="392" Body="&lt;p&gt;I have a problem to train my classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 10 different kinds of music genres, each genre with 100 songs, after making an Mfccs I have a numpy array of &lt;strong&gt;(1293, 20)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all together with np.vstack I have an array of &lt;strong&gt;(1293000, 20)&lt;/strong&gt; and another for the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run &lt;code&gt;model.fit (features, labels)&lt;/code&gt;, it takes a lot of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.manifold import TSNE&#xA;X_embedded = TSNE (n_components = 2).fit_transform(features)&#xA;X_embedded.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've tried to reduce the songs from 1000 to 100 but it's still taking a long time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea how I can classify songs with arrays with so much data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I put some code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = sklearn.preprocessing.StandardScaler()&#xA;y, sr = librosa.load('EXAMPLE1')&#xA;mfcc = librosa.feature.mfcc(y, sr=sr, n_mfcc=20).T&#xA;mfcc_scaled = scaler.fit_transform(mfcc)&#xA;mfcc_scaled.shape # (1293, 20)&#xA;&#xA;y, sr = librosa.load('/Users/josetorronteras/AnacondaProjects/Neural-Networks/genres/pop/pop.00044.au')&#xA;mfcc2 = librosa.feature.mfcc(y, sr=sr, n_mfcc=20).T&#xA;mfcc_scaled2 = scaler.fit_transform(mfcc2)&#xA;mfcc_scaled2.shape # (1293, 20)&#xA;&#xA;tmp_arr = []&#xA;tmp_arr.append(mfcc_scaled)&#xA;tmp_arr.append(mfcc_scaled2)&#xA;mafcc_list = np.vstack(tmp_arr)&#xA;&#xA;mafcc_list.shape # (2586, 20)&#xA;a0 = np.zeros(len(mfcc_scaled))&#xA;a1 = np.ones(len(mfcc_scaled2))&#xA;&#xA;labels = np.concatenate((a0, a1))&#xA;labels.shape # (2586,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerDisplayName="user45581" LastActivityDate="2018-02-01T19:01:49.837" Title="Classifier training long time due to the size data" Tags="|machine-learning|python|neural-network|scikit-learn|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27355" PostTypeId="2" ParentId="27348" CreationDate="2018-02-01T19:21:15.553" Score="3" Body="&lt;p&gt;I actually do not think your method is a &lt;em&gt;good&lt;/em&gt; way to find subtopics.  Consider a document X with a distribution of topics z.  X is made up of a mixed model distribution of topic Z.  If you just give a document the most domiant distribution, and then run lda again, you might find subtopics but you'll also refind the topics that should should perhaps not be considered subtopics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's consider a document that talks about food and exercising and why certain foods and cardo is good for heart health.  Let's assume that we find topics 1 and 2 such that the main topics are food, and exercise.  Suppose that the domiant distribution is topic 1.  Then running LDA on that particular document again, you're going to refine food, exercise and perhaps nutrition, viatims, etc.  However, there's no real reasons what should be considered the topic without inspecting each one and making an inference regarding it.  Also you'll have no real way to discern how deep you should go down this tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://papers.nips.cc/paper/2466-hierarchical-topic-models-and-the-nested-chinese-restaurant-process.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The original paper on hLDA&lt;/a&gt; can prove useful and exploits the chinese restraurant process and its relationship to dirichlet distribution to form the topic relationship and tree depth problem.  Also on David Blei github (bleilab) has an implentation in c++.  You can essentially setup a little shell script after processing the data in a langauge you may be more familar with and use his code.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-02-01T19:21:15.553" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27356" PostTypeId="2" ParentId="24196" CreationDate="2018-02-01T19:23:19.480" Score="3" Body="&lt;ul&gt;&#xA;&lt;li&gt;p2.16xlarge has 16 Tesla K80 GPUs&lt;/li&gt;&#xA;&lt;li&gt;g3.16xlarge has 4 Tesla M60 GPUs&lt;/li&gt;&#xA;&lt;li&gt;p3.16xlarge has 8 Tesla V100 GPUs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I believe the number is a reference to the vCPUs (they all have 64).  Comparing the 1 GPU instance of each type, p3 is the most expensive, then g3, then p2, as expected.&lt;/p&gt;&#xA;" OwnerUserId="45586" LastEditorUserId="29575" LastEditDate="2018-02-01T19:42:31.820" LastActivityDate="2018-02-01T19:42:31.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27357" PostTypeId="2" ParentId="27340" CreationDate="2018-02-01T20:37:21.510" Score="1" Body="&lt;p&gt;There's more than two or three variants with regards to LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A paper that explores these variants can be read here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1503.04069.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM:A Search Space Odyssey&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-02-01T20:37:21.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27358" PostTypeId="2" ParentId="27345" CreationDate="2018-02-01T20:38:54.110" Score="0" Body="&lt;p&gt;No algorithm directly works on the CSV data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even the people that use a single CSV fike will have to &lt;strong&gt;parse it and load&lt;/strong&gt; the data into a suitable in &lt;strong&gt;memory&lt;/strong&gt; representation.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-01T20:38:54.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27359" PostTypeId="1" AcceptedAnswerId="27361" CreationDate="2018-02-01T21:26:28.643" Score="9" ViewCount="8253" Body="&lt;p&gt;I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation to balance the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;code&gt;keras&lt;/code&gt;. I have looked into many examples but I have not found a solution to my problem. I am thinking about replacing the parameters of the &lt;code&gt;model.fit&lt;/code&gt; call by passing a function or generator doing the 'batching'. This function I will design to import from disk and apply augmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea how to implement that. Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="39830" LastEditorUserId="45264" LastEditDate="2019-04-26T12:42:29.187" LastActivityDate="2019-04-26T12:42:29.187" Title="How to train data by batch from disk?" Tags="|python|keras|training|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27360" PostTypeId="2" ParentId="27343" CreationDate="2018-02-01T22:53:37.347" Score="3" Body="&lt;p&gt;There is a way to do this. &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;See the documentation&lt;/a&gt;! It is very good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you mean that you build a network using Keras (which contains recurrent GRU layers) and would like to save the model after some training, then restart from the same point e.g. with new data or just to push the model further.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Assuming you want to analyse some images of shape (150, 150, 3):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Model, load_model&#xA;from keras.layers import Input, GRU, Dense&#xA;&#xA;my_model = Model()&#xA;&#xA;my_input = Input(shape=(150, 150, 3))    # using Tensorflow's dims: 'channels last'&#xA;gru_layer - GRU()(input)&#xA;output = Dense(32)(gru_layer)&#xA;&#xA;# tell the Model class which layers are the start and end of the model&#xA;model = Model(inputs=my_input, outputs=my_output)&#xA;&#xA;# save it!&#xA;my_model.save('my_model.h5')&#xA;&#xA;# some time later.... a new python session&#xA;my_reloaded_model = load_model('my_model.h5')&#xA;&#xA;# continue using the model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Explanation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;At the point the model is built, we can compile and train it the usual way e.g. &lt;code&gt;my_model.fit(...)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once training is finished, we can save the model two different ways:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;just the model weights, using &lt;code&gt;my_model.save_weights('my_weights.h5')&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;the whole model and its meta data, using &lt;code&gt;my_model.save('my_model.h5')&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In either case, you will need the library &lt;code&gt;h5py&lt;/code&gt; (see the documentation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Option 2 preserves more information from the model. Quoting the documentation, it saves:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;the architecture of the model, allowing to re-create the model&lt;/li&gt;&#xA;  &lt;li&gt;the weights of the model&lt;/li&gt;&#xA;  &lt;li&gt;the training configuration (loss, optimizer)&lt;/li&gt;&#xA;  &lt;li&gt;the state of the optimizer, allowing to resume training exactly where you left off.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To continue on the with model where you ended and saved, it is as simple as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_model = keras.models.load_model('my_models.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can train it further or introspect the model and so on.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Links&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Details on the &lt;a href=&quot;https://keras.io/models/model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Model&lt;/code&gt; class&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Details of &lt;a href=&quot;https://keras.io/layers/recurrent/#gru&quot; rel=&quot;nofollow noreferrer&quot;&gt;GRU&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Details of &lt;a href=&quot;https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;saving models&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-02-01T22:53:37.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27361" PostTypeId="2" ParentId="27359" CreationDate="2018-02-01T23:11:56.123" Score="9" Body="&lt;p&gt;As you are working on image classification and would also like to implement some data augmentation, you can combine the two AND load the batches directly from a folder using the mighty 'ImageDataGenerator` class.&lt;/p&gt;&#xA;&lt;p&gt;Have a look at &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;noreferrer&quot;&gt;the execellent documentation!&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I won't copy and paste the example from that link, but I can outline the steps that you go through:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;create the generator class:   &lt;code&gt;data_gen = ImageDataGenerator()&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you want it to perform on-the-fly augmentation for you, that can be    specified when creating the class: &lt;code&gt;data_gen = ImageDataGenerator(samplewise_center=True, ...)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you use augmentation processes that require some statistics about the dataset, e.g. &lt;strong&gt;feature_wise&lt;/strong&gt; normalisation (not sample-wise), you must prepare the generator by showing it some of your data: &lt;code&gt;data_gen.fit(training_data)&lt;/code&gt;. This &lt;code&gt;fit&lt;/code&gt; method simply precomputes things like the mean and standard deviation, which is later used for normalisation.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the generator goes into the model's &lt;code&gt;fit_generator&lt;/code&gt; method, and we call the &lt;code&gt;flow_from_directory&lt;/code&gt; method of the generator:&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;model.fit_generator(training_data=data_gen.flow_from_directory('/path/to/folder/'), ...)&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;You can also create a separate generator using ImageDataGenerator for your validation data, where you should then not apply the augmentation, so that validation tests are done on real data, giving you and accurate picture of the model's performance.&lt;/p&gt;&#xA;&lt;p&gt;In any case, these generators will theoretically run forever, generating batches from your folder. Therefore I recommend using a callback function from Keras to stop when a certain criteria is met. See the &lt;a href=&quot;https://keras.io/callbacks/#earlystopping&quot; rel=&quot;noreferrer&quot;&gt;documentation for the EarlyStopping class&lt;/a&gt;. You could also do this manually, but Keras makes it very simple!&lt;/p&gt;&#xA;&lt;p&gt;If you want fine-grained control, you could do all of the above manually, loading enough samples from disk for a single batch, performing some augmentation and then running the &lt;code&gt;model.train_on_batch()&lt;/code&gt; method. If you want to get into the details, you might be best first learning the Keras way, then progress to your own detailed models combing Tensorflow as required. The two can be used toegether very nicely!&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-06T07:41:25.200" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27362" PostTypeId="1" CreationDate="2018-02-01T23:32:11.823" Score="1" ViewCount="69" Body="&lt;p&gt;At work, I am in a role that does not have a lot of computer permissions. I do have the Microsoft office suite. I have skills with programming and would like to be able to use my experience to do more streamlined and substantial manipulation of large relational data sets. Considering I can't get python or even Bash, is there any good way to do this?&lt;/p&gt;&#xA;" OwnerUserId="45591" LastActivityDate="2018-03-04T07:34:34.473" Title="I only have microsoft office, how can i write code to manipulate relational data sets?" Tags="|python|dataset|data-cleaning|data|programming|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27363" PostTypeId="2" ParentId="27349" CreationDate="2018-02-02T01:39:19.697" Score="5" Body="&lt;p&gt;Gain = Total gains of splits which use the feature. (In my opinion, features with high gain are usually the most important features)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Frequency = Numbers of times the feature is used in a model.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience, these values are not usually correlated all of the time. I have had situations where a feature has the most gain but it was barely checked so there wasn't alot of 'frequency'.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, binary coded variables don't usually have high frequency because there is only 2 possible values. When it comes continuous variables, the model usually is checking for certain ranges so it needs to look at this feature multiple times usually resulting in high frequency.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For future reference, I usually just check the top 20 features by gain, and top 20 by frequency. If a feature appears in both then it is important in my opinion. Also, I wouldn't really worry about 'cover'. I don't think there is much to learn from that.&lt;/p&gt;&#xA;" OwnerUserId="45182" LastActivityDate="2018-02-02T01:39:19.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27365" PostTypeId="2" ParentId="26001" CreationDate="2018-02-02T02:12:08.347" Score="2" Body="&lt;p&gt;there are two ways to parallel your model in multi-gpus. one is &lt;strong&gt;data parallel&lt;/strong&gt;, it replicates whole model in each gpu, and you should input different data into different gpus; the cost you have to pay is that you have to merge gradient into one before updating the weights, so this method is more fit for convolution.&#xA;another is &lt;strong&gt;model parallel&lt;/strong&gt;, you split model into multi-parts, and each gpu is assigned to execute different parts; in this way, you have to pass parts output to another parts, so fully connected layer suits better.&lt;/p&gt;&#xA;" OwnerUserId="45600" LastActivityDate="2018-02-02T02:12:08.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27366" PostTypeId="1" AcceptedAnswerId="27785" CreationDate="2018-02-02T06:19:12.693" Score="5" ViewCount="403" Body="&lt;p&gt;Self Organizing Map is credited to be a very effective tool for exploratory data analysis as it aids in visualization of high-dimensional data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the part where the dimensionality reduction is happening is not clear to me. From &lt;a href=&quot;http://www.ai-junkie.com/ann/som/som1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;, I understand that the information is essentially embedded in the weight vectors of neurons which are of the same dimensions as of the original patterns. The same weight vectors are supposed to be used for visualization (if I follow from the highly popular colors example). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is true, I do not understand how SOM is performing dimensionality reduction and how it is making visualization of high-dimensional data simpler. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone explain (if possible with an example of &gt;3 dimensional data) how SOM performs dimension reduction and makes visualization possible?&lt;/p&gt;&#xA;" OwnerUserId="45603" LastEditorUserId="45603" LastEditDate="2018-02-14T06:48:19.017" LastActivityDate="2020-07-04T10:40:03.990" Title="How do SOMs make visualization of high-dimension data possible?" Tags="|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27367" PostTypeId="2" ParentId="27362" CreationDate="2018-02-02T06:19:20.983" Score="1" Body="&lt;p&gt;It should be possible to install Python without admin privileges. This has been discussed on Stackoverflow &lt;a href=&quot;https://stackoverflow.com/questions/33876657/how-to-install-python-any-version-in-windows-when-youve-no-admin-priviledges&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest method appears to be installing &lt;a href=&quot;https://www.anaconda.com/download/#linux&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anaconda&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And according to &lt;a href=&quot;https://stackoverflow.com/questions/39301675/how-can-i-run-windows-bash-without-being-root&quot;&gt;this answer&lt;/a&gt; you should also be able to use WSL without admin rights, which will get you bash.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For handling relational data you can use &lt;a href=&quot;https://www.sqlite.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sqlite&lt;/a&gt;, which is built-in to Python. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sqlite3&#xA;&#xA;conn = sqlite3.connect('example.db')&#xA;cur = conn.cursor()&#xA;cur.execute('''CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT''')&#xA;conn.commit()&#xA;conn.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-02T06:38:41.297" LastActivityDate="2018-02-02T06:38:41.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27369" PostTypeId="1" CreationDate="2018-02-02T09:03:04.940" Score="1" ViewCount="81" Body="&lt;p&gt;I am new to the machine learning field, but I wanted to try and implement a simple texture classification algorithm with Keras (to give a clear idea about what I mean as 'texture', [this][1] is the most basic texture database that can be found). However, I have a very small set of data, so I thought to try to apply transfer learning to the problem; however, I couldn't find anything on this online, so I wanted to understand which are the best places to look for a pre-trained neural network. Do you have any suggestion in this regard? Which website is best for getting an idea on how to start a machine learning project?&lt;/p&gt;&#xA;" OwnerUserId="45609" LastEditorUserId="45609" LastEditDate="2018-02-06T07:33:38.730" LastActivityDate="2018-02-06T07:33:38.730" Title="Where to find pre-trained models for transfer learning" Tags="|machine-learning|neural-network|classification|keras|transfer-learning|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-04T15:24:23.833" ContentLicense="CC BY-SA 3.0" />
  <row Id="27370" PostTypeId="1" AcceptedAnswerId="27373" CreationDate="2018-02-02T09:12:42.003" Score="5" ViewCount="15124" Body="&lt;p&gt;I'm new to CNNs, starting off with keras. I'm currently using ImageDataGenerator to import my &lt;strong&gt;train/validation folders&lt;/strong&gt; (which each have 2 class subfolders for my binary classification task). Was wondering how can I import my train/validation files without using ImageDataGenerator? I'm aware that ImageDataGenerator is good for accuracy as it does some augmentation, but I want to compare the accuracy to a training set without any augmentations. Essentially I think I need to put all the images into an array, but not sure how to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically I want to know what is the normal way to import training/validation data for images, so I can compare what is the accuracy difference with/without imagedatagen. I know with normal NN tasks it's easy as you can just do pd.read_csv().&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently importing like so: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_datagen = ImageDataGenerator(&#xA;        rescale=1./255,&#xA;        shear_range=0.2,&#xA;        zoom_range=0.2,&#xA;        horizontal_flip=True)&#xA;&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;&#xA;train_generator = train_datagen.flow_from_directory(&#xA;        'data/train',&#xA;        target_size=(150, 150),&#xA;        batch_size=32,&#xA;        class_mode='binary')&#xA;&#xA;validation_generator = test_datagen.flow_from_directory(&#xA;        'data/validation',&#xA;        target_size=(150, 150),&#xA;        batch_size=32,&#xA;        class_mode='binary')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35386" LastActivityDate="2018-02-02T09:55:25.910" Title="How to import image data into python for keras?" Tags="|machine-learning|keras|convolution|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27371" PostTypeId="2" ParentId="24403" CreationDate="2018-02-02T09:14:10.417" Score="1" Body="&lt;p&gt;Yes, you can. The only thing you need to change is the loss function. Implement the loss function used in quantile regression and integrate it. Also, you want to take a look at how you evaluate those intervals. For that, I would use ICP, MIL and RMIL metrics. &lt;/p&gt;&#xA;" OwnerUserId="45613" LastActivityDate="2018-02-02T09:14:10.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27372" PostTypeId="2" ParentId="23132" CreationDate="2018-02-02T09:26:37.983" Score="0" Body="&lt;p&gt;You can check in the documentation which parameters apply to which algorithm. Depending on the algorithm you can optimize different ones, such as c1 and c2 but also epsilon or gamma.&#xA;&lt;a href=&quot;https://sklearn-crfsuite.readthedocs.io/en/latest/api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://sklearn-crfsuite.readthedocs.io/en/latest/api.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45614" LastActivityDate="2018-02-02T09:26:37.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27373" PostTypeId="2" ParentId="27370" CreationDate="2018-02-02T09:30:00.150" Score="6" Body="&lt;p&gt;The &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;noreferrer&quot;&gt;docs&lt;/a&gt; for ImageDataGenerator suggest that no augmentation is done by default. So you could instantiate it without any augmentation parameters and keep the rest of your code for handling your directory structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_datagen = ImageDataGenerator(rescale=1./255)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You are also allowed to write your own custom data generator and pass it to &lt;code&gt;model.fit_generator()&lt;/code&gt;. &lt;a href=&quot;https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; is a nice tutorial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or if your data fits in memory you could write some simpler code possibly using &lt;code&gt;keras.preprocessing.image.load_img&lt;/code&gt; to load all the images into an array and pass them to &lt;code&gt;model.fit&lt;/code&gt; instead.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-02T09:55:25.910" LastActivityDate="2018-02-02T09:55:25.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27374" PostTypeId="2" ParentId="27352" CreationDate="2018-02-02T09:55:10.770" Score="0" Body="&lt;p&gt;The other option is to setup jupyter notebook on GCP. You can use the following command to run jupyter notebook in background.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nohup jupyter notebpook --ip=0.0.0.0 &amp;amp;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can do tunneling by doing an ssh into GCP:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ssh username@&amp;lt;public_ip&amp;gt; -L 8888:127.0.0.1:8888&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you should be able to access jupyter notebook from your local machine with the following url in the browser&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;127.0.0.1:8888&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14364" LastActivityDate="2018-02-02T09:55:10.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27376" PostTypeId="2" ParentId="27349" CreationDate="2018-02-02T10:17:05.773" Score="3" Body="&lt;p&gt;My layman's understanding of those metrics as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Gain = (some measure of) improvement in overall model accuracy by using the feature&lt;/li&gt;&#xA;&lt;li&gt;Frequency = how often the feature is used in the model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It's important to remember that the algorithm builds sequentially, so the two metrics are not always directly comparable / correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example (2 scenarios):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Var1 is relatively predictive of the response. It is included by the algorithm and its &quot;Gain&quot; is relatively high. Once its link to the response has been captured it might not be used again - e.g. there may be other features which are more predictive &lt;strong&gt;at later stages of modelling&lt;/strong&gt; or all of Var1's link to the response may have been captured - and so its &quot;Frequency&quot; is low. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Var1 is extremely predictive across the whole range of response values. We can expect that Var1 will have high &quot;Gain&quot;. Now, since Var1 is so predictive it might be fitted repeatedly (each time using a different split) and so will also have a high &quot;Frequency&quot;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In most cases, we prioritise accuracy and so will likely prioritise &quot;Gain&quot; over &quot;Frequency&quot;, but if you're using the algorithm for feature selection then it may be a good idea to use a mixture of both to inform your decision, much like @bbennett36 suggested.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-02T10:17:05.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27377" PostTypeId="1" CreationDate="2018-02-02T10:42:46.060" Score="3" ViewCount="44" Body="&lt;p&gt;Is there a CONSTANT limit in &lt;a href=&quot;https://en.wikipedia.org/wiki/Differentiable_neural_computer&quot; rel=&quot;nofollow noreferrer&quot;&gt;DNC&lt;/a&gt; external memory usage?  Like a human brain, 7 chunks of information? Or is it some sort of a hyperparameter? How much data it can associate using that Short-Term memory?&lt;/p&gt;&#xA;" OwnerUserId="45617" LastEditorUserId="47488" LastEditDate="2018-04-30T02:18:57.060" LastActivityDate="2018-04-30T02:18:57.060" Title="Is the external memory of Differtiable Neural Computer limited?" Tags="|deep-learning|algorithms|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27378" PostTypeId="1" AcceptedAnswerId="27383" CreationDate="2018-02-02T11:11:55.863" Score="1" ViewCount="222" Body="&lt;p&gt;I want to clarify I have understood how SARSA works in nuances. Consider an original definition taken from ON-LINE Q-LEARNING USING CONNECTIONIST SYSTEMS. G. A. Rummery &amp;amp; M. Niranjan. CUED/F-INFENG/TR 166. September 1994 (which is the first publication where SARSA wss mentioned, according to a Wikipedia article).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors proposed an update rule which &quot;... differs from normal Q-learning in the use of the Qt+1 associated with the action selected, rather than the greedy max(Qt+1 | a) used in Q-learning.&quot; (A citation from page 6.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that this is essential to on-policy nature of the algorithm. The term SARSA mentioned in the footnote for this definition.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And a later pseudo-code of SARSA update used in many readings:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;begin&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  initialize Q[S,A] arbitrarily&#xA;  observe current state s&#xA;  select action using a policy based on Q&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;repeat forever:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;carry out an action a&#xA;observe reward r and state s'&#xA;select action a' using a poicy based on Q&#xA;Q[s,a] &amp;lt;- Q[s,a] + alpha * (r + gamma * Q[s', a'] - Q[s, a])&#xA;s &amp;lt;- s'&#xA;a &amp;lt;- a'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;end-repeat&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;end&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;source: &lt;a href=&quot;http://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to understand if I must use exactly the same Q function (and policy) to get A and A'. If I update Q function in each iteration, it follows that the next action in a subsequent iteration will be derived using the latest Q updated, while the previous action was obtained using the former Q.\&#xA;On the other hand, I really can make A and A' with exactly the same Q, and only after that update the Q. So I will always consider A and A' derived using the same function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which is more orthodox / correct?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="42953" LastEditorUserId="42953" LastEditDate="2018-02-02T12:16:58.450" LastActivityDate="2018-02-02T13:03:48.037" Title="Clarifying my understanding of on-policy RL (online SARSA)" Tags="|reinforcement-learning|online-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27379" PostTypeId="2" ParentId="18840" CreationDate="2018-02-02T11:55:11.087" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;4&quot;&gt;&#xA;  &lt;li&gt;Every decision boundary that can be found by linear SVM can be found by a perceptron.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Concerning 4: In the case where your classes are linearly separable:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are usually infinitely many decision boundaries that achieve perfect classification. The Perceptron is guaranteed to find one of them, but it is virtually impossible to predict which one. (That will depend on the initial weights and the learning rate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SVM (without slack variables) is also guaranteed to find a perfect decision boundary; but not an arbitrary one. It finds one of the boundaries with maximal distance to the closest observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can think of the SVM (without slack variables) picking the 'best' decision boundary among those that the Perceptron could find.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-02T14:20:27.727" LastActivityDate="2018-02-02T14:20:27.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27380" PostTypeId="1" AcceptedAnswerId="27384" CreationDate="2018-02-02T12:14:36.783" Score="0" ViewCount="507" Body="&lt;p&gt;Suppose I use a linear Support Vector Machine &lt;em&gt;with&lt;/em&gt; slack variables on a dataset that is linearly separable. &#xA;Could it happen that the Support Vector Machine reports a solution that does not perfectly separate the classes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an illustration: Is the situation in the picture possible for a Support Vector Machine with slack variables? Although there is a &quot;better&quot; boundary that allows perfect classification, the Support Vector Machine goes for a sub-optimal solution that misclassifies two samples.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mq2Sq.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mq2Sq.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-02T13:19:40.210" Title="Linear SVM with slack variables: Will it find a perfect decision boundary if it exists?" Tags="|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27382" PostTypeId="2" ParentId="26838" CreationDate="2018-02-02T12:26:27.003" Score="1" Body="&lt;p&gt;Hopefully this article &lt;a href=&quot;http://paulbourke.net/miscellaneous/imagefilter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image Filtering in the Frequency Domain&lt;/a&gt; can help you understand the concept of convolution, image and frequency filtering&lt;/p&gt;&#xA;" OwnerUserId="45545" LastActivityDate="2018-02-02T12:26:27.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27383" PostTypeId="2" ParentId="27378" CreationDate="2018-02-02T12:48:53.130" Score="1" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;I want to understand if I must use exactly the same Q function (and policy) to get A and A'. If I update Q function in each iteration, it follows that the next action in a subsequent iteration will be derived using the latest Q updated, while the previous action was obtained using the former Q.\ On the other hand, I really can make A and A' with exactly the same Q, and only after that update the Q. So I will always consider A and A' derived using the same function.&lt;/p&gt;&#xA;&lt;p&gt;Which is more orthodox / correct?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The algorithm pseudocode given is more orthodox, since in order to revise the value of &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; you would have to &amp;quot;roll back&amp;quot; the environment and see where the newly-sampled &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; would take you from state &lt;span class=&quot;math-container&quot;&gt;$S$&lt;/span&gt;. To make this clearer, you can see that:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;select action a' using a policy based on Q&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;could be re-phrased:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;select action a' by sampling epsilon-greedy function over Q(s',*)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;. . . you cannot do that unless you have the value of &lt;span class=&quot;math-container&quot;&gt;$S'$&lt;/span&gt;, and you may only have that value if you have already taken action &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; when in state &lt;span class=&quot;math-container&quot;&gt;$S$&lt;/span&gt;. Changing &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt; at that stage therefore means going back in time . . .&lt;/p&gt;&#xA;&lt;p&gt;In practice it doesn't matter much, even if you have capability to roll back (in a simulator, or in a planning algorithm). If your policy is based on e.g. &lt;span class=&quot;math-container&quot;&gt;$\epsilon$&lt;/span&gt;-greedy over the current Q values, then you are performing SARSA for optimal control (as opposed to prediction). In that case, changing Q means changing the policy. &amp;quot;On-policy&amp;quot; in SARSA for control must allow for the non-stationarity of the policy. Occasionally that means that the &lt;span class=&quot;math-container&quot;&gt;$A'$&lt;/span&gt; value you just chose would have been chosen with a lower probability in a more optimal policy. But you chose it anyway this time, and the agent should choose it less often &lt;em&gt;in future&lt;/em&gt;. The learning-rate based updates will remove estimation bias due to earlier poor/too-frequently-sampled choices over time.&lt;/p&gt;&#xA;&lt;p&gt;Revising a single step &amp;quot;mistake&amp;quot; is possible, but not common practice in a purely online algorithm. I have not seen it in planning look-ahead or offline algorithms either that I have studied. I don't know for certain, but I suspect the occasional boost to learning you might get from revising the immediate part of the trajectory is too small to be worth the loss of generality of the algorithm. You may find it does help sometimes though, and worth an experiment to review whether it is helpful, provided you are working with a simulator/planner where rolling back state is relatively easy.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Note that the way you are thinking does turn up again when using function approximators (e.g. linear functions or neural networks) in semi-gradient versus &amp;quot;true gradient&amp;quot; methods, where instead of this being an issue with which Q values to use, it is an issue with calculating gradients due to the TD error, when your TD target is based on the same parameters that you are are taking the gradient for. In semi-gradient methods, this issue is ignored, and the methods still work OK. However, the &amp;quot;true gradient&amp;quot; methods are more theoretically correct.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-02T13:03:48.037" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27384" PostTypeId="2" ParentId="27380" CreationDate="2018-02-02T13:19:40.210" Score="2" Body="&lt;p&gt;Having linear separable data means that an optimal solution exists. The support vector machine is an approach to identify the optimal solution with multiple steps through Lagrange multiplication. This method is guaranteed to give you the optimal solution if there is one. &lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-02-02T13:19:40.210" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27385" PostTypeId="2" ParentId="27135" CreationDate="2018-02-02T13:36:25.153" Score="1" Body="&lt;p&gt;At coursera, the homework of third week of convolutional networks by professor Andrew Ng, is about this. I recommend you to see that homework. It also implements the &lt;code&gt;YOLO&lt;/code&gt; paper. I can't add the code here, but the architecture of the network is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;____________________________________________________________________________________________________&#xA;Layer (type)                     Output Shape          Param #     Connected to                     &#xA;====================================================================================================&#xA;input_1 (InputLayer)             (None, 608, 608, 3)   0                                            &#xA;____________________________________________________________________________________________________&#xA;conv2d_1 (Conv2D)                (None, 608, 608, 32)  864         input_1[0][0]                    &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_1 (BatchNorm (None, 608, 608, 32)  128         conv2d_1[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_1 (LeakyReLU)        (None, 608, 608, 32)  0           batch_normalization_1[0][0]      &#xA;____________________________________________________________________________________________________&#xA;max_pooling2d_1 (MaxPooling2D)   (None, 304, 304, 32)  0           leaky_re_lu_1[0][0]              &#xA;____________________________________________________________________________________________________&#xA;conv2d_2 (Conv2D)                (None, 304, 304, 64)  18432       max_pooling2d_1[0][0]            &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_2 (BatchNorm (None, 304, 304, 64)  256         conv2d_2[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_2 (LeakyReLU)        (None, 304, 304, 64)  0           batch_normalization_2[0][0]      &#xA;____________________________________________________________________________________________________&#xA;max_pooling2d_2 (MaxPooling2D)   (None, 152, 152, 64)  0           leaky_re_lu_2[0][0]              &#xA;____________________________________________________________________________________________________&#xA;conv2d_3 (Conv2D)                (None, 152, 152, 128) 73728       max_pooling2d_2[0][0]            &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_3 (BatchNorm (None, 152, 152, 128) 512         conv2d_3[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_3 (LeakyReLU)        (None, 152, 152, 128) 0           batch_normalization_3[0][0]      &#xA;____________________________________________________________________________________________________&#xA;conv2d_4 (Conv2D)                (None, 152, 152, 64)  8192        leaky_re_lu_3[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_4 (BatchNorm (None, 152, 152, 64)  256         conv2d_4[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_4 (LeakyReLU)        (None, 152, 152, 64)  0           batch_normalization_4[0][0]      &#xA;____________________________________________________________________________________________________&#xA;conv2d_5 (Conv2D)                (None, 152, 152, 128) 73728       leaky_re_lu_4[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_5 (BatchNorm (None, 152, 152, 128) 512         conv2d_5[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_5 (LeakyReLU)        (None, 152, 152, 128) 0           batch_normalization_5[0][0]      &#xA;____________________________________________________________________________________________________&#xA;max_pooling2d_3 (MaxPooling2D)   (None, 76, 76, 128)   0           leaky_re_lu_5[0][0]              &#xA;____________________________________________________________________________________________________&#xA;conv2d_6 (Conv2D)                (None, 76, 76, 256)   294912      max_pooling2d_3[0][0]            &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_6 (BatchNorm (None, 76, 76, 256)   1024        conv2d_6[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_6 (LeakyReLU)        (None, 76, 76, 256)   0           batch_normalization_6[0][0]      &#xA;____________________________________________________________________________________________________&#xA;conv2d_7 (Conv2D)                (None, 76, 76, 128)   32768       leaky_re_lu_6[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_7 (BatchNorm (None, 76, 76, 128)   512         conv2d_7[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_7 (LeakyReLU)        (None, 76, 76, 128)   0           batch_normalization_7[0][0]      &#xA;____________________________________________________________________________________________________&#xA;conv2d_8 (Conv2D)                (None, 76, 76, 256)   294912      leaky_re_lu_7[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_8 (BatchNorm (None, 76, 76, 256)   1024        conv2d_8[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_8 (LeakyReLU)        (None, 76, 76, 256)   0           batch_normalization_8[0][0]      &#xA;____________________________________________________________________________________________________&#xA;max_pooling2d_4 (MaxPooling2D)   (None, 38, 38, 256)   0           leaky_re_lu_8[0][0]              &#xA;____________________________________________________________________________________________________&#xA;conv2d_9 (Conv2D)                (None, 38, 38, 512)   1179648     max_pooling2d_4[0][0]            &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_9 (BatchNorm (None, 38, 38, 512)   2048        conv2d_9[0][0]                   &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_9 (LeakyReLU)        (None, 38, 38, 512)   0           batch_normalization_9[0][0]      &#xA;____________________________________________________________________________________________________&#xA;conv2d_10 (Conv2D)               (None, 38, 38, 256)   131072      leaky_re_lu_9[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_10 (BatchNor (None, 38, 38, 256)   1024        conv2d_10[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_10 (LeakyReLU)       (None, 38, 38, 256)   0           batch_normalization_10[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_11 (Conv2D)               (None, 38, 38, 512)   1179648     leaky_re_lu_10[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_11 (BatchNor (None, 38, 38, 512)   2048        conv2d_11[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_11 (LeakyReLU)       (None, 38, 38, 512)   0           batch_normalization_11[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_12 (Conv2D)               (None, 38, 38, 256)   131072      leaky_re_lu_11[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_12 (BatchNor (None, 38, 38, 256)   1024        conv2d_12[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_12 (LeakyReLU)       (None, 38, 38, 256)   0           batch_normalization_12[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_13 (Conv2D)               (None, 38, 38, 512)   1179648     leaky_re_lu_12[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_13 (BatchNor (None, 38, 38, 512)   2048        conv2d_13[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_13 (LeakyReLU)       (None, 38, 38, 512)   0           batch_normalization_13[0][0]     &#xA;____________________________________________________________________________________________________&#xA;max_pooling2d_5 (MaxPooling2D)   (None, 19, 19, 512)   0           leaky_re_lu_13[0][0]             &#xA;____________________________________________________________________________________________________&#xA;conv2d_14 (Conv2D)               (None, 19, 19, 1024)  4718592     max_pooling2d_5[0][0]            &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_14 (BatchNor (None, 19, 19, 1024)  4096        conv2d_14[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_14 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_14[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_15 (Conv2D)               (None, 19, 19, 512)   524288      leaky_re_lu_14[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_15 (BatchNor (None, 19, 19, 512)   2048        conv2d_15[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_15 (LeakyReLU)       (None, 19, 19, 512)   0           batch_normalization_15[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_16 (Conv2D)               (None, 19, 19, 1024)  4718592     leaky_re_lu_15[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_16 (BatchNor (None, 19, 19, 1024)  4096        conv2d_16[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_16 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_16[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_17 (Conv2D)               (None, 19, 19, 512)   524288      leaky_re_lu_16[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_17 (BatchNor (None, 19, 19, 512)   2048        conv2d_17[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_17 (LeakyReLU)       (None, 19, 19, 512)   0           batch_normalization_17[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_18 (Conv2D)               (None, 19, 19, 1024)  4718592     leaky_re_lu_17[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_18 (BatchNor (None, 19, 19, 1024)  4096        conv2d_18[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_18 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_18[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_19 (Conv2D)               (None, 19, 19, 1024)  9437184     leaky_re_lu_18[0][0]             &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_19 (BatchNor (None, 19, 19, 1024)  4096        conv2d_19[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;conv2d_21 (Conv2D)               (None, 38, 38, 64)    32768       leaky_re_lu_13[0][0]             &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_19 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_19[0][0]     &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_21 (BatchNor (None, 38, 38, 64)    256         conv2d_21[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;conv2d_20 (Conv2D)               (None, 19, 19, 1024)  9437184     leaky_re_lu_19[0][0]             &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_21 (LeakyReLU)       (None, 38, 38, 64)    0           batch_normalization_21[0][0]     &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_20 (BatchNor (None, 19, 19, 1024)  4096        conv2d_20[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;space_to_depth_x2 (Lambda)       (None, 19, 19, 256)   0           leaky_re_lu_21[0][0]             &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_20 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_20[0][0]     &#xA;____________________________________________________________________________________________________&#xA;concatenate_1 (Concatenate)      (None, 19, 19, 1280)  0           space_to_depth_x2[0][0]          &#xA;                                                                   leaky_re_lu_20[0][0]             &#xA;____________________________________________________________________________________________________&#xA;conv2d_22 (Conv2D)               (None, 19, 19, 1024)  11796480    concatenate_1[0][0]              &#xA;____________________________________________________________________________________________________&#xA;batch_normalization_22 (BatchNor (None, 19, 19, 1024)  4096        conv2d_22[0][0]                  &#xA;____________________________________________________________________________________________________&#xA;leaky_re_lu_22 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_22[0][0]     &#xA;____________________________________________________________________________________________________&#xA;conv2d_23 (Conv2D)               (None, 19, 19, 425)   435625      leaky_re_lu_22[0][0]             &#xA;====================================================================================================&#xA;Total params: 50,983,561&#xA;Trainable params: 50,962,889&#xA;Non-trainable params: 20,672&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Like you, I can't see any flatten layer.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-02T13:36:25.153" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27386" PostTypeId="1" AcceptedAnswerId="27387" CreationDate="2018-02-02T14:27:06.287" Score="0" ViewCount="1072" Body="&lt;p&gt;I have an implicit dataset. It contains which user click which item. I'm doing collaborative filtering and finally i get the item similarites. So now i have data like;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item - SimilarItem - SimilarityValue&#xA;A    - C           - 0.12&#xA;A    - R           - 0.42&#xA;A    - Y           - 0.34&#xA;A    - J           - 0.62&#xA;B    - A           - 0.16&#xA;B    - Y           - 0.83&#xA;C    - J           - 0.23&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Item  --  Similarities&#xA;A     --  C,R,Y,J&#xA;B     --  A,Y&#xA;C     --  J,A,D&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As i know all the relations between 40K item, can i turn this into a nice visualization showing some of them get together at somewhere and someothers are elsewhere. I can do this either with python or some javascript library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What should be the way of doing this? Can you show me some examples?&lt;/p&gt;&#xA;" OwnerUserId="44190" LastEditorUserId="44190" LastEditDate="2018-02-07T10:54:08.123" LastActivityDate="2018-02-07T10:54:08.123" Title="Visualizing item similarities" Tags="|python|clustering|visualization|recommender-system|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-04T15:23:10.993" ContentLicense="CC BY-SA 3.0" />
  <row Id="27387" PostTypeId="2" ParentId="27386" CreationDate="2018-02-02T14:46:12.310" Score="1" Body="&lt;p&gt;You could treat your results as an undirected graph with weighted edges. Your nodes are A, B, C, etc. and your vertices are the connections between items, weighted by similarity value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://qxf2.com/blog/drawing-weighted-graphs-with-networkx/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a Python function that could be a great starting point. It draws thicker connections for vertices with a larger weight. I copied it and replaced the example data with your data points above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[...]&#xA;node_list = ['A', 'B', 'C', 'D', 'J', 'R', 'Y']&#xA;[...]&#xA;G.add_edge(node_list[0],node_list[2],weight=0.12) # A - C&#xA;G.add_edge(node_list[0],node_list[5],weight=0.42) # A - R&#xA;G.add_edge(node_list[0],node_list[6],weight=0.34) # A - Y&#xA;G.add_edge(node_list[0],node_list[4],weight=0.62) # A - J&#xA;G.add_edge(node_list[0],node_list[1],weight=0.16) # A - B&#xA;G.add_edge(node_list[0],node_list[6],weight=0.83) # B - Y&#xA;G.add_edge(node_list[2],node_list[4],weight=0.21) # C - J&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NsjLu.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NsjLu.jpg&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you have 40,000 items, that would probably require additional work to make the plot comprehensible. You could begin by ignoring all connections between nodes with a similarity value of, say, 0.01 or less.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-02T14:46:12.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27388" PostTypeId="1" AcceptedAnswerId="27390" CreationDate="2018-02-02T14:58:16.577" Score="16" ViewCount="5741" Body="&lt;p&gt;If I have a 50 dimensional hypercube. And I define it's boundary by $0&amp;lt;x_j&amp;lt;0.05$ or $0.95&amp;lt;x_j&amp;lt;1$ where $x_j$ is dimension of the hypercube. Then calculating the proportion of points on the boundary of the hypercube will be $0.995$. What does it mean? Does it mean that rest of the space is empty? If $99\%$ of the points are  at the boundary then the points inside the cube must not be uniformly distributed?&lt;/p&gt;&#xA;" OwnerUserId="41149" LastEditorUserId="45507" LastEditDate="2018-11-13T16:00:15.920" LastActivityDate="2020-02-03T10:39:22.597" Title="What does it mean when we say most of the points in a hypercube are at the boundary?" Tags="|machine-learning|mathematics|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27389" PostTypeId="1" CreationDate="2018-02-02T15:26:23.570" Score="1" ViewCount="129" Body="&lt;p&gt;I need to develop a report that will show automated queries in an audit log of queries on a system of the company. The logs have this fashion:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;query_id id          query_time&#xA;1        1           2018-02-01 00:09:02&#xA;2        1           2018-02-01 00:24:55&#xA;3        1           2018-02-01 00:58:55&#xA;4        1           2018-02-01 01:01:49&#xA;5        1           2018-02-01 01:05:42&#xA;6        1           2018-02-01 01:18:56&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;query_id&lt;/code&gt; is an index of the query, the &lt;code&gt;id&lt;/code&gt; is who has queried and &lt;code&gt;query_time&lt;/code&gt; is the time of the query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to make the difference between the time of the queries and count how much was less than 60 seconds, but don't make the job. One other way is see the periodicity of the queries. I was thinking if there was a better way to do this. Maybe something using time series. There is some technique that could identify a query bot?&lt;/p&gt;&#xA;" OwnerUserId="45627" LastEditorUserId="29575" LastEditDate="2018-02-02T15:35:07.553" LastActivityDate="2018-04-03T18:01:41.330" Title="Detect a bot in a audit log" Tags="|anomaly-detection|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ClosedDate="2018-04-06T01:10:24.840" ContentLicense="CC BY-SA 3.0" />
  <row Id="27390" PostTypeId="2" ParentId="27388" CreationDate="2018-02-02T15:34:35.280" Score="33" Body="&lt;p&gt;Speaking of '&lt;em&gt;&lt;span class=&quot;math-container&quot;&gt;$99\%$&lt;/span&gt; of the points in a hypercube&lt;/em&gt;' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The volume of a hypercube is the product of its side lengths.&#xA;For the 50-dimensional unit hypercube we get &lt;span class=&quot;math-container&quot;&gt;$$\text{Total volume} = \underbrace{1 \times 1 \times \dots \times 1}_{50 \text{ times}} = 1^{50} = 1.$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let us exclude the boundaries of the hypercube and look at the '&lt;em&gt;interior&lt;/em&gt;' (I put this in quotation marks because the mathematical term &lt;a href=&quot;https://en.wikipedia.org/wiki/Interior_(topology)&quot; rel=&quot;noreferrer&quot;&gt;interior&lt;/a&gt; has a very different meaning). We only keep the points &lt;span class=&quot;math-container&quot;&gt;$x = (x_1, x_2, \dots, x_{50})$&lt;/span&gt; that satisfy&#xA;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;0.05 &amp;lt; x_1 &amp;lt; 0.95 \,\text{ and }\, 0.05 &amp;lt; x_2 &amp;lt; 0.95 \,\text{ and }\, \dots &#xA;\,\text{ and }\, 0.05 &amp;lt; x_{50} &amp;lt; 0.95.&#xA;$$&lt;/span&gt;&#xA;What is the volume of this '&lt;em&gt;interior&lt;/em&gt;'? Well, the '&lt;em&gt;interior&lt;/em&gt;' is again a hypercube, and the length of each side is &lt;span class=&quot;math-container&quot;&gt;$0.9$&lt;/span&gt; (&lt;span class=&quot;math-container&quot;&gt;$=0.95 - 0.05$&lt;/span&gt; ... it helps to imagine this in two and three dimensions).&#xA;So the volume is &lt;span class=&quot;math-container&quot;&gt;$$\text{Interior volume} = \underbrace{0.9 \times 0.9 \times \dots \times 0.9}_{50 \text{ times}} = 0.9^{50} \approx 0.005.$$&lt;/span&gt;&#xA;Conclude that the volume of the '&lt;em&gt;boundary&lt;/em&gt;' (defined as the unit hypercube without the '&lt;em&gt;interior&lt;/em&gt;') is &lt;span class=&quot;math-container&quot;&gt;$1 - 0.9^{50} \approx 0.995.$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows that &lt;span class=&quot;math-container&quot;&gt;$99.5\%$&lt;/span&gt; of the volume of a 50-dimensional hypercube is concentrated on its '&lt;em&gt;boundary&lt;/em&gt;'.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Follow-up:&lt;/strong&gt; &lt;em&gt;ignatius&lt;/em&gt; raised an interesting question on how this is connected to probability. Here is an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you came up with a (machine learning) model that predicts housing prices based on 50 input parameters. All 50 input parameters are &lt;a href=&quot;https://en.wikipedia.org/wiki/Independence_(probability_theory)#For_real_valued_random_variables&quot; rel=&quot;noreferrer&quot;&gt;independent&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Uniform_distribution_(continuous)&quot; rel=&quot;noreferrer&quot;&gt;uniformly distributed&lt;/a&gt; between &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us say that your model works very well if none of the input parameters is &lt;em&gt;extreme:&lt;/em&gt; As long as every input parameter stays between &lt;span class=&quot;math-container&quot;&gt;$0.05$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$0.95$&lt;/span&gt;, your model predicts the housing price almost perfectly.&#xA;But if one or more input parameters are extreme (smaller than &lt;span class=&quot;math-container&quot;&gt;$0.05$&lt;/span&gt; or larger than &lt;span class=&quot;math-container&quot;&gt;$0.95$&lt;/span&gt;), the predictions of your model are absolutely terrible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any given input parameter is extreme with a probability of only &lt;span class=&quot;math-container&quot;&gt;$10\%$&lt;/span&gt;. So clearly this is a good model, right?&#xA;No! The probability that &lt;em&gt;at least one of the &lt;span class=&quot;math-container&quot;&gt;$50$&lt;/span&gt; parameters is extreme&lt;/em&gt; is &lt;span class=&quot;math-container&quot;&gt;$1 - 0.9^{50} \approx 0.995.$&lt;/span&gt;&#xA;So in &lt;span class=&quot;math-container&quot;&gt;$99.5\%$&lt;/span&gt; of the cases, your model's prediction is terrible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rule of thumb:&lt;/strong&gt; &lt;em&gt;In high dimensions, extreme observations are the rule and not the exception.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2020-02-03T10:39:22.597" LastActivityDate="2020-02-03T10:39:22.597" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="27391" PostTypeId="2" ParentId="21734" CreationDate="2018-02-02T15:41:17.707" Score="12" Body="&lt;p&gt;You can do this by creating a new VGG16 model instance with the new input shape &lt;code&gt;new_shape&lt;/code&gt; and copying over all the layer weights. The code is roughly&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_model = VGG16(weights=None, input_shape=new_shape, include_top=False)&#xA;for new_layer, layer in zip(new_model.layers[1:], model.layers[1:]):&#xA;    new_layer.set_weights(layer.get_weights())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45631" LastEditorUserId="45631" LastEditDate="2018-02-02T19:31:36.990" LastActivityDate="2018-02-02T19:31:36.990" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27392" PostTypeId="1" CreationDate="2018-02-02T15:45:12.373" Score="13" ViewCount="9174" Body="&lt;p&gt;I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying different models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After working with some results and reading more and more about LSTM, it seems like this approach is far superior to anything else I've tried (across multiple datasets). I keep thinking to myself, &quot;why/when would you &lt;em&gt;not&lt;/em&gt; use LSTM?&quot;. The use of the additional gates, inherent to LSTM, makes perfect sense to me after having some models that suffer from vanishing gradients. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what's the catch with LSTM? Where do they not do so well? I know there is no such thing as a &quot;one size fits all&quot; algorithm, so there must be a downside to LSTM.&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2022-12-19T19:38:36.113" Title="So what's the catch with LSTM?" Tags="|neural-network|nlp|lstm|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27393" PostTypeId="1" CreationDate="2018-02-02T15:59:52.620" Score="1" ViewCount="169" Body="&lt;p&gt;I have written a (convenience) class for binary classification in python which takes several models and compares their ROCs, AvPrecisions, and creates several charts (distribution of scores, accuracy/sensitivity/specificity, ROC, PR, etc.).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A key difference that I am observing is in the computation of average precision.  For some examples, using &lt;code&gt;average_precision_score(true labels, prob score)&lt;/code&gt; I get a value which is 0.1% to 0.2% more than the average precision computed as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xaxs,yaxs = df['sens'].values[::-1],df['ppv'].values[::-1]&#xA;ap = yaxs[:-1]*(xaxs[1:]-xaxs[:-1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or equivalently&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xaxs,yaxs = df['sens'].values,df['ppv'].values&#xA;tmp = yaxs[1:]*(xaxs[:-1]-xaxs[1:])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I do the same for AUC and that matches to second place of decimal.  Based on the formula given &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; the above calculation seems to be correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the figure shown, the values computed by &lt;code&gt;scikit-learn&lt;/code&gt; are 0.8977 and 0.8722.  Whereas the values compute by the numpy code above are 0.895 and 0.8697.  I am using 10,001 thresholds between 0 and 1 so I would think that the bumps in the left corner of the chart should be adequately captured.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would welcome input on how to change my calculation to match the one from skl. Thx.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ajXxv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ajXxv.png&quot; alt=&quot;precision recall curve&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32199" LastEditorUserId="1330" LastEditDate="2021-07-22T16:04:31.187" LastActivityDate="2021-07-22T16:04:31.187" Title="matching results with sklearn average_precision_score" Tags="|python|classification|scikit-learn|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27394" PostTypeId="2" ParentId="27241" CreationDate="2018-02-02T16:50:54.743" Score="4" Body="&lt;p&gt;Well, I would not call this &quot;Test&quot; set you defined as &quot;Test&quot;! I would say: Train, Validation, Test. And this &quot;Test&quot; is the dataset that your model has not seen yet. This is pretty standard even explained in details in Andrew Ng courses like this very useful and &lt;a href=&quot;https://www.youtube.com/watch?v=M3qpIzy4MQk&quot; rel=&quot;nofollow noreferrer&quot;&gt;important lesson&lt;/a&gt; on splitting data and the significance of their distributions.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-02-03T08:40:13.223" LastActivityDate="2018-02-03T08:40:13.223" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27396" PostTypeId="2" ParentId="27389" CreationDate="2018-02-02T17:30:12.747" Score="0" Body="&lt;p&gt;It sounds like you have good &lt;em&gt;initial&lt;/em&gt; data but you might need to develop a meta data set. I would start by taking all this info and building behavior profiles. How often do they query per session? When do they query? How many sessions per hour/day/week/month? And so on, I think you get the idea. It just doesn't strike me like the raw file is the way to go, you need a little bit more to work with but you can get there with what you have now. Get down to various behaviors, one ID per row, and you'll be in a much better position to model this out.   &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-02T17:30:12.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27399" PostTypeId="2" ParentId="27388" CreationDate="2018-02-02T17:49:25.853" Score="10" Body="&lt;p&gt;You can see the pattern clearly even in lower dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1st dimension. Take a line of length 10 and a boundary of 1. The length of the boundary is 2 and the interior 8, 1:4 ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2nd dimension. Take a square of side 10, and boundary 1 again. The area of the boundary is 36, the interior 64, 9:16 ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3rd dimension. Same length and boundary. The volume of the boundary is 488, the interior is 512, 61:64 - already the boundary occupies almost as much space as the interior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4th dimension, now the boundary is 5904 and the interior 4096 - the boundary is now larger.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even for smaller and smaller boundary lengths, as the dimension increases the boundary volume will always overtake the interior.&lt;/p&gt;&#xA;" OwnerUserId="45635" LastActivityDate="2018-02-02T17:49:25.853" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27401" PostTypeId="2" ParentId="27392" CreationDate="2018-02-02T19:06:10.937" Score="13" Body="&lt;p&gt;You are right that LSTMs work very well for some problems, but some of the drawbacks are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LSTMs take longer to train&lt;/li&gt;&#xA;&lt;li&gt;LSTMs require more memory to train&lt;/li&gt;&#xA;&lt;li&gt;LSTMs are easy to overfit&lt;/li&gt;&#xA;&lt;li&gt;Dropout is much harder to implement in LSTMs&lt;/li&gt;&#xA;&lt;li&gt;LSTMs are sensitive to different random weight initializations&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;These are in comparison to a simpler model like a 1D conv net, for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first three items are because LSTMs have more parameters.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-02T19:19:51.503" LastActivityDate="2018-02-02T19:19:51.503" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27402" PostTypeId="1" AcceptedAnswerId="27403" CreationDate="2018-02-02T21:21:27.427" Score="1" ViewCount="159" Body="&lt;p&gt;My PHD is about yield of soybeans and it is typical agriculture theme, but I am pretty good with programming and Python programming language and I have already some deep learning programs which I made by myself successfully. Problem is that model is not very easy to show on the paper. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I show model just like a chart or graph? And maybe somebody already solved issue with transporting deep learning model file to &quot;paper&quot;? Is it normal to use deep learning in my agriculture PHD instead common middle age Linear Regression etc?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inputs and outputs is ok on paper but deep learning model is a little bit controversial in my scientific circles and I don't know how to present it so I'm asking maybe somebody already has successful experience by presenting deep learning model on the paper in science.&lt;/p&gt;&#xA;" OwnerUserId="45643" LastEditorUserId="45643" LastEditDate="2018-02-02T21:36:31.243" LastActivityDate="2018-09-01T15:35:03.513" Title="Can i use deep learning in my agriculture PHD?" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="3" ClosedDate="2018-09-01T23:05:49.093" ContentLicense="CC BY-SA 3.0" />
  <row Id="27403" PostTypeId="2" ParentId="27402" CreationDate="2018-02-02T21:49:37.460" Score="8" Body="&lt;p&gt;Please note that the following is my personal opinion; I still hope that you find it useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you cannot properly explain a concept on paper, then it should not be part of your PhD thesis. It is possible to implement Deep Learning algorithms without any conceptual knowledge. While this may be (somewhat) acceptable in the industry, it will get you into trouble in academia. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no doubt that your algorithms are well-programmed and solve many issues that are challenging from a programmer's perspective. But that is not what the reviewers of your thesis are interested in. They are also not interested in the prediction accuracy of your algorithm. &#xA;As scientists, they want to know &lt;em&gt;why&lt;/em&gt; something happens; and to my knowledge, Deep Learning algorithms provide next to nothing on this question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this is where 'middle age' linear regression has its strengths (by the way: a quick way to make an enemy is to tell him that the method he has been using all his life is too old to be useful). Yes, linear regression is unexciting, and it has many shortcomings. But it is extremely well understood. Most importantly, there are tons of books and papers about the connection between &lt;em&gt;assumptions&lt;/em&gt; and &lt;em&gt;valid interpretations&lt;/em&gt;: we know under which assumptions and in which sense the linear regression estimator is optimal; we know which assumptions lead to which kind of confidence intervals for the estimator; we know about robustness with respect to model misspecification; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As long as you cannot address these issues convincingly and on paper (That is: In what sense is my result optimal? How confident can I be about my result? How robust is my model?), you should not use Deep Learning in research as an alternative to classical statistical methods.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-09-01T15:35:03.513" LastActivityDate="2018-09-01T15:35:03.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27404" PostTypeId="2" ParentId="8015" CreationDate="2018-02-02T22:21:53.587" Score="2" Body="&lt;p&gt;Both screengrabs you've posted are not associated with any particular method of calculating probabilities. So they keep the formulae for calculating Naive Bayes very general, without endowing the notation with any specific meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make them &quot;anything&quot;, whether some variant of Bernoulli, Multinomial or so on, involves defining how the $P(...)$s or $p$s are defined and evaluated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you want to use &lt;a href=&quot;http://www2.aueb.gr/users/ion/docs/ceas2006_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, you can simply &quot;plug in&quot; formulae to evaluate them as defined in the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for multivariate Bernoulli Naive Bayes with a Laplacian prior, you would plug in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CvK90.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CvK90.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, for Multinomial Naive Bayes (whether using TF or binary attributes), you would plug in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Rxc1d.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rxc1d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the second formula, $p = \frac{1}{1+e^{\eta}}$, it is simply a more computationally-accurate way to evaluate the general formula Naive Bayes posterior calculation $p = \frac{\prod_{i=1}^N p_i}{\prod_{i=1}^N p_i + \prod_{i=1}^N (1-p_i)}$. This is because performing summation operations in the log space over many operands may often be more accurate than performing multiplication operations in the usual space, due to the way floating-point arithmetic is implemented in computers.&lt;/p&gt;&#xA;" OwnerUserId="45615" LastActivityDate="2018-02-02T22:21:53.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27407" PostTypeId="1" CreationDate="2018-02-03T02:20:33.310" Score="2" ViewCount="1608" Body="&lt;p&gt;I'm trying to use clustering to automate a group-finding process with the aim of being able to automatically detect groups in unseen data. The data are html elements within any given webpage, this includes numerous features including spatial and visual information rendered from a web browser. The detected groups are usually semantically meaningful, for example a group of navigation links or a list of items in the main content area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've implemented a subtle distance measure that captures how similar or different these elements are and I'm passing these values to DBSCAN for clustering. So far it works quite well but I'm trying to think of ways to improve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distance function is actually a weighted sum of 10 separate distance measures that capture different kinds of similarities and differences. I've weighted the more reliable and decisive measures higher than others. In theory, the measures that are weighted lower should only have an effect when they are congruent and are otherwise overpowered by the higher-weighted measures. Additionally each measurement function can itself amplify or attenuate it's own weight for a given item-pair, for example set its weight to zero if it is unsure and wants to abstain, or set it higher if it has found a similarity/difference that is particularly notable or reliable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although it's broadly working quite well, I've noticed that often one or two of the largest clusters should be further split. I've considered two methods for this: (a) attenuating the weights of measures that are mostly giving a low distance value within the cluster (thereby amplifying other measures with higher variance) and then rerunning DBSCAN or (b) searching clusters for subsets with large numbers of co-occurring features (e.g. some css class in the parent element and a particular font-size) and either splitting the cluster based on these, or adjusting the distance measures to reflect this apparent grouping and then rerunning the global DBSCAN with the adjusted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently it all works unsupervised but I'm open to manually creating a set of ground-truth data for improve it. Maybe then this could be used to optimize the weights in such a way that minimizes within-cluster distances and maximizes between-cluster distances? I also like the idea of running clustering multiple times, repeatedly adjusting the weights based on the variances of each measure within the cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for more ideas on how to optimize this, I would appreciate any input. I have seen some comments elsewhere that unsupervised clustering is only suitable for exploring data and not automating decisions, but I haven't been able to find a better way of finding these groups with my limited ML/data-science experience. Given all the time I've spent building this, the ideal thing would be to figure out a clever way of adaptively adjusting these weights in a more intelligent way. &lt;/p&gt;&#xA;" OwnerUserId="45651" LastActivityDate="2019-09-03T18:02:40.370" Title="Clustering with multiple distance measures" Tags="|python|scikit-learn|clustering|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27408" PostTypeId="1" AcceptedAnswerId="27415" CreationDate="2018-02-03T02:25:27.387" Score="1" ViewCount="99" Body="&lt;p&gt;Why in LSTM we calculate gradient w.r.t weights, but not w.r.t the cell state?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it theoretically possible to correct the contents of the cell state, and what would it result in?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that weights are like a &quot;set of skills&quot;, so that network can respond correctly to the input, even gazillions of iterations later. The cell-state is an understanding of what's going on in the past, up to the start of the current minibatch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why not to correct the value stored in the cell state? It would be very useful if we carry the cell state forward, between minibatches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/44183738/9007125]&quot;&gt;https://stackoverflow.com/a/44183738/9007125]&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Generally it is recommended to reset state after each epoch, as the state may grow for too long and &lt;strong&gt;become unstable&lt;/strong&gt;. However in my experience with small size datasets (20,000- 40,000 samples) resetting or not resetting the state after an epoch does not make much of a difference to the end result. For bigger datasets it may make a difference.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-02-08T21:54:31.830" LastActivityDate="2018-02-08T21:54:31.830" Title="Possible to correct an actual cell state in LSTM via gradient?" Tags="|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27409" PostTypeId="2" ParentId="27347" CreationDate="2018-02-03T02:54:44.647" Score="6" Body="&lt;p&gt;I am pretty sure you have to:&lt;/p&gt;&#xA;&lt;p&gt;wait for enough data in the memory (10 000 entries, etc), then take out a minibatch (&lt;strong&gt;for example&lt;/strong&gt;, 64 &lt;strong&gt;random&lt;/strong&gt; elements), run backprop once, using this minibatch.&lt;/p&gt;&#xA;&lt;p&gt;You then take an average of resulted gradients and finally, correct your network with that averaged gradient.&lt;/p&gt;&#xA;&lt;p&gt;After,&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;put the elements back in the memory (where you've taken them from)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Samples that have high priority are likely to be used in training many times. Reducing the weights on these oft-seen samples basically tells the network, &amp;quot;train on these samples, but without much emphasis; they'll be seen again soon.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;or &lt;a href=&quot;https://datascience.stackexchange.com/a/52098/43077&quot;&gt;throw it out, substituting it with new memories instead.&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Then keep playing the game, adding say, another 10 examples before doing another weights-correction session after assembling a batch of 64 random elements.&lt;/p&gt;&#xA;&lt;p&gt;I refer to the &amp;quot;Session&amp;quot; meaning a sequence of backprops, where the result is the average gradient used to finally correct the network.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself, or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Consider using just one network.  Let's say our memory bank contains several elements:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;...&#xA;{...}&#xA;{stateFrom, takenActionIndex, takenActionQval, immediateReward, stateNext }  &amp;lt;-- a single element&#xA;{...}&#xA;{...}&#xA;{...}&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;When using each element in your memory during the Correction session (one element after the other), you need to:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;pick an element. As seen, it contains stateFrom, taken action (action X), qvalue of action X, reward you received, state that it led to.&lt;/li&gt;&#xA;&lt;li&gt;Run forward prop as if you were in &amp;quot;next state&amp;quot; (mentioned in that element).  Get its best [action Y, qvalue]. In fact, your action Y doesn't have to be the action with highest q-value. Instead, it could be the &amp;quot;epsilon-greedy decided action&amp;quot; from the &amp;quot;next state&amp;quot; - you would then have &lt;a href=&quot;https://studywolf.wordpress.com/2013/07/01/reinforcement-learning-sarsa-vs-q-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SARSA instead of Q-learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;obtain delta by which the &amp;quot;qvalue of action X&amp;quot; differs from the &lt;strong&gt;immediate reward + &amp;quot;qvalue of action Y&amp;quot;&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;get deltas of several other elements that you've chosen from the bank&lt;/li&gt;&#xA;&lt;li&gt;Using each delta, compute a gradient tensor for your network's weights. Average those gradients into a single tensor. Punish your network's weights with this tensor.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I've intentionally circumvented 1 thing - you don't actually store takenActionQval, because they might become obsolete by the time you fetch its element from MemoryBank. You have to re-compute these scores during backprop.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Therefore, you are training against the version of your network from the similar but previous correction session.&lt;/p&gt;&#xA;&lt;p&gt;Notice, you don't store [&amp;quot;next state&amp;quot;, action Y] because by the time you select it to train (maybe you didn't select it for several minibatches), the network might have a different q-value for that action Y.&lt;/p&gt;&#xA;&lt;p&gt;You could also &lt;strong&gt;copy your network to the second network (target network), but only, say, every 200 timesteps&lt;/strong&gt;. You would still proceed to punish &lt;strong&gt;your&lt;/strong&gt; network for any differences with the target network in the meantime, after every 30 timesteps.&lt;/p&gt;&#xA;&lt;p&gt;Notice, the intuition of why this works is because: the Q-values are sort-of &amp;quot;flowing&amp;quot; from finish back to the beginning, by a little with every new journey. And you are always updating the current state towards the successor state. And train the successor state against its own (even further) successor.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2023-02-20T17:00:28.053" LastActivityDate="2023-02-20T17:00:28.053" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27410" PostTypeId="1" CreationDate="2018-02-03T05:59:44.923" Score="2" ViewCount="106" Body="&lt;p&gt;I have 12000 audio files which lengths is about 2 seconds and 12000 text files with phonemes and time stamps for audio files. My task is to train NN to synthesize audio for train data set only. I already tried feedforward NN and it works good only for 20 audio files. Which architecture should i choose? What should i do with data(preprocessing e.t.c)?&lt;/p&gt;&#xA;" OwnerUserId="45658" LastActivityDate="2018-02-03T05:59:44.923" Title="How to train neural network for text-to-speech task?" Tags="|machine-learning|python|neural-network|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-04T15:23:57.237" ContentLicense="CC BY-SA 3.0" />
  <row Id="27412" PostTypeId="1" AcceptedAnswerId="32007" CreationDate="2018-02-03T08:47:26.227" Score="6" ViewCount="1479" Body="&lt;p&gt;I'm trying to detect voice in a noisy environment. This is a New Zealand suburban/bush soundscape. We're using low quality microphones (but I don't think this has an effect on my problem) to periodically record the soundscape in an effort to record birdsong. One of the main challenges is we need to remove all potential human voice for privacy reasons. I'm stuck on the problem of how to detect voice in a noisy environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example of a spectogram with a person speaking 8 steps away from the microphone. There is crackle in the recording (don't believe this has an effect) and a bird chirping (activity around 2000hz) in the background. You can see the voice at 256-1024hz from 8sec to 13sec. The activity around 128hz from 15 to 22sec is a vehicle. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mj0bT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mj0bT.png&quot; alt=&quot;voice, 8 steps away&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm trying to find a way of recognising voice activity (with high degrees of precision) just like in this file. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that using a Machine Learning algorithm will have the best success due to the variability of voice and these sound files. &lt;em&gt;(I have also been working on a mathmatical method based on energy proportion in the voice freq ranges per sample but this just doesn't seem to work with the variability of the sound files)&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We only have around 10minutes of voice recorded like in the above spectogram. We then probably have about 24hours of non-voice recordings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q1: &lt;strong&gt;Will this be enough data, if not, would using data from freesound.org be advisable?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q2: &lt;strong&gt;What types of algorithms would work best?&lt;/strong&gt; - processing time is not a worry, the data contains noise from a multitude of sources (vehicles, animals, people, weather), I don't have much voice to work with(but I expect I can just use voice files from freesound.org) and &lt;strong&gt;I want to achieve 100% precision&lt;/strong&gt; with not so much of a worry about recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q3: &lt;strong&gt;Does anyone know of any projects in a similar space to mine where I could see what they've done?&lt;/strong&gt; (there has been work done on voice detection [VAD] but from a few experiments it all isn't suited to as noisy environments as mine ie. &lt;a href=&quot;https://github.com/wiseman/py-webrtcvad&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q4: Last question, &lt;strong&gt;does anyone know any efficient techniques to label discrete events in continuous audio data to use in ML algorithms in Python?&lt;/strong&gt; (At the moment I expect I'll be listening to files on Audacity and recording where voice starts and stops via a spreadsheet)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, any other advice would be awesome!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(FYI I have done courses looking at ML-regression, Neural Nets and SVM/tree's but wouldn't call my self 'ML proficient')&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, are there any other places I can ask these questions? I realise DSP.stack is more suited to solving small problems in the dsp area but I don't know where else to ask this!&lt;/p&gt;&#xA;" OwnerUserId="45602" LastActivityDate="2018-05-23T02:38:15.393" Title="Detecting voice in a noisy environment" Tags="|python|audio-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27414" PostTypeId="2" ParentId="13138" CreationDate="2018-02-03T09:29:07.340" Score="9" Body="&lt;p&gt;There is a nice write up about &lt;a href=&quot;http://torch.ch/blog/2016/07/25/nce.html&quot; rel=&quot;noreferrer&quot;&gt;&lt;em&gt;Language modeling a billion words&lt;/em&gt;&lt;/a&gt;. Below are some excerpts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Word-level&lt;/strong&gt; models have an important advantage over character-level models.&lt;br&gt;&#xA;Take the following sequence as an example (a quote from Robert A. Heinlein):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Progress isn't made by early risers. It's made by lazy men trying to&#xA;  find easier ways to do something.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;After tokenization, the word-level model might view this sequence as containing 22 tokens. On the other hand, the character-level will view this sequence as containing 102 tokens. This longer sequence makes the task of the character model harder than the word model, as it must take into account dependencies between more tokens over more time-steps. Another issue with character language models is that they need to learn spelling in addition to syntax, semantics, etc. In any case, word language models will typically have lower error than character models&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main advantage of &lt;strong&gt;character&lt;/strong&gt; over word language models is that they have a really small vocabulary. For example, the GBW dataset will contain approximately 800 characters compared to 800,000 words (after pruning low-frequency tokens). In practice this means that character models will require less memory and have faster inference than their word counterparts. Another advantage is that they do not require tokenization as a preprocessing step.&lt;/p&gt;&#xA;" OwnerUserId="45663" LastEditorUserId="29575" LastEditDate="2018-02-03T15:06:39.150" LastActivityDate="2018-02-03T15:06:39.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27415" PostTypeId="2" ParentId="27408" CreationDate="2018-02-03T09:32:21.530" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it theoretically possible to correct the contents of the cell state, and what would it result in?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes it is. Using back propagation, it is possible to get the gradient of any value that affects a well-defined output. For training data, that includes all current cell outputs - in fact these are necessary to calculate as an interim step in order to get the gradients of the weights. Once you have the gradients of a cost or error function, then you can perform a step of gradient descent in order to discover a value that would result in a lower error for given training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In usual training scenarios you do not alter neuron outputs after they have been calculated using feed-forward, because these are not parameters of the model. Typical reasons to alter neuron values (or even the input) are in order to view what the ideal state might be in a given scenario. If your state can be visualised through some decoder - maybe even another neural network - then this could allow you to see the difference between actual internal state, and a potentially better one. That could be useful in error analysis for example.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So why not to correct the value stored in the cell state? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's because in usual training scenarios, you are creating a network that predicts an output value. You can calculate the right corrections for training data, but not when predicting, because the whole point of predicting is to estimate a label that you do not already have. As such, you want to alter your function parameters, and not interim values.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It would be very useful if we carry the cell state forward, between minibatches.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Only during training. In a prediction scenario you usually have no way of calculating the necessary gradients. What you don't want is to train a system that then requires using error values and gradients that you do not have in production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some scenarios, such as an online system predicting next item $x_{t+1}$ in a sequence, where you could immediately train based on error after you observed the next item and before you continued the prediction sequence for $x_{t+2}$, you could possibly use the approach. I am not sure whether it would help performance, but &lt;em&gt;in principle it could&lt;/em&gt;. If it did help, you'd have to compare the improvement versus other simpler changes such as different hyper-parameters on a network that didn't correct internal state using gradients.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In summary, it is possible your idea would work quite well in an online system with near-immediate feedback. In that case you could think of a set of weights as being &quot;rules to update a belief state from data&quot;, and the output of hidden layer neurons as being &quot;a current belief state&quot;. When errors occur, it does appear to make sense to update both the rules that led to the error and the current belief that resulted from earlier faulty rules. It is perhaps worth an experiment or two. The main caveat is that the two update processes (for weights and LSTM layer state) would interact and/or adapt to each other, so it may not lead to measurably different performance than just adding more LSTM cells to the layer.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-02-03T14:07:58.517" LastActivityDate="2018-02-03T14:07:58.517" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27417" PostTypeId="1" CreationDate="2018-02-03T11:28:32.107" Score="1" ViewCount="81" Body="&lt;p&gt;I want to use Data Mining/Machine Learning for a problem and I'm not sure if there is a standard algorithm for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is as follows: There is a set of Events and a set of Potential Triggers. Each trigger can give rise to none, one or several Events. I want to classify Potential Triggers based on their features into ones that do not cause any event and ones that do. So far this is a standard classification task. Now comes the twist: For some of the Events, there are multiple Potential Triggers. This usually means that one is the real trigger and the others have been labeled accidentally. If the algorithm generates a rule (assuming rule learning for the moment) that covers one of the potential triggers for an event, this event shall be considered &quot;explained&quot; so that the other potential triggers for this event do not have to be classified as triggers for this event any more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The twist in the problem smells a bit like set cover (&lt;a href=&quot;https://en.wikipedia.org/wiki/Set_cover_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Set_cover_problem&lt;/a&gt;), so I'm wondering if there is a good algorithm at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a published algorithm? And does my problem have a standard name under which I can search for further information?&lt;/p&gt;&#xA;" OwnerUserId="42291" LastActivityDate="2022-12-26T14:03:25.393" Title="Classifying potential triggers for events (many-to-many relationship): Which algorithm to use?" Tags="|machine-learning|classification|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27419" PostTypeId="1" CreationDate="2018-02-03T13:03:26.520" Score="1" ViewCount="92" Body="&lt;p&gt;Is there any way to find out execution time for classification algorithms in orange? &lt;/p&gt;&#xA;" OwnerUserId="45675" LastActivityDate="2018-02-03T15:24:49.017" Title="Classification execution time - Orange" Tags="|orange|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27420" PostTypeId="1" CreationDate="2018-02-03T14:29:42.740" Score="0" ViewCount="5217" Body="&lt;p&gt;I need to use ensemble clustering method by using python in my data set. I already applied k-means clustering by using scikit learn library. I also applied different classification method also find ensemble classification method in scikit-learn. Now I am confused is there any library exist in scikit learn for ensemble clustering or how I can apply ensemble clustering method on my data set? &lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2021-03-02T12:10:33.300" Title="How to apply ensemble clustering method?" Tags="|python|data-mining|scikit-learn|clustering|ensemble-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27421" PostTypeId="1" AcceptedAnswerId="27425" CreationDate="2018-02-03T14:54:18.867" Score="7" ViewCount="15150" Body="&lt;p&gt;In &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot; rel=&quot;noreferrer&quot;&gt;CNNs&lt;/a&gt; when do we update the kernel parameters using back propagation? Suppose I have batch size of 50 and training data of 1000. Do I back propagate after each batch has been presented to network or after each data sample?&lt;/p&gt;&#xA;" OwnerUserId="45679" LastEditorUserId="28175" LastEditDate="2018-02-03T15:44:31.010" LastActivityDate="2018-07-06T12:41:05.663" Title="When are weights updated in CNN?" Tags="|machine-learning|neural-network|deep-learning|backpropagation|cnn|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27422" PostTypeId="1" CreationDate="2018-02-03T15:18:33.053" Score="4" ViewCount="1526" Body="&lt;p&gt;Each problem required its own similarity/dissimilarity measure. Imagine we are dealing with dataset composed with vector of real. I suppose that we mostly use the euclidean distance especially in low dimension. Unfortunately we often have to deal with huge dimensional dataset and we apply divers dimension reduction techniques to make the problem easier to solve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be happy to know your opinion about different similarity/dissimilarity measure you use after made a dimensionality reduction and how they impact classification/regression/clustering metrics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it better to use euclidean distance after a PCA or another distance measure ? Why ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if i prefer to use AutoEncoder to reduce my dataset dimensions, does it exists a special one that guaranty to offer better results with a specific similiarity/dissimilarity measure ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it exist a dimensionality reduction technique that is optimal with a particular distance metrics an which is bounded in $O(n.log(n))$ in time ?&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="9257" LastEditDate="2018-02-08T09:25:36.327" LastActivityDate="2018-07-08T13:50:06.230" Title="Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / ... )?" Tags="|similarity|dimensionality-reduction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27423" PostTypeId="2" ParentId="27421" CreationDate="2018-02-03T15:19:59.140" Score="0" Body="&lt;p&gt;During The training stage, it just takes input and assigns values in a matrix.&#xA;When you start giving it test data during the first iteration it just choose some random weights and try to figure out what it gets and then take that prediction as 'i' value based on the prediction, if it predicts true it doesn't change any of the weights for that class but if it predicted wrong class then it adjust weights according to parameters you have given then iterate again and until the prediction close to data what it is trained on.So, here only while test your data weights are altered and back propagated it may happen in the background by splitting your data of &quot;1000&quot; into training-set &amp;amp; test-set and once the training is completed test data is iterated multiple times until the loss stabilizes to certain point it may be different for each dataset your training on&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2018-02-03T15:19:59.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27424" PostTypeId="2" ParentId="27419" CreationDate="2018-02-03T15:24:49.017" Score="2" Body="&lt;p&gt;you can just use time library and give time counter like time save to variable and after classification take time now save to variable and subtract both and print the result you get the execution time&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2018-02-03T15:24:49.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27425" PostTypeId="2" ParentId="27421" CreationDate="2018-02-03T15:42:51.197" Score="14" Body="&lt;p&gt;Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Gradient Descent&lt;/li&gt;&#xA;&lt;li&gt;Stochastic Gradient Descent&lt;/li&gt;&#xA;&lt;li&gt;Batch Gradient Descent &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients with respect to all the data samples and updates the weights after passing the whole data-set. That means for each epoch, passing the whole data-set through the network, one update occurs. This update is accurate toward descending gradient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second one, updates the weights after passing each data which means if your data sample has one thousand samples, one thousand updates will happen whilst the previous method updates the weights one time per the whole data-sample. This method is not accurate but is so much faster than the previous one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last one tries to find a trade-off between the above approaches. You specify a batch size and you will update the weights after passing the data samples in each batch, means the gradients are calculated after passing each batch. Suppose you have one thousand data sample and you have specified a batch size with one hundred data sample. You will have 10 weight update for each epoch. This method is more accurate than the second approach and is more faster than the first approach. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Do I back propagate after each batch has been presented to network or after each image?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your method is the last one. Consequently, after passing the entire batch, you would update the weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;Based on the comments of one of our friends, the above approaches are named as follows, respectively:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[Batch] Gradient Descent (batch size = all training samples)&lt;/li&gt;&#xA;&lt;li&gt;True SGD (batch size = 1 - weights update for each training sample)&lt;/li&gt;&#xA;&lt;li&gt;Mini-batch SGD (batch size = m out of n training samples). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="-1" LastEditDate="2018-07-06T12:41:05.663" LastActivityDate="2018-07-06T12:41:05.663" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27426" PostTypeId="2" ParentId="23661" CreationDate="2018-02-03T15:45:42.093" Score="0" Body="&lt;p&gt;I think it might help where each and every type of datasets &lt;a href=&quot;https://github.com/ChristosChristofidis/awesome-deep-learning#datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2018-02-03T15:45:42.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27427" PostTypeId="2" ParentId="27421" CreationDate="2018-02-03T16:04:23.573" Score="1" Body="&lt;p&gt;Weight update can be understood as a change in weight to make your error less and less. YOu first assume some weights and get the model prediction and then the error. You then take the derivate of the error wrt to weights and finally update the weights so the error will reduce. you can also see it as a minimum finding problem in calculus. you are finding the values of weight or say the point in weight space that gives you the minimum error. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-02-03T16:04:23.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27428" PostTypeId="2" ParentId="13138" CreationDate="2018-02-03T16:08:03.870" Score="2" Body="&lt;p&gt;In my opinion, the character based RNNs will also perform better but they need much more data than the word based models and character based models need to train for a much longer period of time. I would say it is more of a trial and error as well as a trade-off between data and computation power available. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastEditorUserId="29575" LastEditDate="2018-02-03T16:31:45.290" LastActivityDate="2018-02-03T16:31:45.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27429" PostTypeId="2" ParentId="27285" CreationDate="2018-02-03T16:13:50.470" Score="1" Body="&lt;p&gt;Think of this as varied filter size and varied filter values. It will extract different representation (or say capture different part of the image), and then stack them to get a bigger feature vector. Then, you do the featurisation .Also, check for dilated CNNs used for NLP. They are based on somewhat similar concepts.&lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-02-03T16:13:50.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27431" PostTypeId="2" ParentId="17839" CreationDate="2018-02-03T17:02:36.430" Score="4" Body="&lt;p&gt;Ground truth: That is the reality you want your model to predict.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may have some noise but you want your model to learn the underlying pattern in data that’s causing this ground truth.  Practically, your model will never be able to predict the ground truth as ground truth will also have some noise and no model gives hundred percent accuracy but you want your model to be as close as possible.  &lt;/p&gt;&#xA;" OwnerUserId="26444" LastEditorUserId="26444" LastEditDate="2018-02-04T16:30:49.563" LastActivityDate="2018-02-04T16:30:49.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27432" PostTypeId="1" CreationDate="2018-02-03T17:13:29.017" Score="3" ViewCount="1056" Body="&lt;p&gt;Many people use the mean and variance of the &lt;strong&gt;training set&lt;/strong&gt; to standardize the &lt;strong&gt;test set&lt;/strong&gt;, instead of calculating the mean and variance of the test set and use these. Isn't risky to do that ? If no, why ?&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="28175" LastEditDate="2018-02-03T18:10:58.387" LastActivityDate="2018-02-03T20:04:05.220" Title="Mean and Variance of Feature Scaling" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|feature-scaling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27433" PostTypeId="2" ParentId="761" CreationDate="2018-02-03T17:27:03.130" Score="-2" Body="&lt;p&gt;Go with Kmeans clustering as HBScan will take forever.   I tried it for one of the project and ended but using Kmeans with desired results.  &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-02-03T17:27:03.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27434" PostTypeId="2" ParentId="27432" CreationDate="2018-02-03T18:10:34.707" Score="1" Body="&lt;p&gt;It is not dangerous if you have enough data. If you have enough data, you can somehow estimate the distribution of the phenomenon in hand. If you find the distribution of your sample and its parameters, it means that you know everything about your phenomenon under study. If you are familiar with statistics and probability, you may know that whenever you have enough data, you can estimate the expected value of the random variable using the mean of data samples and standard deviation of the sample will be equal to the standard deviation of the random variable, again if you have enough data. If you have enough data, it means that there will be no difference between the value of mean and expected value also standard deviation of the sample and standard deviation of the population. So if you extract enough data, these values may be so close together. Moreover, the reason we are using standardization of data is that we want to have features with same scale. This is the main reason, consequently there is no need to find the exact value of mean and standard deviation. You may see among machine learning and deep learning practitioners that they may not do this operation because it's a bit time consuming. They just usually divide each feature by the greatest value of the corresponding feature among data samples.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-03T18:17:40.267" LastActivityDate="2018-02-03T18:17:40.267" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27436" PostTypeId="1" CreationDate="2018-02-03T18:40:34.860" Score="1" ViewCount="679" Body="&lt;p&gt;I'm working on an image segmentation algorithm with FCN (Long et al., 2015) as the backbone network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea I have is to use the argmax binary mask obtained from the final score layer (250x250x1) to generate some data (e.g. number of blobs in the mask) to modify the ground truth (e.g. set some pixels in the gt mask to 'ignore' labels) or in some way (partly) extract from the features (similar to RPN layer in FasterRCNN). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this violate any deep learning or machine learning rules?     &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2024-01-04T18:05:55.710" Title="Ground truth/label modification during training (with the data obtained from the" Tags="|deep-learning|labels|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27438" PostTypeId="2" ParentId="27421" CreationDate="2018-02-03T19:22:31.363" Score="3" Body="&lt;p&gt;Back-propagation technically refers to computing the gradient of the loss function with respect to the parameters. According to Section 6.5 of the &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt; book:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi layer neural networks. Actually, back-propagation refers only to the method for computing the gradient, while another algorithm,such as stochastic gradient descent, is used to perform learning using this gradient.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The weights are updated right after back-propagation in each iteration of stochastic gradient descent. From Section 8.3.1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2SMUQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2SMUQ.png&quot; alt=&quot;SGD&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you can see that the parameters are updated by multiplying the gradient by the learning rate and subtracting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The SGD algorithm described here applies to CNNs as well as other architectures.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-03T19:30:21.117" LastActivityDate="2018-02-03T19:30:21.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27439" PostTypeId="2" ParentId="27432" CreationDate="2018-02-03T20:04:05.220" Score="3" Body="&lt;p&gt;It doesn't make sense to standardize your test set with the mean and variance computed on the test set. At best the test mean and variance will be close enough to the training mean and variance to not break your model - but why not guarantee they are the same by using the same standardization procedure for both. Typically it is best to choose one of the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute mean and variance on entire data set (train + test) and use these to standardize each set.&lt;/li&gt;&#xA;&lt;li&gt;Compute mean and variance on train set and use these to standardize each set.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you are not worried about leaking information from the test set then choose (1), otherwise choose (2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you standardize the test set with different values than the training set you might end up with train and test data from different distributions, which will lead to failure to make accurate predictions.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-03T20:04:05.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27440" PostTypeId="1" CreationDate="2018-02-03T20:32:57.663" Score="1" ViewCount="6201" Body="&lt;p&gt;I'm having some trouble interpreting what's going on in the training and validation loss, sensitivity, and specificity for my model. My validation sensitivity and specificity and loss are NaN, and I'm trying to diagnose why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training set has 50 examples of time series with 24 time steps each, and 500 binary labels (shape: (50, 24, 500)). My validation set has shape (12, 24, 500). Of course, I expect a neural network to overfit massively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because I was interested in a stateful LSTM, I followed philipperemy's advice and used &lt;code&gt;model.train_on_batch&lt;/code&gt; with batch_size = 1. I had multiple inputs: one called &lt;code&gt;seq_model_in&lt;/code&gt; that is a time series, and one called &lt;code&gt;feat_in&lt;/code&gt; that is not a time series (and so is concatenated in to the model after the LSTM but before the classification step). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As my classes are highly imbalanced, I also used Keras's &lt;code&gt;class_weights&lt;/code&gt; function. To make this function work in a multi-label setting, I concatenated two columns to the front of my responses (one of all 0s and one of all 1s), such that the final shape of the response is (50, 502).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; feat_in = Input(shape=(1,), batch_shape=(1, 500), name='feat_in')&#xA; feat_dense = Dense(out_dim, name='feat_dense')(feat_in)&#xA; seq_model_in = Input(shape=(1,), batch_shape=(1, 1, 500), name='seq_model_in')&#xA; lstm_layer = LSTM(10, batch_input_shape=(1, 1, 500), stateful=stateful)(seq_model_in)&#xA; merged_after_lstm = Concatenate(axis=-1)([lstm_layer, feat_dense])&#xA; dense_merged = Dense(502, activation=&quot;sigmoid&quot;)(merged_after_lstm)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have coded this model in Keras for time series prediction (multi-label prediction at the next time step):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training and validation metrics and loss do not change per epoch, which is worrisome (and, I think, a symptom of overfitting), but I'm also concerned about understanding the graphs themselves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the TensorBoard graphs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UzlYd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UzlYd.png&quot; alt=&quot;training loss&quot;&gt;&lt;/a&gt; &#xA;&lt;a href=&quot;https://i.stack.imgur.com/xE5nq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xE5nq.png&quot; alt=&quot;validation loss&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/7VYQT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7VYQT.png&quot; alt=&quot;training sensitivity&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/SanjL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SanjL.png&quot; alt=&quot;validation sensitivity&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HDKH8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HDKH8.png&quot; alt=&quot;training specificity&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HG36Q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HG36Q.png&quot; alt=&quot;validation specificity&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training loss should (roughly) be decreasing per epoch, as should the validation loss. The training sensitivity and specificity are 92% and 97.5%, respectively (another possible sign of overfitting).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Am I right in thinking the sensitivity and specificity graphs should all have the same general shape as the train_spec graph?&lt;/li&gt;&#xA;&lt;li&gt;Why does the training sensitivity graph look like this?&lt;/li&gt;&#xA;&lt;li&gt;Why does the validation loss return NaN?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2021-07-30T15:04:05.490" Title="Keras stateful LSTM returns NaN for validation loss" Tags="|keras|lstm|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27441" PostTypeId="1" AcceptedAnswerId="27442" CreationDate="2018-02-03T21:15:49.533" Score="1" ViewCount="249" Body="&lt;p&gt;I'm trying to write a program that can take multiple low-resolution images as inputs and output a high-resolution image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that for single-image superresolution, Convolutional Neural Networks work great. I can just take a network with just three convolution layers as described &lt;a href=&quot;https://software.intel.com/en-us/articles/an-example-of-a-convolutional-neural-network-for-image-super-resolution&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;here&quot;&gt;here&lt;/a&gt;. I can then train the network with a huge dataset of low-res images and their corresponding high-res images (with mean-squared error cost function), and it should theoretically work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to achieve greater accuracy, though, I can gain more information about a potential high-resolution image by looking at multiple low-resolution images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is: How would I modify this neural-network based algorithm to be able to take multiple images (of the same object) as input, and output a more accurate high-resolution image as a result? I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Sorry if anything I said is incorrect; I know virtually nothing about this field, so any advice helps!)&lt;/p&gt;&#xA;" OwnerUserId="45655" LastActivityDate="2018-02-03T22:03:38.230" Title="Multi-image superresolution using CNNs" Tags="|machine-learning|computer-vision|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-04T15:23:40.267" ContentLicense="CC BY-SA 3.0" />
  <row Id="27442" PostTypeId="2" ParentId="27441" CreationDate="2018-02-03T22:03:38.230" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I agree with your analysis that this problem presents different challenges than single-image super-resolution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the single image case, we know exactly where everything should be in the output image, because it is just a higher resolution version of the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why not try and extend this idea to your case? Here is how you might do that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Concatenate each low-res image and a low-res version of the output image as each training and test example&lt;/li&gt;&#xA;&lt;li&gt;For prediction concatenate each low-res image and duplicate one of them to specify the desired orientation of the output&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So if you have &lt;code&gt;n&lt;/code&gt; 3-channel &lt;code&gt;h x w&lt;/code&gt; input images in each training example, your input shape should be &lt;code&gt;(h, w, 3*(n+1))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should give the network enough information to learn the correct transformation.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-03T22:03:38.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27443" PostTypeId="1" CreationDate="2018-02-03T22:13:26.207" Score="3" ViewCount="4024" Body="&lt;p&gt;I want to implement a time-series prediction model using LSTMs like the one mentioned here: &lt;a href=&quot;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model works pretty well using a similar Keras code above, though I want to implement the same using H20.ai&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/deep-learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation here on H2O Deep Water&lt;/a&gt; says &quot;The H2O Deep Water project supports CNNs and &lt;strong&gt;RNNs&lt;/strong&gt; though third-party integrations of other deep learning libraries such as TensorFlow, Caffe and MXNet.&quot;. Though there are no demos for the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this conversation here in Jan '16 on &lt;a href=&quot;https://groups.google.com/forum/#!topic/h2ostream/qU2FezMCyW8&quot; rel=&quot;nofollow noreferrer&quot;&gt;'Recurrent Neural Networks in H2O for time series prediction?'&lt;/a&gt;, they say it hasn't been implemented yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice or links to similar implementations on H2O would be deeply appreciated&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am looking for a data parallel architecture implementation rather than model parallelism&lt;/p&gt;&#xA;" OwnerUserId="23644" LastEditorUserId="23644" LastEditDate="2018-02-04T13:31:27.020" LastActivityDate="2019-01-07T15:20:35.393" Title="LSTM for time series forecasting with H20.ai" Tags="|python|deep-learning|time-series|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27444" PostTypeId="1" AcceptedAnswerId="27445" CreationDate="2018-02-03T23:50:01.190" Score="4" ViewCount="2471" Body="&lt;p&gt;The &lt;code&gt;KNeighborsClassifier&lt;/code&gt; has a method for predicting class probabilities. However, I cannot find any documentation describing how these probabilities are computed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a toy example that returns class probabilites:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.neighbors import KNeighborsClassifier&#xA;import numpy as np&#xA;N = 100&#xA;np.random.seed(1)&#xA;X = np.random.random((N,2))&#xA;Y = np.random.randint(2, size=N)&#xA;model = KNeighborsClassifier(5, weights='distance')&#xA;model.fit(X, Y)&#xA;print(model.predict_proba([[0.5, 0.5]]))&#xA;#&amp;gt;&amp;gt;&amp;gt; [[0.55486525 0.44513475]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="7738" LastActivityDate="2018-02-04T00:35:51.247" Title="How does sklearn KNeighborsClassifier compute class probabilites?" Tags="|scikit-learn|k-nn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27445" PostTypeId="2" ParentId="27444" CreationDate="2018-02-04T00:05:30.627" Score="4" Body="&lt;p&gt;The class probabilities are the normalized weighted average of indicators for the k-nearest classes, weighted by the inverse distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: Say we have 6 classes, and the 5 nearest examples to our test input have class labels 'F', 'B', 'D', 'A', and 'B', with distances 2, 3, 4, 5, and 6, respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the unnormalized class probabilities can by computed by:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1/2) * [0, 0, 0, 0, 0, 1] + (1/3) * [0, 1, 0, 0, 0, 0] + &#xA;(1/4) * [0, 0, 0, 1, 0, 0] + (1/5) * [1, 0, 0, 0, 0, 0] + &#xA;(1/6) * [0, 1, 0, 0, 0, 0] =&#xA;&#xA;[1/5 ,1/2, 0, 1/4, 0, 1/2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And after normalizing we get:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0.13793103, 0.34482758, 0.0, 0.17241379, 0.0, 0.34482758]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you choose &lt;code&gt;weights='uniform'&lt;/code&gt; then it becomes a simple average, which can be computed by multiplying each indicator by &lt;code&gt;(1/k)&lt;/code&gt; and summing, with no need for additional normalization.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-04T00:35:51.247" LastActivityDate="2018-02-04T00:35:51.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27446" PostTypeId="1" CreationDate="2018-02-04T04:17:24.167" Score="0" ViewCount="206" Body="&lt;p&gt;I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to me that the general configuration of the platform is focused on CSV format; so is most of the help material. In fact, I figured a way around when I changed the extension of my .txt file to .csv. Through this post and the answer followed, I am trying to share my experience with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This post is useful, if:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You have created a IBM DSX notebook with “Language=Scala” and “Spark version=2.1 OR 2.0″[did not try with other versions]&lt;/li&gt;&#xA;&lt;li&gt;AND You are trying to access a text file [did not try other formats] that you have uploaded from your personal computer to the IBM Cloud Object Storage&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45698" LastEditorUserId="45698" LastEditDate="2018-02-04T17:01:21.787" LastActivityDate="2018-02-04T17:01:21.787" Title="How to Access a Text File from IBM DSX Scala-Spark Notebook" Tags="|apache-spark|scala|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-04T15:17:38.747" ContentLicense="CC BY-SA 3.0" />
  <row Id="27447" PostTypeId="2" ParentId="27446" CreationDate="2018-02-04T04:18:24.987" Score="1" Body="&lt;p&gt;In order to access files that are in Cloud Object Storage, you need to provide the path of the file in the form of a CloudObjectStorage instance. Add following lines, after the file credentials, to your code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;var soc = new CloudObjectStorage(sc, credentials, null, “bluemix_cos”) var myFile = sc.textFile(soc.url(“your_bucket”,”data.txt”))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second code line above, “your_bucket” is the value of “BUCKET” in the file credentials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is one last thing we need to do before we are good to go. The CloudObjectStorage constructor accepts credentials in a way that is for the CSV files. Luckily the main difference between the two formats is the HashMap key names; hence, we can easily manage this problem by changing the above credentials manually, as mentioned below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“IBM_API_KEY_ID” as “apiKey”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“IAM_SERVICE_ID” as “serviceId”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“ENDPOINT” as “endPoint”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;“IBM_AUTH_ENDPOINT” as “iamServiceEndpoint”&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For further details, you can visit the link below:&#xA;&lt;a href=&quot;http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45698" LastActivityDate="2018-02-04T04:18:24.987" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27449" PostTypeId="1" AcceptedAnswerId="27453" CreationDate="2018-02-04T08:10:30.480" Score="3" ViewCount="3208" Body="&lt;p&gt;I have several thousand text documents and I am currently working on obtaining the latent feature representations of words and generate sentences using variational auto encoder. The main obstacle I am facing is “&lt;em&gt;how to input such large textual vectors into VAE (or say even in a simple auto encoder)&lt;/em&gt;&quot;. I know there are other techniques such as adding layers in the form of LSTMs, but I want to keep it simple and keep the model as just VAE. One way is to use one-hot-encoded vectors or bag of words, but again, this is not the most efficient way since for a vocabulary of 100K unique words, each document will have a 100K input vector. Additionally, we loose the sentence structure. For small datasets however, there is no problem in training an autoencoder using this type of input. Another method is to use word embeddings using a pre-trained Word2Vec. This is what I have been trying to do and the python notebook which can be &lt;a href=&quot;https://www.dropbox.com/s/j2jhnvh0u4v4t8c/VAEText.ipynb?dl=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;DOWNLOADED HERE&lt;/a&gt; uses this technique. The code is too long and has multiple pre-processing steps, so I am unable to embed this code in my post. The following are my questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Now, each sentence (or document) will have different number of words. So the number of word embeddings for each  document will have different lengths. Unfortunately, keras requires all the inputs to be of same length ( if I am right). So, how to handle such cases of varying input lengths?. Currently, in the fifth block of the python notebook, you can see the statement &lt;code&gt;data = [x for x in vect if len(x) == 10]&lt;/code&gt;. That is, I only consider documents that have exactly 10 words to overcome this problem. Ofcourse, this is not practical. Can we pad 0 vectors?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The VAE example shown in &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras blog&lt;/a&gt; uses MNIST data as exmple. Therefore, they use a &lt;em&gt;sigmoid activation&lt;/em&gt; in the final reconstruction layer; consequently, “&lt;em&gt;binary_crossentropy&lt;/em&gt;” as the loss function (along with the KL divergence). Since my inputs are word embeddings, where there are even negative values in the embeded vectors, I believe I should not use the activation as “&lt;em&gt;sigmoid&lt;/em&gt;” in the final reconstruction layer.  Is that right?. Additionally, I have also changed the loss as  “&lt;em&gt;mean_squared_error&lt;/em&gt;” instead of “&lt;em&gt;binary_crossentropy&lt;/em&gt;” in the attached code. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;It will be great if someone who has worked on VAE and autoencoders for text data can provide their inputs regarding the questions mentioned above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: the attached code is a simplified version of the code in &lt;a href=&quot;https://github.com/Toni-Antonova/VAE-Text-Generation/blob/master/vae_nlp.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;THIS LINK&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43921" LastActivityDate="2018-02-04T17:04:47.480" Title="Right Way to Input Text Data in Keras Auto Encoder" Tags="|python|deep-learning|keras|text-mining|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27450" PostTypeId="1" CreationDate="2018-02-04T08:36:59.017" Score="2" ViewCount="200" Body="&lt;p&gt;I'm reading Nielsen's book on neural networks. In the first chapter you construct a neural net to recognize handwritten digits by training the network on 60,000 samples. I wonder how would you build a neural net to recognize a person by his/her silhouette? Is there some special technique to make it possible? &lt;/p&gt;&#xA;" OwnerDisplayName="user45691" LastEditorUserId="28175" LastEditDate="2018-02-04T09:03:15.017" LastActivityDate="2018-02-04T09:03:15.017" Title="Deep learning technique to recognize a person's silhouette" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27451" PostTypeId="2" ParentId="27450" CreationDate="2018-02-04T08:52:20.623" Score="1" Body="&lt;p&gt;For this purpose you have to have a data-set that can be interpreted by the human. I mean an expert should label the data samples without hesitating. Then you can make a typical neural network and train it. In your case you have 10 classes and all 60 thousand images are labeled by an expert. Consequently, you are able to train a network. Your task is supervised and you need labeled data-set.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-04T08:52:20.623" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27452" PostTypeId="2" ParentId="27307" CreationDate="2018-02-04T16:04:44.710" Score="0" Body="&lt;p&gt;Consider that you are doing vector operation, change your cost function to the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(1 / m) * sum(((-y) .* (log(h)) - ((1 - y) .* log((1-h)))));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and your gradient to the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grad = (1./m) * (x' * (h - y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although the latter is just for precedence reassuring. &#xA;&lt;hr /&gt;&#xA;Based on the discussion in the chat, although the code calculates the cost in a wrong way, the reason the cost does not decrease is that the data is not linearly separable. Logistic regression is a simple algorithm which classifies successfully linearly separable data. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/21896/28175&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-04T17:06:15.053" LastActivityDate="2018-02-04T17:06:15.053" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27453" PostTypeId="2" ParentId="27449" CreationDate="2018-02-04T17:04:47.480" Score="1" Body="&lt;p&gt;Here's my take on your questions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, you can zero-pad vectors. However, I would strongly recommend you use an LSTM as part of your encoder if you're using sentences as input. An LSTM can take variable-length input - that's part of its charm!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The loss function depends entirely on your goal. It sounds like you're trying to create a sequence-to-sequence VAE. I would go about this by putting a softmax function on your output layer, to generate the next most probable word in the sentence. So, I would use &lt;code&gt;categorical_crossentropy&lt;/code&gt; as the loss function, and I would use &lt;code&gt;softmax&lt;/code&gt; instead of &lt;code&gt;sigmoid&lt;/code&gt;, because you're choosing the next most likely word from a set vocabulary (this is not a binary issue but rather a categorical issue).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Hope that helps!&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-02-04T17:04:47.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27454" PostTypeId="1" CreationDate="2018-02-04T17:57:15.457" Score="6" ViewCount="139" Body="&lt;p&gt;So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as:&#xA;$((\neg x_1\land x_3) \lor (\neg x_2 \land x_1 \land x_3))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have gotten as far as attempting to construct a 2-layer network which I believe should be able to learn this function, each neuron $h_1$ through $h_3$ performing a single logical operation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ydUmT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ydUmT.png&quot; alt=&quot;Network architecture&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However my lack of understanding in both logic and BDNs as a whole has led me to believe that my network is incorrect when considering the weight setting rules proposed by McCullough Pitts (j is neuron index, k is input index):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{jk} = -1$ if $F(x_k)=\neg x_k $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{jk} = 1$ if $F(x_k)=x_k $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understood it, you set the weight for each input to -1 if that input is negated in the neuron's formula and 1 otherwise. I was able to calculate these for the first layer with ease, but got thrown by a hole in my knowledge of how the second layer works. I assumed that $h_3$ takes $y_1$ and $y_2$ as input, as opposed to an $x_k$, so I don't see how the above rule can be applied. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping maybe someone with a stronger understanding of neural networks could perhaps explain to me how the weight setting rule is applied to the second layer, I currently assumed it meant $w=1$ if $F(y_i)=y_i$ and visa versa, but I believe this is wrong. &lt;/p&gt;&#xA;" OwnerUserId="45721" LastActivityDate="2018-02-04T17:57:15.457" Title="Learning a logical function with a 2 layer BDN network - manual weight setting rule question?" Tags="|neural-network|beginner|theory|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27455" PostTypeId="2" ParentId="27407" CreationDate="2018-02-04T18:03:37.787" Score="1" Body="&lt;p&gt;This is just a wild guess, but I was wondering whether your custom distance function is indeed a &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_(mathematics)#Definition&quot; rel=&quot;nofollow noreferrer&quot;&gt;distance function&lt;/a&gt;. &#xA;In particular, your problem might occur because your distance function $d(x, y)$ does not &lt;em&gt;separate&lt;/em&gt; observations: This is the case if there exist two observations $x_1$ and $x_2$ that are distinct, $x_1 \neq x_2$, but have zero distance, $d(x_1, d_2) = 0$. Then $x_1$ and $x_2 $ would necessarily be put into the same cluster by your algorithm.&#xA;If this happens for many observations in your test set, then you will observe large clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might be able to check on paper that your function satisfies the four axioms of a distance function (see the Wikipedia article above); if that is not possible, you could at least check it by looping over your test set.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-04T18:03:37.787" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27458" PostTypeId="1" CreationDate="2018-02-05T04:45:20.107" Score="1" ViewCount="63" Body="&lt;p&gt;I am looking to detect blink events in real-time single channel EEG. Classification of a moving window of samples to determine whether a blink artifact exists requires feature extraction (except when using deep learning, I am not experienced enough for this). What features would be useful to extract from a window of approx. 50-200 samples of time series data for detecting a blink event. The blink event can be easily seen in the below picture:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/X0rtH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0rtH.png&quot; alt=&quot;Blink event in EEG data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45739" LastActivityDate="2018-02-12T11:48:41.083" Title="Features for blink detection in real-time single channel EEG" Tags="|time-series|feature-extraction|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-18T13:57:37.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="27459" PostTypeId="1" CreationDate="2018-02-05T06:01:29.607" Score="6" ViewCount="133" Body="&lt;p&gt;How is the data classified as long-term memory and short-term memory?&#xA;Is there some standards programmers set?&lt;/p&gt;&#xA;" OwnerUserId="44714" LastEditorUserId="44714" LastEditDate="2018-02-06T01:12:07.910" LastActivityDate="2018-06-04T07:52:42.383" Title="Classifications as long-term memory and short-term memory in LSTM" Tags="|deep-learning|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27460" PostTypeId="1" CreationDate="2018-02-05T06:24:59.650" Score="2" ViewCount="188" Body="&lt;p&gt;I am the beginner of Machine learning.&#xA;&lt;strong&gt;The process is:&lt;/strong&gt;&lt;br&gt;&#xA;I have different log files (System log, MSSQL Server log, Linux log, MySQL Log, FTP log, IIS log).If any input is given, I will find out which type of log using machine learning technique. Each log has a different format. Some logs don't have structure format(Linux, MySQL log, FTP log).&#xA;   In my analysis, these are all implemented using KNN algorithm(machine learning).&#xA;  But I don't know how to implement this? please give any suggestion about this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The log type format:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Linux-syslog:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Jan  5 08:39:01 iei-Virtual-Machine CRON[48622]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] &amp;amp;&amp;amp; [ -x /usr/lib/php5/sessionclean ] &amp;amp;&amp;amp; [ -d /var/lib/php5 ] &amp;amp;&amp;amp; /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MySQL-error log:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2018-01-05 10:55:20 18856 [Warning] Unsafe statement is written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ between master and the slave. Statement: CALL SubmitGetChangeDetectionInfo(@_SubmitGetChangeDetectionInfo_0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Event log:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IE038,System log,Error,20/11/2017 12:47:51 PM,TerminalServices-Printers,1111,None,Driver HP Deskjet 3520 series required for printer IEC057(Mahendran) Printer is unknown. Contact the administrator to install the driver before you log in again.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;FTP log:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2018-01-04 00:00:01 162.254.209.219 INFOEVOL\EC 192.168.0.13 63346 &#xA;DataChannelOpened - - 0 0 c1df3130-60e6-4678-9dcd-39177cc60d06 -IISAppslog:&#xA;2017-11-06 03:25:16 192.168.0.13 GET /IEIAppsLogin.aspx &#xA;param=OwjgKJLT+ikfpFnxYbvZS/QWXTFP4GEXmT+qM7TeXTMqi5D7DKexzYjZc3aJNB0x 90 - &#xA;182.73.50.19 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+&#xA;(KHTML,+like+Gecko)+Chrome/61.0.3163.100+Safari/537.36 &#xA;http://182.73.50.19/AppsLogin.aspx 302 0 0 234&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;MSSQL Server:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;01/10/2018 07:07:07,Logon,Unknown,Login failed for user 'ms SQL'. Reason: Could not find a login matching the name provided. [CLIENT: 49.65.2.226]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45741" LastEditorUserId="45507" LastEditDate="2018-02-05T09:33:05.603" LastActivityDate="2018-02-08T19:16:39.787" Title="Find which type of log using machine learning?" Tags="|machine-learning|k-nn|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27461" PostTypeId="1" AcceptedAnswerId="27462" CreationDate="2018-02-05T07:55:46.140" Score="3" ViewCount="251" Body="&lt;p&gt;When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in my case would be several months of processing time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why I was looking for an alternative. On the &lt;a href=&quot;https://www.kaggle.com/hadend/tuning-random-forest-parameters&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt; website, I found a tutorial that uses a different approach. It almost works like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Test all 6 parameter individually with the other 5 parameters as default value and plot the results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Change the default values for alle 6 parameters to the best value of the associated plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Test all 6 parameter individually with the other 5 parameters as last best value and plot the results&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Repeat step 2 and 3 until the results does not change anymore&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach has the advantage of requiring much fewer runs. Is this a scientifically accepted procedure? And does this approach have a name in the literature?&lt;/p&gt;&#xA;" OwnerUserId="45747" LastEditorUserId="45507" LastEditDate="2018-02-05T11:50:55.923" LastActivityDate="2018-02-05T11:50:55.923" Title="Parameter tuning for machine learning algorithms" Tags="|hyperparameter|hyperparameter-tuning|grid-search|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27462" PostTypeId="2" ParentId="27461" CreationDate="2018-02-05T08:32:42.600" Score="0" Body="&lt;p&gt;In general, your approach will get stuck in local minima. This is why it is not scientifically accepted. &#xA;(Notice that this may be different in very special cases, in particular if the performance of the algorithm is a strictly convex function of all input parameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see how the approach fails, suppose your machine learning algorithm has two parameters, $x$ and $y$, which can be either $0$ or $1$. The default values are $x=1$ and $y=1.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The performance of your machine learning algorithm is $f$ and should be as high as possible.&#xA;Assume the following performance levels $f(x,y)$:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    | x=0 | x=1 &#xA;----|-----|-----&#xA;y=0 | 0.9 | 0.2&#xA;y=1 | 0.1 | 0.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Your approach would do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First, choose the default value $x=1$ and compute $f(x=1, y=0) = 0.2$ and $f(x=1, y=1) = 0.3$. Second, choose the default value $y=1$ and compute $f(x=0,y=1)=0.1$ and $f(x=1, y=1) = 0.3$.&lt;/li&gt;&#xA;&lt;li&gt;Change the default values to the best value. In this case, this requires no change since $x=1$ and $y=1$ are the best values, respectively.&lt;/li&gt;&#xA;&lt;li&gt;The result did not change. Report $(x=1, y=1)$ as the best parameter combination.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;But the global performance maximum occurs at $(x=0, y=0).$&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-05T08:32:42.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27464" PostTypeId="2" ParentId="27307" CreationDate="2018-02-05T08:55:58.433" Score="0" Body="&lt;p&gt;In general, there's nothing wrong with the output of calling the sigmoid being below &lt;code&gt;0.5&lt;/code&gt;. All the sigmoid does is to &quot;squash&quot; the data into a tight range that is consistent with a probability value (which must be between $0$ and $1$) before the next step of fitting the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if you're saying that values fail to converge, or get stuck at some small value during iterations, then I would guess there is a bug in your implementation of the loss/cost/error function, its gradient and related quantities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should definitely update the matrix &lt;code&gt;*&lt;/code&gt; operations to the &lt;a href=&quot;https://uk.mathworks.com/help/matlab/ref/times.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;element-wise&lt;/a&gt; &lt;code&gt;.*&lt;/code&gt; in this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, I'm a bit suspicious of the plus &amp;amp; minus signs in the following equations. You should double-check them against the original mathematical equations to make sure, as I don't think the signs look exactly right:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;j = (1./m) * ( -y' * log( h ) - ( 1 - y' ) * log ( 1 - h))&#xA;grad = 1./m * x' * (h - y);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45615" LastActivityDate="2018-02-05T08:55:58.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27465" PostTypeId="2" ParentId="27460" CreationDate="2018-02-05T09:50:15.487" Score="1" Body="&lt;p&gt;You first need to preprocess your text to convert it into features that can be consumed by a machine learning algorithm. Then you can try this algorithm (a classifier, in this case) with the known examples you have. Then you can use that trained model to recognise logs it hasn't seen before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have a preference of language you'd like to use, python with the scikit-learn library would be a good start. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your problem, work through this tutorial which is very similar to what you're trying to achieve:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45754" LastActivityDate="2018-02-05T09:50:15.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27466" PostTypeId="1" AcceptedAnswerId="31285" CreationDate="2018-02-05T10:24:33.047" Score="3" ViewCount="96" Body="&lt;p&gt;As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet.&#xA;The topic is a regression problem for which several machine learning approaches have been tested:&#xA;&lt;code&gt;SVR, RandomForestRegressor, GradientBoosting, DecisionTrees, Artificial Neural Networks, LSTMs, K-NearestNeighbors&lt;/code&gt;.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has been shown that useful results can be achieved with all approaches. Which of the methods is the best depends heavily on the parameter configuration. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be too much work to test any parameter configuration for any approach. Besides, I think it would be uninteresting to show dozens of plots about the parameter tuning tests, because they also depend heavily on the respective data sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how do you typically compare different types of machine learning approaches in scientific papers that are heavily dependent on the parameters? Just play a bit with grid search and randomized search until you are satisfied with the parameter configuration?&lt;/p&gt;&#xA;" OwnerUserId="45756" LastActivityDate="2018-05-06T13:04:07.417" Title="Comparison of machine learning approaches for a topic in a scientific paper" Tags="|machine-learning|hyperparameter|grid-search|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27467" PostTypeId="2" ParentId="27466" CreationDate="2018-02-05T10:35:49.860" Score="1" Body="&lt;p&gt;Hello I don't know about scientific paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I know that to compare quality of ML algorithms one against another you need to establish a metric : Make it a square mean error or anything else, a final number that can tell you if your algorithm is working as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On a side note : Every algorithm, if used in their correct purpose, is highly parameters dependent. So I'm not sure what you're referring to with &quot;machine learning approaches that are heavily dependent on the parameters&quot;&lt;/p&gt;&#xA;" OwnerUserId="45762" LastActivityDate="2018-02-05T10:35:49.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27468" PostTypeId="1" CreationDate="2018-02-05T10:35:58.543" Score="4" ViewCount="1939" Body="&lt;p&gt;I am working on a product classification problem (E-Commerce) in which I have to identify product category based on keywords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say for example, if input is given as '&lt;strong&gt;Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display&lt;/strong&gt;' , It should correctly identify it as &lt;strong&gt;'Mobile'&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem here is that I have certain key-value pairs on which I have to train the model. Previously, I was doing the same problem by training model on product level using LinearSVC and it was giving satisfactory results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Details about data: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I have 39 classes/ categories currently, which might increase in future.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am using a csv file which is of around 10 MB and it has around 7000 rows.(Refer below structure)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;I am using LinearSVC from sklearn.svm&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have the following structure for training now:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;--------------------------------------------------------------&#xA;| Attribute_Names   |  Attribute_Values        |   Category  |&#xA;--------------------------------------------------------------&#xA;| Brand             |  Samsung, Nokia, OnePlus | Mobile      |&#xA;| RAM, Memory       | 2 GB, 4 GB, 3 GB, 6GB    | Mobile      |&#xA;| Color,Colour      | Black, Golden, White     | Mobile      |&#xA;--------------------------------------------------------------&#xA;| Fabric, Material  | Cloth, Synthetic, Silk   | Ethnic Wear |&#xA;| Pattern, Design   | Digital, floral print    | Ethnic Wear |&#xA;--------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have the following queries:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Which Model to use for this problem ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - How would you handle such problem ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 - Any similar problem you have seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 1) - merge column(attribute_names, attribute_values)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 2) - Cleaned the data(removing stopwords,special characters, steming)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 3) - Feature extraction using TfidfVectorizer(stop_words='english',lowecase=True)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;step 4) - OneVsRestClassifier(LinearSVC(loss='hinge',penalty='l2'))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This are the steps we have performed till now,and the results are not good enough (miss-classification still exist) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;format/structure of data is same as we mentioned above.&lt;/p&gt;&#xA;" OwnerUserId="45763" LastEditorUserId="45763" LastEditDate="2018-02-06T06:31:38.747" LastActivityDate="2018-02-06T06:31:38.747" Title="Product classification in e-commerce using attribute keywords" Tags="|machine-learning|python|classification|scikit-learn|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27470" PostTypeId="2" ParentId="6931" CreationDate="2018-02-05T11:32:43.337" Score="0" Body="&lt;p&gt;Some more libs available for working with SOMs are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1305.1422.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Somoclu&lt;/a&gt; : This is a very fast implementation written to work&#xA;with/without GPUs on a standalone or a distributed system.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pypi.python.org/pypi/MiniSom/1.1.1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Minisom&lt;/a&gt; : Decent library with few visualization capabilities&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both can be installed using pip, so you can quickly try them out.&lt;/p&gt;&#xA;" OwnerUserId="45603" LastEditorUserId="29575" LastEditDate="2018-02-05T12:38:19.410" LastActivityDate="2018-02-05T12:38:19.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27471" PostTypeId="1" AcceptedAnswerId="27490" CreationDate="2018-02-05T11:34:21.933" Score="3" ViewCount="8974" Body="&lt;p&gt;I've been using matlab until now to classify a large number of labelled time series I have. This has been relatively successful but I'd like to try using Tensorflow to apply a Deep Learning paradigm instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm a complete noob at this and so I'm a bit overwhelmed with the literature as I'm struggling to generalise examples such as the 0-9 digits classification examples to my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current code reads in the 1064 time series (length 3125), reads in the labels, converts the labels to onehot_encoding and extracts training and validation sets.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Not all code included&#xA;def read_data():&#xA;    #Get labels from the labels.txt file&#xA;    labels = pd.read_csv('labels.txt', header = None)&#xA;    labels = labels.values&#xA;&#xA;    data = pd.read_csv('ts.txt',header = None)&#xA;&#xA;    return data, labels&#xA;&#xA;data, labels = read_data()&#xA;&#xA;# Default split is 75% train 25% test&#xA;ts_train, ts_test, labels_train, labels_test = train_test_split(data,labels)&#xA;&#xA;onehot_encoder = OneHotEncoder(sparse=False)&#xA;labels_train = onehot_encoder.fit_transform(labels_train)&#xA;labels_test = onehot_encoder.fit_transform(labels_test)&#xA;&#xA;#Construct the graph&#xA;batch_size = 100&#xA;seq_len = 3125&#xA;learning_rate = 0.0001&#xA;epochs = 1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But now I need to construct the graph and I'm a bit lost. If someone could link me some useful examples I'd be very grateful thank you.&lt;/p&gt;&#xA;" OwnerUserId="45766" LastActivityDate="2018-02-05T16:39:53.323" Title="Basic Time Series Classification Examples" Tags="|classification|time-series|tensorflow|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-17T05:56:06.530" ContentLicense="CC BY-SA 3.0" />
  <row Id="27472" PostTypeId="1" CreationDate="2018-02-05T12:53:50.027" Score="0" ViewCount="92" Body="&lt;p&gt;I interested in data mining specialization and have been enrolled in data mining specialization course at Coursera by University of Illinois. This specialization course contains four parts. The very first course is &quot;Data Visualization&quot; which contains more conceptual knowledge rather than technical skills that not as expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anybody can give me suggestion what can I expect after completed that course in the case to find a job in data mining specialization?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I am intended to acquire engineering job, is this course right for me?&lt;/p&gt;&#xA;" OwnerUserId="44018" LastEditorUserId="44083" LastEditDate="2018-02-06T01:12:52.807" LastActivityDate="2018-02-06T01:12:52.807" Title="Need sugestion to accuire a job in data mining" Tags="|data-mining|beginner|" AnswerCount="2" CommentCount="0" ClosedDate="2018-02-06T21:16:44.863" ContentLicense="CC BY-SA 3.0" />
  <row Id="27474" PostTypeId="2" ParentId="27221" CreationDate="2018-02-05T13:24:53.553" Score="1" Body="&lt;p&gt;You can connect the output of the sentiment analysis to &lt;code&gt;Test&amp;amp;Score&lt;/code&gt; widget together with &lt;code&gt;SVM&lt;/code&gt; learner. &lt;code&gt;Test&amp;amp;Score&lt;/code&gt; widget can perform different validations (e.g. cross-validation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gKdrT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gKdrT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to perform testing on different data please use &lt;code&gt;Predictions&lt;/code&gt; widget. For more information please check widgets help available with pressing &lt;code&gt;f1&lt;/code&gt; shortcut in Orange.&lt;/p&gt;&#xA;" OwnerUserId="32745" LastActivityDate="2018-02-05T13:24:53.553" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27475" PostTypeId="2" ParentId="27472" CreationDate="2018-02-05T13:50:23.603" Score="3" Body="&lt;p&gt;Welcome to the site! The course you are taking sounds like a good start; but I am not sure if it will be enough to land you a job as a Data Engineer (depending on your other qualifications, of course).&#xA;If you are willing to invest more time, &lt;a href=&quot;https://www.quora.com/What-should-I-learn-in-data-science-in-100-hours-I-am-free-for-the-next-10-days-and-would-like-to-learn-whatever-I-can-in-the-next-10-days-and-I-can-put-in-10-hours-a-day-What-can-I-learn-to-get-a-hang-of-it-and-get-started?share=cad63442&amp;amp;srid=hRRiT&quot; rel=&quot;nofollow noreferrer&quot;&gt;here is a great text on how to learn Data Science in one hundred hours&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-05T17:19:23.563" LastActivityDate="2018-02-05T17:19:23.563" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27476" PostTypeId="1" CreationDate="2018-02-05T13:57:20.137" Score="5" ViewCount="1779" Body="&lt;p&gt;I am working on project where my task is to find unauthorized access using any machine learning technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me clear my problem definition.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;UserA&lt;/strong&gt; access website using chrome browser from windows PC.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;UserB&lt;/strong&gt; access website using Internet Explorer from windows PC.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;UserA&lt;/strong&gt; never use safari to access website but all of a sudden it uses safari to access website from mac system.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Point 1 and 2 are normal logins but point 3 may be used by attackers because it is not matching with &lt;strong&gt;UserA&lt;/strong&gt; behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus i am am trying to implement an unsupervised model in python to first learn normal behaviors of all users for website access with other features such as -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;username&lt;/li&gt;&#xA;&lt;li&gt;country&lt;/li&gt;&#xA;&lt;li&gt;service_key (unique identifier)&lt;/li&gt;&#xA;&lt;li&gt;system&lt;/li&gt;&#xA;&lt;li&gt;app_type (client, server, enterprise etc.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Below I am sharing sample data-set of mine.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+==========+=========+=============+=========+==========+&#xA;| username | country | service_key | system  | app_type |&#xA;+==========+=========+=============+=========+==========+&#xA;| userA    | india   | e08fe2d     | windows | 2        |&#xA;| userA    | india   | e08fe2d     | android | 2        |&#xA;| usreB    | china   | bb15d36     | windows | 3        |&#xA;| userB    | india   | bb15d36     | windows | 3        |&#xA;| userB    | russia  | bb15d36     | mac     | 3        |&#xA;&#xA;| userA    | usa     | e08fxxx     | mac     | 3        |&#xA;| userB    | china   | bb1xxxx     | ios     | 1        |&#xA;+==========+=========+=============+=========+==========+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Above 5 records are manually analyzed and marked as normal logs and collected from different users system. But the last 2 records are abnormal and should be detected as outlier because their is no match in features. Here to be note is that logins can have different country and system but it cannot have different username, service_key and app_type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest me to model this or can share blog or samples in python?&lt;/p&gt;&#xA;" OwnerUserId="45750" LastEditorUserId="28175" LastEditDate="2018-02-05T14:24:53.273" LastActivityDate="2018-02-05T19:59:18.440" Title="To detect unauthorized access using outlier detection" Tags="|machine-learning|python|unsupervised-learning|anomaly-detection|outlier|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27477" PostTypeId="2" ParentId="27476" CreationDate="2018-02-05T14:08:10.640" Score="2" Body="&lt;p&gt;This is a scenario where you need a meta-dataset. The sample data that you posted is not something that is ready for modeling. I would work towards developing a dataset that would establish the same patterns that you listed in the rest of your post. How many countries have they logged in from? How many times was it during the day? At night? etc. To be clear, this would be a dataset where it's one user ID per row, and each ID only has one record. If you can develop that, then you will be in a much better position to model this out.&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-05T14:08:10.640" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27478" PostTypeId="2" ParentId="27472" CreationDate="2018-02-05T14:09:42.963" Score="0" Body="&lt;p&gt;I like coursera and there are many data science programs to choose from. If I were you, I would look into the 10-course series certificate that is offered by John Hopkins. You will come out of that course being pretty good with R and establishing a good base of knowledge for the rest of your growth as a data scientist. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-05T14:09:42.963" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27480" PostTypeId="2" ParentId="27476" CreationDate="2018-02-05T14:18:12.800" Score="3" Body="&lt;p&gt;I guess you need anomaly detection algorithm. It is like fraud detection for finding abnormal behaviors. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In data mining, anomaly detection (also outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset. Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I highly recommend you taking a look at &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/Mwrni/developing-and-evaluating-an-anomaly-detection-system&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-05T14:18:12.800" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27481" PostTypeId="1" CreationDate="2018-02-05T14:40:36.950" Score="1" ViewCount="256" Body="&lt;p&gt;I have a question about how to create a matrix that will take the lists defined below and make them the entries for the columns and rows. My data involves two lists: one lists originals sources (or papers) and the other lists those sources' individual citations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is done in RStudio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sample of my original dataset looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iPYQz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iPYQz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; # List all the sources&#xA; sources.vertices.levels &amp;lt;- cit.total$Sources&#xA; sources.vertices &amp;lt;- (as.character(sources.vertices.levels))&#xA;&#xA; citation.edges.levels &amp;lt;- cit.total$Citations&#xA; citation.edges &amp;lt;- (as.character(citation.edges.levels))&#xA;&#xA; # Break up the factors into character lists&#xA; list.sources &amp;lt;- levels(sources.vertices.levels)&#xA; list.citations &amp;lt;- levels(citation.edges.levels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Below I have created an empty matrix (50X1497). I would like to make the row names be each individual source, and the columns be each individual citation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I am lost on how to create a binary matrix that matches each individual source (from the rows) to the individual citations (to the columns). &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; # Building an aggregate matrix&#xA; matrix.network &amp;lt;- matrix(cit.citations,ncol=1497, nrow=50)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;An example of what kind of matrix I want to do is below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/txW3g.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/txW3g.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44683" LastEditorUserId="44683" LastEditDate="2018-02-05T15:03:32.110" LastActivityDate="2018-02-05T16:10:09.143" Title="Building a data frame and matrix for citation network" Tags="|r|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27482" PostTypeId="1" AcceptedAnswerId="27485" CreationDate="2018-02-05T15:01:54.487" Score="3" ViewCount="2659" Body="&lt;p&gt;I have a SQL (MS SQL Server) database of ~30 million companies. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+-----------------------+----------------+-----------+&#xA;|     company_name      |    country     | ID_number |&#xA;+-----------------------+----------------+-----------+&#xA;| Mercedes Benz Limited | Germany        |     12345 |&#xA;| Apple Corporation     | United States  |     67899 |&#xA;| Aunt Mary Butcher     | United Kingdom |     56789 |&#xA;+-----------------------+----------------+-----------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I have another list of companies and I would like to assign ID_number based on approximate company name match. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------------------+----------------+&#xA;|      company       |    country     |&#xA;+--------------------+----------------+&#xA;| Mercedes Benz Ltd. | Germany        |&#xA;| Apple Corp.        | United States  |&#xA;| Butcher Aunt Mary  | United Kingdom |&#xA;| Volkswagen Gmbh    | Germany        |&#xA;+--------------------+----------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My goal obviously is to limit the number of comparisons I have to make&lt;/strong&gt;. So I approach it the following way:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Normalize the names - remove Ltd. Corp. etc.&lt;/li&gt;&#xA;&lt;li&gt;Filter by country &lt;/li&gt;&#xA;&lt;li&gt;Filter by name length (I assume that strings of very different length can't be very similar)&lt;/li&gt;&#xA;&lt;li&gt;Filter by first n letters&lt;/li&gt;&#xA;&lt;li&gt;Calculate the similarity with Levenshtein or Jaccard agains all filtered companies&lt;/li&gt;&#xA;&lt;li&gt;Select the best match&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So for &lt;code&gt;Mercedes Benz&lt;/code&gt; I would only take companies from Germany, that start with ME and have length of &lt;code&gt;Mercedes Benz&lt;/code&gt; +-7 letters. However for &lt;code&gt;Aunt Mary Butcher&lt;/code&gt; this wouldn't work because &lt;code&gt;Butcher Aunt Mary&lt;/code&gt; doesn't start with AU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from this being very naive method it's also very slow. I could put more hardware on it but I don't think the method is efficient itself. How do you usually tackle problems like that? &lt;/p&gt;&#xA;" OwnerUserId="45779" LastEditorUserId="45779" LastEditDate="2018-02-05T15:49:26.167" LastActivityDate="2018-02-05T16:01:46.200" Title="compare two lists" Tags="|python|data-mining|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27483" PostTypeId="2" ParentId="27482" CreationDate="2018-02-05T15:16:07.130" Score="0" Body="&lt;p&gt;A solution can be, doing your process, and extract not matched cases. Then, split names on their words, and try to find their distance by Jaccard distance on their word set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In sum, found these cases as an exception, and try to handle them using methods such as mentioned.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-02-05T15:16:07.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27484" PostTypeId="2" ParentId="27329" CreationDate="2018-02-05T15:34:53.773" Score="0" Body="&lt;p&gt;It sounds like your dataset is going to need some work before you can start applying a model to it, for example you will have to translate customer location into something more usable, like distance from depot, or Urban/Suburb/Country labels perhaps?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng's Coursera course&lt;/a&gt; is a popular starting point, many beginners seem to find it useful. You'll get a good understanding of machine learning algorithms and how to arrange your data to get the best out of the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2018-02-05T15:34:53.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27485" PostTypeId="2" ParentId="27482" CreationDate="2018-02-05T15:58:33.367" Score="2" Body="&lt;p&gt;FYI This isn't really a data science question, it's really related more to SQL and should be asked on those forums. But I'm going to try and help you anyways. This is something where you need to do matching by pronunciation using something like SOUNDEX in MySQL (I'm not aware of other RDBMS that offer this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, it's really something that requires an iterative approach where the first pass you're doing straight matches, the next pass you're doing matches with wildcards and then the next pass you use SOUNDEX. In theory, that should leave you with very few (if any) unmatched items for you to do manually. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-05T15:58:33.367" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27486" PostTypeId="2" ParentId="27482" CreationDate="2018-02-05T16:01:46.200" Score="1" Body="&lt;p&gt;Ok,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Disclaimer : I have no knowledge about MS Sql.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Clean the name as you say&lt;/li&gt;&#xA;&lt;li&gt;split each name into words&lt;/li&gt;&#xA;&lt;li&gt;order the result by alphabetical order&lt;/li&gt;&#xA;&lt;li&gt;create a key where the original name is associated to its ordered key&lt;/li&gt;&#xA;&lt;li&gt;newly created key should also linked to the country&lt;/li&gt;&#xA;&lt;li&gt;do this on both tables&lt;/li&gt;&#xA;&lt;li&gt;you should know have a common key to find the appropriate ID_number based on the name and the country.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45762" LastActivityDate="2018-02-05T16:01:46.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27487" PostTypeId="2" ParentId="27481" CreationDate="2018-02-05T16:10:09.143" Score="1" Body="&lt;p&gt;Are the 1s and 0s only to indicate that the source matches the citation? If so, try this :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse); library(magrittr)    &#xA;cit.total %&amp;gt;% &#xA;mutate(score = 1) %&amp;gt;% # so that all 'matches' in your original data frame are tagged '1'&#xA;spread(&quot;Citations&quot;, &quot;score&quot;, fill = 0) # each source and citation pair gets a number, 1 if the match exists, 0 if it doesn't&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="40767" LastActivityDate="2018-02-05T16:10:09.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27490" PostTypeId="2" ParentId="27471" CreationDate="2018-02-05T16:39:53.323" Score="7" Body="&lt;p&gt;What you need to learn are RNNs and LSTMs. Here are some links I used to learn these:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=6niqTuYFZLQ&amp;amp;list=PLzUTmXVwsnXod6WNdg57Yc3zFx_f-RYsq&amp;amp;index=10&quot; rel=&quot;noreferrer&quot;&gt;CS231n RNN+LSTM lecture&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;noreferrer&quot;&gt;Understanding LSTMs&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also I would suggest you to use &lt;a href=&quot;https://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt;, a Tensorflow API. In my experience, it makes working with &lt;a href=&quot;https://keras.io/layers/recurrent/&quot; rel=&quot;noreferrer&quot;&gt;RNNs and LSTMs&lt;/a&gt; way easier, if you're a beginner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found these articles which seemed related to your problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://naveensai.me/project/Multiariate%20TImeSeries%20Classification/&quot; rel=&quot;noreferrer&quot;&gt;Time series classification project by naveen sai on github&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://aboveintelligent.com/time-series-analysis-using-recurrent-neural-networks-lstm-33817fa4c47a&quot; rel=&quot;noreferrer&quot;&gt;https://aboveintelligent.com/time-series-analysis-using-recurrent-neural-networks-lstm-33817fa4c47a&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;noreferrer&quot;&gt;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Keywords to search: Time series classification tensorflow keras&lt;/p&gt;&#xA;" OwnerUserId="20327" LastActivityDate="2018-02-05T16:39:53.323" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27491" PostTypeId="1" AcceptedAnswerId="27507" CreationDate="2018-02-05T17:09:19.010" Score="1" ViewCount="404" Body="&lt;p&gt;I have a collection of 10 features (all numerical) and a single binary outcome variable. I need to train a binary classification model, find the best features and compute thresholds for each feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To find the best subset of features is easy, I use logistic regression with L1 penalisation and it works nicely. However, the next step is to find thresholds for &lt;strong&gt;each&lt;/strong&gt; subselected feature: if the values of a feature &lt;em&gt;i&lt;/em&gt;, &lt;em&gt;j&lt;/em&gt;,...&lt;em&gt;k&lt;/em&gt; are above/below than some numerical values, than chances to be in the class A (rather than in B) are higher. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;FOR EXAMPLE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider, we found that out of 10 original features, there are L1 leaves only 3: F1, F4,F7. And if you take three unseen (new) data with some values {F1_i, F4_i, F7_i}, where &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;F1 &amp;gt; 1.23&#xA;F4 &amp;lt; 7.15&#xA;F7 &amp;gt; 2.74&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then the new data point {F1_i, F4_i, F7_i} belongs to class A. Here {1.23, 7.15, 2.74} are thresholds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to explain the problem as clear as possible, but if it isn't, please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUESTION&lt;/strong&gt; What will be the best approach to solve this problem? How to compute thresholds?&lt;/p&gt;&#xA;" OwnerUserId="14070" LastActivityDate="2018-02-06T05:50:11.617" Title="decision rules for each feature (binary classification)" Tags="|python|classification|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27492" PostTypeId="1" AcceptedAnswerId="27493" CreationDate="2018-02-05T17:44:50.130" Score="5" ViewCount="106" Body="&lt;p&gt;In classical neural nets, we have that each layer is connected only with the following layer.  What if we relaxed this constraint and allowed it to be connected to any or all subsequent layers?  Has this architecture been explored?  It seems to be backprop would still work on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the very least, this type of network could be emulated by artificially creating identity neurons at each layer that bring in the value of every earlier node, where the incoming weights to those layers are fixed at 1.&lt;/p&gt;&#xA;" OwnerUserId="45069" LastActivityDate="2018-02-05T18:39:06.280" Title="Neural Network with Connections to all forward layers" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27493" PostTypeId="2" ParentId="27492" CreationDate="2018-02-05T17:55:12.053" Score="4" Body="&lt;p&gt;What you describe has been explored in Deep Residual Neural Networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A residual block will combine two or more blocks from a standard architecture like a CNN with a skip connection that adds the input to the first block to the output of the last block.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/F50fx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F50fx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuition is that deep networks have a harder and harder time learning the identity function between layers, which has been proven to be useful especially in image recognition tasks. Residual connections also mitigate the problem of &lt;a href=&quot;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;vanishing gradients&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Residual connections help solve the &quot;degradation&quot; problem, where deeper architectures lead to reduced accuracy. For example &lt;a href=&quot;https://research.google.com/pubs/pub43022.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;GoogLeNet&lt;/a&gt; won &lt;a href=&quot;http://www.image-net.org/challenges/LSVRC/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ILSVRC&lt;/a&gt; in 2014 with a &lt;strong&gt;22&lt;/strong&gt;-layer CNN, but in 2015 Microsoft &lt;a href=&quot;https://arxiv.org/abs/1512.03385&quot; rel=&quot;nofollow noreferrer&quot;&gt;ResNet&lt;/a&gt; won with a &lt;strong&gt;152&lt;/strong&gt;-layer Res Net.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-05T18:39:06.280" LastActivityDate="2018-02-05T18:39:06.280" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27494" PostTypeId="2" ParentId="27460" CreationDate="2018-02-05T18:31:21.793" Score="1" Body="&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/45507/elias-strehle&quot;&gt;Elias Strehle&lt;/a&gt; mentioned in the comments, this does not sound like a problem for machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will be able to solve this much faster by identifying a few simple patterns that indicate which type of log you are looking at.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can already see that lines start with date string followed by a few tokens which look different in each log. You can define regex patterns that will only match a certain type of log, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2}&lt;/code&gt; will match your Linux-syslog&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+\d+\s&lt;/code&gt; will match your MySQL-error log&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;^[A-Z]+\d+,&lt;/code&gt; will match your Event log&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}&lt;/code&gt; will match your FTP log&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And &lt;code&gt;^\d{2}/\d{2}/\d{4}&lt;/code&gt; will match your MSSQL Server log.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-08T19:16:39.787" LastActivityDate="2018-02-08T19:16:39.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27495" PostTypeId="1" CreationDate="2018-02-05T18:53:55.097" Score="2" ViewCount="34" Body="&lt;p&gt;I recently wrote &lt;a href=&quot;https://stackoverflow.com/a/47781091/562769&quot;&gt;this&lt;/a&gt; as a list of what I have seen / can think of as problem sources that make it hard to reproduce (replicate?) an experiment. I think I have seen most of them, except for hardware errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thought was this: A typical computer vision experiment might run a training on a GPU for many hours / several days. Even if a bit-flip happens only once every $10^9$ FLOPs, that would be 11500 errors per second with an &lt;a href=&quot;http://www.gamestar.de/artikel/nvidia-geforce-gtx-1080-ti-schneller-als-eine-titan-x-fuer-699-us-dollar,3310460.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nvidia Titan GTX 1080 Ti&lt;/a&gt;. I don't know how this error would affect later calculations (how the problem is numerically conditioned).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So: &lt;strong&gt;Are there any reports on Hardware errors affecting experiments?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Blog posts, journal articles, posters?)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-02-10T13:45:34.123" Title="Are there any reports on Hardware errors affecting experiments?" Tags="|experiments|hardware|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27496" PostTypeId="2" ParentId="27468" CreationDate="2018-02-05T19:11:24.283" Score="4" Body="&lt;p&gt;The problem type you're dealing with is referred to as multiclass classification. Not all algorithms are suited to handle it, but tree based methods and neural networks are popular choices. If you need it to run quickly and probability calibration isn't too important, Naive Bayes also works quite well for some data sets. To see an example of a dataset of this type, check of the &lt;a href=&quot;https://www.kaggle.com/c/spooky-author-identification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Spooky Author Identification competition&lt;/a&gt;. The published kernels give some good examples of feature engineering and modeling choices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I'm assuming the category label is unique. If there can be more than one label per record, it's called multilabel classification, which I would handle by building a separate binary model for each of the 39 labels in your data set. For an example dataset, check out the &lt;a href=&quot;https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Toxic Comments competition&lt;/a&gt;.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for modeling with your data in specific, the structure you have it in now seems a bit odd to feed to a model. A training data set should have each row represent one record, and each column represent a feature with the value in the column describing the record, whereas in your format, any cell phone should have information regarding each value in the corresponding mobile Attribute_Names. When you get the raw input '&lt;strong&gt;Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display&lt;/strong&gt;', how is that transformed into a format that can be fed into the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, where are the Attribute_Names, Attribute_Values coming from, are they manually specified? If so, that is limiting the performance potential from a model since there could be additional words in the data the model could detect if left to generate features on it's own. For a good modeling flow, the training data should contain the inputted raw text, then process the text to generate features, then feed into the actual model to output a label. So one row of raw training data set would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;------------------------------------------------------------------------&#xA;| Description                                            |   Category  |&#xA;------------------------------------------------------------------------&#xA;| 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable   | Mobile      |&#xA;| Upto 256 GB 5.5 inch Full HD Display'                  |             |&#xA;------------------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You would then turn the text into features using some sklearn pre-processors. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;Count Vectorizer&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF Vectorizer&lt;/a&gt; are popular choices. You can also create your own features from keywords from the Attribute key and values you already have, such as creating a Brand indicator column if the words Samsung, Nokia or OnePlus appear in the text, but I would never use only manually specified features when modeling with text features. &lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-05T19:11:24.283" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27497" PostTypeId="2" ParentId="27476" CreationDate="2018-02-05T19:36:14.053" Score="4" Body="&lt;p&gt;This question is quite broad. I'll try to &lt;em&gt;set you on the right path&lt;/em&gt;, more so than providing a truly complete answer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Theoretical background&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As others have mentioned, the task you're trying to do is usually known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Anomaly_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;anomaly detection&lt;/strong&gt;&lt;/a&gt;, also known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Novelty_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;novelty detection&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's many possible ways to approach this kind of task, depending on the assumptions you are willing to make about your data. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Before we start exploring alternatives, it's important to mention here that &lt;strong&gt;all your variables  are &lt;a href=&quot;https://en.wikipedia.org/wiki/Categorical_variable&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical&lt;/a&gt;&lt;/strong&gt; (country, service_key, system, app_type). This is important, since usually algorithms usually handle either &lt;strong&gt;continuous variables&lt;/strong&gt; or &lt;strong&gt;discrete variables&lt;/strong&gt;, and categorical ones fall into the latter type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's some tricks you can do, however, to handle them interchangeably. &lt;a href=&quot;https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;One-hot encoding&lt;/strong&gt;&lt;/a&gt; is simple and popular, and &lt;strong&gt;enables you to transform a categorical variable into N binary variables&lt;/strong&gt;. For some classes of algorithms, you can then use these binary variables as if they were continuous. &lt;a href=&quot;https://stats.stackexchange.com/questions/82923/mixing-continuous-and-binary-data-with-linear-svm&quot;&gt;You can even mix binary and truly continuous variables&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now, there's usually two families of methods to approach this sort of problem: &lt;a href=&quot;http://www.mayo.edu/mayo-edu-docs/center-for-translational-science-activities-documents/berd-5-6.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;parametric&lt;/strong&gt; vs &lt;strong&gt;non-parametric&lt;/strong&gt;&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parametric methods require that you do assumptions on the underlying &lt;a href=&quot;https://en.wikipedia.org/wiki/Probability_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;probability distributions&lt;/a&gt; of your data. You can then estimate these parameters using your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-parametric methods, on the other hand, are simpler to use, as they don't require assuming a underlying distribution. They might also be less powerful / exact, especially if you don't have a lot of data and your data very closely follows a known probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Practical advise&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're willing to do some statistical work, look into &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_inference&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian inference&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a easier non-parametric method, look into into One-Class SVM. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is a code example using python and scikit-learn&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;more practical information on anomaly detection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, honestly, you're giving us a important clue here:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here to be note is that logins can have different country and system&#xA;  but it cannot have different username, service_key and app_type.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you already know all this, you might want to completely skip the complicated parts. Just compare those values directly, summarize them into a distance measure (you can give different weights to service_key and app_type, for example) and see if that's good enough&lt;/p&gt;&#xA;" OwnerUserId="45803" LastEditorUserId="45803" LastEditDate="2018-02-05T19:59:18.440" LastActivityDate="2018-02-05T19:59:18.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27498" PostTypeId="1" AcceptedAnswerId="27530" CreationDate="2018-02-05T19:56:47.897" Score="15" ViewCount="12281" Body="&lt;p&gt;I am trying to implement shared layers in Keras. I do see that Keras has &lt;code&gt;keras.layers.concatenate&lt;/code&gt;, but I am unsure from documentation about its use. Can I use it to create multiple shared layers? What would be the best way to implement a simple shared neural network as shown below using Keras? &lt;a href=&quot;https://dam-prod.media.mit.edu/thumb/2017/12/12/Screen%20Shot%202017-12-12%20at%2011.50.03%20AM.png.1400x1400.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://dam-prod.media.mit.edu/thumb/2017/12/12/Screen%20Shot%202017-12-12%20at%2011.50.03%20AM.png.1400x1400.png&quot; alt=&quot;Shared Neural network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that all the shape of input, output and shared layers for all 3 NNs are the same. There are multiple shared layers (and non-shared layers) in the three NNs. The coloured layers are unique to each NN, and have same shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the figure represents 3 identical NNs with multiple shared hidden layers, followed by multiple non-shared hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am unsure how to share multiple layers as in the Twitter example, there was just one shared layer (example in API doc).&lt;/p&gt;&#xA;" OwnerUserId="45804" LastActivityDate="2018-02-06T15:52:15.133" Title="Multi task learning in Keras" Tags="|machine-learning|neural-network|deep-learning|keras|multitask-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27499" PostTypeId="2" ParentId="9681" CreationDate="2018-02-05T20:55:43.743" Score="4" Body="&lt;p&gt;RStudio seems to recognize subheaders when they lie within functions. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# SECTION ONE -----------------------------------------------------------------&#xA;&#xA;testfunc &amp;lt;- function(input1,input2,input3){&#xA;&#xA;# SUBSECTION ONE --------------------------------------------------------------&#xA;# SUBSECTION TWO --------------------------------------------------------------&#xA;  ss2func &amp;lt;- function(x1,x2,x3){&#xA;&#xA;  }&#xA;}&#xA;&#xA;# SECTION TWO -----------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45809" LastEditorUserId="45809" LastEditDate="2018-02-05T21:06:09.017" LastActivityDate="2018-02-05T21:06:09.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27500" PostTypeId="2" ParentId="27495" CreationDate="2018-02-05T21:21:43.713" Score="1" Body="&lt;p&gt;Memory corruption seems to be an issue that is important enough for companies to buy expensive ECC memory for their clusters.&#xA;The &lt;a href=&quot;https://en.wikipedia.org/wiki/ECC_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia article on ECC memory&lt;/a&gt; lists some causes for memory corruption, including (to my surprise and delight) cosmic rays and ingenious hackers.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-05T21:21:43.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27501" PostTypeId="1" AcceptedAnswerId="27503" CreationDate="2018-02-05T22:00:50.253" Score="6" ViewCount="20248" Body="&lt;p&gt;When hierarchical clustering is preferred over k means clustering?&lt;/p&gt;&#xA;" OwnerUserId="45814" LastEditorUserId="105060" LastEditDate="2020-09-22T07:02:09.930" LastActivityDate="2020-09-22T07:02:09.930" Title="K-Means vs hierarchical clustering" Tags="|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-12T11:34:38.950" ContentLicense="CC BY-SA 4.0" />
  <row Id="27502" PostTypeId="1" CreationDate="2018-02-06T04:36:44.793" Score="1" ViewCount="1572" Body="&lt;p&gt;If we randomly split the data into training data and validation data, and assume the training data and validation data have similar &quot;distributions&quot;, i.e. they are both good representations of the whole data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, should the validation accuracy always be roughly the same as the training accuracy if there is no overfitting? Or is it possible that, for some cases, there could exist an &quot;intrinsic&quot; gap between the training and validation accuracy that is not due to overfitting or bad representation of the validation data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If such &quot;intrinsic&quot; gap exists, how to tell if the gap between the training and validation accuracy is &quot;intrinsic&quot; or caused by overfitting?&lt;/p&gt;&#xA;" OwnerUserId="42599" LastActivityDate="2018-08-06T21:29:49.803" Title="validation/training accuracy and overfitting" Tags="|machine-learning|cross-validation|training|accuracy|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27503" PostTypeId="2" ParentId="27501" CreationDate="2018-02-06T05:21:48.300" Score="10" Body="&lt;p&gt;I would say hierarchical clustering is usually preferable, as it is both more flexible and has fewer hidden assumptions about the distribution of the underlying data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With k-Means clustering, you need to have a sense ahead-of-time what your desired number of clusters is (this is the 'k' value).  Also, k-means will often give unintuitive results if (a) your data is not well-separated into sphere-like clusters, (b) you pick a 'k' not well-suited to the shape of your data, i.e. you pick a value too high or too low, or (c) you have weird initial values for your cluster centroids (one strategy is to run a bunch of k-means algorithms with random starting centroids and take some common clustering result as the final result).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, hierarchical clustering has fewer assumptions about the distribution of your data - the only requirement (which k-means also shares) is that a distance can be calculated each pair of data points.  Hierarchical clustering typically 'joins' nearby points into a cluster, and then successively adds nearby points to the nearest group.  You end up with a 'dendrogram', or a sort of connectivity plot. You can use that plot to decide after the fact of how many clusters your data has, by cutting the dendrogram at different heights.  Of course, if you need to pre-decide how many clusters you want (based on some sort of business need) you can do that too.  Hierarchical clustering can be more computationally expensive but usually produces more intuitive results.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastEditorUserId="45929" LastEditDate="2019-02-25T04:48:09.867" LastActivityDate="2019-02-25T04:48:09.867" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27504" PostTypeId="2" ParentId="27502" CreationDate="2018-02-06T05:30:09.663" Score="3" Body="&lt;p&gt;I'm not sure what you mean by 'intrinsic'.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A higher training accuracy means you have found a model of the training data that doesn't describe the validation data equally well.  Because of this, any statistically significant difference in training-validation accuracy is due to overfitting.   Figuring out what 'statistically significant' means in this context is not totally trivial, unfortunately, but the general point still stands.  &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-02-06T05:30:09.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27505" PostTypeId="1" AcceptedAnswerId="27524" CreationDate="2018-02-06T05:30:55.430" Score="3" ViewCount="730" Body="&lt;p&gt;Is there an R build tool (like Maven or Gradle for Java) to get the dependencies and package an R project?&lt;/p&gt;&#xA;" OwnerUserId="45827" LastEditorUserId="45827" LastEditDate="2018-02-08T00:44:33.503" LastActivityDate="2018-02-08T00:44:33.503" Title="Is there an R build tool like Maven?" Tags="|r|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-06T21:15:00.567" ContentLicense="CC BY-SA 3.0" />
  <row Id="27506" PostTypeId="1" AcceptedAnswerId="27751" CreationDate="2018-02-06T05:38:31.077" Score="26" ViewCount="29303" Body="&lt;p&gt;I have the following CNN:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MehcI.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MehcI.png&quot; alt=&quot;network layour&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I start with an input image of size 5x5&lt;/li&gt;&#xA;&lt;li&gt;Then I apply convolution using 2x2 kernel and stride = 1, that produces feature map of size 4x4.&lt;/li&gt;&#xA;&lt;li&gt;Then I apply 2x2 max-pooling with stride = 2, that reduces feature map to size 2x2. &lt;/li&gt;&#xA;&lt;li&gt;Then I apply logistic sigmoid.&lt;/li&gt;&#xA;&lt;li&gt;Then one fully connected layer with 2 neurons.&lt;/li&gt;&#xA;&lt;li&gt;And an output layer.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the sake of simplicity, let's assume I have already completed the forward pass and computed &lt;strong&gt;&amp;delta;H1=0.25&lt;/strong&gt; and  &lt;strong&gt;&amp;delta;H2=-0.15&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after the complete forward pass and partially completed backward pass my network looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wE1He.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wE1He.png&quot; alt=&quot;network after forward pass&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I compute deltas for non-linear layer (logistic sigmoid):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align}&#xA;&amp;amp;\delta_{11}=(0.25 * 0.61 + -0.15 * 0.02) * 0.58 * (1 - 0.58) = 0.0364182\\&#xA;&amp;amp;\delta_{12}=(0.25 * 0.82 + -0.15 * -0.50) * 0.57 * (1 - 0.57) = 0.068628\\&#xA;&amp;amp;\delta_{21}=(0.25 * 0.96 + -0.15 * 0.23) * 0.65 * (1 - 0.65) = 0.04675125\\&#xA;&amp;amp;\delta_{22}=(0.25 * -1.00 + -0.15 * 0.17) * 0.55 * (1 - 0.55) = -0.06818625\\&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I propagate deltas to 4x4 layer and set all the values which were filtered out by max-pooling to 0 and gradient map look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aaEQ9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aaEQ9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I update kernel weights from there? And if my network had another convolutional layer prior to 5x5, what values should I use to update it kernel weights? And overall, is my calculation correct?&lt;/p&gt;&#xA;" OwnerUserId="45821" LastEditorUserId="29169" LastEditDate="2021-02-13T16:42:41.347" LastActivityDate="2021-02-13T16:42:41.347" Title="back propagation in CNN" Tags="|machine-learning|cnn|convolutional-neural-network|backpropagation|kernel|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27507" PostTypeId="2" ParentId="27491" CreationDate="2018-02-06T05:50:11.617" Score="4" Body="&lt;p&gt;You are describing every binary classifier.  However, you are missing a key point. If your classes are separable by the value of just ONE feature, you can do what you're saying and find e.g. F1 &gt; 1.23 as a threshold.  If classification involves a &lt;em&gt;combination&lt;/em&gt; of features, you will need to describe some combination of thresholds for each feature, or (equivalently) some relationship between the features that tells you about the class label.  It's the job of every binary classifier to do exactly this - they just do it in different ways.  See for example &lt;a href=&quot;http://tjo-en.hatenablog.com/entry/2014/01/06/234155&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post.&lt;/a&gt;  Your desire to have a combination of fixed sets of thresholds will only work if you can have a set of thresholds that will encompass/describe/classify every combination of feature values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a set of easy-to-read thresholds like you mention, you should read about &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision tree classifiers&lt;/a&gt;.  They'll do something like what you want - but will also ensure that you provide a class label for every possible combination fo features values. The nice about about decision trees is that they'll let you leave out your current feature selection step - they just do it for you by (1) picking the feature that best discriminates the two classes overall, (2) picking the threshold value of that feature that gives the most information about the class label (usually), and (3) repeating (1-2) several times.   &lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-02-06T05:50:11.617" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27509" PostTypeId="2" ParentId="27234" CreationDate="2018-02-06T09:18:36.330" Score="1" Body="&lt;p&gt;The essential transformation in @AN6U5's answer ist done in the following lines:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;df['hourfloat']=df.hour+df.minute/60.0&lt;br&gt;&#xA;  df['x']=np.sin(2.&lt;em&gt;np.pi&lt;/em&gt;df.hourfloat/24.)&lt;br&gt;&#xA;  df['y']=np.cos(2.&lt;em&gt;np.pi&lt;/em&gt;df.hourfloat/24.)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;in the first line he transforms minutes into hours by dividing them by 60&#xA;so for example 20 Minutes are converted to 0.3333 hours&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, in line 2 and 3 he converts this float number from polar coordinates to cartesian coordinates (&lt;a href=&quot;https://en.wikipedia.org/wiki/Polar_coordinate_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Polar_coordinate_system&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So chaning this from hour to weekday you just need to adapt the first line.&lt;br&gt;&#xA;Imagining a a clock where 00:00 is Monday, followed by Tuesday (clockwise), and so on ... you need to convert hour into weekday (for simplicity I assume weekday has the values 0-7). So first you divide your hour by 24 which transforms it into days and then further you divide it again by 7 which gives you a float number in weeks. Then add your weekday to the hour and then proceed with line 2 and 3 just as given, except that you correct the 24 to 7.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a formula: hour/(24*7)+weekday = weekfloat&#xA;I haven't tried it out by myself but I think this should do it.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Alternatively, when you have two cyclic features you could transform weekday and hour into &lt;a href=&quot;https://en.wikipedia.org/wiki/Spherical_coordinate_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;spherical coordinates&lt;/a&gt;. This would leave you with three coordinates, x,y and z but also preserves the 'closeness' within one feature itself.&lt;/p&gt;&#xA;" OwnerUserId="43367" LastEditorUserId="43367" LastEditDate="2018-02-09T14:20:03.030" LastActivityDate="2018-02-09T14:20:03.030" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27511" PostTypeId="1" AcceptedAnswerId="27512" CreationDate="2018-02-06T10:12:19.673" Score="2" ViewCount="70" Body="&lt;p&gt;In a document on deep learning about auto encoders, it is said that these networks were used back from 2006 to 2010 for deep neural networks initialization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody explain how this was done?&lt;/p&gt;&#xA;" OwnerUserId="40515" LastActivityDate="2018-02-06T11:02:57.577" Title="How were auto encoders used to intialize deep neural networks?" Tags="|deep-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27512" PostTypeId="2" ParentId="27511" CreationDate="2018-02-06T10:36:59.423" Score="3" Body="&lt;p&gt;There were a few different techniques. One popular one was stacked autoencoders, where each layer was trained separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially this was done by progressively growing the autoencoder, two layers at a time (one encode layer, plus equivalent decode layer), followed by complete training at each step of growth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If learning from a fixed training set, you could store the encoded representation of the whole dataset so far as input into next stage of training, saving some computation when building up the layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After training each encoder layer separately you could use the weights of the encoder section of the autoencoder as the starting weights of the deep NN. Intuitively this made sense as you would have a representation of the input that you knew could be used to reconstruct it, and that typically was compressed, so should in theory have extracted salient details from the training data population. On top of these pre-trained layers, you may add one or two new layers that implemented whatever classification or regression task that you needed the final NN to perform. Then you would train with the labelled data - this is similar to fine-tuning networks and transfer learning that is still done nowadays.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The results from this pre-training stage could be worthwhile. It is still a valid technique if you have a lot of unlabelled data, and a relatively small amount of labelled data. However, the introduction of ReLU and careful controls on weight initialisation meant that deep networks could often be trained more directly. Recent additions such as skip connections and batch normalisation have further improved more direct training approaches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cmgreen.io/2016/01/04/tensorflow_deep_autoencoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is an example with code, using TensorFlow&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-02-06T11:02:57.577" LastActivityDate="2018-02-06T11:02:57.577" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27514" PostTypeId="1" CreationDate="2018-02-06T12:18:34.213" Score="2" ViewCount="422" Body="&lt;p&gt;I have N models, each of them are used to predict on a set of data.&#xA;I am currently combining their predictions by averaging across rows.&#xA;Need suggestions on combining their predictions. End goal is to classify the samples. &lt;/p&gt;&#xA;" OwnerUserId="26329" LastEditorUserId="29575" LastEditDate="2018-02-07T13:19:42.847" LastActivityDate="2018-02-07T13:19:42.847" Title="Combining Different Models" Tags="|machine-learning|ensemble-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27516" PostTypeId="2" ParentId="27514" CreationDate="2018-02-06T13:12:17.677" Score="2" Body="&lt;p&gt;Is it a classification problem or a regression problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use average you assume equal weight for each model. You can use cross validation to identify what weights work best for you.&lt;/p&gt;&#xA;" OwnerUserId="20327" LastActivityDate="2018-02-06T13:12:17.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27519" PostTypeId="1" CreationDate="2018-02-06T13:39:20.757" Score="2" ViewCount="1441" Body="&lt;p&gt;I have a df with many columns that represent the market cap of companies that compose an index. The index of the dataframe is dates. Before the company enters the index or after it leaves it, the market cap of the company in the df is zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know the mean number of days in which the index doesn't change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[in]: &#xA;df = pd.DataFrame(np.array([[1, 1,np.nan], [np.nan,2, 10], [1,3, 100],[4,np.nan, 100]]), &#xA;                           columns=['a', 'b','c'])&#xA;&#xA;df&#xA;[out]:  &#xA;        a   b   c    &#xA;    0   1.0 1.0 NaN    &#xA;    1   NaN 2.0 10.0    &#xA;    2   1.0 3.0 100.0    &#xA;    3   4.0 NaN 100.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;what I want to know is how many rows have entries from the same column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, row 0 has entries from columns a and b. row 1 from columns b and c. row 2 from columns a,b and c. an row 3 from columns a and c. therefore there are 4 rows with unique column combinations and 4 changes. The mean is then 1.&lt;/p&gt;&#xA;" OwnerUserId="45845" LastEditorUserId="45845" LastEditDate="2018-02-06T15:24:47.450" LastActivityDate="2018-02-06T15:37:15.067" Title="how many rows have values from the same columns pandas" Tags="|pandas|dataframe|finance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27520" PostTypeId="1" AcceptedAnswerId="27522" CreationDate="2018-02-06T13:50:48.193" Score="2" ViewCount="2648" Body="&lt;p&gt;My thesis topic is about building a (deep) neural-network classifier to classify the type of a place. I am given both labels and some inputs in string type. So for example the label &quot;Supermarket&quot; might have a feature like &quot;Food&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I feed my string input features to the neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, is there any efficient way to substitute those string inputs with numerical values knowing that there are A LOT of them?&lt;/p&gt;&#xA;" OwnerUserId="45033" LastEditorUserId="29575" LastEditDate="2018-02-06T14:55:18.217" LastActivityDate="2018-02-06T14:55:18.217" Title="Training Neural network classifier using string inputs" Tags="|classification|deep-learning|data-cleaning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27521" PostTypeId="1" AcceptedAnswerId="37989" CreationDate="2018-02-06T13:52:54.357" Score="1" ViewCount="36" Body="&lt;p&gt;I'm new with Octave and I need to plot this function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;% Computes the value of the Powell Sum benchmark function.&#xA;% SCORES = POWELLSUMFCN(X) computes the value of the Powell Sum function at &#xA;% point X. POWELLSUMFCN accepts a matrix of size M-by-N and returns a vetor &#xA;% SCORES of size M-by-1 in which each row contains the &#xA;% function value for the corresponding row of X. &#xA;% &#xA;% Author: Mazhar Ansari Ardeh&#xA;% Please forward any comments or bug reports to mazhar.ansari.ardeh at&#xA;% Google's e-mail service or feel free to kindly modify the repository.&#xA;function scores = powellsumfcn(x)&#xA;    n = size(x, 2);&#xA;    absx = abs(x);&#xA;&#xA;    scores = 0;&#xA;    for i = 1:n&#xA;        scores = scores + (absx(:, i) .^ (i + 1));&#xA;    end&#xA;end&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have copied from &lt;a href=&quot;http://benchmarkfcns.xyz/benchmarkfcns/powellsumfcn.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;BenchmarkFcns&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have no idea about how to plot it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I need to pass all the possible values to the function plot it?&lt;/p&gt;&#xA;" OwnerUserId="31726" LastActivityDate="2018-09-09T08:27:50.213" Title="Plot 2D function: do I need to pass all the values to the function?" Tags="|plotting|octave|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27522" PostTypeId="2" ParentId="27520" CreationDate="2018-02-06T13:54:38.703" Score="1" Body="&lt;p&gt;You always avoid feeding direct strings into neural networks. This thread here explains why you should avoid doing this : &lt;a href=&quot;https://datascience.stackexchange.com/questions/869/neural-network-parse-string-data&quot;&gt;Neural Network parse string data?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you convert the strings you have into vectors or any other form of numerical representation and encoding your labels as categorical, it will solve the problem you have at hand. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need me to elaborate more on this, I would be more than happy to do so.&lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-02-06T13:54:38.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27523" PostTypeId="2" ParentId="27520" CreationDate="2018-02-06T14:16:46.057" Score="1" Body="&lt;p&gt;First thing that comes to my mind is one-hot encoding but if you say that you have so many different strings and if you want to capture the semantics within the encoding step you should consider word embeddings.&lt;/p&gt;&#xA;" OwnerUserId="38355" LastActivityDate="2018-02-06T14:16:46.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27524" PostTypeId="2" ParentId="27505" CreationDate="2018-02-06T14:32:21.767" Score="3" Body="&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;https://rstudio.github.io/packrat/&quot; rel=&quot;nofollow noreferrer&quot;&gt;packrat&lt;/a&gt;&lt;/strong&gt; package is what you're looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It bypasses R's native packages and allows you to build and deploy a bundle of packages and dependencies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it does not do Maven-style dynamic dependency resolution; you need to specify a list of packages that you believe will work together, then the bundle is installed/deployed as a single unit.&lt;/p&gt;&#xA;" OwnerUserId="45615" LastActivityDate="2018-02-06T14:32:21.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27527" PostTypeId="2" ParentId="27514" CreationDate="2018-02-06T15:15:50.907" Score="4" Body="&lt;p&gt;Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying to build an ensemble on top of it might lead to worse performance in deployment since the model is only memorizing the training data. In that case, a simple average might do you better than trying to anything more sophisticated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, assuming you have more data to use, or if you want to use the data you trained on anyways, you can either design a weighted average or create a second model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have an idea of which model performs better, you might want to try manually assigning weights for each model's output. A simple average assigns an equal weight to each output, but you can experiment with shifting the weight around a bit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For an example, given two models, a basic average yields &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_pred =  0.5 * mod_1  + 0.5 * mod_2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, if you see the performance on model one does better overall, you might shift the average to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_pred =  0.9 mod_1  + 0.1 mod_2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If instead you want to create a second model, you can try using any machine learning model suitable for classification.  I would probably stay away from larger models like Random Forests, but doing something like using the predictions as features into a Logistic Regression model can work well. But again, I would emphasize only trying to use the predictions from a holdout data set into the logistic regression model to make sure you aren't over fitting.&lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-06T15:15:50.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27528" PostTypeId="2" ParentId="27519" CreationDate="2018-02-06T15:29:36.400" Score="2" Body="&lt;p&gt;By building a tuple of whether a value is &lt;code&gt;nan&lt;/code&gt; or not, you can then construct each unique like:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_average_run_length(a_df):&#xA;    unique_run_lengths = set(&#xA;        tuple(not np.isnan(x) for x in row[1]) for row in df.iterrows())&#xA;    return len(a_df) / len(unique_run_lengths)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Test Code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame(&#xA;    [[1, 1, np.nan], [np.nan, 2, 10], [1, 3, 100], [4, np.nan, 100]],&#xA;    columns=list('abc'))&#xA;&#xA;print(df)&#xA;print(get_average_run_length(df))&#xA;&#xA;# double the last row&#xA;df = pd.concat([df, df.tail(1)])&#xA;print(df)&#xA;print(get_average_run_length(df))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Results:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     a    b      c&#xA;0  1.0  1.0    NaN&#xA;1  NaN  2.0   10.0&#xA;2  1.0  3.0  100.0&#xA;3  4.0  NaN  100.0&#xA;&#xA;1.0&#xA;&#xA;     a    b      c&#xA;0  1.0  1.0    NaN&#xA;1  NaN  2.0   10.0&#xA;2  1.0  3.0  100.0&#xA;3  4.0  NaN  100.0&#xA;3  4.0  NaN  100.0&#xA;&#xA;1.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29575" LastActivityDate="2018-02-06T15:29:36.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27529" PostTypeId="2" ParentId="27519" CreationDate="2018-02-06T15:37:15.067" Score="1" Body="&lt;p&gt;I am not entirely sure if I understand your question correctly. What result do you want for the following dataframe: &lt;code&gt;2/5&lt;/code&gt; because there are 2 different NaN-combinations in 5 rows, or &lt;code&gt;4/5&lt;/code&gt; because there are 4 changes in the index composition in 5 rows?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    a   b&#xA;0   1   NaN&#xA;1   NaN 2&#xA;2   3   NaN&#xA;3   NaN 4&#xA;4   5   NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Stephen Rauch's answer covers the first scenario.&#xA;This one should cover the second:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;df = pd.DataFrame(np.array([[1, 2, np.nan],&#xA;                            [3, 4, np.nan],&#xA;                            [6, np.nan, 7],&#xA;                            [8, np.nan, np.nan]]))&#xA;&#xA;df_na = np.array(df.isna()[:-1])&#xA;df_na_shifted = np.array(df.isna()[1:])&#xA;&#xA;number_of_good_cols = (df_na == df_na_shifted).sum(axis=1)&#xA;&#xA;number_of_good_rows = sum(number_of_good_cols == df.shape[1])&#xA;print('The index changed on %d days.' % (df.shape[0] - 1 - number_of_good_rows))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-06T15:37:15.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27530" PostTypeId="2" ParentId="27498" CreationDate="2018-02-06T15:52:15.133" Score="12" Body="&lt;p&gt;By using the functional API you can easily share weights between different parts of your network. In your case we have an &lt;code&gt;Input&lt;/code&gt; $x$ which is our input, then we will have a &lt;code&gt;Dense&lt;/code&gt; layer called shared. Then we will have three different &lt;code&gt;Dense&lt;/code&gt; layers called sub1, sub2 and sub3 and then three output layers called out1, out2 and out3.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = Input(shape=(n, ))&#xA;shared = Dense(32)(x)&#xA;sub1 = Dense(16)(shared)&#xA;sub2 = Dense(16)(shared)&#xA;sub3 = Dense(16)(shared)&#xA;out1 = Dense(1)(sub1)&#xA;out2 = Dense(1)(sub2)&#xA;out3 = Dense(1)(sub3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can define our model now like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Model(inputs=x, outputs=[out1, out2, out3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It will expect a tuple/list of three elements now, one for each output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go a lot further with these concepts. Let's say we would like to learn individual weights for the person layers but still want to have the same weights for the linear combination towards the output layer, we could achieve that by doing this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;out = Dense(1)&#xA;out1 = out(sub1)&#xA;out2 = out(sub2)&#xA;out3 = out(sub3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT: Concatenating is basically the opposite of what you want to do, it's pasting (intermediate) outputs of different parts of your network into a new layer. You actually want to split off to multiple different parts.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-06T15:52:15.133" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27533" PostTypeId="1" AcceptedAnswerId="27535" CreationDate="2018-02-06T16:33:32.570" Score="11" ViewCount="20955" Body="&lt;p&gt;I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's &lt;em&gt;Deep Learning for Python&lt;/em&gt;. The data set is ~1000 Time Series with length 3125 with 3 potential classes.&lt;/p&gt;&#xA;&lt;p&gt;I'd like to go beyond the basic &lt;em&gt;Dense&lt;/em&gt; layers which give me about 70% prediction rate and the book goes on to discuss LSTM and RNN layers.&lt;/p&gt;&#xA;&lt;p&gt;All the examples seem to use datasets with multiple features for each timeseries and I'm struggling to work out how to implement my data as a result.&lt;/p&gt;&#xA;&lt;p&gt;If for example, I have 1000x3125 Time Series, how do I feed that into something like the SimpleRNN or LSTM layer? Am I missing some fundamental knowledge of what these layers do?&lt;/p&gt;&#xA;&lt;h3&gt;Current code:&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import os&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.layers import LSTM, Dropout, SimpleRNN, Embedding, Reshape&#xA;from keras.utils import to_categorical&#xA;from keras import regularizers&#xA;from sklearn.model_selection import train_test_split&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def readData():&#xA;    # Get labels from the labels.txt file&#xA;    labels = pd.read_csv('labels.txt', header = None)&#xA;    labels = labels.values&#xA;    labels = labels-1&#xA;    print('One Hot Encoding Data...')&#xA;    labels = to_categorical(labels)&#xA;&#xA;    data = pd.read_csv('ts.txt', header = None)&#xA;&#xA;    return data, labels&#xA;&#xA;print('Reading data...')&#xA;data, labels = readData()&#xA;&#xA;print('Splitting Data')&#xA;data_train, data_test, labels_train, labels_test = train_test_split(data, labels)&#xA;&#xA;print('Building Model...')&#xA;#Create model&#xA;model = Sequential()&#xA;## LSTM / RNN goes here ##&#xA;model.add(Dense(3, activation='softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA; &#xA;print('Training NN...')&#xA;history = model.fit(data_train, labels_train, epochs=1000, batch_size=50,&#xA;    validation_split=0.25,verbose=2)&#xA;&#xA;results = model.evaluate(data_test, labels_test)&#xA;&#xA;predictions = model.predict(data_test)&#xA;&#xA;print(predictions[0].shape)&#xA;print(np.sum(predictions[0]))&#xA;print(np.argmax(predictions[0]))&#xA;&#xA;print(results)&#xA;&#xA;acc = history.history['acc']&#xA;val_acc = history.history['val_acc']&#xA;epochs = range(1, len(acc) + 1)&#xA;&#xA;plt.plot(epochs, acc, 'bo', label='Training acc')&#xA;plt.plot(epochs, val_acc, 'b', label='Validation acc')&#xA;plt.title('Training and Validation Accuracy')&#xA;plt.xlabel('Epochs')&#xA;plt.ylabel('Loss')&#xA;plt.legend()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45766" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-10-07T15:02:11.080" Title="Keras LSTM with 1D time series" Tags="|python|deep-learning|time-series|lstm|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27534" PostTypeId="1" CreationDate="2018-02-06T16:53:27.430" Score="1" ViewCount="1260" Body="&lt;p&gt;I am trying to create a simple Autoencoder to select features based on a high dimensional dataset. I can understand, how AE works and how AE will reduce the dimensionality but I am looking for a sample or tutorial for the AE feature selection which is implemented based on the Keras DL tool. I would really appreciate your help on this.&lt;/p&gt;&#xA;" OwnerUserId="45859" LastActivityDate="2018-02-06T16:53:27.430" Title="How to create a simple Autoencoder to select features from high dimensional dataset" Tags="|feature-selection|autoencoder|" AnswerCount="0" CommentCount="3" ClosedDate="2018-02-18T13:57:52.643" ContentLicense="CC BY-SA 3.0" />
  <row Id="27535" PostTypeId="2" ParentId="27533" CreationDate="2018-02-06T17:34:42.257" Score="11" Body="&lt;p&gt;LSTM layers require data of a different shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should then have 3125 rows and 1 column, where each value can be one of three possible values. So it sounds like you're doing a classification problem. To check this in code, I would do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; X.shape&#xA;(3125, 1000)&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; y.shape&#xA;(1000,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The LSTM class requires each single sample to consist of a 'block' of time. Let's say you want to have a block of 100 time-steps. This means &lt;code&gt;X[0:100]&lt;/code&gt; is a single input sample, which corresponds to the target variable at &lt;code&gt;y[100]&lt;/code&gt;. this means your window size (a.k.a number of time-steps or number of lags) is equal to 100. As stated above, you have 3125 samples, so &lt;code&gt;N = 3125&lt;/code&gt;. To form the first block, we unfortunately have to discard the first 100 samples of &lt;code&gt;y&lt;/code&gt;, as we cannot form an entire block of 100 from the available data (we would end up needing the data points before &lt;code&gt;X[0]&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given all this, an LSTM requires you to deliver batches of shape &lt;code&gt;(N - window_size, window_size, num_features)&lt;/code&gt;, which translates into &lt;code&gt;(3125 - 100, 100, 1000)&lt;/code&gt; == &lt;code&gt;(3025, 100, 1000)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Creating these time-blocks is a bit of a hassle, but create a good function once, then save it  :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is more work to be done, perhaps look at more in depth examples of my explanation above &lt;a href=&quot;https://machinelearningmastery.com/reshape-input-data-long-short-term-memory-networks-keras/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;... or have a read of the &lt;a href=&quot;https://keras.io/layers/recurrent/#lstm&quot; rel=&quot;noreferrer&quot;&gt;LSTM documentation&lt;/a&gt;, (or better still, &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L1906&quot; rel=&quot;noreferrer&quot;&gt;the source code!&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final model would then be simple enough (based on your code):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Create model&#xA;model = Sequential()&#xA;model.add(LSTM(units=32, activation='relu',&#xA;               input_shape=(100, 1000))    # the batch size is neglected!&#xA;model.add(Dense(3, activation='softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam',&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Have a look at &lt;a href=&quot;https://keras.io/getting-started/sequential-model-guide/#specifying-the-input-shape&quot; rel=&quot;noreferrer&quot;&gt;the documentation regarding input shape for the &lt;code&gt;Sequential&lt;/code&gt; model&lt;/a&gt;. It basically says that we don't need to specify the number of batches within &lt;code&gt;input_shape&lt;/code&gt;. It can be done with e.g. &lt;code&gt;batch_size=50&lt;/code&gt;, if you require it to be a fixed number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the &lt;code&gt;input_shape&lt;/code&gt; argument is not in the documentation for &lt;code&gt;LSTM&lt;/code&gt;, but the class itself inherits from &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L212&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;RNN&lt;/code&gt;&lt;/a&gt;, which in turn inherits from &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/engine/topology.py#L191&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;Layer&lt;/code&gt;&lt;/a&gt; - so it will be able to use the info you provide.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;One last tip:&lt;/strong&gt; if you plan on adding several LSTM layers ('stacking' them), then you shall need to add one more argument to all but the &lt;em&gt;last&lt;/em&gt; &lt;code&gt;LSTM&lt;/code&gt;, namely, the &lt;code&gt;return_sequences=True&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-02-06T17:34:42.257" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="27536" PostTypeId="1" AcceptedAnswerId="27584" CreationDate="2018-02-06T18:20:35.000" Score="2" ViewCount="3389" Body="&lt;p&gt;Sorry for vague heading for the question. My question is that, is there any way to compare features (or attributes) used in machine learning algorithm? I have used Naive Bayesian classifier for binary classification which consists of total 6 features. I want to compare this features with one another and also list out features in priority order of their significance in models(how important that feature is). For example we use information gain or gini index to decide how effective feature is in CART. How can I compare this in naive bayes? I have read about confusion matrix and f measure but they are good at defining goodness of models and not features.&lt;/p&gt;&#xA;" OwnerUserId="43679" LastEditorUserId="43679" LastEditDate="2018-02-07T02:07:50.617" LastActivityDate="2018-02-08T07:31:11.273" Title="Feature importance parameter in machine learning models like Naive Bayes" Tags="|machine-learning|decision-trees|naive-bayes-classifier|" AnswerCount="1" CommentCount="6" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27537" PostTypeId="1" CreationDate="2018-02-06T19:14:18.280" Score="1" ViewCount="62" Body="&lt;p&gt;Recall that an AR(p) process can be defined as&#xA;$$&#xA;X_t = \phi_1 X_{t-1} + \cdots + \phi_p X_{t-p} + Z_t&#xA;$$&#xA;where $Z_t$ is I.I.D. white noise. I want to write an algorithm simulating these processes, but I'm not sure if I'm taking the right approach. Here's my idea:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;INPUT:&#xA;  n = length of time series&#xA;  [\phi_1,...,\phi_p] the parameters of the process&#xA;&#xA;1. Construct a white noise vector Z of length n + p&#xA;2. Have the first rolling sum of the X_i's be set as the noise&#xA;3. Construct the rest of the time series from there&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case&#xA;$$&#xA;\begin{align}&#xA;X_0 &amp;amp;= \phi_1Z_{-1} + \cdots + \phi_pZ_{-p} + Z_0 \\&#xA;X_1 &amp;amp;= \phi_1X_0 + \phi_2Z_{-1} + \cdots + \phi_pZ_{-(p-1)} \\&#xA;&amp;amp;\cdots&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another option would be to start by constructing $X_1,\ldots, X_p$ by having the lower order terms in the recursion equation be set to 0. So, for the first few&#xA;$$&#xA;\begin{align}&#xA;X_1 &amp;amp;= Z_1 \\&#xA;X_2 &amp;amp;= \phi_1 X_1 + Z_2 \\&#xA;X_3 &amp;amp;= \phi_1 X_2 + \phi_2 X_1 + Z_3 \\&#xA;&amp;amp;\cdots&#xA;\end{align}&#xA;$$&#xA;Which approach is the &quot;right&quot; approach for constructing the simulation?&lt;/p&gt;&#xA;" OwnerUserId="41894" LastEditorUserId="41894" LastEditDate="2018-02-07T19:55:34.220" LastActivityDate="2018-02-07T19:55:34.220" Title="How can I implement my own AR(p) simulation algorithm?" Tags="|time-series|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27538" PostTypeId="1" AcceptedAnswerId="27540" CreationDate="2018-02-06T19:58:15.913" Score="2" ViewCount="104" Body="&lt;p&gt;I got question about preparation data for my ML algorithm.&#xA;Raw data has format similar to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;  &quot;finances&quot;: [&#xA;    {&#xA;      &quot;assets&quot;: 1230.39,&#xA;      &quot;investments&quot;: 3245.39,&#xA;      &quot;netProfit&quot;: 8765.45,&#xA;      &quot;year&quot;: 2017&#xA;    },&#xA;    {&#xA;      &quot;assets&quot;: 111.11,&#xA;      &quot;investments&quot;: 222.22,&#xA;      &quot;netProfit&quot;: 333.33,&#xA;      &quot;year&quot;: 2016&#xA;    },&#xA;    {&#xA;      &quot;assets&quot;: 1111.11,&#xA;      &quot;investments&quot;: 2222.22,&#xA;      &quot;netProfit&quot;: 3333.33,&#xA;      &quot;year&quot;: 2015&#xA;    }&#xA;  ],&#xA;  &quot;someValue&quot;: 123.45,&#xA;  &quot;title&quot;: &quot;Hello!&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And I am wondering what is best way to pass data about finances to my algorithm (Number of years can differ from 0 to 8).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking about making every parameter for every year new attribute, but this would lead to have many not null values for some cases, because not every record has 8 years of financial history.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: What is best way to handle such data in Input data (In fact every of &lt;code&gt;finances&lt;/code&gt; elements would have more values in it (around 10-15).&lt;/p&gt;&#xA;" OwnerUserId="45869" LastActivityDate="2018-02-06T22:21:56.907" Title="Preprocess list data" Tags="|data-cleaning|preprocessing|finance|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27539" PostTypeId="1" CreationDate="2018-02-06T21:43:44.093" Score="5" ViewCount="11604" Body="&lt;p&gt;I have a question about the type of model which I should use for a dataset I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset has a total of 7 independent variables and 1 dependent variable which I need to predict.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out of the 7 input variables, 6 of them are categorical and 1 is a date column. Now I have encoded the categorical columns using label encoding and converted them into numerical values. Now I’ve used a simple linear regression model on this dataset and achieved a normalized RMSE value of 0.11.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to improve this accuracy, how do I go about doing it? How can I derive upon the kind of models that I can use considering the data set I have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is mostly about forecast revenues for each product group. That's why I have those categorical columns which represent the hierarchy of the product groups.&lt;/p&gt;&#xA;" OwnerUserId="45872" LastActivityDate="2018-02-07T09:11:16.337" Title="Choosing a model for dataset with categorical variables" Tags="|predictive-modeling|model-selection|" AnswerCount="1" CommentCount="2" ClosedDate="2018-02-18T13:58:57.503" ContentLicense="CC BY-SA 3.0" />
  <row Id="27540" PostTypeId="2" ParentId="27538" CreationDate="2018-02-06T22:21:56.907" Score="2" Body="&lt;p&gt;The ultimate end goal of your modeling is going to affect the way you want to format your data. It's a good practice whenever you start a machine learning project to ask yourself, &lt;em&gt;what is the precise question you want to answer&lt;/em&gt;, because whatever model you generate, it's only going to make sense if used in the context of the question asked. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If in your case you want to predict loan approvals, then first you need to check whether that information is even present in the data you have. The JSON you have just shows historical data, but do you know the outcome for each data point? Is that another feature, maybe recorded in the &quot;someValue&quot;, &quot;title&quot; area of the file? Without it, you really can't do anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can get that information, then it's perfectly fine to generate a variable for each year: assets_year1, assets_year2, ... assets_year8, investments_year1, ... etc. True, year8 might have a lot of Null values, but that's not necessarily bad. For example, most models working with text data consist of really sparse training matrices, yet they do very well in practice. Depending on the algorithm, it might weed out those variables anyways. With Null values, you just have to try imputing the Null values differently, and you can also create an indicator variable for whether the column has missing values, and see how different settings change performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take it a step further, you can also create additional features to encapsulate the nuances that are occurring over the years. Create features like the historical average, average change per year, total number of years of history, etc. Creative feature engineering is the hardest part, but can lead to big changes in performance. &lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-06T22:21:56.907" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27541" PostTypeId="1" AcceptedAnswerId="27696" CreationDate="2018-02-06T23:56:08.963" Score="3" ViewCount="23382" Body="&lt;p&gt;I have tried installing profiling pandas profiling library with the following command from the terminal:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install pandas-profiling&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ModuleNotFoundError: No module named 'pandas_profiling'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong?&lt;/p&gt;&#xA;" OwnerUserId="45534" LastActivityDate="2018-02-11T09:13:34.983" Title="Python: Unable to install pandas_profiling" Tags="|python|pandas|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-11T21:59:37.407" ContentLicense="CC BY-SA 3.0" />
  <row Id="27542" PostTypeId="1" CreationDate="2018-02-07T06:19:19.840" Score="1" ViewCount="343" Body="&lt;p&gt;Problem statement:&#xA;Given a set of random variables Xs, give the order of importance of their ability to explain variance in target variable Y. Assumes all Xs are normalised to have 0 mean and std dev of 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can treat this as regression problem(normal or ordinal)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solution for 2 cases:&#xA;Case 1: When the calculated covariance matrix is diagonal, I know that random variables Xs are independent wrt each other. Then I can fit a linear regression model and order Xs by their weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 2: Or If I know that Xs are independent(&lt;a href=&quot;https://en.wikipedia.org/wiki/Independence_(probability_theory)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Independence_(probability_theory)&lt;/a&gt;&#xA;even though their covariance in non zero, I can fit linear regression model and order Xs by weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I solve this problem when I know that random variable Xs are not independent and have non zero covariance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also how can I use a non linear model such as random forest to get variable importance? scikit provides feature_importances_ api for random forest.... Is there an gotcha with this api such as results are not meaningful when variables are dependent or covariant?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Found this question elsewhere: &lt;a href=&quot;https://stats.stackexchange.com/questions/202221/for-linear-classifiers-do-larger-coefficients-imply-more-important-features/202846&quot;&gt;https://stats.stackexchange.com/questions/202221/for-linear-classifiers-do-larger-coefficients-imply-more-important-features/202846&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27108" LastEditorUserId="27108" LastEditDate="2018-02-07T07:00:06.180" LastActivityDate="2018-02-07T07:01:26.947" Title="Find the order of importance of random variables in their ability to explain a variance of Y" Tags="|scikit-learn|random-forest|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27543" PostTypeId="2" ParentId="27542" CreationDate="2018-02-07T06:50:37.583" Score="2" Body="&lt;p&gt;&lt;strong&gt;Decision trees&lt;/strong&gt; are by nature immune to multi-collinearity. So by that principal so is &lt;strong&gt;Random Forest&lt;/strong&gt;. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. - follow &lt;a href=&quot;https://datascience.stackexchange.com/a/12597/30578&quot;&gt;this link&lt;/a&gt; to know more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For other models, solutions can be many, but here are the popular choices - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Regularization&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an automatic way of handling high correlations in your Xs. It decreases the value of the co-efficients by penalizing them against the loss function and introduces bias in the system. Popular implementations are &lt;code&gt;L1&lt;/code&gt;, &lt;code&gt;L2&lt;/code&gt; and &lt;code&gt;Elastic Net&lt;/code&gt;(in case of linear and logistic models), &lt;code&gt;Pruning&lt;/code&gt;(in case of Decision Trees), &lt;code&gt;Dropout&lt;/code&gt; and &lt;code&gt;max pooling&lt;/code&gt; in case of deep learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost all &lt;code&gt;sklearn&lt;/code&gt; implementations support a regularization parameter in 1 way or the other. Depends on the algorithm you are using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Using VIF to detect multi-collinearity&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Variance_inflation_factor&quot; rel=&quot;nofollow noreferrer&quot;&gt;Variance Inflation Factor&lt;/a&gt; is a metric that you can use with all your Xs and then start dropping variables with the highest VIF &lt;em&gt;one by one&lt;/em&gt; (usually the variable with the highest VIF is dropped and then the exercise is repeated till there is no multi-collinearity left or domain experts come in and choose what important variables to keep/drop)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Principal Component Analysis&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the model might lose its interpretability (you won't be able to tell exactly what feature is how important), &lt;code&gt;PCA&lt;/code&gt; has been another effective way of removing multi-collinearity. Basically it projects the data in a way that the output columns are orthogonal (independent)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Some Examples&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ridge Regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn.linear_model import Ridge&#xA; import numpy as np&#xA; n_samples, n_features = 10, 5&#xA; np.random.seed(0)&#xA; y = np.random.randn(n_samples)&#xA; X = np.random.randn(n_samples, n_features)&#xA; clf = Ridge(alpha=1.0)&#xA; clf.fit(X, y) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lasso Regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn import linear_model&#xA; clf = linear_model.Lasso(alpha=0.1)&#xA; clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elastic Net&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn.linear_model import ElasticNet&#xA; from sklearn.datasets import make_regression&#xA;&#xA; X, y = make_regression(n_features=2, random_state=0)&#xA; regr = ElasticNet(random_state=0)&#xA; regr.fit(X, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt;&#xA; - Almost all parameters in scikit-learn can account for regularization here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statsmodels.org/dev/generated/statsmodels.stats.outliers_influence.variance_inflation_factor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Statsmodels&lt;/a&gt; has a rather good implementation of VIF&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.decomposition import PCA&#xA;from sklearn.svm import SVC&#xA;&#xA;X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])&#xA;pca = PCA(n_components=2) # adjust yourself&#xA;pca.fit(X)&#xA;&#xA;X_t_train = pca.transform(X)&#xA;clf = SVC()&#xA;clf.fit(X_t_train, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30578" LastEditorUserId="30578" LastEditDate="2018-02-07T07:01:26.947" LastActivityDate="2018-02-07T07:01:26.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27545" PostTypeId="1" AcceptedAnswerId="27554" CreationDate="2018-02-07T08:15:54.073" Score="0" ViewCount="104" Body="&lt;p&gt;Anyone know the best packages to build a decision tree in Rstudio v3.3?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to look at some data-driven segmentation for my data. I was thinking of doing chaid analysis (mainly because this is why I've done in the past). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've looked around myself, but can't find the best package for this version! &lt;/p&gt;&#xA;" OwnerUserId="45887" LastActivityDate="2018-02-07T09:58:17.167" Title="Decision trees for Rstudio v3.3" Tags="|r|decision-trees|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-12T19:35:04.300" ContentLicense="CC BY-SA 3.0" />
  <row Id="27546" PostTypeId="1" AcceptedAnswerId="27548" CreationDate="2018-02-07T08:38:43.430" Score="2" ViewCount="1553" Body="&lt;p&gt;Lets say I have a feature set of &lt;code&gt;f0 to f1000&lt;/code&gt;. I am thinking of applying PCA on &lt;code&gt;f500 to f1000&lt;/code&gt; &lt;em&gt;reducing their dimensionality&lt;/em&gt;. Can I combine this &lt;em&gt;reduced set&lt;/em&gt; with the features &lt;code&gt;f0 to f499&lt;/code&gt; as the feature space for training a learning algorithm? &lt;/p&gt;&#xA;" OwnerUserId="45458" LastActivityDate="2018-02-07T09:02:57.203" Title="Can PCA be applied to reduce dimensionality of only a subset of features?" Tags="|pca|" AnswerCount="1" CommentCount="2" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27548" PostTypeId="2" ParentId="27546" CreationDate="2018-02-07T09:02:57.203" Score="6" Body="&lt;p&gt;Yes, absolutely. Simply split your data into two sets feature-wise, apply PCA to one of them, and then stick them back together again. How to actually perform this will vary depending on your programming language/frameworks, but it is trivially easy in python + pandas, for example.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-02-07T09:02:57.203" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27549" PostTypeId="2" ParentId="27539" CreationDate="2018-02-07T09:05:44.473" Score="6" Body="&lt;p&gt;I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar &lt;a href=&quot;https://datascience.stackexchange.com/questions/27012/cleaning-input-data-with-pd-get-dummies/27014#27014&quot;&gt;post&lt;/a&gt;, where I discuss some techniques to convert categorical variables to numerical values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you provide little information about your categorical variables, for example how many levels each categorical variable have or how you do label encoding (just out-of-the-box method?) it is hard to give better guidelines. Generally speaking though, label encoding is mostly used for encoding  target variables (dependent variable) if it is categorical, while you mention you use label encoding for other 6 independent variables (inputs). Here your target is rather numerical values (regression), so I would go beyond label encoding to convert your independent categorical variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image the following very simplified scenario (just for educational purposes) to understand why label encoding is not a good idea where there is more than 1 independent categorical variables:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Color   Year&#xA;0    Red   2010&#xA;1   Blue   2011&#xA;2  Green   2012&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now label encoding this would converting this to (naively speaking): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Color  Year&#xA;0   1    1&#xA;1   2    2&#xA;2   3    3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There ways like constructing your own dictionary for each categorical variable and values associated to its levels while label encoding to prevent this obvious mistake. Apart from that, since you hardly go beyond RMSE value of 0.11, one could think of your input values (maybe not encoded properly!) and choice of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe try using this &lt;a href=&quot;https://github.com/scikit-learn-contrib/categorical-encoding&quot; rel=&quot;noreferrer&quot;&gt;code&lt;/a&gt; for handling your independent categorical variables. Please note here, as a general rule of thumb, avoid using one-hot encoding although it is widely used in XGBoost (explained in the post). Maybe try &quot;Target Encoding&quot;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now speaking of the choice of model:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Have you tried regularization (L1, L2)?&lt;/li&gt;&#xA;&lt;li&gt;Have confident you are about the independency of your independent variables? In simple regression you are assuming this holds, but may it is not which is easily the case that your independent variables are correlated. Maybe you should analyze this first using confusion matrix (see &lt;a href=&quot;https://www.datascience.com/blog/introduction-to-correlation-learn-data-science-tutorials&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; for a brief tutorial)&lt;/li&gt;&#xA;&lt;li&gt;Have you tried gradient boosting decision trees (GBDT) for regression?&#xA;Note1: If you want to go with GBDTs, do not leave that other 1 independent variable alone. You said you have 6 independent variables out of 7. I assume the other one is continuous numerical values? If that so, you have to handle that properly in GBDTs like binarizing (put in bins).&#xA;Note2: You may try either &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/&quot; rel=&quot;noreferrer&quot;&gt;XGBoost&lt;/a&gt;, or &lt;a href=&quot;https://tech.yandex.com/catboost/&quot; rel=&quot;noreferrer&quot;&gt;Catboost&lt;/a&gt;, and a nice &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/08/catboost-automated-categorical-data/&quot; rel=&quot;noreferrer&quot;&gt;tutorial&lt;/a&gt; using Catboost. Each has its pros and cons. For the former hyperparameter tuning could be challenging and there is no ways to automatically account for categorical variables but have larger community. While the latter is recently released (mid last year), it is much easier in terms of hyperparameter tuning (often defaults works great), and there are ways to automatically account for categorical features without explicitly encode them, but less support and community is smaller.&#xA;Note3: The good thing about GBDT is that you do not need to worry about the correlation of features per se. Algorithm automatically ignore the coupling of strongly correlated features. &#xA;Note4: You may easily overfit, although there are built-in regularization. Catboost for example have a interactive learning curve for training and validation to control overfitting (see the last tutorial). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I strongly believe taking care of your inputs together with a proper choice of algorithm would boost your RMSE error. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-02-07T09:11:16.337" LastActivityDate="2018-02-07T09:11:16.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27550" PostTypeId="1" CreationDate="2018-02-07T09:19:44.687" Score="2" ViewCount="23" Body="&lt;p&gt;I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of customers, some are lost, some are still active. We have a model which looks through the data and assigns reasons for why they became lost. One of the reasons to determine is if them not using our software caused them to become lost - we have software to automate their orders but they can ignore this and process orders manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created 2 variables; software usage rate and software usage vs their country average (as a ratio). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim is to determine the point, in either or both of these, which is basically the threshold/marker to say the software was an issue. I am stuck as there are in essence no results - we know if a customer is lost, but we can only speculate/model on the reasons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I've been doing so far is looking at the distributions of lost and active manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="27683" LastEditorUserId="27683" LastEditDate="2018-02-07T09:49:08.803" LastActivityDate="2018-02-07T10:54:11.733" Title="Find threshold in rate to determine reason for lost customer" Tags="|prediction|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27551" PostTypeId="1" CreationDate="2018-02-07T09:42:15.853" Score="1" ViewCount="377" Body="&lt;p&gt;I have ensembled 3 algorithms as below,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimators = []&#xA;model1 = MultinomialNB().fit(X,y)&#xA;estimators.append(('Naive Bayes', model1))&#xA;model2 = LinearSVC(random_state=0).fit(X,y)&#xA;estimators.append(('SVM', model2))&#xA;model3 = RandomForestClassifier(bootstrap=True).fit(X,y)&#xA;estimators.append(('Random Forest', model3))&#xA;&#xA;# create the ensemble model&#xA;ensemble = VotingClassifier(estimators)&#xA;results = model_selection.cross_val_score(ensemble, X, y)&#xA;print(results.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while passing new sentence to test the ensembling I have used, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predicted = results.predict(X_new_tfidf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I am getting error:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;AttributeError: 'numpy.ndarray' object has no attribute 'predict'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I fix/debug this?&lt;/p&gt;&#xA;" OwnerUserId="45889" LastEditorUserId="29575" LastEditDate="2018-02-07T13:34:23.790" LastActivityDate="2018-02-07T15:12:18.527" Title="testing new data in model" Tags="|python|scikit-learn|pandas|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27552" PostTypeId="1" CreationDate="2018-02-07T09:42:20.040" Score="0" ViewCount="187" Body="&lt;p&gt;This is the scenario:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Client -&amp;gt; Server&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The client sends multiple voice calls to Server.&#xA;Call info:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;calling number &lt;/li&gt;&#xA;&lt;li&gt;called number&lt;/li&gt;&#xA;&lt;li&gt;call duration&lt;/li&gt;&#xA;&lt;li&gt;source IP &lt;/li&gt;&#xA;&lt;li&gt;destination IP&lt;/li&gt;&#xA;&lt;li&gt;SIP header information (User-Agent, Version, Proxy information)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other patterns:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Numeric Patterns - Destination.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Time Patterns - Calls occurring too regularly, activity at unusual times or dates.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Geographic Patterns - Proximity relationships between apparently unrelated entities.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In case of Toll fraud and client being insecure, the attacker can send calls via original agent hence the Service Provider won't reject calls immediately at least based from IP information. (Toll fraud).&#xA;I'm exploring which approach is the best to implement an ML model, instead of static rules to be able to detect Toll fraud in a live system:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Classification (Fraud or not)&lt;/li&gt;&#xA;&lt;li&gt;Reinforcement Learning (If fraud then reward)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm new to RL, does this scenario makes sense for live systems?&#xA;Can I use offline data to train an RL model? Offline data includes Fraud cases detected historically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.easysol.net/real-time-learning-to-reinforcement-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2018-02-07T15:25:10.607" Title="Multivariable real time system for fraud detection" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27553" PostTypeId="2" ParentId="27537" CreationDate="2018-02-07T09:48:24.973" Score="1" Body="&lt;p&gt;You question boils down to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;should you try to do the simulation in a vectorised manner?&lt;/li&gt;&#xA;&lt;li&gt;should you iterate over a vector?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In general we should always try to vectorise (option 1) as it's much more efficient, and it is easy to lag vectors in R (&lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/stats/html/embed.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;embed&lt;/code&gt;&lt;/a&gt;) and Python Pandas (&lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shift.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;shift&lt;/code&gt;&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I can't see any efficient vectorisation-based solution. The autoregressive nature of the simulated values force us to iterate over a vector to simulate each of $X_1, X_2, \ldots, X_n$ individually (option 2). &lt;/p&gt;&#xA;" OwnerUserId="45615" LastActivityDate="2018-02-07T09:48:24.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27554" PostTypeId="2" ParentId="27545" CreationDate="2018-02-07T09:58:17.167" Score="2" Body="&lt;p&gt;The &quot;best&quot; package depends on your goals and data really.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few tree / forest packages that I've come across:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;randomForest&lt;/code&gt; - an implementation of the original algorithm&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;ranger&lt;/code&gt; - a flexible implementation designed to run on high-dimensional data (my personal favourite forest package)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;party&lt;/code&gt;'s ctree / cforest functions - conditional inference trees / forests&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;rpart&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some of these packages focus on forests rather than trees; you can coerce the forest algorithm into building a tree by flexing appropriate parameters.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-07T09:58:17.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27555" PostTypeId="1" AcceptedAnswerId="98225" CreationDate="2018-02-07T10:51:54.103" Score="1" ViewCount="49" Body="&lt;p&gt;Suppose we operate with state-action pairs called 'S', and a reward function R() as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R(S1) &amp;lt;- 0&#xA;R(S2) &amp;lt;- 0&#xA;...&#xA;R(Sn) &amp;lt;- 1&#xA;&#xA;so that&#xA;&#xA;E(R(S)) &amp;lt;&amp;lt; 1, i.e., non-zero rewards are very sparse&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In that case a RL agent faces a task of accumulating as many as possible 1s in order to maximize &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sum(gamma^i * Ri), i -&amp;gt;&amp;gt; infinity&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now consider immediate rewards introduced by one to make the agent better behaved:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R(S1) &amp;lt;- 0&#xA;R(S2) &amp;lt;- -1&#xA;R(S3) &amp;lt;- -10&#xA;R(S4) &amp;lt;- 0.1&#xA;...&#xA;R(Sn) &amp;lt;- 1&#xA;&#xA;so that &#xA;&#xA;E(R(S)) -&amp;gt;&amp;gt; -10, i.e., a simple average of random rewards is strongly negative.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it the case that in order to maximize the sum of discounted rewards the agent may choose to avoid the states with strongly negative rewards even at the cost of not getting positive rewards in some delayed states?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this really a non-linear problem to the model that approximates the agent's actions in that varying level of negative/positive immediate rewards (let's say, from -1 to 1, VS. from -100 to 100) may distract the agent from getting the delayed rewards which must be maximized under optimal policy in favor of maximizing immediate rewards, even if gamma -&gt;&gt; 1?&lt;/p&gt;&#xA;" OwnerUserId="42953" LastEditorUserId="42953" LastEditDate="2018-02-08T08:55:10.193" LastActivityDate="2021-07-23T12:48:48.410" Title="Does employment of engineered immediate rewards in RL introduce a non-linear problem to an agent?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27556" PostTypeId="2" ParentId="27550" CreationDate="2018-02-07T10:54:11.733" Score="1" Body="&lt;p&gt;You could use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test&quot; rel=&quot;nofollow noreferrer&quot;&gt;two-sample $t$-test&lt;/a&gt; to determine whether software usage is significantly lower for lost users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Plotting 'software usage rate'/'software usage rate relative to country average' against 'percentage of lost users' might give you an idea about a suitable threshold.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-07T10:54:11.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27560" PostTypeId="2" ParentId="27062" CreationDate="2018-02-07T11:59:02.427" Score="1" Body="&lt;p&gt;Since you do not provide any information about the creation of X_train and y_train I cannot be sure that this is the issue, but I would guess that it has to do with some change in the order of the features. Three-based algorithms' results are affected when the order of the input features changes, so make sure that this order does not change every time you run your code (e.g. you can sort them alphabetically before RFE). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/5394&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; discussion interesting.&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-02-07T11:59:02.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27561" PostTypeId="1" AcceptedAnswerId="27562" CreationDate="2018-02-07T13:34:05.250" Score="20" ViewCount="48957" Body="&lt;p&gt;I am using a convolution neural network ,&lt;code&gt;CNN&lt;/code&gt;. At a specific epoch, I only save the best CNN model weights based on improved validation accuracy over previous epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does increasing the number of epochs also increase over-fitting for &lt;code&gt;CNNs&lt;/code&gt; and deep learning in general?&lt;/p&gt;&#xA;" OwnerUserId="45898" LastEditorUserId="29169" LastEditDate="2021-02-13T16:42:25.633" LastActivityDate="2021-02-13T16:42:25.633" Title="Can the number of epochs influence overfitting?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27562" PostTypeId="2" ParentId="27561" CreationDate="2018-02-07T14:00:07.520" Score="20" Body="&lt;p&gt;Yes, it may. In machine-learning there is an approach called early stop. In that approach you plot the error rate on training and validation data. The horizontal axis is the &lt;code&gt;number of epochs&lt;/code&gt; and the vertical axis is the &lt;code&gt;error rate&lt;/code&gt;. You should stop training when the error rate of validation data is minimum. Consequently if you increase the number of epochs, you will have an over-fitted model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In deep-learning era, it is not so much customary to have early stop. There are different reasons for that but one of them is that deep-learning approaches need so much data and plotting the mentioned graph would be so much wavy because these approach use stochastic-gradient-like optimizations. In deep-learning again you may have an over-fitted model if you train so much on the training data. To deal with this problem, another approaches are used for avoiding the problem. Adding noise to different parts of models, like drop out or somehow batch normalization with a moderated batch size, help these learning algorithms not to over-fit even after so many epochs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general too many epochs may cause your model to over-fit the training data. It means that your model does not &lt;em&gt;learn&lt;/em&gt; the data, it memorizes the data. You have to find the accuracy of validation data for each epoch or maybe iteration to investigate whether it over-fits or not.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="60055" LastEditDate="2018-12-19T15:00:01.077" LastActivityDate="2018-12-19T15:00:01.077" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="27563" PostTypeId="1" AcceptedAnswerId="27572" CreationDate="2018-02-07T14:49:20.597" Score="16" ViewCount="19944" Body="&lt;p&gt;I have been trying to understand how to represent and shape data to make a &lt;strong&gt;multidimentional&lt;/strong&gt; and &lt;strong&gt;multivariate&lt;/strong&gt; time series forecast using Keras (or TensorFlow) but I am still very unclear after reading many blog posts/tutorials/documentation about how to present the data in the correct shape (most examples being of slightly less &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Dataset:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;several cities&lt;/li&gt;&#xA;&lt;li&gt;for which I have info about say temperature, car traffic, humidity&lt;/li&gt;&#xA;&lt;li&gt;for say the last 2 years (one record for each day)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I want to do:&lt;/strong&gt;&#xA;I'd like to forecast for each city the temperatures I can expect for the next year using a possibly lagged version of temperature, car traffic and humidity (of course there would be several more features but this is just an example for thought).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I am confused about:&lt;/strong&gt;&#xA;If I have 2 cities, for which I recorded 3 features for 365 days. How should I shape my input so that the model can output a forecast for 365 days for these two cities (i.e. 2 time series of temperatures for 365 days)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively the tensor shape would be &lt;code&gt;(?, 365, 3)&lt;/code&gt; for 365 days and 3 features. But I'm not sure what to stick into the first dimension and, most importantly, I would be surprised if it had to be for the number of cities. But at the same time, I have no idea how to specify into the model that it has to understand the dimensions properly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers will be helpful. I'm pretty familiar with the rest of the problem (i.e. how you build a network in Keras etc since I have done this for other neural networks but more specifically how best to encode the sequence for the desired input.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Oh and also&lt;/em&gt;, I guess I could train and predict for each city independently, but I'm sure everyone will agree there are probably things to be learned that are not particular to any city but that can only be seen if considering several of them, hence why I think it is important to encode it in the model.&lt;/p&gt;&#xA;" OwnerUserId="45892" LastActivityDate="2020-05-09T17:10:47.860" Title="Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras" Tags="|python|keras|rnn|lstm|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27564" PostTypeId="1" CreationDate="2018-02-07T14:56:35.927" Score="0" ViewCount="39" Body="&lt;p&gt;I am trying to do download all the marathon results from 1996 to 2016 from the following link. There are more than 400000 entries, but I can only show 1000 results per page. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way that can let me download all of them quickly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://chicago-history.r.mikatiming.de/2015/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://chicago-history.r.mikatiming.de/2015/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45902" LastActivityDate="2018-02-07T15:15:25.213" Title="How can I download data from online table quickly?" Tags="|dataset|" AnswerCount="1" CommentCount="3" ClosedDate="2018-02-12T19:35:21.300" ContentLicense="CC BY-SA 3.0" />
  <row Id="27565" PostTypeId="2" ParentId="27551" CreationDate="2018-02-07T15:12:18.527" Score="1" Body="&lt;p&gt;Your line &lt;code&gt;results = model_selection.cross_val_score(ensemble, X, y)&lt;/code&gt; just returns an array of scores for each run of the cross validation, so the error is telling you that it's just an array of numbers, not a model itself and has no defined method for fitting things. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use predict, you need to call it on a model object, which I'm assuming you're going for the VotingClassifier. Your &lt;code&gt;ensemble&lt;/code&gt; variable has a fit method, so you want to call&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predicted = ensemble.predict(X_new_tfidf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-07T15:12:18.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27566" PostTypeId="2" ParentId="27564" CreationDate="2018-02-07T15:15:25.213" Score="1" Body="&lt;p&gt;There doesn't appear to be an easy way - there is no mention of an API to get the data programmatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try contacting them via Twitter or something. Perhaps they would be happy to give you a dump of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you might have to go down the route of writing a scraper e.g. using &lt;a href=&quot;http://www.seleniumhq.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Selenium&lt;/a&gt; and &lt;a href=&quot;https://www.crummy.com/software/BeautifulSoup/&quot; rel=&quot;nofollow noreferrer&quot;&gt;BeautifulSoup&lt;/a&gt;. That would allow you to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;execute the search&lt;/li&gt;&#xA;&lt;li&gt;download the file&lt;/li&gt;&#xA;&lt;li&gt;navigate to the next search results page&lt;/li&gt;&#xA;&lt;li&gt;repeat from step 2. until finished&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;They might be fairly easy, but do respect the websites wishes - perhaps they have a file called &lt;code&gt;robots.txt&lt;/code&gt; which will tell you if they mind you doing that or not. In any case, build some &lt;code&gt;time.sleep()&lt;/code&gt;s into your code, as not to bombard their server!&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-02-07T15:15:25.213" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27567" PostTypeId="2" ParentId="27552" CreationDate="2018-02-07T15:25:10.607" Score="1" Body="&lt;p&gt;I think it makes sense to stick with Classification here, since you already have examples of fraudulent and non-fraudulent calls that you can train on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might also be beneficial to train several models for different regions based on IP, as well as your 'global' model, and apply the region specific and global models to incoming calls. Just a few ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand, &lt;a href=&quot;http://blog.easysol.net/real-time-learning-to-reinforcement-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;real-time learning would require immediate feedback&lt;/a&gt;, which most fraud detection systems can't provide. For example, it may take a few days or weeks to have a case of fraud resolved (labelled fraud/not-fraud), and therefore take some time for the learning system to receive the feedback on it's prediction.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2018-02-07T15:25:10.607" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27569" PostTypeId="1" AcceptedAnswerId="27570" CreationDate="2018-02-07T15:39:38.870" Score="0" ViewCount="1911" Body="&lt;p&gt;I have built CART model using sklearn. I'm having total 6 features in training dataset and passing all of them in fit function. I've tested both criteria Gini and entropy. But whenever I plot tree using graphviz, the tree uses only 3 features in case of Gini and 4 features in case of entropy. I've also implemented CART from scratch for cross check purpose and still, Gini uses 3 features and entropy uses 4 features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything else is working fine. I'm getting the result as expected in test dataset with accuracy 97%. I just want to confirm that, is this normal? Does cart chooses best features only and ignore other features to avoid the wrong classification? Also, my dataset is quite small in size, only 220 records.&lt;/p&gt;&#xA;" OwnerUserId="43679" LastActivityDate="2018-02-07T16:13:08.020" Title="Decision tree not using all features from training dataset" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27570" PostTypeId="2" ParentId="27569" CreationDate="2018-02-07T16:13:08.020" Score="4" Body="&lt;p&gt;When a feature is not that informative of your target, the algorithm can choose not to use it. This can be for two reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All the information is already in the other features, meaning it adds nothing to include it&lt;/li&gt;&#xA;&lt;li&gt;There is some regularization going on, meaning your algorithm punishes complexity if it doesn't add enough&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It can also be that the regularization is very explicit, meaning there is a maximum depth set as hyperparameter. In this case the other features are more informative and useful for your task.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-07T16:13:08.020" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27572" PostTypeId="2" ParentId="27563" CreationDate="2018-02-07T17:08:45.123" Score="16" Body="&lt;p&gt;The input shape for an LSTM must be &lt;code&gt;(num_samples, num_time_steps, num_features)&lt;/code&gt;. In your example case, combining both cities as input, &lt;code&gt;num_features&lt;/code&gt; will be 2x3=6.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you lump all your 365 time steps into one sample, then the first dimension will be 1 - one single sample! You can also do sanity check by using the total number of data points. You have 2 cities, each with 365 time-steps and 3 features: 2x365x3= 2190 . This is obviously the same as 1x365x6 (as I said above) - so it would be a possibility (Keras will run) - but it obviously won't learn to generalise at all, only giving it one sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look &lt;a href=&quot;https://datascience.stackexchange.com/questions/27533/keras-lstm-with-1d-time-series/27535#27535&quot;&gt;at this relevant question&lt;/a&gt;, which I recently answered. There I speak a little about using a &lt;em&gt;rolling window&lt;/em&gt; (check the comments of the answer for more info). That will buy you more samples if you need them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to train a single model with data for both cities as input, then making predictions for both cities at each time-step is as simple as defining a final &lt;code&gt;Dense&lt;/code&gt; layer, which outputs 2 units. Your validation/test data must then of course contain a tuple of (city1, city2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A perhaps more sophisticated way to approach this would be to create datasets on a single-city basis, then train several sub-models on each city individually (say for 5 layers), then &lt;code&gt;Merge&lt;/code&gt;/&lt;code&gt;Concatenate&lt;/code&gt; them and put several further layers on top. This will mean you are combining the learnt features of each city, which are in turn being combined to a higher level of abstraction. Here is &lt;a href=&quot;https://1.bp.blogspot.com/-7P_ZGBc8s3c/V4TWwNokHpI/AAAAAAAABsk/P0KChOrEYkAEOFmQNE_EMkrjNAnnruluACLcB/s1600/concat.tif&quot; rel=&quot;noreferrer&quot;&gt;the first image I got from a search engine&lt;/a&gt;, which sketches the idea.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-02-07T17:08:45.123" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="27573" PostTypeId="1" CreationDate="2018-02-07T17:22:03.630" Score="2" ViewCount="587" Body="&lt;p&gt;I've been trying to determine the VC dimension of ellipses which are origin centered and axis aligned. My first approach was to find some equivalence to a threshold classifier function family of the form&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ H=\big\{\text{ sign}\big(\sum_{x}(Wx)+b\big)\ |\ w \in \mathbb{R}^d, b \in \mathbb{R}\ \big\}$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For these functions I know the VC dimension is $d+1$, so for axis-aligned origin-centered ellipses I tried&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ H=\big\{\text{ sign}\big(\sum_{x}(W^2x^2)-R^2\big)\ |\ w \in \mathbb{R}^d, R \in \mathbb{R}\ \big\} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought from here it would be straightforward but I couldn't figure it out. The limitations on the values that $w,x$ and $b$ (which equals $-R^2$) can get now can affect the dimension, but I'm not sure how.&lt;/p&gt;&#xA;" OwnerUserId="45909" LastEditorUserId="26562" LastEditDate="2018-02-08T03:37:12.803" LastActivityDate="2018-02-08T03:37:12.803" Title="How to determine the VC Dimension of axis-aligned, origin-centered ellipses?" Tags="|vc-theory|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27574" PostTypeId="1" AcceptedAnswerId="27600" CreationDate="2018-02-07T19:57:41.273" Score="1" ViewCount="2973" Body="&lt;p&gt;I have made a DCGAN which I am trying to train on custom dataset of only 1200 images. I have tried to gather more, but even gathering these 1200 was hard enough. If you are wondering I used Google Chromes extension &quot;Fakun Batch Download Image&quot; to gather my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TRAINING DETAILS:&lt;/strong&gt;&#xA;In training procedure I am simultaneously updating parameters of both, Generator, and Discriminator network. I've read that it works much better then training only one player ( Discriminator ) for K steps and then other ( Generator ) for one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUESTION:&lt;/strong&gt; Should I perform maybe some kind of transformation on all of those images and then merge transformed images with the initial ones, or something similar? &lt;/p&gt;&#xA;" OwnerUserId="45907" LastEditorUserId="29575" LastEditDate="2018-02-07T20:31:12.223" LastActivityDate="2018-02-08T12:39:49.053" Title="Tips &amp; Tricks on training DCGAN on small dataset" Tags="|deep-learning|convolution|cnn|gan|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27575" PostTypeId="1" CreationDate="2018-02-07T20:59:28.297" Score="3" ViewCount="110" Body="&lt;p&gt;I have a tough problem and need some advice:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a collection of variable length sequences, many of which are unique -- imagine the moves to a chess game, eg&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;d4 Nf6&lt;/li&gt;&#xA;  &lt;li&gt;c4 g6&lt;/li&gt;&#xA;  &lt;li&gt;Nc3 Bg7&lt;/li&gt;&#xA;  &lt;li&gt;e4 d6&lt;/li&gt;&#xA;  &lt;li&gt;Nf3 O-O&lt;/li&gt;&#xA;  &lt;li&gt;Be2 c5&lt;/li&gt;&#xA;  &lt;li&gt;O-O Bg4&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and for each item in this collection, I have another collection of descriptions generated by humans (think comments - &lt;code&gt;comment_1: &quot;cool game&quot;, comment_2: &quot;awesome sacrifice&quot;&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to mine the associations between the comments and the sequences to tag the sequences with human readable labels for search purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've thought about topic modeling for label generation + clustering/grouping the sequences, but I cannot figure out how to do something like cluster the games. I have millions of examples of the sequences if that helps. Any idea how I can measure the distance/similarity between sequences like this? Some kind of embedding? I've thought about trying a word2vec / doc2vec approach, but haven't tested yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally I'd be able to input an unseen sequence and suggest labels/human-readable description for this sequence as well. &lt;/p&gt;&#xA;" OwnerUserId="31879" LastActivityDate="2018-07-08T03:49:04.587" Title="Collection Of Variable Length Sequences and Descriptions: A Search Problem" Tags="|similarity|topic-model|distance|sequential-pattern-mining|vector-space-models|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27576" PostTypeId="1" AcceptedAnswerId="27577" CreationDate="2018-02-07T21:02:54.030" Score="2" ViewCount="575" Body="&lt;p&gt;I thought &lt;code&gt;n_samples&lt;/code&gt; is the number of training examples. But when using GridSearchCV, n_samples becomes 32 rather than 50.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Error when using GridSearchCV:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Expected n_neighbors &amp;lt;= n_samples,  but n_samples = 32, n_neighbors =&#xA;  50&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Training examples:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;print(X_train.shape[0])&lt;/code&gt; =&gt; 50&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;print(len(y_train))&lt;/code&gt; =&gt; 50&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This works:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;neigh = KNeighborsClassifier(n_neighbors=50)&#xA;neigh.fit(X_train, y_train) &#xA;result = neigh.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This fails:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import GridSearchCV&#xA;&#xA;grid_params = { &#xA;    &quot;n_neighbors&quot; : [50]&#xA;}&#xA;&#xA;g = GridSearchCV(KNeighborsClassifier(), grid_params)&#xA;g.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm confused why &lt;code&gt;n_samples&lt;/code&gt; becomes 32 when using GridSearchCV.&lt;/p&gt;&#xA;" OwnerUserId="45916" LastActivityDate="2018-02-07T21:56:36.773" Title="Why does GridSearchCV (sklearn) change the value of n_samples?" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27577" PostTypeId="2" ParentId="27576" CreationDate="2018-02-07T21:56:36.773" Score="4" Body="&lt;p&gt;The CV stands for CrossValidation, meaning it will split up your training set in a number of folds (in this case 3), train on &lt;strong&gt;n-1&lt;/strong&gt; of those folds and test on the remaining one. This is why your training is now done on 32 instead of 50 samples. Crossvalidation is useful for estimating how well your model (including specific hyperparameters) does on unseen data.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-07T21:56:36.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27578" PostTypeId="2" ParentId="27575" CreationDate="2018-02-07T22:22:57.957" Score="0" Body="&lt;p&gt;I don't know a ton about chess notation but it looks like you can encode these moves as pairs of categories in some way. Then every move is a time step in your (input) sequence. What you could try to do is use a seq2seq model to map your input sequence (the game progression) to the output sentence. I do think that modeling your inputs as a recurrent unit will be very possible, but predicting raw text output might be difficult to extract properly just because the signal is so thin and difficult to capture. You could look into classifying these comments into a few categories and turning it into a RNN classification problem, which might lead to better results.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-07T22:22:57.957" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27581" PostTypeId="1" AcceptedAnswerId="27582" CreationDate="2018-02-08T06:28:24.460" Score="4" ViewCount="1814" Body="&lt;p&gt;I have an &lt;a href=&quot;https://archive.ics.uci.edu/ml/machine-learning-databases/00242/ENB2012_data.xlsx&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I am unable to use the K-Fold Validation. I am getting the error raised:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;ValueError(&amp;quot;{0} is not supported&amp;quot;.format(y_type))&lt;/p&gt;&#xA;&lt;p&gt;ValueError: continuous is not supported .&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I do not want to do encoding to int, since it may affect the data, and also I want to understand why K-fold is not working.&lt;/p&gt;&#xA;&lt;p&gt;Below is my python code.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.ensemble import RandomForestRegressor&#xA;from sklearn import cross_validation, metrics&#xA;from sklearn.cross_validation import train_test_split&#xA;from sklearn import model_selection&#xA;from sklearn.metrics import classification_report&#xA;from sklearn.metrics import confusion_matrix&#xA;from sklearn.metrics import accuracy_score&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;from sklearn.neighbors import KNeighborsClassifier&#xA;from sklearn.discriminant_analysis import LinearDiscriminantAnalysis&#xA;from sklearn.naive_bayes import GaussianNB&#xA;&#xA;from sklearn import svm&#xA;from sklearn import preprocessing&#xA;&#xA; - `List item`&#xA;&#xA;url = &amp;quot;https://archive.ics.uci.edu/ml/machine-learning-databases/00242/ENB2012_data.xlsx&amp;quot;&#xA;names=['Relative Compactness','Surface Area','Wall Area','Roof Area','Overall Height','Orientation','Glazing Area','Glazing Area Distribution','Heating Load','Cooling Load']&#xA;df = pd.read_excel(url,names=names)&#xA;&#xA;#Feature selection&#xA;train=df.sample(frac=0.8,random_state=150)&#xA;test=df.drop(train.index)&#xA; &#xA;#save the original values in a dataframe so we can compare later&#xA;test_loads=test[[&amp;quot;Cooling Load&amp;quot;]]&#xA; &#xA;#Create 2 lists of response values to train our model&#xA;Y1=np.array(train['Heating Load'])&#xA;Y2=np.array(train['Cooling Load'])&#xA;&#xA;#Select the features&#xA;train_corr=train[['Overall Height','Relative Compactness','Roof Area','Surface Area']]&#xA;test_corr=test[['Overall Height','Relative Compactness','Roof Area','Surface Area']]&#xA;seed = 7&#xA;scoring = 'accuracy'&#xA;X_train,X_test,y_train,y_test=cross_validation.train_test_split(train_corr,Y1,test_size=0.2)&#xA;kfold = model_selection.KFold(n_splits=10, random_state=seed) &#xA;cv_results = model_selection.cross_val_score(RandomForestRegressor(), X_train, y_train, cv=kfold, scoring=scoring)&#xA;print (cv_results.mean())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45930" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2022-02-22T11:48:19.773" Title="Unable to Use The K-Fold Validation Sklearn Python" Tags="|machine-learning|python|deep-learning|scikit-learn|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27582" PostTypeId="2" ParentId="27581" CreationDate="2018-02-08T06:43:12.743" Score="1" Body="&lt;p&gt;Based on the answer &lt;a href=&quot;https://stackoverflow.com/a/48551629/5120235&quot;&gt;here&lt;/a&gt;, &lt;em&gt;Since you are doing a classification task, you should be using the metric R-squared (co-effecient of determination) instead of accuracy score (accuracy score is used for classification purposes).&lt;/em&gt; You should use something like &lt;code&gt;score&lt;/code&gt; for evaluation because your task is regression.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-08T06:43:12.743" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27583" PostTypeId="1" CreationDate="2018-02-08T06:53:48.523" Score="2" ViewCount="491" Body="&lt;p&gt;I have data set of 50 students. I want to cluster them on their sequential data ( While doing a job they followed multiple sequences A, B, c total 7 stages). I am planning to apply k-means clustering on their first order Markov chain transition probability matrix. So that means, I have 50, 7x7 transition probability matrix. Each 7x7 matrix has 49 data point. So, I can make a 50x49 matrix. If I apply k- means on this matrix, is it a proper approach for clustering sequential data? &lt;/p&gt;&#xA;" OwnerUserId="44808" LastActivityDate="2018-02-08T06:53:48.523" Title="k- means clustering on Markov chain trasition probability" Tags="|machine-learning|r|clustering|sequence|sequential-pattern-mining|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27584" PostTypeId="2" ParentId="27536" CreationDate="2018-02-08T07:31:11.273" Score="0" Body="&lt;p&gt;Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. The feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn for instance), the variability of those probabilities with respect to the classes should express the importance of those features.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-08T07:31:11.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27585" PostTypeId="1" CreationDate="2018-02-08T08:06:06.603" Score="0" ViewCount="875" Body="&lt;p&gt;I have a dataset which has the following 5 columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date, hour, day_of_week, street_id, counts&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My dataset has information about the number of cars that each street (same city) has in a given hour of a certain date, and I want to predict the traffic count that a certain street has in a given hour of a certain date.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I could use certain variables depending on the day and hour that I want to predict, for example, if a want to predict the traffic count of a working Wednesday:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Results of others working days&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Results of others Wednesdays&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to use Spark MLlib to perform the prediction because I have experience with Spark and I have large datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How you deal with this kind of problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="45933" LastEditorUserId="29575" LastEditDate="2018-02-08T13:37:13.440" LastActivityDate="2024-01-30T11:04:35.527" Title="How to Predict/Forecast street's Traffic based on previous values?" Tags="|apache-spark|prediction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27586" PostTypeId="1" AcceptedAnswerId="27587" CreationDate="2018-02-08T09:13:24.217" Score="12" ViewCount="4956" Body="&lt;p&gt;I have just completed the machine learning for R course on cognitiveclass.ai and have begun experimenting with randomforests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have made a model by using the &quot;randomForest&quot; library in R. The model classifies by two classes, good, and bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that when a model is overfit, it performs well on data from its own trainingset but badly on out-of-sample data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train and test my model I have shuffled and split the complete dataset into 70% for training and 30% for testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: I am getting a 100% accuracy out of the prediction done on the testing set. Is this bad? It seems too good to be true.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is waveform recognition on four on each other depending waveforms. The features of the dataset are the cost results of Dynamic Time Warping analysis of waveforms with their target waveform.&lt;/p&gt;&#xA;" OwnerUserId="45937" LastEditorUserId="29575" LastEditDate="2018-02-09T01:23:19.727" LastActivityDate="2019-01-17T18:40:22.130" Title="Is a 100% model accuracy on out-of-sample data overfitting?" Tags="|r|random-forest|prediction|overfitting|" AnswerCount="2" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="27587" PostTypeId="2" ParentId="27586" CreationDate="2018-02-08T09:27:45.480" Score="29" Body="&lt;p&gt;High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You didn't split the data properly and the validation data also occured in your training data, meaning it does indicate overfitting because you are not measuring generalization anymore&lt;/li&gt;&#xA;&lt;li&gt;You use some feature engineering to create additional features and you might have introduced some target leakage, where your rows are using information from it's current target, not just from others in your training set&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-08T09:27:45.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27588" PostTypeId="1" CreationDate="2018-02-08T10:04:45.670" Score="1" ViewCount="37633" Body="&lt;pre&gt;&lt;code&gt;1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone please tell me how to load this data into a dataframe from .dat file. The data is given such that attribute number:value. I want only the values into  the dataframe&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2019-03-09T21:33:45.253" Title="How to load this data from .dat into dataframe using python" Tags="|python|pandas|dataframe|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27590" PostTypeId="2" ParentId="27422" CreationDate="2018-02-08T10:14:05.253" Score="2" Body="&lt;p&gt;This question will invite a lot of opinions and it will be very specific to the problem that you are looking at. But there's no harm in laying out the popular options - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Start with the data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;a. Images&lt;/strong&gt;&#xA; - If you are working with images and plan to use a classifier like &lt;code&gt;ConvNet&lt;/code&gt;, PCA probably isnt a very good idea. You should just resize the image using popular libs like &lt;code&gt;opencv&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;b. Unstructured Text&lt;/strong&gt;&#xA; - I have personally looked at a lot of free text problems, and my goto method is using &lt;code&gt;Cosine Distance&lt;/code&gt; after doing a &lt;code&gt;TruncatedSVD&lt;/code&gt; (dimensionality reduction) on the bag-of-words data (or any way of structuring for that matter)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;c. Structured&lt;/strong&gt;&#xA; - Usually PCA is the go-to solution, but only if you are ok with losing the interpretability in your downstream predictive models. &lt;code&gt;Scree Plots&lt;/code&gt; are a good way to evaluate how good a PCA exercise was / how many Principal Components you need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Methods&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;a. Auto-encoders&lt;/strong&gt;&#xA; - These are neural-network based dimensionality reduction technique and I haven't looked at a single production instance where in a predictive modelling workflow Auto-encoders outperformed PCA. I guess we just have to wait for the right use-case&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;b. Distance metrics&lt;/strong&gt; - &lt;code&gt;Euclidean&lt;/code&gt; is no-doubt the most popular one, but &lt;code&gt;Cosine&lt;/code&gt; catches up well to free text problems. You also have Hamming distance, Mahalanobis distance and countless others but its really upto you to try that. I haven't come across a case where these were performing better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While we are talking about distance metrics, also look at &lt;code&gt;DBScan&lt;/code&gt; for clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;code&gt;Silhouette Distance&lt;/code&gt; is another effective way of measuring how good your clusters did.&lt;/p&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2018-02-08T10:14:05.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27591" PostTypeId="2" ParentId="418" CreationDate="2018-02-08T10:16:45.297" Score="3" Body="&lt;p&gt;With the little information you have provided regarding the nature of your data, I would advise you to follow the following approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Convert text data into categories. You can try different alternatives for how much information the categories should contain, but specific categories have to exist for each variable. As an example, I will assume a variable that came from a text field of a survey questionnaire regarding preferable way of people to get to work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first, we need to make sure that answers with similar meaning are written on the same way and belong to the same category (e.g. &quot;by bike&quot;, &quot;cycling&quot;, &quot;by bicycle&quot; all have the same meaning). Then you can try further merging into less detailed categories (e.g. merge &quot;tram&quot;, &quot;metro&quot; and &quot;bus&quot; into &quot;Means of public transport&quot;) or even more (e.g. &quot;Walking&quot;, &quot;Jogging&quot;, &quot;Cycling&quot; into &quot;Physical activity&quot;) depending on what you are trying to find out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can even put some different combinations in your dataset and then the next steps will determine which ones will be used for the analysis. In cases where the text data can be &quot;translated&quot; in ordered variables make sure you do this (e.g. if you have &quot;small, medium, high&quot; transform it to &quot;1,2,3&quot;).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Turn your categorical variables (not the ordinal ones) into dummy (binary) variables. Most of classification/feature selection algorithms do this automatically, but make sure this is the case with the ones you select. I realise that the dimensionality of the data will become quite big at this point, but this will be handled in the next step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply a feature selection/dimensionality reduction technique on&#xA;your data. You can find a useful review of such techniques &lt;a href=&quot;https://pdfs.semanticscholar.org/310e/a531640728702fce6c743c1dd680a23d2ef4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. If you are using Python, sklearn tools give you a lot of options (see more details &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;). Make sure you use a technique that also considers multicollinearity. I would try Principal Component Analysis or a tree-based algorithm.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For classifying the data, I would go with Decision Tree Classifier (also available &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;via sklearn&lt;/a&gt;). It also performs feature selection setting importance weights to the features. You can set the level of detail on the generated tree depending on your options (e.g. max_depth, min_samples_split) Make sure to adjust the level of detail based on cross-validation to avoid overfitting.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45221" LastEditorUserId="29575" LastEditDate="2018-02-08T13:34:42.773" LastActivityDate="2018-02-08T13:34:42.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27594" PostTypeId="2" ParentId="27585" CreationDate="2018-02-08T10:28:06.143" Score="0" Body="&lt;p&gt;This looks like a &lt;code&gt;Time Series&lt;/code&gt; problem. So based on a variable's past values, you try to predict the future values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually an &quot;unheard of&quot; problem with Spark, but you are in luck ; &lt;a href=&quot;https://github.com/sryza/spark-timeseries&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;spark-ts&lt;/code&gt;&lt;/a&gt; library seems to be doing what you need, so you don't need to code your own using MLlib. I recommend you try it out and then circle back to something in MLlib if things don't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They have introduced a &lt;a href=&quot;https://blog.cloudera.com/blog/2015/12/spark-ts-a-new-library-for-analyzing-time-series-data-with-apache-spark/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;TimeSeriesRDD&lt;/code&gt;&lt;/a&gt; and once you can encode your data in this data structure (Note that this still behaves like a normal &lt;code&gt;RDD&lt;/code&gt;), you can play around with the models available. For example, implementing the &lt;code&gt;ARIMA&lt;/code&gt; model would be as simple as  - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;val arimaModel = ARIMA.fitModel(1, 0, 1, ts)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope that helps!&lt;/p&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2018-02-08T10:28:06.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27597" PostTypeId="1" AcceptedAnswerId="27603" CreationDate="2018-02-08T10:49:20.787" Score="3" ViewCount="1291" Body="&lt;p&gt;Suppose I have 2 classes. One class has 16 samples and the other class has 435 samples. Is it justified to oversample the class with 16 sample to have a 435 number of samples? Or is it better to undersample the class with 435 samples? If so, what should be the number of samples after the undersampling is carried out?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="85045" LastEditDate="2021-02-22T11:20:53.150" LastActivityDate="2021-02-22T11:20:53.150" Title="What is the allowable limit of oversampling?" Tags="|machine-learning|classification|dataset|class-imbalance|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27598" PostTypeId="1" CreationDate="2018-02-08T11:04:52.610" Score="9" ViewCount="385" Body="&lt;p&gt;Picture below shows 7 points around the origin. One of them has been selected by a human based on rules and experience and is colored red (the one in the lower left quadrant).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oJssk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oJssk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we have over 1000 of these sets of points and for each set a human has selected a single point. These conditions apply to all sets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Each set has about 3 - 10 points&lt;/li&gt;&#xA;&lt;li&gt;There are no outliers&lt;/li&gt;&#xA;&lt;li&gt;Points can have positive and negative values&lt;/li&gt;&#xA;&lt;li&gt;No mistakes were made when selecting a point&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My question is: Does there exists a machine learning algorithm to learn from these sets and human made selections so it can automatically decide which point to select when a new set of points is given? This new set satisfies the first 3 conditions from above of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 final remarks:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The example I gave is just a randomly constructed example by me to support the idea about points in a plane around the origin together with a selected one. In real life there might be more structure but for now I am curious and would like to know what is possible for this case.   &lt;/li&gt;&#xA;&lt;li&gt;Would variations be possible? Say it is about 2 selected points or you have circles with a given radius instead of points.  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45901" LastEditorUserId="29575" LastEditDate="2018-02-12T15:23:36.793" LastActivityDate="2018-02-12T15:23:36.793" Title="Which algorithm to apply for choosing the right point" Tags="|machine-learning|" AnswerCount="2" CommentCount="17" ContentLicense="CC BY-SA 3.0" />
  <row Id="27599" PostTypeId="1" CreationDate="2018-02-08T11:40:10.353" Score="1" ViewCount="1376" Body="&lt;p&gt;I'm reading the book &quot;Bayesian Analysis with Python&quot; and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic coin flipping example. In this example we set a prior on the probability of obtaining a head, then given some data our likelihood is the binomial distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So from my understanding using grid search we will break the interval $[0,1]$ into chunks, in our case 100. We have a prior probability $P(\theta=\theta_{0})$ for each of the points in the discretised grid we multiply this by $P(D|\theta=\theta_{0})$ and we multiply these together to get an un-normalised estimate at each location. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def posterior_grid_approx(grid_points=100, heads=6, tosses=9):&#xA;    grid = np.linspace(0, 1, grid_points)&#xA;    prior=np.repeat(5,grid_points)&#xA;    likelihood=stats.binom.pmf(heads, tosses, grid)&#xA;    unstd_posterior = likelihood * prior&#xA;    posterior = unstd_posterior / unstd_posterior.sum()    &#xA;    return grid, posterior&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't understand why the prior is set to a list of $5$'s isn't this meant to be a probability value? From my understanding they've broken the grid into 100 different points obtained a binomial pmf for each of these locations and then multiplied the number by 5. How is this a valid prior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd really appreciate if someone could clearly explain the reasoning behind this as i like to understand the code as clearly as the concept. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="40659" LastActivityDate="2018-02-08T13:24:44.080" Title="Creating a posterior distribution for classic coin flipping in python using grid search" Tags="|python|statistics|bayesian|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27600" PostTypeId="2" ParentId="27574" CreationDate="2018-02-08T12:39:49.053" Score="2" Body="&lt;h1&gt;Extending a small dataset comprised of images&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;A deep learning algorithm will learn a mapping function from your input space to your outputs. The variations in your input images will be learned within this function. Thus, you will want to consider this fact when you augment your dataset. The distribution of your input features should be concise as to what you plan to model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are thus left balancing between adding data and adding variability to your input space. For example, it might not be worth it to rotate by 180 degrees when trying to generate Dragon Ball characters. Their heads should not be where their feet are. You would want your network to understand that the bottom of the image should contain some strange space boots, and the top exotic haircuts or bald heads. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Augmenting images&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Here are some useful transformations that you can use to get more data&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Apply transformations (rotations, translations)&lt;/li&gt;&#xA;&lt;li&gt;Mirror the image&lt;/li&gt;&#xA;&lt;li&gt;Add distortions &lt;/li&gt;&#xA;&lt;li&gt;Change zoom factor&lt;/li&gt;&#xA;&lt;li&gt;Add blurring thus to better generalize the input data&lt;/li&gt;&#xA;&lt;li&gt;Invert colors (skew, add brightness, etc.)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;In Keras&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In Keras you can use the &lt;code&gt;ImageDataGenerator&lt;/code&gt; functions. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.datasets import mnist&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;&#xA;# load data&#xA;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;X_train = X_train.astype('float32')&#xA;&#xA;# set up your data generator&#xA;datagen = ImageDataGenerator(&#xA;   featurewise_center=True,&#xA;   featurewise_std_normalization=True,&#xA;   rotation_range=60,&#xA;   width_shift_range=0.2,&#xA;   height_shift_range=0.2,&#xA;   shear_range=0.2,&#xA;   zoom_range=0.2,&#xA;   horizontal_flip=True,&#xA;   vertical_flip = True)&#xA;&#xA;# Fit the generator using your data&#xA;datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))&#xA;&#xA;# let's look at some generated images&#xA;image = X_train[5]&#xA;&#xA;plt.figure(figsize=(12,12))&#xA;plt.subplot(4, 4, 1)&#xA;plt.imshow(image.reshape((28,28)),  cmap='gray')&#xA;for j in range(15):&#xA;   augmented = datagen.random_transform(image.reshape((28,28,1)))&#xA;    plt.subplot(4, 4, j+2)&#xA;    plt.imshow(augmented.reshape((28,28)),  cmap='gray')&#xA;plt.tight_layout()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-08T12:39:49.053" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27601" PostTypeId="2" ParentId="27599" CreationDate="2018-02-08T13:24:44.080" Score="3" Body="&lt;h1&gt;In short&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The problem assumes a uniform prior distribution function. All possible $P(H)$ are equally likely. Because they are standardizing the probability distribution function at the end it does not matter what value is placed in &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prior=np.repeat(1,grid_points)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Likelihood function&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The likelihood function answers the question, how probable is the prior which we assumes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, when flipping a fair coin we can assume that the probability of getting a heads is $P(H) = 0.5$. Now, if we do an experiment and in 3 flips we observe 3 heads then $P(HHH|P(H)=0.5) = 0.5^3$. This tells us that their is only a 0.125 chance that we selected the correct prior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Posterior \propto Likelihood \times Prior$&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The code&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;As you observed it is strange that they set their entire list with the value $5$. However this will have no effect on the results because the before last line standardizes the results. Try changing the $5$ to a different value and you will see that your results remain unchanged. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's go through the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grid = np.linspace(0, 1, grid_points)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This creates an array between $0$ and $1$. This is the different probabilities we will assume to calculate our likelihood. For example if we use 5 &lt;code&gt;grid_points&lt;/code&gt; we have $[0, 0.25, 0.5, 0.75, 1]$. So we will assume that $P(H)$ is equal to these values in turn and get the resulting likelihood for each value. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;likelihood=stats.binom.pmf(heads, tosses, grid)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can now set out prior however we wish. We can assume that the coin is fair, or we can have no clue and just assume that it's a uniform distribution. This is essentially what they are doing here. The value they chose makes no difference in the results. However, I think for clarity making it equal to 1 is more obvious.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prior=np.repeat(1,grid_points)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To calculate the posterior we multiply the prior by the likelihood&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;unstd_posterior = likelihood * prior&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then to get a probability distribution function we standardize the function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;posterior = unstd_posterior / unstd_posterior.sum() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-08T13:24:44.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27602" PostTypeId="1" CreationDate="2018-02-08T13:36:32.367" Score="4" ViewCount="571" Body="&lt;p&gt;(Please pardon the pun in the question title.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm currently teaching an introductory class to data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the differences between&#xA;a left join, a right join, an inner join and a full join.&#xA;However, I am not sure how to explain to students when&#xA;a particular join (say an inner join)&#xA;is the appropriate join for that particular situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for three real-life examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In example A, an inner join is the most appropriate.&lt;/li&gt;&#xA;&lt;li&gt;In example B, a left/right join is the most appropriate.&lt;/li&gt;&#xA;&lt;li&gt;In example C, a full join is the most appropriate.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't need all the details,&#xA;but I do want the big picture ideas that show that&#xA;in that real-life example, this join is most appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am teaching using the R package dplyr,&#xA;so I have used the names of the &lt;a href=&quot;http://dplyr.tidyverse.org/reference/join.html&quot; rel=&quot;noreferrer&quot;&gt;dplyr join functions&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33777" LastEditorUserId="33777" LastEditDate="2018-02-08T13:58:49.630" LastActivityDate="2018-02-14T11:33:38.527" Title="The right join for the right situation?" Tags="|sql|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27603" PostTypeId="2" ParentId="27597" CreationDate="2018-02-08T13:51:14.753" Score="2" Body="&lt;p&gt;This depends on the nature of your data. If you can effectively simulate 435 samples using any given oversampling methods such as SMOTE or ADASYN for instance, then I would say oversampling would be better. Because it would provide data for various scenarios. But if exact replication is not possible and if replicated there might be an inherent problem with the model then you should choose to undersample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good example for undersampling would be that for example if you're conducting a scientific experiment and the data you have are limited to a few scenarios and the other data is yet to be fully observed, you would choose to undersample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oversampling would be the reverse case, where you can effectively simulate data for various classes based on some parameter and the generated data can mimic actual data and scenarios, then you should oversample.&lt;/p&gt;&#xA;" OwnerUserId="30107" LastActivityDate="2018-02-08T13:51:14.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27604" PostTypeId="1" CreationDate="2018-02-08T13:55:34.887" Score="2" ViewCount="1454" Body="&lt;p&gt;I have a model with this summary:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;___________________________&#xA;Layer (type)                     Output Shape          Param #     Connected to                     &#xA;====================================================================================================&#xA;input_1 (InputLayer)             (None, 30, 37)        0                                            &#xA;____________________________________________________________________________________________________&#xA;s0 (InputLayer)                  (None, 128)           0                                            &#xA;____________________________________________________________________________________________________&#xA;bidirectional_1 (Bidirectional)  (None, 30, 128)       52224       input_1[0][0]                    &#xA;____________________________________________________________________________________________________&#xA;repeat_vector_1 (RepeatVector)   (None, 30, 128)       0           s0[0][0]                         &#xA;                                                                   lstm_1[0][0]                     &#xA;                                                                   lstm_1[1][0]                     &#xA;                                                                   lstm_1[2][0]                     &#xA;                                                                   lstm_1[3][0]                     &#xA;                                                                   lstm_1[4][0]                     &#xA;                                                                   lstm_1[5][0]                     &#xA;                                                                   lstm_1[6][0]                     &#xA;                                                                   lstm_1[7][0]                     &#xA;                                                                   lstm_1[8][0]                     &#xA;____________________________________________________________________________________________________&#xA;concatenate_1 (Concatenate)      (None, 30, 256)       0           bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[0][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[1][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[2][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[3][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[4][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[5][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[6][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[7][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[8][0]            &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   repeat_vector_1[9][0]            &#xA;____________________________________________________________________________________________________&#xA;dense_1 (Dense)                  (None, 30, 1)         257         concatenate_1[0][0]              &#xA;                                                                   concatenate_1[1][0]              &#xA;                                                                   concatenate_1[2][0]              &#xA;                                                                   concatenate_1[3][0]              &#xA;                                                                   concatenate_1[4][0]              &#xA;                                                                   concatenate_1[5][0]              &#xA;                                                                   concatenate_1[6][0]              &#xA;                                                                   concatenate_1[7][0]              &#xA;                                                                   concatenate_1[8][0]              &#xA;                                                                   concatenate_1[9][0]              &#xA;____________________________________________________________________________________________________&#xA;attention_weights (Activation)   (None, 30, 1)         0           dense_1[0][0]                    &#xA;                                                                   dense_1[1][0]                    &#xA;                                                                   dense_1[2][0]                    &#xA;                                                                   dense_1[3][0]                    &#xA;                                                                   dense_1[4][0]                    &#xA;                                                                   dense_1[5][0]                    &#xA;                                                                   dense_1[6][0]                    &#xA;                                                                   dense_1[7][0]                    &#xA;                                                                   dense_1[8][0]                    &#xA;                                                                   dense_1[9][0]                    &#xA;____________________________________________________________________________________________________&#xA;dot_1 (Dot)                      (None, 1, 128)        0           attention_weights[0][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[1][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[2][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[3][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[4][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[5][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[6][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[7][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[8][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;                                                                   attention_weights[9][0]          &#xA;                                                                   bidirectional_1[0][0]            &#xA;____________________________________________________________________________________________________&#xA;c0 (InputLayer)                  (None, 128)           0                                            &#xA;____________________________________________________________________________________________________&#xA;lstm_1 (LSTM)                    [(None, 128), (None,  131584      dot_1[0][0]                      &#xA;                                                                   s0[0][0]                         &#xA;                                                                   c0[0][0]                         &#xA;                                                                   dot_1[1][0]                      &#xA;                                                                   lstm_1[0][0]                     &#xA;                                                                   lstm_1[0][2]                     &#xA;                                                                   dot_1[2][0]                      &#xA;                                                                   lstm_1[1][0]                     &#xA;                                                                   lstm_1[1][2]                     &#xA;                                                                   dot_1[3][0]                      &#xA;                                                                   lstm_1[2][0]                     &#xA;                                                                   lstm_1[2][2]                     &#xA;                                                                   dot_1[4][0]                      &#xA;                                                                   lstm_1[3][0]                     &#xA;                                                                   lstm_1[3][2]                     &#xA;                                                                   dot_1[5][0]                      &#xA;                                                                   lstm_1[4][0]                     &#xA;                                                                   lstm_1[4][2]                     &#xA;                                                                   dot_1[6][0]                      &#xA;                                                                   lstm_1[5][0]                     &#xA;                                                                   lstm_1[5][2]                     &#xA;                                                                   dot_1[7][0]                      &#xA;                                                                   lstm_1[6][0]                     &#xA;                                                                   lstm_1[6][2]                     &#xA;                                                                   dot_1[8][0]                      &#xA;                                                                   lstm_1[7][0]                     &#xA;                                                                   lstm_1[7][2]                     &#xA;                                                                   dot_1[9][0]                      &#xA;                                                                   lstm_1[8][0]                     &#xA;                                                                   lstm_1[8][2]                     &#xA;____________________________________________________________________________________________________&#xA;dense_2 (Dense)                  (None, 11)            1419        lstm_1[0][0]                     &#xA;                                                                   lstm_1[1][0]                     &#xA;                                                                   lstm_1[2][0]                     &#xA;                                                                   lstm_1[3][0]                     &#xA;                                                                   lstm_1[4][0]                     &#xA;                                                                   lstm_1[5][0]                     &#xA;                                                                   lstm_1[6][0]                     &#xA;                                                                   lstm_1[7][0]                     &#xA;                                                                   lstm_1[8][0]                     &#xA;                                                                   lstm_1[9][0]                     &#xA;====================================================================================================&#xA;Total params: 185,484&#xA;Trainable params: 185,484&#xA;Non-trainable params: 0&#xA;____________________________________________________________________________________________________&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The model is further summarised as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fB4UA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fB4UA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the &quot;attention&quot; block summarised as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/inKss.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/inKss.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input is a fuzzy date, e.g. &quot;November 17, 1979&quot; (capped at 30 characters) and the output is the 10 character representation &quot;YYYY-mm-dd&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I would like to plot the values of the &lt;code&gt;attention_weights&lt;/code&gt; layer.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to see which part of &quot;Saturday, 17th November, 1979&quot; the network &quot;looks at&quot; when it predicts each of &lt;code&gt;YYYY&lt;/code&gt;, &lt;code&gt;mm&lt;/code&gt;, and &lt;code&gt;dd&lt;/code&gt;. I'm expecting to see it ignores the day (&quot;Saturday&quot;) completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried following the &lt;a href=&quot;https://keras.io/getting-started/faq/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras documentation&lt;/a&gt; for obtaining the output of an intermediate layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the attention node has 10 inputs, so I have to grab each of those:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f = K.function(model.inputs, [model.get_layer('attention_weights').get_output_at(t) for t in range(10)])&#xA;r = f([source, np.zeros((1,128)), np.zeros((1,128))])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With &lt;code&gt;source&lt;/code&gt; e.g. &quot;17 November 1979&quot; encoded as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[[ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]&#xA;  [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.&#xA;    1.]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;r&lt;/code&gt; is then a matrix of shape &lt;code&gt;(10,1,30,1)&lt;/code&gt; and the attention map I'm plotting it thus:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;attention_map = np.zeros((10, 30))&#xA;for t in range(10):&#xA;    for t_prime in range(30):&#xA;        attention_map[t][t_prime] = r[t][0,t_prime,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;...but all the values are the same! I'm expecting &lt;em&gt;some&lt;/em&gt; variation.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also tried adding &lt;code&gt;K.learning_phase()&lt;/code&gt; to no avail. What am I doing wrong? &lt;/p&gt;&#xA;" OwnerUserId="45945" LastEditorUserId="45945" LastEditDate="2018-02-08T14:03:02.903" LastActivityDate="2024-01-01T05:00:30.067" Title="Keras intermediate layer (attention model) output" Tags="|python|keras|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27605" PostTypeId="2" ParentId="27602" CreationDate="2018-02-08T13:57:39.377" Score="1" Body="&lt;p&gt;You can explain them using Venn diagram. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nok0J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nok0J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AMLhb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AMLhb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/X1Dtj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X1Dtj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See more details about the query and examples &lt;a href=&quot;https://www.w3schools.com/sql/sql_join_full.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-02-08T13:57:39.377" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27606" PostTypeId="2" ParentId="27602" CreationDate="2018-02-08T14:27:33.923" Score="8" Body="&lt;p&gt;You're an online retailer. Like Amazon.&#xA;You keep your purchase data for different categories of items in different tables, but all website users have one account with one ID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Inner Join:&lt;/strong&gt;&#xA;You have two datasets, one with User IDs and purchases of clothing data, the second dataset has User IDs and purchases of books data. You want to find out who purchases both clothes and books from your site. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You &lt;em&gt;inner join&lt;/em&gt; to find the User IDs and purchases for people who bought clothing and books. Any users who didn't buy both of these items will be 'dropped' from the final table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Left Join&lt;/strong&gt;&#xA;You have one dataset with User IDs and account info(e.g. age, name) of all users. You want to build out this table to include some sales data, without dropping people from your complete list of users. So you would left join your purchase dataset to your account dataset (with User ID as the key). Any user who has made no purchases won't be dropped from your final table. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Right Join&lt;/strong&gt; &#xA;The opposite of left join. E.g. you want to keep all purchase data, even if for some reason your Account Info table does not contain the ID of the user who made that purchase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Full (Outer) Join&lt;/strong&gt; &#xA;You want to Join account info and purchases tables, but you don't want to lose any entries in either table. So your final table will include Users who have made no purchases, and (perhaps mysteriously) purchases who have no users associated with them.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2018-02-08T14:27:33.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27607" PostTypeId="2" ParentId="27598" CreationDate="2018-02-08T14:54:21.567" Score="6" Body="&lt;p&gt;This is a fascinating problem!&#xA;Two things make it especially challenging:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How should we compare two point sets? Classical problems in Machine Learning have a fixed number of attributes, and these attributes are not interchangeable: For example, I might have data on different persons with attributes &lt;code&gt;age&lt;/code&gt; and &lt;code&gt;height&lt;/code&gt; (in centimeters). Every sample has one entry for each, and of course &lt;code&gt;(age, height) = (22, 180)&lt;/code&gt; is not the same as &lt;code&gt;(age, height) = (180, 22)&lt;/code&gt;.&#xA;Neither is true in your problem. A point set has between 3 and 10 points, and the order in which we enter the points should not make a difference when comparing two point sets.&lt;/li&gt;&#xA;&lt;li&gt;How do we make a prediction? Say we have found a way to pick point sets from our training set that are similar to your point set above. We face the problem that our prediction must be one of the 7 points in your picture; but none of these points might be contained in the similar point sets.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Let me outline an algorithm that deals with both challenges. The prediction accuracy is not very good; but maybe you see a way how it can be improved. And at least it predicts &lt;em&gt;something&lt;/em&gt;, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Simulating samples&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To be able to test the algorithm, I wrote functions that generate samples and labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Generating samples:&lt;/em&gt;&#xA;Each sample contains between 3 and 10 points. The number of points is random, drawn from a uniform distribution. Each point is of the form &lt;code&gt;(x_coordinate, y_coordinate)&lt;/code&gt;. The coordinates are again random, drawn from a normal distribution.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from random import randint&#xA;&#xA;def create_samples(number_samples, min_points, max_points):&#xA;&#xA;    def create_single_sample(min_points, max_points):&#xA;        n = randint(min_points, max_points)&#xA;        return np.array([np.random.normal(size=2) for _ in range(n)]) &#xA;&#xA;    return np.array([create_single_sample(min_points, max_points) for _ in range(number_samples)])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Generating labels:&lt;/em&gt; As a toy example, let us assume that the rule for choosing a point is: Always pick the point that is closest to &lt;code&gt;(0, 0)&lt;/code&gt;, where 'closest' should be understood in terms of the Euclidean norm.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def decision_function_minnorm(sample):&#xA;    norms = np.apply_along_axis(np.linalg.norm, axis=1, arr=sample)&#xA;    return sample[norms.argmin()]&#xA;&#xA;def create_labels(samples, decision_function):&#xA;    return np.array([decision_function(sample) for sample in samples])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can now create our train and test sets:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_train, n_test = 1000, 100&#xA;dec_fun = decision_function_minnorm&#xA;&#xA;X_train = create_samples(number_samples=n_train, min_points=3, max_points=10)&#xA;X_test = create_samples(number_samples=n_test, min_points=3, max_points=10)&#xA;y_train = create_labels(X_train, dec_fun)&#xA;y_test = create_labels(X_test, dec_fun)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Comparing point sets via Hausdorff distance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us tackle the first problem: How should we compare different point sets?&#xA;The number of points in the point sets is different. &#xA;Also remember that the order in which we write down the points should not matter: Comparing to the point set &lt;code&gt;[(0,0), (1,1), (2,2)]&lt;/code&gt; should yield the same result as comparing to the point set &lt;code&gt;[(2,2), (0,0), (1,1)]&lt;/code&gt;.&#xA;My approach is to compare point sets via their &lt;a href=&quot;https://en.wikipedia.org/wiki/Hausdorff_distance&quot; rel=&quot;noreferrer&quot;&gt;Hausdorff distance&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def hausdorff(A, B):&#xA;&#xA;    def dist_point_to_set(x, A):&#xA;        return min(np.linalg.norm(x - a) for a in A)&#xA;&#xA;    def dist_set_to_set(A, B):&#xA;        return max(dist_point_set(a, B) for a in A)&#xA;&#xA;    return max(dist_set_to_set(A, B), dist_set_to_set(B, A))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Predicting via k-nearest neighbors and averaging&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We now have a notion of distance between point sets.&#xA;This makes it possible to use k-nearest neighbors classification:&#xA;Given a test point set, we find the &lt;code&gt;k&lt;/code&gt; point sets in our training sample that have the smallest Hausdorff distance relative to the test point set, and obtain their labels.&#xA;Now comes the second problem: How do we turn these &lt;code&gt;k&lt;/code&gt; labels into a prediction for the test point set? I took the simplest approach: average the labels and predict the point in the test point set that is closest to the average.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def predict(x, num_neighbors):&#xA;    # Find num_neighbors closest points in X_train.&#xA;    distances_to_train = np.array([hausdorff(x, x_train) for x_train in X_train])&#xA;    neighbors_idx = np.argpartition(distances_to_train, -num_neighbors)[-num_neighbors:]&#xA;&#xA;    # Get labels of the neighbors and calculate the average.&#xA;    targets_neighbors = y_train[neighbors_idx]&#xA;    targets_mean = sum(targets_neighbors) / num_neighbors&#xA;&#xA;    # Find point in x that is closest to targets_mean and use it as prediction.&#xA;    distances_to_mean = np.array([np.linalg.norm(p - targets_mean) for p in x])&#xA;    closest_point = x[distances_to_mean.argmin()]&#xA;&#xA;    return closest_point&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. Testing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything is in place to test the performance of our algorithm.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_neighbors = 70&#xA;successes = 0&#xA;for i, x in enumerate(X_test):&#xA;    print('%d/%d' % (i+1, n_test))&#xA;    prediction = predict(x, num_neighbors)&#xA;    successes += np.array_equal(prediction, y_test[i])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the given decision function and &lt;code&gt;num_neighbors = 70&lt;/code&gt;, we get a prediction accuracy of 84%.&#xA;This is not terribly good, and it is of course specific to our decision function, which seems fairly easy to predict.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To see this, define a different decision function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;decision_function_maxaverage(sample):&#xA;    avgs = (sample[:, 0] + sample[:, 1]) / 2&#xA;    return sample[norms.argmin()]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using this function via &lt;code&gt;dec_fun = decision_function_maxaverage&lt;/code&gt; brings down prediction accuracy to 45%.&#xA;This shows how important it is to think about the decision rules that generate your labels. If you have an idea &lt;em&gt;why&lt;/em&gt; people choose certain points, this will help you find the best algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some ways to improve this algorithm: (1) Use a different distance function instead of Hausdorff distance, (2) use something more sophisticated than k-nearest neighbors, (3) improve how the selected training labels are turned into a prediction.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-08T14:54:21.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27608" PostTypeId="2" ParentId="26435" CreationDate="2018-02-08T15:10:40.510" Score="1" Body="&lt;p&gt;One of the most fundamental assumptions in machine learning is that the training data is 'similar' to the test data. Training makes no sense otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the question is: How similar are reviews of, say, movies and stocks? Maybe somewhat, but not too much.&#xA;Your movie-trained algorithm would certainly be able to deal with statements like 'This company is awesome' or 'Warning, do not buy this stock'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what about 'The stock price will explode/implode' or 'Concerning this stock, I am bearish/bullish'? The words explode, implode, bearish and bullish have probably never been expressions of sentiment in movie reviews.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-08T15:10:40.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27609" PostTypeId="2" ParentId="27586" CreationDate="2018-02-08T15:19:54.827" Score="1" Body="&lt;p&gt;Investigate to see what your most predictive features are. Sometimes you accidentally included your target (or something that is equivalent to your target) amongst your features.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-02-08T15:19:54.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27610" PostTypeId="2" ParentId="9777" CreationDate="2018-02-08T15:32:49.270" Score="2" Body="&lt;p&gt;When to use label encoding versus one-hot encoding.&lt;/p&gt;&#xA;&lt;h3&gt;Tree based methods:&lt;/h3&gt;&#xA;&lt;p&gt;When categorical feature is ordinal label encoding can lead to better quality if it preserves correct order of values. In this case a split made by a tree will divide the feature to values 'lower' and 'higher' that the value chosen for this split.&lt;/p&gt;&#xA;&lt;h3&gt;Non-tree based methods:&lt;/h3&gt;&#xA;&lt;p&gt;One-hot encoding or embedings should be used.&#xA;Unless there is a linear relashionship between the label encoding and the dependent variable non-tree based methods will have a hard time with label encoding.&lt;/p&gt;&#xA;&lt;p&gt;One-hot encoding a categorical feature with huge number of values can lead to high memory consumption. You can use sparse matrices to deal with this problem. You can also ignore a subset of the categories that are rare to decrease the number of new features.&lt;/p&gt;&#xA;" OwnerUserId="45958" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-08T15:58:02.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27611" PostTypeId="1" CreationDate="2018-02-08T16:22:16.650" Score="0" ViewCount="78" Body="&lt;p&gt;I am a data science neophyte struggling to solve a churn prediction problem, i would be grateful to have someone help with the regression model&lt;/p&gt;&#xA;" OwnerUserId="45961" LastActivityDate="2018-02-08T16:46:20.020" Title="Churn Prediction problem" Tags="|regression|prediction|churn|" AnswerCount="1" CommentCount="2" ClosedDate="2018-02-11T22:01:35.470" ContentLicense="CC BY-SA 3.0" />
  <row Id="27612" PostTypeId="2" ParentId="27588" CreationDate="2018-02-08T16:23:57.117" Score="2" Body="&lt;p&gt;Given just one line of the data, it's a little hard to go off of, but I'm assuming you're trying to get at the number after each colon, and the number before it refers to the column name?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If so, you can use read_csv with a little tweaking:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from pandas.compat import StringIO&#xA;&#xA;temp='1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400'&#xA;#after testing replace StringIO(temp) to filename&#xA;&#xA;df = pd.read_csv(StringIO(temp), &#xA;                 sep=&quot;\s+&quot;, #separator whitespace&#xA;                 index_col=0,&#xA;                 header=None) &#xA;&#xA;for c in df.columns.values:&#xA;    df[c] = df[c].apply(lambda x: float(str(x).split(':')[1]))&#xA;&#xA;df.head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which will output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    1           2           3           4       5           6           7           8           9                               &#xA;1   31080.4102  2.871828    5.862267    7.10085 8.283706    -5.427875   -6.667087   -8.888233   28898.9434&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45531" LastEditorUserId="20954" LastEditDate="2019-03-09T21:33:45.253" LastActivityDate="2019-03-09T21:33:45.253" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27613" PostTypeId="1" CreationDate="2018-02-08T16:45:45.757" Score="3" ViewCount="126" Body="&lt;p&gt;Being still a bit new to neural networks, I wish to use some form of machine learning but I am not sure which one is most suited.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a toy-example of what I am working on. Tips are much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming there are chunks of data (not pixels of an image :)) which all are based on a root variable, being 'a'.&#xA;This 'a' is the 'thing' and the other variables are aspects. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also assuming it's a finite set of things and that each thing 'a' has four ('b', 'c' and 'd') aspects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to make a program that learns that thing ('a':1) often occurs with ('b':0.2), ('c' :0.8), and ('d':0.3).&#xA;There is a catch however, I want there to be a certain conditional structure underlying the co-occurrence of a and the respective values of (b, c, d).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('a':1) often     (prob. 0.7) occurs with ('b':0.2), ('c':0.8), and ('d':0.3)&#xA;('a':1) sometimes (prob. 0.6) occurs with ('b':0.2), ('c':0.5), and ('d':0.7)&#xA;('a':1) sometimes (prob. 0.4) occurs with ('b':0.3), ('c':0.4), and ('d':0.2)&#xA;('a':1) rarely    (prob. 0.2) occurs with ('b':0.5), ('c':0.3), and ('d':0.8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a given ('a':2) this would all be different.&#xA;There is no implied necessity for the values of b, c and d to sum to 1. They are mere placeholder values representing aspects (0.2 could be green, 0.8 purple, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to make sure the program understands that there is an inherent dependency between b, c and d, as well as an 'a':x given (b, c, d). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone has suggestions based on this, I would be very grateful.&#xA;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I am working in python and was thinking about dictionaries as the data-structure, but if there are better solutions (also depending on the algorithm perhaps) I am all ears. &lt;/p&gt;&#xA;" OwnerUserId="45962" LastActivityDate="2018-02-13T15:31:36.300" Title="Best machine learning algorithm for understanding specific conditional structures" Tags="|machine-learning|neural-network|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27614" PostTypeId="2" ParentId="27611" CreationDate="2018-02-08T16:46:20.020" Score="2" Body="&lt;p&gt;I really like &lt;a href=&quot;https://ragulpr.github.io/2016/12/22/WTTE-RNN-Hackless-churn-modeling/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; because it gives an overview of common approaches to churn prediction as well as a new state-of-the-art method.  There is also code available. However, the state-of-the-art method uses a custom loss function built on top of recurrent neural nets, which is a complex place to dive in.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-02-08T16:46:20.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27615" PostTypeId="1" AcceptedAnswerId="27616" CreationDate="2018-02-08T16:53:24.653" Score="44" ViewCount="67375" Body="&lt;p&gt;I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.&lt;/p&gt;&#xA;&lt;p&gt;Now when using the classifier, should I normalize test data as well. I feel that the basic aim of normalization is to make the learning algorithm give more weight to more important features while learning. So once it has been trained, it already knows which features are important and which are not. So is there any need to apply normalization to test data as well?&lt;/p&gt;&#xA;&lt;p&gt;I am new to this field. So please ignore if the question appears silly?&lt;/p&gt;&#xA;" OwnerUserId="37063" LastEditorUserId="135950" LastEditDate="2022-06-07T14:58:09.523" LastActivityDate="2022-06-07T14:58:09.523" Title="Should we apply normalization to test data as well?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27616" PostTypeId="2" ParentId="27615" CreationDate="2018-02-08T17:04:25.957" Score="70" Body="&lt;p&gt;Yes you need to apply normalisation to test data, if your algorithm works with or needs normalised training data*. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is because your model works on the representation given by its input vectors. The scale of those numbers is part of the representation. This is a bit like converting between feet and metres . . . a model or formula would work with just one type of unit normally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not only do you need normalisation, but you should apply the exact same scaling as for your training data. That means storing the scale and offset used with your training data, and using that again. A common beginner mistake is to separately normalise your train and test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python and SKLearn, you might normalise your input/X values using the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html&quot; rel=&quot;noreferrer&quot;&gt;Standard Scaler&lt;/a&gt; like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = StandardScaler()&#xA;train_X = scaler.fit_transform( train_X )&#xA;test_X = scaler.transform( test_X )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note how the conversion of &lt;code&gt;train_X&lt;/code&gt; using a function which fits (figures out the params) then normalises. Whilst the &lt;code&gt;test_X&lt;/code&gt; conversion just transforms, using the same params that it learned from the train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tf-idf normalisation you are applying should work similarly, as it learns some parameters from the data set as a whole (frequency of words in all documents), as well as using ratios found in each document.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Some algorithms (such as those based on decision trees) do not need normalised inputs, and can cope with features that have different inherent scales.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2019-11-02T17:08:12.807" LastActivityDate="2019-11-02T17:08:12.807" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="27617" PostTypeId="2" ParentId="27615" CreationDate="2018-02-08T17:07:40.010" Score="13" Body="&lt;p&gt;Definitely you should normalize your data. You normalize the data for the following aims:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For having different features in same scale, which is for accelerating learning process. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For caring different features fairly without caring the scale.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After training, your learning algorithm has learnt to deal with the data in scaled form, so you have to normalize your test data with the normalizing parameters used for training data.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-08T17:07:40.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27618" PostTypeId="1" CreationDate="2018-02-08T17:19:58.883" Score="3" ViewCount="3077" Body="&lt;p&gt;As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my goal is that conceptually similar vectors (regardless of whether they represent documents or words) are close to each other, then would it be better to train them simultaneously?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, a document about Godzilla should be similar to the word Godzilla (as well as to other documents about Godzilla) as well as to the word Japan.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, could such a result being achieved with a typical implementation, when including single words as documents into the training data?&lt;/p&gt;&#xA;" OwnerUserId="45963" LastActivityDate="2018-02-28T14:30:37.193" Title="Training Doc2Vec and Word2Vec at the same time" Tags="|machine-learning|deep-learning|word2vec|word-embeddings|similar-documents|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27619" PostTypeId="1" CreationDate="2018-02-08T18:28:13.573" Score="2" ViewCount="384" Body="&lt;p&gt;I am doing a PCA as a data exploration step and I realize that the two first principal components capture only 25% of the variance, the ten first principle component capture about 60% of the information, does it worth to interpret those axis knowing that they do not capture enough information&#xA;&lt;a href=&quot;https://i.stack.imgur.com/zftry.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zftry.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45957" LastActivityDate="2018-02-08T19:53:21.293" Title="PCA first dimension do not not capture enough variance" Tags="|pca|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27620" PostTypeId="1" CreationDate="2018-02-08T19:14:04.027" Score="0" ViewCount="886" Body="&lt;p&gt;I am trying to predict the winner of a tennis match from the players participating and their respective ranks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Labels of my data:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Winner&lt;/li&gt;&#xA;&lt;li&gt;Loser&lt;/li&gt;&#xA;&lt;li&gt;Winner rank&lt;/li&gt;&#xA;&lt;li&gt;Loser rank&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The problem is that the columns holding the player names in my data are labeled 'Winner' and 'Loser'. What is the best course of action to render this dataset usable for machine learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could I just assign the 'Winner' column/vector as my target variable, and construct two new columns ('Player1', 'Player2') and populate them with random choices from 'Winner' and 'Loser'?&lt;/p&gt;&#xA;" OwnerUserId="45965" LastActivityDate="2019-09-11T21:01:25.203" Title="How to handle the target variable being in the features" Tags="|python|scikit-learn|pandas|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27621" PostTypeId="2" ParentId="27620" CreationDate="2018-02-08T19:45:51.697" Score="1" Body="&lt;p&gt;I think you need to formalize your problem a little bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to predict the winner of a tennis match, I would do something like this. I will be predicting the probability of Player 1 winning (or Player 2, without loss of generality). Then, this is a simple classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, for the features, I will organize it in a way that uses player 1 or player 2 to represent the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the variables are: &lt;code&gt;players-current-ranking&lt;/code&gt; and &lt;code&gt;face-to-face-win-rate&lt;/code&gt;. Then the feature vector will be something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;player1_current_rank: 10&#xA;player1_2_face_to_face_win_rate: 0.7&#xA;player2_current_rank: 20&#xA;player2_1_face_to_face_win_rate: 0.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can never have target variable in your feature, otherwise it won't make any sense predicting it.&lt;/p&gt;&#xA;" OwnerUserId="14152" LastActivityDate="2018-02-08T19:45:51.697" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27622" PostTypeId="2" ParentId="27619" CreationDate="2018-02-08T19:47:08.980" Score="1" Body="&lt;p&gt;You don't state how many original features there are? where there 10, 50, 50 million? The above 10 principle components capture 62.3% of the variance and so we can be sure that there are more that ten original features. However we will assume then that there aren't many more (lets say circa twenty), lets consider what is going on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With PCA you are hoping to separate the useful information from the noise. By taking only the the first two components in you example you are discarding all other information as noise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends upon the application but I would say that 25% is never enough (I think serious questions would be raised as why you had). So the question becomes then what is going on? Well lets consider the two extreme cases: A set of uncorrelated features and a set of fully correlated features. Consider first three uncorrelated features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 0 0 &#xA;0 1 0 &#xA;0 0 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The total variance is 3. The eigenvalues are &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.000&#xA;1.000&#xA;1.000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each component therefore represents 1/3 of the variance and they are equal. Now let use consider the other extreme, all features are highly correlated, i.e. a correlation matrix of&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1.000 1.000 1.000&#xA;1.000 1.000 1.000&#xA;1.000 1.000 1.000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The eigenvalues are&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3.000&#xA;0.000&#xA;0.000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Therefore the first principle component contains all the information and these three features could be characterised by a single latent variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why is this relevant? Well in your data set your eigenvalues are all of similar magnitude. This suggests that the features in your original data are not very correlated (like the first example above, look at the off diagonal elements your original correlation matrix they are likely close to zero) and therefore it is expect that many principle components would be required since there is little shared information in the original feature space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This of course assume that we only have a small number of original features. If you have 50 million features then capturing 62.3% in only ten latent variables is pretty good,&lt;/p&gt;&#xA;" OwnerUserId="13259" LastEditorUserId="13259" LastEditDate="2018-02-08T19:53:21.293" LastActivityDate="2018-02-08T19:53:21.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27623" PostTypeId="2" ParentId="16640" CreationDate="2018-02-08T19:55:11.817" Score="0" Body="&lt;p&gt;You need to train your classifier on a external larger labelled dataset like &lt;a href=&quot;https://www.kaggle.com/therohk/india-headlines-news-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;. (vocab size 1M+)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may need to slice the data to include only sports articles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use the 8000 articles as the test data for the classifier. &lt;/p&gt;&#xA;" OwnerUserId="44998" LastEditorUserId="29575" LastEditDate="2018-02-08T20:20:45.763" LastActivityDate="2018-02-08T20:20:45.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27624" PostTypeId="1" AcceptedAnswerId="27769" CreationDate="2018-02-08T20:02:21.590" Score="10" ViewCount="18617" Body="&lt;p&gt;Here is my understanding of the difference between a Bernoulli and a Multinomial Naive Bayes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bernoulli explicitly models the presence/absence of a feature, whereas Multinomial doesn't.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is there something that I am missing?&lt;/li&gt;&#xA;&lt;li&gt;Can someone explain why this difference matters intuitively? Perhaps using an example where you would obviously use one instead of the other&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;On the wikipedia page, they say:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Why is that?&lt;/p&gt;&#xA;" OwnerUserId="38887" LastEditorUserId="38887" LastEditDate="2018-02-14T07:51:00.340" LastActivityDate="2018-02-14T09:12:22.417" Title="Difference between Bernoulli and Multinomial Naive Bayes" Tags="|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27625" PostTypeId="1" AcceptedAnswerId="27633" CreationDate="2018-02-08T21:36:45.023" Score="3" ViewCount="1756" Body="&lt;p&gt;I am doing semantic segmentation (multi-class classification of image pixels) using convolutional neural networks (CNN) in Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I am applying this to aerial images of crops (vegetation). In Keras, I successfully developed a workflow to segment/classify different crops for one specific dataset (let's call this dataset &lt;code&gt;rural area #1&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I apply my Keras weights trained on &lt;code&gt;rural area #1&lt;/code&gt; for initializing the training of another dataset &lt;code&gt;rural area #2&lt;/code&gt;? Such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = load_model(&quot;weights_ruralarea1.hdf5&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I will proceed to &lt;code&gt;model.fit&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;rural area #2&lt;/code&gt; dataset has little training images for training the CNN. So would using my weights for &lt;code&gt;rural area #1&lt;/code&gt; be a form of transfer learning? or will it be a form of fine-tuning?&lt;/p&gt;&#xA;" OwnerUserId="45898" LastEditorUserId="29169" LastEditDate="2021-02-13T16:42:15.493" LastActivityDate="2021-02-13T16:42:15.493" Title="How to apply my deep learning model to a new dataset?" Tags="|machine-learning|python|deep-learning|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27626" PostTypeId="1" AcceptedAnswerId="27667" CreationDate="2018-02-08T23:18:48.143" Score="2" ViewCount="1146" Body="&lt;p&gt;I am using pytorch to build DCGAN which i aim to train on custom dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have already posted a question on training DCGAN on small dataset, and of course answer was data augmentation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But i have a problem. I am coding in PyTorch and i want to perform several different transfomrations on existing ImageFolder object which represents my loaded dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Idea was to perform, lets say around 5 different transformations, and after performing each transformation i want to expand my dataset by adding the newly transformed images to it. Here is pseudo code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;list_of_transformations = ['random_crop','center_crop','color_jitter','mirror','grayscale']&#xA;dataset = ImageFolder('path_to_dataset')&#xA;for t in list_of_transformations&#xA;       dataset += dataset.t()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am guessing this might not be the best idea, or is even considered 'stupid', but i am new to Deep Learning and Python in general. Any help would be appreciated! Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="45907" LastEditorUserId="45907" LastEditDate="2018-02-10T08:57:52.703" LastActivityDate="2018-02-10T08:59:53.383" Title="Perform several different torchvision.transforms on ImageFolder object" Tags="|python|deep-learning|gan|pytorch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27628" PostTypeId="1" AcceptedAnswerId="27804" CreationDate="2018-02-09T01:10:44.847" Score="22" ViewCount="17672" Body="&lt;p&gt;Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs?&lt;/p&gt;&#xA;&lt;p&gt;For a simplified example, assume that we have to predict the sequence of characters:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;A B C D E F G H I J K L M N O P Q R S T U V W X Y Z&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Is it bad (or better?) if I keep training my LSTM with the following minibatches:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;A B C D E F G H I J K L M N, backprop, erase the cell&#xA;&#xA;B C D E F G H I J K L M N O, backprop, erase the cell&#xA;&#xA; .... and so on, shifting by 1 every time?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Previously, I always trained it as:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;A B C D E F G H I J K L M N,  backprop, erase the cell&#xA;&#xA;O P Q R S T U V W X Y Z,  backprop, erase the cell&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Instead of shifting by one, would it be better to slide the window by 2 entries instead, etc? What would that mean (in terms of precision/overfitting)?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Also, if I were to do the sliding-window approach in a Feed-forward network, would it result in overfitting? I would assume yes, because the network is exposed to the same information regions for a very long time. For example, it is exposed to &lt;code&gt;E F G H I J K&lt;/code&gt; for a long time.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Please remember that cell state is erased between training batches, so the LSTM will have a &amp;quot;hammer to head&amp;quot; at these times. &lt;strong&gt;It's unable to remember&lt;/strong&gt; what was before OPQRSTUVWXYZ. This means that the LSTM is unable to ever learn that &amp;quot;O&amp;quot; follows the &amp;quot;M&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;So, I thought (thus my entire question), why not to give it intermediate (overlapping) batch in between...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? Ultimately, that would mean a sliding window for an LSTM.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Some useful info I've found after answer was accepted:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;from here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The first word of the English translation is probably highly correlated with the first word of the source sentence. But that means decoder has to consider information from 50 steps ago, and that information needs to be somehow encoded in the vector. Recurrent Neural Networks are known to have problems dealing with such long-range dependencies. In theory, architectures like LSTMs should be able to deal with this, but in practice long-range dependencies are still problematic.&lt;/p&gt;&#xA;&lt;p&gt;For example, researchers have found that reversing the source sequence&#xA;(feeding it backwards into the encoder) produces significantly better&#xA;results because it shortens the path from the decoder to the relevant&#xA;parts of the encoder. &lt;strong&gt;Similarly, feeding an input sequence twice also&#xA;seems to help a network to better memorize things.&lt;/strong&gt; For example, if one training example is &amp;quot;John went home&amp;quot;, you would give &amp;quot;John went home John went home&amp;quot; to the network as one input.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit after accepting the answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Several months after, I am more inclined to use the sliding window approach, as it uses the data better. But in that case you probably don't want to train BCDEFGHIJKLMNO right after ABCDEFGHIJKLMN.  Instead, shuffle your examples, to gradually and uniformly &amp;quot;brush-in&amp;quot; &lt;strong&gt;all&lt;/strong&gt; of the information into your LSTM.  Give it HIJKLMNOPQRSTU after ABCDEFGHIJKLMNO etc.  That's directly related to Catastrophic Forgetting. As always, monitor the Validation and Test set closely, and stop as soon as you see their errors steadily increasing&lt;/p&gt;&#xA;&lt;p&gt;Also, the &amp;quot;hammer to head&amp;quot; issue can be improved, by using Synthetic Gradients. See its benefit here: (linked answer discusses its benefit of long sequences) &lt;a href=&quot;https://datascience.stackexchange.com/a/32425/43077&quot;&gt;https://datascience.stackexchange.com/a/32425/43077&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2023-05-25T20:30:59.197" LastActivityDate="2023-05-25T20:30:59.197" Title="Sliding window leads to overfitting in LSTM?" Tags="|lstm|backpropagation|mini-batch-gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27630" PostTypeId="1" CreationDate="2018-02-09T06:03:47.480" Score="2" ViewCount="95" Body="&lt;p&gt;I'm trying to build a classification model where instances are classified based not on some linear combination of their features, but instead based on whether the features are all within some (unknown) ranges. So, for example, if I had two features, and my instances (features, followed by known classification) are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(0, 5): 0&lt;/li&gt;&#xA;&lt;li&gt;(2, 7): 1&lt;/li&gt;&#xA;&lt;li&gt;(3, 2): 0&lt;/li&gt;&#xA;&lt;li&gt;(4, 4): 1&lt;/li&gt;&#xA;&lt;li&gt;(3, 9): 0&lt;/li&gt;&#xA;&lt;li&gt;(6, 6): 0&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;then perhaps the model would learn that the first feature should be between 1 and 5, and the second feature should be between 3 and 8. (I only want &quot;and&quot;s in the model; I'm not looking for any cases where feature 1 is between a and b OR feature 2 is between c and d.) In essence, the model should find some hyperrectangle in feature space with all edges parallel to feature axes, where 0s are expected to be outside the rectangle and 1s are expected to be inside it. (In the example data, the data points are separable; this is &lt;em&gt;not&lt;/em&gt; the case in my real problem. In the spirit of an SVM or logistic regression, the model should do the best it can.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first stab at this involved trying to do least-squares regression with a hypothesis function made up of two sigmoids for each feature (one &quot;forward&quot;, one &quot;backward&quot;), parameterized to slide back and forth, all of which were multiplied together. Maybe my scipy-fu is weak, but above four features, I started having convergence issues with that approach. Is there some tried-and-true way to solve this kind of classification problem? (Even just a name for this kind of problem would be helpful!)&lt;/p&gt;&#xA;" OwnerUserId="45996" LastEditorUserId="45996" LastEditDate="2018-02-09T16:57:41.833" LastActivityDate="2021-08-26T19:58:41.473" Title="Classify based on features being in ranges?" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27631" PostTypeId="1" AcceptedAnswerId="27641" CreationDate="2018-02-09T06:08:09.090" Score="1" ViewCount="5000" Body="&lt;p&gt;&lt;strong&gt;Problem Statement:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Given the details about a product, we need to map it to its category.&lt;/p&gt;&#xA;&lt;p&gt;Currently we are using Product Name as a &lt;strong&gt;feature&lt;/strong&gt; and Product Category as the &lt;strong&gt;Label&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;There are around 50,000 categories available currently and it will grow in future.&lt;/p&gt;&#xA;&lt;p&gt;I created a small dataset which consisted 20 categories and 100 records for each label. so the total record count is 2000. Using RandomForest I got 92% accuracy.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;So I went on to create a model with 1800 categories[labels] and records for each categories varies from 500-1500. When I ran the same model with new dataset I got only 19% accuracy and more than 50% of the predicted value pointed to the same label.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Dataset Sample:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Product_Combined    Category&#xA;2Pcs Led Light Lamp Strip Dimmer Switch Brightness Adjustable Control 12-24V 8A Arts, Crafts &amp;amp; Sewing | Painting, Drawing &amp;amp; Art Supplies | Drawing | Light Boxes&#xA;10 Pcs 1/4&amp;quot; Male To 1/4&amp;quot; Female Screw Adapter For Tripod Camera Flash Bracket Stand Arts, Crafts &amp;amp; Sewing | Painting, Drawing &amp;amp; Art Supplies | Drawing | Light Boxes&#xA;L-Fine A4 Tracing LED Light Pad Box(13.86x9.45 Inches) with Adjustable Light Intensity for Artists,Drawing, Sketching, Animation    Arts, Crafts &amp;amp; Sewing | Painting, Drawing &amp;amp; Art Supplies | Drawing | Light Boxes&#xA;BZONE Solar Powered Operated Copper Wire LED Fairy Light Decorative String Lights for Indoor Outdoor Home Garden Lawn Patio Party Christmas Valentine''s Day (16.4ft, Pink Color)   Arts, Crafts &amp;amp; Sewing | Painting, Drawing &amp;amp; Art Supplies | Drawing | Light Boxes&#xA;LitEnergy 32.5 Inch Diagonal A2 Tracing Table with LED Light and Paper  Arts, Crafts &amp;amp; Sewing | Painting, Drawing &amp;amp; Art Supplies | Drawing | Light Boxes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import string&#xA;import codecs&#xA;import pandas as pd&#xA;import numpy as np&#xA;&#xA;from sklearn.feature_extraction.text import CountVectorizer&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.model_selection import train_test_split&#xA;from stemming.porter2 import stem&#xA;from sklearn.metrics import confusion_matrix&#xA;&#xA;from nltk.stem import PorterStemmer&#xA;from nltk.corpus import stopwords&#xA;&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.externals import joblib&#xA;&#xA;stop = stopwords.words('english')&#xA;&#xA;data_file = &amp;quot;Book3.txt&amp;quot;&#xA;#Reading the input/ dataset&#xA;data = pd.read_csv( data_file, header = 0, &#xA;    delimiter= &amp;quot;\t&amp;quot;, quoting = 3, encoding = &amp;quot;ISO-8859-1&amp;quot;)&#xA;data = data.dropna()&#xA;#Removing stopwords, punctuation and stemming&#xA;data['Product_Combined'] = data['Product_Combined'].apply(&#xA;    lambda x: ' '.join([word for word in x.split() if word not in (stop)]))&#xA;data['Product_Combined'] = data['Product_Combined'].str.replace(&#xA;    '[^\w\s]',' ').replace('\s+',' ')&#xA;data['Product_Combined'] = data['Product_Combined'].apply(&#xA;    lambda x: ' '.join([stem(word) for word in x.split()]))&#xA;&#xA;train_data, test_data, train_label,  test_label = train_test_split(&#xA;    data.Product_Combined, data.Breadcrumb, test_size=0.3, random_state=100)&#xA;&#xA;RF = RandomForestClassifier(n_estimators=100)&#xA;vectorizer = CountVectorizer( max_features = 50000, ngram_range = ( 1,3 ) )&#xA;data_features = vectorizer.fit_transform( train_data )&#xA;&#xA;RF.fit(data_features, train_label)&#xA;test_data_feature = vectorizer.transform(test_data)&#xA;Output_predict = RF.predict(test_data_feature)&#xA;print (&amp;quot;BreadCrumb_Accuracy: &amp;quot; + str(np.mean(Output_predict == test_label)))&#xA;&#xA;with codecs.open(&amp;quot;out_bread_crumb.txt&amp;quot;, &amp;quot;w&amp;quot;, &amp;quot;utf8&amp;quot;) as out:&#xA;    out.write(&amp;quot;Input\tPredicted\tActual\n&amp;quot;)&#xA;    for inp, pred, act in zip(test_data, Output_predict, test_label):&#xA;        try:&#xA;            out.write(&amp;quot;{}\t{}\t{}\n&amp;quot;.format(inp, pred, act))&#xA;        except:&#xA;            continue&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;strong&gt;Output:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Input   Predicted   Actual&#xA;Centuri Duster Dispos Compress Gas Duster 10 oz 2 Pk    Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches Electronics | Computers &amp;amp; Accessories | Computer Accessories &amp;amp; Peripherals | Cleaning &amp;amp; Repair | Compressed Air Dusters&#xA;BB Mall Phone Ring Stand Metal Stainless Steel Univers 360 Rotat Ring Kickstand iPhon 6 6s 6 s plus Samsung Note 5 Note 4 S5 iPad All SmartPhon Tablet Black    Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches Cell Phones &amp;amp; Accessories | Accessories | Mounts &amp;amp; Stands | Stands&#xA;Standard Motor Product 6444 Ignition Wire Set   Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches Automotive | Replacement Parts | Ignition Parts | Spark Plugs &amp;amp; Wires | Wires | Wire Sets&#xA;Walker 52271 Extension Pipe Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches Automotive | Replacement Parts | Exhaust &amp;amp; Emissions | Exhaust Pipes &amp;amp; Tips&#xA;ACDelco KS10640 Profession Time Compon Seal Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches Automotive | Replacement Parts | Bearings &amp;amp; Seals | Seals | Camshafts&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As you can see more than 50% of the actual test data was labeled as &lt;code&gt;Automotive | Exterior Accessories | Towing Products &amp;amp; Winches | Winches&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="15902" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-09T09:30:12.673" Title="Random Forest Multiclass Classification" Tags="|python|random-forest|multiclass-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27632" PostTypeId="1" CreationDate="2018-02-09T06:09:59.483" Score="1" ViewCount="98" Body="&lt;p&gt;Let success metric(for some business use case I am working on) be a continuous random variable S.&#xA;The mean of pdf defined on S indicates the chance of success. Higher the mean more is the chance of success. Let std dev of pdf defined on S indicates risk. Lower the std deviation lower the risk of failure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have data,let's call them X, which affects S. Let X also be modelled as bunch of random variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P(S|X) changes based on X.&#xA;The problem statement is I want to pick Xs such that P(S|X) has mean higher than P(S) and std deviation lower than P(S).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to illustrate my point I have taken X of 1 dimension.&#xA;Scatter plot between X(horizontal) and Y(on vertical):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YXpXM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YXpXM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that P(S|X) changes for different values of X as given in the below plot:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/SUKxY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SUKxY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For X between 4500 and 10225, mean of S is 3.889 and std dev is 0.041 compared to mean of 3.7 and std dev of 0.112 when there is no constraint on X.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am interested in is given S and bunch of Xs... pick range of Xs such that resulting distribution of P(S|X) has higher mean and lower standard deviation... Please help me find a standard technique that would help me achieve this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also I don't want to condition on X such that number of samples are too small to generalise.I want to avoid cases such as on the left most side of tree where number of samples is 1.&lt;/p&gt;&#xA;" OwnerUserId="27108" LastEditorUserId="27108" LastEditDate="2018-02-09T06:38:05.163" LastActivityDate="2021-03-26T17:07:00.733" Title="Convert a pdf into a conditional pdf such that mean increases and std dev falls" Tags="|machine-learning|probability|variance|discriminant-analysis|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27633" PostTypeId="2" ParentId="27625" CreationDate="2018-02-09T06:12:56.177" Score="3" Body="&lt;p&gt;There is a difference between using pre trained weights and transfer learning.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;When you use pre trained weights and re train all the layers in the entire model for the dataset, then you are just working towards converging faster in the direction of better results.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you were to only train the upper layers and lock all the other layers from training, then you are implementing transfer learning.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For your problem at hand, I would use approach 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be more than happy to elaborate in case you want me to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a link that explains how to do transfer learning with Keras : &lt;a href=&quot;https://towardsdatascience.com/transfer-learning-using-keras-d804b2e04ef8&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/transfer-learning-using-keras-d804b2e04ef8&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1029" LastEditorUserId="1029" LastEditDate="2018-02-09T07:53:52.250" LastActivityDate="2018-02-09T07:53:52.250" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27634" PostTypeId="1" CreationDate="2018-02-09T06:20:22.793" Score="4" ViewCount="11732" Body="&lt;p&gt;I was reading &lt;a href=&quot;https://arxiv.org/pdf/1608.04363.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; on environmental noise discrimination using Convolution Neural Networks and wanted to reproduce their results. They convert WAV files into log-scaled mel spectrograms. How do you do this? I am able to convert a WAV file to a mel spectrogram&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y, sr = librosa.load('audio/100263-2-0-117.wav',duration=3)&#xA;ps = librosa.feature.melspectrogram(y=y, sr=sr)&#xA;librosa.display.specshow(ps, y_axis='mel', x_axis='time')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DCAzQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DCAzQ.png&quot; alt=&quot;mel spectrogram&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also able to &lt;em&gt;display&lt;/em&gt; it as a log scaled spectrogram:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;librosa.display.specshow(ps, y_axis='log', x_axis='time')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bpHi8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bpHi8.png&quot; alt=&quot;log scaled image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, they look different, but the actual spectrogram &lt;code&gt;ps&lt;/code&gt; is the same. Using librosa, &lt;strong&gt;how can I convert this melspectrogram into a log scaled melspectrogram?&lt;/strong&gt; Furthermore, what is the use of a log scaled spectrogram over the original? Is it just to reduce the variance of the Frequency domain to make it comparable to the time axis, or something else? &lt;/p&gt;&#xA;" OwnerUserId="30435" LastActivityDate="2019-05-28T03:41:05.693" Title="How to convert a mel spectrogram to log-scaled mel spectrogram" Tags="|python|speech-to-text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27636" PostTypeId="2" ParentId="23706" CreationDate="2018-02-09T07:27:29.770" Score="1" Body="&lt;p&gt;I think you can use a prediction model to predict next value $y_{pred}$ after that you have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$e = |y_{pred} - y_t|$, use error to compare with &quot;$alpha \times std$&quot; of series. $alpha = [1, 10]$.&lt;/p&gt;&#xA;" OwnerUserId="45999" LastEditorUserId="29575" LastEditDate="2018-02-09T14:39:32.503" LastActivityDate="2018-02-09T14:39:32.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27637" PostTypeId="1" CreationDate="2018-02-09T07:36:32.437" Score="1" ViewCount="1485" Body="&lt;p&gt;I have a large dataset which can't be loaded in memory, hence I decided to use incremental learning using Xgboost. What I have done currently is: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tuned num_boosting_rounds using a chunk of data&lt;/li&gt;&#xA;&lt;li&gt;Set early stopping rounds to a value (60) &amp;lt;&amp;lt; num_boosting_rounds to prevent overfitting&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When my training code runs, it runs for a 1000 rounds on the first two chunks optimizing the loss function. It then stops at 60 rounds for each subsequent chunk as the best value for loss function was observed in the 1000th round in the 2nd Chunk. Is this the correct way to configure the Incremental model? Would this result in my model being sub-par owing to early stop for a majority of training chunks.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for idx,df in enumerate(df_pointer):&#xA;    num_round = 1000&#xA;    early_stopping_rounds = 60&#xA;    param = {'max_depth':5, 'eta':0.02, 'silent':1, 'objective':'binary:logistic', 'eval_metric':'logloss', 'max_delta_step':4, 'scale_pos_weight': 4}&#xA;    dtrain, deval = getDMatrixSplit(df)&#xA;    watchlist  = [(deval,'eval')]&#xA;    bst = xgb.train(param, dtrain, num_round, watchlist,&#xA;    early_stopping_rounds=early_stopping_rounds, xgb_model=xgb_model)&#xA;    xgb_model = self.model_path +'/xgb_%s_%s.model'%(ml_algo, idx)&#xA;    bst.save_model(xgb_model)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45458" LastEditorUserId="45458" LastEditDate="2018-02-20T06:13:11.253" LastActivityDate="2018-02-20T10:00:04.927" Title="Configuring Incremental XGBoost model" Tags="|machine-learning|xgboost|overfitting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27639" PostTypeId="1" AcceptedAnswerId="27640" CreationDate="2018-02-09T07:57:07.233" Score="6" ViewCount="8720" Body="&lt;p&gt;A training set has five classes including:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;quot;label-A&amp;quot;, &amp;quot;label-B&amp;quot;, &amp;quot;label-C&amp;quot;, &amp;quot;label-D&amp;quot;, &amp;quot;others&amp;quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But the problem is much simpler - it is to determine whether each input belongs to &lt;code&gt;&amp;quot;label-ABCD&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;others&amp;quot;&lt;/code&gt;. In this case, there are two solutions to solve this problem in my mind.&lt;/p&gt;&#xA;&lt;p&gt;Solution 1: Train a 5-classes classifier, when the classifier predicts the input as &lt;code&gt;&amp;quot;label-A&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-B&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-C&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-D&amp;quot;&lt;/code&gt;, we relabel it as &lt;code&gt;&amp;quot;label-ABCD&amp;quot;&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Solution 2: Train a 2-classes classifier, we relabel the data as &lt;code&gt;&amp;quot;label-ABCD&amp;quot;&lt;/code&gt; which is labeled as &lt;code&gt;&amp;quot;label-A&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-B&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-C&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;label-D&amp;quot;&lt;/code&gt;. And then it becomes a binary-classification problem.&lt;/p&gt;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Which way can the model get a better performance in &amp;quot;&lt;strong&gt;theorem&lt;/strong&gt;&amp;quot;?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In real case, these two cases get almost the same performance by a CNN classification model, and I am wondering if I adopt a weaker classifier like C4.5, Naive Bayes, SVM...which method will win?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Thanks!!&lt;/p&gt;&#xA;" OwnerUserId="41629" LastEditorUserId="29169" LastEditDate="2021-07-05T18:08:12.240" LastActivityDate="2021-07-05T18:08:12.240" Title="Multi-class classification v.s. Binary classification" Tags="|machine-learning|classification|svm|multiclass-classification|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27640" PostTypeId="2" ParentId="27639" CreationDate="2018-02-09T08:15:26.110" Score="8" Body="&lt;p&gt;The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. This means that is A and B are different in some way, but this difference is irrespective of the classification with &quot;others&quot; then there is no need to learn that distinction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: if you want to detect dog, cat, human with features such as weight, height and number of legs. The number of legs feature will have relatively low importance, because cats and dogs will likely all have 4 legs. However if I want to classify cat/dog vs humans, then the number of legs will be the most important feature. It might be the only feature you need.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One caveat may be severe class imbalance. By combining your classes in this way you may end up with an over representation of ABCD. You can use techniques such as anomaly detection to train a model on your ABCD data and then detect whether a novel instance falls within this distribution, or is an outlier in which case you would label it as &quot;others.&quot;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-09T08:15:26.110" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="27641" PostTypeId="2" ParentId="27631" CreationDate="2018-02-09T09:30:12.673" Score="2" Body="&lt;p&gt;There are bunch of things to take care of before you can solve the problem.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How is the label distribution in the training?&lt;/li&gt;&#xA;&lt;li&gt;If the distribution is not appropriate, then you need to sample the training data appropriately.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With regards to the approach:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Using random forest is appropriate.&lt;/li&gt;&#xA;&lt;li&gt;But as features to the random forest it would be better to use word vectors as input to the model. That would take into account products with same labels to have a very strong similarity score based on their names.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I have used this before almost for the exact problem and I saw a big boost in my results. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-02-09T09:30:12.673" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27642" PostTypeId="1" AcceptedAnswerId="27647" CreationDate="2018-02-09T09:46:08.143" Score="2" ViewCount="692" Body="&lt;p&gt;I want to assess the importance of variables in my model using the &lt;code&gt;Importance()&lt;/code&gt; function of R RandomForest package. I have a binary response variable / class and binary feature values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mytree.rf &amp;lt;- randomForest(class ~ ., data=mydata, ntree=1500,keep.forest=FALSE,importance=TRUE)&#xA;importance(mytree.rf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output matrix contains the MeanDecreaseAccuracy and MeanDecreaseGini. I understand those two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is with two other columns in the output. One simply says &quot;TRUE&quot; the other one &quot;FALSE&quot;. Neither in the documentation nor online I was able to find an answer what those values are and how they are calculated...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1: Thanks to Davids answer I realized TRUE and FALSE are my class &quot;names&quot;. I still don't understand how the value given in the matrix is calculated though... Can anyone help with that?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2: Thanks to David again, it turns out the answer is in the documentation. But it cannot be found in the chapter about the importance() function, but rather in the description of objects of class randomForest. Importance is one of these objects.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[...] a matrix with nclass + 2 (for classification) or two (for regression) columns.&#xA;For classification, the first nclass columns are the class-specific measures computed&#xA;as mean descrease in accuracy. The nclass + 1st column is the mean&#xA;descrease in accuracy over all classes. The last column is the mean decrease&#xA;in Gini index.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28937" LastEditorUserId="28937" LastEditDate="2018-02-13T15:21:13.977" LastActivityDate="2018-02-13T15:21:13.977" Title="Meaning of &quot;TRUE&quot; column in R RandomForest output for Importance()?" Tags="|r|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27643" PostTypeId="1" CreationDate="2018-02-09T09:52:10.903" Score="0" ViewCount="88" Body="&lt;p&gt;I want to work in Deep Belief Network using Python. Is there any package available in python similar to darch in R.&lt;/p&gt;&#xA;" OwnerUserId="46003" LastActivityDate="2018-02-09T10:27:50.520" Title="Is there any package in python similar to darch in R?" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27644" PostTypeId="2" ParentId="27643" CreationDate="2018-02-09T10:27:50.520" Score="2" Body="&lt;p&gt;A simple google with keyword    gives me:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A tutorial about implement DBN from scratch&#xA;&lt;a href=&quot;http://deeplearning.net/tutorial/DBN.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/tutorial/DBN.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;A python package about the DBN&#xA;&lt;a href=&quot;https://github.com/albertbup/deep-belief-network&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/albertbup/deep-belief-network&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Some interesting discussion about the DBN (and little history)&#xA;&lt;a href=&quot;https://github.com/keras-team/keras/issues/748&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/issues/748&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;SO discussion &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/23188131/deep-belief-network-with-scikit&quot;&gt;https://stackoverflow.com/questions/23188131/deep-belief-network-with-scikit&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-02-09T10:27:50.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27645" PostTypeId="1" CreationDate="2018-02-09T10:28:03.183" Score="2" ViewCount="1925" Body="&lt;p&gt;I've recently been looking at autoencoders and kernel PCA for unsupervised feature extraction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets consider just for a moment linear PCA. Its my understanding that if a autoencoder (with a single hidden layer) has linear activation functions then the sum-of-squares error function (chosen for convenience rather than statistical relevance) will have a unique solution for the weights. This can be easily seen, consider the sum-of-squares error function for the autoencoder input $\boldsymbol{x}$ and the autoencoder output $\hat{\boldsymbol{x}}$, that we wish to solve: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{aligned} &#xA;&amp;amp; \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \hat{\boldsymbol{x}}^{(n)}\|^2\\&#xA;\end{aligned} &#xA;$$&#xA;An autoencoder is a neural network whose input is its output. Defining the following: Let $\boldsymbol{W}$ be the weights matrix between the input layer and the hidden layer and $\boldsymbol{V}$ be the weights matrix between the hidden layer and the output layer. I'm ignoring the bias terms for now. Let $\psi$ be the activation function for each unit on the hidden layer and let $\phi$ be the activation function for each unit on the output layer. Then we have two mappings in an autoencoder: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\boldsymbol{z} = \psi\left(\boldsymbol{W}\boldsymbol{x}\right), \:\:\:\:\:\:\: \hat{\boldsymbol{x}} = \phi\left(\boldsymbol{V}\boldsymbol{z}\right) &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{aligned} &#xA;&amp;amp; \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \phi\left(\boldsymbol{V}\psi\left(\boldsymbol{W}\boldsymbol{x}^{(n)}\right)\right)\|^2\\&#xA;\end{aligned} &#xA;\label{eqn: 1}&#xA;$$&#xA;If $\psi$ and $\phi$ are linear, i.e  $\phi(\boldsymbol{s})=\boldsymbol{s}$ and $\psi(\boldsymbol{s})=\boldsymbol{s}$, then $\hat{\boldsymbol{x}} = \boldsymbol{V}\boldsymbol{W} \boldsymbol{x} = \boldsymbol{A} \boldsymbol{x}$ (i.e rotation) and&#xA;$$&#xA;\begin{aligned} &#xA;&amp;amp; \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{V}\boldsymbol{W}\boldsymbol{x}^{(n)}\|^2\\&#xA;&amp;amp; \min_{\boldsymbol{W},\boldsymbol{V}} {1\over 2N}\sum^N_{n=1} \|\boldsymbol{x}^{(n)} - \boldsymbol{A}\boldsymbol{x}^{(n)}\|^2\\&#xA;\end{aligned} &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is recognised as principle component analysis. This solution for the weights &quot;forms a basis set which spans the [linear] principle component space&quot; (Bishop, Pattern Recognition and Machine Learning). Bishop then goes on to say that these weight vectors aren't necessarily orthogonal or normalised. I assume that this is because we have not proposed any constraints on $\boldsymbol{A}$ as we would have done in PCA by way of proposing an orthogonal basis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So am I right in thinking that the autoencoder I've described above does not in general give the linear PCA solution, but in some very special cases they may match? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now bishop goes on to say that the only way to handle non-linear inputs is to add more hidden layers to the autoencoder and have some layers use non-linear activation functions. However one would assume that the same problem as above persists? i.e. that one is not guaranteed to get the same solution as PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There seem to be additional issues [Bishop]:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Training requires solving a non-linear optimisation problem. &lt;/li&gt;&#xA;&lt;li&gt;One may end up with a solution that corresponds to a local minima, not a global one.&lt;/li&gt;&#xA;&lt;li&gt;One must specify the number of subspace dimensions as part of the network architecture.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Therefore my question is, why use autoencoders at all for unsupervised feature extraction? Why not just use kernal PCA? Which will reduce to linear PCA when the data permits it. Is there some other advantage that I'm missing? Under what conditions should an autoencoder be chosen over kernel PCA?&lt;/p&gt;&#xA;" OwnerUserId="13259" LastEditorUserId="13259" LastEditDate="2018-02-09T10:35:50.343" LastActivityDate="2018-02-09T10:53:21.280" Title="Under what conditions should an autoencoder be chosen over kernel PCA?" Tags="|deep-learning|feature-extraction|pca|autoencoder|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27646" PostTypeId="2" ParentId="27645" CreationDate="2018-02-09T10:53:21.280" Score="2" Body="&lt;p&gt;I think you can interpret autoencoders as essentially performing kernel PCA, but with an extremely complex kernel. Like, really, stupidly complex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The kinds of things you can accomplish with autoencoders are spectacular. For example, consider &lt;a href=&quot;https://github.com/junyanz/BicycleGAN&quot; rel=&quot;nofollow noreferrer&quot;&gt;BicycleGAN&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If kernel PCA is a step above PCA, autoencoders are miles away. If you have an application where PCA makes sense but isn't working out, try kernel PCA. If you have an application where condensing your input into an embedding of some kind would be valuable but PCA is unlikely to be very helpful, use an autoencoder.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-09T10:53:21.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27647" PostTypeId="2" ParentId="27642" CreationDate="2018-02-09T11:06:30.327" Score="1" Body="&lt;p&gt;I don't see any columns like that. Here's a simple example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(randomForest)&#xA;data(iris)&#xA;iris.rf &amp;lt;- randomForest(Species ~ ., data=iris, importance=TRUE)&#xA;colnames(importance(iris.rf))&#xA;&#xA;# [1] &quot;setosa&quot;               &quot;versicolor&quot;          &#xA;# [3] &quot;virginica&quot;            &quot;MeanDecreaseAccuracy&quot;&#xA;# [5] &quot;MeanDecreaseGini&quot; &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you're seeing columns named &quot;TRUE&quot; and &quot;FALSE&quot; in the matrix returned by &lt;code&gt;importance&lt;/code&gt;, my best guess is that those are the names of columns in your training data.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-09T11:06:30.327" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27648" PostTypeId="1" CreationDate="2018-02-09T11:18:47.883" Score="5" ViewCount="12223" Body="&lt;p&gt;I am new to machine learning and I got this task in my university. I have a dataset with over 100 columns and two target variables: $target1$ is categorical i.e. $0$ or $1$ and $target2$ is continuous i.e. values in range $0 \space to \space 100$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I predict this type of problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using multi-output classification from sklearn using the Random forest as an ensembler and it is predicting nicely for continuous target variable but not for categorical target variable.&lt;/p&gt;&#xA;" OwnerUserId="46007" LastEditorUserId="26562" LastEditDate="2018-02-11T22:47:57.560" LastActivityDate="2019-02-03T13:18:04.530" Title="Multi target classification for different types of target variables" Tags="|python|multilabel-classification|multitask-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27650" PostTypeId="2" ParentId="27632" CreationDate="2018-02-09T11:47:52.290" Score="1" Body="&lt;p&gt;Just apply an optimization to search for the X values that satisfy the criteria you're looking for. Here's a simple demo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;mu_x_true = 1e4&#xA;mu_y_true = 3.75&#xA;n = 1e2&#xA;&#xA;x &amp;lt;- rpois(n, mu_x_true)&#xA;y &amp;lt;- rnorm(n, sqrt(mu_y_true))^2&#xA;&#xA;plot(x, y)&#xA;&#xA;# conditions:&#xA;# E[Y|X] &amp;gt; E[Y]&#xA;# std(Y|x) &amp;lt; std(Y)&#xA;&#xA;mu_y_emp = mean(y)&#xA;sd_y_emp = sd(y)&#xA;&#xA;objective &amp;lt;- function(par, alpha=0.5){ &#xA;    if (par[1]&amp;gt;par[2]) par = rev(par)&#xA;    ix &amp;lt;- which((par[1] &amp;lt; x) &amp;amp; (x &amp;lt; par[2]))&#xA;    k &amp;lt;- length(ix)&#xA;    if (k==0) return(1e12)&#xA;    mu_yx &amp;lt;- mean(y[ix])&#xA;    sd_yx &amp;lt;- sd(y[ix])&#xA;&#xA;    alpha*(mu_y_emp - mu_yx) + (1-alpha)*(sd_yx - sd_y_emp)&#xA;}&#xA;&#xA;init &amp;lt;- mean(x) + c(-sd(x), sd(x))&#xA;test &amp;lt;- optim(objective, par=init)&#xA;&#xA;ix &amp;lt;- which((par[1] &amp;lt; x) &amp;amp; (x &amp;lt; par[2]))&#xA;&#xA;mean(y[ix]) &amp;gt; mean(y) &#xA;# TRUE&#xA;&#xA;sd(y) &amp;gt; sd(y[ix])&#xA;# TRUE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-09T11:47:52.290" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27651" PostTypeId="2" ParentId="27648" CreationDate="2018-02-09T11:57:05.067" Score="2" Body="&lt;p&gt;You should break this down into two models. I would solve this in the following manner:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The first model would predict if its either Target 1 or Target 2&#xA;by looking at 100 columns&lt;/li&gt;&#xA;&lt;li&gt;The second model then would look at the 100 columns and additionally the output of model 1 and then predict 0 or 1 in case of target 1 or 0-100 in case of target 2. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I do not think you can achieve the result with just one single model. &#xA;If you need more information, I could elaborate on it. But this should give you a starting point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an explanation of the 2 above points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Train a classifier, with all the data points you have with labels as Target 1/ Target 2. For this you could use any family of classifier. But you need to be very careful in the evaluation. If this models performs poorly, you will have a problem, as your classification would affect your next model. You also need to check if the distribution between target 1 and target 2 are appropriate before using a model to classify them.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once the classifier is done, you can then use regression with all the input features + class of the entry ( target 1 or 2 ). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1029" LastEditorUserId="1029" LastEditDate="2018-02-12T15:31:18.563" LastActivityDate="2018-02-12T15:31:18.563" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27652" PostTypeId="2" ParentId="27648" CreationDate="2018-02-09T12:15:05.620" Score="0" Body="&lt;p&gt;This depends on the meaning of your 2 response variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your continues variable is actually 100 (integer) class as oppose to meaningful increasing magnitude (hence more of a regression), hence, what you can have a try is combining your 2 categorical response into one by concatenating.  This new variable will be your new response with 200 class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, Nischal's answer would be the way to go.&lt;/p&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-02-09T12:15:05.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27653" PostTypeId="1" AcceptedAnswerId="27675" CreationDate="2018-02-09T12:59:34.347" Score="2" ViewCount="391" Body="&lt;p&gt;I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ on my local machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering what the best way would be to go about development of an application to run on the data that is in the cluster. Things on my mind for consideration are the following;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;IntelliJ is a nice IDE but I can't store the volume of data I want to look at on my local machine&lt;/li&gt;&#xA;&lt;li&gt;I can run spark-shell on the edge node but its not great for data exploration and development&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How is it best to bridge this gap between data and a nice development environment?&lt;/p&gt;&#xA;" OwnerUserId="37059" LastActivityDate="2018-02-10T12:58:51.677" Title="Best practice for developing using Spark" Tags="|apache-spark|apache-hadoop|scala|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-12T19:34:53.080" ContentLicense="CC BY-SA 3.0" />
  <row Id="27654" PostTypeId="1" CreationDate="2018-02-09T13:32:34.083" Score="1" ViewCount="24" Body="&lt;p&gt;Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks? &lt;/p&gt;&#xA;" OwnerUserId="9388" LastActivityDate="2018-02-11T21:02:49.703" Title="Any research on segmentation of non-text contents out of (mostly) text-documents?" Tags="|machine-learning|nlp|text-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27655" PostTypeId="1" CreationDate="2018-02-09T15:40:38.157" Score="1" ViewCount="51" Body="&lt;p&gt;I work for a highly-regulated entity, so I have to obfuscate what I'm working on; I'll provide the following as examples on what I'm doing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training an XGB model for NLP comments about breeds of dogs (Tensorflow, sklearn). I have a dataset about dogs that I'm using to train the algo and cross-validation testing is promising. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my test data is not just about dogs, it's about animals in general. So that data can be about dogs, cats, elephants, whatever. Right now, the XGB model is giving all of the animals it does not recognize a default value (a category). How can I make my XGB just say &quot;I don't know&quot; instead? Is there a way to assign a default value to test values that don't make sense in any category?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I realize that another possibility is to create a model to classify the test data but I want to see if there is a way to solve this within the XGB model. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-09T18:45:01.110" Title="Help with training XGB" Tags="|tensorflow|xgboost|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27656" PostTypeId="1" AcceptedAnswerId="27659" CreationDate="2018-02-09T16:20:03.117" Score="3" ViewCount="532" Body="&lt;p&gt;There must be examples of this, though I haven't been able to find any. Maybe I don't know what to search for.&lt;/p&gt;&#xA;" OwnerUserId="46026" LastActivityDate="2018-02-09T19:50:41.787" Title="Are there any examples of neural networks that take two samples as input with a label of {same class, different class}?" Tags="|neural-network|classification|deep-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27657" PostTypeId="2" ParentId="26351" CreationDate="2018-02-09T17:50:07.383" Score="0" Body="&lt;p&gt;With scaling (or Z-transformation), you need a mean and a variance, which should come from total data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's more, if your model is going to be used on future coming data, then this mean and variance need to be applied to new data as well, as oppose to based on new data's mean or variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One important assumption in modelling is that feature / pattern in training would be the same or similar in testing set. This is the base that we can use history data to predict future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence this requires consistency in mapping values to another using one transformation function.  Different sets of mean and variance, would lead to different transformation. &lt;/p&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-02-09T17:50:07.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27658" PostTypeId="2" ParentId="27655" CreationDate="2018-02-09T18:45:01.110" Score="1" Body="&lt;p&gt;You could e.g. use as objective &quot;multi:softprob&quot; instead of &quot;multi:softmax&quot;. The model is identical, but you get a probability for each label instead if the most probable label.  Then you can specify a threshold, e.g. 0.5 and return &quot;unknown&quot; if no probability is above the limit.&lt;/p&gt;&#xA;" OwnerUserId="36977" LastActivityDate="2018-02-09T18:45:01.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27659" PostTypeId="2" ParentId="27656" CreationDate="2018-02-09T19:09:10.267" Score="2" Body="&lt;p&gt;Yes. This is how a face recognition algorithm might work for example, where two pictures might be of the same person or different person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To build such a system, just pair up your data in a training set, double the input vector space and run a binary classifier that outputs &quot;true&quot; if the two items are the same. Any ML classifier could be adapted to this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, you may want more control over classification, and be robust against new classes that your algorithm has not seen before. A big problem in face recognition is the large number of potential classes, including classes not seen in training data, coupled with an equally large variance of images in the wild. This means that a naive approach as above will have poor performance in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a more sophisticated approach: &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/HuUtN/triplet-loss&quot; rel=&quot;nofollow noreferrer&quot;&gt;Triplet Loss&lt;/a&gt;. This requires that you train with three inputs for each example. Unlike the naive version:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The input is one image at a time. You train in triplets - an &quot;anchor&quot;, and a postive match, and a negative match, in order to calculate one loss value for backprop.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The output is not a same/different class, but a vector description of the object. Similarity of objects is the distance between vectors. You do not need a label for this vector, even though this is supervised learning, thanks to how the loss function works.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The loss function is based on difference of distance from anchor example to positive example compared with a desired higher distance to negative example. This encourages learning key features of the inputs by making distance between vectors low (ideally zero) when they represent the same class, and as high as possible when they represent different classes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In both the naive approach and triplet loss approach, you need to be careful about selecting training data. You want to make the learning algorithm work hard to learn key differences, otherwise it is too easy to get a good loss. So taking MNIST as an example, when looking at negative matches, you want to pair up more (1, 7), (3, 2), (3, 8), (4, 9) etc pairs than (0,1), (9, 5). There are ways to drive this selection based on feedback from previous training epoch. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-02-09T19:50:41.787" LastActivityDate="2018-02-09T19:50:41.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27662" PostTypeId="2" ParentId="27648" CreationDate="2018-02-09T20:18:51.280" Score="0" Body="&lt;p&gt;There may be benefit, i.e. improved performance, in solving for both problems together instead of as separate problems via separate models if the two targets have some sort of relationship. Look up multitask learning and structured prediction. The output layer in neural networks can be of any dimension, including 2 which is your case.&lt;/p&gt;&#xA;" OwnerUserId="26297" LastActivityDate="2018-02-09T20:18:51.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27663" PostTypeId="1" CreationDate="2018-02-10T06:00:41.010" Score="1" ViewCount="115" Body="&lt;p&gt;I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to index ontology concepts. However, as it is very difficult to find the ground truth about the topics in the documents, it is becoming very difficult to evaluate the model. As manually tagging the correct labels will take a huge amount of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any standard ways by which people evaluate models of topic extraction/ entity extraction, when ground truth about the topics is not clear?&lt;/p&gt;&#xA;" OwnerUserId="46054" LastActivityDate="2018-02-10T06:00:41.010" Title="Performance Metric for topic extraction when there is no ground truth" Tags="|nlp|text-mining|model-evaluations|performance|topic-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27664" PostTypeId="1" CreationDate="2018-02-10T06:32:46.710" Score="4" ViewCount="1810" Body="&lt;p&gt;I am seeking guidance on a machine learning problem involving the tagging of data columns. Currently, I have a system where users can add multiple tags to a columns in a table. However, I want to automate the tagging of new columns by using Multi Label Classification. I have extracted 21 features from each column by doing a column analysis on the column values. The features obtained would include statistical values such standard deviation, max,min, kurtosis and etc. Am I on the right path in using these features as inputs for a Multi Label Classification model ? Right now I am focusing on numeric values in columns&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8TN5B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8TN5B.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, the above table on the left represent some arbitrary table which consist of 3 columns. As a user I would tag the column with the appropriate tags. So the Rainfall column would have rainfall and precipitation and Temperature column would have temperature. The table on the right just represents the tags being assigned to a column in a table format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/J0gAw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J0gAw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example sample data set in the above image&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order for me to do multi-label classification to automate the tagging of columns automatically, when tables with similar columns are ingested into the system, I would need to extract some features or properties that describes the already tagged columns to use as input for the multi label model. So I did some column analysis and placed just several example features in the table above. This includes standard deviation, maximum,minimum, median and kurtosis. I have about 21 features in total. The output labels are also represented for each column in the above image where 1 signifies the label is present and 0 is not present.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Z6apu.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Z6apu.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end the model will decide which tags are assigned to a newly discovered column based on its features.&lt;/p&gt;&#xA;" OwnerUserId="46055" LastEditorUserId="46055" LastEditDate="2018-02-12T19:08:49.690" LastActivityDate="2022-01-17T16:00:50.093" Title="Multi Label Classification on Data Columns in Tables" Tags="|machine-learning|classification|algorithms|supervised-learning|multilabel-classification|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="27665" PostTypeId="1" AcceptedAnswerId="27666" CreationDate="2018-02-10T08:15:08.857" Score="5" ViewCount="13022" Body="&lt;p&gt;Can anyone explain what is &lt;code&gt;Saturating Gradient&lt;/code&gt; problem? It would be nice if anyone can provide math details as well. Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="45907" LastEditorUserId="28175" LastEditDate="2018-02-10T08:42:45.643" LastActivityDate="2021-05-18T19:28:10.197" Title="What Is Saturating Gradient Problem" Tags="|machine-learning|neural-network|deep-learning|gradient-descent|backpropagation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27666" PostTypeId="2" ParentId="27665" CreationDate="2018-02-10T08:32:03.427" Score="7" Body="&lt;p&gt;If you use sigmoid-like activation functions, like sigmoid and tanh, after some epochs of training, the linear part of each neuron will have values that are very big or very small. This means that the linear part will have a big output value regardless of its sign. Consequently, the input of sigmoid-like functions in each neuron which adds non-linearity will be far from the center of these functions.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vvy9I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vvy9I.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In those locations, the gradient/derivative value is very small. Consequently, after numerous iterations, the weights get updated so slowly because the value of the gradient is very small. This is why we use the &lt;code&gt;ReLU activation function&lt;/code&gt; for which its gradient doesn't have this problem. Saturating means that after some epochs that learning happens relatively fast, the value of the linear part will be far from the center of the sigmoid and it somehow saturates, and it takes too much time to update the weights because the value of gradient is small. You can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/23493/28175&quot;&gt;here&lt;/a&gt; as a solution for this problem.&lt;/p&gt;&#xA;&lt;p&gt;If I want to explain the math part, suppose that you are using sigmoid as the activation function. If &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; represents sigmoid, its gradient is &lt;span class=&quot;math-container&quot;&gt;$\sigma (1-\sigma$&lt;/span&gt;). Now suppose that your linear part, the input of sigmoid is a positive number which is too large, then sigmoid which is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$ \frac{1}{1+e^{-x}} $$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;will have a value near to one but smaller than that. On the other hand &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; will be so close to zero, multiplying &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$1 - \sigma$&lt;/span&gt; will result in a small value, which means the value of the gradient is very small. If the value of the linear part is very small, then &lt;span class=&quot;math-container&quot;&gt;$\sigma$&lt;/span&gt; will be close to zero and &lt;span class=&quot;math-container&quot;&gt;$1- \sigma$&lt;/span&gt; will be close to 1 but smaller than that. Again, multiplying these will lead to a small value as the gradient.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="110693" LastEditDate="2021-02-21T13:25:25.820" LastActivityDate="2021-02-21T13:25:25.820" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27667" PostTypeId="2" ParentId="27626" CreationDate="2018-02-10T08:59:53.383" Score="3" Body="&lt;p&gt;I've just found out about the class torch.utils.data.ConcatDataset([datasets]) . Its easy to use and documentation is here : &lt;a href=&quot;http://pytorch.org/docs/master/data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pytorch.org/docs/master/data.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45907" LastActivityDate="2018-02-10T08:59:53.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27670" PostTypeId="1" CreationDate="2018-02-10T10:26:46.250" Score="1" ViewCount="8157" Body="&lt;p&gt;I use numpy arrays to work with deep learning images. But as the data gets bigger, I'm facing issue with RAM even before training the model when using techniques like data augmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone suggest me how to work with large data for eg. 30GB of data in my system which has 16gb ram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I'm worried about RAM during preprocessing and training, while i do batch processing with my GPU&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-02-10T20:39:09.467" Title="How to work with large amount of data overcoming RAM issues in python" Tags="|python|deep-learning|keras|numpy|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27671" PostTypeId="1" CreationDate="2018-02-10T11:18:25.340" Score="25" ViewCount="34111" Body="&lt;p&gt;Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it possible to apply it on text classification problem? Which part of the data do you need to oversample? There is already &lt;a href=&quot;https://datascience.stackexchange.com/questions/19553/can-smote-be-applied-over-sequence-of-words-sentences&quot;&gt;another question&lt;/a&gt; about it, but it doesn't have an answer. Where can I possibly learn to get started with this? &lt;/p&gt;&#xA;" OwnerUserId="12496" LastEditorUserId="29575" LastEditDate="2018-02-10T13:02:16.467" LastActivityDate="2021-07-04T03:28:11.967" Title="How do you apply SMOTE on text classification?" Tags="|class-imbalance|text|smote|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27672" PostTypeId="1" CreationDate="2018-02-10T12:12:10.447" Score="3" ViewCount="1347" Body="&lt;p&gt;I have an AlexNet architecture that predicts in a supervised-way, whether a given image belongs into classes 1-10. But it should also recognize if it doesn't fit into any of these classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thought was that I just add an 11th class and label all my negative-data with this class and train the softmax-cross-entropy loss with Gradient Descent Optimizer (Adam) as usual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my question is if this is a valid approach on solving my problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have around 7.000 images per class and could obtain up to 70.000 negative examples. I have the strong feeling that training with 70.000 negative-examples my network would always predict the outcome to belong to class 11 (not matching any other). Am I correct with the assumption? If yes: How many negative examples would do the job?&lt;/p&gt;&#xA;" OwnerUserId="46065" LastEditorUserId="29575" LastEditDate="2018-02-10T12:56:20.833" LastActivityDate="2018-10-10T08:00:31.230" Title="Loss function and data distribution for positive / negative examples" Tags="|neural-network|dataset|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27674" PostTypeId="2" ParentId="27648" CreationDate="2018-02-10T12:36:09.343" Score="4" Body="&lt;p&gt;You have one classification task and one regression task, but sklearn's multioutput meta-estimators only support two tasks of the same type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best solution here is to train two models: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A binary classifier to predict $target1$ &lt;/li&gt;&#xA;&lt;li&gt;A regressor to predict $target2$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import train_test_split&#xA;from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor&#xA;&#xA;# X, y = load training data&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)&#xA;&#xA;clf = RandomForestClassifier()&#xA;clf.fit(X_train, y_train[:,0])&#xA;print(&quot;classifier accuracy:&quot;, clf.score(X_test, y_test[:,0]))&#xA;&#xA;regr = RandomForestRegressor()&#xA;regr.fit(X_train, y_train[:,1)&#xA;print(&quot;regressor R^2:&quot;, regr.score(X_test, y_test[:,1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="40853" LastEditDate="2018-02-12T04:29:54.023" LastActivityDate="2018-02-12T04:29:54.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27675" PostTypeId="2" ParentId="27653" CreationDate="2018-02-10T12:58:51.677" Score="1" Body="&lt;p&gt;What you want to do is develop your code in IntelliJ, and then package your code and dependencies into an executable jar file using SBT or Maven. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have your jar stored locally, you can use &lt;a href=&quot;https://spark.apache.org/docs/latest/submitting-applications.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;spark-submit&lt;/a&gt; to transfer the jar to your cluster (along with some other parameters) for execution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might also want to take a sample of your data and store it locally so you can run spark locally and test/debug your code in IntelliJ. This can speed up development considerably, and having access to a debugger is a huge help.&lt;/p&gt;&#xA;" OwnerUserId="33603" LastActivityDate="2018-02-10T12:58:51.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27676" PostTypeId="1" CreationDate="2018-02-10T13:21:43.947" Score="9" ViewCount="7538" Body="&lt;p&gt;I have been building some models for a project, but I can't wrap my head around the math of Adagrad and Adadelta algorithms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do understand how vanilla gradient descent works and I have written code for making it work successfully.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will be grateful if anyone explain these two things to me or provide some resource to understand them.&lt;/p&gt;&#xA;" OwnerUserId="46068" LastEditorUserId="56981" LastEditDate="2018-09-16T07:51:02.760" LastActivityDate="2018-09-16T07:51:02.760" Title="Understanding the mathematics of AdaGrad and AdaDelta" Tags="|machine-learning|gradient-descent|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27678" PostTypeId="2" ParentId="17202" CreationDate="2018-02-10T14:30:47.690" Score="1" Body="&lt;p&gt;Lime introduced support for regression problems in their latest version. A notebook with examples is available &lt;a href=&quot;https://marcotcr.github.io/lime/tutorials/Using%2Blime%2Bfor%2Bregression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29415" LastActivityDate="2018-02-10T14:30:47.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27679" PostTypeId="2" ParentId="25576" CreationDate="2018-02-10T15:05:21.927" Score="5" Body="&lt;p&gt;Option (b) is a good approach. If you use a classifier that outputs probabilities, you can even multiply the two model outputs together to calculate a risk expectation that's interpretable as the joint likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If cpc is discrete valued and non negative, you could interpret that as a count and use a zero-inflated Poisson model. If it's not, you could convert it by rounding to the nearest cent or something like that. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-10T15:05:21.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27680" PostTypeId="2" ParentId="27672" CreationDate="2018-02-10T15:05:43.517" Score="1" Body="&lt;p&gt;Your approach of adding negative examples to your data set, replacing the 10-output softmax final layer with an 11-output softmax, and retraining is certainly valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need these additional examples because the network can only learn to classify categories that it has seen before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Retraining the final layer only is a good first attempt because it is the cheapest, but if you don't get good results you can try fine tuning one or more additional layers at the end of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these techniques fall under the topic of &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Transfer Learning&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the second part of your question about how many negative examples to include - this is known as &lt;a href=&quot;https://towardsdatascience.com/dealing-with-imbalanced-classes-in-machine-learning-d43d6fa19d2&quot; rel=&quot;nofollow noreferrer&quot;&gt;Class Imbalance Problem&lt;/a&gt; in Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether your model will always predict the negative class if you include many negative examples depends on the data and the model, but it is certainly a possibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The safest approach would be to undersample your negative examples and include only 7000 to match the frequency of other classes in the training set. This should be a good first try, but if it doesn't work you should consider a couple of questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What is the expected distribution of my target population?&lt;/li&gt;&#xA;&lt;li&gt;What is the cost of misclassifying each type of example?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;And then make an informed decision about how to proceed, which may involve either sampling negative examples differently, or adjusting your performance metric to something like &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve&quot; rel=&quot;nofollow noreferrer&quot;&gt;AUC&lt;/a&gt; (which can guide &lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;early-stopping&lt;/a&gt;, for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additional material:&#xA;&lt;a href=&quot;https://arxiv.org/abs/1710.05381&quot; rel=&quot;nofollow noreferrer&quot;&gt;A systematic study of the class imbalance problem in convolutional neural networks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-10T15:05:43.517" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27681" PostTypeId="2" ParentId="27670" CreationDate="2018-02-10T16:26:25.143" Score="1" Body="&lt;p&gt;During image preprocessing in Keras, you may run out of memory when doing &lt;code&gt;zca_whitening&lt;/code&gt;, which involves taking the dot product of an image with itself. This depends on the size of individual images in your dataset, not on the total size of your dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The memory required for &lt;code&gt;zca_whitening&lt;/code&gt; will exceed 16GB for all but very small images, see &lt;a href=&quot;https://github.com/keras-team/keras/issues/1531&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for an explanation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To solve this you can set &lt;code&gt;zca_whitening=False&lt;/code&gt; in &lt;a href=&quot;https://keras.io/preprocessing/image/#imagedatagenerator&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageDataGenerator&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-10T16:26:25.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27682" PostTypeId="2" ParentId="27670" CreationDate="2018-02-10T16:54:53.180" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://dask.pydata.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dask&lt;/a&gt; is designed to manage these types of workloads. It provides an interface like NumPy, Pandas, or Python iterators for larger-than-memory operations. An example of using Dask with TensorFlow can be found &lt;a href=&quot;https://github.com/tflearn/tflearn/blob/master/examples/basics/use_dask.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-02-10T16:54:53.180" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27683" PostTypeId="2" ParentId="27676" CreationDate="2018-02-10T18:16:53.613" Score="2" Body="&lt;p&gt;From &lt;a href=&quot;https://www.quora.com/What-is-an-intuitive-explanation-of-the-AdaDelta-Deep-Learning-optimizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;quora&lt;/a&gt; you'll find a more complete guide, but main ideas are that AdaGrad tries to taggle these problems in gradient learning rate selection in machine learning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Manual selection of the learning rate η.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 The gradient vector gt is scaled uniformly by a scalar learning rate η.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 The learning rate η remains constant throughout the learning process.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It resolves concerns 2 and 3 simply by dividing each current gradient component by an L2 norm of past observed gradients for that particular component.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It has in itself the following issues:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 Continually decaying learning rate η.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 Manual selection of the learning rate η.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;AdaDelta resolves AdaGrad concern 1 by summing the gradients only within a certain window W.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Concern 2 solution relates to mismatch in gradient units and thus&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the actual accumulation process is implemented using a concept from momentum.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The last calculation needs understanding on momentum theory and it was shortly explained there in article.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;My idea was to give the main causes behind what was intended, maybe that makes reading easier.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44114" LastEditDate="2018-02-11T08:07:38.633" LastActivityDate="2018-02-11T08:07:38.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27685" PostTypeId="2" ParentId="27670" CreationDate="2018-02-10T20:39:09.467" Score="2" Body="&lt;p&gt;Do all image preparation and data augmentation during preprocessing and save the result as arrays of one or more samples (up to mini-batch size). Do not convert the arrays back to images. Read these prepared arrays with the wrapper that trains your model. I recommend numpy.save for its simplicity and transparency. Other options are discussed here: &lt;a href=&quot;https://stackoverflow.com/questions/9619199/best-way-to-preserve-numpy-arrays-on-disk&quot;&gt;Stackoverflow - persisting numpy arrays&lt;/a&gt;  .&lt;/p&gt;&#xA;" OwnerUserId="46082" LastActivityDate="2018-02-10T20:39:09.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27686" PostTypeId="1" CreationDate="2018-02-10T22:47:48.910" Score="1" ViewCount="207" Body="&lt;p&gt;I think they share a lot (e.g. machine learning is a subset of both, right?), but maybe both have elements the other doesn't have?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you name some in that case? Or is one a subset of the other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is the relationship between AI and data science?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, when it comes to the relationship of AI and ML, I always say AI is a superset of ML. And the distinguishing set is search algorithms, which I would include in AI but not in ML. Would search algorithms be included in data science?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2019-02-13T07:07:10.260" LastActivityDate="2019-04-24T05:53:37.007" Title="What is the relationship between AI and data science?" Tags="|reference-request|definitions|ai|" AnswerCount="2" CommentCount="3" CommunityOwnedDate="2019-02-15T14:30:33.120" ContentLicense="CC BY-SA 3.0" />
  <row Id="27687" PostTypeId="1" CreationDate="2018-02-10T22:58:37.920" Score="7" ViewCount="1356" Body="&lt;p&gt;I need to train a word2vec embedding model on Wikipedia articles using Gensim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eventually, I will use the entire Wikipedia for that&lt;br /&gt;&#xA;but for the moment, I'm doing some experimentation/optimization to improve the model quality and I was wondering how many articles would be enough to train a meaningful/good model? How many examples needed for each unique word in vocabulary?&lt;/p&gt;&#xA;" OwnerUserId="6433" LastEditorUserId="6433" LastEditDate="2021-02-04T14:42:46.213" LastActivityDate="2021-02-04T14:42:46.213" Title="How much text is enough to train a good embedding model?" Tags="|nlp|training|word-embeddings|performance|embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27688" PostTypeId="1" CreationDate="2018-02-10T23:20:24.793" Score="1" ViewCount="808" Body="&lt;p&gt;I am building a deep neural network based binary classifier, with single output. The loss function I actually want to minimize is&#xA;$$ \mathcal L(\hat y,y) = &#xA;\begin{cases}&#xA;0,  &amp;amp; \text{if $\hat y$ = 0} \\&#xA;1,  &amp;amp; \text{if $\hat y$ = 1 &amp;amp; $y$ = 0} \\&#xA;\gamma \approx -0.7 ,  &amp;amp; \text{if $\hat y$ = 1 &amp;amp; $y$ = 1}&#xA;\end{cases}&#xA;$$&#xA;where $y \in \{0;1\}$ is sample's label, $\hat y \in \{0;1\}$ - classifier's output and $\gamma$ - a hyperparameter. I think this is a case of assymmetric loss. (One can see it as betting: no reward when not betting, stake 1\$, payout 1.7\$ if bet wins)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I know by now, this loss function is probably not suited well for backpropagation and gradient descent. &lt;strong&gt;Question&lt;/strong&gt;: Is there a better-suited formulation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The often used cross-entropy loss doesn't allow for trade-off tuning between precision and recall. LINEX and LINLIN are assymetric by design purpose, but I coundn't find an example of a deep NN trained with them. An alternative could be leaving the loss function as it is and resorting to &lt;a href=&quot;http://www.jhuapl.edu/SPSA/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SPSA&lt;/a&gt;, but I'd like to keep it simple if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: I came up with &#xA;$$ \mathcal L(\hat y,y) = - (\gamma \hat y)^{y}(-\hat y)^{1-y}$$&#xA;At the moment, I have no clue if it is going to &lt;a href=&quot;https://stackoverflow.com/questions/40752801/non-smooth-and-non-differentiable-customized-loss-function-tensorflow#answer-40758135&quot;&gt;work&lt;/a&gt; for NN learning. I'm concerned (maybe unnecessarily) that, without logarithm, it is not &lt;a href=&quot;https://math.stackexchange.com/questions/1582452/logistic-regression-prove-that-the-cost-function-is-convex#answer-1582667&quot;&gt;convex&lt;/a&gt; regarding NN weights. (Last layer has sigmoid activation.) Here is the log loss shown for comparison.&#xA;$$ \mathcal L(\hat y,y) = - (y\log \hat y + (1-y)\log(1-\hat y)) = -\log[\hat y^y (1-\hat y)^{1-y}]$$&lt;/p&gt;&#xA;" OwnerUserId="44654" LastEditorUserId="44654" LastEditDate="2018-02-11T15:18:45.413" LastActivityDate="2018-02-11T19:28:32.910" Title="asymmetric cost function for deep neural network binary classifier" Tags="|classification|deep-learning|supervised-learning|loss-function|cost-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27689" PostTypeId="2" ParentId="27688" CreationDate="2018-02-10T23:57:26.703" Score="1" Body="&lt;p&gt;Rather than 0 use 0.1 or some other small non-zero value. Use the true y as an indicator function, multiply your class probability by your reward component, and sum terms for all possibilities to get a usable loss, like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Loss(y, y_p) = (1-y)(1)(y_p) + (y)(-0.7)(y_p) + (0.1)(1-y_p)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $y_p =P(\hat{y}=1|X)$&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-02-11T19:28:32.910" LastActivityDate="2018-02-11T19:28:32.910" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27690" PostTypeId="1" CreationDate="2018-02-11T01:38:10.520" Score="3" ViewCount="452" Body="&lt;p&gt;What are the pros/cons of using external GPUs (e.g., connected through thunderbolt) vs. internal GPUs for machine learning? &lt;/p&gt;&#xA;" OwnerUserId="843" LastActivityDate="2022-04-06T16:20:54.430" Title="External GPU vs. internal GPU for machine learning" Tags="|performance|gpu|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27692" PostTypeId="2" ParentId="27686" CreationDate="2018-02-11T03:17:24.230" Score="5" Body="&lt;p&gt;Though neither are well defined, as commonly used they are somewhat orthogonal concepts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, AI has a fairly narrow definition - it is about optimization through actions.  AI is about decision making, either in deterministic or probabilistic environments.  Typically, this is operationalized as action selection to maximize some reward function, or equivalently to minimize some loss function.  Supervised or unsupervised learning (i.e. machine learning) about your environment can be helpful in using your experience to aid in selecting optimal actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As it's commonly used, data science has no rigorous definition - businesses use the term to refer to anything from creating charts Excel to deep reinforcement learning models that can win Go.  From the point of view of a practitioner, these have absolutely nothing in common. From the point of view of a business owner, the common thread is extracting meaning, and therefore value, from raw data.  A data scientist is a 'meaning extraction layer'.  How this operation is performed, and the techniques used (again in my experience), make no difference to the employer of a data scientist. The job title may as well be 'data magician'.  But the point is that data + data science = business value, whether that comes in the form of insights into customer trends, causal analysis of marketing campaigns, or an AI bot that is 'rewarded' when it recommends you a movie that you like.  I suppose that means that AI is a subset of data science - but you could also say the same thing about clear communication, so it's a bit of a non-statement.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-02-11T03:17:24.230" CommentCount="0" CommunityOwnedDate="2019-02-15T14:30:33.120" ContentLicense="CC BY-SA 3.0" />
  <row Id="27693" PostTypeId="1" CreationDate="2018-02-11T03:33:11.807" Score="2" ViewCount="433" Body="&lt;p&gt;I'm implementing prediction code for courses of computing fields using Naive Bayes classifier. The output is to predict whether the course is (management, design, database, analysis,…9 classes). I have only about 100 records of courses descriptions. I label 50 by search keyword and I will use them as 80% of them as train data and the 20% as test data where the other 50 courses to predict by the classifier. The classes are not balanced I can’t to change anything in data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The accuracy is 75% how can I increase the accuracy? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it small for training dataset?&lt;/p&gt;&#xA;" OwnerUserId="46089" LastEditorUserId="29575" LastEditDate="2018-02-11T16:41:31.137" LastActivityDate="2018-02-11T17:10:53.913" Title="Text Mining with Naive Bayes" Tags="|machine-learning|data-mining|predictive-modeling|text-mining|naive-bayes-classifier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27694" PostTypeId="1" AcceptedAnswerId="27695" CreationDate="2018-02-11T08:21:22.517" Score="15" ViewCount="16957" Body="&lt;p&gt;If I look at one of the many sources for the Imagenet classes on the &lt;a href=&quot;https://github.com/Lasagne/Recipes/blob/master/examples/resnet50/imagenet_classes.txt&quot; rel=&quot;noreferrer&quot;&gt;Internet&lt;/a&gt; I cannot find a single class related to human beings (and no, &lt;em&gt;harvestman&lt;/em&gt; is not someone who harvests, but it's what I knew as a daddy longlegs, a kind of spider :-). How is that possible? I would have at least expected a &lt;code&gt;person&lt;/code&gt; class, and even something more specific such as &lt;code&gt;man&lt;/code&gt;, &lt;code&gt;woman&lt;/code&gt;, &lt;code&gt;toddler&lt;/code&gt;, etc. Nothing of the sort. Why? Did &lt;em&gt;Fei-Fei Li&lt;/em&gt; and her team make a conscious choice not to have people images in the database? Am I looking at the wrong file? For the sake of the question, we can consider the &lt;code&gt;ImageNet&lt;/code&gt; versions from 2014 onwards.&lt;/p&gt;&#xA;" OwnerUserId="27526" LastEditorUserId="29169" LastEditDate="2021-02-13T16:42:04.160" LastActivityDate="2021-02-13T16:42:04.160" Title="Is there a person class in ImageNet? Are there any classes related to humans?" Tags="|machine-learning|deep-learning|dataset|image-classification|convolutional-neural-network|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27695" PostTypeId="2" ParentId="27694" CreationDate="2018-02-11T08:40:05.703" Score="8" Body="&lt;p&gt;You can also take a look at &lt;a href=&quot;https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; for the labels in the imagenet. I guess you are right, there is no label for &lt;em&gt;human&lt;/em&gt; in the data-set but there is something to notice. There are labels in imagenet like cowboy or some specific hats and other related things to human like shirt and t-shirt. You can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt; and also &lt;a href=&quot;http://yosinski.com/deepvis&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. In the latter link Yosinski et al, has tried to show that the popular AlexNet has learned to recognize human faces although there is no label as human face in the imagenet data-set. In their paper, they have investigated that Convolutional neural networks may try to learn things that are distributed among layers or maybe not and they may not have special label in the training data. As an example, the face of cats and humans can be referred to. Moreover, as you can see &lt;a href=&quot;http://image-net.org/downloads/attributes/README&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; maybe the aim was &lt;em&gt;attribute learning in large-scale datasets&lt;/em&gt;, as quoted in the last line of the page, as the reference.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-11T08:40:05.703" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27696" PostTypeId="2" ParentId="27541" CreationDate="2018-02-11T08:55:45.310" Score="5" Body="&lt;p&gt;Try: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pip install pandas-profiling&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda install -c anaconda pandas-profiling&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="29575" LastEditDate="2018-02-11T09:13:34.983" LastActivityDate="2018-02-11T09:13:34.983" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27699" PostTypeId="2" ParentId="27693" CreationDate="2018-02-11T17:10:53.913" Score="1" Body="&lt;p&gt;Is there any reason you have to use Naive Bayes? While Naive Bayes does handle multi-class modeling quite nicely, sometimes the &quot;naive&quot; assumption that each word in the text is independent of the others is &lt;em&gt;too&lt;/em&gt; naive, especially given the size of your training set. While you may be able to increase the accuracy a little bit by doing more text processing, I wouldn't expect to see significant improvement, especially given the training size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since Naive Bayes itself doesn't have much parameter tweaking, if you want to try more processing of the data itself to improve performance, you can try things like text &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;count vectorization&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF vectorization&lt;/a&gt; to represent the text data more holistically than just keyword flagging.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are able to implement other models, I would take a look at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit learn multi-class models&lt;/a&gt; for ideas. Tree based methods are also inherently multiclass, and have more parameters to tweak, so implementing something like Random Forests may suit you better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the training size itself, that does seem a little small. Especially if you are using text data, the more features you include, the more data you need to be able to stand a chance of detecting a signal in the data.&lt;/p&gt;&#xA;" OwnerUserId="45531" LastActivityDate="2018-02-11T17:10:53.913" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27700" PostTypeId="1" AcceptedAnswerId="31822" CreationDate="2018-02-11T17:45:40.577" Score="3" ViewCount="682" Body="&lt;p&gt;I have multi-label data for semantic segmentation. For semantic segmentation for one class I get a high accuracy but I can't do it for multi-class segmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 6 class labels so my Y train matrix is equal [78,480,480,6] ('channel last'), where 78 - number of images 480,480 -image size, 6-number of masks and X train matrix [78, 480, 480, 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last lines of my CNN model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l = Conv2D(filters=64, kernel_size=(1,1), activation='relu')(l)&#xA;output_layer = Conv2D(filters=6, kernel_size=(1,1), activation='sigmoid')(l)&#xA;&#xA;model = Model(input_layer, output_layer)&#xA;model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know what I hove done wrong that my multi-label semantic segmentation model doesn't have proper results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image and masks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kvJ2p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kvJ2p.png&quot; alt=&quot;Image and Masks&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46104" LastEditorUserId="29575" LastEditDate="2018-02-11T17:53:34.370" LastActivityDate="2018-05-18T17:56:54.693" Title="Sementic segmentation data and model compile in Keras" Tags="|deep-learning|keras|optimization|loss-function|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27701" PostTypeId="1" CreationDate="2018-02-11T17:47:49.227" Score="2" ViewCount="90" Body="&lt;p&gt;Google has a DS / AI / ML / engineering themed &lt;a href=&quot;https://aiyprojects.withgoogle.com/vision&quot; rel=&quot;nofollow noreferrer&quot;&gt;DIY kit&lt;/a&gt; for voice recognition and computer vision.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mCVz0.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mCVz0.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The big manufacturers (Micro Center, Mouser, Seeed, etc) who partnered with Google are sold out and currently don't list any future availability dates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These projects are kind of &lt;a href=&quot;https://liliputing.com/2017/11/google-introduces-45-aiy-vision-kit-diy-computer-vision-hardware-projects.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;publicity&lt;/a&gt;, goodwill, and community-engagement efforts by Google, very much in the vein of Google Cardboard. They're meant to be cheap, accessible, and easily cloned for popular use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All of that is great and it's an attractive hobby for a Data Scientist, but the problem is that Supply &amp;lt; Demand due to low manufacturer price points (part of an agreement with Google to limit &lt;a href=&quot;https://economics.stackexchange.com/questions/10192/what-does-quantity-represent-when-looking-at-supply-and-demand&quot;&gt;quantity&lt;/a&gt; of supply whilst simultaneously restraining price, thus resulting in a shortage). As a result, we are now getting price gouged on eBay by resellers for &gt;$250/each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/d7qdu.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/d7qdu.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike with &lt;a href=&quot;https://vr.google.com/cardboard/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Cardboard&lt;/a&gt;, most of the businesses that might clone and sell their versions of it have not yet gotten into the game (to my knowledge).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to piece together a list of what you'd need to buy to make your own? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's what I have so far. Some of this (the Pi) was meant to be purchased separately anyway:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.microcenter.com/product/463611/8MP_Raspberry_Pi_Camera_Module&quot; rel=&quot;nofollow noreferrer&quot;&gt;Raspberry Pi Zero W board&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.microcenter.com/product/463611/8MP_Raspberry_Pi_Camera_Module&quot; rel=&quot;nofollow noreferrer&quot;&gt;Raspberry Pi Camera 2&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;Raspberry Pi Zero W compatible power supply (SKU: 001917)&lt;/li&gt;&#xA;&lt;li&gt;Micro SD card (available everywhere offline and online)&lt;/li&gt;&#xA;&lt;li&gt;Cardboard (I have questions on this below)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.radioshack.com/products/radioshack-108db-piezo-buzzer?gclid=CjwKCAiA8P_TBRA9EiwAJrpHM9Zr8NPDAQWFhqagk31u0KaVHM322v65YO5t8oCs6ZWj4Wsn7DpSrhoCGa8QAvD_BwE&quot; rel=&quot;nofollow noreferrer&quot;&gt;Piezo buzzer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the cardboard are there any CNC files or do you just transcript/print their design and manually cut your own? For now I'll start with a cereal box or something and cut it to approx. that size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the button and status LED? Can we find the same or sufficiently similar sold separately such that one could follow along with Google AIY projects, or demo projects for AIY users to replicate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yx3o8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yx3o8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Not yet located:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;VisionBonnet accessory board (×1)&lt;/li&gt;&#xA;&lt;li&gt;11mm plastic standoffs (×2)&lt;/li&gt;&#xA;&lt;li&gt;24mm RGB arcade button and nut (×1)&lt;/li&gt;&#xA;&lt;li&gt;Privacy LED (×1)&lt;/li&gt;&#xA;&lt;li&gt;LED bezel (×1)&lt;/li&gt;&#xA;&lt;li&gt;Tripod mounting nut (×1)&lt;/li&gt;&#xA;&lt;li&gt;Lens, lens washer (1 spare), and lens magnet (×1)&lt;/li&gt;&#xA;&lt;li&gt;Button harness (×1)&lt;/li&gt;&#xA;&lt;li&gt;Pi0 camera flat flex cable (×1)&lt;/li&gt;&#xA;&lt;li&gt;MIPI flat flex cable (×1) ... is &lt;a href=&quot;https://www.aliexpress.com/item/Raspberry-Pi-Zero-W-Camera-Cable-30-CM-Ribbon-Line-FFC-Cable-for-Raspberry-Pi-Zero/32820946214.html?src=google&amp;amp;albslr=221224290&amp;amp;isdl=y&amp;amp;aff_short_key=UneMJZVf&amp;amp;source=%7Bifdyn:dyn%7D%7Bifpla:pla%7D%7Bifdbm:DBM&amp;amp;albch=DID%7D&amp;amp;src=google&amp;amp;albch=shopping&amp;amp;acnt=708-803-3821&amp;amp;isdl=y&amp;amp;albcp=653153647&amp;amp;albag=34728528644&amp;amp;slnk=&amp;amp;trgt=75384829977&amp;amp;plac=&amp;amp;crea=en32820946214&amp;amp;netw=g&amp;amp;device=c&amp;amp;mtctp=&amp;amp;gclid=CjwKCAiA8P_TBRA9EiwAJrpHMxY0IDssMiq9fJq668sZ3pmSrUwgaMpmDRcF6dQU6bniIjX6Yp3BARoCzYoQAvD_BwE&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; is?&lt;/li&gt;&#xA;&lt;li&gt;External cardboard box (×1)&lt;/li&gt;&#xA;&lt;li&gt;Internal cardboard frame (×1)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="2723" LastEditorUserId="2723" LastEditDate="2018-02-11T19:13:28.463" LastActivityDate="2018-02-11T19:13:28.463" Title="Piecing together an equivalent of Google's Data Science / Engineering AIY Computer Viz kit" Tags="|computer-vision|google|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27702" PostTypeId="1" CreationDate="2018-02-11T19:13:40.950" Score="2" ViewCount="463" Body="&lt;p&gt;I'm looking at working on a machine learning project for a company, where they are interested in paying in instalments at certain milestones in the project. My initial thoughts are that how to define these milestones?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's development of a recommender system and deployment on their site.  It might take up to 12 months in worst case scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking to ask for payment at these points:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;before starting the project (0 months)&lt;/li&gt;&#xA;&lt;li&gt;when I demo a prototype (low accuracy but bare bones functionality) (3 months)&lt;/li&gt;&#xA;&lt;li&gt;when I get the prototype running on live data (higher accuracy) (6 months)&lt;/li&gt;&#xA;&lt;li&gt;when it passes A/B testing and user evaluation (9 months)&lt;/li&gt;&#xA;&lt;li&gt;when it's fully deployed (12 months)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What I am not sure of is, is it practical to define milestones like this? I am also wary of defining acceptance criteria like percentage accuracy since it's really hard to work out what would be acceptable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am also looking into this standard for ideas for milestones: &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which would suggest Business Understanding, Data Understanding, Data Preparation, Modelling and Evaluation. However these are more fluid &quot;phases&quot; rather than milestones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to find out if there's an established best way to scope and plan projects like this and how to define acceptability at each stage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="46107" LastActivityDate="2018-02-11T19:13:40.950" Title="Milestones of data science project" Tags="|recommender-system|project-planning|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-12T19:33:32.027" ContentLicense="CC BY-SA 3.0" />
  <row Id="27703" PostTypeId="1" CreationDate="2018-02-11T19:31:18.993" Score="0" ViewCount="414" Body="&lt;p&gt;Using tensorflow/Keras, I have built a good model which is currently binary classification.  For example, training labels would be images of a person's knees bent or knees not bent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea behind such a model could be using a continuous video feed, and when it detects either knees bent or not, a certain probability would output.  It works, however, since it's binary, I have probabilities close to 0 or 1 even though a person is not even in the video feed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas on how I can change my model (perhaps multi-class) in over a continuous feed - which could be hundreds of images over a minute would only give me probabilities for those events of knees bent or not?  If going multi-class, would I then have to also train all other potential images?  That seems close to impossible, even with using transfer learning with ImageNet.  Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="32754" LastActivityDate="2018-07-25T22:00:21.287" Title="Using Binary Image Classification on Video" Tags="|keras|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27704" PostTypeId="1" CreationDate="2018-02-11T19:40:45.207" Score="0" ViewCount="1926" Body="&lt;p&gt;I'm trying to use &lt;code&gt;GridSearchCV&lt;/code&gt; with &lt;code&gt;RidgeClassifier&lt;/code&gt;, but I'm getting this error: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is &lt;strong&gt;regression&lt;/strong&gt; type.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;IndexError: too many indices for array&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'm new to Machine Learning, please help me out.&#xA;This is the code I've been trying to implement: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.grid_search import GridSearchCV&#xA;from sklearn.metrics import classification_report&#xA;&#xA;tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],&#xA;                     'C': [1, 10, 100, 1000]},&#xA;                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]&#xA;scores = ['precision', 'recall']&#xA;alphas = np.array([1,0.1,0.01,0.001,0.0001,0])&#xA;model = RidgeClassifier(normalize=True, random_state=100, tol=0.1)&#xA;for score in scores:&#xA;    clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))&#xA;    clf.fit(X, Y)&#xA;    print(&quot;Best parameters set found on development set:&quot;)&#xA;    print(clf.best_params_)&#xA;    for params, mean_score, scores in clf.grid_scores_:&#xA;        print(&quot;%0.3f (+/-%0.03f) for %r&quot;&#xA;              % (mean_score, scores.std() * 2, params))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the complete error log: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------------------------&#xA;IndexError                                Traceback (most recent call last)&#xA;&amp;lt;ipython-input-59-c97c7e0fc6f3&amp;gt; in &amp;lt;module&amp;gt;()&#xA;     12 for score in scores:&#xA;     13     clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))&#xA;---&amp;gt; 14     clf.fit(X, Y)&#xA;     15     print(&quot;Best parameters set found on development set:&quot;)&#xA;     16     print(clf.best_params_)&#xA;&#xA;/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in fit(self, X, y)&#xA;    836 &#xA;    837         &quot;&quot;&quot;&#xA;--&amp;gt; 838         return self._fit(X, y, ParameterGrid(self.param_grid))&#xA;    839 &#xA;    840 &#xA;&#xA;/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)&#xA;    551                                  'of samples (%i) than data (X: %i samples)'&#xA;    552                                  % (len(y), n_samples))&#xA;--&amp;gt; 553         cv = check_cv(cv, X, y, classifier=is_classifier(estimator))&#xA;    554 &#xA;    555         if self.verbose &amp;gt; 0:&#xA;&#xA;/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in check_cv(cv, X, y, classifier)&#xA;   1833         if classifier:&#xA;   1834             if type_of_target(y) in ['binary', 'multiclass']:&#xA;-&amp;gt; 1835                 cv = StratifiedKFold(y, cv)&#xA;   1836             else:&#xA;   1837                 cv = KFold(_num_samples(y), cv)&#xA;&#xA;/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in __init__(self, y, n_folds, shuffle, random_state)&#xA;    568         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):&#xA;    569             for label, (_, test_split) in zip(unique_labels, per_label_splits):&#xA;--&amp;gt; 570                 label_test_folds = test_folds[y == label]&#xA;    571                 # the test split can be too big because we used&#xA;    572                 # KFold(max(c, self.n_folds), self.n_folds) instead of&#xA;&#xA;IndexError: too many indices for array&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45224" LastActivityDate="2018-02-12T11:26:06.543" Title="How to use GridSearchCV with RidgeClassifier" Tags="|python|scikit-learn|grid-search|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27705" PostTypeId="2" ParentId="27654" CreationDate="2018-02-11T21:02:49.703" Score="1" Body="&lt;p&gt;Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper &lt;a href=&quot;https://arxiv.org/pdf/1801.07779.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The WiLI benchmark dataset for written language identification&lt;/a&gt;, page 4 &quot;Single-Character Frequency Analysis&quot;.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-02-11T21:02:49.703" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27707" PostTypeId="1" AcceptedAnswerId="27710" CreationDate="2018-02-11T22:39:19.800" Score="1" ViewCount="426" Body="&lt;p&gt;I am creating a CNN to categorise a sentence into one of N possible labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used the tutorial &lt;a href=&quot;http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/%20here&quot; rel=&quot;nofollow noreferrer&quot;&gt;from WildML&lt;/a&gt; to start the code, and I have modified it to allow multiple outputs instead of just true/false.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using VocabularyProcessor to convert the words into numbers and then using OneHotEncoder to transform the labels for evaluation. I am currently using the same VocabularyProcessor for converting both X and y&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where x_text is the loaded sentences and y_text is the correct labels&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;max_document_length = max([len(x.split(&quot; &quot;)) for x in x_text])&#xA;vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)&#xA;X = np.array(list(vocab_processor.fit_transform(x_text)))&#xA;&#xA;y_val = np.array(list(vocab_processor.transform(y_text)))&#xA;&#xA;from sklearn.preprocessing import OneHotEncoder&#xA;onehotencoder = OneHotEncoder(categorical_features = [0])&#xA;y = onehotencoder.fit_transform(y_val).toarray()&#xA;y = y[:, 1:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This then goes through the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem that I am finding is that during evaluation, if the labels are not exactly the same then the indexes do not match the prediction that is coming out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically what my question is, is how do I match up the output of the evaluation with the text label so that I can use it after training without hard coding the labels up front&lt;/p&gt;&#xA;" OwnerUserId="35528" LastActivityDate="2018-02-12T05:33:55.887" Title="How to match up categorical labels in training and evaluation" Tags="|python|tensorflow|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27708" PostTypeId="1" CreationDate="2018-02-12T00:29:24.637" Score="0" ViewCount="183" Body="&lt;p&gt;I have weights which I use for initializing the training of a convolution neural net model as follows (in Keras): &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = load_model(&quot;my_weights.hdf5&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I want to rerun the entire model with further data augmentation and training using these loaded weights. I would like to know if this is a feasible approach to apply. And also, would this be considered as fine-tuning? or transfer learning?&lt;/p&gt;&#xA;" OwnerUserId="45898" LastActivityDate="2018-06-13T10:40:47.573" Title="How to describe the process of using existing weights for training a deep learning model?" Tags="|machine-learning|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27709" PostTypeId="1" CreationDate="2018-02-12T01:26:35.920" Score="7" ViewCount="7749" Body="&lt;p&gt;I have some movement data sampled over a time series. I am trying to classify the movements in real time as either smooth or shaky. For example, as the movement is smooth it is classified as smooth until it suddenly becomes shaky and is then classified as so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having some serious trouble doing this. Aside from using an RNN and feeding it subsets of the time series data to classify (which I do not have enough data to train), does anybody know of any time series classification algorithms? I plan on feeding subsets of the time series into the algorithm with a single label for training and prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is greatly appreciated. &lt;/p&gt;&#xA;" OwnerUserId="46120" LastActivityDate="2018-08-12T02:01:32.257" Title="How to classify movement data (time series) in real time" Tags="|machine-learning|classification|time-series|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27710" PostTypeId="2" ParentId="27707" CreationDate="2018-02-12T03:07:56.363" Score="3" Body="&lt;p&gt;You are going about it all wrong. You shouldn't transform your labels into numerical values and match them to the output; that's not how classification is done. Instead, for each class (category) you will add a node in the output layer; the predicted class is then the &lt;code&gt;argmax&lt;/code&gt; of the output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let's say you have three classes: &lt;code&gt;simple&lt;/code&gt;, &lt;code&gt;compound&lt;/code&gt; and &lt;code&gt;complex&lt;/code&gt;. We'll assign an integer for each class starting from $0$.&#xA;Your output layer will have three nodes, one for each class, and for each training example $x$, the true label $y$ is a three dimensional &quot;hot-encoded&quot; vector. (i.e $y = (y_0, y_1, y_2)$ where $y_i = 1$ if $x$ belongs to class $i$ otherwise $y_i = 0$ for $i \in \{0,1,2\}$).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let's suppose you feed your (trained) network an input value &lt;code&gt;x&lt;/code&gt; and it outputs the following vector:&#xA;$y' = (0.4, 1.4, 2.5)$. The predicted label in this case is $\arg\max(y') = 2$. So the prediction for the sentence &lt;code&gt;x&lt;/code&gt; is: &lt;code&gt;complex&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: since &lt;code&gt;argmax&lt;/code&gt; is not differentiable, we use &lt;code&gt;softmax&lt;/code&gt; during training to convert the output vector &lt;code&gt;y'&lt;/code&gt; into a vector of probabilities that sums up to $1$ before applying the loss function. The prediction in this case is the class with the highest probability. For example, if we apply &lt;code&gt;softmax&lt;/code&gt; to the same &lt;code&gt;y'&lt;/code&gt; from the example above, we get: $\text{softmax}(y') = (0.08, 0.23, 0.69)$. It's clear that the highest probability ($69\%$) belongs to the class $2$ (&lt;code&gt;complex&lt;/code&gt;) and that these probabilities sum up to $1$.&lt;/p&gt;&#xA;" OwnerUserId="46123" LastEditorUserId="46123" LastEditDate="2018-02-12T05:33:55.887" LastActivityDate="2018-02-12T05:33:55.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27712" PostTypeId="2" ParentId="27628" CreationDate="2018-02-12T06:37:49.927" Score="6" Body="&lt;p&gt;LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs to remember on their own.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LSTM's are already prone to overfitting, and if you feed in lots of redundant data with a sliding window then yes, they are likely to overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, a sliding window is necessary for time series forecasting with Feedforward Neural Networks, because FNNs require a fixed size input and do not have memory, so this is the most natural way to feed them time series data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether or not the FNN will overfit depends on its architecture and your data, but all standard regularization techniques will apply if it does. For example you can try choosing a smaller network, L2 regularization, Dropout, etc.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-12T06:43:03.563" LastActivityDate="2018-02-12T06:43:03.563" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27713" PostTypeId="1" CreationDate="2018-02-12T07:33:11.487" Score="6" ViewCount="1194" Body="&lt;p&gt;I have been reading through &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap3.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this book&lt;/a&gt; and am trying to do the exercises.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is &lt;em&gt;&quot;Connecting regularization and the improved method of weight initialization&quot;&lt;/em&gt; part 3. We have to use a heuristic argument to prove &quot;the weight decay will tail off when the weights are down to a size around &lt;span class=&quot;math-container&quot;&gt;$1/\sqrt{n},$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; is the total number of weights in the network.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more context on the same problem (Parts 1 and 2):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/17921/heuristic-argument-for-weight-decay-and-regularization&quot;&gt;Heuristic argument for Weight decay and regularization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/15016/speed-decay-proof-for-l2-regularization-and-non-normalizied-weight-initiation?answertab=active#comment16776_15016&quot;&gt;Speed decay proof for L2 regularization and non-normalizied weight initiation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevant equation appears to be this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$C= -\frac{1}{n_t} \sum_{xj} [y_j \ln{a^L_j}+(1−y_j)\ln{(1−a^L_j)}]+\frac{\lambda}{2n_t}\sum_{w}w^2.$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The book seems to use the &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; notation to represent two different things. I've changed it so that &lt;span class=&quot;math-container&quot;&gt;$n_t$&lt;/span&gt; refers to the size of the training set, and &lt;span class=&quot;math-container&quot;&gt;$n_w$&lt;/span&gt; refers to the number of weights in the network)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it has something to do with the last term, &lt;span class=&quot;math-container&quot;&gt;$\frac{\lambda}{2n}\sum_{w}w^2$&lt;/span&gt;, because if we substitute &lt;span class=&quot;math-container&quot;&gt;$w=\frac{1}{\sqrt{n_w}}$&lt;/span&gt; then the whole term simplifies to &lt;span class=&quot;math-container&quot;&gt;$\frac{\lambda}{2n}$&lt;/span&gt;. This would mean that the partial derivative with respect to the weight becomes &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial w}$&lt;/span&gt; instead of &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial w}+\frac{\lambda}{n}w$&lt;/span&gt;. Is this enough of an explanation to conclude that weight decay falls off once the weights are around &lt;span class=&quot;math-container&quot;&gt;$\frac{1}{\sqrt{n}}?$&lt;/span&gt;&lt;/p&gt;&#xA;" OwnerUserId="46129" LastEditorUserId="29575" LastEditDate="2018-09-22T01:14:07.727" LastActivityDate="2018-11-08T07:03:00.560" Title="Weight decay in neural network" Tags="|machine-learning|neural-network|deep-learning|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27714" PostTypeId="1" CreationDate="2018-02-12T07:36:32.583" Score="1" ViewCount="250" Body="&lt;p&gt;I set up a neural network model with a training and test data set and saved the model with the &quot;Save Model&quot; - Widget. I wonder what kind of information the saved pickle-file contains? Does it only contain an neural-network-algorithm with all set parameters or does it depends on the used training data set?&lt;/p&gt;&#xA;" OwnerUserId="46127" LastActivityDate="2018-03-18T15:30:30.303" Title="Orange: Save Model" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27715" PostTypeId="1" CreationDate="2018-02-12T09:08:17.440" Score="0" ViewCount="1582" Body="&lt;p&gt;Most generative adversarial networks learn the distribution of the dataset and then generate a sample of 10's to 100's of images with similar distributions. I am curious if there is any research regenerating a single image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked into Super Resolution GAN's and they somewhat model this. They try to estimate a highly resolved image from its lower resolution counterpart. Could that be used to recreate an image?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help.&lt;/p&gt;&#xA;" OwnerUserId="46132" LastEditorUserId="29169" LastEditDate="2019-06-07T04:29:12.707" LastActivityDate="2019-06-07T04:29:12.707" Title="Using Generative Adversarial Network to generate Single Image" Tags="|deep-learning|gan|generative-models|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="27716" PostTypeId="1" CreationDate="2018-02-12T09:12:51.713" Score="3" ViewCount="1098" Body="&lt;p&gt;I studied the perceptron algorithm and I'm trying to prove the convergence by myself. However, I'm wrong somewhere and I am not able to find the error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assumption:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We assume that there is some $\gamma &amp;gt; 0$ such&#xA;that $$y_{t}(\theta ^{*})^{T}x_{t} \geq \gamma $$  for all $t = 1, \ldots , n$. The additional number $\gamma &amp;gt; 0$ is used to ensure that each example is classified correctly with a finite margin.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We will assume that all the (training) images have bounded&#xA;Euclidean norms, i.e., $$\left \| \bar{x_{t}} \right \|\leq R$$ for all $t$ and some finite $R$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;By hypothesis the learning rule is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta ^{(k)}= \theta ^{(k-1)} + \mu y_{t}\bar{x_{t}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, $$(\theta ^{*})^{T}\theta ^{(k)}=(\theta ^{*})^{T}\theta ^{(k-1)} + \mu y_{t}\bar{x_{t}} \geq  (\theta ^{*})^{T}\theta ^{(k-1)} + \mu \gamma $$&#xA;so , by induction &#xA;$$(\theta ^{*})^{T}\theta ^{(k)}\geq k\mu \gamma $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the same time,&#xA;$$\left \| \theta ^{(k)} \right \|^{2} = \left \| \theta ^{(k-1)}+\mu y_{t}\bar{x_{t}}  \right \|^{2} = \left \| \theta ^{(k-1)} \right \|^{2}+2\mu y_{t}(\theta ^{(k-1)^{^{T}}})\bar{x_{t}}+\left \| \mu \bar{x_{t}} \right \|^{2} \leq \left \| \theta ^{(k-1)} \right \|^{2}+\left \| \mu\bar{x_{t}} \right \|^{2}\leq \left \| \theta ^{(k-1)} \right \|^{2}+\mu ^{2}R^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, by induction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\left \| \theta ^{(k)} \right \|^{2} \leq k\mu ^{2}R^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can now combine parts 1) and 2) to bound the cosine of the angle between $\theta^∗$ and $\theta(k)$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\cos(\theta ^{*},\theta ^{(k)}) =\frac{\theta ^{*}\theta ^{(k)}}{\left \| \theta ^{*} \right \|\left \|\theta ^{(k)}  \right \|} \geq \frac{k\mu \gamma }{\sqrt{k\mu ^{2}R^{2}}\left \|\theta  ^{2} \right \|}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since cosine is bounded by one, we get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$k \leq \frac{R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that the correct result should be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$k \leq \frac{\mu ^{2}R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$&lt;/p&gt;&#xA;" OwnerUserId="44101" LastEditorUserId="29575" LastEditDate="2018-02-18T21:16:08.707" LastActivityDate="2018-02-18T21:16:08.707" Title="Perceptron - Convergence proof" Tags="|machine-learning|perceptron|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27717" PostTypeId="1" CreationDate="2018-02-12T09:13:44.400" Score="1" ViewCount="135" Body="&lt;p&gt;I have a number of participants from a study and am trying to classify them into five &quot;perceived effort&quot; zones. However, I am finding that the accuracy is sitting at around 70% - 75% using various machine learning algorithms, which is OK but I think this could be more accurate by making the features relative for each participant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each zone corresponds to the participant's perceived effort:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Zone 1 &amp;lt; 20% effort&#xA;Zone 2 &amp;lt; 40% effort&#xA;Zone 3 &amp;lt; 60% effort&#xA;Zone 4 &amp;lt; 80% effort&#xA;Zone 5 maximum effort&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to run the same algorithms on data that has feature values relative for each participant, because one participant's zone 3 maybe be the equivalent to another's zone 4. In other words, the zones are not uniform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One idea I had was to make the feature's relative to to each participants max of that feature. As an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Participant 1: Max running speed is 20 mph (running speed = Feature 1)&#xA;&#xA;run    zone        actualSpeed      relativeSpeed&#xA;1       5             20            20 / 20 = 1.0&#xA;2       4             17            17 / 20 = 0.85&#xA;3       3             13            13 / 20 = 0.65&#xA;4       2             10            10 / 20 = 0.50&#xA;5       1             8             18 / 20 = 0.40&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, a few of my features have values that range from -3.6 to + 2.5, and I am stuck on how to make such features relative to the participant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question: How to make sample feature values relative to the groups/participants for supervised classification machine learning?&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="41460" LastActivityDate="2018-02-12T09:13:44.400" Title="Ways to Make Feature Values Relative for Each Participant" Tags="|machine-learning|classification|feature-engineering|supervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27718" PostTypeId="2" ParentId="27598" CreationDate="2018-02-12T10:17:15.533" Score="3" Body="&lt;p&gt;Here are a few ways you might use neural networks to solve this problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With a plain Feedforward Neural Network:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Scale your data to fit in the square around the origin from (-1,-1) to (1,1)&lt;/li&gt;&#xA;&lt;li&gt;Represent each point with two inputs corresponding to its x and y coordinates, or 0,0 if the $k$th point is not present&lt;/li&gt;&#xA;&lt;li&gt;Add a third indicator input for each point, indicating whether that point is present&lt;/li&gt;&#xA;&lt;li&gt;Choose the number and size of hidden layers&lt;/li&gt;&#xA;&lt;li&gt;Use a softmax layer of size 10 at the output&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So every input example will be a vector of length 30, where the last 3*(10-$k$) values are zero when there are $k$ points present in the set, and the output is a vector of length 10 summing to 1, whether the largest value corresponds to the predicted point (whose position is corresponds to that position in the input).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With a Convolutional Neural Network:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Divide your plane into grid of $n$ x $n$ squares, and represent your input as an $n$ x $n$ matrix that is $k$ if there are $k$ points in the square ($i,j$) and $0$ otherwise. Hopefully points won't overlap, so you have a matrix of $1$s and $0$s.&lt;/li&gt;&#xA;&lt;li&gt;Train a CNN on your input matrices. Your output shape should be a softmax of size $n*n$, which corresponds to the input shape flattened. Choose the point with the highest value at the corresponding output coordinates. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The CNN might perform better since your data is inherently spatial. However you have to decide what to do if two or more points overlap. The simplest solution is to pick one randomly, which might be OK depending on your specific task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;With a Recurrent Neural Network:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Feed in variable length sequences of scaled (x,y) points and output a size 10 softmax estimation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Yes, it's as easy as that with RNNs! They handle variable length inputs well, but they still lack the advantages of CNNs for handling spatial data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Caveats:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If using a FNN or an RNN, there is also the matter of how you order your input data. If there is no inherent order in your real data, then we don't want our network making different predictions for the same data encoded in different orders. One way to handle this is with &lt;strong&gt;data augmentation&lt;/strong&gt;: duplicate each training example a few times with different input orderings, so hopefully your network can learn the appropriate symmetries.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only have time to try one approach, I would choose the CNN. CNNs are designed to do well with spatial data, and there is no problem with input orderings.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-12T11:42:27.477" LastActivityDate="2018-02-12T11:42:27.477" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27719" PostTypeId="2" ParentId="27709" CreationDate="2018-02-12T10:28:31.737" Score="3" Body="&lt;p&gt;Supervised learning for classification in machine learning trains a model in order to determine which distribution a certain novel instance belongs to. Model complexity is directly correlated with the performance of a specific algorithm given a set number of instances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means you should always start with understanding your data before selecting a model.  &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Your dataset&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Your dataset is comprised of instances $x_i \in \mathbb{R}^{300}, i \in {1, ..., n}$ and $10 &amp;lt; n &amp;lt; 50$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a very small number of instances $n$. Usually I recommend to have &lt;strong&gt;ten times more instances than features&lt;/strong&gt; for a two class classification problem. Furthermore, less than $50,000$ instances is not sufficient for training any deep learning model. You will need to consider using traditional machine learning algorithms (SVM, KNN, Random Forests, etc.). Lastly, the class imbalance in your dataset will introduce some bias, this will lead to poorer results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All hope is not lost, we can still do something to solve this problem!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Feature selection&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The first step which I would suggest is to consider performing some feature reduction. You want to map your $X \in \mathbb{R}^{300}$ to a lower dimensional set $\mathbb{R}^{p}$ where $p&amp;lt;300$. However, in removing features you will evidently be losing information, so you want to be able to learn what are the relevant features to your decision smooth/shaky. For example, when classifying cat/dog, the weight may be a highly informative feature thus its contribution in the decision should be highly considered, whereas the number of feet would not provide any useful information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some techniques to do this include PCA, LDA, and theres others as well. You can try them, its one line in scikit-learn. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Choosing an algorithm&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Due to the heavy class imbalance, I would suggest that classification based algorithms will perform poorly. In contrast, I would attempt to use anomaly detection algorithms. These techniques will learn the distribution of your nominal set (smooth) and attempt to detect when signals differ significantly from this distribution and will then classify them as anomalous (shaky).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many of these techniques exist, please refer to the following answers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/19392/using-time-series-data-from-a-sensor-for-ml/19394#19394&quot;&gt;Using time series data from a sensor for ML&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/18481/how-to-train-model-to-predict-events-30-minutes-prior-from-multi-dimensionnal-t/18483#18483&quot;&gt;How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-12T10:28:31.737" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27720" PostTypeId="1" CreationDate="2018-02-12T11:19:31.880" Score="1" ViewCount="723" Body="&lt;p&gt;I'm trying to use the work (Neural Networks) done in this repo:&#xA;&lt;a href=&quot;https://github.com/jtkim-kaist/VAD&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/jtkim-kaist/VAD&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It says this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note: To apply this toolkit to other speech data, the speech data&#xA;  should be sampled with 16kHz sampling frequency.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I've got speech data at 48khz. I've read in &lt;a href=&quot;https://sound.stackexchange.com/questions/23032/what-is-the-proper-way-to-reduce-audio-sample-rate-and-bit-depth&quot;&gt;places&lt;/a&gt; that reducing sampling rate is a complicated process, you can't just remove every nth datapoint, you have to filter things...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is this necessary if I only intend to use the data in the Neural Network toolkit provided by the repo I linked? &#xA;If so, is there an industry standard method for changing sample rate?&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise that it probably depends on what features are being used. However the feature that is used is this: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;MRCG (multi resolution cochleagra) concatenates the cochleagram features at multiple spectrotemporal resolutions&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a ruddy complicated feature! Lets pretend we're just using a Melspectogram (unless you're willing to answer the question from the perspective of MRCG's).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Neural networks are likely to use features of a Melspectogram that we wouldn't think of. This makes me think it is unwise to train the Neural Net using downsampled speech data unless we intend to predict using 48khz data downsampled to 16khz forever after...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think? Can I use my 48khz data - downsampled with no filtering - with the expectation that the model will work for prediction on real 16khz data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then for future readers sake, how about the other way? Say I had an 8khz file, could I increase the sample rate to 16khz without filtering?&lt;/p&gt;&#xA;" OwnerUserId="45602" LastActivityDate="2024-01-16T20:08:00.940" Title="Downsampling audio files for use in Machine Learning" Tags="|data-cleaning|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27721" PostTypeId="2" ParentId="27704" CreationDate="2018-02-12T11:26:06.543" Score="1" Body="&lt;p&gt;The problem is that Y has shape &lt;code&gt;(15780,1)&lt;/code&gt;. The shape of Y should show &lt;code&gt;(15780,)&lt;/code&gt;. The explanation for this is a bit long, so I just give here the links-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/22053050/difference-between-numpy-array-shape-r-1-and-r&quot;&gt;Difference between shapes (R,1) and (R,)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/31995175/scikit-learn-cross-val-score-too-many-indices-for-array&quot;&gt;Solution for IndexError: too many indices for array&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-02-12T11:26:06.543" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27722" PostTypeId="1" AcceptedAnswerId="27724" CreationDate="2018-02-12T11:29:41.277" Score="16" ViewCount="13407" Body="&lt;p&gt;I know that a neural-network architecture is mostly based on the problem itself and the types of input/output, but still - there's always a &quot;square one&quot; when starting to build one. So my question is - given a input dataset of &lt;em&gt;MxN&lt;/em&gt; (M  is the number of records, N is the number of features) and a C possible output classes - is there a thumb-rule to how many layers/units should we start with? &lt;/p&gt;&#xA;" OwnerUserId="43870" LastActivityDate="2020-08-05T14:57:55.870" Title="Is there a thumb-rule for designing neural-networks?" Tags="|neural-network|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27723" PostTypeId="2" ParentId="27458" CreationDate="2018-02-12T11:48:41.083" Score="0" Body="&lt;p&gt;I would try a combination of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A rolling-window variance to detect unusually large deflections&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Bollinger_Bands&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bollinger bands&lt;/a&gt; to detect the spikes&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Autocorrelation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autocorrelation&lt;/a&gt; to detect the 'up down up' pattern&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-12T11:48:41.083" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27724" PostTypeId="2" ParentId="27722" CreationDate="2018-02-12T12:00:08.887" Score="11" Body="&lt;p&gt;This question has been answered in detail on CrossValidated: &lt;a href=&quot;https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw&quot;&gt;How to choose the number of hidden layers and nodes in a feedforward neural network?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, let me add my own two cents:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no magic rule for choosing the best neural network architecture, but if you can find an architecture someone has used to solve a similar problem this is often an excellent starting point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best places to look are official or unofficial examples using popular neural network libraries  such as Keras, PyTorch, or Tensorflow, and architectures described in academic literature. &lt;a href=&quot;https://github.com/keras-team/keras/tree/master/examples&quot; rel=&quot;noreferrer&quot;&gt;keras/examples&lt;/a&gt; on github is a great resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These architectures were likely chosen after lots of trial and error, so most of the work will have been done for you.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-12T12:00:08.887" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27725" PostTypeId="1" CreationDate="2018-02-12T12:20:00.730" Score="1" ViewCount="645" Body="&lt;p&gt;I have a data set describing water levels of rivers. It has following attributes: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Value                   float64&#xA;Variable Name            object&#xA;Variable Unit Code       object&#xA;Date Pulled              object&#xA;Site Code                 int64&#xA;Site Latitude           float64&#xA;Site Longitude          float64&#xA;Site Name                object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using Linear Regression in trying to predict the &lt;code&gt;Value&lt;/code&gt; using &lt;code&gt;Site Code&lt;/code&gt;, &lt;code&gt;Site Longitude&lt;/code&gt; and &lt;code&gt;Site Latitude&lt;/code&gt; but getting results which are no way near the actual values. I am wondering if it's my choice of independent variables that is creating this problem or should I try other types of Regression? If it's the first case, which approaches are recommended to get higher accuracy rate or more precisely, what should one do if there are not many relevant independent variables?&lt;/p&gt;&#xA;" OwnerUserId="44701" LastActivityDate="2018-02-12T16:25:04.177" Title="Choosing input variables for Linear Regression for higher accuracy" Tags="|python|scikit-learn|linear-regression|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27726" PostTypeId="1" CreationDate="2018-02-12T13:31:46.740" Score="34" ViewCount="37275" Body="&lt;p&gt;In NLP, people tend to use cosine similarity to measure document/text distances. I want to hear what do people think of the following two scenarios, which to pick, cosine similarity or Euclidean?&lt;/p&gt;&#xA;&lt;p&gt;Overview of the task set: The task is to compute context similarities of multi-word expressions. For example, suppose we were given an MWE of &lt;code&gt;put up&lt;/code&gt;, context refers to the words on the left side of &lt;code&gt;put up&lt;/code&gt; and as well as the words on the right side of it in one text. Mathematically speaking, similarity in this task is about calculating&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;sim(context_of_using_&amp;quot;put_up&amp;quot;, context_of_using_&amp;quot;in_short&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Note that context is the feature that built on top of word embeddings, let's assume each word has an embedding dimension of &lt;code&gt;200&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;p&gt;Two scenarios of representing &lt;code&gt;context_of_an_expression&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;concatenate the left and right context words, producing an embedding vector of dimension &lt;code&gt;200*4=800&lt;/code&gt; if picking two words on each side. In other words, a feature vector of [lc1, lc2, rc1, rc2] is build for context, where &lt;code&gt;lc=left_context&lt;/code&gt; and &lt;code&gt;rc=right_context&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;get the mean of the sum of left and right context words, producing a vector of &lt;code&gt;200&lt;/code&gt; dimensions. In other words, a feature vector of [mean(lc1+lc2+rc1+rc2)] is built for context.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;[Edited] For both scenarios, I think Euclidean distance is a better fit. Cosine similarity is known for handling scale/length effects because of normalization. But I don't think there's much to be normalized.&lt;/p&gt;&#xA;" OwnerUserId="41487" LastEditorUserId="85045" LastEditDate="2021-04-27T23:02:38.520" LastActivityDate="2022-05-16T15:54:19.580" Title="When to use cosine simlarity over Euclidean similarity" Tags="|machine-learning|nlp|clustering|similarity|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27727" PostTypeId="1" CreationDate="2018-02-12T13:32:53.903" Score="0" ViewCount="145" Body="&lt;p&gt;In any predictive modelling exercise we first start with defining observation window and perform window for the product/problem. Just wanted to know if the window is different for different predictors or it's same for all the predictor variables. for ex- variable A can be calculated in 3 year of observation window. Can another variable (Variable B) be calculated in 2 year of window ?    &lt;/p&gt;&#xA;" OwnerUserId="43647" LastActivityDate="2023-12-24T18:00:20.070" Title="Observation window in Predictive modelling" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27728" PostTypeId="1" AcceptedAnswerId="27757" CreationDate="2018-02-12T13:43:31.367" Score="2" ViewCount="111" Body="&lt;p&gt;I'm working on a project where I have to build a classifier using Deep Neural Networks and I will be dealing with big data. I am familiar with neural networks, yet not an expert. I can obviously find a huge amount of literature online regarding deep learning but mostly will be &lt;code&gt;convnets&lt;/code&gt; related.&#xA;Are there any good research papers for DNN-based classifiers or should I just stack more layers? I want to get some insight about the architecture and training process. &#xA;P.S: Using Deep Neural Networks is mandatory even if the accuracy is low, but it would be great if it worked out.&lt;/p&gt;&#xA;" OwnerUserId="45033" LastEditorUserId="28175" LastEditDate="2018-02-12T14:12:48.403" LastActivityDate="2018-02-13T08:21:28.700" Title="Recommended papers for deep learning based classification?" Tags="|machine-learning|neural-network|classification|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27729" PostTypeId="2" ParentId="27728" CreationDate="2018-02-12T14:05:35.770" Score="1" Body="&lt;p&gt;Maybe reading papers is a bit hard, because they usually don't explain everything. I suggest you watching &lt;a href=&quot;https://coursera.org/specializations/deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep learning specialization&lt;/a&gt; by professor Andrew Ng, which is brief and easy to figure out. If you insist on reading papers, &lt;a href=&quot;https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner&amp;#39;s-Guide-To-Understanding-Convolutional-Neural-Networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; three sequential articles may help you. Choosing neural networks highly depend on your task. For most computer vision tasks convolutional neural nets are used. You can also take a look at &lt;a href=&quot;https://datascience.stackexchange.com/tags/deep-learning/info&quot;&gt;here&lt;/a&gt; which contains different resources and discussions and many other useful stuff.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-12T14:05:35.770" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27730" PostTypeId="1" CreationDate="2018-02-12T14:26:15.390" Score="2" ViewCount="346" Body="&lt;p&gt;My corpus contains several posts having text for several companies i.e. each post contains information about several companies. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to cluster the information based on few company names that I can specify. Clustering should be based on some similarity matrix such as euclidean or cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm to use based on company name that I can specify and which similarity method to use?&lt;/p&gt;&#xA;" OwnerUserId="37422" LastActivityDate="2023-12-23T23:01:15.273" Title="How to cluster sentences based on company names from a post(s) containing several company names using similarity metric." Tags="|python|nlp|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27731" PostTypeId="2" ParentId="27725" CreationDate="2018-02-12T16:13:41.927" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I am using Linear Regression in trying to predict the Value using Site Code, Site Longitude and Site Latitude&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Unfortunately none of these three variables can go directly into linear regression. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Site Code&lt;/code&gt; looks like a numerical variable, but it is actually categorical. For example &lt;code&gt;Site Code = 10&lt;/code&gt; is probably orthogonal to &lt;code&gt;Site Code = 5&lt;/code&gt;, and should not be interpreted as twice as significant. The correct way to handle this is to create a boolean &lt;a href=&quot;https://en.wikipedia.org/wiki/Dummy_variable_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;dummy indicator variable&lt;/a&gt; for each possible site code. You can also use this method for some of your other &lt;code&gt;object&lt;/code&gt; variables which also appear to be categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Latitude&lt;/code&gt; and &lt;code&gt;Longitude&lt;/code&gt; will not work well in a linear regression because their relationship is highly non-linear. For example two points can have the same latitude/longitude but be very far apart. One typical method is to convert &lt;code&gt;(lat,lng)&lt;/code&gt; pairs into predefined zones, and treat the zone as a categorical variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final type of variable you have is a &lt;code&gt;date&lt;/code&gt;. You could convert this directly to a categorical variable, but it might be better to take only the month to reduce the number of categories and generalize seasonal effects better.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-12T16:13:41.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27732" PostTypeId="2" ParentId="27725" CreationDate="2018-02-12T16:25:04.177" Score="0" Body="&lt;p&gt;While it is always preferred to have more informative variables for building a model, in reality though, perhaps like your case, we have to live with what we have. Although, if I were you I would think of digging other external data relevant to water level like geographical data, climate (there should be many things out there that you can add to your current feature space). Anyhow your current situation, my educated guess is that a simple regression is not a good choice of model anyway. Simply it won't capture any nonlinear correlation between your independent and dependent variables suitable for prediction i.e. water levels of rivers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would strongly suggest, if you want to build a better predictive model hopefully performing better than a simple regression and still fast and easy, to go with Gradient Boosting Decision Trees (GBDT) either using &lt;a href=&quot;https://github.com/Microsoft/LightGBM&quot; rel=&quot;nofollow noreferrer&quot;&gt;LightGBM&lt;/a&gt;, &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost&lt;/a&gt;, or recently my favorite &lt;a href=&quot;https://github.com/catboost/catboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;Catboost&lt;/a&gt; implementations (otherwise you could think of Neural Network as well, but depends how much data you have etc). Each has its own pros and con, please check out this nice &lt;a href=&quot;https://www.youtube.com/watch?v=5CWwwtEM2TA&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; by Mateusz Susikin in PyData Conference 2017 going through some of their differences. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note when building a GBDT model, you need to be careful how to encode your categorical variables, and not least, how to include your independent continuous variables if they exist. Please go through these stackexchange &lt;a href=&quot;https://datascience.stackexchange.com/questions/27012/cleaning-input-data-with-pd-get-dummies/27014#27014&quot;&gt;post1&lt;/a&gt;, &lt;a href=&quot;https://datascience.stackexchange.com/questions/27539/choosing-a-model-for-dataset-with-categorical-variables/27549#27549&quot;&gt;post2&lt;/a&gt; where I discuss ways to handle them, or at least they may give you some hints.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-02-12T16:25:04.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27734" PostTypeId="1" AcceptedAnswerId="27761" CreationDate="2018-02-12T17:49:31.533" Score="1" ViewCount="2929" Body="&lt;p&gt;We have used a derivative of the P-hash algorithm to determine similar images when the user selects an image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This allows the customer to check out very similar looking photos together. It has also helped us cluster pictures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the user selects an image, we would like to show him/her how each of the similar images is different from the selected image, by highlighting the difference, and by classifying the difference.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(eg: this is the hard cover version of the book; The font is different on this cover; original dress in black: same dress in blue, green and yellow ) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure what kind of annotated training data would be useful for Tensorflow. Could someone please point us in the right direction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that we have successfully been able to classify single pictures and even do a bit of object detection by drawing bounded boxes, but we are unable to do pairwise difference identification.&lt;/p&gt;&#xA;" OwnerUserId="46164" LastEditorUserId="46164" LastEditDate="2018-02-12T17:56:30.457" LastActivityDate="2018-02-13T08:47:18.140" Title="Given that two images are similar, how do I find and classify the difference between them?" Tags="|deep-learning|tensorflow|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27735" PostTypeId="2" ParentId="27146" CreationDate="2018-02-12T18:22:29.083" Score="3" Body="&lt;p&gt;You are supposed to perform calculations on cost function element wise. Try using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.multiply&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.dot&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is an inner product for Matrices. &lt;/p&gt;&#xA;" OwnerUserId="46168" LastEditorUserId="2584" LastEditDate="2018-03-06T22:38:21.983" LastActivityDate="2018-03-06T22:38:21.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27740" PostTypeId="1" CreationDate="2018-02-12T19:42:33.897" Score="3" ViewCount="9860" Body="&lt;p&gt;For practicing linear regression, I am generating some synthetic data samples as follows.&lt;/p&gt;&#xA;&lt;p&gt;First it generates 2000 samples with 3 features (represented by &lt;code&gt;x_data&lt;/code&gt;). Then it generates &lt;code&gt;y_data&lt;/code&gt; (results as real y) by a small simulation. i.e. by assuming a linear dependence model: imaginary weights (represented by &lt;code&gt;w_real&lt;/code&gt;), bias (represented by &lt;code&gt;b_real&lt;/code&gt;), and adding some noise.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from mpl_toolkits.mplot3d import Axes3D&#xA;&#xA;&#xA;#create some test data and simulate results&#xA;x_data = np.random.randn(2000,3)&#xA;w_real = [0.3,0.5,0.1]&#xA;b_real = -0.2&#xA;&#xA;noise = np.random.randn(1,2000)*0.1&#xA;y_data = np.matmul(w_real,x_data.T) + b_real + noise&#xA;&#xA;print(len(x_data))&#xA;print(len(y_data[0]))&#xA;&#xA;fig = plt.figure()&#xA;ax = fig.add_subplot(111, projection='3d')&#xA;&#xA;x1 = x_data[:,0]&#xA;x2 = x_data[:,1]&#xA;x3 = x_data[:,2]&#xA;ax.scatter3D(x1, x2, x3, c=x3, cmap='Greens');&#xA;&#xA;plt.show()&#xA;&#xA;#actual implementation of liner regression&#xA;#compute y_pred, compare with y_data above etc etc&#xA;#assume more code here&#xA;&#xA;exit() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am trying to visualize above simulated samples (&lt;code&gt;x_data&lt;/code&gt; and &lt;code&gt;y_data&lt;/code&gt;) using matplotlib. I was able to plot x_data as displayed in following image. I would also like to visualize simulated results (&lt;code&gt;y_data&lt;/code&gt;) on this plot, may be with a different color. Motivation behind it is to visualize relation between x &amp;amp; y. How can I plot it so?&lt;/p&gt;&#xA;&lt;p&gt;Data dimensions:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;x_data&lt;/code&gt;: &lt;span class=&quot;math-container&quot;&gt;$2000 \times 3$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;y_data&lt;/code&gt;: &lt;span class=&quot;math-container&quot;&gt;$2000 \times 1$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Here are how sample data is displayed by above sample,&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qtqhx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qtqhx.png&quot; alt=&quot;synthetic data samples&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here is an example of what I would like to achieve. The image shows two different hypothesis represented by straight lines, in my case I would like to draw a single line representing &lt;code&gt;y_data&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eztrA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eztrA.png&quot; alt=&quot;Example from Web&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42774" LastEditorUserId="98307" LastEditDate="2020-08-01T18:43:42.940" LastActivityDate="2020-08-01T18:43:42.940" Title="Plotting multivariate linear regression" Tags="|machine-learning|python|linear-regression|visualization|matplotlib|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27744" PostTypeId="1" CreationDate="2018-02-12T21:14:31.670" Score="0" ViewCount="156" Body="&lt;p&gt;I am predicting number of vehicles in 4 traffic junctions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have following columns in my dataset :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;DateTime&lt;/li&gt;&#xA;&lt;li&gt;Junction_ID&lt;/li&gt;&#xA;&lt;li&gt;Number_of_vehicles&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;At the first glance , this problem may look like Time series regression. But, the data given seems like Linear Regression problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I have applied linear regression in the following manner :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Used &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;get_dummies&lt;/a&gt; extensively for all the columns. I used dummy variables for 31 days,24 hours ,7 days of weeks and 4 Junction Ids.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then applied Linear Regression model &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;&#xA;x_train, x_test, y_train, y_test = train_test_split(train_data,train_vehicles)&#xA;&#xA;clf.fit(x_train,y_train)&#xA;&#xA;import math&#xA;&#xA;pred=clf.predict(x_test)&#xA;&#xA;pred.shape #got result as (12030,)&#xA;&#xA;result = []&#xA;for x in pred:&#xA;result.append(math.ceil(x))&#xA;&#xA;from sklearn.metrics import mean_squared_error&#xA;&#xA;score=mean_squared_error(y_test, result)&#xA;rmse=math.sqrt(score)&#xA;print('RMSE is :', rmse)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am getting RMSE value as 10.636853077462394&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Since RMSE value is on lower side , can I say this model is decent ?&lt;/li&gt;&#xA;&lt;li&gt;Is there any other approach which I can use on this dataset ?&lt;/li&gt;&#xA;&lt;li&gt;Do I need to check for colinearity ?&lt;/li&gt;&#xA;&lt;li&gt;How can I check if multiple variables are interrelated ?&lt;/li&gt;&#xA;&lt;li&gt;Should I go for non-linear regression on this dataset ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="44477" LastActivityDate="2019-08-20T12:00:49.320" Title="Is linear regression fit for this data" Tags="|machine-learning|regression|linear-regression|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27745" PostTypeId="2" ParentId="27153" CreationDate="2018-02-12T23:22:53.770" Score="1" Body="&lt;p&gt;The predictions you are getting are logits, meaning the sum across all categories is 1. So the largest-number should be the category you are looking for. To get the category, you can use argmax to find the index of the maximum number. The cross entropy loss is a measure of discrepancy between predicted value and labels. In this case you are 95% accurate.&lt;/p&gt;&#xA;" OwnerUserId="33364" LastActivityDate="2018-02-12T23:22:53.770" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27746" PostTypeId="2" ParentId="27744" CreationDate="2018-02-13T01:33:20.190" Score="2" Body="&lt;p&gt;For the first question, it is important to recall that RMSE has the same unit as the dependent variable. It means that there is no absolute good or bad threshold, however you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would do some feature engineering (Create more variables: Time of Day, Day of Week, Month etc...) and run it through a Neural Network and then check for accuracy.  You may want to check if there is any correlation between the 4 junctions as well but if you run a NN, you don't have to.&lt;/p&gt;&#xA;" OwnerUserId="46186" LastEditorUserId="40853" LastEditDate="2018-02-13T01:48:50.723" LastActivityDate="2018-02-13T01:48:50.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27751" PostTypeId="2" ParentId="27506" CreationDate="2018-02-13T05:08:37.293" Score="17" Body="&lt;p&gt;A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from &lt;a href=&quot;http://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/&quot; rel=&quot;noreferrer&quot;&gt;this source&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h1&gt;Forward pass&lt;/h1&gt;&#xA;&lt;p&gt;As you observed the forward pass of the convolutional layer can be expressed as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where in our case &lt;span class=&quot;math-container&quot;&gt;$k_1$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$k_2$&lt;/span&gt; is the size of the kernel, in our case &lt;span class=&quot;math-container&quot;&gt;$k_1=k_2=2$&lt;/span&gt;. So this says for an output &lt;span class=&quot;math-container&quot;&gt;$x_{0,0} = 0.25$&lt;/span&gt; like you found. &lt;span class=&quot;math-container&quot;&gt;$m$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; iterate across the dimensions of the kernel.&lt;/p&gt;&#xA;&lt;h1&gt;Backpropagation&lt;/h1&gt;&#xA;&lt;p&gt;Assuming you are using the mean squared error (MSE) defined as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$E = \frac{1}{2}\sum_p (t_p - y_p)^2$&lt;/span&gt;,&lt;/p&gt;&#xA;&lt;p&gt;we want to determine&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w^l_{m', n'}}$&lt;/span&gt; in order to update the weights. &lt;span class=&quot;math-container&quot;&gt;$m'$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$n'$&lt;/span&gt; are the indices in the kernel matrix not be confused with its iterators. For example &lt;span class=&quot;math-container&quot;&gt;$w^1_{0,0} = -0.13$&lt;/span&gt; in our example. We can also see that for an input image &lt;span class=&quot;math-container&quot;&gt;$H$&lt;/span&gt;x&lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt; the output dimension after the convolutional layer will be&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$(H-k_1+1)$&lt;/span&gt;x&lt;span class=&quot;math-container&quot;&gt;$(W-k_2+1)$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In our case that would be &lt;span class=&quot;math-container&quot;&gt;$4$&lt;/span&gt;x&lt;span class=&quot;math-container&quot;&gt;$4$&lt;/span&gt; as you showed. Let's calculate the error term. Each term found in the output space has been influenced by the kernel weights. The kernel weight &lt;span class=&quot;math-container&quot;&gt;$w^1_{0,0} = -0.13$&lt;/span&gt; contributed to the output &lt;span class=&quot;math-container&quot;&gt;$x^1_{0,0} = 0.25$&lt;/span&gt; and every single other output. Thus we express its contribution to the total error as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \frac{\partial E}{\partial x^l_{i, j}} \frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This iterates across the entire output space, determines the error that output is contributing and then determines the contribution factor of the kernel weight with respect to that output.&lt;/p&gt;&#xA;&lt;p&gt;Let us call the contribution to the error from the output space delta for simplicity and to keep track of the backpropagated error,&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial x^l_{i, j}} = \delta^l_{i,j}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h3&gt;The contribution from the weights&lt;/h3&gt;&#xA;&lt;p&gt;The convolution is defined as&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$&lt;/span&gt;,&lt;/p&gt;&#xA;&lt;p&gt;thus,&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;By expanding the summation we end up observing that the derivative will only be non-zero when &lt;span class=&quot;math-container&quot;&gt;$m=m'$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$n=n'$&lt;/span&gt;. We then get&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Then back in our error term&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \delta_{i,j}^l o^{l-1}_{i+m', j+n'}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;h3&gt;Stochastic gradient descent&lt;/h3&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w^{l(t+1)}_{m', n'} = w^{l(t)}_{m', n'} - \eta \frac{\partial E}{\partial w^l_{m', n'}}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Let's calculate some of them&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from scipy import signal&#xA;o = np.array([(0.51, 0.9, 0.88, 0.84, 0.05), &#xA;              (0.4, 0.62, 0.22, 0.59, 0.1), &#xA;              (0.11, 0.2, 0.74, 0.33, 0.14), &#xA;              (0.47, 0.01, 0.85, 0.7, 0.09),&#xA;              (0.76, 0.19, 0.72, 0.17, 0.57)])&#xA;d = np.array([(0, 0, 0.0686, 0), &#xA;              (0, 0.0364, 0, 0), &#xA;              (0, 0.0467, 0, 0), &#xA;              (0, 0, 0, -0.0681)])&#xA;&#xA;gradient = signal.convolve2d(np.rot90(np.rot90(d)), o, 'valid')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;array([[ 0.044606,  0.094061], [ 0.011262,  0.068288]])&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Now you can put that into the SGD equation in place of &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial E}{\partial w}$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Please let me know if theres errors in the derivation.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Update: Corrected code&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="79595" LastEditDate="2020-07-10T23:13:22.507" LastActivityDate="2020-07-10T23:13:22.507" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27752" PostTypeId="2" ParentId="27153" CreationDate="2018-02-13T05:48:54.163" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;predictions = classifier.predict(x_test)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You have not provided the shape of your x_test but based on the documentation of the predict function that you should provide an array-like item, you are inputting an array-like input. Each output already shows the probability of each corresponding input. It seems that because the low values of predictions, they are smaller than &lt;code&gt;0.5&lt;/code&gt;, the predicted labels for your test data are all zero.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-13T05:48:54.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27753" PostTypeId="1" CreationDate="2018-02-13T06:29:54.903" Score="0" ViewCount="1428" Body="&lt;p&gt;I want to detect abnormal behaviour in a oil pipe where the oil is flowing with some constant pressure. I have a sensor which monitors and pressure over the time and push it to my cloud server.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got the dataset . My requirement is to do data analytics in python and find-out the abnormal pattern in the dataset over the time. And I need to suggest abnormal behaviors present in the dataset ., say for example., from 1 pm to 3:30 pm today , the pressure raises and falls may be due to some leakage in the pipe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can we do it by simple statistical model? or machine learning is required?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can please suggest the best suitable machine learning algorithm for this scenario?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also please mention the web links over here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="46190" LastActivityDate="2019-03-17T16:08:02.117" Title="Find-out abnormal behavior over the time" Tags="|machine-learning|dataset|time-series|statistics|probability|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27754" PostTypeId="1" CreationDate="2018-02-13T06:51:23.260" Score="0" ViewCount="3073" Body="&lt;p&gt;So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on the residuals of the tree from the last round?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have the following iris dataset:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/F4aUo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F4aUo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I get 3 error after running a classification tree. Then how do I fit another classification tree based on what I got wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And after then, how would I weight them to form an ensemble?&lt;/p&gt;&#xA;" OwnerUserId="46192" LastActivityDate="2018-02-13T11:17:50.693" Title="How to find the residuals of a classification tree in xgboost" Tags="|classification|xgboost|gbm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27755" PostTypeId="2" ParentId="27753" CreationDate="2018-02-13T07:11:48.067" Score="3" Body="&lt;h1&gt;Your problem definition&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You have time series data which is used to measure the pressure using your sensor. You wish to identify when the pressure recordings are abnormal. This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use a sliding window approach and use that as your feature space to detect the distribution of your recordings. The window length you will select $m$ is the first of your hyper-parameters that you will need to tune. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Using a statistical model&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The retained time series $X \in \mathbb{R}^m$, you can treat this signal as a queue, first in first out. When a new recording is defined then discard the oldest datapoint. For a given set $X$ of samples, get the mean and the standard deviation $\sigma$. If the new point exceeds a multiple of $\sigma$, usually this is set to $3\sigma$, but it depends on the expected variation of your sensor, then we flag a state change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can further extend this by using the generalized likelihood ratio test (GLRT) to determine when a new sample causes the point to fall significantly outside the distribution of the null hypothesis. In which case this indicates a state change (the new point came from a different distribution than the normal flow through the pipe).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Using machine learning&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;You can collect multiple instances of your time series and annotate them based on your experience as being nominal or anomalous. Then you will have a supervised 2-class classification problem. First you should attempt some feature extraction using PCA, LDA or similar techniques.. Then you can attempt to use all the fun algorithms available to you through scikit-learn (SVM, Random Forests, K-NN, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is a significantly higher number of nominal instances than anomalous ones, this will introduce bias to your model. Anomaly detection algorithms are better suited for these types of problems. These algorithms learn the distribution within which your nominal set should belong. Then for novel instances it evaluates the probability of it being contained in the learned distribution. If the probability is small, then the algorithm will flag the instance as anomalous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information on anomaly detection for time series refer to: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/19392/using-time-series-data-from-a-sensor-for-ml/19394#19394&quot;&gt;Using time series data from a sensor for ML&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/18481/how-to-train-model-to-predict-events-30-minutes-prior-from-multi-dimensionnal-t/18483#18483&quot;&gt;How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-13T07:11:48.067" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27756" PostTypeId="2" ParentId="27153" CreationDate="2018-02-13T07:39:41.380" Score="14" Body="&lt;p&gt;What you have are predicted class probabilities. Since you are doing binary classification, each output is the probability of the first class for that test example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To convert these to class labels you can take a threshold:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;probas = np.array([[0.4],[0.7],[0.2]])&#xA;labels = (probas &amp;lt; 0.5).astype(np.int)&#xA;print(labels)&#xA;[[1]&#xA; [0]&#xA; [1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For multiclass classification  where you want to assign one class from multiple possibilities you can use &lt;code&gt;argmax&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probas = np.array([[0.4, 0.1, 0.5],[0.7, 0.2, 0.1],[0.3, 0.4, 0.3]])&#xA;labels = np.argmax(probas, axis=-1)    &#xA;print(labels)&#xA;[2 0 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And to get these as one-hot encoded arrays you can use &lt;code&gt;LabelBinarizer&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import preprocessing&#xA;&#xA;lb = preprocessing.LabelBinarizer()&#xA;lb.fit_transform(labels)&#xA;array([[0, 0, 1],&#xA;       [1, 0, 0],&#xA;       [0, 1, 0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And for multilabel classification where you can have multiple output classes per example you can use thresholding again:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probas = np.array([[0.6, 0.1, 0.7],[0.7, 0.2, 0.1],[0.8, 0.9, 0.6]])&#xA;labels = (probas &amp;gt; 0.5).astype(np.int)&#xA;print(labels)&#xA;[[1 0 1]&#xA; [1 0 0]&#xA; [1 1 1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some packages provide separate methods for getting probabilities and labels, so there is no need to do this manually, but it looks like you are using Keras which only gives you probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a sidenote, this is not called &quot;normalization&quot; for neural networks. Normalization typically describes scaling your input data to fit in a nice range like [-1,1].&lt;/p&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2019-01-31T14:32:04.817" LastActivityDate="2019-01-31T14:32:04.817" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="27757" PostTypeId="2" ParentId="27728" CreationDate="2018-02-13T08:21:28.700" Score="1" Body="&lt;p&gt;If you are interested in very recent, highly advanced works I would highly recommend keeping an eye on Google Brain's research paper releases. They have categories from Machine Intelligence, NLP, Machine Translation, Machine Perception, etc.. Many of these research papers involve Deep NN's so it should satisfy your desires. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://research.google.com/pubs/papers.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://research.google.com/pubs/papers.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46197" LastActivityDate="2018-02-13T08:21:28.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27758" PostTypeId="2" ParentId="27671" CreationDate="2018-02-13T08:33:15.450" Score="16" Body="&lt;p&gt;Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;SMOTE works in feature space. It means that the output of SMOTE is not a synthetic data which is a real representative of a text inside its feature space.&lt;/li&gt;&#xA;&lt;li&gt;On one side SMOTE works with KNN and on the other hand, feature spaces for NLP problem are dramatically huge. KNN will easily fail in those huge dimensions. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So I can propose you two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do not care about the real text representation of new synthetic samples which I assume should be fine. You need to balance the distribution for your classifier not for a reader of text data. So apply SMOTE as traditional (however I usually use the solution 2 bellow so I do not gaurantee the result!) with some Dimensionality Reduction step.     &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Lets assume you want to make your data samples from minor class double using 3-NN. Ignore the major class(es) and keep only minor class samples.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) For each sample point in feature space choose 5 nearest neighbors. Then choose 3 of them randomly (isn't it unnecessary complicated? if I didn't want to explain the original algorithm I would say just choose 3 neighbors!)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3) For each dimension calculate the distance between sample and neighbors and multiply it in a random number between 0-1 and add it to the original value of sample in that dimension. (this complicated paragraph simply means for each dimension choose a random value between the original sample and that neighbor!)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;But I usually do another oversampling which is on the text (so more intuitive) and is kind of SMOTE.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) Ignore the major class. Get a length distribution of all documents in minor class so that we generate new samples according the the true document length (number of words/phrases). We assume we want to make the size of class triple (so producing $k=2$ synthetic documents per original document)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) Generate a sequence of $n$ random integers according to that distribution. It's used for determining the length of new synthetic document.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3) For each document: Choose one integer from random length sequence and $m$ random document whose length is close to the integer. Put tokens of all $m$ documents in a set and randomly choose $n$ tokens $k$ times. these are your $k$ new documents.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-06-28T23:34:19.067" LastActivityDate="2018-06-28T23:34:19.067" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="27759" PostTypeId="2" ParentId="27708" CreationDate="2018-02-13T08:36:52.373" Score="1" Body="&lt;p&gt;Yes it is definitely doable, and further more, its is recommended in many scenarios. For instance, if you wanted to train some generic image classifier it would be naive idea to start from scratch and, for instance, retrain ImageNets' dataset for 2-3 weeks simply trying to retrieve optimal weights that have already been previously computed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;if you simply take pre-trained weights and use them on a different domain, then that would be referred to as transfer learning. However, since you want to retrain the existing weights with additional training data then, yes, you would be &quot;fine-tuning&quot; the model. When you do this however, there are considerations that need to be taken. Do you want to fine tune every layer? Or only deeper layers that detect very specific features of the dataset? Considerations like these should be taken into account. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check out &lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/transfer-learning/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="46197" LastActivityDate="2018-02-13T08:36:52.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27761" PostTypeId="2" ParentId="27734" CreationDate="2018-02-13T08:47:18.140" Score="1" Body="&lt;p&gt;In order for you to do pairwise difference, you need to do the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate Ground truth data by drawing bounding boxes on elements you need comparison for and label them i.e if you want to compare text on the hard cover, you need to mark the text region and add a label to it like font type.&lt;/li&gt;&#xA;&lt;li&gt;Once you have ground truth data, you can then use object detection networks like different variants of Yolo to identify multi class objects in each document.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This should provide you a starting point to solving the problem at hand. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-02-13T08:47:18.140" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27764" PostTypeId="1" CreationDate="2018-02-13T10:55:38.227" Score="1" ViewCount="81" Body="&lt;p&gt;I am writing a paper on aspect based sentiment analysis and I need datasets appropriate having only positive and negative classes to compare my result to other papers. Two of them are below which have used yelp dataset:&#xA;&lt;a href=&quot;http://onlinepresent.org/proceedings/vol143_2017/41.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://onlinepresent.org/proceedings/vol143_2017/41.pdf&lt;/a&gt;&#xA;&lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-981-10-7605-3_78&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://link.springer.com/chapter/10.1007/978-981-10-7605-3_78&lt;/a&gt;&#xA;If possible I would like to have access to datasets they have used, does anyone has them?&lt;/p&gt;&#xA;" OwnerUserId="46201" LastActivityDate="2018-02-13T11:25:25.080" Title="Yelp dataset needed" Tags="|dataset|" AnswerCount="1" CommentCount="4" ClosedDate="2018-02-15T17:50:10.507" ContentLicense="CC BY-SA 3.0" />
  <row Id="27765" PostTypeId="2" ParentId="27754" CreationDate="2018-02-13T11:17:50.693" Score="2" Body="&lt;p&gt;There are 2 methods to &quot;continue a model training&quot;:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Training from another xgboost model&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If you have a pre-trained xgboost model, you can continue training using the &lt;code&gt;xgb_model&lt;/code&gt; parameter in &lt;code&gt;xgb.train&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;xgb_model  a previously built model to continue the training from.&#xA;  Could be either an object of class xgb.Booster, or its raw data, or&#xA;  the name of a file with a previously saved model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Training residual&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Using xgboost to training a residual or prediction of a model, you can following the demo provided in xgboost:&#xA;&lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simply speaking, you can use the &quot;base_margin&quot; element when preparing data.&lt;/p&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-02-13T11:17:50.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27766" PostTypeId="2" ParentId="27764" CreationDate="2018-02-13T11:21:47.997" Score="1" Body="&lt;p&gt;For you to access the dataset you need to register with your email first. Available in both JSON and SQL Here you can access the dataset. &lt;a href=&quot;https://www.yelp.com/dataset/download&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="25635" LastEditorUserId="25635" LastEditDate="2018-02-13T11:25:25.080" LastActivityDate="2018-02-13T11:25:25.080" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27767" PostTypeId="1" CreationDate="2018-02-13T14:03:39.623" Score="49" ViewCount="103278" Body="&lt;p&gt;I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I keep getting memory errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From your experience is it possible? If not do you know of a better way to go around this? (hive table? increase the size of my RAM to 64? create a database and access it from python)&lt;/p&gt;&#xA;" OwnerUserId="41807" LastEditorUserId="29575" LastEditDate="2018-02-13T14:58:05.327" LastActivityDate="2019-06-07T07:33:34.080" Title="Opening a 20GB file for analysis with pandas" Tags="|python|bigdata|pandas|anaconda|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27768" PostTypeId="2" ParentId="27613" CreationDate="2018-02-13T15:31:36.300" Score="1" Body="&lt;p&gt;To me, this describes &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_probability&quot; rel=&quot;nofollow noreferrer&quot;&gt;conditional probabilities&lt;/a&gt;. And the first thing that springs to my mind when it comes to learning conditional probabilities is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian network (BN)&lt;/a&gt;, which is an example of a probabilistic graphical model. In fact, &lt;a href=&quot;https://www.coursera.org/learn/probabilistic-graphical-models-3-learning/lecture/MOk4x/maximum-likelihood-estimation-for-bayesian-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;BNs can be fitted&lt;/a&gt; using Maximum Likelihood Estimation (MLE), which corresponds to counting occurrences (similar to &quot;Naive Bayes&quot;, if you have used that). An excellent book on this is &lt;a href=&quot;http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online&quot; rel=&quot;nofollow noreferrer&quot;&gt;Barber&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another model that corresponds to your probabilistic problem, and which is more neural network-like, is a &lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Restricted Boltzmann Machine (RBM)&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Mn38h.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Mn38h.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;http://deeplearning.net/tutorial/rbm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once again, this involves learning conditional probabilities, like $p(v_0 | h_0, h_1, h2), p(v_1 | h_0, h_1, h2), p(v_2 | h_0, h_1, h2), \ldots$. This is a &lt;a href=&quot;https://deeplearning4j.org/restrictedboltzmannmachine&quot; rel=&quot;nofollow noreferrer&quot;&gt;nice introduction&lt;/a&gt; to them. Unlike neural networks, which are trained using backpropagation and are not probabilistic, RBMs are indeed probabilistic and are trained using Contrastive Divergence.&lt;/p&gt;&#xA;" OwnerUserId="45615" LastActivityDate="2018-02-13T15:31:36.300" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27769" PostTypeId="2" ParentId="27624" CreationDate="2018-02-13T15:39:20.627" Score="7" Body="&lt;p&gt;Bernoulli models the presence/absence of a feature. Multinomial models the number of counts of a feature. Here's a &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/2uhhbh/difference_between_binomial_multinomial_and/co8xxls/&quot; rel=&quot;noreferrer&quot;&gt;concise explanation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Bernoulli_naive_Bayes&quot; rel=&quot;noreferrer&quot;&gt;Wikipedia&lt;/a&gt; warns that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Note that a naive Bayes classifier with a Bernoulli event model is not&#xA;  the same as a multinomial NB classifier with frequency counts&#xA;  truncated to one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To understand why, we should note that, as &lt;a href=&quot;https://geekyisawesome.blogspot.co.uk/2016/12/bernoulli-vs-binomial-vs-multinoulli-vs.html&quot; rel=&quot;noreferrer&quot;&gt;this page&lt;/a&gt; nicely explains,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Whereas the binomial distribution generalises the Bernoulli&#xA;  distribution across the number of trials, the multinoulli distribution&#xA;  generalises it across the number of outcomes, that is, rolling a dice&#xA;  instead of tossing a coin.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does this imply for us? Multinomial NB cares about counts for multiple features that do occur, whereas Bernoulli NB cares about counts for a single feature that do occur and counts for the same feature that do &lt;em&gt;not&lt;/em&gt; occur.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This means that, for example, Multinomial NB will classify a document based on the counts it finds of multiple keywords; whereas Bernoulli NB can only focus on a single keyword, but will also count how many times that keyword does not occur in the document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So they do model slightly different things. If you have discrete multiple features to worry about, you have to use Multinomial NB. But if you only have a single feature to worry about, then you can make a modelling choice based on the above.&lt;/p&gt;&#xA;" OwnerUserId="45615" LastEditorUserId="45615" LastEditDate="2018-02-14T09:12:22.417" LastActivityDate="2018-02-14T09:12:22.417" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27770" PostTypeId="1" CreationDate="2018-02-13T18:17:02.930" Score="2" ViewCount="180" Body="&lt;p&gt;I want to train a neural network that removes scratches from pictures. I chose a GAN architecture with a generator (G) and a discriminator (D) and two sets of images scratchy and non-scratchy, with similar motives. G in my setting uses mainly convolutional and deconvolutional layers with &lt;code&gt;ReLU&lt;/code&gt;. As input it uses the scratchy images. The D discriminates between the output of G and the non-scratchy images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example the generator would perform the following transformations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(128, 128, 3) &amp;gt; (128, 128, 128) &amp;gt; (128, 128, 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where the tuples contain (width, height, channels). Input and output need to have the same format. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in order to get output that have same global structure as the inputs (i.e. streets, houses etc) it seem I have to use filter-sizes and strides of 1 and basically pass the complete pictures through the network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the features I am looking at are rather small and local. It should be sufficient to use filters of up to 20 pixels in size for the convolutions. And then then apply the network to all parts of the input picture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be a good generator architecture for such a task? Would you agree with the structure of the generator or would you expect different designs to perform better on local changes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code for the generator that I use implemented in tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generator(x, batch_size, reuse=False):&#xA;    with tf.variable_scope('generator') as scope:&#xA;        if (reuse):&#xA;            tf.get_variable_scope().reuse_variables()&#xA;&#xA;        s = 1&#xA;        f = 1&#xA;&#xA;        assert ((WIDTH + f - 1) / s) % 1 == 0&#xA;&#xA;        keep_prob = 0.5&#xA;&#xA;        n_ch1 = 32&#xA;        w = init_weights('g_wc1', [f, f, CHANNELS, n_ch1])&#xA;        b = init_bias('g_bc1', [n_ch1])&#xA;        h = conv2d(x, w, s, b)&#xA;        h = bn(h, 'g_bn1')&#xA;        h = tf.nn.relu(h)&#xA;        h = tf.nn.dropout(h, keep_prob)&#xA;        h1 = h&#xA;&#xA;        n_ch2 = 128&#xA;        w = init_weights('g_wc2', [f, f, n_ch1, n_ch2])&#xA;        b = init_bias('g_bc2', [n_ch2])&#xA;        h = conv2d(h, w, s, b)&#xA;        h = bn(h, &quot;g_bn2&quot;)                &#xA;        h = tf.nn.relu(h)&#xA;        h = tf.nn.dropout(h, keep_prob)&#xA;        h2 = h&#xA;&#xA;        n_ch3 = 256&#xA;        w = init_weights('g_wc3', [f, f, n_ch2, n_ch3])&#xA;        b = init_bias('g_bc3', [n_ch3])&#xA;        h = conv2d(h, w, s, b)&#xA;        h = bn(h, &quot;g_bn3&quot;)                &#xA;        h = tf.nn.relu(h)&#xA;        h = tf.nn.dropout(h, keep_prob)      &#xA;&#xA;        output_shape = [batch_size, HEIGHT//s//s, WIDTH//s//s, n_ch2]&#xA;        w = init_weights('g_wdc3', [f, f, n_ch2, n_ch3])&#xA;        b = init_bias('g_bdc3', [n_ch2])&#xA;        h = deconv2d(h, w, s, b, output_shape)        &#xA;        h = bn(h, &quot;g_bnd3&quot;)                &#xA;        h = tf.nn.relu(h)&#xA;        h = h + h2&#xA;&#xA;        output_shape = [batch_size, HEIGHT//s, WIDTH//s, n_ch1]&#xA;        w = init_weights('g_wdc2', [f, f, n_ch1, n_ch2])&#xA;        b = init_bias('g_bdc2', [n_ch1])&#xA;        h = deconv2d(h, w, s, b, output_shape)        &#xA;        h = bn(h, &quot;g_bnd2&quot;)                &#xA;        h = tf.nn.relu(h)&#xA;        h = h + h1&#xA;&#xA;        output_shape = [batch_size, HEIGHT, WIDTH, CHANNELS]&#xA;        w = init_weights('g_wdc1', [f, f, CHANNELS, n_ch1])&#xA;        b = init_bias('g_bdc1', [CHANNELS])&#xA;        h = deconv2d(h, w, s, b, output_shape)&#xA;&#xA;    return tf.nn.sigmoid(h+x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When you use a stride &lt;code&gt;s&amp;gt;1&lt;/code&gt; you will get an hourglass, where the layers get smaller but deeper. The depth is independently controlled by the &lt;code&gt;n_ch&lt;/code&gt; variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I am running this on a Google colab notebook. Its amazing to have such an engine for free and be able to experiment with deep learning! Fantastic!&lt;/p&gt;&#xA;" OwnerUserId="26202" LastEditorUserId="26202" LastEditDate="2018-07-13T21:21:37.993" LastActivityDate="2019-04-11T23:01:50.347" Title="How to use deep learning to add local (e.g. repairing) transformations to images?" Tags="|machine-learning|neural-network|deep-learning|gan|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27771" PostTypeId="2" ParentId="16620" CreationDate="2018-02-13T19:04:48.437" Score="1" Body="&lt;p&gt;The output dimension of a convolution in deep learning depends on multiple factors &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the size of the filter (aka kernel)&lt;/li&gt;&#xA;&lt;li&gt;the padding (whether you add zeros or not around your image and how many)&lt;/li&gt;&#xA;&lt;li&gt;the numbers of filter that you use&lt;/li&gt;&#xA;&lt;li&gt;the stride&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The simplest dependency is that on the numbers of filters &lt;code&gt;N&lt;/code&gt;. It gives you the numbers of feature maps that your output has. For the input that may be the RGB channels i.e. 3, for the output this number can be chosen freely. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next factor is the zero-padding. If you use a filter size of (3,3) and &quot;valid&quot; padding i.e. adding NO zeros around the image you end up with an output of dimension.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(100, 100, 3) -&amp;gt; (98, 98, N)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because you use a stride of 1. If you move the filter across the image at the end of the picture in each direction the filter will hit the border after 98 steps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you use &quot;SAME&quot; padding you compensate for the filter size -in case of a filter size of (3,3) that would correspond to one line of zeros around the image- you will end up with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(100, 100, 3) -&amp;gt; (100, 100, N)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With a stride of 2 for example you shift the position of the filter by two pixels. Therefore, you get&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(100, 100, 3) -&amp;gt; (50, 50, N)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2018-02-13T19:04:48.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27772" PostTypeId="2" ParentId="17759" CreationDate="2018-02-13T19:34:45.097" Score="18" Body="&lt;p&gt;The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, time values close to each other will be grouped together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Encoding time as categorical gives the model more flexibility, but in some cases, the model may not have enough data to learn well. One technique that may be useful is to group time values together into some number of sets, and use the set as a categorical attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some example groupings:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For month, group into quarters or seasons, depending upon the use case. Eg: Jan-Mar, Apr-Jun, etc.&lt;/li&gt;&#xA;&lt;li&gt;For hour-of-day, group into time-of-day buckets: morning, evening, etc,&lt;/li&gt;&#xA;&lt;li&gt;For day-of-week, group into weekday, weekend.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Each of the above can also be used directly as a categorical attribute as well, given enough data. Further, groupings can also be discovered by data analysis, to complement a domain knowledge based approach.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-02-13T19:34:45.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27773" PostTypeId="2" ParentId="27744" CreationDate="2018-02-13T19:48:54.190" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;Is the target variable likely to be linearly, or additively dependent on the inputs? This means Monday will always have, say 10 more vehicles than Tuesday. If it is more intuitive to say that Monday will have 10% more vehicles than Tuesday, you can consider a log-linear model: transform the target variable by taking a log.&lt;/li&gt;&#xA;&lt;li&gt;In addition to measuring RMSE, you may want to visualize the data and predictions. Plot the actual and predicted vehicles on y-axis and date on x-axis, separately for each junction. This should tell you something about how good your model is, and potentially where it is going wrong.&lt;/li&gt;&#xA;&lt;li&gt;Based on the method of defining the time variables, the features are not likely to correlated to each other.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-02-13T19:48:54.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27774" PostTypeId="1" CreationDate="2018-02-13T20:12:58.267" Score="3" ViewCount="1126" Body="&lt;p&gt;I have a dataset containing 40 input columns and 12 output columns (float values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I supposed it's a regression problem and I am wondering of how to choose the ideal architecture of the Neural Network i didn't find examples of neural Network regression with multiple output &#xA;What is the right way to do it?&lt;/p&gt;&#xA;" OwnerUserId="46219" LastActivityDate="2018-02-13T20:55:41.673" Title="Neural Network for Multiple Float Output" Tags="|neural-network|deep-learning|regression|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27775" PostTypeId="2" ParentId="27774" CreationDate="2018-02-13T20:55:41.673" Score="2" Body="&lt;p&gt;Just use an output layer with 12 neurons instead of 1. Qualitatively there is no difference. For regression the output activations should be linear, and you have a few choices for cost: RMSE, MAE, or Huber Loss to name a few.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-13T20:55:41.673" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27776" PostTypeId="1" AcceptedAnswerId="27778" CreationDate="2018-02-14T02:13:20.363" Score="2" ViewCount="1072" Body="&lt;p&gt;I have a dataset of long/short equity hedge funds returns and their associated benchmarks (market indices). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to form multiple regression on the fund returns using the benchmarks returns as independent variables (i am allowed to form linear combination or manipulation of the indices or even non-linear combinations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, I do not know which independent variables to choose. Are techniques such as subset selection, Lasso, and Ridge supposed to be used in situation like this?&lt;/p&gt;&#xA;" OwnerUserId="46230" LastEditorUserId="40853" LastEditDate="2018-02-14T03:00:00.570" LastActivityDate="2018-02-14T03:07:27.670" Title="How to choose variables for regression" Tags="|machine-learning|regression|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27777" PostTypeId="1" CreationDate="2018-02-14T02:21:47.400" Score="1" ViewCount="695" Body="&lt;pre&gt;&lt;code&gt;   model = Sequential()&#xA;   model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(img_h_w,img_h_w,1)))&#xA;   model.add(Conv2D(32, (3, 3), activation='relu'))&#xA;   model.add(MaxPooling2D(pool_size=(2,2)))&#xA;   model.add(Dropout(0.25))&#xA;   model.add(Flatten())&#xA;   model.add(Dense(128, activation='relu'))&#xA;   model.add(Dropout(0.5))&#xA;   model.add(Dense(1, activation='softmax'))&#xA;   model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])&#xA;   model.fit(x_train, y_train, batch_size=100, epochs=3, verbose=1)&#xA;   score = model.evaluate(x_test, y_test, verbose=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running this returns &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252&#xA;Epoch 2/3&#xA;2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252&#xA;Epoch 3/3&#xA;2500/2500 [==============================] - 6s 2ms/step - loss: 7.5694 - acc: 0.5252&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which shows the accuracy is at 50%, and when I look at the predictions made, I see it only outputs 1. It is meant to classify images into 0 or 1. &lt;/p&gt;&#xA;" OwnerUserId="46231" LastEditorUserId="29169" LastEditDate="2021-02-13T16:41:51.360" LastActivityDate="2021-02-13T16:41:51.360" Title="Why does my Keras Conv Net only return 1?" Tags="|machine-learning|deep-learning|keras|cnn|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27778" PostTypeId="2" ParentId="27776" CreationDate="2018-02-14T03:07:27.670" Score="3" Body="&lt;p&gt;There are a few steps you can take to choose features for linear regression:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Exclude variables that are highly correlated with each other. If variables are highly correlated you are essentially inputting the same information multiple times which can cause over-fitting and does not satisfy the properties no multi-collinearity for linear regression. You can create a Pearson correlation matrix and decided which variables are too highly correlated using some chosen threshold i.e only keep variables with a correlation coefficient of &amp;lt; 0.3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - If you have many variables you could perform principal component analysis (PCA) to reduce the dimensions of the data and use those as your linear regression features. The idea of PCA is reduce dimensions while holding all of the information. Each component from PCA are uncorrelated, satisfying the no multi-collinearity property. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3 - There is also a method known as stepwise linear regression. You allow all variables to enter the model and it will iteratively remove and add variables until the model with the highest R-squared (or whatever your chosen model metric is) is produced. You do have to be cautious using the stepwise  method as it can lead to overfitting, but it can give an indication on what features to use. Here's some info on stepwise: &lt;a href=&quot;https://en.wikipedia.org/wiki/Stepwise_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Stepwise_regression&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 - If you are using R, there is a brilliant package called &quot;caret&quot; that can help with feature selection. Here is a fantastic link to use as a guide: &lt;a href=&quot;https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps out as a starting point&lt;/p&gt;&#xA;" OwnerUserId="41460" LastActivityDate="2018-02-14T03:07:27.670" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27779" PostTypeId="2" ParentId="27770" CreationDate="2018-02-14T04:35:21.337" Score="0" Body="&lt;p&gt;Have you looked into ResNet which modifies image on a pixel level, rather than holistically modifying the image content, preserving the global structure and annotations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also your application sounds a lot like &lt;a href=&quot;https://dmitryulyanov.github.io/deep_image_prior&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep image prior&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33364" LastActivityDate="2018-02-14T04:35:21.337" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27780" PostTypeId="2" ParentId="27770" CreationDate="2018-02-14T05:02:13.077" Score="1" Body="&lt;p&gt;There is a paper called &lt;em&gt;Spatial Transformer Networks&lt;/em&gt; written by &lt;em&gt;Max Jaderberg et al&lt;/em&gt;. What it does is trying to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional network to be spatial invariant. One of the significant achievements of this module is that it tries to enhance distorted inputs. Take a look at &lt;a href=&quot;https://drive.google.com/file/d/0B1nQa_sA3W2iN3RQLXVFRkNXN0k/view&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;I edit the answer because of the request of one of our friends. First I quote something from the popular book written by Pr. &lt;em&gt;Gonzalez&lt;/em&gt;, I hope nothing goes wrong with copyright. Then I suggest my recommendation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Figure 2.40&lt;/em&gt;, the one that I've attached, &lt;em&gt;shows an example of the steps in Fig. 2.39. In this case, the transform used was the Fourier transform, which we mention briefly later in this section and discuss in detail in Chapter 4. Figure 2.40(a) is an image corrupted by sinusoidal interference and Fig. 2.40(b) is the magnitude of its Fourier transform, which is the output of the first stage in Fig. 2.39. As you will learn in Chapter 4, sinusoidal interference in the spatial domain appears as bright bursts of intensity in the transform domain. In this case, the bursts are in a circular pattern that can be seen in Fig. 2.40(b). Figure 2.40(c) shows a mask&#xA;image (called a filter) with white and black representing 1 and 0, respectively.&#xA;For this example, the operation in the second box of Fig. 2.39 is to multiply the mask by the transform, thus eliminating the bursts responsible for the interference. Figure 2.40(d) shows the final result, obtained by computing the inverse of the modified transform. The interference is no longer visible, and important detail is quite clear. In fact, you can even see the fiducial marks (faint crosses) that are used for image alignment.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MA3SZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MA3SZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Okay! After those quotes, I refer to my suggestion. In the &lt;em&gt;ST&lt;/em&gt; network, the authors have claimed that their differentiable module can learn different kinds of transformations other than affine transformation. The point about that is that we usually apply two kinds of transformations. One transformation is applied to the intensity of the image, which is popular by the means of &lt;em&gt;filters&lt;/em&gt;. The other one is called image warping where we change the position of intensities and the intensity values do not change unless they locate between the discrete grids of image entries also called pixels, &lt;em&gt;picture element&lt;/em&gt;. Spatial transformers are good for the second task but they also can be used for the first task. There are studies about using &lt;em&gt;CNNs&lt;/em&gt; for evaluating the images in their frequency domain and not in the spatial domain. You can employ these differentiable modules in those nets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally about your specific task, what I'm seeing in your pictures, your noise has a same behaviour. I guess if that is true for all cases, your task is not learning and can be solved using usual image processing techniques.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-06-12T20:20:19.927" LastActivityDate="2018-06-12T20:20:19.927" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="27781" PostTypeId="1" CreationDate="2018-02-14T06:20:33.537" Score="1" ViewCount="522" Body="&lt;p&gt;I was trying to plot the cluster result of DBSCAN clustering. I cluster data into two cluster and when i write code to plot those it shows 'name error'. But I couldn't understand whats the problem.&#xA;Here's my code with error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(0, reduced_data.shape[0]):&#xA;    if dbscan.labels_[i] == 0:&#xA;        c1 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='r',marker='+')&#xA;    elif dbscan.labels_[i] == 1:&#xA;        c2 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='g',marker='o')&#xA;    elif dbscan.labels_[i] == -1:&#xA;        c3 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='b',marker='*')&#xA;    plt.legend([c1, c2, c3], ['Cluster 1', 'Cluster 2','Noise'])&#xA;    plt.title('DBSCAN finds 2 clusters and noise')&#xA;    plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CCQCs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CCQCs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edited:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rest of my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']&#xA;X = train[feature_cols]&#xA;y = train['label']&#xA;&#xA;# split X and y into training and testing sets&#xA;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)&#xA;&#xA;X_train_scale = scale(X_train)&#xA;X_test_scale = scale(X_test)&#xA;&#xA;reduced_data = PCA(n_components=2).fit_transform(X_train_scale)&#xA;reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)&#xA;&#xA;from pylab import *&#xA;xx, yy = zip(*reduced_data)&#xA;scatter(xx,yy)&#xA;show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9Ljhz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9Ljhz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dbscan = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)&#xA;labels=dbscan.labels_&#xA;print(labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/E5xn1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/E5xn1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)&#xA;print (n_clusters_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result of n_cluster is 2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the main data before PCA reducing is like these:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6wukb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6wukb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="924" LastEditDate="2018-02-14T11:01:46.700" LastActivityDate="2018-02-14T11:01:46.700" Title="Looping problem in python" Tags="|python|clustering|plotting|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27782" PostTypeId="1" CreationDate="2018-02-14T06:28:51.633" Score="3" ViewCount="2853" Body="&lt;p&gt;I wants to create an app which can recognize multiple faces from one image, I planed to built an android app through &lt;code&gt;KIVI&lt;/code&gt;, I am not getting from where to start, it is my first minor project, any help will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="45336" LastEditorUserId="28175" LastEditDate="2018-02-14T06:37:56.287" LastActivityDate="2018-06-14T16:08:02.017" Title="how can I recognize multiple faces from one image in python" Tags="|machine-learning|python|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27783" PostTypeId="2" ParentId="27782" CreationDate="2018-02-14T06:37:16.867" Score="0" Body="&lt;p&gt;Actually there are so many ways for doing so depending on your point of view. You can use deep learning and machine learning approaches or old fashioned vision approaches. For instance, you can use &lt;code&gt;Segnet&lt;/code&gt; neural net if you want to use learning approaches. But because you have limited computation power, vision approaches is more common. Take a look at &lt;a href=&quot;https://www.superdatascience.com/opencv-face-detection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-14T06:37:16.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27784" PostTypeId="1" CreationDate="2018-02-14T06:43:42.860" Score="4" ViewCount="360" Body="&lt;p&gt;I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I would like to use SOM to identify clusters in unlabeled data, I am running it first with some labeled benchmark data sets to make sure that my implementation is bug free. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Iris data set, I can see a clear demarcation between the classes suggesting that the implementation is correct. The classes have been re-labeled as 1,2,3. &lt;a href=&quot;https://i.stack.imgur.com/J3CG2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J3CG2.png&quot; alt=&quot;U-matrix for Iris data set&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I run the same code on another data set, I can see that SOM was able to map members of different classes together (all the 0s are together and 1s are together and there are two distinct regions), but there is no clear demarcation.&lt;a href=&quot;https://i.stack.imgur.com/hd42M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hd42M.png&quot; alt=&quot;U-matrix of the data set in question&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I improve the U-matrix or SOM to produce a clear demarcation between the classes?&lt;/p&gt;&#xA;" OwnerUserId="45603" LastActivityDate="2023-12-18T18:08:41.110" Title="Why is my U-matrix visually not separating the classes?" Tags="|python|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27785" PostTypeId="2" ParentId="27366" CreationDate="2018-02-14T07:02:23.790" Score="2" Body="&lt;p&gt;Consider a data set of 1000 patterns, each with 100 features. It is impossible to plot this data. When this data is run through SOM, the network learns the weights such that each of the neurons summarizes a subset of the vectors. After the training is complete, similar patterns get mapped to neighborhood neurons and farther apart neurons represent dissimilar patterns. Visualizing this mapping tells which patterns lie where, in a 2D space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in the world poverty map&lt;a href=&quot;https://pdfs.semanticscholar.org/4eee/2a5fe88f57578c892d448f93100e25089531.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[1]&lt;/a&gt;&lt;a href=&quot;http://www.cis.hut.fi/research/som-research/worldmap.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;[2]&lt;/a&gt;, each pattern is a vector of length 39 and represents one country. When this data set is run through SOM, the countries(patterns) are mapped to nearest neurons and it can be seen that the countries are arranged together based on their economic wellness.&lt;a href=&quot;https://i.stack.imgur.com/XUN7S.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XUN7S.gif&quot; alt=&quot;World Poverty Map&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After SOM is run until the stopping criterion is met, each pattern/input vector is mapped to the best matching unit. Hence, for example, Belgium (BEL) is closest to the weight vector of the first node in the map (top-right corner) and hence, mapped to it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each node is colored based on the average distance of its weight vector with the weight vectors of its neighbors (U-matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is still not clear how dimensionality reduction is achieved.&lt;/p&gt;&#xA;" OwnerUserId="45603" LastActivityDate="2018-02-14T07:02:23.790" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27786" PostTypeId="2" ParentId="27781" CreationDate="2018-02-14T07:44:50.263" Score="3" Body="&lt;p&gt;In your specific case you only have 2 clusters, however this is not necessarily always going to be the case. I would allow for more flexibility. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume from your sample code that you are following what is shown in the &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt;. Following from what they are doing you should have the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']&#xA;X = train[feature_cols]&#xA;y = train['label']&#xA;&#xA;# split X and y into training and testing sets&#xA;from sklearn.model_selection import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)&#xA;&#xA;X_train_scale = scale(X_train)&#xA;X_test_scale = scale(X_test)&#xA;&#xA;reduced_data = PCA(n_components=2).fit_transform(X_train_scale)&#xA;reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)&#xA;&#xA;from pylab import *&#xA;xx, yy = zip(*reduced_data)&#xA;scatter(xx,yy)&#xA;show()&#xA;&#xA;# Compute DBSCAN&#xA;db = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)&#xA;core_samples_mask = np.zeros_like(db.labels_, dtype=bool)&#xA;core_samples_mask[db.core_sample_indices_] = True&#xA;labels = db.labels_&#xA;&#xA;# Number of clusters in labels, ignoring noise if present.&#xA;n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I just changed your variable name dbscan to db to make it easier to go through the code from the source. From here you can plot all the different clusters that were identified by the DBSCAN method. You should also keep the mask which retains the labels as a list in order for us to access them easily when we plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will identify what are the unique labels which DBSCAN identified and we will map a color to each of these labels. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Plot result&#xA;import matplotlib.pyplot as plt&#xA;&#xA;# Black removed and is used for noise instead.&#xA;unique_labels = set(labels)&#xA;colors = [plt.cm.Spectral(each)&#xA;          for each in np.linspace(0, 1, len(unique_labels))]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For each cluster as determined by the unique labels we will plot all the values associated with it as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for k, col in zip(unique_labels, colors):&#xA;    # -1 is an identifier for noise&#xA;    if k == -1:&#xA;        # Black used for noise.&#xA;        col = [0, 0, 0, 1]&#xA;&#xA;    # Find out what instances belong to this cluster, k&#xA;    class_member_mask = (labels == k)&#xA;&#xA;    # Pull out these instances&#xA;    xy = X[class_member_mask]&#xA;&#xA;    # Plot all of these instances&#xA;    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),&#xA;             markeredgecolor='k', markersize=6)&#xA;&#xA;plt.title('Estimated number of clusters: %d' % n_clusters_)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-14T07:44:50.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27787" PostTypeId="2" ParentId="27715" CreationDate="2018-02-14T08:09:26.043" Score="2" Body="&lt;p&gt;A generative adversarial network is comprised of two parts which are necessary for the training, the generator and the discriminator. They are playing a game against each other, the discriminator tried to learn what is real and what is fake, and the generator tries to create fake instances which will fool the discriminator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the input of the discriminator you will use real images, for example the MNIST numbers. At the input of the generator you will put just random noise vectors. These vectors will then be shaped by the layers of the generator to create some image, hopefully resembling the MNIST dataset. If it fools the discriminator, it wins that round. Otherwise, it will tune its weights using backpropagation in order to produce better quality results in the next round. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once training is complete, you will detach the discriminator and use only the generator. Then for every single random noise vector fed to the input of the generator you will receive an associated image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a GaN implementation. You will need to put a path &quot;./gan/images/&quot; i the directory where you are running this GaN.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import time&#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;&#xA;from keras.layers import Input&#xA;from keras.models import Model&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten&#xA;from keras.layers import Conv2D, MaxPooling2D&#xA;from keras.layers import Dense, Activation, Flatten, Reshape&#xA;from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D&#xA;from keras.layers import LeakyReLU, Dropout&#xA;from keras.layers import BatchNormalization&#xA;from keras.optimizers import Adam, RMSprop&#xA;from keras.datasets import mnist&#xA;&#xA;import matplotlib.pyplot as plt&#xA;&#xA;class GAN():&#xA;    def __init__(self):&#xA;        self.img_rows = 28 &#xA;        self.img_cols = 28&#xA;        self.channels = 1&#xA;        self.img_shape = (self.img_rows, self.img_cols, self.channels)&#xA;&#xA;        optimizer = Adam(0.0002, 0.5)&#xA;&#xA;        # Build and compile the discriminator&#xA;        self.discriminator = self.build_discriminator()&#xA;        self.discriminator.compile(loss='binary_crossentropy', &#xA;            optimizer=optimizer,&#xA;            metrics=['accuracy'])&#xA;&#xA;        # Build and compile the generator&#xA;        self.generator = self.build_generator()&#xA;        self.generator.compile(loss='binary_crossentropy', optimizer=optimizer)&#xA;&#xA;        # The generator takes noise as input and generated imgs&#xA;        z = Input(shape=(28,28,))&#xA;        img = self.generator(z)&#xA;&#xA;        # For the combined model we will only train the generator&#xA;        self.discriminator.trainable = False&#xA;&#xA;        # The valid takes generated images as input and determines validity&#xA;        valid = self.discriminator(img)&#xA;&#xA;        # The combined model  (stacked generator and discriminator) takes&#xA;        # noise as input =&amp;gt; generates images =&amp;gt; determines validity &#xA;        self.combined = Model(z, valid)&#xA;        self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)&#xA;&#xA;    def build_generator(self):&#xA;&#xA;        noise_shape = (28,28,)&#xA;&#xA;        model = Sequential()&#xA;&#xA;        model.add(Reshape((28*28,), input_shape=noise_shape))&#xA;        model.add(Dense(256))&#xA;        model.add(LeakyReLU(alpha=0.2))&#xA;        model.add(BatchNormalization(momentum=0.8))&#xA;        model.add(Dense(512))&#xA;        model.add(LeakyReLU(alpha=0.2))&#xA;        model.add(BatchNormalization(momentum=0.8))&#xA;        model.add(Dense(1024))&#xA;        model.add(LeakyReLU(alpha=0.2))&#xA;        model.add(BatchNormalization(momentum=0.8))&#xA;        model.add(Dense(np.prod(self.img_shape), activation='tanh'))&#xA;        model.add(Reshape(self.img_shape))   &#xA;&#xA;        model.summary()&#xA;&#xA;        noise = Input(shape=noise_shape)&#xA;        img = model(noise)&#xA;&#xA;        return Model(noise, img)&#xA;&#xA;    def build_discriminator(self):&#xA;&#xA;        img_shape = (self.img_rows, self.img_cols, self.channels)&#xA;&#xA;        model = Sequential()&#xA;&#xA;        model.add(Conv2D(32, kernel_size=(3, 3),&#xA;                 input_shape=img_shape))&#xA;        model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;        model.add(LeakyReLU(alpha=0.2))&#xA;        model.add(Dropout(0.25))&#xA;&#xA;        model.add(Flatten())&#xA;        model.add(Dense(64))&#xA;        model.add(LeakyReLU(alpha=0.2))&#xA;        model.add(Dense(1, activation='sigmoid'))&#xA;        model.summary()&#xA;&#xA;        img = Input(shape=img_shape)&#xA;        validity = model(img)&#xA;&#xA;        return Model(img, validity)&#xA;&#xA;    def train(self, epochs, batch_size=128, save_interval=50):&#xA;&#xA;        # Load the dataset&#xA;        (X_train, _), (_, _) = mnist.load_data()&#xA;&#xA;        # Rescale -1 to 1&#xA;        X_train = (X_train.astype(np.float32) - 127.5) / 127.5&#xA;        X_train = np.expand_dims(X_train, axis=3)&#xA;&#xA;        half_batch = int(batch_size / 2)&#xA;&#xA;        for epoch in range(epochs):&#xA;&#xA;            # ---------------------&#xA;            #  Train Discriminator&#xA;            # ---------------------&#xA;&#xA;            # Select a random half batch of images&#xA;            idx = np.random.randint(0, X_train.shape[0], half_batch)&#xA;            imgs = X_train[idx]&#xA;&#xA;            noise = np.random.normal(0, 1, (half_batch,28,28,))&#xA;&#xA;            # Generate a half batch of new images&#xA;            gen_imgs = self.generator.predict(noise)&#xA;&#xA;            # Train the discriminator&#xA;            d_loss_real = self.discriminator.train_on_batch(imgs, np.ones((half_batch, 1)))&#xA;            d_loss_fake = self.discriminator.train_on_batch(gen_imgs, np.zeros((half_batch, 1)))&#xA;            d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)&#xA;&#xA;&#xA;            # ---------------------&#xA;            #  Train Generator&#xA;            # ---------------------&#xA;&#xA;            noise = np.random.normal(0, 1, (batch_size,28,28,))&#xA;&#xA;            # The generator wants the discriminator to label the generated samples&#xA;            # as valid (ones)&#xA;            valid_y = np.array([1] * batch_size)&#xA;&#xA;            # Train the generator&#xA;            g_loss = self.combined.train_on_batch(noise, valid_y)&#xA;&#xA;            # Plot the progress&#xA;            print (&quot;%d [D loss: %f, acc.: %.2f%%] [G loss: %f]&quot; % (epoch, d_loss[0], 100*d_loss[1], g_loss))&#xA;&#xA;            # If at save interval =&amp;gt; save generated image samples&#xA;            if epoch % save_interval == 0:&#xA;                self.save_imgs(epoch)&#xA;&#xA;    def save_imgs(self, epoch):&#xA;        r, c = 5, 5&#xA;        noise = np.random.normal(0, 1, (r * c, 28,28,))&#xA;        gen_imgs = self.generator.predict(noise)&#xA;&#xA;        # Rescale images 0 - 1&#xA;        gen_imgs = 0.5 * gen_imgs + 0.5&#xA;&#xA;        fig, axs = plt.subplots(r, c)&#xA;        cnt = 0&#xA;        for i in range(r):&#xA;            for j in range(c):&#xA;                axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')&#xA;                axs[i,j].axis('off')&#xA;                cnt += 1&#xA;        fig.savefig(&quot;gan/images/mnist_%d.png&quot; % epoch)&#xA;        plt.close()&#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    gan = GAN()&#xA;    gan.train(epochs=30000, batch_size=64, save_interval=200)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-02T02:44:54.050" LastActivityDate="2018-03-02T02:44:54.050" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27788" PostTypeId="2" ParentId="27604" CreationDate="2018-02-14T09:58:30.213" Score="0" Body="&lt;p&gt;The problem was that I tried to plot the attention map of a model which was loaded from a saved model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output when the model was saved was:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;/home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [, ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [, ]}. They will not be included in the&#xA;  serialized model (and thus will be missing at deserialization time).&lt;br&gt;&#xA;  str(node.arguments) + '. They will not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&#xA;  /home/opyate/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py:2361:&#xA;  UserWarning: Layer lstm_1 was passed non-serializable keyword&#xA;  arguments: {'initial_state': [,&#xA;  ]}.&#xA;  They will not be included in the serialized model (and thus will be&#xA;  missing at deserialization time).   str(node.arguments) + '. They will&#xA;  not be included '&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, if I construct the model from code, and just load the saved weights, it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The assumption is that the &lt;code&gt;UserWarning&lt;/code&gt;s when saving the model has something to do with my problem.&lt;/p&gt;&#xA;" OwnerUserId="45945" LastActivityDate="2018-02-14T09:58:30.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27789" PostTypeId="2" ParentId="27767" CreationDate="2018-02-14T10:16:12.650" Score="41" Body="&lt;p&gt;If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The &lt;code&gt;pandas.read_csv&lt;/code&gt; method allows you to read a file in chunks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;for chunk in pd.read_csv(&amp;lt;filepath&amp;gt;, chunksize=&amp;lt;your_chunksize_here&amp;gt;)&#xA;    do_processing()&#xA;    train_algorithm()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is the method's &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10076" LastEditorUserId="14372" LastEditDate="2018-02-15T01:31:24.810" LastActivityDate="2018-02-15T01:31:24.810" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27790" PostTypeId="2" ParentId="27328" CreationDate="2018-02-14T10:25:27.837" Score="1" Body="&lt;p&gt;Hope this helps&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After doing a bit of research into how &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/voc2007/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The PASCAL Visual Object Classes Challenge 2007&lt;/a&gt; image data are stored. This is what I could find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They follow an annotation of each image which creates an xml file of that particular image which contains these details &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/voc2007/guidelines.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PASCAL Visual Object Classes Challenge 2007 (VOC2007) Annotation Guidelines&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of xml file from VOC2007 of one of these images&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;IMAGE&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SSpGc.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SSpGc.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;XML&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;annotation&amp;gt;&#xA;&amp;lt;folder&amp;gt;VOC2007&amp;lt;/folder&amp;gt;&#xA;&amp;lt;filename&amp;gt;000001.jpg&amp;lt;/filename&amp;gt;&#xA;&amp;lt;source&amp;gt;&#xA;    &amp;lt;database&amp;gt;The VOC2007 Database&amp;lt;/database&amp;gt;&#xA;    &amp;lt;annotation&amp;gt;PASCAL VOC2007&amp;lt;/annotation&amp;gt;&#xA;    &amp;lt;image&amp;gt;flickr&amp;lt;/image&amp;gt;&#xA;    &amp;lt;flickrid&amp;gt;341012865&amp;lt;/flickrid&amp;gt;&#xA;&amp;lt;/source&amp;gt;&#xA;&amp;lt;owner&amp;gt;&#xA;    &amp;lt;flickrid&amp;gt;Fried Camels&amp;lt;/flickrid&amp;gt;&#xA;    &amp;lt;name&amp;gt;Jinky the Fruit Bat&amp;lt;/name&amp;gt;&#xA;&amp;lt;/owner&amp;gt;&#xA;&amp;lt;size&amp;gt;&#xA;    &amp;lt;width&amp;gt;353&amp;lt;/width&amp;gt;&#xA;    &amp;lt;height&amp;gt;500&amp;lt;/height&amp;gt;&#xA;    &amp;lt;depth&amp;gt;3&amp;lt;/depth&amp;gt;&#xA;&amp;lt;/size&amp;gt;&#xA;&amp;lt;segmented&amp;gt;0&amp;lt;/segmented&amp;gt;&#xA;&amp;lt;object&amp;gt;&#xA;    &amp;lt;name&amp;gt;dog&amp;lt;/name&amp;gt;&#xA;    &amp;lt;pose&amp;gt;Left&amp;lt;/pose&amp;gt;&#xA;    &amp;lt;truncated&amp;gt;1&amp;lt;/truncated&amp;gt;&#xA;    &amp;lt;difficult&amp;gt;0&amp;lt;/difficult&amp;gt;&#xA;    &amp;lt;bndbox&amp;gt;&#xA;        &amp;lt;xmin&amp;gt;48&amp;lt;/xmin&amp;gt;&#xA;        &amp;lt;ymin&amp;gt;240&amp;lt;/ymin&amp;gt;&#xA;        &amp;lt;xmax&amp;gt;195&amp;lt;/xmax&amp;gt;&#xA;        &amp;lt;ymax&amp;gt;371&amp;lt;/ymax&amp;gt;&#xA;    &amp;lt;/bndbox&amp;gt;&#xA;&amp;lt;/object&amp;gt;&#xA;&amp;lt;object&amp;gt;&#xA;    &amp;lt;name&amp;gt;person&amp;lt;/name&amp;gt;&#xA;    &amp;lt;pose&amp;gt;Left&amp;lt;/pose&amp;gt;&#xA;    &amp;lt;truncated&amp;gt;1&amp;lt;/truncated&amp;gt;&#xA;    &amp;lt;difficult&amp;gt;0&amp;lt;/difficult&amp;gt;&#xA;    &amp;lt;bndbox&amp;gt;&#xA;        &amp;lt;xmin&amp;gt;8&amp;lt;/xmin&amp;gt;&#xA;        &amp;lt;ymin&amp;gt;12&amp;lt;/ymin&amp;gt;&#xA;        &amp;lt;xmax&amp;gt;352&amp;lt;/xmax&amp;gt;&#xA;        &amp;lt;ymax&amp;gt;498&amp;lt;/ymax&amp;gt;&#xA;    &amp;lt;/bndbox&amp;gt;&#xA;&amp;lt;/object&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a link for annotation tool &lt;a href=&quot;https://github.com/tzutalin/labelImg&quot; rel=&quot;nofollow noreferrer&quot;&gt;labelImg&lt;/a&gt; that can help you generate the similar &lt;code&gt;xml&lt;/code&gt; file for an image in your own dataset &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details on how the image data are divided into &lt;code&gt;train&lt;/code&gt;,&lt;code&gt;trainval&lt;/code&gt;,&lt;code&gt;test&lt;/code&gt; and &lt;code&gt;val&lt;/code&gt; check this &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/voc2007/htmldoc/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Development Kit&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45545" LastEditorUserId="45545" LastEditDate="2018-02-14T11:49:06.097" LastActivityDate="2018-02-14T11:49:06.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27791" PostTypeId="2" ParentId="27602" CreationDate="2018-02-14T11:13:42.690" Score="1" Body="&lt;p&gt;Take, for example, an imaginary database of a &quot;school&quot; with three tables:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Students (StudentID, Name)&lt;/li&gt;&#xA;&lt;li&gt;Courses (CourseID, Name)&lt;/li&gt;&#xA;&lt;li&gt;Enrollment (StudentID, CourseID)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In this context you could explain the joins as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Inner join&lt;/em&gt; would be a natural operation for answering queries like &quot;what courses is this student enrolled in&quot; or &quot;what students are enrolled in this course&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if a student is not enrolled in any courses, the query above would simply return no rows, whilst you might still want to have &lt;em&gt;one&lt;/em&gt; row with the name of the student in the output and a NULL/NA value as its only &quot;course&quot;. In this case you could prefer a &lt;em&gt;left join&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The need for left/right/outer joins is even more common if you need to do some aggregations later on. Suppose you need to output a table, listing all courses alongside with the count of students enrolled in them. Using an inner join would only list the courses with at least one student. Using a left join would list all of them, including those with 0 students.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right joins are somewhat esoteric and rarely used (I've written tons of SQL but never used a right join so far I think). Indeed, you can always use a left join in place of a right join, and most of the time people tend to think about data access in a &quot;left to right&quot; manner. The three possible situations for using a right join that I see are: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you somewhy tend to &quot;think&quot; in a right-to-left manner. Suppose you want to list courses along with the number of students enrolled. The &quot;left to right&quot; thinking goes as follows: first select all Courses, then left-join all Enrollments, then aggregate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also think differently. The &quot;important&quot; data source you want to start with is the Enrollments table, after all, so why not start by reading data from it, and perhaps even aggregating it first. &lt;em&gt;Then&lt;/em&gt; you could join the Courses table to get the course names in place of IDs. In this situation you would be right-joining the Courses.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you suspect a right join would be more efficient. Suppose that, in the example above, there are thousands of courses, only a couple of them are non-empty, with hundreds of students. In this case, a &quot;straightforward&quot; manner to execute a &quot;Courses left join Enrollments&quot; query would be to scan the Courses table, trying to match up each course by scanning the Enrollments table (assume the query optimizer is dumb and there are no indexes on the tables, for the sake of example -- is there a query optimzer in dplyr, after all?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only two courses would have matches but the scan would have to proceed for a thousand times. The resulting table would have the two nonempty course names repeated hundreds of times for each enrolled student alongside thousands of course names with NULL students, after which you'd proceed to the aggregation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the contrary, if we first aggregated Enrollments and then right-joined the Course names in the &quot;right-to-left&quot; manner, the aggregated Enrollments table would only have two rows, and thus perform just two ID matches over the Courses table (writing out the remaining course names as-is).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fortunately, most SQL database engines are smart enough nowadays to deal with such optimizations internally without the need for the user to hand-tune the direction and order of joins in the query.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, sometimes you &lt;em&gt;generate SQL programmatically&lt;/em&gt;. In this case again, the need to &quot;join in&quot; the Course or Student names to some previously computed main result table could be quite common, and a right join perfectly appropriate.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;That was a long digression on &lt;em&gt;right situations for right joins&lt;/em&gt;, let's get back to the &lt;em&gt;outer join&lt;/em&gt;. It is also rarely used, in my experience. They say outer joins are harder for SQL engines to optimize (although I can't at the moment exactly see why so). Anyway, in our example one could use an outer join to create a table of the form (Course, List of Students), with two extra conditions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All the empty courses would be included, showing their student list as a single NULL entry, &lt;/li&gt;&#xA;&lt;li&gt;There would be a special NULL course, listing all students who are not enrolled anywhere at all. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That could be a perfectly useful table, right?&lt;/p&gt;&#xA;" OwnerUserId="14519" LastEditorUserId="14519" LastEditDate="2018-02-14T11:33:38.527" LastActivityDate="2018-02-14T11:33:38.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27793" PostTypeId="2" ParentId="24740" CreationDate="2018-02-14T11:48:18.457" Score="0" Body="&lt;p&gt;To measure what features are the &quot;drivers of difference between groups&quot;, you're going to need to frame this as a classification problem. Within this framework, you can use variable importance (and potentially coefficient values, depending on the model) to perform inference on drivers, i.e. to identify and rank them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying PCA as a pre-processing step will make it significantly harder to identify what variables are drivers for specific classes. PCA is agnostic to class assignment: you can use PCA eigenvector components and loadings to interpret what variables are responsible for the bulk of the variance in your data, but that isn't actually what you want. Imagine if your classes were long elipsoidal clusters, each with the exact same covariance matrix but a different center (i.e. the elipses are &quot;parellel&quot;): the components given by PCA would be mainly influenced by the elipsoid shape (i.e. the shared covariance matrix) rather than by differences between groups. If you're interested in understanding the drivers of group differences, I strongly recommend you drop the PCA step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I still don't have a good handle on exactly what you're hoping to get out of &quot;which groups are most similar&quot;, but I suspect the &quot;proximity&quot; measure given by random forests would satisfy your need here. This measure is actually between observations, but you can take averages to get the expected proximity between groups. A benefit of using random forests here is that they have built-in variable importance measures, and you can even introspect them to understand the drivers behind &lt;a href=&quot;https://github.com/andosa/treeinterpreter&quot; rel=&quot;nofollow noreferrer&quot;&gt;individual observations&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a little demo showing how to use a random forest model to detect drivers via variable importance, and measure group similarity via average inter-observation proximity:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, set up the data and fit a model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(randomForest)&#xA;data(iris)&#xA;set.seed(123)&#xA;&#xA;unq_classes &amp;lt;- unique(iris$Species)&#xA;n_classes   &amp;lt;- length(unq_classes)&#xA;&#xA;iris.rf &amp;lt;- randomForest(Species ~ ., data=iris, &#xA;                        importance=TRUE,&#xA;                        proximity=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Variable importances, rescaled to [0,1] with &quot;1&quot; indicating the most important variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var_imp &amp;lt;- importance(iris.rf)[,1:(n_classes+1)]&#xA;var_imp &amp;lt;- apply(var_imp, 2, function(m) m/max(m))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's the result (that last column is the marginalized importance):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                    setosa versicolor virginica MeanDecreaseAccuracy&#xA;Sepal.Length 0.2645863  0.2403256 0.2503047            0.3336813&#xA;Sepal.Width  0.1927240  0.0314708 0.1716495            0.1564093&#xA;Petal.Length 0.9525359  0.9589636 0.9356667            0.9549433&#xA;Petal.Width  1.0000000  1.0000000 1.0000000            1.0000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And our mean proximities, again rescaled to [0,1] with 1 indicating the most similar pair of groups:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prx &amp;lt;- iris.rf$proximity&#xA;mean_prx &amp;lt;- matrix(NA, n_classes, n_classes)&#xA;for (i in 1:(n_classes-1)){&#xA;  for (j in (i+1):n_classes){&#xA;    cls_i &amp;lt;- iris$Species == unq_classes[i]&#xA;    cls_j &amp;lt;- iris$Species == unq_classes[j]&#xA;    mean_prx[j,i] &amp;lt;- mean(prx[cls_i, cls_j])&#xA;  }&#xA;}&#xA;&#xA;mean_prx &amp;lt;- mean_prx/max(mean_prx, na.rm=TRUE)&#xA;rownames(mean_prx) &amp;lt;- unq_classes&#xA;colnames(mean_prx) &amp;lt;- unq_classes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Giving us:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                 setosa versicolor virginica&#xA;setosa               NA         NA        NA&#xA;versicolor 0.0267520374         NA        NA&#xA;virginica  0.0007778552          1        NA&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's what the data looks like to put these results in context:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eWvFu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eWvFu.png&quot; alt=&quot;Iris Data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-02-14T11:59:34.040" LastActivityDate="2018-02-14T11:59:34.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27794" PostTypeId="2" ParentId="27767" CreationDate="2018-02-14T12:03:08.470" Score="29" Body="&lt;p&gt;There are two possibilities: either you &lt;em&gt;need&lt;/em&gt; to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e.g. your algorithm only needs samples of rows or columns at once).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first case, you'll need to &lt;em&gt;solve a memory problem&lt;/em&gt;. Increase your memory size, rent a high-memory cloud machine, use inplace operations, provide information about the type of data you are reading in, make sure to delete all unused variables and collect garbage, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is very probable that 32GB of RAM would not be enough for Pandas to handle your data. Note that the integer &quot;1&quot; is just one byte when stored as text but 8 bytes when represented as &lt;code&gt;int64&lt;/code&gt; (which is the default when Pandas reads it in from text). You can make the same example with a floating point number &quot;1.0&quot; which expands from a 3-byte string to an 8-byte &lt;code&gt;float64&lt;/code&gt; by default. You may win some space by letting Pandas know precisely which types to use for each column and forcing the smallest possible representations, but we did not even start speaking of Python's data structure overhead here, which may add an extra pointer or two here or there easily, and pointers are 8 bytes each on a 64-bit machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To summarize: no, 32GB RAM is probably not enough for Pandas to handle a 20GB file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second case (which is more realistic and probably applies to you), you need to solve a &lt;em&gt;data management problem&lt;/em&gt;. Indeed, having to load all of the data when you really only need parts of it for processing, may be a sign of bad data management. There are multiple options here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Use an SQL database. If you can, it is nearly always the first choice and a decently comfortable solution. 20GB sounds like the size most SQL databases would handle well without the need to go distributed even on a (higher-end) laptop. You'll be able to index columns, do basic aggregations via SQL, and get the needed subsamples into Pandas for more complex processing using a simple &lt;code&gt;pd.read_sql&lt;/code&gt;. Moving the data to a database will also provide you with an opportunity to think about the &lt;em&gt;actual&lt;/em&gt; data types and sizes of your columns.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If your data is mostly numeric (i.e. arrays or tensors), you may consider holding it in a HDF5 format (see &lt;a href=&quot;http://www.pytables.org/&quot; rel=&quot;noreferrer&quot;&gt;PyTables&lt;/a&gt;), which lets you conveniently read only the necessary slices of huge arrays from disk. Basic &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html&quot; rel=&quot;noreferrer&quot;&gt;numpy.save and numpy.load&lt;/a&gt; achieve the same effect via memory-mapping the arrays on disk as well. For GIS and related raster data there are &lt;a href=&quot;http://www.rasdaman.org/&quot; rel=&quot;noreferrer&quot;&gt;dedicated&lt;/a&gt; &lt;a href=&quot;https://www.paradigm4.com/try_scidb/&quot; rel=&quot;noreferrer&quot;&gt;databases&lt;/a&gt;, which might not connect to pandas as directly as SQL, but should also let you do slices and queries reasonably conveniently.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Pandas does not support such &quot;partial&quot; memory-mapping of HDF5 or numpy arrays, as far as I know. If you still want a kind of a &quot;pure-pandas&quot; solution, you can try to work around by &quot;sharding&quot;: either storing the &lt;em&gt;columns&lt;/em&gt; of your huge table separately (e.g. in separate files or in separate &quot;tables&quot; of a single HDF5 file) and only loading the necessary ones on-demand, or storing the &lt;em&gt;chunks of rows&lt;/em&gt; separately. However, you'd then need to implement the logic for loading the necessary chunks, thus reinventing the bicycles already imlpemented in most SQL databases, so perhaps option 1 would still be easier here.  If your data comes in a CSV, though, you can process it in chunks by specifying the &lt;code&gt;chunksize&lt;/code&gt; parameter to &lt;code&gt;pd.read_csv&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="14519" LastEditorUserId="14519" LastEditDate="2018-10-24T14:34:28.990" LastActivityDate="2018-10-24T14:34:28.990" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27795" PostTypeId="1" CreationDate="2018-02-14T12:33:18.393" Score="12" ViewCount="10309" Body="&lt;p&gt;I am working with Python, scikit-learn and keras. I have 3000 thousands images of front-faced watches like the following ones:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oHe6I.jpg&quot; rel=&quot;noreferrer&quot;&gt;Watch_1&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/5L4b0.jpg&quot; rel=&quot;noreferrer&quot;&gt;Watch_2&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/bYvKl.jpg&quot; rel=&quot;noreferrer&quot;&gt;Watch_3&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to write a program which receives as an input a photo of a real watch which maybe taken under less ideal conditions than the photos above (different background colour, darker lightning etc) and find the most similar watches among the 3000 ones to it. By similarity I mean that if I give as an input a photo of a round, brown watch with thin lace then I expect as an output watches of round shape, of dark colour and with thin lace.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the most efficient machine learning algorithm to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, by following &lt;a href=&quot;https://www.quora.com/Whats-the-state-of-the-art-in-image-similarity-with-deep-learning&quot; rel=&quot;noreferrer&quot;&gt;this link&lt;/a&gt; I have two different solutions in my mind:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Using a CNN as a feature extractor and compare the distances between the these features for every pair of images with reference to the input image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Using two CNNs in a Siamese Neural Network to compare the images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are these two options the best ones for this task or would you suggest something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know any pre-trained neural network (with pre-determined hyperparameters) for this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have found some interesting posts on StackOverflow about this but they are pretty old: &lt;a href=&quot;https://stackoverflow.com/questions/4196453/simple-and-fast-method-to-compare-images-for-similarity?noredirect=1&amp;amp;lq=1&quot;&gt;Post_1&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/843972/image-comparison-fast-algorithm/844113#844113&quot;&gt;Post_2&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/23931/algorithm-to-compare-two-images/3445087#3445087&quot;&gt;Post_3&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45610" LastEditorUserId="45610" LastEditDate="2018-02-23T11:22:33.550" LastActivityDate="2021-06-09T11:22:04.113" Title="Neural networks - Find most similar images" Tags="|python|neural-network|keras|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27796" PostTypeId="1" CreationDate="2018-02-14T12:41:58.643" Score="2" ViewCount="1529" Body="&lt;p&gt;I'm using Keras 2.1.3 and I'm trying to fine tune a Inception Resnetv2 with Keras application.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I load the pretrained model from keras.applications&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_tensor = Input(shape=(299,299,3))&#xA;model = applications.inception_resnet_v2.InceptionResNetV2(weights='imagenet', &#xA;                                                        include_top=False,&#xA;                                                        input_tensor=input_tensor,&#xA;                                                        input_shape=(299, 299,3))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I create the bottleneck for my problem :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;top_model = Sequential()&#xA;top_model.add(Flatten(input_shape=model.output_shape[1:]))&#xA;top_model.add(Dense(256, activation='relu'))&#xA;top_model.add(Dropout(0.5))&#xA;top_model.add(Dense(40, activation='softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And finally create a new model to concatenate the two parts : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_model = Sequential()&#xA;for l in model.layers:&#xA;      new_model.add(l)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At this step, I got an error &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Input 0 is incompatible with layer conv2d_7: expected axis -1 of input shape to have value 192 but got shape (None, 35, 35, 64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So I printed each layer shape and I have &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Layer n-1 : Input : (None, 35, 35, 64), Output : (None, 35, 35, 64)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Layer n : Input : (None, 35, 35, 192), Output : (None, 35, 35, 48)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you can see shapes dismatch and it seems weird that come from Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My exact error is &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Input 0 is incompatible with layer conv2d_7: expected axis -1 of input shape to have value 192 but got shape (None, 35, 35, 64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;model.summary() (model is model from Keras.applications.inception_resnet_v2.InceptionResNetV2) give me for the &lt;strong&gt;conv2d_7 layer&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;hr&gt;&#xA;  &#xA;  &lt;p&gt;max_pooling2d_2 (MaxPooling2D)  (None, 35, 35, 192)  0           activation_5[0][0]               &lt;/p&gt;&#xA;  &#xA;  &lt;hr&gt;&#xA;  &#xA;  &lt;p&gt;activation_9 (Activation)       (None, 35, 35, 64)   0           batch_normalization_9[0][0]      &lt;/p&gt;&#xA;  &#xA;  &lt;hr&gt;&#xA;  &#xA;  &lt;p&gt;conv2d_7 (Conv2D)               (None, 35, 35, 48)   9216        max_pooling2d_2[0][0]            &lt;/p&gt;&#xA;  &#xA;  &lt;hr&gt;&#xA;  &#xA;  &lt;p&gt;conv2d_10 (Conv2D)              (None, 35, 35, 96)   55296       activation_9[0][0]               &lt;/p&gt;&#xA;  &#xA;  &lt;hr&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46143" LastEditorUserId="46143" LastEditDate="2018-02-14T13:30:32.137" LastActivityDate="2018-05-11T03:11:34.397" Title="Keras shape error in applications Inception Resnet v2" Tags="|python|deep-learning|keras|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27800" PostTypeId="2" ParentId="27795" CreationDate="2018-02-14T13:27:55.593" Score="4" Body="&lt;p&gt;I don't think that a high level architecture as such is the best fit but it rather depends on many factors and details. From what I am aware of the first approach is promising especially when extended by additional steps as done in the &lt;a href=&quot;https://github.com/paucarre/tiefvision&quot; rel=&quot;nofollow noreferrer&quot;&gt;TiefVision&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;An additional bounding box network is used to discriminate the relevant part of the image from the rest&lt;/li&gt;&#xA;&lt;li&gt;The feature vectors are not simply compared directly but rather used to train a comparison network using triplets (learning similarity based on feature vectors and examples for more and less similar instances).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This work is a lot more recent (2016/17) than what you posted and comes with a nice tool set and a more detailed &lt;a href=&quot;http://users.eecs.northwestern.edu/~jwa368/pdfs/deep_ranking.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why using triplets aka Deep Ranking?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As stated in the comments: Why should one use triplets for image similarity instead of learning feature vectors and calculate their distance? Triplets are a way of formulating the similarity question as a learning problem instead of learning feature vectors that basically do not care for similarity. This approach makes especially sense in cases where human-perceived similarity is important, which might differ from machine perception.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Triplets work like this: You provide 3 images. One to compare to, one similar (close) and one not so similar (distant) image. This is your training/test/validation data. Training your network on those samples and predicting the correct order (classify similar from non-similar images) overall lets the network learn how to order images based on their similarity. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all this approach is comparatively complex. It might be overengineered but you also asked for the best way to do this and Deep Ranking achieves very high precision values.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-02-14T16:24:01.327" LastActivityDate="2018-02-14T16:24:01.327" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27802" PostTypeId="2" ParentId="27795" CreationDate="2018-02-14T13:59:19.050" Score="2" Body="&lt;p&gt;I would focus on data augmentation first. Since your images have a white background you have it a little bit easier. Turn the white background into a transparent background, scale down the image, rotate it and put it in backgrounds similar to your target data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do this a bunch of times with different combination and have a label for each watch. Then I would suggest you use a regular convolutional neural network for the classification. Each label will have a score, pick the one with the highest confidence and that one should be the most &lt;strong&gt;similar&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example lets say you run the classifier with an image and get this result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Watch1: 0.51&#xA;&#xA;Watch2: 0.30&#xA;&#xA;Watch3: 0.25&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The CNN is saying that it has a 51% confidence that Watch1 is the watch in the input image. But also what is true is that it is the one it thinks looks more similar, Watch2 would be the next one more similar and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't get good results, do the usual. Experiment with the parameters and/or add more layers. Try to find out where it is failing. After you have that insight, you can use it to choose a more specialized type of convolutional network for your particular problem. Looking for that without prior knowledge of how it would perform is not the right approach. I would suggest you start with a basic convolutional model and then work from there.&lt;/p&gt;&#xA;" OwnerUserId="42417" LastEditorUserId="42417" LastEditDate="2018-02-14T20:09:16.813" LastActivityDate="2018-02-14T20:09:16.813" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27804" PostTypeId="2" ParentId="27628" CreationDate="2018-02-14T15:09:02.193" Score="13" Body="&lt;p&gt;Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, &lt;a href=&quot;https://machinelearningmastery.com/time-series-forecasting-supervised-learning/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, for framing forecasting as a supervised learning problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of parameters in an LSTM (in fact, there are $4(mn + n^2 + n)$ parameters, where $m$ is the input length and $n$ is the output length, according to &lt;a href=&quot;https://datascience.stackexchange.com/questions/10615/number-of-parameters-in-an-lstm-model&quot;&gt;this post&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since LSTMs do not require fixed size input, they can find the optimal lookback number on their own. However, if you've done a prior autoregressive analysis and decided that, for example, the current time step is most correlated with the 10th previous time step, and not correlated with the 11th or any time steps further in the past, then you could perhaps save yourself some training time by feeding in fixed-length sequences. However, that kind of defeats the purpose of an LSTM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data are not rich enough for an LSTM, I would recommend trying something much simpler, like an autoregressive model, and working your way up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT (responding to a comment):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overlapping sequences are used as input, especially when the sequence is very long (although, of course, 'long' is relative). Although LSTMs are better than a vanilla RNN for long sequences, they can still have some trouble remembering time steps from the beginning of a sequence if the sequence is very long. That led to things like the bidirectional LSTM, which reads the sequence forwards and backwards, improving the exposure of the network to the beginning and end of each input sequence. The principle is the same with overlapping sequences, although I would argue that overlapping sequences is more intuitive.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastEditorUserId="31434" LastEditDate="2018-02-14T18:55:02.217" LastActivityDate="2018-02-14T18:55:02.217" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27805" PostTypeId="1" CreationDate="2018-02-14T15:28:55.437" Score="1" ViewCount="700" Body="&lt;p&gt;Below is an Excel sheet with crops and countries. &quot;1&quot; means that the crop is grown in that country whiles &quot;0&quot; means it is not grown in that country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find where there is an intersection (as in I want to find similar crops that are grown in most of the countries or at least most of the countries.) How do I do this ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EzVP0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EzVP0.png&quot; alt=&quot;Snapshot of spreadsheet with crops and the countries they are grown in&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46259" LastActivityDate="2018-07-14T20:19:00.237" Title="Find intersection of data between rows and columns" Tags="|data-mining|dataset|visualization|pandas|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27806" PostTypeId="1" CreationDate="2018-02-14T15:34:30.523" Score="4" ViewCount="4437" Body="&lt;p&gt;After reading about the most famous object detection CNN based methods: YOLO, YOLO 9000, r-cnn, faster r-cnn, etc., I was wondering if there is an architecture that can calculate the distance to the obstacle after detection?&lt;/p&gt;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T16:39:23.213" LastActivityDate="2021-02-13T16:39:23.213" Title="A deep learning approach that can calculate distance to obstacles" Tags="|deep-learning|neural-network|cnn|computer-vision|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27807" PostTypeId="1" CreationDate="2018-02-14T15:35:44.390" Score="4" ViewCount="323" Body="&lt;p&gt;I need to fine-tune a &lt;code&gt;CNN&lt;/code&gt; to classify two classes: dogs and cats, for example. However, I want the &lt;code&gt;CNN&lt;/code&gt; to be able to tell if there are no dogs nor cats in a given image. Hence, I'm thinking of using a third class called &lt;em&gt;background&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to fine-tune the network with lots of images: images of cats go to the cat class, images of dogs go to the dog class, and every other image goes to the background class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This way, the fine-tuned network would be able to classify a dog as a dog, a cat as a cat, and (hopefully) everything else as background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the right way to do it? Would it work? I can't seem to find reliable information about this online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is a little bit more complicated, but knowing this would be a very good start.&lt;/p&gt;&#xA;" OwnerUserId="46258" LastEditorUserId="28175" LastEditDate="2018-02-14T15:44:54.273" LastActivityDate="2018-10-13T00:01:31.380" Title="Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image" Tags="|machine-learning|neural-network|classification|deep-learning|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27808" PostTypeId="2" ParentId="27807" CreationDate="2018-02-14T15:44:30.743" Score="2" Body="&lt;p&gt;Actually you are in the right path but in the question you are wrong in the second paragraph. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you should do is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;change the output layer to 4 classes, dogs, cats, dogs and cats, none.&lt;/li&gt;&#xA;&lt;li&gt;If you have so many data, don't freeze the convolutional layers and train them all but if not, freeze them and fine tune the last layers, maybe just the dense layers.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-14T15:44:30.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27809" PostTypeId="2" ParentId="27805" CreationDate="2018-02-14T15:50:22.423" Score="1" Body="&lt;p&gt;Suppose that you want to know the name of countries with more than ten &lt;em&gt;Ginger&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# read your data using pd.read_csv and specify appropriate inputs based on your excel file in a typical variable named csvFile and transpose it&#xA;csvFile = csvFile[csvFile['Ginger'] &amp;gt; 10]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also if you want to see the countries with zero &lt;em&gt;Ginger&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;csvFile = csvFile[csvFile['Ginger'] == 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-14T15:58:06.593" LastActivityDate="2018-02-14T15:58:06.593" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27810" PostTypeId="2" ParentId="27806" CreationDate="2018-02-14T15:59:43.283" Score="4" Body="&lt;p&gt;Actually, I don't think this is a problem that could be solved just from the image because it could be a small object that is really close or and big object really far. As far as I know, self driving cars use sensors for this. From what I found &lt;a href=&quot;https://photo.stackexchange.com/questions/12434/how-do-i-calculate-the-distance-of-an-object-in-a-photo&quot;&gt;here&lt;/a&gt;, you would need the actual height of the object to calculate it from one image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I strongly suspect you could also calculate the distance by having two images and then using the distance between the two cameras, and the Pythagorean theorem to calculate it. But that's just conjecture.  &lt;/p&gt;&#xA;" OwnerUserId="42417" LastActivityDate="2018-02-14T15:59:43.283" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27811" PostTypeId="2" ParentId="27806" CreationDate="2018-02-14T16:16:26.567" Score="2" Body="&lt;p&gt;As far as I am aware there are no direct architectures that solve this at this moment in time, but I don't think this is a big issue for CNNs in general. For your training set you could choose to either train directly to estimate a distance if it's always the same object or if there are multiple objects and you want to detect them you can add the distance to the loss function. You need to tune the weighting between these two different objectives however and it might not be obvious how to do this if you don't know how these object detectors work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: Based on your additional information:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It mostly depends on the training data you have available, if you have historical distances between the camera and the vehicle then I think a system like I proposed will work significantly better because it allows you to train the network end-to-end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could imagine you only have bounding boxes, then you might need to find a different solution, either by approximating the size of the car (by identifying the type of car for example) and using the size of the predicted bounding box as an indicator for how far away it really is. Different cameras might make this tricky due to different types of distortion.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="14904" LastEditDate="2018-02-15T14:23:00.953" LastActivityDate="2018-02-15T14:23:00.953" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27817" PostTypeId="2" ParentId="27795" CreationDate="2018-02-14T19:05:23.057" Score="2" Body="&lt;p&gt;I would pick a classifier, like VGG-16, that works well on the imagenet classes. Then, run your watch images through it. For sure, you can expect the output to be mostly &quot;watch&quot; with high probability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you then get extra features: the activation level of all other categories. That gives you a vector of a thousand values between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also extract the activation at various point in the network. Then, the similarity of those activations and outputs should be similar between two cases only if the images are similar.&lt;/p&gt;&#xA;" OwnerUserId="46272" LastActivityDate="2018-02-14T19:05:23.057" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27818" PostTypeId="2" ParentId="18667" CreationDate="2018-02-14T21:00:09.173" Score="-1" Body="&lt;p&gt;Because with MNIST, you are trying to predict based on probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sigmoid function squishes the $x$ value between $0$ and $1$. This helps to pick the most probable digit that matches the label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ReLU function doesn't squish anything. If the $x$ value is less than $0$, the the output is $0$. If its more than $0$, the answer is the $x$ value itself. No probabilities are being created. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Honestly, I'm suprised you got anything more than 10% when you plug it in.&lt;/p&gt;&#xA;" OwnerUserId="46279" LastEditorUserId="29575" LastEditDate="2018-02-14T21:57:27.937" LastActivityDate="2018-02-14T21:57:27.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27819" PostTypeId="1" CreationDate="2018-02-14T23:13:00.410" Score="3" ViewCount="20305" Body="&lt;p&gt;I'm struggling with a Pandas problem. I have the following data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;| symbol | side | status  | origQty | executedQty |     qty     | availableQty |   price    | boughtValue | soldValue  | dcaLevel |&#xA;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;| DGDBTC | BUY  | FILLED  |   0.125 |  0.12500000 |  0.12500000 |   0.12500000 | 0.02000700 |  0.00250088 |            |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.125 |  0.12500000 |  0.12500000 |   0.25000000 | 0.01960100 |  0.00245013 |            |          |&#xA;| DGDBTC | SELL | FILLED  |    0.25 |  0.25000000 | -0.25000000 |   0.00000000 | 0.02005900 |             | 0.00501475 |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.113 |  0.11300000 |  0.11300000 |   0.11300000 | 0.02203000 |  0.00248939 |            |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.113 |  0.11300000 |  0.11300000 |   0.22600000 | 0.02160300 |  0.00244114 |            |          |&#xA;| DGDBTC | BUY  | EXPIRED |   0.226 |  0.00000000 |  0.00000000 |              | 0.02125500 |             |            |          |&#xA;| DGDBTC | BUY  | PARTIAL |   0.226 |  0.15800000 |  0.15800000 |   0.38400000 | 0.02126100 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02196600 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02214300 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02189900 |             |            |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.384 |  0.38400000 |  0.38400000 |   0.76800000 | 0.02082900 |  0.00799834 |            |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.768 |  0.76800000 |  0.76800000 |   1.53600000 | 0.01984300 |  0.01523942 |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02074400 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02094100 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02076800 |             |            |          |&#xA;| DGDBTC | SELL | PARTIAL |   1.536 |  0.30300000 | -0.30300000 |   1.23300000 | 0.02065000 |             |            |          |&#xA;| DGDBTC | SELL | FILLED  |   1.233 |  1.23300000 | -1.23300000 |   0.00000000 | 0.02070000 |             | 0.02552310 |          |&#xA;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This a subset of the data group by symbol. For each symbol I want to populate the last column with a value that complies with the following rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Each buy order (side=BUY) in a series has the value zero (0). &lt;/li&gt;&#xA;&lt;li&gt;For each consecutive buy order the value is increased by&#xA;one (1).&lt;/li&gt;&#xA;&lt;li&gt;When a sell order (side=SELL) is reached it marks a new buy order serie. &lt;/li&gt;&#xA;&lt;li&gt;Rows with status EXPIRED are skipped.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;| symbol | side | status  | origQty | executedQty |     qty     | availableQty |   price    | boughtValue | soldValue  | dcaLevel |&#xA;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;| DGDBTC | BUY  | FILLED  |   0.125 |  0.12500000 |  0.12500000 |   0.12500000 | 0.02000700 |  0.00250088 |            |        0 |&#xA;| DGDBTC | BUY  | FILLED  |   0.125 |  0.12500000 |  0.12500000 |   0.25000000 | 0.01960100 |  0.00245013 |            |        1 |&#xA;| DGDBTC | SELL | FILLED  |    0.25 |  0.25000000 | -0.25000000 |   0.00000000 | 0.02005900 |             | 0.00501475 |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.113 |  0.11300000 |  0.11300000 |   0.11300000 | 0.02203000 |  0.00248939 |            |        0 |&#xA;| DGDBTC | BUY  | FILLED  |   0.113 |  0.11300000 |  0.11300000 |   0.22600000 | 0.02160300 |  0.00244114 |            |        1 |&#xA;| DGDBTC | BUY  | EXPIRED |   0.226 |  0.00000000 |  0.00000000 |              | 0.02125500 |             |            |          |&#xA;| DGDBTC | BUY  | PARTIAL |   0.226 |  0.15800000 |  0.15800000 |   0.38400000 | 0.02126100 |             |            |        2 |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02196600 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02214300 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   0.384 |  0.00000000 |  0.00000000 |              | 0.02189900 |             |            |          |&#xA;| DGDBTC | BUY  | FILLED  |   0.384 |  0.38400000 |  0.38400000 |   0.76800000 | 0.02082900 |  0.00799834 |            |        3 |&#xA;| DGDBTC | BUY  | FILLED  |   0.768 |  0.76800000 |  0.76800000 |   1.53600000 | 0.01984300 |  0.01523942 |            |        4 |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02074400 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02094100 |             |            |          |&#xA;| DGDBTC | SELL | EXPIRED |   1.536 |  0.00000000 |  0.00000000 |              | 0.02076800 |             |            |          |&#xA;| DGDBTC | SELL | PARTIAL |   1.536 |  0.30300000 | -0.30300000 |   1.23300000 | 0.02065000 |             |            |          |&#xA;| DGDBTC | SELL | FILLED  |   1.233 |  1.23300000 | -1.23300000 |   0.00000000 | 0.02070000 |             | 0.02552310 |          |&#xA;+--------+------+---------+---------+-------------+-------------+--------------+------------+-------------+------------+----------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried the following two ways.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;merged_df['dcaLevel'] = merged_df[(merged_df['side'] == 'BUY') &amp;amp; (merged_df['status'].isin(['FILLED', 'PARTIAL']))].groupby(['symbol'])['dcaLevel'].apply(lambda x: x.shift(1) + 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This way throws an error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;merged_df['dcaLevel'] = merged_df[(merged_df['side'] == 'BUY') &amp;amp; (merged_df['status'].isin(['FILLED', 'PARTIAL']))].groupby(['symbol'])['dcaLevel'].apply(lambda x: 0 if x.shift(1) else x.shift(1) + 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried the following alternative approach.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;symbol_df = merged_df.loc[merged_df['symbol'] == 'DGDBTC']&#xA;tmp_df = symbol_df[(symbol_df['side'] == 'BUY') &amp;amp; (symbol_df['status'].isin(['FILLED', 'PARTIAL']))]&#xA;tmp_df['dcaLevel'] = np.where(tmp_df['availableQty'] &amp;lt; tmp_df['availableQty'].shift(1), 0, tmp_df['dcaLevel'].shift(1) + 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It works on some rows and not on others and the first buy order in a serie remains NaN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have coded the following which does work, but I'm sure there is an easier way to do this with Pandas.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;merged_df['dcaLevel'] = np.NaN&#xA;grouped = merged_df[merged_df['status'].isin(['FILLED', 'PARTIAL'])].groupby(['symbol'])&#xA;col_idx = merged_df.columns.get_loc('dcaLevel')&#xA;for name, group in grouped:&#xA;    first = True&#xA;    for index, row in group.iterrows():&#xA;        if row['side'] == 'SELL':&#xA;            first = True&#xA;            dca_level = np.NaN&#xA;        else:&#xA;            if first:&#xA;                first = False&#xA;                dca_level = 0&#xA;            else:&#xA;                dca_level = dca_level + 1&#xA;            merged_df.iloc[index, col_idx] = dca_level&#xA;merged_df[merged_df['symbol'] == 'DGDBTC']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope someone can help solve this problem.&lt;/p&gt;&#xA;" OwnerUserId="46283" LastEditorUserId="46283" LastEditDate="2018-02-14T23:52:40.467" LastActivityDate="2019-04-17T10:52:37.340" Title="Populate column based on previous row with a twist" Tags="|pandas|data-wrangling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27820" PostTypeId="1" CreationDate="2018-02-14T23:23:30.310" Score="2" ViewCount="417" Body="&lt;p&gt;I am a beginner in ML, though I have completed Andrew Ng's Machine Learning course on Coursera and am soon to complete his Deep Learning specialization, also on Coursera. I'm looking for recommendations for things to study after that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They need to be on the easier side since I am still learning. Also in addition to resources on ML or Deep Learning, I'm also interested in any resources on the underlying math. I have a book in probability and another book in calculus, though both are fairly dense. Anything lighter might be a plus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any resources you would recommend? I'm particularly interested in other courses on Coursera since that's what my company currently sponsors, though I'm interested in whatever suggestions you have to offer.&lt;/p&gt;&#xA;" OwnerUserId="44394" LastActivityDate="2018-03-01T20:52:07.517" Title="recommendations for self-study in ML/Deep Learning/the underlying math" Tags="|self-study|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27821" PostTypeId="2" ParentId="27819" CreationDate="2018-02-15T01:01:18.877" Score="2" Body="&lt;p&gt;A simple approach could be the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;counter = 0&#xA;&#xA;def conditional_cumulative_sum(x):&#xA;    global counter&#xA;&#xA;    if x['status'] == 'EXPIRED':&#xA;        return np.nan&#xA;    elif x['side'] == 'BUY':&#xA;        temporal = counter&#xA;        counter += 1&#xA;        return temporal&#xA;    elif x['side'] == 'SELL':&#xA;        counter = 0&#xA;        return np.nan&#xA;&#xA;frame = pd.read_csv('sample.csv')&#xA;frame['dcaLevel'] = frame.apply(conditional_cumulative_sum, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Assuming &lt;strong&gt;sample.csv&lt;/strong&gt; contains the data provided in the examples and that the empty values in the column &lt;code&gt;dcaLevel&lt;/code&gt; correspond to NaN values.&lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2018-02-15T01:01:18.877" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27822" PostTypeId="1" CreationDate="2018-02-15T01:38:58.983" Score="2" ViewCount="1928" Body="&lt;p&gt;I am trying to use &lt;code&gt;tflearn.objectives.roc_auc_score&lt;/code&gt; as a loss function for a GRU network in Keras but I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; ValueError: An operation has `None` for gradient. Please make sure&#xA;&amp;gt; that all of your ops have a gradient defined (i.e. are&#xA;&amp;gt; differentiable). Common ops without gradient: K.argmax, K.round,&#xA;&amp;gt; K.eval.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is surprising as &lt;a href=&quot;http://tflearn.org/objectives/#roc-auc-score&quot; rel=&quot;nofollow noreferrer&quot;&gt;the implementation&lt;/a&gt; is apparently based on an approximation that is supposed to be differentiable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your reference, here is the code from the &lt;a href=&quot;https://github.com/tflearn/tflearn/blob/master/tflearn/objectives.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;tflearn Github&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def roc_auc_score(y_pred, y_true):&#xA;    &quot;&quot;&quot; ROC AUC Score.&#xA;    Approximates the Area Under Curve score, using approximation based on&#xA;    the Wilcoxon-Mann-Whitney U statistic.&#xA;    Yan, L., Dodier, R., Mozer, M. C., &amp;amp; Wolniewicz, R. (2003).&#xA;    Optimizing Classifier Performance via an Approximation to the Wilcoxon-Mann-Whitney Statistic.&#xA;    Measures overall performance for a full range of threshold levels.&#xA;    Arguments:&#xA;        y_pred: `Tensor`. Predicted values.&#xA;        y_true: `Tensor` . Targets (labels), a probability distribution.&#xA;    &quot;&quot;&quot;&#xA;    with tf.name_scope(&quot;RocAucScore&quot;):&#xA;&#xA;        pos = tf.boolean_mask(y_pred, tf.cast(y_true, tf.bool))&#xA;        neg = tf.boolean_mask(y_pred, ~tf.cast(y_true, tf.bool))&#xA;&#xA;        pos = tf.expand_dims(pos, 0)&#xA;        neg = tf.expand_dims(neg, 1)&#xA;&#xA;        # original paper suggests performance is robust to exact parameter choice&#xA;        gamma = 0.2&#xA;        p     = 3&#xA;&#xA;        difference = tf.zeros_like(pos * neg) + pos - neg - gamma&#xA;&#xA;        masked = tf.boolean_mask(difference, difference &amp;lt; 0.0)&#xA;&#xA;return tf.reduce_sum(tf.pow(-masked, p)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="34033" LastEditorUserId="34033" LastEditDate="2018-02-15T01:45:31.110" LastActivityDate="2019-04-22T08:02:35.537" Title="ROC-AUC loss for GRU Model: Cannot use tflearn's loss in keras" Tags="|neural-network|keras|recommender-system|loss-function|tflearn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27823" PostTypeId="1" AcceptedAnswerId="27838" CreationDate="2018-02-15T02:14:52.607" Score="0" ViewCount="375" Body="&lt;p&gt;Let's say you have a data set with GPA (dependent variable) and Amount of alcohol, Amount of study, IQ, and SAT score as the independent variables. And you want to perform the principal component analysis in R for dimension reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the csv file you read in R, do you have to have GPA or do you have to remove it?&lt;/p&gt;&#xA;" OwnerUserId="46230" LastActivityDate="2018-02-15T11:44:50.933" Title="Including the dependent variable in your data to perform principal component analysis?" Tags="|machine-learning|data-mining|dimensionality-reduction|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27824" PostTypeId="2" ParentId="27767" CreationDate="2018-02-15T02:33:27.753" Score="5" Body="&lt;p&gt;In my experience, initializing &lt;code&gt;read_csv()&lt;/code&gt; with parameter &lt;code&gt;low_memory=False&lt;/code&gt; tends to help when reading in large files. I don't think you have mentioned the file type you are reading in, so I am not sure how applicable this is to your situation though.&lt;/p&gt;&#xA;" OwnerUserId="34033" LastActivityDate="2018-02-15T02:33:27.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27826" PostTypeId="1" AcceptedAnswerId="27827" CreationDate="2018-02-15T03:35:41.733" Score="1" ViewCount="167" Body="&lt;p&gt;I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I  would like to calculate the correlation between two glasses on two different people. At the moment I cannot use &lt;code&gt;df.correlate()&lt;/code&gt; because the timestamps are not identical and therefore the data looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;index | ts | r_person | l_person&#xA;-----------------------&#xA;0     | 23 | 3.0      | NAN&#xA;1     | 25 | NAN      | 3.2&#xA;2     | 28 | 3.1      | NAN&#xA;3     | 32 | 3.0      | NAN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there was still any way to calculate a correlation directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column &lt;code&gt;r_person&lt;/code&gt; would become $3.05$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would be less trivial than it seems because even at the start it wasn't always one data point R one data point L and after cleaning the data it has become less so. In other words multiple NAN values might appear in the same column as you can see in the example. I can still deal with that by just spreading out the average. My second technique was going to try and merge the values together if they were close enough. Bearing in mind the data was collected at 50hz. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is whether anyone has a quicker or better way of aligning the data without losing it or changing it too much?&lt;/p&gt;&#xA;" OwnerUserId="34138" LastEditorUserId="29575" LastEditDate="2018-02-15T03:59:09.763" LastActivityDate="2018-02-15T11:00:29.427" Title="Calculating correlation of slightly out of sync data" Tags="|python|pandas|data-cleaning|correlation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27827" PostTypeId="2" ParentId="27826" CreationDate="2018-02-15T05:48:06.727" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The problem here is that the missing data point isn't necessarily associated with a timestamp that is the average of the timestamps above and below. If it were, the procedure you suggested would be equivalent to a linear interpolation, i.e. drawing a line between the points before and after the missing timestamp, and using that line to construct a local prediction for the missing observation. Which is what you should actually do instead. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example in the case above, the line for the point your trying to interpolate is $\text{r_person}(ts) = 3.0 +\frac{.1}{5}(ts-23)$, so $\text{r_person}(25) = 3.04$ (I think... doing this in my head right now).&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-02-15T05:53:14.673" LastActivityDate="2018-02-15T05:53:14.673" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27829" PostTypeId="1" CreationDate="2018-02-15T06:07:56.250" Score="6" ViewCount="1537" Body="&lt;p&gt;I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both columns to index form to simplify the data. As such, the set has 2 columns; 1. A unique ID and 2. A bag of key strings (they are identifying numbers but in string format, separated by spaces). Example format:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ID |  KeyWords&lt;/li&gt;&#xA;&lt;li&gt;0 |   1 2 &lt;/li&gt;&#xA;&lt;li&gt;1 |   2&lt;/li&gt;&#xA;&lt;li&gt;2 |   3&lt;/li&gt;&#xA;&lt;li&gt;3 |   2 3&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I initially vectorised KeyWords by TD-IDF and used a simple and dirty KMeans (via sklearn) to cluster them into 13 clusters (selected by the elbow method), which seemed to work and output intuitive and interesting results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the issue I have come across is that there is always one dominant cluster that holds ~80% of the observations. I assume that KMeans was the limiting factor with assumed constant variance, and looked into Latent Dirichlet Allocation (LDA), as well as Gaussian Mixture Modelling (GMM) and BayesianGMM as alternatives. Using the same data format as above, LDA seemed to produce unintuitive results, for which all the topics were very similar and did not make much sense (vastly different to KMeans clusters, expected to see some key similar topics?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm quite new and out of my depth with clustering and have been reading quite a lot about it, but if anyone could provide any insights into immediate logic flaws or a push in the right direction I would very much appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &#xA;There's about 3300 unique URLs (or KeyWords in this case). The number of tokens/URLs per document is quite skewed actually. The median is 1, mean is ~1.5 (there is a small number of ID's that have a very large number of tokens, some I can deem as outliers but most I cannot). Another key fact is that in the dominant cluster, almost all of these &quot;big players&quot; who have looked at a large number of KeyWords are all within the dominant cluster. I removed them to see how it played out, and the dominant cluster remains.&lt;/p&gt;&#xA;" OwnerUserId="46288" LastEditorUserId="46288" LastEditDate="2018-02-15T23:27:45.753" LastActivityDate="2018-02-19T00:20:53.327" Title="Clustering Observations by String Sequences (Python/Pandas df)" Tags="|python|clustering|text-mining|pandas|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27831" PostTypeId="2" ParentId="27722" CreationDate="2018-02-15T08:29:50.723" Score="2" Body="&lt;p&gt;Following @Imran's answer, I found &lt;a href=&quot;http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; in one of in the comments of the CrossValidated post he linked to. Besides an attempt to find the right architecture using Genetic Models (instead of using a rule-of-thumb), section 2.1 gives some theoretical bounds to how many hidden units should be in a one/two-hidden-layers system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; &lt;a href=&quot;https://medium.com/figuring-data-out/are-genetic-models-better-than-random-sampling-8c678002d392&quot; rel=&quot;nofollow noreferrer&quot;&gt;I've tested this theorem&lt;/a&gt;, and found out that using Genetic Models is just as good as selecting a random architecture.&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="43870" LastEditDate="2018-04-09T13:01:00.513" LastActivityDate="2018-04-09T13:01:00.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27833" PostTypeId="2" ParentId="13429" CreationDate="2018-02-15T09:08:46.613" Score="3" Body="&lt;p&gt;The original &lt;a href=&quot;https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Seq2Seq paper&lt;/a&gt; uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed &lt;strong&gt;&lt;em&gt;teacher forcing&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AB7a1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AB7a1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder &lt;em&gt;peeks&lt;/em&gt; the encoded input at each time step).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The paper &quot;&lt;a href=&quot;https://arxiv.org/pdf/1406.1078.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Phrase Representations using RNN Encoder–Decoder&#xA;for Statistical Machine Translation&lt;/a&gt;&quot; combines these both techniques (so it passes encoded input vector along with time delayed output sequence as inputs to decoder).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ux0Xh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ux0Xh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28054" LastEditorUserId="28054" LastEditDate="2018-02-15T09:50:08.383" LastActivityDate="2018-02-15T09:50:08.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27835" PostTypeId="1" CreationDate="2018-02-15T10:31:58.020" Score="1" ViewCount="865" Body="&lt;p&gt;I am working on texture classification and based on previous works, I am trying to modify the final layer of AlexNET to have 20 classes, and train only that layer for my multi class classification problem.&#xA;I am using Tensorflow-GPU on an NVIDIA GTX 1080, Python3.6 on Ubuntu 16.04.&#xA;I am using the Gradient Descent Optimiser and the class Estimator to build this. I am also using two dropout layers for regularization. Therefore, my hyper parameters are the learning rate, batch_size, and weight_decay. I have tried using batch_size of 50,100,200,weight_decays of 0.005 and 0.0005, and learning rates of 1e-3,1e-4,and 1e-5. All the training loss curves for the above values follow similar trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My training loss curve does not monotonically decrease and instead seems to oscillate. I have provided a tensorboard visualization for learning rate=1e-5, weight decay=0.0005, and batch_size=200. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please assist in understanding what went wrong and how I could possibly rectify it.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/CiGWt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Tensorboard Visualization for the case I specified&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  # Create the Estimator&#xA;  classifier = tf.estimator.Estimator(model_fn=cnn_model)&#xA;  # Set up logging for predictions&#xA;  tensors_to_log = {&quot;probabilities&quot;: &quot;softmax_tensor&quot;}&#xA;  logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=10)&#xA; # Train the model&#xA;  train_input_fn = tf.estimator.inputs.numpy_input_fn(x={&quot;x&quot;: train_data},y=train_labels,batch_size=batch_size,num_epochs=None,shuffle=True)&#xA;  classifier.train(input_fn=train_input_fn, steps=200000, hooks=[logging_hook])&#xA;  # Evaluate the model and print results&#xA;  eval_input_fn = tf.estimator.inputs.numpy_input_fn(&#xA;x={&quot;x&quot;: eval_data},&#xA;  y=eval_labels,&#xA;  num_epochs=1,&#xA;  shuffle=False)&#xA;  eval_results = classifier.evaluate(input_fn=eval_input_fn)&#xA;  print(eval_results)&#xA;&#xA;#Sections of the cnn_model&#xA; #Output Config&#xA; predictions = { &quot;classes&quot;: tf.argmax(input=logits, axis=1),# Generate predictions (for PREDICT and EVAL mode)&#xA;&quot;probabilities&quot;: tf.nn.softmax(logits, name=&quot;softmax_tensor&quot;)}  # Add `softmax_tensor` to the graph. It is used for PREDICT and by the `logging_hook`.&#xA;if mode == tf.estimator.ModeKeys.PREDICT:&#xA;  return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)&#xA;&#xA; # Calculate Loss (for both TRAIN and EVAL modes)&#xA;  onehot_labels = tf.one_hot(indices=tf.cast(labels,tf.int32),depth=20)&#xA;  loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)&#xA;&#xA;&#xA;#Training Config&#xA;  if mode == tf.estimator.ModeKeys.TRAIN:&#xA;    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)&#xA;    tf.summary.scalar('training_loss',loss)&#xA;    summary_hook = tf.train.SummarySaverHook(save_steps=10,output_dir='outputs',summary_op=tf.summary.merge_all())&#xA;    train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())&#xA;    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op,training_hooks=[summary_hook])&#xA;&#xA;&#xA;&#xA;  # Evaluation Metric- Accuracy&#xA;  eval_metric_ops = {&quot;accuracy&quot;: tf.metrics.accuracy(labels=labels, predictions=predictions[&quot;classes&quot;])}&#xA;   print(time.time()-t)&#xA;   tf.summary.scalar('eval_loss',loss)&#xA;   ac=tf.metrics.accuracy(labels=labels,predictions=predictions[&quot;classes&quot;])&#xA;   tf.summary.scalar('eval_accuracy',ac)&#xA;   evaluation_hook= tf.train.SummarySaverHook(save_steps=10,output_dir='outputseval',summary_op=tf.summary.merge_all())&#xA;  return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops,evaluation_hooks=[evaluation_hook])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46301" LastActivityDate="2018-02-15T10:31:58.020" Title="Why does my training loss oscillate while training the final layer of AlexNet with pre-trained weights?" Tags="|python|tensorflow|cnn|alex-net|" AnswerCount="0" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="27836" PostTypeId="2" ParentId="27826" CreationDate="2018-02-15T11:00:29.427" Score="0" Body="&lt;p&gt;You could collect your time points into half-open intervals like $\dots, [20, 25), [25, 30), [30, 35), \dots$ and correlate the &lt;em&gt;average&lt;/em&gt; observations for &lt;code&gt;r_person&lt;/code&gt; and &lt;code&gt;l_person&lt;/code&gt;.&#xA;An example with a few more data points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;df = pd.DataFrame([[0, 23, 3.0, np.nan],&#xA;                   [1, 24, 2.9, np.nan],&#xA;                   [2, 25, np.nan, 3.2],&#xA;                   [3, 27, 3.0, np.nan],&#xA;                   [4, 27, np.nan, 3.3],&#xA;                   [5, 28, 3.1, np.nan],&#xA;                   [6, 29, np.nan, 3.2],&#xA;                   [7, 32, 3.0, np.nan]],&#xA;                  columns=['index', 'tx', 'r_person', 'l_person'])&#xA;&#xA;bin_min = df['tx'].min() // 5 * 5&#xA;bin_max = (df['tx'].max() // 5 + 1) * 5&#xA;bins = range(bin_min, bin_max + 1, 5)&#xA;&#xA;df['tx_bins'] = pd.cut(df['tx'], bins)&#xA;df_binned = df[['tx_bins', 'r_person', 'l_person']].groupby('tx_bins').mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This yields&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df_binned&#xA;          r_person  l_person&#xA;tx_bins                     &#xA;(20, 25]      2.95      3.20&#xA;(25, 30]      3.05      3.25&#xA;(30, 35]      3.00       NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-15T11:00:29.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27837" PostTypeId="2" ParentId="22494" CreationDate="2018-02-15T11:02:38.660" Score="2" Body="&lt;p&gt;One thing that hasn't been mentioned yet and that you can consider for the future: you can still increase your dropout at the fully connected layers.&lt;/p&gt;&#xA;&lt;p&gt;I read a paper once that used 90% dropout rate. Although it had many many nodes (2048 if i recall correctly), I have tried this myself on layers with fewer nodes and it was very helpful in some cases.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;I just looked up which paper it was. I can't recall which paper I just remembered but I found these that also had some success with 90% dropout rates.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., &amp;amp;&#xA;Fei-Fei, L. (2014). Large-scale video classification with&#xA;convolutional neural networks. In Proceedings of the IEEE conference&#xA;on Computer Vision and Pattern Recognition (pp. 1725-1732).&lt;/p&gt;&#xA;&lt;p&gt;Simonyan, K., &amp;amp; Zisserman, A. (2014). Two-stream convolutional&#xA;networks for action recognition in videos. In Advances in neural&#xA;information processing systems (pp. 568-576).&lt;/p&gt;&#xA;&lt;p&gt;Varol, G., Laptev, I., &amp;amp; Schmid, C. (2017). Long-term temporal&#xA;convolutions for action recognition. IEEE transactions on pattern&#xA;analysis and machine intelligence.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46302" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-15T14:27:46.850" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27838" PostTypeId="2" ParentId="27823" CreationDate="2018-02-15T11:44:50.933" Score="1" Body="&lt;p&gt;You need to remove the dependent variable before performing PCA.&#xA;Otherwise you are essentially using the dependent variable to explain itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, think about prediction:&#xA;You might want to use your model to predict unknown GPA scores based on information about amount of alcohol, amount of study, IQ and SAT. In that case, it is impossible to include information on GPA scores.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-15T11:44:50.933" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27839" PostTypeId="2" ParentId="11118" CreationDate="2018-02-15T12:07:26.063" Score="1" Body="&lt;p&gt;Competitive self-play without human database is even possible for complicated, partially observed environments. OpenAI is focusing on this direction.&#xA;According to &lt;a href=&quot;http://ll,%20without%20explicitly%20designing%20an%20environment%20with%20these%20skills%20in%20mind.%20Self-play%20ensures%20that%20the%20environment%20is%20always%20the%20right%20difficulty%20for%20an%20AI%20to%20improve.%20Taken%20alongside%20our%20Dota%202%20self-play%20results,%20we%20have%20increasing%20confidence%20that%20self-play%20will%20be%20a%20core%20part%20of%20powerful%20AI%20systems%20in%20the%20future.&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Self-play ensures that the environment is always the right difficulty for an AI to improve.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's an important reason for the success of self-play.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OpenAI achieved superhuman results for Dota 2 1v1, in August 11th 2017, &lt;a href=&quot;https://blog.openai.com/more-on-dota-2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;beat Dendi&lt;/a&gt; 2-0 under standard tournament rules.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The bot learned the game from scratch by self-play, and does not use imitation learning or tree search. This is a step towards building AI systems which accomplish well-defined goals in messy, complicated situations involving real humans.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not just games, this direction is also promising for robotics tasks.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We’ve found that self-play allows simulated AIs to discover physical skills like tackling, ducking, faking, kicking, catching, and diving for the ball, without explicitly designing an environment with these skills in mind.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the next step, they extend the method to learn how to &lt;a href=&quot;https://blog.openai.com/learning-to-cooperate-compete-and-communicate/&quot; rel=&quot;nofollow noreferrer&quot;&gt;cooperate, compete and communicate&lt;/a&gt;, not just limit to self-play.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-15T12:07:26.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27840" PostTypeId="1" CreationDate="2018-02-15T12:46:47.083" Score="2" ViewCount="376" Body="&lt;p&gt;I'm interested in working on challenging AI problems, and after reading this article (&lt;a href=&quot;https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/&lt;/a&gt;) by DeepMind and Blizzard, I think that developing a robust AI capable of learning to play Starcraft 2 with superhuman level of performance (without prior knowledge or human hard-coded heuristics) would imply a huge breakthrough in AI research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure I know this is an extremely challenging problem, and by no means I pretend to be the one solving it, but I think it's a challenge worth taking on nonetheless because the complexity of the decision making required is much closer to the real world and so this forces you to come up with much more robust, generalizable AI algorithms that could potentially be applied to other domains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, an AI that plays Starcraft 2 would have to be able to watch the screen, identify objects, positions, identify units moving and their trajectories, update its current knowledge of the world, make predictions, make decisions, have short term and long term goals, listen to sounds (because the game includes sounds), understand natural language (to read and understand text descriptions appearing in the screen as well), it should probably be endowed also with some sort of attention mechanism to be able to pay attention to certain regions of interest of the screen, etc. So it becomes obvious that at least one would need to know about Computer Vision, Object Recognition, Knowledge Bases, Short Term / Long Term Planning, Audio Recognition, Natural Language Processing, Visual Attention Models, etc. And obviously it would not be enough to just study each area independently, it would also be necessary to come up with ways to integrate everything into a single system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, does anybody know good resources with content relevant to this problem? I would appreciate any suggestions of papers, books, blogs, whatever useful resource out there (ideally state-of-the-art) which would be helpful for somebody interested in this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="46305" LastEditorUserId="46305" LastEditDate="2018-02-15T13:05:02.367" LastActivityDate="2018-05-17T12:18:47.640" Title="Training an AI to play Starcraft 2 with superhuman level of performance?" Tags="|machine-learning|computer-vision|audio-recognition|nlp|ai|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27841" PostTypeId="2" ParentId="27829" CreationDate="2018-02-15T13:14:06.177" Score="1" Body="&lt;p&gt;As URLs are usually unique strings, I am not sure if NLP methods are proper things here. That is interesting that TFIDF+Kmeans gave you an intuitive results however as the problem is unsupervised, it´s pretty difficult to say what is intuitive that way. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion your problem is a typical &lt;a href=&quot;https://arxiv.org/abs/0906.0612&quot; rel=&quot;nofollow noreferrer&quot;&gt;Community Detection&lt;/a&gt; in &lt;a href=&quot;https://en.wikipedia.org/wiki/Bipartite_graph&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bipartite Graphs&lt;/a&gt; e.g. &lt;a href=&quot;https://arxiv.org/pdf/cs/0108018.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;. This approach is certainly worth to try. Of course the result can be combined with other algorithms e.g. in a voting schema.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last but not the least is to take a clustering method which does not limit to resolution e.g. in Community Detection, any &lt;a href=&quot;https://en.wikipedia.org/wiki/Modularity_(networks)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modularity&lt;/a&gt;-based method is only able to capture clusters larger than a resolution limit. One approach to this is a hierarchical approach to partitioning the data in which you again apply clustering on the significantly large cluster you find.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-02-15T13:14:06.177" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27842" PostTypeId="1" AcceptedAnswerId="27845" CreationDate="2018-02-15T13:33:54.183" Score="0" ViewCount="128" Body="&lt;p&gt;I have a question regarding my machine learning lecture where we had to decide whether $$K(x,y)=x_1y_1-x_2y_2$$ is a valid kernel (e.g. for a SVM). My intuition would say that it is a valid kernel since we can display it with: $$\Phi(x)=(x_1, ix_2)\implies K(x,y)=\Phi(x)\Phi(y)$$ with $i$ being the imaginary number. Is that right?&lt;/p&gt;&#xA;" OwnerUserId="46308" LastActivityDate="2018-02-15T15:51:12.150" Title="Kernel with complex vectors" Tags="|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27843" PostTypeId="1" AcceptedAnswerId="27876" CreationDate="2018-02-15T14:21:15.000" Score="0" ViewCount="1198" Body="&lt;p&gt;I am writing an algorithm to fit a sine wave. I want to have 4 parameters ( amplitude, frequency, phase &amp;amp; centre position). When I tried to program with all 4 parameters I wasn't able to find a good fit. So I tried to first make it work with each parameter separately and making the other 3 constant hardcoded values. I managed to make this work for all parameters except for the frequency. So I think the problem is the derivative of the error function w.r.t the frequency parameter. As seen below all the other parameters of the sine are hardcoded values except the frequency parameter&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%This is iterated for a large number of times&#xA;for i = 1:length(x)&#xA;%computing hypothesis and error&#xA;    TrialY(i) = 4* sin(w2(end) *x(i) + 4) -2;&#xA;    Error(end) = Error(end) + (TrialY(i) - y(i))^2;&#xA;end&#xA;Error(end) = Error(end) / (2*length(x));&#xA;&#xA;dw2 = 0; %derivative value&#xA;&#xA;%derivative of error function with respect to frequency (w2)&#xA;for i = 1: length(x)&#xA;    dw2 = dw2 + (4* sin(w2(end) *x(i) + 4) -2 - y(i)) * 4 * x(i) * cos(w2(end) * x(i) + 4);&#xA;%hardcoded derivative&#xA;end&#xA;dw2 = 2 * dw2 / length(x); %normalising error&#xA;&#xA;w2(end + 1) = 0;&#xA;&#xA;w2(end) = w2(end - 1) - alpha * dw2; % alpha is the update constant&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems like the parameter is not converging on the right value with whatever &lt;code&gt;alpha&lt;/code&gt; and initial values I give it. While the plot of the Error with time is decreasing but converging on a value &gt; 0.&lt;/p&gt;&#xA;" OwnerUserId="46309" LastActivityDate="2018-02-16T06:01:32.110" Title="Linear Regression of Sine wave using Gradient descent Not working" Tags="|linear-regression|gradient-descent|matlab|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-18T13:58:09.983" ContentLicense="CC BY-SA 3.0" />
  <row Id="27845" PostTypeId="2" ParentId="27842" CreationDate="2018-02-15T14:41:00.033" Score="0" Body="&lt;p&gt;Kernels are considered valid if they are &lt;a href=&quot;https://en.wikipedia.org/wiki/Positive-definite_kernel#Definition&quot; rel=&quot;nofollow noreferrer&quot;&gt;positive-definite&lt;/a&gt;, so we must have&#xA;$$&#xA;\sum_{i=1}^n \sum_{j=1}^n c_i c_j K(x_i, x_j) \ge 0&#xA;$$&#xA;for all $n \in \mathbb{N}$ and $c_1, \dots, c_n \in \mathbb{R}$ and (in your case) all $x^1, \dots, x^n \in \mathbb{R}^2.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Letting $n=1$ and $c_1 = 1$ and $x^1 = (0, 1)$ shows that $K$ is not positive-definite, since&#xA;$$&#xA;K\big((0,1), (0,1)\big) = 0\times 0 - 1\times 1 = -1.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with your approach is that you define $\Phi$ on $\mathbb{C}$ but write $K$ as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Dot_product#Algebraic_definition&quot; rel=&quot;nofollow noreferrer&quot;&gt;dot product&lt;/a&gt; for &lt;em&gt;real-valued&lt;/em&gt; vectors.&#xA;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Positive-definite_kernel#Connection_with_reproducing_kernel_Hilbert_spaces_and_feature_maps&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature map argument&lt;/a&gt; for positive definite vectors only works if the product is an inner product on $\Phi$'s codomain.&#xA;The dot product for real-valued vectors is &lt;em&gt;not&lt;/em&gt; an inner product on $\mathbb{C}^2$ (because $i\cdot i = -1 &amp;lt; 0$).&#xA;You would have to use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Dot_product#Complex_vectors&quot; rel=&quot;nofollow noreferrer&quot;&gt;generalized dot product&lt;/a&gt; $x \cdot y = \sum_i x_i \overline{y_i}$, or another inner product on $\mathbb{C}^2$.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-15T15:51:12.150" LastActivityDate="2018-02-15T15:51:12.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27846" PostTypeId="1" CreationDate="2018-02-15T14:52:43.897" Score="5" ViewCount="843" Body="&lt;p&gt;If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run?  If I just compare after a few epochs, will that give me a good idea about how it will perform fully converged (say for example after 1000 epochs).  Is there any research papers where people have studied this question?  &lt;/p&gt;&#xA;" OwnerUserId="45397" LastActivityDate="2020-07-21T15:34:51.300" Title="How many epochs to run during hyperparameter search?" Tags="|training|hyperparameter|hyperparameter-tuning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27849" PostTypeId="2" ParentId="10695" CreationDate="2018-02-15T15:46:40.043" Score="1" Body="&lt;p&gt;I have done it &lt;a href=&quot;https://gist.github.com/AbhishekAshokDubey/054af6f92d67d5ef8300fac58f59fcc9&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; in my github repository.&lt;/p&gt;&#xA;&lt;p&gt;See if this is what you need.&lt;/p&gt;&#xA;" OwnerUserId="18641" LastEditorUserId="29169" LastEditDate="2021-03-11T19:05:37.727" LastActivityDate="2021-03-11T19:05:37.727" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27850" PostTypeId="2" ParentId="9637" CreationDate="2018-02-15T15:54:55.473" Score="1" Body="&lt;p&gt;In order to train an agent to play a board game, the first important task is to create a Reinforcement Learning environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4 essential aspects of an environment is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Observation space: What is the input/state, as well as its shape and range?&lt;/li&gt;&#xA;&lt;li&gt;Action space: What is the possible action, as well as its shape and range?&lt;/li&gt;&#xA;&lt;li&gt;Reward function: What is the reward for a particular pair of state + action? Will it be immediate or sparse reward? The reward function will affect the difficulty of the environment a lot.&lt;/li&gt;&#xA;&lt;li&gt;When will an episode/a game end?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I highly suggest using the same API as &lt;a href=&quot;https://gym.openai.com/docs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenAI Gym&lt;/a&gt; because of its popularity and high-quality. Then you can try to apply direct &lt;a href=&quot;https://github.com/openai/baselines&quot; rel=&quot;nofollow noreferrer&quot;&gt;these algorithms&lt;/a&gt; before trying on your own. They are state-of-the-art algorithms and the quality is guaranteed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the neural network architecture, you can exploit the nature of the game for better result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, in Go, the position is symmetric and the actions are simple and position-independent, which is very suitable to use convolutional neural network (AlphaGo Zero). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, in Chess, because the actions are asymmetric and position-dependent, Google had to &lt;a href=&quot;https://arxiv.org/abs/1712.01815&quot; rel=&quot;nofollow noreferrer&quot;&gt;redesign&lt;/a&gt; the architecture to train an agent to play chess.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-15T15:54:55.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27854" PostTypeId="1" CreationDate="2018-02-15T16:32:45.687" Score="0" ViewCount="3291" Body="&lt;p&gt;I have a dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[['Apple', 'Beer', 'Chicken', 'Rice'],&#xA;['Apple', 'Beer', 'Rice'],&#xA;['Apple', 'Beer'],&#xA;['Apple', 'Bananas']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using the frequent itemsets tools, specifically apriori, to find frequent datasets.  I convert to a OneHotDataset, then to a Data Frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;oht = OnehotTransactions()&#xA;oht_ary = oht.fit(tempList).transform(tempList)&#xA;oht_ary&#xA;&#xA;df = pd.DataFrame(oht_ary, columns=oht.columns_)&#xA;df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I use the Apriori function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;frequent_itemsets = apriori(df, min_support=0.01, use_colnames=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I see this table, which is what I expect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;support                      itemsets&#xA;0      1.00                       [Apple]&#xA;1      0.25                     [Bananas]&#xA;2      0.75                        [Beer]&#xA;3      0.25                     [Chicken]&#xA;4      0.50                        [Rice]&#xA;5      0.25              [Apple, Bananas]&#xA;6      0.75                 [Apple, Beer]&#xA;7      0.25              [Apple, Chicken]&#xA;8      0.50                 [Apple, Rice]&#xA;9      0.25               [Beer, Chicken]&#xA;10     0.50                  [Beer, Rice]&#xA;11     0.25               [Chicken, Rice]&#xA;12     0.25        [Apple, Beer, Chicken]&#xA;13     0.50           [Apple, Beer, Rice]&#xA;14     0.25        [Apple, Chicken, Rice]&#xA;15     0.25         [Beer, Chicken, Rice]&#xA;16     0.25  [Apple, Beer, Chicken, Rice]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The question I have is, is there any way to print a suppport &quot;count&quot; of occurrences of the datasets-meaning how many times that itemset appears in the transactions?&lt;/strong&gt;  One thing I messed around with from the mlxtend site was this being added to add an additional column of length but couldn't get a count piece:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(lambda x: len(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;EDIT:  Ok I think I am confusing folks.  What I would like to see is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      support     itemsets    count&#xA;0      1.00       [Apple]       4&#xA;1      0.25      [Bananas]      1&#xA;2      0.75         [Beer]      3&#xA;3      0.25       [Chicken]     1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I guess I could just create a column that uses the total support and multiplies by the relative support, but was looking for a more systematic way to do it. &lt;/p&gt;&#xA;" OwnerUserId="46307" LastEditorUserId="46307" LastEditDate="2018-02-15T17:58:13.140" LastActivityDate="2019-04-14T20:02:00.387" Title="Getting count of frequent itemsets in Python mlxtend" Tags="|python|data-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27856" PostTypeId="1" CreationDate="2018-02-15T09:01:47.317" Score="2" ViewCount="205" Body="&lt;p&gt;Vowpal Wabbit (VW) uses online normalization as explained here [1].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible or is there any code/paper that explores the idea of &lt;strong&gt;doing online learning with multiple workers in a parameter server setting&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] &lt;a href=&quot;https://arxiv.org/abs/1305.6646&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1305.6646&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="31935" OwnerDisplayName="JC1" LastEditorUserId="1330" LastEditDate="2022-03-25T14:33:47.440" LastActivityDate="2022-03-25T14:33:47.440" Title="Vowpal Wabbit Online Normalization -- Possible to parallelize?" Tags="|machine-learning|gradient-descent|mini-batch-gradient-descent|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27858" PostTypeId="1" CreationDate="2018-02-15T16:58:57.130" Score="3" ViewCount="1136" Body="&lt;p&gt;I am trying to run Tensorflow for image recognition (classification) in Java (JSE not Android).&lt;/p&gt;&#xA;&lt;p&gt;I am using the code from &lt;a href=&quot;https://github.com/emara-geek/object-recognition-tensorflow/blob/master/src/main/java/com/emaraic/ObjectRecognition/Recognizer.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, and &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/src/main/java/org/tensorflow/examples/LabelImage.java&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It works for Inceptionv3 models, and for models retrained from Inceptionv3.&lt;/p&gt;&#xA;&lt;p&gt;But for MobileNet models, it does not work, (such as following this &lt;a href=&quot;https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;The code works but gives the wrong results (wrong classify labels). What code/settings are required to use MobileNet from Java?&lt;/p&gt;&#xA;&lt;p&gt;The code that works for Inceptionv3 is&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;try (Tensor image = Tensor.create(imageBytes)) {&#xA;    float[] labelProbabilities = executeInceptionGraph(graphDef, image);&#xA;    int bestLabelIdx = maxIndex(labelProbabilities);&#xA;    result.setText(&amp;quot;&amp;quot;);&#xA;    result.setText(String.format(&#xA;        &amp;quot;BEST MATCH: %s (%.2f%% likely)&amp;quot;,&#xA;        labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));&#xA;        System.out.println(&#xA;            String.format(&#xA;                &amp;quot;BEST MATCH: %s (%.2f%% likely)&amp;quot;,&#xA;                labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This works of Inceptionv3 models, but not MobileNet,&#xA;Gives the error, &amp;quot;Expects args[0] to be float but the string is provided&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;For MobileNet we tried the code,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;try (Graph g = new Graph()) {&#xA;    GraphBuilder b = new GraphBuilder(g);&#xA;    // Some constants specific to the pre-trained model at:&#xA;    // https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip&#xA;    //&#xA;    // - The model was trained with images scaled to 224x224 pixels.&#xA;    // - The colors, represented as R, G, B in 1-byte each were converted to&#xA;    //   float using (value - Mean)/Scale.&#xA;    final int H = 224;&#xA;    final int W = 224;&#xA;    final float mean = 128f;&#xA;    final float scale = 1f;&#xA;    // Since the graph is being constructed once per execution here, we can use a constant for the&#xA;    // input image. If the graph were to be re-used for multiple input images, a placeholder would&#xA;    // have been more appropriate.&#xA;    final Output&amp;lt;String&amp;gt; input = b.constant(&amp;quot;input&amp;quot;, imageBytes);&#xA;    final Output&amp;lt;Float&amp;gt; output = b.div(&#xA;        b.sub(&#xA;            b.resizeBilinear(&#xA;                b.expandDims(&#xA;                    b.cast(b.decodeJpeg(input, 3), Float.class),&#xA;                    b.constant(&amp;quot;make_batch&amp;quot;, 0)),&#xA;                    b.constant(&amp;quot;size&amp;quot;, new int[] {H, W})),&#xA;                    b.constant(&amp;quot;mean&amp;quot;, mean)),&#xA;                    b.constant(&amp;quot;scale&amp;quot;, scale));&#xA;                    try (Session s = new Session(g)) {&#xA;                        return s.runner().fetch(output.op().name()).run().get(0).expect(Float.class);&#xA;                    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This works but gives the wrong labels.&lt;/p&gt;&#xA;" OwnerUserId="46317" LastEditorUserId="97640" LastEditDate="2021-07-18T18:04:59.180" LastActivityDate="2024-02-20T10:06:55.797" Title="Running Tensorflow MobileNet from Java" Tags="|tensorflow|java|inception|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="27859" PostTypeId="1" AcceptedAnswerId="27877" CreationDate="2018-02-15T18:16:11.770" Score="4" ViewCount="3427" Body="&lt;p&gt;I'm trying to build an neural net in Keras that would look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MWmXc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MWmXc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $x_1$, $x_2$, ... are input vectors that undergo the same transformation $f$. $f$ is itself a layer whose parameters must be learned. The sequence length $n$ is variable across instances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having trouble understanding two things here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;What should the input look like?&lt;/strong&gt;&lt;br&gt;&#xA;I'm thinking of a 2D tensor with shape (number_of_x_inputs, x_dimension), where x_dimension is the length of a single vector $x$. Can such 2D tensor have a variable shape? I know tensors can have variable shapes for batch processing, but I don't know if that helps me here.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;How do I pass each input vector through the same transformation before feeding it to the RNN layer?&lt;/strong&gt;&lt;br&gt;&#xA;Is there a way to sort of extend for example a GRU so that an $f$ layer is added before going through the actual GRU cell?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46319" LastEditorUserId="29575" LastEditDate="2018-02-15T18:55:35.363" LastActivityDate="2018-02-16T06:29:46.343" Title="How to build a recurrent neural net in Keras where each input goes through a layer first?" Tags="|keras|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27860" PostTypeId="1" AcceptedAnswerId="27868" CreationDate="2018-02-15T18:26:21.410" Score="0" ViewCount="34" Body="&lt;p&gt;I'm requesting data from a government body, and they asked me what format I want to receive the data in. This will be a table of about 400,000 rows and about 10 columns. My options are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;comma or tab delimited ASCII, Microsoft Access database, Microsoft Excel file etc&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They also want to know the media by which I want to receive data. My options are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;encrypted CD, DLT Tape, etc.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm used to working with JSON, and doing data analysis via Python and R. These data will have to remain confidential.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers?&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-02-15T20:45:21.343" Title="Looking for advice: data transfer" Tags="|dataset|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27862" PostTypeId="1" AcceptedAnswerId="27867" CreationDate="2018-02-15T19:27:20.987" Score="1" ViewCount="69" Body="&lt;p&gt;I'm trying to find out what I need to research and start learning to try and apply machine learning to this problem:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In multiple offices I have 20 chairs, all of these chairs will need to have a risk assessment carried out, as the chairs become older I want to understand when they become more at risk because of other factors such as; age, usage, last time assessed and areas level of risk. I then want to be able to come up with a simple risk score of each individual chair.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have done some research and I think it points towards using a machine learning technique such as supervised learning with a neural network but I'm uncertain. Given my scenario what should I be looking to start learning and doing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore I believe that from my limited knowledge that providing all of the values I'm using are the same across other items I could apply and scale out this model for other items?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Further info;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I'm familliar with Javascript I'm choosing to use synaptics.js to create a neural network and I'm going to be generating sample data inside a node server. I'm going to be using the following for my datasets that will be used as my model for learning from:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Age (days to years), Location(low, med, high), Office(low, med, high), Last Risk Assessment Date (days-&gt; years capped by twenty), Daily use (Low, Medium, High) and quality(low, med, high).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;All of the above will be given an expected outcome manually so that I can attempt to figure out a way for the neural network to process the above and come to the same conclusion I did.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the above labels I will be adding a weighting that will then be used to classify the risk level it should be associated to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The factors I'm looking for the neural network to understand is that given my factors a chair with a lot of high factors will likely be placed in a higher outcome value of say 1-4 for risk factors. I'm then hoping I can use this learning to be applied to multiple other things other than chairs and increase the labels the network can use to learn from as I go along.&lt;/p&gt;&#xA;" OwnerUserId="46327" LastEditorUserId="46327" LastEditDate="2018-02-15T21:27:22.470" LastActivityDate="2018-02-15T21:27:22.470" Title="Machine Learning to predict risk of items" Tags="|machine-learning|neural-network|unsupervised-learning|supervised-learning|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-18T13:57:25.507" ContentLicense="CC BY-SA 3.0" />
  <row Id="27866" PostTypeId="1" CreationDate="2018-02-15T20:20:38.277" Score="3" ViewCount="227" Body="&lt;p&gt;I've worked-through the back-propagation for the &lt;a href=&quot;https://arxiv.org/pdf/1610.09513.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Phased LSTM (Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu 2016)&lt;/a&gt; and would like to show the notes. It was a relatively difficult task, so I post it here, to help if anyone struggles with derivation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you know how to do the back-prop for the usual LSTM, and so will show only the most critical gradient-pieces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind the gradient flows in to opposite direction of the arrow.&#xA;The Phased LSTM uses Peepholes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, notice, the authors skip some timesteps by referring to the &quot;most recent update time&quot; as &lt;span class=&quot;math-container&quot;&gt;$t_j$&lt;/span&gt;  but I am using just &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt; for everything. That's because during training we need to consider every timestep - there is a &quot;leak&quot; in the time gate. However, as the paper advises, the leak is turned off at runtime, and you can then use &lt;span class=&quot;math-container&quot;&gt;$t_j$&lt;/span&gt; and skip any 'non-j' timesteps to save performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note - there still might be an error in the notes, although I've checked it thoroughly, and also via Gradient Check procedure. Still, please let me know if you spot one.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://math.stackexchange.com/questions/2651437/partial-of-modulo-operator-with-non-integers&quot;&gt;Here is a link regarding partial derivatives through the modulo operator&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tools to be used:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The quotient rule says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial}{\partial{a_i}} \left( \frac{g(a_i)}{h(a_i)} \right) = \frac{\frac{\partial{g(a_i)}}{\partial{a_i}}h(a_i) - g(a_i)\frac{\partial{h(a_i)}}{\partial{a_i}}}{h(a_i)^2}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the the Chain rule says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial}{\partial{a_i}} [ g(h(a_i))] = \frac{\partial{g(h(a_i))}}{\partial{h(a_i)}}\frac{\partial{h(a_i)}}{\partial{a_i}}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as a bonus,product rule says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\partial}{\partial{a_i}}g(a_i)h(a_i) = \frac{\partial{g(a_i)}}{\partial{a_i}} + \frac{\partial{h(a_i)}}{\partial{a_i}}$$&lt;/span&gt; &#xA;&lt;a href=&quot;https://i.stack.imgur.com/K9ay8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K9ay8.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Part #2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aR8E6.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aR8E6.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2020-03-11T01:30:11.820" LastActivityDate="2020-03-11T01:30:11.820" Title="Gradient derivation reference for Phased LSTM" Tags="|gradient-descent|lstm|" AnswerCount="0" CommentCount="6" ClosedDate="2018-02-18T13:58:26.933" ContentLicense="CC BY-SA 4.0" />
  <row Id="27867" PostTypeId="2" ParentId="27862" CreationDate="2018-02-15T20:41:00.127" Score="2" Body="&lt;p&gt;This sounds like a homework problem and those are frowned upon on this web site; so I'm going to be cautious about helping you and just give you general guidance:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - be very, very careful about picking models &lt;em&gt;before&lt;/em&gt; you do anything else. That is a bit of Researcher Bias and you should try to avoid that; let the data tell you where to go, not the other way around. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - you have to think about what &quot;factors&quot; you can use to describe each chair. How old is it? What color is it? Does it have wheels or not? Some of these features will have an effect on your models, others will not. But it should be your goal to start with a long list of factors and then try to model them out and see which ones have an effect on your results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Start putting a list of factors together and then you will be in a better position to continue your studies. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-15T20:41:00.127" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27868" PostTypeId="2" ParentId="27860" CreationDate="2018-02-15T20:45:21.343" Score="3" Body="&lt;p&gt;Either format will work with R and Python, though you'll need a library for MS Access, which is the least common option. If any of these formats is the native format of the data, I'd go for that, which will avoid weird transformation artefacts. If neither of the formats is, I'd choose CSV, which would allow me to use a text editor if needed. However, if one of the columns contains user generated texts, it might be worth the trouble to choose Excel, that would limit the chance that the export would get botched.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is way easier to get your hands on a CD-reader. Your data doesn't sound that impressively large that it warrants using tape.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-15T20:45:21.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27869" PostTypeId="2" ParentId="27840" CreationDate="2018-02-15T22:08:16.777" Score="1" Body="&lt;p&gt;You pose an interesting question. The problem that I see is that even if you develop all the items you mentioned (and I'm not sure you need all of them) you wouldn't have the computing power needed for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You see, most of these items are based on reinforcement learning. Which means that the models are given a relatively small set of data (roughly speaking, the rules of the game) and then are set off to play millions &amp;amp; millions of games. Read the whitepaper that was written by Google on beating Go. They basically admit that there whole point was to write something very, very simple that could learn on it's own and then just set it free on Google's massive power. It's nothing for them to say, &quot;let's have this robot play 10 million games&quot;. So the level of sophistication from your initial model is pretty low, you just need to make it an exceptional learner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you make an exceptional learner of Starcraft - then what? How are you going to have the power to make your algorithm to play millions of games? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-15T22:08:16.777" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27870" PostTypeId="1" CreationDate="2018-02-15T23:22:35.767" Score="1" ViewCount="4747" Body="&lt;p&gt;I don't understand the difference between these terms and google didn't help much. Please answer the question even if you have any slightest idea.&lt;/p&gt;&#xA;" OwnerUserId="46334" LastActivityDate="2018-11-29T08:39:46.993" Title="What is the difference between data singular and data plural ?" Tags="|statistics|data|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27871" PostTypeId="2" ParentId="27870" CreationDate="2018-02-15T23:35:00.293" Score="3" Body="&lt;p&gt;&lt;em&gt;Data&lt;/em&gt; is plural, as in 'These data are depicted below'. Singular is 'datum' but it's rare one would have occasion to use the term. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-02-15T23:35:00.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27872" PostTypeId="1" CreationDate="2018-02-16T01:44:31.977" Score="1" ViewCount="34" Body="&lt;p&gt;Suppose &lt;code&gt;a&lt;/code&gt; is a variable and I have a DSL (Domain Specific language) or a code snippet that says&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;a.size() &amp;gt; 0&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task is to convert it to simple English and print &lt;code&gt;Number of a's is greater than zero&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More examples:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input&lt;/strong&gt;:  &lt;code&gt;a != null&lt;/code&gt; , &lt;strong&gt;Output&lt;/strong&gt; &lt;code&gt;a is not null&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input&lt;/strong&gt;:  &lt;code&gt;a != &quot;&quot;&lt;/code&gt; , &lt;strong&gt;Output&lt;/strong&gt; &lt;code&gt;a is not empty&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input&lt;/strong&gt;: &lt;code&gt;a.parent.children.size()&amp;gt;3&lt;/code&gt;, &lt;strong&gt;Output&lt;/strong&gt;: &lt;code&gt;Parent of 'a' has more than 3 children&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input&lt;/strong&gt;: &lt;code&gt;a.children().forAll(e|e.value&amp;lt;4)&lt;/code&gt;, &lt;strong&gt;Output&lt;/strong&gt;: &lt;code&gt;All the children of 'a' have a value less than 4&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know what the right approach to this kind of problem would be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I basically want to translate the code snippet to simple English, similar to how humans can have a fair/high level understanding of what the code snippet is trying to do by just reading it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've considered the below approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;One approach that I've considered to to write something like a compiler or a parser.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another approach that I'm considering is to use machine translation, similar to how RNNs are used to translate one language to another. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;How would one approach this problem? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked the same question on SO &lt;a href=&quot;https://stackoverflow.com/questions/48814273/what-approach-is-to-be-taken-to-convert-a-dsl-to-simple-english&quot;&gt;here&lt;/a&gt; but I didn't get any response yet, so decided to ask here&lt;/p&gt;&#xA;" OwnerUserId="38520" LastEditorUserId="38520" LastEditDate="2018-02-16T06:37:34.323" LastActivityDate="2018-02-16T06:37:34.323" Title="What approach is to be taken to convert a code snippet to simple English?" Tags="|machine-learning|neural-network|nlp|" AnswerCount="0" CommentCount="3" ClosedDate="2018-02-18T09:47:28.400" ContentLicense="CC BY-SA 3.0" />
  <row Id="27876" PostTypeId="2" ParentId="27843" CreationDate="2018-02-16T06:01:32.110" Score="0" Body="&lt;p&gt;The problem seems to be that in the case of frequency parameter, the error function is not convex, and has multiple local minima of value greater than 0. The gradient descent is converging to one of these minima. The error will reduce to 0 only if the starting estimate lies in the right convex basin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the amplitude and center value parameters, the error functions are convex, and for phase parameter, there are multiple local minima all of value 0.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-02-16T06:01:32.110" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27877" PostTypeId="2" ParentId="27859" CreationDate="2018-02-16T06:29:46.343" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;What should the input look like?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;How do I pass each input vector through the same transformation before feeding it to the RNN layer?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Use &lt;code&gt;TimeDistributed&lt;/code&gt;. The example below passes all vectors $x_i$ through the same feed forward network (&lt;code&gt;Dense(5, ...)&lt;/code&gt;), but you should be able to swap that out for whatever &lt;code&gt;f&lt;/code&gt; you were thinking.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import LSTM, Dense, TimeDistributed&#xA;&#xA;x_dimension = 16&#xA;num_classes = 2&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(TimeDistributed(Dense(5, activation='relu'),&#xA;          input_shape=(None, x_dimension)))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(LSTM(8))&#xA;model.add(Dense(num_classes, activation='softmax'))&#xA;&#xA;print(model.summary(90))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This prints the following model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Layer (type)                            Output Shape                        Param #&#xA;==========================================================================================&#xA;time_distributed_1 (TimeDistributed)    (None, None, 5)                     85&#xA;__________________________________________________________________________________________&#xA;lstm_1 (LSTM)                           (None, None, 32)                    4864&#xA;__________________________________________________________________________________________&#xA;lstm_2 (LSTM)                           (None, 8)                           1312&#xA;__________________________________________________________________________________________&#xA;dense_2 (Dense)                         (None, 2)                           18&#xA;==========================================================================================&#xA;Total params: 6,279&#xA;Trainable params: 6,279&#xA;Non-trainable params: 0&#xA;__________________________________________________________________________________________&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-02-16T06:29:46.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27878" PostTypeId="2" ParentId="6676" CreationDate="2018-02-16T06:46:51.490" Score="4" Body="&lt;p&gt;SGDClassifier, as the name suggests, uses Stochastic Gradient descent as its optimization algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at the implementation of LogisiticRegression in Sklearn there are five optimization techniques(solver) provided and by default it is 'LibLinear' that uses Coordinate Descent(CD) to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than number of iterations, optimization, type of regularization(penalty) and its magnitude(C) also affect the performance of the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are running it on Iris data-set tuning all these hyper-parameters may not bring significant change but for complex data set they do play a meaningful role.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more, you can refer the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sklearn Logistic Regression Documentation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="46349" LastEditorUserId="29575" LastEditDate="2018-02-16T07:01:59.707" LastActivityDate="2018-02-16T07:01:59.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27879" PostTypeId="2" ParentId="26366" CreationDate="2018-02-16T06:54:06.697" Score="104" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;This suggests that all the training examples have a fixed sequence length, namely &lt;code&gt;timesteps&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is not quite correct, since that dimension can be &lt;code&gt;None&lt;/code&gt;, i.e. variable length. Within a single &lt;em&gt;batch&lt;/em&gt;, you must have the same number of timesteps (this is typically where you see 0-padding and masking). But between batches there is no such restriction. During inference, you can have any length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example code that creates random time-length batches of training data.&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import LSTM, Dense, TimeDistributed&#xA;from keras.utils import to_categorical&#xA;import numpy as np&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))&#xA;model.add(LSTM(8, return_sequences=True))&#xA;model.add(TimeDistributed(Dense(2, activation='sigmoid')))&#xA;&#xA;print(model.summary(90))&#xA;&#xA;model.compile(loss='categorical_crossentropy',&#xA;              optimizer='adam')&#xA;&#xA;def train_generator():&#xA;    while True:&#xA;        sequence_length = np.random.randint(10, 100)&#xA;        x_train = np.random.random((1000, sequence_length, 5))&#xA;        # y_train will depend on past 5 timesteps of x&#xA;        y_train = x_train[:, :, 0]&#xA;        for i in range(1, 5):&#xA;            y_train[:, i:] += x_train[:, :-i, i]&#xA;        y_train = to_categorical(y_train &amp;gt; 2.5)&#xA;        yield x_train, y_train&#xA;&#xA;model.fit_generator(train_generator(), steps_per_epoch=30, epochs=10, verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is what it prints. Note the output shapes are &lt;code&gt;(None, None, x)&lt;/code&gt; indicating variable batch size and variable timestep size.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;__________________________________________________________________________________________&#xA;Layer (type)                            Output Shape                        Param #&#xA;==========================================================================================&#xA;lstm_1 (LSTM)                           (None, None, 32)                    4864&#xA;__________________________________________________________________________________________&#xA;lstm_2 (LSTM)                           (None, None, 8)                     1312&#xA;__________________________________________________________________________________________&#xA;time_distributed_1 (TimeDistributed)    (None, None, 2)                     18&#xA;==========================================================================================&#xA;Total params: 6,194&#xA;Trainable params: 6,194&#xA;Non-trainable params: 0&#xA;__________________________________________________________________________________________&#xA;Epoch 1/10&#xA;30/30 [==============================] - 6s 201ms/step - loss: 0.6913&#xA;Epoch 2/10&#xA;30/30 [==============================] - 4s 137ms/step - loss: 0.6738&#xA;...&#xA;Epoch 9/10&#xA;30/30 [==============================] - 4s 136ms/step - loss: 0.1643&#xA;Epoch 10/10&#xA;30/30 [==============================] - 4s 142ms/step - loss: 0.1441&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="33202" LastEditDate="2019-09-06T01:15:11.060" LastActivityDate="2019-09-06T01:15:11.060" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="27880" PostTypeId="2" ParentId="27777" CreationDate="2018-02-16T07:07:32.850" Score="1" Body="&lt;p&gt;It is only outputting 1 because &lt;code&gt;softmax&lt;/code&gt; makes no sense when given a single input. Softmax computed with only one input $x$ is equivalent to $\frac{e^x}{e^x} = 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, the line&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(1, activation='softmax'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;is wrong. Use &lt;code&gt;sigmoid&lt;/code&gt; instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(1, activation='sigmoid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or read more about one-hot encoding your expected output and the helper function &lt;a href=&quot;https://keras.io/utils/#to_categorical&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/utils/#to_categorical&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="33202" LastEditDate="2018-02-16T16:13:51.867" LastActivityDate="2018-02-16T16:13:51.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27881" PostTypeId="1" CreationDate="2018-02-16T07:20:51.617" Score="9" ViewCount="4482" Body="&lt;p&gt;We need to implement a time series problem with the LSTM model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, while implementing the same, the main challenge I am facing is the feature selection issue. Because our data-set contains 2300 observations and 600 features. And we already know there are so many features which are not at all relevant. But, no one has the domain expertise to confirm which feature is relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried with…&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;All 600 features, [shape of X become, 2280 X 20 X 600 (if time step is 20)]. The accuracy is very poor 53-55%.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Considered a single column as a predictor (the feature which we want to predict) [shape of X become, 2280 X 20 X 1 (if the time step is 20)]. The accuracy is also poor.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Also, tried with PCA so that all 600 features are reduced to 20 features [approx 98% data was retained], but not seen any improvement.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally, I have gone through a very lengthy process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. Out of 600 features, each of them is tried out individually with the LSTM algorithm. The feature which was giving the best result is selected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. The selected feature is paired with the rest of 599 features individually. Then, I select the best pair that gives the best output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. The best pair is further tried with rest 598 features individually and this process continues...&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;After doing so, as of now, we get 6 features, the accuracy gets improved to 65%.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;My question is, is it the only approach that I can do feature selection in time series problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of any non-time series problem there are so many ways to do the feature selection. What will be the best way to do feature selection in the time series issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's give another example...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume we developed the LSTM model with 10 predictors/features. The model is working very fine and we get 80% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, somehow 100 more features get added which are not relevant for deriving target y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the accuracy will be dropped as the model will focus on 120 predictors/features, where 100 are not relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, how will we eliminate the 100 unwanted features?&lt;/p&gt;&#xA;" OwnerUserId="46348" LastEditorUserId="83275" LastEditDate="2020-01-24T10:57:26.103" LastActivityDate="2023-02-18T18:35:18.987" Title="LSTM Feature selection process" Tags="|time-series|feature-selection|lstm|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27885" PostTypeId="2" ParentId="27870" CreationDate="2018-02-16T10:45:03.547" Score="0" Body="&lt;p&gt;&lt;em&gt;Datum&lt;/em&gt; is latin for &lt;a href=&quot;https://en.wiktionary.org/wiki/datum#English&quot; rel=&quot;nofollow noreferrer&quot;&gt;'a given'&lt;/a&gt;, and &lt;em&gt;data&lt;/em&gt; is the plural, meaning 'the givens.'&#xA;If you want to be precise, you should refer to a single observation as a &lt;em&gt;datum&lt;/em&gt; and multiple observations as &lt;em&gt;data&lt;/em&gt;. Notice that the latter is a plural word: 'The data &lt;em&gt;are&lt;/em&gt; [not is] hard to interpret.'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But usage has changed. People, especially in Data Science, usually seem to talk about 'a single data point' and say that 'the data is hard to interpret.'&#xA;See &lt;a href=&quot;https://www.theguardian.com/news/datablog/2010/jul/16/data-plural-singular&quot; rel=&quot;nofollow noreferrer&quot;&gt;this discussion&lt;/a&gt; for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation: As long as you are talking to Data Scientists and IT guys, ban &lt;em&gt;datum&lt;/em&gt; from your vocabulary and use &lt;em&gt;data&lt;/em&gt; with the singular.&#xA;Reconsider once you are a professor in Oxford.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-16T10:45:03.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27888" PostTypeId="1" AcceptedAnswerId="27891" CreationDate="2018-02-16T11:09:56.917" Score="9" ViewCount="1796" Body="&lt;p&gt;I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Structure or format of my data is as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;----------------------------------------------------------------------------------------&#xA;| product_title          | Key_value_pairs                               | taxonomy_id |&#xA;----------------------------------------------------------------------------------------&#xA;  Samsung S7 Edge        | Color:black,Display Size:5.5 inch,Internal    | 211 &#xA;                          Storage:128 GB, RAM:4 GB,Primary Camera:12 MP  &#xA;&#xA;  Case cover Honor 8     | Color:transparent,Height:15 mm,width:22 mm    | 212 &#xA;&#xA;  Ruggers Men's T-Shirt  | Size:L,ideal for:men,fit:regular,             | 111&#xA;                          sleeve:half sleeve&#xA;&#xA;  Optimum Nutrition Gold | Flavor:chocolate,form:powder,size:34 gm       | 311&#xA;  Standard Whey Protein  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Data distribution is not normal; it is highly imbalanced:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-------------------------&#xA;| taxonomy_id |   count |&#xA;-------------------------&#xA;          111 |  851750 &#xA;          112 |  355592&#xA;          113 |  379433&#xA;          114 |   23138&#xA;          115 |  117735&#xA;          116 |  145757&#xA;          117 | 1339471&#xA;          121 |  394026&#xA;          122 |  193433&#xA;          123 |   78299&#xA;          124 |  111962&#xA;          131 |    1776&#xA;          132 |    4425&#xA;          133 |     908&#xA;          134 |   23062&#xA;          141 |   22713&#xA;          142 |   42073&#xA;          211 |    7892&#xA;          212 | 1574744&#xA;          221 |    1047&#xA;          222 |  397515&#xA;          223 |   53009&#xA;          231 |    1227&#xA;          232 |    7683&#xA;          251 |     739&#xA;          252 |     327&#xA;          253 |   38974&#xA;          254 |      25&#xA;          311 |    2901&#xA;          321 |    7126&#xA;          412 |     856&#xA;          421 |  697802&#xA;          422 |  414855&#xA;          423 |   17750&#xA;          425 |    1240&#xA;          427 |     658&#xA;          429 |    1058&#xA;          431 |   20760&#xA;          441 |     257       &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see they are highly imbalanced and leading to mis-classifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Steps I have performed till now&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1) Merge product_title and key_value_pairs column and remove stop words and special characters and perform stemming.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2) I have used pipeline for TFIDFvectorizer(), LinearSVC()&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vectorizerPipe = Pipeline([&#xA;                 ('tfidf', TfidfVectorizer(lowercase=True, stop_words='english')),&#xA;                 ('classification', OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge'))),&#xA;                 ])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this I have fit pipeline and stored the classifier in pickle&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prd = vectorizerPipe.fit(df.loc[:, 'description'], df.loc[:, 'taxonomy_id'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;On Testing side I have repeated step 1 as mentioned above and then load the pickle and use predict function&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd = cl.predict([testData])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Issues I am facing &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;A lot of products are being mis-classified into some other categories&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: Ultimate Nutrition Prostar 100% Whey Protein should be classified into category 311 but my classifier is classifying it as 222 which is completely wrong.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am not sure whether to use TFidfVectorizer() or Hashingvectorizer(), can you guys help me in selecting one of this along with  their parameters?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Algorithm I am using is LinearSVC, is it a good choice for multi-class classification problems with large amount of data? Or should I use different algorithms?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As my data is highly imbalanced I tried random undersampling. The results were improved but  they were still not up to the mark.&#xA;Also I am not sure whether this is the right approach to perform random undersampling:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipe = make_pipeline_imb(&#xA;    HashingVectorizer(lowercase=True),&#xA;    RandomUnderSampler(ratio={111: 405805, 112: 170431, 113: 241709, 114: 8341, 115: 50328, 116: 89445, 117: 650020, 121: 320803, 122: 162557, 123: 66156, 124: 36276, 131: 1196, 132: 3365, 133: 818, 134: 15001, 141: 6145, 142: 31783, 211: 24728, 212: 100000, 221: 791, 222: 8000, 223: 35406, 231: 785, 232: 3000, 251: 477, 252: 127, 253: 29563, 254: 33, 311: 2072, 321: 5370, 412: 652, 421: 520973, 422: 99171, 423: 16786, 425: 730, 427: 198, 429: 1249, 431: 13793, 441: 160},random_state=1), &#xA;    OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge')))&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I am new in machine learning so I have used this approach for text classification. If my approach is wrong then please correct me with right one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(It would be great if you give suggestion or solution with examples as it will help me understand better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;***EDIT-1****&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;RndmFrst = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)&#xA;LogReg = LogisticRegression()&#xA;voting = VotingClassifier(estimators=[('LogReg ', LogReg), ('RndmFrst', RndmFrst)], voting='soft', n_jobs=-1)&#xA;&#xA;pipe = Pipeline([('tfidf', TfidfVectorizer(ngram_range=(1,4), max_features=50000)), ('clf', voting)])&#xA;&#xA;pipe = pipe.fit(df.loc[:,'description'], df.loc[:,'taxonomy_id'])&#xA;Preds = pipe.predict(test_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46356" LastEditorUserId="69822" LastEditDate="2019-03-19T17:00:47.510" LastActivityDate="2019-03-20T11:48:12.147" Title="Imbalanced data causing mis-classification on multiclass dataset" Tags="|machine-learning|python|classification|scikit-learn|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27889" PostTypeId="2" ParentId="27840" CreationDate="2018-02-16T11:10:43.513" Score="0" Body="&lt;p&gt;This answer is based on my opinion and others', it might not be the answer you expect.&lt;/p&gt;&#xA;&lt;p&gt;The problem you want to solve is probably in the field of &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_general_intelligence&quot; rel=&quot;nofollow noreferrer&quot;&gt;Artificial General Intelligence&lt;/a&gt;. The problems require AGI to solve are informally known as &lt;a href=&quot;https://en.wikipedia.org/wiki/AI-complete&quot; rel=&quot;nofollow noreferrer&quot;&gt;AI-complete&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;AI-complete problems are hypothesised to include computer vision, natural language understanding, and dealing with unexpected circumstances while solving any real world problem.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Currently, we're not ready to solve these problems, and if Gartner is right, the hype won't come to peak in 10 years.&lt;/p&gt;&#xA;&lt;p&gt;This is the Gartner hype curve in 2017, and AGI is still in its early phase.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qb3ei.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qb3ei.png&quot; alt=&quot;This is the Gartner hype curve in 2017, and AGI is still in its early phase&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;The problem is, at the moment, our state-of-the-art AI models (Deep Learning) don't have the ability to explain their decision or reason about their behavior, which makes learning highly inefficient.&lt;/p&gt;&#xA;&lt;p&gt;In addition, the Reinforcement Learning framework is way too simple and it ignores the complexity of human environment. You might find &lt;a href=&quot;https://www.quora.com/Why-does-a-reinforcement-learning-agent-need-a-discount-factor/answer/Sridhar-Mahadevan-6?share=547cdf09&amp;amp;srid=TC9Z&quot; rel=&quot;nofollow noreferrer&quot;&gt;this answer&lt;/a&gt; useful.&lt;/p&gt;&#xA;&lt;p&gt;Geoffrey Hinton, one of the leaders of the field, said in an interview that &lt;a href=&quot;http://www.i-programmer.info/news/105-artificial-intelligence/11135--geoffrey-hinton-says-ai-needs-to-start-over.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AI needed to start over&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;My view is throw it all away and start again,&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;I don't think it's how the brain works. We clearly don't need all the labeled data.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;That's it. If one day AI can learn how to play Starcraft 2, it probably learns in a very different way from what it learns today.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-16T11:10:43.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27890" PostTypeId="1" CreationDate="2018-02-16T12:00:05.603" Score="1" ViewCount="1023" Body="&lt;p&gt;I am a newbie in Machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found this example using &lt;code&gt;tflearn&lt;/code&gt; somewhere. It is the part of the program where we initialize a dummy model before training it. In the example, there were 4 input layers and 2 output layers. I didn't understand any terms but I ran that program and it worked. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def neural_network_model(input_size):&#xA;    network = input_data(shape=[None, input_size, 1], name='input')&#xA;&#xA;    network = fully_connected(network, 128, activation='relu')&#xA;    network = dropout(network, 0.8)&#xA;&#xA;    network = fully_connected(network, 256, activation='relu')&#xA;    network = dropout(network, 0.8)&#xA;&#xA;    network = fully_connected(network, 512, activation='relu')&#xA;    network = dropout(network, 0.8)&#xA;&#xA;    network = fully_connected(network, 256, activation='relu')&#xA;    network = dropout(network, 0.8)&#xA;&#xA;    network = fully_connected(network, 128, activation='relu')&#xA;    network = dropout(network, 0.8)&#xA;&#xA;    network = fully_connected(network, 2, activation='softmax')&#xA;    network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')&#xA;    model = tflearn.DNN(network, tensorboard_dir='log')&#xA;&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I wanted to reuse this function to create a model that I will train with a data having 24 input layers and 4 output layers but it gave me the following error: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Cannot feed value of shape (64, 4) for Tensor 'targets/Y:0', which has&#xA;  shape '(?, 2)'&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone explain me what does all of the term above mean ? And what should I change to adapt with my 24 by 4 layer?&lt;/p&gt;&#xA;" OwnerUserId="25139" LastEditorUserId="28175" LastEditDate="2018-02-16T12:20:49.113" LastActivityDate="2021-04-08T14:03:20.413" Title="How to create a dummy model in Tensorflow" Tags="|machine-learning|neural-network|deep-learning|tensorflow|tflearn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27891" PostTypeId="2" ParentId="27888" CreationDate="2018-02-16T12:19:05.650" Score="6" Body="&lt;p&gt;Nice question!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Some Remarks&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For imbalanced data you have different approaches. Most well-established one is &lt;a href=&quot;https://datascience.stackexchange.com/questions/27671/how-do-you-apply-smote-on-text-classification/27758#27758&quot;&gt;resampling&lt;/a&gt; (Oversampling small classes /underssampling large classes). The other one is to make your classification hierarchical i.e. classify large classes against all others and then classify small classes in second step (classifiers are not supposed to be the same. Try model selection strategies to find the best).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Practical Answer&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have got acceptable results without resampling the data! So try it but later improve it using resampling methods (statistically they are kind of A MUST).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TFIDF is good for such a problem. Classifiers should be selected through model selection but my experience shows that Logistic Regression and Random Forest work well on this specific problem (however it's just a practical experience).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may follow the code bellow as it worked simply well then you may try modifying it to improve your results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = pd.read_csv(...)&#xA;test = pd.read_csv(...)    &#xA;&#xA;# TFIDF Bag Of Words Model For Text Curpos. Up to 4-grams and 50k Features&#xA;vec = TfidfVectorizer(ngram_range=(1,4), max_features=50000)&#xA;TrainX = vec.fit_transform(train)&#xA;TestX = vec.transform(test)&#xA;&#xA;&#xA;# Initializing Base Estimators&#xA;clf1 = LogisticRegression()&#xA;clf2 = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)&#xA;&#xA;# Soft Voting Classifier For Each Column&#xA;clf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)], voting='soft', n_jobs=-1)&#xA;clf = clf.fit(TrainX, TrainY)&#xA;preds = clf.predict_proba(TestX)[:,1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note that the code is abstract so TianX, TrainY,TestX,etc should be properly defined by you.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Hints&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Be careful about what is StopWord. Practically many people (including myself!) made this mistake to remove stop words according to pre-defined lists. That is not right!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stop words are corpus-sensitive so You need to remove stopwords according to information theoretic concepts (to keep it simple you need to know TFIDF kind of ignores your corpus-specific stopwords. If you need more explanation please let me know to update my answer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;VotingClassifier is a meta-learning strategy in the family of &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensemble Methods&lt;/a&gt;. They take benefit from different classifiers. Try them as they work pretty well in practice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Voting schema simply takes the results of different classifiers and return the output of the one which has the highest probability to be right. So kind of democratic approach against dictatorship ;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="69822" LastEditDate="2019-03-20T11:48:12.147" LastActivityDate="2019-03-20T11:48:12.147" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="27892" PostTypeId="2" ParentId="27890" CreationDate="2018-02-16T12:19:49.973" Score="1" Body="&lt;p&gt;The reason is that in your model, you have specified the input size to vary, as the parameter of your function specifies. You have to change your code as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def neural_network_model(input_size, output_size):&#xA;network = input_data(shape=[None, input_size, 1], name='input')&#xA;&#xA;network = fully_connected(network, 128, activation='relu')&#xA;network = dropout(network, 0.8)&#xA;&#xA;network = fully_connected(network, 256, activation='relu')&#xA;network = dropout(network, 0.8)&#xA;&#xA;network = fully_connected(network, 512, activation='relu')&#xA;network = dropout(network, 0.8)&#xA;&#xA;network = fully_connected(network, 256, activation='relu')&#xA;network = dropout(network, 0.8)&#xA;&#xA;network = fully_connected(network, 128, activation='relu')&#xA;network = dropout(network, 0.8)&#xA;&#xA;network = fully_connected(network, output_size, activation='softmax')&#xA;network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')&#xA;model = tflearn.DNN(network, tensorboard_dir='log')&#xA;&#xA;return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You have hard coded the output size, which is two, you have to change it to be a variable and during calling, you have to specify the size of your new output, which is four.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-16T12:19:49.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27893" PostTypeId="1" AcceptedAnswerId="27905" CreationDate="2018-02-16T13:08:43.983" Score="3" ViewCount="985" Body="&lt;p&gt;Deep Q-learning, A3C, policies evolved with genetic algorithms, they all fail to learn Asteroids, or at least perform way worse than human. From the hardest Atari games according to RL most of the focus is on Montezuma's revenge, which clearly suffers from sparse rewards. However I don't think this is the case of Asteroids (&lt;a href=&quot;https://www.youtube.com/watch?v=9Ydu8UhIjeU&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;), since for every asteroid shot a reward is provided. Why DRL performs that bad then?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some papers that report a bad result on Asteroids (some articles refer to each other):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.cs.swarthmore.edu/~meeden/cs63/s15/nature15b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Human-level control through deep reinforcement&#xA;learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1507.04296.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Massively Parallel Methods for Deep Reinforcement Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1509.06461.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Reinforcement Learning with Double Q-learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1511.06581.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dueling Network Architectures for Deep Reinforcement Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1511.05952.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prioritized Experience Replay&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1710.02298.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rainbow: Combining Improvements in Deep Reinforcement Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1703.03864.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Evolution Strategies as a Scalable Alternative to Reinforcement Learning&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.utexas.edu/users/pstone/Papers/bib2html-links/TCIAIG13-mhauskn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Neuroevolution Approach to General Atari Game Playing&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="17484" LastEditorUserId="17484" LastEditDate="2018-02-19T10:31:08.960" LastActivityDate="2020-08-15T18:49:22.557" Title="Why Deep Reinforcement Learning fails to learn how to play Asteroids?" Tags="|deep-learning|reinforcement-learning|cnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27894" PostTypeId="1" AcceptedAnswerId="27896" CreationDate="2018-02-16T14:38:42.480" Score="1" ViewCount="496" Body="&lt;p&gt;I use keras for training an image classification problem as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;datagen = ImageDataGenerator(&#xA;    featurewise_center=False,&#xA;    featurewise_std_normalization=False,&#xA;    rotation_range=20,&#xA;    width_shift_range=0.2,&#xA;    height_shift_range=0.2,&#xA;    horizontal_flip=True)&#xA;&#xA;# compute quantities required for featurewise normalization&#xA;# (std, mean, and principal components if ZCA whitening is applied)&#xA;datagen.fit(x_train)&#xA;&#xA;&#xA;# alternative to model.fit_generator&#xA;for e in range(epochs):&#xA;    print('Epoch', e)&#xA;    batches = 0&#xA;    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=32):&#xA;        model.fit(x_batch, y_batch)&#xA;        batches += 1&#xA;        if batches &amp;gt;= len(x_train) / 32:&#xA;            # we need to break the loop by hand because&#xA;            # the generator loops indefinitely&#xA;            break&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to know if &lt;code&gt;datagen.fit&lt;/code&gt; is required if I do not apply any data normalization?&lt;/p&gt;&#xA;" OwnerUserId="45898" LastEditorUserId="28175" LastEditDate="2018-02-16T15:55:02.847" LastActivityDate="2018-02-16T15:55:02.847" Title="If I do not do any data normailization, is datagen.fit required in Keras?" Tags="|machine-learning|python|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27895" PostTypeId="1" CreationDate="2018-02-16T14:56:39.837" Score="2" ViewCount="388" Body="&lt;p&gt;I have 2000+ training images(of custom farm structures) and I want to define my own model using Tensorflow. I researched (&lt;a href=&quot;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/defining_your_own_model.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/defining_your_own_model.md&lt;/a&gt;). This is a very high-level documentation. Can anyone help me on how to go about doing this? &lt;/p&gt;&#xA;" OwnerUserId="46371" LastActivityDate="2018-02-16T14:56:39.837" Title="How to define own model using Tensorflow object detection API" Tags="|machine-learning|python|deep-learning|keras|tensorflow|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-18T08:04:58.687" ContentLicense="CC BY-SA 3.0" />
  <row Id="27896" PostTypeId="2" ParentId="27894" CreationDate="2018-02-16T15:06:19.263" Score="1" Body="&lt;p&gt;According to the comment of the &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; in the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# compute quantities required for featurewise normalization&#xA;# (std, mean, and principal components if ZCA whitening is applied)&#xA;datagen.fit(x_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It does the normalization, reducing mean and dividing by standard deviation, and more things like &lt;code&gt;PCA&lt;/code&gt;. So it seems that you don't need to do normalization. That method does that, and normalizing features is required for accelerating training process and caring about all features with different scales the same.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-16T15:31:50.987" LastActivityDate="2018-02-16T15:31:50.987" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27897" PostTypeId="2" ParentId="27714" CreationDate="2018-02-16T15:10:31.920" Score="1" Body="&lt;p&gt;As far as i know, saved model contains the &lt;strong&gt;parameters&lt;/strong&gt; of the &lt;strong&gt;trained&lt;/strong&gt; model. This means, that: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;No, it does not contain the algorithm itself, however it contains all the parameters to run it.&lt;/li&gt;&#xA;&lt;li&gt;The values of these parameters definitely depend on the used training set.&lt;/li&gt;&#xA;&lt;li&gt;If later loaded, can be used with another data set, which contains &quot;&lt;a href=&quot;https://docs.orange.biolab.si/3/visual-programming/widgets/model/savemodel.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;compatible attributes&lt;/a&gt;&quot;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35817" LastActivityDate="2018-02-16T15:10:31.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27898" PostTypeId="1" AcceptedAnswerId="27902" CreationDate="2018-02-16T15:43:22.477" Score="3" ViewCount="652" Body="&lt;p&gt;The convergence of the &quot;simple&quot; &lt;a href=&quot;https://en.wikipedia.org/wiki/Perceptron#Convergence&quot; rel=&quot;nofollow noreferrer&quot;&gt;perceptron&lt;/a&gt; says that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$k\leqslant \left ( \frac{R\left \| \bar{\theta} \right \|}{\gamma } \right )^{2}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $k$ is the number of iterations (in which the weights are updated), $R$ is the maximum distance of a sample from the origin, $\bar{\theta}$ is the final weights vector, and  $\gamma$ is the smallest distance from $\bar{\theta}$ to a sample (= the margin of hyperplane).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many books implicitly say that  $\left \| \bar{\theta} \right \|$ is equal to 1. But why do they normalize it ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="85045" LastEditDate="2021-09-15T07:42:58.267" LastActivityDate="2021-09-15T07:42:58.267" Title="Normalizing the final weights vector in the upper bound on the Perceptron's convergence" Tags="|machine-learning|perceptron|convergence|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27899" PostTypeId="2" ParentId="19105" CreationDate="2018-02-16T15:43:53.873" Score="1" Body="&lt;p&gt;There are a couple of implicit assumptions that are necessary to make the argument reasonable:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Observations should be independent and identically distributed. Otherwise, almost anything could happen.&lt;/li&gt;&#xA;&lt;li&gt;The loss function should be of the form $L(Y, Z) = \frac1N \sum_{i=1}^N l(Y_i, Z_i)$ for some function $l$, where $Z_i = f(\mathbf{X}; \beta)(\mathbf{X}_i)$ is the prediction for $\mathbf{X}_i$ after being trained on $\mathbf{X}$. A counterexample would be $L(Y, Z) = \max_i\{|Y_i-Z_i|\}.$ The individual maxima from a leave-one-out procedure would not be an unbiased estimator for the maximum over the entire sample.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But even then, your claim is not always correct. Here is a counterexample. It has no practical value, but perhaps it is a good illustration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose $\mathbf{X}$ is $(N \times 1)$-dimensional and write $\mathbf{X}_i = X_i.$ To illustrate the point, suppose further that $Y_i = X_i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider the following model $f$, given a test value $X_j$ and a training set $\tilde{\mathbf{X}}$: If $X_j$ is in the test set $\tilde{\mathbf{X}}$, predict $Y_j = X_j.$ Otherwise, predict $Y_j = 42.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I evaluate $L(Y, f(\mathbf{X}))$, I get a perfect fit and therefore zero loss because all $Y_i = X_i$ are in the test set $\mathbf{X}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I evaluate $L(Y_j, f(\mathbf{X}_{-j}))$, I generally get a nonzero loss, because my model always predicts $42$.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-16T15:43:53.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27900" PostTypeId="1" AcceptedAnswerId="28230" CreationDate="2018-02-16T15:44:34.403" Score="3" ViewCount="734" Body="&lt;p&gt;For a nominal categorical variable that has two levels, e.g. Gender (levels = Male,Female), is it feasible to use label encoding instead of One Hot encoding ? If it is, are there any implications of using one encoding method over the other, for such a categorical variable ?&lt;/p&gt;&#xA;" OwnerUserId="46374" LastEditorUserId="46374" LastEditDate="2018-02-23T10:15:58.540" LastActivityDate="2018-02-23T11:34:32.290" Title="Nominal categorical variable with two levels: Label Encoding or One Hot encoding?" Tags="|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27902" PostTypeId="2" ParentId="27898" CreationDate="2018-02-16T15:56:25.160" Score="2" Body="&lt;p&gt;The Perceptron's output $f$ is&#xA;$$&#xA;f(\overline\theta \cdot \overline{x}) = \begin{cases} 1 &amp;amp;\text{ if } \overline{\theta }\cdot \overline{x} &amp;gt; 0 \\ 0 &amp;amp;\text{ if } \overline{\theta }\cdot \overline{x} \le 0\end{cases}&#xA;$$&#xA;Here, $\overline{x} = (1, x_1, \dots, x_n)$ where $(x_1, \dots, x_n)$ is the input vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see that the output only depends on the &lt;em&gt;sign&lt;/em&gt; of the product $\overline{\theta }\cdot \overline{x}.$&#xA;Therefore the output will not change if we multiply $\overline\theta \cdot \overline{x}$ with a positive constant:&#xA;$$&#xA;f(\overline\theta \cdot \overline{x}) = f(c\times \overline\theta \cdot \overline{x}) \qquad \text{ for all } c &amp;gt; 0.&#xA;$$&#xA;In particular, we may choose $c = 1/\lVert \overline\theta \rVert.$ So we get&#xA;$&#xA;f(\overline{\theta }\cdot \overline{x}) = f(\tilde\theta\cdot \overline{x}),&#xA;$&#xA;where $\tilde\theta = \overline\theta / \lVert \overline\theta \rVert$ is the normalized vector.&#xA;Conclude that there is no loss of generality in assuming $\lVert \overline\theta \rVert = 1.$&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-18T07:29:54.717" LastActivityDate="2018-02-18T07:29:54.717" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27903" PostTypeId="1" CreationDate="2018-02-16T16:17:49.943" Score="2" ViewCount="1116" Body="&lt;p&gt;In a multilabel setting a training example could be &lt;code&gt;a, b, (a, b), d, c, (d, c)&lt;/code&gt;, etc. This makes it a bit hard to come up with a helpful confusion matrix because the number of columns or rows could be very large - as I understand it, you wouldn't have a column or row for just &lt;code&gt;a&lt;/code&gt; or &lt;code&gt;b&lt;/code&gt;, but also &lt;code&gt;(a, b)&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do people usually do in these cases? Do they usually create a column or row for every possible combination, or do they simplify somehow? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you'd like to provide code I am using Weka, but my question is primarily about the best practice.&lt;/p&gt;&#xA;" OwnerUserId="44394" LastActivityDate="2018-02-16T16:17:49.943" Title="what's the best way to plot a confusion matrix in a multilabel setting?" Tags="|confusion-matrix|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27905" PostTypeId="2" ParentId="27893" CreationDate="2018-02-16T16:54:06.487" Score="4" Body="&lt;p&gt;I suspect a bug, or some subtle implementation detail.&lt;/p&gt;&#xA;&lt;p&gt;In many ways, Asteroids is as near an ideal environment as one can get without custom design:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unambiguous, simple reward system (get points, don't die)&lt;/li&gt;&#xA;&lt;li&gt;limited options (move or shoot a single weapon type)&lt;/li&gt;&#xA;&lt;li&gt;all enemies are very similar&lt;/li&gt;&#xA;&lt;li&gt;The player has perfect information.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Compare this to DOTA 2, which has not been mastered, but with a moderate reduction in complexity (1v1 instead of 5v5), OpenAI was able to achieve some impressive results, despite being magnitudes of order more complex than asteroids.&lt;/p&gt;&#xA;&lt;p&gt;There are certain compromises made in the 2015 DQN paper, for example:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;Following previous approaches to playing Atari2600 games,we also use&#xA;a simple frame-skipping technique (15). More precisely, the agent sees&#xA;and selects actions on every kth frame instead of every frame, and its&#xA;last action is repeated on skipped frames. Because running the&#xA;emulator forward for one step requires much less computation than&#xA;having the agent select an action, this technique allows the agent to&#xA;play roughly k times more games without significantly increasing the&#xA;runtime.&lt;/p&gt;&#xA;&lt;p&gt;We use k - 4 for all games&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;As a counter-example, top level human players often make single frame level decisions / actions in modern fighting games (rendered both visually and in game logic at 60 FPS), so we know this approach will not work well with all video games.&lt;/p&gt;&#xA;&lt;p&gt;I suspect that even though these were effective generalized approaches, one of them severely failed with asteroids.&lt;/p&gt;&#xA;&lt;p&gt;This is a good question, as diagnosing why asteroids is causing difficulty could give some strong insight into testing procedures or algorithm design, as clearly something is happening. I think to get an accurate answer would require fixing the problem.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.swarthmore.edu/%7Emeeden/cs63/s15/nature15b.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sources&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46277" LastEditorUserId="98307" LastEditDate="2020-08-15T18:49:22.557" LastActivityDate="2020-08-15T18:49:22.557" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27908" PostTypeId="1" CreationDate="2018-02-16T17:14:46.787" Score="2" ViewCount="89" Body="&lt;p&gt;I am building a program to implement a solution for multilabel classification. It's an interesting problem because most of the examples (~90%) actually only have one class, but some have multiple classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally there are certain classes which we know to be mutually exclusive, i.e. if an example is in class A then it cannot be in class B, and vice-versa.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way it's currently implemented is to identify all the classes which are mutually exclusive and place them within their own classifier, then build a separate classifier for each of the others:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Classifier 1: Predicts A (or OTHER)&#xA;Classifier 2: Predicts B (or OTHER)&#xA;Classifier 3: Predicts C,D,E, or F (or OTHER)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When the classifier runs, it gets predictions from all 3 classifiers and then combines them. So if classifier 1 predicts A, and classifier 2 and 3 both predict OTHER, then the result would be A. If instead classifier 2 had predicted B, then the result would be A,B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Does this seem like a reasonable implementation to you?&lt;/li&gt;&#xA;&lt;li&gt;It seems like the downside is that we could end up with &lt;code&gt;A and B&lt;/code&gt; too many times when really we wanted them to be exclusive more often. In that case we could try to improve things with a classifier that controls the degree of exclusivity by predicting &lt;code&gt;A, B, A-and-B&lt;/code&gt; instead. But is that likely to represent an improvement, or would it be unnecessary? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In an ideal world if the A and B classifiers are functioning correctly I think it would be unnecessary, but I don't know if combining the decision into a single classifier would allow the classifier to learn the boundaries &quot;better&quot; in some way.&lt;/p&gt;&#xA;" OwnerUserId="44394" LastActivityDate="2018-02-16T23:29:41.657" Title="multilabel program: separate classifiers vs combined classifiers" Tags="|multilabel-classification|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="27909" PostTypeId="1" CreationDate="2018-02-16T17:29:27.450" Score="3" ViewCount="2804" Body="&lt;p&gt;Every perceptron convergence proof i've looked at implicitly uses a learning rate = 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the book I'm using (&quot;Machine learning with Python&quot;) suggests to use a small learning rate for convergence reason, without giving a proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain how the learning rate influences the perceptron convergence and what value of learning rate should be used in practice?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="29169" LastEditDate="2019-06-07T16:53:40.570" LastActivityDate="2019-06-07T16:53:40.570" Title="Learning rate in the Perceptron Proof and Convergence" Tags="|machine-learning|perceptron|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27910" PostTypeId="1" AcceptedAnswerId="27924" CreationDate="2018-02-16T18:34:33.383" Score="3" ViewCount="338" Body="&lt;p&gt;We know how to determine regression parameters using gradient descent. If &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?Y%20%3D%20a%20&amp;plus;%20bx_1&quot; alt=&quot;Y = a+b*x1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the cost function is &lt;code&gt;C=|Y-Y(X)|^2&lt;/code&gt;, we update &lt;code&gt;b&lt;/code&gt; as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?b%20%3A%3D%20b%20-%20%5Calpha%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20b%7D&quot; alt=&quot;b := b - alpha*dC/db&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;img src=&quot;https://latex.codecogs.com/gif.latex?%5Calpha&quot; alt=&quot;alpha&quot;&gt; is the learning rate and &lt;img src=&quot;https://latex.codecogs.com/gif.latex?%5Cpartial%20C/%5Cpartial%20b&quot; alt=&quot;dC/db&quot;&gt; is the partial differential of the cost function &lt;code&gt;C&lt;/code&gt; with respect to &lt;code&gt;b&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If in multiple regression there exist an interaction and we want to stick on the linear model formulation (not using tree or other non-linear regressors), such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?Y%20%3D%20a%20&amp;plus;%20b%20x_1%20x_2&quot; alt=&quot;Y = a + b*x1*x2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the cost function is still the same, do we just do the same way to update &lt;code&gt;b&lt;/code&gt;? i.e. the existence of interaction terms doesn't have impact on gradient descent. I didn't see any difference of gradient descent between with/without interaction.&lt;/p&gt;&#xA;" OwnerUserId="46384" LastEditorUserId="46384" LastEditDate="2018-02-16T23:20:58.327" LastActivityDate="2018-02-17T10:27:14.670" Title="Is the gradient descent the same if cost function has interaction?" Tags="|machine-learning|linear-regression|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27914" PostTypeId="1" CreationDate="2018-02-16T22:53:08.977" Score="10" ViewCount="18941" Body="&lt;p&gt;I am trying the find the pretrained models (graph.pd and labels.txt) files for Tensorflow (for all of the Inception versions and MobileNet)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After much searching I found some models in, &lt;a href=&quot;https://storage.googleapis.com/download.tensorflow.org/models&quot; rel=&quot;noreferrer&quot;&gt;https://storage.googleapis.com/download.tensorflow.org/models&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;like, &lt;a href=&quot;https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip&quot; rel=&quot;noreferrer&quot;&gt;https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://storage.googleapis.com/download.tensorflow.org/models/inception_dec_2015.zip&quot; rel=&quot;noreferrer&quot;&gt;https://storage.googleapis.com/download.tensorflow.org/models/inception_dec_2015.zip&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I found this as links from various blogs/etc. But is there a way to list all the models/zips in this dir? &lt;a href=&quot;https://storage.googleapis.com/download.tensorflow.org/models&quot; rel=&quot;noreferrer&quot;&gt;https://storage.googleapis.com/download.tensorflow.org/models&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want the latest of Inceptionv3 and MobileNet, does anyone know the file names in models for these?&lt;/p&gt;&#xA;" OwnerUserId="46317" LastActivityDate="2020-08-19T18:04:48.453" Title="Where to find list of Tensorflow pretrained models available in download.tensorflow.org/models" Tags="|tensorflow|image-classification|inception|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27916" PostTypeId="2" ParentId="6054" CreationDate="2018-02-17T00:11:05.627" Score="1" Body="&lt;p&gt;Using the algebraic definition of a vector being orthogonal to a hyperplane:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\forall \ x_1, x_2$ on the separating hyperplane, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ w^T(x_1-x_2)=(w^Tx_1 + b)-(w^Tx_2 + b)=0-0=0 \ \small\Box.$$&lt;/p&gt;&#xA;" OwnerUserId="46282" LastActivityDate="2018-02-17T00:11:05.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27918" PostTypeId="1" CreationDate="2018-02-17T01:27:08.710" Score="2" ViewCount="952" Body="&lt;p&gt;I am trying to reproduce this figure from Bishop:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xkJUH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xkJUH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Residual vs. Alpha (lambda in figure)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is pasted below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn import linear_model&#xA;&#xA;from sklearn.linear_model import Ridge&#xA;from sklearn.preprocessing import PolynomialFeatures&#xA;from sklearn.pipeline import make_pipeline&#xA;from sklearn.metrics import mean_squared_error&#xA;&#xA;x_train = np.linspace(0, 1, 10)&#xA;noise_10 = np.random.normal(0, 0.3, 10)&#xA;y_train = np.sin(2*np.pi*x_train) + noise_10&#xA;x_train = x_train[:, np.newaxis]&#xA;&#xA;x_test = np.linspace(0, 1, 100)&#xA;noise_100 = np.random.normal(0, 0.3, 100)&#xA;y_test = np.sin(2*np.pi*x_test) + noise_100&#xA;x_test = x_test[:, np.newaxis]&#xA;&#xA;&#xA;n_alphas = 200&#xA;alphas = np.logspace(-40, -18, n_alphas)&#xA;&#xA;errors = []&#xA;for a in alphas:&#xA;    ridge = make_pipeline(PolynomialFeatures(degree = 9), &#xA;              Ridge(alpha=a))&#xA;    ridge.fit(x_train, y_train)&#xA;    mse = mean_squared_error(y_test, ridge.predict(x_test))&#xA;    errors.append(np.sqrt(mse))    &#xA;&#xA;print(errors)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the errors array has the same value for all values of alpha. It's taking the first value of &lt;code&gt;alpha = np.exp(-40)&lt;/code&gt; and all the other values seem to be the same for all future iterations of the for loop. How can I correct this error?&lt;/p&gt;&#xA;" OwnerUserId="46396" LastEditorUserId="29575" LastEditDate="2018-02-17T01:35:31.763" LastActivityDate="2018-02-20T09:03:48.983" Title="Ridge regression - varying alpha and observing the residual" Tags="|scikit-learn|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27921" PostTypeId="2" ParentId="27777" CreationDate="2018-02-17T08:11:16.020" Score="0" Body="&lt;p&gt;For binary classification you can use &lt;code&gt;softmax&lt;/code&gt; as the output layer but you should consider that if you do so, your last layer have to have two neurons, each corresponds to one specific class. Moreover, you have to use &lt;code&gt;categorical_crossentropy&lt;/code&gt; as the loss function. Change the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(1, activation='softmax'))&#xA;model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(2, activation='softmax'))&#xA;model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-17T08:11:16.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27922" PostTypeId="1" CreationDate="2018-02-17T08:20:56.567" Score="2" ViewCount="686" Body="&lt;p&gt;I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def create_baseline():&#xA;    model = Sequential()&#xA;    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))&#xA;    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))&#xA;    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;    return model&#xA;&#xA;estimators = []&#xA;estimators.append(('standardize', StandardScaler()))&#xA;estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))&#xA;pipeline = Pipeline(estimators)&#xA;kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)&#xA;scores = cross_val_score(pipeline, X, y, cv=kfold)&#xA;print(&quot;%.2f%%&quot; % (scores[1]*100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am using around 5871 malicious samples and 3488 benign samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to cross check I have implemented without cross validation, a very simple implementation, and I am getting ~99% accuracy in that also. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)&#xA;&#xA;scale = StandardScaler(with_mean=0, with_std=1)&#xA;new_X_train = scale.fit_transform(X_train)&#xA;new_X_test = scale.transform(X_test)&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(Dense(23, input_dim=46, kernel_initializer=&quot;normal&quot;))&#xA;model.add(PReLU(alpha_initializer='zero', weights=None))&#xA;&#xA;model.add(Dense(1, kernel_initializer='normal'))&#xA;model.add(Activation('sigmoid'))&#xA;&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)&#xA;&#xA;scores = model.evaluate(new_X_test, y_test)&#xA;print(&quot;%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&#xA;print(history.history.keys())&#xA;&#xA;plt.plot(history.history['acc'])&#xA;plt.plot(history.history['val_acc'])&#xA;plt.title('model accuracy')&#xA;plt.ylabel('accuracy')&#xA;plt.xlabel('epoch')&#xA;plt.legend(['train', 'test'], loc='upper left')&#xA;plt.show()&#xA;&#xA;plt.plot(history.history['loss'])&#xA;plt.plot(history.history['val_loss'])&#xA;plt.title('model loss')&#xA;plt.ylabel('loss')&#xA;plt.xlabel('epoch')&#xA;plt.legend(['train', 'test'], loc='upper left')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The plots I am getting are: &lt;a href=&quot;https://i.stack.imgur.com/rC2MC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rC2MC.png&quot; alt=&quot;loss&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/J7oS2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J7oS2.png&quot; alt=&quot;accuracy&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46402" LastActivityDate="2018-02-17T18:19:55.483" Title="Simple neural network implementation in keras" Tags="|neural-network|classification|keras|cross-validation|feature-scaling|" AnswerCount="1" CommentCount="4" ClosedDate="2018-02-27T06:30:57.133" ContentLicense="CC BY-SA 3.0" />
  <row Id="27924" PostTypeId="2" ParentId="27910" CreationDate="2018-02-17T10:27:14.670" Score="2" Body="&lt;p&gt;The usual way to use interaction terms in linear regression is to construct new $x_n$, e.g. $x_3 = x_1 x_2$, and treat those identically as any other $x_n$. The learned parameter $b$ does not &quot;know&quot; the difference in how you calculated $x$, and the problem is still considered linear regression even if you create really complex functions of $x_n$ to create an input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking your example, but with slightly different notation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The model estimate for Y is $\hat{y} = a + bx_1x_2$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The value of Y you want to learn is $y$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Mean squared error for a single example is $L = \frac{1}{2}(y-\hat{y})^2$ The factor of 2 does not change this answer, and is commonly used to simplify the gradient. Typically $C$ is the mean of $L$ over all examples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In order to learn optimal value of $b$, for one example the gradient you need is $\frac{\partial L}{\partial b}$. We can get that by expanding the loss function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L = \frac{1}{2}(y-\hat{y})^2$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L = \frac{1}{2}(y^2 - 2y\hat{y} + \hat{y}^2)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could expand further, but typically now we calculate $\frac{\partial L}{\partial \hat{y}}$ and use the chain rule, because that has a simpler, more intuitive-looking result. Terms without $\hat{y}$ are zero:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want $\frac{\partial L}{\partial b}$ for gradient descent&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b}$ (Chain rule)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial L}{\partial b} = (\hat{y} - y)(\frac{\partial}{\partial b} a + bx_1x_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, terms without $b$ in them are constants:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial L}{\partial b} = (\hat{y} - y)(x_1x_2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the $x_1x_2$ term is unchanged from the input. It could be any function $x_n = f(x_1, x_2, x_3 ....)$&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-02-17T10:27:14.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27925" PostTypeId="1" CreationDate="2018-02-17T11:26:39.247" Score="6" ViewCount="322" Body="&lt;p&gt;I have 5 different classes in which I want to classify some data points. I'm using RNN with Echo-state networks (Reservoir computing).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, a straightforward method consists of computing the outputs y and deciding the class using the argmax[k]&lt;strong&gt;(y)&lt;/strong&gt; with k is the dimension of the right class. Now I found another method that consists of computing the argmax[k]&lt;strong&gt;(sum(y))&lt;/strong&gt;, and that is what I didn't understand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've attached an image of the mathematical approach. The latter is understandable, but it appears not logical for me, since the sum of y over the data points timescale gives the same result for each time sequence, meaning that all data points will be classified in the same class(which corresponds to the max of the sum(y)).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone what I don't understand?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cVtRc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cVtRc.png&quot; alt=&quot;Readouts for Classification&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46407" LastEditorUserId="29575" LastEditDate="2018-07-11T23:23:09.773" LastActivityDate="2018-07-11T23:23:09.773" Title="Classification of a time series data" Tags="|classification|time-series|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27927" PostTypeId="1" CreationDate="2018-02-17T16:32:46.120" Score="3" ViewCount="6542" Body="&lt;p&gt;I have a dataset that looks like this :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    UserID  App_open_Time(Hour_ofDay)   Email_Open_Time(hour_ofDay)&#xA;       1        1                         1&#xA;       2        1                         1&#xA;       3        1                         1&#xA;       4        1                         1&#xA;       5        1                         1&#xA;       6        2                         1&#xA;       7        2                         1&#xA;       8        3                         1&#xA;       9        3                         1&#xA;      10        3                         1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to know if the &lt;code&gt;App_open_time&lt;/code&gt; is correlated with &lt;code&gt;Email open time&lt;/code&gt; .&#xA;How can i do this analysis in python.&#xA;I was planning to do a &lt;code&gt;pearson-correlation&lt;/code&gt; in Python using Numpy ,Is this the best approach?&lt;/p&gt;&#xA;" OwnerUserId="13515" LastActivityDate="2019-05-11T03:03:40.643" Title="Calculating correlation between two time variables" Tags="|machine-learning|correlation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27928" PostTypeId="2" ParentId="27927" CreationDate="2018-02-17T16:52:43.043" Score="0" Body="&lt;p&gt;You can use the following code snippet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import cm&#xA;cmap = cm.get_cmap('gnuplot')&#xA;scatter = pd.scatter_matrix(YOUR_TRAINING_DATA, c = YOUR_LABELS_OF_TRAINING, marker = 'o', s = 40, hist_kwds = {'bins':15}, figsize = (12, 12), cmap = cmap)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It plots the scatter plot of each feature separately and together. It is like the correlation matrix. You can take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/25271/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-17T16:52:43.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27929" PostTypeId="2" ParentId="27925" CreationDate="2018-02-17T16:57:35.207" Score="1" Body="&lt;p&gt;The method you have here is for time series, so input and output are a set of values over a time period.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum method sum(y) here is a short hand (abbreviation) for the average over the last mentioned equation, it is not a conventional sum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also do note argmax k over the sums is done in separate dimension for each k representing the individual class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in my understanding sum is needed only because it is a series of data and sum is calculated per class for the whole series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The y target (n) is 1 in only in one dimension when match happen, so argmax k finds with highest value the correct class.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-02-17T16:57:35.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27931" PostTypeId="2" ParentId="27922" CreationDate="2018-02-17T18:19:55.483" Score="1" Body="&lt;p&gt;Considering the nature of your problem (disease classification?), probably you have imbalanced data: more negative/benign w.r.t. positive/malicious. In such cases accuracy is not a good metric to assess the quality of a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider, as example, a dataset with 99 &quot;negative&quot; cases and only one &quot;positive&quot; case, and consider a classifier which predict &quot;negative&quot; all the times. The accuracy of such model is, actually, 99%:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qvOFT.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qvOFT.gif&quot; alt=&quot;accuracy formula&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you want to do in order to deal with unbalanced data is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) just use another metric, as &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow noreferrer&quot;&gt;ROC-AUC&lt;/a&gt;. This can be sufficient if you have a dataset not so much unbalanced, say up to 1:20. This metric is not officially supported in Keras, but you can find a custom callback which calculate the AUC score every end of epoch &lt;a href=&quot;https://github.com/keras-team/keras/issues/3230&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) if the imbalance is very big, say 1:1000, you need to balance the dataset. There are lots of techniques, the simplest one is to make copies of the minority class samples, as many as needed to reach the balance. Other smarter techniques exist, and some of then are included in &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/&quot; rel=&quot;nofollow noreferrer&quot;&gt;imbalanced-learn for SKLearn&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to have a better overview of techniques and metrics used in such cases, I suggest you &lt;a href=&quot;http://www.ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-02-17T18:19:55.483" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27939" PostTypeId="1" CreationDate="2018-02-18T01:26:48.557" Score="4" ViewCount="222" Body="&lt;p&gt;Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scheme 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss)words from each article (for the sake of simplicity let us assume batch size = m)&lt;/li&gt;&#xA;&lt;li&gt;Set the initial hidden state H0 = $[0,0,..,0]$&lt;/li&gt;&#xA;&lt;li&gt;Calculate loss and gradient on this batch and update the parameters&lt;/li&gt;&#xA;&lt;li&gt;Move a word forward and get next S words from each article (hence S2, S3, ... , Ss are the same words as in the previous batch)&lt;/li&gt;&#xA;&lt;li&gt;Use H1 calculated from previous step and use it as H0 in this iteration&lt;/li&gt;&#xA;&lt;li&gt;Do this to the end&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;*In this scheme we would have to use zero padding on the last batch (at the end of articles)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scheme 2&lt;/strong&gt;&#xA;Same as Scheme 1 but in step 5 we reinitialize H0 to a vector of Zeroes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scheme 3&lt;/strong&gt;&#xA;Same as scheme 1 but in step 4 we move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Scheme 4, 5, 6&lt;/strong&gt;&#xA;Same as Scheme 1, 2, 3 but instead of taking s consecutive words we take first sentence from each article and zero pad them to the length S&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the right way to go through the data in feeding it to a RNN, Please give reasons as well. I think it should be Scheme 1 but it could be really slow&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2022-01-17T13:01:00.097" Title="How do we pass data to a RNN?" Tags="|deep-learning|rnn|language-model|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27940" PostTypeId="1" AcceptedAnswerId="28021" CreationDate="2018-02-18T02:30:41.953" Score="0" ViewCount="362" Body="&lt;p&gt;I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any methods to generate lists of related words from a root word? For instance, where I could type &quot;medicine&quot; and it would return a list of words or terms related to medicine, such as &quot;doctor&quot;, &quot;malpractice&quot;, &quot;office&quot;, etc.?&lt;/p&gt;&#xA;" OwnerUserId="46424" LastActivityDate="2018-02-20T10:49:29.650" Title="Guided topic modeling: generating words from topics" Tags="|r|topic-model|lda|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27941" PostTypeId="1" CreationDate="2018-02-18T02:52:08.367" Score="3" ViewCount="2456" Body="&lt;p&gt;Use case is like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a sentence(review data): &lt;code&gt;The staffs were very kind and helpful. The room is ok for its price. There did not seem to be a heater in the room. So, a bit for our January trip.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if I want to know the sentiment of a word for example &lt;code&gt;room&lt;/code&gt; then how should I proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have designed sentiment analysis for a given sentence using &lt;code&gt;bag of words&lt;/code&gt; model, but determining the sentiment for the given &lt;code&gt;word&lt;/code&gt; from the &lt;code&gt;sentence&lt;/code&gt; (review data) is kind of new to me. What approach should I take here ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Providing some approach or any link is enough.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="41266" LastEditorUserId="41266" LastEditDate="2018-02-19T06:42:34.973" LastActivityDate="2018-02-19T06:42:34.973" Title="How to identify sentiment of a given word from a sentence" Tags="|machine-learning|deep-learning|nlp|sentiment-analysis|" AnswerCount="2" CommentCount="3" ClosedDate="2018-02-18T13:58:14.277" ContentLicense="CC BY-SA 3.0" />
  <row Id="27942" PostTypeId="2" ParentId="27941" CreationDate="2018-02-18T03:56:40.567" Score="2" Body="&lt;p&gt;One of the ways to do it is to build bag of words and apply a linear model for sentimental analysis. Then you can look at model's coefficients - positive coefficient means word is positive, negative coefficient - negative word.&lt;/p&gt;&#xA;" OwnerUserId="35541" LastActivityDate="2018-02-18T03:56:40.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27943" PostTypeId="2" ParentId="27941" CreationDate="2018-02-18T04:46:02.380" Score="2" Body="&lt;p&gt;You could use existing indices of positive and negative words. &lt;a href=&quot;https://www.quora.com/Is-there-a-downloadable-database-of-positive-and-negative-words&quot; rel=&quot;nofollow noreferrer&quot;&gt;This link&lt;/a&gt; from Quora lists several such indices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;True sentiment analysis from a single word could be difficult. Words derive their meaning from the context in which they appear. Almost any word could be positive or negative depending on how it's used. Still, there are probabilities at play so it may be a viable approach. &lt;/p&gt;&#xA;" OwnerUserId="32121" LastActivityDate="2018-02-18T04:46:02.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27945" PostTypeId="1" AcceptedAnswerId="27946" CreationDate="2018-02-18T07:21:39.277" Score="11" ViewCount="4670" Body="&lt;p&gt;Currently I am going through Normal Equation in Machine Learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$ &#xA;\hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I see how they use this equation, I found they always add an additional column of 1s in the starting of matrix X before transposing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand why. What's the logic behind this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The places where I found such things&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/2DKxQ/normal-equation&quot; rel=&quot;noreferrer&quot;&gt;Coursera&lt;/a&gt; - Theory&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Implementation&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now let’s compute using the Normal Equation. We will use the &lt;code&gt;inv()&lt;/code&gt;&#xA;  function from NumPy’s Linear Algebra module (&lt;code&gt;np.linalg&lt;/code&gt;) to compute the&#xA;  inverse of a matrix, and the &lt;code&gt;dot()&lt;/code&gt; method for matrix multiplication:&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;code&gt;X_b = np.c_[np.ones(( 100, 1)), X] # add x0 = 1 to each instance&lt;/code&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn and&#xA;  TensorFlow: Concepts, Tools, and Techniques to Build Intelligent&#xA;  Systems (p. 111). O'Reilly Media. Kindle Edition.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46429" LastEditorUserId="89038" LastEditDate="2020-05-13T14:21:31.307" LastActivityDate="2020-05-13T14:21:31.307" Title="What is the use of additional column of 1s in normal equation?" Tags="|machine-learning|linear-algebra|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27946" PostTypeId="2" ParentId="27945" CreationDate="2018-02-18T07:30:00.303" Score="11" Body="&lt;p&gt;The normal equations are designed such that each coefficient in the model has an input of some kind it's being multiplied against. The column of ones is the &quot;input&quot; to the intercept term.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-18T07:30:00.303" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27948" PostTypeId="1" CreationDate="2018-02-18T08:46:14.167" Score="2" ViewCount="599" Body="&lt;p&gt;I want to build two parallel models for image semantic segmentation in Keras.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    input1 = Input(shape=(480,480,3))&#xA;    input2 = Input(shape=(480,480,1))&#xA;&#xA;    c1_1 = Conv2D(filters=64, kernel_size=(3,3), activation='relu',  padding='same')(input1)&#xA;    c1_1 = MaxPool2D(strides=(2,2))(c1_1)&#xA;&#xA;    c2_1 = Conv2D(filters=16, kernel_size=(3,3), activation='relu',  padding='same')(input2)&#xA;    c2_1 = MaxPool2D(strides=(2,2))(c2_1)&#xA;&#xA;   (...)&#xA;&#xA;    # Merge to models:&#xA;    c = concatenate([c1_n, c2_n], axis=3)&#xA;    c = UpSampling2D(size=(2,2))(c)&#xA;    c = Conv2D(filters=512, kernel_size=(3,3), activation='relu', padding='same')(c)&#xA;&#xA;   (...)&#xA;&#xA;   output_layer = Conv2D(6, kernel_size=(1,1), activation='softmax')(c)&#xA;&#xA;   model = Model([input1, input2], output_layer)&#xA;&#xA;   model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['categorical_accuracy'])&#xA;&#xA;&#xA;   def my_generator(x_train, y_train, batch_size):&#xA;     data_generator = ImageDataGenerator(&#xA;        (...).flow(x_train, x_train, batch_size, seed=42)&#xA;     mask_generator = ImageDataGenerator(&#xA;        (...).flow(y_train, y_train, batch_size, seed=42)&#xA;&#xA;     while True:&#xA;        x_batch, _ = data_generator.next()&#xA;        y_batch, _ = mask_generator.next()&#xA;        yield [x_batch[:,:,:,:3], x_batch[:,:,:,3]], y_batch&#xA;&#xA;     #X = [72, 480, 480, 4]&#xA;     #Y = [72, 480, 480, 5]&#xA;     model.fit_generator(my_generator(X, Y, 7),&#xA;                       steps_per_epoch = 60,&#xA;                       validation_data = (X_test, Y_test),&#xA;                       epochs=150, verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But i have an error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[ 0.33891213,  0.37238494,  0.33054393,  0.        ],&#xA;     [ 0.34728033,  0.38493724,  0.33472803,  0.        ],&#xA;     [ 0.35146444,  0.39330544,  0.35983264,  0.        ],&#xA;     ....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can you help me with my error or tell me whether this assumption is correct?&lt;/p&gt;&#xA;" OwnerUserId="46104" LastEditorUserId="46104" LastEditDate="2018-02-18T08:52:39.810" LastActivityDate="2024-03-12T15:06:10.740" Title="Two parallel models for semantic segmentation in Keras" Tags="|python|deep-learning|keras|image-classification|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27949" PostTypeId="1" AcceptedAnswerId="27950" CreationDate="2018-02-18T09:21:55.397" Score="1" ViewCount="8048" Body="&lt;p&gt;Currently, I am working on a project. The dataset is balanced roughly in the ratio of 50:50. I created a decision tree classifier. I am achieving decent accuracy (~75%) on validation data but the precision for the target variable is biased. For class=0 it is approx. 98% while for the class = 1 it is just 17%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried scaling the data using MinMaxScaler still no luck.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = tree.DecisionTreeClassifier(class_weight={1:30}, min_samples_leaf=160, max_depth=10)&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=10)&#xA;&#xA;min_max_scaler = preprocessing.MinMaxScaler()&#xA;X_train_scaled = min_max_scaler.fit_transform(X_train)&#xA;X_test_scaled = min_max_scaler.fit_transform(X_test)&#xA;&#xA;model = model.fit(X_train_scaled, y_train)&#xA;&#xA;prediction = model.predict(X_test_scaled)&#xA;&#xA;print metrics.accuracy_score(y_test, prediction)&#xA;print classification_report(y_test, prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;Size of x_train_scaled = 12600 and x_test_scaled = 5400&#xA;Accuracy: 75%&#xA;Precision: {0:100%, 1:17%}&#xA;Recall: {0:74%, 1:100%}&#xA;F1-Score: {0:85%, 1:29%}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I improve the precision of class=1 while still maintaining the overall precision and accuracy?&lt;/p&gt;&#xA;" OwnerUserId="46431" LastActivityDate="2019-07-23T18:34:37.627" Title="Improve Precision of a binary classifier - Decision Tree in Python" Tags="|machine-learning|python|scikit-learn|decision-trees|performance|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27950" PostTypeId="2" ParentId="27949" CreationDate="2018-02-18T11:05:38.783" Score="7" Body="&lt;p&gt;I guess differences in accuracies between class 0 and class 1 come from the &lt;strong&gt;class_weight&lt;/strong&gt; parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on this parameter to re-balance your results in class 0 and class 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An other idea could be to play on probabilities outputs and decision boundary threshold. Remember than when calling for method &lt;strong&gt;.predict()&lt;/strong&gt;, sklearn decision tree will compare outputed probability with threshold 0.5. If it is greater than 0.5, then it assign class 1. On the contrary, if it is less than 0.5, it will assign class 0. You could try to play on this threshold by outputing probabilities first with &lt;strong&gt;.predict_proba()[:,1]&lt;/strong&gt; and then test results for different thresholds decision boundaries. You can operate such as below :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = clf.fit(df[features], df[label])&#xA;df[&quot;proba&quot;] = model.predict_proba(df[features])[:,1]&#xA;threshold = 0.4 # You can play on this value (default is 0.5)&#xA;df[&quot;pred&quot;] = df[&quot;proba&quot;].apply(lambda el: 1.0 if el &amp;gt;= threshold else 0.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="18434" LastEditDate="2018-02-21T10:19:05.937" LastActivityDate="2018-02-21T10:19:05.937" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27951" PostTypeId="2" ParentId="27918" CreationDate="2018-02-18T11:49:20.250" Score="0" Body="&lt;p&gt;I have taken a look on your code. You obtain same errors results for each alpha value because your regularization strength is too small. Replacing : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alphas = np.logspace(-40, -18, n_alphas)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alphas = np.logspace(-40, -1, n_alphas)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will yields different errors values for alpha values large enough. Are you sure about figure alpha values? Do you have a link to this hands-on?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I would like to highlight the fact that you have to standardize your features before using regularization. Reason is, by creating polynomial features, polynomial features will have different magnitudes. Therefore when fitting model, coefficients to be estimated won't have the same magnitudes neither and so regularization will highly penalize coefficients with large values. Standardization / Normalization is a strong prerequisite to regularization.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-02-18T11:49:20.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27952" PostTypeId="2" ParentId="27940" CreationDate="2018-02-18T13:11:28.793" Score="1" Body="&lt;p&gt;I think instead of building something from scratch, you should utilize the domain knowledge of experts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example: Turn Wikipedia's topic portals (e.g., the &lt;a href=&quot;https://en.wikipedia.org/wiki/Portal:Medicine/Medicine_topics&quot; rel=&quot;nofollow noreferrer&quot;&gt;medicine portal&lt;/a&gt;) into &#xA;a family tree.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-18T13:11:28.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27953" PostTypeId="1" AcceptedAnswerId="28844" CreationDate="2018-02-18T13:14:53.937" Score="8" ViewCount="7932" Body="&lt;p&gt;I have a lack of understanding about this issue. Could anybody explain it or give an advice to a good literature regarding it? I don't understand what is a explicit density model and how it differs from an implicit density one. Could anyone exemplify the difference between these two kinds of models?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LBO7b.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LBO7b.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Here is the picture &lt;a href=&quot;https://arxiv.org/abs/1701.00160&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45271" LastEditorUserId="85045" LastEditDate="2020-12-10T08:58:22.387" LastActivityDate="2020-12-10T08:58:22.387" Title="Difference between explicit and implicit density with and without the relation to neural network" Tags="|neural-network|rnn|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="27954" PostTypeId="1" AcceptedAnswerId="27955" CreationDate="2018-02-18T14:59:58.727" Score="2" ViewCount="366" Body="&lt;p&gt;I have a similarity/distance matrix:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   a | b | c&#xA; a 0 | 1 | 2&#xA; b 1 | 0 | 3&#xA; c 2 | 3 | 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to build an encoder/model that learns an n-dimensional representation of each of the points in the dataset s.t. the euclidean-difference between the representations produces the difference provided in the matrix, e.g. distance(a,b) = 1 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="46439" LastActivityDate="2018-02-18T19:19:37.097" Title="How to build an encoder using a distance matrix" Tags="|autoencoder|distance|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27955" PostTypeId="2" ParentId="27954" CreationDate="2018-02-18T16:21:26.213" Score="1" Body="&lt;p&gt;This is essentially what &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt; does. It supports transforming data from any input dimension to any output dimension, and it tries to preserve distances between all pairs of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the author's &lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;website&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Can I use a pairwise Euclidean distance matrix as input into t-SNE?&lt;/strong&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Yes you can! Download the Matlab implementation, and use your pairwise&#xA;  Euclidean distance matrix as input into the tsne_d.m function.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="15828" LastEditorUserId="15828" LastEditDate="2018-02-18T19:19:37.097" LastActivityDate="2018-02-18T19:19:37.097" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="27957" PostTypeId="1" AcceptedAnswerId="27993" CreationDate="2018-02-18T17:43:56.533" Score="26" ViewCount="30451" Body="&lt;p&gt;I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable like location: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Location &#xA;----------&#xA;Californian&#xA;NY&#xA;Florida&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We have to convert them like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1  0  0&#xA;0  1  0&#xA;0  0  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, it was suggested that we have to discard one dummy variable, no matter how many dummy variables are there. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do we need to discard one dummy variable?&lt;/p&gt;&#xA;" OwnerUserId="42205" LastEditorUserId="29169" LastEditDate="2019-06-09T15:32:24.513" LastActivityDate="2020-02-19T22:03:18.257" Title="Why do we need to discard one dummy variable?" Tags="|machine-learning|regression|categorical-data|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="27958" PostTypeId="1" AcceptedAnswerId="27961" CreationDate="2018-02-18T17:45:03.400" Score="1" ViewCount="1174" Body="&lt;p&gt;Via this &lt;a href=&quot;https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-867-machine-learning-fall-2006/lecture-notes/lec2.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;MIT document&lt;/a&gt; i studied the single layer Perceptron convergence proof (= maximum number of steps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the convergence proof inside this document , the learning rate is implicitly defined as 1 .&#xA;After studying it, by myself i tried to re-do the proof inserting this time a generic learning rate $\eta $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result is that the result remains the same:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$k\leq \frac{R^{2}\left \| \theta ^{*} \right \|^{2}}{\gamma ^{2}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;that is the learning rate $\eta$ cancelled out in the proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible , or i make mistakes in my proof ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-02-18T18:32:02.947" Title="Perceptron learning rate irrelevant in convergence" Tags="|machine-learning|perceptron|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-19T21:29:50.737" ContentLicense="CC BY-SA 3.0" />
  <row Id="27959" PostTypeId="1" CreationDate="2018-02-18T18:02:39.017" Score="2" ViewCount="2257" Body="&lt;p&gt;I have a simple question about convolution layers in CNNs. Consider that we have 32 features map with size &lt;span class=&quot;math-container&quot;&gt;$100\times100$&lt;/span&gt;. So, can we set 16 convolution layer with size &lt;span class=&quot;math-container&quot;&gt;$9\times9\times16$&lt;/span&gt; after features map? There isn't any limit on depth of the convolution layer? Then if we can, so every &lt;span class=&quot;math-container&quot;&gt;$9\times9\times16$&lt;/span&gt; layer produces just one layer? (How does depth=16 work on depth=32)?&lt;/p&gt;&#xA;" OwnerUserId="45494" LastEditorUserId="29575" LastEditDate="2019-08-29T19:20:26.137" LastActivityDate="2019-08-29T19:20:26.137" Title="Convolution over volume in CNNs" Tags="|machine-learning|neural-network|deep-learning|cnn|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="27960" PostTypeId="1" AcceptedAnswerId="27969" CreationDate="2018-02-18T18:14:14.443" Score="4" ViewCount="1951" Body="&lt;p&gt;In &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn documentation&lt;/a&gt; and in &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning#Limitations&quot; rel=&quot;nofollow noreferrer&quot;&gt;decision tree learning Wikipedia article&lt;/a&gt; there is mention of &quot;&lt;strong&gt;There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems.&lt;/strong&gt;&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't recall knowing such problem types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are examples (and possibly related datasets) of such problems?&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2018-02-18T21:21:51.620" Title="What are examples for XOR, parity and multiplexer problems in decision tree learning?" Tags="|dataset|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27961" PostTypeId="2" ParentId="27958" CreationDate="2018-02-18T18:20:21.173" Score="2" Body="&lt;p&gt;Your conclusion is correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the classifier is of the form of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f(x; \theta) = \operatorname{sign}(\theta^Tx)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;while the update rule is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta^{(k+1)}=\theta^{(k)}+\eta y_tx_t$$ which only occurs when there is a misclassification and we only care about the in the classification but &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\operatorname{sign}(\theta^Tx)=\operatorname{sign}(\eta \theta^Tx)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;as long as $\eta$ is positive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It only scales $\theta^Tx$ but our concern is just the sign, hence the learning rate is irrelevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remark: You might like to note the assumptions being used in the proof though. For instance, there is this assumption that $\theta^*$ exists. It is assumed that the data can be perfectly separated by a hyperplane.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2018-02-18T18:32:02.947" LastActivityDate="2018-02-18T18:32:02.947" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27962" PostTypeId="2" ParentId="27959" CreationDate="2018-02-18T18:26:15.577" Score="4" Body="&lt;p&gt;When you have 32 feature maps with height and width equal to 100 and the depth of each equal to one it means that you have 32 planes, a common jargon among vision people, with 100 by 100 entries. You can set the height and width of the next layer and they can be arbitrary. You can also set the number of feature maps but the depth of each feature map would be equal to the number of feature maps of the previous layer. So it should be 16 * 9 * 9 * 32 if you set height and width equal to 9 and the number of feature maps to 16. As you can see in 16 * 9 * 9 * 32, 32 is located at the end, this is called &lt;em&gt;channels last&lt;/em&gt;. You can not set the depth, because it should be equal to the number of channels, features maps, of the previous layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;16 * 9 * 9 * 32 means that you have 16 feature maps of dimension 9 * 9 * 32, so the output of each feature map would be the member-wise product of all the outputs of the feature maps of the previous layer and each of 9 * 9 * 32 kernels. Consequently the result would be 16 planes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I highly recommend you taking a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/ctQZz/convolutions-over-volume&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-18T19:06:07.663" LastActivityDate="2018-02-18T19:06:07.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27963" PostTypeId="1" AcceptedAnswerId="27965" CreationDate="2018-02-18T18:44:06.080" Score="3" ViewCount="71" Body="&lt;p&gt;could you please suggest me a nondeterministic algorithm for dimensionality reduction except t-SNE. &lt;/p&gt;&#xA;" OwnerUserId="43559" LastActivityDate="2018-02-18T19:59:02.080" Title="Non Deterministc Dimensionality reduction" Tags="|machine-learning|deep-learning|cross-validation|dimensionality-reduction|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-25T14:49:03.300" ContentLicense="CC BY-SA 3.0" />
  <row Id="27964" PostTypeId="1" AcceptedAnswerId="27967" CreationDate="2018-02-18T19:31:35.513" Score="8" ViewCount="40677" Body="&lt;p&gt;I have two different files and on the first, I tried to save data to file as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.save(open(Q1_TRAINING_DATA_FILE, 'wb'), q1_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;On second file, i'm trying to load it the same way using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;q1_data = np.load(open(Q1_TRAINING_DATA_FILE, 'rb'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I then get the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;FileNotFoundError: [Errno 2] No such file or directory: 'q1_train.npy'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I searched my google drive but couldn't find this file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Platform: &lt;a href=&quot;https://research.google.com&quot; rel=&quot;noreferrer&quot;&gt;https://research.google.com&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&#xA;I'm trying to run below Kaggle problem on Colab platform. The author has two files (Jupyter and nbs) - one to prepare and 2nd to train. The step on nb1 where it's creating some files - which later to be consumed by file 2 is where I'm struck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/bradleypallen/keras-quora-question-pairs/blob/master/quora-question-pairs-training.ipynb&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/bradleypallen/keras-quora-question-pairs/blob/master/quora-question-pairs-training.ipynb&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="35544" LastEditorUserId="29169" LastEditDate="2019-06-09T21:22:46.457" LastActivityDate="2019-06-09T21:22:46.457" Title="How to download dynamic files created during work on Google Colab?" Tags="|machine-learning|neural-network|deep-learning|cloud-computing|colab|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="27965" PostTypeId="2" ParentId="27963" CreationDate="2018-02-18T19:59:02.080" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autoencoders&lt;/a&gt; are non-deterministic, since they rely on a random weight initialization.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-18T19:59:02.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27966" PostTypeId="1" CreationDate="2018-02-18T20:30:32.577" Score="1" ViewCount="30" Body="&lt;p&gt;So, the loss function needs to be a function of (say) two variables A and B. It needs to be continous and differentiable in terms of A. Basically, I have a set of precomputed values (B), and during training, after every forward pass, I get a value A. A can vary from 0 to 1. And B can vary from 0 to (say) a bit large value, 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, this is how I want the loss function to work :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A - high, B - high&lt;br&gt;&#xA;=&gt; Ideal situation. Loss should be a small value i.e. No change in A required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A - high, B - low&#xA;=&gt; Loss should be such that it moves towards decreasing the value of A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A - low, B - high&#xA;=&gt; Loss should be such that it moves towards increasing the value of A. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A - low, B - low&#xA;=&gt; Ideal situation. Loss should be a small value i.e. No change in A required.&lt;/p&gt;&#xA;" OwnerUserId="46449" LastActivityDate="2018-02-25T15:00:03.167" Title="I am working on a project, and I've been trying to come up with a custom loss/score function. Would really appreciate some help" Tags="|machine-learning|deep-learning|statistics|loss-function|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-26T00:52:40.570" ContentLicense="CC BY-SA 3.0" />
  <row Id="27967" PostTypeId="2" ParentId="27964" CreationDate="2018-02-18T20:31:03.443" Score="4" Body="&lt;p&gt;Based on what I've seen and experienced, the best way is to store and retrieve your data from your drive account. Actually your question is a bit unclear but first I say, try to use the following command to see the current files in your directory, although I guess each 12 hours they all would be deleted automatically. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;!ls&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Anyway I recommend the following instructions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the following code for having permission to access to your drive account:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;!pip install -U -q PyDrive&#xA;&#xA;import tensorflow as tf&#xA;import timeit&#xA;&#xA;config = tf.ConfigProto()&#xA;config.gpu_options.allow_growth = True&#xA;&#xA;from pydrive.auth import GoogleAuth&#xA;from pydrive.drive import GoogleDrive&#xA;from google.colab import auth&#xA;from oauth2client.client import GoogleCredentials&#xA;&#xA;# Authenticate and create the PyDrive client.&#xA;auth.authenticate_user()&#xA;gauth = GoogleAuth()&#xA;gauth.credentials = GoogleCredentials.get_application_default()&#xA;drive = GoogleDrive(gauth)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Use the following code to get the id of contents in your drive:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;file_list = drive.ListFile({'q': &quot;'root' in parents and trashed=false&quot;}).GetList()&#xA;for file1 in file_list:&#xA;  print('title: %s, id: %s' % (file1['title'], file1['id']))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Put the id of the desired file, e.g. a typical text file, in the content of the following dictionary with id key:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;downloaded = drive.CreateFile({'id': 'the id of typical text file'})&#xA;file = downloaded.GetContentString()&#xA;print('Downloaded content &quot;{}&quot;'.format(len(file)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Till now you have copied the text file, then you have to write it in your Colab disk using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;text_file = open(&quot;your desired name.txt&quot;, &quot;w&quot;)&#xA;text_file.write(file)    &#xA;text_file.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Create &amp;amp; upload a file.&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;uploaded = drive.CreateFile({'title': 'filename.csv'})&#xA;uploaded.SetContentFile('filename.csv')&#xA;uploaded.Upload()&#xA;print('Uploaded file with ID {}'.format(uploaded.get('id')))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Downloading from Colab without Uploading to drive&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from google.colab import files&#xA;files.download('your typical h5 file or what ever.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For more information about transferring different data formats there are more explanations in the notebook provided with &lt;a href=&quot;https://colab.research.google.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;em&gt;Colab&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-19T05:23:20.657" LastActivityDate="2018-02-19T05:23:20.657" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27968" PostTypeId="2" ParentId="27716" CreationDate="2018-02-18T21:01:37.043" Score="1" Body="&lt;p&gt;What you presented is the  typical proof of convergence of perceptron proof indeed is independent of $\mu$. Hence the conclusion is right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically $\theta^*x$ represents a hyperplane that perfectly separate the two classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula $k \le \frac{\mu^2 R^2 \|\theta^*\|^2}{\gamma^2}$ doesn't make sense as it implies that if you set $\mu$ to be small, then $k$ is arbitarily close to $0$. It is saying that with small learning rate, it converges immediately. You might want to look at the termination condition for your perceptron algorithm carefully.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-02-18T21:01:37.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27969" PostTypeId="2" ParentId="27960" CreationDate="2018-02-18T21:21:51.620" Score="5" Body="&lt;p&gt;Below is an example of XOR dataset for classification. As you can see, decision trees perform pretty poorly on this dataset. Reason is decision trees splits space into rectangular regions. Therefore they are not pretty good with this kind of distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really want to use trees in that sort of situation, it is interesting to use so-called rotation trees. Rotation is about performing PCA (principal components analysis) on input features while learning trees. Using it, decision trees can then build non-rectangular regions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/v4IfD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/v4IfD.png&quot; alt=&quot;XOR machine learning&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, here is a playground to test gradient boosting algorithms including on XOR dataset. It is really interesting ! You can click on &quot;rotate trees&quot; to activate rotation.&#xA;&lt;a href=&quot;http://arogozhnikov.github.io/2016/07/05/gradient_boosting_playground.html&quot; rel=&quot;noreferrer&quot;&gt;Gradient boosting playground&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-02-18T21:21:51.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27971" PostTypeId="1" CreationDate="2018-02-18T21:32:16.177" Score="0" ViewCount="56" Body="&lt;p&gt;I have a .txt file and I am required to read the bottom table using &lt;strong&gt;read.table()&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pages.stat.wisc.edu/~jgillett/327-1/4/beef.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://pages.stat.wisc.edu/~jgillett/327-1/4/beef.txt&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I clean the text to do that?&lt;/p&gt;&#xA;" OwnerUserId="46451" LastActivityDate="2018-02-19T01:10:39.890" Title="How to read table" Tags="|r|text-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-19T15:06:47.340" ContentLicense="CC BY-SA 3.0" />
  <row Id="27973" PostTypeId="2" ParentId="27971" CreationDate="2018-02-18T21:52:24.767" Score="1" Body="&lt;p&gt;The &quot;comment.char&quot; parameters in read.table() allows you to skip those lines with comments.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; A &amp;lt;- read.table('beef.txt', header = T, comment.char = &quot;%&quot;)&#xA;&amp;gt; str(A)&#xA;'data.frame':   17 obs. of  10 variables:&#xA;$ YEAR : int  1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 ...&#xA;$ PBE  : num  59.7 59.7 63 71 71 74.2 72.1 79 73.1 70.2 ...&#xA;$ CBE  : num  58.6 59.4 53.7 48.1 49 48.2 47.9 46 50.8 55.2 ...&#xA;$ PPO  : num  60.5 63.3 59.9 56.3 55 59.6 57 49.5 47.3 56.6 ...&#xA;$ CPO  : num  65.8 63.3 66.8 69.9 68.7 66.1 67.4 69.7 68.7 62.2 ...&#xA;$ PFO  : num  65.8 68 65.5 64.8 65.6 62.4 51.4 42.8 41.6 46.4 ...&#xA;$ DINC : num  51.4 52.6 52.1 52.7 55.1 48.8 41.5 31.4 29.4 33.2 ...&#xA;$ CFO  : num  90.9 92.1 90.9 90.9 91.1 90.7 90 87.8 88 89.1 ...&#xA;$ RDINC: num  68.5 69.6 70.2 71.9 75.2 68.3 64 53.9 53.2 58 ...&#xA;$ RFP  : int  877 899 883 884 895 874 791 733 752 811 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2018-02-19T01:10:39.890" LastActivityDate="2018-02-19T01:10:39.890" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27976" PostTypeId="2" ParentId="27829" CreationDate="2018-02-19T00:20:53.327" Score="1" Body="&lt;p&gt;TF IDF as well as LDA are meant to work with &lt;strong&gt;much longer documents&lt;/strong&gt;. All documents should have more than 100 tokens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a median of 1, no clustering will be able to do much. Either they visited the same url, or they didn't. That is too little information for a statistical approach.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-19T00:20:53.327" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="27978" PostTypeId="1" AcceptedAnswerId="27982" CreationDate="2018-02-19T03:51:57.977" Score="1" ViewCount="494" Body="&lt;p&gt;There are many metrics to evaluate clustering algorithm like Calinski-Harabaz Index, Dunn index, Rand index, etc. Are there any advantage of using Dunn index over other metrics for evaluating clustering algorithm (K-means in particular)? If yes, what are the advantages and disadvantages?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2018-02-19T07:09:24.007" Title="What is the advantage of using Dunn index over other metrics for evaluating clustering algorithm?" Tags="|machine-learning|classification|clustering|unsupervised-learning|model-evaluations|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-25T15:03:52.913" ContentLicense="CC BY-SA 3.0" />
  <row Id="27979" PostTypeId="1" AcceptedAnswerId="27980" CreationDate="2018-02-19T05:34:59.977" Score="1" ViewCount="80" Body="&lt;pre&gt;&lt;code&gt;trainung_data = [&#xA;    ['Green', 3, 'Apple'],&#xA;    ['Yellow', 3, 'Apple'],&#xA;    ['Red', 1, 'Grape'],&#xA;    ['Red', 1, 'Grape'],&#xA;    ['Yellow', 3, 'Lemon'],&#xA;]&#xA;header = [&quot;color&quot;, &quot;diameter&quot;, &quot;label&quot;]&#xA;def is_numeric(value):&#xA;    return isinstance(value, int) or isinstance(value, float)&#xA;class Question:   &#xA;    def __init__(self, column, value):&#xA;        self.column = column&#xA;        self.value = value&#xA;&#xA;    def match(self, example):&#xA;        val = example[self.column]&#xA;        if is_numeric(val):&#xA;            return val&amp;gt;=self.value&#xA;        else:&#xA;             return val&amp;gt;=self.value&#xA;    def __repr__(self):&#xA;        condition = '=='&#xA;        if is_numeric(self.value):&#xA;            condition = '&amp;gt;='&#xA;        return &quot;Is %s %s %s?&quot; % (header[self.column], condition, str(self.value))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if i call the class like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Question(0, 'Green')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It give me output like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Is color == Green?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46433" LastActivityDate="2018-02-19T06:25:01.040" Title="How the &quot;def match(self, example)&quot; method is automatically calling and how the example arguments is working here?" Tags="|python|jupyter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27980" PostTypeId="2" ParentId="27979" CreationDate="2018-02-19T06:25:01.040" Score="0" Body="&lt;p&gt;Typing &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Question(0,'Green')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should not print out anything. However if you type &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(Question(0,'Green'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then, the $\_\_$repr$\_\_$ method is called.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-02-19T06:25:01.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27981" PostTypeId="2" ParentId="24458" CreationDate="2018-02-19T06:52:40.527" Score="1" Body="&lt;p&gt;One the key advantages that is I use a lot is that is compatible with pdb, so that&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pdb.set_trace # To the rescue&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Permits the use of python data structures&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and let us to use pythonic control flow instead of using the main tf equivalents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also it allows to avoid metaprogramming issues such as &quot;lazy loading&quot; and adding a bunch of operations to my graph. Also autograd similarities&lt;/p&gt;&#xA;" OwnerUserId="46465" LastEditorUserId="11097" LastEditDate="2018-02-19T06:54:11.123" LastActivityDate="2018-02-19T06:54:11.123" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="27982" PostTypeId="2" ParentId="27978" CreationDate="2018-02-19T07:09:24.007" Score="1" Body="&lt;p&gt;They will often give the same preferences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not forget that these are largely &lt;strong&gt;heuristics&lt;/strong&gt;. They won't have much advantages over one another. That is why there are so many.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a guideline, look at the definitions. Choose that index, whose equation is most relevant for your problem. (Yes, you &lt;em&gt;do&lt;/em&gt; need to understand what they do. Clustering is hard, it is easy to get meaningless results by just looking at the code and scores and forgetting the underlying math.)&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-19T07:09:24.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27984" PostTypeId="1" CreationDate="2018-02-19T08:56:06.050" Score="3" ViewCount="6509" Body="&lt;p&gt;I want to use this keras seq2seq example to train my model. But i dont undestand role of encoder and decoder model and why we dont use model which we trained here.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;model.fit([encoder_input_data, decoder_input_data],&#xA;  decoder_target_data,&#xA;            batch_size=batch_size,&#xA;            epochs=epochs,&#xA;            validation_split=0.2)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the second, model dont work after saving , it means that when i load model from file and try to use it.It returns invalid values differing from the values that were predicted immediately after the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please can you describe for me this two problems.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Model,save_model,load_model&#xA;from keras.layers import Input, LSTM, Dense&#xA;import numpy as np&#xA;&#xA;&#xA;batch_size = 64  # Batch size for training.&#xA;epochs = 10  # Number of epochs to train for.&#xA;latent_dim = 256  # Latent dimensionality of the encoding space.&#xA;num_samples = 100    # Number of samples to train on.&#xA;# Path to the data txt file on disk.&#xA;data_path = 'fragments1.txt'&#xA;&#xA;&#xA;# Vectorize the data.&#xA;input_texts = []&#xA;target_texts = []&#xA;input_characters = set()&#xA;target_characters = set()&#xA;with open(data_path, 'r', encoding='utf-8') as f:&#xA;    lines = f.read().split('\n')&#xA;for line in lines[: min(num_samples, len(lines) - 1)]:&#xA;    input_text, target_text = line.split('\t')&#xA;    # We use &quot;tab&quot; as the &quot;start sequence&quot; character&#xA;    # for the targets, and &quot;\n&quot; as &quot;end sequence&quot; character.&#xA;    target_text = '\t' + target_text + '\n'&#xA;    input_texts.append(input_text)&#xA;    target_texts.append(target_text)&#xA;    for char in input_text:&#xA;        if char not in input_characters:&#xA;            input_characters.add(char)&#xA;    for char in target_text:&#xA;        if char not in target_characters:&#xA;            target_characters.add(char)&#xA;&#xA;&#xA;input_characters = sorted(list(input_characters))&#xA;target_characters = sorted(list(target_characters))&#xA;num_encoder_tokens = len(input_characters)&#xA;num_decoder_tokens = len(target_characters)&#xA;max_encoder_seq_length = max([len(txt) for txt in input_texts])&#xA;max_decoder_seq_length = max([len(txt) for txt in target_texts])&#xA;&#xA;&#xA;print('Number of samples:', len(input_texts))&#xA;print('Number of unique input tokens:', num_encoder_tokens)&#xA;print('Number of unique output tokens:', num_decoder_tokens)&#xA;print('Max sequence length for inputs:', max_encoder_seq_length)&#xA;print('Max sequence length for outputs:', max_decoder_seq_length)&#xA;&#xA;input_token_index = dict(&#xA;    [(char, i) for i, char in enumerate(input_characters)])&#xA;target_token_index = dict(&#xA;    [(char, i) for i, char in enumerate(target_characters)])&#xA;&#xA;encoder_input_data = np.zeros(&#xA;    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),&#xA;    dtype='float32')&#xA;decoder_input_data = np.zeros(&#xA;    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),&#xA;    dtype='float32')&#xA;decoder_target_data = np.zeros(&#xA;    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),&#xA;    dtype='float32')&#xA;&#xA;for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):&#xA;    for t, char in enumerate(input_text):&#xA;        encoder_input_data[i, t, input_token_index[char]] = 1.&#xA;    for t, char in enumerate(target_text):&#xA;        # decoder_target_data is ahead of decoder_input_data by one timestep&#xA;        decoder_input_data[i, t, target_token_index[char]] = 1.&#xA;        if t &amp;gt; 0:&#xA;            # decoder_target_data will be ahead by one timestep&#xA;            # and will not include the start character.&#xA;            decoder_target_data[i, t - 1, target_token_index[char]] = 1.&#xA;&#xA;# Define an input sequence and process it.&#xA;encoder_inputs = Input(shape=(None, num_encoder_tokens))&#xA;encoder = LSTM(latent_dim, return_state=True)&#xA;encoder_outputs, state_h, state_c = encoder(encoder_inputs)&#xA;# We discard `encoder_outputs` and only keep the states.&#xA;encoder_states = [state_h, state_c]&#xA;&#xA;# Set up the decoder, using `encoder_states` as initial state.&#xA;decoder_inputs = Input(shape=(None, num_decoder_tokens))&#xA;# We set up our decoder to return full output sequences,&#xA;# and to return internal states as well. We don't use the&#xA;# return states in the training model, but we will use them in inference.&#xA;decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)&#xA;decoder_outputs, _, _ = decoder_lstm(decoder_inputs,&#xA;                                     initial_state=encoder_states)&#xA;decoder_dense = Dense(num_decoder_tokens, activation='softmax')&#xA;decoder_outputs = decoder_dense(decoder_outputs)&#xA;&#xA;# Define the model that will turn&#xA;# `encoder_input_data` &amp;amp; `decoder_input_data` into `decoder_target_data`&#xA;model = Model([encoder_inputs, decoder_inputs], decoder_outputs)&#xA;&#xA;# Run training&#xA;model.compile(optimizer='rmsprop', loss='categorical_crossentropy')&#xA;model.fit([encoder_input_data, decoder_input_data], decoder_target_data,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          validation_split=0.2)&#xA;# Save model&#xA;save_model(model, 'model.h5')&#xA;&#xA;# Next: inference mode (sampling).&#xA;# Here's the drill:&#xA;# 1) encode input  and retrieve initial decoder state&#xA;# 2) run one step of decoder with this initial state&#xA;# and a &quot;start of sequence&quot; token as target.&#xA;# Output will be the next target token&#xA;# 3) Repeat with the current target token and current states&#xA;&#xA;# Define sampling models&#xA;encoder_model = Model(encoder_inputs, encoder_states)&#xA;&#xA;decoder_state_input_h = Input(shape=(latent_dim,))&#xA;decoder_state_input_c = Input(shape=(latent_dim,))&#xA;decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]&#xA;decoder_outputs, state_h, state_c = decoder_lstm(&#xA;    decoder_inputs, initial_state=decoder_states_inputs)&#xA;decoder_states = [state_h, state_c]&#xA;decoder_outputs = decoder_dense(decoder_outputs)&#xA;decoder_model = Model(&#xA;    [decoder_inputs] + decoder_states_inputs,&#xA;    [decoder_outputs] + decoder_states)&#xA;&#xA;# Reverse-lookup token index to decode sequences back to&#xA;# something readable.&#xA;reverse_input_char_index = dict(&#xA;    (i, char) for char, i in input_token_index.items())&#xA;reverse_target_char_index = dict(&#xA;    (i, char) for char, i in target_token_index.items())&#xA;&#xA;&#xA;def decode_sequence(input_seq):&#xA;    # Encode the input as state vectors.&#xA;    states_value = encoder_model.predict(input_seq)&#xA;&#xA;    # Generate empty target sequence of length 1.&#xA;    target_seq = np.zeros((1, 1, num_decoder_tokens))&#xA;    # Populate the first character of target sequence with the start character.&#xA;    target_seq[0, 0, target_token_index['\t']] = 1.&#xA;&#xA;    # Sampling loop for a batch of sequences&#xA;    # (to simplify, here we assume a batch of size 1).&#xA;    stop_condition = False&#xA;    decoded_sentence = ''&#xA;    while not stop_condition:&#xA;        output_tokens, h, c = decoder_model.predict(&#xA;            [target_seq] + states_value)&#xA;&#xA;        # Sample a token&#xA;        sampled_token_index = np.argmax(output_tokens[0, -1, :])&#xA;        sampled_char = reverse_target_char_index[sampled_token_index]&#xA;        decoded_sentence += sampled_char&#xA;&#xA;        # Exit condition: either hit max length&#xA;        # or find stop character.&#xA;        if (sampled_char == '\n' or&#xA;           len(decoded_sentence) &amp;gt; max_decoder_seq_length):&#xA;            stop_condition = True&#xA;&#xA;        # Update the target sequence (of length 1).&#xA;        target_seq = np.zeros((1, 1, num_decoder_tokens))&#xA;        target_seq[0, 0, sampled_token_index] = 1.&#xA;&#xA;        # Update states&#xA;        states_value = [h, c]&#xA;&#xA;    return decoded_sentence&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46284" LastActivityDate="2019-07-23T12:51:14.907" Title="Problem with keras model loading" Tags="|machine-learning|neural-network|predictive-modeling|rnn|machine-translation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27985" PostTypeId="1" CreationDate="2018-02-19T10:06:22.550" Score="0" ViewCount="465" Body="&lt;p&gt;I have a automation system which manages files sent by multiple client inside S3. It so happens client sends file on a regular basis. Let's not consider the content of the file for time being as it can be something of new data or append on old data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically my files are arranged on a pattern: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i) client_a/data_type1/20180610/file_name1_{some_random_text}.csv&#xA;ii) client_a/data_type1/20180611/file_name1_{some_random_text}.csv&#xA;iii) client_a/data_type1/20180610/file_name2_{some_random_text}.csv&#xA;iv) client_a/data_type1/20180611/file_name2_{some_random_text}.csv&#xA;v) client_b/data_type2/20180610/file_name3_{some_random_text}.csv&#xA;vi) client_b/data_type2/20180611/file_name3_{some_random_text}.csv&#xA;vii) client_b/data_type1/20180610/file_name4_{some_random_text}.csv&#xA;viii) client_b/data_type1/20180611/file_name4_{some_random_text}.csv&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a sample of the file management. Now I want to be able to be able to identify unique files. For example case i and ii are same files received on different date. But may be separated by some_random_text (usually date or something). &#xA;The primary assumptions here are :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;File name length varies as per client and datatype &lt;/li&gt;&#xA;&lt;li&gt;Random text maybe at the first or end of file name for different files but   for not same file. i.e file_name1 will always have random text after the name&lt;/li&gt;&#xA;&lt;li&gt;File name extension will vary on different files.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So I want to build a system which takes realtime input of the files received and do an analysis of the name and path to determine if newer version of he same file is received and place it accordingly (same in the sense data may be added to the file). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have very limited understanding of machine learning algorithms as of whole and not sure which set of algorithms this requires. I am thinking of looking at clustering algorithms. I just want to sure i start in the right direction. Suggestions will be highly appreciated. &lt;/p&gt;&#xA;" OwnerUserId="46471" LastEditorUserId="924" LastEditDate="2018-02-21T07:59:37.880" LastActivityDate="2018-02-21T07:59:37.880" Title="file name pattern matching and identifying" Tags="|data|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="27986" PostTypeId="1" AcceptedAnswerId="28020" CreationDate="2018-02-19T10:08:27.240" Score="1" ViewCount="17973" Body="&lt;p&gt;I am trying to apply &lt;strong&gt;Random Forest&lt;/strong&gt; algorithm on a data set using Orange. The target variable is not set in the data set. However, I know which column is the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I specify the target variable in a &lt;strong&gt;.csv&lt;/strong&gt; file using &lt;strong&gt;Orange&lt;/strong&gt; or any other tools?&lt;/p&gt;&#xA;" OwnerUserId="46472" LastEditorUserId="46472" LastEditDate="2018-02-19T11:27:54.947" LastActivityDate="2022-11-16T17:35:40.353" Title="Orange: Data has no target variable error" Tags="|machine-learning|predictive-modeling|random-forest|orange|csv|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27987" PostTypeId="2" ParentId="17997" CreationDate="2018-02-19T12:02:42.573" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/alitouka/spark_dbscan&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/alitouka/spark_dbscan&lt;/a&gt; is n-dimensional.&#xA;you can try also WEKA, SPMF library&lt;/p&gt;&#xA;" OwnerUserId="46478" LastActivityDate="2018-02-19T12:02:42.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27988" PostTypeId="1" AcceptedAnswerId="27989" CreationDate="2018-02-19T12:51:27.787" Score="8" ViewCount="7029" Body="&lt;p&gt;I have a really large data set with mixed variables. I have converted categorical variables to numerical using &lt;code&gt;OneHotEncoding&lt;/code&gt; and it has resulted in more than a couple of thousand different features, combined that is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to apply dimensionality reduction algorithms on OneHotEncoded data which looks like &lt;code&gt;[[1. 0. 1. 0.]...[0. 0. 0. 0.]]&lt;/code&gt; or should it be done by merging with the original data set?&lt;/p&gt;&#xA;" OwnerUserId="44701" LastActivityDate="2020-02-05T21:48:52.980" Title="Applying dimensionality reduction on OneHotEncoded array" Tags="|python|scikit-learn|dimensionality-reduction|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27989" PostTypeId="2" ParentId="27988" CreationDate="2018-02-19T14:49:50.643" Score="4" Body="&lt;p&gt;Following your example, you have different points in a 4-dimensional space. So, yes! you can use any dimensionality reduction technique, from &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt; to &lt;a href=&quot;https://github.com/lmcinnes/umap&quot; rel=&quot;nofollow noreferrer&quot;&gt;UMAP&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, if your data is in a numeric format (and one-hot actually is), all the elements have the same dimensionality, and you don't have undefined values (NAN, inf), you can always use dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-02-19T14:49:50.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27991" PostTypeId="1" CreationDate="2018-02-19T15:18:25.933" Score="3" ViewCount="817" Body="&lt;p&gt;I have previously worked with GAMs, where I was trying to do regression on a log-transformed variable. The log-transformation introduced a negative bias in the average of the predicted variable, and I corrected for this by multiplying each of the predictions $\exp(\hat y)$ by the factor &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\langle \exp( \delta \hat y) \rangle$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\delta \hat y$ was the residuals from the GAM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am using XGB, and trying to do regression on a log-transformed variable $y$ once again. The predictions $\hat y$ satisfy &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac{\sum_i \hat y_i}{\sum_i y_i} = 0.999&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so overall it looks good. However, when I &lt;code&gt;exp&lt;/code&gt;-transform the variables I get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\frac{\sum_i \exp(\hat y_i)}{\sum_i \exp(y_i)} = 0.861&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is considerably worse. I suspect that this is due to the negative bias. Is there a way to correct for the bias in the XGB as in a GAM/GLM?&lt;/p&gt;&#xA;" OwnerUserId="46487" LastActivityDate="2018-12-19T19:02:50.497" Title="Correcting log-bias in the output of an XGB" Tags="|machine-learning|xgboost|bias|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27992" PostTypeId="1" CreationDate="2018-02-19T15:25:18.180" Score="7" ViewCount="20075" Body="&lt;p&gt;ERROR SUMMARY:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError: Expected binary or unicode string, got [&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;BACKGROUND:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have several features that are histories of user activity. I am trying to predict whether a given user will take an action (represented by a 0 or 1 in my &lt;code&gt;y_train&lt;/code&gt; list below) based off of their histories of different types of actions. For example, one feature might be &lt;code&gt;button_A_click_per_day&lt;/code&gt; and the history for a given user would be a vector of button A clicks per day over the past 365 days. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The relevant snippets of my code is as follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;# Build feature columns for classifier&#xA;feature_columns = []&#xA;for key in X_train:&#xA;    col = tf.feature_column.numeric_column(&#xA;        key=key, &#xA;        shape=max_width,&#xA;    )&#xA;    feature_columns.append(col)&#xA;&#xA;# Build classifier&#xA;classifier = tf.estimator.DNNClassifier(&#xA;    feature_columns=my_feature_columns,&#xA;    hidden_units=[10, 10],&#xA;    n_classes=2,&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And X_train is structured as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; X_train['&amp;lt;feature_name&amp;gt;']&#xA;129     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...&#xA;...&#xA;1294    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...&#xA;860     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...&#xA;Name: &amp;lt;feature_name&amp;gt;, Length: 1377, dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My train input function is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train_input_fn(features, labels, batch_size):&#xA;    dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))&#xA;    dataset = dataset.shuffle(buffer_size=1000).repeat(count=None).batch(batch_size)&#xA;    return dataset.make_one_shot_iterator().get_next()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, when issuing the following command... &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier.train(&#xA;    # input_fn is a fn that takes not arguments and returns an iterator.&#xA;    input_fn=lambda: train_input_fn(X_train, y_train, batch_size=100),&#xA;    steps=1000,&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/tensorflow/python/util/compat.py in as_bytes(bytes_or_text, encoding)&#xA;     63   else:&#xA;     64     raise TypeError('Expected binary or unicode string, got %r' %&#xA;---&amp;gt; 65                     (bytes_or_text,))&#xA;     66 &#xA;     67 &#xA;&#xA;TypeError: Expected binary or unicode string, got [0.0, 0.0, 0...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;MY QUESTIONS:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do you know what I'm doing wrong to get this error?&lt;/li&gt;&#xA;&lt;li&gt;Is there a better way to model features that are histories?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46488" LastActivityDate="2019-12-16T05:29:36.817" Title="TypeError: Expected binary or unicode string, got [" Tags="|predictive-modeling|tensorflow|model-selection|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="27993" PostTypeId="2" ParentId="27957" CreationDate="2018-02-19T17:05:46.133" Score="18" Body="&lt;p&gt;Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form &lt;a href=&quot;http://www.statsmodels.org/dev/contrasts.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; &quot;A categorical variable of K categories, or levels, usually enters a regression as a sequence of K-1 dummy variables. This amounts to a linear hypothesis on the level means.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is already discussed at this very nice &lt;a href=&quot;https://stats.stackexchange.com/questions/115049/why-do-we-need-to-dummy-code-categorical-variables&quot;&gt;stats.stackexchange answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was told there is an advanced course by Yandex in Coursera that covers this subject in more details if you still have doubts, see &lt;a href=&quot;https://www.coursera.org/learn/competitive-data-science&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. Note you can always audit the course content for free. ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another nice post if you want a thorough explanation with lots of examples with statistical perspective and not being limited to only dummy coding, see &lt;a href=&quot;https://stats.idre.ucla.edu/r/library/r-library-contrast-coding-systems-for-categorical-variables/&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; from UCLA (in R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that if you using &lt;code&gt;pandas.get_dummies&lt;/code&gt;, there is a parameter i.e. &lt;code&gt;drop_first&lt;/code&gt; so that whether to get k-1 dummies out of k categorical levels by removing the first level. Please note &lt;code&gt;default = False&lt;/code&gt;, meaning that the reference is not dropped and k dummies created out of k categorical levels!&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-02-20T07:22:55.023" LastActivityDate="2018-02-20T07:22:55.023" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="27994" PostTypeId="1" CreationDate="2018-02-19T17:20:47.160" Score="5" ViewCount="148" Body="&lt;p&gt;I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that I train, test and validate the classifier on one month of data, and use it to classify another month. Whether I repeatedly add the next month of data to the training dataset and upset my classifier is open for debate.&#xA;I've spent a little while on this and have something that works well but it works better on some months than others. I am using either RandomForest or LinearSVC(penalty=’l1’) because I have a lot of features, around 400,000 observations and limited computing power.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not have much insight into the processes that produce the input dataset and based on previous analysis, I do not trust their quality control. What worries me is that one month I might receive a dataset that is very different to the data I have been training on and I wouldn’t really know unless the distribution of classes was quite different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure that I cannot be the first person to have this worry but I can’t find much information about it, so maybe I am misguided here. I’ve found the Population Stability Index, which is the right kind of idea, but I am unsure about it because it is dependent on bin size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question(s): &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;How is a worry like this normally dealt with?&lt;/li&gt;&#xA;&lt;li&gt;Can you point me to some ideas/case studies that will help me?&lt;/li&gt;&#xA;&lt;li&gt;Is there anything I can do on the classification side that will make&#xA;my classifier more robust to population instability?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46490" LastActivityDate="2020-04-13T14:02:40.397" Title="Dealing with population instability" Tags="|machine-learning|classification|bigdata|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="27995" PostTypeId="1" CreationDate="2018-02-19T20:28:24.677" Score="4" ViewCount="133" Body="&lt;p&gt;I had a brief experience with machine learning by using a clustering algorithm, i also read the basic ideas and calculations of a simple classification algorithm.&#xA;Now, i would read more about &amp;quot;machine learning&amp;quot; and I found many similar definitions like the following:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;Machine learning is the science of getting computers to act without being explicitly programmed...&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;Machine learning is a type of artificial intelligence (AI) that allows software applications to become more accurate in predicting outcomes without being explicitly programmed... &amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;My understanding is that the ability to learn and adapting deduction (output) without reprogrammation is the main idea, and based on my personal understanding, this &amp;quot;adaptation&amp;quot; is only possible with &amp;quot;supervised algorithm&amp;quot; with a new training which can permit a change/adaptation/improvement on the output model with the same program and source code.&lt;/p&gt;&#xA;&lt;p&gt;So based on my understanding again, this &amp;quot;adaptation&amp;quot; and &amp;quot;learning&amp;quot; definition doesn't fit unsupervised machine learning algorithms, since the model with all calculations is fixed and implemented! Any change will need an update on source code!&lt;/p&gt;&#xA;&lt;p&gt;Therefore, I would have corrections to my misinterpretation, and more clarifications to have a better understanding of &amp;quot;machine learning&amp;quot; and unsupervised/supervised learning relation.&lt;/p&gt;&#xA;" OwnerUserId="46493" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-07-18T20:53:31.740" Title="When using an unsupervised alogirthm, what is the &quot;learning&quot; part since it belong to machine learning field?" Tags="|machine-learning|unsupervised-learning|supervised-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="27997" PostTypeId="2" ParentId="27995" CreationDate="2018-02-19T21:03:25.303" Score="1" Body="&lt;p&gt;You said:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;and based on my personal understanding, this &quot;adaptation&quot; is only possible with &quot;supervised algorithm&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, it is not true. Let's consider a clustering method as an unsupervised algorithm like K-means. Each iteration in K-means algorithm is a basis for the next iteration of the K-means up to the convergence of the algorithm. Hence, it means we are learning the structure of the data in each iteration up to reaching a specific target value. Although it is unsupervised, we are learning the structure of the data in each iteration. Indeed, it is a kind of (unsupervised) learning at all. &lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-02-19T21:03:25.303" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27998" PostTypeId="1" AcceptedAnswerId="28019" CreationDate="2018-02-19T21:40:53.263" Score="1" ViewCount="57" Body="&lt;p&gt;Let's suppose I have a big list of words. I want to turn this list into a vector space of dimension $N$ such that each word is a vector in this vector space. But I have no idea how to go about with that. Some questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the list enough? For each element of the list, do I need $x$ example sentences also?&lt;/li&gt;&#xA;&lt;li&gt;How does the computer deduce the dimensions of the vector space from the list/corpus?&lt;/li&gt;&#xA;&lt;li&gt;Is there a way to figure out whether the dimensions of the vector space correspond to something in English?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46501" LastActivityDate="2018-02-20T10:43:15.313" Title="Help me understand how word-as-vector representations are constructed" Tags="|word2vec|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="27999" PostTypeId="2" ParentId="27939" CreationDate="2018-02-19T22:34:10.453" Score="1" Body="&lt;p&gt;Based on how much i got to know i propose the correct scheme would be third one i.e. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss) words from each article (for the sake of simplicity let us assume batch size = m)&lt;/li&gt;&#xA;&lt;li&gt;Set the initial hidden state H0 = [0,0,..,0]&lt;/li&gt;&#xA;&lt;li&gt;Calculate loss and gradient on this batch and update the parameters&lt;/li&gt;&#xA;&lt;li&gt;We move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration&lt;/li&gt;&#xA;&lt;li&gt;Do this to the end&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Why we wont take any but &lt;strong&gt;Scheme 1&lt;/strong&gt; and &lt;strong&gt;Scheme 3&lt;/strong&gt; is because:&#xA;H0 should only be initialized with a vector of zero when there is no context available (if you have some additional information, why not use it?!). The objective is to maximize the probability of each article and the sentences are not independent. Also, there is no reason to not treat period &quot;.&quot; as a step in itself. We do calculate its word embedding too&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not &lt;strong&gt;Scheme 1&lt;/strong&gt; - Because of &lt;a href=&quot;https://en.wikipedia.org/wiki/Catastrophic_interference&quot; rel=&quot;nofollow noreferrer&quot;&gt;catastrophic interference&lt;/a&gt; ( As the network in &lt;strong&gt;Scheme 1&lt;/strong&gt; fed in same input again and again ) combined with slow learning it would result in&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2018-02-19T22:34:10.453" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28000" PostTypeId="1" CreationDate="2018-02-20T00:25:18.850" Score="1" ViewCount="260" Body="&lt;p&gt;Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the proposed architecture - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/W1jaE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W1jaE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say I have a pre-trained model hosted as a service. The part I'm struggling with is, how do I get the user features from the database in realtime from the RiderID to be used by the prediction service (XGBoost Model)? I'm guessing a lookup in a SQL database will take too long, considering I have 1M+ users and rides. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="46504" LastActivityDate="2023-09-05T17:06:10.760" Title="Retrieve user features in real time from UserId for prediction" Tags="|machine-learning|predictive-modeling|apache-spark|model-selection|ensemble-modeling|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28001" PostTypeId="1" AcceptedAnswerId="28051" CreationDate="2018-02-20T03:54:49.393" Score="1" ViewCount="407" Body="&lt;p&gt;I am trying to import&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;from statsmodels.stats.outliers_influence import variance_inflation_factor&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is working fine upto Scipy 0.19.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But , with Python 3.6.3 ,it's failing due to unavailability of &lt;code&gt;ss&lt;/code&gt; module in Scipy 1.0.0.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;~\Anaconda3\lib\site-packages\statsmodels\regression\linear_model.py in &amp;lt;module&amp;gt;()&#xA;     41 from scipy.linalg import toeplitz&#xA;     42 from scipy import stats&#xA;---&amp;gt; 43 from scipy.stats.stats import ss&#xA;     44 from scipy import optimize&#xA;     45 from scipy.stats import chi2&#xA;&#xA;ImportError: cannot import name 'ss'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone please guide me how to do VIF analysis using Scipy1.0.0 ?&lt;/p&gt;&#xA;" OwnerUserId="44477" LastActivityDate="2018-02-20T16:44:34.993" Title="How to calculate VIF in python 3.6 with scipy 1.0.0?" Tags="|python|scipy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28002" PostTypeId="1" CreationDate="2018-02-20T04:41:55.297" Score="1" ViewCount="269" Body="&lt;p&gt;I am using Keras to make a &amp;quot;set&amp;quot; identifier for the card game &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Set_(game)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Set&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Here is my script (some code may be unnecessary but was used during experimentation):&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Conv2D&#xA;from keras.layers import MaxPooling2D&#xA;from keras.layers import Activation&#xA;from keras.layers import Flatten&#xA;from keras.layers import Dense&#xA;from keras.layers import Dropout&#xA;from keras.callbacks import EarlyStopping&#xA;from keras.optimizers import SGD&#xA;from keras.utils import to_categorical&#xA;from keras.layers.advanced_activations import LeakyReLU&#xA;from keras.regularizers import l2&#xA;&#xA;import os&#xA;import glob&#xA;import shutil&#xA;&#xA;import numpy as np&#xA;from keras.preprocessing.image import load_img, img_to_array&#xA;from keras.applications.imagenet_utils import preprocess_input&#xA;&#xA;earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5,&#xA;                          verbose=1, mode='auto')&#xA;&#xA;callbacks_list = [earlystop]&#xA;&#xA;# opt = SGD(lr=0.01)&#xA;&#xA;setIdentifier = Sequential()&#xA;&#xA;setIdentifier.add(Conv2D(32, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))&#xA;&#xA;setIdentifier.add(MaxPooling2D(pool_size = (2, 2)))&#xA;&#xA;setIdentifier.add(Conv2D(64, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))&#xA;&#xA;# setIdentifier.add(Conv2D(64, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))&#xA;&#xA;setIdentifier.add(MaxPooling2D(pool_size = (2, 2)))&#xA;&#xA;setIdentifier.add(Flatten())&#xA;&#xA;setIdentifier.add(Dense(units=256, activation = 'relu'))&#xA;&#xA;setIdentifier.add(Dropout(rate=0.5))&#xA;&#xA;setIdentifier.add(Dense(units = 2, activation = &amp;quot;softmax&amp;quot;))&#xA;&#xA;setIdentifier.compile(optimizer = &amp;quot;adam&amp;quot;, loss='categorical_crossentropy', metrics = ['accuracy'])&#xA;&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;&#xA;train_datagen = ImageDataGenerator(rescale=1./255,&#xA;                shear_range = 0.2, &#xA;                zoom_range = 0.2, &#xA;                horizontal_flip = True)&#xA;&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;&#xA;training_set = train_datagen.flow_from_directory('Data_Training', &#xA;                                                target_size = (64, 64),&#xA;                                                batch_size = 32,&#xA;                                                class_mode = 'categorical')&#xA;&#xA;test_set = test_datagen.flow_from_directory('Data_Testing',&#xA;                                            target_size = (64, 64),&#xA;                                            batch_size = 32,&#xA;                                            class_mode = 'categorical')&#xA;&#xA;setIdentifier.fit_generator(training_set, &#xA;                                steps_per_epoch = 486,&#xA;                                epochs = 50, &#xA;                                validation_data = test_set,&#xA;                                validation_steps = 60,&#xA;                                callbacks = callbacks_list)&#xA;&#xA;setIdentifier.save(&amp;quot;setIdentifier.h5&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The file &lt;code&gt;Data_Training&lt;/code&gt; has a folder &lt;code&gt;notSet&lt;/code&gt; and a folder &lt;code&gt;Set&lt;/code&gt;, and the file &lt;code&gt;Data_Testing&lt;/code&gt; likewise. There are two times as many &lt;code&gt;notSets&lt;/code&gt; than &lt;code&gt;Sets&lt;/code&gt; in each category.&lt;/p&gt;&#xA;&lt;p&gt;No matter what I do, in both training and testing phases, the output is always &lt;code&gt;notSet&lt;/code&gt;. I am not sure what is the cause for this.&lt;/p&gt;&#xA;&lt;p&gt;I have tried a &lt;code&gt;binary_classifier&lt;/code&gt; with the sigmoid activation function and final density layer with 1 node. I have tried changing activation functions and using &lt;code&gt;leakyRelus&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I have double checked my training steps and test steps.&lt;br /&gt;&#xA;I have &lt;span class=&quot;math-container&quot;&gt;$5184\text{ sets }+ 2 \cdot 5184\text{ notSet = }15552$&lt;/span&gt; train images, which when divided by a batch size of &lt;span class=&quot;math-container&quot;&gt;$32$&lt;/span&gt; gets me &lt;span class=&quot;math-container&quot;&gt;$486$&lt;/span&gt; steps.&lt;br /&gt;&#xA;I have &lt;span class=&quot;math-container&quot;&gt;$648\text{ Sets }+ 2 \cdot 648\text{ notSet = }60.75$&lt;/span&gt; &amp;gt;&amp;gt; rounded to &lt;span class=&quot;math-container&quot;&gt;$60$&lt;/span&gt; validation steps.&lt;/p&gt;&#xA;&lt;p&gt;I am confused by what is going on with the neural network - why isn't anything approving and why is it labelling everything with one class?&lt;/p&gt;&#xA;&lt;p&gt;For more info, &lt;a href=&quot;https://github.com/Keyan-Rhm/MLWithSet&quot; rel=&quot;nofollow noreferrer&quot;&gt;here is the Github repo&lt;/a&gt; which includes the training images.&lt;/p&gt;&#xA;&lt;p&gt;Any help is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="46509" LastEditorUserId="98307" LastEditDate="2020-08-12T12:50:50.500" LastActivityDate="2020-08-12T12:50:50.500" Title="Keras val_acc unchanging when training (same label assigned to all images)" Tags="|machine-learning|python|neural-network|keras|cnn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28003" PostTypeId="1" CreationDate="2018-02-20T04:55:34.080" Score="5" ViewCount="19765" Body="&lt;p&gt;I have a regression problem which I have to predict 3 numerical values from a provided data. For example let's say I have a data set containing &lt;code&gt;X1,X2,X3,X4,X5,X6...X100,Y1,Y2,Y3&lt;/code&gt; columns. In this case I have to predict &lt;code&gt;Y1,Y2,Y3&lt;/code&gt; values. Is there are any way to construct the model to get all the outputs at the same time using &lt;a href=&quot;https://keras.io/&quot; rel=&quot;noreferrer&quot;&gt;Keras&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43958" LastEditorUserId="43958" LastEditDate="2018-02-20T05:08:24.273" LastActivityDate="2020-04-09T05:40:36.950" Title="Get multiple output from Keras" Tags="|machine-learning|python|neural-network|keras|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28004" PostTypeId="2" ParentId="28003" CreationDate="2018-02-20T05:24:45.770" Score="3" Body="&lt;p&gt;You can set your output layer to have 3 nodes. When you train, set your output to be a vector containing &lt;span class=&quot;math-container&quot;&gt;$[Y_1, Y_2, Y_3]$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Dense(3, activation='linear'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In above it is not mentioned that we have a linear regression, also adding dense layer will let model take different &lt;span class=&quot;math-container&quot;&gt;$Y_i$&lt;/span&gt; simultaenously in beginnging, I wil still prefer \textbf{Model} Api from keras, as multioutput model&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="80382" LastEditDate="2019-08-30T10:37:27.980" LastActivityDate="2019-08-30T10:37:27.980" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28005" PostTypeId="1" CreationDate="2018-02-20T05:56:32.693" Score="2" ViewCount="1022" Body="&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to classify whether a document is checked or not checked. I have only text dataset for check document and if a document does not consist the check data it can automatically be classified as not checked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that it can be doe using OneClass SVM. Below are the dataset and program implemented by me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;import pandas as pd&#xA;from nltk.stem import PorterStemmer&#xA;from nltk.corpus import stopwords&#xA;&#xA;from sklearn.model_selection import cross_val_score&#xA;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;ps = PorterStemmer()&#xA;&#xA;stop = stopwords.words('english')&#xA;data_test = pd.DataFrame({&quot;Keywords&quot;:[&quot;Barge ID&quot;, &quot;Check&quot;]})&#xA;data = pd.DataFrame({&quot;Keywords&quot;:[&quot;Vessel Name&quot;, &quot;Barge ID&quot;, &quot;Barge/Vessel&quot;, &quot;VESSEL&quot;]})&#xA;&#xA;data['Keywords'] = data['Keywords'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))&#xA;data['Keywords'] = data['Keywords'].str.replace('[^\w\s]',' ').replace('\s+',' ')&#xA;&#xA;data_test['Keywords'] = data_test['Keywords'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))&#xA;data_test['Keywords'] = data_test['Keywords'].str.replace('[^\w\s]',' ').replace('\s+',' ')&#xA;&#xA;vectorizer = TfidfVectorizer( max_features = 200, ngram_range = ( 1,3 ), sublinear_tf = True )&#xA;data_features = vectorizer.fit_transform( data['Keywords'] )&#xA;&#xA;clf = svm.OneClassSVM(nu=0.1, kernel=&quot;rbf&quot;, gamma=0.1)&#xA;clf.fit(data_features)&#xA;&#xA;test_data_features = vectorizer.transform(data_test['Keywords'])&#xA;&#xA;print(clf.predict(test_data_features))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried it for a small sample code but then to it did not work. it is wrongly indicating as -1 for Barge ID and 1 for Check&lt;/p&gt;&#xA;" OwnerUserId="15902" LastActivityDate="2022-03-25T14:32:15.910" Title="OneClass SVM classification for text data" Tags="|python|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28006" PostTypeId="1" CreationDate="2018-02-20T06:15:04.687" Score="81" ViewCount="31908" Body="&lt;p&gt;What are the differences, if any, between a &quot;data scientist&quot; and a &quot;machine learning engineer&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over the past year or so &quot;machine learning engineer&quot; has started to show up a lot in job postings. This is particularly noticeable in San Francisco, which is arguably where the term &quot;data scientist&quot; originated. At one point &quot;data scientist&quot; overtook &quot;statistician&quot;, and I'm wondering if the same is now slowly beginning to happen to &quot;data scientist&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Career advice is listed as off-topic on this site, but I view my question as highly relevant since I'm asking about definitions; I'm not asking about recommendations given my own career trajectory or personal circumstances like other off-topic questions have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is on-topic because it might someday have significant implications for many users of this site. In fact, this stack-exchange site might not exist if the &quot;statistician&quot; vs &quot;data scientist&quot; evolution had not occurred. In that sense, this is a rather pertinent, potentially existential question. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastEditorUserId="29575" LastEditDate="2018-02-20T13:27:50.850" LastActivityDate="2023-08-10T16:18:29.407" Title="Data scientist vs machine learning engineer" Tags="|machine-learning|" AnswerCount="9" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28007" PostTypeId="2" ParentId="27991" CreationDate="2018-02-20T06:20:12.987" Score="1" Body="&lt;p&gt;From this perspective, modern ML regression techniques are not different to classic ones. A mean unbiased model on the log scale will automatically be mean biased after retransformation (by Jensen's inequality). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the aim of your analysis, applying some retransformation bias correction factor might be useful as well for algorithms like XGBoost. (Be careful with models that impose L1/L2 penalties on the intercept though - a rare occasion.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The specific correction factor can be calculated directly from the residuals in the same way as for OLS.&lt;/p&gt;&#xA;" OwnerUserId="36977" LastActivityDate="2018-02-20T06:20:12.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28008" PostTypeId="2" ParentId="28006" CreationDate="2018-02-20T06:33:50.403" Score="12" Body="&lt;p&gt;[Completely a personal opinion]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the term 'Data Scientist' overtook 'Statistician', it is more towards sounding cool, rather than any major difference. Similarly, the term 'Deep Learning'. It is just neural networks (which is another Machine Learning algorithm) with a couple of more layers. No one can explain when a particular neural net can be called DL, rather than ML, cause the definition itself is fuzzy. So, is the term 'Data Scientist'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as companies are adopting the DevOps mindset to data science, the term ML Engineer evolved. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is the DevOps mindset to data science?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where you build the model, deploy it and also expected to maintain it in production. This helps in avoiding a lot of friction in software teams.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[PS: DevOps is a way of doing software, more like a philosophy. So, using it as a designation, again confuses me].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, ML engineers are supposed to know the nuances of systems engineering, ML, and stats (obviously). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A vague generalization would be Data Engineer + Data Scientist = ML Engineer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, the designations in this space are becoming vague day by day, and the term 'Statistician' is becoming more and more relevant (the irony!).&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2018-02-20T13:50:43.287" LastActivityDate="2018-02-20T13:50:43.287" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28009" PostTypeId="2" ParentId="27995" CreationDate="2018-02-20T08:11:18.373" Score="1" Body="&lt;p&gt;Supervised learning refers to learning a concept from examples. Those examples typically require work from a human, which is usually &quot;expensive&quot;. In fact a huge part of practical ML-work is to come up with smart ways of obtaining training data (think of reCaptcha's for instance).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Un&lt;/strong&gt;supervised learning is learning without labeled data, that is all there is to it. Clustering is one example, PCA is another and autoencoding is the newest hottest thing (imho). In some sense unsupervised learning try to compress the data (by finding subspaces of high dimensional representations along the lines of which the data is ordered, or &lt;em&gt;manifolds&lt;/em&gt;): You are trying to formulate concepts that describe the data on a higher level, this involves finding smarter descriptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be somewhat of a leap from K-means, but consider this: I could easily describe a photograph to you using a set of abstractions (fi: I see a dog with a bone -&gt; 23 bytes). This would give you a picture of the scene without the need of me sending you the full dataset (fi: 1080 * 768 points ~ 10^6 bytes). Conceptually this would be somewhat similar to: 4 examples of cluster one and one example of cluster 4, instead of sending 5 complete instances of, say, 10.000 features.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastEditorUserId="27432" LastEditDate="2018-02-20T09:21:36.677" LastActivityDate="2018-02-20T09:21:36.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28010" PostTypeId="2" ParentId="27985" CreationDate="2018-02-20T08:30:45.427" Score="0" Body="&lt;p&gt;You are looking for the python packages glob and re. I suggest you model the  the logic by hand, of the hip it does't sound like something you want to have searched by a machine learning algorithm.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-20T08:30:45.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28011" PostTypeId="2" ParentId="28006" CreationDate="2018-02-20T08:57:07.257" Score="65" Body="&lt;p&gt;Good question. Actually there is a lot of confusion on this subject, mainly because both are quite new jobs. But if we focus on the semantics, the real meaning of the jobs become clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beforehand is better to compare apples with apples, talking about a single subject, the Data. Machine Learning and its sub-genre (Deep Learning, etc.) are just one aspect of the Data World, together with the statistic theories, the data acquisition (DAQ), the processing (which can be non-machine learning driven), the interpretation of the results, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for my explanation, I will broad the Machine Learning Engineer role to the one of Data Engineer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Science is about experiment, trials and fails, theory building, phenomenological understanding.&#xA;Engineering is about work on what science already knows, perfecting it and carry to the &quot;real world&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think about a proxy: what is the difference between a nuclear scientist and a nuclear engineer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nuclear scientist is the one which know the science behind the atom, the interaction between them, the one which wrote the recipe which allow to get energy from the atoms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nuclear engineer is the guy charged to take the recipe of the scientist, and carry it to the real world. So it's knowledge about the atomic physics is quite limited, but he also know about materials, buildings, economics, and whatever else useful to build a proper nuclear plant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming back to the Data world, here another example: the guys which developed Convolutional Neural Networks (Yann LeCun) is a Data Scientist, the guy which deploy the model to recognize faces in pictures is a Machine Learning Engineer. The guy responsible of the whole process, from the data acquisition to the registration of the .JPG image, is a Data Engineer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, basically, 90% of the Data Scientist today are actually Data Engineers or Machine Learning Engineers, and 90% of the positions opened as Data Scientist actually need Engineers. An easy check: in the interview, you will be asked about how many ML models you deployed in production, not on how many papers on new methods you published.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, when you see announces about &quot;Machine Learning Engineer&quot;, that means that the recruiters are well aware of the difference, and they really need someone able to put some model in production.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="-1" LastEditDate="2019-10-03T20:10:16.887" LastActivityDate="2019-10-03T20:10:16.887" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="28012" PostTypeId="2" ParentId="27918" CreationDate="2018-02-20T09:03:48.983" Score="1" Body="&lt;p&gt;I still haven't figured out what the previously posted code is doing wrong. However, manually populating the alphas array gives me results that are close to the original figure.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt &#xA;from sklearn import linear_model&#xA;from sklearn.linear_model import Ridge&#xA;from sklearn.preprocessing import PolynomialFeatures &#xA;from sklearn.pipeline import make_pipeline&#xA;from sklearn.metrics import mean_squared_error&#xA;np.random.seed(12344)&#xA;x_1000 = np.linspace(0, 1, 1000)&#xA;x_train = np.linspace(0, 1, 10)&#xA;noise_10 = np.random.normal(0, 0.3, 10)&#xA;y_train = np.sin(2*np.pi*x_train) + noise_10&#xA;x_train = x_train[:, np.newaxis]&#xA;x_test = np.linspace(0, 1, 100)&#xA;noise_100 = np.random.normal(0, 0.3, 100)&#xA;y_test = np.sin(2*np.pi*x_test) + noise_100&#xA;x_test = x_test[:, np.newaxis]&#xA;low_alpha = make_pipeline(PolynomialFeatures(degree = 9), Ridge(alpha=np.exp(-18)))&#xA;low_alpha.fit(x_test, y_test)&#xA;plt.figure(1)&#xA;plt.plot(x_train, low_alpha.predict(x_train), label = 'alpha = ln(-18)')&#xA;plt.plot(x_1000, np.sin(2*np.pi*x_1000))&#xA;plt.legend()&#xA;plt.show&#xA;high_alpha = make_pipeline(PolynomialFeatures(degree = 9),&#xA;              Ridge(alpha=np.exp(0)))&#xA;high_alpha.fit(x_test, y_test)&#xA;plt.figure(2)&#xA;plt.plot(x_train, high_alpha.predict(x_train), label = 'alpha = 1')&#xA;plt.plot(x_1000, np.sin(2*np.pi*x_1000))&#xA;plt.legend()&#xA;plt.show&#xA;alphas = np.array([np.exp(-30), np.exp(-29), np.exp(-28), np.exp(-27), np.exp(-26), np.exp(-25), np.exp(-24), np.exp(-23), np.exp(-22), np.exp(-21), np.exp(-20), np.exp(-19), np.exp(-18), np.exp(-17), np.exp(-16), np.exp(-15), np.exp(-14), np.exp(-13), np.exp(-12), np.exp(-11), np.exp(-10), np.exp(-9), np.exp(-8), np.exp(-7), np.exp(-6), np.exp(-5), np.exp(-4), np.exp(-3), np.exp(-2), np.exp(-1), np.exp(-1), np.exp(0), np.exp(1)])&#xA;&#xA;test_errors = []&#xA;train_errors = []&#xA;&#xA;for a in np.nditer(alphas):&#xA;    ridge = make_pipeline(PolynomialFeatures(degree = 9),&#xA;              Ridge(alpha=a))&#xA;    ridge.fit(x_train, y_train)&#xA;    mse_train = mean_squared_error(y_train, ridge.predict(x_train))&#xA;    mse_test = mean_squared_error(y_test, ridge.predict(x_test))&#xA;    train_errors.append(np.sqrt(mse_train))&#xA;    test_errors.append(np.sqrt(mse_test))&#xA;plt.figure(3)&#xA;plt.plot(alphas, test_errors, 'g^', label = 'Test Error')&#xA;plt.plot(alphas, train_errors, 'bs', label = 'Train Error')&#xA;plt.xscale('log')&#xA;plt.xlabel('Regression coefficient Lambda')&#xA;plt.ylabel('Residuals')&#xA;plt.legend()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output (only for the third figure) is shown below: &lt;a href=&quot;https://i.stack.imgur.com/dBKRb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dBKRb.png&quot; alt=&quot;Reproduction of Figure from Bishop&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46396" LastActivityDate="2018-02-20T09:03:48.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28013" PostTypeId="1" CreationDate="2018-02-20T09:36:39.847" Score="0" ViewCount="1453" Body="&lt;p&gt;I am working on a clustering problem. I am not able to find the right similarity metric for my system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have n nodes with ordered vector (eg: [1,0,0,0,1,0,.....] &quot;1&quot; represents represent presence of ith object and &quot;0&quot; represents absence of the object). I want to cluster them.&lt;/p&gt;&#xA;" OwnerUserId="46522" LastActivityDate="2018-02-21T11:37:22.347" Title="Similarity metric for clustering" Tags="|machine-learning|clustering|similarity|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28015" PostTypeId="2" ParentId="27637" CreationDate="2018-02-20T10:00:04.927" Score="1" Body="&lt;p&gt;The &quot;optimal&quot; stopping point for XGBoost really depends on the data you feed into it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using Chunk N and Chunk N+1 for instance, consider the two scenarios:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;data in Chunk N and Chunk N+1 is &lt;strong&gt;very different&lt;/strong&gt; - it's likely&#xA;that the optimal stopping points for each chunk are very different.&lt;/li&gt;&#xA;&lt;li&gt;data in Chunk N and Chunk N+1 is &lt;strong&gt;very similar&lt;/strong&gt; - it's likely that the optimal stopping points for each chunk are quite similar.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would suggest using a different approach - so for each chunk:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use the &lt;code&gt;xgb.cv&lt;/code&gt; function to first determine the optimal stopping point&lt;/li&gt;&#xA;&lt;li&gt;use the results from &lt;code&gt;xgb.cv&lt;/code&gt; in &lt;code&gt;xgb.train&lt;/code&gt; [i.e. set &lt;code&gt;nrounds&lt;/code&gt; = &lt;code&gt;xgb.cv$best.iteration&lt;/code&gt;]&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As a side-note, have you explored any other techniques / tools to handle large data sets? In R, there are packages designed to tackle similar problems - e.g.&lt;code&gt;bigmemory&lt;/code&gt; and &lt;code&gt;ff&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-20T10:00:04.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28016" PostTypeId="2" ParentId="25348" CreationDate="2018-02-20T10:02:06.797" Score="1" Body="&lt;p&gt;If you are using R, have you considered the &lt;code&gt;bigmemory&lt;/code&gt; and &lt;code&gt;ff&lt;/code&gt; packages?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't have much experience using these myself, but would be interested to see if they help with the issue at hand.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-20T10:02:06.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28017" PostTypeId="2" ParentId="27985" CreationDate="2018-02-20T10:25:44.193" Score="1" Body="&lt;p&gt;This does not sound like a machine learning problem. It seems to me that you could come up with a list of rules that determine whether two files belong together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As S van Balen said, you should take a look at the &lt;code&gt;glob&lt;/code&gt; and &lt;code&gt;re&lt;/code&gt; packages in Python. (If you are on Linux, a Bash script would probably be much more elegant than a solution in Python).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following links might be helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Walk through a directory and its subdirectories to get a list of all files: &lt;a href=&quot;https://stackoverflow.com/a/954522/4743630&quot;&gt;https://stackoverflow.com/a/954522/4743630&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Group strings in a list based on partial matches (for instance: turn &lt;code&gt;['file_name1_xyz', 'file_name2_xyz', 'file_name1_abc', 'file_name2_def']&lt;/code&gt; into &lt;code&gt;[ ['file_name1_xyz', 'file_name1_abc'], ['file_name2_xyz', 'file_name2_def'] ]&lt;/code&gt;): &lt;a href=&quot;https://stackoverflow.com/questions/40167651/python-iterate-through-a-list-of-strings-and-group-partial-matching-strings&quot;&gt;https://stackoverflow.com/questions/40167651/python-iterate-through-a-list-of-strings-and-group-partial-matching-strings&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You mention that files may have different file types. Be careful when converting between types: Make sure for instance that separators and quote characters are treated correctly.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-20T10:25:44.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28018" PostTypeId="2" ParentId="28013" CreationDate="2018-02-20T10:29:21.013" Score="1" Body="&lt;p&gt;Cosine similarity is a popular choice: &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Cosine_similarity&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-20T10:29:21.013" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28019" PostTypeId="2" ParentId="27998" CreationDate="2018-02-20T10:43:15.313" Score="1" Body="&lt;p&gt;I am assuming that you mean a vector representation of words, not to be confused by the vector representation produced in a bag of words approach that represent a document in vector space. Word2vec is an approach in which you train a model to represent words as a function of the provided context. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answers that follow are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;1) No: You'll need some representation of context in which the word is used. For example: Skip-grams.&lt;/li&gt;&#xA;&lt;li&gt;2) No: That is a user defined parameter&lt;/li&gt;&#xA;&lt;li&gt;3) &amp;lt; I do not understand this question &gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-20T10:43:15.313" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28020" PostTypeId="2" ParentId="27986" CreationDate="2018-02-20T10:48:03.657" Score="3" Body="&lt;p&gt;You can specify the target by choosing target in the feature column &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lVKTu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lVKTu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-02-20T10:48:03.657" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28021" PostTypeId="2" ParentId="27940" CreationDate="2018-02-20T10:49:29.650" Score="0" Body="&lt;p&gt;You could try using word2vec: It orders words in vector space according to context, it allows you to retrieve words with similar context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find pretrained models on the web.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-20T10:49:29.650" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28022" PostTypeId="1" AcceptedAnswerId="28271" CreationDate="2018-02-20T10:50:17.893" Score="1" ViewCount="1022" Body="&lt;p&gt;A client would like to sort out his filesystem (~ 1,000,000,000 files), which has been fed by numerous workers over the years, each with their own unknown naming convention, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;[DATE]-[CLIENT]-[FILENAME]&lt;/li&gt;&#xA;&lt;li&gt;[TYPE]-[CLIENT].[DATE][FILENAME]&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here are four examples (out of ~1,000,000,000 files) to make things clearer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;JPM_TPD0001662_2009124012302000451.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;JPMF_STA_1712010832_18001_LUX_approval.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CHACN05CTRP_00111.001.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CHACN63CJO1_00018.001.pdf&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The purpose is to find out patterns in the naming conventions, but I can't use regular expressions, since the conventions are unknown a priori.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering whether there was a kind of clustering algorithm to be able to group files according to their naming conventions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any K-Mean philosophy applied to strings?&lt;/p&gt;&#xA;" OwnerUserId="44865" LastEditorUserId="44865" LastEditDate="2018-02-22T06:09:32.120" LastActivityDate="2018-02-24T21:00:11.280" Title="Clustering algorithm to sort filenames" Tags="|classification|clustering|k-means|" AnswerCount="5" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28025" PostTypeId="1" AcceptedAnswerId="28029" CreationDate="2018-02-20T11:43:18.410" Score="10" ViewCount="17443" Body="&lt;p&gt;I am trying to implement decision tree classifier to classify my data set. I am using Python. Now it is easy to implement in scikit learn, but how can I implement this in tensorflow.&lt;/p&gt;&#xA;" OwnerUserId="46433" LastEditorUserId="29575" LastEditDate="2018-02-20T13:38:22.940" LastActivityDate="2020-03-12T19:41:43.907" Title="Does Tensorflow support a Decision Tree Classifier?" Tags="|machine-learning|python|deep-learning|tensorflow|decision-trees|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28027" PostTypeId="1" CreationDate="2018-02-20T12:22:40.103" Score="1" ViewCount="822" Body="&lt;p&gt;i try haar_cascade for face detection and LBPH for face recognition , but the result wasn't good enough, please suggest good ways to detect and recognize faces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my aim is to create an app which take a photograph of students and by scanning this one photograph, it will predict which student is present and which is absent&lt;/p&gt;&#xA;" OwnerUserId="45336" LastActivityDate="2022-10-28T06:43:26.510" Title="which is the most effective(accurate) face detection method in python" Tags="|machine-learning|python|deep-learning|computer-vision|open-source|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28028" PostTypeId="2" ParentId="28027" CreationDate="2018-02-20T12:30:02.580" Score="0" Body="&lt;p&gt;Did you look into a (pre)trained convolutional net? I have not used any pre-trained facial model (trained other objects with tensorflow though) but I'm pretty sure a quick search will yield quite a few semi off the shelf models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That should be pretty much the state of the art, quite effective even with simpler models, but also widely available pretrained (specially for more widely used objects such as faces)&lt;/p&gt;&#xA;" OwnerUserId="27432" LastEditorUserId="27432" LastEditDate="2018-02-20T12:35:25.363" LastActivityDate="2018-02-20T12:35:25.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28029" PostTypeId="2" ParentId="28025" CreationDate="2018-02-20T12:38:32.183" Score="10" Body="&lt;p&gt;Basically I guess &lt;code&gt;TensorFlow&lt;/code&gt; does not support decision trees. I quote from &lt;a href=&quot;https://datascience.stackexchange.com/a/12921/28175&quot;&gt;here&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;This is a big oversimplification, but there are essentially two types of machine learning libraries available today, &lt;code&gt;Deep learning&lt;/code&gt;&#xA;  (CNN,RNN, fully connected nets, linear models) and Everything else&#xA;  (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc). The reason for&#xA;  this is that deep learning is much more computationally intensive than&#xA;  other more traditional training methods, and therefore requires&#xA;  intense specialization of the library (e.g., using a GPU and&#xA;  distributed capabilities). If you're using Python and are looking for&#xA;  a package with the greatest breadth of algorithms, try scikit-learn.&#xA;  In reality, if you want to use deep learning and more traditional&#xA;  methods you'll need to use more than one library. There is no&#xA;  &quot;complete&quot; package.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can see from &lt;a href=&quot;https://stackoverflow.com/a/43031536/5120235&quot;&gt;here&lt;/a&gt; that there are other learning algorithms implemented in &lt;code&gt;TensorFlow&lt;/code&gt; which are not deep models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take a look at &lt;a href=&quot;https://terrytangyuan.github.io/2016/08/06/tensorflow-not-just-deep-learning/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; for tracking algorithms implemented in &lt;code&gt;TensorFlow&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="69165" LastEditDate="2019-03-08T20:11:21.047" LastActivityDate="2019-03-08T20:11:21.047" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="28030" PostTypeId="1" CreationDate="2018-02-20T12:48:46.263" Score="3" ViewCount="1162" Body="&lt;p&gt;I work with python and images of watches (examples: &lt;a href=&quot;https://i.stack.imgur.com/BOTES.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;watch_1&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/qPNEb.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;watch_2&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/uISDl.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;watch_3&lt;/a&gt;). My aim is to take a photo of a random watch and then find the most similar watches to it in my database. Obviously, one main feature which distinguishes the watches are their shape (square, rectangular, round, oval) but there are also other ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I am just running a PCA and a KNN on rgb images of watches to find the most similar ones among them. My source code is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;import numpy as np&#xA;import os&#xA;from glob import glob&#xA;from sklearn.decomposition import PCA&#xA;from sklearn import neighbors&#xA;from sklearn import preprocessing&#xA;&#xA;&#xA;data = []&#xA;&#xA;# Read images from file&#xA;for filename in glob('Watches/*.jpg'):&#xA;&#xA;    img = cv2.imread(filename)&#xA;    height, width = img.shape[:2]&#xA;    img = np.array(img)&#xA;&#xA;    # Check that all my images are of the same resolution&#xA;    if height == 529 and width == 940:&#xA;&#xA;        # Reshape each image so that it is stored in one line&#xA;        img = np.concatenate(img, axis=0)&#xA;        img = np.concatenate(img, axis=0)&#xA;        data.append(img)&#xA;&#xA;# Normalise data&#xA;data = np.array(data)&#xA;Norm = preprocessing.Normalizer()&#xA;Norm.fit(data)&#xA;data = Norm.transform(data)&#xA;&#xA;# PCA model&#xA;pca = PCA(0.95)&#xA;pca.fit(data)&#xA;data = pca.transform(data)&#xA;&#xA;# K-Nearest neighbours&#xA;knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)&#xA;distances, indices = knn.kneighbors(data)&#xA;&#xA;print(indices)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However when I try to run this script for more than 1500 rgb images then I get a &lt;code&gt;MemoryError&lt;/code&gt; at the point where the data are processed by the PCA method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this normal for a pc with 24GB RAM and 3.6GHz Intel Core CPU without any discrete GPU?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I overcome this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shall I use another method like Incremental PCA (or a deep learning algorithm) or simply shall I buy a discrete GPU?&lt;/p&gt;&#xA;" OwnerUserId="45610" LastEditorUserId="45610" LastEditDate="2018-02-20T15:55:25.340" LastActivityDate="2018-02-20T21:39:53.357" Title="How to run PCA and KNN on big-data" Tags="|machine-learning|python|computer-vision|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28031" PostTypeId="2" ParentId="28030" CreationDate="2018-02-20T13:02:21.677" Score="3" Body="&lt;p&gt;KNN is instance based so it will store all training instances in memory. Since you are using images this will add up quickly. KNN on untransformed images might not perform that well anyway, you could look into filter banks to transform your images to a bag-of-word-representation (which is smaller and more invariant).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if it is accuracy you are aiming for I would recommend skipping all that (it is very 2012 anyway) in favor of using deep learning, fi: construct an auto-encoder and determine similarity on the encoded representation of an image (which could in turn be done using knn btw). &lt;/p&gt;&#xA;" OwnerUserId="27432" LastEditorUserId="27432" LastEditDate="2018-02-20T21:39:53.357" LastActivityDate="2018-02-20T21:39:53.357" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="28032" PostTypeId="2" ParentId="28027" CreationDate="2018-02-20T13:04:57.990" Score="1" Body="&lt;p&gt;You can use the Facial landmarks library, it's very easy to use &#xA;you will get 68 point and compare them from your dataSet very dirty and easy way of doing things note that&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The users must take the pictures at the same position to keep the same depth &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-02-20T13:04:57.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28033" PostTypeId="2" ParentId="19767" CreationDate="2018-02-20T13:31:39.527" Score="2" Body="&lt;p&gt;To cast that column of your data-frame as type float try: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k_train = train['cols_to_use'].astype(float)&#xA;target = train['final_status'].astype(float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More documentation can be found &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.astype.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; or you can cast them when loading the csv file &lt;code&gt;sep=','&lt;/code&gt; maybe usful assuming that your data is separated by a , in your CSV file&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train = pd.read_csv(&quot;datasets/train2.csv&quot;, &#xA;                    dtype={'cols_to_use': float , 'final_status': float})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that converting unix TimeStamp to readable date is very simple with the &lt;code&gt;datetime&lt;/code&gt; library example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import datetime &#xA;&#xA;datetime.datetime.fromtimestamp(1526972723).strftime('%Y-%m-%d %H:%M:%S')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I hope it's helpful.&lt;/p&gt;&#xA;" OwnerUserId="46527" LastEditorUserId="46527" LastEditDate="2018-05-22T08:07:46.637" LastActivityDate="2018-05-22T08:07:46.637" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28034" PostTypeId="2" ParentId="28006" CreationDate="2018-02-20T13:36:56.087" Score="7" Body="&lt;p&gt;It may vary from company to company, but &lt;em&gt;Data Scientist&lt;/em&gt; as a designation has &lt;strong&gt;been around for some time now&lt;/strong&gt; and is usually &lt;strong&gt;meant for extracting knowledge and insights from data&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen &lt;em&gt;data scientists&lt;/em&gt; doing&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Writing Image processing and image recognition algorithms,&lt;/li&gt;&#xA;&lt;li&gt;Design and implement decision trees for a business use case,&lt;/li&gt;&#xA;&lt;li&gt;Or simply design and implement some reports or write ETLs for data transformations.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Data_science&quot; rel=&quot;noreferrer&quot;&gt;Data science&lt;/a&gt;, &lt;strong&gt;however, is a super-domain of &lt;em&gt;machine learning&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;It employs techniques and theories drawn from many fields within the&#xA;  broad areas of mathematics, statistics, information science, and&#xA;  computer science, in particular from the subdomains of &lt;strong&gt;machine&#xA;  learning, classification, cluster analysis, uncertainty&#xA;  quantification, computational science, data mining, databases, and&#xA;  visualization&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Machine learning engineer&lt;/strong&gt; seems to be a designation where your employer has already narrowed down to the &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Approach,&lt;/li&gt;&#xA;&lt;li&gt;Tools, &lt;/li&gt;&#xA;&lt;li&gt;and a rough model (of what to deliver)  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;to extract knowledge or insights from data using machine learning and your &lt;strong&gt;work will be to design and implement machine learning algorithms to deliver the same&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="46293" LastActivityDate="2018-02-20T13:36:56.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28036" PostTypeId="1" CreationDate="2018-02-20T15:03:54.753" Score="1" ViewCount="264" Body="&lt;p&gt;If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 examples per minibatch)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Details:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 12600 time series examples, each with 24 time steps, and each time step has 972196 binary labels. This is a multilabel problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming float32 numbers:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;loading the entire dataset should take about 1095 GB (32*12600*24*972196) = 9.4x10^12 bits)&lt;/li&gt;&#xA;&lt;li&gt;loading a minibatch of size 32 should take about 2.7 GB (32*32*24*972196 bits)&lt;/li&gt;&#xA;&lt;li&gt;loading one training example should take about 89 MB (32*24*972196 bits)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm currently using an LSTM with online learning, and it's taking about 10 seconds per training example. I'm looking to speed this up.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other related answers:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One answer to &lt;a href=&quot;https://stats.stackexchange.com/questions/182376/online-learning-in-practice&quot;&gt;this question&lt;/a&gt; implies that minibatch learning will not be helpful when there are millions of features or millions of observations (I think they're just talking about the memory limitations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another question is &lt;a href=&quot;https://stats.stackexchange.com/questions/223829/online-learning-in-lstm&quot;&gt;related&lt;/a&gt; to what I'm asking, but on the implementation side of things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read &lt;a href=&quot;https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network&quot;&gt;this question&lt;/a&gt; and the answers, and it's very good but doesn't quite answer my question.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2024-01-06T01:02:55.373" Title="Online vs minibatch training for speed" Tags="|machine-learning|deep-learning|online-learning|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28037" PostTypeId="1" CreationDate="2018-02-20T15:08:30.137" Score="2" ViewCount="1725" Body="&lt;p&gt;I understand that gradient descent is local and it deals only with the inputs to the neuron, what it outputs and what it should output. In all I've seen, gradient descent needs the activation function to be differentiable, so a threshold function cannot be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet, the biological neurons either fire or they don't. The input to the neuron, in my understanding, is the equivalent of the membrane potential. Once it passes a certain threshold, the neuron fires (one or multiple times) and the input is reset. Considering this, the step function appears to be enough to reproduce the behavior of biological neurons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm thinking about the usual weights (integer or even float) paired with the step function, trained with the following form of backpropagation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;looking at one neuron and its connections and backpropagating, it has an output of 1 (it fired) when it should have been 0. The neuron has a threshold of 5 and its net input did sum up to be higher than 5. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;to train it, subtract from the weights so that the net input decreases by an arbitrary learning rate, so that the threshold is not exceeded or is closer to not being exceeded. For example, the net input was 5.5, so the &quot;error&quot; is 0.5. Multiply this &quot;error&quot; by the learning rate of 0.1 and the result is 0.05. This means that the weights need to be adjusted by 5% (multiplied times 0.95). The weights are changed only a little for each training example. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In my understanding, this is similar to backpropagation with gradient descent, but it's applied to step activation functions and it works even if they're not differentiable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, where are the implementations of this very simple training method? Why wouldn't it work? &lt;/p&gt;&#xA;" OwnerUserId="46538" LastEditorUserId="46538" LastEditDate="2018-02-20T15:24:15.507" LastActivityDate="2024-02-11T13:03:30.397" Title="Backpropagation with step or threshold activation function" Tags="|training|backpropagation|binary|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28038" PostTypeId="2" ParentId="26165" CreationDate="2018-02-20T15:11:17.050" Score="4" Body="&lt;p&gt;Reading the article, it seems like they define VGG54 as the loss calculated from the euclidean distance between the $\phi_{5,4}$ feature maps derived from both the high and low resolution images using the VGG19 network. Where $\phi_{i,j}$ is defined as &quot;&lt;em&gt;the feature map obtained by the j-th convolution (after activation) and before the i-th max-pooling layer within the VGG19 network&lt;/em&gt;&quot;.&lt;/p&gt;&#xA;" OwnerUserId="46542" LastEditorUserId="29575" LastEditDate="2018-02-20T15:33:15.863" LastActivityDate="2018-02-20T15:33:15.863" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28039" PostTypeId="2" ParentId="28006" CreationDate="2018-02-20T15:14:11.613" Score="13" Body="&lt;h1&gt;The terms are nebulous because they are new&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Being in the middle of a job search in the 'data science' field, I think that there are two things going on here. First, the jobs are new, and there is no set definitions of various terms, so no commonly agreed upon matching of terms with job descriptions. Compare this to 'web developer' or 'back-end developer.' These are two similar jobs that have reasonably well agreed upon and distinct descriptions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, a lot of people doing the job posting and initial interviews don't know that well what they are hiring for. This is particularly true in the case of small to medium sized-companies that hire recruiters to find applicants for them. It is these intermediaries that are posting the job descriptions on CareerBuilder or whatever forum. This isn't to say that many of them don't know their stuff, many of them are quite knowledgeable about the companies they represent and the requirements of the workplace. But, without well defined terms to describe different specific jobs, nebulous job titles are often the result.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;There are three general divisions of the field&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In my experience, there are three general divisions of the 'job space' of data science. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first is the development of the mathematical and computational techniques that make data science possible. This covers things like statistical research into new machine learning methods, the implementation of these methods, and the building of computational infrastructure to employ these methods in the real world. This is the division farthest separated from the customer, and the smallest division. Much of this work is done by either academics or researchers at the big companies (Google, Facebook, etc). This is for things like developing Google's TensorFlow, IBM's SPSS neural nets, or whatever the next big graph database is going to be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second division is using the underlying tools to create application specific packages to perform whatever data analysis needs to be done. People are hired to use Python or R or whatever to build analysis capability on some set of data. A lot of this work, in my experience, involves doing the 'data laundry,' turning raw data in whatever form into something usable. Another big chunk of this work is databasing; figuring out how to store the data in a way that it can be accessed in whatever timeline you need it in. This job isn't so much taking tools, but using existing database, statistics, and graphical analysis libraries to produce some results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The third division is producing analysis from the newly organized and accessible data. This is the most customer facing side, depending on your organization. You have to produce analysis that business leaders can use to make decisions. This would be the least technical of the three divisions; many jobs are hybrids between the second and third divisions at this point, since data science is in its infancy. But in the future, I strongly suspect that there will be a more clean division between these two jobs, with people win the second job needing a technical, computer science or statistics based education, and this third job needing only a general education. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, all three could describe themselves as 'data scientist', but only the first two could reasonably describe themselves as 'machine learning engineer.'&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Conclusion&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For the time being, you will have to find out yourself what each job entails. My current job hired me on as an 'analyst,' to do some machine learning stuff. But as we got to work, it became apparent that the company's databasing was inadequate, and now probably 90% of my time is spent working on the databases. My machine learning exposure is now just quickly running stuff through whatever scikit-learn package seems most appropriate, and shooting csv files to the third division analysts to make powerpoint presentations for the customer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The field is in flux. A lot of organizations are trying to add data science decision making to their processes, but without knowing clearly what that means. Its not their fault, its pretty hard to predict the future, and the ramifications of a new technology are never very clear. Until the field is more established, many jobs themselves will be as nebulous as the terms used to describe them. &lt;/p&gt;&#xA;" OwnerUserId="30472" LastEditorUserId="30472" LastEditDate="2018-02-20T15:19:52.797" LastActivityDate="2018-02-20T15:19:52.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28040" PostTypeId="1" AcceptedAnswerId="28055" CreationDate="2018-02-20T15:23:45.180" Score="4" ViewCount="12905" Body="&lt;p&gt;I noticed something strange while I was conducting a multiple label classification problem via keras neural network. My data set consist of imbalance data with 12 features and 25 possible labels. When I instantiate my model with no class weight I get a precision of 97%, recall of 13%, subset accuracy of 14%, f1-score of 23% using the micro average. &#xA;When I apply class weight these scores are significantly reduced to the below. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('Accuracy', 0.1757093081134893)&#xA;('Precision:', 0.19632925472747498)&#xA;('Recall', 0.1637291280148423)&#xA;F1 -score 0.178553363682&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also I calculate the weights with below code that I copied and modify from a previous post:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def class_out(s):&#xA;      y_classes = s#.idxmax(1, skipna=False)&#xA;&#xA;      # Instantiate the label encoder&#xA;      le = LabelEncoder()&#xA;&#xA;      # Fit the label encoder to our label series&#xA;      le.fit(list(y_classes))&#xA;&#xA;      # Create integer based labels Series&#xA;      y_integers = le.transform(list(y_classes))&#xA;&#xA;      #print y_integers&#xA;      # Create dict of labels : integer representation&#xA;      labels_and_integers = dict(zip(y_classes, y_integers))&#xA;&#xA;      print labels_and_integers&#xA;      class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)&#xA;      sample_weights = compute_sample_weight('balanced', y_integers)&#xA;&#xA;      class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))&#xA;      class_sweights_dict = dict(zip(le.transform(list(le.classes_)), sample_weights))&#xA;&#xA;&#xA;      print class_weights_dict&#xA;&#xA;      return class_weights_dict&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also see a sample of the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = 100&#xA;&#xA;weights = class_out(df_all['tag'])&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(10, activation=&quot;relu&quot;, input_shape=(12,)))&#xA;&#xA;#model.add(Dense(10, activation='relu'))&#xA;&#xA;#model.add(Dense(8, activation='relu'))&#xA;#model.add(Dropout(0.50))&#xA;model.add(Dense(25, activation=&quot;sigmoid&quot;))&#xA;model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy',precision,mcor,recall, f1])&#xA;&#xA;model.fit(X_train, Y_train, batch_size=batch_size, epochs=15,class_weight=weights,&#xA;      verbose=1,validation_data=(test, target_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a reason to believe that the model performance is best without class weights ?&lt;/p&gt;&#xA;" OwnerUserId="46055" LastEditorUserId="381" LastEditDate="2018-02-20T18:14:36.300" LastActivityDate="2018-03-29T07:05:26.003" Title="Class weight degrades Multi Label Classification Performance" Tags="|machine-learning|neural-network|keras|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28041" PostTypeId="2" ParentId="19747" CreationDate="2018-02-20T15:25:31.747" Score="1" Body="&lt;p&gt;Months later, after struggling with the same task, I've managed to load it!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TLDR version:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Replace all double-double-quotes (&quot;&quot;) within the values in the file by something harmless (i.e. &#xA;'). Orange then manages to load it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explanation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The comment by &quot;CodesInChaos&quot; did indeed point in the right direction:&#xA;This is a bug (in Orange or some library it uses) in parsing the quoted values. To be specific, it does not correctly handle correctly escaped double quotes inside the values of the columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RFC 4180 says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If double-quotes are used to enclose fields, then a double-quote&#xA;         appearing inside a field must be escaped by preceding it with&#xA;         another double quote.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, occurrence of this kind of escaping in the file causes the observed error in the Orange.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;NOTE&lt;/strong&gt;: Careful, simple search &amp;amp; replace may not cut it in general case - if the value itself starts with the double-quote, you would replace &quot;&quot;&quot; with '&quot;, which would produce invalid csv.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Also of NOTE&lt;/strong&gt;: &lt;strong&gt;Weka&lt;/strong&gt; seems to have this limitation too. Additionally, it does not tolerate line-breaks inside the quoted values - which again, according to the RFC are valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Moral of the story&lt;/strong&gt;: &lt;a href=&quot;https://news.ycombinator.com/item?id=7796268&quot; rel=&quot;nofollow noreferrer&quot;&gt;CSV parsing is not easy&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="35817" LastEditorUserId="35817" LastEditDate="2018-02-20T15:48:59.270" LastActivityDate="2018-02-20T15:48:59.270" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28042" PostTypeId="2" ParentId="28022" CreationDate="2018-02-20T15:30:58.853" Score="0" Body="&lt;h1&gt;Parse using regular expressions&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I work a project where we get thousands of data files per day from on the order of 10 different systems. The filenames are all a jumble, and have a tendency to change over time. This is a job for regular expressions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The primary thing I use to organize is functional groupings. I simply use a small set of indicators common to each file set. In my case, the server that aggregates and sends the data files to me appends its name to the filename (zs2101, or something like that). So then I search file names for this limited set of regular expressions (I am currently using 20). In your situation, it seems like the client name in the file headers is something you could search for an use for the base of your organization&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I divide files by date of generation of the data. Each file comes with a timestamp. When I read files into my archive, I find the date field in every file and convert it to a standard format, and then change the filename to reflect that standard format. Now I build a directory tree for each server name, organized by date (a folder for each year, and a sub-folder for each month, in my case). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation is that you use regular expressions to organize your data. These are deterministic in the sense that if you receive a strange file header or malformed filename, you know where it will end up (in my case, there is an 'Undetermined' folder that accepts everything that doesn't match a server or date regex). The problem with k-means is that if you get something you didn't plan for, it can be pretty hard to tell where the clustering algorithm will put it, leading to lost data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck, hope this helps. &lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-02-20T15:30:58.853" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28043" PostTypeId="1" AcceptedAnswerId="28050" CreationDate="2018-02-20T15:33:28.547" Score="2" ViewCount="1417" Body="&lt;p&gt;If I have a regression problem that can also be a classification problem by converting a continuous variable to a binary depending on a threshold, which model would be more accurate if I only care about being above or under the threshold?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: I have a column with house prices [20, 30, 40, 70] but I only care if the house is expensive (more than 40) or cheap (less than 40)&lt;/p&gt;&#xA;" OwnerUserId="46543" LastEditorUserId="46543" LastEditDate="2018-02-20T15:51:12.700" LastActivityDate="2018-02-20T16:38:59.787" Title="Classification or regression? Which model is more accurate if I only care about being above or under the threshold?" Tags="|machine-learning|classification|regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28044" PostTypeId="1" AcceptedAnswerId="28049" CreationDate="2018-02-20T15:43:12.233" Score="2" ViewCount="41" Body="&lt;p&gt;I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc.  The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others.  Each model has roughly 150 input variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model scores have been used to group the customers by decile and measure the churn rate of each group.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The present situation is that the scores appear to be too good which suggests we may have a data leakage problem.  For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business.  What I would like to know is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What are some basic analyses that I can perform to address the data&#xA;leakage issue. &lt;/li&gt;&#xA;&lt;li&gt;How can I leverage the model metadata to better&#xA;understand model performance? &lt;/li&gt;&#xA;&lt;li&gt;What other important questions should I&#xA;know to ask in order to fully address this task?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30613" LastActivityDate="2018-02-20T16:28:08.290" Title="How to approach model reporting task" Tags="|machine-learning|ensemble-modeling|scoring|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28045" PostTypeId="2" ParentId="25029" CreationDate="2018-02-20T15:51:40.433" Score="14" Body="&lt;p&gt;I think the best solution is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;add the weights to the second column of y_true &#xA;and then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def custom_loss(y_true, y_pred)&#xA;    weights = y_true[:,1]&#xA;    y_true = y_true [:,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That way it's sure to be assigned to the correct sample when they are shuffled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the metric functions will need to be customized as well by adding &lt;code&gt;y_true = y_true[:,0]&lt;/code&gt; at the top.&lt;/p&gt;&#xA;" OwnerUserId="32860" LastActivityDate="2018-02-20T15:51:40.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28046" PostTypeId="1" CreationDate="2018-02-20T16:05:11.397" Score="1" ViewCount="219" Body="&lt;p&gt;I am searching for a dataset that contains a machine's (any mechanical machine) operational variables (e.g. vibration, temprature, etc. ) versus it's operational status (e.g. functioning, malfunction, breakdown).&#xA;I have failed to find such dataset with googling.&#xA;Is there such a dataset?&lt;/p&gt;&#xA;" OwnerUserId="46545" LastActivityDate="2018-02-20T21:25:53.337" Title="Looking for a machine's breakdown dataset" Tags="|dataset|prediction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28047" PostTypeId="2" ParentId="28013" CreationDate="2018-02-20T16:06:05.650" Score="1" Body="&lt;p&gt;As you can imagine, the answer is &quot;it depends&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in your case you can choose between two: Euclidean or &lt;a href=&quot;https://en.wikipedia.org/wiki/Cosine_similarity&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cosine&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the dimensionality grows every point approach the border of the multi dimensional space where they lie, so the Euclidean distances between points tends asymptotically to be the same, which in similarity terms means that the points are all very similar to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in those cases, you go for the cosine similarity; let's say for dimensionality bigger than 100.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise you can go for the traditional, Euclidean similarity.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-02-20T16:06:05.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28049" PostTypeId="2" ParentId="28044" CreationDate="2018-02-20T16:28:08.290" Score="2" Body="&lt;h1&gt;Remove input data to test for leakage&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is very generalized question, so without knowing the types and provenance of the input data, this can be hard to answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, in general, to check for leakage, you can use the model on some subsets of the input variables while removing other input variables. If you get data from multiple sources, then try removing all input variables from a single source, then re-run your models. You may be able to identify the source of the data leakage. Alternately, if computational power allows, you can brute force it by running the model with each of the 150 input variables removed, or all sets of two variables, etc.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Use customer-centered time data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Regarding model meta-data, again I would investigate data provenance. Are you predicting churn using the complete patterns of customers who stopped using the service? What I mean to say is, instead of looking backwards from a fixed real-time period, like today, to all customers who did or did not stop using the service, try looking from a fixed customer-time. Use only data from the first year that each customer used the service, and attempt to predict whether each customer will remain with the service for another year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The warning signs of a customer dropping the service may be obvious in the months leading up to that customer dropping the service, but by then, the predictive power of your model may be too late to stop that customer from leaving. Instead, index the time component of each customer's history to zero when the first start using the service, and run your model on this data.&lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-02-20T16:28:08.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28050" PostTypeId="2" ParentId="28043" CreationDate="2018-02-20T16:38:59.787" Score="4" Body="&lt;p&gt;Classification is the more direct approach and it will likely give better results. This is because the model's goal is exactly the same as your goal - i.e. predicting whether the price is above or below the threshold - and it will maximize this accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A regression model that is trying to minimize MSE, for example, could give you a model that doesn't accurately put predictions on the correct side of the threshold - which it doesn't know or care about. The model does not have the same goal as you do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's construct a very simple example to see why this might be the case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Training     Test &#xA;x      y     x      y&#xA;1.0    20    1.8    ?&#xA;2.0    20    2.4    ?&#xA;3.0    200   2.8    ?&#xA;4.0    200   3.2    ?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I used the rule &lt;code&gt;if x&amp;lt;2.5: y=20, else y=200&lt;/code&gt; to generate the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A maximum-margin classifier would get all of the test examples right, but a linear regression model fit on the training data would get all of the test examples wrong.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-02-20T16:38:59.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28051" PostTypeId="2" ParentId="28001" CreationDate="2018-02-20T16:44:34.993" Score="1" Body="&lt;p&gt;First, &lt;code&gt;ss&lt;/code&gt; module is in &lt;code&gt;scipy.stats&lt;/code&gt; not &lt;code&gt;scipy.stats.stats&lt;/code&gt; as of &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.stats.ss.html#scipy.stats.ss&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy 0.17&lt;/a&gt;. Also, it is deprecated, but you seem to have figured that out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, in the current &lt;code&gt;linear_model.py&lt;/code&gt; on &lt;a href=&quot;https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/linear_model.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;github&lt;/a&gt;, the &lt;code&gt;ss&lt;/code&gt; package is no longer imported:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.linalg import toeplitz&#xA;from scipy import stats&#xA;from scipy import optimize&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What version is your statsmodels? Sounds like you may need to update it.&lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-02-20T16:44:34.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28052" PostTypeId="2" ParentId="27994" CreationDate="2018-02-20T16:54:39.403" Score="1" Body="&lt;p&gt;There could be significant problems from using one time chunk in a time series to predict the next chunk. For example, if you were using data from September and October to predict retail shopping expenditure in November and December, you would be very wrong. Instead, you need to note the annual trend of increased shopping between Thanksgiving and Christmas to make correct predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, while this sort of skirts the questions you asked, I think that if you are having difficulty using one month to predict the next month, you need to start looking at longer-term trends, especially annual ones. I would investigate this first before looking into population instability. &lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-02-20T16:54:39.403" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28053" PostTypeId="1" CreationDate="2018-02-20T16:57:10.453" Score="5" ViewCount="3372" Body="&lt;p&gt;This is quite a general question, perhaps somewhat opinion-based. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. Violinplots are made by performing a kernel density estimation on your distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there objective arguments to use one over the other? Or maybe there are specific situations in which one would prefer one over the other? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of both can be found &lt;a href=&quot;https://matplotlib.org/examples/statistics/boxplot_vs_violin_demo.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;: &lt;a href=&quot;https://i.stack.imgur.com/zcKyA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zcKyA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14372" LastActivityDate="2021-06-25T10:36:46.593" Title="Boxplots or violinplots?" Tags="|data-mining|visualization|distribution|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28054" PostTypeId="2" ParentId="28053" CreationDate="2018-02-20T17:16:54.557" Score="3" Body="&lt;p&gt;Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One main reason to go for a violin plot is to give more detail about the distribution, as box plots just give hard stops at the mean, stddev and 2 stddevs. Therefore if you think there is interesting information contained in the distribution between those points go violin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other main reason is they're more eye catching on Kaggle ;)&lt;/p&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-20T17:16:54.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28055" PostTypeId="2" ParentId="28040" CreationDate="2018-02-20T17:23:42.877" Score="5" Body="&lt;p&gt;Adding class weight but not changing the way you measure performance will usually degrade overall performance as it is designed to allow increased loss on lower-weighted classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend also weighting your accuracy measures. This is a bit tricky with accuracy/precision etc. so maybe calculated the weighted logloss and compare it to the unweighted logloss of the unweighted model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically it comes down to the question: are you happy with a model that performs worse overall but better on your heavily weighted classes? &lt;/p&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-20T17:23:42.877" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28056" PostTypeId="1" AcceptedAnswerId="28067" CreationDate="2018-02-20T17:30:55.590" Score="3" ViewCount="2036" Body="&lt;p&gt;I'm running a regression model on a pretty large data set and getting a fairly woeful $R^2$ score of ~0.2 (see plot below), despite the plot looking like the model is &lt;em&gt;generally&lt;/em&gt; pointing in the right direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, when you have over a million data points, how high can you realistically expect the $R^2$ to go in real world data with a decent amount of noise?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What prompts by scepticism of such traditional measures are articles such as &lt;a href=&quot;https://www.datasciencecentral.com/profiles/blogs/the-death-of-the-statistical-test-of-hypothesis&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; that discuss how quantity of data can degrade statistical tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know what you think and any regression examples using $R^2$ score as a quality metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MD09K.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MD09K.jpg&quot; alt=&quot;Predicted values vs. actual&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38271" LastEditorUserId="85045" LastEditDate="2021-03-03T16:35:20.597" LastActivityDate="2021-03-03T16:35:20.597" Title="Is R2 score a reasonable regression measure on huge datasets?" Tags="|regression|metric|performance|r-squared|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28057" PostTypeId="1" AcceptedAnswerId="28058" CreationDate="2018-02-20T17:52:13.627" Score="10" ViewCount="3835" Body="&lt;pre&gt;&lt;code&gt;Every 1.0s: nvidia-smi                                                                                                                                                                Tue Feb 20 12:49:34 2018&#xA;&#xA;Tue Feb 20 12:49:34 2018&#xA;+-----------------------------------------------------------------------------+&#xA;| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |&#xA;|-------------------------------+----------------------+----------------------+&#xA;| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |&#xA;| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |&#xA;|===============================+======================+======================|&#xA;|   0  Quadro M1000M       Off  | 00000000:01:00.0 Off |                  N/A |&#xA;| N/A   59C    P0    N/A /  N/A |   1895MiB /  2002MiB |     64%      Default |&#xA;+-------------------------------+----------------------+----------------------+&#xA;&#xA;+-----------------------------------------------------------------------------+&#xA;| Processes:                                                       GPU Memory |&#xA;|  GPU       PID   Type   Process name                             Usage      |&#xA;|=============================================================================|&#xA;|    0      1166      G   /usr/lib/xorg/Xorg                           239MiB |&#xA;|    0      1864      G   compiz                                        80MiB |&#xA;|    0      6755      C   python                                      1408MiB |&#xA;|    0     25674      G   ...-token=5769296849603E2A1B668201DBB31D6A   149MiB |&#xA;+-----------------------------------------------------------------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm a new practitioner of keras+gpu based deep learning.&lt;br&gt;&#xA;What is this &lt;code&gt;watch -n 1 nvidia-smi&lt;/code&gt;telling me?&lt;br&gt;&#xA;Is there any value proposition in it, can it be used for improving performance of the gpu?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the backgrpound a small Keras model is training.   &lt;/p&gt;&#xA;" OwnerUserId="40051" LastEditorUserId="40051" LastEditDate="2018-02-20T17:58:29.187" LastActivityDate="2019-06-14T23:42:32.043" Title="interpret results of nvidia-smi" Tags="|deep-learning|nvidia|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28058" PostTypeId="2" ParentId="28057" CreationDate="2018-02-20T18:18:59.550" Score="3" Body="&lt;p&gt;&lt;code&gt;watch&lt;/code&gt; is linux command and not related to &lt;code&gt;nvidia-smi&lt;/code&gt; command,which using that the &lt;code&gt;nvidia-smi&lt;/code&gt; command will be updated every 1 second and it give you a view to see how gpu parameters changing like: RAM usage, temperature, P0(means performance,which p0 is best performance and performance decrease when you get p1,p2,...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;about &quot;&lt;em&gt;can it be used for improving performance of the gpu?&lt;/em&gt;&quot; the performance of gpu is showing by p0 so your gpu currently is in max performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;note that minimum interval for &lt;code&gt;watch&lt;/code&gt; command is 0.1 second so if you use &lt;code&gt;watch -n 0.1 nvidia-smi&lt;/code&gt; then every 0.1 second the output will be update.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for further information type &lt;code&gt;man watch&lt;/code&gt; or &lt;code&gt;man nvidia-smi&lt;/code&gt; in terminal.&lt;/p&gt;&#xA;" OwnerUserId="45494" LastEditorUserId="45494" LastEditDate="2018-02-20T18:30:48.600" LastActivityDate="2018-02-20T18:30:48.600" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28059" PostTypeId="2" ParentId="28056" CreationDate="2018-02-20T19:14:31.963" Score="1" Body="&lt;p&gt;There is no general answer of what to expect for an $R^2$ score.  And there is no general answer for whether a model with this $R^2$ score is a &quot;good&quot; model.  There are many cases where (1) this kind of $R^2$ score is not unreasonable and (2) the model is still useful.  Looking at this data, $R^2=0.243$ feels about right, with most of the data concentrated in an almost disk-like blob near the origin.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The article you link to deals with statistical confidence, which is a very different issue than $R^2$.  I would guess that given the amount of data, despite a $R^2$ score you consider is low, the p-value will be very high.  The article touches upon p-hacking, which is an issue if you consider many possible inputs with competing models.  With just one $x$, as is here, if this is the only model you have built, this is reasonable.&lt;/p&gt;&#xA;" OwnerUserId="8005" LastActivityDate="2018-02-20T19:14:31.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28060" PostTypeId="2" ParentId="28006" CreationDate="2018-02-20T20:47:09.947" Score="1" Body="&lt;p&gt;I don't disagree with any of the answers given. However, I do think that there is a role of Data Scientist that is being glossed over in virtually all of the answers here. Most of these answers say something to the effect of, &quot;Well, an engineer just writes and deploys the model . . . &quot;. Hold on a sec - there's &lt;strong&gt;&lt;em&gt;A LOT&lt;/em&gt;&lt;/strong&gt; of work in those two steps!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My core &lt;strong&gt;definition of a Data Scientist is someone that applies the scientific method to working with data.&lt;/strong&gt; So I'm constantly thinking of hypostheses, designing tests, collecting my data and executing those tests, checking my cross validation results, trying new approaches, transforming my data, etc, etc. That's essentially what goes into &quot;just writes and deploys the model&quot; in a professional setting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for your answer, I think &quot;the devil is in the details&quot; because you can't just gloss over some of these steps/terms. Also, if you are job hunting, you should be careful because &quot;data engineer&quot; and &quot;data scientist&quot; can have woefully different pay scales - you do not want to be a data scientist on a data engineer salary! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I always put myself out there as a data scientist, I tell companies that I work on predictive models (not just analytical) and that I'm not an Excel jockey - I write in programming languages (R, Python, etc). If you can find a position that let's you do both of those, then you're on your way to being a data scientist. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-20T20:47:09.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28061" PostTypeId="1" CreationDate="2018-02-20T20:48:33.710" Score="1" ViewCount="2032" Body="&lt;p&gt;What is the difference between blocking and clustering? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I know clustering originates from the Machine Learning branch and refers to methods of grouping data points together in a &quot;We and them&quot; way: Points in the same cluster are related, points of different clusters are not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Blocking, however, seems to come from the statistics and picked up later in data analysis. I encountered Blocking/Indexing etc. primarily in Record Linkage, where it is used to partition data into blocks before they are analyzed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One could even have both blocking and clustering methods used in Record Linkage. However, I find, I don't understand their difference too well yet and thus would like to have a different perspective on that matter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My interpretation thus far: The goal of blocking and clustering is different:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Blocking is used to throw potential unnecessary data out - distinguishing between important and non-important data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering however does make that distinction, all data points are important. is used for classification purposed - identifying all data. &lt;/p&gt;&#xA;" OwnerUserId="41344" LastActivityDate="2018-02-20T22:55:04.443" Title="Difference between Blocking and Clustering?" Tags="|machine-learning|clustering|bigdata|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28062" PostTypeId="1" AcceptedAnswerId="28124" CreationDate="2018-02-20T20:52:12.383" Score="2" ViewCount="943" Body="&lt;p&gt;I am working on an image classification problem and using data augmentation in Keras.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.preprocessing.image import ImageDataGenerator&#xA;&#xA;datagen = ImageDataGenerator(&#xA;    rotation_range=2,&#xA;       horizontal_flip=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to know if the &lt;code&gt;ImageDataGenerator&lt;/code&gt;applies the transformation randomly to image patches. That is for example, rotation may be applied on one image, whilst flip may be applied to another image. I want to know if the decision to apply rotation or a flip is randomly determined.&lt;/p&gt;&#xA;" OwnerUserId="45898" LastActivityDate="2018-03-05T10:23:42.470" Title="Are the image data augmentation generators in Keras randomly applied" Tags="|machine-learning|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28063" PostTypeId="2" ParentId="28046" CreationDate="2018-02-20T21:25:53.337" Score="1" Body="&lt;p&gt;I would recommend the NSSDC NASA data of 550 space spacecraft  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pds.jpl.nasa.gov/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pds.jpl.nasa.gov/&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-02-20T21:25:53.337" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28064" PostTypeId="2" ParentId="28056" CreationDate="2018-02-20T21:27:01.263" Score="1" Body="&lt;p&gt;I think Paul's answer is a really good one. The one additional point I'd make about the $R^2$ score is that you should only compare different $R^2$ scores between models estimated on the same set of data. Conceptually, it does not make any sense to compare $R^2$ scores between models derived from different data because $R^2$ is itself just a measure of variance in the outcome explained by the model. Different sets of data will have different amounts of variance explainable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a chief reason why defining a &quot;good&quot; $R^2$ score is hard. The $R^2$ score is of course dependent on your model's fit, but also on the data itself (and the set of data is dependent on its collection, as well as the domain from which it is drawn). &lt;/p&gt;&#xA;" OwnerUserId="46546" LastActivityDate="2018-02-20T21:27:01.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28065" PostTypeId="2" ParentId="23067" CreationDate="2018-02-20T21:35:18.240" Score="1" Body="&lt;p&gt;I have encountered case in which potent tree learners acted like Nearest Neighbor variants: They would learn to divide the search space so that only examples remained that where close on some meaningful axis (in my case lat and long :)). This could still generalize to examples in the test set that shared those features but not to genuinely unseen examples. Or to put it in another way: The method is potent enough to find rules that work well, but do not generalize in the way you want it to (aka over fitting, but the maybe in a specific way). What helped me were two things: First of all I tested this characteristic of the domain by just using KNN,only feeding it subsets of the feature space (and sure enough KNN over lat, lng worked like a charm). This helped me understand that examples that were in the same space time coords shared certain history, so the second action was to eliminate those examples in the train space and evaluate on an a set of examples totally outside of the coordinates (space and time) of the train set. It is hard to say where this behavior could lie in your case without having a peek at the code, but perhaps my anecdote helps.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-20T21:35:18.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28066" PostTypeId="1" CreationDate="2018-02-20T21:50:46.160" Score="3" ViewCount="112" Body="&lt;p&gt;I work in a large company on several data science projects. For each of the projects me and my colleagues construct features that have some predictive value for the specific target in that project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some project are similar in that they a predict something for the same kind of entity, for example customers or goods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would make sense to me to share features between projects that are about the same entity. Or at the least, make it easy to reuse features from another project. For example, in some project someone could construct the feature &quot;customer since&quot; which would indicate the number of years someone is a customer. In some other project some constructed a feature &quot;estimated age&quot; that is the outcome of some machine learning pipeline. In a third project I might want to use both features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are best practices in sharing these features? Should I share code or materialized outcomes? Are there packages that aid this process? How does your company solve this? &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2018-02-20T21:50:46.160" Title="What are best practices for collaborative feature engineering?" Tags="|machine-learning|feature-extraction|feature-engineering|feature-scaling|feature-construction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28067" PostTypeId="2" ParentId="28056" CreationDate="2018-02-20T22:11:29.460" Score="4" Body="&lt;p&gt;The coefficient of determination $r^2$ is defined in terms of variance: it is the proportion of variance in the dependent variable that is explained by the independent variable. Variance is a property of normal distributed data. Hence, the coefficient of determination can only be used when you assume that  both the dependent and independent variables are normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just like other properties of normal distributed data, the estimation of $r^2$ improves when the amount of data increases. With very little data a coincidental colinearity might be the case, but this is not possible with large amounts of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Back to your example. Your data is cleary not normal distributed, it has right-skew and has large outliers. For this reason it is not advisable to use   $r^2$. Imagine for example that in the lower left corner (where the majority of the data is) you would observe a negative trend, but in general there is a positive trend. The regression line would be the same and the $r^2$ would be in the same range. This is known as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Simpson%27s_paradox?wprov=sfla1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Simpsons paradox&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, if your data is normal distributed you can use $r^2$ for any size of dataset. If it is not normal distributed you cannot use $r^2$. &lt;/p&gt;&#xA;" OwnerUserId="20372" LastActivityDate="2018-02-20T22:11:29.460" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28068" PostTypeId="2" ParentId="28061" CreationDate="2018-02-20T22:55:04.443" Score="2" Body="&lt;p&gt;As you mentioned their goals are different. In clustering, we try to group data such that they have the same variability. For example, clustering customers of a company into different clusters, somehow, members of each cluster have the same behavior in their buying. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other side, in blocking we try to reduce the variability, to record linkage, as an instance. &lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-02-20T22:55:04.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28069" PostTypeId="1" AcceptedAnswerId="28092" CreationDate="2018-02-20T22:56:39.053" Score="1" ViewCount="27" Body="&lt;p&gt;I have a Pandas dataframe that has some fields that contain very verbose text. I want to be able to iterate through the DF but only display a limited set of words. I have code similar to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for index, row in df.iterrows() :&#xA;    print(row['A'], row['B'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I make sure that I only print the first 300 characters from 'A'?&lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-21T09:39:15.887" Title="Get a portion of a long field in Pandas?" Tags="|python|pandas|" AnswerCount="1" CommentCount="2" ClosedDate="2018-02-22T22:42:49.317" ContentLicense="CC BY-SA 3.0" />
  <row Id="28070" PostTypeId="1" AcceptedAnswerId="28100" CreationDate="2018-02-20T23:25:17.277" Score="1" ViewCount="1140" Body="&lt;p&gt;I'm trying to implement a model of a recurrent neural network to solve a temporal version of the XOR problem, but I am not still able to do that. Any hints?&lt;/p&gt;&#xA;" OwnerUserId="46567" LastEditorUserId="46567" LastEditDate="2023-05-24T07:45:36.540" LastActivityDate="2023-05-24T07:45:36.540" Title="How to obtain with a RRN a version of a temporal XOR function using keras/tensorflow?" Tags="|machine-learning|neural-network|keras|rnn|mlp|" AnswerCount="2" CommentCount="3" ClosedDate="2018-02-25T23:51:01.607" ContentLicense="CC BY-SA 4.0" />
  <row Id="28071" PostTypeId="1" AcceptedAnswerId="28090" CreationDate="2018-02-21T00:12:58.247" Score="1" ViewCount="37" Body="&lt;p&gt;Consider a dataset from 1990 to 2017 that contains the crime rate per 100,000 people in some cities from Latin America.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to measure which city is more complex according to this data and other indicators. I'm using The analytic hierarchy process proposed by Saaty.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;city_code  r1990  r1991 r1992 r1993 r1994 r1994&#xA;  12345     120     80    91   110   105   99&#xA;  23456      10     15    16    12     7   11&#xA;  34567      90     91    85    75    77   65&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to my question, I want to &lt;code&gt;sum&lt;/code&gt; each rate in each year, and then get for each year the &lt;code&gt;mean&lt;/code&gt; of each rate based on the total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the example above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;city_code  total&#xA;  12345     605&#xA;  23456      71&#xA;  34567     483&#xA;&#xA;city_code  mean1990  mean1991 mean1992 mean1993 mean1994 mean1994&#xA;  12345      0.20      0.13     0.15     0.18     0.17     0.16&#xA;  23456      0.14      0.21     0.23     0.17     0.10     0.15&#xA;  34567      0.19      0.19     0.18     0.16     0.16     0.13&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, is this a good practice? I couldn't find an example following this method? Any orientation about this subject will be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="17636" LastEditorUserId="17636" LastEditDate="2018-02-21T15:31:49.307" LastActivityDate="2018-02-21T15:31:49.307" Title="Is a good practice to sum each rate (i.e. crime rate per 100.000 people)?" Tags="|dataset|statistics|data-cleaning|methodology|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28072" PostTypeId="2" ParentId="28006" CreationDate="2018-02-21T01:29:09.980" Score="3" Body="&lt;h3&gt;TL;DR: It depends on who is asking.&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The answer to this question depends largely on the expectations, knowledge, and experience of whomever is asking. An analogous question with just as fuzzy of an answer is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the difference between a software developer, a software&#xA;  engineer, and a computer scientist?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To some people, particularly people who study or teach computer science and software engineering, there is a large and defined difference between these fields. But to the average HR worker, technical recruiter, or manager, these are all just &quot;Computer People&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I love &lt;a href=&quot;http://www.datasciencecentral.com/profiles/blogs/difference-between-machine-learning-data-science-ai-deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;this quote by Vincent Granville&lt;/a&gt;, emphasis mine:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Earlier in my career (circa 1990) I worked on image remote sensing&#xA;  technology, among other things to identify patterns (or shapes or&#xA;  features, for instance lakes) in satellite images and to perform image&#xA;  segmentation: &lt;strong&gt;at that time my research was labeled as computational&#xA;  statistics, but the people doing the exact same thing in the computer&#xA;  science department next door in my home university, called their&#xA;  research artificial intelligence. Today, it would be called data&#xA;  science&lt;/strong&gt; or artificial intelligence, the sub-domains being signal&#xA;  processing, computer vision or IoT.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46573" LastEditorUserId="46573" LastEditDate="2018-08-17T18:42:20.337" LastActivityDate="2018-08-17T18:42:20.337" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28073" PostTypeId="1" AcceptedAnswerId="28388" CreationDate="2018-02-21T01:40:12.573" Score="1" ViewCount="2478" Body="&lt;p&gt;I am using Weka Time Series Forecasting to forecast the trend of the topic &lt;code&gt;NLP&lt;/code&gt; in &lt;code&gt;2018&lt;/code&gt;. For that I used &lt;code&gt;evaluate on training&lt;/code&gt; option in &lt;code&gt;Weka&lt;/code&gt; to obtain Root Mean Squared Error (RMSE). The RMSE value I obtain is &lt;code&gt;0.785&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What does this imply?&#xA;Is there a threshold value (e.g., RMSE &amp;lt; &lt;code&gt;0.2&lt;/code&gt;) that we can consider to &#xA;&lt;code&gt;accept/reject&lt;/code&gt; this prediction?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45262" LastEditorUserId="35644" LastEditDate="2018-02-28T11:03:52.740" LastActivityDate="2018-02-28T11:03:52.740" Title="RMSE in Weka Time Series Forecasting" Tags="|data-mining|time-series|text-mining|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28074" PostTypeId="1" AcceptedAnswerId="28076" CreationDate="2018-02-21T02:40:58.780" Score="1" ViewCount="169" Body="&lt;p&gt;K-means may give different results, because the initial choice of centroids is random. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if I were to choose k=1, will the algorithm always provide the same answer equal to the &quot;barycentre&quot; of my data?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="13736" LastEditDate="2018-02-21T02:51:38.030" LastActivityDate="2018-02-21T03:19:17.237" Title="k-means with one cluster" Tags="|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28076" PostTypeId="2" ParentId="28074" CreationDate="2018-02-21T03:19:17.237" Score="5" Body="&lt;p&gt;Yes. The centroid will converge to the center of all your data and this will occur in a single iteration. This is due to all the data points belonging to a single centroid, thus it will be centered according to all these instances immediately. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.cluster import KMeans&#xA;&#xA;# Generate data&#xA;class1 = np.random.randn(1000, 2)&#xA;&#xA;class2 = np.random.randn(1000, 2)&#xA;class2[:,0] = 4+class2[:,0]&#xA;class2[:,1] = 4+class2[:,1]&#xA;&#xA;class3 = np.random.randn(1000, 2)&#xA;class3[:,0] = -4+class3[:,0]&#xA;class3[:,1] = 4+class3[:,1]&#xA;&#xA;data = np.append( class1, class2, axis= 0)&#xA;data = np.append( data, class3, axis= 0)&#xA;print(data.shape)&#xA;&#xA;# Plot the data&#xA;plt.scatter(data[:,0], data[:,1])&#xA;plt.show()&#xA;&#xA;# Cluster&#xA;kmeans = KMeans(n_clusters=1, random_state=0, verbose = 1).fit(data)&#xA;&#xA;# Plot clustered results&#xA;plt.scatter(data[:,0], data[:,1], c=kmeans.labels_)&#xA;plt.scatter(kmeans.cluster_centers_[:,0], kmeans.cluster_centers_[:,1], c = 'r')&#xA;plt.show()&#xA;&#xA;# Show the cluster centers&#xA;print(kmeans.cluster_centers_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5jQ2N.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5jQ2N.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Initialization complete Iteration  0, inertia 81470.055 Iteration  1,&#xA;  inertia 48841.695 Converged at iteration 1: center shift 0.000000e+00&#xA;  within tolerance 8.140283e-04&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-21T03:19:17.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28077" PostTypeId="2" ParentId="28006" CreationDate="2018-02-21T03:52:25.743" Score="8" Body="&lt;p&gt;Machine Learning Engineers and engineering focused Data Scientist are the same, but not all Data Scientist are engineering focused. About 5 years ago almost all Data Scientist were engineering focused, e.g, they had to write production code. Now, however, there are many Data Scientist roles that are for most part: playing in Jupyter notebook, understanding data, making pretty graphs, explaining to clients, managers, analysts... They don't do any engineering. And I believe that term Machine Learning Engineers came up to underline that this an engineering position. &lt;/p&gt;&#xA;" OwnerUserId="3070" LastActivityDate="2018-02-21T03:52:25.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28078" PostTypeId="2" ParentId="5175" CreationDate="2018-02-21T04:18:20.433" Score="1" Body="&lt;p&gt;Another way of editing alias for a member having multiple aliases:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Right click the 'Measure Names' in Dimension panel.&lt;/li&gt;&#xA;&lt;li&gt;Select 'Aliases...' &lt;/li&gt;&#xA;&lt;li&gt;A small edit box appears with all the members having aliases.&lt;/li&gt;&#xA;&lt;li&gt;Edit/update alias for each member, as preferred.&lt;/li&gt;&#xA;&lt;li&gt;Press OK to save any changes made.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46579" LastEditorUserId="29575" LastEditDate="2018-02-21T04:24:18.393" LastActivityDate="2018-02-21T04:24:18.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28079" PostTypeId="2" ParentId="28053" CreationDate="2018-02-21T07:09:26.177" Score="1" Body="&lt;p&gt;Well, that is not true that Boxplot only gives hard stops! Violin plots are rather contemporary version of Boxplots I would say, and easy for the eyes to see the distribution of data. Boxplots also can reveal how data is distributed. For example, here you see a Boxplot of a normally distributed data that is symmetrical with the mean and median in the center (top), as well as a non-normal data (bottom). The source is &lt;a href=&quot;http://www.statisticshowto.com/assumption-of-normality-test/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0kZvk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0kZvk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details of how to interpret a Boxplot in terms of a normal distribution, look &lt;a href=&quot;https://ms.mcmaster.ca/peter/s2ma3/s2ma3_9798/boxplots.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7lBB3.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7lBB3.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sidenote: Boxplots usually are used for easy and quick outliers detection as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Said that I totally agree with jshep that it is the matter of taste of both user and audience, and usually academics lean towards less fancier presentations like Boxplots. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-02-21T09:28:00.773" LastActivityDate="2018-02-21T09:28:00.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28080" PostTypeId="2" ParentId="9073" CreationDate="2018-02-21T07:37:26.943" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;No, images are present both in the leaves and in the innernodes.&#xA;A more general category like 'dog.n.01' would be a hypernym to a&#xA;more specific category like 'harrier.n.02' (which would be one of&#xA;the hyponyms of dog) but both the synsets would have representative&#xA;images.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;According to &lt;a href=&quot;https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Mechanical Turk brought its own slew of hurdles, with much of the work fielded by two of Li’s Ph.D students, Jia Deng and Olga&#xA;  Russakovsky . For example, how many Turkers needed to look at each&#xA;  image? Maybe two people could determine that a cat was a cat, but an&#xA;  image of a miniature husky might require 10 rounds of validation.&#xA;  What if some Turkers tried to game or cheat the system? Li’s team&#xA;  ended up creating a batch of statistical models for Turker’s&#xA;  behaviors to help ensure the dataset only included correct images.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;According to &lt;a href=&quot;http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Karpathy&lt;/a&gt;, Turkers query on search engines for&#xA;images corresponding to a label like 'harrier' and then give a&#xA;binary 'YES/NO' answer for inclusion of the image in the set. This&#xA;is a noisy process and sacrifices some correctness to scale at that&#xA;level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If multiple objects are present, it could be classified with the&#xA;label for a less dominating object in the image since Turkers are&#xA;unaware of other classes in the dataset and hence wouldn't realize&#xA;it if a more accurate label exists in wordnet for the given image.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Synsets can be inner nodes or leaves. For example, as of today, the&#xA;'dog' synset contains 1603 images whereas its child synset 'harrier'&#xA;contains 217 images.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;All images are not in the leaves. They can be under a leaf or an&#xA;inner node. The specific number of leaves can be estimated using&#xA;their &lt;a href=&quot;http://image-net.org/download-API&quot; rel=&quot;nofollow noreferrer&quot;&gt;api&lt;/a&gt; for full hyponym (synset of a whole subtree) on all&#xA;the high level categories&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46587" LastActivityDate="2018-02-21T07:37:26.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28083" PostTypeId="2" ParentId="28013" CreationDate="2018-02-21T07:53:50.380" Score="2" Body="&lt;p&gt;Cosine is for &lt;em&gt;continuous&lt;/em&gt; values. It's not the most appropriate thing here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For binary values, look at&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Simple matching distance&lt;/li&gt;&#xA;&lt;li&gt;Hamming distance&lt;/li&gt;&#xA;&lt;li&gt;Jaccard distance&lt;/li&gt;&#xA;&lt;li&gt;Many many many more.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Don't expect clustering to just work. If you have binary data, you only have &lt;em&gt;very&lt;/em&gt; few bits of information. The resulting distances will show discrete steps, and clustering will likely still fail to provide good results. Roughly, you have &quot;identical&quot;, &quot;one difference&quot;, &quot;two differences&quot;, and then probably everything already is connected.&#xA;It's because of the data, and a not very well defined task (what would be a &lt;em&gt;good&lt;/em&gt; cluster?).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-21T07:53:50.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28085" PostTypeId="1" AcceptedAnswerId="28099" CreationDate="2018-02-21T08:07:26.737" Score="2" ViewCount="254" Body="&lt;p&gt;I have a data.table base that has many variables to use them to forecasting sales for &lt;strong&gt;the next 6 weeks of daily sales&lt;/strong&gt;. In fact, all the database is arranged by date as you can see here.Note that here I just show you some of variables.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Data_train[order(Date)]&#xA;         Store DayOfWeek       Date Sales Customers Open Promo StateHoliday SchoolHoliday&#xA;      1:     1         2 2013-01-01     0         0    0     0            a             1&#xA;      2:     2         2 2013-01-01     0         0    0     0            a             1&#xA;      3:     3         2 2013-01-01     0         0    0     0            a             1&#xA;      4:     4         2 2013-01-01     0         0    0     0            a             1&#xA;      5:     5         2 2013-01-01     0         0    0     0            a             1&#xA;     ---                                                                                 &#xA;1017205:  1111         5 2015-07-31  5723       422    1     1            0             1&#xA;1017206:  1112         5 2015-07-31  9626       767    1     1            0             1&#xA;1017207:  1113         5 2015-07-31  7289       720    1     1            0             1&#xA;1017208:  1114         5 2015-07-31 27508      3745    1     1            0             1&#xA;1017209:  1115         5 2015-07-31  8680       538    1     1            0             1 .&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is about the &lt;strong&gt;arrangement&lt;/strong&gt; of the data according to the target goal. My problem is really about the &lt;strong&gt;Date&lt;/strong&gt; variable. &#xA;In fact, I suggest this path:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I sum all sales by each date (because I have many types of stores).&lt;/li&gt;&#xA;&lt;li&gt;I order my database according to the Date 's ascending order.&lt;/li&gt;&#xA;&lt;li&gt;I didn't need duplicated rows in the date variables so I delete them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Just to show you the new base for considered variables. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; Data_train[,SumSaleseachDay:=sum(Sales),by=c('Date')][order(Date)][!duplicated(Date)][,-c('Sales','Customers'),with=FALSE]&#xA;     Store DayOfWeek       Date Open Promo StateHoliday SchoolHoliday SumSaleseachDay&#xA;  1:     1         2 2013-01-01    0     0            a             1           97235&#xA;  2:     1         3 2013-01-02    1     0            0             1         6949829&#xA;  3:     1         4 2013-01-03    1     0            0             1         6347820&#xA;  4:     1         5 2013-01-04    1     0            0             1         6638954&#xA;  5:     1         6 2013-01-05    1     0            0             1         5951593&#xA; ---                                                                                 &#xA;938:     1         1 2015-07-27    1     1            0             1        10707292&#xA;939:     1         2 2015-07-28    1     1            0             1         9115073&#xA;940:     1         3 2015-07-29    1     1            0             1         8499962&#xA;941:     1         4 2015-07-30    1     1            0             1         8798854&#xA;942:     1         5 2015-07-31    1     1            0             1        10109742&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ADDED INFORMATION:&lt;/strong&gt; &#xA;I have a database of &lt;code&gt;1017209&lt;/code&gt; rows. And for each Store, I have its historic of  Sales between &lt;code&gt;2013-01-01&lt;/code&gt; and &lt;code&gt;2015-07-31&lt;/code&gt;. And I have also &lt;strong&gt;17 variables&lt;/strong&gt; included to build the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The steps above just lead to forecast by day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I want to forecast for &lt;strong&gt;each Store&lt;/strong&gt; and &lt;strong&gt;by day&lt;/strong&gt;, what should I do?&lt;br&gt;&#xA;thank you in advance!  &lt;/p&gt;&#xA;" OwnerUserId="46590" LastEditorUserId="85045" LastEditDate="2021-03-03T21:06:37.510" LastActivityDate="2021-03-03T21:06:37.510" Title="How do I arrange my data to predict 6 weeks of daily sales" Tags="|machine-learning|time-series|forecasting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28086" PostTypeId="1" CreationDate="2018-02-21T08:27:11.190" Score="4" ViewCount="2547" Body="&lt;p&gt;I am trying to learn a very simple sequence using an RNN (implemented in Keras)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input sequence is randomly generated integers between 0 to 100:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;x=np.random.randint(0,100, size=2000)&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;while the expected output value for time t is the (t-2)th input term i.e:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y&lt;sub&gt;t&lt;/sub&gt;=x&lt;sub&gt;t-2&lt;/sub&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;such that an example dataset looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;+------+------+&#xA;|  X   |  Y   |&#xA;+------+------+&#xA;|    0 |   NA |&#xA;|   24 |   NA |&#xA;|   33 |    0 |&#xA;|    6 |   24 |&#xA;|   78 |   33 |&#xA;|   11 |    6 |&#xA;|    . |    . |&#xA;|    . |    . |&#xA;+------+------+&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: I drop the NA rows before training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to train a simple RNN to learn this sequence as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xtrain=np.reshape(df['X'], (df.shape[0], 1, 1))&#xA;#to match dimension of input shape for SimpleRNN layer.  &#xA;&#xA;model=Sequential()&#xA;model.SimpleRNN(2, input_shape=(None,1)))&#xA;model.add(Dense(1))&#xA;model.compile(optimizer='adam', loss='mse')&#xA;model.fit(x=xtrain, y=df['Y'], epochs=200, batch_size=5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;however, I find that this implementation results in a local minima which predicts constant value(~50) for all test observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone help me with the right way of implementing a basic RNN in Keras to learn this sequence? &lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="46535" LastEditorUserId="46535" LastEditDate="2018-02-21T09:11:36.397" LastActivityDate="2018-09-02T01:00:54.390" Title="Learning a simple sequence with RNN (Keras)" Tags="|python|neural-network|keras|time-series|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28087" PostTypeId="1" CreationDate="2018-02-21T08:43:47.870" Score="1" ViewCount="85" Body="&lt;p&gt;Inspired from this &lt;a href=&quot;https://datascience.stackexchange.com/questions/12909/definition-of-a-model-in-machine-learning&quot;&gt;question&lt;/a&gt;, given that there is no standard definition of a data-science model, is there a way to standardize the communication about any &lt;em&gt;data-science model&lt;/em&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just like &lt;em&gt;material design&lt;/em&gt; is a &lt;em&gt;design language&lt;/em&gt; to communicate a &lt;em&gt;UI design&lt;/em&gt;, is there a &lt;em&gt;similar design&lt;/em&gt; language to talk about &lt;strong&gt;&lt;em&gt;data-science models&lt;/em&gt;&lt;/strong&gt; as well?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or any &lt;strong&gt;other alternative&lt;/strong&gt; or &lt;strong&gt;industry standard artifacts&lt;/strong&gt; that communicates intent, purpose, approach and design of &lt;em&gt;data-science model&lt;/em&gt;?&lt;/p&gt;&#xA;" OwnerUserId="46293" LastActivityDate="2018-02-21T08:43:47.870" Title="Design language or artifact for data science model" Tags="|machine-learning|data-science-model|machine-learning-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28088" PostTypeId="1" AcceptedAnswerId="28170" CreationDate="2018-02-21T08:50:41.497" Score="1" ViewCount="571" Body="&lt;p&gt;I am getting the below error when i run the R code for glm():&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; fmla = 'status_id~ratings'&#xA;&amp;gt; logisticmodel &amp;lt;- glm(fmla, data = playdata, family=binomial(link=&quot;logit&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could you please correct me where I have gone wrong, I googled a lot on it and the only answer is generally naming convention, however, this is not case for me since I tried with changing the data by placing all the values as 2's and it worked -  so it is not a naming issue with variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the data description:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str(playdata)&#xA;&#xA;'data.frame':   24160 obs. of  13 variables:&#xA; $ idd         : int  57251659 63385939 51939145 64339389 33725679 47000250 62738883 33725679 53589441 36670488 ...&#xA; $ status_id       : int  1 1 1 1 1 1 1 1 1 1 ...&#xA; $ id              : int  22820543 22953283 22919397 22699949 22658030 22720403 22581860 22915483 22621108 22651736 ...&#xA; $ group_id        : int  2 2 2 2 2 2 2 2 2 2 ...&#xA; $ created_date    : POSIXct, format: &quot;2017-03-31 12:45:10&quot; &quot;2017-04-04 10:50:11&quot; &quot;2017-04-03 16:40:04&quot; &quot;2017-03-28 14:40:48&quot; ...&#xA; $ question_id     : int  20073221 20219031 20185301 19948471 19906458 18651404 19816152 20175471 19865888 19897662 ...&#xA; $ surftoanswertime: int  533 484 98 476 388 0 741 757 2222 381 ...&#xA; $ skiprate        : num  0.981 0.95 0.993 0.875 0.966 ...&#xA; $ ratings         :integer64 0 0 1 2 0 0 1 1 ... &#xA; $ pos             :integer64 0 0 1 2 0 0 1 0 ... &#xA; $ negratings      :integer64 0 0 0 0 0 0 0 1 ... &#xA; $ cf              : num  NA NA 1 1 NA NA 1 0 NA 1 ...&#xA; $ cf1             : num  -1 -1 1 1 -1 -1 1 0 -1 1 ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Actual formula is as below -  only the ratings and negratings are throwing the above error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fmla = 'status_id~skiprate+ratings+negratings+surftoanswertime+cf1'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Could you please help me in understanding and correcting the error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; logisticmodel &amp;lt;- glm(fmla, data = playdata, family=binomial(link=&quot;logit&quot;))&#xA;Error in names(coef) &amp;lt;- xnames : names() applied to a non-vector&#xA;In addition: Warning messages:&#xA;1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :&#xA;  non-finite coefficients at iteration 1&#xA;2: glm.fit: algorithm did not converge &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I use more columns I am getting the below error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Error in `*tmp*`[fit$pivot] : object of type 'closure' is not subsettable&#xA;In addition: Warning messages:&#xA;1: In glm.fit(x = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  :&#xA;  non-finite coefficients at iteration 1&#xA;2: glm.fit: algorithm did not converge &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23731" LastEditorUserId="23731" LastEditDate="2018-02-21T09:15:52.117" LastActivityDate="2018-02-22T08:33:58.940" Title="r glm - Error in names(coef) &lt;- xnames only for 2 columns in data" Tags="|r|glm|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28090" PostTypeId="2" ParentId="28071" CreationDate="2018-02-21T09:02:37.753" Score="1" Body="&lt;p&gt;This can be fairly good if you deal with few years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general it's not a good practice, since asymptotically will lead you to have almost the same fraction of crimes per all the considered years. But unless you have data from the last 10000 years, you can simply ignore the asymptotic behavior a continue the way you are doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Otherwise, you can normalize the data to a reference value, say the maximum number of crimes on the series. This is more robust.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-02-21T09:02:37.753" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28091" PostTypeId="1" CreationDate="2018-02-21T09:09:52.130" Score="6" ViewCount="191" Body="&lt;p&gt;I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token based document representation (TF-IDF or embeddings) and computing similarity on these but somehow I find this approach a bit clumsy. I want to detect &quot;handprints&quot; (characteristic comment and abbreviation style, folder structure, used 3rd party tools, order of elements in the code etc.) that seem out of the scope of this approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, I cannot find place for any proper machine learning here. Clearly, finding weights for the any future quantitative features would be nice but this similarity task is not classification/regression, so how to define the target? Clustering seems to be a better tool but we cannot define as many categories as potential authors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you kindly suggest any more reliable method? Does any literature exist for this topic?&lt;/p&gt;&#xA;" OwnerUserId="21560" LastActivityDate="2020-04-17T01:00:09.277" Title="How to compute document similarities in case of source codes?" Tags="|machine-learning|nlp|text-mining|similar-documents|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28092" PostTypeId="2" ParentId="28069" CreationDate="2018-02-21T09:39:15.887" Score="2" Body="&lt;p&gt;Fairly simply, for 'A' only:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;max_chars = 300&#xA;for index, row in df.iterrows() :&#xA;    print(row['A'][:max_chars], row['B'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-21T09:39:15.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28093" PostTypeId="2" ParentId="28085" CreationDate="2018-02-21T09:46:21.193" Score="1" Body="&lt;p&gt;I guess it depends on exactly what you want - do you want a forecast per store? If so you would need to leave the data set aggregated only to a store level. If not, and you want the overall sales by day, then you are OK to leave it aggregated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another consideration is whether there are many store-level features e.g. location, weather, local demographics. These features may be useful for the model to learn from and would lose resolution if store level data is aggregated, therefore leaving it un-aggregated would be better. The counter point to this is whether you have enough data for each store to learn representative trends! So it does get kind of circular and depends on the size of your data set.&lt;/p&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-21T09:46:21.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28094" PostTypeId="1" AcceptedAnswerId="29118" CreationDate="2018-02-21T10:26:25.940" Score="5" ViewCount="982" Body="&lt;p&gt;How is the closed form solution to linear regression derived using matrix derivatives as opposed to using the trace method as Andrew Ng does in his Machine learning lectures.  Specifically, I am trying to understand how Nando de Frietas does it &lt;a href=&quot;https://youtu.be/DHspIG64CVM?t=38m46s&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to find the value of $ \theta $ that minimizes $ J(\theta)=(X\theta-Y)^{T}(X\theta-Y) $, where $\theta \in \mathbb{R}^{N \times 1},  X \in \mathbb{R}^{M \times N}$, and $Y \in \mathbb{R}^{M \times 1}$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nabla_{\theta}J(\theta) = \nabla_{\theta} (X\theta-Y)^{T}(X\theta-Y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ = \nabla_{\theta} (\theta^{T} X^{T}-Y^{T})(X\theta-Y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ = \nabla_{\theta} (\theta^{T} X^{T}X\theta-\theta^{T} X^{T}Y - Y^{T}X\theta + Y^{T}Y) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $\theta^{T} X^{T}Y$ is a scalar, so $\theta^{T} X^{T}Y = (\theta^{T} X^{T}Y)^{T} = Y^{T} X \theta$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nabla_{\theta}J(\theta) = \nabla_{\theta}(\theta^{T} X^{T}X\theta-Y^{T} X \theta - Y^{T}X\theta + Y^{T}Y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ = \nabla_{\theta}(\theta^{T} X^{T}X\theta- 2 Y^{T} X \theta + Y^{T}Y)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta + \nabla_{\theta} Y^{T}Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ = \nabla_{\theta} \theta^{T} X^{T}X\theta - \nabla_{\theta} 2 Y^{T} X \theta $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I apply the matrix derivatives described in that video to solve this?  He skip steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;Below is the suggested strategy of removing theta by differentiating, then taking the inverse of both sides.  So looking at one term at a time, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \nabla_{\theta} \theta^{T} X^{T}X\theta = ? $&#xA;How do I differntiate this?  This is like differntiating $x\alpha_{1} \alpha_{2} x$ w.r.t. x in the scalar case. I need to combine those $\theta$ terms to hit them with the derivative. Transposing seems to result in the same expression:&#xA;$$ (\nabla_{\theta} \theta^{T} X^{T}X\theta)^{T} = \nabla_{\theta} \theta^{T} X^{T}X\theta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking at the second term, we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \nabla_{\theta} 2 Y^{T} X \theta = 2 X^{T} Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting together this, we have:&#xA;$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T} Y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Knowing the solution is $\theta = (X^{T}X)^{-1}X^{T}Y$ we can reverse engineer the problem, but I am just not seeing it.  And how do we get rid of that 2 factor?&lt;/p&gt;&#xA;" OwnerUserId="46595" LastEditorUserId="46595" LastEditDate="2018-02-22T01:50:54.550" LastActivityDate="2018-04-06T13:20:24.813" Title="Closed form solution of linear regression via least squares using matrix derivatives" Tags="|machine-learning|linear-regression|linear-algebra|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28095" PostTypeId="1" AcceptedAnswerId="28113" CreationDate="2018-02-21T11:00:18.077" Score="4" ViewCount="9108" Body="&lt;p&gt;after reading the state of the art about object detection using CNN (R-CNN Faster R-CNN ,YOLO, SSD...) I was wondering if there is a method that use RNN's or that combine the use of CNN's and RNN's for object detection ??&#xA;Thank you &lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T16:38:37.513" LastActivityDate="2021-02-13T16:38:37.513" Title="Is There any RNN method used for Object detection" Tags="|deep-learning|neural-network|computer-vision|convolutional-neural-network|rnn|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28097" PostTypeId="2" ParentId="28013" CreationDate="2018-02-21T11:37:22.347" Score="0" Body="&lt;p&gt;I would strongly recommend to also have a look at Graph Theoretic measures as your problem can be easily and properly seen as a &lt;a href=&quot;https://datascience.stackexchange.com/a/27841/8878&quot;&gt;Bipartite-Graph Clustering&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-02-21T11:37:22.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28099" PostTypeId="2" ParentId="28085" CreationDate="2018-02-21T12:19:29.673" Score="0" Body="&lt;p&gt;I guess you are trying to predict the future sales based on the data from two years that each store gathered. So I would suggest you to train model on each individual set for each store which will predict the forecast of the day for that store and feed the prediction to the separate model which will predict the overall forecast on all the stores based upon the individual store prediction, thus you can get individual predictions as well as overall forecast from the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt; Based on your comments&lt;/strong&gt;&#xA;Remembers it is a bit time consuming but i think it worth spending for good results&#xA;you can skip it and learn from tutorials where i learned alot &lt;strong&gt;(not promoting)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;They are not for your specific purpose but close to yours&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can learn the whole process of prediction in the following tutorials&#xA; &lt;a href=&quot;https://pythonprogramming.net/machine-learning-pattern-recognition-algorithmic-forex-stock-trading/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tutorials&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here we go step by step what i had explained&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; First normalize all the data sets from each store &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; then Choose the algorithm you want to use&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; then Arrange the store data in a organized way because you may get bit confused while training&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; Initially Just pick one data set and tune the hyper parameters and understand working of those parameters carefully because your going to them a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; When you get the point Start from the top to bottom of store list and &#xA;Store the trained model for each store with similar names&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;-&gt;&lt;/strong&gt; When your done with all the stores you will be getting predictions for each as you requested and then collect those prediction and average them and you would be getting approximate predication from all the stores &lt;/p&gt;&#xA;" OwnerUserId="43438" LastEditorUserId="43438" LastEditDate="2018-02-22T19:34:38.067" LastActivityDate="2018-02-22T19:34:38.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28100" PostTypeId="2" ParentId="28070" CreationDate="2018-02-21T12:24:31.327" Score="0" Body="&lt;p&gt;I think following &lt;a href=&quot;https://gist.github.com/stewartpark/187895beb89f0a1b3a54&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt; helps you, I have gone trough those tutorials previously.&#xA;Also take a look at &lt;a href=&quot;https://machinelearningmastery.com/recurrent-neural-network-algorithms-for-deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial &lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43438" LastEditorUserId="28175" LastEditDate="2018-02-21T13:08:25.970" LastActivityDate="2018-02-21T13:08:25.970" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28101" PostTypeId="1" CreationDate="2018-02-21T12:42:03.177" Score="1" ViewCount="856" Body="&lt;p&gt;Is there any Package for PCA for data having Categorical variable ?&lt;/p&gt;&#xA;" OwnerUserId="16879" LastActivityDate="2018-04-22T22:41:39.753" Title="PCA with Catagorical Variable in R" Tags="|r|pca|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28102" PostTypeId="2" ParentId="11825" CreationDate="2018-02-21T12:45:44.873" Score="1" Body="&lt;p&gt;Here's a &lt;a href=&quot;https://www.wiley.com/en-us/Fraud+Analytics+Using+Descriptive%2C+Predictive%2C+and+Social+Network+Techniques%3A+A+Guide+to+Data+Science+for+Fraud+Detection-p-9781119133124&quot; rel=&quot;nofollow noreferrer&quot;&gt;great book on Fraud Analytics&lt;/a&gt;. It's quite comprehensive and has a detailed list of original research references and other textbooks at the end of each chapter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.wiley.com/en-us/Fraud+Analytics+Using+Descriptive%2C+Predictive%2C+and+Social+Network+Techniques%3A+A+Guide+to+Data+Science+for+Fraud+Detection-p-9781119133124&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fraud Analytics Using Descriptive, Predictive, and Social Network Techniques: A Guide to Data Science for Fraud Detection&#xA;Bart Baesens, Veronique Van Vlasselaer, Wouter Verbeke&#xA;ISBN: 978-1-119-13312-4&#xA;Aug 2015&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39233" LastActivityDate="2018-02-21T12:45:44.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28103" PostTypeId="2" ParentId="28070" CreationDate="2018-02-21T13:00:55.003" Score="0" Body="&lt;p&gt;Use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;import numpy as np&#xA;&#xA;&#xA;&#xA;a = np.array([[1, 1], [0, 1], [1, 0], [0, 0]])&#xA;b = np.array([[0], [1], [1], [0]])&#xA;&#xA;model = keras.Sequential()&#xA;model.add(keras.layers.Dense(2, activation = 'sigmoid', input_shape = (2, )))&#xA;model.add(keras.layers.Dense(1, activation = 'sigmoid'))&#xA;&#xA;model.compile(optimizer = keras.optimizers.rmsprop(), metrics = ['accuracy'], loss = 'binary_crossentropy')&#xA;model.fit(x = a, y = b, epochs = 750)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you don't get &lt;code&gt;100%&lt;/code&gt; accuracy increase the number of epochs. Also take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt; which may help you how increasing the size of each layer affects the learning process.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-02-21T13:05:56.647" LastActivityDate="2018-02-21T13:05:56.647" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28106" PostTypeId="2" ParentId="28101" CreationDate="2018-02-21T13:48:46.690" Score="0" Body="&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any dimensionality reduction technique suppose you are dealing with algebraic values, and categorical variables do not have such meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a categorical variable which indicate - as example - families' pets in one of (cat, dog, red fish) mapped to (0, 1, 2), you simply cannot say that the distance between cat and dog is equal to one, because those categories do not lie in an algebraic space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in those cases, you do the one-hot encoding of the variables, a.k.a. dummyfication. Then you can safely apply PCA, or any other dimensionality reduction technique.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-02-21T13:48:46.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28108" PostTypeId="2" ParentId="28091" CreationDate="2018-02-21T14:39:49.563" Score="1" Body="&lt;p&gt;Why not use &lt;em&gt;all&lt;/em&gt; non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, I have a tendency in python to copy entire import blocks from one script to another, whether or not I actually need all the imports or not. If someone uses a copied import or include statement, then their imports will all be in the same order. If you track the order of imports, you may be able to find patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also issues of whether you use spaces or tabs for indentation, preferred number of spaces or tabs, etc. For languages with braces, you can see if braces are used following an if statement, or at the start of the next line. I believe this will give you enough 'handprints' to identify people distinctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you do this, you can use multilevel clustering to attempt to assign documents first to a group of possible originators (i.e. all originators who use a certain IDE or text editor). Then you can go through and look for certain patterns within each group, clustering again within each cluster.&lt;/p&gt;&#xA;" OwnerUserId="30472" LastActivityDate="2018-02-21T14:39:49.563" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28109" PostTypeId="2" ParentId="22736" CreationDate="2018-02-21T14:50:20.993" Score="1" Body="&lt;p&gt;Is pandas itself available on the cluster?&#xA;If so, you may try to go with the in-built &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;read_excel()&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may also try the &lt;a href=&quot;https://github.com/ZuInnoTe/hadoopoffice/wiki&quot; rel=&quot;nofollow noreferrer&quot;&gt;HadoopOffice&lt;/a&gt; library, it contains a &lt;a href=&quot;https://github.com/ZuInnoTe/spark-hadoopoffice-ds/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spark DataSource&lt;/a&gt;, also available as &lt;a href=&quot;https://spark-packages.org/package/ZuInnoTe/spark-hadoopoffice-ds&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spark Package&lt;/a&gt;, you can easily test it out without any installation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;$SPARK_HOME/bin/pyspark --packages com.github.zuinnote:spark-hadoopoffice-ds_2.11:1.0.4&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some people also &lt;a href=&quot;https://stackoverflow.com/a/44197455/3613057&quot;&gt;recommend the &lt;code&gt;Spark Excel&lt;/code&gt; dependency&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="46616" LastEditorUserId="29575" LastEditDate="2018-02-21T15:46:52.983" LastActivityDate="2018-02-21T15:46:52.983" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28110" PostTypeId="2" ParentId="26743" CreationDate="2018-02-21T14:58:46.010" Score="0" Body="&lt;p&gt;You'll have to do some feature engineering on your data. Beneficiary and Customer ID has no meaning for machine learning, so you first have to join all other data, taking care of date types (split hour, day of week, month) and should be ordereed by the ones you will not use for training (beneficiary, customerID). than you can create new features like for each transaction the time spent since the last one for each customer, count of transcactions by day/day of wee/month, min, max, median amout spent and son on. It'll be a binary classification (SVM, random forest, logistic regression, etc.).&lt;/p&gt;&#xA;" OwnerUserId="46617" LastActivityDate="2018-02-21T14:58:46.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28112" PostTypeId="1" AcceptedAnswerId="28184" CreationDate="2018-02-21T15:20:31.640" Score="1" ViewCount="556" Body="&lt;p&gt;I'm working on an LSTM model that uses dating conversation dataset. Any suggestion as to where I can get such data? Any ideas will be much appreciated.&lt;/p&gt;&#xA;" OwnerUserId="46618" LastEditorUserId="46618" LastEditDate="2018-02-22T13:41:17.700" LastActivityDate="2018-11-04T14:43:49.997" Title="Speed Date or Dating Conversation Dataset / Corpora" Tags="|dataset|" AnswerCount="2" CommentCount="0" ClosedDate="2021-03-12T01:31:55.613" ContentLicense="CC BY-SA 3.0" />
  <row Id="28113" PostTypeId="2" ParentId="28095" CreationDate="2018-02-21T15:31:10.993" Score="7" Body="&lt;p&gt;Yes, there have been many attempts, but perhaps the most noteable one is the approach described in the paper of &lt;a href=&quot;https://arxiv.org/abs/1412.2306&quot; rel=&quot;noreferrer&quot;&gt;Andrej Karpathy and  Li Fei-Fei&lt;/a&gt; where they connect a CNN and RNN in series (CNN over image region + bidirectional RNN + Multimodal RNN) and use this for labeling a scene with a whole sentence. Though, this one is more than just object detection as it leverages a data set of scenes and their descriptions to generate natural language descriptions of new unseen images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example is &lt;a href=&quot;http://ieeexplore.ieee.org/document/7298958/&quot; rel=&quot;noreferrer&quot;&gt;Ming Liang and Xiaolin Hu's approche&lt;/a&gt; where they mix a CNN with an RNN and use this architecture for better object detection. As Ming and Xiaolin explained in their paper (linked above), the RNN is used to improve the CNN:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A prominent difference is that CNN is typically a feed-forward architecture while in the visual system recurrent connections are abundant. Inspired by this fact, we propose a recurrent CNN (RCNN) for object recognition by incorporating recurrent connections into each convolutional layer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46123" LastEditorUserId="13413" LastEditDate="2018-06-15T00:30:14.583" LastActivityDate="2018-06-15T00:30:14.583" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28114" PostTypeId="1" CreationDate="2018-02-21T15:35:39.467" Score="2" ViewCount="2971" Body="&lt;p&gt;I have a dataset of past emails sent, when they were sent and if they were opened. What would be the best model to predict the best time to send email?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of modeling it as time series with values 1(email opened), 0(email not sent) and -1(email not opened).&lt;/p&gt;&#xA;" OwnerUserId="46620" LastActivityDate="2021-06-07T12:59:11.400" Title="Predicting best time to send email" Tags="|machine-learning|machine-learning-model|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28116" PostTypeId="1" AcceptedAnswerId="28119" CreationDate="2018-02-21T16:02:03.713" Score="0" ViewCount="195" Body="&lt;p&gt;I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed. &#xA;For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was just thinking that during data analysis, does it cause any error or these errors can be ignored?&#xA;Or is there a better option for the same purpose&lt;/p&gt;&#xA;" OwnerUserId="46623" LastActivityDate="2018-02-21T16:23:04.910" Title="Inappropriate stemming in nltk.stem" Tags="|nltk|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28117" PostTypeId="2" ParentId="28114" CreationDate="2018-02-21T16:04:28.527" Score="4" Body="&lt;p&gt;Good question, probably a good handful of ways to look at this as it sounds like a classic classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Define the label&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First step, before you choose a model, would be to define the label. As you suggested, I would go for email opened = 1 and email not opened = 0. I would then think about throwing the email not sent (-1) data away, as I'm not sure you can learn anything about a probability of opening when it wasn't sent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would then be important to check the balance of the label: are they mostly opened or mostly ignored? This will influence any decision to add a class weight if it is imbalanced towards one class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Define features&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So time series could work, but I think another good angle might be to bin the send times into half hourly slots (or shorter, depending on your needs) and treat them as a categorical variable. You would then one-hot encode this feature. Anything else you can add to this will probably help, such as the kind of person it was sent to and what country they're in etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Choose model&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's classification, so have a play around with either logistic regression or something tree based (random forests, gradient boosted trees) to get something going quickly. It's usually a case of trial and error to find the best one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use traditional metrics of precision/recall/accuracy to measure how good it is. Make sure you know which metric is most important to your problem when optimising the model.&lt;/p&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-21T16:04:28.527" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28118" PostTypeId="2" ParentId="28114" CreationDate="2018-02-21T16:08:47.310" Score="0" Body="&lt;p&gt;Have you spent any time exploring your data descriptively? Have you look at at simple counts and percentages of emails opened vs not opened by year, month, day, and perhaps time of day? The reason I'm asking is that you may find some answers simply by conducting an exploratory analysis and reviewing descriptives before you jump into modeling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regard to modeling, you could start with a logistic regression model where the outcome is email opened (1) vs email not opened (0). I don't think you should include email not sent as you do not need to predict that action; you know if the email was sent or not. For your predictors, you could partition your datetimes into multiple features, such as month, day, time of day, year, etc. You could also consider including additional predictors in your model, such as recipients' age, gender, location, etc., if these data are available to you.&lt;/p&gt;&#xA;" OwnerUserId="46546" LastActivityDate="2018-02-21T16:08:47.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28119" PostTypeId="2" ParentId="28116" CreationDate="2018-02-21T16:17:08.013" Score="0" Body="&lt;p&gt;The renown algorithm for stemming is &lt;a href=&quot;http://snowball.tartarus.org/algorithms/porter/stemmer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Porter stemming algorithm&lt;/a&gt;. Hence, you can use &lt;code&gt;stemmer = nltk.stemmer.PorterStemmer()&lt;/code&gt; for the stemming. you can test it using &lt;code&gt;stemmer.stem('poorly')&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, you can see &lt;a href=&quot;https://stackoverflow.com/a/24663617/3768871&quot;&gt;this post&lt;/a&gt; for more details.&lt;/p&gt;&#xA;" OwnerUserId="27665" LastEditorUserId="27665" LastEditDate="2018-02-21T16:23:04.910" LastActivityDate="2018-02-21T16:23:04.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28120" PostTypeId="1" AcceptedAnswerId="28155" CreationDate="2018-02-21T16:17:54.603" Score="2" ViewCount="2949" Body="&lt;p&gt;I'm a complete begginer at Keras. In the Inception v3 example at &lt;a href=&quot;https://keras.io/applications/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/applications/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# create the base pre-trained model&#xA;base_model = InceptionV3(weights='imagenet', include_top=False)&#xA;&#xA;# add a global spatial average pooling layer&#xA;x = base_model.output&#xA;x = GlobalAveragePooling2D()(x)&#xA;# let's add a fully-connected layer&#xA;x = Dense(1024, activation='relu')(x)&#xA;# and a logistic layer -- let's say we have 200 classes&#xA;predictions = Dense(200, activation='softmax')(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As at first step it adds a GlobalAveragePooling2D layer, which is described as:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Global average pooling operation for spatial data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What GlobalAveragePooling2D does and why the example uses it instead of something like Flatten? Which information is averaged?&lt;/p&gt;&#xA;" OwnerUserId="46622" LastActivityDate="2018-02-22T01:58:45.647" Title="GlobalAveragePooling2D in inception v3 example" Tags="|python|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28123" PostTypeId="2" ParentId="10399" CreationDate="2018-02-21T17:04:05.567" Score="2" Body="&lt;p&gt;Replace softmax activation with sigmoid activation function in the last layer. Sigmoid convert the scores to the range of [-1 to 1]. Then you can apply the thresholds on this score value. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;While training, you have to consider the multiple ground truths for that image.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then while evaluating, you will get multiple predictions of the image unlike softmax which makes sure the sum of the probabilities of each class is equal to 1.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46628" LastActivityDate="2018-02-21T17:04:05.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28124" PostTypeId="2" ParentId="28062" CreationDate="2018-02-21T17:13:02.560" Score="1" Body="&lt;p&gt;I suggest having a look at &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;the relevant documentation&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;There, it states:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;rotation_range: Int. Degree range for random rotations.&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;horizontal_flip: Boolean. Randomly flip inputs horizontally.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Saying each of the operations is applied randomly, I would say, means your images will be generated sometime with and sometimes without the augmentation steps, independently from one another.&lt;/p&gt;&#xA;&lt;p&gt;If that doesn't convince you, here is the &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/preprocessing/image.py#L679&quot; rel=&quot;nofollow noreferrer&quot;&gt;relevant snippet from the source code&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;if self.horizontal_flip:&#xA;    if np.random.random() &amp;lt; 0.5:&#xA;        x = flip_axis(x, img_col_axis)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The code for the rotation step is a little more involved, but contained within the same class that I liked above.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-05T10:23:42.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28126" PostTypeId="1" AcceptedAnswerId="28127" CreationDate="2018-02-21T17:51:10.707" Score="1" ViewCount="84" Body="&lt;p&gt;If I use a rnn model for time series forecasting how frequently do I have to retrain the model.&lt;/p&gt;&#xA;" OwnerUserId="40793" LastActivityDate="2018-02-21T18:06:34.337" Title="What should be the training frequency of a rnn model for timeseries prediction?" Tags="|time-series|rnn|" AnswerCount="1" CommentCount="0" ClosedDate="2018-02-25T15:17:01.250" ContentLicense="CC BY-SA 3.0" />
  <row Id="28127" PostTypeId="2" ParentId="28126" CreationDate="2018-02-21T18:06:34.337" Score="1" Body="&lt;p&gt;It depends on any number of factors. What kind of accuracy are you currently getting? How often are you getting new predictions from your model? Do you have human intervention at some point to check your errors, relabel, and improve your training sets? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible that you don't need a schedule, per-se. Depending on your data, the pipeline and the programming of your model, you could also just have continuous learning built into your model with a new release of said model after every epoch. So, really, a lot of the answers you are seeking are very specific to your implementation. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-21T18:06:34.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28128" PostTypeId="1" CreationDate="2018-02-21T18:15:59.147" Score="-1" ViewCount="88" Body="&lt;p&gt;I'm creating a system to evaluate a risk level that grows as it approaches in time to the crisis event. This risk level ranges from 0 to 100, it's a self made index, totally arbitrary.  I have a matrix containing features (X) for  each time window. Is there any method to obtain the best model for the values of Y? I mean, knowing how Y should increase (linear, exponential, gaussian) according to the variables I have. &lt;/p&gt;&#xA;" OwnerUserId="46633" LastEditorUserId="29575" LastEditDate="2018-02-21T18:37:26.647" LastActivityDate="2020-08-25T08:01:45.220" Title="Assigning a value to Y for regression" Tags="|machine-learning|regression|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28129" PostTypeId="1" AcceptedAnswerId="28174" CreationDate="2018-02-21T18:34:29.463" Score="1" ViewCount="129" Body="&lt;p&gt;I have a large training set of ~300GB (which is a subset of an even larger dataset ~15TB). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying train a Convnet with Keras (Tensorflow backend) to do something similar to semantic segmentation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't find any valuable resources to handle such large data. Any suggestions for best practices for such humungous data is appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="18465" LastEditorUserId="29169" LastEditDate="2021-02-13T16:38:27.800" LastActivityDate="2021-02-13T16:38:27.800" Title="Training a Convnet on 300GB data" Tags="|bigdata|convolutional-neural-network|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28130" PostTypeId="1" AcceptedAnswerId="28139" CreationDate="2018-02-21T18:59:37.527" Score="1" ViewCount="151" Body="&lt;p&gt;For regression type problem we know the result is a continuous value, so how is it be cross validated?&#xA;In classification type problem we know the class label so easy to compare, but how is it compare in regression type problem? &lt;/p&gt;&#xA;" OwnerUserId="46433" LastActivityDate="2018-02-21T20:11:21.743" Title="How cross validation works for regression?" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28131" PostTypeId="1" AcceptedAnswerId="28140" CreationDate="2018-02-21T19:06:46.830" Score="1" ViewCount="286" Body="&lt;p&gt;Here is the sample data I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the values are strings and I need to encode them into vectors for training ML models using this data.&#xA;How do I make sure that the strings are always converted to the &quot;same&quot; vectors everytime?&#xA;I notice that when I try a test input with the same value as the training data, it gets vectorized into a different integer.&#xA;What is the standard procedure for preserving the mapping between String &amp;lt;--&gt; Hashed Integer representation so that i get the same hash everytime?&lt;/p&gt;&#xA;" OwnerUserId="39076" LastActivityDate="2018-02-21T20:18:58.877" Title="Vectorizing text data for ML models" Tags="|machine-learning|python|scikit-learn|random-forest|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28132" PostTypeId="2" ParentId="20274" CreationDate="2018-02-21T19:26:09.827" Score="1" Body="&lt;p&gt;I might suggest a change to your theoretical setup:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It sounds like you are trying to maximize an equation f(t), where t is the time of dispatch, and f(t) is the risk adjusted chance of purchasing (as you note it can be 0-many).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case, it is likely better to use a setup like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;===========================================================&#xA;  customer campaign    campaign_time       total_responses_within_1week   &#xA;-----------------------------------------------------------&#xA;1   100       a     2017-01-01 06:50:01       0&#xA;2   101       a     2017-01-01 06:50:01       3&#xA;3   101       a     2017-01-01 06:50:01       2&#xA;4   100       b     2017-01-07 06:30:21       1&#xA;5   101       b     2017-01-07 06:30:21       1&#xA;6   100       c     2017-01-14 06:43:55       1&#xA;7   101       d     2017-01-21 14:02:01       1&#xA;-----------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unless you have specific internal data or journal research that directly indicates a relationship between speed of response and likelihood/number of purchases, I might avoid including it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if you need to estimate time of response for issues like labor or inventory, you could use a probabilistic model, but I'll avoid making specific recommendations there, as my relative strength is in regression models. You'd want the function to output a probabilistic curve over a given time period, and then sum all of those curves together (e.g. to estimate that during Weds morning, you will have 80-109 orders, so need 5 customer service reps).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your goal is purely response frequency (within a relevant time), time of response doesn't need to be modeled, however.&lt;/p&gt;&#xA;" OwnerUserId="46277" LastActivityDate="2018-02-21T19:26:09.827" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28133" PostTypeId="2" ParentId="28131" CreationDate="2018-02-21T19:31:33.183" Score="0" Body="&lt;p&gt;Have you looked into a tokenizer? By your question, I can't tell if either (1) you don't know about tokenizing or (2) you are looking for an alternative to tokenizing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will assume the former and suggest that you read about tokenizing and go down that road. Keras provides a very simple approach to tokenizing and sequences that you can implement quickly and should get what you're looking for. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-21T19:31:33.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28134" PostTypeId="2" ParentId="27914" CreationDate="2018-02-21T19:53:55.703" Score="3" Body="&lt;p&gt;It seems the models hosted on &lt;a href=&quot;https://storage.googleapis.com/download.tensorflow.org/models&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; are not listed anywhere publically (or at least my research failed!). However, there are official and several nonofficial &lt;a href=&quot;https://github.com/topics/pretrained-models&quot; rel=&quot;nofollow noreferrer&quot;&gt;GitHub&lt;/a&gt; repositories with high-level TensorFlow model definitions and pretrained weights. For example:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/Cadene/pretrained-models.pytorch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nonofficial&lt;/a&gt;: that includes &lt;strong&gt;NASNet&lt;/strong&gt;, &lt;strong&gt;ResNeXt&lt;/strong&gt;, &lt;strong&gt;ResNet&lt;/strong&gt;, &lt;strong&gt;InceptionV4&lt;/strong&gt;, &lt;strong&gt;InceptionResnetV2&lt;/strong&gt;, &lt;strong&gt;Xception&lt;/strong&gt;, &lt;strong&gt;DPN&lt;/strong&gt; (need PyTorch).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tensorflow/models&quot; rel=&quot;nofollow noreferrer&quot;&gt;Official&lt;/a&gt;: contains a wide range of official and research models such as &lt;strong&gt;resnet&lt;/strong&gt;, &lt;strong&gt;wide-deep&lt;/strong&gt;, &lt;strong&gt;inception&lt;/strong&gt;, &lt;strong&gt;delf&lt;/strong&gt;, and &lt;strong&gt;tcn&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="46505" LastEditorUserId="26019" LastEditDate="2020-08-19T18:04:48.453" LastActivityDate="2020-08-19T18:04:48.453" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28135" PostTypeId="2" ParentId="28130" CreationDate="2018-02-21T19:57:20.967" Score="2" Body="&lt;p&gt;The idea behind cross validation is to understand the performance of some measure of your model's performance on unseen data. This can be applied to loads of different statistics, not just ones relevant to classification like accuracy. Common measures for holdout performance of regression models include MSE and MAPE, either (or both) of which can be cross-validated over.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-21T19:57:20.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28138" PostTypeId="2" ParentId="28128" CreationDate="2018-02-21T20:07:53.287" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;it's a self made index, totally arbitrary&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Therefore how it should increase is totally arbitrary too.  I think the best answer to your question is that you need to think about the behavior you want to see out of your Y variable, and then write an algorithm that exhibits that behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you want the value to suddenly explode as you reach the crisis event? use an exponential. Do you want to gradually approach larger values? Use a linear model. Do you want a function that gets larger until you reach the crisis event and then slowly dissipates? Use a gaussian.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-02-21T20:07:53.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28139" PostTypeId="2" ParentId="28130" CreationDate="2018-02-21T20:11:21.743" Score="1" Body="&lt;p&gt;In both scenarios, we pick one or more performance &lt;a href=&quot;https://www.kaggle.com/wiki/Metrics&quot; rel=&quot;nofollow noreferrer&quot;&gt;measures&lt;/a&gt; and validate the model based on them. In classification, one may choose to use &lt;em&gt;accuracy&lt;/em&gt;, &lt;em&gt;precision&lt;/em&gt;, &lt;em&gt;recall&lt;/em&gt;, or &lt;em&gt;F-score&lt;/em&gt;. In regression, other metrics such as &lt;em&gt;Root Mean Squared Error (RMSE)&lt;/em&gt;, &lt;em&gt;Mean Absolute Error (MAE)&lt;/em&gt;, and &lt;em&gt;R Squared Score (R^2)&lt;/em&gt; might be useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In regression/classification problems that the order of data points matters (eg time series), we cannot use conventional cross-validation. Instead, some &lt;a href=&quot;https://robjhyndman.com/hyndsight/tscv/&quot; rel=&quot;nofollow noreferrer&quot;&gt;special variations&lt;/a&gt; of cross-validation procedure must be used to make sure we do not train our model on the current samples and validate on past instances.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;p.s. In regression problems, the labels do not need to be &quot;continuous&quot; (as we define in the continuity of a function in calculus). They can be discrete but real-valued.&lt;/p&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-02-21T20:11:21.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28140" PostTypeId="2" ParentId="28131" CreationDate="2018-02-21T20:18:58.877" Score="0" Body="&lt;p&gt;Regardless of the &quot;hashing&quot; algorithm that is used in your code, the same strings should be always mapped to unique values unless they only look the same but not really the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please look for common errors such as capital vs non-capital letters, type of whitespaces (tab, blank, cr, ln), number of whitespaces, 1 vs l, 0 vs O, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this does not solve the issue, please provide some real examples (real tags with the different integer hashing) and a little bit more information about the code/algorithm that you used in your project.&lt;/p&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-02-21T20:18:58.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28141" PostTypeId="1" CreationDate="2018-02-21T20:27:14.407" Score="2" ViewCount="2531" Body="&lt;p&gt;I have a partial neural network with several layers of various types, with weights $\theta$; let's call it $F_{\theta}$; and the dimensions of input and output arrays are the same, implemented in Keras with Tensorflow backend. Within this framework, how do train a model of the form $y = F_{\theta}(F_{\theta}(x))$, or more generally some $n$-th iteration of $F_{\theta}$? In other words, I reuse an entire subset of the neural network, with all the same weights, so the full model is deeper but does not have more parameters than the shallower model.&lt;/p&gt;&#xA;" OwnerUserId="46643" LastActivityDate="2018-04-04T15:06:37.483" Title="Reusing a portion of a neural network (with shared weights) in Keras" Tags="|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28145" PostTypeId="2" ParentId="13607" CreationDate="2018-02-21T21:41:33.733" Score="5" Body="&lt;p&gt;It seems that you use MSE as the loss function, from a glimpse on the paper it seems they use NLL (cross entropy), MSE is considered prone to be sensitive to data imbalance among other issues and it may be the cause of the problem you experience, I would try training using categorical_crossentropy loss in your case, moreover learning rate of 0.01 seems too large I would try to play with it and try 0.001 or even 0.0001&lt;/p&gt;&#xA;" OwnerUserId="41421" LastActivityDate="2018-02-21T21:41:33.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28146" PostTypeId="2" ParentId="20274" CreationDate="2018-02-21T21:47:44.427" Score="1" Body="&lt;p&gt;As there are important questions to the scenario and data, I'm sharing some thoughts together with assumed answers to some questions rather than a complete solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, in sample data there is not clear how cases when user has not responded are modeled and/or whether each campaign has been sent to all users (which in real live is never the case as user base is changing over the time). But we can safely assume that there is this information given (either all campaign has been sent to all users - for simplicity or we now which users received with campaign).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then it can be easily imagined that some campaigns can be better suited to be sent earlier in the day, e.g. B2B services - during working hours, and some - rather in the evening, e.g. trip last minute offers. This can heavily influence the response rate for many users and we don't have any information about the campaigns themselves - no features to try to model the content of the campaign and it's impact on responsiveness. If we now that the same type of campaign has been sent always but at various time to various users, that could create a global (audience-wide) trend in responsive rate in different times of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second aspect that influence responsiveness globally is the quality of the campaign - how it is appealing to the users. That could be partially inferred observing responsiveness of users when they receive various campaigns at the same times of the day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That poses a question what is the distribution of the times of the day each campaign separately and in comparison between them - whether there is enough data to try to infer that information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming for starters that these are similar campaigns regarding the quality and they are time-of-the-day agnostic, we can focus on each user separately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look to a distribution of number of responses over the time of the day (e.g. averaged by the number of campaigns sent in the given point of time), you can spot the maximum there, which is the best one-shot candidate for the next campaign. &#xA;This approach however has following caveats. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For the longer term, it will create a &quot;time point bubble&quot; for the user &lt;/li&gt;&#xA;&lt;li&gt;trying to always send him on this time point. That would not reflect the change of the user preferences.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Here you could apply moving average, e.g. over fixed time window (like &quot;last 3 months&quot;) and/or trying other data points from time to time and probably other techniques to gain more diverse and future-proof strategy.&#xA;- The maximum time point might have been an exceptional behaviour while there is much more evidence for a bit smaller number of responses but supported by more time points in other time range. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To take into account various qualities of campaigns, their rating could be created by comparing responsiveness of users to which more than one campaign has been sent in the same time point. Then weigh number of responses by inverted rating of the campaign on the distribution of number of responses for the given user.&lt;/p&gt;&#xA;" OwnerUserId="46624" LastEditorUserId="29575" LastEditDate="2018-02-21T22:37:11.027" LastActivityDate="2018-02-21T22:37:11.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28147" PostTypeId="2" ParentId="28101" CreationDate="2018-02-21T21:57:41.893" Score="2" Body="&lt;p&gt;No, there aren't.  I wish there were.  It would make life a lot easier.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A categorical variable is just that:  a category.  It might be represented by a number but it cannot be treated like one.  For example, if you have a list of cars with their colors, and the colors are denoted this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;1 = red&#xA;2 = blue&#xA;3 = green&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can't really work with them as numbers.  You can't say that blue is the average color, for instance, because the average of 1 + 2 + 3 is 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is create dummy variables, usually a 0 and 1, which then describe the category.  In our example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;0 = not red&#xA;1 = red&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then create dummies for the other colors along similar lines:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;0 = not blue&#xA;1 = blue&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;0 = not green&#xA;1 = green&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done across different columns to make things easier.  It's all part of the mungification of data, getting it into shape so you can run an analysis.&lt;/p&gt;&#xA;" OwnerUserId="46652" LastActivityDate="2018-02-21T21:57:41.893" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28150" PostTypeId="1" CreationDate="2018-02-21T23:06:18.007" Score="2" ViewCount="337" Body="&lt;p&gt;For those who do AI or Machine Learning, is it possible to analyze video? For example, here’s a tidbit from an article I’ve been reading:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;in basketball, data can help coaches determine if a player executed a jump shot well because he was skilled, the pass was good or the defense was bad. If he is skilled, he can be used to lead key strategies. If the pass is good, it identifies players that work well together and use them for the same. If the defense is bad, coaches can strategize plays that take advantage of this weakness.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How would I measure this with AI? I’d like to find the true value of a high school player and how they’d affect their college team.&lt;/p&gt;&#xA;" OwnerUserId="46654" LastEditorUserId="67412" LastEditDate="2019-04-16T15:12:04.370" LastActivityDate="2023-10-18T08:33:45.393" Title="Can I analyze Video in AI?" Tags="|machine-learning|deep-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="28152" PostTypeId="2" ParentId="12384" CreationDate="2018-02-21T23:53:25.837" Score="0" Body="&lt;p&gt;There is no general relationship between size of train dataset and accuracy or coverage on test dataset. To see this in a trivial way, you can augment your training data with unlimited copies of a single training example but this will (probably) not help your accuracy or coverage on the test dataset.&lt;/p&gt;&#xA;" OwnerUserId="1037" LastActivityDate="2018-02-21T23:53:25.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28155" PostTypeId="2" ParentId="28120" CreationDate="2018-02-22T01:58:45.647" Score="3" Body="&lt;p&gt;Prior to GAP, one would flatten your tensor and then add a few fully connected layers in your model.  The problem is that a bunch of parameters in your model end up being attributed to the dense layers and could potentially lead to overfitting.  A natural solution was to add dropout to help regulate that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However a few years ago, the idea of having a Global Average Pooling came into play.  GAP can be viewed as alternative to the whole flatten FC Dropout paradigm.  GAP helps prevent overfitting by doing an extreme form of reduction.  Given a H X W X D tensor, GAP will average the H X W features into a single number and reduce the tensor into a 1 X 1 X D tensor.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The original paper simply applied GAP and then a softmax.  However, it's now common to have GAP followed by a FC layer.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-02-22T01:58:45.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28156" PostTypeId="1" CreationDate="2018-02-22T02:08:22.290" Score="2" ViewCount="3216" Body="&lt;p&gt;Here is the sample data I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started by putting the Tags into a dataframe df and the Label into a separate dataframe df_label.&#xA;Then used a HashingVectorizer to prepare the text for processing by ML models (I want to hash the strings into a unique numerical value so that the ML Models can train on it)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vectorizer = HashingVectorizer()&#xA;&#xA;X_train = vectorizer.transform(df)&#xA;&#xA;y_train = vectorizer.transform(df_label)&#xA;&#xA;clf = RandomForestClassifier(n_jobs=2, random_state=0)&#xA;&#xA;clf.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I execute this, I get: &lt;code&gt;ValueError: Unknown label type: 'unknown' on y_train.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to both Python and ML and I am not sure whether the problem is with my basic logic or whether it is a trivial implementation issue. Appreciate your insight and support.&lt;/p&gt;&#xA;" OwnerUserId="39076" LastEditorUserId="201" LastEditDate="2019-04-07T12:31:53.713" LastActivityDate="2019-04-07T12:31:53.713" Title="Using HashingVectorizer for text vectorization" Tags="|python|scikit-learn|nlp|random-forest|dataframe|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28157" PostTypeId="1" CreationDate="2018-02-22T04:44:07.310" Score="0" ViewCount="1656" Body="&lt;p&gt;Here is the sample data I have:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Scikit Learn Random Forest Classifier.&#xA;I want the Classifier to give higher priority to certain inputs.&#xA;For Example: 100x priority to Tag 1: 10x priority to Tag 2: 1x priority to Tag 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does Scikit-Learn have any parameters where I can specify this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[Since the purpose of the Random Forest is to learn the priority of the inputs, is my question nonsensical?]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Appreciate your insight and support.&lt;/p&gt;&#xA;" OwnerUserId="39076" LastActivityDate="2018-02-22T08:09:02.717" Title="Giving higher priority to certain inputs in SKLearn Random Forest" Tags="|machine-learning|python|scikit-learn|nlp|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28158" PostTypeId="1" AcceptedAnswerId="28159" CreationDate="2018-02-22T05:23:43.347" Score="32" ViewCount="54627" Body="&lt;p&gt;I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?&lt;/p&gt;&#xA;" OwnerUserId="46433" LastEditorUserId="53476" LastEditDate="2019-05-16T04:33:18.523" LastActivityDate="2019-05-16T04:33:18.523" Title="How to calculate the fold number (k-fold) in cross validation?" Tags="|machine-learning|python|scikit-learn|cross-validation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28159" PostTypeId="2" ParentId="28158" CreationDate="2018-02-22T05:38:42.550" Score="25" Body="&lt;p&gt;The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-fold cross validation is used for two main purposes, to tune hyper parameters and to better evaluate the performance of a model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both of these cases selecting $k$ depends on the same thing. You must ensure that the training set and testing set are drawn from the same distribution. And that both sets contain sufficient variation such that the underlining distribution is represented. In a 10-fold cross validation with only 10 instances, there would only be 1 instance in the testing set. This instance does not properly represent the variation of the underlying distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, selecting $k$ is not an exact science because it's hard to estimate how well your fold represents your overall dataset. I usually use 5-fold cross validation. This means that 20% of the data is used for testing, this is usually pretty accurate. However, if your dataset size increases dramatically, like if you have over 100,000 instances, it can be seen that a 10-fold cross validation would lead in folds of 10,000 instances. This should be sufficient to reliably test your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, yes the number of folds depends on the data size. I usually stick with 4- or 5-fold. Make sure to shuffle your data, such that your folds do not contain inherent bias. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-22T05:38:42.550" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28160" PostTypeId="2" ParentId="28158" CreationDate="2018-02-22T05:40:31.687" Score="13" Body="&lt;p&gt;Depends on how much CPU juice you are willing to afford for the same. Having a  lower K means less variance and thus, more bias, while having a higher K means more variance and thus, and lower bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, one should keep in mind the computational costs for the different values. High K means more folds, thus higher computational time and vice versa. So, one needs to find a sweet spot between those by doing a hyper tuning analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you need to keep the size of your data in mind. If your data is very less, then even using a k-fold crossval wouldn't make sense. So, you might want to use a leave-one-out CV (LOOCV).&lt;/p&gt;&#xA;" OwnerUserId="11097" LastActivityDate="2018-02-22T05:40:31.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28161" PostTypeId="1" CreationDate="2018-02-22T05:54:48.737" Score="2" ViewCount="5804" Body="&lt;p&gt;I'm looking for Jupyter extension to plot interactive graphs. For example I need to plot twenty time series lines with order to examine data. Only plugin which I've found is bqplot &lt;a href=&quot;https://github.com/bloomberg/bqplot&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/bloomberg/bqplot&lt;/a&gt; but its documentation is too complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could you advice some other free tool or help with bqplot?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't plot lines with different colors. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;unitIds = commonSeries['UnitId'].unique()&#xA;commonSeries = commonSeries.sort_values(by=['Date','UnitId'], ascending=True)&#xA;commonSeries = commonSeries.set_index(pd.DatetimeIndex(commonSeries['Date']))&#xA;&#xA;plt.clear()&#xA;x_ord = OrdinalScale()&#xA;y_sc = LinearScale()&#xA;lines = []&#xA;&#xA;for unitId in unitIds:    &#xA;    timeseries = commonSeries.loc[commonSeries['UnitId'] == unitId]&#xA;    line = plt.Lines(x=timeseries.index, y=timeseries['Value'], scales={'x': x_ord, 'y': y_sc}, colors=['red','green','blue'])&#xA;    lines.append(line)&#xA;&#xA;plt.Figure(marks=lines) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/O97JW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/O97JW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46673" LastEditorUserId="11097" LastEditDate="2018-02-22T05:58:27.250" LastActivityDate="2018-02-22T14:52:56.300" Title="Interactive plots in Jupyter" Tags="|python|visualization|jupyter|ipython|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28162" PostTypeId="2" ParentId="28161" CreationDate="2018-02-22T05:58:10.420" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://plot.ly/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plotly&lt;/a&gt; is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are &lt;a href=&quot;https://plot.ly/python/offline/#generating-offline-graphs-within-jupyter-notebook&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorials&lt;/a&gt; on Plotly's docs which help you integrate it with Jupyter. &lt;a href=&quot;https://plot.ly/python/ipython-notebook-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Another tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="29575" LastEditDate="2018-02-22T14:09:54.030" LastActivityDate="2018-02-22T14:09:54.030" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28164" PostTypeId="2" ParentId="28156" CreationDate="2018-02-22T06:25:03.350" Score="0" Body="&lt;p&gt;Although it is very difficult to understand your data sample i will try to correct you from what i have understood from your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whenever You use any vectorizer make sure you first apply fit on your corpus/data and then transform it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you have apply transform() without applying fit on your X_train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Important note : You don't have to perform HashingVectorizer on your Label .&#xA;correct your code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   vectorizer = HashingVectorizer()&#xA;   X_train = vectorizer.fit_transform(df)&#xA;   clf = RandomForestClassifier(n_jobs=2, random_state=0)&#xA;   clf.fit(X_train, df_label)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would suggest to use TfidfVectorizer() instead if HashingVectorizer() but before that do some research on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Always refer sklearn documentation so it will help you&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="46356" LastActivityDate="2018-02-22T06:25:03.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28165" PostTypeId="2" ParentId="28161" CreationDate="2018-02-22T06:38:37.357" Score="1" Body="&lt;p&gt;I've found simple, useful and opensource plugin: &lt;a href=&quot;http://holoviews.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://holoviews.org&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46674" LastActivityDate="2018-02-22T06:38:37.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28166" PostTypeId="1" CreationDate="2018-02-22T06:54:39.697" Score="1" ViewCount="32" Body="&lt;p&gt;&lt;em&gt;Consider the following words taken from the lecture notes:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Hilbert-Schmidt Independence Criterion (HSIC) measures the dependence of the two random variables $X$ and $Y$. An empirical estimate of the HSIC is proportional to the $trace(KHLH)$, where&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$K$ is a kernel matrix on $X$,&lt;/li&gt;&#xA;&lt;li&gt;$L$ is a kernel on $Y$,&lt;/li&gt;&#xA;&lt;li&gt;$H$ is a centering matrix with $H_{ij} = δ(i, j) − \frac{1}{n}$.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;$HSIC(X, Y ) = 0$ if and only if $X$ and $Y$ are independent. The larger $HSIC(X, Y)$, the larger the dependence between $X$ and $Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;For me the only unclear part of this definition is the &quot;[K] kernel matrix on a distribution [X]&quot; sentence. Can someone explain what it means to apply the kernel on a distribution?&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="43888" LastActivityDate="2018-02-22T06:54:39.697" Title="How is Kernel Matrix on a distribution defined?" Tags="|feature-selection|feature-extraction|kernel|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28167" PostTypeId="2" ParentId="28157" CreationDate="2018-02-22T08:09:02.717" Score="2" Body="&lt;p&gt;You've kind of answered your own question there!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the strengths of machine learning with algorithms like Random Forests is that they are able to determine the feature (input) importance themselves. My advice would not be to weight them for now (not sure you actually can with SKLearn's RF) and let the model determine it itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is give additional weight to the rows (observations) of your data set, or give additional weight to different classes of the label (for example, if there aren't many 'P's in your label but they're important, you can bump up their weight relative to the other classes).&lt;/p&gt;&#xA;" OwnerUserId="38271" LastActivityDate="2018-02-22T08:09:02.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28168" PostTypeId="1" CreationDate="2018-02-22T08:19:29.943" Score="1" ViewCount="31" Body="&lt;p&gt;Are there any cloud computing services that allow for processing and storing data exclusively in Switzerland? Are there any that have machine-learning-specific functionality?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most do not seem to have this capability:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cloud.google.com/compute/docs/regions-zones/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cloud.google.com/compute/docs/regions-zones/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://aws.amazon.com/about-aws/global-infrastructure/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://aws.amazon.com/about-aws/global-infrastructure/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://azure.microsoft.com/en-us/regions/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://azure.microsoft.com/en-us/regions/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In some industries, such as finance, this is a strict requirement for regulatory and compliance reasons.&lt;/p&gt;&#xA;" OwnerUserId="39233" LastActivityDate="2018-02-22T08:19:29.943" Title="Cloud computing with country-specific region for Switzerland" Tags="|machine-learning|bigdata|cloud-computing|hardware|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28169" PostTypeId="2" ParentId="28022" CreationDate="2018-02-22T08:27:10.087" Score="0" Body="&lt;p&gt;If you don't know the patterns in advance, then it is going to be quite difficult to do the automated grouping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can, however, take the following approach&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Step - 1&lt;/strong&gt; -&lt;br&gt;&#xA; &lt;strong&gt;Define basic rules&lt;/strong&gt; (data-driven) based on which grouping can be done (like starts-with &lt;code&gt;JPM_&lt;/code&gt;, &lt;em&gt;ends-with&lt;/em&gt; a number in a &lt;em&gt;specific range&lt;/em&gt;, file-size, source, etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Step - 2&lt;/strong&gt; -&lt;br&gt;&#xA; &lt;strong&gt;Unless a direct match towards a group&lt;/strong&gt;, rules should direct towards a group but with a &lt;strong&gt;staging state&lt;/strong&gt; where a &lt;em&gt;manual verification&lt;/em&gt; can be done. If there is a direct match, directly apply &lt;strong&gt;step 4&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Step - 3&lt;/strong&gt; -&lt;br&gt;&#xA; After &lt;strong&gt;manual verification&lt;/strong&gt;, change their state to &lt;strong&gt;verified&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Step - 4&lt;/strong&gt; -&lt;br&gt;&#xA; &lt;strong&gt;Remember the inputs and the verified output&lt;/strong&gt; so that next-time when the same input comes, directly put them in the same verified group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, your manual verification will be your &lt;strong&gt;machine-learning&lt;/strong&gt; and there are two advantages&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In due time machine will be able to take decisions by itself.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Process is easy to debug, verify and port to another system.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46293" LastActivityDate="2018-02-22T08:27:10.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28170" PostTypeId="2" ParentId="28088" CreationDate="2018-02-22T08:33:58.940" Score="1" Body="&lt;p&gt;The problem is with datatype &lt;code&gt;integer64&lt;/code&gt;. I don't know why this is the problem, but change it to type &lt;code&gt;integer&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ratings = as.integer(ratings)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, I have not seen anyone using integer64 datatype.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-02-22T08:33:58.940" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28172" PostTypeId="2" ParentId="28022" CreationDate="2018-02-22T10:01:23.873" Score="0" Body="&lt;p&gt;Your problem here isn't in choosing an appropriate clustering algorithm, its defining an appropriate similarity metric. Edit distance and jaro winkler distance will cover a lot of ground for you, but you should still anticipate needing to do a fair amount of pre-processing and customization here. Also, as great as text-based metrics are, you have much more information here that can be leveraged. Your data is obviously going to have implicit clusters in it already based on documents that are contained in the same folders, and going further those folders exist in a hierarchy which also implies certain groupings. You should make sure that your clustering and/or similarity scoring incorporates these topological features in addition to any text similarity you do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your first step is going to be understanding your data more. I'd recommend constructing a tree visualization of the folder hierarchies you're going to be dealing with, taking a sample of 5-10 filenames from within each folder so you can better understand what your dealing with. From here, start trying to understand what kinds of naming conventions are in place that you can take advantage of. There are probably lots of files with dates at the beginning or end, maybe commonly occurring client names, words that are suggestive of document classifications like &quot;report&quot;, &quot;newsletter&quot;, &quot;resume&quot; etc. The more of these you can capture and deal with directly, the better. Next, you may start seeing some patterns that suggest ways you can further tokenize filenames. spaces, hyphens, and underscores are probably good places to start (after dealing with dates/timestamps, obviously), and CamelCasing would be worth looking out for as well. Also, different filetypes might have different naming conventions. For example, *.png files are probably more likely to have all numeric names starting with dates (i.e. someone dumped their camera to a folder).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to just get really quick and dirty with it, something you could try would be to parse each filename into n-grams (e.g. all sequential 3-letter sequences that occur in a filename) and then score pairwise filename similarity based on the jaccard distance of the n-grams that appear in each filename. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've figured out a couple of different approaches you want to try, you should start thinking about how to evaluate your results. Obviously you're going to start out evaluating things qualitatively, but that doesn't really help you compare the strengths/weaknesses of different approaches. One thing you could try would be to use the naming conventions learned by a particular method to try to predict whether or not randomly sampled filenames appear in the same folder or not, and score your methods based on how well the resulting classifiers perform.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ultimately, your going to have to custom tailor the solution to what you see in your clients filenames. Hopefully, I gave you a few ideas to work with here.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-22T10:01:23.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28173" PostTypeId="1" CreationDate="2018-02-22T10:09:44.167" Score="0" ViewCount="354" Body="&lt;p&gt;In boosting process we get a better accuracy for training data, but there is a lot of chance to over fit. For bagging ensemble method over fitting chance is lower than boosting. &#xA;Why we use boosting and for what condition ??&lt;/p&gt;&#xA;" OwnerUserId="46433" LastActivityDate="2018-02-22T21:21:33.177" Title="For what condition boosting work better than bagging in Ensemble Learner?" Tags="|machine-learning|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28174" PostTypeId="2" ParentId="28129" CreationDate="2018-02-22T10:10:17.467" Score="4" Body="&lt;p&gt;You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the &lt;a href=&quot;https://keras.io/models/sequential/#fit_generator&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;fit_generator&lt;/code&gt;&lt;/a&gt; method rather than &lt;code&gt;fit&lt;/code&gt; to pass in an iterator that feeds samples to your model from disk rather than loading all of that data at once. &lt;a href=&quot;https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a tutorial&lt;/a&gt; that discusses this more.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-22T10:10:17.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28175" PostTypeId="2" ParentId="26886" CreationDate="2018-02-22T10:16:15.273" Score="0" Body="&lt;p&gt;See if one of the answers in &lt;a href=&quot;https://stackoverflow.com/questions/31323499/sklearn-error-valueerror-input-contains-nan-infinity-or-a-value-too-large-for&quot;&gt;this discussion&lt;/a&gt; is helpful.&#xA;In my case, the error was caused by a &lt;code&gt;df = df.reindex(index=my_index)&lt;/code&gt;: The dataframe's index started at 1, but &lt;code&gt;my_index&lt;/code&gt; contained a 0, so pandas silently inserted a row full of NaNs...&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-22T10:16:15.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28176" PostTypeId="1" CreationDate="2018-02-22T10:16:38.630" Score="2" ViewCount="5549" Body="&lt;p&gt;I have a dataset which contains various columns: numerical and categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset &lt;a href=&quot;https://gist.github.com/anonymous/1eb36b1c319062b185c092589c76a94a&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was able to process the categorical data using &lt;code&gt;.astype('category')&lt;/code&gt; and &lt;code&gt;cat.codes&lt;/code&gt; features in Pandas dataframe as explained &lt;a href=&quot;http://pbpython.com/categorical-encoding.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; in Approach #2.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def process_categorical(self, dataset):&#xA;        &quot;&quot;&quot;&#xA;&#xA;        :param dataset:&#xA;        :return:&#xA;        &quot;&quot;&quot;&#xA;        # Label Encoding.&#xA;        for categorical_feature in LABEL_ENCODED_FEATURES:&#xA;            categorical_feature = _to_string(categorical_feature)&#xA;            dataset[categorical_feature] = dataset[categorical_feature].astype('category')&#xA;            dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes&#xA;        # Drop previous values.&#xA;        dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)&#xA;        # Rename back categories. {'to_user_cat': 'to_user' ... }&#xA;        dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)&#xA;        return dataset&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm able to train my model and test data.&#xA;When I generate my training dataset where categorical features were encoded. (Using &lt;code&gt;.cat.codes&lt;/code&gt;), this numerical data is different than my predictions. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataset['user_agent_numerical'].value_counts()&#xA;&#xA;10    1002&#xA;11     850&#xA;1      288&#xA;8       18&#xA;7       17&#xA;2       16&#xA;6       14&#xA;3        5&#xA;0        4&#xA;9        2&#xA;5        2&#xA;4        2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which corresponds to: &lt;code&gt;dataset.groupby(['user_agent']).size()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;user_agent&#xA;24gt2wreg24h                4&#xA;Avaya one-X Deskphone     288&#xA;Deskphone                  16&#xA;FreePBX 1.8                 5&#xA;HiDude UA v3.81             2&#xA;SimpleSIP V4.3              2&#xA;Twilio Gateway             14&#xA;caller                     17&#xA;eyeBeam release 3006o      18&#xA;friendly-scanner            2&#xA;pplsip                   1002&#xA;sipcli/v1.8               850&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port&#xA;sip:789011972592277524@13.57.9.131,789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to do pre-processing, My prediction data contains &lt;code&gt;pplsip&lt;/code&gt; as &lt;code&gt;user-agent&lt;/code&gt; hence it ends up with 0, instead of 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported &lt;a href=&quot;https://stackoverflow.com/questions/48901427/predict-with-categorical-features&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Complete code &lt;a href=&quot;https://gist.github.com/spicyramen/f8fa050db7f24b56ff10fa7295f44be4&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2019-03-22T16:03:15.797" Title="Predicting with categorical data" Tags="|machine-learning|python|feature-engineering|prediction|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28177" PostTypeId="1" AcceptedAnswerId="29555" CreationDate="2018-02-22T10:19:01.260" Score="11" ViewCount="22030" Body="&lt;p&gt;What happens when we do repartition on a PySpark dataframe based on the column. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataframe.repartition('id')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does this moves the data with the similar &lt;code&gt;'id'&lt;/code&gt; to the same partition? How does the &lt;code&gt;spark.sql.shuffle.partitions&lt;/code&gt; value affect the repartition?&lt;/p&gt;&#xA;" OwnerUserId="43240" LastActivityDate="2018-06-25T02:54:05.297" Title="PySpark dataframe repartition" Tags="|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28178" PostTypeId="2" ParentId="28176" CreationDate="2018-02-22T11:10:01.310" Score="6" Body="&lt;p&gt;Yep this is a common problem. What I would do is use SKLearns &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;label encoder&lt;/a&gt;. With a similar API to SKLearn models, it can be &lt;code&gt;fit&lt;/code&gt; to your category - meaning that any new data passed through the encoder object is encoded in the same fashion. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Import the encoder&#xA;from sklearn.preprocessing import LabelEncoder&#xA;&#xA;# Fit it to your training set + transform&#xA;encoder = LabelEncoder()&#xA;train[categorical_feature] = encoder.fit_transform(train[categorical_feature])&#xA;&#xA;# Use it on the test set to ensure the same transformation&#xA;test[categorical_feature] = encoder.transform(test[categorical_feature])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Huge caveat # 1&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This assumes every possible category is present in your training set. If you think this isn't the case, fit the encoder to a dataset that does have them all.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Huge caveat # 2&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Depending on the model you're using, categoricals still aren't 'processed' when they've been label-encoded , you probably still need to one hot encode (SKLearn has a package for that too).&lt;/p&gt;&#xA;" OwnerUserId="38271" LastEditorUserId="69822" LastEditDate="2019-03-22T16:03:15.797" LastActivityDate="2019-03-22T16:03:15.797" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28179" PostTypeId="1" CreationDate="2018-02-22T12:27:21.753" Score="0" ViewCount="85" Body="&lt;p&gt;I have a data.table that contains many timing variables.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Date: it gives the date of Sales&lt;/li&gt;&#xA;&lt;li&gt;Promo2(week, year): It describes the calendar week and the year when the store has started to participate in the Promo2 (reduction of prices).&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CompetitionOpenSince(Month, Year): Gives the approximate year and&#xA;the month of the moment when the nearest competitor was opened.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;PromoInterval: Describes the consecutive intervals Promo2 is launched,&#xA;naming the months when the promotion was launched again. For example.&#xA;&quot;Feb, May, Aug, Nov&quot; means that each round starts in February, May,&#xA;August, November of a given year for this store.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The Goal is to predict sales for next few week. I have other factor and binary variables such as the type of store(a b,c..), the level of Assortment, CompetitionDistance, number of Customers...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my work I suggest some variable that uses timing variables above:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Compute the number of days between the day of sales and the first&lt;br&gt;&#xA;day when the store started the promo2(Day-Promo2).&lt;/li&gt;&#xA;&lt;li&gt;Compute the number of days between the day of Promo2 and the day&#xA;   when the competitor opened the promo2(Promo2-CompetitionOpenSince).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Dont surprised when I wrote (Day-Promo2) and (Promo2-CompetitionOpenSince) because I transform Promo2 and CompetitionOpenSince to date type using some hypothesis to simplify the work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can you suggest more than that? I find it really hard to create new other useful explaining variables mostly the  &lt;strong&gt;PromoInterval&lt;/strong&gt; variable!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your suggestions!&lt;/p&gt;&#xA;" OwnerUserId="46590" LastActivityDate="2023-12-30T02:00:50.103" Title="Need recommandations of using timing variables for forecasting sales" Tags="|machine-learning|forecasting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28180" PostTypeId="2" ParentId="28179" CreationDate="2018-02-22T13:41:31.223" Score="0" Body="&lt;p&gt;Generally it works well to include some lagged variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sales last week&lt;/li&gt;&#xA;&lt;li&gt;Sales a month ago&lt;/li&gt;&#xA;&lt;li&gt;Inventory a month ago&lt;/li&gt;&#xA;&lt;li&gt;etc&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="800" LastEditorUserId="29575" LastEditDate="2018-02-22T13:46:14.793" LastActivityDate="2018-02-22T13:46:14.793" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28181" PostTypeId="2" ParentId="28161" CreationDate="2018-02-22T14:01:01.140" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://bokeh.pydata.org/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bokeh&lt;/a&gt; and &lt;a href=&quot;https://plot.ly/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Plotly&lt;/a&gt; both feature interactive visualizations and can be used in a Jupyter notebook. &lt;/p&gt;&#xA;" OwnerUserId="27015" LastEditorUserId="27015" LastEditDate="2018-02-22T14:13:05.923" LastActivityDate="2018-02-22T14:13:05.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28182" PostTypeId="1" CreationDate="2018-02-22T14:11:14.620" Score="2" ViewCount="231" Body="&lt;p&gt;I have a fairly simple LSTM models that achieves ok results.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/e05or.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/e05or.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to add auxiliary data as appeared in the &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras docs&lt;/a&gt;. I added only one binary variable. I expected this model to be at least as good, assuming that if it has no information its weights will go to zero. In reality, the model fails and just predicts zeros for my entire test set.&#xA;The second model:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/EL2Ya.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EL2Ya.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;More specifically, I'm predicting a time series of human-behaviour, and my binary feature tags whether the time point is in a weekend. I'd expect it capture some of the weekend (black background) reduction, and if not, at least not to ruin the model completely.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/FG2NW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FG2NW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9910" LastEditorUserId="9910" LastEditDate="2018-02-22T14:52:57.977" LastActivityDate="2018-02-22T14:52:57.977" Title="LSTM with auxiliary data achieves abismal results" Tags="|keras|time-series|lstm|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-26T00:52:50.730" ContentLicense="CC BY-SA 3.0" />
  <row Id="28183" PostTypeId="2" ParentId="28182" CreationDate="2018-02-22T14:39:21.720" Score="0" Body="&lt;p&gt;Is there a reason why this new information &lt;strong&gt;&lt;em&gt;has&lt;/em&gt;&lt;/strong&gt; to be a part of your model? There's no rule that says that all data must be fed into your model. Maybe you have just found a feature that shouldn't be included in your predictive phases? And if so, that's perfectly fine, there is nothing wrong with that. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-22T14:39:21.720" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28184" PostTypeId="2" ParentId="28112" CreationDate="2018-02-22T14:40:48.213" Score="2" Body="&lt;p&gt;I have a feeling that you are never going to come across this, most of this data is closely guarded. It's no secret that dating sites like OK Cupid use data science on their app, so I think that most of these sites aren't quick to make their data public. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-22T14:40:48.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28186" PostTypeId="2" ParentId="28161" CreationDate="2018-02-22T14:52:56.300" Score="1" Body="&lt;p&gt;Simply use &lt;code&gt;%matplotlib notebook&lt;/code&gt; the same way you would use &lt;code&gt;%matplotlib inline&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.reddit.com/r/IPython/comments/36p360/try_matplotlib_notebook_for_interactive_plots/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.reddit.com/r/IPython/comments/36p360/try_matplotlib_notebook_for_interactive_plots/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-02-22T14:52:56.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28187" PostTypeId="1" AcceptedAnswerId="28197" CreationDate="2018-02-22T14:57:37.907" Score="0" ViewCount="533" Body="&lt;p&gt;I'm trying to train a CNN for MNIST, everything goes well except for the loss stays very high in my model, while it's very low in example(which has different structure).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;this model still yields high accuracy though there is high loss.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I attached my code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with tf.name_scope(&quot;inputs&quot;):&#xA;    X = inputs = tf.placeholder(tf.float32, &#xA;                                shape=[None, 28, 28, 1], name=&quot;X&quot;)&#xA;    y = tf.placeholder(&#xA;                    tf.int32,&#xA;                    name=&quot;y&quot;)&#xA;&#xA;    training = tf.placeholder_with_default(False, [], name=&quot;training&quot;)&#xA;&#xA;conv1 = tf.layers.conv2d(&#xA;                        inputs,&#xA;                        filters=6,&#xA;                        kernel_size=3,&#xA;                        strides=(1, 1),&#xA;                        padding='SAME',&#xA;                        activation=tf.nn.selu,&#xA;                        name=&quot;conv1&quot;,&#xA;                    )&#xA;&#xA;conv2 = tf.layers.conv2d(&#xA;                        conv1,&#xA;                        filters=12,&#xA;                        kernel_size=3,&#xA;                        strides=(1, 1),&#xA;                        padding='SAME',&#xA;                        activation=tf.nn.selu,&#xA;                        name=&quot;conv2&quot;,&#xA;                    )&#xA;&#xA;max_pool3 = tf.layers.max_pooling2d(&#xA;                            conv2,&#xA;                            pool_size=8,&#xA;                            strides=2,&#xA;                            padding='SAME',&#xA;                            name=&quot;max_pool3&quot;&#xA;                        )&#xA;with tf.name_scope(&quot;conv4&quot;):&#xA;    conv4 = tf.layers.conv2d(&#xA;                            max_pool3,&#xA;                            filters=12,&#xA;                            kernel_size=3,&#xA;                            strides=(1, 1),&#xA;                            padding='SAME',&#xA;                            activation=tf.nn.selu,&#xA;                            name=&quot;conv4&quot;,&#xA;                        )&#xA;&#xA;    num_ele = int(conv4.shape[1]*conv4.shape[2]*conv4.shape[3])&#xA;    conv4_flat = tf.reshape(&#xA;                        conv4,&#xA;                        shape=[-1, num_ele],&#xA;                        name=&quot;conv4_flat&quot;&#xA;                      )&#xA;    conv4_flat_dropout = tf.layers.dropout(conv4_flat, rate=dropout_rate,&#xA;                                        training=training,&#xA;                                        name=&quot;conv4_flat_dropout&quot;)&#xA;&#xA;with tf.name_scope(&quot;fc5&quot;):&#xA;    fc5 = tf.layers.dense(&#xA;                            conv4_flat_dropout,&#xA;                            conv4_flat_dropout.shape[1]//2,&#xA;                            activation=tf.nn.selu,&#xA;                            name=&quot;fc5&quot;,&#xA;                        )&#xA;    fc5_dropout = tf.layers.dropout(fc5, rate=dropout_rate,&#xA;                                   training=training,&#xA;                                   name=&quot;fc5_dropout&quot;)&#xA;&#xA;logits = tf.layers.dense(&#xA;                        fc5_dropout,&#xA;                        n_outputs,&#xA;                        name=&quot;logits&quot;,&#xA;                    )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the training process&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# It starts with very low accuracy, but instead in the sample after the first epoch the accuracy for training sets reaches 1.&#xA; 0 train loss:1.8484&#xA;    train acc:0.751745&#xA;                     validation loss:1.7019&#xA;                      validation acc:0.7656&#xA; 1 train loss:0.0745&#xA;    train acc:0.978927&#xA;                     validation loss:0.0782&#xA;                      validation acc:0.9764&#xA; 2 train loss:0.0958&#xA;    train acc:0.972818&#xA;                     validation loss:0.1072&#xA;                      validation acc:0.9706&#xA; 3 train loss:0.1186&#xA;    train acc:0.971727&#xA;                     validation loss:0.1292&#xA;                      validation acc:0.9714&#xA; 4 train loss:0.1397&#xA;    train acc:0.969836&#xA;                     validation loss:0.1422&#xA;                     validation acc:0.9738&#xA;# Accuracy for some reason always drops dramaticlt here. Which I don't understand why.&#xA; 5 train loss:0.8394&#xA;    train acc:0.939564&#xA;                     validation loss:0.8237&#xA;                      validation acc:0.9470&#xA; 6 train loss:0.3108&#xA;    train acc:0.979182&#xA;                     validation loss:0.3345&#xA;                      validation acc:0.9786&#xA; 7 train loss:0.6576&#xA;    train acc:0.967382&#xA;                     validation loss:0.8300&#xA;                      validation acc:0.9652&#xA; 8 train loss:0.2005&#xA;    train acc:0.987273&#xA;                     validation loss:0.3021&#xA;                      validation acc:0.9832&#xA; 9 train loss:0.2915&#xA;    train acc:0.984145&#xA;                     validation loss:0.4509&#xA;                      validation acc:0.9812&#xA;10 train loss:0.7932&#xA;    train acc:0.968273&#xA;                     validation loss:1.1119&#xA;                      validation acc:0.9634&#xA;11 train loss:0.2778&#xA;    train acc:0.988636&#xA;                     validation loss:0.4988&#xA;                      validation acc:0.9848&#xA;12 train loss:0.4892&#xA;    train acc:0.982982&#xA;                     validation loss:0.6407&#xA;                      validation acc:0.9826&#xA;13 train loss:0.5457&#xA;    train acc:0.983382&#xA;                     validation loss:0.9361&#xA;                      validation acc:0.9806&#xA;14 train loss:0.3998&#xA;    train acc:0.989527&#xA;                     validation loss:0.7423&#xA;                      validation acc:0.9876&#xA;15 train loss:0.3925&#xA;    train acc:0.985745&#xA;                     validation loss:0.7599&#xA;                      validation acc:0.9788&#xA;16 train loss:0.2093&#xA;    train acc:0.993236&#xA;                     validation loss:0.5771&#xA;                      validation acc:0.9850&#xA;17 train loss:0.5663&#xA;    train acc:0.989855&#xA;                     validation loss:1.2298&#xA;                      validation acc:0.9846&#xA;18 train loss:0.6623&#xA;    train acc:0.988927&#xA;                     validation loss:1.3572&#xA;                      validation acc:0.9824&#xA;19 train loss:0.1555&#xA;    train acc:0.994891&#xA;                     validation loss:0.6606&#xA;                      validation acc:0.9872&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the sample code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with tf.name_scope(&quot;inputs&quot;):&#xA;    X = tf.placeholder(tf.float32, shape=[None, n_inputs], name=&quot;X&quot;)&#xA;    X_reshaped = tf.reshape(X, shape=[-1, height, width, channels])&#xA;    y = tf.placeholder(tf.int32, shape=[None], name=&quot;y&quot;)&#xA;    training = tf.placeholder_with_default(False, shape=[], name='training')&#xA;&#xA;conv1 = tf.layers.conv2d(X_reshaped, filters=conv1_fmaps, kernel_size=conv1_ksize,&#xA;                     strides=conv1_stride, padding=conv1_pad,&#xA;                     activation=tf.nn.relu, name=&quot;conv1&quot;)&#xA;conv2 = tf.layers.conv2d(conv1, filters=conv2_fmaps, kernel_size=conv2_ksize,&#xA;                     strides=conv2_stride, padding=conv2_pad,&#xA;                     activation=tf.nn.relu, name=&quot;conv2&quot;)&#xA;&#xA;with tf.name_scope(&quot;pool3&quot;):&#xA;    pool3 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding=&quot;VALID&quot;)&#xA;    pool3_flat = tf.reshape(pool3, shape=[-1, pool3_fmaps * 14 * 14])&#xA;    pool3_flat_drop = tf.layers.dropout(pool3_flat, conv2_dropout_rate, training=training)&#xA;&#xA;with tf.name_scope(&quot;fc1&quot;):&#xA;    fc1 = tf.layers.dense(pool3_flat_drop, n_fc1, activation=tf.nn.relu, name=&quot;fc1&quot;)&#xA;    fc1_drop = tf.layers.dropout(fc1, fc1_dropout_rate, training=training)&#xA;&#xA;with tf.name_scope(&quot;output&quot;):&#xA;    logits = tf.layers.dense(fc1, n_outputs, name=&quot;output&quot;)&#xA;    Y_proba = tf.nn.softmax(logits, name=&quot;Y_proba&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I found that most of the classifications are classified with very high probability(like 1). Both those correct ones and false ones.&#xA;&lt;br&gt;&#xA;&#xA;(softmax probablity)&#xA;[0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0.000000e+00 1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00&#xA; 0.000000e+00 0.000000e+00 0.000000e+00 6.966008e-33 0.000000e+00]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]&#xA;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5TFcn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5TFcn.png&quot; alt=&quot;![Image of classes]()&quot;&gt;&lt;/a&gt;&#xA;The probability are corresponidng to each of the images&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is&#xA;&lt;li&gt;Why will this happen what exactly caused this, was it a bug in my code or it's natural for this to happen?&#xA;&lt;li&gt;If this has to do with the architecture, what potential problem might cause it and how can I fix this problem and avoid this in the future?&#xA;&lt;li&gt;Why is the accuracy after first epoch so low and why the accuracy drops a lot after 3 to 6 epoches?&#xA;&lt;li&gt;Was the styling of my code acceptable, and what should I improve?&#xA;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Original codes are here&#xA;&lt;br&gt; &#xA;&lt;a href=&quot;https://github.com/Dovermore/handson-ml/blob/master/chapter_13_exer/question_7.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Dovermore/handson-ml/blob/master/chapter_13_exer/question_7.ipynb&lt;/a&gt;&#xA;&lt;a href=&quot;https://github.com/Dovermore/handson-ml/blob/master/13_convolutional_neural_networks.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Dovermore/handson-ml/blob/master/13_convolutional_neural_networks.ipynb&lt;/a&gt;&#xA;(go to the bottom to find the related segment in the second link)&lt;/p&gt;&#xA;" OwnerUserId="46703" LastEditorUserId="46703" LastEditDate="2018-02-23T01:36:00.530" LastActivityDate="2018-02-23T01:36:00.530" Title="What is the reason that CNN classify some images horribly wrong" Tags="|tensorflow|cnn|mnist|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28188" PostTypeId="1" CreationDate="2018-02-22T15:09:40.490" Score="3" ViewCount="340" Body="&lt;p&gt;I have population data from Census.gov:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total US population by age by year from 1940 through 2010&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the range of decades, the data is missing discrete population values for ages greater than a certain age.  Instead an aggregate amount is provided that represents all ages greater than the cutoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically it follows this pattern:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;1940 to 1979:  Discrete data from 0 to 84 and aggregate for ages 85&#xA;and greater &lt;/li&gt;&#xA;&lt;li&gt;1980 to 1999:  Discrete data from 0 to 99 and aggregate for ages 100 and greater &lt;/li&gt;&#xA;&lt;li&gt;2000 to 2010:  Discrete data from 0 to 84 and aggregate for ages 85 and greater&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The desired outcome is to have discrete data points for each age and year from 0-99 and then an aggregated lump sum figure for ages 100 and greater.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I want to input the missing discrete population values for ages 85 to 100 for years 1940 through 1979 and years 2000 through 2010.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I want to use the actual discrete population values for ages 85 to 100 for years 1980 through 1989 to achieve that outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some Observations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The pattern of missing values is MNAR (Missing Not At Random) - these were systematically omitted but the aggregate value representing the missing detail is provided&lt;/li&gt;&#xA;&lt;li&gt;Population data for this time frame is deterministic:  Population levels rise linearly each year;  The duty cycle of a human body is finite and constraints and limits are well known.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Looking at the data we can see that each of the three subsets of years have very similar patterns.  More variation in younger ages and variation flattens out for ages greater than 60&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8fQag.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8fQag.png&quot; alt=&quot;All available population values ages 0-99 for years 1940 through 2010&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then if we focus on the years 1980 through 1989 we can fit a nice curve for ages 0 through 100 with a Multiple-R-Squared of .979.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cBJu4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cBJu4.png&quot; alt=&quot;Average of age per each year and fitted curve of 4th order&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then if we narrow the focus to ages 60 to 100 and even narrower to ages 85 to 100 the Multiple-R-Squared increases to .9996.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TcVfm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TcVfm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cu2nK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cu2nK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we flip our focus and look at the increasing levels of population we can observe that these relationships are linear.  Population rises at a steady rate year over year.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Total population 1940 through 2010:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7jSvA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7jSvA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ages 85 through 100 for years 1980 through 1999.  Each age is linear.  Each age has a slightly lower rate of increase (lesser slope).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/C9qR3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/C9qR3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;My question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is where I could use some guidance to move forward:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When imputing discrete missing population values by age and year, how do I combine the fitted curve that models changes in population when age increases with the linear regression that models changes in population year over year?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does one or more documented methods naturally apply to the problem as I have described it?&#xA;For example:  KNN, PCA, BPCA, Mean, MICE, other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there is recommended method can you point me to available R or Python packages and documentation that describes the mechanics of applying a given approach?&lt;/p&gt;&#xA;" OwnerUserId="14376" LastEditorUserId="13023" LastEditDate="2018-12-21T20:12:37.490" LastActivityDate="2024-02-05T00:02:25.367" Title="Missing population values in census data" Tags="|r|missing-data|data-imputation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28189" PostTypeId="2" ParentId="26511" CreationDate="2018-02-22T15:27:55.503" Score="1" Body="&lt;p&gt;Interesting question. I think that you can illustrate this by thinking about different use cases. The one example I've heard that I like is around lending decisions for loan applications. That's an algorithm but, because of regulations, it can't be strictly &quot;black box&quot;. The decision has to be, effectively, interpretable because the bank has to give you a reason for decline on the loan. So, there's certainly better algos out there for loans that can give a binary result, but do you want a bank to just tell you yes or no? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-22T15:27:55.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28190" PostTypeId="2" ParentId="9702" CreationDate="2018-02-22T15:28:45.723" Score="0" Body="&lt;p&gt;Instead of exporting the weights, you can export the model to a pickle file and use a &lt;a href=&quot;https://www.xlwings.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;xlwings&lt;/a&gt; to read the data from the spreadsheet, load the pickled model and run a prediction &lt;a href=&quot;https://stackoverflow.com/questions/3567365/calling-python-script-from-excel-vba&quot;&gt;Here's a similar questions&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="10076" LastActivityDate="2018-02-22T15:28:45.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28191" PostTypeId="1" CreationDate="2018-02-22T15:34:03.793" Score="1" ViewCount="373" Body="&lt;p&gt;To design a neural network based predictor for &lt;code&gt;f(x) = sin(x)&lt;/code&gt; I designed the following network. &lt;code&gt;Input ----&amp;gt; Dense(num_neurons=1) ---&amp;gt; Relu() ----&amp;gt; Dense(num_neurons=100) ---&amp;gt; Relu() ---&amp;gt; Dense(num_neurons=1) ----&amp;gt; Relu()--&amp;gt; Output&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on &lt;code&gt;X_train = numpy.arange(0.0, 314.1, 0.1)&lt;/code&gt; and &lt;code&gt;Y_train = numpy.sin(X_train)&lt;/code&gt;. It is subsequently tested on &lt;code&gt;X_test= numpy.arange(-10.0, 10.0, 0.001)&lt;/code&gt; and &lt;code&gt;Y_test = numpy.sin(X_test)&lt;/code&gt; The predictor however performs badly on the test data. What could have gone wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training data is too small to train the network. Adding more data especially when X &amp;lt; 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L2 loss is not a good metric to measure loss when regressing for sin(x)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This type of neural network architecture can fundamentally not learn a function like sin(x).&lt;/p&gt;&#xA;" OwnerUserId="39331" LastEditorUserId="11097" LastEditDate="2018-04-10T11:29:26.650" LastActivityDate="2018-04-10T11:29:26.650" Title="Error in Neural Network" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28192" PostTypeId="1" CreationDate="2018-02-22T16:32:57.690" Score="4" ViewCount="9589" Body="&lt;p&gt;I'm new in machine learning and I am willing to know better what is the difference between biased and unbiased learners?&#xA;Anyone can provide some examples?&lt;/p&gt;&#xA;" OwnerUserId="46640" LastEditorUserId="28175" LastEditDate="2018-02-22T18:07:13.127" LastActivityDate="2023-03-24T15:21:54.397" Title="What are the differences between biased and unbiased learners?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28193" PostTypeId="2" ParentId="28188" CreationDate="2018-02-22T16:41:39.190" Score="0" Body="&lt;p&gt;I think you need to be wary of using curves to extrapolate beyond the age thresholds - specifically I think you should consider:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;mortality increases with age; I would imagine it increases at an increasing rate with age (especially at higher ages). Would you be able to capture this effectively?&lt;/li&gt;&#xA;&lt;li&gt;there is obviously some sort of overlap between populations in subsequent years (e.g. someone who is included in 1985 data could also be in 1986 data). What effect would this have on the data imputation?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would suggest a different approach. Actuaries have traditionally produced &quot;life tables&quot; which capture mortality in population cohorts. If you can find a set of tables which is applicable to the period in question, you could use these to calculate population numbers.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-22T16:41:39.190" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28194" PostTypeId="1" CreationDate="2018-02-22T16:49:23.590" Score="2" ViewCount="209" Body="&lt;p&gt;I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was wondering why more advanced methods don't seemed to be used, such as SQP algorithms or interior-point methods. Is it because training a neutral net is always a simple unconstrained optimization problem, and the above-mentioned methods would be unnecessary? Any insight would be great, thanks.&lt;/p&gt;&#xA;" OwnerUserId="46712" LastActivityDate="2021-09-21T16:01:40.137" Title="Optimization methods used in machine learning" Tags="|machine-learning|neural-network|training|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28196" PostTypeId="1" CreationDate="2018-02-22T16:56:09.293" Score="1" ViewCount="196" Body="&lt;p&gt;In have two classes in my data, and one of my features with Gini importance 0.023 (8th highest out of 95 features) shows a p-value of 0.44 when doing a t-test on the values from the two classes. I'm wondering if this makes sense. Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: If it's helpful, all of the Gini importances range between 0.0011 to 0.0295, and the p-values ranged from 5.13e-109 to 1. These p-values were seen for both t-tests and rank-sum tests. &lt;/p&gt;&#xA;" OwnerUserId="33057" LastEditorUserId="29781" LastEditDate="2018-03-15T16:27:52.027" LastActivityDate="2018-05-14T19:04:59.063" Title="In random forest, can features with high Gini importance show insignificant p-values in t-tests?" Tags="|machine-learning|feature-selection|random-forest|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28197" PostTypeId="2" ParentId="28187" CreationDate="2018-02-22T17:28:11.093" Score="2" Body="&lt;p&gt;Always remember classification is computational process with group of numbers in matrices and when you train a model for particular set it goes through the data and build a matrices that is called as trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you give a data set to classify, It tries to associate the values of test data with trained models and it just classifies mathematically,If any labels in the model comes close to that of test data it tries to associate that test data to the highest accurate that it identifies not true output but predicted mathematically&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes they may fall into into the wrong label based on the accuracy of your model and when your training a data like MNIST,sometimes it does classify to wrong labels when parameters are not tuned as such its accuracy is low&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps&lt;/p&gt;&#xA;" OwnerUserId="43438" LastActivityDate="2018-02-22T17:28:11.093" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28198" PostTypeId="1" AcceptedAnswerId="28212" CreationDate="2018-02-22T17:31:42.300" Score="1" ViewCount="391" Body="&lt;p&gt;I'm building a testing project to get an introduction to DS &amp;amp; ML. &#xA;As a person part of the working force, sometimes finding a job is harder than it should be. I thought I could built a testing project to help workers find a job that best match their interests and their skills.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use a classifier, but I do not know if regression if the best way to approach this as a first pass. I was thinking of a GA system that could find some way to find way to approach this over time in a X generation. Maybe this isn't the best way for recommending jobs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for isn't the code for the problem but more of algorithms ideas I should take a look at and implement in any given programming language. I'm looking to implement a system that can take a peak at job descriptions, interests and skills of an individual. I think this doesn't sound too crazy to be able to train an agent and then make it look at every jobs there is on LinkedIn for instance and take a look at 'Software Engineer' and give me 45% matching and here's the three skills that you have to work on or 98% match, please apply.&lt;/p&gt;&#xA;" OwnerUserId="39542" LastEditorUserId="23240" LastEditDate="2019-03-16T10:32:37.163" LastActivityDate="2019-03-16T10:32:37.163" Title="Which algorithm should be used for an accurate job recommendation system" Tags="|machine-learning|recommender-system|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="28199" PostTypeId="2" ParentId="28192" CreationDate="2018-02-22T18:06:10.700" Score="3" Body="&lt;p&gt;Biased in the context that you are speaking means, that your model overfits the training data and can not generalize well. It means your model performs very well on your training data, but can not do well on cross-validation and test data. It is customary to say that biased learners memorize the training data which is really true. Biased learners don't &lt;em&gt;learn&lt;/em&gt; the data, they &lt;em&gt;fit&lt;/em&gt; the data. For understanding the other usages of &lt;em&gt;bias&lt;/em&gt; take a look at this &lt;a href=&quot;https://datascience.stackexchange.com/q/18123/28175&quot;&gt;question&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is something that may be worth mentioning. You may have heard people saying that your model has a high-bias problem. It just means that your model can not learn the training data, whilst the biased learners overfits the training data, means fits the training data. The latter can not generalize well because it has fitted the training data, memorized it, the former can not generalize because it has not learnt even the training data so it has not learnt so much and can not generalize.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="-1" LastEditDate="2019-01-12T17:30:01.080" LastActivityDate="2019-01-12T17:30:01.080" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28200" PostTypeId="1" CreationDate="2018-02-22T18:10:43.190" Score="5" ViewCount="5534" Body="&lt;p&gt;I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?&lt;/p&gt;&#xA;" OwnerUserId="9802" LastEditorUserId="9802" LastEditDate="2018-02-22T18:58:35.120" LastActivityDate="2019-03-15T16:26:58.537" Title="When should you balance a time series dataset?" Tags="|machine-learning|classification|time-series|class-imbalance|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28201" PostTypeId="2" ParentId="26511" CreationDate="2018-02-22T18:33:21.103" Score="1" Body="&lt;p&gt;Another question you can ask yourself is whether you have a business goal to understand a system in such a way that gives you information about the inputs and their relationship and how changes to those inputs impact your outcome (prediction). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recent example of a problem I worked on that falls into this case is predicting the number of market leads by month (week, day) using spend by channel (TV, radio, digital). Here the goal wasn't just to predict how many leads would be generated given spend but also to have a framework to use to optimize lead generation around spend distribution (i.e., what is the most cost effective distribution of spend across TV, radio, and digital to generate the largest number of leads). Because of this business requirement, a neural network or SVM would not have met our goals because, while they would have provided a prediction of lead generation, they would not have provided the understanding of the inputs (spend by channel).&lt;/p&gt;&#xA;" OwnerUserId="46546" LastEditorUserId="29575" LastEditDate="2018-02-22T19:05:10.533" LastActivityDate="2018-02-22T19:05:10.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28202" PostTypeId="1" CreationDate="2018-02-22T19:04:49.520" Score="1" ViewCount="55" Body="&lt;p&gt;Let's pretend we have a series of events, which have binary results $ y_1, \ldots, y_n $.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For every event we have a number of people with more or less stable features $ f_1, \ldots, f_k $ (like sex for given client for every event) and a volatile feature $ g $ (for instance, the amount of spent money for the corresponding event).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We want to predict the target $ y_t $ using this data. Empirically we found that values like $$ \sum \{ g | f_1 &amp;gt; c \} $$ have some predictive power and want to generalize them (in both ways, using different conditions and different aggregations).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of machine learning algorithm can be useful for this setting?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, essentially, that we want to find a cluster based on features $ f_1, \ldots, f_k $ that have predictive power in terms of feature $ g $.&lt;/p&gt;&#xA;" OwnerUserId="46691" LastEditorUserId="924" LastEditDate="2018-03-02T07:58:42.147" LastActivityDate="2018-03-02T07:58:42.147" Title="Classify people, predict event" Tags="|machine-learning|classification|time-series|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28203" PostTypeId="2" ParentId="5885" CreationDate="2018-02-22T19:27:53.453" Score="1" Body="&lt;p&gt;Before you do that, you may want to check for outliers. Say 99% of the data lie in range (-5, 5), but one little guy takes a value of 25.0. Your normalized array would  cluster around (0, 0.3), and that would cause problem for the neural net to learn.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastActivityDate="2018-02-22T19:27:53.453" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28206" PostTypeId="2" ParentId="28156" CreationDate="2018-02-22T20:23:44.740" Score="0" Body="&lt;p&gt;From &lt;code&gt;ValueError: Unknown label type: 'unknown' on y_train&lt;/code&gt; I guess you have some unsupported/invalid data type in your &lt;code&gt;y_train&lt;/code&gt; or the type is indeed valid but does not match with the &lt;code&gt;RandomForestClassifier&lt;/code&gt; expectation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please do the followings:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;print/plot the &lt;code&gt;y_train&lt;/code&gt; and make sure all values make sense. There is a fair chance that &lt;code&gt;HashingVectorizer&lt;/code&gt; could not hash some special cases (eg foreign language characters or missing values).&lt;/li&gt;&#xA;&lt;li&gt;use &lt;code&gt;type&lt;/code&gt; or &lt;code&gt;dtype&lt;/code&gt; to find out the &lt;code&gt;v_train&lt;/code&gt; data type. Make sure it is consistent with &lt;code&gt;RandomForestClassifier&lt;/code&gt; &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;expectation&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;p.s. as @outlier mentioned, we usually do not transform our labels (&lt;code&gt;y_train&lt;/code&gt; in your case). Many classifiers can handle strings/letters as class labels. In case they cannot, we can use a simple mapping such as P -&gt; 0, Q -&gt; 1, etc. The following does the job for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import LabelEncoder&#xA;encoder = LabelEncoder()&#xA;encoder.fit(y)&#xA;encoded_y = encoder.transform(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46505" LastEditorUserId="46505" LastEditDate="2018-02-25T06:51:48.867" LastActivityDate="2018-02-25T06:51:48.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28207" PostTypeId="2" ParentId="28173" CreationDate="2018-02-22T21:21:33.177" Score="1" Body="&lt;p&gt;Bagging (and features sampling) aim to reduce variance by providing low-correlated trees. Estimators can then be aggregated together to reduce variance. Reason is simple decision trees tend to quickly overfit in the bottom nodes. Bagging and features sampling only adress high-variance (overfitting) problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, boosting is a meta-algorithm that aims to reduce both bias and variance by training estimators in a sequential way. Each predictor tries to reduce residuals from previous estimators.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, the two meta-algorithms can't be easily compared that way. They will give you different results depending of data distributions, potential outliers, usecase... Potentially, boosting is more sensitive to outliers than bagging for example. I think trying the 2 methods is your best chance and you don't have to restrict to only one model.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-02-22T21:21:33.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28210" PostTypeId="1" AcceptedAnswerId="28211" CreationDate="2018-02-22T21:32:37.800" Score="31" ViewCount="52304" Body="&lt;p&gt;Can someone please post a straightforward example of Keras using a callback to save a model after every epoch? I can find examples of saving weights, but I want to be able to save a completely functioning model after every training epoch. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2023-01-24T12:27:37.847" Title="Keras Callback example for saving a model after every epoch?" Tags="|python|keras|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28211" PostTypeId="2" ParentId="28210" CreationDate="2018-02-22T22:06:29.840" Score="30" Body="&lt;p&gt;Setting 'save_weights_only' to False in the Keras &lt;a href=&quot;https://keras.io/callbacks/&quot; rel=&quot;noreferrer&quot;&gt;callback&lt;/a&gt; 'ModelCheckpoint' will save the full model; this example taken from the link above will save a full model every epoch, regardless of performance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some more examples are found &lt;a href=&quot;https://machinelearningmastery.com/check-point-deep-learning-models-keras/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;, including saving only improved models and loading the saved models. &lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2018-02-22T22:06:29.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28212" PostTypeId="2" ParentId="28198" CreationDate="2018-02-22T22:44:51.013" Score="2" Body="&lt;p&gt;You are much, much too early in your process to even begin thinking about your models. At this stage, you should be thinking about what the data looks like and how much of it do you need. For starters, what are you measuring? What is the answer that you seek? Is it job satisfaction? Likelihood to land an interview? Likelihood to get a job offer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have that, how are you quantifying that? Is it a categorical variable? A continuous variable? Then you would have to decide on the likely list of factors that go into that answer and determine how you're going to get that data. Do you need to run a custom survey? Are there other datasets that you can leverage? You need to think about all of these answers before you're ready for any sort of modeling discussion. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-22T22:44:51.013" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28213" PostTypeId="1" CreationDate="2018-02-22T23:08:01.743" Score="5" ViewCount="4316" Body="&lt;p&gt;I am trying to implement negative sampling in Keras. I wrote the following code that just compute the loss and I plan to add an additional output for the logits once I get it up and running. Here is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Dense&#xA;from keras.layers import Embedding&#xA;from keras.layers import Input, Concatenate, Masking, Layer&#xA;from keras.layers import LSTM&#xA;from keras.models import Model&#xA;import numpy as np&#xA;from keras.utils import to_categorical&#xA;import tensorflow as tf&#xA;&#xA;&#xA;vocab_size = 20&#xA;n_features = 3&#xA;batch_size = 16&#xA;sequence_len = 10&#xA;embedding_size = 10&#xA;&#xA;# Inputs&#xA;looked = Input((sequence_len,), dtype='int32', name='in_seq')&#xA;features = Input((sequence_len, n_features,), dtype='float', name='in_aux')&#xA;labels = Input((1,), dtype='int32', name='labels_')&#xA;&#xA;looked_m = Masking(mask_value=0)(looked)&#xA;in_embed = Embedding(output_dim=embedding_size, input_dim=vocab_size, input_length=sequence_len)(looked_m)&#xA;&#xA;in_merged = Concatenate()([in_embed, features])&#xA;&#xA;la_lstm = LSTM(256)(in_merged)&#xA;la_dense1 = Dense(128)(la_lstm)&#xA;&#xA;loss = MyLayer(10, vocab_size, mode='train', name='my_layer')([la_dense1, labels])&#xA;&#xA;model = Model(inputs=[looked, features, labels], outputs=[loss])&#xA;model.compile(loss=lambda loss, y_true: loss, optimizer='Adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is my custom layer&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class MyLayer(Layer):&#xA;    def __init__(self, num_sampled, num_classes, mode, **kwargs):&#xA;        self.num_sampled = num_sampled&#xA;        self.num_classes = num_classes&#xA;        self.mode = mode&#xA;        super(MyLayer, self).__init__(**kwargs)&#xA;&#xA;    def build(self, input_shape):&#xA;        dense_shape, classes_shape = input_shape&#xA;        self.kernel = self.add_weight(name='kernel',&#xA;                                      shape=(self.num_classes, dense_shape[1]),&#xA;                                      initializer='uniform',&#xA;                                      trainable=True)&#xA;        self.bias = self.add_weight(name='bias',&#xA;                                      shape=(self.num_classes,),&#xA;                                      initializer='uniform',&#xA;                                      trainable=True)  # Maybe zero&#xA;&#xA;        super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!&#xA;&#xA;    def call(self, inputs_and_labels):&#xA;        inputs, labels = inputs_and_labels&#xA;        if self.mode == &quot;train&quot;:&#xA;            loss = tf.nn.sampled_softmax_loss(&#xA;                weights=self.kernel,&#xA;                biases=self.bias,&#xA;                labels=labels,&#xA;                inputs=inputs,&#xA;                num_sampled=self.num_sampled,&#xA;                num_classes=self.num_classes,&#xA;                num_true=1)&#xA;&#xA;        elif self.mode == &quot;eval&quot;:&#xA;            logits = tf.matmul(inputs, tf.transpose(self.kernel))&#xA;            logits = tf.nn.bias_add(logits, self.bias)&#xA;            labels_one_hot = tf.one_hot(labels, self.num_classes)&#xA;            loss = tf.nn.softmax_cross_entropy_with_logits(&#xA;                labels=labels_one_hot,&#xA;                logits=logits)&#xA;&#xA;        return loss&#xA;&#xA;    def compute_output_shape(self, input_shape):&#xA;        dense_shape, classes_shape = input_shape&#xA;        return (dense_shape[0], )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Everything fine with dummy inputs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Test&#xA;y = np.arange(batch_size)&#xA;y_one_hot = to_categorical(y, vocab_size)&#xA;x_looked = np.array([np.random.choice(np.arange(vocab_size),&#xA;                                          sequence_len,&#xA;                                          replace=True) for _ in range(batch_size)]&#xA;                        )&#xA;x_features = np.random.rand(batch_size, sequence_len, n_features)&#xA;&#xA;layer_name = 'my_layer'&#xA;intermediate_layer_model = Model(inputs=model.input,&#xA;                                 outputs=model.get_layer(layer_name).output)&#xA;intermediate_output = intermediate_layer_model.predict([x_looked, x_features, y])  # OK&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I train&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Here comes the problem&#xA;model.fit([x_looked, x_features, y], y_one_hot)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py&quot;, line 2862, in run_code&#xA;    exec(code_obj, self.user_global_ns, self.user_ns)&#xA;  File &quot;&amp;lt;ipython-input-2-af27c0c040b8&amp;gt;&quot;, line 102, in &amp;lt;module&amp;gt;&#xA;    model.fit([x_looked, x_features, y], y_one_hot)&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/keras/engine/training.py&quot;, line 1689, in fit&#xA;    self._make_train_function()&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/keras/engine/training.py&quot;, line 990, in _make_train_function&#xA;    loss=self.total_loss)&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py&quot;, line 91, in wrapper&#xA;    return func(*args, **kwargs)&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/keras/optimizers.py&quot;, line 440, in get_updates&#xA;    grads = self.get_gradients(loss, params)&#xA;  File &quot;/anaconda3/lib/python3.6/site-packages/keras/optimizers.py&quot;, line 80, in get_gradients&#xA;    raise ValueError('An operation has `None` for gradient. '&#xA;ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46731" LastEditorUserId="46731" LastEditDate="2018-02-23T08:51:09.587" LastActivityDate="2019-07-15T21:40:06.637" Title="Keras negative sampling with custom layer" Tags="|neural-network|keras|tensorflow|sampling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28214" PostTypeId="1" CreationDate="2018-02-22T23:16:25.023" Score="1" ViewCount="50" Body="&lt;p&gt;In artificial neural networks, activation functions are used for neurons, i.e. the &lt;strong&gt;sigmoid&lt;/strong&gt; activation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/owDzr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/owDzr.png&quot; alt=&quot;Sigmoid function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which can be implemented in code as (in Python):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def sigmoid(x):&#xA;    return 1 / (1 + math.exp(-x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can we implement a biological activation function, such as the &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hodgkin-Huxley model&lt;/a&gt;&lt;/strong&gt;, whose mathematical form is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uSh00.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uSh00.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cm: Capacitance&lt;/li&gt;&#xA;&lt;li&gt;Vm: Membrane potential&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As mentioned on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt; page,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The typical Hodgkin–Huxley model treats each component of an excitable&#xA;  cell as an electrical element (as shown in the figure). The lipid&#xA;  bilayer is represented as a capacitance (Cm). Voltage-gated ion&#xA;  channels are represented by electrical conductances (gn, where n is&#xA;  the specific ion channel) that depend on both voltage and time. Leak&#xA;  channels are represented by linear conductances (gL). The&#xA;  electrochemical gradients driving the flow of ions are represented by&#xA;  voltage sources (En) whose voltages are determined by the ratio of the&#xA;  intra- and extracellular concentrations of the ionic species of&#xA;  interest. Finally, ion pumps are represented by current sources&#xA;  (Ip).[clarification needed] The membrane potential is denoted by Vm.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="32511" LastEditorUserId="1330" LastEditDate="2021-07-31T23:51:42.467" LastActivityDate="2021-07-31T23:51:42.467" Title="How to Implement Biological Neuron Activations in Artificial Neural Networks" Tags="|python|neural-network|activation-function|implementation|bioinformatics|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28216" PostTypeId="2" ParentId="25403" CreationDate="2018-02-23T00:52:04.260" Score="0" Body="&lt;p&gt;&lt;strong&gt;&lt;em&gt;API models exist which can achieve this.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://askmacgyver.com/explore/program/universal-topic-classifier/5S2Q5x8K&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://askmacgyver.com/explore/program/universal-topic-classifier/5S2Q5x8K&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It takes an array of categories or &quot;bag of words&quot; and a text string to analyze. It then returns a sorted percentage of relevance for this provided keywords.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Input Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  {&#xA;    &quot;text&quot;: &quot;this bank provides an excelent service to its clients when opening a new account and with other operations&quot;,&#xA;    &quot;classes&quot;: [&#xA;        &quot;bank account&quot;,&#xA;        &quot;online banking&quot;,&#xA;        &quot;technical support&quot;,&#xA;        &quot;mortgage&quot;,&#xA;        &quot;retirement savings&quot;,&#xA;        &quot;mutual funds&quot;,&#xA;        &quot;student loan&quot;,&#xA;        &quot;credit card&quot;,&#xA;        &quot;financial news&quot;&#xA;    ],&#xA;    &quot;minCutOff&quot;: &quot;0.001&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;API Response&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA;    &quot;bank account&quot;: 0.6448822158372491,&#xA;    &quot;technical support&quot;: 0.40099627067600924,&#xA;    &quot;financial news&quot;: 0.28635987039897565,&#xA;    &quot;mortgage&quot;: 0.2676284175575462,&#xA;    &quot;student loan&quot;: 0.257628495744561,&#xA;    &quot;online banking&quot;: 0.32395217514082025,&#xA;    &quot;credit card&quot;: 0.2144582134037077,&#xA;    &quot;mutual funds&quot;: 0.09250890827081894,&#xA;    &quot;retirement savings&quot;: 0.13690496892541437&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46734" LastActivityDate="2018-02-23T00:52:04.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28217" PostTypeId="2" ParentId="25336" CreationDate="2018-02-23T02:54:30.777" Score="3" Body="&lt;p&gt;I would suggest to have a look at &lt;a href=&quot;http://fasttext.cc&quot; rel=&quot;nofollow noreferrer&quot;&gt;fasttext&lt;/a&gt;, it will learn word embeddings for you from your corpus (advantage over word2vec embedding is that it learns representations based on character ngrams, so if a word is there, but its plural is not, it will still have a similar representation) and then in the supervised learning mode it uses something very simple like regression for classification, provided you have examples of labelled data.&#xA;I tried it with trivial 20newsgroups and it works well for distinct categories. Maybe not a production-friendly solution (it's in c++, but can be stretched to python, albeit not without pain), but can give an idea.&#xA;Given the nature of the algorithm, it's quick, compared to nets.&lt;/p&gt;&#xA;" OwnerUserId="46736" LastActivityDate="2018-02-23T02:54:30.777" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28220" PostTypeId="1" AcceptedAnswerId="28226" CreationDate="2018-02-23T05:58:27.347" Score="1" ViewCount="588" Body="&lt;p&gt;How can I use the shape of a distribution as a feature in machine learning ? Do I use something like the standard deviation ?&lt;/p&gt;&#xA;" OwnerUserId="46055" LastActivityDate="2018-02-23T08:39:16.573" Title="Shape of a distribution as a feature" Tags="|machine-learning|data-mining|statistics|algorithms|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28221" PostTypeId="1" AcceptedAnswerId="28224" CreationDate="2018-02-23T06:23:27.443" Score="-3" ViewCount="8206" Body="&lt;p&gt;I am working on my pandas tutorial. Below is my dataframe&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RDY3J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RDY3J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to drop duplicated header row using &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df_FDNY_dataset.drop_duplicates(subset=['FacilityName','FacilityAddress','Borough'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But drop_duplicates() function doesn't work in this case. Could you help me understand the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="41504" LastActivityDate="2018-10-11T07:08:53.727" Title="drop_duplicates() doesn't work in pandas" Tags="|pandas|dataframe|" AnswerCount="2" CommentCount="2" ClosedDate="2018-02-23T17:08:02.347" ContentLicense="CC BY-SA 3.0" />
  <row Id="28222" PostTypeId="2" ParentId="28220" CreationDate="2018-02-23T06:33:25.550" Score="0" Body="&lt;p&gt;You can use &lt;strong&gt;shape of distribution&lt;/strong&gt; as augmented features (extra columns) when your records (ie each row of the table) contains a number of homogeneous values that form a population. Otherwise, the shape of the whole training dataset does not help in differentiating the data points from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use mean and std as well as other parameters that define a statistical dustribution (depends which distribution you try to fit on your data).&lt;/p&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-02-23T06:33:25.550" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28223" PostTypeId="2" ParentId="28221" CreationDate="2018-02-23T06:42:46.260" Score="1" Body="&lt;p&gt;The header row is not duplicated, it is a row of the data frame (see index 0 attached with it, The actual columns don't have any index number). That's why you can't remove it using &lt;code&gt;drop_duplicates&lt;/code&gt;. If you want to remove it after having it in data frame, then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.iloc[1:,:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;df&lt;/code&gt; is your data frame.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-02-23T06:42:46.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28224" PostTypeId="2" ParentId="28221" CreationDate="2018-02-23T06:43:15.473" Score="2" Body="&lt;p&gt;So basically you want to drop the 1st row, which is indexed as 0 in the DataFrame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done by &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.drop(df.index[0], inplace = True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="34269" LastEditDate="2018-10-11T07:08:53.727" LastActivityDate="2018-10-11T07:08:53.727" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28225" PostTypeId="2" ParentId="28191" CreationDate="2018-02-23T06:47:23.680" Score="2" Body="&lt;p&gt;Since we know that &lt;code&gt;sin(x)&lt;/code&gt; can have -ve values as well,&#xA;Relu will kill all those mercilessly..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also we can't have relu just before the output layer as it doesn't makes any sense..(Atleast I have never seen this architecture before..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can make that 100 drop to around 40&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this &lt;a href=&quot;https://stats.stackexchange.com/q/307210&quot;&gt;link&lt;/a&gt; also&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-10T10:14:36.703" LastActivityDate="2018-04-10T10:14:36.703" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28226" PostTypeId="2" ParentId="28220" CreationDate="2018-02-23T08:39:16.573" Score="2" Body="&lt;p&gt;If this distribution is row specific (each sample has a different associated distribution) or category specific this is not a bad approach to encode more information in your features. It's unclear how you have these distributions, are there empirical samples or do you have a parameterized distribution? A few approaches you could take to encoding this could be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fit a distribution family (or if you have one already) and use the location/shape/scale parameters as features&lt;/li&gt;&#xA;&lt;li&gt;Add a few moments and other statistics of the distribution&lt;/li&gt;&#xA;&lt;li&gt;Or similarly take a few percentiles&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Depending on the parameterization of the distribution, the first might be less direct but it should be easy to test.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-02-23T08:39:16.573" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28227" PostTypeId="1" AcceptedAnswerId="28228" CreationDate="2018-02-23T08:51:12.860" Score="7" ViewCount="11625" Body="&lt;p&gt;I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while after balancing (i.e., 100 samples in both the classes)  it is 60%. What are the possible reasons for this?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastActivityDate="2018-02-23T09:21:10.390" Title="Why will the accuracy of a highly unbalanced dataset reduce after oversampling?" Tags="|classification|dataset|unsupervised-learning|supervised-learning|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28228" PostTypeId="2" ParentId="28227" CreationDate="2018-02-23T09:05:06.027" Score="6" Body="&lt;p&gt;Imagine that your data is not easily separable. Your classifier isn't able to do a very good job at distinguishing between positive and negative examples, so it usually predicts the majority class for any example. In the unbalanced case, it will get 100 examples correct and 20 wrong, resulting in a 100/120 = 83% accuracy. But after balancing the classes, the best possible result is about 50%. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem here is that accuracy is not a good measure of performance on unbalanced classes. It may be that your data is too difficult, or the capacity of your classifier is not strong enough. It's usually better to look at the confusion matrix to better understand how the classifier is working, or look at metrics other than accuracy such as the precision and recall, $F_1$ score (which is just the harmonic mean of precision and recall), or AUC. These are typically all easy to use in common machine learning libraries like &lt;code&gt;scikit-learn&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-02-23T09:05:06.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28229" PostTypeId="2" ParentId="28227" CreationDate="2018-02-23T09:21:10.390" Score="6" Body="&lt;p&gt;Accuracy is probably not a good metric for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the original dataset, if the model just makes a dummy prediction that all samples belong to the bigger class, the accuracy will be 83% (100/120). But that's usually not what we want to predict in an imbalanced dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a fraud detection problem. The probability that a transaction is a fraud is very small (let's say 0.01%) but the loss of an undetected fraud transaction is enormous (e.x. 1 millions dollars). On the other hand, the cost of manually verifying if a transaction is relatively small. In that case, we would like to detect all possible frauds, even if we have to make a lot of false positive predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To tackle an imbalanced dataset, first you have to choose which question you want to answer. Then, what's the good metric for this question. Answer these 2 question first before deciding which technique you should use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Come back to the original question. Why does accuracy reduce when we oversample the smaller class?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's because this technique puts more weight to the small class, makes the model bias to it. The model will now predict the small class with higher accuracy but the overall accuracy will decrease.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-23T09:21:10.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28230" PostTypeId="2" ParentId="27900" CreationDate="2018-02-23T11:34:32.290" Score="2" Body="&lt;p&gt;If you use one-hot here, you're just adding an unnecessary variable that is perfectly correlated with another variable in your model. Rather than thinking of it as a label encoding of gender where &quot;0=male, 1=female&quot;, think of it as a binary flag for is_female, where &quot;0=false, 1=true&quot;.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-02-23T11:34:32.290" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28231" PostTypeId="1" CreationDate="2018-02-23T11:42:51.090" Score="1" ViewCount="636" Body="&lt;p&gt;I am working with Python, scikit-learn, keras and with 450x540 rgb images of front-faced watches (e.g. &lt;a href=&quot;https://i.stack.imgur.com/dieB7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Watch_1&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/X8lnb.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Watch_2&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My aim to run an autoencoder or a Siemese Neural Network to find the most similar watches among them. However, I am not sure if I will get better results by comparing pairs of images or triplets of images. As it is defined in this &lt;a href=&quot;https://arxiv.org/abs/1404.4661&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper&lt;/a&gt;, triplets of images consist of one target image, one image which is (more) similar to the target image and one image which is not (so) similar to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain me in simple terms why using triplets of images will (necessarily) yield better results than using pair images, as research papers like the previous one claim?&lt;/p&gt;&#xA;" OwnerUserId="45610" LastEditorUserId="45610" LastEditDate="2018-02-23T11:56:32.423" LastActivityDate="2024-01-16T03:06:01.097" Title="Deep Learning - Find most similar images - Triplets vs Pairs" Tags="|python|neural-network|deep-learning|similarity|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28232" PostTypeId="1" CreationDate="2018-02-23T14:30:41.437" Score="4" ViewCount="8509" Body="&lt;p&gt;Since Orange is just running Python code under the hood...is it possible to set up and do my analysis in Orange but then export or output the underlying code for documentation or replication?&lt;/p&gt;&#xA;" OwnerUserId="46756" LastEditorUserId="29169" LastEditDate="2022-09-04T23:56:16.570" LastActivityDate="2022-09-04T23:56:16.570" Title="Export Python Code from Orange" Tags="|python|orange|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28234" PostTypeId="2" ParentId="21758" CreationDate="2018-02-23T15:35:58.480" Score="0" Body="&lt;p&gt;Each distribution could be estimated with &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;kernel density estimation (KDE)&lt;/a&gt;. KDE is a non-parametric way to estimate the probability density function of a random variable. Kernel density estimation is smooths the data / summarizing the data. It has the advantage of not having to pick a specific distribution apriori. The disadvantage of a KDE is the data is not summarized in a single variable, the data is summarized as a function.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-02-23T15:35:58.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28236" PostTypeId="1" CreationDate="2018-02-23T15:42:34.330" Score="2" ViewCount="35" Body="&lt;p&gt;i am building a face recognition app for my class attendance system , i collect training data from social website like facebook, instagram and other, as you can see the images i got from there is not usually front facial but at variety of angle. i was using haar_cascade for face detection , but it is not good for tilted face, can anyone suggest me the good algorithm for face detection through which i can detect face at variety of angle.&lt;/p&gt;&#xA;" OwnerUserId="45336" LastActivityDate="2018-02-23T15:42:34.330" Title="which algorithm will be good for detecting and recognition of faces from variety of angles" Tags="|deep-learning|tensorflow|computer-vision|open-source|object-detection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28237" PostTypeId="1" CreationDate="2018-02-23T16:05:18.793" Score="1" ViewCount="39" Body="&lt;p&gt;I have the string &quot;Ihaveadog&quot; and i need to recover space-characters in the string.I tried to use seq2seq model but failed.In my case i've got data like &quot;Ihaveadog&quot; -&gt; &quot;010000101000&quot; encoded by one-hot method.What type of DL model is good for the task or the best way is the using classic ML methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your time.&lt;/p&gt;&#xA;" OwnerUserId="46284" LastEditorUserId="46284" LastEditDate="2018-02-23T16:16:37.857" LastActivityDate="2018-02-23T16:16:37.857" Title="Type of model for space-characters recovery?" Tags="|deep-learning|nlp|keras|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28238" PostTypeId="2" ParentId="28213" CreationDate="2018-02-23T17:01:48.700" Score="4" Body="&lt;p&gt;Ok, I just flipped the arguments in the loss:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(loss=lambda loss, y_true: loss, optimizer='Adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(loss=lambda y_true, loss: loss, optimizer='Adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Enjoy&lt;/p&gt;&#xA;" OwnerUserId="46731" LastActivityDate="2018-02-23T17:01:48.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28239" PostTypeId="1" CreationDate="2018-02-23T17:40:26.140" Score="0" ViewCount="1734" Body="&lt;p&gt;I need to find anomalies in logs created by network monitor tool &lt;a href=&quot;https://www.bro.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bro&lt;/a&gt;. Right now I am referring to the example given by &lt;a href=&quot;https://github.com/Kitware/bat/blob/master/examples/anomaly_detection.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bro Analyze Tool&lt;/a&gt;. But the example only uses recent logs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I need to make it work with full log meaning to train on past logs and perform anomaly detection in real time on each new log. In the example, as far as I understand, unsupervised machine learning with IsolationForest and KMeans in &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt; python are used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have the following question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For unsupervised machine learning is it possible to store the model with trained data which is already created on previous run?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If possible can I add new data with existing model and then perform anomaly detection in real time?&lt;/p&gt;&#xA;" OwnerUserId="46766" LastEditorUserId="44456" LastEditDate="2018-02-25T07:00:20.220" LastActivityDate="2018-05-29T15:39:46.760" Title="Machine learning anomaly detection for continues data" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28240" PostTypeId="2" ParentId="28239" CreationDate="2018-02-23T17:48:40.367" Score="3" Body="&lt;p&gt;You can save &lt;code&gt;sklearn&lt;/code&gt; models with &lt;a href=&quot;https://docs.python.org/3/library/pickle.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pickle&lt;/code&gt;&lt;/a&gt;. &lt;a href=&quot;https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This blog post&lt;/a&gt; shows how.&lt;/p&gt;&#xA;" OwnerUserId="43870" LastEditorUserId="43870" LastEditDate="2018-02-24T10:33:02.330" LastActivityDate="2018-02-24T10:33:02.330" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28241" PostTypeId="1" AcceptedAnswerId="28390" CreationDate="2018-02-23T18:21:19.747" Score="0" ViewCount="1094" Body="&lt;p&gt;After reading the state of the Art of object detection using the CNN's(R-CNN,Faster R-CNN,YOLO,YOLOv2,SSD) I was wondering if there is an efficient method that use deep learning with reinforcement learning for object detection ?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T16:38:17.777" LastActivityDate="2021-02-13T16:38:17.777" Title="Reinforcement Deep Learning for object detection" Tags="|deep-learning|neural-network|reinforcement-learning|computer-vision|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ClosedDate="2018-03-05T06:27:27.580" ContentLicense="CC BY-SA 3.0" />
  <row Id="28243" PostTypeId="2" ParentId="28022" CreationDate="2018-02-23T19:38:54.910" Score="1" Body="&lt;p&gt;This is not at all a typical clustering problem, so I doubt &lt;em&gt;any&lt;/em&gt; of these algorithms will help. If you want to try clustering, you will need to do appropriate &lt;strong&gt;feature extraction&lt;/strong&gt;. Don't expect things to work on the raw data. But I guess once you have good features, the problem will already be solved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of trying to frame this as a clustering problem, look at it either from a &lt;em&gt;sequential&lt;/em&gt; pattern view point, or even better: look at the few questions on how to &lt;strong&gt;learn regexps from a set of strings&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-23T19:38:54.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28244" PostTypeId="1" CreationDate="2018-02-23T22:13:30.560" Score="2" ViewCount="2190" Body="&lt;p&gt;Context: I'm building a CNN on &lt;code&gt;MATLAB&lt;/code&gt; to classify &lt;a href=&quot;https://en.wikipedia.org/wiki/Wallpaper_group&quot; rel=&quot;nofollow noreferrer&quot;&gt;wallpaper groups&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the following network type.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;CONV -&gt; ReLU -&gt; POOL -&gt; CONV -&gt; ReLU -&gt; POOL -&gt; FC -&gt; DROPOUT -&gt; FC -&gt; SOFTMAX &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Should the parameters for the first CONV or POOL layers be the same as the second (or later) CONV or POOL layers succeeding it? If not, is there a meaningful way to choose parameters for the repeated succeeding layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for the first CONV layer you may pick a 5x5 filter. Perhaps for the second CONV layer it's more meaningful to pick a certain different size filter?&lt;/p&gt;&#xA;" OwnerUserId="32288" LastEditorUserId="28175" LastEditDate="2018-02-24T06:48:35.757" LastActivityDate="2018-06-24T08:53:25.070" Title="Choosing layer hyper-parameters of a CNN" Tags="|machine-learning|neural-network|deep-learning|classification|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28245" PostTypeId="2" ParentId="13011" CreationDate="2018-02-23T22:22:13.840" Score="2" Body="&lt;p&gt;This is fun! Here is an implementation of Emre's idea in Python. I tried to avoid loops wherever possible to make the code fast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, some imports and constants.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import Counter&#xA;import urllib.request&#xA;&#xA;ALPHABET = 'abcdefghijklmnopqrstuvwxyz'&#xA;LEN_ALPHABET = len(ALPHABET)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Download a comprehensive list of English words.&lt;/strong&gt; (I avoid calling it a 'dictionary' to avoid mix ups with the Python type of the same name).&#xA;I found one &lt;a href=&quot;http://www-personal.umich.edu/~jlawler/wordlist.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;Remove all words that contain non-alphabetic characters (including hyphenated words ... maybe not optimal).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;response = urllib.request.urlopen('http://www-personal.umich.edu/~jlawler/wordlist')&#xA;words = str(response.read()).replace('\\r', '').split('\\n')&#xA;words = {w for w in words if set(w).issubset(set(ALPHABET))}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Create the hash function.&lt;/strong&gt;&#xA;Map a given word to a tuple of length &lt;code&gt;LEN_ALPHABET&lt;/code&gt;. The first entry is the character count for &lt;code&gt;a&lt;/code&gt;, the second is the character count for &lt;code&gt;b&lt;/code&gt;, etc.&#xA;For instance, the word 'agaze' is hashed to the tuple &lt;code&gt;(2, 0, 0, 0, 1, 0, 1, 0, ...0, 0, 1)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;char_to_int = {c: i for i, c in enumerate(ALPHABET)}&#xA;&#xA;def hasher(w):&#xA;    w = [char_to_int[c] for c in w]&#xA;    count = Counter(w)&#xA;    return tuple(count[i] for i in range(LEN_ALPHABET))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3. Create the dictionary of hashed words.&lt;/strong&gt; The dictionary maps each tuple to the set of corresponding words.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;h_words = {}&#xA;for w in words:&#xA;    h_words.setdefault(hasher(w), set()).add(w)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;4. Look up anagrams.&lt;/strong&gt; Finally, a wrapper function to make lookups more convenient.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_anagrams(w):&#xA;    return h_words.setdefault(hasher(w), set())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A small test:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(get_anagrams('olleh'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This returns &lt;code&gt;{'hello'}&lt;/code&gt; as desired.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh, and make sure to execute this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print('The largest set of anagrams in the English language is:', max(h_words.values(), key=len))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-23T22:22:13.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28247" PostTypeId="1" CreationDate="2018-02-23T23:39:16.433" Score="1" ViewCount="93" Body="&lt;p&gt;I recently tried to make a CNN which could play the game Set: &lt;a href=&quot;https://en.wikipedia.org/wiki/Set_(game)&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Set_(game)&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I quickly learned after using my own convolutional neural network and trying to retrain Inception-V3 and AlexNet to classify pictures of three cards into &quot;set&quot; and &quot;notSet&quot;, that a CNN is not useful for this task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which machine learning algorithm would be useful for playing this game? Would an SVM work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are links to my last question (with some code) and the Github repo which includes sample images of the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/28002/keras-val-acc-unchanging-when-training-same-label-assigned-to-all-images&quot;&gt;Keras val_acc unchanging when training (same label assigned to all images)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/Keyan-Rhm/MLWithSet&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Keyan-Rhm/MLWithSet&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46509" LastActivityDate="2018-06-24T10:53:25.047" Title="Using Machine Learning to play the game SET" Tags="|machine-learning|neural-network|game|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28249" PostTypeId="1" CreationDate="2018-02-24T03:22:37.157" Score="6" ViewCount="768" Body="&lt;p&gt;Currently working on a project that requires multi-class image segmentation to identify distinct anomaly types in various sheet metals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have had moderate success with various NN segmentation architectures, however inference performance is far from ideal given the native resolution of the images (even troublesome with extensive down-sampling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have also attempted to hand-craft several features based on filter banks of various convolutional kernels and then subsequently applying clustering algorithms to build a sort of 'anomaly database'. I can readily apply this defined filter bank to extract features and compared to the features to those in the labeled database. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2QZPY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2QZPY.png&quot; alt=&quot;Example anomaly.&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shown in the above image, the center horizontal line needs to be identified as an anomaly. Intuition would lead me to believe simple shadow filters would be able to detect its presence, however given the inherent inconsistencies in the sheet metal - it doesn't appear to be simple (despite human identification being very simple). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not necessarily looking for a perfect solution to solve this, however if anyone was able to provide potential avenues for investigation. I want to exhaust all possible (promising) options of segmentation methods.&lt;/p&gt;&#xA;" OwnerUserId="46779" LastEditorUserId="46779" LastEditDate="2018-02-26T08:48:08.217" LastActivityDate="2018-02-27T10:53:18.787" Title="Image segmentation - handcrafted features vs DNN?" Tags="|machine-learning|deep-learning|feature-selection|feature-engineering|computer-vision|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28251" PostTypeId="1" AcceptedAnswerId="28258" CreationDate="2018-02-24T03:53:04.467" Score="1" ViewCount="295" Body="&lt;p&gt;I am using Two-Class Boosted Decision Tree to train model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Evaluation result I'd say really good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I am using real dataset - the result is very bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can possibly go wrong that makes such huge difference? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the screenshot of my model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8SBw2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8SBw2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two Class Boosted Decision Tree parameters (default):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JCBhg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JCBhg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LCDbR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LCDbR.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46783" LastEditorUserId="46783" LastEditDate="2018-02-24T19:56:06.923" LastActivityDate="2018-02-24T19:56:06.923" Title="Why training model give great result but real data gives very bad result: Azure ML Studio" Tags="|machine-learning|decision-trees|accuracy|azure-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28252" PostTypeId="1" CreationDate="2018-02-24T04:14:29.187" Score="6" ViewCount="14481" Body="&lt;p&gt;I am just trying to use pre-trained vgg16 to make prediction in Keras like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy import ndimage&#xA;from keras.preprocessing import image&#xA;from keras.applications.vgg16 import preprocess_input&#xA;&#xA;im = scipy.misc.imread(&quot;cat_dog/validation/cats/cat.1362.jpg&quot;).astype(np.float32)&#xA;im = scipy.misc.imresize(im, (224, 224)).astype(np.float32)&#xA;&#xA;#im /= 255.0&#xA;#im = im - np.mean(im, axis=2, keepdims=True)&#xA;&#xA;im = np.expand_dims(im, axis=0)&#xA;im = preprocess_input(im)&#xA;&#xA;out = vgg16_model.predict(im)&#xA;np.argmax(out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seemed that im /= 255.0 give very bad prediction. I commented it out and it started making good prediction. I also added preprocess_input(...) but that doesn't seem to affect prediction for the few random trials I did. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is that according to this great blog:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;noreferrer&quot;&gt;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;under &quot;Using the bottleneck features of a pre-trained network: 90% accuracy in a minute&quot;, pre-trained VGG16 is in a transfer learning context. And if you look at this &lt;a href=&quot;https://gist.github.com/fchollet/f35fbc80e066a49d65f1688a7e99f069&quot; rel=&quot;noreferrer&quot;&gt;gist&lt;/a&gt;, you see this line of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def save_bottlebeck_features():&#xA;    datagen = ImageDataGenerator(rescale=1. / 255)&#xA;    etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The preprocessing of input seemed to be 1/255.0 during caching of features from the last conv layer. This is sort of puzzling. I further also looked up how preprocess_input(...) is defined in the code I have, and found for 'tf':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x /= 127.5&#xA;x -= 1.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which you can check &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/python/keras/_impl/keras/applications/imagenet_utils.py&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="46782" LastActivityDate="2018-10-22T15:45:17.383" Title="Why is input preprocessing in VGG16 in Keras not 1/255.0" Tags="|keras|prediction|preprocessing|transfer-learning|vgg16|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28253" PostTypeId="1" AcceptedAnswerId="37405" CreationDate="2018-02-24T05:25:52.463" Score="2" ViewCount="2504" Body="&lt;p&gt;Currently i'm working on a sentiment analysis research project using LSTM networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the input I convert sentences into set of vectors using word2vec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And there are some well  pretrained word vectors like Google word2vec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is, is there are any advantages of using custom trained word2vecs(train using a dataset which related to our domain, such as user reviews of electronic items)  over pretrained ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whats the best option &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;use a pretrained word2vec&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train our own word2vec using a dataset related to the domain&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can any one help me on this&#xA;Thanks &lt;/p&gt;&#xA;" OwnerUserId="44950" LastActivityDate="2018-09-04T13:05:25.683" Title="Use pretrained word vectors over custom trained word2vecs" Tags="|word2vec|lstm|word-embeddings|word|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28254" PostTypeId="2" ParentId="28244" CreationDate="2018-02-24T06:47:55.610" Score="1" Body="&lt;p&gt;Finding an appropriate architecture is somehow practical. Those hyper-parameters you are talking may be different. Try to use a base architecture and then train your model. If it does not learn your model try to change the hyper parameters. It is an iterative operation to find a good model. There are a lot of debates, but there is not exactly consensus for making good models in &lt;code&gt;CNNs&lt;/code&gt;. You can think of if your model does not learn your data, it needs more features to be learned by the &lt;code&gt;CNN&lt;/code&gt;. The behavior of the layers are somehow as follows:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Max Pooling&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It is used for adding spacial invariance to the inputs of its layer. It is also used for decreasing the input size. By increasing its size, the two mentioned behavior would increase.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Convoluional Layers&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;These layers are used for extracting features to reduce the cost function in order to learn data. If you increase the size of these layers they behavior would be more vast, they would find features of a larger region.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-24T06:47:55.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28255" PostTypeId="1" CreationDate="2018-02-24T07:36:45.440" Score="6" ViewCount="7410" Body="&lt;p&gt;I would like to train a generative model that generates artificial handwritten text as output. which architectures would you recommend to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training input could be either images of handwritten letters, not words, or maybe sequences of points for each letter.&#xA;I thought of using some kind of combination of GAN+LSTM/GRU.&#xA;Already found:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/&quot; rel=&quot;noreferrer&quot;&gt;http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://distill.pub/2016/handwriting/&quot; rel=&quot;noreferrer&quot;&gt;https://distill.pub/2016/handwriting/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Would appreciate any further recommendations.&lt;/p&gt;&#xA;" OwnerUserId="46785" LastEditorUserId="13023" LastEditDate="2018-12-30T14:27:36.307" LastActivityDate="2020-03-12T23:01:00.327" Title="Best way to generate artificial handwritten text?" Tags="|generative-models|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28256" PostTypeId="2" ParentId="24739" CreationDate="2018-02-24T08:13:11.180" Score="0" Body="&lt;p&gt;Honestly, this does not sound like a machine learning problem.&#xA;I can think of two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have an analytic model for $dX/dt$, you can integrate it over $t$ to obtain a model for $X$. Then find the parameters that minimize squared distance to the data (plus some regularization perhaps). A grid search might be sufficient.&lt;/li&gt;&#xA;&lt;li&gt;Use standard methods of interpolating a function, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Spline_interpolation&quot; rel=&quot;nofollow noreferrer&quot;&gt;spline interpolation&lt;/a&gt;. The advantage of this is that you get a smooth closed-form solution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please also keep in mind that &lt;em&gt;the&lt;/em&gt; derivative for a finite set of points $X(t_1), X(t_2), ..., X(t_n)$ does not exist. There are infinitely many functions that pass through all points (and almost all of them are not differentiable). In other words: You have to make additional assumptions on the function that you are looking for. With the methods above, you make these assumptions explicitly, which I think is better than burying them deep inside a machine learning algorithm.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-24T08:13:11.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28257" PostTypeId="2" ParentId="28247" CreationDate="2018-02-24T09:15:48.380" Score="1" Body="&lt;p&gt;So you would like to play this game using images?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that's the case, I suggest dividing your problem into 2 sub-problems: Image classification (supervised learning) and reinforcement learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first sub-problem will detect the right card and the second sub-problem takes that as an input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to game, it's definitely worth trying reinforcement learning. You will only need to create a reinforcement learning environment for this game and use some off-the-shelf algorithms (like OpenAI baselines).&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-24T09:15:48.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28258" PostTypeId="2" ParentId="28251" CreationDate="2018-02-24T09:27:32.823" Score="1" Body="&lt;p&gt;Your question is not clear. There's 2 ways to understand it. Which dataset did you use to train your model?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You trained and tested on a premade dataset. The result  is great. Then you applied this model to real dataset and the result is really bad.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If this is the case, you should retrain on your real dataset or apply some Transfer Learning techniques to your current model.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;You trained and tested on a premade dataset. The result is great. Using the same model, you trained and tested on real dataset but the result is much worse.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I can't tell exactly the reason for this. Normally, real data is much more noisy. Did you handle missing data and do some feature engineering before training? &lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-24T09:27:32.823" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28259" PostTypeId="1" CreationDate="2018-02-24T11:26:34.550" Score="3" ViewCount="2062" Body="&lt;p&gt;The connection between cross entropy and log likelihood is widely expressed for the case when sample multi-class labels are one hot binary vectors (basically the same). Cross entropy is defined when labels are not one-hot but any valid probability distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the relation between the general cross entropy and log-likelihood? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;One-hot case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both log likelihood and cross entropy come down to: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$-\sum\limits_{i}\sum\limits_k1\{{y_i^{(k)}=1\}}{\log{\hat{y}_i^{(k)}}}(\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$i$ - the sample index&lt;br&gt;&#xA;$k$ - the class index&lt;br&gt;&#xA;$\theta$ - model parameters&lt;br&gt;&#xA;$y$ - samples labels (one-hot vector)&lt;br&gt;&#xA;$\hat{y}$ - predictions (probability distribution like vector)&lt;br&gt;&#xA;$1\{\}$ - indicator function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General case&lt;/strong&gt; ($y_i$ is probability distribution)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross entropy can be reduced to:&lt;br&gt;&#xA;$-\sum\limits_{i}y_i\log\hat{y}_i(\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does this relate to log-likelihood?&lt;/p&gt;&#xA;" OwnerUserId="38672" LastEditorUserId="38672" LastEditDate="2018-02-24T14:08:13.003" LastActivityDate="2022-02-28T11:26:18.683" Title="Connection between cross entropy and likelihood for multi-class soft label classification" Tags="|machine-learning|neural-network|predictive-modeling|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28267" PostTypeId="2" ParentId="25990" CreationDate="2018-02-24T17:56:13.563" Score="1" Body="&lt;p&gt;Media's explanation is true for &lt;em&gt;regression problems&lt;/em&gt;. These are problems where you predict a continuous target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your image shows a &lt;em&gt;classification problem&lt;/em&gt;. Here, the target variable takes only two values (typically &lt;code&gt;-1&lt;/code&gt; and &lt;code&gt;1&lt;/code&gt; in the Perceptron algorithm).&#xA;In that case, an optimal solution $w^*$ is a vector of weights that perfectly separates both classes.&#xA;If such a solution exists, the Perceptron algorithm will find it. But: If there is one optimal solution, there are usually infinitely many other optimal solutions. You can easily see this in your image: You can move the line a little to the left or to the right, and you can rotate it a little, and it still perfectly separates the classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So while the Perceptron algorithm will find an optimal solution if there is one, you cannot know &lt;em&gt;which one&lt;/em&gt; it will find. That depends on the random starting parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is different e.g. for &lt;a href=&quot;https://en.wikipedia.org/wiki/Support_vector_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;support vector machines&lt;/a&gt;. Here, there is either no optimal solution or exactly one optimal solution.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-24T17:56:13.563" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="28269" PostTypeId="1" CreationDate="2018-02-24T20:00:00.547" Score="1" ViewCount="109" Body="&lt;p&gt;I am currently doing a project in python. This project is Aspect based sentiment analysis. I don't know how to prepare training file for that &lt;/p&gt;&#xA;" OwnerUserId="46810" LastActivityDate="2018-02-24T20:00:00.547" Title="Sentiment Analysis" Tags="|machine-learning|python|nlp|" AnswerCount="0" CommentCount="0" ClosedDate="2018-02-25T16:29:50.397" ContentLicense="CC BY-SA 3.0" />
  <row Id="28270" PostTypeId="1" CreationDate="2018-02-24T20:17:50.240" Score="1" ViewCount="25" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2Agjt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2Agjt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;I have three different histograms (Impact parameter distributions) corresponding to three groups of the same particle with different properties. However, the three distribution have more or less the same shape. Now I want to predict a fourth distribution (with another property), which should have more or less the same shape as the three other ones. I have only a part of this latter distribution in a certain range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A first attempt was to predict the rest of the distribution using a Transfer Factor (TF), which is the ratio of integrals in two different ranges, and by multiplying this TF to the integral of the known range in the fourth distribution, I could estimate the number of events in the unknown range. I did this using the three known distribution, but I obtained three different results, which were not even close to each other. I think that one of the three must be the good one, but I don't know which one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: Is there a statistical method to determine the best one? or is there another way/method to approximate my last distribution?&lt;/p&gt;&#xA;" OwnerUserId="39911" LastEditorUserId="39911" LastEditDate="2018-02-25T01:10:10.453" LastActivityDate="2021-08-04T18:22:55.937" Title="Number of events estimation" Tags="|statistics|parameter-estimation|distribution|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28271" PostTypeId="2" ParentId="28022" CreationDate="2018-02-24T21:00:11.280" Score="1" Body="&lt;p&gt;For those interested in a solution for similar problems, I found a solution with these steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Splitting the filenames on &quot;_&quot;, generating n strings&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Taking the length of each string&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Running KMeans (optimal K using Gap Statistics)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Taking one sample per cluster and reverse-engineering it to a generic regex, via a customized function&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In practice, here is an example of 10 files:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/naUfc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/naUfc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DIuvH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DIuvH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;File 0 splits on &quot;&lt;em&gt;&quot; into groups of lengths 3, 10 and 23 respectively.&#xA;File 3 splits on &quot;&lt;/em&gt;&quot; into groups of lengths 11 and 13 respectively.&#xA;File 5 splits on &quot;_&quot; into groups of lengths 4, 3, 10, 5, 3 and 12 respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Files 0, 1 and 2 belong to the same cluster and have identical naming convention. Files 3, 4, 6, 7, 8 and 9 belong to the same cluster and have identical naming convention. File 5 belongs to another cluster and has yet another naming convention.&lt;/p&gt;&#xA;" OwnerUserId="44865" LastActivityDate="2018-02-24T21:00:11.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28272" PostTypeId="1" CreationDate="2018-02-24T21:03:18.753" Score="3" ViewCount="1561" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/84FwU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/84FwU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;blue: training loss&lt;br&gt;&#xA;black: validation loss&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If modelling gives me such plot, what does it imply? Is it overfitting?&#xA;How should I make it better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier = Sequential()&#xA;classifier.add(LSTM(&#xA;    input_dim=72,&#xA;    output_dim=50,&#xA;    return_sequences=True))&#xA;classifier.add(Dropout(0.20))&#xA;&#xA;classifier.add(LSTM(&#xA;    100,&#xA;    return_sequences=False))&#xA;classifier.add(Dropout(0.20))&#xA;&#xA;classifier.add(Dense(&#xA;    output_dim=1))&#xA;classifier.add(Activation('relu'))&#xA;classifier.compile(loss='mse', optimizer='rmsprop')&#xA;&#xA;X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))&#xA;X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))&#xA;&#xA;#classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 10)&#xA;cls_return = classifier.fit(X_train, y_train, batch_size=300, nb_epoch=500, validation_split=0.05)&#xA;history = cls_return.history&#xA;plt.plot(history['loss'], label='Training Loss', color='blue')&#xA;plt.plot(history['val_loss'], label='Validation Loss', color='black')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12217" LastActivityDate="2018-09-01T15:22:49.663" Title="Stock prediction through LSTM" Tags="|deep-learning|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28274" PostTypeId="1" CreationDate="2018-02-25T05:28:09.720" Score="1" ViewCount="296" Body="&lt;p&gt;&lt;strong&gt;My understanding of l2 regularisation:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;My problem statement:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am making a reasonable assumption(based on domain knowledge) that my features are independent which means I can use the weights of the features to infer the importance of features in influencing Y.&lt;/p&gt;&#xA;&lt;p&gt;From domain knowledge, I want to assume priors about the ratio of weights.&#xA;Eg:&lt;/p&gt;&#xA;&lt;p&gt;W1/W2 is a guassian distribution with mean 0.7&lt;/p&gt;&#xA;&lt;p&gt;W2/W5 is a guassian distribution with mean 2.4&lt;/p&gt;&#xA;&lt;p&gt;And then MAP estimate over data will give me a cost function, with extra penalty added.&lt;/p&gt;&#xA;&lt;p&gt;Is my thought process correct? My intent is weights should be close to priors computed from some heuristic. This will also help me handle sparse data.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Implementation Details:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Is there any library(in any language) where it is easy to do this? Or do I have to compute the cost function and it's gradient myself and implement gradient descent over it?&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Help with Math:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Also I have started learning stats(and maths in general) more rigorously, but I have lots of ground yet to cover. Assuming all Wi/Wj are given, can someone please give me the new cost function. I will write a program to solve for weights based on that cost function.&lt;/p&gt;&#xA;&lt;p&gt;EDIT:&#xA;As I know Wi/Wj, can I assume  sum(Wi) = 100 and fit a bayesian logistics regression?&#xA;Also bayesian logistic regression fit method api requires a hessian matrix.&#xA;&lt;a href=&quot;http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic&lt;/a&gt; calls hessian matrix covariance matrix of fitted MAP parameters. Assuming that features are independent and variance of a feature to be 1, can I assume hessian matrix to be an identity matrix?&lt;/p&gt;&#xA;" OwnerUserId="27108" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2024-01-10T16:03:20.410" Title="Custom regularisation for logistics regression" Tags="|logistic-regression|regularization|bayesian|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28275" PostTypeId="1" CreationDate="2018-02-25T05:32:57.343" Score="1" ViewCount="210" Body="&lt;p&gt;I have a dataset in which one column is as given in the picture. &#xA;What will be the best way to handle such a column?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PTjfl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PTjfl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46799" LastActivityDate="2018-04-26T11:02:19.713" Title="How to manipulate this column of less than/greater than?" Tags="|dataset|feature-selection|data-cleaning|feature-extraction|excel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28276" PostTypeId="2" ParentId="28275" CreationDate="2018-02-25T06:11:59.993" Score="2" Body="&lt;p&gt;This is a categorical variable. There are multiple ways to handle categorical variables in the literature - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dummification (mostly for nominal variables)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The variable is split and converted into multiple binary variables (the number of variables should be &lt;code&gt;1-(number of classes)&lt;/code&gt; ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python example - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;series = pd.Series(['&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', &#xA;                    '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;lt;=50k'])&#xA;&#xA;print(pd.get_dummies(series).iloc[:, 1:])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;50k&#xA;0      0&#xA;1      0&#xA;2      0&#xA;3      0&#xA;4      0&#xA;5      0&#xA;6      0&#xA;7      1&#xA;8      1&#xA;9      1&#xA;10     1&#xA;11     1&#xA;12     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice how the information that was string has been now &lt;em&gt;encoded&lt;/em&gt; in numbers. The reason was dropping a column is so that there is no perfect multicollinearity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ranking(in case of ordinal variables)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are the cases where you can clearly &lt;code&gt;rank&lt;/code&gt; the variables. In your case it would make more sense to rank them as we know then &lt;code&gt;&amp;gt;50k&lt;/code&gt; is actually more than &lt;code&gt;&amp;lt;50k&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use the ranking method in you salary case and the dummification method for nominal cases (like &lt;code&gt;location&lt;/code&gt; variables - where it can't be ordered)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python example - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;series = pd.Series(['&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', '&amp;lt;=50k', &#xA;                    '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;gt;50k', '&amp;lt;=50k'])&#xA;uniq = series.unique()&#xA;ranks = { uniq[i]:i+1 for i in range(len(uniq)) }&#xA;print(ranks)&#xA;series_ranked = series.apply(lambda x: ranks[x])&#xA;print(series_ranked)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0     1&#xA;1     1&#xA;2     1&#xA;3     1&#xA;4     1&#xA;5     1&#xA;6     1&#xA;7     2&#xA;8     2&#xA;9     2&#xA;10    2&#xA;11    2&#xA;12    1&#xA;dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again, in no way is this an exhaustive answer, but rest assured these are the most popular methods used to deal with categorical variables. Hope that helps. Cheers!&lt;/p&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2018-02-25T06:11:59.993" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28277" PostTypeId="1" AcceptedAnswerId="28279" CreationDate="2018-02-25T06:25:18.973" Score="0" ViewCount="1844" Body="&lt;p&gt;I have a pandas dataframe like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0    15.55&#xA;1    15.55&#xA;2    15.55&#xA;3    15.55&#xA;4    20.84&#xA;Name: Y1, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to convert the values of Y1 to categorical (i.e) if its greater than 18.25, I want it 1 else 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please help me on how to do it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what i tried so far&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for temp in TRAIN_ID1:&#xA;    train_ID1.loc[(train_ID1['Y1'] &amp;gt; 18.250000), 'Y1'] = 1&#xA;    train_ID1.loc[(train_ID1['Y1'] &amp;lt; 18.250000), 'Y1'] = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But im getting an error&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---------------------------------------------------------------------------&#xA;TypeError                                 Traceback (most recent call last)&#xA;pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;TypeError: an integer is required&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;&amp;lt;ipython-input-118-2cccb791d834&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 for temp in train_ID1:&#xA;----&amp;gt; 2     train_ID1.loc[(train_ID1['Y1'] &amp;gt; 18.250000), 'Y1'] = 1&#xA;      3     train_ID1.loc[(train_ID1['Y1'] &amp;lt; 18.250000), 'Y1'] = 0&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\series.py in __getitem__(self, key)&#xA;    621         key = com._apply_if_callable(key, self)&#xA;    622         try:&#xA;--&amp;gt; 623             result = self.index.get_value(self, key)&#xA;    624 &#xA;    625             if not is_scalar(result):&#xA;&#xA;~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\indexes\base.py in get_value(self, series, key)&#xA;   2558         try:&#xA;   2559             return self._engine.get_value(s, k,&#xA;-&amp;gt; 2560                                           tz=getattr(series.dtype, 'tz', None))&#xA;   2561         except KeyError as e1:&#xA;   2562             if len(self) &amp;gt; 0 and self.inferred_type in ['integer', 'boolean']:&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()&#xA;&#xA;KeyError: 'Y1'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="44680" LastEditDate="2018-02-25T06:49:15.817" LastActivityDate="2018-02-25T07:17:05.857" Title="How to iterate and modify rows in a dataframe( convert numerical to categorical)" Tags="|machine-learning|dataset|pandas|numpy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28279" PostTypeId="2" ParentId="28277" CreationDate="2018-02-25T07:17:05.857" Score="1" Body="&lt;p&gt;As the error suggests, you don't have a column called &lt;code&gt;Y1&lt;/code&gt;. Hence the error. Here is my suggestion to fix this. Assuming you input data looks like this - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;15.55&#xA;15.55&#xA;15.55&#xA;15.55&#xA;20.84&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Read it in pandas this way - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.read_csv('path/to/file.csv', header=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Provide a column name for this - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.columns = ['Y1']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you have more columns, just fill the &lt;code&gt;df.columns&lt;/code&gt; list accordingly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, use the &lt;code&gt;pandas&lt;/code&gt; best practices as per their latest documentation to &lt;code&gt;assign&lt;/code&gt; a new column - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.assign(Y2= (df['Y1'] &amp;gt; 18.250000).astype(int))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(df)&#xA;&#xA; Y1  Y2&#xA;0  15.55   0&#xA;1  15.55   0&#xA;2  15.55   0&#xA;3  15.55   0&#xA;4  20.84   1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: Since I don't have full visibility on what you are working on, I have assumed what &lt;em&gt;might&lt;/em&gt; be the problems you are facing. If this doesn't work let me know.&lt;/p&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2018-02-25T07:17:05.857" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28281" PostTypeId="2" ParentId="28272" CreationDate="2018-02-25T09:18:59.267" Score="8" Body="&lt;p&gt;Yes, this is overfitting. Financial time series exhibit many peculiarities, including &lt;a href=&quot;https://en.wikipedia.org/wiki/Heteroscedasticity&quot; rel=&quot;nofollow noreferrer&quot;&gt;heteroscedasticity&lt;/a&gt;, high &lt;a href=&quot;https://www.investopedia.com/terms/t/tailrisk.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;tail risk&lt;/a&gt;, all kinds of &lt;a href=&quot;https://www.investopedia.com/terms/s/seasonality.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;seasonality&lt;/a&gt; and all kinds of &lt;a href=&quot;https://www.investopedia.com/terms/m/momentum.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;momentum&lt;/a&gt;. Machine learning algorithms tend to pick up these patterns and overfit on them. To my knowledge, there is no machine learning algorithm that can compete with classical time series analysis when it comes to understanding stock prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please also keep in mind that if someone &lt;em&gt;were&lt;/em&gt; able to make predictions like 'tomorrow the stock price is going to rise by 5%', they could earn billions within a few weeks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; You ask if there is any chance of training a network that is better than random guessing. Yes, for two reasons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, if you try hundreds of parameter combinations and evaluate them all on the same test set, you become the victim of what might be called 'overfitting of the second degree.' Performance on the test set becomes part of your decision process because you discard models that perform poorly on the test set and keep models that perform well. You are essentially training on the test set and will predict it with 100% accuracy eventually. But use this seemingly perfect model on new, previously unseen test data, and you will be disappointed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, heteroscedasticity, seasonality and momentum all introduce a little bit of 'real' predictability into your time series. If you get &lt;em&gt;very&lt;/em&gt; lucky, your algorithm might identify these patterns and at the same time ignore all the weird residual noise.&#xA;This is incredibly unlikely, and it is probably irrelevant. You should not compare your prediction accuracy to random guessing, but to a reasonable baseline model. For monthly prices of individual stocks, this could be the &lt;a href=&quot;https://en.wikipedia.org/wiki/Fama%E2%80%93French_three-factor_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fama-French three-factor model&lt;/a&gt;. (Daily or intraday stock prices will send you a bit deeper into the rabbit hole that is quantitative finance). I bet you five dollars that you will not be able to beat that model's performance on previously unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advice to you: Don't clean the lion's cage on your first day as a zookeeper. Start with something nice and easy.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-09-01T15:22:49.663" LastActivityDate="2018-09-01T15:22:49.663" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="28282" PostTypeId="1" CreationDate="2018-02-25T09:43:26.337" Score="5" ViewCount="4667" Body="&lt;p&gt;When we care more that there should be no false negatives, as far as possible… ie. higher recall (video is suitable for kid or not), we should use (receiver operating characteristic) ROC (area under the curve) AUC and try to maximize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scikit-Learn provides a function to compute this directly:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import roc_auc_score&#xA;roc_auc_score(y_train_5, y_scores)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly, when we care more about the false positives than the false negatives, example shop lifting case then what should we do?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should we just try to maximize the recall ignoring the precision or is there a better metric?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had read somewhere &quot;Use the precision vs recall curve and get that max&quot; but not very sure what that means... Can you please explain the same?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there a direct function for this metric in sci-kit learn, like above for the first case, please do let me know.&lt;/p&gt;&#xA;" OwnerUserId="46824" LastEditorUserId="29575" LastEditDate="2018-06-26T23:46:54.213" LastActivityDate="2018-07-11T15:33:40.967" Title="Sci-kit learn function to select threshold for higher recall than precision" Tags="|machine-learning|neural-network|deep-learning|classification|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28283" PostTypeId="2" ParentId="28282" CreationDate="2018-02-25T13:22:33.037" Score="0" Body="&lt;p&gt;The easiest way is to replace your labels. The other way is to set importance of the more important class to a higher value so the cost function moves toward direction to take much care for your desired label. You can set the &lt;em&gt;class_weight&lt;/em&gt;. Take a look at &lt;a href=&quot;https://stackoverflow.com/questions/30972029/how-does-the-class-weight-parameter-in-scikit-learn-work&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-25T13:22:33.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28284" PostTypeId="2" ParentId="16342" CreationDate="2018-02-25T13:27:51.203" Score="6" Body="&lt;p&gt;Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too!&lt;/p&gt;&#xA;&lt;p&gt;I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, &lt;a href=&quot;https://datascience.stackexchange.com/a/9493/37156&quot;&gt;here&lt;/a&gt; goes.&lt;/p&gt;&#xA;&lt;p&gt;Thanks &lt;a href=&quot;https://datascience.stackexchange.com/users/13023/wacax&quot;&gt;wacax&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37156" LastEditorUserId="25180" LastEditDate="2020-12-19T16:00:38.533" LastActivityDate="2020-12-19T16:00:38.533" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28285" PostTypeId="1" CreationDate="2018-02-25T14:02:12.957" Score="5" ViewCount="11841" Body="&lt;p&gt;There are a lot of way to deal with class-imbalanced data like undersampling, oversampling, changing cost function etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&quot; rel=&quot;noreferrer&quot;&gt;https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the post talking about them all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working with an imbalanced multiclass classification problem and trying to solve it using XGBoost algorithm. I wanted to understand which method works best here. Since XGBoost already has a parameter called weights (which gives weight to each train record), would it be wise to directly use it instead of undersampling, oversampling, writing a cost function etc.?&lt;/p&gt;&#xA;" OwnerUserId="37156" LastActivityDate="2018-02-25T23:59:07.530" Title="What is the best way to deal with imbalanced data for XGBoost?" Tags="|classification|r|xgboost|multilabel-classification|class-imbalance|" AnswerCount="1" CommentCount="1" ClosedDate="2018-02-26T01:02:08.353" ContentLicense="CC BY-SA 3.0" />
  <row Id="28286" PostTypeId="2" ParentId="28282" CreationDate="2018-02-25T14:05:18.927" Score="1" Body="&lt;p&gt;As far as a fuction in scikit to implement a certain threshold for a higher recall, I don't think there is one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, depending on what model you're using, you can vary the threshold for probability outputs of the model to obtain a higher Recall.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say you are using a &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logistic Regression&lt;/a&gt; classifier model, then you can use the &lt;code&gt;predict_proba&lt;/code&gt; function on your data set; this will return probabilities for each sample belonging to a particular class. If you then create a new array from the output of &lt;code&gt;predict_proba&lt;/code&gt; using your threshold, then the model will have a Recall with your desired accuracy when it classifies new unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g pseudo code for binary classification&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = sklearn.ensemble.RandomForestClassifier()&#xA;model = fit(X,y) # fit model to training datset&#xA;probs = model.predict_proba(X_new) # prediction on a new dataset X_new&#xA;&#xA;threshold = 0.7 # threshold we set where the probability prediction must be above this to be classified as a '1'&#xA;classes = probs[:,1] # say it is the class in the second column you care about predictint&#xA;classes[classes&amp;gt;=threshold] = 1&#xA;classes[classes&amp;lt;threshold] = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-02-25T14:05:18.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28287" PostTypeId="2" ParentId="27966" CreationDate="2018-02-25T15:00:03.167" Score="0" Body="&lt;p&gt;Instead of computing $A$, make your algorithm compute $10\times A.$&#xA;Then use your favourite loss function for continuous variables. The most common one is the squared $L^2$-norm $l(10\times A, B) = (10\times A-B)^2.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a plot of $l$ that I made with &lt;a href=&quot;http://www.wolframalpha.com/input/?i=plot(10*A-B)%5E2,%20A%20in%20(0,1),%20B%20in%20(0,10)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wolfram Alpha&lt;/a&gt;:&lt;a href=&quot;https://i.stack.imgur.com/kUE4w.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kUE4w.jpg&quot; alt=&quot;Quadratic loss function&quot;&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-25T15:00:03.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28288" PostTypeId="1" CreationDate="2018-02-25T17:30:47.807" Score="0" ViewCount="377" Body="&lt;p&gt;After applying clustering algorithm I need to extract those data which exists centre of the cluster and which exists border of the cluster. How could I do this by using python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use k-means clustering algorithm and divide those data into 19 cluster. I am also using scikit learn library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edited:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my plot after clustering: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/Pr07o.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Pr07o.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="43455" LastEditDate="2018-02-25T18:00:14.110" LastActivityDate="2018-04-27T10:02:37.077" Title="How to collect data which exists centre or border of the cluster?" Tags="|python|data-mining|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28290" PostTypeId="2" ParentId="25141" CreationDate="2018-02-25T19:10:15.607" Score="0" Body="&lt;p&gt;From your comments, it sounds like you are trying to predict the next value of a series $X_1, X_2, \dots,$ where each $X_i$ is $(1000,2)$-dimensional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about this: To predict $X_i$, you feed in the tensor $(X_{i-k}, X_{i-(k+1)}, \dots, X_{i-1})$, which has dimensions $(1000, 2, k)$. You apply convolution with stride $1$ and appropriate padding to the first and the second dimension: this does not change dimensions.&#xA;And you apply pooling to the third dimension only (for example, set $k=8$ and use three pooling layers with window size $2\times 2$).&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-25T19:10:15.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28291" PostTypeId="2" ParentId="26025" CreationDate="2018-02-25T21:02:53.940" Score="0" Body="&lt;p&gt;Expected value and standard deviation are 'independent' in the sense that knowing one gives you no additional information about the other. Therefore it is more accurate and much simpler to train two separate networks - one for the expected value and one for the standard deviation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also has the advantage that you need to be explicit about what you mean by 'standard deviation of a (heteroscedastic) time series'.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-26T07:36:45.477" LastActivityDate="2018-02-26T07:36:45.477" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28292" PostTypeId="2" ParentId="28285" CreationDate="2018-02-25T23:59:07.530" Score="2" Body="&lt;p&gt;I think using something like this could help in your case.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-train.R#L17&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-train.R#L17&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps at least a bit!&lt;/p&gt;&#xA;" OwnerUserId="46785" LastActivityDate="2018-02-25T23:59:07.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28293" PostTypeId="1" CreationDate="2018-02-26T01:38:24.120" Score="8" ViewCount="5749" Body="&lt;p&gt;I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of 10 time-series; each time-series consists of data collected from torque value of a tire (so 10 tires in total) and the sets &lt;strong&gt;may not contain same number of data points (set size differ)&lt;/strong&gt;. Each time-series data is pretty much just the tire_id, timestamp, and the sig_value (value from the signal, or the sensor). Sample data for one time-series looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tire_id        timestamp        sig_value&#xA;tire_1           23:06.1            12.75&#xA;tire_1           23:07.5                0&#xA;tire_1           23:09.0            -10.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I have 10 of them, and 2 of them behave strangely. I understand that this is an anomaly detection problem, but most of the articles I read online are detecting anomaly points within the same time-series (aka if at some points the torque values are not normal for that tire). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To detect which 2 tires are behaving abnormally, I tried using clustering method, basically k-means clustering (since it's unsupervised). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To prepare the data to feed into the k-means clustering, for each time-series (aka for each tire), I calculated:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The top 3 sets of adjacent local maximum and local minimum with&#xA;highest amplitude (difference)&lt;/li&gt;&#xA;&lt;li&gt;Mean of torque value &lt;/li&gt;&#xA;&lt;li&gt;Standard Deviation of torque values&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I also set the number of clusters to be only 2, so either cluster 1 or 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my end result (after assigning clusters) looks like following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        amplitude  local maxima  local minima  sig_value_std  \&#xA;tire_0     558.50        437.75       -120.75      77.538645   &#xA;tire_0     532.75        433.75        -99.00      77.538645   &#xA;tire_0     526.25        438.00        -88.25      77.538645   &#xA;tire_1     552.50       -116.50        436.00      71.125912   &#xA;tire_1     542.75        439.25       -103.50      71.125912   &#xA;&#xA;        sig_value_average  cluster  &#xA;tire_0          12.816990        0  &#xA;tire_0          12.816990        0  &#xA;tire_0          12.816990        0  &#xA;tire_1          11.588038        1  &#xA;tire_1          11.588038        0 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I have a question of what to do with this result ... so each tire has 3 rows of data, as I've picked the top 3 pairs of local max/min with 3 largest amplitudes, and that means each row can be assigned to a cluster, and sometimes they are assigned to different clusters for 1 tire even. Also the cluster size is normally larger than just 2. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to do anomaly detection about &quot;set of time-series&quot; not just individual&#xA;data points? &lt;/li&gt;&#xA;&lt;li&gt;Is my approach reasonable/logical? If it is, how can I clean up my result to get what I want? And if not, what can I do to improve?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="17423" LastEditorUserId="17423" LastEditDate="2018-03-04T03:35:49.600" LastActivityDate="2018-03-04T03:35:49.600" Title="how to compare different sets of time series data" Tags="|python|clustering|time-series|anomaly-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28294" PostTypeId="1" CreationDate="2018-02-26T02:05:36.530" Score="1" ViewCount="97" Body="&lt;p&gt;I am working on a project where I need to denoise images, and my dataset is composed of a big chunk of pairs &lt;code&gt;&amp;lt;natural image, same image with synthetically added noise&amp;gt;&lt;/code&gt;. The fact is, I have multiple sources of noise (with labels), e.g. Gaussian noise, salt&amp;amp;pepper, distortion, saturation to name a few. Different noises types are on the same original images, meaning that for each undistorted image, I have one pair for each noise type, which I believe is relevant for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to the recent success of GANs for image translation tasks, I was looking into recent architectures and how I can adapt them for my task. The question is: do you think that is possible for a GAN to learn a many-to-one mapping between distributions, i.e. different noise distributions (the many) and the undistorted image distribution (the one) or do I need to train multiple networks for different kinds of noise?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that I have paired images I was looking into the &lt;code&gt;pix2pix&lt;/code&gt; architecture. What about the recent &lt;code&gt;CycleGAN&lt;/code&gt;? Learning the backward mapping function looks a lot tougher if there are multiple distributions, which I guess is why when they release the pre-trained models they indeed have multiple models for multiple tasks.&lt;/p&gt;&#xA;" OwnerUserId="46854" LastEditorUserId="29575" LastEditDate="2018-02-26T02:09:26.377" LastActivityDate="2019-03-10T14:00:34.690" Title="Generative adversarial networks for multiple distribution noise removal" Tags="|machine-learning|computer-vision|gan|generative-models|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28295" PostTypeId="2" ParentId="27953" CreationDate="2018-02-26T03:41:21.530" Score="3" Body="&lt;p&gt;The main difference between explicit density models and implicit density models is that explicit density models, use an explicit density function and implicit density models don't. In other words explicit models assume some prior distribution about the data. For example the work of &lt;a href=&quot;https://arxiv.org/pdf/1605.08803.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dinh et. al.&lt;/a&gt;, which in the reference link is classified as an explicit model, use an isotropic unit norm Gaussian as the prior (page 7, of the above link). One example of implicit density model is the work of &lt;a href=&quot;https://arxiv.org/pdf/1306.1091.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bengio et. al&lt;/a&gt;, also mentioned in the reference link, to estimate the distribution of the data they learn the transition operator of a Markov&#xA;chain whose stationary distribution estimates the data distribution.     &lt;/p&gt;&#xA;" OwnerUserId="26243" LastActivityDate="2018-02-26T03:41:21.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28296" PostTypeId="1" AcceptedAnswerId="28313" CreationDate="2018-02-26T04:00:44.703" Score="5" ViewCount="5230" Body="&lt;p&gt;I am reading a book on Pattern Recognition (by Prof V Susheela Devi and Prof Murty) where in the chapter of data representation 2.3.3 the non metric similarity function is defined as those which do not obey either the triangular inequality or symmetry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that context, it further adds &lt;strong&gt;&lt;em&gt;The squared Euclidean distance is itself an example of a non-metric, but it gives the same ranking as the Euclidean distance which is a metric.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the squared Euclidean distance different from the Euclidean distance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is the squared Euclidean distance non-metric? &lt;/p&gt;&#xA;" OwnerUserId="46326" LastEditorUserId="44139" LastEditDate="2018-02-26T05:52:00.263" LastActivityDate="2018-02-26T13:31:57.177" Title="How the squared Euclidean distance is an example of non-metric function?" Tags="|machine-learning|data-mining|distance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28299" PostTypeId="1" AcceptedAnswerId="28301" CreationDate="2018-02-26T05:34:36.810" Score="3" ViewCount="3737" Body="&lt;p&gt;If we consider two conditions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Number of data is huge &lt;/li&gt;&#xA;&lt;li&gt;Number of data is low&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For what condition does boosting or bagging overfit more compared to the other one?&lt;/p&gt;&#xA;" OwnerUserId="46433" LastEditorUserId="45507" LastEditDate="2018-02-26T17:18:15.687" LastActivityDate="2018-02-26T20:39:13.677" Title="When does boosting overfit more than bagging?" Tags="|machine-learning|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28301" PostTypeId="2" ParentId="28299" CreationDate="2018-02-26T05:45:18.803" Score="6" Body="&lt;p&gt;I read your question as: 'Is boosting more vulnerable to overfitting than bagging?'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, you need to understand that &lt;strong&gt;bagging decreases variance&lt;/strong&gt;, while &lt;strong&gt;boosting decreases bias&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, to be noted that under-fitting means that the model has low variance and high bias and vice versa for overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, boosting is more vulnerable to overfitting than bagging.&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="45726" LastEditDate="2018-02-26T20:39:13.677" LastActivityDate="2018-02-26T20:39:13.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28302" PostTypeId="2" ParentId="28296" CreationDate="2018-02-26T05:54:39.620" Score="1" Body="&lt;p&gt;First of all, I looked at the context, it is in 4.3 (Similarity between vectors) in point 4 (Proximity measures) in chapter 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is the squared Euclidean distance different from the Euclidean distance?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, simply stated, yes it is different, the difference being same as the difference between Variance and Standard Deviation. One is a &lt;em&gt;number&lt;/em&gt; and another is &lt;em&gt;square root of that number&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How is the squared Euclidean distance non-metric?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author says- &lt;strong&gt;It is tacitly assumed that the distance function, d(x, y) where x and y are patterns, is a metric....&lt;/strong&gt;. Now look at the definition of metric, he says a metric is one which satisfies the following properties-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Positive reflexivity&lt;/li&gt;&#xA;&lt;li&gt;Symmetry&lt;/li&gt;&#xA;&lt;li&gt;Triangular inequality&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;He also proves that &lt;code&gt;squared euclidean distance&lt;/code&gt; is not a metric by giving  an example. Because it does not satisfy the property 3 of metric definition, squared euclidean distance is not a metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-02-26T05:54:39.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28303" PostTypeId="1" CreationDate="2018-02-26T06:31:46.587" Score="3" ViewCount="755" Body="&lt;p&gt;My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we handle picture issue, manpower is essentially needed to cut raw photo, label on the photos, and scan on the server for fundamental labeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it sounds weird, but i'm just curious if there are any algorithms/system to make a label automatically for supervised learning.&lt;/p&gt;&#xA;" OwnerUserId="46862" LastActivityDate="2024-02-24T04:03:50.490" Title="Is it possible to make a label automatically in supervised learning(Machine Learning)?" Tags="|machine-learning|deep-learning|algorithms|unsupervised-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28304" PostTypeId="1" AcceptedAnswerId="28437" CreationDate="2018-02-26T06:40:05.627" Score="0" ViewCount="2253" Body="&lt;p&gt;I'm trying to find similar data columns (text and numerical) in other tables by using classification in machine learning. What are some good features that can represent a column for my feature space ?&lt;/p&gt;&#xA;" OwnerUserId="46055" LastActivityDate="2018-03-01T04:07:31.350" Title="Finding similar data columns with machine learning" Tags="|machine-learning|classification|algorithms|multiclass-classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28305" PostTypeId="2" ParentId="28303" CreationDate="2018-02-26T07:11:07.037" Score="1" Body="&lt;p&gt;The problem you ask is probably &lt;strong&gt;Semi-supervised learning&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a Semi-supervised learning problem, you have a small amount of labeled data with a large amount of unlabeled data. The goal of this problem is to build a model that has higher accuracy than using labeled data alone.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you have no labeled data at the beginning, then you might have to use some automatic system (e.x: sensors) to determine the label of your data. In &lt;a href=&quot;https://arxiv.org/abs/1603.02199&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper about robotic grasping&lt;/a&gt;, Sergey Levine automatically detects an attempt of grasp as success or failure based on the openness of the grippers (aka fingers) of the robotic arm and the difference of images before and after the attempt. He called this type of problem &lt;strong&gt;Self-supervised learning&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this help.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-26T07:11:07.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28306" PostTypeId="1" AcceptedAnswerId="28308" CreationDate="2018-02-26T08:35:00.863" Score="2" ViewCount="625" Body="&lt;p&gt;I'm very new to machine learning approaches. I'm reading a tutorial for build a predictive model using random forests.&#xA;One of the transformations implemented was transform categorical variables to binary.&#xA;Imagine (short sample):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Field_Desc  Field_Value&#xA;A               32&#xA;A               100&#xA;B               1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then the developer pass this dataset into:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Field_A1    Field_B1    Field_Value&#xA;1               0                32&#xA;1               0                100&#xA;0               1                 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the advantage to make this transformation for Random Forest Prediction? And for K-Means there will have any advantage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24790" LastEditorUserId="28175" LastEditDate="2018-02-26T09:26:21.703" LastActivityDate="2018-02-26T09:37:45.647" Title="Transform Categorical Variables into Numerical" Tags="|machine-learning|neural-network|deep-learning|predictive-modeling|categorical-data|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28307" PostTypeId="2" ParentId="28288" CreationDate="2018-02-26T08:46:58.013" Score="1" Body="&lt;p&gt;You can use &lt;code&gt;numpy.where&lt;/code&gt; documentation can be fouind here :&#xA;&lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.where.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.where.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can define a function to extract your data ID somthing like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def ClusterIndicesNumpy(clustNum, labels_array): &#xA;    return np.where(labels == cluster)[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To get samples from cluster 3 for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ClusterIndicesNumpy(3, km.labels_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-02-26T08:46:58.013" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28308" PostTypeId="2" ParentId="28306" CreationDate="2018-02-26T09:25:32.297" Score="0" Body="&lt;p&gt;Suppose that you want to have &lt;code&gt;k-means&lt;/code&gt; algorithm, in the formulation of average, you have to take the average of each cluster, and then reassign the centers. If you have categorical data, how do you want to take mean? Changing categorical data to numeric data is for translating situations which don't have numerical features to be suited to be used for such algorithms.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-26T09:25:32.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28309" PostTypeId="2" ParentId="28306" CreationDate="2018-02-26T09:37:45.647" Score="0" Body="&lt;p&gt;Most random forest algorithms I have come across don't require that categorical variables are converted as you describe above, generally because of how the tree does its splitting:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Categorical variable with split point x: splitting rule is data point = x / data point != x.&lt;/li&gt;&#xA;&lt;li&gt;Numeric variable with split point x: splitting rule is data point &amp;lt; x / data point &gt;= x.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;That being said, some algorithms have constraints on the cardinality of categorical variables - e.g. &lt;code&gt;randomForest&lt;/code&gt; in R only accepts categorical variables with less than 53 levels.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-02-26T09:37:45.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28310" PostTypeId="1" AcceptedAnswerId="28312" CreationDate="2018-02-26T09:47:35.590" Score="4" ViewCount="1953" Body="&lt;p&gt;I want to use a pre-trained convolutional network for image classification. My base data has resolutions of 500x500px up to 1000x1000px. Pre-trained architectures often expect less (between 255 and 299px in case of Googles Inception network).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly: Would it potentially have a big impact to use higher resolution images? I.e. is it worthwhile investigating it?&#xA;Secondly: Does it make sense and is it possible to use a pre-trained network on low resolution and re-training the last layer/classifier with higher resolutions?&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="28175" LastEditDate="2018-02-26T10:23:32.557" LastActivityDate="2018-02-26T10:23:32.557" Title="Why do pre-trained CNNs use low image resolution?" Tags="|machine-learning|neural-network|deep-learning|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28311" PostTypeId="1" CreationDate="2018-02-26T10:03:47.257" Score="2" ViewCount="123" Body="&lt;p&gt;I have a project which is to predict in next time, which option player will select for betting. This prediction based on his history. For example, the player have time series data, the last options they selected, how much money they spend, did they loose or win...ect. There are many options for one bet, around 300. Based on that data, I want to predict next time which option (or possibility of options) they will choose for betting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know which kind of problem it is. Is it kind of product recommendation, multi-class classification or multi-label classification? Any keyword, or suggestion to solve this kind of problem? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="46869" LastActivityDate="2018-12-27T13:01:15.207" Title="Predict the next player's option for betting based on playing history?" Tags="|time-series|recommender-system|multiclass-classification|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28312" PostTypeId="2" ParentId="28310" CreationDate="2018-02-26T10:22:23.650" Score="5" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Would it potentially have a big impact to use higher resolution images?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, if you increase the input size to your convolutional neural network, the size of each activation map for each layer increases, so you will have more computation. Also if you use same architecture, the number of neurons and consequently the number of parameters, in dense layers increases. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Does it make sense and is it possible to use a pre-trained network on low resolution and re-training the last layer/classifier with higher resolutions?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The answer is no. When you train a network with a special size of input, you reserve variables to hold the weights and middle variables. If you increase the size of input, dense layers will have different size, so their number of wights should vary too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wrap up, for networks with classification tasks, it is appropriate to pass the network small size of images. For other tasks like edge detection where the information of edges can be destroyed by resizing, you have to be careful. In those cases you have to find an appropriate size of the image in order to keep the important information. The small size of the inputs is for reducing number of operations and number of parameters.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-26T10:22:23.650" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28313" PostTypeId="2" ParentId="28296" CreationDate="2018-02-26T11:44:30.410" Score="5" Body="&lt;p&gt;Let $x, y \in \mathbb{R}^n$. The Euclidean distance $d$ &lt;a href=&quot;https://en.wikipedia.org/wiki/Euclidean_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;is defined as&lt;/a&gt;&#xA;$$&#xA;d(x,y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}.&#xA;$$&#xA;The squared Euclidean distance is therefore&#xA;$$&#xA;d(x,y)^2 = \sum_{i=1}^n (x_i - y_i)^2.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;We know that Euclidean distance is a metric.&#xA;Let us check whether squared Euclidean distance is also a metric. I will use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Metric_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;definition from Wikipedia&lt;/a&gt; (Ankit Seth's definition is equivalent).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Non-negativity: $d(x,y)^2 \ge 0$. This one is obvious.&lt;/li&gt;&#xA;&lt;li&gt;Identity of indiscernibles: $d(x,y)^2 = 0$ if and only if $x=y.$ This is true because $d$ is a metric, and $d(x,y)^2 = 0$ if and only if $d(x,y) = 0$.&lt;/li&gt;&#xA;&lt;li&gt;Symmetry: $d(x,y)^2 = d(y,x)^2$. This is again true because $d$ is a metric and therefore $d(x,y) = d(y,x).$ &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;(You can also see that 2. and 3. are true directly from the definition of $d(x,y)^2$).&#xA;Looks fine so far. But:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;4&quot;&gt;&#xA;&lt;li&gt;Triangle inequality: Do we have $d(x,z)^2 \le d(x,y)^2 + d(y, z)^2$ for all $x, y, z \in \mathbb{R}^n$? No. Pick an arbitrary $x \in \mathbb{R}^n \setminus \{0\}$ and set $y = 2x$ and $z=3x$. Then&#xA;$$&#xA;d(x,z)^2 = \sum_{i=1}^n (x_i - 3x_i)^2 = 4 \sum_{i=1}^n x_i^2&#xA;$$ and &#xA;$$d(x,y)^2 + d(y,z)^2 = \sum_{i=1}^n x_i^2 + \sum_{i=1}^n x_i^2 = 2 \sum_{i=1}^n x_i^2.&#xA;$$&#xA;Since $4\sum_{i=1}^n x_i^2 &amp;gt; 2 \sum_{i=1}^n x_i^2$, we have found a counterexample that shows that the triangle inequality does not hold.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;What does it mean that squared Euclidean distance gives the same ranking as Euclidean distance?&#xA;Suppose we have $x, y, z$ such that $d(x,y) &amp;lt; d(x,z)$.&#xA;Then $d(x,y)^2 &amp;lt; d(x,z)^2$ as well: It ranks points in the same way as Euclidean distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is good to know. For instance, this tells us that the &lt;a href=&quot;https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;$k$-nearest neighbors classifiers&lt;/a&gt; gives the exact same results for squared Euclidean distance and Euclidean distance.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-02-26T13:31:57.177" LastActivityDate="2018-02-26T13:31:57.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28315" PostTypeId="2" ParentId="28311" CreationDate="2018-02-26T12:58:03.870" Score="1" Body="&lt;p&gt;this is a multi class classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are many approaches you can use to solve this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can first try Gradient Boosted Trees with libraries like xgboost or lightgbm. Both of those can be used in R or Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also use neural networks for multi class classification problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there are many ways to solve a multi classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can turn it into a multi binary classification problem. Since you have 300 options, you can train 300 binary classification algorithms on your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try K nearest neighbours.&lt;/p&gt;&#xA;" OwnerUserId="38582" LastActivityDate="2018-02-26T12:58:03.870" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28316" PostTypeId="2" ParentId="27664" CreationDate="2018-02-26T13:36:44.677" Score="1" Body="&lt;p&gt;@DPascal Here is something you could definitely try doing:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Using the features you generated, you could add a label to each of the column for these features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You could generate these feature value for different time slices.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Once that is done, you can then run something as simple as Random Forest Classifier on this data. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-02-26T13:36:44.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28317" PostTypeId="1" CreationDate="2018-02-26T14:11:35.903" Score="4" ViewCount="8564" Body="&lt;p&gt;If I have two classifiers for example Neural network and support vector machine , now I want to know what would be the best way to identify which is good classifier , should it be based on classification accuracy or by analyzing confusion matrix or average precision score, f1 score etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen in most of the papers related to image classification that they only take classification accuracy as the parameter to compare different models. &lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="28175" LastEditDate="2018-02-26T14:43:00.713" LastActivityDate="2019-01-08T09:19:52.480" Title="What is the best way to compare various classification models?" Tags="|machine-learning|neural-network|deep-learning|classification|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28319" PostTypeId="1" CreationDate="2018-02-26T14:41:04.890" Score="0" ViewCount="441" Body="&lt;p&gt;I have dataset of 36 folders 1 image each(total 36 images) the dataset is too small but these are character images which i want to train my &lt;code&gt;val_acc= 0.0229&lt;/code&gt; and &lt;code&gt;y_pred=model.predict(x_test)&lt;/code&gt; always predict &lt;code&gt;class=4&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How I can improve this accuracy for such a small dataset? Why does it always predict class 4?&lt;/p&gt;&#xA;" OwnerUserId="46880" LastEditorUserId="46880" LastEditDate="2018-02-26T14:52:54.413" LastActivityDate="2018-02-26T15:10:56.040" Title="Convolution neural network: small dataset affecting accuracy" Tags="|python|deep-learning|keras|tensorflow|convolution|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28320" PostTypeId="2" ParentId="28319" CreationDate="2018-02-26T15:10:56.040" Score="1" Body="&lt;p&gt;If I understand the question correctly, you have 36 class and 1 sample/class. &#xA;Then I believe it's impossible to use Machine Learning for this dataset to predict the label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reason: When you split train/test set, there will some classes that only appear in test set and not train set. The model doesn't learn about them at all. A major assumption in Machine Learning problems is that the train set and test set must have the same statistical distribution.&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-02-26T15:10:56.040" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28324" PostTypeId="2" ParentId="13011" CreationDate="2018-02-26T16:42:49.363" Score="2" Body="&lt;p&gt;@Emre's response and @elias-strehle's implementation are correct. I have made a similar implementation using an elegant (imho) hashing function I borrowed from &lt;a href=&quot;https://twitter.com/fermatslibrary/status/958700402647674880&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tweet&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The class hashes each word to the product of all it's letters, in which each letter is mapped to a prime number. &lt;em&gt;Products of prime numbers only collide when using the exact same numbers.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is quite fast (as requested), around 200k anagram lookups per second (on my machine)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://github.com/sbalen/anagrammer/blob/master/Anagrammer.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for a full example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Anagrammer:&#xA;&#xA;    alphabet = {&quot;a&quot;:2,&quot;b&quot;:3,&quot;c&quot;:5,&quot;d&quot;:7,&quot;e&quot;:11,&quot;f&quot;:13,&quot;g&quot;:17,&quot;h&quot;:19,&quot;i&quot;:23,&quot;j&quot;:29,&quot;k&quot;:31,&quot;l&quot;:37,&quot;m&quot;:41,&quot;n&quot;:43,&quot;o&quot;:47,&quot;p&quot;:53,&quot;q&quot;:59,&quot;r&quot;:61,&quot;s&quot;:67,&quot;t&quot;:71,&quot;u&quot;:73,&quot;v&quot;:79,&quot;w&quot;:83,&quot;x&quot;:89,&quot;y&quot;:97,&quot;z&quot;:101}&#xA;&#xA;    def __init__(self, corpus):&#xA;        self.index = {}&#xA;        self.createIndex(corpus)&#xA;&#xA;    def createAnagramIndexNumber(self, word):&#xA;        index = 1&#xA;        for x in list(word):&#xA;            index *= self.alphabet[x]&#xA;        return index&#xA;&#xA;    def createIndex(self, corpus):&#xA;        for word in corpus:&#xA;            self.index.setdefault(self.createAnagramIndexNumber(word),set())&#xA;            self.index[self.createAnagramIndexNumber(word)].add(word);&#xA;&#xA;    def getAnagrams(self, word):&#xA;        return self.index[self.createAnagramIndex(word)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-26T16:42:49.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28326" PostTypeId="1" CreationDate="2018-02-26T17:36:19.817" Score="1" ViewCount="20" Body="&lt;p&gt;Many of you have probably seen the turtle from LabSix that gets mistaken for a rifle in Google's InceptionV3 image classifier. I read &lt;a href=&quot;https://arxiv.org/pdf/1707.07397.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the paper&lt;/a&gt; and I understand how they apply EOT to 2d images and on the individual pixel values, but I am still unsure how they implement the EOT algorithm to the 3d model. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are they using EOT to perturb the individual coordinates in the 3d model's mesh? Or are they perturbing images of a turtle and then printing the turtle from the images?&lt;/li&gt;&#xA;&lt;li&gt;How do they check the InceptionV3 output iteratively without having to 3d print the object each time and check the probabilities given?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any examples that someone can point to would also be very helpful.&lt;/p&gt;&#xA;" OwnerUserId="46886" LastActivityDate="2018-02-26T17:36:19.817" Title="How to apply EOT algorithm to 3d model /xpost from AI beta" Tags="|neural-network|image-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28328" PostTypeId="1" CreationDate="2018-02-26T19:20:41.930" Score="22" ViewCount="14049" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;noreferrer&quot;&gt;Multicollinearity&lt;/a&gt; is a problem for linear regression because the results become unstable / depend too much on single elements (&lt;a href=&quot;http://blog.datadive.net/selecting-good-features-part-ii-linear-models-and-regularization/&quot; rel=&quot;noreferrer&quot;&gt;source&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Also, the inverse of $X^TX$ doesn't exist so the standard OLS estimator does not exist ... I have no idea how, but &lt;a href=&quot;https://gist.github.com/MartinThoma/6e7714fd4774b5185ac7c3ace9a5b780&quot; rel=&quot;noreferrer&quot;&gt;sklearn deals with it just fine&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is (perfect) multicollinearity also a problem for neural networks?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2021-12-13T14:09:04.457" Title="How does multicollinearity affect neural networks?" Tags="|neural-network|correlation|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28329" PostTypeId="2" ParentId="28328" CreationDate="2018-02-26T20:02:04.937" Score="3" Body="&lt;p&gt;Multi colinearity affects the learning of Artificial Neural network.  Since the information in the dependent variable is very less compared to the other variables, the neural network will take more time to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In packages like sklearn, the dependent variables are identified and omitted from the calculation.  I have used the lm function in R and it marks the coefficient of the dependent variable with NA.  one can remove the variable from the calculation and still the coefficients are going to be same.  In these cases, the rank of the x matrix will be less than the number of columns.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though there are no inverse exists for xTx, most of the packages will not calculate the inverse directly, but they will calculate the pseudo inverse.  &lt;/p&gt;&#xA;" OwnerUserId="46891" LastActivityDate="2018-02-26T20:02:04.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28331" PostTypeId="1" CreationDate="2018-02-26T20:29:22.630" Score="26" ViewCount="24773" Body="&lt;p&gt;I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this?&lt;/p&gt;&#xA;" OwnerUserId="46893" LastEditorUserId="29575" LastEditDate="2018-02-26T20:50:08.780" LastActivityDate="2018-02-28T12:02:06.963" Title="Different Test Set and Training Set Distribution" Tags="|preprocessing|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28334" PostTypeId="1" AcceptedAnswerId="28336" CreationDate="2018-02-27T00:47:10.343" Score="0" ViewCount="122" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/pdf/1511.08400.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper (page 1 abstract)&lt;/a&gt; which considers regularization technique, the author used the word &quot;norm&quot; - what does it stand for?&#xA;Is it related to Batch Normalization / L1 or L2 Normalization? &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing&#xA;  the squared distance between successive hidden states’ norms&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Googling &quot;what is a norm Neural Net&quot; doesn't provide any links&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-02-27T01:01:57.540" LastActivityDate="2018-02-27T04:38:20.170" Title="What does &quot;Norm&quot; term mean?" Tags="|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28336" PostTypeId="2" ParentId="28334" CreationDate="2018-02-27T04:38:20.170" Score="2" Body="&lt;p&gt;A norm is a concept in linear algebra which assigns a &lt;em&gt;size&lt;/em&gt; to a vector. Many different norms exist you can read up on their many uses &lt;a href=&quot;https://en.wikipedia.org/wiki/Norm_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this paper, they are training a recurrent neural network. These are used for time series data. The assumption is a hidden state should be similar for successive time inputs $t$ and $t-1$. Thus, the author proposes an additional constraint based on the relative &lt;em&gt;size&lt;/em&gt; of the successive hidden state vectors as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$||h_{t}||_2 - ||h_{t-1}||_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The more different the norm of these successive states, the greater the cost will be.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-02-27T04:38:20.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28337" PostTypeId="1" AcceptedAnswerId="28341" CreationDate="2018-02-27T04:53:46.433" Score="0" ViewCount="1959" Body="&lt;p&gt;The objective/loss function of K-Means algorithm is to minimize the sum of squared distances, written in a math form, it looks like this:&#xA;$$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we have different distance metric, for instance, cosine (I realize there's a conversion between cosine and Euclidean but let's forget it for now), manhattan etc, does it mean we will have a different loss function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, the traditional K-Means based on expectation maximization won't be working right? Because for every iteration we revise centroids, normally, by computing the average. However, for some of the metrics other than Euclidean, average might not be a legit representation for the center.&lt;/p&gt;&#xA;" OwnerUserId="41487" LastActivityDate="2018-02-28T01:13:32.473" Title="Does K-Means' objective function imply distance metric is Euclidean" Tags="|machine-learning|clustering|k-means|expectation-maximization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28338" PostTypeId="2" ParentId="28331" CreationDate="2018-02-27T05:16:09.177" Score="-2" Body="&lt;p&gt;There is a good package in python (scikit learn)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can  subsample your observations from training set using this package.&lt;/p&gt;&#xA;" OwnerUserId="46433" LastActivityDate="2018-02-27T05:16:09.177" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28339" PostTypeId="1" AcceptedAnswerId="28369" CreationDate="2018-02-27T05:58:54.620" Score="9" ViewCount="9455" Body="&lt;p&gt;I am trying to understand an article &lt;a href=&quot;http://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Backpropagation In Convolutional Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I can not wrap my head around that diagram: &lt;a href=&quot;https://i.stack.imgur.com/ZSWj6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZSWj6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first layer has &lt;strong&gt;3&lt;/strong&gt; feature maps with dimensions &lt;strong&gt;32x32&lt;/strong&gt;. The second layer has &lt;strong&gt;32&lt;/strong&gt; feature maps with dimensions &lt;strong&gt;18x18&lt;/strong&gt;. How is that even possible ? If a convolution with a kernel &lt;strong&gt;5x5&lt;/strong&gt; applied for &lt;strong&gt;32x32&lt;/strong&gt; input, the dimension of the output should be $(32-5+1)$ by $(32-5+1)$ = $28$ by $28$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, if the first layer has only 3 feature maps, the second layer should have multiple of &lt;strong&gt;3&lt;/strong&gt; feature maps, but &lt;strong&gt;32&lt;/strong&gt; is not multiple of &lt;strong&gt;3&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, why is the size of the third layer is &lt;strong&gt;10x10&lt;/strong&gt; ? Should it be &lt;strong&gt;9x9&lt;/strong&gt; instead ? The dimension of the previouse layer is &lt;strong&gt;18x18&lt;/strong&gt;, so &lt;strong&gt;2x2&lt;/strong&gt; max pooling should reduce it to &lt;strong&gt;9x9&lt;/strong&gt;, not &lt;strong&gt;10x10&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45821" LastEditorUserId="29169" LastEditDate="2021-02-13T16:38:06.780" LastActivityDate="2021-05-08T05:57:52.240" Title="Convolutional Neural Networks layer sizes" Tags="|neural-network|cnn|convolutional-neural-network|convolution|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28340" PostTypeId="1" CreationDate="2018-02-27T07:32:10.793" Score="1" ViewCount="89" Body="&lt;p&gt;How can I perform &lt;a href=&quot;https://en.wikipedia.org/wiki/Conceptual_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;conceptual clustering&lt;/a&gt; in sklearn? My use case is that I have English Wikipedia articles that I'm doing unsupervised learning on (tfidf -&gt; truncated svd -&gt; l2 normalize), and I'd like to create a hierarchy for them such that the nodes at the top are the most general articles (e.g. Programming Languages -&gt; Functional Languages -&gt; Haskell). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using &lt;code&gt;hierarchy.linkage&lt;/code&gt;, but it seems that the algorithm uses &lt;code&gt;n^2&lt;/code&gt; space, and I ran out of memory. I also tried using a &lt;code&gt;KDTree&lt;/code&gt; on the &lt;code&gt;l2&lt;/code&gt; normalized vectors, and then setting each node to be the normalized sum of its children recursively, but this did not produce desirable results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the right way to perform conceptual clustering with cosine similarity in scikit-learn without using quadratic space?&lt;/p&gt;&#xA;" OwnerUserId="26480" LastActivityDate="2024-02-10T22:05:35.673" Title="Conceptual clustering with sklearn?" Tags="|scikit-learn|clustering|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28341" PostTypeId="2" ParentId="28337" CreationDate="2018-02-27T07:46:08.287" Score="2" Body="&lt;p&gt;You're right and you're wrong.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The objective/loss function of K-Means algorithm is to minimize the sum of squared distances&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, absolutely.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;written in a math form, it looks like this:&#xA;  $$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Err... sort of. This is definitely the most popular formulation of kmeans, but a more appropriate formulation would be: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$J(X,Z) = min\ \sum_{z\in centroids}\sum_{x \in data} distance(x,z)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your formulation, you're concretely defining distance as euclidean distance, i.e. the L2 norm. But you can swap out L2 for any distance kernel and apply kmeans. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The caveat here is that kmeans is a &quot;hill climbing&quot; algorithm, which means each iteration should always be at least as good as the previous iteration, and so it must be the case that this improvement will be true for both the E and M steps. For most common distance metrics (L1, L2, cosine, hamming...) this is the case and you're good to go, but there are infinite possible distance metrics and if we're going to be technical about it, the probability that a random distance metric will satisfy this criterion is almost surely 0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to circle back to your question: does the objective function &lt;em&gt;as you formulated it&lt;/em&gt; imply the distance metric is euclidean? Yes. But &lt;em&gt;does kmeans only apply to euclidean space&lt;/em&gt;? No, absolutely not. Use whatever distance metric you want and throw the EM algorithm at it and bam: you've got yourself a non-euclidean kmeans. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, when people say &quot;kmeans&quot;, they're talking about euclidean kmeans. But Kmeans is super easy to modify with a different distance metric and I think only the most pedantic people would argue that you shouldn't call it &quot;kmeans&quot; after such a modification. Although it's generally always described with the objective you posted (which, yes, does imply euclidean distance), you can really drop in pretty much any useful distance metric, throw EM at it, and it'll work. Some people might call it &quot;___ kmeans&quot; depending on the distance metric, but really it's all kmeans.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think part of the reason kmeans often isn't described this way isn't formulated like this is because its often compared with gaussian mixture models (GMM). With a euclidean norm, kmeans is equivalent to a GMM with diagonal covariance (and hard cluster assignment), and is consequently often described as an easy way to fit a GMM with spherical clusters. But this comparison fails if we use a different distance metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you check out the &lt;a href=&quot;https://stats.stackexchange.com/questions/81481/why-does-k-means-clustering-algorithm-use-only-euclidean-distance-metric&quot;&gt;CrossValidated discussion&lt;/a&gt; on this topic. Full disclosure: the highest voted answers disagree with me, but as you probably guessed I think they're being fairly pedantic.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-02-27T08:26:38.867" LastActivityDate="2018-02-27T08:26:38.867" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28342" PostTypeId="1" AcceptedAnswerId="28348" CreationDate="2018-02-27T08:48:42.567" Score="2" ViewCount="433" Body="&lt;p&gt;In the Coursera course &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;machine learning&lt;/a&gt; in the section on Multivariate Linear Regression, Andrew Ng provides the following tips on gradient descent:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use Feature Scaling to converge quicker&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Get feature into an approx -1 &amp;lt; x &amp;lt; 1 range&lt;/li&gt;&#xA;&lt;li&gt;Mean normalization&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Andrew Ng also provides some other tips:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Plot cost vs iterations&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;to ensure cost decreases on every iteration (try smaller alpha)&lt;/li&gt;&#xA;&lt;li&gt;to identify if convergence is too slow (try larger alpha)&lt;/li&gt;&#xA;&lt;li&gt;to identify approximately the number of iterations to converge&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Are these tips applicable to all problems involving gradient descent using different machine/deep learning algorithms or just to Multivariate Linear Regression?&lt;/p&gt;&#xA;" OwnerUserId="13237" LastActivityDate="2018-02-28T08:21:34.523" Title="Should I consider feature scaling for all gradient descent based algorithms?" Tags="|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28343" PostTypeId="2" ParentId="28339" CreationDate="2018-02-27T09:18:23.120" Score="1" Body="&lt;p&gt;It could be a case of padding in combination with convolution strides: if you would pad the first layer with 2 zeroes on either side and use a stride of 2, you would end up with an 18 * 18 * x. The 3 channels on the input are most probably RG&amp;amp;B, which are fairly commonly scaled up to 32 feature maps.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-02-27T09:18:23.120" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28344" PostTypeId="1" CreationDate="2018-02-27T09:23:20.580" Score="1" ViewCount="298" Body="&lt;p&gt;I am analysing a bunch of tweets and I want to understand which political party the authors support.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Mathematica but I am thinking to re-write my code in Python. If you have any suggestion, do not esitate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let say, for example that I have five parties labelled by a number: &lt;code&gt;Party1, Party2,...,Party5&lt;/code&gt; and I provide a list of tweets which are for sure associated to the parties. This list is roughly &lt;code&gt;trainingList = {tweet1-&amp;gt; &quot;Party1&quot;,tweet2-&amp;gt;&quot;Party2&quot;,...,tweet5-&amp;gt;&quot;Party5&quot;,tweetK-&amp;gt;&quot;Neutral&quot;}&lt;/code&gt;. Moreover, I added a class &lt;code&gt;&quot;Neutral&quot;&lt;/code&gt; because some tweets are not classifiable. Actually, I provide more than one tweet par party. In fact, the &lt;code&gt;trainingList&lt;/code&gt; is much longer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that I remove stop-words from the tweets and I apply a stemming-words algorithm to simplify the expressions. Then, every tweets is a list of potentially-significant words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I create the classifier function, providing a &lt;code&gt;prior&lt;/code&gt; (the explicit for of the prior is not important)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c = Classify[trainingList, ClassPriors-&amp;gt; prior]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, I want to do some checks. I take a bunch of tweets belonging to other party-supporters and try &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c[{word1,word2,word3,word4,word5,word7},&quot;Probabilities&quot;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives me a list of probabilities $p_i$. For example, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;| 'Party1' -&amp;gt; p1, 'Party2' -&amp;gt; p2, 'Party3' -&amp;gt; p3, 'Party4' -&amp;gt; p4, 'Party5' -&amp;gt; p5, 'Neutral' -&amp;gt; p6 |&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question1&lt;/strong&gt; Is there a way to understand how the algorithm associate these probabilites? More specifically, how each word is associated to a party&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question2&lt;/strong&gt; Is there a way to see which are the most frequent n-grams of words associated to a given party (class)?&lt;/p&gt;&#xA;" OwnerUserId="46920" LastActivityDate="2021-04-29T11:09:25.923" Title="Using Machine Learning techniques for text-analysis" Tags="|machine-learning|neural-network|data-mining|machine-learning-model|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28345" PostTypeId="2" ParentId="28344" CreationDate="2018-02-27T09:39:59.527" Score="0" Body="&lt;p&gt;Take a few words you know are linked with e.g republicans and with democrats. &#xA;Extract their word embedding. That is the vector representation of the word which will be high dimensional.&#xA;This will assume you have trained word embedding, either pretrained, or even better trained them yourself on your data. The word embeddings could have been trained/extracted through e.g LSTM training or matrix factorisations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the word embeddings, do dimension reduction(PCA as an example) to 2d. Plot! Those who are close together will be &quot;similar&quot; for the objective they are trained for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pretrained word embeddings exist to download. However, they might not be the optimal embeddings for your objective, but better than nothing still.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2018-02-27T09:39:59.527" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28346" PostTypeId="2" ParentId="28331" CreationDate="2018-02-27T09:43:46.273" Score="25" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Great question, this is what is known in Machine Learning paradigm as either&#xA;  &quot;Covariate Shift&quot;, or &quot;Model Drift&quot; or &quot;Nonstationarity&quot; and so on.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking &lt;a href=&quot;https://blog.bigml.com/2014/01/03/simple-machine-learning-to-detect-covariate-shift/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/07/covariate-shift-the-hidden-problem-of-real-world-data-science/&quot; rel=&quot;noreferrer&quot;&gt;post&lt;/a&gt;! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;This &lt;a href=&quot;https://maxhalford.github.io/blog/subsampling-1/&quot; rel=&quot;noreferrer&quot;&gt;blog post&lt;/a&gt; gives you a simple way to handle the subsampling of training data with code provided in Python!&lt;/li&gt;&#xA;&lt;li&gt;Check this &lt;a href=&quot;https://f1000research.com/articles/5-597/v3&quot; rel=&quot;noreferrer&quot;&gt;research paper&lt;/a&gt;. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using &lt;strong&gt;Kullback-Leibler Importance Estimation Procedure&lt;/strong&gt; base on &quot;&lt;em&gt;Kullback-Leibler divergence&lt;/em&gt;&quot; theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;QUICK update (a good solution)&lt;/strong&gt;: I found a &lt;a href=&quot;https://github.com/srome/pykliep&quot; rel=&quot;noreferrer&quot;&gt;Python implementation of KLIEP algorithm&lt;/a&gt; of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-02-28T12:02:06.963" LastActivityDate="2018-02-28T12:02:06.963" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28347" PostTypeId="1" CreationDate="2018-02-27T09:52:33.837" Score="3" ViewCount="204" Body="&lt;p&gt;In the &lt;a href=&quot;https://arxiv.org/pdf/1512.03385.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Residual learning&lt;/a&gt; paper by He et al., there are a number of plots of training/test error vs. backprop iteration. I've only ever seen &quot;smooth&quot; curves on these plots, while in this paper's application, there are sudden jumps in improvement. In this figure (Fig. 4a in the paper linked above), they are at around 15e4 and 30e4 iterations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GWVAY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GWVAY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happened here? My intuition would say that the backpropagation hit a plateau with a gradient of close to zero, and then &lt;em&gt;very suddenly&lt;/em&gt; found a path steeply downward - but is that a realistic shape for a cost function?&lt;/p&gt;&#xA;" OwnerUserId="38133" LastEditorUserId="29169" LastEditDate="2021-02-13T16:37:54.453" LastActivityDate="2024-03-20T13:07:40.957" Title="How to interpret sudden jumps of improvement in training error vs. iteration?" Tags="|neural-network|gradient-descent|convolutional-neural-network|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28348" PostTypeId="2" ParentId="28342" CreationDate="2018-02-27T09:56:56.160" Score="2" Body="&lt;p&gt;About the tips regarding plot cost vs. iteration, they are generally applicable to gradient descent approaches, including deep learning, where hyperparameter tuning (e.g. learning rate) is crucially important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the proper input scaling, it is not only related to the machine learning approach, but to the specific problem under consideration. Sometimes machine learning algorithms rely on distances to compute the similarity between individuals. Scaling changes some of these distances. In these cases, the resulting distance after scaling should be assessed to check whether it is more appropriate than without scaling. &lt;a href=&quot;https://stats.stackexchange.com/q/89809/40048&quot;&gt;Here&lt;/a&gt; you can find examples for clustering. &lt;a href=&quot;https://stats.stackexchange.com/q/244507/40048&quot;&gt;For some machine learning algorithms&lt;/a&gt;, you need standardized features, e.g. regularized linear/logistic regression. For most optimization-based machine learning algorithms, it makes sense to have feature scaling. On the other hand, there are problems where scaling doesn't even make sense (e.g. discrete input problems, like token-based natural language processing).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-02-28T08:21:34.523" LastActivityDate="2018-02-28T08:21:34.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28349" PostTypeId="2" ParentId="28249" CreationDate="2018-02-27T10:28:29.947" Score="1" Body="&lt;p&gt;I would still go with Deep Neural Networks because they perform very well for segmentation tasks. The only thing that you have to be sure about is that you select the perfect patch size and your DNN should be deep enough to learn all the features.I myself have been doing research on image segmentation at the start I was having lots of issues in segmentation using deep learning but once I got all the parameters right then everything went well.&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-02-27T10:28:29.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28350" PostTypeId="2" ParentId="28249" CreationDate="2018-02-27T10:53:18.787" Score="2" Body="&lt;p&gt;You have an object vs background task, i.e. two classes and instance (or m.b. class segmentation problem). If you have a labelled dataset, you can finetune one of the publicly available networks (e.g. FCN, MaskRCNN) to it to produce an image mask (i.e. an array size of the input image where every pixel is labelled as either object or background). So yeah, in 2018 feature engineering can't beat a (fully convolutional, 140M parameter, 16-layer) ConvNN.    &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2018-02-27T10:53:18.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28351" PostTypeId="2" ParentId="16105" CreationDate="2018-02-27T11:27:41.783" Score="1" Body="&lt;p&gt;I would suggest you use Python with Ski-Image for Image Related operation. For Machine Learning typically for Deep Learning, you should use &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; with theano/Tensorflow backend with GPU capabilities whichever suits you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to generate masked images, you should use data augmentation.&lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Augmentation Documentation for Keras&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-02-27T11:27:41.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28352" PostTypeId="2" ParentId="28293" CreationDate="2018-02-27T11:52:44.803" Score="1" Body="&lt;p&gt;Pretty interesting question!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First of all&lt;/strong&gt; have a look at my edit as your question was a bit unclear according to the standard terminology. you have a set of Time-Series and you want to detect the outliers (abnormalities).&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Your approach is pretty clear and logic and shows understanding of the problem and solution. The point is about the way you chose to apply it.&lt;/li&gt;&#xA;&lt;li&gt;K-means is not the best way. I would like to point out that choice of 2 clusters are very smart here as you hope the clusters get formed based on a normal/abnormal structure. It just does not work well in practice if the features you extracted from your time-series are not inhibiting the abnormal behavior.&lt;/li&gt;&#xA;&lt;li&gt;I assume Embedding algorithms are the right way to do so. Most probably if you apply a simple PCA you will see abnormal time-series somewhere further than others. Bellow I write the code. Try it and drop me a line if it didn't work so I go for more sophisticated solutions (e.g. you may construct a phase-space and see your data there or monitor the recurrence of time-series, etc.) &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.decomposition import PCA&#xA;import matplotlib.pyplot as plt&#xA;&#xA;pca = PCA(n_components=2)&#xA;X_new = pca.fit_transform(X)&#xA;plt.figure(figsize=(10,10))&#xA;plt.plot(X_new[:,0],X_new[:,1],&quot;*&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;where X is a 10xN matrix in which each row is one time-series.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may choose more components for PCA and check different PCs against each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways the problem is not that difficult and if it did not work I will update my answer with another solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps and good luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-02-27T11:52:44.803" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="28353" PostTypeId="1" CreationDate="2018-02-27T12:28:35.403" Score="9" ViewCount="24563" Body="&lt;p&gt;Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to always drop the first column after performing One Hot encoding, regardless of the algorithm of choice ?&lt;/p&gt;&#xA;" OwnerUserId="46374" LastEditorUserId="46374" LastEditDate="2018-03-01T12:07:01.730" LastActivityDate="2021-04-01T03:54:40.643" Title="Always drop the first column after performing One Hot Encoding?" Tags="|machine-learning|dataset|feature-selection|categorical-data|encoding|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28354" PostTypeId="2" ParentId="28353" CreationDate="2018-02-27T12:39:19.380" Score="4" Body="&lt;p&gt;This question, in a slightly different form, was discussed &lt;a href=&quot;https://datascience.stackexchange.com/questions/27957/why-do-we-need-to-discard-one-dummy-variable/27993#27993&quot;&gt;herein&lt;/a&gt; earlier.&lt;/p&gt;&#xA;&lt;p&gt;You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you want to drop at the very beginning?&lt;/p&gt;&#xA;&lt;p&gt;In fact in &lt;code&gt;pandas.get_dummies&lt;/code&gt; there is a parameter i.e. &lt;code&gt;drop_first&lt;/code&gt; allows you whether to keep or remove the reference (whether to keep k or k-1 dummies out of k categorical levels). Please note &lt;code&gt;drop_first = False&lt;/code&gt; meaning that the reference is not dropped and k dummies created out of k categorical levels! You set &lt;code&gt;drop_first = True&lt;/code&gt;, then it will drop the reference column after encoding.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="-1" LastEditDate="2021-02-09T07:12:33.550" LastActivityDate="2021-02-09T07:12:33.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28356" PostTypeId="2" ParentId="27420" CreationDate="2018-02-27T13:43:37.047" Score="3" Body="&lt;p&gt;You might find the following libraries helpful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Python: &lt;a href=&quot;https://github.com/GGiecold/Cluster_Ensembles&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cluster Ensembles&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;R: &lt;a href=&quot;https://github.com/AlineTalhouk/diceR&quot; rel=&quot;nofollow noreferrer&quot;&gt;diceR&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46928" LastEditorUserId="29575" LastEditDate="2018-02-27T18:35:02.167" LastActivityDate="2018-02-27T18:35:02.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28360" PostTypeId="1" AcceptedAnswerId="28367" CreationDate="2018-02-27T15:48:53.610" Score="7" ViewCount="2682" Body="&lt;p&gt;I was very recently asked in a job interview about solutions to fix an imbalance of classes in the training dataset. Let's focus on a binary classification case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I offered two solutions: oversampling the minority class by feeding the classifier balanced batches of data, or partitioning the abundant class such as to train many classifiers with a balanced training set, a unique subset of the abundant and the same set of the minority. The interviewers noded, but I was later cut off and one of the knowledge gaps they mentioned was this answer. I know now that I could have discussed changed the metric..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the question that  pops in my mind now is: is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion? The rule of thumb of machine learning seems to be that the training set needs to be as similar as possible to the testing for best prediction performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Isn't just in the cases with have no idea (no prior) about the distribution of the testing that we need to balance the classes? Maybe I should have raised this point in the interview.. &lt;/p&gt;&#xA;" OwnerUserId="39830" LastActivityDate="2018-02-27T17:17:38.943" Title="How to fix class imbalance in training sample?" Tags="|training|class-imbalance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28361" PostTypeId="1" CreationDate="2018-02-27T16:08:16.603" Score="1" ViewCount="642" Body="&lt;p&gt;As a newbie, I am interested what the major drawbacks of traditional clustering algorithms are. The purpose is to understand for which class of data or in which scenarios I shouldn't use traditional clustering methods (e.g. K-means)&lt;/p&gt;&#xA;" OwnerUserId="46935" LastActivityDate="2021-03-03T17:05:30.633" Title="What are the drawbacks of traditional clustering algorithms?" Tags="|clustering|" AnswerCount="1" CommentCount="1" ClosedDate="2018-03-05T06:30:24.387" ContentLicense="CC BY-SA 3.0" />
  <row Id="28362" PostTypeId="2" ParentId="28361" CreationDate="2018-02-27T16:23:24.257" Score="0" Body="&lt;p&gt;Check out this great answer for K-means in particular:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/133656/how-to-understand-the-drawbacks-of-k-means&quot;&gt;How to understand the drawbacks of K-means&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;My favorite resource for comparing the strengths and weaknesses of clustering algos is from sklearn's docs:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Comparing different clustering algorithms on toy datasets&#xA;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38271" LastEditorUserId="85045" LastEditDate="2021-03-03T17:05:30.633" LastActivityDate="2021-03-03T17:05:30.633" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28364" PostTypeId="2" ParentId="28360" CreationDate="2018-02-27T16:37:42.280" Score="1" Body="&lt;p&gt;To your question &quot;is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion?&quot;, the answer is &quot;it depends on the cost of misclassifications&quot;, but normally it is certainly a problem, because your classifier will be biased to classify an individual into the over-represented class regardless of the values of its features because, &lt;em&gt;in average&lt;/em&gt;, doing so increased the chances of correct classification during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different &lt;a href=&quot;https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;oversampling and undersampling strategies&lt;/a&gt;. Maybe the most popular one is &lt;a href=&quot;https://www.jair.org/media/953/live-953-2037-jair.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt; (Synthetic Minority Over-sampling Technique), which creates synthetic training data based on k-nearest neighbors.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-02-27T17:13:34.517" LastActivityDate="2018-02-27T17:13:34.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28366" PostTypeId="1" AcceptedAnswerId="28377" CreationDate="2018-02-27T17:01:52.420" Score="1" ViewCount="294" Body="&lt;p&gt;In what way would exploratory data analysis aid in feature selection, other than to preprocess the data ? Say, if a bivariate analysis was conducted for each predictor variable w.r.t. the target variable, in what way would this help with feature selection, if possible ?&lt;/p&gt;&#xA;" OwnerUserId="46374" LastActivityDate="2018-03-01T10:49:11.617" Title="Exploratory Data Analysis and selecting good predictor variables ?" Tags="|feature-selection|feature-engineering|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28367" PostTypeId="2" ParentId="28360" CreationDate="2018-02-27T17:11:07.273" Score="4" Body="&lt;p&gt;Actually what they mentioned is right. The idea of oversampling is right and is one of, in general, Resampling methods to cope with such problem. Resampling can be done through oversampling the minorities or undersampling the majorities. You may have a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/27758/8878&quot;&gt;SMOTE&lt;/a&gt; algorithm as a well-stablished method of resampling.&lt;/p&gt;&#xA;&lt;p&gt;But about your main question: No it's not only about the consistency of distributions between test and train set. It is a bit more.&lt;/p&gt;&#xA;&lt;p&gt;As you mentioned about metrics, Just imagine accuracy score. If I have a binary classification problem with 2 classes, one 90% of the population and the other class 10%, then with no need of Machine Learning I can say my prediction is always the majority class and I have 90% accuracy! So it just does not work regardless of the consistency between train-test distributions. In such cases you may pay more attention to Precision and Recall. Usually you would like to have a classifier which minimizes the mean (usually harmonic mean) of Precision and Recall i.e. the error rate is where FP and FN are fairly small and close to each other.&lt;/p&gt;&#xA;&lt;p&gt;Harmonic mean is used instead of arithmetic mean because it supports the condition that those errors are as equal as possible. For instance if Precision is &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt; and Recall is &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; the arithmetic mean is &lt;span class=&quot;math-container&quot;&gt;$0.5$&lt;/span&gt; which is not illustrating the reality inside the results. But harmonic mean is &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt; which says however one of the metrics is good the other one is super bad so in general the result is not good.&lt;/p&gt;&#xA;&lt;p&gt;But there are situations in practice in which you DO NOT want to keep the errors equal. Why? See the example bellow:&lt;/p&gt;&#xA;&lt;h2&gt;An Additional Point&lt;/h2&gt;&#xA;&lt;p&gt;This is not exactly about your question but may help understanding.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/neG3q.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/neG3q.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In practice you may sacrifice an error to optimize the other one. For instance diagnosis of HIV might be a case (I am just mocking-up an example). It is highly imbalanced classification as, of course, the number of people with no HIV is dramatically higher than the ones who are carrier. Now let's look at errors:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;False Positive: Person does not have HIV but test says they have.&lt;/p&gt;&#xA;&lt;p&gt;False Negative: Person does have HIV but test says they don't.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;If we assume that wrongly telling someone that &lt;em&gt;he got HIV&lt;/em&gt; simply leads to another test, we may take much care about not wrongly say someone &lt;em&gt;he is not a carrier&lt;/em&gt; as it may result in propagating the virus. Here your algorithm should be sensitive on False Negative and punishes it much more than False Positive i.e. according to the figure above, you may end up with higher rate of False Positive.&lt;/p&gt;&#xA;&lt;p&gt;Same happens when you want to automatically recognize people faces with a camera to let them enter in an ultra-security site. You don't mind if the door is not opened once for someone who has permission (False Negative) but I'm sure you don't want to let a stranger in! (False Positive)&lt;/p&gt;&#xA;&lt;p&gt;Hope it helped.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-02-27T17:17:38.943" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28368" PostTypeId="1" AcceptedAnswerId="28384" CreationDate="2018-02-27T17:37:05.320" Score="0" ViewCount="1464" Body="&lt;p&gt;I using two different classifiers to predict a binary target (Random Forests and Decision Trees). Now I want to evaluate my model creating a confusion matrix. For example, for predicting the binary value using random forests I've:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_features, test_features, training_target, test_target, = train_test_split(df.drop(['score_goal'], axis=1),&#xA;                                              df['score_goal'],&#xA;                                              test_size = .3,&#xA;                                              random_state=12)&#xA;&#xA;clf_rf = RandomForestClassifier(n_estimators=25, random_state=12)&#xA;clf_rf.fit(training_features, training_target)&#xA;print(&quot;Accuracy using Random Forest Classifier is &quot;, clf_rf.score(test_features, test_target)*100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm confusing because I don't know how I can compare the predicted values to identify how many False Positives, etc. I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyone knows how can I build that function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24790" LastActivityDate="2018-02-27T23:22:25.770" Title="Python - Get FP/TP from Confusion Matrix using a List" Tags="|python|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28369" PostTypeId="2" ParentId="28339" CreationDate="2018-02-27T17:37:09.670" Score="4" Body="&lt;p&gt;Actually I guess you are making mistake about the second part. The point is that in &lt;code&gt;CNN&lt;/code&gt;s, convolution operation is done over volume. Suppose the input image is in three channels and the next layer has 5 kernels, consequently the next layer will have five feature maps but the convolution operation consists of convolution over volume which has this property: each kernel will have its width and height, moreover, a depth. its depth is equal to the number of feature maps, here channels of the image, of the previous layer. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/27962/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-27T17:37:09.670" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28370" PostTypeId="1" AcceptedAnswerId="28371" CreationDate="2018-02-27T17:43:13.457" Score="0" ViewCount="83" Body="&lt;p&gt;I know that clustering can be used for unsupervised learning and some people told me &lt;em&gt;for many more techniques&lt;/em&gt;, but I was left with no answer, when I asked for what else clustering is used.&lt;/p&gt;&#xA;" OwnerUserId="46935" LastEditorUserId="28175" LastEditDate="2018-02-27T17:54:11.197" LastActivityDate="2018-02-27T17:54:11.197" Title="What is clustering used for?" Tags="|machine-learning|clustering|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28371" PostTypeId="2" ParentId="28370" CreationDate="2018-02-27T17:53:46.187" Score="1" Body="&lt;p&gt;Labeling data is not always an easy task. There are occasion that the data in hand does not have label and you need to make a model using them. You have to find the similarities and differences in your input data. Clustering approaches try to find these similarities and differences to find similar data. Also they are used as a pre-processing before doing supervised classification. In cases that the input data does not have any label, employing clustering approaches can be a way to label the data and use them for training supervised models.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-27T17:53:46.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28372" PostTypeId="1" AcceptedAnswerId="28382" CreationDate="2018-02-27T18:19:56.260" Score="1" ViewCount="1792" Body="&lt;p&gt;How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count. &lt;/p&gt;&#xA;" OwnerUserId="46374" LastEditorUserId="28175" LastEditDate="2018-02-27T18:34:01.847" LastActivityDate="2018-03-01T15:14:13.380" Title="How would one impute missing values for a Discrete variable?" Tags="|machine-learning|data-mining|missing-data|data-imputation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28373" PostTypeId="2" ParentId="28372" CreationDate="2018-02-27T18:33:32.903" Score="1" Body="&lt;p&gt;It depends, if you have the distribution of that feature, you can take the marginal distribution over that feature which its interpretation is to use the expected value of that feature. If you don't have the distribution you can take the mean of the sample in hand of those samples which have value for that feature and add the mean for those which don't have. Another solution is to separate the data of each class and find the mean of the feature of those data samples having the value and putting the mean for each entry of the corresponding class which does not have value in that entry.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-27T18:33:32.903" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28375" PostTypeId="1" AcceptedAnswerId="28381" CreationDate="2018-02-27T18:46:36.117" Score="1" ViewCount="390" Body="&lt;p&gt;Suppose I have a dictionary:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{apple:large apple, apple:red apple, apple:aple, orange:mandarin, orange:orang, orange:blood orange}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and so on...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then I want to replace a large document of entries with the keys. However, occasionally a new value will come up, i.e. {apple:green apple}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a method where I can replace all values with the corresponding key, but then also replace 'close' values like the one given if they appear?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example document:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var1&#xA;_____&#xA;aple&#xA;apple&#xA;orange&#xA;Apple&#xA;Red apple&#xA;gren Apple&#xA;blood Orange&#xA;orang&#xA;&#xA;var1_replaced&#xA;______________&#xA;apple&#xA;apple&#xA;orange&#xA;apple&#xA;apple&#xA;apple&#xA;orange&#xA;orange&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46886" LastEditorUserId="46886" LastEditDate="2018-02-27T21:00:35.390" LastActivityDate="2018-02-27T21:37:02.443" Title="NLP grouping word categories" Tags="|nlp|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28376" PostTypeId="1" AcceptedAnswerId="28448" CreationDate="2018-02-27T19:54:24.817" Score="0" ViewCount="1098" Body="&lt;p&gt;The input data in the model includes column ControlNo.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eqULH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eqULH.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;But I don't want this column being part of learning process so I'm using &lt;code&gt;Select Columns in Dataset&lt;/code&gt; to exclude &lt;code&gt;ControlNo&lt;/code&gt; column.&lt;/p&gt;&#xA;&lt;p&gt;But as a output I want those columns:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;ControlNo, Score Label, Score Probability&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So basically I need NOT to include column &lt;code&gt;ControlNo&lt;/code&gt; into learning process,&#xA;but have it as output along with &lt;code&gt;Score Label&lt;/code&gt; column.&lt;/p&gt;&#xA;&lt;p&gt;How can I do that?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iPlpa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iPlpa.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46783" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-01T10:19:54.667" Title="How to bypass ID column without being used in the training model but have it as output - Azure ML" Tags="|machine-learning|azure-ml|machine-learning-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28377" PostTypeId="2" ParentId="28366" CreationDate="2018-02-27T20:08:43.013" Score="1" Body="&lt;p&gt;This is an interesting but broad question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine PCA. You yse it for exploring the data embedded in lower dimensional space but the first $n$ principale components are also used as the features (after projection of data on them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you use correlation analysis and remove (deselect) features with high correlation with an existing feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You calculate the variance of each feature abd low variances tell you that there is no infirmation in this feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You inspect feature distributions according to target to determine how much they contribute to the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And of course much more ...&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-03-01T10:49:11.617" LastActivityDate="2018-03-01T10:49:11.617" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28378" PostTypeId="1" CreationDate="2018-02-27T20:45:43.000" Score="3" ViewCount="514" Body="&lt;p&gt;I am using a deep CNN to predict the class an image belongs to (N classes). However, the number of classes is not stationary. I.e. over the time the network will be used, some new classes may emerge while others may not be relevant anymore (I do not need to identify these in any way but I know the relevant classes at any time step).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, I am facing the constraint that the data I can use for this task &quot;vanishes&quot;. This means that, at a point in time, I can only use the data of the previous M time steps (e.g. months) where M remains constant. This prohibits simply retraining the network on the entire dataset here and then while dropping some classes/adding new classes (i.e. changing the size of the output layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of using the feature extracting part of the CNN to generate the feature vector and then use binary classifiers (e.g. logistic regressions) on that. This would enable me to simply discard a binary classifier when a class has become irrelevant or add a new binary classifier in case a new class comes up. This approach would take significantly more time in the inference stage (training stage is not critical) as inference would need to be performed for each class separately. Unfortunately, I am also facing tight time constraints at the inference stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I was wondering whether (A) my idea generally makes sense and (B) if there is a more elegant/efficient multi-class classification approach (i.e. a single inference step) that is applicable in my very special case.&lt;/p&gt;&#xA;" OwnerUserId="31582" LastEditorUserId="31582" LastEditDate="2018-02-27T22:36:20.163" LastActivityDate="2018-06-08T23:42:37.677" Title="Deep CNN with variable number of classes and &quot;vanishing&quot; data" Tags="|multiclass-classification|cnn|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28379" PostTypeId="2" ParentId="27088" CreationDate="2018-02-27T21:01:57.103" Score="0" Body="&lt;p&gt;Each sample (read: each row in your table/matrix) can be seen as the realization of an $m$-dimensional random variable $X_{i}$, where $m$ is the number of features (read: number of columns in your table/matrix).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Independence means that the random variables $X_i, i = 1, \dots, n,$ that generate your samples are &lt;a href=&quot;https://en.wikipedia.org/wiki/Independence_(probability_theory)#For_random_variables&quot; rel=&quot;nofollow noreferrer&quot;&gt;independent&lt;/a&gt;. StatsSorceress explains the intuition behind independence wonderfully. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A different way of looking at it is: 'Does knowing sample $X_i$ give me additional information about another sample $X_j$?'&#xA;One case where this is true (and the samples are not independent) is for time series. If your samples are stock prices, and you know that the stock price one second ago was USD 100, then you can be fairly sure that the stock price right now is also going to be close to USD 100.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-02-27T21:01:57.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28380" PostTypeId="2" ParentId="28331" CreationDate="2018-02-27T21:26:05.737" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to subsample observations from training set which closely&#xA;  resembles test set.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises well to unseen data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, one should adapt its &lt;em&gt;test&lt;/em&gt; data to its &lt;em&gt;train&lt;/em&gt; data (e.g. &lt;a href=&quot;https://sebastianraschka.com/faq/docs/scale-training-test.html&quot; rel=&quot;noreferrer&quot;&gt;standardising test data according to train data&lt;/a&gt;) and not the other way around. In practice, you &lt;strong&gt;don't&lt;/strong&gt; know your test data.&lt;/p&gt;&#xA;" OwnerUserId="45774" LastActivityDate="2018-02-27T21:26:05.737" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28381" PostTypeId="2" ParentId="28375" CreationDate="2018-02-27T21:37:02.443" Score="2" Body="&lt;p&gt;Well ... the simplest approach is using &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Approximate_string_matching&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fuzzy String Matching&lt;/a&gt; and it will work. Just go through the examples in python implementation of it (&lt;a href=&quot;https://github.com/seatgeek/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;fuzzywuzzy&lt;/a&gt;) and you will understand how it works. You need to find a threshold by practice to determine if two strings are similar enough to be considered as same concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it didn't work please drop a line in the comments so I can propose more sophisticated algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-02-27T21:37:02.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28382" PostTypeId="2" ParentId="28372" CreationDate="2018-02-27T22:59:56.353" Score="1" Body="&lt;p&gt;Apart from the methods @Media mentioned, here are some more:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Imputing with info from other variables&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This method is to create a (multi-class) model based on target variable.  So that missing values would be predicted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The steps are likely to be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Subset data without missing value in the variable you want to impute&lt;/li&gt;&#xA;&lt;li&gt;Machine learning on the data with predict model&lt;/li&gt;&#xA;&lt;li&gt;Predict on data with the missing value from model created&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Clustering&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Whether missing values are mainly related to a combination of variables? Unsupervised methods may help here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example using randomForest:&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/107530/using-cluster-information-in-multiple-imputation&quot;&gt;https://stats.stackexchange.com/questions/107530/using-cluster-information-in-multiple-imputation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Domain Knowledge&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If we known that the reason for missing value, we can assign the missing value to a proper level. For example, a survey data are collected from web where given choices are not applicable for some cases, hence leave blank.  In this case, it would be better to leave as a separate value. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Implementation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There are some R package can impute the data for you;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;MICE&lt;/li&gt;&#xA;&lt;li&gt;Amelia&lt;/li&gt;&#xA;&lt;li&gt;missForest&lt;/li&gt;&#xA;&lt;li&gt;Hmisc&lt;/li&gt;&#xA;&lt;li&gt;mi&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/tutorial-powerful-packages-imputing-missing-values/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.analyticsvidhya.com/blog/2016/03/tutorial-powerful-packages-imputing-missing-values/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="17012" LastEditorUserId="17012" LastEditDate="2018-03-01T15:14:13.380" LastActivityDate="2018-03-01T15:14:13.380" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28383" PostTypeId="1" CreationDate="2018-02-27T23:10:24.410" Score="12" ViewCount="4868" Body="&lt;p&gt;How would you &lt;strong&gt;optimize&lt;/strong&gt; a &lt;strong&gt;pre-trained&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;neural network&lt;/code&gt;&lt;/strong&gt; to apply it to a separate problem? Would you just add more layers to the &lt;strong&gt;pre-trained&lt;/strong&gt; model and test it on your data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if the task was to use a &lt;strong&gt;CNN&lt;/strong&gt; to classify &lt;a href=&quot;https://en.wikipedia.org/wiki/Wallpaper_group&quot; rel=&quot;noreferrer&quot;&gt;wallpaper groups&lt;/a&gt;, I'm sure that it wouldn't work to directly classify off a &lt;strong&gt;pre-trained&lt;/strong&gt; network trained on images cats and dogs, even though both are image classifiers.&lt;/p&gt;&#xA;" OwnerUserId="32288" LastEditorUserId="29169" LastEditDate="2021-02-13T16:37:43.657" LastActivityDate="2021-02-13T16:37:43.657" Title="Using a pre trained CNN classifier and apply it on a different image dataset" Tags="|deep-learning|neural-network|classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28384" PostTypeId="2" ParentId="28368" CreationDate="2018-02-27T23:22:25.770" Score="0" Body="&lt;p&gt;Looks like you're using scikit-learn. So why not explore a bit more? Scikit has a &lt;code&gt;metrics&lt;/code&gt; module, that can be of use for your problem. Essentially, what you need is to have two separate arrays - one with real labels and another with predicted labels. And then you're good to go, you could call &lt;code&gt;metrics.classification_report&lt;/code&gt; or &lt;code&gt;metrics.confusion_matrix&lt;/code&gt; or &lt;code&gt;metrics.accuracy_score&lt;/code&gt;, all of them use the real labels and the predicted labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's nothing wrong with using &lt;code&gt;clf_rf.score(test_features, test_target)&lt;/code&gt;, but it will only give you a single value. If you look at the source code, what happens is that the &lt;code&gt;score&lt;/code&gt; method calls a &lt;code&gt;predict&lt;/code&gt; method with &lt;code&gt;test_feature&lt;/code&gt; for prediction of labels, which occurs behind the scenes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's better to actually capture those predicted labels, so that you can reuse them.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf_rf.fit(training_features, training_target)&#xA;predicted_target = clf_rf.predict(test_features)&#xA;accuracy = sklearn.metrics.accuracy_score(test_target, predicted_target)&#xA;cnf_matrix = sklearn.metrics.confusion_matrix(test_target, predicted_target)&#xA;class_report = sklearn.metrics.classification_report(test_target, predicted_target)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then you could do whichever you like with the calculated metrics, print, plot, etc. &#xA;Have a look at the examples that are included for each model/metrics you're using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#examples-using-sklearn-ensemble-randomforestclassifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;3&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report&quot; rel=&quot;nofollow noreferrer&quot;&gt;4&lt;/a&gt;, etc.&lt;/p&gt;&#xA;" OwnerUserId="46736" LastActivityDate="2018-02-27T23:22:25.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28385" PostTypeId="2" ParentId="28331" CreationDate="2018-02-27T23:50:39.017" Score="4" Body="&lt;p&gt;Train set subsampling might not be the best solution!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The differences between test/execution set and training set distribution/features are very common in supervised learning tasks (this is one of the reasons that competitions such as Kaggle are challenging). That is why we say &lt;em&gt;the past performance may be (only) used as a guide for estimating the future performance but it does not indicate/guarantee it&lt;/em&gt;. Therefore, &lt;strong&gt;generalizable&lt;/strong&gt; models have always been preferred over fine-tuned models that may perform very well on the train (sub)set but do poorly on unseen data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While such difference is normal, the too large gap between the past and future sample may be referred as examples of &lt;strong&gt;concept drift&lt;/strong&gt; which is an active research field by itself. Given your question, I cannot judge that your case is a normal ML case or the concept drift is really happening.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are my suggestions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Train a number of models with high generalization capability. Using &lt;strong&gt;&lt;em&gt;bootstrap&lt;/em&gt;&lt;/strong&gt; sampling from your train dataset, you can easily calculate &lt;strong&gt;&lt;em&gt;bias&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;variance&lt;/em&gt;&lt;/strong&gt; components of errors. Recall that you are looking for a &lt;strong&gt;low-variance&lt;/strong&gt; model (where the changes in data would have a marginal effect on its performance) rather than &lt;strong&gt;low-bias&lt;/strong&gt; but &lt;strong&gt;high-variance&lt;/strong&gt; models (that might overfit to your training (sub)set). Now, you can select the best algorithms and evaluate them against the test set. Note that in the training time we supposed to not look at the test set!&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of several random downsampling, look for standardization/normalization and feature selection/engineering. These techniques might be practical in learning more general models. For example, sometimes the range of feature domain may change over time while the shape of distribution (whatever it is) remains almost the same (eg same distribution that is shifted towards left or right). In such case, a simple &lt;strong&gt;standardization&lt;/strong&gt; (ie mapping the train and test samples to a predefined space such as [0,1] using different mapping functions) can reduce the symptoms.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Systematic &lt;strong&gt;downsampling&lt;/strong&gt; can only be an appropriate solution if you do it based on some knowledge about the problem (not just for the purpose of getting a better accuracy on the test dataset). For example, you might know that some of the records in the train data are sampled a long time ago, from far field, or affected by particular factors which none of them will happen in future (in test data collection). In such case, you may remove those samples that can be irrelevant as you are confident that you will not see such patterns in the future (I mean you should have a &lt;strong&gt;rationale&lt;/strong&gt; behind the selection of the training subset rather than looking into the test set that in reality, you do not have access to it). In such case, I call it &lt;strong&gt;outlier removal&lt;/strong&gt; rather than downsampling.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-02-27T23:50:39.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28386" PostTypeId="2" ParentId="28337" CreationDate="2018-02-28T01:13:32.473" Score="1" Body="&lt;p&gt;Yes, the &lt;em&gt;mean&lt;/em&gt; is crucial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you just plug in another distance function instead of the sum-of-squares, the algorithm may fail to converge (and will not find a local optimum). The k-means algorithm relies on &lt;strong&gt;both steps&lt;/strong&gt; (reassignment and mean recomputation) to &lt;strong&gt;optimize the &lt;em&gt;same&lt;/em&gt; function.&lt;/strong&gt; If they optimize different functions, you can get an infinite loop on the worst case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use any Bergman divergence. And in some cases such as Euclidean instead of squared Euclidean nothing really bad will happen (no infinite loop etc.). It just won't find the best solution, because the mean isn't minimizing Euclidean distances.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-02-28T01:13:32.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28387" PostTypeId="2" ParentId="28383" CreationDate="2018-02-28T02:27:01.157" Score="15" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/30871/35644&quot;&gt;Do Read - When Transfer Learning is Disadvantageous ?(thanks to @media)&lt;/a&gt;&#xA;(looks very informative to me, so added here to make this answer a complete one...)&lt;/p&gt;&#xA;&lt;p&gt;Answer to your Question..(starts here)&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h2&gt;Transfer Learning is What you are Looking for..&lt;/h2&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on a dataset of images, our first instinct would be to train the network from scratch. However, in practice, deep neural networks like Covnet has a huge number of parameters, often in the range of millions. Training a Covnet on a small dataset (one that is smaller than the number of parameters) greatly affects the Covnet’s ability to generalize, often result in overfitting.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Therefore, more often in practice, one would fine-tune existing networks that are trained on a large dataset like the ImageNet (1.2M labeled images) by continue training it (i.e. running back-propagation) on the smaller dataset we have. Provided that our dataset is not drastically different in context to the original dataset (e.g. ImageNet), the pre-trained model will already have learned features that are relevant to our own classification problem.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h2&gt;When to fine tune Models?&lt;/h2&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;In general, if our dataset is not drastically different in context from the dataset which the pre-trained model is trained on, we should go for fine-tuning. Pre-trained network on a large and diverse dataset like the ImageNet captures universal features like curves and edges in its early layers, that are relevant and useful to most of the classification problems.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Of course, if our dataset represents some very specific domain, say for example, medical images or Chinese handwritten characters, and that no pre-trained networks on such domain can be found, we should then consider training the network from scratch.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;One other concern is that if our dataset is small, fine-tuning the pre-trained network on a small dataset might lead to overfitting, especially if the last few layers of the network are fully connected layers, as in the case for VGG network. Speaking from my experience, if we have a few thousand raw samples, with the common data augmentation strategies implemented (translation, rotation, flipping, etc), fine-tuning will usually get us a better result.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If our dataset is really small, say less than a thousand samples, a better approach is to take the output of the intermediate layer prior to the fully connected layers as features (bottleneck features) and train a linear classifier (e.g. SVM) on top of it. SVM is particularly good at drawing decision boundaries on a small dataset.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h2&gt;Fine-tuning Techniques&lt;/h2&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Below are some general guidelines for fine-tuning implementation:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The common practice is to truncate the last layer (softmax layer) of the pre-trained network and replace it with our new softmax layer that are relevant to our own problem. For example, pre-trained network on ImageNet comes with a softmax layer with 1000 categories.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If our task is a classification on 10 categories, the new softmax layer of the network will be of 10 categories instead of 1000 categories. We then run back propagation on the network to fine-tune the pre-trained weights. Make sure cross validation is performed so that the network will be able to generalize well.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use a smaller learning rate to train the network. Since we expect the pre-trained weights to be quite good already as compared to randomly initialized weights, we do not want to distort them too quickly and too much. A common practice is to make the initial learning rate 10 times smaller than the one used for scratch training.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is also a common practice to freeze the weights of the first few layers of the pre-trained network. This is because the first few layers capture universal features like curves and edges that are also relevant to our new problem. We want to keep those weights intact. Instead, we will get the network to focus on learning dataset-specific features in the subsequent layers.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;You need to train them again in this case as if I am not wrong then the wallpapers is not a class of Image-net models.. It won't be tough to build a model from scratch to do so(preferably a shallower one will also do here..)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;h3&gt;Source of my answer is translation from these amazing courses..&lt;/h3&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs231n.github.io/transfer-learning/&quot; rel=&quot;noreferrer&quot;&gt;CS231N&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.fast.ai&quot; rel=&quot;noreferrer&quot;&gt;fast.ai&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;For Further Readings,&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://datascience.stackexchange.com/a/27633/35644&quot;&gt;There is a difference between using pre-trained weights and transfer learning.&lt;/a&gt;,&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://karpathy.github.io/2019/04/25/recipe/&quot; rel=&quot;noreferrer&quot;&gt;Karpathy's  Latest NN's Training Recepie BlogPost&lt;/a&gt;..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-05-16T06:31:02.097" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28388" PostTypeId="2" ParentId="28073" CreationDate="2018-02-28T03:36:24.403" Score="0" Body="&lt;h2&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Root-mean-square_deviation&quot; rel=&quot;nofollow noreferrer&quot;&gt;RMSE&lt;/a&gt;(Source)&lt;/h2&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;&lt;p&gt;The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. RMSE is an absolute measure of fit. As the square root of a variance, RMSE can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. &lt;strong&gt;Lower values of RMSE indicate better fit&lt;/strong&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;  &lt;li&gt;&lt;p&gt;RMSE is a good measure of how accurately the model predicts the response, and is the most important criterion for fit if the main purpose of the model is prediction.&lt;/p&gt;&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The best measure of model fit depends on the researcher’s objectives, and more than one are often useful.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you are badly overffiting the data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-02-28T06:57:30.753" LastActivityDate="2018-02-28T06:57:30.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28389" PostTypeId="2" ParentId="20274" CreationDate="2018-02-28T03:58:06.530" Score="0" Body="&lt;h2&gt;Data:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Given for example &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;===========================================================&#xA;  customer campaign    campaign_time       response_time   &#xA;-----------------------------------------------------------&#xA;1   100       a     2017-01-01 06:50:01 2017-01-01 08:02:21&#xA;2   101       a     2017-01-01 06:50:01 2017-01-01 16:45:31&#xA;3   101       a     2017-01-01 06:50:01 2017-01-02 07:20:00&#xA;4   100       b     2017-01-07 06:30:21 2017-01-08 08:15:21&#xA;5   101       b     2017-01-07 06:30:21 2017-01-07 17:00:12&#xA;6   100       c     2017-01-14 06:43:55 2017-01-14 07:59:44&#xA;7   101       d     2017-01-21 14:02:01 2017-01-21 16:50:01&#xA;-----------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As others have already answered, i will like to add to them few points,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;What About &lt;strong&gt;Feature Engineering&lt;/strong&gt;? We can add a lot of other features to the given data-set.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Week Differences.&lt;/li&gt;&#xA;&lt;li&gt;Day, Month etc.&lt;/li&gt;&#xA;&lt;li&gt;Time Differences as well.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And then creating &lt;strong&gt;group by&lt;/strong&gt; views of the data-set as well&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;After that we can use a &lt;strong&gt;RF&lt;/strong&gt; to help use see what are the &lt;strong&gt;feature importance&lt;/strong&gt; of different attributes/columns etc.&lt;/li&gt;&#xA;&lt;li&gt;After that you can also go for &lt;strong&gt;Dendograms(hierarchial clustering)&lt;/strong&gt; to see what columns are irrelevant.&lt;/li&gt;&#xA;&lt;li&gt;And the process continues with different models, doing EDA's etc..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-02-28T03:58:06.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28390" PostTypeId="2" ParentId="28241" CreationDate="2018-02-28T07:18:30.003" Score="2" Body="&lt;p&gt;To my extent of my knowledge, RL is used as a model for attention mechanism in object detection field (particularly the REINFORCE algorithm which is a 'flavor' of policy gradient methods). You can take a look at one of the papers that first proposed the method: &lt;a href=&quot;https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;RAM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-02-28T07:18:30.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28391" PostTypeId="1" CreationDate="2018-02-28T07:52:29.153" Score="0" ViewCount="105" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Given a semi-structured document with only texts and images, and some style properties present on the text, What is&#xA;  the best possible way to classify the text present in the document to&#xA;  any formal HTML standard class such as Heading, Body, Caption, List&#xA;  etc.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;PS: Please forgive me if you think the question is highly abstract, but the task I am attempting can be broadly categorized in this way only. I need to extract structure from a loosely structured document. Any suggestions to this incorporating ML/Heuristics or links to any such existing problems/existing solutions will be deeply appreciated as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wAvIz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wAvIz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46974" LastEditorUserId="46974" LastEditDate="2018-02-28T08:16:10.540" LastActivityDate="2018-02-28T08:16:10.540" Title="Categorize text as Body, Heading in a loosely formatted document" Tags="|machine-learning|neural-network|text|" AnswerCount="0" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="28393" PostTypeId="2" ParentId="17428" CreationDate="2018-02-28T09:42:15.810" Score="1" Body="&lt;p&gt;There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If either of these factors is smaller than &lt;code&gt;1&lt;/code&gt;, then the gradients may vanish in time; if larger than &lt;code&gt;1&lt;/code&gt;, then exploding might happen. For example, the &lt;code&gt;tanh&lt;/code&gt; derivative is &amp;lt;&lt;code&gt;1&lt;/code&gt; for all inputs except &lt;code&gt;0&lt;/code&gt;; sigmoid is even worse and is always ≤&lt;code&gt;0.25&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the recurrency of the LSTM the activation function is the identity function with a derivative of &lt;code&gt;1.0&lt;/code&gt;. So, the backpropagated gradient neither vanishes or explodes when passing through, but remains constant.&#xA;The effective weight of the recurrency is equal to the forget gate activation. So, if the forget gate is on (activation close to &lt;code&gt;1.0&lt;/code&gt;), then the gradient does not vanish. Since the forget gate activation is never &gt;&lt;code&gt;1.0&lt;/code&gt;, the gradient can't explode either.&#xA;So that's why LSTM is so good at learning long range dependencies.&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-02-28T09:42:15.810" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28394" PostTypeId="2" ParentId="26374" CreationDate="2018-02-28T09:42:30.577" Score="4" Body="&lt;p&gt;I also thought about the very same question recently and I think I might have a possible explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we need to calculate distances between k-nearest neighbors, we have to provide some synthetic value which denotes the difference between nominal features. As a matter of fact, it could be any value. For example, imagine you have your nominal features one-hot encoded. In this case the difference between two different nominal features would be 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you would also like to keep the calculated distances more or less on the same scale as if you would have by using only continuous features. I believe that using median of standard deviations of continuous features helps to achieve exactly that and is the reason why it was chosen as a measure of difference between nominal features.&lt;/p&gt;&#xA;" OwnerUserId="46936" LastActivityDate="2018-02-28T09:42:30.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28395" PostTypeId="2" ParentId="17368" CreationDate="2018-02-28T10:05:33.200" Score="1" Body="&lt;p&gt;Ok, this is a bit late, but two points which will hopefully be of help for someone in the future. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The labels that you are looking for should be returned by &lt;code&gt;clf.predict&lt;/code&gt; (see also in the tutorial that you posted)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As can be seen in this quesion &lt;a href=&quot;https://datascience.stackexchange.com/questions/26958/why-wont-my-svm-learn-a-sequence-of-repeated-elements/26990#26990&quot;&gt;Why won&amp;#39;t my SVM learn a sequence of repeated elements&lt;/a&gt; your testset actually must contain outliers. This contradicts the statement in scikit-learn that they are doing novelty detection. To me, the given link proves that the method does not (not that this would be a problem, it is just important to know).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43469" LastActivityDate="2018-02-28T10:05:33.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28396" PostTypeId="1" AcceptedAnswerId="29314" CreationDate="2018-02-28T10:19:08.687" Score="2" ViewCount="1266" Body="&lt;p&gt;Given the following data set:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UCKu3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UCKu3.png&quot; alt=&quot;Sample data set&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering what kind of plotting technique can be used to produce such a visualization:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9Gfnb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9Gfnb.png&quot; alt=&quot;Desired visualization of the data set&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;Skill&lt;/code&gt; attribute is mapped to the y-axis, while the &lt;code&gt;Participant&lt;/code&gt; is on the x-axis. The other two attributes are encoded as colour and size features of each point in the visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although I am trying to get this done with &lt;code&gt;Pandas&lt;/code&gt; and &lt;code&gt;pdvega&lt;/code&gt;, I'd be happy to learn another tool that could render that. But most importantly, I would like to know the &lt;em&gt;proper technical terms&lt;/em&gt; used to describe this type of visualization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have reviewed various Pandas tutorials, but the problem is that the examples provided are targeting numeric data, rather than categories. I am thinking that perhaps I could partially fake it by turning each category into a number, then somehow overriding the labels on the axis - but it sounds like a convoluted solution to a mundane problem, so there ought to be a nicer way to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the raw data that can be used to produce a Pandas dataframe with the data above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;raw = {&quot;Age&quot;:{&quot;0&quot;:&quot;27..35&quot;,&quot;1&quot;:&quot;18..26&quot;,&quot;2&quot;:&quot;18..26&quot;,&quot;3&quot;:&quot;18..26&quot;,&quot;4&quot;:&quot;18..26&quot;},&#xA;       &quot;Skill&quot;:{&quot;0&quot;:&quot;intermediate&quot;,&quot;1&quot;:&quot;expert&quot;,&quot;2&quot;:&quot;novice&quot;,&quot;3&quot;:&quot;intermediate&quot;,&quot;4&quot;:&quot;expert&quot;},&#xA;       &quot;Gender&quot;:{&quot;0&quot;:&quot;M&quot;,&quot;1&quot;:&quot;M&quot;,&quot;2&quot;:&quot;F&quot;,&quot;3&quot;:&quot;F&quot;,&quot;4&quot;:&quot;M&quot;}}&#xA;df = pd.DataFrame(raw)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46981" LastEditorUserId="46981" LastEditDate="2018-02-28T11:39:26.967" LastActivityDate="2018-03-20T14:43:01.847" Title="How to plot categorical data and encode it through colour and shape size?" Tags="|visualization|pandas|plotting|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28398" PostTypeId="1" CreationDate="2018-02-28T10:34:21.550" Score="0" ViewCount="257" Body="&lt;p&gt;Days ago,One AI financial service provider offered us a lesson and mentioned that you are supposed to perform specific feature engineering according to the specific algorithm you are using.For example,when using logistics regression,fitting more features（uncorrelated）like binning the continuous variable into discrete ones are often suggested.Because logistics regression is a simple algorithm and we try to raise dimension in the way that samples will be separated better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I searched a lot(maybe not yet),most of materials are &quot;why/what feature engineering important&quot;,&quot;scaling/standardization/binning continuous variable&quot;,&quot;dealing with null value&quot; or some theoretical comments with no discrete manipulation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;why and how the specific feature engineering should work on specific algorithm.Or any advice on this saying,is it right or wrong?what do you think.any comments are appreciated.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;（I am not good at English,sorry about that if I am not clear enough）&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am not looking forward a detailed answer,some deep thinking about this part is good.&lt;/p&gt;&#xA;" OwnerUserId="46980" LastEditorUserId="46980" LastEditDate="2018-03-01T06:29:51.873" LastActivityDate="2018-08-28T12:02:05.260" Title="specific feature engineering for specific algorithm" Tags="|machine-learning|feature-engineering|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28403" PostTypeId="2" ParentId="28398" CreationDate="2018-02-28T11:24:58.530" Score="0" Body="&lt;p&gt;Some algorithms like boosting trees (XGBoost, for example) easy deals with almost anything &quot;strange&quot; in data: NaN values, outliers, different scalings. And such algorithms can find complex feature interactions in data by itself.&#xA;But if you use some different or simple algorithms such as linear regression, you have to do some features preprocessing to get good results from it: fill or drop NaNs and outliers; make similar scales for all features; try to make polynomial features and so on. Otherwise you can get strange or very inaccurate results.&lt;/p&gt;&#xA;" OwnerUserId="43374" LastActivityDate="2018-02-28T11:24:58.530" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28405" PostTypeId="1" AcceptedAnswerId="28410" CreationDate="2018-02-28T13:12:11.137" Score="1" ViewCount="291" Body="&lt;p&gt;I have this huge mixed data set consisting of both numerical and categorical attributes which upon OneHotEncoding results into a data set with very high dimensionality. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it wise to apply machine learning algorithms like K-means clustering, dimensionality reduction and regression on subsets of data set? For example applying K-means clustering to numerical columns first and join the result with categorical data set later.&lt;/p&gt;&#xA;" OwnerUserId="44701" LastActivityDate="2018-02-28T15:38:15.987" Title="Applying machine learning algorithms to subset of attributes in dataframe" Tags="|python|clustering|bigdata|categorical-data|dimensionality-reduction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28407" PostTypeId="2" ParentId="27618" CreationDate="2018-02-28T13:57:00.153" Score="8" Body="&lt;p&gt;You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-words) and Doc2vec has analogous PV-DM and CBOW models:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Word2vec&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Continuous-bag-of-words (&lt;strong&gt;CBOW&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Skip-gram&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Doc2vec&lt;/strong&gt; (Paragraph Vector)&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Distributed Memory (&lt;strong&gt;PV-DM&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Distributed Bag-of-words (&lt;strong&gt;PV-DBOW&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In skip-gram the inputs are pairs of words generated by a moving window of size &lt;em&gt;n&lt;/em&gt; trough the data where we use one word to predict the other, whereas in CBOW the task is predicting a center word from the surrounding words. In the second case the context word vectors are typically averaged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Doc2vec extension works analogously. The DBOW model is analog to skip-gram and what it does is calculating the probability of words in a paragraph given a randomly selected word from that paragraph. The DM however, tries to predict the central word based on a context of words and the context paragraph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, not necessarily the doc2vec models need to calculate word embeddings first.  In fact, in the gensim &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;noreferrer&quot;&gt;doc2vec function&lt;/a&gt; there is an parameter for the dbow model that says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;dbow_words (int {1,0}) – If set to 1 trains word-vectors (in&#xA;  skip-gram fashion) simultaneous with DBOW doc-vector training; If 0,&#xA;  only trains doc-vectors (faster).&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Meaning that during the training phase not even the dbow requires producing the word vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a great &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-IMDB.ipynb&quot; rel=&quot;noreferrer&quot;&gt;notebook&lt;/a&gt; from rare-technologies that gives a lot of insights about how these models are trained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your specific questions, you can train a word2vec and doc2vec model and use the results to compare words and paragraphs (look at this &lt;a href=&quot;https://arxiv.org/pdf/1607.05368.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt;). But take into account that doc2vec is not training words and then doing some fancy stuff with them. The training process involves calculating paragraphs directly.&lt;/p&gt;&#xA;" OwnerUserId="45166" LastEditorUserId="45166" LastEditDate="2018-02-28T14:30:37.193" LastActivityDate="2018-02-28T14:30:37.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28410" PostTypeId="2" ParentId="28405" CreationDate="2018-02-28T15:38:15.987" Score="1" Body="&lt;p&gt;Applying a machine learning algorithm on only a subset of the data and including other subsets later does not allow the algorithm to assess the importance of each attribute equally. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say you have a data set called A, which has subsets B and C. Without loss of genearality, if you fit a model ('apply an algorithm') on subset B, and then include subset C later, then you're saying 'given subset B is already in the model, assess the impact of including subset C'. Instead, if you apply the entire algorithm on the entire data set (A), then you're allowing the algorithm to discover which features are most important for the desired outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, it may be wise to process the different elements of your data set differently. That is, categorical covariates may be modelled differently from continuous covariates. If you're using something like a feed-forward neural network, then it's not a big deal, but if you're using a more traditional statistical model you may need to take that into account. For example, in R, you need to specify that a categorical covariate is in fact a 'factor' variable. &lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-02-28T15:38:15.987" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28411" PostTypeId="1" AcceptedAnswerId="28415" CreationDate="2018-02-28T16:00:18.873" Score="8" ViewCount="13042" Body="&lt;p&gt;One of the recommendations in the &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/home/week/2&quot; rel=&quot;noreferrer&quot;&gt;Coursera Machine Learning&lt;/a&gt; course when working with gradient descent based algorithms is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Debugging gradient descent.&lt;/strong&gt; Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J(θ) ever increases, then you probably need to decrease α.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Do gradient descent based models in scikit-learn provide a mechanism for retrieving the cost vs the number of iterations?&lt;/p&gt;&#xA;" OwnerUserId="13237" LastEditorUserId="28175" LastEditDate="2018-02-28T16:12:41.757" LastActivityDate="2018-02-28T18:17:41.113" Title="How to plot cost versus number of iterations in scikit learn?" Tags="|machine-learning|neural-network|deep-learning|scikit-learn|gradient-descent|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28412" PostTypeId="1" AcceptedAnswerId="28436" CreationDate="2018-02-28T16:03:31.040" Score="1" ViewCount="316" Body="&lt;p&gt;I've this code (part of predictive model):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_features, test_features, training_target, test_target, = train_test_split(df.drop(['target'], axis=1),&#xA;                                              df['target'],&#xA;                                              test_size = .3,&#xA;                                              random_state=12)&#xA;    coef_gini = DecisionTreeClassifier(criterion = &quot;gini&quot;, random_state = 100, max_depth = 3, min_samples_leaf = 5)&#xA;    coef_gini.fit(training_features, training_target)&#xA;&#xA;    y_pred = coef_gini.predict(test_features)&#xA;    y_pred&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I export my dataset with the predictive value in order to create a confusion matrix to determine the False Positive value?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="23811" LastEditorUserId="40853" LastEditDate="2018-03-01T02:04:38.043" LastActivityDate="2018-03-01T02:21:15.520" Title="Export dataset with predicted target - Python" Tags="|python|predictive-modeling|training|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28413" PostTypeId="2" ParentId="24319" CreationDate="2018-02-28T16:04:31.333" Score="0" Body="&lt;p&gt;It seems that &lt;code&gt;Intel&lt;/code&gt; lets users use its AI &lt;code&gt;DevCloud&lt;/code&gt; for free for thirty days I guess. &lt;a href=&quot;https://software.intel.com/ai-academy/tools/devcloud?cid=&amp;amp;utm_content=Machine_Learning_Academic&amp;amp;utm_medium=Banner_Ad&amp;amp;utm_source=StackOverflow&amp;amp;utm_campaign=AI_Student_ASMO_1H_18_MixedMedia&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is the instructions.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-28T16:04:31.333" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28415" PostTypeId="2" ParentId="28411" CreationDate="2018-02-28T16:12:17.493" Score="4" Body="&lt;p&gt;Based on the answer &lt;a href=&quot;https://stackoverflow.com/a/44453621/5120235&quot;&gt;here&lt;/a&gt;, use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;old_stdout = sys.stdout&#xA;sys.stdout = mystdout = StringIO()&#xA;clf = SGDClassifier(**kwargs, verbose=1)&#xA;clf.fit(X_tr, y_tr)&#xA;sys.stdout = old_stdout&#xA;loss_history = mystdout.getvalue()&#xA;loss_list = []&#xA;for line in loss_history.split('\n'):&#xA;    if(len(line.split(&quot;loss: &quot;)) == 1):&#xA;        continue&#xA;    loss_list.append(float(line.split(&quot;loss: &quot;)[-1]))&#xA;plt.figure()&#xA;plt.plot(np.arange(len(loss_list)), loss_list)&#xA;plt.savefig(&quot;warmstart_plots/pure_SGD:&quot;+str(kwargs)+&quot;.png&quot;)&#xA;plt.xlabel(&quot;Time in epochs&quot;)&#xA;plt.ylabel(&quot;Loss&quot;)&#xA;plt.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also take a look at &lt;a href=&quot;https://stackoverflow.com/q/38179687/5120235&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-02-28T16:12:17.493" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28416" PostTypeId="2" ParentId="24319" CreationDate="2018-02-28T16:19:38.660" Score="14" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Check out &lt;a href=&quot;https://www.crestle.com&quot; rel=&quot;noreferrer&quot;&gt;Crestle&lt;/a&gt;.(Free one hour GPU compute time)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://colab.research.google.com/notebook&quot; rel=&quot;noreferrer&quot;&gt;Google's colab&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.floydhub.com/pricing&quot; rel=&quot;noreferrer&quot;&gt;Floyd-hub&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Paperspace (not free but it's easy to use referrals and earn 15$ compute time..)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Azure ($200)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.deepcognition.ai&quot; rel=&quot;noreferrer&quot;&gt;deepcognition.ai&lt;/a&gt; (2 hours on different GPU enabled machines)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.cognitiveclasses.ai&quot; rel=&quot;noreferrer&quot;&gt;IBM Cloud&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For a more updated list you can star this github repo:&#xA; &lt;a href=&quot;https://github.com/binga/cloud-gpus&quot; rel=&quot;noreferrer&quot;&gt;Cloud GPUs&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="-1" LastEditDate="2019-06-26T14:55:54.327" LastActivityDate="2019-06-26T14:55:54.327" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28417" PostTypeId="2" ParentId="28411" CreationDate="2018-02-28T16:31:26.563" Score="5" Body="&lt;h3&gt;Note The Code Is Self Explanatory(It's Hard-coded)..&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here's the function which we are considering &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def f(a,b):&#xA;    return a**2 + b**2&#xA;&#xA;fig = plt.figure(figsize=(10, 6))&#xA;ax = fig.gca(projection='3d')&#xA;plt.hold(True)&#xA;a = np.arange(-2, 2, 0.25)&#xA;b = np.arange(-2, 2, 0.25)&#xA;a, b = np.meshgrid(a, b)&#xA;c = f(a,b)&#xA;surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3, &#xA;                       linewidth=0, antialiased=False,cmap='rainbow')&#xA;ax.set_zlim(-0.01, 8.01)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rX35X.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rX35X.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Here's a 3D view of Gradient Descent Reaching The Optimum(in case interested, it will not always work, Have a look at the end plot..)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gradient_descent(theta0, iters, alpha):&#xA;    history = [theta0] # to store all thetas&#xA;    theta = theta0     # initial values for thetas&#xA;    # main loop by iterations:&#xA;    for i in range(iters):&#xA;        # gradient is [2x, 2y]:&#xA;        gradient = [2.0*x for x in theta] &#xA;        # update parameters:&#xA;        theta = [a - alpha*b for a,b in zip(theta, gradient)]&#xA;        history.append(theta)&#xA;    return history&#xA;&#xA;history = gradient_descent(theta0 = [-1.8, 1.6], iters = 30, alpha = 0.03)&#xA;&#xA;fig = plt.figure(figsize=(10, 6))&#xA;ax = fig.gca(projection='3d')&#xA;plt.hold(True)&#xA;a = np.arange(-2, 2, 0.25)&#xA;b = np.arange(-2, 2, 0.25)&#xA;a, b = np.meshgrid(a, b)&#xA;c = f(a,b)&#xA;surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3, &#xA;                       linewidth=0, antialiased=False)&#xA;ax.set_zlim(-0.01, 8.01)&#xA;&#xA;a = np.array([x[0] for x in history])&#xA;b = np.array([x[1] for x in history])&#xA;c = f(a,b)&#xA;ax.scatter(a, b, c, color=&quot;r&quot;); &#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And Here's the output we will get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/l4Nsc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/l4Nsc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When Gradient Descent will fail (unfortunately)..&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unfortunately, if the function has many extrema, then the Gradient Descent could find the local minimum instead of global minimum. One trick is to overcome this disadvantage is to run SGD several times with different initial guessed values for $x$.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/vgYMW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vgYMW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-02-28T18:17:41.113" LastActivityDate="2018-02-28T18:17:41.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28418" PostTypeId="1" CreationDate="2018-02-28T16:55:43.977" Score="3" ViewCount="968" Body="&lt;p&gt;I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I run on my machine on RStudio. I am thinking another tool to reduce this expensive time-cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I am talking mainly about the distributed computing tools like spark regardless of the type of algorithm and the size of my data.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help is appreciated !&lt;/p&gt;&#xA;" OwnerUserId="43461" LastEditorUserId="43461" LastEditDate="2018-03-02T09:53:24.257" LastActivityDate="2018-03-02T09:53:24.257" Title="What are the tools to speed up the running time of machine learning algorithms?" Tags="|machine-learning|deep-learning|apache-spark|" AnswerCount="1" CommentCount="4" ClosedDate="2018-03-01T17:44:27.193" ContentLicense="CC BY-SA 3.0" />
  <row Id="28419" PostTypeId="2" ParentId="25231" CreationDate="2018-02-28T18:02:10.327" Score="2" Body="&lt;p&gt;The output of a convolution layer is computed as the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the &lt;strong&gt;depth&lt;/strong&gt; (No of feature maps) is equal to the number of filters applied in this layer (because each added channel is a result of one filter's feature map)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the &lt;strong&gt;width&lt;/strong&gt; ( the same for height) is computed according to the following equation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$W=\frac{W−F+2P}{S} + 1$&lt;/span&gt; where &lt;span class=&quot;math-container&quot;&gt;$F$&lt;/span&gt; is the receptive field (filter width), &lt;span class=&quot;math-container&quot;&gt;$P$&lt;/span&gt; is the padding and &lt;span class=&quot;math-container&quot;&gt;$S$&lt;/span&gt; is the stride. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more details see the examples in the following link&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs231n.github.io/convolutional-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n Convolutional Neural Networks for Visual Recognition&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="67001" LastEditDate="2019-02-05T02:02:16.477" LastActivityDate="2019-02-05T02:02:16.477" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28420" PostTypeId="1" CreationDate="2018-02-28T18:49:33.867" Score="0" ViewCount="2271" Body="&lt;p&gt;I am new to machine learning, though I have a background in statistics. But I had a question about $k$-fold cross-validation. So I understand the basic idea that we divide the dataset into $k$ partitions and then train a model on $k-1$ partitions while testing on the $k$th partition that was left out. So we don't want to train the model over the entire dataset, but instead over just our $k-1$ partitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question was how do we handle changes in the model parameters with each fold in the process. Another way of asking this question is how do we choose the model to test? So when you train on $k-1$ folds, in each iteration there will be a subtle change in the parameter estimates--namely the $\beta_{0...p}$s will change. So when we test the model against the testing partition, we are not testing the same model each time--because the parameter coefficients are different. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that the $\beta$s change for each fold, how do we choose which model to use? And once we choose a model, do we need to test only this model for the $k-fold$ cross-validation, for otherwise we are testing different models and then averaging their prediction error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any clarification would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="40791" LastActivityDate="2020-04-03T20:05:46.620" Title="k-fold cross-validation: model selection or variation in models when using k-fold cross validation" Tags="|cross-validation|" AnswerCount="3" CommentCount="13" ContentLicense="CC BY-SA 3.0" />
  <row Id="28422" PostTypeId="2" ParentId="26449" CreationDate="2018-02-28T19:15:55.243" Score="1" Body="&lt;p&gt;I cannot tell from your question how adept you are at mathematics or where your learning stops. I'll assume since you are a computer software engineer that you're familiar with algebra, geometry, and perhaps some calculus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd recommend you start your learning by reading up on statistics and understanding concepts like descriptives, exploratory data analysis, correlation, distributions, and so on. I see that you prefer books rather than videos, so I'll meet you half way and provide you with a few books that are online, as well as a book or two that you can buy in print.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, I'd recommend &lt;a href=&quot;https://onlinecourses.science.psu.edu/statprogram/programs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Penn State's online graduate course curriculum in statistics&lt;/a&gt;. You can explore each of their courses using the menu on the left. Once you select a course, scroll down on the course's webpage and click on the link that reads &quot;online course notes&quot;. The course notes for these courses are much more than notes and read like full books. They are very instructive. Also, check out &lt;a href=&quot;https://onlinecourses.science.psu.edu/statprogram/node/106&quot; rel=&quot;nofollow noreferrer&quot;&gt;Penn State's online undergraduate course curriculum in statistics&lt;/a&gt;, too, in case you find something in the graduate coursework that is too advanced and want a &quot;simpler&quot; explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, review the &lt;a href=&quot;http://www.biostathandbook.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Handbook of Biological Statistics&lt;/a&gt; by John H. McDonald. Don't let the title fool you; this book is an excellent primer on statistics and data analysis that is applicable to any domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third, review &lt;a href=&quot;http://www.jerrydallal.com/lhsp/lhsp.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Little Handbook of Statistics&lt;/a&gt; by Gerard Dallal. Again, don't let the title fool you; this book is another gem that walks you through some important statistics fundamentals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fourth, check out the book &lt;a href=&quot;http://greenteapress.com/thinkstats/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Think Stats&lt;/a&gt; by Allen Downey. There's a free version online of an earlier edition; the most recent edition you'll have to buy. It's worth it though, especially if you work in Python. In this book, the author teaches you statistics and data analysis using Python to analyze real-world (toy) datasets. This is a really great book to work through.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, check out &lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/149190142X&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science from Scratch&lt;/a&gt; by Joel Grus. This book focuses more on data analysis (instead of statistics fundamentals) and places a greater emphasis on machine learning and modeling. It uses Python (and the Python data science stack) to walk you through analyzing and conducting predictive analytics on real-world (toy) datasets. Another great book to work through.&lt;/p&gt;&#xA;" OwnerUserId="46546" LastActivityDate="2018-02-28T19:15:55.243" CommentCount="0" CommunityOwnedDate="2018-02-28T19:15:55.243" ContentLicense="CC BY-SA 3.0" />
  <row Id="28423" PostTypeId="2" ParentId="28420" CreationDate="2018-02-28T19:50:05.970" Score="2" Body="&lt;p&gt;There are two main (set of) things your model needs to learn from training: &lt;strong&gt;Parameters&lt;/strong&gt; and &lt;strong&gt;Hyperparameters&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Parameters are those that can be inferred (learnt) from data. These are internal to the model and will be calculated in the training phase during. The hyperparameters however, cannot be learnt from data and need to be decided (make assumptions) before training. As you come from statistics, an example could be modelling a distribution from a set of samples. Parameters could be mean and std, and hyperparameter could be selecting a Normal distribution or a Beta distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have hyperparameters (e.g. simple regression model), you only need to train the model and let the algorithm find the optimal weights (parameters) using the data. However, if you want to use let's say Support Vector Machines, then you need to provide some hyperparameters such as the cost, kernel, parameters of the kernel, and so on... before starting the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross Validation can be used to find the optimal hyperparameters. It is done by creating a grid of potential hyperparameter values and training each combination of them against the whole data, performing k trainings of k-1 portions (or &lt;em&gt;folds&lt;/em&gt;) of data. For each specific combination of hyperparameters the model is then trained and evaluated k times, getting a number of predictions equal to the number of folds (k) for each hyperparameter combination. The combination that yields better results based on the specified metric, is considered the best option (final model).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at this nice &lt;a href=&quot;https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; for further clarification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross validation procedure:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/epAqA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/epAqA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45166" LastEditorUserId="86627" LastEditDate="2020-04-03T20:05:46.620" LastActivityDate="2020-04-03T20:05:46.620" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28425" PostTypeId="2" ParentId="28420" CreationDate="2018-02-28T21:03:57.483" Score="1" Body="&lt;p&gt;I think @TitoOrt gets at the bulk of the confusion here, you're trying to explain how to get back the training parameters.  When it seems like the point the authors are making is that you can use X-Fold (for avoiding confusion with my example KNN) to find a hyperparameter.  Let me better explain with K-Nearest Neighbor:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you were trying to find the best K to use with the Iris set, here are error rates for K {2, 3, 4, 5}:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import cluster, datasets, mixture&#xA;from sklearn.neighbors import KNeighborsClassifier&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;data = sklearn.datasets.load_iris()&#xA;X = data[&quot;data&quot;]&#xA;Y = data[&quot;target&quot;]&#xA;&#xA;accuracy = []&#xA;for n in range(2, 6):&#xA;    # 70% holdout for training&#xA;    x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.7)&#xA;&#xA;    model = KNeighborsClassifier(n_neighbors=n)&#xA;    model.fit(x_train, y_train)&#xA;&#xA;    score = model.score(x_test, y_test)&#xA;    accuracy.append((n, score))&#xA;&#xA;# accuracy, number of clusters and the accuracy of the model&#xA;# [(2, 0.96190476190476193),&#xA;#  (3, 0.92380952380952386),&#xA;#  (4, 0.97142857142857142),&#xA;#  (5, 0.94285714285714284)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However this brings up a really easy criticism, how do I know we didn't just get lucky in picking the data to pass into the model when K was set at 4?  (Thankfully for this example, that's probably the seeing as there is only suppose to be 3 classes).  So in order to overcome this we use X-Fold to average out the bias that comes from sampling data, and thus better trust the accuracy that we get from picking K.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;accuracy = []&#xA;for n in range(2, 6):&#xA;    k_accuracy = []&#xA;&#xA;    # splitting and running this loop 10 times&#xA;    kf = KFold(n_splits=10)&#xA;    for train_index, test_index in kf.split(X):&#xA;&#xA;        model = KNeighborsClassifier(n_neighbors=n)&#xA;        model.fit(X[train_index], Y[train_index])&#xA;&#xA;        score = model.score(X[test_index], Y[test_index])&#xA;        k_accuracy.append(score)&#xA;&#xA;    # get the averaging of each 10 kfold runs&#xA;    score = sum(k_accuracy) / 10&#xA;    accuracy.append((n, score))&#xA;&#xA;# accuracy&#xA;# [(2, 0.93333333333333335),&#xA;#  (3, 0.94666666666666688),&#xA;#  (4, 0.92666666666666675),&#xA;#  (5, 0.93333333333333335)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;First of all we've eliminated the K=4 option, and the actual K=3 emerges as the winner.  With more folds our accuracy can be better trusted as well.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's connect this back to your original question though.  Each radius attached to each cluster will be different every time you run this algorithm because we're fitting to the data.  So yes you are right that you will never be able to exactly replicate the radius (parameter) choices that were made to get a certain accuracy, but that is not the goal of cross-validation.  The more important aspect of folding was to pick the best K.  Once we know that we can train to our hearts content.&lt;/p&gt;&#xA;" OwnerUserId="46540" LastActivityDate="2018-02-28T21:03:57.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28426" PostTypeId="1" AcceptedAnswerId="28427" CreationDate="2018-02-28T21:07:32.770" Score="16" ViewCount="79398" Body="&lt;p&gt;After I developed my predictive model using Random Forest I get the following metrics:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        Train Accuracy ::  0.9764634601043997&#xA;        Test Accuracy  ::  0.7933284397683713&#xA;         Confusion matrix  [[28292  1474]&#xA;                            [ 6128   889]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the results from this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  training_features, test_features, training_target, test_target, = train_test_split(df.drop(['bad_loans'], axis=1),&#xA;                                                  df['target'],&#xA;                                                  test_size = .3,&#xA;                                                  random_state=12)&#xA;clf = RandomForestClassifier()&#xA;trained_model = clf.fit(training_features, training_target)&#xA;trained_model.fit(training_features, training_target)&#xA;predictions = trained_model.predict(test_features)      &#xA;&#xA;Train Accuracy: accuracy_score(training_target, trained_model.predict(training_features))&#xA;Test Accuracy: accuracy_score(test_target, predictions)&#xA;Confusion Matrix: confusion_matrix(test_target, predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However I'm getting a little confuse to interpret and explain this values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What exactly this 3 measures tell me about my model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24719" LastActivityDate="2019-01-11T11:26:39.380" Title="Train Accuracy vs Test Accuracy vs Confusion matrix" Tags="|python|predictive-modeling|accuracy|confusion-matrix|classifier|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28427" PostTypeId="2" ParentId="28426" CreationDate="2018-02-28T22:12:02.537" Score="25" Body="&lt;h1&gt;Definitions&lt;/h1&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Accuracy: The amount of correct classifications / the total amount&#xA;of classifications. &lt;/li&gt;&#xA;&lt;li&gt;The train accuracy: The accuracy of a model on examples it was constructed on. &lt;/li&gt;&#xA;&lt;li&gt;The test accuracy is the accuracy of a model on examples it hasn't seen. &lt;/li&gt;&#xA;&lt;li&gt;Confusion matrix: A tabulation of the predicted class (usually&#xA;vertically) against the actual class (thus horizontally).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h1&gt;Overfitting&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;What I would make up of your results is that your model is &lt;strong&gt;overfitting&lt;/strong&gt;. You can tell that from  the large difference in accuracy between the test and train accuracy. Overfitting means that it learned rules  specifically for the train set, those rules do not generalize well beyond the train set.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your confusion matrix tells us how much it is overfitting, because your largest class makes up over 90% of the population. Assuming that you test and train set have a similar distribution, any useful model would have to score more than 90% accuracy:  A simple 0R-model would. Your model scores just under 80% on the test set.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;In depth look at the confusion matrix&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If you would look at the confusion matrix relatively (in percentages) it would look like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;               Actual    TOT&#xA;               1    2&#xA;Predicted 1 | 77% | 4% | 81%  &#xA;Predicted 2 | 17% | 2% | 19%&#xA;TOT         | 94% | 6% |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can infer from the total in the first row that your model predicts &lt;strong&gt;Class 1&lt;/strong&gt; 81% of the time, while the actual occurrence of &lt;strong&gt;Class 1&lt;/strong&gt; is 94%. Hence your model is underestimating this class. &lt;em&gt;It could be the case that it learned specific (complex) rules on the train set, that work against you in the test set.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It could also be worth noting that even though the false negatives of &lt;strong&gt;Class 1&lt;/strong&gt; (17%-point, row 2, column 1)) are hurting your overall performance most, the false negatives of &lt;strong&gt;Class 2&lt;/strong&gt; (4%-point, row 1 column 2) are actually more common with respect to the total population of the respective classes (94%, 6%). This means that your model is bad at predicting &lt;strong&gt;Class 1&lt;/strong&gt;, but even worse at predicting &lt;strong&gt;Class 2&lt;/strong&gt;. The accuracy just for &lt;strong&gt;Class 1&lt;/strong&gt; is 77/99 while the accuracy for &lt;strong&gt;Class 2&lt;/strong&gt; is 2/6. &lt;/p&gt;&#xA;" OwnerUserId="27432" LastEditorUserId="27432" LastEditDate="2018-03-01T10:39:20.850" LastActivityDate="2018-03-01T10:39:20.850" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28428" PostTypeId="2" ParentId="27152" CreationDate="2018-02-28T22:28:06.180" Score="1" Body="&lt;p&gt;There's a couple of tools that already do this. The ones that I'm familiar with are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;IBM Watson Analytics&lt;/li&gt;&#xA;&lt;li&gt;Datarobot&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Both of those frameworks will handle what you're looking for. Of course, they're not free, but they will do the work you seek. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-28T22:28:06.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28429" PostTypeId="2" ParentId="26449" CreationDate="2018-02-28T22:32:10.463" Score="1" Body="&lt;p&gt;Keep in mind that while I have a Masters in Applied Statistics, I'm going to give you a very simple answer: take a course on probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of the modern ML programming frameworks take a large majority of the math out of data science; you really just won't need it in most scenarios. But you will always need the ability to understand your results and the majority of results are expressed in probabilities. If I was new to data science I would take a (brief) course on probabilities, seek to understand what proportions and percentages really mean and then I would work to know a framework (like Tensorflow) really, really well. If you can do that, you can write some really interesting algorithms and not have to be obsessive about the math. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-28T22:32:10.463" CommentCount="0" CommunityOwnedDate="2018-02-28T22:32:10.463" ContentLicense="CC BY-SA 3.0" />
  <row Id="28430" PostTypeId="2" ParentId="28418" CreationDate="2018-02-28T22:35:42.743" Score="1" Body="&lt;p&gt;It all depends on the model you are running and your chosen sample sizing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sample sizing is obvious, so I won't go into that (or at least it should be obvious to you vis-a-vis how R handles data). But the chosen model can have an effect. For example, neural networks are one of the rare problems in computing that you actually &lt;em&gt;can&lt;/em&gt; throw more hardware at to improve. That's going to be a while different experience than, say, gradient boosting or something along those lines. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-02-28T22:35:42.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28431" PostTypeId="2" ParentId="20261" CreationDate="2018-02-28T23:51:57.743" Score="1" Body="&lt;p&gt;Following up on the comment about deep learning, with high dimensional time series data you would be much better served with a recurrent-type of deep model. For example, an LSTM is a very good starting point with high-dimensional data. This may be a good place to start: &lt;a href=&quot;https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although CNNs are very useful for high-dimensional data, when you have a time series, it's best to start with a model that is designed for a time series. A CNN may do well, and you should compare your results to a CNN, but it is not a time series model.&lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-02-28T23:51:57.743" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28432" PostTypeId="1" AcceptedAnswerId="28433" CreationDate="2018-02-28T23:57:25.473" Score="0" ViewCount="54" Body="&lt;p&gt;I will try to implement a k-means algorithm over this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Team_Categorical  CreditAmount_Numeric   Retired?_Binary&#xA;       A                 15.3                   1&#xA;       B                 12                     0&#xA;       C                 6.2                    1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In order to apply k-means algorithm I need to transform the categorical variables into numeric. So I will apply binning transformaton over the Team_Categorical field. So I will have this dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A    B   C  CreditAmount_Numeric   Retired?_Binary&#xA;1    0   0          15.3                 1&#xA;0    1   0          12                   0&#xA;0    0   1          6.2                  1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: Should I transform CreditAmount_Numeric to binary too? &lt;/p&gt;&#xA;" OwnerUserId="24790" LastEditorUserId="32492" LastEditDate="2018-03-01T08:16:54.327" LastActivityDate="2018-03-01T08:16:54.327" Title="Binning which variables?" Tags="|predictive-modeling|preprocessing|binary|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28433" PostTypeId="2" ParentId="28432" CreationDate="2018-03-01T01:21:27.780" Score="1" Body="&lt;p&gt;To answer your question, you need not transform the numeric to binary variable(you meant binning right?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will try to explain it with the above example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5Gc1B.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5Gc1B.png&quot; alt=&quot;snapshot&quot;&gt;&lt;/a&gt;&#xA;Let's start with row-1 where A = 1 , B =0 , C=0 so it means that you are talking about A. So the value of CreditAmount_Numeric belongs to A and the retired binary value also belongs to A. Similarly, you can deduce for the rest records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion is that you need not do any transformation on your numeric variable.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-01T01:28:21.390" LastActivityDate="2018-03-01T01:28:21.390" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28434" PostTypeId="2" ParentId="19575" CreationDate="2018-03-01T01:46:19.050" Score="6" Body="&lt;p&gt;The problem occurs due to &lt;code&gt;DMatrix..num_col()&lt;/code&gt; only returning the amount of non-zero columns in a sparse matrix. Hence, if both train &amp;amp; test data have the same amount of non-zero columns, everything works fine. Otherwise, you end up with different feature names lists. There're currently three solutions to work around this problem:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;realign the columns names of the train dataframe and test dataframe using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_df = test_df[train_df.columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;save the model first and then load the model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;change the test data into array before feeding into the model, ie: use, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_df.values instead of test_df&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47022" LastEditorUserId="29575" LastEditDate="2018-03-01T02:12:40.393" LastActivityDate="2018-03-01T02:12:40.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28435" PostTypeId="2" ParentId="28420" CreationDate="2018-03-01T02:15:40.723" Score="0" Body="&lt;p&gt;The primary use of CV is for tuning the model.  You use the average CV value to see how your model might behave on new data. Once you’ve settled for the best hyper parameters, your retrain on the whole train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then people noticed that you can also use the models trained on each fold, by averaging their predictions on the test set.  That’s the second way of using CV.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best approach depend son the data.  Sometimes retraining on the full set is better than averaging fold predictions, sometimes the converse is true.&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-01T02:15:40.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28436" PostTypeId="2" ParentId="28412" CreationDate="2018-03-01T02:21:15.520" Score="0" Body="&lt;p&gt;If you are just looking for generating Confusion Matrix, then you can try using this command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#for generating confusion matrix    &#xA;sklearn.metrics.confusion_matrix(y_true, y_pred, labels=None, sample_weight=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are just looking for binding predicted and actual values, you can use this command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#For Arrays you can use this&#xA;numpy.concatenate() &#xA;&#xA;#for dataframe you can use this&#xA;DataFrame.append(other, ignore_index=False, verify_integrity=False)[source]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for better understaning you can go through this &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;,&lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.concatenate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-2&lt;/a&gt;,&lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.append.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-3&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-03-01T02:21:15.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28437" PostTypeId="2" ParentId="28304" CreationDate="2018-03-01T04:07:31.350" Score="1" Body="&lt;p&gt;If I'm not mistaken, you are looking for some metrics to investigate if two columns of values represent similar information. Right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the order of numbers is fixed for both columns (such that the ith elements of both columns relate to the same record) and hence both columns have the same length, then you can use a wide range of techniques to measure the similarity/distance between the two vectors such as &lt;strong&gt;cosine similarity&lt;/strong&gt;, &lt;strong&gt;Pearson correlation coefficient&lt;/strong&gt;, &lt;strong&gt;mutual information&lt;/strong&gt;, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the order of numbers in the columns are different or they basically refer to different concepts, then the above techniques are meaningless. In this case, you can look at these columns as two populations of samples. In this case, you may compare their statistical distributions using their distribution parameters (eg &lt;strong&gt;mean&lt;/strong&gt; and &lt;strong&gt;std&lt;/strong&gt; for a normal distribution) or statistical tests (eg &lt;strong&gt;ANOVA&lt;/strong&gt;). In some cases, the number of items (population size) can be different.&lt;/p&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-03-01T04:07:31.350" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28438" PostTypeId="1" CreationDate="2018-03-01T05:10:04.007" Score="3" ViewCount="3127" Body="&lt;p&gt;I am using &lt;strong&gt;opencv&lt;/strong&gt; module in python to extract sequences of images(frames) from video (from a file)&#xA;My code for extracting images from video goes like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2     &#xA;import os&#xA;os.chdir('D:/Python images/videofolder') &#xA;cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')&#xA;cap.isOpened()&#xA;success,image = cap.read()&#xA;count = 1&#xA;success = True&#xA;while success:&#xA;     success,image = cap.read()&#xA;     cv2.imwrite(&quot;image%d.jpg&quot; % count, image)     # save frame as JPEG file&#xA;count += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want my code to select multiple of 'n' images &#xA;eg: to select &amp;amp; save multiple of 5 images(to extract multiple of 5) so that I have 5th, 10th, 15th... images.&#xA;Please guide me.&lt;/p&gt;&#xA;" OwnerUserId="30006" LastActivityDate="2019-04-29T01:01:35.287" Title="to extract multiple of n images using open cv in python" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28439" PostTypeId="2" ParentId="28438" CreationDate="2018-03-01T05:32:02.623" Score="1" Body="&lt;p&gt;You can use the modulo operator to identify multiples of count. Whenever it is a multiple of 5 save the image, otherwise just disregard it and move to the next iteration of the while loop.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure the counter is outside the if statement. It will count every iteration. That is how we will determine if the current image is a multiple of 5.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2     &#xA;import os&#xA;os.chdir('D:/Python images/videofolder') &#xA;cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')&#xA;cap.isOpened()&#xA;&#xA;count = 0&#xA;success = True&#xA;while success:&#xA;    success,image = cap.read()&#xA;    if count % 5 == 0:&#xA;        cv2.imwrite(&quot;image%d.jpg&quot; % count, image) # save frame as JPEG file&#xA;    count += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-01T05:32:02.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28440" PostTypeId="1" CreationDate="2018-03-01T05:32:03.123" Score="5" ViewCount="12023" Body="&lt;p&gt;I'm looking for a way to create a conditional loss function that looks like this: there is a vector of labels, say l (l has the same length as the input x), then for a given input (y_true, y_pred, l) the loss should be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def conditional_loss_function(y_true, y_pred, l):&#xA;    loss = if l is 0: loss_funtion1&#xA;           if l is 1: loss_funtion2&#xA;    return loss&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is like a kind of semi-supervised loss funtion.&lt;/p&gt;&#xA;" OwnerUserId="47036" LastEditorUserId="47036" LastEditDate="2018-03-01T05:51:31.803" LastActivityDate="2018-03-02T05:17:28.730" Title="Custom conditional loss function in Keras" Tags="|keras|loss-function|semi-supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28441" PostTypeId="1" AcceptedAnswerId="28442" CreationDate="2018-03-01T06:13:32.277" Score="28" ViewCount="23599" Body="&lt;p&gt;I understand &lt;code&gt;cross_validate&lt;/code&gt; and how it works, but now I am confused about what  &lt;code&gt;cross_val_score&lt;/code&gt; actually does. Can anyone give me some example?&lt;/p&gt;&#xA;" OwnerUserId="46433" LastEditorUserId="29169" LastEditorDisplayName="user10640" LastEditDate="2021-03-11T19:25:17.320" LastActivityDate="2021-03-11T19:25:54.597" Title="What is the difference between cross_validate and cross_val_score?" Tags="|machine-learning|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28442" PostTypeId="2" ParentId="28441" CreationDate="2018-03-01T06:34:23.093" Score="22" Body="&lt;p&gt;&lt;code&gt;cross_val_score&lt;/code&gt; is a helper function on the estimator and the dataset.&lt;/p&gt;&#xA;&lt;p&gt;Would explain it with an example:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.model_selection import cross_val_score&#xA;&amp;gt;&amp;gt;&amp;gt; clf = svm.SVC(kernel='linear', C=1)&#xA;&amp;gt;&amp;gt;&amp;gt; scores = cross_val_score(clf, iris.data, iris.target, cv=5)&#xA;&amp;gt;&amp;gt;&amp;gt; scores                                              &#xA;array([ 0.96...,  1.  ...,  0.96...,  0.96...,  1.        ])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;This example demonstrates how to estimate the accuracy of a linear kernel support vector machine on the iris dataset by splitting the data, fitting a model and computing the score 5 consecutive times (with different splits each time)&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;cross_validate&lt;/code&gt; function differs from &lt;code&gt;cross_val_score&lt;/code&gt; in two ways -&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It allows specifying multiple metrics for evaluation.&lt;/li&gt;&#xA;&lt;li&gt;It returns a dict containing training scores, fit-times and score-times in&#xA;addition to the test score.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Note: When the cv argument is an integer, &lt;code&gt;cross_val_score&lt;/code&gt; uses the &lt;strong&gt;KFold&lt;/strong&gt; or &lt;strong&gt;StratifiedKFold&lt;/strong&gt; strategies by default, the latter being used if the estimator derives from ClassifierMixin&lt;/p&gt;&#xA;&lt;p&gt;You can go through this &lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; for better understanding&lt;/p&gt;&#xA;&lt;p&gt;Different &lt;a href=&quot;https://www.programcreek.com/python/example/75177/sklearn.cross_validation.cross_val_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;examples&lt;/a&gt; using &lt;code&gt;cross_val_score&lt;/code&gt;, you can go through about its different implementations.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="29169" LastEditDate="2021-03-11T19:25:54.597" LastActivityDate="2021-03-11T19:25:54.597" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28443" PostTypeId="1" AcceptedAnswerId="28444" CreationDate="2018-03-01T06:51:03.637" Score="1" ViewCount="224" Body="&lt;p&gt;MNIST is a famous data set of hand written digits. Suppose we knew who wrote digits for example, female, left handed, 25 yrs old. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would I use these information in CNN in tensorflow? or any other library. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Digits are images and that what CNN handles well but gender, dominant hand, and age are not images. How would you use that information?&lt;/p&gt;&#xA;" OwnerUserId="47040" LastEditorUserId="40853" LastEditDate="2018-03-01T06:57:26.817" LastActivityDate="2018-03-01T07:00:06.313" Title="What if MNIST dataset had another feature" Tags="|python|tensorflow|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28444" PostTypeId="2" ParentId="28443" CreationDate="2018-03-01T07:00:06.313" Score="3" Body="&lt;p&gt;Convolutional layers are useful for images because they take into consideration the neighborhood of pixels. However, for labels like gender and handedness a convolutional layer may not be particularly useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, after the convolutional layers you usually tend to place some densely connected layers. It is there that you may want to add these additional features. When you reshape the 2D matrices which results from the convolutions, you can concatenate the additional features, then feed this new vector to your Dense layer.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-01T07:00:06.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28445" PostTypeId="2" ParentId="28440" CreationDate="2018-03-01T07:17:53.863" Score="1" Body="&lt;p&gt;You should be able to solve this with currying. Make a function that takes the label as input and returns a function which takes &lt;code&gt;y_true&lt;/code&gt; and &lt;code&gt;y_pred&lt;/code&gt; as input. Note that the label needs to be a constant or a tensor for this to work.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def conditional_loss_function(l):&#xA;    def loss(y_true, y_pred):&#xA;        if l == 0: &#xA;            return loss_funtion1(y_true, y_pred)&#xA;        else: &#xA;            return loss_funtion2(y_true, y_pred)&#xA;    return loss&#xA;&#xA;model.compile(loss=conditional_loss_function(l), optimizer=...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Small working example with different loss function depending on the label:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# load pima indians dataset&#xA;dataset = numpy.loadtxt(&quot;pima-indians-diabetes.csv&quot;, delimiter=&quot;,&quot;)&#xA;data = dataset[:,0:8]&#xA;label = dataset[:,8]&#xA;&#xA;X = Input(shape=(8,))&#xA;Y = Input(shape=(1,))&#xA;x = Dense(12, input_dim=8, activation='relu')(X)&#xA;x = Dense(8, activation='relu')(x)&#xA;predictions = Dense(1, activation='sigmoid')(x)&#xA;&#xA;def custom_loss(l):&#xA;    def loss(y_true, y_pred):&#xA;        if l == 0:&#xA;            return binary_crossentropy(y_true, y_pred)&#xA;        else: &#xA;            return mean_squared_error(y_true, y_pred)&#xA;    return loss    &#xA;&#xA;# Compile model&#xA;model = Model(inputs=[X, Y], outputs=predictions)&#xA;model.compile(loss=custom_loss(Y), optimizer='adam', metrics=['accuracy'])&#xA;&#xA;# Fit the model&#xA;model.fit(x=[data, label], y=label, epochs=150)&#xA;&#xA;# evaluate the model&#xA;scores = model.evaluate([data, label], label)&#xA;print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32492" LastEditorUserId="32492" LastEditDate="2018-03-02T05:17:28.730" LastActivityDate="2018-03-02T05:17:28.730" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28446" PostTypeId="1" AcceptedAnswerId="28469" CreationDate="2018-03-01T09:26:44.967" Score="4" ViewCount="5502" Body="&lt;p&gt;I have multiple time series sequences and I want for each new time series to find the most alike old one.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I found that I can use the sum of errors between points. Is this a good approach?&lt;/li&gt;&#xA;&lt;li&gt;Is there a way to be able to compare sequences with different lengths (maybe a sequence look like a subsequence of another sequence)?&lt;/li&gt;&#xA;&lt;li&gt;Will scaling the data before comparing make difference?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="44637" LastEditorUserId="29575" LastEditDate="2018-03-01T14:45:13.803" LastActivityDate="2018-11-02T10:55:15.307" Title="How to compare Timeseries Sequences?" Tags="|time-series|beginner|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28447" PostTypeId="2" ParentId="28446" CreationDate="2018-03-01T10:13:40.303" Score="1" Body="&lt;p&gt;I think you are looking for the distance between two functions, which to my knowledge is a rather complex mathematical field (sorry for not being able to give a reference, but I know I read a book about it once).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answere your second point: I got some decent results using dynamic time warping (&lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Dynamic_time_warping&lt;/a&gt;). It should be available for every software kit. In python, there is a package fastdtw (&lt;a href=&quot;https://pypi.python.org/pypi/fastdtw&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pypi.python.org/pypi/fastdtw&lt;/a&gt;) that does a good job.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I think that scaling will make a difference, no matter which method you use.&lt;/p&gt;&#xA;" OwnerUserId="43469" LastActivityDate="2018-03-01T10:13:40.303" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28448" PostTypeId="2" ParentId="28376" CreationDate="2018-03-01T10:19:54.667" Score="1" Body="&lt;p&gt;You have two options to do this, one less elegant than the other:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can drop whatever column you don't want to use and add them together after getting the scores. Though you might need to reorder columns etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4OOCc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4OOCc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or...&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;There is a block called &lt;strong&gt;Edit Metadata&lt;/strong&gt; which can be used to do what you need. Select whatever columns from the column selector you don't want to use and use the option &lt;code&gt;Fields &amp;gt; Clear Feature&lt;/code&gt; from the block properties. These columns won't be used in the calculations but will be present when you visualise the Score Model output together with the rest of the columns and the scores. Moreover, columns in that output statistics will be labelled as &lt;em&gt;Feature&lt;/em&gt; type if they were used in the calculations. So you straightaway know which columns were actually used in the calculations or not. Block &lt;a href=&quot;https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/edit-metadata&quot; rel=&quot;nofollow noreferrer&quot;&gt;help&lt;/a&gt; is quite useful here.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YXsZ7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YXsZ7.png&quot; alt=&quot;Edit Metadata Block&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UykMc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UykMc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-01T10:19:54.667" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28449" PostTypeId="1" CreationDate="2018-03-01T10:29:16.843" Score="2" ViewCount="583" Body="&lt;p&gt;I have several random forest models that work well. Now, I would like to do feature selection based on this models. How do I find out which features were used frequently by a model e.g. RandomForestClassifier in scikit-learn?&lt;/p&gt;&#xA;" OwnerUserId="26202" LastEditorUserId="26202" LastEditDate="2019-04-23T16:52:16.670" LastActivityDate="2019-04-23T16:52:16.670" Title="How to find out which features were used frequently by RandomForestClassiferer in scikit-learn?" Tags="|scikit-learn|random-forest|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28450" PostTypeId="2" ParentId="28449" CreationDate="2018-03-01T10:33:28.613" Score="2" Body="&lt;p&gt;Feature importance it is a property of the random forrest classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See an example &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-03-01T10:33:28.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28451" PostTypeId="1" CreationDate="2018-03-01T10:43:52.663" Score="2" ViewCount="449" Body="&lt;p&gt;I need to create an app to predict events.&lt;/p&gt;&#xA;&lt;p&gt;I'll explain my requirements: I have a set of intermediate events (eg.: 1, 2, 3...9) and a set of final events (eg.: A for OK and B for KO). These events bring a correlation id to relate each other.&lt;/p&gt;&#xA;&lt;p&gt;What ML algorithm can I use to predict if a sequence of events could cause a future KO event (B in my example) or not?&lt;/p&gt;&#xA;&lt;p&gt;I need an unsupervised algorithm.&lt;/p&gt;&#xA;&lt;p&gt;Any idea about the algorithm?&lt;/p&gt;&#xA;" OwnerUserId="47048" LastEditorUserId="97640" LastEditDate="2021-06-23T13:49:31.027" LastActivityDate="2023-09-04T21:00:55.083" Title="Event prediction sequence" Tags="|machine-learning|predictive-modeling|prediction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28454" PostTypeId="1" CreationDate="2018-03-01T12:03:44.437" Score="1" ViewCount="1034" Body="&lt;p&gt;1- Would it always be beneficial to remove highly correlated features prior to training a model? If not, why not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Would you perform One Hot encoding where applicable, prior to removing highly correlated features ? (taking into account a dummy variable may be highly correlated with another variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="46374" LastActivityDate="2018-03-05T18:31:53.967" Title="Is it good practice to always remove highly correlated variables?" Tags="|machine-learning|categorical-data|correlation|encoding|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28456" PostTypeId="2" ParentId="28449" CreationDate="2018-03-01T12:55:10.053" Score="0" Body="&lt;p&gt;Just to add to the feature Importance of the RF's , don't forget to use the &lt;strong&gt;hierarchical clustering&lt;/strong&gt; from &lt;strong&gt;scipy&lt;/strong&gt; and plot the &lt;strong&gt;dendograms&lt;/strong&gt;..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will help you find the similar columns..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-01T12:55:10.053" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28457" PostTypeId="1" CreationDate="2018-03-01T13:05:24.403" Score="0" ViewCount="1386" Body="&lt;p&gt;I'm developing a multi-label classifier using the Keras library, but I am stuck with a relatively low accuracy of about 2% and my loss value per epoch is around -10,000 with little change between epoch. I'm relatively new to Keras but need to use it for my university work, and I am wondering what is going wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model currently looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(1024, activation='relu', input_shape=X.shape[1:]))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(512, activation='relu'))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(3, activation='sigmoid'))&#xA;&#xA;sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)&#xA;model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])&#xA;&#xA;model.fit(X, y, batch_size=20, epochs=100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this case, the input_shape[1:] is 4096, a flattened feature array and the multi-label I'm trying to classify has a first value that ranges from 1 - 163, the second value ranges from 1 - 2000, and the third value ranges from 2007 - 2014.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue that I am having is while running my model I get a result that does not change during each epoch as is shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MfTmS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MfTmS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know what I'm doing wrong, I'll include how the features and the labels look below in respective order: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[['0.0' ' 0.0' ' 0.0' ..., ' 5.7333' ' 0.0' ' 2.58643']&#xA; ['0.0' ' 3.12623' ' 0.0' ..., ' 0.0' ' 0.0' ' 2.93147']&#xA; ['0.0' ' 0.0' ' 0.0' ..., ' 7.49419' ' 0.0' ' 1.55746']&#xA; ['0.0' ' 0.0' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.3666']&#xA; ['0.0' ' 4.67996' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.0']]&#xA;&#xA;[['78' '1' '2010']&#xA; ['78' '1' '2010']&#xA; ['78' '1' '2014']&#xA; ['78' '1' '2013']&#xA; ['78' '1' '2012']]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm at a loss of what to try and as I have mentioned, I'm new to Keras and deep learning so any help will be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="47061" LastEditorUserId="29575" LastEditDate="2018-03-01T14:49:10.907" LastActivityDate="2018-04-04T17:07:41.700" Title="Why does my loss value start at approximately -10,000 and my accuracy not improve?" Tags="|deep-learning|keras|loss-function|accuracy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28458" PostTypeId="1" CreationDate="2018-03-01T13:28:36.977" Score="0" ViewCount="279" Body="&lt;p&gt;As the question entails, I would like to know as a rule of thumb, an upper limit on the number of features I could have for a data set with 1,000,000 rows/observations, before hitting the &lt;em&gt;curse of dimensionality&lt;/em&gt;. If not a million observations, an answer with a rough estimate for the number of features for a data set with a specified size would be great too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="46374" LastActivityDate="2018-03-05T06:09:00.010" Title="Curse of dimensionality: Data set with 1 million rows, max number of features can I have approximately?" Tags="|machine-learning|dataset|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28461" PostTypeId="1" AcceptedAnswerId="28526" CreationDate="2018-03-01T14:17:47.957" Score="0" ViewCount="565" Body="&lt;p&gt;I've this code to print the importance of each variable on my model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;importances = trained_model.feature_importances_&#xA;std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)&#xA;indices = np.argsort(importances)[::-1]&#xA;&#xA;# Print the feature ranking&#xA;print(&quot;Feature ranking:&quot;)&#xA;&#xA;for f in range(training_features.shape[1]):&#xA;    print(&quot;%d. feature %d (%f)&quot; % (f + 1, indices[f], importances[indices[f]]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I print a lot of variables with feature ranking as 0.0. Should I remove that variables? I can I do it using Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.drop('Col_A', 1) WHERE importances[indices[f]] = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24719" LastActivityDate="2018-03-02T16:54:31.147" Title="Python - Feature Selection - Should I remove bad variables?" Tags="|python|feature-selection|dataframe|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28463" PostTypeId="1" CreationDate="2018-03-01T14:33:45.893" Score="9" ViewCount="5175" Body="&lt;p&gt;I want to do one-step-ahead predictions for time series with LSTM.&#xA;To understand the algorithm, I built myself a toy example: A simple autocorrelated process.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def my_process(n, p, drift=0, displacement=0):&#xA;    x = np.zeros(n)&#xA;&#xA;    for i in range(1, n):&#xA;        x[i] = drift * i + p * x[i-1] + (1-p) * np.random.randn()&#xA;    return x + displacement&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I built an LSTM model in Keras, following &lt;a href=&quot;https://www.kaggle.com/pablocastilla/predict-stock-prices-with-lstm&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt;.&#xA;I simulated processes with high autocorrelation &lt;code&gt;p=0.99&lt;/code&gt; of length &lt;code&gt;n=10000&lt;/code&gt;, trained the neural network on the first 80% of it and let it do one-step-ahead predictions for the remaning 20%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I set &lt;code&gt;drift=0, displacement=0&lt;/code&gt;, everything works fine:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/m3efr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m3efr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I set &lt;code&gt;drift=0, displacement=10&lt;/code&gt; and things went pear-shaped (notice the different scale on the y-axis):&#xA;&lt;a href=&quot;https://i.stack.imgur.com/VaMQw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VaMQw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not terribly surprising: LSTMs should be fed with normalized data!&#xA;So I normalized the data by rescaling it to the interval &lt;span class=&quot;math-container&quot;&gt;$[-1, 1]$&lt;/span&gt;.&#xA;Phew, things are fine again:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9nGs9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9nGs9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I set &lt;code&gt;drift=0.00001, displacement=10&lt;/code&gt;, normalized the data again and ran the algorithm on it.&#xA;This does not look good:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/iwrH5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iwrH5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently the LSTM cannot deal with a drift. &#xA;What to do? (Yes, in this toy example I could simply subtract the drift; but for real-world time series, this is much harder).&#xA;Maybe I could run my LSTM on the difference &lt;span class=&quot;math-container&quot;&gt;$X_{t} - X_{t-1}$&lt;/span&gt; instead of the original time series &lt;span class=&quot;math-container&quot;&gt;$X_t$&lt;/span&gt;. This will remove any constant drift from the time series.&#xA;But running the LSTM on the differenced time series does not work at all:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/B4yLg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B4yLg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: Why does my algorithm break down when I use it on the differenced time series?&#xA;What is a good way to deal with drifts in time series?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the full code for my model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;np.random.seed(42)&#xA;&#xA;from keras.layers.core import Dense, Activation, Dropout&#xA;from keras.layers.recurrent import LSTM&#xA;from keras.models import Sequential&#xA;&#xA;&#xA;# The LSTM model&#xA;my_model = Sequential()&#xA;&#xA;my_model.add(LSTM(input_shape=(1, 1), units=50, return_sequences=True))&#xA;my_model.add(Dropout(0.2))&#xA;&#xA;my_model.add(LSTM(units=100, return_sequences=False))&#xA;my_model.add(Dropout(0.2))&#xA;&#xA;my_model.add(Dense(units=1))&#xA;my_model.add(Activation('linear'))&#xA;&#xA;my_model.compile(loss='mse', optimizer='rmsprop')&#xA;&#xA;&#xA;def my_prediction(x, model, normalize=False, difference=False):&#xA;    # Plot the process x&#xA;    plt.figure(figsize=(15, 7))&#xA;    plt.subplot(121)&#xA;    plt.plot(x)&#xA;    plt.title('Original data')&#xA;&#xA;    n = len(x)&#xA;    thrs = int(0.8 * n)    # Train-test split&#xA;    # Save starting values for test set to reverse differencing&#xA;    x_test_0 = x[thrs + 1]&#xA;    # Save minimum and maximum on test set to reverse normalization&#xA;    x_min = min(x[:thrs])  &#xA;    x_max = max(x[:thrs])&#xA;&#xA;    if difference:&#xA;        x = np.diff(x)   # Take difference to remove drift&#xA;    if normalize:&#xA;        x = (2*x - x_min - x_max) / (x_max - x_min)   # Normalize to [-1, 1]&#xA;&#xA;    # Split into train and test set. The model will be trained on one-step-ahead predictions.&#xA;    x_train, y_train, x_test, y_test = x[0:(thrs-1)], x[1:thrs], x[thrs:(n-1)], x[(thrs+1):n]&#xA;&#xA;    x_train, x_test = x_train.reshape(-1, 1, 1), x_test.reshape(-1, 1, 1)&#xA;    y_train, y_test = y_train.reshape(-1, 1), y_test.reshape(-1, 1)&#xA;&#xA;    # Fit the model&#xA;    model.fit(x_train, y_train, batch_size=200, epochs=10, validation_split=0.05, verbose=0)&#xA;&#xA;    # Predict the test set&#xA;    y_pred = model.predict(x_test)&#xA;&#xA;    # Reverse differencing and normalization&#xA;    if normalize:&#xA;        y_pred = ((x_max - x_min) * y_pred + x_max + x_min) / 2&#xA;        y_test = ((x_max - x_min) * y_test + x_max + x_min) / 2  &#xA;    if difference:&#xA;        y_pred = x_test_0 + np.cumsum(y_pred)&#xA;        y_test = x_test_0 + np.cumsum(y_test)&#xA;&#xA;    # Plot estimation&#xA;    plt.subplot(122)&#xA;    plt.plot(y_pred[-100:], label='One-step-ahead-predictions')&#xA;    plt.plot(y_test[-100:], label='Actual data')&#xA;    plt.title('Prediction on test set')&#xA;    plt.legend()&#xA;    plt.show()&#xA;&#xA;# Make plots&#xA;x = my_process(10000, 0.99, drift=0, displacement=0)&#xA;my_prediction(x, my_model, normalize=False, difference=False)&#xA;&#xA;x = my_process(10000, 0.99, drift=0, displacement=10)&#xA;my_prediction(x, my_model, normalize=False, difference=False)&#xA;&#xA;x = my_process(10000, 0.99, drift=0, displacement=10)&#xA;my_prediction(x, my_model, normalize=True, difference=False)&#xA;&#xA;x = my_process(10000, 0.99, drift=0.00001, displacement=10)&#xA;my_prediction(x, my_model, normalize=True, difference=False)&#xA;&#xA;x = my_process(10000, 0.99, drift=0.00001, displacement=10)&#xA;my_prediction(x, my_model, normalize=True, difference=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2019-01-20T17:41:06.913" LastActivityDate="2019-01-20T17:41:06.913" Title="LSTM: How to deal with nonstationarity when predicting a time series" Tags="|python|keras|time-series|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28464" PostTypeId="1" CreationDate="2018-03-01T15:04:19.647" Score="1" ViewCount="454" Body="&lt;p&gt;I am trying to automatically categorize news articles according to their primary topics, i.e. politics, entertainment, sports, business, technology, health, etc.&lt;/p&gt;&#xA;&lt;p&gt;There are some labeled datasets out there, but ideally I would like to create my own (for potential commercial usage later on). I am using python, but an answer clear enough with relation to any language would be sufficient.&lt;/p&gt;&#xA;&lt;p&gt;So, what would be the best way to go about this task?&lt;/p&gt;&#xA;&lt;p&gt;My current thoughts are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Determine the most popular keywords for each category, then associate&#xA;each keyword/keyword set with each respective category, and then use&#xA;an algorithm to apply a &amp;quot;category&amp;quot; label to a large set of scraped&#xA;articled based on the predefined keyword sets.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Another option is to scrape article from specific sections of news&#xA;sites where the categories are already specified, and apply them to&#xA;each individual constituent article.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;After I have the trained dataset, I plan to implement Naive Bayes classification method to automatically categorize future articles.&lt;/p&gt;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&lt;p&gt;As you can see I have some ideas, but because the web is a large and magical place, I assume someone with previous experience doing something like this may be able to reduce my effort expenditure by guiding me toward the most feasible solution.&lt;/p&gt;&#xA;" OwnerUserId="40467" LastEditorUserId="29169" LastEditDate="2021-08-08T17:14:21.683" LastActivityDate="2023-01-11T13:05:00.593" Title="How can I create a &quot;trained&quot; dataset for categorizing news articles?" Tags="|python|classification|data-mining|nlp|naive-bayes-classifier|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28465" PostTypeId="1" CreationDate="2018-03-01T15:09:18.897" Score="0" ViewCount="36" Body="&lt;p&gt;Take the given values: &lt;code&gt;-2.998, 8.0, 2.5&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My normalisation function takes a sum of the absolute values &lt;code&gt;(13.498)&lt;/code&gt;, let's call that &lt;code&gt;x&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, each absolute value is divided by &lt;code&gt;x&lt;/code&gt; (essentially calculating the mean distribution) - here's the result: &lt;code&gt;0.222, 0.593, 0.185&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So our result values are spread between 0-1, (it's what I need), and works great. But what's the precise terminology for this type of normalisation? I want to be able to document and comment my code accurately.&lt;/p&gt;&#xA;" OwnerUserId="46924" LastEditorUserId="46924" LastEditDate="2018-03-01T15:31:38.570" LastActivityDate="2018-03-01T17:36:35.513" Title="Correct normalisation terminology" Tags="|machine-learning|neural-network|deep-learning|normalization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28467" PostTypeId="1" CreationDate="2018-03-01T15:53:13.183" Score="1" ViewCount="1343" Body="&lt;p&gt;I used Tensorflow Object Detection API for a custom dataset based on the instructions at this &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/using_your_own_dataset.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;help&lt;/a&gt; document.As required , collected the dataset,annotated it in PASCAL VOC XML format,split into training and test sets,generated tfrecords. Pets configuration for custom object detection and localization is used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just want to proceed further and get segmentation boundary rather than bounding box. The &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_locally.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; is clear till this point on directory structure and mechanism to prepare input data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instance Segmentation &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/instance_segmentation.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;help&lt;/a&gt; is unclear to me on two aspects: &#xA;  -Directory structure&#xA;  -Is it required to supply segment boundaries equal in number to that of images in training and test sets ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice that tfrecords are created using &lt;a href=&quot;https://gist.github.com/saghiralfasly/ee642af0616461145a9a82d7317fb1d6&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; script. Can this be reused to include segmentation data.&lt;/p&gt;&#xA;" OwnerUserId="35503" LastActivityDate="2018-03-01T15:53:13.183" Title="How to perform Instance Segmentation using Tensorflow?" Tags="|machine-learning|deep-learning|object-detection|faster-rcnn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28468" PostTypeId="1" AcceptedAnswerId="28471" CreationDate="2018-03-01T16:10:08.610" Score="0" ViewCount="463" Body="&lt;p&gt;I've this matrix confusion:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[9779  107]&#xA;[2227  148]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the accuracy of my model? My doubt is because the confusion matrix is calculated based on Test dataset so how can it evaluate the accuracy of my model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24790" LastActivityDate="2018-03-02T14:53:36.957" Title="Matrix Confusion - Get Model Precision" Tags="|accuracy|confusion-matrix|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28469" PostTypeId="2" ParentId="28446" CreationDate="2018-03-01T16:17:32.443" Score="7" Body="&lt;p&gt;The answer to your questions depend a lot on the &lt;em&gt;nature&lt;/em&gt; of the data represented in the time series. You should ask yourself some questions to better understand what might or might not work, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are the time sequences perfectly aligned?&lt;/li&gt;&#xA;&lt;li&gt;Are two slightly shifted time series considered similar or not?&lt;/li&gt;&#xA;&lt;li&gt;Are two time series with the same shape but different scale considered similar?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Normally&lt;/em&gt;, the answers to those questions are that series are not perfectly aligned and that variations in scale are also fine as long as the shape is similar. For these scenarios, the classical measure is &lt;a href=&quot;https://en.wikipedia.org/wiki/Dynamic_time_warping&quot; rel=&quot;noreferrer&quot;&gt;Dynamic Time Warping&lt;/a&gt; (DTW). There are lower bounds for DTW that are very efficient computationally. The &lt;a href=&quot;http://www.cs.ucr.edu/~eamonn/LB_Keogh.htm&quot; rel=&quot;noreferrer&quot;&gt;research of Professor Keogh&lt;/a&gt; might be interesting if you need theoretical foundation for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, &lt;em&gt;normally&lt;/em&gt; euclidean distance and Manhattan distance are not very appropriate for time series due to their sensitivity to signal transformations (e.g. shifts), but actually they are often used in practice.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-03-01T16:53:47.163" LastActivityDate="2018-03-01T16:53:47.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28470" PostTypeId="2" ParentId="28468" CreationDate="2018-03-01T16:19:50.633" Score="1" Body="&lt;p&gt;Assuming you have the elements 9779 = True Positive, 148 = True Negative, you can obtain the accuracy by adding the diagonal elements, then dividing by the sum of all the numbers within the matrix. So for your example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(9779 + 148)/(9779 + 148 + 107 + 2227) = 0.80964..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, Accuracy = 80.96%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More info: &lt;a href=&quot;http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46374" LastEditorUserId="46374" LastEditDate="2018-03-02T14:53:36.957" LastActivityDate="2018-03-02T14:53:36.957" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28471" PostTypeId="2" ParentId="28468" CreationDate="2018-03-01T16:22:54.673" Score="2" Body="&lt;p&gt;A confusion matrix gives you the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[TP, FP]&#xA;[FN, TN]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where TP = 'true positives'; FP = 'false positives'; FN = 'false negatives'; TN = 'true negatives'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read more here: &lt;a href=&quot;http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By taking TP+TN and dividing by TP+FP+FN+TN, you can get the classification accuracy of your model. In your case, that means (9779+148)/(9779+107+2227+148) = about 81%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More details:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This type of confusion matrix is used for binary classification. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A 'true positive' in this case means a true instance of class 0; that is, the model predicts that a given example belongs to class 0 and it really does belong to class 0. &lt;/li&gt;&#xA;&lt;li&gt;A 'true negative' means a true instance of class 1.&lt;/li&gt;&#xA;&lt;li&gt;A 'false positive' means the model predicts that the example belongs to class 0 when it really belongs to class 1;&lt;/li&gt;&#xA;&lt;li&gt;A 'false negative' means the model predicts that the example belongs to class 1 when it really belongs to class 0.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-03-01T16:22:54.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28472" PostTypeId="1" AcceptedAnswerId="28515" CreationDate="2018-03-01T16:23:04.153" Score="4" ViewCount="3618" Body="&lt;p&gt;I have two groups of images, each one with 1000 samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The speckle pattern, in this context, is the same as a random pattern or &quot;white noise&quot; image. So these images are fundamentally different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In group one, each figure is generated by considering a random function that returns something similar to a speckle pattern (see fig. 1).&#xA;In group two we follow the same procedure as group 1, but we plot a small point above that can be positioned anywhere and with any color (see fig. 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to classify both groups and I already tried to do it with simple neural networks, but I have been unsuccessful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best technique for this kind of problems?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fig. 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ju1Ru.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ju1Ru.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fig. 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UZAOs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UZAOs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47074" LastEditorUserId="35644" LastEditDate="2018-03-01T18:41:30.757" LastActivityDate="2018-03-02T10:35:43.910" Title="Classification of very similar images" Tags="|python|neural-network|classification|computer-vision|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28473" PostTypeId="2" ParentId="28472" CreationDate="2018-03-01T16:44:14.460" Score="2" Body="&lt;p&gt;You're going to have to do some experimenting to figure out what is 'best', but I would recommend starting with a convolutional neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you're only detecting a very small difference, though, the pixel values themselves should give you a good indication of where the colour is, and whether there is colour at all. I'm a bit surprised that a simpler architecture hasn't given you any luck. &lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-03-01T16:44:14.460" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28475" PostTypeId="2" ParentId="28472" CreationDate="2018-03-01T17:10:42.760" Score="2" Body="&lt;p&gt;Based on the given Example&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If they are literally the same everywhere except in a small region, just subtract image 1 from image 2 to find the differences...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And then we can check the positive and negative values where they are differing by a considerable margin to &lt;strong&gt;classify them&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This Will (Probably) Work...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-01T17:33:01.403" LastActivityDate="2018-03-01T17:33:01.403" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28476" PostTypeId="1" AcceptedAnswerId="28479" CreationDate="2018-03-01T17:21:15.977" Score="4" ViewCount="5286" Body="&lt;p&gt;I have a lot of images and I would like to be able to classify them into two groups: one containing images with watermarks and one containing images without any watermark.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are about 40 different watermarks. I created &quot;fake&quot; watermarked images to train a CNN and it worked very well on the &quot;fake&quot; validation set but not on the real images. Plus it was a long shot because I would have needed to train a model for each watermark (and I don't have the original watermark) or train a big model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I quit the watermark approach to try and find text. So I tried OpenCV text detection and it really wasnt working since the text is crooked and not that different from the background. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an easy solution I missed? Any idea is welcolme. I am kinda new to machine learning :)&lt;/p&gt;&#xA;" OwnerUserId="47078" LastEditorUserId="29169" LastEditDate="2021-02-13T16:37:33.567" LastActivityDate="2021-02-13T16:37:33.567" Title="Watermark detection in Python" Tags="|deep-learning|keras|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28477" PostTypeId="2" ParentId="28472" CreationDate="2018-03-01T17:31:14.450" Score="5" Body="&lt;p&gt;Have you tried rule-based approaches?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on your example, I can think of two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For each picture, get a list of all RGB values it contains.&#xA;The first contains only grays (which have RGB values &lt;code&gt;(x, x, x)&lt;/code&gt;), the second one contains color (so some pixel with RGB value &lt;code&gt;(x, y, z)&lt;/code&gt;, where $x \neq y$ or $y \neq z$). Or - if that is faster - convert all pictures to gray scale and check if the converted picture is identical to the original.&lt;/li&gt;&#xA;&lt;li&gt;Scan the image for areas (larger than one pixel) with identical colors. If your dots are large enough and the background noise is uncorrelated for neighboring pixels, it is very likely that these areas are the dots you are looking for.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-03-01T17:31:14.450" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28478" PostTypeId="2" ParentId="28465" CreationDate="2018-03-01T17:36:35.513" Score="2" Body="&lt;p&gt;This is a bit unusual because you are throwing away information on the sign: The vectors &lt;code&gt;(-2.998, 8.0, 2.5)&lt;/code&gt; and &lt;code&gt;(2.9998, 8.0, 2.5)&lt;/code&gt; yield the same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I do not think there is a standard term for this procedure.&#xA;A clear comment in your code could be 'Take the absolut value and normalize by the sum of all values' or, if you are feeling mathematical today, 'Take the absolute value and normalize by the &lt;a href=&quot;https://en.wikipedia.org/wiki/Lp_space#The_p-norm_in_finite_dimensions&quot; rel=&quot;nofollow noreferrer&quot;&gt;$L^1$-norm&lt;/a&gt;'.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-03-01T17:36:35.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28479" PostTypeId="2" ParentId="28476" CreationDate="2018-03-01T17:43:31.290" Score="5" Body="&lt;p&gt;Interesting question! Maybe the &lt;a href=&quot;https://keras.io/applications/&quot; rel=&quot;noreferrer&quot;&gt;pretrained models&lt;/a&gt; in Keras can help. Either by means of transfer learning, so that you might have to label only a small number of images by hand to retrain the higher layers. Or by using them for feature extraction and see if a certain keyword appears frequently for watermarked images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or just upload the pictures some place that does not allow watermarks and see if they get flagged ;-)&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-03-01T18:20:43.867" LastActivityDate="2018-03-01T18:20:43.867" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28480" PostTypeId="1" CreationDate="2018-03-01T18:11:33.763" Score="2" ViewCount="378" Body="&lt;p&gt;The target is a probability between N classes, I don't want it to predict the class with the highest probability but the 'actual' probability per class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|    | Class 1 | Class 2 | Class 3 |&#xA;------------------------------------&#xA;|  1 |     0.9 |    0.05 |    0.05 |&#xA;|  2 |     0.2 |     0.8 |       0 |&#xA;|  3 |     0.3 |     0.3 |     0.4 |&#xA;|  4 |     0.7 |       0 |     0.3 |&#xA;------------------------------------&#xA;|  + |     2.1 |    1.15 |    0.75 | &amp;lt;- correct this imbalance?&#xA;| &amp;gt;0 |       4 |       3 |       3 | &amp;lt;- or this one?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some classes have 'more' samples in the sense that the sum of probabilities is higher than other classes. Do I have to balance this out with weights in the loss function? Or do I only correct for the imbalance in &gt;0 as normally?&lt;/p&gt;&#xA;" OwnerUserId="47011" LastEditorUserId="47011" LastEditDate="2018-03-02T09:40:23.587" LastActivityDate="2024-02-14T22:05:45.650" Title="How to weigh imbalanced softlabels?" Tags="|class-imbalance|labels|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28481" PostTypeId="1" AcceptedAnswerId="28579" CreationDate="2018-03-01T18:20:58.723" Score="0" ViewCount="168" Body="&lt;p&gt;I'm training a cNN for binary classification. I used a batch size of 128, and the loss is decreasing and accuracy is increasing over epoches. The accuracy reached over 0.99 eventually, and the loss reached below 0.3. But after a few more epoches, the model converges to loss 0.6 and accuracy 0.5. An inspection of the model shows that it always predicts 0.5. I'm using binary cross entropy as the loss function. For each epoch, all data points are shuffled. I'm using SGD, learning rate is 0.01.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I hitting a local minimum with low accuracy rate but fairly good loss value? What is the suggested approach to deal with this? Also, why is it possible to have a low loss function with one single predicted class?&lt;/p&gt;&#xA;" OwnerUserId="47082" LastEditorUserId="47082" LastEditDate="2018-03-02T22:28:49.827" LastActivityDate="2018-03-03T20:57:01.110" Title="Reason for having both low loss and same predicted class?" Tags="|deep-learning|loss-function|accuracy|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 3.0" />
  <row Id="28482" PostTypeId="1" AcceptedAnswerId="28483" CreationDate="2018-03-01T18:23:23.323" Score="0" ViewCount="89" Body="&lt;p&gt;When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume the original state is $(0.1, 0.1, 0.2, 5.0)$ &#xA;and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the original length is $$&#xA;\begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1  + 0.2\times 0.2 + 5\times 5)} &amp;amp;= \sqrt{25.06}&#xA;\approx 5.006.\end{align*}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Applying the compensation gives $5.004 / 0.5 = 10.008.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like &lt;a href=&quot;https://stats.stackexchange.com/questions/201569/difference-between-dropout-and-dropconnect&quot;&gt;DropConnect&lt;/a&gt; does).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my actual test, the state-vector of $192$ elements has length of  $0.885$ and the masked vector, with compensation has length of $1.305$&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-05-16T10:57:34.967" LastActivityDate="2019-05-16T10:57:34.967" Title="Dropout dividing by compensation term = overshoots the result?" Tags="|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28483" PostTypeId="2" ParentId="28482" CreationDate="2018-03-01T18:46:09.027" Score="2" Body="&lt;p&gt;Your example is cherry-picked: You mask out small numbers and keep a large one.&#xA;But dropout is applied randomly. Each of the following six masks, and of the corresponding values for the vector length, is equally likely to appear:&#xA;$$&#xA;\begin{align*}&#xA;&amp;amp;(1, 1, 0, 0): &amp;amp;\sqrt{0.1^2 + 0.1^2} &amp;amp;\approx 0.1414,\\&#xA;&amp;amp;(1, 0, 1, 0): &amp;amp;\sqrt{0.1^2+ 0.2^2} &amp;amp;\approx 0.2236,\\&#xA;&amp;amp;(1, 0, 0, 1): &amp;amp;\sqrt{0.1^2+ 5^2} &amp;amp;\approx 5.0010,\\&#xA;&amp;amp;(0, 1, 1, 0): &amp;amp;\sqrt{0.1^2 + 0.2^2} &amp;amp;\approx 0.2236,\\&#xA;&amp;amp;(0, 1, 0, 1): &amp;amp;\sqrt{0.1^2 + 5^2} &amp;amp;\approx 5.0010,\\&#xA;&amp;amp;(0, 0, 1, 1): &amp;amp;\sqrt{0.2^2 + 5^2} &amp;amp;\approx 5.0040.\\&#xA;\end{align*}&#xA;$$&#xA;The average vector length is&#xA;$$&#xA;\frac16 (0.1414+0.2236+5.0010+0.2236+5.0010+5.0040) = 2.5991,&#xA;$$&#xA;which is roughly half of the original vector length $5.006$. So it makes sense to divide it by the dropout rate of $50\%$.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-03-01T18:46:09.027" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28484" PostTypeId="2" ParentId="28480" CreationDate="2018-03-01T19:16:19.420" Score="0" Body="&lt;p&gt;If you have imbalanced classes (for example, if you have 3 classes and 100 examples of class 1 and 1000 examples of class 2 and 5000 examples of class 3), then yes, I would weight the loss function (I would use weighted categorical cross-entropy). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you mean some classes have a higher probability than others, then this is normal and expected behaviour. For example, if you were doing a 10-class classification problem like on MNIST, and you're trying to predict a given image, if the image has some rounded sections then it's much more likely to be a 3 or an 8 than a 1. &lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-03-01T19:16:19.420" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28485" PostTypeId="2" ParentId="27245" CreationDate="2018-03-01T19:51:30.373" Score="1" Body="&lt;p&gt;The innovation number is generated by a single, unique counter for all genomes. The purpose of the innovation number is to have an ID that indicates corresponding genes in two different genomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, a gene may originate in a genome due to mutation in generation 1. In future generations this gene may have been passed on to other genomes through crossover. So now we have multiple genomes with our gene in it. If two genomes with this gene are selected for crossover with each other, we want to know that this gene is identical in both genomes. We can tell that these genes are identical, because they have matching innovation numbers. If each genome had their own counter, then each genome would have its own unrelated gene for each innovation number. This tells us nothing about the nature of our network and negates the reason we added innovation number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside, here is a more thorough outline of &lt;a href=&quot;http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;NEAT&lt;/a&gt; straight from the author of the paper&lt;/p&gt;&#xA;" OwnerUserId="47085" LastActivityDate="2018-03-01T19:51:30.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28487" PostTypeId="1" CreationDate="2018-03-01T20:28:23.800" Score="1" ViewCount="139" Body="&lt;p&gt;The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've built a system which converges nicely with momentum at 0 and learning rate 0.01, even with 100 layers stacked. With dropout, the error decreases, but jumps up and down due to dropout knocking-out certain neurons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it usual to see fluctuations of error during backprop with dropout, similar to SGD?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would that  imply I can train on huge batches instead of minibatches, without the fear of overfitting by default?&#xA;For example, using Resilient Propagation (rProp) - in my test-example the LSTM trains in just 40 iterations using rProp, instead of 1000 using SGD&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-10-26T06:04:27.113" LastActivityDate="2018-10-26T06:04:27.113" Title="Dropout implies stochastic descent?" Tags="|dropout|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28488" PostTypeId="2" ParentId="27820" CreationDate="2018-03-01T20:52:07.517" Score="1" Body="&lt;p&gt;If you're looking for an introduction to mathematics for data science, take a look at the Coursera course &lt;a href=&quot;https://www.coursera.org/learn/datasciencemathskills&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science Math Skills&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="46768" LastActivityDate="2018-03-01T20:52:07.517" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28491" PostTypeId="1" AcceptedAnswerId="28503" CreationDate="2018-03-01T22:04:04.707" Score="1" ViewCount="2426" Body="&lt;p&gt;A lot of people rotate images to create a larger training set for neural networks. For most nets, all of the inputs have to be the same size so the image rotation function has to crop the newly rotated images to match the input size. So, say you have &lt;span class=&quot;math-container&quot;&gt;$32\times32$&lt;/span&gt; resolution images and do 45 degree rotations. Some of the output images will look diamond shaped with black (zero values) in the corners. So, my question is: should you leave these zero values alone or change them in some way and if so, how?&lt;/p&gt;&#xA;" OwnerUserId="28567" LastEditorUserId="67546" LastEditDate="2020-01-27T20:06:31.187" LastActivityDate="2020-01-27T20:06:31.187" Title="Data augmentation: rotating images and zero values" Tags="|neural-network|data-augmentation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28492" PostTypeId="2" ParentId="28487" CreationDate="2018-03-01T22:05:38.967" Score="2" Body="&lt;p&gt;It depends a bit on your definition, usually the stochastic part of Stochastic Gradient Descent refers to the fact that you sample mini batches and estimate the true gradient with this sample. Dropout adds stochastic behaviour by sampling masks that comes down to sampling new network architectures every time. You could certainly see this as a form of stochastic gradient descent but whether this is the case or not depends solely on the definition.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-03-01T22:05:38.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28493" PostTypeId="1" AcceptedAnswerId="28500" CreationDate="2018-03-01T23:36:54.603" Score="9" ViewCount="57947" Body="&lt;p&gt;After run my python code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(confusion_matrix(x_test, x_pred))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get this:&#xA;[100 32&#xA;211  21]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I get the following list:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;True positive = 100 &lt;/li&gt;&#xA;&lt;li&gt;False positive = 32 &lt;/li&gt;&#xA;&lt;li&gt;False negative = 211 &lt;/li&gt;&#xA;&lt;li&gt;True negative = 21&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is this possible?&lt;/p&gt;&#xA;" OwnerUserId="24790" LastActivityDate="2023-03-31T06:27:25.163" Title="Confusion Matrix - Get Items FP/FN/TP/TN - Python" Tags="|python|confusion-matrix|" AnswerCount="9" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28495" PostTypeId="2" ParentId="22335" CreationDate="2018-03-02T00:08:10.463" Score="2" Body="&lt;p&gt;Machine Learning can be rightly considered Black boxes, solutions for the XOR problem using neural networks can be modelled but as the number of inputs grow, so does the complexity and dimensions.  If it is too complex to understand and explain, then it is a black box, whether or not we can calculate the results or not&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can only perceive them upto 3 dimensions but this is sufficient because we can extrapolate this upto higher dimensions using the 3d model as a point of reference.  We can imagine local minimums, as well as parts of datasets that are partially learnt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have toyed with the idea for a while and so I produced animations of neural networks at work and improved my understanding of neural networks.  I have produced animations with 1 and 2 hidden layers (3rd is mostly done) and how they learn data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The animation is slow and the top right animation showing the upper layers is worth watching, you can speed the animations on Youtube if you like, significant changes can be seen on the top right animation with the Blue and Red Mesh at 3:20 Orange and Red mesh at 6 mins and the Blue, Orange and Red mesh at 8:20.  The directions of the weight changes are obviously in the bottom left animation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=UhQJbFDtcoc&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=UhQJbFDtcoc&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47101" LastEditorUserId="47101" LastEditDate="2018-03-02T00:41:13.273" LastActivityDate="2018-03-02T00:41:13.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28497" PostTypeId="1" CreationDate="2018-03-02T00:32:56.100" Score="3" ViewCount="1198" Body="&lt;p&gt;Imagine you have 2 people at 2 different microphones but in the same room. Each microphone is going to pick up some sound from the other person. Is there a good neural network based approach to isolating the signals so that the sound from each microphone only captures 1 person?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember hearing a solution to this a few years back, but Im not sure if I remember that correctly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I ask because a similar problem was mentioned to me today. During EEG brain wave data collection, each electrode can pick up signal from multiple sources in the brain. In that world they try to isolate the sources and reduce the &quot;noise&quot; from other brain areas, and its common to use ICA for such a task. The problem with ICA is that the post-processing stage is very time consuming, so I'm wondering if theres a better ANN/DNN approach that could solve the problem more efficiently, or maybe with better accuracy&lt;/p&gt;&#xA;" OwnerUserId="12206" LastActivityDate="2024-02-17T15:00:30.123" Title="Neural network approach to the cocktail party effect" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28499" PostTypeId="1" AcceptedAnswerId="28595" CreationDate="2018-03-02T01:55:22.400" Score="1" ViewCount="9846" Body="&lt;p&gt;&lt;code&gt;flow_from_directory&lt;/code&gt; in Keras requires images to be in different subdirectories. However, I have the images in a single directory with a csv file specifying the image name and target classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I use flow from directory directly from csv files named train.csv and test.csv?&lt;/p&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="29575" LastEditDate="2018-03-02T02:08:09.507" LastActivityDate="2020-05-24T17:13:14.087" Title="How to use flow_from_directory in keras for csv" Tags="|python|keras|image-classification|cnn|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28500" PostTypeId="2" ParentId="28493" CreationDate="2018-03-02T02:13:31.093" Score="11" Body="&lt;p&gt;Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def perf_measure(y_actual, y_pred):&#xA;    TP = 0&#xA;    FP = 0&#xA;    TN = 0&#xA;    FN = 0&#xA;&#xA;    for i in range(len(y_pred)): &#xA;        if y_actual[i]==y_pred[i]==1:&#xA;           TP += 1&#xA;        if y_pred[i]==1 and y_actual[i]!=y_pred[i]:&#xA;           FP += 1&#xA;        if y_actual[i]==y_pred[i]==0:&#xA;           TN += 1&#xA;        if y_pred[i]==0 and y_actual[i]!=y_pred[i]:&#xA;           FN += 1&#xA;&#xA;return(TP, FP, TN, FN)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if confusion matrix is a 2x2 matrix (named cm), you can use &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TP = cm[0][0]&#xA;FP = cm[0][1]&#xA;FN = cm[1][0]&#xA;TN = cm[1][1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-03-02T02:13:31.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28501" PostTypeId="2" ParentId="28493" CreationDate="2018-03-02T02:14:13.330" Score="2" Body="&lt;p&gt;In your case you can use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conf = confusion_matrix(x_test, x_pred)&#xA;TP = conf[0,0]&#xA;FP = conf[0,1]&#xA;TN = conf[1,0]&#xA;FN = conf[1,1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-02T02:14:13.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28502" PostTypeId="2" ParentId="28493" CreationDate="2018-03-02T02:16:54.437" Score="4" Body="&lt;p&gt;Create a method that does the printing for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def print_confusion_matrix(y_true, y_pred):&#xA;    cm = confusion_matrix(y_true, y_pred)&#xA;    print('True positive = ', cm[0][0])&#xA;    print('False positive = ', cm[0][1])&#xA;    print('False negative = ', cm[1][0])&#xA;    print('True negative = ', cm[1][1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And use it like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print_confusion_matrix(x_test, x_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you want the values return and not only printed you can do it like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_confusion_matrix_values(y_true, y_pred):&#xA;    cm = confusion_matrix(y_true, y_pred)&#xA;    return(cm[0][0], cm[0][1], cm[1][0], cm[1][1])&#xA;&#xA;TP, FP, FN, TN = get_confusion_matrix_values(x_test, x_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32492" LastActivityDate="2018-03-02T02:16:54.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28503" PostTypeId="2" ParentId="28491" CreationDate="2018-03-02T02:31:16.587" Score="1" Body="&lt;p&gt;Keeping the values as zeros will introduce some bias to your network. Given, you have this corner effect for the majority of your dataset, you do not want the network to identify a high probability of making the corners black. Thus, you should fill them, you can extend the edge, do a reflection, wrapping. You can also do some more complex function, like take average of a few patches in your image then place them in the missing areas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keras has a very nice function that can do all this for you.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from keras.datasets import mnist&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;&#xA;# load data&#xA;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;X_train = X_train.astype('float32')&#xA;&#xA;# set up your data generator&#xA;datagen = ImageDataGenerator(&#xA;    featurewise_center=True,&#xA;    featurewise_std_normalization=True,&#xA;    rotation_range=15,&#xA;    width_shift_range=0.2,&#xA;    height_shift_range=0.2,&#xA;    shear_range=0.2,&#xA;    zoom_range=0.2,&#xA;    horizontal_flip=True,&#xA;    vertical_flip = False)&#xA;&#xA;# Fit the generator using your data&#xA;datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))&#xA;&#xA;# Black images&#xA;image = X_train[5]&#xA;plt.imshow(image,  cmap='gray')&#xA;plt.show()&#xA;&#xA;plt.figure(figsize=(12,12))&#xA;plt.subplot(4, 4, 1)&#xA;plt.imshow(image.reshape((28,28)),  cmap='gray')&#xA;&#xA;for j in range(15):&#xA;    augmented = datagen.random_transform(image.reshape((28,28,1)))&#xA;    plt.subplot(4, 4, j+2)&#xA;    plt.imshow(augmented.reshape((28,28)),  cmap='gray')&#xA;plt.tight_layout()&#xA;plt.show()&#xA;&#xA;# White images&#xA;image = -1*X_train[5]&#xA;plt.imshow(image,  cmap='gray')&#xA;plt.show()&#xA;&#xA;plt.figure(figsize=(12,12))&#xA;plt.subplot(4, 4, 1)&#xA;plt.imshow(image.reshape((28,28)),  cmap='gray')&#xA;&#xA;for j in range(15):&#xA;    augmented = datagen.random_transform(image.reshape((28,28,1)))&#xA;    plt.subplot(4, 4, j+2)&#xA;    plt.imshow(augmented.reshape((28,28)),  cmap='gray')&#xA;plt.tight_layout()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-02T02:31:16.587" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28504" PostTypeId="1" CreationDate="2018-03-02T03:05:44.847" Score="1" ViewCount="225" Body="&lt;p&gt;I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Clustering papers and extracting features from them in order to find other authors with similar research interests in PUBMED&lt;/li&gt;&#xA;&lt;li&gt;Develop a classification algorithm to determine if a given author&#xA;has the similar research interest to those in my original set. &#xA;Really, I am trying find new authors and be able to tell if an&#xA;author that I discovered, is similar to my original set of authors.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Unfortunately because my data is only one class, I have an extreme data imbalance problem.  How should I fix this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus as far as I have looked at the frequency of terms in the papers with mixed results.  I have also been using unsupervised learning to extract features with mixed results.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My set of papers and authors are assumed to share similar research interests. Do I need papers and authors that I know aren’t in the set in order to build a classification algorithm?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I look at a paper as a whole or should I look at the papers written by each author for classification, instead of looking at the papers as a group?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suppose I could randomly pick other papers and use those as the examples of other fields. Of course, picking paper at random there is the chance of false negatives.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using R with the tm package to do the mining.  &lt;/p&gt;&#xA;" OwnerUserId="47107" LastEditorUserId="8878" LastEditDate="2018-03-02T12:54:27.223" LastActivityDate="2018-03-02T12:54:27.223" Title="Text classification and clustering with complete date imbalance" Tags="|classification|clustering|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28506" PostTypeId="1" AcceptedAnswerId="28507" CreationDate="2018-03-02T06:34:03.923" Score="0" ViewCount="38" Body="&lt;p&gt;i have labeled data with goods like:&#xA;&quot;Double chamber refrigerator Hitachi R-WB 482 PU2 GBW&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to predict category like: laptops, household appliances etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can i do this?&lt;/p&gt;&#xA;" OwnerUserId="47110" LastActivityDate="2018-03-02T07:06:12.167" Title="Classification of goods by category" Tags="|classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28507" PostTypeId="2" ParentId="28506" CreationDate="2018-03-02T06:55:00.483" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Random forests should be able to handle categorical values natively so look for a different implementation so you don't have to encode all of those features and use up all your memory..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/parameter.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/q/4942/35644&quot;&gt;Dimensionality Reduction&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/08/catboost-automated-categorical-data/&quot; rel=&quot;nofollow noreferrer&quot;&gt;CatBoost&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hierarchical Clustering Using Scipy&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;Target Encoding&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-02T07:06:12.167" LastActivityDate="2018-03-02T07:06:12.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28509" PostTypeId="1" CreationDate="2018-03-02T07:35:00.963" Score="4" ViewCount="192" Body="&lt;p&gt;I have been trying out an NLP problem where I have to predict multi-label-sentiments for some text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 8 labels and 170k training examples, and 140k for the test set.&#xA;My final dictionary size is around 190k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Keras for trying out an NN approach, although I am not sure if my architecture is right, Below is the model which gives me a 95 % on the test set, I am testing out accuracies on 0.7 - 0.3 split while training:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential&#xA;model.add(Embedding(max_indexes + 1, 100, weights=[embeddings], &#xA;                  input_length=100))&#xA;&#xA;model.add(Bidirectional(LSTM(256, return_sequences=True)))&#xA;&#xA;model.add(Convolution1D(256, 5, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;&#xA;model.add(Convolution1D(256, 5, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;&#xA;model.add(Convolution1D(256, 5, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;model.add(MaxPool1D())&#xA;&#xA;model.add(Convolution1D(256, 4, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;&#xA;model.add(Convolution1D(256, 3, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;&#xA;model.add(Convolution1D(256, 3, padding='same'))&#xA;model.add(Activation('relu'))&#xA;model.add(BatchNormalization())&#xA;model.add(MaxPool1D())&#xA;&#xA;model.add(Flatten())&#xA;&#xA;model.add(Dense(64, activation='relu',&#xA;                kernel_regularizer=keras.regularizers.l2(0.02)))&#xA;&#xA;model.add(Dense(target_classes_len, activation='sigmoid',&#xA;                kernel_regularizer=keras.regularizers.l2(0.02)))&#xA;model.add(Dropout(0.1))&#xA;&#xA;adam_opt = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)&#xA;&#xA;early_stopping = EarlyStopping(monitor='val_loss', patience=5, mode='min')&#xA;&#xA;save_best = ModelCheckpoint('model_x.hdf', save_best_only=True, &#xA;                               monitor='val_loss', mode='min')&#xA;reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,&#xA;                              patience=5, min_lr=0.0001)&#xA;&#xA;model.compile(adam_opt, 'binary_crossentropy', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Over the internet, and while watching a video of CSE231N and other StackOverflow questions, I found out I should follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[(CONV-RELU)*n - POOL? ]*m - (FC-RELU)*K, Softmax&#xA; N~5, M~very large, K&amp;gt;=0 and K&amp;lt;=2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Although it was for ImageNet classification problem and I have an NLP problem at hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I found out that should place DropOut after final FullyConnected and BatchNormalization in a sequence with CONV-RELU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I justed wanted to know what would a good organization of layers for an NLP problem, or any way of evaluating different architectures?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="20766" LastActivityDate="2018-11-17T04:04:31.007" Title="Organization of layers in Keras for a NLP problem" Tags="|deep-learning|nlp|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28510" PostTypeId="1" AcceptedAnswerId="28513" CreationDate="2018-03-02T09:48:40.430" Score="1" ViewCount="585" Body="&lt;p&gt;I would like to make a &lt;a href=&quot;https://python-graph-gallery.com/271-custom-your-bubble-plot/&quot; rel=&quot;nofollow noreferrer&quot;&gt;'bubble'&lt;/a&gt; plot, but with rectangular markers. Is it possible to do this in Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On x-axis should be the day of the week (Mon, Tue, Wed,...), on y-axis - counts, how many times people came to a restaurant, the area of rectangles - the number of people came to a restaurant. A height of rectangles is always the same, only a width changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, one rectangle on the plot shows how many people how many times came to a restaurant on a particular day of the week. As axis y may have values &gt; 30, bubbles are not suitable and I consider that rectangles works better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A sketch of what I would like to draw is below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/D1vYA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/D1vYA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to draw rectangular markers or is there another good way for visualizing the same concept?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&#xA;I've already tried box and violin plots. &#xA;The problem with boxplots is that in my case a height of a box is quite small and a tail is long. There are a lot of points indicating outliers. I would prefer to see some difference in outliers instead of a single point. With a bubble plot, it is possible to use some customized function for bubble size, e,g, logarithm. &lt;/p&gt;&#xA;" OwnerUserId="43473" LastEditorUserId="43473" LastEditDate="2018-03-02T11:03:00.097" LastActivityDate="2023-05-26T14:16:51.187" Title="rectangular markers in bubble plot (Python)" Tags="|python|visualization|plotting|matplotlib|seaborn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28512" PostTypeId="1" AcceptedAnswerId="28517" CreationDate="2018-03-02T10:14:00.557" Score="13" ViewCount="19862" Body="&lt;p&gt;Let's say I've trained my model and made my predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is... How can I append some new data to my pre-trained model without retrain the model from the beginning.&lt;/p&gt;&#xA;" OwnerUserId="44044" LastEditorUserId="40853" LastEditDate="2018-03-02T10:15:22.973" LastActivityDate="2020-01-31T14:55:53.100" Title="Train new data to pre-trained model" Tags="|machine-learning|python|deep-learning|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28513" PostTypeId="2" ParentId="28510" CreationDate="2018-03-02T10:27:08.443" Score="1" Body="&lt;p&gt;I can imagine a transposed boxplot() working well... in a figure with seven subplots...&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/olisteadman/ga_pre/blob/master/archive/bar_as_rectangular_bubble.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/olisteadman/ga_pre/blob/master/archive/bar_as_rectangular_bubble.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NWKjz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NWKjz.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46625" LastEditorUserId="55122" LastEditDate="2023-05-26T14:16:51.187" LastActivityDate="2023-05-26T14:16:51.187" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28515" PostTypeId="2" ParentId="28472" CreationDate="2018-03-02T10:35:43.910" Score="0" Body="&lt;p&gt;I found the answer in the paper linked above. &#xA;The authors use a CNN to solve the problem. I will post the code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://link.springer.com/article/10.1007/s00170-017-0882-0&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://link.springer.com/article/10.1007/s00170-017-0882-0&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47074" LastActivityDate="2018-03-02T10:35:43.910" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28516" PostTypeId="2" ParentId="28504" CreationDate="2018-03-02T11:51:38.243" Score="1" Body="&lt;p&gt;The question is not very clear but I will give a try anyways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First of all&lt;/strong&gt; please note that the major approach to tackle the analysis of &lt;a href=&quot;http://www.pnas.org/content/101/suppl_1/5200&quot; rel=&quot;nofollow noreferrer&quot;&gt;co-authorship networks&lt;/a&gt; is not what you mentioned, but &lt;a href=&quot;http://barabasi.com/networksciencebook/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Network Science&lt;/a&gt; techniques as &lt;a href=&quot;https://pdfs.semanticscholar.org/f53b/5d58a2eba3c6ee40bffe2239d7194cb07cc9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;co-authorship is a Social Network&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this necessary tip let's back to the question itself which is &lt;a href=&quot;https://watermark.silverchair.com/57.pdf? &quot; rel=&quot;nofollow noreferrer&quot;&gt;tackling the problem from NLP point of view&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Unfortunately because my data is all of one class already I have an extreme data imbalance problem. How should go about fixing this?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You don't have a &lt;a href=&quot;https://arxiv.org/ftp/arxiv/papers/1305/1305.1707.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;class imbalanced problem&lt;/a&gt;. Class imbalanced problem means that you have more than one class in which one or more classes are highly larger than others. But you have only one class! This is not a class imbalanced problem. You only need to define your question precisely.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;My set of papers and authors are assumed to share similar research interests. To build a classification algorithm do I need papers and authors that I know aren’t in the set? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Depends on what you want to do! Classification of what? You can set your question like &lt;strong&gt;classification of authors in the same field&lt;/strong&gt; then you do not need additional papers. You extract text features according to different papers of all individuals and train your classifier:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Concatenate all sentences of the papers of one individual and call it data. the name of individual is the label. Of course if a paper has two authors, the text will be assigned to both of them.&lt;/li&gt;&#xA;&lt;li&gt;Extract features from your text data. Unsupervised feature extraction does not care about the labels (&lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TF-IDF&lt;/a&gt; is an example) and &lt;a href=&quot;https://hal.inria.fr/inria-00321520/document&quot; rel=&quot;nofollow noreferrer&quot;&gt;supervised methods&lt;/a&gt; do care.&lt;/li&gt;&#xA;&lt;li&gt;Choose a classifier and start your learning phase as always.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;but if you set your question like &lt;strong&gt;classification of authors from different field&lt;/strong&gt; then sure you need data from other fields! In this case instead of each individual being a class, each field (including all authors in that field) becomes a single label.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Should I look at the paper as a whole or should I look at the papers written by each author for classification and instead of looking at the papers as a group?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's super difficult to understand what you mean but I guess I answered it above. Depends on your question! If you are going to classify authors inside a community, then all papers of that author is one text data object.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I suppose I could randomly pick other papers and use those at the examples to one that don’t share research interests. Of course, picking paper at random there is the chance of false negatives. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Again difficult to understand but anyways. In this case you better gather data from another field. Randomly mine some papers from other fields and construct a new class of &lt;strong&gt;NOT THIS FIELD&lt;/strong&gt;. Then go for a binary classification(&lt;a href=&quot;https://en.wikipedia.org/wiki/Logistic_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;Logistic Regression&lt;/a&gt; works well on TF-IDF features). It should be pointed that as your data is text you can not cover &lt;strong&gt;every text which is not in your main class&lt;/strong&gt; so limit your question to a specific border in domain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope I could help :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; The question needs a surgery. I will edit it soon and please check if my edit is conceptually correct.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-02T11:51:38.243" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28517" PostTypeId="2" ParentId="28512" CreationDate="2018-03-02T12:45:20.780" Score="1" Body="&lt;p&gt;I cannot comment yet. If you just load the model and use a fit method it will update the weights, not reinstance all the weights. It will just perform a number of weights update that you can chose, using the new data.&lt;/p&gt;&#xA;" OwnerUserId="47078" LastActivityDate="2018-03-02T12:45:20.780" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28519" PostTypeId="2" ParentId="28463" CreationDate="2018-03-02T14:14:13.867" Score="1" Body="&lt;p&gt;Looking again at your autocorrelated process:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    def my_process(n, p, drift=0, displacement=0):&#xA;        x = np.zeros(n)&#xA;&#xA;        for i in range(1, n):&#xA;            x[i] = drift * i + p * x[i-1] + (1-p) * np.random.randn()&#xA;    return x + displacement&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It looks like things are breaking down when the value of &lt;code&gt;displacement&lt;/code&gt; is high. This makes sense, as you say, because LSTMs need normalized data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;drift&lt;/code&gt; parameter is a bit different. When a small amount of drift is included, since &lt;code&gt;p&lt;/code&gt; is large, the amount of drift is similar to the amount of random noise being added via &lt;code&gt;np.random.randn()&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the plots for &lt;code&gt;drift=0.00001, displacement=10&lt;/code&gt;, it looks like the predictions would be fine except for the y-shift. Because of this, I think the root of the problem is still in the &lt;code&gt;displacement&lt;/code&gt; parameter, not the &lt;code&gt;drift&lt;/code&gt; parameter. Differencing, as has been done, will not help with the &lt;code&gt;displacement&lt;/code&gt; parameter; instead, it corrects for drift. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't tell from your code, but it looks like perhaps the &lt;code&gt;displacement&lt;/code&gt; was not accounted for in &lt;code&gt;model.predict&lt;/code&gt;. That's my best guess. &lt;/p&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-03-02T14:14:13.867" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28520" PostTypeId="2" ParentId="9848" CreationDate="2018-03-02T14:42:14.563" Score="1" Body="&lt;p&gt;I was looking for the same thing in Python and I came the &lt;a href=&quot;https://pypi.python.org/pypi/prince/0.2.6&quot; rel=&quot;nofollow noreferrer&quot;&gt;prince&lt;/a&gt; package that has FAMD implemented.&lt;/p&gt;&#xA;" OwnerUserId="40057" LastEditorUserId="29575" LastEditDate="2018-03-02T14:59:35.560" LastActivityDate="2018-03-02T14:59:35.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28522" PostTypeId="1" AcceptedAnswerId="28523" CreationDate="2018-03-02T16:07:45.333" Score="0" ViewCount="76" Body="&lt;p&gt;My knowledge is that gradient actually gets to the global minimum and gradient descent try to take steps to the direction that he judges to be the lowest.&#xA;I know that calculate the gradient from complex functions is non trivial problem, but let's say that we are a using neural net with a quadratic cost function, would be that hard to calculate the gradient and get actual global minimum?&lt;/p&gt;&#xA;" OwnerUserId="43791" LastActivityDate="2022-03-22T21:28:58.863" Title="Why use Gradient Descent when Gradient just solve the problem? (With Neural Nets)" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28523" PostTypeId="2" ParentId="28522" CreationDate="2018-03-02T16:37:59.640" Score="4" Body="&lt;p&gt;You might be mistaking the gradient itself with the mathematical approach to find the critical points of a differentiable function.&lt;/p&gt;&#xA;&lt;p&gt;In the &lt;a href=&quot;https://en.wikipedia.org/wiki/Derivative_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;latter&lt;/a&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Second_partial_derivative_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;approach&lt;/a&gt;, you take the derivative of the function with respect to its parameters and find the values of the parameters that make it zero. These points in parameter space are &lt;em&gt;critital points&lt;/em&gt;, that is, where the function has either a local maxima, a local minima or a saddle point. In order to know the actual type, you have to take the second derivative.&lt;/p&gt;&#xA;&lt;p&gt;The problem with applying such an approach to neural networks is not finding the gradient of the loss function with respect to the parameters; currently we have automatic differentiation software that computes the gradients for you. The actual problem is solving the equation where such gradient equals to zero. We don't know how to do that but in the trivial cases. Furthermore, this only gives you local optima, not global ones.&lt;/p&gt;&#xA;&lt;p&gt;A &lt;em&gt;solution&lt;/em&gt; to that problem is using numerical optimization techniques, like the gradient descent family. They basically explore the parameter space in the descending direction of the gradient of the loss function, &lt;em&gt;hoping&lt;/em&gt; to reach a minimum. This cannot be assured to converge for non-convex problems. Nevertheless, in practice it works quite well.&lt;/p&gt;&#xA;&lt;p&gt;The reasons why gradient descent techniques work well in non-convex problems are an active line of research.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2022-03-22T21:28:58.863" LastActivityDate="2022-03-22T21:28:58.863" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28526" PostTypeId="2" ParentId="28461" CreationDate="2018-03-02T16:54:31.147" Score="0" Body="&lt;p&gt;Assuming your features are df[X] and your target is df[y], I would just do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keepfeatures = X[trained_model.feature_importances_ &amp;gt; 0]&#xA;keepfeatures.append(y)&#xA;&#xA;df = df[keepfeatures]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="46490" LastActivityDate="2018-03-02T16:54:31.147" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28528" PostTypeId="1" CreationDate="2018-03-02T17:12:26.423" Score="1" ViewCount="70" Body="&lt;p&gt;It's quite a challenging task of &lt;em&gt;aspect extraction&lt;/em&gt; in the field of opinion mining if you look at the number of related papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what is &lt;em&gt;an aspect&lt;/em&gt; in the field of opinion mining?&lt;/p&gt;&#xA;" OwnerUserId="16162" LastActivityDate="2019-08-02T03:02:00.190" Title="What is an aspect in opinion mining?" Tags="|sentiment-analysis|terminology|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28529" PostTypeId="1" AcceptedAnswerId="28534" CreationDate="2018-03-02T17:12:28.610" Score="0" ViewCount="1638" Body="&lt;p&gt;I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;coef_gini = DecisionTreeClassifier(criterion = &quot;gini&quot;, random_state = 100, max_depth = 3, min_samples_leaf = 5)&#xA;coef_gini.fit(training_features, training_target)&#xA;&#xA;y_pred = coef_gini.predict(test_features)&#xA;y_pred&#xA;&#xA;for name, importance in zip(training_features.columns, coef_gini.feature_importances_):&#xA;    print(name, importance)&#xA;&#xA;print ( &quot;Train Accuracy using Decision Trees Classifier is : &quot;, accuracy_score(training_target, coef_gini.predict(training_features)))&#xA;print ( &quot;Test Accuracy using Decision Trees Classifier is : &quot;, accuracy_score(test_target, y_pred))&#xA;print ( &quot;Confusion matrix using Decision Trees Classifier is &quot;, confusion_matrix(test_target, y_pred))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the cost matrix? Is this the money that company will lost for each wrong predictive target value? Anyone have an example?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24719" LastActivityDate="2018-03-03T12:59:17.770" Title="Python - Calculate Cost profitability and benefit of the model" Tags="|python|predictive-modeling|model-evaluations|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28530" PostTypeId="1" CreationDate="2018-03-02T17:30:30.807" Score="1" ViewCount="551" Body="&lt;p&gt;I am new to deep learning. I have a collection of website texts and the activity class each company belongs to. A company can list multiple activity classes (NACE-codes)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to build a recommender that would recommend the five best activity classes based on a website the user provides. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is trained on the top 10K most frequent words that occur in the websites. The words are represented as scores ranging from 0 to 1. Each text is a line in the matrix, with 3000 columns (the first 3K words per website are taken into account).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The labels are k-hot encoded (activity class labels are columns and have 1 if the company lists the activity class; otherwise 0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a significant imbalance between the labels, so each label is weighted by dividing the total number of classes by the time each class occurs in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I train the model, the loss goes down and my custom 'precision' score (correct activity class is among the top-5 predictions) goes up somewhat during training. However, when I use the model to predict, it will always return the same scores between 0 and 1 (what you would expect, given the sigmoid layer) for the classes, no matter what the input is. &lt;strong&gt;So, it seems that all the model has learned is that some classes are more likely to occur than others.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions or guidelines would be appreciated. I have read Chollet's &quot;Deep learning with python&quot;, but it is not exactly a great source of advice on this particular topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; library(keras)&#xA; metric_precision &amp;lt;- function(y_true, y_pred) {&#xA;   metric_top_k_categorical_accuracy(y_true, y_pred, k = 5) &#xA; }&#xA;&#xA;    model &amp;lt;- keras_model_sequential()&#xA;    model %&amp;gt;%&#xA;    layer_embedding(input_dim=10000, output_dim=512, &#xA;     input_length=3000, mask_zero = FALSE, trainable=TRUE ) %&amp;gt;%&#xA;    layer_dropout(0.3) %&amp;gt;%&#xA;    layer_conv_1d(filters,kernel_size, activation = &quot;relu&quot;) %&amp;gt;%&#xA;    layer_max_pooling_1d(5) %&amp;gt;%&#xA;    layer_dropout(0.3) %&amp;gt;%&#xA;    layer_conv_1d(filters,kernel_size, activation = &quot;relu&quot;) %&amp;gt;%&#xA;    layer_global_max_pooling_1d() %&amp;gt;%&#xA;    layer_dense(ncol(train_y),&quot;sigmoid&quot;)&#xA;&#xA;  model %&amp;gt;% compile(&#xA;    loss = 'binary_crossentropy',&#xA;    optimizer = 'rmsprop',&#xA;    metrics = c(precision=metric_precision)&#xA;  )&#xA;&#xA;  model %&amp;gt;% fit(&#xA;    train_x, train_y,&#xA;    batch_size = batch_size,&#xA;    epochs = epochs,&#xA;    validation_data = list(test_x, test_y),&#xA;    class_weight = classweights&#xA;  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47141" LastEditorUserId="47141" LastEditDate="2018-03-03T11:31:24.917" LastActivityDate="2018-03-03T11:31:24.917" Title="Multilabel multiclass classifier returns same probabilities for any input" Tags="|r|keras|rstudio|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28531" PostTypeId="1" CreationDate="2018-03-02T17:39:37.003" Score="2" ViewCount="79" Body="&lt;p&gt;We are currently developing a Deep Convolutional Neural Network to extract road surfaces from aerial orthophoto's. Our current workflow involves using an existing road centreline to mask a large enough area to determine which of the pixels are road surface. The probability binary raster output are reasonably accurate. The final step within our workflow is that we use the original road centreline and average pixel width to determine the road width to buffer the road centreline as the output the road surface area. The problem with the following approach is that collector and arterial roads are not uniform in road width as well as the road bellmouth is lost at road intersections. Currently the team that has developed the following model is using Matlab, which I'm not familiar with, hence the following question to find out what morphological operations could be used to extract a better representation of the road surface by approximating road edges (straight lines), varying road widths and bellmouths at road intersections from the probability raster output from the CNN model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dmygG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dmygG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN: Current Workflow&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NebH7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NebH7.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNN: Red (Manually Derived Road Surface \ Training Dataset) Yellow (Vectorised Road Surface Output)&lt;/p&gt;&#xA;" OwnerUserId="46211" LastActivityDate="2018-07-30T20:00:49.340" Title="What morphological operations can be used to vectorise road surface binary rasters?" Tags="|cnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28533" PostTypeId="2" ParentId="28531" CreationDate="2018-03-02T18:21:14.317" Score="1" Body="&lt;p&gt;Perhaps another solution to your question: There are datasets that contain road surfaces, such as the (Dutch) BGT-database. &lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2018-03-02T18:21:14.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28534" PostTypeId="2" ParentId="28529" CreationDate="2018-03-02T18:28:56.117" Score="1" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;Confusion Matrix&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;A Confusion Matrix is an important tool to measure accuracy of a classification algorithm. It compares predicted class of an outcome and actual outcome.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Scenario 1: Credit Risk&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Based on a credit risk scorecard, application for credit card are classified as “Good” and “Bad”. “Good” indicates applicants paying back dues on credit card and “Bad” indicates customers defaulting on the dues. Now, the customers are compared against actual performance of the customer payment behaviour after say 18 months. So, comparison of Predicted Class (“Good” or “Bad”) to actual customer behaviour state (“Defaulted” or “Regular”).&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TmJye.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TmJye.jpg&quot; alt=&quot;Something Like This&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;There is always a trade-off between Type I error/False Positives (accepting Bad Customers) and Type II Error/False Negatives (Rejecting Good Customers).&lt;/p&gt;&#xA;&lt;p&gt;We generally require to optimize between False Positive Rate (Type I Error) and False Negative Rate (Type II Error).&lt;/p&gt;&#xA;&lt;p&gt;So, role of &lt;a href=&quot;https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.5.0/com.ibm.im.model.doc/c_cost_matrix.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cost matrix&lt;/a&gt; comes in picture to find the optimal cut off value for a classification rule.  Now, going back to Credit Risk Model. The cut off value optimize between cost of an opportunity loss (miss to accept a good customer/Type II Error) and cost of accepting a potential defaulter (involved in loss due to default).&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;Referencing Example is from a blog and image from Google..&lt;/code&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Cost Matrix&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Cost Matrix is similar of confusion matrix. It’s just, we are here more concerned about false positives and false negatives .There is no cost penalty associated with True Positive and True Negatives as they are correctly identified.&lt;/p&gt;&#xA;&lt;p&gt;The goal of this method is to choose a classifier with lowest total cost.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;Total Cost = C(FN)xFN + C(FP)xFP&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;where,&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;FN&lt;/code&gt; is the number of positive observations wrongly predicted&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;FP&lt;/code&gt; is the number of negative examples wrongly predicted&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;C(FN) and C(FP) &lt;/code&gt;corresponds to the costs associated with False Negative and False Positive respectively.&lt;/p&gt;&#xA;&lt;p&gt;Remember, &lt;code&gt;C(FN) &amp;gt; C(FP). &lt;/code&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Hope this Helps..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-03T12:59:17.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28537" PostTypeId="1" AcceptedAnswerId="28546" CreationDate="2018-03-02T19:58:14.137" Score="0" ViewCount="2569" Body="&lt;p&gt;I have found out the most profitable products in my dataframe by using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('ProductName')['ProfitPerOrder'].sum().sort_values().tail()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It gives me the output below. each ProductName has a ProductCategory. How do I display the category next to the product name in the output below?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ProductName&#xA;WFS Shoes           29033.887659&#xA;RDL Suit            45845.253318&#xA;Tennis Suit         46848.513342&#xA;Davenport Shoes    127103.915707&#xA;Halter Dress       314155.742025&#xA;Name: ProfitPerOrder, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-03-03T17:17:08.767" Title="Groupby product, return tuple" Tags="|pandas|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28538" PostTypeId="1" AcceptedAnswerId="28560" CreationDate="2018-03-02T20:33:17.210" Score="1" ViewCount="2660" Body="&lt;p&gt;I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model.  I found  most definition of these functions to be not 'clear'  so I modified the notation to make sure I understand it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is the following  formulation correct?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(w_{-t} | w_{t} ; \theta) = softmax(score(w_{-t}, w_t))$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(w_{-t} | w_{t} ; \theta) = \frac{exp(score(w_{-t}, w_t))}{\sum_{w' \in \theta} exp(score(w', w_{t}))}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{-t} =$ context&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{t} =$ target word&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$score(A,B)$ a measure of similarity between vectors A and B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta = $ vector representation for all words in vocabulary &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the simplest case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$score(A ,B) = A \cdot B$$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="47149" LastActivityDate="2018-03-03T11:50:25.200" Title="Word2Vec, softmax function" Tags="|machine-learning|nlp|word2vec|language-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28539" PostTypeId="1" CreationDate="2018-03-02T20:42:12.360" Score="10" ViewCount="14076" Body="&lt;p&gt;I'm using transfer learning to build an image recognition model using a pre-trained VGG network in Keras and excluding the final fully-connected layer to get the output weights.  I'm then using these output weights to feed into my new model which has a few layers along with a new fully-connected layer of my own that I'm training. The fully-connected layer maps to the number of output classes that I'm trying to predict.&lt;/p&gt;&#xA;&lt;p&gt;Everything is working fine.  However, when I run:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;results = model.predict(img_tensor)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I get output probabilities corresponding to each class, similar to the below:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;print(results)&#xA;&#xA;[[0.1426621  0.6193871  0.23795079]&#xA; [0.11187755 0.6208466  0.2672758 ]&#xA; [0.10050113 0.3768951  0.52260375]&#xA; [0.1338948  0.59470254 0.27140263]&#xA; [0.06612041 0.69726    0.2366195 ]&#xA; [0.12080433 0.495977   0.38321865]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;How can I find out what class each of the columns in the probabilities output correspond to?&lt;/p&gt;&#xA;&lt;p&gt;Does Keras have anything built-in to identify which column of the output probabilities corresponds to which class?&lt;/p&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="43000" LastEditDate="2021-11-26T20:00:16.550" LastActivityDate="2021-11-26T20:00:16.550" Title="How can I find out what class each of the columns in the probabilities output correspond to using Keras for a multi-class classification problem?" Tags="|neural-network|keras|tensorflow|image-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28540" PostTypeId="1" CreationDate="2018-03-02T20:57:53.900" Score="7" ViewCount="386" Body="&lt;p&gt;While learning batch normalization, I was thinking why can't we solve the &quot;gradient scale problem&quot; by using an appropriate activation function ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like can't we delay and scale the activation function instead of scaling the whole dataset and ensure that the variance is preserve through it ?&lt;/p&gt;&#xA;" OwnerUserId="9461" LastEditorUserId="9461" LastEditDate="2018-03-02T21:04:52.727" LastActivityDate="2018-03-02T21:13:41.240" Title="Is there any activation function that can make batch normalization obsolete?" Tags="|machine-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28541" PostTypeId="2" ParentId="28540" CreationDate="2018-03-02T21:08:14.470" Score="6" Body="&lt;p&gt;What you describe sounds a lot like Scaled-Exponential Linear Units (SELUs), that are the core of &lt;a href=&quot;https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf&quot; rel=&quot;noreferrer&quot;&gt;Self-Normalizing Neural Networks&lt;/a&gt;, which where presented at NIPS 2017.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A short summary from &lt;a href=&quot;http://shaofanlai.com/post/29&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt; is that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If the mean and variance of the input is in certain range, then the mean and variance of the output should (1) also in that range and (2) converge to a fixed point after iteratively applying the activation function.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You might want to have a look at the &lt;a href=&quot;https://redd.it/6g5tg1&quot; rel=&quot;noreferrer&quot;&gt;reddit post comments&lt;/a&gt;. If you want to fully understand them, you can go ahead with the 90 page-long appendix of the &lt;a href=&quot;https://arxiv.org/pdf/1706.02515.pdf&quot; rel=&quot;noreferrer&quot;&gt;arxiv preprint&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They got a lot of attention when they were presented, but I think they have not delivered up to the expectations, as no one seems to be talking about them lately &lt;em&gt;on the internet&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-03-02T21:13:41.240" LastActivityDate="2018-03-02T21:13:41.240" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28542" PostTypeId="1" AcceptedAnswerId="28651" CreationDate="2018-03-02T21:44:30.657" Score="5" ViewCount="3521" Body="&lt;p&gt;When using &quot;K-Fold Cross Validtion&quot; for Neural Net, do we:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Pick &lt;strong&gt;and save&lt;/strong&gt; initial weights of the network randomly (let's call it $W_0$)&lt;/li&gt;&#xA;&lt;li&gt;Split data into $N$ equal chunks&lt;/li&gt;&#xA;&lt;li&gt;Train model on $N-1$ chunks, validating against the left-out chunk (the $K$'th chunk)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Get validation error and revert the weights back to&lt;/strong&gt; $W_0$&lt;/li&gt;&#xA;&lt;li&gt;shift $K$ by 1 and repeat from 3.&lt;/li&gt;&#xA;&lt;li&gt;Average-out the validtion errors, to get a much better understanding of how network will generalize using this data.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Revert back to $W_0$ one last time, and train the network using the ENTIRE dataset&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realize 7 is possible, because we have a very good understanding of how network will generalize with the help of step 6. - Is this assumption correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is reverting back to the initial $W_0$ a necessity, else we would overfit? (revert like we do in step 4. and 7.) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3, most important:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume we've made it to step 7, and will train the model using ENTIRE data. By now, we don't intend to validate it after we will finish. In that case how do we know when to stop training the model during step 7? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, we can train with same number of epochs as we did during Cross validation. But then how can we be sure that Cross Validation was trained with an appropriate number of epochs in the first place? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please notice - during steps 3, 4, 5 we only had $K$'th chunk to evaluate Training vs Validation loss. $K$'th chunk is very small, so during the actual Cross-Validation it was unclear when to Early-Stop... To make things worse, it will be even more difficult in case of Leave-One-Out (also know as All-But-One), where K is simply made from a single training example&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-03-05T16:35:11.013" LastActivityDate="2020-12-07T11:56:41.307" Title="Cross Validation how to determine when to Early Stop?" Tags="|cross-validation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28544" PostTypeId="2" ParentId="28539" CreationDate="2018-03-02T22:26:28.073" Score="5" Body="&lt;p&gt;The &lt;code&gt;class_indices&lt;/code&gt; attribute in Keras’ &lt;code&gt;flow_from_directory(directory)&lt;/code&gt; creates a dictionary of the classes and their index in the output array:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;classes: optional list of class subdirectories (e.g. &lt;code&gt;[‘dogs’, ‘cats’]&lt;/code&gt;). Default: None. If not provided, the list of classes will be&#xA;automatically inferred from the subdirectory names/structure under&#xA;directory, where each subdirectory will be treated as a different&#xA;class (and the order of the classes, which will map to the label&#xA;indices, will be alphanumeric). The dictionary containing the mapping&#xA;from class names to class indices can be obtained via the attribute&#xA;&lt;code&gt;class_indices&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Source&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I would be interested to see how people have incorporated this into their workflow/scripts though...&lt;/p&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="98307" LastEditDate="2020-08-02T11:20:04.927" LastActivityDate="2020-08-02T11:20:04.927" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28546" PostTypeId="2" ParentId="28537" CreationDate="2018-03-03T02:36:47.223" Score="1" Body="&lt;p&gt;Add more columns when you are doing group by in the first parentheses..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we should understand why it's giving this result..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's similar to Sql,we are applying an aggregate function on a grouped by value,&#xA;That's why it's giving only one value,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to have further information,&#xA;Add the columns name in the first parentheses in a $list$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will be something like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby([Col1,Col2...,Coln])(another col).sum().sort_values().tail()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit -1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recently enough, I came across that we can cascade group by together by playing smart and using the &lt;code&gt;.join(second group by(). aggregate ())&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here's the working example (ignore the column names...it's relevant to my dataset)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.groupby('Year_of_Release')[['Global_Sales']].sum().join( df.groupby('Year_of_Release')[['Name']].count())&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-03T17:17:08.767" LastActivityDate="2018-03-03T17:17:08.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28547" PostTypeId="2" ParentId="28539" CreationDate="2018-03-03T03:01:10.040" Score="1" Body="&lt;p&gt;You can then select the most probable classes using the &lt;code&gt;probas_to_classes()&lt;/code&gt; utility function. Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_proba = model.predict(x)&#xA;y_classes = keras.np_utils.probas_to_classes(y_proba)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is equivalent to &lt;code&gt;model.predict_classes(x)&lt;/code&gt; on the Sequential model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When Using Generators,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;generator= train_datagen.flow_from_directory(&quot;train&quot;, batch_size=batch_size)&#xA; label_map = (generator.class_indices)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The label_map variable is a dictionary,&#xA;Then from this the relation can be derived between the probability scores and class names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, you can create this dictionary by this code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from glob import glob&#xA;class_names = glob(&quot;*&quot;) # Reads all the folders in which images are present&#xA;class_names = sorted(class_names) # Sorting them&#xA;name_id_map = dict(zip(class_names, range(len(class_names))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-03T03:07:33.643" LastActivityDate="2018-03-03T03:07:33.643" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28548" PostTypeId="1" CreationDate="2018-03-03T03:25:42.850" Score="0" ViewCount="1395" Body="&lt;p&gt;I'm using XGBoost for a multiclass classification problem in R. I'm trying different combinations of the shrinkage parameter and number of iterations to try and settle on optimal values for these parameters (using cross-validation). However, I'm having trouble with exactly interpreting the significance of the parameters. The XGBoost algorithm isn't completely a black box for me - I know how boosting works at a high-level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The questions I have though are as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Does a low shrinkage (e.g. 0.01) and high nrounds (e.g. 500) yield the same results as a high shrinkage (e.g. 0.3) and low nrounds (e.g. 50)? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does nrounds correspond to the number of weak classifiers being fit?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The xgb.cv function in R returns test mean errors and standard deviations for each of the iterations. For the nth iteration, is this the mean over all iterations less than equal to n over all the folds? Apologies if I'm interpreting this incorrectly. Also, what is the significance of the standard deviation here?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there some blog post/link I can refer to for a comprehensive guide on tuning XGBoost parameters? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47158" LastEditorUserId="53001" LastEditDate="2018-06-04T02:01:26.477" LastActivityDate="2018-06-04T02:01:26.477" Title="Interpretation of tuning parameters (shrinkage and nrounds) in XGBoost" Tags="|machine-learning|classification|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28549" PostTypeId="1" CreationDate="2018-03-03T03:36:45.990" Score="1" ViewCount="64" Body="&lt;p&gt;Can Adaboost's ensemble classifier perform worse than the best of the weak learners considered? If so when in what case of weak learner the ensemble learning does not perform better?&lt;/p&gt;&#xA;" OwnerUserId="30641" LastActivityDate="2018-03-03T03:36:45.990" Title="AdaBoost - Ensemble model perform poor than best weak classifier" Tags="|machine-learning|classification|xgboost|ensemble-modeling|ensemble-learning|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28553" PostTypeId="1" CreationDate="2018-03-03T08:00:16.383" Score="3" ViewCount="80" Body="&lt;p&gt;I have been playing around trying to implement my own feedforward neural network. To try it out I decided on an easy example. 3 inputs, 3 output. When you send in &lt;code&gt;(1,0,0)&lt;/code&gt; the expected output is &lt;code&gt;(0,1,0)&lt;/code&gt;. When you send in &lt;code&gt;(0,1,0)&lt;/code&gt; the expected output is &lt;code&gt;(0,0,1)&lt;/code&gt;. And so on, you see the pattern.&lt;/p&gt;&#xA;&lt;p&gt;When working with a test data size of one I get a perfect result. After 100 training sessions with the input of &lt;code&gt;(1,0,0)&lt;/code&gt; i get &lt;code&gt;(0.0192068, 0.980584, 0.0187576)&lt;/code&gt; and the more I train the better the result is.&lt;/p&gt;&#xA;&lt;p&gt;However when I try with 2 sets of training data &lt;code&gt;(1,0,0)&lt;/code&gt; and &lt;code&gt;(0,1,0)&lt;/code&gt;, I get the same result for trying both inputs after the training: &lt;code&gt;(0.0189567, 0.499995, 0.500002)&lt;/code&gt; which isn't really what I wanted. The more I train the more even the result gets.&lt;/p&gt;&#xA;&lt;p&gt;When I try with a 100 diffrent training examples (using an equal distribution of all 7 combinations: &lt;code&gt;(1, 0 ,0), (0, 1 ,0), (0, 0 ,1), (1, 1 ,0), (0, 1 ,1), (1, 0 ,1), (1, 1 ,1)&lt;/code&gt;) the results get just more and more uniform : &lt;code&gt;(0.420004, 0.429997, 0.430002)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;So my question is this: Could this result be in response to a general weakness of feedforward neural networks, in other words, it can't handle training data that is too uniform? Or do you think this could be because of a math error that has snuck in somewhere? I am afraid I am new enough to the concept of feedforward neural networks I can't tell one from the other.&lt;/p&gt;&#xA;&lt;p&gt;Ps.: For those who are curious about what code I am using I am following this code:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network2.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network2.py&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am doing a port to C++ from that code example. I find porting code is a good way to learn because you really have to dig in deep into what every piece of the code is doing to get a clear understanding of how to port it. I also happen to be one of those weird ones that are more comfortable with C++ rather than Python DS.&lt;/p&gt;&#xA;" OwnerUserId="47169" LastEditorUserId="97640" LastEditDate="2021-05-26T04:41:02.733" LastActivityDate="2021-05-26T04:41:02.733" Title="Can you have too uniform test data in a feedforward neural network?" Tags="|machine-learning|deep-learning|neural-network|dataset|implementation|" AnswerCount="0" CommentCount="14" ContentLicense="CC BY-SA 4.0" />
  <row Id="28554" PostTypeId="1" CreationDate="2018-03-03T08:44:50.823" Score="1" ViewCount="323" Body="&lt;p&gt;I have a vector and want to detect outliers in it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need an outlier detection method (a non-parametric method) which can detect red points as outliers.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/myz42.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/myz42.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;I have a lot of vectors like this. The pattern is similar to this but the values are different. I mean we can see the height since the values are not constant but overall the patterns of outliers against the normal data are like this. This means in one place they beak the continuity of the blues points and go in different directions. Here is another figure.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/3RnzZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3RnzZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="12345" LastEditDate="2018-03-05T20:41:04.690" LastActivityDate="2018-03-10T13:08:08.880" Title="An outlier detection for this data" Tags="|machine-learning|data-mining|anomaly-detection|outlier|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="28556" PostTypeId="2" ParentId="28554" CreationDate="2018-03-03T10:40:00.870" Score="2" Body="&lt;p&gt;Not sure if this is time series data, but it looks like it might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For any given narrow window of time there is a distribution that looks like it's centered somewhere a bit higher than y=200 and has a reasonably stable spread. So whatever that distribution is, that's your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then monitor the recent window, and treat that as a sample. Perform a goodness of fit test against the model. If it fails then the points in the window are anomalous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; Based on what you say about the data representing roughly a minute of time, it looks like you could use a window that's about 1 second wide. Run a hypothesis test to see if the data in the window comes from the typical distribution, and if not, just mark all the points in that window as being anomalous. If you need finer resolution than 1s, simply shrink the window.&lt;/p&gt;&#xA;" OwnerDisplayName="user10803" LastEditorDisplayName="user10803" LastEditDate="2018-03-10T13:08:08.880" LastActivityDate="2018-03-10T13:08:08.880" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28558" PostTypeId="2" ParentId="28255" CreationDate="2018-03-03T10:53:58.163" Score="1" Body="&lt;p&gt;Found some implementation of lstm-based handwriter. Maybe I will use some parts.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/sjvasquez/handwriting-synthesis&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/sjvasquez/handwriting-synthesis&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46785" LastActivityDate="2018-03-03T10:53:58.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28559" PostTypeId="1" CreationDate="2018-03-03T11:30:20.160" Score="0" ViewCount="86" Body="&lt;p&gt;Goal: Generate a list of 100 products per vertical (e.g. fashion, electronics) that the teams should source, discount, and list on the website over a specific period. You may assume all customers are online only. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thinking so far:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predict the customers that will come to the website during the&#xA;specific period (time series). Only 30-40% customers return YOY. &lt;/li&gt;&#xA;&lt;li&gt;Understand what they want (use search data, add to basket but didn't checkout etc). Potentially segment it further by looking at those customers who generate the highest revenue in general vs one off purchasers.&lt;/li&gt;&#xA;&lt;li&gt;Further filter those products that these customers add (or take from 'viewed or saved' state to the 'checkout' state) once the product is on deal.&lt;/li&gt;&#xA;&lt;li&gt;Potentially use clustering to recommend products similar to those from step 3 but that have never been on a deal? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I cannot influence the amount of discount. I can only influence &lt;em&gt;what&lt;/em&gt; products we source. Therefore I want to source the ones with the highest potential of purchasers from the customer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on the approach above? &lt;/p&gt;&#xA;" OwnerUserId="41377" LastEditorUserId="41377" LastEditDate="2018-03-03T11:38:23.673" LastActivityDate="2024-01-17T12:07:45.970" Title="Sourcing (discounted) products customers want" Tags="|machine-learning|clustering|predictive-modeling|time-series|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28560" PostTypeId="2" ParentId="28538" CreationDate="2018-03-03T11:50:25.200" Score="1" Body="&lt;p&gt;Your definition is correct. For the reference you can compare it with the probabilistic model from &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec#scaling_up_with_noise-contrastive_training&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow &quot;Vector Representations of Words&quot; tutorial&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$$&#xA;\begin{align}&#xA;P(w_t | h) &amp;amp;= \text{softmax} (\text{score} (w_t, h)) \\&#xA;\\&#xA;           &amp;amp;= \frac{\exp \{ \text{score} (w_t, h) \} }&#xA;             {\sum_\text{Word w' in Vocab} \exp \{ \text{score} (w', h) \}}&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's the same as yours, but they generalize the condition $h$ as a &lt;em&gt;history&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of skip-gram, the target word $w_t$ is any &lt;em&gt;context word&lt;/em&gt; ($w_{-t}$ in your notation) and the history is the &lt;em&gt;center word&lt;/em&gt; ($w_{t}$ in your notation). You specify $\theta$ explicitly in the condition, but that is usually omitted, because there is only one vocabulary in a given problem.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-03-03T11:50:25.200" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28562" PostTypeId="1" CreationDate="2018-03-03T12:05:07.227" Score="0" ViewCount="745" Body="&lt;p&gt;I am seeking theoretical suggestions more than else. This is my first actual practical work, and I am kinda stuck now that I have done a few things: I don't know in which direction to look in order to improve my model.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;What kind of data I have&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a dataset of information about a population. Each entry is a percentage of people belonging to a certain category (my features) over the whole number of inhabitants. I have &lt;code&gt;52&lt;/code&gt; observation and &lt;code&gt;35&lt;/code&gt; features, so a &lt;code&gt;52x35 input matrix&lt;/code&gt;. So my target is a &lt;code&gt;1D-vector&lt;/code&gt; of &lt;code&gt;52&lt;/code&gt; elements, and these are percentages too.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       male  female  students  employees  freelancers  unemployed  catholics...&#xA;city1  ##%  ##%     ##%     ##%...&#xA;city2&#xA;city3&#xA;...&#xA;city52 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and my target vector is the results of the winning candidate of the last presidential election in each city.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;What I did&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I thought that, having such a small dataset, it would be better to stick to simple models, staying away from polynomial models etc. in order to avoid overfitting. Some engineering work to do on the features, too. &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Feature selection&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I removed redundant information and I agglomerated variables, for example by summing up &lt;code&gt;&quot;irrelevant&quot;&lt;/code&gt; percentage (compared to other values of the same conceptual categories) and filing them under new &quot;other&quot; features.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Model selection&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;As I said, I decided to keep it simple, start with some simple linear regression and then checking how other regression models performed compared to linreg. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset was splitted into &lt;code&gt;train $(30%)$ and test set&lt;/code&gt;. For each model I selected the related parameter which maximized the score. I then did &lt;code&gt;Random Feature Elimination (with cross validation)&lt;/code&gt; for each model, using the parameters selected in the previous step, leaving just a few most relevant features (since I know I have few observations compared to the number of features), and took this as final output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I focused mainly on linear and ridge regression, with the latter performing better. My best &lt;code&gt;R2&lt;/code&gt; is around &lt;code&gt;0.7xx&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see a bit of $overfitting$, but no clear path in the residuals. The target points with the worst predictions actually make sense in the context of the work (they are all belonging to a certain category).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After this I tried with some trees and randomforest regressors, but they performed worse than ridge regression.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;What to do next&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Well, this is where I lose my path. I think I did the right things until now, considering the kind of data I have (but please feel free to correct me), but.. is that it? Am I done or do I have something more to do, or maybe do again something in a different way? When can I call my self satisfied?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my very first ML work so any suggestion is very welcome.&lt;/p&gt;&#xA;" OwnerUserId="45764" LastEditorUserId="45764" LastEditDate="2018-03-03T15:28:05.257" LastActivityDate="2018-04-02T15:58:46.633" Title="How can I improve my regression model?" Tags="|machine-learning|predictive-modeling|regression|prediction|model-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28564" PostTypeId="1" AcceptedAnswerId="28573" CreationDate="2018-03-03T13:18:29.290" Score="0" ViewCount="141" Body="&lt;p&gt;I read &lt;a href=&quot;https://arxiv.org/pdf/1706.01399.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper, but I am having trouble understanding what Teacher Helping technique (page 3) is in context of RNN.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain to me what it is? Please assume I don't have much experience in statistics. I'll be glad if this explanation would be as simple as possible (I'll be even more grateful for some example).&lt;/p&gt;&#xA;" OwnerUserId="47182" LastEditorUserId="29575" LastEditDate="2018-03-03T15:19:23.807" LastActivityDate="2018-03-03T19:07:38.993" Title="What is Teacher Helping technique?" Tags="|rnn|lstm|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28566" PostTypeId="2" ParentId="26998" CreationDate="2018-03-03T14:13:15.223" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;It is possible to predict the solution for arbitrary equations provided the at least one real root exists?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You‘ll just need a sufficiently large training dataset and network that is flexible enough (start with one hidden layer). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You’re right your problem is not linear, thus you’ll need non-linear activation function. &lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2018-03-03T14:13:15.223" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28568" PostTypeId="2" ParentId="11554" CreationDate="2018-03-03T15:19:23.247" Score="1" Body="&lt;p&gt;Note that using np.linalg.eigh will produce wrong results since np.linalg.inv(S_W).dot(S_B) is not Hermitian. It still should have real eigenvalues and eigenvectors, and nonzero imaginary parts are rounding errors. It should therefore be safe to just use the real parts, but you can ad a check if the imaginary parts are indeed small.&lt;/p&gt;&#xA;" OwnerUserId="47186" LastActivityDate="2018-03-03T15:19:23.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28569" PostTypeId="2" ParentId="28562" CreationDate="2018-03-03T15:50:08.200" Score="1" Body="&lt;p&gt;I have a similar dataset atm and have been doing some research, not yet finished.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What needs to be kept in mind is the need for generalization given that there is such few samples and many dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The linear regression model is not robust enough given the dataset, it does not handle a high number of dimensions, compared to samples, well. Even if you could tune in order to get a good result, you are still probably overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stepwise feature/model selectors such as Cp-statistics does not seem to do well with so many dimensions vs samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ridge, Lasso, and Elastic net seems to be the way to go for feature selection. These penalized linear regression models handles the high number of dimensions vs samples much better. The linear regression is penalized in order to generalize better and not overfit, creating a less complex solution. &#xA;&lt;a href=&quot;https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;R: GLMNET - Penalized regression&lt;/a&gt;&#xA;Depending on alpha, you may receive a different feature set, coefficient that are not zero. E.g Lasso is biased for not returning features that are very correlated with each other while ridge might return a set of variables/dimensions/predictors/features that are correlated with each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;K-Fold Cross-validation could be used in order to check the features selected are selected over and over again. This could be used in order to make sure that you don't overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this gives you a few ideas.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastEditorUserId="31086" LastEditDate="2018-03-03T15:55:36.407" LastActivityDate="2018-03-03T15:55:36.407" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28573" PostTypeId="2" ParentId="28564" CreationDate="2018-03-03T19:07:38.993" Score="1" Body="&lt;p&gt;In normal RNNs, you can train using either back-propagation through time (BPTT) or teacher forcing. With BPTT the network receives as input for each time step the output of the previous time step. In teacher forcing, the network receives the gold data tokens directly; this induces &lt;em&gt;exposure bias&lt;/em&gt; on the trained network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With textual GANs, you normally don't feed the gold data to the generator, because the generator is generating the sequence so there is no gold data to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the article you refer, they are taking subsequences from real data, feeding them to the generator and having it only generate the final token. Then the discriminator receives the concatenation of the real sequence prefix and the generated final token. This is what they call &quot;teacher helping&quot;.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-03T19:07:38.993" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28574" PostTypeId="1" AcceptedAnswerId="28576" CreationDate="2018-03-03T19:47:23.473" Score="4" ViewCount="39744" Body="&lt;p&gt;I've this code in order to visualize the most important feature of each model:&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dtc = DecisionTreeClassifier(min_samples_split=7, random_state=111)&#xA;rfc = RandomForestClassifier(n_estimators=31, random_state=111)&#xA;trained_model = dtc.fit(features_train, labels_train)&#xA;trained_model.fit(features_train, labels_train)&#xA;predictions = trained_model.predict(features_test)&#xA;importances = trained_model.feature_importances_&#xA;    std = np.std([trained_model.feature_importances_ for trained_model in &#xA; trained_model.estimators_], axis=0)&#xA; indices = np.argsort(importances)[::-1]&#xA;    for f in range(features_train.shape[1]):&#xA;        print(&quot;%d. feature %d (%f)&quot; % (f + 1, indices[f], importances[indices[f]]))&#xA;        plt.figure()&#xA;        plt.title(&quot;Feature importances&quot;)&#xA;        plt.bar(range(features_train.shape[1]), importances[indices], color=&quot;r&quot;, yerr=std[indices], align=&quot;center&quot;)&#xA;        plt.xticks(range(features_train.shape[1]), indices)&#xA;        plt.xlim([-1, features_train.shape[1]])&#xA;        plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using RandomForestClassifier this code runs good but when I try it using Decison Trees classifier I get the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)&#xA;&#xA;builtins.AttributeError: 'DecisionTreeClassifier' object has no attribute 'estimators_'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which attribute should I use see the most important feature of each model?&lt;/p&gt;&#xA;" OwnerUserId="24719" LastEditorUserId="29575" LastEditDate="2018-04-07T14:12:38.127" LastActivityDate="2018-04-07T14:12:38.127" Title="'DecisionTreeClassifier' object has no attribute 'importances_'" Tags="|python|predictive-modeling|feature-selection|decision-trees|estimators|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28575" PostTypeId="1" CreationDate="2018-03-03T20:01:20.237" Score="1" ViewCount="1126" Body="&lt;p&gt;Polynomial regression and multilayer perceptrons have different structures and different learning procedures. What are these two algorithms pros and cons? Are there some situations where one should perform better than the other?&lt;/p&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="28175" LastEditDate="2018-04-03T11:52:59.500" LastActivityDate="2018-04-03T11:52:59.500" Title="Polynomial regression vs. multilayer perceptron" Tags="|machine-learning|neural-network|deep-learning|mlp|normal-equation|" AnswerCount="1" CommentCount="4" ClosedDate="2018-04-05T21:07:09.753" ContentLicense="CC BY-SA 3.0" />
  <row Id="28576" PostTypeId="2" ParentId="28574" CreationDate="2018-03-03T20:08:38.020" Score="4" Body="&lt;p&gt;Do Checkout this &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To Visualise The Tree Itself&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import export_graphviz&#xA;import graphviz&#xA;&#xA;export_graphviz(tree, out_file=&quot;mytree.dot&quot;)&#xA;with open(&quot;mytree.dot&quot;) as f:&#xA;    dot_graph = f.read()&#xA;graphviz.Source(dot_graph)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OR&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import convert_to_graphviz&#xA;convert_to_graphviz(tree)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OR&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import convert_to_graphviz&#xA;import graphviz&#xA;&#xA;graphviz.Source(export_graphviz(tree))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The Visualisation You Can Get Will be Whole Tree Itself..&#xA;&lt;a href=&quot;https://i.stack.imgur.com/YlrzS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YlrzS.png&quot; alt=&quot;RF&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;To Display Feature Importances&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.tree import DecisionTreeClassifier&#xA;classifier = DecisionTreeClassifier()&#xA;classifier.fit(features, labels)&#xA;for name, importance in zip(features.columns, classifier.feature_importances_):&#xA;    print(name, importance)&#xA;    ## Now You Can Do Whatever You Want(plot them using a Barplot etc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-03T20:14:32.990" LastActivityDate="2018-03-03T20:14:32.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28578" PostTypeId="2" ParentId="16306" CreationDate="2018-03-03T20:47:59.293" Score="3" Body="&lt;p&gt;Actually yes! While PySpark's built-in data frames are optimized for large datasets, they actually performs worse (i.e. slower) on small datasets, typically less than 500gb. Running Pandas in Spark can be very useful if you are working with a different sizes of datasets, some of which are small and can be held on a local machine. That being said, if you have data that is too big to fit on one machine, you will need to use spark data frames. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some good links to learn more about Pandas for Spark: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/47975534/are-pyspark-and-pandas-certified-to-work-together&quot;&gt;stack overflow question&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://databricks.com/session/data-wrangling-with-pyspark-for-data-scientists-who-know-pandas&quot; rel=&quot;nofollow noreferrer&quot;&gt;video explanation&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://medium.com/@chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2&quot; rel=&quot;nofollow noreferrer&quot;&gt;differences between pandas &amp;amp; spark data frames&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44624" LastActivityDate="2018-03-03T20:47:59.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28579" PostTypeId="2" ParentId="28481" CreationDate="2018-03-03T20:57:01.110" Score="1" Body="&lt;p&gt;From your description, it might be an issue of too high learning rate. When this happens, the weights cannot reach closer to the minimum and the loss does not go down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Accordingly, I would suggest reducing the learning rate to 0.001 and using Adam. &lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-03T20:57:01.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28582" PostTypeId="1" CreationDate="2018-03-03T23:53:07.283" Score="6" ViewCount="127" Body="&lt;p&gt;I’d like to ask about a case when we would like to predict the best class of some input variable, so that the probability of event will be maximized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example the advertisement type for a given customer, which will maximize the probability of purchase.&#xA;In collected data we have many various ad types, customer descriptors and informations if the purchase was made or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution which comes to my mind is to treat the ad type as an input variable, train regular probability model, then make predictions with all ad type configuration and then pick one giving the best estimation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the other options?&lt;/p&gt;&#xA;" OwnerUserId="47202" LastActivityDate="2020-08-24T12:02:40.307" Title="How can we model the class which maximizes the event probability?" Tags="|classification|predictive-modeling|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28583" PostTypeId="1" AcceptedAnswerId="28586" CreationDate="2018-03-04T02:28:41.967" Score="1" ViewCount="1184" Body="&lt;p&gt;I'm new in Machine learning and I'm working on a problem related to text. I know that in ML we can use features as numerical values as input to neural network, but I don't know how to use features as words. In some papers I read that we take features to be n words with some property. I really don't understand how is that possible. Please, if it is not a problem, just to tell me some good papers or textbooks or links where it is explained how to do that.&lt;/p&gt;&#xA;" OwnerUserId="47204" LastEditorUserId="28175" LastEditDate="2018-03-04T09:44:42.917" LastActivityDate="2018-03-04T09:44:42.917" Title="Words as features of a neural networks" Tags="|machine-learning|neural-network|deep-learning|lstm|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28584" PostTypeId="1" CreationDate="2018-03-04T03:18:43.390" Score="1" ViewCount="107" Body="&lt;p&gt;I am working with an LSTM nn built with Keras. I have the need to pass in a history of events as a single factor and I'm thinking of doing it with as a list. But I've never passed in a list in this manner - is that something that is possible with Keras? Or do I need to go through some sort of categorization process for this? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastEditorUserId="23240" LastEditDate="2018-03-04T04:07:22.883" LastActivityDate="2018-03-06T22:48:12.970" Title="How would you use a series of events as a factor for a NN?" Tags="|python|keras|lstm|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28585" PostTypeId="1" AcceptedAnswerId="28589" CreationDate="2018-03-04T04:51:11.443" Score="1" ViewCount="1639" Body="&lt;p&gt;In one &lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-319-28397-5_1&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper on ML&lt;/a&gt; I read that chi square distribution is used to reduce the number of features. In that paper, features are words. That paper is related to &lt;strong&gt;Sentiment Analysis&lt;/strong&gt;, so we have &lt;strong&gt;&quot;positive&quot;, &quot;negative&quot; and &quot;neutral&quot; category.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;How to calculate chi square distribution in that case? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In Python there is &lt;code&gt;scipy.stats.chisquare&lt;/code&gt; which gives chi_square value and a p_value. What do we do then with these two pieces of information? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What to do for example with word &quot;good&quot; as a feature?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to calculate chi square distribution, and what to do with that? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What does it mean to exclude some feature from the set of features, because in that paper it is mentioned that we take n of them with top chi square. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I really don't know how to do it. If there is any paper or book or link to learn that, please tell me.&lt;/p&gt;&#xA;" OwnerUserId="47204" LastEditorUserId="29575" LastEditDate="2018-03-04T06:07:01.660" LastActivityDate="2018-03-04T15:09:52.480" Title="Chi square distribution for feature selection" Tags="|machine-learning|neural-network|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28586" PostTypeId="2" ParentId="28583" CreationDate="2018-03-04T05:05:18.047" Score="1" Body="&lt;p&gt;You need to make a dictionary of words. It means you have to make a dictionary which you assign each word a unique value. then you can use one-hot-encoding to represent each word uniquely. If this is what you need it will do what you want. But this has a big problem. When you think about cats and dogs, you may find similarities and differences between them. This is because you have more knowledge than the only representation of words in your brain. Consequently, you should use approaches to assign a unique number to each word, and put near concepts as neighbors. For the first part take a look at &lt;a href=&quot;https://www.coursera.org/learn/nlp-sequence-models/lecture/gw1Xw/language-model-and-sequence-generation&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and the second part, take a look at &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-04T05:05:18.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28588" PostTypeId="1" CreationDate="2018-03-04T06:20:40.273" Score="-3" ViewCount="656" Body="&lt;p&gt;This is an assignment question. Can someone someone give me some clue on how to get through: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A generative approach is used for a binary classification problem and it turns out that the resulting classifier predicts &lt;em&gt;positive&lt;/em&gt; at &lt;strong&gt;&lt;em&gt;all&lt;/em&gt;&lt;/strong&gt; points &lt;strong&gt;&lt;em&gt;x&lt;/em&gt;&lt;/strong&gt; in the input space. What can we conclude for sure? Check all that apply.&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;There are no negative points in the training set. &lt;/li&gt;&#xA;  &lt;li&gt;The positive points are spread out     over the space, while the negative  points are    concentratd in a small region.     &lt;/li&gt;&#xA;  &lt;li&gt;There are fewer  negative points than positive points in the training set. &lt;/li&gt;&#xA;  &lt;li&gt;The density   of  positive points is greater than the density of  negative points    everywhere in   the    space.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You need to select points that apply out of those four bullet points. Thanks&lt;/p&gt;&#xA;" OwnerUserId="47212" LastEditorUserId="47212" LastEditDate="2018-03-04T21:16:13.143" LastActivityDate="2018-05-04T02:33:28.573" Title="A binary classifier predicts only one type of class" Tags="|machine-learning|generative-models|homework|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28589" PostTypeId="2" ParentId="28585" CreationDate="2018-03-04T07:49:01.313" Score="2" Body="&lt;p&gt;There are different ways for feature selection. A very good read in &lt;a href=&quot;https://machinelearningmastery.com/feature-selection-machine-learning-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;machinelearningmastery&lt;/a&gt;, to recap:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Univariate Selection.&lt;/li&gt;&#xA;&lt;li&gt;Recursive Feature Elimination.&lt;/li&gt;&#xA;&lt;li&gt;Principal Component Analysis.&lt;/li&gt;&#xA;&lt;li&gt;Feature Importance.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Chi-Squared test For Feature Selection&lt;/strong&gt; goes under the Univariate Selection method for non-negative features. My favorite explanation of chi-Squared in one photo taken from this &lt;a href=&quot;https://chrisalbon.com/machine_learning/feature_selection/chi-squared_for_feature_selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blogpost&lt;/a&gt; is:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/iqNv2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iqNv2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see &lt;code&gt;scikit-learn&lt;/code&gt; has an &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation&lt;/a&gt; for feature_selection using chi2 (perhaps according to &lt;code&gt;scipy.stats.chisquare&lt;/code&gt;) as was shown very briefly in the above-mentioned blog post. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want a more thorough explanation and details how test ranks features based on statistics according to chi2 distribution and p-value etc., and also how to build your own chi2 class for feature selection in Python see this great &lt;a href=&quot;http://www.handsonmachinelearning.com/blog/2AeuRL/chi-square-feature-selection-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt;. Obviously one can read about the basics of chi2 distribution and test in &lt;a href=&quot;https://en.wikipedia.org/wiki/Chi-squared_test&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-03-04T15:09:52.480" LastActivityDate="2018-03-04T15:09:52.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28590" PostTypeId="2" ParentId="18402" CreationDate="2018-03-04T08:46:29.367" Score="2" Body="&lt;p&gt;Also the paper said, we do not have a complete explanation to this phenomenon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But here is kinda explanation about that.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;While we do not have a complete explanation to this phenomenon, we&#xA;  believe that it is caused by the introduction of many short term&#xA;  dependencies to the dataset. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Normally, when we concatenate a source sentence with a target&#xA;  sentence, each word in the source sentence is far from its&#xA;  corresponding word in the target sentence. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;As a result, the problem has a large “minimal time lag” [17]. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;By reversing the words in the source sentence, the average distance&#xA;  between corresponding words in the source and target language is&#xA;  unchanged. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;However, the first few words in the source language are now very close&#xA;  to the first few words in the target language, so the problem’s&#xA;  minimal time lag is greatly reduced. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Thus, backpropagation has an easier time “establishing communication”&#xA;  between the source sentence and the target sentence, which in turn&#xA;  results in substantially improved overall performance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="47215" LastActivityDate="2018-03-04T08:46:29.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28591" PostTypeId="2" ParentId="28582" CreationDate="2018-03-04T09:11:20.033" Score="0" Body="&lt;p&gt;If you want to predict the best class of input variable, why don't you take the input variable itself as the output? And say the probability or a boolean of whether the event is happening or not as a input. I guess with enough training data, it may predict right.&#xA;But that might not be an ideal solution, since you already specify the model the best class for which the event happens, so your approach is a better one.&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-03-04T09:11:20.033" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28592" PostTypeId="2" ParentId="28559" CreationDate="2018-03-04T09:27:09.797" Score="0" Body="&lt;p&gt;Clustering is not well suited for &lt;em&gt;prediction&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use a recommender system instead.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-04T09:27:09.797" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28593" PostTypeId="1" AcceptedAnswerId="28895" CreationDate="2018-03-04T09:43:05.547" Score="5" ViewCount="1407" Body="&lt;p&gt;I have a convolutional layer &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1CdqYkHYYXox0QDzfC-uIKgwQA_ZWbMUUiVg5pJ3_j2s/edit#gid=0&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;(link)&lt;/strong&gt;&lt;/a&gt; with an input &lt;strong&gt;5x5x2&lt;/strong&gt; (width, height, depth):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The layer has 3 filters with dimensions &lt;strong&gt;3x3x2&lt;/strong&gt;, it produces an output with dimensions &lt;strong&gt;3x3x3&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have completed the forward pass:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vO6t1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vO6t1.png&quot; alt=&quot;Forward Pass&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a backward pass I calculated gradients of weights $dL/dw$, but I am not sure how to calculate $dL/dh$ (gradients for the previous layer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/szzpQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/szzpQ.png&quot; alt=&quot;Backward Pass&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First or all, is my calculation of $dL/dw$ valid ? (please refer to external spread sheet linked to the question) And most importantly, how do I propagate deltas to the previous layer ?&lt;/p&gt;&#xA;" OwnerUserId="45821" LastEditorUserId="45821" LastEditDate="2018-03-09T16:55:00.657" LastActivityDate="2018-04-05T13:45:17.833" Title="How to propagate error back to previous layer in CNN?" Tags="|machine-learning|neural-network|backpropagation|cnn|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="28594" PostTypeId="1" AcceptedAnswerId="28698" CreationDate="2018-03-04T09:45:10.040" Score="13" ViewCount="11981" Body="&lt;p&gt;I'm looking for tools that would help me and my team annotate training sets. I work in an environment with large sets of data, some of which are un- or semi-structured. In many cases there are registration that help in finding a grounded truth. In many cases however a curated set is needed, even if it just were for evaluation. A complicating factor is that some of the data can not leave the premise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We are looking to annotate an object detection task, but I anticipate an image segmentation task, a text classification task and a sentiment detection task in the near future. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm looking for is a system that can help a group make an annotation, preferably in a way that motivates the annotators by showing group progress, relative individual progress and perhaps personal inter annotator agreement.&lt;/p&gt;&#xA;" OwnerUserId="27432" LastActivityDate="2020-07-02T13:18:09.680" Title="What are helpful annotation tools (if any)" Tags="|classification|tools|annotation|" AnswerCount="5" CommentCount="1" CommunityOwnedDate="2019-01-31T00:05:48.343" ContentLicense="CC BY-SA 3.0" />
  <row Id="28595" PostTypeId="2" ParentId="28499" CreationDate="2018-03-04T09:52:06.320" Score="2" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;em&gt;Something Like This Should Do The Job&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;When you are doing something new, Mistakes are likely..&#xA;&lt;strong&gt;Use At Your own Risk Or Try It Out on A Sample And Try it on a Seperate Directory Completely&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import os&#xA;import numpy as np&#xA;import shutil&#xA;&#xA;# source is the current directory&#xA;# Open dataset file&#xA;dataset = pd.read_csv('dataset.csv')&#xA;file_names = list(dataset['filenames'].values)&#xA;img_labels = list(dataset['labels'].values)&#xA;&#xA;folders_to_be_created = np.unique(list(dataset['labels'])).values&#xA;&#xA;source = os.getcwd()&#xA;&#xA;for new_path in folders_to_be_created:&#xA;    if not os.path.exists(&amp;quot;.//&amp;quot; + new_path):&#xA;        os.makedirs(new_path)&#xA;&#xA;## Be sure that there is nothing else in your directory except the data, csv and the code file, IT's Better to only have your data in that directory and reference the CSV file from a different Directory...&#xA;&#xA;folders = folders_to_be_created.copy()&#xA;&#xA;for f in range(len(file_names)):&#xA;&#xA;  current_img = file_names[f]&#xA;  current_label = img_labels[f]&#xA;&#xA;   ## **Check this Line Accordingly** &#xA;&#xA;  shutil.move(&amp;quot;path//to//current//file&amp;quot;, &amp;quot;path//to//new//destination//folder//current_label//&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/8858026&quot;&gt;Link for Shutil&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://automatetheboringstuff.com/chapter9/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Another Link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-05-16T06:23:01.740" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28596" PostTypeId="1" CreationDate="2018-03-04T09:53:59.677" Score="1" ViewCount="564" Body="&lt;p&gt;I have a multilabel data which I've trained using different classifiers with &lt;strong&gt;MEKA&lt;/strong&gt; (multilabel version of &lt;strong&gt;WEKA&lt;/strong&gt;) and the evaluation results (e.g. accuracy) that &lt;strong&gt;MEKA&lt;/strong&gt; gives me are different from those I get from the same classifiers of &lt;strong&gt;scikit-learn&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Binary Relevance method (which is both in MEKA and scikit-multilearn), because of my data being multilabel. And for instance, I am using SVM which is &lt;em&gt;SMO&lt;/em&gt; in MEKA (and WEKA) and &lt;em&gt;linearSVC&lt;/em&gt; in sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the accuracy I get from &lt;em&gt;10-fold cross-validation&lt;/em&gt; with &lt;em&gt;BR&gt;SMO&lt;/em&gt; in MEKA is about &lt;strong&gt;50%&lt;/strong&gt; where I get something around &lt;strong&gt;20%&lt;/strong&gt; accuracy from &lt;em&gt;BinaryRelevanvce&gt;LinearSVC&lt;/em&gt;. Other evaluation measures (e.g. f1-score, hamming loss etc.) don't match as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to give all the parameters I could find in the input of &lt;em&gt;LinearSVC&lt;/em&gt;, the same value as those of &lt;em&gt;SMO&lt;/em&gt; but there's no luck. My Python code would be like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from skmultilearn.problem_transform import BinaryRelevance&#xA;from sklearn.svm import LinearSVC&#xA;&#xA;my_classifier = BinaryRelevance(&#xA;                classifier=LinearSVC(C=1, random_state=1, tol=0.001, max_iter=10000, dual=False),&#xA;                require_dense=[True, True])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I must add that using &lt;em&gt;BR&gt;SMO&lt;/em&gt; in MEKA to train and evaluate the model would take almost &lt;strong&gt;an hour&lt;/strong&gt; on my data, but sklearn's &lt;em&gt;linearSVC&lt;/em&gt; (with skmultilern's &lt;em&gt;BinaryRelevance&lt;/em&gt;) takes less than &lt;strong&gt;10 minutes&lt;/strong&gt; on the same data (hence the lower accuracy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And by the way, I'm using &lt;code&gt;accuracy_score&lt;/code&gt; method in &lt;code&gt;sklearn.metrics&lt;/code&gt; for accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is what am I missing? How can I get the same accuracy from &lt;strong&gt;SVM&lt;/strong&gt; in &lt;strong&gt;MEKA&lt;/strong&gt; and &lt;strong&gt;scikit-learn&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="47217" LastEditorUserId="29575" LastEditDate="2018-08-02T00:20:08.203" LastActivityDate="2018-10-01T08:01:05.773" Title="different results with MEKA vs Scikit-learn!" Tags="|scikit-learn|svm|accuracy|weka|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28597" PostTypeId="2" ParentId="28582" CreationDate="2018-03-04T11:54:17.580" Score="0" Body="&lt;p&gt;As you are looking for a conditional distribution of a variable given another one, &lt;a href=&quot;https://en.wikipedia.org/wiki/Graphical_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;graphical models&lt;/a&gt; come to mind.&lt;/p&gt;&#xA;" OwnerUserId="47219" LastActivityDate="2018-03-04T11:54:17.580" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28598" PostTypeId="1" CreationDate="2018-03-04T12:07:33.313" Score="16" ViewCount="33712" Body="&lt;p&gt;When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences.  But there does not seem to be a way to specify weights for the words calculated for instance using TF-IDF.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the usual practice to multiply the word vector embeddings with the associated TF-IDF weight? Or can word2vec organically take advantage of these somehow?&lt;/p&gt;&#xA;" OwnerUserId="47149" LastEditorUserId="18375" LastEditDate="2018-03-05T21:25:01.120" LastActivityDate="2019-01-15T20:58:49.727" Title="Word2Vec embeddings with TF-IDF" Tags="|machine-learning|nlp|word2vec|language-model|tfidf|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28600" PostTypeId="2" ParentId="28596" CreationDate="2018-03-04T13:35:13.060" Score="2" Body="&lt;p&gt;According to &lt;a href=&quot;https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent&quot;&gt;this question&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;LinearSVC is not linear SVM, do not use it if do not have to.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and furthermore&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;LinearSVC [is] one of the mistakes of sklearn developers.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZaX5M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;In this picture&lt;/a&gt; from the question you can observe how it is inaccurate, even compared to SVC with kernel=linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further differences may be explained with differences how SMO and SVC with linear kernel are implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/130293/svm-and-smo-main-differences&quot;&gt;This post&lt;/a&gt; compare the two, and the answer is SMO should be the fastest one, otherwise your dataset maybe is not suitable for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would test SVC with linear kernel and see if things change.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-03-04T13:35:13.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28602" PostTypeId="1" AcceptedAnswerId="28605" CreationDate="2018-03-04T16:57:07.827" Score="-1" ViewCount="142" Body="&lt;p&gt;Is it possible to train a neural network classifier with only one class, and after that with only another class? &#xA;For example, first train it only on recognizing dogs, and after finishing that training, only train it on recognizing cats, so in the end the net can classify between dogs and cats.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do the classes always need to be mixed for training?&#xA;Please tell me any scientific papers explaining why classes have to be mixed or explaining why they do not need to be mixed.&lt;/p&gt;&#xA;" OwnerUserId="45618" LastActivityDate="2018-03-04T17:30:13.480" Title="Training neural network classifier with one class after another" Tags="|neural-network|classification|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28603" PostTypeId="2" ParentId="27135" CreationDate="2018-03-04T17:14:43.643" Score="2" Body="&lt;p&gt;It doesn't matter, with or without flattening, a Dense layer takes the whole previous layer as input. The spatial structure information is not used anymore. Some Neural Network implementations might not be able to map a spatial structure directly into a dense layer, which is why you would need the Flatten in between. Mathematically it is exactly the same in this case.&lt;/p&gt;&#xA;&lt;p&gt;EDIT: As I expected, in the &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/layers/core.py#L743&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt; of Keras it mentions that if the rank is higher like with convolutions, they implicitly Flatten it first.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastEditorUserId="98307" LastEditDate="2020-08-13T13:56:51.303" LastActivityDate="2020-08-13T13:56:51.303" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="28604" PostTypeId="1" AcceptedAnswerId="28696" CreationDate="2018-03-04T17:15:36.313" Score="1" ViewCount="247" Body="&lt;p&gt;I was reading a paper on &lt;strong&gt;traffic flow optimization&lt;/strong&gt; using Multi-Agent Q learning. the paper proposes the following method:&#xA;Deploy a Reinforcement learning controller at each intersection with traffic lights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;first the Q value equation is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$Q^{t}(s,a) = (1- \alpha )Q^{t-1}(s,a) +  \alpha (R{t} +  \gamma max_{a}(Q^{t-1}(s,a))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;second the state is:&#xA;the sum of vehicle queues lengths at the current intersection and one hop intersections &lt;/p&gt;&#xA;&#xA;&lt;p&gt;third the action space is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mvaCh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mvaCh.png&quot; alt=&quot;the action space&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here the actions represent the possible moves of vehicles at the intersection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;forth the reward at time t is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R^{t} = -( w_1\sum q_{current intersection} + w_2\sum q_{neighbors} )$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the q refers to vehicle queues lengths, w1 and w2 are constants.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;fifth there is the algorithm in the image below where it acquires the action required for maximizing the Q value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uoNOW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uoNOW.png&quot; alt=&quot;Algorithm&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am a newbie to reinforcement learning, so please if you find my question naive, cosedire referring me to a proper textbook.&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="47233" LastEditorUserId="47233" LastEditDate="2018-03-05T17:35:47.533" LastActivityDate="2018-03-06T12:25:28.123" Title="Reinforcement Learning (Q Learning)" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28605" PostTypeId="2" ParentId="28602" CreationDate="2018-03-04T17:30:13.480" Score="0" Body="&lt;p&gt;It will probably work  but less &lt;strong&gt;Efficiently&lt;/strong&gt;(or lower Accuracy) if you train the Network one after the another (like trained for Dog's first , followed by Cats etc..) because &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;It will &lt;code&gt;Change The Weights of The Hidden Layers&lt;/code&gt; which in turn will destroy the same Recognition Capability of The Network ...&lt;/li&gt;&#xA;&lt;li&gt;Also Dogs and Cats &lt;code&gt;differ in their natural looks a lot&lt;/code&gt;, So the Net Will Have a tough Time Doing So Simultaneously..&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What You Can Do is either of the followings -&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Train Two Neural Nets Simultaneously or Train them one after the another but saving the Weights of the Layers somewhere for all your Training.. And Then Predict What the Net thinks the input image to be..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Better Will Be To Train a CNN to do the same Task with Ease and will Achieve Better Accuracy.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you only have dogs and cats, then it will be sufficient to only train on one of them i.e if it isn't a dog, then its surely a cat..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Why we prefer mixing of similar Image Groups?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's more of Human Nature to do a task which can en-compass similar things at once, rather than doing the particular tasks repeatedly..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-04T17:30:13.480" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28606" PostTypeId="2" ParentId="28598" CreationDate="2018-03-04T17:34:07.617" Score="13" Body="&lt;p&gt;Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute &lt;em&gt;word&lt;/em&gt; embeddings. The distinction becomes important when one needs to work with sentences or &lt;em&gt;document&lt;/em&gt; embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, according to some papers, is pretty successful. From &lt;a href=&quot;https://stackoverflow.com/q/47727078/712995&quot;&gt;this question from StackOverflow&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this work, tweets were modeled using three types of text representation. The first one is a bag-of-words model weighted by tf-idf (term frequency - inverse document frequency) (Section 2.1.1). The second represents a sentence by averaging the word embeddings of all words (in the sentence) and the third represents a sentence by averaging the weighted word embeddings of all words, the weight of a word is given by tf-idf (Section 2.1.2).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-03-04T17:34:07.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28607" PostTypeId="1" AcceptedAnswerId="28619" CreationDate="2018-03-04T18:28:12.100" Score="0" ViewCount="940" Body="&lt;p&gt;Having a lot of categorical features and other numerics why we need to transform the categorical to binary values? Is it for using the values in mathematics functions of the algorithms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24719" LastEditorUserId="8878" LastEditDate="2018-03-04T22:09:49.977" LastActivityDate="2018-03-05T01:57:46.977" Title="Data binning - Why we need to transform Categorical Variables?" Tags="|categorical-data|preprocessing|binary|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28608" PostTypeId="1" CreationDate="2018-03-04T18:38:25.880" Score="4" ViewCount="66839" Body="&lt;p&gt;My objective: Using pandas, check a column for matching text [not exact] and update new column if TRUE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a csv file, a data frame was created and values of a particular column - COLUMN_to_Check, are checked for a matching text pattern - 'PEA'. Based on whether pattern matches, a new column on the data frame is created with YES or NO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following data in file DATA2.csv&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY&#xA;GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness)&#xA;RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure&#xA;RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure&#xA;RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access&#xA;RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And using this code....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df=pd.read_csv('DATA2.csv')&#xA;&#xA;Search_for_These_values = ['PEA', 'DEF', 'XYZ'] #creating list&#xA;&#xA;pattern = '|'.join(Search_for_These_values)     # joining list for comparision&#xA;&#xA;IScritical=df['COLUMN_to_Check'].str.contains(pattern)&#xA;for CHECK in IScritical:&#xA;    if not CHECK:&#xA;        print CHECK&#xA;        df['NEWcolumn']='NO'&#xA;    else:&#xA;        print CHECK&#xA;        df['NEWcolumn']='YES'&#xA;&#xA;df.to_csv('OUPUT.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Printing the value of 'CHECK' returns correct values, i.e., first row returns false.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C:\Users\ME\Documents\SandBox (master)&#xA;λ python numpytest_pub.py&#xA;False&#xA;True&#xA;True&#xA;True&#xA;True&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the output csv file shows all values of 'NEWColumn' as 'YES', where on 'NEWcolumn', row[0], value should be 'NO' as the 'COLUMN_to_Check' here should not match the pattern.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;,ASSIGNMENT,Open date,Resolved date,COLUMN_to_Check,NUMBER,Open Time,RESOLVED_GROUP,RESOLVED_TIME,SUBCATEGORY,NEWcolumn&#xA;0,GBL_IS_GRC_PROCESSCONTROL,3/1/2017 13:39,11/1/2017 13:09,APAC_LT-ERP-FICO-BOKADABISH_PRD,IM-17-001200,3/1/2017 13:39,GBL_GSO_MQG,11/1/2017 13:09,Security (breach or weakness),YES&#xA;1,RSP_SERVICEDESK,12/1/2017 0:08,12/1/2017 0:27,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006462,12/1/2017 0:08,RSP_SERVICEDESK,12/1/2017 0:27,failure,YES&#xA;2,RSP_SERVICEDESK,10/1/2017 5:27,12/1/2017 0:52,APAC_LT-ERP-SUPPLY-PEA_PRD,IM-17-004667,10/1/2017 5:27,RSP_PCS_INCIDENTS,12/1/2017 0:52,failure,YES&#xA;3,RSP_SERVICEDESK,12/1/2017 2:35,12/1/2017 3:03,APAC_LT-ERP-SALES-PEA_PRD,IM-17-006483,12/1/2017 2:35,RSP_SERVICEDESK,12/1/2017 3:03,access,YES&#xA;4,RSP_SAP_BI,10/1/2017 21:04,12/1/2017 6:01,APAC_LT-ERP-SALES-PEA_PRD,IM-17-005498,10/1/2017 21:04,RSP_SAP_SALES,12/1/2017 6:01,SAP Sales,YES&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I can sense that something is missing in the CHECK part, but not able to figure out what. Can anyone help ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if the question needs rephrasing for better understanding or future community use.&lt;/p&gt;&#xA;" OwnerUserId="47205" LastActivityDate="2021-12-05T17:21:04.997" Title="Using pandas, check a column for matching text and update new column if TRUE" Tags="|python|pandas|data-cleaning|dataframe|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28611" PostTypeId="1" CreationDate="2018-03-04T19:41:28.763" Score="1" ViewCount="2083" Body="&lt;p&gt;I want to find out what people also buy when they buy a &quot;Bike Helmet&quot;. There are only 77 different products in my dataframe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might solve this using a for loop, and creating a counter. So whenever &lt;code&gt;ProductName == 'Bike Helmet&lt;/code&gt;, look at the &lt;code&gt;OrderID&lt;/code&gt; and increase the counter by 1 for all those product names. Then find the next instance of a bike helmet purchase. This approach, however, is slow.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  OrderID           ProductName  PricePerOrder  ProfitPerOrder  Quantity  \&#xA;239    10337           Bike Helmet     805.200000      178.080000        24   &#xA;238    10337           Mehmet-Skor       5.200000        0.626667        28   &#xA;237    10337          Mehmet-Tröja       7.990000        1.540000        20   &#xA;236    10337          Oyaki Kimono      10.025000        2.558333        25   &#xA;240    10337           DSW Sandals       3.600000        0.628000        40   &#xA;40     10262          Runner Shoes      14.362500        1.946250        15   &#xA;42     10262       O-Man Underwear       0.535680       -0.030720        12   &#xA;41     10262         Slip-on Shoes       4.606667        0.050667         2   &#xA;336    10374           Onesy Dress      38.677500       10.800000        15   &#xA;337    10374  Rossi Bermuda Shorts      13.665000        2.370000        30  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-03-04T22:08:52.523" Title="What is a product most frequently bought together with" Tags="|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28612" PostTypeId="2" ParentId="28608" CreationDate="2018-03-04T19:45:13.420" Score="1" Body="&lt;p&gt;&lt;code&gt;df['NEWcolumn']='NO'&lt;/code&gt; sets the whole column to the value &lt;code&gt;'NO'&lt;/code&gt;. So you see the result for the last row in your table, distributed over the whole column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a way to achieve what you want:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['NEWcolumn'][IScritical]='YES'&#xA;df['NEWcolumn'][~IScritical]='NO'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47239" LastEditorUserId="47239" LastEditDate="2018-03-04T21:01:43.440" LastActivityDate="2018-03-04T21:01:43.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28613" PostTypeId="1" AcceptedAnswerId="28628" CreationDate="2018-03-04T20:27:47.047" Score="1" ViewCount="158" Body="&lt;p&gt;I have a dataset that gives data on infertitlity and causes. The dataset is mainly 0,1 to represent &quot;yes&quot; and &quot;no&quot;. However, some fields have &quot;Sometimes&quot;, &quot;Often&quot; which would be represented by -1 or 2. I've only learnt how to do categorical data i.e. 1,0 and Numberical data. So my question is since there are more options other than 1 and 0, which type of regression do I use? Logistic regression or Linear regression?&lt;/p&gt;&#xA;" OwnerUserId="46648" LastActivityDate="2018-03-05T23:49:35.820" Title="What type of regression should I use" Tags="|dataset|regression|logistic-regression|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28614" PostTypeId="1" CreationDate="2018-03-04T21:00:45.410" Score="0" ViewCount="4594" Body="&lt;p&gt;I am trying to find out what is optimum number of neurons that can be used in MNIST dataset(60,000 training and 10,000 testing data). I build a single hidden layer model using keras,with relu activation but when I increase the neurons in hidden layer the accuracy improves slightly nearly .4% per 100 neurons ,even if I use 2000 neurons instead of 784 neurons it doesn't has any considerable change.I thought that increasing neurons above certain limit will decrease the classification accuracy because of overfitting.but It doesn't seem to look happening I even tried using 2000 neurons but the accuracy doesn't decrease on the test data.exceeding this limit I think is not possible for me because it can crash my computer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is hit and trial the only way to find the optimum number of neurons in a MLP hidden layer. How can I further tweak the parameters , can it be some other hyper parameters that I can play with like intialzing hidden layer weights initially.or adding more layers.(but adding more layer doesn't help much) I tried it.&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-03-05T12:57:44.553" Title="number of neurons for mnist dataset using mlp?" Tags="|machine-learning|keras|image-classification|perceptron|mlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28615" PostTypeId="1" CreationDate="2018-03-04T21:18:02.130" Score="0" ViewCount="1333" Body="&lt;p&gt;I have documents,  and I calculated the word vectors using word2vec for all the terms in my corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to compute similarity between documents using the bag-of-words model.  After  creating the Bag-of-words ( consisting of  vector representations). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;how do I now compute similarity between a sequence of vectors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do I simply just take the mean? and calculate the cosine similarity between the mean vectors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;or is there a better approach for computing similarity between two bags?&lt;/p&gt;&#xA;" OwnerUserId="47149" LastEditorUserId="47149" LastEditDate="2018-03-04T21:31:17.897" LastActivityDate="2018-05-04T00:32:38.470" Title="Bag of words and word2vec clarifications" Tags="|nlp|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28616" PostTypeId="2" ParentId="28615" CreationDate="2018-03-04T21:50:56.090" Score="1" Body="&lt;p&gt;BoW is a text representation like word2vec or doc 2vec. If you already have the word2vec vectors then you can take the mean for each document and calculate the similarity but it's not the direct approach. The direct approach is to calculate &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;doc2vec&lt;/a&gt; which models the documents directly and with more detailed precision than word2vec.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-04T21:50:56.090" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28617" PostTypeId="2" ParentId="28613" CreationDate="2018-03-04T21:58:42.280" Score="0" Body="&lt;p&gt;well ... some points first: Logistic Regression is for classification and Linear Regression is for regression tasks. They are conceptually different so be careful what you want to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Categorical variables can be vectorized using &lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-categorical-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;encoding&lt;/a&gt;. If there are many categories you may apply a dimensionality reduction before feeding the data to the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithm is chosen based on a &lt;a href=&quot;https://en.wikipedia.org/wiki/Model_selection&quot; rel=&quot;nofollow noreferrer&quot;&gt;model selection&lt;/a&gt; process so, in general, you do not know in advance which one is better.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-04T21:58:42.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28618" PostTypeId="2" ParentId="28611" CreationDate="2018-03-04T22:08:52.523" Score="1" Body="&lt;p&gt;Two approaches, both based on discrete mathematics. The first one is a set-theoretic approach to &lt;a href=&quot;http://www-ai.cs.uni-dortmund.de/LEHRE/SEMINARE/SS09/AKTARBEITENDESDM/FOLIEN/Frequent_Itemset_Mining_Methods.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;frequent item-set mining&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;association rule mining&lt;/a&gt;. For instance &lt;a href=&quot;https://en.wikipedia.org/wiki/Apriori_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;A-priori&lt;/a&gt; algorithm works here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the next approach is creating a graph based on orders and do graph mining on them. A-priori might be a better approach as in this case you probably need to deal with hypergraphs which makes life a bit complicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you first extract sets of different orders from your dataframe. As your problem is not general and you already know that there is only association to one product is important, you can prune your data much. Then simply rank other products bought together with Bike Helmet by counting number of their occurences and dividing by number of all (it gives you the probability estimate of each product is bought together with bike helmet).&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-04T22:08:52.523" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28619" PostTypeId="2" ParentId="28607" CreationDate="2018-03-04T22:10:41.420" Score="0" Body="&lt;p&gt;Yes. If you need the information of your categorical variables then you need to represent them by numbers as most of machine learning algorithms are math-based.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-04T22:10:41.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28621" PostTypeId="2" ParentId="11372" CreationDate="2018-03-04T23:57:31.377" Score="1" Body="&lt;p&gt;In The Visual Display of Quantitative Information, page 116-7, Tufte explains &lt;a href=&quot;https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=00040Z&quot; rel=&quot;nofollow noreferrer&quot;&gt;both the concept of a 'duck' and its provenance&lt;/a&gt;. There is a small service building in Flanders NY called 'The Big Duck' whose form is the shape of ... a duck. It is a form without function, that is its own end and shows, in data graphics, hardly any data at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/r2g8p.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/r2g8p.jpg&quot; alt=&quot;Th Big Duck in Flanders NY&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47250" LastActivityDate="2018-03-04T23:57:31.377" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28623" PostTypeId="2" ParentId="6553" CreationDate="2018-03-05T00:55:58.780" Score="1" Body="&lt;p&gt;It sounds like Doc2Vec (or paragraph/context vectors) might be the right fit for this problem.&lt;/p&gt;&#xA;&lt;p&gt;In a nutshell, in addition to the word vectors, you add a &amp;quot;context vector&amp;quot; (in your case, an embedding for the author) that is used to predict the center or context words.&lt;/p&gt;&#xA;&lt;p&gt;This means that you would benefit from all the data about &amp;quot;democracy&amp;quot; but also extract an embedding for that author, which combined should allow you to analyze the bias of each author with limited data about each author.&lt;/p&gt;&#xA;&lt;p&gt;You can use &lt;a href=&quot;https://radimrehurek.com/gensim/models/doc2vec.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim's implementation&lt;/a&gt;. The doc includes links to the source papers.&lt;/p&gt;&#xA;" OwnerUserId="47252" LastEditorUserId="85045" LastEditDate="2021-01-01T12:54:57.057" LastActivityDate="2021-01-01T12:54:57.057" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28624" PostTypeId="2" ParentId="28607" CreationDate="2018-03-05T01:57:46.977" Score="0" Body="&lt;p&gt;You need to transfer the categorical variable to numerical to feed to the model and then comes the real question, why we convert it the way we do. We convert an n level of the categorical variable to n-1 dummy variables. There are two main reasons for it:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do avoid the collinearity into the created dummy variables&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;To avoid the artificial ordering in directly converting the categorical levels to numeric levels&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-03-05T01:57:46.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28625" PostTypeId="2" ParentId="28588" CreationDate="2018-03-05T02:00:30.663" Score="1" Body="&lt;p&gt;While training your binary classifier, check for class imbalance. That's the only way you can take any final decision about the test data. As you don't know the actual result of test data, therefore, you need to make sure the classifier is good enough to generalize any new data. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-03-05T02:00:30.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28626" PostTypeId="1" AcceptedAnswerId="28640" CreationDate="2018-03-05T02:13:58.853" Score="2" ViewCount="131" Body="&lt;p&gt;When I have a dataset where each datum has &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt;,&#xA;and the &lt;code&gt;(x,y)&lt;/code&gt; has a relation of one of &lt;code&gt;y = a_i*x + b_i (i=1,2,...)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the process written below available? and which algorithm does it belong to?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The process is.....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have many points &lt;code&gt;(x, y)&lt;/code&gt;.&#xA;The machine finds 2 linear functions which represents the points.&#xA;The machine eliminates points which are far from the 3 lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, I think I put parameters the number of linear functions and criterion to judge if a point is on a line or not.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first figure is my dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to have a machine which accepts the number of line (3 in this case)&#xA;and finds 3 lines(as the second figure (the lines in the figure are just put for idea without computation)), and then finally suggests points which may belong to neither of them. (In this case, for example, (71.6, 22))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I, for instance, extend the &lt;code&gt;k-means&lt;/code&gt; algorithm to achieve this procedure?///&lt;a href=&quot;https://i.stack.imgur.com/1rjwD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1rjwD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;a href=&quot;https://i.stack.imgur.com/yi0JP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yi0JP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47254" LastActivityDate="2019-04-22T14:12:35.127" Title="Unsupervised clustering without of Data which is supposed to be on a linear function" Tags="|clustering|unsupervised-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28628" PostTypeId="2" ParentId="28613" CreationDate="2018-03-05T04:19:50.947" Score="1" Body="&lt;p&gt;Using  Logistic regression or Linear regression depend on the dependent variable(DV). Based on your question, I believe that your DV will be infertitlity(Yes/No) so you should use logistic regression because linear regression is for the continuous variables(e.g:exam score) and logistic regression is for categorical variables(e.g.L Yes/NO) &lt;/p&gt;&#xA;" OwnerUserId="47255" LastActivityDate="2018-03-05T04:19:50.947" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28630" PostTypeId="2" ParentId="28626" CreationDate="2018-03-05T05:54:56.847" Score="0" Body="&lt;p&gt;I would rather look at mixture models. Or, if there is additional noise besides the lines, use computer vision algorithms, for example, some variation of the Hough transform.&lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="6550" LastEditDate="2018-03-05T06:10:00.840" LastActivityDate="2018-03-05T06:10:00.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28631" PostTypeId="2" ParentId="28458" CreationDate="2018-03-05T06:09:00.010" Score="1" Body="&lt;p&gt;The curse of dimensionality means that your intuition fails at a certain number of features. See &lt;a href=&quot;https://martin-thoma.com/average-distance-of-points/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Average Distance of Random Points in a Unit Hypercube&lt;/a&gt; for some examples. It does not matter how many points you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, at around 100 dimensions every two points randomly sampled points are really close.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-03-05T06:09:00.010" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28632" PostTypeId="2" ParentId="5197" CreationDate="2018-03-05T06:30:12.613" Score="1" Body="&lt;p&gt;interestMeasure function from arules v1.5-5 let’s you find additional measures and also some invariant ones. Here’s the link to the R documentation: &lt;a href=&quot;https://www.rdocumentation.org/packages/arules/versions/1.5-5/topics/interestMeasure&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.rdocumentation.org/packages/arules/versions/1.5-5/topics/interestMeasure&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47260" LastActivityDate="2018-03-05T06:30:12.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28633" PostTypeId="2" ParentId="28626" CreationDate="2018-03-05T09:10:42.867" Score="0" Body="&lt;p&gt;Looking at the plot, it seems that you could project it down to 1D.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From here you simply have to calculate the distances to the lines, which will be a cluster center, or point in the 1D. &#xA;Since running K-Means, or GMM might not give you cluster centers corresponding to the placement of the line it does not seem to be the right thing to do. &lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2018-03-05T09:10:42.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28636" PostTypeId="1" AcceptedAnswerId="28646" CreationDate="2018-03-05T11:05:12.863" Score="4" ViewCount="7242" Body="&lt;p&gt;What are &lt;code&gt;.nii&lt;/code&gt; files, and how is data stored in them? I have some of these and I want to know how can I create &lt;code&gt;3D&lt;/code&gt; image of &lt;code&gt;MRI&lt;/code&gt; scan from them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can load the file in my python script using nibabel. Where to go from here next?&lt;/p&gt;&#xA;" OwnerUserId="43387" LastEditorUserId="29575" LastEditDate="2018-03-05T14:08:48.557" LastActivityDate="2018-03-05T14:08:48.557" Title="How to create 3D images from .nii file" Tags="|machine-learning|python|neural-network|classification|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28637" PostTypeId="1" AcceptedAnswerId="28638" CreationDate="2018-03-05T11:12:06.660" Score="3" ViewCount="1048" Body="&lt;p&gt;In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By intuition, I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers. For example, passing from a &lt;code&gt;p_keep = 0.8&lt;/code&gt; to &lt;code&gt;p_keep=0.5&lt;/code&gt;, especially when the inputs features are all relevant. Where &lt;code&gt;p_keep&lt;/code&gt; is the probability of keeping a node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Layer1: p_keep=0.9&#xA;Layer2: p_keep=0.7&#xA;Layer3: p_keep=0.6&#xA;Layer4: p_keep=0.55&#xA;Layer5-n: p_keep=0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="9461" LastEditorUserId="32492" LastEditDate="2018-03-05T12:14:20.547" LastActivityDate="2018-04-09T13:41:09.863" Title="Probability of dropout growth" Tags="|neural-network|deep-learning|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28638" PostTypeId="2" ParentId="28637" CreationDate="2018-03-05T11:40:21.833" Score="1" Body="&lt;p&gt;&lt;code&gt;Hinton advocates tuning dropout in conjunction with tuning the size of your hidden layer. Increase your hidden layer size(s) with dropout turned off until you perfectly fit your data. Then, using the same hidden layer size, train with dropout turned on. This should be a nearly optimal configuration&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A good initial configuration for the hidden layers is 50%. If applying dropout to an input layer, it's best to not exceed 25%.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;By intuition I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to try to generalize, I'd say that it's all about &lt;strong&gt;balancing an increase in the number of parameters of your network without over-fitting&lt;/strong&gt;. So if e.g. you start with a reasonable architecture and amount of dropout, and you want to try increasing the number of neurons in some layer, you will likely want to increase the dropout proportionately, so that the same number of neurons are not-dropped. If &lt;code&gt;p=0.5&lt;/code&gt; was optimal for &lt;code&gt;100&lt;/code&gt; neurons, a good first guess for &lt;code&gt;200&lt;/code&gt; neurons would be to try &lt;code&gt;p=0.75&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I get the feeling that dropout near the top of the network can be more damaging than dropout near the bottom of the network, because it prevents all downstream layers from accessing that information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to why &lt;code&gt;0.5&lt;/code&gt; is generally used, I think its because tuning things like the dropout parameter is really something to be done when all of the big choices about &lt;strong&gt;architecture&lt;/strong&gt; have been settled. It might turn out that &lt;code&gt;0.10&lt;/code&gt; is better, but it takes a lot of work to discover that, and if you change the filter size of a conv-net or change the overall number of layers or do things like that, you're going to have to re-optimize that value. &#xA;So &lt;code&gt;0.5&lt;/code&gt; is seen as a sort of placeholder value until we are at the stage where we are chasing down percentage points or fractions of a percentage point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Original Paper and the opening paragraph is from the original paper...&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-09T13:41:09.863" LastActivityDate="2018-04-09T13:41:09.863" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28639" PostTypeId="1" AcceptedAnswerId="28643" CreationDate="2018-03-05T11:59:59.483" Score="0" ViewCount="908" Body="&lt;p&gt;I would like to understand, how a clustering algorithm can be used (if possible) to identify naturally occurring groups within a data set, prior to building predictive models/model, and to hence improve accuracy of models/model&lt;/p&gt;&#xA;" OwnerUserId="46374" LastActivityDate="2018-03-05T19:47:26.457" Title="Clustering algorithm prior to model building?" Tags="|machine-learning|clustering|dataset|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28640" PostTypeId="2" ParentId="28626" CreationDate="2018-03-05T12:29:39.270" Score="0" Body="&lt;p&gt;Very interesting question!&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;First Approach: &lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt; + &lt;a href=&quot;https://en.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;K-means&lt;/a&gt;&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Your data will be explained very well on the &lt;a href=&quot;https://gerardnico.com/data_mining/pca&quot; rel=&quot;nofollow noreferrer&quot;&gt;second principle component&lt;/a&gt;. If you apply PCA on your data the first PC captures the data along the lines in which you completely lose the differentiation but the second PC is prependicular to the first one so your data will be projected in a way that points correspondig to each line are placed closer to each other. As you know the number of lines (number of clusters) a priori then you simply apply k-means and that's it! &#xA;See image in the link to have an idea how the second pc vector would be. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Second approach: GMMs&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://statweb.stanford.edu/~tibs/stat315a/LECTURES/em.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian Mixture Models&lt;/a&gt; are fitted to clusters in a data using maximum likelihood estimation (you use &lt;a href=&quot;https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Expectation Maximization algorithm&lt;/a&gt; for that). Your clusters along second PC are &lt;a href=&quot;https://de.mathworks.com/matlabcentral/mlc-downloads/downloads/submissions/34839/versions/2/screenshot.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;pretty gaussian&lt;/a&gt; so you will get a good soft-clustering if you fit a mixture of $n$ (number of lines again) Gaussian kernels to them.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Variant: $a$s Are Not Equal&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In this case your lines cross each other as slopes are different. Your image does not show that but I include it here anyways. In this case you fit a &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;linear regression&lt;/a&gt; to each line and keep the coefficients of the line. The you have a $2D$ data in which each line is described by just a slope and an intercept. Then the &lt;a href=&quot;https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line&quot; rel=&quot;nofollow noreferrer&quot;&gt;prependicular distance&lt;/a&gt; between each point and all lines tells you which line is closer so that is the cluster. (The distance can also simply be the &lt;a href=&quot;https://mathbitsnotebook.com/Algebra1/StatisticsReg/ST2Residuals.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;residual of that point from regression line&lt;/a&gt;. You just need a distance metric to determine the closest line)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need implementation as well, please drop a comment here so I can update answer with Python code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-03-05T12:39:06.137" LastActivityDate="2018-03-05T12:39:06.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28641" PostTypeId="2" ParentId="12851" CreationDate="2018-03-05T12:34:17.970" Score="12" Body="&lt;p&gt;The Python package &lt;code&gt;conx&lt;/code&gt; can visualize networks with activations with the function &lt;code&gt;net.picture()&lt;/code&gt; to produce SVG, PNG, or PIL Images like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nhHjO.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nhHjO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conx is built on Keras, and can read in Keras' models. The colormap at each bank can be changed, and it can show all bank types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More information can be found at: &lt;a href=&quot;http://conx.readthedocs.io/en/latest/&quot; rel=&quot;noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47270" LastActivityDate="2018-03-05T12:34:17.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28642" PostTypeId="2" ParentId="28614" CreationDate="2018-03-05T12:57:44.553" Score="2" Body="&lt;p&gt;If you train long enough and have &quot;too many&quot; hidden layer units, then you will eventually have over-training. Usually, the goal is to find the smallest number of hidden units that can do reasonably well, as that will generally give you good generalization. In fact, there is the &quot;bottle neck theory&quot; that attempts to show this effect (&lt;a href=&quot;https://openreview.net/forum?id=ry_WPG-A-&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://openreview.net/forum?id=ry_WPG-A-&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more on comparisons between flat hidden layer networks versus convolutional networks, see:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://conx.readthedocs.io/en/latest/MNIST.html#A-multi-layer-perceptron-network-for-MNIST-classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/MNIST.html#A-multi-layer-perceptron-network-for-MNIST-classification&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47270" LastActivityDate="2018-03-05T12:57:44.553" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28643" PostTypeId="2" ParentId="28639" CreationDate="2018-03-05T13:05:55.110" Score="2" Body="&lt;p&gt;In clustering the outcome variable or the response is unknown, this is why it's called clustering. Irrespective, of the fact the data being labeled or unlabelled, clustering can be applied as a data preprocessing algorithm.&#xA;Essentially, you must proceed by employing the initial data preprocessing tasks (like missing value treatment, collinearity, skewness etc). Once, the data is &quot;statistically clean&quot;, then you can apply any clustering technique. However, remember clustering requires data to be &quot;grouped&quot; such that data points within a group are related to each other and unrelated to other data points belonging to another group. This can be achieved only if you have a &quot;statistically clean&quot; data. The next important point to consider is, &quot;how to determine the possible number of clusters&quot;. Because any clustering algorithm will divide the data points into groups oblivious of the fact whether the groups exist or not. Therefore, you will have to prove mathematically/statistically the occurrence of groups in the dataset. In literature, there exist several methods like the &quot;Principal Component Analysis (PCA)&quot; or the &quot;elbow method&quot;.&#xA;Once, you have determined such groups, you can then label the groups and perform predictive analytics.&lt;/p&gt;&#xA;" OwnerUserId="11061" LastActivityDate="2018-03-05T13:05:55.110" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28644" PostTypeId="2" ParentId="28457" CreationDate="2018-03-05T13:07:35.690" Score="3" Body="&lt;p&gt;You are using the &lt;code&gt;sigmoid&lt;/code&gt; activation function on the output layer that squashes the values between 0 and 1, but you are setting the targets outside of that range. I suggest scaling your outputs between 0 and 1 for each output unit. By the way, using a library like &lt;code&gt;conx&lt;/code&gt; that is built on Keras would give you a warning about this very issue. For more, see: &lt;a href=&quot;http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html#A-Simple-Network&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html#A-Simple-Network&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47270" LastActivityDate="2018-03-05T13:07:35.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28645" PostTypeId="2" ParentId="28141" CreationDate="2018-03-05T13:20:34.563" Score="1" Body="&lt;p&gt;When you build your Keras model using the functional interface, you can also build additional models on any subset of the paths through the network by reusing the intermediary functions. Then you can train on just portions of the network (given that you have targets for the outputs). I haven't tried to train on sub-networks of a network, but I do use these intermediary models to propagate activations between internal layers. The Python package &lt;code&gt;conx&lt;/code&gt; that is built on top of Keras will build these intermediary models for you, by the way. For more, see: &lt;a href=&quot;http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html?highlight=propagate_from#Propagation-functions&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html?highlight=propagate_from#Propagation-functions&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47270" LastActivityDate="2018-03-05T13:20:34.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28646" PostTypeId="2" ParentId="28636" CreationDate="2018-03-05T13:24:52.573" Score="4" Body="&lt;p&gt;The most common way of processing images in python through numpy arrays. Since you have already loaded your image through nibabel, you need to get the data from the image object and then cast it as a numpy array.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import nibabel as nib&#xA;import numpy as np&#xA;&#xA;# Get nibabel image object&#xA;img = nib.load(&quot;path/to/image.nii&quot;)&#xA;&#xA;# Get data from nibabel image object (returns numpy memmap object)&#xA;img_data = img.get_data()&#xA;&#xA;# Convert to numpy ndarray (dtype: uint16)&#xA;img_data_arr = np.asarray(img_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By default this should be a 3D numpy array with a shape of (height,width,image). If you want to access one image you can always use PIL. For example to save the first image of the .nii file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from PIL import Image&#xA;img = Image.fromarray(img_data_arr_norm[:,:,0], 'L')&#xA;img.save(&quot;image.jpeg&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-03-05T13:24:52.573" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28647" PostTypeId="2" ParentId="28086" CreationDate="2018-03-05T13:26:55.797" Score="2" Body="&lt;p&gt;Using the raw integers as inputs and targets will make this a very difficult task. A better approach would be to come up with a vector for each number. You can simply encode each number directly as a vector, use a &quot;onehot&quot; representation, or use an Embedding layer. You can see an example of embeddings in &lt;code&gt;conx&lt;/code&gt; that is built on Keras here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://conx.readthedocs.io/en/latest/Predicting%20and%20Generating%20Texts.html?highlight=embeddinglayer&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/Predicting%20and%20Generating%20Texts.html?highlight=embeddinglayer&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;and and example without embeddings using onehot representations here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://conx.readthedocs.io/en/latest/AliceInWonderland.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conx.readthedocs.io/en/latest/AliceInWonderland.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47270" LastActivityDate="2018-03-05T13:26:55.797" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28650" PostTypeId="1" CreationDate="2018-03-05T16:05:31.800" Score="3" ViewCount="1386" Body="&lt;p&gt;This is more of a design question regarding linear regression. Here is some info on our dataset:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Our dataset has 8 features; 3 of them being categorical. We are willing to perform linear regression to fit our target data.&lt;/li&gt;&#xA;&lt;li&gt;We have tried including all of our 8 features (categorical ones being encoded in integer) and doing the linear regression.&lt;/li&gt;&#xA;&lt;li&gt;We have also tried taking out the categorical features and running the linear regression algorithm for each possible combination of our 3 categorical features. That of course yields in a lot of regression runs (each category has 4 possible values; so 64 to be exact).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The latter approach gave us better results. So, fixing the categorical features and creating a new dataset for each combination turned out to give better estimations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does this tell us about our data?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Categorical features make the data non-linear, so they should indeed be taken out? For example, there is a complex relationship between categorical feature A and categorical feature B that can't be captured in linear regression?&lt;/li&gt;&#xA;&lt;li&gt;If I am forced to step out of linear regression, which algorithm should I try to apply? I would prefer to have one dataset with 8 features instead of having 64 datasets with 5 features. There should be an algorithm that can capture this model.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47144" LastEditorUserId="40853" LastEditDate="2018-03-06T02:18:30.907" LastActivityDate="2018-03-08T16:41:07.137" Title="Removing Categorial Features in Linear Regression" Tags="|scikit-learn|linear-regression|feature-engineering|" AnswerCount="6" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28651" PostTypeId="2" ParentId="28542" CreationDate="2018-03-05T16:14:10.710" Score="2" Body="&lt;p&gt;First, I would like to emphasize, that cross-validation on itself does not give you any insights about overfitting. This would require comparing training and validation errors over the epochs. Typically you make such comparison with your eye and you can start with one train/validation split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt; By getting validation error N times, you develop a reasonable (whether it is very or not very good is a question) understanding of how your network will perform (= what error it will give) on the new unseen data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Often you do cross validation as a part of grid search of hyper-parameters. Then averaging errors at step 6 is mainly for choosing the best hyper-parameters: you believe that the hyper-parameters are best if the corresponding network produces the smallest average validation error. Simultaneously this error is your estimation on what error the final model will give you on the new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want, you can proceed with your exploration and compare validation errors (for one and the same hyper-parameter set) with each other, calculate standard deviation in order to get further insights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept &quot;model generalizes well&quot; is more related to the absence of overfitting. To make this conclusion, you need to compare train and validation errors. If they are close to each other, then the model generalizes well. This is not directly related to cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2:&lt;/strong&gt;  The only purpose to take the whole dataset is to train the model on more data. And more data is always good. On the other side, if you are happy with one of the models produced during cross-validation, you can take it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3:&lt;/strong&gt; You can take the number of epochs as one of the parameters in grid-search of hyper-parameters. This search usually goes with cross-validation inside. When at step 7 you take the whole data set, you take more data. Thus, overfitting, if at all, at this stage can only be reduced. If it bothers you, that each chunk is small, replace K-fold cross validation with, for example, K times 50/50 train/test splits. And I would never worry about leave-one-out. It was developed for small (very small) datasets. For Neural Net to be good, you typically need large or very large dataset.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastEditorUserId="9085" LastEditDate="2018-03-05T19:58:11.660" LastActivityDate="2018-03-05T19:58:11.660" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28652" PostTypeId="2" ParentId="19527" CreationDate="2018-03-05T16:27:59.510" Score="1" Body="&lt;p&gt;Time for a more general answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your approach should be pragmatic, based on your objectives and time scales. Are you trying to find nuanced links between predictors and response, are you hoping for more of a rough idea of &quot;important&quot; variables? Do you have the time and resources to prepare your data and run more complicated algorithms / processes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something to bear in mind is that there are often many ways to achieve the same goal. You may find that &lt;strong&gt;for your particular data and task&lt;/strong&gt;, simple (&quot;traditional&quot;) tests may perform as well as more complicated techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, perhaps consider something along the lines of the following - in increasing order of complexity:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Chi-square / Cramer's V for categorical predictors&lt;/li&gt;&#xA;&lt;li&gt;ANOVA test for continuous predictors&lt;/li&gt;&#xA;&lt;li&gt;Forward selection (GLM technique)&lt;/li&gt;&#xA;&lt;li&gt;Decision trees / bagged trees / random forest importance.&lt;/li&gt;&#xA;&lt;li&gt;Boosted trees&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you're looking for R packages to help you with feature selection, have a look at &lt;code&gt;caret&lt;/code&gt; and &lt;code&gt;FSelector&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-03-05T16:27:59.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28654" PostTypeId="2" ParentId="28650" CreationDate="2018-03-05T17:05:59.440" Score="1" Body="&lt;p&gt;Sounds like you have a lot of complex categorical variables in your model. Here's what I would do to see which ones are significant and which ones are not. For each of the 4 categorical variables, you will only need 3 binary variables to represent the options. If all 3 binary options are 0, then the fourth category is 1, so it simplifies the model a little. Here's what I would do:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Run a regression model for each categorical variable using the binary variables. You'll have 4 models in total.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Run these models with &lt;a href=&quot;https://gerardnico.com/data_mining/stepwise_regression#backward&quot; rel=&quot;nofollow noreferrer&quot;&gt;backwards stepwise regression&lt;/a&gt;. You should analyze these four models to look for similarities or patterns, maybe something will jump out at you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) After all four models are run using this selection method, run a final regression using backwards stepwise selection using only the significant variables from the previous 4 runs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will leave you with a final model (and results) without cluttering the regression with all 64 iterations of the categorical variables. If the significant variables from this are too cumbersome, maybe only discuss or highlight the most significant independent variables or trim it down some other way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck, let us know how it goes!&lt;/p&gt;&#xA;" OwnerUserId="47282" LastActivityDate="2018-03-05T17:05:59.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28656" PostTypeId="1" AcceptedAnswerId="28657" CreationDate="2018-03-05T17:48:23.020" Score="2" ViewCount="1169" Body="&lt;p&gt;I'm performing some simulations, and at the end I get a CSV file with three columns. One column holds the values for the x-axis, which was also input to the simulation and theoretical calculations, second one holds theoretically expected values, and the other column holds the values obtained by the simulation. I was planning to plot something like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/N7jic.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N7jic.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that does not look good in my case, as the values in y-axis normally keep doubling, and the values for the x-axis exponentially increase, so most of the points end up getting collected at the lower left part, near the intersection of x-axis and y-axis of the plot. Therefore, I need a different way to plot my data, which will be more visually appealing and inform how close the simulation results are to the theoretical expected ones. For example, some of my values can be seen below (and they keep increasing in such a way):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x         = [2, 4, 8, 16, 32, 64] # partially removed for brevity&#xA;expected  = [47.9995, 95.9783, 191.9127, 383.9708, 767.8831] # partially removed for brevity &#xA;simulated = [48, 96, 191.8, 383.8, 767.4] # partially removed for brevity&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is a good way to plot such a data that doubles in the y-axis and exponentially increases on the x-axis all the time, and to view how similar the two datasets actually are?&lt;/p&gt;&#xA;" OwnerUserId="47288" LastActivityDate="2018-03-26T05:39:40.670" Title="How to plot similarity of two datasets?" Tags="|plotting|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28657" PostTypeId="2" ParentId="28656" CreationDate="2018-03-05T18:31:32.587" Score="2" Body="&lt;p&gt;Here is an example of &lt;code&gt;r&lt;/code&gt; lattice &lt;code&gt;xyplot&lt;/code&gt; using log scale on the x axis and the difference of your two measures &lt;code&gt;I(expected - simulated)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df &amp;lt;- data.frame(&#xA;x         = c(2, 4, 8, 16, 32, 64),&#xA;expected  = c(47.9995, 95.9783, &#xA;191.9127, 383.9708, 767.8831, &#xA;1457.2771),&#xA;simulated = c(48, 96, 191.8, 383.8, &#xA;767.4, 1458.1228))&#xA;xy &amp;lt;- xyplot(I(expected - simulated) ~ x ,   &#xA;auto.key=TRUE,&#xA;data =  df ,    type=c(&quot;p&quot;,&quot;g&quot;),&#xA;scales=list(x=list(log = 10) ),   &#xA;ylab=&quot;difference expected - simulated&quot;, xlab=&quot;x&quot;, main=&quot;Simulation Results&quot; )&#xA;print (xy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note, that I added a 6th result to your sample data,that was missing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/shHfC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/shHfC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10620" LastEditorUserId="35644" LastEditDate="2018-03-26T05:39:40.670" LastActivityDate="2018-03-26T05:39:40.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28658" PostTypeId="2" ParentId="28454" CreationDate="2018-03-05T18:31:53.967" Score="2" Body="&lt;p&gt;(1) &lt;strong&gt;No&lt;/strong&gt;. For example, if you have computer vision problems, then each pixel of the image is a feature. most neighboring pixels are highly correlated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, see the following image from &lt;a href=&quot;https://arxiv.org/pdf/1701.08380.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The HASYv2 dataset&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rdn19.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rdn19.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-03-05T18:31:53.967" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28659" PostTypeId="2" ParentId="28656" CreationDate="2018-03-05T18:50:06.583" Score="2" Body="&lt;p&gt;Here we have &lt;code&gt;50000&lt;/code&gt; points, &lt;code&gt;10000&lt;/code&gt; in each of &lt;code&gt;five&lt;/code&gt; categories with associated numerical values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of using Logarithms, you can also use &#xA;&lt;code&gt;O( log* N )&lt;/code&gt; is &quot;&lt;a href=&quot;http://en.wikipedia.org/wiki/Iterated_logarithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;iterated logarithm&lt;/a&gt;&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In computer science, the iterated logarithm of n, written log* n (usually read &quot;log star&quot;), is the number of times the logarithm function must be iteratively applied before the result is less than or equal to 1.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Checkout &lt;strong&gt;Datashader&lt;/strong&gt; (This is what you Need) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://nbviewer.jupyter.org/github/bokeh/bokeh-notebooks/blob/master/tutorial/A2%20-%20Visualizing%20Big%20Data%20with%20Datashader.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference Notebook&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Generating Something Random(you will get the idea)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;np.random.seed(1)&#xA;num=10000&#xA;&#xA;dists = {cat: pd.DataFrame(dict(x=np.random.normal(x,s,num),&#xA;                                y=np.random.normal(y,s,num),&#xA;                                val=val,cat=cat))&#xA;         for x,y,s,val,cat in &#xA;         [(2,2,0.01,10,&quot;d1&quot;), (2,-2,0.1,20,&quot;d2&quot;), (-2,-2,0.5,30,&quot;d3&quot;), (-2,2,1.0,40,&quot;d4&quot;), (0,0,3,50,&quot;d5&quot;)]}&#xA;&#xA;df = pd.concat(dists,ignore_index=True)&#xA;df[&quot;cat&quot;]=df[&quot;cat&quot;].astype(&quot;category&quot;)&#xA;df.tail()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&#xA;        cat val     x           y&#xA;49995   d5  50  -1.397579   0.610189&#xA;49996   d5  50  -2.649610   3.080821&#xA;49997   d5  50  1.933360    0.243676&#xA;49998   d5  50  4.306374    1.032139&#xA;49999   d5  50  -0.493567   -2.242669&#xA;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%time tf.shade(ds.Canvas().points(df,'x','y'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Output Image&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ebV8H.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ebV8H.png&quot; alt=&quot;The Picture Clearly Shows 5 Normal Distributions&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-05T18:55:14.653" LastActivityDate="2018-03-05T18:55:14.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28661" PostTypeId="1" CreationDate="2018-03-05T19:45:35.897" Score="2" ViewCount="469" Body="&lt;p&gt;I am putting together a multi-category classification algorithm. Since it's NLP, the training data is very simple with one column for labels and another column for text. However, because it's NLP, some training records can fit multiple categories. Should I iterate through &amp;amp; have multiple entries (same data, different labels) or should each piece of data only have 1 category assigned? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-03-08T08:47:24.493" Title="Training data for multi-category classification algorithm" Tags="|neural-network|nlp|multiclass-classification|lstm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28662" PostTypeId="2" ParentId="28639" CreationDate="2018-03-05T19:47:26.457" Score="2" Body="&lt;p&gt;Clustering is primarily useful for &lt;em&gt;you&lt;/em&gt; to understand your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It probably does not help much to automate that and try to use it for prediction. You can try, but I don't think it will even improve results much to use cluster information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two things that you need to consider:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Clustering is very difficult, expensive, and hard to parameterize. That means it won't be easy to automate in a useful way - it breaks all the time when data changes.&lt;/li&gt;&#xA;&lt;li&gt;Most good clustering (not kmeans) cannot predict the cluster label on new data. So using it will mean you first need to train a classifier to predict the cluster, then use this in another classifier to predict your class.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-05T19:47:26.457" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28664" PostTypeId="1" CreationDate="2018-03-05T20:16:04.390" Score="5" ViewCount="1860" Body="&lt;p&gt;We noticed we had a biased sample in our A/B test and was wondering if difference-in-differences would help us make valid conclusions about the data, or if there was another way to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We ran an new experiment on our site, where we offered 50% of our users a new feature. We assigned users with odd ids into the experiment group and users with even ids into the control group and then ran the experiment. However, we saw that even prior to running the experiment, there was a statistically difference between the two groups. We think this is because we ran many experiments where we segmented based on odd/even of the id, so people in the experiment group have seen many treatments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So lesson learned, we'll flip a coin next time instead of using the id. However, we'd like to see if we can still make inferences from the current experiment. I've heard of something called difference-in-differences. Would this work in this case, or is there a different approach that would work better? Ideally, we don't want to scrap the test and start over since 50% of our users have already seen the new feature.&lt;/p&gt;&#xA;" OwnerUserId="21613" LastActivityDate="2022-03-10T18:43:48.670" Title="Can we use difference-in-differences with a biased A/B test?" Tags="|experiments|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28665" PostTypeId="2" ParentId="16786" CreationDate="2018-03-05T20:24:17.373" Score="1" Body="&lt;p&gt;This is a great place to start! While not catalogued in a &quot;process flow&quot;, Daniel Jurafsky's book, &lt;a href=&quot;https://web.stanford.edu/~jurafsky/slp3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&quot;Speech and Language Processing&quot;&lt;/a&gt; talks through the various calculations and steps related to analyzing text that you will find useful. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I say that a process flow is not provided is because Jurafsky - in great detail - explains the pros and cons of particular methods applied throughout a pipeline, and how this could change results. As an example, when calculating perplexity (an inverse metric that quantifies how well a language model can predict the next word in a statement), you should capture beginnings, ends, and stop words of statements - as opposed to other methods that require the removal of stop words.&lt;/p&gt;&#xA;" OwnerUserId="8251" LastEditorUserId="29575" LastEditDate="2018-03-05T20:46:17.433" LastActivityDate="2018-03-05T20:46:17.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28666" PostTypeId="2" ParentId="28650" CreationDate="2018-03-05T20:31:25.050" Score="1" Body="&lt;p&gt;Encoding categorical variables as integers is generally bad for linear regression, because the model will interpret that to mean that category 2 is twice as significant as category 1, and so on, which is not necessarily true. It isn't surprising that you got bad results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A better approach is to encode your categories with dummy variables. Let's say your categorical variables are C1, C2, and C3, each taking values from 1 to 4. Then we can have twelve 0/1 dummy variables corresponding to each possible category for each categorical variable. For any input exactly four dummy variables will be 1 and the rest will be zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your linear regression now looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat{y}=a_1*d_{C11}+a_2*d_{C12}+a_3*d_{C13}+a_4*d_{C14} + a_5*d_{C21}+a_6*d_{C22}+a_7*d_{C23}+a_8*d_{C24} +&#xA;a_9*d_{C31}+a_{10}*d_{C32}+a_{11}*d_{C33}+a_{12}*d_{C34} +&#xA;a_{13}*x_1+a_{14}*x_2+a_{15}*x_3+a_{16}*x_4+a_{17}*x_5&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x_1$ through $x_5$ are your numerical inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a given input has C1=1, C2=4, and C3=3, for example, then this would reduce to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{y}=a_1*1+a_8*1+a_{11}*1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also possible to do the same thing with 64 dummy variables for each possible combination of the categorical variables as you were doing, but in a single linear regression as above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the Wikipedia entry on &lt;a href=&quot;https://en.wikipedia.org/wiki/Dummy_variable_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;dummy variables&lt;/a&gt; for more info.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are still not getting good results with linear regression, then consider using Gradient Boosting Regression Trees.&lt;/p&gt;&#xA;" OwnerUserId="15828" LastActivityDate="2018-03-05T20:31:25.050" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28667" PostTypeId="2" ParentId="28664" CreationDate="2018-03-05T20:36:02.230" Score="0" Body="&lt;p&gt;At this point in the experiment, nothing you can do can make the two groups equal.  What you can do is make assumptions about how the two groups will behave.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;If&lt;/em&gt; you can safely assume that previous treatments that you have applied using the even/odd sampling method will not interact with the new A/B test, then a difference-in-differences method would be appropriate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple example of when this assumption (and a difference-in-differences) is &lt;em&gt;inappropriate&lt;/em&gt;:&#xA;Let's say in your first test you send group A (the odds) a book titled &quot;The importance of eating healthy.&quot;  In the second test, you send group A (the odds) dietary supplements.  Group A has been compromised by prior treatments that could impact the effects on the second test.&lt;/p&gt;&#xA;" OwnerUserId="46949" LastEditorUserId="46949" LastEditDate="2018-03-06T14:15:25.883" LastActivityDate="2018-03-06T14:15:25.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28669" PostTypeId="1" AcceptedAnswerId="28673" CreationDate="2018-03-05T21:36:08.097" Score="0" ViewCount="44" Body="&lt;p&gt;I need to find patterns experimental data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns are &quot;experiments&quot; which are chemical treatments for growth experiments.  The rows are individual gene names, the values are a fitness-defect score, which reflect the genes contribution to growth. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find patterns that are reflected across all experiments using some type of PCA or clustering.  I have been trying to use sklearn but have not been successful in applying a model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;gene     SGTC_1                   SGTC_2                 SGTC_3 &#xA;YAL002W  3.56420220283773        1.80774301690328       0.431491057210906&#xA;YAL004W -0.885645399324204      -1.76020417788351       0.883034190306176&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 4000 rows for genes and 30 columns for experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="47299" LastActivityDate="2018-03-05T22:34:52.813" Title="PCA or cluster table of experimental fitness scores" Tags="|scikit-learn|clustering|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28670" PostTypeId="2" ParentId="28650" CreationDate="2018-03-05T22:00:20.947" Score="1" Body="&lt;p&gt;What I would do to optimise the performance of linear regression&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One Hot encode the categorical features&lt;/li&gt;&#xA;&lt;li&gt;Use PCA to reduce the dimensionality of the data&lt;/li&gt;&#xA;&lt;li&gt;Scale the data (subtract the mean, divide by the standard deviation)&lt;/li&gt;&#xA;&lt;li&gt;Train the regression model on the reduced scaled dataset&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If you have enough data (&gt;10k examples), you could even train a neural network on the data to capture the complex relationships between features which linear regression wouldn’t capture.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-03-05T22:00:20.947" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28671" PostTypeId="2" ParentId="28613" CreationDate="2018-03-05T22:05:35.720" Score="0" Body="&lt;p&gt;Choosing the learning algorithm depends on the problem type: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Linear regression&lt;/em&gt; is typically used for &lt;em&gt;regression problems&lt;/em&gt; (i.e., predicting results within a continuous output)&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Logistic regression&lt;/em&gt; is typically used for &lt;em&gt;classification problems&lt;/em&gt; (i.e., predicting results in a discrete output)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In our scenario, we want to predict if patients with certain causes (features) are highly likely to be diagnosed with infertility (&quot;yes&quot;), are often diagnosed with infertility (&quot;often&quot;), are sometimes diagnosed wit infertility (&quot;sometimes&quot;), or are highly likely to be &lt;em&gt;not&lt;/em&gt; diagnosed with infertility (&quot;no&quot;). So, as a result, we can have one of the four discrete outputs: &quot;yes&quot;,  &quot;often&quot;, &quot;sometimes&quot;, &quot;no&quot;. That is, we have a classification problem and therefore we should choose logistic regression over linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, logistic regression does binary classification (two classes) and we have four classes. Still, we can use logistic regression by learning four different models:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predicts &quot;yes&quot; or &quot;the rest&quot; (&quot;the rest&quot; includes &quot;often&quot;, &quot;sometimes&quot;, &quot;no&quot;)&lt;/li&gt;&#xA;&lt;li&gt;Predicts &quot;often&quot; or &quot;the rest&quot; (&quot;the rest&quot; includes &quot;yes&quot;, &quot;sometimes&quot;, &quot;no&quot;)&lt;/li&gt;&#xA;&lt;li&gt;Predicts &quot;sometimes&quot; or &quot;the rest&quot;&lt;/li&gt;&#xA;&lt;li&gt;Predicts &quot;no&quot; or &quot;the rest&quot;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For a given patient, all four models are then evaluated. Ideally, three of the models will predict &quot;the rest&quot; and one will predict the actual class for the patient. This strategy is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-vs.-rest transformation&lt;/a&gt;. There is also &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-one&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-vs.-one transformation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, we can use one of the learning algorithms that natively support multiclass classification and do not require a transformation: decision trees, support vector machines, neural networks, multinomial logistic regression, etc.&lt;/p&gt;&#xA;" OwnerUserId="42142" LastEditorUserId="42142" LastEditDate="2018-03-05T23:49:35.820" LastActivityDate="2018-03-05T23:49:35.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28672" PostTypeId="1" CreationDate="2018-03-05T22:28:57.380" Score="0" ViewCount="400" Body="&lt;p&gt;I read a lot about random forest and gradient boosting, but I do not know how these two algorithms really work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, see the simple picture about basketball (picture 1) from &lt;a href=&quot;https://www.r-bloggers.com/how-random-forests-improve-simple-regression-trees/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this link&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does Random Forest and how does Gradient Boosting work?&#xA;Has each tree in the random forest different trainings data AND different features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry about m question but I don‘t find an easy non-mathematical answer.&lt;/p&gt;&#xA;" OwnerUserId="20903" LastEditorUserId="20903" LastEditDate="2018-03-05T22:39:21.790" LastActivityDate="2018-05-07T17:46:03.090" Title="Regression - How Random Forest and Gradient Boosting really works?" Tags="|python|regression|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28673" PostTypeId="2" ParentId="28669" CreationDate="2018-03-05T22:34:52.813" Score="0" Body="&lt;p&gt;PCA is a dimensionality reduction algorithm - it projects your high dimensional data onto a lower dimensional plane. This is useful for either visualisation (if you reduce to 2 or 3 dimensions to plot), or for training machine learning models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You say you want to find patterns in your data - I’m not quite sure what you mean by this. Do you want to visualise your data, train a model on it and make some prediction, or something else?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To visualise high dimensional data, you could use either the tSNE (t-stochastic neighbour embedding) algorithm, or PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on the type of data you have, you can “find patterns” in different ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data is unlabelled (you don’t know the classes of each sample or there is no dependent variable) you can use unsupervised learning algorithms such as K-means clustering, K nearest neighbours, Gaussian mixture model. If your data has dependent variables, depending on whether your dependent variable is categorical or continuous, you could use classification algorithms for the former and regression algorithms for the latter. Classification algorithms include logistic regression or decision trees. Regression models include linear regression.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-03-05T22:34:52.813" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28675" PostTypeId="1" AcceptedAnswerId="28708" CreationDate="2018-03-06T01:26:21.337" Score="1" ViewCount="1809" Body="&lt;p&gt;I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the samples in class C is actually belong to class A and class B. For now, I need to run some supervised learning (logistic regression, decision tree and random forests) models. According to the confusion matrix, the classification between A and B is relatively accurate, but the classification between C and other two classes is not acceptable.&#xA;I wondered whether is any way to deal with this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I'm considering to use a clustering algorithm for the samples in class C before runnning the model. After dividing the samples in class C into 3 groups, try to find a relatively better group as the datasets for class C.&lt;/p&gt;&#xA;" OwnerUserId="32002" LastActivityDate="2018-03-06T14:28:08.290" Title="How to treat the datasets with unreliable labels" Tags="|classification|data-cleaning|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28676" PostTypeId="1" AcceptedAnswerId="30084" CreationDate="2018-03-06T01:49:00.123" Score="10" ViewCount="10942" Body="&lt;p&gt;Can you build complex (in terms of complex numbers) neural networks in Keras or Tensorflow or something similar?&lt;/p&gt;&#xA;&lt;p&gt;This would mean the inputs, weights, activation functions, and outputs would all potentially use complex numbers. I know this can be done in theory, but all the software packages I have come across seem to assume only real numbers are used for the weights and inputs.&lt;/p&gt;&#xA;" OwnerUserId="47309" LastEditorUserId="47309" LastEditDate="2022-08-27T18:44:01.550" LastActivityDate="2022-08-27T18:44:01.550" Title="Are there neural networks packages that use complex numbers?" Tags="|python|neural-network|keras|software-recommendation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="28677" PostTypeId="2" ParentId="28650" CreationDate="2018-03-06T02:07:47.320" Score="2" Body="&lt;p&gt;I think using Linear Regression is not a good option as, &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;This performs very well on numeric variables(categorical -&gt; binary).&lt;/li&gt;&#xA;&lt;li&gt;Cannot handle Missing Data(suggestible to ignore those records).&lt;/li&gt;&#xA;&lt;li&gt;When you are trying to predict, By chance a category is not trained then it cannot predict. &#xA;For example: A variable has 4 categories and in the random sample it&#xA;could pick only 3 categories, if the test has 4th category then it&#xA;will throw an error. So, we should be careful when the data is&#xA;divided between Test and Train&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now, what are the other algorithms which are available:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Random Forest(you cannot use this when any category variable has more than 52 categories, in your case it shouldn't be an issue)&lt;/li&gt;&#xA;&lt;li&gt;XGBoost&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Do let me know if you need any additional explanations. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-03-06T02:07:47.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28678" PostTypeId="2" ParentId="26222" CreationDate="2018-03-06T02:40:57.800" Score="2" Body="&lt;p&gt;In my experience, the example code for a low number of classes (&amp;lt;200) works well. When moving to more classes the imbalance data makes the network converge to the same numbers. You have imbalance data because now each output is a binary classifier by its own, this doesn't happen with softmax. The way to mitigate the problem is to use &lt;code&gt;weighted_cross_entropy_with_logits&lt;/code&gt; and set &lt;code&gt;pos_weight&lt;/code&gt; to a positive number &gt; 1 (10 works). But I still don't get very good results.&lt;/p&gt;&#xA;" OwnerUserId="20933" LastActivityDate="2018-03-06T02:40:57.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28680" PostTypeId="2" ParentId="18751" CreationDate="2018-03-06T02:55:26.243" Score="8" Body="&lt;p&gt;You can get the answer &lt;a href=&quot;https://keras.rstudio.com/articles/faq.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Here it is:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- ...  # create the original model&#xA;layer_name &amp;lt;- 'my_layer'&#xA;intermediate_layer_model &amp;lt;- keras_model(inputs = model&lt;span class=&quot;math-container&quot;&gt;$input,&#xA;                                    outputs = get_layer(model, layer_name)$&lt;/span&gt;output)&#xA;intermediate_output &amp;lt;- predict(intermediate_layer_model, data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47308" LastEditorUserId="98307" LastEditDate="2020-08-09T09:54:35.743" LastActivityDate="2020-08-09T09:54:35.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28681" PostTypeId="1" AcceptedAnswerId="28684" CreationDate="2018-03-06T03:02:27.333" Score="5" ViewCount="6085" Body="&lt;p&gt;I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, then the histogram of this variable in train and test data set looks pretty much the same. See the following figure.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/bwnkx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bwnkx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;strong&gt;I still need to use the training data (with both features and labels) to make predictions on the test data (with features only). In this case, how should I deal with this feature variable?&lt;/strong&gt; I was thinking about removing the top 1% data in both training and test data set, but as I still need to make predictions on that 1% test data, so this is not a good idea I guess. In this example, as the empirical distribution of this variable in both training and test data sets look the same before and after removing the &quot;outlier&quot;, should we just leave this variable unchanged? Also, &lt;strong&gt;in general, how should we handle the outlier before we put the feature into the machine learning algorithm?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="30170" LastEditorUserId="81999" LastEditDate="2020-02-19T04:50:09.690" LastActivityDate="2020-02-19T04:50:09.690" Title="When to remove outlier in preparing features for machine learning algorithm" Tags="|data-cleaning|outlier|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28684" PostTypeId="2" ParentId="28681" CreationDate="2018-03-06T03:56:32.800" Score="3" Body="&lt;p&gt;Dealing with outliers requires knowledge about the outlier, the dataset and possibly domain knowledge. Given this, there are many options to handle outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without taking a look at your specific data, it could be that this outlier represents a total? Perhaps the data source you have included totals, which should be removed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, figuring out what to do with outliers requires investigating the outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the outlier is a data processing or entry error, it can generally be removed, or replaced with say, the mean (without the outlier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If errors have been ruled out think about whether the outlier could be a legitimate value? There are no Right or Wrong answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, documentation of your decisions is really important. I'd suggest running your models with the outlier, and without, comparing the results and document any decision to remove or transform the outlier. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on your purpose, it should become relatively clear as to whether the outlier should be removed, eg if you're visualising the data, removing the outlier will likely have more a meaningful impact than retaining it.&lt;/p&gt;&#xA;" OwnerUserId="47311" LastEditorUserId="47311" LastEditDate="2018-03-07T00:01:01.457" LastActivityDate="2018-03-07T00:01:01.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28689" PostTypeId="1" CreationDate="2018-03-06T07:04:19.473" Score="1" ViewCount="141" Body="&lt;p&gt;We are facing issue in our project. We have a data set of around 25000 rows, we have a column name title, it contains text data and we have a score column in the data set.We want to use Machine Learning techniques to know what are the factors that make the label High, by factor i mean what are the key words etc.., as use can see we have a short text so what is the most proper technique to extract the maximum knowledge related to the label from this short text,I did some pre processing on the text by cleaning it then extracting ngram features using tf-idf weighting function, then i tried using decision trees algorithm for classification of text &#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Score Label                       Tactic Title&#xA;1.  High                        Opportunity Movement.   &#xA;2.  Low                         Partner Launch.  &#xA;3.  High                        Implement Mix panel stories.&#xA;4.  Low                         Improve app performance and reduce multiple API Calls.&#xA;5.  High                        Review Comments.&#xA;6.  Low                         Support SimpleStarta Team Goal. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30427" LastEditorUserId="30427" LastEditDate="2018-03-06T11:03:16.187" LastActivityDate="2018-03-06T13:03:20.330" Title="Extract Pattern using Short Text Processing" Tags="|machine-learning|classification|clustering|text-mining|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28690" PostTypeId="1" CreationDate="2018-03-06T08:15:57.640" Score="1" ViewCount="403" Body="&lt;p&gt;So far I was using R on my home pc: i3 CPU, two cores, 4 threads. In order to run the code faster I was using the package &quot;DoSnow&quot;, utilizing 3 out of the 4 cores in order not to choke my system completely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, now I got a server which has a two single core Xenons and I wonder: can I use the same multi core code for multi cpus? If no, how can I run them together and at about 70% (again, in order to not choke the system)? So far, all the codes I found on the net refer to cores rather then cpus&lt;/p&gt;&#xA;" OwnerUserId="33861" LastEditorUserId="33861" LastEditDate="2018-03-07T18:24:16.067" LastActivityDate="2018-03-07T18:24:16.067" Title="R studio, one multi core CPU vs dual single cores" Tags="|r|parallel|rstudio|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28691" PostTypeId="1" CreationDate="2018-03-06T08:36:48.220" Score="0" ViewCount="1976" Body="&lt;p&gt;I'm on the first chapter of &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nielsen's book on neural networks&lt;/a&gt;. He gives an example of a gradient descent and that change in function &lt;span class=&quot;math-container&quot;&gt;$C(v_1,v_2)$&lt;/span&gt; is described by this formula: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\begin{eqnarray}\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 +\frac{\partial C}{\partial v_2} \Delta v_2\tag{7}\end{eqnarray}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Though I'm familiar with partial derivatives, i'm confused about how you would compute partial derivatives of this function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone give an example please?  &lt;/p&gt;&#xA;" OwnerDisplayName="user45691" LastEditorUserId="29169" LastEditDate="2019-06-09T15:22:51.890" LastActivityDate="2019-06-09T15:22:51.890" Title="Gradient descent and partial derivatives" Tags="|neural-network|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28692" PostTypeId="1" CreationDate="2018-03-06T08:55:20.673" Score="0" ViewCount="2966" Body="&lt;p&gt;I've been reading about one-hot encoding for categorical values. Could similar mechanism make sense for continuous values? I'm looking at a problem, where I try to predict the volume of product X, with a variables being the price of product Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason is to increase the strength of the signal from the price of Y, which otherwise could be understated because overall prices fluctuate in relatively low range. The idea would be to e.g. remove attribute of &quot;Price Y&quot; and replace it with &quot;Promotion Y&quot; based on a set threshold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Price X | Price Y | Vol X | Vol Y  | Promotion X  | Promotion Y&#xA;100     | 90      | 40    | 50     | 0            |  0&#xA;75      | 75      | 60    | 60     | 1            |  1&#xA;100     | 75      | 30    | 70     | 0            |  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I could just test it out on a dataset, but this could be very specific to this very set and I just want some more theoretical suggestion if and where such approaches would make sense?&lt;/p&gt;&#xA;" OwnerUserId="47322" LastActivityDate="2020-09-08T07:25:39.217" Title="Encoding continuous values" Tags="|machine-learning|machine-learning-model|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28696" PostTypeId="2" ParentId="28604" CreationDate="2018-03-06T10:57:56.340" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The reward function does not choose the action. It gives the immediate &lt;em&gt;consequences&lt;/em&gt; for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only &lt;em&gt;depend&lt;/em&gt; on most recent state, previous action plus a random factor at most. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case reward depends directly on the resulting state = how many cars at each junction after taking an action, and not &lt;em&gt;directly&lt;/em&gt; on the previous action = which traffic flow to enable. It does still depend on the action indirectly, via however the environment works. But that does not need to be expressed mathematically (as a model of the system) for Q learning to work. Although not stated, the reward probably also depends on a random factor outside of your control = how many cars turn up at each junction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using &lt;a href=&quot;http://banditalgs.com/2016/09/18/the-upper-confidence-bound-algorithm/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Upper Confidence Bound&lt;/a&gt; (or UCB), which is a relatively sophisticated way of balancing exploration of actions that &lt;em&gt;might&lt;/em&gt; be better than your current best estimate, versus simply using the &quot;best&quot; action so far. If you always took the &quot;best&quot; action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the standard textbooks for all this is &lt;a href=&quot;http://incompleteideas.net/book/the-book-2nd.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sutton &amp;amp; Barto, &lt;em&gt;Reinforcement Learning: An Introduction&lt;/em&gt;&lt;/a&gt;. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem in easy-to-understand detail.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-06T12:25:28.123" LastActivityDate="2018-03-06T12:25:28.123" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28698" PostTypeId="2" ParentId="28594" CreationDate="2018-03-06T11:40:29.810" Score="4" Body="&lt;p&gt;You can try &lt;a href=&quot;https://prodi.gy/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prodigy&lt;/a&gt; by explosion.ai, creators of spacy or &lt;a href=&quot;http://brat.nlplab.org&quot; rel=&quot;nofollow noreferrer&quot;&gt;brat&lt;/a&gt; an open source alternative to it. &#xA;You may also refer to &lt;a href=&quot;https://www.quora.com/What-are-the-best-tools-for-manually-annotating-a-text-corpus-with-entities-and-relationships&quot; rel=&quot;nofollow noreferrer&quot;&gt;this post&lt;/a&gt; on qoura.&lt;/p&gt;&#xA;" OwnerUserId="47318" LastActivityDate="2018-03-06T11:40:29.810" CommentCount="1" CommunityOwnedDate="2019-01-31T00:05:48.343" ContentLicense="CC BY-SA 3.0" />
  <row Id="28699" PostTypeId="2" ParentId="28691" CreationDate="2018-03-06T11:45:20.367" Score="7" Body="&lt;h1&gt;What's the point?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;First, it is good to understand what we are doing that leads us to need these tools. When we are trying to apply machine learning we want to infer some meaning from data. This means that given an instance we want to put it through our model and then we can have some output that tell us something about this data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the square of the difference between the predicted label and the ground truth label for the $N$ images that we trained over. We will want to minimize this in some way. The cost is also commnly called the loss function.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;So we have a minimization problem on our hands!&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is &quot;how can we minimize a function&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's minimize the following function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = x^2-4x+6$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we plot this we can see that there is a minimum at $x = 2$. To do this analytically we can take the derivative of this function as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dy}{dx} = 2x - 4 = 0$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x = 2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is &lt;strong&gt;gradient descent&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How does gradient descent work?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take our function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = x^2-4x+6$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we will determine the $x$ for which $y$ is minimized. Gradient descent algorithm first says we will pick a random value for $x$. Let us initialize at $x=8$. Then the algorithm will do the following iteratively until we reach convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x^{new} = x^{old} - \nu \frac{dy}{dx}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\nu$ is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too large we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dy}{dx} = 2x - 4$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nu = 0.1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Iteration 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $ &lt;br/&gt;&#xA;$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $ &lt;br/&gt;&#xA;$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $ &lt;br/&gt;&#xA;$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $ &lt;br/&gt;&#xA;$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $ &lt;br/&gt;&#xA;$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $ &lt;br/&gt;&#xA;$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $ &lt;br/&gt;&#xA;$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $ &lt;br/&gt;&#xA;$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $ &lt;br/&gt;&#xA;$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $ &lt;br/&gt;&#xA;$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $ &lt;br/&gt;&#xA;$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $ &lt;br/&gt;&#xA;$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $ &lt;br/&gt;&#xA;$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $ &lt;br/&gt;&#xA;$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $ &lt;br/&gt;&#xA;$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $ &lt;br/&gt;&#xA;$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $ &lt;br/&gt;&#xA;$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $ &lt;br/&gt;&#xA;$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $ &lt;br/&gt;&#xA;$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $ &lt;br/&gt;&#xA;$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $ &lt;br/&gt;&#xA;$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $ &lt;br/&gt;&#xA;$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $ &lt;br/&gt;&#xA;$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $ &lt;br/&gt;&#xA;$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $ &lt;br/&gt;&#xA;$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $ &lt;br/&gt;&#xA;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $ &lt;br/&gt;&#xA;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $ &lt;br/&gt;&#xA;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $ &lt;br/&gt;&#xA;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $ &lt;br/&gt;&#xA;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $ &lt;br/&gt;&#xA;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $ &lt;br/&gt;&#xA;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $ &lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we see that the algorithm converges at $x = 2$!&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Where do partial derivatives come into this?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;We just minimized $y$ for $x$. But if we had a multi-dimensional $x$ then we will need to identify where is the steepest slope in each dimension. To do this we will take the derivative across each dimension. In 2 dimensions gradient descent is performed using the gradient of the function $\nabla y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = 2x_1^2 + 5x_1 - 4x_2^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we will take the gradient in both dimensions defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x^{new} = x^{old} - \nu \nabla y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The operator $\nabla$ is called the gradient it is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nabla y = \frac{\partial y}{\partial x_1} \hat{\textbf{x}_1} + \frac{\partial y}{\partial x_2} \hat{\textbf{x}_2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{\text{x}_1}$ and $\hat{\text{x}_2}$ are the two dimensions that we are optimizing over. So in our analogy we will go down this 2D surface a bit in the $x_1$ dimension and the $x_2$ dimension. Kind of like going down the mountain in the North-West direction. A bit in front and a bit to our left. That's a 2D direction!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The total size of the step that we take across all dimensions is then described by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Delta y = \frac{\partial y}{\partial x_1} \Delta x_1 + \frac{\partial y}{\partial x_2} \Delta x_2$.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Relating all this to your question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Change our function $y(x_1, x_2)$ by $C(v_1, v_2)$. The size of the step we are taking is defined by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Delta C = \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At each gradient descent iteration we will update $v$ as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v^{new} = v^{old} - \nu \nabla C$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nabla C = \frac{\partial C}{\partial v_1} \hat{\textbf{v}_1} + \frac{\partial C}{\partial v_2} \hat{\textbf{v}_2} = (\frac{\partial C}{\partial v_1}, \frac{\partial C}{\partial v_2})^T$&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Supplemental&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;2D gradient descent of a function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = x_1^2 - x_1 + x_2^2 - 3x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The partial derivatives are&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial y}{\partial x_1} = 2x_1 - 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial y}{\partial x_2} = 2x_2 - 3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nabla y = (2x_1 - 1, 2x_2 - 3)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our update rule with a learning rate $\nu = 0.1$ will then be&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.0 \\ &#xA;0.0 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.0 - 1 \\ &#xA;2 * 0.0 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.1 \\ &#xA;0.3 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.1 \\ &#xA;0.3 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.1 - 1 \\ &#xA;2 * 0.3 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.18 \\ &#xA;0.54 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.18 \\ &#xA;0.54 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.18 - 1 \\ &#xA;2 * 0.54 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.24 \\ &#xA;0.73 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.24 \\ &#xA;0.73 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.24 - 1 \\ &#xA;2 * 0.73 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.29 \\ &#xA;0.88 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.29 \\ &#xA;0.88 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.29 - 1 \\ &#xA;2 * 0.88 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.33 \\ &#xA;1.00 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.33 \\ &#xA;1.00 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.33 - 1 \\ &#xA;2 * 1.00 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.36 \\ &#xA;1.10 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.36 \\ &#xA;1.10 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.36 - 1 \\ &#xA;2 * 1.10 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.39 \\ &#xA;1.18 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.39 \\ &#xA;1.18 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.39 - 1 \\ &#xA;2 * 1.18 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.41 \\ &#xA;1.24 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.41 \\ &#xA;1.24 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.41 - 1 \\ &#xA;2 * 1.24 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.43 \\ &#xA;1.29 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.43 \\ &#xA;1.29 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.43 - 1 \\ &#xA;2 * 1.29 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.44 \\ &#xA;1.33 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.44 \\ &#xA;1.33 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.44 - 1 \\ &#xA;2 * 1.33 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.45 \\ &#xA;1.37 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.45 \\ &#xA;1.37 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.45 - 1 \\ &#xA;2 * 1.37 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.46 \\ &#xA;1.39 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.46 \\ &#xA;1.39 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.46 - 1 \\ &#xA;2 * 1.39 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.47 \\ &#xA;1.41 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.47 \\ &#xA;1.41 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.47 - 1 \\ &#xA;2 * 1.41 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.47 \\ &#xA;1.43 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.47 \\ &#xA;1.43 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.47 - 1 \\ &#xA;2 * 1.43 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.44 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.44 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.48 - 1 \\ &#xA;2 * 1.44 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.45 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.45 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.48 - 1 \\ &#xA;2 * 1.45 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.46 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.48 \\ &#xA;1.46 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.48 - 1 \\ &#xA;2 * 1.46 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.47 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.47 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.47 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.47 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.47 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.47 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.48 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.48 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.48 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.48 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.49 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.49 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.49 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ &#xA;\left(\begin{array}{cc}&#xA;x_1^{new} \\ &#xA;x_2^{new} &#xA;\end{array}\right) = &#xA;\left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right) &#xA;- 0.1 &#xA;\left(\begin{array}{cc} &#xA;2 * 0.49 - 1 \\ &#xA;2 * 1.49 - 3 &#xA;\end{array}\right) &#xA; = \left(\begin{array}{cc}   &#xA;0.49 \\ &#xA;1.49 &#xA;\end{array}\right)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we have convergence in 2D at those points above!&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-06T13:04:08.267" LastActivityDate="2018-03-06T13:04:08.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28700" PostTypeId="2" ParentId="28689" CreationDate="2018-03-06T12:45:43.903" Score="2" Body="&lt;h2&gt;Problem 1: &quot;Tactic Titles&quot; Are Really TEXT&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;i.e. they are not standard &quot;tags&quot;. I say this because they seem so structured and tag-like. For instance in text analysis using NLP techniques you mostly deal with texts in which &quot;Review Comments&quot; and &quot;Reviewing Comments&quot; and &quot;Comment Review&quot; are supposed to be considered the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Actually the way you already tried is a valid way to do it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another idea could be using information theoretic approaches. It means using something like Tf-IDF approach but this time taking classes into account. Here you are going to look for words/n-grams which appear a lot in desired class (&lt;strong&gt;High&lt;/strong&gt;) and not a lot in the other class. A simple formulation could be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Score_{w_i} = \frac{N_{high}}{N_{low}+1}$$&#xA;where $N_{high}$ is the number of times the word $w_i$ appeared in class &lt;strong&gt;High&lt;/strong&gt; and $N_{low}$ is the number of times the word $w_i$ appeared in class &lt;strong&gt;Low&lt;/strong&gt;. The constant $1$ is just a smoother to avoid a zero denominator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use it be careful about overfitting as you are learning features in a very implicit and hand-crafted way. To avoid overfitting you may prune your found words based on a &lt;em&gt;min_count&lt;/em&gt; or &lt;em&gt;max_count&lt;/em&gt; limitation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course this is a simple score and you can improve it by modification (e.g. i did not consider the normalization which is better to be considered, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For prediction you can also simply use naive Bayes. Count words/phrases based on Bayesian approach (just count the frequency of words happening in each class) and normalize them to probabilities. Now you have most likely data given classes and the probability of classes. A new data comes and $P(C|D)=P(D|C)\times P(C)$ is calculated easily.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Problem 2: The &quot;Tactic Titles&quot; Are Just Fixed Tags (Maybe irrelevant to score)&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Means that semantics are not correlated with classes i.e. two titles with similar terms or meanings might be in two different classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then it's not a learning problem. Either you have the label of the new data in the dictionary so you just extract the class, or you don't have it so you don't know about it!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;How would you want to know that &quot;Partner Launch&quot; is low priority but &quot;Review Comments&quot; is high if you have no meta-information form the organization behind it? How can you predict that based on this most probably &quot;Write Comments&quot; is &lt;strong&gt;High&lt;/strong&gt; or &lt;strong&gt;Low&lt;/strong&gt;? .If you argue that Machine Learning is supposed to learn that latent variable behind, then I would say yes! but please note that maybe that latent phenomenon is not correlated with &lt;strong&gt;Text&lt;/strong&gt;. For instance if you get meta-data about which department is doing any of this tasks, you may end up with a more correlated/causal set of features. &lt;strong&gt;&lt;em&gt;The long story short: Maybe the information is not in TEXT but some other aspects.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-06T12:45:43.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28701" PostTypeId="2" ParentId="10404" CreationDate="2018-03-06T12:48:01.363" Score="2" Body="&lt;p&gt;In many real-world problems (especially in cases where data are the result of measurements) the amount of data available is relatively small and it is very costful to get more measurements. Luckily, modern data science is based on computers, where we can repeat simulations of our experiment to produce more data. Bootstrapping actually creates bigger samples out of your existing data and can even do this in multiple repetitions to allow significance testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bootstrapping respects the distribution that your existing data have, so you don't need to make any assumptions regarding the distribution of your data. This is important because in most statistical approaches there are such assumptions, which sometimes we don't even realise or check before we apply them. (e.g. have a look &lt;a href=&quot;https://www.investopedia.com/ask/answers/073115/what-assumptions-are-made-when-conducting-ttest.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; or at &lt;a href=&quot;https://stats.stackexchange.com/questions/19675/what-normality-assumptions-are-required-for-an-unpaired-t-test-and-when-are-the&quot;&gt;this&lt;/a&gt; post)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bootstrapping is commonly used for the calculation of confidence intervals or for hypothesis testing. If you are using python, you might find the following links useful:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-&lt;a href=&quot;https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Calculation of confidence intervals with bootstrapping example&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-&lt;a href=&quot;https://stats.stackexchange.com/questions/92542/how-to-perform-a-bootstrap-test-to-compare-the-means-of-two-samples&quot;&gt;2-paired hypothesis testing with bootstrapping&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-03-06T12:48:01.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28703" PostTypeId="2" ParentId="28689" CreationDate="2018-03-06T13:03:20.330" Score="0" Body="&lt;p&gt;Have you tried the boring, straightforward approach? Get a list of all words and count how often they occur with a high or a low label. (Exclude words that occur only once or twice, and also the words that occur very often).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Score Label     Tactic Title&#xA; 1.   High            Build Batmobile&#xA; 2.   Low             Repair Batcave&#xA; 3.   High            Paint Batmobile&#xA; 4.   Low             Paint Batcave&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The words that occur more than once are &lt;code&gt;Paint, Batmobile, Batcave&lt;/code&gt;. &#xA;&lt;code&gt;Batmobile&lt;/code&gt; has 100% &lt;code&gt;High&lt;/code&gt; labels, &lt;code&gt;Paint&lt;/code&gt; has 50% and &lt;code&gt;Batcave&lt;/code&gt; has 0%. You get the idea.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-03-06T13:03:20.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28705" PostTypeId="1" CreationDate="2018-03-06T13:29:06.350" Score="1" ViewCount="3102" Body="&lt;p&gt;I have a convolutional network, taken from &lt;a href=&quot;https://github.com/imlab-uiip/keras-segnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; github, the one in &lt;code&gt;build.py&lt;/code&gt;. Because it was made in an outdated version of keras I am trying to rewrite this to version 2.1.4. I came a long but I have some trouble with (hopefully) the last step. Below I posted the code I have now but what I need to do yet is the final reshaping. I receive the error &lt;code&gt;ValueError: total size of new array must be unchanged&lt;/code&gt; which according to &lt;a href=&quot;https://stackoverflow.com/questions/48610981/segnet-in-keras-total-size-of-new-array-must-be-unchanged-error/48611204#48611204&quot;&gt;this&lt;/a&gt; answer I got this error because the shape of my output is not corresponding to what I want to reshape it to. However my height and width are both factors of 32 so I do not understand the problem. The final layer of the network (a batch normalization) returns a shape of &lt;code&gt;(Bat (None, 0, 256, 2)&lt;/code&gt;. According to the code I found I have to reshape this using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;autoencoder.add(Reshape((n_labels, img_h*img_w))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which would be &lt;code&gt;(2, (256 * 256))&lt;/code&gt;. This sounds incorrect the 256 * 256 part is very large, but I don't fully understand the reshape function. After all I took this from github so it should be correct but the notation can be changed with the introduction of the new keras versions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody shed some light on what this reshaping does and maybe suggest to what dimensions I need to reshape? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is the entire model I'm using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras import models&#xA;from keras.layers.core import Activation, Reshape, Permute&#xA;from keras.layers.convolutional import Conv2D, MaxPooling2D, UpSampling2D&#xA;from keras.layers.normalization import BatchNormalization&#xA;import json&#xA;&#xA;img_w = 256&#xA;img_h = 256&#xA;n_labels = 2&#xA;&#xA;kernel = 3&#xA;&#xA;encoding_layers = [&#xA;    Conv2D(64, (kernel,kernel), padding='same', input_shape=(1, img_h, img_w)),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(64, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    MaxPooling2D(),&#xA;&#xA;    Conv2D(128, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(128, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    MaxPooling2D(),&#xA;&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    MaxPooling2D(),&#xA;&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    MaxPooling2D(),&#xA;&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    MaxPooling2D(),&#xA;]&#xA;&#xA;autoencoder = models.Sequential()&#xA;autoencoder.encoding_layers = encoding_layers&#xA;&#xA;for l in autoencoder.encoding_layers:&#xA;    autoencoder.add(l)&#xA;&#xA;decoding_layers = [&#xA;    UpSampling2D(),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;&#xA;    UpSampling2D(),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(512, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;&#xA;    UpSampling2D(),&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(256, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(128, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;&#xA;    UpSampling2D(),&#xA;    Conv2D(128, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(64, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;&#xA;    UpSampling2D(),&#xA;    Conv2D(64, (kernel,kernel), padding='same'),&#xA;    BatchNormalization(),&#xA;    Activation('relu'),&#xA;    Conv2D(n_labels, (1, 1), padding='valid'),&#xA;    BatchNormalization(),&#xA;]&#xA;autoencoder.decoding_layers = decoding_layers&#xA;for l in autoencoder.decoding_layers:&#xA;    autoencoder.add(l)&#xA;&#xA;autoencoder.add(Reshape((n_labels, img_h * img_w)))&#xA;autoencoder.add(Permute((2, 1)))&#xA;autoencoder.add(Activation('softmax'))&#xA;&#xA;with open('model_5l.json', 'w') as outfile:&#xA;    outfile.write(json.dumps(json.loads(autoencoder.to_json()), indent=2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="47339" LastActivityDate="2019-09-18T06:59:52.707" Title="Reshape output of convolutional layer to which dimensions?" Tags="|python|neural-network|keras|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28706" PostTypeId="2" ParentId="28584" CreationDate="2018-03-06T14:01:35.063" Score="1" Body="&lt;p&gt;Your input matrix should have dimensions &lt;strong&gt;(# of instances, length of history of events, 1)&lt;/strong&gt;. For example, I have text data that is sequential and I will cut it up into sequences which are sequentially dependent. Each row in the matrix is the sequence, the history of events. My label for each of these is the next sequential letter, or event in your case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input $X$ has the following shape&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(X.shape)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(163717, 100, 1)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The LSTM model in Keras. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# define the LSTM model&#xA;model = Sequential()&#xA;model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))&#xA;model.add(Dropout(0.2))&#xA;model.add(Dense(y.shape[1], activation='softmax'))&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam') &#xA;model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;$y$ is my targets. Can be any kind of classification.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If you have 1000 instances of these sequences of events which have 50 historical values then your matrix should have shape &lt;strong&gt;(1000, 50, 1)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If you want to add structured data you can always do this after the LSTM layers into the densely connected layer. You can have two models whos outputs are concatenated and then added together. This is a lot easier to do with the functional API of Keras as it is much more flexible and allows you to interact with the tensors more directly. For example&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential, Model&#xA;from keras.layers import Concatenate, Dense, LSTM, Input, concatenate&#xA;from keras.optimizers import Adagrad&#xA;&#xA;seq_input = Input(shape=(X.shape[1], X.shape[2]))&#xA;seq_model = LSTM(256)(seq_input)&#xA;seq_model = Dropout(0.2)(seq_model)&#xA;seq_model = Dense(100, activation = 'softmax')(seq_model)&#xA;&#xA;metadata_input = Input(shape=(8, ))&#xA;metamodel = Dense(10, )(metadata_input)&#xA;&#xA;merged = concatenate([seq_model, metamodel])&#xA;merged = Dense(64, activation = 'softmax')(merged)&#xA;merged = Dense(y.shape[1], activation = 'softmax')(merged)&#xA;&#xA;model = Model(inputs=[seq_input, metadata_input], outputs=merged)&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This example is as above. However, I add a metadata input of size 8 to the model after the LSTM layers. This will contribute in the output as it will embed the information from those features in the last few layers of the model.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-06T17:38:00.530" LastActivityDate="2018-03-06T17:38:00.530" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28707" PostTypeId="2" ParentId="28584" CreationDate="2018-03-06T14:23:27.567" Score="1" Body="&lt;p&gt;If I understand you correctly, you are wondering whether it is possible to have inputs where &lt;em&gt;one element&lt;/em&gt; is a list or a matrix.&#xA;It is not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, this would require a complete rewrite of all activation and decision functions in Keras. Take a linear activation for instance. Keras expects that it gets a bunch of numbers $x_1, \dots, x_n$ and can calculate a weighted sum $w_1 x_1 + \dots + w_n x_n$.&#xA;What then should it do if you pass it a couple of lists $(x_1^1, \dots, x_1^{m_1}), \dots, (x_n^1, \dots, x_n^{m_n})$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 1:&lt;/strong&gt; Let me be a bit clearer about this point. Every component of a neural network is built on the assumption that the input tensors contain real values. Mathematically: The input must be of the form $X \in \mathbb{R}^{n_1 \times \dots \times n_k}.$&#xA;What you would like instead are elements of the set containing all real vectors of arbitrary length, $\mathcal{R} := \{x \in \mathbb{R}^m \mid m \in \mathbb{N}\}$: You want to input $X \in \mathcal{R}^{n_1 \times \dots \times n_k}.$&#xA;This would require you to rethink and redefine &lt;em&gt;absolutely everything:&lt;/em&gt; Activations, loss functions, forward propagation, back propagation... &lt;strong&gt;End of edit 1&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, it is unnecessary. &#xA;Say your input consists of a list $(x_1^1, \dots, x_1^m)$ and two scalar inputs $x_2$ and $x_3$. Then you should simply pass Keras the $m+2$ scalar inputs $x_1^1, \dots, x_1^m, x_2, x_3$. See JahKnows' answer for a nice implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Things get a bit more tricky if your input lists have different lengths ... so your first observation contains a list with five items, your second observation a list with seven items, etc.&#xA;In that case you need to use &lt;a href=&quot;https://www.quora.com/What-is-masking-in-a-recurrent-neural-network-RNN&quot; rel=&quot;nofollow noreferrer&quot;&gt;padding and masking&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit 2:&lt;/strong&gt; You say in the comments that you are unhappy with padding because some of your sequences are very long. Let me point out some alternatives:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use batch padding with variable length. This is common for networks that take sentences as input. The test set is ordered by sentence length and split into batches. Each batch is padded individually with as few pads as possible.&lt;/li&gt;&#xA;&lt;li&gt;Cut off sequences, either by dropping the oldest entries (for a time series), or the smallest, or the largest, or the most extreme, or the least extreme, or whatever makes sense for your data.&lt;/li&gt;&#xA;&lt;li&gt;Feed your network with derived properties of the sequence, not the sequence itself. The more I think about it, the more this seems like the best approach for your problem. A distribution is fully described by its &lt;a href=&quot;https://en.wikipedia.org/wiki/Moment_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;moments&lt;/a&gt;. By feeding enough moments (and the sequence length) into your network, you retain almost all information about the it. Notice also that moments remain unchanged for permutations of a sequence. Since you say that your sequences are not time series, this might be a very important property (and one that padding does not have!)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-03-06T22:48:12.970" LastActivityDate="2018-03-06T22:48:12.970" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28708" PostTypeId="2" ParentId="28675" CreationDate="2018-03-06T14:28:08.290" Score="1" Body="&lt;p&gt;You have a common issue in which your learning data is noisy. Of course if you can do anything to make your data better/use more data this is the way to go. But I assume manual cleaning at large is not possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reasons for the noise?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different approaches in machine learning to deal with noisy data. It depends a lot on where your noise is coming from. It helps to understand why your labels are incorrect. In your case it sounds like there is a structural mistake where you systematically misclassify under certain conditions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ML methods to solve the noisy data issue&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Besides simple rules (if applicable) there are more complex ways to approach the issue. There are many papers (e.g. &lt;a href=&quot;https://arxiv.org/pdf/1701.01619.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://www-labs.iro.umontreal.ca/~bengioy/cifar/NCAP2014-summerschool/slides/Sainbayar_cifar_sainaa_compressed.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://vision.cornell.edu/se3/wp-content/uploads/2017/04/DeepLabelCleaning_CVPR.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;) that suggest different variants of the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Manually create a small subset of cleaned labels&lt;/li&gt;&#xA;&lt;li&gt;Train the network on clean and noisy labels&lt;/li&gt;&#xA;&lt;li&gt;Use an additional network (or layer) to either create a mapping from noisy to clean labels or to learn about the noise distribution&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This way you try to make the network recognizing its own mistake. Some of these models are rather complex, others are quite complex. There is even a paper that trains the network to also &lt;a href=&quot;http://www.ee.cuhk.edu.hk/~xgwang/papers/xiaoXYHWcvpr15.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;classify the noise&lt;/a&gt; and than make predictions based on features + noise classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure if there are any best practises yet which architecture works best in which case. You may need to try out some alternatives. &lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-03-06T14:28:08.290" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28710" PostTypeId="2" ParentId="28705" CreationDate="2018-03-06T15:08:20.457" Score="2" Body="&lt;p&gt;Your data format is not the default data format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default, &lt;code&gt;Conv2D&lt;/code&gt;, &lt;code&gt;MaxPooling2D&lt;/code&gt;, and &lt;code&gt;UpSampling2D&lt;/code&gt; expect inputs of the form &lt;code&gt;[batch, height, width, channels]&lt;/code&gt;.&#xA;Your input is of the form &lt;code&gt;[batch, channels, height, width]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So your algorithm tries to apply convolution, pooling and upsampling to the &lt;code&gt;channels&lt;/code&gt; and &lt;code&gt;height&lt;/code&gt; dimensions, not to the &lt;code&gt;height&lt;/code&gt; and &lt;code&gt;width&lt;/code&gt; dimensions as intended.&#xA;The fix is easy: Add the option &lt;code&gt;data_format='channels_first'&lt;/code&gt; to all convolution, pooling and upsampling layers. (Or change your data format).&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2019-09-18T06:59:52.707" LastActivityDate="2019-09-18T06:59:52.707" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28711" PostTypeId="1" AcceptedAnswerId="28718" CreationDate="2018-03-06T15:28:42.777" Score="1" ViewCount="118" Body="&lt;p&gt;I've a list of 1,300 news events, represented by only three terms coming from running LDA topic model on thousands of tweets. Here's some of them as an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['manchester,bony,city', 'attack,claims,responsibility', 'police,officers,nypd',&#xA; 'goal,arsenal,liverpool', 'test,pakistan,sunday', 'obama,ukraine,merkel', ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to group them in more general domains (Politics, Sport, Health, Economy, etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which kind of clustering algorithms could I use (in Python)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or maybe, can I use LDA topic model, even if I don't have documents but only three words?&lt;/p&gt;&#xA;" OwnerUserId="46313" LastEditorUserId="24000" LastEditDate="2018-03-06T15:53:30.450" LastActivityDate="2018-03-06T17:05:11.750" Title="Cluster algorithm to group events in more general domains" Tags="|python|clustering|lda|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28712" PostTypeId="1" CreationDate="2018-03-06T15:35:47.237" Score="0" ViewCount="359" Body="&lt;p&gt;I know the plot of &lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperbolic_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;span class=&quot;math-container&quot;&gt;$\tanh$&lt;/span&gt;&lt;/a&gt; activation function looks like. I also know that its output has a range of &lt;span class=&quot;math-container&quot;&gt;$[-1, 1]$&lt;/span&gt;. Furthermore, I also know the it is defined as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$&#xA;\tanh(x) = \frac{\sinh(x)}{cosh(x)} = \frac{e^x - e^{-x}}{e^x + e^{-x}}&#xA;$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What exactly is the meaning of term &quot;hyperbolic&quot;? &lt;/li&gt;&#xA;&lt;li&gt;What is the interpretation of the word hyperbolic in this context? How is the equation above derived? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1151" LastEditorUserId="1151" LastEditDate="2019-02-11T03:53:48.113" LastActivityDate="2019-02-11T03:53:48.113" Title="What exactly is the &quot;hyperbolic&quot; tanh function used in the context of activation functions?" Tags="|machine-learning|neural-network|deep-learning|activation-function|linear-algebra|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28713" PostTypeId="2" ParentId="28712" CreationDate="2018-03-06T16:04:58.800" Score="2" Body="&lt;p&gt;The hyperbolic trig functions follow the equation for a Rectangular hyperbola, which is something you should be familiar from analytical geometry.  The recentgular hyperbola is defined by $x^2 - y^2 = 1$ and if you let $x = cosh(t)$ and $y = sinh(t)$ and plug it into the rectangular hyperbola equation you can verify this fact quite easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this idea you can derive the rest of the hyperbolic trig functions.  (Also they reason they are given trig names is because they resemble the properties trig functions have).&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-06T16:04:58.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28714" PostTypeId="1" AcceptedAnswerId="44684" CreationDate="2018-03-06T16:06:27.507" Score="6" ViewCount="8285" Body="&lt;p&gt;Has anyone seen &lt;a href=&quot;https://arxiv.org/abs/1706.03762&quot; rel=&quot;nofollow noreferrer&quot;&gt;this model's&lt;/a&gt; implementation &lt;strong&gt;using Keras&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;inb4&lt;/strong&gt;: &lt;a href=&quot;https://github.com/Kyubyong/transformer&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt;, &lt;a href=&quot;https://github.com/jadore801120/attention-is-all-you-need-pytorch&quot; rel=&quot;nofollow noreferrer&quot;&gt;pytorch&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14048" LastEditorUserId="29575" LastEditDate="2019-04-22T15:19:56.247" LastActivityDate="2021-10-11T14:57:50.403" Title="Is there &quot;Attention Is All You Need&quot; implementation in Keras?" Tags="|deep-learning|nlp|keras|machine-translation|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28716" PostTypeId="1" CreationDate="2018-03-06T16:47:39.310" Score="2" ViewCount="92" Body="&lt;p&gt;Are there any papers or research showing a correlation between walking stride and human height? My purpose is to estimate height from walking stride of a person.&lt;/p&gt;&#xA;" OwnerUserId="47354" LastActivityDate="2018-03-06T20:01:55.410" Title="Human Height Estimation using walking stride" Tags="|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28717" PostTypeId="1" CreationDate="2018-03-06T16:56:49.913" Score="1" ViewCount="299" Body="&lt;p&gt;Is there a way of making custom Python packages generally available within Jupyter lab?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While their documentation suggests to make install-able versions, and going through PIP, this is rather inconvenient for exploratory data science (especially, while I keep extending/developing these packages).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually I would point to the location of packages as a PYTHONPATH in .bash_profile (MacOS) – a step that would make them accessible in other contexts (e.g.: Jupyter notebooks) - but does not work with Jupyter lab (on the same machine).&lt;/p&gt;&#xA;" OwnerUserId="23437" LastEditorUserId="23240" LastEditDate="2019-03-15T04:13:41.167" LastActivityDate="2019-03-15T04:13:41.167" Title="Adding custom packages to Jupyter Lab?" Tags="|python|jupyter|pip|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28718" PostTypeId="2" ParentId="28711" CreationDate="2018-03-06T17:05:11.750" Score="0" Body="&lt;p&gt;Some creative post-processing can be done. For instance applying &lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;Named Entity Recognition&lt;/a&gt; and simplify some parts (&lt;strong&gt;Manchester&lt;/strong&gt; is a &lt;strong&gt;City&lt;/strong&gt;). Using Knowledge Graph Analysis also gives some meta-info e.g. mapping your word to Wikipedia graph or using DBpedia may help you to recognize Named Entitiy categories (&lt;strong&gt;&lt;a href=&quot;http://dbpedia.org/page/Barack_Obama&quot; rel=&quot;nofollow noreferrer&quot;&gt;Obama&lt;/a&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;a href=&quot;http://dbpedia.org/page/Angela_Merkel&quot; rel=&quot;nofollow noreferrer&quot;&gt;Merkel&lt;/a&gt;&lt;/strong&gt; are politicians and NER does not necessarily capture their profession).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the combination of named entity recognition and knowledge-base (Wikipedia, DBpedia, etc.) mapping is called &lt;a href=&quot;https://en.wikipedia.org/wiki/Entity_linking&quot; rel=&quot;nofollow noreferrer&quot;&gt;Entity Linking&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regardless of statistical learning for NLP techniques, all structures above are actually &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;graphs&lt;/a&gt; so they can &lt;a href=&quot;https://github.com/gsi-upm/sematch&quot; rel=&quot;nofollow noreferrer&quot;&gt;give you also the semantic similarity measure&lt;/a&gt; based on which you can use other clustering algorithms like &lt;a href=&quot;https://en.wikipedia.org/wiki/Spectral_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spectral Clustering&lt;/a&gt; and go to a semantically higher level of clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-06T17:05:11.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28719" PostTypeId="1" CreationDate="2018-03-06T17:31:15.207" Score="2" ViewCount="1667" Body="&lt;p&gt;I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was working through Ian Goodfellow's famous Deep Learning text. However, I found their exposition of back propagation unsatisfactory. They model the computational graph the wrong way, making the weights into the vertices and the operators into the edges. But this makes no sense, as it is the operations (not the variables) which must have an in-degree and out-degree to them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if there is a better reference, either in a textbook, a paper, or a blog that would rigorously outline the details of back propagation in its full generality.&lt;/p&gt;&#xA;" OwnerUserId="44256" LastActivityDate="2019-03-27T19:58:59.303" Title="A good reference for the back propagation algorithm?" Tags="|backpropagation|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28720" PostTypeId="2" ParentId="28692" CreationDate="2018-03-06T17:50:08.513" Score="2" Body="&lt;p&gt;No. There is no similar mechanism for continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it worries you, that &lt;code&gt;overall prices fluctuate in relatively low range&lt;/code&gt;,&#xA;you can&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) demean the price, that is subtract mean price from all price values. Then negative values will clearly show below-average and positive above-average prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) after demeaning you can divide values by the standard deviation of the price. (1) and (2) together is called &quot;standardization&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively you can &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) rescale your price to the range of values you want. Usual choice is (0,1) range.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do this for one feature (price in your case), then it makes sense to do the same for other features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whether or not this will help to get better prediction results depends on the model. Some models, a typical example would be SVM, do require such transformation.&lt;/p&gt;&#xA;" OwnerUserId="9085" LastActivityDate="2018-03-06T17:50:08.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28721" PostTypeId="2" ParentId="15135" CreationDate="2018-03-06T17:53:59.203" Score="2" Body="&lt;p&gt;Here's another approach (assumes equal three-way split):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# randomly shuffle the dataframe&#xA;df = df.reindex(np.random.permutation(df.index))&#xA;&#xA;# how many records is one-third of the entire dataframe&#xA;third = int(len(df) / 3)&#xA;&#xA;# Training set (the top third from the entire dataframe)&#xA;train = df[:third]&#xA;&#xA;# Testing set (top half of the remainder two third of the dataframe)&#xA;test = df[third:][:third]&#xA;&#xA;# Validation set (bottom one third)&#xA;valid = df[-third:]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This can be made more concise but I kept it verbose for explanation purposes.&lt;/p&gt;&#xA;" OwnerUserId="17397" LastActivityDate="2018-03-06T17:53:59.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28723" PostTypeId="2" ParentId="28446" CreationDate="2018-03-06T18:35:44.327" Score="5" Body="&lt;p&gt;There is this very cool idea in a paper by Ryabko[1] which is not yet very well known. This is called the telescope distance $D_H$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make a valid assessment about two time series, just looking at the data is not enough. You need to compare the underlying stochastic process that generates them, i.e. you want to compare two probability distributions. And the telescope distance is precisely a metric on the space of probability distributions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It goes like this (kind of formidable to the uninitiated). For a set of functions $\mathbf{H} = (\mathcal{H_1, H_2, \ldots})$, the telescope distance is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$D_{\mathbf{H}}(P, Q) \equiv \sum_{k=1}^{\infty} w_k \sup_{h \in \mathcal{H_k}} | E_P [h(X_1,\ldots,X_k)] - E_Q [h(Y_1, \ldots,Y_k)]|$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $P,Q$ are the distribution that generates $X$ and $Y$ respectively, and $w_k$ are some exponentially decaying weights (see the paper for details). You don't know $P$ and $Q$; all you got are the time series. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out you could use the following empirical quantity $\hat{D}$ to estimate the true telescope distance,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\small&#xA;\hat{D}_{\mathbf{H}}(X_{1:n}, Y_{1:m}) \equiv \sum_{k=1}^{min(m,n)} w_k \sup_{h \in  \mathcal{H_k}} \big|\frac{1}{n-k+1} \sum_{i=1}^{n-k+1} h(X_{i:i+k-1})- \frac{1}{m-k+1} \sum_{i=1}^{m-k+1} h(Y_{i:i+k-1}) \big|&#xA;$$&#xA;where $X_{1:n}$ and $Y_{1:m}$ are your observed time series. Notice the nice thing about this metric is, the two time series don't have to be equal in length. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, everything seems to be fine except that you notice you need the $h(\ldots)$? And it's not just one function, but a sequence of functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The cool idea is that these $h(\ldots)$s could be modeled as a binary-classifier well known in machine learning. For example, one could use SVM for these $h$ to discriminate between a subsequence $X$ and a subsequence of $Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've trained these binary classifiers, and there are $min(n,m)$ of them, you run them through the subsequences of the same length of $X$ and $Y$, sum them up and you're done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]Ryabko, D., &amp;amp; Mary, J. (2013). A binary-classification-based metric between time-series distributions and its use in statistical and learning problems. The Journal of Machine Learning Research, 14(1), 2837-2856.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastEditorUserId="27250" LastEditDate="2018-03-06T18:49:54.180" LastActivityDate="2018-03-06T18:49:54.180" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28724" PostTypeId="2" ParentId="28719" CreationDate="2018-03-06T19:26:23.227" Score="2" Body="&lt;p&gt;I made little search some days ago to get familiar with some Backpropagation related thing and came across to find &lt;a href=&quot;https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this pdf&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the beginning the author says that the approach is there taken so that it is most understandable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Directly from Author:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this chapter we present a proof of the backpropagation algorithm based&#xA;  on a graphical approach in which the algorithm reduces to a graph labeling&#xA;  problem. This method is not only more general than the usual analytical&#xA;  derivations, which handle only the case of special network topologies, but&#xA;  also much easier to follow.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;(Author: R. Rojas, 1996)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-03-06T19:26:23.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28726" PostTypeId="2" ParentId="28716" CreationDate="2018-03-06T20:01:55.410" Score="3" Body="&lt;p&gt;I did not find scientific data, but interesting ratio is mentioned on several sports sites like &lt;a href=&quot;https://www.walkingwithattitude.com/articles/features/how-to-measure-stride-or-step-length-for-your-pedometer&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Women have ratio 0.413 and men 0.415 and it is used by multiplying height with ratio and we get the stride length.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;My lenght 182cm, &#xA;ratio 0.415&#xA;stride length 182 x 0.415 = 75.53cm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-03-06T20:01:55.410" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28727" PostTypeId="1" AcceptedAnswerId="28728" CreationDate="2018-03-06T20:33:14.777" Score="1" ViewCount="1103" Body="&lt;p&gt;This question is about best practices for working in Pandas dataframes.  Speed, ease of use, and memory consumed could all impact any answers you might have.  I start by pulling a data set into a dataframe like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Date    Location    Value&#xA;3/4/2018    1        4795&#xA;3/5/2018    1        4795&#xA;3/4/2018    2        5022&#xA;3/5/2018    2        5088&#xA;3/4/2018    3        100&#xA;3/5/2018    3        100&#xA;3/4/2018    4        117154&#xA;3/5/2018    4        117154&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to sum the Value based on some other criteria.  For this example, lets use two states, SD and ND.  Location 1,2,4 are in ND, and Location 3 is in SD.  As I see it, I have two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Have Pandas post process the location numbers.  IE Pythonicly: ND = Sum(Loc 1,2,4), SD = Sum(Loc 4).  Then build/pivot the time series based on state&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Build a lookup table and have Pandas append the state to each row in the dataframe.  Then filter/group by state for the time series.  Lookup table would look like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Location    State&#xA;1           ND&#xA;1           ND&#xA;2           ND&#xA;2           ND&#xA;3           SD&#xA;3           SD&#xA;4           ND&#xA;4           ND&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In option one, would Pandas add a new row to the dataframe for each day of the timeseries with the state totals?  Or would the output be a pivot table like structure with only the state totals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If option two, what would be the best type of way to host the lookup table?  CSV?  JSON?  Table in SQL DB?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm concerned in option one changes would need to be made directly to the code.  Whereas in option two an addition to the lookup table could add the information required to add location to the correct group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I know this is open ended, I hope someone is willing to provide thoughts on efficient structure for this type of data flow.  &lt;/p&gt;&#xA;" OwnerUserId="47352" LastActivityDate="2018-03-06T20:40:53.167" Title="Most Efficient Post Processing with Python and Pandas" Tags="|python|pandas|sql|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28728" PostTypeId="2" ParentId="28727" CreationDate="2018-03-06T20:40:53.167" Score="0" Body="&lt;p&gt;I'd do it this way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;helper dictionary:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [79]: d = {'SD':[3], 'ND':[1,2,4]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;let's convert it to a Pandas Series:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [80]: lkp = pd.Series({el:key for key,lst in d.items() for el in lst})&#xA;&#xA;In [81]: lkp&#xA;Out[81]:&#xA;1    ND&#xA;2    ND&#xA;3    SD&#xA;4    ND&#xA;dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now we can map &lt;code&gt;Location&lt;/code&gt; into State and group by it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [82]: df.assign(Location=df['Location'].map(lkp)).groupby('Location')['Value'].sum()&#xA;Out[82]:&#xA;Location&#xA;ND    254008&#xA;SD       200&#xA;Name: Value, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-03-06T20:40:53.167" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28729" PostTypeId="1" AcceptedAnswerId="28743" CreationDate="2018-03-06T21:23:44.933" Score="1" ViewCount="972" Body="&lt;p&gt;I have a data set of 11 variables with allot of observations for each one. I want to make linear regression on the variables with the observed $\vec{y}=\alpha +\beta*\vec{X}$ when X is matrix. I'm trying to reduce my parameters so I activate pca algorithm on X. I get the &quot;loading&quot; data but i don't understand how to use it to get only four (for example) variables to estimate instead of 11.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;somebody can help? &lt;/p&gt;&#xA;" OwnerUserId="47366" LastEditorUserId="40853" LastEditDate="2018-03-07T08:08:54.250" LastActivityDate="2018-03-08T01:01:55.363" Title="How to use pca results for linear regression" Tags="|linear-regression|dimensionality-reduction|pca|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28730" PostTypeId="2" ParentId="28729" CreationDate="2018-03-06T21:48:50.753" Score="1" Body="&lt;p&gt;In general, I would suggest to use a regularization technique for reducing the dimensionality ofa data set in linear regression cases. Please refer to &lt;a href=&quot;http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;L1 regularization&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to decrease the number variables using PCA, you should look at the lambda values that describe the variations in the principle components, then, select the a few components with the largest corresponding lambda values (eg the first four). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do a scaling if necessary.&lt;/li&gt;&#xA;&lt;li&gt;Sometimes the very first component is not very relevant and can be eliminated.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-03-06T21:48:50.753" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28732" PostTypeId="1" CreationDate="2018-03-04T21:22:02.617" Score="1" ViewCount="51" Body="&lt;p&gt;&lt;br/&gt;&#xA;I'm trying to implement &lt;strong&gt;RMSProp&lt;/strong&gt; in my own Neural Network library so I can undertand the 'under-the-hood' operations, but this specific implementation is not working / converging, and I can't figure out why. &lt;br/&gt;I'm pretty sure I followed the &lt;a href=&quot;https://medium.com/100-days-of-algorithms/day-69-rmsprop-7a88d475003b&quot; rel=&quot;nofollow noreferrer&quot;&gt;formula&lt;/a&gt; (RMSProp + Momentum), and here is my code :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;//Step 1 - Compute a hidden neuron (this == ClNeuron) error gradient (TanH)&#xA;double tmpBuffer = 0.00;&#xA;for (std::size_t i = 0; i&amp;lt;this-&amp;gt;m_output_connections.size(); i++)&#xA;{&#xA;    ClNeuron* target_neuron = (ClNeuron*)m_output_connections[i]-&amp;gt;m_target_neuron;&#xA;    tmpBuffer += (target_neuron-&amp;gt;m_error_gradient * this-&amp;gt;m_output_connections[i]-&amp;gt;m_weight);&#xA;}&#xA;&#xA;//Get the TanH derivative of this neuron's last output&#xA;this-&amp;gt;m_error_gradient = tmpBuffer * this-&amp;gt;TanHDerivative(this-&amp;gt;m_result_buffer);&#xA;&#xA;&#xA;&#xA;&#xA;//Step 2 - For each of this neuron's input weights, compute the new weight's change&#xA;for(std::size_t i=0;i&amp;lt;this-&amp;gt;m_input_connections.size();i++)&#xA;{&#xA;    double new_weight_delta = this-&amp;gt;m_input_connections[i]-&amp;gt;m_learning_rate * this-&amp;gt;m_error_gradient * this-&amp;gt;m_input_connections[i]-&amp;gt;m_data;&#xA;    this-&amp;gt;m_input_connections[i]-&amp;gt;m_last_error_gradient_mean_square = 0.9 * this-&amp;gt;m_input_connections[i]-&amp;gt;m_last_error_gradient_mean_square + (1 - 0.9) * (new_weight_delta * new_weight_delta);&#xA;    this-&amp;gt;m_input_connections[i]-&amp;gt;m_momentum = this-&amp;gt;m_input_connections[i]-&amp;gt;m_momentum_constant * this-&amp;gt;m_input_connections[i]-&amp;gt;m_momentum + this-&amp;gt;m_input_connections[i]-&amp;gt;m_learning_rate * new_weight_delta / std::sqrt(this-&amp;gt;m_input_connections[i]-&amp;gt;m_last_error_gradient_mean_square)+0.000000001;&#xA;&#xA;    //Make the actual weight update&#xA;    this-&amp;gt;m_input_connections[i]-&amp;gt;m_weight -= this-&amp;gt;m_input_connections[i]-&amp;gt;m_momentum;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would anyone be kind enough to point me in the right direction ?&#xA;Many thanks !!&lt;/p&gt;&#xA;" OwnerUserId="47816" OwnerDisplayName="Charles-Ugo Brouillard" LastEditorUserId="1330" LastEditDate="2021-08-10T18:57:54.273" LastActivityDate="2021-08-10T18:57:54.273" Title="I Can't find my RMSProp implementation bug?" Tags="|backpropagation|implementation|c++|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28733" PostTypeId="1" AcceptedAnswerId="28909" CreationDate="2018-03-07T00:22:10.737" Score="2" ViewCount="556" Body="&lt;p&gt;Typical algorithms involve learning and applying a single mapping e.g. $ f: X \mapsto Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any algorithms that learn multiple mappings given an extra variable e.g. $ f(Z): X \mapsto Y$&#xA;(This feels like an abuse of notation.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we learn a mapping, given $Z$, that is then applied to $X$ to produce $Y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution (although intractable) would be to train an algorithm (e.g. a neural network) whose output is then used as the weights of a second network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions would be much appreciated and feel free to ask for more details. Thanks. &lt;/p&gt;&#xA;" OwnerUserId="37793" LastActivityDate="2018-03-10T06:32:32.110" Title="Are there any algorithms that learn to learn a function mapping?" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28734" PostTypeId="2" ParentId="26332" CreationDate="2018-03-07T00:35:45.103" Score="2" Body="&lt;p&gt;Because it makes the network learn the desired output (usually 1 or 0) much easier. It is possible to omit the logistic function (aka sigmoid function) and achieve desired results, but the network has to learn to specifically map each input to one or zero which is hard when it has the ability to map it to any real number. &#xA;The logistic function makes it easier because the network just has to learn to map it to a positive or negative number. &lt;a href=&quot;https://i.stack.imgur.com/PNO1x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PNO1x.png&quot; alt=&quot;Logistic function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the network can map an input to -6 if it wanted to and still produce the desired output -1, whereas before it has to output specifically -1.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, @Media logistic regression doesn't need a non-linearity for this purpose. It will still only be able to separate the data in a linear fashion. &lt;/p&gt;&#xA;" OwnerUserId="37793" LastActivityDate="2018-03-07T00:35:45.103" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28736" PostTypeId="2" ParentId="28676" CreationDate="2018-03-07T03:35:01.517" Score="3" Body="&lt;p&gt;Neural networks learn a function mapping from the input space to the output. This can be framed as a classification or a regression problem. In both of these cases the function determines the most information rich inputs which will lead to a minimized loss. We minimize the loss with respect to the input space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we have a complex valued input this is no problem. Let's assume two complimentary extreme cases. If we have a complex image, however, the complex part is completely randomly distributed. Then this problem would be equivalent to only using the real part of the image as the input. This is due to the complex part not having any pertinent information which would lead to the classification or regression. This is equally true, if the real part of the image part is completely random but the complex part contains all the information. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the in between situation where we do not know the information distribution in the real and imaginary space then you can input of these components into the network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a 64*64 complex image. This will result in $2*64^2$ input nodes each accepting a floating point value, if the first layer is a densely connected layer. This is commonly used for MRI images. So the nodes can identify which real or complex values are needed to infer meaning and provide an output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not seen any implementation that directly applies neural networks on complex values. But derivatives of complex values exist. You would need nodes which have complex functions such that the gradient is applied to both the real and imaginary part. From a guess, I suppose it could be possible. But the common method is just to split the real and imaginary part and feeding them to the network.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="40853" LastEditDate="2018-03-07T03:36:57.543" LastActivityDate="2018-03-07T03:36:57.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28737" PostTypeId="1" AcceptedAnswerId="28752" CreationDate="2018-03-07T03:55:23.023" Score="0" ViewCount="676" Body="&lt;p&gt;What is intuition behind multimodal distribution? and&#xA;How does GANs generate samples from it?&lt;/p&gt;&#xA;" OwnerUserId="47375" LastActivityDate="2018-03-07T11:26:39.767" Title="Multimodal distribution and GANs" Tags="|deep-learning|statistics|probability|gan|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-08T20:59:06.180" ContentLicense="CC BY-SA 3.0" />
  <row Id="28738" PostTypeId="2" ParentId="28719" CreationDate="2018-03-07T05:57:10.740" Score="3" Body="&lt;p&gt;Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;One neuron network is called the perceptron&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(z) = \frac{1}{1+exp(z)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $w$ is the associated weight for each input $x$ and we have a bias $b$.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How do we train these weights&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we can then update the weights through gradient descent as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\eta$ is the learning rate.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;What if we have 2 layers?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Take a look at the following answer: &lt;a href=&quot;https://datascience.stackexchange.com/questions/26874/compute-backpropagation/26905#26905&quot;&gt;Compute backpropagation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Why model neural networks as a graph&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-07T05:57:10.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28739" PostTypeId="1" CreationDate="2018-03-07T06:26:40.813" Score="2" ViewCount="2085" Body="&lt;p&gt;I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why and how is Chunking helpful?&lt;/li&gt;&#xA;&lt;li&gt;Plus aren't POS tags enough?&lt;/li&gt;&#xA;&lt;li&gt;Why are three tags necessary for the representation? Why not just use 'inside' tag and 'outside' tag?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39338" LastEditorUserId="39338" LastEditDate="2018-03-07T06:32:06.587" LastActivityDate="2018-03-08T17:44:07.543" Title="Why are Chunking and IOB tags necessary?" Tags="|nltk|named-entity-recognition|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28740" PostTypeId="1" AcceptedAnswerId="28744" CreationDate="2018-03-07T06:53:07.590" Score="2" ViewCount="292" Body="&lt;p&gt;I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to prevent overfitting. While tuning the dropout rate, I am increasing the value from 0.2 -&gt; 0.3 -&gt; 0.4 -&gt; 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know if I should increase the training data size to have a more accurate comparison. What I mean is suppose I am having training data of size 1 million for a dropout rate 0.2. Should I increase the training data size to 1.5 million for dropout rate 0.3?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1000000 * 0.8 * 0.8 * 0.8 = 512000 (for dropout ratio 0.2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1500000 * 0.7 * 0.7 * 0.7 = 514500 (for dropout ratio 0.3)&lt;/p&gt;&#xA;" OwnerUserId="47377" LastEditorUserId="47377" LastEditDate="2018-03-07T09:25:12.850" LastActivityDate="2018-03-07T10:11:10.770" Title="Do we need to increase training data size when increasing dropouts?" Tags="|keras|hyperparameter|dropout|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28742" PostTypeId="1" AcceptedAnswerId="28759" CreationDate="2018-03-07T07:42:50.563" Score="1" ViewCount="5629" Body="&lt;p&gt;I am trying to predict a categorial variable given a set of input variables, which are also categorical. Both the target and features variables only take the class values [below mean, mean, above mean].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have used one-hot encoding on both the target and feature vectors, but I am not sure if this is correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have 2 feature vectors I get 2x3 columns after one-hot encoding. For example, the first instance may be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X[0] = [0,1,0,    1,0,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;indicating the first feature has mean value ([0,1,0]) and the second feature is below mean ([1,0,0]), with the corresponding target&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y[0] = [1,0,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;indicating the target is below mean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the encoded X and y, I have tried to do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svm_clf = SVC()&#xA;svm_clf.fit(X_train, y_train)&#xA;y_pred = svm_clf.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but it gives me 'ValueError(&quot;bad input shape {0}&quot;.format(shape))'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have given the input and output below for the real situation, where the input has 9 features (so 9x3 columns after encoding). The error message indicates there is a problem with the 'y' vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I be one-hot encoding the target?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I tell the classifier that the output can only take a &quot;single value&quot; i.e. the 3 columns in the target are not independent as they all belong to the one variable. For example, the output for a given instance cannot be [1,1,0] as this indicates it is both &quot;below mean&quot; and &quot;mean&quot; which is not possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also tried a random forest classifier which ran OK, but the results were not plausible, so I assume I am doing something wrong,                      &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input features:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X_train type:  and shape: (872, 27) and head(5):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]&#xA; [ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]&#xA; [ 0.  1.  0.  0.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]&#xA; [ 1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.  1.  1.  0.  0.]&#xA; [ 1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  1.  0.  1.  0.  0.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Target:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_train type:  and shape: (872, 3) and head(5):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[ 0.  0.  1.]&#xA; [ 0.  0.  1.]&#xA; [ 1.  0.  0.]&#xA; [ 0.  0.  1.]&#xA; [ 0.  1.  0.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error message:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  File &quot;analyse.py&quot;, line 287, in dummy_test&#xA;    svm_clf.fit(X_train, y_train)&#xA;  File &quot;/usr/lib64/python3.6/site-packages/sklearn/svm/base.py&quot;, line 149, in fit&#xA;    X, y = check_X_y(X, y, dtype=np.float64, order='C', accept_sparse='csr')&#xA;  File &quot;/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py&quot;, line 578, in check_X_y&#xA;    y = column_or_1d(y, warn=True)&#xA;  File &quot;/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py&quot;, line 614, in column_or_1d&#xA;    raise ValueError(&quot;bad input shape {0}&quot;.format(shape))&#xA;  ValueError: bad input shape (872, 3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47380" LastEditorUserId="47380" LastEditDate="2018-03-07T10:15:30.667" LastActivityDate="2018-03-07T13:31:27.840" Title="How to structure data and model for multiclass classification in SVM?" Tags="|classification|scikit-learn|svm|multiclass-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28743" PostTypeId="2" ParentId="28729" CreationDate="2018-03-07T08:01:53.603" Score="0" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the outcome which you get from PCA explain the most of your original dataset. You need to name them based on your business understanding(Assuming that you know about data, as you mentioned that you wanted to apply, Linear Regression) else you might need some Subject Matter Experts expertise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of-course, the Features won't be same with the original data or else what is the point in performing PCA(I know that you understood this part). To decide on the number of features, you need to look at Scree Plot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PCA is a Dimensionality Reduction algorithm which helps you to derive new features based on the existing ones. PCA is an Unsupervised Learning Method,  used when the has many features, when you don't understand anything about the data, no data dictionary etc.For better understanding on PCA you can go through this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link-1&lt;/a&gt;,&lt;a href=&quot;https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues&quot;&gt;link-2&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now before performing Linear Regression, you need to check if these new features are explaining the Target Variable by applying Predictor Importance test(PI Test), you can go through the Feature Selection test in the &lt;a href=&quot;https://machinelearningmastery.com/feature-selection-in-python-with-scikit-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;python&lt;/a&gt;,&lt;a href=&quot;https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the outcome of PI Test you can go ahead and use those important feature for modeling and discarding the features which are not explaining the target variable well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, you can achieve the results which you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you are stuck somewhere.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-08T01:01:55.363" LastActivityDate="2018-03-08T01:01:55.363" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28744" PostTypeId="2" ParentId="28740" CreationDate="2018-03-07T09:57:44.270" Score="1" Body="&lt;p&gt;No, you should only tune one hyperparameter at a time. If you change two hyperparameters and the performance increases, how do you know which of the two parameters is responsible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have the time, do a full grid search on the dropout rate and the training data size.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastActivityDate="2018-03-07T09:57:44.270" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28747" PostTypeId="1" AcceptedAnswerId="28749" CreationDate="2018-03-07T10:11:34.480" Score="5" ViewCount="2817" Body="&lt;p&gt;I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term &quot;weight&quot; being used a lot, although it is a little confusing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know, that before the inputs are summed and passed to activation functions, they are separately weighted, after some research, i found out that the purpose of the weight function was to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;ensure orthonormality&lt;/li&gt;&#xA;&lt;li&gt;avoid data loss&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know that two inputs are only orthonormal if they are orthogonal unit vectors, if they are orthogonal then their dot product is always 0. &#xA;Dot product:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ytKHz.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ytKHz.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where theta is angle between two unit vectors and absolute value of a is norm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if two unit vectors are perpendicular to each other, then they are orthonormal and their dot product is 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But what does this have to do with data loss? I've also heard that sometimes the value of input might be zero, and we know that multiplication by zero outputs zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is the real purpose of the weight in neural networks? and what does it have to do with orthonormality? For example, what would be the purpose of weights in linear regression?&lt;/p&gt;&#xA;" OwnerUserId="47359" LastEditorUserId="47359" LastEditDate="2018-03-07T15:38:46.083" LastActivityDate="2018-03-27T04:18:19.130" Title="Purpose of weights in neural networks" Tags="|neural-network|data|weighted-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28749" PostTypeId="2" ParentId="28747" CreationDate="2018-03-07T10:55:50.713" Score="7" Body="&lt;p&gt;The reason for weights in machine learning is actually a lot easier than it seems. It's the way by which our model learns some underlining function and performs the classification or regression. We tune these weights in order to model some underlining function which can map our input to a desired output. Either a class in classification, or a range of values in a regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's look at an easier machine learning model so that we can understand why weights are needed. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Linear Regression&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is just a straight line which splits data into two sections. Let us apply this model in 2D, where we have two features $x = [x_1, x_2]$, for example weight and height. And labels $y$ which will split our data into $y \in \{men, women\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A random line in this space is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -2x_2 + x_1 + 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume this is our boundary line. Let's just assume that women fall above this line and men fall below it. Like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/biPEW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/biPEW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now if we get a new data point $x_{new} = [2, 8]$ we will label this as being a woman. So the entire decision is based on the numbers $1, -2, 1$ from our linear equation. We need to tune these values using the training data. We usually call these trainable parameters the weights $w$ associated with the features $x$ and we also add a bias $b$. In general the linear separator in 2D is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = w_1x_1 + w_2x_2 + b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our predicted label is $\hat{y} = w_1x_1 + w_2x_2 + b$. If $\hat{y} &amp;gt; 0$ then woman, else man.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In $n$ dimensions this can be written as a matrix multiplication as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{y} = w^Tx + b$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Obviously, a linear separator is not sufficient for most classification tasks. Things are not always linearly separable. So we use more complex models. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Neural networks&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In neural networks each node is associated with a function much like the linear separator. However, we will use the sigmoid function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\sigma(w^Tx) = \frac{1}{1 + exp^{-(w^Tx + b)}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weights here have the same effect. They will modulate the input values $x$ such that we are able to learn some classification or regression. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;How do the weights affect the decision boundary. We want the two different classes, circles and x's, to be on opposite sides of this boundary in order for us to be able to correctly classify them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The weights are trained iteratively using gradient descent. We can see that the decision boundary starts off terribly and then gets progressively better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -1.0 - 1.0x_1 - 1.0x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/b0QuQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/b0QuQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -16.0 - -39.0x_1 - 71.0x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/I7Jmd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/I7Jmd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -36.0 - -94.0x_1 - 61.0x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pdx8z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pdx8z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -83.5 - -134.0x_1 - 76.0x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/q858N.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q858N.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$0 = -88.5 - -114.0x_1 - 98.5x_2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TgSe4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TgSe4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see we changed the weights until we were able to find this ideal boundary between the two classes. If you want to find our more about how the gradient descent algorithm works to tune these weights you can look &lt;a href=&quot;https://datascience.stackexchange.com/questions/28691/gradient-descent-and-partial-derivatives/28699#28699&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-27T04:18:19.130" LastActivityDate="2018-03-27T04:18:19.130" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28750" PostTypeId="1" CreationDate="2018-03-07T11:09:51.527" Score="1" ViewCount="23" Body="&lt;p&gt;are there any companies with a proven track record to help in big data services or big data consulting services. &lt;/p&gt;&#xA;" OwnerUserId="47390" LastActivityDate="2018-03-07T11:09:51.527" Title="Which are the Best Big Data Services companies in 2018" Tags="|bigdata|" AnswerCount="0" CommentCount="0" ClosedDate="2018-03-07T11:18:25.727" ContentLicense="CC BY-SA 3.0" />
  <row Id="28752" PostTypeId="2" ParentId="28737" CreationDate="2018-03-07T11:26:39.767" Score="0" Body="&lt;p&gt;You can think of a multimodal distribution as a union of multiple unimodal distributions. In the case of GANs for image processing, each mode could be a category of images. To significantly simplify what's going on to motivate the idea, if you have a dataset of cat pictures and dog pictures, you can think of that as bimodal. Just cat pictures, that's unimodal. Training a GAN on a multimodal dataset of images means that it should be able to generate members of any image category in your dataset, and it should generate different categories with the same frequency with which they appeared in the training data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;GANs are trained by taking a random vector as input and attempt to construct a feasible member of the data distribution as output. In effect, the GAN learns a (surjective) mapping from the random space onto the multimodal distribution, such that random inputs will generate samples from the multimodal data distribution as outputs.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-07T11:26:39.767" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28753" PostTypeId="2" ParentId="28733" CreationDate="2018-03-07T11:40:40.157" Score="0" Body="&lt;p&gt;If $Z$ denotes a set of attributes such that for any $Z$, $f(Z)$ defines a family of functions, then you've basically stumbled on to hyperparameter search. As an example, consider a neural network with parameters $\theta$. We can reformulate your mapping $ f: X \mapsto Y$ as trying to learn the function $f(X;\theta) = Y$. But we have a lot of structural decisions to make here. What's our cost function? How many layers? What dimensions? What types of layers and activations? Do we want to have recurrence or convultions? Dropout? Normalization? Regularization? What learning rate? Should we use momentum? An annealing schedule? How many epochs? How much coffee should I drink before I start coding? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we wrap all of those structural decisions into $Z$, then $f(Z)$ is a specification for a family of functions for whom we are trying to learn the parameters $\theta$ that minimizes error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hyperparameter search is a big problem and there's a lot of interesting research into the topic. I think the current SOTA is bayesian optimization with bandit algorithms, although simple approaches like grid search, random search, and even guess-and-check are probably the most common.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-03-07T11:52:43.810" LastActivityDate="2018-03-07T11:52:43.810" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28754" PostTypeId="1" CreationDate="2018-03-07T11:51:32.513" Score="1" ViewCount="934" Body="&lt;p&gt;I am fitting a model with 100,000 samples x 10 features (6 ints and 4 floats), using SKLearn KernelRidge:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = KernelRidge(kernel='linear')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looking at the task manager, 'Python' process takes ~40GB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please explain why is there such a high demand?&#xA;What kind of matrix is built in the background?&lt;/p&gt;&#xA;" OwnerUserId="43705" LastActivityDate="2018-09-03T23:01:06.933" Title="SKLearn KernelRidge memory demand" Tags="|python|scikit-learn|regression|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28756" PostTypeId="2" ParentId="23082" CreationDate="2018-03-07T12:12:45.560" Score="2" Body="&lt;p&gt;ELM is actually solving a generalized linear problem :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;H\beta = Y &#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $H$ is the matrix of the hidden layer output, $\beta$ is the weight vector (or matrix) connecting the hidden layer and the output layer and $Y$ symbolizes the target of your training. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to find $\beta$ using the pseudo-inverse of H.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solving this problem is equivalent of finding the $\beta$ achieving the minimum of the Least-square problem $||H\beta - T ||$ : &#xA;\begin{equation}&#xA;\beta = \underset{\beta}{\text{min}} \left \vert\left \vert H\beta - T \right\vert\right\vert &#xA;\end{equation}&#xA;To answer your question &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How is it possible to assign values to links without actually seeing the Training set?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since $H$ depends on the activation functions, on the initialized weight matrix linking Input layer and hidden layer and on each hidden node bias, it encompasses all the input information and ensures a learning by tuning $\beta$ coherently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find lot more details in &lt;a href=&quot;http://www.di.unito.it/~cancelli/retineu11_12/ELM-NC-2006.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ELM Theory and Application&lt;/a&gt; or in this &lt;a href=&quot;http://www-lia.deis.unibo.it/phd/materials/courses/Reti%20Neurali%20Teoria%20e%20Applicazioni/Fariselli2/ELM-Workshop-NUS.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;ELM presentation&lt;/a&gt; for example.&lt;/p&gt;&#xA;" OwnerUserId="46749" LastActivityDate="2018-03-07T12:12:45.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28757" PostTypeId="1" CreationDate="2018-03-07T12:41:57.170" Score="1" ViewCount="134" Body="&lt;p&gt;If I will train one autoencoder with one vector only and a second autoencoder with a second vector only, does it mean if vectors were similar, that the hidden layer vectors of both autoencoders will be similar as well?&lt;/p&gt;&#xA;&lt;p&gt;Autoencoder structure is identical. The quantity of hidden layer neurons is smaller than input. So I want to reduce dimensionality. I understand that it will be overfitting. But for me, it looks like you do not learn features but just mimic your input function with the smaller dimensional vector while losing information. So you receive a kind of low dimensional approximation of the learned vector.&lt;/p&gt;&#xA;&lt;p&gt;So if vector V1 was similar to V2 and I will use V1 to train Autoencoder A1 and V2 to train Autoencoder A2, does it mean that the hidden layer vector of A1 and hidden Layer vector of A2 won't be that similar like V1 and V2 but still approximately quite similar?&lt;/p&gt;&#xA;" OwnerUserId="47395" LastEditorUserId="29575" LastEditDate="2021-06-16T23:49:30.243" LastActivityDate="2024-01-07T00:01:50.217" Title="two autoencdoers learned by two similar vectors (each one with its own). Similarity of hidden layer vectors will be the same?" Tags="|autoencoder|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28758" PostTypeId="1" AcceptedAnswerId="28762" CreationDate="2018-03-07T13:07:59.737" Score="1" ViewCount="500" Body="&lt;p&gt;I am using MLPClassifer example from&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/neural_networks_supervised.html#classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for training: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.neural_network import MLPClassifier&#xA;X = [[0., 0.], [1., 1.]]&#xA;y = [0, 1]&#xA;clf = MLPClassifier(solver='lbfgs', alpha=1e-5,&#xA;                    hidden_layer_sizes=(5, 2), random_state=1)&#xA;&#xA;clf.fit(X, y)                         &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At the predict step, we use test data [2., 2.], [-1., -2.] in&#xA;&lt;code&gt;clf.predict([[2., 2.], [-1., -2.]])&lt;/code&gt;. The output of this function is&#xA;&lt;code&gt;array([1, 0])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we observe, the test data [2.,2.] is not in the train dataset we passed. Still, we got the closest match as label 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What i am trying to find is if the test data i supplied is not in the train dataset, i should print a message to user that data is not valid instead of telling him the wrong label as 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, in knn classification, i have &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier.kneighbors&quot; rel=&quot;nofollow noreferrer&quot;&gt;kneighbours function&lt;/a&gt; which tells the distance of my closest neighbours to the test data i supplied in a 0 to 1 scale. So, i could easily eliminate the test data samples which are highly distant from my train data samples by keeping threshold at 0.6 or 0.7. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any criteria/threshold like this i could do with MLPClassifier or with any one of &lt;a href=&quot;http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Incremental Classifiers mentioned here&lt;/a&gt; which can restrict my test samples if not present in train dataset ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question migrated from &lt;a href=&quot;https://stackoverflow.com/questions/49149278/mlpclassifier-threshold-factor-to-eliminate-test-samples-that-are-not-in-match-w&quot;&gt;SO&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="15884" LastActivityDate="2018-03-07T16:11:57.117" Title="MLPClassifier threshold factor to eliminate test samples that are not in match with train data" Tags="|machine-learning|python|scikit-learn|data-science-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28759" PostTypeId="2" ParentId="28742" CreationDate="2018-03-07T13:31:27.840" Score="1" Body="&lt;p&gt;See the svm documentation &lt;a href=&quot;http://scikit-learn.org/stable/modules/svm.html#svm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It does support multiclass problem. The only thing is that your &lt;strong&gt;y&lt;/strong&gt; should be like this &lt;strong&gt;[0, 1, 2, 3]&lt;/strong&gt;. If you do not use one-hot encoding and use the y label as it is, it might work.&lt;/p&gt;&#xA;" OwnerUserId="28931" LastActivityDate="2018-03-07T13:31:27.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28760" PostTypeId="2" ParentId="28754" CreationDate="2018-03-07T13:53:42.517" Score="1" Body="&lt;p&gt;This is rather common.  The algorithm for KernelRidge requires a SVD to be performed.  Sadly, the SVD cannot handle a sparse matrix so the _pre_compute_svd function in sklearn just converts the matrix into a dense matrix and moves on.  This tends to blow up memory rather quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have a couple of choices.  Rewrite the method to handle sparse matrices or just use a different method.  SVR would be the most similiar alternative.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-07T13:53:42.517" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28761" PostTypeId="1" CreationDate="2018-03-07T15:25:12.180" Score="1" ViewCount="194" Body="&lt;p&gt;I would like to build a simple sentiment analysis classifier using logistic regression. I downloaded a list of positive and negative words from &lt;a href=&quot;http://www.cs.uic.edu/%7Eliub/FBS/sentiment-analysis.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cs.uic.edu&lt;/a&gt;. There are more than 6000 words both positive and negative. Linear Classifier has the form: (&lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_classifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia Reference&lt;/a&gt;)&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\sum_{1}^{N} w_j*x_j$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; is the weight of the vector &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and N is the number of words. So for example, if the weight of vector &lt;code&gt;awesome&lt;/code&gt; is &lt;code&gt;3&lt;/code&gt;, then in the following sentence:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Food is awesome and music is awesome.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;according to the formula, it will become:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$3 * 2$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where 3 is the weight of the vector &lt;code&gt;awesome&lt;/code&gt; and 2 is the vector itself (denotes the number of times it occurs in a sentence)&lt;/p&gt;&#xA;&lt;p&gt;My question is how do I decide the coefficients to start with? Will it be a manual process? There are more than 6000 words. What is the way to approach this?&lt;/p&gt;&#xA;" OwnerUserId="47403" LastEditorUserId="14518" LastEditDate="2023-02-13T09:00:09.817" LastActivityDate="2024-03-24T19:04:00.387" Title="Deciding Initial Weights In A Linear Classifier For Sentiment Analysis" Tags="|machine-learning|classification|logistic-regression|sentiment-analysis|machine-learning-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28762" PostTypeId="2" ParentId="28758" CreationDate="2018-03-07T16:11:57.117" Score="1" Body="&lt;p&gt;SGDClassifier has &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier.decision_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;desicion_function&lt;/a&gt; which tells the distance to the hyperplane, where the values are compared to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This value could imply too big and too low values.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastActivityDate="2018-03-07T16:11:57.117" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28764" PostTypeId="1" CreationDate="2018-03-07T16:14:49.050" Score="2" ViewCount="2267" Body="&lt;p&gt;How does one-sided label smoothing make the discriminator more robust by reducing the confidence in correct class?&lt;/p&gt;&#xA;" OwnerUserId="47375" LastActivityDate="2019-02-06T09:39:34.620" Title="One sided label smoothing in GANs" Tags="|deep-learning|statistics|probability|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28766" PostTypeId="2" ParentId="21638" CreationDate="2018-03-07T16:41:00.940" Score="0" Body="&lt;p&gt;It depends a bit on what uncertainty measures here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it means that there is a probability &lt;em&gt;p&lt;/em&gt; that it is the given outcome, then it's not that difficult to approach. If you have 4 outcomes and you know that outcome &lt;strong&gt;a&lt;/strong&gt; has 60%, then the other three together have 40%. You could either spread this 40% uniformly over the other three categories, use the prior probabilities to spread it out or use another model to predict how it would be spread out. Once you have a model over the other probabilities, you could use a neural network classification approach where you pass these probabilities instead of one hot encoded targets. Directly passing them to a tree based method will not work although XGBoost multiclas classification might be able to be tweaked to handle this. That said, another approach would be to then resample your dataset to make it significantly bigger, and sample the target from this distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If uncertainty is some metric that is more a feeling than a direct mathematical thing, weighting your loss is probably better but it might not necessarily be best to take this linearly.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-03-07T16:41:00.940" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28767" PostTypeId="2" ParentId="16088" CreationDate="2018-03-07T17:17:40.227" Score="1" Body="&lt;p&gt;I studied in in class and never really understood. I found teh best way : implement your own.&#xA;At the end this is a very &quot;pragmatic&quot; algorithm and it is more an execution routine than a full totalitarian method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for examples, you can see it in action on the classical (and very visual!) travelling salesman problem like here&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.theprojectspot.com/tutorial-post/applying-a-genetic-algorithm-to-the-travelling-salesman-problem/5&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.theprojectspot.com/tutorial-post/applying-a-genetic-algorithm-to-the-travelling-salesman-problem/5&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you do not need much than understanding the components for starters : chromosomes (= solutions), mutation function (altering one solution), crossover (mixing two solutions, meaning selecting parents and actually mixing them), fintess function (quality of the solution), and generation function (candidate solutions generation) and (optionally) elitism policy (how much we protect best solutions from disappearance). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After having understood it, you will see the modular aspect of GA and you will be able to dig in more advanced content (evolutionnary strategy, coding, etc.) if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thing is that in GA you almost define everything, they are only classical execution routines. This is why I do not think books will add a lot to your understanding in this case.&lt;/p&gt;&#xA;" OwnerUserId="47411" LastActivityDate="2018-03-07T17:17:40.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28768" PostTypeId="1" AcceptedAnswerId="28896" CreationDate="2018-03-07T17:35:07.520" Score="0" ViewCount="144" Body="&lt;p&gt;I have a neo4j database that stores forum posts. Replies to specific posts are encoded using a &lt;code&gt;replyto&lt;/code&gt; relationship. I am trying to retrieve all the information from the database, convert the relationships into node properties, then output all the nodes with the new &lt;code&gt;replyto&lt;/code&gt; property in JSON format. I think the &lt;code&gt;replyto&lt;/code&gt; property in the JSON object should be an array of all the post IDs that are replies to the given post, but please suggest something better if you think of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What cypher query should I use to retrieve this data and how should I convert the relationships into node properties using javascript?&lt;/p&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2018-03-09T17:31:12.430" Title="Convert graph database relationships into node properties" Tags="|data-cleaning|javascript|json|neo4j|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-10T17:52:44.877" ContentLicense="CC BY-SA 3.0" />
  <row Id="28769" PostTypeId="2" ParentId="12909" CreationDate="2018-03-07T18:01:35.323" Score="2" Body="&lt;p&gt;In machine learning paradigm, model refers to a mathematical expression of model parameters along with input place holders for each prediction, class and action for regression, classification and reinforcement categories respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This expression is embedded in the single neuron as a model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For single layer perceptron and deep learning model, one needs to extract this model by carefully walking the neurons and layers to collect and stitch activation function in an ordered fashion. &lt;/p&gt;&#xA;" OwnerUserId="47413" LastActivityDate="2018-03-07T18:01:35.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28770" PostTypeId="1" AcceptedAnswerId="39189" CreationDate="2018-03-07T18:12:53.917" Score="1" ViewCount="58" Body="&lt;p&gt;1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- What would be the benefits of transforming variables in such a way ?&lt;/p&gt;&#xA;" OwnerUserId="46374" LastActivityDate="2018-10-04T13:45:35.327" Title="Algorithms that would benefit from variable transformations?" Tags="|machine-learning|dataset|distribution|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28771" PostTypeId="2" ParentId="28770" CreationDate="2018-03-07T18:19:18.693" Score="0" Body="&lt;p&gt;Models with direct regularization on their weights benefit from this. These regularizations add a prior to the model and punishes high weights. If your variables are not in the same range then the regularization is not the same for each input. Search for weight regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another form is where in neural networks the weights are initialized in a way that they expect the inputs to be normally distributed. If they are far from that, the impact in the later layers can be fairly big, greatly impacting convergence and numerical stability.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, tree based methods don't care at all usually.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-03-07T18:19:18.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28772" PostTypeId="2" ParentId="10103" CreationDate="2018-03-07T18:23:39.137" Score="14" Body="&lt;p&gt;You must look at &lt;a href=&quot;https://github.com/DmitryUlyanov/Multicore-TSNE&quot; rel=&quot;noreferrer&quot;&gt;this Multicore implementation&lt;/a&gt; of t-SNE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I actually tried it and can vouch for its superior performance. &lt;/p&gt;&#xA;" OwnerUserId="47414" LastEditorUserId="29575" LastEditDate="2018-03-07T20:08:19.063" LastActivityDate="2018-03-07T20:08:19.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28773" PostTypeId="2" ParentId="5345" CreationDate="2018-03-07T18:44:40.313" Score="1" Body="&lt;p&gt;I made a &lt;a href=&quot;https://alternativeto.net/list/2063/guis-to-save-from-typing-r-code/&quot; rel=&quot;nofollow noreferrer&quot;&gt;list of all GUIs&lt;/a&gt; that produce R code through point-click dialogues. Most of these are not full IDEs and only complement. Rkward is a FOSS multiplatform competitor to Rstudio. R AnalyticFlow specializes in 2d graphical layout of icons of code. At end of link is code to install and run 6 IDE helpers.&lt;/p&gt;&#xA;" OwnerUserId="30507" LastEditorUserId="30507" LastEditDate="2018-03-09T16:29:12.117" LastActivityDate="2018-03-09T16:29:12.117" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28774" PostTypeId="2" ParentId="28757" CreationDate="2018-03-07T18:50:10.400" Score="0" Body="&lt;p&gt;I'd say if you are training with a single vector, then no they wouldn't be the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems unintuitive at first but one has to realize it doesn't make sense to learn the features of a single vector, only the features of a set of vectors. The goal of an auto-encoder it to use its parameters to compress its dataset into a lower-dimension representation (A.K.A latent space) as efficiently as possible. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By efficiently, I mean that the most amount of useful information is encoded into the latent space. This means the vectors close in input space will also be close in the latent space.  The auto-encoder achieves this by learning to utilize groups of commonly occurring pixels (e.g. features). For images, an A.E. might learn to describe a picture in terms of these-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BS9tV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BS9tV.png&quot; alt=&quot;http://colah.github.io/posts/2014-07-Conv-Nets-Modular/&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now instead of describing an image as a with all its pixels (i.e. the original image) we can instead describe it with lines of blobs of colors, effectively reducing the amount of information needed. Lines and blobs are learned instead of random pixels because lines and lines and blobs can describe any picture, but a random group of pixels can not. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, it is necessary when we have multiple vectors that we learn features that can approximate all the vectors in the dataset. And hence when we have only one vector we can use any combination of features because any combination of features will be able to approximate the original vector.&lt;/p&gt;&#xA;" OwnerUserId="37793" LastActivityDate="2018-03-07T18:50:10.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28775" PostTypeId="1" CreationDate="2018-03-07T19:15:10.657" Score="1" ViewCount="1349" Body="&lt;p&gt;I am using a transfer learning approach. For this I followed the &lt;a href=&quot;https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow for poets tutorial&lt;/a&gt;. I use a pre-trained InceptionV3 architecture trained on the Imagenet dataset. The last layer and the softmax classification is replaced and retrained, using a new set of 7 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Per class I have around 4.000 - 5.000 images. I tried multiple training parameters with an AdamOptimizer. The labels are noisy, about 15-20% of the labels are incorrect. The images show products of a certain category (e.g. cars) and the labels classify different type of a feature (e.g. 7 different types of tires, wheels).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Parameters&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;learning rate: 0.001&lt;/li&gt;&#xA;&lt;li&gt;iterations: 7.000&lt;/li&gt;&#xA;&lt;li&gt;batch size: 100&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Performance&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The test accuracy is 50%, train accuracy 68%. Visualising the learning rate, the network already reached 50% after 2000 iterations. What surprises me is the overall low performance as well as the lack of further improvement during the training time. Its also noteworthy that the network seems to make very hard to understand errors (not only mixing up similar classes but clearly distinguishable ones as well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I wonder: Is this potentially because retraining only a single layer is too limited to pick up the subtle differences in certain parts of the images? How would you go about it to improve?&lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2024-01-26T17:09:51.230" Title="Transfer learning: Poor performance with last layer replaced" Tags="|cnn|performance|transfer-learning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="28777" PostTypeId="1" CreationDate="2018-03-07T20:47:19.043" Score="1" ViewCount="626" Body="&lt;p&gt;i begin to work with tensorflow, and now i want to update a matrix (in the matrix, every row is one prediction, all the matrix is updated after a batch), i want to know what is the max value in the output of the mlp, and i have two problems, the first is how i can update a matrix, in the doc not says about update a matrix, i want to try update a simple array, and after update the complete row in the matrix (i'm not sure if allows that, or i will reassign all the row)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;...MLP...&#xA;&#xA;unbatched_states = tf.split(hidden, batch_size, 0) #hidden is the last layer from one MLP, the shape is 100,100, size of 100, and batch of 100&#xA;unbatched_sense_ids = tf.split(train_sense_one_hot, batch_size, 0) ##creates a list of tensors, every tensor is a case tagged&#xA;self.predictions = tf.Variable(tf.zeros([batch_size, embedding_size], dtype=tf.int64), trainable=False)#variable, where is stored the prediction)&#xA;&#xA;for i in range(batch_size):            &#xA;        predicted_sense_i = tf.argmax(unbatched_states[i], 1, name='prediction')  #get the index of the max value in the case i of the batch          &#xA;        predicted_sense=tf.Variable(tf.zeros([embedding_size], dtype=tf.int64), trainable=False) #a tensor updated marking one the max value&#xA;        predicted_sense = tf.scatter_update(predicted_sense, predicted_sense_i), tf.constant(1, dtype=tf.int64))   #update with the index of max value                   &#xA;        self.predictions = tf.scatter_update(self.predictions[i], predicted_sense, tf.constant(1, dtype=tf.int64)) #update the matrix&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Exist another way for update a matrix.&lt;/p&gt;&#xA;" OwnerUserId="47419" LastActivityDate="2018-03-07T20:47:19.043" Title="tensorflow tf.scatter_update over a matrix" Tags="|neural-network|tensorflow|supervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28778" PostTypeId="1" AcceptedAnswerId="28846" CreationDate="2018-03-07T21:22:01.607" Score="1" ViewCount="1093" Body="&lt;p&gt;I've been reading about word2vec and it's ability to encode words into vector representations. Coordinates (probabilities) of these words are clustered together with their usual context-neighbor words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if we have 10k unique words in our dataset, we will feed 10 000 features into the network. The network can have for example 300 hidden neurons, each with linear activation function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output of the net are 10 000 neurons, each with a softmax classifier. Every such a &quot;soft-maxed output&quot; represents probability to select the appropriate word from our 10k dictionary.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/DcSPa.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DcSPa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt; Well, I was able to get benefit of &quot;semantically grouped&quot; words via that final softmaxed-output layer. However, it seems way too large - it still has dimension of 10 000 probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I actually use &lt;strong&gt;that 300 neurons result&lt;/strong&gt; in &lt;strong&gt;some other networks of mine&lt;/strong&gt;? Say, in my LSTM, etc? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then weights of my LSTM wouldn't take so much space on disk. However, I would need to decode the 300-dimensional state back into my 10k-dimensional state, so I can look it up&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2:&lt;/strong&gt; ...are the &quot;Encoded&quot; vectors actually already sitting in the Hidden layer or in the Output Layer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;References:&#xA;&lt;a href=&quot;https://www.quora.com/Why-is-word2vec-a-log-linear-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;first&lt;/a&gt;, &lt;a href=&quot;https://stackoverflow.com/questions/42501035/questions-about-word-embeddingword2vec&quot;&gt;second&lt;/a&gt;, &lt;a href=&quot;http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;third&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-03-09T05:20:24.580" LastActivityDate="2019-01-20T18:29:21.037" Title="use hidden layer of word2vec instead of 'one-hot', to reduce number of weights in other nets?" Tags="|word2vec|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28779" PostTypeId="2" ParentId="28778" CreationDate="2018-03-07T23:06:45.867" Score="1" Body="&lt;p&gt;Are you asking if the 300 dimension 'middle layer' has learned to encode words into an embedding like word2vec? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the case, I'd say no. There's nothing in the one-hot vector that specifies distance between two words. For instance, the one-hot vectors for &lt;em&gt;cat&lt;/em&gt; and &lt;em&gt;table&lt;/em&gt; will be the same distance away for &lt;em&gt;cat&lt;/em&gt; and &lt;em&gt;lion&lt;/em&gt;. With embeddings, clearly you want to have the second example closer together. &lt;/p&gt;&#xA;" OwnerUserId="37793" LastEditorUserId="40853" LastEditDate="2018-03-08T03:40:28.143" LastActivityDate="2018-03-08T03:40:28.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28780" PostTypeId="1" CreationDate="2018-03-08T00:27:18.173" Score="1" ViewCount="193" Body="&lt;p&gt;&lt;strong&gt;PROBLEM&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have a list of 100,000+ google queries related to travel bookings. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;hotels in london&#xA;barcelona flight&#xA;city breaks to berlin&#xA;khao san road hostel&#xA;luxury holiday to paris&#xA;new york business class flight price&#xA;disneyland trip...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How can I extract the location i.e., London&lt;/li&gt;&#xA;&lt;li&gt;Classify the Line of Business i.e., flight, hotel, package, etc&lt;/li&gt;&#xA;&lt;li&gt;Classify the affinity i.e., luxury, family, city break, beach etc&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use this info to record the frequency of various pattern that exists in the keywords &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i.e. **keyword pattern**                **frequency**&#xA;(destination) hotel                          xxx&#xA;flight to (destination)                      yyy&#xA;(theme) (destination) hotel                  zzz&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;POTENTIAL SOLUTIONS&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Manual&lt;/strong&gt; - get an exhaustive as possible list of locations (most of&#xA;the queries will be for tourist destinations) and look for a match&#xA;against the keywords. Again, compare the keywords against possible&#xA;Line of Business Identifiers and Affinity Identifiers&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Google Cloud Natural Language API&lt;/strong&gt; - This can be used to analyse&#xA;Entities and Sentiment of text. e.g.&#xA;     hotels in london -&gt; entity(hotels), entity(london)&#xA;     barcelona flight -&gt; entity(flight), entity(barcelona)&#xA;This isn't very powerful and only supports english.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Machine learning&lt;/strong&gt; - seems difficult as I haven't got any descriptors&#xA;for the keywords. Would Naive Bayes be applicable or SVM?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd preferably like to run any solution in R too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone please suggest a direction/potential solution?&lt;/p&gt;&#xA;" OwnerUserId="47425" LastEditorUserId="40853" LastEditDate="2018-03-08T01:50:06.437" LastActivityDate="2018-03-08T12:08:48.263" Title="Data Mining - Intent matching and classification of text" Tags="|classification|data-mining|nlp|text-mining|svm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28781" PostTypeId="1" CreationDate="2018-03-08T01:13:38.697" Score="1" ViewCount="144" Body="&lt;p&gt;Can anyone help identify what kind of Visualization this is? &#xA;(google image search was not informative, unfortunately...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NPj9Y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NPj9Y.png&quot; alt=&quot;Image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41834" LastEditorUserId="8878" LastEditDate="2018-03-08T09:27:57.343" LastActivityDate="2018-03-08T09:27:57.343" Title="What kind of Visualization is this?" Tags="|visualization|plotting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28782" PostTypeId="1" CreationDate="2018-03-08T01:17:52.977" Score="1" ViewCount="698" Body="&lt;p&gt;I'm building a binary sound classifier using the ESC-50 dataset. I have taken one class, &quot;dog bark&quot;, to be positive and the rest of the 49 classes to be negative. As the dataset is imbalanced I'm running into lot of training issues. I tried building a model but couldn't get a f1-score greater than 0.3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using mfcc and fft as features. I have tried used LR and SVM to train without much success. Can't use Deep learning models as it's a real-time system and can't have much delay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I approach this problem?&lt;/p&gt;&#xA;" OwnerUserId="25297" LastEditorUserId="29575" LastEditDate="2018-03-08T14:21:10.273" LastActivityDate="2018-05-07T14:43:52.873" Title="ESC-50 Audio data for binary classifier" Tags="|machine-learning|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28783" PostTypeId="2" ParentId="28781" CreationDate="2018-03-08T01:30:02.327" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it is called as &lt;strong&gt;Flow Visualization&lt;/strong&gt;. I remember seeing something similar when I was running a Campaign using Google Analytics. That graph is used for showing the Click-Path Analysis, Visit Flow, Navigation Flow etc. You can go through this &lt;a href=&quot;http://cutroni.com/blog/2011/10/19/path-analysis-in-google-analytics-with-flow-visualization/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link-1&lt;/a&gt;, &lt;a href=&quot;https://www.e-nor.com/blog/general/google-analytics-flow-visualization-not-your-traditional-path-analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;link-2&lt;/a&gt; for better understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outcome of this graph was to see which path was mostly used and where can we find some room to improve it the ones which are not used mostly, finding the root cause and provide best experience to the users.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-08T03:34:29.630" LastActivityDate="2018-03-08T03:34:29.630" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28785" PostTypeId="1" CreationDate="2018-03-08T06:33:00.117" Score="1" ViewCount="1425" Body="&lt;p&gt;I would like to know the meaning of an autocorrelation graph of a sine wave. When the time lag is 0, then the autocorrelation should give the highest value of 1 since a copy of the signal is completely correlated to itself. By this logic, after a time equal to the period of the signal, the correlation should again be maximum since the shifted signal is again the signal itself. However, when I plot the correlation in python, I get a function which keeps increasing which goes against my intuition that the correlation function should be periodic. Can anyone please explain why the autocorrelation shows this trend in its graphical form?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;time = np.arange(0, 10, 0.1);&#xA;y = np.sin(time)&#xA;result = np.correlate(y, y, mode='full')&#xA;plt.plot(result[:int(result.size/2 )])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qdfkk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qdfkk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30397" LastEditorUserId="30397" LastEditDate="2018-03-08T06:38:17.710" LastActivityDate="2019-07-01T19:50:29.607" Title="Autocorrelation of a sine wave" Tags="|time-series|correlation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28786" PostTypeId="2" ParentId="25463" CreationDate="2018-03-08T07:25:45.767" Score="1" Body="&lt;p&gt;Actually the shapes are for simplification. If you want to know the correct behavior you have to take a look at the formulas of each &lt;code&gt;LSTM&lt;/code&gt; cell. To answer your first question, There may be different answers. What the pictures are depicting belongs to tasks which are many to many, and for each input you need exactly one output. There are different tasks for sequences that can be defined:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One to one&lt;/li&gt;&#xA;&lt;li&gt;One to many&lt;/li&gt;&#xA;&lt;li&gt;Many to one&lt;/li&gt;&#xA;&lt;li&gt;Many to many&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can take a look at &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For illustrating the formulas of each &lt;code&gt;LSTM&lt;/code&gt; cell I've given the following picture from professor Andrew Ng course about deep learning:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OuRuR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OuRuR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, each node in the &lt;code&gt;LSTM&lt;/code&gt; cell can be connected indirectly to the output of the adjacent cells of the previous time-step. It's indirect because there are gates between them. Also consider that &lt;code&gt;LSTM&lt;/code&gt; cell shares the weights for all the inputs of different time steps. Consequently each neuron in &lt;code&gt;LSTM&lt;/code&gt; cell is dependent to the input of the current time-step and the output of the adjacent nodes of the previous time-steps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the third question, the input size will be equal to the number of features of the input for each time-step. The number of the outputs depends on your task as I referred to at first. Take a look at the first link.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-08T07:25:45.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28787" PostTypeId="1" CreationDate="2018-03-08T07:48:00.917" Score="0" ViewCount="52" Body="&lt;p&gt;How did anaek created chat bots using slack for HR Tools and how much time will it take as a fresher to make a similar product?&lt;/p&gt;&#xA;" OwnerUserId="33960" LastActivityDate="2018-03-08T18:46:09.177" Title="Make a chatbot using slack" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-08T20:58:25.177" ContentLicense="CC BY-SA 3.0" />
  <row Id="28788" PostTypeId="2" ParentId="28661" CreationDate="2018-03-08T08:07:26.970" Score="5" Body="&lt;p&gt;No, it is perfectly possible to train on multiple categories. What you need, though, is an exhaustive list of these categories (in supervised learning, that is).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you are trying to associate sentences with topics, and you have a list of possible topics &lt;code&gt;topics = ['sports', 'soccer', 'politics']&lt;/code&gt;. &#xA;It sounds like your data look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentence                       | topics&#xA;-------------------------------|----------------------------------&#xA;'Barack Obama loves soccer'    | ['politics', 'sports', 'soccer']&#xA;'The parliament is important'  | ['politics']&#xA;'Soccer is fun'                | ['sports', 'soccer']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you need to one-hot encode the topics:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = [['Barack Obama loves soccer'], ['The parliament is important'], ['Soccer is fun']]&#xA;&#xA;Y = [[1, 1, 1], [1, 0, 0], [0, 1, 1]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then you train a neural network to predict not one but three (= number of topics) values.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-03-08T08:47:24.493" LastActivityDate="2018-03-08T08:47:24.493" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28789" PostTypeId="2" ParentId="25463" CreationDate="2018-03-08T08:26:39.720" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Is this correct?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes the diagrams both look correct to me. The key thing to understanding both diagrams is that the inputs and outputs of an LSTM cell are vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The circles in the first diagram do represent the concept that the layer contains multiple individual artificial neurons, and that might make you assume that the second diagram is a picture of &lt;em&gt;one&lt;/em&gt; of those neurons. Arguably there are multiple &quot;neurons&quot; or sub-layers with different roles inside a cell, because there are multiple places in which calculations of the form $f(W\mathbf{x} +b)$ occur, performing slightly different roles. I think the term &quot;cell&quot; is used to refer to this architecture of neurons. As a short-hand when I say &quot;neuron&quot;, I tend to think of the output stage of the hidden layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in the second diagram, all of the operations shown work with vectors. Most importantly the left-to-right arrows in the second diagram represent vectors of hidden state from timestep to timestep from the &lt;em&gt;whole layer&lt;/em&gt;. So each neuron in the cell is recurrently connected to every other neuron in that cell - twice as it happens in an LSTM, because LSTMs have both an internal cell state and a layer output.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;2&quot;&gt;&#xA;  &lt;li&gt;Is each unit in the cell independent from the others? Or do they share information?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To match your description of the diagrams, let's define a &quot;unit&quot; as a collection of one of each type of neuron/gate used to make up the cell, that in theory could be wired together to make a working LSTM cell layer with a single scalar cell state and output value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These units &lt;em&gt;are&lt;/em&gt; independent in that each has its own weight parameters. There are no shared parameters for the connections between input and the units, or for the recursive connections that forward state from one time step to the next. In that sense the units do not share information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the connections do mean that on each time step, input data and hidden state plus output from last output from all other units in the cell are combined are used in calculations. Any cell unit can base its new internal state plus its output on the values of all other outputs and internal states from other units in the cell. In this sense, the units do share information. I guess from your question that it is probably this second issue that you are concerned about, as the second diagram makes you think of a wiring diagram for a single neuron, but as explained above that is not the case.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol start=&quot;3&quot;&gt;&#xA;  &lt;li&gt;Imagine I have the following configuration: Number of samples = 1000 Number of time-steps = 10 Number of features = 5&#xA;  In this case, each unit in a cell will take as input a vector of size 5 right?&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Almost. Each neuron inside the cell will take an input of 5 from $\mathbf{x}$, plus an input of the hidden layer output, $\mathbf{h}$. So if in your case the LSTM cell size was 10, then each neuron would take a combined vector of 15. In addition, a second cell state vector is maintained, not labelled in your diagram. That is not used directly as an input to any neurons (i.e. components of the form $f(W\mathbf{x}+b)$), but does interact with the other values and can get changed itself, through the various gates. In your second diagram, it is the uppermost arrow going left to right.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But what will be the size of the output for one unit? 1?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The cell as a whole will have the output of whatever size you have made the layer. That is what the diagram number 2 tries to show. However, using our working definition of &quot;unit&quot;, the output of each unit will be two scalar values - the hidden layer output, and the cell state - which will be part of their respective vectors shown in the diagrams.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-08T12:16:29.890" LastActivityDate="2018-03-08T12:16:29.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28790" PostTypeId="2" ParentId="28003" CreationDate="2018-03-08T08:53:23.097" Score="7" Body="&lt;p&gt;it's quite easy to make! One thing to note I'm using the Keras functional API.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Input, Dense&#xA;&#xA;Input_1= Input(shape=(shape, ))&#xA;&#xA;x = Dense(100, activation='relu')(Input_1)&#xA;x = Dense(100, activation='relu')(x)&#xA;x = Dense(100, activation='relu')(x)&#xA;&#xA;out1 = Dense(1,  activation='linear')(x)&#xA;out2 = Dense(1,  activation='linear')(x)&#xA;out3 = Dense(1,  activation='linear')(x)&#xA;&#xA;&#xA;model = Model(inputs=Input_1, outputs=[out1,out2,out3])&#xA;model.compile(optimizer = &quot;rmsprop&quot;, loss = 'mse')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29767" LastEditorUserId="29767" LastEditDate="2020-04-09T05:40:36.950" LastActivityDate="2020-04-09T05:40:36.950" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28791" PostTypeId="1" AcceptedAnswerId="28793" CreationDate="2018-03-08T09:42:37.023" Score="2" ViewCount="370" Body="&lt;p&gt;Consider a New user which has never rated any movie on the Website or the System has never seen the user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the System recommend Movies to the User and based on what ? &#xA;How will we evaluate the Recommendation , is the recommendation accurate or appropriate and how confident the system is about the Recommendation .&lt;/p&gt;&#xA;" OwnerUserId="40277" LastActivityDate="2018-03-08T16:17:00.273" Title="How does a Recommender System recommend movies to a New User?" Tags="|machine-learning|neural-network|deep-learning|data-mining|recommender-system|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28792" PostTypeId="1" CreationDate="2018-03-08T09:50:46.203" Score="0" ViewCount="6569" Body="&lt;p&gt;I have a dataset like the following one:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xYn5g.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xYn5g.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each column is a different numerical feature. Each row represents a timestamp. I want to create an LSTM model that can make prediction of the future time-steps for all the features. For example, I want to use the first 2000 examples to train my model and use the next 1000 to test it. The problem is that I do not know how to proceed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since we do not have y's value in this dataset, I was thinking of creating them by shifting the time t+1 to t has a new column y. I explain myself: for example I will have a new column 14, with the value 996.52 for timestamp 0, which is the value at time-step 1 for the feature 0. And so on for all the time-steps and all the features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is after that I do not know how to feed my LSTM using Keras to make several steps predictions with such a dataset.&lt;/p&gt;&#xA;" OwnerUserId="47442" LastEditorUserId="29575" LastEditDate="2018-04-08T15:07:05.180" LastActivityDate="2018-04-12T00:55:48.840" Title="Predict multi-steps for Multi-Time series output with Keras" Tags="|keras|time-series|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28793" PostTypeId="2" ParentId="28791" CreationDate="2018-03-08T09:57:02.863" Score="3" Body="&lt;p&gt;In general, a safe default option for recommender systems is to recommend the most popular product, so two options for the recommendation:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;simply recommend the most popular movie in a time window around the current moment (maybe taking into account momentum),&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;do the same but marginalizing over account info, e.g. age, gender, country, language; also, IP location is an option to get such type of info.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Confidence assessment can be done based on recommendation success rate over the results of the same recommendation to previous users with the same profile characteristics, if such data is available.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-08T09:57:02.863" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28794" PostTypeId="2" ParentId="28782" CreationDate="2018-03-08T10:48:34.523" Score="1" Body="&lt;p&gt;On the &lt;a href=&quot;https://github.com/karoldvl/ESC-50&quot; rel=&quot;nofollow noreferrer&quot;&gt;github page&lt;/a&gt; of the ESC-50 dataset, there is a list of tried classifiers as well as links to the relevant papers. The best ones are all using some kind of deep learning, mostly CNNs, and currently the best score is 85.5%. A baseline Random forest (notebook available &lt;a href=&quot;https://github.com/karoldvl/paper-2015-esc-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) achived 44.3% and SVM (available &lt;a href=&quot;https://github.com/karoldvl/paper-2015-esc-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) achieved 39.6%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you take a look at the approaches linked above and see if you can improve upon them. However, for higher accuracy you probably need some kind of deep neural network.&lt;/p&gt;&#xA;" OwnerUserId="32492" LastActivityDate="2018-03-08T10:48:34.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28795" PostTypeId="2" ParentId="28672" CreationDate="2018-03-08T10:59:44.280" Score="0" Body="&lt;p&gt;According to Mueller in &lt;em&gt;Introduction to machine learning with Python: a guide for data scientists&lt;/em&gt; a Random Forest consists in several Decision Trees whose input is a reorganization of the initial datasets. This reorganization is called bootstrap.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still according to him, each tree will be trained on a feature space subset. During the training each tree will try to fit the data according to its feature subset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Gradient Boosting tree (a part of what is called Gradient boosting machine) the idea is to take shallows trees (shallow here means that each tree will have a very thin subset of the feature space) without &quot;boostraping&quot; the dataset. The different trees are set in a series circuit and each tree will correct the previous tree error. You may tune the learning rate between each tree for a better learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: Gradient Boosting Machine, in general, combines several &quot;simple&quot; models. In Gradient Boosting Tree the simple model are shallow trees etc.&#xA;You can have a look to this non mathematical article &lt;a href=&quot;https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d&quot; rel=&quot;nofollow noreferrer&quot;&gt;GB from Scratch&lt;/a&gt;   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not a complete answer but I hope it will be completed and still useful&lt;/p&gt;&#xA;" OwnerUserId="46749" LastActivityDate="2018-03-08T10:59:44.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28797" PostTypeId="2" ParentId="28681" CreationDate="2018-03-08T11:18:37.010" Score="0" Body="&lt;p&gt;Will &lt;code&gt;remove&lt;/code&gt; the answer once the &lt;code&gt;op&lt;/code&gt; sees it(please comment when you have done so)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Scatter Plot&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(8,6))&#xA;plt.scatter(range(df.shape[0]), np.sort(df.loan_amount.values))&#xA;plt.xlabel('index', fontsize=12)&#xA;plt.ylabel('loan_amount', fontsize=12)&#xA;plt.title(&quot;Loan Amount Distribution&quot;)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Try Box plot Also..&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;It does really help to correctly predict the outlier ones. We remove them for aesthetic visualisation of the data. However We can choose to (not) include them in our modeling activity because this is what we want to correct or improve on...&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-08T11:35:41.653" LastActivityDate="2018-03-08T11:35:41.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28798" PostTypeId="2" ParentId="28780" CreationDate="2018-03-08T12:08:48.263" Score="1" Body="&lt;p&gt;The questions you have listed are more or less independent from each other.&#xA;To &lt;strong&gt;extract location&lt;/strong&gt;, you should be looking into the class of solutions called Named Entity Recognition. It is widely supported, and, for example, NLTK should have location extraction support for languages but English.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second task, the &lt;strong&gt;line of business&lt;/strong&gt;, seems cryptic to me, since you are solving the task, you have complete information on the full list of those lines, so probably writing any kind of detector (custom NER would work) is a way to go, however, if all the queries are formed as LOB + location, by solving the first one, you can subtract the location out of query to get the line of business.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the examples of queries you have provided, I don't see any way for a human being to extract any affinity, so you should investigate it to understand if those queries actually carry such kind of information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With (1) and (2) given, the fourth task should be trivial to count.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2018-03-08T12:08:48.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28799" PostTypeId="1" AcceptedAnswerId="28801" CreationDate="2018-03-08T12:42:35.623" Score="12" ViewCount="7949" Body="&lt;p&gt;1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event occurring? (assuming we have an accurate model)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) The same question as in 1), but for other models that can output raw probabilities.&lt;/p&gt;&#xA;" OwnerUserId="46374" LastEditorUserId="55122" LastEditDate="2019-07-03T14:04:59.167" LastActivityDate="2019-08-07T07:19:59.870" Title="Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?" Tags="|machine-learning|classification|xgboost|probability|probability-calibration|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28800" PostTypeId="1" CreationDate="2018-03-08T12:46:31.747" Score="1" ViewCount="21" Body="&lt;p&gt;I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(related to this survey &lt;a href=&quot;https://datascience.stackexchange.com/questions/18572/machine-learning-learn-to-work-well-on-future-data-distribution&quot;&gt;Machine learning learn to work well on future data distribution?&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="16050" LastActivityDate="2018-06-24T14:00:34.330" Title="a simple way to test wether a tree-based classifier would transfer well to a target population?" Tags="|decision-trees|probability|transfer-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28801" PostTypeId="2" ParentId="28799" CreationDate="2018-03-08T13:24:09.287" Score="14" Body="&lt;p&gt;It depends on the definition of &lt;em&gt;accurate model&lt;/em&gt;, but in general the answer to your question 1) is No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding your second question (based on results in the paper of Niculescu-Mizil &amp;amp; Caruana linked below):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;boosted trees and stumps - NO&lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes - NO&lt;/li&gt;&#xA;&lt;li&gt;SVM - NO&lt;/li&gt;&#xA;&lt;li&gt;bagged trees - YES&lt;/li&gt;&#xA;&lt;li&gt;neural nets - YES&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can test whether it is the case for your particular model and dataset by looking at the so called &lt;em&gt;reliability plot&lt;/em&gt;: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Create N bins based on the model output (e.g. 10-20)&lt;/li&gt;&#xA;&lt;li&gt;Create a scatter plot with average model output for each bin along X axis and average true probability for each bin along Y axis&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ideally, your X-Y points should lie near the diagonal Y=X, otherwise the output of your classifier cannot be interpreted as a probability of an event. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, not all is lost and if needed, one can try to modify (calibrate) the output of the model in a such way that it better reflects the observed probability. In order to assess whether the calibration exercise was successful one might look at the reliability plot based on the calibrated model output (instead of using raw model output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two most widely used techniques for the calibration of classifier outputs are &lt;em&gt;Platt scaling&lt;/em&gt; and &lt;em&gt;isotonic regression&lt;/em&gt;, see the links below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that it is not advisable to cailbrate the classifier using the training dataset (you might need to reserve a separate subset of your data for the purpose of calibration).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some relevant links&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf&quot; rel=&quot;noreferrer&quot;&gt;Predicting Good Probabilities With Supervised Learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://danielnee.com/tag/platt-scaling/&quot; rel=&quot;noreferrer&quot;&gt;CALIBRATING CLASSIFIER PROBABILTIES&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://fastml.com/classifier-calibration-with-platts-scaling-and-isotonic-regression/&quot; rel=&quot;noreferrer&quot;&gt;Classifier calibration with Platt's scaling and isotonic regression&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="65675" LastEditDate="2019-08-07T07:19:59.870" LastActivityDate="2019-08-07T07:19:59.870" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="28802" PostTypeId="2" ParentId="28650" CreationDate="2018-03-08T13:43:33.317" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;We have tried including all of our 8 features (categorical ones being&#xA;  encoded in integer) and doing the linear regression.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If it is not dummy encoding and your categories can not be ranged - that is wrong. For example, doing this:&#xA;[apples, bananas, strawberries] -&gt; [0, 1, 2]&#xA;for almost all tasks will be not correct.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We have also tried taking out the categorical features and running the&#xA;  linear regression algorithm for each possible combination of our 3&#xA;  categorical features. That of course yields in a lot of regression&#xA;  runs (each category has 4 possible values; so 64 to be exact).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also needed to be revised. If some of your combinations have very small amount of cases you can not trust the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Our dataset has 8 features; 3 of them being categorical. We are&#xA;  willing to perform linear regression to fit our target data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Do dummy encoding. If you see some strict relationship between categories you can also add one-hot dummy encoding:&#xA;category A - category B&#xA;1 - 1&#xA;2 - 0&#xA;2 - 3&#xA;1 - 1&#xA;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Possibly here category A and category B are strongly correlated while both of them have category 1. Create new feature for this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general: play with your data. Find possible relationships and add them to your model.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I am forced to step out of linear regression, which algorithm&#xA;  should I try to apply? I would prefer to have one dataset with 8&#xA;  features instead of having 64 datasets with 5 features. There should&#xA;  be an algorithm that can capture this model.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Forests, XGBoost as mentioned earlier. For this you do not need one-hot or dummy encoding. By the way, usage of simple Decision Trees may give you beautiful pattern of relationships between categories and it's influence on target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try simple neural network after dummy and one-hot encoding too.&lt;/p&gt;&#xA;" OwnerUserId="46464" LastActivityDate="2018-03-08T13:43:33.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28804" PostTypeId="2" ParentId="28650" CreationDate="2018-03-08T15:43:55.630" Score="1" Body="&lt;p&gt;All the answers mentioned are great, but what I will do is ( a noob )&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Go with RF first to get the feature Importance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then plot the Hierarchical Clustering and plotting the &lt;strong&gt;Dendograms&lt;/strong&gt; using Scipy to see which columns are close to each other as the model this them to be..&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;After That, Go with &lt;code&gt;CatBoost&lt;/code&gt; to give your model the final touch.. CatBoost is really very effective when you have categorical data... It will handle all of them automatically...(try it if you haven't)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-08T16:41:07.137" LastActivityDate="2018-03-08T16:41:07.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28805" PostTypeId="1" AcceptedAnswerId="28824" CreationDate="2018-03-08T15:54:47.903" Score="0" ViewCount="207" Body="&lt;p&gt;I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an algorithm to dynamically update learning rate based on total error of the network without relying on any hyper-parameters ?&lt;/p&gt;&#xA;" OwnerUserId="45821" LastEditorUserId="45821" LastEditDate="2018-03-09T23:53:22.750" LastActivityDate="2018-03-09T23:53:22.750" Title="Learning Rate based on error of the network" Tags="|machine-learning|neural-network|deep-learning|optimization|learning-rate|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28808" PostTypeId="2" ParentId="28672" CreationDate="2018-03-08T16:43:55.367" Score="1" Body="&lt;p&gt;&lt;strong&gt;Random Forest&lt;/strong&gt;:  Build a decision tree.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Sample N examples from your population with replacement (meaning examples can appear multiple times).&lt;/li&gt;&#xA;&lt;li&gt;At each node do the following&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Select m predictors from all predictors&lt;/li&gt;&#xA;&lt;li&gt;Split based on predictor that performs best via some objective function&lt;/li&gt;&#xA;&lt;li&gt;Go to the next node, select another m predictors and repeat&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Combine all trees as an average or weighted via some scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Gradient Boosting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One key note is that random forest trees essentially indepedent of each other.  Boosting algorithms add a certain depedency to the model.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Initialize a model by finding the minimizer of a certain objective function&lt;/li&gt;&#xA;&lt;li&gt;For each iteration&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute the partial derivative of -$L(y_{i}, F(x_{i}))$ with respect to $F(x_{i})$ for all i to n&lt;/li&gt;&#xA;&lt;li&gt;Fit a tree $h_m$ to the the result from above.&lt;/li&gt;&#xA;&lt;li&gt;solve $\lambda_m =arg min \sum_{i=1}^{n}L(y_{i}, F_{m-1}(x_{i}) + \lambda h_{m}(x_{i})$&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;Update model via  $F_m(x) = F_{m-1} + \lambda_m h_{m} (x)$ &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-08T16:43:55.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28809" PostTypeId="1" CreationDate="2018-03-08T16:44:45.973" Score="2" ViewCount="82" Body="&lt;p&gt;How can you calculate better video recommendations for a SmartTV app which is used by multiple users in a household. I don't know which user is currently watching the video because the account for the app is shared. So the current user might get a recommendation based on video usage of an other user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of using principal component analysis to find out how many users live in the household and which user is likely to watch the video now. I could map the usage data to this specific user and then build recommendations for this hypothetical user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be the best approach to solve this problem. Are there best practices or papers which describe a solution?&lt;/p&gt;&#xA;" OwnerUserId="47465" LastActivityDate="2018-04-06T20:50:13.877" Title="distinguish users for recommender system" Tags="|recommender-system|pca|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28810" PostTypeId="1" AcceptedAnswerId="28812" CreationDate="2018-03-08T16:49:07.533" Score="1" ViewCount="304" Body="&lt;p&gt;I'm trying to understand can we implement a simple linear regression model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we are predicting price currencies. We want to know whether the currency will raise or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As i understand, we need to define two vectors for this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x=[1,2,3,4,5,6,7,8,9,10,11,12]$ - months&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=[2.30,2.33,2.29,2.30,2.36,2.40,2.46,2.50,2.48, 2.43,2.38,2.35]$ - average prices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's plot this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wPlXv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wPlXv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before sigmoid separator, i will try a simple linear separator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm guessing that at first, I need to choose some random slope (and bias will be the smallest scalar in vector).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$f(x) = 0.026x+2.3$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R1Irq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R1Irq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As we see, the separator is inaccurate, let's try the quadratic cost function at f(1):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = \frac{1}{1} \sum_{i=0}^{1}(2.3259999999999996-2.30)^2=0.0006759999999999897$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems accurate in the beginning, but it gets worse as it progresses, so somehow i need to improve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my knowledge, the next step is to find the derivative of the function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, gradient descent algorithm is used for this, but finding a slope of the tangent line is very easy here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{dy}{dx} = 0.026$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the next step? How can i use this derivative to use proper weights to minimize the cost function?&lt;/p&gt;&#xA;" OwnerUserId="47359" LastActivityDate="2018-03-08T17:15:29.743" Title="How to use correct weights in linear regression model" Tags="|linear-regression|loss-function|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28811" PostTypeId="2" ParentId="28809" CreationDate="2018-03-08T16:51:33.310" Score="1" Body="&lt;p&gt;I was interestedin this problem awhile back ago.  I still have this paper and should serve as a good primer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/ftp/arxiv/papers/1408/1408.2055.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Guess Who Rated Thos Movie:...&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-08T16:51:33.310" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28812" PostTypeId="2" ParentId="28810" CreationDate="2018-03-08T17:06:53.603" Score="2" Body="&lt;p&gt;You are correct for a linear separator line the derivative seems trivial. In this case gradient descent is not necessary because a closed form solution exists for the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w = (X^TX)^{-1}X^Ty$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We only use optimization techniques such as gradient descent for models where a closed form solution does not exist. However, if you perform gradient descent on the weights using the derivative and doing the following equation iteratively &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w^{new} = w^{old} - \nu \frac{dy}{dx}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;would yield the same result as the closed form solution. This is however very unnecessary when not necessary. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-08T17:15:29.743" LastActivityDate="2018-03-08T17:15:29.743" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="28814" PostTypeId="2" ParentId="130" CreationDate="2018-03-08T17:31:00.617" Score="0" Body="&lt;p&gt;Several great answers on here, in particular, @damienfrancois's answer very succinctly captures the general idea.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't see any examples of feature engineering for relational or time-series data. In that case, data scientists generally extract statistical patterns across relationships and over time. For instance, in order to predict what customers will by in the future in an ecommerce database, one might extract quantities like the average historical purchase amount, or the frequency of prior purchases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote a piece on this topic that goes into much more detail with several examples here: &lt;a href=&quot;https://www.featurelabs.com/blog/feature-engineering-vs-feature-selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.featurelabs.com/blog/feature-engineering-vs-feature-selection/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47468" LastActivityDate="2018-03-08T17:31:00.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28815" PostTypeId="2" ParentId="28739" CreationDate="2018-03-08T17:44:07.543" Score="3" Body="&lt;p&gt;BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take English as an example, some words will (most likely) never end a Named Entity, like adjectives, so the model will never tag them as the L(ast) part of a named entity. The same applies to the L-tag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is crucial, is that many models, like Conditional Random Fields, learn not only tags themselves, but also the transition probability, so, if you will get some chunk of text that is tagged as &lt;code&gt;B_ O_ L_&lt;/code&gt;, that sequence is incorrect, but when you learn the transitions as well, model will figure out, that if you get a strong Beginning and End, the inside part should also be a entity part.&lt;/p&gt;&#xA;" OwnerUserId="2573" LastActivityDate="2018-03-08T17:44:07.543" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28816" PostTypeId="2" ParentId="27822" CreationDate="2018-03-08T18:13:20.410" Score="0" Body="&lt;p&gt;for me at least this problem was caused outside the above function because I was using &lt;code&gt;tf.argmax&lt;/code&gt; to obtain &lt;code&gt;y_pred&lt;/code&gt; which is not differentiable (and also incorrect). I replaced by &lt;code&gt;y_pred[:, 1]&lt;/code&gt; which solved the problem.&lt;/p&gt;&#xA;" OwnerUserId="47470" LastActivityDate="2018-03-08T18:13:20.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28818" PostTypeId="1" AcceptedAnswerId="28831" CreationDate="2018-03-08T18:31:31.410" Score="4" ViewCount="1643" Body="&lt;p&gt;I have a random forest classifier and Multinomial Naive Bayes. For feature importance, I used gini index for random forest and for Multinomial Naive Bayes I used the coefficients of each feature. Then normalized to compare the two lists but there is a big difference between the two. Is this normal? Something like &lt;/p&gt;&#xA;&#xA;&lt;p&gt;RF vs NB&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A - C &lt;/li&gt;&#xA;&lt;li&gt;B - D&lt;/li&gt;&#xA;&lt;li&gt;C - A&lt;/li&gt;&#xA;&lt;li&gt;D - B&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47473" LastEditorUserId="47473" LastEditDate="2018-03-09T11:56:09.717" LastActivityDate="2018-03-09T14:04:40.347" Title="Can feature importance change a lot between models?" Tags="|random-forest|feature-engineering|naive-bayes-classifier|ranking|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28821" PostTypeId="2" ParentId="28787" CreationDate="2018-03-08T18:46:09.177" Score="1" Body="&lt;p&gt;I don't know the specific tools that you mentioned. But generally speaking, to create a chatbot, you would need to connect an API from Slack to your own API layer that handles the NLP side of it. So, if you're just starting out, your focus should be on creating an NLP algorithm that handles the types of questions that you want your bot to answer and converse with. You need to create code that both recognizes new inputs and understands how those inputs fit to everything else (i.e. an ongoing conversation). From there, you can create an API that you consume in any number of programs an you can practice with it (even in a terminal screen) for your own testing &amp;amp; algorithm training purposes. At that point, you should be able to integrate that into any number of platforms, slack being only one option. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-03-08T18:46:09.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28822" PostTypeId="1" CreationDate="2018-03-08T18:49:27.823" Score="3" ViewCount="4163" Body="&lt;p&gt;Why do we have two neurons in the output layer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does each neuron mean? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If our classifier is a binary classifier, will we have only one neuron in the output layer?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a picture of the neural network:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WOqMz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WOqMz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47474" LastEditorUserId="28175" LastEditDate="2018-03-09T08:14:58.333" LastActivityDate="2018-03-09T08:14:58.333" Title="Why do we have multiple neurons in the output layer of a neural network?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28824" PostTypeId="2" ParentId="28805" CreationDate="2018-03-08T18:55:17.250" Score="2" Body="&lt;p&gt;Your intuition is on point, and shrinking the learning rate like this is often referred to as &quot;annealing&quot;. But linking the learning rate to error magnitude neglects certain problematic error surface topologies. An excellent motivating example is the &lt;a href=&quot;https://en.wikipedia.org/wiki/Rosenbrock_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rosenbrock &quot;Banana&quot; Function&lt;/a&gt;, which is often used as a test case for optimization algorithms. The &quot;banana&quot; is a low error valley which hides the global minimum. If an optimization path finds its way into this valley, the path to the global minimum is along a nearly flat gradient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yaD3E.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yaD3E.png&quot; alt=&quot;The Rosenbrock Function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use an optimization algorithm that naively shrinks the learning rate relative to the error, you're going to get stuck as soon as you hit the valley. On the one hand: congrats! You've achieved a low error solution. But you're not necessarily anywhere near the global minimum. So how can we do better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An approach used by modern gradient-based methods like Adagrad, RMSProp, and Adam is to separately assign learning rates to each parameter, and tie the learning rate to the magnitude of the respective parameter's &lt;em&gt;update&lt;/em&gt;. The &lt;a href=&quot;http://cs231n.github.io/neural-networks-3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS231n lecture notes&lt;/a&gt; explains:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Adagrad&lt;/strong&gt; is an adaptive learning rate method originally proposed by Duchi et al..&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;code&gt;# Assume the gradient dx and parameter vector x&lt;br&gt;&#xA;  cache += dx**2&lt;br&gt;&#xA;  x += - learning_rate * dx / (np.sqrt(cache) + eps)&lt;/code&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Notice that the variable cache has size equal to the size of the gradient, and keeps track of per-parameter sum of squared gradients. This is then used to normalize the parameter update step, element-wise. Notice that the weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-03-08T19:30:07.937" LastActivityDate="2018-03-08T19:30:07.937" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28825" PostTypeId="1" AcceptedAnswerId="29267" CreationDate="2018-03-08T18:55:29.587" Score="1" ViewCount="380" Body="&lt;p&gt;I need to apply classifier algorithm after clustering. Now after clustering I find the id numbers that which id belongs which cluster. I clustered them into 2 cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I need to collect those data by using those id. But I don't know how I can collect all information by using those id.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I use jupyeter notebook and in main data I have no attribute named id and those id assigned jupyter notebook when I load data from main data file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is my main data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/411Mi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/411Mi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code to find which data belongs which cluster.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = 0.10&#xA;i=0&#xA;C_i = np.where(labels == i)[0].tolist()&#xA;n_i = len(C_i) # number of points in cluster i&#xA;&#xA;# (2) indices of the points from X to be sampled from cluster i&#xA;sample_i = np.random.choice(C_i, int(x * n_i)) &#xA;print (i, sample_i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and after clustering I find these ids&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L7ApH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L7ApH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New addition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose my loading file name is train.&#xA;now using &lt;code&gt;train.loc[26]&lt;/code&gt; command I get the info of that specific id.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/krIh5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/krIh5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I need to collect all info into a new data frame like as &lt;code&gt;train&lt;/code&gt; dataframe&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="43455" LastEditDate="2018-03-19T15:56:26.393" LastActivityDate="2018-03-19T15:56:26.393" Title="How to extract all information by using id" Tags="|machine-learning|python|clustering|jupyter|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28826" PostTypeId="2" ParentId="28822" CreationDate="2018-03-08T18:59:49.653" Score="3" Body="&lt;p&gt;If you want multiple things out of your network you need multiple output nodes. In the case of multiclass classification you want multiple outputs, one for each class. These represent the probability distribution over the different classes. For binary you can get away with only one output because the other class has &lt;code&gt;1-P&lt;/code&gt; as the probability. You could say the same for multiclass that you need one less, however this is commonly parameterized with a softmax which needs all values for the denominator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other cases might be for example one output per pixel, for example for image segmentation.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-03-08T18:59:49.653" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28827" PostTypeId="1" AcceptedAnswerId="28829" CreationDate="2018-03-08T19:10:16.697" Score="1" ViewCount="2237" Body="&lt;p&gt;I am working a classification problem. The dataset was collected from  Painters by number, a competition hosted by kaggle. The task is to identify painter,style and genre given paintings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I trained individual models to predict painter,style,genre given paintings. Now i would like to incorporate Multi task learning (i.e) Developing a single model which can predict all three tasks.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue i am currently facing is in designing architecture.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Model                         No of classes (Softmax)&#xA;   (Individual Models)                   &#xA;   --------------------             -----------------------&#xA; Model predicts painter                     8 (8 - painters)&#xA; given paintings &#xA;&#xA; Model predicts style                       10 (10 style classes)&#xA; given paintings      &#xA;&#xA;&#xA; Model predicts genre                       23 (23 genre classes)&#xA; given paintings    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't know how to combine the above models in keras. Any suggestions or feedback would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="2893" LastActivityDate="2018-03-08T19:32:49.957" Title="Multi task learning architecture for Multi-label classification" Tags="|keras|multitask-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28828" PostTypeId="2" ParentId="28825" CreationDate="2018-03-08T19:12:59.137" Score="1" Body="&lt;p&gt;Define a new column, then use these ids to select the relevant rows and set that column to the appropriate id:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame,Series&#xA;from numpy.random import rand&#xA;df = DataFrame(rand(10,10)).assign(cluster=0)&#xA;clusters = Series([[1,3,5],[0,2,4,6,7,8,9]])&#xA;for cluster,rows in clusters.iteritems():&#xA;    df.loc[rows][&quot;cluster&quot;] = cluster&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you can do your &lt;code&gt;groupby&lt;/code&gt;s on column &quot;cluster&quot;.&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2018-03-08T19:20:58.233" LastActivityDate="2018-03-08T19:20:58.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28829" PostTypeId="2" ParentId="28827" CreationDate="2018-03-08T19:32:49.957" Score="3" Body="&lt;p&gt;You should design a multi-task model (MTM).&#xA;MTM has the ability to share learned representations from input between several tasks. More precisely, we try to simultaneously optimize a model with &lt;code&gt;m&lt;/code&gt; types of loss function, one for each task. Consequently, MTM will learn more generic features, which should be used for several tasks, at its earlier layers. Then, subsequent layers, which become progressively more specific to the details of the desired task, can be divided into multiple branches, each for a specific task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need an architecture like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Your input and hidden layers&#xA;inp = Input(...)     &#xA;x = Layer1(...)(inp)&#xA;x = Layer2(...)(x)&#xA;...&#xA;x = Layer_N-1(...)(x) &#xA;&#xA;# Your 'm' output layers&#xA;out_1 = Layer_N(x) &#xA;out_2 = Layer_N(x)&#xA;...&#xA;out_m = Layer_N(x)&#xA;&#xA;# your model&#xA;MTM = Model(inputs=inp, outputs=[out_1,out_2,...,out_m])&#xA;MTM.compile(loss=[loss_func_1, loss_func_2,..., loss_func_m], &#xA;      optimizer='xxx', &#xA;      metrics=xxx)&#xA;&#xA;MTM.fit(train_data, [train_labels_1,train_labels_2,...,train_labels_m],                &#xA;          batch_size = xxx,&#xA;          epochs = xxx)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want a real example in Keras, I have implemented a 2-task ConvNet for my project &lt;a href=&quot;https://github.com/mmalekzadeh/motion-sense/blob/master/codes/gen_paper_codes/1_MotionSense_Trial.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2018-03-08T19:32:49.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28831" PostTypeId="2" ParentId="28818" CreationDate="2018-03-08T20:06:04.527" Score="6" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is this normal?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is not surprising. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, you are using different measures of feature importance. It’s like measuring the &lt;em&gt;importance&lt;/em&gt; of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, the performance of your models is likely to be different. In extreme case, output of one of your models could be completely rubbish (in your case it is more likely to be NB). Then the feature importance metrics produced with such model is not credible. In less extreme scenarios when the difference in models‘ performance is not so dramatic the trustworthiness of importances produced by two different models is more comparable. Still the importances might be quite different due to the first argument, i.e. different &lt;em&gt;language&lt;/em&gt; used to capture the importance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have not asked about it in your question, but there are feature importance  approaches which are model agnostic, i.e. can be applied to any predictive model. For example, check the &lt;em&gt;permutation importance&lt;/em&gt; approach, described in chapter &quot;15.3.2 Variable Importance&quot; in &lt;a href=&quot;https://web.stanford.edu/~hastie/Papers/ESLII.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Elements of Statistical Learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2018-03-09T14:04:40.347" LastActivityDate="2018-03-09T14:04:40.347" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28834" PostTypeId="2" ParentId="27634" CreationDate="2018-03-08T22:22:22.883" Score="1" Body="&lt;p&gt;here is example code to extract audio features as log spectrogram using python scipy: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from scipy.io import wavfile&#xA;from scipy import signal&#xA;&#xA;sample_rate=16000&#xA;window_size=20&#xA;step_size=10&#xA;eps=1e-10&#xA;rate, data = wavfile.read('filename.wav')&#xA;if data.ndim &amp;gt; 1 : # ignore  channels 2+&#xA;    data = data[:, 0]&#xA;nperseg = int(round(window_size * sample_rate / 1e3))&#xA;noverlap = int(round(step_size * sample_rate / 1e3))&#xA;freqs, times, spec = signal.spectrogram(data,fs=sample_rate,window='hann',nperseg=nperseg,noverlap=noverlap)&#xA;log_specgram = np.log(spec.T.astype(np.float32) + eps)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47483" LastActivityDate="2018-03-08T22:22:22.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28835" PostTypeId="1" CreationDate="2018-03-08T22:53:59.387" Score="5" ViewCount="4828" Body="&lt;p&gt;I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have just started using tensorboard &amp;amp; noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IdRAo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IdRAo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change &amp;amp; the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Am I interpreting the plots correctly that I do indeed have vanishing gradients &amp;amp; thus my convolutional layers arent learning? Does this mean they are currently essentially worthless? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) What can I do to remedy this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 3/13/18&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Few comments:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I have tried the network w/ just 1 layer and no layers (RNN--&gt;FC), and having 2 kayers does empirically improve performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?&lt;/p&gt;&#xA;" OwnerUserId="47279" LastEditorUserId="29169" LastEditDate="2021-02-13T16:37:22.133" LastActivityDate="2021-02-13T16:37:22.133" Title="How to fix these vanishing gradients?" Tags="|deep-learning|neural-network|tensorflow|nlp|convolutional-neural-network|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28836" PostTypeId="1" CreationDate="2018-03-08T22:54:06.967" Score="2" ViewCount="5850" Body="&lt;p&gt;I have a classic &lt;code&gt;User-Item dataset&lt;/code&gt; where each row (i.e., &lt;code&gt;(user, item)&lt;/code&gt;) indicates the action of a user clicking/selecting an item. Now, the dataset only provides &lt;em&gt;positive samples&lt;/em&gt; and does not specifically indicate whether a user has disliked an item. In order to create a &lt;em&gt;balanced dataset&lt;/em&gt;, I would like to create random negative samples (for instance randomly pick a set of items which the user has never clicked). Of course, I can achieve this by writing a program; however, I found this library in python Scikits called &lt;a href=&quot;https://pypi.python.org/pypi/imbalanced-learn#id32&quot; rel=&quot;nofollow noreferrer&quot;&gt;imbalanced-learn&lt;/a&gt;, which seems to provide various sampling techniques. Can someone provide a small code-snippet that can use this library (or some other python library) to achieve this?. &lt;/p&gt;&#xA;" OwnerUserId="43921" LastActivityDate="2020-04-04T14:02:30.337" Title="Creating Balanced Dataset Using Scikits" Tags="|python|scikit-learn|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28837" PostTypeId="2" ParentId="28835" CreationDate="2018-03-08T23:20:53.693" Score="0" Body="&lt;h2&gt;Activation functions&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;What activation function are you using?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common solution to the vanishing gradient issue in deep networks is to use activation functions that don't have this problem, i.e ReLU instead of sigmoid or tanh activation.&lt;/p&gt;&#xA;" OwnerUserId="15518" LastActivityDate="2018-03-08T23:20:53.693" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28838" PostTypeId="2" ParentId="28836" CreationDate="2018-03-08T23:52:32.877" Score="0" Body="&lt;p&gt;&lt;strong&gt;Simple Version&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate 2d classification dataset&#xA;&#xA;X, y = make_blobs(n_samples=100, centers=3, n_features=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Moons Classification Problem&lt;/strong&gt; (swirl like data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# generate 2d classification dataset&#xA;&#xA;X, y = make_moons(n_samples=100, noise=0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;make_circles&lt;/strong&gt; generates a binary classification problem with datasets that fall into concentric circles and few more are also there.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;References&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/datasets/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/datasets/plot_make_imbalance.html#sphx-glr-auto-examples-datasets-plot-make-imbalance-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Imbalanced learn&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="45256" LastEditDate="2018-05-14T11:04:51.857" LastActivityDate="2018-05-14T11:04:51.857" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28839" PostTypeId="1" CreationDate="2018-03-08T23:52:51.680" Score="1" ViewCount="210" Body="&lt;p&gt;Suppose that we have a dataset of n elements each element is composed of m sub elements, is there any existing algorithm that we can use to calculate the complexity of this dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean by the complexity anything that includes the following elements:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Diversity of dataset (number of distinct elements)&lt;/li&gt;&#xA;&lt;li&gt;Distance between each element&lt;/li&gt;&#xA;&lt;li&gt;Correlation between the elements of this dataset&lt;/li&gt;&#xA;&lt;li&gt;Size of the dataset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42029" LastActivityDate="2018-03-09T09:44:53.163" Title="Is there any existing algorithm or formula that can calculate the complexity of a dataset?" Tags="|data-mining|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28840" PostTypeId="1" AcceptedAnswerId="28861" CreationDate="2018-03-08T23:58:33.523" Score="6" ViewCount="26695" Body="&lt;p&gt;I'm using a dataset of movies and would like to group if a movie is the same across different retailers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Movie&lt;/strong&gt;: Beauty and the Beast&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Platforms&lt;/strong&gt;: Google, Netflix, iTunes, Amazon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have access to signals like:&#xA;Studio, Movie Name, Runtime, Language, Release Year, etc. However, in this case some movies, which are not the same and signals mentioned before, are not capable of finding the right match. I need to do what a human would do: Check Movie cover. Example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Beauty and the Beast &lt;a href=&quot;http://www.imdb.com/title/tt2771200/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.imdb.com/title/tt2771200/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Beauty and the Beast &lt;a href=&quot;http://www.imdb.com/title/tt6305650/?ref_=fn_al_tt_1&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.imdb.com/title/tt6305650/?ref_=fn_al_tt_1&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have access to the art image.&#xA;I'm using Python to do this comparison.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a library that can help me compare 2 images and determine if they are similar?&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="29169" LastEditDate="2019-06-07T17:08:12.427" LastActivityDate="2020-02-26T05:39:46.623" Title="Compare image similarity in Python" Tags="|python|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="28843" PostTypeId="2" ParentId="10077" CreationDate="2018-03-09T02:03:26.293" Score="5" Body="&lt;p&gt;Sounds like this is more extractive summarization if you are looking for key words. Here are a few papers which probably have implementations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1603.07252&quot; rel=&quot;noreferrer&quot;&gt;Neural Summarization by Extracting Sentences and Words&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1708.04439.pdf&quot; rel=&quot;noreferrer&quot;&gt;Extractive Summarization using Deep Learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1504.01255.pdf&quot; rel=&quot;noreferrer&quot;&gt;Semi-supervised Convolutional Neural Networks for&#xA;Text Categorization via Region Embedding&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, SpaCy (not affiliated) has a good &lt;a href=&quot;https://explosion.ai/blog/deep-learning-formula-nlp&quot; rel=&quot;noreferrer&quot;&gt;blog&lt;/a&gt; on the general architecture of of text extraction tasks.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-03-09T02:03:26.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28844" PostTypeId="2" ParentId="27953" CreationDate="2018-03-09T02:09:23.743" Score="5" Body="&lt;p&gt;This paragraph in &lt;a href=&quot;http://arxiv.org/abs/1610.03483&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; may be of help, methinks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7LLJ2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7LLJ2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9191" LastEditorUserId="9191" LastEditDate="2018-03-09T02:15:26.137" LastActivityDate="2018-03-09T02:15:26.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28845" PostTypeId="2" ParentId="10091" CreationDate="2018-03-09T03:05:24.763" Score="0" Body="&lt;p&gt;If you have a fairly large set of 'good' and 'bad' images, You can use a convolutional neural network (CNN) with a package like pytorch or tensorflow (pytorch is more pythonic). As I understand it, nowadays for image tasks, CNNs are what all the cool kids use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a package to preload models and you can take something small like vgg16 &lt;a href=&quot;http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;as shown here&lt;/a&gt; and then replace the last layer with a fully connected output of size 2 as shown toward the end of the tutorial, or download the notebook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its nice that You don't have care what 'good' and 'bad' are in this case (so it can be very true to your question), as long as you have like 10K images...&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-03-09T03:05:24.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28846" PostTypeId="2" ParentId="28778" CreationDate="2018-03-09T03:37:40.600" Score="1" Body="&lt;p&gt;&lt;strong&gt;Answering Question 1 of my own post:&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can I actually use that 300 neurons result in some other networks of mine? Say, in my LSTM, etc?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, I can, because the outcome of the hidden (orange in the picture) already represents the actual embedded vector. It should be now used in any other custom networks (LSTM, etc), NOT the one-hot encoded vector.  Using this orange vector will indeed greately reduce number of required weights if there are multiple custom networks under development.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice, connections between one-hot vector and the orange layer are represented by a special matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The matrix is special because each column (or rows, depending on your preferred notation) already represents these pre-activations in those 300 neurons - a response to a corresponding incoming one-hot vector.  So you just take a column, and treat it as the entire orange layer, which is super convenient&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More on that here: &lt;a href=&quot;https://datascience.stackexchange.com/a/29161/43077&quot;&gt;https://datascience.stackexchange.com/a/29161/43077&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Answering Question 2:&lt;/strong&gt; Indeed the encoded values are already in the result of a Hidden Layer (orange on on the picture). You can use them in any other algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The red output layer is only needed during training. The weights leading from InputVector to HiddenLayer is what you ultimately care about&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-01-20T18:29:21.037" LastActivityDate="2019-01-20T18:29:21.037" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28847" PostTypeId="1" AcceptedAnswerId="29721" CreationDate="2018-03-09T03:54:16.150" Score="1" ViewCount="3087" Body="&lt;p&gt;I am using a RandomForest for multiclass classification. I would like to use the oob_decision_function to explore precision/recall, but I don't understand the OOB results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using 25,000 trees (n_estimators=25000) and the possible class values are 0,1 or 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My results are (truncated to the first 10 rows):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_train = [1 2 1 1 0 0 2 2 2 2]&#xA;y_pred  = [2 2 2 1 0 2 2 2 2 2]&#xA;&#xA;oob_decision_function:&#xA;[[ 0.25377529  0.28080796  0.46541674]&#xA; [ 0.32162915  0.3250808   0.35329005]&#xA; [ 0.34463485  0.27709584  0.3782693 ]&#xA; [ 0.31091392  0.2982096   0.39087648]&#xA; [ 0.34932553  0.28632762  0.36434685]&#xA; [ 0.31535905  0.19570567  0.48893528]&#xA; [ 0.25472683  0.35845451  0.38681866]&#xA; [ 0.32521156  0.31721116  0.35757728]&#xA; [ 0.30706625  0.32703203  0.36590172]&#xA; [ 0.29785305  0.22490485  0.4772421 ]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dataset is not uniform: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class 0:  32% of samples&#xA;class 1:  27% of samples&#xA;class 2:  41% of samples&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The predictions don't seem to agree with the decision function. See for example the 4th sample: the prediction (y_pred[3]) is class 1, but the OOB values are 0.3109 (class 0), 0.2982 (class 1) and 0.3908 (class 2). Why is the prediction class 1 and not class 2? I thought after 25,000 trees the prediction should closely match the OOB probabilities, or am I not understanding OOB properly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;how are the OOB values calculated? I thought it was as follows: for&#xA;each of the 25,000 trees it creates a new bagged training set. Let's&#xA;say a given sample was not included (ie. out-of-bag) in 6000 of those&#xA;trees. When the sample is out-of-bag, the current tree is used to predict the&#xA;class for that sample. So for our given sample, it was estimated 6000&#xA;times and let's say the predictions were class 0 (1000 times), class&#xA;1 (2000 times) and class 2 (3000 times). Therefore the OOB values&#xA;would be 1000/6000 (class 0), 2000/6000 (class 1) and 3000/6000&#xA;(class 2). Is this correct?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the OOB values are calculated using the results of the predictions&#xA;from &lt;em&gt;individual&lt;/em&gt; trees (as explained above), but the output&#xA;predictions (ie. y_pred) are from the final &lt;em&gt;ensemble&lt;/em&gt; of trees. Is&#xA;this correct?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;should the OOB values converge to the cross validation values as the&#xA;number of trees increases? AFAIK &lt;em&gt;k&lt;/em&gt;-fold cross validation splits the&#xA;data into &lt;em&gt;k&lt;/em&gt; subsets and uses the entire forest on each subset.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;is there some sort of weighting being applied (e.g. if my data are&#xA;not uniformly distributed)?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47380" LastActivityDate="2018-03-30T17:37:39.980" Title="OOB decision function doesn't match prediction in scikit-learn RandomForest" Tags="|classification|scikit-learn|random-forest|multiclass-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28848" PostTypeId="2" ParentId="28847" CreationDate="2018-03-09T04:46:45.663" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;From The Docs&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The RandomForestClassifier is trained using bootstrap aggregation, where each new tree is fit from a bootstrap sample of the training observations&lt;code&gt; z_i = (x_i, y_i)&lt;/code&gt;. The &lt;code&gt;out-of-bag (OOB)&lt;/code&gt; error is the average error for each &lt;code&gt;z_i&lt;/code&gt; calculated using predictions from the trees that do not contain &lt;code&gt;z_i&lt;/code&gt; in their respective bootstrap sample. This allows the RandomForestClassifier to be fit and validated whilst being trained.. So we don't actually need to a Validation dataset or k-fold in this case&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(answer isn't strictly acc to your Question ordering)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Intuitive Understanding(The Docs is quite clear if not read below..)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Suppose our training data set is represented by T and suppose data set has M features (or attributes or variables).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;T = {(X1,y1), (X2,y2), ... (Xn, yn)} and Xi is input vector {xi1, xi2, ... xiM} and yi is the label (or output or class).&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Summary of RF:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random Forests algorithm is a classifier based on primarily two methods - bagging and random subspace method. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose we decide to have &lt;code&gt;S&lt;/code&gt; number of trees in our forest then we first create S datasets of &quot;same size as original&quot; created from random resampling of data in &lt;code&gt;T&lt;/code&gt; with-replacement (&lt;code&gt;n&lt;/code&gt; times for each dataset). This will result in &lt;code&gt;{T1, T2, ... TS}&lt;/code&gt; datasets. Each of these is called a bootstrap dataset. Due to &quot;with-replacement&quot; every dataset &lt;code&gt;Ti&lt;/code&gt; can have duplicate data records and Ti can be missing several data records from original datasets. This is called Bagging. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, &lt;code&gt;RF&lt;/code&gt; creates &lt;code&gt;S&lt;/code&gt; trees and uses &lt;code&gt;m (=sqrt(M) or =floor(lnM+1))&lt;/code&gt; random subfeatures out of &lt;code&gt;M&lt;/code&gt; possible features to create any tree. This is called &lt;code&gt;random subspace method. &lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for each &lt;code&gt;Ti&lt;/code&gt; bootstrap dataset you create a tree &lt;code&gt;Ki&lt;/code&gt;. If you want to classify some input data &lt;code&gt;D = {x1, x2, ..., xM}&lt;/code&gt; you let it pass through each tree and produce &lt;code&gt;S&lt;/code&gt; outputs (one for each tree) which can be denoted by &lt;code&gt;Y = {y1, y2, ..., ys}.&lt;/code&gt; Final prediction is a majority vote on this set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Out-of-bag error:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After creating the classifiers&lt;code&gt;(S trees)&lt;/code&gt;, for each&lt;code&gt;(Xi,yi)&lt;/code&gt;in the original training set i.e. &lt;code&gt;T&lt;/code&gt;, select all &lt;code&gt;Tk&lt;/code&gt;which does not include &lt;code&gt;(Xi,yi)&lt;/code&gt;. This subset, pay attention, is a set of boostrap datasets which does not contain a particular record from the original dataset. This set is called out-of-bag examples. There are n such subsets (one for each data record in original dataset T). &lt;code&gt;OOB classifier is the aggregation of votes ONLY over Tk such that it does not contain (xi,yi). &lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Out-of-bag estimate for the generalization error is the error rate of the out-of-bag classifier on the training set (compare it with known yi's).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Also your n_estimators is Quite High which will lead you to Overffiting&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also &lt;strong&gt;RF&lt;/strong&gt; isn't a good option when we have severe imbalanceness As Random forests are built on decision trees, and decision trees are sensitive to class imbalance. Therefore each tree will be biased in the same direction and magnitude (on average) by class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Just a side note,&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Bagging and Boosting&lt;/em&gt; might sound &lt;em&gt;similar&lt;/em&gt; but aren't &lt;em&gt;similar&lt;/em&gt;..&#xA;Xgboost can take care of Imbalanceness whereas RF's don't..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-09T05:27:29.520" LastActivityDate="2018-03-09T05:27:29.520" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28849" PostTypeId="1" AcceptedAnswerId="28872" CreationDate="2018-03-09T05:00:42.317" Score="4" ViewCount="1042" Body="&lt;p&gt;&lt;strong&gt;Short version: despite lots of reading, machine learning still feels like being a monkey in the dark. Any advice?&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;For background, I'm a researcher in computer science, in a field non-related to machine learning.&lt;/p&gt;&#xA;&lt;p&gt;I have been trying to get more proficient in machine learning*, yet no matter how much I read and fiddle with code/toy datasets, when I try to go to a harder problem, I always feel overwhelmed by the choices I need to make:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have to choose the algorithm: This is the part I typically  find the most straightforward;&lt;/li&gt;&#xA;&lt;li&gt;For said algorithm, I have to choose the objective function :  usually, many are applicable, and I find it difficult to gain a good intuition of what makes an objective function adapted in some cases rather than others, apart from the very classical ones for linear or logistic regression&lt;/li&gt;&#xA;&lt;li&gt;And then, I should devise the features: this still feels completely arcane to me, apart from using content-based features readily available in the data.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;I am under the impression that I have to &amp;quot;create&amp;quot; the tailored algorithm &lt;em&gt;and&lt;/em&gt; the data.&lt;/p&gt;&#xA;&lt;p&gt;Concerning the algorithm, I have spent some time into studying gradient boosting and the math behind it, to the point that I have a reasonably solid comprehension of &lt;em&gt;how it works&lt;/em&gt;, and an intuition of parameter tuning for simple datasets. However, that knowledge does not generalize.&lt;/p&gt;&#xA;&lt;p&gt;How are these issues typically approached? Are there any resources that can help?&lt;/p&gt;&#xA;&lt;p&gt;* By taking the Machine Learning  Coursera course and its more &lt;a href=&quot;https://www.youtube.com/playlist?list=PLA89DCFA6ADACE599&quot; rel=&quot;nofollow noreferrer&quot;&gt;in-depth version&lt;/a&gt;, reading more XgBoost-specific material (on its internals and parameter tuning and intuition), as well as playing with the Titanic dataset, and a housing market dataset.&lt;/p&gt;&#xA;" OwnerUserId="47497" LastEditorUserId="98307" LastEditDate="2020-08-18T16:24:33.393" LastActivityDate="2020-08-18T16:24:33.393" Title="Tactics to avoid feeling overwhelmed by machine learning" Tags="|machine-learning|xgboost|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="28850" PostTypeId="2" ParentId="28835" CreationDate="2018-03-09T05:18:35.373" Score="2" Body="&lt;p&gt;Edit: Definitely try Xavier initialization first, as the other answerer said.&lt;/p&gt;&#xA;&lt;p&gt;In other cases, where you have to increase the gradient manually...&lt;/p&gt;&#xA;&lt;p&gt;Gradient means rate of change of your loss function. If your loss function is not changing much with respect to certain weights, then changing those weights doesn't change your loss function.&lt;/p&gt;&#xA;&lt;p&gt;The weights just determine the type of linear combination from the previous layer. If the loss doesn't change when you change the linear combination, then you need to amplify the effect of increasing or decreasing the linear combination. So what you want is, whenever the weights increase, you want the linear combination to increase by more, and whenever the weights decrease, you want the linear combination to decrease by more.&lt;/p&gt;&#xA;&lt;p&gt;Let's say you multiplied a weight by a constant &lt;em&gt;k&lt;/em&gt;. Then if you increased that weight the linear combination would increase more, and if you decreased the weight then the linear combination would decrease more. So it must be that multiplying your weights by a constant &lt;em&gt;k&lt;/em&gt; &amp;gt; &lt;strong&gt;1&lt;/strong&gt; would increase the effects on the linear combination from changing the weights.&lt;/p&gt;&#xA;&lt;p&gt;If you multiply all your weights by a constant &lt;em&gt;k&lt;/em&gt;, then that's the same thing as multiplying the entire linear combination by &lt;em&gt;k&lt;/em&gt;. So you want to multiply the linear combination by &lt;em&gt;k&lt;/em&gt; before squishing it with your activation function.&lt;/p&gt;&#xA;&lt;p&gt;Your new activation function then would be this:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;activation = ReLU( linear combination x &lt;em&gt;k&lt;/em&gt; )&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;activation = 0 x (linear combination x &lt;em&gt;k&lt;/em&gt;), if linear combination &amp;lt;= 0&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;activation = 1 x (linear combination x &lt;em&gt;k&lt;/em&gt;), if linear combination &amp;gt; 0&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Compare to regular ReLU, which is:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;activation = 0 x linear combination, if linear combination &amp;lt;= 0&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;activation = 1 x linear combination, if linear combination &amp;gt; 0&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So you can see that no matter what the case is, you want to multiply the ReLU activation by &lt;em&gt;k&lt;/em&gt;. Specifically, you want to multiply the ReLU activation by &lt;em&gt;k&lt;/em&gt; &amp;gt; &lt;strong&gt;1&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;In practical terms, this increases the slope of the right side of the ReLU function.&lt;/p&gt;&#xA;" OwnerUserId="39308" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-09T22:08:29.773" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28852" PostTypeId="1" CreationDate="2018-03-09T05:44:51.620" Score="2" ViewCount="121" Body="&lt;p&gt;I'm working on a research problem where I need to perform classification for coarse prediction in a feature space and then fine grained regression for getting more precise values. I know that this way of regression should work. I also will essentially deal with feature maps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am thinking of using a 'stacked hourglass network'. Do I need to identify this by sheer experimentation or can someone intuitively remove some possibilities saying a particular architecture may not be suitable for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found stacked hourglass network to upscale and downscale essentially the heatmaps but now am confused with changing the model for sequential classification and regression task. Any clues would be welcomed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1603.06937&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stacked Hourglass Networks for Human Pose Estimation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47500" LastEditorUserId="29169" LastEditDate="2021-02-13T16:37:07.843" LastActivityDate="2024-02-03T09:03:30.427" Title="How to get intuitive understanding which deep learning architecture suits for my problem" Tags="|deep-learning|regression|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28853" PostTypeId="2" ParentId="5345" CreationDate="2018-03-09T06:10:04.737" Score="25" Body="&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://r-brain.io/&quot; rel=&quot;noreferrer&quot;&gt;RIDE&lt;/a&gt;&lt;/strong&gt; -   R-Brain IDE (RIDE) for R &amp;amp; Python, Other Data Science R IDEs, Other Data Science Python IDEs. Flexible layout. Multiple language support.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://jupyter.org/&quot; rel=&quot;noreferrer&quot;&gt;Jupyter notebook&lt;/a&gt;&lt;/strong&gt; -   The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed on a local desktop&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/jupyterlab/jupyterlab&quot; rel=&quot;noreferrer&quot;&gt;Jupyter lab&lt;/a&gt;&lt;/strong&gt; - An extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Radiant&lt;/strong&gt; – Open-source platform-independent browser-based interface for business analytics in R, based on the Shiny package and can be run locally or on a server.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.visualstudio.com/vs/features/rtvs/&quot; rel=&quot;noreferrer&quot;&gt;R Tools for Visual Studio (RTVS)&lt;/a&gt;&lt;/strong&gt; -&#xA;A free, open-source extension for Visual Studio 2017, RTVS is presently supported only in Visual Studio on Windows and not Visual Studio for Mac.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.getarchitect.io/&quot; rel=&quot;noreferrer&quot;&gt;Architect&lt;/a&gt;&lt;/strong&gt; -  Architect is an integrated development environment (IDE) that focuses specifically on the needs of the data scientist. All data science tasks from analyzing data to writing reports can be performed in a single environment with a common logic.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://www.displayr.com/features/&quot; rel=&quot;noreferrer&quot;&gt;displayr&lt;/a&gt;&lt;/strong&gt; - Simple and powerful. Automation by menu or code. Elegant visualizations. Instant publishing.&#xA;Collaboration. Reproducibility. Auto-updating. Secure cloud platform.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://atom.io/packages/rbox&quot; rel=&quot;noreferrer&quot;&gt;Rbox&lt;/a&gt;&lt;/strong&gt; - This package is a collection of several packages to run R via Atom editor.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Use below for more IDEs:&lt;/strong&gt;&lt;br /&gt;&#xA;&lt;a href=&quot;https://rkward.kde.org/&quot; rel=&quot;noreferrer&quot;&gt;RKWard&lt;/a&gt; - an easy to use and easily extensible IDE/GUI for R&lt;br /&gt;&#xA;&lt;a href=&quot;https://sourceforge.net/projects/tinn-r/&quot; rel=&quot;noreferrer&quot;&gt;Tinn-R&lt;/a&gt; - Tinn-R Editor - GUI for R Language and Environment&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://r.analyticflow.com/en/&quot; rel=&quot;noreferrer&quot;&gt;R AnalyticFlow&lt;/a&gt; - data analysis software that utilizes the R environment for statistical computing.&lt;br /&gt;&#xA;&lt;a href=&quot;https://rgedit.sourceforge.net/&quot; rel=&quot;noreferrer&quot;&gt;Rgedit&lt;/a&gt; - a text-editor plugin.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/jalvesaq/Nvim-R&quot; rel=&quot;noreferrer&quot;&gt;Nvim-R&lt;/a&gt; - Vim plugin for editing R code.&lt;br /&gt;&#xA;&lt;a href=&quot;https://rattle.togaware.com/&quot; rel=&quot;noreferrer&quot;&gt;Rattle&lt;/a&gt; - A Graphical User Interface for Data Mining using R.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.freecodecamp.org/turning-vim-into-an-r-ide-cd9602e8c217&quot; rel=&quot;noreferrer&quot;&gt;How to Turn Vim Into an IDE for R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14145" LastEditorUserId="29169" LastEditDate="2021-02-07T20:42:21.957" LastActivityDate="2021-02-07T20:42:21.957" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28856" PostTypeId="2" ParentId="28835" CreationDate="2018-03-09T07:21:26.287" Score="6" Body="&lt;p&gt;In order to fix the problem of vanishing gradients, you can use &lt;a href=&quot;http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization&quot; rel=&quot;noreferrer&quot;&gt;Xavier Initilization&lt;/a&gt;. Also, the implementation of Xavier Initialization in tensorflow can be done by following &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/layers/xavier_initializer&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; thread.&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-03-09T07:21:26.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28857" PostTypeId="2" ParentId="27694" CreationDate="2018-03-09T07:32:13.637" Score="6" Body="&lt;p&gt;I found the class 7846 (name=&quot;n00007846&quot;) is for person. To access to class description, read &lt;a href=&quot;http://image-net.org/download-API&quot; rel=&quot;noreferrer&quot;&gt;http://image-net.org/download-API&lt;/a&gt;. Even better, the following text files contain everything you ever need to understand the classes in ImageNet dataset (class = WordNet ID):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://image-net.org/archive/words.txt&quot; rel=&quot;noreferrer&quot;&gt;http://image-net.org/archive/words.txt&lt;/a&gt; maps between WordNet ID and words for all synsets&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://image-net.org/archive/gloss.txt&quot; rel=&quot;noreferrer&quot;&gt;http://image-net.org/archive/gloss.txt&lt;/a&gt; : maps between WordNet ID and glosses for all synsets&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;n00007846 maps to &lt;em&gt;person, individual, someone, somebody, mortal, soul&lt;/em&gt;. The corresponding gloss is: &lt;em&gt;a human being; &quot;there was too much for one person to do&quot;&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="47506" LastEditorUserId="45507" LastEditDate="2018-03-10T07:02:41.303" LastActivityDate="2018-03-10T07:02:41.303" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28858" PostTypeId="1" AcceptedAnswerId="28864" CreationDate="2018-03-09T07:55:29.750" Score="1" ViewCount="819" Body="&lt;p&gt;I am working on a card game for openai gym and currently I ask myself how to shape the reward function for it. One round of the game consists of each player picking a card from its hand, whereas not every card can be played depending on the card which has been played by one of the players before. For every set of played cards, there is a total order such that the player with the highest card wins the round.&lt;br&gt;&#xA;In the situation in which cards are rejected I want to give the agent some reward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of an invalid card, it is hard to say if that card is any nearer to one of the valid cards than any other. Also the agent should learn that this card is not playable at this point. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For completeness, the agent gets a discrete observation of everything it can remember of the game (its own cards, cards played in current round, cards played in past rounds, game mode (defines the total order of cards)). Then it should play a discrete action which either is a game mode in the beginning or a card during the round. Then it either gets a reward because its card got rejected or it gets a reward based on whether it wins the round or not. The game accounts a certain amount of points for a won round depending on the constellation of played cards in that round.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case any more details are required, just ask for them.&lt;/p&gt;&#xA;" OwnerUserId="24351" LastActivityDate="2018-03-10T09:06:37.407" Title="Card game for Gym: Reward shaping" Tags="|reinforcement-learning|openai-gym|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28859" PostTypeId="2" ParentId="28822" CreationDate="2018-03-09T08:11:27.533" Score="0" Body="&lt;p&gt;Neural networks can be used for classification and regression tasks. They are also used for transcription and clustering. Each of them can have their own characteristics. For classification tasks you may have different classes. The trained network should flag the output with the greatest value among other outputs which corresponds to the appropriate input. In regression tasks, there are different situations. Suppose that you are given the input feature containing age, gender, height and other related things and you try to find the size of foot and size of length of hands simultaneously. I mean you have numerous inputs and numerous outputs. As an other example, in detection tasks, you are asked the place of the desired object. If you want to do that using neural networks, the output of your system consists of the center of the detected item, the height and the width of the detected item, which &lt;em&gt;is&lt;/em&gt; a regression task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The interpretation of outputs for regression tasks is like estimating a real value, this is why linear activation functions are used as the last layers' activation. For classification tasks it is like finding the probability. The probability that your network finds depends on the activation function used in the last layer. If your classes are mutually exclusive, and each input should have just one label, for &lt;code&gt;OCR&lt;/code&gt; tasks each image of number refers to just a single number so classes are mutually exclusive, the activation should be &lt;code&gt;Softmax&lt;/code&gt; and it represents the probability of belonging to each class. If the input may contain different classes, e.g. an image containing both cat and dog, you can use &lt;code&gt;Sigmoid&lt;/code&gt; activation which each output represents the chance of existing of each class. You can also take a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/25315/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-09T08:11:27.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28860" PostTypeId="2" ParentId="28836" CreationDate="2018-03-09T09:03:36.687" Score="1" Body="&lt;p&gt;If I understood correctly, you want to &lt;em&gt;invent&lt;/em&gt; new negative samples to have a balanced dataset, because your current dataset only have positive samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the library you mentioned only re-samples a dataset where all classes are present. The &lt;em&gt;sample creation process&lt;/em&gt; is by definition data-specific, as there can be impossible attribute combinations, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to &lt;strong&gt;blindly&lt;/strong&gt; create combination of attribute values, you can use the stuff in module &lt;code&gt;random&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To create integer attributes:&lt;pre&gt;&#xA;def random_integer(min_val, max_val):&#xA;    return randint.randint(0, 9))&#xA;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;To create real-valued attributes:&lt;pre&gt;&#xA;def random_float(min_val, max_val):&#xA;    return random.uniform(min_val, max_val)&#xA;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;To create discrete attributes:&lt;pre&gt;&#xA;def random_discrete(value_list):&#xA;    return random.choice(value_list)&#xA;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;And then, with a loop, generate your negative samples in a loop:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;    def generate_negative_sample():&#xA;        return {'attr1': random_discrete(['a', 'b', 'c']),&#xA;                'attr2': random_float(-10., 10.),&#xA;                'attr3': random_integer(0, 4)}&#xA;&#xA;    random_samples = [generate_negative_sample() for _ in range(1000)]&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-09T09:03:36.687" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28861" PostTypeId="2" ParentId="28840" CreationDate="2018-03-09T09:38:58.817" Score="4" Body="&lt;p&gt;The problem you mention is not trivial. There is no library that out of the box will compare the pictures for you and give you a reliable similarity value. Therefore, you need to develop a system that works for both your problem and your dataset.&lt;/p&gt;&#xA;&lt;p&gt;Having said that, since neural networks work better than any other method for image recognition you can try:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Autoencoders: (In case your data is unlabeled) The idea is that the model extracts the features for you and then you omit the output layers so you have a new representation of your image but in a new feature space the model has learnt from data. Once your images are in this new feature space, you can use whatever technique to compute similarity. You can have an example on how to do this &lt;a href=&quot;https://towardsdatascience.com/find-similar-images-using-autoencoders-315f374029ea&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hash binary codes: (In case your data is labeled). This is a supervised method based on CNNs that seems to work quite nice to find relevant features in your images. Have a look at this &lt;a href=&quot;https://www.cv-foundation.org/openaccess/content_cvpr_workshops_2015/W03/papers/Lin_Deep_Learning_of_2015_CVPR_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Working with images is normally not quite straighforward and it requires some effort and experimentation to master these techniques. However, it is absolutely worth it and fun.&lt;/p&gt;&#xA;" OwnerUserId="45166" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-09T09:38:58.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28862" PostTypeId="2" ParentId="28839" CreationDate="2018-03-09T09:44:53.163" Score="2" Body="&lt;p&gt;well ... &lt;em&gt;&quot;if n elements each element is composed of m sub elements&quot;&lt;/em&gt; sound like you have a set of sets from discrete objects. If this is the case you can go through Combinatorial Analysis like Graph Theory. You can model it in two ways:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;Using &lt;a href=&quot;https://en.wikipedia.org/wiki/Hypergraph&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hypergraphs&lt;/a&gt;&lt;/strong&gt;: You have your elements as hyper-edges of a Hypergraph. This model lets you compute distances, diversities, sizes, and many more graph theoretical measures e.g. look at &lt;a href=&quot;http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1000385&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;, &lt;a href=&quot;http://toilers.mines.edu/RMCWiC/2014/sites/default/files/posters/csuconf-final20.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://www.math.iisc.ernet.in/~ifcam/new_avenue/Slides/Seminars/Ravindran.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Simple &lt;a href=&quot;https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Graphs&lt;/a&gt;&lt;/strong&gt;: You can create simple complete graphs from each sub-element and connect them to other &lt;a href=&quot;https://en.wikipedia.org/wiki/Complete_graph&quot; rel=&quot;nofollow noreferrer&quot;&gt;complete graph&lt;/a&gt; (forming &lt;a href=&quot;https://en.wikipedia.org/wiki/Clique_(graph_theory)&quot; rel=&quot;nofollow noreferrer&quot;&gt;cliques&lt;/a&gt;) of other sub-elements. See the example bellow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;element1 = [[a,b,c],[b,c,d],[e,f,g]]  &#xA;Graph = [(a,b),(b,c),(a,c),(b,c),(c,d),(b,d),(e,f),(e,g),(f,g)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;using this you will have a huge graph (network) of your object on which you can calculate &lt;a href=&quot;https://en.wikipedia.org/wiki/Network_science&quot; rel=&quot;nofollow noreferrer&quot;&gt;tones of structural and statistical measures&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume my thought were naive as your question is not clear and you need to provide more detailed example to get a more accurate answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good Luck!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-09T09:44:53.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28863" PostTypeId="2" ParentId="28800" CreationDate="2018-03-09T10:13:08.017" Score="1" Body="&lt;p&gt;Not sure if you've tried this already, but you might dig into &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/R-package/discoverYourData.html#feature-importance&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBOOST feature importance&lt;/a&gt; to establish how your model is making predictions, then do a more in-depth comparison between the two populations at those splits. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This isn't very different from what you've proposed, but it does make for a more focused analysis; you might have time to look at higher order interactions between the more important features. (I'm assuming it's not possible to cheat by training a classifier on your population B and comparing   how each model makes predictions!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/TeamHG-Memex/eli5&quot; rel=&quot;nofollow noreferrer&quot;&gt;ELI5&lt;/a&gt; and &lt;a href=&quot;https://github.com/marcotcr/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;LIME&lt;/a&gt; also come to mind for debugging feature importance and explaining model prediction.&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2018-03-09T10:13:08.017" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28864" PostTypeId="2" ParentId="28858" CreationDate="2018-03-09T10:35:38.863" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In reinforcement learning, you must set rewards so that they are maximised when the agent achieves the &lt;em&gt;goals&lt;/em&gt; of the problem. You should avoid trying to &quot;help&quot; the agent by setting interim rewards for things that &lt;em&gt;might&lt;/em&gt; help it achieve those goals.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For card rejection, if that is part of the game (i.e. it is valid to play a &quot;wrong&quot; card, and you lose your turn), then either no reward, or a negative one might suffice. Probably you should go with no reward, because the punishment would be in not winning that round anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If an invalid card cannot actually be played according to the rules of the game, and there is no &quot;pass&quot; move or equivalent, then you should not allow the agent to select it. Simply remove the action from consideration when making action selection. It is OK for the agent/environment to enforce this in a hard-coded fashion: A common way to do that, if your agent outputs a discrete set of action probabilities or preferences, is to filter that set by the environment's set of allowed actions, and renormalise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What if you want the agent to learn about correct card selection?&lt;/em&gt; Once you have decided that, then it becomes a learning objective and you can use a reward scheme. The action stops being &quot;play a card&quot; and becomes &quot;propose a card to play&quot;. If the proposal is valid, then the state change and reward for the round's play are processed as normal. If the proposal is not valid, then the state will not change, and the agent should receive some negative reward. Two things to note about this approach:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Turns in the game and time steps for the agent are now separate. That's not a problem, just be aware of the difference. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This will probably not encourage the agent to play better (in fact for same number of time steps, it will probably have learned less well how to win, because it is busy learning how to filter cards based on the observed features), but it will enable it to learn to propose correct cards without that being forced on it in a hard-coded fashion.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For winning a round, then you  &lt;em&gt;might&lt;/em&gt; want to reward the agent according to the game score it accumulates. Assuming that the winner of the overall game is the player with the highest score, this should be OK. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there is a caveat to that: &lt;em&gt;If&lt;/em&gt; by making certain plays the agent opens up &lt;em&gt;other&lt;/em&gt; players to score even higher, then simply counting how many points the agent gets is not enough to make it competitive. Instead, you want very simple sparse rewards: e.g. +1 for winning the game, 0 for drawing, -1 for losing. The main advantage of using RL approach in the first place is that the algorithms can and should be able to figure out how to use this sparse information and turn it into an optimal strategy. This is entirely how AlphaGo Zero works for instance - it has absolutely no help evaluating interim positions, it is rewarded only for winning or losing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you go with +1 win, -1 lose rewards, then you could maybe make players' current scores part of the state observation. That may help in decision making if there is an element of risk/gambling where a player behind in the scores might be willing to risk everything on the last turns just for a small chance to win overall.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-10T09:06:37.407" LastActivityDate="2018-03-10T09:06:37.407" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28865" PostTypeId="1" CreationDate="2018-03-09T10:49:50.060" Score="2" ViewCount="143" Body="&lt;p&gt;I am working on sales forecasting problem.I am able to provide data about which items got sold and not sold to the algorithm.&lt;strong&gt;How to provide algorithm information about items that are not present in the store.Is there any way we could encode this information in data or any other algorithms accepts this kind of information&lt;/strong&gt;.Currently, I am using Neural Networks and Random Forest to forecast &#xA;Sales.&lt;/p&gt;&#xA;" OwnerUserId="37626" LastActivityDate="2024-01-30T09:03:55.473" Title="Store's unseen items sales forecasting" Tags="|machine-learning|predictive-modeling|statistics|probability|forecast|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28868" PostTypeId="1" CreationDate="2018-03-09T12:13:45.883" Score="2" ViewCount="65633" Body="&lt;p&gt;While converting json file to csv,I got the above error&#xA;My code is shown below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import csv&#xA;data = open(&quot;SRInFlow.json&quot;,encoding=&quot;utf8&quot;)&#xA;rows = json.loads(data.read())&#xA;r=rows.values()&#xA;fieldnames = [b'dateTimeValues', b'timeSeries']&#xA;with open('C:/DeepLearning/tensorflowpoc/Codes/SRInFlow.csv', 'w') as f:&#xA;    dict_writer = csv.DictWriter(f, fieldnames=fieldnames)&#xA;    dict_writer.writeheader()&#xA;    for d in r:&#xA;  ---&amp;gt;    dict_writer.writerow(d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error displays in the last line. How to rectify the error? Thanks in advance &lt;/p&gt;&#xA;" OwnerUserId="44688" LastEditorUserId="28175" LastEditDate="2018-03-09T12:14:47.597" LastActivityDate="2018-03-09T12:41:26.247" Title="AttributeError: 'str' object has no attribute 'keys'" Tags="|python|json|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28869" PostTypeId="2" ParentId="28868" CreationDate="2018-03-09T12:41:26.247" Score="3" Body="&lt;p&gt;This error is because you are initialising a dictionary writer &lt;code&gt;dict_writer = csv.DictWriter(f, fieldnames=fieldnames)&lt;/code&gt; but then you are passing a normal row containing only the values &lt;code&gt;r=rows.values()&lt;/code&gt; -&gt; &lt;code&gt;for d in r:&lt;/code&gt; -&gt;&lt;code&gt;dict_writer.writerow(d)&lt;/code&gt;, when the writer expects a dictionary including the keys. That is what the error means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know how your JSON looks but you can try something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import json&#xA;import csv&#xA;&#xA;data = '{&quot;dateTimeValues&quot;: [1,3,4], &quot;timeSeries&quot;: [0,0,0]}'&#xA;rows = json.loads(data)&#xA;&#xA;r=zip(*rows.values())&#xA;&#xA;fieldnames = ['dateTimeValues', 'timeSeries']&#xA;with open('c:/Users/jsz.P3-GROUP/Desktop/shit.csv', 'w') as f:&#xA;    dict_writer = csv.writer(f)&#xA;    dict_writer.writerow(fieldnames)&#xA;    for d in r:&#xA;        dict_writer.writerow(d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This initialises a simple csv writer &lt;code&gt;dict_writer = csv.writer(f)&lt;/code&gt;(not dictionary), then inputs the fieldnames as a normal row for the header &lt;code&gt;dict_writer.writerow(fieldnames)&lt;/code&gt; and finally inserts only the values as in your example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that for my json string I had to transpose the values first as in &lt;code&gt;r=zip(*rows.values())&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-09T12:41:26.247" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28870" PostTypeId="1" CreationDate="2018-03-09T13:50:23.537" Score="0" ViewCount="137" Body="&lt;p&gt;I have a sample dataset as below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data_1,Data_2,Data_3,Data_4,Data_5,Data_6,Data_7,Data_8&#xA;93.966589,6.033411,93.966589,6.033411,93.966589,6.033411,1296.257118,1044.073206&#xA;93.966589,6.033411,93.966589,6.033411,93.966589,6.033411,1296.257118,1044.073206&#xA;93.966589,6.033411,93.966589,6.033411,93.966589,6.033411,1296.257118,1044.073206&#xA;93.966589,6.033411,93.966589,6.033411,93.966589,6.033411,1296.257118,1044.073206&#xA;93.966589,6.033411,93.966589,6.033411,93.966589,6.033411,1296.257118,1044.073206&#xA;96.853313,3.146687,96.853313,3.146687,96.853313,3.146687,1233.525837,1076.147926&#xA;96.853313,3.146687,96.853313,3.146687,96.853313,3.146687,1233.525837,1076.147926&#xA;96.853313,3.146687,96.853313,3.146687,96.853313,3.146687,1233.525837,1076.147926&#xA;96.853313,3.146687,96.853313,3.146687,96.853313,3.146687,1233.525837,1076.147926&#xA;96.853313,3.146687,96.853313,3.146687,96.853313,3.146687,1233.525837,1076.147926&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has two data points repeated in a uniform way. As per the clustering logic, the ideal number of clusters for this dataset should be two. Is this correct or should the identical values be considered duplicate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does clustering work better for heterogeneous data or it should work for these kind of data too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="40370" LastEditorUserId="46505" LastEditDate="2018-04-09T05:06:41.760" LastActivityDate="2018-04-09T05:06:41.760" Title="KMeans Clustering on Heterogeneous Dataset" Tags="|machine-learning|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28871" PostTypeId="1" CreationDate="2018-03-09T13:58:32.023" Score="1" ViewCount="72" Body="&lt;p&gt;As a personal project, I'm trying to build a classifier which attempts to predict the metacritic score of a game based purely on its cover. I figured it would be a fun project to learn Keras image classification with (2D-Convolution to be more specific).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm definitely new to this so if I've made any &lt;em&gt;rookie mistakes&lt;/em&gt;, please tell me. Here are some notes before I post the CNN:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I've written a Metacritic &lt;em&gt;Scrubber&lt;/em&gt; which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Games with square cover art (DLC, non-retail, etc) have been omitted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model I've built has been based on the work by Iwana et al in this paper:&#xA;&lt;a href=&quot;https://arxiv.org/abs/1610.09204&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1610.09204&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv2D(64, (2, 3),activation='relu', input_shape=(123,98,3)))&#xA;model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))&#xA;&#xA;model.add(Conv2D(128, (2, 3),activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))&#xA;&#xA;model.add(Conv2D(256, (2, 3),activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))&#xA;&#xA;model.add(Conv2D(256, (2, 3),activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))&#xA;&#xA;model.add(Conv2D(256, (2, 3),activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))&#xA;&#xA;model.add(Flatten())&#xA;&#xA;model.add(Dense(720, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(720, activation='relu'))&#xA;model.add(Dense(720, activation='relu'))&#xA;model.add(Dense(10, activation='softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.001,momentum=0.1,decay=0.0005,nesterov=True), metrics=['accuracy'])&#xA;model.summary()&#xA;&#xA;#Save output to file&#xA;csv_logger = CSVLogger('training.log')&#xA;#Save best model when possible&#xA;checkpoint = ModelCheckpoint('training_model_best.hdf5', monitor='val_acc', verbose=1, save_best_only=True, mode='max')&#xA;&#xA;model.fit(np.array(images), np.array(labels), epochs=225, batch_size=10,&#xA;    validation_split=0.50,verbose=2,shuffle=True,callbacks=[csv_logger,checkpoint])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using quite a harsh validation split of 0.5 because I really wanted to avoid overfitting. But regardless of whether I use between 0.2 - 0.5, I typically reach a validation accuracy of approximately 35%, which is pretty disappointing. A lot of the hyper-parameters I have no idea what to set to however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry for the long post. Ultimately, my result is &lt;em&gt;a little&lt;/em&gt; better than random guessing but I'm hoping through some pointers, I can bring this accuracy up a bit. I wonder if actually, the cover files I've downloaded are too small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading.&lt;/p&gt;&#xA;" OwnerUserId="45766" LastEditorUserId="28175" LastEditDate="2018-03-09T15:42:10.737" LastActivityDate="2018-03-09T15:48:56.973" Title="Improvements to video-game cover CNN classifier (keras)" Tags="|keras|image-classification|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28872" PostTypeId="2" ParentId="28849" CreationDate="2018-03-09T13:58:35.333" Score="3" Body="&lt;p&gt;We have to climb up a steep learning curve when we learn about machine learning. Your question is quite general: One of the tactics I use when learning is divide and conquer. Get some coarse overview about the whole area, then pick some particular area and dig deeper only there.&lt;br&gt;&#xA;Perhaps the question is too general, the best tactic may vary and depend on the area you address. &#xA;But I am not sure if learning the math is always helpful (although it may always be interesting for those who care). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The algorithms can often be applied in a black box approach, and it may be sometimes not necessary to understand an algorithm in math terms (white box), but sufficient to know it's function, strengths and weaknesses (black box).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may be the first one that tests that algorithm for the domain, so pure experimentation is useful in the end. &lt;/p&gt;&#xA;" OwnerUserId="21163" LastActivityDate="2018-03-09T13:58:35.333" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28873" PostTypeId="2" ParentId="28870" CreationDate="2018-03-09T14:18:17.867" Score="0" Body="&lt;p&gt;Technically, it is allowed to duplicate observations (records) to cluster, yet it is far from correct. &#xA;What I'd do is first apply grouping similar records and use unique observations with a weight being their frequency.&#xA;Weight of observation can be used in many clustering algorithms.&lt;/p&gt;&#xA;" OwnerUserId="47522" LastActivityDate="2018-03-09T14:18:17.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28874" PostTypeId="1" AcceptedAnswerId="28897" CreationDate="2018-03-09T15:36:05.657" Score="21" ViewCount="33344" Body="&lt;p&gt;I'm a little bit new to machine learning.&lt;/p&gt;&#xA;&lt;p&gt;I am using a neural network to classify images. There are two possible classes. I am using a &lt;code&gt;Sigmoid&lt;/code&gt; activation at the last layer so the scores of images are between 0 to 1.&lt;/p&gt;&#xA;&lt;p&gt;I expected the scores to be sometimes close to 0.5 when the neural net is not sure about the class of the image, but all scores are either 1.0000000e+00 (due to rounding I guess) or very close to zero (for exemple 2.68440009e-15). In general, is that a good or bad thing ? How can this behaviour be avoided?&lt;/p&gt;&#xA;&lt;p&gt;In my use case I wanted to optimize for recall by setting a lower threshold but this has no impact because of what I described above.&lt;/p&gt;&#xA;&lt;p&gt;More generally, how can I minimize the number of false negatives when in training the neural net only cares about my not ad-hoc loss ? I am ok with decreasing accuracy a little bit to increase recall.&lt;/p&gt;&#xA;" OwnerUserId="47078" LastEditorUserId="47078" LastEditDate="2021-03-04T10:08:34.680" LastActivityDate="2021-04-23T19:13:58.213" Title="How to maximize recall?" Tags="|machine-learning|neural-network|deep-learning|keras|image-classification|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28875" PostTypeId="1" CreationDate="2018-03-09T15:48:01.790" Score="1" ViewCount="52" Body="&lt;p&gt;I'm trying to implement what is explained in a paper on audio signal processing. The guys who wrote this paper tried a Convolutional Neural Network and here is how they explain it :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The CNN are constructed using two different building blocks: block A consists of two layers with &lt;code&gt;32 3*3&lt;/code&gt; filters, and block &lt;code&gt;B&lt;/code&gt; consists of two layers with &lt;code&gt;64 3*3&lt;/code&gt; filters ; both in combination with batch normalization, and each followed by a &lt;code&gt;3*3&lt;/code&gt; max-pooling layer and a drop-out layer.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://ismir2017.smcnus.org/wp-content/uploads/2017/10/123_Paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;see the full paper here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They state that they apply this model to &lt;code&gt;84*9&lt;/code&gt; images (actually it's data from a spectogram, &lt;code&gt;84&lt;/code&gt; is a number of frequency bins, &lt;code&gt;9&lt;/code&gt; is the number of frames)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have some trouble to find out &lt;strong&gt;precisely&lt;/strong&gt; the different layers to add to the model :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Does the Batch Normalization layer occurs after &lt;strong&gt;each&lt;/strong&gt; convolutional layer ?&lt;/li&gt;&#xA;&lt;li&gt;Does the max-pooling layer occurs after &lt;strong&gt;each&lt;/strong&gt; convolutional layer ?&lt;/li&gt;&#xA;&lt;li&gt;Does the drop-out layer occurs after &lt;strong&gt;each&lt;/strong&gt; convolutional layer ?&lt;/li&gt;&#xA;&lt;li&gt;Is there any implicit padding ? Implicit stride ?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;My guess is : if they say they are using Batch Normalization, max-pooling and drop-out, then maybe it is implemented after each Convolutional layer (the four of them) (I've often seen that it's applied after each layer)&#xA;But with those layers it's impossible to have an input_shape of &lt;code&gt;(84, 9)&lt;/code&gt; with no padding ...&#xA;I'm quite stuck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot :)&lt;/p&gt;&#xA;" OwnerUserId="47526" LastEditorUserId="1330" LastEditDate="2021-08-12T15:27:13.550" LastActivityDate="2021-08-12T15:27:13.550" Title="Understand the shape of this Convolutional Neural Network" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28876" PostTypeId="2" ParentId="28871" CreationDate="2018-03-09T15:48:56.973" Score="1" Body="&lt;p&gt;Here are some points to consider:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Good!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This also should be fine.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would normalise the values between 0-1 (&lt;code&gt;norm_values = values/255&lt;/code&gt;). Regarding the size of the files I don't think this should be a problem. For example, Cifar-10 are 32x32 or the average image resolution on ImageNet is 469x387 pixels although most approaches resize them to 256x256. I think your size should be fine.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;4) Games with square cover art (DLC, non-retail, etc) have been omitted.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not much to add here.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.&#xA;  The model I've built has been based on the work by Iwana et al in this paper: &lt;a href=&quot;https://arxiv.org/abs/1610.09204&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1610.09204&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here is where I actually think we have some problems. First, your dataset is quite small. 3816 reduced to 1908 when splitting train/test... This is not good especially with such structure. In the paper you mentioned, they are using a network with around 2.5M parameters and they used ~137K samples. Your 1908 dataset seems tiny compared and you are using the same model structure... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion your model is not able to do better, simply put. It doesn't matter what parameters you choose you need (a lot) more data. You might try reducing the size of your network, also create more data using noise addition, mirror samples, etc... and see if these help somehow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, apart from the size, you have no way to tell if your data is representative enough so that any model can learn from it. Therefore, 35% accuracy is as good as any other value I am afraid. Anyway, your mission seemed to be able to complete your experiment and you did. And you learnt a lot about limitations of deep models, so I'd say: good work!&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-09T15:48:56.973" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28877" PostTypeId="1" AcceptedAnswerId="28878" CreationDate="2018-03-09T15:56:37.183" Score="2" ViewCount="639" Body="&lt;p&gt;I am trying to build a simple multi layer perceptron Neural Network in Java, but apparently my calculations are off. I am looking for a beginner-level tutorial which can help me to understand how to properly calculate forward and backward pass, preferably with examples.&lt;/p&gt;&#xA;" OwnerUserId="47432" LastActivityDate="2019-09-27T16:02:11.753" Title="Neural Network beginner level tutorial" Tags="|machine-learning|neural-network|" AnswerCount="3" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28878" PostTypeId="2" ParentId="28877" CreationDate="2018-03-09T15:58:59.257" Score="1" Body="&lt;p&gt;One of articles which helped me a lot is : &lt;a href=&quot;https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Step by Step Backpropagation Example&lt;/a&gt; by &lt;strong&gt;Matt Mazur&lt;/strong&gt;. It covers forward and backward pass of MLP. I hope that helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another great source is &lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.deeplearningbook.org/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45821" LastActivityDate="2018-03-09T15:58:59.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28879" PostTypeId="2" ParentId="28877" CreationDate="2018-03-09T16:18:42.227" Score="0" Body="&lt;p&gt;Apart the mentioned resources, this also might be of help: &lt;a href=&quot;https://www.cse.unsw.edu.au/~cs9417ml/MLP2/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLP Java example&lt;/a&gt;. It's from the University of Sydney and includes theory and a Java implementation. &lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-09T16:18:42.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28880" PostTypeId="1" CreationDate="2018-03-09T16:31:13.517" Score="3" ViewCount="4076" Body="&lt;p&gt;I read a couple of posts (like &lt;a href=&quot;https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;) about the encoder-decoder architecture and their implementation. But I still don't understand a couple of things.&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the difference between a basic CNN or RNN and encoder decoder ?&#xA;Are there some properties that the encoder and decoder need to satisfy ? As far as I understood the encoder encodes the input in another dimension and creates a context vector. Later this context vector is decoded by the decoder.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I know that their are different types of encoders and decoders, what is the most simple architecture to implement ?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I use predefined word embeddings like word2vect in the encoder(if i have text as an input) ?&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47528" LastEditorUserId="98307" LastEditDate="2020-08-15T19:59:49.613" LastActivityDate="2020-08-15T19:59:49.613" Title="Basic encoder-decoder architecture" Tags="|keras|rnn|cnn|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28881" PostTypeId="1" CreationDate="2018-03-09T16:42:03.557" Score="4" ViewCount="4820" Body="&lt;p&gt;While I understand the concept of dilated convolution as there are lot of papers explaining about it, I have heard less about dilated pooling. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Can someone explain what it is? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What is the internal implementation of it? Preferably with an example.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46628" LastEditorUserId="45507" LastEditDate="2018-03-10T07:05:44.910" LastActivityDate="2020-07-02T15:19:37.843" Title="What is dilated pooling and how it works mathematically?" Tags="|machine-learning|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28882" PostTypeId="2" ParentId="28874" CreationDate="2018-03-09T16:43:37.530" Score="12" Body="&lt;p&gt;To answer the last question, suppose that you have a binary classification problem. It is customary to label the class as positive if the output of the &lt;code&gt;Sigmoid&lt;/code&gt; is more than &lt;code&gt;0.5&lt;/code&gt; and negative if it's less than &lt;code&gt;0.5&lt;/code&gt;. For increasing recall rate you can change this threshold to a value less than &lt;code&gt;0.5&lt;/code&gt;, e.g. &lt;code&gt;0.2&lt;/code&gt;. For tasks which you may want a better precision you can increase the threshold to bigger value than &lt;code&gt;0.5&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the first part of your question, it highly depends on your data and its feature space. There are problems which the data is linearly separable in higher dimensions which means you can easily employ just a single neuron for classifying the data by a single hyper-plane. If it has happened that you have such a good accuracy you can not say anything unless you try to find the value of cross validation error. By interpreting the difference between the value of training data, and cross-validation or maybe test data, you can figure out whether your classifier performs well or not.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-09T16:43:37.530" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="28883" PostTypeId="1" AcceptedAnswerId="28906" CreationDate="2018-03-09T16:53:01.060" Score="2" ViewCount="1155" Body="&lt;p&gt;I have a dataset of ~5000 elements represented by vectors composed by ~30 binary values (0 or 1) &#xA;on which I am performing binary classification with SVM with linear kernel (I use the Scikit learn lib).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For curiosity, I tried to add an extra feature that consists in a AND between two others (remember that all my features are boolean). The result was that the performance of the SVM improved. I was surprised by this improvement because the AND operation is equivalent to a multiplication, therefore I would expect that my SVM, as every linear classifier, was somehow naturally already taking into account mutiplications between features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is wrong with my theoretic understanding of SVM ?&lt;/p&gt;&#xA;" OwnerUserId="43650" LastEditorUserId="28175" LastEditDate="2018-03-09T17:00:00.423" LastActivityDate="2018-03-09T22:43:27.617" Title="Why adding combinations of features would increase performance of linear SVM?" Tags="|classification|feature-selection|svm|feature-construction|normal-equation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28884" PostTypeId="1" AcceptedAnswerId="28894" CreationDate="2018-03-09T16:56:42.007" Score="2" ViewCount="2352" Body="&lt;p&gt;I'm trying to implement an algorithm to find the minimal value of a function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before moving to sigmoid activation functions, i'm trying to understand linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have two vectors:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x=[1,2,3,4,5,6,7,8,9,10,11,12]&#xA;&#xA;y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wPlXv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wPlXv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Between these points, I would like to add a linear separator with least squares.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say I have some imperfect linear function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$f(x)=0.026x+2.3$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I know, there are two ways to find this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w = (X^TX)^{-1}X^Ty$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and a gradient descent algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w^{new} = w^{old} - \nu \frac{dy}{dx}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although for linear models finding derivative is trivial, thus second method is not necessary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now i've used the first equation on the vectors, in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the output was irrelevant:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then i've tried using the second method for 500 iterations, in Python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i in range(1,5000):&#xA;    x_old = x_new&#xA;    x_new = x_old - v*dydx&#xA;    print(&quot;x_new = {0} - {1}({2}) = {3}&quot;.format(x_old, v, dydx, x_new))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, i'm not sure how to know when it reaches a convergence point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?&lt;/p&gt;&#xA;" OwnerUserId="47359" LastEditorUserId="29169" LastEditDate="2019-06-09T19:05:50.453" LastActivityDate="2019-06-09T19:05:50.453" Title="Finding optimal weights for models" Tags="|neural-network|logistic-regression|linear-regression|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="28885" PostTypeId="2" ParentId="28880" CreationDate="2018-03-09T16:57:37.013" Score="2" Body="&lt;p&gt;Encoder and decoder are highly overloaded terms. As a generic definition, an encoder-decoder neural architecture has a part of the network called &quot;encoder&quot; that receives an input and generates a &lt;em&gt;code&lt;/em&gt; (i.e. expresses the input in a different representation space) and another part called &quot;decoder&quot; that takes a given &lt;em&gt;code&lt;/em&gt; and converts it to the output representation space. Normally, the dimensionality of the &lt;em&gt;code&lt;/em&gt; is much less than that of the input/output representation spaces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In principle, the architecture of the encoder and the decoder is arbitrary, i.e. they can be CNN's, RNN's, multilayer perceptrons or whatever other thing. This highly depends on the task. This should answer question (1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are two contexts where encoder-decoders network organization is usual:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;image autoencoders: normally, the encoder is a CNN and the decoder is the analogous deconvolution. &lt;/li&gt;&#xA;&lt;li&gt;sequence-to-sequence tasks: normally, both encoder and decoder are GRUs or LSTMs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;As you mention word embeddings and a blog post about text summarization, I guess your problem relates to NLP, so you may start with the &lt;a href=&quot;https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras blog seq2seq tutorial&lt;/a&gt;. This should answer question (2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The appropriateness of using pre-trained word embeddings depends on the task and the amount of training data you have. When you don't have a lot of data, using pre-trained embeddings is a form of &lt;em&gt;data augmentation&lt;/em&gt; to get better results. For translation, it is not frequent to see pre-trained word embeddings. This should answer question (3).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastEditorUserId="14675" LastEditDate="2018-03-09T17:28:07.263" LastActivityDate="2018-03-09T17:28:07.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28886" PostTypeId="2" ParentId="28883" CreationDate="2018-03-09T16:59:38.193" Score="0" Body="&lt;p&gt;Suppose that you have a learning problem and it's just for fitting a function which depends on only one feature, and the function to be predicted is a quadratic shape. you can have a good performance by just having the input feature if you use linear &lt;code&gt;SVM&lt;/code&gt; but it will have errors. adding extra features, polynomial features as the input may be useful but it adds more complexity to the classifier and it causes more computational overheads. By the way that's true. You can have better estimate of functions by adding the high order polynomials of the features in hand. I don't know if you are familiar with normal equation or not but what that does is to add high order polynomials to better fit the function which makes the current data. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-09T16:59:38.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28888" PostTypeId="2" ParentId="28880" CreationDate="2018-03-09T17:03:31.640" Score="1" Body="&lt;p&gt;The autoencoder which uses convolutional layers is essentially a subset of CNN architectures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea of an encoder is exactly as you stated you want to go from a space $\mathbb{R}^n$ to $\mathbb{R}^m$ where $m&amp;lt;n$. There are a lot of ways to compress information in such a way and these techniques used to be called compressed sensing. The idea was then to find automated ways to generate this mapping from n-dimensions to m-dimensions. As the name suggests it is an &lt;strong&gt;auto&lt;/strong&gt;encoder. Lately the most popular method for doing this is using a CNN let's just refer to this specific type of autoencoder as an autoencoder for simplicity, but remember there are other ways of doing this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The autoencoder has a special structure where we will constrain the number of parameters at the center layer. The encoder is the part of the network which compresses the information into the m-dimensional space. We then use a decoder to reconstruct the input from the compressed data. An autoencoder is trained by feeding the same input and output. We want the network to recreate the input as closely as possible. If the output is equal to the input then we have perfect reconstruction and all the information contained in the input is contained. When we constrain the dimensions there will always be some information loss. We will want to minimize this loss. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of a vanilla autoencoder&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_img = Input(shape=(28, 28, 1))  # adapt this if using `channels_first` image data format&#xA;&#xA;x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)&#xA;x = MaxPooling2D((2, 2), padding='same')(x)&#xA;x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)&#xA;x = MaxPooling2D((2, 2), padding='same')(x)&#xA;x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)&#xA;x = MaxPooling2D((2, 2), padding='same')(x)&#xA;encoded = MaxPooling2D((2, 2), padding='same')(x)&#xA;&#xA;# at this point the representation is compressed&#xA;&#xA;x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)&#xA;x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)&#xA;x = UpSampling2D((2, 2))(x)&#xA;x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)&#xA;x = UpSampling2D((2, 2))(x)&#xA;x = Conv2D(16, (3, 3), activation='relu')(x)&#xA;x = UpSampling2D((2, 2))(x)&#xA;decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)&#xA;&#xA;autoencoder = Model(input_img, decoded)&#xA;autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;What happens if the network just learn the identity function!!&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For this problem the denoising autoencoder was conceived. Add white Gaussian noise to the input and train using the original image at the output. This forces the network to learn the underlining distribution of the images and not just copy them to the output. You can use the same network as above. The noise can be added as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;noise_factor = 0.5 &#xA;x_train_noisy = x_train_reshaped + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train_reshaped.shape) &#xA;x_test_noisy = x_test_reshaped + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test_reshaped.shape) &#xA;x_train_noisy = np.clip(x_train_noisy, 0., 1.) &#xA;x_test_noisy = np.clip(x_test_noisy, 0., 1.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-09T17:03:31.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28889" PostTypeId="5" CreationDate="2018-03-09T17:14:20.317" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2018-03-09T17:14:20.317" LastActivityDate="2018-03-09T17:14:20.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28890" PostTypeId="4" CreationDate="2018-03-09T17:14:20.317" Score="0" Body="Convolutional Neural Networks (CNN, also called ConvNets) are a tool used for classification tasks and image recognition.  The name giving first step is the extraction of features from the input data. " OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-03-10T20:18:18.213" LastActivityDate="2018-03-10T20:18:18.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28891" PostTypeId="2" ParentId="1002" CreationDate="2018-03-09T17:16:24.927" Score="0" Body="&lt;p&gt;R has &lt;a href=&quot;https://alternativeto.net/list/2063/guis-to-save-from-typing-r-code/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pnc dialogue GUIs&lt;/a&gt; like SPSS. They print R code so you can learn and combine their efforts. I would recommend BlueSky for it's dialogues for everything and rattle. While these software are great for EDA, statistics and visualization, machine learning they don't do well. &lt;/p&gt;&#xA;" OwnerUserId="30507" LastActivityDate="2018-03-09T17:16:24.927" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28892" PostTypeId="2" ParentId="28884" CreationDate="2018-03-09T17:16:58.100" Score="1" Body="&lt;p&gt;Also you have misunderstood the &lt;code&gt;-1&lt;/code&gt;..&#xA;It's not the exponent rather the &lt;strong&gt;symbol of inverse of a Matrix&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also use &lt;code&gt;.T&lt;/code&gt; for Transpose...(a bit Pythonic Convenience)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to help you out,&#xA;Search &lt;code&gt;np.linalg.inv&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your second Query,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer &lt;a href=&quot;https://datascience.stackexchange.com/q/28411/35644&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the &lt;code&gt;5th decimal place..&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps..&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-09T17:16:58.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28893" PostTypeId="2" ParentId="28883" CreationDate="2018-03-09T17:26:03.760" Score="0" Body="&lt;p&gt;Intuitively, the linear classifier you built is only trying to find a local minimum meaning he is trying a lot of different operation but it is not looking at the whole space of possibilities, if an &quot;and&quot; operation is relevent to your problem and you add it as a feature it will improve the SVM performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fallback of adding features like this is that it can help the model overfit : it is easier to find a linear bound that separates well the data if the input space has a lot of dimensions.&lt;/p&gt;&#xA;" OwnerUserId="47078" LastActivityDate="2018-03-09T17:26:03.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28894" PostTypeId="2" ParentId="28884" CreationDate="2018-03-09T17:26:07.160" Score="2" Body="&lt;p&gt;In this case your feature matrix $X$ has a single dimension. Each point in your graph has a $y$ value that depends only on 1 value of $x$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ok let's go through the code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x=[1,2,3,4,5,6,7,8,9,10,11,12]&#xA;y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's convert these to matrices. We will also add a column of 1's to the end of the $X$ matrix. This will be used to train the bias value.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;temp = np.ones((len(x), 2))&#xA;temp[:,0] = np.asarray(x)&#xA;x = temp&#xA;y = np.asarray(y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we will calculate our weights as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w = (X^TX)^{-1}X^Ty$&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([ 0.01174825,  2.30530303])&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Look at the dimensions of our weights vector. It only has 2 values. One value associated with $x$, the first column of our $X$ matrix, and a bias, associated with the 1's column that we added. The equation of this line is described as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = 0.01174825 x_1 + 2.30530303$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qUSfY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qUSfY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see that this line indeed describes the data pretty well for a linear regression. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Deeper&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;However, your data looks like it would be better fit using a polynomial. You should try &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = w_1 x_1^2 + w_2 x_1 + b$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this add a new feature in the $X$ matrix which corresponds to $x^2$. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x=[1,2,3,4,5,6,7,8,9,10,11,12]&#xA;y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35] &#xA;temp = np.ones((len(x), 3))&#xA;temp[:,0] = np.power(np.asarray(x), 2)&#xA;temp[:,1] = np.asarray(x)&#xA;x = temp&#xA;y = np.asarray(y)&#xA;&#xA;w = np.matmul(np.matmul(np.linalg.inv(np.matmul(np.transpose(x), x)), np.transpose(x)), y)&#xA;&#xA;[![xx = range(1,15,1)&#xA;yy = \[0\]*len(xx)&#xA;for ix, i in enumerate(xx):&#xA;    yy\[ix\] = w\[0\]*i**2 + w\[1\]*i + w\[2\]][2]][2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wKJ7u.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wKJ7u.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Even deeper&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;And going one step further by adding the $x^3$ term in the same way we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FOBz5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FOBz5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Make sure not to add too high of a degree to your polynomial or you will be overfitting!! This means although you characterize your training data perfectly, it will not generalize well to new instances. Thus this will be a useless model. That is why you need to split your training and testing data, that way you can verify if the model you build using your training data can generalize. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-09T17:26:07.160" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28895" PostTypeId="2" ParentId="28593" CreationDate="2018-03-09T17:30:40.453" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Since calculating the backward gradients that too in CNN is very math heavy and it's a bit different from normal MLP as in CNN'S we have Tensors not scalar values to multiply so People use Correlation to Calculate the Convolution Operation...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Just adding some links which I refer to from time to time thinking that someday I will be able to do this from scratch (seems impossible with deepening of Layers though...)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference 1&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference 2&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;code&gt;CS231N Slides (Lecture 4-6 specially, the computational graphs construction are very helpful to understand the Back-propagation Algorithm..)&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It would be great if Someone can &lt;code&gt;transfer the Equations into Latex here&lt;/code&gt;&#xA;So that the question can be answered formally..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the above Links, here's what I got,(Came to know that Convolutions are equivalent to Correlation)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/file/d/172Ymua0dKPMtmUpPqARWdsCU5C96N-C5/view?usp=drivesdk&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image 1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://drive.google.com/file/d/1VUNu_71bMSlxLCAh-jzLtAH2pBG9193Q/view?usp=drivesdk&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image 2&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Stack doesn't allow me uploading image so using gdrive..(do rotate one of the images, the first one actually)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it will give you an idea...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-05T13:45:17.833" LastActivityDate="2018-04-05T13:45:17.833" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28896" PostTypeId="2" ParentId="28768" CreationDate="2018-03-09T17:31:12.430" Score="0" Body="&lt;p&gt;The cypher query that ended up working for me was &lt;code&gt;MATCH (n) OPTIONAL MATCH (n)-[r]-&amp;gt;(m) RETURN n,r,m&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;function makeAllBlocksQuery(cb){&#xA;  db.cypher({&#xA;    query: &quot;MATCH (n) OPTIONAL MATCH (n)-[r]-&amp;gt;(m) RETURN n,r,m&quot;,&#xA;  }, function(err,results){&#xA;    var processedResults = handleAllBlocks(err, results);&#xA;    cb(processedResults);&#xA;  });  &#xA;}&#xA;&#xA;function handleAllBlocks(err, results) {&#xA;    if (err) throw err;&#xA;    if (!results) {&#xA;      console.log('No blocks found.');&#xA;      return [];&#xA;    } else {&#xA;      //first, form [_fromId, _toId] relationshipArray&#xA;      var relationshipArray = new Array();&#xA;      for (count = 0; count &amp;lt; Object.keys(results).length; count++) {&#xA;        if(results[count].r !== null) {&#xA;          var relationship = [results[count].r._fromId, results[count].r._toId];&#xA;          relationshipArray.push(relationship);&#xA;        }&#xA;      }&#xA;&#xA;      var resultArray = new Array();&#xA;      var repliedIdsArray = new Array();&#xA;      //add nodes found in results to resultArray&#xA;      //and add id property to properties for later&#xA;      for (count = 0; count &amp;lt; Object.keys(results).length; count++) {&#xA;        results[count].n.properties.id = results[count].n._id;&#xA;&#xA;        //iterate through the relationshipArray for each result&#xA;        //and add all the replied Ids to the repliedIdsArray&#xA;        for (relationshipcount = 0; relationshipcount &amp;lt; relationshipArray.length; relationshipcount++){&#xA;          if (relationshipArray[relationshipcount][1] == results[count].n.properties.id) {&#xA;            repliedIdsArray.push(relationshipArray[relationshipcount][0]);&#xA;          }&#xA;        }&#xA;        //add the repliedIdsArray to the result's properties&#xA;        results[count].n.properties.hasreplyto = repliedIdsArray;&#xA;        repliedIdsArray = [];&#xA;        //stringify so it can be sent to client more easily&#xA;        resultArray.push(JSON.stringify(results[count].n.properties));&#xA;      }&#xA;      return resultArray;&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13165" LastActivityDate="2018-03-09T17:31:12.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28897" PostTypeId="2" ParentId="28874" CreationDate="2018-03-09T17:43:30.933" Score="10" Body="&lt;p&gt;&lt;strong&gt;Train to avoid false negatives&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What your network learns depends on the loss function you pass it. By choosing this function you can emphasize various things - overall accuracy, avoiding false negatives, false positives etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case you probably use a cross entropy loss in combination with a softmax classifier. While softmax squashes the prediction values to be 1 when combined across all classes, the cross entropy loss will penalise the distance between the actual ground truth and the prediction. In this calculation it will not take into account what the values of the &quot;false negative&quot; predictions are. In other words: The loss function only cares for the correct class and its related prediction, not for the values of all other classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since you want to avoid false negatives this behaviour is probably the exact thing you need. But if you also want the distance between the actual class and the false predictions another loss function that also takes into account the false values might even serve you better. Give your high accuracy this poses the risk that your overall performance will drop. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What to do then?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Making the wrong prediction and being very sure about it is not uncommon. There are millions of things you could look at, so your best guess probably is to investigate the error. E.g. you could use a confusion matrix to recognize patterns which classes are mixed with which. If there is structure you might need more samples of a certain class or there are probably labelling errors in your training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to go ahead would be to manually look at all (or some) examples of errors. Something very basic as listing the errors in a table and trying to find specific characteristics can guide you towards what you need to do. E.g. it would be understandable if your network usually gets the &quot;difficult&quot; examples wrong. But maybe there is some other clear systematic your network did not pick up yet due to lack of data? &lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-03-10T07:23:45.090" LastActivityDate="2018-03-10T07:23:45.090" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28898" PostTypeId="2" ParentId="28881" CreationDate="2018-03-09T18:23:06.643" Score="1" Body="&lt;p&gt;From &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/3492&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow Github&lt;/a&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Dilated max-pooling is simply regular max-pooling but the pixels/voxels you use in each &amp;quot;application&amp;quot; of the max-pooling operation are exactly the same pixels/voxels you would select with dilated convolution.&lt;/p&gt;&#xA;&lt;p&gt;Dilated convolution/pooling are useful for connectomics and 3D shape datasets (3D deep learning).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;So, similarly as in convolution every nth pixel is selected for the operation.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Selecting points like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/IYALG.png&quot; alt=&quot;picture&quot; /&gt;&lt;/p&gt;&#xA;&lt;p&gt;making pooling for them like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/vgds2.png&quot; alt=&quot;image&quot; /&gt;&#xA;Picture sources:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Is-a-pooling-layer-necessary-in-CNN-Can-it-be-replaced-by-convolution&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/Is-a-pooling-layer-necessary-in-CNN-Can-it-be-replaced-by-convolution&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-is-max-pooling-in-convolutional-neural-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/What-is-max-pooling-in-convolutional-neural-networks&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Edit2:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;2 x 2 max pooling with 3 x 3 dilation:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0 (5) 0  0 [6] 0  0 (3) 0  0 [6] 0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0 (3) 0  0 (4) 0  0 (2) 0  0 (4) 0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0 (1  0  0 (2) 0  0 (2) 0  0 (1) 0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0 (3) 0  0 [4] 0  0 [6] 0  0 (3) 0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;0  0  0  0  0  0  0  0  0  0  0  0  0  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;becomes&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;6  6&#xA;4  6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-11T12:22:38.047" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28899" PostTypeId="1" AcceptedAnswerId="28913" CreationDate="2018-03-09T19:31:40.433" Score="2" ViewCount="5042" Body="&lt;p&gt;I built an multi classification in CNN using keras with Tensorflow in the backend. It nicely predicts cats and dogs. However, when it comes to an image which does not have any object-white background image-, it still finds a dog ( lets say probability for dog class 0.75…, cats 0.24… ). I am a quite newbie learner in learning built with neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if I am asking a silly question, even though I have searched the internet I could not find any answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is my exception from the case of the white background image as an input to prediction method, is 0 probability for dog and cat classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion would make me so happy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The below is how I implemented the training.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier = Sequential()&#xA;&#xA;classifier.add(Conv2D(32, 3, 3, input_shape=(64, 64, 3), activation='relu'))&#xA;classifier.add(MaxPool2D(pool_size=(2, 2)))&#xA;&#xA;classifier.add(Conv2D(32, 3, 3, activation='relu'))&#xA;classifier.add(MaxPool2D(pool_size=(2, 2)))&#xA;&#xA;classifier.add(Flatten())&#xA;&#xA;classifier.add(Dense(units=128, activation='relu'))&#xA;&#xA;classifier.add(Dense(units=2, activation='softmax'))&#xA;&#xA;# Metrics will be categorical_accuracy&#xA;classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])&#xA;&#xA;&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;&#xA;train_datagen = ImageDataGenerator(&#xA;    rescale=1. / 255,&#xA;    shear_range=0.2,&#xA;    zoom_range=0.2,&#xA;    horizontal_flip=True)&#xA;&#xA;test_datagen = ImageDataGenerator(rescale=1. / 255)&#xA;training_set = train_datagen.flow_from_directory(&#xA;    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/training_set',&#xA;    target_size=(64, 64),&#xA;    batch_size=32,&#xA;    class_mode='categorical')&#xA;&#xA;test_set = test_datagen.flow_from_directory(&#xA;    '/Users/ozercevikaslan/Desktop/Convolutional_Neural_Networks/dataset/test_set',&#xA;    target_size=(64, 64),&#xA;    batch_size=32,&#xA;    class_mode='categorical')&#xA;&#xA;classifier.fit_generator(&#xA;    training_set,&#xA;    steps_per_epoch=8000,&#xA;    epochs=25,&#xA;    validation_data=test_set,&#xA;    validation_steps=2000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47531" LastEditorUserId="47531" LastEditDate="2018-03-09T19:38:27.247" LastActivityDate="2019-11-19T21:28:14.687" Title="Multi-Class classification with CNN using keras - trained model predicts object even in a fully white picture" Tags="|keras|tensorflow|multiclass-classification|cnn|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28903" PostTypeId="2" ParentId="28870" CreationDate="2018-03-09T20:42:16.130" Score="0" Body="&lt;p&gt;It will be faster to merge duplicates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you &lt;em&gt;can&lt;/em&gt; run k-means with duplicates, obviously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget preprocessing &amp;amp; result validation!&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-09T20:42:16.130" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28906" PostTypeId="2" ParentId="28883" CreationDate="2018-03-09T22:35:38.093" Score="2" Body="&lt;p&gt;Multiplication is &lt;em&gt;not&lt;/em&gt; a linear operation. Your linear SVM constructs a (hyper-)plane&#xA;$$&#xA;w_0 = w_1 x_1 + w_2 x_2&#xA;$$&#xA;for some weights $w_0, w_1, w_2.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By introducing the AND-feature, you add another dimension:&#xA;$$&#xA;w_0 = w_1 x_1 + w_2 x_2 + w_3 x_1 x_2.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might well be that your two-dimensional data set is not linearly separable, but the three-dimensional data set is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A small addition: Would adding the OR-feature increase performance even further? No, because it is a linear combination of the other three features: $x \vee y = x + y - (x \wedge y)$ where $\vee$ is OR and $\wedge$ is AND.&lt;/p&gt;&#xA;" OwnerUserId="45507" LastEditorUserId="45507" LastEditDate="2018-03-09T22:43:27.617" LastActivityDate="2018-03-09T22:43:27.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28907" PostTypeId="2" ParentId="17134" CreationDate="2018-03-09T23:04:24.910" Score="0" Body="&lt;ul&gt;&#xA;&lt;li&gt;1x512 input array means: recurrent network processes&#xA;electrode voltage 512 times, in other words you have single feature&#xA;to process.&lt;/li&gt;&#xA;&lt;li&gt;CNN with one feature is useless.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47483" LastActivityDate="2018-03-09T23:04:24.910" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28908" PostTypeId="2" ParentId="23923" CreationDate="2018-03-10T01:43:50.323" Score="1" Body="&lt;p&gt;A thought:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe an approach like &lt;a href=&quot;https://arxiv.org/pdf/1411.4555.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Show and Tell: A neural image caption generator&lt;/a&gt; would be useful to look at. There they use exactly as you say a combination of CNN (for image) features and LSTM (for text) + search to generate text given the features from the image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible that then a discriminator can be used to learn whether 'generated text' is close enough to 'given label' in order to give the final result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also possible that you do not need the discriminator, and instead of using the output of the show and tell network directly (like removing the last linear layer) and feed one of the earlier steps (which represents a probability distribution of possible texts to generate) into the discriminator so that it has more possible combinations of text outputs to work with...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is probably more supervised than you want, but maybe starting with &lt;a href=&quot;https://github.com/KranthiGV/Pretrained-Show-and-Tell-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;pre-trained&lt;/a&gt; weights of this model can get you somewhere faster.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-03-10T01:43:50.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28909" PostTypeId="2" ParentId="28733" CreationDate="2018-03-10T06:32:32.110" Score="1" Body="&lt;p&gt;This is equivalent to thinking of this as learning a function of two inputs, i.e.,  $g : (X,Z) \mapsto Y$ (i.e., $g(X,Z)=Y$).  Any machine learning method for learning functions can be used for that; you just treat the feature vector (the input) as the concatenation of the vector $X$ and the vector $Z$, and then use standard ML techniques on the resulting instances.  So, you don't need anything special or fancy to handle this -- you can use any existing machine learning method that you are comfortable with.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-10T06:32:32.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28910" PostTypeId="1" CreationDate="2018-03-10T06:33:29.980" Score="2" ViewCount="63" Body="&lt;p&gt;&lt;strong&gt;Problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to maximize performance for social media posts by optimizing the time when they are published. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current model:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X: publishing_datetime, post_attribute_1, ..., post_attribute_n&#xA;y: performance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Desired model:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X: post_attribute_1, ..., post_attribute_n&#xA;y: publishing_datetime&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The desired model should predict the optimal publishing_datetime for maximizing performance. Once the data can be modeled like this,  the problem is solved with a regression neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I've tried:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Filtering the posts with above-average performance and using their attributes and publishing_datetime to form my desired model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is not ideal as a lot of data is unused and posts with particularly great performance influence just as much as posts with barely above-average performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Any suggestion on how to achieve this model transition?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All ideas and alternative approaches are very welcome. Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="41777" LastActivityDate="2024-01-10T02:06:00.793" Title="Modeling social media post scheduling optimization" Tags="|machine-learning|neural-network|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28911" PostTypeId="1" CreationDate="2018-03-10T08:52:04.097" Score="0" ViewCount="91" Body="&lt;p&gt;I have a small restricted dataset, it is not very small but accuracy will be much better if I will have more data. I have split it to train and test datasets:  85%/15%. I have chosen NN model and trained it, measured loss, it is ok. Now I should use it in production. Should I train it additionally on the test data, because in such way I will get better accuracy? &lt;/p&gt;&#xA;" OwnerUserId="21319" LastActivityDate="2018-06-11T19:52:33.090" Title="When it is a good idea to train a model also on the test data?" Tags="|machine-learning|neural-network|dataset|training|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28913" PostTypeId="2" ParentId="28899" CreationDate="2018-03-10T10:34:09.153" Score="1" Body="&lt;p&gt;As I mentioned in my question post, the post is a bit silly even for a new learner. In this case, the world has only 2 classes which are dogs and cats so the output must be either a dog or a cat.&lt;/p&gt;&#xA;" OwnerUserId="47531" LastEditorUserId="47531" LastEditDate="2018-03-12T23:20:12.927" LastActivityDate="2018-03-12T23:20:12.927" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28914" PostTypeId="2" ParentId="16992" CreationDate="2018-03-10T10:49:13.710" Score="1" Body="&lt;p&gt;I can only guess that since it is binary classifier based recommender model, it predicts whether a product is recommendable to a user or not with a probability score which can be set to a threshold.&lt;/p&gt;&#xA;" OwnerUserId="47331" LastActivityDate="2018-03-10T10:49:13.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28915" PostTypeId="1" AcceptedAnswerId="29071" CreationDate="2018-03-10T11:03:06.680" Score="10" ViewCount="1677" Body="&lt;p&gt;I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;repeat many thousand times {&#xA;  repeat until game is over {&#xA;    if(board layout is unknown or exploring) {&#xA;      move randomly&#xA;    } else {&#xA;      move in location which historically gives highest reward&#xA;    }&#xA;  }&#xA;&#xA;  for each step in the game {&#xA;    determine board layout for current step&#xA;    if(board layout is unknown) {&#xA;      add board layout to memory&#xA;    }&#xA;    update reward for board layout based on game outcome&#xA;  }&#xA;}&#xA;&#xA;now play a human and win :-)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Exploration: in the beginning the algorithm explores aggresively, and this reduces linearly. After say a thousand games it only explores in 10% of the moves. All other moves are based on exploitation of previous rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rewards: if the game resulted in a win, then award 10 points. If the game resulted in a draw, 0 points, otherwise -5 points.  Actually, these rewards can be &quot;tuned&quot;, so that if the game was shorter and it was won, then award more points or if it was longer award less points.  This way the algorithm prefers winning quickly. That means that it learns to win as soon as possible, rather than aiming to win later on.  That is important so that it doesn't miss winning immediately - if it missed such a move the opponent would likely a) move there to avoid letting the AI win next time, and b) think the algorithm was stupid because it missed an &quot;obvious&quot; win.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This algorithm does indeed learn, so I can class it as a maching learning algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think, but I am not sure, that it is a reinforced learning algorithm. However, according to &lt;a href=&quot;https://www.cse.unsw.edu.au/~cs9417ml/RL1/introduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.cse.unsw.edu.au/~cs9417ml/RL1/tdlearning.html&lt;/a&gt; it is not temporal difference learning, because it doesn't estimate the rewards until the end, and it should be estimating the reward as it goes along. That might mean that it is not reinforced learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I noticed that the algorithm is pretty useless unless I train it with at least every move which an opponent tries. So in a way it feels like it's using brute force rather than really &quot;learning&quot;. This makes me question whether or not machine learning tictactoe is really learning.  I agree that using a neural network to learn image recognition can be classed as learning because when it sees an unknown image it is able to state its classification. But that is quite useless for games like tictactoe where similar looking board layouts are totally unrelated (one may lead to a win, the other may lead to a loss). So...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; regarding rewards... when the algorithm is deciding where to go, it works out the reward for each position as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;var total = winRewards + drawRewards + lossRewards;&#xA;move.reward = (100*(winRewards/total)) + (10*(drawRewards/total)) + (-1*(lossRewards/total));&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I divide by the total number of points (for each move), because otherwise it seems to learn that one place is GREAT and doesn't give other ones a chance.  This way, we work out the win ratio regardless of how often it's been played. It's normalised in comparison to the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code is available here: &lt;a href=&quot;https://github.com/maxant/tictactoe/blob/master/ai.js&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/maxant/tictactoe/blob/master/ai.js&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE #2: I have since figured out that this algorithm cannot be classed as using brute force because it doesn't actually learn that many games before becoming an expert. Details here: &lt;a href=&quot;http://blog.maxant.co.uk/pebble/2018/04/11/1523468336936.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.maxant.co.uk/pebble/2018/04/11/1523468336936.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47557" LastEditorUserId="47557" LastEditDate="2018-04-11T17:40:21.287" LastActivityDate="2018-04-11T17:40:21.287" Title="Is this a Q-learning algorithm or just brute force?" Tags="|machine-learning|neural-network|reinforcement-learning|q-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28917" PostTypeId="2" ParentId="25651" CreationDate="2018-03-10T13:38:09.760" Score="0" Body="&lt;p&gt;Try modeling the time series with a midpoint estimate (expected value at each time) and a band estimate (permissible spread at any given midpoint). Then being an anomaly is just a matter of being outside the band.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How you estimate the midpoints depends on the problem. For example if your time series has one or more seasonalities, you could use time series decomposition (e.g., seasonal-trend decomposition using loess). The band estimate is also problem-dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did some work like this for a bookings data time series. I wrote about it here: &lt;a href=&quot;https://techblog.expedia.com/2016/07/28/applying-data-science-to-monitoring/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://techblog.expedia.com/2016/07/28/applying-data-science-to-monitoring/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user10803" LastActivityDate="2018-03-10T13:38:09.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28918" PostTypeId="1" CreationDate="2018-03-10T14:11:02.463" Score="3" ViewCount="76" Body="&lt;p&gt;I have about 30 lists of unequal length (some of which are triplicates of the data), corresponding to metrics relating to nodes of different graphs. I want to compare their similarity using a distance metric, but was unsure which method I can use given the data lists are of unequal length. I was exploring using dynamic time warping, but was wondering if there is any other more basic method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I was a considering creating histograms with same bin edges and number of bins for each list and using a distance metric on the frequency, but I am not sure how to go about this using python, or if there's a function/package that does this already. Is this even a &quot;good&quot; way of doing it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm also interested in finding a way to measure the statistical significance of the distance measures between the different graphs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a lot in one question, I am new to this and appreciate any help.  Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="47568" LastActivityDate="2018-03-10T15:42:52.900" Title="Statistically Robust Distance Measure/Metric for comparing more than two network data series" Tags="|data-mining|distance|descriptive-statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28920" PostTypeId="2" ParentId="28918" CreationDate="2018-03-10T15:42:52.900" Score="1" Body="&lt;p&gt;Just to clarify the question:&#xA;Do the lists describe different graphs, or do you need the similarities to learn which lists refer to the same graph? &#xA;Is data tri- or duplicated within single lists or are lists duplicating other lists? Would you consider removing redundant metrics?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry, this was rather a comment than an answer, but I am not yet entitled to comment here, so I am starting the discussion like this.&lt;/p&gt;&#xA;" OwnerUserId="21163" LastActivityDate="2018-03-10T15:42:52.900" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28921" PostTypeId="2" ParentId="326" CreationDate="2018-03-10T16:34:16.250" Score="0" Body="&lt;p&gt;I do not think Python has &lt;a href=&quot;https://alternativeto.net/list/2063/guis-to-save-from-typing-r-code/&quot; rel=&quot;nofollow noreferrer&quot;&gt;point-click GUI&lt;/a&gt; that turn it into SPSS and SAS. Playing around with those is genuinely fun. &lt;/p&gt;&#xA;" OwnerUserId="30507" LastActivityDate="2018-03-10T16:34:16.250" CommentCount="0" CommunityOwnedDate="2018-03-10T16:34:16.250" ContentLicense="CC BY-SA 3.0" />
  <row Id="28922" PostTypeId="1" CreationDate="2018-03-10T17:14:12.530" Score="0" ViewCount="77" Body="&lt;p&gt;Can it really be called Generalization if we remove/modify data points to suit our model?&lt;/p&gt;&#xA;" OwnerUserId="47572" LastActivityDate="2018-05-10T16:54:41.593" Title="Why is it necessary to clean/preprocess data if the unseen real data is bound to consist missing values, outliers etc.?" Tags="|machine-learning|data-cleaning|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28923" PostTypeId="2" ParentId="28922" CreationDate="2018-03-10T17:30:41.297" Score="1" Body="&lt;p&gt;It depends. Actually there are research papers finding that neural network can sometimes cope very well with sparse or noisy labels. But basically when you apply any machine learning approach, you want your model to pick up patterns in your data. In order to be a pattern something needs to be repetitive and predictable. Outliers and errors in your data might not adhere to that. They rather break the pattern. It can still work but it is harder to recognize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Making a model &quot;learn&quot; outliers/missing values etc. will only work, if these incidences also appear in a pattern which makes them unlikely to be outliers. As a rule of thumb it is therefore a good idea to remove this noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Finding patterns in exceptions &amp;amp; errors&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I mentioned some researches try to address the noise issue for example by classifying if there is random (&quot;white&quot;) noise or structured noise. They not only predict the target value but also the type of noise and then try to make another prediction based on both pieces of information. So there are some active approaches to address noise but usually this is due to the fact that it would be to cumbersome or even impossible to remove the noise.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-03-10T17:30:41.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28924" PostTypeId="2" ParentId="13550" CreationDate="2018-03-10T17:40:28.750" Score="2" Body="&lt;p&gt;Use the below instructions as it was worked for me&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import accuracy_score&#xA;&#xA;predictions = nordan_tree.predict(X_test)&#xA;&#xA;print(accuracy_score(y_test, predictions))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38842" LastEditorUserId="11097" LastEditDate="2018-03-10T20:18:07.173" LastActivityDate="2018-03-10T20:18:07.173" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28928" PostTypeId="2" ParentId="28915" CreationDate="2018-03-10T20:12:26.723" Score="6" Body="&lt;p&gt;Kudos for figuring out a working tic-tac-toe playing algorithm from scratch!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;First things first, this is definitely &lt;em&gt;not&lt;/em&gt; Q-learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I do think it classifies as Reinforcement Learning. You have implemented these key components of RL:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;A state (the current board), used as input on each step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An action (desired next board arrangement), used as output. When the action is effectively to choose the next state directly, this is sometimes called the afterstate representation. It is commonly used in RL for deterministic games.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Rewards generated by the environment, where the agent's goal is to maximise expected reward.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;An algorithm that can take data about states, actions and rewards, and learn to optimise expected reward through gaining experience within the environment.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Your algorithm is closest IMO to &lt;a href=&quot;http://www.incompleteideas.net/book/ebook/node53.html&quot; rel=&quot;noreferrer&quot;&gt;Monte Carlo Control&lt;/a&gt;, which is a standard RL approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the big advantages of Q Learning is that it will learn an optimal policy even whilst exploring - this is known as &lt;em&gt;off-policy&lt;/em&gt; learning, whilst your algorithm is on-policy, i.e. it learns about the values of how it is currently behaving. This is why you have to reduce the exploration rate over time - and that can be a problem because the exploration rate schedule is a hyper-parameter of your learning algorithm that may need careful tuning.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, I suppose it would be technically. However, it is unlikely to scale well to more complex problems just from adding a neural network to estimate action values, unless you add in some of the more sophisticated elements, such as using temporal-difference learning or policy gradients. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A discount factor is not important for episodic problems. It is only necessary for continuous problems, where you need to have some kind of time horizon otherwise the predicted reward would be infinite (although you could also replace the discount mechanism with an average reward approach in practice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate is an important omission. You don't explain what you have in its stead. You have put &lt;code&gt;update reward for board layout based on game outcome&lt;/code&gt; - that update step typically has the learning rate in it. However, for tic-tac-toe and Q-Learning, you can actually set the learning rate to 1.0, which I guess is the same as your approach, and it works. &lt;a href=&quot;https://github.com/neilslater/game_playing_scripts/blob/master/tictactoe_q.py&quot; rel=&quot;noreferrer&quot;&gt;I have written example code that does exactly that&lt;/a&gt; - see &lt;a href=&quot;https://github.com/neilslater/game_playing_scripts/blob/master/tictactoe_q.py#L233&quot; rel=&quot;noreferrer&quot;&gt;this line which sets learning rate to 1.0&lt;/a&gt;. However, more complex scenarios, especially in non-deterministic environments, would learn badly with such a high learning rate.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your algorithm is definitely learning something from experience, albeit inefficiently compared to a human. A lot of the more basic RL algorithms have similar issues though, and often need to see each possible state of a system multiple times before they will converge on an answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would say that an exhaustive tree search from the current position during play was &quot;brute force&quot;. In a simple game like tictactoe, this is probably more efficient than RL. However, as games get more and more sophisticated, the machine learning approach gets competitive with search. Often both RL and some form of search are used together.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-10T20:25:39.227" LastActivityDate="2018-03-10T20:25:39.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28929" PostTypeId="1" CreationDate="2018-03-10T20:34:28.097" Score="1" ViewCount="24" Body="&lt;p&gt;I am looking for papers which could give a greater understanding on differences between Tree based and linear based models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anybody can gove me some pointers?&lt;/p&gt;&#xA;" OwnerUserId="45411" LastEditorUserId="134" LastEditDate="2018-12-27T14:56:49.023" LastActivityDate="2018-12-27T14:56:49.023" Title="Papers on tree based and linear models" Tags="|algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28931" PostTypeId="2" ParentId="28929" CreationDate="2018-03-10T21:19:35.853" Score="2" Body="&lt;p&gt;I suggest chapters „3 Linear Regression“ and „8 Tree-Based Methods“ in &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;An introduction to statistical learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2018-03-10T21:19:35.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28937" PostTypeId="1" CreationDate="2018-03-11T09:18:14.603" Score="2" ViewCount="1528" Body="&lt;p&gt;I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Architecture: InceptionV3&lt;/li&gt;&#xA;  &lt;li&gt;Pre-trained InceptionV3 with weights from image net&lt;/li&gt;&#xA;  &lt;li&gt;replaced last layer, retrained the second half of the network&lt;/li&gt;&#xA;  &lt;li&gt;Classification: Softmax with cross entropy ploss&lt;/li&gt;&#xA;  &lt;li&gt;Adam optimizer with keras default parameters&lt;/li&gt;&#xA;  &lt;li&gt;Hyper parameters: Learning rate 0.0001, batch size 64&lt;/li&gt;&#xA;  &lt;li&gt;Goal: Multi-label classification with exclusive labels&lt;/li&gt;&#xA;  &lt;li&gt;Data: 17 classes ranging from 0-16 with 48.600 images in total&lt;/li&gt;&#xA;  &lt;li&gt;Train, Test, Validation: 80%, 10%, 10%&lt;/li&gt;&#xA;  &lt;li&gt;Training: 2 epochs training only last layer + 2 epochs training second half of network&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Results&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my validation set I created a confusion matrix with the following results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Rnu69.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rnu69.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are also some more per-class performance metrics:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A4ylQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A4ylQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="28175" LastEditDate="2018-03-11T09:51:13.747" LastActivityDate="2018-03-11T09:51:13.747" Title="Interpreting confusion matrix and validation results in convolutional networks" Tags="|machine-learning|deep-learning|cnn|performance|confusion-matrix|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="28939" PostTypeId="2" ParentId="28937" CreationDate="2018-03-11T09:41:34.043" Score="1" Body="&lt;p&gt;These are my recommendations: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly. &lt;/li&gt;&#xA;&lt;li&gt;Try to change the architecture. If your task does not have the same scale as &lt;em&gt;image-net&lt;/em&gt; surely you would need a network with simpler network architecture, consequently the number of parameters diminishes. &lt;/li&gt;&#xA;&lt;li&gt;Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.&lt;/li&gt;&#xA;&lt;li&gt;For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes. &lt;/li&gt;&#xA;&lt;li&gt;Because your data-set is not balanced, use other metrics to evaluate your model, like &lt;code&gt;F1&lt;/code&gt; score. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-11T09:41:34.043" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28940" PostTypeId="1" CreationDate="2018-03-11T13:36:10.970" Score="1" ViewCount="2030" Body="&lt;p&gt;So every time I try to write a CNN or RNN I have problems understanding how the dimensions work. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I don't understand why does the model compile, which in my opinion means that the output dimension of the previous layer is the input dimension of the next one and so on(the layers are compatible with each other). So how is it possible that when I fit my model I get an error in the 3rd or 4th layer(or any other layer except the first one) saying that for ex it &lt;strong&gt;expects 3 dimensions but got 2&lt;/strong&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;On a more precise example, lets say I have the following &lt;a href=&quot;https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-shot model&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vocab_size =10000&#xA;src_txt_length =10&#xA;sum_txt_length = 100&#xA;&#xA;inputs = Input(shape=(src_txt_length,))&#xA;encoder1 = Embedding(vocab_size, 128)(inputs)&#xA;encoder2 = LSTM(128)(encoder1)&#xA;encoder3 = RepeatVector(sum_txt_length)(encoder2)&#xA;&#xA;decoder1 = LSTM(128, return_sequences=True)(encoder3)&#xA;outputs = TimeDistributed(Dense(vocab_size, activation='softmax'))(decoder1)&#xA;model = Model(inputs=inputs, outputs=outputs)&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam')&#xA;model.fit(X,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I get an error in the TimeDistributed layer saying that it expects 3 dimensions but got 2.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X.shape == (20000,10)&#xA;len(y) == 20000  # y is a list of labels 0s and 1s&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So if I want to calculate the dimension of each layer when I fit my data on paper, how will I do that ?&lt;/p&gt;&#xA;" OwnerUserId="47599" LastEditorUserId="29575" LastEditDate="2018-03-11T14:27:09.137" LastActivityDate="2018-05-10T22:50:09.787" Title="Dimension problem in keras neural networks" Tags="|machine-learning|neural-network|keras|rnn|cnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28941" PostTypeId="2" ParentId="28940" CreationDate="2018-03-11T14:21:23.683" Score="1" Body="&lt;p&gt;Ok I was able to recreate the error. The TimeDistributed layer applies the dense layer to each of the time slices. In this model you will have 100 different time slices as defined by &lt;strong&gt;sum_txt_length&lt;/strong&gt; and the RepeatVector layer. These time slices will be propagated through the network all the way to the TimeDistributed layer. Thus, your $y$ vector must have dimensions $(None, 100, 10000)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;code&gt;model.summary()&lt;/code&gt; to get the expected dimensions after each layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to fix the output layer to match the label dimensions. I believe you will need to add a &lt;code&gt;Flatten()&lt;/code&gt; layer before the densely connected layer, and do not use the TimeDistributed layer at the output. That way your output will be of size $vocab_size$, thus predicting the next vocabulary word given an input.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-11T14:21:23.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28942" PostTypeId="1" CreationDate="2018-03-11T16:42:29.757" Score="1" ViewCount="180" Body="&lt;p&gt;I have some problems understanding/interpreting the C-Index cluster quality measure. So, if we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c(x_i, x_j) = 1 $ if $ x_i, x_j $ in the same cluster; $0$ else&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\Gamma = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n d(x_i,x_j)*c(x_i,x_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\alpha = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n c(x_i,x_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$min=$ sum of the $\alpha$ smallest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$max=$ sum of the $\alpha$ largest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then the C-Index is defined as $C=\frac{\Gamma - min}{max - min}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The result is a value in $[0, 1]$, where lower values indicate a better cluster quality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here are some things I get from this value:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if all elements in a cluster are close together and all clusters are far apart, we can get  $\Gamma=min$, which means $C=0$&lt;/li&gt;&#xA;&lt;li&gt;Analogously, in a worst case scenario, all the observations that are furthers apart might be in the same cluster, so we would get $\Gamma=max$, which means $C=1$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now, these are the things I'm unsure about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First: If we only have a single cluster in our clustering (e.g. k-Means for $k=1$), then $\alpha$ is equal to the number of distinct pairs of observations, so $max=min$, which means $C=\frac{\Gamma - min}{max - min} = \frac{\Gamma - min}{min - min} = \frac{\Gamma - min}{0}$ and we get a division by 0. A similar problem occurs if we have $N$ observations in $N$ different clusters, since $c$ is always $0$ in that case. So, is it correct to say that the C-Index can only be used for clusterings with $k$ many clusters where $1 &amp;lt; k &amp;lt; N$, for $N$ observations?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second: Is it reasonable to say that the C-Index is agnostic to the number of clusters (e.g. the value of $k$ in k-Means)? For instance, we might have 5 observations $x_1...x_5$ close to each other, but each put into a separate cluster $C_1...C_5$. Then, we might have a clusters $C_6=\{x_6, x_7\}$ where $x_6, x_7$ are very close to each other but far apart from all other observations. In that case, $\Gamma=d(x_6,x_7)$, $\alpha=1$, $\min=d(x_6,x_7)$, so $\Gamma=min$, which means $C=0$. That is, we have the best possible C-Index-Value, even though, intuitively, it might have been better to put $x_1...x_5$ into a single cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, this is more about k-Means: if we use normal k-Means (not global k-Means), are we always guaranteed to reach $C=0$, for an unbounded number of iterations? I can't seem to find an example that wouldn't result in this.&lt;/p&gt;&#xA;" OwnerUserId="47561" LastEditorUserId="47561" LastEditDate="2018-03-11T19:18:03.787" LastActivityDate="2019-12-04T02:00:35.683" Title="Interpreting the C-Index" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28944" PostTypeId="1" AcceptedAnswerId="28955" CreationDate="2018-03-11T18:17:31.520" Score="1" ViewCount="1828" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/k6gNb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k6gNb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I'm doing the Udemy course for Data Science in Python, and there's something weird happening with the plots. The first one in the above picture is mine, the second is the instructors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I get rid of the white space on the sides, they seem to be default when using the plotting function in pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &#xA;This is all the code. Everything is basically default, so I'm guessing the matplotlib has some default settings changed in newer version of the package. The file df3 is here &lt;a href=&quot;https://github.com/sxsheng/Udemy&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/sxsheng/Udemy&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;import matplotlib.pyplot as plt&#xA;&#xA;df3 = pd.read_csv('df3')&#xA;&#xA;%matplotlib inline (This line is to be used in jupyter notebooks to display plots)&#xA;&#xA;df3['a'].plot.hist()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47610" LastEditorUserId="40853" LastEditDate="2018-03-12T03:34:56.777" LastActivityDate="2018-03-12T03:44:05.520" Title="Plots getting rid of whitespace" Tags="|visualization|plotting|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28948" PostTypeId="1" CreationDate="2018-03-11T21:57:18.210" Score="0" ViewCount="916" Body="&lt;p&gt;Suppose I have a dataset with $M$ observations of dimension $N$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've seen many people use Kohonen networks (or self-organizing maps, SOMs) to do this task. In this approach, the neurons have weights of the same dimension of the observations, $N$. The idea is to train the network so that each node $w_i$ becomes a centroid of a cluster, and thus identify the clusters with the neurons of the network. As we don't know the number of clusters a priori, one should establish a large number of neurons, $W$ such that $W&amp;lt;&amp;lt; M$, and then identify the nodes that represent the centroids (checking which ones are located in more dense areas and other criteria similar to that one).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is why one would do this. I could just say that I have $M$ possible centroids (each datapoint) and then use the discriminating criteria to check which ones are the actual centroids. In this approach there are $M$ centroid candidates and in the SOM one, $W$. But in both cases the dimension is $N$. Why is then using a SOM favorable? I believe I would want to have &lt;em&gt;more&lt;/em&gt; data when performing clustering, so I think that having $M$ samples instead of $W$ would be better. Also, the dimensionality of the problem stays the same, so no advantage there. On top of it, it would be much faster to do it without the SOM, as training it may take a while, but the output (i.e. the values of the neuron weights) would just be pretty similar to a subset of the original dataset, which could be obtained just by identifying possible centroids from the $M$ observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is it then that SOMs are used for clustering?&lt;/p&gt;&#xA;" OwnerUserId="35479" LastActivityDate="2018-04-11T08:46:40.413" Title="Why use SOM for clustering?" Tags="|neural-network|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28950" PostTypeId="2" ParentId="28940" CreationDate="2018-03-11T22:10:27.720" Score="1" Body="&lt;p&gt;The shape of the input:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = Input(shape=(src_txt_length,))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = Input(shape=(src_txt_length,vocab_size))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47483" LastActivityDate="2018-03-11T22:10:27.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28951" PostTypeId="2" ParentId="28899" CreationDate="2018-03-11T23:03:51.590" Score="1" Body="&lt;p&gt;in categorical_crossentropy the sum of predictions are equal to one, in your case either cat or dog, &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;maybe using accuracy threshold, (or) adding third class for unknowns (other images than dog/cat).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47483" LastActivityDate="2018-03-11T23:03:51.590" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28952" PostTypeId="1" AcceptedAnswerId="28954" CreationDate="2018-03-11T23:06:21.643" Score="1" ViewCount="848" Body="&lt;p&gt;Given a predicted sequence and actual sequence I want to compute it's precision and accuracy, for example: Note that these sequences will only contain &lt;code&gt;0, 1 or -1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;predicted sequence: &lt;code&gt;-1,0,1,1,-1,0,1,1,0,-1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;actual sequence:     &lt;code&gt;-1,1,0,1,-1,1,0,1,0,-1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that precision is computed using this &lt;code&gt;tp/tp+fp&lt;/code&gt; and accuracy is computed using &lt;code&gt;tp + tn /tp + tn + fp + fn&lt;/code&gt;. But because I have &lt;code&gt;-1&lt;/code&gt; in it I am unsure how I would compute true positives? My understanding that a true positive is if I predicted a &lt;code&gt;1&lt;/code&gt; and it's corresponding actual value is a &lt;code&gt;1&lt;/code&gt;. A walk through of the computation for precision and accuracy would help. &lt;/p&gt;&#xA;" OwnerUserId="28660" LastEditorUserId="28660" LastEditDate="2019-01-11T08:57:24.283" LastActivityDate="2019-01-11T08:57:24.283" Title="How to compute precision and accuracy of a sequence that is not strictly binary?" Tags="|machine-learning|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="28954" PostTypeId="2" ParentId="28952" CreationDate="2018-03-12T01:44:45.580" Score="3" Body="&lt;p&gt;Welcome to the Site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that this problem is Multi-Class Classification Problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To get a confusion matrix for the same you can use the following command:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from mlxtend.evaluate import confusion_matrix&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#import the required packages&#xA;from mlxtend.evaluate import confusion_matrix &#xA;from mlxtend.evaluate import plot_confusion_matrix&#xA;&#xA;#Actual Target Values&#xA;y_target =    [-1,1,0,1,-1,1,0,1,0,-1]&#xA;#Predicted Values&#xA;y_predicted = [-1,0,1,1,-1,0,1,1,0,-1]&#xA;&#xA;#creation of confusion matrix&#xA;cm = confusion_matrix(y_target=y_target, &#xA;                      y_predicted=y_predicted, &#xA;                      binary=False)&#xA;#to print the calculated values  of Confusion Matrix&#xA;cm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Outcome:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array([[3, 0, 0],&#xA;   [0, 1, 2],&#xA;   [0, 2, 2]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For visualizing the cm you can use the following command:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fig, ax = plot_confusion_matrix(conf_mat=cm)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FgVje.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FgVje.png&quot; alt=&quot;OutcomeViz&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go through this &lt;a href=&quot;https://rasbt.github.io/mlxtend/user_guide/evaluate/confusion_matrix/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; for better understanding of &lt;strong&gt;mlextend&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can get the Precision and Accuracy values by using the following formulas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{kji}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ijk}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Go through these &lt;a href=&quot;http://text-analytics101.rxnlp.com/2014/10/computing-precision-and-recall-for.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;,&lt;a href=&quot;https://stats.stackexchange.com/questions/51296/how-do-you-calculate-precision-and-recall-for-multiclass-classification-using-co&quot;&gt;Link-2&lt;/a&gt; for better understanding on how to compute the same, in the &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/7391&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-3&lt;/a&gt; is GitHub link which explains on how they implemented for a 1-D array, looking at that you can try expanding it for your outcome.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-12T05:44:41.343" LastActivityDate="2018-03-12T05:44:41.343" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="28955" PostTypeId="2" ParentId="28944" CreationDate="2018-03-12T03:28:35.150" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;what you where thinking is right, the new version plot looks a bit different, to get exactly the same plot as above you can use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import matplotlib.pyplot as plt    &#xA;df3 = pd.read_csv('df3')&#xA;#%matplotlib inline (This line is to be used in jupyter notebooks to display plots)&#xA;&#xA;ax = plt.subplot(111)&#xA;ax.hist(df3['a'],ec='black')&#xA;&#xA;#this is x axis limiter since your value ranges in between 0 to 1 I gave these values&#xA;plt.xlim([0,1])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Outcome:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JnY1e.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JnY1e.png&quot; alt=&quot;Outcome&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you need anything else. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is what you were looking for, then you can accept the answer by clicking on the green tick mark.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-12T03:44:05.520" LastActivityDate="2018-03-12T03:44:05.520" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28956" PostTypeId="1" CreationDate="2018-03-12T03:44:47.053" Score="1" ViewCount="219" Body="&lt;p&gt;I am looking for a right NN architecture(probably, based on LSTM/GRU) for the classification problem I faced. I have an alphabet of events {A, B, C, D, ..., N} and sequences of these events for each training example. It is obviously that Embedding + LSTM/GRU gives me some baseline model, but my problem is a little bit more complex. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also have timestamps attached to each event, so the same sequence in terms of order my represent different scenarios in reality for example {B, C, C,  &amp;lt;2 days after&gt; A } and {B, &amp;lt;2 days after&gt; C, C, A }. I tried to replace embedded vectors representing events with just events' timestamps(or duration from previous event) in the LSTM/GRU layer input and it gave me also some model with 75%-80% quality from the baseline model(and about ~6% of examples were correctly classified which were not correctly classified in baseline model). Now I am looking for an NN architecture to combine these two datasets(sequence of events and sequence of timestamps). I have already tried the following approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Simply concatenate timestamp information into embedding vectors(with&#xA;and without training), for example if embedding size = 32, we would&#xA;have 33 dimensional vector for LSTM input - 97%-100% quality from&#xA;the baseline model &lt;/li&gt;&#xA;&lt;li&gt;NN with 2 inputs for event sequence and for&#xA;timestamps sequence with concatenation before LSTM - ~90% quality&#xA;from the baseline model &lt;/li&gt;&#xA;&lt;li&gt;NN with 2 inputs for event sequence and for&#xA;timestamp sequence with 2 LSTMs and concatenation after - ~95%&#xA;quality from the baseline model&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks for any comments or recommendations.&lt;/p&gt;&#xA;" OwnerUserId="47623" LastActivityDate="2018-03-12T03:44:47.053" Title="Unevenly stretched sequences with LSTM/GRU" Tags="|neural-network|deep-learning|rnn|lstm|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28957" PostTypeId="1" AcceptedAnswerId="28973" CreationDate="2018-03-12T05:48:16.757" Score="0" ViewCount="168" Body="&lt;p&gt;How is the performance of Fischer projection compared to other LDA methods of dimension reduction? I thought that Fischer projection was a great method of dimension reduction by maximizing class separation, but when I looked at the LDA methods in scikit learn, Fischer projection wasn't even in the list. This got me thinking, is it any good compared to the other methods out there?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit (answer): My bad, they are the same. Fischer projection is a 2 class special case of LDA. Projection using LDA can be performed using the fit_transform and transform methods in sklearn&lt;/p&gt;&#xA;" OwnerUserId="32288" LastEditorUserId="32288" LastEditDate="2018-03-17T10:56:51.743" LastActivityDate="2018-03-17T10:56:51.743" Title="Performace of Fischer projection as dimension reduction compared to other LDA methods" Tags="|dimensionality-reduction|pca|discriminant-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28958" PostTypeId="1" AcceptedAnswerId="28959" CreationDate="2018-03-12T06:08:54.713" Score="0" ViewCount="128" Body="&lt;p&gt;I am working on a dataset that has a dependent variable that is binary, but it contains 98% of 0's and 2% of 1's. I am trying to use Logistic regression to predict purchase of a product. But because of the huge number of 0's, the model is not predicting well and getting a large number of false positive result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly suggest how can I approach this.&lt;/p&gt;&#xA;" OwnerUserId="45519" LastActivityDate="2018-03-12T06:42:08.830" Title="Re: Logistic Regression" Tags="|classification|logistic-regression|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ClosedDate="2018-03-12T18:36:23.783" ContentLicense="CC BY-SA 3.0" />
  <row Id="28959" PostTypeId="2" ParentId="28958" CreationDate="2018-03-12T06:31:56.943" Score="0" Body="&lt;p&gt;This kind of problem is call Data Imbalance issue, this is a very common issue in Financial Industry, Health Care Industry(Cancer Cell Detection) like Banks and Insurance (for Fraud Detection) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To overcome such issues, we use different techniques like Over-sampling or Under-sampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Over-sampling tries to increase that minority records by duplicating those records to make balance in the data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under-sampling tries to decrease the majority records by removing some records which are not significant to make balance in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different algorithms for implementing the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can go through these &lt;a href=&quot;https://datascience.stackexchange.com/questions/24610/smote-and-multi-class-oversampling/24664#24664&quot;&gt;Link-1&lt;/a&gt;,&lt;a href=&quot;https://datascience.stackexchange.com/questions/24905/best-methods-to-solve-class-imbalance-problem-and-why/24912#24912&quot;&gt;Link-2&lt;/a&gt;, for Explanation and Implementation of the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you need anything else.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-03-12T06:31:56.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28960" PostTypeId="2" ParentId="28958" CreationDate="2018-03-12T06:42:08.830" Score="1" Body="&lt;p&gt;This is a class imbalance problem. Your data has more number of 0's, thats why the model is also predicting 0's only. There are many solutions to this problem, like over/under sampling, SMOTE etc. Here are the links which show how to tackle the problem-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/235808/binary-classification-with-strongly-unbalanced-classes&quot;&gt;Binary classification with strongly unbalanced classes&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Imbalanced classification problem&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-12T06:42:08.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28961" PostTypeId="2" ParentId="28948" CreationDate="2018-03-12T08:31:21.280" Score="1" Body="&lt;p&gt;I disagree with the assertion that SOM are frequently used. IMHO they are difficult to use and rather obscure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless they have some interesting properties, although I'd rather call them a visualization than a clustering technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are ignoring is the&quot;map&quot; property. Neurons aren't independent but connected in a particular layout. Without this, the method is useless AFAICT. You need nodes to usually not be empty, but rather the average of many points, and thus M&amp;lt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-12T08:31:21.280" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28962" PostTypeId="1" CreationDate="2018-03-12T08:55:45.130" Score="2" ViewCount="584" Body="&lt;p&gt;I want to get the Principal components of a dataset and apply K mean clustering on them. Do I need to Normalized the PCA output before applying Kmeans on them ?&lt;/p&gt;&#xA;" OwnerUserId="16879" LastActivityDate="2022-04-04T16:06:10.560" Title="Should Principal components be normalized before applying K means on them?" Tags="|k-means|pca|normalization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28964" PostTypeId="1" AcceptedAnswerId="28967" CreationDate="2018-03-12T09:26:14.260" Score="1" ViewCount="767" Body="&lt;p&gt;I am trying to split number string to two to digit numbers &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I get two different Numbers out that string&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want separate two numbers&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(&quot;-26755.22-50150.60&quot;) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-26755.22&#xA;-50150.60 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have tried &lt;code&gt;stringr::str_split&lt;/code&gt; but I din't manage to keep the digits. &lt;/p&gt;&#xA;" OwnerUserId="47630" LastEditorUserId="40853" LastEditDate="2018-03-12T10:20:13.567" LastActivityDate="2018-03-12T10:45:53.907" Title="How do I split number string with digit pattern?" Tags="|data-wrangling|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28966" PostTypeId="1" CreationDate="2018-03-12T10:04:21.423" Score="0" ViewCount="437" Body="&lt;p&gt;I'm writing a university report on for the Toxici comment classification Kaggle competition, comparing different attempt made with different models and &lt;strong&gt;I want to know if Convolutional Neural Networks are One model, multiple output layers model&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ONMO are also known as multi-task learning, this approach would have one input layer, one set of hidden layers, and one output layer for each label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, CNN are several layers of convolutions with nonlinear activation functions like ReLU or tanh applied to the results. It uses convolutions over the input layer to compute the output rather than a fully connected layer, or affine layer.&lt;/p&gt;&#xA;" OwnerUserId="41692" LastActivityDate="2018-03-12T12:56:02.250" Title="Are Convolutional Neural Networks a one model, multiple output layers.?" Tags="|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28967" PostTypeId="2" ParentId="28964" CreationDate="2018-03-12T10:18:28.653" Score="0" Body="&lt;p&gt;I think this should do the thing for you:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#the input&#xA;str &amp;lt;- &quot; -26755.22-50150.60&quot;    &#xA;&#xA;&#xA;#you can use this command to split the data into 2 columns&#xA;#this doesn't have any constraint in the number of digits before and after the decimal&#xA;strspl &amp;lt;- as.numeric(unlist(regmatches(str,gregexpr(&quot;(?&amp;gt;-)*[[:digit:]]+\\.*[[:digit:]]*&quot;,str, perl=TRUE))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Outcome:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; strspl&#xA;[1] -26755.22 -50150.60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to store the outcome in a data-frame then you can use the below command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#everything is same but instead of as.numeric we replace it with as.data.frame&#xA;str_df &amp;lt;- as.data.frame(unlist(regmatches(str,gregexpr(&quot;(?&amp;gt;-)*[[:digit:]]+\\.*[[:digit:]]*&quot;,str, perl=TRUE))))&#xA;&#xA;#it is not necessary to change the column but it is a good practice to do so&#xA;colnames(str_df) &amp;lt;- &quot;variable_1&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Outcome:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/HwOOC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HwOOC.png&quot; alt=&quot;Outcome in Dataframe&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if you have any doubt, would help you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you got what you are looking for, you can accept the answer by clicking on the green tick mark.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-12T10:45:53.907" LastActivityDate="2018-03-12T10:45:53.907" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="28968" PostTypeId="1" AcceptedAnswerId="28974" CreationDate="2018-03-12T10:31:22.503" Score="1" ViewCount="3986" Body="&lt;p&gt;Hi I am studying tensorflow for cifar-10 image classification using the code &lt;a href=&quot;https://github.com/exelban/tensorflow-cifar-10&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35597" LastEditorUserId="35597" LastEditDate="2018-03-26T10:02:30.177" LastActivityDate="2018-03-26T10:02:30.177" Title="Get the probabilities of Tensorflow" Tags="|machine-learning|neural-network|deep-learning|tensorflow|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28969" PostTypeId="2" ParentId="28964" CreationDate="2018-03-12T10:34:39.370" Score="1" Body="&lt;p&gt;I just did this-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; library(stringr)&#xA;&amp;gt; x &amp;lt;- c(&quot;-26755.22-50150.60&quot;)&#xA;&amp;gt; matc = &quot;\\-[0-9]*\\.[0-9]{2}&quot;&#xA;&amp;gt; as.numeric(unlist(str_match_all(x, matc)))&#xA;[1] -26755.22 -50150.60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After this, you can select the numbers by using respective indexes.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-12T10:34:39.370" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28971" PostTypeId="1" AcceptedAnswerId="41379" CreationDate="2018-03-12T10:45:15.853" Score="3" ViewCount="3912" Body="&lt;p&gt;I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: &lt;a href=&quot;http://imanmalik.com/cs/2017/06/05/neural-style.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://imanmalik.com/cs/2017/06/05/neural-style.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for &lt;strong&gt;I., a note activation array&lt;/strong&gt;; for &lt;strong&gt;II. the predicted velocities&lt;/strong&gt; and for &lt;strong&gt;III. the true, expected velocities&lt;/strong&gt;). As you can see, the &lt;em&gt;predicted&lt;/em&gt; velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xK1hh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xK1hh.png&quot; alt=&quot;y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Using TensorFlow backend.&#xA;Loading numpy data ...&#xA;173 train sequences&#xA;10 test sequences&#xA;x_train shape: (173,)&#xA;y_train shape: (173,)&#xA;Setting up model ...&#xA;_________________________________________________________________&#xA;Layer (type)                 Output Shape              Param #&#xA;=================================================================&#xA;bidirectional_1 (Bidirection (4, None, 128)            123392&#xA;_________________________________________________________________&#xA;bidirectional_2 (Bidirection (4, None, 128)            98816&#xA;_________________________________________________________________&#xA;bidirectional_3 (Bidirection (4, None, 128)            98816&#xA;_________________________________________________________________&#xA;time_distributed_1 (TimeDist (4, None, 88)             11352&#xA;=================================================================&#xA;Total params: 332,376&#xA;Trainable params: 332,376&#xA;Non-trainable params: 0&#xA;_________________________________________________________________&#xA;None&#xA;Training model ...&#xA;Epoch 1/1&#xA;2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX&#xA;44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783&#xA;Saving model ...&#xA;Loss and metrics: [0.002714238129556179, 0.0027141105383634567]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;number_of_notes = 88&#xA;input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained).&#xA;output_size = number_of_notes # 88 notes.&#xA;&#xA;units = 64&#xA;dropout = 0.2 # Drop 20% of units for linear transformation of inputs.&#xA;&#xA;model = Sequential()&#xA;model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout),&#xA;                        input_shape=(None,input_size),&#xA;                        batch_input_shape=(args.batch_size,None,input_size)))&#xA;model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))&#xA;model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))&#xA;model.add(TimeDistributed(Dense(output_size, activation='sigmoid')))&#xA;model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse'])&#xA;&#xA;print('Training model ...')&#xA;&#xA;number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size))&#xA;model.fit_generator(batch_generator(x_train, y_train),&#xA;                    steps_per_epoch=number_of_train_batches,&#xA;                    epochs=args.epochs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is how I predict:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prediction_data = np.load(path)&#xA;&#xA;# Copy prediction input N times to create a batch of the right size.&#xA;tiled = np.tile(prediction_data, [args.batch_size,1,1])&#xA;&#xA;raw_prediction = model.predict(tiled, batch_size=args.batch_size)[0]&#xA;prediction = (raw_prediction * 127).astype(int) # Float -&amp;gt; MIDI velocity.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!&lt;/p&gt;&#xA;" OwnerUserId="47637" LastActivityDate="2018-11-18T22:30:31.557" Title="My Keras bidirectional LSTM model is giving terrible predictions" Tags="|keras|tensorflow|rnn|lstm|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28972" PostTypeId="1" AcceptedAnswerId="29106" CreationDate="2018-03-12T10:47:09.513" Score="1" ViewCount="280" Body="&lt;p&gt;I am working with the titanic dataset and using decision trees for analyzing the age covariate. I'd like just to see whether kids are more likely to survive than adults. I implemented my own Gini coefficient and I had plot the coefficient by age: dataset here &lt;a href=&quot;https://github.com/juan-barragan/titanic/blob/master/titanic_ds.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;titanic ds&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import seaborn&#xA;import matplotlib.pyplot as plt&#xA;from sklearn import tree&#xA;import graphviz&#xA;import numpy as np&#xA;&#xA;def gini_by_age(df, t):&#xA;    df['age_group'] = df['age'].apply(lambda row : 0 if row &amp;lt;= t else 1)&#xA;    kids = df[df['age_group'] == 0]&#xA;    kids0 = kids[kids['survived'] == 0]&#xA;    kids1 = kids[kids['survived'] == 1]&#xA;    adults = df[df['age_group'] == 1]&#xA;    adults0 = adults[adults['survived'] == 0]&#xA;    adults1 = adults[adults['survived'] == 1] &#xA;    gk = 1 - (len(kids0)**2 + len(kids1)**2)/float(len(kids))**2&#xA;    ga = 1 - (len(adults0)**2 + len(adults1)**2)/float(len(adults))**2&#xA;&#xA;    return gk + ga&#xA;&#xA;def plot_gini_by_age(df):    &#xA;    ages = range(2,25)&#xA;    y = [gini_by_age(df, a) for a in ages]&#xA;    plt.plot(ages, y)&#xA;    plt.show()&#xA;&#xA;def use_tree(df):&#xA;    X = np.array(df['age']).reshape((len(df['age']),1))&#xA;    y = df['survived']&#xA;    clf = tree.DecisionTreeClassifier(max_depth=1).fit(X,y)    &#xA;    dot_data = tree.export_graphviz(clf, out_file=None)&#xA;    graph = graphviz.Source(dot_data)&#xA;    graph.render(&quot;age&quot;)&#xA;&#xA;titanic_df = pd.read_csv(&quot;titanic_ds.csv&quot;)&#xA;ages_cov = titanic_df[['age', 'survived']].dropna()&#xA;plot_gini_by_age(ages_cov)&#xA;use_tree(ages_cov)&#xA;print gini_by_age(ages_cov, 5)&#xA;print gini_by_age(ages_cov, 8.5)&#xA;print gini_by_age(ages_cov, 15)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output:&#xA;    0.925844132419&#xA;    0.937732003001&#xA;    0.963875889772&#xA;I see from the plot that gini coefficient has local minima at roughly 5, 8 and 15 years old and the best is at 5. But scikit gives me 8.5 years old as the best split. What is wrong here?&lt;/p&gt;&#xA;" OwnerUserId="47635" LastActivityDate="2018-03-15T06:52:18.260" Title="Decision Trees split in scikit" Tags="|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28973" PostTypeId="2" ParentId="28957" CreationDate="2018-03-12T10:47:28.310" Score="1" Body="&lt;p&gt;I didn't get what you meant by &quot;other LDA Methods&quot;. To the best of my knowledge, Fisher method is just one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sklearn has the implementation and you can find it &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it good or bad? When it comes to dimensionality reduction you don't know as dimensionality reduction methods are usually unsupervised. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_vs_lda.html#sphx-glr-auto-examples-decomposition-plot-pca-vs-lda-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;So if PCA works better? No one knows&lt;/a&gt;. At least in the context of two-class classification, Fisher method is still brilliant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helped!&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-03-12T10:47:28.310" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28974" PostTypeId="2" ParentId="28968" CreationDate="2018-03-12T10:47:40.553" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to ask you that how to predict the probabilities of each class in test images.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;At line 27 in the &lt;em&gt;train.py&lt;/em&gt; you have the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;correct_prediction = tf.equal(y_pred_cls, tf.argmax(y, axis=1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It tries to find whether the predicted values are the same as the real ones. You can run y_pred_cls to see the probability of each class for your desired input.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to use the code to predict the probabilities of new data's labels, how to save and load the model we have trained which used the train data.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;for saving your model and its weights you can take a look at &lt;a href=&quot;https://stackoverflow.com/q/33759623/5120235&quot;&gt;here&lt;/a&gt;. As you can see from there, you have to make a saver object:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;#Prepare to feed input, i.e. feed_dict and placeholders&#xA;w1 = tf.placeholder(&quot;float&quot;, name=&quot;w1&quot;)&#xA;w2 = tf.placeholder(&quot;float&quot;, name=&quot;w2&quot;)&#xA;b1= tf.Variable(2.0,name=&quot;bias&quot;)&#xA;feed_dict ={w1:4,w2:8}&#xA;&#xA;#Define a test operation that we will restore&#xA;w3 = tf.add(w1,w2)&#xA;w4 = tf.multiply(w3,b1,name=&quot;op_to_restore&quot;)&#xA;sess = tf.Session()&#xA;sess.run(tf.global_variables_initializer())&#xA;&#xA;#Create a saver object which will save all the variables&#xA;saver = tf.train.Saver()&#xA;&#xA;#Run the operation by feeding input&#xA;print sess.run(w4,feed_dict)&#xA;#Prints 24 which is sum of (w1+w2)*b1 &#xA;&#xA;#Now, save the graph&#xA;saver.save(sess, 'my_test_model',global_step=1000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And for loading that you have to restore it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;sess=tf.Session()    &#xA;#First let's load meta graph and restore weights&#xA;saver = tf.train.import_meta_graph('my_test_model-1000.meta')&#xA;saver.restore(sess,tf.train.latest_checkpoint('./'))&#xA;&#xA;&#xA;# Access saved Variables directly&#xA;print(sess.run('bias:0'))&#xA;# This will print 2, which is the value of bias that we saved&#xA;&#xA;&#xA;# Now, let's access and create placeholders variables and&#xA;# create feed-dict to feed new data&#xA;&#xA;graph = tf.get_default_graph()&#xA;w1 = graph.get_tensor_by_name(&quot;w1:0&quot;)&#xA;w2 = graph.get_tensor_by_name(&quot;w2:0&quot;)&#xA;feed_dict ={w1:13.0,w2:17.0}&#xA;&#xA;#Now, access the op that you want to run. &#xA;op_to_restore = graph.get_tensor_by_name(&quot;op_to_restore:0&quot;)&#xA;&#xA;print sess.run(op_to_restore,feed_dict)&#xA;#This will print 60 which is calculated &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;hr /&gt;&#xA;Edit: Actually the code is a bit strange, anyway. The following part:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;can help you. It outputs the probabilities for each class.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-03-12T11:05:10.513" LastActivityDate="2018-03-12T11:05:10.513" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="28975" PostTypeId="2" ParentId="27767" CreationDate="2018-03-12T11:07:13.450" Score="12" Body="&lt;p&gt;I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was obtained as 20GB gzipped CSV files from energy meters, time series data at several seconds intervals. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;File IO: &lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_root = r&quot;/media/usr/USB STICK&quot;&#xA;fname = r&quot;meters001-050-timestamps.csv.gz&quot;&#xA;this_file = os.path.join(data_root,fname)&#xA;assert os.path.exists(this_file), this_file&#xA;this_file&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Create a chunk iterator directly over the gzip file (do not unzip!)&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cols_to_keep = [0,1,2,3,7]&#xA;column_names = ['METERID','TSTAMP','ENERGY','POWER_ALL','ENERGY_OUT',]&#xA;parse_dates = ['TSTAMP']&#xA;dtype={'METERID': np.int32, &#xA;       'ENERGY': np.int32,&#xA;       'POWER_ALL': np.int32,&#xA;       'ENERGY_OUT': np.int32,&#xA;      }&#xA;df_iterator = pd.read_csv(this_file, &#xA;                        skiprows=0, &#xA;                        compression='gzip',&#xA;                        chunksize=1000000, &#xA;                        usecols=cols_to_keep,&#xA;                        delimiter=&quot;;&quot;,&#xA;                        header=None,&#xA;                        names = column_names,&#xA;                      dtype=dtype,&#xA;                     parse_dates=parse_dates,&#xA;                     index_col=1,&#xA;                     )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Iterate over the chunks&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;new_df = pd.DataFrame()&#xA;count = 0&#xA;for df in df_iterator:&#xA;    chunk_df_15min = df.resample('15T').first()&#xA;    #chunk_df_30min = df.resample('30T').first()&#xA;    #chunk_df_hourly = df.resample('H').first()&#xA;    this_df = chunk_df_15min&#xA;    this_df = this_df.pipe(lambda x: x[x.METERID == 1])&#xA;    #print(&quot;chunk&quot;,i)&#xA;    new_df = pd.concat([new_df,chunk_df_15min])&#xA;    print(&quot;chunk&quot;,count, len(chunk_df_15min), 'rows added')&#xA;    #print(&quot;chunk&quot;,i, len(temp_df),'rows added')&#xA;    #break&#xA;    count += 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Inside the chunk loop, I am doing some filtering and re-sampling on time. Doing this I reduced the size from 20GB to a few hundred MB HDF5 for further offline data exploration. &lt;/p&gt;&#xA;" OwnerUserId="47636" LastEditorUserId="47636" LastEditDate="2018-10-24T13:00:33.433" LastActivityDate="2018-10-24T13:00:33.433" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="28977" PostTypeId="2" ParentId="28451" CreationDate="2018-03-12T11:35:01.970" Score="1" Body="&lt;p&gt;Maybe sequential association rules could be applied here. Please check &lt;a href=&quot;http://www.philippe-fournier-viger.com/spmf/index.php?link=documentation.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;spmf&lt;/a&gt; java library, section 'Sequential Rule Mining'.&lt;/p&gt;&#xA;" OwnerUserId="43473" LastActivityDate="2018-03-12T11:35:01.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28978" PostTypeId="1" AcceptedAnswerId="28980" CreationDate="2018-03-12T11:45:59.167" Score="0" ViewCount="464" Body="&lt;p&gt;if Word2Vec is nothing but a transformation of one-hot into a dense vector, why can't I just feed one-hot into LSTM (or for that matter sacrifice first dense layer, in any network that will end up using the embedding) and call it a day?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would I actually spend time pre-computing Word2Vec embeddings?&#xA;Yes, the resulting embedings are vectors that are clustered together if the words have similar meaning. But I presume a feed-forward classifier would figure out a good internal model anyway?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-03-12T11:57:13.533" LastActivityDate="2018-03-12T12:07:30.823" Title="Word2Vec benefit in LSTM" Tags="|neural-network|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28980" PostTypeId="2" ParentId="28978" CreationDate="2018-03-12T12:07:30.823" Score="1" Body="&lt;p&gt;You don't need word embeddings. Actually, in neural machine translation is frequent not to use them and simply train the embeddings along with the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, word embeddings work as a &lt;em&gt;data augmentation&lt;/em&gt; technique, as you normally use a different (and much larger) dataset to train them, so they can be useful when you don't have much training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the decision of using pre-trained word embeddings or not should be driven by the available data.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-12T12:07:30.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28981" PostTypeId="2" ParentId="28966" CreationDate="2018-03-12T12:56:02.250" Score="2" Body="&lt;p&gt;Convolutional neural networks implement the convolutional filter technique. This is more or less a sliding window, usually (but not limited to) convolving across the width, heights and depths of images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a single task or multi learning learning task depends on what you want this network to output. Convolution rather describes the behavior of a single layer processing some inputs into some intermediate outputs. The characteristics of your last layer determine if it is used for single or multi tasks. To my understanding this is rather independent of the convolution.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastActivityDate="2018-03-12T12:56:02.250" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28982" PostTypeId="1" CreationDate="2018-03-12T13:05:48.243" Score="1" ViewCount="366" Body="&lt;p&gt;What is the current state of the art when it comes to extracting tabular data from documents (images/ocr, not pdf/html)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come to ICDAR, but the last table detection challenges were from 2015.&lt;/p&gt;&#xA;" OwnerUserId="43384" LastActivityDate="2018-03-12T13:05:48.243" Title="Table extraction state of the art (beginning of 2018)" Tags="|machine-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="28983" PostTypeId="1" CreationDate="2018-03-12T15:04:48.993" Score="1" ViewCount="261" Body="&lt;p&gt;I am writing a paper about machine learning and I need to create some neural network diagrams and basic math functions I am describing. I need a program to create visually decent technical picture without spending days to learn how to use the program. &lt;/p&gt;&#xA;" OwnerUserId="47659" LastActivityDate="2024-02-12T23:05:40.673" Title="What program to use to visualise neural network diagram and math functions" Tags="|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="28984" PostTypeId="1" AcceptedAnswerId="29005" CreationDate="2018-03-12T15:37:06.910" Score="0" ViewCount="44" Body="&lt;p&gt;I have read some articles about &lt;em&gt;image hashing&lt;/em&gt;, and I would like to know if we could apply this technique for general purpose images classification tasks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially I would like to know which could be the drawbacks in using image hashing for this kind of tasks.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2018-03-13T05:31:40.923" Title="Could we use image hasing techniques for images classification tasks?" Tags="|image-classification|image-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28985" PostTypeId="1" CreationDate="2018-03-12T16:33:12.450" Score="1" ViewCount="140" Body="&lt;p&gt;I was able to install &lt;a href=&quot;https://github.com/biolab/orange3/blob/master/doc/development/source/tutorial-channels.rst&quot; rel=&quot;nofollow noreferrer&quot;&gt;the demo widget&lt;/a&gt; in Orange 3.9.1 and made it work with these learners&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Naive bayes&lt;/li&gt;&#xA;&lt;li&gt;Logistic regression&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In other words I was able to connect the nodes to the newly installed &quot;Learning curve A&quot; widget.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the catch: Those two were the only I was able to connect.&#xA;For example, I can't connect the tree learner or the random forest learnear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see what I mean from the picture below.&#xA;Find here &lt;a href=&quot;https://github.com/biolab/orange3/blob/master/doc/development/source/orange-demo/orangedemo/OWLearningCurveA.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;the source&lt;/a&gt; of the demo widget&#xA;In the tutorial you can see other learners are connected successfully.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea why I can't connect the other learners? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jiZAi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jiZAi.png&quot; alt=&quot;orange canvas&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44760" LastEditorUserId="44760" LastEditDate="2018-03-12T18:42:29.830" LastActivityDate="2018-03-12T18:42:29.830" Title="Orange Biolab custom widget learner input" Tags="|orange|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28987" PostTypeId="1" CreationDate="2018-03-12T17:49:31.750" Score="1" ViewCount="7743" Body="&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;from matplotlib import style&#xA;style.use(&quot;ggplot&quot;)&#xA;from sklearn import svm&#xA;&#xA;&#xA;X=[[1,0,0,0,0],&#xA;           [0,1,0,0,0],&#xA;           [0,0,1,0,0],&#xA;           [1,0,0,1,0],&#xA;           [1,0,0,0,1]] &#xA;&#xA;y=[0,1,1,1,0]&#xA;&#xA;model=svm.SVC()&#xA;model.fit(X,y)&#xA;print(model.predict([1,0,1,0,0]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was working on this but Iam getting an error as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;if it contains a single sample.&quot;.format(array))&#xA;ValueError: Expected 2D array, got 1D array instead:&#xA;array=[1. 0. 1. 0. 0.].&#xA;Reshape your data either using array.reshape(-1, 1) if your data has a single fe&#xA;ature or array.reshape(1, -1) if it contains a single sample.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Iam new to this can u guys help me out?&lt;/p&gt;&#xA;" OwnerUserId="47670" LastActivityDate="2018-04-12T16:28:15.713" Title="SVM with multiple features" Tags="|python|classification|scikit-learn|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28988" PostTypeId="1" CreationDate="2018-03-12T18:07:07.893" Score="2" ViewCount="1321" Body="&lt;p&gt;When I am doing cross validation using Python's Sklearn and take the score of different metrics (accuracy, precision, etc.) like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;result_accuracy = cross_val_score(classifier, X_train, y_train, scoring='accuracy', cv=10)&#xA;result_precision = cross_val_score(classifier, X_train, y_train, scoring='precision', cv=10)&#xA;result_recall = cross_val_score(classifier, X_train, y_train, scoring='recall', cv=10)&#xA;result_f1 = cross_val_score(classifier, X_train, y_train, scoring='f1', cv=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Did every execution of &lt;code&gt;cross_val_score()&lt;/code&gt; function for different metrics made the same 10 folds of the training data or not? If not, do I need to make the general 10-folds first using KFold, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;seed = 7&#xA;kf = KFold(n_splits=10, random_state=seed)&#xA;&#xA;result_accuracy = cross_val_score(classifier, X_train, y_train, scoring='accuracy', cv=kf)&#xA;result_precision = cross_val_score(classifier, X_train, y_train, scoring='precision', cv=kf)&#xA;result_recall = cross_val_score(classifier, X_train, y_train, scoring='recall', cv=kf)&#xA;result_f1 = cross_val_score(classifier, X_train, y_train, scoring='f1', cv=kf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43592" LastActivityDate="2018-03-12T19:55:47.983" Title="Cross Validation for Different Metrics - Sklearn" Tags="|python|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28991" PostTypeId="2" ParentId="28987" CreationDate="2018-03-12T18:39:35.603" Score="0" Body="&lt;p&gt;Your inputs are lists but they need to be arrays. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X=np.array([[1,0,0,0,0],&#xA;           [0,1,0,0,0],&#xA;           [0,0,1,0,0],&#xA;           [1,0,0,1,0],&#xA;           [1,0,0,0,1]])&#xA;&#xA;y=np.array([0,1,1,1,0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-03-12T18:39:35.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28992" PostTypeId="1" CreationDate="2018-03-12T19:27:19.440" Score="1" ViewCount="651" Body="&lt;p&gt;I'm trying to fine-tune a pre-trained InceptionV3 on the &lt;a href=&quot;https://lampsrv02.umiacs.umd.edu/projdb/project.php?id=72&quot; rel=&quot;nofollow noreferrer&quot;&gt;tobacco-3482 document dataset&lt;/a&gt; (I'm only using the first 6 classes), but I'm getting accuracies under 20% on the validation set (&gt; 90% accuracy on the training set). I've tried numerous batch sizes, epochs, etc., any ideas? Here is my code for Keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import sys&#xA;import os&#xA;from keras.layers import *&#xA;from keras.optimizers import *&#xA;from keras.applications import *&#xA;from keras.models import Model&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras.callbacks import ModelCheckpoint, EarlyStopping&#xA;from keras import backend as k&#xA;import keras&#xA;import glob&#xA;&#xA;def get_nb_files(directory):&#xA;  &quot;&quot;&quot;Get number of files by searching directory recursively&quot;&quot;&quot;&#xA;  if not os.path.exists(directory):&#xA;    return 0&#xA;  cnt = 0&#xA;  for r, dirs, files in os.walk(directory):&#xA;    for dr in dirs:&#xA;      cnt += len(glob.glob(os.path.join(r, dr + &quot;/*&quot;)))&#xA;  return cnt&#xA;&#xA;# fix seed for reproducible results (only works on CPU, not GPU)&#xA;seed = 9&#xA;np.random.seed(seed=seed)&#xA;tf.set_random_seed(seed=seed)&#xA;&#xA;# hyper parameters for model&#xA;nb_classes = 6  # number of classes&#xA;based_model_last_block_layer_number = 126  # value is based on based model selected.&#xA;img_width, img_height = 299, 299  # change based on the shape/structure of your images&#xA;batch_size = 32  # try 4, 8, 16, 32, 64, 128, 256 dependent on CPU/GPU memory capacity (powers of 2 values).&#xA;nb_epoch = 50  # number of iteration the algorithm gets trained.&#xA;learn_rate = 1e-4  # sgd learning rate&#xA;momentum = .9  # sgd momentum to avoid local minimum&#xA;transformation_ratio = 0  # how aggressive will be the data augmentation/transformation&#xA;nb_train_samples = get_nb_files('tobacco')  # Total number of train samples. NOT including augmented images&#xA;nb_validation_samples = get_nb_files('tobacco-test')  # Total number of train samples. NOT including augmented images.&#xA;&#xA;&#xA;def train(train_data_dir, validation_data_dir, model_path):&#xA;    if K.image_data_format() == 'channels_first':&#xA;        input_shape = (3, img_width, img_height)&#xA;    else:&#xA;        input_shape = (img_width, img_height, 3)&#xA;&#xA;    base_model = InceptionV3(input_shape=input_shape, weights='imagenet', include_top=False)&#xA;&#xA;    x = base_model.output&#xA;    x = GlobalAveragePooling2D()(x)&#xA;    x = Dense(256, activation='relu', name='fc1')(x)&#xA;    x = Dropout(0.5)(x)&#xA;&#xA;    predictions = Dense(nb_classes, activation='softmax', name='predictions')(x)&#xA;&#xA;    # add your top layer block to your base model&#xA;    model = Model(base_model.input, predictions)&#xA;    print(model.summary())&#xA;&#xA;&#xA;    # first: train only the top layers (which were randomly initialized)&#xA;    # i.e. freeze all layers of the based model that is already pre-trained.&#xA;    for layer in base_model.layers:&#xA;        layer.trainable = False&#xA;&#xA;    # Read Data and Augment it: Make sure to select augmentations that are appropriate to your images.&#xA;    # To save augmentations un-comment save lines and add to your flow parameters.&#xA;    train_datagen = ImageDataGenerator(rescale=1. / 255,&#xA;                                       rotation_range=transformation_ratio,&#xA;                                       shear_range=transformation_ratio,&#xA;                                       zoom_range=transformation_ratio,&#xA;                                       cval=transformation_ratio,&#xA;                                       horizontal_flip=False,&#xA;                                       vertical_flip=False)&#xA;&#xA;    validation_datagen = ImageDataGenerator(rescale=1. / 255)&#xA;&#xA;    train_generator = train_datagen.flow_from_directory(train_data_dir,&#xA;                                                        target_size=[img_width, img_height],&#xA;                                                        batch_size=batch_size,&#xA;                                                        class_mode='categorical')&#xA;&#xA;    validation_generator = validation_datagen.flow_from_directory(validation_data_dir,&#xA;                                                                  target_size=[img_width, img_height],&#xA;                                                                  batch_size=batch_size,&#xA;                                                                  class_mode='categorical')&#xA;&#xA;    model.compile(optimizer='nadam',&#xA;                  loss='categorical_crossentropy',  # categorical_crossentropy if multi-class classifier&#xA;                  metrics=['accuracy'])&#xA;&#xA;    # save weights of best training epoch: monitor either val_loss or val_acc&#xA;&#xA;    top_weights_path = os.path.join(os.path.abspath(model_path), 'top_model_weights.h5')&#xA;    callbacks_list = [&#xA;        ModelCheckpoint(top_weights_path, monitor='val_acc', verbose=1, save_best_only=True),&#xA;        EarlyStopping(monitor='val_acc', patience=5, verbose=0),&#xA;        keras.callbacks.TensorBoard(log_dir='tensorboard/inception-v3-train-top-layer', histogram_freq=0, write_graph=False, write_images=False)&#xA;    ]&#xA;&#xA;    # Train Simple CNN&#xA;    model.fit_generator(train_generator,&#xA;                        steps_per_epoch=nb_train_samples // batch_size,&#xA;                        epochs=nb_epoch / 5,&#xA;                        validation_data=validation_generator,&#xA;                        validation_steps=nb_validation_samples // batch_size,&#xA;                        callbacks=callbacks_list,&#xA;                        verbose=1)&#xA;&#xA;    # verbose&#xA;    print(&quot;\nStarting to Fine Tune Model\n&quot;)&#xA;&#xA;    # add the best weights from the train top model&#xA;    model.load_weights(top_weights_path)&#xA;&#xA;    for layer in model.layers[:based_model_last_block_layer_number]:&#xA;        layer.trainable = False&#xA;    for layer in model.layers[based_model_last_block_layer_number:]:&#xA;        layer.trainable = True&#xA;&#xA;    # compile the model with a SGD/momentum optimizer&#xA;    # and a very slow learning rate.&#xA;    model.compile(optimizer='nadam',&#xA;                  loss='categorical_crossentropy',&#xA;                  metrics=['accuracy'])&#xA;&#xA;    # save weights of best training epoch: monitor either val_loss or val_acc&#xA;    final_weights_path = os.path.join(os.path.abspath(model_path), 'model_weights.h5')&#xA;    callbacks_list = [&#xA;        ModelCheckpoint(final_weights_path, monitor='val_acc', verbose=1, save_best_only=True),&#xA;        EarlyStopping(monitor='val_loss', patience=5, verbose=0),&#xA;        keras.callbacks.TensorBoard(log_dir='tensorboard/inception-v3-fine-tune', histogram_freq=0, write_graph=False, write_images=False)&#xA;    ]&#xA;&#xA;    # fine-tune the model&#xA;    model.fit_generator(train_generator,&#xA;                        steps_per_epoch=nb_train_samples // batch_size,&#xA;                        epochs=nb_epoch,&#xA;                        validation_data=validation_generator,&#xA;                        validation_steps=nb_validation_samples // batch_size,&#xA;                        callbacks=callbacks_list,&#xA;                        verbose=1)&#xA;&#xA;    # save model&#xA;    model_json = model.to_json()&#xA;    with open(os.path.join(os.path.abspath(model_path), 'model.json'), 'w') as json_file:&#xA;        json_file.write(model_json)&#xA;&#xA;&#xA;train('tobacco', 'tobacco-test', '.')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="245" LastActivityDate="2018-03-12T19:27:19.440" Title="Pretrained InceptionV3 - very low accuracy on Tobacco dataset" Tags="|keras|inception|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="28993" PostTypeId="1" CreationDate="2018-03-12T19:30:21.757" Score="0" ViewCount="47" Body="&lt;p&gt;Assume we train a KMeans model using data X. This will give a set of centroids that can be used to cluster data X* using a Nearest Centroid Classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we use a density-based model such as DBSCAN to train a model using data X, how can we use it to classify future data X*? Is KNN suitable?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="47676" LastActivityDate="2018-04-12T08:08:59.670" Title="Testing unsupervised clustering" Tags="|classification|clustering|k-means|dbscan|k-nn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28994" PostTypeId="2" ParentId="28988" CreationDate="2018-03-12T19:55:47.983" Score="2" Body="&lt;p&gt;It only really matters if you want to shuffle your data in the cross-validation. The default for both &lt;code&gt;cross_val_score&lt;/code&gt; and &lt;code&gt;KFold&lt;/code&gt; is to NOT shuffle.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do want to shuffle your second option is best if you want to make sure that you are comparing scores across the same splits on the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind that in the default &lt;code&gt;KFold(shuffle=False)&lt;/code&gt;, so if you want to shuffle your data make sure that you set Kfold to:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;kf = KFold(n_splits=10, shuffle=True, random_state=seed)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="47677" LastActivityDate="2018-03-12T19:55:47.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28995" PostTypeId="2" ParentId="28993" CreationDate="2018-03-12T22:23:25.567" Score="1" Body="&lt;p&gt;General speaking, you can assign the cluster to the new value using the method of that clustering method. For example, if you are using the DBSACAN algorithm, you can assign the cluster to the new value, if it is reachable from one core points of a cluster (in a tie situation, we can assign the more density cluster or many other methods).&lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-03-12T22:23:25.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28996" PostTypeId="1" CreationDate="2018-03-13T00:08:48.223" Score="2" ViewCount="178" Body="&lt;p&gt;I have a dataset with following data format:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;3 -&amp;gt; a -&amp;gt; b -&amp;gt; c -&amp;gt; d -&amp;gt; ikd&lt;/p&gt;&#xA;&lt;p&gt;a -&amp;gt; c -&amp;gt; 3 -&amp;gt; dk -&amp;gt; 2 -&amp;gt; l2i&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Each row represents a path from start to end. Let's take the first row as an example. The start point is &lt;code&gt;3&lt;/code&gt; and the endpoint is &lt;code&gt;ikd&lt;/code&gt;. I have millions of rows like that. And each row may have a different length. What I want to do is let users input a path with the above format and I predict a new path based on the existent dataset. The idea is to find the closest path for each user. Each node on the path may have a weight to indicate how important this node is. So the closest path should respect this weight.&lt;/p&gt;&#xA;&lt;p&gt;Is there any ML algorithm for this kind of problem? I am new to ML so I need some guidance on solving this problem.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; There is no specific meaning for the type of each node, like number, letter, etc. All these nodes are of type string and the user's input is from an existent string list.&lt;/p&gt;&#xA;" OwnerUserId="47687" LastEditorUserId="120463" LastEditDate="2021-08-19T04:05:19.947" LastActivityDate="2024-01-26T08:02:00.283" Title="Looking for a ML algorithm to predict a path based on millions of data" Tags="|machine-learning|dataset|predictive-modeling|prediction|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="28997" PostTypeId="1" CreationDate="2018-03-13T00:18:43.920" Score="2" ViewCount="194" Body="&lt;p&gt;I've just realized my prediction approach for LSTM might not be correct. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to predict character by character, by reading over the book. The way I've approached the problem is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;   b                                    c                d                e&#xA;   ^     carry cell state forward       ^                ^                ^&#xA;LSTM_t0  -------------------------&amp;gt;  LSTM_t1  -----&amp;gt;  LSTM_t2  -----&amp;gt;  LSTM_t3&#xA;   ^                                    ^                ^                ^&#xA;   a                                    b                c                d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This means I have 4 timesteps, at at each one I feed next letter into LSTM, expecting it to immediately predict the next letter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I instead do this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;ignore          ignore           ignore             e&#xA;  ^               ^                ^                ^&#xA;LSTM_t0  ----&amp;gt;  LSTM_t1  -----&amp;gt;  LSTM_t2  -----&amp;gt;  LSTM_t3&#xA;  ^               ^                ^                ^&#xA;  a               b                c                d&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In the first case, I am able to get 4 loss-values, but in the second example, I only have 1 source of gradient, at _t3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main concern is in first example, I demand LSTM to make prediction of 'b' and 'c' without supplying it enough previous context.  It's fine for 'd' and 'e', but asking for answer at timestep 0 and 1 is a bit unfair? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would be best for this particular example?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-04-26T08:17:52.867" LastActivityDate="2018-12-26T01:01:25.707" Title="LSTM querying approach" Tags="|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="28999" PostTypeId="1" AcceptedAnswerId="29016" CreationDate="2018-03-13T01:16:57.590" Score="0" ViewCount="952" Body="&lt;p&gt;Suppose I have a data frame with 2 columns, which are sales and promotions. I want to predict the next day sales based on the past sales and promotion info of 3 days, plus the promotion to be applied at the next day? How do I process and reshape the dataframe? I mean if only previous promotions need to be considered, then, after some shifting, the data frame could be reshaped into (sample size, 3, 2), but it becomes a problem if I also need to consider the promotion at the next day. It is a pretty common issue, does anyone has any thought about this?&lt;/p&gt;&#xA;" OwnerUserId="47693" LastActivityDate="2018-03-13T09:46:35.683" Title="How to use LSTM to make prediction with both feature from the past and the current ones?" Tags="|machine-learning|python|keras|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29001" PostTypeId="1" AcceptedAnswerId="29004" CreationDate="2018-03-13T03:35:20.723" Score="0" ViewCount="68" Body="&lt;p&gt;I have a problem at hand to identify Good/Bad Products using given parameters. The number of parameters are in the order of 5000s and there are multiple values for the parameters. However I do not have a labelled set of data which says these are the products that are good or bad.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Example, Say the parameters are AX, AY, AZ, B, C, DX, DY, etc.&#xA;Each of them has a different range. Is decision trees the right approach? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can classification be applied to this problem?&lt;/p&gt;&#xA;" OwnerUserId="31443" LastEditorUserId="31443" LastEditDate="2018-03-15T22:21:04.883" LastActivityDate="2018-03-15T22:21:04.883" Title="Identify Bad Products from given parameters using neural networks" Tags="|neural-network|classification|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29002" PostTypeId="2" ParentId="25405" CreationDate="2018-03-13T03:53:04.517" Score="1" Body="&lt;p&gt;Yes you could add a time component if you have dates in your data. Use the functions found in this &lt;a href=&quot;https://www.google.co.in/search?q=date%20time%20r&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;, for your work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially you would want to see if there are day based effects such as Are deliveries late on Fridays / Friday evenings / Early mornings / Late nights / are etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you'd have to do for that is to strip out the day of the week from the relevant date variables and add it as a ordinal predictor to your data. If you want to strip out individual weekday effects, use an &lt;code&gt;if else&lt;/code&gt; function to classify the day of the week as Monday / Not Monday, Tuesday / Not Tuesday, etc. You could do something similar to month of the year, week of the year, season, etc. &lt;/p&gt;&#xA;" OwnerUserId="47698" LastEditorUserId="29575" LastEditDate="2018-03-13T04:14:51.060" LastActivityDate="2018-03-13T04:14:51.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29004" PostTypeId="2" ParentId="29001" CreationDate="2018-03-13T05:26:38.453" Score="2" Body="&lt;p&gt;No.  Classification requires labelled data.  Without labelled data there is no way to solve this.  How would you do anything at all, if you don't know which of the products in the training set are good and which are bad?  There's no basis for making a decision of any sort.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-13T05:26:38.453" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29005" PostTypeId="2" ParentId="28984" CreationDate="2018-03-13T05:31:40.923" Score="1" Body="&lt;p&gt;You could, but it won't be very effective.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image hashing is aimed at detecting two instances of almost the same image.  So, if your training set contains an image of a dog, and the test set contains an almost-identical image, then using image hashing you could use that to learn the label of the test set image.  But in practice that doesn't provide much generalization power.  In practice, we want to take hundreds of different images of dogs, and use that to learn to recognize new images of docs, so that if in the test set we are given a totally new image of a dog, we can still classify it as a dog.  Image hashing won't help with that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, image hashing isn't designed for this sort of thing and will work poorly.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-13T05:31:40.923" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29006" PostTypeId="1" AcceptedAnswerId="29018" CreationDate="2018-03-13T05:32:15.480" Score="21" ViewCount="34985" Body="&lt;p&gt;Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am  right. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which technique would be &lt;em&gt;preferred&lt;/em&gt; over the other and when? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking, since feature selection does not modify the original data and it's properties, I assume that you will use feature selection when it's important that the features you're training on be unchanged. But I can't imagine why you would want something like this.. &lt;/p&gt;&#xA;" OwnerUserId="32288" LastEditorUserId="35644" LastEditDate="2019-02-01T02:43:11.763" LastActivityDate="2020-02-23T01:17:32.997" Title="Feature selection vs Feature extraction. Which to use when?" Tags="|feature-selection|feature-extraction|dimensionality-reduction|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29009" PostTypeId="2" ParentId="29006" CreationDate="2018-03-13T06:15:50.560" Score="4" Body="&lt;p&gt;I think they are 2 different things,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets start with &lt;strong&gt;Feature Selection&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This technique is used for selecting the features which explain the most of the target variable(has a correlation with the target variable).This test is ran just before the model is applied on the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain it better let us go by an example: there are 10 feature and 1 target variable, 9 features explain 90% of the target variable and 10 features together explains 91% of the target variable. So the 1 variable is not making much of a difference so you tend to remove that before modelling(It is subjective to the business as well). I can also be called as Predictor Importance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now lets talk about &lt;strong&gt;Feature Extraction&lt;/strong&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is used in Unsupervised Learning,extraction of contours in images, extraction of Bi-grams from a text, extraction of phonemes from recording of spoken text. &#xA;When you don't know anything about the data like no data dictionary, too many features which means the data is not in understandable format. Then you try applying this technique to get some features which explains the most of the data. Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can apply Feature Extraction on the given data to extract features and then apply Feature Selection with respect to the Target Variable to select the subset which can help in making a good model with good results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you can go through these &lt;a href=&quot;https://www.featurelabs.com/blog/feature-engineering-vs-feature-selection/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-1&lt;/a&gt;,&lt;a href=&quot;https://datascience.stackexchange.com/questions/130/what-is-dimensionality-reduction-what-is-the-difference-between-feature-selecti&quot;&gt;Link-2&lt;/a&gt; for better understanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we can implement them in R, Python, SPSS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let me know if need any more clarification.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-13T06:45:36.560" LastActivityDate="2018-03-13T06:45:36.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29010" PostTypeId="2" ParentId="28993" CreationDate="2018-03-13T07:19:29.910" Score="0" Body="&lt;p&gt;Clustering is not predictive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A new data point could cause DBSCAN clusters to merge, so it could drastically change the result. So if you want to classify using clusters, it usually means you are solving the wrong problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really really need this (despite the warnings) just train any classifier on the cluster labels. For example, a SVM with RBF kernel could work very well for DBSCAN cluster (at least with Euclidean distance).&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-13T07:19:29.910" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29013" PostTypeId="2" ParentId="28942" CreationDate="2018-03-13T08:49:32.843" Score="1" Body="&lt;p&gt;For the first Q you already give a counterexample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is biased to k, it prefers k = N. And it will also overrate N-1, N-2, ... So it is not agnostic to k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If k-means would always find the best C index, then the C index would just be redundant to SSQ, which is much cheaper to compute... But you probably have just been looking at way too simple toy datasets. Use real data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-13T08:49:32.843" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29015" PostTypeId="2" ParentId="21643" CreationDate="2018-03-13T09:43:07.423" Score="1" Body="&lt;p&gt;From the same question you mentioned, you will find an interesting &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; to Keras documentation. If you navigate to &lt;strong&gt;Multi-input and multi-output models&lt;/strong&gt; you'll find an example code that does exactly this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EJ1uL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EJ1uL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-13T09:43:07.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29016" PostTypeId="2" ParentId="28999" CreationDate="2018-03-13T09:46:35.683" Score="0" Body="&lt;p&gt;Have a look at this &lt;a href=&quot;https://datascience.stackexchange.com/questions/17099/adding-features-to-time-series-model-lstm&quot;&gt;question&lt;/a&gt;. There is a nice discussion about how to implement this. The idea in to create a separated Input to your model and concatenate it AFTER the recurrent layer(s). Also in the Keras documentation, there is an &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; on how to build such models with a few lines of code. &lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-13T09:46:35.683" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29017" PostTypeId="2" ParentId="28997" CreationDate="2018-03-13T09:59:28.257" Score="1" Body="&lt;p&gt;Your first example is basically not a sequential model. You have an input and an output and that's it. You don't need a recurrent layer for that...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would suggest you do is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    e               f               g                         &#xA;    ^               ^               ^ &#xA; LSTM_t1  ----&amp;gt;  LSTM_t2  -----&amp;gt; LSTM_t3  -----&amp;gt;  ...&#xA;    ^               ^               ^  &#xA;  abcd            bcde            cdef               &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Obviously, you will lose the ability of predict the first characters but hopefully your dataset will include a lot more examples using those specific characters so it will learn how to use them eventually. I would say this is the most general way of producing char to char text generation models. A couple of examples including implementations: &lt;a href=&quot;https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2018/03/text-generation-using-python-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-13T09:59:28.257" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29018" PostTypeId="2" ParentId="29006" CreationDate="2018-03-13T10:00:33.400" Score="24" Body="&lt;p&gt;Adding to The answer given by Toros,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;feature extraction and feature engineering&lt;/strong&gt;: transformation of raw data into features suitable for modeling;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;feature transformation&lt;/strong&gt;: transformation of data to improve the accuracy of the algorithm;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;feature selection&lt;/strong&gt;: removing unnecessary features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Just to add an Example of the same,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Feature Extraction and Engineering(we can extract something from them)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Texts(ngrams, word2vec, tf-idf etc)&lt;/li&gt;&#xA;&lt;li&gt;Images(CNN'S, texts, q&amp;amp;a)&lt;/li&gt;&#xA;&lt;li&gt;Geospatial data(lat, long etc)&lt;/li&gt;&#xA;&lt;li&gt;Date and time(day, month, week, year, rolling based)&lt;/li&gt;&#xA;&lt;li&gt;Time series, web, etc&lt;/li&gt;&#xA;&lt;li&gt;Dimensional Reduction Techniques (PCA, SVD, Eigen-Faces etc)&lt;/li&gt;&#xA;&lt;li&gt;Maybe we can use Clustering as well (DBSCAN etc)&lt;/li&gt;&#xA;&lt;li&gt;.....(And Many Others)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Feature transformations(transforming them to make sense)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Normalization and changing distribution(Scaling)&lt;/li&gt;&#xA;&lt;li&gt;Interactions&lt;/li&gt;&#xA;&lt;li&gt;Filling in the missing values(median filling etc)&lt;/li&gt;&#xA;&lt;li&gt;.....(And Many Others)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Feature selection(building your model on these selected features)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Statistical approaches&lt;/li&gt;&#xA;&lt;li&gt;Selection by modeling&lt;/li&gt;&#xA;&lt;li&gt;Grid search &lt;/li&gt;&#xA;&lt;li&gt;Cross Validation&lt;/li&gt;&#xA;&lt;li&gt;.....(And Many Others)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do look at the links shared by others.&#xA;They are Quite Nice...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2020-02-23T01:17:32.997" LastActivityDate="2020-02-23T01:17:32.997" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="29019" PostTypeId="1" CreationDate="2018-03-13T10:05:57.480" Score="13" ViewCount="5259" Body="&lt;p&gt;Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix factorization of the ppmi of the co-occurrences of the words in the corpus. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, I can't understand why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V ? Is it a problem with the gradient descent or is there another reason ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone told me it might be because the embeddings u and v of one word should be far enough to represent the fact that a word rarely appears in its own context. But it is not clear to me.   &lt;/p&gt;&#xA;" OwnerUserId="17208" LastActivityDate="2021-05-14T23:01:57.073" Title="Why do we need 2 matrices for word2vec or GloVe" Tags="|word2vec|word-embeddings|matrix-factorisation|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29021" PostTypeId="1" CreationDate="2018-03-13T10:58:25.813" Score="1" ViewCount="351" Body="&lt;p&gt;I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you so much Yarin!):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/yaringal/ConcreteDropout&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/yaringal/ConcreteDropout&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;He implements a Keras Layer called Concrete Dropout, which can be applied to arbitrairy layers. Nevertheless all his examples are feed forward networks, so here is my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Can I use it in a straight forward manner to regularize RNNs?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example Code in Keras for this would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(ConcreteDropout(GRU(32)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2018-03-13T10:58:25.813" Title="Concrete Dropout for Recurrent Neural Networks (Keras)" Tags="|keras|rnn|regularization|dropout|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29023" PostTypeId="1" CreationDate="2018-03-13T12:00:27.870" Score="2" ViewCount="265" Body="&lt;p&gt;I have a code like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lstm_cell = tf.contrib.rnn.BasicLSTMCell(256, state_is_tuple = True)&#xA;c_in = tf.placeholder(tf.float32, [1, lstm_cell.state_size.c], &quot;c_in&quot;)&#xA;h_in = tf.placeholder(tf.float32, [1, lstm_cell.state_size.h], &quot;h_in&quot;)&#xA;rnn_state_in = (c_in, h_in)&#xA;rnn_in = tf.expand_dims(previous_layer, [0])&#xA;sequence_length = #size of my batch&#xA;rnn_state_in = tf.contrib.rnn.LSTMStateTuple(c_in, h_in)&#xA;lstm_outputs, lstm_state = tf.nn.dynamic_rnn(lstm_cell,&#xA;                                                    rnn_in,&#xA;                                                    initial_state = rnn_state_in,&#xA;                                                    sequence_length = sequence_length,&#xA;                                                    time_major = False)&#xA;lstm_c, lstm_h = lstm_state&#xA;rnn_out = tf.reshape(lstm_outputs, [-1, 256])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, I use dynamic_rnn to simulate the time steps from the batch.&#xA;While each forward pass, I can get &lt;code&gt;lstm_c, lstm_h&lt;/code&gt; which I can store anywhere outside.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, suppose I have done a forward pass for N items in a sequence in my network and have the final cell state and hidden state provided from the dynamic_rnn. Now, to perform back propagation, what should be my input to the LSTMs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By default, &lt;strong&gt;does backprop happen across time steps in dynamic_rnn?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(say, no. of time steps = batch_size=N)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So is it enough for me to provide the input as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sess.run(_train_op, feed_dict = {_state: np.vstack(batch_states),&#xA;                                            ...&#xA;                                            c_in: prev_rnn_state[0],&#xA;                                            h_in: prev_rnn_state[1]&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(where &lt;code&gt;prev_rnn_state&lt;/code&gt; is a tuple of &lt;code&gt;cell state, hidden state&lt;/code&gt;, which I got from the dynamic_rnn from forward propagation for the previous batch.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do I have unroll the LSTM layer across time series explicitly and train it by providing a vector of the cell states and hidden gathered across the previous time series?&lt;/p&gt;&#xA;" OwnerUserId="47724" LastEditorUserId="47724" LastEditDate="2018-03-14T08:41:59.177" LastActivityDate="2018-03-14T08:41:59.177" Title="Back Propagation in time for tf.nn.dynamic_rnn for sequential input (from batch)" Tags="|tensorflow|rnn|lstm|backpropagation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29026" PostTypeId="2" ParentId="25405" CreationDate="2018-03-13T12:47:06.860" Score="1" Body="&lt;p&gt;You would probably want to place more weight on more recent training observations, as the performance in the upcoming 1-2 months is likely to be more influenced by performance over the past (say) 6 months than performance 2-3 years ago.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a &quot;date&quot; feature, you could create a weight based on this. Then - if your random forest algorithm allows it - use the weight feature to influence the bootstrap sampling (observations with higher weights are more likely to be sampled). In R, the &lt;code&gt;ranger&lt;/code&gt; random forest algorithm has a parameter called &lt;code&gt;case.weights&lt;/code&gt; which does exactly this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way of capturing a time-sensitive trend would be to create features based on &quot;rolling&quot; time periods - e.g. fraction of deliveries which are late within past month / 3 months / 6 months, etc. I would suggest doing this overall with reference to the date of analysis (perhaps group by delivery company), and for the individual training instances with reference to the instance's original delivery date.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-03-13T12:47:06.860" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29028" PostTypeId="1" AcceptedAnswerId="48937" CreationDate="2018-03-13T12:54:33.827" Score="2" ViewCount="9326" Body="&lt;p&gt;I have train and test data in two separate files.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;OneHotEncoder gives different number of features for Train and Test Data based on the different values they have. But the classifier requires that the number of features for test and train data should be equal, how can I solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="46805" LastActivityDate="2019-05-09T08:01:50.320" Title="Different number of features after using OneHotEncoder" Tags="|scikit-learn|feature-engineering|feature-scaling|" AnswerCount="3" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="29029" PostTypeId="1" CreationDate="2018-03-13T12:55:20.807" Score="0" ViewCount="578" Body="&lt;p&gt;I have started a project of classifying a dataset using deep learning. I have tried transfer learning on pretrained models. Now, I want to design a CNN model which can do this work of classification but everywhere it says it needs large data. My question is: Is it possible to design a deep CNN model on an image dataset of size less than 1000 images?&lt;/p&gt;&#xA;" OwnerUserId="47728" LastActivityDate="2018-03-18T04:51:20.057" Title="Is it possible to design a deep CNN model on a small size image dataset" Tags="|deep-learning|cnn|" AnswerCount="2" CommentCount="11" ContentLicense="CC BY-SA 3.0" />
  <row Id="29030" PostTypeId="2" ParentId="29028" CreationDate="2018-03-13T13:31:53.530" Score="0" Body="&lt;p&gt;Data preprocessing (including creation of dummy variables from categorical features) needs to be done before splitting the data into train and test set. This would solve your issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't have much details on what you are trying to do - so if what I say is irrelevant just skip it; but, the fact that your test set contains less categories than the training set is something that I would try to avoid. If this happens for multiple categories, maybe you should try to group some of the categories together into more general ones (e.g. if the variable contains &quot;ways of going to work&quot;, you could merge &quot;bus&quot;, &quot;tram&quot;, &quot;metro&quot; into &quot;public transport&quot;). Also, why don't you try cross-validation instead of having a static test set? (Useful &lt;a href=&quot;http://scikit-learn.org/stable/modules/cross_validation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; for cross-validation with scikit-learn)&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-03-13T13:31:53.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29031" PostTypeId="2" ParentId="29006" CreationDate="2018-03-13T14:05:00.720" Score="5" Body="&lt;p&gt;As Aditya said, there are 3 feature-related terms that sometimes are confused with each other. I will try and give summary explanation to each one of them:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature extraction:&lt;/strong&gt; Generation of features from data that are in a format that is difficult to analyse directly/are not directly comparable (e.g. images, time-series, etc.) In the example of a time-series, some simple features could be for example: length of time-series, period, mean value, std, etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature transformation:&lt;/strong&gt; Transformation of existing features in order to create new ones based on the old ones. A very popularly used technique for dimensionality reduction is Principal Component Analysis (pca) that uses some orthogonal transformation in order to produce a set of linearly non-correlated variables based on the initial set of variables.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature selection:&lt;/strong&gt; Selection of the features with the highest &quot;importance&quot;/influence on the target variable, from a set of existing features. This can be done with various techniques: e.g. Linear Regression, Decision Trees, calculation of &quot;importance&quot; weights (e.g. Fisher score, ReliefF)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the only thing you want to achieve is dimensionality reduction in an existing dataset, you can use either feature transformation or feature selection methods. But if you need to know the physical interpretation of the features you identify as &quot;important&quot; or you are trying to limit the amount of data that need to be collected for your analysis (you need all the initial set of features for feature transformation), then only feature selection can work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find more details on &lt;strong&gt;Feature Selection and Dimensionality Reduction&lt;/strong&gt; in the following links:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.1478&amp;amp;rep=rep1&amp;amp;type=pdf&quot; rel=&quot;noreferrer&quot;&gt;A summary of Dimension Reduction methods&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://eprints.kku.edu.sa/170/1/feature_selection_for_classification.pdf&quot; rel=&quot;noreferrer&quot;&gt;Classification and Feature Selection: A Review&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/2163330/difference-between-feature-selection-feature-extraction-feature-weights&quot;&gt;Relevant question and answers in Stack Overflow&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45221" LastEditorUserId="35644" LastEditDate="2018-03-13T14:41:32.917" LastActivityDate="2018-03-13T14:41:32.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29032" PostTypeId="2" ParentId="29029" CreationDate="2018-03-13T14:23:09.727" Score="1" Body="&lt;p&gt;According to your description of the data, it is highly probable that training any neural network of reasonable size will overfit the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One option is to apply any possible &lt;strong&gt;regularization&lt;/strong&gt; approach; these come to mind:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use L1/L2 weight regularization.&lt;/li&gt;&#xA;&lt;li&gt;Reduce the number of parameters as much as you can (less layers, less number of filters).&lt;/li&gt;&#xA;&lt;li&gt;Use dropout.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Also, try any &lt;strong&gt;data augmentation&lt;/strong&gt; trick: rotate images, mirror images, crop and scale images, etc.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-13T14:23:09.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29034" PostTypeId="2" ParentId="28911" CreationDate="2018-03-13T14:41:00.653" Score="2" Body="&lt;p&gt;The test set is there to show you how the model can perform to unknown cases and basically reassures that it is not overfitted. The selection of the &quot;optimal&quot; model needs to be made based on its performance on the test set and NOT on the training set, because what you need is the model with the best behavior in unknown cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my humble opinion, just splitting the file in training and test set is not the best approach, as there might be bias in the accuracy calculation just because of the random way data were splitted. I would go with cross-validation in order to ensure that the model has been extensively tested. In cases with small amount of data, leave-one-out-cross-validation is often used in order to maximize the amount of available data for the training. You can read more on Cross validation and Leave-one-out-cross-validation (theory and python implementation) &lt;a href=&quot;http://leave-one-out-cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have found the model with the best performance using cross-validation, I believe you can feel free to train it on the whole set of data before giving it to production.&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-03-13T14:41:00.653" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29035" PostTypeId="2" ParentId="29006" CreationDate="2018-03-13T14:41:13.267" Score="3" Body="&lt;p&gt;The two are very different: Feature Selection indeed reduces dimensions, but feature extraction adds dimensions which are computed from other features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For panel or time series data, one usually has the datetime variable, and one does not want to train the dependent variable on the date itself as those do not occur in the future. So you should eliminate the datetime: feature elimination. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, weekday/weekend day may be very relevant, so we need to compute the weekday status from the datetime: feature extraction. &lt;/p&gt;&#xA;" OwnerUserId="18179" LastActivityDate="2018-03-13T14:41:13.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29037" PostTypeId="1" CreationDate="2018-03-13T15:38:50.827" Score="3" ViewCount="469" Body="&lt;p&gt;I'm running a linear regression model as baseline for a specific estimation problem. Based on the resulting R-squared, regressor coefficients and their respective p-values, I can conclude that specific independent variables can be removed from the model.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;What is the induced risk of removing these variables from the feature set?&lt;/li&gt;&#xA;&lt;li&gt;Can other models—that are better at modelling nonlinear relationships—suffer from this decision?&lt;/li&gt;&#xA;&lt;li&gt;How can I be sure that I am not loosing valuable nonlinear information without running &quot;nonlinear&quot; regressors?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47739" LastActivityDate="2018-05-21T14:47:41.043" Title="What is the risk of removing a feature with low correlation...?" Tags="|regression|feature-selection|feature-extraction|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29041" PostTypeId="2" ParentId="5990" CreationDate="2018-03-13T17:50:52.727" Score="0" Body="&lt;p&gt;Ideally you do not need any transformation. Relative time difference between two points can be used as distance function. Where classification can be based on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in java:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;public class TimeDistanceMeasurer implements DistanceMeasure {&#xA;&#xA;    @Override&#xA;    public double compute(double[] a, double[] b) throws DimensionMismatchException {&#xA;        String time1 = String.format(&quot;%02d&quot;, (int)a[0]) + String.format(&quot;%02d&quot;, (int)a[0]);&#xA;        String time2 = String.format(&quot;%02d&quot;, (int)b[0]) + String.format(&quot;%02d&quot;, (int)b[0]);&#xA;&#xA;        SimpleDateFormat format = new SimpleDateFormat(&quot;HHmm&quot;);&#xA;        try {&#xA;            Date date1 = format.parse(time1);&#xA;            Date date2 = format.parse(time2);&#xA;            return Math.abs(date2.getTime() - date1.getTime());&#xA;        } catch (Exception e) {&#xA;            throw new IllegalStateException(&quot;Something went wrong.&quot;, e);&#xA;        }&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47748" LastEditorUserId="47748" LastEditDate="2018-03-13T22:14:44.040" LastActivityDate="2018-03-13T22:14:44.040" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29044" PostTypeId="1" CreationDate="2018-03-13T19:08:11.793" Score="0" ViewCount="6725" Body="&lt;p&gt;So I wrote Linear regression from scratch using y = mx+b and ran the algorithm for 50 epochs (times) to minimize the cost and get the best parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use Scikit Learn, I just call the Linear Regression method and fit the data-set to it then start predicting. How many epochs does fit method run ? Not only for Linear Regression but also other ML methods in general.&lt;/p&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2018-03-14T00:21:45.553" Title="How many epochs does fit method run" Tags="|machine-learning|scikit-learn|predictive-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29045" PostTypeId="2" ParentId="16036" CreationDate="2018-03-13T19:14:28.377" Score="1" Body="&lt;p&gt;A common and simple method to match &quot;documents&quot; is to use TF-IDF weighting, as you have described. However, as I understand your question, you want to rank each career (-document) based on a set of users skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you create a &quot;query vector&quot; from the skills, you can multiply the vector with your term-career matrix (with all the tf-idf weights as values). The resulting vector would give you a ranking score per career-document which you can use to pick the top-k careers for the set of &quot;query skills&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. if your query vector $\bar{q}$ consists of zeros and ones, and is of size $1 \times |terms|$, and your term-document matrix $M$ is of size $|terms| \times |documents|$, then $\bar{v} M$ would result in a vector of size $1 \times |documents|$ with elements equal to the sum of every query term's TF-IDF weight per career document.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This method of ranking is one of the simplest and many variations exist. The TF-IDF entry on &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt; also describes this ranking method briefly. I also found this &lt;a href=&quot;https://stackoverflow.com/questions/27802170/best-way-to-match-2-text-documents&quot;&gt;Q&amp;amp;A&lt;/a&gt; on SO about matching documents.&lt;/p&gt;&#xA;" OwnerUserId="47739" LastEditorUserId="29575" LastEditDate="2018-03-13T19:42:20.947" LastActivityDate="2018-03-13T19:42:20.947" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29046" PostTypeId="2" ParentId="20249" CreationDate="2018-03-13T21:38:45.607" Score="1" Body="&lt;p&gt;A couple of vocab words:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;layer: An entire set of receptive units with shared weights that creates a convolutional feature map&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unit: a single nxn neighborhood filter operator in a given layer of a CNN (say 3x3 or 5x5, whatever given filter size you are using)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;stride length: The distance, or padding, in between the center of every unit of a layer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a convolutional layer, the convolution operation can be thought of like a sliding window, in which the filter from a given layer is applied to the top left corner patch of pixels, then slid to the left 1 horizontal stride length and repeated until the end of the row, then it hops down 1 vertical stride length and starts again, and so on and so forth. This is implemented in a CNN as a 2D grid of units, all sharing the same weights for convolution (this produces the same result as a sliding window).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The RCL takes the sliding window concept of convolution and adds memory to it, so the filter within each unit will be modified based on the response of the previous unit. The time step means how many strides back you want to directly include in the calculation of the current filter. So, if I have an RCL with a filter size of 5x5, and a stride of 1 for an image size of 100x100, a single convolutional layer (assuming it is padded for the filter radius) will have a grid of 100x100 units. Utilizing a time step of N means that you will include response results from the previous N units on the grid to calculate the new unit response (or, more accurately, the unit weights in backprop). &lt;/p&gt;&#xA;" OwnerUserId="47762" LastEditorUserId="29575" LastEditDate="2018-03-13T22:04:38.700" LastActivityDate="2018-03-13T22:04:38.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29048" PostTypeId="2" ParentId="29044" CreationDate="2018-03-13T21:55:31.740" Score="5" Body="&lt;p&gt;In scikit-learn's linear regression, the parameters that minimise the squared error loss aren't estimated using gradient descent, they are computed exactly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The minimisation problem for linear least squares is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat{\beta} = \underset{\beta}\arg \min || \mathbf{y} - \beta \mathbf{X} ||^2&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which has a unique solution (assuming the columns of $\mathbf{X}$ are linearly independent):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat{\beta} = (\mathbf{X}^\text{T}\mathbf{X})^{-1}\mathbf{X}^\text{T}\mathbf{y}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For classifiers that are fitted with an iterative optimisation process like gradient descent, &lt;em&gt;e.g.&lt;/em&gt;, &lt;code&gt;MLPClassifier&lt;/code&gt;, there is a parameter called &lt;code&gt;max_iter&lt;/code&gt; which sets the maximum number of epochs. If &lt;code&gt;tol&lt;/code&gt; is set to 0, the optimisation will run for &lt;code&gt;max_iter&lt;/code&gt; epochs.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-03-14T00:21:45.553" LastActivityDate="2018-03-14T00:21:45.553" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29049" PostTypeId="1" CreationDate="2018-03-13T22:48:31.607" Score="0" ViewCount="324" Body="&lt;p&gt;I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One piece of data (&lt;a href=&quot;https://www.pandasecurity.com/mediacenter/src/uploads/2014/07/Pandalabs-2015-anual-EN.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.pandasecurity.com/mediacenter/src/uploads/2014/07/Pandalabs-2015-anual-EN.pdf&lt;/a&gt; on page 9) it states that the 'infection rates' of Sweden is 20.88% (bottom 3 ranking), USA at 29.48% (middle ranking) and China (first rank) having 57.24%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another (&lt;a href=&quot;http://www.virusradar.com/en/home/world&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.virusradar.com/en/home/world&lt;/a&gt;) , uses a different measurement to define the 'threat rates', which is different to the one above, which has Sweden at 2.13%, US at 2.87%, and China at 15.17%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another piece of data (&lt;a href=&quot;https://www.symantec.com/content/dam/symantec/docs/reports/istr-22-2017-en.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.symantec.com/content/dam/symantec/docs/reports/istr-22-2017-en.pdf&lt;/a&gt; &#xA; on page 50) states 'identity theft' in Sweden is 6 million(low ranking), US 791 million (high ranking), and China 11 million (middle ranking)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm unsure how to compare these, because by just looking with our eyes we can see Sweden is the lowest in all these statistics which I can use to further discuss my argument. However, these are clearly different measurements (one is overall infections, one is just identity theft, etc), and I have much more similar data which provides numbers/percentages of certain types of cyber crime by country.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to analyse the results so I can compare them to further the goals of the thesis about where cyber crime is most/least prominent and to create visualisations of this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So would just simply ranking them be fine? (Sweden 3 because it was lowest all times?, USA 2 because it was second twice?, China 1 because it was first twice?) but that sounds very incorrect. Should I convert all my numeric data sets (like the identity theft one) to percentages then compare them all by percentage to rank/discuss accordingly? And compare them then even though they're different measurements of the same thing(country)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for any advice.&lt;/p&gt;&#xA;" OwnerUserId="47757" LastEditorUserId="47757" LastEditDate="2018-03-13T23:03:25.693" LastActivityDate="2021-12-06T10:02:15.187" Title="Comparing data sets with different measurements" Tags="|dataset|statistics|visualization|data-cleaning|descriptive-statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29050" PostTypeId="2" ParentId="26087" CreationDate="2018-03-13T23:00:47.480" Score="5" Body="&lt;p&gt;This is something which is &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/6557&quot; rel=&quot;noreferrer&quot;&gt;planned&lt;/a&gt; to be done. Setting the minimum number of samples required at a leaf node or a split as well as setting the maximum depth of the tree are how you want to work around this.&lt;/p&gt;&#xA;" OwnerUserId="2511" LastActivityDate="2018-03-13T23:00:47.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29051" PostTypeId="1" AcceptedAnswerId="29061" CreationDate="2018-03-13T23:33:08.593" Score="0" ViewCount="1771" Body="&lt;p&gt;Some papers report the correlation between features when building a model and some don't. Is there a need to check the correlation between features and target feature? It won't be easy if the number of features is high. &lt;/p&gt;&#xA;" OwnerUserId="47473" LastEditorUserId="47473" LastEditDate="2018-03-14T00:02:04.180" LastActivityDate="2018-03-16T08:40:51.280" Title="Is correlation needed when building a model?" Tags="|machine-learning|predictive-modeling|feature-selection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29052" PostTypeId="1" CreationDate="2018-03-13T23:40:17.697" Score="3" ViewCount="969" Body="&lt;p&gt;I need to group items by their approximity to each other in multi (&amp;lt;5) dimensional space. The items also have a categorical feature. I need to form groups (clusters) such that none of the records from the same category would appear in thr same cluster.  Is there a class of clustering algorithms that can do that? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My way of thinking is to use custom distances that would measure two records in the same category further then ones in different categories. That works to some extent, but it does not guarantee satisfaction of the given requirement.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple one dimension (x) + category (c) example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;    x       c&#xA;0   0.80    0&#xA;1   0.90    1&#xA;2   0.10    0&#xA;3   0.30    1&#xA;4   0.20    0&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The goal is to group records into two clusters [0, 1];and  [2 or 4, 3]; then record 4 or 2 respectively should remain outside of the second cluster because a record with c=0 is already present in the cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions?&lt;/p&gt;&#xA;" OwnerUserId="47766" LastEditorUserId="47766" LastEditDate="2018-03-14T18:51:32.027" LastActivityDate="2018-03-16T12:35:33.627" Title="How to use cluster analysis with grouped data so one cluster may only have not more than one item from each group?" Tags="|clustering|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29053" PostTypeId="1" AcceptedAnswerId="29060" CreationDate="2018-03-14T01:53:39.920" Score="0" ViewCount="254" Body="&lt;p&gt;I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My scraper can navigate through the university websites and find their tuition fees pages perfectly , but when trying to extract specific fees like : &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Resident &lt;/li&gt;&#xA;&lt;li&gt;Non Resident &lt;/li&gt;&#xA;&lt;li&gt;Per Credit Hour&lt;/li&gt;&#xA;&lt;li&gt;Per Semester &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm running into trouble due to the data being so unstructured from site to site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried using NLTK to parse data based on parts of speech tags and regex chunking to try to extract sentences such as &quot;tuition cost for resident: $12,500&quot; but colleges can display this data in a number of ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any better ideas/methodologies that I should be looking into that can help me with extracting this type of data?&lt;/p&gt;&#xA;" OwnerUserId="47773" LastEditorUserId="40853" LastEditDate="2018-03-14T02:23:58.607" LastActivityDate="2018-03-14T06:03:15.267" Title="Proper/Possible methods for extracting unstructured data from websites" Tags="|machine-learning|nlp|feature-extraction|nltk|scraping|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29054" PostTypeId="2" ParentId="15989" CreationDate="2018-03-14T02:56:08.200" Score="28" Body="&lt;p&gt;In a multi-class setting micro-averaged precision and recall are always the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;P = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FP_c}\\&#xA;R = \frac{\sum_c TP_c}{\sum_c TP_c + \sum_c FN_c}&#xA;$$&#xA;where c is the class label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since in a multi-class setting you count &lt;em&gt;all&lt;/em&gt; false instances it turns out that&#xA;$$&#xA;\sum_c FP_c = \sum_c FN_c&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence P = R. In other words, every single False Prediction will be a False Positive for a class, and every Single Negative will be a False Negative for a class. If you treat a binary classification case as a bi-class classification and compute the micro-averaged precision and recall they will be same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer given by Rahul is in the case of averaging binary precision and recall from multiple dataset. In which case the micro-averaged precision and recall are different.&lt;/p&gt;&#xA;" OwnerUserId="39811" LastEditorUserId="40853" LastEditDate="2018-03-14T03:17:11.753" LastActivityDate="2018-03-14T03:17:11.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29055" PostTypeId="1" AcceptedAnswerId="29059" CreationDate="2018-03-14T03:24:22.987" Score="1" ViewCount="792" Body="&lt;p&gt;Assume that we have a training data set (with both features and labels) and a test data set (with only features). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we build a machine learning model that requires normalization of the features, the correct way of doing normalization (to prevent information leakage) is to only use the training data set. Namely, a wrong way of doing normalization is to stack the training (exclude the Y column) and test data set together, and perform normalization (i.e., using the mean and variance of the entire training + test data set). The intuition here is very clear: if you want to get an unbiased estimator of your model performance in production, then when you train your model, you shouldn't instill any information in the test data set that will be used to gauge the actual performance of your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is the following:&lt;/strong&gt; When we have trained the model correctly and about to use this model to do prediction, how should we normalize the test data set? I believe the correct way to normalize the test data set is: using the mean and variance obtained from the training data set to do normalization on the test data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, why not just normalize the test data using its own mean and variance? Or why not stack train and test data set together and using the overall mean and variance to normalize the test data set? In the prediction stage, the idea and intuition of data snooping and information leakage is not clear to me.&lt;/p&gt;&#xA;" OwnerUserId="30170" LastActivityDate="2018-03-14T05:58:18.087" Title="Data Snooping, Information Leakage When Performing Feature Normalization" Tags="|machine-learning|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29057" PostTypeId="1" CreationDate="2018-03-14T05:16:06.710" Score="0" ViewCount="199" Body="&lt;p&gt;I am able to Install Anaconda and able to write Python and R code without any issues based on below help. &#xA;&lt;a href=&quot;https://gist.github.com/simecek/019d87c55fec3839d95bbf8489dde61d&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://gist.github.com/simecek/019d87c55fec3839d95bbf8489dde61d&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I extracted my data from MS-SQLServer via pypyodbc&lt;/li&gt;&#xA;&lt;li&gt;Converted the Python dataset to R and maniputed the data (as per my usecase)&lt;/li&gt;&#xA;&lt;li&gt;Now I am ready to convert my R dataframe back to Python -&gt; Got stuck&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The question is..is it possible to convert R dataframe to Python dataframe (as a new dataframe)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, my final preparations should be done via Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I search for above issue for 3 hours and no luck. Please help.&lt;/p&gt;&#xA;" OwnerUserId="9663" LastActivityDate="2018-06-13T07:34:52.860" Title="Jupyter notebook - Converted Python dataset to R Dataframe. Got stuck in converting back from R to Python Dataframe" Tags="|machine-learning|python|r|jupyter|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29058" PostTypeId="1" CreationDate="2018-03-14T05:54:55.350" Score="0" ViewCount="72" Body="&lt;p&gt;I have some data that contains students answer, score, and time spent in many exams. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to estimate each students level according to this data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;student ID, topic ID, score, time spend&#xA;&#xA;  sid1    ,   tid1  ,  10  , 5s&#xA;&#xA;  sid1    ,   tid2  ,  4   , 15s&#xA;&#xA;  sid1    ,   tid3  ,  3   , 25s&#xA;&#xA;  sid1    ,   tid4  ,  10  , 5s&#xA;&#xA;  sid2    ,   tid1  ,  11  , 15s&#xA;&#xA;  sid2    ,   tid2  ,  6   , 11s&#xA;&#xA;  sid2    ,   tid3  ,  7   , 24s&#xA;&#xA;  sid2    ,   tid4  ,  0   , 53s&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47777" LastEditorUserId="40853" LastEditDate="2018-03-14T08:46:18.057" LastActivityDate="2018-08-12T03:02:40.660" Title="How to estimate student's level according to score and time spend" Tags="|machine-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29059" PostTypeId="2" ParentId="29055" CreationDate="2018-03-14T05:58:18.087" Score="2" Body="&lt;p&gt;The normalization parameters you fitted in training are now part of your model. You fitted the model weights on the training data: the normalization step is part of your model now, and the &quot;parameters&quot; for that step are the mean and variance of the training data, not the test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To help make this more concrete, let's pretend that this model is in production. You fit your model on whatever training data you had available, and now it's running as a scoring service. You can evaluate your model accuracy over time as data you've scored gets labeled to ground truth, but you're just evaluating the model not refitting it. How do you normalize these incoming predictions? You're going to need to use the mean and variance from whatever dataset you used to fit the model. If you'd rather do something like using the last K observations to estimate mean/variance, then that's the procedure you need to be testing when you fit the model.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-14T05:58:18.087" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29060" PostTypeId="2" ParentId="29053" CreationDate="2018-03-14T06:03:15.267" Score="0" Body="&lt;p&gt;You need to build a couple of classifiers. First, you need a classifier to give you thumbs up that you want to parse the page at all. Call this the &quot;is_relevant&quot; model. Once you've determined that a page is relevant, you should pass it through a separate classifier for each data element you're hoping to capture (or a multiclass classifier capable of recognizing each of those elements &lt;em&gt;and distinguishing them from content you're not interested in&lt;/em&gt;).&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-14T06:03:15.267" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29061" PostTypeId="2" ParentId="29051" CreationDate="2018-03-14T06:16:20.810" Score="7" Body="&lt;p&gt;Not really, no. Sort of. It depends on how complex your model/data is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's entirely possible to have a situation where a feature taken in isolation will not be correlated with the target variable, but multiple features considered together will. This is why univariate correlation is unreliable for feature selection. A trivial case that demonstrates this is a bivariate model performing a binary classification where the positive class is bounded by the right upper and left lower quadrants, and the negative class is bounded by the left upper and right lower quadranta (i.e. the &quot;XOR&quot; pattern):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/L5huJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/L5huJ.png&quot; alt=&quot;XOR data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the input features have the same sign (x&gt;0 &amp;amp; y&gt;0 or x&amp;lt;0 &amp;amp; y&amp;lt;0), it's the positive class, else it's the negative class. But either feature in isolation is completely useless and uncorrelated with the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, modern models like deep neural networks are effectively capable of &quot;learning their own features&quot;, i.e. constructing extremely complex features by developing abstractions from the raw inputs. The &quot;final&quot; features learned by such a model will likely be correlated with the target, but the input features need not be. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you consider the imagenet task (classifying a photo as a member of one of 10,000 classes), I'd be very surprised to learn that there's any correlation between the values of specific pixels and any target class. That's just not how we interpret photos. The value of the pixel in position [25, 10] should not have any correlation with whether or not the picture is a photo of a dog. But, if we think of the entire network before the output layer as a feature engineering system (such that the &quot;classifier&quot; is just the output layer), then the &quot;features&quot; provided by the penultimate layer (the inputs to &quot;The Classifier&quot;) probably have some correlation with the target. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;TL;DR: If a feature is correlated with the target, it probably contains information about the target that will be useful for modeling. But that does not mean uncorrelated features are useless. Reporting correlation when it's there is a simple way to demonstrate that there's a signal in that variable. But lack of correlation doesn't necessarily mean you should throw those features away. In fact, correlation doesn't even mean you should necessarily use that feature either: you can have multiple features correlated with the target that are highly correlated with each other, in which case you would probably only want one or a handful of that group in your model.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-03-16T08:40:51.280" LastActivityDate="2018-03-16T08:40:51.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29062" PostTypeId="1" CreationDate="2018-03-14T08:31:55.170" Score="-1" ViewCount="125" Body="&lt;p&gt;I am new in neural network and deep learning, trying to create a deep learning model to classify images. While reading blogs and videos, a question comes in my mind and not able to find the correct answer on internet. So that's the main reason to post question here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is &lt;strong&gt;how can i generate alert on unseen data?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;E.g.&lt;/strong&gt;&#xA;My trained model is able to classify following objects in the image:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Truck&lt;/li&gt;&#xA;&lt;li&gt;Car&lt;/li&gt;&#xA;&lt;li&gt;Bike etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But i want to generate an alert for given &lt;strong&gt;bicycle&lt;/strong&gt; image to trained model. The point here is to be note that intentionally i am giving bicycle image. The current model predict it as a bike because its most of the features are matching with bike along with probability values for truck and car.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest me how can i implement this in deep learning?&lt;/p&gt;&#xA;" OwnerUserId="45750" LastActivityDate="2018-03-14T15:23:23.027" Title="Alert generation on unseen data using deep learning" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29063" PostTypeId="1" CreationDate="2018-03-14T08:41:34.253" Score="0" ViewCount="712" Body="&lt;p&gt;Excuse if this has been answered before. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to extract features and parse from a piece of text and run some analysis.&#xA;For e.g. &quot;Plot the past 5-year sales of Apple&quot; should give me the following &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Information:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Company - Apple &lt;/li&gt;&#xA;&lt;li&gt;Item - Sales &lt;/li&gt;&#xA;&lt;li&gt;Period - past 5 years &lt;/li&gt;&#xA;&lt;li&gt;Action - Plot&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What deep learning techniques / algorithms should I be looking to use? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers are highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="47782" LastEditorUserId="29169" LastEditDate="2021-02-13T16:36:45.750" LastActivityDate="2021-02-13T16:36:45.750" Title="Text analysis - classification, parsing" Tags="|deep-learning|convolutional-neural-network|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29064" PostTypeId="2" ParentId="19538" CreationDate="2018-03-14T08:53:48.293" Score="1" Body="&lt;p&gt;I was going to try this for codes of the form A12.3;&#xA;Take the first letter, as a position in the alphabet, add 10 and then multiply by 10, giving you 26 values between 100 and 360 &#xA;Append the second two digits to this number, i.e add 12 from the example above, to give 10012.&#xA;Then append the final digit, i.e. 100123.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This should give you numerically comparable distances that you'd get from analysing the ICD10 codes as a tree, i.e. all codes that are near each other on one branch are numerically near each other on another branch, and shouldn't spill over too much. By that I mean that the numbers at the high end of one letter code shouldn't be nearer the lower numbers of another letter code than they are to other members of the group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone has any comments, I'd be interested?&lt;/p&gt;&#xA;" OwnerUserId="47783" LastActivityDate="2018-03-14T08:53:48.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29065" PostTypeId="2" ParentId="1165" CreationDate="2018-03-14T09:54:05.197" Score="2" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Limitations of ML-Powered Predictive Models&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Predictive Models on Conversion Studies for the Level Premium Term Plans&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data Management&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Business intelligence(BI) predictive Analytics&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Big data and Market Intelligence &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Educational data mining&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Automation of predictive models&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;predictive data quality using ML&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Open source Data Mining&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data Mining System&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data mining Trends&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Health data mining&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Web Analytics solution&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data Mining marketing&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data Mining in Search Engine Analytics&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fraud &amp;amp; Security&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Risk Management&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47785" LastEditorUserId="29575" LastEditDate="2018-03-14T12:28:17.830" LastActivityDate="2018-03-14T12:28:17.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29067" PostTypeId="1" AcceptedAnswerId="29073" CreationDate="2018-03-14T10:31:39.837" Score="5" ViewCount="18694" Body="&lt;p&gt;I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into dummies variables, creating 29 features. My dataset has shape of 6552 rows and 34 features. What is the recommend approach to tune the parameters of XGBClassifier, since I created the model using default values, i.e., model=XGBClassifier()? Should I use a brute-force looping the values in some parameters until I find a optimal prediction value? In this case what is recommended?&lt;/p&gt;&#xA;" OwnerUserId="47792" LastActivityDate="2023-06-24T19:03:50.990" Title="How to optimize XGBoost performance accuracy?" Tags="|machine-learning|xgboost|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29068" PostTypeId="2" ParentId="29063" CreationDate="2018-03-14T10:59:33.080" Score="1" Body="&lt;p&gt;Named-Entity Recognition (NER) is one of the techniques you could look at. Different techniques exist, you could first look at pre-trained models of Spacy in Python &lt;a href=&quot;https://spacy.io/usage/linguistic-features&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Otherwise, you could train a model on your own training data if you have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, more systematic approaches can be efficient. For exemple, looking at POS tags values, do regex identification...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More generally, this is the field of Information Extraction (IE) in text mining. Some good resources can be found &lt;a href=&quot;http://philipperemy.github.io/information-extract/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://www.nltk.org/book/ch07.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Sorry, everything is in Python as I work mainly with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-03-14T10:59:33.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29071" PostTypeId="2" ParentId="28915" CreationDate="2018-03-14T13:11:48.383" Score="6" Body="&lt;p&gt;Questions about definitions are always fun, so let me try to offer another answer here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Firstly, let us model mathematically what you are doing. At the highest level, you are estimating some measure of &quot;reward&quot; $R(s)$ for each board state $s$. You are doing it by interacting with the environment and updating your internal parameters (i.e. the table of $R$ values) towards &lt;em&gt;reinforcing&lt;/em&gt; the favorable behaviours. Consequently, by most standard definitions, your algorithm should indeed be categorized as &lt;em&gt;reinforcement learning&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand &lt;em&gt;what kind&lt;/em&gt; of reinforcement learning you are doing and whether it is &quot;good&quot;, we should go a bit deeper. One of the key notions in reinforcement learning is the &lt;em&gt;value function&lt;/em&gt; $V$ (or its alter-ego, the $Q$ function), which reports the best possible total reward you may expect to gain if you play optimally from a given board state. If you can show that your algorithm is, at least in some sense, estimating $V$, you can claim a certain &quot;goodness&quot; guarantee, and proceed to classify it into any of the known &quot;good&quot; algorithm types (which all essentially either aim to estimate $V$ directly, or to act as if they estimated it implicitly).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that when we speak about two-player games, there does not necessarily exist a unique $V$ to aim for. For example, assuming the reward 1 for winning, 0 for losing, 0.5 for a draw, and no discounting, $V(\text{empty board})$ is $0.5$ if you are playing against an optimal opponent, but it is probably close to $1$ if your opponent is random. If you play against a human, your $V$ may differ depending on the human. Everyone knows that the first move into the center is the safest one, however I myself have never won with such a move - the game always ends in a draw. I did win a couple of times by making the first move to the corner, because it confuses some opponents and they make a mistake on their first turn. This aside, assuming $V$ to denote the game against an optimal opponent is a reasonable choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Getting back to your algorithm, the crucial step is the update of $R$, which you did not really specify. Let me assume you are simply accumulating the scores there. If it is the case, we may say that, strictly speaking, you are &lt;em&gt;not&lt;/em&gt; doing Q-learning, simply because that's not how the $Q$ function is updated in the classical definition. It still does not mean you are not implicitly estimating the correct $V$, though, and it is rather hard to prove or disprove whether you do it or not eventually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me tune your algorithm a bit for clarity. Instead of adding up the final reward to $R(s)$ for each state $s$, which occurred in the game, let us track the &lt;em&gt;average&lt;/em&gt; reward ever reached from each state. Obviously, the position where you always win, although rarely reach, should be more valued than a position where you rarely win, even if you often reach it, so we probably won't break the algorithm by this change, and the overall spirit of learning stays the same anyway. After this change, $R(s)$ becomes easy to interpret - it is the &lt;em&gt;average expected reward&lt;/em&gt; reachable from position $s$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;em&gt;average expected reward&lt;/em&gt; is not really the value function $V$ we are interested in estimating. Our target $V$ should tell us the &lt;em&gt;best expected reward&lt;/em&gt; for each position, after all. Interestingly, though, when your policy is already optimal then your average reward is equal to your optimal reward (because you always do optimal moves anyway), hence it may be the case that even though you are kind-of learning the wrong metric in your algorithm, if the learning process pushes your algorithm ever so slightly towards optimal play, as  you gradually improve your policy, the &quot;average expected reward&quot; metric itself slowly becomes &quot;more correct&quot; and eventually you start converging to the correct value function. This is pure handwaving, but it should illustrate the claim about it being hard to prove or disprove whether your algorithm formally learns what it should learn. Maybe it does.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, let us, instead of tracking the &lt;em&gt;average&lt;/em&gt; reward for each state, change your algorithm to track the &lt;em&gt;best possible&lt;/em&gt; reward so far. This means, you'll check all the alternative moves from each position and only update $R(s)$ if your current move resulted in an improved score down the road (in comparison to alternative options you could have taken from this state). Congratulations, now your algorithm &lt;em&gt;is&lt;/em&gt; equivalent to the usual Q-learning method (it is the &quot;value iteration&quot; method, to be more precise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, &quot;is it learning or brute force&quot; is a valid question. The word &quot;learning&quot; can be interpreted in at least two different ways. Firstly, learning may denote simplistic &lt;em&gt;memorization&lt;/em&gt;. For example, if I discover that the first move to the center is good, I may write this fact down in a table and use this fact later directly. People call such memorization &quot;learning&quot;, but this learning is really quite dumb.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second, different meaning often ascribed to &quot;learning&quot; is &lt;em&gt;generalization&lt;/em&gt;. It would be the case when, besides simply writing down which moves are good, your algorithm could generalize this information to previously unseen moves. This is the &quot;intelligent&quot; kind of learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Q-learning, as well as many other RL algorithms are typically formulated in terms of updates to the tables $V$ or $Q$. As such, they are inherently &quot;dumb learning&quot; algorithms, which do not even aim to generalize the state information. True generalization (aka &quot;smart learning&quot;) emerges only when you start &lt;em&gt;modeling&lt;/em&gt; the state or the policy using something with a built-in generalization ability, such as a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to summarize. Yes, your algorithm is reinforcement learning. No, it is not Q-learning, but it becomes that with a minor change. Yes, it is more &quot;brute force&quot; rather than &quot;intelligent learning&quot;, but so is the default Q-learning as well. Yes, adding generalization by modeling states with a neural network makes the algorithm &quot;more intelligent&quot;.&lt;/p&gt;&#xA;" OwnerUserId="14519" LastEditorUserId="14519" LastEditDate="2018-03-14T19:37:11.170" LastActivityDate="2018-03-14T19:37:11.170" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29072" PostTypeId="1" AcceptedAnswerId="29078" CreationDate="2018-03-14T14:02:46.757" Score="2" ViewCount="6530" Body="&lt;p&gt;I'm implementing a 1D CNN in keras by following the keras tutorial on the same -&#xA; &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;. Once the model is built, when I execute model.summary(), I get the following output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_________________________________________________________________&#xA;Layer (type)                 Output Shape              Param #   &#xA;=================================================================&#xA;input_1 (InputLayer)         (None, 1000)              0         &#xA;_________________________________________________________________&#xA;embedding_1 (Embedding)      (None, 1000, 100)         17410600  &#xA;_________________________________________________________________&#xA;conv1d_1 (Conv1D)            (None, 996, 128)          64128     &#xA;_________________________________________________________________&#xA;max_pooling1d_1 (MaxPooling1 (None, 199, 128)          0         &#xA;_________________________________________________________________&#xA;conv1d_2 (Conv1D)            (None, 195, 128)          82048     &#xA;_________________________________________________________________&#xA;max_pooling1d_2 (MaxPooling1 (None, 39, 128)           0         &#xA;_________________________________________________________________&#xA;conv1d_3 (Conv1D)            (None, 35, 128)           82048     &#xA;_________________________________________________________________&#xA;max_pooling1d_3 (MaxPooling1 (None, 1, 128)            0         &#xA;_________________________________________________________________&#xA;global_max_pooling1d_1 (Glob (None, 128)               0         &#xA;_________________________________________________________________&#xA;dense_1 (Dense)              (None, 128)               16512     &#xA;_________________________________________________________________&#xA;dense_2 (Dense)              (None, 20)                2580      &#xA;=================================================================&#xA;Total params: 17,657,916&#xA;Trainable params: 247,316&#xA;Non-trainable params: 17,410,600&#xA;_________________________________________________________________&#xA;None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The conv1d_1 has the total number of parameters as 64128. But since the conv1d_1 was initialized with filters = 128, kernel_size = 5, padding = 'valid' (which means no padding), shouldn't the number of parameters be &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;=&gt; kernel_size * kernel_size * num_filters + num_filters * bias&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;=&gt; 5 * 5 * 128 + 128 * 1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;=&gt; 26 * 128 &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;=&gt; 3328&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="10717" LastActivityDate="2022-01-20T17:04:18.633" Title="counting number of parameters keras" Tags="|machine-learning|keras|cnn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29073" PostTypeId="2" ParentId="29067" CreationDate="2018-03-14T14:24:30.103" Score="5" Body="&lt;p&gt;There are three main techniques to tune up hyperparameters of any ML model, included XGBoost:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Grid search: you let your model run with different sets of hyperparameter, and select the best one between them. Packages like SKlearn have routines already implemented. But also in this case you have to pre-select the nodes of your grid search, i.e. which values have to be tried by the routine&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Random search: similar to Grid Search, but you basically only choose the parameters boundaries, and the routine randomly try different sets of hyperparameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;more informations about method 1 and 2 are &lt;a href=&quot;http://scikit-learn.org/stable/modules/grid_search.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Bayesian optimization algorithms; this is the way I prefer. Basically this algorithms guesses the next set hyperparameter to try based on the results of the trials it already executed. An easy to use and powerful is &lt;a href=&quot;https://github.com/automl/SMAC3&quot; rel=&quot;noreferrer&quot;&gt;SMAC&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="29134" LastEditDate="2018-03-14T15:39:43.153" LastActivityDate="2018-03-14T15:39:43.153" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29074" PostTypeId="2" ParentId="29067" CreationDate="2018-03-14T14:56:00.673" Score="4" Body="&lt;p&gt;When using GridSearchCV with XGBoost, be sure that you have the latest versions of XGBoost and SKLearn and take particular care with njobs!=1 &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/issues/6627&quot; rel=&quot;nofollow noreferrer&quot;&gt;explanation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import xgboost as xgb&#xA;from sklearn.grid_search import GridSearchCV&#xA;&#xA;xgb_model = xgb.XGBClassifier()&#xA;optimization_dict = {'max_depth': [2,4,6],&#xA;                     'n_estimators': [50,100,200]}&#xA;&#xA;model = GridSearchCV(xgb_model, optimization_dict, &#xA;                     scoring='accuracy', verbose=1)&#xA;&#xA;model.fit(X,y)&#xA;print(model.best_score_)&#xA;print(model.best_params_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Word of advice with XGBoost, careful with overfitting. In your particular situation, I suggest &lt;a href=&quot;https://github.com/catboost/catboost&quot; rel=&quot;nofollow noreferrer&quot;&gt;Catboost&lt;/a&gt; since it is optimized for categoricals.&lt;/p&gt;&#xA;" OwnerUserId="47802" LastActivityDate="2018-03-14T14:56:00.673" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29075" PostTypeId="2" ParentId="29049" CreationDate="2018-03-14T14:56:50.317" Score="1" Body="&lt;p&gt;You're right, you cannot directly compare the scores, since they are extracted from different samples, using different metrics and parameters, etc. Is exactly as comparing apples with airplanes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But is not so wrong to play around with the placement in the rank: you are comparing placements with placements, so it makes perfectly sense. If Sweden is always at the last place, this definitely means something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can also think of weight those placements by some parameter/set of parameters which takes in account the subject of the ranks.&#xA;As example, seriousness of the threats (ex. infection by AdWare can be less important than identity theft, etc), coverage of the population by the authority which published the results, reputation of the authority, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also extract uncertainties on those ranks, maybe based on the statistical samples taken in account by the authorities, and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-03-14T14:56:50.317" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29078" PostTypeId="2" ParentId="29072" CreationDate="2018-03-14T15:08:38.123" Score="3" Body="&lt;p&gt;In fact, you use 1D convolution. &#xA;Given that the dimension of the output of embedding layer is 100, that the kernel size is 5, and that the number of filters is 128,&#xA;You have 100x5x128 = 64000 weights. Add to this 128 biases and you get 64128 parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that parameter sharing is used, so that there is only one set of weights and biases per filter, in depth.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastEditorUserId="41728" LastEditDate="2018-03-14T15:14:32.993" LastActivityDate="2018-03-14T15:14:32.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29079" PostTypeId="2" ParentId="29062" CreationDate="2018-03-14T15:23:23.027" Score="0" Body="&lt;p&gt;There is nothing easy that comes to my mind here.&#xA;Adding a garbage class is not good idea as it would deteriorate the rest of the model. Two solutions come here:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You add a bicycle class or other classes that can be problematic in your opinion due to close features with your base classes, and train again. Complicated to be exhaustive with all situations.&lt;/li&gt;&#xA;&lt;li&gt;You define a threshold of confidence of your model: for example if there is no probability more than 95% then the model is not confident in its choice and you should not trust him. Good and larged models should stay at equiprobable choices at the softmax mater when the features are not recognized well.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-03-14T15:23:23.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29080" PostTypeId="2" ParentId="29058" CreationDate="2018-03-14T15:47:58.087" Score="1" Body="&lt;p&gt;I think there will be different approaches, and different results, to this problem.&lt;/p&gt;&#xA;&lt;p&gt;The first which comes to my mind is simple:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;evaluate the difficulty of each topic, by the average of the time spent by the students on it;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;use the evaluations at point 1) to weight the time spent by the students on the topics;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;calculate the average weighted time spent on all the topics;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;a) you can rank the students based on the average weighted times, or&lt;/p&gt;&#xA;&lt;p&gt;b) you can plot an histogram of the average weighted times, and choose some proper tresholds to classify the students&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="29134" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-14T23:46:03.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29082" PostTypeId="1" AcceptedAnswerId="29092" CreationDate="2018-03-14T16:24:03.030" Score="1" ViewCount="20" Body="&lt;p&gt;I am currently using sklearn &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;scale&lt;/code&gt;&lt;/a&gt; to preprocess my X data before being put into a perceptron - &lt;code&gt;mean/stddev&lt;/code&gt; so as to prevent the data converging to infinity or 0. My perceptron returns the weights + bias after the network has been trained:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = preprocessing.scale(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After processing the X and Y data through my perceptron I am returned with weights. From these weights I can calculate the line of best fit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ls = cp.linspace(cp.min(X), cp.max(X))&#xA;best_fit = w[1]+w[0]*ls&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;w[1]&lt;/code&gt; is a bias. This &lt;code&gt;best_fit&lt;/code&gt; line is accurate but it is relative to the preprocessed X rather than the original X which I would like to plot. What is the technique to make these weights relative to the original X values if it is possible?&lt;/p&gt;&#xA;" OwnerUserId="47290" LastActivityDate="2018-03-14T21:18:42.573" Title="Weights and bias' relative to preprocessed X" Tags="|plotting|perceptron|bias|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29084" PostTypeId="2" ParentId="27709" CreationDate="2018-03-14T17:25:29.610" Score="0" Body="&lt;p&gt;I have to wonder why you are classifying entire instances as either smooth or shaky? Would it be possible to model each instance as a hidden Markov model (HMM) with two states, smooth vs shaky? Your task would be to model the transition probabilities and estimate the current state of the instances. Do instances ever transition from shaky back to smooth? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Estimating the model parameters can be heavy work, but you have a small dataset. I recently ran an analysis similar to this using the &lt;code&gt;msm&lt;/code&gt; package in R. I had around 3000 instances of varying lengths (typically &amp;lt; 20 each). The code ran quickly (under a minute but noticeable wait).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the effect is strong -- i.e. the measurements really change when you go from &quot;smooth&quot; to &quot;shaky&quot;, you should still be able to estimate the parameters even with the &quot;class imbalance&quot; that you have. If not, you could take a subset of the data that achieves better balance (or better, randomly select subsets of the smooth data and bootstrap the parameter estimates). The transition probabilities are conditional on the state you are in, so you can change the overall proportion of being in one state or another without harm to your analysis. Of course, global estimates of how long the system remains in one state would need to be adjusted in light of population prevalence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the transition to the shaky state is anticipated by changes in the observed movement data, then you need to consider one of those Poisson intensity models where the intensity function depends on the observations. Check out R package PtProcess and the references therein. It offers a variety of models, which could meet your needs.&lt;/p&gt;&#xA;" OwnerUserId="3077" LastActivityDate="2018-03-14T17:25:29.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29085" PostTypeId="2" ParentId="26451" CreationDate="2018-03-14T17:27:43.687" Score="5" Body="&lt;p&gt;Instead of using &lt;code&gt;tf.nn.conv2d_transpose&lt;/code&gt; you can use &lt;code&gt;tf.layers.conv2d_transpose&lt;/code&gt;&#xA;It is a wrapper layer and there is no need to input output shape or if you want to calculate output shape you can use the formula: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;H = (H1 - 1)*stride + HF - 2*padding&#xA;H - height of output image i.e H = 28 &#xA;H1 - height of input image i.e H1 = 7 &#xA;HF - height of filter &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43692" LastEditorUserId="29575" LastEditDate="2018-03-14T17:45:50.233" LastActivityDate="2018-03-14T17:45:50.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29087" PostTypeId="1" AcceptedAnswerId="29088" CreationDate="2018-03-14T20:02:02.517" Score="0" ViewCount="900" Body="&lt;p&gt;The original string is like this &quot;965 - Vehicle parking area&quot;, how do I get rid of the &quot;965 - &quot; part in R? (There are two spaces before and after the dash.)&#xA;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="47813" LastActivityDate="2018-03-15T05:31:17.870" Title="How to get rid of some parts in a string variable?" Tags="|r|" AnswerCount="2" CommentCount="1" ClosedDate="2018-03-15T09:29:07.673" ContentLicense="CC BY-SA 3.0" />
  <row Id="29088" PostTypeId="2" ParentId="29087" CreationDate="2018-03-14T20:32:16.410" Score="1" Body="&lt;p&gt;One possibility is to use gsub function to find the first ' - ' and remove everything before and also remove the dash with the space before and after that.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tmp&#xA;[1] &quot;965 - Vehicle parking area&quot;&#xA;&amp;gt; gsub(&quot;^.*? - &quot;,&quot;&quot;, tmp)&#xA;[1] &quot;Vehicle parking area&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If there are two spaces: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; tmp &amp;lt;- &quot;965  -  Vehicle parking area&quot;&#xA;&amp;gt; sub(&quot;^.*?  -  &quot;,&quot;&quot;, tmp)&#xA;[1] &quot;Vehicle parking area&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2018-03-14T20:52:53.097" LastActivityDate="2018-03-14T20:52:53.097" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29089" PostTypeId="1" AcceptedAnswerId="29091" CreationDate="2018-03-14T20:52:08.050" Score="0" ViewCount="552" Body="&lt;p&gt;Ok guys, I might be very tired here, but I can't figure out why this matrix multiplication by a scalar gives the following result (python)&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;Matrix named 'dx'&lt;/strong&gt;&lt;br/&gt;&#xA;        [ 1.6,  3.6,  0.4, 14.4, 25.6],&lt;br/&gt;&#xA;        [10. , 10. ,  0.4, 14.4,  3.6],&lt;br/&gt;&#xA;        [ 0.4,  0. ,  0. ,  1.6, 10. ],&lt;br/&gt;&#xA;        [ 6.4,  0. ,  3.6,  1.6,  0.4],&lt;br/&gt;&#xA;        [14.4,  0. , 25.6,  0.4,  6.4]&lt;br/&gt;&#xA;  &lt;br/&gt;&#xA;  &lt;strong&gt;10 * dx, in python, gives&lt;/strong&gt; &lt;br/&gt;&#xA;        [ -40.,  -60.,  -20., -120., -160.],&lt;br/&gt;&#xA;        [-100., -100.,  -20., -120.,  -60.],&lt;br/&gt;&#xA;        [ -20.,    0.,    0.,  -40., -100.],&lt;br/&gt;&#xA;        [ -80.,    0.,  -60.,  -40.,  -20.],&lt;br/&gt;&#xA;        [-120.,    0., -160.,  -20.,  -80.]&lt;br/&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;From what I understand, each member should be multiplicated by 10, but it's clearly not the case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using &lt;strong&gt;Python 3.6 &amp;amp; numpy&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What Am I missing ? &#xA;Link to matrix multiplication by scalar , pretty basic stuff :&#xA;&lt;a href=&quot;http://www.purplemath.com/modules/mtrxmult.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.purplemath.com/modules/mtrxmult.htm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks !&lt;/p&gt;&#xA;" OwnerUserId="47816" LastActivityDate="2018-03-14T21:02:02.397" Title="Can't understand this simple matrix multiplication in python" Tags="|matrix-factorisation|numpy|" AnswerCount="1" CommentCount="2" ClosedDate="2018-03-18T19:43:32.817" ContentLicense="CC BY-SA 3.0" />
  <row Id="29090" PostTypeId="2" ParentId="26556" CreationDate="2018-03-14T20:54:24.040" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a way to quantify the likelihood in ANN similar to logistic regression wherein regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Good question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, there is a way. The approach that can help you is called &lt;em&gt;partial dependence plot&lt;/em&gt; (PDP), see the links below for further details and examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach is model agnostic, i.e. works well for any predictive model, powerful yet simple. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main steps for one-dimensional partial dependence plot are as follows&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fit your model as usual&lt;/li&gt;&#xA;&lt;li&gt;Select the predictor of interest and a set of values to be investigated (e.g. income as in the article you refer to and values of say 50k, 70k, 80k, ..., 120k)&lt;/li&gt;&#xA;&lt;li&gt;For all observations in your dataset replace the values of your predictor with the first value from the set above (50k). &lt;/li&gt;&#xA;&lt;li&gt;Calculate the model output for the modified dataset from the previous step and calculate the average over all observations. &lt;/li&gt;&#xA;&lt;li&gt;Repeat steps 3-4 for the remaining values (70k, 80k, ...) and plot the values of your predictor along X axis and the corresponding averaged model predictions along Y axis. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;With one-dimensional PDP described above you can easily see the marginal impact of a predictor being analysed on the model output. Furthermore, one can use similar technique to perform multi-dimensional analysis, e.g. to investigate the impact of interactions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;partial dependence plots- scikit-learn documentation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/dansbecker/partial-dependence-plots&quot; rel=&quot;nofollow noreferrer&quot;&gt;partial dependence plot - tutorial by Dans Becker on Kaggle&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2018-03-14T20:54:24.040" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29091" PostTypeId="2" ParentId="29089" CreationDate="2018-03-14T21:02:02.397" Score="7" Body="&lt;p&gt;It works for me.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import numpy as np&#xA;&amp;gt;&amp;gt;&amp;gt; dx = np.matrix([[ 1.6, 3.6, 0.4, 14.4, 25.6],&#xA;... [10. , 10. , 0.4, 14.4, 3.6],&#xA;... [ 0.4, 0. , 0. , 1.6, 10. ],&#xA;... [ 6.4, 0. , 3.6, 1.6, 0.4],&#xA;... [14.4, 0. , 25.6, 0.4, 6.4]])&#xA;&amp;gt;&amp;gt;&amp;gt; 10 * dx&#xA;matrix([[  16.,   36.,    4.,  144.,  256.],&#xA;        [ 100.,  100.,    4.,  144.,   36.],&#xA;        [   4.,    0.,    0.,   16.,  100.],&#xA;        [  64.,    0.,   36.,   16.,    4.],&#xA;        [ 144.,    0.,  256.,    4.,   64.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You might want to check if you have redefined dx before the multiplication.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-03-14T21:02:02.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29092" PostTypeId="2" ParentId="29082" CreationDate="2018-03-14T21:18:42.573" Score="1" Body="&lt;p&gt;If $$y = w_1 + w_0 \left(\frac{x_{}-\mu_1}{\sigma_1} \right),$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then we have $$y = \left(w_1-\frac{\mu_1}{\sigma_1}\right)+ \left(\frac{w_0}{\sigma_1}\right)x$$&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-03-14T21:18:42.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29093" PostTypeId="1" CreationDate="2018-03-14T21:40:13.840" Score="2" ViewCount="15357" Body="&lt;p&gt;Let's stay I have a field with a continuous variable, like a count of people waiting in line. I want to take those values and create a categorical value based on quartiles. Let's say my range of values is 1 to 80 and the quartiles tell me that a &quot;very short&quot; line is less than 5 people, a &quot;short&quot; line in 6 to 30, a &quot;long&quot; line is 31 to 50 and a &quot;very long&quot; line is &gt;=51&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of different ways to write this in python/pandas/numpy but something tell me that one of you can come up with a method/snippet that is short and elegant. Note that I want the method to also generate the quartile values, something that I haven't really done in Python before. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2019-09-25T12:40:39.760" Title="Continuous variable to categorical by quartiles?" Tags="|python|statistics|pandas|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29094" PostTypeId="2" ParentId="29093" CreationDate="2018-03-14T21:54:58.800" Score="1" Body="&lt;p&gt;Unless I misunderstood you completely, I think &lt;a href=&quot;https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.mquantiles.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;mquantiles&lt;/a&gt; in scipy.stats easily does what you want.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-03-14T21:54:58.800" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29095" PostTypeId="1" AcceptedAnswerId="29134" CreationDate="2018-03-15T00:31:40.543" Score="0" ViewCount="65" Body="&lt;p&gt;While reading &lt;a href=&quot;https://www.geeksforgeeks.org/decision-tree/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; material on decision trees, I came across the following statement:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The construction of decision tree classifiers does not require any domain knowledge or &lt;strong&gt;parameter setting&lt;/strong&gt;, and therefore is&#xA;  &lt;strong&gt;appropriate for exploratory knowledge discovery&lt;/strong&gt;. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I have couple of doubts related to this statement. First one is the &lt;strong&gt;definition of parameter setting&lt;/strong&gt; in this context and another one is &lt;strong&gt;how being without parameter setting is apt for exploratory knowledge discovery&lt;/strong&gt;?&lt;/p&gt;&#xA;" OwnerUserId="47826" LastEditorUserId="47826" LastEditDate="2018-03-15T00:41:26.803" LastActivityDate="2018-11-23T12:25:36.860" Title="Formal definition for parameter setting in data mining context" Tags="|decision-trees|definitions|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29096" PostTypeId="2" ParentId="28196" CreationDate="2018-03-15T00:32:09.647" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I understood you correctly, these two facts perfectly reconcile in case your classes are not separable in one-dimensional feature space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following exaggerated toy example should illustrate such situation (and power of RF).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#uncomment the following line to remove all variables&#xA;# rm(list=ls())&#xA;&#xA;library(ggplot2)&#xA;library(caret)&#xA;library(MASS)&#xA;library(GGally)&#xA;library(dplyr)&#xA;&#xA;SEED &amp;lt;- 123456&#xA;SIMS &amp;lt;- 1000&#xA;&#xA;set.seed(SEED)&#xA;&#xA;x.signal &amp;lt;- rbind(&#xA;    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,0.9,0.9,1),2,2)),&#xA;    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,-0.9,-0.9,1),2,2)))&#xA;&#xA;x.noise &amp;lt;- mvrnorm(n = SIMS, c(0,0), matrix(c(1,0.9,0.9,1),2,2))&#xA;y &amp;lt;- rep(c(TRUE, FALSE), each=SIMS/2)&#xA;df &amp;lt;- data.frame(&#xA;    x1.signal=x.signal[,1],&#xA;    x2.signal=x.signal[,2], &#xA;    x1.noise=x.noise[,1], &#xA;    x2.noise=x.noise[,2], &#xA;    y=factor(y))&#xA;&#xA;yName &amp;lt;- &quot;y&quot;&#xA;xName &amp;lt;- setdiff(colnames(df), yName)&#xA;&#xA;ggpairs(df, aes(colour=y))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uTHrG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uTHrG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ctrl &amp;lt;- trainControl(&quot;cv&quot;, number = 5, verboseIter = TRUE)&#xA;rffit &amp;lt;- train(df[,xName], df[,yName], &#xA;    method = &quot;rf&quot;, trControl = ctrl, tuneLength = 5)&#xA;&#xA;varImpPlot(rffit$finalModel)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Y1zSg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y1zSg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## t = 0.061767, df = 997.98, p-value = 0.9508&#xA;t.test(filter(df, y==&quot;TRUE&quot;)$x1.signal, &#xA;    filter(df, y==&quot;FALSE&quot;)$x1.signal,&#xA;    var.equal = FALSE, paired = FALSE)&#xA;&#xA;## t = 0.22695, df = 996.04, p-value = 0.8205&#xA;t.test(filter(df, y==&quot;TRUE&quot;)$x2.signal,&#xA;    filter(df, y==&quot;FALSE&quot;)$x2.signal, &#xA;    var.equal = FALSE, paired = FALSE)&#xA;&#xA;&#xA;dplyr::group_by(df, y) %&amp;gt;% &#xA;    dplyr::summarise(mx1=mean(x1.signal), &#xA;    mx2=mean(x2.signal))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29781" LastEditorUserId="29781" LastEditDate="2018-03-15T01:27:28.100" LastActivityDate="2018-03-15T01:27:28.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29097" PostTypeId="2" ParentId="29093" CreationDate="2018-03-15T01:07:18.367" Score="7" Body="&lt;p&gt;Panda's &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/categorical.html&quot; rel=&quot;noreferrer&quot;&gt;Categorical Data Type&lt;/a&gt; is designed for that type of analysis,  &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.cut.html&quot; rel=&quot;noreferrer&quot;&gt;pandas.cut&lt;/a&gt; can divide by user-defined bins and &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.qcut.html&quot; rel=&quot;noreferrer&quot;&gt;pandas.qcut&lt;/a&gt; can create quantile-based discretization. Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;&#xA;df = pd.DataFrame({'value': np.random.randint(1, 80, 20)}) &#xA;df['group'] = pd.cut(df.value,&#xA;                     bins=[0, 5, 31, 51, 80],&#xA;                     labels=[&quot;very short&quot;, &quot;short&quot;, &quot;long&quot;, &quot;very long&quot;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-03-15T01:37:20.780" LastActivityDate="2018-03-15T01:37:20.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29098" PostTypeId="1" CreationDate="2018-03-15T02:52:11.873" Score="0" ViewCount="28" Body="&lt;p&gt;In the last video of his course on Convolutional Neural Networks, Andrew Ng was discussing using ConvNets on 3D input data. He mostly discussed cat scans, but also mentioned you could treat a movie as 3D data and run a ConvNet on it (where the z axis is time, i.e. the # frame of the movie). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find this pretty fascinating. I'm just curious if anybody is aware of anybody doing something like this in practice, and what they accomplished.&lt;/p&gt;&#xA;" OwnerUserId="44394" LastEditorUserId="29169" LastEditDate="2021-02-13T16:36:32.960" LastActivityDate="2021-02-13T16:36:32.960" Title="Has anybody created interesting ConvNets on 3D movie data?" Tags="|cnn|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29100" PostTypeId="2" ParentId="27126" CreationDate="2018-03-15T04:20:52.427" Score="1" Body="&lt;p&gt;A couple of things comes to mind.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;get_dummies can transform a dataframe with many columns, whereas LabelBinarizer will only do one column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;get_dummies outputs a dataframe (if the input is a dataframe) with a nicely formatted columns, whereas LabelBinarizer outputs a numpy array, so if you want to attach labels to them, you'd need to get them from the fitted instance of the LabelBinarizer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Inverse transform is more intuitive with LabelBinarizer, it has a method named inverse_transform, whereas with get_dummies you would need to do something like dummies.idxmax(axis=1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So overall, get_dummies seems to be a better choice&lt;/p&gt;&#xA;" OwnerUserId="39811" LastActivityDate="2018-03-15T04:20:52.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29101" PostTypeId="1" CreationDate="2018-03-15T05:04:29.340" Score="0" ViewCount="108" Body="&lt;p&gt;So, I am trying to create a Neural Network which will effectively separate 2 Gaussian curves with somewhat different means and standard deviations. My basic aim is, for some given input vector the Neural Network will select 2 points from the 2 Gaussian curves (selection based on the input vector) and compare them, and output a value depending on the Gaussian curve which gives a greater value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if I implement a simple Neural Network with 2 hidden layers the accuracy of my predictions is about 68.75% for 2 Gaussian curves with &lt;strong&gt;&lt;em&gt;N(0,1)&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;N(0,2000)&lt;/em&gt;&lt;/strong&gt; respectively and 100% for &lt;strong&gt;&lt;em&gt;N(0,1)&lt;/em&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;em&gt;N(500,1)&lt;/em&gt;&lt;/strong&gt;  (my NN outputs &lt;code&gt;0, 1&lt;/code&gt; and hence I can easily measure accuracy by &lt;code&gt;true_positives/total_samples&lt;/code&gt;). Clearly more the overlapping curves, less the accuracy as the NN cannot distinguish between the 2 curves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried increasing the number of hidden layers, changing learning rate but still there is no improvement in accuracy. I am using back-propagation with momentum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is what structure, learning algorithm and modifications in the Neural Network will help me to effectively separate 2 curves which is kind of overlapping for a better accuracy?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarification 1: I have generated the data myself for testing purposes, in real situation I will only have an input/output pair from a black-box, and all I know the black-box follows some Gaussian distribution. Also the input vector has to be arranged in some sort of way, and then if we fit the curve to it, it turns out to be Gaussian.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clarification 2: I am looking for possible hints. Any experienced person in this field might make a guess for some better methods. I know you cannot have a definite method, so suggestion of methods which might work are welcome.&lt;/p&gt;&#xA;" OwnerUserId="47832" LastActivityDate="2018-03-15T05:17:10.790" Title="How to separate overlapping curves more effectively?" Tags="|machine-learning|neural-network|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29102" PostTypeId="2" ParentId="29101" CreationDate="2018-03-15T05:17:10.790" Score="1" Body="&lt;p&gt;The problem here isn't your chosen neural network architecture, it's that you're working with literally unseparatable data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's isn't any practical way to distinguish observations near the mode of multiple distributions that share a mode. The best you can do is approximate the distance from the mode at which the high variance distribution is more likely than the low variance distribution, and classify observations relative to that. You can do that analytically, and your neural network isn't going to do any better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have you tried visualizing your data to better understand what you're tasking your model with here? This is a bit more extreme than &quot;kind of overlapping&quot; distributions.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-15T05:17:10.790" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29103" PostTypeId="2" ParentId="29057" CreationDate="2018-03-15T05:22:27.180" Score="1" Body="&lt;p&gt;I find the answer for my own question. I hope this helps to others. Thanks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/r_interface.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pandas.pydata.org/pandas-docs/stable/r_interface.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9663" LastActivityDate="2018-03-15T05:22:27.180" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29104" PostTypeId="2" ParentId="29087" CreationDate="2018-03-15T05:31:17.870" Score="0" Body="&lt;p&gt;Use &lt;code&gt;str_replace&lt;/code&gt; function in &lt;code&gt;stringr&lt;/code&gt; library. You just need the alphabets and spaces between them, so this will work-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; str_replace(&quot;965 - Vehicle parking area&quot;, &quot;[^A-Za-z]+&quot;, &quot;&quot;)&#xA;[1] &quot;Vehicle parking area&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-15T05:31:17.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29105" PostTypeId="1" AcceptedAnswerId="29108" CreationDate="2018-03-15T05:41:32.030" Score="5" ViewCount="1387" Body="&lt;p&gt;I think I understand both types of units in terms of just the math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is, how is it possible in practice for a GRU to perform as well as or better than an LSTM (which is what seems to be the norm)? I don't intuitively get how the GRU is able to make up for the missing cell content. The gates seem to be pretty much the same as an LSTM's gates, but with a missing part. Does it just mean that the cell in an LSTM is actually nearly useless?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Other questions have asked about the differences between GRU and LSTM. None of them (in my opinion) explain well enough &lt;em&gt;why&lt;/em&gt; a GRU works as well as an LSTM even without the memory unit, only that the lack of a memory unit is one of the differences that makes a GRU faster.&lt;/p&gt;&#xA;" OwnerUserId="47835" LastEditorUserId="47835" LastEditDate="2018-03-15T05:49:56.573" LastActivityDate="2018-03-15T10:57:03.097" Title="How can a GRU perform as well as an LSTM?" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29106" PostTypeId="2" ParentId="28972" CreationDate="2018-03-15T06:52:18.260" Score="1" Body="&lt;p&gt;I got it thanks to the scikit team, I put the answer here for the people to come. The split used in scikit uses weights in calculating the Gini coefficient, just add the following lines before returning:&#xA;    ....&#xA;    gk *= len(kids)/len(df)&#xA;    ga *= len(adults)/len(df)&lt;/p&gt;&#xA;" OwnerUserId="47635" LastActivityDate="2018-03-15T06:52:18.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29108" PostTypeId="2" ParentId="29105" CreationDate="2018-03-15T08:23:02.693" Score="3" Body="&lt;p&gt;GRU and LSTM are two popular RNN variants out of many possible similar architectures motivated by similar theoretical ideas of having a &quot;pass through&quot; channel where gradients do not degrade as much, and a system of sigmoid-based control gates to manage signals passing between time steps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even with LSTM, there are variations which may or may not get used, such as adding &quot;peephole&quot; connections between previous cell state and the gates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LSTM and GRU are the two architectures explored so far that do well across a wide range of problems, as verified by experiment. I suspect, but cannot show conclusively, that there &lt;em&gt;is no strong theory&lt;/em&gt; that explains this rough equivalence. Instead we are left with more intuition-based theories or conjectures:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;GRU has less parameters per &quot;cell&quot;, allowing it in theory to generalise better from less examples, at the cost of less flexibility.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;LSTM has a more sophisticated memory in the form of separating internal cell state from cell output, allowing it to output features useful for a task without needing to memorise those features. This comes at the cost of needing to learn extra gates which help map between state and features.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When considering performance of these architectures in general, you have to allow that some problems will make use of these strengths better, or it may be a wash. For instance, in a problem where forwarding the layer output between time steps is already a good state representation &lt;em&gt;and&lt;/em&gt; feature representation, then there is little need for the additional internal state of the LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In effect the choice between LSTM and GRU is yet another hyperparameter to consider when searching for a good solution, and like most other hyperparameters, there is no strong theory to guide an a priori selection.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-03-15T08:23:02.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29109" PostTypeId="1" AcceptedAnswerId="29111" CreationDate="2018-03-15T08:23:03.107" Score="1" ViewCount="867" Body="&lt;p&gt;I'm working with huge data sheet, and start learning Pandas, but i hit this challenge i have a loop and trying to move everything from my loop into Pandas but i not all i can find a way around.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;panda_dataframe = pd.read_sql(sql=sql, con=mysql_cnx, index_col='UUID')&#xA;&#xA;logging.debug('__setupProducts() - after mysql query : run time {time}'.format(time=datetime.datetime.now() - start_time))&#xA;logging.debug('__setupProducts() - Product found to handle: {count}'.format(count=len(panda_dataframe)))&#xA;&#xA;panda_dataframe['Price'] = panda_dataframe['Price'].apply(lambda x:float(x/100))&#xA;panda_dataframe['PriceNext'] = panda_dataframe['PriceNext'].apply(lambda x:float(x/100))&#xA;panda_dataframe['CostPrice'] = panda_dataframe['CostPrice'].apply(lambda x:float(x/100))&#xA;panda_dataframe['CostPriceReal'] = panda_dataframe['CostPriceReal'].apply(lambda x:float(x/100))&#xA;panda_dataframe['InStoreStock'] = panda_dataframe['InStoreStock'].apply(lambda x:int(x))&#xA;&#xA;logging.debug('__setupProducts() - restructer dataframe to right types : run time {time}'.format(time=datetime.datetime.now() - start_time))&#xA;&#xA;for product_uuid, product in panda_dataframe.iterrows():&#xA;    logging.info('Product: {title} loading and prepare...'.format(title=product['Title']))&#xA;&#xA;    try:&#xA;        product_data = store_remote_stock_dataframe.get_group(product_uuid)&#xA;&#xA;        product_data_onstock = product_data.loc[product_data['Stock'] &amp;gt; 0, ['Stock', 'CostPriceReal', 'CostPrice', 'Expected', 'DistributorUUID', 'Country']]&#xA;        product_data_outstock = product_data.loc[product_data['Stock'] &amp;lt;= 0, ['Stock', 'CostPriceReal', 'CostPrice', 'Expected', 'DistributorUUID', 'Country']]&#xA;        product_data = None&#xA;&#xA;        if len(product_data_onstock) &amp;gt; 0:&#xA;            stock_cost_price = product_data_onstock.sort_values(by=['CostPrice'], ascending=True).iloc[0,:]&#xA;            stock_cost_real_price = product_data_onstock.sort_values(by=['CostPriceReal'], ascending=True).iloc[0,:]&#xA;        elif len(product_data_outstock) &amp;gt; 0:&#xA;            stock_cost_price = product_data_outstock.sort_values(by=['CostPrice'], ascending=True).iloc[0,:]&#xA;            stock_cost_real_price = product_data_outstock.sort_values(by=['CostPriceReal'], ascending=True).iloc[0,:]&#xA;        else:&#xA;            stock_cost_price = None&#xA;            stock_cost_real_price = None&#xA;&#xA;        stock_cost_price = stock_cost_price.drop(['CostPriceReal','CostPrice']) if stock_cost_price is not None else None&#xA;        stock_cost_real_price = stock_cost_real_price.drop(['CostPriceReal','CostPrice']) if stock_cost_real_price is not None else None&#xA;    except:&#xA;        stock_cost_price = None&#xA;        stock_cost_real_price = None&#xA;&#xA;&#xA;&#xA;&#xA;    products.append({&#xA;        'uuid' : product_uuid,&#xA;        'title' : product['Title'],&#xA;        'price' : product['Price'],&#xA;        'price-next' : product['PriceNext'],&#xA;        'price-cost' : product['CostPrice'],&#xA;        'price-cost-real' : product['CostPriceReal'],&#xA;        'overwrites' : product['Overwrites'],&#xA;        'distributor-stock' : {&#xA;            'cost-price' : {&#xA;                'distributor' : stock_cost_price['DistributorUUID'] if stock_cost_price is not None else None,&#xA;                'stock' : stock_cost_price['Stock'] if stock_cost_price is not None else 0,&#xA;                'expected' : stock_cost_price['Expected']  if stock_cost_price is not None else -1,&#xA;                'country' : stock_cost_price['Country']  if stock_cost_price is not None else None,&#xA;            },&#xA;            'cost-price-real' : {&#xA;                'distributor' : stock_cost_real_price['DistributorUUID'] if stock_cost_real_price is not None else None,&#xA;                'stock' : stock_cost_real_price['Stock'] if stock_cost_real_price is not None else 0,&#xA;                'expected' : stock_cost_real_price['Expected'] if stock_cost_real_price is not None else -1,&#xA;                'country' : stock_cost_real_price['Country']  if stock_cost_real_price is not None else None&#xA;            }&#xA;        },&#xA;        'stock' : {&#xA;            'store' : int(product['InStoreStock']) if product['InStoreStock'] is not None else 0,&#xA;        },&#xA;        'manufacturer' : manufacturers[product['ManufacturerUUID']]['_id'] if product['ManufacturerUUID'] in manufacturers else None,&#xA;        'category' : categorys[product['CategoryUUID']]['_id'] if product['CategoryUUID'] in categorys else None,&#xA;    })&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;what i wich is move my &quot;try&quot; code shut be before my for loop becures then i can remove the for loop and continue whitout the for loop at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope sombody can help me to be better to Pandas so i can use the panda power :)&lt;/p&gt;&#xA;" OwnerUserId="47842" LastActivityDate="2018-03-15T09:20:04.293" Title="How to move for loop away and go pure Pandas?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29110" PostTypeId="1" CreationDate="2018-03-15T09:11:51.920" Score="1" ViewCount="98" Body="&lt;p&gt;I am working with a big data set (millions of observations) where for each observation I am trying to predict a probability (or score it) of being of a class. I haven't any labeled training data and it is impossible to collect it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I found a service that allows to query several times per hour with min amount of allowed observations starting from several thousands and with no upper boundary, ultimately service returns the proportion of the class for each tested group of observations. I assume here that an observation with a target class is more likely to appear in corresponding groups, and I would like to preserve this assumtion here, as without it the idea doesn't make any sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data can be non-randomly grouped into multiple slices by attributes where some of attributes has quite good positive or negative relationship with the class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;way to think about it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;| observation_id  | attr1 | attr2 | attr3 | … | attrN |&#xA;|-----------------|-------|-------|-------|---|-------|&#xA;| _1              | 1     | 0     | 1     | … | 0     |&#xA;| _2              | 0     | 0     | 1     | … | 1     |&#xA;| _3              | 0     | 0     | 1     | … | 0     |&#xA;| _4              | 1     | 0     | 1     | … | 1     |&#xA;| _5              | 1     | 0     | 0     | … | 0     |&#xA;| _6              | 1     | 0     | 0     | … | 1     |&#xA;| …               | …     | …     | …     | … | …     |&#xA;| M               | 1     | 0     | 1     | … | 0     |&#xA;&#xA;&#xA;If I group observations by attrs I receive sets of a different cardinality of observation and I can gather such info:&#xA;&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;attr1 = {_1,_4,_5,_6,...,M} and has 73% proportion of a target class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;attr3 = {_1,_2,_3,_4,...,M} and has 46% proportion of a target class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;attrN = {has some observations} and has some proportion of a class&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it turns out that any observation is a member of a group with a known probability (proportion) to have a target class if you pick one observation at random. The objective is to find a way to infer which observation relates to a target class from from class proportion for every observation given knowledge of the class proportion for every group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any model type or scoring approach to optimise this process to infer the likelihood of observation being of the class from knowing the proportion of the class in multiple groups where this observation lives? By optimisation I mostly mean minimisation of testing iterations as it takes quite a lot of time, and if possible gaining more information from lower amount of features (in this case - affinity to a group by attribute).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any hint or reference to an article/book/page will be super helpful :) Thanks a lot!&lt;/p&gt;&#xA;" OwnerUserId="47844" LastEditorUserId="47844" LastEditDate="2018-03-15T12:11:22.300" LastActivityDate="2018-03-15T12:11:22.300" Title="Predict class having only class proportions for every attribute (non labeled data)" Tags="|classification|data-mining|predictive-modeling|probability|scoring|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29111" PostTypeId="2" ParentId="29109" CreationDate="2018-03-15T09:20:04.293" Score="1" Body="&lt;p&gt;I always avoid for loops when operating on pandas rows - it’s slow and inefficient. If possible, try to creat some functions (&lt;code&gt;def func(x): ...&lt;/code&gt;) and then apply these functions to the rows of the dataframe: &lt;code&gt;df[‘col1’].apply(func)&lt;/code&gt; which will vectorise the operation of applying &lt;code&gt;func&lt;/code&gt; to each row, making it much faster and more efficient.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-03-15T09:20:04.293" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29114" PostTypeId="2" ParentId="29105" CreationDate="2018-03-15T10:57:03.097" Score="3" Body="&lt;p&gt;As other people pointed, there is no clear superior approach. As commonly happens in ML, more complex approaches are somehow more powerful than simpler ones (they can model more complex relationships) but this always comes at a cost. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the specific example of GRUs and LSTMs is exactly the same, LSTM have a more complex structure so it has the ability to model more temporal related features. GRUs are simpler (see picture below), so they will perhaps outperform LSTMs when data is scarce or when there is a high risk of overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at this &lt;a href=&quot;https://arxiv.org/pdf/1412.3555v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; with a nice empirical evaluation of this two approaches. If you go to the conclusion you will read:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;...we could not make concrete conclusion on which of the two gating units&#xA;  was better.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QloSR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QloSR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-15T10:57:03.097" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29115" PostTypeId="1" AcceptedAnswerId="29116" CreationDate="2018-03-15T12:05:15.310" Score="3" ViewCount="58261" Body="&lt;p&gt;I have my code where i want to apply a function on and overwirte the inputs based on the return data, here is my &lt;code&gt;apply&lt;/code&gt; section&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;panda_dataframe[[&#xA;    'distributor-stock-cost-price-real-distributor',&#xA;    'distributor-stock-cost-price-stock',&#xA;    'distributor-stock-cost-price-expected',&#xA;    'distributor-stock-cost-price-country',&#xA;    'distributor-stock-cost-price-real-distributor',&#xA;    'distributor-stock-cost-price-real-stock',&#xA;    'distributor-stock-cost-price-real-expected',&#xA;    'distributor-stock-cost-price-real-country'&#xA;]] = panda_dataframe[['uuid']].apply(lambda x:__getDistributorStockData(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then i got my function i apply to my dataframe.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def __getDistributorStockData(x):&#xA;    in_product_uuid = x&#xA;&#xA;&#xA;    out_cost_price_dist = None&#xA;    out_cost_price_stock = None&#xA;    out_cost_price_expected = None&#xA;    out_cost_price_country = None&#xA;    out_cost_price_real_dist = None&#xA;    out_cost_price_real_stock = None&#xA;    out_cost_price_real_expected = None&#xA;    out_cost_price_real_country = None&#xA;&#xA;&#xA;    try:&#xA;        product_data = store_remote_stock_dataframe.get_group(in_product_uuid)&#xA;&#xA;        product_data_onstock = product_data.loc[product_data['Stock'] &amp;gt; 0, ['Stock', 'CostPriceReal', 'CostPrice', 'Expected', 'DistributorUUID', 'Country']]&#xA;        product_data_outstock = product_data.loc[product_data['Stock'] &amp;lt;= 0, ['Stock', 'CostPriceReal', 'CostPrice', 'Expected', 'DistributorUUID', 'Country']]&#xA;&#xA;        if len(product_data_onstock) &amp;gt; 0:&#xA;            stock_cost_price = product_data_onstock.sort_values(by=['CostPrice'], ascending=True).iloc[0,:]&#xA;            stock_cost_real_price = product_data_onstock.sort_values(by=['CostPriceReal'], ascending=True).iloc[0,:]&#xA;&#xA;            out_cost_price_dist = stock_cost_price['DistributorUUID']&#xA;            out_cost_price_stock = stock_cost_price['Stock']&#xA;            out_cost_price_expected =  stock_cost_price['Expected']&#xA;            out_cost_price_country =  stock_cost_price['Country']&#xA;            out_cost_price_real_dist = stock_cost_real_price['DistributorUUID']&#xA;            out_cost_price_real_stock = stock_cost_real_price['Stock']&#xA;            out_cost_price_real_expected = stock_cost_real_price['Expected']&#xA;            out_cost_price_real_country = stock_cost_real_price['Country']&#xA;&#xA;        elif len(product_data_outstock) &amp;gt; 0:&#xA;            stock_cost_price = product_data_outstock.sort_values(by=['CostPrice'], ascending=True).iloc[0,:]&#xA;            stock_cost_real_price = product_data_outstock.sort_values(by=['CostPriceReal'], ascending=True).iloc[0,:]&#xA;&#xA;            out_cost_price_dist = stock_cost_price['DistributorUUID']&#xA;            out_cost_price_stock = stock_cost_price['Stock']&#xA;            out_cost_price_expected =  stock_cost_price['Expected']&#xA;            out_cost_price_country =  stock_cost_price['Country']&#xA;            out_cost_price_real_dist = stock_cost_real_price['DistributorUUID']&#xA;            out_cost_price_real_stock = stock_cost_real_price['Stock']&#xA;            out_cost_price_real_expected = stock_cost_real_price['Expected']&#xA;            out_cost_price_real_country = stock_cost_real_price['Country']&#xA;        else:&#xA;            stock_cost_price = None&#xA;            stock_cost_real_price = None&#xA;    except:&#xA;        stock_cost_price = None&#xA;        stock_cost_real_price = None&#xA;&#xA;    return [&#xA;        out_cost_price_dist,&#xA;        out_cost_price_stock,&#xA;        out_cost_price_expected,&#xA;        out_cost_price_country,&#xA;        out_cost_price_real_dist,&#xA;        out_cost_price_real_stock,&#xA;        out_cost_price_real_expected,&#xA;        out_cost_price_real_country&#xA;    ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My big issue is everything run fin, i have try it out on a single row, and its work fine, but its return inside a array, and now i want to apply it on every single row but i got this error.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;  File &quot;test.py&quot;, line 12, in &amp;lt;module&amp;gt;&#xA;    pricerule_calculate_run = PriceruleCalculate.run(method='business')&#xA;  File &quot;.....&quot;, line 940, in run&#xA;    __setupProducts(productUUID)&#xA;  File &quot;.....&quot;, line 507, in __setupProducts&#xA;    ]] = panda_dataframe[['uuid']].apply(lambda x:__getDistributorStockData(x), axis=1)&#xA;  File &quot;/usr/local/lib/python3.6/site-packages/pandas/core/frame.py&quot;, line 2514, in __setitem__&#xA;    self._setitem_array(key, value)&#xA;  File &quot;/usr/local/lib/python3.6/site-packages/pandas/core/frame.py&quot;, line 2544, in _setitem_array&#xA;    self.loc._setitem_with_indexer((slice(None), indexer), value)&#xA;  File &quot;/usr/local/lib/python3.6/site-packages/pandas/core/indexing.py&quot;, line 599, in _setitem_with_indexer&#xA;    raise ValueError('Must have equal len keys and value '&#xA;ValueError: Must have equal len keys and value when setting with an iterable&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;can eny how help me here and find my mistake, i'm still very new for Pandas and i'm optimize my data handler right away.&lt;/p&gt;&#xA;" OwnerUserId="47842" LastActivityDate="2019-05-25T06:36:57.730" Title="Pandas apply return: Must have equal len keys and value when setting with an iterable" Tags="|python|pandas|dataframe|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29116" PostTypeId="2" ParentId="29115" CreationDate="2018-03-15T13:05:33.753" Score="8" Body="&lt;p&gt;I found the issue, I need to return a &lt;code&gt;pd.Series()&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return pd.Series([&#xA;    out_cost_price_dist,&#xA;    out_cost_price_stock,&#xA;    out_cost_price_expected,&#xA;    out_cost_price_country,&#xA;    out_cost_price_real_dist,&#xA;    out_cost_price_real_stock,&#xA;    out_cost_price_real_expected,&#xA;    out_cost_price_real_country&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47842" LastEditorUserId="8501" LastEditDate="2019-05-25T06:36:57.730" LastActivityDate="2019-05-25T06:36:57.730" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29117" PostTypeId="1" AcceptedAnswerId="30144" CreationDate="2018-03-15T13:17:31.737" Score="9" ViewCount="1156" Body="&lt;p&gt;I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine learning or deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had searched for approaches, but unfortunately I didn't find any approach to take.  I had tried with bag of words model and glove word embedding to predict whether a string is name or city etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But, I didn't succeed with bag of words model and with GloVe there are a lot of names which are not covered in the embedding example :- lauren is present in Glove and laurena isn't&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did find this post &lt;a href=&quot;https://datascience.stackexchange.com/questions/8542/appropriate-algorithm-for-string-not-document-classification&quot;&gt;here&lt;/a&gt;, which had a reasonable answer but I couldn't the approached used to solve that problem apart from the fact that NLP and SVM were used to solve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions are appreciated&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks and Regards,&#xA;Sai Charan Adurthi.&lt;/p&gt;&#xA;" OwnerUserId="46754" LastActivityDate="2018-05-10T19:50:09.653" Title="what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc" Tags="|machine-learning|deep-learning|text-mining|nlp|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29118" PostTypeId="2" ParentId="28094" CreationDate="2018-03-15T13:25:13.330" Score="1" Body="&lt;p&gt;Matrix derivatives work a bit different than regular ones. The scalar parallel of $\nabla_{\theta} \theta^{T} X^{T}X\theta$ you make should be more like $\theta x^2 \theta$ which in turn is just $x^2 \theta^2$. (Note that I changed $\alpha$ for $x$ to avoid confusion with $X$.) You do not need to modify $\theta^{T} X^{T}X\theta$ so that the two $\theta$'s are beside each other, this actually is what you want.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just like you usually have $\frac{d}{d\theta} (x^2 \theta^2) = 2 x^2 \theta$, then in matrix notation the rule is&#xA;$$\nabla_{\theta} \theta^{T} X^{T}X\theta = 2 X^{T}X \theta.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I feel he's not really &quot;skipping steps&quot;, but applying a different step than the one you expected.&lt;/p&gt;&#xA;" OwnerUserId="47308" LastActivityDate="2018-03-15T13:25:13.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29122" PostTypeId="1" AcceptedAnswerId="29265" CreationDate="2018-03-15T15:17:44.897" Score="0" ViewCount="262" Body="&lt;p&gt;I am a little confused with using encog to create a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying text classification with a basic feed forward network. For the input data I have 200 unique words (features/inputs) to input into the network and 100 different pieces of text. so I have a matrix of 100x230, essentially 100 different training items each with the frequency of 230 words. I then have an output which classifies the data as A, B, C, D or E, so one column and 5 different outputs, therefore this is an ordinal column and is part of the CSV data set I read in thus I actually have a matrix of 100X231 where the 231 column is the output I desire to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can train and run this neural network no problem but I am confused about the number of output neurons to have. I would have thought there are 5 different classifications therefore I should have 5 different neurons, however I can't setup the code like that, as it complains:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;IMLDataSet trainingSet = new BasicMLDataSet(input, ideal);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My input variable is a double[100][230] and the ideal is just the column of expected classifications so double[100][1]. Because I only have one column and the data is normalised those A,B,C... values are turned into numbers between -1 and 1 and therefore when I run the neural net with 5 output neruons it complains that it should be only 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run the neural net with one neuron output it gives me what looks like correct and accurate answers but the value it gives is between -1 and 1. I am assuming the activation level is being matched to the output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question therefore is either how to denormalise this output to get back to my actual letter classification or how to use 5 neurons on the output so the network chooses the appropriate classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My neural network is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; BasicNetwork network = new BasicNetwork();&#xA; network.AddLayer(new BasicLayer(null, true, 230));&#xA; network.AddLayer(new BasicLayer(new ActivationTANH(), true, 16));&#xA; network.AddLayer(new BasicLayer(new ActivationTANH(), false, 1));&#xA; network.Structure.FinalizeStructure();&#xA; network.Reset();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47869" LastActivityDate="2018-03-19T14:51:36.263" Title="Encog neural network multiple outputs" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29125" PostTypeId="2" ParentId="16060" CreationDate="2018-03-15T15:42:06.953" Score="11" Body="&lt;p&gt;Just adding my 2 cents&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding an image classification task solved with a typical CNN Architecture consisting of a Backend (Convolutions + NL + possibly Spatial Pooling) which performs Representation Learning and of a Frontend (e.g. Fully Connected Layers, MLP) which solves the specific task, in this case image classification, the idea is to build a function $ f : I \rightarrow L $ able to map from the Spatial Domain $ I $ (Input Image) to the Semantic Domain $ L $ (Label Set) in a 2 step process which is &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Backend (Representation Learning) : $ f : I \rightarrow \mathcal{L} $ maps the Input to the Latent Semantic Space &lt;/li&gt;&#xA;&lt;li&gt;Frontend (Task Specific Solver) : $ f : \mathcal{L} \rightarrow L $ maps from the Latent Semantic Space to the Final Label Space  &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and it is performed using the following properties  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;spatial equivariance, regarding ConvLayer (Spatial 2D Convolution+NonLin e.g. ReLU) as a shift in the Layer Input produces a shift in the Layer Output (Note: it is about the Layer, not the single Convolution Operator)&lt;/li&gt;&#xA;&lt;li&gt;spatial invariance, regarding the Pooling Operator (e.g. Max Pooling passes over the max value in its receptive field regardless of its spatial position)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The closer to the input layer, the closer to the purely spatial domain $ I $ and the more important the spatial equivariance property which allows to build spatially equivariant hierarchical (increasingly) semantic representation &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The closer to the frontend, the closer to the latent purely semantic domain $ \mathcal{L} $ and the more important the spatial invariance as the specific meaning of the image is desired to be independent from the spatial positions of the features &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using fully connected layers in the frontend makes the classifier sensitive to feature position at some extent, depending on the backend structure : the deeper it is and the more the translation invariant operator (Pooling) used &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It has been shown in &lt;a href=&quot;http://cs231n.stanford.edu/reports/2016/pdfs/107_Report.pdf&quot; rel=&quot;noreferrer&quot;&gt;Quantifying Translation-Invariance in Convolutional Neural Networks&lt;/a&gt; that to improve the CNN Classifier Translation Invariance, instead of acting on the inductive bias (architecture hence depth, pooling, …) it's more effective to act on the dataset bias (data augmentation)&lt;/p&gt;&#xA;" OwnerUserId="33025" LastActivityDate="2018-03-15T15:42:06.953" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29127" PostTypeId="1" CreationDate="2018-03-15T15:51:15.890" Score="1" ViewCount="350" Body="&lt;p&gt;Using &lt;strong&gt;full-batch&lt;/strong&gt; gradient descent, &lt;strong&gt;stacking 100 layers&lt;/strong&gt; and using alpha 0.0001 results in steadily decreasing error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, after I implemented Batch Norm, the same scenario results in fluctuations. My implementation was verified by several people, so now I am wondering - why batch norm adds this stochastisity effect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I am no longer able to stack 100+ layers, only approximatelly 10 layer before stochasticity becomes very apparent and hard to control.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;What's more interesting -it seems to get worse with smaller learning rate! 0.4 is good, and 0.0001 results in smaller updates (as expected) but larger relative fluctuation (caught me by surprise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is it so?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; Just tried &lt;strong&gt;100 layers (each 63 neurons)&lt;/strong&gt;. It's very noisy, but I am able to more-or-less steadily reduce the error. It's less stochastic if I set learning rate to &lt;strong&gt;10&lt;/strong&gt; (&quot;ten&quot;, lol!) and will get only noisier if it's say &lt;strong&gt;0.5&lt;/strong&gt; which is incredibly weird&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, this learning rate is so high that the error does occasionally snap to a high value, but seems to work with 100 layers...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Notice - I am using full-batch gradient descent, with 50 elements&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using 100 LSTM units, stacked like a pillar onto each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number of timesteps is 50. Feature size 50 (lstm state has dimension of 50), because my sentence has 50 distinct characters, each character is encountered only once per epoch. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Performing backpropagation after 50 timesteps Using vanilla gradient descent, all fancy things like accelerated momentum or L2 norm, dropout are disabled. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure there is no error, but interested if anyone can tell why Batch Norm has this stochastic property, if that's common with people.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-03-19T09:14:48.710" LastActivityDate="2018-09-20T18:40:20.987" Title="is Batch Norm a little bit stochastic by default?" Tags="|backpropagation|batch-normalization|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29128" PostTypeId="1" CreationDate="2018-03-15T17:10:57.660" Score="0" ViewCount="352" Body="&lt;p&gt;I have been working with machine learning for about a year now, but mostly with large datasets. However, I am currently working on a problem with a &lt;em&gt;very&lt;/em&gt; small dataset. Here is my problem: I am creating a rocket fuel with 4 ingredients, x1, x2, x3, x4, and I want to maximize reaction strength, y. I have already mixed them in the arrangements below to get the corresponding values.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;(0.9)x1 + (0.0)x2 + (0.1)x3 + (0)x4 = 16.5&lt;/li&gt;&#xA;&lt;li&gt;(0.0)x1 + (0.9)x2 + (0.1)x3 + (0)x4 = 8.6&lt;/li&gt;&#xA;&lt;li&gt;(.45)x1 + (.45)x2 + (0.0)x3 + (0.1)x4 = 12.6&lt;/li&gt;&#xA;&lt;li&gt;(0.6)x1 + (0.3)x2 + (0.05)x3 + (.05)x4 = 18.9&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(0.3)x1 + (0.9)x2 + (0.05)x3 + (.05)x4 = 9.8&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My next question is, how should I design my next few mixtures to maximize the reaction strength?  Can you suggest any algorithms or statistical frameworks to get me started? Much appreciated. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerDisplayName="user47874" LastActivityDate="2019-01-22T15:13:13.483" Title="Statistical inference on a very small datasets" Tags="|predictive-modeling|statistics|bayesian|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29130" PostTypeId="2" ParentId="29128" CreationDate="2018-03-15T19:48:33.137" Score="1" Body="&lt;p&gt;There are two separate issues: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Sampling - Picking the optional ingredient level for next experiment to run. Given you have only have 4 explanatory variables, just plot them. Either all pairwise or a couple of 3d charts. With the outcome variable on the y or z axis. You'll then see the trend in the data. You can decide to get more data for interpolation (between the data points you already have) or extrapolation (data outside of the current range). There are frameworks, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Optimization&lt;/a&gt;, but that is too much work given the small dimensionality.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Inference - Predicting performance for new data. Given the data you have seen thus far (sample data), estimate parameters. In your example that would the estimating the contribution of each of the 4 ingredients, either individually or interaction. Those parameters could be scalar coefficients or &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_linear_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;distributions&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-15T19:48:33.137" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29131" PostTypeId="1" AcceptedAnswerId="29137" CreationDate="2018-03-15T20:24:43.197" Score="4" ViewCount="22164" Body="&lt;p&gt;I want to know feature names that a LogisticRegression() Model has used along with their corresponding weights in scikit-learn. I can access to weights using &lt;code&gt;coef_&lt;/code&gt;, but i did not know how can pair them with their corresponding weights. &lt;/p&gt;&#xA;" OwnerUserId="33245" LastActivityDate="2019-12-08T13:26:27.563" Title="feature names in LogisticRegression()" Tags="|python|scikit-learn|logistic-regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29132" PostTypeId="1" CreationDate="2018-03-15T20:28:02.687" Score="2" ViewCount="1443" Body="&lt;p&gt;The Nipals PCA algorithm calculates the scores and loadings of a data array iteratively.  If only (say) 3 scores and loadings are calculated from a data array with more than 3 variables, there is a residual matrix created (called E).  How can I extract this matrix from the SciKit Learn PCA algorithm so that I can create contribution charts?&lt;/p&gt;&#xA;" OwnerUserId="47882" LastActivityDate="2021-12-03T00:04:15.420" Title="How can I extract the residual array from the Scikit Learn PCA routine?" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29134" PostTypeId="2" ParentId="29095" CreationDate="2018-03-15T20:56:14.993" Score="1" Body="&lt;p&gt;First part about parameter setting regards to the fact that you don't need to define any parameters of the model, which is the case in per se linear regression which takes the form of ŷ =θ0+θ1x where &lt;span class=&quot;math-container&quot;&gt;$θ_0,θ_1$&lt;/span&gt; are parameters. Decision trees create those n-dimensional boundaries for future classification (regression) based on certain criteria. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are familiar with statistical inference, a good analogy would be good old parametric vs non-parametric tests. &lt;/p&gt;&#xA;" OwnerUserId="47883" LastEditorUserId="47826" LastEditDate="2018-11-23T12:25:36.860" LastActivityDate="2018-11-23T12:25:36.860" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29136" PostTypeId="2" ParentId="29098" CreationDate="2018-03-15T22:02:59.803" Score="1" Body="&lt;p&gt;The &lt;a href=&quot;http://www.thumos.info/&quot; rel=&quot;nofollow noreferrer&quot;&gt;2015 Thumos challenge&lt;/a&gt; was about action recognition in video collected from youtube - totalling 13k videos (430 hrs or 45 million frames) and 101 'action classes'. The video is of activities like people brushing their teeth, playing basketball, and surfing, and some video clips included multiple activities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The video is untrimmed, so it's closer to a real-world setting than video edited down to exclusively a single action of interest. The &lt;a href=&quot;http://storage.googleapis.com/www.thumos.info/thumos15_notebooks/TH15_UTS&amp;amp;CMU.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;winning&lt;/a&gt; and &lt;a href=&quot;http://storage.googleapis.com/www.thumos.info/thumos15_notebooks/TH15_MSRAsia.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;second place&lt;/a&gt; entries for action classification included CNNs in various guises (after a quick scan I couldn't find any entries that didn't). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another use you might be interested in is video captioning (not classification): &lt;a href=&quot;https://arxiv.org/abs/1511.03476&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; incorporates CNNs to describe in words what is taking place in each video (examples from p9 of the article, I love the descriptions at bottom right): &lt;a href=&quot;https://i.stack.imgur.com/iQpHm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iQpHm.png&quot; alt=&quot;HRNE video descriptions&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2018-03-15T22:02:59.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29137" PostTypeId="2" ParentId="29131" CreationDate="2018-03-15T23:09:29.160" Score="0" Body="&lt;p&gt;I made a scenario:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;from sklearn.linear_model.logistic import LogisticRegression&#xA;from sklearn.metrics import accuracy_score&#xA;&#xA;max_features = 100&#xA;tfidf = TfidfVectorizer(max_features=max_features)#stop_words='english',)# norm = None)#)&#xA;&#xA;#Simple&#xA;texts_train = ['positive sample', 'again positive', 'negative sample', 'again negative']&#xA;target_train = [1,1,0,0]&#xA;texts_test = ['negative', 'positive']&#xA;target_test = [0,1]&#xA;texts_train1 = tfidf.fit_transform(texts_train)&#xA;texts_test1 = tfidf.transform(texts_test)&#xA;classifier = LogisticRegression()&#xA;classifier.fit(texts_train1, target_train)&#xA;predictions = classifier.predict(texts_test1)&#xA;&#xA;print('accuracy (simple):', accuracy_score(target_test, predictions))&#xA;tfidf.get_feature_names()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;['again', 'negative', 'positive', 'sample']&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier.coef_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;array([[ 0.        , -0.56718183,  0.56718183,  0.        ]])&#xA;that makes sense!&lt;/p&gt;&#xA;" OwnerUserId="33245" LastActivityDate="2018-03-15T23:09:29.160" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29139" PostTypeId="1" CreationDate="2018-03-16T00:03:28.077" Score="-1" ViewCount="57" Body="&lt;p&gt;When working with prediction problems, is there a need to consider the change in time or not? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, when trying to predict the price of a house, we just have the current features and the current price. The classification problem is to predict if the house price will be above a threshold or not. The regression problem is to predict the exact price. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a need to have datasets with some time interval between them (for example X days or Y years) or we can predict with a dataset that was taken at a certain time? And in this case, is there a difference between classification and regression?&lt;/p&gt;&#xA;" OwnerUserId="47473" LastActivityDate="2018-05-15T08:05:20.367" Title="How many datasets are required in a prediction problem?" Tags="|classification|predictive-modeling|dataset|time-series|regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29140" PostTypeId="1" AcceptedAnswerId="29146" CreationDate="2018-03-16T00:55:12.120" Score="7" ViewCount="208" Body="&lt;p&gt;I understand that the models are only as good as the data you get, and bad design can generate really bad data. Nonrandom sampling, unbalanced/incomplete designs, confounding, can make data analysis really hard. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At what point should one be confident that they ran a &lt;em&gt;useful&lt;/em&gt; model? Do you just do a cross-validation with a training/test dataset and call it a day? Obviously &quot;all models are wrong, some are useful&quot; but at some point the tradeoffs with excluding too many parameters by LASSOing and strange transformations by getting BIC down become glaring.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tl;dr at the end of the day what makes you go &quot;I did the right thing for my company/project, and this should work&quot;&lt;/p&gt;&#xA;" OwnerUserId="47675" LastActivityDate="2018-03-16T09:29:11.877" Title="What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors?" Tags="|performance|accuracy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29141" PostTypeId="1" AcceptedAnswerId="29211" CreationDate="2018-03-16T01:35:16.840" Score="0" ViewCount="6674" Body="&lt;p&gt;I have a dataframe with a bunch of columns (words).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df&#xA;&#xA;        arg1 predicate&#xA;    0   PERSON        be&#xA;    1       it      Pick&#xA;    2  details      Edit&#xA;    3    title   Display&#xA;    4    title   Display&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I used a pretrained word2vec model to create a new df with all words replaced by vectors (1-D numpy arrays).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; get updated_df&#xA;&#xA;    updated_df = df.applymap(lambda x: self.filterWords(x))&#xA;    def filterWords(self, x):&#xA;        model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)&#xA;        if x in model.vocab:&#xA;            return model[x]&#xA;        else:&#xA;            return model['xxxxx']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;updated_df print:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;             arg1  \&#xA;        0  [0.16992188, -0.48632812, 0.080566406, 0.33593...   &#xA;        1  [0.084472656, -0.0003528595, 0.053222656, 0.09...   &#xA;        2  [0.06347656, -0.067871094, 0.07714844, -0.2197...   &#xA;        3  [0.06640625, -0.032714844, -0.060791016, -0.19...   &#xA;        4  [0.06640625, -0.032714844, -0.060791016, -0.19...   &#xA;&#xA;                                                   predicate  &#xA;        0  [-0.22851562, -0.088378906, 0.12792969, 0.1503...  &#xA;        1  [0.018676758, 0.28515625, 0.08886719, 0.213867...  &#xA;        2  [-0.032714844, 0.18066406, -0.140625, 0.115722...  &#xA;        3  [0.265625, -0.036865234, -0.17285156, -0.07128...  &#xA;        4  [0.265625, -0.036865234, -0.17285156, -0.07128...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to train a SVM(sklearn Linear SVC) with this data.&#xA;When I pass the updated_df as X_Train, I get &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.fit(updated_df, out_df.values.ravel())    &#xA;array = np.array(array, dtype=dtype, order=order, copy=copy)&#xA;ValueError: setting an array element with a sequence&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What is the right way of passing this as the input data to the classifier? &#xA;My y_train is fine.&#xA;If I get a hash of the words to create the updated_df like below, it works fine.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;updated_df = df.applymap(lambda x: hash(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I need to pass the word2vec vectors to establish a relationship between the words. I am new to python/ML and appreciate the guidance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Editing with the current status based on Theudbald's suggestion:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class ConcatVectorizer(object):&#xA;def __init__(self, word2vec):&#xA;    self.word2vec = word2vec&#xA;    # if a text is empty we should return a vector of zeros&#xA;    # with the same dimensionality as all the other vectors&#xA;    self.dim = len(word2vec.itervalues().next())&#xA;    print &quot;self.dim = &quot;, self.dim&#xA;&#xA;def fit(self, X, y):&#xA;    print &quot;entering concat embedding fit&quot;&#xA;    print &quot;fit X.shape = &quot;, X.shape&#xA;    return self&#xA;&#xA;def transform(self, X):&#xA;    print &quot;entering concat embedding transform&quot;&#xA;    print &quot;transform X.shape = &quot;, X.shape&#xA;    dictionary = {':': 'None', '?': 'None', '': 'None', ' ': 'None'}&#xA;    X = X.replace(to_replace=[':','?','',' '], value=['None','None','None','None'])&#xA;    X = X.fillna('None')&#xA;    print &quot;X = &quot;, X&#xA;    X_array = X.values&#xA;    print &quot;X_array = &quot;, X_array&#xA;&#xA;    vectorized_array = np.array([&#xA;        np.concatenate([self.word2vec[w] for w in words if w in self.word2vec]&#xA;                or [np.zeros(self.dim)], axis=0)&#xA;        for words in X_array&#xA;    ])&#xA;&#xA;    print &quot;vectorized array&quot;, vectorized_array&#xA;    print &quot;vectorized array.shape&quot;, vectorized_array.shape&#xA;    return vectorized_array&#xA;&#xA;&#xA;model = gensim.models.KeyedVectors.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)&#xA;    w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}&#xA;etree_w2v_concat = Pipeline([&#xA;    (&quot;word2vec vectorizer&quot;, ConcatVectorizer(w2v)),&#xA;    (&quot;extra trees&quot;, ExtraTreesClassifier(n_estimators=200))])&#xA;rf.testWordEmbClassifier(etree_w2v_concat)&#xA;&#xA;       def testWordEmbClassifier(self, pipe_obj):&#xA;    kb_fname = 'kb_data_3.csv'&#xA;    test_fname = 'kb_test_data_3.csv'&#xA;    kb_data = pd.read_csv(path + kb_fname, usecols=['arg1',&#xA;                                                        'feature_word_0',&#xA;                                                        'feature_word_1',&#xA;                                                        'feature_word_2',&#xA;                                                        'predicate'])&#xA;    kb_data_small = kb_data.iloc[0:5]&#xA;    kb_data_out = pd.read_csv(path + kb_fname, usecols=['output'])&#xA;    kb_data_out_small = kb_data_out.iloc[0:5]&#xA;    print kb_data_small&#xA;    pipe_obj.fit(kb_data_small, kb_data_out_small.values.ravel())&#xA;    print pipe_obj.predict(kb_data_small)&#xA;    self.wordemb_predictResult(pipe_obj, test_fname, report=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="39076" LastEditorUserId="39076" LastEditDate="2018-03-20T04:52:24.817" LastActivityDate="2018-03-20T04:52:24.817" Title="Use a dataframe of word vectors as input feature for SVM" Tags="|python|scikit-learn|pandas|svm|numpy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29143" PostTypeId="2" ParentId="29139" CreationDate="2018-03-16T07:05:16.920" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there a need to have datasets with some time interval between them (for example X days or Y years) or we can predict with a dataset that was taken at a certain time?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you have some data that displays pattern of changing prices (or your target) over time, then sure, you have to consider the time change. This data will then become a time-series data, which has to be processed differently than normal data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And in this case, is there a difference between classification and regression?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The problem of predicting prices at first-hand is regression, because the prices take continuous values. But you can convert it to classes according to your threshold. So in this case, (as far as I understand), there is no difference between regression and classification, as you are just putting numbers (prices) in different classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please comment if something is not clear.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-16T07:05:16.920" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29144" PostTypeId="2" ParentId="20012" CreationDate="2018-03-16T07:06:04.880" Score="2" Body="&lt;p&gt;I'd say BN goes after the ReLU and not before, in general it should be put between 2 layers so to normalize the layer output PDF before becoming another layer input &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The convolutive layer processing is composed of a Lin (Conv Operator) + NonLin (e.g. ReLU) processing (as the Artificial Neuron Processing) and a sparsifying nonlin like ReLU produces an output PDF which is non-negative as a result of filtering, so before passing it as the next layer input the BN can help renormalizing it &lt;/p&gt;&#xA;" OwnerUserId="33025" LastActivityDate="2018-03-16T07:06:04.880" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29145" PostTypeId="2" ParentId="29052" CreationDate="2018-03-16T07:21:13.390" Score="0" Body="&lt;p&gt;Use a constraint optimizer instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define your objective - what is a good result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then define your constraints (all points in exactly one group, no duplicate labels in any group) and run it.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-16T07:21:13.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29146" PostTypeId="2" ParentId="29140" CreationDate="2018-03-16T07:47:39.397" Score="3" Body="&lt;p&gt;Hey Welcome to the Site!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are saying is right, Data Science din't reach to the stage where it has some standard methods for achieving this(standard procedures, don't know we would be able to reach that stage in near future). But we have some general standards like:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Forecasting: ETS,ARIMA,SARIMA etc &lt;/li&gt;&#xA;&lt;li&gt;Prediction: Linear Regression,Random Forest, GLM, Neural Network etc&lt;/li&gt;&#xA;&lt;li&gt;Classification: Logistic Regression, Random Forest etc&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;When you go to granular level it is hard to generalize, as every business problem is different and one single method cannot be used for solving all the business problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to answer the next question how do you get confidence that the outcome is good enough, I assume that you have heard about RMSE,MAPE and many more for predictions and confusion matrix for classification problem. We use these metrics to see access the models performance, for example if you are trying to classify whether the given cell is cancer cell or not, there are 100 records in which 90 are non-cancer cells and 10 are cancer cell, your model gives 99% accuracy but could classify 5 out of 9 literally 55% of the total in such scenarios you need to look cannot use accuracy, you need to use F1 score etc. As you were asking about a model right, all models are not useful. True not all models built are going to go for production level, you would choose the best one and productionize it. You can re-train your model on a basis(Daily, Weekly,Monthly based on business requirement). Would you call it a day off post completion of validation? I wouldn't, I would go to the Subject Matter Expert present him the results ask him/her for their insights, if they both are inline then I would do a Beta testing on some actual data and then productionize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now to address your last question, There is no standard saying that this is good or bad, if it works for you, your Business then that is a Good Model. To convenience your mangers and subject matter(Data) experts, you need to dig deep into the data try all different scenarios ask as many questions as possible. Try understanding the data very well. So, you can answer Business Questions with data supportive answers(This is possible only when you are well-worse with data). As they are very good with business they would be asking questions with respect to business, you need to be ready with all such scenarios by understanding business and data well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I do have a feeling like you do. I did alot of things but nothing worked but you shouldn't be unhappy as you understood that these are the ways which would lead you to Unsuccessful results(best example is Thomas Alva Edison has used 1000 diff metals before using Tungsten to make a bulb). Similarly all the methods which we have tried are different steps you have tried to get the solution. My funda is, did I try something different/new everyday or not. Crucial part of this process is, maintenance of clear documentation at each and every step. Which would come in handy in the near future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anything in R&amp;amp;D is never a waste it is just an other try or experiment, so your work is never waste. Your are trying to build a solid base for the bright future of your company.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-03-16T09:29:11.877" LastActivityDate="2018-03-16T09:29:11.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29147" PostTypeId="2" ParentId="29140" CreationDate="2018-03-16T09:13:44.023" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What makes you confident in your results?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The appropriate method to evaluate whether you have modeled a real signal or noise is completely dependent on the question you are asking and the modeling approach you've used to address it. Many very thick books have been written on this topic, often constraining their attention to one problem domain and/or type of model. The complexities associated with model evaluation are a big component of why data scientists generally have graduate degrees. Which brings us to the second part of your question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;At what point do you think you can present your work to tech illiterate superiors?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your tech illiterate superiors don't have that graduate degree that would inform them how to evaluate your analysis. They're trusting you to present honest and accurate results. It is extremely easy to mislead people who aren't stats-fluent into believing whatever narrative you want to present. It's your responsibility to make sure your results are air tight, or at least in synch with your client's risk tolerance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your results are ready to be shared when you are satisfied that you are interpreting them correctly, and you have a plan for how to communicate them plainly. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;at the end of the day what makes you go &quot;I did the right thing for my company/project, and this should work&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have built a model that accomplishes my goal.&lt;/li&gt;&#xA;&lt;li&gt;I am satisfied that my modeling approach is sound and repeatable. If I'm working on a prediction task, I also want to be sure that my model generalizes well to out-of-sample data.&lt;/li&gt;&#xA;&lt;li&gt;I have evaluated the projected impact of utilizing my model and am reasonably confident that the benefits of applying it justify the time and effort I am putting in building it.&lt;/li&gt;&#xA;&lt;li&gt;I have a clear path forward to implementation. I have a technical plan for making my results actionable, and know whose support I need to make it happen.&lt;/li&gt;&#xA;&lt;li&gt;I am confident that I can communicate my results in a way that will convince non-technical stakeholders that my results are real and assuage their concerns.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-16T09:13:44.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29148" PostTypeId="1" CreationDate="2018-03-16T09:31:19.573" Score="0" ViewCount="164" Body="&lt;p&gt;To read Context Free Grammars we can use &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nltk.CFG.fromstring(&quot;&quot;&quot; &#xA;S-&amp;gt; NP VP`...&#xA;    &quot;&quot;&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it can't read a feature based grammar which is like?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;S -&amp;gt; NP[NUM=?n] VP[NUM=?n]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So how to read feature based grammar from a string without actually having to save it in a fcfg file to load it later?&lt;/p&gt;&#xA;" OwnerUserId="39338" LastActivityDate="2019-01-09T10:01:15.237" Title="How to read Feature Based Grammar from a string" Tags="|python|nlp|nltk|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29149" PostTypeId="2" ParentId="29132" CreationDate="2018-03-16T09:37:26.853" Score="1" Body="&lt;p&gt;I'm not super familiar with PLS/NIPALS, but I think you need to construct it yourself. Multiply the scores by the loadings and subtract the resulting matrix from the data to get the residual matrix. See here for a more detailed discussion: &lt;a href=&quot;https://learnche.org/pid/latent-variable-modelling/projection-to-latent-structures/how-the-pls-model-is-calculated&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://learnche.org/pid/latent-variable-modelling/projection-to-latent-structures/how-the-pls-model-is-calculated&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-16T09:37:26.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29150" PostTypeId="1" CreationDate="2018-03-16T10:03:20.503" Score="0" ViewCount="45" Body="&lt;p&gt;To data scientists who use python for development, how do you demo your work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used to demo my work in Shiny as my developments was mainly in R. Now I develop 99% in Python but not sure of the best way to demo work (predictive modelling, data analytics side, pattern detection,..etc)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I use Dash, but the amount if time I spend on html and css makes it less appealing.&lt;/p&gt;&#xA;" OwnerUserId="27537" LastActivityDate="2018-03-16T11:47:59.903" Title="Demoing data science work" Tags="|visualization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29152" PostTypeId="2" ParentId="29150" CreationDate="2018-03-16T11:47:59.903" Score="1" Body="&lt;p&gt;Use a Jupyter notebook. Write good descriptions in Markdown between blocks of code. Here are some examples showing machine learning in Python:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ogrisel/notebooks&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ogrisel/notebooks&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/ogrisel/notebooks/blob/master/Labeled%20Faces%20in%20the%20Wild%20recognition.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ogrisel/notebooks/blob/master/Labeled%20Faces%20in%20the%20Wild%20recognition.ipynb&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2018-03-16T11:47:59.903" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29153" PostTypeId="2" ParentId="27782" CreationDate="2018-03-16T12:10:40.463" Score="1" Body="&lt;p&gt;In general, you perform object detection to find potential faces, and then you perform face recognition on the face(s) detected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This article walks through the various steps quite well:&#xA;&lt;a href=&quot;https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The author also has a Python library that does recognition of multiple faces in a scene with video examples:&#xA;&lt;a href=&quot;https://github.com/ageitgey/face_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ageitgey/face_recognition&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2018-03-16T12:10:40.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29154" PostTypeId="2" ParentId="29052" CreationDate="2018-03-16T12:15:42.060" Score="1" Body="&lt;p&gt;You can write your own algorithm.  I drafted something up quickly. It can be significantly optimized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's make some random data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;&#xA;n = 9&#xA;x = np.random.rand(n,2)&#xA;y = np.zeros((n,))&#xA;y[n//3:2*n//3] = 1&#xA;y[2*n//3::] = 2&#xA;&#xA;plt.scatter(x[:,0], x[:,1], c=y)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/r4e1T.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/r4e1T.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's get the interdistance of each of these points.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dists = np.asarray([np.linalg.norm(i-j) for i in x for j in x]).reshape(n,n)&#xA;&#xA;plt.imshow(dists)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MrjFt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MrjFt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will make a list which will hold the radius of each circle for each point. For each point we will iterate over the other points by their relative nearness. If an associated label is not yet seen, add it to the temporary list. Otherwise, we end the function and take the average of the last distance and the current illegal point.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;radii = []&#xA;&#xA;for row in dists:&#xA;    labels = []&#xA;    dis = []&#xA;    for i in np.argsort(row):&#xA;        if y[i] not in labels:&#xA;            labels.append(y[i])&#xA;            dis = row[i]&#xA;        else:&#xA;            dis = (dis + row[i])/2&#xA;            break&#xA;    radii.append(dis)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we can plot these circles by &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fig, ax = plt.subplots(figsize=(10,10))&#xA;&#xA;for ix, i in enumerate(radii):&#xA;    circle = plt.Circle((x[ix, 0], x[ix, 1]), i, color='b', fill=False, alpha = 0.5)&#xA;    ax.add_artist(circle)&#xA;&#xA;plt.scatter(x[:,0], x[:,1], c=y)&#xA;plt.xlim([-0.2,1.4])&#xA;plt.ylim([-0.2,1.4])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/RfqkO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RfqkO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If we increase the number of datapoints $n$ we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qG3Tp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qG3Tp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-16T12:35:33.627" LastActivityDate="2018-03-16T12:35:33.627" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29155" PostTypeId="2" ParentId="29127" CreationDate="2018-03-16T12:34:21.057" Score="1" Body="&lt;p&gt;What about trying to use a BN implementation provided by a widely use lib e.g. Keras instead of yours, to test your implementation correctness? &#xA;If after this change you get different observations then it's possible your implementation is in some way different to the standard one &lt;/p&gt;&#xA;&#xA;&lt;p&gt;About the learning rate, one of the advantages of BN is that it should allow to use higher learning rate because it should combat possible scale explosion happening because of this &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having more information about the NN architecture could help &lt;/p&gt;&#xA;" OwnerUserId="33025" LastActivityDate="2018-03-16T12:34:21.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29156" PostTypeId="1" AcceptedAnswerId="29158" CreationDate="2018-03-16T13:53:15.990" Score="0" ViewCount="60" Body="&lt;p&gt;I am trying ML techniques in language processing.&#xA;I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most of them occurs once or twice.&#xA;So for example from texts:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;0: One text here&lt;/p&gt;&#xA;&lt;p&gt;1: Another text there&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I got&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;    One Another  Text  Here  There   Target&#xA;  0   1       0     1      1     0     True&#xA;  1   0       1     1      0     1    False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;So if word &amp;quot;one&amp;quot; occured once I have got as a column and it is False for all other 2999 texts.&#xA;Should I drop these columns? Or should use different technique?&#xA;This amount of columns make me some problems becuase it takes a lot of time to build a classify.&lt;/p&gt;&#xA;" OwnerUserId="41451" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-16T14:13:57.743" Title="Should I remove features that occur very rarely to build a model?" Tags="|feature-selection|preprocessing|naive-bayes-classifier|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29158" PostTypeId="2" ParentId="29156" CreationDate="2018-03-16T14:13:57.743" Score="2" Body="&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;yes, if they occur so rarely, they can only lead to overfit, so it's better to ignore them as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Longer answer:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually one puts all those unique occurrence in a single feature/token, and that's the way I suggest you to proceed. So if you have two features that appears only once, you can join them to create a feature that appears twice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But anyway, 3000 short text are too few to create a NLP model that can work well. &#xA;To mitigate your lack of data, you can use pre-trained word embeddings like the ones available &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. Doing so you can also keep most of the single occurring words/features, because in those word embeddings they already have a defined semantic meaning.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-03-16T14:13:57.743" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29159" PostTypeId="1" AcceptedAnswerId="29166" CreationDate="2018-03-16T14:21:43.977" Score="1" ViewCount="328" Body="&lt;p&gt;I'm researching a regression model to predict a target value that has four features, all of which are categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The categories are not fixed, e.g. one is a customer identifier. How could my model handle making a prediction for a customer identifer it &lt;em&gt;hasn't&lt;/em&gt; seen before, based on the remaining features it has been trained with?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have considered having a model for each of the features that could predict which category label is most similar based on the other three remaining features (or multiple similar category labels could be used and take an average of the target values for those).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My only concern with with this method is that it's not that scalable, I'm going to want to extend the model with more and more categorical features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some technique that could create an 'unknown' label for each of the features so the model can handle this case or would the prediction likely be completely inaccurate?&lt;/p&gt;&#xA;" OwnerUserId="47923" LastActivityDate="2018-03-16T15:43:34.017" Title="How can I make a prediction in a regression model if a category has not been observed already?" Tags="|machine-learning|regression|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29160" PostTypeId="1" AcceptedAnswerId="29163" CreationDate="2018-03-16T14:47:03.320" Score="0" ViewCount="28327" Body="&lt;p&gt;I'm working with a data set of movies which has various info on them. One of the columns contains the various genres a movie may belong to like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JrxpN.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JrxpN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to do is count how often a genre occurs in each column, in above example a corresponding series would look like (created the series myself):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t7cpP.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t7cpP.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I extract this information from the original dataframe using pandas?&lt;/p&gt;&#xA;" OwnerUserId="47924" LastActivityDate="2018-03-16T17:42:57.167" Title="Counting the occurrence of each string in a pandas dataframe column" Tags="|dataset|pandas|dataframe|" AnswerCount="1" CommentCount="2" ClosedDate="2018-03-19T01:13:34.000" ContentLicense="CC BY-SA 3.0" />
  <row Id="29161" PostTypeId="2" ParentId="29019" CreationDate="2018-03-16T14:55:35.657" Score="9" Body="&lt;p&gt;Might not be the answer you are seeking, but I'll still have a go:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, quick review of word2Vec, assume we are using skip gram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input:  10 000&lt;/li&gt;&#xA;&lt;li&gt;Hidden:  300 &lt;/li&gt;&#xA;&lt;li&gt;Output: 10 000&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There is a matrix &lt;strong&gt;E&lt;/strong&gt; between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons -  a response to a corresponding incoming 1-hot vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning.  And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This second matrix will make &lt;strong&gt;an entirely different&lt;/strong&gt; one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. &lt;strong&gt;In other words, this output won't be your original one-hot.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why a second matrix is needed. &#xA;At the output, we perform a softmax, like in a classification problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This allows us to express a relation &quot;word&quot;--&gt;embedding--&gt;&quot;context-neighbor-word&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, backpropagation can be done, to correct the &lt;strong&gt;Input-Hidden&lt;/strong&gt; weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that case you &lt;strong&gt;wouldn't&lt;/strong&gt; use this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;strong&gt;You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why a second matrix is needed during training:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once again, recall, there is no activation at the hidden layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can dictate the network what &quot;one-hot&quot; must have been created in response to your original &quot;input one-hot&quot;, and can punish network if it fails to generate a correct answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-10-19T03:53:36.157" LastActivityDate="2018-10-19T03:53:36.157" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="29162" PostTypeId="1" AcceptedAnswerId="29167" CreationDate="2018-03-16T14:55:49.313" Score="0" ViewCount="1586" Body="&lt;p&gt;I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the &quot;#&quot; prior to tokenizing? What about the &quot;@&quot; symbol? &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-03-16T17:31:33.507" Title="How do NLP tokenizers handle hashtags?" Tags="|nlp|tokenization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29163" PostTypeId="2" ParentId="29160" CreationDate="2018-03-16T15:07:05.743" Score="1" Body="&lt;p&gt;Something Equivalent to this should work:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the function(&lt;code&gt;pandas.Series.str.count&lt;/code&gt;) in this &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.count.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; will even make it easier...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.split.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;split&lt;/code&gt;&lt;/a&gt; by regex :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#using your sample&#xA;df[['class1', 'class2', 'class3',..]] = df['genre'].str.split('|', expand=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternative to the above method (but iterating the dataframe)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  l = list(df[index-key].values)&#xA;  l = (&quot;,&quot;.join(l))&#xA;  l.split('|')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And if need remove column &lt;code&gt;genre&lt;/code&gt; add &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;drop&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = df.drop('genre', axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then you can use &lt;code&gt;value_counts()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But this assumes that you have same length of each &lt;code&gt;genre&lt;/code&gt; or apply a check first and proceed accordingly..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try using &lt;code&gt;Counter&lt;/code&gt; from the collections module(after splitting them, just update each key's value)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example &lt;a href=&quot;https://docs.python.org/3/library/collections.html#collections.Counter&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Counter&lt;/code&gt;&lt;/a&gt; dict &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from collections import Counter&#xA;&amp;gt;&amp;gt;&amp;gt; Counter(['apple','red','apple','red','red','pear'])&#xA;Counter({'red': 3, 'apple': 2, 'pear': 1})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="29575" LastEditDate="2018-03-16T17:42:57.167" LastActivityDate="2018-03-16T17:42:57.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29164" PostTypeId="1" AcceptedAnswerId="31042" CreationDate="2018-03-16T15:37:56.283" Score="1" ViewCount="1545" Body="&lt;p&gt;I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test. &#xA;I train my model using random forest and logistic regression and it worked fine.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(123)&#xA;tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 )&#xA;randPridct = predict(tweetRand, newdata = test_sparse)&#xA;table(test_sparse$label,randPridct &amp;gt;=0.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Its is working fine on test data which divided from raw content. &#xA;But when i am running it on a new unseen data it is throwing an exception.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; predicrRand_test=predict(tweetRand, newdata=sparse_4testing)&#xA;Error in eval(predvars, data, env) : object 'run' not found&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;SO how should i deal with these situation. I am new to data science. Please help. &lt;/p&gt;&#xA;" OwnerUserId="47926" LastEditorUserId="47926" LastEditDate="2018-03-16T16:19:55.043" LastActivityDate="2018-05-30T14:59:53.630" Title="Text classification- What to do when train and test data have different features" Tags="|machine-learning|r|text-mining|random-forest|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29166" PostTypeId="2" ParentId="29159" CreationDate="2018-03-16T15:43:34.017" Score="1" Body="&lt;p&gt;Technically, you can't. That is one of the limitations of regression models; they are really only effective for values/ranges that they have seen before. Your use of categorical values makes it even more complex. But even with continuous variables, it is not recommended to use regression models for these &quot;unknown&quot; values. &lt;/p&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-03-16T15:43:34.017" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29167" PostTypeId="2" ParentId="29162" CreationDate="2018-03-16T15:44:37.497" Score="4" Body="&lt;p&gt;The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider this example tweet: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;Hi, we need you!&lt;/code&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;code&gt;#Hi #Weneedyou&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you use &lt;em&gt;TreeBank&lt;/em&gt; or &lt;em&gt;WordPunct&lt;/em&gt; tokenizers the output will be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However if you use &lt;em&gt;Whitespace&lt;/em&gt; Tokenizer, the result is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; ['Hi', 'we', 'need', 'you!', '#Hi', '#Weneedyou']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similar discrepancies can be found in terms such as &lt;code&gt;can't&lt;/code&gt; or &lt;code&gt;pre-order&lt;/code&gt; for instance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you need to consider what is a token for your task at hand. In my previous example &lt;code&gt;Hi&lt;/code&gt; make sense either with or without &lt;code&gt;#&lt;/code&gt;. However &lt;code&gt;Weneedyou&lt;/code&gt; without the hash is just a poorly written word. Maybe you want to keep the hashtags in your corpus or maybe not, but this depends on what you want to do with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So first you need to know how your tokenizer handles these cases and decide if you want to remove them beforehand or later to keep hashtags in your corpus or only (sometimes weird) words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of the &lt;code&gt;@&lt;/code&gt; is exactly the same, you can keep it, remove it or maybe delete the whole instance &lt;code&gt;@user&lt;/code&gt; as you don't want to keep user names in your corpus. As I said, it all depends on your task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: In case you want to play around with different tokenizers, try &lt;a href=&quot;http://text-processing.com/demo/tokenize/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45166" LastEditorUserId="45166" LastEditDate="2018-03-16T17:31:33.507" LastActivityDate="2018-03-16T17:31:33.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29169" PostTypeId="1" CreationDate="2018-03-16T16:43:30.257" Score="-1" ViewCount="102" Body="&lt;p&gt;&lt;strong&gt;Overview&lt;/strong&gt;&#xA;The data set I am working with considers a team that annually plays a 5-game home schedule. My goal is to identify the fans that are most likely to defect for the upcoming season, meaning not renew their season pass. This is my Y variable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Description of Dataset&lt;/strong&gt;&#xA;The dataset contains 2 sheets. The first sheet has 17 x variables that influence whether or not someone will likely defect. (i.e., price of ticket, whether or not they used their ticket for games 1-5, age, income, and so on). This sheet is data taken from the previous year, so I also have the actual Y values (those who did and did not renew their season pass for the previous season)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second sheet is where I will make my predictions for the upcoming season on whether or not someone will renew their season pass. The X variables are updated and already given for this current year. All I need to do is used the given data and predict the Y variable (whether someone will renew their season pass or default)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Suggestions Wanted&lt;/strong&gt;&#xA;I need advice on which classification method would provide me with the highest accuracy. These are the three models I know, so please only suggest one of the three: Decision tree, Random Forests, or Logistic Regression?&lt;/p&gt;&#xA;" OwnerUserId="47929" LastActivityDate="2018-03-16T17:24:33.443" Title="Need Advice, Classification Problem in Python: Should I use Decision tree, Random Forests, or Logistic Regression?" Tags="|python|classification|random-forest|logistic-regression|decision-trees|" AnswerCount="1" CommentCount="2" ClosedDate="2018-03-19T19:02:20.720" ContentLicense="CC BY-SA 3.0" />
  <row Id="29170" PostTypeId="1" CreationDate="2018-03-16T16:47:44.047" Score="1" ViewCount="419" Body="&lt;p&gt;I am following &lt;a href=&quot;https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd&quot; rel=&quot;nofollow noreferrer&quot;&gt;this example&lt;/a&gt; to learn a bit about the use of auto-encoders in fraud detection.&#xA;Now that I reached the end of the article, two questions rose in mind:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Can we train the network in an unsupervised way, freeze the layers weights, add a sigmoid layer as an output then finally use this model as a binary classifier?&lt;/li&gt;&#xA;&lt;li&gt;Are auto-encoders considered as the state-of-the-art method in fraud detection? What other alternatives can be used in this case?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="40515" LastEditorUserId="1330" LastEditDate="2021-08-23T18:24:04.820" LastActivityDate="2021-08-23T18:24:04.820" Title="Fraud detection using auto-encoders and Keras" Tags="|classification|keras|class-imbalance|autoencoder|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29171" PostTypeId="2" ParentId="29169" CreationDate="2018-03-16T17:24:33.443" Score="2" Body="&lt;p&gt;In Machine Learning, you can not say that a particular model will perform better than all other in every condition. Some times Logistic Regression may outperform a Random Forest/SVM, sometimes SVM can outperform Logistic Regression and RF. So, use every model you can for your purpose. There is no shortcut here.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-16T17:24:33.443" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29173" PostTypeId="1" CreationDate="2018-03-16T20:36:20.670" Score="2" ViewCount="13836" Body="&lt;p&gt;I have been trying to understand this &lt;em&gt;sliding window technique&lt;/em&gt; but to no avail and really unsure as to how I would implement it.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;My dataset:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I have hourly values for the electric load for a year (over 8700 data points) - image below. I'm going to split the dataset into a training set  (1 Jan to 30 Sept) and a test set (1 Oct to 31 Dec).&#xA;&lt;a href=&quot;https://i.stack.imgur.com/0IeS3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0IeS3.png&quot; alt=&quot;First few values of dataset&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I'm going to use &lt;em&gt;supervised learning&lt;/em&gt; techniques such as &lt;em&gt;Regression trees&lt;/em&gt; and &lt;em&gt;random forests&lt;/em&gt; (basically anything that is available to me in &lt;a href=&quot;/questions/tagged/scikit-learn&quot; class=&quot;post-tag&quot; title=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; aria-label=&quot;show questions tagged &amp;#39;scikit-learn&amp;#39;&quot; rel=&quot;tag&quot; aria-labelledby=&quot;tag-scikit-learn-tooltip-container&quot;&gt;scikit-learn&lt;/a&gt;), train them on the training set then make predictions on the test-set.&lt;/p&gt;&#xA;&lt;p&gt;I understand that I need to use &lt;em&gt;historical known values&lt;/em&gt; as input features to input into a model. As a result, I created &lt;code&gt;&amp;quot;Load_lagN&amp;quot;&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Qs:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Am I correct in saying that because I have created 10 lagged variables (&lt;span class=&quot;math-container&quot;&gt;$Xt-1$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$Xt-10$&lt;/span&gt;), this is the equivalent of using a sliding window of size 10?&lt;/li&gt;&#xA;&lt;li&gt;How would I then just simply train the model on the training set and make predictions on the test set without using the sliding model?&lt;/li&gt;&#xA;&lt;li&gt;With the sliding window model, does that assume only the past n values (10 in my case) are relevant?&lt;/li&gt;&#xA;&lt;li&gt;If I have split my dataset into a training and test set, how does the regression tree become trained and then make predictions in the test set (I'll stick to &lt;em&gt;one-step-ahead&lt;/em&gt; forecasting for now) whilst implementing this sliding window technique?&lt;/li&gt;&#xA;&lt;li&gt;I've come across this method - &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;TimeSeriesSplit()&lt;/code&gt;&lt;/a&gt; - is this what I need to use for the sliding window technique or is it only a cross validator?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;My understanding of the sliding window method:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;As you can see in the image below, I use values of 10:00 to 19:00 to predict the value at 20:00, move the window so that this new value is now included, and then predict the value for 21:00. This keeps happening until I have exhausted the training set. I then make predictions.&lt;/p&gt;&#xA;&lt;p&gt;What are your thoughts?&lt;/p&gt;&#xA;" OwnerUserId="44668" LastEditorUserId="69263" LastEditDate="2024-04-01T15:50:19.443" LastActivityDate="2024-04-01T15:50:19.443" Title="time series forecasting - sliding window method" Tags="|scikit-learn|time-series|supervised-learning|forecasting|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 4.0" />
  <row Id="29174" PostTypeId="1" CreationDate="2018-03-16T20:45:55.560" Score="3" ViewCount="988" Body="&lt;p&gt;I have a categorical feature that I one-hot encoded and used in my XGBoost model, but it consistently underperforms as a predictor compared to the other predictors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Then&lt;/strong&gt; I created a new variable that contains the same kind of information that the categorical feature has... &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Imagine I'm interested in predicting price of a house and the&#xA;  categorical feature is the town that it is in, and the numerical&#xA;  feature is a feature I generated ranking the town's &lt;em&gt;relative&#xA;  expensiveness&lt;/em&gt;, based on some prior knowledge of the towns.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TownA - 100&#xA;TownB - 40&#xA;TownC - 65&#xA;TownD - 15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now all of a sudden the new numerical variable that was directly derived from the categorical one is outperforming. Is this because XGBoost just works better with numerical variables and maybe some predictive capability is lost when I one-hot encode the variable?&lt;/p&gt;&#xA;" OwnerUserId="46886" LastActivityDate="2018-03-16T21:25:42.640" Title="Is XGBoost better with numeric predictors?" Tags="|xgboost|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29175" PostTypeId="1" CreationDate="2018-03-16T20:50:51.933" Score="2" ViewCount="21" Body="&lt;p&gt;Dynamic Memory networks are described &lt;a href=&quot;https://arxiv.org/pdf/1506.07285.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&#xA;. I understand what is going on for question answering task but when it comes to sequence to sequence modeling, they describe it in 4th paragraph of 2.4 answer module.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In the sequence modeling task, we wish to label each word&#xA;  in the original sequence. To this end, the DMN is run in&#xA;  the same way as above over the input words. For word t,&#xA;  we replace Eq. 8 with $e^i = h^{i}_{t}$&#xA;  . Note that the gates for the&#xA;  first pass will be the same for each word, as the question is the same. This allows for speed-up in implementation&#xA;  by computing these gates only once. However, gates for&#xA;  subsequent passes will be different, as the episodes are different.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I could not understand how the global gates are computed and why would they be equal for each word for first pass. They are going to be computed as a function of question vector (same) and word vector (different for each word). This is how vague i am on what is going on here. Can somebody explain how the piece in the paragraph mentioned come together?&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2018-03-16T20:50:51.933" Title="How are dynamic memory networks employed in sequence to sequence modelling" Tags="|deep-learning|nlp|stanford-nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29176" PostTypeId="2" ParentId="29174" CreationDate="2018-03-16T21:25:42.640" Score="5" Body="&lt;p&gt;By default, feature importance in &lt;code&gt;xgboost&lt;/code&gt; is given by how many times a given feature appears as a split feature across all trees in the ensemble. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When one-hot encoded, each newly created dummy variable can only take the values 0 and 1, and so can only appear once in each (sub)tree. However, when combining the values into one numeric feature by giving each category a different value, the feature can appear many more times on different levels in each tree, which brings up the importance score.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-03-16T21:25:42.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29177" PostTypeId="1" CreationDate="2018-03-16T22:30:10.380" Score="1" ViewCount="37" Body="&lt;p&gt;How would gather requirements for a data science project? How do you convince stakeholders that whatever you develop is going to benefit them? How to build trust to make use of data science for the business operations?&lt;/p&gt;&#xA;" OwnerUserId="47242" LastActivityDate="2018-03-18T06:15:40.797" Title="Requirements collecting" Tags="|python|bigdata|" AnswerCount="2" CommentCount="0" ClosedDate="2018-03-20T00:04:33.593" ContentLicense="CC BY-SA 3.0" />
  <row Id="29179" PostTypeId="1" AcceptedAnswerId="29307" CreationDate="2018-03-17T04:03:19.480" Score="0" ViewCount="22" Body="&lt;p&gt;I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the model accuracy. I want to know the reason, I thought that maybe there is a different distribution in my training and testing data, that lead to this, and checked this hypothesis, the result was that distributions are different. In general case any idea is appreciated.&lt;/p&gt;&#xA;" OwnerUserId="33245" LastEditorUserId="33245" LastEditDate="2018-03-20T12:37:42.497" LastActivityDate="2018-03-20T13:17:52.697" Title="Feature has a pattern in relation to class but does not enhance classifier to predict class" Tags="|machine-learning|logistic-regression|distribution|feature-construction|machine-learning-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29180" PostTypeId="1" CreationDate="2018-03-17T06:23:34.670" Score="1" ViewCount="650" Body="&lt;p&gt;I have a sample of 50,000 images, some of which are shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Vy4Is.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Vy4Is.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; $\qquad$&#xA; &lt;a href=&quot;https://i.stack.imgur.com/zZiuY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zZiuY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; $\qquad$ &lt;a href=&quot;https://i.stack.imgur.com/Ar8QI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ar8QI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; $\qquad$ &lt;a href=&quot;https://i.stack.imgur.com/X0kkG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X0kkG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; $\qquad$ &lt;a href=&quot;https://i.stack.imgur.com/f4UbB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f4UbB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Associated to these images are labels for the digit with the largest pixel size. My goal is to build a machine learning model to predict the largest digit in an image by pixel size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To that end, I used transfer learning on the resnext model, but only found an accuracy of 60%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that &lt;a href=&quot;https://github.com/kkweon/mnist-competition&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; implementation uses transfer learning to train a model to predict MNIST digits, I would now like to crop each training image to retain only the largest digit and then train the model using the &lt;a href=&quot;https://github.com/kkweon/mnist-competition&quot; rel=&quot;nofollow noreferrer&quot;&gt;linked&lt;/a&gt; implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is, how I do crop the training images to retain only the digit in each image with the largest size.&lt;/p&gt;&#xA;" OwnerUserId="45297" LastEditorUserId="45297" LastEditDate="2018-03-17T18:38:50.307" LastActivityDate="2018-05-17T20:19:37.210" Title="Multiple digits MNIST and transfer learning" Tags="|neural-network|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29183" PostTypeId="1" AcceptedAnswerId="29203" CreationDate="2018-03-17T08:41:07.300" Score="1" ViewCount="49" Body="&lt;p&gt;I want to do a quick computation in R that involves estimating the ideal gas constant from experimental data.  Ideally I'd have data from a monatomic gas at different pressures, volumes, temperatures and moles (or just pressure and volume, keeping the others fixed).  Anyone have an idea of where I might find a data set like this?  An R data frame would be nice, but I'll take excel, tab delimited, or whatever.&lt;/p&gt;&#xA;" OwnerUserId="47958" LastActivityDate="2018-03-18T06:02:29.693" Title="Ideal gas data needed" Tags="|data|" AnswerCount="1" CommentCount="1" ClosedDate="2018-03-19T19:03:10.657" ContentLicense="CC BY-SA 3.0" />
  <row Id="29184" PostTypeId="2" ParentId="18667" CreationDate="2018-03-17T08:44:35.517" Score="2" Body="&lt;p&gt;I got around 98% accuracy using ReLu activation function. I have used the following architecture :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;fully connected layer with 300 hidden units&lt;/li&gt;&#xA;&lt;li&gt;ReLu activation &lt;/li&gt;&#xA;&lt;li&gt;fully connected layer with 10 hidden units&lt;/li&gt;&#xA;&lt;li&gt;Softmax layer&lt;/li&gt;&#xA;&lt;li&gt;Output Clipping 1e-10 to 0.999999 to avoid log(0) and value greater than 1&lt;/li&gt;&#xA;&lt;li&gt;Cross entropy loss&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think you should add output clipping and then train it, hope that will work fine.&lt;/p&gt;&#xA;" OwnerUserId="43692" LastEditorUserId="29575" LastEditDate="2018-03-17T16:33:04.767" LastActivityDate="2018-03-17T16:33:04.767" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29185" PostTypeId="1" CreationDate="2018-03-17T09:11:56.397" Score="0" ViewCount="444" Body="&lt;p&gt;I would like to predict next 12 months of employee count based on 3 or more years of data that includes salaries, profit and employee count. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also interpolated monthly values for these three variables for known years to better suit to the problem and generate more training examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on analysis I would like to apply polynomial prediction model for predicting employee count at step &lt;strong&gt;t&lt;/strong&gt; from &lt;strong&gt;t-1&lt;/strong&gt;, then from &lt;strong&gt;t&lt;/strong&gt; to &lt;strong&gt;t+1&lt;/strong&gt; and up to &lt;strong&gt;t+11&lt;/strong&gt; which corresponds to December. I do not have available salaries and profit data at prediction times so I cannot use them as &lt;strong&gt;X&lt;/strong&gt; vector to predict employee count &lt;strong&gt;y&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using &lt;strong&gt;Arima&lt;/strong&gt; and it works fine, but it works with univariate series.&#xA;I tried also to reframe these three time series in pattern sequences with window or lag 1 and prediction steps 12 which gives (13,39) matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have some ideas how this problem can be approached, how it can be trained and how prediction can be triggered? Also what could be other options for reframing the time series?&lt;/p&gt;&#xA;" OwnerUserId="47959" LastEditorUserId="29575" LastEditDate="2018-03-17T16:43:00.420" LastActivityDate="2018-03-17T16:43:00.420" Title="Ideas for using polynomial regression for multivariate time series prediction" Tags="|machine-learning|scikit-learn|time-series|regression|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29186" PostTypeId="2" ParentId="29177" CreationDate="2018-03-17T12:00:57.877" Score="0" Body="&lt;p&gt;An answer to the third question: &quot;How to build trust to make use of data science for the business operations?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It heavily depends on the type and structure of data that you want to get access and the application that you want to provide. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if data owner has a concern about the privacy of its users, and you just want to execute some statistical queries over their data, you should design and propose a privacy-preserving method for accessing to their data. There are some techniques like &lt;a href=&quot;https://en.wikipedia.org/wiki/Differential_privacy&quot; rel=&quot;nofollow noreferrer&quot;&gt;Differential Privacy (or Randomized Response)&lt;/a&gt; which provide a well-defined privacy guarantee for data owner.&lt;/p&gt;&#xA;" OwnerUserId="38657" LastActivityDate="2018-03-17T12:00:57.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29190" PostTypeId="1" CreationDate="2018-03-17T15:40:47.157" Score="0" ViewCount="64" Body="&lt;p&gt;Based on the question, what are they and what are their differences? I've seen a lot those terms but cannot find an appropriate answer.&lt;/p&gt;&#xA;" OwnerUserId="48969" LastActivityDate="2018-03-17T17:06:11.823" Title="What are temporal and local data?" Tags="|machine-learning|deep-learning|data-mining|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-19T01:04:20.017" ContentLicense="CC BY-SA 3.0" />
  <row Id="29192" PostTypeId="1" AcceptedAnswerId="29193" CreationDate="2018-03-17T16:31:39.967" Score="0" ViewCount="956" Body="&lt;p&gt;My dataset consists of short videos of 4/5 time-steps each (frames), and the problem is classifying this video (multi-label classification). The idea is to use an LSTM but I'm wondering if the sequence length is not enough.&#xA; What are the suggested sequence length? May 4/5 time steps be enough?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. could you please post some link to scientific articles to endorse your claims?&lt;/p&gt;&#xA;" OwnerUserId="48970" LastEditorUserId="48970" LastEditDate="2018-03-18T13:42:32.103" LastActivityDate="2018-03-18T13:42:32.103" Title="What is the minimum/suggested sequence length for training an LSTM?" Tags="|deep-learning|rnn|lstm|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-19T19:06:44.053" ContentLicense="CC BY-SA 3.0" />
  <row Id="29193" PostTypeId="2" ParentId="29192" CreationDate="2018-03-17T16:58:01.567" Score="0" Body="&lt;p&gt;LSTMs would run into problems beyond 500 time steps. &#xA;original paper by &lt;a href=&quot;http://www.bioinf.jku.at/publications/older/2604.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hochreiter and Schmidhuber&lt;/a&gt;: &quot;LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: Your video is 60 seconds long. I would start by encoding 5 frames per second using a CNN (use resnet (SEnets have better performance but i havent read the paper) till the layer just before final affine layer which feeds into the softmax encode your images. Use these encodings as inputs in a GRU (Hence a 300 time steps for this GRU) and use its final state to make the class prediction (use a separate loss for each label?)&lt;/p&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="32169" LastEditDate="2018-03-18T07:53:28.690" LastActivityDate="2018-03-18T07:53:28.690" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29194" PostTypeId="2" ParentId="29190" CreationDate="2018-03-17T17:06:11.823" Score="1" Body="&lt;p&gt;Temporal data is across time. Data points are correlated and you model them using a variant of RNNs (LSTMs, GRUs, Tree recursive NNs etc). example would be text script of speech. Notice how each data point would be correlated with the previous data point&lt;/p&gt;&#xA;&#xA;&lt;p&gt;local data is ambiguous and could mean a lot of things. local to a particular location? local data as opposed to server data? local to a particular task? etc etc. Please provide context&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2018-03-17T17:06:11.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29197" PostTypeId="1" CreationDate="2018-03-17T21:36:12.880" Score="1" ViewCount="1577" Body="&lt;p&gt;I have an input array  X, which is of the shape &lt;code&gt;(38000,32,1)&lt;/code&gt;; the output array Y is of &lt;code&gt;(38000,1)&lt;/code&gt;, the element of Y can be &lt;code&gt;0&lt;/code&gt; or an numerical value, such as &lt;code&gt;0.040139&lt;/code&gt; or &lt;code&gt;1.075341&lt;/code&gt;, or some other numerical values, etc. A typical vector in X looks like as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AqGdx.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AqGdx.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to build an model to predict Y based on X. This is the model, two convolutional layer, followed by max-pooling, then one convolutional layer, one pooling, then a fully connected layer. The detailed architecture is as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;outX = conv_filter(20, 50, strides=1, activation='relu')(inputs)&#xA;outX = conv_filter(5, 60, activation='relu')(outX)&#xA; outX = MaxPooling1D(2)(outX)&#xA;outX = conv_filter(2, 70, activation='relu')(outX)&#xA;outX = MaxPooling1D(2)(outX)&#xA;&#xA;outX = Dense(300, activation='relu')(outX)&#xA;outX = Flatten()(outX)&#xA;predictions = Dense(1,activation='linear')(outX)&#xA;model = Model(inputs=[inputs],outputs=predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The loss function is 'mean_squared_error'.The result does not look very good. One of the reason I can think is about the numerical values for the input vector, which tends to be very small. Can it cause any numerical issue? Are there any suggestions to pre-process them? Besides, are there any improvement can be made on the network architecture? Any suggestions would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="21767" LastActivityDate="2018-07-16T09:59:51.183" Title="building a prediction model using CNN" Tags="|deep-learning|predictive-modeling|keras|tensorflow|cnn|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="29201" PostTypeId="2" ParentId="29029" CreationDate="2018-03-18T04:51:20.057" Score="1" Body="&lt;p&gt;As &lt;a href=&quot;https://datascience.stackexchange.com/users/14675/ncasas&quot;&gt;ncasas&lt;/a&gt; answered, layer stacking and lesser number of filters do help. I didn't use dropout as I have very few parameters. Regularization is my next approach. I used &lt;strong&gt;Batch Normalization&lt;/strong&gt; to generalize the model. It works very well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without Batch Normalization, model overfitted around 65% but using a single Batch Normalization generalized well upto 70%. Use it after &lt;strong&gt;convolutional layer&lt;/strong&gt; for better results.&lt;/p&gt;&#xA;" OwnerUserId="47728" LastActivityDate="2018-03-18T04:51:20.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29202" PostTypeId="1" CreationDate="2018-03-18T05:30:29.403" Score="4" ViewCount="839" Body="&lt;p&gt;I'm fairly new to ML, so as a learning exercise to get familiar with Keras I'm trying to learn some word2vec style embeddings from a dataset. I'm confused about the behavior of the &lt;a href=&quot;https://keras.io/preprocessing/sequence/#skipgrams&quot; rel=&quot;noreferrer&quot;&gt;skipgram utility&lt;/a&gt;, in particular the sampling table argument that's supposed to be populated from &lt;a href=&quot;https://keras.io/preprocessing/sequence/#make_sampling_table&quot; rel=&quot;noreferrer&quot;&gt;make_sampling_table&lt;/a&gt;. I understand the idea behind it, but I'm seeing some strange outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a toy example, let's pretend that we have the following parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;vocab_size = 3&#xA;sequence = [0,1,0,1,1,0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So a small vocabulary with 3 words in it, only 2 of which show up in the sequence. Generating the skipgrams &amp;amp; labels works fine without a sampling table at all:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pairs, labels = skipgrams(sequence, vocab_size)&#xA;print(&quot;pairs: &quot; + str(pairs) + &quot; labels: &quot; + str(labels))&#xA;&#xA;OUTPUT: pairs: [[1, 1], [1, 2], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 2], [1, 1], [1, 1], [1, 1], [1, 1]] labels: [0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But when I try to pass in a sampling table so as to not oversample the common words, I get no outputs at all:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pairs, labels = skipgrams(sequence, vocab_size, sampling_table=make_sampling_table(vocab_size))&#xA;print(&quot;pairs: &quot; + str(pairs) + &quot; labels: &quot; + str(labels))&#xA;&#xA;OUTPUT: pairs: [] labels: []&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is: what's going on here? Why does the act of passing a sampling table into the skipgrams method make it produce no output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="48997" LastActivityDate="2018-12-14T02:02:30.547" Title="Confusion about Keras' skipgram and sampling table utilities" Tags="|machine-learning|nlp|keras|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29203" PostTypeId="2" ParentId="29183" CreationDate="2018-03-18T06:02:29.693" Score="0" Body="&lt;p&gt;The National Institute of Standards and Technology (NIST) has ideal gas datasets &lt;a href=&quot;https://catalog.data.gov/dataset/nist-computational-chemistry-comparison-and-benchmark-database-srd-101&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-18T06:02:29.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29204" PostTypeId="2" ParentId="29180" CreationDate="2018-03-18T06:10:02.817" Score="1" Body="&lt;p&gt;You need to manually or automatically draw bounding boxes around each digit. Then compute the area for each box and take the largest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given they are non-overlapping black digits on a white background, &lt;a href=&quot;https://en.wikipedia.org/wiki/Template_matching&quot; rel=&quot;nofollow noreferrer&quot;&gt;template matching&lt;/a&gt; would work.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-18T06:10:02.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29205" PostTypeId="2" ParentId="29177" CreationDate="2018-03-18T06:15:40.797" Score="0" Body="&lt;p&gt;Ask a lot of questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;What is in your biggest pain point?&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;What do you want to automate to make your job easier?&quot;&lt;/li&gt;&#xA;&lt;li&gt;&quot;What data do you have?&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then listen to the answers. Build things that are useful. Teach people how to use the new systems.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-18T06:15:40.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29208" PostTypeId="1" CreationDate="2018-03-18T08:59:43.963" Score="1" ViewCount="41" Body="&lt;p&gt;Encoded text at character level:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;tvletwgzkrqvuhtwamuluhpkskpmpmiwtvuhamqvmviwlrvikquhtwamuluhqgvipmmvulkriwpmqvtwleuhamqvmviwlrvikquhtwamuluhqgqvqvtwviezlemvxeuhamqvmviwlrvikquhtwamuluhpkskvieniwlrvikquhqvmvuhqgpmpmiwletwulenokuhxepmuhtwiwululentvuhtwamuluhvimvuhsktwlemvezskenuhtwtvuhulqvkrezuhamypmvamdfuhulenamguuhraskvipmyptwqvuhtwamuluhxepmuhvimvenulgzenypuhenuhsatvuhvipmdfuhqgletwsklepmuhulqvlemvxeuhtwamuluhxepmuhtwiwululentvuhenuhqvmvuhpmpmiwletwulenok&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Map to one of 10 classes: 0-9.&lt;/p&gt;&#xA;" OwnerUserId="47867" LastEditorUserId="29575" LastEditDate="2018-03-19T13:56:54.693" LastActivityDate="2018-03-19T13:56:54.693" Title="How do I learn encoding of a text that is encoded at character level?" Tags="|deep-learning|nlp|" AnswerCount="0" CommentCount="4" ClosedDate="2020-07-13T14:58:57.523" ContentLicense="CC BY-SA 3.0" />
  <row Id="29211" PostTypeId="2" ParentId="29141" CreationDate="2018-03-18T10:19:31.407" Score="1" Body="&lt;p&gt;In my opinion, scikit-learn raises an error because &lt;em&gt;updated_df&lt;/em&gt; is composed of 2 features (columns) with list formats. Therefore, for a given observation x_i : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_i = [arg1_i, predicate_i] = [[vector_arg1_i], [vector_predicate_i]].&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Scikit-learn can't handle this format of input features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are mutiple ways to train a suprevised machine learning model after Word2Vec text processing. A common one is to sum or to average columns &lt;em&gt;arg1&lt;/em&gt; and &lt;em&gt;predicate&lt;/em&gt; in order to have following observation x_i structure : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_i = [(arg1_i + predicate_i) / 2] = [(vector_arg_i + vector_predicate_i) / 2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;More explanations and a gentle comparison between Word2Vec and CountVectorizer features engineering approaches for text classification : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-03-18T10:19:31.407" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29214" PostTypeId="2" ParentId="16177" CreationDate="2018-03-18T12:04:40.210" Score="1" Body="&lt;p&gt;I think that &lt;a href=&quot;https://hackernoon.com/latest-deep-learning-ocr-with-keras-and-supervisely-in-15-minutes-34aecd630ed8&quot; rel=&quot;nofollow noreferrer&quot;&gt;this &lt;em&gt;Tutorial&lt;/em&gt;&lt;/a&gt; is what you need.&lt;/p&gt;&#xA;&lt;p&gt;The architecture is described in &lt;a href=&quot;https://www.youtube.com/watch?v=uVbOckyUemo&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt;. It is based on a CNN and some LSTM neural networks.&lt;/p&gt;&#xA;&lt;p&gt;I dont know if its the best architecture for this kind of stuf, but I think that it gives a good understanding on how solving the problem.&lt;/p&gt;&#xA;" OwnerUserId="38510" LastEditorUserId="98307" LastEditDate="2020-08-13T12:32:43.363" LastActivityDate="2020-08-13T12:32:43.363" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29218" PostTypeId="1" CreationDate="2018-03-18T13:52:41.277" Score="2" ViewCount="757" Body="&lt;p&gt;Is there a similar tool as the tfidf vectorizer which converts a collection of raw documents to a matrix of TF-IDF features in a R package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for reference.&lt;/p&gt;&#xA;" OwnerUserId="30903" LastEditorUserId="30903" LastEditDate="2018-03-19T11:43:11.987" LastActivityDate="2018-05-18T12:22:25.400" Title="Tfidfvectorizer in R for character n-grams" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29219" PostTypeId="2" ParentId="16177" CreationDate="2018-03-18T15:19:18.843" Score="1" Body="&lt;p&gt;What I'm going to refer to is introducing some papers which are about this context. The papers have available data-set and there code can be accessed easily. Actually there are a lot of works in this context, but I suggeest you reading the following papers which are relevant to your question. In English, the nature of the language is in a way that you make a word using combination of letters and usually it is not needed to connect the letters to construct words. There are other languages that letters have to be connected to make words. In &lt;code&gt;OCR&lt;/code&gt; problems of those languages it is a commen practice to consider the &lt;em&gt;connected components&lt;/em&gt;. The papers that I'm going to refer to have a good data-set that can be used for both supervised and unsupervised methods. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.spiedigitallibrary.org/conference-proceedings-of-spie/9445/94450Z/Sub-word-image-clustering-in-Farsi-printed-books/10.1117/12.2181404.short&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sub-word image clustering in Farsi printed books&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.spiedigitallibrary.org/conference-proceedings-of-spie/9402/94020C/Clustering-of-Farsi-sub-word-images-for-whole-book-recognition/10.1117/12.2075931.short&quot; rel=&quot;nofollow noreferrer&quot;&gt;Clustering of Farsi sub-word images for whole-book recognition&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sid.ir/En/Journal/ViewPaper.aspx?ID=352817&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sub-word Image Clustering in Old Printed Documents Using Template Matching&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.spiedigitallibrary.org/conference-proceedings-of-spie/9402/94020H/A-comparison-of-1D-and-2D-LSTM-architectures-for-the/10.1117/12.2075930.short&quot; rel=&quot;nofollow noreferrer&quot;&gt;A comparison of 1D and 2D LSTM architectures for the recognition of handwritten Arabic&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The last paper is a great paper and is somehow the result of the previous ones.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-03-18T15:26:25.690" LastActivityDate="2018-03-18T15:26:25.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29223" PostTypeId="1" CreationDate="2018-03-18T17:27:08.967" Score="9" ViewCount="13755" Body="&lt;p&gt;In Machine Learning Kernels on Kaggle I often see EDAs with structured data. So, I was wondering, if there are any recommended/standard procedures for EDA with image datasets. What kind of statistical analyses do you conduct, what kind of plots do you draw and what aims do you have in mind?&lt;/p&gt;&#xA;" OwnerUserId="47948" LastActivityDate="2020-09-02T12:33:46.823" Title="Exploratory Data Analysis with Image Datset" Tags="|machine-learning|python|neural-network|r|computer-vision|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29224" PostTypeId="2" ParentId="29202" CreationDate="2018-03-18T18:02:27.240" Score="1" Body="&lt;p&gt;After looking through the code for the skipgram method, I figured it out: it has to do with the sampling_factor parameter. It defaults to 1e-5, which means that for such a small sampling factor it's almost never going to actually pick a sample unless your dataset &amp;amp; vocabulary size is big enough to actually get some samples. If you have a small vocabulary (like in the toy example), you should set that to something much higher: when I set it to 1.0 (i.e. take a sample as often as possible) I got lots of pairs, even for the toy example.&lt;/p&gt;&#xA;" OwnerUserId="48997" LastEditorUserId="48997" LastEditDate="2018-03-18T18:07:33.957" LastActivityDate="2018-03-18T18:07:33.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29230" PostTypeId="2" ParentId="29180" CreationDate="2018-03-18T19:49:18.023" Score="1" Body="&lt;p&gt;Find connected components of black pixels, then for each connected component, find its bounding box, and keep the one with the largest bounding box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(If you run into issues with noise, preprocessing with &lt;a href=&quot;https://en.wikipedia.org/wiki/Mathematical_morphology&quot; rel=&quot;nofollow noreferrer&quot;&gt;morphological operators&lt;/a&gt; might help.  But only try that if the simple approach above doesn't work.)&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-18T19:49:18.023" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29232" PostTypeId="1" CreationDate="2018-03-18T20:59:22.367" Score="1" ViewCount="38" Body="&lt;p&gt;I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the number of reviews for said hotel. In my data set, some hotels have a rating of 5 for example, but with only one or two reviews. On the other hand, some hotels who are also rated at 5 have over 300 reviews. Is there a way to give priority or more influence to hotels that have been rated more frequently?&#xA;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="49031" LastActivityDate="2022-02-15T09:01:55.207" Title="Recommender system: Give a feature more significance than another" Tags="|bigdata|recommender-system|pyspark|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29233" PostTypeId="2" ParentId="19654" CreationDate="2018-03-18T23:21:09.200" Score="1" Body="&lt;p&gt;There is only one neural network to train in word2vec. CBOW, continuous-bag-of-words', and skip-gram are two different methods of constructing the training data for the neural network. You have to pick &lt;strong&gt;one&lt;/strong&gt; of those two training methods.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-18T23:21:09.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29235" PostTypeId="2" ParentId="29218" CreationDate="2018-03-19T00:24:08.637" Score="1" Body="&lt;p&gt;You should check out the text2vec R package. It has TF-IDF capabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the link to the package info: &#xA;&lt;a href=&quot;https://cran.r-project.org/web/packages/text2vec/text2vec.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;text2vec&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="41460" LastActivityDate="2018-03-19T00:24:08.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29237" PostTypeId="1" AcceptedAnswerId="29277" CreationDate="2018-03-19T00:54:18.157" Score="6" ViewCount="1842" Body="&lt;p&gt;The example below is taken from the lectures in &lt;a href=&quot;http://deeplearning.ai&quot; rel=&quot;nofollow noreferrer&quot;&gt;deeplearning.ai&lt;/a&gt; shows that the result is the sum of the &lt;strong&gt;element-by-element product&lt;/strong&gt; (or &amp;quot;element-wise multiplication&amp;quot;.  The red numbers represent the weights in the filter:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$(1*1)+(1*0)+(1*1)+(0*0)+(1*1)+(1*0)+(0*1)+(0*0)+(1*1) &#xA;= 1+0+1+0+1+0+0+0+1 &#xA;= 4 $&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/MkFSC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MkFSC.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;HOWEVER, most resources say that it's the &lt;em&gt;&lt;strong&gt;dot product&lt;/strong&gt;&lt;/em&gt; that's used:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;…we can re-express the output of the neuron as , where is the bias&#xA;term. In other words, we can compute the output by y=f(x*w) where b is&#xA;the bias term. In other words, we can compute the output by&#xA;performing the dot product of the input and weight vectors, adding in&#xA;the bias term to produce the logit, and then applying the&#xA;transformation function.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Buduma, Nikhil; Locascio, Nicholas. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (p. 8). O'Reilly Media. Kindle Edition.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;We take the 5&lt;em&gt;5&lt;/em&gt;3 filter and slide it over the complete image and&#xA;along the way take the dot product between the filter and chunks of&#xA;the input image.  For every dot product taken, the result is a&#xA;scalar.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad5df8&quot; rel=&quot;nofollow noreferrer&quot;&gt;The best explanation of Convolutional Neural Networks on the Internet!&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;Each neuron receives some inputs, performs a dot product and&#xA;optionally follows it with a non-linearity.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://CS231n%20Convolutional%20Neural%20Networks%20for%20Visual%20Recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n Convolutional Neural Networks for Visual Recognition&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;The result of a convolution is now equivalent to performing one large&#xA;matrix multiply np.dot(W_row, X_col), which evaluates the dot product&#xA;between every filter and every receptive field location.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;http://CS231n%20Convolutional%20Neural%20Networks%20for%20Visual%20Recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n Convolutional Neural Networks for Visual Recognition&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;However, when I research &lt;a href=&quot;https://www.khanacademy.org/math/precalculus/precalc-matrices/properties-of-matrix-multiplication/a/matrix-multiplication-dimensions&quot; rel=&quot;nofollow noreferrer&quot;&gt;how to compute the dot product of matrics&lt;/a&gt;, it seems that the dot product is &lt;strong&gt;not&lt;/strong&gt; the same as summing the element-by-element multiplication. What operation is actually used (element-by-element multiplication or the dot product?) and what is the primary difference?&lt;/p&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="29169" LastEditDate="2021-02-13T16:41:34.060" LastActivityDate="2021-02-13T16:41:34.060" Title="In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?" Tags="|deep-learning|neural-network|cnn|convolutional-neural-network|image-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29238" PostTypeId="1" CreationDate="2018-03-19T01:24:58.580" Score="1" ViewCount="58" Body="&lt;p&gt;I'm trying to build an image classifier where people can take a picture of a tool or part and have the image classified. Much like bixby or amazon's tool to do something similar, but with only 30,000 different items to classify instead of a huge shopping catalog. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was hoping there would be an AI service that takes a lot of the work out as I am trying to avoid training my own CNN because the manpower necessary to take the amount of pictures I would need is not achievable for my project. I can definitely get the manpower to take some pictures, to the tune of maybe 300,000 to 1 million, but I don't believe that would be enough while training a brand-new CNN.&lt;/p&gt;&#xA;" OwnerUserId="49036" LastActivityDate="2018-05-18T09:22:08.183" Title="Is there an AI service that could be used to classify 30,000 different tools and parts?" Tags="|neural-network|classification|image-classification|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29239" PostTypeId="1" CreationDate="2018-03-19T02:53:33.543" Score="1" ViewCount="387" Body="&lt;p&gt;I am trying out isolation forest to detect outliers in a specific target column of my dataset. The dataset contains 188 rows of data with 178 rows with the same value for that target column and the isolation forest gives out every single value -1. Is that a bug or should I take it as that the values are fine? Here is a piece of the code. (I know I need to stop using &lt;code&gt;ix&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.ensemble import IsolationForest&#xA;&#xA;df1 = pd.read_csv('C:/Users/smotapar/Desktop/ase/source/data.csv')&#xA;clf = IsolationForest(n_estimators=200, random_state=10, bootstrap=False)&#xA;clf.fit(df1.ix[:,&quot;target&quot;].values.reshape(-1, 1))&#xA;clf.predict(df1.ix[:,&quot;target&quot;].values.reshape(-1, 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives out an output:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1])&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, what did I do wrong? If I didn't, why is the output like this?&lt;/p&gt;&#xA;" OwnerUserId="40010" LastEditorUserId="24000" LastEditDate="2018-03-19T07:53:54.097" LastActivityDate="2019-03-13T18:22:04.993" Title="Isolation forest results every value -1" Tags="|python|scikit-learn|pandas|outlier|" AnswerCount="0" CommentCount="2" FavoriteCount="0" ClosedDate="2021-08-28T03:35:36.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="29240" PostTypeId="2" ParentId="29237" CreationDate="2018-03-19T03:20:41.450" Score="2" Body="&lt;p&gt;The convolution can be written as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$I(x,y) = \sum_{i=0}^n \sum_{j=0}^m k(i,j) I(x+i, y+j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the kernel $k$ is of size $n$ by $m$. The image is size $N$ by $M$. Ok so in your example we are evaluating $I(0,0)$, our kernel is $n=3$ and $m=3$. Then our image is $N=5$ and $M=5$. If you go through the equation then you will see that you get exactly the result you showed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a dot product as defined as being a sum of products. We multiply element-wise and then we add them all together. The result is the answer you got $4$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The entire convolution operation across the image is a matrix where each entry is this dot product.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-19T03:33:01.023" LastActivityDate="2018-03-19T03:33:01.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29241" PostTypeId="1" CreationDate="2018-03-19T04:26:08.343" Score="2" ViewCount="410" Body="&lt;p&gt;In a convolutional neural network (CNN), since the RGB values get multiplied in the first convolutional layer, does this mean that color is essentially only extracted in the very first layer?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Snippets from &lt;a href=&quot;http://cs231n.github.io/understanding-cnn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS231n Chapter on CNN&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[...] One dangerous pitfall that can be easily noticed with this&#xA;  visualization is that some activation maps may be all zero for many&#xA;  different inputs, which can indicate dead filters, and can be a&#xA;  symptom of high learning rates [...] Typical-looking activations on&#xA;  the first CONV layer (left), and the 5th CONV layer (right) of a&#xA;  trained AlexNet looking at a picture of a cat. Every box shows an&#xA;  activation map corresponding to some filter. Notice that the&#xA;  activations are sparse (most values are zero, in this visualization&#xA;  shown in black) and mostly local.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="28175" LastEditDate="2018-03-19T05:07:53.357" LastActivityDate="2019-12-11T18:01:45.227" Title="Is color information only extracted in the first input layer of a convolutional neural network?" Tags="|machine-learning|neural-network|deep-learning|cnn|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29242" PostTypeId="1" AcceptedAnswerId="29244" CreationDate="2018-03-19T04:58:04.433" Score="0" ViewCount="515" Body="&lt;p&gt;I'm reading the &lt;a href=&quot;http://ect.bell-labs.com/who/tkh/publications/papers/odt.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;random forest paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;binary decision trees use a single feature at each nonterminal node. A test point is assigned to the left or right branch by its value of that feature. Geometrically this corresponds to assigning the point to one side of the hyperplane that is parallel to one axis of the feature space.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone explain why the hyperplane is parallel to the axis? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say the feature is represented by the $x$ axis, and that the decision is $x &amp;lt; 5$ vs $x &amp;gt;5$. To me it seems like the hyperplane should be vertical to the $x$ axis so that $x = 4$ is on one side and $x=6$ is on the other. What's wrong with my understanding? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="44295" LastEditorUserId="38892" LastEditDate="2018-03-19T07:53:39.740" LastActivityDate="2018-03-19T07:53:39.740" Title="binary decision tree hyperplane parallel to axis?" Tags="|machine-learning|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29243" PostTypeId="2" ParentId="29241" CreationDate="2018-03-19T05:07:39.417" Score="1" Body="&lt;p&gt;The quote that you have brought is speaking about something else. Zero means that most of your linear part of neurons have a negative value and after passing through the &lt;code&gt;ReLU&lt;/code&gt; non-linearity, they become zero. That's the explanation of that part. About your question, not really. I don't know if you've ever visualized a convolutional network or not but you've done that you may have seen that there may be layers that get activated for special colors even in deep layers. The reason is that the convolutional layer just tries to do cross-correlation operation on the input which has three channels. Next layers of convolution also do this considering e.g. the input has 64 channels. To wrap up, there may be other deep-layers that use the information of colors, Although these operations really do not care about the colors, they care about the local patterns that may happen in the input. The reason for learning colors is that similar colors have similar patterns in their RGB channels. These patterns are what convolutional networks try to learn. Take a look at the tool box that have been discussed &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-19T05:07:39.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29244" PostTypeId="2" ParentId="29242" CreationDate="2018-03-19T05:44:07.450" Score="0" Body="&lt;p&gt;Your understanding is fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hyperplane is $x=5$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In $3$ dimension, it is a plane that is parallel to the the $y$-$z$ plane, which can be written as $x=0$. It is parallel to the $y$-axis and $z$-axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are right that it is perpendicular to the $x$-axis. The $x$-axis is the normal direction for the plane $x=5$.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2018-03-19T06:28:40.957" LastActivityDate="2018-03-19T06:28:40.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29245" PostTypeId="1" AcceptedAnswerId="29247" CreationDate="2018-03-19T05:44:33.477" Score="8" ViewCount="12937" Body="&lt;p&gt;In the paper ImageNet Classification with Deep Convolutional Neural Networks, the size of input image is 224x224. The following figure shows the input size.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/b51OM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/b51OM.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;From caffe, deploy.prototxt file from the directory of the bvlc_alexnet says that the input size is 227x227.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/DP5wl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DP5wl.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Why is the input size different?&lt;/p&gt;&#xA;" OwnerUserId="46242" LastEditorUserId="29169" LastEditDate="2021-02-13T16:36:19.317" LastActivityDate="2021-06-23T16:58:50.160" Title="What is the input size of Alex net" Tags="|machine-learning|deep-learning|neural-network|cnn|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29246" PostTypeId="1" CreationDate="2018-03-19T05:59:08.663" Score="2" ViewCount="3459" Body="&lt;p&gt;How to use k-fold cross validation for MNIST dataset?&#xA;I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import cross_val_score&#xA;clf = svm.SVC(kernel='linear', C=1)&#xA;scores = cross_val_score(clf, iris.data, iris.target, cv=5)&#xA;scores                                              &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for example while importing mnist dataset in keras &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.datasets import mnist&#xA;(Xtrain,Ytrain),(Xtest,Ytest)=mnist_load()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in this dataset is already divided in test and train , so to apply cross validation on the entire dataset do we need to make Xtrain and Xtest as one entity to exploit the whole data.&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2022-01-13T22:06:00.587" Title="how to prepare data for cross validation in mnist dataset?" Tags="|scikit-learn|keras|cross-validation|mnist|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="29247" PostTypeId="2" ParentId="29245" CreationDate="2018-03-19T06:01:58.330" Score="13" Body="&lt;p&gt;I guess it has been a mistake. Take a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/MmYe2/classic-networks&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The other author's were Ilya Sutskever and Geoffrey Hinton. So, AlexNet input starts with 227 by 227 by 3 images. And if you read the paper, the paper refers to 224 by 224 by 3 images. But if you look at the numbers, I think that the numbers make sense only of actually 227 by 227. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-19T06:01:58.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29248" PostTypeId="2" ParentId="29246" CreationDate="2018-03-19T06:34:22.880" Score="0" Body="&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import cross_val_score&#xA;clf = svm.SVC(kernel='linear', C=1)&#xA;scores = cross_val_score(clf, iris.data, iris.target, cv=5)&#xA;scores&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;They are not using the whole data for cross validation as such ( it's just an illusion)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the &lt;code&gt;cv&lt;/code&gt; argument is an integer, cross_val_score uses the &lt;code&gt;KFold&lt;/code&gt; or &lt;code&gt;StratifiedKFold&lt;/code&gt; strategies by default, the latter being used if the estimator derives from &lt;code&gt;ClassifierMixin&lt;/code&gt;..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's kind of automated inside the call..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check this kaggle kernel &lt;a href=&quot;https://www.kaggle.com/dansbecker/cross-validation&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-19T06:37:56.737" LastActivityDate="2018-03-19T06:37:56.737" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="29249" PostTypeId="1" CreationDate="2018-03-19T07:02:59.750" Score="5" ViewCount="730" Body="&lt;p&gt;Please help answer this question or point me to any resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a model in an environment where training happens with new data and the data is discarded after training is completed.&#xA;This keeps on happening in cycles. Hence we dealing with models allowing &quot;partial fit&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this model, there is a training variable X , which is a categorical variable.&#xA;X has some previously known categories C and its encoded as &quot;one hot vector&quot; of length D .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now suppose we are now observing a new category of X, so we need to extend encoding of C to length D+1 .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is, how to do this without losing all the previously learned knowledge by model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training again is not an option since we have discarded all the previously held data and only new data can be used for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is already a question &lt;a href=&quot;https://datascience.stackexchange.com/questions/19051/updating-one-hot-encoding-to-account-for-new-categories&quot;&gt;Updating One-Hot Encoding to account for new categories&lt;/a&gt;&#xA;However the answer is either to use entire data to create the &quot;one hot encoding&quot; (which is not possible in this environment since new data is in future) , or to omit the usage entirely (thus loosing some information). &lt;/p&gt;&#xA;" OwnerUserId="49049" LastEditorUserId="49049" LastEditDate="2018-03-19T07:45:31.617" LastActivityDate="2019-05-01T17:33:53.903" Title="How to continue incremental learning when a categorical variable has been assigned additional category labels?" Tags="|machine-learning|categorical-data|online-learning|machine-learning-model|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29250" PostTypeId="2" ParentId="29223" CreationDate="2018-03-19T07:28:12.013" Score="0" Body="&lt;p&gt;Idea from Lev Manovich, shown on &lt;a href=&quot;https://youtube.com/watch?v=GIVK0-SNUgU&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; shows how most effectively plotting picture data to chart is made by showing the images as the dots in the chart.&lt;/p&gt;&#xA;&lt;p&gt;When something interesting behavior is on one point, you can observe the image right away.&lt;/p&gt;&#xA;&lt;p&gt;There was an example about plotting time series data on x-y space and from the form of daily curve there was a anomality in one day. Directly zooming on the pictures on that revealed a fire taken place in the landscape. From average value black and white plots you would not get that kind of immediate observations.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="98307" LastEditDate="2020-08-20T17:06:06.903" LastActivityDate="2020-08-20T17:06:06.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29251" PostTypeId="1" CreationDate="2018-03-19T07:44:33.263" Score="0" ViewCount="80" Body="&lt;p&gt;I've got a &lt;a href=&quot;https://drive.google.com/file/d/1piM5Em09b-uSHFq245sRpPTnSt0k3xhV/view?usp=sharing&quot; rel=&quot;nofollow noreferrer&quot;&gt;Excel file&lt;/a&gt; having columns &lt;strong&gt;Name, Surname&lt;/strong&gt; and &lt;strong&gt;Community&lt;/strong&gt;, having around 31 thousand entries. For my college project, the task I have is to write a program to predict &quot;Community&quot; of a person, given their name and surname. For example, the first entry in the file is the name: &lt;em&gt;Yogita Singh&lt;/em&gt;. If someone enters a name like &lt;em&gt;Yogta Singha&lt;/em&gt;, then approximately it should predict that that person belongs to the &lt;em&gt;Punjabi&lt;/em&gt; communtiy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did manage to write a program, which does the prediction using the &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance&lt;/a&gt;. I gave more weightage to the surname (10 times more), than the first name, because surname and community has a greater correlation than first-name and community (naturally, because some first-names are very common and used by people from many different communities).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to improve the accuracy of my model? Has there been any relevant theoretical study regarding name - ethnicity mapping/correlation models? I think Levenshtein  distance is a good start but I'm not sure the exact weights I should give to the surname and first name. Also, perhaps I need to give more weight to the characters towards the beginning of the surname, rather than characters towards the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions regarding how to improve the accuracy of my model will be greatly appreciated. Thanks.&lt;/p&gt;&#xA;" OwnerDisplayName="user49048" LastActivityDate="2018-03-19T07:52:20.450" Title="How to improve accuracy of ethnicity prediction using name?" Tags="|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29252" PostTypeId="2" ParentId="29251" CreationDate="2018-03-19T07:52:20.450" Score="1" Body="&lt;h1&gt;Feature engineering&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The question you want to ask yourself is &quot;When you look at a name how do you as a human deduce the community?&quot; I would claim that this is possible by looking at the way the letters are grouped together. For example &quot;aoui&quot; and &quot;wi&quot; is very common in India and Arabic, whereas &quot;Zh&quot; would be very common in Chinese. So how can we extract these types of features?&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;n-grams&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;n-grams&lt;/strong&gt; is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The segmentation works as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The String: Hello World&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2-gram: &quot;He&quot;, &quot;el&quot;, &quot;ll&quot;, &quot;lo&quot;, &quot;o &quot;, &quot; W&quot;, &quot;Wo&quot;, &quot;or&quot;, &quot;rl&quot;, &quot;ld&quot; &lt;br/&gt;&#xA;3-gram: &quot;Hel&quot;, &quot;ell&quot;, &quot;llo&quot;, &quot;lo &quot;, &quot;o W&quot;, &quot; Wo&quot;, &quot;Wor&quot;, &quot;orl&quot;, &quot;rld&quot;&lt;br/&gt;&#xA;4-gram: &quot;Hell&quot;, &quot;ello&quot;, &quot;llo &quot;, &quot;lo W&quot;, &quot;o Wo&quot;, &quot; Wor&quot;, &quot;Worl&quot;, &quot;orld&quot;&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would capture the distribution of letter combinations which are present for each &lt;strong&gt;community&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-19T07:52:20.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29253" PostTypeId="2" ParentId="29238" CreationDate="2018-03-19T09:03:04.427" Score="1" Body="&lt;p&gt;Use pre-trained models and transfer learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;take &lt;strong&gt;keras&lt;/strong&gt; as example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://keras.io/applications/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/applications/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;you just need prepare dozens of pictures for each category and can have a pretty good performance.&lt;/p&gt;&#xA;" OwnerUserId="43732" LastActivityDate="2018-03-19T09:03:04.427" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29254" PostTypeId="1" CreationDate="2018-03-19T09:32:31.363" Score="1" ViewCount="91" Body="&lt;p&gt;I am working on 3D object detection in the context of self driving cars and I was wondering if there are other challenging benchmarks for this beside &lt;a href=&quot;http://www.cvlibs.net/datasets/kitti/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kitti&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="11097" LastEditDate="2018-03-19T12:13:32.790" LastActivityDate="2018-03-19T14:12:06.723" Title="What are the challenging benchmarks for self driving cars object detection beside Kitti" Tags="|machine-learning|computer-vision|self-driving|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29255" PostTypeId="1" CreationDate="2018-03-19T09:52:15.473" Score="1" ViewCount="178" Body="&lt;p&gt;I was going through a Kaggle kernel, where train data was of different data types, so it consisted of separating out features for different types like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_cat = Input((len(a), ))&#xA;input_num = Input((len(num_feat), ))&#xA;input_words = Input((maxlen, ))&#xA;&#xA;#More layers are added to the inputs&#xA;#x_cat, x_num, x_words are updated vars for the 3 inputs&#xA;&#xA;x = concatenate([x_cat, x_num, x_words])&#xA;#some final layers added&#xA;predictions = Dense(1, activation=&quot;sigmoid&quot;)(x)&#xA;&#xA;model = Model(inputs=[input_cat, input_num, input_words], outputs=predictions)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Later the model is used to fit training data like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit([X_train_cat, X_train_num, X_train_words], X_train_targe, #other params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get how the separated feature files are trained separately, and then their weights are concatenated, which are then used to pull out predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just wanted to get a bit intuition behind this or is it as simple as processing different types of feature differently and then combining them for a better model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks! &lt;/p&gt;&#xA;" OwnerUserId="20766" LastEditorUserId="11097" LastEditDate="2018-03-19T11:14:59.923" LastActivityDate="2018-03-19T11:14:59.923" Title="How does multiple feature files work in Keras" Tags="|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29256" PostTypeId="1" CreationDate="2018-03-19T11:16:20.557" Score="1" ViewCount="208" Body="&lt;p&gt;When I write a network, do I have to write the whole forward property in &lt;code&gt;nn.Module.forward()&lt;/code&gt;? I mean if I do some operations outside the net, does grad correctly flow?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have two networks, in which the output of one is the input of the other(&lt;code&gt;net1 -&amp;gt; midresults -&amp;gt; net2&lt;/code&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I do some operations on midresults(&lt;code&gt;net1 -&amp;gt; midresults -&amp;gt; operations -&amp;gt; net2&lt;/code&gt;), can (net1+net2) be trained end to end?&lt;/p&gt;&#xA;" OwnerUserId="49064" LastEditorUserId="50727" LastEditDate="2020-07-13T11:45:00.373" LastActivityDate="2023-09-05T09:02:05.193" Title="Forward and backward process in pyTorch" Tags="|deep-learning|pytorch|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29257" PostTypeId="1" AcceptedAnswerId="29261" CreationDate="2018-03-19T11:19:44.287" Score="3" ViewCount="118" Body="&lt;p&gt;Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of up to 32 objects at once. If the sensor can't find 32 objects (which is always the case) it sets the x and y coordinates of the objects not found to zero. Could there be a problem training a neural network on this kind of data? The sensor is set on a car, the objects are other cars and the networks job is to predict the next move.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem is that when the sensor finds a new object, already existing/found object might change id. Any tips on that? I am thinking about making randomly permute the indices of the object since that should not make any difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any standard solutions to these kind of problems? Especially Setting the distance of non existing objects to zero.&lt;/p&gt;&#xA;" OwnerUserId="49061" LastEditorUserId="49061" LastEditDate="2018-03-19T13:35:35.070" LastActivityDate="2018-03-20T12:11:12.000" Title="Setting &quot;missing&quot; distance values to zero when training a neural network" Tags="|machine-learning|neural-network|deep-learning|missing-data|rnn|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29258" PostTypeId="2" ParentId="6186" CreationDate="2018-03-19T11:56:48.773" Score="4" Body="&lt;p&gt;Although the definitions of the two concepts are fuzzy, there is a slight difference between online learning and incremental learning approaches. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In online learning approach, model is updated to adapt to the new data. It is possible that the model can forget the previously learned inferences which is called as &lt;a href=&quot;https://en.wikipedia.org/wiki/Catastrophic_interference&quot; rel=&quot;nofollow noreferrer&quot;&gt;Catastrophic Interference&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whereas in the incremental approach, even as the model is updated, previous inferences are not forgotten.&lt;/p&gt;&#xA;" OwnerUserId="49068" LastActivityDate="2018-03-19T11:56:48.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29259" PostTypeId="2" ParentId="20291" CreationDate="2018-03-19T12:40:23.210" Score="1" Body="&lt;p&gt;Distributed Memory model preserves the word order in a document whereas Distributed Bag of words just uses the bag of words approach, which doesn't preserve any word order.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has been explained in details in this &lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;research paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49072" LastEditorUserId="29575" LastEditDate="2018-03-19T13:56:11.517" LastActivityDate="2018-03-19T13:56:11.517" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29260" PostTypeId="2" ParentId="14843" CreationDate="2018-03-19T13:33:19.057" Score="0" Body="&lt;p&gt;The first paragraph in &lt;a href=&quot;http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt;, explains it pretty well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also the time complexity will not grow linearly if you increase the size of your network. It will grow squared and so will the number of parameters. &lt;/p&gt;&#xA;" OwnerUserId="49061" LastEditorUserId="29575" LastEditDate="2018-03-19T13:55:26.293" LastActivityDate="2018-03-19T13:55:26.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29261" PostTypeId="2" ParentId="29257" CreationDate="2018-03-19T13:47:57.667" Score="1" Body="&lt;p&gt;The short answer is &quot;it depends&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is zero better than one or infinity?  It depends on the range of x-y coordinates that you use.  It also depends on your output.  If you go from (x,y) to something like z = x^2, then you're placing your &quot;null&quot; at a local minimum. So, if it's a car, I'm assuming you're using a forward-facing camera.  Do you really want your algorithm to see a bunch of &quot;cars&quot; bunched up at the origin?&#xA;By the way, where is your origin?  Upper left (like most image indexing)? Lower left (like a cartesian plane)? Center (if so, what direction is positive x and y?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for re indexing, what you can do is look at the previous frame (or frames) where your 32 objects are indexed, along with their position and velocity.  Then, look at the current frame and compare each object's position to the positions in the previous frame.  Assign the objects in the current frame the same id as the object nearest it in the previous frame.  You can break ties with velocity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;following up on your comments:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a neural network can learn any arbitrary function to any arbitrary precision.  Eventually.  If you want certain behavior at (0,0) that is dramatically different from behavior at (0+ε,0+ε), then your network will take a long time to converge.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest seeing what kind of results you get by converting nulls to zero and then comparing it to other techniques.  Maybe replace null with the average of all the other cars, or set it immediately behind your ego vehicle (since cars primarily move forward, your network will probably put less importance on cars behind the ego car).&lt;/p&gt;&#xA;" OwnerUserId="21232" LastEditorUserId="21232" LastEditDate="2018-03-20T12:11:12.000" LastActivityDate="2018-03-20T12:11:12.000" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29262" PostTypeId="2" ParentId="29127" CreationDate="2018-03-19T14:05:25.617" Score="0" Body="&lt;p&gt;I don't necessarily think it's that batch normalization is necessarily stochastic, but rather just how batch normalization works when compared to the inference stage.  As you may know, during the training phase batch normalization depends on the mini-batch.  However, that dependency is undesirable for inference so instead a moving average over all mini-batches is taken.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing this obviously can cause problems since what your model is inference on is different than what it was trained on. Furthermore, the smaller your mini-batch the worse the performance gets with more depth since the the inaccuracies computed for the smaller batch just get compounded more and more.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you insist on having a batch normalization phase, a way to help mitigate this might include implementing a batch re-normalization setup instead.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-19T14:05:25.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29263" PostTypeId="2" ParentId="29254" CreationDate="2018-03-19T14:12:06.723" Score="1" Body="&lt;p&gt;You may find &lt;a href=&quot;http://apolloscape.auto/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apolloscape dataset&lt;/a&gt; interesting. It was recently released by Baidu. From the &lt;a href=&quot;https://arxiv.org/abs/1803.06184&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; describing the dataset:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this paper, we present a large-scale open dataset,&#xA;  ApolloScape, that&#xA;  consists of RGB videos and corresponding dense 3D point&#xA;  clouds.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Apart from the &lt;a href=&quot;https://arxiv.org/abs/1803.06184&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;,&#xA;there have been &lt;a href=&quot;https://medium.com/@Synced/baidu-apollo-releases-massive-self-driving-dataset-teams-up-with-berkeley-deepdrive-5e785ab4053b&quot; rel=&quot;nofollow noreferrer&quot;&gt;some blog posts&lt;/a&gt; with summarized information on the dataset.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-19T14:12:06.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29264" PostTypeId="1" AcceptedAnswerId="29270" CreationDate="2018-03-19T14:50:38.033" Score="0" ViewCount="249" Body="&lt;p&gt;I am classifying documents I have around 4000 of them that I am trying to categorise into 5 categories. I am using a bag of words model which equates to about 18,000 unique words (features) and therefore I have an input layer of a neural network with 18,0000 inputs which doesn't seem right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is taking a huge amount of memory to try and train this network and so much time it will never converge!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of reducing the number of input neurons seeing as a large portion of this data will be nulls?&lt;/p&gt;&#xA;" OwnerUserId="47869" LastActivityDate="2018-03-19T16:05:16.453" Title="Text classification with neural network number of input neurons" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29265" PostTypeId="2" ParentId="29122" CreationDate="2018-03-19T14:51:36.263" Score="1" Body="&lt;p&gt;Needed to normalise the data which encodes the column and then denormalise on the output which gives the correct classification.&lt;/p&gt;&#xA;" OwnerUserId="47869" LastActivityDate="2018-03-19T14:51:36.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29266" PostTypeId="1" CreationDate="2018-03-19T15:01:10.050" Score="4" ViewCount="1125" Body="&lt;p&gt;Any Thoughts on improving the Model. So far i was able to achieve around accuracy 0.20 on each task specific dense network of a Multi task Learning Architecture. I have posted model  and validation Accuracy &lt;a href=&quot;https://i.stack.imgur.com/sxBRG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Accuracy&lt;/a&gt; and Model and validation loss &lt;a href=&quot;https://i.stack.imgur.com/hLOA6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Loss&lt;/a&gt;.Currently from graph, i can see no learning is happening. Below is complete description of the problem.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Problem Description&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am working on a classification problem. The dataset was collected from the competition hosted by kaggle, &lt;strong&gt;Painters by number&lt;/strong&gt;. The task is to identify painter,style and genre given paintings.So far, I trained individual models to predict painter,style,genre given paintings. Now i would like to incorporate Multi task learning (i.e) developing a single model which can predict all three tasks. I came up with the  architecture specified by&#xA;&lt;a href=&quot;https://datascience.stackexchange.com/questions/28932/is-it-possible-to-pass-multiple-data-generators-for-multiple-inputs-to-model-fit&quot;&gt;Multi Task Learning Architecture&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; input_layer   = Input(shape=(64,64,3))&#xA;&#xA; x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(input_layer)&#xA; x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)&#xA; x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)&#xA;&#xA; # Block 2&#xA; x = Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv1')(x)&#xA; x = Conv2D(128, (3, 3), activation='relu', padding='same',   name='block2_conv2')(x)&#xA; x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool')(x)&#xA;&#xA; # Block 3&#xA; x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv1')(x)&#xA; x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2')(x)&#xA; x = Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv3')(x)&#xA; x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool')(x)&#xA;&#xA; # Block 4&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv1')(x)&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2')(x)&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv3')(x)&#xA; x = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool')(x)&#xA;&#xA; # Block 5&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv1')(x)&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2')(x)&#xA; x = Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv3')(x)&#xA; x = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool')(x)&#xA;&#xA; x = Flatten()(x)&#xA; x = Dense(100)(x)&#xA;&#xA; out_style_1 = Dense(&#xA;         100,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero',&#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation='relu'&#xA;     )(x)&#xA;&#xA; out_genre_1 = Dense(&#xA;         100,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero',&#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation='relu'&#xA;     )(x)&#xA;&#xA; out_painter_1 = Dense(&#xA;         100,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero',&#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation='relu'&#xA;     )(x)&#xA;&#xA; out_style_2 = Dense(&#xA;         19,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero',&#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation = 'softmax',&#xA;     )(out_style_1)&#xA;&#xA;  out_genre_2 = Dense(&#xA;         32,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero', &#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation = 'softmax',&#xA;     )(out_genre_1)&#xA;&#xA; out_painter_2 = Dense(&#xA;         10,&#xA;         kernel_initializer=glorot_normal(seed=seed_val),&#xA;         bias_initializer = 'zero',&#xA;         kernel_regularizer = l2(l=0.0001),&#xA;         activation = 'softmax',&#xA;     )(out_painter_1)&#xA;&#xA; multi_tasking_model = Model(inputs=[input_layer],outputs=[out_style_2,out_genre_2,out_painter_2])&#xA; multi_tasking_model.summary()&#xA;&#xA;&#xA; multi_tasking_model.compile(&#xA;   loss = 'categorical_crossentropy',&#xA;   optimizer=Adam(lr=0.0001, beta_1=0.9, beta_2=0.999,   epsilon=0.00000001),&#xA; metrics=['accuracy']&#xA;  )&#xA;&#xA;&#xA; history = multi_tasking_model.fit_generator(&#xA;    generator = train_generator,&#xA;    steps_per_epoch= 2920//50,&#xA;    epochs = 100,&#xA;    validation_data = valid_generator,&#xA;    validation_steps = 690//50&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Data Provider&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Using keras data provider for this task was not helpful. So I have created a custom data provider. Reference(&lt;a href=&quot;https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford data provider&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;class DataGenerator(object):&#xA;&#xA;  def __init__(self,dim_x = 64,dim_y=32,dim_z=32,batch_size=32,&#xA;shuffle=True,data=None,style=None,genre=None,painter=None):&#xA;&#xA;    self.dim_x = dim_x&#xA;    self.dim_y = dim_y&#xA;    self.dim_z = dim_z&#xA;    self.batch_size = batch_size&#xA;    self.shuffle=shuffle&#xA;    self.data = data&#xA;    self.y_style = style&#xA;    self.y_genre = genre&#xA;    self.y_painter = painter&#xA;&#xA; @property&#xA; def data(self):&#xA;    return self._data&#xA;&#xA; @data.setter&#xA; def data(self,value):&#xA;    self._data = value&#xA;&#xA; @property&#xA; def y_style(self):&#xA;    return self._y_style&#xA;&#xA; @y_style.setter&#xA; def y_style(self,value):&#xA;    self._y_style=value&#xA;&#xA; @property&#xA; def y_genre(self):&#xA;    return self._y_genre&#xA;&#xA; @y_genre.setter&#xA; def y_genre(self,value):&#xA;    self._y_genre = value&#xA;&#xA; @property&#xA; def y_painter(self):&#xA;    return self._y_painter&#xA;&#xA; @y_painter.setter&#xA; def y_painter(self,value):&#xA;    self._y_painter = value&#xA;&#xA; def __get_exploration_order(self,len_list_ids):&#xA;    indexes = np.arange(len_list_ids)&#xA;    if self.shuffle:&#xA;        np.random.shuffle(indexes)&#xA;    return indexes&#xA;&#xA; def __data_generation(self,list_ids_temp):&#xA;    X = np.empty((self.batch_size,self.dim_x,self.dim_y,self.dim_z))&#xA;    y_style = np.empty((self.batch_size), dtype=int)&#xA;    y_genre = np.empty((self.batch_size), dtype=int)&#xA;    y_painter = np.empty((self.batch_size), dtype=int)&#xA;&#xA;    for i, ID in enumerate(list_ids_temp):&#xA;        X[i,:,:,:] = self.data[ID]&#xA;&#xA;        y_style[i] = self.y_style[ID]&#xA;        y_genre[i] = self.y_genre[ID]&#xA;        y_painter[i] = self.y_painter[ID]&#xA;&#xA;    return X,[y_style,y_genre,y_painter]&#xA;&#xA;&#xA; def generate(self,len_list_ids):&#xA;    while 1:&#xA;        indexes = self.__get_exploration_order(len_list_ids)&#xA;&#xA;        imax = int(len(indexes)/self.batch_size)&#xA;        for i in range(imax):&#xA;            list_ids_temp = [ k for k in indexes[i*self.batch_size:(i+1)*self.batch_size]]&#xA;&#xA;            X,y = self.__data_generation(list_ids_temp)&#xA;&#xA;            yield X,y&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Data Provider Initialization&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  from DataGenerator import DataGenerator&#xA;&#xA;  params = {&#xA;  'dim_x': 224,&#xA;  'dim_y': 224,&#xA;  'dim_z': 3,&#xA;  'batch_size':10,&#xA;  'shuffle':True&#xA; }&#xA;&#xA; data_train_gen = DataGenerator(**params)&#xA; data_train_gen.data = np.load('data.npy')&#xA; data_train_gen.y_style = np.load('y_style.npy')&#xA; data_train_gen.y_genre = np.load('y_genre.npy')&#xA; data_train_gen.y_painter = np.load('y_painter.npy')&#xA; train_generator=data_train_gen.generate(data_train_gen.data.shape[0])&#xA;&#xA;&#xA; data_gen_valid = DataGenerator(**params)&#xA; data_gen_valid.data = np.load('data_valid.npy')&#xA; data_gen_valid.y_style = np.load('y_style_valid.npy')&#xA; data_gen_valid.y_genre = np.load('y_genre_valid.npy')&#xA; data_gen_valid.y_painter = np.load('y_painter_valid.npy')&#xA; valid_generator= data_gen_valid.generate(data_gen_valid.data.shape[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Model Summary&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Layer (type)                    Output Shape         Param #     Connected to&#xA;   input_4 (InputLayer)         (None, 64, 64, 3)         0&#xA;   block1_conv1 (Conv2D)           (None, 64, 64, 64)   1792        input_3[0][0]&#xA;   block1_conv2 (Conv2D)           (None, 64, 64, 64)   36928       block1_conv1[0][0]&#xA;   block1_pool (MaxPooling2D)      (None, 32, 32, 64)   0           block1_conv2[0][0]&#xA;   block2_conv1 (Conv2D)           (None, 32, 32, 128)  73856       block1_pool[0][0]&#xA;   block2_conv2 (Conv2D)           (None, 32, 32, 128)  147584      block2_conv1[0][0]&#xA;   block2_pool (MaxPooling2D)      (None, 16, 16, 128)  0           block2_conv2[0][0]&#xA;   block3_conv1 (Conv2D)           (None, 16, 16, 256)  295168      block2_pool[0][0]&#xA;   block3_conv2 (Conv2D)           (None, 16, 16, 256)  590080      block3_conv1[0][0]&#xA;   block3_conv3 (Conv2D)           (None, 16, 16, 256)  590080      block3_conv2[0][0]&#xA;   block3_pool (MaxPooling2D)      (None, 8, 8, 256)    0           block3_conv3[0][0]&#xA;   block4_conv1 (Conv2D)           (None, 8, 8, 512)    1180160     block3_pool[0][0]&#xA;   block4_conv2 (Conv2D)           (None, 8, 8, 512)    2359808     block4_conv1[0][0]&#xA;   block4_conv3 (Conv2D)           (None, 8, 8, 512)    2359808     block4_conv2[0][0]&#xA;   block4_pool (MaxPooling2D)      (None, 4, 4, 512)    0           block4_conv3[0][0]&#xA;   block5_conv1 (Conv2D)           (None, 4, 4, 512)    2359808     block4_pool[0][0]&#xA;   block5_conv2 (Conv2D)           (None, 4, 4, 512)    2359808     block5_conv1[0][0]&#xA;   block5_conv3 (Conv2D)           (None, 4, 4, 512)    2359808     block5_conv2[0][0]&#xA;   block5_pool (MaxPooling2D)      (None, 2, 2, 512)    0           block5_conv3[0][0]&#xA;   flatten_3 (Flatten)             (None, 2048)         0           block5_pool[0][0]&#xA;   dense_15 (Dense)                (None, 100)          204900      flatten_3[0][0]&#xA;   dense_16 (Dense)                (None, 100)          10100       dense_15[0][0]&#xA;   dense_17 (Dense)                (None, 100)          10100       dense_15[0][0]&#xA;   dense_18 (Dense)                (None, 100)          10100       dense_15[0][0]&#xA;   dense_19 (Dense)                (None, 19)           1919        dense_16[0][0]&#xA;   dense_20 (Dense)                (None, 32)           3232        dense_17[0][0]&#xA;   dense_21 (Dense)                (None, 10)           1010        dense_18[0][0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Data Description&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Data(train)                shape&#xA;   --------------            --------&#xA;   data_train.npy            (2920,224,244,3)&#xA;   y_style_train.npy         (2920, 19)&#xA;   y_genre_train.npy         (2920, 32)&#xA;   y_painter_train.npy       (2920,10)&#xA;&#xA;   Data(valid)                shape&#xA;   --------------            --------&#xA;   data_valid.npy            (690,224,244,3)&#xA;   y_style_valid.npy         (690, 19)&#xA;   y_genre_valid.npy         (690, 32)&#xA;   y_painter_valid.npy       (690,10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I believe i have posted all the information that is available to me.Any tips would be greatly helpful&lt;/p&gt;&#xA;" OwnerUserId="2893" LastEditorUserId="2893" LastEditDate="2018-03-19T23:10:27.370" LastActivityDate="2018-09-02T20:01:44.063" Title="Thoughts on improving the Multitask Learning Model" Tags="|keras|multitask-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29267" PostTypeId="2" ParentId="28825" CreationDate="2018-03-19T15:41:52.873" Score="1" Body="&lt;p&gt;&lt;strong&gt;Solution:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create a list by using those index number like. Suppose I need 10% data's index number. At first I collect 0 (where i=0) number cluster's index numbers then 1(where i=1) number cluster index numbers&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x = 0.10&#xA; i=0&#xA; C_i = np.where(labels == i)[0].tolist() &#xA; n_i = len(C_i)&#xA; #indices of the points from X to be sampled from cluster i&#xA; sample_i = np.random.choice(C_i, int(x * n_i))&#xA; print (i, sample_i)&#xA; list1=(sample_i)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;x = 0.10&#xA; i=1&#xA; C_i = np.where(labels == i)[0].tolist() &#xA; n_i = len(C_i)&#xA; # indices of the points from X to be sampled from cluster i&#xA; sample_i = np.random.choice(C_i, int(x * n_i))&#xA; print (i, sample_i)&#xA; list2=(sample_i)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fD11G.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fD11G.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After found two list for 2 cluster I merge those two list into 1 list&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;new_list = np.concatenate((list1,list2))&#xA;new_train_data=train.loc[new_list]&#xA;new_train_data.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0JAYS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0JAYS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="43455" LastEditDate="2018-03-19T15:55:10.777" LastActivityDate="2018-03-19T15:55:10.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29269" PostTypeId="1" CreationDate="2018-03-19T16:00:09.523" Score="5" ViewCount="1892" Body="&lt;p&gt;I have a dataset where I have the &lt;code&gt;Outcome&lt;/code&gt; of a case and the &lt;code&gt;Country&lt;/code&gt; under review, as well as whether a judge (among a panel of judges) reviewing the case was also from the country under review, &lt;code&gt;Country_Judge&lt;/code&gt;, categorised as &lt;code&gt;TRUE&lt;/code&gt; or &lt;code&gt;FALSE&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I measure the relationship between the &lt;code&gt;Country_Judge&lt;/code&gt; and the &lt;code&gt;Outcome&lt;/code&gt; of a case? I want to know whether a judge's nationality has an impact on the outcome of the cases. &lt;/p&gt;&#xA;" OwnerUserId="49088" LastActivityDate="2018-04-20T21:17:45.040" Title="How to find the relationship between categorical variables?" Tags="|r|statistics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29270" PostTypeId="2" ParentId="29264" CreationDate="2018-03-19T16:05:16.453" Score="1" Body="&lt;p&gt;Yes, actually what usually people do is to map the unique tokens in a space with fixed dimensionality, obtaining what is called &quot;words embeddings&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And actually, using already trained word embeddings like &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GloVE&lt;/a&gt; is usually it's a best practice: those vectors are trained on huge datasets like Wikipedia or Common Crawl. A nicety of those vectors is that, thanks to the way they are built, they include also the relation between the words, a sort of semantic.&#xA;This is the way I definitely suggest you to start.&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-03-19T16:05:16.453" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29271" PostTypeId="2" ParentId="6115" CreationDate="2018-03-19T16:16:44.813" Score="0" Body="&lt;p&gt;There were many suggested solutions on stackoverflow that may have worked but not anymore in the newest versions of wordcloud and tm.&#xA;The root cause is that the text data contain emoji characters which are encoded in surrogates that are messy.&#xA;For more detailed explanation, please read this &lt;a href=&quot;http://www.textasdata.com/2015/02/encoding-headaches-emoticons-and-rs-handling-of-utf-816/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; that contains the solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The conversion that works in this case is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words &amp;lt;- iconv(words, &quot;ASCII&quot;, &quot;UTF-8&quot;, sub=&quot;byte&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's what failed:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to convert the text BEFORE and AFTER creating the corpus with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words.corpus &amp;lt;- Corpus(VectorSource(words))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;BEFORE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Converting to UTF-8 on the text didn't work with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words &amp;lt;- sapply(words, function(x) iconv(enc2utf8(x), sub = &quot;byte&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for (i in 1:length(words))&#xA;{&#xA;    Encoding(words[[i]])=&quot;UTF-8&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;AFTER:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Converting to UTF-8 on the corpus didn't work with:&#xA;&#xA;words.corpus &amp;lt;- tm_map(words.corpus, removeWords, remove_words)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words.corpus &amp;lt;- tm_map(words.corpus, content_transformer(stringi::stri_trans_tolower))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words.corpus &amp;lt;- tm_map(words.corpus, function(x) iconv(x, to='UTF-8'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words.corpus &amp;lt;- tm_map(words.corpus, enc2utf8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;nor&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words.corpus &amp;lt;- tm_map(words.corpus, tolower)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;All these solutions may have worked at a certain point in time, so I don't want to discredit the authors. They may work some time in the future. But why they didn't work is almost impossible to say because there were good reasons why they were supposed to work.&#xA;Anyway, just remember to convert the text before creating the corpus with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words &amp;lt;- iconv(words, &quot;ASCII&quot;, &quot;UTF-8&quot;, sub=&quot;byte&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49089" LastActivityDate="2018-03-19T16:16:44.813" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29273" PostTypeId="1" CreationDate="2018-03-19T16:53:38.037" Score="1" ViewCount="112" Body="&lt;p&gt;Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used the random input arrays as the features in the dataset and the output of the decision tree as the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feed my dataset to the neural network, and start it training. If there are 6 features, logically, a two-layer network can't learn a million-bit decision tree, because there simply isn't enough &quot;information capacity&quot;, essentially entropy, in the neural network structure to store such a complex ruleset. With decision trees, calculating entropy is fairly simple. But how would you do it for a neural network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would you calculate the minimum size neural network needed to learn a given ruleset?&lt;/p&gt;&#xA;" OwnerUserId="49092" LastEditorUserId="49092" LastEditDate="2018-03-19T17:00:36.823" LastActivityDate="2018-03-19T17:00:36.823" Title="How do you calculate the information capacity of a neural network?" Tags="|neural-network|decision-trees|information-theory|theory|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29275" PostTypeId="1" CreationDate="2018-03-19T18:20:32.467" Score="-1" ViewCount="319" Body="&lt;p&gt;I have text data representing sensor outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dataset&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1458996986002;    11.43,-15.86,11.20,508.26;        -1.59,-0.22,6.17,40.68;        126.0,-150.9,-105.0,49671.81;        Walk&#xA;1459002923002;    16.69,-12.68,13.96,634.65;        -2.55,2.13,4.87,34.87;         126.0,-150.9,-105.0,49671.81;        Walk&#xA;&#xA;&#xA;timestamp;    acc_x,acc_y,acc_z;    gyro_x,gyro_y,gyro_z;    magn_x,magn_y,magn_z;    ActivityName&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Goal&lt;/strong&gt;: I would like to extract features from the text lines before feeding it into a Recurrent Neural Network (GRU/LSTM). So, my goal is &lt;strong&gt;automatic feature extraction&lt;/strong&gt;. Those extracted features (encoder network) will be used before the neural network for an activity recognition task (classification).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Question&lt;/strong&gt;:  Which Autoencoder (denoising, variational, sparse) is suitable for such dataset? Or should I use RBM instead? After choosing a feature extraction method, how do I compare the output with input, since the input is not 0s and 1s?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have read&lt;/strong&gt;: I read that RBM is generative model, which, even if you give some similar input, it can generate similar correct output. And training autoencoder is &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/20ue0z/what_are_the_practical_difference_between_an_rbm/cg74tpy/?st=jex5e27b&amp;amp;sh=5cb31722&quot; rel=&quot;nofollow noreferrer&quot;&gt;said(1)&lt;/a&gt; &lt;a href=&quot;http://qr.ae/TU1pGg&quot; rel=&quot;nofollow noreferrer&quot;&gt;to be easier(2)&lt;/a&gt; than RBM. On the other hand, variational autoencoder can do something similar (&lt;a href=&quot;https://stats.stackexchange.com/a/328181/193154&quot;&gt;generative&lt;/a&gt;). &lt;strong&gt;First question would be is having &lt;em&gt;generative&lt;/em&gt; ability has any advantage for above problem, because at the end of pretraining i would just use encoder part and throw decoder layers&lt;/strong&gt;? If no, denoising autoencoder seems right approach I'm my opinion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last thing: I know to force network to learn important features from data (instead of memorizing), you can choose following methods: 1) &lt;a href=&quot;https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798&quot; rel=&quot;nofollow noreferrer&quot;&gt;use sandwich like layers&lt;/a&gt;, 2) add noise to input 3) &lt;a href=&quot;https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798&quot; rel=&quot;nofollow noreferrer&quot;&gt;regularize autoencoder&lt;/a&gt;=make only some nodes active at the same time &lt;/p&gt;&#xA;" OwnerUserId="49075" LastEditorUserId="29169" LastEditDate="2019-09-15T21:17:18.240" LastActivityDate="2020-08-08T08:58:19.780" Title="Suitable Autoencoder for Activity Recognition dataset Feature Extraction" Tags="|neural-network|classification|feature-extraction|autoencoder|rbm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29276" PostTypeId="1" AcceptedAnswerId="29421" CreationDate="2018-03-19T18:24:43.783" Score="3" ViewCount="440" Body="&lt;p&gt;I am reading the paper on &lt;a href=&quot;https://arxiv.org/pdf/1410.5401.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neural Turing Machine, page 9,&lt;/a&gt; and got stuck on just 1 pesky point.&lt;/p&gt;&#xA;&lt;p&gt;I can't understand how the shift is done:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Each head emits a shift weighting &lt;span class=&quot;math-container&quot;&gt;$\vec{s_t}$&lt;/span&gt; that defines a normalized distribution over the allowed integer shifts &lt;strong&gt;(Authors mean entries in s will add up to 1)&lt;/strong&gt;. For example, if shifts between -1 and 1 are allowed, &lt;span class=&quot;math-container&quot;&gt;$\vec{s_t}$&lt;/span&gt; has three elements corresponding to the degreee to which shifts of -1, 0 and 1 are performed.&lt;/p&gt;&#xA;&lt;p&gt;If we index the &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; memory locations from 0 to &lt;span class=&quot;math-container&quot;&gt;$N-1$&lt;/span&gt;, the rotation applied to &lt;span class=&quot;math-container&quot;&gt;$\vec{w_t^g}$&lt;/span&gt; by &lt;span class=&quot;math-container&quot;&gt;$\vec{s_t}$&lt;/span&gt; can be expressed as the following circular convolution:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$w_{new}(i) \leftarrow \sum_{j=0}^{N-1}w^g_t(j)s_t(i-j)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where all index arithmetic is computed modulo &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The authors use notation &lt;span class=&quot;math-container&quot;&gt;$w_t(i)$&lt;/span&gt; to signify i'th element in the vector.&lt;/p&gt;&#xA;&lt;p&gt;I cannot understand how &lt;span class=&quot;math-container&quot;&gt;$s_t$&lt;/span&gt; is applied to the &lt;span class=&quot;math-container&quot;&gt;$w^g_t$&lt;/span&gt; and what makes it possible to actually perform shift, say by +1&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;For example, assume we are working on the the i'th element that is at index 3 and &lt;span class=&quot;math-container&quot;&gt;$w^g_t$&lt;/span&gt; has 5 entries.&lt;/p&gt;&#xA;&lt;p&gt;Like in the paper, let's allow &lt;span class=&quot;math-container&quot;&gt;$s$&lt;/span&gt; to be able to perform a shift of -1, 0 or 1. Thus, &lt;span class=&quot;math-container&quot;&gt;$s$&lt;/span&gt; has 3 dimensions. Now, I want to shift every element in &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; by +1 forward.&lt;/p&gt;&#xA;&lt;p&gt;I have a feeling I will get out of bounds error, and modulo won't help too much. We can unfold this computation for the 3rd element:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;w[3] = wg[0]*s[(3-0)%5] + wg[1]*s[(3-1)%5] + wg[2]*s[(3-2)%5] + wg[3]*s[(3-3)%5] + wg[4]*s[(3-4)%5] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;when simplified this means:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;w[3] = wg[0]*s[3] + wg[0]*s[2] + wg[0]*s[1] + wg[0]*s[0] + wg[0]*s[4]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;As you can see, I got s[3] at the start and s[4] at the end. ...But we've just agreed &lt;strong&gt;s&lt;/strong&gt; will only have 3 entries {-1, 0, 1} and this means out of bounds error.&lt;/p&gt;&#xA;&lt;p&gt;And, I can't really understand how what we've done would &amp;quot;shift&amp;quot; the entire &lt;span class=&quot;math-container&quot;&gt;$w^g$&lt;/span&gt; into &lt;span class=&quot;math-container&quot;&gt;$w_{new}$&lt;/span&gt; once we do it for all 5 entries (we've done the above just for 3rd entry). Can someone give me the intuition for it as well?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-08-13T13:47:10.393" Title="Neural Turing Machine - how the shifting works?" Tags="|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29277" PostTypeId="2" ParentId="29237" CreationDate="2018-03-19T19:00:41.863" Score="2" Body="&lt;p&gt;I believe the key is that when the filter is convolving some part of the image (the &quot;receptive field&quot;) each number in the filter (i.e. each weight) is &lt;strong&gt;first flattened into vector format&lt;/strong&gt;.  Likewise, the pixels of the image are also &lt;strong&gt;flattened into vector format&lt;/strong&gt;.  THEN, the dot product is calculated.  Which is the exact same thing as finding the sum of the element-by-element (element-wise) multiplication. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, these  flattened vectors can also be combined in a matrix format, as the below image shows.  In this case then true matrix multiplication can be used, but it's important to note that the flattening the image pixels from each convolution and also the weights filter is the precursor.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IQWdM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IQWdM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;image credit: &lt;a href=&quot;https://www.youtube.com/watch?v=u4alGiomYP4&amp;amp;t=641s&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow and Deep Learning without a PhD, Part 1 (Google Cloud Next '17)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="10548" LastEditDate="2018-03-25T00:51:52.957" LastActivityDate="2018-03-25T00:51:52.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29278" PostTypeId="1" AcceptedAnswerId="29282" CreationDate="2018-03-19T19:07:15.473" Score="1" ViewCount="998" Body="&lt;p&gt;I am working on an NLP project about classifying offensive text data in social media. By offensive I especially mean threat words that one say to another.&#xA;&lt;br/&gt;&lt;br/&gt;&#xA;&lt;strong&gt;Some examples:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&amp;quot;Stop doing this or else you will pay for it.&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;Just wait until you see what's coming&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;&amp;quot;I will break your legs next time I see you.&amp;quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;As an initial approach, I considered semantic and syntactic keyword matching. However, doing this on this very problem seemed harder because threating is an action and it is expressed in so many different ways.&#xA;&lt;br/&gt;&lt;br/&gt;&#xA;My main goal is classifying text data by offensive/non-offensive text by using Machine Learning and Deep Learning algorithms. After weeks of online searching, I could not find a ready-to-use dataset. I considered manually labelling the data. However, I don't know where I should start.&lt;/p&gt;&#xA;&lt;p&gt;What would be the best approach for making progress in this task? I also plan to do this in both English and German languages.&lt;/p&gt;&#xA;&lt;p&gt;Also, below is a related article for fully understanding of the problem:&lt;br/&gt;&#xA;&lt;a href=&quot;https://link.springer.com/content/pdf/10.1007%2Fs41060-017-0088-4.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep learning for detecting inappropriate content in text&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49097" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-20T19:05:12.757" Title="Detecting Offensive Text Content in English and German" Tags="|classification|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29279" PostTypeId="1" AcceptedAnswerId="29281" CreationDate="2018-03-19T19:16:20.233" Score="0" ViewCount="170" Body="&lt;p&gt;I know handcrafted features are features which are being created by a human made algorithm. But what are semi-handcrafted features? &lt;/p&gt;&#xA;" OwnerUserId="49098" LastActivityDate="2018-03-19T19:23:24.890" Title="What is the meaning of semi-handcrafted features?" Tags="|machine-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29281" PostTypeId="2" ParentId="29279" CreationDate="2018-03-19T19:23:24.890" Score="0" Body="&lt;p&gt;It means part of it handled by the human at first, and then, base on them running a machinery learning algorithms to extract new features. For example, gather some features from the human about some pictures, and using some machinery algorithms to find some new features by some non-linear combinations of those features. &lt;/p&gt;&#xA;" OwnerUserId="27665" LastActivityDate="2018-03-19T19:23:24.890" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29282" PostTypeId="2" ParentId="29278" CreationDate="2018-03-19T19:59:32.413" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge&quot; rel=&quot;nofollow noreferrer&quot;&gt;Toxic comment classification challenge&lt;/a&gt; might be a good place to start.  It contains a set of comments and 6 binary classifications indicting if it's a toxic comment and of which type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imagine this would be a sufficient start.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-19T19:59:32.413" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29283" PostTypeId="1" AcceptedAnswerId="29302" CreationDate="2018-03-19T20:16:25.640" Score="4" ViewCount="6695" Body="&lt;p&gt;I've got a collection of yearly data (one value per year per category), and I'd like to find series that are most similar to one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://gist.github.com/jacobpatton/642b4e80dbd3bab445504dd9c91cd116&quot; rel=&quot;nofollow noreferrer&quot;&gt;Example data is here.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't know much about data science, but it seems like cosine similarity might be the way to go? If so, how do I account for the nil values in the datasets?&lt;/p&gt;&#xA;" OwnerUserId="49102" LastEditorUserId="45221" LastEditDate="2018-03-20T15:23:57.573" LastActivityDate="2018-04-19T16:08:58.847" Title="How to find similar time series?" Tags="|time-series|similarity|cosine-distance|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29284" PostTypeId="1" CreationDate="2018-03-19T21:09:49.350" Score="-1" ViewCount="16846" Body="&lt;p&gt;I was trying to implement a model to distinguish between low or high pass filters acting on a white noise signal by using Scikit Learn's logistic regression. It seems to be working fine but when I extract the parameters &lt;code&gt;b=intercept_&lt;/code&gt;, and &lt;code&gt;m=coef_&lt;/code&gt; and use them to plot &lt;code&gt;1/(1+np.exp(-m*x-b)&lt;/code&gt;, the plot differs from when I use the predict function of the logistic regression. I can get it to be exactly by multiplying everything in &lt;code&gt;np.exp()&lt;/code&gt; by an arbitrarily large number but I am not sure what the exact value is. My aim was to use the &lt;code&gt;intercept_&lt;/code&gt; parameter as an estimate for the cut-off frequency that's why I need the exact value of the parameter, the other one is just for determining between high and low pass so all I need is the sign. Anyone by any chance know what it is? I looked everywhere but I still cannot seem to find the answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your time.&lt;/p&gt;&#xA;" OwnerUserId="49104" LastActivityDate="2018-04-19T23:15:47.477" Title="Coefficients from Logistic Regression using Scikit-Learn" Tags="|classification|scikit-learn|regression|logistic-regression|parameter|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29285" PostTypeId="1" AcceptedAnswerId="29289" CreationDate="2018-03-19T21:29:04.307" Score="1" ViewCount="207" Body="&lt;p&gt;For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems to work fine when benchmarking in on the MNIST dataset, and using only sigmoid activation functions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately I seem to get issues when replacing these with &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)&quot; rel=&quot;nofollow noreferrer&quot;&gt;relus&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what my learning curve looks like for 50 epochs on a training dataset of ~500 examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sh2EW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sh2EW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Everything is fine for the first ~8 epochs and then I get a complete collapse on the score of a dummy classifier (~0.1 accuracy). I checked the code of the relu and it seems fine. Here are my forward and backward passes:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fprop(self, inputs):&#xA;    return np.maximum( inputs, 0.)&#xA;&#xA;def bprop(self, inputs, outputs, grads_wrt_outputs):&#xA;    derivative = (outputs &amp;gt; 0).astype( float)&#xA;    return derivative * grads_wrt_outputs&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The culprit seems to be in the numerical stability of the relu. I tried different learning rates and many parameter initializers for the same result. &lt;code&gt;Tanh&lt;/code&gt; and &lt;code&gt;sigmoid&lt;/code&gt; work properly. Is this a known issue? Is it a consequence of non-continuous derivative of the &lt;code&gt;relu&lt;/code&gt; function?&lt;/p&gt;&#xA;" OwnerUserId="39830" LastActivityDate="2018-03-21T11:14:01.207" Title="Homemade deep learning library: numerical issue with relu activation" Tags="|machine-learning|deep-learning|backpropagation|activation-function|numerical|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29287" PostTypeId="1" CreationDate="2018-03-20T00:37:41.097" Score="10" ViewCount="22233" Body="&lt;p&gt;I have a &lt;strong&gt;streaming data&lt;/strong&gt; along with timestamp dataset that looks like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/viQBw.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/viQBw.png&quot; alt=&quot;Sample Data Look alike&quot; /&gt;&lt;/a&gt;1.png&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Timestamp&lt;/strong&gt; can be inclusive of &amp;quot;seconds&amp;quot; too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Column1, Column2 .... ColumnN&lt;/strong&gt; correspond to the variables (they change over time) and &amp;quot;Label&amp;quot; shows the different samples. You can assume that the &lt;strong&gt;values tend to decrease over time for a particular label&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Labels A1,B1,C1.........A2,...&lt;/strong&gt; M labels.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Note : Values of timeNew of a Label depends on values of  timeOld of that Label and Labels belong to its cluster.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;I need to group &lt;strong&gt;Labels&lt;/strong&gt; with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).&lt;/p&gt;&#xA;&lt;p&gt;I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.&lt;/p&gt;&#xA;" OwnerUserId="49105" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2020-06-24T20:56:34.533" Title="Multivariate Time-Series Clustering" Tags="|machine-learning|python|clustering|time-series|" AnswerCount="3" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29288" PostTypeId="2" ParentId="29284" CreationDate="2018-03-20T02:25:36.793" Score="1" Body="&lt;p&gt;Can you post some sample code of what you're trying to do?  I can match predict_proba exactly when using the coefficients and intercept.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xmin, xmax = -5, 5&#xA;n_samples = 100&#xA;np.random.seed(0)&#xA;X = np.random.normal(size=n_samples)&#xA;y = (X &amp;gt; 0).astype(np.float)&#xA;X[X &amp;gt; 0] *= 4&#xA;X += .3 * np.random.normal(size=n_samples)&#xA;&#xA;X = X[:, np.newaxis]&#xA;x_train, x_test, y_train, y_test = train_test_split(X,y)&#xA;clf = linear_model.LogisticRegression(C=1e5)&#xA;clf.fit(x_train, y_train)&#xA;clf.predict_proba(x_test)&#xA;&#xA;def model(x):&#xA;    return 1 / (1 + np.exp(-x))&#xA;&#xA;model(x_test * clf.coef_ + clf.intercept_).ravel()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-20T02:25:36.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29289" PostTypeId="2" ParentId="29285" CreationDate="2018-03-20T02:33:38.033" Score="2" Body="&lt;p&gt;I don't know exactly what the problem is, but maybe you could try checking the value of your gradients and see if they change a lot around the 8th epoch.&lt;/p&gt;&#xA;" OwnerUserId="49109" LastActivityDate="2018-03-20T02:33:38.033" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29291" PostTypeId="1" CreationDate="2018-03-20T04:04:06.270" Score="1" ViewCount="33" Body="&lt;p&gt;I just recently got interested in Data Science, and just encountered concepts such as embedding from high dimensions to low dimensions. Maybe on an irrelevant note, but in this video: &lt;a href=&quot;https://www.youtube.com/watch?v=CQbbsKK1kus&quot; rel=&quot;nofollow noreferrer&quot;&gt;Geometry of Singular Value Decomposition (SVD) | Lê Nguyên Hoang&lt;/a&gt;, he mentioned that we can think of SVD as mapping of movies into concepts and mapping of users into concepts. Furthermore, he also mentioned that such concepts are to be data-driven. That makes me wonder if the theorem of SVD imply that there exists intrinsic categorizations of data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone give me any analysis on why this might be true or false ? Also it would be appreciated if you can appeal to intuitions, since I am very new to this area.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; The url was corrected, i mistook it for another video&lt;/p&gt;&#xA;" OwnerUserId="49113" LastActivityDate="2018-03-20T04:04:06.270" Title="SVD and intrinsic categorization" Tags="|embeddings|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29292" PostTypeId="1" AcceptedAnswerId="29384" CreationDate="2018-03-20T04:31:46.240" Score="0" ViewCount="4962" Body="&lt;p&gt;I have been getting poor results on my time series predictions with a LSTM network. I'm looking for any ideas to improve the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iM6sn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iM6sn.png&quot; alt=&quot;Predictions VS True Data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The above graph shows the True Data vs. Predictions. The True Data is smooth zig zag shaped, from 0 to 1. However the predictions rarely reach 0 or 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The distribution in the prediction data-set rarely reaches 0 or 1 and it's centered around 0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/okpFO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/okpFO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However the distributions in the True Data set is evenly distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/P0kh4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/P0kh4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the LSTM model built in keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dropout(0.4, input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))&#xA;model.add(Bidirectional(LSTM(30, dropout=0.4, return_sequences=False, recurrent_dropout=0.4), input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))&#xA;model.add(Dense(1))&#xA;model.compile(loss='mae', optimizer='adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I get the predictions to be more similar to the true data?&lt;/p&gt;&#xA;" OwnerUserId="49114" LastEditorUserId="29575" LastEditDate="2018-03-20T14:50:43.503" LastActivityDate="2018-03-21T17:15:53.083" Title="Improving LSTM Time-series Predictions" Tags="|keras|time-series|regression|lstm|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29293" PostTypeId="1" CreationDate="2018-03-20T07:32:29.060" Score="3" ViewCount="10504" Body="&lt;p&gt;I am applying regression to a dataset comprising 110 rows and 7 columns each with targets. When I applied Lasso regression to the data and calculated the RMSE value, the RMSE value was &lt;code&gt;13.11&lt;/code&gt;. I think the RMSE value should be close to zero. What is the permissible values of RMSE in a regression model? What could have gone wrong in the computation?&lt;/p&gt;&#xA;&lt;p&gt;My code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import linear_model&#xA;reg = linear_model.Lasso(alpha = .00001)&#xA;reg.fit(Xt,Yt)&#xA;ans=reg.predict(Xts)&#xA;print(ans)&#xA;from sklearn.metrics import mean_squared_error&#xA;print(mean_squared_error(Yts, ans))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Whereas when I try cross validation the MSE scores are way below &lt;code&gt;0.35&lt;/code&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;kfold = KFold(n_splits=10)&#xA;results = cross_val_score(reg, full_data, target, cv=kfold)&#xA;print(&amp;quot;Results: %.2f (%.2f) MSE&amp;quot; % (results.mean(), results.std()))&#xA;results&#xA;Results: -0.13 (0.45) MSE&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="29169" LastEditDate="2021-09-23T01:42:52.610" LastActivityDate="2021-09-23T01:44:22.653" Title="High root mean squared error in regression model" Tags="|machine-learning|scikit-learn|regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29294" PostTypeId="2" ParentId="29293" CreationDate="2018-03-20T08:42:07.247" Score="8" Body="&lt;p&gt;RMSE does not work that way. A RMSE of &lt;code&gt;13&lt;/code&gt; might actually be great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range &lt;code&gt;[0,1e9]&lt;/code&gt;, then a RMSE of &lt;code&gt;13&lt;/code&gt; is spectacular. On the other hand, if your target is in the range &lt;code&gt;[0,1]&lt;/code&gt;, a RMSE of &lt;code&gt;0.5&lt;/code&gt; is terrible. If you would like to try a metric that can be more readily interpretable as having a &amp;quot;good&amp;quot; or &amp;quot;bad&amp;quot; score, try the Mean Average Percent Error (MAPE).&lt;/p&gt;&#xA;&lt;p&gt;As far as why you get a lower MSE when you cross validate: you do not show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I would recommend using bootstrap cross validation instead of k-fold.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="29169" LastEditDate="2021-09-23T01:44:22.653" LastActivityDate="2021-09-23T01:44:22.653" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="29295" PostTypeId="1" AcceptedAnswerId="29296" CreationDate="2018-03-20T08:52:11.477" Score="4" ViewCount="4456" Body="&lt;p&gt;Here I try to construct a classifier using DNN(deep neural network) with its inputs being many portfolios. In essence, each portfolio contains several stocks which are labeled by there inner-code, for example &quot;1430&quot; or &quot;5560&quot;, etc. Since the inner-code is discrete digital number, I prefer to use on-hot encoding to represent each portfolio. However, there are as many as a few thousand different inner-code which means that the dimension of on-hot code may also be that large. I wonder whether the dimension is too large for real training and is there any way to solve or ease this problem, such as using PCA afterward.&lt;/p&gt;&#xA;" OwnerUserId="49119" LastActivityDate="2018-03-20T09:14:27.407" Title="Is it effective to use one-hot encoding when its dimension is as large as thousands" Tags="|machine-learning|neural-network|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29296" PostTypeId="2" ParentId="29295" CreationDate="2018-03-20T09:14:27.407" Score="4" Body="&lt;p&gt;In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the &lt;em&gt;vocabulary&lt;/em&gt; (the set of supported tokens) is around 32K. These tokens are typically one-hot encoded. Therefore, &lt;strong&gt;there is not inherent problem in having one-hot encoded vectors with thousands of components&lt;/strong&gt;. However, the amount and variety of training data should &lt;em&gt;support&lt;/em&gt; the dimensionality of the model for it not to overfit. Note that in NLP, these one hot encodings are used as expected outputs of the model, over which to compute maximum likelihood (categorical cross-entropy) together with the output of the model (a softmax).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Depending on how you are using this portfolio information, &lt;strong&gt;another option would be to have an &lt;a href=&quot;https://stats.stackexchange.com/q/182775/40048&quot;&gt;embedding layer&lt;/a&gt;&lt;/strong&gt;, which &lt;em&gt;embeds&lt;/em&gt; discrete values into a continuous representation space (whose dimensionality is a hyperparameter of the model). This would be possible if this information is used as &lt;em&gt;input&lt;/em&gt; to the model. In those cases, you input an integer number, which is used to index the embedded vector table.&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-20T09:14:27.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29297" PostTypeId="2" ParentId="29292" CreationDate="2018-03-20T10:11:18.513" Score="1" Body="&lt;p&gt;I have just used LSTM to train a model predict time series value and get good result as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# reshape input to be [samples, time steps, features]&#xA;trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))&#xA;testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))&#xA;&#xA;model = Sequential()&#xA;model.add(LSTM(40, input_shape=(trainX.shape[1], trainX.shape[2])))&#xA;model.add(Dense(1))&#xA;model.compile(loss='mean_absolute_error', optimizer='adam')&#xA;model.fit(trainX, trainY, epochs=10, batch_size=1, verbose=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is my result: blue is real value, orange is predict.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qntd1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qntd1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45999" LastActivityDate="2018-03-20T10:11:18.513" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29299" PostTypeId="2" ParentId="29287" CreationDate="2018-03-20T11:01:10.667" Score="0" Body="&lt;p&gt;It depends a bit if the timestamps have any connection to each other (is t2 impacted by t1 as example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general this looks like a classification problem and you can use for example &lt;a href=&quot;http://scikit-learn.org/stable/supervised_learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;. If you want to distinguish A and C and all other cases you would end up with a multi-class classification problem and not all algorithms support these. If not You can just transform the Label into target (A+B) /non_target (the rest). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Three additional advice: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You can use pandas Dataframes for the pre-processing.&lt;/li&gt;&#xA;&lt;li&gt;You should at least have a basic understanding of test and training samples before you start anything.&lt;/li&gt;&#xA;&lt;li&gt;The correct success metric is both important and sometimes challenging to find. Quick example: If you choose accuracy and 99% of your values belong to a class any algorithm that per default predicts this class will have a very high accuracy. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If timestamps are connected you might want to do some pre-processing and add data from previous timestamps to the current one (example: average value of column 1 of last x previous tstamps).&lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2018-03-20T11:01:10.667" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29300" PostTypeId="2" ParentId="29287" CreationDate="2018-03-20T11:06:47.833" Score="2" Body="&lt;p&gt;Aleksandr Blekh's answer in &lt;a href=&quot;https://datascience.stackexchange.com/questions/3738/how-to-deal-with-time-series-which-change-in-seasonality-or-other-patterns&quot;&gt;this older question&lt;/a&gt; provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1512.06747.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;An approach on the use of DTW with multivariate time-series&lt;/a&gt; (the paper actual refers to classification but you might want to use the idea and adjust it for clustering)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://jmlr.org/papers/volume17/khaleghi16a/khaleghi16a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A paper on clustering of time-series&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://infolab.usc.edu/DocsDemos/mmdb04.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A PCA-based similarity measure for multivariate time-series&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://pdfs.semanticscholar.org/d74f/a52a7ddd41b175378dbe0604e635dab8a708.pdf?_ga=2.170588256.1353094283.1521625278-401163560.1521625278&quot; rel=&quot;nofollow noreferrer&quot;&gt;A review on feature extraction and pattern recognition methods in time-series data&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Before proceeding with any method, I believe it is important to spend some time to think of the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Depending on what you are trying to find out, you need to decide&#xA;whether Dynamic time warping (DTW) is useful for you. For example, if&#xA;2 time-series have the exact same pattern but one of the 2 has a&#xA;time-delay, should they still belong in the same cluster? (and how&#xA;small/big time-delay is acceptable to put them in the same cluster?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45221" LastEditorUserId="45221" LastEditDate="2018-03-21T09:49:55.713" LastActivityDate="2018-03-21T09:49:55.713" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29301" PostTypeId="2" ParentId="10430" CreationDate="2018-03-20T12:05:12.577" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://automl.github.io/SMAC3/stable/index.html#&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMAC&lt;/a&gt; has a parallel version available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure it handles the single runs as you want, but it sounds strange to me if it don't. Give a try!&lt;/p&gt;&#xA;" OwnerUserId="29134" LastActivityDate="2018-03-20T12:05:12.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29302" PostTypeId="2" ParentId="29283" CreationDate="2018-03-20T12:07:09.187" Score="4" Body="&lt;p&gt;Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in comparing the patterns, a very simple approach would be Pearson's correlation. Keep in mind that this will not compare the actual values but the patterns (i.e. if the values have similar fluctuations with the years, so for example time-series [1 2 3 4] would have higher correlation with [5 6 7 8] than with [1 1 2 2])&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are interested in both values and pattern, I would use a distance-based metric: Euclidean Distance, Manhattan distance etc. I believe you will find &lt;a href=&quot;https://stats.stackexchange.com/questions/36152/converting-similarity-matrix-to-euclidean-distance-matrix/36158#36158&quot;&gt;this&lt;/a&gt; post interesting, where the mathematical background of similarity is explained. Also, python implementations of several distance metrics in python (including cosine-similarity) can be found in &lt;a href=&quot;http://dataaspirant.com/2015/04/11/five-most-popular-similarity-measures-implementation-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; blog-post.&lt;/p&gt;&#xA;" OwnerUserId="45221" LastActivityDate="2018-03-20T12:07:09.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29306" PostTypeId="1" CreationDate="2018-03-20T13:12:59.290" Score="3" ViewCount="1117" Body="&lt;p&gt;I tried to follow the example in this link:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/programmers_guide/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/programmers_guide/datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but I am totally lost about how to run the session. I understand the first argument is the operations to run, and feed_dict is the placeholders (my understanding is the batches of the training or test dataset), &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = 100&#xA;handle_mix = tf.placeholder(tf.float64, shape=[])&#xA;handle_src0 = tf.placeholder(tf.float64, shape=[])&#xA;handle_src1 = tf.placeholder(tf.float64, shape=[])&#xA;handle_src2 = tf.placeholder(tf.float64, shape=[])&#xA;handle_src3 = tf.placeholder(tf.float64, shape=[])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I create the dataset from mp4 tracks and stems, reading mixture and sources magnitudes, and pad them to be suitable to batching&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataset = tf.data.Dataset.from_tensor_slices(&#xA;    {&quot;x_mixed&quot;:padded_lbl, &quot;y_src0&quot;: padded_src[0], &quot;y_src1&quot;:   padded_src[1],&quot;y_src2&quot;: padded_src[1], &quot;y_src3&quot;: padded_src[1]})&#xA;&#xA;dataset = dataset.shuffle(1000).repeat().batch(batch_size)&#xA;&#xA;iterator = tf.data.Iterator.from_structure(dataset.output_types, dataset.output_shapes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;from the example I should do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;next_element = iterator.get_next()&#xA;&#xA;training_init_op = iterator.make_initializer(dataset)&#xA;for _ in range(20):&#xA;    # Initialize an iterator over the training dataset.&#xA;    sess.run(training_init_op)&#xA;    for _ in range(100):&#xA;        sess.run(next_element)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I have a loss, summaries, and optimiser operations and need to feed the data as batches, following another example as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;l, _, summary = sess.run([loss_fn, optimizer, summary_op], feed_dict={handle_mix: batch_mix, handle_src0: batch_src0,&#xA;                             handle_src1: batch_src1, handle_src2: batch_src2, handle_src3: batch_src3})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So I thought something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_mix, batch_src0, batch_src1, batch_src2, batch_src3 = data.train.next_batch(batch_size)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or maybe a separate run to fetch the batches first, then run the optimisation as above, such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_mix, batch_src0, batch_src1, batch_src2, batch_src3 = sess.run(next_element)&#xA;l, _, summary = sess.run([loss_fn, optimizer, summary_op], feed_dict={handle_mix: batch_mix, handle_src0: batch_src0, handle_src1: batch_src1, handle_src2: batch_src2, handle_src3: batch_src3})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This last attempt, returned string names of the batches as created in the tf.data.Dataset.from_tensor_slices (&quot;x_mixed&quot;, &quot;y_src0&quot;, ... etc) and failed to cast to tf.float64 placeholders in the session.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please let me know how to create this dataset, there might be an error in the structure from tensor slices in the first place, then how to batch them, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you very much, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Manal&lt;/p&gt;&#xA;" OwnerUserId="49137" LastEditorUserId="49137" LastEditDate="2018-03-24T16:59:16.327" LastActivityDate="2018-06-23T16:12:02.223" Title="Tensorflow dataset batching for complex data" Tags="|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29307" PostTypeId="2" ParentId="29179" CreationDate="2018-03-20T13:17:52.697" Score="1" Body="&lt;p&gt;There are multiple things at play here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, if your training and testing data are different then it becomes difficult to learn to generalize, because it will learn the relationships in your training data and if those relationships are significantly different then it will learn the wrong things. It could be that the data density is just different in your testing set which is not as much of a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second of all, if the relationship that you see is not linear, your logistic regression might not be able to pick up the relationship due to the underlying linear assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Third of all, maybe other features are highly (co)correlated and adding the new feature does not add any signal but does add the potential for fitting additional noise, thus explaining the performance reduction.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-03-20T13:17:52.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29311" PostTypeId="1" CreationDate="2018-03-20T13:37:26.070" Score="4" ViewCount="4505" Body="&lt;p&gt;I am very confused between one hot encoding and word embedding in terms of structure of the network and how it reduces the dimensionality.&lt;/p&gt;&#xA;&lt;p&gt;I am currently using encog with c# which has some documentation but not many differing examples.&lt;/p&gt;&#xA;&lt;p&gt;If I take an example of text classification by neural networks and have:&#xA;1000 different texts&#xA;10,000 unique words&lt;/p&gt;&#xA;&lt;p&gt;What I was previously doing was taking each text and then working out the unique words and putting them in a list. I would then work out the number of times each unique word appeared in a text and I would end up with a matrix of:&lt;/p&gt;&#xA;&lt;p&gt;1000 x 10,000 i.e. for each text a count of the number of times each unique word appears (text as rows and unique words (features) as columns).&lt;/p&gt;&#xA;&lt;p&gt;I would then normalise this and use &lt;a href=&quot;https://www.heatonresearch.com/2017/02/10/equilateral.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;equilateral encoding&lt;/a&gt; (similar to one hot).&lt;/p&gt;&#xA;&lt;p&gt;My network would then consist of 10,000 input neurons (one for each word (feature)) and then a hidden layer and then an output layer trying to predict a classification of an output column with 5 different values.&lt;/p&gt;&#xA;&lt;p&gt;This appears to be totally wrong but it did train and pretty accurately. I'm trying to work out with one hot encoding what exactly that would mean in terms of matrix sizes, what those matrices consist of and the layout of the network.&lt;/p&gt;&#xA;&lt;p&gt;This is also related to using word embedding as when I use this I get the following size: 1000 (number of texts), 872 (text size), 300 (vector size). I therefore have 3 dimensions so that made me think in one hot encoding I should probably have: 1000, 872, 10000 (unique words).&lt;/p&gt;&#xA;&lt;p&gt;If someone could explain where I am going wrong here and the difference between how I would approach one hot encoded text classification vs an embedded layer using:&lt;/p&gt;&#xA;&lt;p&gt;1000 texts, 10000 unique words, 872 text length and 300 vector size&lt;/p&gt;&#xA;" OwnerUserId="47869" LastEditorUserId="25180" LastEditDate="2022-02-24T18:11:04.730" LastActivityDate="2022-02-24T18:11:04.730" Title="One hot encoding vs Word embedding" Tags="|neural-network|word-embeddings|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29312" PostTypeId="1" AcceptedAnswerId="29313" CreationDate="2018-03-20T13:59:45.657" Score="-1" ViewCount="57" Body="&lt;p&gt;I'm looking for the exact value of epsilon to run the &lt;code&gt;DBSCAN&lt;/code&gt; clustering algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the &lt;code&gt;KNN&lt;/code&gt; distance plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R09KN.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R09KN.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This chart has two flex points. I need the second flex point. I'm using the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# evaluate kNN distance&#xA;dist &amp;lt;- dbscan::kNNdist(iris, 4)&#xA;&#xA;# order result&#xA;dist &amp;lt;- dist[order(dist)]&#xA;&#xA;# scale&#xA;dist &amp;lt;- dist / max(dist)&#xA;&#xA;# derivative&#xA;ddist &amp;lt;- diff(dist) / ( 1 / length(dist))&#xA;&#xA;# get first point where derivative is higher than 1&#xA;knee &amp;lt;- dist[length(ddist)- length(ddist[ddist &amp;gt; 1])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can improve my code to get second point where derivative is higher than 1?&lt;/p&gt;&#xA;" OwnerUserId="49127" LastEditorUserId="924" LastEditDate="2018-03-22T08:14:17.983" LastActivityDate="2018-03-22T08:14:17.983" Title="How to select the second point where derivative is greater than 1 in r?" Tags="|r|clustering|dbscan|" AnswerCount="1" CommentCount="1" ClosedDate="2018-03-22T20:55:53.673" ContentLicense="CC BY-SA 3.0" />
  <row Id="29313" PostTypeId="2" ParentId="29312" CreationDate="2018-03-20T14:35:54.503" Score="0" Body="&lt;p&gt;Check out the which method, it returns you the index where it satisfies the criteria.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; a &amp;lt;- c(-0.123, 1.2, 0.3, 8, -0.678)&#xA;&amp;gt; b &amp;lt;- which(a&amp;gt;1)&#xA;[1] 2 4&#xA;&amp;gt; b[2]&#xA;[1] 4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-03-20T14:35:54.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29314" PostTypeId="2" ParentId="28396" CreationDate="2018-03-20T14:43:01.847" Score="0" Body="&lt;p&gt;The easiest solution I found is &lt;a href=&quot;https://altair-viz.github.io/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Altair&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import altair as alt&#xA;&#xA;# assume df contains the DataFrame from the question + a numeric column `Participant`&#xA;alt.Chart(df).mark_point().encode(y='Skill', x='Participant', size='Age', color='Gender')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is what gets rendered out of the box:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UWHj3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UWHj3.png&quot; alt=&quot;Image generated with default settings&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46981" LastActivityDate="2018-03-20T14:43:01.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29315" PostTypeId="1" AcceptedAnswerId="36381" CreationDate="2018-03-20T14:53:30.840" Score="3" ViewCount="172" Body="&lt;p&gt;Say I have a tagging system on an electrical circuit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;&lt;b&gt;Name&lt;/b&gt;          &lt;b&gt;Description&lt;/b&gt;&#xA;-------       --------------&#xA;BT104         Battery. Power source&#xA;SW104         Circuit switch&#xA;LBLB-F104     Fluorescent light bulb&#xA;LBLB104       Light bulb&#xA;...           ...&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a hundreds of tags created by people who should have followed my naming conventions but sometimes they make mistakes add unnecessary additional characters to tag names (i.e. BTwq104 etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Up until now I used regular expressions, that I built over time whilst observing various inconsistencies that users introduce whilst naming different parts of their circuits, to parse the names and tell me what the different elements are. For example: name '&lt;em&gt;BT104&lt;/em&gt;' would tell me it's a &lt;em&gt;battery&lt;/em&gt; on &lt;em&gt;circuit 104&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to investigate or use a machine learning technique to identify what a tag name is (same way I used regular expressions). Any suggestions and approaches are welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I tried &lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;Named-entity recognition&lt;/a&gt; suggested technique &quot;&lt;strong&gt;Bag of words&lt;/strong&gt;&quot;. Followed a few tutorials &lt;a href=&quot;https://pythonprogramminglanguage.com/bag-of-words/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://machinelearnings.co/text-classification-using-neural-networks-f5cd7b8765c6&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; (latter being the most useful in learning). None of them produced wanted results if any. I think that &quot;Bag of words&quot; are mostly used for real word rather than made up words.&lt;/p&gt;&#xA;" OwnerUserId="38467" LastEditorUserId="29575" LastEditDate="2018-08-02T16:51:38.527" LastActivityDate="2018-08-08T07:34:49.883" Title="Recognising made up terms" Tags="|machine-learning|python|neural-network|classification|supervised-learning|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="29316" PostTypeId="1" CreationDate="2018-03-20T15:17:28.010" Score="-2" ViewCount="399" Body="&lt;p&gt;After applying k-means clustering we can easily find the closest index of the cluster. Now if I want to find the distance index from cluster center / index that is staying border of the cluster, how could I find it? &lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2018-03-28T07:21:29.553" Title="How to find data that is the maximum distance from cluster center or stayed border of the cluster" Tags="|machine-learning|python|scikit-learn|clustering|k-means|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29317" PostTypeId="1" CreationDate="2018-03-20T15:19:23.737" Score="1" ViewCount="1061" Body="&lt;p&gt;I am running pam function for kmedoid under cluster package in R. My dataset has 1 million entries.&#xA;It's throwing error  saying &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Have 1 million observation but not more than 65536 observation allowed&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Please suggest the solution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance :)&lt;/p&gt;&#xA;" OwnerUserId="16879" LastEditorUserId="16879" LastEditDate="2018-03-20T17:16:16.423" LastActivityDate="2018-10-19T04:02:44.837" Title="R: Pam function under cluster package not accepting data set having more than 65536 entries" Tags="|r|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29324" PostTypeId="1" CreationDate="2018-03-20T16:39:49.220" Score="0" ViewCount="965" Body="&lt;p&gt;I have time series data of the following properties:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input shape:  (num_timesteps, num_features)&#xA;output shape: (num_timesteps, num_outputs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I reshape it to batch form:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input shape:  (num_batches, num_timesteps_in_batch, num features)&#xA;output shape: (num_batches, num_timesteps_in_batch, num outputs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a stateful RNN in Keras:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;modelinput = Input(batch_shape=(num_batches,None,num_features))&#xA;prediction = GRU(10,return_sequences=True,stateful=True)(inputs)&#xA;model = Model(inputs=modelinput,outputs=prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;After trainig (which works fine) I would like to predict on a sequence without cutting the data, so input shape (num_timesteps, num_features). How can I do that?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought about having a second model that shares the weights with the RNN and that has dynamic input shapes. Is that possible?&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2024-02-01T01:06:35.763" Title="Predictions with arbitrairy sequence length for stateful RNN (LSTM/GRU) in Keras" Tags="|keras|rnn|prediction|reshape|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29328" PostTypeId="2" ParentId="29324" CreationDate="2018-03-20T17:12:06.600" Score="0" Body="&lt;p&gt;I found a possible solution: You can save the weights, create a complete new model and load the weights again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Original Stateful Model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;modelinput = Input(batch_shape=(num_batches,None,num_features))&#xA;prediction = GRU(10,return_sequences=True,stateful=True)(inputs)&#xA;model = Model(inputs=modelinput,outputs=prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Save Weights after Training&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(...)&#xA;model.save_weights('weights.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Create New Model (same structure, just not stateful and dynamic batch size)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;modelinput_pred = Input(batch_shape=(None,None,num_features))&#xA;prediction_pred = GRU(10,return_sequences=True,stateful=False)(inputs_pred)&#xA;model_pred = Model(inputs=modelinput,outputs=prediction_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Load Save Parameters and Predict&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model_pred.load_weights('weights.h5')&#xA;model_pred.predict(...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Nevertheless I hope for a better solution through sharing weights between a stateful GRU and a standard GRU. Is that possible?&lt;/p&gt;&#xA;" OwnerUserId="26643" LastActivityDate="2018-03-20T17:12:06.600" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29329" PostTypeId="1" CreationDate="2018-03-20T18:08:42.260" Score="2" ViewCount="949" Body="&lt;p&gt;I am trying to tag Hindi text using python(&lt;code&gt;nltk&lt;/code&gt; library). I have been successful but I am unable to understand some of the tags.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried searching for tagset but the only information I could find is about some &lt;code&gt;upenn_tagset&lt;/code&gt;. The tags that I get matches only some of the tags from &lt;code&gt;upenn_tagset&lt;/code&gt;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how to get the tagset for hindi?&lt;/p&gt;&#xA;" OwnerUserId="37063" LastEditorUserId="52196" LastEditDate="2019-07-18T19:30:09.683" LastActivityDate="2020-12-15T01:02:10.417" Title="How to get the tagset for hindi pos tagging?" Tags="|nlp|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29332" PostTypeId="2" ParentId="29223" CreationDate="2018-03-20T19:04:41.150" Score="3" Body="&lt;p&gt;Have a look at &lt;a href=&quot;https://arxiv.org/pdf/1701.08380.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The HASYv2 dataset&lt;/a&gt;. I tried to do as much of the exploratory work as possible to make sure that others can directly try more interesting thing with the dataset.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Image format specific stuff&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;(Minimum / Median / Mean / Maximum) (width / height / area)&lt;/li&gt;&#xA;&lt;li&gt;Image formats&lt;/li&gt;&#xA;&lt;li&gt;Timestamps&lt;/li&gt;&#xA;&lt;li&gt;Exiff meta data&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For this kind of stuff, you might want to have a look at &lt;a href=&quot;https://github.com/MartinThoma/edapy&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;edapy&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Image/ML specific stuff&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Things you can do with images:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Compute the mean image&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean image by class&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Eigenface&quot; rel=&quot;nofollow noreferrer&quot;&gt;Eigenfaces&lt;/a&gt; (or rather &quot;Eigenimages&quot;)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Facial_recognition_system#Traditional&quot; rel=&quot;nofollow noreferrer&quot;&gt;Fisher-Faces&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can compute the correlation of pixels, e.g. &lt;a href=&quot;https://arxiv.org/pdf/1701.08380.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Figure 3&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cOhqp.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cOhqp.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Classification-specific stuff&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Plot the distribution of classes.&lt;/li&gt;&#xA;&lt;li&gt;Behaviour of standard classification algorithms (CNNs, VGG-16)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Confusion Matrix Ordering&lt;/a&gt; (page 48 - 52, especially Figure 5.12 and 5.13): Find similar classes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UPoVg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UPoVg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-03-20T21:17:52.713" LastActivityDate="2018-03-20T21:17:52.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29333" PostTypeId="2" ParentId="12721" CreationDate="2018-03-20T19:11:09.113" Score="4" Body="&lt;p&gt;I’ve come to the same conclusion as yourself and others, traditional forecasting is still probably the most applicable and maybe reliable for time series of numeric values. There is some slight bleed in deep learning in discussion where time series for numeric values gets mixed into deep learning, where deep learning (currently) applies to modern challenges in pattern recognition for image, sound, clean text, or anomaly detection. I often have good results with VAR / VECM for daily transactional data, which could probably be applied to your signal processing use case. &lt;/p&gt;&#xA;" OwnerUserId="27519" LastActivityDate="2018-03-20T19:11:09.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29336" PostTypeId="1" AcceptedAnswerId="29342" CreationDate="2018-03-20T20:12:30.817" Score="0" ViewCount="1848" Body="&lt;p&gt;I am creating a very basic decision tree, the dataset being as follows (columns 1 to 11 are features and column 12 is prediction, I am slicing away column 0 in processing phase as in code below):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;|       domain       | A | AAAA | MX | NS | SOA | TXT | CAA | SSL | ccTLD | TTL | WHOIS | mal_or_ben |&#xA;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;| fedoraproject.org  | 2 |  2   | 1  | 2  |  1  |  0  |  2  |  0  |   0   |  0  |   0   |     1      |&#xA;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;| blackswanstore.com | 1 |  0   | 1  | 2  |  1  |  2  |  0  |  0  |   0   |  1  |   1   |     0      |&#xA;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;|    comcast.net     | 1 |  0   | 1  | 2  |  1  |  1  |  0  |  0  |   0   |  0  |   1   |     1      |&#xA;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;|     achren.org     | 1 |  0   | 1  | 2  |  1  |  0  |  0  |  0  |   0   |  1  |   1   |     0      |&#xA;|--------------------+---+------+----+----+-----+-----+-----+-----+-------+-----+-------+------------|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;although when I am passing the dataset, I dont actually pass the top row with the column names, just:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;fedoraproject.org,2,2,1,2,1,0,2,0,0,0,0,1&#xA;blackswanstore.com,1,0,1,2,1,2,0,0,0,1,1,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is how I am preparing my decision tree:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# read data from csv&#xA;balance_data = pd.read_csv(&quot;training_data.csv&quot;, sep=',', header=None)&#xA;&#xA;# x is predictor variable and y is outcome&#xA;X = balance_data.values[:, 1:12]&#xA;Y = balance_data.values[:, 12]&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=200)&#xA;y_train = y_train.astype('int')&#xA;y_test = y_test.astype('int')&#xA;&#xA;# criterion as gini index&#xA;clf_gini = DecisionTreeClassifier(criterion=&quot;gini&quot;, random_state=200,&#xA;                                  max_depth=5, min_samples_leaf=5)&#xA;clf_gini.fit(X_train, y_train)&#xA;&#xA;# export gini criterion&#xA;with open(&quot;graph.dot&quot;, &quot;w&quot;) as f:&#xA;    f = tree.export_graphviz(clf_entropy, out_file=f)&#xA;&#xA;# plot using graphviz - needs pydot and graphviz installed and set to system path&#xA;graph = Source(tree.export_graphviz(clf_entropy, out_file=None,&#xA;                                    feature_names=[&quot;A&quot;, &quot;AAAA&quot;, &quot;MX&quot;, &quot;NS&quot;, &quot;SOA&quot;,&#xA;                                                   &quot;TXT&quot;, &quot;CAA&quot;, &quot;SSL&quot;, &quot;ccTLD&quot;,&#xA;                                                   &quot;TTL&quot;, &quot;WHOIS&quot;]))&#xA;graph.format = &quot;png&quot;&#xA;graph.render(&quot;graph_render&quot;, view=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, in order to pass the feature name, what I am doing is just passing the column names in order as in the csv input. Is this correct (I am very new to this) or am I missing something here?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="23632" LastActivityDate="2018-03-20T23:41:40.267" Title="Correct Way of Displaying Features in Decision Tree" Tags="|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29337" PostTypeId="1" CreationDate="2018-03-20T20:15:47.177" Score="3" ViewCount="397" Body="&lt;p&gt;When using/building a system for &lt;a href=&quot;https://en.wikipedia.org/wiki/Entity_linking&quot; rel=&quot;nofollow noreferrer&quot;&gt;Entity Linking&lt;/a&gt;, is there a well-known measure for &quot;ambiguity degree&quot; of an entity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some approach to compare named entities regarding how difficult to disambiguate?&lt;/p&gt;&#xA;" OwnerUserId="6433" LastEditorUserId="6433" LastEditDate="2021-02-04T14:42:35.270" LastActivityDate="2021-02-04T14:42:35.270" Title="How to measure Entity Ambiguity?" Tags="|nlp|text-mining|metric|named-entity-recognition|research|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29339" PostTypeId="1" CreationDate="2018-03-20T22:11:21.317" Score="1" ViewCount="227" Body="&lt;p&gt;I would like to use out-of-bag training/validation with a classifier such as RandomForestClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to get the out-of-bag predictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want the OOB predictions so I can compute precision/recall on instances not used to train the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are alternative approaches to getting the precision/recall, please let me know.&lt;/p&gt;&#xA;" OwnerUserId="47380" LastActivityDate="2021-08-24T15:37:00.250" Title="Precision/recall from out-of-bag predictions in RandomForestClassifier" Tags="|classification|scikit-learn|random-forest|c|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29341" PostTypeId="2" ParentId="29284" CreationDate="2018-03-20T22:48:24.730" Score="1" Body="&lt;p&gt;In logistic regression, we don't fit a linear line to our data points. Instead, we fit a linear line to the log-likelihood of our data point. Therefore, you need to take exp to fit it accurately. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-03-20T22:48:24.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29342" PostTypeId="2" ParentId="29336" CreationDate="2018-03-20T23:41:40.267" Score="1" Body="&lt;p&gt;If you pass &lt;code&gt;header=None&lt;/code&gt;, &lt;code&gt;pandas.read_csv&lt;/code&gt; assumes that the first row contains data and names the columns '0' to '12'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead you should pass &lt;code&gt;header=0&lt;/code&gt; to specify that the column names are in the first row or equivalently skip the header argument.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then still continue with &lt;code&gt;X = balance_data.values[:, 1:12]&lt;/code&gt;, because calling &lt;code&gt;values&lt;/code&gt; returns a &lt;code&gt;numpy&lt;/code&gt; array without the column names.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you could also select your feature columns like so: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;feature_names = ['A','AAAA',....]&#xA;X = balance_data[feature_names].values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can then pass the same list of &lt;code&gt;feature_names&lt;/code&gt; to graphviz.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also note that you don't have to pass a &lt;code&gt;numpy&lt;/code&gt; array to &lt;code&gt;scikit-learn&lt;/code&gt;'s functions. It can handle &lt;code&gt;pandas&lt;/code&gt; DataFrames as well, so &lt;code&gt;values&lt;/code&gt; is optional.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-03-20T23:41:40.267" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29345" PostTypeId="1" AcceptedAnswerId="29351" CreationDate="2018-03-21T03:42:35.543" Score="6" ViewCount="4775" Body="&lt;p&gt;Recently, in an interview I got this question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Design a convnet that sorts numbers. Operators are ReLU, Conv, and Pooling. E.g. input: 5, 3, 6, 2; output: 2, 3, 5, 6&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not sure how can you sort a list of numbers using CNN. I know it is possible using RNN. Is it even possible? &lt;/p&gt;&#xA;" OwnerUserId="10947" LastEditorUserId="29169" LastEditDate="2021-02-13T16:36:06.243" LastActivityDate="2021-11-06T00:08:14.357" Title="How to sort numbers using Convolutional Neural Network?" Tags="|deep-learning|rnn|convolutional-neural-network|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29347" PostTypeId="1" AcceptedAnswerId="29360" CreationDate="2018-03-21T05:29:13.337" Score="0" ViewCount="61" Body="&lt;p&gt;Hi I'm currently trying to predict if an item will be successful in my store, this means (How much is going to sale in USD)&#xA;My training dataset contains many features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Item name&lt;/li&gt;&#xA;&lt;li&gt;Item weight&lt;/li&gt;&#xA;&lt;li&gt;Item category&lt;/li&gt;&#xA;&lt;li&gt;Item country of origin&lt;/li&gt;&#xA;&lt;li&gt;Item sales overall&lt;/li&gt;&#xA;&lt;li&gt;Item sales per store&lt;/li&gt;&#xA;&lt;li&gt;Item rating&lt;/li&gt;&#xA;&lt;li&gt;Item price&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Etc....&#xA;Since I will be introducing a new item for sale, I know very little about this new item:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Item name&lt;/li&gt;&#xA;&lt;li&gt;Item weight&lt;/li&gt;&#xA;&lt;li&gt;Item category&lt;/li&gt;&#xA;&lt;li&gt;Item country of origin&lt;/li&gt;&#xA;&lt;li&gt;Item price&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Not all the features present in training data/test data will be present when I will be making predictions. Is this normal in ML ? What is the rule of thumb when doing feature engineering for this type of cases.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastActivityDate="2018-03-21T09:35:57.903" Title="Data preparation for Regression Model" Tags="|dataset|prediction|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29348" PostTypeId="1" CreationDate="2018-03-21T05:30:18.500" Score="1" ViewCount="170" Body="&lt;p&gt;I have several sequences of univariate real-valued time-series data. The sequences are of different lengths and right now I cannot batch them and feed them to a network.&#xA;What is the correct procedure to pad these sequences? Is it even possible in this case since I can't use any number as a special symbol?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE 1&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working with arbitrary univariate time-series data (not related to one specific domain, unbounded range). To give example of one such a series consider standardized stock dataset (only first 10 elements shown):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = array([-0.37807043, 0.14321786, -0.37807043, 0.13478392, 0.18733381,&#xA;   1.19576774, 0.25675156, 0.26064414, 0.30930144, 0.38650436])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47858" LastEditorUserId="47858" LastEditDate="2018-03-21T06:15:17.123" LastActivityDate="2024-02-15T08:00:55.143" Title="How to pad real-valued sequences" Tags="|tensorflow|sequence-to-sequence|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29349" PostTypeId="2" ParentId="29117" CreationDate="2018-03-21T05:48:20.283" Score="2" Body="&lt;p&gt;You could apply character grams - Intuitively, there might be a huge difference in character set between a phone number and an email address. and then pass the character gram vector to SVM to make a prediction. You could implement this using in sklearn using the below feature extractors. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;TfIdfVectorizer(analyzer='character')&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CountVectorizer(analyzer='character')&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Cross validate on the ngram range and slack variables of SVM to fine tune your model. &lt;/p&gt;&#xA;" OwnerUserId="2608" LastActivityDate="2018-03-21T05:48:20.283" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29351" PostTypeId="2" ParentId="29345" CreationDate="2018-03-21T06:35:45.687" Score="9" Body="&lt;p&gt;I have a solution however I use a densely connected layer at the output to simplify the reshaping. If you can manipulate the sizes of this model such that you have 4 output parameters this should work as well. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import print_function&#xA;import keras&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten&#xA;from keras.layers import Conv1D, MaxPooling1D, Reshape&#xA;from keras.callbacks import ModelCheckpoint&#xA;from keras.models import model_from_json&#xA;from keras import backend as K&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Preparing the data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;We will generate some random lists containing integers between [0,49], we will take a random permutation of the list and then take the first 4 values. We will then set our targets $y$ as the sorted rows of $x$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;n = 100000&#xA;x_train = np.zeros((n,4))&#xA;for i in range(n):&#xA;    x_train[i,:] = np.random.permutation(50)[0:4]&#xA;&#xA;x_train = x_train.reshape(n, 4, 1)&#xA;y_train = np.sort(x_train, axis=1).reshape(n, 4,)&#xA;&#xA;n = 1000&#xA;x_test = np.zeros((n,4))&#xA;for i in range(n):&#xA;    x_test[i,:] = np.random.permutation(50)[0:4]&#xA;&#xA;x_test = x_test.reshape(n, 4, 1)&#xA;y_test = np.sort(x_test, axis=1).reshape(n, 4,)&#xA;&#xA;print(x_test[0][0].T)&#xA;print(y_test[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 44.  36.  13.   0.] &lt;br/&gt; [  0.  13.  36.  44.]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h1&gt;The model&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I tried different combinations of parameters. This worked out not bad.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_shape = (4,1)&#xA;&#xA;model = Sequential()&#xA;model.add(Conv1D(32, kernel_size=(2),&#xA;                 activation='relu',&#xA;                 input_shape=input_shape,&#xA;                 padding='same'))&#xA;&#xA;model.add(Conv1D(64, (2), activation='relu', padding='same'))&#xA;model.add(MaxPooling1D(pool_size=(2)))&#xA;model.add(Reshape((64,2)))&#xA;&#xA;model.add(Conv1D(32, (2), activation='relu', padding='same'))&#xA;model.add(MaxPooling1D(pool_size=(2)))&#xA;&#xA;model.add(Flatten())&#xA;model.add(Dense(4))&#xA;&#xA;model.compile(loss=keras.losses.mean_squared_error,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&#xA;epochs = 10&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;history = model.fit(x_train, y_train,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Epoch 10/10 &lt;br/&gt; 100000/100000 [==============================] - 6s&#xA;  56us/step - loss: 0.9061 - acc: 0.9973 - val_loss: 0.5302 - val_acc:&#xA;  0.9950&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jDHZJ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jDHZJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GWwN0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GWwN0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Results&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;So for a new list of values, I get the predicted output. Then I determine which value in the original list is closest to each of these and replace them. I could have just rounded the predicted values, however this caused so +/-1 errors due to rounding the wrong way. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_list = [1,45,3,18]&#xA;pred = model.predict(np.asarray(test_list).reshape(1,4,1))&#xA;print(test_list)&#xA;print(pred)&#xA;&#xA;print([np.asarray(test_list).reshape(4,)[np.abs(np.asarray(test_list).reshape(4,) - i).argmin()] for i in list(pred[0])])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1, 45, 3, 18] &lt;br/&gt; [[  0.87599814   3.43058085  17.36335754  45.21624374]] &lt;br/&gt;&#xA;  [1, 3, 18, 45]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And for the sequence you suggested as a test case&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_list = [5,3,6,2]&#xA;pred = model.predict(np.asarray(test_list).reshape(1,4,1))&#xA;print(test_list)&#xA;print(pred)&#xA;&#xA;print([np.asarray(test_list).reshape(4,)[np.abs(np.asarray(test_list).reshape(4,) - i).argmin()] for i in list(pred[0])])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[5, 3, 6, 2] &lt;br/&gt; [[ 1.85080266  2.95598722  4.92955017  5.88561296]] &lt;br/&gt;[2,&#xA;  3, 5, 6]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-21T06:35:45.687" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29352" PostTypeId="1" AcceptedAnswerId="29382" CreationDate="2018-03-21T07:18:26.937" Score="3" ViewCount="18628" Body="&lt;p&gt;I apologize if this question is misplaced -- I'm not sure if this is more of a &lt;code&gt;re&lt;/code&gt; question or a &lt;code&gt;CountVectorizer&lt;/code&gt; question. I'm trying to exclude any would be token that has one or more numbers in it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from sklearn.feature_extraction.text import CountVectorizer&#xA;&amp;gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt;&amp;gt; docs = ['this is some text', '0000th', 'aaa more 0stuff0', 'blahblah923']   &#xA;&amp;gt;&amp;gt;&amp;gt; vec = CountVectorizer()&#xA;&amp;gt;&amp;gt;&amp;gt; X = vec.fit_transform(docs)&#xA;&amp;gt;&amp;gt;&amp;gt; pd.DataFrame(X.toarray(), columns=vec.get_feature_names())&#xA;   0000th  0stuff0  aaa  blahblah923  is  more  some  text  this&#xA;0       0        0    0            0   1     0     1     1     1&#xA;1       1        0    0            0   0     0     0     0     0&#xA;2       0        1    1            0   0     1     0     0     0&#xA;3       0        0    0            1   0     0     0     0     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I want instead is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   aaa  is  more  some  text  this&#xA;0    0   1     0     1     1     1&#xA;1    0   0     0     0     0     0&#xA;2    1   0     1     0     0     0&#xA;3    0   0     0     0     0     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My thought was to use &lt;code&gt;CountVectorizer&lt;/code&gt;'s &lt;code&gt;token_pattern&lt;/code&gt; argument to supply a regex string that will match anything except one or more numbers:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; vec = CountVectorizer(token_pattern=r'[^0-9]+')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but the result &lt;em&gt;includes&lt;/em&gt; the surrounding text matched by the negated class:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   aaa more   blahblah  stuff  th  this is some text&#xA;0          0         0      0   0                  1&#xA;1          0         0      0   1                  0&#xA;2          1         0      1   0                  0&#xA;3          0         1      0   0                  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also, replacing the default pattern &lt;code&gt;(?u)\b\w\w+\b&lt;/code&gt; obviously messes with the tokenizer's normal function which I want to preserver.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I really want is to use the normal &lt;code&gt;token_pattern&lt;/code&gt;, but apply a secondary screening of those tokens to only include those that have strictly letters in them. How can this be done?&lt;/p&gt;&#xA;" OwnerUserId="36935" LastActivityDate="2020-02-14T07:46:59.677" Title="sklearn CountVectorizer token_pattern -- skip token if pattern match" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29353" PostTypeId="2" ParentId="29348" CreationDate="2018-03-21T07:31:12.247" Score="0" Body="&lt;p&gt;How you pad it (and even whether you do so) would depend on what you expect of the data. This imposes boundary conditions on the data which will induce artifacts in any transform you make. How bad this effect depends on how well geared your data is to accepting a particular padding method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Padding methods include zero padding or a periodic bound.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Padding doesn't have to be done in the time domain. Eg interpolating in the frequency domain and back transforming allows you to extrapolate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your analytics has a finite history (eg FIR filters) then you can isolate time regions where padding is unnecessary and draw comparisons therefrom.&lt;/p&gt;&#xA;" OwnerUserId="49195" LastActivityDate="2018-03-21T07:31:12.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29355" PostTypeId="1" CreationDate="2018-03-21T08:07:36.573" Score="2" ViewCount="3486" Body="&lt;p&gt;How can I draw CNN Architecture like this one here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KVeZi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KVeZi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43959" LastEditorUserId="43959" LastEditDate="2018-03-21T09:17:50.660" LastActivityDate="2018-06-19T11:35:34.177" Title="Drawing 1D CNN architecture" Tags="|deep-learning|graphs|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29356" PostTypeId="2" ParentId="29355" CreationDate="2018-03-21T08:26:08.357" Score="1" Body="&lt;p&gt;You can find a python script over here: &lt;a href=&quot;https://github.com/gwding/draw_convnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/gwding/draw_convnet&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it's helpful &lt;/p&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-03-21T08:26:08.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29360" PostTypeId="2" ParentId="29347" CreationDate="2018-03-21T09:35:57.903" Score="1" Body="&lt;p&gt;Usually you can only predict with the variables you have trained on. But in a case like this, I would suggest you check the &lt;a href=&quot;https://en.wikipedia.org/wiki/Multicollinearity&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multicollinearity&lt;/a&gt; of these missing variables with the ones you will have. May be they are already highly correlated with the features you already have. In that case, you can just model using the available features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is not the case and the missing variable looks like a significant variable in your regression, then you might need to pick some other variables which can help you indirectly derive the missing features. For example may be category helps you derive the rating with some accuracy.&lt;/p&gt;&#xA;" OwnerUserId="16368" LastActivityDate="2018-03-21T09:35:57.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29361" PostTypeId="1" CreationDate="2018-03-21T09:43:42.060" Score="1" ViewCount="207" Body="&lt;p&gt;I want to identify subtle patterns in images using a convolutional neural net. I have seen several examples where people gave up reasoning that the pattern is not dominant or consistent enough to be picked up by a neural network. Since tuning these networks is very time consuming I wonder: What are methods to validate that the problem is solvable/the pattern can be recognised (or at least give an indication of the same)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background: My problem&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to identify subtle differences in products based on their images. I want to break down a product image into a large variety of describing attributes. Now I need to find out which product attributes are well suited for identification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I want to distinguish collar types in clothing images. Therefore I need to figure out if the human understanding of collar types (waterfall, V-shaped, round etc.) are distinct and consistent enough to be properly identified by a neural net with human level performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course images always differ a little and the solvability is clearly related to the data available. But before collecting and cleaning hundreds of thousands of images I would like to find out if results will be any good.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-03-21T16:13:49.020" LastActivityDate="2018-03-21T18:09:39.007" Title="How to determine if problem is solvable by neural network?" Tags="|neural-network|convolution|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29362" PostTypeId="1" AcceptedAnswerId="29394" CreationDate="2018-03-21T09:49:16.453" Score="3" ViewCount="54" Body="&lt;p&gt;I am new to Machine learning and trying to learn by practicing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a situation where I am reading a set of N data. Each of N data will have independent state at any moment of time. I want to use these N data to find a relation between them. In case if I am not receiving one or more sets of data, I should be able to use available relation between N data to predict the missing data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of machine learning algorithm can be used?&lt;/p&gt;&#xA;" OwnerUserId="24532" LastEditorUserId="29781" LastEditDate="2018-03-22T01:14:23.977" LastActivityDate="2019-04-18T10:00:46.963" Title="How to find relation between N components and predict the value of any one component using the predicted relation?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29364" PostTypeId="1" CreationDate="2018-03-21T10:25:58.073" Score="0" ViewCount="1637" Body="&lt;p&gt;I'm creating a deep network with 3 hidden layers for classification of the MNIST dataset. The network architecture is something like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;784 input nodes -&amp;gt; 150 hidden nodes (layer 1) -&amp;gt; 125 hidden nodes (layer 2) -&amp;gt; 100 hidden nodes (layer 3) -&amp;gt; 10 output nodes (1 for each possible digit)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For the output layer, I need to decide on an &lt;strong&gt;activation function&lt;/strong&gt;. I've previously used the &lt;strong&gt;sigmoid&lt;/strong&gt; activation function, but it seems many people are using &lt;strong&gt;softmax&lt;/strong&gt; in this particular implementation. I'd like to know which is best suited for the task and why.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks&lt;/p&gt;&#xA;" OwnerUserId="46413" LastActivityDate="2018-03-21T12:28:22.977" Title="Softmax vs Sigmoid in RBM/Auto Encoder final layer" Tags="|autoencoder|rbm|mnist|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29365" PostTypeId="2" ParentId="29285" CreationDate="2018-03-21T11:04:50.273" Score="1" Body="&lt;p&gt;One reason might be “exploding gradients”. Although your loss function seems to output quite stable values, it can perhaps be relevant to investigate your gradients and see how they change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe this blogpost can help you out: &lt;a href=&quot;https://machinelearningmastery.com/exploding-gradients-in-neural-networks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;machinelearningmastery&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47739" LastEditorUserId="47739" LastEditDate="2018-03-21T11:14:01.207" LastActivityDate="2018-03-21T11:14:01.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29368" PostTypeId="1" CreationDate="2018-03-21T12:24:52.127" Score="0" ViewCount="28" Body="&lt;p&gt;I have two time series representing scores, lets call it score A - score B, &#xA;score A is related to a company and it is observed every year from 1990 (about 27 observation)&#xA;score B is related to a product  and it is observed  every quarter from 2016 &#xA;I would like to check if changes in score A influence change in the score B, but I don't know where to start.&lt;/p&gt;&#xA;" OwnerUserId="49154" LastActivityDate="2018-07-19T17:02:55.050" Title="Check if change in time serie influence change in another time serie" Tags="|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29369" PostTypeId="2" ParentId="29364" CreationDate="2018-03-21T12:28:22.977" Score="2" Body="&lt;p&gt;This choice mainly depends on what your output represents. Given a vector $\mathbf{x}$, the sigmoid function is given by&#xA;$$ \sigma(x_i) = \frac{\exp(x_i)}{1 + \exp(x_i)} $$&#xA;while the softmax is given by&#xA;$$ \mathrm{softmax}(x_i) = \frac{\exp(x_i)}{\sum_{j=0}^N \exp(x_j)}$$&#xA;A key difference is that the output of the sigmoid function applied to $x_i$ only depends on $x_i$. The output of the softmax function depends on &lt;em&gt;all&lt;/em&gt; elements of the vector $\mathbf{x}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sigmoid will squash each $x_i$ into the range $(0, 1)$, which enables you to interpret $\sigma(x_i)$ as the probability of $x_i$.&#xA;But, if you have multiple classes, e.g. 0-9 in MNIST, each probability is independent, that means you could have probability $p=0.9$ for the digit 1 and $p=0.5$ for the digit 3. This is undesirable if you want to distinguish between multiple classes. However, if multiple classes can appear at the same time, then sigmoid is well suited.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, the softmax is basically a sigmoid function which is normalized such that $\sum_{j=0}^N \mathrm{softmax}(x_j) = 1$. This means that the output of a softmax layer is a valid probability mass function, i.e. the output of your neural network is the probability of the input belonging to a certain class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if you want to classify between e.g. the 10 digits as in MNIST, then softmax is the way to go. If the classes are independent, i.e. it is possible that an image shows a cat &lt;em&gt;and&lt;/em&gt; a dog at the same time, then sigmoid is the way to go.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2018-03-21T12:28:22.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29370" PostTypeId="2" ParentId="29368" CreationDate="2018-03-21T12:41:43.017" Score="1" Body="&lt;p&gt;Avoiding the fine details, what you are looking for is a cross correlation. This will give you whether there's any influence (whether it be correlated or anticorrelated) and a time lag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep in mind the arrow of time (effect doesn't precede a cause) and remember that correlation does not imply causality. You will want to get a large enough data set that you can perform an analysis of variation to determine how reliable your conclusion is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As your data set for B is miniscule, I would seriously doubt any meaningful conclusions can be drawn. When it comes to statistics, as Stalin has ashamedly said in his approach to warfare, quantity is a quality in and of itself.&lt;/p&gt;&#xA;" OwnerUserId="49195" LastActivityDate="2018-03-21T12:41:43.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29372" PostTypeId="1" AcceptedAnswerId="29374" CreationDate="2018-03-21T14:02:19.583" Score="0" ViewCount="56" Body="&lt;p&gt;This piece of code should be self-explanatory and it's also a choke-point in my process. Any way to do this better?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;geoset['MonthInt'] = pd.to_datetime(geoset['Date received']).dt.month # Add a month&#xA;geoset['YearInt'] = pd.to_datetime(geoset['Date received']).dt.year # Add a year&#xA;geoset['DayInt'] = pd.to_datetime(geoset['Date received']).dt.day # Add a day of month&#xA;geoset['DayOfWeekInt'] = pd.to_datetime(geoset['Date received']).dt.dayofweek # Add a day of week&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-03-22T02:22:15.537" Title="Better way to break out a date?" Tags="|python|pandas|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29374" PostTypeId="2" ParentId="29372" CreationDate="2018-03-21T14:12:05.400" Score="3" Body="&lt;p&gt;Obvious optimisation would be converting 'Date received' to pandas datetime once, for example (if you need to retain original column):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;geoset['pddate_received'] = pd.to_datetime(geoset['Date received'])&#xA;geoset['MonthInt'] = geoset['pddate_received'].dt.month # Add a month&#xA;geoset['YearInt'] = geoset['pddate_received'].dt.year # Add a year&#xA;geoset['DayInt'] = geoset['pddate_received'].dt.day # Add a day of month&#xA;geoset['DayOfWeekInt'] = geoset['pddate_received'].dt.dayofweek # Add a day of week&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="2573" LastEditorUserId="29575" LastEditDate="2018-03-22T02:22:15.537" LastActivityDate="2018-03-22T02:22:15.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29378" PostTypeId="2" ParentId="8162" CreationDate="2018-03-21T15:22:40.173" Score="1" Body="&lt;p&gt;Note that in the linked presentation, on the slide titled &quot;Plotting the effects&quot;, the &lt;code&gt;treat&lt;/code&gt; object has only 2 rows.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, because &lt;code&gt;frame&lt;/code&gt; has 57k rows, &lt;code&gt;fitObjGrouped&lt;/code&gt; has predictions for each row of &lt;code&gt;newdata&lt;/code&gt;.  You can verify this with &lt;code&gt;fitObjGrouped$n&lt;/code&gt;.  To fix the problem, try:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;frame &amp;lt;- data.frame(group = unique(dataset$group))&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="49217" LastEditorUserId="49217" LastEditDate="2018-03-21T16:29:05.997" LastActivityDate="2018-03-21T16:29:05.997" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29379" PostTypeId="2" ParentId="28509" CreationDate="2018-03-21T16:00:53.700" Score="2" Body="&lt;p&gt;In this kind of problems, one needs to be sure what model he/she wants to use. A convolution network would be great with final softmax layer giving 8 outputs. you don't need a bidirectional layer in your network architecture. Try a simple conv network and make sure your data pipeline is working then you can add more layers and make it complex. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-03-21T16:00:53.700" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29381" PostTypeId="2" ParentId="29269" CreationDate="2018-03-21T16:21:24.937" Score="2" Body="&lt;p&gt;Is &lt;code&gt;Outcome&lt;/code&gt; also a boolean variable?  If so, a simple &lt;code&gt;prop.test&lt;/code&gt; will do.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's a toy dataset where a judge from the same country is less likely to give a guilty verdict.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse)&#xA;n&amp;lt;-1000&#xA;dataset&amp;lt;-tibble(country_judge = sample(c(TRUE,FALSE), n, &#xA;                                           replace=T, prob=c(0.2,0.8))) %&amp;gt;%&#xA;  mutate(outcome = ifelse(country_judge,&#xA;                          sample(c(&quot;Guilty&quot;, &quot;Innocent&quot;), n, &#xA;                                     replace=T, prob=c(0.4,0.6)),&#xA;                          sample(c(&quot;Guilty&quot;, &quot;Innocent&quot;), n, &#xA;                                     replace=T, prob=c(0.5,0.5))))&#xA;&#xA;dataset %&amp;gt;%&#xA;  group_by(country_judge) %&amp;gt;%&#xA;  summarise(p_guilty=mean(outcome==&quot;Guilty&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# A tibble: 2 x 2&#xA;  country_judge  p_guilty&#xA;          &amp;lt;lgl&amp;gt;     &amp;lt;dbl&amp;gt;&#xA;1         FALSE 0.5108835&#xA;2          TRUE 0.3698630&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, pull out vectors of trials, and &quot;successes&quot;, and feed those into &lt;code&gt;prop.test&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;trials &amp;lt;- dataset %&amp;gt;%&#xA;  group_by(country_judge) %&amp;gt;%&#xA;  count() %&amp;gt;%&#xA;  pull(n)&#xA;&#xA;successes &amp;lt;- dataset %&amp;gt;%&#xA;  filter(outcome==&quot;Guilty&quot;) %&amp;gt;%&#xA;  group_by(country_judge) %&amp;gt;%&#xA;  count() %&amp;gt;%&#xA;  pull(n)&#xA;&#xA;prop.test(successes, trials)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which gives something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    2-sample test for equality of proportions with continuity correction&#xA;&#xA;data:  successes out of trials&#xA;X-squared = 13.068, df = 1, p-value = 0.0003003&#xA;alternative hypothesis: two.sided&#xA;95 percent confidence interval:&#xA; 0.06517776 0.21686317&#xA;sample estimates:&#xA;   prop 1    prop 2 &#xA;0.5108835 0.3698630 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49217" LastActivityDate="2018-03-21T16:21:24.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29382" PostTypeId="2" ParentId="29352" CreationDate="2018-03-21T16:25:23.597" Score="1" Body="&lt;p&gt;Found &lt;a href=&quot;https://stackoverflow.com/questions/29375632/regular-expression-for-match-all-words-without-numbers?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&quot;&gt;this SO post&lt;/a&gt; which says to use the following regex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;\b[^\d\W]+\b/g&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;yielding the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; vec = CountVectorizer(token_pattern=r'\b[^\d\W]+\b')&#xA;&amp;gt;&amp;gt;&amp;gt; X = vec.fit_transform(docs)&#xA;&amp;gt;&amp;gt;&amp;gt; pd.DataFrame(X.toarray(), columns=vec.get_feature_names())&#xA;   aaa  is  more  some  text  this&#xA;0    0   1     0     1     1     1&#xA;1    0   0     0     0     0     0&#xA;2    1   0     1     0     0     0&#xA;3    0   0     0     0     0     0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I needed in my regex were the &lt;code&gt;\b&lt;/code&gt; word boundary characters of which I was not aware of. That &lt;em&gt;does&lt;/em&gt; make this a misplaced question as it has nothing to do with data science or that discipline's tools (sklearn). &lt;/p&gt;&#xA;" OwnerUserId="36935" LastActivityDate="2018-03-21T16:25:23.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29383" PostTypeId="2" ParentId="29361" CreationDate="2018-03-21T16:31:27.290" Score="2" Body="&lt;p&gt;Neural network will pick up on any patterns as long as your environment is not fully stochastic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what authors of Using Recurrent Neural Networks to Frecasting of Forex used to determine the amount of stochasticity:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/ftp/cond-mat/papers/0304/0304469.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;page 7, bottom. &quot;Hurst Exponent&quot;.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastActivityDate="2018-03-21T16:31:27.290" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29384" PostTypeId="2" ParentId="29292" CreationDate="2018-03-21T17:15:53.083" Score="-1" Body="&lt;p&gt;Ok it seems i calculated the outputs wrongly. It was not calculated fairly across the entire data-set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am getting better results after improving the output calculations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZzsSO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZzsSO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It still can be improved, but it's a great start.&lt;/p&gt;&#xA;" OwnerUserId="49114" LastActivityDate="2018-03-21T17:15:53.083" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29388" PostTypeId="2" ParentId="29361" CreationDate="2018-03-21T18:09:39.007" Score="1" Body="&lt;p&gt;It's much easier to identify projects where a neural network won't work than to identify projects where it will work. For example, if you have fewer than 10,000 records (highly subjective rule of thumb), then a neural net will probably overfit and not work. If a human struggles with the labels, then a neural network will probably struggle too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my subjective opinion, your problem sounds like it will be difficult for a neural network (or any model) to learn without a substantial amount of records. You would probably need several hundred thousand at a minimum, not including data augmentations like flipping images, etc, or something on the scale of an ImageNet dataset (i.e., millions). I say that because even as a human being I don't know what those collar types are that you described. On a dataset in the low tens of thousands you could probably train a convnet to identify shirts vs pants, but I highly doubt it could accurately detect collar types. &lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2018-03-21T18:09:39.007" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29391" PostTypeId="1" CreationDate="2018-03-21T18:17:54.100" Score="5" ViewCount="3513" Body="&lt;p&gt;I would like to detect defects/anomalies in images. Due to the lack of images with anomalies, I try to solve the problem in an unsupervised manner. Until now, I trained a variational autoencoder together with an generative adversarial network with “good” images. Now, I have an encoder/decoder network which is able to produce images without anomalies. Here are two examples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;original image | cropped image &amp;amp; input of the encoder | output of the decoder&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kGBNn.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kGBNn.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XucZc.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XucZc.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In those images, if the object is rotated (not vertical), then it is an anomaly (like the second image). To measure the difference between the input/output of the encoder/decoder network, I tried the structural similarity metric SSIM. However, the result is not satisfying enough as many images without an anomaly also have a low similarity value. Also the pixel-wise MAE or MSE metrics are not good as they do not look specifically on the structure and are very sensitive to the brightness. Here is the result with the SSIM metric (green are the good examples, red are the anomalies):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0ZXKl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0ZXKl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to avoid is to add specific rules for the detection, in this example to detect vertical lines because there are many other objects with different anomalies which I would like to detect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you have a suggestion for another image similarity metric? Am I on the right path or are there other/better methods to do unsupervised anomaly detection? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: There are only a few images with anomalies (&amp;lt;10 images), but many without an anomaly..&lt;/p&gt;&#xA;" OwnerUserId="49228" LastEditorUserId="49228" LastEditDate="2018-03-22T22:22:02.960" LastActivityDate="2019-04-08T15:00:50.127" Title="Unsupervised Anomaly Detection in Images" Tags="|machine-learning|neural-network|deep-learning|similarity|anomaly-detection|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29392" PostTypeId="2" ParentId="9454" CreationDate="2018-03-21T18:51:28.020" Score="1" Body="&lt;p&gt;Recoding the NAs to 0's is not what you want. Think of what this means: a missing value on an item for a particular user means that said user hasn't seen or listened to or bought (depending on what kind of data you have) that item, while a zero value means that the user disliked that item; so they have completely different meanings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal of a recommender system is basically to fill those NA values in a way that &quot;makes sense&quot;. It is very common to have sparse matrices in recommender systems. I don't know exactly what kind of approach you're using or why you're using a perceptron, but I would recommend you to check how matrix factorization for recommender systems works (&quot;Matrix factorization for recommender systems&quot; by Robert Bell, Chris Volinsky and Yehuda Koren is a good paper) and see if this approach works for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also try first a simple baseline model that takes into account the biases of the users and the biases of the items; Bell, Volinsky and Koren's paper is a good start.&lt;/p&gt;&#xA;" OwnerUserId="49109" LastActivityDate="2018-03-21T18:51:28.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29394" PostTypeId="2" ParentId="29362" CreationDate="2018-03-21T19:19:10.257" Score="2" Body="&lt;p&gt;I understand your task can be formulated as an &lt;em&gt;imputation&lt;/em&gt; task, see the link below for further details. Also before you proceed look up the concepts of &lt;em&gt;missing data&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to fill in missing data you’ll have to build N distinct models using available observations. Each of N models will have N-1 inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that because more than one component can be missing your models should be able to handle the missing data. According to my knowledge neural networks cannot handle the missing data out of the box. Tree based models, such that random forests or GBMs are more suitable for this kind of tasks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.stat.columbia.edu/~gelman/arm/missing.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.stat.columbia.edu/~gelman/arm/missing.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29781" LastActivityDate="2018-03-21T19:19:10.257" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29396" PostTypeId="1" CreationDate="2018-03-21T20:55:08.857" Score="0" ViewCount="143" Body="&lt;p&gt;Can a dataset with only  binary digits 0's and 1's perform good classification such as decision tree and naive Bayes&lt;/p&gt;&#xA;" OwnerUserId="49003" LastActivityDate="2018-03-21T23:29:25.530" Title="Performing Binary classification using binary dataset" Tags="|data-mining|dataset|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29397" PostTypeId="2" ParentId="9858" CreationDate="2018-03-21T20:58:34.353" Score="1" Body="&lt;p&gt;H-W's algorithm, from the 1979 paper, takes as input initial clusters. However, the authors suggest a method for obtaining them in their last section. They write that &lt;em&gt;it is guaranteed that no cluster will be empty after the initial assignment in the subroutine&lt;/em&gt;. It goes as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute the overall mean $\bar{x}$.&lt;/li&gt;&#xA;&lt;li&gt;Order the observations with respect to their distance to $\bar{x}$, that is $||x_i - \bar{x}||_2$ (in ascending order I guess?).&lt;/li&gt;&#xA;&lt;li&gt;Take the points in position $\{ 1 + (L-1) [M/K] \}$, where $L=1, \dots, K$, as initial centroids. ($[\ \cdot\ ]$ most probably refer to the floor function, hence the $1$ at the beginning.)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As for the main algorithm, it is described in a paper called &lt;em&gt;Hartigan's K-Means Versus Lloyd's K-Means-Is It Time for a Change?&lt;/em&gt; by N Slonim, E Aharoni, K Crammer, published in 2013 by &lt;em&gt;AJCAI&lt;/em&gt;. Note that this version simply uses a random initial partition. It goes as follows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the vectors $x \in \mathcal{X}$ and a target number of clusters $K$,&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Set $\mathcal{C}$ to be a random partition of $\mathcal{X}$ into $K$ clusters and compute the centroid vector associated to each $C \in \mathcal{C}$, denote them $v_C$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Scan $\mathcal{X}$ in a random order, and for all $x \in \mathcal{X}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1 Set the stopping indicator to $s = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.1. Tentatively remove $x$ from its cluster $C$, letting $C^{-} = C \setminus \{ x \}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.2 Find&#xA;\begin{align*}&#xA;C^+ = \Big\{ \mathrm{argmin}_{C^* \in (\mathcal{C} \setminus C) \cup C^{-}}\ \frac{1}{n} d(x,v_C^*) + \frac{1}{n} \sum_{y \in C^*} [d(y,v_{C^* \cup x}) - d(y,v_{C^*})] \Big\} \cup \{ x \} &#xA;\end{align*}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2.3 If $C^{+} \neq C$, then reset $C \leftarrow C^{-}$ and $C^* \leftarrow C^{+}$ and update their respective vectors $v_{C}$ and $v_{C^*}$. Also reset $s \leftarrow 0$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If $s=0$, go back to 2.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Here I make a slight abuse of notation letting $C^*$ be the solution of the $\mathrm{argmin}$ in 2.2. This step, 2.2, simply computes the change in the loss obtained if $x$ is added to $C^*$, as opposed to it staying alone in its own cluster (once removed). $d$ stands for distance. Finally, the particular ways to compute the updated vectors $v_C$, $v_{C^* \cup \{x \}}$ and so on is provided in the paper cited above. They can be done in linear time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the answers to all your questions are implicit in the above algorithm... However, I still have to make sure this implementation of the algorithm is &lt;em&gt;standard&lt;/em&gt;. In particular if it is the one implemented in R. Any comments/edit are welcomed.&lt;/p&gt;&#xA;" OwnerUserId="47308" LastEditorUserId="47308" LastEditDate="2018-03-21T21:25:15.160" LastActivityDate="2018-03-21T21:25:15.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29398" PostTypeId="1" CreationDate="2018-03-21T22:41:23.290" Score="1" ViewCount="1136" Body="&lt;p&gt;So I understand the general idea of how isolation forests works, but I'm having trouble understanding how the model makes predictions on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it pass every new point (separately) through every tree in the trained model, and then runs the exact same splits with the exact same original subset data (plus this single new point), to determine the number of steps (anomaly score) until this new point gets isolated? &#xA;Then this score gets compared back to the anomaly score threshold that was set when the model was trained?&lt;/p&gt;&#xA;" OwnerUserId="49242" LastActivityDate="2019-12-10T13:01:05.137" Title="Isolation Forest Prediction Mechanics: Does it compare value with every tree (and the original training subset)?" Tags="|random-forest|decision-trees|anomaly-detection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29399" PostTypeId="2" ParentId="29396" CreationDate="2018-03-21T23:29:25.530" Score="1" Body="&lt;p&gt;Yes. Many data sets are one-hot encoded into only 0s and 1s because some ML algorithms work best that way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2018-03-21T23:29:25.530" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29401" PostTypeId="1" CreationDate="2018-03-22T00:14:52.130" Score="5" ViewCount="1364" Body="&lt;p&gt;In search, position of the search result affects the click-through rate a great deal. How do people usually deal with this ? In practice how to remove such bias to create unbiased training data for training learning to rank model ?&lt;/p&gt;&#xA;" OwnerUserId="44064" LastActivityDate="2020-03-09T21:02:00.257" Title="How to deal with position bias in search?" Tags="|machine-learning|recommender-system|search|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29404" PostTypeId="1" CreationDate="2018-03-22T07:04:15.063" Score="2" ViewCount="132" Body="&lt;p&gt;I recently developed a DNN model and I want to know what exactly is training time and what all steps are included in it? For ex I carried out the following steps&#xA;1) Determined best Network Architecture by 5 fold CV ( using Back-propagation)&#xA;2) Once I got the Best architecture I used unsupervised learning to get initial weights using unlabled data&#xA;3) I applied back-propagation to fine tune the network with initial weights given by step 2 (It ran for 500 epochs)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the training time in this procedure? Does it include all the steps or only steps 2 or 3? Or in general terms does the training time include time to run the code for the best architecture or it also includes the time taken to determine the best network architecture?&lt;/p&gt;&#xA;" OwnerUserId="49259" LastActivityDate="2021-09-27T01:03:30.260" Title="What does the training time for a Neural Network include?" Tags="|deep-learning|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29408" PostTypeId="1" CreationDate="2018-03-22T08:50:57.537" Score="1" ViewCount="854" Body="&lt;p&gt;I have some observations belonging to groups and I would like to compute the similarity of them within different groups in order to tell which observations, within specific groups, have similar characteristics or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which metrics can be useful to do such things and under which conditions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there are many metrics to compute the similarities between individuals in a group, but there is no threshold to tell whether individuals in a cluster share similar patterns or not.&lt;/p&gt;&#xA;" OwnerUserId="49154" LastEditorUserId="29169" LastEditDate="2019-06-07T05:55:42.227" LastActivityDate="2019-06-07T05:55:42.227" Title="Intra-cluster similarity metric" Tags="|machine-learning|clustering|similarity|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29409" PostTypeId="1" AcceptedAnswerId="29425" CreationDate="2018-03-22T09:02:20.557" Score="1" ViewCount="67" Body="&lt;p&gt;Policy Evaluation is computing the state-value function for an arbitary policy $\pi$.(suton &amp;amp; barto book).&#xA;Now &lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{equation}&#xA;v_{\pi} = E_{\pi}[\;G_{t}\,|\;S_{t}=s] \qquad\qquad\qquad\qquad(4.1)\\\&#xA;        =  E_{\pi}[R_{t+1}+\gamma G_{t+1} |S_{t}=s]\qquad\qquad\quad(4.2)\\&#xA;        =  E_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s]\qquad\qquad(4.3)\\&#xA;        = \sum_{a} \pi(a|s) \sum_{s^{'},r} p(s^{'},r|s,a)[r+\gamma v_{\pi}(s^{'})]\quad(4.4)&#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone help me to figure out from (4.3) to (4.4). And If equation (4.4) is bellman's equation for state value function, then how is it different from below equation&#xA;\begin{equation}&#xA;v_{\pi} = \sum_{a\in A} \pi(a|s)[R_{s}^{a}+\gamma \sum_{s'\in S} P_{ss^{'}}^{a}v_{\pi}({s^{'})}] &#xA;\end{equation} &lt;/p&gt;&#xA;" OwnerUserId="2893" LastActivityDate="2018-03-25T08:27:30.347" Title="Need help in deriving Policy Evaluation (Prediction)" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29410" PostTypeId="1" AcceptedAnswerId="29426" CreationDate="2018-03-22T10:08:10.827" Score="2" ViewCount="17014" Body="&lt;p&gt;I want to use scikit-learn's GridSearchCV to optimise a BaggingClassifier that uses a support vector classifier (SVC). I want the grid search to search over parameters for both the BaggingClassifier and the SVC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried this setup:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;svc_pipe = Pipeline([&#xA;     ('svc', SVC(probability=True)),&#xA;])&#xA;pipe = Pipeline([&#xA;     ('bag', BaggingClassifier(svc_pipe, no_estimators=50)),&#xA;])&#xA;&#xA;params = {&#xA;    'bag__bootstrap_features' : [True, False],&#xA;    'bag__svc__kernel': ['linear', 'rbf'],&#xA;    'bag__svc__decision_function_shape': ['ovo', 'ovr']&#xA;}&#xA;&#xA;rnd_search = GridSearchCV(pipe, param_grid=params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I get this error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: Invalid parameter svc for estimator BaggingClassifier(base_estimator=Pipeline(memory=None,&#xA;     steps=[('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,&#xA;  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',&#xA;  max_iter=-1, probability=True, shrinking=True,&#xA;  tol=0.001, verbose=False))]),&#xA;         bootstrap=True, bootstrap_features=True, max_features=1.0,&#xA;         max_samples=1.0, n_estimators=50, n_jobs=-1, oob_score=False,&#xA;         verbose=0, warm_start=False). Check the list of available parameters with `estimator.get_params().keys()`.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone show me what I have done wrong?&lt;/p&gt;&#xA;" OwnerUserId="47380" LastActivityDate="2018-03-22T15:01:57.093" Title="How to set parameters to search in scikit-learn GridSearchCV" Tags="|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29411" PostTypeId="1" AcceptedAnswerId="29416" CreationDate="2018-03-22T10:12:13.213" Score="2" ViewCount="2422" Body="&lt;p&gt;What are some techniques that I can use for anomaly detection given a non-Normal distribution? I have less than twenty available observations.&lt;/p&gt;&#xA;" OwnerUserId="49154" LastEditorUserId="29169" LastEditDate="2019-06-07T04:31:22.280" LastActivityDate="2019-06-07T04:31:22.280" Title="Outlier detection with non normal distribution" Tags="|anomaly-detection|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="29412" PostTypeId="2" ParentId="13669" CreationDate="2018-03-22T10:22:44.890" Score="17" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/27994137/how-to-save-the-output-of-a-cell-in-ipython-notebook&quot;&gt;See this stack question&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use %%capture Jupyter notebook' magic command to catch output of cell and then paste it to your text file with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with open('output.txt', 'w') as out:&#xA;   out.write(cap.stdout)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if you want to cell code to specific file for example code.txt you can use magic function %%writefile&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%%writefile example.txt&#xA;ab = 'This is code'&#xA;a = 5&#xA;print(a+ 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also if you want to append to file you must use -a parameter&lt;/p&gt;&#xA;" OwnerDisplayName="user38487" LastEditorDisplayName="user38487" LastEditDate="2018-05-11T10:45:59.983" LastActivityDate="2018-05-11T10:45:59.983" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="29415" PostTypeId="1" CreationDate="2018-03-22T10:31:28.620" Score="0" ViewCount="541" Body="&lt;p&gt;I've built a feedforward net that predicts 2 classes (0,1). I've used the validation_split attribute like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;history = model.fit(X, Y, validation_split=validation_split_ratio, epochs=700, batch_size=10, verbose=1, callbacks=[tb, checkpoint]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I got strange results (validation performs better than the train data):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DhIKZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DhIKZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/joE82.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/joE82.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then took the validation data used for the training (20% from the last samples in my dataset) and tested the model to build a confusion matrix:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3rdRA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3rdRA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I trying to build a confusion matrix from new unseen data, my model performs poorly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/la41i.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/la41i.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why is that?&lt;/strong&gt; I thought the validation data used by Keras is also &quot;unseen&quot; by the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE Visualization code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Dense, Dropout&#xA;from keras.models import Sequential&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA;from sklearn.metrics import classification_report, confusion_matrix&#xA;import itertools&#xA;&#xA;def create_model():&#xA;    ## Create our model&#xA;    model = Sequential()&#xA;    # 0.81250&#xA;&#xA;    model.add(Dense(44, input_dim=18, kernel_initializer='uniform', activation='relu'))&#xA;    model.add(Dropout(0.8))&#xA;    model.add(Dense(44, activation='relu'))&#xA;    model.add(Dropout(0.4))&#xA;    model.add(Dense(3, activation='relu'))&#xA;    model.add(Dropout(0.2))&#xA;    model.add(Dense(1, activation='sigmoid'))&#xA;    return model&#xA;&#xA;&#xA;def load_trained_model(weights_path):&#xA;    model = create_model()&#xA;    model.load_weights(weights_path)&#xA;    return model&#xA;&#xA;def plot_confusion_matrix(cm, classes,&#xA;                          normalize=False,&#xA;                          title='Confusion matrix',&#xA;                          cmap=plt.cm.Blues):&#xA;    &quot;&quot;&quot;&#xA;    This function prints and plots the confusion matrix.&#xA;    Normalization can be applied by setting `normalize=True`.&#xA;    &quot;&quot;&quot;&#xA;    if normalize:&#xA;        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]&#xA;        print(&quot;Normalized confusion matrix&quot;)&#xA;    else:&#xA;        print('Confusion matrix, without normalization')&#xA;&#xA;    print(cm)&#xA;&#xA;    plt.imshow(cm, interpolation='nearest', cmap=cmap)&#xA;    plt.title(title)&#xA;    plt.colorbar()&#xA;    tick_marks = np.arange(len(classes))&#xA;    plt.xticks(tick_marks, classes, rotation=45)&#xA;    plt.yticks(tick_marks, classes)&#xA;&#xA;    fmt = '.2f' if normalize else 'd'&#xA;    thresh = cm.max() / 2.&#xA;    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):&#xA;        plt.text(j, i, format(cm[i, j], fmt),&#xA;                 horizontalalignment=&quot;center&quot;,&#xA;                 color=&quot;white&quot; if cm[i, j] &amp;gt; thresh else &quot;black&quot;)&#xA;&#xA;    plt.tight_layout()&#xA;    plt.ylabel('True label')&#xA;    plt.xlabel('Predicted label')&#xA;&#xA;&#xA;dataset = np.loadtxt(&quot;./validation/validation.csv&quot;, skiprows=1, delimiter=&quot;,&quot;)&#xA;model = load_trained_model('./checkpoints/weights.best.hdf5')&#xA;X = dataset[:, :18]&#xA;Y = dataset[:, [18]]&#xA;&#xA;&#xA;Y_pred = model.predict(X)&#xA;print(Y_pred)&#xA;&#xA;y_pred = model.predict_classes(X)&#xA;print(y_pred)&#xA;&#xA;p = model.predict_proba(X)  # to predict probability&#xA;&#xA;target_names = ['class 0','class 1']&#xA;print(classification_report(Y, y_pred, target_names=target_names))&#xA;cnf_matrix = confusion_matrix(Y, y_pred)&#xA;print(cnf_matrix)&#xA;plt.figure()&#xA;plot_confusion_matrix(cnf_matrix, classes=target_names,normalize=True,&#xA;                      title='Confusion matrix, without normalization')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dataset sample:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0,1,0,0,0,0,0,1,0,0,0,5,1,6.01,4.86,43.0,1,2,1&#xA;1,0,0,0,0,0,0,0,1,0,117,3,1,5.02,4.86,54.0,0,2,1&#xA;0,1,0,0,0,0,1,0,0,0,1,2,1,6.01,4.86,70.0,1,2,0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Last column is the label (1 or 0)&lt;/p&gt;&#xA;" OwnerUserId="25696" LastEditorUserId="25696" LastEditDate="2018-03-22T15:14:00.650" LastActivityDate="2018-03-22T15:14:00.650" Title="Keras - Why Validation data produce good results, while unseen data is performing poorly" Tags="|machine-learning|neural-network|keras|overfitting|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="29416" PostTypeId="2" ParentId="29411" CreationDate="2018-03-22T10:51:24.143" Score="2" Body="&lt;p&gt;I would suggest a nearest neighbors approach. This technique is non-parametric, such that it does not assume your features follow any given distribution. The degree from which a novel instance can be classified as anomalous can set through some p-value estimation. These techniques are computationally expensive however due to your small dataset this may be well suited. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Check out:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Learning Minimum Volume Sets&#xA;&lt;a href=&quot;http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anomaly Detection with Score functions based on Nearest Neighbor Graphs&#xA;&lt;a href=&quot;https://arxiv.org/abs/0910.5461&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/0910.5461&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New statistic in P-value estimation for anomaly detection&#xA;&lt;a href=&quot;http://ieeexplore.ieee.org/document/6319713/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ieeexplore.ieee.org/document/6319713/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-22T10:51:24.143" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29418" PostTypeId="1" CreationDate="2018-03-22T11:40:33.957" Score="2" ViewCount="777" Body="&lt;p&gt;Is there a threshold where it is computationally more efficient than one hot encoding to create separate keras embedding layers for each categorical feature &gt; than x categories?&lt;/p&gt;&#xA;" OwnerUserId="49274" LastActivityDate="2018-03-22T11:40:33.957" Title="Embedding layers for categorical features" Tags="|keras|embeddings|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29419" PostTypeId="2" ParentId="29408" CreationDate="2018-03-22T11:41:54.537" Score="0" Body="&lt;p&gt;This seems to me like a hypothesis testing problem where the null hypothesis can be formulated as &quot;two data sets are drawn from the same population distribution function&quot;. If we can disprove this statement with sufficient certainty then we can assume that they were in fact drawn from different populations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can bin your features into ranges and then apply the &lt;strong&gt;Chi-Squared Test&lt;/strong&gt; defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\chi^2 = \sum_i^n \frac{(O_i-E_i)^2}{E_i}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $i$ is each bin you have for a feature, $O_i$ is the number of observations you have for the $i^{th}$ bin, $E_i$ is the expected number of observations. $E_i = Np_i$, where $N$ is your number of observations and $p_i$ is the probability of obtaining bin $i$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example assume a simple coin flip experiment where we want to determine if a coin is fair. The baseline distribution should be uniform 50 heads, 50 tails. We will conduct a chi-squared test to see if our coin is fair. When flipping the coin we observed 30 heads and 70 tails. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\chi^2 = \frac{(30 - 100*0.5)^2}{100*0.5} + \frac{(70-100*0.5)^2}{100*0.5} = 16$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This example has 1 degree of freedom. There are two possible outcomes, $r = (n-1)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we can look at our trusty chi-squared table. The rows is usually for the degrees of freedom $\nu$, which is 1 in our example. And the columns is for the level of significance, for a significant result look under $p&amp;lt;0.05$ for a highly significant result look under $p&amp;lt;0.01$. We can see that our calculated $\chi^2$ result is much higher than the value for a $p&amp;lt;0.01$ thus we can say witth certainty that the distribution of our Group 2 is not the same as that of Group 1. They are statistically different. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Alternatively, you can get the probability distribution function for each variable and get the overlapping area between two groups. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-22T11:41:54.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29421" PostTypeId="2" ParentId="29276" CreationDate="2018-03-22T13:18:06.487" Score="2" Body="&lt;p&gt;Ok, it makes sense if we don't think about the formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at the bottom half of the following slide, taken from &lt;a href=&quot;https://www.slideshare.net/KihoSuh/neural-turing-machine-77087607&quot; rel=&quot;nofollow noreferrer&quot;&gt;Page 20, &quot;Neural Turing Machine&quot; side-presentation by Kiho Suh; June 19, 2017&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sL59d.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sL59d.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-08-13T13:47:10.393" LastActivityDate="2018-08-13T13:47:10.393" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29423" PostTypeId="2" ParentId="29037" CreationDate="2018-03-22T13:55:53.367" Score="2" Body="&lt;p&gt;Variable selection in linear regression is based on partial correlations, not zero order correlations. The partial correlation is the one resulting after a conditioning variable effect is removed from both X and Y. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While a variable may be non highly correlated with Y, it may be highly correlated with an X variable in such a way that the X variable after being partialled out of the conditioning effect becomes highly significant. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This effect is known as suppression among other names. &#xA;HTH&lt;/p&gt;&#xA;" OwnerUserId="49284" LastActivityDate="2018-03-22T13:55:53.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29425" PostTypeId="2" ParentId="29409" CreationDate="2018-03-22T14:12:35.320" Score="2" Body="&lt;p&gt;Going from step (4.3) to (4.4) is turning the expectation (dependent on following policy $\pi$) into a more concrete calculation. To do this, you must resolve your random variables ($R_{t+1}$ and $S_{t+1}$) into the specific values from the finite sets of $\mathcal{R}$ and $\mathcal{S}^+$ - where individual set members are noted $r$ and $s$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recall that $\pi(a|s)$ is the probability of taking action $a$ given state $s$, so by taking a sum of any function depending on $(s,a)$ e.g. $\sum_{a} \pi(a|s)F(s,a)$ you will get the &lt;em&gt;expected&lt;/em&gt; value of $F(s,a)$ starting from state $s$ and following policy $\pi$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you could write (4.3b) as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v(s) = \sum_{a} \pi(a|s)\mathbb{E}_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s, A_t=a]\qquad\qquad(4.3b)\\$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, now you have written the expectation when given $s$ and $a$ (so you have all the possible cases of $a$ available to work with), you can use the transition and reward probabilities to fully resolve the expectation and substitute the random variables ($R_{t+1}$ and $S_{t+1}$) for their MDP model distributions described in $p(s',r|s,a)$, leading to equation (4.4). E.g.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v(s) = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)\mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1})| S_{t+1}=s', R_{t+1}=r]\qquad\qquad(4.3c)\\$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\qquad = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)(r + \gamma v_{\pi}(s'))\qquad\qquad(4.4)\\$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your last equation is a slightly different formulation of the same result, where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$R_s^a$ is the &lt;em&gt;expected&lt;/em&gt; reward when taking action $a$ in state $s$. It is equal to $\sum_{s',r} p(s',r|s,a)r$. It is also equal to $\mathbb{E}[R_{t+1}|S_t=s, A_t=a]$, by definition and independently of $\pi$.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$P_{ss'}^a$ is the state transition probability - the probability of ending up in state $s'$ when taking action $a$ in state $s$. It is equal to $\sum_r p(s',r|s,a)$ given $s'$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Technically using $R_s^a$ loses some information about the dynamics of the underlying MDP. Although not anything important to reinforcement learning, which deals with maximising expected reward, so in some ways it is more convenient to already characterise the MDP with expected rewards. &lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-03-25T08:27:30.347" LastActivityDate="2018-03-25T08:27:30.347" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29426" PostTypeId="2" ParentId="29410" CreationDate="2018-03-22T15:01:57.093" Score="2" Body="&lt;p&gt;There is a typo in &lt;code&gt;pipe&lt;/code&gt;, &lt;code&gt;no_estimators&lt;/code&gt; should be &lt;code&gt;n_estimators&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address your problem, if you run the following piece of code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for param in rnd_search.get_params().keys():&#xA;    print(param)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will show you how the parameters are passed to different parts of the pipeline, the parameters of interest are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;bag__base_estimator__svc__kernel&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;bag__base_estimator__svc__decision_function_shape&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So you were almost there, you were just missing &lt;code&gt;base_estimator__&lt;/code&gt; in the &lt;code&gt;svc&lt;/code&gt; pipeline parameters. All you need to do is change the &lt;code&gt;svc&lt;/code&gt; parameters like so:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = {&#xA;    'bag__bootstrap_features' : [True, False],&#xA;    'bag__base_estimator__svc__kernel': ['linear', 'rbf'],&#xA;    'bag__base_estimator__svc__decision_function_shape': ['ovo', 'ovr']&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20429" LastActivityDate="2018-03-22T15:01:57.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29427" PostTypeId="1" AcceptedAnswerId="29624" CreationDate="2018-03-22T15:25:43.097" Score="0" ViewCount="1883" Body="&lt;p&gt;LSTM networks can be used to generate new text. Given a sequence, I can predict the next word. Is there a way to get a score associated to each word predicted?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, if the new word has never been seen by the LSTM network, can we train the LSTM to output a score of &quot;no confidence&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, this &lt;a href=&quot;https://arxiv.org/pdf/1602.06291.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; gives the following example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;For example, let us consider the following three text segments:&#xA;&#xA;1) Sir Ahmed Salman Rushdie is a British Indian novelist&#xA;and essayist. He is said to combine magical realism with&#xA;historical fiction.&#xA;&#xA;2) Calvin Harris &amp;amp; HAIM combine their powers for a magical&#xA;music video.&#xA;&#xA;3) Herbs have enormous magical power, as they hold the&#xA;earth’s energy within them.&#xA;&#xA;Consider an LM that is trained on a dataset having the&#xA;example sentences given above — given the word “magical”,&#xA;what should be the most likely next word: realism, music, or power?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Say, that, in fact, the next word is neither one, but &quot;power&quot;, but my LSTM has never seen that word before. So, the LSTM is going to predict one of the three words it has seen, but I would like it to output a low confidence score. Is this possible?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastActivityDate="2018-03-28T01:40:09.117" Title="Can LSTM have a confidence score for each word predicted?" Tags="|neural-network|nlp|lstm|prediction|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29430" PostTypeId="1" CreationDate="2018-03-22T15:55:45.573" Score="0" ViewCount="278" Body="&lt;p&gt;Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do &lt;code&gt;fillna.(alldata.mean())&lt;/code&gt; or will that cause data leakage. What would be a better way because doing them separately would have different mean for same column. &lt;/p&gt;&#xA;" OwnerUserId="39331" LastActivityDate="2018-04-27T01:02:29.933" Title="Filling in missing values" Tags="|machine-learning|data-cleaning|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29431" PostTypeId="1" CreationDate="2018-03-22T16:26:13.097" Score="1" ViewCount="213" Body="&lt;p&gt;I'm curious to know whether boosting, random forests or other types of ensemble models can perform multi-dimensional regression.&#xA;To be precise: That means multiple outputs (multi-dimensional labels) that are not intended for classification. For example predicting a persons height and weight from some data where there are two outputs each corresponding to one of height/weight (the labels and output values can be normalized if necessary). You can easily do this with neural networks of course. I am not interested in the trivial case where you simply have multiple models, one per regression output.&#xA;If yes, are there any libraries available that support this (the ones I've looked at do not)?&lt;/p&gt;&#xA;" OwnerUserId="21896" LastActivityDate="2018-09-19T08:00:29.670" Title="Multi-dimensional regression with ensemble models?" Tags="|regression|ensemble-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29432" PostTypeId="1" AcceptedAnswerId="29452" CreationDate="2018-03-22T16:36:02.553" Score="1" ViewCount="146" Body="&lt;p&gt;I found &lt;a href=&quot;https://github.com/davidadamojr/TextRank&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; python implementation on github with 400+ stars which use levenshtein distance between each nodes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But &lt;a href=&quot;http://www.aclweb.org/anthology/W04-3252&quot; rel=&quot;nofollow noreferrer&quot;&gt;original paper (page 4)&lt;/a&gt; said: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Next, all lexical units that pass the syntactic filter&#xA;  are added to the graph, and an &lt;strong&gt;edge is added between&#xA;  those lexical units that co-occur within a window of N word&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So the question is: is levenshtein distance legit for this algorithm or better to rewrite with windowed edges?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively levenshtein distance must not work because it's not denote importance of word...&lt;/p&gt;&#xA;" OwnerUserId="44623" LastEditorUserId="44623" LastEditDate="2018-03-22T20:15:41.220" LastActivityDate="2018-03-23T08:27:55.883" Title="What distance should I use for edges weights in textrank algorithm" Tags="|nlp|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29434" PostTypeId="1" CreationDate="2018-03-22T17:51:29.447" Score="3" ViewCount="1476" Body="&lt;p&gt;The &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt; state that &lt;code&gt;token_pattern&lt;/code&gt; is &lt;strong&gt;only used if analyzer == 'word'&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;token_pattern : string&#xA;&#xA;    Regular expression denoting what constitutes a “token”, only used if&#xA;    analyzer == 'word'. The default regexp selects tokens of 2 or more &#xA;    alphanumeric characters (punctuation is completely ignored and always &#xA;    treated as a token separator).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Below is my desired pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;analyzer = TfidfVectorizer().build_analyzer()&#xA;stemmer = SnowballStemmer('english')&#xA;&#xA;def processed_words(doc):&#xA;    return (stemmer.stem(w) for w in analyzer(doc))&#xA;&#xA;vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode',&#xA;            stop_words='english', token_pattern=r'\b[^_\d\W]+\b')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I include &lt;code&gt;analyzer=processed_words&lt;/code&gt;, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How about if I want to apply stemming AND lemmatization using two analyzers (include also &lt;code&gt;WordNetLemmatizer()&lt;/code&gt;) -- how does one handle that?&lt;/p&gt;&#xA;" OwnerUserId="36935" LastActivityDate="2021-08-11T02:08:00.713" Title="Scikitlearn - TfidfVectorizer - how to use a custom analyzer AND still use token_pattern" Tags="|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29435" PostTypeId="1" AcceptedAnswerId="29441" CreationDate="2018-03-22T18:17:01.680" Score="0" ViewCount="58" Body="&lt;p&gt;How do I represent a color as an activation value within a neuron? ( might be off-topic)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I want to detect colors Using Neural Nets&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my Knowledge any activation function which is generally used will push the values in between &lt;code&gt;0-1&lt;/code&gt; which isn't of any use to me...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just wondering how Windows 10 decides on the theme color change when we switch themes.. (led me to think about this or its just simple averaging on pixel values of the channels?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So seems like an impossible task for Neural Nets then?&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-22T18:27:41.627" LastActivityDate="2018-03-22T20:03:50.090" Title="Can colors be detected using Neural Nets?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29436" PostTypeId="2" ParentId="29430" CreationDate="2018-03-22T18:24:09.077" Score="1" Body="&lt;p&gt;No you shouldn't do that as the statistics of your training set and your test set isn't the same..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We apply the same transformations on both the data-sets but not using the same statistics..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also why always filling by mean is preferred?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search for interpolation in data-frames (requires scipy)&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-22T18:24:09.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29437" PostTypeId="1" CreationDate="2018-03-22T18:38:40.543" Score="2" ViewCount="442" Body="&lt;p&gt;I am working on a live sensor data set and looking for abnormal patterns (leading to a machine fault condition) from the available data set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am learning and new to the world of data science, but comfortable with Python. I have few questions that I am looking to get suggestions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;What kind of algorithms would be best suited for this case?&lt;/li&gt;&#xA;&lt;li&gt;What are the basic steps for doing a predictive analysis in python?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Please correct me if my questions are not correctly framed.&lt;/p&gt;&#xA;" OwnerUserId="49297" LastEditorUserId="29575" LastEditDate="2019-04-06T18:23:25.333" LastActivityDate="2019-04-06T18:23:25.333" Title="Anomaly Detection from available sensor data set?" Tags="|machine-learning|scikit-learn|predictive-modeling|anomaly-detection|data-science-model|" AnswerCount="2" CommentCount="7" ClosedDate="2019-04-07T02:38:06.597" ContentLicense="CC BY-SA 4.0" />
  <row Id="29438" PostTypeId="2" ParentId="29431" CreationDate="2018-03-22T18:45:41.423" Score="1" Body="&lt;p&gt;I think there are some algorithms in scikit-learn that support this kind of multi-output with correlated inputs/outputs (e.g. not using one learner per output, this can be achieved using the MultiOutputRegressor meta-estimator): RandomForest, DecisionTree and LinearRegression.&#xA;See : &lt;a href=&quot;https://stats.stackexchange.com/questions/153853/regression-with-scikit-learn-with-multiple-outputs-svr-or-gbm-possible&quot;&gt;https://stats.stackexchange.com/questions/153853/regression-with-scikit-learn-with-multiple-outputs-svr-or-gbm-possible&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py&lt;/a&gt;&#xA;for a comparison between the two approaches.&lt;/p&gt;&#xA;" OwnerUserId="27204" LastActivityDate="2018-03-22T18:45:41.423" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29439" PostTypeId="1" AcceptedAnswerId="29518" CreationDate="2018-03-22T19:02:56.887" Score="0" ViewCount="1152" Body="&lt;p&gt;Suppose we have feature transformation $\Phi(x) = [1, x_1, x_2, x_1x_2]$. Now we want to find the kernel corresponding to $\Phi$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have done is using kernel decomposition, we have:&#xA;$$&#xA;K(x, y) = \Phi(x) .\Phi(y)\\&#xA;K(x, y) = 1.1 + x_1y_1 + x_2y_2 + x_1x_2y_1y_2 \\&#xA;K(x, y) = 1 + \sum_{i=1}^{N}x_iy_i + \prod_{i=1}^{N}x_iy_i&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$x, y$ are in $\Bbb{R}^2$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is that kernel valid?&lt;/p&gt;&#xA;" OwnerUserId="43818" LastActivityDate="2018-03-25T16:30:12.663" Title="Finding a kernel with feature transformation" Tags="|svm|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29440" PostTypeId="1" CreationDate="2018-03-22T19:17:57.923" Score="3" ViewCount="6062" Body="&lt;p&gt;Given two sets of samples drawn from two different distributions, is it computationally possible to get an estimate of KL-Divergence between the two distribution using these samples?&lt;/p&gt;&#xA;&lt;p&gt;Here I am assuming the dimensionality of the two distributions is high (say &lt;em&gt;d&lt;/em&gt;). To compute the estimate, we first need to discretize the entire space and then estimate probabilities based on the frequencies. Let us say, we discretize each dimension into &lt;em&gt;p&lt;/em&gt; bins. Then the total number of grids in the space will be &lt;span class=&quot;math-container&quot;&gt;$p^d$&lt;/span&gt;. So we need to compute the probabilities of the two distributions for &lt;span class=&quot;math-container&quot;&gt;$p^d$&lt;/span&gt; grids, which is exponential in time. Hence I assume we cannot compute an estimate of KL Divergence using the samples for any practical problem.&lt;/p&gt;&#xA;&lt;p&gt;I wanted to check if this explanation is correct or if I am missing anything. Could someone assert if this rationale is correct?&lt;/p&gt;&#xA;" OwnerUserId="49112" LastEditorUserId="1330" LastEditDate="2020-06-17T15:15:58.650" LastActivityDate="2021-07-16T11:08:00.873" Title="Calculating an estimate of KL Divergence using the samples drawn from distributions" Tags="|machine-learning|probability|metric|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="29441" PostTypeId="2" ParentId="29435" CreationDate="2018-03-22T19:51:05.697" Score="2" Body="&lt;p&gt;Normally color spaces are not considered to be one dimensional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given three types of human cone cells, the most natural approach is probably not to use a single but three neurons. If color input is encoded as RGB values, then one neuron would be for red, one for green, and one for the blue channel.&lt;/p&gt;&#xA;" OwnerUserId="16213" LastEditorUserId="16213" LastEditDate="2018-03-22T20:03:50.090" LastActivityDate="2018-03-22T20:03:50.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29445" PostTypeId="2" ParentId="16807" CreationDate="2018-03-22T21:33:12.463" Score="2" Body="&lt;p&gt;Aside from the other answers I think it's worth pointing out that there are two quantities which are distinct but often coupled:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The number of inputs used to compute the gradient of the parameters in each step.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As others have pointed out, the gradient with respect to a minibatch is an approximation of the true gradient.  The larger the minibatch, the better the approximation.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;The number of inputs collected into an array and computed &quot;at the same time&quot;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The trade off here is purely about performance (memory/cycles).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These quantities are typically the same, i.e. the minibatch size, but in principle they can be decoupled.&lt;/p&gt;&#xA;" OwnerUserId="49306" LastActivityDate="2018-03-22T21:33:12.463" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29447" PostTypeId="1" AcceptedAnswerId="29456" CreationDate="2018-03-22T21:50:41.557" Score="0" ViewCount="3166" Body="&lt;p&gt;I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using someone else's code, whose goal is to perform cross-validation to choose the best tree-based algorithm for a binary classification. It is probably worth saying that classes are highly skewed (93% / 7%).&#xA;The metric which is used is the following: the classifier is trained and then the probability associated with each test element is computed.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probas = probas[:,list(clf.classes_).index(1)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, these probabilities are ordered from the highest to the lowest and put in the x-axis. On the y-axis, being y the cumulative sum of entries associated with each probability.&#xA;Then, they compute the area under the obtained curve, as in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;joint = zip(probas, truth)&#xA;joint = sorted(list(joint), key=lambda x:x[0], reverse=True)&#xA;probas = [x[0] for x in joint]&#xA;truth = [x[1] for x in joint]&#xA;&#xA;# Calculate accumulated number of true labels at each probability point.&#xA;# Also calculate Area Under Curve (AUC) (higher is better model)&#xA;truth_cumulative = np.cumsum(truth) / np.sum(truth)&#xA;area = np.trapz(truth_cumulative, dx=1) / len(truth)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me an intuition of what this metric is about and a pointer to some resources to better understand it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="49308" LastActivityDate="2018-03-23T18:52:18.147" Title="Area Under Curve with probability" Tags="|python|metric|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29448" PostTypeId="2" ParentId="23029" CreationDate="2018-03-22T22:26:02.613" Score="2" Body="&lt;p&gt;RNNs aren't necessarily faster. It's hard to compare different architectures, but in a text-mining application I'm doing, my RNN takes 3-5x longer than my CNN to train and score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long-distance relationships will take a lot of data to learn with any model. And it's not clear to me that paying more attention to more recent data is a bad thing. Systems change over time and so paying a lot of attention to ancient data probably isn't useful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you mean by &quot;understand&quot;: Comprehended in the abstract, visualize, probe about specific predictions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a cool RNN visualization (text): &lt;a href=&quot;https://arxiv.org/abs/1506.02078&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1506.02078&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The choice isn't &quot;LSTMs are all hype&quot; versus &quot;LSTMs ROCK&quot;. For example, in text use (categorical time series of words or characters), CNNs are better at some things, while LSTMs are better at others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And theoretically, RNNs (including LSTMs, of course) can process varying-length series straightforwardly, while CNNs or other NNs need some work. (In practice -- for example if you're using tensorflow as your underlying engine -- you won't be able to handle absolutely any length of series with RNNs, but the principle is there.)&lt;/p&gt;&#xA;" OwnerUserId="16613" LastActivityDate="2018-03-22T22:26:02.613" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29449" PostTypeId="1" AcceptedAnswerId="29451" CreationDate="2018-03-22T22:51:57.963" Score="2" ViewCount="3586" Body="&lt;p&gt;In the &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/overview.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;overview page&lt;/a&gt; of the &lt;a href=&quot;https://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas&lt;/a&gt; documentation the Series data structure is described as 'homogeneously-typed'. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data Structures&#xA;Dimensions  Name    Description&#xA;1   Series  1D labeled homogeneously-typed array&#xA;2   DataFrame   General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed column&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However it is possible to create Series objects with multiple data-types.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])&#xA;#=&amp;gt;a    1&#xA;#  b    2&#xA;#  c    3&#xA;#  d    4&#xA;#  e    5&#xA;#  f    x&#xA;#  dtype: object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So what could be the meaning of &lt;code&gt;homogeneously-typed&lt;/code&gt;mentioned in pandas documentation?&lt;/p&gt;&#xA;" OwnerUserId="46898" LastActivityDate="2018-03-23T00:35:53.943" Title="Series data structure in pandas" Tags="|pandas|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29451" PostTypeId="2" ParentId="29449" CreationDate="2018-03-22T23:25:28.503" Score="1" Body="&lt;p&gt;If you have multiple different types in a Series, say &lt;code&gt;int&lt;/code&gt; and &lt;code&gt;string&lt;/code&gt;, all of the data will get upcasted to &lt;strong&gt;the same&lt;/strong&gt; &lt;code&gt;dtype=object&lt;/code&gt; (as you can see from your example). &lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-03-22T23:25:28.503" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29452" PostTypeId="2" ParentId="29432" CreationDate="2018-03-22T23:45:02.637" Score="0" Body="&lt;p&gt;As lib author said using lev-dist was just en experiment and might not work (and it really doesn't), but for some reason people use broken lib w/o questions ¯\_(ツ)_/¯&lt;/p&gt;&#xA;" OwnerUserId="44623" LastEditorUserId="44623" LastEditDate="2018-03-23T08:27:55.883" LastActivityDate="2018-03-23T08:27:55.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29453" PostTypeId="2" ParentId="27784" CreationDate="2018-03-23T00:27:29.713" Score="0" Body="&lt;p&gt;Is the number of predictor variables in your new dataset large? If so, Euclidean distance can be unreliable due to the curse of dimensionality. You might want to try reducing the dimension of your dataset using Principal Component Analysis or feature selection. Alternatively, you might want to try using a different distance metric, like the cosine similarity.&lt;/p&gt;&#xA;" OwnerUserId="49299" LastActivityDate="2018-03-23T00:27:29.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29454" PostTypeId="2" ParentId="29449" CreationDate="2018-03-23T00:35:53.943" Score="2" Body="&lt;p&gt;Your series is indeed homogeneously-typed and you can check it's type:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;s = pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])&#xA;s.dtype&#xA;&amp;gt; dtype('O')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where 'O' is for &quot;object&quot;. However, if you check the type of the individual elements of your series, they are different:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;type(s['a'])&#xA;&amp;gt; int&#xA;type(s['f'])&#xA;&amp;gt; str&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think the key thing to remember is that if your Series or DataFrame Column is non homogenously-typed (eventhough technically homogeneously-typed of type &quot;object&quot;), then there are certain pandas functions that won't work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a better explanation from the pandas documentation's &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/basics.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Essential Basic Functionality&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If a DataFrame or Panel contains homogeneously-typed data, the ndarray&#xA;  can actually be modified in-place, and the changes will be reflected&#xA;  in the data structure. For heterogeneous data (e.g. some of the&#xA;  DataFrame’s columns are not all the same dtype), this will not be the&#xA;  case. The values attribute itself, unlike the axis labels, cannot be&#xA;  assigned to.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all of the data&#xA;  involved. For example, if strings are involved, the result will be of&#xA;  object dtype. If there are only floats and integers, the resulting&#xA;  array will be of float dtype.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="49177" LastActivityDate="2018-03-23T00:35:53.943" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29456" PostTypeId="2" ParentId="29447" CreationDate="2018-03-23T02:47:34.420" Score="1" Body="&lt;p&gt;In your code the Area Under the Curve (AUC) is used to calculate the area under the Cumulative Distribution Function (CDF). Let's go through the code to see how this is done. However, in the last section, I do not agree with the two lines used to calculate the area. If all the labels have a ground truth label of 1, and we have a perfect classifier where the probability of being Class 1 is always 100%, we should have an AUC of 1. However, we will get 50%. I will show this later. You should note that the way this code is written will only work for a binary classifier, with labels 0 and 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will first calculate the probability of each instance being in class 0 and class 1 by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probs = forest.predict_proba(X_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we will extract the second column, this contains the probability of each instance being in Class 1.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probas = probs[:,list(forest.classes_).index(1)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we will join the probabilities of each instance with its ground truth label. Then we will sort the groupings based on their probability. The instances with a probability of 1 will be at the start of the list and those with a probability of 0 will be at the end of the list. This is done using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;joint = zip(probas, truth)&#xA;joint = sorted(list(joint), key=lambda x:x[0], reverse=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we will separate the probabilities and the ground truth labels into separate lists using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;probas = [x[0] for x in joint]&#xA;truth = [x[1] for x in joint]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The blue line is the probabilities and the ground truth labels is the orange line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NC0TB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NC0TB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This is where the problems start. The next two lines do not include the probabilities but only the ground truth labels.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;truth_cumulative = np.cumsum(truth)&#xA;truth_cumulative = truth_cumulative / np.sum(truth)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This means that if we had all class 1, with probability 1. We would get an AUC of 0.5. This does not make any sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I force it by doing this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;probas[100:100000] = 1&#xA;truth[100:100000] = 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There will only be a few 0 values left. Then after the ordering and cumulative sum we get the plot as follow, which is obviously wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f4rtZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f4rtZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;We need to do exactly what is written in the comment but not done through code. We will get the false positive rates for each false negative rate. Much like what is described below in the Receiver Operator Curve (ROC).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be done with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Training Set&#xA;probas_train = forest.predict_proba(X_train)&#xA;fpr_t, tpr_t, thresholds_t = roc_curve(y_train, probas_train[:, 1])&#xA;&#xA;# Testing Set&#xA;probas_ = forest.predict_proba(X_test)&#xA;fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])&#xA;roc_auc = auc(fpr, tpr)&#xA;&#xA;train, = plt.plot(fpr_t, tpr_t, label = 'Train')&#xA;test, = plt.plot(fpr, tpr, label = 'Test')&#xA;plt.xlabel('False Positive Rate')&#xA;plt.ylabel('True Positive Rate')&#xA;plt.legend()&#xA;plt.show()&#xA;&#xA;print('Area Under the Curve (AUC): ', roc_auc)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Training Set &lt;br/&gt; Area Under the Curve (AUC):  0.998140359403 &lt;br/&gt; Testing Set &lt;br/&gt;&#xA;  Area Under the Curve (AUC):  0.937729144675&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/u1zB1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/u1zB1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Receiver Operator Curve (ROC)&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This was invented during WWII as a means to detect aircraft using radar. For example, if we have a sensor which requires some kind of threshold to detect planes. We can determine the true positive rate (TPR) and false positive rate (FPR) which results in our experiment. We will then plot these as the x-axis as the false positive rate, and the y-axis as the true positive rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$TPR = \frac{\text{True Positives}}{\text{Positives}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$FPR = \frac{\text{False Positives}}{\text{Negatives}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the sensor is perfect we will always have $0$ false positives and always a $100\%$ true positives. This results in a curve which looks like the blue curve. If the sensor was completely random and garbage, then you would have a random guess which is the diagonal line.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UM9Og.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UM9Og.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good classifier will be as close to the blue line as possible. Note, that a line below the diagonal is also good, you should just invert your decision. You have a consistent classifier but you are inverting the decision. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, consider a sensor which reads a value and it is above a threshold $\theta$ we call it a detection. If we lower the threshold of detection, we should expect more detection, however if we end up with an equal amount of true positives and false negatives, then we are going along the diagonal line. This is a bad classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q7AEE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q7AEE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we have 2 sensors, red and yellow, we want to decide which sensor is better. Just looking at the curves is not very effective because it's hard to see which one is definitely higher. So we can devise a metric named the &lt;strong&gt;Area Under the Curve (AUC)&lt;/strong&gt;. This is exactly what it means in the name. Then we an say the curve with the higher AUC is the better one.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-23T18:52:18.147" LastActivityDate="2018-03-23T18:52:18.147" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29458" PostTypeId="1" CreationDate="2018-03-23T04:35:32.790" Score="1" ViewCount="82" Body="&lt;p&gt;In &lt;a href=&quot;https://onlinelibrary.wiley.com/doi/full/10.1002/widm.1097&quot; rel=&quot;nofollow noreferrer&quot;&gt;'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures'&lt;/a&gt; they have mentioned;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;There are two slightly different classes of measure: &lt;strong&gt;lexical cohesion (sometimes called ‘unithood’ or ‘phraseness’)&lt;/strong&gt;, which quantifies the expectation of co-occurrence of words in a phrase (e.g., back-of-the-book index is significantly more cohesive than term name); and &lt;strong&gt;semantic informativeness (sometimes called ‘termhood’)&lt;/strong&gt;, which highlights phrases that are representative of a given document or domain.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the review does not include the ways to calculate/derive these measures. Can someone please specify how to get these two measurements for a given text documents?&lt;/p&gt;&#xA;" OwnerUserId="45262" LastActivityDate="2024-02-29T14:04:45.573" Title="How to calculate lexical cohension and semantic informaticveness for a given dataset?" Tags="|data-mining|nlp|text-mining|statistics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29460" PostTypeId="2" ParentId="25004" CreationDate="2018-03-23T06:22:26.413" Score="0" Body="&lt;p&gt;Your intuition that the distributions of words in the subject and email body are likely to be different has merit. Regarding and &quot;out of the box&quot; solution, the easiest approach would be to construct two separate classifiers -- one each for the subject and body -- and then ensemble them together. A very simple ensemblification approach would be to just use the probability scores outputted by those two models as inputs to a logistic regression classifier.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-03-23T06:22:26.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29461" PostTypeId="1" CreationDate="2018-03-23T07:48:55.603" Score="0" ViewCount="198" Body="&lt;p&gt;I applied PCA on MNIST data and found that the first 64 components are able to retain 86% of variance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any problem while applying pca to a big dataset like MNIST. Because in most of the papers I have read to take the components which can explain upto 99% of variance. But it would be pointless to apply pca if such variance comes at 120 pca components. &lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-03-23T09:56:35.353" Title="very low variance explained after applying pca" Tags="|machine-learning|pca|mnist|variance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29462" PostTypeId="1" CreationDate="2018-03-23T08:24:21.930" Score="2" ViewCount="170" Body="&lt;p&gt;Newbie alert to data science and ML.&#xA;I'm learning Supervised and Unsupervised learning at the moment and Supervised learning is easy to digest and I can relate to a lot of practical use cases. Unsupervised learning is where I just couldn't correlate to a the real-world use cases (although I found numerous of quotes where people say they use it for Customer Segmentation, Fraud Detection etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For argument sake, I'll just quote a sample taken from one of the MS Azure Studio examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset contains Countries and their average protein intakes in various forms of food).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When this dataset is run through a KMeans algorithm, it creates 3 clusters and fits the country names in these clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in this specific example what is the problem I'm trying to solve?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I looking to find similar countries based on protein intake habits?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I creating groups based on the given dataset and then there is a human intelligence to qualify these groups (or clusters) to say &quot;Vegetarian Rich Countries&quot;, &quot;Red Meat Rich Countries&quot; etc. Then when a new country comes we predict whether this country falls in which cluster?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, there is an intermediate human intelligence is needed in the workflow which requires labelling the cluster (as opposed to labelling each datapoint in classification). Is this a correct understanding?&lt;/p&gt;&#xA;" OwnerUserId="44861" LastActivityDate="2020-02-21T17:46:37.113" Title="How does ML Clustering put to a practical real-world use?" Tags="|machine-learning|classification|clustering|k-means|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29464" PostTypeId="1" CreationDate="2018-03-23T08:57:42.450" Score="8" ViewCount="1781" Body="&lt;p&gt;I am a novice seeking help to gain experience in Data Science.&lt;/p&gt;&#xA;&lt;p&gt;Let us take a scenario where a big company would like to forecast its sales (a specific product) across different stores in different geographic locations.&#xA;As an Analyst, a task is given to forecast a few hundred series (sales) for the next 3 months. Since, we are forecasting sales across different geographic&#xA;locations, the nature of the series would not be same for all of them. There would hundreds of models to check with.&#xA;What are the suggested approaches for this scenario with your experience in this field? Also, how important it is know the nature of each series in this scenario?&lt;/p&gt;&#xA;" OwnerUserId="49058" LastEditorUserId="29169" LastEditDate="2021-09-18T17:02:04.080" LastActivityDate="2021-10-19T22:06:55.603" Title="Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros" Tags="|time-series|regression|lstm|rnn|forecasting|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29466" PostTypeId="2" ParentId="29461" CreationDate="2018-03-23T09:56:35.353" Score="1" Body="&lt;p&gt;It is entirely correct to apply PCA to a dataset like MNIST. Intuitively, corner pixels should almost never contain any information as to what digit is contained in the center of the image. So we should disregard them. You should expect similar results as with other datasets. PCA lowers the dimensionality of your data, thus allowing for a less complex model, however, this is at the cost of some information that is rejected when retaining only $n$ components. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When data is limited a less complex model will result in a lower bias and thus better accuracy when applied to novel data. However, the MNIST dataset is plentiful, thus the benefit of removing some features and losing even minimal information may cause your performance to degrade.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-23T09:56:35.353" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29467" PostTypeId="1" CreationDate="2018-03-23T10:00:11.403" Score="1" ViewCount="844" Body="&lt;p&gt;I have CNN architecture for object detection ( one object in image ) in KERAS.&#xA;It has 22 Convolutional layers ( layer includes max pool , LeakyRelu and Batchnorm )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last layes are following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(Flatten())&#xA;model.add(Dense(6))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basicly i want to output vector that reresents &lt;code&gt;( probability that is is ObjECT , probability that it isnt , x , y , xmax, ymax )&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am feeding it data that has one object in it and labels&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;( 1, 0, xmin , ymin , xmax , ymax )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This has around 0.60% accuracy. However the sum of (isOBject + isntObject ) is greater than 1.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[   1.02664185    0.34075582  231.69602966  241.59547424  280.6262207&#xA;   266.35855103]] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hos is this possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also when i feed network another data that does NOT have object in it and labl them&lt;/p&gt;&#xA;&#xA;&lt;p&gt;( 0, 1 , 0, 0, 0, 0 ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get 10% accuracy. How can it be? It has more data to learn from and added false data on top of that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate all tips and tricks / explanation, basicly all help. Thanks&lt;/p&gt;&#xA;" OwnerUserId="40162" LastEditorUserId="29169" LastEditDate="2021-02-13T16:35:51.820" LastActivityDate="2021-02-13T16:35:51.820" Title="The sum of probabilities is more than 1" Tags="|machine-learning|neural-network|training|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29468" PostTypeId="2" ParentId="29467" CreationDate="2018-03-23T10:11:49.407" Score="2" Body="&lt;p&gt;You need to use appropriate activations. If you were using softmax for those two components, they'd be constrained to sum to one. LeakyReLU not only doesn't impose this constraint: it can output any real value, so it isn't even constrained to [0,1] for those components individually.&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-03-23T10:13:12.440" LastActivityDate="2018-03-23T10:13:12.440" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29469" PostTypeId="1" CreationDate="2018-03-23T11:39:45.433" Score="1" ViewCount="413" Body="&lt;p&gt;Until now I have used LDA only for &lt;em&gt;topic modelling&lt;/em&gt;. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2024-01-06T22:02:15.227" Title="Implementation of LDA (Latent Dirichlet Allocation) for classification tasks" Tags="|classification|nlp|lda|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29470" PostTypeId="1" CreationDate="2018-03-23T11:44:51.010" Score="0" ViewCount="806" Body="&lt;p&gt;I have a dataset (~52k rows) with a column containing just pure sentences (upper and lowercase, with punctuation and stop words) in each row. What can I do to represent this data in a meaningful way for my model. I tried using a &lt;a href=&quot;https://www.google.com/url?q=http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&amp;amp;sa=U&amp;amp;ved=0ahUKEwi5j6epqYLaAhXrL8AKHb0FDyMQFggEMAA&amp;amp;client=internal-uds-cse&amp;amp;cx=016639176250731907682:tjtqbvtvij0&amp;amp;usg=AOvVaw0ENAbFAIkFHRcqWD5qbg3v&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;CountVectorizer&lt;/code&gt;&lt;/a&gt; but it's giving me a dataframe with way too many columns and the model is having a hard time using it. What other options do I have?&lt;/p&gt;&#xA;" OwnerUserId="49341" LastActivityDate="2018-05-24T17:23:30.183" Title="Feature extraction from pure text" Tags="|python|scikit-learn|feature-extraction|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29471" PostTypeId="1" CreationDate="2018-03-23T12:38:15.440" Score="5" ViewCount="1646" Body="&lt;p&gt;Optimization algorithms such as gradient descent or particle swarm can find a minima in a function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, learning methods such as back-prop define learning as an optimization problem and are used to learn weights in deep neural networks and etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that DL models can learn/memorize basically anything in the training data [2], even random noise, which brings us to the question of then how we can achieve generalization (as defined in [5]) &lt;sup&gt;*&lt;sub&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In [7, 1, 3] authors tried to show the connection between generalization and the loss landscape (sharp minima/flat minima) plus its effect from the batchsize. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect of training data on generalization studied in [4] shows that models that have high influence from more data points generalize better than models that rely/influenced on less data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand [6] sheds some light on the role of the model's parameters from a neuroscience point of view in terms of effect of neurons in generalization (Networks that generalize better are harder to break with neuron deletion.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although all of these papers investigated generalization up to a certain point, the effect of optimization used in the learning algorithm on the generalization is not yet clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any study done in this direction? What are the possible related work on this? Or do you have an answer to this question?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*:&#xA;Let me point out that although many believe &lt;em&gt;explicit regularization&lt;/em&gt; is crucial for generalization, [2] already explains explicit regularization (l1/l2/dropout) does not play a big role in generalization. Many tricks that were known as a generalizer, are shown to be a myth. They also show that interestingly, SGD works as an implicit regularizer which might be a connection to the effect of optimization alg. to generalization.&lt;br&gt;&#xA;&lt;/sub&gt;&lt;/sup&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: Li, Hao, et al. &quot;Visualizing the Loss Landscape of Neural Nets.&quot; arXiv preprint arXiv:1712.09913 (2017).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]: Zhang, Chiyuan, et al. &quot;Understanding deep learning requires rethinking generalization.&quot; arXiv preprint arXiv:1611.03530 (2016).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3]: Keskar, Nitish Shirish, et al. &quot;On large-batch training for deep learning: Generalization gap and sharp minima.&quot; arXiv preprint arXiv:1609.04836 (2016).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[4]: Koh, Pang Wei, and Percy Liang. &quot;Understanding black-box predictions via influence functions.&quot; arXiv preprint arXiv:1703.04730 (2017).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[5]: Kawaguchi, Kenji, Leslie Pack Kaelbling, and Yoshua Bengio. &quot;Generalization in deep learning.&quot; arXiv preprint arXiv:1710.05468 (2017).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[6]: Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, Matthew Botvinick, &quot;On the importance of single directions for generalization.&quot; arXiv preprint arXiv:1803.06959 (2018). &lt;a href=&quot;https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[7]: Dinh, Laurent, et al. &quot;Sharp minima can generalize for deep nets.&quot; arXiv preprint arXiv:1703.04933 (2017).&lt;/p&gt;&#xA;" OwnerUserId="49290" LastEditorUserId="49290" LastEditDate="2018-03-23T15:35:40.190" LastActivityDate="2018-05-22T18:59:03.747" Title="The connection between optimization and generalization" Tags="|machine-learning|neural-network|deep-learning|optimization|generalization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29472" PostTypeId="1" CreationDate="2018-03-23T12:49:56.827" Score="5" ViewCount="1032" Body="&lt;p&gt;As I understand &lt;strong&gt;STN&lt;/strong&gt; as described by the the deepmind paper &lt;a href=&quot;https://arxiv.org/abs/1506.02025&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1506.02025&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;allow a neural network to learn how to perform spatial transformations on the input image in order to enhance the geometric invariance of the model. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;whereas &lt;strong&gt;Deformable Convolutions&lt;/strong&gt; &lt;a href=&quot;https://arxiv.org/abs/1703.06211&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1703.06211&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;add 2D offsets to the regular grid sampling locations in the standard convolution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From the paper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;...deformable convolution does&#xA;not adopt a global parametric transformation and feature&#xA;warping. Instead, it samples the feature map in a local&#xA;and dense manner. To generate new feature maps, it has&#xA;a weighted summation step, which is absent in STN.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me STNs augment the input image whereas DC augment the kernel shape. Results should be similar/same in terms of field of view of the network? &lt;strong&gt;Please correct me if mistaken.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="47730" LastActivityDate="2018-06-11T22:21:36.143" Title="Spatial Transformer Networks vs Deformable Convolutions" Tags="|neural-network|deep-learning|computer-vision|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29474" PostTypeId="2" ParentId="19674" CreationDate="2018-03-23T14:04:38.277" Score="-1" Body="&lt;p&gt;hstak is a good solution. Using this I combined these features.&lt;/p&gt;&#xA;" OwnerUserId="33245" LastActivityDate="2018-03-23T14:04:38.277" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29475" PostTypeId="2" ParentId="29470" CreationDate="2018-03-23T14:16:38.783" Score="0" Body="&lt;p&gt;There's a myriad of approaches you can do here but each really depend on your end goal.  You can use a BOW approach and create a large sparse matrix like you've seen before with CountVectorizer or TfidfVectorizer.  You can computer similarity scores between sentences using cosine distances.  You can score them using bm25+.  You can show average idf score.  You can show co-occurance of terms in sentence when compared to values in same target or different target or all targets.  You can embed the words into a continious vector space using word2vec, fasttext etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the end, it really comes down to what you're overall goal.  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-03-23T14:16:38.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29477" PostTypeId="2" ParentId="29471" CreationDate="2018-03-23T14:48:33.000" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What part the optimization alg. (Grad. Descent) plays in generalization of the learning algorithm?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Actually for generalizing you have to find a model that does not overfits the training data. For doing so, there are numerous approaches, like L1/L2 regularization which adds noise to the cost function and somehow weights to prohibit the network from having large weights which may lead to overfitting. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/23287/28175&quot;&gt;here&lt;/a&gt;. Other techniques are drop-out which adds noise to the activations to help the network not to depend on special nodes. These techniques add some noise to different parts of the network which lead to a cost function with a high error value. After finding the error back prop techniques try to set the parameters/weights to go downhill of the cost function. These techniques help the algorithm not to overfit which means the constructed model will be able to generalize, although you have to test it using cross-validation data and test data. Consequently, optimization techniques by themselves always try to reduce the amount of error and they somehow always lead to overfitting because they try to reduce the cost and this is regularization techniques and other variants that have to be used to construct a cost function which its optimum points do not lead to overfitting. This &lt;a href=&quot;https://datascience.stackexchange.com/a/24537/28175&quot;&gt;link&lt;/a&gt; also may help you.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Or the influence of the data [4] plays a more important role? Although [6] sheds the light on the role of the model parameters from a neuroscience point of view in terms of the effect of neurons in generalization, the question on how to learn such models remains unsolved.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Data is the most important part of the learning process. Your data have to be representative enough. Deep learning problems are considered those problems which may have better results if their training data increases. If the model should be able to generalize, it is vital to use data from the real distribution. This is just for training. You can use some techniques in order not to overfit the data using your training data-set. It is customary to add small noises to the input to let the model generalize well. Transforming data by translation, rotation and even distorting image inputs are examples of adding noise to the input to avoid overfitting. Although changing the data is dangerous because it may change the distribution of the real population. So, you can do something with your input data to avoid overfitting and let your model to generalize well.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So how can we really better understand the generalization considering different factors as mentioned above?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generalization techniques are used to make the network generalize well. The customary approaches are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Drop Out&lt;/li&gt;&#xA;&lt;li&gt;L1/L2 Regularization&lt;/li&gt;&#xA;&lt;li&gt;Early Stop&lt;/li&gt;&#xA;&lt;li&gt;Adding Noise and data-augmentation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It should be considered that adding noise to the input should be without changing the distribution. The ratio of signal to noise also should not be small because the information may be lost compeletely. Depending on your problem each of them make work well but I guess there is no consensus which one is the best but in deep-learning the first one is so helpful. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is the connection between optimization and generalization (as defined in [5])?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What I'm saying is based on my experience. Optimization itself always lead to overfitting. You have to use generalization techniques to avoid that. To help you figure out the problem, suppose that deep-learning algorithms are able to learn all functions. If you provide them with relatively small number of data, they will memorize a hypothesis, and they do not learn the problem. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-23T14:48:33.000" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="29478" PostTypeId="2" ParentId="11390" CreationDate="2018-03-23T15:46:20.127" Score="3" Body="&lt;p&gt;Bit late to the party, but here are some heuristics. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For &lt;em&gt;training&lt;/em&gt; of linear classifiers, 3 - 5 independent cases per class and feature are recommended. This limit gives you reliably &lt;em&gt;stable&lt;/em&gt; models, it doesn't guarantee a &lt;em&gt;good&lt;/em&gt; model (this is not possible: you could have uninformative data where no model could achieve good generalization performance)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;However, for sample sizes as small as your scenario, &lt;em&gt;verification (validation)&lt;/em&gt; rather than training is the bottleneck, and verification depends on absolute number of test cases rather than cases relative to model complexity: as a rule of thumb, you need ≈ 100 test cases in the denominator to estimate a proportion with a confidence interval that is not more than 10 %points wide.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately this also means that you basically cannot get the empirical learning curve for your application: you cannot measure it precisely enough, and in practice you'd anyways have huge difficulties extrapolating it because for training you react to the small sample size by restricting your model complexity - and you'd relax this with increasing sample size.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See our paper for details: &lt;a href=&quot;http://dx.doi.org/10.1016/j.aca.2012.11.007&quot; rel=&quot;nofollow noreferrer&quot;&gt;Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.&lt;br&gt;&#xA;DOI: 10.1016/j.aca.2012.11.007&lt;/a&gt;&lt;br&gt;&#xA;&lt;a href=&quot;http://arxiv.org/abs/1211.1323&quot; rel=&quot;nofollow noreferrer&quot;&gt;accepted manuscript on arXiv: 1211.1323&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I've never had anything close to these recommendations (spectroscopy data, also for medical applications). What I do then is: I very closely measure model stability as part of the modeling and the verification process.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5268" LastActivityDate="2018-03-23T15:46:20.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29479" PostTypeId="1" CreationDate="2018-03-23T16:33:34.023" Score="1" ViewCount="217" Body="&lt;p&gt;I'm trying to learn a sufficient statistic from data. My input are sequences of samples &lt;code&gt;x_samples&lt;/code&gt; which are real numbers. The length of the input sequence varies and has no fixed value. There is no memory between the sequences - each sequence is completely independent from all the others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output &lt;code&gt;y&lt;/code&gt; is the sufficient statistic given by the sum of &lt;code&gt;log(p_1(x)/p_2(x))&lt;/code&gt; for all the samples in a given sequence (thus a RNN is needed). &lt;code&gt;p_1&lt;/code&gt; and &lt;code&gt;p_2&lt;/code&gt; are two unknown functions (they are the PDFs of some process). For example if a sequence has 3 samples &lt;code&gt;x1,x2,x3&lt;/code&gt; the output is given by &lt;code&gt;y = log(p_1(x1)/p_2(x1)) + log(p_1(x2)/p_2(x2)) + log(p_1(x3)/p_2(x3))&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering how to input the sequences into my network. I've seen a few solutions to this problem but none of them seem relevant for me.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I don't want to use zero padding. The reason is that the sequence has no maximum length - it depends on other factors independent of this problem and may theoretically take any value. The LSTM is part of a bigger RL system. The data is generated along the way  so i don't have a fixed data set.&lt;/li&gt;&#xA;&lt;li&gt;I don't want to use 1 hot encoding for my input since &lt;code&gt;x_samples&lt;/code&gt; are just real values. It is possible to quantize the input but i think this is a bad solution.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I think it makes sense to input the the samples one by one with a start and end tokens for each sequence. Does this make sense?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally I wonder if during the training I should reset the memory of the LSTM in the end of each sequence (with &lt;code&gt;model.reset_states()&lt;/code&gt;, have I understood it correctly?) or it is enough to set a start and end tokens in the begging and the end of every sequence.&lt;/p&gt;&#xA;" OwnerUserId="49358" LastActivityDate="2018-03-23T16:33:34.023" Title="LSTM with keras/tensorflow implementation" Tags="|python|keras|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29480" PostTypeId="1" CreationDate="2018-03-23T18:52:28.867" Score="23" ViewCount="131354" Body="&lt;p&gt;I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder into my notebook and use it?&lt;/p&gt;&#xA;" OwnerUserId="43878" LastEditorUserId="28175" LastEditDate="2018-03-24T05:36:54.847" LastActivityDate="2020-10-13T13:53:46.513" Title="Uploading images folder from my system into Google Colab" Tags="|machine-learning|neural-network|deep-learning|dataset|colab|" AnswerCount="6" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29482" PostTypeId="2" ParentId="29480" CreationDate="2018-03-23T19:45:50.157" Score="3" Body="&lt;p&gt;You don't need to upload them if you have a download link ...( it would be faster if you can upload them all as either ways you have to do so.. So its better to upload them first and then download them in your notebook every-time you run it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a download link then just this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;! wget &amp;lt;Link&amp;gt;&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Else upload then to your drive and then just use the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from google.colab import files&#xA;&#xA;uploaded = files.upload()&#xA;&#xA;##files.upload returns a dictionary of the files which were uploaded. The &#xA;##dictionary is keyed by the file name, the value is the data which was &#xA;##uploaded.&#xA;&#xA;for fn in uploaded.keys():&#xA;  print('User uploaded file &quot;{name}&quot; with length {length} bytes'.format(&#xA;      name=fn, length=len(uploaded[fn]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-23T19:45:50.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29485" PostTypeId="1" AcceptedAnswerId="32816" CreationDate="2018-03-23T23:49:50.477" Score="5" ViewCount="1448" Body="&lt;p&gt;I am running a GAN on the MNIST dataset. As the GAN continues to train, the quality of the generator seems to get worse, and it even seems as if it is starting to converge to just one value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch 7:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/YBofC.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YBofC.png&quot; alt=&quot;Epoch 7&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch 21:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4urxt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4urxt.png&quot; alt=&quot;Epoch 21&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is this happening? I am running the code from this repo here:&#xA;&lt;a href=&quot;https://github.com/bstriner/keras-adversarial/blob/master/examples/example_gan.py&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/bstriner/keras-adversarial/blob/master/examples/example_gan.py&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49377" LastActivityDate="2018-06-08T09:45:40.400" Title="GAN discriminator converging to one output" Tags="|machine-learning|neural-network|image-classification|gan|mnist|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29486" PostTypeId="2" ParentId="12909" CreationDate="2018-03-24T00:56:04.713" Score="4" Body="&lt;p&gt;A model, loosely speaking, is a simplification of some thing or process. For example, the shape of the Earth not actually a sphere, but we might treat it as one if we are designing a globe. Similarly, assuming the universe is deterministic, there is some natural process which determines if a customer will buy a product on a website. We might construct a something that approximates that process, which we could give some information about a customer and which tells us if it thinks that customer will buy a product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &quot;machine learning model&quot;, then, is a model constructed by a machine learning system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Apologies for this not being a rigorous answer, but I hope this is still useful.)&lt;/p&gt;&#xA;" OwnerUserId="49378" LastActivityDate="2018-03-24T00:56:04.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29487" PostTypeId="1" AcceptedAnswerId="29489" CreationDate="2018-03-24T03:51:41.090" Score="2" ViewCount="84" Body="&lt;p&gt;My friend is in the business of getting cats to the top of mountains. He currently uses a set of heuristics to decide which cats are most likely to get to the top. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the ones he likes, he feeds, at great expense, in the hope that they will be strong enough to get to the top. For the ones he doesn't like, he thinks it not worth his time / money to feed, and he's happy and surprised if they get to the top anyway. From looking at the data, we know that on average, the ones he doesn't feed get to the top less often.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am training a classifier to help him decide which to feed. Does it make sense to include rows in the training data where he fed them and where he didn't? When I excluded the rows where he did not feed them, the model did worse, which I did not expect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forgive the example. It's contrived, but I think retains all of the characteristics of the actual situation.&lt;/p&gt;&#xA;" OwnerUserId="13179" LastEditorUserId="13179" LastEditDate="2018-03-24T03:57:00.577" LastActivityDate="2018-03-24T16:42:08.827" Title="Including biased data in training model" Tags="|machine-learning|predictive-modeling|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29488" PostTypeId="1" CreationDate="2018-03-24T04:32:54.607" Score="1" ViewCount="106" Body="&lt;p&gt;I've been pulled onto my first Data Science project at work. Classic problem of predicting sales based on web traffic data, etc. While I don't know about the specific techniques I will be using in my model, I think that PCA will probably be tried at some point, and if I use it successfully as a preprocessing step before I chug the data into some estimator, I'm thinking about how I might go about defending/explaining my work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, what is an intuitive explanation for PCA that doesn't rely heavily on the linear algebra that makes up the process? Perhaps an analogy that draws a parallel to something that any layman can readily understand? &lt;/p&gt;&#xA;" OwnerUserId="41255" LastActivityDate="2018-05-25T18:42:56.570" Title="Whats an explanation of PCA that is intuitive for someone in senior leadership who doesn't have a technical background?" Tags="|prediction|pca|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29489" PostTypeId="2" ParentId="29487" CreationDate="2018-03-24T05:23:46.403" Score="1" Body="&lt;p&gt;That's a wonderful question, and beautifully posed.  I love how you capture the essence of the issue in such a clean way.  Unfortunately, the answer is that your data set does not have enough information to help you decide which cats you should feed.  Without a properly controlled experiment, you don't have a way to infer causality; you can't rule out the possibility of confounding factors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that 70% of the cats he feeds make it to the top, and 20% of the cats he doesn't feed make it to the top.  Let me make up four alternative explanations for why that might happen:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Hypothesis #1: There's only one kind of cat in the world.  The characteristics of the cat are irrelevant.  If you feed them, they will have a 70% chance of making it to the top; if you don't, they'll have only a 20% chance.  This is true for all cats, regardless of their attributes.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hypothesis #2: There are two kinds of cats in the world: the strong, and the weak.  The strong have a 70% chance of making it to the top of the mountain, regardless of whether you feed them or not.  The weak have a 20% chance of making it to the top, regardless of whether you feed them.  You can tell the two kinds of cats apart by their characteristics, and your friend happens to like the strong ones and dislike the weak ones.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hypothesis #3: There are two kinds of cats in the world: the athletes, and the champion nappers.  The former have a 70% chance of making it to the top of the mountain if you feed them, or a 50% chance if you don't.  The latter have a 40% chance of making it to the top if you feed them, or a 20% chance if you don't.  You can tell the two apart by their characteristics, and your friend happens to like the athletes and dislike the nappers.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Hypothesis #4: There are two kinds of cats in the world: the hungry, and the overweight.  The former have a 70% chance of making it to the top of the mountain if you feed them (they're pretty fit, but they already have all the food they need, so why bother exploring?), or a 100% chance if you don't (hunger drives them to greater heights).  The latter have a 21% chance of making it to the top, if you feed them, or a 20% chance, if you don't (either way, they probably won't make it, because they are so overweight and out of shape).  You can tell the two apart by their characteristics, and your friend happens to like the hungry ones and dislike the overweight ones.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;All four hypotheses are equally consistent with the data.  The data set is useless at helping you distinguish between them.  Yet each hypothesis leads to a different prediction about which cats to feed: the first two say it doesn't matter, the third says you should feed the ones your friend likes, and the fourth says you should feed the ones your friend dislikes.  No amount of statistics or machine learning is going to enable you to make a decision based solely on the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To distinguish between these possibilities, you either need some kind of prior that will give you a basis for choosing among the models that are consistent with the data (perhaps based on your domain knowledge), or you need to conduct a controlled experiment (where the choice of whether to feed a cat is randomized, rather than being selected based on whether your friend likes the cat).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;What if we have a prior that says that feeding a cat can only help its chances of making it to the top, and not hurt its chances?  Well, that's an example of the first of two options I suggest in the last paragraph.  However, it's still not informative enough to let us decide which cats to feed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that we need to tell whether feeding the cats your friend likes improves their odds &lt;em&gt;more&lt;/em&gt; than it improves the odds of feeding the cats your friend doesn't like.  We just don't have any information on that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The chances could be 70% vs 69% for cats he likes and 90% vs 20% for cats he doesn't (i.e., cats he like have a 70% chance if fed or a 69% chance if not fed, etc.); or it could be 70% vs 0% for cats he likes and 21% vs 20% for cats he doesn't.  In the former case, you should feed the cats he &lt;em&gt;doesn't&lt;/em&gt; like, as they see a 70 percentage point improvement from feeding, while the ones he likes see only a 1 p.p. improvement.  In the latter case, you should feed the cats he does like, because they see a 70 p.p. improvement, while the ones he dislikes see only a 1 p.p. improvement.  You just can't distinguish between these two possibilities from the dataset you have.  So your prior would need to be considerably more detailed than that, before you can make any headway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you'd like to learn more about the subject and under what circumstances we &lt;em&gt;can&lt;/em&gt; draw conclusions about causality despite the lack of a controlled experiment, you might start by reading Judea Pearl's work.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2018-03-24T16:42:08.827" LastActivityDate="2018-03-24T16:42:08.827" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29490" PostTypeId="2" ParentId="29480" CreationDate="2018-03-24T05:35:54.003" Score="0" Body="&lt;p&gt;I recommend you uploading a zip file containing your images to your drive and downloading the content from drive to Colab. Then you will be able to extract them. The code for uploading is &lt;a href=&quot;https://datascience.stackexchange.com/a/27967/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-24T05:35:54.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29491" PostTypeId="2" ParentId="29488" CreationDate="2018-03-24T06:14:09.043" Score="0" Body="&lt;p&gt;Assume you are measuring the number of page visits as x and the time spent on the page as y, let’s say you measure it for each visitor/customer for a specific page.&#xA;If you create a scatter plot of x vs y, then you’ll get a Gaussian cloud of points in a 2-dimensional space. PCA can transform your data (cloud) from 2D to 1D, you can explain your data with a single variable rather than 2 variables. Once you decrease data dimensionality your problem (for example clustering or classifying the customers) would be easier to solve.&lt;/p&gt;&#xA;" OwnerUserId="47750" LastActivityDate="2018-03-24T06:14:09.043" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29492" PostTypeId="1" CreationDate="2018-03-24T06:24:15.430" Score="1" ViewCount="185" Body="&lt;p&gt;I have a model that predicts the level of injury over 3 classes: Low, Medium and High. I wish to optimize the model parameters on the scoring basis of precision. However, precision is class specific, we can determine the precision of low, medium and high separately. Is there a way to determine something like &quot;Overall Precision&quot; from the confusion matrix?&lt;/p&gt;&#xA;" OwnerUserId="49382" LastActivityDate="2022-06-02T07:01:30.220" Title="Evaluate a model based on precision for multi class classification" Tags="|multiclass-classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29494" PostTypeId="2" ParentId="9325" CreationDate="2018-03-24T12:01:13.697" Score="0" Body="&lt;p&gt;Look at &lt;a href=&quot;https://github.com/TUT-ARG/sed_eval&quot; rel=&quot;nofollow noreferrer&quot;&gt;sed_eval&lt;/a&gt; library. It is developed for evaluating event detection in audio which is a multi-label problem (as in each audio, multiple events exist). They have many evaluation options, which might fit to your needs. &#xA;You can get the true-positive rate, ... and from there computing the confusion matrix is not that hard.&lt;/p&gt;&#xA;" OwnerUserId="49290" LastActivityDate="2018-03-24T12:01:13.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29495" PostTypeId="1" AcceptedAnswerId="29500" CreationDate="2018-03-24T14:10:05.110" Score="11" ViewCount="31739" Body="&lt;p&gt;If I know the time of a given validation with set values, can I estimate the time GridSearchCV will take for n values I want to cross-validate?&lt;/p&gt;&#xA;" OwnerUserId="49278" LastEditorUserId="29169" LastEditDate="2021-07-12T16:23:05.123" LastActivityDate="2023-01-16T15:05:21.327" Title="How to estimate GridSearchCV computing time?" Tags="|cross-validation|grid-search|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29497" PostTypeId="2" ParentId="23259" CreationDate="2018-03-24T16:08:22.227" Score="3" Body="&lt;p&gt;As Kyle said on his answer word2vec can be run with the data dump data and you would get a mapping that shows the closest words, that are possible synonym candidates. Same approach is on this &lt;a href=&quot;https://www.quora.com/What-are-good-ways-to-automatically-find-synonyms-using-machine-learning-ML-techniques-What-are-good-ways-to-automatically-find-antonyms-using-ML-techniques&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora post&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://community.rapidminer.com/t5/RapidMiner-Studio-Knowledge-Base/Synonym-Detection-with-Word2Vec/ta-p/43860&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is explained&lt;/a&gt; how word2vec makes a vector of probabilities of different words and with cosine similiary (highest cosine distances) you can find the nearest ones = the synonym candidates. A code example is on &lt;a href=&quot;https://github.com/huyingxi/Synonyms/blob/master/synonyms/word2vec.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;this Github&lt;/a&gt;. There is a KDT tree used and its cosine distance. (KDT = k dimensional tree)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically a synonym is a word with enough little distance, and you can set some threshold to find all enough near ones or only the nearest. All that in code of course.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the mentioned Quora WordNet was mentioned as a source of synonyms too, but then I came up also with idea of using SE Tag Synonym dump (&lt;a href=&quot;https://meta.stackexchange.com/questions/57514/tag-synonyms-and-data-dump&quot;&gt;see here&lt;/a&gt;), where we have a superwised source of common synonyms. Those can be used as alternative source of synonyms, or as a database to verify the ones found by the distance method.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44114" LastEditDate="2018-03-26T17:03:05.023" LastActivityDate="2018-03-26T17:03:05.023" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29498" PostTypeId="1" CreationDate="2018-03-24T18:02:42.353" Score="1" ViewCount="1091" Body="&lt;p&gt;When training a neural network, I usually plot the accuracy obtained on the validation data (validation accuracy) as an intermediate measure of the network's performance – the final measure being test accuracy. This validation accuracy is measured one or more times during a single epoch, depending on the size of the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually, I stop training when the validation accuracy &lt;a href=&quot;https://stackoverflow.com/questions/34603987/when-to-stop-training-neural-networks&quot;&gt;starts dropping&lt;/a&gt; – a sign of overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for large datasets I often don't have enough computational power to wait for the validation accuracy to drop. I merely stop the training after the validation accuracy (or validation loss) stays constant for a while, like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V57jd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V57jd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically, I do not know if the model can improve further, but is it reasonable to assume that it won't? Again, this is only for large datasets with limited computing power available. With sufficient computing power available, I can afford to wait until validation accuracy drops.&lt;/p&gt;&#xA;" OwnerUserId="27583" LastActivityDate="2020-06-21T09:18:15.693" Title="Validation accuracy for neural network" Tags="|neural-network|deep-learning|cross-validation|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29499" PostTypeId="2" ParentId="29462" CreationDate="2018-03-24T19:45:39.287" Score="2" Body="&lt;p&gt;You want to &lt;em&gt;understand&lt;/em&gt; the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you run a clustering, then study how the points in a cluster differ from the points not in a cluster. Then based on these observations, you form a hypothesis. For example, you may notice that a cluster contains countries who eat a lot of fast-food and who are overweight. Then you can formulate the hypothesis that fast-food causes overweightedness, and then &lt;em&gt;test&lt;/em&gt; that hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a form of &lt;em&gt;explorative&lt;/em&gt; data analysis. There is not a mathematical function to maximize, but it is a tool for &lt;em&gt;humans&lt;/em&gt; to understand their data and then be able to formulate new hypotheses that would not have sprung to your mind otherwise.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-24T19:45:39.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29500" PostTypeId="2" ParentId="29495" CreationDate="2018-03-25T00:47:36.937" Score="12" Body="&lt;p&gt;You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search.&lt;/p&gt;&#xA;&lt;p&gt;E.g. suppose you want to use a grid search to select the hyperparameters &lt;em&gt;a&lt;/em&gt;, &lt;em&gt;b&lt;/em&gt; and &lt;em&gt;c&lt;/em&gt; of your pipeline.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;params = {'a': [1, 2, 3, 4, 5],&#xA;          'b': [1, 2, 3, 4],&#xA;          'c': [1, 2, 3]}&#xA;&#xA;cv = GridSearchCV(pipeline, params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;By default this should run a search for a grid of &lt;span class=&quot;math-container&quot;&gt;$5 \cdot 4 \cdot 3 = 60$&lt;/span&gt; different parameter combinations. The default cross-validation is a 3-fold cv so the above code should train your model &lt;span class=&quot;math-container&quot;&gt;$60 \cdot 3 = 180$&lt;/span&gt; times. By default GridSearch runs parallel on your processors, so depending on your hardware you should divide the number of iterations by the number of processing units available. Let's say for example I have 4 processors available, each processor should fit the model &lt;span class=&quot;math-container&quot;&gt;$ 180 / 4 = 45$&lt;/span&gt; times. Now, if on average my model takes &lt;span class=&quot;math-container&quot;&gt;$10 sec$&lt;/span&gt; to train, I'm estimating around &lt;span class=&quot;math-container&quot;&gt;$45 \cdot 10 / 60 = 7.5min$&lt;/span&gt; training time. In practice it should be closer to &lt;span class=&quot;math-container&quot;&gt;$8min$&lt;/span&gt; due to overhead.&lt;/p&gt;&#xA;&lt;p&gt;Finally, because some parameters heavily affect the training time of that algorithm, I would suggest using the &lt;code&gt;max_iter&lt;/code&gt; argument whenever available so that your estimation doesn't fall far off.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Please note :&#xA;As of July 2021, the default folds is 5.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;From sklearn documentation :&#xA;Changed in version 0.22: cv default value if None changed from 3-fold to 5-fold.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="34269" LastEditorUserId="118856" LastEditDate="2021-07-12T06:50:42.600" LastActivityDate="2021-07-12T06:50:42.600" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="29501" PostTypeId="1" CreationDate="2018-03-25T02:48:47.183" Score="6" ViewCount="5665" Body="&lt;p&gt;I have many images that I want to plot as a result of running t-SNE and I want to be able to interactively explore them.  matplotlib does not allow enough interactivity to explore, and plotly is &lt;a href=&quot;https://github.com/plotly/plotly.py/issues/978&quot; rel=&quot;noreferrer&quot;&gt;too slow&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there an easy way to plot thousands of images in a way that different patterns can be explored?&lt;/p&gt;&#xA;" OwnerUserId="49411" LastEditorUserId="49411" LastEditDate="2018-03-25T04:51:14.480" LastActivityDate="2018-06-05T12:02:11.633" Title="How to create interactive plot of thousands of images as output of t-SNE?" Tags="|python|visualization|dimensionality-reduction|tsne|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29502" PostTypeId="2" ParentId="29501" CreationDate="2018-03-25T03:08:19.710" Score="7" Body="&lt;p&gt;It is certainly possible to create interactive plots of many thousands of images, as Google has done in their interactive &lt;a href=&quot;https://artsexperiments.withgoogle.com/tsnemap/&quot; rel=&quot;noreferrer&quot;&gt;art t-SNE Map&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, as far as I have found, there is not any canned way to do this.  &lt;a href=&quot;https://github.com/YaleDHLab/pix-plot&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/YaleDHLab/pix-plot&lt;/a&gt; is an open source application that also does an impressive job at plotting thousands of images, and if you are using python (e.g. with scikit-learn's t-SNE), it does not look too bad to modify to your use case.&lt;/p&gt;&#xA;" OwnerUserId="49411" LastEditorUserId="49411" LastEditDate="2018-03-25T03:33:09.157" LastActivityDate="2018-03-25T03:33:09.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29503" PostTypeId="2" ParentId="29462" CreationDate="2018-03-25T03:30:53.217" Score="1" Body="&lt;p&gt;Clustering is a really subjective problem. In most cases, you have a data set of unlabeled samples. One scenario is that you know there are k groups or clusters in the data, you just need to find those k clusters. Here, k-means or Gaussian mixture models (or any other relevant method) can be used to discover your clusters. Another scenario would be that you even don’t know how many clusters or groups exists in your data, you want to find the number of clusters using a clustering method. For example, a clustering algorithm which doesn't need to know the number of clusters beforehand like x-means. And in some cases you want to preprocess your data, partition it and then use the results in a supervised learning algorithm.&lt;/p&gt;&#xA;" OwnerUserId="47750" LastActivityDate="2018-03-25T03:30:53.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29505" PostTypeId="2" ParentId="29501" CreationDate="2018-03-25T04:30:06.133" Score="7" Body="&lt;p&gt;&lt;a href=&quot;http://datashader.org/&quot; rel=&quot;noreferrer&quot;&gt;Datashader&lt;/a&gt; is a Python visualization library designed to handle large datasets. A tutorial to plot t-SNE with datashader can be found &lt;a href=&quot;https://jlorince.github.io/viz-tutorial/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-25T04:30:06.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29507" PostTypeId="1" AcceptedAnswerId="29510" CreationDate="2018-03-25T09:42:07.623" Score="7" ViewCount="3573" Body="&lt;p&gt;Can &lt;code&gt;CNNs&lt;/code&gt; predict well if they are trained on canonical-like images but tested on a version of images that are little bit shifted? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried it using &lt;code&gt;mnist&lt;/code&gt; dataset and found the contrary. The accuracy of the test set that was shifted was very low as compared to &lt;code&gt;MLPs&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="38506" LastEditorUserId="28175" LastEditDate="2018-12-28T22:39:27.153" LastActivityDate="2018-12-28T22:39:27.153" Title="Are CNNs insensitive to rotations and shifts in images?" Tags="|machine-learning|neural-network|deep-learning|cnn|mnist|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29510" PostTypeId="2" ParentId="29507" CreationDate="2018-03-25T10:47:50.597" Score="9" Body="&lt;p&gt;If you use &lt;code&gt;max-pooling&lt;/code&gt; layers, they may be insensetive to small shifts but not that much. If you want your network to be able to be invariant to transformations, such as translations and shifts or other types of customary transformations, you have two solutions, at least as far as I know:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Increasing the size of data-set&lt;/li&gt;&#xA;&lt;li&gt;Using spatial transformers &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/27237/28175&quot;&gt;&#xA;What is the state-of-the art ANN architecture for MNIST&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/a/26291/28175&quot;&gt;Why do convolutional neural networks work&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks to one of our friends, another way is to use transfer learning after data-augmentation. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-12-28T22:38:21.833" LastActivityDate="2018-12-28T22:38:21.833" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="29515" PostTypeId="2" ParentId="29507" CreationDate="2018-03-25T14:38:09.630" Score="3" Body="&lt;p&gt;Convolution is shift-equivariant except for border effects. Fully connected layers aren't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pooling (without subsampling/stride) can be seen as a kind of smoothing, and its output is often the same for many neighboring positions. Subsampling this (applying a stride) results in an operation that is fairly invariant to small shifts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Global pooling is fully shift invariant, again except for border effects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for rotations: standard architectures aren't inherently robust to those.&lt;/p&gt;&#xA;" OwnerUserId="49426" LastActivityDate="2018-03-25T14:38:09.630" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29518" PostTypeId="2" ParentId="29439" CreationDate="2018-03-25T16:30:12.663" Score="0" Body="&lt;p&gt;Yes, it is valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kernel is just an inner product in the feature space. In fact, here you have explicitly started from the feature transformation and we understand the feature transformation. &lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-03-25T16:30:12.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29520" PostTypeId="1" AcceptedAnswerId="29834" CreationDate="2018-03-25T18:31:12.973" Score="10" ViewCount="12808" Body="&lt;p&gt;I would appreciate if you could let me know in the following example code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import Counter&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import train_test_split,StratifiedKFold,learning_curve,validation_curve,GridSearchCV&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.metrics import classification_report&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def plot_learning_curve(train_sizes, train_scores, test_scores, title, alpha=0.1):&#xA;    train_mean = np.mean(train_scores, axis=1)&#xA;    train_std = np.std(train_scores, axis=1)&#xA;    test_mean = np.mean(test_scores, axis=1)&#xA;    test_std = np.std(test_scores, axis=1)&#xA;    plt.plot(train_sizes, train_mean, label='train score', color='blue', marker='o')&#xA;    plt.fill_between(train_sizes, train_mean + train_std,&#xA;                     train_mean - train_std, color='blue', alpha=alpha)&#xA;    plt.plot(train_sizes, test_mean, label='test score', color='red', marker='o')&#xA;    plt.fill_between(train_sizes, test_mean + test_std, test_mean - test_std, color='red', alpha=alpha)&#xA;    plt.title(title)&#xA;    plt.xlabel('Number of training points')&#xA;    plt.ylabel('F-measure')&#xA;    plt.grid(ls='--')&#xA;    plt.legend(loc='best')&#xA;    plt.show()&#xA;&#xA;&#xA;def plot_validation_curve(param_range, train_scores, test_scores, title, alpha=0.1):&#xA;    train_mean = np.mean(train_scores, axis=1)&#xA;    train_std = np.std(train_scores, axis=1)&#xA;    test_mean = np.mean(test_scores, axis=1)&#xA;    test_std = np.std(test_scores, axis=1)&#xA;    plt.plot(param_range, train_mean, label='train score', color='blue', marker='o')&#xA;    plt.fill_between(param_range, train_mean + train_std,&#xA;                     train_mean - train_std, color='blue', alpha=alpha)&#xA;    plt.plot(param_range, test_mean, label='test score', color='red', marker='o')&#xA;    plt.fill_between(param_range, test_mean + test_std, test_mean - test_std, color='red', alpha=alpha)&#xA;    plt.title(title)&#xA;    plt.grid(ls='--')&#xA;    plt.xlabel('Parameter value')&#xA;    plt.ylabel('F-measure')&#xA;    plt.legend(loc='best')&#xA;    plt.show()&#xA;&#xA;X, y = make_classification(n_classes=2, class_sep=2,weights=[0.9, 0.1], n_informative=3, n_redundant=1, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)&#xA;print('Original dataset shape {}'.format(Counter(y)))&#xA;&#xA;ln = X.shape&#xA;names = [&quot;x%s&quot; % i for i in range(1, ln[1] + 1)]&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)&#xA;st=StandardScaler()&#xA;&#xA;rg = LogisticRegression(class_weight = { 0:1, 1:6.5 }, random_state = 42, solver = 'saga',max_iter=100,n_jobs=-1)&#xA;&#xA;param_grid = {'clf__C': [0.001,0.01,0.1,0.002,0.02,0.005,0.0007,.0006,0.0005],&#xA;              'clf__class_weight':[{ 0:1, 1:6 },{ 0:1, 1:4 },{ 0:1, 1:5.5 },{ 0:1, 1:4.5 },{ 0:1, 1:5 }]&#xA;              }&#xA;&#xA;pipeline = Pipeline(steps=[('scaler', st),&#xA;                           ('clf', rg )])&#xA;&#xA;cv=StratifiedKFold(n_splits=5,random_state=42)&#xA;rg_cv = GridSearchCV(pipeline, param_grid, cv=cv, scoring =  'f1')&#xA;rg_cv.fit(X_train, y_train)&#xA;print(&quot;Tuned rg best params: {}&quot;.format(rg_cv.best_params_))&#xA;&#xA;ypred = rg_cv.predict(X_train)&#xA;print(classification_report(y_train, ypred))&#xA;print('######################')&#xA;ypred2 = rg_cv.predict(X_test)&#xA;print(classification_report(y_test, ypred2))&#xA;&#xA;plt.figure(figsize=(9,6))&#xA;param_range1=[i / 10000.0 for i in range(1, 11)]&#xA;param_range2=[{0: 1, 1: 6}, {0: 1, 1: 4}, {0: 1, 1: 5.5}, {0: 1, 1: 4.5}, {0: 1, 1: 5}]&#xA;&#xA;if __name__ == '__main__':&#xA;    train_sizes, train_scores, test_scores = learning_curve(&#xA;              estimator= rg_cv.best_estimator_ , X= X_train, y = y_train,&#xA;                train_sizes=np.arange(0.1,1.1,0.1), cv= cv,  scoring='f1', n_jobs= - 1)&#xA;&#xA;    plot_learning_curve(train_sizes, train_scores, test_scores, title='Learning curve for Logistic Regression')&#xA;&#xA;    train_scores, test_scores = validation_curve(&#xA;        estimator=rg_cv.best_estimator_, X=X_train, y=y_train, param_name=&quot;clf__C&quot;, param_range=param_range1,&#xA;        cv=cv, scoring=&quot;f1&quot;, n_jobs=-1)&#xA;&#xA;    plot_validation_curve(param_range1, train_scores, test_scores, title=&quot;Validation Curve for C&quot;, alpha=0.1)&#xA;&#xA;    train_scores, test_scores = validation_curve(&#xA;        estimator=rg_cv.best_estimator_, X=X_train, y=y_train, param_name=&quot;clf__class_weight&quot;, param_range=param_range2,&#xA;        cv=cv, scoring=&quot;f1&quot;, n_jobs=-1)&#xA;&#xA;    plot_validation_curve(param_range2, train_scores, test_scores, title=&quot;Validation Curve for class_weight&quot;, alpha=0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Why when the  best estimator of GridSearchCv is passed into the learning curve function, it prints all the previous print lines several times?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How to plot validation curve for class weight? &lt;code&gt;TypeError: float() argument must be a string or a number, not 'dict'&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="29169" LastEditDate="2019-06-07T16:36:00.770" LastActivityDate="2019-12-14T23:17:04.600" Title="How to plot learning curve and validation curve while using pipeline" Tags="|classification|scikit-learn|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="29521" PostTypeId="1" AcceptedAnswerId="29523" CreationDate="2018-03-25T19:39:20.800" Score="1" ViewCount="334" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/28894/47359&quot;&gt;This&lt;/a&gt; answer shows that linear and polynomial function weights can be trained using this matrix operation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w = (X^TX)^{-1}X^Ty$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, algorithms such as gradient descent are not necessary for these functions. By my understanding, gradient descent for linear regression model finds perfect derivative for each weight so that cost function is minimal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before asking for connections between gradient descent and equation above, let's separate the equation in smaller steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X = [1,2,3,4,5,6,7,8,9,10,11,12]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's turn $X$ vector to matrix by adding column of 1's which will be used to train the bias value:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{bmatrix}&#xA;    1 &amp;amp; 1 \\&#xA;    2 &amp;amp; 1 \\&#xA;    3 &amp;amp; 1 \\&#xA;    4 &amp;amp; 1 \\&#xA;    5 &amp;amp; 1 \\&#xA;    6 &amp;amp; 1 \\&#xA;    7 &amp;amp; 1 \\&#xA;    8 &amp;amp; 1 \\&#xA;    9 &amp;amp; 1 \\&#xA;    10 &amp;amp; 1 \\&#xA;    11 &amp;amp; 1 \\&#xA;    12 &amp;amp; 1&#xA;  \end{bmatrix}&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Transpose: $X^T$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{bmatrix}&#xA;     1 &amp;amp; 2 &amp;amp; 3 &amp;amp; 4 &amp;amp; 5 &amp;amp; 6 &amp;amp; 7 &amp;amp; 9 &amp;amp; 10 &amp;amp; 11 &amp;amp; 12 \\&#xA;     1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &amp;amp; 1 &#xA;  \end{bmatrix}&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Matrix multiplication $X^TX$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{bmatrix}&#xA;     650 &amp;amp; 78 \\&#xA;     78 &amp;amp; 12&#xA;  \end{bmatrix}&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Inverse matrix operation $(X^TX)^{-1}$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{bmatrix}&#xA;     0.00699301 &amp;amp; -0.04545455 \\&#xA;     -0.04545455 &amp;amp; 0.37878788&#xA;  \end{bmatrix}&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Matrix multiplication $(X^TX)^{-1}X^T$:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;array([[-0.03846154, -0.03146853, -0.02447552, -0.01748252, -0.01048951,&#xA;        -0.0034965 ,  0.0034965 ,  0.01048951,  0.01748252,  0.02447552,&#xA;         0.03146853,  0.03846154],&#xA;       [ 0.33333333,  0.28787879,  0.24242424,  0.1969697 ,  0.15151515,&#xA;         0.10606061,  0.06060606,  0.01515152, -0.03030303, -0.07575758,&#xA;        -0.12121212, -0.16666667]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Matrix vector multiplication $(X^TX)^{-1}X^Ty$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{bmatrix}&#xA;     0.01174825 &amp;amp; 2.30530303 &#xA;  \end{bmatrix}&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This seems to be the perfect slope for minimizing cost. But I'm unable to understand how does it exactly work. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like that the algorithm such as gradient descent would've took extremely large amount of iterations to estimate perfect slope and bias value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How/why does this equation exactly work? Is it somehow related to differentiation? Could it be compared to algorithms such as gradient descent? If so, how? Is it possible to use this equation with sigmoid functions?&lt;/p&gt;&#xA;" OwnerUserId="47359" LastEditorUserId="47359" LastEditDate="2018-03-25T22:14:10.270" LastActivityDate="2018-03-25T22:14:10.270" Title="How does &quot;linear algebraic&quot; weight training function work?" Tags="|machine-learning|linear-regression|linear-algebra|normal-equation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29522" PostTypeId="2" ParentId="29521" CreationDate="2018-03-25T20:09:15.260" Score="1" Body="&lt;p&gt;First of all, gradient descent cannot find the global optimum. If your function has just one extremum, it can find it but if it has lots of them there is no guarantee that it finds the best one. If you are familiar with derivative and slope of functions, the Normal equation, tries to find the point which all the derivative is equal to zero for all directions, variables. It is the result of the derivatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are somehow right, mathematically we know the optimal solution of all ML problems but in practice, normal equations do not work. The reason is that if you increase the number of features, you will have a matrix which gets larger and larger. Most problems are not linearly separable. Consequently, you have to add high order polynomials and the point is that you don't know which polynomial to use. If you use just the second order polynomial you will get a matrix that won't be located in your computer's memory because of being so much large. suppose that you have &lt;code&gt;100&lt;/code&gt; features. If you just add the combination of multiplication of two variables, compute how many entries your matrix will have. And if you do that you can be sure that it does not add too much complexity to your model, because you've not added complex higher order polynomials whilst you need them. ML algorithms like deep-nets try to learn so many complicated functions using the smaller number of entries.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How/why does this equation exactly work? Could it be compared to algorithms such as gradient descent? If so, how? Is it possible to use this for more complex activation functions such as hyperbolic tangent?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is the result of taking the multivariable derivative, gradient. Yes the goal of them is to find the local minima, but they have a different approach. Gradient descent tries to go downhill whilst normal equation tries to find the location by finding where the derivative is zero for all features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't figure out the last question to help you address it.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-25T20:09:15.260" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29523" PostTypeId="2" ParentId="29521" CreationDate="2018-03-25T20:12:08.403" Score="2" Body="&lt;p&gt;Your goal is to find a $w$ such that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Xw \approx y$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and way to model this problem is to minimize the objective function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\min_w\|Xw-y\|^2.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiating with respect to $w$ and equate it to zero gives us&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$2X^T(Xw-y)=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X^TXw-X^Ty=0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$X^TXw=X^Ty$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if $X^TX$ is invertible, then we have &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$w=(X^TX)^{-1}(X^Ty)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remark:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We tend to avoid computing inverse and prefer gradient-based method. Complexity of the normal equation method is cubic.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-03-25T20:12:08.403" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29526" PostTypeId="1" AcceptedAnswerId="29538" CreationDate="2018-03-25T20:32:20.150" Score="14" ViewCount="4474" Body="&lt;p&gt;In the famous &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap1.html&quot; rel=&quot;noreferrer&quot;&gt;Deep Learning Book&lt;/a&gt;, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; is the set of all weights and &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt; the set of all biases, &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; is the number of training inputs, x is the set of all training inputs, y(x) is the expected output of the network for input x, and &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; is the actual output of the network for input &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;, with respect to &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most of this formula seems very clear to be, except the &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt; in the denominator. If I understand it correctly, we are summing up the squared vector length of (the actual output minus its expected output), for each training input (giving us the total squared error for the training set) and then divide this by the number of training samples, to get the mean squared error of all training samples. Why do we divide this by &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt; as well then?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other places I've seen that Andrew Ng's lecture defines the Mean Square cost in a similar way, also with the &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt; in the denominator, so this seems to be a common definition.&lt;/p&gt;&#xA;" OwnerUserId="47561" LastEditorUserId="29575" LastEditDate="2019-04-05T13:35:29.113" LastActivityDate="2020-04-13T08:16:09.720" Title="Why is there a $2$ at the denominator of the mean squared error function?" Tags="|neural-network|loss-function|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29527" PostTypeId="1" CreationDate="2018-03-25T20:43:33.720" Score="9" ViewCount="6983" Body="&lt;p&gt;I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to use the same auto encoders as the ones I used for the images I get bad results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess the reason for this is that my text is sparse and I have a big vocabulary size of 500K words. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Do you have a link of a working example of an auto encoder for text in Keras?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I saw that in most papers they use cross-entropy as a loss function. How does cross-entropy calculate the loss exactly ? Does it make sense to use cross-entropy even if I do a character by character auto encoder?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47600" LastEditorUserId="29575" LastEditDate="2018-03-26T00:34:25.553" LastActivityDate="2018-11-22T00:01:35.257" Title="Which type auto encoder gives best results for text" Tags="|python|keras|autoencoder|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29529" PostTypeId="1" AcceptedAnswerId="29537" CreationDate="2018-03-25T23:34:19.823" Score="0" ViewCount="4706" Body="&lt;p&gt;I have a very little data, so my word2vec model does not perform well. My intention is to identify words similar to technical terms such as 'support vector machine', 'machine learning', 'artificial intelligence' etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in knowing if I can use the Google's wikipedia model for this. But according to my model most of the words I will be dealing with are n-grams. Hence, how can I utilise this Google's wikipedia model that is based on unigrams to achieve my task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am happy to provide more examples if needed :)&lt;/p&gt;&#xA;" OwnerUserId="45262" LastActivityDate="2018-03-27T14:48:36.777" Title="How to alter word2vec wikipedia model for n-grams?" Tags="|neural-network|nlp|text-mining|word2vec|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29531" PostTypeId="2" ParentId="29527" CreationDate="2018-03-26T01:25:08.453" Score="4" Body="&lt;p&gt;A working example of a Variational Autoencoder for Text Generation in Keras can be found &lt;a href=&quot;https://github.com/Toni-Antonova/VAE-Text-Generation/blob/master/vae_nlp.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability value between 0 and 1 for classification. Cross-entropy loss goes up as the predicted probability diverges from the actual label. In the case of character-by-character autoencoder, each character in the vocabulary would be a label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross-entropy works if the input and output are the same size, that is the case in character-by-character autoencoder. Often times in text analysis, the input and output sequences are different lengths so a second term, &lt;a href=&quot;https://cs224d.stanford.edu/reports/OshriBarak.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;encoder loss&lt;/a&gt;, is added to the objective function. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-26T01:25:08.453" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29533" PostTypeId="1" CreationDate="2018-03-26T04:58:48.380" Score="0" ViewCount="215" Body="&lt;p&gt;In the popular example for Bayesian Belief network &lt;strong&gt;&lt;a href=&quot;https://ocw.mit.edu/courses/health-sciences-and-technology/hst-950j-biomedical-computing-fall-2010/lectures-and-readings/MITHST_950JF10_lec14.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Burglary Alarm&lt;/a&gt;&lt;/strong&gt; how is the probability for burglary P(B) and earthquake P(E) calculated as 0.001 and 0.002 respectively?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it an assumption made or there is some calculation involved ? I can understand the conditional probabilities for the child nodes but not sure how the probability for the nodes Burglary and Earthquake are getting calculated?&lt;/p&gt;&#xA;" OwnerUserId="46326" LastEditorUserId="46326" LastEditDate="2018-03-26T10:17:50.380" LastActivityDate="2019-03-29T11:52:26.250" Title="How to calculate the Probability for the Unconditional Node in the Bayesian Belief Network?" Tags="|bayesian|bayesian-networks|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29534" PostTypeId="2" ParentId="29526" CreationDate="2018-03-26T05:35:50.117" Score="1" Body="&lt;p&gt;It's just that it helps when we do the backprop of error while differentiating, as the &lt;code&gt;2&lt;/code&gt; in the denominator cancels out with the &lt;code&gt;2&lt;/code&gt; which we will get from the exponent..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it's just a matter of  Mathematical Convenience...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-26T06:28:15.417" LastActivityDate="2018-03-26T06:28:15.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29537" PostTypeId="2" ParentId="29529" CreationDate="2018-03-26T06:42:05.580" Score="1" Body="&lt;p&gt;To clarify, an n-gram usually refers to a sequence of characters, i.e. the word &quot;clear&quot; is comprised of the trigrams {cle, lea, ear}. I think the term you are looking for is &quot;multi-word phrases&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Embedding collections of words is referred to a couple different ways, including &quot;sent2vec&quot;, &quot;doc2vec&quot; or &quot;thought vectors&quot;. These terms generally refer to embedding a &quot;complete&quot; set of words,  either a sentence, paragraph, collection of paragraphs, or a document. A common --&#xA;if somewhat inelegant -- approach to using a pre-trained word2vec model to embed multiple words is to embed each word separately and then take the average as the embedding for their combination. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you'll find this article relevant: &lt;a href=&quot;https://arxiv.org/abs/1607.00570&quot; rel=&quot;nofollow noreferrer&quot;&gt;Representation learning for very short texts using weighted word embedding aggregation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="4683" LastEditorUserId="4683" LastEditDate="2018-03-26T15:01:12.313" LastActivityDate="2018-03-26T15:01:12.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29538" PostTypeId="2" ParentId="29526" CreationDate="2018-03-26T06:49:18.967" Score="16" Body="&lt;p&gt;This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also watch &lt;a href=&quot;https://www.youtube.com/watch?v=_PwhiWxHK8o&quot; rel=&quot;noreferrer&quot;&gt;this video on SVM lecture&lt;/a&gt; by Patrick Winston, where he uses a similar formula and then tells that he is using $2$ in denominator just for mathematical convenience.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-26T06:49:18.967" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29541" PostTypeId="1" AcceptedAnswerId="29718" CreationDate="2018-03-26T10:11:59.727" Score="6" ViewCount="2203" Body="&lt;p&gt;I am performing a comparison among time series by using Dynamic Time Warping (DTW). However, it is not a real distance, but a distance-like quantity, since it doesn't assure the triangle inequality to hold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reminder:&lt;code&gt;d:MxM-&amp;gt;R&lt;/code&gt; is a distance if for all x,y in M:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 - d(x,y) ≥ 0, and d(x,y) = 0 if and only if x = y&#xA;2 - It is symmetric: d(x,y) = d(y,x)&#xA;3 - Triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is any equivalent measure that ensures the condition of &lt;em&gt;distance&lt;/em&gt; in a matemathical sense? Obviously, I am not looking for a Euclidean distance, but one that ensures the proper classification of my series in a future clustering. &#xA;If so, there is any solid implementation in a R or Python package?&lt;/p&gt;&#xA;" OwnerUserId="49226" LastActivityDate="2018-03-30T10:26:06.020" Title="Alternative distance to Dynamic Time Warping" Tags="|machine-learning|time-series|distance|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29542" PostTypeId="1" CreationDate="2018-03-26T10:13:48.657" Score="4" ViewCount="719" Body="&lt;p&gt;I've the following somewhat unusual background and I've managed (probably by luck) to get an industry job of a computer vision researcher using deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My background:&lt;/strong&gt; I've a PhD in pure math, and have the following machine learning experience: linear and logistic regressions, support vector machines (SVM), linear and quadratic discriminant analyses (LDA/QDA), decision trees, PCA. I'm familiar with the mathematical theory of all the above, and implemented the LDA/QDA, SVM, PCA in matlab, where I'm proficient in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my experience with Python and deep learning are very limited to zero. I don't know scikit learn, only somewhat familiar with NumPy arrays, but still have problems with the basics like concatenation with empty arrays. I don't know for examples the data structures like classes or lists in Python-just to give you an idea. I know what a neural network is, but don't know anything else, e.g. backpropagation or autoencoder. Hence I don't know the theory of deep learning as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Time constraint:&lt;/strong&gt; So I'm in a situation where I'll have to gain some hands on experience in both Python as well as deep learning, possibly in 2 months (the company has a 4 months trial period and use only Python, but they'll get a sense of my learning in 2 months, hence I've to learn fast).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; Given my expertise and time constraints, could you mention a &lt;strong&gt;road map&lt;/strong&gt; (if possible) so that I can get a &lt;em&gt;fast introduction to deep learning and its Python libraries&lt;/em&gt; (e.g. Tensor Flow or Keras etc.)? I know there're tons of resources out there, but not all of them can be learnt within a short time span.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks so much in advance!!!&lt;/p&gt;&#xA;" OwnerUserId="33198" LastEditorUserId="33198" LastEditDate="2018-03-26T10:20:18.500" LastActivityDate="2022-07-25T12:15:13.603" Title="Fast introduction to deep learning in Python, with advanced math and some machine learning backgrounds, but not much Python experience" Tags="|machine-learning|python|neural-network|deep-learning|" AnswerCount="6" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29543" PostTypeId="1" AcceptedAnswerId="29545" CreationDate="2018-03-26T10:20:57.750" Score="-1" ViewCount="36" Body="&lt;p&gt;I use a convnet to classify two types of objects: class A and B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created the data set myself and have around 1000 examples per class. Some are really obvious and clear, some others are very difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: After the training, is it a good idea to retrain the model just on the data that are not correctly classified ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fore example, by taking 50% of the examples that are the least correctly classify and retrain for some epochs on them only. My intuition is that it would help the network focus on non-obvious features that could have been learnt during the first training phase .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I am an idiot. A training on a subset of the data will necessarily result in an increase of the loss for the rest of the data (except if there are some magic tricks I don't know about). The only good solution is to design a custom loss function that fits my needs for the first training.&lt;/p&gt;&#xA;" OwnerUserId="28812" LastEditorUserId="29169" LastEditDate="2021-02-13T16:35:40.087" LastActivityDate="2021-02-13T16:35:40.087" Title="Fine train a convnet on difficult data only?" Tags="|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29544" PostTypeId="2" ParentId="29542" CreationDate="2018-03-26T11:09:47.307" Score="1" Body="&lt;p&gt;First of all, you CAN learn them in 2 months time, if you are going to devote everyday for this. As per your background, you don't need much of the basics of ML, you just need Python. There are many answers for the same here-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/What-is-the-best-way-to-learn-Python-from-scratch&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/What-is-the-best-way-to-learn-Python-from-scratch&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simple roadmap for python here would be-&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Download and install python in your PC&lt;/li&gt;&#xA;&lt;li&gt;Start with a course and book (related to basic python concepts and programming) and start coding. I recommend both course and book because not everything is in a book and not everything is told in the courses.&lt;/li&gt;&#xA;&lt;li&gt;As soon as you start creating smaller working scripts, go to &lt;a href=&quot;http://www.hackerrank.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;hackerrank&lt;/a&gt; and start doing exercises there. This will develop your understanding of data structures and which data structure to use in which type of problem. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This will help you in writing scripts in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For understanding deep learning, you can go to this-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/tutorial/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This tutorial uses &lt;a href=&quot;http://deeplearning.net/software/theano/introduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano&lt;/a&gt; for building neural network models. It also explains many terms related to deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally (you already know), practice is what you want to learn in this short time. And your biggest positive point is that you have a PhD in Maths, so you can easily understand the mathematics behind deep learning models.&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-26T11:09:47.307" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29545" PostTypeId="2" ParentId="29543" CreationDate="2018-03-26T11:21:06.640" Score="0" Body="&lt;p&gt;I am an idiot. A training on a subset of the data will necessarily result in an increase of the loss for the rest of the data (except if there are some magic tricks I don't know about). The only good solution is to design a custom loss function that fits my needs for the first training.&lt;/p&gt;&#xA;" OwnerUserId="28812" LastActivityDate="2018-03-26T11:21:06.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29546" PostTypeId="2" ParentId="26639" CreationDate="2018-03-26T11:28:38.323" Score="0" Body="&lt;p&gt;For regression, typically the activation function stays linear&lt;/p&gt;&#xA;" OwnerUserId="49470" LastActivityDate="2018-03-26T11:28:38.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29548" PostTypeId="2" ParentId="9681" CreationDate="2018-03-26T14:24:27.193" Score="4" Body="&lt;p&gt;It's kind of ugly but if you want a subsection to fold up inside the section but also be foldable on its own you can use curly braces at either end of the subsection.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# SECTION ONE --------------------------------------------------------------&#xA;y &amp;lt;- 11:20&#xA;&#xA;{# SUBSECTION ONE ----------------------------------------------------------&#xA;x &amp;lt;- 1:10&#xA;}   &#xA;&#xA;# SECTION TWO --------------------------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This way you can have subsection one folded while still seeing the code in section one. But if you fold section one subsection one will also be closed.&lt;/p&gt;&#xA;" OwnerUserId="49477" LastActivityDate="2018-03-26T14:24:27.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29549" PostTypeId="2" ParentId="26639" CreationDate="2018-03-26T14:29:20.540" Score="2" Body="&lt;p&gt;If you want to use an LSTM for time series prediction, you have to try every possible parameter to find the best possible result. In order to check your model performance, you should also check your train/validation loss (this will probably show that 10 epochs is too few). Check your network architecture. Why do you use three layers? More layers won't mean better results. Try different optimizers with different learning rates. Finally, ask yourself why you are even using an LSTM for prediction? Have you tried other methods? LSTMs aren't magic, they won't have perfect prediction without any pattern to draw from, and you have pretty small samples for training.&lt;/p&gt;&#xA;" OwnerUserId="19161" LastActivityDate="2018-03-26T14:29:20.540" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29551" PostTypeId="1" CreationDate="2018-03-26T14:48:06.253" Score="3" ViewCount="648" Body="&lt;p&gt;Suppose I have a neural network which accepts two sets of features as inputs and generates corresponding outputs, for instance, generate average final grade from: 1. working hours for N students in a class, 2. mid-term grade for N students in the same class.  During training, the neural network is trained with many different classes. What I would like to during inference is to turn off one set of the features (i.e. feed only working hours as inputs) and predict outputs using the same neural network.  Obviously, setting the mid-term grades to be all 0s would not be a good option. I wonder if anyone know what would be a good way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="24887" LastEditorUserId="1330" LastEditDate="2018-09-23T17:44:50.230" LastActivityDate="2019-08-24T23:02:20.323" Title="turn off parts of features in a neural network?" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29552" PostTypeId="2" ParentId="29529" CreationDate="2018-03-26T14:51:38.470" Score="0" Body="&lt;p&gt;If you want to model the unique meaning of commonly occurring n-grams, often times called &lt;a href=&quot;https://en.wikipedia.org/wiki/Collocation&quot; rel=&quot;nofollow noreferrer&quot;&gt;collocations&lt;/a&gt;, the best solution is to train a new embedding space to model those specific semantics. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aggregating existing embeddings will only capture part of the collocation meaning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do not have very much data, you can do fine-tuning/transfer learning. Fine-tuning/transfer learning takes an existing model architecture and weights, then does additionally training with more data. In this case, take Google's Wikipedia model and train it with your collocations. An example of fine-tuning for word2vec in Keras can be found &lt;a href=&quot;https://github.com/PacktPublishing/Deep-Learning-with-Keras/blob/f558d37187514b003a98cb2c5def35d7928b923f/Chapter05/finetune_word2vec_embeddings.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-03-27T14:48:36.777" LastActivityDate="2018-03-27T14:48:36.777" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29553" PostTypeId="2" ParentId="29551" CreationDate="2018-03-26T15:10:57.437" Score="1" Body="&lt;p&gt;As far as I know, you cannot do that.&lt;br&gt;&#xA;First : &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Obviously setting the mid-term grades to be all 0s would be be a good&#xA;  option&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, actually that is a really bad option, neural networks do not understand magical parameters, meaning, if you put a 0 there it will think that the mid-term grades is 0, therefore it will probably give you a very low final grade prediction for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My advise, if you are very interested in doing that, create boolean factors, training inputs:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;working_hours&lt;/li&gt;&#xA;&lt;li&gt;mid-term_grade&lt;/li&gt;&#xA;&lt;li&gt;mid-term_grade_present&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;mid-term_grade_present should be a 1 when you have the mid-term_grade training data, and 0 when you do not have it. BTW, yes, generate training data without the mid-term_grade.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are working with non linear machine learning algorithms that should be enough, if you are using linear algorithms you should one hot encode the mid-term_grade_present and multiply it by the mid-term_grade, ending with something like this:  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;working_hours&lt;/li&gt;&#xA;&lt;li&gt;mid-term_grade_present_1 * mid-term_grade&lt;/li&gt;&#xA;&lt;li&gt;mid-term_grade_present_0 * mid-term_grade&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="45431" LastActivityDate="2018-03-26T15:10:57.437" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29555" PostTypeId="2" ParentId="28177" CreationDate="2018-03-26T17:25:06.093" Score="12" Body="&lt;p&gt;The default value for &lt;code&gt;spark.sql.shuffle.partitions&lt;/code&gt; is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dataframe.repartition('id')&lt;/code&gt; creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's, you'll end up with inconsistently sized partitions.&lt;/p&gt;&#xA;" OwnerUserId="49483" LastEditorUserId="24000" LastEditDate="2018-03-27T01:39:12.860" LastActivityDate="2018-03-27T01:39:12.860" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29557" PostTypeId="2" ParentId="14524" CreationDate="2018-03-26T20:13:42.567" Score="1" Body="&lt;p&gt;Thought I'd add a selection of what's of some of the 'best' stuff out there to suit different learning styles :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Literature&#xA;- &lt;a href=&quot;https://arxiv.org/abs/1710.09829&quot; rel=&quot;nofollow noreferrer&quot;&gt;Original paper&lt;/a&gt; by Sara Sabour, Nicholas Frosst, Geoffrey E Hinton&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Blog post&#xA;- &lt;a href=&quot;https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b&quot; rel=&quot;nofollow noreferrer&quot;&gt;Understanding Hinton’s Capsule Networks. Part I: Intuition&lt;/a&gt; by Max Pechyonkin&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Youtube video&#xA;- &lt;a href=&quot;https://www.youtube.com/watch?v=pPN8d0E3900&quot; rel=&quot;nofollow noreferrer&quot;&gt;Capsule Networks (CapsNets) – Tutorial&lt;/a&gt; by Aurélien Géron&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Implementation&#xA;- &lt;a href=&quot;https://github.com/XifengGuo/CapsNet-Keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras implementation&lt;/a&gt; by Xifeng Guo&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More reading&#xA;- &lt;a href=&quot;https://github.com/loretoparisi/CapsNet&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github repository containing comprehensive list of CapsNet info&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42531" LastActivityDate="2018-03-26T20:13:42.567" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29558" PostTypeId="1" CreationDate="2018-03-26T20:17:37.530" Score="1" ViewCount="61" Body="&lt;p&gt;I have a situation where I need to propose a solution along with stack of technologies that need to be used for the below business case&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Business Case:&#xA;I am receiving a car manufacturing data set from various car manufacturing companies, the data set for each car model is different and have different number of attributes but I do receive them in csv files every 2 minutes. I need to choose a particular storage format so that it is easy to query by each car model and based on the model, i select, i should be able to display the different features of that car model. Is there a recommended architecture for this type of data analysis?&lt;/p&gt;&#xA;" OwnerUserId="47242" LastActivityDate="2018-06-25T10:58:16.257" Title="Architecture Question" Tags="|machine-learning|predictive-modeling|multilabel-classification|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29559" PostTypeId="1" AcceptedAnswerId="30338" CreationDate="2018-03-26T20:29:32.570" Score="0" ViewCount="245" Body="&lt;p&gt;I'm rather new to Orange, so apologies in advance if I'm missing something obvious or am trying to fit a square peg in a round hole. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two files, Summary and Detail, that I'm importing into an Orange workflow. Summary has one row per Contact, and Detail has multiple rows per Contact, each row corresponding to a particular action they've taken. Contacts are identified in both files by Contact ID.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done a bunch of analysis to identify Contacts in Summary that I'm interested in using Orange's machine learning algorithms. I'd like to be able to step through each row in Summary (that is, each Contact) to see what actions in Detail that the Contact took. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought I'd be able to use the Merge Data widget to do this by hooking up Selected Data from Summary as the Data and Data from Detail as the Extra Data using a Find Matching Rows join type. However, since each unique Contact ID appears multiple times in Detail, I'm not able to select it in the Merge Data widget. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know it's nonsensical, but I did try to wire Data from Detail as the Data and Selected Data from Summary as the Extra Data with the only Merge Data option that worked (Append columns from Extra Data). This showed me all the data from Detail, but appended with columns from Summary when the appropriate row was selected in Summary. Based on my understanding of Merge Data, this is predicted but (in my use case) unhelpful behavior. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In database terms, I'm sort of trying to set up an inner join with a one-to-many relationship between Summary and Detail and, by selecting a row in Summary, to be able to see the related rows in Detail.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the bottom line is, is this sort of behavior doable in Orange, or am I trying to paint a house with a chainsaw? Is there another widget that would be more appropriate for the task? (I also tried the Select Rows widget, but couldn't figure out how to populate the filters based on the row that I had selected in Summary.) &lt;/p&gt;&#xA;" OwnerUserId="49491" LastActivityDate="2018-04-15T10:39:22.487" Title="Filtering detail table based on selection in summary table in Orange 3.11" Tags="|orange|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29560" PostTypeId="2" ParentId="26655" CreationDate="2018-03-26T21:18:13.217" Score="1" Body="&lt;p&gt;There are two solutions to this problem, though I'm only going explain the second one because I think is the best option.&lt;/p&gt;&#xA;&lt;h2&gt;First option:&lt;/h2&gt;&#xA;&lt;p&gt;10 Scatterplots of speed of vehicle on the x-axis and predicted speed of vehicle on the y-axis. The only scenario I see this plot been useful is for clustering different types of vehicles with their speeds (if there actually are many vehicle models). Naturally you could also aggregate all data points in a single scatterplot and color-code the data set (ex. dataset1 = blue, dataset 2 = red, etc.)&lt;/p&gt;&#xA;&lt;h2&gt;Second option:&lt;/h2&gt;&#xA;&lt;p&gt;Calculate the difference of vehicle speed and predicted speed for each data point and then bind all the results into a single data table so that each column contains the difference between actual speed and predicted speed for each data set.&#xA;Then plot each column as a bar in a boxplot, here you will be able to describe all differences including signs for outliers and mean. As you might already know boxes represent standard deviation (variance) and the central line represents the mean. You can find more information about boxplots here:&#xA;&lt;a href=&quot;http://%20en.wikipedia.org/wiki/Box_plot&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h3&gt;Sample code in R and ggplot2&lt;/h3&gt;&#xA;&lt;pre&gt;&lt;code&gt;dt1 &amp;lt;- data.frame(data = c(runif(20, min = -1, max = 1),&#xA;                       runif(20, min = -1, max = 1), &#xA;                       runif(20, min = -2, max = 1)), &#xA;              datasetName = c(rep(&amp;quot;dataset1&amp;quot;, 20),&#xA;                              rep(&amp;quot;dataset2&amp;quot;, 20), &#xA;                              rep(&amp;quot;dataset3&amp;quot;, 20)))&#xA;&#xA;&#xA;ggplot(dt1, aes(x = datasetName, y = data)) +&#xA;geom_boxplot(fill = &amp;quot;white&amp;quot;, colour = &amp;quot;#3366FF&amp;quot;) +&#xA;labs(x = &amp;quot;Dataset Name&amp;quot;, y = &amp;quot;Difference Between Speed and Predicted Speed&amp;quot;,  title = &amp;quot;Datasets distribution&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The resulting image should look something like this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6IzKP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6IzKP.png&quot; alt=&quot;Sample Image&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-26T21:18:13.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29561" PostTypeId="1" CreationDate="2018-03-26T23:29:41.190" Score="1" ViewCount="1132" Body="&lt;p&gt;I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type &quot;Animal&quot;, &quot;Building&quot;, &quot;Imagery&quot;, etc., which are not covered in Stanford CoreNLP entity extraction method. Also, I want to detect organization entities which are not included in CoreNLP entity model. I have the data about the organizations with me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do this?&lt;/p&gt;&#xA;" OwnerUserId="46054" LastActivityDate="2020-09-20T19:01:50.203" Title="How to extract entities from text using existing ontologies?" Tags="|nlp|text-mining|named-entity-recognition|stanford-nlp|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29565" PostTypeId="2" ParentId="29561" CreationDate="2018-03-27T06:52:06.710" Score="0" Body="&lt;p&gt;There are multiple ways to extract NER. Primarily, based on the construction of statements NER was extracted with the use of POS  tags. But overtime with the change of how information was being conveyed, there has been a migration from traditional methods to learning methods. Currently, take a look at sequence to sequence tagging for NER detection. If you have the appropriate dataset, you have the capability to extract anything you consider as NER. &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-03-27T06:52:06.710" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29566" PostTypeId="2" ParentId="29558" CreationDate="2018-03-27T06:57:07.307" Score="1" Body="&lt;p&gt;Given that the data model you receive is different, it makes best sense for you to use a NoSql store like mongodb.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the tech stack I would use ( python ) :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;To receive requests with csv file, I would use flask microweb framework.&lt;/li&gt;&#xA;&lt;li&gt;I would run flask with gunicorn and gevent.&lt;/li&gt;&#xA;&lt;li&gt;For every csv file received, I would parse the rows based on varied columns and store them in a mongodb. &lt;/li&gt;&#xA;&lt;li&gt;I would write queries on the collection in mongodb to extract information. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would be more than happy to elaborate more if required . &lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-03-27T06:57:07.307" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29567" PostTypeId="1" CreationDate="2018-03-27T07:00:56.113" Score="1" ViewCount="1091" Body="&lt;p&gt;As the title suggested. By default sns.heatmap will not include categorical features which are not factorized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since factorized categorical features do not behave like a scale (eg. Quality: 0-10), I assume we can't it's not advisable on categorical features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other good practice that we can use to detect strong correlation between the categorical features (among themselves and with the label)?&lt;/p&gt;&#xA;" OwnerUserId="47736" LastActivityDate="2021-10-07T18:30:25.357" Title="Is it advisable or okay to view heatmap of categorical features?" Tags="|correlation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29568" PostTypeId="2" ParentId="29558" CreationDate="2018-03-27T07:42:42.637" Score="1" Body="&lt;p&gt;AWS Cloud Solution Steps- &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dump those files in S3&lt;/li&gt;&#xA;&lt;li&gt;Initiate lambda function (serverless) for pre-processing data according to your need&lt;/li&gt;&#xA;&lt;li&gt;Store this data into Redshift. &#xA;You can create generalized schema to store car details.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="25223" LastActivityDate="2018-03-27T07:42:42.637" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29569" PostTypeId="1" CreationDate="2018-03-27T08:06:21.537" Score="2" ViewCount="439" Body="&lt;p&gt;I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that my distribution is not normal, &#xA;given the qqplot that I have, and the number of observations can I say that my variable is approximately normal? I want to use parametric test but all the test rely on the normality assumption&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Jtk84.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Jtk84.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/lXkBK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/lXkBK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49354" LastActivityDate="2018-03-27T21:27:39.733" Title="can I say that my variable is &quot;approximately&quot; normally distributed?" Tags="|machine-learning|data-mining|distribution|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29570" PostTypeId="1" AcceptedAnswerId="29573" CreationDate="2018-03-27T08:58:50.690" Score="0" ViewCount="1790" Body="&lt;p&gt;I am trying to develop a NN for a very simple classification model with keras/tensorflow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;input: &quot;Do&quot; =&gt; class output: &quot;Dog&quot;&lt;/li&gt;&#xA;&lt;li&gt;input: &quot;Ca&quot; =&gt; class output: &quot;Cat&quot;&lt;/li&gt;&#xA;&lt;li&gt;input: &quot;Mo&quot; =&gt; class output: &quot;Mouse&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I train the model with many &quot;Do&quot;, &quot;Ca&quot;, etc. (as dictionary), indexing the input to categorical arrays (&lt;code&gt;[0,1,0]&lt;/code&gt; for example is &quot;Cat&quot;, &lt;code&gt;[1,0,0]&lt;/code&gt; is &quot;Dog&quot;, &lt;code&gt;[0,0,1]&lt;/code&gt; is &quot;Mouse&quot; in case of three classes).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that for this type of problem every other traditional classification algorithm should be used, and not DL, but I'm doing it with a purpose (I need to merge it with an image classifier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: since the classification is easy (&quot;Do&quot; is always &quot;Dog&quot;, &quot;Ca&quot; is always &quot;Cat&quot;) a Decision tree would have an accuracy of 100% always. Why do Neural Networks need tons of data and epochs to get an acceptable accuracy? With 10 classes, it takes 10 epochs and 8 thousand entries in order to get an accuracy higher than 90%. I'm using sparse categorical crossentropy as loss, and SGD as the optimizer. (2 Dense layer, relu - softmax). Also, I am a bit lost on how to choose the number of neurons, I guess trial and error is the way.&lt;/p&gt;&#xA;" OwnerUserId="49517" LastEditorUserId="24000" LastEditDate="2018-03-27T12:13:25.703" LastActivityDate="2018-03-27T12:13:25.703" Title="Neural network accuracy for simple classification" Tags="|machine-learning|python|neural-network|keras|tensorflow|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29571" PostTypeId="2" ParentId="29569" CreationDate="2018-03-27T09:21:17.730" Score="2" Body="&lt;p&gt;Your data is already binned. You should do a &lt;strong&gt;Chi-Squared test&lt;/strong&gt; to see if the hypothesis that your data is normally distributed is correct. You can refer to &lt;a href=&quot;https://datascience.stackexchange.com/questions/29408/intra-cluster-similarity-metric/29419#29419&quot;&gt;this question&lt;/a&gt; to see how you can go about this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will set your baseline as the Normal distribution with the mean and variance from your dataset. Then you will test the difference in expected value between the Normal distribution and yours.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-03-27T09:21:17.730" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29572" PostTypeId="1" AcceptedAnswerId="29579" CreationDate="2018-03-27T09:27:35.267" Score="7" ViewCount="26311" Body="&lt;p&gt;I started looking for ways to do feature selection in machine learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;By having a quick look at this  &lt;a href=&quot;https://machinelearningmastery.com/an-introduction-to-feature-selection/&quot; rel=&quot;noreferrer&quot;&gt;post &lt;/a&gt;, I made the assumption that feature selection is only manageable for supervised learning problems:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BdkfF.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BdkfF.png&quot; alt=&quot;Image from link&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still, I have to ask: are there methods to do feature selection without having a known variable that will be used for a classification/regression problem? &lt;/p&gt;&#xA;" OwnerUserId="40515" LastEditorUserId="40515" LastEditDate="2019-02-04T17:09:02.027" LastActivityDate="2020-04-02T20:14:14.333" Title="is it possible to do feature selection for unsupervised machine learning problems?" Tags="|feature-selection|unsupervised-learning|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29573" PostTypeId="2" ParentId="29570" CreationDate="2018-03-27T09:39:41.730" Score="2" Body="&lt;p&gt;I will add my 2 cents at the end of this answer. However, this is how it can be done using a neural network. Firstly, yes, you should expect to need more data to train even a simple neural network because their are more parameters that need tuning. Think of them like little faucets that you need to tune in order to get the right output volume based on an input. If you have millions of these faucets you an imagine that this is an arduous process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need some of the following imports&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import print_function&#xA;import keras&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten&#xA;from keras.layers import Conv2D, MaxPooling2D&#xA;from keras.callbacks import ModelCheckpoint&#xA;from keras.models import model_from_json&#xA;from keras import backend as K&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, in your case you know what your data should be so you can simulate it. I do this as follows and make a training and testing set. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;n = 10&#xA;m = 2&#xA;&#xA;x_train = np.zeros((n, m))&#xA;y_train = np.zeros((n,))&#xA;for i in range(n):&#xA;    label = np.random.randint(0,m)&#xA;    y_train[i] = label&#xA;    x_train[i, label] = 1&#xA;&#xA;x_test = np.zeros((n//3, m))&#xA;y_test = np.zeros((n//3,))&#xA;for i in range(n//3):&#xA;    label = np.random.randint(0,m)&#xA;    y_test[i] = label&#xA;    x_test[i, label] = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we will have a training set which contains $n$ instances and a testing set with a third as many. $m$ is the number of possible inputs. For cat vs. dog this would be $m=2$. You will be using your more general case where $m=10$. Each entry in the matrix $x$ has the vector with one-hot encoded vector where the index in accordance with the label is 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to reshape the data for it to fit with the Keras structure.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# The known number of output classes.&#xA;num_classes = m&#xA;&#xA;# Channels go last for TensorFlow backend&#xA;x_train_reshaped = x_train.reshape(x_train.shape[0], m,)&#xA;x_test_reshaped = x_test.reshape(x_test.shape[0], m,)&#xA;input_shape = (m,)&#xA;&#xA;# Convert class vectors to binary class matrices. This uses 1 hot encoding.&#xA;y_train_binary = keras.utils.to_categorical(y_train, num_classes)&#xA;y_test_binary = keras.utils.to_categorical(y_test, num_classes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We then build our model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(64, activation='relu', input_shape = input_shape))&#xA;model.add(Dense(num_classes, activation='softmax'))&#xA;&#xA;model.compile(loss=keras.losses.categorical_crossentropy,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We then train our model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs = 4&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;history = model.fit(x_train_reshaped, y_train_binary,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test_reshaped, y_test_binary))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Epoch 4/4 &lt;br/&gt;10/10 [==============================] - 0s 251us/step -&#xA;  loss: 0.6247 - acc: 0.6000 - val_loss: 0.5311 - val_acc: 1.0000&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Voila, now you have perfect classification with this network. You can play around with the model and see a summary of the model using.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.summary()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;For $m = 10$&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Due to the higher complexity that this set has you will need to increase the number of instances in your training set. I will also 2 layers to our model and make them less wide. Furthermore, we will add more epochs so we will train longer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;n = 1000&#xA;m = 10&#xA;&#xA;x_train = np.zeros((n, m))&#xA;y_train = np.zeros((n,))&#xA;for i in range(n):&#xA;    label = np.random.randint(0,m)&#xA;    y_train[i] = label&#xA;    x_train[i, label] = 1&#xA;&#xA;x_test = np.zeros((n//3, m))&#xA;y_test = np.zeros((n//3,))&#xA;for i in range(n//3):&#xA;    label = np.random.randint(0,m)&#xA;    y_test[i] = label&#xA;    x_test[i, label] = 1&#xA;&#xA;# The known number of output classes.&#xA;num_classes = m&#xA;&#xA;# Channels go last for TensorFlow backend&#xA;x_train_reshaped = x_train.reshape(x_train.shape[0], m,)&#xA;x_test_reshaped = x_test.reshape(x_test.shape[0], m,)&#xA;&#xA;input_shape = (m,)&#xA;&#xA;# Convert class vectors to binary class matrices. This uses 1 hot encoding.&#xA;y_train_binary = keras.utils.to_categorical(y_train, num_classes)&#xA;y_test_binary = keras.utils.to_categorical(y_test, num_classes)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(32, activation='relu', input_shape = input_shape))&#xA;model.add(Dense(32, activation='relu'))&#xA;model.add(Dense(num_classes, activation='softmax'))&#xA;&#xA;model.compile(loss=keras.losses.categorical_crossentropy,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&#xA;epochs = 10&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;history = model.fit(x_train_reshaped, y_train_binary,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test_reshaped, y_test_binary))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Epoch 10/10 &lt;br/&gt;1000/1000 [==============================] - 0s 49us/step&#xA;  - loss: 1.5977 - acc: 1.0000 - val_loss: 1.5235 - val_acc: 1.0000&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;My suggestions&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I would not use a NN for such a case. Most of the frameworks allow you to add information throughout your model. Such that if you have images you can run a CNN over them, then when you are ready to convert your layers to a densely connected layer you can in additional information, such as vectorized text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can thus use a random forests approach or something even simpler to get your 100% classification even faster. Then you can feed the output of this model to your deep learning framework which has already &quot;extracted the features&quot; from the images and concatenate these additional features to that tensor. Then you will pass this larger tensor through the subsequent Dense layers to get your final output.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-03-27T09:56:03.743" LastActivityDate="2018-03-27T09:56:03.743" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29574" PostTypeId="1" AcceptedAnswerId="29580" CreationDate="2018-03-27T09:40:36.900" Score="0" ViewCount="121" Body="&lt;p&gt;I've this Python code that allows me to collect data from Todoist API. Now I want to store this data into a Dictionary in order to make some data processing and some advanced analytics. Using this code how can I store this values into the structure ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pytodoist import todoist&#xA;user = todoist.login('####','&quot;&quot;&quot;&quot;')&#xA;projects = user.get_projects()&#xA;for project in projects:&#xA;    print(project.name)&#xA;&#xA;tasks = user.get_tasks()&#xA;for task in tasks:&#xA;    print(&quot;Project: &quot;, task.project.name, &quot;, Task: &quot;, task.content, &quot;, Date: &quot;, task.date_string, &quot;, Priority: &quot;, task.priority, &quot;, Added in: &quot;, task.date_added)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My key_pais: The strings that I've in For loop &quot;Project&quot;, &quot;Task&quot;, etc.&#xA;Values: the values from taks&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="24790" LastEditorUserId="24790" LastEditDate="2018-03-27T11:02:05.047" LastActivityDate="2018-03-27T16:16:43.253" Title="Python - Todoist API - Store data into Dictionary" Tags="|python|dataset|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29576" PostTypeId="2" ParentId="29570" CreationDate="2018-03-27T09:44:28.257" Score="0" Body="&lt;p&gt;For every problem there is proportional solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that you want to build your classifier inside an image recognition algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may need a NN for the image (lots of inputs + lots of possibilities) but not for the whole system. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is usually done for this kind of problematic is to build a &quot;pipe&quot;. A sequence of ML algorithms taking as input the output of the previous one:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First block is a text zone detection within the image &lt;/li&gt;&#xA;&lt;li&gt;Second block read the zones into text&lt;/li&gt;&#xA;&lt;li&gt;Third block do the actual classification.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This method also make the debugging process easier, as you can evaluate your prediction rate at every step making easy to know which part can be optimized/fixed if a problem is encountered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="45762" LastActivityDate="2018-03-27T09:44:28.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29577" PostTypeId="2" ParentId="29572" CreationDate="2018-03-27T09:49:32.717" Score="11" Body="&lt;p&gt;Take a look at these links-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/108743/methods-in-r-or-python-to-perform-feature-selection-in-unsupervised-learning&quot;&gt;https://stats.stackexchange.com/questions/108743/methods-in-r-or-python-to-perform-feature-selection-in-unsupervised-learning&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.jmlr.org/papers/volume5/dy04a/dy04a.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.jmlr.org/papers/volume5/dy04a/dy04a.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/post/What_is_the_best_unsupervised_method_for_feature_subset_selection&quot; rel=&quot;noreferrer&quot;&gt;https://www.researchgate.net/post/What_is_the_best_unsupervised_method_for_feature_subset_selection&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-03-27T09:49:32.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29579" PostTypeId="2" ParentId="29572" CreationDate="2018-03-27T10:22:42.343" Score="4" Body="&lt;p&gt;Feature Selection is a technique which is used when we you know the target variable(Supervised Learning)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we talk with respect to Unsupervised Learning, there is no exact technique which could do that. But there is something which can help us in those lines i.e., Dimensionality Reduction, this technique is used to reduce the number of features and give us the features which explains the most about the dataset. The features would be derived from the existing features and might or might not be the same features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different techniques which are available for doing so:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;PCA&lt;/li&gt;&#xA;&lt;li&gt;Linear discriminant analysis&lt;/li&gt;&#xA;&lt;li&gt;Non-negative Matrix Factorization&lt;/li&gt;&#xA;&lt;li&gt;Generalized discriminant analysis and many more.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The outcome of Feature Selection would be the same features which explain the most with respect to the target variable but the outcome of the Dimensionality Reduction might or might not be the same features as these are derived from the given input. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-03-27T10:22:42.343" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29580" PostTypeId="2" ParentId="29574" CreationDate="2018-03-27T10:51:59.207" Score="0" Body="&lt;p&gt;At first remember that a dictionary is just bunches of keys and value pairs...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if the key is $a$ and the value is $10$ , you can access the value by doing &lt;code&gt;d[a]&lt;/code&gt; (d is the dictionary)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beaware that there is no order to storing the values, you can use OrderedDict for that..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also you can have two values for the same key in a simple dictionary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's the complete &lt;a href=&quot;https://stackoverflow.com/q/27102573/6524169&quot;&gt;answer&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-27T16:16:43.253" LastActivityDate="2018-03-27T16:16:43.253" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29581" PostTypeId="1" AcceptedAnswerId="37231" CreationDate="2018-03-27T10:54:41.630" Score="5" ViewCount="1107" Body="&lt;p&gt;Tensorflow has built-in implementations for both, the &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss&quot; rel=&quot;noreferrer&quot;&gt;Connectionist Temporal Classification (CTC) loss &lt;/a&gt; and a special &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/sequence_loss&quot; rel=&quot;noreferrer&quot;&gt;seq2seq loss (weighted cross-entropy)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since CTC loss is also intended to deal with seq2seq mappings, I wonder about how the two loss functions compare. I went through an &lt;a href=&quot;https://distill.pub/2017/ctc/&quot; rel=&quot;noreferrer&quot;&gt;excellent explanation&lt;/a&gt; of CTC loss, finding out that the target sequence is restricted not to be longer than the input sequence whereas this restriction does not exist for seq2seq loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some questions I have (ranked by relevance):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Are bidirectional LSTMs suited for seq2seq loss? I have only found b-LSTM implementations with CTC or even simpler loss functions. There is a &lt;a href=&quot;https://github.com/tensorflow/tensorflow/issues/3585&quot; rel=&quot;noreferrer&quot;&gt;post from 2016&lt;/a&gt; saying that this was not possible at that time. Any updates?&lt;/li&gt;&#xA;&lt;li&gt;Are there conceptual differences apart from the many-to-one restriction in CTC loss that seq2seq does not have? E.g. does the seq2seq loss make a similar conditional independence assumption like the CTC loss?&lt;/li&gt;&#xA;&lt;li&gt;Do the loss functions have preferences regarding the optimizer (RMSProp, ADAM, Momentum etc.)&lt;/li&gt;&#xA;&lt;li&gt;Do the loss functions work better for particular sets of tasks?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Theoretical arguments and practical experiences are both appreciated.&lt;/p&gt;&#xA;" OwnerUserId="43061" LastActivityDate="2018-08-21T11:23:48.977" Title="Theoretical and practical comparison of CTC and seq2seq loss in Tensorflow" Tags="|machine-learning|neural-network|deep-learning|nlp|tensorflow|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29583" PostTypeId="1" CreationDate="2018-03-27T11:06:45.513" Score="1" ViewCount="30" Body="&lt;p&gt;How do Dynamic Memory Network, for example from the paper &lt;a href=&quot;https://arxiv.org/pdf/1506.07285.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ask Me Anything:&#xA;Dynamic Memory Networks for Natural Language Processing&lt;/a&gt;, scale to large inputs? The paper states the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In these cases, the input may be a sentence, a long story, a&#xA;  movie review, a news article, or several Wikipedia articles.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Did anyone actually implement this network architecture to read an arbitrarily large corpus of text (say 250 Wikipedia articles)? The only example I could find online use rather small inputs (a few lines of text).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the bottleneck of using this type of network with large inputs? One solution I see is to encode paragraphs instead of sentences.&lt;/p&gt;&#xA;" OwnerUserId="33312" LastActivityDate="2018-03-27T11:06:45.513" Title="How do Dynamic Memory Network scale to large inputs?" Tags="|neural-network|deep-learning|nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29584" PostTypeId="2" ParentId="9269" CreationDate="2018-03-27T11:21:32.357" Score="1" Body="&lt;p&gt;I think your post is missing a clear question. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll answer your concerns anyway:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For a implicit feedback problem like the one described RMSE, indeed, doesn't make much sense. First of all it is not clear what the difference between the real value (number of interactions, e.g. watched movie 4 times) and the estimation would mean. Second, in recommender systems you usuallly want to give the user 10-20 relevant recommendations and you don't care for the rest. Hence, it makes sense to optimize your algorithm to get these top recommendations right while disregarding your performance further down in the ranking. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;See my first answer. Also, the authors of your cited paper argue for a rank-based metric.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44089" LastActivityDate="2018-03-27T11:21:32.357" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29585" PostTypeId="1" AcceptedAnswerId="29588" CreationDate="2018-03-27T12:46:00.947" Score="2" ViewCount="171" Body="&lt;p&gt;So I came across a &lt;a href=&quot;http://iai.asm.org/content/79/1/23.full.pdf+html&quot; rel=&quot;nofollow noreferrer&quot;&gt;bioinformatics paper&lt;/a&gt;, where I found a line which says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One potential problem with using a training set with&#xA;  equal numbers of positive and negative examples in cross-validation&#xA;  is that it can artificially inflate performance estimates&#xA;  because the number of false-positive classifications is&#xA;  proportional to the number of examples classified. So applying&#xA;  these methods to all proteins in an organism may&#xA;  result in a large number of false-positive identifications.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I am unable to understand how classification of balanced dataset is a problem. Can someone please explain this to me?&lt;/p&gt;&#xA;" OwnerUserId="8013" LastEditorUserId="28175" LastEditDate="2018-03-27T13:24:02.333" LastActivityDate="2018-03-27T13:24:02.333" Title="Does classification of a balanced data-set lead to any problem?" Tags="|machine-learning|deep-learning|class-imbalance|distribution|bayes-error|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29586" PostTypeId="1" CreationDate="2018-03-27T12:47:48.753" Score="1" ViewCount="896" Body="&lt;p&gt;Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-customers.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the mailings, we select the top 100,000 scoring (highest predicted probabilities) records. We are expecting a low response rate but we want to make sure we send mail pieces to all of the highest predicted prospects.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We bin the model scores into deciles and only select from decile 1 (highest scoring) which we'll say is &gt;= 0.10 predicted probability. The model has a 95% recall and 5% precision at this threshold.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense in this scenario to get the highest recall possible? Or would precision be important here and I should optimize on that or just optimize them both together.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The goal is to get the most customer conversions within that 100k and we already know there will be a good amount of false positives but we want as many true positives as possible.&lt;/p&gt;&#xA;" OwnerUserId="45182" LastActivityDate="2018-05-27T05:40:41.970" Title="Is recall more important than precision for mass mailings?" Tags="|machine-learning|predictive-modeling|probability|model-evaluations|information-retrieval|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29587" PostTypeId="1" AcceptedAnswerId="29592" CreationDate="2018-03-27T13:16:26.920" Score="5" ViewCount="9230" Body="&lt;p&gt;i'm currently experimenting with scikit and the DBSCAN algorithm. And i'm wondering how to combine the data with the labels to write them into a new file. I'd also like to understand how the labels array is used to filter the examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me anytime i say something wrong because i'd like to understand the whole process better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example my data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;city    x    y&#xA;A       1    1&#xA;B       1    1&#xA;C       5    5&#xA;D       8    8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So if i understand it correctly i first need to split the data i'd like to cluster. If i use the data like above it will consider the &lt;em&gt;city&lt;/em&gt; column aswell (or even fail).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my next array would look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x    y&#xA;1    1&#xA;1    1&#xA;5    5&#xA;8    8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now i'll use the DBSCAN on my Array and it will create a cluster model. The clusterlabels are now stored in the array &lt;em&gt;foo.labels_&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as i know i can filter the data with those labels to get the items within the clusters. Let's assume my data is in the dataframe &lt;em&gt;cities&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cluster0 = cities[foo.labels_ == 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What i don't understand is how this works. I somehow don't get what exactly happens here. I know that the labels are in an array where the column is a number and the value of the column is the cluster. So how do i get the correct index of my cities?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after the clustering i'd like to export my data back into a CSV file with the following format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;city    x     y     cluster&#xA;A       1     1     0&#xA;B       1     1     0&#xA;C       5     5     1&#xA;D       8     8     2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My guess is to use to original dataframe and add another column like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cities = cities.assign(cluster=p.Series(labels_))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But i'm absolutely unsure if that's the correct way to achieve what i want to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd really appreciate some opinions and explanations.&lt;/p&gt;&#xA;" OwnerUserId="41656" LastActivityDate="2018-03-27T14:30:11.830" Title="Python clustering and labels" Tags="|python|scikit-learn|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29588" PostTypeId="2" ParentId="29585" CreationDate="2018-03-27T13:22:59.817" Score="2" Body="&lt;p&gt;Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't work well because of changing the distribution of each class. In cases like a disease prediction where the number of each class varies for different places, a model that is trained in the U.S won't work in African countries at all. The reason is that the distribution of classes has been changed. So in such cases that usually the negative and positive classes are not balanced in practice, balancing them will cause the problem of distribution changes. On these occasions, people usually use the real data-set which is not balanced and use &lt;code&gt;F1&lt;/code&gt; score for evaluation. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-27T13:22:59.817" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29589" PostTypeId="1" AcceptedAnswerId="29591" CreationDate="2018-03-27T13:31:41.600" Score="1" ViewCount="142" Body="&lt;p&gt;I want to know what is Big Data? Can I have practical example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How big data can be? I need numbers where big data term is applicable. If you can provide link for case study with actual numbers, with reference to V's of Big Data.&lt;/p&gt;&#xA;" OwnerUserId="47795" LastActivityDate="2018-03-27T14:01:19.220" Title="What is Big Data?" Tags="|data-mining|dataset|bigdata|visualization|" AnswerCount="1" CommentCount="0" ClosedDate="2018-03-29T07:05:59.743" ContentLicense="CC BY-SA 3.0" />
  <row Id="29590" PostTypeId="1" CreationDate="2018-03-27T13:40:34.470" Score="0" ViewCount="45" Body="&lt;p&gt;I would like to know if it's possible to detect outliers in a time-serie with an outlier score computed given the whole dataset and not given windows &lt;/p&gt;&#xA;" OwnerUserId="49354" LastActivityDate="2018-03-27T20:09:23.090" Title="outlier detection in time serie without using windows" Tags="|time-series|outlier|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ClosedDate="2018-03-28T03:41:49.963" ContentLicense="CC BY-SA 3.0" />
  <row Id="29591" PostTypeId="2" ParentId="29589" CreationDate="2018-03-27T14:01:19.220" Score="3" Body="&lt;p&gt;If I want to quote from &lt;a href=&quot;https://en.wikipedia.org/wiki/Big_data&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;, &lt;em&gt;Big data is data sets that are so voluminous and complex that traditional data-processing application software are inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source. There are five concepts associated with big data: volume, variety, velocity and, the recently added, veracity and value.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Big data can be described by the following characteristics:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Volume&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Variety&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The type and nature of the data. This helps people who analyze it to effectively use the resulting insight. Big data draws from text, images, audio, video; plus it completes missing pieces through data fusion.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Velocity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;In this context, the speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Variability&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Inconsistency of the data set can hamper processes to handle and manage it.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Veracity&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;The data quality of captured data can vary greatly, affecting the accurate analysis.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr /&gt;&#xA;&#xA;&lt;p&gt;To me, big data is highly connected to the deep-learning era. The reason is that during past decades, people could make good descriptions and models of data using machine-learning and data-mining but because everyday new data is coming out, social networks increase rapidly and digital gadgets' popularity is increasing among different nations, the demand for processing data and converting them to information and knowledge is increasing. If we want to use previous techniques to gather information from raw data, it will take too much time, if possible, to reach to appropriate results. In big data and deep-learning era, we need more complicated algorithms and more powerful hardware to deal with difficulties. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also take a look at &lt;a href=&quot;https://www.sas.com/en_us/insights/big-data/what-is-big-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;http://searchcloudcomputing.techtarget.com/definition/big-data-Big-Data&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; which have relatively different perspective. &lt;em&gt;Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-03-27T14:01:19.220" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29592" PostTypeId="2" ParentId="29587" CreationDate="2018-03-27T14:30:11.830" Score="4" Body="&lt;p&gt;As the algorithm should not change the order of the lists you could just add the clusters list&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; cities[&quot;cluster&quot;] = cluster &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are really paranoid you can add your input parameters a second time to the dataframe in the same way and compare the diff in values (should be 0). &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2018-03-27T14:30:11.830" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29595" PostTypeId="1" CreationDate="2018-03-27T15:55:42.840" Score="0" ViewCount="361" Body="&lt;p&gt;I am trying to train a &lt;strong&gt;VGG-19&lt;/strong&gt; neural network on &lt;strong&gt;STL-10&lt;/strong&gt; dataset containing &lt;strong&gt;5000&lt;/strong&gt; images(&lt;strong&gt;500&lt;/strong&gt; for each class). And the number of output classes is &lt;strong&gt;10&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have made no changes to the architecture except that I reduced the size of fully-connected layer from &lt;strong&gt;4096&lt;/strong&gt; to &lt;strong&gt;2048&lt;/strong&gt;, keeping the &lt;strong&gt;dropout&lt;/strong&gt;(0.5) same &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[The reason behind doing this is that, since the number of training images is less, so to avoid over-fitting, I decreased the size of fully-connected layer, but I don't know if this is the right thing to do].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also used &lt;strong&gt;Adam optimiser&lt;/strong&gt;(learning rate = 0.001) instead of &lt;strong&gt;SGD&lt;/strong&gt; as mentioned in the paper.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have only run the code for only 4 epochs. And I have observed that although the cost is decreasing by very small amount, but the training accuracy is decreasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the &lt;em&gt;1st&lt;/em&gt; epoch, &lt;em&gt;cost&lt;/em&gt;:: &lt;strong&gt;2.304091&lt;/strong&gt; and &lt;em&gt;training accuracy&lt;/em&gt;:: &lt;strong&gt;11.99%&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the &lt;em&gt;2nd&lt;/em&gt; epoch, &lt;em&gt;cost&lt;/em&gt;:: &lt;strong&gt;2.303365&lt;/strong&gt; and &lt;em&gt;training accuracy&lt;/em&gt;:: &lt;strong&gt;11.249%&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the &lt;em&gt;3rd&lt;/em&gt; epoch, &lt;em&gt;cost&lt;/em&gt;:: &lt;strong&gt;2.301936&lt;/strong&gt; and &lt;em&gt;training accuracy&lt;/em&gt;:: &lt;strong&gt;10.5625%&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After the &lt;em&gt;4th&lt;/em&gt; epoch, &lt;em&gt;cost&lt;/em&gt;:: &lt;strong&gt;2.30415&lt;/strong&gt; and &lt;em&gt;training accuracy&lt;/em&gt;:: &lt;strong&gt;8.1249%&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to know, that is this behavior natural or is it due to some fault in the changes I made in the architecture?&#xA;Or using less number of layers would have been better? (For example VGG-16)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is my first hands-on neural network experience)&lt;/p&gt;&#xA;" OwnerUserId="49454" LastEditorUserId="29169" LastEditDate="2021-02-13T16:35:28.077" LastActivityDate="2021-02-13T16:35:28.077" Title="Training Error decreasing with each epoch" Tags="|neural-network|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29596" PostTypeId="1" CreationDate="2018-03-27T15:56:17.173" Score="1" ViewCount="6403" Body="&lt;p&gt;I have a list of events topic retrieved from a tweets collection.&#xA;A set of features have been extracted and their values normalized between 0 and 1. An example of an event:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;paris_attack_news-20150107_100842-20150107_112852&quot;: {&#xA;    &quot;ages&quot;: 0.5557594006583049,&#xA;    &quot;density&quot;: 0.0012022814250710345,&#xA;    &quot;followers&quot;: 0.1144661871115895,&#xA;    &quot;friends&quot;: 0.13507755010659472,&#xA;    &quot;hashtagCount&quot;: 0.033270950301517985,&#xA;    &quot;lifespan&quot;: 0.29613227044582224,&#xA;    &quot;mediaCount&quot;: 0.1095890410958904,&#xA;    &quot;mentionCount&quot;: 0.020275919732441472,&#xA;    &quot;objectivity&quot;: 0.2850584551023736,&#xA;    &quot;polarity&quot;: 0.2963684492294102,&#xA;    &quot;retweetCount&quot;: 0.21431767337807606,&#xA;    &quot;status_count&quot;: 0.09222093073720204,&#xA;    &quot;truth&quot;: 1.0,&#xA;    &quot;tweetCount&quot;: 0.01300578034682081,&#xA;    &quot;urlCount&quot;: 0.29494007989347537,&#xA;    &quot;verified&quot;: 0.3392857142857143&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I need to represent each event as an array of its features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;paris_attack_news-20150107_100842-20150107_112852 = [0.5557594006583049, 0.5557594006583049, 0.1144661871115895, 0.13507755010659472, ...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After that, I need to manipulate/aggregate the array values in some ways to get a specific events sorting based on the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data are already in a Python Pandas DataFrame (event name as index, features as columns). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the best way (data structure for further storing or libraries such as NumPy, sklearn or similar) to build the arrays starting from this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: Then I'll need to apply some machine learning alghoritms to detect if an event is TRUE or FALSE, using the feature named &quot;truth&quot;: 1 or 0 as label classification. &lt;/p&gt;&#xA;" OwnerUserId="46313" LastEditorUserId="46313" LastEditDate="2018-03-27T19:01:37.963" LastActivityDate="2018-03-29T02:35:24.203" Title="Best way to represent data as features vectors in Python" Tags="|machine-learning|python|classification|pandas|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29597" PostTypeId="2" ParentId="14999" CreationDate="2018-03-27T16:01:59.357" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://www.philippe-fournier-viger.com/spmf/&quot; rel=&quot;nofollow noreferrer&quot;&gt;SPMF&lt;/a&gt; sounds like a useful library for pattern mining.&lt;/p&gt;&#xA;" OwnerUserId="49535" LastEditorUserId="98307" LastEditDate="2020-08-04T16:59:59.177" LastActivityDate="2020-08-04T16:59:59.177" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29598" PostTypeId="2" ParentId="12851" CreationDate="2018-03-27T16:04:47.000" Score="21" Body="&lt;p&gt;I would add ASCII visualizations using &lt;a href=&quot;https://github.com/stared/keras-sequential-ascii&quot; rel=&quot;noreferrer&quot;&gt;keras-sequential-ascii&lt;/a&gt; (disclaimer: I am the author).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A small network for CIFAR-10 (from &lt;a href=&quot;https://blog.deepsense.ai/deep-learning-hands-on-image-classification/&quot; rel=&quot;noreferrer&quot;&gt;this tutorial&lt;/a&gt;) would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       OPERATION           DATA DIMENSIONS   WEIGHTS(N)   WEIGHTS(%)&#xA;&#xA;           Input   #####     32   32    3&#xA;          Conv2D    \|/  -------------------       896     2.1%&#xA;            relu   #####     30   30   32&#xA;    MaxPooling2D   Y max -------------------         0     0.0%&#xA;                   #####     15   15   32&#xA;          Conv2D    \|/  -------------------     18496    43.6%&#xA;            relu   #####     13   13   64&#xA;    MaxPooling2D   Y max -------------------         0     0.0%&#xA;                   #####      6    6   64&#xA;         Flatten   ||||| -------------------         0     0.0%&#xA;                   #####        2304&#xA;           Dense   XXXXX -------------------     23050    54.3%&#xA;         softmax   #####          10&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For VGG16 it would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       OPERATION           DATA DIMENSIONS   WEIGHTS(N)   WEIGHTS(%)&#xA;&#xA;          Input   #####      3  224  224&#xA;     InputLayer     |   -------------------         0     0.0%&#xA;                  #####      3  224  224&#xA;  Convolution2D    \|/  -------------------      1792     0.0%&#xA;           relu   #####     64  224  224&#xA;  Convolution2D    \|/  -------------------     36928     0.0%&#xA;           relu   #####     64  224  224&#xA;   MaxPooling2D   Y max -------------------         0     0.0%&#xA;                  #####     64  112  112&#xA;  Convolution2D    \|/  -------------------     73856     0.1%&#xA;           relu   #####    128  112  112&#xA;  Convolution2D    \|/  -------------------    147584     0.1%&#xA;           relu   #####    128  112  112&#xA;   MaxPooling2D   Y max -------------------         0     0.0%&#xA;                  #####    128   56   56&#xA;  Convolution2D    \|/  -------------------    295168     0.2%&#xA;           relu   #####    256   56   56&#xA;  Convolution2D    \|/  -------------------    590080     0.4%&#xA;           relu   #####    256   56   56&#xA;  Convolution2D    \|/  -------------------    590080     0.4%&#xA;           relu   #####    256   56   56&#xA;   MaxPooling2D   Y max -------------------         0     0.0%&#xA;                  #####    256   28   28&#xA;  Convolution2D    \|/  -------------------   1180160     0.9%&#xA;           relu   #####    512   28   28&#xA;  Convolution2D    \|/  -------------------   2359808     1.7%&#xA;           relu   #####    512   28   28&#xA;  Convolution2D    \|/  -------------------   2359808     1.7%&#xA;           relu   #####    512   28   28&#xA;   MaxPooling2D   Y max -------------------         0     0.0%&#xA;                  #####    512   14   14&#xA;  Convolution2D    \|/  -------------------   2359808     1.7%&#xA;           relu   #####    512   14   14&#xA;  Convolution2D    \|/  -------------------   2359808     1.7%&#xA;           relu   #####    512   14   14&#xA;  Convolution2D    \|/  -------------------   2359808     1.7%&#xA;           relu   #####    512   14   14&#xA;   MaxPooling2D   Y max -------------------         0     0.0%&#xA;                  #####    512    7    7&#xA;        Flatten   ||||| -------------------         0     0.0%&#xA;                  #####       25088&#xA;          Dense   XXXXX ------------------- 102764544    74.3%&#xA;           relu   #####        4096&#xA;          Dense   XXXXX -------------------  16781312    12.1%&#xA;           relu   #####        4096&#xA;          Dense   XXXXX -------------------   4097000     3.0%&#xA;        softmax   #####        1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="289" LastActivityDate="2018-03-27T16:04:47.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29599" PostTypeId="1" CreationDate="2018-03-27T16:30:20.513" Score="1" ViewCount="709" Body="&lt;p&gt;I'm looking to find a way where I can extract rules from my decision tree. I have 8 predictors and they are all categorical variables and the response variable has three outputs A B and C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've developed the model and can also create decision tree but I'm having trouble in understanding the tree and be able to extract rules from my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 8 predictors I have can have values a, b, c, d, e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an example, I want to extract rules from my decision tree like this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Rule 1:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Predictor 1 - a  &#xA;Predictor 2 - b  &#xA;Predictor 3 - c  &#xA;Predictor 4 - b  &#xA;Predictor 5 - a  &#xA;Predictor 6 - d  &#xA;Predictor 7 - e  &#xA;Predictor 8 - a  &#xA;&#xA;Result: A&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Rule 2: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Predictor 1 - a  &#xA;Predictor 2 - b  &#xA;Predictor 3 - d  &#xA;Predictor 4 - b  &#xA;Predictor 5 - a  &#xA;Predictor 6 - d  &#xA;Predictor 7 - b  &#xA;Predictor 8 - a  &#xA;&#xA;Result: A&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Rule 3: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Predictor 1 - a  &#xA;Predictor 2 - b  &#xA;Predictor 3 - d  &#xA;Predictor 4 - a  &#xA;Predictor 5 - a  &#xA;Predictor 6 - a  &#xA;Predictor 7 - b  &#xA;Predictor 8 - a  &#xA;&#xA;Result: B  &#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and so on and so forth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way I have rules from my decision tree like I explained above?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the result of my model:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;124 samples&lt;/li&gt;&#xA;&lt;li&gt;8 predictors&lt;/li&gt;&#xA;&lt;li&gt;3 classes: 'A', 'B', 'C' &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;No pre-processing.&lt;br&gt;&#xA;Resampling: Cross-Validated (10 fold, repeated 10 times).&lt;br&gt;&#xA;Summary of sample sizes: 112, 111, 111, 112, 110, 112, ... &lt;br&gt;&#xA;Resampling results across tuning parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  cp           Accuracy   Kappa  &#xA;  0.00000000  1.0000000  1.0000000  &#xA;  0.06329114  1.0000000  1.0000000  &#xA;  0.12658228  1.0000000  1.0000000  &#xA;  0.18987342  1.0000000  1.0000000  &#xA;  0.25316456  1.0000000  1.0000000  &#xA;  0.31645570  1.0000000  1.0000000  &#xA;  0.37974684  1.0000000  1.0000000  &#xA;  0.44303797  0.7261846  0.5696935  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Accuracy was used to select the optimal model using the largest value.&#xA;The final value used for the model was cp = 0.3797468.&#xA;n= 124 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;node), split, n, loss, yval, (yprob)&#xA;      * denotes terminal node&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1) root 124 79 A (0.3629032 0.2741935 0.3629032)    &#xA;  2) Activity4e&amp;lt; 0.5 45  0 A (1.0000000 0.0000000 0.0000000) *  &#xA;    3) Activity4e&amp;gt;=0.5 79 34 C (0.0000000 0.4303797 0.5696203)    &#xA;      6) Activity7c&amp;gt;=0.5 34  0 B (0.0000000 1.0000000 0.0000000) *  &#xA;      7) Activity7c&amp;lt; 0.5 45  0 C (0.0000000 0.0000000 1.0000000) * &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VHKKe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VHKKe.png&quot; alt=&quot;visualising my decision tree&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49534" LastEditorUserId="24000" LastEditDate="2018-03-28T05:36:02.540" LastActivityDate="2019-04-28T15:03:00.767" Title="Fetching rules from rpart using caret package" Tags="|classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29600" PostTypeId="1" AcceptedAnswerId="29995" CreationDate="2018-03-27T16:35:06.533" Score="2" ViewCount="341" Body="&lt;p&gt;We have data of several news sites, having quite literally millions of entries. As each news site publishes their own version of the news (also each news site may publish several different version of the same news), we have several entries that are variants of a single news. I am currently working on separating out &quot;Unique&quot; news from our repository. That means if a single news has several variants, only a single variant (most likely the one reported earliest) will be considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe, clustering of the news articles can be used to group together similar news. I am currently exploring DBSCAN, and Hierarchical clustering (Ward's Method). I am wondering whether am I moving in the right direction, is clustering the best solution for solving our problem? If yes, which other algorithms and techniques should I explore?&lt;/p&gt;&#xA;" OwnerUserId="49537" LastEditorUserId="924" LastEditDate="2018-04-06T19:16:31.647" LastActivityDate="2018-04-06T19:16:31.647" Title="Clustering of variants of similar news articles" Tags="|machine-learning|clustering|dbscan|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29602" PostTypeId="1" AcceptedAnswerId="29606" CreationDate="2018-03-27T17:04:01.827" Score="3" ViewCount="3906" Body="&lt;p&gt;I am a student getting started with Tableau for the first time. My proficiency with ggplot2 is intermediate. I can create custom versions of the most popular kinds of charts in ggplot2 but nothing too fancy (and not very time-efficiently). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in creating more polished visuals of my data in Tableau. My ggplot2 ones aren't bad, but they are not what I would call aesthetically pleasing either. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I would like to know is under what situations I should stick with ggplot2. The only advantages that come to mind with my current (albeit limited) understanding of Tableau is to keep my work reproducible and open source. ggplot2 is also probably more customizable, but I am continually impressed by Tableau's offerings as I learn to use it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is not intended to be a flippant jab at ggplot2. As I did not hear any talk about Tableau in the academic circles I was in, I am wondering what obvious drawbacks to Tableau may exist that would explain its unpopularity there. &lt;/p&gt;&#xA;" OwnerUserId="45701" LastActivityDate="2018-03-27T18:17:44.073" Title="When Should I Use ggplot2 Instead of Tableau?" Tags="|visualization|tableau|ggplot2|" AnswerCount="2" CommentCount="0" ClosedDate="2018-03-28T03:41:36.407" ContentLicense="CC BY-SA 3.0" />
  <row Id="29604" PostTypeId="2" ParentId="29541" CreationDate="2018-03-27T17:48:42.750" Score="-1" Body="&lt;p&gt;When you say &quot;However, it is not a real distance, but a distance-like quantity,&quot; you really mean, it is a measure, not a metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why do you think you need a metric? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the following common American girl names:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[Lisabeth, Beth, Lisa, Maryanne, Anne, Mary]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If asked to cluster these names into two groups, we would surely expect [{Lisabeth, Lisa, Beth}, {Maryanne, Mary, Anne}]. &lt;/p&gt;&#xA;&#xA;&lt;h2&gt;However, no distance measure that insists on the triangular inequality would give us “Beth” and “Lisa” in the same group; since they do not share a single character with each other. Yet, both share one character with “Anne”.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There is a tutorial on DTW here &lt;a href=&quot;http://www.cs.unm.edu/~mueen/DTW.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.unm.edu/~mueen/DTW.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49544" LastActivityDate="2018-03-27T17:48:42.750" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29605" PostTypeId="2" ParentId="29602" CreationDate="2018-03-27T18:06:23.933" Score="1" Body="&lt;p&gt;This doesn't answer your Tableau questions, but I can say a few things about ggplot2. ggplot2's has reasonable defaults but nearly everything about the plots can be changed to make the results more clean aesthetically. One package I use all the time is called 'cowplot': &lt;a href=&quot;https://cran.r-project.org/web/packages/cowplot/vignettes/introduction.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/cowplot/vignettes/introduction.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ggplot2 also has built-in themes (&lt;a href=&quot;http://ggplot2.tidyverse.org/reference/ggtheme.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ggplot2.tidyverse.org/reference/ggtheme.html&lt;/a&gt;) and there's a package for community-develeoped themes (&lt;a href=&quot;https://cran.r-project.org/web/packages/ggthemes/vignettes/ggthemes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/ggthemes/vignettes/ggthemes.html&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-03-27T18:06:23.933" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29606" PostTypeId="2" ParentId="29602" CreationDate="2018-03-27T18:17:44.073" Score="5" Body="&lt;p&gt;There is one big economic difference between the two: ggplot2 is an open source package for an open source programming language. On the contrary Tableau is a proprietary software. That might be a dependency that you might not want to risk, e.g. you do everything in Tableau but then the license gets more expensive and your organization does not want to spend money for it anymore. Academics would often stress this reason, another reason of low adoption there is that Tableau has always been the BI tool and R the &quot;research&quot; tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from this, they are simply different tools (in the same domain of data viz) and as such will be suited for different problems. This can be subject to personal preference, but I would say that there is a few basic considerations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Speed: how fast can you get what you need?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use the tool that gets you results. Tableau might often be faster.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reproducibility &amp;amp; automation: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you know you might want to plot this chart many times in future in a consistent manner or embed your plots in some sort of notebook like R Markdown or dashboard, then R is most likely the better choice. You write the code once and produce thousands of plots within seconds if you need to.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Type of project:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you are starting with messy data, or you are doing some modeling work, it seems as a good strategy for me to just stick to R, I see it as more efficient to have everything in one place.&lt;/li&gt;&#xA;&lt;li&gt;If you need to get awesome visualizations from clean data, Tableau will excel&lt;/li&gt;&#xA;&lt;li&gt;Based on the type of visualizations and output you want, one or the other might be better, ggplot2 is extremely flexible, Tableau has a lot of amazing functionality which is ready to go.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="49542" LastActivityDate="2018-03-27T18:17:44.073" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29607" PostTypeId="1" CreationDate="2018-03-27T18:20:00.417" Score="0" ViewCount="110" Body="&lt;p&gt;When K-fold validation should be used and how to decide the value of the K.&#xA;I see most of the text books suggesting implementation of algorithms using 2/3 of data as train and 1/3 of data as test&lt;/p&gt;&#xA;" OwnerUserId="47747" LastActivityDate="2018-03-27T18:53:23.417" Title="How to decide to use K-fold validation or not?" Tags="|machine-learning|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29608" PostTypeId="2" ParentId="29607" CreationDate="2018-03-27T18:53:23.417" Score="1" Body="&lt;p&gt;The main criterion is that you need enough data in your training set to get a good model fit - which is both a function of the data quantity and the model complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you don't have much training data, you want as much data in your training set as possible - in this case, you can do &quot;leave one out&quot; cross-validation.  This is like k-fold, except it's N-fold, where N is the number of observations you have. This means you hold one observation out, build a model using everything else, validate using that one sample, and repeat N times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you have more data with a simple model, you can use a much more loose validation strategy. For example, if you have a million records and are doing linear regression with one predictor, you can get away with using a few thousand samples to build your model, since you'll get really good parameter estimates from thousands of samples (or fewer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Those are the extremes. The suggestions you've seen live somewhere in-between.  When you pick high K, you have to build lots of models (this takes time), but each model is built with more data (good for complex models).  When you pick low K, you have to build fewer models, but each model is built with less data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I pick a low value for K when I'm doing model exploration, but a higher value when I'm ready for validation of a good candidate model.  Actually, in this case you should have three sets - a training set, a validation set (that you use to find a good set of parameters), and a second validation set that you've held out the ENTIRE time (since, in a way, the other validation sets are helping you 'train' the model parameters).&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-03-27T18:53:23.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29610" PostTypeId="2" ParentId="29586" CreationDate="2018-03-27T20:05:28.713" Score="1" Body="&lt;p&gt;Neither.  What you really need is for the probability estimate to be accurate.  It's not enough to optimize just one of them (and you can't optimize for two metrics simultaneously, since better precision tends to lead to worse recall, and vice versa).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you minimize the cross-entropy loss function.  This is designed to help produce accurate estimates of the probability.  In some sense, that will optimize both precision and recall and trade them off in a particular way.  The specifics will depend on what classifier you use.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-27T20:05:28.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29611" PostTypeId="2" ParentId="29590" CreationDate="2018-03-27T20:09:23.090" Score="1" Body="&lt;p&gt;You can try to remove trend and seasonality and check whether the time series is stationary. If it is, you can calculate mean ($\mu$) and standard deviation ($\sigma$). Now for each data point, you calculate the distance from the mean ($\|y-\mu\|$) and check if it exceeds some threshold. The default value for threshold is $3 \sigma$ what comes from three-sigma rule motivated by the assumption about the normality of the data (residuals after time series decomposition in our case). You can choose any threshold which is reasonable for you. One solution is to find the distribution of the data and derive a rule for this particular distribution.&lt;/p&gt;&#xA;" OwnerUserId="45077" LastActivityDate="2018-03-27T20:09:23.090" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29612" PostTypeId="1" AcceptedAnswerId="29664" CreationDate="2018-03-27T20:10:41.733" Score="0" ViewCount="510" Body="&lt;p&gt;I am building a bot with python and I need some system to solve captchas like these:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f2KJu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f2KJu.png&quot; alt=&quot;captcha&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I need a deep learning algorithm, but coding one is a pain in the ass. Is there any easy solution to this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can code the part that screenshots the images and clicks on the correct answer, image 4 or 5 in the example. The images get rotated and change every time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="49550" LastActivityDate="2018-03-28T19:54:44.753" Title="Image Matching to solve captcha" Tags="|machine-learning|python|neural-network|similarity|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29614" PostTypeId="2" ParentId="10724" CreationDate="2018-03-27T21:25:21.160" Score="0" Body="&lt;p&gt;The weight matrix $W_1$ for input (document) and weight matrix $W_2$ for output (words) do not need to be of the same dimension. Say you have $M$ documents and $V$ words, then $W_1$ would be of dimension $M \times H$ and $W_2$ would be of dimension $H \times V$. $H$ is the dimension for embeddings. In other words, as long as the input and output of the neural network shares the same embedding dimension, then the model can work.&lt;/p&gt;&#xA;" OwnerUserId="48977" LastActivityDate="2018-03-27T21:25:21.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29615" PostTypeId="2" ParentId="29569" CreationDate="2018-03-27T21:27:39.733" Score="2" Body="&lt;p&gt;Many will tell you that normality tests are  overly sensitive, especially given that most statistical tests are robust to even gross departures from normality. If you are very concerned, conduct a parametric test anyway along with a nonparametric, and if the end results agree, stick with the parametric. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-03-27T21:27:39.733" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29616" PostTypeId="1" CreationDate="2018-03-27T21:33:42.157" Score="2" ViewCount="144" Body="&lt;pre&gt;&lt;code&gt;def distance_metric(seed, base):&#xA;    num = 0.0&#xA;    den = 0.0&#xA;    num = sum(numpy.minimum(seed,base))&#xA;    den = sum(numpy.maximum(seed,base))&#xA;    dist = round(1.0 - 1.0*num/den,4)&#xA;    return dist&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The metric is used to gauge similarity in the context of locality sensitive hashing. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Items within a bucket are kept if their distance is &amp;lt; 0.16.&lt;/p&gt;&#xA;" OwnerUserId="49556" LastEditorUserId="1330" LastEditDate="2020-06-17T15:14:39.223" LastActivityDate="2020-07-17T19:04:30.393" Title="What is the name of this similarity distance metric?" Tags="|clustering|algorithms|implementation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29618" PostTypeId="2" ParentId="29498" CreationDate="2018-03-27T21:58:35.713" Score="1" Body="&lt;p&gt;You can't assume the loss wouldn't drop with additional cycles, but it's generally safe to assume there are diminishing returns. This is why stopping rules are defined in terms of both number of iterations and change in the loss function. You want to stop when the gains become vanishingly small. At that point, increasing your accuracy from, say, 90.1% to 90.2% probably have little practical benefit, and performance on a hold-out sample probably won't differ much. But, that is an empirical question you can actually test. Run a model to near completion then run another with more stringent stopping criteria and compare on a hold-out. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-03-27T21:58:35.713" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29619" PostTypeId="2" ParentId="29437" CreationDate="2018-03-27T22:02:22.017" Score="0" Body="&lt;p&gt;You are talking about anomaly detection, and there are many approaches. If you can create a training set, one-class SVM is a place to start, but even simple control charts can be useful, particularly with live streaming data. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-03-27T22:02:22.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29620" PostTypeId="2" ParentId="29430" CreationDate="2018-03-27T22:06:59.847" Score="2" Body="&lt;p&gt;This is a controversial subject with no clear best answer and myriad options, some of which are model-specific. You can drop them, replace them with extreme values, interpolate, replace with median, impute with nearest neighbors, etc. In all cases besides dropping them altogether, you make assumptions and create data where no data exists.  I can't tell you the best approach, but I can advise you to try multiple approaches and hope the end results are robust to choice. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-03-27T22:06:59.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29622" PostTypeId="2" ParentId="29616" CreationDate="2018-03-27T22:59:47.977" Score="1" Body="&lt;p&gt;This is the weighted Jaccard Index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is different from the regular Jaccard Index (Similarity). &lt;/p&gt;&#xA;" OwnerUserId="49556" LastEditorUserId="49556" LastEditDate="2018-12-05T20:53:29.140" LastActivityDate="2018-12-05T20:53:29.140" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29624" PostTypeId="2" ParentId="29427" CreationDate="2018-03-28T01:40:09.117" Score="1" Body="&lt;p&gt;Couldn't you require your softmax output to exceed a threshold for the prediction or you call it &quot;not confident&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have not tried word prediction, but the softmax gives you a value for each output, where all of those values sum to 1. So pick the column with the highest value as your predicted word and use that value to determine your confidence. (It may be fairly low, since there might be a small chance of lots of words and then larger chances of your most-common three.) &lt;/p&gt;&#xA;" OwnerUserId="16613" LastActivityDate="2018-03-28T01:40:09.117" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29628" PostTypeId="2" ParentId="25782" CreationDate="2018-03-28T06:54:43.420" Score="1" Body="&lt;p&gt;The different dictionaries are in the &lt;a href=&quot;https://cran.r-project.org/web/packages/SentimentAnalysis/SentimentAnalysis.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation for the function&lt;/a&gt;: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example DictionaryHE is the &quot;Dictionary with opinionated words from Henry’s Financial dictionary&quot; (see DictionaryHE).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentiment &amp;lt;- analyzeSentiment(document)   # Document is a Corpus&#xA;sentimentData &amp;lt;- sentiment$SentimentHE   # Generates analysis based on Henry's dictionary&#xA;plotSentiment(sentimentData) # Quick line plot of sentiment&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49571" LastActivityDate="2018-03-28T06:54:43.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29629" PostTypeId="1" CreationDate="2018-03-28T07:11:57.830" Score="0" ViewCount="1952" Body="&lt;p&gt;I need to put the prediction into a complex function to calculate loss.&#xA;This means that i can't build the loss function by tensorflow's operator and can't get  gradient from Automatic Differentiation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to get gradient from finite difference in my training step.&lt;/p&gt;&#xA;" OwnerUserId="49570" LastEditorUserId="14675" LastEditDate="2018-03-28T11:15:22.403" LastActivityDate="2018-04-27T12:02:37.313" Title="How to use the finite difference to compute gradient for very complex loss function" Tags="|machine-learning|python|tensorflow|gradient-descent|loss-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29630" PostTypeId="2" ParentId="29316" CreationDate="2018-03-28T07:21:29.553" Score="0" Body="&lt;p&gt;Just sort the list descending, rather than ascending.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the farthest points come first.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-03-28T07:21:29.553" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29631" PostTypeId="1" CreationDate="2018-03-28T07:27:24.970" Score="2" ViewCount="408" Body="&lt;p&gt;I am trying to estimate a parameter by repeated sampling (Monte Carlo simulation). Each time I sample, I have to run a particular operation on a &lt;code&gt;data.table&lt;/code&gt; object. Here is an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;visits.site &amp;lt;- site.visits.map[site.name %in% sample.sites, &#xA;                              .(length(unique(visits))),&#xA;                              by = week]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since the number of samplings often run into millions, this is taking a lot of time. I have made the loop execute in parallel on the multiple cores of the CPU using the &lt;code&gt;parApply&lt;/code&gt; function. This has appreciably improved the performance. (I ran it over 8 cores). Note: This is an embarrassingly parallel problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to improve the performance even further by running this on GPU. I have looked at &lt;a href=&quot;https://cran.r-project.org/package=gpuR&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;gpuR&lt;/code&gt;&lt;/a&gt; but I am not sure if it has a function similar to &lt;code&gt;parApply&lt;/code&gt; for GPUs. Are there any packages that have this functionality in R?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If there are some other ways in which I can improve the performance, please let me know.&lt;/p&gt;&#xA;" OwnerUserId="49572" LastEditorUserId="24000" LastEditDate="2018-03-28T14:06:31.437" LastActivityDate="2018-03-28T14:06:31.437" Title="GPU Processing in R (using openCL)" Tags="|r|gpu|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29632" PostTypeId="2" ParentId="28040" CreationDate="2018-03-28T08:24:33.817" Score="1" Body="&lt;p&gt;It can also depend on how imbalanced the data is. If one class has 97% of the instances, then the model will always want to predicts that class. Have a close look at the prediction from your model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pred_class = model.predict_classes(X_test)&#xA;pred = model.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, if the prediction is always the same class you have a problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I notice some odd choices in your above code. When compiling your model change &lt;code&gt;loss='binary_crossentropy'&lt;/code&gt; to &lt;code&gt;loss='categorical_crossentropy'&lt;/code&gt; for multiclass classifications. I would also use softmax and not sigmoid for the exact same reason. &lt;/p&gt;&#xA;" OwnerUserId="49577" LastEditorUserId="49577" LastEditDate="2018-03-29T07:05:26.003" LastActivityDate="2018-03-29T07:05:26.003" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29633" PostTypeId="1" AcceptedAnswerId="29643" CreationDate="2018-03-28T08:44:31.073" Score="0" ViewCount="345" Body="&lt;p&gt;for my supervised classification problem, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a train dataset which contains past purchase data of customers and 5 new products are purchased by these customers. I have a test dataset which contains past purchase data of customers. They never bought from these 5 products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train a predictive model, I need to convert integer purchased data and new products. There are so many different products in my purchased data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KlXvK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KlXvK.png&quot; alt=&quot;my train dataset&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UX7qC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UX7qC.png&quot; alt=&quot;my test dataset&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I saw that the mapping was made for true-false to convert 1-0, but in this case what can I do that?&lt;/p&gt;&#xA;" OwnerUserId="49522" LastActivityDate="2018-03-28T11:15:19.747" Title="How do I convert strings in CSV into integer in Pandas?" Tags="|classification|predictive-modeling|pandas|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29634" PostTypeId="1" CreationDate="2018-03-28T08:49:04.513" Score="29" ViewCount="13672" Body="&lt;p&gt;Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. It seems that simply using concat(A, B) is not a good choice because A, B are totally different kinds of data. For example, unlike B, there is no numerical order in A. So my question is how to combine such two kinds of data or is there any conventional method to handle them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I propose a naive structure as presented in the picture&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QgQFq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QgQFq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you see, the first few layers are used to change(or map) data A to some middle output in continuous space and it is then being concated with data B which forms a new input feature in continuous space for later layers. I wonder whether it is reasonable or it is just a &quot;trial and error&quot; game. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="49119" LastActivityDate="2024-01-10T10:56:09.207" Title="How to combine categorical and continuous input features for neural network training" Tags="|neural-network|feature-selection|categorical-data|feature-construction|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29637" PostTypeId="2" ParentId="15149" CreationDate="2018-03-28T09:51:41.783" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/43103265/how-to-do-a-reverse-operation-to-tf-nn-embedding-lookup&quot;&gt;This answer&lt;/a&gt; describes how you go from a vector in the embedding space back to the the most similar class (e.g. word or character).&lt;/p&gt;&#xA;" OwnerUserId="49583" LastActivityDate="2018-03-28T09:51:41.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29638" PostTypeId="2" ParentId="29629" CreationDate="2018-03-28T09:55:29.957" Score="0" Body="&lt;p&gt;Finite difference is a way to calculate the derivative of a function according to its value. It is based on Taylor's theorem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have a good idea &lt;a href=&quot;https://en.wikipedia.org/wiki/Finite_difference&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and may be have look in &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/contrib/integrate/odeint&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tensorflow API&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's suppose &lt;strong&gt;J&lt;/strong&gt; is your cost function. The first question is to defined the definition domain of your &lt;strong&gt;J&lt;/strong&gt; in which apply finite difference discretization, I should have taken your different layers (but I have never tried this).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second question is linked to the fact that your error is &quot;backpropagated&quot; so when constructing your definition domain you should first think of the direction of your domain (from First layer to last or from last to first). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The you have to customize your gradient to use it in your TensorFlow implementation. So I suggest you to see &lt;a href=&quot;https://stackoverflow.com/questions/46876063/tensorflow-custom-gradients&quot;&gt;here&lt;/a&gt; and to see in StackOverflow if other people has already ask for what you want and if you can find more elaborate answers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it will help you &lt;/p&gt;&#xA;" OwnerUserId="46749" LastActivityDate="2018-03-28T09:55:29.957" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29640" PostTypeId="2" ParentId="13750" CreationDate="2018-03-28T10:59:45.113" Score="2" Body="&lt;p&gt;You would first have to uninstall tensorflow and after that install tensorflow-gpu. After that run your code and it would run on GPU provided you have installed gpu libraries such as CUDA and cuDNN.&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-03-28T10:59:45.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29642" PostTypeId="2" ParentId="29629" CreationDate="2018-03-28T11:13:41.420" Score="1" Body="&lt;p&gt;You may consider using a score function estimator (also known as REINFORCE), which defines an estimator of the gradient of a scoring function that does not need to be differentiable. This can be achieved thanks to the &lt;a href=&quot;http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/&quot; rel=&quot;nofollow noreferrer&quot;&gt;log derivative trick&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may find its mathematical foundations in &lt;a href=&quot;http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the original REINFORCE article by Williams&lt;/a&gt; or in Ian Goodfellow's seminal book on Deep Learning (section &lt;a href=&quot;http://www.deeplearningbook.org/contents/inference.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;20.9.1 Back-Propagating through Discrete Stochastic Operations&lt;/a&gt;).&lt;/p&gt;&#xA;" OwnerUserId="14675" LastActivityDate="2018-03-28T11:13:41.420" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29643" PostTypeId="2" ParentId="29633" CreationDate="2018-03-28T11:15:19.747" Score="0" Body="&lt;p&gt;You cannot convert your string product names to integers and expect    it to work, if you convert it to integers the algorithm you use will    expect a linear relationship between these integers, a relationship    that in fact does not exist.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is categorical data, and you should treat it like that, you have to one-hot encode it. If you have too many products to one-hot    encode all of them try grouping them in categories and then one-hot encoding the categories.&lt;/p&gt;&#xA;" OwnerUserId="45431" LastActivityDate="2018-03-28T11:15:19.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29647" PostTypeId="1" CreationDate="2018-03-28T12:43:40.490" Score="3" ViewCount="4320" Body="&lt;p&gt;I'm currently working on a common image classification with CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use both normalization (substract mean / divide by std per channel) and data augmentation (rotation, color, blur, ...) but I don't know how to use them together. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which order should I use ? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;First normalize with parameters based only on original images and then augment it (augment a normalized image is relevant ? should I ban some type of augmentation like color ?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Augment data and apply normalization based on all image (compute mean/ std with augmented images) which seems to be counterintuitive.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Augment data and apply normalization based on only original image which means that data are not really normalized&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Or don't use both methods  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="46143" LastActivityDate="2018-05-28T18:46:37.573" Title="Order of normalization / augmentation for image classification" Tags="|machine-learning|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29649" PostTypeId="2" ParentId="29595" CreationDate="2018-03-28T13:16:08.897" Score="1" Body="&lt;p&gt;I am not very sure about your case, usually I have seen the same problem but in the other direction, you can take a look here:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/Loss-cross-entropy-is-decreasing-but-accuracy-remains-the-same-while-training-convolutional-neural-networks-How-can-it-happen&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora loss drecrease, accuaracy constant&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stats.stackexchange.com/questions/202485/what-could-be-possible-reason-for-accuracy-in-convnets-being-constant-while-trai&quot;&gt;Cross validated loss drecrease, accuaracy constant&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;I would guess that your DNN is getting more answers right but without impoving its confidence  a lot.&lt;/p&gt;&#xA;&lt;p&gt;Regarding your other questions, I would advise you to start with simplest possible model, for example, in this case it would be using the vgg16 (which is widely more used than the vgg19) and the default parameters ( number of neurons, optimization function...) Once you have something working you can start tuning and modifying things.&lt;/p&gt;&#xA;" OwnerUserId="45431" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-03-28T13:16:08.897" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29650" PostTypeId="1" AcceptedAnswerId="29653" CreationDate="2018-03-28T13:21:10.100" Score="-1" ViewCount="146" Body="&lt;p&gt;I'm a complete newbie to the world of machine learning, and I'm currently working on implementing a model that will need to incorporate feedback, as well as changes to the data set (both feature &amp;amp; label changes over time).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequency of change isn't yet entirely known, but for simplicity could probably be rolled into a batch every day or so.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aware of how I can build a training &amp;amp; test set, and get a classifier up and running.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My primary issue is that it's probably not going to be ideal to run a completely fresh training every time there's a change.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Users will be interacting with the system via &quot;this was helpful / not helpful&quot; type feedback, which I want to use to strengthen / weaken its association model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm absolutely in the dark as to how once you have the model from the initial data, you can then get it to refine over time from this sort of feedback, and how to update (i.e. add/remove features &amp;amp; labels) without starting from scratch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tl;dr:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What sort of classifier is best suited to this sort of refinement-over-time problem?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I'll also add that the model needs to support multi-label classification, so any caveats / gotchas / information on how to do this in the broader context of my question would be helpful too.&lt;/p&gt;&#xA;" OwnerUserId="49597" LastActivityDate="2018-03-28T13:30:41.340" Title="How do machine learning models (e.g. neural networks) get better over time from new data?" Tags="|machine-learning|neural-network|classification|training|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29651" PostTypeId="2" ParentId="29647" CreationDate="2018-03-28T13:21:59.053" Score="3" Body="&lt;p&gt;You can use both methods, in the vast majority of examples I have seen and worked with the usual order is.  &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Augmentate your data, rotations, zoomings, flips....&lt;/li&gt;&#xA;&lt;li&gt;Normalize your images based on the new and augmented dataset.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Doing it like this you can also see the results of the augmentation phase, while if you normalize it first you cannot visualize it&lt;/p&gt;&#xA;" OwnerUserId="45431" LastActivityDate="2018-03-28T13:21:59.053" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29652" PostTypeId="2" ParentId="29647" CreationDate="2018-03-28T13:25:45.550" Score="0" Body="&lt;p&gt;Normalisation helps your neural net because it ensures that your input data always is within certain numeric boundaries, basically making it easier for the network to work with the data and to treat data samples equally. Augmentation creates &quot;new&quot; data samples that should be ideally as close as possible to &quot;real&quot; rather than synthetic data points. Therefore:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Apply augmentation to create synthetic data. &lt;/li&gt;&#xA;&lt;li&gt;Apply pre-processing on the whole data set. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think in practice it sometimes actually doesn't matter in which order you do it. Depending on which transformations you apply to the images they might still be normalized afterwards (e.g. if they are mirrored). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Effect on distribution&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A concern could be that the augmentation messes with the distribution of your data. However, it is likely that you perform linear transformations on your images. When your data is initially in a Gaussian distribution it will remain so after the transformation as explained by &lt;a href=&quot;http://www.stats.ox.ac.uk/~steffen/teaching/gm09/gauss.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="37822" LastEditorUserId="37822" LastEditDate="2018-03-29T14:47:41.060" LastActivityDate="2018-03-29T14:47:41.060" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29653" PostTypeId="2" ParentId="29650" CreationDate="2018-03-28T13:30:41.340" Score="0" Body="&lt;p&gt;If you only want to add more examples you can retrain the machine learning algorithm you had the day before. But if you want to add new features training it from the beginning is needed, you could train a new ML algorithm using only the new features and mix the outputs, but that is not a good solution, retrain the whole ML algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would use a neural network, which is very intuitive for your case, you calculate a set of weights and save them. When you get new data you load your old network and calculated set of weights and tune it using the new examples you have.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NN natively  support multilabel classification, and if one day you decide that you one to add a new label you dont need to retrain the whole NN, you could erase the last layer, add a new one, and only train this last layer weights.&lt;/p&gt;&#xA;" OwnerUserId="45431" LastActivityDate="2018-03-28T13:30:41.340" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29654" PostTypeId="1" AcceptedAnswerId="29658" CreationDate="2018-03-28T13:31:23.617" Score="4" ViewCount="5039" Body="&lt;p&gt;I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months).  I have a sequential dataset with 50 customers i.e. the rows contain sequence of 20 states for each of the 50 customers (dataset has 50 rows and 20 columns excluding the headers). I am trying to determine the next state using markov chains and all the literature in the web is focused around examples of text strings. I am looking something specific to the kind of example I have.&#xA;Can somebody please help me come up with the initial probability matrix and then consider the 20 states to predict the next state?&lt;/p&gt;&#xA;" OwnerUserId="49598" LastActivityDate="2018-06-19T05:22:12.440" Title="Markov Chains for sequential data" Tags="|python|markov-process|markov-hidden-model|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29655" PostTypeId="1" AcceptedAnswerId="29712" CreationDate="2018-03-28T14:40:45.193" Score="1" ViewCount="102" Body="&lt;p&gt;Features marked as boolean values (around 50,000) , need to score a set of fixed output (around 25,000). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have wikipedia topics relevant to a particular url as features. And the output is the advertisement keywords which will perform well on those url. If I have the training data, i.e. advertisement keywords performance on urls. Given a url (I can extract the features i.e. relevant wikipedia topics), I need to predict which keyword might perform well on that url.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the scientific name for such kind of problems in which there is a big boolean input set and I need to score a big fixed output set. Can you name few algorithms for such kind of problems. &lt;/p&gt;&#xA;" OwnerUserId="49603" LastActivityDate="2019-12-22T11:01:20.140" Title="Scientific name for the problem given in the description. (In machine learning what kind of problem it is termed as)" Tags="|machine-learning|deep-learning|algorithms|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29658" PostTypeId="2" ParentId="29654" CreationDate="2018-03-28T15:49:36.293" Score="3" Body="&lt;p&gt;If you know what the state history is, you don't need a 'hidden' Markov model, you just need a Markov model (or some other mechanism).  The 'hidden' part implies a distinction between some sequence of unobservable states, and some observations that are related to them.  In your case, you say you have observed the past states for each customer, so you don't necessarily need to infer anything 'hidden'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest way to proceed in your case would be to calculate a transition matrix, i.e. probability of state given previous state.  That's a very simple model but it might do what you want.  To do this, just look at all state pairs, and count to get p(s2 | s1) = p(s1 &amp;amp; s2)/p(s1).  This is equivalent to a 1-gram model that you've probably read about. Each state is akin to a word.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could also make a more complex model, like a 2-gram model or even an RNN.  Honestly, since you have a fixed amount of history, you can just throw your data into an scikit-learn model or xgboost or something, where each customer's history is the vector of predictors and the next state is the outcome.  It won't know the sequential dependencies, but you are essentially indexing the past states by time, so it may work pretty well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you need more clarification about part of this, just ask.&lt;/p&gt;&#xA;" OwnerUserId="41886" LastActivityDate="2018-03-28T15:49:36.293" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29661" PostTypeId="2" ParentId="29654" CreationDate="2018-03-28T18:29:40.143" Score="3" Body="&lt;p&gt;Because there is very little data HMM will probably overfit (depends on the number of states and letters). I would go with a simple markov chain as it has less parameters and you dont need to tune things like hidden states. If you are going with HMM I would recommend a package called Pomegranate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would also recommend try and using Multinomial model, which can be viewed as 0-memory markov model, maybe your data doesnt have past dependencies.&lt;/p&gt;&#xA;" OwnerUserId="49617" LastActivityDate="2018-03-28T18:29:40.143" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="29664" PostTypeId="2" ParentId="29612" CreationDate="2018-03-28T19:54:44.753" Score="1" Body="&lt;p&gt;Here's one simple yet effective solution without using the deep learning algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Divide the problem into 2 parts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Segmentation: You'll need to do some image processing like edge detection and segment all the objects in the given captcha and save the region of interest (OpenCV can help). Assign labels to the objects.&lt;/li&gt;&#xA;&lt;li&gt;Similarity Measurement: Use some scale and rotational invariant feature descriptor algorithm to extract features from the objects, and a matcher to compare the objects for a similarity measure (check SIFT/SURF features). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Based on the similarity scores, make a decision. &lt;/p&gt;&#xA;" OwnerUserId="42248" LastActivityDate="2018-03-28T19:54:44.753" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29665" PostTypeId="1" CreationDate="2018-03-28T19:55:39.177" Score="1" ViewCount="1925" Body="&lt;p&gt;I was adviced to write in this group regarding my question about modeling categorical database. &#xA;I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns(200 because most of the cases the questions were multiple choice and we had to split it into columns). Majority of variables are categorical or binary. I do not have continous variables at all. My task is to do customer segmentation, clustering. There is no initial assumptions although as I have also the questionnaire so can logically seperate the important questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I face several issues regarding the modeling&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I need to validate the choice of variables i use&lt;/li&gt;&#xA;&lt;li&gt;I am trying to find associations, pairwise associations and trends, as I do not have initial assumtions who can be my segments&lt;/li&gt;&#xA;&lt;li&gt;Clustering models are not working good for categorical variables and the ones I tried for example kmods, ignore the associations, correlations and return me not clear picture.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Can you please suggest how to approach, or from where to start. I am new in data analytics and I need some hints to go on with the analysis and I will be grateful to have some guidance at least high level what can be done.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="49620" LastActivityDate="2019-04-25T21:01:25.330" Title="customer segmentation with categorical variables" Tags="|r|clustering|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29666" PostTypeId="2" ParentId="27984" CreationDate="2018-03-28T20:38:32.823" Score="2" Body="&lt;p&gt;There seems to be an issue in Keras &lt;strong&gt;save_weights&lt;/strong&gt; and &lt;strong&gt;load_model&lt;/strong&gt; functions.&#xA;You can read more about it &lt;a href=&quot;https://github.com/alexander-rakhlin/CNN-for-Sentence-Classification-in-Keras/issues/13&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;here&lt;/strong&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/keras-team/keras/issues/4875&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;here&lt;/strong&gt;&lt;/a&gt; and &lt;a href=&quot;https://github.com/keras-team/keras/issues/4904&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;here&lt;/strong&gt;&lt;/a&gt;. So far  there is no workaround about it.&lt;/p&gt;&#xA;" OwnerUserId="49622" LastEditorUserId="49622" LastEditDate="2018-03-28T21:06:44.327" LastActivityDate="2018-03-28T21:06:44.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29668" PostTypeId="2" ParentId="29665" CreationDate="2018-03-28T20:49:07.437" Score="1" Body="&lt;p&gt;You can use mixture model with variable selection. In this framework, the challenge of variable selection consists in model selection. Thus, detection of the relevant features and estimation of the number of clusters can be done according to information criteria (like BIC or ICL).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To perform this analysis, you can use the R package VarSelLCM. Because you consider categorical variables, your dataset must be a data.frame and each column must be a factor. Here is an example of the script. Your dataset is denoted by &quot;my.data&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;## Clustering by considering all the variables as discriminative&#xA;# Number of clusters is between 1 and 6&#xA;res.all &amp;lt;- VarSelCluster(my.data, 1:6, vbleSelec = FALSE)&#xA;&#xA;# partition&#xA;res.all@partitions@zMAP&#xA;&#xA;# shiny application&#xA;VarSelShiny(res.all)&#xA;&#xA;&#xA;## Clustering with variable selection&#xA;# Number of clusters is between 1 and 6&#xA;res.selec &amp;lt;- VarSelCluster(my.data, 1:6, vbleSelec = TRUE)&#xA;&#xA;# partition&#xA;res.selec@partitions@zMAP&#xA;&#xA;# shiny application&#xA;VarSelShiny(res.selec)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49625" LastActivityDate="2018-03-28T20:49:07.437" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29671" PostTypeId="1" AcceptedAnswerId="29684" CreationDate="2018-03-28T21:17:59.260" Score="3" ViewCount="49469" Body="&lt;p&gt;I have a data frame of 3000 rows x 101 columns like as follow:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Time   id0  id1  id2     ………… id99&#xA;&#xA;1      1.71 6.99 4.01    ………… 4.98&#xA;&#xA;2      1.72 6.78 3.15    ………… 4.97&#xA;&#xA;.&#xA;&#xA;.&#xA;&#xA;3000   0.36 0.23 0.14    ………… 0.28&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using Python, how could we add a column that counts for each row the number of values (in column &lt;code&gt;id0&lt;/code&gt;, to &lt;code&gt;id99&lt;/code&gt;) that are within a specific range?&lt;/p&gt;&#xA;" OwnerUserId="49624" LastEditorUserId="24000" LastEditDate="2018-03-29T05:38:08.747" LastActivityDate="2018-03-29T07:08:41.363" Title="How to count occurrences of values within specific range by row" Tags="|python|pandas|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29672" PostTypeId="1" CreationDate="2018-03-28T21:51:47.933" Score="0" ViewCount="3252" Body="&lt;p&gt;I'm trying to implement an autoencoder for text. But I don't know which loss function I should use ? I tried using the &lt;strong&gt;mse&lt;/strong&gt; but I get a huge loss 1063442. I'm trying to build a very simple autoencoder using only LSTM layers in Keras. On the &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras-blog&lt;/a&gt; they use binary-crossentropy but I think the reason for this is because they use black and white images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="47600" LastActivityDate="2020-08-28T18:04:54.227" Title="What loss function should I use for auto encoders for text" Tags="|keras|loss-function|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29673" PostTypeId="1" CreationDate="2018-03-28T21:51:56.847" Score="2" ViewCount="12080" Body="&lt;p&gt;I want to replace the values in a data set (sample in the picture) using numbers instead of words, e.g., &lt;code&gt;1&lt;/code&gt; instead of &lt;code&gt;D&lt;/code&gt;, &lt;code&gt;-1&lt;/code&gt; instead of &lt;code&gt;R&lt;/code&gt;, &lt;code&gt;0&lt;/code&gt; for all other values. How can I do it with a loop? I know it can be done doing this instead: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(suppose &lt;code&gt;d&lt;/code&gt; is index name) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d[d$Response == &quot;R&quot;,]$Response = -1&#xA;&#xA;d[d$Response == &quot;D&quot;,]$Response = 1&#xA;&#xA;... (other values code it and assign value of) = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cXNlG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cXNlG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49628" LastEditorUserId="24000" LastEditDate="2018-03-29T13:06:11.173" LastActivityDate="2018-12-25T18:03:00.560" Title="Replacing words by numbers in multiple columns of a data frame in R" Tags="|r|dataframe|rstudio|numerical|word|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29677" PostTypeId="1" CreationDate="2018-03-28T23:15:56.240" Score="1" ViewCount="45" Body="&lt;p&gt;I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal convolutional network &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(more or less this one)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ka3mU.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ka3mU.jpg&quot; alt=&quot;screenshot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For X spatio-temporal feature extraction. The fact is I have data with different timesteps (for example 1 dataset is 500 years long and as a 10 years timestep, and another is 100 years long ans has a 1 year timestep). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if someone has an idea of how I can include length and timestep as inputs of my model in order my model will be able to deal with different length and timesteps?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much for your help,&#xA;Marin&lt;/p&gt;&#xA;" OwnerUserId="49630" LastEditorUserId="29169" LastEditDate="2021-02-13T16:35:16.387" LastActivityDate="2021-02-13T16:35:16.387" Title="Variable TimeStep in Spatio Temporal 3DConvNet" Tags="|time-series|preprocessing|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29678" PostTypeId="2" ParentId="29596" CreationDate="2018-03-28T23:26:01.653" Score="1" Body="&lt;p&gt;Numpy is best.  You can just take &lt;/p&gt;&#xA;&#xA;&lt;p&gt;np.array(dict.values()). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Eg,  dict = Paris_attck&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most ML libraries operate on numpy arrays.&lt;/p&gt;&#xA;" OwnerUserId="35437" LastActivityDate="2018-03-28T23:26:01.653" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29681" PostTypeId="2" ParentId="29596" CreationDate="2018-03-29T02:35:24.203" Score="1" Body="&lt;p&gt;Lists in Python are extremely powerful and working with a list of lists is not a complicated procedure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Something like the following may be something you'd like to investigate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;events = [] &#xA;for event in range(0,len(df.index)):&#xA;   an_event = []&#xA;   an_event.append(df[event,'ages'])&#xA;   an_event.append(df[event,'density'])&#xA;   events.append(an_event)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="45216" LastActivityDate="2018-03-29T02:35:24.203" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29682" PostTypeId="1" AcceptedAnswerId="29787" CreationDate="2018-03-29T04:11:21.467" Score="2" ViewCount="169" Body="&lt;p&gt;This has been a spot of confusion for me lately upon reading more into regularisation in neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have always viewed weights as being a measure of the importance of a feature in a model. E.g. A model of ice cream sales starts off with random weights and eventually learns to increase the weights connected to the temperature feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With regularisation (L2, I have been looking at most recently), small weights are favoured more by the model. This makes good sense to me when we are trying to prevent overfitting in our models. The model generalises well between different pieces of input data because the smaller weights give their associated feature less of an impact in the model. This will draw out the true patterns of the model as opposed to a few cases that may skew the weights. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now my confusion is what if a feature like temperature is a consistent predictor of ice cream sales in our model, what does its associated weights look like after regularisation? Is their impact on the cost function so great that the model doesn't prefer smaller weights instead? Also how do folks usually set a value for lambda? &lt;/p&gt;&#xA;" OwnerUserId="45216" LastActivityDate="2018-04-03T05:16:32.487" Title="Understanding regularisation and a preference for small weights" Tags="|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29684" PostTypeId="2" ParentId="29671" CreationDate="2018-03-29T05:37:11.177" Score="1" Body="&lt;p&gt;You can apply a function to each row of the DataFrame with &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;apply&lt;/code&gt;&lt;/a&gt; method. In the applied function, you can first transform the row into a boolean array using &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.between.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;between&lt;/code&gt;&lt;/a&gt; method or with standard relational operators, and then count the &lt;code&gt;True&lt;/code&gt; values of the boolean array with &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sum.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;sum&lt;/code&gt;&lt;/a&gt; method.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame({&#xA;    'id0': [1.71, 1.72, 1.72, 1.23, 1.71],&#xA;    'id1': [6.99, 6.78, 6.01, 8.78, 6.43],&#xA;    'id2': [3.11, 3.11, 4.99, 0.11, 2.88]})&#xA;&#xA;&#xA;def count_values_in_range(series, range_min, range_max):&#xA;&#xA;    # &quot;between&quot; returns a boolean Series equivalent to left &amp;lt;= series &amp;lt;= right.&#xA;    # NA values will be treated as False.&#xA;    return series.between(left=range_min, right=range_max).sum()&#xA;&#xA;    # Alternative approach:&#xA;    # return ((range_min &amp;lt;= series) &amp;amp; (series &amp;lt;= range_max)).sum()&#xA;&#xA;&#xA;range_min, range_max = 1.72, 6.43&#xA;&#xA;df[&quot;n_values_in_range&quot;] = df.apply(&#xA;    func=lambda row: count_values_in_range(row, range_min, range_max), axis=1)&#xA;&#xA;print(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Resulting DataFrame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    id0   id1   id2  n_values_in_range&#xA;0  1.71  6.99  3.11                  1&#xA;1  1.72  6.78  3.11                  2&#xA;2  1.72  6.01  4.99                  3&#xA;3  1.23  8.78  0.11                  0&#xA;4  1.71  6.43  2.88                  2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2018-03-29T05:37:11.177" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29685" PostTypeId="2" ParentId="29671" CreationDate="2018-03-29T07:08:41.363" Score="1" Body="&lt;p&gt;IIUC you can use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.isin.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataFrame.isin()&lt;/a&gt; method:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [41]: given_set = {3,8,11,18,22,24,35,36,42,47}&#xA;&#xA;In [42]: df&#xA;Out[42]:&#xA;    a   b   c   d   e&#xA;0  36  38  27  12  35&#xA;1  45  33   8  41  18&#xA;4  32  14   4  14   9&#xA;5  43   1  31  11   3&#xA;6  16   8   3  17  39&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Solution:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [44]: df['new'] = df.isin(given_set).sum(1)&#xA;&#xA;In [45]: df&#xA;Out[45]:&#xA;    a   b   c   d   e  new&#xA;0  36  38  27  12  35    2&#xA;1  45  33   8  41  18    2&#xA;4  32  14   4  14   9    0&#xA;5  43   1  31  11   3    2&#xA;6  16   8   3  17  39    2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Explanation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [49]: df.isin(given_set)&#xA;Out[49]:&#xA;       a      b      c      d      e&#xA;0   True  False  False  False   True&#xA;1  False  False   True  False   True&#xA;4  False  False  False  False  False&#xA;5  False  False  False   True   True&#xA;6  False   True   True  False  False&#xA;&#xA;In [50]: df.isin(given_set).sum(1)&#xA;Out[50]:&#xA;0    2&#xA;1    2&#xA;4    0&#xA;5    2&#xA;6    2&#xA;dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if you want check for existence instead of counting, you can do it this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [6]: df.isin(given_set).any(1)&#xA;Out[6]:&#xA;0     True&#xA;1     True&#xA;4    False&#xA;5     True&#xA;6     True&#xA;dtype: bool&#xA;&#xA;In [7]: df.isin(given_set).any(1).astype(np.uint8)&#xA;Out[7]:&#xA;0    1&#xA;1    1&#xA;4    0&#xA;5    1&#xA;6    1&#xA;dtype: uint8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-03-29T07:08:41.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29686" PostTypeId="2" ParentId="9747" CreationDate="2018-03-29T07:16:33.620" Score="4" Body="&lt;p&gt;&lt;strong&gt;What is Apache Flume?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Apache Flume is a tool for designed for streaming data ingestion in HDFS. &#xA;Objective: The main objective of Flume is to capture streaming data from various web servers to HDFS.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Applications of Flume&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The applications of flume are,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Flume is used in e-commerce Company to analyze the customer behavior of different regions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;It is used to feed huge log data generated by application servers into &lt;code&gt;HDF5&lt;/code&gt; at a higher speed.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What are prerequisites for learning it?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Basics of Hadoop, Big-data is must.&lt;/li&gt;&#xA;&lt;li&gt;Basics of Linux and scripting&lt;/li&gt;&#xA;&lt;li&gt;Main thing is interest towards technology.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For more information refer &lt;a href=&quot;http://beyondcorner.com/learn-apache-flume/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Apache Flume&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="49648" LastEditorUserId="35644" LastEditDate="2018-03-29T13:06:06.583" LastActivityDate="2018-03-29T13:06:06.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29688" PostTypeId="2" ParentId="29673" CreationDate="2018-03-29T09:24:33.337" Score="1" Body="&lt;p&gt;I guess this is off-topic but here's a reproducible code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Preparing mocking data&#xA;&amp;gt;set.seed(123) #for reproducibility&#xA;&amp;gt;Party &amp;lt;- sample(c('D', 'N','R'), 10, replace = TRUE) #sample random values&#xA;&amp;gt;d &amp;lt;- data.frame(Party) #put values into a dataframe&#xA;&#xA;# Here's how you substitute the values in just one line. This operation is vectorised.&#xA;&amp;gt;d$Response &amp;lt;- ifelse(d$Party=='R', -1, ifelse(d$Party=='D', 1, 0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically in new &lt;code&gt;Response&lt;/code&gt; column, you check if &lt;code&gt;R&lt;/code&gt; and assign -1, if not you check if &lt;code&gt;D&lt;/code&gt; and assign 1; otherwise you assign 0.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; d&#xA;   Party Response&#xA;1      D        1&#xA;2      R       -1&#xA;3      N        0&#xA;4      R       -1&#xA;5      R       -1&#xA;6      D        1&#xA;7      N        0&#xA;8      R       -1&#xA;9      N        0&#xA;10     N        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-03-29T09:24:33.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29689" PostTypeId="1" CreationDate="2018-03-29T10:02:30.140" Score="2" ViewCount="81" Body="&lt;p&gt;I am going to build a 1,000,000 * 2 dataframe in Python and the input I have is in the form of 1*2. So obviously the most straightforward method is to use loop and append a 1*2 pair into the existing dataframe for 1,000,000 times... However, it really takes forever and as a Matlab user, I am almost forbidden in Matlab to do this and always told to initiate the final size of the matrix (or dataframe in Python) first and put elements into the matrix... Is this the same in Python? Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="46454" LastActivityDate="2018-03-29T14:11:36.073" Title="Should I initiate size before loop and append in Python" Tags="|python|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="29691" PostTypeId="1" CreationDate="2018-03-29T13:27:09.953" Score="3" ViewCount="95" Body="&lt;p&gt;Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,&#xA;I want to know how can we take the derivative of $f \text{ w.r.t. } X$? I understand differentiation w.r.t one variable, but how does differentiation of another vector/matrix w.r.t a vector work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Differentiating a function w.r.t a variable gives us the rate at with the function changes when we change the variable by a small amount. What does differentiating w.r.t a vector signify?&lt;/p&gt;&#xA;" OwnerUserId="49664" LastEditorUserId="26562" LastEditDate="2018-04-29T11:33:55.750" LastActivityDate="2018-09-27T04:32:06.563" Title="Derivates with respect to a vector" Tags="|linear-algebra|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29692" PostTypeId="2" ParentId="27124" CreationDate="2018-03-29T13:27:18.187" Score="0" Body="&lt;p&gt;You could explicitly provide correlation as an nn input, as calculated in &lt;a href=&quot;https://pdfs.semanticscholar.org/6cc7/7f1d1860a8e3766f2a9ece65f90c31632582.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Local Correlation Tracking in Time Series&lt;/a&gt;, with code available via &lt;a href=&quot;http://dovgalecs.com/blog/matlab-local-correlation-tracking-in-time-series/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vlad's Blog&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also read a paper ( unfortunately I can't remember its title or names of authors for a google search ) where a prediction of future correlation is an explicit nn target as part of a multivariate target nn target. The authors claimed inclusion of correlation as a target improved the accuracy on the non-correlation part of the multivariate target.&lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2018-03-29T13:27:18.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29693" PostTypeId="1" AcceptedAnswerId="29734" CreationDate="2018-03-29T13:29:17.510" Score="2" ViewCount="365" Body="&lt;p&gt;I'm pretty new to machine learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I can represent a set of discrete values as a vector of 0/1 values. For instance, in the set of features {a, b, c, d, e}, the subset containing &lt;code&gt;{a, c}&lt;/code&gt; can be represented as &lt;code&gt;[1, 0, 1, 0, 0]&lt;/code&gt; and the subset containing &lt;code&gt;{c, d, e}&lt;/code&gt; can be represented as &lt;code&gt;[0, 0, 1, 1, 1]&lt;/code&gt;, meaning I have as many dimensions as elements, which is workable when you have a finite (and small) number of elements.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now, for a clustering task, I want to represent sets of sets, like, for instance, representing the set &lt;code&gt;{{a, c}, {c, d, e}}&lt;/code&gt;. How can I do that? Here, the basic 0/1 approach won't work, as I'll have &lt;code&gt;2^n&lt;/code&gt; possible combinations. What is the workaround, if any?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: here is the transcription as a less abstract, more business problem. I want to find clusters of people according to the trips they made. A trip consists in a set of cities visited, and a set of transportation used. For instance, people might have visited cities in the set &lt;code&gt;{Rabat, Alger, Marrakech, Tunis, Hammamet}&lt;/code&gt; with transportation such as &lt;code&gt;{car, plane, train}&lt;/code&gt;. A trip could be &lt;code&gt;{Rabat, Marrakech, plane}&lt;/code&gt; or &lt;code&gt;{Alger, Marrakech, Tunis, car, train}&lt;/code&gt;. Note the order in which cities were visited, or the order in which vehicles were used, is not considered.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of the items I want to find clusters of could be a person having made those two trips, represented as &lt;code&gt;p1 = {{Rabat, Marrakech, plane}, {Alger, Marrakech, Tunis, car, train}}&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49665" LastEditorUserId="49665" LastEditDate="2018-03-30T08:35:36.260" LastActivityDate="2018-04-12T15:10:06.767" Title="How to represent a set of sets as a vector" Tags="|clustering|feature-engineering|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29695" PostTypeId="1" AcceptedAnswerId="29709" CreationDate="2018-03-29T13:39:52.310" Score="2" ViewCount="688" Body="&lt;p&gt;I'm a beginner in NNs and the first thing I don't understand with batch norm is the following 2 steps:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;First we normalize the batch data on a z parameter to Mu=0, sigma^2=1&lt;/li&gt;&#xA;&lt;li&gt;Then we change z via the coefficients of Mu, sigma^2 (usu. alpha, beta) by updating them as learnable parameters.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't understand why the first step is necessary if we change the distribution in the second step anyway. Could someone explain please?&lt;/p&gt;&#xA;" OwnerUserId="49667" LastEditorUserId="49667" LastEditDate="2018-03-29T14:22:05.007" LastActivityDate="2018-03-30T06:25:06.777" Title="Batch norm: why the initial normalization?" Tags="|neural-network|gradient-descent|batch-normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29696" PostTypeId="2" ParentId="29689" CreationDate="2018-03-29T14:11:36.073" Score="1" Body="&lt;p&gt;Yes it will be just as slow as in matlab. You’re going to be copying the whole data frame every time you append something. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-03-29T14:11:36.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29698" PostTypeId="1" CreationDate="2018-03-29T15:43:39.860" Score="1" ViewCount="991" Body="&lt;p&gt;I'm working on a multiclass classification problem using Keras. I have nearly 17k training data, and approximately 14k testing data (validation comes from the 17k training data). I have a number of issues with my model which I would describe as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Validation accuracy starts with a high of ~51% after three epochs and then starts decreasing to the point of after 100 epochs the validation accuracy is ~23%.&lt;/li&gt;&#xA;&lt;li&gt;Validation loss value starts of at 6.2354, decreases for two epochs, increases for 10 epochs and then starts lowering again.&lt;/li&gt;&#xA;&lt;li&gt;Training accuracy also decreases after 20 epochs reaching a high of ~40% and then slowly dropping with slight fluctations. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above issues all occur while the loss value for the model is continually dropping. If anyone could help with this model it will be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my code, I am each piece of data contains 4096 values each of which are floating point values, and I am trying to classify a class that comes in the form of a size 10 array that contains a number of 0s and 1s depending on the class that it is trying to classify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also included a visual depictions of the way in which the loss value is behaving in the model for 100 epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;final_model = Sequential()&#xA;&#xA;final_model.add(Dense(2048, activation='relu', input_shape=X.shape[1:]))&#xA;final_model.add(Dropout(0.5))&#xA;&#xA;final_model.add(Dense(1024, activation='relu'))&#xA;final_model.add(Dropout(0.5))&#xA;&#xA;final_model.add(Dense(1024, activation='relu'))&#xA;final_model.add(Dropout(0.5))&#xA;&#xA;&#xA;final_model.add(Dense(512, activation='relu'))&#xA;final_model.add(Dropout(0.5))&#xA;&#xA;final_model.add(Dense(10, activation='softmax'))&#xA;&#xA;cust = RMSprop(lr=0.00001, rho=0.9)&#xA;&#xA;final_model.compile(optimizer=cust, loss='categorical_crossentropy', metrics=['categorical_accuracy'])&#xA;&#xA;history = final_model.fit(X, y, epochs=100, batch_size=128, verbose=2, validation_split=0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5QxCm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5QxCm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47061" LastActivityDate="2018-03-29T15:43:39.860" Title="Why does my model accuracy rise and then drop, with the loss sharing similar characteristics?" Tags="|deep-learning|keras|multiclass-classification|multilabel-classification|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29699" PostTypeId="1" CreationDate="2018-03-29T16:20:13.723" Score="3" ViewCount="2276" Body="&lt;p&gt;While this may be slightly off-topic, this question does pertain to data science and machine learning. I want to train a VGG16 model on Imagenet from scratch. For this purpose, I am looking into either buying four to six GTX 1080 Ti's or one Tesla V100. I have a feeling that four GTX 1080 Ti's will perform far better than a single V100, but has there been any practical research into this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT - Each GTX 1080 Ti has 11.3 teraflops of FP32 performance (or 45.2 teraflops for four 1080 Ti's, or slightly less since the scaling isn't linear), whereas the V100 is 100 teraflops, or more than double the performance of four GTX 1080 Ti's. Anyone have any benchmarks for these two on imagenet performance?&lt;/p&gt;&#xA;" OwnerUserId="245" LastEditorUserId="245" LastEditDate="2018-03-29T18:56:05.490" LastActivityDate="2018-06-02T22:37:15.280" Title="Performance of four GTX 1080 Ti's versus one Tesla V100 for Deep Neural Network training" Tags="|machine-learning|hardware|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29701" PostTypeId="2" ParentId="29695" CreationDate="2018-03-29T17:41:33.810" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/io7Mf.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/io7Mf.jpg&quot; alt=&quot;The image says it all&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope I got your question correctly...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It’s called “Batch” Normalization because we perform this transformation and calculate the statistics only for a subpart (a batch) of the entire training set not as a whole.&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-03-29T18:12:07.457" LastActivityDate="2018-03-29T18:12:07.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29702" PostTypeId="2" ParentId="29691" CreationDate="2018-03-29T17:58:26.120" Score="-1" Body="&lt;p&gt;Derivative of a univariate vector is the same as sum of derivatives of its component(Addition rule for differentiation). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for an extra $Y$ matrix in multiplication with $X^T$, the derivative of $f$ is calculated using partial derivative rule. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please see the link &lt;a href=&quot;https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/partial-derivative-and-gradient-articles/a/introduction-to-partial-derivatives&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43770" LastEditorUserId="38892" LastEditDate="2018-03-30T03:54:05.857" LastActivityDate="2018-03-30T03:54:05.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29703" PostTypeId="2" ParentId="29542" CreationDate="2018-03-29T18:24:26.053" Score="2" Body="&lt;p&gt;Giving links some good tutorial which will help you understand concepts of deep learning and give you enough background to be able to implement simple models for your own problems.&lt;p&gt;&#xA;&lt;a href=&quot;https://in.udacity.com/course/deep-learning--ud730&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://in.udacity.com/course/deep-learning--ud730&lt;/a&gt;&lt;br&gt;&#xA;It's approx. duration is three months but with a little more effort you should be able to finish it in two.&lt;br&gt;&#xA;Another one is Andrew Ng course on Coursera. You should really check it out.&lt;br&gt;&#xA;Quick intro to Deep Learning:&#xA;&lt;a href=&quot;https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43770" LastEditorUserId="43770" LastEditDate="2018-03-30T06:01:37.780" LastActivityDate="2018-03-30T06:01:37.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29704" PostTypeId="1" CreationDate="2018-03-29T18:41:55.737" Score="0" ViewCount="2530" Body="&lt;p&gt;I am using crypto currency chart data and was wondering what would be the best process for scaling the time series data. The issue is that some of these currencies are highly volatile and can see extreme swings. The standard process that I see is something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;from sklearn.preprocessing import MinMaxScaler&#xA;x_train, y_train, x_test, y_test = train_test_split(x,y, test_size=0.2)&#xA;scaler = MinMaxScaler()&#xA;scaler.fit(x_train)&#xA;x_train = scaler.transform(x_train)&#xA;x_test = scaler.transform(x_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By default MinMaxScaler scales between 0 and 1 but there are cases where x_test may contain a max that could be double or more what the max was when the scaler was fit. Will this cause model issues or is this acceptable? How do you handle scaling data that you don't know the min/max for? Is min/max scaling best for this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, some currencies have drastically different prices. For example, one could be trading +10x or -10x or more per unit than another. Will having several time series on drastically different scales effect the models? Its seems to me that scaling each time series and each feature to 0-1 would be the best but I am not completely sure. Lets say I have n different time series that have drastically different values. Would something like the following be a decent approach?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#assume data is a list of padas DataFrames&#xA;#e.g. data = [pd.DataFrame(asset1), pd.DataFrame(asset2)]&#xA;scaled = []&#xA;for asset in data:&#xA;    fit_data = data.iloc[:int(len(data*0.2))]&#xA;    scaler = MinMaxScaler()&#xA;    scaler.fit(fit_data)&#xA;    scaled.append(scaler.transform(asset))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49621" LastActivityDate="2018-08-03T08:02:39.917" Title="Scaling multiple time series data" Tags="|scikit-learn|time-series|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29706" PostTypeId="2" ParentId="5226" CreationDate="2018-03-29T19:52:16.987" Score="2" Body="&lt;p&gt;Turn them to numbers, for example for each unique country assingn a unique number (like 1,2,3 and ...) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;also you &lt;strong&gt;Don't&lt;/strong&gt; need to use &lt;strong&gt;One-Hot Encoding&lt;/strong&gt; (aka dummy variables) when working with random forest, because trees don't work like other algorithm (such as linear/logistic regression) and they don't work by distant (they work with finding good split for your features) so &lt;strong&gt;NO NEED&lt;/strong&gt; for One-Hot Encoding&lt;/p&gt;&#xA;" OwnerUserId="29752" LastActivityDate="2018-03-29T19:52:16.987" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29709" PostTypeId="2" ParentId="29695" CreationDate="2018-03-29T22:20:17.747" Score="0" Body="&lt;p&gt;The first step helps to reduce something called “internal covariate shift” of the network. Normalizing the layer inputs before applying the shift and scaling in step two, speeds up the training process (see &lt;a href=&quot;https://arxiv.org/pdf/1502.03167.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;BN paper&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This normalization comes with a cost, namely, it can reduce the number of possible representations a layer can provide.  E.g.  Normalized inputs of a sigmoid are constrained to the linear regime of the function. See &lt;a href=&quot;https://arxiv.org/pdf/1502.03167.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;BN paper&lt;/a&gt; on page three.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second step is there to address this problem. Scaling and shifting the values to “not just” linear domains of the nonlinearities solves the representional issue while keeping the internal covariate shift at a minimum. &lt;/p&gt;&#xA;" OwnerUserId="47739" LastEditorUserId="47739" LastEditDate="2018-03-30T06:25:06.777" LastActivityDate="2018-03-30T06:25:06.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29710" PostTypeId="1" CreationDate="2018-03-29T22:46:15.660" Score="2" ViewCount="3031" Body="&lt;p&gt;I have a JSON dataset, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&quot;candidate00001&quot;: &quot;graceful ones.&quot;&#xA; &quot;candidate00002&quot;:&quot;One more,Marvelous said, sounding royally bored from his seat.&quot;&#xA; &quot;candidate00003&quot;:&quot;She tired,Joe said, though not unkindly.&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;in which, &lt;code&gt;candidate00001&lt;/code&gt; is my first class, &lt;code&gt;candidate00002&lt;/code&gt; is the second class and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to python, so i want to implement a character level classification by &lt;strong&gt;Keras&lt;/strong&gt; using &lt;a href=&quot;http://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; tutorial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my problem is that I don't now how to convert my JSON file as &lt;em&gt;x_train&lt;/em&gt; and &lt;em&gt;y_train&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49604" LastEditorUserId="29575" LastEditDate="2018-04-08T15:53:29.017" LastActivityDate="2018-06-07T17:41:49.857" Title="How to feed my JSON dataset in Keras for character level text classification" Tags="|classification|dataset|keras|text-mining|cnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29712" PostTypeId="2" ParentId="29655" CreationDate="2018-03-30T00:12:38.167" Score="0" Body="&lt;p&gt;Ok. I think I'm getting a clearer picture. Firstly any name isn't going to be different just because the data is large. Of course in the big data community anyone is free to coin a term and time will show if it gets adopted (same goes with what we might consider a scientific name for that matter too).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing is input output. These terms can be confusing as what is input by a comp scientist into a training model is both the input and output of what a statistician would define.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not aware of a name for the problems but names for models addressing such a problem include logistic regression and probit regression. Both deal with a(n) (set of) independent variable(s) - the feature name in your case - and a resultant dependent variable - the binary relevancy. The regression looks at fitting a linear function to a hidden variable; whose threshold gives the binary result. Fitting an &quot;input&quot; training set then allows the scoring based on this linear function, whether you want a binary score or a real valued one (from the hidden variable) that will be more indicative of the probability of your prediction.&lt;/p&gt;&#xA;" OwnerUserId="49195" LastActivityDate="2018-03-30T00:12:38.167" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29713" PostTypeId="2" ParentId="28608" CreationDate="2018-03-30T02:59:56.993" Score="1" Body="&lt;p&gt;You may use directly the &lt;code&gt;IScritical&lt;/code&gt; feature you created:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df=pd.read_csv('DATA2.csv')&#xA;&#xA;Search_for_These_values = ['PEA', 'DEF', 'XYZ'] #creating list&#xA;&#xA;pattern = '|'.join(Search_for_These_values)     # joining list for comparision&#xA;&#xA;IScritical=df['COLUMN_to_Check'].str.contains(pattern)&#xA;&#xA;df['NEWcolumn'] = IScritical.replace((True,False), ('YES','NO'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12490" LastActivityDate="2018-03-30T02:59:56.993" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29714" PostTypeId="1" CreationDate="2018-03-30T04:34:26.070" Score="2" ViewCount="483" Body="&lt;p&gt;Iam training a Keras model for end-to-end speech recognition. I have my own dataset of speech containing about 400 wave files. Text transcriptions is also given as input. Model summary is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;_________________________________________________________________&#xA;Layer (type)                 Output Shape              Param #   &#xA;=================================================================&#xA;the_input (InputLayer)       (None, None, 26)          0         &#xA;_________________________________________________________________&#xA;layer_1_conv (Conv1D)        (None, None, 30)          3930      &#xA;_________________________________________________________________&#xA;conv_batch_norm (BatchNormal (None, None, 30)          120       &#xA;_________________________________________________________________&#xA;rnn_bi (GRU)                 (None, None, 40)          8520      &#xA;_________________________________________________________________&#xA;bt_rnn_bi (BatchNormalizatio (None, None, 40)          160       &#xA;_________________________________________________________________&#xA;bidirectional_15 (Bidirectio (None, None, 40)          19440     &#xA;_________________________________________________________________&#xA;bt_rnn_final (BatchNormaliza (None, None, 40)          160       &#xA;_________________________________________________________________&#xA;time_distributed_15 (TimeDis (None, None, 29)          1189      &#xA;_________________________________________________________________&#xA;softmax (Activation)         (None, None, 29)          0         &#xA;=================================================================&#xA;Total params: 33,519&#xA;Trainable params: 33,299&#xA;Non-trainable params: 220&#xA;_________________________________________________________________&#xA;None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Optimiser used: &lt;code&gt;Adadelta()&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Loss function: &lt;code&gt;ctc_loss&lt;/code&gt; function.&lt;/li&gt;&#xA;&lt;li&gt;Dropout: 0.5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Training and validation loss in last epochs is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 392/400&#xA;27/27 [==============================] - 36s - loss: 19.9499 - val_loss: 16.5945&#xA;&#xA;Epoch 393/400&#xA;27/27 [==============================] - 34s - loss: 18.9789 - val_loss: 14.1015&#xA;&#xA;Epoch 394/400&#xA;27/27 [==============================] - 36s - loss: 17.9598 - val_loss: 14.2997&#xA;&#xA;Epoch 395/400&#xA;27/27 [==============================] - 34s - loss: 17.1506 - val_loss: 15.1215&#xA;&#xA;Epoch 396/400&#xA;27/27 [==============================] - 35s - loss: 17.4900 - val_loss: 14.0334&#xA;&#xA;Epoch 397/400&#xA;27/27 [==============================] - 35s - loss: 17.7459 - val_loss: 14.7812&#xA;&#xA;Epoch 398/400&#xA;27/27 [==============================] - 35s - loss: 18.3460 -  val_loss: 14.4461&#xA;&#xA;Epoch 399/400&#xA;27/27 [==============================] - 35s - loss: 17.4311 - val_loss: 15.5965&#xA;&#xA;Epoch 400/400&#xA;27/27 [==============================] - 35s - loss: 17.6892 - val_loss: 12.4165&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anybody explain to me how this loss is interpreted? What could be the correct values of training loss and validation loss, so my model correctly predicts the output values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried reducing the loss and it did reduce, with the difference of plus or minus 1 unit, but test tests were totally incorrect. Can anyone suggest me ways to gain correct test results?&lt;/p&gt;&#xA;" OwnerUserId="49690" LastEditorUserId="24000" LastEditDate="2018-03-30T14:46:59.680" LastActivityDate="2019-05-18T07:27:01.047" Title="Validation loss is less than training loss by 5 units. How this result is interpreted?" Tags="|deep-learning|keras|speech-to-text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29716" PostTypeId="1" AcceptedAnswerId="29717" CreationDate="2018-03-30T06:39:00.600" Score="1" ViewCount="152" Body="&lt;p&gt;I was using the r.quantile method in sagemath to find boundaries for a box plot.&#xA;The plot was taking a long time using r.quantile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;r.quantile took more than 20 seconds to find the quartiles for a data set that could be sorted and plotted point by point in less than half a second on the same machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is a faster alternative? &lt;/p&gt;&#xA;" OwnerUserId="49696" LastActivityDate="2022-09-14T15:01:40.220" Title="sagemath: compared to r.quantile, what is a faster way to find boundaries for a boxplot?" Tags="|python|r|statistics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29717" PostTypeId="2" ParentId="29716" CreationDate="2018-03-30T06:43:52.703" Score="0" Body="&lt;p&gt;The following (crude) code is at least 50 times faster than r.quantile:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def findFences (orderedList, outlierC = 1.5, farOutlierC = 3.0):&#xA;   &quot;&quot;&quot;findFences: an ordered list of ints or floats -&amp;gt; tuple of 9 floats&#xA;   (aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax)&#xA;   keys: float for outlier constant [outlierC] and far outlier constant [farOutlierC]&quot;&quot;&quot;&#xA;&#xA;   lenMod4, half, quarter = len(orderedList) % 4, int(len(orderedList)/2), int(len(orderedList)/4)&#xA;&#xA;   aMin, aMax = orderedList[0], orderedList[-1]&#xA;&#xA;   # find quartiles&#xA;   if not lenMod4: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, (orderedList[half] + orderedList[half-1])/2.0, (orderedList[half+quarter] + orderedList[half+quarter-1])/2.0&#xA;   elif lenMod4 == 1: Q1, Q2, Q3 = float(orderedList[half-quarter]), float(orderedList[half]), float(orderedList[half+quarter])&#xA;   elif lenMod4 == 2: Q1, Q2, Q3 = float(orderedList[half-quarter-1]), (orderedList[half] + orderedList[half-1])/2.0, float(orderedList[half+quarter])&#xA;   else: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, float(orderedList[half]), (orderedList[half+quarter] + orderedList[half+quarter+1])/2.0&#xA;&#xA;   IQR = Q3 - Q1&#xA;   outDist = IQR * outlierC &#xA;   farOutDist = IQR * farOutlierC&#xA;   innerLoFence, innerHiFence = Q1 - outDist, Q3 + outDist&#xA;   outerLoFence, outerHiFence = Q1 - farOutDist, Q3 + farOutDist&#xA;&#xA;   return aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49696" LastEditorUserId="49696" LastEditDate="2018-03-31T16:43:00.953" LastActivityDate="2018-03-31T16:43:00.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29718" PostTypeId="2" ParentId="29541" CreationDate="2018-03-30T06:49:28.263" Score="3" Body="&lt;p&gt;Like suggested in one answer on &lt;a href=&quot;https://stackoverflow.com/questions/8844487/alternatives-to-the-dynamic-time-warping-dtw-method&quot;&gt;this SO question&lt;/a&gt;, you could use &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Elastic_matching&quot; rel=&quot;nofollow noreferrer&quot;&gt;elastic matching&lt;/a&gt; with &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance&lt;/a&gt; to your task. &lt;a href=&quot;http://richardminerich.com/2012/09/levenshtein-distance-and-the-triangle-inequality/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance obeys triangle inequality&lt;/a&gt; and is therefore a metric distance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an &lt;a href=&quot;https://github.com/SAGNIKMJR/Handwritten-Digit-Recognition-by-Elastic-Matching-in-Python&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation of elastic matching&lt;/a&gt; and &lt;a href=&quot;https://github.com/miohtama/python-Levenshtein&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance calculation&lt;/a&gt; in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To put them together you most probably need to build your own implementation. &lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44114" LastEditDate="2018-03-30T10:26:06.020" LastActivityDate="2018-03-30T10:26:06.020" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29719" PostTypeId="1" AcceptedAnswerId="29726" CreationDate="2018-03-30T06:53:30.373" Score="59" ViewCount="114414" Body="&lt;p&gt;I am starting to learn CNNs using Keras. I am using the theano backend.&lt;/p&gt;&#xA;&lt;p&gt;I don't understand how to set values to:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;batch_size&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;steps_per_epoch&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;validation_steps&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;What should be the value set to &lt;code&gt;batch_size&lt;/code&gt;, &lt;code&gt;steps_per_epoch&lt;/code&gt;, and &lt;code&gt;validation_steps&lt;/code&gt;, if I have 240,000 samples in the training set and 80,000 in the test set?&lt;/p&gt;&#xA;" OwnerUserId="49697" LastEditorUserId="29169" LastEditDate="2021-07-13T00:54:23.633" LastActivityDate="2021-07-13T00:54:23.633" Title="How to set batch_size, steps_per epoch, and validation steps?" Tags="|machine-learning|keras|cnn|theano|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29720" PostTypeId="1" AcceptedAnswerId="29728" CreationDate="2018-03-30T07:39:23.170" Score="0" ViewCount="161" Body="&lt;p&gt;I have data in following form: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a: 1,2,3,2,3&#xA;a: 2,4,5,6,7,8,0,9,7,6,5,6,2&#xA;a: 7,8,9,3,4&#xA;b: 4,5,3,5,6,3,5,1,2&#xA;b: 1,6,3,2,4,5&#xA;b: 2,4,5,6,7,8,0,9,7,6,5,6,2&#xA;c: 7,8,9,3,4&#xA;c: 4,5,3,5,6,3,5,1,2&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(in reality, each case has about 100-200 numbers, though the length is variable)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, a, b and c are groups (their number is fixed - taken as 3 here) and the numbers indicate a vector associated with each case. How can I apply supervised machine learning with such a data so that if I get a new series of numbers, e.g. following: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;3,2,3,4,1,5,6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I should be able to determine which group (a, b or c) does this case belongs to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following features of each list of numbers may be important: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;length of series&#xA;mean value of series&#xA;variance of series&#xA;maximum of series&#xA;minimum of series&#xA;type of distribution of series (normal or non-normal)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I apply machine learning methods to such data. Thanks for your insight.&lt;/p&gt;&#xA;" OwnerUserId="49700" LastActivityDate="2018-03-30T12:01:50.093" Title="Supervised learning for variable length feature-less data" Tags="|machine-learning|prediction|supervised-learning|distribution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29721" PostTypeId="2" ParentId="28847" CreationDate="2018-03-30T08:32:08.157" Score="3" Body="&lt;p&gt;From your example I understand that $y_{pred}$ are the predictions of the random forest on the training set. If this is the case, $y_{pred}$ would give you training performance whereas the oob values will give you validation performance. Because you are overfitting your training predictions are close to the true values whereas your validation/oob predictions are not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When bagging, on average each bag contains 2/3 of the samples leaving the remaining 1/3 in the oob. When you use the random forest to predict on the training set, for each sample there will be about 17 000 (~2/3*25000) trees which have seen that datapoint already since it's in their bag and results will not be reliable in terms of generalisation. In other words, when you predict on the training set, 2/3 of the trees will give you training predictions and 1/3 (oob) will give you validation predictions so overall it's like getting training performance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Answering your last four questions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Random forest in sklearn gets the probability of each class for each tree and then averages them across all bags to get the final probabilities. This is usually called 'soft voting', 'hard voting' being the method you describe.&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/ensemble.html#forest&quot; rel=&quot;nofollow noreferrer&quot;&gt;See 1.11.2.1. Random Forests&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I haven't seen it stated clearly in the documentation but I would expect that the same method that is used to make general predictions ('soft voting') will be used to get the oob values. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Yes, cross validation and oob scores should be rather similar since both use data that the classifier hasn't seen yet to make predictions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Most sklearn classifiers have a hyperparameter called class_weight which you can use when you have imbalanced data but by default in random forest each sample gets equal weight.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="49699" LastEditorUserId="29575" LastEditDate="2018-03-30T17:37:39.980" LastActivityDate="2018-03-30T17:37:39.980" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29723" PostTypeId="2" ParentId="29693" CreationDate="2018-03-30T09:15:45.777" Score="0" Body="&lt;p&gt;Since you do not have a similarity measure for trips, two trips are different if the sets that represent the trips are different. So you can simply represent different trips by different numbers and represent a person's trips by a set of those numbers.&lt;/p&gt;&#xA;" OwnerUserId="16213" LastActivityDate="2018-03-30T09:15:45.777" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29724" PostTypeId="2" ParentId="29542" CreationDate="2018-03-30T10:19:56.370" Score="4" Body="&lt;p&gt;Stanfords CS231n has a great python+numpy tutorial.&#xA;&lt;a href=&quot;http://cs231n.github.io/python-numpy-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://cs231n.github.io/python-numpy-tutorial/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for Neural Networks and Keras + Tensorflow, I can recommend the Deep Learning specialization on Coursera. It is free for one week. If you do not want the certificates you can download all videos and stop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want the certficates you can enroll in all courses simultaneously and finish them faster than advertised. They assume 3-6h per week for 16 weeks. If you have 6h per day, it is possible to go through a weeks content per day. Andrew Ng speaks clearly and the videos are understandable at 1.5x-2x speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the roadmap, i would suggest to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;start with python + numpy until you have a basic understanding of array slicing and read up on unknown commands when you encounter them.&lt;/li&gt;&#xA;&lt;li&gt;start a DL course of your choice but do not skip classes. That is I think it is important to know the basics even if you will only use high level frameworks later on.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="49704" LastEditorUserId="49704" LastEditDate="2018-03-30T10:30:43.297" LastActivityDate="2018-03-30T10:30:43.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29725" PostTypeId="1" CreationDate="2018-03-30T10:26:36.097" Score="2" ViewCount="188" Body="&lt;p&gt;I am having a list of 10 different items a user has bought in the past. Each item has been bought multiple times. I would like to find a pattern in which the user buys a particular item and predict what he will be purchasing next.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I am running a cloth factory and sell clothes to different retailers. I would like to identify what clothes a particular retailer might be asking me next, based on the history of how the retailer has purchased different clothes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We cannot use time series as the date of purchases has no equal intervals. Also I feel simply passing the sequence of purchased products ordered by date to a neural network will not be of much help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard algorithm to identity a pattern?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am having two columns in my data: 1. item_id  2. date_of_purchase&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to create a model which will predict top 3 items which the user will be buying again next. Input to my model will be present date. Output should be top 3 items which the user might be interested in buying.&lt;/p&gt;&#xA;" OwnerUserId="49705" LastEditorUserId="49705" LastEditDate="2018-03-30T11:10:19.920" LastActivityDate="2018-04-29T17:51:46.557" Title="Is there any standard pattern recognition algorithm in predicting an item which a user will be buying next, given I have the history of the purchases" Tags="|machine-learning|predictive-modeling|recommender-system|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29726" PostTypeId="2" ParentId="29719" CreationDate="2018-03-30T11:07:58.800" Score="57" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;batch_size&lt;/strong&gt; determines the number of samples in each mini batch. Its maximum is the number of all samples, which makes gradient descent accurate, the loss will decrease towards the minimum if the learning rate is small enough, but iterations are slower. Its minimum is 1, resulting in stochastic gradient descent: Fast but the direction of the gradient step is based only on one example, the loss may jump around. batch_size allows to adjust between the two extremes: accurate gradient direction and fast iteration. Also, the maximum value for batch_size may be limited if your model + data set does not fit into the available (GPU) memory.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;steps_per_epoch&lt;/strong&gt; the number of batch iterations before a training epoch is considered finished. If you have a training set of fixed size you can ignore it but it may be useful if you have a &lt;em&gt;huge&lt;/em&gt; data set or if you are generating random data augmentations on the fly, i.e. if your training set has a (generated) infinite size. If you have the time to go through your whole training data set I recommend to skip this parameter.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;validation_steps&lt;/strong&gt; similar to steps_per_epoch but on the validation data set instead on the training data. If you have the time to go through your whole validation data set I recommend to skip this parameter.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="49704" LastEditorUserId="49704" LastEditDate="2018-03-30T11:16:43.653" LastActivityDate="2018-03-30T11:16:43.653" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29727" PostTypeId="2" ParentId="29542" CreationDate="2018-03-30T11:28:06.290" Score="2" Body="&lt;p&gt;Okay, first I would tell you that deep learning is very easy as compared conventional machine learning so you don't have to panic with your 2-month deadline provided you have a thorough understanding of machine learning algorithms that you have mentioned above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now &lt;a href=&quot;https://www.coursera.org/specializations/deep-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is a course by Andrew NG that is free if you would audit it. Once you open this course just go through it you don't' even have to do the assignments just understand the concept which is very easy to grasp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After you have gone through the course. Setup python environment &lt;a href=&quot;http://inmachineswetrust.com/posts/deep-learning-setup/&quot; rel=&quot;nofollow noreferrer&quot;&gt;for windows use this&lt;/a&gt;. &lt;a href=&quot;https://www.pyimagesearch.com/2016/11/14/installing-keras-with-tensorflow-backend/&quot; rel=&quot;nofollow noreferrer&quot;&gt;For linux use this&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that search for keras tutorials from &lt;a href=&quot;https://github.com/keras-team/keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;kerasGitHubb&lt;/a&gt; page which can get you going.&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-03-30T11:28:06.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29728" PostTypeId="2" ParentId="29720" CreationDate="2018-03-30T11:38:09.350" Score="2" Body="&lt;p&gt;This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use a recurrent many-to-one RNN/GRU/LSTM network: After feeding each word it generates a feature vector which can be fed into a classifier.&#xA;&lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Recurrent_neural_network&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49704" LastEditorUserId="49704" LastEditDate="2018-03-30T12:01:50.093" LastActivityDate="2018-03-30T12:01:50.093" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29730" PostTypeId="2" ParentId="29542" CreationDate="2018-03-30T11:47:25.357" Score="2" Body="&lt;p&gt;If you are looking for a nice solid tutorial for Deep Learning in the domain of NLP (Natural Language Processing), then Robert Gutherie's &lt;a href=&quot;https://github.com/rguthrie3/DeepLearningForNLPInPytorch/blob/master/Deep%20Learning%20for%20Natural%20Language%20Processing%20with%20Pytorch.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;notebook&lt;/a&gt; would be a great start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It teaches you how to use vectors and how does multi-dimensional vectors look like and how to deal with them. He uses the deep-learning library of Facebook, Pytorch to explain the concepts. This library is more pythonic and less verbose than tf, so it is easier to follow. (personal opinion though).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, you can take up any library like gensim, spacy or Fasttext and put your vectors to work there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A nice follow-up, also would be to go through &lt;a href=&quot;http://pytorch.org/tutorials/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pytorch's Deep Learning tutorials.&lt;/a&gt; They are very nicely written and often very intuitive and the code is very easy to read (thanks to Python and pythocicity of Pytorch :D).&lt;/p&gt;&#xA;" OwnerUserId="11097" LastEditorUserId="11097" LastEditDate="2018-03-30T11:52:45.737" LastActivityDate="2018-03-30T11:52:45.737" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29732" PostTypeId="2" ParentId="29725" CreationDate="2018-03-30T12:30:13.937" Score="0" Body="&lt;p&gt;You could train a recurrent many-to-many network using GRU/LSTM cells.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;input&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;an item as a one-hot encoding&lt;/li&gt;&#xA;&lt;li&gt;a date encoded such that there is no discontinuity when switching from day 365 to 1., e.g. the days can be encoded as $(x,y)$-positions on a 1-ring. The year can be added as a third number.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;output&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;a probability vector for the next item bought. Select the top 3 for your question.&lt;/li&gt;&#xA;&lt;li&gt;(optional) a date in the same encoding as above. It may be useful to predict when the user comes back.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;loss&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When predicting the date, the loss function must be adapted to reflect the combined input, e.g. $||year_{diff}||_2$ for the year part and e.g. distance on the 1-ring or $||(x,y)_{diff}||_2$ for the $(x,y)$ part.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;missing&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This network is missing an important part for a real world scenario: Adding new items changes the input size and requires retraining. This can be solved by using embeddings instead of one-hot encoding of the items.&lt;/p&gt;&#xA;" OwnerUserId="49704" LastEditorUserId="49704" LastEditDate="2018-03-30T12:48:10.430" LastActivityDate="2018-03-30T12:48:10.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29733" PostTypeId="1" CreationDate="2018-03-30T13:50:39.340" Score="0" ViewCount="101" Body="&lt;p&gt;I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfIDF&lt;/a&gt; model. When a query is input, we can infer its TfIDF vector. Finally, we compare the query's TfIDF vector to all the vectors of the documents in the corpus using cosine similarity, which finds us the &quot;most similar&quot; texts.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question is, is there a way to incorporate more micro structure features into the pipeline such that similar document retrieval will perform better, and if so how?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By this I mean, can we use Part of Speech tagging, intent recognition or other methods, as extra features in the pipeline. I am looking for more unsupervised methods here as I have a lot of data (1TB) and it is all unstructured and un-tagged / un-labelled, but supervised suggestions will also be welcome. I appreciate that by incorporating new features, we will most likely have to move to a completely different model paradigm as TfIDF and cosine similarities will not work depending on the structure of the new features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NOTE: I have already performed a significant amount of text pre-processing on the corpus e.g. stemming, tokenizing, word replacement, stop word removal etc.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastEditorUserId="41929" LastEditDate="2018-03-30T14:18:51.557" LastActivityDate="2018-03-30T14:18:51.557" Title="Incorporating new features in document similarity task" Tags="|nlp|feature-extraction|similar-documents|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29734" PostTypeId="2" ParentId="29693" CreationDate="2018-03-30T14:08:53.107" Score="1" Body="&lt;p&gt;You are describing &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot&lt;/a&gt; encoding. There is a slot for each element. If the element is present, the slot has a one, and if the element if not present, the slot has a zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, people will encode orthogonal features in different dimensions. In your case, cities would be one dimension and transportation type would be another dimension. A given data point would be one-hot encoded in a matrix (a 2D collection of vectors). If you want to add people, you would another dimension. That would create a 3D tensor with each person in a row.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another way to compress your data is to encode not the cities (nodes) but the path between the cities (edges). From that encoding, you can create a &lt;a href=&quot;https://en.wikipedia.org/wiki/Laplacian_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;Laplacian matrix&lt;/a&gt; that sets up &lt;a href=&quot;https://en.wikipedia.org/wiki/Spectral_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;spectral clustering&lt;/a&gt;. Since you have multiple transportation methods for the routes, you can create clusters with multi-dimensional spectral clustering.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-04-12T15:10:06.767" LastActivityDate="2018-04-12T15:10:06.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29735" PostTypeId="1" AcceptedAnswerId="29792" CreationDate="2018-03-26T23:54:41.043" Score="8" ViewCount="23337" Body="&lt;p&gt;I recently did a homework where I had to learn a model for the MNIST 10-digit classification. The HW had some scaffolding code and I was supposed to work in the context of this code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My homework works / passes tests but now I'm trying to do it all from scratch (my own nn framework, no hw scaffolding code) and I'm stuck applying the grandient of softmax in the backprop step, and even think what the hw scaffolding code does might not be correct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hw has me use what they call 'a softmax loss' as the last node in the nn. Which means, for some reason they decided to join a softmax activation with the cross entropy loss all in one, instead of treating softmax as an activation function and cross entropy as a separate loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The hw loss func then looks like this (minimally edited by me):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class SoftmaxLoss:&#xA;    &quot;&quot;&quot;&#xA;    A batched softmax loss, used for classification problems.&#xA;    input[0] (the prediction) = np.array of dims batch_size x 10&#xA;    input[1] (the truth) = np.array of dims batch_size x 10&#xA;    &quot;&quot;&quot;&#xA;    @staticmethod&#xA;    def softmax(input):&#xA;        exp = np.exp(input - np.max(input, axis=1, keepdims=True))&#xA;        return exp / np.sum(exp, axis=1, keepdims=True)&#xA;&#xA;    @staticmethod&#xA;    def forward(inputs):&#xA;        softmax = SoftmaxLoss.softmax(inputs[0])&#xA;        labels = inputs[1]&#xA;        return np.mean(-np.sum(labels * np.log(softmax), axis=1))&#xA;&#xA;    @staticmethod&#xA;    def backward(inputs, gradient):&#xA;        softmax = SoftmaxLoss.softmax(inputs[0])&#xA;        return [&#xA;            gradient * (softmax - inputs[1]) / inputs[0].shape[0],&#xA;            gradient * (-np.log(softmax)) / inputs[0].shape[0]&#xA;        ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, on forward it does softmax(x) and then cross entropy loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But on backprop, it seems to only do the derivative of cross entropy and not of softmax. Softmax is left as such. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Shouldn't it also take the derivative of softmax with respect to the input to softmax?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that it should take the derivative of softmax, I'm not sure how this hw actually passes the tests...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, in my own implementation from scratch, I made softmax and cross entropy separate nodes, like so (p and t stand for predicted and truth):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class SoftMax(NetNode):&#xA;    def __init__(self, x):&#xA;        ex = np.exp(x.data - np.max(x.data, axis=1, keepdims=True))&#xA;        super().__init__(ex / np.sum(ex, axis=1, keepdims=True), x)&#xA;&#xA;    def _back(self, x):&#xA;        g = self.data * (np.eye(self.data.shape[0]) - self.data)&#xA;        x.g += self.g * g&#xA;        super()._back()&#xA;&#xA;class LCE(NetNode):&#xA;    def __init__(self, p, t):&#xA;        super().__init__(&#xA;            np.mean(-np.sum(t.data * np.log(p.data), axis=1)),&#xA;            p, t&#xA;        )&#xA;&#xA;    def _back(self, p, t):&#xA;        p.g += self.g * (p.data - t.data) / t.data.shape[0]&#xA;        t.g += self.g * -np.log(p.data) / t.data.shape[0]&#xA;        super()._back()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see, my cross entropy loss (LCE) has the same derivative as the one in the hw, because that is the derivative for the loss itself, without getting into the softmax yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, I would still have to do the derivative of softmax to chain it with the derivative of loss. This is where I get stuck. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For softmax defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?%24%24smax%28x_%7Bi%7D%29%3D%5Cfrac%7Be%5E%7Bx_%7Bi%7D%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%20e%5E%7Bx_%7Bj%7D%7D%7D%24%24&quot; alt=&quot;a&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The derivative is usually defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?%24%24%5Cfrac%7B%5Cpartial%20smax%28x_%7Bi%7D%29%7D%7B%5Cpartial%20x_%7Bj%7D%7D%3Dx_%7Bi%7D%281-x_%7Bj%7D%29%20%5Ctextit%7B%20if%20i%3Dj%20else%20%7D%20-x_%7Bi%7Dx_%7Bj%7D%24%24&quot; alt=&quot;b&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I need a derivative that results in a tensor of the same size as the input to softmax, in this case, batch_size x 10. So I'm not sure how the above should be applied to only 10 components, since it implies that I would diferentiate for all inputs with respect to all outputs (all combinations) or &lt;img src=&quot;https://latex.codecogs.com/gif.latex?%24%24D_%7Bj%7DS_%7Bi%7D%3DS_%7Bi%7D%28I-S_%7Bj%7D%29%20%24%24&quot; alt=&quot;c&quot;&gt; in matrix form.&lt;/p&gt;&#xA;" OwnerUserId="49780" OwnerDisplayName="SaldaVonSchwartz" LastActivityDate="2018-04-01T22:28:11.367" Title="How to apply the gradient of softmax in backprop" Tags="|backpropagation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29739" PostTypeId="1" CreationDate="2018-03-30T18:10:52.763" Score="2" ViewCount="662" Body="&lt;p&gt;Let's say I am training a neural net (e.g. convolutional network or LSTM).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, the longer the training (more epochs) leads to better accuracy, albeit at times at the expense of overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach is to duplicate the data. Again, if I have 1000 examples, I could duplicate them 10 times and randomise them, and the neural net will learn better (but again, it may not be able to generalise well).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any difference between duplicating data and adding epochs when training a neural net in terms of its learning?&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-13T16:35:05.087" LastActivityDate="2021-02-13T16:35:05.087" Title="Any difference between adding epochs and duplicating data for neural nets?" Tags="|deep-learning|neural-network|lstm|training|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29740" PostTypeId="1" AcceptedAnswerId="29768" CreationDate="2018-03-30T18:48:19.957" Score="2" ViewCount="1718" Body="&lt;p&gt;I am looking for a benchmark result or any kaggle competition held using MovieLens(20M or latest) dataset.&#xA;&lt;br&gt;Similar question has been asked &lt;a href=&quot;https://datascience.stackexchange.com/questions/6835/bechmark-for-movielens&quot;&gt;here&lt;/a&gt; but, provided links are dead so re-raising the question.&lt;br&gt;&lt;/p&gt;&#xA;" OwnerUserId="46800" LastActivityDate="2018-03-31T18:23:10.677" Title="benchmark Result for MovieLens dataset?" Tags="|machine-learning|data-mining|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29742" PostTypeId="2" ParentId="22163" CreationDate="2018-03-30T20:17:30.263" Score="6" Body="&lt;p&gt;If your weights are diverging then your optimizer or your gradients aren't behaving well. A common reason for diverging weights is &lt;a href=&quot;https://en.wikipedia.org/wiki/Vanishing_gradient_problem&quot; rel=&quot;noreferrer&quot;&gt;exploding gradients&lt;/a&gt;, which can result from:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;too many layers, or&lt;/li&gt;&#xA;&lt;li&gt;too many recurrent cycles if you're using an RNN. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can verify if you have exploding gradients as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;grad_magnitude = tf.reduce_sum([tf.reduce_sum(g**2)&#xA;                                for g in tf.gradients(loss, weights_list)])**0.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Some approaches to solving the problem of exploding gradients are:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)&quot; rel=&quot;noreferrer&quot;&gt;RELU&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#ELUs&quot; rel=&quot;noreferrer&quot;&gt;ELU&lt;/a&gt; activations&lt;/li&gt;&#xA;&lt;li&gt;Use &lt;a href=&quot;http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization&quot; rel=&quot;noreferrer&quot;&gt;Xavier initialization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Use a &lt;a href=&quot;https://www.quora.com/How-does-deep-residual-learning-work&quot; rel=&quot;noreferrer&quot;&gt;Deep Residual architecture&lt;/a&gt;. This will keep the gradients from being squished by subsequent layers.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="39308" LastEditorUserId="29575" LastEditDate="2018-03-30T20:40:24.657" LastActivityDate="2018-03-30T20:40:24.657" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29743" PostTypeId="2" ParentId="19519" CreationDate="2018-03-30T20:54:29.047" Score="3" Body="&lt;p&gt;If I were you I would use deep learning. You can use an &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;autoencoder&lt;/a&gt; format for this. Essentially you would feed the image in, then each layer yields a smaller output. Then you would feed the output in reverse, using transposed weight matrices, which would yield progressively larger outputs. The final output from the backward pass would be the filled in image. Since your image has such a small color range I would recommend representing pixel colors using &lt;a href=&quot;https://en.wikipedia.org/wiki/One-hot&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-hot&lt;/a&gt; encoding.&lt;/p&gt;&#xA;" OwnerUserId="39308" LastActivityDate="2018-03-30T20:54:29.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29744" PostTypeId="2" ParentId="25074" CreationDate="2018-03-30T21:35:24.177" Score="2" Body="&lt;p&gt;Yes you &quot;can&quot; use least squares to do that. However, you would probably need some parametric assumptions to do it, which in turn might require some knowledge about the correlation matrix...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To go back to your original problem, I would actually argue against your reference about pairwise deletion. The weird behavior he identifies mostly happens because the estimator is computed with a dataset that is too small. The empirical estimator of Pearson's correlation is NOT guaranteed to be positive-definite. Also, there is some sort of pattern in its missing data: entries for the $X_3$ are missing only when $X_1 \neq X_2$. Let us instead assume that entries are missing purely randomly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The estimator you compute using $\hat\Sigma$ := &lt;code&gt;cor(X, use=&quot;pairwise.complete.obs&quot;)&lt;/code&gt;, for a data matrix $X$ still makes a lot of sense. The biggest difference with the non-missing data case is that you now get some entries of $\hat\Sigma$ that are computed on less data; but they still estimate the quantity you wish to estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is less clear is how to use $\hat\Sigma$ in the desired application: the fact that some entries were computed on less data means that some entries might be less reliable than others. However, notice that even in the classical case, some entries are less reliable than others anyway (this depends on the underlying distribution).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand the impact of using all available pairwise data to compute $\hat\Sigma$ in the presence of missing data, consider the two following situations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Situation 1: You have a $n \times d$ data matrix $X$ and compute the empirical correlation with $\hat\Sigma^{(1)}$ := &lt;code&gt;cor(X)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Situation 2: You have a $[nd(d-1)/2] \times d$ data matrix $X$ in which you have $n$ pairwise observations for all $d(d-1)/2$ pairs of variables and &lt;code&gt;NA&lt;/code&gt; everywhere else. in this case you compute the empirical correlation with $\hat\Sigma^{(2)}$ := &lt;code&gt;cor(X, use=&quot;pairwise.complete.obs&quot;)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In both situations, you have the same number of non &lt;code&gt;NA&lt;/code&gt; pairs (although more non &lt;code&gt;NA&lt;/code&gt; entries in the second case). However, you have, for $i_1,j_1 \neq i_2,j_2$&#xA;\begin{align*}&#xA;\mathrm{cor}(\hat\Sigma^{(1)}_{i_1,j_1},\hat\Sigma^{(1)}_{i_2,j_2}) \neq 0&#xA;\end{align*}&#xA;assuming the your variables are indeed related, and&#xA;\begin{align*}&#xA;\mathrm{cor}(\hat\Sigma^{(2)}_{i_1,j_1},\hat\Sigma^{(2)}_{i_2,j_2}) = 0.&#xA;\end{align*}&#xA;Somehow, I tend to think the zero correlation between the estimates might be a good thing more that a bad one...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this to say, pairwise deletion is not necessarily a bad way to proceed.&lt;/p&gt;&#xA;" OwnerUserId="47308" LastActivityDate="2018-03-30T21:35:24.177" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29745" PostTypeId="1" AcceptedAnswerId="29750" CreationDate="2018-03-30T21:50:15.910" Score="2" ViewCount="213" Body="&lt;p&gt;I am running analysis on data for this type of sensor my company makes. I want to quantify the health of the sensor based on three features using the following formula:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sensor health index = feature1 * A + feature2 * B + feature3 *C&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also need to pick a threshold so that if this index exceeds the threshold, the sensor is considered as bad sensor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We only have a legacy list which shows about 100 sensors are bad. But now we have data for more than 10,000 sensors. Anything not in that 100 sensor list is NOT necessarily bad. So I guess the linear regression methods don't work in this scenario.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only way I can think of is the brute force fitting. Pseudo code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# class definition for params(coefficients)&#xA;class params{&#xA;  a&#xA;  b&#xA;  c&#xA;  th&#xA;}&#xA;&#xA;&#xA;# dictionary of parameter and accuracy rate&#xA;map = {}&#xA;&#xA;for thold in range (1..20):&#xA;   for a in range (1..10):&#xA;      for b in range (1..10):&#xA;        for b in range (1..10):&#xA;           # bad sensor list&#xA;           bad_list = []&#xA;           params = new params[a, b, c, thold]&#xA;           for each sensor:&#xA;             health_index = sensor.feature1*a+sensor.feature2*b+sensor.feature3*c&#xA;             if health_index &amp;gt; thold:&#xA;               bad_list.append(sensor.id)&#xA;           accuracy = percentage of common sensors between bad_list and known_bad_sensors&#xA;           map[params] = accuracy&#xA;&#xA;# rank params based on accuracy&#xA;rank(map)&#xA;# the params with most accuracy is the best model&#xA;print map.index(0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I really don't like this method since it is using 5 for loops which is very efficient. I wonder if there is a better way to do it. Using existing library such as sk-learn perhaps?&lt;/p&gt;&#xA;" OwnerUserId="49731" LastActivityDate="2018-03-31T04:24:15.817" Title="Should I fit my parameters with brute force" Tags="|python|classification|scikit-learn|parameter-estimation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29746" PostTypeId="1" AcceptedAnswerId="29747" CreationDate="2018-03-30T22:51:20.473" Score="6" ViewCount="5898" Body="&lt;p&gt;I have around 4000 customer records and 6000 user records and about 3000 customer records match leaving 1000 unmatched customers. I have created a fuzzy matching algorithm using Levenshtein and Hamming and added weights to certain properties, but I want to be able to match the remaining records without manually doing this. Ideally I want to implement an algorithm to take a customer and user and output match/no match. However, wouldn't I need to train with true negatives? Is there an algorithm that can train with just 1 label?&#xA;Thanks&lt;/p&gt;&#xA;" OwnerUserId="49733" LastActivityDate="2018-03-30T23:21:17.843" Title="Data Matching Using Machine Learning" Tags="|machine-learning|fuzzy-logic|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29747" PostTypeId="2" ParentId="29746" CreationDate="2018-03-30T23:00:06.027" Score="4" Body="&lt;p&gt;You can obtain one negative example by taking one of the 3000 customer records and pairing it with any user record that is known not to match.  In this way, you can obtain $3000$ positives and $3000 \times 5999$ negatives.  You could then train a boolean classifier on this entire training set.  This might work better than using one-class classification on just the positives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even better might be to use techniques for &lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank&quot; rel=&quot;nofollow noreferrer&quot;&gt;learning to rank&lt;/a&gt;.  If $c$ is a customer record that is known to match a user record $u$, and $u'$ is any other user record (which $c$ doesn't match), then you want your classifier to rank the pair $(c,u)$ higher than $(c,u')$.  In this way you can obtain $3000 \times 5999$ such ranking-pairs, and try to train a classifier to learn to rank, then use that to find the best match for each of the 1000 customer records.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2018-03-30T23:21:17.843" LastActivityDate="2018-03-30T23:21:17.843" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29748" PostTypeId="2" ParentId="19519" CreationDate="2018-03-30T23:05:21.383" Score="1" Body="&lt;p&gt;It looks like there is special structure in your images that might allow you to do better than using generic image inpainting methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's hard to know from just one example image, but it is possible that you might be able to build a satisfactory method using only very simple features of the input image.  In particular, you could train a classifier that, given a black pixel, tries to predict the correct color for that pixel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your specific example, some features that look useful are: &quot;the most common color in the same column&quot;, &quot;the most common color in the same row&quot;, &quot;the color of the closest non-black pixel above your pixel (in the same column)&quot;, &quot;the color of the closest non-black pixel below your pixel (in the same column)&quot;, &quot;the color of the closest non-black pixel to the right of your pixel (in the same row)&quot;, &quot;the color of the closest non-black pixel to the left of your pixel (in the same row)&quot;.  I suggest trying to train a simple classifer on a few simple features like this, and see how well it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get fancy, you could try modelling this as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_random_field&quot; rel=&quot;nofollow noreferrer&quot;&gt;Markov random field&lt;/a&gt; over a grid graph.  In other words, each pixel is a vertex in the graph, and two pixels are connected by an edge if they are adjacent.  There are then standard methods to learn the underlying density functions given some training examples, and once you know the density functions, you could then use maximum likelihood methods to infer the values of the black pixels.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-30T23:05:21.383" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29749" PostTypeId="2" ParentId="25050" CreationDate="2018-03-30T23:49:46.400" Score="0" Body="&lt;p&gt;Yes.  You can use a CapsuleNet as the discriminator in a GAN.  See the following recent paper, which claims this improves over using CNNs as the discriminator:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1802.06167&quot; rel=&quot;nofollow noreferrer&quot;&gt;CapsuleGAN: Generative Adversarial Capsule Network&lt;/a&gt;.&#xA;Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, Premkumar Natarajan.&#xA;arXiv:1802.06167&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The improvement appears to be modest on the tasks they tested.  It's not clear how to use a CapsuleNet as the generator, so the paper uses a CNN-based generator network.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-30T23:49:46.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29750" PostTypeId="2" ParentId="29745" CreationDate="2018-03-31T00:45:24.470" Score="1" Body="&lt;p&gt;If you don't know which of the 10,000 sensors are good and which are bad, the data from those 10,000 sensors is useless for training a regression line / classifier.  You need &lt;em&gt;labelled&lt;/em&gt; data, where you know both the value of the features &lt;em&gt;and&lt;/em&gt; the health of the sensor.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, to be effective, you probably need your training set to contain both healthy sensors and bad sensors (where you know which ones are healthy and which ones are bad); it's not enough to just have data from bad sensors, because that doesn't tell you what healthy sensors look like.  In a pinch you could use &lt;a href=&quot;https://en.wikipedia.org/wiki/One-class_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;one-class classification&lt;/a&gt;, but I don't recommend it -- your results will probably be poor, and you'll probably be better off obtaining labelled data from both healthy sensors and bad sensors.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-03-31T00:45:24.470" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29752" PostTypeId="2" ParentId="29745" CreationDate="2018-03-31T04:24:15.817" Score="1" Body="&lt;p&gt;Given that only have labels for a small subset of data, you should use unsupervised methods. You can cluster all the sensors data. Then see if there is a pattern to where the 100 bad sensors are and can generalize to the other sensors. If a majority of the 100 bad sensors are in the same region, you can label that cluster as bad and make a threshold based on that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also the field of &lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;semi-supervised learning&lt;/a&gt; that might be appropriate. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-31T04:24:15.817" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29754" PostTypeId="2" ParentId="17704" CreationDate="2018-03-31T06:08:31.797" Score="0" Body="&lt;p&gt;You have two options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;chaining 12 classifiers together. First to split between $Sport$ and $Politics$ then further classify each of them.&#xA;I find this redundant and their might not be enough training data for each model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;using sub-class labels to train data, as you mentioned. &#xA;This &lt;a href=&quot;http://www.cse.unsw.edu.au/~achim/PDFs/ECML01.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; shows decision tress(C4.5) perform better when we have more classes to distinguish.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43770" LastEditorUserId="29575" LastEditDate="2018-03-31T09:03:33.640" LastActivityDate="2018-03-31T09:03:33.640" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29757" PostTypeId="2" ParentId="17286" CreationDate="2018-03-31T06:49:22.237" Score="1" Body="&lt;p&gt;While training a convNet, total memory required include following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Memory for parameters&lt;/li&gt;&#xA;&lt;li&gt;Memory for the output of intermediate layers&lt;/li&gt;&#xA;&lt;li&gt;Memory for the gradient of each parameter&lt;/li&gt;&#xA;&lt;li&gt;Extra memory needed if you are using optimizer like Momentum, RMSprop, Adams etc&lt;/li&gt;&#xA;&lt;li&gt;Miscellaneous memory for implementation&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A good rough approximation is number of parameters x 3 x 4(if you are using 32-bit float) bytes&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, now this is how you calculate the number of parameters:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Conv layer: (kernel width x kernel height) x number of channels x depth + depth (add depth only if bias is there)&lt;/li&gt;&#xA;&lt;li&gt;FC layer: numb of input*numb of output + output (output is added to include the number of bias)&lt;/li&gt;&#xA;&lt;li&gt;Max pool layer: no parameter&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now just sum the number of all the parameters and use the formula I mentioned.&lt;/p&gt;&#xA;" OwnerUserId="49739" LastEditorUserId="49739" LastEditDate="2018-03-31T06:54:31.680" LastActivityDate="2018-03-31T06:54:31.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29760" PostTypeId="1" CreationDate="2018-03-31T12:32:27.280" Score="1" ViewCount="978" Body="&lt;p&gt;I am trying to implement VGG-16 architecture in TensorFlow.&lt;/p&gt;&#xA;&lt;p&gt;As mentioned in the paper, they changed the learning rate 3 time during their 74 epochs of training.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;def optimizer(lr):&#xA;    return tf.train.MomentumOptimizer(learning_rate = lr)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;But when I tried to change the learning rate in tensorflow,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;lr=lr/10&#xA;opt_momentum = optimizer(lr)&#xA;opt = opt_momentum.minimize(cost)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I get the following error&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;FailedPreconditionError (see above for traceback): Attempting to use &#xA;uninitialized value fully_connected_2/biases/Momentum_1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I use &lt;code&gt;builder = tf.saved_model.builder.SavedModelBuilder(export_dir)&lt;/code&gt; to save the model.&lt;/p&gt;&#xA;&lt;p&gt;However. I can't understand how to load the trained weights to initialize the layers after changing the &lt;code&gt;learning_rate&lt;/code&gt;?&lt;/p&gt;&#xA;&lt;p&gt;For changing the learning, I am calling the function &lt;code&gt;optimizer&lt;/code&gt; with a new learning rate as argument, which is return me a new &lt;code&gt;op&lt;/code&gt; but the error says that the layers are uninitialized.&lt;/p&gt;&#xA;" OwnerUserId="49454" LastEditorUserId="1330" LastEditDate="2021-09-09T10:58:35.307" LastActivityDate="2021-09-09T10:58:35.307" Title="How to change learning rate of MomentumOptimizer in tensorflow" Tags="|python|neural-network|tensorflow|convolutional-neural-network|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29761" PostTypeId="1" AcceptedAnswerId="29836" CreationDate="2018-03-31T13:31:20.597" Score="4" ViewCount="3188" Body="&lt;p&gt;In the context of time series data mining, I have read about time series segmentation and time series clustering, but I couldn't differentiate between both. In case they are different, how these methods are correlated with each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well from my understanding (please correct me if I am wrong), the segmentation is considered as a preprocessing step for the clustering phase. I mean that the segmentation step is used mainly to partition your time series data into segments, let's say into states. After that, a conventional clustering algorithm can be applied to group these segments into clusters (similar segments belong to the same cluster).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, let's say that the segmentation process represents a given time series into the following segments: (S1, S2, S3, S4, S5, S6). Then after applying the segmentation process, a conventional clustering method is applied to cluster the extracted segments. So we might end up with something like this: If k = 3: then K1 {S1, S5}, K2 {S3, S6}, K3 {S2, S4}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I am mistaken, and provide links for more clarification if you have any. &lt;/p&gt;&#xA;" OwnerUserId="49752" LastEditorUserId="29575" LastEditDate="2018-03-31T18:27:44.240" LastActivityDate="2018-04-03T06:43:25.733" Title="Difference between Time series clustering and Time series Segmentation" Tags="|data-mining|clustering|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29763" PostTypeId="2" ParentId="28449" CreationDate="2018-03-31T14:31:07.787" Score="0" Body="&lt;p&gt;Try looking at the &lt;a href=&quot;https://pbiecek.github.io/DALEX/&quot; rel=&quot;nofollow noreferrer&quot;&gt;DALEX&lt;/a&gt; package for R. It is designed to specifically investigate the importance of input variables to perhaps otherwise opaque ML algos. &lt;/p&gt;&#xA;" OwnerUserId="30048" LastActivityDate="2018-03-31T14:31:07.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29764" PostTypeId="2" ParentId="27277" CreationDate="2018-03-31T14:49:19.550" Score="0" Body="&lt;p&gt;A pixels does map to 9 anchors. But the 3 by 3 convolutional layer before 1 by 1 maybe aim to extra non-linear(the extra 3 by 3 con layer produce &lt;code&gt;same&lt;/code&gt; width and height as input).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, not 9 pixels map to 9 anchors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://www.youtube.com/watch?v=_GfPYLNQank&amp;amp;t=3126s&quot; rel=&quot;nofollow noreferrer&quot;&gt;video from CS231n&lt;/a&gt;, shows the above idea at 51min.&lt;/p&gt;&#xA;" OwnerUserId="49755" LastEditorUserId="29575" LastEditDate="2018-03-31T15:14:25.110" LastActivityDate="2018-03-31T15:14:25.110" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29768" PostTypeId="2" ParentId="29740" CreationDate="2018-03-31T18:23:10.677" Score="3" Body="&lt;p&gt;One result for MovieLens 20M using Factorization Machine can be found &lt;a href=&quot;https://docs.treasuredata.com/articles/hivemall-movielens20m-fm&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. They got MAE: 0.60 and RMSE: 0.80.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another result for MovieLens 20M using Autoencoders can be found &lt;a href=&quot;https://arxiv.org/pdf/1606.07659.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. They got RMSE: 0.81.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-03-31T18:23:10.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29769" PostTypeId="1" CreationDate="2018-03-31T20:31:57.673" Score="7" ViewCount="242" Body="&lt;p&gt;this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain.&#xA;I took a large &quot;training&quot; subset of MNIST as my guinea pig.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Converted every image in MNIST into &quot;black-and-white&quot; (pixels values only 0 or 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Summed over all data images to build a histogram over the pixels - I counted the number of times each pixel gets a 1 value in the dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Normalized histogram to get an estimate of the &quot;true&quot; probability distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) From this, I got the following probability distribution (shown as a heatmap with matplotlib):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[&lt;img src=&quot;https://i.stack.imgur.com/o3Dgf.png&quot; alt=&quot;Probability distribution for an MNIST training set[1]&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) Now I calculated the entropy and got: $191$ bits&lt;/p&gt;&#xA;&#xA;&lt;p&gt;6) According to David MacKay in his Information Theory book, we could interpret a neural network as a noisy channel and consider each neuron as having a 2 bit capacity. Although he does state to use this idea with care. Chapter 40 of his book &lt;a href=&quot;http://www.inference.org.uk/itila/book.html&quot; rel=&quot;noreferrer&quot;&gt;http://www.inference.org.uk/itila/book.html&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;7) So, as a rough estimate (and with care) we could say we would need a neural network of 95 neurons in order to be able to encode the labeling of this MNIST training set (190/2)&#xA;8) Now we can get to my question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if this is a very &quot;back-of-the-envelope&quot; calculation, shouldn't a neural network capable of learning the labelling be at least in the ballpark of 95 neurons?&#xA;Why do we need, for instance, a neural network with 21840 parameters to get 99% accuracy? (considering the one in PyTorch's example for MNIST: &lt;a href=&quot;https://github.com/pytorch/examples/blob/master/mnist/main.py&quot; rel=&quot;noreferrer&quot;&gt;https://github.com/pytorch/examples/blob/master/mnist/main.py&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="49770" LastActivityDate="2018-04-01T07:05:00.203" Title="Are deep learning models way over the required capacity for their datasets' estimated entropies?" Tags="|neural-network|deep-learning|information-theory|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29770" PostTypeId="2" ParentId="17921" CreationDate="2018-03-31T23:42:33.607" Score="2" Body="&lt;p&gt;$$ C = C_0 + {\lambda \over 2n} \sum_{i=1}^{N_w}w^2  $$&#xA;$$ C = {\lambda N_w w^2 \over 2n}$$&#xA;$$ W = \sqrt{2nC \over \lambda} {1 \over \sqrt{N_w}} $$&#xA;$$ W \propto {1 \over \sqrt{N_w}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/F5E5F.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/F5E5F.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you go for question 3.&lt;/p&gt;&#xA;" OwnerUserId="49779" LastEditorUserId="29575" LastEditDate="2018-04-01T00:12:30.587" LastActivityDate="2018-04-01T00:12:30.587" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29772" PostTypeId="1" CreationDate="2018-04-01T01:47:18.537" Score="1" ViewCount="55" Body="&lt;p&gt;I have this set of data that looks like this&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/02nG4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/02nG4.png&quot; alt=&quot;data&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I was asked o build a &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt;-nearest neighbors algorithm for it which I just finished building. I have this question in regards to the data that I do not understand: Do you notice any spatial or temporal trends in error?&lt;/p&gt;&#xA;&lt;p&gt;I am not sure how to proceed in answering that question. Any suggestions would be appreciated.&lt;/p&gt;&#xA;" OwnerUserId="49781" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2022-06-25T23:04:30.327" Title="k nearest neighbors method, temporal trend in error" Tags="|dataset|data|error-handling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29774" PostTypeId="2" ParentId="29769" CreationDate="2018-04-01T07:05:00.203" Score="5" Body="&lt;p&gt;The current thinking is that it is easier to fit an overparameterized neural network, since the local extrema are different ways of expressing the same thing, whereas in a minimal neural network you have to worry about getting to the global extremum: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The subtle reason behind this is that smaller networks are harder to&#xA;  train with local methods such as Gradient Descent: It’s clear that&#xA;  their loss functions have relatively few local minima, but it turns&#xA;  out that many of these minima are easier to converge to, and that they&#xA;  are bad (i.e. with high loss). Conversely, bigger neural networks&#xA;  contain significantly more local minima, but these minima turn out to&#xA;  be much better in terms of their actual loss. Since Neural Networks&#xA;  are non-convex, it is hard to study these properties mathematically,&#xA;  but some attempts to understand these objective functions have been&#xA;  made, e.g. in a recent paper The Loss Surfaces of Multilayer Networks.&#xA;  In practice, what you find is that if you train a small network the&#xA;  final loss can display a good amount of variance - in some cases you&#xA;  get lucky and converge to a good place but in some cases you get&#xA;  trapped in one of the bad minima. On the other hand, if you train a&#xA;  large network you’ll start to find many different solutions, but the&#xA;  variance in the final achieved loss will be much smaller. In other&#xA;  words, all solutions are about equally as good, and rely less on the&#xA;  luck of random initialization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://cs231n.github.io/neural-networks-1/&quot; rel=&quot;noreferrer&quot;&gt;CS231n Convolutional Neural Networks for Visual Recognition&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="381" LastActivityDate="2018-04-01T07:05:00.203" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29775" PostTypeId="1" CreationDate="2018-04-01T08:51:55.923" Score="2" ViewCount="4637" Body="&lt;p&gt;I have user generated text containing names of ports often containing typos and the actual port names. I would like to correct the misspelling of user generated text containing the names of ports. Can anyone help me think of some ideas or steps to take?&#xA;Perhaps using word2vec? RNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thank you in advance.&lt;/p&gt;&#xA;" OwnerUserId="49762" LastEditorUserId="13023" LastEditDate="2018-04-01T16:48:17.640" LastActivityDate="2018-04-01T16:56:01.320" Title="How to check and correct misspelling in the data of pairs of words?" Tags="|machine-learning|deep-learning|data-cleaning|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29776" PostTypeId="1" AcceptedAnswerId="29807" CreationDate="2018-04-01T08:57:51.267" Score="1" ViewCount="667" Body="&lt;p&gt;I have a CSV file with 20 columns and 785 rows. The 785th row for each column is a label describing the encoded image. The encoded image is either &lt;strong&gt;3&lt;/strong&gt; or &lt;strong&gt;5&lt;/strong&gt;. So 1-784 row is the encoded image and 785th row is the label that names the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I loaded the CSV file &lt;a href=&quot;http://www.sharecsv.com/s/1df9354a61321fa8a82d98dfa3c7c982/3_5_small.csv&quot; rel=&quot;nofollow noreferrer&quot;&gt;3_5_small.csv&lt;/a&gt; and segregated the labels from the encoded data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which as you see is the image of number &lt;strong&gt;3&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I decided to use logistic regression to predict the images from the encoded data. I used Stochastic Gradient Descent as explained in the &lt;a href=&quot;https://www.coursera.org/learn/machine-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;machine learning course by Andrew NG&lt;/a&gt;. But I do not think, I got it right. Before following the code, here are the steps I did:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Transformed &lt;code&gt;train_labels_3_5&lt;/code&gt; which contained only &lt;code&gt;3&lt;/code&gt; and &lt;code&gt;5&lt;/code&gt; to &lt;code&gt;1&lt;/code&gt; and &lt;code&gt;0&lt;/code&gt; respectively. So I want to predict the image of &lt;code&gt;3&lt;/code&gt;. If the output probability is &amp;lt; 0.5, it will be 5 and &gt; 0.5 will be 3.&lt;/li&gt;&#xA;&lt;li&gt;Randomly shuffled the &lt;code&gt;train_data_3_5&lt;/code&gt; and &lt;code&gt;train_labels_3_5&lt;/code&gt; to the same degree.&lt;/li&gt;&#xA;&lt;li&gt;Randomly generated the &lt;code&gt;theta&lt;/code&gt; vector&lt;/li&gt;&#xA;&lt;li&gt;Passed the &lt;code&gt;theta&lt;/code&gt; vector and &lt;code&gt;X&lt;/code&gt; vector into the &lt;code&gt;hypothesis&lt;/code&gt; function&lt;/li&gt;&#xA;&lt;li&gt;Updated the &lt;code&gt;theta&lt;/code&gt; vector.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is all I did. Here is the code to what I have done.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train &amp;lt;- function(data, labels, alpha = 0.001) {&#xA;  #browser()&#xA;&#xA;  #Initialize the theta vector&#xA;  theta &amp;lt;- seq(from = 0, to = 1, length.out = nrow(data))&#xA;&#xA;  number_of_iterations = 10&#xA;  for(noi in 1:number_of_iterations) {&#xA;    for(i in seq(1:ncol(data))) {&#xA;      x = as.vector(data[,i]) #Create a x vector&#xA;      h = hypothesis(x, theta) #Call the hypothesis function to get the probability&#xA;      y = labels[1,i]&#xA;      theta &amp;lt;- theta - (alpha * ((h - y) * x))&#xA;    }&#xA;  }&#xA;  return(theta)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But on the test data and even on the training data, this does not predict correct at all. I do not know where have I gone wrong. I have revisited the algorithm, the lecture but cannot figure out, what am I doing incorrectly. It always predicts &lt;code&gt;1&lt;/code&gt; no matter I pass the vector for &lt;strong&gt;3&lt;/strong&gt; or &lt;strong&gt;5&lt;/strong&gt;!&lt;/p&gt;&#xA;" OwnerUserId="49787" LastEditorUserId="49787" LastEditDate="2018-04-03T17:09:42.440" LastActivityDate="2018-04-03T17:09:42.440" Title="Cannot get the prediction right using Stochastic Gradient Descent: Always predicts 1" Tags="|machine-learning|r|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29779" PostTypeId="1" CreationDate="2018-04-01T12:37:15.127" Score="1" ViewCount="1208" Body="&lt;p&gt;I am plotting a group of time series plots with CI's in seaborn using tsplot:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XaaiE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XaaiE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These were produced from 10 traces each:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XWE27.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XWE27.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The confidence bars shown (99% confidence) appear to be the CI's for the mean. However I am really after 99% confidence for the prediction interval. For the examples above these should be much wider and cross the red dotted line on occasion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can I plot 99% prediction intervals in seaborn tsplot?&lt;/p&gt;&#xA;" OwnerUserId="37172" LastActivityDate="2018-04-01T12:37:15.127" Title="How to plot prediction intervals in seaborn tsplot?" Tags="|python|time-series|plotting|seaborn|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29783" PostTypeId="1" CreationDate="2018-04-01T15:32:34.037" Score="1" ViewCount="2208" Body="&lt;p&gt;Currently I'm working in WEKA, using the SMO classifier (an implementation of SVM).&#xA;For an assignment I am requested to use a polynomial kernel, and report the results for degrees varying from 1 up to 50. I need to comment on overfitting, and my understanding is that higher degrees lead up to a higher risk of overfitting. (And hence, the percentage of correctly classified instances, using 10-fold cross validation should increase as the degree increases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition is right for the 'first' degrees, resulting in a 99.6% rate at degree=10. However, when I set the degree to 50, WEKA reports a 9% rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is it possible that this rate drops (and by so much)?&lt;/p&gt;&#xA;" OwnerUserId="49808" LastActivityDate="2018-04-20T03:39:35.883" Title="SVM with polynomial kernel: strange results with high degree?" Tags="|machine-learning|svm|weka|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29784" PostTypeId="2" ParentId="29775" CreationDate="2018-04-01T16:06:50.070" Score="0" Body="&lt;p&gt;Spell checking is not really in the realm of train-and-predict models in data science. Mainly because embeddings require known words in the case of word2vec/GloVe, known contexts in the case of doc2vec or known stems in the case of Fasttext plus a significantly large training dataset. Spell checking is instead more traditional software engineering. It's broad but fortunately, you can start small and increase it bit by bit. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If words are commonly known i.e. common nouns and pronouns you can use a package similar to &lt;a href=&quot;https://pypi.python.org/pypi/autocorrect&quot; rel=&quot;nofollow noreferrer&quot;&gt;autocorrect in Python&lt;/a&gt; but if you are looking for a specific list of words, in your case ports which are proper nouns, you must use a real spell check procedure. Since it's a broad topic you should start by following the answer provided by &lt;a href=&quot;https://stackoverflow.com/questions/13928155/spell-checker-for-python/13938115#13938115&quot;&gt;Stack Overflow&lt;/a&gt; about &lt;a href=&quot;http://norvig.com/spell-correct.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Peter Norvig's post&lt;/a&gt; and start building from there according to your needs.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2018-04-01T16:45:36.160" LastActivityDate="2018-04-01T16:45:36.160" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29785" PostTypeId="1" AcceptedAnswerId="29817" CreationDate="2018-04-01T16:43:36.280" Score="2" ViewCount="539" Body="&lt;p&gt;What are the benefits of having ML in JavaScript I.e. the deeplearn.js (now tensorflow) stuff, as opposed to implementing the ML steps in a python backend? &lt;/p&gt;&#xA;" OwnerUserId="33856" LastEditorUserId="28175" LastEditDate="2018-04-02T10:42:34.950" LastActivityDate="2018-05-10T19:37:46.927" Title="What are the benefits of having ML in js?" Tags="|machine-learning|neural-network|deep-learning|tensorflow|javascript|" AnswerCount="3" CommentCount="1" ClosedDate="2018-04-06T01:09:37.220" ContentLicense="CC BY-SA 3.0" />
  <row Id="29786" PostTypeId="2" ParentId="29775" CreationDate="2018-04-01T16:56:01.320" Score="2" Body="&lt;p&gt;Another way to reframe the problem is &lt;a href=&quot;https://en.wikipedia.org/wiki/Approximate_string_matching&quot; rel=&quot;nofollow noreferrer&quot;&gt;approximate/fuzzy string matching&lt;/a&gt;. &lt;a href=&quot;https://pypi.python.org/pypi/fuzzywuzzy&quot; rel=&quot;nofollow noreferrer&quot;&gt;FuzzyWuzzy&lt;/a&gt; is an approximate/fuzzy string matching package for Python. The package can find the nearest element in a set to a user's inputted value with a level of confidence. Something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from fuzzywuzzy import process&#xA;&#xA;ports = {&quot;Houston&quot;, &quot;New York&quot;, &quot;Newark&quot;, &quot;Long Beach&quot;}&#xA;user_input = &quot;Strong Beach&quot;&#xA;best_guess = process.extractOne(user_input, ports)&#xA;print(f&quot;The best match for '{user_input}' is '{best_guess[0]}' which is a {best_guess[1]}% match.&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The best match for 'Strong Beach' is 'Long Beach' which is a 82% match.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-01T16:56:01.320" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29787" PostTypeId="2" ParentId="29682" CreationDate="2018-04-01T18:26:04.327" Score="1" Body="&lt;p&gt;Ridge or L2 regularization is used to prevent over-fitting when having multi-col-linearity in your features. Here is an example : suppose you want to train a linear regression model on following dataset: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  x1      x2      y&#xA; 1.0    -1.0      5&#xA;-1.0     1.0      5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Training a linear regression model, we are searching for an hypothesis function with following format :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/imfWW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/imfWW.png&quot; alt=&quot;linear regression&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can observe, there are multiple solutions to this problem. For example :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta0 = 5, theta1 = 1 and theta2 = 1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But, there are also multiple other solutions to this problem. Including :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;theta0 = 5, theta1 = 1000000 and theta2 = 1000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now imagine you have a new observation such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      x1      x2      y&#xA; 1.00001    -1.0      5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A solution with small theta coefficients is most likely to yield better results than a solution with large theta coefficients. To understand L2 regularization, we have to understand that multicollinearity can yield to less robust solutions because is can appear ridges in cost function surface and so result in a less good solution. By adding a penalization term, we will smooth the cost function surface to make learning focus on small theta coefficients :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Nay6x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Nay6x.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, going back to your temperature example, ridge regression will reduce temperature theta coefficient as much as possible in order to avoid a potential multicollinearity with other features. By adding L2 regularization, it is now possible to address overfitting issues in model if there is multicollinearity in your features. However, if your regularization strength is too large, you risk to force your learning to focus on small values of theta only and so it could yield a bias emergence in your model and so an underfitting situation.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastEditorUserId="29575" LastEditDate="2018-04-03T05:16:32.487" LastActivityDate="2018-04-03T05:16:32.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29788" PostTypeId="1" AcceptedAnswerId="29795" CreationDate="2018-04-01T19:37:57.090" Score="1" ViewCount="63" Body="&lt;p&gt;I have the below sets of data per application, you can call them as software metrics. These metrics vary depending on the size of an application.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Bugs&#xA;CodeSmells&#xA;Vulnerability&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The size of the application is determined by LOC (Lines of code), how can i showcase the complexity of each app relative to the lines of code if i visualize each of these parameters.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Example&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;           Bugs   LOC&#xA;SweetApp   10     10000&#xA;SourApp    120    5660000&#xA;SaltyApp   55     1500&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do i visualize Bugs per app in relation to the LOC, because LOC determines their complexity a higher number in bugs doesn't necessarily be bad considering a higher number in LOC.&lt;/p&gt;&#xA;" OwnerUserId="49814" LastActivityDate="2018-04-02T01:39:05.533" Title="Visualizing software metrics" Tags="|visualization|software-development|data-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29791" PostTypeId="1" CreationDate="2018-04-01T21:27:26.850" Score="0" ViewCount="420" Body="&lt;p&gt;The predictors are latitude and longitude, and target variable is region.&lt;/p&gt;&#xA;" OwnerUserId="49717" LastActivityDate="2018-04-01T22:28:16.770" Title="Is there a way to standardize latitude and longitude to be used as predictors in KNN algorithm?" Tags="|k-nn|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-09T13:23:36.040" ContentLicense="CC BY-SA 3.0" />
  <row Id="29792" PostTypeId="2" ParentId="29735" CreationDate="2018-04-01T22:28:11.367" Score="5" Body="&lt;p&gt;After further working on this, I figured out that:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an activation function is also valid.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The homework implementation is indeed missing the derivative of softmax for the backprop pass.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The gradient of softmax with respect to its inputs is really the partial of each output with respect to each input:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?%5Cbegin%7Balign*%7D%20%5Cfrac%7B%5Cpartial%20smax%28x_i%29%7D%7B%5Cpartial%20x_i%7D%20%26%3D%5Cfrac%7Bf%27%28x%29g%28x%29-f%28x%29g%27%28x%29%7D%7Bg%28x%29%5E2%7D%20%5C%5C%5C%5C%20%26%3D%20%5Cfrac%7Be%5E%7Bx_i%7D%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%20-%20e%5E%7Bx_i%7D%5Cfrac%7B%5Cpartial%7D%7Bx_i%7D%7B%7D%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%20%7D%7B%28%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%29%5E2%7D%7D%5C%5C%5C%5C%20%26%3D%20%5Cfrac%7Be%5E%7Bx_i%7D%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%20-%20%28e%5E%7Bx_i%7D%29%5E2%7D%7B%28%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%29%5E2%7D%7D%5C%5C%5C%5C%20%26%3D%20%5Cleft%20%28%20%5Cfrac%7Be%5E%7Bx_i%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cfrac%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cright%20%29%20-%20%5Cleft%20%28%20%5Cfrac%7Be%5E%7Bx_i%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cfrac%7Be%5E%7Bx_i%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cright%20%29%5C%5C%5C%5C%20%26%3D%5Cfrac%7Be%5E%7Bx_i%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cleft%20%281-%20%5Cfrac%7Be%5E%7Bx_i%7D%7D%7B%5Csum_%7Bj%3D1%7D%5E%7B%7CX%7C%7D%7Be%5E%7Bx_j%7D%7D%7D%5Cright%20%29%5C%5C%5C%5C%20%26%3D%20smax%28x_i%29%281-smax%28x_i%29%29%20%5Cend%7Balign*%7D&quot; alt=&quot;gradient1&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for the vector (gradient) form:&#xA;&lt;img src=&quot;https://latex.codecogs.com/gif.latex?%5Cnabla%20smax%28%5Cbold%7Bx%7D%29%20%3D%20%5Cbegin%7Bbmatrix%7D%20smax%28x_0%29%281-smax%28x_0%29%29%2C%20%26%20smax%28x_1%29%281-smax%28x_1%29%29...%2C%20%26%20smax%28x_%7B%7C%5Cbold%7Bx%7D%7C%7D%29%281-smax%28x_%7B%7C%5Cbold%7Bx%7D%7C%7D%29%20%5Cend%7Bbmatrix%7D&quot; alt=&quot;gradient2&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which in my vectorized numpy code is simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;self.data * (1. - self.data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &lt;code&gt;self.data&lt;/code&gt; is the softmax of the input, previously computed from the forward pass.&lt;/p&gt;&#xA;" OwnerUserId="49780" LastActivityDate="2018-04-01T22:28:11.367" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29793" PostTypeId="2" ParentId="29791" CreationDate="2018-04-01T22:28:16.770" Score="1" Body="&lt;p&gt;Rather than modifying your data, use great circle distance for those features' conmponent of the cost function. &lt;/p&gt;&#xA;" OwnerUserId="4683" LastActivityDate="2018-04-01T22:28:16.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29794" PostTypeId="2" ParentId="29785" CreationDate="2018-04-01T23:57:50.510" Score="1" Body="&lt;p&gt;JavaScript is very a popular language, especially for web developers. Machine learning in a web-native language allows additional programmers to use machine learning more easily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;JavaScript is a client-side language that allows deep learning models to predict without server-side resources.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-10T19:37:46.927" LastActivityDate="2018-05-10T19:37:46.927" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29795" PostTypeId="2" ParentId="29788" CreationDate="2018-04-02T01:25:53.187" Score="1" Body="&lt;p&gt;You could use the ratios of bugs (or any other variable) divided by lines of code (LOC). Since ratios may vary a lot for instance bugs vs. vulnerabilities the resulting plot oftentimes doesn't look as good so you could use a normalization procedure, in fact, is the recommended procedure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case Bugs and a few vulnerabilities divided by LOC doesn't look too bad so I included the plot of raw ratios too in this example. I'm using a &lt;a href=&quot;http://ggplot2.tidyverse.org/reference/geom_count.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;count overlapping points plot&lt;/a&gt; which is a variant of a barplot that doesn't fill bars and only cares about the value as a point. You could use any normalization that you feel gives you the best result. Here I'm only centering the data. In both plots the highest value means higher complexity according to the ratio metric.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Raw Ratios of variables divided by LOC&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/94NSt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/94NSt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Normalized Ratios of variables divided by LOC&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mAVrq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mAVrq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Code in R needed to replicate these plots&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(&quot;reshape2&quot;)&#xA;require(&quot;ggplot2&quot;)&#xA;&#xA;df1 &amp;lt;- data.frame(App = c(&quot;SweetApp&quot;, &quot;SourApp&quot;, &quot;SaltyApp&quot;), &#xA;                  LOC = c(10000, 5660000, 1500), &#xA;                  Bugs = c(10, 120, 55), &#xA;                  CodeSmells = c(50, 30, 20), &#xA;                  Vulnerabilities = c(2, 3, 10))&#xA;&#xA;#Define ratios&#xA;df1$RatiosBugs &amp;lt;- df1$Bugs / df1$LOC&#xA;df1$RatiosCodeSmells &amp;lt;- df1$CodeSmells / df1$LOC&#xA;df1$RatiosVulnerabilities &amp;lt;- df1$Vulnerabilities / df1$LOC&#xA;&#xA;#Normalize Ratios&#xA;df1$NormRatiosBugs &amp;lt;- scale(df1$RatiosBugs, scale = FALSE)&#xA;df1$NormRatiosCodeSmells &amp;lt;- scale(df1$RatiosCodeSmells, scale = FALSE)&#xA;df1$NormRatiosVulnerabilities &amp;lt;- scale(df1$RatiosVulnerabilities, scale = FALSE)&#xA;&#xA;&#xA;dfRaw &amp;lt;- df1[, c(&quot;App&quot;, &quot;RatiosBugs&quot;, &quot;RatiosCodeSmells&quot;, &quot;RatiosVulnerabilities&quot;)]&#xA;dfNorm &amp;lt;- df1[, c(&quot;App&quot;, &quot;NormRatiosBugs&quot;, &#xA;              &quot;RatiosCodeSmells&quot;, &quot;RatiosVulnerabilities&quot;)]&#xA;dfRaw.m &amp;lt;- melt(dfRaw, id.vars = c(&quot;App&quot;))&#xA;dfNorm.m &amp;lt;- melt(dfNorm, id.vars = c(&quot;App&quot;))&#xA;&#xA;#Plot Raw Ratios&#xA;ggplot(dfRaw.m, aes(App, value)) + &#xA;  geom_count(aes(color = variable), position = position_dodge(width =  0.4), stat = &quot;identity&quot;) &#xA;&#xA;#Plot Normalized Ratios&#xA;ggplot(dfNorm.m, aes(App, value)) + &#xA;  geom_count(aes(color = variable), position = position_dodge(width = 0.4), stat = &quot;identity&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="13023" LastEditDate="2018-04-02T01:39:05.533" LastActivityDate="2018-04-02T01:39:05.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29796" PostTypeId="1" CreationDate="2018-04-02T02:19:21.477" Score="4" ViewCount="77" Body="&lt;p&gt;I try orange today, and want to use logloss as the classifier performance measures when &quot;&lt;a href=&quot;https://docs.orange.biolab.si/3/visual-programming/widgets/evaluation/testandscore.html#classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Test &amp;amp; Score&lt;/a&gt;&quot;, but I don't know how to do that.&lt;/p&gt;&#xA;" OwnerUserId="49826" LastActivityDate="2018-12-30T13:01:05.173" Title="Could I use logloss as classifier performance measures when &quot;Test &amp; Score&quot;?" Tags="|orange|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29797" PostTypeId="1" CreationDate="2018-04-02T03:00:02.403" Score="2" ViewCount="398" Body="&lt;p&gt;&lt;code&gt;tf.nn.dynamic_rnn()&lt;/code&gt; and &lt;code&gt;tf.nn.raw_rnn()&lt;/code&gt; take in an argument called parallel_iterations. The documentation says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;parallel_iterations: (Default: 32). The number of iterations to run in parallel. Those operations which do not have any temporal dependency and can be run in parallel, will be. This parameter trades off time for space. Values &gt;&gt; 1 use more memory but take less time, while smaller values use less memory but computations take longer.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If memory is not the constraint, should i not always keep its value equal to the number of processors available with me on GPU. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here i have my first question - Am i right in thinking that number of processors on GPU mean number of parallel floating point computations it can undertake (i don't suppose GPU's too implement 2 threads per processors as in CPUs)?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;GRU&lt;/a&gt; machinery is: (please note the biases are not showcased in the image but are to be used for better performance)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qXMDp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qXMDp.png&quot; alt=&quot;image&quot;&gt;&lt;/a&gt;, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To me the value 32 seems too low to be kept as default. If i have a GRU with 250 dimensional hidden state and 100 dimensional input i would have (2 x 250 x 350 x batch_size) floating multiplications that i can run in parallel (computing r and u) followed by (250 x batch_size) floating point additions that i can run in parallel followed by (250 x batch_size) sigmoid applications which are at least (250 x batch_size) potential parallel computations. Then it is a (250 x batch_size) parallel computations in application of reset gate followed by (1 x 250 x 350 x batch_size) parallel computations followed by (250 x batch_size) parallel computations for biases. Now in the final step it is (250 x batch_size) floating point computations to calculate (1 - update gate) followed by (500 x batch_size) parallel floating point computations for output state. To me it seems the bottlenecks in parallel implementations using GPU would be presence of too many low sized layers which, for instance here, are bias additions after matrix multiplications (at any rate they are still 250 x batch sized big here so will anyway use GPU to the maximum)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is the way i am thinking in the second paragraph correct while analyzing a deep network for a vague idea on how much the hardware would be able to impact its running time&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="32169" LastActivityDate="2018-04-02T03:00:02.403" Title="What should be the value of parallel iterations in tensorflow RNN implementations?" Tags="|deep-learning|tensorflow|rnn|gpu|parallel|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29798" PostTypeId="1" CreationDate="2018-04-02T05:14:10.107" Score="3" ViewCount="4132" Body="&lt;p&gt;I am a beginner in Machine Learning and I have a situation where an Image needs to be classified into first a super class, then a subclass.&#xA;For Example,&#xA;I have a set of images of fruits, containing images of Mangoes, Banana, Peaches, Apples etc. Then each fruit can either be fresh or rotten. I have training set containing images of rotten and fresh fruits for each Fruit type.&#xA;So for a test image I need to classify which fruit it is (Mango, Banana, etc) and then whether it is fresh or rotten. &#xA;I want to know how to approach this kind of problem&lt;/p&gt;&#xA;" OwnerUserId="49832" LastActivityDate="2019-07-09T19:46:18.933" Title="How to Classify an Image in a Class and a Subclass?" Tags="|machine-learning|image-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29799" PostTypeId="1" CreationDate="2018-04-02T05:31:58.243" Score="1" ViewCount="1966" Body="&lt;p&gt;While training the IMDB movie review dataset for sentiment analysis, the model will give a memory error if I set the features to more than 20,000. Is there any way to pass the memory error? &lt;/p&gt;&#xA;" OwnerUserId="49833" LastEditorUserId="14372" LastEditDate="2018-04-02T19:25:43.880" LastActivityDate="2018-04-03T07:19:28.163" Title="Python giving memory error while fitting the text into vector using various vectorizer" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29800" PostTypeId="1" CreationDate="2018-04-02T06:38:04.000" Score="1" ViewCount="34" Body="&lt;p&gt;Suppose we want to design a neural network that can diagnose skin cancer. We want this neural network to consider the possibility that the doctor we hired misclassified some of our images while labeling. How can we design our neural network?&lt;/p&gt;&#xA;" OwnerUserId="49835" LastEditorUserId="28175" LastEditDate="2018-04-02T10:44:23.007" LastActivityDate="2018-04-02T10:44:23.007" Title="What to do when we can't trust our human classifiers?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29801" PostTypeId="1" CreationDate="2018-04-02T07:13:46.717" Score="0" ViewCount="234" Body="&lt;p&gt;(This is not a question on ways to handle missing data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a &quot;large proportion&quot; (more than 20%) of the records.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general way to decide the % of the total rows for which a feature has to be missing, so that we can consider it useless and remove it from our analysis? Or does this depend on the context? Indirectly, I want to know the % of missing values for a feature so that imputation is meaningless.&lt;/p&gt;&#xA;" OwnerUserId="43193" LastActivityDate="2023-10-21T01:27:26.550" Title="Percentage of missing values so that we can't perform imputation" Tags="|feature-selection|data-imputation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29802" PostTypeId="1" AcceptedAnswerId="29803" CreationDate="2018-04-02T08:01:28.420" Score="1" ViewCount="240" Body="&lt;h3&gt;Which measurement(s) should one choose to compare two regression models?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;After modifying a learning algorithm(specifically, a regression algorithm, let's call it &lt;strong&gt;M1&lt;/strong&gt;) to generate another learning algorithm &lt;strong&gt;M2&lt;/strong&gt;, how to validate if the above modification is efficient?&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;here is what I did(with 10-fold cross-validation)&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I choose &lt;strong&gt;MSE&lt;/strong&gt; as the only measurement, at each run, for &lt;strong&gt;M1&lt;/strong&gt; and &lt;strong&gt;M2&lt;/strong&gt;, calculate the &lt;strong&gt;MSE&lt;/strong&gt; of both the training and testing set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the result shows that:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;average MSE of the training set of 10 runs: &lt;strong&gt;M2&lt;/strong&gt; &amp;lt; &lt;strong&gt;M1&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;average MSE of the testing set of 10 runs: &lt;strong&gt;M2&lt;/strong&gt; &amp;lt; &lt;strong&gt;M1&lt;/strong&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;according to the above list, can we draw a conclusion that &lt;strong&gt;M2&lt;/strong&gt; is better than &lt;strong&gt;M1&lt;/strong&gt;? thus, the modification of algorithm &lt;strong&gt;M1&lt;/strong&gt; is efficient(at least on this dataset)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Or:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?&lt;/p&gt;&#xA;" OwnerUserId="24005" LastActivityDate="2019-05-09T19:04:24.363" Title="How to compare two regression models?" Tags="|machine-learning|regression|cross-validation|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29803" PostTypeId="2" ParentId="29802" CreationDate="2018-04-02T08:51:58.057" Score="1" Body="&lt;p&gt;There are two things to consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Sampling bias&lt;/li&gt;&#xA;&lt;li&gt;Metric&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The sampling bias problem is that your test set is likely not the complete set of things you're interested in. So, no, you can't simply check MSE_1 &amp;lt; MSE_2 and conclude it is always the case when it's &quot;just&quot; for your dataset the case. This is what significance tests are for. (Although this kind of reasoning is super common in machine learning and I did it myself already )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the question if the metric is the correct one for your application. Typical choices are: MSE, mean absolute error, custom cost functions&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-04-02T09:13:27.260" LastActivityDate="2018-04-02T09:13:27.260" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29804" PostTypeId="2" ParentId="29800" CreationDate="2018-04-02T09:01:23.300" Score="2" Body="&lt;p&gt;First, neural networks are good in dealing with&quot;label noise&quot;. I'm currently on mobile/vacation, so remind me to search the paper on Friday.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see the ways: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-04-02T09:01:23.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29805" PostTypeId="2" ParentId="29798" CreationDate="2018-04-02T09:07:34.773" Score="3" Body="&lt;h2&gt;The simple solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Have one classifier which can classify all the combinations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fresh banana&lt;/li&gt;&#xA;&lt;li&gt;Rotten banana&lt;/li&gt;&#xA;&lt;li&gt;Fresh mango&lt;/li&gt;&#xA;&lt;li&gt;Rotten mango&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;The good solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Or have one network with multiple outputs. See the master Thesis of Marvin Teichmann for example. &#xA;&lt;a href=&quot;https://arxiv.org/abs/1612.07695&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1612.07695&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So one path of the network would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Fresh vs Rotten&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The other path of the network would be &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Banana vs Mango vs ...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The difference to the simple solution is that you can give a probability to the cases, e.g. for the ground truth &quot;fresh banana&quot; you might say 99% banana but only 42% fresh.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;The obvious solution&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This one should not be done: have multiple classifiers&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2019-07-09T19:46:18.933" LastActivityDate="2019-07-09T19:46:18.933" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="29807" PostTypeId="2" ParentId="29776" CreationDate="2018-04-02T09:17:50.273" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;My network does always predict the same class. What is the problem?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I had this a couple of times. Although I'm currently too lazy to go through your code, I think I can give some general hints which might also help others who have the same symptom but probably different underlying problems.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Debugging Neural Networks&lt;/h2&gt;&#xA;&#xA;&lt;h3&gt;Fitting one item datasets&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;For every class i the network should be able to predict, try the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a dataset of only one data point of class i.&lt;/li&gt;&#xA;&lt;li&gt;Fit the network to this dataset.&lt;/li&gt;&#xA;&lt;li&gt;Does the network learn to predict &quot;class i&quot;?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If this doesn't work, there are four possible error sources:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Buggy training algorithm&lt;/strong&gt;: Try a smaller model, print a lot of values which are calculated in between and see if those match your expectation.&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Dividing by 0: Add a small number to the denominator&lt;/li&gt;&#xA;&lt;li&gt;Logarithm of 0 / negativ number: Like dividing by 0&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Data&lt;/strong&gt;: It is possible that your data has the wrong type. For example, it might be necessary that your data is of type &lt;a href=&quot;https://docs.scipy.org/doc/numpy/user/basics.types.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;float32&lt;/code&gt;&lt;/a&gt; but actually is an integer.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Model&lt;/strong&gt;: It is also possible that you just created a model which cannot possibly predict what you want. This should be revealed when you try simpler models.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Initialization / Optimization&lt;/strong&gt;: Depending on the model, your initialization and your optimization algorithm might play a crucial role. For beginners who use standard stochastic gradient descent, I would say it is mainly important to initialize the weights randomly (each weight a different value). - see also: &lt;a href=&quot;https://stackoverflow.com/a/41565985/562769&quot;&gt;this question / answer&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;Learning Curve&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; for details.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mhAUB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mhAUB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to start with a tiny training dataset (probably only one item). Then the model should be able to fit the data perfectly. If this works, you make a slightly larger dataset. Your training error should slightly go &lt;strong&gt;up&lt;/strong&gt; at some point. This reveals your models capacity to model the data.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Data analysis&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Check how often the other class(es) appear. If one class dominates the others (e.g. one class is 99.9% of the data), this is a problem. Look for &quot;outlier detection&quot; techniques.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;More&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning rate&lt;/strong&gt;: If your network doesn't improve and get only slightly better than random chance, try reducing the learning rate. For computer vision, a learning rate of &lt;code&gt;0.001&lt;/code&gt; is often used / working. This is also relevant if you use Adam as an optimizer.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Preprocessing&lt;/strong&gt;: Make sure you use the same preprocessing for training and testing. You might see differences in the confusion matrix (see &lt;a href=&quot;https://stackoverflow.com/q/42705400/562769&quot;&gt;this question&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h3&gt;Common Mistakes&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;This is inspired by &lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/6xvnwo/d_my_neural_network_isnt_working_what_should_i_do/&quot; rel=&quot;nofollow noreferrer&quot;&gt;reddit&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You forgot to apply preprocessing&lt;/li&gt;&#xA;&lt;li&gt;Dying ReLU&lt;/li&gt;&#xA;&lt;li&gt;To small / to big learning rate&lt;/li&gt;&#xA;&lt;li&gt;Wrong activation function in final layer:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Your targets are not in sum one? -&gt; Don't use softmax&lt;/li&gt;&#xA;&lt;li&gt;Single elements of your targets are negative -&gt; Don't use Softmax, ReLU, Sigmoid. tanh might be an option&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Too deep network: You fail to train. Try a simpler neural network first.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-04-02T09:17:50.273" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29809" PostTypeId="2" ParentId="29799" CreationDate="2018-04-02T09:46:24.617" Score="2" Body="&lt;p&gt;to answer that question you need to get and give us more information, no chance to advise without knowing where the OOM is thrown and how the memory profile looks like. You may tune the memory usage of your program also in other parts than the part where the error came from, to raise the limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just one hint for a cause that can be resolved without getting more physical memory: &#xA;Sometimes you can &quot;lazy load&quot; large data sources. Get it record by record or chunk by chunk, then process it, and write results to the sink, before you get the next record into memory. This is done with python generators, watch out for the keyword &quot;yield&quot; that is used instead of return. &#xA;&lt;a href=&quot;https://stackoverflow.com/questions/1756096/understanding-generators-in-python&quot;&gt;https://stackoverflow.com/questions/1756096/understanding-generators-in-python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;further you may be better off using low level libraries for certain tasks, see i.e. &lt;a href=&quot;http://www.nltk.org/book/ch06.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.nltk.org/book/ch06.html&lt;/a&gt;  2.4 &lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-02T11:03:53.120" LastActivityDate="2018-04-02T11:03:53.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29810" PostTypeId="2" ParentId="29800" CreationDate="2018-04-02T10:32:01.273" Score="0" Body="&lt;p&gt;I have the following solutions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you have abundant data you can shuffle them and make validation and training data. After that, your neural network should exploit generalization techniques not to overfit the training data. By doing so, you may have relatively acceptable performance which works in noisy situations. &lt;/li&gt;&#xA;&lt;li&gt;The other technique is evaluating Bayes error. This does not have any relation to the neural nets. It just tries to investigate in the feature space of the problem what percentage of your data is misleading, having same input patterns with contradictory labels.&lt;/li&gt;&#xA;&lt;li&gt;Another approach can be using an existing model for validating the data-set.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-02T10:32:01.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29811" PostTypeId="2" ParentId="29801" CreationDate="2018-04-02T10:40:16.710" Score="2" Body="&lt;p&gt;In my opinion, that depends on the context, like the question you want to answer, and also on the domain.&lt;/p&gt;&#xA;&lt;p&gt;Suppose the remaining absolute data set is still big and diverse enough to allow conclusions about that feature. In that case, the percentage of records that cannot add to that imputation is not a problem, in my understanding.&lt;/p&gt;&#xA;&lt;p&gt;It might also be that features sometimes are not just missing but do not apply to the particular kind of record.&lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="69263" LastEditDate="2023-10-21T01:27:26.550" LastActivityDate="2023-10-21T01:27:26.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29812" PostTypeId="2" ParentId="29785" CreationDate="2018-04-02T10:41:48.207" Score="0" Body="&lt;p&gt;It is a bit technical and depends where you use your &lt;code&gt;JavaScript&lt;/code&gt; code. It is used in both backend and frontend apps. My opinion is that using that in frontend apps can help your ML algorithms run on distributed devices. Take a look at &lt;a href=&quot;http://cs.stanford.edu/people/karpathy/convnetjs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. You can use this language to run ML codes on hosts' computers.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-02T10:41:48.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29814" PostTypeId="2" ParentId="24319" CreationDate="2018-04-02T11:29:55.713" Score="1" Body="&lt;p&gt;Microsoft's &lt;a href=&quot;https://studio.azureml.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Azure Machine Learning Studio&lt;/a&gt; has an &quot;always free&quot; tier, subject to &lt;a href=&quot;https://azure.microsoft.com/en-gb/pricing/details/machine-learning-studio/&quot; rel=&quot;nofollow noreferrer&quot;&gt;certain limitations&lt;/a&gt;, including&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;100 modules per experiment (a &quot;module&quot; in Azure-speak is any discrete operation such as &quot;load data&quot; or &quot;train model&quot;, so you can do quite a bit with 100 of them)&lt;/li&gt;&#xA;&lt;li&gt;10Gb of storage&lt;/li&gt;&#xA;&lt;li&gt;1 hour per experiment&lt;/li&gt;&#xA;&lt;li&gt;No parallel execution on multiple nodes&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Training will continue to run while you are not connected, to answer your second question. You can set up your experiments via the web interface or &lt;a href=&quot;https://github.com/hning86/azuremlps&quot; rel=&quot;nofollow noreferrer&quot;&gt;on the command line&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32402" LastEditorUserId="32402" LastEditDate="2018-04-02T11:38:59.663" LastActivityDate="2018-04-02T11:38:59.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29815" PostTypeId="1" CreationDate="2018-04-02T11:58:54.950" Score="4" ViewCount="705" Body="&lt;p&gt;I understand how the convolution layers are applied after selective search finds the regions of interest in vanilla R-CNN and so the back-propagation or any weight updating is done in the individual convolution networks. But in Faster R-CNN a convNet is applied first and then RPN works out the regions of interest in a separate convNet. Where exactly are the weights updated? Is it between the initial convNet and the RPN. But weights can only be updated after comparison with the ground truth values. Or is it that after a full forward pass of both networks, a backward pass is done. In this case how are both considered simultaneously? If not, Where and how is this comparison done? Any pointers to any resource will be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="43967" LastActivityDate="2022-12-01T12:18:49.473" Title="Backpropagation in Faster R-CNN" Tags="|machine-learning|neural-network|cnn|faster-rcnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29816" PostTypeId="1" AcceptedAnswerId="29873" CreationDate="2018-04-02T12:34:55.033" Score="3" ViewCount="1591" Body="&lt;p&gt;Inverse reinforcement learning is about using expert trajectories to learn a reward function. Now the most successful method is &lt;strong&gt;Maximum Entropy Inverse Reinforcement Learning&lt;/strong&gt;. But in that, you need a model-based reinforcement learning. But most of the practical world problems are model-free which is hard to calculate the state transition probabilities. So how can we use this &lt;strong&gt;inverse reinforcement learning&lt;/strong&gt; in real-world problems? &lt;/p&gt;&#xA;" OwnerUserId="33338" LastActivityDate="2018-04-03T21:17:10.557" Title="What are the practical applications of inverse reinforcement learning?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ClosedDate="2018-04-06T01:08:09.460" ContentLicense="CC BY-SA 3.0" />
  <row Id="29817" PostTypeId="2" ParentId="29785" CreationDate="2018-04-02T14:09:35.107" Score="2" Body="&lt;p&gt;There are a lot of services that offer free or very cheap hosting of static websites. If you are able to implement your ML model in JS this allows you to deploy your product/app/whatever easily and with low cost. In comparison, requiring a backend server running your model is harder to setup and maintain, in addition to costing more. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-04-02T14:09:35.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29818" PostTypeId="1" CreationDate="2018-04-02T14:32:10.570" Score="0" ViewCount="773" Body="&lt;p&gt;I have two vector-space models,  with different dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The number of vectors in one model is the same as the number of vectors in the other. I.E: if I have vector representation for a &lt;em&gt;car&lt;/em&gt; in one model,  I have vector representation for a &lt;em&gt;car&lt;/em&gt; in the other model, but the number of dimensions can be different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to combine these models (and then cluster using the combined model),  I cannot average (BoW) or add these models together as stated earlier they have different dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was going to simply concatenate the vectors. Is this valid or  is there better way to do this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also before you concatenate should you normalize? &lt;/p&gt;&#xA;" OwnerUserId="47149" LastActivityDate="2020-09-27T13:05:00.473" Title="Stacking/Concatenating/Combining two vector space models" Tags="|machine-learning|nlp|vector-space-models|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29819" PostTypeId="2" ParentId="15784" CreationDate="2018-04-02T14:45:43.780" Score="0" Body="&lt;p&gt;There is a possibility of refining word2vec vectors, which as research shows capture both, semantic relatedness and semantic similarity, in such a way, that they would capture the relations between words such as antonymy or negation. You can take a look at &lt;a href=&quot;https://arxiv.org/abs/1603.00892&quot; rel=&quot;nofollow noreferrer&quot;&gt;Counter-Fitting&lt;/a&gt; method (or methods in it's related work). Their implementation should be available online.&#xA;This may improve results of your sentiment analysis method.&lt;/p&gt;&#xA;" OwnerUserId="49851" LastActivityDate="2018-04-02T14:45:43.780" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29820" PostTypeId="2" ParentId="29798" CreationDate="2018-04-02T15:03:41.233" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I want to know how to approach this kind of problem&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes this is definately possible... (Probably Known as &lt;code&gt;Classification + Localization&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Split the conv-nets into 2 parallel arch's at the very end , before We generally add a single-dense-layer after Feature Extraction, here you will have a split into two...(the image attached will make it clear) &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One to &lt;strong&gt;detect&lt;/strong&gt; the Class i.e Which Fruit it is...&lt;/li&gt;&#xA;&lt;li&gt;Second to &lt;strong&gt;classify&lt;/strong&gt; them as Rotten/Fresh&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Have a look at this Image(From CS-231N Some credits apply)(in your case, replace Animals class and species with Fruits Class and State)(you don't need a bbox, so a modification to the arch will yield the results as desired provided that the &lt;strong&gt;loss is calculated and used correctly&lt;/strong&gt; on both the ends of the network)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This Paper &lt;a href=&quot;https://arxiv.org/abs/1601.02913&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; probably address the same&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9MdRY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9MdRY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="29169" LastEditDate="2019-06-09T15:15:15.607" LastActivityDate="2019-06-09T15:15:15.607" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29821" PostTypeId="2" ParentId="29818" CreationDate="2018-04-02T15:05:52.300" Score="1" Body="&lt;p&gt;It is perfectly valid to concatenate the vectors from two different models. It will be necessary however to experiment with the two approaches, i.e. either using individual vectors from one of the models or concatenating the vectors, before you can determine which method will produce better clustering results.  It is also valid to normalize word vectors, but again requires experimentation to determine whether this approach will provide superior results.&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-04-02T15:05:52.300" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29823" PostTypeId="1" CreationDate="2018-04-02T17:27:05.003" Score="-1" ViewCount="916" Body="&lt;p&gt;I have a contingency table listing individuals with certain traits. For the sake of simplicity, let's say the table has individuals in rows, and the type of food they like in columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        | Pizza | Spinach | Cheese |&#xA;|-------|-------|---------|--------|&#xA;| Tom   |   0   |    1    |    1   |&#xA;| Jerry |   1   |    0    |    0   |&#xA;| Marie |   0   |    0    |    1   |&#xA;| Alex  |   1   |    0    |    1   |&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to cluster individuals with similar tastes together. What's the best approach for this? Would hierarchical clustering be appropriate for this kind of data? Would k-modes work?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also want to know which foods are the best separators for the clusters. Could correspondence analysis help me with that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, I want to know which foods are correlated. Can I use Spearman's coefficient on this kind of data? &lt;/p&gt;&#xA;" OwnerUserId="49856" LastActivityDate="2018-05-03T20:31:03.843" Title="Clustering a contingency table" Tags="|clustering|correlation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29824" PostTypeId="1" CreationDate="2018-04-02T17:38:24.867" Score="1" ViewCount="7577" Body="&lt;p&gt;I'm trying to use L1 regularization to select features in XGBoost classifier. However, I don't see any example code on how to specify the penalty of l1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I do in sklearn's LogisticRegression. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;C = [10, 1, .1, .001]&#xA;&#xA;for c in C:&#xA;    clf = LogisticRegression(penalty='l1', C=c)&#xA;    clf.fit(X_train, y_train)&#xA;    print('C:', c)&#xA;    print('Training accuracy:', clf.score(X_train, y_train))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should I specify penalty and C in XGBoost?&lt;/p&gt;&#xA;" OwnerUserId="37932" LastEditorUserId="61563" LastEditDate="2019-07-08T22:47:41.283" LastActivityDate="2019-07-08T22:47:41.283" Title="Using L1 penalty in XGBoost" Tags="|xgboost|regularization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29825" PostTypeId="1" AcceptedAnswerId="29835" CreationDate="2018-04-02T18:45:40.180" Score="4" ViewCount="80504" Body="&lt;p&gt;I have a large data set (4.5 million rows, 35 columns). The columns of interest are &lt;code&gt;company_id&lt;/code&gt; (string) and &lt;code&gt;company_score&lt;/code&gt; (float). There are approximately 10,000 unique &lt;code&gt;company_id&lt;/code&gt;'s.  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;company_id    company_score    date_submitted    company_region&#xA;AA            .07              1/1/2017          NW&#xA;AB            .08              1/2/2017          NE&#xA;CD            .0003            1/18/2017         NW&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My goal is to create approximately 10,000 new dataframes, by unique &lt;code&gt;company_id&lt;/code&gt;, with only the relevant rows in that data frame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first idea I had was to create the collection of data frames shown below, then loop through the original data set and append in new values based on criteria.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;company_dictionary = {}  &#xA;for company in df['company_id']:  &#xA;    company_dictionary[company_id] = pd.DataFrame([])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a better way to do this by leveraging pandas?  i.e., is there a way I can use a built-in pandas function to create new filtered dataframes with only the relevant rows?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I tried a new approach, but I'm now encountering an error message that I don't understanding.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    [In]  unique_company_id = np.unique(df[['ID_BB_GLOBAL']].values)&#xA;    [In]  unique_company_id&#xA;    [Out] array(['BBG000B9WMF7', 'BBG000B9XBP9', 'BBG000B9ZG58', ..., 'BBG00FWZQ3R9',&#xA;       'BBG00G4XRQN5', 'BBG00H2MZS56'], dtype=object)&#xA;    [In]  for id in unique_company_id:&#xA;    [In]      new_df = df[df['id'] == id]&#xA;---------------------------------------------------------------------------&#xA;KeyError                                  Traceback (most recent call last)&#xA;C:\get_loc(self, key, method, tolerance)&#xA;   2133             try:&#xA;-&amp;gt; 2134                 return self._engine.get_loc(key)&#xA;   2135             except KeyError:&#xA;&#xA;pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4433)()&#xA;&#xA;pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4279)()&#xA;&#xA;pandas\src\hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:13742)()&#xA;&#xA;pandas\src\hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:13696)()&#xA;&#xA;KeyError: 'id'&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;&amp;lt;ipython-input-50-dce34398f1e1&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      1 for id in unique_bank_id:&#xA;----&amp;gt; 2     new_df = df[df['id'] == id]&#xA;&#xA;C:\ in __getitem__(self, key)&#xA;   2057             return self._getitem_multilevel(key)&#xA;   2058         else:&#xA;-&amp;gt; 2059             return self._getitem_column(key)&#xA;   2060 &#xA;   2061     def _getitem_column(self, key):&#xA;&#xA;C:\ in _getitem_column(self, key)&#xA;   2064         # get column&#xA;   2065         if self.columns.is_unique:&#xA;-&amp;gt; 2066             return self._get_item_cache(key)&#xA;   2067 &#xA;   2068         # duplicate columns &amp;amp; possible reduce dimensionality&#xA;&#xA;C:\ in _get_item_cache(self, item)&#xA;   1384         res = cache.get(item)&#xA;   1385         if res is None:&#xA;-&amp;gt; 1386             values = self._data.get(item)&#xA;   1387             res = self._box_item_values(item, values)&#xA;   1388             cache[item] = res&#xA;&#xA;C:\ in get(self, item, fastpath)&#xA;   3541 &#xA;   3542             if not isnull(item):&#xA;-&amp;gt; 3543                 loc = self.items.get_loc(item)&#xA;   3544             else:&#xA;   3545                 indexer = np.arange(len(self.items))[isnull(self.items)]&#xA;&#xA;C:\ in get_loc(self, key, method, tolerance)&#xA;   2134                 return self._engine.get_loc(key)&#xA;   2135             except KeyError:&#xA;-&amp;gt; 2136                 return self._engine.get_loc(self._maybe_cast_indexer(key))&#xA;   2137 &#xA;   2138         indexer = self.get_indexer([key], method=method, tolerance=tolerance)&#xA;&#xA;pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4433)()&#xA;&#xA;pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4279)()&#xA;&#xA;pandas\src\hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:13742)()&#xA;&#xA;pandas\src\hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:13696)()&#xA;&#xA;KeyError: 'id'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49861" LastEditorUserId="35644" LastEditDate="2018-04-03T16:43:46.183" LastActivityDate="2020-04-05T19:25:56.037" Title="Create new data frames from existing data frame based on unique column values" Tags="|python|pandas|dataframe|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29828" PostTypeId="2" ParentId="29824" CreationDate="2018-04-02T20:17:39.660" Score="5" Body="&lt;p&gt;L2 and L1 regularization are controlled via the &lt;code&gt;lambda (=reg_lambda)&lt;/code&gt; and &lt;code&gt;alpha (=reg_alpha)&lt;/code&gt; parameter respectively. Higher values of &lt;code&gt;alpha&lt;/code&gt; mean more L1 regularization. See the documentation &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest///parameter.html#parameters-for-tree-booster&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-04-02T20:17:39.660" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29829" PostTypeId="1" CreationDate="2018-04-02T20:20:00.337" Score="1" ViewCount="87" Body="&lt;p&gt;I am using a implementation that use both camera and Lidar for 3D obstacle detection for self driving cars but I have no idea how to calculate the distance to the obstacles since the Lidar uses the Bird's Eye View ?&lt;/p&gt;&#xA;" OwnerUserId="41297" LastEditorUserId="29169" LastEditDate="2021-02-13T16:34:21.817" LastActivityDate="2021-02-13T16:34:21.817" Title="How to estimate Distance to Obstacles using Lidar's BEV(Bir's Eye view ) Representation?" Tags="|machine-learning|deep-learning|neural-network|computer-vision|convolutional-neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29832" PostTypeId="2" ParentId="29739" CreationDate="2018-04-03T01:22:41.497" Score="1" Body="&lt;p&gt;The most obvious difference is that by adding epochs, you (still) never get the same observation twice in a same batch, which isn't the case when you duplicate data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm no expert in NNs, but it seems that if you do not introduce any noise in the duplicated data, then you might as well just add epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, if you fit two models &lt;em&gt;over a single epoch&lt;/em&gt;,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;model 1: using your n=1000 observations&lt;/p&gt;&#xA;&#xA;&lt;p&gt;model 2: using your n*10 = 10,000 observations (with duplications)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;your second model will most probably return a lower loss value. However, a fairer comparison would be to train model 1 with 10 epochs so that in both cases you expose your net to your observations the same number of times in total.&lt;/p&gt;&#xA;" OwnerUserId="47308" LastActivityDate="2018-04-03T01:22:41.497" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29833" PostTypeId="2" ParentId="28575" CreationDate="2018-04-03T01:51:54.603" Score="1" Body="&lt;p&gt;Polynomial regression can have multiple entries in the normal equation and it is not easy to say which polynomials you have to use in advance. Moreover, if you have lots of features you cannot handle memory errors most of the time. Nowadays people use &lt;code&gt;MLP&lt;/code&gt;s and use batch normalization among layers for learning better. Those that you are referring to are a bit old algorithms but the former one is the logical mathematical solution for learning problems and the latter one is a beginning point for deep neural networks. I recommend taking a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/21896/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/q/18802/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-03T01:51:54.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29834" PostTypeId="2" ParentId="29520" CreationDate="2018-04-03T04:48:53.463" Score="5" Body="&lt;p&gt;With respect to the first and second question, the code should change into:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import Counter&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.model_selection import train_test_split, StratifiedKFold, learning_curve, validation_curve, GridSearchCV&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.linear_model import LogisticRegression&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.metrics import classification_report&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;&#xA;def plot_learning_curve(train_sizes, train_scores, test_scores, title, alpha=0.1):&#xA;    train_mean = np.mean(train_scores, axis=1)&#xA;    train_std = np.std(train_scores, axis=1)&#xA;    test_mean = np.mean(test_scores, axis=1)&#xA;    test_std = np.std(test_scores, axis=1)&#xA;    plt.plot(train_sizes, train_mean, label='train score', color='blue', marker='o')&#xA;    plt.fill_between(train_sizes, train_mean + train_std,&#xA;                     train_mean - train_std, color='blue', alpha=alpha)&#xA;    plt.plot(train_sizes, test_mean, label='test score', color='red', marker='o')&#xA;&#xA;    plt.fill_between(train_sizes, test_mean + test_std, test_mean - test_std, color='red', alpha=alpha)&#xA;    plt.title(title)&#xA;    plt.xlabel('Number of training points')&#xA;    plt.ylabel('F-measure')&#xA;    plt.grid(ls='--')&#xA;    plt.legend(loc='best')&#xA;    plt.show()&#xA;&#xA;&#xA;def plot_validation_curve(param_range, train_scores, test_scores, title, alpha=0.1):&#xA;    param_range = [x[1] for x in param_range] &#xA;    sort_idx = np.argsort(param_range)&#xA;    param_range=np.array(param_range)[sort_idx]&#xA;    train_mean = np.mean(train_scores, axis=1)[sort_idx]&#xA;    train_std = np.std(train_scores, axis=1)[sort_idx]&#xA;    test_mean = np.mean(test_scores, axis=1)[sort_idx]&#xA;    test_std = np.std(test_scores, axis=1)[sort_idx]&#xA;    plt.plot(param_range, train_mean, label='train score', color='blue', marker='o')&#xA;    plt.fill_between(param_range, train_mean + train_std,&#xA;                 train_mean - train_std, color='blue', alpha=alpha)&#xA;    plt.plot(param_range, test_mean, label='test score', color='red', marker='o')&#xA;    plt.fill_between(param_range, test_mean + test_std, test_mean - test_std, color='red', alpha=alpha)&#xA;    plt.title(title)&#xA;    plt.grid(ls='--')&#xA;    plt.xlabel('Weight of class 2')&#xA;    plt.ylabel('Average values and standard deviation for F1-Score')&#xA;    plt.legend(loc='best')&#xA;    plt.show()&#xA;&#xA;&#xA;if __name__ == '__main__':&#xA;    X, y = make_classification(n_classes=2, class_sep=2, weights=[0.9, 0.1], n_informative=3, n_redundant=1, flip_y=0,&#xA;                               n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)&#xA;    print('Original dataset shape {}'.format(Counter(y)))&#xA;&#xA;    ln = X.shape&#xA;    names = [&quot;x%s&quot; % i for i in range(1, ln[1] + 1)]&#xA;&#xA;    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)&#xA;    st = StandardScaler()&#xA;&#xA;    rg = LogisticRegression(class_weight={0: 1, 1: 6.5}, random_state=42, solver='saga', max_iter=100, n_jobs=-1)&#xA;&#xA;    param_grid = {'clf__C': [0.001, 0.01, 0.1, 0.002, 0.02, 0.005, 0.0007, .0006, 0.0005],&#xA;                  'clf__class_weight': [{0: 1, 1: 6}, {0: 1, 1: 4}, {0: 1, 1: 5.5}, {0: 1, 1: 4.5}, {0: 1, 1: 5}]&#xA;                  }&#xA;&#xA;    pipeline = Pipeline(steps=[('scaler', st),&#xA;                               ('clf', rg)])&#xA;&#xA;    cv = StratifiedKFold(n_splits=5, random_state=42)&#xA;    rg_cv = GridSearchCV(pipeline, param_grid, cv=cv, scoring='f1')&#xA;    rg_cv.fit(X_train, y_train)&#xA;    print(&quot;Tuned rg best params: {}&quot;.format(rg_cv.best_params_))&#xA;&#xA;    ypred = rg_cv.predict(X_train)&#xA;    print(classification_report(y_train, ypred))&#xA;    print('######################')&#xA;    ypred2 = rg_cv.predict(X_test)&#xA;    print(classification_report(y_test, ypred2))&#xA;&#xA;    plt.figure(figsize=(9, 6))&#xA;    param_range1 = [i / 10000.0 for i in range(1, 11)]&#xA;    param_range2 = [{0: 1, 1: 6}, {0: 1, 1: 4}, {0: 1, 1: 5.5}, {0: 1, 1: 4.5}, {0: 1, 1: 5}]&#xA;&#xA;    train_sizes, train_scores, test_scores = learning_curve(&#xA;        estimator=rg_cv.best_estimator_, X=X_train, y=y_train,&#xA;        train_sizes=np.arange(0.1, 1.1, 0.1), cv=cv, scoring='f1', n_jobs=- 1)&#xA;&#xA;    plot_learning_curve(train_sizes, train_scores, test_scores, title='Learning curve for Logistic Regression')&#xA;&#xA;    train_scores, test_scores = validation_curve(&#xA;        estimator=rg_cv.best_estimator_, X=X_train, y=y_train, param_name=&quot;clf__C&quot;, param_range=param_range1,&#xA;        cv=cv, scoring=&quot;f1&quot;, n_jobs=-1)&#xA;&#xA;    plot_validation_curve(param_range1, train_scores, test_scores, title=&quot;Validation Curve for C&quot;, alpha=0.1)&#xA;&#xA;    train_scores, test_scores = validation_curve(&#xA;        estimator=rg_cv.best_estimator_, X=X_train, y=y_train, param_name=&quot;clf__class_weight&quot;, param_range=param_range2,&#xA;        cv=cv, scoring=&quot;f1&quot;, n_jobs=-1)&#xA;&#xA;    plot_validation_curve(param_range2, train_scores, test_scores, title=&quot;Validation Curve for class_weight&quot;, alpha=0.1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="26019" LastEditDate="2018-06-23T07:59:08.430" LastActivityDate="2018-06-23T07:59:08.430" CommentCount="18" ContentLicense="CC BY-SA 4.0" />
  <row Id="29835" PostTypeId="2" ParentId="29825" CreationDate="2018-04-03T05:24:10.557" Score="2" Body="&lt;p&gt;You can &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;groupby&lt;/code&gt;&lt;/a&gt; &lt;code&gt;company_id&lt;/code&gt; column and convert its result into a dictionary of DataFrames:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.DataFrame({&#xA;    &quot;company_id&quot;: [&quot;AA&quot;, &quot;AB&quot;, &quot;AA&quot;, &quot;CD&quot;, &quot;AB&quot;],&#xA;    &quot;company_score&quot;: [.07, .08, .06, .0003, .09],&#xA;    &quot;company_region&quot;: [&quot;NW&quot;, &quot;NE&quot;, &quot;NW&quot;, &quot;NW&quot;, &quot;NE&quot;]})&#xA;&#xA;# Approach 1&#xA;dict_of_companies = {k: v for k, v in df.groupby('company_id')}&#xA;&#xA;# Approach 2&#xA;dict_of_companies = dict(tuple(df.groupby(&quot;company_id&quot;)))&#xA;&#xA;import pprint&#xA;pprint.pprint(dict_of_companies)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'AA':   company_id company_region  company_score&#xA;0         AA             NW           0.07&#xA;2         AA             NW           0.06,&#xA; 'AB':   company_id company_region  company_score&#xA;1         AB             NE           0.08&#xA;4         AB             NE           0.09,&#xA; 'CD':   company_id company_region  company_score&#xA;3         CD             NW         0.0003}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2018-04-03T05:24:10.557" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29836" PostTypeId="2" ParentId="29761" CreationDate="2018-04-03T06:43:25.733" Score="4" Body="&lt;p&gt;Actually there is no fixed terminology and these two terms sometimes used in the same meaning and sometimes different. I would suggest following the terminology bellow for yourself, then you can differentiate methods according to this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://sclab.yonsei.ac.kr/courses/10TPR/10TPR.files/Gesture%20spotting%20with%20body-worn%20inertial%20sensors%20to%20detect%20user%20activities.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time-Series Segmentation&lt;/a&gt; means partitioning an individual time series to similar segments i.e. clustering within an individual time-series (e.g. i have a video in which someone is reading a book for a while, then starts walking and then starts cycling. now I want to segment these three actions).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Suggestion: State-Space reconstruction, moving Autocorrelation, moving DTW, Fourier Analysis, Visibility Graphs or any other method which can measure the similarity of a time-series with itself. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://perso.eleves.ens-rennes.fr/~xshan192/doc/time_series.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time-Series Clustering&lt;/a&gt; (or &lt;a href=&quot;https://pdfs.semanticscholar.org/20fa/a2ef4bb4e84b1d68750cda28d0a45fb16075.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;) means finding similar time-series within a dataset of time-series (e.g. i have 10 brain signals, 5 from healthy subjects 5 from patients without knowing who is patient and who is healthy. Now I want to cluster this dataset into two clusters)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Suggestion: Build a similarity matrix between time-series using e.g. DTW and then apply Spectral Clustering (just improvised. If you search literature there should be more mature solutions)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helped :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-04-03T06:43:25.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29837" PostTypeId="2" ParentId="29799" CreationDate="2018-04-03T07:19:28.163" Score="0" Body="&lt;p&gt;In order to overcome out of memory error during training, you can either reduce the batch size that would go in for training or if you have a large dataset you can use h5py matrix.&lt;/p&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-04-03T07:19:28.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29838" PostTypeId="2" ParentId="29796" CreationDate="2018-04-03T07:43:10.533" Score="1" Body="&lt;p&gt;Hover on the scoring header and use right click. This should open a window where you can select logloss as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/6Mntg.png&quot; alt=&quot;screenshot&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="23708" LastActivityDate="2018-04-03T07:43:10.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29840" PostTypeId="1" AcceptedAnswerId="29842" CreationDate="2018-04-03T07:55:17.413" Score="4" ViewCount="24842" Body="&lt;p&gt;My dataset has 2 columns:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Gender( that has 1 or 2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Rating1(That has 3 ratings (1,2,3))&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rating1: 1,2,3,3,2&#xA;&#xA;gender: 2,1,2,2,1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want output as:&lt;a href=&quot;https://i.stack.imgur.com/6QqIl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6QqIl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nutri.groupby(['Rating1','Gender']).nunique()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;but I get an output which looks wrong.&lt;/p&gt;&#xA;" OwnerUserId="49878" LastEditorUserId="49878" LastEditDate="2019-04-16T02:29:12.060" LastActivityDate="2019-04-16T02:29:12.060" Title="How to count grouped occurrences?" Tags="|python|pandas|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29841" PostTypeId="2" ParentId="25470" CreationDate="2018-04-03T08:02:25.367" Score="1" Body="&lt;p&gt;There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.&lt;/li&gt;&#xA;&lt;li&gt;The state updates as the model moved &quot;forward&quot; through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).&lt;/p&gt;&#xA;" OwnerUserId="49583" LastActivityDate="2018-04-03T08:02:25.367" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29842" PostTypeId="2" ParentId="29840" CreationDate="2018-04-03T08:40:16.747" Score="6" Body="&lt;p&gt;You can do that by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt;&amp;gt; m = pd.DataFrame({'gender': [1, 2, 2, 1, 1, 2, 1], 'rating': [3, 4, 2, 1, 3, 1, 5]})&#xA;&amp;gt;&amp;gt;&amp;gt; m.groupby(['rating','gender']).size().to_frame('count').reset_index()&#xA;   rating  gender  count&#xA;0       1       1      1&#xA;1       1       2      1&#xA;2       2       2      1&#xA;3       3       1      2&#xA;4       4       2      1&#xA;5       5       1      1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this is what you want to pull.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: &#xA;As mentioned, I did not account for zero values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to do one more step additional to get what you want. Finding the combinations that are missing and then joining it. The one liner solution would be&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from itertools import product&#xA;&amp;gt;&amp;gt;&amp;gt; m.groupby(['rating', 'gender']).size().to_frame('count').reset_index().merge(&#xA;        pd.DataFrame(list(set([i for i in product(*[m.gender, m.rating])])), columns=['gender', 'rating']),&#xA;        on=['rating', 'gender'],&#xA;        how='right').fillna(value=0)&#xA;   rating  gender  count&#xA;0       1       1    1.0&#xA;1       1       2    1.0&#xA;2       2       2    1.0&#xA;3       3       1    2.0&#xA;4       4       2    1.0&#xA;5       5       1    1.0&#xA;6       2       1    0.0&#xA;7       4       1    0.0&#xA;8       3       2    0.0&#xA;9       5       2    0.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Explanation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Get the original grouped counts as mentioned before the first edit but this time you need to join with the combinations that are missing to get the zero counts. Use &lt;code&gt;itertools.product&lt;/code&gt; to get all combinations of gender and rating and right join it with original grouped frame on &lt;code&gt;rating&lt;/code&gt; and &lt;code&gt;gender&lt;/code&gt; to get merged &lt;code&gt;DataFrame&lt;/code&gt; which has &lt;code&gt;numpy.na&lt;/code&gt; values if no count is present and then use &lt;code&gt;fillna&lt;/code&gt; method to fill it with zero. The only loophole is if a rating say &lt;code&gt;4&lt;/code&gt; is not there in the original data, there is no combination that follows it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="26394" LastEditDate="2018-04-03T12:26:04.503" LastActivityDate="2018-04-03T12:26:04.503" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29843" PostTypeId="1" AcceptedAnswerId="29847" CreationDate="2018-04-03T09:34:03.850" Score="1" ViewCount="6543" Body="&lt;p&gt;I have a problem which I explained in other &lt;a href=&quot;https://stats.stackexchange.com/questions/337439/clustering-customers-by-their-orders-sequence-patterns&quot;&gt;question&lt;/a&gt;. I've understood that my dataset is a sequence of states or something like that. Is there libraries to analyze sequence with python? And is it right way to use Hidden Markov Models to cluster sequences?&lt;/p&gt;&#xA;" OwnerUserId="46674" LastEditorUserId="28175" LastEditDate="2018-04-16T07:31:34.573" LastActivityDate="2018-04-16T07:31:34.573" Title="Are there any python libraries for sequences clustering?" Tags="|machine-learning|neural-network|deep-learning|lstm|sequence|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29844" PostTypeId="1" AcceptedAnswerId="29846" CreationDate="2018-04-03T10:24:43.827" Score="5" ViewCount="9177" Body="&lt;p&gt;Normally when I draw bar plot its simple as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;from pylab import rcParams&#xA;import seaborn as sb&#xA;&#xA;%matplotlib inline&#xA;rcParams['figure.figsize'] = 5, 4&#xA;sb.set_style('whitegrid')&#xA;&#xA;x = range(1, 10)&#xA;y = [1,2,3,4,0.5,4,3,2,1]&#xA;&#xA;plt.bar(x, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I aggregate data on basis of &lt;code&gt;age&lt;/code&gt; feature with the following command&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_ag = data.groupby('age')['age'].count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It returns the output that is the people belonging to a particular age e.g. 11 people with age 22 years.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;age&#xA;22     11&#xA;23      8&#xA;27     28&#xA;28      1&#xA;29     70&#xA;30     13&#xA;31     45&#xA;Name: age, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I treat those as &lt;code&gt;x&lt;/code&gt; and &lt;code&gt;y&lt;/code&gt; points to draw a bar plot?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = # what should I write here for age data&#xA;y = # what for count &#xA;&#xA;plt.bar(x, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49884" LastEditorUserId="24000" LastEditDate="2018-04-03T20:15:51.043" LastActivityDate="2018-04-03T20:15:51.043" Title="How can I draw bar graph in python on aggregated data?" Tags="|python|visualization|pandas|matplotlib|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29845" PostTypeId="2" ParentId="29844" CreationDate="2018-04-03T10:52:08.430" Score="5" Body="&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.Series(data=[11, 8, 28, 1, 70, 13, 45],&#xA;               index=[22, 23, 27, 28, 29, 30, 31],&#xA;               name=&quot;age&quot;).rename_axis(&quot;age&quot;, axis=0)&#xA;&#xA;x = df.index&#xA;y = df.values&#xA;&#xA;plt.bar(x, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mnWPn.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mnWPn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24000" LastActivityDate="2018-04-03T10:52:08.430" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29846" PostTypeId="2" ParentId="29844" CreationDate="2018-04-03T11:31:43.770" Score="2" Body="&lt;p&gt;I solved my issue using &lt;code&gt;size()&lt;/code&gt; and &lt;code&gt;reset_index()&lt;/code&gt; functions.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g1 = data.groupby( [&quot;age&quot;] ).size().reset_index(name='count')&#xA;&#xA;x= g1['count']&#xA;y=g1['age']&#xA;&#xA;plt.bar(x, y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49884" LastActivityDate="2018-04-03T11:31:43.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29847" PostTypeId="2" ParentId="29843" CreationDate="2018-04-03T11:50:15.793" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is there libraries to analyze sequence with python&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You can take a look at &lt;a href=&quot;http://biopython.org/DIST/docs/tutorial/Tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. You can also use &lt;code&gt;TensorFlow&lt;/code&gt; if your task is sequence classification, but based on comments you have referred that your task is unsupervised. Actually, &lt;code&gt;LSTMs&lt;/code&gt; can be used for unsupervised tasks too depending on what you want. Take a look at &lt;a href=&quot;https://scholar.google.com/scholar?q=lstm+for+clustering&amp;amp;hl=en&amp;amp;as_sdt=0&amp;amp;as_vis=1&amp;amp;oi=scholart&amp;amp;sa=X&amp;amp;ved=0ahUKEwiiuNmBhZ7aAhXRIlAKHV4xBlMQgQMIJDAA&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And is it right way to use Hidden Markov Models to cluster sequences?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Markov hidden models are those that your current state does not depend on all previous states. If you your task has longterm dependencies, you can use &lt;code&gt;LSTM&lt;/code&gt; networks. If your data does not have longterm dependencies you can use simple &lt;code&gt;RNNs&lt;/code&gt;. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-03T11:50:15.793" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29848" PostTypeId="1" AcceptedAnswerId="29948" CreationDate="2018-04-03T13:23:32.610" Score="5" ViewCount="2175" Body="&lt;p&gt;I am using 3-fold cross validation and a grid search of the C and gamma parameters for a SVC using the RBF kernel I have achieved a classification score of 84%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When testing against live data the accuracy rate is 70% (1500 samples used). However, when testing against an un-seen hold out set the accuracy is 86% (8800 samples, 20% of the original dataset). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The training and holdout data set have even distribution of the 3 classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be the cause of this large discrepancy? It does not seem to be over fitting judging by the performance of the model with the hold out set?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How did you split the data set?&lt;/strong&gt; &#xA;The data was originally in sequential order. I wrote a script to randomly split each sample between the train and hold out set, making use of a CSPRNG. Then at the end a report was automatically generated to display the distribution of each class in each set. The distribution very near equal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How did you do the grid search?&lt;/strong&gt; &#xA;Through the SKlearn SVC grid search method (&lt;code&gt;GridSearchCV&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is there any overlap between the data points used during grid search and the un-seen hold out set?&lt;/strong&gt; &#xA;No overlap, they are all from unique time stamps in the initial set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Does the live data come from the same distribution as the other?&lt;/strong&gt; &#xA;Yes the live data comes from the same source and the distribution is roughly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do you know?&lt;/strong&gt; &#xA;I have a script to count up the occurrences of each class in the data set.&lt;/p&gt;&#xA;" OwnerUserId="47570" LastEditorUserId="47570" LastEditDate="2018-04-05T15:15:50.117" LastActivityDate="2018-10-19T18:00:00.237" Title="Detecting over fitting of SVM/SVC" Tags="|svm|overfitting|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="29849" PostTypeId="1" AcceptedAnswerId="29883" CreationDate="2018-04-03T13:59:21.020" Score="0" ViewCount="294" Body="&lt;p&gt;I'm using Sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;KNN&lt;/a&gt; to build a classifier and was wondering if there is any way to visualize the decision tree that the algorithm builds. Maybe something of this fashion&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oQsYg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oQsYg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40123" LastActivityDate="2020-04-26T16:13:53.317" Title="Visualizing Decision Tree of K-Nearest-Neighbours classifier" Tags="|scikit-learn|visualization|k-nn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29851" PostTypeId="1" AcceptedAnswerId="29853" CreationDate="2018-04-03T14:13:28.643" Score="16" ViewCount="31463" Body="&lt;p&gt;A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values)&lt;/p&gt;&#xA;&lt;p&gt;The usual data science approach would be to perform a One-Hot Encoding.&#xA;However, wouldn't it be a bit extreme to perform some One-Hot Encoding with a dictionary quite large (+/- 300 values)? Is there any best practice on when to choose Embedding vectors or One-Hot Encoding?&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Additional, information: how would you handle the previous case if new values can be added to the dictionary. Re-training seems the only solution, however with One-Hot Encoding, the data dimension will simultaniously increase which may lead to additional troubles, embedding vectors, on the opposite side, can keep the same dimension even if new values appears.&lt;/p&gt;&#xA;&lt;p&gt;How would you handle such a case ? Embedding vectors clearly seem more appropriate to me, however I would like to validate my opinion and check if there is another solution that could be more apporiate.&lt;/p&gt;&#xA;" OwnerUserId="39198" LastEditorUserId="43000" LastEditDate="2021-05-21T10:37:39.820" LastActivityDate="2021-05-21T10:38:44.810" Title="One Hot Encoding vs Word Embedding - When to choose one or another?" Tags="|preprocessing|word-embeddings|embeddings|encoding|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29853" PostTypeId="2" ParentId="29851" CreationDate="2018-04-03T14:47:19.587" Score="28" Body="&lt;p&gt;One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the &amp;quot;curse of dimensionality&amp;quot; by creating a new dimension for each category.&lt;/p&gt;&#xA;&lt;p&gt;Embedding is a method that requires large amounts, both in the total amount of data and repeated occurrences of individual exemplars, and long training time. The result is a dense vector with a fixed, arbitrary number of dimensions.&lt;/p&gt;&#xA;&lt;p&gt;They also differ at the prediction stage: a One-Hot Encoding tells you nothing of the semantics of the items; each vectorization is an orthogonal representation in another dimension. Embeddings will group commonly co-occurring items together in the representation space.&lt;/p&gt;&#xA;&lt;p&gt;If you have enough training data, enough training time, and the ability to apply the more complex training algorithm (e.g., word2vec or GloVe), go with Embeddings. Otherwise, fall back to One-Hot Encoding.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="43000" LastEditDate="2021-05-21T10:38:44.810" LastActivityDate="2021-05-21T10:38:44.810" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="29854" PostTypeId="2" ParentId="29851" CreationDate="2018-04-03T14:47:44.153" Score="6" Body="&lt;p&gt;It seems that Embedding vector is the best solution here. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick&quot;.&#xA;In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One advantage in your use case is that you may perform online encoding. If you have not encountered every vocabulary words yet, you may still assign a hash. Then later, new words may be added to the vocabulary. One pitfall though is &quot;hash collisions&quot;. Indeed there is a probability that two different words end up with the same hash.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found an example of this hashing trick in the excellent &lt;a href=&quot;https://www.manning.com/books/deep-learning-with-python&quot; rel=&quot;noreferrer&quot;&gt;&quot;Deep learning with Python&quot; from François Chollet&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;samples = ['The cat sat on the mat.', 'The dog ate my homework.']&#xA;dimensionality = 1000&#xA;max_length = 10&#xA;&#xA;results = np.zeros((len(samples), max_length, dimensionality))&#xA;for i, sample in enumerate(samples):&#xA;    for j, word in list(enumerate(sample.split()))[:max_length]:&#xA;        # the word is given a hash with a random integer&#xA;        index = abs(hash(word)) % dimensionality&#xA;        results[i, j, index] = 1.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The resulting array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;results.shape&#xA;(2, 10, 1000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can observe that common words between the two sentences are given the same index: &lt;code&gt;The&lt;/code&gt; in position 0 of the two sentences (positions 0 and 6 of dimension 2).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.where(results &amp;gt; 0)&#xA;(array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1], dtype=int64),&#xA; array([0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4], dtype=int64),&#xA; array([ 23,  58, 467, 442,  83,  77,  23, 798, 618, 301, 942], dtype=int64))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12490" LastEditorUserId="12490" LastEditDate="2018-04-03T18:47:07.413" LastActivityDate="2018-04-03T18:47:07.413" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29857" PostTypeId="1" CreationDate="2018-04-03T15:18:33.350" Score="5" ViewCount="3836" Body="&lt;p&gt;I have the following sentence&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;query = u'tell me about people in konoha who have wind style chakra and are above jonin level'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to split the above sentence into the following three parts&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA; &quot;tell me about people in konoha&quot;,&#xA; &quot;who have wind style chakra&quot;,&#xA; &quot;and are above jonin level&quot;&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have looked into sent tokenizers in &lt;strong&gt;spacy&lt;/strong&gt; and &lt;strong&gt;nltk&lt;/strong&gt;, but they are not giving me the desired results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am just a beginner in nlp and machine learning and have very limited knowledge so far. It would be awesome if you could direct me to some techniques or available packages through which I could achieve the above results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'll be having question-like queries like the one above where I would be asking for details about the subject based on some constraints&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the above query the subject was &lt;strong&gt;people in Konoha&lt;/strong&gt; and the constraints are &lt;strong&gt;have wind style chakra&lt;/strong&gt; and &lt;strong&gt;above jonin level&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to extract such relationships from similar queries.&lt;/p&gt;&#xA;" OwnerUserId="49899" LastEditorUserId="49899" LastEditDate="2018-04-03T16:05:26.380" LastActivityDate="2018-04-04T19:06:24.970" Title="Splitting a sentence to meaningful parts" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="6" ClosedDate="2018-04-06T01:09:14.033" ContentLicense="CC BY-SA 3.0" />
  <row Id="29858" PostTypeId="1" AcceptedAnswerId="29915" CreationDate="2018-04-03T15:36:39.910" Score="4" ViewCount="601" Body="&lt;p&gt;The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the general predictive power of x over y when the relationship between x and y is not linear?&lt;/p&gt;&#xA;" OwnerUserId="49893" LastActivityDate="2018-04-04T21:59:45.157" Title="Generalization of Correlation Coefficient" Tags="|predictive-modeling|statistics|correlation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29862" PostTypeId="1" AcceptedAnswerId="30169" CreationDate="2018-04-03T16:27:34.043" Score="4" ViewCount="1205" Body="&lt;p&gt;I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was to use a neural network (e.g. adding multiple outputs to a pre-trained covnet) and I would try keras first (having used it before). I assume it would be useful to try to do this all in a single covnet (especially based on a TWiML talk on &lt;a href=&quot;http://dx.doi.org/10.1038/s41551-018-0195-0&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, where the author suggested that in general one prediction target will help to improve features for another and given that e.g. how one would recognise gender will differ by breed etc.).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question arises, because for some training/validation/test images some (but not all) of the multiple prediction targets may be missing. Sometimes we will just know the breed, for some images we will not know the gender and the exact age may oftne be unavailable (but only that the animal is adult versus younger than about 4 months, which a human can easily label by just looking at the animal).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to guess, I would expect the missingess to be missing at random (but likely not missing completely at random). For example, for some breeds and ages it is harder to tell the gender by just looking at the animal (perhaps the neural network will manage to do it). So, someone who just takes a photo of an animal may not know the gender and humans are no good at labelling gender from photos. Another example: age might be unavailable, because someone just took a photo of an animal they do not have detailed information on or because someone bought an animal secondhand and never found out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some standard way of handling this? My intuition is that even records with one or two missing prediction target still should provide useful information for the other ones. Thus, listwise deletion of all records with any missing target information would be extremely inefficient (and possibly even otherwise problematic?!).&lt;/p&gt;&#xA;" OwnerUserId="44389" LastEditorUserId="44389" LastEditDate="2018-04-03T17:28:42.403" LastActivityDate="2018-04-11T12:37:34.663" Title="Missing outputs in multiple-output neural net" Tags="|neural-network|keras|cnn|missing-data|multitask-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29863" PostTypeId="1" CreationDate="2018-04-03T16:58:45.107" Score="9" ViewCount="5647" Body="&lt;p&gt;I am learning Tensorflow and came across different &lt;a href=&quot;https://www.tensorflow.org/guide/feature_columns&quot; rel=&quot;noreferrer&quot;&gt;feature columns&lt;/a&gt; used in Tensorflow . Out of these types, two are &lt;a href=&quot;https://www.tensorflow.org/guide/feature_columns#categorical_identity_column&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;categorical_identity_column&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;https://www.tensorflow.org/guide/feature_columns#indicator_and_embedding_columns&quot; rel=&quot;noreferrer&quot;&gt;&lt;code&gt;indicator_column&lt;/code&gt;&lt;/a&gt;. Both have been defined in the same way. As far as I understand, both convert categorical column to one-hot encoded column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is what is the difference between the two? When to use one and when to use the other?&lt;/p&gt;&#xA;" OwnerUserId="44139" LastEditorUserId="29575" LastEditDate="2018-07-21T13:09:28.880" LastActivityDate="2020-08-02T12:23:40.057" Title="Difference between indicator column and categorical identity column in tensorflow" Tags="|machine-learning|tensorflow|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29864" PostTypeId="2" ParentId="13385" CreationDate="2018-04-03T18:34:49.957" Score="0" Body="&lt;p&gt;I provide you some steps to perform this task:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;extract plain content from the news, for example using &lt;a href=&quot;https://github.com/seomoz/dragnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;dragnet&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;tokenize each text and represent them with vectors with the bag of words technique. A simple way to perform this is using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; from sklearn.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clusterize them using some classification technique like &lt;a href=&quot;http://scikit-learn.org/stable/modules/neighbors.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-NN&lt;/a&gt;(k nearest neighbors). You will find the k-NN sklearn implementation very helpfull.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The key to perform the task is using the &lt;strong&gt;TfidfVectorizer&lt;/strong&gt; which weigths more the tokens that only appear in a few notices, like &quot;Apple&quot; and &quot;Tim Cook&quot;. Using a simple count vectorizer will not be usefull.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="29575" LastEditDate="2018-04-03T19:00:18.437" LastActivityDate="2018-04-03T19:00:18.437" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29865" PostTypeId="2" ParentId="12954" CreationDate="2018-04-03T18:45:01.050" Score="0" Body="&lt;p&gt;A binary classification between &lt;em&gt;business&lt;/em&gt; and &lt;em&gt;non-business&lt;/em&gt; is very simple:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Extract plain content from the news, for example using &lt;a href=&quot;https://github.com/seomoz/dragnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;dragnet&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Tokenize each text and represent them with vectors with the bag of words technique. A simple way to perform this is using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; from sklearn.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clusterize them using some classification technique like &lt;a href=&quot;http://scikit-learn.org/stable/modules/neighbors.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-NN&lt;/a&gt;(k nearest neighbors). You will find the k-NN sklearn implementation very helpfull.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;To perform a multilabel classification like &lt;em&gt;companies related news&lt;/em&gt;, you have to use &lt;strong&gt;TfidfVectorizer&lt;/strong&gt; which weights more the rare words that appear only in few news, like &lt;em&gt;Apple&lt;/em&gt; or &lt;em&gt;Coca-Cola&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="29575" LastEditDate="2018-04-03T19:30:34.007" LastActivityDate="2018-04-03T19:30:34.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29866" PostTypeId="2" ParentId="29600" CreationDate="2018-04-03T18:48:37.227" Score="0" Body="&lt;p&gt;I am working in the same topic right now. I am using the following algorithm:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) extract plain content from the news, for example using &lt;a href=&quot;https://github.com/seomoz/dragnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;dragnet&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) tokenize each text and represent them with vectors with the bag of words technique. A simple way to perform this is using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; from sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Clusterize them using some classification technique like &lt;a href=&quot;http://scikit-learn.org/stable/modules/neighbors.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-NN&lt;/a&gt;(k nearest neighbors). You will find the k-NN sklearn implementation very helpfull.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key to perform the task is using the &lt;strong&gt;TfidfVectorizer&lt;/strong&gt; which weigths more the tokens that only appear in a few notices, and so I can recognize similar news that talk about same topics.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-03T18:48:37.227" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29867" PostTypeId="2" ParentId="29823" CreationDate="2018-04-03T18:56:09.833" Score="0" Body="&lt;p&gt;You can use the Latent Class Model (see Goodman, L., 1974. Exploratory latent structure analysis using both identifiable and unidentifiable&#xA;models. Biometrika 61 (2), 215–231). This approach achieves the clustering purpose by a mixture of multinomial distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to know which foods are the best separators for the clustering, you can check the parameters of the multinomial distributions for each component. Moreover, you can perform a variable selection simultanously with model estimation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach (with and without variable selection) is implemented in the R package &lt;code&gt;VarSelLCM&lt;/code&gt;. Here is an example (considering that your data are stored in object &quot;x&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(VarSelLCM)&#xA;&#xA;x &amp;lt;- as.data.frame(x)&#xA;for (j in 1:ncol(x)) x[,j] &amp;lt;- as.factor(x[,j])&#xA;&#xA;# Clustering with variable selection and a number of cluster between 1 and 4&#xA;# Model selection is BIC (to use MICL, the option must be specified)&#xA;out &amp;lt;- VarSelCluster(x, 1:4, nbcores = 2)&#xA;&#xA;# partition&#xA;out@partition@zMAP&#xA;&#xA;# summary&#xA;summary(out)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49625" LastActivityDate="2018-04-03T18:56:09.833" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29869" PostTypeId="1" CreationDate="2018-04-03T19:12:19.210" Score="5" ViewCount="136" Body="&lt;p&gt;I am performing an online news classification. The idea is to recognize group of news of the same topic.&#xA;My algorithm has these steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I go through a group of feeds from news sites and I recognize news links.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) For each new link, I extract the content using &lt;a href=&quot;https://github.com/seomoz/dragnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;dragnet&lt;/a&gt;, and then I tokenize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) I find the vector representation of all the &lt;em&gt;old news&lt;/em&gt; and the &lt;em&gt;last one&lt;/em&gt; using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;TfidfVectorizer&lt;/a&gt; from sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) I find the nearest neighbor in my dataset computing euclidean distance from the &lt;em&gt;last news&lt;/em&gt; vector representation and all the vector representations of the &lt;em&gt;old news&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a problem using TfidfVectorizer because it weights more the special words that only appear in a few news, like &lt;strong&gt;Apple&lt;/strong&gt;, and news that talk about &lt;strong&gt;Aple&lt;/strong&gt; are grouped together even when they deal with different topics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Is there a common approach more efficient than the one I am using?&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-04-04T19:27:49.470" LastActivityDate="2018-06-18T15:34:08.087" Title="Online news classification" Tags="|classification|scikit-learn|text-mining|tfidf|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29870" PostTypeId="1" CreationDate="2018-04-03T19:37:36.817" Score="7" ViewCount="29764" Body="&lt;p&gt;I am having a little trouble understanding the difference between what a &quot;Node&quot; of a tree and a &quot;Leaf&quot; of a tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I am trying to decide the size of coffee a person may like. There are three categories: small, medium, and large based off the peoples age, height, weight, income.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have four predictors and 3 possible outcomes. When looking at many gradient boosting algorithms, there are parameters that can increase the number of leaves. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of this (correct me if wrong), but I will illustrate with a picture. Assuming each yes/no split is 50/50&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/EcrpP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EcrpP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does increasing the number of leaves to lets say 3 leaves change it from yes/no aka 50/50 to 33/33/33? This is a little confusing to me. Thank you for any clarification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sam&lt;/p&gt;&#xA;" OwnerUserId="49916" LastActivityDate="2018-10-11T13:55:44.167" Title="Decision Trees Nodes vs Leaves Definition" Tags="|machine-learning|decision-trees|gbm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29873" PostTypeId="2" ParentId="29816" CreationDate="2018-04-03T21:17:10.557" Score="2" Body="&lt;p&gt;I think that the field has moved on from that paper. There is a trend to use the data from the expert to either &quot;precondition&quot; the agent or extracting a policy directly from the data. You can search for imitation learning or behavioral cloning. Some of these algorithms: &lt;a href=&quot;https://arxiv.org/pdf/1606.03476.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Learning&lt;/a&gt;, &lt;a href=&quot;https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;DAGGER&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/pdf/1704.03732.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Q-learning from Demonstrations &lt;/a&gt;. AlphaGo used Supervised Learning as well to get to a good policy before getting trained in a RL setting. So instead of trying to recover a complicated reward function from the data you can use the above methods to get a good policy or initialize the parameters of your agent to more promising directions. Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-04-03T21:17:10.557" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="29874" PostTypeId="2" ParentId="29863" CreationDate="2018-04-03T22:20:32.763" Score="1" Body="&lt;p&gt;You would use &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical_column_with_*&lt;/a&gt; to get a &lt;code&gt;_CategoricalColumn&lt;/code&gt; to feed into a linear model; this column returns identity values, often using a vocabulary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column&quot; rel=&quot;nofollow noreferrer&quot;&gt;indicator_column&lt;/a&gt; is a multi-hot representation of a given categorical column and would be used if you want to feed the feature into a DNN, for example; it produces an &lt;code&gt;_IndicatorColumn&lt;/code&gt;. &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column&quot; rel=&quot;nofollow noreferrer&quot;&gt;embedding_column&lt;/a&gt; is analogous but you'd use it if your input is sparse.&lt;/p&gt;&#xA;" OwnerUserId="49923" LastActivityDate="2018-04-03T22:20:32.763" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29875" PostTypeId="2" ParentId="12919" CreationDate="2018-04-03T22:50:49.693" Score="6" Body="&lt;p&gt;Ryan Zotti offers a good answer, but this is changing. With the addition of &lt;a href=&quot;https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/tensor_forest&quot; rel=&quot;noreferrer&quot;&gt;Random Forest&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/abs/1710.11555&quot; rel=&quot;noreferrer&quot;&gt;Gradient Boosting&lt;/a&gt;, and Bayesian methods to &lt;code&gt;TensorFlow&lt;/code&gt;, it is headed in the direction of becoming a one-stop solution. More traditional algorithms are listed &lt;a href=&quot;https://stackoverflow.com/questions/43030393/is-tensorflow-only-limited-to-neural-networks/43031536#43031536&quot;&gt;here&lt;/a&gt;. TensorFlow has particular promise, as it is designed to scale well and supports GPU operations. However, &lt;code&gt;scikit learn&lt;/code&gt; is the traditional one-stop shop where you can find many standard algorithms. They usually aren't the latest and greatest, so you will likely want specialized libraries as well.&lt;/p&gt;&#xA;" OwnerUserId="49925" LastEditorUserId="49925" LastEditDate="2018-04-03T22:59:42.470" LastActivityDate="2018-04-03T22:59:42.470" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29877" PostTypeId="2" ParentId="28025" CreationDate="2018-04-03T22:58:44.773" Score="3" Body="&lt;p&gt;Similar to what I wrote on the &lt;a href=&quot;https://datascience.stackexchange.com/questions/12919/is-tensorflow-a-complete-machine-learning-library/12921#12921&quot;&gt;other post&lt;/a&gt;, TensorFlow does in fact have implementations of Random Forest and Gradient Boosting, in addition to other non-deep learning algorithms. The links can be found at that post.&lt;/p&gt;&#xA;" OwnerUserId="49925" LastActivityDate="2018-04-03T22:58:44.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29879" PostTypeId="1" AcceptedAnswerId="29888" CreationDate="2018-04-04T00:31:09.753" Score="6" ViewCount="1880" Body="&lt;p&gt;I have a categorical variable which has thousands of values, for a dataset which has millions of records. The data is being used to create a binary classification model. I am in the early steps of feature selection, but I am trying out Random Forest, Boosted Trees, and Logistic Regression to see what works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I find the frequency of each category and sort that, I see that about 50 values make up the top 80%. Is it valid to condense this feature as a binary on whether the value is in that set of values or not. By 'valid', I mean is it likely that this sort of transformation retain any useful information for a model? I have a concern that sorting these categorical values which do not have any order to them creates some incorrect assumptions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The frequecy distribution looks a little like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A;10%&#xA;D;5%&#xA;E;1.2%&#xA;B;1.1%&#xA;...&#xA;Z;0.004%&#xA;W;0.0037%&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Going one step further, is it valid to profile each class in my dataset and do the same? Say Categories A-F comprise the top 80% of class 0 and Categories D-H are the top 80% of class 1. I would convert:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_id;cat_var&#xA;1;B&#xA;2;F&#xA;3;H&#xA;4;Z&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_id;cat_var_top80class0;cat_var_top80class1&#xA;1;1;0&#xA;2;1;1&#xA;3;0;1&#xA;4;0;0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Adding picture to hopefully clear up this idea. In yellow are the pre-calculated distributions of cat_var (***_id in the picture) for classes 0 and 1 based on the training set. On the right shows how the transformation would be applied:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4QVwg.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4QVwg.png&quot; alt=&quot;sorted ***_id distribution by class&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37776" LastEditorUserId="37776" LastEditDate="2018-04-06T03:19:36.570" LastActivityDate="2018-04-06T03:19:36.570" Title="Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?" Tags="|machine-learning|data-mining|feature-selection|feature-engineering|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29880" PostTypeId="2" ParentId="29870" CreationDate="2018-04-04T01:27:53.290" Score="6" Body="&lt;p&gt;Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In your example tree diagram, the nodes that say 'Large', 'Medium' or 'Small' are leaf nodes. The other nodes in the tree are interchangeably called &lt;em&gt;split nodes&lt;/em&gt;, &lt;em&gt;decision nodes&lt;/em&gt; or &lt;em&gt;internal nodes&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In gradient boosting algorithms, a number of decision trees are grown. Each tree is grown until some stopping criteria is met. One kind of stopping criteria is the maximum number of leaves in the tree. At each stage of growing a decision tree, a leaf node is turned into a split node by creating a yes/no question (we call this a &lt;em&gt;binary split&lt;/em&gt;), and two new leaf nodes are created which correspond to each side of the split. Once the total number of leaf nodes in the tree reaches the limiting value, the tree building algorithm stops and begins building the next tree.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-04T01:27:53.290" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29881" PostTypeId="1" CreationDate="2018-04-04T02:28:11.777" Score="2" ViewCount="95" Body="&lt;p&gt;I have a time series data containing hotel occupancy rates from January 2014 to December 2017. I have run a forecasting model (Holtswinter in R). Now, I want to predict the significant external factors driving this occupancy rate. I have considered weather data (Temperature, Visibility etc.,) and Google Trends data (based on certain key words for trend tracking) as Independent variables and Occupancy rate as Dependent variable. This is clearly a non-linear regression model. What is the best algorithm to predict the significant variables impacting occupancy rate? Please refer to the sample data in the image.&lt;a href=&quot;https://i.stack.imgur.com/fm5Ba.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fm5Ba.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any insights on how to proceed with the model and/or any code snippets in R is/are surely appreciated. Thank you!&lt;/p&gt;&#xA;" OwnerUserId="49922" LastActivityDate="2018-04-04T02:28:11.777" Title="How to build a predictive model on Time series data" Tags="|r|predictive-modeling|time-series|regression|visualization|" AnswerCount="0" CommentCount="0" ClosedDate="2018-04-05T21:04:53.253" ContentLicense="CC BY-SA 3.0" />
  <row Id="29883" PostTypeId="2" ParentId="29849" CreationDate="2018-04-04T03:13:25.770" Score="1" Body="&lt;p&gt;$k$-NN does not build a decision tree to classify a new instance, it looks at the class of the most similar examples (the nearest neighbours) in the training set. So, in short, no you cannot get a decision tree from $k$-NN. You can build a decision tree for your dataset directly by using scikit-learn's &lt;code&gt;DecisionTreeClassifier&lt;/code&gt; instead if you need a decision tree.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-04T03:13:25.770" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29884" PostTypeId="2" ParentId="29869" CreationDate="2018-04-04T05:14:41.967" Score="2" Body="&lt;p&gt;The usual rule is to ask only one question per post.  I will answer your first question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Build a dictionary, once, in advance.  With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, and it's likely to contain all the words you need; the likelihood that you're missing some important word is pretty low.  Now, each time you see an article, throw away any words not in the dictionary.  That way, you won't need to vectorize the old news over and over every time; you can do it once and be done.  That should improve performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another variation: if you see a new word on some day, you can add that word to the dictionary; at the end of the day retokenize everything you saw that day.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2018-04-04T19:50:22.950" LastActivityDate="2018-04-04T19:50:22.950" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29885" PostTypeId="1" AcceptedAnswerId="29894" CreationDate="2018-04-04T07:24:37.020" Score="4" ViewCount="424" Body="&lt;p&gt;I am having a hard time trying to understand the MSE loss function given in the &lt;a href=&quot;https://xgboost.readthedocs.io/en/latest/model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Boosted Trees&lt;/a&gt; (beware! My maths skills are the equivalent of a very sparse matrix):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$&#xA;\begin{split}\text{obj}^{(t)} &amp;amp; = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\&#xA;          &amp;amp; = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + constant&#xA;\end{split}&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second equality sign implies that one could easily derive the second equation from the first one, but I cannot see how.&#xA;My first naïve attempt was to:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;express $y_i$ as $a$&lt;/li&gt;&#xA;&lt;li&gt;express $(\hat{y}_i^{(t-1)} + f_t(x_i))$ as $b$&lt;/li&gt;&#xA;&lt;li&gt;and then expand $(a-b)^2$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I wasn't successful. Any help is really appreciated. &lt;/p&gt;&#xA;" OwnerUserId="49937" LastActivityDate="2018-04-04T11:42:22.480" Title="XGBoost equations (for dummies)" Tags="|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29886" PostTypeId="1" CreationDate="2018-04-04T07:27:36.287" Score="2" ViewCount="84" Body="&lt;p&gt;I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am interested in knowing if there are any per-existing tools to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am happy to provide examples if needed :) &lt;/p&gt;&#xA;" OwnerUserId="45262" LastActivityDate="2024-01-16T04:02:45.273" Title="Extract details from bibliometrics data" Tags="|data-mining|nlp|text-mining|social-network-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29888" PostTypeId="2" ParentId="29879" CreationDate="2018-04-04T07:43:05.010" Score="4" Body="&lt;p&gt;&lt;em&gt;(Edited after @D.W. suggestion).&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To the best of my knowledge, there is nothing wrong with what you have in mind; thus it is certainly valid.. As you said, you have to try out all possible ways you can think and see which one works better. The most important point adopting a particular encoding is to be aware of you have certain amount of information getting lost and it varies one case to another depending on the problem at hand. For example in the case of binary coding based on high/low frequency sublevels, there will large of information (details) lost which could help the algorithm to do classification. I liked your idea of percentile coding based on cumulative density distribution. Maybe you want to look at Quantile-based discretization which is available in &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.qcut.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas.qcut&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The rest is my earlier answer as it was (below). I intended to suggest trying other techniques as well on top of what you have had in mind; but apparently the message was not clear. Please note that I do not seek to get my answer marked as final answer, as I aware that still it does not fully answer your question; it is simply to brainstorm and exchange ideas in length. ;-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps you have already digged out enough about ways to convert categorical variables into continuous data. In case you did not and missed checking this &lt;a href=&quot;https://datascience.stackexchange.com/questions/27012/cleaning-input-data-with-pd-get-dummies/27014#27014&quot;&gt;answer&lt;/a&gt;, check it out. As discussed, they are many ways to convert cat-to-num, and your problem is one of the hardest yet the most common across many domains. You have a high cardinality in your categorical variable, and as far as I understood imbalance distributions of those sublevels and you are not sure whether messing up the order of those sublevels matters or not. You may need to try Ordinal Encoding (if order really matters) or weight of evidence (WoE) transformation (see this &lt;a href=&quot;https://pkghosh.wordpress.com/2017/10/09/combating-high-cardinality-features-in-supervised-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; for instance) that I have heard but not tried or even going beyond mixing them in a meaningful way to represent your categorical data properly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have learned, despite all efforts in the field, that this problem is still an open challenge in data science and machine learning. Thus there is no best solution, or well-established method as far as I checked. Please do let me know if you come across one.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-04-05T08:11:20.867" LastActivityDate="2018-04-05T08:11:20.867" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29889" PostTypeId="2" ParentId="29886" CreationDate="2018-04-04T08:36:41.780" Score="0" Body="&lt;p&gt;To learn from traditional software in that domain you could start at &lt;a href=&quot;https://en.wikipedia.org/wiki/Comparison_of_reference_management_software&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Comparison_of_reference_management_software&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="21163" LastActivityDate="2018-04-04T08:36:41.780" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29890" PostTypeId="1" CreationDate="2018-04-04T08:45:24.617" Score="1" ViewCount="206" Body="&lt;p&gt;I have an imbalanced classification problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first partitioned my data into a training set (Dataset A) and a test set (Dataset B).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then used the R package &lt;code&gt;ROSE&lt;/code&gt; to simultaneously undersample the majority class in Dataset A and oversample the minority class. This produced a balanced set (Dataset C) which has as many rows as Dataset A.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have trained my model on Dataset C and computed &lt;em&gt;training&lt;/em&gt; AUC. I now want to estimate (i.e., not using Dataset B) the &lt;em&gt;test&lt;/em&gt; AUC using k-fold cross-validation. How should I best do this?&lt;/p&gt;&#xA;" OwnerUserId="49941" LastEditorUserId="24000" LastEditDate="2018-04-04T15:00:08.860" LastActivityDate="2019-09-30T16:02:30.353" Title="Estimating test AUC using k-fold CV for imbalanced classification problem" Tags="|machine-learning|classification|r|cross-validation|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29892" PostTypeId="2" ParentId="29542" CreationDate="2018-04-04T08:58:22.917" Score="2" Body="&lt;p&gt;This would actually be more suitable as a comment, but I am lacking the reputation to do so. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your background, I recommend this page&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.python-course.eu/python3_course.php&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.python-course.eu/python3_course.php&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's a good introduction to Python and it also contains a part about NumPy (and I would highly recommend to get more familiar with those before doing the ML part). I would not recommend the part about machine learning because it teaches you how to program the models, but usually your job in industry is to use whatever is there, not to write things yourself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the &quot;normal&quot; machine learning the scikit-learn homepage gives good examples. This leaves deep-learning, for which I cannot recommend anything. &lt;/p&gt;&#xA;" OwnerUserId="43469" LastActivityDate="2018-04-04T08:58:22.917" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29893" PostTypeId="1" AcceptedAnswerId="29896" CreationDate="2018-04-04T09:02:27.417" Score="17" ViewCount="69272" Body="&lt;p&gt;I'm building a sentiment analysis program in python using Keras Sequential model for deep learning&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my data is 20,000 tweets:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;positive tweets: 9152 tweets&lt;/li&gt;&#xA;&lt;li&gt;negative tweets: 10849 tweets&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I wrote a sequential model script to make the binary classification as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=Sequential()&#xA;model.add(Embedding(vocab_size, 100, input_length=max_words))&#xA;model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))&#xA;model.add(MaxPooling1D(pool_size=2))&#xA;model.add(Flatten())&#xA;model.add(Dense(250, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;# Fit the model&#xA;&#xA;print(model.summary())&#xA;history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;however I get very strange results! The model accuracy is almost perfect (&gt;90)&#xA;whereas the validation accuracy is very low (&amp;lt;1) (shown bellow)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train on 9417 samples, validate on 4036 samples&#xA; Epoch 1/2&#xA;- 13s - loss: 0.5478 - acc: 0.7133 - val_loss: 3.6157 - val_acc: 0.0243&#xA; Epoch 2/2&#xA;- 11s - loss: 0.2287 - acc: 0.8995 - val_loss: 5.4746 - val_acc: 0.0339&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried to increase the number of epoch, and it only increases the model accuracy and lowers the validation accuracy&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any advice on how to overcome this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is how I handle my data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#read training data&#xA;pos_file=open('pos2.txt', 'r', encoding=&quot;Latin-1&quot;)&#xA; neg_file=open('neg3.txt', 'r', encoding=&quot;Latin-1&quot;)&#xA;# Load data from files&#xA;pos = list(pos_file.readlines())&#xA;neg = list(neg_file.readlines()) &#xA;x = pos + neg&#xA;docs = numpy.array(x)&#xA;#read Testing Data&#xA;pos_test=open('posTest2.txt', 'r',encoding=&quot;Latin-1&quot;)&#xA;posT = list(pos_test.readlines())&#xA;neg_test=open('negTest2.txt', 'r',encoding=&quot;Latin-1&quot;)&#xA;negT = list(neg_test.readlines())&#xA;xTest = posT + negT&#xA;total2 = numpy.array(xTest)&#xA;&#xA;CombinedDocs=numpy.append(total2,docs)&#xA;&#xA;# Generate labels&#xA;positive_labels = [1 for _ in pos]&#xA;negative_labels = [0 for _ in neg]&#xA;labels = numpy.concatenate([positive_labels, negative_labels], 0)&#xA;&#xA;# prepare tokenizer&#xA;t = Tokenizer()&#xA;t.fit_on_texts(CombinedDocs)&#xA;vocab_size = len(t.word_index) + 1&#xA;# integer encode the documents&#xA;encoded_docs = t.texts_to_sequences(docs)&#xA;#print(encoded_docs)&#xA;&#xA;# pad documents to a max length of 140 words&#xA;max_length = 140&#xA;padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here I used Google public word2vec&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# load the whole embedding into memory&#xA;embeddings_index = dict()&#xA;f = open('Google28.bin',encoding=&quot;latin-1&quot;)&#xA;for line in f:&#xA;values = line.split()&#xA;word = values[0]&#xA;coefs = asarray(values[1:], dtype='str')&#xA;embeddings_index[word] = coefs&#xA;f.close()&#xA;&#xA;# create a weight matrix for words in training docs&#xA;embedding_matrix = zeros((vocab_size, 100))&#xA;&#xA;for word, i in t.word_index.items():&#xA;embedding_vector = embeddings_index.get(word)&#xA;if embedding_vector is not None:&#xA;    embedding_matrix[i] = embedding_vector&#xA;&#xA;&#xA;#Convert to numpy&#xA;NewTraining=numpy.array(padded_docs)&#xA;NewLabels=numpy.array(labels)&#xA;encoded_docs2 = t.texts_to_sequences(total2)&#xA;&#xA;# pad documents to a max length of 140 words&#xA;&#xA;padded_docs2 = pad_sequences(encoded_docs2, maxlen=max_length, padding='post')&#xA;&#xA;&#xA;# Generate labels&#xA;positive_labels2 = [1 for _ in posT]&#xA;negative_labels2 = [0 for _ in negT]&#xA;yTest = numpy.concatenate([positive_labels2, negative_labels2], 0)&#xA;NewTesting=numpy.array(padded_docs2)&#xA;NewLabelsTsting=numpy.array(yTest)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49944" LastEditorUserId="49944" LastEditDate="2018-04-05T08:44:41.297" LastActivityDate="2021-08-18T02:34:10.560" Title="High model accuracy vs very low validation accuarcy" Tags="|python|deep-learning|classification|keras|overfitting|" AnswerCount="5" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="29894" PostTypeId="2" ParentId="29885" CreationDate="2018-04-04T09:11:28.383" Score="3" Body="&lt;p&gt;I recall I was struggling for some time deriving the second equation. That constant keeps many of your missing elements. Let's break it down using your $(a-b)^{2}$ notation. We will have $a^{2}$, $b^{2}$, and $2ab$:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$a^{2}$: $y_{i}$ is constant since it is your true labels/values, thus $a^{2}$ i.e. $y_{i}^{2}$  goes to the constant.&lt;/li&gt;&#xA;&lt;li&gt;$b^{2}$: $y\hat{}_{i}^{t-1}$+$f_{t}^{2}$+$2y\hat{}_{i}^{t-1}f_{t}$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;$y\hat{}_{i}^{t-1}$ is constant as it is prediction in a step before $(t-1)$ that we know already, thus goes to the constant term. The other two terms remain as it is.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$2ab$: $2y_{i}(y\hat{}_{i}^{t-1}+f_{t}) = -2y_{i}y\hat{}_{i}^{t-1}-2y_{i}f_{t}$. Here also the first term is constant. Only the second term remains.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The rest should be pretty straightforward, just add things that are left and clean it up and you see the second equation comes out beautifully.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-04-04T11:42:22.480" LastActivityDate="2018-04-04T11:42:22.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29895" PostTypeId="1" CreationDate="2018-04-04T09:51:28.307" Score="2" ViewCount="451" Body="&lt;p&gt;The mean square loss function is the standard for regression neural networks. However, if I have a neural network learning two tasks (two outputs) at once, is it more advisable to train on the sum of the relative errors for the different outputs or the sum of the mean square errors of both tasks?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, the mean square loss function favours the task with larger expected values so it's &quot;unfair&quot; to the other task. I'm leaning towards using relative error but are there any caveats to using it as the loss function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; Problem with using relative error is that it's a percentage and therefore the gradient updates will be very small. I think it may be more practical to use an error function proportional to the mean relative errors but scaled so that they are large enough. &lt;/p&gt;&#xA;" OwnerUserId="49501" LastEditorUserId="49501" LastEditDate="2018-12-26T08:38:41.963" LastActivityDate="2018-12-26T08:38:41.963" Title="Cost/loss functions for multi-tasking regression neural networks" Tags="|machine-learning|neural-network|regression|loss-function|cost-function|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="29896" PostTypeId="2" ParentId="29893" CreationDate="2018-04-04T10:26:31.697" Score="11" Body="&lt;p&gt;You should try to shuffle all of your data and split them to the train and test and valid set then train again.&lt;/p&gt;&#xA;" OwnerUserId="29645" LastEditorUserId="29575" LastEditDate="2018-12-09T16:37:43.533" LastActivityDate="2018-12-09T16:37:43.533" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="29897" PostTypeId="2" ParentId="29890" CreationDate="2018-04-04T10:26:38.963" Score="0" Body="&lt;p&gt;Cross validation is normally done on training dataset as oppose to a method of measuring calculation on test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use a 5-fold cross validation on training dataset, this means your training dataset will be split into 5 parts and training on 4 portion of data while testing on the rest for 5 times.  The prediction from the 1 portion (with 5 times) would be used to calculate the chosen measure as cross validated score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ROSE&lt;/code&gt; package looks like does not provide cross validation function, but you can write your own function to implement:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; for( i in 1:5){&#xA;    train on 80% data&#xA;    predict on 20% data and save prediction&#xA; }  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17012" LastActivityDate="2018-04-04T10:26:38.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29898" PostTypeId="2" ParentId="11797" CreationDate="2018-04-04T10:57:36.477" Score="3" Body="&lt;p&gt;If you want counts, instead of the Boolean values, you can try like this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pandas.Series([('Adventure', 'Drama', 'Fantasy','Fantasy'), ('Comedy', 'Family'), ('Drama', 'Comedy', 'Romance'), (['Drama']), &#xA;                    (['Documentary','Documentary']), ('Adventure','Adventure' ,'Biography', 'Drama', 'Thriller')]).apply(list).to_frame(name='genre')&#xA;for genre in set.union(*df.genre.apply(set)):&#xA;    df[genre] = df.apply(lambda _: int(_.genre.count(genre)), axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49952" LastEditorUserId="29575" LastEditDate="2018-04-04T13:57:50.787" LastActivityDate="2018-04-04T13:57:50.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29899" PostTypeId="1" CreationDate="2018-04-04T11:18:12.343" Score="5" ViewCount="2699" Body="&lt;p&gt;I am trying to estimate the predictive uncertainty for a deep neural network. While I do have a labeled training set, I´m trying to measure uncertainty for some unlabeled production data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1612.01474&quot; rel=&quot;noreferrer&quot;&gt;This paper&lt;/a&gt; proposes the use of Deep Ensembles and Adversarial Training to compute a measurement of uncertainty. However, it uses Brier Score as a metric which requires me to know the real label of my production data.&#xA;Is there a similar way or metric which does not require labeled data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach was described by &lt;a href=&quot;http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html&quot; rel=&quot;noreferrer&quot;&gt;Yarin Gal&lt;/a&gt; utilizing Monte Carlo Dropout. However I can´t get any useful results using that technique on my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use my model in an online learning task. As the given data may very over time, I need to detect examples where my model is highly uncertain, so I can manually classify those examples.&lt;/p&gt;&#xA;" OwnerUserId="49951" LastEditorUserId="49951" LastEditDate="2018-04-04T11:34:33.247" LastActivityDate="2020-05-13T14:04:35.523" Title="Estimating Predictive Uncertainty for unlabeled data" Tags="|neural-network|deep-learning|unsupervised-learning|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29900" PostTypeId="1" CreationDate="2018-04-04T11:46:28.690" Score="3" ViewCount="376" Body="&lt;p&gt;I have a data set where I look at the cooling of a process. The starting temperature may vary between 580 and 180 degrees. I know that at some point the cooling system failed (see examples in the plot) and I tried to predict this failure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the data is completely unlabeled (the failure itself had to be detected manually, too), I ran unsupervised algorithms on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give an idea about the data, here is a plot including the results of one example of unsupervised learning. The top plot show two parameters that I extracted for each process, which varies in duration. The colors which resemble the position in the bottom plot result from an unsupervised learning algorithm using only 30 minutes of data, append the last value as often as possible if less than 30 minutes were available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BQQYe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BQQYe.png&quot; alt=&quot;The top plot shows the aim temperature of the process, the middel one shows the smallest value of the derivative, giving a clear indication that around process no 800 the cooling system was broken. The different colors results from unsupervised clustering using AgglomerativeClustering, Ward, and 8 clusters as the aim&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The example results of three clusters would look like this (to give an idea about the shape of the data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cqsW2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cqsW2.png&quot; alt=&quot;The temperatures of different clusters within the observed 30 Minutes&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Up to know, I have changed the algorithms between &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;AgglomerativeClustering&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.Birch.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Birch&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBSCAN&lt;/a&gt; with various parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost all clusters indicated that the failing of the cooling system occured by showing new or unusal clusters during the time the problems occured (see here clusters 4 and 7), but none of them showed a similar behaviour predicting the failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This led me to the following quesions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Would the same procedure using (obviously unsupervised) neural networks be an option and if so, how (I am working in python)? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;What other approaches could there be to handling this problem?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;At what point can I savely say that predicting the failure (not merely detecting it) is not contained in the data? I assume this is true as the processes are very different and any minor change in cooling due to a system that is about to fail would remain undiscovered as other parameters have a much bigger influence, but I'd very much appreciate any opinion on this. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Sample Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   [[565. , 565. , 564. , 555. , 542. , 527. , 511.5, 496. , 460. ,&#xA;    434. , 413. , 393. , 376. , 359. , 344. , 329. , 315. , 303. ,&#xA;    291. , 279. , 268. , 258. , 249. , 239. , 231. , 222. , 214. ,&#xA;    207. , 200. , 193. ], #would go on with 188,...&#xA;   [540. , 540. , 539. , 531. , 520. , 508. , 496. , 494. , 456. ,&#xA;    436. , 420. , 404. , 390. , 377. , 364. , 353. , 341. , 331. ,&#xA;    321. , 312. , 303. , 295. , 286. , 279. , 271. , 263. , 263. ,&#xA;    263. , 263. , 263. ], #the process was ended too early, 263 got repeated to match the format&#xA;   [530. , 530. , 529. , 520. , 509. , 495. , 455. , 427. , 405. ,&#xA;    384. , 365. , 348. , 332. , 317. , 302. , 288. , 275. , 263. ,&#xA;    252. , 242. , 232. , 222. , 213. , 204. , 196. , 188. , 181. ,&#xA;    174. , 168. , 161. ], #would go on with 154&#xA;   [181. , 174. , 165. , 158. , 152. , 147. , 146. , 141. , 137. ,&#xA;    132. , 128. , 125. , 121. , 118. , 114. , 111. , 109. , 106. ,&#xA;    103. , 101. ,  98. ,  96. ,  94. ,  92. ,  91. ,  89. ,  87. ,&#xA;     85. ,  84. ,  82. ]] #would go on with 81&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43469" LastEditorUserId="43469" LastEditDate="2018-04-05T07:41:04.427" LastActivityDate="2018-04-05T07:41:04.427" Title="Anomaly detection in cooling process data without exact labels" Tags="|python|time-series|anomaly-detection|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29902" PostTypeId="1" CreationDate="2018-04-04T14:46:02.430" Score="1" ViewCount="783" Body="&lt;p&gt;I have been using different deep learning models and extracting features from different layers for the given images. My code goes like this:&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;#load libraries&#xA;from keras import applications&#xA;from keras.preprocessing import image&#xA;from keras.applications.resnet50 import preprocess_input&#xA;from keras.models import Model&#xA;import numpy as np&#xA;&#xA;#declare models&#xA;model1 = applications.VGG16(weights='imagenet', include_top=False)&#xA;model1.summary()&#xA;model1 = Model(inputs=model1.input, outputs=model1.get_layer('block5_conv3').output)&#xA;model1.summary()&#xA;img_path = 'elephant.jpg'&#xA;img = image.load_img(img_path, target_size=(100,100))&#xA;x = image.img_to_array(img)&#xA;x = np.expand_dims(x, axis=0)&#xA;x = preprocess_input(x)&#xA;&#xA;block5_conv3_features = model1.predict(x) #dimensions (1,6,6,512)&#xA;&#xA;model2 = applications.VGG19(weights='imagenet', include_top=False)&#xA;model2.summary()&#xA;model2 = Model(inputs=model2.input, outputs=model2.get_layer('block2_conv1').output)&#xA;model2.summary()&#xA;img_path = 'elephant.jpg'&#xA;img = image.load_img(img_path, target_size=(100,100))&#xA;x = image.img_to_array(img)&#xA;x = np.expand_dims(x, axis=0)&#xA;x = preprocess_input(x)&#xA;&#xA;block2_conv1_features = model2.predict(x) #dimensions (1,50,50,128)&#xA;&#xA;model3 = applications.ResNet50(weights='imagenet',include_top=False)&#xA;model3.summary()&#xA;model3 = Model(inputs=model3.input, outputs=model3.get_layer('avg_pool').output)&#xA;model3.summary()&#xA;img_path = 'elephant.jpg'&#xA;img = image.load_img(img_path, target_size=(224, 224))&#xA;x = image.img_to_array(img)&#xA;x = np.expand_dims(x, axis=0)&#xA;x = preprocess_input(x)&#xA;&#xA;preds = model3.predict(x) #dimensions (1,1,1,248)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I'm getting feature vectors of dimensions (1,6,6,512), (1,50,50,128) and (1,1,1,128) from three different models. How can I concatenate these feature vectors in Python?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="1330" LastEditDate="2021-07-10T23:32:28.093" LastActivityDate="2021-07-10T23:32:28.093" Title="How to concatenate feature vectors of different dimensions?" Tags="|machine-learning|deep-learning|feature-extraction|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29903" PostTypeId="1" CreationDate="2018-04-04T15:00:41.253" Score="7" ViewCount="247" Body="&lt;p&gt;I have been using R and RStudio for prototyping and model building and due to some persisting problems (which would only be applicable to the environment that I am using in) we have decided to use python. I am looking to know:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;is there a python development environment like &quot;RStudio Server&quot;? &lt;/li&gt;&#xA;&lt;li&gt;how easy/hard to enable multi-threading/multi-core processing in&#xA;Python for decision trees/gradient boosting?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;can data persist in python dev environment server? Meaning: Can I save the dev environment with code and data frames and come back the&#xA;next&#xA;    day to access/pick up where I left?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can you connect to Oracle database from Python? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a way I can import .RData to python development&#xA;    environment?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why ask these questions here?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While looking up on search engines, I am mostly getting unreliable results, training insitutes promotions, outdated blogs and whitepapers from industry giants. I need a reliable answer.&lt;/p&gt;&#xA;" OwnerUserId="49963" LastActivityDate="2018-04-05T07:47:39.990" Title="migrating to python from R: specific questions" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29904" PostTypeId="1" CreationDate="2018-04-04T15:41:01.853" Score="1" ViewCount="99" Body="&lt;p&gt;I keep reading that convolution neural net (CNN) performs best with lots and lots (100k+) of data.  Is there any rule of thumb, or lower limit for data size during the grid search phase?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I run a CNN with 100 data points, vary just one parameter (say add an extra layer, or increase a filter size), and get better results, can I reasonably expect better results with those parameters during the actual training phase?&lt;/p&gt;&#xA;" OwnerUserId="21232" LastActivityDate="2023-12-22T00:06:50.820" Title="tuning a convolution neural net, sample size" Tags="|machine-learning|cnn|convolution|hyperparameter-tuning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29905" PostTypeId="2" ParentId="29903" CreationDate="2018-04-04T15:45:29.473" Score="1" Body="&lt;p&gt;1) Anaconda Spyder, maybe&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) sklearn, random forest has an option to select the number of jobs, and it will take care of parallelizing&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) i don't think so, but you can pickle objects and load them up. you can probably do something like create a variable_name - value dictionary and just pickle that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) never tried&lt;/p&gt;&#xA;&#xA;&lt;p&gt;5) according to &lt;a href=&quot;https://stackoverflow.com/questions/21288133/loading-rdata-files-into-python&quot;&gt;https://stackoverflow.com/questions/21288133/loading-rdata-files-into-python&lt;/a&gt;, I don't think so&lt;/p&gt;&#xA;" OwnerUserId="21232" LastActivityDate="2018-04-04T15:45:29.473" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29907" PostTypeId="1" AcceptedAnswerId="29913" CreationDate="2018-04-04T17:05:44.723" Score="1" ViewCount="81" Body="&lt;p&gt;I am building a neural network to solve a regression problem. The output is a single numerical value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, the output is censored: the values below 0 were recorded as 0, and postive values remained unchanged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What activation function should I use for the output layer (maybe ReLU)? How to define the loss function, should I just use RMSE? (because the output is censored, we want the neural network to be able to generate 0 output, and positive values).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is to predict the electricity demand time series based on multiple input variables. Only the values above a certain threshold are being recorded, hence the output is censored. We have a lot of numerical/categorical input variables: time of the day, air temperature, days in a week, holiday/workday, etc. We want to build a neural network model to predict electricity demand (0 or positive) based on the input variables.&lt;/p&gt;&#xA;" OwnerUserId="16866" LastEditorUserId="40853" LastEditDate="2018-04-04T17:52:27.037" LastActivityDate="2018-04-04T19:53:51.627" Title="Censored output data, which activation function for the output layer and which loss function to use?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29908" PostTypeId="1" AcceptedAnswerId="29917" CreationDate="2018-04-04T17:14:02.240" Score="6" ViewCount="2060" Body="&lt;p&gt;I'm having trouble graphing Pandas grouped data in Bokeh.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;company_id    company_score    date_submitted    company_region&#xA;AA            .07              1/1/2017          NW&#xA;AB            .08              1/2/2017          NE&#xA;CD            .0003            1/18/2017         NW&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've successfully grouped the data by &lt;code&gt;company_id&lt;/code&gt;, and calculated the simple moving average of company_score using 10 days / inputs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;score_by_company = df['company_score'].groupby(df['company_id']).rolling(10).mean()&#xA;&#xA;company_id    index           SMA     &#xA;BBG000B9WMF7  7613            NaN&#xA;              1911663         NaN&#xA;              1911664         NaN&#xA;              1911665         NaN&#xA;              1911666         NaN&#xA;              1911667         NaN&#xA;              1911668         NaN&#xA;              1911669         NaN&#xA;              1911670         NaN&#xA;              1911671    0.000002&#xA;              1911672    0.000002&#xA;              etc.       etc.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I translate this data into a time series graph, with each &lt;code&gt;company_id&lt;/code&gt; being its own line?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like I need to create a data dictionary off of the grouped data, but I'm not sure if that's the right approach.&lt;/p&gt;&#xA;" OwnerUserId="49861" LastEditorUserId="8878" LastEditDate="2018-04-05T09:21:19.800" LastActivityDate="2020-08-01T16:00:52.110" Title="Pandas grouped data to Bokeh graph" Tags="|python|visualization|pandas|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29911" PostTypeId="2" ParentId="29857" CreationDate="2018-04-04T19:06:24.970" Score="7" Body="&lt;p&gt;You are looking to create a &lt;a href=&quot;https://en.wikipedia.org/wiki/Parse_tree&quot; rel=&quot;noreferrer&quot;&gt; parse tree&lt;/a&gt; to find multi-token clauses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is code to generate a parse tree:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import spacy&#xA;from nltk import Tree&#xA;&#xA;nlp = spacy.load('en')&#xA;&#xA;def to_nltk_tree(node):&#xA;    if node.n_lefts + node.n_rights &amp;gt; 0:&#xA;        return Tree(node.orth_, [to_nltk_tree(child) for child in node.children])&#xA;    else:&#xA;        return node.orth_&#xA;&#xA;query = u'tell me about people in konoha who have wind style chakra and are above jonin level'&#xA;doc = nlp(query)&#xA;[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    tell                          &#xA;  ____|______                       &#xA; |         about                   &#xA; |           |                      &#xA; |         people                  &#xA; |     ______|_____                 &#xA; |    |           have             &#xA; |    |       _____|____________    &#xA; |    |      |     |     |     are &#xA; |    |      |     |     |      |   &#xA; |    |      |     |   chakra above&#xA; |    |      |     |     |      |   &#xA; |    in     |     |   style  level&#xA; |    |      |     |     |      |   &#xA; me konoha  who   and   wind  jonin&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;From that parse tree, you can then select the phrases.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-04T19:06:24.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29912" PostTypeId="1" CreationDate="2018-04-04T19:45:26.177" Score="1" ViewCount="113" Body="&lt;p&gt;My use case is doing a regression on images, which is different than most examples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to use one of the canned data generators to yield images from a directory for x and a single integer for y? How would I do so? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd also like to shuffle the dataset and do an 80 : 20 validation split.&lt;/p&gt;&#xA;" OwnerUserId="49980" LastEditorUserId="24000" LastEditDate="2018-04-05T10:49:05.383" LastActivityDate="2018-04-05T10:49:05.383" Title="Keras data generator for images x and continuous y" Tags="|neural-network|keras|regression|" AnswerCount="0" CommentCount="1" ClosedDate="2018-04-09T15:26:43.280" ContentLicense="CC BY-SA 3.0" />
  <row Id="29913" PostTypeId="2" ParentId="29907" CreationDate="2018-04-04T19:53:51.627" Score="0" Body="&lt;p&gt;We can't tell you what loss function to use.  That is based on business needs.  In particular, what is the cost of being wrong?  Is the cost of being wrong proportional to relative error?  absolute error?  Something else?  That will drive the choice of loss function.  You should try to choose a loss function where the value of the loss function is proportional to the cost of the error (e.g., the monetary cost to your company).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using ReLU as the activation function in the final layer would be sensible given that your data is censored to replace all negative outputs with 0.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-04-04T19:53:51.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29915" PostTypeId="2" ParentId="29858" CreationDate="2018-04-04T21:59:45.157" Score="5" Body="&lt;p&gt;I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\dots,X_d)$. Also let $X_{ij}$ be the $i$th observation for variable $j$. These measures are called rank correlations because they can be computed using the ranks only. What I mean is that if you sort all $X_{ij}$, $i=1,\dots,n$, and replace the biggest observation by $n$, the second biggest by $n-1$, and so on (do that for all columns $j$) and call you new observations $R_{ij}$, then&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;the empirical Spearman's $\rho$ (matrix) is simply the Pearson linear correlation (matrix) of $(R_1,\dots,R_d)$; and&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the empirical (pairwise) Kendall's $\tau$ between $X_{i_1}$ and $X_{i_2}$ is the probability of concordance minus the probability of discordance between two iid observations, say $(X_{1 i_1},X_{1 i_2})$ and $(X_{2 i_1},X_{2 i_2})$, which  can equivalently be computed from the ranks $(R_{1 i_1},R_{1 i_2})$ and $(R_{2 i_1},R_{2 i_2})$ instead.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A rank correlation between $X_{i_1}$ and $X_{i_2}$ of one indeed means perfect concordance (i.e. $X_{i_1}$ always increases with $X_{i_2}$), but that does not necessarily means they are linearly related. The ranks are linearly related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just to make the concept of concordance clearer, here the (bivariate) observations are all concordant&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/brmHb.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/brmHb.png&quot; alt=&quot;Concordant observations from a bivariate random vector&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and here they are all discordant&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NzBRh.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NzBRh.png&quot; alt=&quot;Discordant observations from a bivariate random vector&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So that when you consider a cloud of points you have some pairs that are concordant and others that are discordant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this answer provides examples, but there are many other ways to approach the question. As commented by Emre, information-theoretic measures are also an option.&lt;/p&gt;&#xA;" OwnerUserId="47308" LastActivityDate="2018-04-04T21:59:45.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29916" PostTypeId="1" CreationDate="2018-04-05T01:44:21.203" Score="0" ViewCount="93" Body="&lt;p&gt;I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. &lt;a href=&quot;https://i.stack.imgur.com/zV56U.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here are my clustering results for one search.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, there is clearly three different groups showing up at the moment. However, when you look at the &lt;a href=&quot;https://i.stack.imgur.com/afSzy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;top terms of each cluster&lt;/a&gt;, there is a clear issue. A lot of the terms are repeating in each cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is another issue, when I try to have the model predict which clusters a few test items are in, which are clearly different products, it says that they are in the same group. Here are the test terms:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Adjustable DC-DC LM 2596 Converter Buck Step Down Regulator Power Module LW SZUS&#xA;10pcs Mini360 3A DC Voltage Step Down Power Converter Buck Module 3.3V 5V 9V 12V&#xA;5V Out, 6V to 12V In AMS1117-5.0 5.0V Step-Down Linear Voltage Regulator Module&#xA;10pcs Mini LM2596s 3A DC to DC Buck Converter Power Supply Step Down Module&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have thought about using the grammar in each title as a feature, but as there is an 80 character limit for the titles, there rarely is any grammar. I am at a loss for where to go next in terms of trying to pick more features that would get me better results. I am doing all of this in python using scikitlearn and nltk.&lt;/p&gt;&#xA;" OwnerUserId="44396" LastActivityDate="2018-07-12T11:55:49.143" Title="How to select features for clustering to detect the number of different unique products in a search result?" Tags="|scikit-learn|clustering|k-means|nltk|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29917" PostTypeId="2" ParentId="29908" CreationDate="2018-04-05T02:04:18.973" Score="2" Body="&lt;p&gt;You can make the plots by looping over the groups from &lt;code&gt;groupby&lt;/code&gt;...&lt;/p&gt;&#xA;&lt;p&gt;Or this should also work..&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib&#xA;import matplotlib.pyplot as plt&#xA;df['company_score'].groupby(df['company_id']).rolling(10).mean().unstack().plot()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;See &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/visualization.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for more information on plotting with pandas dataframes and &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/groupby.html#iterating-through-groups&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for looping over a groupby-object. Finally &lt;a href=&quot;http://blog.yhat.com/posts/aggregating-and-plotting-time-series-in-python.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; link is super helpful.&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="98307" LastEditDate="2020-08-01T16:00:52.110" LastActivityDate="2020-08-01T16:00:52.110" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29920" PostTypeId="2" ParentId="10509" CreationDate="2018-04-05T05:20:14.747" Score="8" Body="&lt;p&gt;This is an old question. I am surprised that I don't see anyone mentioned &lt;strong&gt;Mean Encoding&lt;/strong&gt; (a.k.a &lt;strong&gt;Target Encoding&lt;/strong&gt;). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the &lt;strong&gt;cdf of the frequency&lt;/strong&gt; (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really explain why it works rigorously.&lt;/p&gt;&#xA;" OwnerUserId="49932" LastActivityDate="2018-04-05T05:20:14.747" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29922" PostTypeId="2" ParentId="23061" CreationDate="2018-04-05T06:00:48.857" Score="1" Body="&lt;p&gt;I don't think there is a well-defined (or well enough defined) terminology for that. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multi-variate means that it involves multiple (dependent or independent) variables. Which is pretty clear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multi-dimensional in mathematical sense means that the data points are from a multi-dimensional space (typically vector space). This is &lt;strong&gt;exactly the same&lt;/strong&gt; as multi-variate. In this sense, your visualization is both multi-variate and multi-dimensional. And both the size of the markers and the color represent individual dimensions or variables, which is then equivalent. Also, time would be an individual dimension. Personally, I would go with this interpretation of the term multi-dimensional.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to restrict the term multi-dimensional to spatial dimensions of the plot, then size and color of markers would not be dimensions in that sense. I personally find this interpretation confusing. Time then also would not be a dimension in this sense, unless you represent it by a 3rd axis in the graph. This, again, is confusing as many people actually often put time into the same category as spatial dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, all in all I would stick to the mathematical interpretation of dimensions here so that multi-variate and multi-dimensional will be equivalent. &lt;/p&gt;&#xA;" OwnerUserId="29912" LastActivityDate="2018-04-05T06:00:48.857" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29923" PostTypeId="1" CreationDate="2018-04-05T06:01:20.780" Score="1" ViewCount="1328" Body="&lt;p&gt;I am using the dot product as a way to measure the similarly of two facial-model vectors extracted by a ML algorithm (OpenFace in fact).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to convert the L2 norm to a probability U[0,1] in order to compare with other solutions from other providers that map directly to probabilities but in a blackbox fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By &lt;em&gt;probability&lt;/em&gt; &lt;strong&gt;I mean the probability that the two vectors represent the same person&lt;/strong&gt;. So an identical image would give a probability of 1; images of the same person would give a high probability; and orthogonal images (different people, or some suitable toy case) would in the limit have such a probability of zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like a uniform distribution for comparison with other providers, but this boils down to knowing what the underlying probability distribution of the feature vectors is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do this?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Rayleigh distribution CDF&lt;/li&gt;&#xA;&lt;li&gt;Cosine similarity&lt;/li&gt;&#xA;&lt;li&gt;(1) is the same as (2)&lt;/li&gt;&#xA;&lt;li&gt;Draw from a simulated distribution&lt;/li&gt;&#xA;&lt;li&gt;Arbitrarily scale to U[0,1]&lt;/li&gt;&#xA;&lt;li&gt;Train into the comparison probability space directly (skipping the feature-vector step)&lt;/li&gt;&#xA;&lt;li&gt;None of the above&lt;/li&gt;&#xA;&lt;li&gt;Something else?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Some useful links are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/744/cosine-similarity-versus-dot-product-as-distance-metrics&quot;&gt;Cosine similarity versus dot product as distance metrics&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://houseofjeff.com/2015/05/10/using-probability-distribution-and-cosine-similarity-to-automatically-detect-data-contents&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://houseofjeff.com/2015/05/10/using-probability-distribution-and-cosine-similarity-to-automatically-detect-data-contents&lt;/a&gt; - this one is almost there?&lt;/p&gt;&#xA;" OwnerUserId="37113" LastEditorUserId="37113" LastEditDate="2018-04-05T09:52:01.453" LastActivityDate="2018-04-05T10:41:25.697" Title="How do I convert an L2 norm to a probability?" Tags="|machine-learning|statistics|probability|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29924" PostTypeId="2" ParentId="26663" CreationDate="2018-04-05T06:26:45.740" Score="3" Body="&lt;p&gt;See this post: &lt;a href=&quot;https://stackoverflow.com/questions/47594861/predicting-a-multiple-time-step-forward-of-a-time-series-using-lstm&quot;&gt;https://stackoverflow.com/questions/47594861/predicting-a-multiple-time-step-forward-of-a-time-series-using-lstm&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be training on a shifted X for your Y. Then passing through the new weights.h5 and predicting as shown.&lt;/p&gt;&#xA;" OwnerUserId="31046" LastActivityDate="2018-04-05T06:26:45.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29927" PostTypeId="2" ParentId="29903" CreationDate="2018-04-05T07:47:39.990" Score="0" Body="&lt;p&gt;I can try to answer the 3rd question. You can use Jupyter + Python Kernel (install &lt;a href=&quot;https://anaconda.org/anaconda/python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Anaconda&lt;/a&gt; to use conda commands with python Kernel). Jupyter allows keep your code and notes and graphs in one notebook and save it. &lt;/p&gt;&#xA;" OwnerUserId="46674" LastActivityDate="2018-04-05T07:47:39.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29930" PostTypeId="1" CreationDate="2018-04-05T09:42:47.767" Score="1" ViewCount="544" Body="&lt;p&gt;Maybe it's a trivial question, but I'm a bit confused right now... I'll explain: I've some elements in my data, each with a value between 0 and 1 and an associated label (1, 0).&#xA;I need to test some thresholds and then compute the ROC-AUC curve.&#xA;For example with a threshold = 0.4, all the values greater than 0.4 will be predicted as true (1) and all the values under 0.4 will be predicted as false (0), then I'll compare the result with the actual labels to calculate the True Positive Rate and the False Positive Rate and finally build the ROC-AUC curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is, considering that, based on the threshold chosen, I already know under which prediction an element will fall, I don't need a machine learning classifier to do the default training and testing stages, right?&lt;/p&gt;&#xA;" OwnerUserId="46313" LastEditorUserId="85045" LastEditDate="2021-09-14T20:24:43.537" LastActivityDate="2021-09-14T20:24:43.537" Title="ROC-AUC curve as metric for binary classifier without machine learning algorithm" Tags="|classification|auc|roc|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29931" PostTypeId="2" ParentId="28849" CreationDate="2018-04-05T10:10:22.907" Score="0" Body="&lt;p&gt;The road to Machine Learning enlightenment is highly non-linear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You already took a great step, which is coming to Data Science exchange to ask questions. :)&#xA;This is a great place, full of resources and good pointers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As mentioned before, you need to slowly pave your way through different topics, from practical decisions about datasets, computer, training time, to deeper decisions about the model and overall approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My recommendation as someone trying to go through the same path is to look for great leaders in the area and go through their courses thoroughly. The first one I can recommend is David MacKay, with his free book (&lt;a href=&quot;http://www.inference.org.uk/itila/book.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.inference.org.uk/itila/book.html&lt;/a&gt; and lectures (&lt;a href=&quot;http://videolectures.net/david_mackay/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://videolectures.net/david_mackay/&lt;/a&gt;). David's approach to Machine Learning is that of a &quot;connector&quot;, trying to show connections between different fields. Take a look at his lectures and try to follow them with exercises on the book. This will give you a solid foundation on which to draw upon when you are in trouble. He saved me more than once.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another recommendation is Andrew Ng's Coursera course on Machine Learning. Although not as deep, it is full of practical advice for different methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having a good understanding will strengthen your foundations so when tides, such as Deep Learning, come towards you, you will know how to keep your stand and make the most of it. You will understand how tides come and go and allow them to transform you instead of overwhelming you.&lt;/p&gt;&#xA;" OwnerUserId="49770" LastActivityDate="2018-04-05T10:10:22.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29932" PostTypeId="2" ParentId="29923" CreationDate="2018-04-05T10:41:25.697" Score="1" Body="&lt;p&gt;I guess you either need to train directly into probability space, or train another model on top of the feature vectors to learn the probability of two vectors belonging to the same person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the other metrics you could use on top of the feature vectors are a bit arbitrary, as they were not considered by the cost function used when training the original model.&lt;/p&gt;&#xA;" OwnerUserId="29912" LastActivityDate="2018-04-05T10:41:25.697" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29933" PostTypeId="1" AcceptedAnswerId="29939" CreationDate="2018-04-05T12:14:41.737" Score="1" ViewCount="462" Body="&lt;p&gt;I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To understand the logic, I am reviewing some of the present code samples. This one seemed quite interesting, so I went through it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/keon/deep-q-learning/blob/master/dqn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keon/deep-q-learning/blob/master/dqn.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the gist of the code that does the actual training of the underlying deep neural network: &lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def replay(self, batch_size):&#xA;    minibatch = random.sample(self.memory, batch_size)&#xA;    for state, action, reward, next_state, done in minibatch:&#xA;        target = reward&#xA;        if not done:&#xA;            target = (reward + self.gamma *&#xA;                      np.amax(self.model.predict(next_state)[0]))&#xA;        target_f = self.model.predict(state)&#xA;        target_f[0][action] = target&#xA;        self.model.fit(state, target_f, epochs=1, verbose=0)&#xA;    if self.epsilon &amp;gt; self.epsilon_min:&#xA;        self.epsilon *= self.epsilon_decay&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the 5th line of the code, (after the &lt;code&gt;if not done&lt;/code&gt; line) we are adding the discounted reward of the next step, to the present step, and setting it as the target reward of the executed action to be trained. So, the way I see it, we have the reward of the executed action, and discounted possible reward of the following action, combined. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understand, in each iteration, Q-learning algorithm predicts the future reward &lt;strong&gt;of next step (and next step only)&lt;/strong&gt; using the machine learning technique in use (be it the CNN, DNN etc.). And we are multiplying the reward of &lt;strong&gt;next step (and that specific next step only)&lt;/strong&gt; with discount rate, to make it less important than the immediate reward (with the ratio we specified). So, my question is, how does the algorithm takes even further steps (say, 5 steps) ahead into account?&lt;/p&gt;&#xA;" OwnerUserId="50007" LastEditorUserId="50007" LastEditDate="2018-04-05T13:21:18.833" LastActivityDate="2018-04-05T14:14:35.380" Title="How does a Q algorithm consider future rewards?" Tags="|machine-learning|deep-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29934" PostTypeId="2" ParentId="29930" CreationDate="2018-04-05T12:54:51.773" Score="1" Body="&lt;p&gt;Your intuition is correct: you don't necessarily need a machine learning algorithm to calculate a ROC curve and, from what you say, it seems you already have the necessary ingredients in place.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the ROC curve was historically developed under circumstances that had nothing to do with machine learning; from &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields and was soon introduced to psychology to account for perceptual detection of stimuli. ROC analysis since then has been used in medicine, radiology, biometrics, forecasting of natural hazards, meteorology, model performance assessment, and other areas for many decades and is increasingly used in machine learning and data mining research.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43000" LastEditorUserId="43000" LastEditDate="2018-04-05T13:41:51.540" LastActivityDate="2018-04-05T13:41:51.540" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29938" PostTypeId="1" AcceptedAnswerId="36114" CreationDate="2018-04-05T14:14:12.703" Score="4" ViewCount="4844" Body="&lt;h1&gt;Background&lt;/h1&gt;&#xA;&lt;p&gt;Using Python, I need to score the existence of a quote, containing around 2-7 words, a longer text. The quote doesn't have to match the text precisely, but similar words should have the &lt;em&gt;same order&lt;/em&gt;.&lt;/p&gt;&#xA;&lt;p&gt;For example, given the following &lt;strong&gt;long text&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The most beautiful things in the world cannot be seen or touched, they are felt with the heart&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The following quotes should be scored &lt;strong&gt;high&lt;/strong&gt; (say, above 80 / 100):&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The beautiful thing in our world&lt;/p&gt;&#xA;&lt;p&gt;World cannot see&lt;/p&gt;&#xA;&lt;p&gt;They feel with the heart&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Since they are not precise, but they preserve the order.&lt;/p&gt;&#xA;&lt;p&gt;While, on the other hand, these quotes should be scored &lt;strong&gt;lower&lt;/strong&gt; (say, below 50 / 100):&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;The beautiful heart cannot be felt or seen&lt;/p&gt;&#xA;&lt;p&gt;They are the most seen in the world&lt;/p&gt;&#xA;&lt;p&gt;These words don't even appear on this text&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Because (the first 2) appear entirely in the text, but do not preserve the order.&lt;/p&gt;&#xA;&lt;h1&gt;The problem&lt;/h1&gt;&#xA;&lt;p&gt;This task cannot be accomplished by simply checking the existence of each word in the text. I don't know which algorithm fits best for this task.&lt;/p&gt;&#xA;&lt;h1&gt;What I have tried&lt;/h1&gt;&#xA;&lt;p&gt;Most of the functions in &lt;code&gt;fuzzywuzzy&lt;/code&gt; (&lt;code&gt;partial_token_sort_ratio&lt;/code&gt;, &lt;code&gt;token_sort_ratio&lt;/code&gt; and etc) scored the later terms higher.&#xA;&lt;code&gt;partial_ratio&lt;/code&gt; did score the earlier terms higher, but the quote&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;These words don't even appear on this text&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Got 52 / 100 which is unreasonably high.&lt;/p&gt;&#xA;&lt;h1&gt;My question&lt;/h1&gt;&#xA;&lt;p&gt;How can I use python to score the existence of short quotes in longer texts as mentioned above?&lt;/p&gt;&#xA;" OwnerUserId="10972" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-10-11T16:11:16.550" Title="Plagiarism detection with Python" Tags="|python|nlp|fuzzy-logic|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29939" PostTypeId="2" ParentId="29933" CreationDate="2018-04-05T14:14:35.380" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;As far as I understand, in each iteration, Q-learning algorithm predicts the future reward of next step (and next step only) using the machine learning technique in use (be it the CNN, DNN etc.). &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The Q values should eventually converge to the &lt;em&gt;expected sum&lt;/em&gt;, &lt;em&gt;future&lt;/em&gt;, &lt;em&gt;discounted&lt;/em&gt; reward when taking action A in state S and following the optimal policy. Breaking it down:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Expected sum&lt;/em&gt; is not exactly the same as &quot;predicted&quot;, but close enough for our purposes. And it really does mean sum of the rewards, not a single reward. To differentiate, this is often called the &quot;return&quot; or &quot;utility&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Future&lt;/em&gt; -&gt; from the step being evaluated onwards until end of episode, or the limit as time goes to infinity for continuous tasks with discounting.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;em&gt;Discounted&lt;/em&gt; -&gt; a discount factor is only necessary for continuous tasks.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And we are multiplying the reward of next step (and that specific next step only) with discount rate, to make it less important than the immediate reward (with the ratio we specified).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No, there is no multiplication of the reward. Let's take a look at the line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;target = (reward + self.gamma *&#xA;                  np.amax(self.model.predict(next_state)[0]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;reward&lt;/code&gt; is not being multiplied by anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is being multiplied by $\gamma$ is the Q value of the next state. That value represents the total sum of all rewards following on from that point - not a single reward value at all.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So, my question is, how does the algorithm takes even further steps (say, 5 steps) ahead into account?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is in the Q values. The pseudocode for the code you are looking at is not:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;target_for_Q(s,a) = next_step_reward * gamma&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;target_for_Q(s,a) = next_reward + gamma * current_value_of_Q(s',a')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;target_for_Q(s,a) = next_reward + gamma * estimate_all_future_return&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is closely related to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Bellman_equation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bellman function for policy evaluation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively what is happening is that you start with (really poor) estimates for expected &lt;em&gt;return&lt;/em&gt; (not expected reward), and update them by inserting observed values of &lt;code&gt;next_reward&lt;/code&gt;, &lt;code&gt;s'&lt;/code&gt; and &lt;code&gt;a'&lt;/code&gt; into the update rule above. The values always represent a learned estimate of total expected &lt;em&gt;return&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-04-05T14:14:35.380" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29941" PostTypeId="1" AcceptedAnswerId="29996" CreationDate="2018-04-05T14:30:42.107" Score="1" ViewCount="449" Body="&lt;p&gt;Can Q Learning work with a static state for each step? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over again. Of course the different actions lead to different rewards, but is Q Learning the right concept to use in this case? What other types could you suggest?&lt;/p&gt;&#xA;" OwnerUserId="50014" LastActivityDate="2018-05-06T20:39:29.180" Title="Reinforcement Learning with static state" Tags="|reinforcement-learning|q-learning|dqn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29942" PostTypeId="2" ParentId="29266" CreationDate="2018-04-05T15:01:03.853" Score="1" Body="&lt;p&gt;Adjust the optimizer hyper-parameters&#xA;1) use amsgrad = True&#xA;2) try lower beta_1&#xA;3) adjust lr from 1e-6 to .1  or use a learning rate finder&#xA;4) use either the weight_decay(learning schedule) in the optimizer or add a Callback that will adjust the learning rate while training&lt;/p&gt;&#xA;&#xA;&lt;p&gt;adjust the parameters here.....&#xA;optimizer=Adam(lr=0.0001, beta_1=0.9, beta_2=0.999,   epsilon=0.00000001)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dropouts, batch normalization can also help as mentioned above. I would also suggest using a leaky relu or prelu in place of the relu activation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also initialize the conv weights using the 'he_uniform' initializer instead of the default.&lt;/p&gt;&#xA;" OwnerUserId="50016" LastActivityDate="2018-04-05T15:01:03.853" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29944" PostTypeId="1" AcceptedAnswerId="29945" CreationDate="2018-04-05T15:15:57.030" Score="0" ViewCount="185" Body="&lt;p&gt;I'm self learning data science so bear with me as I try and make my question as clear as possible. Lets assume I have a dataset of a dependent variable y and some independent variables x1, x2 and x3 etc. &#xA;Is the correlation between these variables simply a mathematical calculation? I mean if i ran a correlation test in R for example and the test came back with no correlation between the variables is that necessarily true? What if I knew or suspected that one of the variables is correlated to another how would I define that in my model or would relying on a correlation test be sufficient.&lt;/p&gt;&#xA;" OwnerUserId="50018" LastActivityDate="2018-04-09T08:52:09.313" Title="How do I determine if variables are correlated? Is it simply a mathematical calculation?" Tags="|machine-learning|r|data|correlation|data-science-model|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29945" PostTypeId="2" ParentId="29944" CreationDate="2018-04-05T15:33:55.463" Score="3" Body="&lt;p&gt;Yes, correlation is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow noreferrer&quot;&gt;mathematical concept&lt;/a&gt;, and it as well known as &lt;strong&gt;Pearson correlation&lt;/strong&gt;. This is probably the one you are obtaining in your analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are other correlation analysis you can try in order to be totally sure about your result. The most famous ones out of Pearson are &lt;strong&gt;Kendall&lt;/strong&gt; correlation and &lt;strong&gt;Spearman&lt;/strong&gt; correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, sometimes relation between variables is less evident than a correlation analysis. For example, if the variables are time-related, one may have a &lt;strong&gt;lag&lt;/strong&gt; over the other. For example, in finances it is known that the GDP is correlated with a lag of some months of the Employment Rate. It is part of a good Data Science knowledge to have a business knowledge, an intuition of why the variables may be related in some way and try to find how to prove that this non evident dependence exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I noticed you put R as a tag. You can study as well libraries polycor and ggm to learn about &lt;strong&gt;polychoric correlations&lt;/strong&gt; and &lt;strong&gt;partial correlations&lt;/strong&gt;, although the may not apply in your case.&lt;/p&gt;&#xA;" OwnerUserId="49226" LastEditorUserId="49226" LastEditDate="2018-04-05T15:47:34.340" LastActivityDate="2018-04-05T15:47:34.340" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29947" PostTypeId="1" CreationDate="2018-04-05T15:52:18.043" Score="1" ViewCount="33" Body="&lt;p&gt;I have data of 90 persons.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data of a person:&lt;br&gt;&#xA;- x coordinate&lt;br&gt;&#xA;- y coordinate&lt;br&gt;&#xA;- score (1 to 6)  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to form groups of 9 people each so that:&lt;br&gt;&#xA;- the distance between people in one group is as small as possible&lt;br&gt;&#xA;- people with the same coordinates are in different groups&lt;br&gt;&#xA;- people within a group should have different scores  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not expecting a full-fledged algorithm but rather hints in the right direction. So far I am using a simple k-means algorithm which forms groups so that the distance between people gets minimized however the two other constraints are not considered, unfortunately.&lt;/p&gt;&#xA;" OwnerUserId="50021" LastActivityDate="2018-04-06T19:04:36.577" Title="Cluster algorithm which minimizes a distance while fulfilling a constraint" Tags="|clustering|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29948" PostTypeId="2" ParentId="29848" CreationDate="2018-04-05T16:17:08.617" Score="1" Body="&lt;p&gt;It seems likely that the live data is different somehow from your other data.  Cross-validation shows a 84% accuracy, and accuracy on the held-out set is 86%, which is pretty consistent and does not indicate overfitting.  Accuracy on the live data is 70%, which is significantly different.  That suggests that live data is somehow different in ways that are important to the classifier.  Perhaps &lt;a href=&quot;https://en.wikipedia.org/wiki/Concept_drift&quot; rel=&quot;nofollow noreferrer&quot;&gt;concept drift&lt;/a&gt; has occurred.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-04-05T16:17:08.617" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29950" PostTypeId="1" CreationDate="2018-04-05T16:18:36.850" Score="1" ViewCount="68" Body="&lt;p&gt;I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer.  By this I mean handling the following on my own:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;preprocess big data at low latency&lt;/li&gt;&#xA;&lt;li&gt;design &amp;amp; train deep learning models on massive data&lt;/li&gt;&#xA;&lt;li&gt;deploy models to serve predictions at massive scale&lt;/li&gt;&#xA;&lt;li&gt;stream/preprocess incoming data to update models in real time&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which cloud platform would you choose to do this?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;GCP:  Allows me to do the above with the minimum effort (serverless model hosting, model versioning etc), however, it ties me to tensorflow (I'm an MXNet fan).  Looks like I need to pick up apache beam for distributed data preprocessing...&lt;/li&gt;&#xA;&lt;li&gt;AWS: Maximum flexibility but seems far less clean.  Appears to be more suited to a team of 5 experts wanting to achieve the above.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What software would you choose?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I'm essentially looking to pickup the minimum amount of things to have the maximum impact. &lt;/li&gt;&#xA;&lt;li&gt;Currently spend most of my time using python + MXNet + EC2 and am comfortable with (2)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="50025" LastEditorUserId="50025" LastEditDate="2018-04-05T16:28:30.987" LastActivityDate="2018-04-05T17:11:50.330" Title="Which cloud platform to maximize my impact as a data scientist?" Tags="|deep-learning|preprocessing|distributed|google-prediction-api|cloud-computing|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-06T01:10:59.553" ContentLicense="CC BY-SA 3.0" />
  <row Id="29951" PostTypeId="1" CreationDate="2018-04-05T16:55:08.433" Score="0" ViewCount="105" Body="&lt;p&gt;We have a large amount (Billions) of high cardinality, mixed nominal &amp;amp; numerical data, and are performing some clustering on it as an experiment.  There is a small subset of these data, however, that we know should belong in its own cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to purposely make sure these data end up in their own cluster, while still clustering the remaining data?  If so, how does one approach doing so?&lt;/p&gt;&#xA;" OwnerUserId="50030" LastActivityDate="2018-04-06T20:22:16.260" Title="Can You Purposely Bias A Clustering Model?" Tags="|clustering|unsupervised-learning|semi-supervised-learning|bias|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29952" PostTypeId="1" CreationDate="2018-04-05T17:10:36.590" Score="4" ViewCount="4165" Body="&lt;p&gt;In the context of time series prediction, I have read that time series is a series of data that taken at successive equally spaced points in time (which means its in order). What if I have a &lt;strong&gt;discontinuous&lt;/strong&gt; time series data, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have data that represnt a room temperature within the working hours, specifically from 7:00 am - 3:00 pm. And this data is repeated on the day of the week. So we have something similar to this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saturday: [ Time series data from 7:00 am - 3:00 pm] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sunday: [ Time series data from 7:00 am - 3:00 pm] &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Monday: [ Time series data from 7:00 am - &lt;strong&gt;12:00 pm&lt;/strong&gt;]  -- &lt;strong&gt;Only up to 12 pm&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first question:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Is that considered as time series data&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Now if I want to predict the room temperature on Monday @ 1:00 pm. How can I do this using given this type of datasets?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="49752" LastActivityDate="2019-06-26T23:10:27.360" Title="Time series prediction of discontinuous data" Tags="|data-mining|time-series|prediction|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="29953" PostTypeId="2" ParentId="29950" CreationDate="2018-04-05T17:11:50.330" Score="4" Body="&lt;p&gt;It has been my experience that transitioning from local modeling to large-scale distributed programming is a lot more work than most data scientists realize, and leaves little room for anything BUT data engineering, similar to what @Emre said above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're developing an infrastructure yourself (say, spark) using GCP or AWS VMs, installing and maintaining that is a LOT of work.  This is doubly true if you're running a multi-tenant system and/or supporting production jobs.  You will constantly be trying to solve the 'why didn't my job run?' or 'why does it take my job 14 days to run?' problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using the built-in data science infrastructure to those systems (Redshift, Athena, ElasticSearch, etc), you can save some time but it is still EXTREMELY non-trivial to manage.  There is a reason why, year after year, Data Scientists' favorite tools always include Databricks or similar -- because managing these things is a pain, and requires a completely different skillset than actual modeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that being said, I have two suggestions.  First, AWS is more mature and has more community answers that help overcome hurdles than GCP does.  You will run into issues with its IAM systems (which become a necessary evil of responsible big data engineering), and the quirkiness of its various products (lambda, for instance, will only run &amp;lt; 3m scripts), but overall everything you're trying to do has already been done and documented by someone else.  It is the better choice, IMO.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'd urge you to take a closer look at a managed platform like Qubole or Databricks (I work for neither).  I set up and maintained a Qubole / AWS environment for over a year.  I learned a ton about data engineering, architecture, elastic spark infrastructure and all the nuances / pitfalls / limitations of distributed computing that they don't tell you about in the rote documentation, but was still able to maintain a functioning system.  My use case didn't include deep learning, but they could have, by me turning a few knobs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, once you have taken your lessons from these environments that work, you can architect, deploy &amp;amp; support your own big data infrastructure to your heart's content (which will now be your full time job).  Hope that helps.&lt;/p&gt;&#xA;" OwnerUserId="50030" LastActivityDate="2018-04-05T17:11:50.330" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29954" PostTypeId="2" ParentId="29947" CreationDate="2018-04-05T17:13:55.493" Score="1" Body="&lt;p&gt;I found &lt;a href=&quot;https://cs.gmu.edu/~carlotta/teaching/CS775-s10/readings/ConsTutorialFinal.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; on &quot;Clustering with Constraints&quot; to be a helpful resource to what you're talking about:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Credit:&#xA;Clustering with Constraints&#xA;Incorporating Prior Knowledge&#xA;into Clustering&#xA;Adapted from a Tutorial of Sugato Basu and Ian Davidson (SDM 2005)&lt;/p&gt;&#xA;" OwnerUserId="50030" LastEditorUserId="50030" LastEditDate="2018-04-06T18:43:19.517" LastActivityDate="2018-04-06T18:43:19.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29955" PostTypeId="1" CreationDate="2018-04-05T17:55:09.940" Score="1" ViewCount="473" Body="&lt;p&gt;What all should I learn in sequence to be expert in Tensorflow and Google Keras provided I know Python and Numpy, Pandas, Pyplot basics? Please provide the resources too as most online tutorials are chargeable&lt;/p&gt;&#xA;" OwnerUserId="47569" LastEditorUserId="28175" LastEditDate="2018-04-05T18:31:44.287" LastActivityDate="2018-05-22T09:40:11.703" Title="Resources to learn Tensorflow and Keras" Tags="|machine-learning|neural-network|deep-learning|keras|tensorflow|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-05T21:09:30.713" ContentLicense="CC BY-SA 3.0" />
  <row Id="29956" PostTypeId="2" ParentId="28809" CreationDate="2018-04-05T18:02:50.163" Score="0" Body="&lt;p&gt;Generally, there is a big risk to the 2-stage approach of (1) classifying who watches at the moment and (2) serving recommendations to the predicted user: if you get stage 1 wrong your recommendations will be terribly off. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you should consider calculating and serving recommendations to the complete household, i.e. a mixture of recommendations to the Dad, Daughter, etc. Additionally, you could change the order based on your user classification. This would give you superior personalization but you won't be completely off if you predict the wrong user.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for specific strategies you could check early Netflix Papers and Blog Posts. If I remember correctly they used to have only one user per account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="44089" LastEditorUserId="44089" LastEditDate="2018-04-06T20:50:13.877" LastActivityDate="2018-04-06T20:50:13.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29957" PostTypeId="2" ParentId="29955" CreationDate="2018-04-05T18:31:14.607" Score="1" Body="&lt;p&gt;Take a look at professor Andrew Ng's course about deep learning. Its homework is written in &lt;code&gt;Keras&lt;/code&gt; and &lt;code&gt;TensorFlow&lt;/code&gt;. It is a very good point to begin.&#xA;&lt;hr /&gt;&#xA;I guess it is not fair to mention non-free lectures because the owners of other non-free lectures may get disappointed but take a look at &lt;a href=&quot;https://www.udemy.com/complete-guide-to-tensorflow-for-deep-learning-with-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;complete guide to TensorFlow for deep learning with python&lt;/a&gt;, which is very useful.&#xA;&lt;hr /&gt;&#xA;I have also found &lt;a href=&quot;http://web.stanford.edu/class/cs20si/syllabus.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;these&lt;/a&gt; lectures from Stanford university very useful. Furthermore, you can take a look at the official paper of google research team about &lt;a href=&quot;https://arxiv.org/pdf/1603.04467&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow&lt;/a&gt; where you can read &lt;em&gt;TensorFlow is an interface for expressing machine learning&#xA;algorithms, and an implementation for executing such algorithms.&#xA;A computation expressed using TensorFlow can be&#xA;executed with little or no change on a wide variety of heterogeneous&#xA;systems, ranging from mobile devices such as phones&#xA;and tablets up to large-scale distributed systems of hundreds&#xA;of machines and thousands of computational devices such as&#xA;GPU cards. The system is flexible and can be used to express&#xA;a wide variety of algorithms, including training and inference&#xA;algorithms for deep neural network models, and it has been&#xA;used for conducting research and for deploying machine learning&#xA;systems into production across more than a dozen areas of&#xA;computer science and other fields, including speech recognition,&#xA;computer vision, robotics, information retrieval, natural&#xA;language processing, geographic information extraction, and&#xA;computational drug discovery. This paper describes the TensorFlow&#xA;interface and an implementation of that interface that&#xA;we have built at Google. The TensorFlow API and a reference&#xA;implementation were released as an open-source package under&#xA;the Apache 2.0 license in November, 2015 and are available at&#xA;www.tensorflow.org.&lt;/em&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-22T09:40:11.703" LastActivityDate="2018-05-22T09:40:11.703" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="29958" PostTypeId="1" AcceptedAnswerId="29961" CreationDate="2018-04-05T18:31:54.900" Score="2" ViewCount="7258" Body="&lt;p&gt;I have an MLP with 3072 input nodes which are for 1024 rgb pixels. My datasets is in an array with each row representing one image and looking like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[red_pix1, red_pix2, ..., red_pix1024, green_pix1, green_pix2, ..., green_pix1024, blue_pix1, blue_pix2, ..., blue_pix1024]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each array value is an integer between 0 and 255.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, before training the network, should I &quot;normalize&quot; my dataset by dividing each element by 255? That way, each input element would have values between 0 and 1. Is this better than having values between 0 and 255? &lt;/p&gt;&#xA;" OwnerUserId="46413" LastActivityDate="2018-04-05T19:37:16.380" Title="When inputting image rgb values to MLP, should I divide by 255?" Tags="|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29959" PostTypeId="1" CreationDate="2018-04-05T18:55:16.337" Score="-1" ViewCount="1093" Body="&lt;p&gt;I am doing regression analysis on a data set with over 20000 samples using scikit learn. Trying to use regression models to fit three features to label which is a score ranges from 0 to 10. Problem is only 100 of the data has a known score. The rest are all unlabeled.&lt;/p&gt;&#xA;&lt;p&gt;Semi-supervised learning seems to work well with classification problems using methods like label-propagation. I wonder if it works for regression problems as well. If so, where can I find any examples for labeling unlabeled data based on similarity?&lt;/p&gt;&#xA;" OwnerUserId="49731" LastEditorUserId="29169" LastEditDate="2021-02-06T18:45:14.207" LastActivityDate="2021-02-06T18:45:14.207" Title="How to do semi-supervised learning for regression" Tags="|scikit-learn|regression|semi-supervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29960" PostTypeId="2" ParentId="29959" CreationDate="2018-04-05T19:21:30.287" Score="0" Body="&lt;p&gt;I also had a similar dataset, I came across &lt;strong&gt;&lt;em&gt;covarite shift&lt;/em&gt;&lt;/strong&gt; technique of Machine Learning. As you have changes in the distribution of the input variables in training data. Though the predictions might also be wrong using this.&lt;/p&gt;&#xA;" OwnerUserId="47569" LastActivityDate="2018-04-05T19:21:30.287" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29961" PostTypeId="2" ParentId="29958" CreationDate="2018-04-05T19:37:16.380" Score="3" Body="&lt;p&gt;The component values are often stored as integer numbers in the range 0 to 255, the range that a single 8-bit byte can offer,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, If you divide by 255  the range can be described with a 0.0-1.0 where 0.0 means 0 (0x00) and 1.0 means 255 (0xFF). &#xA;Normalization will help you to remove distortions caused by lights and shadows in an image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Refer to this &lt;a href=&quot;http://aishack.in/tutorials/normalized-rgb/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Normalize RGB&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47569" LastActivityDate="2018-04-05T19:37:16.380" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29962" PostTypeId="1" AcceptedAnswerId="29963" CreationDate="2018-04-05T19:55:54.780" Score="1" ViewCount="258" Body="&lt;p&gt;What is the difference between Hyper-parameter Tuning and k-NN algorithm? Is k-NN also a type of Hyper-parameter tuning?&lt;/p&gt;&#xA;" OwnerUserId="50038" LastEditorUserId="26562" LastEditDate="2018-04-08T22:05:00.733" LastActivityDate="2018-04-08T22:05:00.733" Title="Hyperparameter Tuning in Machine Learning" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29963" PostTypeId="2" ParentId="29962" CreationDate="2018-04-05T20:01:01.397" Score="0" Body="&lt;p&gt;In kNN algorithm, you only try to find a suitable value of parameter k. And some models may have many parameters that can be modified. Normal parameters are optimized by loss functions and Hyperparameter tuning allows you to set various parameters to get the best model. You set them before training. Its 2 methods: &lt;strong&gt;&lt;em&gt;GRID METHOD AND RANDOM SAMPLING&lt;/em&gt;&lt;/strong&gt; might work well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Grid Method&lt;/strong&gt;:&#xA;Impose a grid on possible space of a hyperparameter and then go over each cell of grid one by one and evaluate your model against values from that cell. Grid method tends to vast resources in trying out parameter values which would not make sense at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Random Sampling Method&lt;/strong&gt;: &#xA;In random method, we have a high probability of finding a good set of params quickly. After doing random sampling for a while, we can zoom into the area indicative of the good set of params.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random sampling allows efficient search in hyperparameter space. But sampling at random does not guarantee uniformity over the range of valid values. Therefore, it is important to pick an appropriate scale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can read this: &lt;a href=&quot;http://datalya.com/blog/2017/hyperparameter-tuning-of-deep-learning-algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hyperparameter Tuning of Deep Learning Algorithm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="47569" LastEditorUserId="47569" LastEditDate="2018-04-07T18:31:06.493" LastActivityDate="2018-04-07T18:31:06.493" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29964" PostTypeId="1" AcceptedAnswerId="29966" CreationDate="2018-04-05T20:07:48.097" Score="1" ViewCount="168" Body="&lt;p&gt;I working on a binary classification task. The dataset is quite small ~1800 rows and ~60 columns. There are no duplicates in the rows. I am comparing different classifiers amongst the canonical ones: random forest, logistic regression, boosted tree and SVC. I am training the hyperparameters by a CV on 90% (train) with 10% held out to measure the generalization error (test). The dataset is slightly unbalances (1 to 3 ratio of classes) hence I used a stratified fold for all splits. I also use roc-auc as a metric for my CV.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get the following results for roc-auc score and accuracy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; DummyClassifier&#xA;Train&#xA;ROC-AUC score: 0.50000&#xA;Accuracy: 0.69705&#xA;Test&#xA;ROC-AUC score: 0.50000&#xA;Accuracy: 0.69545&#xA;&#xA; LogisticRegression&#xA;Train&#xA;ROC-AUC score: 0.88459&#xA;Accuracy: 0.78666&#xA;Test&#xA;ROC-AUC score: 0.72559&#xA;Accuracy: 0.69545&#xA;&#xA; RandomForestClassifier&#xA;Train&#xA;ROC-AUC score: 1.00000&#xA;Accuracy: 0.99695&#xA;Test&#xA;ROC-AUC score: 0.81748&#xA;Accuracy: 0.80455&#xA;&#xA; XGBClassifier&#xA;Train&#xA;ROC-AUC score: 1.00000&#xA;Accuracy: 0.99949&#xA;Test&#xA;ROC-AUC score: 0.80617&#xA;Accuracy: 0.79545&#xA;&#xA; SVC&#xA;Train&#xA;ROC-AUC score: 0.89900&#xA;Accuracy: 0.83248&#xA;Test&#xA;ROC-AUC score: 0.73515&#xA;Accuracy: 0.73182&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is always a significant gap between train and test scores. I am clearly overfitting. I guess it is a consequence of the low number of rows but I am not sure about what to do about that? Force the CV grid search for hyperparameters to a range with strong regularization?&lt;/p&gt;&#xA;" OwnerUserId="39830" LastActivityDate="2018-04-05T20:44:49.223" Title="Significant overfitting with CV" Tags="|classification|class-imbalance|overfitting|binary|metric|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29965" PostTypeId="2" ParentId="20071" CreationDate="2018-04-05T20:12:01.983" Score="2" Body="&lt;p&gt;The FastText binary format (which is what it looks like you're trying to load) isn't compatible with Gensim's &lt;code&gt;word2vec&lt;/code&gt; format; the former contains additional information about subword units, which &lt;code&gt;word2vec&lt;/code&gt; doesn't make use of.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There's some discussion of the issue (and a workaround), on the FastText Github page. In short, you'll have to load the text format (available at &lt;a href=&quot;https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you've loaded the text format, you can use Gensim to save it in binary format, which will dramatically reduce the model size, and speed up future loading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/facebookresearch/fastText/issues/171#issuecomment-294295302&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/facebookresearch/fastText/issues/171#issuecomment-294295302&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13232" LastActivityDate="2018-04-05T20:12:01.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29966" PostTypeId="2" ParentId="29964" CreationDate="2018-04-05T20:44:49.223" Score="2" Body="&lt;p&gt;For the problem of overfitting, you could look train models that employ regularization. For instance &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/svm/plot_svm_scale_c.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; examples shows how to regularize an SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing I noted is that you have used the tag &quot;unbalanced-classes&quot;. If that is the case, accuracy isn't a very good metric. While AUC is good at this, I've personally had trouble with this metric in the past. My suggestion would be to include a metirc like F1-score and most importantly in each case calculate the confusion matrix. This will show you if you are missing one class more than the other. If that is the case you might want to incorporate an oversampling method (e.g. SMOTE) into your pipeline.&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-04-05T20:44:49.223" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29969" PostTypeId="1" CreationDate="2018-04-05T22:50:36.617" Score="5" ViewCount="11918" Body="&lt;p&gt;I'm trying to understand the difference between xgboost.XGBRegressor and  xgboost.sklearn.XGBClassifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain the difference in a concise manner?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because when I fit both classifiers with the exact same data, I get pretty different performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I fit the data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = xgboost.XGBRegressor(alpha=c)&#xA;#clf = xgboost.sklearn.XGBClassifier(alpha=c)&#xA;&#xA;clf.fit(X_train, y_train)&#xA;y_pred = clf.predict(X_test)&#xA;print('Model roc auc score: %0.3f' % roc_auc_score(y_test, y_pred, average='macro', sample_weight=None))&#xA;print('Model pr auc score: %0.3f' % average_precision_score(y_test, y_pred))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;when  clf = xgboost.sklearn.XGBClassifier(alpha=c)&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Model roc auc score: 0.544&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model pr auc score: 0.303&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;when clf = xgboost.XGBRegressor(alpha=c)&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Model roc auc score: 0.703&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model pr auc score: 0.453&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would cause this performance difference?&lt;/p&gt;&#xA;" OwnerUserId="37932" LastEditorUserId="71442" LastEditDate="2022-02-16T10:18:50.203" LastActivityDate="2022-02-16T10:18:50.203" Title="Difference between XGBRegressor and XGBClassifier" Tags="|xgboost|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="29970" PostTypeId="2" ParentId="29969" CreationDate="2018-04-05T23:06:21.443" Score="14" Body="&lt;p&gt;XGBRegressor is for continuous target/outcome variables. These are often called &quot;regression problems.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBClassifier is for categorical target/outcome variables. These are often called &quot;classification problems.&quot;&lt;/p&gt;&#xA;" OwnerUserId="13373" LastActivityDate="2018-04-05T23:06:21.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29972" PostTypeId="1" AcceptedAnswerId="29976" CreationDate="2018-04-06T03:05:30.573" Score="-2" ViewCount="189" Body="&lt;p&gt;I am applying a active learning with a SGDClassifier (log loss function) as the base learner on some data and I have the following graphs representing the learning curve of queries vs error rate. The green is the validation error and blue is the training error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is my model overfitting or has high variance in both graphs? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/M0TQd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/M0TQd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ruOAU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ruOAU.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46055" LastEditorUserId="46055" LastEditDate="2018-04-06T03:45:50.860" LastActivityDate="2018-04-06T06:27:52.763" Title="Help Interpreting a Active Learning Learning Curve" Tags="|machine-learning|data-mining|active-learning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="29975" PostTypeId="1" CreationDate="2018-04-06T06:25:08.840" Score="3" ViewCount="1261" Body="&lt;p&gt;I am working on multi-class classification problem on an image dataset. There is one class with 80% of the images and rest 20% is divided into rest 6 remaining classes. If I have to apply the image-augmentation technique, what ratio of count of images between each class I have to maintain?&lt;/p&gt;&#xA;" OwnerUserId="50056" LastEditorUserId="29169" LastEditDate="2021-02-13T16:34:11.333" LastActivityDate="2021-04-19T00:06:41.010" Title="How to deal with skewed imbalanced image dataset to work with CNN?" Tags="|deep-learning|neural-network|cnn|convolutional-neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29976" PostTypeId="2" ParentId="29972" CreationDate="2018-04-06T06:27:52.763" Score="2" Body="&lt;p&gt;&lt;strong&gt;Overfitting&lt;/strong&gt; looks more likely because:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;After some queries your validation errors are systematically higher than training errors, which is probably not what you want.&lt;/li&gt;&#xA;&lt;li&gt;After some queries, your training errors &lt;strong&gt;slowly fall&lt;/strong&gt; while validation errors remain &lt;strong&gt;constant&lt;/strong&gt;. It's like your classifier is memorising your data set at a slow pace.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;When the number of queries is small, it looks like your classifier did better simply because it was a simpler data set. Again, overfitting can also be an issue.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2018-04-06T06:27:52.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29979" PostTypeId="1" CreationDate="2018-04-06T09:53:48.620" Score="1" ViewCount="987" Body="&lt;p&gt;I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training steps: 20,000&#xA;learning rate: 0.075&#xA;test accuracy :72.9%&#xA;train accuracy for final iteration : 98.0%&#xA;val accuracy for final iteration : 75.0%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;clearly overfitting, so I tried &lt;strong&gt;L2 regularization&lt;/strong&gt;. Here are my parameters and results, for the highest accuracy until now :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training steps: 40,000&#xA;learning rate: 0.1&#xA;test accuracy :71.1%&#xA;alpha for L2 : 0.00075&#xA;train accuracy for final iteration : 91.0%&#xA;val accuracy for final iteration : 71.0%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looked promising so I decided to go further with the following hoping to get better accuracy :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training steps: 80,000&#xA;learning rate: 0.075&#xA;alpha for L2 : 0.005&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;go the following graph :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SvaSi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SvaSi.png&quot; alt=&quot;tensorboard graph&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since, the graph was getting saturated so I quit at 50,000 steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My question&lt;/strong&gt; : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes&lt;/p&gt;&#xA;" OwnerUserId="44673" LastActivityDate="2018-08-04T14:00:56.407" Title="trying to decrease overfitting with regularisation in CNN" Tags="|deep-learning|cnn|accuracy|regularization|hyperparameter-tuning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29980" PostTypeId="2" ParentId="29979" CreationDate="2018-04-06T10:35:51.987" Score="1" Body="&lt;p&gt;Steps for overcoming overfitting are the followings:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; 1. Add more data  &#xA; 2. Use data augmentation  &#xA; 3. Use architectures that generalize well  &#xA; 4. Add regularization  &#xA; 5. Reduce architecture complexity.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.&lt;/p&gt;&#xA;" OwnerUserId="45431" LastActivityDate="2018-04-06T10:35:51.987" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="29982" PostTypeId="1" AcceptedAnswerId="29983" CreationDate="2018-04-06T10:58:59.973" Score="0" ViewCount="69" Body="&lt;p&gt;I believe some month ago I have read somewhere that autoencoders can respond better to sparse input data when trained with such. For example when the training data is not sparse, but many features of the test data are zero. Apparently randomly setting some features of the training data to zero can help to perform the model better on the test set. I desperatly tried to find the book/paper where I read it again, but I was not able to and I cannot remember the name for this. Does somebody know what this technique is called?&lt;/p&gt;&#xA;" OwnerUserId="50069" LastActivityDate="2018-04-06T11:16:31.373" Title="Technique to make NN (specifically autoencoders) more robust by artifcally making input data more sparse" Tags="|neural-network|deep-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29983" PostTypeId="2" ParentId="29982" CreationDate="2018-04-06T11:16:31.373" Score="1" Body="&lt;p&gt;I've finally found it. The term I was looking for was '&lt;strong&gt;masking corruption'&lt;/strong&gt; and is mentioned in 'Practical Recommendations for Gradient-Based Training of Deep Architectures' by Yoshua Bengio.&lt;/p&gt;&#xA;" OwnerUserId="50069" LastActivityDate="2018-04-06T11:16:31.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29984" PostTypeId="1" CreationDate="2018-04-06T12:39:26.443" Score="3" ViewCount="1723" Body="&lt;p&gt;I have a very large dataset with timestamp data. Till now, I loaded the whole dataset in order to train some models using python (statsmodels.api,  statsmodels.formula.api, keras.models.Sequential).&#xA;Now, I would like to train models with all historical data just one time, and then update the model just with new data (each day I have new samples).&#xA;Is it possible and how with these python libraries?&#xA;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50074" LastActivityDate="2019-07-27T09:04:21.137" Title="Incremental learning with new samples" Tags="|machine-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29985" PostTypeId="2" ParentId="29984" CreationDate="2018-04-06T13:02:29.657" Score="2" Body="&lt;p&gt;This is what Machine Learning models are used for..(to predict what they think will happen in the near time based on the data they have been fed into....)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple answer is this &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first couple days of data are the most important , you need to gauge and have a watch over your model..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As with any incremental learning, you can learn more recent stuff but also underfit the past significantly more than without learning the new stuff.&#xA;Therefore, it's why there's a monitoring to perform on models which use incremental learning (especially on production systems).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Provided that other things remains similar with time..(same Preprocessing, same scaling etc..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But let's say if you have tons of data everyday, then nearly at the month end, the model can be completely trained on the last months data or else checkpoints needs to be kept &lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-06T17:51:40.553" LastActivityDate="2018-04-06T17:51:40.553" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29987" PostTypeId="1" AcceptedAnswerId="30034" CreationDate="2018-04-06T13:36:27.980" Score="-3" ViewCount="165" Body="&lt;p&gt;I would like to analyse some text and most of my Reviews are german. Does anyone know if python has a good gunning fog index function for german language? I couldnt find anything&lt;/p&gt;&#xA;&#xA;&lt;p&gt;best regards&lt;/p&gt;&#xA;" OwnerUserId="50077" LastActivityDate="2018-04-08T15:19:02.770" Title="german gunning fog index function" Tags="|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29988" PostTypeId="2" ParentId="29975" CreationDate="2018-04-06T13:50:37.127" Score="1" Body="&lt;p&gt;I think that you first need to consider what are the class proportions in the data that you are going to eventually use the model for. It is important to have a test set that comes from the same data generating distribution as the real data. Now, in case the proportions in this data are the same as you wrote than it is actually a good thing to have more examples for the classes that are more common. It allows your model to give those classes a bigger prior and focus more on those classes, extract more features relevant to those classes particularly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case you'd like to reduce this skew (for example because the proportions in the real data are different or the imbalance is so big that your model learns to assign always just the most common class and ignores others) you can use resampling: define what class proportions would suit you and randomly sample examples using those ratios.&lt;/p&gt;&#xA;" OwnerUserId="50076" LastActivityDate="2018-04-06T13:50:37.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29990" PostTypeId="1" CreationDate="2018-04-06T16:15:33.807" Score="2" ViewCount="3891" Body="&lt;p&gt;I'm interested in exporting a correlation matrix to csv. I've tried using the to.csv functionality, but my matrix is obtained from a function and I get an &quot;object has no attribute 'to_csv'&quot; error. I appreciate any guidance. Code is below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from matplotlib import pyplot as plt&#xA;from matplotlib import cm as cm&#xA;&#xA;def correlation_matrix(df):&#xA;&#xA;    fig = plt.figure()&#xA;    ax1 = fig.add_subplot(111)&#xA;    cmap = cm.get_cmap('jet', 30)&#xA;    cax = ax1.imshow(df.corr(), interpolation=&quot;nearest&quot;, cmap=cmap)&#xA;    ax1.grid(True)&#xA;    plt.title('Feature Correlation')&#xA;    labels = None&#xA;    # Add colorbar, and make sure to specify tick locations to match desired ticklabels.&#xA;    cbar = fig.colorbar(cax, ticks=[-1.1, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6,.8,1])&#xA;    plt.show()&#xA;&#xA;correlation_matrix(pd.DataFrame(np.c_[Xs,Y]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50083" LastEditorUserId="26562" LastEditDate="2018-04-06T23:22:05.403" LastActivityDate="2020-09-05T12:00:30.043" Title="Exporting Correlation Matrix (from function)" Tags="|machine-learning|feature-selection|numpy|matplotlib|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29991" PostTypeId="1" CreationDate="2018-04-06T16:19:02.620" Score="0" ViewCount="95" Body="&lt;p&gt;I have a sequential data from time &lt;code&gt;T1&lt;/code&gt; to &lt;code&gt;T6&lt;/code&gt;. The rows contain the sequence of states for 50 customers. There are only 3 states in my data. For example, it looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        T1  T2  T3  T4  T5  T6&#xA;Cust1   C   B   C   A   A   C&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My transition matrix &lt;code&gt;X&lt;/code&gt; looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    A   B   C&#xA;A   0.3 0.6 0.1&#xA;B   0.5 0.2 0.3&#xA;C   0.7 0.1 0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, we see that at time &lt;code&gt;T6&lt;/code&gt; the state is at &lt;code&gt;C&lt;/code&gt; which corresponds to &lt;code&gt;c=[0 0 1]&lt;/code&gt; vector.&#xA;I am now predicting &lt;code&gt;T7&lt;/code&gt; by doing the matrix multiplication: &lt;code&gt;c * X&lt;/code&gt; which gives me &lt;code&gt;[0.7 0.1 0.2]&lt;/code&gt;. Based on this, I decide that the state at &lt;code&gt;T7&lt;/code&gt; would be &lt;code&gt;A&lt;/code&gt; (highest prob. value).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For &lt;code&gt;T8&lt;/code&gt;, I use the result of the probability vector I got above and do &lt;code&gt;[0.7 0.1 0.2]*X = [0.4 0.44 0.14]&lt;/code&gt;  and decide that the state is &lt;code&gt;B&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: Am I doing something wrong? Am I contradicting the memoryless property of the Markov Chains?&lt;/p&gt;&#xA;" OwnerUserId="49598" LastEditorUserId="24000" LastEditDate="2018-04-07T21:01:02.617" LastActivityDate="2023-02-28T11:07:00.847" Title="Simple Markov Chains Memoryless Property Question" Tags="|markov-process|markov|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29993" PostTypeId="2" ParentId="26597" CreationDate="2018-04-06T16:40:41.213" Score="5" Body="&lt;p&gt;Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Look for similar problems that have also been modeled with feed-forward networks and study their architectures.&lt;/li&gt;&#xA;&lt;li&gt;Begin with that configuration, train the data set and evaluate the test set.&lt;/li&gt;&#xA;&lt;li&gt;Perform &lt;a href=&quot;http://www.inference.vc/pruning-neural-networks-two-recent-papers/&quot; rel=&quot;noreferrer&quot;&gt;pruning&lt;/a&gt; in your architecture and compare results in the data set with the previous results. If the accuracy of your model is not affected so you can infer that the original model is overfitting the data.&lt;/li&gt;&#xA;&lt;li&gt;Otherwise, try adding more degrees of freedom (i.e. more layers).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The general approach is to try different architectures, compare results and take the best configuration. Experience gives you more intuition in the first architecture guess.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-06T16:40:41.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29994" PostTypeId="2" ParentId="29947" CreationDate="2018-04-06T19:04:36.577" Score="0" Body="&lt;p&gt;You are describing a variation on &lt;a href=&quot;https://en.wikipedia.org/wiki/Mathematical_optimization&quot; rel=&quot;nofollow noreferrer&quot;&gt;optimization&lt;/a&gt;, maximize a real function by systematically choosing input values from within an allowed set and computing the value of the function. Specifically, a variation called &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_programming&quot; rel=&quot;nofollow noreferrer&quot;&gt;linear programming&lt;/a&gt; where you to maximize the similarity (similarity is the inverse of distance) subject to a couple of constraints:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QOIE6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QOIE6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-06T19:04:36.577" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="29995" PostTypeId="2" ParentId="29600" CreationDate="2018-04-06T19:16:09.203" Score="2" Body="&lt;p&gt;You don't want clustering.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you are looking for is &lt;strong&gt;near duplicate detection&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use minhash. Apparently that is what Google News uses for exactly this purpose.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-06T19:16:09.203" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="29996" PostTypeId="2" ParentId="29941" CreationDate="2018-04-06T20:16:03.310" Score="1" Body="&lt;p&gt;The framework you describe is the Bandits framework. Bandits are algorithms that solve stateless MDPs. So the problem is not sequential in the sense that actions affect the transition to the next state. Take a look at a nice &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-1-5-contextual-bandits-bff01d1aad9c&quot; rel=&quot;nofollow noreferrer&quot;&gt;Contextual Bandits&lt;/a&gt; tutorial which is accompanied with Python Jupyter notebook. Please note that Contextual Bandits are a special category of Bandits but at least you will be exposed to the general Bandit idea and formalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try Q learning but with the Rescorla-Wagner rule $r(a) - Q(a)$ as the reward correction $\gamma\max_a' Q(s',a')$ does not make any sense as there is no transition to a subsequent state $s'$.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="18626" LastEditDate="2018-04-06T20:23:53.670" LastActivityDate="2018-04-06T20:23:53.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29997" PostTypeId="2" ParentId="29951" CreationDate="2018-04-06T20:22:16.260" Score="1" Body="&lt;p&gt;In &lt;strong&gt;constrained clustering&lt;/strong&gt; you can provide examples of objects that should, or that must not, be in the same cluster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This can be used, e.g., for model selection: run several times and return the result with the fewest violations. Or to guide cluster extraction from a hierarchical result.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-06T20:22:16.260" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="29998" PostTypeId="1" CreationDate="2018-04-06T20:30:33.243" Score="1" ViewCount="25" Body="&lt;p&gt;I have 128 bin FFT data for an audio file. I want to convert these FFT into MFCC of size 14, 20, 40. Is there some library which converts directly FFT to MFCC.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S - I dont have the actual audio file.&lt;/p&gt;&#xA;" OwnerUserId="25297" LastActivityDate="2018-04-06T20:30:33.243" Title="Converting MFCC to FFT" Tags="|machine-learning|feature-extraction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30001" PostTypeId="2" ParentId="29990" CreationDate="2018-04-07T00:30:55.510" Score="2" Body="&lt;p&gt;Don't use the matrix obtained from your function rather,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.corr.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;correlation&lt;/code&gt; is a DataFrame too&lt;/a&gt;, so you can simply use &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;to_csv&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;correlation_df(within your function itself).to_csv(&quot;C:\destinationfolder\file.csv&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-04-07T00:30:55.510" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30006" PostTypeId="1" AcceptedAnswerId="30087" CreationDate="2018-04-07T04:12:08.127" Score="2" ViewCount="3193" Body="&lt;p&gt;Is there a simple example to start with for using &lt;a href=&quot;https://pypi.python.org/pypi/kohonen&quot; rel=&quot;nofollow noreferrer&quot;&gt;kohonen 1.1.2&lt;/a&gt; or is it only the &lt;a href=&quot;https://github.com/lmjohns3/kohonen/blob/master/kohonen_test.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;test file&lt;/a&gt; that will be the reference? &lt;/p&gt;&#xA;" OwnerUserId="31443" LastActivityDate="2018-04-09T17:26:27.877" Title="Examples on kohonen self organizing maps" Tags="|python|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30007" PostTypeId="1" CreationDate="2018-04-07T07:06:46.060" Score="3" ViewCount="50" Body="&lt;p&gt;When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is there a better way to articulate this reasoning?&lt;/p&gt;&#xA;" OwnerUserId="50102" LastActivityDate="2018-05-24T10:15:37.920" Title="Can the 'bin size' in a histogram be thought of as a regularity constraint?" Tags="|machine-learning|statistics|visualization|probability|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30008" PostTypeId="1" AcceptedAnswerId="30009" CreationDate="2018-04-07T09:02:07.377" Score="1" ViewCount="130" Body="&lt;p&gt;So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data for that particular input in training data set.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; x1          x2           x3          y&#xA; 10          15           20          0&#xA; 20           7           10          1&#xA; 5           10           4           0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So imagine this is an example training set, we send these inputs to activation function, and for the first input it returns correct output (0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for second output it again returns 0, so the weights are adjusted until the activation function returns 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now my doubt is, if the new updated weights returns the wrong output for the third input, its weight gets changed again, but will there be a situation where these weights will not satisfy for the previously tested inputs, like for example the first input in this case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a chance that new weights will return 1 for the first input, which is wrong? &lt;/p&gt;&#xA;" OwnerUserId="30653" LastEditorUserId="29575" LastEditDate="2018-04-07T14:17:28.690" LastActivityDate="2018-04-07T18:38:22.917" Title="Weights in neural network" Tags="|neural-network|deep-learning|activation-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30009" PostTypeId="2" ParentId="30008" CreationDate="2018-04-07T09:41:06.983" Score="2" Body="&lt;p&gt;In parametric models such as linear regression, logistic regression and multi-layers perceptrons, weights are updated with regards to the &quot;&lt;em&gt;difference&lt;/em&gt;&quot; between the output of your model and the real label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More precisely, weights are updated using the gradient descent / backpropagation procedure. It is composed of two parts : &lt;strong&gt;the forward pass&lt;/strong&gt; and &lt;strong&gt;the backward pass&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a given observation (or a set of observations), the forward pass is about feeding the model with observations and output a result &lt;strong&gt;a&lt;/strong&gt;. This output &quot;a&quot; is then compared with the real value, the label &lt;strong&gt;y&lt;/strong&gt;. Using some cost function metrics (such as Absolute Error or Square Error for regression purposes,  cross-entropy for classification purposes...), we can then compute &lt;strong&gt;j(y,a)&lt;/strong&gt; which is the error between the output and the real value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3gxBy.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3gxBy.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can now run &lt;strong&gt;the backward pass&lt;/strong&gt; which is about computing the derivative of the cost function with regards to any weight / bias coefficient in the logistic regression / neural network. We can the update coefficients such as :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/g9H2F.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/g9H2F.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/9wP3J.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9wP3J.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where alpha is the learning rate. So to answer your question, weights are not updated until they reach the expected value. We are just trying to reach the minimum in cost function surface by running gradient descent procedure.&lt;/p&gt;&#xA;" OwnerUserId="18434" LastActivityDate="2018-04-07T09:41:06.983" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30010" PostTypeId="2" ParentId="30008" CreationDate="2018-04-07T09:48:41.093" Score="1" Body="&lt;p&gt;Given the sampled three rows,  x1 and x2 look as if they can determine y, and x3 can be left unconsidered, or in other words, x1 and x2 seem to correlate with y&#xA;.&lt;br&gt;&#xA;As we are not discussing any semantics, do not know what these figure mean, new training data can void the model, even make y practically unpredictable, finally. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is just an assumption, not guaranteed, that there is a significant correlation at allx1,x2,and x3 are enough data points to ćreate a model that can be used to predict y with sufficient precision. &lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-07T18:38:22.917" LastActivityDate="2018-04-07T18:38:22.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30013" PostTypeId="1" AcceptedAnswerId="30135" CreationDate="2018-04-07T10:54:46.880" Score="6" ViewCount="1663" Body="&lt;p&gt;I have age, gender, height, weight and some other similar parameters of 15000 subjects. I also have one column showing if they had a medical condition (present in about 20% subjects). I now want to analyze the factors which are most contributing to the medical condition. The usual test for this is Logistic regression analysis. However, that assumes that the relationships are linear. What other machine learning tests can I apply for this purpose with this kind of data? Thanks for your insight.&lt;/p&gt;&#xA;" OwnerUserId="49700" LastEditorUserId="28175" LastEditDate="2018-04-10T15:10:29.803" LastActivityDate="2018-04-11T11:37:52.450" Title="Alternatives to Logistic Regression" Tags="|machine-learning|neural-network|deep-learning|classification|machine-learning-model|" AnswerCount="4" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="30014" PostTypeId="2" ParentId="29991" CreationDate="2018-04-07T11:07:57.867" Score="0" Body="&lt;p&gt;I think you are not doing anything wrong, the markov property is satisfied when the prediction can be solely based on the present state. I do not think you are contradicting this &quot;memoryless&quot; condition (which applies to the whole chain) when you predict what could probaly happen after a last state that is already just predicted.   &lt;/p&gt;&#xA;" OwnerUserId="21163" LastActivityDate="2018-04-07T11:07:57.867" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30015" PostTypeId="1" AcceptedAnswerId="30317" CreationDate="2018-04-07T11:20:34.323" Score="3" ViewCount="261" Body="&lt;p&gt;Traditionally, a state for RNN is computed as &#xA;$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single matrix times a concatenated vector:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$W_m[x, h_{t-1}]$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $[...]$ is concatenation.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In other words, we would end up with a long vector like $\{x_1, x_2, x_3, h_{1,t-1}, h_{2,t-1}, h_{3,t-1} \}$ multiplied by $W_m$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like the second approach would have a significantly larger matrix, which has more elements than $W$ and $U$ combined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does that mean $W$ and $U$ are a simplification, &lt;strong&gt;what do we lose by using them, and adding up the results?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-04-12T16:37:02.853" LastActivityDate="2019-04-12T16:37:02.853" Title="RNN: why Wx + Uh instead of W[x,h]" Tags="|machine-learning|linear-algebra|rnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30017" PostTypeId="2" ParentId="2464" CreationDate="2018-04-07T13:04:34.663" Score="0" Body="&lt;p&gt;Complementary to the previous answer you should better not just run kmeans directly on the compositional data derived from the lda topic-doc distribution, instead use some compositional data transformation to project them to the euclidean space like ilr or clr. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(&lt;a href=&quot;https://arxiv.org/pdf/1704.06150.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Example&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="50118" LastEditorUserId="29575" LastEditDate="2018-04-07T14:11:10.877" LastActivityDate="2018-04-07T14:11:10.877" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30020" PostTypeId="1" AcceptedAnswerId="30707" CreationDate="2018-04-07T17:01:48.320" Score="1" ViewCount="46" Body="&lt;p&gt;In R Markdown, I would like to be able to conditionally include certain chunks depending on what I am outputting to. For example if I am knitting a PDF I would like to include a small bit of LaTeX to do a page break, but if I am knitting HTML from the same document, I would like to skip that, or perhaps include something else. So how can a document &quot;know&quot; where it is going, and selectively include things?&lt;/p&gt;&#xA;" OwnerUserId="32402" LastActivityDate="2018-04-23T18:32:01.887" Title="Different R Markdown in same document for each output type" Tags="|r|rstudio|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30022" PostTypeId="1" CreationDate="2018-04-07T18:06:36.943" Score="1" ViewCount="1452" Body="&lt;p&gt;I am new to pytorch and Deep learning. I am trying to do image segmentation.&#xA;But , I am stuck at how to label training set images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please help me ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one of my training image&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/awyRU.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/awyRU.jpg&quot; alt=&quot;Crop and weed classification&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have two kinds of plants here - one is weed and another one is a good crop. I need to label them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me how can I do this ?&lt;/p&gt;&#xA;" OwnerUserId="44477" LastEditorUserId="44477" LastEditDate="2018-04-07T18:26:51.673" LastActivityDate="2020-07-02T12:57:46.323" Title="Image segmentation training set labeling" Tags="|machine-learning|python|deep-learning|pytorch|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30023" PostTypeId="2" ParentId="30022" CreationDate="2018-04-07T18:24:11.263" Score="1" Body="&lt;p&gt;If you have the labeled training data with you then, you will have to label each pixel in the training set, compute the loss and backpropagate it. The link given below should help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Image_segmentation#Supervised_image_segmentation_using_MRF_and_MAP&quot; rel=&quot;nofollow noreferrer&quot;&gt;Suprevised Image Segmentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50130" LastActivityDate="2018-04-07T18:24:11.263" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30024" PostTypeId="2" ParentId="155" CreationDate="2018-04-07T18:45:33.587" Score="2" Body="&lt;p&gt;These are the most used ones:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.kaggle.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;: The Home of Data Science &amp;amp; Machine Learning. Kaggle helps you learn, work, and play. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://data.gov.in&quot; rel=&quot;nofollow noreferrer&quot;&gt;Govt. Repository Data&lt;/a&gt;: USCG Marine Casualty and Pollution Data&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quandl.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quandl&lt;/a&gt;: Quandl is a marketplace for financial, economic and alternative data delivered in modern formats for today's analysts, including Python, Excel, Matlab, R, and via our API.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;UCI WEB&lt;/a&gt;: Classification - Regression - Adult - Abalone - Life Sciences&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47569" LastActivityDate="2018-04-07T18:45:33.587" CommentCount="0" CommunityOwnedDate="2018-04-07T18:45:33.587" ContentLicense="CC BY-SA 3.0" />
  <row Id="30025" PostTypeId="1" CreationDate="2018-04-07T21:47:42.460" Score="1" ViewCount="34" Body="&lt;p&gt;I curiously want to know why in many references &lt;sup&gt;&lt;strong&gt;[1][2]&lt;/strong&gt;&lt;/sup&gt; on back propagation, authors use the &lt;em&gt;partial derivative&lt;/em&gt; symbol, i.e $\frac{\partial}{\partial W}$, instead of correctly alternating between it and &lt;em&gt;total derivative symbol&lt;/em&gt;, i.e $\frac{\text{d}}{\text{dW}}$?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doesn't it make the back-propagation formula's derivation context more ambiguous?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1]: Wikipedia - Back propagation - &lt;a href=&quot;https://en.wikipedia.org/wiki/Backpropagation&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Backpropagation&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2]: Stanford Wiki - UFLDL- &lt;a href=&quot;http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50106" LastEditorUserId="1330" LastEditDate="2021-07-10T23:29:27.477" LastActivityDate="2021-07-10T23:29:27.477" Title="Should there be 'total derivative' symbol in the mathematical representation of back-propagation algorithm's formula?" Tags="|machine-learning|backpropagation|mathematics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30026" PostTypeId="1" AcceptedAnswerId="30058" CreationDate="2018-04-07T22:39:29.150" Score="2" ViewCount="6975" Body="&lt;p&gt;I want to find $(x,y)$ which minimizes $x^2+y^2$ with GA to apply it for another function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know any example of GA with deap (Python) like that?&lt;/p&gt;&#xA;" OwnerUserId="49685" LastEditorDisplayName="user10640" LastEditDate="2020-11-05T16:59:43.280" LastActivityDate="2020-11-05T16:59:43.280" Title="How do I find the minimum value of $x^2+y^2$ with a genetic algorithm?" Tags="|genetic-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30027" PostTypeId="2" ParentId="28499" CreationDate="2018-04-08T04:39:24.783" Score="1" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;training_set = train_datagen.flow_from_directory(training_path,&#xA;                                             target_size = (64, 64),&#xA;                                             batch_size = 32,&#xA;                                             class_mode = 'binary')&#xA;&#xA;&#xA;imgs, labels = next(training_set)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50143" LastActivityDate="2018-04-08T04:39:24.783" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30028" PostTypeId="1" CreationDate="2018-04-08T05:49:21.583" Score="2" ViewCount="294" Body="&lt;p&gt;So I am following this &lt;a href=&quot;https://github.com/llSourcell/ethereum_future&quot; rel=&quot;nofollow noreferrer&quot;&gt;notebook&lt;/a&gt;(at least for the data portion)and have a trained model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to do is true future predictions as LSTM are capable of generating data for unseen time steps(if they are stateful). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Trying to follow this post &lt;a href=&quot;https://stackoverflow.com/questions/47594861/predicting-a-multiple-time-step-forward-of-a-time-series-using-lstm&quot;&gt;here&lt;/a&gt; as it is the only thing I can find on this concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;here is the default function: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def load_data(filename, sequence_length):&#xA;    # Read the data file&#xA;    raw_data = pd.read_csv(filename, dtype=float)&#xA;    raw_data = raw_data.values&#xA;    print (raw_data)&#xA;&#xA;    # Change all zeros to the number before the zero occurs&#xA;    for x in range(0, raw_data.shape[0]):&#xA;        for y in range(0, raw_data.shape[1]):&#xA;            if (raw_data[x][y] == 0):&#xA;                raw_data[x][y] = raw_data[x - 1][y]&#xA;&#xA;    # Convert the file to a list&#xA;    data = raw_data.tolist()&#xA;&#xA;    # Convert the data to a 3D array (a x b x c)&#xA;    # Where a is the number of days, b is the window size, and c is the number of features in the data file&#xA;    result = []&#xA;    for index in range(len(data) - sequence_length):&#xA;        result.append(data[index: index + sequence_length])&#xA;&#xA;    # Normalizing data by going through each window&#xA;    # Every value in the window is divided by the first value in the window, and then 1 is subtracted&#xA;    d0 = np.array(result)&#xA;    dr = np.zeros_like(d0)&#xA;    dr[:, 1:, :] = d0[:, 1:, :] / d0[:, 0:1, :] - 1&#xA;&#xA;    # Keeping the unnormalized prices for Y_test&#xA;    # Useful when graphing bitcoin price over time later&#xA;    start = 2400&#xA;    end = int(dr.shape[0] + 1)&#xA;    unnormalized_bases = d0[start:end, 0:1, 7]&#xA;&#xA;    # Splitting data set into training (First 90% of data points) and testing data (last 10% of data points)&#xA;    split_line = round(0.9 * dr.shape[0])&#xA;    training_data = dr[:int(split_line), :]&#xA;&#xA;    # Shuffle the data&#xA;    np.random.shuffle(training_data)&#xA;&#xA;    # Training Data&#xA;    X_train = training_data[:, :-1]&#xA;    Y_train = training_data[:, -1]&#xA;    Y_train = Y_train[:, 7]&#xA;&#xA;    # Testing data&#xA;    X_test = dr[int(split_line):, :-1]&#xA;    Y_test = dr[int(split_line):, 6, :]&#xA;    Y_test = Y_test[:, 7]&#xA;&#xA;    # Get the day before Y_test's price&#xA;    Y_daybefore = dr[int(split_line):, 5, :]&#xA;    Y_daybefore = Y_daybefore[:, 7]&#xA;&#xA;    # Get window size and sequence length&#xA;    sequence_length = sequence_length&#xA;    window_size = sequence_length - 1  # because the last value is reserved as the y value&#xA;&#xA;    return X_train, Y_train, X_test, Y_test, Y_daybefore, unnormalized_bases, window_size&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and trying to refactor according to the post:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def load_data_future(filename, sequence_length):&#xA;    # Read the data file&#xA;    raw_data = pd.read_csv(filename, dtype=float)&#xA;    raw_data = raw_data.values&#xA;    print (raw_data)&#xA;&#xA;    # Change all zeros to the number before the zero occurs&#xA;    for x in range(0, raw_data.shape[0]):&#xA;        for y in range(0, raw_data.shape[1]):&#xA;            if (raw_data[x][y] == 0):&#xA;                raw_data[x][y] = raw_data[x - 1][y]&#xA;&#xA;    # Convert the file to a list&#xA;    data = raw_data.tolist()&#xA;&#xA;    # Convert the data to a 3D array (a x b x c)&#xA;    # Where a is the number of days, b is the window size, and c is the number of features in the data file&#xA;    result = []&#xA;    for index in range(len(data) - sequence_length):&#xA;        result.append(data[index: index + sequence_length])&#xA;&#xA;    # Normalizing data by going through each window&#xA;    # Every value in the window is divided by the first value in the window, and then 1 is subtracted&#xA;    d0 = np.array(result)&#xA;    dr = np.zeros_like(d0)&#xA;    dr[:, 1:, :] = d0[:, 1:, :] / d0[:, 0:1, :] - 1&#xA;&#xA;    # Keeping the unnormalized prices for Y_test&#xA;    # Useful when graphing bitcoin price over time later&#xA;    start = 2400&#xA;    end = int(dr.shape[0] + 1)&#xA;    unnormalized_bases = d0[start:end, 0:1, 7]&#xA;&#xA;    # Splitting data set into training (First 90% of data points) and testing data (last 10% of data points)&#xA;    split_line = round(1 * dr.shape[0])&#xA;    training_data = dr[:int(split_line), :]&#xA;&#xA;    # Shuffle the data&#xA;    np.random.shuffle(training_data)&#xA;&#xA;    #Reformatting training data&#xA;    X_train = training_data[:,:-1]&#xA;    Y_train = training_data[:,1]&#xA;    #Y_train = Y_train[:, 1]&#xA;&#xA;&#xA;    # Get the day before Y_test's price&#xA;    Y_daybefore = dr[int(split_line):, 5, :]&#xA;    Y_daybefore = Y_daybefore[:, 7]&#xA;&#xA;    # Get window size and sequence length&#xA;    sequence_length = sequence_length&#xA;    window_size = sequence_length - 1  # because the last value is reserved as the y value&#xA;    entire_data = dr&#xA;&#xA;    return X_train, Y_train, Y_daybefore, unnormalized_bases, window_size,entire_data&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What am I doing wrong??&#xA;I can see that its formatted samples, outputs,timesteps per feature. Not sure how to manipulate this to acheive what I am going for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even high level pointers would be helpful in the comments! &lt;/p&gt;&#xA;" OwnerUserId="50145" LastEditorUserId="50145" LastEditDate="2018-04-10T22:29:00.537" LastActivityDate="2018-04-10T22:29:00.537" Title="Reformatting data for future time series prediction LSTM(Keras)" Tags="|keras|time-series|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30029" PostTypeId="1" CreationDate="2018-04-08T10:08:01.360" Score="1" ViewCount="80" Body="&lt;p&gt;I want to create a content based filtering recommender system which has multiple seeds. All that I have read about is having an initial seed from which the recommendations should be similar to. This is particularly helpful in situations where you want to create movie recommendations that should be interesting to a group of people. So I need a &lt;code&gt;query seed&lt;/code&gt; which is a single row of movie features to feed to the recommender. So how do I create a &lt;code&gt;query seed&lt;/code&gt; based from multiple rows of movie features?&lt;/p&gt;&#xA;" OwnerUserId="50155" LastEditorUserId="50155" LastEditDate="2018-04-09T04:43:47.113" LastActivityDate="2018-04-09T04:43:47.113" Title="Multiple Seeds Content Based filtering Recommender System" Tags="|data-mining|dataset|recommender-system|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30030" PostTypeId="1" AcceptedAnswerId="33223" CreationDate="2018-04-08T10:23:32.257" Score="2" ViewCount="1011" Body="&lt;p&gt;Is there any R package that supports fitting an HMM using multiple sequences of observations? to the best of my knowledge depmixS4 does not support this feature&lt;/p&gt;&#xA;" OwnerUserId="50156" LastActivityDate="2018-08-04T05:48:57.977" Title="depmixS4 - How to fit HMM usgin multiple sequences of observation" Tags="|r|markov-hidden-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30031" PostTypeId="2" ParentId="29952" CreationDate="2018-04-08T10:43:24.713" Score="1" Body="&lt;p&gt;Yes, it is time series with missing values. In fact you have process $X_t$, which is room temperature in moment $t$, but you have observations in specific time intervals. So your goal is to find out missing values between non-observed intervals.  There are several methods and packages for solving problems like this, &lt;a href=&quot;https://www.rdocumentation.org/packages/imputeTS/versions/2.6&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; is an implementation of some methods in R:&lt;/p&gt;&#xA;" OwnerUserId="50154" LastEditorUserId="29575" LastEditDate="2018-04-08T15:57:49.077" LastActivityDate="2018-04-08T15:57:49.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30032" PostTypeId="1" CreationDate="2018-04-08T11:34:57.240" Score="4" ViewCount="1687" Body="&lt;p&gt;In the MNIST dataset, you have 10 defined classes, one for each digit. But you don't have a &quot;not a digit&quot; class. It seems that most image classification datasets are the same. But in a business setting, for a production model, you could certainly get invalid images that don't correspond to any of the defined classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So let's say you were to create a handwritten digit image classifying model to be used in a real-world project. If you don't have a &quot;not a digit&quot; class, then if somebody submits a picture of the letter &quot;M&quot;, then it will incorrectly be classified as one of the 10 digit classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, in this example, should you define a &quot;not a digit&quot; class and train the model on a set of images that the model may be expected to receive that have nothing to do with handwritten digits such that invalid images are correctly classified as &quot;not a digit&quot;?&lt;/p&gt;&#xA;" OwnerUserId="50158" LastEditorUserId="28175" LastEditDate="2018-04-08T11:51:38.847" LastActivityDate="2018-05-08T12:48:22.677" Title="&quot;other&quot; class in Image classification" Tags="|machine-learning|neural-network|deep-learning|image-classification|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30033" PostTypeId="2" ParentId="30032" CreationDate="2018-04-08T11:51:15.253" Score="1" Body="&lt;p&gt;You can have that but I recommend something else for your case. Suppose that you have 10 classes of digits that they are mutually exclusive. Consequently, your output vector will always be on-hot encoded. In a simple case, your output vector should contain ten entries for each class. To make your model distinguish between those inputs which are digits and those which are not, do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Increase the size of output to eleven. Provide data-set that contains eleven-entries labels. the first entry corresponds to the existence of a digit. If it is a digit the first entry should be one and next 10 entries should be activated for the according to the value of the digit. So, your labels will not be one-hot encoded. If the input is not a digit the first entry of the output should be zero and other entries are don't cares. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-08T11:51:15.253" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30034" PostTypeId="2" ParentId="29987" CreationDate="2018-04-08T12:59:28.003" Score="1" Body="&lt;p&gt;This might be rather in the field of linguistics than related to data science, but nevertheless:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To my knowledge, the &quot;gunning fog index&quot; as measure for understandability is intended for english language only, per definition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For some tools intended for german, see &#xA;&lt;a href=&quot;https://klartext.uni-hohenheim.de/hix&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://klartext.uni-hohenheim.de/hix&lt;/a&gt; (page in german), which refers to a german SMOG index. There are implementations of SMOG listed in PyPI, but you may need to find the appropriate values to configure them for german language.&lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-08T15:19:02.770" LastActivityDate="2018-04-08T15:19:02.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30035" PostTypeId="2" ParentId="29710" CreationDate="2018-04-08T13:22:27.740" Score="2" Body="&lt;p&gt;Looking into the tutorial you refer to, it seems to be made for character recognition of handwritten letters and numbers, taken from the MNIST database. Character classification means in this tutorial: Take an image and classify it as a represenation of a certain Unicode or Asci defined character. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The candidates you want to process are strings, well defined characters that need no recognition at all. So the tutorial does not match your interest. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you define your interest, what do you want to get out of these candidates, which would be a good classification for these three samples? &#xA;If you have that, I would suggest to raise it as a new question. &lt;/p&gt;&#xA;" OwnerUserId="21163" LastActivityDate="2018-04-08T13:22:27.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30038" PostTypeId="1" CreationDate="2018-04-08T17:25:04.793" Score="0" ViewCount="963" Body="&lt;p&gt;I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from langdetect import detect&#xA;&#xA;df = pd.read_csv(&quot;annotation_sample.csv&quot;, nrows=10)&#xA;df = df.drop(['id', 'profile_id', 'created', 'document_id'], axis=1)&#xA;df = df[pd.notnull(df['txt'])]&#xA;df&#xA;&#xA;df.txt.apply(lambda x: detect(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Above works (although I've not figured out how to count a specific language) although it only works as its limited to 10 rows, when using the entire 10,000 it throws the 'No features in text' error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pic of the data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SK5SA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SK5SA.png&quot; alt=&quot;pic of data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I need help with is;&#xA;How do I avoid running this on rows that contain just numbers, urls etc? Is there a way to skip error'd ones and continue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, how do I then count how many have come back as a specific language?&lt;/p&gt;&#xA;" OwnerUserId="50171" LastEditorUserId="29575" LastEditDate="2018-04-08T20:53:55.267" LastActivityDate="2018-04-09T05:04:40.257" Title="Counting how many rows contain chinese in mixed value dataset using python" Tags="|python|pandas|dataframe|" AnswerCount="1" CommentCount="0" ClosedDate="2018-04-11T03:24:34.827" ContentLicense="CC BY-SA 3.0" />
  <row Id="30039" PostTypeId="1" AcceptedAnswerId="30416" CreationDate="2018-04-08T18:06:24.233" Score="1" ViewCount="1460" Body="&lt;p&gt;I am trying to implement (as a toy project) some aspects of speech recognition in Tensorflow. The audio files I want to use as training and test data have different sample rates (16, 20, 44 and 44.1 kHz).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following list is not exhaustive, just listing a few options I could think of.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I can resample the whole signal (which is quite un-periodic) using &lt;code&gt;scipy.signal.resample&lt;/code&gt;, probably after padding with zeros to achieve a signal length which is a power of 2, chopping of the equivalent amount afterwards.&lt;/li&gt;&#xA;&lt;li&gt;I can try to chop the signal into short, non-overlapping windows, which I modify using &lt;code&gt;scipy.signal.resample&lt;/code&gt; before glueing them back together and applying the rest of the chain.&lt;/li&gt;&#xA;&lt;li&gt;I can chop the signal up into overlapping windows, &lt;code&gt;scipy.signal.resample&lt;/code&gt; them and stick them together in a weighted fashion.&lt;/li&gt;&#xA;&lt;li&gt;I can take the spectrogram tensor obtained using &lt;code&gt;tf.contrib.signal.stft&lt;/code&gt; and try to stretch it, for example using rational function interpolation (reflecting tube models of the vocal tract)&lt;/li&gt;&#xA;&lt;li&gt;I can assume that the model will learn to abstract away from sample rates, maybe even helping it (or hindering?) by adding convolution layers before the actual model I am interested in.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In which step of the model should I normalize sampling rates, and how should I do it?&lt;/p&gt;&#xA;" OwnerUserId="50039" LastEditorUserId="50039" LastEditDate="2018-04-08T20:49:21.140" LastActivityDate="2018-09-02T21:07:41.217" Title="Working with audio data with different sample rates in Tensorflow" Tags="|tensorflow|sampling|audio-recognition|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30040" PostTypeId="1" AcceptedAnswerId="30285" CreationDate="2018-04-08T18:35:02.847" Score="2" ViewCount="1032" Body="&lt;p&gt;I have a very basic convolutional neural net built in Keras with a TensorFlow backend. The model is based of this &lt;a href=&quot;https://www.kaggle.com/anokas/simple-keras-starter/code&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle kernel full model&lt;/a&gt;. The input training data are 256x256 images with a corresponding set of class labels in a csv file. For example, here are the class labels in the csv file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;image_name  |  tags&#xA;----------------------------------------&#xA;train_0     |  class1 class2 class3&#xA;train_1     |  class2&#xA;train_2     |  class3&#xA;and so on ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following code shows how I am building, training, and predicting results.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Build model&#xA;model = Sequential()&#xA;model.add(Conv2D(32, kernel_size=(3, 3),&#xA;                 activation='relu',&#xA;                 input_shape=(32, 32, 3))) # Originally (32, 32, 3)&#xA;&#xA;model.add(Conv2D(64, (3, 3), activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;model.add(Flatten())&#xA;model.add(Dense(128, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(3, activation='sigmoid')) # Originally 17&#xA;&#xA;model.compile(loss='binary_crossentropy', # We NEED binary here, since categorical_crossentropy l1 norms the output before calculating loss.&#xA;              optimizer='adam',&#xA;              metrics=['accuracy'])&#xA;&#xA;# Train the model              &#xA;model.fit(x_train, y_train,&#xA;          batch_size=128,&#xA;          epochs=4,&#xA;          verbose=1,&#xA;          validation_data=(x_valid, y_valid))&#xA;&#xA;# Predict&#xA;p_valid = model.predict(x_valid, batch_size=128)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can see the predicted results are an array of class probabilities:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p_valid&#xA;Out[29]: &#xA;array([[2.4571007e-01, 1.4929530e-02, 9.3575776e-01],&#xA;       [2.6206359e-01, 1.2429485e-02, 9.5172155e-01],&#xA;       [3.3679003e-01, 2.8344743e-02, 8.5209453e-01],&#xA;       ...,&#xA;       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],&#xA;       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],&#xA;       [8.2605667e-03, 2.0513131e-07, 9.9999821e-01]], dtype=float32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I know which probability array is corresponding to which image?&lt;/p&gt;&#xA;" OwnerUserId="17539" LastEditorUserId="17539" LastEditDate="2018-04-08T19:11:55.973" LastActivityDate="2018-04-13T19:36:40.610" Title="How to associate array of class probabilities with corresponding image in keras?" Tags="|python|predictive-modeling|keras|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30041" PostTypeId="1" CreationDate="2018-04-08T19:23:56.427" Score="1" ViewCount="526" Body="&lt;p&gt;I am implementing Scott Reed's paper on Generative Adversarial Text to Image Synthesis.(&lt;a href=&quot;https://arxiv.org/pdf/1605.05396.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1605.05396.pdf&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset I am using is a simple one, consisting of images of circles, equilateral triangles and rectangles. The caption of for example a circle is &quot;This is a circle of radius 10&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead of the GAN used in the paper, I am using a BiGAN(this shouldnt make that much of a difference in the losses) and training it for 200 epochs. The fake images generated by the generator look quite real in the end. However, the generator loss doesn't seem to decrease, while the discriminator loss decrease fairly rapidly. I can attach more details if needed. Here are the plots of both the generator and the discriminator loss: &lt;a href=&quot;https://i.stack.imgur.com/li2A7.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://i.stack.imgur.com/li2A7.jpg&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas as to why this is happening?&lt;/p&gt;&#xA;" OwnerUserId="26083" LastActivityDate="2018-04-08T19:23:56.427" Title="Generator loss not decreasing- text to image synthesis" Tags="|neural-network|deep-learning|gan|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30042" PostTypeId="2" ParentId="30038" CreationDate="2018-04-08T19:42:16.973" Score="1" Body="&lt;p&gt;That should be feasible using some more python and pandas features:&#xA;- errors can be caught when you wrap the detect in your own function and return None (or what you prefer) instead of a language code&#xA;- results of the detect can be added to the df and counted, filtered, you name it. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def mydetect(txt):&#xA;    try:&#xA;        return detect(txt)&#xA;    except:&#xA;        return None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Pandas allows you to add the results to a second column of the dataframe, but I do not have the command handy at the moment, please check a tutorial.&#xA;Then you can filter the dataset or count occurences of a language. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can also limit the except clause to certain errors if you think that would make more sense. &lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-09T05:04:40.257" LastActivityDate="2018-04-09T05:04:40.257" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30043" PostTypeId="2" ParentId="29991" CreationDate="2018-04-08T19:46:57.527" Score="1" Body="&lt;p&gt;I am not sure what you actually want to do, but if you want to &lt;strong&gt;simulate&lt;/strong&gt; the Markov chain, than you really have to bring probabilities into play. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, in each step, you would use the transition matrix to determine the probability for each of the possible target states - A, B and C in your example. You would then draw a value - let me call it U - from a uniform distribution on [0,1] and determine the next state based on that value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example, you would proceed from C to A if U is between 0 and 0.7 (and therefore with probability 0.7), to B if U is between 0.7 and 0.8 (and therefore with probability 0.8 - 0.7 = 0.1) and stay at C if U is between 0.8 and 1.0 (with probability 1.0 - 0.8 = 0.2)&lt;/p&gt;&#xA;" OwnerUserId="49886" LastActivityDate="2018-04-08T19:46:57.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30045" PostTypeId="2" ParentId="23194" CreationDate="2018-04-08T20:03:37.577" Score="1" Body="&lt;p&gt;From my point of view, both answers are correct - it depends on what you want to achieve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to use an RBM is as a &lt;strong&gt;generative model&lt;/strong&gt;. If you wanted to do this, you would train the network in order to obtain weights generating a distribution the network which is as close as possible to the real distribution of the input data (not exactly true as CD is known not to converge to the sample distribution, but this just as an aside). You could then, for instance, reconstruct images from partial or sample completely new images. In this context, the main purpose of the hidden units is to increase the power of the network - a Boltzmann machine without hidden units is not able to distinguish target distributions with the same second moments. However, the actual value of the hidden units during and after is less important, as your primary goal is to generate samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to use the model to detect &lt;strong&gt;features&lt;/strong&gt;, the hidden units have an additional role. To explain this, suppose you use contrastive divergence as a learning algorithm. In each iteration, you would then sample the hidden units from the visible units and the visible units from the new values of the hidden units. If the result is close to the original value of the visible units, i.e. if your &lt;strong&gt;reconstruction error&lt;/strong&gt; is small, then, apparently, the hidden units and the weights have learned enough about the training data to reconstruct the value of the visible units from the value of the hidden units. As you say, this is in fact similar to a dimensional reduction or an autoencoder. You could then build a second RBM on top the hidden units, i.e. you could start to stack RBMs to build deep networks, where each new layer of hidden units learns more and more abstract features from the training data. &lt;/p&gt;&#xA;" OwnerUserId="49886" LastActivityDate="2018-04-08T20:03:37.577" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30048" PostTypeId="1" CreationDate="2018-04-08T21:25:01.753" Score="1" ViewCount="441" Body="&lt;p&gt;I created two categorical variables (Fac1 and Fac2) which can take values 1 or 2.  So, I have 4 combinations.  For the combination (1,1) I created 25 gaussian random numbers with mean 12.5 and variance 3, for (1,2) I created 25 Gaussian Random numbers with (37.5,3), for (2,1) another 25 with (62.5,3) and for (2,2) with (87.5,3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I created the R program and ran the lm Model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;twodimfac = read.csv(&quot;TwoDimFac.csv&quot;)&#xA;twodimfac$Fac1 &amp;lt;- as.factor(twodimfac$Fac1)&#xA;twodimfac$Fac2 &amp;lt;- as.factor(twodimfac$Fac2)&#xA;&#xA;A = twodimfac[,1:2]&#xA;x_with_int = lm(twodimfac[,3] ~ . , data = A)&#xA;summary(x_with_int)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result I am getting is&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Call:&#xA;lm(formula = twodimfac[, 3] ~ ., data = A)&#xA;&#xA;Residuals:&#xA;    Min      1Q  Median      3Q     Max &#xA;-8.9121 -2.3011  0.2431  1.8639 23.5385 &#xA;&#xA;Coefficients:&#xA;            Estimate Std. Error t value Pr(&amp;gt;|t|)    &#xA;(Intercept)  12.3355     0.6839   18.04   &amp;lt;2e-16 ***&#xA;Fac12        49.5762     0.7897   62.78   &amp;lt;2e-16 ***&#xA;Fac22        25.4342     0.7899   32.20   &amp;lt;2e-16 ***&#xA;---&#xA;Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1&#xA;&#xA;Residual standard error: 3.948 on 97 degrees of freedom&#xA;Multiple R-squared:  0.9812,    Adjusted R-squared:  0.9808 &#xA;F-statistic:  2530 on 2 and 97 DF,  p-value: &amp;lt; 2.2e-16&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am interpreting Intercept, 12.33554 as a predictor for (1,1).  For other values, the program shows for (1,2) a value which is 12.3355 + 25.4342 = 37.7697, for (2,1) it predicts 61.91175 (which is sum of the intercept coeff and Fac12 coefficient) and for (2,2) it predicts 87.34596 (which is the sum of all coefficients).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How I can interpret the values?  If I have combination of more than 2 factor variables, how one can interpret the coefficients.  Is there any easy formula/function to get it, rather than guessing?&lt;/p&gt;&#xA;" OwnerUserId="46891" LastActivityDate="2018-04-08T21:25:01.753" Title="Interpreting the lm results in R for categorical variables" Tags="|r|linear-regression|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30049" PostTypeId="1" CreationDate="2018-04-08T23:20:07.840" Score="0" ViewCount="31" Body="&lt;p&gt;I'm building an AI that works with natural language processing integrated in my school project application (an app dedicated to traders). I'm wondering what set of word is used most to express negativity (or positivity) in stock market? If a certain stock is decreasing in value (or increasing) what words are most likely to be appear on the press?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Ultimate Goal&lt;/strong&gt;: Classifying an article fetched from the web as describing stock increase or decrease.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Are there any public free data sets tackling this subject? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Else, is there a data set that classifies sentences as trending or not trending? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion, help or even collaboration would be very appreciated!&lt;/p&gt;&#xA;" OwnerUserId="50184" LastActivityDate="2018-04-09T02:32:52.330" Title="Classification of articles fetched from the web as describing stock (trending) increase or decrease" Tags="|dataset|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30050" PostTypeId="1" AcceptedAnswerId="30053" CreationDate="2018-04-09T01:54:54.903" Score="0" ViewCount="3324" Body="&lt;p&gt;I have an atypical time format that I need to convert into a datetime index for time series analysis.  I'm working in Python / Pandas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The column is 'BC_DT', and the format is &quot;27-MAR-18&quot;.  Example is below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;BC_DT&#xA;27-MAR-18&#xA;28-MAR-18&#xA;29-MAR-18&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I tried this method, but I'm getting an error: ValueError: time data '27-MAR-18' does not match format '%d-%b-%Y'&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['Converted_Date'] = df['BC_DT'].apply(lambda x: dt.datetime.strptime(x, '%d-%b-%Y'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49861" LastActivityDate="2019-08-29T11:29:54.430" Title="Convert Atypical Date Format for Time Series in Python" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30051" PostTypeId="2" ParentId="30049" CreationDate="2018-04-09T02:32:52.330" Score="1" Body="&lt;p&gt;You can use a deep learning network to generate &lt;strong&gt;sentiment analysis&lt;/strong&gt;. Pretrained models can be found which have been trained using vectorized word lists which are commonly associated with positivity and negativity. However, these may not generalize well to your specific use case. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the historic nature of stocks you can easily compile your own database from online sources. For example, you can construct instances in your dataset as follows. Tokenize all the words found in articles pertaining to Google for February 5th 2001, then the target label can be the stock value for February 7th 2001. You can choose the time lag as you please. If you do this for a large amount of instances, your dataset should be able to contain some notion of what words found in articles will cause a stock price to rise or lower. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T02:32:52.330" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30052" PostTypeId="2" ParentId="28870" CreationDate="2018-04-09T02:40:34.527" Score="1" Body="&lt;p&gt;K-means attempts to group distributions into $k$ similar categories based on some metric of nearness. If your dataset contains repeated values this is not a problem. It simply means that you have 2 distributions from which you are drawing instances, and the variance of these distributions is essentially zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you have a dataset from which you want to cluster two species of fish with 2 features: color and length. However, the fish of each species are all clones (ignore other biology stuff i know nothing about like environmental factors). Thus they will all have exactly the same color and the same length in either group. Each measurement of each individual will be exactly the same. This is fine. We simply have 0 variance in each distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clustering is very well suited for very tight distributions because they are easily separable. However, distributions with high variance will have significant overlap and will thus cause many instances to be wrongfully classified as being a part of the other cluster.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The ideal number of clusters&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This value can be calculated by determining the average distance between each instance and its nearest cluster. You will see that once you surpass 2, the average distance will not decrease in any further in this case, which means a single distribution is being split into 2 groups. Thus 2 is the ideal number of clusters. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T02:40:34.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30053" PostTypeId="2" ParentId="30050" CreationDate="2018-04-09T02:52:38.143" Score="3" Body="&lt;p&gt;Let pandas determine what datetime format you are using automatically. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;raw_data = pd.DataFrame(data={'BC_DT':['27-MAR-18','28-MAR-18','29-MAR-18']})&#xA;raw_data['BC_DT'] =  pd.to_datetime(raw_data['BC_DT'])&#xA;print(raw_data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;BC_DT &lt;br/&gt;0:   2018-03-27 &lt;br/&gt;1:  2018-03-28 &lt;br/&gt;2:  2018-03-29&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T02:52:38.143" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30054" PostTypeId="1" CreationDate="2018-04-09T02:52:51.233" Score="0" ViewCount="39" Body="&lt;ol&gt;&#xA;&lt;li&gt;Given the &lt;strong&gt;job title&lt;/strong&gt; return the skillset required for the job &lt;/li&gt;&#xA;&lt;li&gt;If a user is lacking some skills required for the job, we have to suggest the courses the user has to take to bridge the gap.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Problem with 1) is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) The provider is returning the skillset only if the keyword(tile of the job) is exact. For instance: For &quot;Accountant&quot; it gives the skillset whereas for &quot;Accounting&quot; it didn't give any result. One another example is: It gives result for &quot;Developer&quot; not for &quot;Software Engineer&quot; or &quot;Software Developer&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does NLP or machine learning solves this issue? If yes, can you give some recommendation on how this problem could be approached.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 2): What is the way to handle recommendations?&lt;/p&gt;&#xA;" OwnerUserId="50192" LastEditorUserId="50192" LastEditDate="2018-04-09T03:22:36.300" LastActivityDate="2018-04-09T03:22:36.300" Title="Data normalisation and recommendation based on skillset" Tags="|machine-learning|nlp|recommender-system|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30055" PostTypeId="2" ParentId="30054" CreationDate="2018-04-09T02:59:38.230" Score="1" Body="&lt;h1&gt;Root words&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The problem here is that language has many different ways of distorting words to give slightly more information without changing the root of the word. Your database which contains the needed courses for a specific profession should ideally be tokenized into a numerical value. Or if that is not possible you can get the stem of a word as follows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk.stem.snowball import SnowballStemmer&#xA;stemmer = SnowballStemmer(&quot;english&quot;)&#xA;print(stemmer.stem(&quot;Accountant&quot;))&#xA;print(stemmer.stem(&quot;Accounting&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;account &lt;br/&gt;account&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h1&gt;Recommendations&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;There's a lot of resources on how to do this, especially using deep learning in recen tyears. You can look at some of the publications from Netflix, Amazon, SoundCloud. All these groups have very powerful recommendation systems. In essence what you want to do is determine some user metrics such as click through sequence, search history. Then build a database with these features and a target such as the next item that is played. From this you can train your model to predict a set of items the user is likely to click on next given his feature set. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T02:59:38.230" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30056" PostTypeId="1" CreationDate="2018-04-09T04:00:05.670" Score="6" ViewCount="3658" Body="&lt;p&gt;I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I present to you the triangle of death (in red), in the context of a computation graph for the batch norm equation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ChLtZ.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ChLtZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is that applying chain rule to the triangle results in &lt;code&gt;1 - 1 = 0&lt;/code&gt;, which kills the gradient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is I am messing up one or more of:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the derivatives.&lt;/li&gt;&#xA;&lt;li&gt;the relationships I think the graph implies with respect to the nodes.&lt;/li&gt;&#xA;&lt;li&gt;the graph representation I chose for BN as a composition of functions.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;but I'm not sure where my error(s) is/are.&lt;/p&gt;&#xA;" OwnerUserId="49780" LastEditorUserId="49780" LastEditDate="2018-04-09T04:17:27.723" LastActivityDate="2018-07-29T22:10:05.457" Title="deriving the gradient of batch normalization" Tags="|backpropagation|batch-normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30057" PostTypeId="1" CreationDate="2018-04-09T04:08:14.367" Score="1" ViewCount="6255" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Any recommendation on the libraries/methods to extract the skill set required for the job from the job description (raw text) ?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;And also to extract the skillset from the resume ? The resume could be of any format raw text, pdf, docx.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nZoQT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nZoQT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have attached the job description format. &lt;/p&gt;&#xA;" OwnerUserId="50192" LastEditorUserId="50192" LastEditDate="2018-04-10T03:36:14.707" LastActivityDate="2018-04-10T03:36:14.707" Title="To extract the skills required for the job given the job description" Tags="|machine-learning|data-mining|nlp|" AnswerCount="3" CommentCount="2" ClosedDate="2018-04-09T13:37:55.233" ContentLicense="CC BY-SA 3.0" />
  <row Id="30058" PostTypeId="2" ParentId="30026" CreationDate="2018-04-09T04:19:47.007" Score="5" Body="&lt;h1&gt;Genetic algorithm&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This consists in 4 crucial steps: initialization, evaluation, selection and combination.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Initialization&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Each individual in the population is encoded by some genes. In our case the genes represent our $[x, y]$ values. We will then set our search range to [0, 1000] for this specific problem. Usually you will know what is naturally possible based on your problem. For example, you should know the range of possible soil densities in nature. We will create 100 individuals in our population.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Evaluation of the fitness&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This step simply asks you to put the $[x,y]$ values into your function and get its result. Pretty standard stuff.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Selection&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There are many ways with which you can select parents. I will always keep the alpha male. The best individual in the population, he will be cloned to the next. Then I will use tournament selection. We will repeat the following until the next generation population is full. Pick four parents at random, take the best individual from the first two and the best from the last two. These will be the two parents which will gives us our next offspring.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Combination&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;From the two parents we will build the new genome for the child using the binary values of the $[x,y]$ values of the parents. The resulting binary value for each codon in the genome of the child is selected from the two parent genes by uniform random.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The code&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class Genetic(object):&#xA;&#xA;    def __init__(self, f, pop_size = 100, n_variables = 2):&#xA;        self.f = f&#xA;        self.minim = -100&#xA;        self.maxim = 100&#xA;        self.pop_size = pop_size&#xA;        self.n_variables = n_variables&#xA;        self.population = self.initializePopulation()&#xA;        self.evaluatePopulation()&#xA;&#xA;    def initializePopulation(self):&#xA;        return [np.random.randint(self.minim, self.maxim, size=(self.n_variables)) &#xA;                           for i in range(self.pop_size)]&#xA;&#xA;    def evaluatePopulation(self):&#xA;        return [self.f(i[0], i[1]) for i in self.population]&#xA;        #return [(i[0]-4)**2 + i[1]**2 for i in self.population]&#xA;&#xA;    def nextGen(self):&#xA;        results = self.evaluatePopulation()&#xA;        children = [self.population[np.argmin(results)]]&#xA;&#xA;        while len(children) &amp;lt; self.pop_size:&#xA;            # Tournament selection&#xA;            randA, randB = np.random.randint(0, self.pop_size), \&#xA;                           np.random.randint(0, self.pop_size)&#xA;            if results[randA] &amp;lt; results[randB]: p1 = self.population[randA]&#xA;            else: p1 = self.population[randB]&#xA;&#xA;            randA, randB = np.random.randint(0, self.pop_size), \&#xA;                           np.random.randint(0, self.pop_size)  &#xA;            if results[randA] &amp;lt; results[randB]: p2 = self.population[randA]&#xA;            else: p2 = self.population[randB]   &#xA;&#xA;            signs = []&#xA;            for i in zip(p1, p2):&#xA;                if i[0] &amp;lt; 0 and i[1] &amp;lt; 0: signs.append(-1)&#xA;                elif i[0] &amp;gt;= 0 and i[1] &amp;gt;= 0: signs.append(1)&#xA;                else: signs.append(np.random.choice([-1,1]))&#xA;&#xA;            # Convert values to binary&#xA;            p1 = [format(abs(i), '010b') for i in p1]&#xA;            p2 = [format(abs(i), '010b') for i in p2]&#xA;&#xA;            # Recombination&#xA;            child = []&#xA;            for i, j in zip(p1, p2):&#xA;                for k, l in zip(i, j):&#xA;                    if k == l: child.append(k)&#xA;                    else: child.append(str(np.random.randint(min(k, l), &#xA;                                                             max(k,l))))&#xA;&#xA;            child = ''.join(child)&#xA;            g1 = child[0:len(child)//2] &#xA;            g2 = child[len(child)//2:len(child)]&#xA;            children.append(np.asarray([signs[0]*int(g1, 2), &#xA;                                        signs[1]*int(g2, 2)]))&#xA;        self.population = children&#xA;&#xA;    def run(self):&#xA;        ix = 0&#xA;        while ix &amp;lt; 1000:&#xA;            ix += 1&#xA;            self.nextGen()&#xA;        return self.population[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you can use the code by&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f = lambda x, y: (x)**2 + y**2&#xA;gen = Genetic(f)&#xA;minim = gen.run()&#xA;print('Minimum found      X =', minim[0], ', Y =', minim[1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Minimum found      X = 0 , Y = 0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f = lambda x, y: (x-6)**2 + y**2&#xA;gen = Genetic(f)&#xA;minim = gen.run()&#xA;print('Minimum found      X =', minim[0], ', Y =', minim[1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Minimum found      X = 6 , Y = 0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-04-09T04:40:01.163" LastActivityDate="2018-04-09T04:40:01.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30060" PostTypeId="2" ParentId="30013" CreationDate="2018-04-09T05:01:29.080" Score="2" Body="&lt;p&gt;Any classifier that naturally provides feature importance can be applied here. For example, take a look at &lt;a href=&quot;https://datascience.stackexchange.com/questions/16693/interpreting-decision-tree-in-context-of-feature-importances&quot;&gt;Decision Trees&lt;/a&gt; and &lt;a href=&quot;http://blog.datadive.net/selecting-good-features-part-iii-random-forests/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forrest&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For other algorithms that do not provide such information (eg KNN and Neural Nets), you may take a (forward/backward/bidirectional) feature selection. For example, chose a subset of features, train a model and evaluate it with any preferred metrics (better to use bootstrapping or cross-validation). Choose another subset if features and try again. After several trials, you can find the subset with the best prediction power. This is the set of the mist contributing features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that different techniques/measures may give you different results.&lt;/p&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-04-09T05:01:29.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30061" PostTypeId="1" CreationDate="2018-04-09T05:04:19.293" Score="0" ViewCount="63" Body="&lt;p&gt;Hi I want to extract words of english texts using click rate with machine learning model. Now I know the click rate of text, and I know How to extract words(unigram) of each text, for example, there are about 10000 texts,and the click rate of each text is provided. How to extract words features for click rate. How to extract key words and compute the importance of each word for computing click rate.&lt;/p&gt;&#xA;" OwnerUserId="35597" LastActivityDate="2018-04-09T05:39:01.793" Title="machine learning mode are suitable for using word features as x, and y as click rate of text" Tags="|machine-learning|python|linear-regression|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30062" PostTypeId="2" ParentId="30061" CreationDate="2018-04-09T05:39:01.793" Score="1" Body="&lt;p&gt;You can build a dictionary of character sequences (words) and for each instance of text you will count the occurrence of these words. You can either use groupings of characters &lt;strong&gt;n-grams&lt;/strong&gt; or words them selves using &lt;strong&gt;bag-of-words&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;n-grams&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;n-grams&lt;/strong&gt; is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The segmentation works as follows: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The String: Hello World&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2-gram: &quot;He&quot;, &quot;el&quot;, &quot;ll&quot;, &quot;lo&quot;, &quot;o &quot;, &quot; W&quot;, &quot;Wo&quot;, &quot;or&quot;, &quot;rl&quot;, &quot;ld&quot;&#xA;3-gram: &quot;Hel&quot;, &quot;ell&quot;, &quot;llo&quot;, &quot;lo &quot;, &quot;o W&quot;, &quot; Wo&quot;, &quot;Wor&quot;, &quot;orl&quot;, &quot;rld&quot;&#xA;4-gram: &quot;Hell&quot;, &quot;ello&quot;, &quot;llo &quot;, &quot;lo W&quot;, &quot;o Wo&quot;, &quot; Wor&quot;, &quot;Worl&quot;, &quot;orld&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Bag-of-words&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Bag-of-Words&lt;/strong&gt; builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.  &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T05:39:01.793" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30063" PostTypeId="2" ParentId="30056" CreationDate="2018-04-09T05:41:00.673" Score="0" Body="&lt;p&gt;Some image credits apply......&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's from a blog which applies the idea of Computational Graphs as explained in CS-231N by Karapathay..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What differs from your graph is that we need to take into account $*,+,/,-$ symbols also..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will have a careful look at yours also and update here, but till then, this graph works as I have also derived the expressions from this one...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NpUz2.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NpUz2.jpg&quot; alt=&quot;Check This Image Out&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-09T06:08:11.717" LastActivityDate="2018-04-09T06:08:11.717" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30064" PostTypeId="1" AcceptedAnswerId="30147" CreationDate="2018-04-09T05:47:33.937" Score="0" ViewCount="653" Body="&lt;p&gt;While predicting, what happens if we pass the &lt;code&gt;newdata&lt;/code&gt; along with the target variable? Do we need to isolate the target variable before feeding into predict function?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = read.csv(&quot;diabetes.csv&quot;)  #768 observations of 9 variables&#xA;&#xA;#I split the data into train and test&#xA;set.seed(123)&#xA;IDS = 1:nrow(data)&#xA;train_data = sample(x = IDS,size = nrow(data)*0.7)&#xA;test_data = sample(x = setdiff(IDS,train_data),size = nrow(data)*0.3)&#xA;train_data = data[train_data,]&#xA;test_data = data[test_data,]&#xA;&#xA;#Build logistic regression model&#xA;log_model1 = glm(formula = Outcome ~ .,data = train_data,family = 'binomial')&#xA;&#xA;#Predict it on test or train&#xA;train_predict = predict(object = log_model1,newdata = train_data,type = &quot;response&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="24000" LastEditDate="2018-04-09T06:22:09.403" LastActivityDate="2018-05-11T00:50:37.657" Title="predict() function in R" Tags="|logistic-regression|prediction|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30065" PostTypeId="2" ParentId="30057" CreationDate="2018-04-09T05:51:41.303" Score="0" Body="&lt;p&gt;You will need to do some web scraping. &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; resource may be useful to you, it explains how to do web scraping with Beautiful Soup, a very common python package. However, web scraping is never an easy task because webpages are rarely formatted in a friendly way. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-09T05:51:41.303" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30066" PostTypeId="2" ParentId="30057" CreationDate="2018-04-09T07:04:09.077" Score="4" Body="&lt;p&gt;Assuming you already have the raw text, you can do the followings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Create train data:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to create set words and bigrams labeled as skills. There might be some available lists to help you out. Otherwise, generate your list using resources such as wordnet and thesaurus. You can also start with a short list based on the data you have and then expand it using word2vec or similar word embedding techniques. For example, we start with a list that contains &lt;em&gt;coding&lt;/em&gt; as one of the skills. Then, we query the word2vec pretrained model for closest words/bigrams. There is a fair chance you will end up with &lt;em&gt;programming&lt;/em&gt;, &lt;em&gt;software coding&lt;/em&gt;, and &lt;em&gt;computer programming&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach would be to cluster the words and bigrams in your dataset using word embedding techniques. Then, look into your clusters to see which ones contain the skill set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that any word/expression not on your list will not be considered as required skills. Therefore, you may need to expand your list after a few trials.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Detecting the Skills&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tokenize your raw text into words and expressions&lt;/li&gt;&#xA;&lt;li&gt;Remove stop words&lt;/li&gt;&#xA;&lt;li&gt;Encode your tokens using an embedding (Word2Vec, FastText, etc)&lt;/li&gt;&#xA;&lt;li&gt;Use the list from the previous step to add labels to your data (anything on the list is True, other as False)&lt;/li&gt;&#xA;&lt;li&gt;Train a binary classifier (Naive Bayes classifier should be good enough)&lt;/li&gt;&#xA;&lt;li&gt;Evaluate your model, feature set, and labels. If needed, refine and repeat.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="46505" LastActivityDate="2018-04-09T07:04:09.077" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30068" PostTypeId="2" ParentId="30057" CreationDate="2018-04-09T08:42:14.663" Score="0" Body="&lt;p&gt;In what format are your input texts provided? &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If it is &lt;em&gt;HTML&lt;/em&gt; then use a library that selects (for instance) from the Document Object Model DOM all paragraphs following &quot;Requirements&quot;. In javascript you could use jQuery, or on the linux command line you could use the xmllint binary (which uses xml2 internally). Many other libraries are available.&lt;/li&gt;&#xA;&lt;li&gt;If it is &lt;em&gt;PDF&lt;/em&gt; then doing this is more difficult.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After that you could do some bag-of-words modeling as user &quot;Borhan Kazimipour&quot; has suggested in another answer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could topic-model them. Probabilistic Topic Modeling is a more difficult technique which would take me at least a week to re-learn the principles of, and to get familiar with available command line tools and libraries.&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2018-04-09T08:42:14.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30069" PostTypeId="2" ParentId="29944" CreationDate="2018-04-09T08:52:09.313" Score="1" Body="&lt;p&gt;If you suspect  that adding another variable does not improve the model, just leave it out, run the updated model again and check, for instance, the adjusted R-square value.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;anova()&lt;/code&gt; function takes two models and compares them more formally.&lt;/p&gt;&#xA;&lt;p&gt;See this set of comprehensive answers to &amp;quot;&lt;a href=&quot;https://stats.stackexchange.com/questions/13686/what-are-modern-easily-used-alternatives-to-stepwise-regression&quot;&gt;What are modern, easily used alternatives to stepwise regression?&amp;quot;&lt;/a&gt; on the Cross-Validated site.&lt;/p&gt;&#xA;&lt;p&gt;Good models give you insight into your problem (there may be several that do this).&lt;/p&gt;&#xA;&lt;p&gt;(From R.Cotton (2011) Learning R book p 271)&lt;/p&gt;&#xA;" OwnerUserId="14588" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-09T08:52:09.313" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30070" PostTypeId="1" AcceptedAnswerId="42184" CreationDate="2018-04-09T09:18:04.743" Score="1" ViewCount="10577" Body="&lt;p&gt;I am building a Convolutional Neural Network with Keras. I want to use my own dataset. This dataset consists of matrices of specified size and filled with 0's which contain a submatrix of specified size and filled with 1's. The submatrix is optional and the goal is to train the NN to predict whether a matrix contains the submatrix or whether it doesnt. To make it more difficult to detect, I am adding various types of noise to the matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a picture of what the matrices look like, the black parts are 0's and the white part are 1's. There is a 1:1 correspondance between the pixels of the image and the entries in the matrix.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/PCS5E.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PCS5E.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to create a CNN which can classify these images. The easiest approach so far was to use&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_set = test_datagen.flow_from_directory('dataset/training_set',&#xA;                                            target_size = (imageSize, imageSize),&#xA;                                            batch_size = batchSize,&#xA;                                            class_mode = 'binary')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And then I fill the directory &lt;code&gt;training_set&lt;/code&gt; with 1000's of images. This works well, but has one major problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is really inefficient, as I'm first creating images, saving them on a hard-dsik, then loading them again. These are a lot of unecessary steps, considering that keras transforms the images back into matrices (or tensors, idk), and my images were initially matrices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus I was thinking about improving the efficiency by having the matrices in a &lt;code&gt;.csv&lt;/code&gt; or even having my &lt;code&gt;matrixgenerator.py&lt;/code&gt; &quot;feed&quot; a list or array directly into keras. I did some research and found the method &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;flow&lt;/a&gt; from Keras which specifies as parameter an input matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately this is where I am stuck. &lt;strong&gt;I do not understand how to get my matrices into the form keras wants. I also don't know the exact shape the matrices and labels need to have to be able to use the flow method.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would really help if someone could explain me how to use the flow method/tell me where I find more information on it, or give me a better option to feed my dataset into a keras CNN. Thank you for your help. I'm also ok with using a lower-level tensorflow implementation as long as I understand it.&lt;/p&gt;&#xA;" OwnerUserId="50206" LastActivityDate="2018-12-05T18:00:58.310" Title="How to use keras flow method?" Tags="|dataset|keras|tensorflow|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30071" PostTypeId="1" AcceptedAnswerId="30072" CreationDate="2018-04-09T09:44:29.503" Score="0" ViewCount="247" Body="&lt;p&gt;I've come accross the following paragraph in the &lt;a href=&quot;https://arxiv.org/pdf/1101.0891.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;To Explain or To Predict?&lt;/a&gt; paper by &lt;em&gt;Galit Shmueli&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In explanatory modeling, data partitioning is less common [than in predictive modeling] because of&#xA;  the reduction in statistical power. When used, it is usually done for&#xA;  the retrospective purpose of assessing the robustness of ˆf. A rarer&#xA;  yet important use of data partitioning in explanatory modeling is for&#xA;  strengthening model validity, by demonstrating some predictive power.&#xA;  Although one would not expect an explanatory model to be optimal in&#xA;  terms of predictive power, it should show some degree of accuracy.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I understand why data partitioning is useful in the case of a predictive model, which is to assess the generalization capacity of a model. However, in the case of an explanatory model, I don't understand why it should show some degree of accuracy in terms of predictive power since it's not the objective of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here comes my question: &lt;strong&gt;is data partitioning necessary for an explanatory model and why?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="50208" LastActivityDate="2018-04-09T10:12:42.200" Title="Is data partitioning necessary for an explanatory model and why?" Tags="|data-mining|predictive-modeling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30072" PostTypeId="2" ParentId="30071" CreationDate="2018-04-09T10:12:42.200" Score="0" Body="&lt;p&gt;An explanatory model is used to identify and explain what causes some particular outcome (i.e. identify/quantify the drivers of effect).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though the model will not be used for prediction, it needs to be &lt;strong&gt;accurate enough&lt;/strong&gt; to adequately predict the response so that you can be sure the conclusions you draw from the model are valid. As an extreme example, if your explanatory model does not predict the outcome at all, then all the inferences you make from this model are useless.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So then, how does one validate model accuracy? In most cases, data partitioning and validation on the hold-out sample is used; in your case the validation is more of a sense-check for accuracy.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-04-09T10:12:42.200" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30073" PostTypeId="2" ParentId="27927" CreationDate="2018-04-09T10:46:45.983" Score="2" Body="&lt;p&gt;I don't think you can use Pearson's correlation because it is used for continuous variables. Your variables are ordinal, so a test like Spearman's would be more appropriate. However I don't think that tests for ordinal variables would be appropriate either, because your variables are also &lt;strong&gt;cyclical&lt;/strong&gt;, in the sense that &lt;code&gt;Hour_ofDay=23&lt;/code&gt; and &lt;code&gt;Hour_ofDay=1&lt;/code&gt; are really just 2 hours apart, but for the Spearman's test they would be considered 22 hours apart.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think in this case it would be more appropriate to look at the distribution of the distance (measured in hours) between the two variables.&#xA;The appropriate distance metric in this case is defined as follows (distance originally defined in the accepted answer to &lt;a href=&quot;https://gamedev.stackexchange.com/questions/4467/comparing-angles-and-working-out-the-difference&quot;&gt;this other question&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;distance = np.sign(a1-a2)*(12 - abs(abs(a1 - a2) - 12))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where a1 and a2 are your App and Email opening time variables. Note that the variables need to be in the range 0 to 23 for this distance to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Compute this distance for each row, add it as a column to your dataframe and plot it with an histogram.&#xA;This histogram will tell you quite a lot about the &quot;correlation&quot; between the two variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;No correlation: the histogram will be uniform between -12 and 12&lt;/li&gt;&#xA;&lt;li&gt;Instantaneous correlation, i.e. user opens email and app at the same time: the histogram will have a peak at 0&lt;/li&gt;&#xA;&lt;li&gt;Anticipated delay, i.e. user opens email 1 hour before app: the histogram will have a peak at 1&lt;/li&gt;&#xA;&lt;li&gt;Delayed correlation, i.e. user opens email 1 hour after app: the histogram will have a peak at -1&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This visualization will allow you to draw rich conclusions about the relation between App- and Email opening times.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: if your variables also include minutes and seconds in a date format you have to convert the variables to numerical. E.g. 01:30 (hour 1 and 30 minutes) becomes 1.5 . Also pay attention to the date format in case you have time expressed in the 12-hour clock (e.g. 6 PM, 1 AM)&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="29575" LastEditDate="2018-04-09T13:41:27.027" LastActivityDate="2018-04-09T13:41:27.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30075" PostTypeId="1" AcceptedAnswerId="30082" CreationDate="2018-04-09T11:01:02.373" Score="7" ViewCount="1318" Body="&lt;p&gt;I am learning word2vec and word embedding , I have downloaded GloVe pre-trained word embedding (shape 40,000 x 50)  and using this function to extract information from that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;def loadGloveModel(gloveFile):&#xA;    print (&quot;Loading Glove Model&quot;)&#xA;    f = open(gloveFile,'r')&#xA;    model = {}&#xA;    for line in f:&#xA;        splitLine = line.split()&#xA;        word = splitLine[0]&#xA;        embedding = np.array([float(val) for val in splitLine[1:]])&#xA;        model[word] = embedding&#xA;    print (&quot;Done.&quot;,len(model),&quot; words loaded!&quot;)&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if I call this function for word 'hello'  something like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(loadGloveModel('glove.6B.100d.txt')['hello'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it gives me 1x50 shape vector like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[ 0.26688    0.39632    0.6169    -0.77451   -0.1039     0.26697&#xA;  0.2788     0.30992    0.0054685 -0.085256   0.73602   -0.098432&#xA;  0.5479    -0.030305   0.33479    0.14094   -0.0070003  0.32569&#xA;  0.22902    0.46557   -0.19531    0.37491   -0.7139    -0.51775&#xA;  0.77039    1.0881    -0.66011   -0.16234    0.9119     0.21046&#xA;  0.047494   1.0019     1.1133     0.70094   -0.08696    0.47571&#xA;  0.1636    -0.44469    0.4469    -0.93817    0.013101   0.085964&#xA; -0.67456    0.49662   -0.037827  -0.11038   -0.28612    0.074606&#xA; -0.31527   -0.093774  -0.57069    0.66865    0.45307   -0.34154&#xA; -0.7166    -0.75273    0.075212   0.57903   -0.1191    -0.11379&#xA; -0.10026    0.71341   -1.1574    -0.74026    0.40452    0.18023&#xA;  0.21449    0.37638    0.11239   -0.53639   -0.025092   0.31886&#xA; -0.25013   -0.63283   -0.011843   1.377      0.86013    0.20476&#xA; -0.36815   -0.68874    0.53512   -0.46556    0.27389    0.4118&#xA; -0.854     -0.046288   0.11304   -0.27326    0.15636   -0.20334&#xA;  0.53586    0.59784    0.60469    0.13735    0.42232   -0.61279&#xA; -0.38486    0.35842   -0.48464    0.30728  ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I am not getting what actually these values represent , ( I know its result of hidden layer of single layer neural network ) but my confusion is what actually these weights represent and how it is useful for me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because what I was getting suppose if I have :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5gAnY.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5gAnY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I understand because each word is mapping to each column category label, But in the GloVe there are no columns labels for 50 columns, it just returns 50 values vector, so what actually these vectors represent and what i can do with it? I am trying to find this since 4-5 hours but everyone/every tutorial on the internet explaining what are word embedding and how they looks like but no one explaining what actually these weights represent?&lt;/p&gt;&#xA;" OwnerUserId="24883" LastEditorUserId="29575" LastEditDate="2018-04-09T13:45:16.177" LastActivityDate="2018-04-09T13:45:16.177" Title="what actually word embedding dimensions values represent?" Tags="|deep-learning|nlp|word2vec|lstm|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30077" PostTypeId="2" ParentId="30013" CreationDate="2018-04-09T11:20:27.523" Score="2" Body="&lt;p&gt;Besides the obvious ideas regarding machine learning methods such as tree based approaches (e.g. xgboost), I would note that logistic regression is not limited to linear effects. For example, spline based methods can be applied quite easily (see e.g. Frank Harrell's &quot;Regression Modeling Strategies - With  Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis&quot; for more details). There are also generalized additive models, that may be yet another option. In such models it may be slightly easier to interpret the contribution of different variables than in tree based methods, where this can be a bit harder.&lt;/p&gt;&#xA;" OwnerUserId="44389" LastActivityDate="2018-04-09T11:20:27.523" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30079" PostTypeId="2" ParentId="24608" CreationDate="2018-04-09T12:06:05.143" Score="3" Body="&lt;p&gt;I can't say which is more efficient or easier but &lt;a href=&quot;https://mxnet.incubator.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MXNet&lt;/a&gt; exists, has interfaces to most languages (python, scala, R, C++, perl, Julia), and the code looks fairly similar to keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://mxnet.incubator.apache.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://mxnet.incubator.apache.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another one that I've come across is &lt;a href=&quot;http://caffe.berkeleyvision.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;caffe&lt;/a&gt;, mostly because people seem to share a lot of pre-trained nets in their &lt;a href=&quot;http://caffe.berkeleyvision.org/model_zoo.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;model zoo&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://caffe.berkeleyvision.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://caffe.berkeleyvision.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: I didn't see &lt;a href=&quot;https://caffe2.ai/&quot; rel=&quot;nofollow noreferrer&quot;&gt;caffe 2&lt;/a&gt; but I think this is quite separate to the original caffe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally I've not fully had time to check out H2O's offering in &lt;a href=&quot;https://www.h2o.ai/deep-water/&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep water&lt;/a&gt; which is an alternative to Keras but still sits on top of TF/MXNet/Caffe&lt;/p&gt;&#xA;" OwnerUserId="50221" LastEditorUserId="50221" LastEditDate="2018-04-09T12:58:40.873" LastActivityDate="2018-04-09T12:58:40.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30080" PostTypeId="1" CreationDate="2018-04-09T12:24:57.037" Score="1" ViewCount="1148" Body="&lt;p&gt;Is there any way to set a threshold value that can be set for voting results for Random Forest Classifier in sklearn package in order to prevent any misclassification for incoming test data.&lt;/p&gt;&#xA;" OwnerUserId="49833" LastEditorUserId="64041" LastEditDate="2020-02-06T04:18:54.397" LastActivityDate="2020-02-06T04:18:54.397" Title="Threshold Value for Random Forest Classifier" Tags="|machine-learning|scikit-learn|random-forest|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30081" PostTypeId="2" ParentId="13525" CreationDate="2018-04-09T12:49:31.823" Score="4" Body="&lt;p&gt;There is a tutorial &lt;a href=&quot;https://github.com/mmckerns/tutmom&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modern Optimization in Python&lt;/a&gt; and a corresponding video &lt;a href=&quot;https://www.youtube.com/watch?v=geFER2oVvvU&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modern Optimization in Python | SciPy 2017 Tutoria&lt;/a&gt;. Although they are not books, I think they may be helpful to you. &lt;/p&gt;&#xA;" OwnerUserId="50223" LastEditorUserId="29575" LastEditDate="2018-04-09T13:40:05.063" LastActivityDate="2018-04-09T13:40:05.063" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30082" PostTypeId="2" ParentId="30075" CreationDate="2018-04-09T13:26:46.627" Score="3" Body="&lt;p&gt;These columns are actually arbitrary, they do not represent anything for humans. However, it does not mean they are useless, quite the opposite - computers can extract features from this highly dimensional space more easily, such as in neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To extract information which is useful for humans, I recommend to have a look at examples in &lt;a href=&quot;https://arxiv.org/pdf/1301.3781.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;original paper by Tomas Mikolov&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can have a look at common capital city.  Athens is to Greece (in sense that they have similar relationship) as is Oslo to Norway.&#xA;We can use afterwards vectors obtained using GloVe and construct&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v(Athens) - v(Greece) + v(Norway)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and this vector in 'globally' (meaning that trained on universal dataset, such as wikipedia) trained model would be closest to $v(Oslo)$.&lt;/p&gt;&#xA;" OwnerUserId="46700" LastActivityDate="2018-04-09T13:26:46.627" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30083" PostTypeId="2" ParentId="24608" CreationDate="2018-04-09T13:28:15.720" Score="1" Body="&lt;p&gt;I reccomend pytorch. You can find good tutorials &lt;a href=&quot;http://pytorch.org/tutorials/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="29575" LastEditDate="2018-04-09T13:46:13.767" LastActivityDate="2018-04-09T13:46:13.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30084" PostTypeId="2" ParentId="28676" CreationDate="2018-04-09T14:27:04.013" Score="7" Body="&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1705.09792&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt; they implement Complex-Valued Deep Neural Network architectures in Keras with Theano backend, and compare them with real-valued counterparts. &lt;/p&gt;&#xA;" OwnerUserId="50235" LastEditorUserId="29575" LastEditDate="2018-04-09T14:55:25.400" LastActivityDate="2018-04-09T14:55:25.400" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30085" PostTypeId="1" CreationDate="2018-04-09T14:33:53.113" Score="3" ViewCount="164" Body="&lt;h1&gt;Data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: &lt;em&gt;date, txVolume(USD), txCount, marketcap(USD), price(USD), exchangeVolume(USD)&lt;/em&gt;. The date frequency is daily. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some stocks begin trading in 2014 and some in 2017 so the length of the date column is not equal for all stocks. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Goal&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I want to see how an index fund would develop. The fund's strategy would be to on the first of each month buy the 10 largest stocks (based on market cap). Rebalancing occurs iff there is a &quot;shift&quot; in which stocks are top 10. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to use python.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Initial attempt&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I'm familiar with &lt;code&gt;pandas&lt;/code&gt; &lt;code&gt;glob&lt;/code&gt; and little bit with &lt;code&gt;numpy&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My sketch at solving this is the following&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;import all files using the &lt;code&gt;glob&lt;/code&gt; package &lt;/li&gt;&#xA;&lt;li&gt;create a matrices for: &lt;strong&gt;market cap&lt;/strong&gt; and &lt;strong&gt;price&lt;/strong&gt;. They have &lt;em&gt;T&lt;/em&gt; rows and &lt;em&gt;M&lt;/em&gt; columns, where M is the nr of stocks and T is the length of the longest series (circa 4 years). &lt;/li&gt;&#xA;&lt;li&gt;Use pandas &lt;code&gt;.resample('M')&lt;/code&gt; to go from daily to monthly data.&lt;/li&gt;&#xA;&lt;li&gt;(ideally the column in the matrix above would have names, therefore using pandas is good since I can set it by &lt;code&gt;df.columns = list_of_names&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;li&gt;create a &lt;strong&gt;returns&lt;/strong&gt; matrix based on the price matrix, and a &lt;strong&gt;weigth&lt;/strong&gt; matrix based on the market cap matrix. &lt;/li&gt;&#xA;&lt;li&gt;return of index = diagonal(Return matrix * Weight matrix transposed)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But I'm a very inexperienced programmer so maybe this approach is stupid. I would like to see if there is a better way of solving the problem I have. &lt;/p&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-04-09T14:33:53.113" Title="Create top 10 index fund based on &gt;100 stocks" Tags="|python|pandas|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30086" PostTypeId="2" ParentId="24608" CreationDate="2018-04-09T14:38:17.843" Score="1" Body="&lt;p&gt;Keras is a high-level API that can be used on top of &lt;a href=&quot;https://github.com/tensorflow/tensorflow&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;TensorFlow&lt;/code&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/Microsoft/cntk&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;CNTK&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;https://github.com/Theano/Theano&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Theano&lt;/code&gt;&lt;/a&gt;. You can use each of the low-level APIs but the problem of those is that you can get complicated if you design very deep nets whilst dealing with &lt;code&gt;Keras&lt;/code&gt; is much easier. Consequently, &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Keras&lt;/code&gt;&lt;/a&gt; is designed for accelerating deep nets' designing. &lt;code&gt;Keras&lt;/code&gt; is opensource like the underlying libraries it comes for and I guess its project is not for &lt;code&gt;Google&lt;/code&gt;. &lt;code&gt;TensorFlow&lt;/code&gt; itself has a high-level API, namely &lt;a href=&quot;http://tflearn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;TFLearn&lt;/code&gt;&lt;/a&gt;. I cannot say which is &lt;a href=&quot;https://datascience.stackexchange.com/a/25318/28175&quot;&gt;better&lt;/a&gt; but the point is that try to master one of them perfectly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that &lt;code&gt;plaidML&lt;/code&gt; Keras backend is also available which enables training on &lt;strong&gt;AMD&lt;/strong&gt; graphics.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2019-03-16T21:47:29.817" LastActivityDate="2019-03-16T21:47:29.817" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30087" PostTypeId="2" ParentId="30006" CreationDate="2018-04-09T14:54:30.267" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Are-there-any-good-python-implementations-of-Kohonen-Self-Organizing-Maps-and-Neural-Gas&quot; rel=&quot;nofollow noreferrer&quot;&gt;There are alternative solutions&lt;/a&gt; for self organizing maps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of them I found &lt;a href=&quot;http://www.pymvpa.org/examples/som.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pymvpa&lt;/a&gt; where the example is easy to read and understand. It is also maintained quite activately as you can see from their &lt;a href=&quot;https://github.com/PyMVPA/PyMVPA&quot; rel=&quot;nofollow noreferrer&quot;&gt;Github&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to run the kohonen 1.1.2 test file, but it did not run after two days of trying. So, let's have a try for the another solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To run pyMVPA example som.py, you have to do the following (at least):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1 install some pre-requisites: at least numpy, scipy, nibabel and swig to be able to run setup.py&#xA;2 run the commands: &#xA;   python setup.py build&#xA;   python setup.py install&#xA;3 modify example under doc/examples (som.py) to have &#xA; import matplotlib.pyplot as pl&#xA;4 run command:&#xA;   python som.py&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note that this works only in python 2, not python3 can be used as command.&lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44114" LastEditDate="2018-04-09T17:26:27.877" LastActivityDate="2018-04-09T17:26:27.877" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30088" PostTypeId="1" CreationDate="2018-04-09T14:55:46.970" Score="4" ViewCount="166" Body="&lt;p&gt;I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1 - Taking a look at &lt;a href=&quot;http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog post&lt;/a&gt; and &lt;a href=&quot;https://www.coursera.org/learn/nlp-sequence-models/lecture/8CZiw/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng's Deep Learning Specialization&lt;/a&gt;, I understood that, for each word, we generate &lt;strong&gt;one training sample for each context word&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if we have the sentence &quot;cat sat on the mat&quot; we will have samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;(cat, sat)&#xA;(sat, cat)&#xA;(sat, on)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you train your network which will have the dimensions.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Input: $(V,1)$&lt;/li&gt;&#xA;&lt;li&gt;Weights 1: $(d,V)$&lt;/li&gt;&#xA;&lt;li&gt;Hidden layer: $(d,1)$&lt;/li&gt;&#xA;&lt;li&gt;Weights 2: $(V,d)$&lt;/li&gt;&#xA;&lt;li&gt;Output layer: $(V,1)$&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ok, so these dimensions match and we are good. &lt;strong&gt;For one given input, given that the weights are the same, we always have the same estimation in the output layer&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this definition we have symmetric samples (for example, &lt;code&gt;(cat, sat)&lt;/code&gt; and &lt;code&gt;(sat, cat)&lt;/code&gt;) and &lt;strong&gt;saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2 - Watching &lt;a href=&quot;https://www.youtube.com/watch?v=ERibwqs9p38&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford's NLP with Deep Learning class&lt;/a&gt; (38:54), it seems like for the same center word we can get different outputs:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/g5w44.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/g5w44.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have &lt;strong&gt;one training example for each center word&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="50238" LastActivityDate="2018-04-22T08:44:07.757" Title="Skipgram - multiple formulations?" Tags="|machine-learning|nlp|word2vec|word-embeddings|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30090" PostTypeId="1" AcceptedAnswerId="45641" CreationDate="2018-04-09T15:42:16.253" Score="5" ViewCount="152" Body="&lt;p&gt;I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a missing word after a sentence. Does this model only work for a sentence with four words which the first three are known and the last is unknown? If I have a sentence in 10 words. The first nine words are known, can I use 9 words as input to predict the last missing word in that sentence?&lt;/p&gt;&#xA;" OwnerUserId="50242" LastEditorUserId="39338" LastEditDate="2019-02-18T16:07:30.610" LastActivityDate="2019-02-18T16:07:30.610" Title="Can CBOW model only accept fixed number of words?" Tags="|deep-learning|word2vec|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30091" PostTypeId="2" ParentId="25705" CreationDate="2018-04-09T17:38:10.053" Score="6" Body="&lt;p&gt;In &lt;a href=&quot;https://arxiv.org/pdf/1609.07009.pdf&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;, theauthors have provided a deep explanation, basically its the exact same thing. sub-pixel uses o&lt;em&gt;r^2 conv kernels all at res H/r,W/r while the regular transpose conv uses o conv kernels all at res H,W the authors prove that one can rearrange the feature map (O&lt;/em&gt;r^2,h,w) to a feature map (O,h&lt;em&gt;r,w&lt;/em&gt;r) without losing information.&lt;/p&gt;&#xA;" OwnerUserId="50249" LastEditorUserId="29575" LastEditDate="2018-04-09T19:07:46.840" LastActivityDate="2018-04-09T19:07:46.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30092" PostTypeId="1" AcceptedAnswerId="30544" CreationDate="2018-04-09T17:41:37.493" Score="2" ViewCount="138" Body="&lt;p&gt;How do we decide which kernel needs to be used for a particular dataset? &#xA;Is there any criteria needs to be followed?&#xA;And also what is the criteria to select C and gamma values?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly excuse me if this question looks silly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-20T00:39:44.373" Title="How do we decide which kernel needs to be used in SVM?" Tags="|svm|optimization|kernel|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30093" PostTypeId="5" CreationDate="2018-04-09T17:51:52.183" Score="0" Body="" OwnerUserId="-1" LastEditorUserId="-1" LastEditDate="2018-04-09T17:51:52.183" LastActivityDate="2018-04-09T17:51:52.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30094" PostTypeId="4" CreationDate="2018-04-09T17:51:52.183" Score="0" Body="Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human–computer interaction. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation. See NLP." OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-10T01:04:47.607" LastActivityDate="2018-04-10T01:04:47.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30096" PostTypeId="2" ParentId="30092" CreationDate="2018-04-09T18:38:31.217" Score="0" Body="&lt;p&gt;Which has a minimum test or &lt;a href=&quot;https://en.wikipedia.org/wiki/Cross-validation_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;CV&lt;/a&gt; error (Or maximum accuracy). &lt;/p&gt;&#xA;" OwnerUserId="50154" LastActivityDate="2018-04-09T18:38:31.217" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30097" PostTypeId="1" CreationDate="2018-04-09T19:20:57.697" Score="6" ViewCount="24512" Body="&lt;p&gt;I'm trying to build a 2 class image classifier using the architecture suggested in first part of this blog &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;noreferrer&quot;&gt;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset has 1500 images of class1 and 500 images of class2. I created 4 copies of class2 to make the number of images same in both classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also use ImageDataGenerator to augment images&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;datagen = ImageDataGenerator(&#xA;        featurewise_center=True,&#xA;        featurewise_std_normalization=True,&#xA;        shear_range=0.2,&#xA;        zoom_range=0.5,&#xA;        rotation_range=45,&#xA;        horizontal_flip=True,&#xA;        vertical_flip=True)&#xA;datagen.fit(train_imgs)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm using AdaDelta,RMSProp, SGD, Adam, AdaGrad optimizers and tried adding/removing Conv2D and Dense layers. Tried BatchNormalizationa and Dropout. The results are coming out almost same:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For first few epochs(about 20) training and validation errors keep reducing until log loss reaches about 0.4 (best I have got till now) after that the model starts to overfit and validation loss keeps increasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I can prevent overfitting by reducing the network complexity and adding dropouts but that reduces the training accuracy too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest some tips to improve the accuracy and avoid overfitting.&lt;/p&gt;&#xA;" OwnerUserId="50256" LastEditorUserId="29169" LastEditDate="2021-02-13T04:24:33.083" LastActivityDate="2023-05-29T11:08:35.507" Title="How to improve loss and avoid overfitting" Tags="|deep-learning|image-classification|accuracy|convolutional-neural-network|overfitting|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30098" PostTypeId="1" CreationDate="2018-04-09T19:43:51.067" Score="-1" ViewCount="118" Body="&lt;p&gt;I have created a classification ensemble with random forest as a base classifier. Each random forest has 500 trees. There are total 100 such forests in the ensemble. Majority voting is being used as the voting scheme. I was expecting the classifier to perform marginally better than a single random forest. However, its performance is drastically poorer than the random forest. Can anyone explain the reason?&lt;/p&gt;&#xA;" OwnerUserId="50257" LastEditorUserId="50257" LastEditDate="2018-04-09T21:44:17.003" LastActivityDate="2018-09-07T02:00:24.167" Title="Classification Ensemble with Random Forest as base classifier" Tags="|random-forest|ensemble-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30099" PostTypeId="2" ParentId="29464" CreationDate="2018-04-09T19:56:20.577" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;I can suggest &lt;a href=&quot;https://www.rdocumentation.org/packages/forecast/versions/8.1/topics/auto.arima&quot; rel=&quot;nofollow noreferrer&quot;&gt;auto.arima&lt;/a&gt; function from &lt;a href=&quot;https://www.rdocumentation.org/packages/forecast/versions/8.1&quot; rel=&quot;nofollow noreferrer&quot;&gt;forecast&lt;/a&gt; library, if you are R user, if you are Python user then follow &lt;a href=&quot;https://stackoverflow.com/questions/22770352/auto-arima-equivalent-for-python&quot;&gt;this&lt;/a&gt; link. All you need is to write simple for loop, which allows you to built best ARIMA models in different geographic locations:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   for (my_time_series in set_of_all_time_series){&#xA;        model=auto.arima(my_time_series )&#xA;         }&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can cluster your time series by correlation (make sure that your time series are stationary to avoid spurious correlation). If this reduces the number of time series (which will depend on threshold on correlation), you can take any 1 member from each class, build any model (not only ARIMA) and apply model results on each member of that class.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Contract &lt;a href=&quot;https://en.wikipedia.org/wiki/Vector_autoregression&quot; rel=&quot;nofollow noreferrer&quot;&gt;VAR&lt;/a&gt; model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50154" LastEditorUserId="50154" LastEditDate="2018-04-09T20:06:12.517" LastActivityDate="2018-04-09T20:06:12.517" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30102" PostTypeId="2" ParentId="30097" CreationDate="2018-04-09T21:43:59.400" Score="5" Body="&lt;p&gt;I am not familiar with the software you are using but keep in mind: You EXPECT accuracy to drop if you reduce over fitting. It is not a bad thing. Over-fitting is essentially &quot;fake accuracy&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some good approaches in general to avoid over-fitting though: Use cross-validation, normalize your features, increase size of data-set and dont just increase your data-set by copying data. &lt;/p&gt;&#xA;" OwnerUserId="50262" LastActivityDate="2018-04-09T21:43:59.400" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30103" PostTypeId="2" ParentId="30098" CreationDate="2018-04-09T21:46:28.720" Score="0" Body="&lt;p&gt;Ok first of all: Training set should not be Test-set. Use cross-validation or something similiar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then try different voting techniques, you will see differrent results. But the way you are doing this there is much of a reason why the average opinion of 500 forests should be better than the opinion of one forest. &lt;/p&gt;&#xA;" OwnerUserId="50262" LastActivityDate="2018-04-09T21:46:28.720" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30106" PostTypeId="1" AcceptedAnswerId="30153" CreationDate="2018-04-10T02:44:35.077" Score="0" ViewCount="88" Body="&lt;p&gt;i have a dataset of work period from a lot of people during multiple years and i would like to find the overlapping date between these periods, the data would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Name;Start;End;Valid&#xA;John Doe;2015-01-01;2017-01-01;Yes&#xA;Jonh Doe;2016-02-03;2016-06-04;No&#xA;Jane Doe;1980-01-01;2018-01-01;Yes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see i can have multiple lines for the same person and are mapped as valid or not, there are a few more rules like if the person is a professor he can have overlapping work periods but the core is to find these wrong periods created by human error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since i am a beginner in machine learning i would to try to implement this as a learning project although i intend to use and expand in the future if all goes well, at least to my eyes seems like a somewhat easy task to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like your opinion in what kind of algorithm i should use, or what framework i should or not use, or even if this is not very cost effective to implement on ML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of somekind of supervised learning using tensorflow, maybe is too much?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also any tips, tricks or tutorials would be much appreciate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT: More Information&lt;/strong&gt;&#xA;At first i am providing the answer, which would be the &quot;Yes&quot; or &quot;No&quot; on Valid, but when i provide something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Row;ID;Name;Start;End;&#xA;1;1;John Doe;2015-01-01;2017-01-01;&#xA;2;1;Jonh Doe;2016-02-03;2016-06-04;&#xA;3;1;Jonh Doe;2017-06-02;2018-04-01;&#xA;4;1;Jonh Doe;1990-01-01;2017-07-01;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So i would like it to point out, hey, John Doe has 2 invalid periods, row 2 and 4, &lt;strong&gt;why should it give me that?&lt;/strong&gt; because 2016-02-03 to 2016-06-04 is already covered by row 1 and row 4 overlap with row 1,2,3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At first i thought on making this with regular programming to check for everything, but there are some cases where human error make this quite a difficult task to do, thats why ml got in to my mind, the algorithm does not need to have really high accuracy, this is mostly a learning project.&lt;/p&gt;&#xA;" OwnerUserId="50269" LastEditorUserId="50269" LastEditDate="2018-04-11T03:22:31.300" LastActivityDate="2018-04-11T09:25:37.343" Title="Guidance to create a machine learning alg to find multiple overlapping date over multiple period" Tags="|machine-learning|tensorflow|supervised-learning|beginner|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30108" PostTypeId="1" CreationDate="2018-04-10T05:18:18.843" Score="1" ViewCount="80" Body="&lt;p&gt;I have been using the mlp library on the sklearn on some unsupervised data. I have been getting training set scores between 0.7 - 0.9 .However my test set scores have been really low. I assume this is due to over fitting and have read and understood that regularization (to help smooth weight/bias) and early stopping would help prevent this issue. However I am finding it hard to implement it, could someone help me with an implementation or guide me on the right path with a suitable link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(The tutorial on regularization for clf on sklearn did not really help me as I found it to be confusing with little explanation)&lt;/p&gt;&#xA;" OwnerUserId="50273" LastEditorUserId="50273" LastEditDate="2018-04-10T16:22:46.757" LastActivityDate="2018-04-10T16:22:46.757" Title="The MLP library on sklearn" Tags="|machine-learning|neural-network|scikit-learn|" AnswerCount="0" CommentCount="4" ClosedDate="2018-04-11T09:11:24.117" ContentLicense="CC BY-SA 3.0" />
  <row Id="30110" PostTypeId="2" ParentId="26528" CreationDate="2018-04-10T05:21:55.130" Score="8" Body="&lt;h1&gt;Implementations&lt;/h1&gt;&#xA;&lt;p&gt;These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.&lt;/p&gt;&#xA;&lt;h2&gt;Bad&lt;/h2&gt;&#xA;&lt;p&gt;WangZ's &lt;a href=&quot;https://github.com/wangz10/tensorflow-playground/blob/master/doc2vec.py&quot; rel=&quot;noreferrer&quot;&gt;implementation&lt;/a&gt; looks complete from a brief glance, although it only offers Adagrad and Stochastic Gradient Descent, two of the least reliable gradient descents. You really want minibatch gradient descent, which will predictably converge close to the local minimum where you initialize your system (think: optimization problems from calculus class). Beyond that, there's nothing really notable about his or her implementation.&lt;/p&gt;&#xA;&lt;h2&gt;Better&lt;/h2&gt;&#xA;&lt;p&gt;Jonathan's &lt;a href=&quot;https://github.com/JonathanRaiman/PVDM&quot; rel=&quot;noreferrer&quot;&gt;implementation&lt;/a&gt; is a bit better than WangZ's but worse than the Machine Learning Cookbook solution. &lt;strong&gt;He also uses Gensim instead of TensorFlow&lt;/strong&gt;, which is not bad except from an engineering standpoint (it doesn't scale quite as well). He uses mini-batch gradient descent, but with hierarchical softmax instead of NCE. This is not going to make a &lt;em&gt;huge&lt;/em&gt; difference except perhaps increasing training time.&lt;/p&gt;&#xA;&lt;h2&gt;Best&lt;/h2&gt;&#xA;&lt;p&gt;The Machine Learning Cookbook &lt;a href=&quot;https://github.com/PacktPublishing/TensorFlow-Machine-Learning-Cookbook/blob/master/Chapter%2007/doc2vec.py&quot; rel=&quot;noreferrer&quot;&gt;implementation&lt;/a&gt; is really just incredible. It follows all of the recommendations from Google's original papers on the topic, even improving on the gradient descent (this uses mini-batch gradient descent). Notably, this also uses NCE (Noise Contrastive Estimation), which I don't typically see, instead of hierarchical softmax to obtain the error for gradient descent, which Google showed produces slightly better results and works faster. A lot of thought clearly went into this, and whoever was behind it knew what s(he) was doing. &lt;strong&gt;I would go with this one if I were you.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;h1&gt;PV-DM vs PV-DBOW&lt;/h1&gt;&#xA;&lt;p&gt;PV stands for Paragraph Vectors, the model you are talking about. It is also know as doc2vec. It comes in two flavors, CBOW (Continuous Bag of Words), sometimes also called DBOW (Distributed Bag of Words), and DM (Distributed Memory).&lt;/p&gt;&#xA;&lt;p&gt;These are both distributed models in that each &amp;quot;neuron&amp;quot; contributes to the training task, but nothing has any meaning without the other &amp;quot;neurons&amp;quot; to give it context.&lt;/p&gt;&#xA;&lt;p&gt;To paraphrase Google's &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;noreferrer&quot;&gt;documentation&lt;/a&gt;: The CBOW/DBOW model predicts the target word from a bag of context words (e.g. &amp;quot;mat&amp;quot; from &amp;quot;the cat sits on the&amp;quot;) while the DM model predicts a &lt;a href=&quot;https://en.wikipedia.org/wiki/N-gram#Skip-gram&quot; rel=&quot;noreferrer&quot;&gt;skip-gram&lt;/a&gt; from a single word (e.g. &amp;quot;sits .. mat&amp;quot; from &amp;quot;cat&amp;quot;).&lt;/p&gt;&#xA;&lt;p&gt;In practice, research shows that the skip-gram/DM model works better if you have a lot of data and the CBOW/DBOW model works better with less. That said, this technique (and deep learning in general) isn't particularly effective without lots of training data.&lt;/p&gt;&#xA;&lt;h1&gt;IMDB&lt;/h1&gt;&#xA;&lt;p&gt;Any of these implementations can be trained on comment data from IMDB. For example, the Machine Learning Cookbook implementation is hard-coded to pull data from a similar &lt;a href=&quot;http://www.cs.cornell.edu/people/pabo/movie-review-data/&quot; rel=&quot;noreferrer&quot;&gt;dataset&lt;/a&gt; . To change where it gets the data from, you would modify the load_movie_data() function in &lt;strong&gt;text_helpers.py&lt;/strong&gt; (same directory in the repository as your link, but a different file) to pull from your preferred source. For instance, you might want to modify it to pull data from the local file-system instead.&lt;/p&gt;&#xA;" OwnerUserId="50274" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-11T01:54:20.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30111" PostTypeId="1" CreationDate="2018-04-10T05:56:18.477" Score="1" ViewCount="197" Body="&lt;p&gt;So after you train your model with millions of rows of data, what is the model being saved in disk look like? Is it file with millions of weights and biases that would do the best predictions?&lt;/p&gt;&#xA;" OwnerUserId="50275" LastEditorUserId="50275" LastEditDate="2018-04-10T20:02:15.723" LastActivityDate="2018-04-10T20:02:15.723" Title="What does a deep learning model look like in file?" Tags="|deep-learning|" AnswerCount="2" CommentCount="3" ClosedDate="2018-04-14T21:40:20.037" ContentLicense="CC BY-SA 3.0" />
  <row Id="30113" PostTypeId="1" CreationDate="2018-04-10T05:59:36.020" Score="2" ViewCount="277" Body="&lt;p&gt;I have a data set in which I have to predict the price of a building. Among many features there is a feature called &lt;strong&gt;Availability&lt;/strong&gt; which has two type values like :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&quot;Ready To Move&quot; and a date (e.g. &quot;20-Jan&quot;, &quot;18-May&quot;, etc.)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Availability           Price&#xA;------------           ----- &#xA;Ready To Move           55&#xA;18-Apr                  38&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How would you engineer this column to find its relationship with price and to draw other conclusions.&lt;/p&gt;&#xA;" OwnerUserId="37716" LastEditorUserId="8878" LastEditDate="2018-04-10T13:45:04.493" LastActivityDate="2019-05-25T03:09:36.187" Title="Feature Engineering of mixed data type column" Tags="|data-cleaning|feature-extraction|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30115" PostTypeId="2" ParentId="30111" CreationDate="2018-04-10T07:50:21.170" Score="0" Body="&lt;p&gt;It depends, there is no general rule. Can be as simple as a List or a JSON File though.&lt;/p&gt;&#xA;" OwnerUserId="50262" LastActivityDate="2018-04-10T07:50:21.170" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30118" PostTypeId="1" CreationDate="2018-04-10T09:17:30.420" Score="1" ViewCount="35" Body="&lt;p&gt;I am working on text analysis problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Person X can log in his goals and his actions to achieve his goal. Also their  score is calculated based on some formula to measure progress of the goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For ex:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Goal1 : Loose weight  ... Score: 70%&lt;/li&gt;&#xA;&lt;li&gt;Goal2: Finish reading this book.... Score: 10%&lt;/li&gt;&#xA;&lt;li&gt;Action1: Go to the gym&lt;/li&gt;&#xA;&lt;li&gt;Action2: Drink water&lt;/li&gt;&#xA;&lt;li&gt;Action3: Read one page daily&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Goals and actions don't have any relationship in my DB, so I joined them based on the person.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;I have as an input&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+--------------------------+---------------------+--------+-------+&#xA;|           Goal           |       Action        | Person | Score |&#xA;+--------------------------+---------------------+--------+-------+&#xA;| Lose weight              | Go to the gym       | X      |    70 |&#xA;| Lose weight              | Drink water         | X      |    70 |&#xA;| Lose weight              | Read one page daily | X      |    70 |&#xA;| Finish reading this book | Go to the gym       | X      |    10 |&#xA;| Finish reading this book | Drink water         | X      |    10 |&#xA;| Finish reading this book | Read one page daily | X      |    10 |&#xA;+--------------------------+---------------------+--------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to detect what are the good actions that can have high score and what are the bad actions that can have low score.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My Approach:&lt;/h2&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Apply clustering algorithm on Goals to find similar groups of goals then study their actions.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;My Text data dont have the type of action in it, it usually starts with a noun, so I manually created a new column for Actions like below, to make it appear in action form, also to unify the meaning of other actions into one.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+----------------------------------------+--------------------------+&#xA;|              Action Title              | Actions (Manually filled)|&#xA;+----------------------------------------+--------------------------+&#xA;| Sending Video Via whatsapp             | send video               |&#xA;| WhatsApp the video                     | send video               |&#xA;| WhatsApp Video Broadcasting for Higher | send video               |&#xA;| Sending Video Via Whatsapp             | send video               |&#xA;| Send video                             | send video               |&#xA;| Video Broadcast                        | send video               |&#xA;| Send the video                         | send video               |&#xA;| Sending Video via Whatsapp             | send video               |&#xA;| Share video with 10 contact customers  | send video               |&#xA;| New Strategy Owners Contacts           | create strategy contacts |&#xA;| New strategy unit contact              | create strategy contacts |&#xA;+----------------------------------------+--------------------------+&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I used R to build a term document matrix using bigram tokenizer (combination of 2 words).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;I found words appearing only in high score (0-75).&lt;/li&gt;&#xA;&lt;li&gt;I found words appearing only in low score (0-75).&lt;/li&gt;&#xA;&lt;li&gt;I calculated probability of common words to know if it will occur more in high or low.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Conclusion : I concluded step 4 are result of good actions while step 5 are results of bad actions. The common words are either added to step 4 or 5 depending on probability.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Suggestion&lt;/h2&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Am I going in the right path?&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Is there a ready API or a way to automate step 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did step 2 because when I count the words appearing in either high or low, I will find big number. Otherwise, depending only on bigrams will give me a word frequency of 1 for each word.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="32220" LastEditorUserId="8878" LastEditDate="2018-04-10T14:20:09.327" LastActivityDate="2018-04-10T14:20:09.327" Title="Find words related to high or low score" Tags="|machine-learning|r|clustering|text-mining|naive-bayes-classifier|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30123" PostTypeId="2" ParentId="30111" CreationDate="2018-04-10T11:42:28.507" Score="5" Body="&lt;p&gt;Yes you are right. It depends on which frameworks you are using for training the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if you use tensorflow framework, the trained model will be saved into several files. You can save and load model into two ways:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Checkpoint&lt;/li&gt;&#xA;&lt;li&gt;Savedmodel (freeze)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please check this link. &lt;a href=&quot;https://www.tensorflow.org/programmers_guide/saved_model&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In keras, Model weights are saved to HDF5 format. This is a grid format that is ideal for storing multi-dimensional arrays of numbers. you can save and load your model to file:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Save Model to JSON.&lt;/li&gt;&#xA;&lt;li&gt;Save Model to YAML.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The JSON format of the model looks like the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    {  &#xA;   &quot;keras_version&quot;:&quot;2.0.2&quot;,&#xA;   &quot;backend&quot;:&quot;theano&quot;,&#xA;   &quot;config&quot;:[  &#xA;      {  &#xA;     &quot;config&quot;:{  &#xA;        &quot;dtype&quot;:&quot;float32&quot;,&#xA;        &quot;bias_regularizer&quot;:null,&#xA;        &quot;activation&quot;:&quot;relu&quot;,&#xA;        &quot;bias_constraint&quot;:null,&#xA;        &quot;use_bias&quot;:true,&#xA;        &quot;bias_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;&#xA;           },&#xA;           &quot;class_name&quot;:&quot;Zeros&quot;&#xA;        },&#xA;        &quot;kernel_regularizer&quot;:null,&#xA;        &quot;activity_regularizer&quot;:null,&#xA;        &quot;kernel_constraint&quot;:null,&#xA;        &quot;trainable&quot;:true,&#xA;        &quot;name&quot;:&quot;dense_1&quot;,&#xA;        &quot;kernel_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;              &quot;maxval&quot;:0.05,&#xA;              &quot;minval&quot;:-0.05,&#xA;              &quot;seed&quot;:null&#xA;           },&#xA;           &quot;class_name&quot;:&quot;RandomUniform&quot;&#xA;        },&#xA;        &quot;batch_input_shape&quot;:[  &#xA;           null,&#xA;           8&#xA;        ],&#xA;        &quot;units&quot;:12&#xA;     },&#xA;     &quot;class_name&quot;:&quot;Dense&quot;&#xA;  },&#xA;  {  &#xA;     &quot;config&quot;:{  &#xA;        &quot;kernel_regularizer&quot;:null,&#xA;        &quot;bias_regularizer&quot;:null,&#xA;        &quot;activation&quot;:&quot;relu&quot;,&#xA;        &quot;bias_constraint&quot;:null,&#xA;        &quot;use_bias&quot;:true,&#xA;        &quot;bias_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;&#xA;           },&#xA;           &quot;class_name&quot;:&quot;Zeros&quot;&#xA;        },&#xA;        &quot;activity_regularizer&quot;:null,&#xA;        &quot;kernel_constraint&quot;:null,&#xA;        &quot;trainable&quot;:true,&#xA;        &quot;name&quot;:&quot;dense_2&quot;,&#xA;        &quot;kernel_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;              &quot;maxval&quot;:0.05,&#xA;              &quot;minval&quot;:-0.05,&#xA;              &quot;seed&quot;:null&#xA;           },&#xA;           &quot;class_name&quot;:&quot;RandomUniform&quot;&#xA;        },&#xA;        &quot;units&quot;:8&#xA;     },&#xA;     &quot;class_name&quot;:&quot;Dense&quot;&#xA;  },&#xA;  {  &#xA;     &quot;config&quot;:{  &#xA;        &quot;kernel_regularizer&quot;:null,&#xA;        &quot;bias_regularizer&quot;:null,&#xA;        &quot;activation&quot;:&quot;sigmoid&quot;,&#xA;        &quot;bias_constraint&quot;:null,&#xA;        &quot;use_bias&quot;:true,&#xA;        &quot;bias_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;&#xA;           },&#xA;           &quot;class_name&quot;:&quot;Zeros&quot;&#xA;        },&#xA;        &quot;activity_regularizer&quot;:null,&#xA;        &quot;kernel_constraint&quot;:null,&#xA;        &quot;trainable&quot;:true,&#xA;        &quot;name&quot;:&quot;dense_3&quot;,&#xA;        &quot;kernel_initializer&quot;:{  &#xA;           &quot;config&quot;:{  &#xA;              &quot;maxval&quot;:0.05,&#xA;              &quot;minval&quot;:-0.05,&#xA;              &quot;seed&quot;:null&#xA;           },&#xA;           &quot;class_name&quot;:&quot;RandomUniform&quot;&#xA;        },&#xA;        &quot;units&quot;:1&#xA;     },&#xA;     &quot;class_name&quot;:&quot;Dense&quot;&#xA;  }&#xA;&#xA;   ],&#xA;   &quot;class_name&quot;:&quot;Sequential&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50292" LastActivityDate="2018-04-10T11:42:28.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30128" PostTypeId="2" ParentId="30113" CreationDate="2018-04-10T13:26:12.643" Score="0" Body="&lt;p&gt;You can create a new column with name &quot;Number of days in which the building is going to be available&quot; (Values will be (20-Jan / 18-Apr )-current date).&lt;/p&gt;&#xA;" OwnerUserId="43983" LastActivityDate="2018-04-10T13:26:12.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30129" PostTypeId="2" ParentId="30113" CreationDate="2018-04-10T13:59:07.640" Score="1" Body="&lt;p&gt;You have several choices. Among them try these:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Discretization_of_continuous_features&quot; rel=&quot;nofollow noreferrer&quot;&gt;Discretize&lt;/a&gt; your target and use &lt;a href=&quot;http://uregina.ca/~gingrich/ch11a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;chi-square association measure&lt;/a&gt; or &lt;a href=&quot;https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cramer's V&lt;/a&gt; to see the association between variables.&lt;/li&gt;&#xA;&lt;li&gt;The other way around i.e. convert availability labels to numerics and simply try different association analysis methods like Correlation (&lt;a href=&quot;https://en.wikipedia.org/wiki/Pearson_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pearson&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spearman&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kendall&lt;/a&gt;) or &lt;a href=&quot;https://en.wikipedia.org/wiki/Mutual_information&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mutual Information&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;PS: Please note that Spearman or Kendall Rank Correlations can also be applied to your categorical variables as they are ordered (dates). &lt;/p&gt;&#xA;" OwnerUserId="8878" LastEditorUserId="8878" LastEditDate="2018-04-10T14:05:29.527" LastActivityDate="2018-04-10T14:05:29.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30130" PostTypeId="1" AcceptedAnswerId="30131" CreationDate="2018-04-10T14:14:03.840" Score="1" ViewCount="380" Body="&lt;p&gt;Below is the code I am trying to execute.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = np.random.normal(size=1000)&#xA;fig, ax = plt.subplots()&#xA;H = ax.hist(x, bins=50, alpha=0.5, histtype='stepfilled')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anybody elaborate what fig, ax = plt.subplots() doing here ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="28175" LastEditDate="2018-04-10T14:48:35.203" LastActivityDate="2020-08-01T16:28:19.960" Title="What plt.subplots() doing here?" Tags="|python|matplotlib|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30131" PostTypeId="2" ParentId="30130" CreationDate="2018-04-10T14:51:08.197" Score="3" Body="&lt;p&gt;As you can read from &lt;a href=&quot;https://stackoverflow.com/a/34162641/5120235&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;&lt;code&gt;plt.subplots()&lt;/code&gt; is a function that returns a tuple containing a figure&#xA;and axes object(s). Thus when using &lt;code&gt;fig, ax = plt.subplots()&lt;/code&gt; you&#xA;unpack this tuple into the variables &lt;code&gt;fig&lt;/code&gt; and &lt;code&gt;ax&lt;/code&gt;. Having &lt;code&gt;fig&lt;/code&gt; is useful&#xA;if you want to change figure-level attributes or save the figure as an&#xA;image file later (e.g. with &lt;code&gt;fig.savefig('yourfilename.png')&lt;/code&gt;. You&#xA;certainly don't have to use the returned figure object but many people&#xA;do use it later so it's common to see. Also, all axes objects (the&#xA;objects that have plotting methods), have a parent figure object.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The parameter &lt;code&gt;bins&lt;/code&gt; is used to set the number of ranges to be used to accumulate the data in those ranges. I don't know where else you have question.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="98307" LastEditDate="2020-08-01T16:28:19.960" LastActivityDate="2020-08-01T16:28:19.960" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30132" PostTypeId="1" CreationDate="2018-04-10T14:54:26.510" Score="1" ViewCount="2644" Body="&lt;p&gt;How can i &lt;strong&gt;Save&lt;/strong&gt; the &lt;strong&gt;TensorFlow model&lt;/strong&gt; using &lt;code&gt;estimator.export_savedmode()&lt;/code&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Especially, what should i put inside the &lt;code&gt;serving_input_receiver_fn()&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have created a Custom Estimator based on VGGNet Architecture, i am using my own images and doing some transformation (you can see them in &lt;code&gt;_parse_function()&lt;/code&gt;) on the images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read the documentation &lt;a href=&quot;https://www.tensorflow.org/programmers_guide/saved_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, but i am exactly not sure what to write for my code (please see below). Ultimately i want to save the model and use TensorFlow Serving.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    from __future__ import absolute_import&#xA;    from __future__ import division&#xA;    from __future__ import print_function&#xA;&#xA;    import tensorflow as tf&#xA;    import numpy as np&#xA;    from sklearn.model_selection import train_test_split&#xA;    import os&#xA;    import matplotlib.pyplot as plt&#xA;    import matplotlib.image as mpimg&#xA;    import scipy&#xA;    from scipy import ndimage&#xA;    import scipy.misc&#xA;&#xA;    tf.logging.set_verbosity(tf.logging.INFO)&#xA;&#xA;    def cnn_model_fn(features, labels, mode):&#xA;        &quot;&quot;&quot;Model function for CNN.&quot;&quot;&quot;&#xA;        # Input Layer&#xA;        input_layer = tf.reshape(features, [-1, 224, 224, 3])&#xA;&#xA;    # Convolutional Layer #1&#xA;    conv1 = tf.layers.conv2d(&#xA;      inputs=input_layer,&#xA;      filters=64,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;    # Convolutional Layer #2&#xA;    conv2 = tf.layers.conv2d(&#xA;      inputs=conv1,&#xA;      filters=64,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;    # Pooling Layer #1&#xA;    pool1 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)&#xA;&#xA;    # Convolutional Layer #3&#xA;    conv3 = tf.layers.conv2d(&#xA;      inputs=pool1,&#xA;      filters=128,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #4&#xA;    conv4 = tf.layers.conv2d(&#xA;      inputs=conv3,&#xA;      filters=128,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;    pool2 = tf.layers.max_pooling2d(inputs=conv4, pool_size=[2, 2], strides=2)&#xA;&#xA;    # Convolutional Layer #5&#xA;    conv5 = tf.layers.conv2d(&#xA;      inputs=pool2,&#xA;      filters=256,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #6&#xA;    conv6 = tf.layers.conv2d(&#xA;      inputs=conv5,&#xA;      filters=256,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #7&#xA;    conv7 = tf.layers.conv2d(&#xA;      inputs=conv6,&#xA;      filters=256,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    pool3 = tf.layers.max_pooling2d(inputs=conv7, pool_size=[2, 2], strides=2)&#xA;&#xA;    # Convolutional Layer #8&#xA;    conv8 = tf.layers.conv2d(&#xA;      inputs=pool3,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #9&#xA;    conv9 = tf.layers.conv2d(&#xA;      inputs=conv8,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #10&#xA;    conv10 = tf.layers.conv2d(&#xA;      inputs=conv9,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;    pool4 = tf.layers.max_pooling2d(inputs=conv10, pool_size=[2, 2], strides=2)&#xA;&#xA;    # Convolutional Layer #11&#xA;    conv11 = tf.layers.conv2d(&#xA;      inputs=pool4,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #12&#xA;    conv12 = tf.layers.conv2d(&#xA;      inputs=conv11,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    # Convolutional Layer #12&#xA;    conv13 = tf.layers.conv2d(&#xA;      inputs=conv12,&#xA;      filters=512,&#xA;      kernel_size=[3, 3],&#xA;      padding=&quot;same&quot;,&#xA;      activation=tf.nn.relu)&#xA;&#xA;    pool5 = tf.layers.max_pooling2d(inputs=conv13, pool_size=[2, 2], strides=2)&#xA;    # Dense Layer&#xA;    pool5_flat = tf.reshape(pool5, [-1, 7 * 7 * 512])&#xA;    dense1 = tf.layers.dense(inputs=pool5_flat, units=4096, activation=tf.nn.relu)&#xA;    dense2 = tf.layers.dense(inputs=dense1, units=4096, activation=tf.nn.relu)&#xA;    dense3 = tf.layers.dense(inputs=dense2, units=1024, activation=tf.nn.relu)&#xA;&#xA;    dropout = tf.layers.dropout(&#xA;      inputs=dense3, rate=0.001, training=mode == tf.estimator.ModeKeys.TRAIN)&#xA;&#xA;    logits1 = tf.layers.dense(inputs=dropout, units=2)&#xA;    logits2 = tf.layers.dense(inputs=dropout, units=4)&#xA;&#xA;    predictions = {&#xA;        &quot;classes1&quot;: tf.argmax(input=logits1, axis=1),&#xA;        &quot;classes2&quot;: tf.argmax(input=logits2, axis=1),&#xA;        &quot;probabilities1&quot;: tf.nn.softmax(logits1, name=&quot;softmax_tensor_1&quot;),&#xA;        &quot;probabilities2&quot;: tf.nn.softmax(logits2, name=&quot;softmax_tensor_2&quot;)&#xA;    }&#xA;&#xA;    if mode == tf.estimator.ModeKeys.PREDICT:&#xA;        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)&#xA;&#xA;    # Calculate Loss (for both TRAIN and EVAL modes)&#xA;    loss1 = tf.losses.sparse_softmax_cross_entropy(labels=labels[:,0], logits=logits1)&#xA;    loss2 = tf.losses.sparse_softmax_cross_entropy(labels=labels[:,1], logits=logits2)&#xA;    loss = loss1 + loss2&#xA;&#xA;    # Configure the Training Op (for TRAIN mode)&#xA;    if mode == tf.estimator.ModeKeys.TRAIN:&#xA;        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)&#xA;        train_op = optimizer.minimize(&#xA;            loss=loss,&#xA;            global_step=tf.train.get_global_step())&#xA;        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)&#xA;&#xA;    # Add evaluation metrics (for EVAL mode)&#xA;    eval_metric_ops = {&#xA;        &quot;accuracy1&quot;: tf.metrics.accuracy(&#xA;            labels=labels[:,0], predictions=predictions[&quot;classes1&quot;]),&#xA;        &quot;accuracy2&quot;: tf.metrics.accuracy(&#xA;            labels=labels[:,1], predictions=predictions[&quot;classes2&quot;]),&#xA;        &quot;precision1&quot;: tf.metrics.precision(labels=labels[:,0], predictions=predictions[&quot;classes1&quot;]),&#xA;        &quot;precision2&quot;: tf.metrics.precision(labels=labels[:,1], predictions=predictions[&quot;classes2&quot;]),&#xA;        &quot;recall1&quot;: tf.metrics.recall(labels=labels[:,0], predictions=predictions[&quot;classes1&quot;]),&#xA;        &quot;recall2&quot;: tf.metrics.recall(labels=labels[:,1], predictions=predictions[&quot;classes2&quot;])&#xA;    }&#xA;&#xA;    return tf.estimator.EstimatorSpec(&#xA;      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)&#xA;&#xA;    def _parse_function(filename, label):&#xA;        image_string = tf.read_file(filename)&#xA;        image_decoded = tf.image.decode_image(image_string)&#xA;        image_typecasted = tf.cast(image_decoded, tf.float32)&#xA;        image_reshaped = tf.reshape(image_typecasted, [-1, 224, 224, 3])&#xA;        return image_reshaped, label&#xA;&#xA;    def _parse_function(filename):&#xA;        image_string = tf.read_file(filename)&#xA;        image_decoded = tf.image.decode_image(image_string)&#xA;        image_typecasted = tf.cast(image_decoded, tf.float32)&#xA;        image_reshaped = tf.reshape(image_typecasted, [-1, 224, 224, 3])&#xA;        return image_reshaped&#xA;&#xA;    def stratified_train_test_split_():&#xA;        filenamelist = []&#xA;        labelslist = []&#xA;        DIRECTORY = 'path_to'&#xA;&#xA;    for filename in os.listdir(DIRECTORY):&#xA;        fullfilename = DIRECTORY + filename&#xA;        if filename.endswith('.back.0.jpg'):&#xA;            #back image, original orientation&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [0,0]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.back.90.jpg'):&#xA;            #back image, rotated clockwise 90&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [0,1]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.back.180.jpg'):&#xA;            #back image, rotated clockwise 180&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [0,2]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.back.270.jpg'):&#xA;            #back image, rotated clockwise 270&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [0,3]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.front.0.jpg'):&#xA;            #front image, rotated clockwise 0&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [1,0]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.front.90.jpg'):&#xA;            #front image, rotated clockwise 90&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [1,1]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.front.180.jpg'):&#xA;            #front image, rotated clockwise 180&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [1,2]&#xA;            labelslist.append(temp)&#xA;&#xA;        elif filename.endswith('.front.270.jpg'):&#xA;            #front image, rotated clockwise 270&#xA;            filenamelist.append(fullfilename)&#xA;            temp = [1,3]&#xA;            labelslist.append(temp)&#xA;&#xA;    X_train, X_test, y_train, y_test = train_test_split(filenamelist, labelslist, test_size=0.20, random_state=42, shuffle=True, stratify=labelslist)&#xA;    return X_train, X_test, y_train, y_test&#xA;&#xA;&#xA;def my_input_fn_train(X_train, y_train):&#xA;    filenames = tf.constant(X_train)&#xA;    labels = tf.constant(y_train)&#xA;    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))&#xA;    dataset = dataset.map(_parse_function)&#xA;    # # Shuffle, repeat, and batch the examples.&#xA;    dataset = dataset.shuffle(5000).repeat().batch(64)&#xA;    # # Build the Iterator, and return the read end of the pipeline.&#xA;    return dataset.make_one_shot_iterator().get_next()&#xA;&#xA;def my_input_fn_test(X_test, y_test):&#xA;    filenames = tf.constant(X_test)&#xA;    labels = tf.constant(y_test)&#xA;    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))&#xA;    dataset = dataset.map(_parse_function)&#xA;    # # Shuffle, repeat, and batch the examples.&#xA;    dataset = dataset.shuffle(5000).repeat(1).batch(64)        &#xA;    # # Build the Iterator, and return the read end of the pipeline.&#xA;    return dataset.make_one_shot_iterator().get_next()&#xA;&#xA;def my_input_fn_predict(filename):    &#xA;    filenames = tf.constant(filename)&#xA;    dataset = tf.data.Dataset.from_tensors((filenames))&#xA;    dataset = dataset.map(_parse_function)&#xA;    return dataset.make_one_shot_iterator().get_next()&#xA;&#xA;def main(unused_argv):&#xA;&#xA;    # Create the Estimator&#xA;    mnist_classifier = tf.estimator.Estimator(&#xA;    model_fn=cnn_model_fn, &#xA;    model_dir=&quot;path_to_model_directory&quot;,&#xA;    config = tf.estimator.RunConfig( save_checkpoints_steps=None, save_checkpoints_secs=600, save_summary_steps=5))&#xA;    # Set up logging for predictions&#xA;    tensors_to_log_1 = {&quot;probabilities1&quot;: &quot;softmax_tensor_1&quot;}&#xA;    tensors_to_log_2 = {&quot;probabilities2&quot;: &quot;softmax_tensor_2&quot;}&#xA;    logging_hook_1 = tf.train.LoggingTensorHook(&#xA;        tensors=tensors_to_log_1, every_n_iter=100)&#xA;    logging_hook_2 = tf.train.LoggingTensorHook(&#xA;        tensors=tensors_to_log_2, every_n_iter=100)&#xA;&#xA;    #Splitting the train test split seperately&#xA;    X_train, X_test, y_train, y_test = stratified_train_test_split_()&#xA;&#xA;    #Removed the training, testing and prediction calls.&#xA;&#xA;    #Code for exporting the models using &#xA;    def serving_input_receiver_fn():&#xA;      #????&#xA;&#xA;    mnist_classifier.export_savedmodel(export_dir_base, serving_input_fn)&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;  tf.app.run()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50300" LastActivityDate="2018-04-10T14:54:26.510" Title="How to Save TensorFlow model using estimator.export_savemodel()" Tags="|machine-learning|deep-learning|tensorflow|estimators|" AnswerCount="0" CommentCount="1" ClosedDate="2020-03-09T19:17:33.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="30134" PostTypeId="2" ParentId="30013" CreationDate="2018-04-10T15:02:26.170" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I now want to analyze the factors which are most contributing to the medical condition&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If this is your purpose I suggest you plotting covariance matrix for input features. By doing so you will be able to see the correlation of your features together. That may help you diminish redundant correlated features &lt;a href=&quot;https://datascience.stackexchange.com/a/25271/28175&quot;&gt;1&lt;/a&gt;.  Moreover, you can plot the correlation of your outcomes and each feature to see their relationship. That helps you figure out which features are correlated or uncorrelated to the outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;PCA&lt;/code&gt; helps you find major components are needed to describe the phenomena &lt;a href=&quot;https://datascience.stackexchange.com/a/26719/28175&quot;&gt;2&lt;/a&gt;. Neural nets are useful for making a model that can predict for labeled data-sets. For your case, I've not seen any paper but there might be.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-10T15:02:26.170" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30135" PostTypeId="2" ParentId="30013" CreationDate="2018-04-10T15:33:10.053" Score="2" Body="&lt;p&gt;If you choose your alternative to Tree based models, then you really have an upper edge here as compared to all other linear/logistic Regressions etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;People generally use&lt;code&gt;Co-relation, Co-variance and heat maps.. etc&lt;/code&gt; as a process which often generates tables of distances with even more numbers than the original data, but making &lt;code&gt;Dendrograms&lt;/code&gt; in fact simplifies our understanding of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Distances between objects can be visualized in many simple and evocative ways, one of them is &lt;code&gt;Hierarchial Clustering&lt;/code&gt;..&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What is Hierarchical Clustering?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hierarchical clustering is where you build a cluster tree (a dendrogram) to represent data, where each group (or “node”) links to two or more successor groups. The groups are nested and organized as a tree, which ideally ends up as a meaningful classification scheme.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So now, What is a Dendrogram?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A dendrogram is a type of tree diagram showing hierarchical clustering — relationships between similar sets of data. They are frequently used in biology to show clustering between genes or samples, but they can represent any type of grouped data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns under the same split at the leaves are somewhat having relationships between them or have similar attributes.., that is what we try to Explore and deepen out understanding about in order to cut down redundant Features....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sample Dendrogram looks like this one...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dwUTK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dwUTK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So Let's try to interpret the Diagram Now, &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Removing redundant features&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing that makes it harder to interpret a variable is that there seem to be some variables with very similar meanings(redundant features...)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's try removing some of these related features to see if the model can be simplified without impacting the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_oob(df):&#xA;    m = RandomForestRegressor(n_estimators=30, min_samples_leaf=5, max_features=0.6, n_jobs=-1, oob_score=True)&#xA;    x, _ = split_vals(df, n_trn)&#xA;    m.fit(x, y_train)&#xA;    return m.oob_score_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here's our baseline.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;get_oob(df_keep)&#xA;0.88999425494301454&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now we try removing each variable one at a time....&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for c in ('saleYear', 'saleElapsed', 'fiModelDesc', 'fiBaseModel', 'Grouser_Tracks', 'Coupler_System'):&#xA;    print(c, get_oob(df_keep.drop(c, axis=1)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output(s)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;saleYear 0.889037446375&#xA;saleElapsed 0.886210803445&#xA;fiModelDesc 0.888540591321&#xA;fiBaseModel 0.88893958239&#xA;Grouser_Tracks 0.890385236272&#xA;Coupler_System 0.889601052658&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It looks like we can try one from each group for removal. Let's see what that does.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;to_drop = ['saleYear', 'fiBaseModel', 'Grouser_Tracks']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Looks good even after dropping some of the columns....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;References -:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Dendrogram&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wiki Link&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://nbviewer.jupyter.org/github/fastai/fastai/blob/master/courses/ml1/lesson2-rf_interpretation.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Notebook Link - fast.ai(Jeremy is Just Awesome..)&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.nonlinear.com/support/progenesis/comet/faq/v2.0/dendrogram.aspx&quot; rel=&quot;nofollow noreferrer&quot;&gt;Blog Link&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Edit - (just to make the answer complete)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also have something known as &lt;strong&gt;Partial Dependencies&lt;/strong&gt; while using RF's which is also a very helpful insight to explore even further...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-11T11:37:52.450" LastActivityDate="2018-04-11T11:37:52.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30136" PostTypeId="1" CreationDate="2018-04-10T16:10:16.187" Score="1" ViewCount="631" Body="&lt;p&gt;Is it possible to use the Levenshtein (edit) distance of two strings as the error function my model attempts to optimize?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The inputs are images representing the letters in a text document. The output should be the predicted letters from the image. I want to combine them into a text document, and then adjust the model weights to minimize the levenshtein distance of the input document, and the predicted document. Can I do this by extending &lt;code&gt;nn.Module&lt;/code&gt; to create a loss function in &lt;code&gt;pytorch&lt;/code&gt;, and then using &lt;code&gt;loss.backward&lt;/code&gt; to update the weights in the model? &lt;/p&gt;&#xA;" OwnerUserId="43048" LastEditorUserId="1330" LastEditDate="2021-07-10T23:27:16.683" LastActivityDate="2021-07-10T23:27:16.683" Title="Pytorch loss function for Levenshtein (edit) distance between letters in images" Tags="|machine-learning|pytorch|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30137" PostTypeId="2" ParentId="18393" CreationDate="2018-04-10T16:22:37.853" Score="1" Body="&lt;p&gt;I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given I work for Talend, I would suggest our data fabric. =)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here’s a case study with the Panama Papers. &lt;a href=&quot;https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The concept in that case study of combining a data fabric with analytic tools is the general concept to use regardless of which data fabric you use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can find the trial and open versions of Talend at &lt;a href=&quot;https://www.talend.com/download/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.talend.com/download/&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: Here’s another example, which &lt;em&gt;implements a backend and UI showing the ebooks of the Gutenberg project. It allows to import the whole gutenberg index using a camel route.&lt;/em&gt; &lt;a href=&quot;https://github.com/Talend/tesb-rt-se/tree/master/examples/tesb/ebook&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/Talend/tesb-rt-se/tree/master/examples/tesb/ebook&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="50307" LastEditorUserId="50307" LastEditDate="2018-04-11T01:22:54.690" LastActivityDate="2018-04-11T01:22:54.690" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30138" PostTypeId="2" ParentId="12230" CreationDate="2018-04-10T17:33:25.820" Score="0" Body="&lt;p&gt;One approach could be the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Make a vector respresentation of all texts in the data set, for example with tfidf technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Take the first vector and put it in a pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Enter in the following loop:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3a) take the next vector and compute the cosine similarity between this vector and the centroid of each built pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3b) if one of this cosine similarity falls below a predefined threshold, stack this document representation in the corresponding pile. Another case, build a new pile with this vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3c) recompute the centroid of each modified pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This algorithm is going to find similar tweets, which we suppose that are related with same topic.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-10T17:33:25.820" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30139" PostTypeId="2" ParentId="12281" CreationDate="2018-04-10T17:36:45.297" Score="1" Body="&lt;p&gt;Have you found a good approach?&#xA;I am envolved in the same work right now.&#xA;My approach is the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Make a vector respresentation of all texts in the data set, for example with tfidf technique.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Take the first vector and put it in a pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Enter in the following loop:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3a) take the next vector and compute the cosine similarity between this vector and the centroid of each built pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3b) if one of this cosine similarity falls below a predefined threshold, stack this document representation in the corresponding pile. Another case, build a new pile with this vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3c) recompute the centroid of each modified pile.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This algorithm is going to find similar tweets, which we suppose that are related with same topic.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-10T17:36:45.297" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30140" PostTypeId="2" ParentId="11553" CreationDate="2018-04-10T17:45:11.710" Score="1" Body="&lt;p&gt;If you are trying to add weights to rare or infrequent terms, which appear only in few texts, definetly you should use the tf-idf technique, which computes the frequency of each word on all the data set and after that computes a weight of each word in each text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another case, if you want to add weights to specific words, you just can modify the tf-idf technique.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-10T17:45:11.710" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30141" PostTypeId="1" CreationDate="2018-04-10T18:00:20.480" Score="5" ViewCount="506" Body="&lt;p&gt;Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E.g. An SVM with a Gaussian kernel is linear in infinite-dimensional space.&lt;/p&gt;&#xA;" OwnerUserId="50312" LastActivityDate="2019-04-09T00:01:25.633" Title="Are all classifiers linear in some high dimensional space?" Tags="|classification|theory|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30142" PostTypeId="1" AcceptedAnswerId="30145" CreationDate="2018-04-10T18:07:04.577" Score="2" ViewCount="123" Body="&lt;p&gt;I am involved in twitter analysis data. I want to find trending topics in tweets  with some hashtags, like #finance or #technology. I have a hugh data set of tweets and now I need to analyze them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to recognize topics, if there are. They way I'm approaching this is, first, performing a vector representation of each tweet, with a tfidf technique, and then, build groups of them based on their cosine similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there common techniques in tweets analysis?&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-04-10T19:06:01.510" LastActivityDate="2018-04-12T18:39:39.137" Title="twitter data analysis?" Tags="|classification|scikit-learn|clustering|bigdata|twitter|" AnswerCount="2" CommentCount="4" ClosedDate="2018-04-14T21:42:34.557" ContentLicense="CC BY-SA 3.0" />
  <row Id="30144" PostTypeId="2" ParentId="29117" CreationDate="2018-04-10T19:12:47.677" Score="1" Body="&lt;p&gt;Applying common categorical labels to words is typically called &lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;Named-entity recognition (NER)&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;NER can be done by static rules (e.g., regular expressions) or learned rules (e.g., decision trees). These rules are often brittle and do not generalize. &lt;a href=&quot;https://en.wikipedia.org/wiki/Conditional_random_field&quot; rel=&quot;nofollow noreferrer&quot;&gt;Conditional Random Fields (CRF)&lt;/a&gt; are often a better solution because they are able to model the latent states of languages. Current state-of-the-art performance in &lt;a href=&quot;https://arxiv.org/abs/1707.05928&quot; rel=&quot;nofollow noreferrer&quot;&gt;NER is done with a combination of Deep Learning models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://nlp.stanford.edu/software/CRF-NER.shtml#Download&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Named Entity Recognizer&lt;/a&gt; and &lt;a href=&quot;https://spacy.io/api/entityrecognizer&quot; rel=&quot;nofollow noreferrer&quot;&gt;spaCy&lt;/a&gt; are packages to perform NER.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-10T19:12:47.677" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30145" PostTypeId="2" ParentId="30142" CreationDate="2018-04-10T20:49:58.207" Score="3" Body="&lt;p&gt;I believe that the algorithm that you want to use is something called a latent dirichlet allocation (LDA) model. This model is designed to uncover the topics in a corpus of documents. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit learn has an implementation.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They even have a &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/applications/plot_topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-plot-topics-extraction-with-nmf-lda-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; which teaches you how to extract topics. The tutorial also describes Non-negative Matrix Factorization (NNMF) as a method to extract the topics. I can't vouch for this algorithm, because I haven't used it personally (as opposed to LDA which I have used before), but from their tutorial NNMF does seem to give reasonable results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using cosine similarity will help you to group tweets that are most similar, but it wouldn't give you their topics. Which may be what you want? It really is hard to say, because only you would know how you should have the system behave. Unfortunately, that doesn't help you figure out what is trending, and you will need to do some heavy post-processing to make whatever algorithm you use spit out something that is useful to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-04-10T20:49:58.207" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30147" PostTypeId="2" ParentId="30064" CreationDate="2018-04-10T22:15:53.870" Score="2" Body="&lt;p&gt;As is common in R the answer depends and you should always double check to make sure including the target (or other) doesn’t affect the answer. But generally it won’t. Superfluous features are ignored as long as newdata includes all expected predictors present during training. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-04-10T22:15:53.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30150" PostTypeId="1" AcceptedAnswerId="30151" CreationDate="2018-04-11T03:39:04.470" Score="2" ViewCount="136" Body="&lt;p&gt;I am trying to use Machine Learning to predict the load of a residence at any point in time for a whole year. I have past data pertaining to that house. So I have the training data and I need the algorithm to predict future loads of the house.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on my knowledge, I have found the &quot;supervised&quot; machine learning technique to be the one I must adapt. I figured this out since I have labelled test data, I have a prediction requirement and I can get feedback for my prediction (cross-checking with the actual value). Am I correct here? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I read online that &quot;Unsupervised&quot; learning is to be used at places where we need to find &quot;Hidden data structure&quot;. I assume it means pattern. If so, what is the difference between the unsupervised and supervised learning in my case. Both of them will give me a prediction about the future load pertaining to that house at any point in time.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;My background&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I am doing my Masters in EE (Power systems). I am new to Machine Learning as well. &lt;/p&gt;&#xA;" OwnerUserId="50325" LastEditorUserId="8820" LastEditDate="2018-04-18T07:21:12.203" LastActivityDate="2018-04-18T08:53:59.603" Title="Which is the best Machine learning technique for this Load forecasting problem?" Tags="|machine-learning|unsupervised-learning|supervised-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30151" PostTypeId="2" ParentId="30150" CreationDate="2018-04-11T03:48:48.187" Score="2" Body="&lt;p&gt;I think this is one of the time series forecasting because you want to predict the future load of residence with the past dataset of residence's load by time. By my experience I recommend using LSTM RNN for the solution. An LSTM is well-suited to classify, process and predict time series given time lags of unknown size and duration between important events. Of course, this is supervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for difference between supervised learning and unsupervised learning on your case, you don't need to use unsupervised learning for this solution because you have already dataset with input/correct output. if you are training your machine learning task for every input with corresponding target, it is called supervised learning, which will be able to provide target for any new input after sufficient training. Contrary, if you are training your machine learning task only with a set of inputs, it is called unsupervised learning, which will be able to find the structure or relationships between different inputs. It says you should use supervised learning - RNN or CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: as for framework, I recommend using Tensorflow or Keras.&lt;/p&gt;&#xA;" OwnerUserId="50292" LastActivityDate="2018-04-11T03:48:48.187" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30153" PostTypeId="2" ParentId="30106" CreationDate="2018-04-11T05:31:28.423" Score="0" Body="&lt;p&gt;Sorting and cleaning this dataset from overlapping periods is a task you can do  with plain python, perhaps a bit more comfortable if you use pandas.  Machine learning would be an additional overkill, it adds more complexity and is of no help for that task. Although I understand you are looking for a learning project, I do not recommend to use this one, and rather recommend  to pick a task that is more often solved with machine learning.   &lt;/p&gt;&#xA;" OwnerUserId="21163" LastEditorUserId="21163" LastEditDate="2018-04-11T06:11:52.467" LastActivityDate="2018-04-11T06:11:52.467" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30155" PostTypeId="1" AcceptedAnswerId="30562" CreationDate="2018-04-11T07:40:45.377" Score="1" ViewCount="350" Body="&lt;p&gt;I checked this &lt;a href=&quot;https://www.quora.com/What-are-the-best-tools-for-manually-annotating-a-text-corpus-with-entities-and-relationships&quot; rel=&quot;nofollow noreferrer&quot;&gt;old question&lt;/a&gt; on tools for text corpus annotation and there are many good tools there, however the tools there seem not to cover my problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a linear text segmentation problem, in which a text needs to be segmented in different sections (topics). I'm building a model for it, but I have no training data. Therefore, the team decided to manually label some text, by annotating blocks in the text that represent each section. I tried some NER or POS labelling tools, but they are not very convenient for selecting several lines and paragraphs to annotate a label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a good tool for human annotation of text segmentation? &lt;/p&gt;&#xA;" OwnerUserId="50338" LastEditorUserId="26562" LastEditDate="2018-04-18T00:08:31.077" LastActivityDate="2018-04-20T08:03:44.897" Title="What are the best tools for manually annotating textual topic segmentation?" Tags="|machine-learning|nlp|annotation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30156" PostTypeId="1" AcceptedAnswerId="30157" CreationDate="2018-04-11T08:00:28.160" Score="0" ViewCount="37" Body="&lt;p&gt;I finished my Economics thesis using RStudio, but my script was very slow due to massive RAM consumption during the process.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;My Case&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I had a massive dataset (stock prices in daily frequency for 10 years, ~700 stocks i.e. $3500\times700$) and I was picking each stock as a vector to decompose it into wavelets and CF filter (2 datasets $28000\times700$) and apply benford's law (two datasets $9\times700$).&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;The Problem&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;RStudio was storing my datasets in memory and they were consuming a significant proportion just by touching them.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h2&gt;Question&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I started learning basic SQL commands and I found out that I can call specific columns from a certain table. Would my script be more efficient if I was calling my stocks one by one as vectors from there instead of picking them directly from RStudio? In other words, do queries call the whole dataset and then retrieve the requested values or do they follow a kind of shortcut to be memory efficient? If not, what's the purpose of using databases for domestic use?&lt;/p&gt;&#xA;" OwnerUserId="24984" LastActivityDate="2018-04-11T08:38:45.470" Title="Do DBMS decrease Memory requirements?" Tags="|r|efficiency|rstudio|relational-dbms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30157" PostTypeId="2" ParentId="30156" CreationDate="2018-04-11T08:38:45.470" Score="1" Body="&lt;p&gt;Welcome to the site!&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Would my script be more efficient if I was calling my stocks one by&#xA;  one as vectors from there instead of picking them directly from&#xA;  RStudio?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes as you told, calling the specific columns from Database is better than extracting everything. I generally use Dataframes rather than vectors. Dataframes are very efficient and the transformation using Dataframes is much easier and better. You can go through this &lt;a href=&quot;https://stackoverflow.com/questions/2280276/vector-vs-data-frame-in-r&quot;&gt;link&lt;/a&gt;, for better understanding on Vector Vs Dataframe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I use 1 year data and consists of 100,000 records. Query takes like 8 minutes to extract the data in Rstudio using SQL DB and store it in an R-dataframe. After the extraction is over I don't hit the database meaning no read and write on the database by which the DB Server is not effected. After doing all the modeling and finally I commit the data into SQL DB to store the committed data(results).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In other words, do queries call the whole dataset and then retrieve&#xA;  the requested values or do they follow a kind of shortcut to be memory&#xA;  efficient?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It depends on how you write the query, in my experience when I tried using temp-tables and is much efficient but when tried running that query in R but it wasn't supporting so had to stick to nested queries. If you don't write the nested queries in an optimized manner then it takes more time to extract the data(memory lekage).&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-04-11T08:38:45.470" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30159" PostTypeId="2" ParentId="30106" CreationDate="2018-04-11T09:22:57.243" Score="1" Body="&lt;p&gt;This is quite easy and you don't need machine learning at all. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Order by Person and Start date. Then simply iterate over each data line for a person and see if the start dates are earlier than the end dates for the previous entry. Add rules as necessary.&lt;/p&gt;&#xA;" OwnerUserId="50262" LastEditorUserId="40853" LastEditDate="2018-04-11T09:25:37.343" LastActivityDate="2018-04-11T09:25:37.343" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30160" PostTypeId="1" CreationDate="2018-04-11T09:32:52.763" Score="1" ViewCount="627" Body="&lt;p&gt;I have a big dataset with nearly 200 features. However, I do not have class labels for these data. I want to divide these data into two classes based on these features. I know, when we do not have class labels we have to use some clustering method. However, since I do not have any labels, I am just wondering how to measure the accuracy of the models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please let me know the most suitable approach that I should follow?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am happy to provide more details about my featureset if needed :)&lt;/p&gt;&#xA;" OwnerUserId="45262" LastActivityDate="2018-06-10T16:26:47.047" Title="How to classify features into two classes without labels?" Tags="|machine-learning|data-mining|nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30163" PostTypeId="2" ParentId="30160" CreationDate="2018-04-11T09:55:47.013" Score="0" Body="&lt;p&gt;Applying machine learning without labels is called &lt;strong&gt;unsupervised learning&lt;/strong&gt;. These methods are definitely harder to train and evaluate as you have pointed out than supervised learning. I will warn that 200 features is quite a lot, the more features you have the higher the dimensionality and thus the higher the complexity. Unsupervised learning techniques are not well suited for highly complex data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, unsupervised learning assumes that your data is separated into $k$ separate classes. Each with a distinct distribution. The model you will choose will try to estimate the distribution parameters which describes each of the $k$ classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do to measure a &lt;em&gt;fitness&lt;/em&gt; for your model? You can try to find an adequate balance between the similarity of instances within clusters and dissimilarity between instances in different clusters. The distance between the points within a cluster (intra-cluster distance) should be minimized. Whereas the distance between instances in different clusters should be maximized. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-11T09:55:47.013" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30165" PostTypeId="2" ParentId="30160" CreationDate="2018-04-11T10:12:12.653" Score="1" Body="&lt;p&gt;As you know, clustering is Unsupervised learning algorithm. Since you don't know the number of clusters, it becomes hard to find the best possible separation (number of clusters). There is a very good paper published on validating clustering techniques. There are 3 criteria defined in &lt;a href=&quot;http://web.itu.edu.tr/sgunduz/courses/verimaden/paper/validity_survey.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; for validating your clusters. You can take a look at that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the link below gives you some code in R for cluster validation. If you want you can try this too-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.sthda.com/english/articles/29-cluster-validation-essentials/97-cluster-validation-statistics-must-know-methods/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.sthda.com/english/articles/29-cluster-validation-essentials/97-cluster-validation-statistics-must-know-methods/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-04-11T10:12:12.653" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30167" PostTypeId="1" CreationDate="2018-04-11T11:43:24.513" Score="6" ViewCount="98" Body="&lt;p&gt;I want to build a model that can detect which driver is driving now the car based on a dataset that contains 20 driver records for 3600s each driver ( the dataset contains all the car sensors values every second ) .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So , now i have that dataset that contain the drivers records . &#xA;How can i train a model that can identify the driver based on 60 seconds ( or more for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Means , i want to make predictions with a dataframe of rows and not a single row .&#xA;because we can't identify a driver with a single row .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;:) &lt;/p&gt;&#xA;" OwnerUserId="50352" LastEditorUserId="1330" LastEditDate="2020-12-16T14:58:52.960" LastActivityDate="2020-12-16T14:58:52.960" Title="Classify driver based on time-series sensor data" Tags="|machine-learning|classification|time-series|multiclass-classification|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="30169" PostTypeId="2" ParentId="29862" CreationDate="2018-04-11T12:37:34.663" Score="4" Body="&lt;p&gt;Since we are talking about multiple different types of targets (classes versus numerical for example) we already need a composite loss function. I will consider how to balance the different composite parts of the loss function outside of the scope of this answer but if you look into multi-task learning there are solutions to this. What you could do (both in training and evaluation) is to mask the parts of the loss function that are unknown. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can do this fairly easily (but hacky) by passing weights as input during training or prediction for each output in your sample whether it was available or not as 1 or 0 and multiply that component with the weight. For analysis of the training and prediction you need to make sure to take the mean of only the available labels because otherwise your loss values are too optimistic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do think this approach is very valuable in a lot of cases, this can also help with semi-supervised learning where you do have labels available for a similar task.&lt;/p&gt;&#xA;" OwnerUserId="14904" LastActivityDate="2018-04-11T12:37:34.663" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30170" PostTypeId="1" CreationDate="2018-04-11T13:32:31.317" Score="4" ViewCount="381" Body="&lt;p&gt;I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements that I will now report. This discussion will be based on Keras only for practical reasons.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First of all: Keras initially supported various metrics now removed, like for example &lt;strong&gt;precision, recall, f1, fbeta&lt;/strong&gt;. As stated &lt;a href=&quot;https://github.com/keras-team/keras/issues/5794#issuecomment-287641301&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; they were removed only for a clarity reason: they were calculated as local metrics and so they were (almost) meaningless. &lt;/li&gt;&#xA;&lt;li&gt;Those metrics clearly used the labels/classes to be computed. However in that post nobody complained about this problem&lt;/li&gt;&#xA;&lt;li&gt;Now comes the hard part: Keras allows to specify both class and sample weights during the training (with the &lt;code&gt;class_weight&lt;/code&gt; and &lt;code&gt;sample_weight&lt;/code&gt; arguments). These both influence the training loss and the metrics listed in &lt;code&gt;weighted_metrics&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;However, &lt;code&gt;class_weight&lt;/code&gt; does not influence the metrics listed in &lt;code&gt;weighted_metrics&lt;/code&gt; during the validation nor can be used in the &lt;code&gt;evaluate&lt;/code&gt; method, where it is only possibile to specify the &lt;code&gt;sample_weight&lt;/code&gt; argument but not the &lt;code&gt;class_weight&lt;/code&gt; one.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;François Chollet clearly explained this design choice (quote):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;You'll notice that evaluate has a sample_weight argument but no class_weight argument. Validation evaluation in fit follows the same pattern. The reason is that class weighting is meant to help training (by modulating contributions of different classes), but not to affect evaluation. Evaluation, because it reflects production constraints, implies that labels are not available when making predictions, and thus can't be used to modulate evaluation metrics.&lt;/em&gt; &lt;a href=&quot;https://github.com/keras-team/keras/pull/8591&quot; rel=&quot;nofollow noreferrer&quot;&gt;(link)&lt;/a&gt; &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Additionally I have noticed that in the case where validation_split is used, class_weight is applied to the validation data, which is not the expected behavior (classes are of course supposed unknown for the validation data). I am also fixing this.&lt;/em&gt; &lt;a href=&quot;https://github.com/keras-team/keras/issues/496&quot; rel=&quot;nofollow noreferrer&quot;&gt;(link)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;I mean that weighting by classes in a test setup is a form of label information leak, because it cannot be reproduced in a real test situation (when you don't have the labels, such as when using model.predict()). It's acceptable during training, by definition, but should not be done at testing time.&lt;/em&gt; &lt;a href=&quot;https://github.com/keras-team/keras/issues/736&quot; rel=&quot;nofollow noreferrer&quot;&gt;(link)&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I cannot understand why this is the case during the validation phase:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A class leaking like this is not such a serious problem: there is no backprop during the validation phase and so the potential leaking is quite limited. In addition I think this is not more serious than the standard leaking due to the continuous model tuning on the same validation set&lt;/li&gt;&#xA;&lt;li&gt;Without using the classes, during the validation we could not compute interesting quantities such as the precision, the recall, the confusion matrix and so on. Computing them on the training set would produce a biased estimate.&lt;/li&gt;&#xA;&lt;li&gt;Using the classes info would be thus extremely useful in order to obtain good estimates.&lt;/li&gt;&#xA;&lt;li&gt;In addition, as stated before, those metrics were available and nobody complained about them.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I know we could obtain a similar effect by playing with the &lt;code&gt;sample_weight&lt;/code&gt; argument but I am more interested in understanding this problem.&#xA;I know that the classes will not be available (theoretically) during the testing phase but I think that, during the development and tuning, these information and statistics could help better understanding the errors done by the model.&lt;/p&gt;&#xA;" OwnerUserId="50354" LastEditorUserId="50354" LastEditDate="2018-04-11T14:19:15.103" LastActivityDate="2018-05-11T18:54:41.813" Title="Class leaking on validation set" Tags="|neural-network|classification|keras|model-evaluations|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30171" PostTypeId="1" CreationDate="2018-04-11T14:20:40.090" Score="1" ViewCount="22" Body="&lt;p&gt;I am trying to find a vector that would describe the effects of wind on a multirotor. I have a bunch of datalogs from a single frame of multirotor and am of the mind to digg.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that during flight a multirotor has 2 vectors to fight against:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Gravity - to stay in the air and level height&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Wind&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In a world without wind the multirotor would only fight against gravity when hovering and when flying in some direction. Each command the pilot makes is realized as gravity would dictate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we add wind, a certain deflection to multirotor angles occurs. If the wind is constant then the deflection is a function of desired movement and the wind parameters (strength and angle).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My idea is that across many flight logs the wind changes. You get average winds of different angles and different strengths. I would like to average out the wind effects and find a model that is the expected flight pattern when there is no wind. Using that model I would estimate the wind vector and using that knowledge I could do all sorts of clever stuff like choosing optimal flight speed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I have worked on creating a database of data from the logs. Now I need to figure out how to proceed. How do I proceed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of trying to separate the positional (angles, height, speed, current consumption) data from the database and the desired parameters (direction, height, speed). Can this work as a sort of a set of labled data? The angle and current consumption show the effort needed to achieve the desired values. The multirotor algorithms try to be as close to the desired as possible and thus taking the angles and consuming as much current as is optimal in that point in time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would already be happy if I could label the wind strength into a few different levels to rough calculations for speed and other things. Levels like none, low, medium, high, toooo much to fly really.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers are welcome and since I am not well versed with this subject any keywords that I could read up on are much apreciated.&lt;/p&gt;&#xA;" OwnerUserId="49569" LastActivityDate="2018-04-11T14:20:40.090" Title="modelling multirotor aerodynamics using datalogs of flights" Tags="|data-mining|data-stream-mining|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30173" PostTypeId="2" ParentId="27158" CreationDate="2018-04-11T15:13:44.157" Score="0" Body="&lt;p&gt;It would be too lengthy to do the derivation here. Would highly recommend watching &lt;a href=&quot;https://www.youtube.com/watch?v=lekCh_i32iE&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt;, where Hugo Larochelle has derived the conditional independence of hidden units in RBMs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this video answers all of your questions, which seem to spur from conditional independence of RBMs. If not, it would be helpful if you could elaborate your question and provide some more context.&lt;/p&gt;&#xA;" OwnerUserId="47649" LastActivityDate="2018-04-11T15:13:44.157" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30175" PostTypeId="1" CreationDate="2018-04-11T15:34:05.170" Score="3" ViewCount="3158" Body="&lt;p&gt;I want to use the pre-trained VGG16 model of Keras, along with another TensorFlow model. I want to take the output from one of the layers of VGG16 in Keras, put it into the TensorFlow model and train only the latter. Is this possible if I use a TensorFlow backend for Keras?&lt;/p&gt;&#xA;" OwnerUserId="27616" LastActivityDate="2019-01-10T16:05:25.850" Title="Is it possible to integrate Keras and TensorFlow code?" Tags="|keras|tensorflow|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30176" PostTypeId="1" CreationDate="2018-04-11T15:38:16.330" Score="5" ViewCount="2775" Body="&lt;p&gt;I wanted to know what is the correct way to train the SGDClassier model on new data observations? Should I use the &lt;code&gt;fit&lt;/code&gt; function or the &lt;code&gt;partial_fit&lt;/code&gt; function? How are they different?&lt;/p&gt;&#xA;" OwnerUserId="46055" LastEditorUserId="29575" LastEditDate="2019-05-18T22:43:06.080" LastActivityDate="2019-05-19T11:59:49.523" Title="SGDClassifier fit and partial_fit functions" Tags="|machine-learning|scikit-learn|bigdata|algorithms|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30177" PostTypeId="2" ParentId="30160" CreationDate="2018-04-11T15:53:34.027" Score="1" Body="&lt;p&gt;You have to use &lt;strong&gt;unsupervised learning&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, in order to measure accuracy of your model, you should use &lt;a href=&quot;https://en.wikipedia.org/wiki/Cluster_analysis#Evaluation_and_assessment&quot; rel=&quot;nofollow noreferrer&quot;&gt;cluster quality&lt;/a&gt; &lt;strong&gt;intrinsic&lt;/strong&gt; and &lt;strong&gt;extrinsic&lt;/strong&gt; metrics. Compute the similarity of the data in each cluster (intrinsic metric), and the dissimilarity between the data of different clusters (extrinsic metric). A good clustering throws data with great similarity in each cluster and great dissimilarity between clusters.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-11T15:53:34.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30178" PostTypeId="1" AcceptedAnswerId="30188" CreationDate="2018-04-11T16:06:36.053" Score="2" ViewCount="1646" Body="&lt;p&gt;I have a text dataset which I vectorize using a tfidf technique and now in order to make a cluster analysis I am measuring distances between these vector representations. I have found that a common technique is to measure distance using cosine similarity, and when I ask why euclidean distance is not used, the common answer is that cosine similarity works better when vectors have different magnitude.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since my text vectorized representation is &lt;strong&gt;normalized&lt;/strong&gt; I wonder which is the advantage of using cosine similarity over euclidean distance in order to cluster my data?&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-11T19:35:39.073" Title="Euclidean vs. cosine similarity" Tags="|machine-learning|data-mining|clustering|text-mining|bigdata|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30179" PostTypeId="2" ParentId="30170" CreationDate="2018-04-11T16:10:37.200" Score="1" Body="&lt;p&gt;You could calculate these metrics and then attempt to clarify &lt;em&gt;how&lt;/em&gt; your model is failing during validation, but then you're effectively turning your simulated test environment into a second round of training. Any further tuning decisions made from calculating these metrics defeat the purpose of using the validation set as a chance to prove your model's ability to generalize. &lt;/p&gt;&#xA;" OwnerUserId="50363" LastActivityDate="2018-04-11T16:10:37.200" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30180" PostTypeId="1" AcceptedAnswerId="30198" CreationDate="2018-04-11T16:26:30.563" Score="0" ViewCount="219" Body="&lt;p&gt;After tuning hyper-parameters for a gradient boosted model, I have found that the best tree count (iterations) is a few thousand. &#xA;I'm worried that such a high count might impact prediction performance.&#xA;Can someone explain the relation between trees count and prediction time?&lt;/p&gt;&#xA;" OwnerUserId="26882" LastEditorUserId="26562" LastEditDate="2018-04-11T23:38:54.840" LastActivityDate="2018-04-11T23:47:05.823" Title="How does the number of trees effect the prediction time in gradient boost classification trees?" Tags="|predictive-modeling|decision-trees|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30181" PostTypeId="1" AcceptedAnswerId="77741" CreationDate="2018-04-11T16:51:31.033" Score="4" ViewCount="247" Body="&lt;p&gt;As a signal processing engineering and being new to NLP, I am confused with giving input to CNN network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With my knowledge of &lt;a href=&quot;https://github.com/raady07/CNN-for-bearing-fault-diagnosis/blob/master/main_all.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN&lt;/a&gt;, I am trying to build a classifier for ethnicity with inputs as text (last name(LN), middle name(MN), first name(FN)).&#xA;I have a list of 8,000,000 samples with last, middle, first names and class information&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;array = [['person1_LN','person1_MN','person1_FN','Person1_class'],&#xA;         ['person1_LN','person1_MN','person1_FN','Person2_class'], &#xA;         ....]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to apply conv layer (CL) followed by Pooling Layer(PL) on LN,MN,FN respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/dennybritz/cnn-text-classification-tf/blob/master/text_cnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text processing&lt;/a&gt; example demonstrates with the sentences to convert to word embedding. I am trying to understand this code snippet &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;W = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0), name='W')&#xA;self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)&#xA;self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://rare-technologies.com/word2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; says training to be done using word2vec.&#xA;Reading these two blogs I could understand nothing. In Either of the cases, my data is not a sentence, moreover in the second case on what things I have to train?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will the CL operate if I directly give the input of words without word embeddings? &#xA;If not any example on how to embed the words in my case to give the input to CNN?&lt;/p&gt;&#xA;" OwnerUserId="49268" LastEditorUserId="49268" LastEditDate="2018-04-12T13:43:48.157" LastActivityDate="2020-08-14T19:05:40.593" Title="How to do give input to CNN when doing a text processing?" Tags="|machine-learning|python|nlp|cnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30182" PostTypeId="1" CreationDate="2018-04-11T17:02:39.227" Score="6" ViewCount="17750" Body="&lt;p&gt;What is the difference between sequence data and time series data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding is that sequence data is any data where the order matters and time series is a special type of sequence data ordered by the time stamps. Is this correct?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a paper or book that defines it so I can cite it in a research paper?&lt;/p&gt;&#xA;" OwnerUserId="50365" LastActivityDate="2021-08-20T06:33:02.593" Title="Sequence data vs time series data" Tags="|machine-learning|data-mining|time-series|sequence|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30185" PostTypeId="1" CreationDate="2018-04-11T18:50:00.753" Score="-3" ViewCount="380" Body="&lt;p&gt;I need help in reading the following data file into Python and separate the data into variables keeping in mind each variable is one column and the radiance (which is the last variable) has many columns. After separating them graph a 2D graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like it much if you'd show me how to put these variables in columns and radians in many columns.You can pick any arbitrary name for many columns.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;1. 165.08  75.16 200.07 18.08  0.36 561556726 1 -9.034E-08  1.121E-06  9.695E-07  7.981E-07  1.575E-05  2.284E-06  1.912E-06  1.998E-06  2.053E-06  2.116E-06  2.149E-06  2.197E-06  2.209E-06  2.163E-06  2.149E-06  2.218E-06  2.166E-06  2.190E-06  2.185E-06  2.105E-06  2.110E-06  2.086E-06  2.053E-06  1.996E-06  2.020E-06  1.992E-06  1.905E-06  1.897E-06  1.784E-06  1.697E-06  1.718E-06  1.602E-06  1.579E-06  1.536E-06  1.480E-06  1.415E-06  1.345E-06  1.341E-06  1.274E-06  1.249E-06  1.204E-06  1.170E-06  1.139E-06  1.119E-06  1.032E-06  9.872E-07  8.550E-07  6.789E-07  6.203E-07  6.594E-07  5.504E-07  5.858E-07  6.771E-07  7.255E-07  6.678E-07  6.082E-07  5.895E-07  5.551E-07  5.206E-07  5.001E-07  4.303E-07  4.685E-07  3.865E-07  4.051E-07  3.697E-07  3.725E-07  3.390E-07  3.316E-07  2.626E-07  2.794E-07  2.775E-07  2.738E-07  2.654E-07  2.272E-07  2.030E-07  2.040E-07  2.272E-07  1.816E-07  1.565E-07  1.714E-07  1.267E-07  1.201E-07  1.807E-07  1.220E-07  1.108E-07  1.583E-07  1.248E-07  1.220E-07  8.661E-08  1.052E-07  1.099E-07  9.499E-08  1.341E-07  8.568E-08  5.588E-08  5.867E-08  6.333E-08  7.357E-08  7.078E-08  2.328E-08  5.495E-08  6.147E-08  6.519E-08  2.701E-08  3.353E-08  9.499E-08  3.818E-08  5.681E-08  8.196E-08  3.539E-08  5.309E-08  1.024E-08  5.029E-08  8.009E-08  4.657E-08 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50315" LastEditorUserId="50315" LastEditDate="2018-04-14T17:08:48.897" LastActivityDate="2018-04-15T00:53:18.040" Title="How to plot 2D or 3D graph using Python?" Tags="|python|pandas|" AnswerCount="1" CommentCount="3" ClosedDate="2018-04-12T21:03:30.863" ContentLicense="CC BY-SA 3.0" />
  <row Id="30186" PostTypeId="1" AcceptedAnswerId="30193" CreationDate="2018-04-11T18:50:16.043" Score="1" ViewCount="4147" Body="&lt;p&gt;I’m trying to understand, and eventually build a Restricted Boltzmann Machine. I understand that the update rule - that is the algorithm used to change the weights - is something called “contrastive divergence”. I looked &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Restricted_Boltzmann_machine&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; up on Wikipedia and found these steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Take a training sample v, compute the probabilities of the hidden units and sample a hidden activation vector h from this probability distribution.&lt;/li&gt;&#xA;&lt;li&gt;Compute the outer product of v and h and call this the positive gradient.&lt;/li&gt;&#xA;&lt;li&gt;From h, sample a reconstruction v' of the visible units, then resample the hidden activations h' from this. (Gibbs sampling step)&lt;/li&gt;&#xA;&lt;li&gt;Compute the outer product of v' and h' and call this the negative gradient.&lt;/li&gt;&#xA;&lt;li&gt;...&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I don’t understand step 3 and I’m struggling to grasp the concept of Gibbs sampling. Would someone explain this simply to me? I have covered neural networks if that helps you.&lt;/p&gt;&#xA;" OwnerUserId="37363" LastActivityDate="2019-01-25T01:40:05.127" Title="Understanding Contrastive Divergence" Tags="|machine-learning|python|neural-network|unsupervised-learning|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30187" PostTypeId="2" ParentId="694" CreationDate="2018-04-11T19:28:20.627" Score="3" Body="&lt;p&gt;I recommend you to use &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt; which is under strong development and supports deep learning. You can use the high-level neural networks API &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; that runs on top of tensorflow and is very simply to use, just try a &lt;a href=&quot;https://elitedatascience.com/keras-tutorial-deep-learning-in-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt; and you are going to love it.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-11T19:28:20.627" CommentCount="0" CommunityOwnedDate="2018-04-11T19:28:20.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="30188" PostTypeId="2" ParentId="30178" CreationDate="2018-04-11T19:35:39.073" Score="4" Body="&lt;p&gt;On L2 normalized data it is an easy and good exercise to prove that they are equivalent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you should try to solve the math yourself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hint: use squared Euclidean.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that it is common with tfidf to &lt;em&gt;not&lt;/em&gt; have normalized data because of various technical reasons, e.g., when using inverted indexes in text search. Furthermore, cosine is faster on very sparse data.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-11T19:35:39.073" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30189" PostTypeId="1" CreationDate="2018-04-11T19:58:50.233" Score="1" ViewCount="50" Body="&lt;p&gt;I would like to cluster some user reviews and I'm doing this with k-means. In my dataset I have English and German reviews. Is this manipulating the cluster result if I don't seperate them? Or should I do a k-means prediction on each language?&lt;/p&gt;&#xA;" OwnerUserId="50077" LastEditorUserId="26562" LastEditDate="2018-04-11T23:35:06.727" LastActivityDate="2020-02-15T05:01:00.200" Title="Should I seperate the data for a k-means model?" Tags="|machine-learning|k-means|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30190" PostTypeId="2" ParentId="30189" CreationDate="2018-04-11T20:06:32.997" Score="0" Body="&lt;p&gt;Using text documents in different languages you are going to have different vector representations, unless you translate the documents previously. For example, &lt;em&gt;house&lt;/em&gt; and &lt;em&gt;maison&lt;/em&gt; are going to be related to different features. So a cluster algorithm is not going to recognize them as synonymous.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should try a previous translation of your reviews. The quality of that translation is going to affect the clustering algorithm depending on the algorithms you are using. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you tell me the steps you are performing in your cluster I could help you better.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-11T20:06:32.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30191" PostTypeId="1" AcceptedAnswerId="30218" CreationDate="2018-04-11T20:23:48.573" Score="0" ViewCount="625" Body="&lt;p&gt;I will explain with an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have 2 factories that produce pulp paper. Each have similar processes where the laws of physics give the same outcome. Now let's say this 2 factories have equipment and sensors from different manufacturers, so the output of those sensors is not comparable in any way (different number of variables, different metric system etc.). Although for both factories I can caculate the output easily and determine the learning target in a comparable way (eg. metric tonnes of paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way of using deep learning to learn from both datasets at the same time? I mean increase the predictive power upon a sample from factory 1 due to insights on factory 2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about having 3 DNN, 2 for reducing feature representation and standardizing output representation and the third one for learning the general pattern common to both and predicting the final output?&lt;/p&gt;&#xA;" OwnerUserId="26448" LastActivityDate="2018-04-12T12:00:06.557" Title="How to learn from multiple data sources with different input variables but the same underlying pattern?" Tags="|neural-network|deep-learning|representation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30193" PostTypeId="2" ParentId="30186" CreationDate="2018-04-11T20:50:54.157" Score="3" Body="&lt;p&gt;Gibbs sampling is an example for the more general &lt;strong&gt;Markov chain Monte Carlo&lt;/strong&gt; methods to sample from distribution in a high-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To explain this, I will first have to introduce the term &lt;strong&gt;state space&lt;/strong&gt;. Recall that a Boltzmann machine is built out of binary units, i.e. every unit can be in one of two states - say 0 and 1. The overall state of the network is then specified by the state for every unit, i.e. the states of the network can be described as points in the space &lt;span class=&quot;math-container&quot;&gt;$\{0,1\}^N$&lt;/span&gt;, where N is the number of units in the network. This point is called the &lt;strong&gt;state space&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, on that state space, we can define a &lt;strong&gt;probability distribution&lt;/strong&gt;. The details are not so important, but what you essentially do is that you define energy for every state and turn that into a probability distribution using a &lt;strong&gt;Boltzmann distribution&lt;/strong&gt;. Thus there will be states that are likely and other states that are less likely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A Gibbs sampler is now a procedure to produce a sample, i.e. a sequence &lt;span class=&quot;math-container&quot;&gt;$X_n$&lt;/span&gt; of states such that, roughly speaking, the distribution of these states across the state space reflects the probability distribution. Thus you want most of the &lt;span class=&quot;math-container&quot;&gt;$X_n$&lt;/span&gt; to be in regions of the state space with high probability (and low energy), and few of them to be in regions with low probability (and high energy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do this, a naive Gibbs sampling approach would proceed as follows. You start with some state &lt;span class=&quot;math-container&quot;&gt;$X_0$&lt;/span&gt;. To find the state &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt;, you would pick some unit and calculate the conditional probability for that unit to be in state 1 (&quot;on&quot;) &lt;strong&gt;conditional on the current value of all other units&lt;/strong&gt;. Call this number p. You would then set the unit to 1 with probability p and pick the next unit to repeat this to get from &lt;span class=&quot;math-container&quot;&gt;$X_1$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$X_2$&lt;/span&gt; and so forth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the special case of a restricted Boltzmann machine, this can be greatly simplified. Instead of going through, say, first all hidden units and then all visible units and update them like this one by one, you can, in fact, update all hidden units in one step and all visible units in one step, because any two hidden units and any two visible units are independent. Thus, for a full cycle through all units of the state space, you would:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;calculate the probability for all hidden units to be 1, given the value of the visible units&lt;/li&gt;&#xA;&lt;li&gt;set the hidden units to 1 with this probability&lt;/li&gt;&#xA;&lt;li&gt;calculate the probability for the visible units to be 1, again conditional on the value of the hidden units, and&lt;/li&gt;&#xA;&lt;li&gt;set the visible units to 1 with this probability&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This constitutes one full Gibbs sampling step and is your step 1 + the first part of 3 (the second part is then needed for the further calculation and not part of the sampling). The reason why we do this in the CD algorithm is that we actually want to approximate an expectation value and use a sampler for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a complex topic and hard to summarize in a few sentences. If you want to learn more about the mathematics behind this (Markov chains) and on the application to RBMs (contrastive divergence and persistent contrastive divergence), you might find &lt;a href=&quot;https://github.com/christianb93/MachineLearning/blob/master/doc/MarkovChains/MarkovChainsIntroduction.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;https://github.com/christianb93/MachineLearning/blob/master/doc/BoltzmannMachine/BoltzmannMachinesAndHopfieldNetworks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; document helpful - these are some notes that I put together while learning about this.&lt;/p&gt;&#xA;" OwnerUserId="49886" LastEditorUserId="-1" LastEditDate="2019-01-25T01:40:05.127" LastActivityDate="2019-01-25T01:40:05.127" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30195" PostTypeId="2" ParentId="30182" CreationDate="2018-04-11T22:44:06.177" Score="5" Body="&lt;p&gt;Sequential Data is any kind of data where the order matters as you said. So we can assume that time series is a kind of sequential data, because the order matters. &#xA;A time series is a sequence taken at successive equally spaced points in time and it is not the only case of sequential data. In the latter the order is defined by the dimension of time. There are other cases of sequential data as data from text documents, where you can take into account the order of the terms or biological data (DNA sequence etc.). &#xA;The fact that you have sequential data is important for two reasons. First, you can into account for the representation of the data and also you can take it into account for the data modeling (e.g. Conditional Random Fields, Hidden Markov Models for text or genes and ARIMA Models for time series problems). &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-04-11T22:44:06.177" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30196" PostTypeId="2" ParentId="22302" CreationDate="2018-04-11T23:21:00.240" Score="4" Body="&lt;p&gt;&lt;strong&gt;Fine-tuning&lt;/strong&gt; is the process in which the parameters of a trained model must be adjusted very precisely while we are trying to validate that model taking into account a small data set that does not belong to the train set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That small validation data set comes from the same distribution as the data set used for the training of the model. The split of the available data to train and validation set is random. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Transfer Learning&lt;/strong&gt; or Domain Adaptation is related to the difference in the distribution of the train and test set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is something broader than Fine tuning, which means that we know a priori that the train and test come from different distribution and we are trying to tackle this problem with several techniques depending on the kind of difference, instead of just trying to adjust some parameters (usually we are doing this for reasons as preventing overfitting etc.)  &lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="61023" LastEditDate="2019-01-14T16:43:06.373" LastActivityDate="2019-01-14T16:43:06.373" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30198" PostTypeId="2" ParentId="30180" CreationDate="2018-04-11T23:47:05.823" Score="3" Body="&lt;p&gt;The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take about twice as long as 500 trees, and about half as long as 2000 trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'll need to test it yourself and check if it's fast enough for your use case. Modern libraries like &lt;code&gt;xgboost&lt;/code&gt; can handle high numbers of trees with remarkable efficiency. One thing to keep in mind is that it's generally faster to get predictions for a whole block of test examples at once than it is to get the predictions one at a time.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-11T23:47:05.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30199" PostTypeId="2" ParentId="26531" CreationDate="2018-04-12T00:39:54.417" Score="1" Body="&lt;p&gt;Based on your description, it looks like different models have different biases. Two important questions: do you have any data imbalance problem? What kind of models you are using? Using stacking based classifier is beneficial if you have different biases. Try to use a simple stack based classifier.&#xA;For your level-1 classifier, use different models (e.g. SVM-L, SVM-NL, DT, RF, ... etc). For your meta-data, use probabilities and for the meta-classifier use Random Forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a data imbalance problem using stack based classifier is a little bit more challenging.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="82019" LastEditDate="2019-09-11T21:26:49.430" LastActivityDate="2019-09-11T21:26:49.430" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30200" PostTypeId="2" ParentId="28792" CreationDate="2018-04-12T00:55:48.840" Score="4" Body="&lt;p&gt;LSTM inputs should be a tensor of size (#samples, time_steps, #features). Considering your dataset; you will have #samples = 2000-1 (for the training); time_steps = 1 (this is the lag-time you choose), and #features = 14. Then for the testing/validation #samples = 1000-1. But here, I suggest you take due consideration to the autocorrelation of the dataset; it seems that you choose the lag-time = 1 (just for the sake of creating the response variables); however, check out the temporal correlation + temporal structure of the data. Similar, to other Deep Neural networks, LSTM requires large dataset to train and test; checkout if you can increase the lag-time and get more predictor data. Have a look at &lt;a href=&quot;https://datascience.stackexchange.com/questions/27563/multi-dimentional-and-multivariate-time-series-forecast-rnn-lstm-keras&quot;&gt;Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras&lt;/a&gt; this similar question for more information. &lt;/p&gt;&#xA;" OwnerUserId="50383" LastActivityDate="2018-04-12T00:55:48.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30201" PostTypeId="2" ParentId="30185" CreationDate="2018-04-12T01:30:39.607" Score="2" Body="&lt;p&gt;First let's store your data in a variable:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = '165.08 75.16 200.07 18.08 0.36 561556726 1 -9.034E-08 1.121E-06 9.695E-07 7.981E-07 1.575E-05 2.284E-06 1.912E-06 1.998E-06 2.053E-06 2.116E-06 2.149E-06 2.197E-06 2.209E-06 2.163E-06 2.149E-06 2.218E-06 2.166E-06 2.190E-06 2.185E-06 2.105E-06 2.110E-06 2.086E-06 2.053E-06 1.996E-06 2.020E-06 1.992E-06 1.905E-06 1.897E-06 1.784E-06 1.697E-06 1.718E-06 1.602E-06 1.579E-06 1.536E-06 1.480E-06 1.415E-06 1.345E-06 1.341E-06 1.274E-06 1.249E-06 1.204E-06 1.170E-06 1.139E-06 1.119E-06 1.032E-06 9.872E-07 8.550E-07 6.789E-07 6.203E-07 6.594E-07 5.504E-07 5.858E-07 6.771E-07 7.255E-07 6.678E-07 6.082E-07 5.895E-07 5.551E-07 5.206E-07 5.001E-07 4.303E-07 4.685E-07 3.865E-07 4.051E-07 3.697E-07 3.725E-07 3.390E-07 3.316E-07 2.626E-07 2.794E-07 2.775E-07 2.738E-07 2.654E-07 2.272E-07 2.030E-07 2.040E-07 2.272E-07 1.816E-07 1.565E-07 1.714E-07 1.267E-07 1.201E-07 1.807E-07 1.220E-07 1.108E-07 1.583E-07 1.248E-07 1.220E-07 8.661E-08 1.052E-07 1.099E-07 9.499E-08 1.341E-07 8.568E-08 5.588E-08 5.867E-08 6.333E-08 7.357E-08 7.078E-08 2.328E-08 5.495E-08 6.147E-08 6.519E-08 2.701E-08 3.353E-08 9.499E-08 3.818E-08 5.681E-08 8.196E-08 3.539E-08 5.309E-08 1.024E-08 5.029E-08 8.009E-08 4.657E-08 '&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we'll split this string and convert each number to float.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [float(x) for x in a.split()]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Finally, we'll plot these points.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;plt.plot(a[7:])  # I don't plot the first 7 numbers as they are &#xA;                 # really high and would dominate the gragh&#xA;                 # feel free to remove the square brackets&#xA;                 # if you want all the points to be plotted&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now if you want to view the plot add &lt;code&gt;plt.show()&lt;/code&gt; and if you want to save it just type &lt;code&gt;plt.savefig('aa.png', bbox_inches='tight')&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With these I get the following image:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kPItY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kPItY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerDisplayName="user50384" LastEditorDisplayName="user50384" LastEditDate="2018-04-15T00:53:18.040" LastActivityDate="2018-04-15T00:53:18.040" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30203" PostTypeId="2" ParentId="30189" CreationDate="2018-04-12T03:03:22.483" Score="1" Body="&lt;p&gt;Yes you should &lt;strong&gt;definitely&lt;/strong&gt; split these two up. When calculating the TF-TDF matix, it will give different terms to objects of the same entity, because of the difference in language. This &lt;strong&gt;will&lt;/strong&gt; affect your clustering results.&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-12T03:03:22.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30205" PostTypeId="1" AcceptedAnswerId="30337" CreationDate="2018-04-12T04:38:12.667" Score="2" ViewCount="1185" Body="&lt;p&gt;While evaluating the Linear regression model, which metric do we need to consider R square or Mean Squared Error (or which needs to be given more importance)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-15T09:30:38.810" Title="Linear Regression - Metrics to consider?" Tags="|regression|linear-regression|metric|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30206" PostTypeId="1" AcceptedAnswerId="30353" CreationDate="2018-04-12T05:12:40.107" Score="1" ViewCount="2520" Body="&lt;p&gt;I'm trying to manipulate some data in Biolabs Orange, using the built in Python Script widget and information at &lt;a href=&quot;https://docs.orange.biolab.si/3/data-mining-library/tutorial/data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Biolabs Orange tutorial on scripting&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;However, I'm struggling with taking the results of SMOTE and putting them into a format for Orange:&lt;/p&gt;&#xA;&lt;p&gt;This is the code I'm using in the Python Script widget:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Get libraries&#xA;import Orange&#xA;import numpy as np&#xA;from Orange.data import Domain, Table&#xA;from imblearn.over_sampling import SMOTE&#xA;&#xA;#in_data = Orange.data.Table('WORKING_temp.csv')&#xA;&#xA;df = in_data.copy()&#xA;&#xA;# set variables for SMOTE&#xA;sm = SMOTE(random_state=42)&#xA;&#xA;# get table of data (X) and class variables (y)&#xA;X, y = df.X, df.Y&#xA;&#xA;# resample data and classes&#xA;X_res, y_res = sm.fit_sample(X, y)&#xA;&#xA;&#xA;df.X = X_res&#xA;df.Y = y_res&#xA;&#xA;temp = Orange.data.Table(df.X, df.Y)&#xA;temp.domain = df.domain&#xA;&#xA;out_data = Orange.data.Table(temp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The result is a ValueError, which I think is related to changing the length of the class variables and data table, while leaving the original index length?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;quot;ValueError: could not broadcast input array from shape (3724,10) into shape (3724)&amp;quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26748" LastEditorUserId="29169" LastEditDate="2020-12-13T17:18:05.257" LastActivityDate="2022-10-04T15:13:03.473" Title="Scripting code for class imbalance in Biolabs Orange" Tags="|python|orange|class-imbalance|smote|anaconda|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30208" PostTypeId="1" CreationDate="2018-04-12T05:55:42.307" Score="0" ViewCount="7688" Body="&lt;p&gt;I am using a large dataset with 4 different multilabel classes. I was trying to apply Random forest algorithm on those data set. After preparing data set I separate X (feature columns) and Y(feature class), split those data into train and testing data, fit training data to model, predict test data and was trying to find the accuracy by using testing data then I faced valueError: 'multiclass-multioutput is not supported'. My data set is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/m52dH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/m52dH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is my code after pre-processing: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;#define X and y&#xA;feature_cols=['MFCCs_ 1','MFCCs_ 2','MFCCs_ 3','MFCCs_ 4','MFCCs_ 5','MFCCs_ 6','MFCCs_ 7','MFCCs_ 8','MFCCs_ 9', 'MFCCs_10', 'MFCCs_11', 'MFCCs_12', 'MFCCs_13','MFCCs_14','MFCCs_15','MFCCs_16','MFCCs_17','MFCCs_18','MFCCs_19','MFCCs_20','MFCCs_21','MFCCs_22']&#xA;feature_class=['RecordID','Family','Genus','Species']&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# X is a matrix, hence we use [] to access the features we want in feature_cols&#xA;X = train[feature_cols]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;# y is a vector, hence we use dot to access 'label'&#xA;y = train[feature_class]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model= RandomForestClassifier(n_estimators=100)&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=42)&#xA;model.fit(X_train, y_train)&#xA;preds = model.predict(X_test)&#xA;print(accuracy_score(y_test, preds,normalize=False))&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and then I faced these value error: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/QodXp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QodXp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I solve these problem? &lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2019-06-30T22:01:25.540" Title="Problem to classify multilabel dataset while using random forest algorithm" Tags="|machine-learning|python|random-forest|multilabel-classification|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30210" PostTypeId="1" CreationDate="2018-04-12T06:22:25.523" Score="0" ViewCount="175" Body="&lt;p&gt;like this, like shown in &lt;a href=&quot;https://www.youtube.com/watch?v=z2DHpW79w0Y&amp;amp;t=7s&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="50394" LastEditorUserId="26019" LastEditDate="2020-08-20T17:10:11.240" LastActivityDate="2020-08-20T17:11:10.587" Title="how to make moving moving graphs that vary in time like a video?" Tags="|dataset|visualization|data|" AnswerCount="2" CommentCount="0" ClosedDate="2020-08-20T21:09:40.177" ContentLicense="CC BY-SA 4.0" />
  <row Id="30211" PostTypeId="1" CreationDate="2018-04-12T06:43:21.537" Score="2" ViewCount="1306" Body="&lt;p&gt;I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model :&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Exception: fastText: Cannot load lid.176.bin due to C++ extension failed to allocate the memory&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any ideas? I've already installed all prerequisites and still get this error.&lt;/p&gt;&#xA;" OwnerUserId="44044" LastEditorUserId="29575" LastEditDate="2018-07-21T13:10:37.340" LastActivityDate="2018-07-21T13:10:37.340" Title="Fasttext exception error" Tags="|machine-learning|language-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30212" PostTypeId="2" ParentId="30210" CreationDate="2018-04-12T07:59:22.523" Score="3" Body="&lt;p&gt;You can find a list of great library &lt;a href=&quot;https://github.com/d3/d3/wiki/Gallery&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;I would recommend &lt;a href=&quot;https://piratepeel.github.io/sportskills.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt; but you have to write some css to have that same effect.&lt;/p&gt;&#xA;" OwnerUserId="46527" LastEditorUserId="26019" LastEditDate="2020-08-20T17:11:10.587" LastActivityDate="2020-08-20T17:11:10.587" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30214" PostTypeId="1" CreationDate="2018-04-12T09:09:26.407" Score="2" ViewCount="1015" Body="&lt;p&gt;I am using the negative sampling approach used in Word2Vec to train some image embeddings. From what I have read, for every positive example, we are creating a number of negative examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question:&lt;/strong&gt; Why do we use an imbalanced dataset here? Presumably we will get the normal issue where the algorithm ends up predicting the negative label to minimise the cost function? I understand that the aim isn't really to use it as a prediction model, but rather to extract the embeddings, but what is the benefit of having an imbalanced class here?&lt;/p&gt;&#xA;" OwnerUserId="50403" LastActivityDate="2018-04-24T13:30:38.353" Title="Image Embeddings - Negative Sampling and Imbalanced Class Issues" Tags="|machine-learning|python|neural-network|deep-learning|word2vec|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30215" PostTypeId="1" AcceptedAnswerId="30217" CreationDate="2018-04-12T09:42:32.367" Score="15" ViewCount="21677" Body="&lt;p&gt;I am currently doing a course in tensorflow in which they used tf.one_hot(indices, depth). Now I don't understand how these indices change into that binary sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody please explain to me the exact process???&lt;/p&gt;&#xA;" OwnerUserId="50406" LastEditorUserId="50406" LastEditDate="2019-04-07T05:57:57.850" LastActivityDate="2019-04-07T05:57:57.850" Title="What is one hot encoding in tensorflow?" Tags="|machine-learning|python|neural-network|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30216" PostTypeId="1" CreationDate="2018-04-12T09:59:42.433" Score="1" ViewCount="523" Body="&lt;p&gt;I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Title:&#xA;literacy rates Africa&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Description:&#xA;What are literacy rates in African countries&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.&lt;/p&gt;&#xA;&lt;p&gt;Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of &lt;em&gt;n-grams&lt;/em&gt;. In fact there is a similar example in &lt;a href=&quot;https://cntk.ai/pythondocs/CNTK_303_Deep_Structured_Semantic_Modeling_with_LSTM_Networks.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNKT&lt;/a&gt; where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.&lt;/p&gt;&#xA;&lt;p&gt;So far I have found a way to do the &lt;em&gt;n-grams&lt;/em&gt; but this is not what I want as for example in the sentence:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;the cow jumps over the moon&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I get the following code:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;the_cow_jumps&#xA;cow_jumps_over&#xA;jumps_over_the&#xA;over_the_moon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;When I would be interested in the &lt;em&gt;4-gram:&lt;/em&gt;&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;cow_jumps_over_moon&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one &lt;em&gt;n-gram&lt;/em&gt; which is not what I want as my main objective is get the final vocabulary to train my model.&lt;/p&gt;&#xA;&lt;p&gt;As an example of what I want could be the words:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;book &#xA;book_character &#xA;book_editions_published &#xA;book_subject &#xA;books_published &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Instead of:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;book&#xA;character&#xA;editions&#xA;published&#xA;subject&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50405" LastEditorUserId="98307" LastEditDate="2020-08-11T12:06:59.907" LastActivityDate="2020-08-11T12:06:59.907" Title="Extract relevant vocabulary from a document" Tags="|python|nlp|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30217" PostTypeId="2" ParentId="30215" CreationDate="2018-04-12T10:05:46.050" Score="21" Body="&lt;p&gt;Suppose you have a categorical feature in your dataset (e.g. color). And your samples can be either red, yellow or blue. In order to pass this argument to a ML algorithm, you first need to encode it so that instead of strings you have numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to do such a thing is to create a mapping where: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;red --&gt; 1&lt;br&gt;&#xA;yellow --&gt; 2&lt;br&gt;&#xA;blue --&gt; 3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and replace each string with its mapped value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However this might create &lt;strong&gt;unwanted side effects&lt;/strong&gt; in our ML model as when dealing with numbers it might think that blue &gt; yellow (because 3 &gt; 2) or that red + yellow = blue (because 1 + 2 = 3). The model has no way of knowing that these data were categorical and then were mapped as integers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The solution to this problem is &lt;strong&gt;one-hot encoding&lt;/strong&gt; where we create &lt;em&gt;N&lt;/em&gt; &lt;strong&gt;new features&lt;/strong&gt;, where &lt;em&gt;N&lt;/em&gt; is the number of unique values in the original feature. In our exampel &lt;em&gt;N&lt;/em&gt; would be equal to 3, because we have 3 unique colors (red, yellow and blue).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each of these features be binary and would correspond to &lt;strong&gt;one&lt;/strong&gt; of these unique values. In our example the first feature would be a binary feature telling us if that sample is red or not, the second would be the same thing for yellow and the third for blue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example of such a transformation is illustrated below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/CHKvT.png&quot; alt=&quot;&quot;&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note, that because this approach increases the dimensionality of the dataset, if we have a feature that takes many unique values, we may want to use a more sparse encoding (like the one I presented above).  &lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-04-12T10:05:46.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30218" PostTypeId="2" ParentId="30191" CreationDate="2018-04-12T12:00:06.557" Score="1" Body="&lt;p&gt;What you are referring to is multi-view learning. Multi-view learning basically tells us how multiple data sources or multiple feature subsets can be combined to create a more robust learning curve for the algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In recent years, starting from 2013 a lot of research has been carried in this rapidly growing field. A good introduction to the topic can be found in the link below. It contains a more theoretical and mathematical approach to understanding the method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://research.ics.aalto.fi/airc/reports/R1011/msml.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://research.ics.aalto.fi/airc/reports/R1011/msml.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32958" LastActivityDate="2018-04-12T12:00:06.557" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30219" PostTypeId="2" ParentId="30215" CreationDate="2018-04-12T12:05:20.037" Score="3" Body="&lt;p&gt;depth: A scalar defining the depth of the one hot dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;indices: A Tensor of indices.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This the example given in tensorflow documentation.&lt;br&gt;&#xA;1. Only Specifying indices and depth(Default Values of on_value is 1 and off_value is 0)&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; indices = [0, 1, 2]&#xA;    depth = 3&#xA;    tf.one_hot(indices, depth)  # output: [3 x 3]&#xA;    # [[1., 0., 0.],&#xA;    #  [0., 1., 0.],&#xA;    #  [0., 0., 1.]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Specifying on_value and off_value&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;pre&gt;&#xA;indices = [0, 2, -1, 1]&#xA;depth = 3&#xA;tf.one_hot(indices, depth, on_value=5.0, off_value=0.0, axis=-1)&#xA;##output: [4 x 3]&#xA;    # [[5.0, 0.0, 0.0],  # one_hot(0)&#xA;    #  [0.0, 0.0, 5.0],  # one_hot(2)&#xA;    #  [0.0, 0.0, 0.0],  # one_hot(-1)&#xA;    #  [0.0, 5.0, 0.0]]  # one_hot(1)&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also see the &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/r1.7/tensorflow/python/ops/array_ops.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;code on GitHub&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="44401" LastActivityDate="2018-04-12T12:05:20.037" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30220" PostTypeId="2" ParentId="30216" CreationDate="2018-04-12T12:57:16.633" Score="1" Body="&lt;p&gt;Not sure if this is what you are looking for. Couldn't understand your question properly do you want 4-grams then this is the way you could do it. Assuming you have cleaned your text file by eliminating articles.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from nltk import ngrams&#xA;file=open('abc.txt','r')&#xA;txt = file.read()&#xA;n = 4&#xA;fourgrams = ngrams(txt.split(), n)&#xA;for grams in fourgrams:&#xA;  print (grams)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In case your file abc.txt contains &quot;This is random text to demonstrate the use of n-grams&quot;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;OUTPUT:&#xA;('This', 'is', 'random', 'text')&#xA;('is', 'random', 'text', 'to')&#xA;('random', 'text', 'to', 'demonstrate')&#xA;('text', 'to', 'demonstrate', 'the')&#xA;('to', 'demonstrate', 'the', 'use')&#xA;('demonstrate', 'the', 'use', 'of')&#xA;('the', 'use', 'of', 'n-grams')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you do not want repetition then &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;words = 'This is random text we’re going to split apart'&#xA;x=[]&#xA;for word in words.split():&#xA;    x.append(word)&#xA;    if len(x) == 4:&#xA;        print(x)&#xA;        x=[]&#xA;print(x)&#xA;&#xA;OUTPUT:&#xA;['This', 'is', 'random', 'text']&#xA;['we’re', 'going', 'to', 'split']&#xA;['apart']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44401" LastActivityDate="2018-04-12T12:57:16.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30222" PostTypeId="1" CreationDate="2018-04-12T14:00:40.450" Score="1" ViewCount="49" Body="&lt;p&gt;I have been reading &quot;drop&quot; is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this method useful in minibatch at all? &lt;/p&gt;&#xA;" OwnerUserId="50419" LastActivityDate="2018-04-12T15:47:49.720" Title="Point of dropping weights in mini batch for purpose of regularization" Tags="|neural-network|regularization|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30223" PostTypeId="2" ParentId="30210" CreationDate="2018-04-12T14:43:17.607" Score="0" Body="&lt;p&gt;If you happen to work with R, you can use the &lt;a href=&quot;https://github.com/dgrtwo/gganimate&quot; rel=&quot;nofollow noreferrer&quot;&gt;gganimate&lt;/a&gt; package which works with &lt;a href=&quot;http://ggplot2.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ggplot2&lt;/a&gt;. It allows you to to save the animation in gif, mp4 or swf afterwards.&lt;/p&gt;&#xA;" OwnerUserId="50423" LastActivityDate="2018-04-12T14:43:17.607" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30226" PostTypeId="2" ParentId="30222" CreationDate="2018-04-12T15:47:49.720" Score="1" Body="&lt;p&gt;Yes dropout is useful this case too. The purpose of dropout is to make the model generalize better by causing it to use &lt;strong&gt;all available&lt;/strong&gt; weights from a given layer. It is supposed to make the subsequent layer &lt;strong&gt;not rely on a small number of activations&lt;/strong&gt; from the previous layer, but to use them all. &lt;/p&gt;&#xA;" OwnerDisplayName="user50384" LastActivityDate="2018-04-12T15:47:49.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30227" PostTypeId="1" AcceptedAnswerId="30730" CreationDate="2018-04-12T16:01:33.937" Score="2" ViewCount="3075" Body="&lt;p&gt;I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer which is a fully connected layer. However, I observed that without dropout I get 97.75% accuracy on the test data and with dropout of 0.5 I get 95.36%. I want to ask whether I am doing something wrong or what is the reason for this phenomena? &#xA;I change it into eval mode in test but I reach to 96.44% accuracy. Still it is less than without dropout.&#xA;Thanks a lot&lt;/p&gt;&#xA;&#xA;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# RNN Model (Many-to-One)&#xA;class RNN(nn.Module):&#xA;   def __init__(self, input_size, hidden_size, num_layers, num_classes):&#xA;    super(RNN, self).__init__()&#xA;    self.hidden_size = hidden_size&#xA;    self.num_layers = num_layers&#xA;    self.lstm = nn.LSTM(input_size, hidden_size, num_layers,&#xA;                        batch_first=True,bidirectional=True)&#xA;&#xA;    self.fc = nn.Sequential(&#xA;        nn.Dropout(0.1),&#xA;       nn.Linear(hidden_size*2, num_classes),&#xA;        nn.Softmax(dim=1)&#xA;    )&#xA;def init_hidden(self,x):&#xA;&#xA;     return(Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)).cuda(), &#xA;     Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).cuda()))&#xA;&#xA;def forward(self, x):&#xA;    # Set initial states &#xA;&#xA;&#xA;    # Forward propagate RNN&#xA;&#xA;    hidden = self.init_hidden(x)&#xA;    #print(len(hidden))&#xA;    out, _ = self.lstm(x, hidden)  &#xA;&#xA;    # Decode hidden state of last time step&#xA;    out = self.fc(out[:, -1, :])  &#xA;    return out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50428" LastActivityDate="2018-04-24T05:09:21.773" Title="Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch" Tags="|lstm|pytorch|dropout|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30228" PostTypeId="2" ParentId="28987" CreationDate="2018-04-12T16:10:10.783" Score="2" Body="&lt;p&gt;After having performed &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X=np.array([[1,0,0,0,0],&#xA;           [0,1,0,0,0],&#xA;           [0,0,1,0,0],&#xA;           [1,0,0,1,0],&#xA;           [1,0,0,0,1]])&#xA;&#xA;y=np.array([0,1,1,1,0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you have to do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y = y.reshape(1, -1)&#xA;&#xA;model=svm.SVC()&#xA;model.fit(X,y)&#xA;test = np.array([1,0,1,0,0])&#xA;test = test.reshape(1,-1)&#xA;print(model.predict(test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In future you have to scale your dataset. You can use either Standard Scaler (suggested) or MinMax Scaler. &lt;/p&gt;&#xA;" OwnerUserId="50429" LastEditorUserId="50429" LastEditDate="2018-04-12T16:28:15.713" LastActivityDate="2018-04-12T16:28:15.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30229" PostTypeId="2" ParentId="30080" CreationDate="2018-04-12T16:17:00.080" Score="1" Body="&lt;p&gt;Yes and the most important parameter is the tree depth. It's a pre-pruning technique that allows to prevent overfitting. Specifically for sklearn is: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;estimator.tree_.max_depth&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I suggest you to perform GridSearch on max_depth:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = {'max_depth':[1,50]}&#xA;gs = GridSearchCV(DecisionTreeClassifier(), params)&#xA;gs.fit(X,y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where X is you training set containing instances&#xA;and y are the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are packages that also support post-pruning like this one &lt;a href=&quot;https://svaante.github.io/decision-tree-id3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://svaante.github.io/decision-tree-id3/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50429" LastActivityDate="2018-04-12T16:17:00.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30230" PostTypeId="1" CreationDate="2018-04-12T16:25:11.360" Score="1" ViewCount="261" Body="&lt;p&gt;I am doing an experiment on word2vec hyperparameter optimization using Grid Search, Bayesian Optimization and Population Based Training.&#xA;For this experiment, the code for Grid Search and Bayesian method has been written and working. Therefore I need help with writing the code for the Population Based Training Method. Please i need help with sample python code for hyperparameter optimization using Population Based training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="49758" LastActivityDate="2018-04-12T16:25:11.360" Title="python sample code for hyperparameter optimization using Population Based Training" Tags="|machine-learning|python|data-science-model|" AnswerCount="0" CommentCount="0" ClosedDate="2018-04-13T21:49:22.627" ContentLicense="CC BY-SA 3.0" />
  <row Id="30231" PostTypeId="1" CreationDate="2018-04-12T17:13:26.670" Score="1" ViewCount="147" Body="&lt;p&gt;I have multiple sensors providing time series sources with slightly different time stamps and different sampling rates. For example I have a breathing rate feature from 1 sensor at 1 Hz, EKG from another at 100 Hz and temperature for another at 10Hz.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I were to use this data with a 1D CNN or LSTM, how would I prepare it? I tried resampling all the sources to 10Hz and merging them by closest timestamp but that didn’t work out too well. What are common methods of tackling this problem?&lt;/p&gt;&#xA;" OwnerUserId="50433" LastActivityDate="2018-04-12T17:13:26.670" Title="How can I prepare my data from multiple time series sources for time series regression?" Tags="|keras|time-series|feature-engineering|lstm|cnn|" AnswerCount="0" CommentCount="0" ClosedDate="2018-04-17T22:35:28.913" ContentLicense="CC BY-SA 3.0" />
  <row Id="30232" PostTypeId="1" CreationDate="2018-04-12T17:18:37.230" Score="1" ViewCount="81" Body="&lt;p&gt;I have some test results in an NxN matrix which counts total fail results for that cell. I want to see if the failures generate any geometric patterns that can be identified automatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach I am planning is, I want to use the result matrix and use a Neural Network to Generate Classification. From the classification I want to see if it was actually able to find out some classes that the failure patterns can be identified spatially.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I was planning to use &lt;a href=&quot;http://www.pymvpa.org/examples/som.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;kohonen SOM&lt;/a&gt; to classify the data. From kohonen SOM with automatic class detection how to find out the only the number of classes inferred?&lt;/p&gt;&#xA;" OwnerUserId="31443" LastEditorUserId="31443" LastEditDate="2018-04-13T16:44:20.740" LastActivityDate="2018-04-15T02:54:36.813" Title="Number of classes in SOM" Tags="|neural-network|classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30233" PostTypeId="1" AcceptedAnswerId="30431" CreationDate="2018-04-12T17:59:31.640" Score="5" ViewCount="5387" Body="&lt;p&gt;For example, taking the image from &lt;a href=&quot;https://sebastianraschka.com/faq/docs/bagging-boosting-rf.html#boosting&quot; rel=&quot;noreferrer&quot;&gt;sebastian raschkas post &quot;Machine Learning FAQ&quot;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YknB5.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YknB5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would expect a very similar (if not exactly the same) result for a decision tree: Given only two features, it finds the optimal feature (and value for that feature) to split the classes. Then, the decision tree does the same for each child considering only the data which arrives in the child. Of course, boosting considers all the data again, but at least in the given sample it leads to exactly the same decision boundary. &lt;strong&gt;Could you make an example where a decision tree would have a different decision boundary on the same training set than boosted decision stumps?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the intuition that boosted decision stumps are less likely to overfit because the base classifier is so simple, but I couldn't exactly pin point why.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-04-12T20:11:35.553" LastActivityDate="2018-04-17T15:37:05.830" Title="How are boosted decision stumps different from a decision tree?" Tags="|decision-trees|boosting|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30235" PostTypeId="2" ParentId="30142" CreationDate="2018-04-12T18:39:39.137" Score="1" Body="&lt;p&gt;As mentioned by @Ryan the LDA is a way to go but I am not sure it will provide robust results on documents that are fundamentally limited to 140 characters in length. I tried it in the past on summaries of news articles and got mixed results. One alternative idea might be to test the performance of a supervised model like SVM or KNN when hash-tags are used as the classes? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an aside if you are committed to the LDA check out the gensim and LDAviz packages in python. &lt;/p&gt;&#xA;" OwnerUserId="29801" LastActivityDate="2018-04-12T18:39:39.137" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30237" PostTypeId="1" AcceptedAnswerId="30280" CreationDate="2018-04-12T20:04:46.017" Score="7" ViewCount="976" Body="&lt;p&gt;Following is a plot in Bishop's Pattern Recognition and Machine Learning explaining why $L_1$ regularization would lead to sparse solution, and I understand that in this case $w_1$ would be automatically constrained to be 0.  However, it seems to me that it is not always true that the corner point is closest to the optimal unconstrained solution.  For instance, if the optimal unconstrained solution is at the red cross, then there exists some point on the side that is closer to the red cross than corner points.  In this case, do we still have sparse solutions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A9yFq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A9yFq.png&quot; alt=&quot;A figure from Pattern Recognition and Machine Learning&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50440" LastEditorUserId="9123" LastEditDate="2018-04-13T02:42:17.857" LastActivityDate="2018-04-13T18:49:30.187" Title="Does L1 regularization always generate a sparse solution?" Tags="|machine-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30238" PostTypeId="1" AcceptedAnswerId="30242" CreationDate="2018-04-12T20:19:53.473" Score="8" ViewCount="940" Body="&lt;p&gt;So I was wondering how does one, for example, can best optimize the model they are trying to build when confronted with issues presented by high bias or high variance. Now, of course, you can play with the regularization parameter to get to a satisfying end, but I was wondering whether it is possible to do this without relying on regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If b is the bias estimator of a model and v of its variance, wouldn't it make sense to try to minimize b*v?&lt;/p&gt;&#xA;" OwnerUserId="50441" LastActivityDate="2018-04-12T22:04:55.727" Title="Question on bias-variance tradeoff and means of optimization" Tags="|machine-learning|variance|bias|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30239" PostTypeId="1" AcceptedAnswerId="30411" CreationDate="2018-04-12T20:37:53.160" Score="0" ViewCount="106" Body="&lt;p&gt;So far I have read 2 papers on GANs and it seems like they are unsupervised networks that only uses supervision for the discriminator. If I want to translate an image from a picture to line art all I need to do is to prepare my ground truth &quot;the unmodified picture&quot; and the corresponding line art correct? Do I have to pre-classify any facial feature for the generator to learn the corresponding line art or is that what Conditional GANs are for? I am trying to brainstorm my own project but the data preprocessing stage is a little hazy.&lt;/p&gt;&#xA;" OwnerUserId="46645" LastActivityDate="2018-04-17T08:40:11.473" Title="Image to image translation from image to sketch using GANs, do I need to annotated the face at all?" Tags="|data-cleaning|preprocessing|gan|generative-models|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30240" PostTypeId="1" CreationDate="2018-04-12T20:55:38.243" Score="1" ViewCount="629" Body="&lt;p&gt;I'm in a situation where I need to limit the number of support vectors (SVs) in my support vector machine binary classifier. A simple way to do this would be to manually remove the least important SVs. Does anyone know of a way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried manually modifying the following fields of the SVM to remove SVs, but without success:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.dual_coef_&#xA;clf.intercept_&#xA;clf.n_support_&#xA;clf.support_&#xA;clf.support_vectors_&#xA;clf.shape_fit_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;clf&lt;/code&gt; is my RBF SVM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that increasing regularization (by decreasing C) does not necessarily reduce the number of SVs (&lt;a href=&quot;https://dgroppe.com/2018/01/21/increased-regularization-does-not-necessarily-decrease-the-number-of-support-vectors/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dgroppe.com/2018/01/21/increased-regularization-does-not-necessarily-decrease-the-number-of-support-vectors/&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="26907" LastEditorUserId="26562" LastEditDate="2018-04-12T22:38:34.407" LastActivityDate="2018-11-14T18:29:17.540" Title="Is there a way to manually remove support vectors from a scikit-learn model?" Tags="|scikit-learn|svm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30241" PostTypeId="2" ParentId="29916" CreationDate="2018-04-12T21:18:34.150" Score="1" Body="&lt;p&gt;You &lt;strong&gt;cannot&lt;/strong&gt; expect clustering to understand what a &quot;product&quot; is. So it will not make one cluster per product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, the lines are very blurry: is the same mobile phone in different colors the same product, or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An &lt;em&gt;unsupervised&lt;/em&gt; method cannot do this. &lt;strong&gt;You need to provide training data and use supervised learning.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-12T21:18:34.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30242" PostTypeId="2" ParentId="30238" CreationDate="2018-04-12T22:04:55.727" Score="10" Body="&lt;p&gt;There are a lot of ways bias and variance can be minimized and despite the popular saying it isn't always a tradeoff.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The two main reasons for &lt;strong&gt;high bias&lt;/strong&gt; are &lt;strong&gt;insufficient model capacity&lt;/strong&gt; and &lt;strong&gt;underfitting&lt;/strong&gt; because the training phase wasn't complete. For example, if you have a very complex problem to solve (e.g. image recognition) and you use a model of low capacity (e.g. linear regression) this model would have high bias as a result of the model not being able of grasp the complexity of the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main reason for &lt;strong&gt;high variance&lt;/strong&gt; is &lt;strong&gt;overfitting&lt;/strong&gt; on the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said there are ways of &lt;strong&gt;reducing both bias and variance&lt;/strong&gt; on a ML model. For example the easiest way of achieving this is getting more data (in some cases even synthetic data help).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What we tend to do in practice is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;First, we increase the capacity of the model in order to &lt;strong&gt;reduce the variance&lt;/strong&gt; on the training set as much as possible. In other words we want to make the model &lt;strong&gt;overfit&lt;/strong&gt; (even reach a loss of 0 on the training set). This is done because we want to make sure the model has the &lt;strong&gt;capacity&lt;/strong&gt; of sufficiently understanding the data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then we try to &lt;strong&gt;reduce the bias&lt;/strong&gt;. This is done through &lt;strong&gt;regularization&lt;/strong&gt; (&lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;noreferrer&quot;&gt;early stopping&lt;/a&gt;, &lt;a href=&quot;https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c&quot; rel=&quot;noreferrer&quot;&gt;norm penalties&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Dropout_(neural_networks)&quot; rel=&quot;noreferrer&quot;&gt;dropout&lt;/a&gt;, etc.)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-04-12T22:04:55.727" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30244" PostTypeId="2" ParentId="16322" CreationDate="2018-04-12T22:27:18.337" Score="3" Body="&lt;p&gt;You could use the following function to determine how much memory your model requires:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_model_memory_usage(batch_size, model):&#xA;    import numpy as np&#xA;    from keras import backend as K&#xA;&#xA;    shapes_mem_count = 0&#xA;    for l in model.layers:&#xA;        single_layer_mem = 1&#xA;        for s in l.output_shape:&#xA;            if s is None:&#xA;                continue&#xA;            single_layer_mem *= s&#xA;        shapes_mem_count += single_layer_mem&#xA;&#xA;    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])&#xA;    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])&#xA;&#xA;    total_memory = 4.0*batch_size*(shapes_mem_count + trainable_count + non_trainable_count)&#xA;    gbytes = np.round(total_memory / (1024.0 ** 3), 3)&#xA;    return gbytes&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I copied this from ZFTurbo's answer on &lt;a href=&quot;https://stackoverflow.com/questions/43137288/how-to-determine-needed-memory-of-keras-model&quot;&gt;this&lt;/a&gt; post.&lt;/p&gt;&#xA;" OwnerDisplayName="user50384" LastActivityDate="2018-04-12T22:27:18.337" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30245" PostTypeId="1" AcceptedAnswerId="30263" CreationDate="2018-04-12T23:50:23.170" Score="2" ViewCount="20959" Body="&lt;p&gt;I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is &lt;a href=&quot;http://depositfiles.com/files/lamp9yxa7&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for the help.&lt;/p&gt;&#xA;" OwnerUserId="50449" LastEditorUserId="28175" LastEditDate="2018-04-13T00:07:52.190" LastActivityDate="2021-04-17T14:59:29.927" Title="Is there a way to replace existing values with NaN" Tags="|machine-learning|python|data-cleaning|data|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30246" PostTypeId="2" ParentId="30245" CreationDate="2018-04-13T00:06:29.583" Score="1" Body="&lt;p&gt;Depending on the data structure you are keeping the values there might be different solutions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using &lt;code&gt;Numpy&lt;/code&gt; arrays, you can employ &lt;code&gt;np.insert&lt;/code&gt; method which is referred &lt;a href=&quot;https://stackoverflow.com/a/39848762/5120235&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;a = np.arrray([(122.0, 1.0, -47.0), (123.0, 1.0, -47.0), (125.0, 1.0, -44.0)]))&#xA;np.insert(a, 2, np.nan, axis=0)&#xA;array([[ 122.,    1.,  -47.],&#xA;       [ 123.,    1.,  -47.],&#xA;       [  nan,   nan,   nan],&#xA;       [ 125.,    1.,  -44.]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you are using &lt;code&gt;Pandas&lt;/code&gt; you can use instance method &lt;code&gt;replace&lt;/code&gt; on the objects of the &lt;code&gt;DataFrames&lt;/code&gt; as referred &lt;a href=&quot;https://stackoverflow.com/a/34794112/5120235&quot;&gt;here&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [106]:&#xA;df.replace('N/A',np.NaN)&#xA;&#xA;Out[106]:&#xA;    x    y&#xA;0  10   12&#xA;1  50   11&#xA;2  18  NaN&#xA;3  32   13&#xA;4  47   15&#xA;5  20  NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the code above, the first argument can be your arbitrary input which you want to change.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-13T00:06:29.583" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30249" PostTypeId="1" AcceptedAnswerId="30252" CreationDate="2018-04-13T01:28:07.543" Score="8" ViewCount="69670" Body="&lt;p&gt;I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Age  Height  Weight  Gender&#xA;12    5'7     NaN     M&#xA;NaN   5'8     160     M&#xA;32    5'5     165     NaN&#xA;21    NaN     155     F&#xA;55    5'10    170     NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to remove all the rows where 'Gender' has NaN values. The output i'd like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Age  Height  Weight Gender&#xA;12   5'7     NaN     M&#xA;NaN  5'8     160     M&#xA;21   NaN     155     F&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="50449" LastActivityDate="2018-04-13T02:09:55.977" Title="How to delete entire row if values in a column are NaN" Tags="|machine-learning|python|data-cleaning|data|" AnswerCount="2" CommentCount="1" ClosedDate="2018-04-14T21:41:55.083" ContentLicense="CC BY-SA 3.0" />
  <row Id="30251" PostTypeId="2" ParentId="30249" CreationDate="2018-04-13T02:08:36.150" Score="5" Body="&lt;p&gt;If you are working on Python Dataframe, Please try this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = df[np.isfinite(df['Gender'])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Or you can use notnull instead of isfinite.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = df[pd.notnull(df['Gender'])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50292" LastActivityDate="2018-04-13T02:08:36.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30252" PostTypeId="2" ParentId="30249" CreationDate="2018-04-13T02:09:55.977" Score="15" Body="&lt;p&gt;Well if the dataset is not too large I would suggest using pandas to clean the data. So you would need to first do &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;python2 -m pip install pandas&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Python3&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;python3 -m pip install pandas&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you already have anaconda installed you can skip the above step. Next you could go through an IDE (like jupyter) or through the shell type the following commands&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.read_csv(&quot;filename&quot;, dtype=str)&#xA;#or if excel file&#xA;#df = pd.read_excel(&quot;filename&quot;, dtype=str)&#xA;&#xA;df = df[pd.notnull(df['Gender'])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then you would want to save your result in a file with &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.to_csv(&quot;newfile&quot;); &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49985" LastActivityDate="2018-04-13T02:09:55.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30255" PostTypeId="1" AcceptedAnswerId="36563" CreationDate="2018-04-13T06:28:02.897" Score="2" ViewCount="1484" Body="&lt;p&gt;I just read Demand-Driven Forecasting: A Structured Approach to Forecasting(Wiley and SAS Business Series) and have a few doubts in Holt-Winters Model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Unlike OLS Regression Modeling technique or ARIMA, no assumptions were checked before running Holt-Winters. For instance, in ARIMA, we first make the data stationary before running ARIMA or in OLS, we check normality, auto-correlation etc. However, as per the book, no test was conducted before and after running Holt-Winters. We just calculate MAPE and check if it is acceptable. So, can someone confirm if there are any tests that we should do before and after running Holt-Winters?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Can Holt-Winters model incorporate causal factors such as Price, Promotion, Marketing etc?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) When we are running Holt-Winters multiplicative or additive model, we don't need to explicitly adjust for seasonality before?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) How to treat for missing values or outliers in time series forecasting? We obviously cannot remove missing values rows as the time interval will break. Hence, for missing values, I have thought to use CAGR for that period(weekly, monthly or yearly) in the last 5 years and use it to estimate missing value. Can someone confirm if this correct? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I believe we shouldn't remove outliers and rather look why it is there in the first place. It might be because of some external factors like Price Change or Promotion which is causing the spike. This will ofcourse be not detected by Holt-Winters. So, if it is a rare promotion, then we should remove that value and insert the normal demand again by using CAGR of past data in order to maintain the historical pattern. Hence, the spike will be gone and the historical trend will be intact, thus increasing our MAPE. Can someone confirm if this correct? &lt;/p&gt;&#xA;" OwnerUserId="50217" LastActivityDate="2018-08-09T16:58:38.027" Title="What are the prerequisites before running Holt Winters Model?" Tags="|time-series|forecasting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30258" PostTypeId="2" ParentId="30097" CreationDate="2018-04-13T07:46:01.283" Score="1" Body="&lt;p&gt;There are a few things you can do to reduce over-fitting.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use Dropout increase its value and increase the number of training epochs&lt;/li&gt;&#xA;&lt;li&gt;Increase Dataset by using Data augmentation&lt;/li&gt;&#xA;&lt;li&gt;Tweak your CNN model by adding more training parameters. Reduce Fully Connected Layers.&lt;/li&gt;&#xA;&lt;li&gt;Change the whole Model&lt;/li&gt;&#xA;&lt;li&gt;Use Transfer Learning (Pre-Trained Models)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43147" LastActivityDate="2018-04-13T07:46:01.283" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30260" PostTypeId="2" ParentId="20452" CreationDate="2018-04-13T09:09:10.417" Score="1" Body="&lt;p&gt;&lt;strong&gt;How to approach the solution using a NN ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given your data, it doesn't look like an RNN/LSTM is the most sensible choice.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like you have two features and a binary output. RNN/LSTM are useful with variable-length input, otherwise, they are a bit overkill.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, make sure that you have a reasonable amount of data before using a deep learning approach. They usually have many parameters to train, and if you don't have enough data points, this might be difficult to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to stick to NN, I'd say a regular multilayer perceptron might do the job. If you are open to other options, I would try decision trees for this particular problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How do I convert the categorical value of OUI to numerical entities&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use one-hot encoding. There are plenty of implementations out there (i.e. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;), or you could do it yourself. &lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-04-13T09:09:10.417" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30263" PostTypeId="2" ParentId="30245" CreationDate="2018-04-13T09:47:19.347" Score="3" Body="&lt;h2&gt;Randomly replace values in a &lt;code&gt;numpy&lt;/code&gt; array&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# The dataset&#xA;data = pd.read_csv('iris.data')&#xA;mat = data.iloc[:,:4].as_matrix()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Set the number of values to replace. For example 20%:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Edit: changed len(mat) for mat.size&#xA;prop = int(mat.size * 0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Randomly choose indices of the numpy array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;i = [random.choice(range(mat.shape[0])) for _ in range(prop)]&#xA;j = [random.choice(range(mat.shape[1])) for _ in range(prop)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Change values with NaN&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mat[i,j] = np.NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Dropout for any array dimension&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Another way to do that with an array of more than 2 dimensions would be to use the &lt;code&gt;numpy.put()&lt;/code&gt; function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import random&#xA;from sklearn import datasets&#xA;&#xA;data = datasets.load_iris()['data']&#xA;&#xA;def dropout(a, percent):&#xA;    # create a copy&#xA;    mat = a.copy()&#xA;    # number of values to replace&#xA;    prop = int(mat.size * percent)&#xA;    # indices to mask&#xA;    mask = random.sample(range(mat.size), prop)&#xA;    # replace with NaN&#xA;    np.put(mat, mask, [np.NaN]*len(mask))&#xA;    return mat&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This function returns a modified array:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;modified = dropout(data, 0.2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We can verify that the correct number of values have been modified:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.sum(np.isnan(modified))/float(data.size)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;[out]:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0.2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="12490" LastEditorUserId="12490" LastEditDate="2018-04-14T05:29:33.410" LastActivityDate="2018-04-14T05:29:33.410" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30264" PostTypeId="1" CreationDate="2018-04-13T10:00:30.390" Score="1" ViewCount="1838" Body="&lt;p&gt;For this dataset, it seems that the predictions of my k-means model only consider the horizontal axis, although the cluster centers seem reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is something wrong with this classification? Please note the color of the grid in the background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use scikit-learn, here is the code fragment of classification and visualization.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = KMeans(n_clusters = 5)    &#xA;model.fit(df_stuff[['Stuff','Other Stuff']])&#xA;&#xA;fig = plt.figure()&#xA;ax = fig.add_axes([0,0,1,1])&#xA;&#xA;ax.scatter(df_stuff['Stuff'], df_stuff['Other Stuff'],c=model.labels_,s=80,cmap='rainbow')&#xA;ax.set_xlabel('Stuff')&#xA;ax.set_ylabel('Other Stuff')&#xA;ax.set_title('Strange Clusters')&#xA;&#xA;# Draw Cluster Centers&#xA;for center in model.cluster_centers_:&#xA;    ax.scatter(center[0],center[1],c='black',s=5120,alpha=0.2)&#xA;&#xA;# Draw Cluster Grid&#xA;cluster_grid = {'x': [], 'y': [], 'cluster': []}&#xA;for x in np.linspace(df_stuff['Stuff'].min(),df_stuff['Stuff'].max(),25):&#xA;    for y in np.linspace(0.35,0.6,25):&#xA;        cluster_grid['x'].append(x)&#xA;        cluster_grid['y'].append(y)&#xA;        cluster_grid['cluster'].append(model.predict([[x,y]])[0])&#xA;&#xA;ax.scatter(cluster_grid['x'],cluster_grid['y'],c=cluster_grid['cluster'],cmap='rainbow',alpha=0.4,s=10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GW2Ni.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GW2Ni.png&quot; alt=&quot;Plot of my strange clusters&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50471" LastEditorUserId="29575" LastEditDate="2018-04-13T14:28:53.843" LastActivityDate="2018-04-13T21:32:04.183" Title="k-means: Only one-dimensional cluster predictions in two-dimensional space" Tags="|scikit-learn|k-means|prediction|clustering|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30266" PostTypeId="1" CreationDate="2018-04-13T10:27:50.960" Score="1" ViewCount="281" Body="&lt;p&gt;I need to split my datasets based on my own feature column in order to hold together certain data rows (e.g. from one patient, or compound) for cross-validation (CV).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Orange 3.11, Test&amp;amp;Score contains &quot;Cross validation by feature&quot;, but it is disabled. The widget documentation contains a screenshot still without it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this a feature that is currently under development? Or do I have to define the relevant column differently to enable the option? (It's &quot;meta&quot; right now.) Or is there a different way to achieve this particular CV, i.e. use a pre-defined column for a &quot;Leave-group-out&quot; CV?&lt;/p&gt;&#xA;" OwnerUserId="50473" LastEditorUserId="50473" LastEditDate="2018-06-27T12:43:28.983" LastActivityDate="2019-06-18T15:37:22.790" Title="Orange: Group samples by a &quot;splitting&quot; feature for cross-validation?" Tags="|cross-validation|orange|sampling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30267" PostTypeId="1" CreationDate="2018-04-13T10:48:43.583" Score="1" ViewCount="73" Body="&lt;p&gt;It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have a lot of data, that represents a sequence of state, action, reward. We can train our agent on this data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We also need continuous actions, as the set of actions is big. Policy gradient is therefore the way to go but it generally uses an actor-critic that requires an emulator. We can not emulate, what would be the other options?&lt;/p&gt;&#xA;" OwnerUserId="45527" LastActivityDate="2018-05-15T18:12:42.060" Title="Policy gradient on data only, without emulators" Tags="|machine-learning|data-mining|dataset|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30268" PostTypeId="2" ParentId="23608" CreationDate="2018-04-13T11:47:14.870" Score="1" Body="&lt;p&gt;The text is obfuscated for you but to the computer, this is as incomprehensible as English for instance. When classifying text, you always first vectorize your input, meaning that your text becomes a vector. This vector typically represents the frequency of the tokens in your vocabulary in that particular string (bag-of-words representation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is represent each document/string as a vector representing the frequencies of characters, character n-grams, or even words. One you have your input, you can deal with it as any classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Naive Bayes has shown to be a good lower bound for these types of tasks.&lt;/p&gt;&#xA;" OwnerUserId="38887" LastActivityDate="2018-04-13T11:47:14.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30269" PostTypeId="1" CreationDate="2018-04-13T12:00:28.497" Score="1" ViewCount="594" Body="&lt;p&gt;I have a dataframe in the following format &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     symbol     name        date   close      market&#xA;0    BTC  Bitcoin  2013-04-28  134.21  1500520000&#xA;1    BTC  Bitcoin  2013-04-29  144.54  1491160000&#xA;2    BTC  Bitcoin  2013-04-30  139.00  1597780000&#xA;3    BTC  Bitcoin  2013-05-01  116.99  1542820000&#xA;4    BTC  Bitcoin  2013-05-02  105.21  1292190000&#xA;...&#xA;          symbol    name        date     close  market&#xA;752109    W3C  W3Coin  2018-04-07  0.000277       0&#xA;752110    W3C  W3Coin  2018-04-08  0.000286       0&#xA;752111    W3C  W3Coin  2018-04-09  0.000270       0&#xA;752112    W3C  W3Coin  2018-04-10  0.000286       0&#xA;752113    W3C  W3Coin  2018-04-11  0.000300       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to create a matrix with prices and a matrix with marketcaps. Because I'm interested in re-creating the S&amp;amp;P500 index. I have done this with a for loop below. &lt;strong&gt;Question: How do I do this without a for loop?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;edit: Note that some symbols have different number of dates, for example BTC might have 1000 days but W3Coin only 50 days. I think that is what the error message is telling me.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;import data: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.read_csv('data-coins-scrape/CryptoData.csv', parse_dates=True)&#xA;cols = ['symbol', 'name', 'date', 'close', 'market']&#xA;df = df[cols]&#xA;symbols = df.symbol.unique()&#xA;nr_coins = df.symbol.nunique()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for loop to get prices and makretcaps &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prices = []&#xA;marketscaps = []&#xA;shapes = []&#xA;for s in symbols:&#xA;    # slice out one coin&#xA;    dfc = df.loc[df.symbol == s]&#xA;    shapes.append(dfc.shape)&#xA;    prices.append(dfc['close'])&#xA;    marketcaps.append(df['market'])&#xA;del dfc&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;check the output &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;len(prices), len(marketcaps) &#xA;# (1531, 3065) is the output &#xA;&#xA;type(prices[0]), prices[0].shape, prices[0].name &#xA;# output: (pandas.core.series.Series, (1810,), 'close')&#xA;&#xA;df.loc[df.symbol == 'BTC', 'close'].head() == prices[0].head() &#xA;# returns True &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastEditorUserId="45191" LastEditDate="2018-04-20T12:25:57.670" LastActivityDate="2018-04-20T12:25:57.670" Title="Create price matrix from tidy data without for loop" Tags="|python|pandas|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30271" PostTypeId="2" ParentId="30269" CreationDate="2018-04-13T15:19:11.087" Score="2" Body="&lt;p&gt;You can do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1 = df.set_index(['name', 'date'])&#xA;prices = df1['close'].unstack()&#xA;marketcaps = df1['marketcaps'].unstack()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="14265" LastActivityDate="2018-04-13T15:19:11.087" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30272" PostTypeId="2" ParentId="22194" CreationDate="2018-04-13T15:23:27.967" Score="2" Body="&lt;p&gt;Well, absolutely. I had the same issue recently (and ended up here). So I just simulated a dataset and an independant label (&lt;a href=&quot;https://www.thekerneltrip.com/statistics/what-to-expect-from-a-model-when-there-is-nothing-to-learn/&quot; rel=&quot;nofollow noreferrer&quot;&gt;full article here&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, when there is nothing to learn, your model will produce predictions uncorrelated to the label. In turn, these predictions (when repeated many times) show an interesting distribution of the error rate (evaluated on a validation set or using cross validation).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in your case, I am quite surprised that one specific model does not work correctly, when the others seem to show relevant predictions.&lt;/p&gt;&#xA;" OwnerUserId="50478" LastActivityDate="2018-04-13T15:23:27.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30273" PostTypeId="2" ParentId="30237" CreationDate="2018-04-13T15:34:35.937" Score="0" Body="&lt;p&gt;Note that the plot only depicts two dimensions. In that example the $w_1$ coefficient is 0 while $w_2$ is something different from 0. Thus, you could say that the sparsity is 50% since half of your coefficients are 0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having said that, you could have problems where the optimal solution has a 0% sparsity and problems where the sparsity is 100%, even with $L_1$ regularization. It depends on the data and the amount of regularization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, the short answer is that with $L_1$ is just easier for the coefficients to be exactly $0$ than with $L_2$, where it is almost impossible. In fact, it is easy to compute the exact $\lambda_1$ for which all the coefficients of the solution are $0$. If you decrease that value you start observing that some of them are different from $0$.&lt;/p&gt;&#xA;" OwnerUserId="14265" LastActivityDate="2018-04-13T15:34:35.937" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30274" PostTypeId="2" ParentId="30264" CreationDate="2018-04-13T15:55:33.137" Score="0" Body="&lt;p&gt;It might not be your clustering that is the problem. But the visual representation of your clustering. It is unclear why you are plotting elements on a hardcoded grid. You should just plot the actual values of the raw data. Something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;&#xA;# Plot the data&#xA;X = df_stuff[['Stuff','Other Stuff']]&#xA;plt.scatter(X[:, 0], X[:, 1], c=model.predict(X))&#xA;&#xA;# Plot k-means clusters centers&#xA;centers = kmeans.cluster_centers_&#xA;plt.scatter(centers[:, 0], centers[:, 1], c='black')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-13T15:55:33.137" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30275" PostTypeId="1" CreationDate="2018-04-13T16:01:27.820" Score="1" ViewCount="752" Body="&lt;p&gt;When reading papers on neural networks, I occasionally stumble upon the following notation with a semicolon:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\text{tanh}(\mathbf{W_c}[\mathbf{c}_t;\mathbf{h}_t])&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless otherwise noted, does this by default mean the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;vector $\mathbf{c}_t$ is &lt;strong&gt;appended&lt;/strong&gt; to vector $\mathbf{h}_t$&lt;/li&gt;&#xA;&lt;li&gt;the resulting long vector is dot-producted with a Weight matrix $\mathbf{W}_t$&lt;/li&gt;&#xA;&lt;li&gt;Finally, the resulting vector is component-wise activated by a hyperbolic tangent function &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first bullet point is my main question.&#xA;Googling for &quot;Vector concatenation notation&quot; doesn't return answers that would resemble the image I've attached above. However, many papers seem to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example &lt;a href=&quot;https://math.stackexchange.com/questions/298648/is-there-a-common-symbol-for-concatenating-two-finite-sequences&quot;&gt;of notation that I googled&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="26562" LastEditDate="2018-04-14T01:07:02.270" LastActivityDate="2018-04-14T01:07:02.270" Title="Does this notation mean vector-concatenation?" Tags="|neural-network|notation|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30276" PostTypeId="2" ParentId="30264" CreationDate="2018-04-13T16:26:14.973" Score="2" Body="&lt;p&gt;You are overfitting your data. You are fitting 5 clusters for ~20 data points. The red and blue clusters only have a single data point. Either get more data or fit fewer clusters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Elbow_method_(clustering)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Elbow method&lt;/a&gt; will help decide how many clusters are appropriate. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-13T16:26:14.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30277" PostTypeId="1" CreationDate="2018-04-13T17:18:21.183" Score="7" ViewCount="5549" Body="&lt;p&gt;In short:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the policy gradient method, if the reward is always positive (never negative), the policy gradient will always be positive, hence it will keep making our parameters larger. This makes the learning algorithm meaningless. How do we get around this problem?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In detail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &quot;RL Course by David Silver&quot; lecture 7 (on YouTube), he introduced the REINFORCE algorithm for policy gradient (here just showing 1 step):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/c9eKs.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/c9eKs.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual policy update is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2yc42.gif&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2yc42.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that $v_t$ here stands for the reward we get. Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence $\theta$ will keep increasing! So how do we deal with rewards that never change sign?&lt;/p&gt;&#xA;" OwnerUserId="50482" LastActivityDate="2018-04-15T08:28:55.133" Title="RL Policy Gradient: How to deal with rewards that are strictly positive?" Tags="|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30278" PostTypeId="2" ParentId="23323" CreationDate="2018-04-13T17:43:43.533" Score="0" Body="&lt;p&gt;There is another big news dataset in Kaggle called &lt;strong&gt;All The News&lt;/strong&gt; you can dwnload it &#xA;&lt;a href=&quot;https://www.kaggle.com/snapcrack/all-the-news&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;All The News&quot;&gt;Here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data primarily falls between the years of 2016 and July 2017.&#xA;And were scraped with beautiful soup from big US news sites like:&#xA;New York Times, Breitbart, CNN, Business Insider, the Atlantic, Fox News, Talking Points Memo, Buzzfeed News and many more.&lt;/p&gt;&#xA;" OwnerUserId="42460" LastActivityDate="2018-04-13T17:43:43.533" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30280" PostTypeId="2" ParentId="30237" CreationDate="2018-04-13T18:27:00.717" Score="2" Body="&lt;p&gt;I think with your red cross you have in mind the case where $w_1 = w_2$. You are right that in this scenario the solution will not be sparse, since you will also get $w^*_1 = w^*_2$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, whenever the coordinates of the red cross are off the line $y=x$ ($w_1\neq w_2$), you can shrink the regularization region so much that eventually you will end up with a similar picture as Figure 3.4 above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say your red cross has the coordinates $w_1=r_1$ and $w_2=r_2$ and say your regularization strength is such that $||w||_1 = |w_1|+|w_2| \leq C$ where $C$ defines the size of the shaded region.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cfaBC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cfaBC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, for the circular contours, the optimal solution will be where the line $w^*_2 = C- w^*_1$ (for the boundary region) and the line through the red cross at $(r_1,r_2)$ running &lt;em&gt;orthogonal&lt;/em&gt; to the boundary region intersect. You can see that the latter has the form $w_2 = f(w_1) = 1\cdot w_1 + r_2 - r_1$. (Orthogonality gives you slope +1 because the boundary region has slope -1.) Setting them equal gives you the intersection at coordinates $w^*_1 = \frac{C+r_1-r_2}{2}$ and $w^*_2 = \frac{C-r_1+r_2}{2}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if $r_2=r_1$ you have $w^*_1=w^*_2$ independent of $C$ and the solution won't be sparse.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $r_1&amp;gt;r_2$ (i.e. you are below the first diagonal) shrinking $C$ to $r_1-r_2&amp;gt;0$ yields $w^*_2=0$ but $w^*_1&amp;gt;0$. You end up in the right corner of the regularization region. (Similar to my image.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If $r_2&amp;gt;r_1$ (i.e. you are above the first diagonal) shrinking $C$ to $r_2-r_1&amp;gt;0$ yields $w^*_1=0$ but $w^*_2&amp;gt;0$. You end up in the top corner of the regularization region. (Similar to your image.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As Emre mentioned in the comments, this holds in general and you can &quot;see&quot; this from the Kuhn-Tucker conditions of the optimization problem. &lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-04-13T18:49:30.187" LastActivityDate="2018-04-13T18:49:30.187" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30282" PostTypeId="1" CreationDate="2018-04-13T19:08:02.640" Score="2" ViewCount="44" Body="&lt;p&gt;I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine portion. The target is the quality of the wine, rated from 1-10. I would like to change the values in the column 'quality' such that if the value is from 1-4, it is replaced with 'low', if the value is from 5-6, the value is replaced with 'medium', and if the value is 8-10, it is replaced with 'high'. My code is as such:&lt;/p&gt;&#xA;&lt;p&gt;#Read in data&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;white = pd.read_csv('Wine_data.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;#Rename columns to exlude the u&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;columns = ['fixed acidity', 'volatile acidity', 'citric acid','residual sugar', 'chlorides', 'free sulfur dioxide','total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol', 'quality']&#xA;white.columns = columns&#xA;&#xA;quality = np.array(white['quality'])&#xA;&#xA;for i in range(len(quality)):&#xA;    if quality[i] in [1,2,3,4]:&#xA;        quality[i] = 'low'&#xA;    elif quality[i] in [5,6,7]:&#xA;        quality[i] = 'med'&#xA;    else:&#xA;        quality[i] = 'high'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;However, I am getting this error:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Error in lines 13-19&#xA;Traceback (most recent call last):&#xA;  File &amp;quot;/cocalc/lib/python2.7/site-packages/smc_sagews/sage_server.py&amp;quot;, line 1013, in execute&#xA;    exec compile(block+'\n', '', 'single') in namespace, locals&#xA;  File &amp;quot;&amp;quot;, line 5, in &amp;lt;module&amp;gt;&#xA;ValueError: invalid literal for long() with base 10: 'med'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://cocalc.com/share/d9ac106e-6ccf-46b6-b1bc-e97bcbf46956/Wine%20Quality%20Project.sagews?viewer=share&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a link to my code.&lt;/p&gt;&#xA;&lt;p&gt;Help this nOOb please.&lt;/p&gt;&#xA;" OwnerUserId="50489" LastEditorUserId="98307" LastEditDate="2020-08-14T08:15:30.977" LastActivityDate="2020-08-14T08:15:30.977" Title="Help with replacing values in a data frame" Tags="|machine-learning|python|pandas|random-forest|dataframe|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30283" PostTypeId="2" ParentId="19187" CreationDate="2018-04-13T19:21:23.237" Score="1" Body="&lt;p&gt;Exciting stuff that you're getting into! You may find this video helpful about &lt;a href=&quot;http://www.decisivedata.net/blog/what-is-predictive-analytics&quot; rel=&quot;nofollow noreferrer&quot;&gt;Predictive Analytics and its fundamentals&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They offer a ton of helpful insight on predictive analytics fundamentals and how predictive analytics fit into the spectrum of analytics and data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hopefully, this will help clarify!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best Wishes, Bethany&lt;/p&gt;&#xA;" OwnerUserId="50492" LastActivityDate="2018-04-13T19:21:23.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30284" PostTypeId="1" CreationDate="2018-04-13T19:31:15.607" Score="0" ViewCount="234" Body="&lt;p&gt;I have a set of clusters which each cluster contains a list of short documents. I want to compute how coherent and cohesive each cluster is and filter out the incoherent and in-cohesive ones. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of &lt;strong&gt;intra-cluster distance&lt;/strong&gt; and &lt;strong&gt;within-cluster dispersion&lt;/strong&gt;, which are part of computing clustering evaluation metrics &lt;strong&gt;Silhouette Coefficient&lt;/strong&gt; and &lt;strong&gt;Calinski-Harabaz Index&lt;/strong&gt;, respectively. My question is, is there any other metrics or ways to compute such intra-cluster coherence or cohesion? Is there any standard ways for this that I am not aware of? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="2454" LastActivityDate="2018-04-14T09:46:40.623" Title="Measuring document cluster cohesion" Tags="|nlp|clustering|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30285" PostTypeId="2" ParentId="30040" CreationDate="2018-04-13T19:36:40.610" Score="1" Body="&lt;p&gt;Keras &lt;code&gt;Model.predict()&lt;/code&gt; method doesn't shuffle the data, so each row in &lt;code&gt;p_valid&lt;/code&gt; corresponds to the row in &lt;code&gt;x_valid&lt;/code&gt;. For instance, &lt;code&gt;p_valid[2]&lt;/code&gt; holds the probabilities for the image &lt;code&gt;x_valid[2]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, note that the values in each row don't necessarily sum up to &lt;code&gt;1&lt;/code&gt;, because they correspond to the probability of class &lt;code&gt;i&lt;/code&gt; being present, thus are not exclusive.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-04-13T19:36:40.610" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30286" PostTypeId="2" ParentId="30282" CreationDate="2018-04-13T19:50:58.353" Score="2" Body="&lt;p&gt;Looks like &lt;code&gt;smc_sagews&lt;/code&gt; requires numeric values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should replace strings &lt;code&gt;low&lt;/code&gt;/&lt;code&gt;med&lt;/code&gt;/&lt;code&gt;high&lt;/code&gt; with numbers 0/1/2 or -1/0/1.&lt;/p&gt;&#xA;" OwnerUserId="9867" LastActivityDate="2018-04-13T19:50:58.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30287" PostTypeId="1" CreationDate="2018-04-13T20:44:33.227" Score="3" ViewCount="159" Body="&lt;p&gt;I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;There are two states ($a$ and $b$) stepping to $a$ is terminal.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;All rewards are zero, discount for stepping from $b \to b$ is $1$, and discount for all other steps is zero&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The possible scenarios are: $a\to b$ (probability $1$), $b\to a$ (probability $p$) and $b\to b$ (probability $1-p$).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The first question I have is whether or not its true that the optimal values for each state here are zero? If not how did you derive this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second question I have is if we want a parameter $\lambda$ and have a feature $\phi$ such that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\lambda \times \phi(a)$ and $\lambda \times \phi(b)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;approximate the optimal values at the states a and b and we attempt to approximate such a $\lambda$ by means of TD(0) starting with $\lambda_0 = 1$ how can I find the expected value of $\lambda$ after one episode of updating in terms of $p$? ($E[\lambda_T]$ where $T$ is a random number representing the duration of the episode)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By an episode I mean if we go back from state b to a the episode is over.&lt;/p&gt;&#xA;" OwnerUserId="50494" LastEditorUserId="836" LastEditDate="2018-04-14T07:57:34.340" LastActivityDate="2018-07-15T02:24:45.083" Title="What is the optimal value of a Markov Decision process with Single actions at each state?" Tags="|machine-learning|reinforcement-learning|q-learning|markov-process|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30288" PostTypeId="2" ParentId="30264" CreationDate="2018-04-13T21:26:02.003" Score="3" Body="&lt;p&gt;KMeans does correctly do what it is supposed to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just plot your data correctly, with the same scale on both axes...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Y deviations do not matter, they are tiny compared to the X axis. Deviations there are 100x larger, so squared deviations even 10000x. Since KMeans minimized squared errors, only x matters&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When plotted correctly, your data more looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/GW2Ni.png&quot; width=&quot;800&quot; height=&quot;10&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2018-04-13T21:32:04.183" LastActivityDate="2018-04-13T21:32:04.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30290" PostTypeId="2" ParentId="30282" CreationDate="2018-04-13T22:06:08.807" Score="3" Body="&lt;p&gt;Assuming that the data is small enough that efficiency is not an issue, I would use the Pandas data frame and do something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;white['quality_code'] = 'high'&#xA;white.loc[quality &amp;lt; 5, 'quality_code'] = 'low'&#xA;white.loc[(quality &amp;gt; 4) &amp;amp; (quality &amp;lt; 8) , 'quality_code'] = 'med'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="15127" LastActivityDate="2018-04-13T22:06:08.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30292" PostTypeId="1" AcceptedAnswerId="30293" CreationDate="2018-04-13T23:15:35.763" Score="0" ViewCount="687" Body="&lt;p&gt;I have a (&lt;strong&gt;.csv&lt;/strong&gt;) file with more than &lt;strong&gt;35,000&lt;/strong&gt; rows and &lt;strong&gt;100&lt;/strong&gt; columns, where the rows represent the attributes and columns represent the instances. In Excel the maximum number of columns is &lt;strong&gt;16,384&lt;/strong&gt;. Therefore, it is impossible to do this in one single Excel sheet. I need my dataset to be transposed in order to perform some machine learning algorithms in WEKA. Is there any tool allow such thing like this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, is there a way to transpose (&lt;strong&gt;.arff&lt;/strong&gt;) file in WEKA?&lt;/p&gt;&#xA;" OwnerUserId="41350" LastEditorUserId="29575" LastEditDate="2018-04-13T23:50:46.953" LastActivityDate="2018-04-14T00:55:02.817" Title="How can I transpose a high dimensional dataset?" Tags="|machine-learning|csv|weka|excel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30293" PostTypeId="2" ParentId="30292" CreationDate="2018-04-14T00:55:02.817" Score="0" Body="&lt;p&gt;This is possible to do in WEKA, but it only works if:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the first column of your data is of string or nominal type, and &lt;/li&gt;&#xA;&lt;li&gt;the rest of your columns are numeric. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Open the dataset in the explorer, and first remove the class attribute by choosing 'No Class' from the drop-down menu on the right. Then, apply the filter from &lt;code&gt;filters.unsupervised.attribute.Transpose&lt;/code&gt;. After this, you should re-apply the class attribute by re-selecting the class attribute in the drop-down menu.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data doesn't meet the requirements listed above, I would recommend using this short python script to do it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.read_csv('your_file_here.csv', index=False)&#xA;df.transpose().to_csv('your_new_file_here.csv', index=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then loading the new file into WEKA.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-14T00:55:02.817" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30294" PostTypeId="2" ParentId="30277" CreationDate="2018-04-14T01:23:01.480" Score="3" Body="&lt;p&gt;Had a struggle with this recently as well, and here is what I came away with: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe you can think of the policy update result as similar to total loss over an episode. Recall that in a vanilla neural net, eg a perceptron, loss will be positive as well, and the optimizer is calculating how much each parameter in theta would have to change to achieve that loss (via gradient descent), and then this is backpropagated or the final update. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference here is that in this case we are trying to maximize a reward. Why then does the reward not go to infinity at some point? because there is a finite number of episodes in each simulation, and each action can give a maximum of 1, so there is an upper bound on the total reward given by your setup.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am not really great on theory, but code snippets make more sense to me, so seeing how they are implemented gives some intuition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pytorch version&lt;/a&gt; - see the finish_episode() function - which is fairly clear. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/awjuliani/DeepRL-Agents/blob/master/Vanilla-Policy.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow Example&lt;/a&gt; (sorry I do not know a better one) - see the bottom of the session loop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here a simulation is run over several episodes which entail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each step in an episode, an action is chosen by sampling policy action scores, and these are stored, along with reward from the game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At end of episode, a total policy loss is calculated with the policy action log probs and actual rewards (averaging over all steps in this case). Gradients are calculated w.r.t to the final policy loss, and backpropagated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, here is the [&lt;a href=&quot;http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf&lt;/a&gt; ](original paper) on REINFORCE, see pages 4-5 which explains that the update vector is not necessarily increasing, but &quot;lies in a direction for which [the] performance measure is increasing&quot;.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-14T01:23:01.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30295" PostTypeId="1" CreationDate="2018-04-14T02:19:32.127" Score="1" ViewCount="633" Body="&lt;p&gt;I'd like to know if there are any libraries that allow imputation by &lt;strong&gt;clustering&lt;/strong&gt;, &lt;strong&gt;regression&lt;/strong&gt; and &lt;strong&gt;stochastic&lt;/strong&gt; &lt;strong&gt;regression&lt;/strong&gt;. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the best imputation method for an small dataset (Iris in this case). I had to delibrately create NaN values since Iris set has none.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code for KNN imputation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; import pandas as pd&#xA; import numpy as np&#xA; import random&#xA; from fancyimpute import KNN&#xA;&#xA; data = pd.read_csv(&quot;D:/Iris_classification/train.csv&quot;)&#xA; mat = data.iloc[:,:4].as_matrix()&#xA;&#xA; prop = int(mat.size * 0.5) #Set the % of values to be replaced&#xA; i = [random.choice(range(mat.shape[0])) for _ in range(prop)] #Randomly choose indices of &#xA; j = [random.choice(range(mat.shape[1])) for _ in range(prop)] #the numpy array &#xA;&#xA; mat[i,j] = np.NaN #replace values with NaN&#xA;&#xA;&#xA;&#xA; mat_filled = pd.DataFrame(KNN(3).complete(mat)) #converted the array back to df&#xA;&#xA; data_col = data.drop('species', axis = 1)&#xA; mat_filled.columns = data_col.columns  #added column names that went missing in mat_filled&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a similar way to impute with the other 3 methods?&lt;/p&gt;&#xA;" OwnerUserId="50449" LastActivityDate="2024-02-16T00:07:20.967" Title="Is there are way to impute missing values by clustering, regression and stochastic regression" Tags="|machine-learning|python|data-cleaning|data|data-imputation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30296" PostTypeId="1" CreationDate="2018-04-14T04:16:25.530" Score="1" ViewCount="193" Body="&lt;p&gt;I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1-&lt;strong&gt;Which have rmse closer to average rmse:&lt;/strong&gt; pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2-&lt;strong&gt;Best performance:&lt;/strong&gt; Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="50491" LastActivityDate="2018-04-14T07:53:47.503" Title="Best model for Machine Learning" Tags="|machine-learning|classification|scikit-learn|tensorflow|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30297" PostTypeId="1" CreationDate="2018-04-14T04:34:53.397" Score="1" ViewCount="468" Body="&lt;p&gt;Greeting, everyone! I have $X = (x_1,x_2,...,x_n)$, and $Y = (y_1,y_2,...,y_n)$ where $x_i$ - is a time series, and $y_i$ - is the output time series for $x_i$. e.g. for $x_{ik}$ &lt;em&gt;($i$ - index of time series, $i = 1,n$, $k$ - number of pattern(idk how it's named), $k=1,t$, $t$ - size of $x_i$)&lt;/em&gt;, I have $y_{ik}$, I'll pray that you understand me))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to do: I want to train LSTM-based NN, for predicting $X$ to $Y$. As you seen, for each $x_ik$, I have $y_ik$, so I think that it will be great, if NN will learn on whole time series, not just on the last point, question is how I should set my NN in &lt;strong&gt;Keras&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the end, once again, I try to explain:&#xA;$x_i = (x_{i1}, x_{i2},...,x_{ik})$ and output is $y_i = (y_{i1}, y_{i2},...,y_{ik})$, but for now I train my model with $x_i = (x_{i1}, x_{i2},...,x_{ik})$ and $y_i = y_{ik}$ - so it's just one last point. &lt;/p&gt;&#xA;" OwnerUserId="42730" LastActivityDate="2018-04-14T04:34:53.397" Title="How to set LSTM for train on multiple time series on keras?" Tags="|neural-network|deep-learning|keras|time-series|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30298" PostTypeId="1" CreationDate="2018-04-14T05:13:32.947" Score="5" ViewCount="1822" Body="&lt;p&gt;This might be a weird question but I'm trying to have a deep understanding of how neural networks work theoretically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was doing some tests with my perceptron and I decided to test it on a single input single output dataset. What I was looking for was 100% accuracy since what I was testing on was a trivial separable dataset with binary output. However, the accuracy was below 50% (it wasn't balanced). I realized that this is due to the fact that there is a single weight that when trained is either going to be =&gt;0 or &amp;lt;0. So after going through the activation function (e.g. hardlim), unless the input is scaled to be [-1,1] range, it's always going to give the same output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does my explanation make sense? Are there any theoretical aspects I'm missing?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, are neural networks useful at all for single input problems? They seems useless to me since we're basically classifying by putting a point or points (for multiclass problems) on a single line to separate outputs, which is a pretty simple problem that doesn't need the intricacy of neural networks.&lt;/p&gt;&#xA;" OwnerUserId="50506" LastActivityDate="2020-06-16T13:39:20.403" Title="Are single input single output neural networks possible?" Tags="|neural-network|classification|perceptron|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30299" PostTypeId="2" ParentId="30227" CreationDate="2018-04-14T05:36:25.727" Score="1" Body="&lt;p&gt;It might be because of learning rate. You need to reduce your learning rate and let it run for more epochs.&lt;/p&gt;&#xA;" OwnerUserId="50428" LastActivityDate="2018-04-14T05:36:25.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30300" PostTypeId="1" CreationDate="2018-04-14T05:55:17.420" Score="3" ViewCount="190" Body="&lt;p&gt;I have data set of sequences of user executed commands sorted in the order of its occurrence. The data looks like this.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;user1 : cmd1-&amp;gt;cmd2-&amp;gt;cmd3-&amp;gt;...-&amp;gt;cmdn&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;user2 : cmd1-&amp;gt;cmd2-&amp;gt;cmd3-&amp;gt;cmd5-&amp;gt;...-&amp;gt;cmdn&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;user3 : cmd6-&amp;gt;cmd2-&amp;gt;cmd4-&amp;gt;cmd5-&amp;gt;...-&amp;gt;cmdn&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So the idea is to predict the next user command ordered by probability based on the previous command or sequence of commands similar to the clickstream prediction.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried &lt;code&gt;1st-order Markov Chain&lt;/code&gt; but it is only based on previous command, and higher order Markov Chain would not be efficient because the sequences could be really long, with many different numbers of command.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any other way to solve this kind of problem?&lt;/p&gt;&#xA;" OwnerUserId="50507" LastEditorUserId="35644" LastEditDate="2018-04-16T00:30:14.660" LastActivityDate="2018-04-16T00:30:14.660" Title="Next events prediction based on previous events" Tags="|prediction|sequence|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30301" PostTypeId="2" ParentId="30296" CreationDate="2018-04-14T07:53:47.503" Score="2" Body="&lt;p&gt;If you use the test set for model selection (it does not matter whether you use some random state, or do hyperparameter tuning) then it is dirty and can no longer be used to predict the prediction performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general workflow you should follow is this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Split into train and test. Lock &quot;test&quot; in a safe.&lt;/li&gt;&#xA;&lt;li&gt;Use cross-validation on the &lt;em&gt;train set only&lt;/em&gt; to tune parameters.&lt;/li&gt;&#xA;&lt;li&gt;Fix the hyperparameter to the best setting found. Write them in boldface as 'final parameters' into a &lt;strong&gt;stone&lt;/strong&gt; plate on your wall.&lt;/li&gt;&#xA;&lt;li&gt;Train a new classifier on the entire training set using exactly these parameters&lt;/li&gt;&#xA;&lt;li&gt;Get the test set from the safe, and run the classifier on the test set exactly once to get a prediction of how well your classifier is going to work in the future&lt;/li&gt;&#xA;&lt;li&gt;If the results are substantially worse than those from 2.: panic. It's not going to work.&lt;/li&gt;&#xA;&lt;li&gt;Optional: If you are &lt;em&gt;confident&lt;/em&gt; that hyperparameters do not depend on the data size but that you need every little bit of training data, you could train a classifier on train+test with the in-stone parameters. This is risky though, a you don't have a test set anymore.&lt;/li&gt;&#xA;&lt;li&gt;Deploy, but monitor that it really performs as good as predicted.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The golden rule is: if you used the test set, you must not change the model anymore. You've got one shot to score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is why you split the train set again in CV.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-14T07:53:47.503" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30302" PostTypeId="1" AcceptedAnswerId="30485" CreationDate="2018-04-14T07:54:10.320" Score="10" ViewCount="6590" Body="&lt;p&gt;The following plot shows coefficients obtained with linear regression (with &lt;code&gt;mpg&lt;/code&gt; as the target variable and all others as predictors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For mtcars dataset (&lt;a href=&quot;https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/mtcars.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://gist.github.com/seankross/a412dfbd88b3db70b74b&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) both with and without scaling the data: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZEDri.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZEDri.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I interpret these results? The variables &lt;code&gt;hp&lt;/code&gt; and &lt;code&gt;disp&lt;/code&gt; are significant only if the data is scaled. Are &lt;code&gt;am&lt;/code&gt; and &lt;code&gt;qsec&lt;/code&gt; equally important or is &lt;code&gt;am&lt;/code&gt; more important than &lt;code&gt;qsec&lt;/code&gt;? Which variable should one say are important determinants of &lt;code&gt;mpg&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your insight. &lt;/p&gt;&#xA;" OwnerUserId="49700" LastEditorUserId="25180" LastEditDate="2020-05-08T20:08:11.000" LastActivityDate="2020-05-08T20:08:11.000" Title="Linear Regression and scaling of data" Tags="|feature-selection|linear-regression|feature-scaling|" AnswerCount="2" CommentCount="5" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30303" PostTypeId="1" CreationDate="2018-04-14T09:12:29.373" Score="1" ViewCount="333" Body="&lt;p&gt;Logistic regression is generally performed if there are 2 categories in outcome variables. I just tried it for iris dataset with species as y variable which has 3 categories. I used following code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import matplotlib.pylab as plt&#xA;&#xA;from sklearn.linear_model import LogisticRegression&#xA;clf = LogisticRegression()&#xA;&#xA;from sklearn import datasets&#xA;iris = datasets.load_iris() &#xA;&#xA;clf.fit(iris.data, iris.target)&#xA;logcoefdf = pd.DataFrame(data=clf.coef_, &#xA;     columns=[&quot;SL&quot;, &quot;SW&quot;, &quot;PL&quot;, &quot;PW&quot;], &#xA;     index=['setosa','versicolor','virginica'])&#xA;print(logcoefdf)&#xA;logcoefdf.plot.bar()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The printout and plot of coefficients is as follows: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                  SL        SW        PL        PW&#xA;setosa      0.414988  1.461297 -2.262141 -1.029095&#xA;versicolor  0.416640 -1.600833  0.577658 -1.385538&#xA;virginica  -1.707525 -1.534268  2.470972  2.555382&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(I have labelled rows by names of species but I am not sure if this is correct).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From above output I get following plot: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Zj84C.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Zj84C.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the interpretation of these results? Does it mean that petal length (PL) is lowest in setosa and highest in virginica group? And both sepal width and petal width are less in versicolor species? Thanks for your insight.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: &#xA;If I use only 2 categories of iris dataset, I get only one set of coefficients: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf.fit(iris.data[0:100,:], iris.target[0:100])&#xA;print(clf.coef_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[[-0.40731745 -1.46092371  2.24004724  1.00841492]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is it that Logistic regression is being performed for all possible combinations of categories, i.e. setosa vs versicolor, versicolor vs virginica and virginica vs setosa?&lt;/p&gt;&#xA;" OwnerUserId="49700" LastEditorUserId="49700" LastEditDate="2018-04-14T10:56:35.277" LastActivityDate="2018-04-14T14:22:58.100" Title="Logistic regression if 3 categories in outcome variable" Tags="|logistic-regression|categorical-data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30304" PostTypeId="2" ParentId="22" CreationDate="2018-04-14T09:24:39.837" Score="8" Body="&lt;p&gt;You should not use k-means clustering on a dataset containing mixed datatypes. Rather, there are a number of clustering algorithms that can appropriately handle mixed datatypes. Some possibilities include the following: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Partitioning-based algorithms: k-Prototypes, Squeezer&lt;br&gt;&#xA;2) Hierarchical algorithms: ROCK, Agglomerative single, average, and complete linkage&lt;br&gt;&#xA;3) Density-based algorithms: HIERDENC, MULIC, CLIQUE&lt;br&gt;&#xA;4) Model-based algorithms: SVM clustering, Self-organizing maps &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to learn more about these algorithms, the manuscript 'Survey of Clustering Algorithms' written by Rui Xu offers a comprehensive introduction to cluster analysis.&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-04-14T09:24:39.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30305" PostTypeId="2" ParentId="30284" CreationDate="2018-04-14T09:46:40.623" Score="1" Body="&lt;p&gt;The Silhouette Coefficient is a good option. The Calinski-Harabaz Index has been shown to be unstable for evaluating noisy datsets, so I would recommend the Silhouette Coefficient over Calinski-Harabaz Index.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The Clustering Validation index based on Nearest Neighbors (CVNN) index has been demonstrated to work well for many different datasets containing high noise, arbitrarily shaped clusters, and skewed distributions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few other validation measures you can try using are the SD validity index, the S_Dbw validitiy index, and the Xie-Beni index. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following is a good manuscript for interal clustering validation metrics &lt;a href=&quot;http://datamining.rutgers.edu/publication/internalmeasures.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;manuscript&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-04-14T09:46:40.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30306" PostTypeId="2" ParentId="30269" CreationDate="2018-04-14T10:02:08.313" Score="0" Body="&lt;p&gt;create a price matrix with pd.concat, some works and some give a type error &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.concat([prices[0], prices[1], prices[2], prices[3]], axis=1, join='outer').tail() &#xA;# works!&#xA;&#xA;pricemat = pd.concat(prices, keys=[btc.index], axis=1) &#xA;# TypeError: unhashable type: 'DatetimeIndex'&#xA;&#xA;pricemat = pd.concat(prices, axis=1) &#xA;#TypeError: unhashable type: 'DatetimeIndex'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;create a price matrix with for loop, give a value error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;priced_df = prices[0]&#xA;K = len(prices)&#xA;for i in range(1, K):&#xA;    priced_df = pd.concat([priced_df, prices[i]], axis=1)&#xA;    print(priced_df.shape[1])&#xA;# at 25 it stops and says:&#xA;&#xA;# ValueError: Shape of passed values is (25, 2057), indices imply (25, 1810)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-04-14T10:02:08.313" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30307" PostTypeId="2" ParentId="5224" CreationDate="2018-04-14T10:11:20.977" Score="2" Body="&lt;p&gt;While wacax's answer is complete and really explanatory, I would like to add a couple of things in case anyone stumbles on this answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, most &lt;code&gt;scipy.misc&lt;/code&gt; image related functions (&lt;code&gt;imread&lt;/code&gt;, &lt;code&gt;imsave&lt;/code&gt;, &lt;code&gt;imresize&lt;/code&gt; erc) have become deprecated in favor of either &lt;a href=&quot;https://pypi.python.org/pypi/imageio&quot; rel=&quot;nofollow noreferrer&quot;&gt;imageio&lt;/a&gt; or &lt;a href=&quot;http://scikit-image.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;skimage&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, I would strongly recommend the python library &lt;a href=&quot;http://imgaug.readthedocs.io/en/latest/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;imgaug&lt;/a&gt; for any augmentation task. It is really easy to use and has virtually all augmentation techniques you might want to use.&lt;/p&gt;&#xA;" OwnerDisplayName="user50384" LastActivityDate="2018-04-14T10:11:20.977" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30308" PostTypeId="2" ParentId="30277" CreationDate="2018-04-14T10:37:34.903" Score="6" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence θ will keep increasing! So how do we deal with rewards that never change sign?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is true. However, in many policy functions and in most situations, the gradient part $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ will tend to zero as you reach a deterministic policy. This happens for a softmax action selection based on &quot;preferences&quot; (a matrix of softmax weights per action for each state) or as the output layer of a neural network. And it will counteract the tendency for the final layer preferences (or the logits of the neural network last layer) to grow uncontrollably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have identified a true weakness of REINFORCE. For instance, using the softmax action selection as an example and your always-positive returns:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;When the agent selects a non-maximising action, this will result in a positive return, the agent will add to its preference for that action.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;When the agent selects a maximising action, this will result in a &lt;em&gt;larger&lt;/em&gt; positive return, the agent will add to its preference for that action &lt;em&gt;more&lt;/em&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;REINFORCE works by increasing the preferences of better actions &lt;em&gt;faster&lt;/em&gt; than preferences of worse actions. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;This leads to a feedback process where better actions are chosen more often, increasing their preference values even faster.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Ultimately the preference for best actions will be so much higher than the alternatives that the softmax function will saturate. The gradient  $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ for all actions will be close to zero.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This does not always happen quickly, and basic REINFORCE implementations can be numerically unstable for exactly the reason in your question. To improve stability (and often learning speed), you can use &lt;a href=&quot;https://github.com/dennybritz/reinforcement-learning/blob/master/PolicyGradient/CliffWalk%20REINFORCE%20with%20Baseline%20Solution.ipynb&quot; rel=&quot;noreferrer&quot;&gt;REINFORCE with baseline&lt;/a&gt;, which starts to address your concern by using offsetting values ($v_t - \bar{v}_t$ or similar). You can also then take that idea further and use Actor-Critic.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-04-15T08:28:55.133" LastActivityDate="2018-04-15T08:28:55.133" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30309" PostTypeId="1" AcceptedAnswerId="30328" CreationDate="2018-04-14T10:42:07.767" Score="0" ViewCount="288" Body="&lt;p&gt;So I have a dataset in which I have to predict class binary label (1 or 0), the problem, out of 120k data points, only 200 have the label '1'.&#xA;the aim is to minimize FN and FP.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Which  ML model should I use? Gradient boost or XGBoost or logistic regression. How do I calculate the class weights?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Which accuracy metric captures the minimization of FP and FN?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50514" LastEditorUserId="26562" LastEditDate="2018-04-14T21:37:00.273" LastActivityDate="2018-05-07T09:21:03.560" Title="Right ML mode and metric to minimize FN and FP on imbalanced dataset" Tags="|machine-learning|random-forest|sampling|class-imbalance|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30310" PostTypeId="1" CreationDate="2018-04-14T11:19:09.577" Score="1" ViewCount="209" Body="&lt;p&gt;Reading the literature around deep learning adversarial attacks it appears to be wholly concentrated on attacks of image classification models. Are there papers that describe attacks on non image data ? Searching archive for deep learning adversarial attacks appears to contain results that are just related to image classification field.&lt;/p&gt;&#xA;" OwnerUserId="237" LastActivityDate="2022-01-19T16:05:20.567" Title="Adversarial attacks on non image data" Tags="|machine-learning|deep-learning|research|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30312" PostTypeId="1" AcceptedAnswerId="30894" CreationDate="2018-04-14T12:23:49.790" Score="3" ViewCount="5437" Body="&lt;p&gt;On looking at various machine learning methods at the &lt;a href=&quot;http://scikit-learn.org/stable/modules/classes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;, it appears that some modules such as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;LinearRegression&lt;/code&gt;&lt;/a&gt; provide coefficients (&lt;code&gt;coef_&lt;/code&gt;), others such as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;AdaBoostRegressor&lt;/code&gt;&lt;/a&gt; provide &lt;code&gt;feature_importances_&lt;/code&gt; while some e.g. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;BaggingRegressor&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;BernoulliNB&lt;/code&gt;&lt;/a&gt; do not provide either of these.&lt;/p&gt;&#xA;&lt;p&gt;Why this difference? Are &lt;code&gt;coef_&lt;/code&gt; similar to &lt;code&gt;feature_importances_&lt;/code&gt; to assess the contribution of a variable in prediction? How to assess the feature importance for modules where neither of these is available?&lt;/p&gt;&#xA;" OwnerUserId="49700" LastEditorUserId="24000" LastEditDate="2021-03-31T13:36:43.220" LastActivityDate="2021-03-31T13:36:43.220" Title="Regression coefficients vs feature_importances_ vs none" Tags="|machine-learning|scikit-learn|feature-selection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30313" PostTypeId="1" CreationDate="2018-04-14T12:39:21.350" Score="1" ViewCount="250" Body="&lt;p&gt;Which specific performance evaluation metrics are used in training, validation and testing and &lt;strong&gt;why&lt;/strong&gt;? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically I am actually after deciding when to use (i.e. in training, validation or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally I am using performance evaluation for environmental data where I am using ANN to model. I have continuous features and am predicting a continuous variable.&lt;/p&gt;&#xA;" OwnerUserId="50515" LastEditorUserId="50515" LastEditDate="2018-04-14T14:20:17.220" LastActivityDate="2018-04-14T14:20:17.220" Title="Performance Evaluation Metrics used in Training, Validation and Testing" Tags="|machine-learning|predictive-modeling|performance|model-selection|machine-learning-model|" AnswerCount="0" CommentCount="6" ClosedDate="2018-04-20T08:21:54.527" ContentLicense="CC BY-SA 3.0" />
  <row Id="30314" PostTypeId="2" ParentId="30303" CreationDate="2018-04-14T14:22:58.100" Score="1" Body="&lt;p&gt;Logistic regression can work on multi-class. Frankly it is not big different from binary classifier. The question is how does it find the coefficients for each class which you had displayed. By default to find the coefficients for a single class, it goes with &lt;code&gt;one vs rest combination&lt;/code&gt;. So &lt;code&gt;setosa&lt;/code&gt; vs remaining all classes. It gets coefficients iteratively by minimizing cross entropy. So ultimately you will end up with a matrix having a size of &lt;code&gt;n_classes vs features&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Coming to the interpretation part of the question. To interpret the importance of a single feature, you can do it as - &lt;em&gt;Given the coefficients of every other feature to be same (in comparing a class with another) including intercept, the class that has the feature with highest coefficient has greater chance of engulfing the new point.&lt;/em&gt; This being said the features having &lt;code&gt;high petal length and width&lt;/code&gt; and &lt;code&gt;low sepal length and width&lt;/code&gt; contribute good for a data point to belong to class &lt;code&gt;virginica&lt;/code&gt;. But it does not make much sense to interpret each coefficient individually. Hope I made some sense.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-04-14T14:22:58.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30316" PostTypeId="1" CreationDate="2018-04-14T15:32:41.310" Score="1" ViewCount="430" Body="&lt;p&gt;After reading different articles about ML and algorithms, scientist tends to use different words when describing the different aspects in ML.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So now I'm a bit confused myself and I hope you can correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) So to my understanding supervised/unsupervised learning are different categories of machine learning algorithms. Each category contains different algorithms such as Neural Networks and Bayesian?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Regression, Classification and Clustering are types of models? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) A model is the result of a trained algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that I'm not completely wrong, thanks! :)&lt;/p&gt;&#xA;" OwnerUserId="50528" LastEditorUserId="50528" LastEditDate="2018-04-14T15:53:26.390" LastActivityDate="2018-04-14T23:57:02.237" Title="Confused about the different aspects in Machine Learning" Tags="|machine-learning|unsupervised-learning|supervised-learning|machine-learning-model|" AnswerCount="2" CommentCount="4" ClosedDate="2018-04-17T22:31:25.770" ContentLicense="CC BY-SA 3.0" />
  <row Id="30317" PostTypeId="2" ParentId="30015" CreationDate="2018-04-14T15:33:05.760" Score="2" Body="&#xA;&#xA;&lt;p&gt;Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often.&#xA;In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Tensorflow&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is implemented there (&lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn_cell_impl.py#L390&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tensorflow/python/ops/rnn_cell_impl.py&lt;/code&gt;&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def call(self, inputs, state):&#xA;    &quot;&quot;&quot;Most basic RNN: output = new_state = act(W * input + U * state + B).&quot;&quot;&quot;&#xA;&#xA;    gate_inputs = math_ops.matmul(&#xA;        array_ops.concat([inputs, state], 1), self._kernel)&#xA;    gate_inputs = nn_ops.bias_add(gate_inputs, self._bias)&#xA;    output = self._activation(gate_inputs)&#xA;    return output, output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A single matrix multiplication is more efficient, so it's applied, even though the comment describes the expanded formula with two matrices.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Keras&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;On the other hand, Keras often chooses simplicity and clarity over performance. Here's its implementation (&lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L887&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;keras/layers/recurrent.py&lt;/code&gt;&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def call(self, inputs, states, training=None):&#xA;    prev_output = states[0]&#xA;&#xA;    ...&#xA;&#xA;    if dp_mask is not None:&#xA;        h = K.dot(inputs * dp_mask, self.kernel)&#xA;    else:&#xA;        h = K.dot(inputs, self.kernel)&#xA;    if self.bias is not None:&#xA;        h = K.bias_add(h, self.bias)&#xA;&#xA;    ...&#xA;    output = h + K.dot(prev_output, self.recurrent_kernel)&#xA;    if self.activation is not None:&#xA;        output = self.activation(output)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The class thus makes its it easy to access two matrices separately (&lt;code&gt;self.kernel&lt;/code&gt; and &lt;code&gt;self.recurrent_kernel&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Pytorch&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://pytorch.org/docs/master/_modules/torch/nn/modules/rnn.html#RNNCell&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pytorch approach&lt;/a&gt; is closer to keras. Moreover, not only they use separate kernel matrices, they also have two bias vectors, and these four arrays are accessible for the client code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite these differences all three libraries are functionally equivalent.&lt;/p&gt;&#xA;" OwnerUserId="18375" LastActivityDate="2018-04-14T15:33:05.760" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30320" PostTypeId="1" CreationDate="2018-04-14T18:15:57.667" Score="3" ViewCount="313" Body="&lt;p&gt;I'm working on a project where the task is to &lt;strong&gt;classify videos&lt;/strong&gt; of birds and predict the species. I have tried to use machine learning models originally designed for &lt;strong&gt;deep action recognition&lt;/strong&gt; and tweaked for my dataset.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Dataset&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The dataset consists of 9 different classes with around 3 000 videos with labels:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Björktrast (Fieldfare)&lt;/li&gt;&#xA;&lt;li&gt;Koltrast (Blackbird)&lt;/li&gt;&#xA;&lt;li&gt;Nötväcka (Nuthatch)&lt;/li&gt;&#xA;&lt;li&gt;Pilfink (Tree sparrow)&lt;/li&gt;&#xA;&lt;li&gt;Mus (Mouse)&lt;/li&gt;&#xA;&lt;li&gt;Blåmes (Blue tit)&lt;/li&gt;&#xA;&lt;li&gt;Rödhake (Robin)&lt;/li&gt;&#xA;&lt;li&gt;Katt (Cat)&lt;/li&gt;&#xA;&lt;li&gt;Inget objekt (No object)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://www.youtube.com/watch?v=4iNpw7J5q9I&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; of a sample dataset containing the different classes that my deep learning model should be able to predict:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=4iNpw7J5q9I&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ExQEP.jpg&quot; alt=&quot;https://www.youtube.com/watch?v=4iNpw7J5q9I]&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Method&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Currently I have used this codebase for &lt;a href=&quot;https://github.com/yjxiong/temporal-segment-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;temporal segment networks&lt;/a&gt; to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have expanded the dataset to around 30 000 videos using the following techniques:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Adding RGB noise to videos already in the training dataset.&lt;/li&gt;&#xA;&lt;li&gt;Adding random distortions and warping to videos in the training dataset.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2&gt;Problems&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I have a few questions about doing video classification to predict bird species:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Are there better techniques for doing video classification compared to using temporal segment networks?&lt;/li&gt;&#xA;&lt;li&gt;Are there any preprocessing steps that should be done to improve accuracy?&lt;/li&gt;&#xA;&lt;li&gt;How to train the model to generalize better to be less sensitive to lighting and weather?&lt;/li&gt;&#xA;&lt;li&gt;How can the model be trained to classify unknown species?&lt;/li&gt;&#xA;&lt;li&gt;Are there other ways to expand the dataset except those mentioned above?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In short: What is the best deep learning model for video classification for bird species?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2018/04/17:&lt;/strong&gt; I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.&lt;/p&gt;&#xA;" OwnerUserId="50531" LastEditorUserId="50531" LastEditDate="2018-04-16T22:34:04.007" LastActivityDate="2018-04-17T09:02:04.953" Title="Video classification of birds" Tags="|machine-learning|deep-learning|classification|preprocessing|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30321" PostTypeId="2" ParentId="30316" CreationDate="2018-04-14T19:12:00.087" Score="6" Body="&lt;p&gt;Good question and welcome to Datascience&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine you have the tree as follows.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;                 Machine Learning Models&#xA;                          |&#xA;           ----------------------------------------------------&#xA;           |                                                  |&#xA;       Supervised                                         Unsupervised&#xA;           |                                                  |&#xA;   - --------------------                                  Clustering&#xA;   |                    |&#xA;Regression          Classification&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;A model is indeed a trained version of the algorithm you are chosing,&#xA;regardless of whether its Regression, Classification or Clustering. The above&#xA;tree is ofcourse a simplified version of all ML algorithms.&lt;/p&gt;&#xA;" OwnerUserId="26674" LastEditorUserId="26674" LastEditDate="2018-04-14T20:52:27.473" LastActivityDate="2018-04-14T20:52:27.473" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30323" PostTypeId="1" CreationDate="2018-04-14T19:30:13.650" Score="1" ViewCount="38" Body="&lt;p&gt;I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All these predictors have the same unit, as they are all car usage percentage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem comes from the fact that some zip codes have very 'inactive' car usage: the values is often 0, or much smaller compared with others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I use general linear models, such as LASSO or ridge, I usually standardize my data first. Through standardization, these 'inactive' predictors actually got amplified.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am wondering if there is a systematic way to restrict the beta's of these inactive predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any insight?&lt;/p&gt;&#xA;" OwnerUserId="31006" LastEditorUserId="18375" LastEditDate="2018-04-14T21:37:49.380" LastActivityDate="2021-11-15T15:06:30.817" Title="Linear Model: How to deal with predictors with a lot of missing/small values?" Tags="|data-cleaning|missing-data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30324" PostTypeId="2" ParentId="19744" CreationDate="2018-04-14T19:54:13.387" Score="1" Body="&lt;p&gt;The pdf describes the following way:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us call the variable as var&#xA;where &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;m = mean(var)&#xA;&#xA;sd  = sd(var)&#xA;&#xA;x = (var-m)/sd&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will create a list of values which are Ipsative standardized, which is called Z&lt;em&gt;i&lt;/em&gt; score&lt;/p&gt;&#xA;" OwnerUserId="50533" LastEditorUserId="29575" LastEditDate="2018-04-14T20:11:38.873" LastActivityDate="2018-04-14T20:11:38.873" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30325" PostTypeId="1" CreationDate="2018-04-14T21:35:26.763" Score="5" ViewCount="260" Body="&lt;p&gt;I was wondering if there's a good way to use ensembling when I have two or more algoritims producing ranked lists.  That is, suppose I have the following datasets consisting of ordered lists (higher to the top means more relevant):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Method1_Rankings  Method2_Rankings GoldStandard_Rankings&#xA;item1             item2             item1&#xA;item3             item1             item3&#xA;item2             item10            item5&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to optimally combine methods 1 and 2 (e.g., give the rankings some weights or similar)?  Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50535" LastActivityDate="2024-02-01T13:00:45.167" Title="Methods for ensembling ranked lists?" Tags="|ranking|ensemble|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30326" PostTypeId="2" ParentId="23739" CreationDate="2018-04-14T23:52:17.620" Score="4" Body="&lt;p&gt;I did some extensive experimenting to address the asked question. My experiments indicated that the encoding path (left leg of the NN) should have less, but wider layers. I usually take half as many layers, but double the number of nodes for the encoding path. I have no explanation for this, just these configurations often led to faster convergence.&lt;/p&gt;&#xA;" OwnerUserId="50539" LastEditorUserId="29169" LastEditDate="2021-03-11T20:10:32.847" LastActivityDate="2021-03-11T20:10:32.847" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30327" PostTypeId="2" ParentId="30316" CreationDate="2018-04-14T23:57:02.237" Score="3" Body="&lt;p&gt;1) Supervised learning is most of the time the process of learning a mapping, e.g relation, of input features &lt;em&gt;x&lt;/em&gt; (sample) to an output &lt;em&gt;y&lt;/em&gt; (often labels).&#xA;Unsupervised learning doesn’t not use labels /output &lt;em&gt;y&lt;/em&gt; to learn a relation between the samples and possible labels (ex: clustering).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Classification and regression are two types of supervised learning (discrete output labels vs continuous).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Very good resources exist on the forum and web to go deeper with it if you’d like, don’t hesitate.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-04-14T23:57:02.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30328" PostTypeId="2" ParentId="30309" CreationDate="2018-04-15T00:42:57.373" Score="0" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;If you use class weights, I don't think it would make much of a difference which model you use (regarding the imbalance). If you are familiar with python, I would suggest &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; sklearn function that could help you out in computing the class weights. However, I have found that &lt;strong&gt;oversampling&lt;/strong&gt; (or undersampling) often works better than class weights. Consider using &lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/generated/imblearn.over_sampling.SMOTE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt; for this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;While there are metrics and loss functions better suited at handling class imbalance (&lt;a href=&quot;http://contrib.scikit-learn.org/imbalanced-learn/stable/generated/imblearn.metrics.geometric_mean_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;GMS&lt;/a&gt;, &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;quadratic kappa&lt;/a&gt;, etc.) I don't think you would have an issue with using any loss &#xA;if you considered oversampling your data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerDisplayName="user50384" LastActivityDate="2018-04-15T00:42:57.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30329" PostTypeId="1" CreationDate="2018-04-15T01:27:19.730" Score="2" ViewCount="1707" Body="&lt;p&gt;My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, I have the number of customers in a store at any given time. I want to create a three dimensional array with axes [month, day_of_week, hour] where the values are the mean of the historical values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One example data point, where the data covers several years but each axis here is just 12 months, 7 days, and 24 hours. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Array[January,Friday,1pm] = 3 customers on average &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried Pandas Groupby but I can't figure out how to translate that information from a Groupby object or Panel to a useful 3D array. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My closest attempt so far:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dataframe.groupby([&quot;month&quot;,&quot;day_of_week&quot;,&quot;hour&quot;])[&quot;count&quot;].mean().unstack().to_panel()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This outputs a Panel object, but I can't find a way to translate this to just a 3D array. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me figure out a way to solve my problem and build this 3D array without for loops? &lt;/p&gt;&#xA;" OwnerUserId="44299" LastEditorUserId="44299" LastEditDate="2018-04-15T01:39:28.810" LastActivityDate="2018-04-15T01:39:28.810" Title="Multidimensional Groupby to ND Array" Tags="|python|pandas|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30330" PostTypeId="2" ParentId="12727" CreationDate="2018-04-15T02:07:56.093" Score="2" Body="&lt;p&gt;Augmenting Aneel's answer, I had to add &lt;code&gt;escape='&quot;'&lt;/code&gt; option get this working properly. Spark 2.3&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;spark.read.csv(DATA_FILE, sep=',', escape='&quot;', header=True, &#xA;               inferSchema=True, multiLine=True).count()&#xA;&#xA;159571&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Interestingly, Pandas can read this without any additional instructions. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.read_csv(DATA_FILE).shape&#xA;&#xA;(159571, 8)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="43768" LastEditorUserId="29575" LastEditDate="2018-04-15T02:27:25.017" LastActivityDate="2018-04-15T02:27:25.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30334" PostTypeId="2" ParentId="17157" CreationDate="2018-04-15T08:50:19.990" Score="2" Body="&lt;p&gt;The 1., 2., 3. approach mentioned by HEITZ is in my experience possible extending with an alternative for 3. where you write your data from R (data.table) back into MySQL.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So full steps are MySQL-&gt;data.table-&gt;MySQL&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you ensure you use data.table syntax where you don't copy the DT its also RAM-friendly. &lt;/p&gt;&#xA;" OwnerUserId="50555" LastActivityDate="2018-04-15T08:50:19.990" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30336" PostTypeId="1" CreationDate="2018-04-15T09:27:32.033" Score="2" ViewCount="3001" Body="&lt;p&gt;I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it possible to run such a script using spark-submit? In the python script I included this block of code for spark context. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import findspark&#xA;findspark.init()&#xA;import pyspark&#xA;from pyspark import SparkContext&#xA;APP_NAME = 'kineticPCA'&#xA;conf = SparkConf().setAppName(APP_NAME)&#xA;conf = conf.setMaster('spark://192.168.xx.xxx:7077')&#xA;sc = SparkContext(conf=conf)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the how submit the job on command prompt, on the spark bin directory &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;spark-submit G:/guru orders/pca kinetics/files/kineticPCA.py&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the error I get:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Error: Cannot load main class from JAR G:/guru with URI G. Please&#xA;  s through --class.&#xA;  Run with --help for usage help or --verbose for debug output&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZEhBR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZEhBR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be the cause of this error?&lt;/p&gt;&#xA;" OwnerUserId="50556" LastEditorUserId="29575" LastEditDate="2018-04-15T15:13:08.367" LastActivityDate="2018-04-24T07:55:27.770" Title="Run python script using spark-submit on windows 7" Tags="|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30337" PostTypeId="2" ParentId="30205" CreationDate="2018-04-15T09:30:38.810" Score="4" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/users/35644/aditya&quot;&gt;Aditya&lt;/a&gt; already mentioned rightly so in the comments RMSE, MSE or MAE is preferred while evaluating a linear regression model. So to answer your questions I will provide with the pros and cons of using RMSE and MAE and further more put forward some points to why R-squared is not preferred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. MAE vs. RMSE - Which metric is better ?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ideally this depends on the loss function of your model, the most common one being L2 - euclidean distance (you can create your own loss function too). RMSE gives relatively higher weight to large errors. It does not necessarily increase with the variance of the errors but increases with the variance of the frequency distribution of error magnitudes. While, MAE is comparatively steady.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;typically, it makes more sense to give a higher weight to points away from the mean in which RMSE would be preferred. In a scenario where being away from the mean by any magnitude would be penalized equally, MAE would be preferred.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RMSE can be troublesome when calculating on different sized test samples (As is the case when training a model). RMSE has a tendency to be increasingly larger than MAE as the test sample size increases ([RMSE] ≤ [MAE * sqrt(n)].&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. Is R-squared useless ?&lt;/strong&gt; (Kinda harsh but sorry not sorry)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To keep it short and simple.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;R-squared does not measure goodness of fit.&lt;/li&gt;&#xA;&lt;li&gt;R-squared does not measure predictive error.&lt;/li&gt;&#xA;&lt;li&gt;R-squared does not allow you to compare models using transformed&#xA;responses.&lt;/li&gt;&#xA;&lt;li&gt;R-squared does not measure how one variable explains another.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="32958" LastActivityDate="2018-04-15T09:30:38.810" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30338" PostTypeId="2" ParentId="29559" CreationDate="2018-04-15T10:39:22.487" Score="0" Body="&lt;p&gt;A workflow like this might do the trick, although it is static (does not adjust to varying actions).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Load the two files, Summary and Details.&lt;/li&gt;&#xA;&lt;li&gt;I suppose &quot;Details&quot; has columns such as &quot;Action&quot; and some value. In my example, Details will contain &quot;Subject&quot; and &quot;Grade&quot; for 3 highschool subjects.&lt;/li&gt;&#xA;&lt;li&gt;&quot;Select columns&quot; is optional (slim down unused columns)&lt;/li&gt;&#xA;&lt;li&gt;You need a &quot;Select rows&quot; for each subject/action, filtering that specific one.&lt;/li&gt;&#xA;&lt;li&gt;Every &quot;Edit Domain&quot; is used to rename the generic feature name to a specific on (like &quot;Grade&quot; to &quot;Grade_Math&quot;).&lt;/li&gt;&#xA;&lt;li&gt;Results are then merged step-wise to the Summary (which in my example contains students, your summary contains some other people).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If the number of actions is limited, the time needed is reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is a one-time job, you might as well save the output (CSV) and use that in a different project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it needs to be more dynamic and adaptive, you'll have to dig into Python and use the Python code widget to create a list of actions, create a list of columns based on that, and then populate those with matching values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/B19nM.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/B19nM.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50473" LastActivityDate="2018-04-15T10:39:22.487" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30340" PostTypeId="1" CreationDate="2018-04-15T12:36:56.170" Score="4" ViewCount="4681" Body="&lt;p&gt;What are some examples of linear and non-linear machine learning models (algorithms) for purposes of comparison between the two categories? Which are the parameters (or scalars in a linear algebraic sense) and which are the predictors/factors (or vectors in a linear algebraic sense again)?&lt;/p&gt;&#xA;" OwnerUserId="50563" LastEditorUserId="26019" LastEditDate="2020-03-31T16:25:03.123" LastActivityDate="2020-03-31T16:25:03.123" Title="Distinguising features of linear vs, non-linear machine learning models (algorithms)" Tags="|machine-learning|deep-learning|classifier|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-17T06:03:18.107" ContentLicense="CC BY-SA 4.0" />
  <row Id="30342" PostTypeId="1" CreationDate="2018-04-15T14:29:10.700" Score="0" ViewCount="54" Body="&lt;p&gt;Hi I would like to ask how to sample out 50 instances from 150 instances of iris data by using Jackknife. &#xA;Is it possible? &#xA;Thanks in advance&lt;/p&gt;&#xA;" OwnerUserId="50570" LastActivityDate="2018-04-15T17:02:34.340" Title="R programming (Jackknife)" Tags="|machine-learning|r|sampling|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-16T06:00:45.587" ContentLicense="CC BY-SA 3.0" />
  <row Id="30343" PostTypeId="2" ParentId="29863" CreationDate="2018-04-15T16:11:59.407" Score="10" Body="&lt;p&gt;&lt;code&gt;indicator_column&lt;/code&gt; encodes the input to a &lt;code&gt;multi-hot&lt;/code&gt; representation, not &lt;code&gt;one-hot&lt;/code&gt; encoding.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; clarifies more:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;name = indicator_column(categorical_column_with_vocabulary_list(&#xA;    'name', ['bob', 'george', 'wanda'])&#xA;columns = [name, ...]&#xA;features = tf.parse_example(..., features=make_parse_example_spec(columns))&#xA;dense_tensor = input_layer(features, columns)&#xA;&#xA;dense_tensor == [[1, 0, 0]]  # If &amp;quot;name&amp;quot; bytes_list is [&amp;quot;bob&amp;quot;]&#xA;dense_tensor == [[1, 0, 1]]  # If &amp;quot;name&amp;quot; bytes_list is [&amp;quot;bob&amp;quot;, &amp;quot;wanda&amp;quot;]&#xA;dense_tensor == [[2, 0, 0]]  # If &amp;quot;name&amp;quot; bytes_list is [&amp;quot;bob&amp;quot;, &amp;quot;bob&amp;quot;] &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The last two examples describe what is meant by &lt;code&gt;multi-hot&lt;/code&gt; encoding. for example if the input be &lt;code&gt;[&amp;quot;bob&amp;quot;, &amp;quot;wanda&amp;quot;]&lt;/code&gt; the encoding will be &lt;code&gt;[[1, 0, 1]]&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="43130" LastEditorUserId="98307" LastEditDate="2020-08-02T12:23:40.057" LastActivityDate="2020-08-02T12:23:40.057" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30344" PostTypeId="1" AcceptedAnswerId="30347" CreationDate="2018-04-15T16:55:34.020" Score="52" ViewCount="93484" Body="&lt;p&gt;It seems the &lt;a href=&quot;http://ruder.io/optimizing-gradient-descent/&quot; rel=&quot;noreferrer&quot;&gt;Adaptive Moment Estimation (Adam)&lt;/a&gt; optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why not always use Adam? Why even bother using RMSProp or momentum optimizers?&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2020-08-05T08:48:02.517" Title="Why not always use the ADAM optimization technique?" Tags="|neural-network|optimization|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30345" PostTypeId="2" ParentId="30342" CreationDate="2018-04-15T17:02:34.340" Score="1" Body="&lt;p&gt;here you go:&#xA;&lt;a href=&quot;http://www.rexamples.com/14/Sample()&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.rexamples.com/14/Sample()&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sample(x, size, replace = FALSE, prob = NULL)&lt;/p&gt;&#xA;" OwnerUserId="6667" LastActivityDate="2018-04-15T17:02:34.340" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30347" PostTypeId="2" ParentId="30344" CreationDate="2018-04-15T17:36:51.447" Score="42" Body="&lt;p&gt;&lt;a href=&quot;https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/&quot; rel=&quot;noreferrer&quot;&gt;Here&lt;/a&gt;’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.&lt;/p&gt;&#xA;&lt;p&gt;There is often a value to using more than one method (an ensemble), because every method has a weakness.&lt;/p&gt;&#xA;" OwnerUserId="50307" LastEditorUserId="98307" LastEditDate="2020-08-05T08:48:02.517" LastActivityDate="2020-08-05T08:48:02.517" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30348" PostTypeId="2" ParentId="30267" CreationDate="2018-04-15T17:40:24.937" Score="1" Body="&lt;p&gt;I think your best approach is to use Imitation Learning. Many techniques in imitation learning use Supervised Learning so you do not need to use emulator. Check &lt;a href=&quot;https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats11-NoRegret.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;DAGGER&lt;/a&gt; which is used in continuous action scenarios or the recent &lt;a href=&quot;https://arxiv.org/pdf/1703.01030.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;AggreVated&lt;/a&gt; algorithm (just ignore the theoretical parts of the paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a start you can use Supervised Learning just for experimentation and then use the above algorithms. I would suggest though to use even a poor simulator just to have an idea of how your implementations behave after the. Bare in mind that RL tries to solve an optimization problem (maximize an utility/cost function) whereas the Supervised Learning methods try to optimize the difference between model's prediction and ground truth. Just be cautious of the algorithm's behavior before you go live.&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-04-15T17:40:24.937" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30349" PostTypeId="2" ParentId="12585" CreationDate="2018-04-15T19:10:39.397" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;So I guess that if I train a GloVe I could use it just for finding synonyms, or something more?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;GloVe (vectors) are not the perfect word representations if you were to find synonyms. &#xA;Since GloVe vectors are generated on the basis of the contexts in which they appear (context implies words before and after the word in study within a sentence), you may find antonyms will also have similar GloVe vectors since they too will appear in the same context. For eg.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the two words &quot;good&quot; and &quot;bad&quot; used in the following sentences:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;John was a good boy who was injured in the war.&#xA;John was a bad boy who was injured in the war. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, both the words &quot;good&quot; and &quot;bad&quot; appear in the same context. The context before is &quot;John was a&quot; and the context after is &quot; who was injured in the war.&quot;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, their GloVe vectors won't have much of a distinction and you might change the meaning of your sentence completely when spinning for a new word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To address the above, a very good alternative to GloVe vectors is &quot;retro-fit vectors&quot;. &lt;a href=&quot;https://www.cs.cmu.edu/~hovy/papers/15HLT-retrofitting-word-vectors.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here is the link of the original article.&lt;/a&gt; and &lt;a href=&quot;https://github.com/mfaruqui/retrofitting&quot; rel=&quot;nofollow noreferrer&quot;&gt;here is the GitHub link of the actual vectors.&lt;/a&gt; since they are learned for semantic similarity. I suggest using these retro-fit vectors since you seem to be interested in lexical substitution of synonyms. &lt;/p&gt;&#xA;" OwnerUserId="3357" LastEditorUserId="3357" LastEditDate="2018-06-11T08:40:34.637" LastActivityDate="2018-06-11T08:40:34.637" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30350" PostTypeId="1" CreationDate="2018-04-15T20:38:13.463" Score="3" ViewCount="95" Body="&lt;p&gt;Consider the equations relating to the diagonal approximation for the hessian matrix for a neural network in &lt;em&gt;&quot;Pattern Recognition and Machine Learning - Christopher Bishop&quot;&lt;/em&gt; &lt;em&gt;(on pg. 250 eq. 5.80)&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;$\frac{\partial^2 E_{n}}{\partial w_{ji}^2} = \frac{\partial^2 E_{n}}{\partial a_{j}^2} z_{i}^2$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\frac{\partial^2 E_{n}}{\partial a_{j}^2} = ({h}'(a_{j}))^2 \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}} + h''(a_{j})\sum_{k}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;$a_{j}$ - &lt;em&gt;input to a unit j&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h$ - &lt;em&gt;activation function&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$z_i = h(a_i)$ &lt;em&gt;i.e output of a unit i&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w_{kj}$ - &lt;em&gt;edge from j to k&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested in how &lt;em&gt;eq. 2&lt;/em&gt; follows from:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial E_{n}}{\partial a_{j}} = h'(j)\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;where k runs through all the units to which unit j &lt;strong&gt;directly&lt;/strong&gt; sends a connection to (eq 5.56, pg 244).&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, I am getting a different result for the first term on the RHS of &lt;em&gt;eq. 2&lt;/em&gt;, i.e&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;$({h}'(a_{j}))^2 \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So, &lt;em&gt;eq. 3&lt;/em&gt; should follow from:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;4&quot;&gt;&#xA;&lt;li&gt;$h'(j)\frac{\partial }{\partial a_{j}}\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Following is the way how I tried to derive &lt;em&gt;eq. 3&lt;/em&gt; from &lt;em&gt;eq. 4&lt;/em&gt; using the chain rule of differentiation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$h'(a_{j})\frac{\partial }{\partial a_{j}}\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$= h'(a_{j})\sum_{k}\frac{\partial a_{k}}{\partial a_{j}} \frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$=h'(a_{j})\sum_{k}h'(a_{j})w_{kj}\frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Used:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(a_k = \sum_jh(a_j)w_{kj}) \wedge (\frac{\partial w_{kj}}{\partial a_j} = 0 )\rightarrow \frac{\partial a_{k}}{\partial a_j} = h'(j)w_{kj}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what remains to be shown is:&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;5&quot;&gt;&#xA;&lt;li&gt;$\sum_{k}w_{kj}\frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}} = \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}}$&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;which should complete the proof.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;&lt;strong&gt;QUESTION 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to establish &lt;em&gt;eq. 5&lt;/em&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One may want to use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial a_k} {\partial w_{kj}} = z_j$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&#xA;&lt;strong&gt;QUESTION 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$y = f(x_1, x_2,...)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then, is it &lt;em&gt;always&lt;/em&gt; true that&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial y} {\partial x_k} = 1 / \frac{\partial x_k} {\partial y}$&lt;/p&gt;&#xA;" OwnerUserId="50574" LastEditorUserId="50574" LastEditDate="2018-04-15T20:57:51.370" LastActivityDate="2018-04-15T20:57:51.370" Title="Deriving diagonal approximation of Hessian in a neural network" Tags="|machine-learning|neural-network|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30351" PostTypeId="2" ParentId="30344" CreationDate="2018-04-15T22:25:53.807" Score="19" Body="&lt;p&gt;You should also take a look at &lt;a href=&quot;http://ruder.io/optimizing-gradient-descent/&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; post comparing different gradient descent optimizers. As you can see below Adam is clearly not the best optimizer for some tasks as many converge better.&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-15T22:25:53.807" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30352" PostTypeId="1" AcceptedAnswerId="30368" CreationDate="2018-04-15T22:31:44.387" Score="1" ViewCount="374" Body="&lt;p&gt;I have a large amount of categorical and dummy variables (36) and I would like to remove a number of them based on their multicollinearity (or just collinearity). Instead of using Chi Square tests over and over again, are there any functions that can check for (multi)collinearity in my variables and return variables with multicollinearity (or collinearity)?&lt;/p&gt;&#xA;" OwnerUserId="50584" LastActivityDate="2018-05-26T15:54:03.167" Title="Is there a quick way to check for multicollinearity between categorical variables in R?" Tags="|r|logistic-regression|categorical-data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30353" PostTypeId="2" ParentId="30206" CreationDate="2018-04-15T23:09:45.193" Score="0" Body="&lt;p&gt;With some help from the Orange team, I was able to solve the problem!&#xA;&lt;a href=&quot;https://github.com/biolab/orange3/issues/2995&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link to thread on github&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following code can be pasted into a Python Script widget, and will balance your classes before training a classifier in Orange:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Get libraries&#xA;import Orange&#xA;import numpy as np&#xA;from Orange.data import Domain, Table&#xA;from imblearn.over_sampling import SMOTE&#xA;&#xA;# How Orange passes data to widget&#xA;df = in_data.copy()&#xA;&#xA;# set variables for SMOTE&#xA;sm = SMOTE(random_state=42)&#xA;&#xA;# get table of data (X) and class variables (y)&#xA;X, y = df.X, df.Y&#xA;&#xA;# resample data and classes&#xA;X_res, y_res = sm.fit_sample(X, y)&#xA;&#xA;# Get the target and feature variables&#xA;d = Domain(df.domain.attributes, df.domain.class_vars)&#xA;&#xA;# Create a new Orange Table object with the appropriate headers&#xA;# This is how Orange passes the data on to the next widget&#xA;out_data = Orange.data.Table(d, X_res, y_res)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26748" LastActivityDate="2018-04-15T23:09:45.193" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30354" PostTypeId="2" ParentId="30340" CreationDate="2018-04-15T23:32:03.027" Score="3" Body="&lt;p&gt;Actually, there are many linear and non-linear machine learning algorithms. Selecting a right algorithm highly depends on your data-set and the nature of your data. for instance, in machine-learning era, it was customary to estimate functions by assigning a typical model to the problem and reducing the error by predicting the appropriate coefficients using a cost function for regression tasks. In such cases, you should have bias-variance trade-off. That means you should not fit the data nor miss it. you should find a good estimate, a good model that has generalization capability. For doing all of these you have to choose features which help you describe the problem better for making a model. In deep-learning, we usually do not have this trade-off. If you increase the size of your training set, you can almost be sure that you can have better results. In machine-learning, you can always be sure that by making complex non-linear models, you overfit your data while using complex deep-learning models does not necessarily mean that if you employ generalization techniques which avoid overfitting. For choosing a right model, it is customary to use a simple linear model and make it complicated step by step. Problems which their inputs have numerous features, you can not see and visualize the data-set to check whether they are linearly separable in that space or not. Consequently, beginning with a simple model and making it complex step by step is a logical solution. For measuring performance there are different solutions that all depend on your goal. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26855/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-15T23:32:03.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30355" PostTypeId="2" ParentId="30287" CreationDate="2018-04-16T00:25:24.660" Score="1" Body="&lt;p&gt;Hi guys thanks for your comments and sorry for the slow reply. I cannot reply to them directly because I am not allowed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Constantinos, to your point regarding b: b--&gt;b, b--&gt;a being different actions.  I think you misinterpreted what I am asking. There is only one action (what you call it is arbitrary) but two possible outcomes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@Neil Slater, Yes I think the question is badly written. An MRP is defined as an MDP with only one possible action in each state (or no action dependant on how you look at it). Reward is a real value that is the output of a function that maps a pair of states (so say (a,b) or (b,b) etc ) to a real value. Discount is defined similarly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Really the crux of the question is: find a linear function approximation parameter $\lambda$ such that $\lambda* \phi(s) \approx v(s)$ where $v(s)$ is the value associated with a certain state $s$. I believe that such values are defined as $v(s) = \sum_{s'}P(s,s')(R(s,s') + \gamma v(s'))$ where $R(.,.)$ is a reward transition function and $P(.,.)$ is a function mapping state pairs to probabilities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the main reason I am lost with this is because the question itself is pretty ambiguous so I am trying to fill in the gaps heuristically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks Guys&lt;/p&gt;&#xA;" OwnerUserId="50586" LastActivityDate="2018-04-16T00:25:24.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30356" PostTypeId="1" AcceptedAnswerId="30359" CreationDate="2018-04-16T01:21:22.073" Score="2" ViewCount="678" Body="&lt;p&gt;I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to try some primitive batching but in order to one hot encode I need to find number of all unique values, which i can't do without loading data into a dataframe first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other tools that I can use to attempt loading the file into a dataframe? Does Pyspark also have a limit of when it starts crashing? I know that its capable of breaking down operations into stages, does that help with RAM management or just execution?&lt;/p&gt;&#xA;" OwnerUserId="40123" LastActivityDate="2018-04-16T03:48:35.793" Title="How to load a csv file into [Pandas] dataframe if computer runs out of RAM?" Tags="|pandas|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30357" PostTypeId="1" AcceptedAnswerId="30360" CreationDate="2018-04-16T01:46:47.950" Score="1" ViewCount="185" Body="&lt;p&gt;Which and all Machine Learning algorithms needs the data to be standardised/normalised before feeding into the model. How do we determine whether the particular model/data needs to be  standardised/normalised.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-08-14T07:01:54.423" Title="Standardization and Normalization" Tags="|classification|regression|normalization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30358" PostTypeId="1" CreationDate="2018-04-16T03:42:49.250" Score="1" ViewCount="929" Body="&lt;p&gt;I want to predict the probability that an individual credit_balance is more than value N is true&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to compare 3 classification methods: Logistic regression, Least discriminant and quadratic discriminant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For 1 sample prediction I get &lt;code&gt;predict_proba&lt;/code&gt; for each of them with different values, how can I decide which model is the best for my prediction?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;linear regression &lt;code&gt;predict_proba&lt;/code&gt; result is &lt;code&gt;[[0.93227393 0.06772607]]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;LDA &lt;code&gt;predict_proba&lt;/code&gt; is &lt;code&gt;[[0.94144572 0.05855428]]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;QDA &lt;code&gt;precit_proba&lt;/code&gt; is &lt;code&gt;[[9.99999999e-01 1.24419207e-09]]&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What parameters should we look at to decide which classification is the best to predict the model?&lt;/p&gt;&#xA;" OwnerUserId="50544" LastEditorUserId="26562" LastEditDate="2018-04-16T04:30:13.673" LastActivityDate="2018-04-16T15:27:14.230" Title="predict_proba compared" Tags="|classification|linear-regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30359" PostTypeId="2" ParentId="30356" CreationDate="2018-04-16T03:48:35.793" Score="3" Body="&lt;p&gt;As noted many times by the writers of pandas, the ideal size of memory for analyzing with pandas is around 5-10 times the load of data you are giving. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said, if you can afford to load data is chunks for pre-processing and dump only the columns that are needed, I recommended &lt;code&gt;pandas.read_table&lt;/code&gt; option to load data in pieces (check option &lt;code&gt;chunks&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;pyspark&lt;/code&gt; as you mentioned might be good to go with. But there is &lt;code&gt;dask&lt;/code&gt; which is built around &lt;code&gt;python&lt;/code&gt; stack of &lt;code&gt;pandas&lt;/code&gt; and &lt;code&gt;numpy&lt;/code&gt; for distributed work loads which has support for &lt;code&gt;pandas.DataFrame&lt;/code&gt; and &lt;code&gt;numpy.array&lt;/code&gt; . But I never followed anyone who was successful in using it in production (or atleast mentioned that they had used it). May be some of the people here can vouch for it. There is another library called &lt;code&gt;sframe&lt;/code&gt; to load the data way out of memory as the author suggested in one of his &lt;a href=&quot;https://www.youtube.com/watch?v=2QZu1JRqmDk&amp;amp;t=2042s&quot; rel=&quot;nofollow noreferrer&quot;&gt;presentations&lt;/a&gt;. It keeps on serializing data onto disk. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope I summarized it well.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-04-16T03:48:35.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30360" PostTypeId="2" ParentId="30357" CreationDate="2018-04-16T03:51:55.220" Score="1" Body="&lt;p&gt;Whenever you have features that they have different scale and it is significant for some features, you should standardize your feature. Take a look at &lt;a href=&quot;https://stats.stackexchange.com/q/207108/179078&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-16T03:51:55.220" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30362" PostTypeId="1" AcceptedAnswerId="31948" CreationDate="2018-04-16T06:03:09.990" Score="6" ViewCount="3276" Body="&lt;p&gt;An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked like they need to be checked in ARIMA model.Do we have to check for it in LSTM??&lt;/p&gt;&#xA;" OwnerUserId="50595" LastActivityDate="2018-05-21T19:10:04.850" Title="Checking for stationarity in LSTM" Tags="|python|rnn|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30363" PostTypeId="2" ParentId="30233" CreationDate="2018-04-16T06:32:45.297" Score="1" Body="&lt;p&gt;The representation power of the models is different.&#xA;Consider the case of XOR for many features.&#xA;Assume the all the features are binary, just randomly sampled fro B(0.5).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A decision tree will be able to represent that concept, in &lt;a href=&quot;http://www.chegg.com/homework-help/decision-graph-generalization-decision-tree-allows-nodes-e-a-chapter-18-problem-7e-solution-9780136042594-exc&quot; rel=&quot;nofollow noreferrer&quot;&gt;exponential size&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any of the features alone is useless (or misleading).&#xA;A decision stump is restricted to a single feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if the decision stumps will choose the right features, one cannot represent XOR &lt;a href=&quot;https://crypto.stackexchange.com/questions/12634/reason-why-xor-is-a-linear-operation-but-ordinary-addition-isn-t&quot;&gt;using a linear weight of the features&lt;/a&gt;.&#xA;In simpler words, the reason the XOR is not linear is that there is no weight w such that&#xA;a xor b = w*a + (1-w)*b.&lt;/p&gt;&#xA;" OwnerUserId="13727" LastEditorUserId="13727" LastEditDate="2018-04-16T09:01:31.900" LastActivityDate="2018-04-16T09:01:31.900" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="30364" PostTypeId="1" CreationDate="2018-04-16T07:10:30.160" Score="1" ViewCount="917" Body="&lt;p&gt;I have a set of 20 observations, the target variable which is a binary variable (0 or 1), in this set has one unique value which is 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to predict this variable for a large sample, so I want to use a regression algorithm. The problem is that the target variable take one value, there is no observation that can describe the 0 class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any solution to handle this situation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance. &lt;/p&gt;&#xA;" OwnerUserId="50599" LastEditorUserId="40853" LastEditDate="2018-04-16T07:55:17.817" LastActivityDate="2020-04-03T14:41:49.443" Title="one single value target variable" Tags="|machine-learning|regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30365" PostTypeId="2" ParentId="30364" CreationDate="2018-04-16T07:53:20.610" Score="1" Body="&lt;p&gt;Welcome to the site! As Media has mentioned it is a classification problem not prediction problem. If you meant Logistic Regression then it makes sense. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to remember one thing if the outcome is only one value then algorithm would also return the same value irrespective of anything. if you don't have negative cases then you cannot use Logistic regression and it doesn't know when to say 0 as the model wasn't trained on it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If possible using Business Matter Experts advice populate some dummy data for 0's and balance the data. Train your model and see what the model is going to say.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Else you cannot use any kind of classification algorithm. &lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-04-16T07:53:20.610" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30366" PostTypeId="1" CreationDate="2018-04-16T08:07:00.763" Score="11" ViewCount="149" Body="&lt;p&gt;My pursuit is to generate something like a grottesque(a kind of painting producing human-animals and plants hybrids).  I need to do something like this painting in order to create an art exhibition. I don't need that produced images are ordinate or beautiful, just monstrous images produced automatically by a computer. Can someone help me? And is it possible to do it in a month?&lt;/p&gt;&#xA;" OwnerUserId="50602" LastEditorUserId="1330" LastEditDate="2021-03-17T20:45:31.893" LastActivityDate="2021-03-30T20:10:19.083" Title="Automatically generating images for an art show with deep learning software" Tags="|neural-network|deep-learning|algorithms|gan|" AnswerCount="2" CommentCount="1" ClosedDate="2021-04-05T09:40:12.893" ContentLicense="CC BY-SA 4.0" />
  <row Id="30367" PostTypeId="1" CreationDate="2018-04-16T08:36:33.490" Score="3" ViewCount="272" Body="&lt;p&gt;After reinstalling Keras and Tensorflow on a Virtual Machine, I have noticed that models fail to train correctly (at the very least those I have checked). &#xA;For example, taking this &lt;a href=&quot;https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt; from the &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras blog&lt;/a&gt; as a test case, instead of decreasing loss and final validation accuracy of ~0.8, loss quickly increased to ~7.5 and accuracy stayed at ~0.5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It should be noted that when I tried it with older versions of Keras and Tensorflow (on the same computer), I did not observe this aberrant behavior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have repeated this experiment at three other setups in which the same behavior was observed - a two other computers using a VM, and a third computer, this time a laptop with no suitable GPU. &#xA;On the other hand, trying this on two computers with GPUs, the training progressed as expected, with loss decreasing and accuracy increasing to the level advertised in the Keras blog and observed with the previous versions of Keras and Tensorflow while using a CPU.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the evidence I can speculate that either Keras's or Tensorflow's interaction with the CPU was changed in a manner disrupting the training processes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone here faced a similar problem? &#xA;What can be done to solve this?     &lt;/p&gt;&#xA;" OwnerUserId="43472" LastActivityDate="2018-04-24T00:38:06.987" Title="Strange behavior in current Keras setting (tensorflow-1.7.0 Keras-2.1.5) when using CPU" Tags="|keras|tensorflow|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30368" PostTypeId="2" ParentId="30352" CreationDate="2018-04-16T09:14:13.060" Score="2" Body="&lt;p&gt;In my work I usually use Normalized Mutual Information (NMI) to get an understanding of how &quot;correlated&quot; two categorical variables are.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normalized Mutual Information is an information-theoretic measure that tells you how much information is shared by two variables. If NMI is close to 1, the two variables are very &quot;correlated&quot;, while if NMI is close to 0 the two variables are &quot;uncorrelated&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wrote this function that computes the NMI between the first two variables in a data.table. It is quite fast thanks to data.table. Feel free to use it!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;compute_mutualinfo &amp;lt;- function(df){&#xA;  require(data.table)&#xA;  require(entropy)&#xA;  var1 = names(df)[1]&#xA;  var2 = names(df)[2]&#xA;  tmp_tab &amp;lt;- df[,.N,by=c(var1,var2)]&#xA;  names(tmp_tab)[1:2] &amp;lt;- c('V1','V2')&#xA;  cross_tab &amp;lt;- tapply(tmp_tab$N,list(addNA(as.factor(tmp_tab$V2),ifany = T), addNA(as.factor(tmp_tab$V1),ifany = T)), sum)&#xA;  cross_tab[is.na(cross_tab)] &amp;lt;- 0&#xA;  mi.plugin(cross_tab)/sqrt(entropy(df[,.N,by=var1,with=T][,N])*entropy(df[,.N,by=var2,with=T][,N]))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50211" LastActivityDate="2018-04-16T09:14:13.060" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30369" PostTypeId="1" AcceptedAnswerId="30493" CreationDate="2018-04-16T09:35:53.563" Score="4" ViewCount="790" Body="&lt;p&gt;I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First: Why do we need to add alpha and gamma every time we sample the equation below? What if we delete the alpha and gamma from the equation? Would it still be possible to get the result? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Lp5ve.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Lp5ve.png&quot; alt=&quot;LDA sampling formular&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Second: In LDA, we randomly assign a topic to every word in the document. Then, we try to optimize the topic by observing the data. Where is the part which is related to posterior inference in the equation above?&lt;/p&gt;&#xA;" OwnerUserId="50605" LastActivityDate="2018-04-18T21:10:07.740" Title="Why do we need the hyperparameters beta and alpha in LDA?" Tags="|topic-model|lda|hyperparameter|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30370" PostTypeId="1" AcceptedAnswerId="30382" CreationDate="2018-04-16T09:41:06.617" Score="6" ViewCount="12274" Body="&lt;p&gt;Suppose we want to predict what customer will buy during his next visit to the Electronic Shop based on his past purchase history. I know that it is a very broad question, but I am new to machine learning and don't have much idea about how to approach this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest thing that comes to my mind is to find the most frequent items that customer has bought and suggest it. However, I don't think that this is a very robust approach as it doesn't consider this scenario:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Computer (1st Purchase) -&amp;gt; Mouse (2nd Purchase) -&amp;gt; Mouse Pad (3rd Purchase)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am looking for a simple model to get started and scale in terms of features and training data. I would love to hear suggestions of experienced Data Scientist as it is a most common problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50603" LastEditorUserId="29169" LastEditDate="2019-06-07T06:17:02.540" LastActivityDate="2019-06-07T06:17:02.540" Title="How to predict customer's next purchase" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30371" PostTypeId="2" ParentId="30358" CreationDate="2018-04-16T10:01:17.150" Score="2" Body="&lt;p&gt;You need to take a step back in order to decide which model would suit best for your use case. Before doing that predict_proba is definitely important to calculate the posterior probability of the class labels, but it isn't great for comparing against other model outputs (especially to decide which models suits best for your prediction).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logistic Regression, QDA and LDA all have different approaches. Logistic regression is based on maximum likelihood estimation while LDA and QDA are based on Bayes theorem. To understand which classifier best suits your model we need to go over the assumptions (assuming you know the mathematical expressions) and then you can judge which best apply to you.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. Logistic Regression&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Logistic regression, it is possible to directly get the probability of an observation for a class (Y=k) for a particular observation (X=x). There is nothing to assume to run logistic regression for classification. Its generally a safe go to method which is not exigent and is robust.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2. LDA &amp;amp; QDA&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LDA and QDA algorithm is based on Bayes theorem and classification of an observation is done in following two steps.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Identify the distribution for input X for each of the class (or groups ex Y=k1, k2, k3 etc )&lt;/li&gt;&#xA;&lt;li&gt;Flip the distribution using Bayes theorem to calculate the probability Pr(Y=k|X=x)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Following are the assumption required for LDA and QDA:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;LDA Assumption:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Common covariance across all response classes σ2 ( for ex σk1 = σk2 =&#xA;σk3 for k1, k2 , k3 response classes )&lt;/li&gt;&#xA;&lt;li&gt;Distribution of observation in each of the response classes is normal&#xA;with a class-specific mean (µk) and common covariance σ.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;QDA Assumption:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Different covariance for each of the response classes. For ex – σk1,&#xA;σk2, σk3 for response class k1, k2, k3 etc.&lt;/li&gt;&#xA;&lt;li&gt;Distribution of observation in each of the response class is normal&#xA;with a class-specific mean (µk) and class-specific covariance (σk2).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Notes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LDA (Linear Discriminant Analysis) is used when a linear boundary is required between classifiers.&lt;/li&gt;&#xA;&lt;li&gt;QDA (Quadratic Discriminant Analysis) is used to find a non-linear boundary between classifiers.&lt;/li&gt;&#xA;&lt;li&gt;LDA/QDA, when all its requirements met classifies better than logistic regression (more efficient).&lt;/li&gt;&#xA;&lt;li&gt;Logistic regression is not sensitive to outliers while LDA/QDA is.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To conclude:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LDA and QDA work well when class separation and normality assumption holds.&lt;/li&gt;&#xA;&lt;li&gt;Logistic regression has an edge on LDA/QDA for dataset that is not normal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="32958" LastEditorUserId="32958" LastEditDate="2018-04-16T15:27:14.230" LastActivityDate="2018-04-16T15:27:14.230" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30374" PostTypeId="1" CreationDate="2018-04-16T11:45:17.520" Score="2" ViewCount="1116" Body="&lt;p&gt;I want to emphasise (increase the weight) of only a subset of data. Lets say I have old and fresh data, I would like to say that old data has to have more weight and therefore has more influence in the decision than the new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In scikit-learn I found only &lt;code&gt;class-weight&lt;/code&gt; parameter, but it does not change the weight of the samples, only of all samples within the class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to incorporate this emphasis into the gradient boosted trees in spark or xgboost in python? &lt;/p&gt;&#xA;" OwnerUserId="33501" LastActivityDate="2019-04-14T04:04:20.997" Title="emphasise some observation weights more than the others" Tags="|weighted-data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30376" PostTypeId="1" CreationDate="2018-04-16T13:03:56.417" Score="10" ViewCount="23565" Body="&lt;p&gt;I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there other methods?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My search results came up with a solution&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['col']=(df['col']-df['col'].min())/(df['col'].max()-df['col'].min())&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this works only with panda dataframes (they have column names). Is there a way to implement column headers in the matrix?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import random&#xA;import sklearn.preprocessing import StandardScaler&#xA;&#xA;data = pd.DataFrame({'sepal_length': [3.4, 4.5, 3.5], &#xA;                     'sepal_width': [1.2, 1, 2],&#xA;                'petal_length': [5.5, 4.5, 4.7],&#xA;                'petal_width': [1.2, 1, 3],&#xA;                    'species': ['setosa', 'verginica', 'setosa']})&#xA;&#xA;#Shuffle the data and reset the index&#xA;from sklearn.utils import shuffle&#xA;data = shuffle(data).reset_index(drop = True)  &#xA;&#xA;#Create Independent and dependent matrices&#xA;X = data.iloc[:, [0, 1, 2, 3]].values &#xA;y = data.iloc[:, 4].values&#xA;&#xA;#train_test_split&#xA;from sklearn.cross_validation import train_test_split&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1, random_state = 0)&#xA;&#xA;&#xA;#Impute missing values at random&#xA;prop = int(X_train.size * 0.5) #Set the % of values to be replaced&#xA;prop1 = int(X_test.size * 0.5)&#xA;&#xA;a = [random.choice(range(X_train.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array&#xA;b = [random.choice(range(X_train.shape[1])) for _ in range(prop)]&#xA;c = [random.choice(range(X_test.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array&#xA;d = [random.choice(range(X_test.shape[1])) for _ in range(prop)]&#xA;X_train[a, b] = np.NaN&#xA;X_test[c, d] = np.NaN&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is where I get the error :&lt;strong&gt;Input contains NaN, infinity or a value too large for dtype('float64').&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;from sklearn.preprocessing import StandardScaler #importing the library that does feature scaling&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sc_X = StandardScaler() # created an object with the scaling class&#xA;&#xA;X_train = sc_X.fit_transform(X_train)  # Here we fit and transform the X_train matrix&#xA;X_test = sc_X.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50449" LastEditorUserId="55122" LastEditDate="2020-08-04T15:28:28.793" LastActivityDate="2020-08-04T16:27:35.100" Title="How do we standardize arrays with NaN?" Tags="|machine-learning|python|scikit-learn|normalization|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30377" PostTypeId="1" CreationDate="2018-04-16T13:25:40.237" Score="7" ViewCount="271" Body="&lt;p&gt;Let's say we want to predict the probability of rain. So just the binary case: rain or no rain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this will be enough. And it is actually desired to make the classifier not too confident. Hence cross entropy (CE) is chosen:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But cross entropy practically makes it very hard for the classifier to learn to predict 0. Is there another objective function that does not behave that extreme around 0?&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Why it matters&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;There might be cases where it is possible to give the prediction 0% (or at least something much closer to 0 like $10^{-6}$). Like in a desert. And there might be applications where one needs this (close to) zero predictions. For example, when you want to predict the probability that something happens at least once. If the classifier always predicts at least a 1% chance, then having rain at least once in 15 days is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - (1-0.05)^{15} \approx 54\%$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but if the classifier can &lt;em&gt;practically&lt;/em&gt; output 0.1% as well, then this is only&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$1 - (1-0.001)^{15} \approx 1.5\%$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could also imagine this to be important for medical tests or for videos.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-04-16T13:46:39.517" LastActivityDate="2023-06-01T09:01:40.200" Title="What is a good objective function for allowing close to 0 predictions?" Tags="|xgboost|optimization|objective-function|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30378" PostTypeId="2" ParentId="14648" CreationDate="2018-04-16T13:40:06.177" Score="2" Body="&lt;p&gt;This would be easier if you have multiple columns:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pyspark.sql.functions import when   &#xA;cols = df.columns # list of all columns&#xA;for col in cols:&#xA;    df= df.withColumn(col, when(df[col]&amp;gt;0,1).otherwise(0))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17000" LastEditorUserId="17000" LastEditDate="2018-04-17T01:49:01.537" LastActivityDate="2018-04-17T01:49:01.537" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30379" PostTypeId="2" ParentId="30376" CreationDate="2018-04-16T13:54:04.340" Score="3" Body="&lt;p&gt;You can use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.preprocessing.Imputer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import datasets as ds&#xA;from sklearn.model_selection import train_test_split&#xA;&#xA;# load Iris data set    &#xA;data = ds.load_iris()&#xA;&#xA;X = data.data&#xA;y = data.target&#xA;&#xA;# artificially set 33% of [X] data set to NaN's    &#xA;X.ravel()[np.random.choice(X.size, int(X.shape[0]*.33), replace=False)] = np.nan&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [137]: X&#xA;Out[137]:&#xA;array([[5.1, 3.5, nan, nan],&#xA;       [nan, 3. , 1.4, 0.2],&#xA;       [4.7, nan, 1.3, 0.2],&#xA;       ...,&#xA;       [6.5, 3. , nan, 2. ],&#xA;       [6.2, 3.4, 5.4, 2.3],&#xA;       [5.9, 3. , nan, 1.8]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now we can impute and standardize it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;imp = Imputer(strategy=&quot;mean&quot;, axis=0)&#xA;scale = StandardScaler()&#xA;&#xA;In [139]: X_new = scale.fit_transform(imp.fit_transform(X))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [160]: X_new&#xA;Out[160]:&#xA;array([[-1.03733263e+00,  1.22587069e+00, -1.37398311e-15, -3.17837019e-16],&#xA;       [ 1.18191646e-15, -5.32987255e-02, -1.43399195e+00, -1.35522269e+00],&#xA;       [-1.56962048e+00,  2.27226133e-15, -1.49587065e+00, -1.35522269e+00],&#xA;       ...,&#xA;       [ 8.25674859e-01, -5.32987255e-02, -1.37398311e-15,  1.22131653e+00],&#xA;       [ 4.26458969e-01,  9.70036804e-01,  1.04115598e+00,  1.65073974e+00],&#xA;       [ 2.72430791e-02, -5.32987255e-02, -1.37398311e-15,  9.35034396e-01]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Demo2, using Pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import Imputer, StandardScaler&#xA;from sklearn.pipeline import Pipeline&#xA;&#xA;#...    &#xA;&#xA;estimator = Pipeline([(&quot;impute&quot;, Imputer(strategy=&quot;mean&quot;, axis=0)),&#xA;                      (&quot;scale&quot;, StandardScaler()),&#xA;                      (&quot;forest&quot;, RandomForestRegressor(random_state=0,&#xA;                                                       n_estimators=100))])&#xA;&#xA;estimator.fit(X_train, y_train)&#xA;#...    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastEditorUserId="20492" LastEditDate="2018-04-16T18:29:58.763" LastActivityDate="2018-04-16T18:29:58.763" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30380" PostTypeId="2" ParentId="30376" CreationDate="2018-04-16T13:55:07.980" Score="1" Body="&lt;p&gt;Working with NaNs is always a bit difficult. Maybe it would be useful if you try to enrich NaN values. For example, by averaging the considered feature for groups like an age class. If only a few records have NaN values, you might simply drop these (pandas dropna).&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-16T13:55:07.980" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30381" PostTypeId="1" CreationDate="2018-04-16T13:58:55.280" Score="1" ViewCount="36" Body="&lt;p&gt;I am in an ML course and one of our tasks is to predict the helpfulness of Amazon reviews.  Currently, I am doing what most people seem to .  That is, a hashvectorizer(2-gram) on the text, tfidf, several scalar features, and (in my case) a Ridge classifier (parameters chosen by grid search).&#xA;  Honestly, it has been frustrating since I have tried to read and apply best practice from the course and any published works I can find.  That said, I am still only achieving marginal results:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'Pos': 6588, 'Neg': 84412, 'TP': 2755, 'TN': 78990, 'FP': 5422, 'FN': 3833, 'Accuracy': 0.8982967032967033, 'Precision': 0.33692063103827807, 'Recall': 0.41818457802064357}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;...which isn't great. I don't need help coding*, but if anyone can suggest useful case studies, or other sklearn classifiers/techniques I could research, I would be grateful.  I'm rapidly running out of directions to follow up on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if relevant, you can find my latest iteration &lt;a href=&quot;http://github.com/wolfm2/jobout/blob/master/minibook/amazon.2.1/output.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="50619" LastEditorUserId="40853" LastEditDate="2018-04-16T16:57:49.583" LastActivityDate="2018-04-26T13:44:44.240" Title="Determining helpful Amazon feedback" Tags="|text-mining|feature-selection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30382" PostTypeId="2" ParentId="30370" CreationDate="2018-04-16T14:00:08.563" Score="4" Body="&lt;p&gt;Take a look at association rule learning (&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Association_rule_learning&lt;/a&gt;). A really common algorithm is the Apriori agorithm. You could use the package apyori, it works great: &lt;a href=&quot;https://pypi.python.org/pypi/apyori/1.1.1&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pypi.python.org/pypi/apyori/1.1.1&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-16T14:00:08.563" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30383" PostTypeId="1" CreationDate="2018-04-16T14:33:25.563" Score="2" ViewCount="607" Body="&lt;p&gt;I have a dataset with both numerical and categorical features (variables), I converted all the categorical variables in dummies, then I split the train and test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am at the step where I want to standardize the features before fitting the model.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Should I apply the standardisation to all the features or only to the numerical ones? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In this case, is it preferable to use the MinMax scaler on a range 0 to 1? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50623" LastEditorUserId="29575" LastEditDate="2018-04-17T00:10:10.123" LastActivityDate="2018-06-16T00:19:32.550" Title="standardize dataset with numerical and dummy features" Tags="|machine-learning|preprocessing|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30384" PostTypeId="1" AcceptedAnswerId="30392" CreationDate="2018-04-16T15:32:25.740" Score="6" ViewCount="638" Body="&lt;p&gt;I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to &lt;strong&gt;&lt;em&gt;segment the scripts into parts&lt;/em&gt;&lt;/strong&gt; such that each part falls into one of the following categories: &lt;br&gt;&#xA;&lt;code&gt;MainConceptDescription&lt;/code&gt;-&gt; Explanation of the Main concept(s)&lt;br&gt;&#xA;&lt;code&gt;SubConceptDescription&lt;/code&gt;-&gt; Explanation of Subconcept related to the main concept&lt;br&gt;&#xA;&lt;code&gt;Methodology / Technique&lt;/code&gt;-&gt; To achieve something, what should one do&lt;br&gt;&#xA;&lt;code&gt;Summary&lt;/code&gt;-&gt; Summary of the discussed material or of the whole course&lt;br&gt;&#xA;&lt;code&gt;Application&lt;/code&gt;-&gt; Practical advise for the concept&lt;br&gt;&#xA;&lt;code&gt;Example&lt;/code&gt;-&gt; Concept example&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, for the first 2 i think I should try to apply &lt;strong&gt;Latent Dirichlet Allocation&lt;/strong&gt; to extract the topics. Another idea was to look in the resource name and search for these words in the text. Another idea was to read some of the resources and manually fix some sort of a dictionary for every category and then create regex patterns and search for them in the text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the latter seems too lame. So now I am not sure what can I do. I've seen similar works for research papers, however research papers have their own specific expressions etc that are more or less constant and seen in most of the papers, but that is not the case with my video scripts where it's 100% spoken natural language I need to work on. Do you have any ideas how can I approach this? I do have a list of keywords that kind of signifies whether an example follows or a concept is explained, but I am doing this manually which is surely not what I want to do for 563 files that may as well become many more. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I'd like to connect topics found to &lt;strong&gt;ontologies&lt;/strong&gt; in order to enrich the metadata about each file. I have no idea how can I approach this either. Any advice would be VERY appreciated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forgive me if my explanations make no sense. I am not too familiar with the terminology. So if you also explain some of the terminology you use, i'd appreciate that too. And please advise on &lt;strong&gt;algorithms&lt;/strong&gt; I can try with. &lt;/p&gt;&#xA;" OwnerUserId="50625" LastEditorUserId="50625" LastEditDate="2018-04-16T18:42:40.183" LastActivityDate="2018-04-16T18:54:42.150" Title="How to split natural language script into segments?" Tags="|python|data-mining|topic-model|lda|processing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30385" PostTypeId="1" AcceptedAnswerId="30422" CreationDate="2018-04-16T16:14:13.553" Score="0" ViewCount="2117" Body="&lt;p&gt;I attempted to merge a VGG-16 and ResNet-50 model in Keras to benefit from the combined feature representations toward a binary classification task. I was successful in building and saving the merged model. However, while training, I am getting an error. Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#load libraries&#xA;import numpy as np&#xA;from keras import applications&#xA;from keras.layers import GlobalAveragePooling2D, Dense&#xA;from keras.layers import Concatenate&#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from sklearn.metrics import log_loss&#xA;from keras.models import Model&#xA;from keras.optimizers import SGD&#xA;from sklearn.metrics import roc_curve, auc&#xA;from sklearn.metrics import classification_report,confusion_matrix,accuracy_score&#xA;import matplotlib.pyplot as plt&#xA;from sklearn.preprocessing import label_binarize&#xA;from evaluation import plot_confusion_matrix&#xA;from keras.models import load_model&#xA;from scipy import interp&#xA;from itertools import cycle&#xA;###############################################################################&#xA;#image dimensions and loading&#xA;img_width, img_height = 1024,1024&#xA;train_data_dir = 'china_whole/train'&#xA;validation_data_dir = 'china_whole/test'&#xA;epochs = 30 &#xA;batch_size = 2 &#xA;num_classes= 2&#xA;###############################################################################&#xA;''' Building a merge of two models'''&#xA;base_model1 = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_width,img_height,3))&#xA;#get the model summary&#xA;base_model1.summary()&#xA;#addind the top layers&#xA;x1 = base_model1.output&#xA;x1 = GlobalAveragePooling2D()(x1) #analogous to flatten()&#xA;model1 = Model(inputs=base_model1.input, outputs=x1)&#xA;model1.summary()&#xA;#load the second model ########################################################&#xA;base_model2 = applications.ResNet50(weights='imagenet', include_top=False, input_shape=(img_width,img_height,3))&#xA;#get the model summary&#xA;base_model2.summary()&#xA;#addind the top layers&#xA;x2 = base_model2.output&#xA;x2 = GlobalAveragePooling2D()(x2) #analogous to flatten()&#xA;model2 = Model(inputs=base_model2.input, outputs=x2)&#xA;model2.summary()&#xA;###############################################################################&#xA;'''merge the models'''&#xA;mergedOut = Concatenate()([model1.output,model2.output]) #this is concatenating the two GAP layers&#xA;#add a new dense layer and softmax&#xA;out=Dense(2048, activation='relu')(mergedOut) &#xA;out = Dense(num_classes, activation='softmax', name='predictions')(out)&#xA;#create the new model with three branches and one dense layer&#xA;model = Model(inputs=[model1.input,model2.input], outputs=out)&#xA;model.summary()&#xA;#fix the optimizer&#xA;sgd = SGD(lr=0.0001, decay=1e-6, momentum=0.9, nesterov=True) &#xA;#the values are computed by a randomized gird search method&#xA;#compile the gpu model&#xA;model.compile(optimizer=sgd,&#xA;              loss='categorical_crossentropy',&#xA;              metrics=['accuracy'])&#xA;###############################################################################&#xA;#declaring image data generators&#xA;train_datagen = ImageDataGenerator(rotation_range=2.0,rescale=1./255)&#xA;test_datagen = ImageDataGenerator(rescale=1./255)&#xA;train_generator = train_datagen.flow_from_directory(&#xA;    train_data_dir,&#xA;    target_size=(img_height, img_width),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical')&#xA;validation_generator = test_datagen.flow_from_directory(&#xA;    validation_data_dir,&#xA;    target_size=(img_height, img_width),&#xA;    batch_size=batch_size,&#xA;    class_mode='categorical',&#xA;    shuffle=False)&#xA;nb_train_samples = len(train_generator.filenames)&#xA;nb_validation_samples = len(validation_generator.filenames)&#xA;#check the class indices&#xA;train_generator.class_indices&#xA;validation_generator.class_indices&#xA;###############################################################################&#xA;#start training&#xA;&#xA;model.fit_generator(&#xA;    train_generator,&#xA;    steps_per_epoch=nb_train_samples // batch_size,&#xA;    epochs=epochs,&#xA;    validation_data=validation_generator,&#xA;    validation_steps=nb_validation_samples // batch_size,verbose=1, workers=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error when running model.fit_generator was:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;&#xA;  File &quot;&amp;lt;ipython-input-62-ab7292930748&amp;gt;&quot;, line 6, in &amp;lt;module&amp;gt;&#xA;    validation_steps=nb_validation_samples // batch_size,verbose=1, workers=1)&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\keras\legacy\interfaces.py&quot;, line 91, in wrapper&#xA;    return func(*args, **kwargs)&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py&quot;, line 2224, in fit_generator&#xA;    class_weight=class_weight)&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py&quot;, line 1877, in train_on_batch&#xA;    class_weight=class_weight)&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py&quot;, line 1476, in _standardize_user_data&#xA;    exception_prefix='input')&#xA;&#xA;  File &quot;C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py&quot;, line 86, in _standardize_input_data&#xA;    str(len(data)) + ' arrays: ' + str(data)[:200] + '...')&#xA;&#xA;ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[0.17478053, 0.17478053, 0.17478053],&#xA;         [0.17482202, 0.17482202, 0.17482202],&#xA;         [0.17394349, 0.17394349, 0.17394349],&#xA;         ...,&#xA;         [0.15611881, 0.15611881, 0.15611881...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can i resolve this issue in the code?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastActivityDate="2019-04-29T14:29:55.527" Title="Error in training a merged model in Keras" Tags="|keras|image-classification|prediction|training|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30386" PostTypeId="2" ParentId="30381" CreationDate="2018-04-16T16:47:54.777" Score="0" Body="&lt;p&gt;If you want to stick with vanilla Machine Learning, SVMs hav been known to work really well on text classification problems. You can experiment with various kernels. Alternatively, you can try neural networks with word embeddings.  However from your evaluation metrics, it seems you have a data imbalance problem, hence the low precision and recall. Accuracy becomes completely meaningless here. Before you try anything else, I would suggest that you try to balance your samples by getting more data, if that doesnt seem to be possible then try bagging. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastEditorUserId="21608" LastEditDate="2018-04-21T03:06:39.363" LastActivityDate="2018-04-21T03:06:39.363" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30387" PostTypeId="2" ParentId="30233" CreationDate="2018-04-16T17:08:02.203" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_stump&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision stump&lt;/a&gt; are decision trees with one step from root to leaves whereas &lt;a href=&quot;https://en.wikipedia.org/wiki/Decision_tree_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decision trees&lt;/a&gt; can have several steps between root and leaves.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Easy example of these two is that a decision stump could be which side of coin faces up when thrown and a decision tree would be that if the coin could is touching the ground already (states are interconnected):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    stump                         tree&#xA;Is the coin thrown       Is it touching ground&#xA;        |                            |&#xA;  50         50           can turn?       lays still?&#xA;                               |              |&#xA;                      to right   to left    0    0&#xA;                            |         |&#xA;                          0  100   100 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Boosting can't help if decision tree in my example exactly knows &lt;strong&gt;before decision&lt;/strong&gt; for example the side of turning possibility and for stump always one step &lt;strong&gt;after&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, stump can help in finding a statistical pattern in that example but not the underlying external facts affecting the system in certain move, if the conditions vary randomly in time. &lt;/p&gt;&#xA;" OwnerUserId="44114" LastEditorUserId="44114" LastEditDate="2018-04-16T20:56:28.940" LastActivityDate="2018-04-16T20:56:28.940" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30388" PostTypeId="1" CreationDate="2018-04-16T17:12:26.240" Score="0" ViewCount="94" Body="&lt;p&gt;I have written some code for an SVM and the final output isn’t brilliant. &lt;a href=&quot;https://i.stack.imgur.com/aUzjN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aUzjN.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt; The data is linearly separable so I don’t understand why the data isn’t perfectly separated. Here is the code: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from matplotlib import pyplot as plt, numpy as np&#xA;&#xA;x = np.array([[-2, 4, -1], [4, 1, -1], [1, 6, -1], [2, 4, -1], [6, 2, -1]])&#xA;&#xA;y = np.array([-1, -1, 1, 1, 1])&#xA;&#xA;def svm_sgd(X, Y):&#xA;    w = np.zeros(len(X[0]))    # Parameters &#xA;    eta = 1&#xA;    epochs = 100000&#xA;&#xA;    # Update rules&#xA;&#xA;    for epoch in range(1, epochs):&#xA;        for i, x in enumerate(X):&#xA;            if (Y[i] * np.dot(X[i], w)) &amp;lt; 1:&#xA;                w = w + eta * ((X[i] * Y[i]) + (-2 * (1/epoch) * w))&#xA;            else:&#xA;                w = w + eta * (-2 * (1/epoch) * w)&#xA;    return w&#xA;&#xA;w = svm_sgd(x, y)&#xA;print(w)&#xA;&#xA;# Print Hyperplane —————————————&#xA;&#xA;for d, sample in enumerate(x):&#xA;    if d &amp;lt; 2:&#xA;        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)&#xA;    else:&#xA;        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)&#xA;&#xA;plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')&#xA;plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')&#xA;&#xA;x2=[w[0],w[1],-w[1],w[0]]&#xA;x3=[w[0],w[1],w[1],-w[0]]&#xA;&#xA;x2x3 = np.array([x2,x3])&#xA;X,Y,U,V = zip(*x2x3)&#xA;ax = plt.gca()&#xA;ax.quiver(X,Y,U,V,scale=1, color='blue')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The bottom portion just uses matplotlib to print the data, so it’s not part of the SVM algorithm, and can therefore be ignored. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The problem ~ can anyone explain to me, based on my code, why the line doesn’t separate the data more perfectly?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="37363" LastActivityDate="2018-04-16T18:04:53.273" Title="Improve performance of SVM" Tags="|machine-learning|python|svm|supervised-learning|matplotlib|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30389" PostTypeId="1" AcceptedAnswerId="42185" CreationDate="2018-04-16T17:38:36.590" Score="3" ViewCount="4143" Body="&lt;h2&gt;I'm trying to build a CNN similar to this:&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KtOko.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KtOko.png&quot; alt=&quot;figure 1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this purpose I chose to use Keras since I worked with it before (simple RNN and FFNN only).&#xA;My training data consists of the input data (25 x 25px images) and the output, being 25 x 25px images as well.&#xA;The problem is, that I don't know how to build the Model using Keras. How do I specify the output to be of the same shape as the input?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I have so far:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;batch_size = 5&#xA;input_shape = (batch_size, 28, 28,  32)&#xA;&#xA;model = Sequential()&#xA;model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),&#xA;                 activation='relu',&#xA;                 input_shape=input_shape))&#xA;model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))&#xA;model.add(Conv2D(64, (5, 5), activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Flatten())&#xA;model.add(Dense(1000, activation='relu'))&#xA;model.add(Conv2D(64, (5, 5), activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))&#xA;model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),&#xA;                 activation='relu',&#xA;                 input_shape=input_shape))&#xA;&#xA;model.compile(loss=keras.losses.categorical_crossentropy,&#xA;              optimizer=keras.optimizers.SGD(lr=0.01),&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know, that the code will not work like that. I just want to clarify what I'm trying to do. The code above was taken from: &lt;a href=&quot;http://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras tutorial – build a convolutional neural network in 11 lines&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As of now I just tried to reverse the code from the point, where the tutorial would output the predicted label for the given image to transform the result back to the wanted output image.&lt;/p&gt;&#xA;" OwnerUserId="50627" LastEditorUserId="29575" LastEditDate="2018-04-17T00:11:22.453" LastActivityDate="2019-01-04T19:02:15.777" Title="Keras CNN image input and output" Tags="|keras|tensorflow|cnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30390" PostTypeId="2" ParentId="30388" CreationDate="2018-04-16T18:04:53.273" Score="1" Body="&lt;p&gt;This is the output when I run your code. It seems to be doing fine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ba6TJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ba6TJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[  1.58876117   3.17458055  11.11863105]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Try to run it again? Perhaps you changed some parameters accidentally before posting it?&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-04-16T18:04:53.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30391" PostTypeId="2" ParentId="30383" CreationDate="2018-04-16T18:18:16.740" Score="1" Body="&lt;p&gt;1) Dummy variables doesn't need to be standardized, just numerical ones, but if you use MaxMin scaler you can pass both numerical and dummy variables because this scaler doesn't change the values of dummy variables, try to apply the equation to dummy variables to see.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) That's a trick question and the response is: it depends. If you have some outlier in your data MaxMin doesn't fit well, because this outlier will be replaced by 0 or 1 and the rest of data will be confined in a very restricted range of values, in this case you should use a normal scaler.&lt;/p&gt;&#xA;" OwnerUserId="47872" LastEditorUserId="47872" LastEditDate="2018-04-16T18:53:59.067" LastActivityDate="2018-04-16T18:53:59.067" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30392" PostTypeId="2" ParentId="30384" CreationDate="2018-04-16T18:20:10.603" Score="4" Body="&lt;p&gt;I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your broader categories; description, methodology, classifier, application, and example. That is, I would manually label some transcripts at the paragraph level. If you don't have paragraph-segmented text, smooth the classification probability over the sentences so they share the same label. Or use a paragraph &lt;a href=&quot;https://en.wikipedia.org/wiki/Text_segmentation&quot; rel=&quot;nofollow noreferrer&quot;&gt;segmentation&lt;/a&gt; model as in &lt;a href=&quot;http://www.cstr.ed.ac.uk/downloads/publications/2016/laic2016para.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Automatic Paragraph Segmentation with Lexical and Prosodic Features&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The classifier could be a CRF or an RNN. The modern way to induce the ontology would be through word embeddings; cf. e.g, &lt;a href=&quot;http://ir.hit.edu.cn/~car/papers/acl14embedding.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Semantic Hierarchies via Word Embeddings&lt;/a&gt;. Formerly I would have recommended a hierarchical topic model, such as hLDA; cf. e.g, &lt;a href=&quot;http://jmlr.csail.mit.edu/papers/volume12/zavitsanos11a/zavitsanos11a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Non-Parametric Estimation of Topic Hierarchies from Texts with Hierarchical Dirichlet Processes&lt;/a&gt; or &lt;a href=&quot;https://github.com/XiaofengZhu/hrLDA&quot; rel=&quot;nofollow noreferrer&quot;&gt;Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Welcome to the site and good luck!&lt;/p&gt;&#xA;" OwnerUserId="381" LastEditorUserId="381" LastEditDate="2018-04-16T18:54:42.150" LastActivityDate="2018-04-16T18:54:42.150" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30393" PostTypeId="1" CreationDate="2018-04-16T18:28:18.320" Score="1" ViewCount="69" Body="&lt;p&gt;I wonder whether &lt;code&gt;orange3&lt;/code&gt; allows to get commands that are running &quot;behind&quot; a certain statistical procedure? This would be like in &lt;code&gt;R&lt;/code&gt;, when using &lt;code&gt;RCmdr&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="15381" LastActivityDate="2018-04-16T18:28:18.320" Title="Are interactive command lines from orange3 possible" Tags="|orange|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30394" PostTypeId="2" ParentId="30374" CreationDate="2018-04-16T18:51:42.817" Score="0" Body="&lt;p&gt;There might be a fancier way to create dynamic weights but I would probably start with oversampling the subset and see how that goes. So if you've got classes A, B, and C and want to emphasize C, make a duplicate copy of C and insert that into your training data. In other words, assume you have six records to train on:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A1&lt;/li&gt;&#xA;&lt;li&gt;A2&lt;/li&gt;&#xA;&lt;li&gt;B1&lt;/li&gt;&#xA;&lt;li&gt;B2&lt;/li&gt;&#xA;&lt;li&gt;C1&lt;/li&gt;&#xA;&lt;li&gt;C2&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;add:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;7&quot;&gt;&#xA;&lt;li&gt;C1&lt;/li&gt;&#xA;&lt;li&gt;C2&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30212" LastActivityDate="2018-04-16T18:51:42.817" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30396" PostTypeId="1" AcceptedAnswerId="38648" CreationDate="2018-04-16T20:16:16.030" Score="1" ViewCount="2376" Body="&lt;p&gt;I have been seeing a lot where images are generally scaled down to either &lt;span class=&quot;math-container&quot;&gt;$64\times64$&lt;/span&gt;, &lt;span class=&quot;math-container&quot;&gt;$32\times32$&lt;/span&gt; or other lower resolutions. Can someone please help me with this and answer a few questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Don't we lose image details by doing so?&lt;/li&gt;&#xA;&lt;li&gt;What would be the consequences if we scale down the image to some other higher resolution such as &lt;span class=&quot;math-container&quot;&gt;$512\times512$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$1024\times1024$&lt;/span&gt; or something?&lt;/li&gt;&#xA;&lt;li&gt;Can we feed the network without &lt;span class=&quot;math-container&quot;&gt;$1:1$&lt;/span&gt; square images?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50406" LastEditorUserId="28175" LastEditDate="2018-09-22T15:53:36.483" LastActivityDate="2018-09-22T15:53:36.483" Title="Why do we scale down images before feeding them to the network?" Tags="|machine-learning|neural-network|deep-learning|dataset|cnn|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30397" PostTypeId="1" CreationDate="2018-04-16T23:08:19.493" Score="6" ViewCount="377" Body="&lt;p&gt;The paper I read is &lt;a href=&quot;http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Glorot et al (2010)&lt;/a&gt;. And the math part is in Section 4.2.1.&#xA;Formula (5) and (10) make sense to me but I cannot derive formula (6) and (7) myself from (2) and (3).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found many tutorials on the internet used the formula&#xA;$$Var[XY] = Var[X]Var[Y] + (E[X])^2 Var[Y] + Var[X](E[Y])^2$$&#xA;which requires the independence between X and Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in formula (2) and (3) the gradients are not independent of W and Z, because all of them are related to each other through the output from the last layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate it if anyone can give me a derivation of the formula (6) and (7).&#xA;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="4767" LastEditorUserId="1330" LastEditDate="2021-02-15T15:28:03.333" LastActivityDate="2021-02-15T15:28:03.333" Title="Maths of Xavier initialization" Tags="|deep-learning|neural-network|statistics|mathematics|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30399" PostTypeId="1" CreationDate="2018-04-17T00:22:48.560" Score="1" ViewCount="119" Body="&lt;p&gt;I have tried WEKA's Experimenter. However, it's for classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking for a way to apply the k-means algorithm on the same dataset but with multiple '&lt;strong&gt;k&lt;/strong&gt;' values. Is there any option in WEKA's GUI allow this?&lt;/p&gt;&#xA;" OwnerUserId="41350" LastActivityDate="2024-01-17T00:01:10.140" Title="How do I simultaneously select multiple values for k-means in WEKA?" Tags="|clustering|weka|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30400" PostTypeId="1" CreationDate="2018-04-17T01:02:22.933" Score="2" ViewCount="89" Body="&lt;p&gt;I am absolutely new in R, and my problem is that I do not have any Real World experience in it. I mean, I have learnt a lot but I am always struggling when I get a new task to deal with. Generally speaking I am talking about, how to start to deal with a new task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sometimes the dataset is so big (surprisingly :)) that I am not able to get the panorama about it and the usually used functions such as &lt;code&gt;str()&lt;/code&gt;, &lt;code&gt;summarise()&lt;/code&gt;, &lt;code&gt;head()&lt;/code&gt;, &lt;code&gt;tail()&lt;/code&gt; maybe &lt;code&gt;sample_n&lt;/code&gt; from package &lt;code&gt;dplyr&lt;/code&gt; are not enough to fill me in satisfactorily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost every example that I found on the net, were about datasets which were almost perfect. If we need to clean the data at all, we can relatively easily identify the basic problems because the problems are unambiguous and you can realize them when you check the &lt;code&gt;head()&lt;/code&gt; or something.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What about the real world data? What if the columns shifted in the middle of your dataset, or there are some rows where the values consists an inappropriate symbol or space or something (salary, price, phone-number etc)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary: - What is your general method to getting familiar with your dataset (lets assume that we are already know what is the meaning of the variables because we have a description about it)? - Do you have a general examining method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there are no two similar projects, I am really interested in YOUR basic workflow (with some examples or explanations) though.&lt;/p&gt;&#xA;" OwnerUserId="50634" LastEditorUserId="26562" LastEditDate="2018-04-17T03:28:20.283" LastActivityDate="2018-04-17T09:34:48.037" Title="R basic workflow" Tags="|r|data-mining|data-cleaning|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30401" PostTypeId="1" AcceptedAnswerId="30413" CreationDate="2018-04-17T03:54:40.467" Score="1" ViewCount="310" Body="&lt;p&gt;I am studying the ensemble machine learning and when I read some articles online, I encountered 2 questions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this &lt;a href=&quot;https://www.kdnuggets.com/2017/02/stacking-models-imropved-predictions.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;, it mentions&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Instead, model 2 may have a better overall performance on all the data&#xA;  points, but it has worse performance on the very set of points where&#xA;  model 1 is better. The idea is to combine these two models where they&#xA;  perform the best. This is why creating out-of-sample predictions have&#xA;  a higher chance of capturing distinct regions where each model&#xA;  performs the best.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eGfXW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eGfXW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I still cannot get the point, why not train all training data can avoid the problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this &lt;a href=&quot;http://supunsetunga.blogspot.com/2016/06/stacking-in-machine-learning.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;, in the prediction section, it mentions&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Simply, for a given input data point, all we need to do is to pass it&#xA;  through the M base-learners and get M number of predictions, and send&#xA;  those M predictions through the meta-learner as inputs&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But in the training process, we use k -fold train data to train M base-learner, so should I also train M base-learner based on all train data for the input to predict?&lt;/p&gt;&#xA;" OwnerUserId="50638" LastActivityDate="2018-04-17T09:27:18.387" Title="Questions on ensemble technique in machine learning" Tags="|machine-learning|python|data-mining|scikit-learn|ensemble-modeling|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30402" PostTypeId="1" CreationDate="2018-04-17T06:16:50.543" Score="1" ViewCount="218" Body="&lt;p&gt;Let's say I have a df with the below columns&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Feed     ItemName&#xA;----     --------&#xA;FeedA      ItemA&#xA;FeedA      ItemB&#xA;FeedA      ItemC&#xA;FeedA      ItemA&#xA;FeedA      ItemB&#xA;FeedA      ItemA&#xA;FeedA      ItemC&#xA;FeedA      ItemC&#xA;FeedA      ItemC&#xA;FeedA      ItemA&#xA;FeedA      ItemB&#xA;FeedB      ItemC&#xA;FeedB      ItemB&#xA;FeedB      ItemB&#xA;FeedB      ItemB&#xA;FeedB      ItemC&#xA;FeedB      ItemA&#xA;FeedB      ItemA&#xA;    .....&#xA;    .....&#xA;&#xA;&#xA;    .....&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to get the count of each &lt;code&gt;ItemName&lt;/code&gt; grouped by &lt;code&gt;Feed&lt;/code&gt; and build a Facet grid (for each &lt;code&gt;Feed&lt;/code&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst I tried &#xA;&lt;code&gt;df.groupby(['Feed','ItemName'])['ItemName'].count()&lt;/code&gt;&#xA;to the get the count split by &lt;code&gt;ItemName&lt;/code&gt; &amp;amp; grouped by &lt;code&gt;Feed&lt;/code&gt;, but am not sure on how plot that as graph.&lt;/p&gt;&#xA;" OwnerUserId="50641" LastEditorUserId="50641" LastEditDate="2018-04-17T08:50:47.857" LastActivityDate="2018-04-17T08:50:47.857" Title="Pandas df - FacetGrid - Need help with plotting" Tags="|pandas|plotting|matplotlib|seaborn|" AnswerCount="0" CommentCount="1" ClosedDate="2018-04-22T11:10:26.987" ContentLicense="CC BY-SA 3.0" />
  <row Id="30403" PostTypeId="2" ParentId="30400" CreationDate="2018-04-17T06:17:21.483" Score="0" Body="&lt;p&gt;Go with Analytics vidya where there are many competitions are ongoing, and I recommend you to work on a problem which is still live (Practice problem on Time Series).&#xA;I hope you like it.&lt;/p&gt;&#xA;" OwnerUserId="47646" LastActivityDate="2018-04-17T06:17:21.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30404" PostTypeId="2" ParentId="30399" CreationDate="2018-04-17T06:17:43.983" Score="0" Body="&lt;p&gt;Try writing a program.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's easy to do a for loop over k.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will also want to use multiple random seeds.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-17T06:17:43.983" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30406" PostTypeId="1" CreationDate="2018-04-17T07:12:52.437" Score="1" ViewCount="3524" Body="&lt;p&gt;what are the default kernels used in convolution done in cnn &lt;/p&gt;&#xA;&#xA;&lt;p&gt;for example in this code of keras there are 32 filters of size 5x5 each.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my question is 1) What will be the different default kernels in those 32 filters. &#xA;for example in an tutorial I read that the &#xA;\begin{bmatrix}&#xA;    1 &amp;amp; 1 \\&#xA;    -1       &amp;amp; -1\\&#xA;\end{bmatrix} could be taken as horizontal edge detector and &#xA;\begin{bmatrix}&#xA;1&amp;amp; -1\\&#xA;1 &amp;amp; -1&#xA;\end{bmatrix}&#xA;could be taken as vertical edge detector .&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def baseline_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu'))&#xA;    model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;    model.add(Conv2D(32, (5, 5), activation='relu'))&#xA;    model.add(Dropout(0.2))&#xA;    model.add(Flatten())&#xA;    model.add(Dense(128, activation='relu'))&#xA;    model.add(Dense(num_classes, activation='softmax'))&#xA;    # Compile model&#xA;    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-04-17T08:27:06.723" Title="default kernels in cnn in keras?" Tags="|machine-learning|keras|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30408" PostTypeId="2" ParentId="13151" CreationDate="2018-04-17T07:21:36.863" Score="28" Body="&lt;p&gt;In general, the performance of classifiers are compared using accuracy, this is a measure of the number of correctly classified instances divided by the total number of instances. However, from the training data we can get a better approximation of the expected error from our classifier when we are using ensemble learning or bagging techniques.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Out-of-bag error&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This metric is the accuracy of examples $x_i$ using all the trees in the random forest ensemble for which it was omitted during training. Thus it kind of acts as a semi-testing instance. You can get a sense of how well your classifier can generalize using this metric. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To implement oob in sklearn you need to specify it when creating your Random Forests object as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import RandomForestClassifier &#xA;forest = RandomForestClassifier(n_estimators = 100, oob_score = True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we can train the model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;forest.fit(X_train, y_train)&#xA;print('Score: ', forest.score(X_train, y_train))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Score:  0.979921928817&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As expected the accuracy of the model when evaluating the training set is very high. However, this is meaningless because you can very well be overfitting your data and thus your model is rubbish. However, we can use the out-of-bag score as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(forest.oob_score_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.86453272101&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is the accuracy whilst evaluating our instances in the training set using only the trees for which they were omitted. Now let's calculate the score on the testing set as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print('Score: ', forest.score(X_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Score:  0.86517733935&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We see that the accuracy measured by oob is very similar to that obtained with the testing set. It thus follows through the theory that the oob accuracy is a better metric by which to evaluate the performance of your model rather than just the score. This is a consequence of bagging models and cannot be done with other types of classifiers. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Calculating oob using different metrics&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Yes, you can do this! However, it depends how exactly your code is structured. I am not sure how you can include the oob and AUC all together with the &lt;code&gt;cross_val_score&lt;/code&gt; function. However, if you are doing the cross validation folds manually you can do the following, the random forests algorithm in sklearn provides you the decision function of the oob as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(forest.oob_decision_function_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The class can then be obtained using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import metrics&#xA;pred_train = np.argmax(forest.oob_decision_function_,axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we can calculate the AUC by using the following&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;metrics.roc_auc_score(y_train, pred_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.86217157846471204&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-04-17T08:07:39.807" LastActivityDate="2018-04-17T08:07:39.807" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30410" PostTypeId="2" ParentId="30406" CreationDate="2018-04-17T08:27:06.723" Score="1" Body="&lt;p&gt;The initial value of the CNN kernels can be seen from the documentation found &lt;a href=&quot;https://keras.io/layers/convolutional/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The default Conv2D layer looks like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;keras.layers.Conv2D(32, (5, 5), activation='relu', kernel_initializer='glorot_uniform', bias_initializer='zeros')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we see that the kernel weights are initialized by Glorot uniform method an dthe bias is initialized as all zeros. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Glorot Uniform&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Uniform sampling from $[-limit, limit]$ where &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$limit = \sqrt{\frac{6}{fan\_in + fan\_out}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $fan\_in$ is the number of input neurons to the weight tensor and $fan\_out$ is the number of output neurons of the weight tensor.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-17T08:27:06.723" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30411" PostTypeId="2" ParentId="30239" CreationDate="2018-04-17T08:40:11.473" Score="0" Body="&lt;p&gt;Transferring an original image to a different style of image is called &lt;strong&gt;style transfer&lt;/strong&gt;. Although you can use a GAN for such a case, I am not convinced this is the best approach. Furthermore, there have been many papers recently published suggesting very interesting ways for which to train models for style transfer. An amazing implementation of a file transfer algorithm which can convert video in real time can be found &lt;a href=&quot;https://github.com/lengstrom/fast-style-transfer&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is another method that I have come across more recently which uses the same approach but a different loss function. The paper can be found &lt;a href=&quot;https://arxiv.org/pdf/1603.08155.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The underlining method&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;These methods use a combination of two networks. The first network acts as a transformation function $f_W$ which maps your image $x$ to a transformed image $\hat{y}$. The purpose is to train this network such that the output $\hat{y}$ is the original input image in the new style, in your case as line art. To train this network we will use the second network on the right side which learns how to mix in the &lt;em&gt;Style Target&lt;/em&gt; and the &lt;em&gt;Content Target&lt;/em&gt;. We want the transformed image $\hat{y}$ is still similar in content, but we want the transformed image to have a similar style as the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To train this network we will use a massive amount of input images using some online database source. This training process will be very long. However, once it is complete then we can strip the network on the right side and you will only need the network which approximates $f_W$. You will then be able to put a new image into the input of this network and the output will be the line art version of the input image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to tune the extent at which the input image is &lt;em&gt;line artified&lt;/em&gt; by tuning the weight on the style loss during the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/q8Up8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/q8Up8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-17T08:40:11.473" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30412" PostTypeId="2" ParentId="30401" CreationDate="2018-04-17T08:51:08.423" Score="2" Body="&lt;p&gt;1- The idea of ensemble methods is to reduce variance mostly, which means &quot;overfitting&quot;. The idea behind it is that we train different models ( in this case 2 models ) that don't necesseraly see all the training data points, so it won't overfit on it ( method commonly called as bagging ). Now for the prediction we can do a vote for classification problems or mean value for regression problems or even a whole learner for that. That ensures that the model will be stable through training and test phase. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if we train on all training data we may get overfitting and thus low accuracy on test cases.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- have a look at this one :&lt;a href=&quot;https://datascience.stackexchange.com/questions/18339/why-use-both-validation-set-and-test-set&quot;&gt;Why use both validation set and test set?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50647" LastEditorUserId="50647" LastEditDate="2018-04-17T08:59:31.683" LastActivityDate="2018-04-17T08:59:31.683" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30413" PostTypeId="2" ParentId="30401" CreationDate="2018-04-17T08:57:05.010" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Instead, model 2 may have a better overall performance on all the data&#xA;  points, but it has worse performance on the very set of points where&#xA;  model 1 is better. The idea is to combine these two models where they&#xA;  perform the best. This is why creating out-of-sample predictions have&#xA;  a higher chance of capturing distinct regions where each model&#xA;  performs the best.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's not about training on all the data or not. Both models trained on all the data. But each of them is better than the other at different points. If I and my older brother are tying to guess the exact year of a song, I will do better in 90s songs and he in 80s songs - it's not a perfect analogy but you get the point - imagine my brain just can't process 80s songs, and his can't process 90s songs. The best is to deploy us both knowing we each have learnt different regions of the input space better. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Simply, for a given input data point, all we need to do is to pass it&#xA;  through the M base-learners and get M number of predictions, and send&#xA;  those M predictions through the meta-learner as inputs&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;k-fold is still just one learner. But you're training multiple times to chose parameters that minimize error in the left-out fold. This is like training only me on all the songs showing me k-1 folds of data, and I calibrate my internal model the best I can... but I'll still never be very good at those 80s songs. I'm just one base learner whose functional form (my brain) isn't fit for those songs. If we could bring the second learner along, that would improve things. &lt;/p&gt;&#xA;" OwnerUserId="31457" LastEditorUserId="40853" LastEditDate="2018-04-17T09:27:18.387" LastActivityDate="2018-04-17T09:27:18.387" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30414" PostTypeId="2" ParentId="30400" CreationDate="2018-04-17T09:00:14.887" Score="1" Body="&lt;p&gt;Here is a great recent video of Hadley Wickham doing exploratory analysis of a medium-sized dataset. It contains many simple but clever hacks, and demonstrates strategies such as &quot;doing a single case first, then doing something for all instances&quot; (here: one city -&gt; all cities)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=go5Au01Jrvs&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=go5Au01Jrvs&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly showing the &quot;whole game&quot; of data analysis. See code and data at &lt;a href=&quot;https://github.com/hadley/building-permits&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/hadley/building-permits&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14588" LastActivityDate="2018-04-17T09:00:14.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30415" PostTypeId="2" ParentId="30320" CreationDate="2018-04-17T09:02:04.953" Score="1" Body="&lt;h1&gt;Are there better techniques for doing video classification compared to using temporal segment networks?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I would suggest to train the network to detect bird species in image space without the temporal dimension. This will make the network have much less parameters to tune and you will likely get better results. Then for a novel video, you can detect the species for each frame and do a majority vote to decide what species was in the video.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Are there any preprocessing steps that should be done to improve accuracy?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If you train in image space then you can use a DataGenerator on the image and transform it in all kinds of ways, the position and rotation of the bird does not affect the species. This can be done through the Keras pipeline. You can check how to do this in general &lt;a href=&quot;https://datascience.stackexchange.com/questions/28491/data-augmentation-rotating-images-and-zero-values/28503#28503&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How to train the model to generalize better to be less sensitive to lighting and weather?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You can try to pass your images through some pretrained segmentation network to identify background and foreground and then set that area to a static color. Perhaps black (i.e. all 0's). This will make the decision boundary for the network easier. However, I think you should focus on the other points, the network should inherently detect what features are the most salient for discriminating different species, if you have sufficient training samples the weather and background will be low information features and will thus not be used.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How can the model be trained to classify unknown species?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This would be called anomaly detection. This is a much harder problem. If you have birds of &lt;em&gt;other&lt;/em&gt; species available in your dataset then you can add a class label for other. This is still standard supervised learning. If this is not available it will be very difficult. Most anomaly detection algorithms do not work well for high dimensional datasets. Then you will need a way to compress the information in your input images. This can be achieved with some transfer learning on a pretrained autoencoder network trained on natural images such as ImageNET, then you can use transfer learning using your bird images to further tune the network for your specific use case. You can then use this compressed feature space using one of the anomaly detection techniques I described &lt;a href=&quot;https://datascience.stackexchange.com/questions/29411/outliers-detection-with-non-normal-distribution&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-17T09:02:04.953" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30416" PostTypeId="2" ParentId="30039" CreationDate="2018-04-17T09:31:27.923" Score="0" Body="&lt;p&gt;Is there a reason you want to do the re-sampling inside the tensorflow computation graph? Otherwise, there are well-tested and established tools to do re-sampling as a separate step, consider running&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;$ sox unnormalized_file.wav -r 44100 normalized_file.wav&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;on all your data files.&lt;/p&gt;&#xA;" OwnerUserId="50039" LastActivityDate="2018-04-17T09:31:27.923" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30417" PostTypeId="2" ParentId="30400" CreationDate="2018-04-17T09:34:48.037" Score="2" Body="&lt;p&gt;There are several packages on R that provides you full reports on the status of your dataframe. Not only the number of missings, but also the type of the columns, the number of outliers (if it is continuous), the distribution along categories (if it is categorical),...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I have to choose one of them, a great help is for sure &lt;a href=&quot;http://blog.revolutionanalytics.com/2018/02/dataexplorer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataExplorer&lt;/a&gt;, give it a try! I usually combine it with a set of checks prepared by myself, once I know a bit about the data frame and the information I expect from a &quot;business&quot; side.&lt;/p&gt;&#xA;" OwnerUserId="49226" LastActivityDate="2018-04-17T09:34:48.037" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30422" PostTypeId="2" ParentId="30385" CreationDate="2018-04-17T12:11:09.080" Score="2" Body="&lt;p&gt;I have found the answer to my question. while training, I need to pass the training and validation twice like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit([X_train, X_train], Y_train,..., validation_data=([X_valid, X_valid], Y_valid)). &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41557" LastEditorUserId="52291" LastEditDate="2019-04-29T14:29:55.527" LastActivityDate="2019-04-29T14:29:55.527" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30423" PostTypeId="1" AcceptedAnswerId="30441" CreationDate="2018-04-17T12:19:25.567" Score="0" ViewCount="1741" Body="&lt;p&gt;I'm attempting to merge a VGG-16 and ResNet-50 through concatenation. I was successful in training and saving the merged model. Here is the code snippet:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras import applications&#xA;from keras.models import Model&#xA;from keras.layers import Dense, GlobalAveragePooling2D&#xA;from keras.layers import Concatenate&#xA;from keras.models import load_model&#xA;# Loading the training data&#xA;img_rows=300&#xA;img_cols=300&#xA;channel = 3&#xA;num_classes = 3&#xA;batch_size = 10 &#xA;nb_epoch = 10&#xA;#load the first model ######################################################### &#xA;base_model1 = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))&#xA;#get the model summary&#xA;base_model1.summary()&#xA;#addind the top layers&#xA;x1 = base_model1.output&#xA;x1 = GlobalAveragePooling2D()(x1) #analogous to flatten()&#xA;model1 = Model(inputs=base_model1.input, outputs=x1)&#xA;model1.summary()&#xA;#load the second model ########################################################&#xA;base_model2 = applications.ResNet50(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))&#xA;#get the model summary&#xA;base_model2.summary()&#xA;#addind the top layers&#xA;x2 = base_model2.output&#xA;x2 = GlobalAveragePooling2D()(x2) #analogous to flatten()&#xA;model2 = Model(inputs=base_model2.input, outputs=x2)&#xA;model2.summary()&#xA;&#xA;'''merge the models'''&#xA;mergedOut = Concatenate()([model1.output,model2.output])&#xA;#add a new dense layer and softmax&#xA;out=Dense(2048, activation='relu')(mergedOut) &#xA;out = Dense(num_classes, activation='softmax', name='predictions')(out)&#xA;#create the new model with three branches and one dense layer&#xA;model = Model(inputs=[model1.input,model2.input], outputs=out)&#xA;model.summary()&#xA;###############################################################################&#xA;#training the model&#xA;hist=model.fit([X_train, X_train], Y_train, batch_size=batch_size, epochs=nb_epoch, &#xA;                     shuffle=True, verbose=1, validation_data=([X_valid, X_valid], Y_valid)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, the inputs to both the models are the same. I would like to use this as a common input to the merged model to avoid passing X_train and X_valid twice during the model.fit. How can I do this?&lt;/p&gt;&#xA;" OwnerUserId="41557" LastActivityDate="2018-04-17T18:41:38.487" Title="How to pass common inputs to a merged model in Keras" Tags="|classification|keras|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30428" PostTypeId="1" AcceptedAnswerId="30434" CreationDate="2018-04-17T14:33:49.113" Score="1" ViewCount="2325" Body="&lt;p&gt;matlab function detrend subtracts the mean from data. If data contains several data columns, detrend treats each data column separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the normalization technique is subtracting the mean and dividing it by standard deviation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the normalization already subtract the mean from the data, in such case, is it essential to perform (before or after) detrend operation? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the significance of each operation? &lt;/p&gt;&#xA;" OwnerUserId="50664" LastActivityDate="2018-04-17T16:16:57.433" Title="What is difference between detrend and normalization?" Tags="|machine-learning|matlab|normalization|" AnswerCount="1" CommentCount="1" ClosedDate="2018-04-20T08:23:51.280" ContentLicense="CC BY-SA 3.0" />
  <row Id="30429" PostTypeId="2" ParentId="23423" CreationDate="2018-04-17T14:46:55.050" Score="3" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/danFromTelAviv/bounding-box-labeler-yolo&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a great simple tool&lt;/a&gt;. It's fully in Python so you can play around with it to fit your needs more properly.&lt;/p&gt;&#xA;" OwnerUserId="42634" LastEditorUserId="42634" LastEditDate="2018-05-07T13:28:12.350" LastActivityDate="2018-05-07T13:28:12.350" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30430" PostTypeId="1" AcceptedAnswerId="84177" CreationDate="2018-04-17T15:11:00.177" Score="5" ViewCount="288" Body="&lt;p&gt;Concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With reference to the classic house price prediction use case:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;House prices change over time thus the model I use today could make no sense in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best approach to address concept drift?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do we keep updating the input replacing older house prices of yesteryear?&lt;/li&gt;&#xA;&lt;li&gt;Do we add an extra feature for Date of Sale - by including a temporal aspect as a feature with larger data sets? &lt;/li&gt;&#xA;&lt;li&gt;Do we eventually change model hyperparameters during training to build a model that fits better the new data?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47412" LastEditorUserId="33551" LastEditDate="2018-05-08T12:06:43.713" LastActivityDate="2020-10-18T15:30:52.310" Title="Temporal Aspects in Machine Learning" Tags="|machine-learning-model|" AnswerCount="1" CommentCount="23" ContentLicense="CC BY-SA 4.0" />
  <row Id="30431" PostTypeId="2" ParentId="30233" CreationDate="2018-04-17T15:29:32.653" Score="4" Body="&lt;p&gt;The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After step (1), a decision tree would only operate on the bottom orange part since the top blue part is already perfectly separated. The top blue part would be left unchanged. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The boosted stumps, however,  operate (as you mentioned) on the full dataset again, which can lead to different results.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-04-17T15:37:05.830" LastActivityDate="2018-04-17T15:37:05.830" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30432" PostTypeId="1" AcceptedAnswerId="30436" CreationDate="2018-04-17T15:54:54.190" Score="0" ViewCount="1867" Body="&lt;p&gt;I am running a CNN, on the 1st epoch my training set accuracy is 15% and validation set is 12%, by the 51st epoch my training accuracy is 87% and validation set is 13%. What is happening? What does it mean if my validation set is lower or greater than my training set ?? (my test set gives 12%)&lt;/p&gt;&#xA;" OwnerUserId="50532" LastEditorUserId="29169" LastEditDate="2021-02-13T04:24:21.620" LastActivityDate="2021-02-13T04:24:21.620" Title="What if my validation set is worse than my training?" Tags="|neural-network|cross-validation|training|convolutional-neural-network|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30434" PostTypeId="2" ParentId="30428" CreationDate="2018-04-17T16:10:10.543" Score="1" Body="&lt;p&gt;You &lt;code&gt;detrend&lt;/code&gt; data in order to &lt;strong&gt;get rid of the linear trend&lt;/strong&gt; in your data, which might cause &lt;a href=&quot;https://en.wikipedia.org/wiki/Spurious_relationship&quot; rel=&quot;nofollow noreferrer&quot;&gt;spurious regression&lt;/a&gt;, misleading evidence that there is some relation between variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Normalization&lt;/code&gt; means adjusting values measured on different scales to a notionally &lt;strong&gt;common scale&lt;/strong&gt;. &#xA;Example you give: &lt;a href=&quot;https://en.wikipedia.org/wiki/Standard_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;Standard score&lt;/a&gt; $\frac{X - \mu}{\sigma}$ is just one possible method of normalization . It allows you to see where value lies in comparison to mean. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience you would only &lt;code&gt;detrend&lt;/code&gt; data in order to create time series model. On the other hand, &lt;code&gt;normalization&lt;/code&gt; is frequently used to compare previously not comparable statistics or detect anomalies (in case of standard score). As a result, usage really depends on the use case and you rarely use both at the same time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wikipedia is good enough source to clarify this unambiguity. &lt;/p&gt;&#xA;" OwnerUserId="49349" LastEditorUserId="49349" LastEditDate="2018-04-17T16:16:57.433" LastActivityDate="2018-04-17T16:16:57.433" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30435" PostTypeId="2" ParentId="13676" CreationDate="2018-04-17T16:25:58.577" Score="2" Body="&lt;p&gt;Pooling (max/mean/etc) has two primary benefits: it significantly reduces computational complexity (at the cost of potentially important data) and it helps the network achieve spatial invariance by making the spatial relativity between features less relevant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the spatial invariance achieved through pooling is not always ideal.    For instance, given a CNN trained to identify images of automobiles, it would probably classify an image containing a hood/trunk, tires, windows, car doors, etc. as an automobile, regardless of the components' relative positioning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, I do not know of significant costs to using padding.  Zero padding is critical for deep networks; without it, our volumes would rapidly collapse through the layers.  It therefore helps to maintain desirable volume sizes and to preserve the border data.  It also helps the geometry of CNNs to work out smoothly by preventing mismatched dimensions between layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might find &lt;a href=&quot;https://arxiv.org/pdf/1710.09829.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; interesting.  Hinton and his team propose a new network called Capsule Network (CapsNet) that addresses the problems with pooling and provides an alternative solution for acquiring spatial invariance.  One of the many exciting parts of their algorithm is that the location of features relative to one another is a primary asset in the classification process, bringing computers closer to a &quot;true understanding&quot; of objectness.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="29575" LastEditDate="2018-04-17T17:51:00.483" LastActivityDate="2018-04-17T17:51:00.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30436" PostTypeId="2" ParentId="30432" CreationDate="2018-04-17T16:43:22.393" Score="3" Body="&lt;p&gt;This is a common high-variance problem due to overfitting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Simply put:&lt;/strong&gt; Good training accuracy together with low dev set (or in your terminology the validation set) accuracy it means you are expressing your training data very well, whereas your model fails to perform well on unseen data (High-variance Problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;More general:&lt;/strong&gt; there are few other combinations of this &lt;em&gt;high-/low-bias&lt;/em&gt; together with &lt;em&gt;high-/low-variance&lt;/em&gt; that are explained in length in the context of Bias-Variance Trade-off by Andrew Ng in his first Machine Learning course and even more relevant to Convolutional Neural Networks in his last DeepLearning course, see this &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/ZhclI/bias-variance&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;. You also easily find many blog posts discussing about these stuffs like this &lt;a href=&quot;https://kevinzakka.github.io/2016/09/26/applying-deep-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt;, or &lt;a href=&quot;https://medium.com/@rishavsapahia/5-min-recap-for-andrew-ng-deep-learning-specialization-course-3-90e89a332b0c&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you see here is quite alright. If fact it is highly recommended practice that you start by a model capable of describing your training set very well first (even overfit). Then if overfit meaning there is a large difference in accuracy between train/dev sets (similar to your scenario), you start penalizing the model (aka regularization). Once again you can find tons of materials on how to regularize CNN models. On the top of your search you will find Dropout, Max-Pooling. See &lt;a href=&quot;https://towardsdatascience.com/deep-learning-3-more-on-cnns-handling-overfitting-2bd5d99abe5d&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also suggest to pay attention to your data split distributions. It is a very fundamental concept that often paid less attention to. Violating this assumption (aka data split mismatch) could lead to problems that could appear in any forms like large accuracy difference of model on data splits. Maybe check Ng's short &lt;a href=&quot;https://www.coursera.org/learn/machine-learning-projects/lecture/78P8f/train-dev-test-distributions&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-04-18T05:24:24.183" LastActivityDate="2018-04-18T05:24:24.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30437" PostTypeId="2" ParentId="989" CreationDate="2018-04-17T16:47:11.257" Score="1" Body="&lt;p&gt;I just had a similar issue with a dataset which contains only 115 elements and only one single feature (international airline data). The solution was to scale the data. What I missed in answers so far was the usage of a Pipeline:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.svm import SVR&#xA;from sklearn.pipeline import Pipeline&#xA;from sklearn.preprocessing import StandardScaler, MinMaxScaler&#xA;&#xA;model = Pipeline([('scaler', StandardScaler()),&#xA;                  ('svr', SVR(kernel='linear'))])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can train &lt;code&gt;model&lt;/code&gt; like a usual classification / regression model and evaluate it the same way. Nothing changes, only the definition of the model.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="38887" LastEditDate="2020-04-08T10:21:17.367" LastActivityDate="2020-04-08T10:21:17.367" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30439" PostTypeId="1" CreationDate="2018-04-17T17:28:45.820" Score="2" ViewCount="1984" Body="&lt;p&gt;I have a convolutional network which consists of multiple CNN layers (which could work with any size) and a fully connected layer at the end which can only work with fixed input size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have trained the network for certain image size and the fully connected layer can only work for those input sizes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't want to retrain the entire network when operating on different input sizes. Is there any way to achieve training of only the newly added fully connected layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code for inference looks like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def inference(images, reuse=False, trainable=True):&#xA;    coarse1_conv = conv2d('coarse1', images, [11, 11, 3, 96], [96], [1, 4, 4, 1], padding='VALID', reuse=reuse, trainable=trainable)&#xA;    coarse1 = tf.nn.max_pool(coarse1_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID', name='pool1')&#xA;    coarse2_conv = conv2d('coarse2', coarse1, [5, 5, 96, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)&#xA;    coarse2 = tf.nn.max_pool(coarse2_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1')&#xA;    coarse3 = conv2d('coarse3', coarse2, [3, 3, 256, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)&#xA;    coarse4 = conv2d('coarse4', coarse3, [3, 3, 384, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)&#xA;    coarse5 = conv2d('coarse5', coarse4, [3, 3, 384, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)&#xA;    coarse6 = fc('coarse6', coarse5, [6*10*256, 4096], [4096], reuse=reuse, trainable=trainable)&#xA;    coarse7 = fc('coarse7', coarse6, [4096, 4070], [4070], reuse=reuse, trainable=trainable)&#xA;    coarse7_output = tf.reshape(coarse7, [-1, 55, 74, 1])&#xA;    return coarse7_output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50669" LastEditorUserId="28175" LastEditDate="2018-04-17T18:08:39.460" LastActivityDate="2018-04-17T18:08:39.460" Title="Changing the shape of the input layer in tensorflow" Tags="|machine-learning|neural-network|deep-learning|tensorflow|cnn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30440" PostTypeId="2" ParentId="30439" CreationDate="2018-04-17T18:08:19.193" Score="2" Body="&lt;p&gt;Actually, if you change the input the input size, nothing goes wrong with the convolutional layers but the outputs of these layers increases and that will cause to the increased number of inputs to the dense layers &lt;a href=&quot;https://datascience.stackexchange.com/a/28312/28175&quot;&gt;1&lt;/a&gt;. Consequently, you will have to have extra weights and you have to train them. That is why it is better to have a fixed size input. But there are other solutions that can be expanded to your task. Take a look at the contents of the third week of &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; course. As you will see, you have to input patches of the image to the network but what is done there is for detection tasks, you have to expand it to your task, but maybe resizing all inputs to a predefined size is simpler.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-17T18:08:19.193" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30441" PostTypeId="2" ParentId="30423" CreationDate="2018-04-17T18:41:38.487" Score="0" Body="&lt;p&gt;I found the answer to my question. I need to pass the common input shape to the individual models and concatenate the outputs like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_shape=(img_rows,img_cols,3)&#xA;commonInput = Input(input_shape)&#xA;out1 = model1(commonInput)    &#xA;out2 = model2(commonInput)  &#xA;mergedOut = Concatenate()([out1,out1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41557" LastActivityDate="2018-04-17T18:41:38.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30443" PostTypeId="1" CreationDate="2018-04-17T19:02:55.260" Score="4" ViewCount="206" Body="&lt;p&gt;I am looking for any straightforward solution to work with Jupyter Notebooks and git since the cached data on the .ipynb files makes the diff difficult to read.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know any standalone alternative and not server hosted?&lt;/p&gt;&#xA;" OwnerUserId="50674" LastEditorUserId="1330" LastEditDate="2018-04-17T19:14:05.827" LastActivityDate="2018-04-18T15:10:42.043" Title="Jupyter notebooks and git" Tags="|jupyter|" AnswerCount="1" CommentCount="2" ClosedDate="2018-04-20T06:09:53.360" ContentLicense="CC BY-SA 3.0" />
  <row Id="30445" PostTypeId="2" ParentId="30443" CreationDate="2018-04-17T19:15:58.633" Score="3" Body="&lt;p&gt;One option is to strip the output from the .ipynb file. Then the git diff would only track the cell data. One package that strips the output Jupyter Notebook is &lt;a href=&quot;https://github.com/kynan/nbstripout&quot; rel=&quot;nofollow noreferrer&quot;&gt;nbstripout&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-04-18T15:10:42.043" LastActivityDate="2018-04-18T15:10:42.043" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30447" PostTypeId="1" AcceptedAnswerId="30455" CreationDate="2018-04-17T20:14:59.613" Score="2" ViewCount="451" Body="&lt;p&gt;I want to use t-SNE in WEKA just for visualization purposes. I tried to look at the package manager but could not find any implementation of it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything that I can do to achieve it?&lt;/p&gt;&#xA;" OwnerUserId="50685" LastActivityDate="2018-04-18T02:33:05.247" Title="Is there t-SNE in WEKA?" Tags="|weka|tsne|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30448" PostTypeId="1" CreationDate="2018-04-17T21:19:04.727" Score="6" ViewCount="9576" Body="&lt;p&gt;I'm using xgboost package in R with early stopping at 75 rounds. To monitor the progress the algorithm I print the F1 score from the training and test set after each round. After the algorithm has done 75 rounds, xgboost returns the model with the highest score on the test set, not the training set. My guess is that this has to do with the monitoring functionality and the watchlist parameter of xgboost &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intention of giving the algorithm access to the test set during training (using the watchlist parameter) was to monitor the training progress, and not to  select the best performing classifier with respect to the test set. That would be cheating right? My question is two-fold:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is this really cheating? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;and if it is cheating,&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Why is it programmed into the xgboost package?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50689" LastEditorUserId="29575" LastEditDate="2018-04-18T02:03:12.717" LastActivityDate="2018-04-18T22:59:14.750" Title="Is the early stopping of xgboost using correct" Tags="|xgboost|model-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30449" PostTypeId="2" ParentId="19997" CreationDate="2018-04-17T22:21:34.087" Score="1" Body="&lt;p&gt;Yes they are related. As an example, consider Gaussian smoothing (en.wikipedia.org/wiki/Gaussian_blur) which is a convolution with a kernel of Gaussian values. A CNN learns the weights of filters (i.e. kernels), and thus can learn to perform smoothing if needed.&lt;/p&gt;&#xA;" OwnerUserId="10683" LastActivityDate="2018-04-17T22:21:34.087" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30450" PostTypeId="2" ParentId="19997" CreationDate="2018-04-17T23:23:27.100" Score="1" Body="&lt;p&gt;Although &lt;code&gt;CNN&lt;/code&gt; stands for convolutional neural networks, what they do is named cross-correlation in mathematics and not convolution. Take a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/wfUhx/strided-convolutions&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Now, before moving on there is a technical comment I want to make about cross-correlation versus convolutions and just for the facts what you have to do to implement convolutional neural networks. If you reading different math textbook or signal processing textbook, there is one other possible inconsistency in the notation which is that, if you look at the typical math textbook, the way that the convolution is defined before doing the element Y's product and summing, there's actually one other step ...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-17T23:23:27.100" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30453" PostTypeId="1" AcceptedAnswerId="30454" CreationDate="2018-04-18T01:41:04.280" Score="0" ViewCount="557" Body="&lt;p&gt;Below is the dataset for which I am trying to implement Linear regression in &#xA;python.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   age     sex     bmi  children smoker     region      charges&#xA;0   19  female  27.900         0    yes  southwest  16884.92400&#xA;1   18    male  33.770         1     no  southeast   1725.55230&#xA;2   28    male  33.000         3     no  southeast   4449.46200&#xA;3   33    male  22.705         0     no  northwest  21984.47061&#xA;4   32    male  28.880         0     no  northwest   3866.85520&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am confused what to do with the columns children, smoker, sex as they are of type &quot;object&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data columns (total 7 columns):&#xA;age         1338 non-null int64&#xA;sex         1338 non-null object&#xA;bmi         1338 non-null float64&#xA;children    1338 non-null int64&#xA;smoker      1338 non-null object&#xA;region      1338 non-null object&#xA;charges     1338 non-null float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Do I have to convert this to numeric before building my model ?&#xA;Please provide your suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="40853" LastEditDate="2018-04-18T02:08:44.427" LastActivityDate="2018-04-18T05:28:14.430" Title="Linear Regression in Python" Tags="|python|linear-regression|feature-engineering|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30454" PostTypeId="2" ParentId="30453" CreationDate="2018-04-18T02:23:33.837" Score="1" Body="&lt;p&gt;Yes, you will have to convert everything to numeric. That requires thinking about what these attributes represent accordingly you can use either the below 3 options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are three options:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;One-Hot encoding for categorical data&lt;/li&gt;&#xA;&lt;li&gt;Arbitrary numbers for ordinal data&lt;/li&gt;&#xA;&lt;li&gt;Use something like group means for categorical data (e. g. mean&#xA;prices for city districts).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You have to be careful to not infuse information you do not have in the application case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm expanding on option 1 and 3, if you want to know about option 2 you can go through the links attached at last.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;One hot encoding&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If you have categorical data, you can create dummy variables with 0/1 values for each possible value. Similarly you could implement for children, smoker.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;E. g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id  Sex&#xA;0   Male&#xA;1   Feamle&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;id   Male   Female &#xA;0      1       0    &#xA;1      0       1   &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This can easily be done with pandas:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;data = pd.DataFrame({'Sex': ['Male', 'Female']})&#xA;print(pd.get_dummies(data))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;will result in:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        Sex_Male  Sex_Female &#xA;0           1            0         &#xA;1           0            1        &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Using categorical data for groupby operations&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is an additional usecase but in your case it is not necessary to use this but if you feel so, you can try implementing this as well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use the mean for each category over past (known events).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say you have a DataFrame with the last known mean prices for cities:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prices = pd.DataFrame({&#xA;    'city': ['A', 'A', 'A', 'B', 'B', 'C'],&#xA;    'price': [1, 1, 1, 2, 2, 3],&#xA;})&#xA;mean_price = prices.groupby('city').mean()&#xA;data = pd.DataFrame({'city': ['A', 'B', 'C', 'A', 'B', 'A']})&#xA;&#xA;print(data.merge(mean_price, on='city', how='left'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  city  price&#xA;0    A      1&#xA;1    B      2&#xA;2    C      3&#xA;3    A      1&#xA;4    B      2&#xA;5    A      1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For better understanding you can go through this &lt;a href=&quot;https://stackoverflow.com/questions/34007308/linear-regression-analysis-with-string-categorical-features-variables/34008270#34008270&quot;&gt;Link-1&lt;/a&gt;, &lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.dtypes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link-2&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-04-18T02:23:33.837" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30455" PostTypeId="2" ParentId="30447" CreationDate="2018-04-18T02:33:05.247" Score="2" Body="&lt;p&gt;Sadly no, there is not a T-SNE implementation for WEKA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can install python packages in your environment, then you can use the &lt;code&gt;wekaPython&lt;/code&gt; package (in WEKA's package manager) to run scikit-learn's T-SNE implementation on data you have loaded into WEKA. Use this code in the 'CPython Scripting' panel (which appears after successfully installing &lt;code&gt;wekaPython&lt;/code&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = py_data.iloc[:, :-1]&#xA;y = py_data.iloc[:, -1]&#xA;&#xA;from sklearn.manifold import TSNE&#xA;import matplotlib.pyplot as plt&#xA;&#xA;tX = TSNE().fit_transform(X)&#xA;&#xA;plt.scatter(tX[:, 0], tX[:, 1])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-18T02:33:05.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30456" PostTypeId="1" AcceptedAnswerId="30458" CreationDate="2018-04-18T03:28:40.770" Score="1" ViewCount="664" Body="&lt;p&gt;I have around 9500 cities that I want to put in a pandas dataframe, then store in a file for later use. For example, below, these are 3 cities that I have data for. Some range in size based on year. Lovell_Wyoming only has 9 years of data points, corresponding to the years, as opposed to the 15 that Wheatland and Worland have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My original idea was putting the quantitative data (Arson..Year) in a map, then putting the city name, as a key, into a larger map, with that quantitative data. Building the larger map that way. Then, converting the map to a dataframe, then to a csv. I am a bit inexperienced with pandas so I am not sure how to do this correctly, if this is even the best way to do it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day, I would like this data in a csv file that is easily accessible by loading it into a dataframe and calling on whatever value I need.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;City 'Lovell_Wyoming' &#xA;Arson [0, 0, 0, 0, 0, 0, 0, 1, 0] &#xA;Assaults [6, 6, 3, 4, 3, 28, 3, 2, 2] &#xA;Auto_thefts [1, 1, 1, 0, 0, 1, 2, 0, 1] &#xA;Burglaries [6, 11, 5, 2, 0, 15, 11, 7, 7] &#xA;Murders [0, 0, 0, 0, 1, 0, 0, 0, 0] &#xA;Rapes [0, 0, 3, 0, 0, 1, 1, 0, 1] &#xA;Robberies [0, 0, 0, 0, 0, 0, 0, 1, 0] &#xA;Thefts [23, 49, 35, 39, 28, 37, 54, 35, 10] &#xA;Year [2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2014]&#xA;&#xA;City 'Wheatland_Wyoming' &#xA;Arson [0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0] &#xA;Assaults [9, 2, 6, 5, 6, 6, 2, 4, 2, 4, 3, 11, 5, 4, 8] &#xA;Auto_thefts [4, 8, 3, 3, 4, 4, 5, 3, 4, 6, 4, 8, 12, 7, 3] &#xA;Burglaries [17, 17, 14, 9, 10, 17, 12, 26, 51, 12, 15, 21, 32, 31, 13] &#xA;Murders [1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0] &#xA;Rapes [0, 0, 0, 4, 2, 1, 2, 0, 2, 1, 1, 0, 2, 0, 0] &#xA;Robberies [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] &#xA;Thefts [109, 95, 146, 81, 108, 100, 82, 85, 106, 128, 48, 85, 66, 56, 47] &#xA;Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]&#xA;&#xA;City 'Worland_Wyoming' &#xA;Arson [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] &#xA;Assaults [12, 17, 19, 17, 11, 15, 16, 2, 9, 1, 4, 7, 2, 1, 3] &#xA;Auto_thefts [2, 1, 2, 1, 1, 8, 1, 1, 1, 0, 1, 0, 0, 0, 1] &#xA;Burglaries [6, 10, 10, 10, 9, 10, 10, 0, 6, 1, 0, 2, 0, 11, 18] &#xA;Murders [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] &#xA;Rapes [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3] &#xA;Robberies [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] &#xA;Thefts [44, 41, 47, 29, 30, 25, 27, 27, 23, 30, 36, 45, 54, 46, 43] &#xA;Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I apologize in advance if the way I formatted this is a bit weird! Please let me know if you would like any more information.&lt;/p&gt;&#xA;" OwnerUserId="50703" LastEditorUserId="50703" LastEditDate="2018-04-18T03:42:21.267" LastActivityDate="2018-04-18T06:28:44.537" Title="Formatting categories of data with pandas in Python" Tags="|python|pandas|dataframe|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30457" PostTypeId="2" ParentId="30448" CreationDate="2018-04-18T03:45:30.683" Score="1" Body="&lt;p&gt;That's not cheating. If your test set is a representative sample of the future data you'll want to make predictions on, you'll want to have the lowest possible error there!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you maximized performance on the training set, instead, you might overfit. It's fairly easy for a boosted algorithm to inadvertently memorize its training data rather than learn a meaningful mapping of inputs to output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Per the comment below, the &quot;test set&quot; you describe is actually functioning like a validation set here. &lt;/p&gt;&#xA;" OwnerUserId="50706" LastEditorUserId="50706" LastEditDate="2018-04-18T10:25:03.160" LastActivityDate="2018-04-18T10:25:03.160" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30458" PostTypeId="2" ParentId="30456" CreationDate="2018-04-18T04:47:02.060" Score="4" Body="&lt;p&gt;This is how I would do it. However, a DataFrame can be structured in a number of way which best suits your needs. I believe that this method allows for the greatest flexibility because you can easily use grouping functions to restructure this format on the go. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First you need to set up your data in a way that is compatible with Python. I use a dictionary of dictionaries&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cities = {'Lovell_Wyoming':&#xA;          {'Crimes':&#xA;           {&#xA;            'Arson': [0, 0, 0, 0, 0, 0, 0, 1, 0],&#xA;            'Assaults': [6, 6, 3, 4, 3, 28, 3, 2, 2] ,&#xA;            'Auto_thefts': [1, 1, 1, 0, 0, 1, 2, 0, 1] ,&#xA;            'Burglaries': [6, 11, 5, 2, 0, 15, 11, 7, 7] ,&#xA;            'Murders': [0, 0, 0, 0, 1, 0, 0, 0, 0] ,&#xA;            'Rapes': [0, 0, 3, 0, 0, 1, 1, 0, 1] ,&#xA;            'Robberies': [0, 0, 0, 0, 0, 0, 0, 1, 0] ,&#xA;            'Thefts': [23, 49, 35, 39, 28, 37, 54, 35, 10]&#xA;           },&#xA;            'Years': [2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2014]&#xA;          },&#xA;&#xA;         'Wheatland_Wyoming':&#xA;          {'Crimes':&#xA;           {&#xA;            'Arson': [0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0] ,&#xA;            'Assaults': [9, 2, 6, 5, 6, 6, 2, 4, 2, 4, 3, 11, 5, 4, 8] ,&#xA;            'Auto_thefts': [4, 8, 3, 3, 4, 4, 5, 3, 4, 6, 4, 8, 12, 7, 3] ,&#xA;            'Burglaries': [17, 17, 14, 9, 10, 17, 12, 26, 51, 12, 15, 21, 32, 31, 13] ,&#xA;            'Murders': [1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0] ,&#xA;            'Rapes': [0, 0, 0, 4, 2, 1, 2, 0, 2, 1, 1, 0, 2, 0, 0] ,&#xA;            'Robberies': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,&#xA;            'Thefts': [109, 95, 146, 81, 108, 100, 82, 85, 106, 128, 48, 85, 66, 56, 47]&#xA;            },&#xA;              'Years': [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]&#xA;            }&#xA;         }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then compile this data into rows of your DataFrame&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = []&#xA;for city in cities:&#xA;    for ix, year in enumerate(cities[city]['Years']):&#xA;        for crime in cities[city]['Crimes']:&#xA;            temp = {'City':city, &#xA;                    'Crime':crime, &#xA;                    'Year':year, &#xA;                    'Count':cities[city]['Crimes'][crime][ix]}&#xA;            data.append(temp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then into the DataFrame structure as.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.DataFrame(data=data)&#xA;df&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JdVm4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JdVm4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Queries&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You can then query this DataFrame in a number of ways for example if you want to know arsons in 2002 you would do &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[(df['Crime']=='Arson') &amp;amp; (df['Year']==2002)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qAcWi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qAcWi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Counting&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You can count the number of arsons throughout the years as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[(df['Crime']=='Arson')].groupby(['City'])['Count'].agg('sum')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;City &lt;br/&gt;Lovell_Wyoming       1 &lt;br/&gt;Wheatland_Wyoming    3 &lt;br/&gt;Name: Count, dtype:&#xA;  int64&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Writing your DataFrame to a CSV file&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This can be done directly as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.to_csv('filename.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-04-18T06:28:44.537" LastActivityDate="2018-04-18T06:28:44.537" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30459" PostTypeId="2" ParentId="30453" CreationDate="2018-04-18T05:28:14.430" Score="1" Body="&lt;p&gt;First notice that the column &quot;children&quot; is already numeric, so you don't need to do anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns &quot;Sex&quot; and &quot;Smoker&quot; are categorical and yes you need to change them to numeric form. This is because many ML algorithms like Linear Regression require data in numeric form, they can't operate directly on text data. You can check more answers here-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Why-do-we-have-to-convert-the-categorical-value-into-factor-in-R-or-dummy-variables-before-applying-machine-learning-algorithms-especially-linear-regression-Does-it-impact-our-results&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/Why-do-we-have-to-convert-the-categorical-value-into-factor-in-R-or-dummy-variables-before-applying-machine-learning-algorithms-especially-linear-regression-Does-it-impact-our-results&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to convert categorical to numeric form?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This has been answered by &lt;a href=&quot;https://datascience.stackexchange.com/users/40853/toros91&quot;&gt;Toros91&lt;/a&gt;, you have &lt;code&gt;get_dummies&lt;/code&gt; function in pandas which create dummy columns for each category of categorical feature in your data. So for your case,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; pd.get_dummies(df, columns=['sex','smoker'])&#xA;age    bmi     children   region      charges   sex_female  sex_male  smoker_no  smoker_yes    &#xA;0  19  27.900     0      southwest  16884.92400      1         0           0           1    &#xA;1  18  33.770     1      southeast   1725.55230      0         1           1           0    &#xA;2  28  33.000     3      southeast   4449.46200      0         1           1           0    &#xA;3  33  22.705     0      northwest  21984.47061      0         1           1           0    &#xA;4  32  28.880     0      northwest   3866.85520      0         1           1           0    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44139" LastActivityDate="2018-04-18T05:28:14.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30460" PostTypeId="1" CreationDate="2018-04-18T05:40:49.820" Score="1" ViewCount="49" Body="&lt;p&gt;I have following sort of data coming every day:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(0)(3,4,5)(6,9,1)(5,35,12,232)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1)(5,1,4)(6,2)(12,54,12,43)(8,23,65)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2)(6,7,2)(34,3)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3)(4323,23,12,4543)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4)(987,32,324,23,224,12,213,21)(1,2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(5)(3242,23,23434,34,324,322)(4,342,423,4)(3,1,30)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(6)(1,2,3,4,5)(6,7)(8,9,10)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(0)(1,2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1)(54,12)(45,21,5,19)(9,8,6,41)(432,1,431,2)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In each sequence, the first number indicates the day of a week (0-6).&#xA;All other numbers indicate user IDs.&#xA;The meaning of the first sequence is:&#xA;On Sunday (0), following users met as 3 different groups: (3,4,5), (6,9,1), (5,35,12,232)&#xA;What is the best method for predicting user groups for next day?&#xA;Can I use RNN? Any specific method that I should look into? Any classical problems closely related to this?&lt;/p&gt;&#xA;" OwnerUserId="50711" LastEditorUserId="50711" LastEditDate="2018-04-18T12:33:25.240" LastActivityDate="2018-05-18T14:22:27.810" Title="Group prediction" Tags="|deep-learning|rnn|prediction|sequence|sequence-to-sequence|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30461" PostTypeId="2" ParentId="30460" CreationDate="2018-04-18T06:22:41.927" Score="1" Body="&lt;p&gt;RNN can be used for such tasks, But there is another non deep learning based approach which might be helpful here See &lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Association rule learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38492" LastActivityDate="2018-04-18T06:22:41.927" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30462" PostTypeId="2" ParentId="30396" CreationDate="2018-04-18T06:37:31.510" Score="1" Body="&lt;p&gt;Because it saves lots of computation time. Yes, we do loose image details, but it depends on your task whether the details are really important. Let's say your task is to detect and count the total number of circles in an image, So you are only concerned with the shape circle, not its dimension. Hence, reducing image resolution a bit would be more computationally efficient.&lt;/p&gt;&#xA;" OwnerUserId="38492" LastActivityDate="2018-04-18T06:37:31.510" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30465" PostTypeId="1" AcceptedAnswerId="30486" CreationDate="2018-04-18T07:04:30.120" Score="3" ViewCount="35532" Body="&lt;p&gt;I have just run a Linear regression model on the Dataset having 7 independent variable and 1 target variable. Below is the R squared and MSE values.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean squared error for training set : 36530921.0123&lt;/li&gt;&#xA;&lt;li&gt;$R^2$ value for training set : 0.7477&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Can anybody please give me some tips to increase the efficiency of this model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I have just implemented the same problem using Linear regression with Normalization of the features. &#xA;I got the below output:&#xA;Mean squared error for training set : 5.468490570335696e-10&#xA;R2 value for training set : 0.9275088299658416&#xA;Mean squared error for training set : 4.111793316375822e-10&#xA;R2 value for training set : 0.9342888671422529&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So can we consider normalizing the dataset to get better accuracy ?&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="50146" LastEditDate="2018-04-21T07:54:09.890" LastActivityDate="2018-04-21T07:54:09.890" Title="Tips to improve Linear Regression model" Tags="|machine-learning|python|regression|linear-regression|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 3.0" />
  <row Id="30467" PostTypeId="1" AcceptedAnswerId="30468" CreationDate="2018-04-18T08:01:30.100" Score="2" ViewCount="74" Body="&lt;p&gt;I am wondering how I can manage a test data after using PCA or normalization and another thing like that in the classification because our model works on the representation given by its input vectors. For example, suppose you have used PCA in your training dataset to gain better accuracy or you have normalized (min-max) data. Now, you have developed a model and want to install it and label the new coming samples. You need to somehow apply PCA to each coming record and normalize that record. Applying PCA to one record will not yield the same effect of the PCA  used in training phase and I think even it doesn't make sense to apply PCA to just one sample. So how can we manage these preprocessing techniques in the training phase in the test data, too?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="12345" LastActivityDate="2018-04-18T08:37:43.503" Title="Basic classification question" Tags="|machine-learning|classification|data-mining|training|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30468" PostTypeId="2" ParentId="30467" CreationDate="2018-04-18T08:37:43.503" Score="0" Body="&lt;p&gt;PCA is a matrix transformation from your original dataset to a set of orthogonal features. The transformation matrix which is applied to the training set is maintained and used in the future with your testing data such that the original testing set features will be mapped to same space as the training set transformed by PCA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the training set has $n$ instances and their are $m$ features, the training matrix is of size $n \times m$. The PCA transformation matrix is of dimensions $m \times k$, where $k$ is the number of retained PCA features, the top eigenvalues. Thus we can transform a single instance $1 \times m$, by the $m \times k$ transformation matrix. This results in a $1 \times k$ vector.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I have some text files that I vectorized using bag-of-words. The training set is shown below on the left side and the testing set is on the right side. Each row is a text file and the columns is the word count.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OBwxS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OBwxS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we plot the first 2 features of this dataset we get &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/vSmuA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vSmuA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we will fit our PCA transformation matrix and we will apply this transformation to both the training and testing set.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.decomposition import PCA&#xA;pca = PCA(n_components=2, copy=True)&#xA;pca.fit(X_train)&#xA;train_PCA = pca.transform(X_train)&#xA;test_PCA = pca.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives the following plot. The purple and yellow points are the 2 different classes from the training set. Then the light blue points are from the testing set. You can see that the points in the testing set after being transformed by PCA will line up alongside the training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kRSsJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kRSsJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-18T08:37:43.503" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30469" PostTypeId="1" AcceptedAnswerId="30478" CreationDate="2018-04-18T08:42:01.380" Score="0" ViewCount="389" Body="&lt;p&gt;I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When there are not many states QLearning seems to be working well, but as soon as the environment gets bigger and the amount of states gets bigger it is performing rather bad. Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model? Is it possible with reinforcement in general or does it require to learn all possible states? &lt;/p&gt;&#xA;" OwnerUserId="50719" LastActivityDate="2018-04-18T13:18:33.847" Title="Can you interpolate with QLearning or Reinforcement learning in general?" Tags="|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30470" PostTypeId="2" ParentId="30150" CreationDate="2018-04-18T08:53:59.603" Score="1" Body="&lt;p&gt;The differnece between supervised and unsupervised learning is that in supervised learning we have a labelled data (with the correct output) i.e there is one dependent variable, and the job of the ML task is to produce an output that is close to the actual output. In unsupervised learning we dont have any dependent variable or we dont have a labelled data or the actual output and the job of the ML task is to find hidden patterns and group the data according to the patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of ML algorithm will you use largely depend on the type of the data, that you can find by visualizing the data. If there is a linear correlation between the inputs and the output you can use simple linear regression and that will suffice and there is there is not a linear correlation than you can use SVM'S and if you dont want to use that you can use hit and try method and apply all simple alogorithms like naive bayes , linear regression , SVM,s and pick the that gives the best result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding LSTM'S they are a type of deeplearning networks and mainly used for sequential data in time and a very complicated one like audio data or words in a sentence and this is an overcomplication of this task. If you want to go towards deepl learning than you can use a simple ANN for this task&lt;/p&gt;&#xA;" OwnerUserId="50720" LastActivityDate="2018-04-18T08:53:59.603" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30473" PostTypeId="1" AcceptedAnswerId="30479" CreationDate="2018-04-18T10:28:15.793" Score="3" ViewCount="9556" Body="&lt;p&gt;I would like to know how the &lt;code&gt;alpha&lt;/code&gt; parameter, in Multinomial Bayes, affects the text classification task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that this parameter is correlated to the algorithm's ability in classifying unseen words during training. How changes text classification using low or high values of alpha?&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2018-04-18T13:20:25.207" Title="How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?" Tags="|classification|naive-bayes-classifier|text|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30474" PostTypeId="1" CreationDate="2018-04-18T11:54:37.870" Score="1" ViewCount="411" Body="&lt;p&gt;After reading a while, I am confused now about my LSTM data structure. Assuming that I have a supervised learning problem with 1000 samples and 40 features as input. Now I want to create 10 timesteps of x. My resulting dimension of Keras data structure is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x: (1000, 10, 40)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with every two-dimensional matrix (1000, 40) shifted row-wise and ten times. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is now:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Which dimension has my target y to be?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dimension is &lt;code&gt;y=[1000,1]&lt;/code&gt; for one resulting target, but I also read &lt;code&gt;y=[1000,10,1]&lt;/code&gt;. I suppose, the second dimension with value 10 should be the row-wise shifted target-vector. But, do I really need this? Shouldn't my solution be the right one?&lt;/p&gt;&#xA;" OwnerUserId="40108" LastEditorUserId="29575" LastEditDate="2018-04-19T05:55:11.190" LastActivityDate="2024-02-28T02:04:05.437" Title="Keras data structure for LSTM-networks" Tags="|keras|lstm|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30475" PostTypeId="1" AcceptedAnswerId="30502" CreationDate="2018-04-18T12:24:40.873" Score="2" ViewCount="301" Body="&lt;p&gt;From my observations and little experience it appears that most of the ML project are about classifying stuff. Is there cancer signs on the photo? Does the picture show car, whale or banana? Etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to implement a model for face identification. Not detection/recognition, but identification: having two different photos of the same person, my model should determine if in the pictures is depicted the same person. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to achieve that using Tensorflow with convolutional nets. I've read this paper: &lt;a href=&quot;http://ydwen.github.io/papers/WenECCV16.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://ydwen.github.io/papers/WenECCV16.pdf&lt;/a&gt; and center loss looks promising. What do you think about that? Are there any new ideas/papers/implementations regarding that problem that are worth attention?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked this question also on MachineLearning reddit (&lt;a href=&quot;https://www.reddit.com/r/MachineLearning/comments/8cysrx/d_what_are_the_stateoftheart_models_for/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.reddit.com/r/MachineLearning/comments/8cysrx/d_what_are_the_stateoftheart_models_for/&lt;/a&gt;) and got an useful link with FaceNet implementation, trying here also :)&lt;/p&gt;&#xA;" OwnerUserId="50732" LastActivityDate="2018-04-24T06:22:47.393" Title="What are the state-of-the-art models for identifying objects in photos?" Tags="|deep-learning|tensorflow|cnn|convolution|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30476" PostTypeId="1" AcceptedAnswerId="30481" CreationDate="2018-04-18T12:28:01.727" Score="5" ViewCount="10473" Body="&lt;p&gt;In pandas I can set the date as index, and then run &lt;code&gt;df.plot()&lt;/code&gt; to see a line chart. How do I make that line chart &lt;strong&gt;stacked&lt;/strong&gt; as in the picture below?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xcBWf.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xcBWf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2020-08-10T08:18:03.803" Title="Stacked time series plot in python" Tags="|python|visualization|pandas|seaborn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30478" PostTypeId="2" ParentId="30469" CreationDate="2018-04-18T13:11:37.020" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;When Q learning is described as &quot;model free&quot;, it means that the agent does not need access to (or use) a predictive model of the environment. It cannot refer to state transitions and rewards in advance, but has to experience them in order to learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;em&gt;does not&lt;/em&gt; mean that you have to avoid using a learning data model (such as a neural network) in order to generalise to new unseen data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, yes, Q learning can interpolate from unseen states and predict their Q value. To do this, you replace the state/action table with a supervised learning method based on descriptions of state $s$ and action $a$ as inputs, that you train as a regression model to predict $Q(s,a)$ (as a variant you can also have just state as input and predict $Q(s,a)$ for all possible actions as a vector in one go).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, Q learning with a neural network suffers from instability. See &lt;a href=&quot;https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Mind's DQN paper&lt;/a&gt; for example of a system that solves that instability. In short:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use experience replay - store S, A, R, S' data for each step and run the Q learning update on random mini-batches of the stored data, instead of online.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Keep two copies of the Q estimator neural network. Train one continuously, and copy it to a &quot;frozen&quot; version every now and then (e.g. every 100 mini-batches). Use the &quot;frozen&quot; copy to calculate the new $Q(s,a)$ targets.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This still might not match your learning scenario. If you want to solve mazes, think carefully about what data is truly available to the agent and how you might use it. For instance if you are using Q learning to solve a maze where you have a map, it is very inefficient approach. This is often shown as a toy problem, because it is possible to view the learning data very easily. But in that toy problem, the agent is not given the map, nor any knowledge of what a grid is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are a couple of suggestions that may still help, separate to using a neural network value estimator:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If you do have a model of your environment (but not a map or other data that could be directly analysed for a solution), joining Q learning with a planning algorithm might work better for you than Q learning, as &lt;a href=&quot;https://medium.com/@ranko.mosic/online-planning-agent-dyna-q-algorithm-and-dyna-maze-example-sutton-and-barto-2016-7ad84a6dc52b&quot; rel=&quot;nofollow noreferrer&quot;&gt;in Dyna-Q&lt;/a&gt;. This is relevant where you have an agent exploring in real time that would benefit from &quot;looking&quot; ahead before taking actions.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If your problem is very sparse rewards (due to larger maze, and only getting different reward at the end), then a worthwhile improvement is to look into multi-step TD learning, where the rewards are propagated back to previous steps more efficiently. &lt;a href=&quot;https://stackoverflow.com/questions/40862578/how-to-understand-watkinss-q%CE%BB-learning-algorithm-in-suttonbartos-rl-book&quot;&gt;Maybe look into $Q(\lambda)$&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-04-18T13:18:33.847" LastActivityDate="2018-04-18T13:18:33.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30479" PostTypeId="2" ParentId="30473" CreationDate="2018-04-18T13:12:50.627" Score="2" Body="&lt;p&gt;Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a  tweet made by Trump or not. You are given a tweet, &quot;I have huge respect for women&quot; and your goal is to classify it to 'Trump' or 'Not Trump'. Moreover, you are given 5 other random tweets and you also know whether those tweets were by Trump or not (Basically, they are already classified) this is your training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;in other words you are calculating,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(I\,have\,huge\,respect\,for\,women) = P(I)\times P(have)\times P(huge)\times P(respect)\times P(for)\times P(women)\\$$&#xA;$$\\$$&#xA;$$P(I\,have\,huge\,respect\,for\,women|Trump) = P(I|Trump)\times P(have|Trump)\times P(huge|Trump)\times P(respect|Trump)\times P(for|Trump)\times P(women|Trump)\\$$&#xA;$$\\$$&#xA;$$P(I\,have\,huge\,respect\,for\,women|Not\,Trump) = P(I|Not\,Trump)\times P(have|Not\,Trump)\times P(huge|Not\,Trump)\times P(respect|Not\,Trump)\times P(for|Not\,Trump)\times P(women|Not\,Trump)$$&#xA;$$\\$$&#xA;Depending on the probablity of equations (2) and (3) the user would make a decision whether the statement was made by Trump or not. Now lets say that none of the 5 training set tweets have the word huge in it, in which case,&#xA;$$\\$$&#xA;$$P(huge|Trump)=P(huge|Not\,Trump)=0$$&#xA;$$\\$$&#xA;And hence, equations (2) and (3) are now zero and this is bad. A solution to this is to do smoothing or rather Laplace smoothing. The basic idea is to increase the probablities of all bigrams in your non-maximum likelihood equation (Since we are changing the counts from what they occurred in hopes to make it better) by 1 to make everything non-zero. That increase by 1 is called adding the pseudocount or as you know it as $\alpha$. Now, $\alpha=1$ may not always give the most accurate probablities and hence they could attain any finite non-negative integer. The way to know what $\alpha$ gives the most accurate responses is through iterating over all values of $\alpha$ on the training set unfortunately. The way to do that is whole different topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS : Trump did say he has huge respect for women. Such irony.&lt;/p&gt;&#xA;" OwnerUserId="32958" LastEditorUserId="32958" LastEditDate="2018-04-18T13:20:25.207" LastActivityDate="2018-04-18T13:20:25.207" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30480" PostTypeId="2" ParentId="30460" CreationDate="2018-04-18T13:43:03.283" Score="1" Body="&lt;p&gt;I see your problem consisting of two parts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predicting which users will participate.&lt;/li&gt;&#xA;&lt;li&gt;Predicting the number of groups and what individual group composition will be.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For the first part, you can use &lt;a href=&quot;https://www.cs.sjsu.edu/~stamp/RUA/HMM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hidden Markov Models (HMMs)&lt;/a&gt; for each user id which basically model the probabilities such that given the prior knowledge of user participation ('PNNPPPN') it predicts how likely a user will participate next time.&#xA;[where 'P' stands for Participated, 'N' stands for Not-participated and the sequence is a 1 week history of a particular user.]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second part, you can use &lt;a href=&quot;https://arxiv.org/pdf/1301.3781.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Word2vec&lt;/a&gt; with which you can get vector embeddings of each user. When you visualise these vectors (in a 2-D space) you could see the clusters where the users who tend to form groups more often would be closer to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, from above two things now you know that whether a particular user will participate today or not and which other users, who have similar vector embeddings, are also likely to participate, you can predict the groups for a particular day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(This is one approach, of-course there can be others.)&lt;/p&gt;&#xA;" OwnerUserId="41058" LastEditorUserId="41058" LastEditDate="2018-04-18T13:48:30.543" LastActivityDate="2018-04-18T13:48:30.543" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30481" PostTypeId="2" ParentId="30476" CreationDate="2018-04-18T13:48:43.510" Score="7" Body="&lt;p&gt;You can simply use&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.plot.area()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Found &lt;a href=&quot;https://python-graph-gallery.com/254-pandas-stacked-area-chart/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; after a quick google search.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="98307" LastEditDate="2020-08-10T08:18:03.803" LastActivityDate="2020-08-10T08:18:03.803" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="30482" PostTypeId="1" AcceptedAnswerId="30509" CreationDate="2018-04-18T15:32:53.290" Score="1" ViewCount="144" Body="&lt;p&gt;I have a df with nearly 40 million rows and ~20 columns (total size is 2.2+GB). 15 of my features are categorical. I figured that the most reasonable way to go about this problem without making the df any bigger would be fit/transform each with LabelEncoder and then convert each feature to category data type.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only thing is that none of the categorical features are ordinal. Should I fit/transform them with StandardScaler or is that unnecessary?&lt;/p&gt;&#xA;" OwnerUserId="50746" LastActivityDate="2018-04-19T06:30:05.240" Title="Handling categorical variables in large df" Tags="|python|classification|pandas|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30485" PostTypeId="2" ParentId="30302" CreationDate="2018-04-18T16:33:12.830" Score="4" Body="&lt;p&gt;The fact that the coefficients of hp and disp are low when data is unscaled and high when data are scaled means that these variables help explaining the dependent variable but their magnitude is large, so the coefficients in the unscaled case have to be low.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In terms of &quot;importance&quot;, I would say that the absolute value of the coefficients in the scaled case is a good measure of the importance, more than in the unscaled case, since there the magnitude of the variable is also relevant, and it should not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course the more important variable is wt.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-18T16:33:12.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30486" PostTypeId="2" ParentId="30465" CreationDate="2018-04-18T17:20:34.720" Score="6" Body="&lt;p&gt;You can build more complex models to try to capture the remaining variance. Here are several options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Add &lt;a href=&quot;https://en.wikipedia.org/wiki/Interaction_(statistics)#In_regression&quot; rel=&quot;noreferrer&quot;&gt;interaction terms&lt;/a&gt; to model how two or more&#xA;independent variables together impact the target variable&lt;/li&gt;&#xA;&lt;li&gt;Add &lt;a href=&quot;https://en.wikipedia.org/wiki/Polynomial_regression&quot; rel=&quot;noreferrer&quot;&gt;polynomial terms&lt;/a&gt; to model the nonlinear relationship between an independent variable and the target variable&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Add &lt;a href=&quot;http://people.stat.sfu.ca/~cschwarz/Consulting/Trinity/Phase2/TrinityWorkshop/Workshop-handouts/TW-04-Intro-splines.pdf&quot; rel=&quot;noreferrer&quot;&gt;spines&lt;/a&gt; to approximate piecewise linear models&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fit &lt;a href=&quot;https://en.wikipedia.org/wiki/Isotonic_regression&quot; rel=&quot;noreferrer&quot;&gt;isotonic regression&lt;/a&gt; to remove any assumption of the target function form&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Fit non-parametric models, such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines&quot; rel=&quot;noreferrer&quot;&gt;MARS&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-18T17:20:34.720" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30487" PostTypeId="1" CreationDate="2018-04-18T18:45:01.210" Score="0" ViewCount="4103" Body="&lt;p&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=ByjaPdWXKJ4&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.youtube.com/watch?v=ByjaPdWXKJ4&lt;/a&gt; This is the time-stamped video of &quot;deconvolution&quot;. I can understand normal convolution but not so much with upsampling convolution. In the video he explained that you plop down the filter and use each individual scalar as the weight to apply to each value in the filter. I am having a hard time understanding how he got an output shape of 4x4? Is there a special formula for calculating the output for &quot;upconvolution&quot;? I am also confused on the padding and how it affects upconvolution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My crappy paint image &lt;a href=&quot;https://i.stack.imgur.com/3EH6A.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3EH6A.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="46645" LastActivityDate="2018-04-21T12:02:09.560" Title="I still don't know how deconvolution works after watching CS231 lecture, I need help" Tags="|deep-learning|convolution|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30489" PostTypeId="2" ParentId="30465" CreationDate="2018-04-18T19:04:51.767" Score="1" Body="&lt;p&gt;Multicollinearity could be a reason for poor perfomance when using Linear Regression Models. Multicollinearity refers to a situation where a number of independent variables in a Linear Regression model are closely correlated to one another and it can lead to skewed results. In general, multicollinearity can lead to wider confidence intervals and less reliable probability values for the independent variables. &#xA;Also maybe other assumptions of Linear Regrresion do not hold. Linear regression needs the relationship between the independent and dependent variables to be linear.  It is also important to check for outliers since linear regression is sensitive to outlier effects.  The linearity assumption can best be tested with scatter plots. Linear regression analysis requires that there is little or no autocorrelation in the data.  Autocorrelation occurs when the residuals are not independent from each other.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="35679" LastEditDate="2018-04-18T21:10:33.663" LastActivityDate="2018-04-18T21:10:33.663" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30490" PostTypeId="1" AcceptedAnswerId="31070" CreationDate="2018-04-18T20:19:38.973" Score="2" ViewCount="164" Body="&lt;p&gt;Let $v \in \{0,1\}^M$ be the visible layer, $h \in \{0,1\}^N$ be the hidden layer, where $M$ and $N$ are natural numbers. Given the biases $b \in \Re^M$, $c \in \Re^N$ and weights $W \in \Re^{M \times N}$, the energy and probability of an RBM is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Energy, $E(v,h; b,c,W) = -b^T v - c^T h - h^T W^T v$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probability, $p(v, h; b, c, W) = \frac{e^{-E(v,h; b,c,W)}}{Z}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $Z = \sum_{v,h} e^{-E(v,h; b,c,W)}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The negative log likelihood error for a Restricted Boltzmann Machine (RBM) is given by:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\mathcal{L}(b,c,W) = \frac{1}{T} \sum_{t=1}^{T} \left( -\log \sum_{h} e^{-E(v^t, h; b,c,W)} \right) + \log Z$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$T$ is size of training dataset; and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$v^t$ represents $t^{th}$ data point in the training dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is clear that computing $Z$ (and hence $\log Z$) is intractable because we have to sum over $2^{M+N}$ configurations of $v$ and $h$ - exponential time algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, shouldn't computing $\sum_{h} e^{-E(v^t, h; b,c,W)}$ be intractable as well? Aren't we summing over all the $2^N$ configurations of $h$ here? If say $N = 64$, then we are already reaching exascale computations ($2^{64} = 1.84 \times 10^{19}$)!&lt;/p&gt;&#xA;" OwnerUserId="47649" LastActivityDate="2018-05-01T06:43:03.400" Title="Isn't computing the &quot;tractable error&quot; in Restricted Boltzmann Machines (RBM) intractable?" Tags="|neural-network|deep-learning|clustering|training|rbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30491" PostTypeId="2" ParentId="30487" CreationDate="2018-04-18T20:32:29.310" Score="2" Body="&lt;p&gt;OP here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d&lt;/a&gt;&#xA;This has helped me understand how deconvolutional layer or transposed convolution works! Note to self don't google deconvolution or upconvolution google transposed convolution instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update 2: &lt;a href=&quot;http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It turns out &quot;deconvolution&quot; is just convolution but with different arithmetics. You can take the transpose or add enough padding so that &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) You can upsample instead of downsampling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Keep the previously linked relation, what I mean is you need to make sure each upsampled filter still contains a symbolic link with the smaller input. &lt;/p&gt;&#xA;" OwnerUserId="46645" LastEditorUserId="46645" LastEditDate="2018-04-21T12:02:09.560" LastActivityDate="2018-04-21T12:02:09.560" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30492" PostTypeId="2" ParentId="30474" CreationDate="2018-04-18T20:50:57.490" Score="0" Body="&lt;p&gt;Your target &lt;strong&gt;y&lt;/strong&gt; can be whatever you need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to do sequence to sequence for example, &lt;strong&gt;y&lt;/strong&gt; will have the same number of time steps than the inputs (you predict something for each timestep).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But you can also define an output with one timestep only, for text classification for example.&#xA;Your input data has 10 timesteps, but the output is only a prediction at the last timestep.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It depends on the problem you’re trying to solve.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-04-18T20:50:57.490" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30493" PostTypeId="2" ParentId="30369" CreationDate="2018-04-18T21:10:07.740" Score="1" Body="&lt;p&gt;LDA is a bayesian model. The equation that you gave is the posterior distribution of the model. The alpha and beta parameters come from the fact that the dirichlet distribution, (a generalization of the beta distribution) takes these as parameters in the prior distribution. So to answer your first question, will the formula above work without the alpha and gamma, yes, you would implicitly be assuming these parameters from the prior distribution are both equal to zero. You will still get a result, but the prior you are using might be problematic on a conceptual level (maybe it would cause some numerical problems too, I'm not sure?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer question 2, I want to point out that in LDA we use the conjugate prior to simplify the calculation of the optimal parameter values. That is what you are seeing in the above equation. What you are seeing is the posterior value is proportional to the left hand side. Adding the &quot;c&quot; terms allows you to calculate the posterior.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-04-18T21:10:07.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30494" PostTypeId="2" ParentId="13882" CreationDate="2018-04-18T22:01:00.007" Score="1" Body="&lt;p&gt;Actually both roles are recommended for someone from coding background. It depends more on the specific characteristics of each role in a company. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data engineering is more about infrastructure work, which means parsing data files, storing data in particular databases (SQL or NoSQL e.g. Mongo-DB), designing databases or designing the pipeline of the data process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Science is more about building models, selecting appropriate variables, performing exploration or validation of statistical models, hypothesis testing etc.. All these need good knowledge of at least one scripting programming language like Python, Matlab and R. In some cases there is also a need of Software Engineer skills for the implementation of applications related to Predictive Analytics or Machine Learning(or Statistical Learning). &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-04-18T22:01:00.007" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30495" PostTypeId="2" ParentId="30448" CreationDate="2018-04-18T22:59:14.750" Score="2" Body="&lt;p&gt;The early stopping and watchlist parameters in xgboost can be used to prevent overfitting. If the watchlist is given two data-sets, then the algorithm will perform hold out validation as described &lt;a href=&quot;http://codingwiththomas.blogspot.se/2016/03/xgboost-validation-and-early-stopping.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting the test set in the watchlist will cause the algorithm to select the model with the best performance against the test set which can be considered as cheating.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was calling my algorithm like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train &amp;lt;- data[selection,]&#xA;test &amp;lt;- data[-selection,]&#xA;&#xA;watchlist &amp;lt;- list(train=train,test=test)&#xA;&#xA;bstSparse &amp;lt;- xgb.train(data=train, &#xA;                       ...,&#xA;                       watchlist=watchlist,&#xA;                       early_stopping_rounds=75)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I should have been calling it like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_and_val &amp;lt;- data[selection,]&#xA;inds &amp;lt;- createDataPartition(temp_and_val$labels, p = 0.9)&#xA;&#xA;train &amp;lt;- train_and_val[inds,]&#xA;validation &amp;lt;- train_and_val[-inds,]&#xA;test &amp;lt;- data[-selection,]&#xA;&#xA;watchlist &amp;lt;- list(validation=validation,train=train)&#xA;&#xA;bstSparse &amp;lt;- xgb.train(data=train, &#xA;                       ...,&#xA;                       watchlist=watchlist,&#xA;                       early_stopping_rounds=75)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I found this article very informative with regards to early stopping and xgboost &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://codingwiththomas.blogspot.se/2016/03/xgboost-validation-and-early-stopping.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost Validation and Early Stopping in R&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50689" LastActivityDate="2018-04-18T22:59:14.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30496" PostTypeId="1" CreationDate="2018-04-18T23:18:52.167" Score="6" ViewCount="324" Body="&lt;p&gt;I am aware that an ensemble machine learning model is a stack of two or more machine learning models. Is there a word to refer to those individual models that go into the ensemble model? (i.e. a specific term/jargon?) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wish to refer to them and am not aware of any specific terminology to describe them. I've considered: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;sub-models&quot; &lt;/li&gt;&#xA;&lt;li&gt;&quot;component models&quot; &lt;/li&gt;&#xA;&lt;li&gt;even &quot;pancake models&quot; (since people&#xA;generally understand what's being referring to when they hear 'one of the&#xA;pancakes in the model')&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45768" LastEditorUserId="45768" LastEditDate="2018-04-18T23:41:09.497" LastActivityDate="2018-05-11T14:20:57.277" Title="What are the individual models within a machine learning ensemble called?" Tags="|machine-learning|ensemble-modeling|machine-learning-model|" AnswerCount="3" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30497" PostTypeId="2" ParentId="30496" CreationDate="2018-04-18T23:39:01.853" Score="5" Body="&lt;p&gt;I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer &quot;components&quot;, &quot;individual/constituent models&quot;, or something like that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you definitely want to find a &quot;correct&quot; term, one way to discover it (if it exists) is to look into an ensemble learning early paper. &#xA;To arrive at one, a good way is to search for a survey paper (e.g. &lt;a href=&quot;https://link.springer.com/content/pdf/10.1007/s10462-009-9124-7.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) and go on a journey to the past to find that original study.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a quick skim of the above, you might want to look at Exploratory Data Analysis (Tukey 77), if you manage to find the pdf.&lt;/p&gt;&#xA;" OwnerUserId="50765" LastEditorUserId="34269" LastEditDate="2018-04-19T03:15:40.757" LastActivityDate="2018-04-19T03:15:40.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30498" PostTypeId="2" ParentId="9325" CreationDate="2018-04-19T00:14:29.887" Score="3" Body="&lt;p&gt;Also take a look at &lt;a href=&quot;http://scikit.ml/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-multilearn&lt;/a&gt;. It is a very good library that extends sklearn for multi-label learning. However, I'm not sure how the confusion matrix works for multi-label problems...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This guy &lt;a href=&quot;https://stackoverflow.com/questions/25503260/build-confusion-matrix-for-multiclass-multilabel-classification&quot;&gt;claims&lt;/a&gt; he has solved it.&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-19T00:14:29.887" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30500" PostTypeId="1" CreationDate="2018-04-19T00:54:30.843" Score="1" ViewCount="8351" Body="&lt;p&gt;I want to plot some data that contains timedeltas. However, the numpy &lt;code&gt;plot()&lt;/code&gt; and &lt;code&gt;errorbar()&lt;/code&gt; return:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError: float() argument must be a string or a number, not 'Timedelta'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The question is how can I efficiently convert the timedeltas to a float or integer number. E.g. when I use minutes as base:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 days 01:30:00 -&amp;gt; 90&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;or hours&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 days 01:30:00 -&amp;gt; 1.5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similar to &lt;code&gt;Series.dt.days&lt;/code&gt; which would return 0 in this case.&lt;/p&gt;&#xA;" OwnerUserId="26202" LastActivityDate="2020-10-30T11:30:33.327" Title="How to turn a series that contains pandas.datetime.timedelta to floats e.g. hours or days?" Tags="|python|pandas|numpy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30501" PostTypeId="2" ParentId="30500" CreationDate="2018-04-19T00:57:59.713" Score="2" Body="&lt;p&gt;One way to do this would be to use &lt;code&gt;Series.dt.seconds&lt;/code&gt; and ´Series.dt.days´ and  multiply with a factor for the desired unit:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(Series.dt.seconds/3600) + (Series.dt.days*24)  # for values with [hours]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26202" LastEditorUserId="26202" LastEditDate="2018-04-19T01:08:09.847" LastActivityDate="2018-04-19T01:08:09.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30502" PostTypeId="2" ParentId="30475" CreationDate="2018-04-19T01:12:15.943" Score="1" Body="&lt;p&gt;EDIT : &lt;a href=&quot;https://arxiv.org/abs/1804.06655v1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Face Recognition: A Survey&lt;/a&gt; New on arxiv 4/18/2018 looks like best survey of methods over face related tasks :).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Beyond Facenet, there are a few approaches which may be good to look at how many faces do you intend to have your system know about - aka netowrks that have a network to directly output which face it is (~ &amp;lt;10K), vs a feature map for clustering (~ 10k to 100K), or comparing any 2 faces (~ &gt;100K) - below are examples of approaches to each...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper just came out:&#xA;&lt;a href=&quot;https://arxiv.org/pdf/1804.03487v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Exploring Disentangled Feature Representation Beyond Face Identification- 2018&lt;/a&gt; - Reported Accuracy 99.816 . Use an encoder-decoder like scheme to compute features of a face. Then given all the faces you computed this on, do clustering to find which ones are the same face (TSNE - distance in feature space). This paper is cool since it also gives features of each face like 'smiling', and these are used to augment the search. Along similar line is &lt;a href=&quot;https://arxiv.org/pdf/1803.11366v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before that (other than Facenet) - &lt;a href=&quot;https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf?&quot; rel=&quot;nofollow noreferrer&quot;&gt;DeepFace&lt;/a&gt; - Accuracy 97.35. Its the facebook lib. If its state-of-the-art enough for them, its state of the art enough for me. Approach is given two images, put them into siamese network, for first detect, then 3d model the face, then project to 2d featuremap, which then combine to label saying whether they are same person or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1509.00244.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Robust Face Recognition via Multimodal Deep&#xA;Face Representation&lt;/a&gt; - 2016, Reported accuracy 98.43. This is interesting because they trained on a relatively small dataset (CASIA WebFace). However this has the last layer being the number of identities in the dataset - so this could be a llimiting factor if you want to recognize millions of identities like facebook does. Otherwise, this looks easiest to implement/mess with.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1804.01417v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Patch-based Face Recognition using a Hierarchical&#xA;Multi-label Matcher&lt;/a&gt; - not sure what is going on here, but looks interesting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would think another limiting factor for you is how many examples do you have per face - eg do you have 4K identities and 4mil images like the facebok dataset, or 10k identities and .5mil images (CASIA WebFace), or LFW with like ~5K identities and ~15k images. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-04-24T06:22:47.393" LastActivityDate="2018-04-24T06:22:47.393" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30504" PostTypeId="2" ParentId="853" CreationDate="2018-04-19T02:02:47.127" Score="5" Body="&lt;p&gt;In this 2014 &lt;a href=&quot;http://ir.hit.edu.cn/~jguo/papers/emnlp2014-semiemb.pdf&quot; rel=&quot;noreferrer&quot;&gt;paper&lt;/a&gt; (&lt;a href=&quot;https://github.com/jiangfeng1124/emnlp14-semi&quot; rel=&quot;noreferrer&quot;&gt;GitHub&lt;/a&gt;), the authors compared multiple strategies of incorporating word embeddings in a CRF-based NER system, including dense embedding, binerized embedding, cluster embedding, and a novel &lt;em&gt;prototype&lt;/em&gt; method. Using dense vectors directly as suggested by vlad is the most straightforward way but also the least effective in multiple evaluations. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I implemented the prototype idea in my domain-specific NER project and it works pretty well for me. &lt;/p&gt;&#xA;" OwnerUserId="50768" LastActivityDate="2018-04-19T02:02:47.127" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30506" PostTypeId="2" ParentId="17146" CreationDate="2018-04-19T05:32:45.997" Score="1" Body="&lt;p&gt;As you said, you cannot prove mathematically that esembling increases performance, but it generally does. That's reason why gradient boosting and random forests are so popular in kaggle competitions, because they outperform what a decision tree can learn in many ways.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a curiosity, even Neural Networks can be used as &quot;weak&quot; learners, as can be seen in &lt;a href=&quot;https://arxiv.org/abs/1704.00109&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1704.00109&lt;/a&gt;. So, ensembling is a very powerful technique that can be applied in many areas of machine learning. The main problem is that ensembles are not easily interpretable, being way more black-boxy than its weak learners.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T05:32:45.997" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30508" PostTypeId="2" ParentId="30496" CreationDate="2018-04-19T06:17:32.840" Score="3" Body="&lt;p&gt;I have heard people calling them &quot;weak learners&quot; many times, but this is only when they are not very acurate themselves.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T06:17:32.840" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30509" PostTypeId="2" ParentId="30482" CreationDate="2018-04-19T06:30:05.240" Score="1" Body="&lt;p&gt;No. If, as you said, the variables are categorical, performing a scaling does not make any sense. Plain LabelEncoder already does what you want.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T06:30:05.240" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30510" PostTypeId="1" AcceptedAnswerId="30511" CreationDate="2018-04-19T07:48:18.650" Score="0" ViewCount="154" Body="&lt;p&gt;I built a model for time series in order to forecast new values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way to choose the correct model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are better criteria like AIC and BIC or the comparison between prediction errors? In this last case I have to split dataset in train and test while in the first case is not necessary, reducing time of implementation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50074" LastEditorUserId="40853" LastEditDate="2018-04-19T08:29:16.470" LastActivityDate="2018-04-19T08:29:16.470" Title="Choice of time series models" Tags="|time-series|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30511" PostTypeId="2" ParentId="30510" CreationDate="2018-04-19T08:18:14.317" Score="2" Body="&lt;p&gt;First thing first, when ever you use Time Series data you call it as Forecasting not Prediction as it is time dependent. To understand why you can go through this &lt;a href=&quot;https://stats.stackexchange.com/questions/65287/difference-between-forecast-and-prediction&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;Metrics to compare models&lt;/h1&gt;&#xA;&lt;p&gt;When you are trying to compare between models you need to use AIC,BIC, AUC etc values. you can go though this &lt;a href=&quot;https://stats.stackexchange.com/questions/81427/aic-guidelines-in-model-selection&quot;&gt;link&lt;/a&gt; to understand better&lt;/p&gt;&#xA;&lt;h1&gt;Metrics to Access the Model&lt;/h1&gt;&#xA;&lt;p&gt;When you are accessing the performance of the model then you need to check for Error Rates(RMSE, MAE, MAPE, MSE etc). Yes, in this case you need to divide the data into Train and Test to access the model. You can go through this &lt;a href=&quot;http://www.edscave.com/forecasting---time-series-metrics.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; for better understanding&lt;/p&gt;&#xA;&lt;h1&gt;Improve the Forecast&lt;/h1&gt;&#xA;&lt;p&gt;To take it to the next level, you can use an ensemble to get better the result. This might or might not decrease the error rate. In most of the cases it is helpful. You can combine 2/3 moderately performing hotels outcome to get best results i.e., Ensemble Model.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-19T08:18:14.317" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30513" PostTypeId="1" CreationDate="2018-04-19T09:07:00.963" Score="4" ViewCount="3032" Body="&lt;p&gt;I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding traffic and I fetch them with a PHP script whenever this is appropriate depending on the interaction of the user with the chatbot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder how to deal with the case when the user asks variations of the same question which therefore can be answered with the same answer.&#xA;For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why is there traffic at High Street?&lt;/li&gt;&#xA;&lt;li&gt;What is the cause of traffic at High Street?&lt;/li&gt;&#xA;&lt;li&gt;Why did I encounter traffic at High Street?&lt;/li&gt;&#xA;&lt;li&gt;I am stuck in traffic at High Street. Why is this?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously, I can start by removing stopwords (e.g. did), by naming entities (e.g. road -&gt; High Street), by defining synonyms and by applying a text similarity measure (e.g. Levenshtein distance etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I feel like reinventing the wheel if I do this. Therefore, my question is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Are there any APIs which can compare strings in terms of semantic similarity (without even requiring training)?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there software platforms such as Dialogflow which are suitable for these tasks but still you must explicitly state all the variations of the same question so that you will get the same answer. Therefore, I look for a API where you will explicitly state only one of these variations of the same question (e.g. Why is there traffic at High Street?) and then the API will figure out by itself which other variations are identical to it in terms of meaning or not.&lt;/p&gt;&#xA;" OwnerUserId="45610" LastEditorUserId="45610" LastEditDate="2018-04-19T15:30:07.570" LastActivityDate="2020-02-05T22:13:46.887" Title="Are there any good NLP APIs for comparing strings in terms of semantic similarity?" Tags="|machine-learning|python|nlp|similarity|software-recommendation|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30514" PostTypeId="1" AcceptedAnswerId="30515" CreationDate="2018-04-19T09:49:05.340" Score="1" ViewCount="867" Body="&lt;p&gt;I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with and how RF imputation works in PySpark. &lt;/p&gt;&#xA;" OwnerUserId="44701" LastActivityDate="2018-04-19T10:21:11.237" Title="Use of Random Forest algorithm in PySpark for imputation" Tags="|random-forest|pyspark|data-imputation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30515" PostTypeId="2" ParentId="30514" CreationDate="2018-04-19T10:21:11.237" Score="1" Body="&lt;p&gt;You can do the following: use all the other features as input and the missing data as the label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Train using all the rows that have the column filled with data and classify the others that don't. Use the values predicted by the Random Forest as the value of that field on the subsequent models and transformations.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastActivityDate="2018-04-19T10:21:11.237" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30516" PostTypeId="1" AcceptedAnswerId="31851" CreationDate="2018-04-19T10:22:48.633" Score="0" ViewCount="5335" Body="&lt;p&gt;I am currently working on a sentiment analysis project where the end goal is to try and predict who wins the Nigerian 2019 presidential election. My goal is to create a model that analyses tweets and detects the tone of a tweet (whether positive, negative or neutral). This model can in turn be used to predict who wins. Here's what I have done so far.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have a program running on a server streaming tweets daily&lt;/li&gt;&#xA;&lt;li&gt;I have labelled about 5000 tweets manually into different categories &#xA;(positive, negative and neutral tweets).&lt;/li&gt;&#xA;&lt;li&gt;I previously used text blob to determine sentiment on each tweet. But it &#xA;was not accurate enough.&lt;/li&gt;&#xA;&lt;li&gt;I was then advised to train a model on the way Nigerians speak on twitter, &#xA;which can sometimes be a little different from normal English&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;From my research, I found out I have to use feature extraction before I can train the model with my labelled data. My problem now is that I don't understand how to go about these feature extraction. I have read materials about what people use (bag of words, bigrams, unigrams, part of speech tags etc). I don't understand the explanation I got so far, how it all comes together or which to use. Please could anyone explain these in layman's terms or point me to resources that explains this in layman's terms. Thank you&lt;/p&gt;&#xA;" OwnerUserId="50783" LastEditorUserId="50783" LastEditDate="2018-04-19T10:34:59.887" LastActivityDate="2018-05-19T13:34:33.083" Title="How does one go about feature extraction for training labelled tweets for sentiment analysis?" Tags="|machine-learning|python|data-mining|sentiment-analysis|twitter|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30517" PostTypeId="2" ParentId="30516" CreationDate="2018-04-19T10:39:56.847" Score="0" Body="&lt;p&gt;I'll throw at you some standard names that the are you're building a project on has, every term that you don't understand, research it, it'll improve your knowledge a lot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, there are great resources out there on NLP (Natural Language Processing)regarding classification problems such as this one: &lt;a href=&quot;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which algorithm to use? &lt;strong&gt;It depends on your data&lt;/strong&gt;, there are two topics that you have to pay attention: the over or underfitting of your model and the distribution of your classes, those two things if managed correctly can make your model way better than those that don't look at it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try to explore your data, plot the distribution of the classes based on a single word, two words, a phrase, a hashtag, or something like that, try to see if they can describe one or more classes, your model will probably improving by doing that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are two articles about feature extraction on text data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13370" LastActivityDate="2018-04-19T10:39:56.847" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30518" PostTypeId="2" ParentId="19092" CreationDate="2018-04-19T10:42:55.190" Score="0" Body="&lt;p&gt;The theoretical estimation of the error depends on the data, and the fact that the labels can be predicted by the variable X. There's no way to a priori know which will be the best option regarding the two models, so just try.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I would bet that the second option will work better if y2 is correlated in some way to y1, and I would bet for the first option if they are independent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when you train your model, I highly recommend sampling the data in a more balanced way, so that the model not learns to predict (0, 0) always regardless of the values of X (as a model that does these dummy predictions would have high accuracy). Another way to work around the imbalanced classes is to use AUC in ROC as a metric to choose among different models.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T10:42:55.190" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30519" PostTypeId="1" AcceptedAnswerId="30520" CreationDate="2018-04-19T10:52:14.350" Score="2" ViewCount="416" Body="&lt;p&gt;What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of bedrooms, living rooms and bathrooms. I want to identify outliers in this data, where some of the variables are discrete and some of them are continuous. Any ideas on how to do this?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-26T11:53:54.467" LastActivityDate="2021-08-29T03:58:49.483" Title="General methods outlier detection" Tags="|machine-learning|statistics|anomaly-detection|outlier|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30520" PostTypeId="2" ParentId="30519" CreationDate="2018-04-19T11:08:08.827" Score="2" Body="&lt;p&gt;&lt;code&gt;Dbscan&lt;/code&gt; seems a great choice for you, look at &lt;code&gt;scikit-learn&lt;/code&gt; &lt;a href=&quot;https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation&lt;/a&gt; for further discovery.&lt;/p&gt;&#xA;&lt;p&gt;About being discrete or continuous, it actually doesn't matter, what you have to look at it is if the scale is the best suited for the algorithm in hand (and &lt;code&gt;scikit-learn&lt;/code&gt; has algorithms to handle that).&lt;/p&gt;&#xA;&lt;p&gt;Another tip is to actually see if the attributes fit on a distribution, some of them might, and parametric methods of detecting outliers are better suited for the task.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="85045" LastEditDate="2021-08-29T03:58:49.483" LastActivityDate="2021-08-29T03:58:49.483" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30522" PostTypeId="1" AcceptedAnswerId="30554" CreationDate="2018-04-19T12:53:05.640" Score="1" ViewCount="79" Body="&lt;p&gt;I have the following 2 class data, as shown below. &lt;a href=&quot;https://i.stack.imgur.com/yr734.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yr734.png&quot; alt=&quot;figure&quot;&gt;&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Its a hand crafted example using two ellipse equations. I want to know what might be a recommended kernel to be used with this problem if I want to use an SVM based binary classifier.&lt;/p&gt;&#xA;" OwnerUserId="8528" LastActivityDate="2018-04-20T07:01:22.620" Title="What Kernel is suitable for the following data for SVM classification?" Tags="|classification|svm|binary|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30523" PostTypeId="1" CreationDate="2018-04-19T12:55:38.637" Score="1" ViewCount="314" Body="&lt;p&gt;I am new into Caffe and I have some trouble in understanding how should I create a dataset for object detection. I have downloaded some images and bounding boxes from ImageNet. The bounding boxes are in Pascal VOC format (xml files). As I understood, this is not the only format for bounding boxes. However, I want to build a dataset that can be fed to a Caffe network. Again, I know that there are several suitable formats. For now, I am not interested in performance. What is the easiest method to build the dataset?&lt;/p&gt;&#xA;" OwnerUserId="45420" LastActivityDate="2018-04-19T12:55:38.637" Title="How to build a Caffe dataset for object detection with ImageNet data" Tags="|dataset|caffe|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30524" PostTypeId="2" ParentId="26533" CreationDate="2018-04-19T14:29:01.560" Score="1" Body="&lt;p&gt;Of course, this makes complete sense, as long as all the other variables are the same. In fact, if you think of logistic regression as a linear regression with the log-odds of the output variable (as in &lt;a href=&quot;http://www.wright.edu/~thaddeus.tarpey/ES714glm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wright.edu/~thaddeus.tarpey/ES714glm.pdf&lt;/a&gt;), then it is trivial that comparing the weights is the same as comparing them in linear regression, which makes perfect sense.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T14:29:01.560" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30525" PostTypeId="1" AcceptedAnswerId="38663" CreationDate="2018-04-19T14:59:49.990" Score="2" ViewCount="1042" Body="&lt;p&gt;Seems like &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/009813549280051A&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autoassociative neural networks&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoencoder&quot; rel=&quot;nofollow noreferrer&quot;&gt;Autoencoders&lt;/a&gt; do the same thing.&lt;/p&gt;&#xA;&lt;p&gt;Are they the same, or is there some subtlety I'm not picking up on?&lt;/p&gt;&#xA;" OwnerUserId="21232" LastEditorUserId="85045" LastEditDate="2021-01-17T17:11:23.373" LastActivityDate="2021-01-17T17:11:23.373" Title="what's the difference between autoencoder and autoassociative neural networks?" Tags="|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30530" PostTypeId="1" AcceptedAnswerId="30536" CreationDate="2018-04-19T16:43:44.833" Score="0" ViewCount="715" Body="&lt;p&gt;Going through this &lt;a href=&quot;http://shop.oreilly.com/product/0636920052289.do&quot; rel=&quot;nofollow noreferrer&quot;&gt;book&lt;/a&gt;, I am familiar with the following:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For each training instance the backpropagation algorithm first makes a&#xA;  prediction (forward pass), measures the error, then goes through each&#xA;  layer in reverse to measure the error contribution from each&#xA;  connection (reverse pass), and finally slightly tweaks the connection&#xA;  weights to reduce the error.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However I am not sure how this differs from the reverse-mode autodiff implementation by TensorFlow. As far as I know the above algorithm first goes through the graph in the forward direction and then in the second pass computes all partial derivatives for the outputs with respect to the inputs. This is very similar to the propagation algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;How does backpropagation differ from reverse-mode autodiff ?&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="49032" LastActivityDate="2018-04-19T19:27:29.527" Title="How does backpropagation differ from reverse-mode autodiff" Tags="|machine-learning|deep-learning|tensorflow|backpropagation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30531" PostTypeId="1" CreationDate="2018-04-19T16:59:44.193" Score="2" ViewCount="1487" Body="&lt;p&gt;I'm starting a very beginner tutorial for DS/ML. One of the first things to do is to poke at the data they gave me. After doing the typical head() and describe() in pandas, I want to look more carefully at the features that are numerical, to see if there's any correlation between them and the labels (for example, a simple linear relationship).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used matplotlib and stuff before, but I want to do this the &quot;correct&quot; way. The tutorial I'm looking at suggests using seaborn, but to be honest it seems difficult to apply to what I'm suggesting (they're looking at more complicated relationships/etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use matplotlib, but I've never found a smooth, seamless way of making a grid of several plots simultaneously. Doing the &quot;subplot&quot; thing was really clunky. Is there a better way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like to do is, let's say I have 4 numerical features, and I want to plot each of them (or their average, etc) vs the label, and display it, so I can quickly see patterns. What's the best way to do this?&lt;/p&gt;&#xA;" OwnerUserId="50805" LastActivityDate="2018-04-20T07:21:22.573" Title="What's the easiest way to quickly plot several features simultaneously for new data?" Tags="|plotting|seaborn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30532" PostTypeId="1" CreationDate="2018-04-19T18:00:11.290" Score="6" ViewCount="305" Body="&lt;p&gt;I read this introduction about AdaBoost (&lt;a href=&quot;http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf&lt;/a&gt;), and am curious why confidence for each model is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\alpha_j=\frac{1}{2}\log\frac{1-\epsilon_j}{\epsilon_j}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(from page 48 of &lt;a href=&quot;http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf&quot; rel=&quot;noreferrer&quot;&gt;http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone provide any source for explaining this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="50440" LastActivityDate="2023-07-06T12:45:22.400" Title="Confidence value in AdaBoost?" Tags="|machine-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30533" PostTypeId="2" ParentId="30530" CreationDate="2018-04-19T18:32:59.267" Score="0" Body="&lt;p&gt;They are more or less specify the same thing. I think the difference is that back-propagation refers to the updating of weights with respect to their gradient to minimize a function; &quot;back-propagating the gradients&quot; is a typical term used. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conversely, reverse-mode diff merely means calculating the gradient of a function. Nothing here in this process implies that we aim to the weights/values of a function. &lt;/p&gt;&#xA;" OwnerUserId="37793" LastActivityDate="2018-04-19T18:32:59.267" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30534" PostTypeId="1" CreationDate="2018-04-19T19:02:14.190" Score="1" ViewCount="94" Body="&lt;p&gt;In &lt;a href=&quot;https://static.googleusercontent.com/media/research.google.com/de//pubs/archive/46180.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google Vizier: A Service for Black-Box Optimization&lt;/a&gt; it is show what kind of optimizers google uses internally.&#xA;Figure 6 shows some benchmark results compared to Random Search. Am I right in understanding that &quot;GP Bandit&quot; has better results as &quot;CMA-ES&quot; in 8 Dimensions? So is a lower number better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S6BY0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S6BY0.png&quot; alt=&quot;Figure 6 from the paper&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="50810" LastActivityDate="2018-04-19T19:02:14.190" Title="results from &quot;Google Vizier: A Service for Black-Box Optimization&quot;" Tags="|optimization|gaussian|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30535" PostTypeId="2" ParentId="26409" CreationDate="2018-04-19T19:16:06.973" Score="2" Body="&lt;p&gt;You can do the following:&lt;/p&gt;&#xA;&lt;p&gt;A Pipeline can be made of other pipelines. Isn't that great?&#xA;A &lt;a href=&quot;http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.Pipeline&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pipeline&lt;/a&gt; inherit from the Estimator class and by definition, a &lt;a href=&quot;http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.PipelineStage&quot; rel=&quot;nofollow noreferrer&quot;&gt;PipelineStage&lt;/a&gt; can be either an Estimator or a Transformer.&lt;/p&gt;&#xA;&lt;p&gt;This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform on the DataFrame.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="85045" LastEditDate="2021-05-28T12:47:43.933" LastActivityDate="2021-05-28T12:47:43.933" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30536" PostTypeId="2" ParentId="30530" CreationDate="2018-04-19T19:27:29.527" Score="0" Body="&lt;p&gt;Thanks to the above answer for the valid contribution however I have found the answer to this question by the author of the book himself:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Bakpropagation refers to the whole process of training an artificial neural network using multiple backpropagation steps, each of which computes gradients and uses them to perform a Gradient Descent step. In contrast, reverse-mode auto diff is simply a technique used to compute gradients efficiently and it happens to be used by backpropagation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="49032" LastActivityDate="2018-04-19T19:27:29.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30537" PostTypeId="1" CreationDate="2018-04-19T19:37:05.040" Score="7" ViewCount="2187" Body="&lt;p&gt;I am new to DL and Keras. I am trying to solve a regression problem with multivariate outputs (y shape (?, 2)) using Keras (tensorflow backend). I am having my confusion about how the loss is calculated. I use mean absolute error as the loss function. However, since my target data has 2 dimensions, is the loss value calculated as the reduced mean on all dimensions (a scalar as the result)? I checked the Keras source code, it uses K.mean(..., axis=-1) for MAE calculation. If K.mean is the same to numpy.mean, &quot;axis=-1&quot; should do the column mean (for my case, it should return a tensor with shape (?,2) but not a scalar).  If this is the case, how could the loss value be a single number (as outputed in the training process log)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the MAE return is indeed a scalar (reduced mean), this gives me another problem. The data from each dimension of my target is not in a same range. A reduced mean would be biased towards the high value dimension. Shall I change my model to a multi-task learning model then?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot for your help on this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;L. &lt;/p&gt;&#xA;" OwnerUserId="50809" LastActivityDate="2020-03-24T14:01:28.360" Title="Keras Loss Function for Multidimensional Regression Problem" Tags="|keras|tensorflow|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30538" PostTypeId="2" ParentId="19840" CreationDate="2018-04-19T20:19:07.080" Score="1" Body="&lt;p&gt;Is this &lt;a href=&quot;http://www.statsmodels.org/dev/imputation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;statsmodels MICE implementation&lt;/a&gt; an option?  I also heard that an implementation of MICE is being merged into the development branch of scikit-learn but I can't find any info on that.&lt;/p&gt;&#xA;" OwnerUserId="50811" LastActivityDate="2018-04-19T20:19:07.080" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30539" PostTypeId="2" ParentId="20321" CreationDate="2018-04-19T22:03:45.970" Score="1" Body="&lt;p&gt;You are right, increasing the dropout proportion will help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this looks like a setting where early stopping will be a very good choice. Although dropout, weight decay and batch-norm can work propperly, the fact that you easily overfit your training set would make it an appropiate scenario to try early stopping.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, as the neural network takes very short to be trained you can train many of them (on some subset of the training set, making them weak learners) and create an ensemble to make the final predictions.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T22:03:45.970" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30540" PostTypeId="2" ParentId="22002" CreationDate="2018-04-19T22:14:48.050" Score="0" Body="&lt;p&gt;Usually it helps to have data centered at 0 and with standard deviation 1. I would reescale it such thar its standard deviation is 1. Appart from that, everything looks good.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-19T22:14:48.050" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30541" PostTypeId="2" ParentId="30531" CreationDate="2018-04-19T22:45:26.837" Score="1" Body="&lt;p&gt;I believe &lt;a href=&quot;https://seaborn.pydata.org/generated/seaborn.pairplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;seaborn's pairplot&lt;/a&gt; is what you are looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://seaborn.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Seaborn&lt;/a&gt; is a plotting library built on top of matplotlib for exactly the reason you want. To make quick aesthetically pleasing graphs for data analysis. Seaborn also work very well with pandas DataFrames.&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-19T22:45:26.837" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30542" PostTypeId="1" CreationDate="2018-04-19T22:57:19.167" Score="-3" ViewCount="71" Body="&lt;p&gt;I have trained a CNN, in Keras, to remove noise from an image, the input shape is (5,5) and I trained it by using patches from an image with noise with the expected output for the center pixel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To enumerate, the input shape is (5,5) the output is one value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to use my CNN on a full, large, image. Is there any way to take my input image (1024x1024) and run it through the CNN without having to split this image into 5x5 patches? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50815" LastActivityDate="2018-04-20T01:40:48.193" Title="Using small CNN for de-noising on a full image" Tags="|python|deep-learning|keras|cnn|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30543" PostTypeId="1" CreationDate="2018-04-20T00:06:27.417" Score="2" ViewCount="729" Body="&lt;p&gt;I have developed a model in Keras that works perfectly when reading data stored locally.  However, I now want to take advantage of Google Cloud Platform's GPUs for training the model.  I have set up the GPU on GCP and am working in a Jupyter notebook.  I have moved my images to Google Cloud Storage.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I access these images (specifically the directories -  training, validation, test) directly from Cloud Storage using the Keras' flow_from_directory method of the &lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;ImageDataGenerator&lt;/a&gt; class?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;here's my directory structure in Google Cloud Storage (GCS):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mybucketname/&#xA;      class_1/&#xA;          img001.jpg&#xA;          img002.jpg&#xA;          ...&#xA;      class_2/&#xA;          img001.jpg&#xA;          img002.jpg&#xA;          ...&#xA;      class_3/&#xA;          img001.jpg &#xA;          img002.jpg&#xA;          ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10548" LastActivityDate="2019-06-14T17:58:55.313" Title="How to access images directly from Google Cloud Storage (GCS) when using Keras?" Tags="|keras|tensorflow|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30544" PostTypeId="2" ParentId="30092" CreationDate="2018-04-20T00:39:44.373" Score="2" Body="&lt;p&gt;Without knowing the geometry of your data, determining a kernel is generally achieved through trial and error.  The radial basis function (RBF) kernel is a good starting choice because most data are not linearly separable.  Fortunately training an SVM is fast, so brute-forcing the kernel search is not a terrible method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For selecting the C and gamma values, we generally use a grid search:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gBXjH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gBXjH.png&quot; alt=&quot;example from scikit-learn&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Every cell in the graph represents a SVM model trained on your data given the respective C and gamma parameters.  The color represents its accuracy depicted by the scale on the right.  Note that our scales are logarithmic, so we are able to search a broad solution space while only training 169 models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this example, we can predict that the optimal (C,gamma) value combination would be in the region around $( 10, 0.1 )$ or $( 10^{10}, 1e^{-7}) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can further refine our C and gamma parameters by &quot;zooming&quot; into the regions with greatest validation accuracy.  In other words, we can repeat the grid search with finer granularity and with boundaries determined by the regions of interest until we find the optimal C/gamma combination.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-20T00:39:44.373" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30545" PostTypeId="1" CreationDate="2018-04-20T01:14:54.007" Score="1" ViewCount="9792" Body="&lt;p&gt;I have been implementing a DecisionTreeRegressor model in Anaconda environment with a data set sourced from a 20 million row, 12-dimensional CSV file. I could get the chunks off of the data set with chunksize set to 500,000 rows and process the computation of the R-Squared score on the training/test split data sets in each iteration of 500,000 rows till iteration #20. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sklearn.__version__: 0.19.0 &#xA;pandas.__version__: 0.20.3 &#xA;numpy.__version__: 1.13.1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;GridSearchCV()&lt;/code&gt; instance uses parameter grid with parameter &lt;code&gt;max_depth&lt;/code&gt; set to values [4, 6]. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then see memory errors in numpy module with the Anaconda Python interpreter throwing an exception. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the exception after iteration #20: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Score for model trained on Test Dataset:  -0.000287864727209&#xA;Best Parameters:  {'max_depth': 4} &#xA;Best Cross-Validation Accuracy:  -0.00037759422675 &#xA;&#xA;Traceback (most recent call last):&#xA;&#xA;  File &quot;&amp;lt;ipython-input-1-a28a1b71d60d&amp;gt;&quot;, line 1, in &amp;lt;module&amp;gt;&#xA;    runfile('C:/Kal/Stat-Work/Stat-Code/SciKit/Final/DecisionTreeRegression-MaximumDepthFour.py', wdir='C:/Kal/Stat-Work/Stat-Code/SciKit/Final')&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py&quot;, line 710, in runfile&#xA;    execfile(filename, namespace)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py&quot;, line 101, in execfile&#xA;    exec(compile(f.read(), filename, 'exec'), namespace)&#xA;&#xA;  File &quot;C:/Kal/Stat-Work/Stat-Code/SciKit/Final/DecisionTreeRegression-MaximumDepthFour.py&quot;, line 21, in &amp;lt;module&amp;gt;&#xA;    for piece in chunker:&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 978, in __next__&#xA;    return self.get_chunk()&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 1042, in get_chunk&#xA;    return self.read(nrows=size)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py&quot;, line 1023, in read&#xA;    df = DataFrame(col_dict, columns=columns, index=index)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py&quot;, line 275, in __init__&#xA;    mgr = self._init_dict(data, index, columns, dtype=dtype)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py&quot;, line 411, in _init_dict&#xA;    return _arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py&quot;, line 5506, in _arrays_to_mgr&#xA;    return create_block_manager_from_arrays(arrays, arr_names, axes)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals.py&quot;, line 4309, in create_block_manager_from_arrays&#xA;    blocks = form_blocks(arrays, names, axes)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals.py&quot;, line 4381, in form_blocks&#xA;    int_blocks = _multi_blockify(int_items)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals.py&quot;, line 4450, in _multi_blockify&#xA;    values, placement = _stack_arrays(list(tup_block), dtype)&#xA;&#xA;  File &quot;C:\Users\bkalahas\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals.py&quot;, line 4493, in _stack_arrays&#xA;    stacked = np.empty(shape, dtype=dtype)&#xA;&#xA;MemoryError&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Read the dataset into a DataFrame from the Test Regression CSV file. &#xA;chunker = pd.read_csv('C:/Kal/Stat-Work/Stat-Code/SciKit/Test_Data_Set_Regression.csv',&#xA;             names=['ROW', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'RT'],&#xA;             chunksize=500000, low_memory=False)&#xA;&#xA;for piece in chunker:&#xA;    # Create Training and Test Datasets. &#xA;    X_train, X_test, y_train, y_test = train_test_split(&#xA;        piece[['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']], piece['RT'], random_state=0)&#xA;&#xA;    param_grid = {&#xA;                 'max_depth': [4, 6]&#xA;                 }&#xA;&#xA;    # Instantiate the GridSearchCV class with the model DecisionTreeRegressor, the parameter&#xA;    # grid to search param_grid, and the cross-validation strategy we want to use, &#xA;    # say 5 fold (stratified) cross-validation.&#xA;    grid_search = GridSearchCV(DecisionTreeRegressor(), param_grid, cv=5)&#xA;&#xA;    # Call fit method to run cross-validation for each combination&#xA;    # of parameters we specified in param_grid&#xA;    grid_search.fit(X_train, y_train)&#xA;&#xA;    # Print the best_score, best_parameters and the test_score.&#xA;    print(&quot;Score for model trained on whole Training Dataset: &quot;, grid_search.score(X_train, y_train))&#xA;&#xA;    # Evaluate the generalization performance by calling score method on the Test Dataset.&#xA;    print(&quot;Score for model trained on Test Dataset: &quot;, grid_search.score(X_test, y_test))&#xA;    print(&quot;Best Parameters: &quot;, grid_search.best_params_)&#xA;    print(&quot;Best Cross-Validation Accuracy: &quot;, grid_search.best_score_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Please point to ways to overcome the Python memory error&#xA;exception.&lt;/li&gt;&#xA;&lt;li&gt;What is the best way to implement a DecisionTreeRegressor model on a&#xA;cluster of 4 16-GB RAM, 2.5-GHz CPU machines (linux or windows)? I&#xA;see memory errors with scale even with DecisionTreeRegressor model.&#xA;SVM model does not even compute fully beyond 20,000 rows in a chunk&#xA;with hung CPUs being seen by the Python interpreter. But, it is&#xA;known that SVM is computationally tedious. Is there a way around&#xA;such memory errors for DecisionTree and Ensemble models when&#xA;combined with Pandas? Pandas is our memory data analytic engine.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The dataset is particularly huge (actual total of 100 million rows in source database as produced almost everyday). Using cross validation against the entire 100 million row dataset is computationally onerous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have come up with this Bagging-like model where the dataset is broken into samples of 500,000 rows each and the MSE is computed for each sample. The goal is to compute the averaged out MSE across all samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This question is important since it deals with scale which is an everyday computational problem in ML algorithms. I would also appreciate critical answers on various aspects of my code above versus down votes without reason. Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50820" LastEditorUserId="29575" LastEditDate="2020-04-11T20:44:29.810" LastActivityDate="2024-02-14T23:06:10.707" Title="How to avoid memory error with Pandas pd.read_csv method call with GridSearchCV usage for DecisionTreeRegressor model?" Tags="|python|scikit-learn|pandas|decision-trees|ensemble-modeling|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30546" PostTypeId="2" ParentId="17146" CreationDate="2018-04-20T01:21:04.823" Score="2" Body="&lt;p&gt;Under Ensemble you can use Majority Votes, Average, Weights etc to get the final outcome from Ensemble model. To understand it better you can go through this &lt;a href=&quot;https://stats.stackexchange.com/questions/18891/bagging-boosting-and-stacking-in-machine-learning&quot;&gt;Link&lt;/a&gt;, explained well by Alexander.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, let us consider that you have 3 models which has an accuracy of 65-70%. Now by stacking these 3 models there is very high chance that you models accuracy would increase. In another scenario you have 3 models model-1: 95%, model-2: 55%, model-3: 45% accuracy, then if you stack them then there is a very good chance it can worsen the result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Conclusion, it all depends on the individual models performance, Ensemble performs well when you combine moderately performing models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically there is no proof saying that this method is suitable for this scenario but trail and error might help you to get good results. It is subjective to the business scenario. Similarly, for bagging and boosting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my experience with Bagging when the model accuracy is bad, I tried using bagging to fit the data better but EOD training accuracy(20% to 10% approx) was decreased but test accuracy was worsened(11% to 20% approx). So, you have to decide which suits your business problem better and take it forward.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastActivityDate="2018-04-20T01:21:04.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30547" PostTypeId="2" ParentId="18099" CreationDate="2018-04-20T01:32:28.213" Score="3" Body="&lt;p&gt;Your interpretation is correct.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$ F_n $&lt;/span&gt; represents the number of &lt;strong&gt;false negatives&lt;/strong&gt; whereas &lt;span class=&quot;math-container&quot;&gt;$ F_p $&lt;/span&gt; represents the number of &lt;strong&gt;false positives&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;h1&gt;Recall&lt;/h1&gt;&#xA;&lt;p&gt;&lt;strong&gt;answers the question:&lt;/strong&gt;&#xA;&lt;em&gt;when presented a positive example, how often does the classifier get it right?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;aliases:&lt;/strong&gt; sensitivity, true positive rate (TPR)&lt;/p&gt;&#xA;&lt;h1&gt;Precision&lt;/h1&gt;&#xA;&lt;p&gt;&lt;strong&gt;answers the question:&lt;/strong&gt; &lt;em&gt;out of all the examples the classifier thought were positive, how often were the examples actually positive?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;aliases:&lt;/strong&gt; positive predictive value (PPV)&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-20T01:32:28.213" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30548" PostTypeId="2" ParentId="30542" CreationDate="2018-04-20T01:40:48.193" Score="0" Body="&lt;p&gt;No, since you trained on patches of the image, you will need to feed in patches. This will make the denoising process of a large image very slow.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-20T01:40:48.193" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30551" PostTypeId="2" ParentId="29783" CreationDate="2018-04-20T03:39:35.883" Score="1" Body="&lt;p&gt;Setting the polynomial kernel degree to 50 is likely causing the SVM to severely overfit to the data, which would explain the 9% you are seeing.  Increasing the degree helps the SVM make an appropriate generalization, but when you start to see the validation/test accuracy decrease, then the SVM is starting to overfit.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-20T03:39:35.883" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30552" PostTypeId="2" ParentId="15509" CreationDate="2018-04-20T04:24:24.767" Score="2" Body="&lt;p&gt;The task you are trying to perform is called semantic segmentation (or pixel-wise segmentation).  There exists extensive literature on the subject and numerous tutorials/online resources to get you started, such as &lt;a href=&quot;http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Fully convolutional networks (FCN) popularized the core components of state-of-the-art semantic segmentation techniques.  They have since evolved into numerous superior, yet related, networks, many of which are covered in the link above.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although the linked website is quite comprehensive, it does not mention &lt;a href=&quot;https://arxiv.org/abs/1703.06870&quot; rel=&quot;nofollow noreferrer&quot;&gt;mask rcnn&lt;/a&gt;, which has performed very well and at relatively fast speeds (~5 fps).  It is definitely worth exploring!&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-20T04:24:24.767" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30554" PostTypeId="2" ParentId="30522" CreationDate="2018-04-20T07:01:22.620" Score="1" Body="&lt;p&gt;I think the Rbf (Radial basis function) kernel (&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Radial_basis_function_kernel&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia for rbf&lt;/a&gt;) is the most suited for this problem, as its contour plots are circles.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance,  this &lt;a href=&quot;http://arun-aiml.blogspot.com.es/2017/07/kernel-svm-support-vector-machine.html?m=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog&lt;/a&gt; solves very similar problems using the same kernel.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-20T07:01:22.620" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30555" PostTypeId="1" AcceptedAnswerId="30565" CreationDate="2018-04-20T07:19:45.343" Score="4" ViewCount="3712" Body="&lt;p&gt;I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a patience of 5 epochs when my validation loss is increasing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does it make sense to use early stopping in combination with dropout?&lt;/p&gt;&#xA;" OwnerUserId="50832" LastEditorUserId="50727" LastEditDate="2018-05-17T10:31:19.660" LastActivityDate="2019-05-15T20:21:34.027" Title="Regularization - Combine drop out with early stopping" Tags="|machine-learning|deep-learning|rnn|dropout|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30556" PostTypeId="2" ParentId="30531" CreationDate="2018-04-20T07:21:22.573" Score="0" Body="&lt;p&gt;What you could do is apply a dimensionality reduction technique such as &lt;a href=&quot;https://en.m.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;tSNE&lt;/a&gt; to visualise all your features at once in 2 or 3 dimensions. &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scikit learn&lt;/a&gt; has an excellent implementation of tSNE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, tSNE is used more to find local structure and clusters in high dimensions, rather than relationships between features. Furthermore, distances between clusters and the reduced X-Y axes &lt;a href=&quot;https://distill.pub/2016/misread-tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;mean nothing&lt;/a&gt; in tSNE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you know your data has some relationship between features, it may be best to use seaborns pairplot, as suggested by M Sef, to plot several features at once.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-04-20T07:21:22.573" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30557" PostTypeId="1" CreationDate="2018-04-20T07:25:09.083" Score="10" ViewCount="19483" Body="&lt;p&gt;In the fast R-CNN paper (&lt;a href=&quot;https://arxiv.org/abs/1504.08083&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/abs/1504.08083&lt;/a&gt;) by Ross Girshick, the bounding box parameters are continuous variables. These values are predicted using regression method. Unlike other neural network outputs, these values do not represent the probability of output classes. Rather, they are physical values representing position and size of a bounding box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exact method of how this regression learning happens is not clear to me. Linear regression and image classification by deep learning are well explained separately earlier. But how the linear regression algorithm works in the CNN settings is not explained so clearly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain the basic concept for easy understanding?&lt;/p&gt;&#xA;" OwnerUserId="50828" LastEditorUserId="50727" LastEditDate="2020-07-13T11:44:10.627" LastActivityDate="2020-07-13T11:44:10.627" Title="How does the bounding box regressor work in Fast R-CNN?" Tags="|deep-learning|image-recognition|yolo|object-recognition|faster-rcnn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30558" PostTypeId="1" CreationDate="2018-04-20T07:27:12.757" Score="0" ViewCount="2013" Body="&lt;p&gt;We have 1000's of csv, word files in different folders. Is there any open source data mining software which will look into the csv files for&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BrDwo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BrDwo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) First has to check for the &lt;strong&gt;&lt;em&gt;Project names&lt;/em&gt;&lt;/strong&gt;. If present..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Look for the &lt;strong&gt;&lt;em&gt;County&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Then, &lt;strong&gt;&lt;em&gt;State -&gt; Country&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4)If all the conditions are satisfied, assign &lt;strong&gt;&lt;em&gt;Order&lt;/em&gt;&lt;/strong&gt; to the file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've brief experience working with Orange data mining software. Tried that, but couldn't even able to check project names within the csv's. Just installed Rapidminer and i can't even comprehend what i can do with it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can shed light on what sort of software/process i have to look, will be greatly helpful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: Most of the csv's &amp;amp; word documents are having more than 20 pages&lt;/p&gt;&#xA;" OwnerUserId="50831" LastEditorUserId="50831" LastEditDate="2018-04-20T08:19:13.520" LastActivityDate="2018-04-20T15:31:32.413" Title="Search for keywords within csv/word files which are listed in an excel sheet" Tags="|data-mining|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30559" PostTypeId="1" AcceptedAnswerId="30566" CreationDate="2018-04-20T07:52:35.620" Score="1" ViewCount="365" Body="&lt;p&gt;I am new with Machine Learning and I started with some lessons in Kaggle. There, I learnt how to use &lt;code&gt;DecisionTreeRegressor()&lt;/code&gt; and &lt;code&gt;RandomForestRegressor()&lt;/code&gt; from &lt;code&gt;sklearn&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;However, I cannot really understand how I can verify that my explanatory variables do not overfit the model. For example, the lessons included evaluation with the use of &lt;em&gt;Mean Absolute Error&lt;/em&gt;. MAE and MRSE can evaluate whether my Decision Tree depth is optimal or not, but not if my explanatory data are even relevant.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I come from Economics, so I am used to deal with such problems using diagnostics or $R^2$. Is there any equivalent to $R^2$ benchmark to determine whether my explanatory variables are overfitting my model or not?&lt;/p&gt;&#xA;" OwnerUserId="24984" LastEditorUserId="40853" LastEditDate="2018-04-20T09:34:10.507" LastActivityDate="2018-05-04T15:24:30.247" Title="RF and DT overfitting" Tags="|python|random-forest|decision-trees|predictor-importance|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30560" PostTypeId="1" CreationDate="2018-04-20T07:58:11.790" Score="1" ViewCount="81" Body="&lt;p&gt;I have a list of our college students' high school courses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to recommend college degrees to current high school students based on their courses - that is, predict a class based on a vector. For instance someone who studied algebra, calculus and statistics could be recommended software engineering, accountancy or mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There seem to be a few approaches I could take: market basket analysis, collaborative filtering, clustering or even neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can structure my data in a sparse matrix of courses, with each row having a class representing the student's eventual degree, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DEGREE       English    Calc    Algebra    Geography    History ... etc&#xA;Soft.Eng.    0          1       1          0            0&#xA;Comms        1          0       0          1            1&#xA;Mech.Eng.    1          1       1          0            0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How should I approach this?&lt;/p&gt;&#xA;" OwnerUserId="50835" LastActivityDate="2018-04-25T21:56:04.527" Title="Recommending college degrees based on high school subjects" Tags="|classification|clustering|recommender-system|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30561" PostTypeId="2" ParentId="30557" CreationDate="2018-04-20T07:58:44.627" Score="4" Body="&lt;p&gt;The paper cited does not mention linear regression at all. What it does is using a neural network to predict continuous variables, and refers to that as regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression that is defined (which is not linear at all), is just a CNN with convolutional layers, and fully connected layers, but in the last fully connected layer, it does not apply sigmoid or softmax, which is what is typically used in classification, as the values correspond to probabilities. Instead, what this CNN outputs are four values $(r, c, h, w)$, where $(r, c)$ specify the values of the position of the left corner and $(h, w)$ the height and width of the window. In order to train this NN, the loss function will penalize when the outputs of the NN are very different from the labelled $(r, c, h, w)$ in the training set.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-20T07:58:44.627" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30562" PostTypeId="2" ParentId="30155" CreationDate="2018-04-20T08:03:44.897" Score="0" Body="&lt;p&gt;Since I couldn't find a suitable tool, so I made a tool for textual topic segmentation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To anyone interested in this question, I forked YEDDA's entity recognition project and changed it, accordingly. You can check it here in GitHub: &lt;a href=&quot;https://github.com/ThiagoSousa/YEEDASeg&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/ThiagoSousa/YEEDASeg&lt;/a&gt;. My project is called YEDDASeg. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: People mentioned to use BRAT, but it is good to select a word and tag with with a label. My case is a bit more complicated, because it can be many paragraphs. In YEDDASeg, you basic just click on the text and press a key to label a topic in the text. It is simpler for this purpose.&lt;/p&gt;&#xA;" OwnerUserId="50338" LastActivityDate="2018-04-20T08:03:44.897" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30563" PostTypeId="2" ParentId="30559" CreationDate="2018-04-20T08:04:25.550" Score="2" Body="&lt;p&gt;Yes, and that benchmark is called validation data. The idea is to split your data in some training data, which is what you are going to fit to your models in order to estimate the parameters in them (and make them learn), and some validation data, which is the data you are going to use in order to evaluate your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you can do is compute the error on your validation data (which the models haven't been trained on) for the different models you want to use, and you would like to keep the model with the lowest validation error. &lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-20T08:04:25.550" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30565" PostTypeId="2" ParentId="30555" CreationDate="2018-04-20T08:54:36.297" Score="2" Body="&lt;p&gt;It does make sense, they are just two different things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dropout only makes your model learning harder, and by this it helps the parameters of the model act in different ways and detect different features, but even with dropout you can potentially overfit your traning set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, early stopping prevents your model from overfitting by taking the best model on your validation data so far.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;However&lt;/em&gt;, for the sake of simplicity, I think it is easier to just use dropout (training a neural network is not easy and the training may not be successful due to many different reasons, it is a good practice to reduce the possible reasons why the training is failing as much as possible). Unless you have short time to train your network, with a sufficiently high amount of dropout you will ensure that your model is not overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My final recommendation is: just use dropout. If using a 0.5 dropout rate still overfits, set a higher dropout rate.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-20T08:54:36.297" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30566" PostTypeId="2" ParentId="30559" CreationDate="2018-04-20T09:00:47.067" Score="2" Body="&lt;p&gt;I think you can perform Predictor Importance test and see which are the variable explaining the most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is this package named &lt;a href=&quot;https://www.kaggle.com/tilii7/boruta-feature-elimination&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt;, you can go through the link for implementation in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can eliminate the variables which are highly correlated. For example if you have age as the target variable and you have DOB as a feature then it makes no sense to build a model. So, you need to make sure to eliminate the variable which are highly correlated to the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my Scenario I had this following visualization&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SgDSo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SgDSo.png&quot; alt=&quot;Predictor Importance&quot;&gt;&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see the 2 variables(underlined with red dash) are highly correlated with the target variable, before removing these variables the MAE was 0.9(approx) after removing those features(Backward Stepwise Elimination) and the MAE was 3.5(approx) but that is the actual error. Currently working on getting some external features to explain the data and to improve the accuracy. Every time it is not about accuracy/error rate of the model, it is also about how good our model could be generalized and robust our model should be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To check if the data is overfitting, then I tried testing it by taking those 2 variables and try modelling and the MAE was 1.6(approx) from this we can understand that these 2 variables explain the most.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, try applying and see how the features are correlated with the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the methods used to address over-fitting in decision tree is called pruning which is done after the initial training is complete. In pruning, you trim off the branches of the tree, i.e., remove the decision nodes starting from the leaf node such that the overall accuracy is not disturbed. This is done by segregating the actual training set into two sets: training data set, D and validation data set, V. Prepare the decision tree using the segregated training data set, D. Then continue trimming the tree accordingly to optimize the accuracy of the validation data set, V.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go thorough this &lt;a href=&quot;https://datascience.stackexchange.com/questions/6380/how-to-avoid-overfitting-in-random-forest&quot;&gt;link&lt;/a&gt;, about how we can avoid over fitting by tuning the parameters.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-05-04T03:53:24.650" LastActivityDate="2018-05-04T03:53:24.650" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="30568" PostTypeId="2" ParentId="30519" CreationDate="2018-04-20T09:31:21.650" Score="3" Body="&lt;p&gt;You can use proximity-based outlier detection methods. These methods fall under 2 different categories, 1) Cluster-based, and 2) Distance-based.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cluster-based methods will define a point as an outlier if its distance to cluster centroids is large. If you have both categorical and continuous variables, then it is important to ensure that the clustering method you use is appropriate for mixed data-types. Clustering methods appropriate for both types include k-Prototypes or Squeezer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use a distance-based method, you could use the k-nearest neighbor algorithm, and define an outlier as a data point whose distance from its k neighbors is much larger than normal data points.&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-04-20T09:31:21.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30570" PostTypeId="1" CreationDate="2018-04-20T10:54:05.070" Score="1" ViewCount="74" Body="&lt;p&gt;Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. Assume, for some reason (probably some training problem or declaring classes), C is confused and doesn't perform well. The way we assign a class using C to each test data $I$ is: $class(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$, where $p_j(I)$ is the probability estimate of $I$ corresponding to the $j$-th class, given by C.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, on top of this previous classifier C, I'd like to use a Hidden Markov Model (HMM) to &quot;correct&quot; the mistakes made by the previous context-free classifier C, by taking into account the contextual/sequential information not used by C. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence let in my HMM, the hidden state $Z_i$ denote the true class of the $i$-th sample $I_i$, and $X_i$ be the predicted class by C. My question is: how could we use the probabilistic information $cl(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$ to train this HMM? I understand that the &lt;em&gt;confusion matrix of C&lt;/em&gt; can be used to define the emission prob. of the HMM, but how do we define the transition and start/prior prob.? I'm tempted to define the start/prior prob. vector as $\pi:=(p_1(x_1), ..., p_K(x_1))$.  But I may be wrong. &lt;strong&gt;This is my main question&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A follow up question:&lt;/strong&gt; One can define an HMM in the above way (using confusion matrix and the prob. information from C); call the resulting parameter set $\Theta_0$. But after doing so, is it advisable to estimate the parameters to best fit the data $I$ used for C, while initializing a parameter set with the values mentioned in the previous paragraph?&lt;/p&gt;&#xA;" OwnerUserId="50802" LastEditorUserId="29575" LastEditDate="2018-04-21T01:06:16.950" LastActivityDate="2023-12-28T11:05:55.850" Title="How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?" Tags="|machine-learning|neural-network|markov-process|markov-hidden-model|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30571" PostTypeId="1" CreationDate="2018-04-20T12:48:17.093" Score="1" ViewCount="521" Body="&lt;p&gt;I followed these two posts to understand about restoring a saved model and then extracting variables from it:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/45562755/extracting-weights-values-from-a-tensorflow-model-checkpoint&quot;&gt;Extracting weights values from a tensorflow model checkpoint&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/questions/46131410/how-to-examine-the-feature-weights-of-a-tensorflow-linearclassifier&quot;&gt;How to examine the feature weights of a Tensorflow LinearClassifier?&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But now I am unable to understand as to what do those variables mean exactly and how to understand the relative importance given by the model to the features in the learning process? Below is the output of &lt;code&gt;tf.train.list_variables&lt;/code&gt; and &lt;code&gt;tf.train.load_variables&lt;/code&gt; (respectively) from my code:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;gt;&amp;gt; tf.train.list_variables('./dnn_fe_trial1/model.ckpt-1')&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            [('dnn/hiddenlayer_0/biases', [50]),&#xA;            ('dnn/hiddenlayer_0/biases/enlayer_0/biases/part_0/Adam', [50]),&#xA;            ('dnn/hiddenlayer_0/biases/enlayer_0/biases/part_0/Adam_1', [50]),&#xA;            ('dnn/hiddenlayer_0/weights', [61, 50]),&#xA;            ('dnn/hiddenlayer_0/weights/nlayer_0/weights/part_0/Adam', [61, 50]),&#xA;            ('dnn/hiddenlayer_0/weights/nlayer_0/weights/part_0/Adam_1', [61, 50]),&#xA;            ('dnn/input_from_feature_columns/brand_embedding/weights', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/brand_embedding/weights/mbedding/weights/part_0/Adam', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/brand_embedding/weights/mbedding/weights/part_0/Adam_1', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/city_embedding/weights', [12, 3]),&#xA;            ('dnn/input_from_feature_columns/city_embedding/weights/mbedding/weights/part_0/Adam', [12, 3]),&#xA;            ('dnn/input_from_feature_columns/city_embedding/weights/mbedding/weights/part_0/Adam_1', [12, 3]),&#xA;            ('dnn/input_from_feature_columns/dow_embedding/weights', [7, 3]),&#xA;            ('dnn/input_from_feature_columns/dow_embedding/weights/mbedding/weights/part_0/Adam', [7, 3]),&#xA;            ('dnn/input_from_feature_columns/dow_embedding/weights/mbedding/weights/part_0/Adam_1', [7, 3]),&#xA;            ('dnn/input_from_feature_columns/l_cat_embedding/weights', [11, 3]),&#xA;            ('dnn/input_from_feature_columns/l_cat_embedding/weights/mbedding/weights/part_0/Adam', [11, 3]),&#xA;            ('dnn/input_from_feature_columns/l_cat_embedding/weights/mbedding/weights/part_0/Adam_1', [11, 3]),&#xA;            ('dnn/input_from_feature_columns/product_id_embedding/weights', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/product_id_embedding/weights/mbedding/weights/part_0/Adam', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/product_id_embedding/weights/mbedding/weights/part_0/Adam_1', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/type_id_embedding/weights', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/type_id_embedding/weights/mbedding/weights/part_0/Adam', [10000, 16]),&#xA;            ('dnn/input_from_feature_columns/type_id_embedding/weights/mbedding/weights/part_0/Adam_1', [10000, 16]),&#xA;            ('dnn/logits/biases', [1]),&#xA;            ('dnn/logits/biases/nn/logits/biases/part_0/Adam', [1]),&#xA;            ('dnn/logits/biases/nn/logits/biases/part_0/Adam_1', [1]),&#xA;            ('dnn/logits/weights', [50, 1]),&#xA;            ('dnn/logits/weights/n/logits/weights/part_0/Adam', [50, 1]),&#xA;            ('dnn/logits/weights/n/logits/weights/part_0/Adam_1', [50, 1]),&#xA;            ('dnn/regression_head/dnn/learning_rate', []),&#xA;            ('dnn/regression_head/train_op/dnn/beta1_power', []),&#xA;            ('dnn/regression_head/train_op/dnn/beta2_power', []),&#xA;            ('global_step', [])]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;&amp;gt;&amp;gt; tf.train.load_variable('./dnn_fe_trial1/model.ckpt-1','dnn/hiddenlayer_0/weights')&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            array([[ 0.14350541,  0.18532775, -0.03176343, ..., -0.07279533,&#xA;            -0.08580479, -0.07619692],&#xA;            [ 0.16894072, -0.10593006,  0.06088932, ..., -0.01411209,&#xA;            -0.26995516,  0.15667924],&#xA;            [-0.10020741, -0.03164399, -0.14427225, ..., -0.02787848,&#xA;            -0.15646952, -0.1361219 ],&#xA;            ...,&#xA;            [ 0.15014522,  0.15378515, -0.05414914, ..., -0.16788298,&#xA;            -0.14711154, -0.226382  ],&#xA;            [-0.16823539,  0.2009476 , -0.271177  , ..., -0.10694946,&#xA;            -0.22870012, -0.13458726],&#xA;            [-0.13175508,  0.15535942, -0.18468232, ..., -0.1362714 ,&#xA;            -0.27476427, -0.21606216]], dtype=float32)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50852" LastActivityDate="2018-04-20T12:48:17.093" Title="How to extract relative importance of features from a tensorflow DNNRegressor model?" Tags="|python|tensorflow|regression|feature-extraction|feature-engineering|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30572" PostTypeId="1" CreationDate="2018-04-20T13:35:05.470" Score="1" ViewCount="186" Body="&lt;p&gt;I want to understand what kind of regression does the DNNRegressor estimator apply in the background, specifically is it linear?&lt;/p&gt;&#xA;" OwnerUserId="50852" LastActivityDate="2018-04-20T13:35:05.470" Title="What regression approach does tensorflow's DNNRegressor apply under the hood?" Tags="|machine-learning|python|deep-learning|tensorflow|regression|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30573" PostTypeId="1" AcceptedAnswerId="30721" CreationDate="2018-04-20T13:44:57.703" Score="0" ViewCount="502" Body="&lt;p&gt;I have a dataframe of X values and one with the y values in what I think is the proper format. I took the single feature y and converted it to dummies so it's binary and is m rows by n features. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;&#xA;X = df.drop('target',axis=1)&#xA;y = pd.get_dummies(df['target'], sparse=True, drop_first=True)&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=99, train_size=0.60&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I've been trying to figure out how exactly the data should look from the &lt;a href=&quot;http://scikit.ml/api/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; but am not having much luck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first tried the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import skmultilearn.problem_transform import BinaryRelevance&#xA;from sklearn.naive_bayes import GaussianNB&#xA;&#xA;classifier = BinaryRelevance(GaussianNB())&#xA;classifier.fit(X_train, y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where X, y train sets are pandas dataframes. This results in the following error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TypeError: no suppoerted conversion for types: (dytype('O'),)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I converted to matrices:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train = X_train.asmatrix()&#xA;y_train = y_train.asmatrix()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I received the same error. Then I tried&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train = np.matrix(X_train)&#xA;y_train = np.matrix(y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Same error again. Is there an easy way to go from dataframe to proper format for skmultilearn models?&lt;/p&gt;&#xA;" OwnerUserId="50746" LastEditorUserId="50746" LastEditDate="2018-04-20T13:56:30.493" LastActivityDate="2018-04-23T22:52:33.467" Title="Formatting Data for use in SKmultilearn" Tags="|python|scikit-learn|multilabel-classification|data-formats|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30574" PostTypeId="1" AcceptedAnswerId="30595" CreationDate="2018-04-20T13:46:49.837" Score="8" ViewCount="6171" Body="&lt;p&gt;Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than dropout?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-26T11:53:27.877" LastActivityDate="2018-05-26T11:53:27.877" Title="Dropout vs weight decay" Tags="|machine-learning|deep-learning|overfitting|regularization|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30575" PostTypeId="1" CreationDate="2018-04-20T14:22:50.367" Score="3" ViewCount="240" Body="&lt;p&gt;The data is a huge set of observations of dozens of variables, all (potentially, somehow) related to a dichotomous outcome variable, and all (potentially) correlated to each other, or to unknown / unobserved things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that &quot;merely&quot; applying logistic regressions or any related extension or improvement (such as Classification And Regression Trees etc.) might not be sufficient, because there is a &quot;basic&quot; bayesian network structure that seems to exist (based on expert opinion) - this means that the observed variables are somehow describing distinct nodes in a bayesian network, but the exact formula HOW they describe these nodes is unclear, as is the eventual predictive power of these nodes towards the outcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is this exact problem in a highly simplified example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have two runners who are going to compete in a 400m sprint against each other. We have a dataset of observations of thousands of such competitions with two runners, each observation containing, for each of the two runners, (a) age, (b) size, (c) number of training runs in the past 6 weeks and (d) average mileage per training run in the past 6 weeks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we believe that (a) and (b) will determine (e) &quot;talent for sprinting&quot; and (c) and (d) will determine (f) &quot;current form&quot;, and that (e) and (f) together will give the probability of winning - but we do not know HOW (a) and (b) determine (e) or how (c) and (d) determine (f) (e.g., is it &quot;age times size&quot; or is it &quot;age times 2 while age lower than 45&quot; etc.). Further, we also do not know how (e) and (f) together determine the probability of winning...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any &quot;known&quot; approach to such a problem, i.e. a way to determine the formula that connects the initial variables to the later variables that determine the outcome, and then further determining the predictive power of the resulting network? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or is that unnecessary (i.e. using the original variables only in some algorithm - which one? - should give the same result)? Or is it impossible (too many unknown things)? What should I do / read about / learn to solve this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And in addition, is there a way to CHECK whether my original assumption (that (a) and (b) determine (e) etc.) was correct, e.g. via some sort of clustering algorithm, PCA, ...?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would you do to approach such a question?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much indeed in advance for helping a rather new data science learner by pointing me into the right direction!&lt;/p&gt;&#xA;" OwnerUserId="50412" LastActivityDate="2020-09-30T09:06:00.673" Title="Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?" Tags="|machine-learning|neural-network|clustering|logistic-regression|bayesian-networks|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30576" PostTypeId="1" AcceptedAnswerId="30588" CreationDate="2018-04-20T14:30:24.127" Score="2" ViewCount="183" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Dropout_(neural_networks)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dropout&lt;/a&gt; is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any similar technique in linear regression, SVMs or tree-based methods?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="16213" LastEditDate="2018-04-20T22:52:56.223" LastActivityDate="2018-04-20T22:52:56.223" Title="Dropout in other machine learning models" Tags="|svm|regularization|dropout|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30577" PostTypeId="2" ParentId="30558" CreationDate="2018-04-20T15:31:32.413" Score="1" Body="&lt;p&gt;Try something like the following in python:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;&#xA;df = pd.read_csv(path_to_file, header=0) # header=0 means the first row contains column names&#xA;&#xA;# locate the records where none of the following columns are null ('~' means not)&#xA;mask = ~df[['Project Name', 'County', 'State', 'Country']].isnull()&#xA;df[ mask ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm not entirely sure what you want to do with the 'Order' columns, but if it's to get just the &quot;Order&quot; column from the above statement, then just slightly modify it to the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['Order'][mask]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to set the value of &quot;Order&quot; for those columns that aren't null, then just do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# we need to adjust our mask because the current version won't work with the .loc function that we'll use below&#xA;&#xA;mask = df[(~df['PROJECTNAME'].isnull())&amp;amp;(~df['County'].isnull())&amp;amp;(~df['State'].isnull())&amp;amp;(~df['Country'].isnull())]&#xA;&#xA;df.loc[ mask, &quot;Order&quot;] = whatever you want&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to save all the orders where the specified columns aren't null to a separate csv, do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df[ mask ].to_csv(name_your_file)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50746" LastActivityDate="2018-04-20T15:31:32.413" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30578" PostTypeId="1" AcceptedAnswerId="30582" CreationDate="2018-04-20T16:17:20.770" Score="3" ViewCount="1067" Body="&lt;p&gt;In my next academic year at university, I have the option to take a course in Advanced Functional Programming. A basic description of the course is this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;You’ll focus on a number of more advanced functional programming topics such as: programming with effects; reasoning about programs; control flow; advanced libraries; improving efficiency; type systems; and functional pearls.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore I'd like to know if functional programming is useful in Data Science. If so, why it is useful, and by extension, whether this course will ultimately be useful in the pursuit of becoming a Data Scientist.&lt;/p&gt;&#xA;" OwnerUserId="50862" LastActivityDate="2018-04-20T18:23:37.717" Title="What can functional programming be used for in data science?" Tags="|programming|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30579" PostTypeId="2" ParentId="22126" CreationDate="2018-04-20T16:59:53.547" Score="3" Body="&lt;p&gt;I believe it does, for example, if you flatten both features into 1-D array and then concatenate them, they should still preserve most of the information. Because total number of feature vector still remains the same.&lt;/p&gt;&#xA;" OwnerUserId="50863" LastActivityDate="2018-04-20T16:59:53.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30580" PostTypeId="1" CreationDate="2018-04-20T17:03:51.900" Score="1" ViewCount="2837" Body="&lt;p&gt;I am working with the cancer data everything is working perfect but just after the window update i am getting this error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is my python code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df=pd.read_csv('breast-cancer-wisconsin-data.csv')&#xA;mg.plots.plot_knn_classification(n_neighbors=3)&#xA;&#xA;x_train,x_test,y_train,y_test=train_test_split(cancer.data,cancer.target,stratify=cancer.target,random_state=42)&#xA;log_reg100=LogisticRegression(C=100)&#xA;log_reg100.fit(x_train,y_train)&#xA;print(&quot;Accuracy on training set {}&quot;,format(log_reg.score(x_train,y_train)))&#xA;print(&quot;Accuracy on training set {}&quot;,format(log_reg.score(x_test,y_test)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50864" LastActivityDate="2018-04-20T17:03:51.900" Title="How to resolve name 'log_reg' is not defined" Tags="|ipython|" AnswerCount="0" CommentCount="2" ClosedDate="2018-04-23T08:13:21.930" ContentLicense="CC BY-SA 3.0" />
  <row Id="30581" PostTypeId="1" CreationDate="2018-04-20T18:21:09.750" Score="1" ViewCount="1944" Body="&lt;p&gt;How to apply CSR Matrix on DBSCAN algorithm in python without using any libraries?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update: Matrix size (8580, 126356)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have given a shot and implemented the algorithm. It runs rather slow. I guess its because regionQuery function calculates the euclidean distance  between a chosen point and every other point in the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any way this problem can be solved?&lt;/p&gt;&#xA;" OwnerUserId="50869" LastEditorUserId="9257" LastEditDate="2018-04-22T09:39:00.217" LastActivityDate="2018-09-23T15:44:06.320" Title="What is slowing down classic DBSCAN algorithm" Tags="|python|clustering|dbscan|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30582" PostTypeId="2" ParentId="30578" CreationDate="2018-04-20T18:23:37.717" Score="5" Body="&lt;p&gt;One reason why functional programming could be useful for data science is that it lends itself more easily to parallel and distributed programming, e.g. the popular frameworks &lt;a href=&quot;http://spark.apache.org/&quot; rel=&quot;noreferrer&quot;&gt;Apache Spark&lt;/a&gt; for cluster computing and &lt;a href=&quot;http://kafka.apache.org/&quot; rel=&quot;noreferrer&quot;&gt;Apache Kafka&lt;/a&gt; for stream-processing are both written in Scala (and Java). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other than that &quot;functional programming&quot; as a skill is not directly related to data science. It's a tool that may facilitate some practicalities of data science and therefore more relevant for the &quot;data engineering&quot; aspect of data science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's useful but probably not necessary. It depends on your interests.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-04-20T18:23:37.717" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30583" PostTypeId="1" AcceptedAnswerId="30586" CreationDate="2018-04-20T19:24:26.007" Score="0" ViewCount="185" Body="&lt;p&gt;Can anybody explain me what are the effects on the model if we have sparse data in our dataset. And also how to deal these sparse matrices ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-20T21:24:52.617" Title="Sparse Matrix - Effect and Solution" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30584" PostTypeId="1" CreationDate="2018-04-20T20:15:34.663" Score="1" ViewCount="170" Body="&lt;p&gt;I read this paper (&lt;a href=&quot;https://arxiv.org/pdf/1206.2944.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1206.2944.pdf&lt;/a&gt;) discussing about practical issues of Bayesian optimization and they mentioned that integrating out hyperparameters of Gaussian process using fully-Bayesian treatment would be sensible and give more stable results.  The way they do that is to use following equation (Eq. 6 in page 4 of the paper):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\hat{a}(x;\{x_n,y_n\})=\int a(x;\{x_n,y_n,\theta\})p(\theta|\{x_n,y_n\})d\theta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how this integral is implemented?  I know that $a(x;\{x_n,y_n,\theta\})$ can be computed using Gaussian process regression, and in theory $p(\theta|\{x_n,y_n\})$ can be expressed as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p(\theta|\{x_n,y_n\})=\frac{p(\{x_n,y_n\}|\theta)p(\theta)}{p(\{x_n,y_n\})}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and I expect $p(\{x_n,y_n\}|\theta)$ should be tractable (it is just a multi-variate Gaussian), but what is prior for $p(\theta)$ and how to compute $p(\{x_n,y_n\})$ and more importantly how to combine them to implement the complete integral?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50440" LastEditorUserId="1330" LastEditDate="2021-09-14T12:51:31.700" LastActivityDate="2021-09-14T12:51:31.700" Title="How to integrate out hyperparameters of Gaussian process for Bayesian optimization?" Tags="|machine-learning|mathematics|bayesian|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30586" PostTypeId="2" ParentId="30583" CreationDate="2018-04-20T21:24:52.617" Score="1" Body="&lt;p&gt;The idea is really simple, just look at some online resources like &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Sparse_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.m.wikipedia.org/wiki/Sparse_matrix&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The implementation is also really simple. For pandas this page might help you &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/sparse.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pandas.pydata.org/pandas-docs/stable/sparse.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-20T21:24:52.617" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30587" PostTypeId="2" ParentId="14999" CreationDate="2018-04-20T21:39:25.463" Score="8" Body="&lt;p&gt;I am actively maintaining an efficient &lt;a href=&quot;https://github.com/chuanconggao/PrefixSpan-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;implementation&lt;/a&gt; of both PrefixSpan and BIDE in Python 3, supporting mining both frequent and top-k (closed) sequential patterns.&lt;/p&gt;&#xA;" OwnerUserId="47080" LastEditorUserId="98307" LastEditDate="2020-08-04T17:00:05.437" LastActivityDate="2020-08-04T17:00:05.437" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="30588" PostTypeId="2" ParentId="30576" CreationDate="2018-04-20T22:21:49.490" Score="2" Body="&lt;p&gt;Random forests could be thought of as using a kind of dropout-esque technique as each split node only considers a random subset of the features, effectively 'dropping out' the other ones. Also, sometimes in large tree ensembles, each tree is only given a random subset of features to begin with, akin to dropout on the input layer of a neural network.&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-20T22:21:49.490" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30589" PostTypeId="1" CreationDate="2018-04-20T22:24:53.213" Score="3" ViewCount="6591" Body="&lt;p&gt;I am playing with ROC and trying to draw some curves. I am using example from &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this scikit&lt;/a&gt; page. I do not understand one thing: when I print out the content of tpr and fpr, I see two arrays of numbers (21 elems in each) - why is that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TPR (True Positive Ratio) is a proportion of those tuples classified as positives to all real positive tuples. So, it should be one number&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see it as follow: I take classifier (like Decision Tree), train it on some data and finally test it. Then I can calculate TPR and FPR and I should have only two values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead, I receive arrays. Why is that? I tried to change classifier from that example to Decision Tree and then I receive only one point (which is usually [0, 1]). How can I draw ROC curve on decision tree? Or - to be more precise with my question - why do I get an array of values as my TRP/FPR instead of single values?&lt;/p&gt;&#xA;" OwnerUserId="41740" LastActivityDate="2021-08-29T03:59:15.430" Title="How to interpret FPR and TPR in ROC curve?" Tags="|classification|scikit-learn|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30590" PostTypeId="2" ParentId="30589" CreationDate="2018-04-20T22:53:21.377" Score="2" Body="&lt;p&gt;As explained here in the docs of &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve&quot; rel=&quot;nofollow noreferrer&quot;&gt;the function returning the ROC Curve&lt;/a&gt;,&#xA;they return an array because when you interpret a ROC curve what you're looking at the performance of the predictor &lt;strong&gt;given a threshold&lt;/strong&gt;. So, it means that, if I'm looking for only classifications that the model is above 80% sure, the ROC curve will have a different TPR/FPR than a threshold of 90%.&lt;/p&gt;&#xA;&lt;p&gt;If you have further questions, here's a great &lt;a href=&quot;http://www.dataschool.io/roc-curves-and-auc-explained/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; explaining it.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="85045" LastEditDate="2021-08-29T03:59:15.430" LastActivityDate="2021-08-29T03:59:15.430" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30591" PostTypeId="1" AcceptedAnswerId="30593" CreationDate="2018-04-21T00:36:18.360" Score="4" ViewCount="2178" Body="&lt;p&gt;If momentum optimizer independently keeps a custom &quot;inertia&quot; value for each weight, then why do we ever need to bother with learning rate? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surely, momentum would catch up its magnutude pretty quickly to any needed value anyway, why to bother scaling it with learning rate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_{dw} = \beta v_{dw} +(1-\beta)dW$$&#xA;$$W = W-\alpha v_{dw}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\alpha$ is the learning rate (0.01 etc) and $\beta$ is the momentum coefficient (0.9 etc)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the answer! To put it more plain: momentum controls &quot;how well we retain&quot; the movement, and learning rate is &quot;how fast do we reGain&quot; the movement&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-05-11T17:21:25.680" LastActivityDate="2018-05-11T17:21:25.680" Title="Why does momentum need learning rate?" Tags="|neural-network|backpropagation|momentum|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30592" PostTypeId="2" ParentId="27104" CreationDate="2018-04-21T01:50:24.487" Score="0" Body="&lt;p&gt;A regression technique that allow to predict a multidimensional output can be the &lt;strong&gt;PLS&lt;/strong&gt; ( partial least square ). I implemented it in scala and it will be soon available on &lt;a href=&quot;https://github.com/Clustering4Ever/Clustering4Ever/blob/master/README.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;Clustering4Ever&lt;/a&gt; repo. In fact we went a bit further by applying it with the clusterwise pattern which generate k-clusters driving by PLS regression which result with one regression model per cluster which can be much more accurate but also most costly due to approximatively 1000.n regressions. You can look on it with, &lt;strong&gt;A new micro batch approach for partial least square clusterwise regression&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastActivityDate="2018-04-21T01:50:24.487" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30593" PostTypeId="2" ParentId="30591" CreationDate="2018-04-21T02:00:57.803" Score="5" Body="&lt;p&gt;To answer the first question about why we need the learning rate even if we have momentum, let's consider an example in which we are &lt;strong&gt;not using the momentum term&lt;/strong&gt;.  The weight update is therefore:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \Delta w_{ij} = \frac{\partial E}{\partial w_{ij}} \cdot l $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$ \Delta w_{ij} $ is the weight update&lt;/li&gt;&#xA;&lt;li&gt;$ \frac{\partial E}{\partial w_{ij}} $ is the gradient of the error with respect to the weight&lt;/li&gt;&#xA;&lt;li&gt;$ l \space $ is the learning rate coefficient&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Our weight update is determined by the gradient of our current error with respect to the weight at node $ ij $.  Therefore, our prior weight deltas are not factored into our weight update equation.  If we were to eliminate the learning rate, our weights would not update.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now let's consider an example &lt;strong&gt;using the momentum term&lt;/strong&gt; in its derived form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \Delta w_{ij} = (\frac{\partial E}{\partial w_{ij}} \cdot l) + (\mu \cdot \Delta w^{t-1}_{ij}) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$ \mu $ is the momentum coefficient&lt;/li&gt;&#xA;&lt;li&gt;$ \Delta w^{t-1}_{ij} $ is the weight update of node $ ij $ from the previous epoch&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Now we are factoring the previous weight delta in our weight update equation.  In this form, it is easier to see that the learning rate and momentum are effectively independent terms.  However, without a learning rate, our weight delta would still be zero.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Now you might ask: &lt;em&gt;what if we remove the learning rate after getting an initial momentum value so that momentum is the sole influence of the weight delta?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This destroys the backpropagation algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective of backprop is to optimize the weights to minimize error.  We achieve this minimization by adjusting the weights according to the error gradient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Momentum, on the other hand, aims to improve the rate of convergence and to avoid local minimas.  The momentum term does not explicitly include the error gradient in its formula.  Therefore, momentum by itself does not enable learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were to only use momentum after establishing an initial weight delta, the weight update equation would look as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \Delta w_{ij} = (\mu \cdot \Delta w^{t-1}_{ij}) $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \lim_{t \to \infty} \Delta w^t_{ij} =&#xA;\begin{cases}&#xA;0 &amp;amp; | \space \mu &amp;lt; 1 \space \lor \space (\mu = 1 \space \land \space \Delta w^{t=0}_{ij} &amp;lt; 1) \\&#xA;1 &amp;amp; | \space \mu = 1 \land \space \Delta w^{t=0}_{ij} = 1\\&#xA;\infty &amp;amp; | \space otherwise&#xA;\end{cases}&#xA;$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although there exists a scenario where the weight delta approaches zero, this descent is not based on the error gradient and is in fact predetermined by the momentum coefficient and the initial weight delta: this weight delta does not achieve our objective to minimize the error and is therefore useless.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate is critical for updating the weights to minimize error.  Momentum is used to help the learning rate, but not replace it.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-21T04:36:30.533" LastActivityDate="2018-04-21T04:36:30.533" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30594" PostTypeId="2" ParentId="11950" CreationDate="2018-04-21T02:03:42.000" Score="1" Body="&lt;p&gt;Broadcasting your data and learn on it with different learning parameters per spark partition is a solution &lt;strong&gt;only&lt;/strong&gt; if your data isn t so big it can fit in each machine memory. If you desire apply ML models at scale you have to deal with subquadratic complexity, if not you will to increase number of nodes accordingly to the complexity, but its something difficult from quadratic to bigger complexity. With &lt;a href=&quot;https://github.com/Clustering4Ever/Clustering4Ever/blob/master/README.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;Clustering4Ever&lt;/a&gt; we try to propose scalable clustering algorithms in scala and scala/spark. We will soon add some new algorithms that you may enjoy. Don t hesitate to ask for specific algorithms or add those you already implemented.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastActivityDate="2018-04-21T02:03:42.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30595" PostTypeId="2" ParentId="30574" CreationDate="2018-04-21T03:12:42.840" Score="12" Body="&lt;p&gt;These techniques are not mutually exclusive; combining dropout with weight decay has become pretty standard for deep learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, where weight decay applies a linear penalty, dropout can cause the penalty to grow exponentially.  This property of dropout can lead to hypothetical failures as proposed and proven in &lt;strong&gt;section 4.2 of &lt;a href=&quot;https://www.researchgate.net/publication/301846616_Fundamental_differences_between_Dropout_and_Weight_Decay_in_Deep_Networks&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, research has consistently shown the benefits of dropout (with and without weight decay) for training deep networks.  A practical scenario in which weight decay is exclusively preferred over dropout would be quite the anomaly.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-21T03:12:42.840" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30596" PostTypeId="1" CreationDate="2018-04-21T04:33:14.610" Score="2" ViewCount="1212" Body="&lt;p&gt;Is it possible to predict the products which are going to be out of stock in next coming days/weeks.?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If Yes, what could be the possible data points that I should collect?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How is this problem different from sales prediction?&lt;/p&gt;&#xA;" OwnerUserId="50885" LastEditorUserId="29575" LastEditDate="2018-04-21T06:19:38.480" LastActivityDate="2018-04-21T06:19:38.480" Title="product out of stock prediction" Tags="|machine-learning|predictive-modeling|bigdata|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30599" PostTypeId="2" ParentId="30537" CreationDate="2018-04-21T05:53:15.873" Score="0" Body="&lt;p&gt;I fall on same issue with RMSE which by the way may be a good complementary choice of MAE. Thus in order to measure error prediction on multidimentional output the way i implemented was as follow. Measure the index score per dimension then do the average on all dimensions which gives a single scalar value.&#xA;If your data isn t on same scale, apply a standardization on it. And in order to have same order of magnitude on all your output dimensions you also can divide each column by their corresponding test set standard deviation before applying the mean reduce.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="9257" LastEditDate="2018-04-21T05:59:57.807" LastActivityDate="2018-04-21T05:59:57.807" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30600" PostTypeId="2" ParentId="30596" CreationDate="2018-04-21T06:18:14.880" Score="1" Body="&lt;p&gt;You may don’t need any machine learning for that, just simple math.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate the average use per day, define your desired minimum amount which should be on stock and collect the time for a supplier to send you new products.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that, you can predict when to order your product from the supplier to always have your desired amount on stock.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Search for order point system :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the system you could estimate when a product will be out of stock and when to order to prevent that.&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-21T06:18:14.880" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30601" PostTypeId="2" ParentId="28252" CreationDate="2018-04-21T06:37:09.140" Score="1" Body="&lt;p&gt;I have a new thinking on this. I think it maybe ok to use a different but reasonable preprocessing (such as 1/255.) in the context of transfer learning (pre-training), than whats originally used to train the VGG16. As long as the top most conv-conv-...pool representation is useful for your new task, and it empirically checked out fine for what F. Chollet did in his blog.&lt;/p&gt;&#xA;" OwnerUserId="46782" LastActivityDate="2018-04-21T06:37:09.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30602" PostTypeId="1" CreationDate="2018-04-21T06:50:42.490" Score="5" ViewCount="2221" Body="&lt;p&gt;I read through &amp;quot;ImageNet Classification with Deep Convolutional Neural Networks&amp;quot; again specifically for details on how to implement PCA color augmentation.&#xA;I am unsure if I have it right. Here is how I did this in numpy:&lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;# original image is 224x224x3 of dtype uint8 &#xA;&#xA;renorm_image = np.reshape(original_image,(original_image.shape[0]*original_image.shape[1],3))&#xA;&#xA;renorm_image = renorm_image.astype('float32')&#xA;renorm_image -= np.mean(renorm_image, axis=0)&#xA;renorm_image /= np.std(renorm_image, axis=0)&#xA;&#xA;cov = np.cov(renorm_image, rowvar=False)&#xA;&#xA;lambdas, p = np.linalg.eig(cov)&#xA;alphas = np.random.normal(0, 0.1, 3)&#xA;&#xA;delta = np.dot(p, alphas*lambdas)&#xA;&#xA;delta = (delta*255.).astype('int8')&#xA;&#xA;pca_color_image = np.maximum(np.minimum(original_image + delta, 255), 0).astype('uint8')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;One serious doubt is the line &amp;quot;delta = (delta*255.)&amp;quot;. I have to do this to rescale things such that the numbers make sense. I hope someone can give me feedback if this is right.&lt;/p&gt;&#xA;" OwnerUserId="46782" LastEditorUserId="32433" LastEditDate="2021-06-29T17:00:05.233" LastActivityDate="2021-06-29T17:00:05.233" Title="How to implement PCA color augmentation as discussed in AlexNet" Tags="|pca|convolutional-neural-network|data-augmentation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30603" PostTypeId="1" AcceptedAnswerId="30608" CreationDate="2018-04-21T06:52:13.553" Score="2" ViewCount="187" Body="&lt;p&gt;I am trying to build an &lt;code&gt;MLP&lt;/code&gt; classifier model on a dataset containing 30000 samples and 23 features. &#xA;What are the standards I need to consider while selecting the number of hidden layers and number of nodes in each hidden layer?&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="28175" LastEditDate="2018-04-21T14:53:45.480" LastActivityDate="2018-04-21T14:53:45.480" Title="Neural Network Hidden Layer Selection" Tags="|machine-learning|python|neural-network|deep-learning|mlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30604" PostTypeId="2" ParentId="30581" CreationDate="2018-04-21T06:52:53.907" Score="4" Body="&lt;ol&gt;&#xA;&lt;li&gt;Write the DBSCAN code yourself.&lt;/li&gt;&#xA;&lt;li&gt;Run the code.&lt;/li&gt;&#xA;&lt;li&gt;Observe that your code likely will be a lot slower than the libraries code.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-21T06:52:53.907" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30605" PostTypeId="1" CreationDate="2018-04-21T07:06:28.130" Score="21" ViewCount="3614" Body="&lt;p&gt;When reading &lt;a href=&quot;https://arxiv.org/abs/1607.01759&quot; rel=&quot;noreferrer&quot;&gt;this paper&lt;/a&gt; there is a line which says &quot;linear  classifiers  do not  share  parameters  among  features  and  classes.&quot; What is the meaning of this statement? Does it mean that linear classifiers such as logistic regression need features that are mutually independent?&lt;/p&gt;&#xA;" OwnerUserId="39423" LastEditorUserId="34269" LastEditDate="2019-06-23T21:39:38.270" LastActivityDate="2019-06-23T21:39:38.270" Title="What does it mean to &quot;share parameters between features and classes&quot;" Tags="|machine-learning|logistic-regression|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30606" PostTypeId="2" ParentId="30211" CreationDate="2018-04-21T07:17:38.670" Score="2" Body="&lt;p&gt;You are probably trying to load word vectors that are shared in the fasttext.cc website or are trying to load using models that are trained with the latest one on master. There is &lt;a href=&quot;https://github.com/salestock/fastText.py/issues/115&quot; rel=&quot;nofollow noreferrer&quot;&gt;an issue in github&lt;/a&gt; with a lot of comments. One solution is to try to load from the text files the code for which is given &lt;a href=&quot;https://fasttext.cc/docs/en/crawl-vectors.html#format&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. I was able to work around this using java but had to train it in java only. Take a look at the examples in this package.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/vinhkhuc/JFastText/tree/master/examples/api&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/vinhkhuc/JFastText/tree/master/examples/api&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39423" LastActivityDate="2018-04-21T07:17:38.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30607" PostTypeId="1" AcceptedAnswerId="33826" CreationDate="2018-04-21T08:21:17.357" Score="1" ViewCount="17551" Body="&lt;p&gt;I have 52 CSV files in a folder. I want to build a model based on this data. That's why I want to Leave one out cross-validation on these data. How can I do this using sci-kit learn in python?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried from sci kit document and also search many resources.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import glob&#xA;import numpy as np&#xA;import pandas as pd&#xA;from sklearn.cross_validation import LeaveOneOut&#xA;path=r'...................\Data\New &#xA;design process data'&#xA;filelist=glob.glob(path + &quot;/*.csv&quot;)&#xA;loo=LeaveOneOut()&#xA;for train,test in loo.split(filelist):&#xA;   print(&quot;%s %s&quot; % (train, test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But it showed errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;init&lt;/strong&gt;() missing 1 required positional argument: 'n'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new in python as well as sci-kit learn. If anyone can help me, It would be a great convenience.&lt;/p&gt;&#xA;" OwnerUserId="44808" LastActivityDate="2020-06-13T18:12:22.910" Title="Leave one out Cross validation using sklearn (Multiple CSV)" Tags="|machine-learning|python|scikit-learn|cross-validation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30608" PostTypeId="2" ParentId="30603" CreationDate="2018-04-21T09:50:57.457" Score="3" Body="&lt;ol&gt;&#xA;&lt;li&gt;First try a simple model: The input layer and the output layers dimension are defined by your data / your problem definition. Then train a model without any hidden layer.&lt;/li&gt;&#xA;&lt;li&gt;See how good it performs. Is it good enough? If yes, you're done. If no, continue&lt;/li&gt;&#xA;&lt;li&gt;Add a hidden layer of reasonable size or adjust a hidden layers size. Go to step (2).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The &quot;reasonable&quot; size part might be difficult. As a guidance: If you have only a single node, it is certainly too small for a 1000 class problem. It might be big enough for a 2 - 3 class problem. I would usually suggest to keep the size of the features per layer roughly constant or at most reduce it by 1/10 or triple it. But that is only gut feeling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for my preference for simple models is &lt;a href=&quot;https://en.wikipedia.org/wiki/Occam%27s_razor&quot; rel=&quot;nofollow noreferrer&quot;&gt;Occam's razor&lt;/a&gt;, the fact that they are often faster, easier to analyze and to manually improve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more information about topology learning and rules how to design neural networks, see:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Thoma, Martin. &quot;&lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Analysis and Optimization of Convolutional Neural Network Architectures&lt;/a&gt;.&quot; arXiv preprint arXiv:1707.09725 (2017).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Especially chapter 2.5 and chapter 3.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-04-21T10:19:43.833" LastActivityDate="2018-04-21T10:19:43.833" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30609" PostTypeId="2" ParentId="30214" CreationDate="2018-04-21T10:50:39.383" Score="2" Body="&lt;p&gt;I don't think we are creating a number of negative examples for every positive example. Negative sampling is done to efficiently compute the softmax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Word2vec tries to maximize the similarity for words in similar contexts. For e.g. given drink the NN should predict water with maximum probability.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have V (typically &gt; 1 million) words in your training data, so the last layer of your model has V neurons. For every word in your training data, you would have to compute output from all the V neurons (to compute softmax). It is computationally very expensive. Negative sampling is one way to address this problem. Instead of computing the all the V outputs, we just sample few words and approximate the softmax. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negative sampling can be used to speed up neural networks where the number of output neurons is very high. Hierarchical softmax is another technique that's used for training word2vec.      &lt;/p&gt;&#xA;" OwnerUserId="29788" LastActivityDate="2018-04-21T10:50:39.383" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30610" PostTypeId="1" AcceptedAnswerId="30627" CreationDate="2018-04-21T11:40:16.237" Score="2" ViewCount="765" Body="&lt;p&gt;I am using a CNN for sentiment analysis of news articles. It is a binary classification with outputs: Interesting &amp;amp; Uninteresting.&#xA;In my dataset, there are around 50,000 Uninteresting articles and only about 200 Interesting articles. I know the ratio is badly skewed.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;My question is what should be the ratio in such a scenario.&lt;/li&gt;&#xA;&lt;li&gt;One approach that I want to try is to cluster the Uninteresting news&#xA;articles and take a sample from each cluster for training. Is there&#xA;a better approach?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="37585" LastActivityDate="2020-08-07T14:12:23.560" Title="What should be the ratio of True vs False cases in a binary classifier dataset?" Tags="|machine-learning|deep-learning|nlp|dataset|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30613" PostTypeId="1" CreationDate="2018-04-21T14:10:53.300" Score="1" ViewCount="44" Body="&lt;p&gt;I have a BSc in Applied Mathematics and I was thinking about an MSc in Data Science or Data Engineering, but I have no ideia what are the best programs available in Europe.&#xA;Any suggestions of good universities with MSc's in this field?&lt;/p&gt;&#xA;" OwnerUserId="49678" LastActivityDate="2018-04-21T14:10:53.300" Title="Best masters degree in data science and engineering in Europe" Tags="|data|" AnswerCount="0" CommentCount="1" ClosedDate="2018-04-23T01:39:24.157" ContentLicense="CC BY-SA 3.0" />
  <row Id="30614" PostTypeId="1" CreationDate="2018-04-21T14:26:46.380" Score="3" ViewCount="48" Body="&lt;p&gt;Is there any formula to find the output dimensions of a capsule network similar to that of a Convolutional Neural Network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For Example: In CNN, we know that &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(W - F + 2P/S)+1 = output_size where W = Dimensions of Input, F= Filter Size, p= Padding and S = Stride&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Kindly note that the same formula does not work for capsule networks as well.&lt;/p&gt;&#xA;" OwnerUserId="49441" LastActivityDate="2018-04-21T14:26:46.380" Title="Formula to calculate size of Capsule output similar to the formula for CNN?" Tags="|machine-learning|cnn|computer-vision|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30615" PostTypeId="2" ParentId="30088" CreationDate="2018-04-21T14:42:28.220" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=diUiV48q-5c&amp;amp;t=536s&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng word2vec&lt;/a&gt; @1:29. In the video, his choice of window size is more than one. In his example, if &quot;orange&quot; is the center word, the target word could be &quot;juice&quot;, &quot;glass&quot;, &quot;my&quot;, which are all within the window size of 5. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In your example, &quot;cat sat on the mat&quot;, if you choose a window size of one, you can get 7 training samples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;(cat, sat),&#xA;(sat, on),(sat, cat),&#xA;(on, sat), (on, the), &#xA;(the, on), (the, mat)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If this is the entire corpus, p(cat|sat), probability of &lt;em&gt;sat&lt;/em&gt; occurs as the center word, &lt;em&gt;cat&lt;/em&gt; being the context, is 0.5, p(sat|cat), &lt;em&gt;cat&lt;/em&gt; being the center word, &lt;em&gt;sat&lt;/em&gt; being the context is 1. The distributed word representation of &quot;cat&quot; and &quot;sat&quot; being the center word should be different.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&quot;it seems like for the same center word we can get different outputs:&quot; That's because they are predicting more than one context words. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;@31:35 &lt;a href=&quot;https://www.youtube.com/watch?v=ERibwqs9p38&amp;amp;t=4201s&quot; rel=&quot;nofollow noreferrer&quot;&gt;224 2017 lec2&lt;/a&gt;, the slide says &quot;predict surrounding words of radius m for every word&quot;. That is, the problem is to find a distribution of context words given the center word, so for one center word, the model will have more than one output words (unless you set &lt;em&gt;m&lt;/em&gt; to 1 and specify the direction as well). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;@39:37 &lt;a href=&quot;https://www.youtube.com/watch?v=ERibwqs9p38&amp;amp;t=4201s&quot; rel=&quot;nofollow noreferrer&quot;&gt;224 2017 lec2&lt;/a&gt; The professor did a dot product between the matrix output and the word representation. It's also shown on the top of the note: &quot;$u_o ^T v_c$&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multiply the hidden layer (d * 1) by Weights 2 (V * d) will get a V*1 vector. That's $u_o$, output vector representation. Then we do $u_o ^T v_c$ to get the output for each context word. After that, we apply softmax and compare that value with the context word's one-hot representation ($y$ vs $\hat{y}$), three comparisons in the prof's example.&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;That sounds like the idea of the Continuous Bag of Words model.  &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50893" LastEditorUserId="50893" LastEditDate="2018-04-22T08:44:07.757" LastActivityDate="2018-04-22T08:44:07.757" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30616" PostTypeId="2" ParentId="30603" CreationDate="2018-04-21T14:52:52.183" Score="1" Body="&lt;p&gt;Because your inputs have numerous features, it is not clear whether your data-set is linearly separable using hyperplanes as a hypothesis or not due to the fact that you can not visualize your data and see the outcomes. First, try to add one simple neuron which has a linear calculation and a nonlinearity activation. The cost function of a single neuron is convex and it shows you whether your data is linearly separable or not in that feature space. If you have high error rate then you have to add neurons. For figureing out how you should add extra neurons and layers take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/26642/28175&quot;&gt;here&lt;/a&gt; which may help you.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-21T14:52:52.183" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30617" PostTypeId="1" CreationDate="2018-04-21T15:22:50.920" Score="3" ViewCount="1587" Body="&lt;p&gt;Why is RMSProp in many cases converging faster than Momentum?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Momentum:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_{dW} := \beta v_{dw} +(1-\beta)dW$$&#xA;$$W := W-\alpha v_{dw}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;RMSProp:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ S_{dw} := B \cdot S_{dw} + (1-B)\cdot (dW)^2$$&#xA;  $$W := W- \alpha \frac{dW}{\sqrt{S_{dw}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $\alpha$ is the learning rate (0.01 etc), $\beta$ is the momentum term (0.9 etc), similar to B&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my point of view, both momentum and RMSProp have &quot;tendency to keep moving&quot;. Well, I can see how RMSprop will naturally accelerate on flat surfaces due to&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{1}{\sqrt{S_{dw}}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;when $S_{dw}$ is small, but is there another benefit that RMSprop provides?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-07-31T19:46:28.427" LastActivityDate="2020-09-29T18:11:27.103" Title="Why RMSProp converges faster than Momentum?" Tags="|optimization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30618" PostTypeId="2" ParentId="30617" CreationDate="2018-04-21T15:38:35.783" Score="3" Body="&lt;p&gt;The basic intuition is that you should not have the same learning rate for different dimensions. For instance, you can have a high slope in one direction but not for another. Consequently, you should not have the same speed for the two directions. Momentum adds acceleration. Suppose gradient is your instant velocity and the average is your average velocity. Momentum is actually viscosity or somehow friction. Suppose that you are near your optimal points, your gradients become zero and you have low average which means your speed changes slowly. They have both alpha term but what is going to be used is the running average, just a kind of average which is simple to be calculated. Take a look at &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/BhJlm/rmsprop&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/y0m1f/gradient-descent-with-momentum&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for making an analogy.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2020-09-29T18:11:27.103" LastActivityDate="2020-09-29T18:11:27.103" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="30619" PostTypeId="1" CreationDate="2018-04-21T16:07:36.777" Score="2" ViewCount="52" Body="&lt;p&gt;I am trying to build machine learning models (GBM, RF, Staking)&#xA;on top of a dataset that is about 3G in size on my local computer.&#xA;However, I only have 4G memory (only 2G are available). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is : is it logical to split the whole data on 20% for &#xA;training set, 10% on validation set, and the 70% for the testing part?&#xA;I split also the test set on 7 equal subsets with same distributions.&#xA;I am doing this due to the fact I cannot test a model on full dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still I am not really convinced about this solution, and I am not sure&#xA;that it is good enough to get a robust final model. &#xA; What can I do? I am new to machine learning and big data.&lt;/p&gt;&#xA;" OwnerUserId="50900" LastEditorUserId="29575" LastEditDate="2018-04-21T19:18:05.883" LastActivityDate="2018-04-21T19:18:05.883" Title="Split large dataset for predictive modeling using rsparkling -sparklyr" Tags="|predictive-modeling|bigdata|rstudio|machine-learning-model|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30620" PostTypeId="1" AcceptedAnswerId="30625" CreationDate="2018-04-21T16:53:20.980" Score="3" ViewCount="5529" Body="&lt;p&gt;I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: X has 4877 features per sample; expecting 2799&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a way to delete any features in x_test that were not used in x_train?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried imputation but couldn't get it to work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="50899" LastActivityDate="2018-04-21T18:48:11.580" Title="Python Sklearn TfidfVectorizer Feature not matching; delete?" Tags="|python|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30622" PostTypeId="1" CreationDate="2018-04-21T18:00:55.907" Score="1" ViewCount="151" Body="&lt;p&gt;I was studying about Association Rule Learning and got to know about FP-Growth algorithm. I am wondering that how is FP-Tree maintained for a really large dataset, for example, that of Amazon and eBay.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Is the whole dataset stored in just one FP-Tree?&lt;/li&gt;&#xA;&lt;li&gt;If not, then how are multiple FP-Tree generated and stored? Like, is the dataset divided or something?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="50907" LastActivityDate="2018-04-21T18:00:55.907" Title="How is FP-Tree used in FP-Growth maintained for large dataset" Tags="|machine-learning|data-mining|dataset|association-rules|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30623" PostTypeId="1" CreationDate="2018-04-21T18:02:20.363" Score="3" ViewCount="133" Body="&lt;p&gt;Sorry for the vagueness of the title; I'll explain what I mean. I'm doing the &lt;a href=&quot;https://www.kaggle.com/c/titanic&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle Titanic beginner tutorial&lt;/a&gt;. The label you're interested in is the &quot;survived&quot; rate (0 or 1), which you want to predict in the test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the first things I did was plot the average survival rate for each of the features, for each one, taking the average survival rate for that value of the feature. For example, for the &quot;Sex&quot; feature, you can look at the average survival rate for males and females, which gives you good info you can use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For features like &quot;Fare&quot; and &quot;Age&quot;, which are more continuous, it wouldn't really make sense to take the average in the same way, because there aren't necessarily many people with the same exact age. So, I did &quot;banding&quot;, where I do a histogram-like thing of saying that everyone from 0-10 is one band, 10-20 another band, etc, and those bands are now categorical features that can be averaged over well (this is a good move, right? at least initially?).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tYTtt.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tYTtt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's the background. Now, here's my question. I want to start with a simple model like linear regression. For some features, like Sex, or Pclass, it's clear that a linear model could fit it very well. For some however, like AgeBand, there's not really a clear, monotonic trend for increasing AgeBand...but there definitely is a difference in the survival rate between different bands!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what you could do is, sort those &quot;non monotonic&quot; features &lt;em&gt;by&lt;/em&gt; the survival rate, so it is monotonic, and then use them for linear regression. Is that a good idea or no?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, here's the AgeBand feature sorted by the survival rate:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LNxcM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LNxcM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50805" LastActivityDate="2018-07-22T15:10:53.280" Title="Does it make sense to &quot;reorder&quot; a categorical feature to make it monotonic?" Tags="|linear-regression|feature-extraction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30624" PostTypeId="2" ParentId="30620" CreationDate="2018-04-21T18:05:46.263" Score="1" Body="&lt;p&gt;I think you would need to delete exactly those features (columns) that are not known to your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine that you trained your model on the following features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;human weight&lt;/li&gt;&#xA;&lt;li&gt;human height&lt;/li&gt;&#xA;&lt;li&gt;sex&lt;/li&gt;&#xA;&lt;li&gt;age&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;and now you want to feed it a test data set containing the following features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;human weight&lt;/li&gt;&#xA;&lt;li&gt;human height&lt;/li&gt;&#xA;&lt;li&gt;sex&lt;/li&gt;&#xA;&lt;li&gt;age&lt;/li&gt;&#xA;&lt;li&gt;eye color&lt;/li&gt;&#xA;&lt;li&gt;name&lt;/li&gt;&#xA;&lt;li&gt;zip code&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;so if if you randomly reduce it to the correct shape and get:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;eye color  (should be: human weight)&lt;/li&gt;&#xA;&lt;li&gt;human height&lt;/li&gt;&#xA;&lt;li&gt;zip code  (should be: sex)&lt;/li&gt;&#xA;&lt;li&gt;name (should be: age)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;what kind of accuracy would you expect in this case?&lt;/p&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-04-21T18:05:46.263" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30625" PostTypeId="2" ParentId="30620" CreationDate="2018-04-21T18:48:11.580" Score="3" Body="&lt;p&gt;You should not experience this problem if you use TfidfVectorizer properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Demo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [58]: from sklearn.feature_extraction.text import TfidfVectorizer&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;source text&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [59]: text = &quot;&quot;&quot;I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.&#xA;    ...:&#xA;    ...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error&quot;&quot;&quot;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;let's tokenize it to a list of sentenses:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [60]: from nltk import  sent_tokenize&#xA;&#xA;In [61]: vect = TfidfVectorizer()&#xA;&#xA;In [62]: data = sent_tokenize(text)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;yields:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [63]: data&#xA;Out[63]:&#xA;['I trained a classifier using TfidfVectorizer in Sklearn.',&#xA; 'I then pickled the model for future use.',&#xA; 'The new x_test that I want to make predictions on, has more features than the x_train from the model.',&#xA; 'This is the resulting error']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now we can fit and transform our data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [64]: X = vect.fit_transform(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [65]: X&#xA;Out[65]:&#xA;&amp;lt;4x31 sparse matrix of type '&amp;lt;class 'numpy.float64'&amp;gt;'&#xA;        with 34 stored elements in Compressed Sparse Row format&amp;gt;&#xA;&#xA;In [66]: vect.get_feature_names()&#xA;Out[66]:&#xA;['classifier',&#xA; 'error',&#xA; 'features',&#xA; 'for',&#xA; 'from',&#xA; 'future',&#xA; 'has',&#xA; 'in',&#xA; 'is',&#xA; 'make',&#xA; 'model',&#xA; 'more',&#xA; 'new',&#xA; 'on',&#xA; 'pickled',&#xA; 'predictions',&#xA; 'resulting',&#xA; 'sklearn',&#xA; 'tfidfvectorizer',&#xA; 'than',&#xA; 'that',&#xA; 'the',&#xA; 'then',&#xA; 'this',&#xA; 'to',&#xA; 'trained',&#xA; 'use',&#xA; 'using',&#xA; 'want',&#xA; 'x_test',&#xA; 'x_train']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;now let's feed it a data set with unknown words (features):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [67]: new_dataset = [&quot;let's see what happens to unknown words&quot;, &quot;Yet another sentence.&quot;]&#xA;&#xA;In [68]: X2 = vect.transform(new_dataset)&#xA;&#xA;In [69]: X2&#xA;Out[69]:&#xA;&amp;lt;2x31 sparse matrix of type '&amp;lt;class 'numpy.float64'&amp;gt;'&#xA;        with 1 stored elements in Compressed Sparse Row format&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;it worked properly - all unknown features (words) have been ignored:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)&#xA;Out[70]:&#xA;   classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \&#xA;0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0&#xA;1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0&#xA;&#xA;   x_test  x_train&#xA;0     0.0      0.0&#xA;1     0.0      0.0&#xA;&#xA;[2 rows x 31 columns]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-04-21T18:48:11.580" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30626" PostTypeId="2" ParentId="17362" CreationDate="2018-04-21T20:13:09.877" Score="3" Body="&lt;p&gt;The best way to use brat on windows (10) is by using windows linux subsystem.&#xA;S1. Enable Developer mode&#xA;S2. Enable Windows Linux Subsystem by turning the windows feature on (control panel)&#xA;S3. Install Ubuntu from windows store.&#xA;S4. From bash, install python (sudo apt install python-minimal)&#xA;S5. Get brat downloaded on your favorite drive and folder, unpack it (7Zip).&#xA;S6. Go to the folder from bash (cd /mnt//...)&#xA;S7. ./install.sh, followed by python standalone.py, then copy and paste the localhost on your favorite browser!&lt;/p&gt;&#xA;" OwnerUserId="50913" LastActivityDate="2018-04-21T20:13:09.877" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30627" PostTypeId="2" ParentId="30610" CreationDate="2018-04-21T22:40:30.723" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Ideal true vs false ratios don't exist and they should reflect the the reality the best they can, you can always remove negatives if the ratio is too skewed to improve training speed though. Let me explain it with an example. Ads CTR is as old as the internet and it's skewed to less than 1% positives vs. plus 99% negatives. Yet, data scientists prefer to train it on the entire dataset because many negatives will include information that models couldn't find otherwise. They might not provide a lot of information as a positive one but they are still somewhat important. There are approaches where CTR ratios get artificially rebalanced by sampling in case you want a swifter training and it will still work. In your case, positives are 0.4% which resemble CTR on ads so you can: gather more data to increase the number of positives in order to better understand what makes an article interesting. In case that is not possible trying ensembles which often improve prediction performance.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Clustering is an unsupervised approach so you would be losing information by doing so (training labels) besides, sentence embeddings (representations)  of one big cluster of negatives and a tiny cluster of positives do not convey information as well as word embeddings which have already been trained on billions of documents.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;In addition, running k-means on categorical variables will yield anomalous clusters because it's meant to be used with continuous variables. You can find more information about the topic on the following links:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://Kmeans:%20Whether%20to%20standardise?%20Can%20you%20use%20categorical%20variables?%20Is%20Cluster%203.0%20suitable?&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kmeans: Whether to standardise? Can you use categorical variables? Is Cluster 3.0 suitable?&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://My%20data%20set%20contains%20a%20number%20of%20numeric%20attributes%20and%20one%20categorical.&quot; rel=&quot;nofollow noreferrer&quot;&gt;My data set contains a number of numeric attributes and one categorical&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/general/19741&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.quora.com/Why-does-K-means-clustering-perform-poorly-on-categorical-data-The-weakness-of-the-K-means-method-is-that-it-is-applicable-only-when-the-mean-is-defined-one-needs-to-specify-K-in-advance-and-it-is-unable-to-handle-noisy-data-and-outliers&quot; rel=&quot;nofollow noreferrer&quot;&gt;Why does K means clustering perform poorly on categorical data The weakness of the K means method is that it is applicable only when the mean is defined one needs to specify K in advance and it is unable to handle noisy data and outliers&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Therefore, you should use high dimensional embeddings or representations to cluster meanings together, this has been explored in word meanings but for sentences or articles, a vector representation becomes more complicated to implement. One possible approach is the Word Movers’ Distance but there are many more possible approaches, you should google them. In addition a non-linear clustering algorithm such as t-sne will  probably yield better results than k-means using the embeddings approach.&lt;/p&gt;&#xA;&lt;p&gt;A better approach is:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;to use multiple models and compare their performance on this dataset. I have the impression that there will be certain keywords that make articles interesting, so a bag of words will still be helpful, even as a starter model.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use feature engineering. Your model might be overloooking important features, such as article length, reading time, number of paragraphs, ratio of complex words  (measured by length), etc. Feature engineering is always important in case you haven't used it yet.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use pretrained embeddings. CNN and RNN models can use pretrained embeddings such as GloVe, Word2Vec or FastText so you use better representations plus other complex layers later on in the architecture. This is extremely important to increase accuracy.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use metrics to measure improvement and ranks to check for the best predicted interesting articles.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="13023" LastEditorUserId="98307" LastEditDate="2020-08-07T14:12:23.560" LastActivityDate="2020-08-07T14:12:23.560" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="30628" PostTypeId="1" CreationDate="2018-04-21T23:42:51.487" Score="2" ViewCount="222" Body="&lt;p&gt;I'm a beginner to machine learning. I have a large dataset of items, each with a limited set of most similar items in the dataset, which I would like to create clusters of, preferably each having the same size, although this isn't necessary. Could someone point me in the right direction? Thanks.&lt;/p&gt;&#xA;" OwnerUserId="50917" LastActivityDate="2018-04-23T04:53:18.823" Title="Clustering algorithms based on lists of most similar items?" Tags="|machine-learning|data-mining|clustering|dataset|" AnswerCount="3" CommentCount="2" ClosedDate="2018-04-23T08:12:59.367" ContentLicense="CC BY-SA 3.0" />
  <row Id="30629" PostTypeId="2" ParentId="30581" CreationDate="2018-04-22T00:22:52.927" Score="1" Body="&lt;p&gt;As @Anony-Mousse pointed it, on DBSCAN index structures are often used in order to decrease execution times. K-d-trees are one example but this one works well just in small dimensions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You had right intuition about what slowing down, the computation of every distance from all point to one is O(n) time complexity but applied to every points it becomes O(n&lt;sup&gt;2&lt;/sup&gt;) which is something we desire to avoid on important set of points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover you may reconsider your clustering approach because DBSCAN is  known to works well with small dimensional datasets due to hyperball volume fast decrease with higher dimensions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keep trying implementing algorithms by yourself it is best way to learn even if existing mature library do often a better job but are also more difficult to understand, one step after another you will be able to write better algorithms and contribute to open source libraries.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="924" LastEditDate="2018-04-22T17:47:04.750" LastActivityDate="2018-04-22T17:47:04.750" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30630" PostTypeId="1" CreationDate="2018-04-22T00:55:15.767" Score="4" ViewCount="5191" Body="&lt;p&gt;I'm using the newest version of xgboost package in python 2.7 and based on my problem, I'm going to change xgboost cost function to use my own defined cost function. Couple of questions:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In which part of the xgboost package should I find and change the cost function?&#xA;After changing the cost function, how can I add the updated code to the xgboost package to use the new training function instead of xgboost's pre-defined function?&lt;/p&gt;&#xA;" OwnerUserId="50919" LastActivityDate="2020-09-09T15:22:13.023" Title="changing cost function in xgboost" Tags="|machine-learning|python|xgboost|" AnswerCount="1" CommentCount="1" ClosedDate="2020-09-09T16:18:28.387" ContentLicense="CC BY-SA 3.0" />
  <row Id="30631" PostTypeId="2" ParentId="30628" CreationDate="2018-04-22T04:04:48.707" Score="0" Body="&lt;p&gt;You can go for K-means clustering to cluster the data based on the pattern.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-22T04:04:48.707" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30632" PostTypeId="2" ParentId="27288" CreationDate="2018-04-22T05:54:21.070" Score="4" Body="&lt;p&gt;There are functions in the Estimator class that handle this, namely: &lt;code&gt;get_variable_names&lt;/code&gt; and &lt;code&gt;get_variable_value&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess they want to avoid polluting the global scope. Note that you can pass a &lt;code&gt;name&lt;/code&gt; parameter to your layers on creation to disambiguate them.&lt;/p&gt;&#xA;" OwnerUserId="50925" LastEditorUserId="29575" LastEditDate="2018-04-22T14:03:24.280" LastActivityDate="2018-04-22T14:03:24.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30633" PostTypeId="2" ParentId="30560" CreationDate="2018-04-22T06:23:56.397" Score="-1" Body="&lt;p&gt;This problem is perfectly suited for a neural network. Your model will have 40 input nodes (this is fine), then you will have some arbitrary hidden layers, you need to tune this, and 20 outputs. After the training process you can even get a probability for each of them. This can be used to rank suggestions for potential students!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;How to do this&lt;/h1&gt;&#xA;&#xA;&lt;h2&gt;Load the data to memory&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Depending on the means by which your data is stored this step will be different. However, the goal is to go from the raw file source to either Python Numpy array or a Pandas DataFrame. I will assume your data is structured as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Px0mH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Px0mH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and is saved as a .csv file.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's load our data into an X and Y matrix. We will be encoding the labels for 'degree' as values. Make sure these are all well spelled otherwise a new label will be created for the mispelled ones. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;&#xA;df = pd.read_csv('test1.csv')&#xA;df['Degree'] = df['Degree'].astype('category')&#xA;df[&quot;Degree_encoding&quot;] = df[&quot;Degree&quot;].cat.codes&#xA;&#xA;X = np.asarray(df.loc[:, df.columns != 'Degree'])&#xA;Y = df['Degree_encoding']&#xA;&#xA;print(X.shape)&#xA;print(Y.shape)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(39, 7) &lt;br/&gt;(39,)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h2&gt;Applying a neural network to this dataset&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;First we will split the data randomly into a training and testing set. This is used to evaluate our model while maintaining that we are not overfitting. Then we identify the number of output classes. Then we reshape the input matrices such that they have a channel in their last dimension, this is how data flows through the model. Then we will categorize our outputs as one-hot encoded vectors.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import train_test_split&#xA;import keras&#xA;&#xA;# Split the data&#xA;x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)&#xA;&#xA;# The known number of output classes.&#xA;num_classes = len(set(df[&quot;Degree_encoding&quot;]))&#xA;# Input dimensions&#xA;shape = X.shape[1::]&#xA;&#xA;# We need to add a channels dimension to our data&#xA;# Channels go last for TensorFlow backend in Keras&#xA;x_train_reshaped = x_train.reshape((x_train.shape[0],) + shape)&#xA;x_test_reshaped = x_test.reshape((x_test.shape[0],) + shape)&#xA;input_shape = shape&#xA;print(input_shape)&#xA;&#xA;# Convert class vectors to binary class matrices. This uses 1 hot encoding.&#xA;y_train = keras.utils.to_categorical(y_train, num_classes)&#xA;y_test = keras.utils.to_categorical(y_test, num_classes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We then design our model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Dense(32,&#xA;                 activation='relu',&#xA;                 input_shape=input_shape))&#xA;model.add(Dense(128, activation='relu'))&#xA;model.add(Dense(num_classes, activation='softmax'))&#xA;&#xA;model.compile(loss=keras.losses.categorical_crossentropy,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can use &lt;code&gt;model.summary()&lt;/code&gt; to get a description of these layers. Then we are ready to train our model!&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs = 100&#xA;batch_size = 128&#xA;model.fit(x_train_reshaped, y_train,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test_reshaped, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PORhj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PORhj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-22T06:23:56.397" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30634" PostTypeId="2" ParentId="30630" CreationDate="2018-04-22T06:49:56.397" Score="3" Body="&lt;p&gt;This is discussed in &lt;a href=&quot;https://stackoverflow.com/questions/37436629/how-to-write-a-custom-evaluation-metric-in-python-for-xgboost&quot;&gt;stackoverflow&lt;/a&gt;, just to recap: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Define your your customized cost function, e.g.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; def new_cost(y_pred, y_true):&#xA;    # perform calculation for new cost&#xA;    return 'new_cost', score&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then pass it to the feval argument (see &lt;a href=&quot;http://xgboost.readthedocs.io/en/latest/python/python_api.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;official doc&lt;/a&gt;) for training like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = xgboost.train(params, dtrain, num_rounds, watchlist, feval=new_cost)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There are many examples out there if you search harder for more concrete use cases.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-04-22T06:49:56.397" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30635" PostTypeId="1" CreationDate="2018-04-22T07:19:39.510" Score="3" ViewCount="1073" Body="&lt;p&gt;So I have a trained LSTM model with which I am trying to predict future values. The model is stateful as seen below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(GRU(units=50,batch_size=1, input_shape=(None, 1),stateful=True, return_sequences=True))&#xA;model.add(Activation('tanh'))&#xA;model.add(GRU(units=50,stateful=True,return_sequences=True))&#xA;model.add(Dropout(.1))&#xA;model.add(GRU(1, return_sequences=True,stateful=True))&#xA;model.add(Lambda(lambda x: x * 1.0012))&#xA;model.load_weights(&quot;weights.h5&quot;)&#xA;model.compile(loss='mse', optimizer='adam')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I try to predict on the entire data and then pass that through in a loop to predict future vals I get the title error. This ONLY occurs if steps is set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I set steps with a batch_size it tells me to set batch_size to None and if I do that I get the error above. Very strange. What's going on here? It also will work if I do not set steps and set batch to any int divisible by the # of samples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the prediction code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.reset_states()&#xA;predictions = model.predict(entire_data,batch_size=1)&#xA;futureElement = predictions[:,-1:,:]&#xA;futureElements = []&#xA;forecast1 = []&#xA;futureElements.append(futureElement)&#xA;for i in range(time):&#xA;    futureElement = model.predict(futureElement,batch_size=None,steps=60)&#xA;    forecast1.append(futureElement)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31046" LastEditorUserId="31046" LastEditDate="2018-04-24T02:44:30.697" LastActivityDate="2018-04-24T02:44:30.697" Title="TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'(Stateful LSTM Keras)" Tags="|deep-learning|keras|lstm|sequence|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30636" PostTypeId="2" ParentId="27870" CreationDate="2018-04-22T10:19:05.003" Score="-1" Body="&lt;p&gt;Data (singular): The value of the variable associated with one element of a population or&#xA;sample. This value may be a number, a word, or a symbol.&#xA;• Data (plural): The set of values collected for the variable from each of the elements belonging&#xA;to the sample&lt;/p&gt;&#xA;" OwnerUserId="50928" LastActivityDate="2018-04-22T10:19:05.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30637" PostTypeId="2" ParentId="30628" CreationDate="2018-04-22T11:07:42.097" Score="0" Body="&lt;p&gt;All clustering algorithms, by default cluster the &lt;strong&gt;most similar items&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are searching for a specific similarity (e.g. similar size) you need to select an &lt;strong&gt;appropriate distance metric&lt;/strong&gt; (e.g. one that compares sizes).&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-22T11:07:42.097" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30639" PostTypeId="2" ParentId="30605" CreationDate="2018-04-22T12:23:44.453" Score="24" Body="&lt;p&gt;I will try to answer this question through &lt;strong&gt;logistic regression&lt;/strong&gt;, one of the simplest linear classifiers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The simplest case of logistic regression is if we have a &lt;strong&gt;binary classification&lt;/strong&gt; task ($y \in\{0,1\})$and only &lt;strong&gt;one input feature&lt;/strong&gt; ($x \in R$). In this case the output of logistic regression would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat y = σ(w \cdot x + b)&#xA;$$&#xA;where $w$ and $b$ are both &lt;strong&gt;scalars&lt;/strong&gt;. The output of the model $\hat y \in [0,1]$ corresponds to the probability that $x$ will be of class $1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We'll try to break down the phrase &lt;em&gt;&quot;linear classifiers do not share parameters among features and classes&quot;&lt;/em&gt; into two parts. We will examine the cases of multiple features and multiple classes separately to see if logistic regression shares parameters for any those tasks:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do linear classifiers share parameters among features?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, for each example, $y$ is a scalar that takes binary values (like before), while $x$ is a &lt;strong&gt;vector&lt;/strong&gt; of length $N$ (where $N$ is the number of features). Here, the the output is a linear combination of the input features (i.e. a weighted sum of these features plus the biases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat y = σ \left(\sum_i^N{(w_i \cdot x_i)} + b\right) \;\; or \;\; σ( \mathbf w \cdot \mathbf x + b)&#xA;$$&#xA;where $ \mathbf x $ and $ \mathbf w $ are vectors of length $N$. The product $\mathbf x \cdot \mathbf w$ produces a scalar. As you can see from above there is a &lt;strong&gt;separate weight&lt;/strong&gt; $w_i$ for each input feature $x_i$ and these weights are &lt;strong&gt;independent&lt;/strong&gt; by all means. From this we can conclude that there is &lt;strong&gt;no parameter sharing among features&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Do linear classifiers share parameters among classes?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case $x$ is a scalar, however $y$ is a &lt;strong&gt;vector&lt;/strong&gt; of length $M$ (where $M$ is the number of classes). To tackle this, logistic regression essentially produces a separate output $y_j$ for each of the $M$ classes. Each output is a scalar $y_j \in [0,1]$ and corresponds to the probability of $x$ belonging to class $j$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mathbf{ \hat y} = w \cdot \mathbf x + \mathbf b, \;\; where \;\; \mathbf{ \hat y} = {\hat y_1, \hat y_2, ..., y_M}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to think of this is as $M$ simple &lt;strong&gt;independent&lt;/strong&gt; logistic regressions each with an output of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat y_j = σ(w_j \cdot x + b_j)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the above it is obvious that &lt;strong&gt;no weights are shared among the different classes&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;multi-feature and multi-class&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By combining the two cases above we can finally reach the most general case of multiple features and multiple classes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\mathbf{ \hat y} = σ( \mathbf W \cdot \mathbf x + \mathbf b)&#xA;$$&#xA;where $\mathbf{ \hat y}$ is a vector with a size of $M$, $\mathbf x$ is a vector with a size of $N$, $\mathbf b$ is a vector with a size of $M$ and $W$ is a matrix with a size of $(N \times M)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In any case, &lt;strong&gt;linear classifiers do not share any parameters among features or classes&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your second question, linear classifiers &lt;strong&gt;do have an underlying assumption that features need to be independent&lt;/strong&gt;, however this is &lt;strong&gt;not&lt;/strong&gt; what the author of the paper intended to say.&lt;/p&gt;&#xA;" OwnerUserId="34269" LastEditorUserId="34269" LastEditDate="2018-09-18T08:22:20.757" LastActivityDate="2018-09-18T08:22:20.757" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30642" PostTypeId="2" ParentId="12851" CreationDate="2018-04-22T13:48:43.237" Score="33" Body="&lt;p&gt;There is an open source project called &lt;a href=&quot;https://github.com/lutzroeder/Netron&quot; rel=&quot;noreferrer&quot;&gt;Netron&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Netron is a viewer for neural network, deep learning and machine learning models.&lt;/p&gt;&#xA;&lt;p&gt;Netron supports ONNX (.onnx, .pb), Keras (.h5, .keras), CoreML (.mlmodel) and TensorFlow Lite (.tflite). Netron has experimental support for Caffe (.caffemodel), Caffe2 (predict_net.pb), MXNet (-symbol.json), TensorFlow.js (model.json, .pb) and TensorFlow (.pb, .meta).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R7PT1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R7PT1.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50932" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-22T13:48:43.237" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30644" PostTypeId="1" AcceptedAnswerId="30668" CreationDate="2018-04-22T17:17:39.753" Score="1" ViewCount="1577" Body="&lt;p&gt;Can someone help me to formulate an MDP for the below problem ?.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem Definition&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bunny wakes up in a strange room with 2 doors; one on the left, and one on the&#xA;right.  In front of him is a map of the building, laying out clearly what lies behind each door.  Behind one of the doors is a room with the way outside, and behind the other one a room with a hungry tiger.  Bunny does not like tigers.  Especially hungry ones.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Consider  the  control  problem  where  the  current  state  is  specified  by  the current  room  Bunny  is  in,  and  the  actions  Bunny  can  take  are  to  move through  a  door  to  another  room.   Assume  that  there  are  2  doors  in  the starting room,  each leading to its own room,  and that one of these other rooms (specifically the one on the left) has a single door that leads to the exit, while the other one (specifically the one on the right) has a tiger in it.&#xA;Moreover, assume that Bunny has full knowledge of these details but that&#xA;there is a 10% chance that when trying to move through one of the doors,&#xA;he gets confused and goes through the other door instead.  Lastly, assume&#xA;that the episode does not end when Bunny enters the room with the tiger.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Formulate a Markov Decision Process (MDP) for the problem of con-&#xA;trolling Bunny’s actions in order to avoid the tiger and exit the building.&#xA;(Give the transition and reward functions in tabular format, or give the&#xA;transition graph with rewards).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to know, is there any procedures or rules, that needs to be considered before formulating an MDP for a problem&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What i have done so far&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/njZ4K.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/njZ4K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;State A - Starting room&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;State B - Room with an exit door&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;State C - Room that has a tiger in it&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I don't know how to proceed further.&lt;/p&gt;&#xA;" OwnerUserId="2893" LastActivityDate="2018-04-23T08:27:27.900" Title="Formulate a MDP for a problem given below" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30645" PostTypeId="1" CreationDate="2018-04-22T17:26:03.397" Score="0" ViewCount="44" Body="&lt;p&gt;Hopefully someone may point me in the right direction for this problem of mine, what would be the best ML algorithm to use in order to match an image with another (or very similar) image on the web? When i say best i am thinking about performance and also speed on the user's end&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50939" LastActivityDate="2018-04-22T17:26:03.397" Title="Match a uploaded image with another image using machine learning?" Tags="|machine-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30646" PostTypeId="2" ParentId="29311" CreationDate="2018-04-22T19:22:56.143" Score="1" Body="&lt;p&gt;Work with a Embedding Matrix. The Embedding Matrix Dimension = all of your unique Tokens x Vector Dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your Embedding Layer should have the Dimension WordVektor x length of your Text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Watch this: watch &lt;a href=&quot;https://www.youtube.com/watch?v=eZ_HocUaJhU&quot; rel=&quot;nofollow noreferrer&quot;&gt;this video&lt;/a&gt; Part 16 and Part 17. Easy explanation and short videos. &lt;/p&gt;&#xA;" OwnerUserId="50944" LastEditorUserId="29575" LastEditDate="2018-04-22T19:40:57.920" LastActivityDate="2018-04-22T19:40:57.920" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30647" PostTypeId="1" CreationDate="2018-04-22T19:49:36.230" Score="1" ViewCount="780" Body="&lt;p&gt;I am trying to calculate BIC in python. In python, there is no inbuilt library for computing BIC. I referenced the following link to compute variance and BIC further:- &lt;a href=&quot;https://stats.stackexchange.com/questions/90769/using-bic-to-estimate-the-number-of-k-in-kmeans?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&quot;&gt;https://stats.stackexchange.com/questions/90769/using-bic-to-estimate-the-number-of-k-in-kmeans?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&lt;/a&gt;&#xA;The variance formula given in the highest voted answer is not working for my code design as centers is a two-dimensional array and in my case it's a single dimensional array. So I converted the variance formula given in the link to calculate but I learned it's not the correct way. &#xA;Here is my python code:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def compute_variance_grayscale(self, clusters, centroids, gImage, k):&#xA;    # compute variance&#xA;    variance = 0&#xA;&#xA;    for i in range(len(clusters)): #iterate over the available clusters&#xA;        cluster = clusters[i]&#xA;        sum = 0;&#xA;        centroid = centroids[i]&#xA;        for j in cluster: #for each data point in cluster j compute the sum&#xA;            sum += (gImage.getImageValueAt(j[0], j[1]) - centroid) ** 2&#xA;&#xA;        variance += sum / (len(cluster) - k); #add the sum to the given variance&#xA;&#xA;    return variance&#xA;&#xA;        def compute_variance_rgb(self, clusters, centroids, cImage, k):&#xA;    # compute variance&#xA;    variance = 0&#xA;    for i in range(len(clusters)):&#xA;        cluster = clusters[i]&#xA;        sum = 0;&#xA;        centroid = centroids[i]&#xA;        for j in cluster:&#xA;            b,g,r = cImage.getImageValueAt(j[0], j[1])&#xA;            b_e = (b - centroid[0]) ** 2&#xA;            g_e = (g - centroid[1]) ** 2&#xA;            r_e = (r - centroid[2]) ** 2&#xA;&#xA;            sum += (b_e + g_e + r_e)&#xA;&#xA;        variance += sum / (len(cluster) - k);&#xA;&#xA;    return variance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone please point to the right way to calculate variance? &#xA;Also, it would be great if someone could explain to me the computation of BIC and use of const terms as mentioned in the highest voted answer in the above link:-&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;np.sum([n[i] * np.log(n[i]) -&#xA;               n[i] * np.log(N) -&#xA;             ((n[i] * d) / 2) * np.log(2*np.pi*cl_var) -&#xA;             ((n[i] - 1) * d/ 2) for i in range(m)]) - const_term&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50945" LastEditorUserId="1330" LastEditDate="2021-09-18T12:20:24.257" LastActivityDate="2021-09-18T12:20:24.257" Title="Correct calculation of BIC (Bayesian Information Criterion) to determine K for K-Means" Tags="|python|scikit-learn|k-means|bayesian|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30648" PostTypeId="1" CreationDate="2018-04-22T20:07:23.737" Score="3" ViewCount="1760" Body="&lt;p&gt;I have images with and without watermark. There is only one type of watermark. I have tried &lt;code&gt;VGG16&lt;/code&gt; transfer learning, but results were bad. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are the methods to do image classification if it has a watermark or not?&lt;/p&gt;&#xA;" OwnerUserId="50946" LastEditorUserId="1330" LastEditDate="2018-04-24T19:52:09.423" LastActivityDate="2018-06-23T20:53:22.523" Title="Watermark detection using Deep Learning" Tags="|machine-learning|neural-network|deep-learning|cnn|computer-vision|" AnswerCount="2" CommentCount="2" ClosedDate="2018-06-24T21:16:40.643" ContentLicense="CC BY-SA 3.0" />
  <row Id="30649" PostTypeId="2" ParentId="30648" CreationDate="2018-04-22T20:52:28.677" Score="0" Body="&lt;p&gt;You could consider finding watermark an &lt;a href=&quot;https://en.wikipedia.org/wiki/Object_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;object detection&lt;/a&gt; problem, not an image classification problem. Faster R-CNN and &quot;You only look once&quot; (YOLO) are both general object detection architectures.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-22T20:52:28.677" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30650" PostTypeId="2" ParentId="30648" CreationDate="2018-04-22T21:20:07.793" Score="0" Body="&lt;p&gt;One of the good approaches for dealing with such problems is using landmark detection. In this case, your problem will look like a combination of classification and regression task using deep neural networks. Take a look at &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/OkD3X/landmark-detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://pdfs.semanticscholar.org/46b2/c6727f5ccf06f53020844aa0157237ac5a89.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. You should specify points which have the same identity among all data-set. Then your network will try to find the position of each point. Also take a look at &lt;a href=&quot;http://www.robots.ox.ac.uk/~vgg/software/via/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-22T21:20:07.793" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30652" PostTypeId="1" AcceptedAnswerId="31296" CreationDate="2018-04-22T23:43:09.317" Score="2" ViewCount="4087" Body="&lt;p&gt;I want to do spell correction for the portuguese language, specifically for restaurant bots. The problem is that food names aren't always in portuguese as well, and for that reason are the most likely to need correction as the user would not always know how to spell it correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I though of a few things. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Training the model with portugese words + food words spelled wrong,&#xA;but it'd be very difficult to find these food words&lt;/li&gt;&#xA;&lt;li&gt;Training the model for several different languages (but I guess would&#xA;make it confused and actually correct a lot of things wrong).&lt;/li&gt;&#xA;&lt;li&gt;Training the model with portuguese words and for the food words, use&#xA;something that picks words that the user wrote and try to approximate&#xA;them to the list of food words. (woudln't it be slow?)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What would be a good solution to this problem that can be fast to be used with lots of requests?&lt;/p&gt;&#xA;" OwnerUserId="50950" LastActivityDate="2018-05-14T15:49:39.887" Title="How to do spelling correction for a language but also correct some words in other language" Tags="|machine-learning|deep-learning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30654" PostTypeId="1" CreationDate="2018-04-23T00:31:19.570" Score="0" ViewCount="58" Body="&lt;p&gt;I created a convolutional network to recognise certain substrings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, the following phrases would be mapped to the &quot;What&quot; class:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;What are you doing?&#xA;What you doing?&#xA;Whatcha doing?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The following to the &quot;How&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;How are you doing?&#xA;How you doing?&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, say I have millions of examples and thousands of classes, my network learns relatively well. However, sometimes a new &quot;phrase&quot; that belongs to a never seen class appears. Of course, the model cannot map it to something it has never seen, however, when I look at the output layer values, I was hoping to see all neurons to have small values. My idea was that, I would use the softmax and then if the max value is less than 0.95 or some high value, I would assume the network is unsure, therefore remove that result for manual observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in many cases, I have seen my network to be almost sure (a neuron with value close to 1) even when it has never seen that class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, say I have the two classes What and How, I would expect:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Whatcha doing? --&amp;gt; &quot;What&quot;= 0.95, &quot;How&quot;= 0.05&#xA;How you doing? --&amp;gt; &quot;What&quot; = 0.02, &quot;How&quot;= 0.98&#xA;Where you going?--&amp;gt; &quot;What&quot;= 0.6, &quot;How&quot;=0.4&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the last case, the network is not too sure on either, and I would check the input. Unfortunately, this is not working, and I am getting high values (close to 1) neurons for some new input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I fix this? Please, bear in mind, I am happy to sacrifice some accuracy in order to ensure the network never tells me it is sure about new inputs, in other words, I am happy to go and manually check more inputs (even if seen before) rather that let even one unseen input go unnoticed.&lt;/p&gt;&#xA;" OwnerUserId="13736" LastEditorUserId="29169" LastEditDate="2021-02-13T04:24:01.413" LastActivityDate="2021-02-13T04:24:01.413" Title="How can we create neural net to detect false predictions?" Tags="|neural-network|accuracy|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30657" PostTypeId="2" ParentId="30628" CreationDate="2018-04-23T04:53:18.823" Score="0" Body="&lt;p&gt;Look at graph clustering algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have an edge from each item to it's most similar items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use e.g. spectral clustering.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-23T04:53:18.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30658" PostTypeId="2" ParentId="30575" CreationDate="2018-04-23T04:56:33.983" Score="0" Body="&lt;p&gt;There exist approaches for learning the &lt;em&gt;structure&lt;/em&gt; of Bayesian networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They just don't seem to be very popular, so I don't think you'll find them in standard toolkits.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-23T04:56:33.983" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30659" PostTypeId="1" AcceptedAnswerId="30671" CreationDate="2018-04-23T05:25:07.953" Score="13" ViewCount="18320" Body="&lt;p&gt;I am trying to fine tune some code from a &lt;a href=&quot;https://www.kaggle.com/fujisan/use-keras-pre-trained-vgg16-acc-98&quot; rel=&quot;noreferrer&quot;&gt;Kaggle kernel&lt;/a&gt;. The model uses pretrained VGG16 weights (via 'imagenet') for transfer learning. However, I notice there is no layer freezing of layers as is recommended in a &lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;noreferrer&quot;&gt;keras blog&lt;/a&gt;. One approach would be to freeze the all of the VGG16 layers and use only the last 4 layers in the code during compilation, for example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for layer in model.layers[:-5]:&#xA;    layer.trainable = False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Supposedly, this will use the imagenet weights for the top layers and train only the last 5 layers. What are the consequences of not freezing the VGG16 layers? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential, Model, load_model&#xA;from keras import applications&#xA;from keras import optimizers&#xA;from keras.layers import Dropout, Flatten, Dense&#xA;&#xA;img_rows, img_cols, img_channel = 224, 224, 3&#xA;&#xA;base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows, img_cols, img_channel))&#xA;&#xA;add_model = Sequential()&#xA;add_model.add(Flatten(input_shape=base_model.output_shape[1:]))&#xA;add_model.add(Dense(256, activation='relu'))&#xA;add_model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;model = Model(inputs=base_model.input, outputs=add_model(base_model.output))&#xA;model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),&#xA;              metrics=['accuracy'])&#xA;&#xA;model.summary()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17539" LastEditorUserId="50727" LastEditDate="2020-07-13T11:41:24.617" LastActivityDate="2021-01-29T21:44:20.377" Title="What are the consequences of not freezing layers in transfer learning?" Tags="|machine-learning|deep-learning|keras|tensorflow|transfer-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30660" PostTypeId="1" AcceptedAnswerId="30663" CreationDate="2018-04-23T05:31:30.127" Score="0" ViewCount="405" Body="&lt;p&gt;I want to rescale the features of my data to be between [0,1] and [-1,1]? Is their a clear cut way that works every time for these ranges? I think the below equation works for [0,1] but when it is describe people say generally it works, so I am not certain it works every single time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1mvBP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1mvBP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28660" LastActivityDate="2018-04-24T01:51:49.247" Title="How to do Feature Scaling for these ranges [0,1] and [-1,1]?" Tags="|machine-learning|feature-selection|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30662" PostTypeId="1" CreationDate="2018-04-23T06:04:34.787" Score="0" ViewCount="268" Body="&lt;p&gt;I've created a simple 2-2-1 feedforward ANN to predict an XOR using Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The activation function I'm using on all layers is a &lt;em&gt;tanh&lt;/em&gt;, so in order to make use of the entire range of the function, &lt;em&gt;i.e.&lt;/em&gt; [-1, 1], I've decided to use -1 instead of 0 as the symbol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My input data is, thus, &lt;code&gt;[[-1, -1], [-1, 1], [1, -1], [1, 1]]&lt;/code&gt;, for an output of &lt;code&gt;[[-1], [1], [1], [-1]]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought this would give better results since I'm using the entire range of the function and it supposedly would converge better because of that. Also, since I'm just using a different symbol, it should be the same as with using 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, my network can not converge (giving a 0.5 accuracy), and what baffles me the most is that using 0 and 1 as symbols converges, and at a much faster rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a reason for such a counter-intuitive (at least in my conception) thing to be happening?&lt;/p&gt;&#xA;" OwnerUserId="50484" LastActivityDate="2018-04-23T06:47:54.790" Title="Neural network does not converge with negative symbols" Tags="|machine-learning|neural-network|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30663" PostTypeId="2" ParentId="30660" CreationDate="2018-04-23T06:33:10.987" Score="2" Body="&lt;p&gt;What you said is right, the above equation is for normalizing the data with-in the range of [0,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we can generalize using the below equation &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To normalize in $[-1,1]$ you can use:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;x'' = 2\frac{x - \min{x}}{\max{x} - \min{x}} - 1&#xA;$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, you can always get a new variable $x'''$ in $[a,b]$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;x''' = (b-a)\frac{x - \min{x}}{\max{x} - \min{x}} + a&#xA;$$ &lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-04-24T01:51:49.247" LastActivityDate="2018-04-24T01:51:49.247" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30664" PostTypeId="1" CreationDate="2018-04-23T06:33:25.700" Score="2" ViewCount="82" Body="&lt;p&gt;I have a set of unlabeled numeric data and i want to generate rules from it for classification purpose.  Which technique can i use? &lt;/p&gt;&#xA;" OwnerUserId="50963" LastEditorUserId="50963" LastEditDate="2018-04-26T18:30:27.230" LastActivityDate="2018-04-26T18:30:27.230" Title="Generating rules from data for classification" Tags="|machine-learning|data-mining|dataset|" AnswerCount="1" CommentCount="0" ClosedDate="2018-04-23T15:53:57.310" ContentLicense="CC BY-SA 3.0" />
  <row Id="30665" PostTypeId="1" AcceptedAnswerId="30690" CreationDate="2018-04-23T06:46:06.630" Score="2" ViewCount="1513" Body="&lt;p&gt;According to Wikipedia, &quot;the distance matrix of size $\frac{(n^2-n)}{2}$ can be materialized to avoid distance recomputations, but this needs $O(n^2)$ memory, whereas a non-matrix based implementation of DBSCAN only needs $O(n)$ memory.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{(n^2-n)}{2}$ is basically the triangular matrix. However, it says that a non-matrix based implementation only requires $O(n)$ memory. How does that work? Regardless of what data structure you use, don't you always have to have $\frac{(n^2-n)}{2}$ distance values? It would still be $O(n^2)$ space complexity, no? Is there something I'm missing here? I'm working with a huge dataset and I would really like to cut down on memory usage.&lt;/p&gt;&#xA;" OwnerUserId="49501" LastEditorUserId="26562" LastEditDate="2018-04-23T09:08:58.720" LastActivityDate="2018-04-24T22:54:04.587" Title="DBSCAN - Space complexity of O(n)?" Tags="|clustering|scalability|dbscan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30666" PostTypeId="2" ParentId="30662" CreationDate="2018-04-23T06:47:54.790" Score="1" Body="&lt;p&gt;I tried this experiment and was able to get some positive results. I will describe what I tried then perhaps you can specify where the differences may lie and we can further explore them. From what I tried I would assume that you are simply not training long enough.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Creating the data&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;n = 100000&#xA;x_train = np.zeros((n,2))&#xA;y_train = np.zeros((n,))&#xA;for i in range(n):&#xA;    x_train[i,0] = np.random.choice([-1,1])&#xA;    x_train[i,1] = np.random.choice([-1,1])&#xA;    if x_train[i,0] == 1 and x_train[i,1] == 1 or x_train[i,0] == -1 and x_train[i,1] == -1:&#xA;        y_train[i] = -1&#xA;    else:&#xA;        y_train[i] = 1&#xA;&#xA;x_train = x_train.reshape(n, 2,)&#xA;&#xA;n = 1000&#xA;x_test = np.zeros((n,2))&#xA;y_test = np.zeros((n,))&#xA;for i in range(n):&#xA;    x_test[i,0] = np.random.choice([-1,1])&#xA;    x_test[i,1] = np.random.choice([-1,1])&#xA;    if x_test[i,0] == 1 and x_test[i,1] == 1 or x_test[i,0] == -1 and x_test[i,1] == -1:&#xA;        y_test[i] = -1&#xA;    else:&#xA;        y_test[i] = 1&#xA;&#xA;x_test = x_test.reshape(n, 2,)&#xA;&#xA;print(x_test[0].T) &#xA;print(y_test[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[ 1.  1.] &lt;br/&gt;&#xA;  -1.0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h1&gt;Build the model&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;As you describe the model is 2 input nodes, 2 hidden nodes and 1 output node. Every node is using tanh as its activation function.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_shape = (2,)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(2, activation='tanh',&#xA;                 input_shape=input_shape))&#xA;model.add(Dense(2, activation='tanh'))&#xA;model.add(Dense(1, activation='tanh'))&#xA;&#xA;model.compile(loss=keras.losses.mean_squared_error,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Train the model&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Because I generated many instances of the data I am only training for 10 epochs. However, if your input space is only the four possible inputs you may want thousands of epochs. Neural networks do take a long time to converge.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs = 10&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;history = model.fit(x_train, y_train,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Epoch 10/10 &lt;br/&gt;100000/100000 [==============================] - 1s&#xA;  9us/step - loss: 9.3983e-05 - acc: 1.0000 - val_loss: 7.9096e-05 -&#xA;  val_acc: 1.0000&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ukswR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ukswR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-23T06:47:54.790" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30668" PostTypeId="2" ParentId="30644" CreationDate="2018-04-23T07:06:08.590" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I would like to know, is there any procedures or rules, that needs to be considered before formulating an MDP for a problem&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For a simple problem like this, I would do the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;List the states.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You have missed a state - the terminal state of leaving the building&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Mark terminal states&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In this case, only exiting the building is a terminal state. The hungry tiger doesn't actually do anything to Bunny, the problem states that the MDP should just continue.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;List the actions possible in each state.&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In your starting diagram, you do not show actions, and this is already limiting your ability to express the MDP.  &lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;List the possible transitions and probabilities of them&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For most action choices in the example, there are two possible outcomes from each action. The problem doesn't actually describe what happens when there is only one door, for the room with the tiger, but it seems reasonably safe to say that it is not possible for Bunny to get confused when there is only one door, so exiting the room with the tiger is one exception.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Assign rewards. Rewards should relate to the goals stated in the problem definition. Negative rewards for things to avoid happening (or in general to avoid time wasting). Positive rewards for reaching desired targets.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In your case you have two goals - avoid tiger and exit building. There does not appear to be any time constraint.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Typically you would show action choices as labelled arrows going from states to an action node (usually just a small solid circle) where the state transitions then take effect, with further arrows labelled with probabilities going to states. So State A should have two arrows going out &quot;choose left door&quot; and &quot;choose right door&quot;, then these choices will branch again - &quot;choose left door&quot; will go to state B with p=0.9 and state C with p=0.1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have some free choices over where to put reward labels, and there are a few different standard approaches. In this case, I would put the non-zero rewards on the state transitions that lead to goal-affecting states. E.g. maybe a +1 reward on the state transition from B taking the exit door and ending up outside. Note that for both the exit and the room with the tiger, that there are two possible ways to get to them due to the random transitions (thanks to Bunny getting confused).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note there are no actions, and no possibility of different transitions, where Bunny starts in state C and ends up in state C on the next timestep. I think you are confusing state C as being a terminal state - although this is understandable as some encounters with hungry tigers in the real world could well be terminal, the problem clearly states that everything just carries on for Bunny if he encounters the tiger. It is just something he wants to avoid. If he ends up in state C, then he still has an action he can take to go through the door. When he takes this action, he cannot &quot;get confused&quot; and pick the wrong door, so should always end up in state B with p=1.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you decide to draw a transition diagram, you have free choice of layout. There is no fixed approach for this, and for many MDPs it may not be possible to draw a nice 2D diagram. However, in your case, it is definitely possible to layout the states in a simple 2D arrangement. A good place to start would be to place the states in locations that correspond to the imaginary room map. Leave plenty of space initally, as the states do not link directly (as you have started), but link via action nodes which themselves branch to the next state.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this enough guidance for you to complete the problem by yourself. Solving the remainder on your own will be much better for your learning than turning it into a worked example here.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-04-23T08:27:27.900" LastActivityDate="2018-04-23T08:27:27.900" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30669" PostTypeId="2" ParentId="30664" CreationDate="2018-04-23T07:21:05.787" Score="3" Body="&lt;p&gt;If the data is unlabeled you either&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;manually label the data (if you have not too many samples)&lt;/li&gt;&#xA;&lt;li&gt;manually label part of the data, train a classification model on it (and validate your model!) and use the model to label the unlabeled part, to speed up the labeling process&lt;/li&gt;&#xA;&lt;li&gt;use pretrained classification/segmentation models (if your data is images)&lt;/li&gt;&#xA;&lt;li&gt;use unsupervised learning algorithms to find patterns in the data, then manually inspect the groups obtained to find recognizable patterns and label the groups accordingly. The algorithm you should use depends on the type of features you have. With numerical features you can use distance-based clustering techniques like k-means, or dimensionality reduction techniques like Self-Organizing Maps or t-SNE&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Basically, it depends on what kind of data you have and how many samples. I believe that more information on your data is required for providing more specific answers.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2018-04-23T12:14:17.457" LastActivityDate="2018-04-23T12:14:17.457" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30670" PostTypeId="2" ParentId="30623" CreationDate="2018-04-23T07:51:42.010" Score="1" Body="&lt;p&gt;I would not recommend doing this in general as it will likely lead to overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While a particular reordering might improve fit on your dataset, the goal is to create a model with good predictive power. If you received never before seen data from an independent second, third, one hundredth Titanic, would the ordering still be the same? I doubt it. You'd probably be incorrectly using the somewhat arbitrary ordering fixed on the first, real Titanic on new data where the ordering is no longer the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the story goes that the young and the elderly got preferential treatment and this is why their survival rate was higher. Therefore your AgeBand curve should be approximately U-shaped. (Which it is, approximately.) If you insist on &lt;em&gt;linear&lt;/em&gt; regression, I don't see how you could ever (directly) include this nonlinear (U-shaped) effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The fewer assumptions (particularities) go into the choice of features, the better. It would then make more sense to me to reorder such that you pick bands alternately from the two ends (starting from the middle): &lt;code&gt;3, 4, 2, 5, 1, 6, 0, 7&lt;/code&gt; (or in reverse order, starting from the ends). Or, even better, pool the age bands into coarser groups: &lt;code&gt;{1,2,3}, {4,5,6,7}, {0}&lt;/code&gt;. I think these are justified by your exploratory data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I look at the range of values in the AgeBand plot ($0.33$ to $0.55$), this variable does not actually explain as much as Pclass, Sex, FareBand. In the end you might end up dropping it from your variables anyway, so once again, the more attention you pay to it, the more you're at risk of overfitting.&lt;/p&gt;&#xA;" OwnerUserId="47856" LastActivityDate="2018-04-23T07:51:42.010" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30671" PostTypeId="2" ParentId="30659" CreationDate="2018-04-23T08:10:01.670" Score="11" Body="&lt;p&gt;I think that the main consequences are the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Computation time: If you freeze all the layers but the last 5 ones, you only need to backpropagate the gradient and update the weights of the last 5 layers. In contrast to backpropagating and updating the weights all the layers of the network, this means a huge decrease in computation time. For this reason, if you unfreeze all the network, this will allow you to see the data fewer epochs than if you were to update only the last layers weights'.&lt;/li&gt;&#xA;&lt;li&gt;Accuracy: Of course, by not updating the weights of most of the network your are only optimizing in a subset of the feature space. If your dataset is similar to any subset of the imagenet dataset, this should not matter a lot, but, if it is very different from imagenet, then freezing will mean a decrease in accuracy. If you have enough computation time, unfreezing everything will allow you to optimize in the whole feature space, allowing you to find better optima. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To wrap up, I think that the main point is to check if your images are comparable to the ones in imagenet. In this case, I would not unfreeze many layers. Otherwise, unfreeze everything but get ready to wait for a long training time.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-23T08:10:01.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30672" PostTypeId="1" CreationDate="2018-04-23T08:20:33.670" Score="3" ViewCount="386" Body="&lt;p&gt;I want to build a model which is able to detect the presence of a specific object in the image without caring about the location (i.e. no bounding boxes, just 0/1 output). More specifically, the object is always the same and it is characterized by a specific pattern (e.g. a national flag or a bar code). The algorithm should be robust to scale, rotation and distortion.&#xA;What is the best architecture in this case? Is there something better than a generic CNN binary classifier?&lt;/p&gt;&#xA;" OwnerUserId="45420" LastEditorUserId="45420" LastEditDate="2018-04-23T12:20:51.357" LastActivityDate="2018-04-23T15:22:20.837" Title="Best neural network architecture for object detection without location" Tags="|neural-network|object-detection|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30673" PostTypeId="2" ParentId="22853" CreationDate="2018-04-23T08:30:03.320" Score="20" Body="&lt;p&gt;Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition for this critical point to be a minimum is that the Hessian matrix is positive definite. As the Hessian is a symmetric matrix, we can diagonalize it. If we write the diagonal matrix corresponding to the Hessian as:&#xA;$$&#xA;  D =&#xA;  \begin{bmatrix}&#xA;    d_{1} &amp;amp; &amp;amp; \\&#xA;    &amp;amp; \ddots &amp;amp; \\&#xA;    &amp;amp; &amp;amp; d_{n}&#xA;  \end{bmatrix}&#xA;$$&#xA;the Hessian being positive definite is equivalent to $d_1 &amp;gt; 0, \dots, d_n&amp;gt;0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now let's think about deep learning cost functions. Deep learning cost functions depend on lots of parameters in a very complicated manner, so the Hessian will have a complicated expression itself. For this reason, we can think that the values of $d_1,\dots,d_n$ are not biased towards negative or positive values. For this reason, given any critical point, the probability of every value $d_i$ to be positive can be assumed to be $1/2$. Moreover, it is reasonable to assume that the values of $d_i$ do not depend easily on the values of $d_j$, due to the high non-linearity of the Hessian matrix, so we will take the probabilities of them being positive as independent events.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, given a critical point, the probability of it being a minimum is:&#xA;$$&#xA;P(d_1 &amp;gt; 0, \dots, d_n &amp;gt; 0) = P(d_1 &amp;gt; 0)\cdot \cdots \cdot P(d_n &amp;gt; 0) = \frac{1}{2^n}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The probability of any critical point being a minimum decreases exponentially with the dimension of the input space. In deep learning,&#xA;this space can range from 1000 to $10^8$, and in both cases $1/2^n$ is ridiculously small. Now we are convinced that, given any critical point that we come across, it is &lt;em&gt;very&lt;/em&gt; unlikely that it is a minimum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;But what about maxima?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maxima of a function are the minima of minus the function. For this reason, all the arguments used previously can be used to minus the cost function and we conclude that every critical point has probability of $1/2  ^n$ to be a maximum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, given a critical point, the probability of it being a saddle point is &#xA;$$P(saddle) = 1 - P(maximum) - P(minimum) = 1 - \frac{1}{2^n} - \frac{1}{2^n} = 1 - \frac{1}{2^{n-1}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is very close to 1 if $n$ is large enough (which typically is in deep learning).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-07-28T11:38:28.080" LastActivityDate="2018-07-28T11:38:28.080" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30674" PostTypeId="2" ParentId="30672" CreationDate="2018-04-23T09:14:08.507" Score="2" Body="&lt;p&gt;May not really answer you question, but to be honest, if you just want to find one specific object that you know beforehand, I wouldn't use a neural network at all (in my opinion you should use the right tool for a task).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For easier tasks like this, you can use more &quot;old school&quot; methods (boo) like &lt;a href=&quot;https://en.wikipedia.org/wiki/Template_matching&quot; rel=&quot;nofollow noreferrer&quot;&gt;template matching&lt;/a&gt;. With such approaches you just take an isolated image of your object and, for example, extract a set of features (e.g. using opencv) and then move a rectangle over images where you want to find that object and just compare features. The advantage of such approaches is that you do not really need training data, like you would for training a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you really, really, really want to use a CNN, you could try to just paste an image of your object into thousands of random images and then take even more images to train a classifier that examines if the object is in the image or not.&lt;/p&gt;&#xA;" OwnerUserId="21086" LastActivityDate="2018-04-23T09:14:08.507" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30676" PostTypeId="1" AcceptedAnswerId="30679" CreationDate="2018-04-23T09:38:48.060" Score="32" ViewCount="13330" Body="&lt;p&gt;I try to understand role of derivative of sigmoid function in neural networks. &#xA;&lt;a href=&quot;https://i.stack.imgur.com/N8vbm.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/N8vbm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this derivative exactly? &#xA;&lt;a href=&quot;https://i.stack.imgur.com/inMoa.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/inMoa.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def sigmoid(x):&#xA;    return 1 / (1 + np.exp(-x))&#xA;&#xA;def derivative(x, step):&#xA;    return (sigmoid(x+step) - sigmoid(x)) / step&#xA;&#xA;x = np.linspace(-10, 10, 1000)&#xA;&#xA;y1 = sigmoid(x)&#xA;y2 = derivative(x, 0.0000000000001)&#xA;&#xA;plt.plot(x, y1, label='sigmoid')&#xA;plt.plot(x, y2, label='derivative')&#xA;plt.legend(loc='upper left')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49067" LastEditorUserId="29587" LastEditDate="2018-04-23T09:40:18.677" LastActivityDate="2020-08-20T20:06:52.197" Title="Role derivative of sigmoid function in neural networks" Tags="|machine-learning|neural-network|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30678" PostTypeId="2" ParentId="30676" CreationDate="2018-04-23T09:58:11.243" Score="0" Body="&lt;p&gt;The derivative you see here is  &lt;strong&gt;important&lt;/strong&gt; in neural networks. It's the reason why people generally prefer something else such as &lt;strong&gt;rectified linear unit&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Do you see the derivative drop for the two ends? What if your network is on the very left side, but it needs to move to the right side? Imagine you're on -10.0 but you want 10.0. The gradient will be too small for your network to converge quickly. We don't want to wait, we want quicker convergence. RLU doesn't have this problem.&lt;/p&gt;&#xA;&lt;p&gt;We call this problem &amp;quot;&lt;strong&gt;Neural Network Saturation&lt;/strong&gt;&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;Please see &lt;a href=&quot;https://www.quora.com/What-is-special-about-rectifier-neural-units-used-in-NN-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="98307" LastEditDate="2020-08-20T20:06:52.197" LastActivityDate="2020-08-20T20:06:52.197" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30679" PostTypeId="2" ParentId="30676" CreationDate="2018-04-23T09:59:51.650" Score="34" Body="&lt;p&gt;The use of derivatives in neural networks is for the training process called &lt;strong&gt;backpropagation&lt;/strong&gt;. This technique uses &lt;strong&gt;gradient descent&lt;/strong&gt; in order to find an optimal set of model parameters in order to minimize a loss function. In your example you must use the &lt;strong&gt;derivative of a sigmoid&lt;/strong&gt; because that is the activation that your individual neurons are using.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;The loss function&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The essence of machine learning is to optimize a cost function such that we can either minimize or maximize some target function. This is typically called the loss or cost funtion. We typically want to minimize this function. The cost function, &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt;, associates some penalty based on the resulting errors when passing data through your model as a function of the model parameters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the square of the difference between the predicted label and the ground truth label for the &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt; images that we trained over. We will want to minimize this in some way.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Minimizing a loss function&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is &quot;how can we minimize a function&quot;? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's minimize the following function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$y = x^2-4x+6$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we plot this we can see that there is a minimum at &lt;span class=&quot;math-container&quot;&gt;$x = 2$&lt;/span&gt;. To do this analytically we can take the derivative of this function as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{dy}{dx} = 2x - 4 = 0$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x = 2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is &lt;strong&gt;gradient descent&lt;/strong&gt;. This is the technique used by neural networks.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Gradient Descent&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take our function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$y = x^2-4x+6$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we will determine the &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; for which &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; is minimized. Gradient descent algorithm first says we will pick a random value for &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;. Let us initialize at &lt;span class=&quot;math-container&quot;&gt;$x=8$&lt;/span&gt;. Then the algorithm will do the following iteratively until we reach convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = x^{old} - \nu \frac{dy}{dx}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$\nu$&lt;/span&gt; is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too small we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain. &lt;strong&gt;The derivative is the means by which we travel down this slope towards our minimum.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{dy}{dx} = 2x - 4$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\nu = 0.1$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Iteration 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $&lt;/span&gt; &lt;br/&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $&lt;/span&gt; &lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And we see that the algorithm converges at &lt;span class=&quot;math-container&quot;&gt;$x = 2$&lt;/span&gt;! We have found the minimum.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Applied to neural networks&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The first neural networks only had a single neuron which took in some inputs &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and then provide an output &lt;span class=&quot;math-container&quot;&gt;$\hat{y}$&lt;/span&gt;. A common function used is the sigmoid function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\sigma(z) = \frac{1}{1+exp(z)}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\hat{y}(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt; is the associated weight for each input &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and we have a bias &lt;span class=&quot;math-container&quot;&gt;$b$&lt;/span&gt;. We then want to minimize our cost function &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;How to train the neural network?&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; and train on batches of data of size &lt;span class=&quot;math-container&quot;&gt;$N$&lt;/span&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\hat{y}$&lt;/span&gt; is the predicted class obtained from the sigmoid function and &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights &lt;span class=&quot;math-container&quot;&gt;$w$&lt;/span&gt;. To make life easier we will split the derivative as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we have that &lt;span class=&quot;math-container&quot;&gt;$\hat{y} = \sigma(w^Tx)$&lt;/span&gt; and the derivative of the sigmoid function is &lt;span class=&quot;math-container&quot;&gt;$\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$&lt;/span&gt; thus we have,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we can then update the weights through gradient descent as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$\eta$&lt;/span&gt; is the learning rate.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="-1" LastEditDate="2020-03-28T07:19:20.220" LastActivityDate="2020-03-28T07:19:20.220" CommentCount="16" ContentLicense="CC BY-SA 4.0" />
  <row Id="30680" PostTypeId="2" ParentId="30676" CreationDate="2018-04-23T09:59:59.160" Score="3" Body="&lt;p&gt;During the phase where the neural network generates its prediction, it feeds the input forward through the network. For each layer, the layer's input $X$ goes first through an affine transformation $W \cdot X + b$ and then is passed through the sigmoid function $σ(W \cdot X + b)$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to train the network, the output $\hat y$ is then compared to the expected output (or label) $y$ through a &lt;strong&gt;cost function&lt;/strong&gt; $L(y, \hat y)=L\left(y, σ(W \cdot X + b)\right)$. The goal of the whole training procedure is to &lt;strong&gt;minimize&lt;/strong&gt; that cost function. In order to do that, a technique called &lt;strong&gt;gradient descent&lt;/strong&gt; is performed which calculates how we should change $W$ and $b$ so that the cost reduces.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_descent&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient Descent&lt;/a&gt; requires calculating the derivative of the cost function w.r.t $W$ and $b$. In order to do that we must apply the &lt;a href=&quot;https://en.wikipedia.org/wiki/Chain_rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;chain rule&lt;/a&gt;, because the derivative we need to calculate is a composition of two functions. As dictated by the chain rule we must calculate the &lt;strong&gt;derivative of the sigmoid function&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the reasons that the sigmoid function is popular with neural networks, is because its derivative is &lt;strong&gt;easy to compute&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-23T09:59:59.160" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30681" PostTypeId="1" AcceptedAnswerId="30718" CreationDate="2018-04-23T11:27:52.800" Score="1" ViewCount="733" Body="&lt;p&gt;Is there a package or function in NLP which can be used to convert the word into an one hot vector.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-23T22:12:51.420" Title="Package/function to convert the word to one hot vector in Python NLP" Tags="|machine-learning|python|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30682" PostTypeId="1" AcceptedAnswerId="30687" CreationDate="2018-04-23T11:38:56.540" Score="3" ViewCount="714" Body="&lt;p&gt;I am solving for a &lt;strong&gt;&lt;code&gt;regression&lt;/code&gt;&lt;/strong&gt; use case using tensorflow's &lt;code&gt;DNNRegressor&lt;/code&gt;. For EDA purpose, I referred to this &lt;a href=&quot;https://machinelearningmastery.com/quick-and-dirty-data-analysis-with-pandas/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; and used &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.boxplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas boxplot&lt;/a&gt; to plot my numerical predictors and target variable(here, &lt;em&gt;pid demand&lt;/em&gt;) and &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/visualization.html#scatter-matrix-plot&quot; rel=&quot;nofollow noreferrer&quot;&gt;scatter_matrix&lt;/a&gt; for plotting the distributions and here are the results :&#xA;&lt;a href=&quot;https://i.stack.imgur.com/t6osb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;predictor_target_boxplot&lt;/a&gt; ; &lt;a href=&quot;https://i.stack.imgur.com/y7lwm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;features_label_pdf_scatter_matrix&lt;/a&gt;&#xA;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need help in interpreting these two plots, specifically on these fronts:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How come the boxplot shows so many points beyond whiskers (~10%), can there be so many outliers in a dataset?&lt;/li&gt;&#xA;&lt;li&gt;How do I handle those outliers?&lt;/li&gt;&#xA;&lt;li&gt;Based on the second plot (feature, label pdf), should I normalize my features to exhibit Gaussian distribution? If so, why? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50852" LastActivityDate="2018-04-23T13:11:11.263" Title="What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?" Tags="|machine-learning|python|predictive-modeling|regression|data-cleaning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30683" PostTypeId="1" CreationDate="2018-04-23T11:49:16.773" Score="1" ViewCount="1435" Body="&lt;p&gt;I have a pdf file that contains information . I would like to extract few key terms/phrase along with a value for example (&lt;code&gt;current balance : CHF (swiss francs) 1,000&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can convert pdf file to text using pdfminer . But how i can extract the above keyword using tensorflow text classification or other methods. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest how i can start with it? I haven't come across a single example with tensorflow. There is a question here (&lt;a href=&quot;https://datascience.stackexchange.com/questions/10077/keyword-phrase-extraction-from-text-using-deep-learning-libraries&quot;&gt;Keyword/phrase extraction from Text using Deep Learning libraries&lt;/a&gt;&#xA;). but this doesn't help me to understand much as I am a beginner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I used NLTK to tokenize words and read no of words by using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;text = re.sub(r'[_&quot;\-;%()|+&amp;amp;=*.!?:#$@\[\]/]', ' ', f)&#xA;from nltk.tokenize import word_tokenize&#xA;from nltk.corpus import stopwords&#xA;&#xA;tokens = word_tokenize(text)&#xA;stop_words = set(stopwords.words('english'))&#xA;words = [w for w in tokens if not w in stop_words]&#xA;freq = nltk.FreqDist(words) &#xA;for key,val in freq.items():&#xA;    frq = str(key) + ':' + str(val)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So i have this text that i could like to extract the key phrase &lt;code&gt;current balance : CHF (swiss francs) 1,000&lt;/code&gt; along with the value using Neural Network. How can i do that? I came across many posts but couldnt get what i need. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS: every time i upload a new file, i want my algorithm to find out the same keyword and use the value next to it.&lt;/p&gt;&#xA;" OwnerUserId="32544" LastActivityDate="2018-04-23T11:49:16.773" Title="Keyword Extraction from a text followed by a key value using tensorflow" Tags="|python|neural-network|data-mining|tensorflow|text-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30684" PostTypeId="1" CreationDate="2018-04-23T11:50:14.250" Score="3" ViewCount="3575" Body="&lt;p&gt;I am trying to solve the San Francisco Crime Problem on Kaggle. To begin with, here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;from sklearn import preprocessing&#xA;from pandas import read_csv&#xA;from sklearn.model_selection import train_test_split&#xA;from sklearn.preprocessing import LabelEncoder&#xA;from sklearn.preprocessing import StandardScaler&#xA;from sklearn.decomposition import PCA&#xA;from sklearn.ensemble import RandomForestClassifier&#xA;from sklearn.metrics import log_loss&#xA;&#xA;data = pd.read_csv('/home/limafoxtrottango/Downloads/sf_crime_train.csv')&#xA;data_test = pd.read_csv('/home/limafoxtrottango/Downloads/Mozilla Downloads/Test.csv')&#xA;&#xA;#pre-processing training data&#xA;le = LabelEncoder()&#xA;data['Category'] = le.fit_transform(data['Category'])&#xA;&#xA;categorical_variables = pd.get_dummies(data[['DayOfWeek','PdDistrict','Resolution']])  &#xA;&#xA;data = data.join(categorical_variables)&#xA;&#xA;scaler1 = StandardScaler()&#xA;scaler2 = StandardScaler()&#xA;scaler3 = StandardScaler()&#xA;scaler4 = StandardScaler()&#xA;&#xA;data['X'] = scaler1.fit_transform(data[['X']])&#xA;data['Y'] = scaler2.fit_transform(data[['Y']])&#xA;&#xA;data['Date'], data['Time'] = data['Dates'].str.split(' ', 1).str&#xA;data['Year'], data['Month'], data['Day'] = data['Date'].str.split('-', 2).str&#xA;data['Hours'], data['Minutes'], data['Seconds'] = data['Time'].str.split(':', 2).str&#xA;data[&quot;Hours&quot;] = data[&quot;Hours&quot;].map(str) + data[&quot;Minutes&quot;]&#xA;&#xA;del data['Minutes']&#xA;del data['Seconds']&#xA;del data['Time']&#xA;del data['Day']&#xA;del data['Year']&#xA;del data['Id']&#xA;del data['Address']&#xA;del data['Descript']&#xA;del data['PdDistrict']&#xA;del data['Resolution']&#xA;del data['DayOfWeek']&#xA;del data['Date']&#xA;del data['Dates']&#xA;&#xA;data['Month'] = scaler1.fit_transform(data[['Month']])&#xA;data['Hours'] = scaler2.fit_transform(data[['Hours']])&#xA;&#xA;labels = data.columns[1:]&#xA;train = data.loc[:, labels].values&#xA;&#xA;#doing pca&#xA;pca = PCA(n_components=3)&#xA;principalComponents = pca.fit_transform(train)&#xA;principalDf = pd.DataFrame(data = principalComponents, columns = ['pc 1', 'pc 2', 'pc 3'])&#xA;&#xA;final_data_train = pd.concat([data[['Category']], principalDf], axis = 1)&#xA;&#xA;target = np.array(final_data_train['Category'])&#xA;features = final_data_train.drop('Category', axis = 1)&#xA;&#xA;# Saving feature names for later use&#xA;feature_list = list(final_data_train.columns)&#xA;&#xA;# Convert to numpy array&#xA;features = np.array(features)&#xA;&#xA;# Instantiate model with 10 decision trees&#xA;rf = RandomForestClassifier(n_estimators = 10, random_state = 0)&#xA;&#xA;# Train the model on training data&#xA;rf.fit(features, target);&#xA;&#xA;#pre-processing testing data&#xA;categorical_variables = pd.get_dummies(data_test[['DayOfWeek','PdDistrict','Resolution']])  &#xA;&#xA;data_test = data_test.join(categorical_variables)&#xA;&#xA;scaler5 = StandardScaler()&#xA;scaler6 = StandardScaler()&#xA;scaler7 = StandardScaler()&#xA;scaler8 = StandardScaler()&#xA;&#xA;data_test['X'] = scaler5.fit_transform(data_test[['X']])&#xA;data_test['Y'] = scaler6.fit_transform(data_test[['Y']])&#xA;&#xA;&#xA;data_test['Date'], data_test['Time'] = data_test['Dates'].str.split(' ', 1).str&#xA;data_test['Year'], data_test['Month'], data_test['Day'] = data_test['Date'].str.split('-', 2).str&#xA;data_test['Hours'], data_test['Minutes'], data_test['Seconds'] = data_test['Time'].str.split(':', 2).str&#xA;data_test[&quot;Hours&quot;] = data_test[&quot;Hours&quot;].map(str) + data_test[&quot;Minutes&quot;]&#xA;&#xA;row_id_test = data_test['Id']&#xA;&#xA;del data_test['Minutes']&#xA;del data_test['Seconds']&#xA;del data_test['Time']&#xA;del data_test['Day']&#xA;del data_test['Year']&#xA;del data_test['Address']&#xA;del data_test['PdDistrict']&#xA;del data_test['Resolution']&#xA;del data_test['DayOfWeek']&#xA;del data_test['Date']&#xA;del data_test['Dates']&#xA;&#xA;data_test['Month'] = scaler7.fit_transform(data_test[['Month']])&#xA;data_test['Hours'] = scaler8.fit_transform(data_test[['Hours']])&#xA;&#xA;data_test_trans = pca.transform(data_test)&#xA;&#xA;predictions = rf.predict_proba(data_test_trans)&#xA;&#xA;final_predictions_file = pd.concat([row_id_test, pd.DataFrame(predictions)], axis = 1)&#xA;&#xA;np.savetxt(&quot;predictions.csv&quot;, final_predictions_file, delimiter=&quot;,&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have done PCA on my training data, and using that object to transform the testing data using: pca.transform(). But I am getting the following error:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;ValueError: operands could not be broadcast together with shapes (60002,39) (38,) during pca.transform&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please point out what I am doing wrong? When I use the training data itself as testing data, and do not remove the target variable from the dataframe, the program runs just fine. When I use the actual testing data, that does not have any target column, this error is thrown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have just started with ML, so please excuse if this question appears a bit too naive. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="50991" LastActivityDate="2019-05-04T18:23:26.533" Title="ValueError: operands could not be broadcast together with shapes (60002,39) (38,) during pca.transform" Tags="|machine-learning|python|scikit-learn|random-forest|pca|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="30686" PostTypeId="1" CreationDate="2018-04-23T12:42:39.437" Score="2" ViewCount="463" Body="&lt;p&gt;I've been combing through this code for a week now trying to figure out why my cost function is increasing as in the following image. Reducing the learning rate does help but very little. Can anyone spot why the cost function isn't working as expected?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I realise a CNN would be preferable, but I still want to understand why this simple network is failing.&#xA;Please help:)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1zRai.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1zRai.png&quot; alt=&quot;Runaway Cost Function&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;&#xA;import matplotlib.pyplot as plt&#xA;&#xA;mnist = input_data.read_data_sets(&quot;MNIST_DATA/&quot;,one_hot=True)&#xA;&#xA;def createPlaceholders():&#xA;    xph = tf.placeholder(tf.float32, (784, None))&#xA;    yph = tf.placeholder(tf.float32, (10, None))&#xA;    return xph, yph&#xA;&#xA;def init_param(layers_dim):&#xA;    weights = {}&#xA;    L = len(layers_dim)&#xA;&#xA;    for l in range(1,L):&#xA;        weights['W' + str(l)] = tf.get_variable('W' + str(l), shape=(layers_dim[l],layers_dim[l-1]), initializer= tf.contrib.layers.xavier_initializer())&#xA;        weights['b' + str(l)] = tf.get_variable('b' + str(l), shape=(layers_dim[l],1), initializer= tf.zeros_initializer())&#xA;&#xA;    return weights&#xA;&#xA;def forward_prop(X,L,weights):&#xA;    parameters = {}&#xA;    parameters['A0'] = tf.cast(X,tf.float32)&#xA;&#xA;    for l in range(1,L-1):&#xA;        parameters['Z' + str(l)] = tf.add(tf.matmul(weights['W' + str(l)], parameters['A' + str(l-1)]), weights['b' + str(l)])&#xA;        parameters['A' + str(l)] = tf.nn.relu(parameters['Z' + str(l)])&#xA;&#xA;    parameters['Z' + str(L-1)] = tf.add(tf.matmul(weights['W' + str(L-1)], parameters['A' + str(L-2)]), weights['b' + str(L-1)])&#xA;    return parameters['Z' + str(L-1)]&#xA;&#xA;def compute_cost(ZL,Y):&#xA;    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = tf.cast(Y,tf.float32), logits = ZL))&#xA;    return cost&#xA;&#xA;def randomMiniBatches(X,Y,minibatch_size):&#xA;    m = X.shape[1]&#xA;    shuffle = np.random.permutation(m)&#xA;    temp_X = X[:,shuffle]&#xA;    temp_Y = Y[:,shuffle]&#xA;&#xA;    num_complete_minibatches = int(np.floor(m/minibatch_size))&#xA;&#xA;    mini_batches = []&#xA;&#xA;    for batch in range(num_complete_minibatches):&#xA;        mini_batches.append((temp_X[:,batch*minibatch_size: (batch+1)*minibatch_size], temp_Y[:,batch*minibatch_size: (batch+1)*minibatch_size]))&#xA;&#xA;    mini_batches.append((temp_X[:,num_complete_minibatches*minibatch_size:], temp_Y[:,num_complete_minibatches*minibatch_size:]))&#xA;&#xA;    return mini_batches&#xA;&#xA;def model(X, Y, layers_dim, learning_rate = 0.001, num_epochs = 20, minibatch_size = 64):&#xA;    tf.reset_default_graph()&#xA;    costs = []&#xA;&#xA;    xph, yph = createPlaceholders()&#xA;    weights = init_param(layers_dim)&#xA;    ZL = forward_prop(xph, len(layers_dim), weights)&#xA;    cost = compute_cost(ZL,yph)&#xA;    optimiser = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)&#xA;&#xA;    init = tf.global_variables_initializer()&#xA;&#xA;    with tf.Session() as sess:&#xA;        sess.run(init)&#xA;&#xA;        for epoch in range(num_epochs):&#xA;            minibatches = randomMiniBatches(X,Y,minibatch_size)&#xA;            epoch_cost = 0&#xA;&#xA;            for b, mini in enumerate(minibatches,1):&#xA;                mini_x, mini_y = mini&#xA;                _,c = sess.run([optimiser,cost],feed_dict={xph:mini_x,yph:mini_y})&#xA;                epoch_cost += c&#xA;            print('epoch: ',epoch+1,'/ ',num_epochs)&#xA;&#xA;            epoch_cost /= len(minibatches)&#xA;            costs.append(epoch_cost)&#xA;&#xA;    plt.plot(costs) &#xA;    print(costs)&#xA;&#xA;&#xA;&#xA;X_train = mnist.train.images.T&#xA;n_x = X_train.shape[0]&#xA;Y_train = mnist.train.labels.T&#xA;n_y = Y_train.shape[0]&#xA;layers_dim = [n_x,10,n_y]&#xA;&#xA;model(X_train, Y_train, layers_dim)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50992" LastActivityDate="2024-01-22T15:08:11.160" Title="MNIST - Vanilla Neural Network - Why Cost Function is Increasing?" Tags="|neural-network|tensorflow|mnist|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30687" PostTypeId="2" ParentId="30682" CreationDate="2018-04-23T12:49:47.837" Score="0" Body="&lt;p&gt;This comments aren't all mine, I have asked on a slack forum,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;boxplot is shouting at you:  skewness, and also high dispersion.  Can not ask much more to a boxplot than location, dispersion and skewness..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also checkout this term heteroscedasticity(completely suits your case)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try switching the transformation to a log plot or lower..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also your eda can't be boxplots dependent as prices are involved here..one of the remedy would be doing box-cox transformation&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.differencebetween.com/difference-between-dispersion-and-vs-skewness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.differencebetween.com/difference-between-dispersion-and-vs-skewness/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.statsmakemecry.com/smmctheblog/confusing-stats-terms-explained-heteroscedasticity-heteroske.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.statsmakemecry.com/smmctheblog/confusing-stats-terms-explained-heteroscedasticity-heteroske.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To have a look at it,&#xA;&lt;a href=&quot;https://datascienceplus.com/how-to-detect-heteroscedasticity-and-rectify-it/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://datascienceplus.com/how-to-detect-heteroscedasticity-and-rectify-it/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-23T13:11:11.263" LastActivityDate="2018-04-23T13:11:11.263" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30688" PostTypeId="2" ParentId="27822" CreationDate="2018-04-23T13:00:00.600" Score="1" Body="&lt;p&gt;This worked for me when I changed the declaration of the function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def roc_auc_score(y_true, y_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;instead:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def roc_auc_score(y_pred, y_true)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I defined it wrong and it worked, when I fixed it I got the same error as you do. I don't really understand your code logic, but you can if this works for you.&lt;/p&gt;&#xA;" OwnerUserId="50994" LastEditorUserId="29575" LastEditDate="2018-04-23T14:08:08.233" LastActivityDate="2018-04-23T14:08:08.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30689" PostTypeId="1" AcceptedAnswerId="38139" CreationDate="2018-04-23T13:29:03.863" Score="6" ViewCount="5062" Body="&lt;p&gt;I like Python, and I like Spark, but they don't fit very well together.  In particular,&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;it is very hard to use python functions in spark (have to create JVM binding for function in python)&lt;/li&gt;&#xA;&lt;li&gt;it is hard to debug pyspark, with py4j in the middle&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;So I wonder if there are any alternatives to pyspark that supports python natively instead of via an adapter layer?&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/34412182/842860&quot;&gt;Reference&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="28423" LastEditorUserId="98307" LastEditDate="2020-08-11T08:18:51.347" LastActivityDate="2020-08-11T08:18:51.347" Title="What are the alternatives to Python + Spark (pyspark)?" Tags="|python|apache-spark|pyspark|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30690" PostTypeId="2" ParentId="30665" CreationDate="2018-04-23T13:29:29.240" Score="2" Body="&lt;p&gt;You can run DBSCAN without storing the distances in a matrix. This has the drawback that each time you visit a point, you have to recalculate all the relevant distances, which requires more time. However, the space complexity stays $O(n)$, since the only things you have in memory at any single time are the positions of the n points, their various labels, the neighbors of the current point and the neighbors of a particular neighbour in the case that the point turns out to be a core point.&lt;/p&gt;&#xA;" OwnerUserId="50993" LastEditorUserId="50993" LastEditDate="2018-04-24T22:54:04.587" LastActivityDate="2018-04-24T22:54:04.587" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30691" PostTypeId="1" AcceptedAnswerId="30694" CreationDate="2018-04-23T14:01:27.993" Score="0" ViewCount="37" Body="&lt;p&gt;I am tackling a RL problem (relaxed version of &lt;a href=&quot;https://www.youtube.com/watch?v=tmRITTg70VE&quot; rel=&quot;nofollow noreferrer&quot;&gt;Space Fortress&lt;/a&gt;) with DQN. The usual approach would be to feed pixels into a CNN but that is usually very slow. I am considering feeding symbolic features such as coordinates, velocity and angles of the relevant elements into a Multi Layer Perceptron because then the training would be faster and I could try more things quicker. My idea is that the perceptual part of the task is not that interesting to learn here and I would like to focus on the RL part instead (I am experimenting with Hierarchical RL).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given that with all the symbolic features that I would provide it would possible to reconstruct the full image and hence the state of the environment, would there be any reason that a CNN pixel based approach would be better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PT: The environment has a fixed number of elements&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="17484" LastActivityDate="2018-04-23T14:40:53.580" Title="Is there any reason why providing symbolic features into an MLP wouldn't outperform feeding raw pixels to a CNN in a RL task?" Tags="|neural-network|reinforcement-learning|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30692" PostTypeId="1" CreationDate="2018-04-23T14:32:02.853" Score="2" ViewCount="107" Body="&lt;p&gt;There exists a mapping from input image to out image. Say input image is a piece of paper with a square hole in the center, and output image is the shadow of the input image when light shines on the paper, which also looks like a square, but with some blurs and different in size. (The real pattern is far more complicated than this square hole) I'm trying to implement a neural network that can learning this kind of mapping, when given the input image, the network can predict the &quot;shadow&quot; of the input image. I've tried to use a convolutional autoencoder-like structure (like this &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://blog.keras.io/building-autoencoders-in-keras.html&lt;/a&gt;) by replace the target image from the input image it self to the output image training set. But it didn't looks work well. I would like to know what kind of neural network is suitable for this task? And what kind of loss  function is suitable?    &lt;/p&gt;&#xA;" OwnerUserId="29865" LastEditorUserId="29169" LastEditDate="2021-02-13T04:23:50.137" LastActivityDate="2021-02-13T04:23:50.137" Title="What kind of neural network structure is suitable for image to image learning?" Tags="|machine-learning|deep-learning|neural-network|image-classification|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30693" PostTypeId="1" CreationDate="2018-04-23T14:33:44.253" Score="4" ViewCount="5786" Body="&lt;p&gt;I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some examples in stackoverflow and other sources as well, however, I could not find a decent explanation about the interpretation of outliers returned by isolationforest. First, what I did is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.ensemble import IsolationForest&#xA;&#xA;X_train = trbb[check_cols]&#xA;&#xA;clf = IsolationForest(n_jobs=6,n_estimators=500, max_samples=256, random_state=23)&#xA;clf.fit(X_train)&#xA;y_pred_train = clf.predict(X_train)&#xA;y_pred_train&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This returns array([1, 1, 1, ..., 1, 1, 1]) where -1's are the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The shape of y_pred_train is 5000, which is identical with X_train[0]. So the index of -1 corresponds to the index of X_train. For exp. one of the outlier indices returned by IF is 532. So this means that the point (in 800 dimensional space) in this index is detected as an outlier. After detecting this, how can I approach and use the results to dig more? For example, can I reach the most important features causing the outliers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To this end, I applied PCA and used the newly created (50) components as new variables. Then, by plotting component pairs with -1 &amp;amp; 1s returned by IF, I tried to get some insight of possible outliers. Indeed, I observed some points as shown below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yDHIL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yDHIL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, the black dots are -1s (outliers) and the yellow ones are 1s (inliers). I interpret this that if both components have very high values, it might cause an outlier. However, how can I decide which original values are the most effective ones for those outliers? For example, can I use the original variables with the highest weights (let's pick top 10) in the related components? Or could you suggest any other method?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!   &lt;/p&gt;&#xA;" OwnerUserId="33335" LastEditorUserId="33335" LastEditDate="2018-04-23T14:43:02.567" LastActivityDate="2019-01-03T16:01:55.430" Title="Multivariate outlier detection with isolation forest..How to detect most effective features?" Tags="|python|scikit-learn|anomaly-detection|outlier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30694" PostTypeId="2" ParentId="30691" CreationDate="2018-04-23T14:40:53.580" Score="1" Body="&lt;p&gt;If the features are able to fully define the image, there would be no reason to use a pixel-based approach.  You are essentially performing lossless compression and (nearly) instantly acquiring a feature set, so you would see a dramatic performance difference between the CNN and MLP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you needed a quick solution, CNNs are great out-of-the-box algorithms, but if you are willing to put in the time to develop an MLP, it would provide a performance boost without losing accuracy.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-23T14:40:53.580" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30695" PostTypeId="1" CreationDate="2018-04-23T15:20:04.860" Score="4" ViewCount="510" Body="&lt;p&gt;I'm getting my feet wet with data science and machine learning. Please bear with me as I try to explain my problem. I haven't been able to find anything about this method, but I suspect there's a simple name for what I'm trying to do. To give an idea about my level of knowledge, I've had some basic statistics training in my university studies (social sciences), and I work as a programmer at the moment. So when it comes to statistics I'm not a complete layperson, but my understanding leaves a lot to be desired.&lt;/p&gt;&#xA;&lt;p&gt;I need to make predictions on a certain dataset. All features and the target are continuous variables (interval/ratio level) and I'm using a regression multi-layer perceptron with a single hidden layer. The &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt; is relatively low when fitting on the entire dataset, so I was hoping to improve the predictive power by doing cluster analysis and fitting multiple regressors on each cluster separately. This didn't work for me. The highest cluster &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt; was the same as for the entire set so I dropped that approach. I've tried multiple clustering algorithms and haven't noticed much of a difference among them; but, I haven't done an exhaustive search so I may have missed something.&lt;/p&gt;&#xA;&lt;p&gt;What I ended up doing is fitting the model on the training subset, identifying the data point for which the prediction error in the test subset was greatest, and removing that one data point. The train-test split is done randomly so the probability of some &amp;quot;bad&amp;quot; data points remaining by chance is pretty low. I repeated this process until I got an &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt; I'm satisfied with, after which I designated the remaining data points as belonging to e.g. &amp;quot;Group 1&amp;quot;. This whole process was then repeated on the entire dataset &lt;em&gt;minus&lt;/em&gt; Group 1, and ultimately all of the data should be divided into groups inside which reasonably reliable predictions can be made. To give an idea about the data: &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt; on the entire set of about 11000 data points hovers around 0.7. In Group 1, I kept 7000 data points for which I can get up to 0.9. The remaining groups also have acceptable (for my standards) &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt;s.&lt;/p&gt;&#xA;&lt;p&gt;After all of the data is split into groups in this way, I expect to be able to train a classifier on the features, to predict the group that a certain data point belongs to and use the appropriate regression model for that group to predict the final target variable. My question is, &lt;strong&gt;is there some methodological flaw in this approach&lt;/strong&gt;, specifically the removal of certain data points based on prediction errors? Am I introducing some artifacts into the data or something like that? As far as I can tell there's no information leakage about the target. What this &lt;em&gt;looks&lt;/em&gt; like to me is, some roundabout way of doing cluster analysis, as it seems to exclude &amp;quot;outliers&amp;quot; in a certain sense, but I couldn't be more precise and that may not be the case at all.&lt;/p&gt;&#xA;&lt;p&gt;Note: all mentions of &lt;span class=&quot;math-container&quot;&gt;$R^{\textrm{2}}$&lt;/span&gt; refer to &lt;em&gt;predictions&lt;/em&gt;, i.e. the score I get on the test set, not training.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; I removed data points only from the test subset (which changes randomly in each iteration), but now that I think about it I guess there's no reason to limit exclusions to the test subset as a worse prediction may happen in the training subset. I'll update when I try it. Also, I haven't yet tried fitting the classifier so I suppose I may not get any better results with this final model. Regardless of the results I get, I'm interested in the validity of this approach. Also, if someone knows of a theoretical limitation here, if there's a reason why I couldn't get better results in principle with this approach, I'd like to know.&lt;/p&gt;&#xA;" OwnerUserId="51000" LastEditorUserId="29169" LastEditDate="2021-07-07T18:12:41.683" LastActivityDate="2021-07-07T18:12:41.683" Title="Is removing poorly predicted data points a valid approach?" Tags="|machine-learning|predictive-modeling|regression|methodology|" AnswerCount="3" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="30696" PostTypeId="2" ParentId="30672" CreationDate="2018-04-23T15:22:20.837" Score="4" Body="&lt;p&gt;It mostly depends on the amount of data you have available for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, unless you have an extremely limited dataset and you are unable to generate synthetic data, &lt;strong&gt;CNNs are currently the best approach for solving your task&lt;/strong&gt;.  With sufficient data, they are very robust.  To improve their performance, you can augment your dataset by randomly rotating, flipping, brightening, contrasting, scaling, cropping, blurring, and adding noise to the images throughout training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also depends on your speed requirements and computational resources.  Traditional methods have the potential to be quite fast, but if you only care about accuracy, a deep network would probably perform better.  Note that some CNNs are also quite fast, such as &lt;a href=&quot;https://arxiv.org/pdf/1704.04861.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;MobileNets&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are &lt;strong&gt;a few reasons you might go &quot;old school&quot;&lt;/strong&gt; and use template matching and more traditional approaches as suggested by @TheDjentleman:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You do not have enough data to make a CNN robust AND you cannot generate synthetic data&lt;/li&gt;&#xA;&lt;li&gt;CNNs do not meet your speed requirements AND the traditional methods do&lt;/li&gt;&#xA;&lt;li&gt;You do not have sufficient resources to train a deep network, such as GPU(s).  FYI Google developed a &lt;a href=&quot;https://cloud.google.com/gpu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;cloud computing tool&lt;/a&gt; for researchers and developers at a pretty reasonable cost.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If none of those criteria fit your situation, go with a CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, you do not necessarily need a generic binary classification CNN.  You can use an object detection network or even a semantic segmentation network and just ignore the location data if you do not need it.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-23T15:22:20.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30697" PostTypeId="1" CreationDate="2018-04-23T15:29:29.560" Score="12" ViewCount="6647" Body="&lt;p&gt;I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I am sorry, question may have been a bit misleading. What I meant by &quot;unfamiliar environment&quot; is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?&lt;/p&gt;&#xA;" OwnerUserId="50956" LastEditorUserId="9257" LastEditDate="2018-04-29T07:22:43.543" LastActivityDate="2018-08-13T10:11:26.897" Title="Unsupervised image segmentation" Tags="|machine-learning|deep-learning|cnn|computer-vision|object-detection|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30698" PostTypeId="1" CreationDate="2018-04-23T15:32:17.810" Score="3" ViewCount="670" Body="&lt;p&gt;I'm working with some time-series data that is clearly non stationary. At first glance, it looks like there are 3 sections of my data. Within each section, the data have similar mean and variance, but between each section the variance of the data is quite different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a well known algorithm to detect and label different sections of stationarity in time-series data?&lt;/p&gt;&#xA;" OwnerUserId="51002" LastActivityDate="2018-05-07T18:24:21.660" Title="Detect different &quot;sections&quot; of stationarity in time-series data?" Tags="|python|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30699" PostTypeId="2" ParentId="30697" CreationDate="2018-04-23T15:37:13.867" Score="2" Body="&lt;p&gt;Actually, your task is supervised. &lt;a href=&quot;https://arxiv.org/abs/1511.00561&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Segnet&lt;/code&gt;&lt;/a&gt; can be good architecture for your purpose which one of its implementations can be accessed &lt;a href=&quot;https://github.com/alexgkendall/caffe-segnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;em&gt;SegNet learns to predict pixel-wise class labels from supervised learning. Therefore we require a dataset of input images with corresponding ground truth labels. Label images must be single channel, with each pixel labelled with its class &lt;a href=&quot;http://mi.eng.cam.ac.uk/projects/segnet/tutorial.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;...&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, take a look at &lt;a href=&quot;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;Fully Convolutional Networks&lt;/strong&gt;&lt;/a&gt; which are well suited for your task.&lt;/p&gt;&#xA;&#xA;&lt;hr /&gt;&#xA;&#xA;&lt;p&gt;Based on the edits in the question, I add extra information. There are numerous methods that can be applied for this task. Basically the easiest one is to use a &lt;em&gt;background&lt;/em&gt; label and classify those classes that you don't know as background by employing the mentioned architectures. By doing so you will have labels which can have overlap for background class which is a probable downside of this approach but its advantage is that in cases where your trained labels are frequently used in the inputs, you can have a relatively light version of architecture which recognizes the unknown classes. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-04-26T13:06:33.690" LastActivityDate="2018-04-26T13:06:33.690" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30700" PostTypeId="2" ParentId="26833" CreationDate="2018-04-23T16:06:42.227" Score="23" Body="&lt;p&gt;You can use any metric function that you specified when compiling the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have the following metric function:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def my_metric(y_true, y_pred):&#xA;     return some_metric_computation(y_true, y_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The only requirement to this function is that it takes accepts the true y and the predicted y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(metrics=['accuracy', my_metric], ...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;EarlyStopping(monitor='my_metric', mode='min')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Make sure to specify the mode (min if lower is better, max if higher is better).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.&lt;/p&gt;&#xA;" OwnerUserId="51003" LastEditorUserId="51003" LastEditDate="2018-04-23T16:32:35.563" LastActivityDate="2018-04-23T16:32:35.563" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30701" PostTypeId="1" CreationDate="2018-04-23T16:12:19.140" Score="2" ViewCount="27995" Body="&lt;p&gt;I am getting the error:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Using the model:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;X = np.reshape(x_train_tfidf.shape[0], 1, x_train_tfidf.shape[1])&#xA;print(X.shape)&#xA;model.add(LSTM(30, return_sequences=True,&#xA;               input_shape=X))&#xA;&#xA;model.add(Flatten())&#xA;model.add(Dropout(0.25))&#xA;model.add(Dense(100,activation = 'relu'))&#xA;model.add(Dropout(0.25))&#xA;model.add(Dense(1,activation='sigmoid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50935" LastEditorUserId="29169" LastEditDate="2020-12-15T18:11:42.313" LastActivityDate="2020-12-17T15:52:20.980" Title="Error: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2" Tags="|machine-learning|python|keras|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="30702" PostTypeId="1" AcceptedAnswerId="30731" CreationDate="2018-04-23T16:53:58.853" Score="4" ViewCount="11112" Body="&lt;p&gt;I am using Scikit-Learn for this classification problem. The dataset has 3 features and 600 data points with labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First I used &lt;code&gt;Nearest Neighbor&lt;/code&gt; classifier. Instead of using cross-validation, I manually run the fit 5 times and everytime resplit the dataset (80-20) to training set  and test set. The average score turns out to be &lt;code&gt;0.61&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf = KNeighborsClassifier(4)&#xA;score = 0&#xA;for i in range(5):&#xA;    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)&#xA;        clf.fit(X_train, y_train)&#xA;        score += clf.score(X_test, y_test)&#xA;print(scores / 5.0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However when I ran cross-validation, the average score is merely &lt;code&gt;0.45&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf =  KNeighborsClassifier(4)&#xA;scores = cross_val_score(clf, X, y, cv=5)&#xA;scores.mean()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why does cross-validation produce significantly lower score than manual resampling?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried &lt;code&gt;Random Forest&lt;/code&gt; classifier. This time using &lt;code&gt;Grid Search&lt;/code&gt; to tune the parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;param_grid = {&#xA;    'bootstrap': [True],&#xA;    'max_depth': [8, 10, 12],&#xA;    'min_samples_leaf': [3, 4, 5],&#xA;    'min_samples_split': [8, 10, 12],&#xA;    'n_estimators': [62, 64, 66, 68, 70]&#xA;}&#xA;clf_ = RandomForestClassifier()&#xA;grid_search = GridSearchCV(estimator = clf, param_grid = param_grid, &#xA;                          cv = 5, n_jobs = -1, verbose = 2)&#xA;grid_search.fit(X, y)&#xA;grid_search.best_params_, grid_search.best_score_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The best score turned out to be &lt;code&gt;0.508&lt;/code&gt; with the following parameters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;({'bootstrap': True,&#xA;  'max_depth': 10,&#xA;  'min_samples_leaf': 4,&#xA;  'min_samples_split': 10,&#xA;  'n_estimators': 64},&#xA; 0.5081967213114754)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I went ahead make prediction on the whole 600 data points and accuracy is quite high &lt;code&gt;0.7688&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;best_grid = grid_search.best_estimator_&#xA;y_pred = best_grid.predict(X)&#xA;accuracy_score(y, y_pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know &lt;code&gt;.best_score_&lt;/code&gt; is the 'Mean cross-validated score of the best_estimator.` But I don't understand why it seems so much lower than the prediction accuracy on the whole set.  &lt;/p&gt;&#xA;" OwnerUserId="49731" LastActivityDate="2021-04-14T12:14:06.080" Title="Why is cross-validation score so low?" Tags="|scikit-learn|random-forest|cross-validation|k-nn|grid-search|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30703" PostTypeId="1" CreationDate="2018-04-23T17:06:36.410" Score="7" ViewCount="269" Body="&lt;p&gt;I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot the clusters in an n-dimensional space. Which options do I have?&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-04-25T09:24:36.140" Title="How to plot clusters in nice a way?" Tags="|clustering|text-mining|plotting|matplotlib|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30704" PostTypeId="1" CreationDate="2018-04-23T17:47:36.297" Score="2" ViewCount="4700" Body="&lt;p&gt;My dataset has 140k rows with 5 attributes and 1 Attrition as target variable (value can either be 0 (Customer churn) or 1 (Customer Does not churn)). I divided my dataset in 80% training and 20% testing. My dataset is heavily imbalanced. 84% of my dataset has 0 as target variable and only 16% has 1 as target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The feature importance of my training dataset is as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ColumnA = 28%, ColumnB = 27%, AnnualFee- 17%, ColumnD - 17% and ColumnE - 11%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I initially wanted to do a very simple check of my model. After creating a Random Forest Classifier I tested the model on a dataset with just 5 rows. I kept all variables constant except Column AnnualFee. Below is a snapshot of my test dataset:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Column A   Column B    AnnualFee   ColumnD ColumnE&#xA; 4500       3.9          5%         2.1      7&#xA; 4500       3.9          10%        2.1      7&#xA; 4500       3.9          15%        2.1      7&#xA; 4500       3.9          20%        2.1      7&#xA; 4500       3.9          25%        2.1      7&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I expected that as annual fee increases the probability of customer churn also increases. But my &lt;strong&gt;rf.predict_proba(X_test)&lt;/strong&gt; seems to be all over the place. I am not sure why this is happening:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried two different codes but the anomaly seems to be happening on both the codes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code 1:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;rf = RandomForestClassifier(n_estimators = 400,random_state = 0, &#xA;min_samples_split=2,min_samples_leaf=5,&#xA;                      class_weight = {0:.0001,1:.9999})&#xA;rf.fit(X_train, Y_train )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code 2: Not My Code - Got it Online&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.model_selection import StratifiedKFold&#xA;from sklearn.model_selection import GridSearchCV&#xA;clf_4 = RandomForestClassifier(class_weight = {0:1,1:5})&#xA;estimators_range = np.array([2,3,4,5,6,7,8,9,10,15,20,25])&#xA;depth_range = np.array([11,21,35,51,75,101,151,201,251,301,401,451,501])&#xA;kfold = 5&#xA;skf = StratifiedKFold(n_splits = kfold,random_state = 42)&#xA;&#xA;model_grid = [{'max_depth': depth_range, 'n_estimators': estimators_range}]&#xA;grid = GridSearchCV(clf_4, model_grid, cv = StratifiedKFold(n_splits = 5, &#xA;random_state = 42),n_jobs = 8, scoring = 'roc_auc')&#xA;grid.fit(X_train,Y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would really appreciate any help on this!&lt;/p&gt;&#xA;" OwnerUserId="51007" LastActivityDate="2019-04-24T18:09:09.717" Title="Random Forest Classifier Probabilities" Tags="|python|predictive-modeling|random-forest|class-imbalance|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30706" PostTypeId="2" ParentId="26881" CreationDate="2018-04-23T18:29:12.120" Score="7" Body="&lt;p&gt;We first center our data by subtracting the mean of the batch.  We also divide by the standard deviation, so our formula becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ z = \frac{x - \mu}{\sigma} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ x $ is the pixel value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \mu $ is the arithmetic mean of the channel distribution&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \sigma $ is the standard deviation of the channel, calculated as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \sigma = \sqrt{\frac{\sum^N_{i=1} |x_i-\mu|^2}{N}} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By standardizing the data in this fashion, we obtain a mean ($\mu$) = 0 and a standard deviation ($\sigma$) = 1.  This allows SGD to converge faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is easier to see how feature scaling can benefit simpler models such as SVMs, but CNNs benefit similarly.  (Mini-)Batch normalization makes every pixel have a similar distribution, which helps SGD converge faster.  This is because networks generally have shared hyperparameters.  For instance, the learning rate is a multiplier for all weights in a network, rather than having a learning rate for every weight; therefore, we want a single learning rate to have proportional influence in all regions.  Otherwise, the learning rate could make some weights over-correct and other weights under-correct.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://becominghuman.ai/image-data-pre-processing-for-neural-networks-498289068258&quot; rel=&quot;noreferrer&quot;&gt;This article&lt;/a&gt; does a great job of explaining CNN preprocessing.  I'll try to highlight the key points (the following images are pulled from it)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider a face recognition challenge:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/yZTn3.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yZTn3.png&quot; alt=&quot;Labeled Faces in the Wild&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Taking the mean (left) and standard deviation (right) of the batch, we get the following:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/HtRe8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HtRe8.png&quot; alt=&quot;mean (left) and standard deviation (right)&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our centered data resembles a face, but more importantly our standard deviation image shows great variance along the borders and everywhere except the face.  From our z-score formula, we can predict that the standardized values near the border and irrelevant areas will be relatively squashed more than values around the face.  The resultant normalization appears as such:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/oHtCK.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oHtCK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Normalizing the input images is widely used and often considered an essential preprocessing step.  However, making an absolute statement such as, &quot;normalization is &lt;em&gt;always&lt;/em&gt; beneficial&quot; or &quot;normalization is &lt;em&gt;never&lt;/em&gt; beneficial&quot; would be misguided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the scenario you presented, normalizing the data would not benefit the CNN, but the network would still be able to learn the data well.  However, this is a very primitive situation, and most real-world problems have significantly more noise and variance for which to account.  In general, normalizing the input image will improve SGD convergence time.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-23T18:29:12.120" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30707" PostTypeId="2" ParentId="30020" CreationDate="2018-04-23T18:32:01.887" Score="2" Body="&lt;p&gt;I was able to get different inclusions and rendering with the following document:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;---&#xA;output: html_document&#xA;---&#xA;&#xA;```{r, results='asis', echo=FALSE}&#xA;library(knitr)&#xA;pandocto &amp;lt;- opts_knit$get(&quot;rmarkdown.pandoc.to&quot;)&#xA;if (pandocto == &quot;docx&quot;) {&#xA;  cat(&quot;## Word Heading\n\nHello world&quot;)&#xA;} else if (pandocto == &quot;html&quot;) {&#xA;  cat(&quot;## HTML Heading\n\nHello world-wide-web&quot;)&#xA;} else if (pandocto == &quot;latex&quot;) {&#xA;  cat(&quot;## LaTeX/PDF Heading\n\nHeLlO wOrLd&quot;)&#xA;}&#xA;```&#xA;&#xA;```{r}&#xA;str(opts_knit$get()&#xA;```&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you vary the head YAML with what you need, perhaps one of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output: html_document&#xA;output: pdf_document&#xA;output: word_document&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;you can see the difference. The tricks are knowing that &lt;code&gt;rmarkdown&lt;/code&gt; tells you the format to which &lt;code&gt;pandoc&lt;/code&gt; is rendering, and the use of &lt;code&gt;results='asis'&lt;/code&gt; in the block header to capture the output as raw-markdown (or latex, as appropriate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I included the second block (&lt;code&gt;str(...)&lt;/code&gt;) so you can see what else is &quot;known&quot; during document compilation/rendering.&lt;/p&gt;&#xA;" OwnerUserId="31084" LastActivityDate="2018-04-23T18:32:01.887" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30708" PostTypeId="2" ParentId="30697" CreationDate="2018-04-23T19:13:18.930" Score="2" Body="&lt;p&gt;The state-of-the-art (SOTA) for image segmentation would be Facebook's &lt;a href=&quot;https://arxiv.org/abs/1703.06870&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mask-RCNN&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it is usually trained on dataset such like &lt;a href=&quot;http://cocodataset.org/#home&quot; rel=&quot;nofollow noreferrer&quot;&gt;COCO&lt;/a&gt; or &lt;a href=&quot;http://host.robots.ox.ac.uk/pascal/VOC/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pascal&lt;/a&gt; which feature real-life objects, you can re-trained it on a dataset of your choice, real or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Facebook provides an implementation (&lt;a href=&quot;https://github.com/facebookresearch/Detectron&quot; rel=&quot;nofollow noreferrer&quot;&gt;Detectron&lt;/a&gt;) under the Apache2 license. Give it a try!&lt;/p&gt;&#xA;" OwnerUserId="51017" LastActivityDate="2018-04-23T19:13:18.930" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30709" PostTypeId="1" CreationDate="2018-04-23T19:55:16.350" Score="2" ViewCount="135" Body="&lt;p&gt;&lt;br&gt;&#xA;I am writing a scientific paper that - among other things - deals with logistic regression in the context of machine learning.&lt;br&gt;&#xA;I read this &lt;a href=&quot;https://www.csie.ntu.edu.tw/~cjlin/papers/liblinear.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt; where the author states that, given a set of instance-label pairs $(\boldsymbol{x_i},y_i),\  i=1,...,l\ \ \boldsymbol(x_i) \in R^n,\ y_i \in \{-1,+1\}$, Logistic Regression solves the following optimization problem:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{equation}\label{eq:log}&#xA;  \min_{w} \frac{1}{2}\boldsymbol{w}^T\boldsymbol{w} + C\sum\limits_{i=1}^l log(1+e^{-y_i\boldsymbol{w}^T \boldsymbol{x_i}})&#xA;\end{equation}\ \ \ \ \ \ \ \ \ \  (Eq.1)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it correct if I say that: &quot;&lt;em&gt;The first summand of Equation 1 is a regularization term while the second represents the negative log-likelihood loss function&lt;/em&gt;&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for your kind assistance.&lt;/p&gt;&#xA;" OwnerUserId="49937" LastEditorUserId="1330" LastEditDate="2020-06-17T15:14:07.050" LastActivityDate="2021-03-17T16:02:30.287" Title="Understanding logistic regression loss function equation" Tags="|scikit-learn|logistic-regression|mathematics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30710" PostTypeId="1" AcceptedAnswerId="30720" CreationDate="2018-04-23T20:20:07.720" Score="1" ViewCount="307" Body="&lt;p&gt;When evaluating results using cross-validation, several strategies can be adopted, as using 5 or 10 folds, or doing leave one out cross-validation, as well as doing a 80/20 split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Under which are general conditions should I try one or another? &lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-26T11:52:49.367" LastActivityDate="2018-05-26T11:52:49.367" Title="Size of folds in k-fold cross-validation" Tags="|machine-learning|cross-validation|model-evaluations|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30711" PostTypeId="2" ParentId="30698" CreationDate="2018-04-23T20:22:03.363" Score="3" Body="&lt;p&gt;I hear my business forecasting professor in my head right now, suggesting the Cumulative Sum (CUSUM) Control Chart for this. Basically, this is a method borrowed from control/logistics studies that tracks a value over time. If a quantity is stable over time, then the series is &quot;in control&quot;, and it falls &quot;out of control&quot; when the quantity starts to deviate systematically (that is, it stops looking like noise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the context of time series, you might compute a windowed stationarity metric from $t_{(0)}$ to $t_{(i)}$, then $t_{(i+1)}$ and so on until $t_{(n)}$, and create a CUSUM chart to watch for when the series seems to break stationarity. The same process might be applied to tracking ARIMA model parameters over time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of this thinking has already been done, of course, and it seems that R Bloggers got a relatively nice result pretty quickly by using the &lt;code&gt;changepoint&lt;/code&gt; package. You can dig into the details in the links below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I applied this method to see if health insurance membership changed drastically across the implementation of the Affordable Care Act in the USA. This would answer the question: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;If we had models built on past membership data in 2012, would we have given an acceptable forecast for post-ACA membership?&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;References:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.r-bloggers.com/changepoint-analysis-of-time-series/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.r-bloggers.com/changepoint-analysis-of-time-series/&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://cran.r-project.org/web/packages/changepoint/changepoint.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cran.r-project.org/web/packages/changepoint/changepoint.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://pdfs.semanticscholar.org/97d5/0c538be1d366f85826d440006d8de65a4f84.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pdfs.semanticscholar.org/97d5/0c538be1d366f85826d440006d8de65a4f84.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="50706" LastEditorUserId="50706" LastEditDate="2018-05-07T18:24:21.660" LastActivityDate="2018-05-07T18:24:21.660" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30712" PostTypeId="2" ParentId="30703" CreationDate="2018-04-23T20:26:43.077" Score="1" Body="&lt;p&gt;You can use a dimensionality reduction algorithm (like principal component analysis) to reduce the number of dimensions of the data to 2 or 3, and then perform scatter plots using the reduced variables, coloring them according to the cluster they belong to. In &lt;a href=&quot;https://www.kdnuggets.com/2016/02/qlucore-comparison-pca-hierarchical-clustering.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blogpost&lt;/a&gt; a similar thing is done.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-23T20:26:43.077" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30713" PostTypeId="2" ParentId="30710" CreationDate="2018-04-23T20:36:08.677" Score="3" Body="&lt;p&gt;I think cross-validation is almost always superior to a simple train-test split. The only problem is computation time (for a 5-fold cross validation you have to train your model 5 times).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your problem allows it, you should always use cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only reason you shouldn't use cross-validation is if your model takes too long to train (e.g. state-of-the-art image recognition networks require weeks of training on GPU clusters).&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-23T20:36:08.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30714" PostTypeId="1" AcceptedAnswerId="30763" CreationDate="2018-04-23T20:44:14.323" Score="4" ViewCount="5625" Body="&lt;p&gt;I have come across a David Silver's slide which contains both the terms &quot;bootstrapping&quot; and &quot;sampling&quot;. Is there any realistic example which helps me to understand the concepts better.&lt;/p&gt;&#xA;" OwnerUserId="2893" LastEditorDisplayName="user10640" LastEditDate="2019-02-18T16:39:32.640" LastActivityDate="2020-03-19T13:56:00.240" Title="What is the difference between bootstrapping and sampling in reinforcement learning?" Tags="|reinforcement-learning|terminology|difference|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30715" PostTypeId="1" CreationDate="2018-04-23T21:39:47.243" Score="9" ViewCount="16258" Body="&lt;p&gt;I'm using the &lt;a href=&quot;https://www.kaggle.com/c/titanic/data&quot; rel=&quot;noreferrer&quot;&gt;Kaggle Titanic dataset&lt;/a&gt;. One feature is &quot;Embarked&quot;, the city the passenger embarked from. The survival rate appears to correlate with it, but I'm worried it may just be correlated with the ticket Fare (which the survival rate definitely correlates with).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to plot a histogram of the fares. That would be easy. However, I also want to, &lt;em&gt;on the same plot&lt;/em&gt;, have the histograms for the three embarked values (Q,C,S), labeled by different colors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've searched but can't figure out how. I can achieve something relatively similar with FacetGrid:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.FacetGrid(train_df, col='Embarked')&#xA;g.map(plt.hist, 'Fare', bins=20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which gives three separate histograms, almost what I want:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NCSGB.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NCSGB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that's not quite what I want. For example, it's not great for comparing them. I could calculate the averages, but ideally I'd see them overlaid, so the three histograms are on the same plot, and you can tell them apart because they're labeled different colors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like PairGrid is &lt;em&gt;almost&lt;/em&gt; what I want, but it appears to be giving some strange axis for &quot;Fare&quot; (which is &lt;em&gt;not&lt;/em&gt; between 0 and 1), and the legend is blank, and the colors are solid (so you can kind of only see the top layer).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.PairGrid(train_df[[&quot;Embarked&quot;,&quot;Fare&quot;]], hue=&quot;Embarked&quot;,size=8)&#xA;g = g.map_diag(plt.hist)&#xA;g = g.add_legend()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7pWcu.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7pWcu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm very confused and not sure how to even go about finding the right answer. It seems like Seaborn can do anything, some really cool things, but it's just impossible to find the specific thing I want to do for a given circumstance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I achieve this?&lt;/p&gt;&#xA;" OwnerUserId="50805" LastActivityDate="2018-11-22T13:00:04.287" Title="How can I change the transparency of a histogram plot in Seaborn using Pairgrid?" Tags="|pandas|seaborn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30716" PostTypeId="1" AcceptedAnswerId="30726" CreationDate="2018-04-23T21:43:19.063" Score="1" ViewCount="2168" Body="&lt;p&gt;The code here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically line 53:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is the cross entropy multiplied by &lt;code&gt;original_dim&lt;/code&gt;? Also, does this function calculate cross entropy only across the batch dimension (I noticed there is no &lt;code&gt;axis&lt;/code&gt; input)? It's hard to tell from the documentation...&lt;/p&gt;&#xA;" OwnerUserId="51024" LastActivityDate="2018-04-24T16:20:27.233" Title="Keras VAE example loss function" Tags="|keras|autoencoder|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30717" PostTypeId="2" ParentId="30715" CreationDate="2018-04-23T22:09:06.010" Score="4" Body="&lt;p&gt;With a pandas dataframe, the following should do it without seaborn:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_df.groupby('Embarked')['Fare'].plot.hist(alpha=0.5)&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have to use seaborn you can use &lt;code&gt;FacetGrid&lt;/code&gt; without the &lt;code&gt;col&lt;/code&gt; and &lt;code&gt;row&lt;/code&gt; argument:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.FacetGrid(train_df, hue='Embarked')&#xA;g = g.map(plt.hist, 'Fare', alpha=0.5)&#xA;g = g.add_legend()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-04-23T22:56:55.080" LastActivityDate="2018-04-23T22:56:55.080" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30718" PostTypeId="2" ParentId="30681" CreationDate="2018-04-23T22:12:51.420" Score="2" Body="&lt;p&gt;There are several ways to convert words to one hot encoded vectors. Since I do not know the data structure you store your data. I assume it is going to be a list&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;from keras.preprocessing.text import Tokenizer &lt;br&gt;&#xA;  samples = ['The', 'dog','mouse','elephant']&lt;br&gt;&#xA;  tokenizer = Tokenizer(num_words=len(samples))&lt;br&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This builds the word index &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;tokenizer.fit_on_texts(samples)&lt;br&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;one hot representation&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')&lt;br&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;By changing the mode from binary to 'tfidf' or 'count', you can make a matrix of any type, apart from one hot. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can achieve the same result using other packages like sklearn. But it does involve a bit more lines of code.&lt;/p&gt;&#xA;" OwnerUserId="45958" LastActivityDate="2018-04-23T22:12:51.420" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30720" PostTypeId="2" ParentId="30710" CreationDate="2018-04-23T22:37:14.803" Score="2" Body="&lt;p&gt;I generally advocate for cross validation in addition to a hold-out sample. As for the number of folds, that depends heavily on your data. Generally you start to approach diminishing returns after some point, but you should try, and evaluate, several regimes. This is very much an empirical question with no hard and fast best answer. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-04-23T22:37:14.803" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30721" PostTypeId="2" ParentId="30573" CreationDate="2018-04-23T22:52:33.467" Score="0" Body="&lt;p&gt;Some of the features were categorical, so they had to be converted to integers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, I ran this to get the data in the right format:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import scipy.sparse as sp&#xA;X_train = sp.csr_matrix(X_train)&#xA;y_train = sp.csr_matrix(y_train)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50746" LastActivityDate="2018-04-23T22:52:33.467" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30723" PostTypeId="2" ParentId="22476" CreationDate="2018-04-23T23:56:22.620" Score="1" Body="&lt;p&gt;I assume that when you say&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;values that are originally small get changed the same ratio&lt;/em&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;you are standardizing your features so that every dimension is on the same scale from [-1,1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is common practice as it helps many models converge faster.  It would not prevent your models from generalizing the data, so here are some other possible causes for your problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The ML algorithm you are using is not appropriate for your task&lt;/li&gt;&#xA;&lt;li&gt;The data is either insufficient in quantity or quality (e.g. noisy data)&lt;/li&gt;&#xA;&lt;li&gt;The preprocessing stage is lacking some steps such as dimensionality reduction&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-23T23:56:22.620" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30724" PostTypeId="1" CreationDate="2018-04-24T00:19:30.190" Score="1" ViewCount="59" Body="&lt;p&gt;i am using a geographic dataset and i intend to use SVR as machine learning method for predicting spatiotemporal patterns from this dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is does SVR canensure spatiotemporal prediction from geographic datasets?&lt;/p&gt;&#xA;" OwnerUserId="51031" LastActivityDate="2018-04-24T00:19:30.190" Title="spatiotemporal prediction using support vector regression" Tags="|machine-learning|regression|prediction|geospatial|svr|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30725" PostTypeId="2" ParentId="30367" CreationDate="2018-04-24T00:38:06.987" Score="3" Body="&lt;p&gt;I recently had a similar trouble with porting working models to tensorflow 1.7. I found that the initialization was the culprit. Adding &#xA;&lt;code&gt;model.add(Dense(2048, activation='relu', kernel_initializer=initializers.glorot_normal()))&lt;/code&gt; instead of &lt;code&gt;model.add(Dense(2048, activation='relu'))&lt;/code&gt; fixed the problem.&lt;/p&gt;&#xA;" OwnerUserId="51033" LastActivityDate="2018-04-24T00:38:06.987" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30726" PostTypeId="2" ParentId="30716" CreationDate="2018-04-24T02:05:40.377" Score="1" Body="&lt;p&gt;Keras metrics.binary_crossentropy computes the cross entropy averaged across all inputs (pseudocode):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;original_dim = 3 &#xA;x = [1,1,0]&#xA;x_decoded = [0.2393,0.7484,-1.1399]&#xA;&#xA;average_BCE = binary_crossentropy(x, x_decoded)&#xA;print(average_BCE)&#xA;&amp;gt;&amp;gt;&amp;gt;0.1186&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For this part of autoencoder loss we need the sum, not the average over all squared differences between input and output pixels, which is equivalent to  average_crossentropy_of_pixels * num_pixels (original_dim)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(original_dim * average_BCE)&#xA;&amp;gt;&amp;gt;&amp;gt;0.3559&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another way of writing this part would (which I think is more illustrative of what's happening, but probably less performant in Keras land):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xent_loss = K.sum(K.square(x - sigmoid(x_decoded)))&#xA;print(xent_loss)&#xA;&amp;gt;&amp;gt;&amp;gt;0.3559&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Regarding the second part, since the first operation you are really doing is subtraction, it implied the tensors for input and output are the same size. This can be found if you check the implementation code - which is better than the docs in this case - go to line 3056:&#xA;&lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-04-24T02:22:12.803" LastActivityDate="2018-04-24T02:22:12.803" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30727" PostTypeId="2" ParentId="30704" CreationDate="2018-04-24T02:18:51.217" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;What do you mean by 'all over the place'? Have you tried actually using scoring measures such as accuracy scores, AUC, etc? These will play a vital role in determining how your model is performing. I'd suggest ROC AUC as to start off since accuracy will be pretty misleading due to the class imbalance.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The class weights you used make little sense. You usually would use something like &lt;code&gt;n_samples / (n_classes * np.bincount(y))&lt;/code&gt; for your weighting, but since sklearn provides automatic balancing of classes, you should probably use that by setting &lt;code&gt;class_weights='balanced'&lt;/code&gt;: &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23495" LastActivityDate="2018-04-24T02:18:51.217" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30728" PostTypeId="2" ParentId="30676" CreationDate="2018-04-24T03:07:51.343" Score="1" Body="&lt;p&gt;In simple words: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Derivative shows neuron's &lt;strong&gt;ability to learn&lt;/strong&gt; on particular&#xA;  input.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For example if input is &lt;strong&gt;0&lt;/strong&gt; or &lt;strong&gt;1&lt;/strong&gt; or &lt;strong&gt;-2&lt;/strong&gt;, the derivative (the &quot;learning ability&quot;) is high and back-propagation will improve neuron's weights for this sample dramatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On other hand, if input is &lt;strong&gt;20&lt;/strong&gt;, the the derivative will be very close to &lt;strong&gt;0&lt;/strong&gt;. &#xA;It means that back-propagation on this sample will not &quot;teach&quot; this neuron to produce a better result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The things above are valid for a single sample. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's look at the bigger picture, for all samples in the training set. &#xA;Here we have several situations:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If derivative is 0 for all samples in your training set AND neuron always produces &lt;strong&gt;wrong&lt;/strong&gt; results - it means the neuron is saturated (dumb) and will not improve.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If derivative is 0 for all samples in your training set AND neuron always produces &lt;strong&gt;correct&lt;/strong&gt; results - it means the neuron have been studying really well and already as smart as it could (side note: this case is good but it may indicate potential overfitting, which is not good)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If derivative is 0 on some samples, non-0 on other samples AND neuron produces &lt;strong&gt;mixed&lt;/strong&gt; results - it indicates that this neuron doing some good work and potentially may improve from further training (though not necessarily as it depends on other neurons and training data you have)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So, when you are looking at the derivative plot, you can see how much the neuron prepared to learn and absorb the new knowledge, given a particular input.&lt;/p&gt;&#xA;" OwnerUserId="51037" LastEditorUserId="40853" LastEditDate="2018-05-02T08:19:01.183" LastActivityDate="2018-05-02T08:19:01.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30729" PostTypeId="2" ParentId="30703" CreationDate="2018-04-24T04:35:15.163" Score="3" Body="&lt;p&gt;T-SNE is another dimensionality reduction algorithm not mentioned in the article in the other answer. Used for VERY high dimensional data, if you have trained some embeddings for your dataset. &lt;a href=&quot;https://lvdmaaten.github.io/tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reference Here&lt;/a&gt; . Python standard library &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;cheers &lt;/p&gt;&#xA;" OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-04-24T05:17:28.000" LastActivityDate="2018-04-24T05:17:28.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30730" PostTypeId="2" ParentId="30227" CreationDate="2018-04-24T05:09:21.773" Score="1" Body="&lt;p&gt;From what I have seen dropout for LSTMs should not be so high as 0.5 Recommendations are 0.1 or less.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=6&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwjAyezvi9LaAhURmuAKHQjKA2QQFgh4MAU&amp;amp;url=https%3A%2F%2Fpdfs.semanticscholar.org%2F3061%2Fdb5aab0b3f6070ea0f19f8e76470e44aefa5.pdf&amp;amp;usg=AOvVaw1X48aCvayLEsFLsldsT0cA&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper - Where to Apply Dropout in Recurrent Neural&#xA;Networks for Handwriting Recognition&lt;/a&gt; has a detailed study on how to use Dropout with RNNs, and one of the recommendations is that 'it is almost always better before the lstm layer than inside or after it' (pg 3), so you can try that. However, I believe this depends on kind of data you have. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I remember reading a paper that had dropout for LSTM only being useful for a large LSTM (like 4096 unit x 4 layers). I cannot find it now, but in &lt;a href=&quot;https://www.google.com/url?sa=t&amp;amp;rct=j&amp;amp;q=&amp;amp;esrc=s&amp;amp;source=web&amp;amp;cd=5&amp;amp;cad=rja&amp;amp;uact=8&amp;amp;ved=0ahUKEwix-PChkdLaAhVINd8KHe5NB2AQFghjMAQ&amp;amp;url=https%3A%2F%2Fwww.danielpovey.com%2Ffiles%2F2017_interspeech_dropout.pdf&amp;amp;usg=AOvVaw1OkTKu-VedhKaSnNNs_Opa&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;, the authors suggest something similar - showing dropout having better results on a 1500 unit x 2 layer lstm than a 650 unit by 2 layer, while the smaller network would just overfit. The lesson I guess depends on how big your lstm is and what you are trying to achieve - the use case being preventing overfitting while increasing model size.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-24T05:09:21.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30731" PostTypeId="2" ParentId="30702" CreationDate="2018-04-24T05:28:15.107" Score="2" Body="&lt;p&gt;In your random forest, this is due to the fact that your final model is overfitting. Sklearn's &lt;code&gt;GridSearchCV&lt;/code&gt; has a default argument &lt;code&gt;refit = True&lt;/code&gt;, that takes the model with the best performance based on cross-validation and retrains it in the whole dataset. Your &lt;code&gt;accuracy score&lt;/code&gt; is very high due to the fact that it is only measured on your training data, and the &lt;code&gt;best_score&lt;/code&gt; is a measure that incorporates how your model performs in models that it has not seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wrap up, in your random forest you are overfitting very badly, as there is a big gap between your validation and training error. Try to use &lt;code&gt;refit = False&lt;/code&gt; and you will not see this gap anymore (but you still have a problem since you are still overfitting your training set with this model).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T05:28:15.107" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30732" PostTypeId="2" ParentId="30686" CreationDate="2018-04-24T05:52:50.693" Score="0" Body="&lt;p&gt;Tensorflow's softmax function only works if the number of batches are in the rows and the output in the columns. If these are reversed, then you need to transpose the tensors in the cost function.&lt;/p&gt;&#xA;" OwnerUserId="50992" LastActivityDate="2018-04-24T05:52:50.693" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30734" PostTypeId="1" AcceptedAnswerId="45088" CreationDate="2018-04-24T07:29:07.937" Score="2" ViewCount="820" Body="&lt;p&gt;Apologies for the newbie question: I've just downloadd the GoogleNews word2vec bin file and used convertvec to convert it to a text file to look at the vector values. I see they are values between -1 and 1. Can somebody explain how these values map back to related terms?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would eventually like to write some code to take this bin/text file, transform it and load it into a MongoDB database for further manipulation.&lt;/p&gt;&#xA;" OwnerUserId="51044" LastActivityDate="2019-07-05T13:01:25.120" Title="In Word2Vec, how to the vector values translate back to related words?" Tags="|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30735" PostTypeId="2" ParentId="410" CreationDate="2018-04-24T07:46:32.870" Score="3" Body="&lt;p&gt;Let me give a brief introduction to another approach on choosing the learning rate, based on Jeremy Howard's &lt;a href=&quot;http://www.fast.ai/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning&lt;/a&gt; course 1. If you want to dig deeper, see &lt;a href=&quot;https://techburst.io/improving-the-way-we-work-with-learning-rate-5e99554f163b&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blogpost&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The learning rate proposed in Jeremy Howard's course is based on a systematic way to try different learning rates and choose the one that makes the loss function go down the most. This is done by feeding many batches to the mini-batch gradient descent method, and increasing the learning rate every new batch you feed to the method. When the learning rate is very small, the loss function will decrease very slowly. When the learning rate is very big, the loss function &#xA;will increase. Inbetween these two regimes, there is an optimal learning rate for which the loss function decreases the fastest. This can be seen in the following figure:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/x0zef.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/x0zef.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We see that the loss decreases very fast when the learning rate is around $10^{-3}$. Using this approach, we have a general way to choose an approximation for the best constant learning rate for our netowork.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T07:46:32.870" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30736" PostTypeId="2" ParentId="30336" CreationDate="2018-04-24T07:55:27.770" Score="1" Body="&lt;p&gt;use this command &#xA;.\bin\spark-submit \guruorders\pcakinetics\files\kineticPCA.py&lt;/p&gt;&#xA;" OwnerUserId="51048" LastActivityDate="2018-04-24T07:55:27.770" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30737" PostTypeId="1" CreationDate="2018-04-24T08:01:08.137" Score="1" ViewCount="44" Body="&lt;p&gt;I have a semi structured data set. I need to collect some data (unlabeled) randomly for labeling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As initiative at first I separated labeled and unlabeled data. Then I convert those data from string to numeric as all data were string. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;After that I cluster unlabeled data into 2(cluster 0 and cluster 1) by applying k-means clustering algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now when I collect some data randomly from both cluster it takes indexes from both labeled and unlabeled data. But my expectation was, indexes will come just from unlabeled data as cluster was created by using unlabeled data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I couldn't figured out the problem. I am giving my code below. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_label.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LktiC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LktiC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The indexes of labeled data stats from 0 to 343 (344 data) and unlabeled data starts from 344 to 863 (519 data)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;train_unlabeled.head()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/eLZ1y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/eLZ1y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# define X and y&#xA;feature_cols = ['TopLeft', 'TopMiddle', 'TopRight', 'MiddleLeft','MiddleMiddle', 'MiddleRight', 'BottomLeft', 'BottomMiddle', 'BottomRight']                   &#xA;&#xA;# X is a matrix, hence we use [] to access the features we want in feature_cols&#xA;X =train_unlabeled[feature_cols]`&#xA;&#xA;# y is a vector, hence we use dot to access 'label'&#xA;y =train_unlabeled['Class']&#xA;&#xA;X_scale = scale(X)&#xA;reduced_data = PCA(n_components=None).fit_transform(X_scale)&#xA;kmeans = KMeans(init='k-means++', n_clusters=2, n_init=10)&#xA;model=kmeans.fit(reduced_data)&#xA;reduced_data.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7yjiE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7yjiE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;labels=kmeans.labels_&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I was trying to collect 95% data from each cluster (cluster was created by &lt;code&gt;train_unlabeld&lt;/code&gt; data)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = 0.95&#xA;i=0&#xA;C_i = np.where(labels == i)[0].tolist() &#xA;n_i = len(C_i)&#xA;sample_i = np.random.choice(C_i, int(x * n_i)) &#xA;print (i, sample_i)&#xA;list1=(sample_i)&#xA;x = 0.95&#xA;i=1&#xA;C_i = np.where(labels == i)[0].tolist() &#xA;n_i = len(C_i)&#xA;sample_i = np.random.choice(C_i, int(x * n_i)) &#xA;print (i, sample_i)&#xA;list2=(sample_i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I print the index while collecting data randomly depending on cluster(0 and 1) I found these indexes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JSabd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JSabd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As the index of unlabeled data is started from 344, so how the index of labeled data comes in cluster 0 and ? I didn't understand it.&lt;/p&gt;&#xA;" OwnerUserId="43455" LastEditorUserId="34269" LastEditDate="2018-04-25T02:50:49.823" LastActivityDate="2018-04-25T02:50:49.823" Title="Problem faced when collect data randomly from cluster" Tags="|machine-learning|python|clustering|k-means|semi-supervised-learning|" AnswerCount="1" CommentCount="6" ClosedDate="2018-04-28T06:26:17.820" ContentLicense="CC BY-SA 3.0" />
  <row Id="30739" PostTypeId="2" ParentId="30007" CreationDate="2018-04-24T08:16:12.207" Score="1" Body="&lt;p&gt;Yes, I think that the idea is to think of a histogram as a particular case of &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kernel Density Estimation&lt;/a&gt;. In this case, the kernel function for a KDE to be a histogram is an &lt;a href=&quot;https://en.wikipedia.org/wiki/Indicator_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;indicator function&lt;/a&gt;. In this case, the bin size is equivalent to the bandwidth of the KDE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The bandwidth of the KDE is indeed a regularity measure: KDEs with very high bandwidth have huge bias (they say that the distribution is uniform), while KDEs with very low bandwidth have huge variance (they adapt to the training data but do not generalize well). As the bandwidth of a KDE is equivalent to the bin size, we can say that the bin size is a regularity measure.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T08:16:12.207" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30740" PostTypeId="1" CreationDate="2018-04-24T08:41:25.700" Score="1" ViewCount="2655" Body="&lt;p&gt;Sorry to start such an unspecific question, but I am slightly lost in the big topic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My tutor proposed to chose neural networks for my final project, and we started by building a CNN for detecting hand written digits in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My own idea was to build another neural network that can play video games of ascending difficulty, for example pong -&gt; snake -&gt; megaman [with tools for extracting in game data] or driving in GTA San Andreas similar to what Sentdex did. (For GTA, I would try something different to Sentdex as I would train the neural network by reinforcement learning in artificial conditions, starting with staying in a straight lane and then moving on to curves, traffic etc.) &#xA;I made my own version of snake, which I could use for additional input and seems like a realistic minimal goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now, I am not so sure which approach I should take for this. I took a look at Sethblings MarI/O and read the NEAT paper. Then, I read the book &quot;Machine Learning in Python&quot; explaining a lot of concepts, and realised that neural networks and especially GA's are only a small part of possible algorithms. Also, I am not sure how much Sethblings network actually thinks and how much it just randomly adapted to the level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you think that the following idea is realistic?:&#xA;&quot;I build my own neural network in python based on NEAT, then test it on video games / tasks of increasing difficulty and try to tune parameters for performance&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could also imagine comparing NEAT to an implementation of Alexnet for the same tasks.&#xA;The whole goal is not to create the perfect NEAT but instead just make one and see how far it takes me. I have a lot of time and motivation at hand, but I don't want to exceed the boundaries of a high school final and my own possibilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feedback welcome and thank you in advance&lt;/p&gt;&#xA;" OwnerUserId="51049" LastActivityDate="2018-05-24T13:18:57.153" Title="Is it possible to use NEAT networks for solving video games?" Tags="|machine-learning|neural-network|genetic-algorithms|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30741" PostTypeId="1" CreationDate="2018-04-24T09:07:01.407" Score="1" ViewCount="87" Body="&lt;p&gt;I have a length 200000 time series of six dimensional data describing the evolution of a chaotic dynamical system. I am exploring ways to partition the state space of the data into minimally interacting regions; I want the least transitions from cluster to another in my time series. Other than not being interested in the trivial case of only 1 cluster I have no restrictions on cluster number though I expect it to be small (K&amp;lt;6 say).&#xA;The purpose of this is to try and identify quasi-stable basins of attraction in the chaotic system.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried Hidden Markov Models with both Gaussian and Gaussian Mixture emissions, but they do not well capture the very non-gaussian structure of my data in phase space. I have also tried using DBSCAN with some success but since it didnt treat data as a time series, the parameters had to be very carefully tuned to get good results, which left me dissatisfied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do any other approaches leap to mind?&lt;/p&gt;&#xA;" OwnerUserId="51051" LastActivityDate="2023-01-11T07:06:25.697" Title="What is an efficient clustering approach for grouping multivariate timeseries data into optimally persistent states?" Tags="|machine-learning|clustering|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30742" PostTypeId="1" AcceptedAnswerId="30743" CreationDate="2018-04-24T09:12:28.423" Score="0" ViewCount="1290" Body="&lt;p&gt;I have implemented a Linear regression model on a dataset of 7 independent variable and 1 target  with the below 2 approaches&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Without normalization of the data, resulted in a Mean squared error of 36530921.0123 and R2  value 0.7477&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) With normalization of the data, resulted in a Mean squared error of 5.468490570335696e-10 and R2 value 0.92750882996584. But in this case even the Target variable is also normalized.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which is the better approach adding new features to the 1st case or considering the 2nd case and it has a better R2 and MSE value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we are considering the 2nd case, what is the steps need to be taken to denormalize the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-24T10:03:12.390" Title="De-normalization in Linear Regression" Tags="|machine-learning|python|linear-regression|normalization|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30743" PostTypeId="2" ParentId="30742" CreationDate="2018-04-24T10:03:12.390" Score="1" Body="&lt;p&gt;Unless you normalize the MSE in scenario 1 (or denormalize the MSE in scenario 2), &lt;strong&gt;comparing two MSE with two different scales is irrelevant&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have data with values varying from 10 to 30 millions, centered then normalized to -1/+1. Suppose you have a MSE of 1000 in the first case, and 0.1 in the second, you will easily see that the second MSE is way more impacting than the first one after (de)normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, if you want to retrieve the target from scenario 2, you need to &lt;strong&gt;apply the reverse operations&lt;/strong&gt; to what has been done to get the &quot;normalized&quot; target. Assuming for instance that you centered / reduced the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;Z = \frac{Y-\bar{Y}}{\bar{Y}} &#xA;$$&#xA;$$&#xA;Z = \alpha X_1 + \beta X_2 + \gamma X_3&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $Y$ is your initial target, $\bar{Y}$ its average, $Z$ your normalized target and $X_i$ your predictors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you apply your model and get a prediction, say $\hat{z}$, you can calculate its corresponding value $\hat{y}$ after applying the reverse transformations to your normalization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\hat{z} = \alpha x_1 + \beta x_2 + \gamma x_3&#xA;$$&#xA;$$&#xA;\frac{\hat{y}-\bar{y}}{\bar{y}}  = \alpha x_1 + \beta x_2 + \gamma x_3&#xA;$$&#xA;$$&#xA;\hat{y} = \bar{y} *(\alpha x_1 + \beta x_2 + \gamma x_3 ) + \bar{y}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now in this particular case, as Ankit Seth pointed out, you need not normalizing your target. The linear model would have adjusted its coefficients automatically, because you have been using linear operands. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, if you proceed to non linear operations for target normalization, the same logic should apply. Say for instance that your model predicts the logarithm of your target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;log(Y) = \alpha X_1 + \beta X_2 + \gamma X_3&#xA;$$&#xA;The reverse operation would be to apply its reciprocal function, e.g. exponential.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;Y = exp(\alpha X_1 + \beta X_2 + \gamma X_3)&#xA;$$&lt;/p&gt;&#xA;" OwnerUserId="42829" LastActivityDate="2018-04-24T10:03:12.390" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30745" PostTypeId="1" AcceptedAnswerId="30746" CreationDate="2018-04-24T10:57:57.240" Score="1" ViewCount="350" Body="&lt;p&gt;Which way is better for drawing the ground truth boxes for object detection?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Drawing as tight as the sides of the box touch the border of the object, or&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0m52Y.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0m52Y.jpg&quot; alt=&quot;1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Make a little space between the box and the object?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BiuHo.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BiuHo.jpg&quot; alt=&quot;2&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="51060" LastEditorUserId="29575" LastEditDate="2018-04-24T13:59:13.333" LastActivityDate="2018-04-24T13:59:13.333" Title="Space between an object and the ground truth bounding box" Tags="|deep-learning|dataset|object-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30746" PostTypeId="2" ParentId="30745" CreationDate="2018-04-24T11:29:14.273" Score="2" Body="&lt;p&gt;The first image is a better bounding box.  A perfect bounding box has no space in between the edges of the object and the box, but it also fully contains the object.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-24T11:29:14.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30747" PostTypeId="2" ParentId="30737" CreationDate="2018-04-24T11:32:48.120" Score="0" Body="&lt;p&gt;The problem was understanding the index between &lt;code&gt;train_unlabeled&lt;/code&gt; data and the cluster. There are no connection between index of &lt;code&gt;train_unlabeled&lt;/code&gt; data set and the data that belongs to cluster. As I am using jupyter notebook, so the indexes given to itself. Now when I clustered those unlabelled data (index start from 344 to 863) into 2, then which data belongs which cluster it already takes its new index for cluster which is not same as previous index.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For better understanding I'm giving an example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;suppose for unlabelled data (&lt;code&gt;train_unlabeled&lt;/code&gt;) the data those indexes are 344,345,346,347 and 348. After clustering 344 and 346 took their place at cluster 0 and 345,347,348 at cluster 1. While clustering the new index of 344 will be 0, for 345 will be 1, for 346 will be 2 and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why when I print the indexes of data, belongs to cluster 0 and 1 (by using &lt;code&gt;print (i, sample_i)&lt;/code&gt; this command) it just shows the index of data of its own cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My mistake was I though that both indexes are same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for keep similarity of indexes before and after cluster, I reset the the index of &lt;code&gt;train_unlabeled&lt;/code&gt; data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The command is: &lt;code&gt;train_unlabeled = train_unlabeled.reset_index()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here comes another problem, that is a new column named:index is added.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4Uxkz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Uxkz.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, after resetting index (now, index starts from 0), I drop the extra index column by using this command : &lt;code&gt;train_unlabeled =train_unlabeled.drop('index', axis=1, inplace=True)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And at last I cluster that unlabeled data (&lt;code&gt;train_unlabeled&lt;/code&gt;) and would be able to take 95% data randomly from cluster by using cluster index.&lt;/p&gt;&#xA;" OwnerUserId="43455" LastActivityDate="2018-04-24T11:32:48.120" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30748" PostTypeId="2" ParentId="16751" CreationDate="2018-04-24T11:57:13.323" Score="5" Body="&lt;p&gt;According to &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/linear_model/logistic.py#L1176&quot; rel=&quot;noreferrer&quot;&gt;sklearn's Logistic source code&lt;/a&gt;, the solver used to minimize the loss function is the SAG solver (Stochastic Average Gradient). &lt;a href=&quot;https://hal.inria.fr/hal-00860051/document&quot; rel=&quot;noreferrer&quot;&gt;This paper&lt;/a&gt; defines this method, and in &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b464d094d2c468a16ea9f8bf8d42d949f84/sklearn/linear_model/sag.py&quot; rel=&quot;noreferrer&quot;&gt;this link&lt;/a&gt; there is the implementation of the sag solver. This implementation of the solver uses a method to obtain the step size (learning rate), so there is not a way that you can change the learning rate (unless you want to change the source code).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T11:57:13.323" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30749" PostTypeId="2" ParentId="30740" CreationDate="2018-04-24T12:11:09.057" Score="3" Body="&lt;p&gt;Implementing the NEAT algorithm for playing video games is quite realistic, especially considering that it is not novel, so there are extensive resources and tutorials to help you.  In fact, there are publications that will walk you through exactly what you want to do, such as &lt;a href=&quot;https://www.cs.swarthmore.edu/~meeden/cs81/s14/papers/Luis.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; on using the NEAT algorithm for playing Pong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course designing a self-driving car would be significantly more advanced than playing the 2D games you mentioned, but it appears that much of the work for developing AI in GTA V exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The person you mentioned, Sentdex, has a &lt;a href=&quot;https://github.com/Sentdex/pygta5&quot; rel=&quot;nofollow noreferrer&quot;&gt;github page&lt;/a&gt; for developing GTA V code.  However, note that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We read frames directly from the desktop, rather than working with the game's code itself. This means it works with more games than just GTA V, and it will basically learn (well, attempt to learn...) whatever you put in front of it based on the frames as input and key presses as output.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is a very primitive approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am by no means an expert on self-driving cars, but after some Googling, &lt;strong&gt;&lt;a href=&quot;https://deepdrive.berkeley.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;deep drive&lt;/a&gt; appears to be the premier simulated self-driving car development platform.&lt;/strong&gt;  If you are going to develop the GTA network, I suggest checking out it out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;hr&gt;&#xA;As for comparing to an implementation of AlexNet:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again, I am not an expert on self-driving cars, but I would expect an &lt;a href=&quot;https://link.springer.com/chapter/10.1007%2F3-540-45014-9_1&quot; rel=&quot;nofollow noreferrer&quot;&gt;ensemble&lt;/a&gt; of networks to be used for this type of task, so I imagine an object detection network or &lt;a href=&quot;http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review&quot; rel=&quot;nofollow noreferrer&quot;&gt;semantic segmentation&lt;/a&gt; network to be used in conjunction with a network designed with the NEAT algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, you can use a &lt;a href=&quot;https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;fully convolutional&lt;/a&gt; AlexNet model to detect objects such as pedestrians/cars/sidewalks/etc and feed that data into another network that also takes other inputs such as your current speed, wheel angle, road conditions etc that combines all this information and translates it into actions such as applying the brakes, turning etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;AlexNet is a Convolutional Neural Network (CNN), which is primarily used in the field of vision processing.  The NEAT algorithm typically generates Multilayer Perceptrons (MLP), which are not used individually for machine vision challenges; they are different tools for different tasks.  It would be like asking, &quot;which is better: a hammer or a screwdriver&quot;.  The answer depends on the task.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, using NEAT for video game AI development is a good introductory project for some of the more advanced topics in machine learning and deep learning.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-24T12:17:16.183" LastActivityDate="2018-04-24T12:17:16.183" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30751" PostTypeId="2" ParentId="441" CreationDate="2018-04-24T12:23:45.790" Score="0" Body="&lt;p&gt;It would be fair to &lt;a href=&quot;https://www.scnsoft.com/blog/spark-vs-hadoop-mapreduce&quot; rel=&quot;nofollow noreferrer&quot;&gt;compare Spark with MapReduce&lt;/a&gt; - Hadoop's processing framework. &#xA;In the majority of cases, Spark may outperform MapReduce. The former enables in-memory data processing, which makes it possible to process data up to 100 times faster. For this reason, Spark is a preferred option if you need insights quickly, for example, if you need to&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;run customer analytics, e.g. compare the behavior of a customer with&#xA;the behavior patterns of a particular customer segment and trigger&#xA;certain actions;&lt;/li&gt;&#xA;&lt;li&gt;manage risks and forecast various possible scenarios;  &lt;/li&gt;&#xA;&lt;li&gt;detect fraud in real-time;&lt;/li&gt;&#xA;&lt;li&gt;run industrial big data analytics and predict anomalies and machine failures.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, MapReduce is good at processing really huge datasets (if you are fine with the time required for processing). Besides, it's a more economical solution, as MapReduce reads from/writes to a disk. And disks are generally cheaper than memory. &lt;/p&gt;&#xA;" OwnerUserId="51065" LastEditorUserId="29575" LastEditDate="2018-04-24T13:53:38.660" LastActivityDate="2018-04-24T13:53:38.660" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30752" PostTypeId="1" AcceptedAnswerId="30761" CreationDate="2018-04-24T12:41:16.470" Score="1" ViewCount="95" Body="&lt;p&gt;I have been working on online learning for a few weeks now, especially with Vowpal Wabbit and logistic regression. My understanding of the online learning algorithms and the problem is alright but I can't get my head straight about the regularisation issue. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a standard machine learning problem, one uses a validation dataset in order to tune regularisations parameters for L1 and L2. But how do you choose those regularisation parameters in an online setting? Do you just fix them from start, or should I update them while training occurs?&lt;/p&gt;&#xA;" OwnerUserId="49293" LastEditorUserId="29575" LastEditDate="2018-04-24T14:00:06.340" LastActivityDate="2018-04-24T14:32:22.710" Title="Should I update my regularisation L1 and L2 regularisation parameters in online setting?" Tags="|regularization|online-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30753" PostTypeId="2" ParentId="17508" CreationDate="2018-04-24T12:41:18.677" Score="2" Body="&lt;p&gt;No, &lt;code&gt;coeff1/sum(coeff)&lt;/code&gt; cannot be a probability due to the fact that it can be bigger than 1. For instance:&#xA;&lt;code&gt;coeff1 = 1.&lt;/code&gt;, &lt;code&gt;coeff2 = -.5&lt;/code&gt;, then &lt;code&gt;coeff1/sum(coeff) = 2&lt;/code&gt;. The interpretation of coefficients in linear regression is the derivative of the output variable with respect to the given predictor.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T12:41:18.677" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30754" PostTypeId="1" CreationDate="2018-04-24T12:51:14.393" Score="5" ViewCount="1411" Body="&lt;p&gt;I'm trying to detect anomalies in an univariate time series.&#xA;I trained a RNN LSTM and currently I get one-step-ahead predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone explain if it's possible to output a confidence interval (or maybe a prediction interval) with RNN LSTM instead of just a predicted value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I want to flag observations as anomulous on real-time data I think I need more information than just the predicted and observed value for point x (f.e. I need a predicted confidence interval).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please point me into the right direction.&#xA;Currently I'm using an Adam optimizer, MSE as the loss function and a Dense output layer with a linear activation function.&lt;/p&gt;&#xA;" OwnerUserId="50832" LastEditorUserId="50832" LastEditDate="2018-04-24T17:46:23.000" LastActivityDate="2018-04-30T12:25:06.823" Title="Anomaly detection using RNN LSTM" Tags="|time-series|statistics|rnn|lstm|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30756" PostTypeId="2" ParentId="30695" CreationDate="2018-04-24T13:10:34.677" Score="3" Body="&lt;p&gt;The process you described is &lt;em&gt;not&lt;/em&gt; valid, it is almost certainly overfitting. However the overall idea may very well be valid. What you should do instead is take a portion of your dataset as the testing which &lt;em&gt;is not looked at during algorithm development&lt;/em&gt;, and split the rest between training and validation sets however you please.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking only at the training and validation sets, develop your entire end to end algorithm, including the classifier!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you are happy with the algorithm, try running it on the test set to get a more accurate measure of performance.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-04-24T13:10:34.677" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30757" PostTypeId="1" AcceptedAnswerId="30768" CreationDate="2018-04-24T13:21:25.807" Score="4" ViewCount="1439" Body="&lt;p&gt;When using ensembling methods for regression, a common approach is to average (using the arithmetic mean) the outputs of the weak learners in order to obtain the output of the ensemble. Is there a reason why the average that people take is the arithmetic mean? Why not performing a geometric or harmonic mean?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-06-12T12:09:53.227" LastActivityDate="2018-06-12T12:09:53.227" Title="Geometric and harmonic means in ensembling methods" Tags="|machine-learning|statistics|ensemble-modeling|ensemble|ensemble-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30758" PostTypeId="2" ParentId="30214" CreationDate="2018-04-24T13:30:38.353" Score="0" Body="&lt;p&gt;Reading through the theoretical part of the tutorial, I understand that ideally, in the Word2Vec Kares architecture, for each target word, you would input all the available context words in order to make the network learn (through reinforcement) the words typically appearing in the same context as the target words (the &quot;positive examples&quot; in your question) and reduce the similarity between the target word and words not typically appearing in its context (the &quot;negative examples&quot; in your question).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You typically go through all the &quot;positive examples&quot; (they are not many) so that the corresponding words end up being placed in the same region of the 300-D vector space. However, going through all the &quot;negative examples&quot; would be a big computational burden due to their sheer amount (they are too many of them). So you &quot;randomly&quot; select some of them (the &quot;negative samples&quot;) and train only for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So indeed there is some &quot;imbalance&quot; (you select &quot;some&quot; instead of &quot;all&quot; the &quot;negative examples&quot; for each target word) since the method makes sure that the target word is more &quot;similar&quot; to words of the same context in comparison to some (instead of all) the words of different contexts, but the same imbalance gives a substantial computational gain. After you have completed the training, you end up with a very nice 300-D representation of your words, readily available in the embedding layer, effectively solving the word embedding problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See also &lt;a href=&quot;https://www.quora.com/What-is-negative-sampling&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; answer from Quora.&lt;/p&gt;&#xA;" OwnerUserId="50993" LastActivityDate="2018-04-24T13:30:38.353" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30759" PostTypeId="1" CreationDate="2018-04-24T14:18:57.557" Score="1" ViewCount="90" Body="&lt;p&gt;I want to modify the cost function such that the model is &lt;strong&gt;penalized more&lt;/strong&gt; if it gives bad predictions for certain important entities (say top selling products in case of demand prediction). Is it possible?&lt;/p&gt;&#xA;" OwnerUserId="50852" LastActivityDate="2018-04-24T14:18:57.557" Title="How to penalize a tensorflow dnn regressor model based on specific conditions?" Tags="|machine-learning|python|tensorflow|regression|optimization|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30760" PostTypeId="1" CreationDate="2018-04-24T14:22:29.517" Score="3" ViewCount="392" Body="&lt;p&gt;I'm a bit confused about how data in audio files can be processed for a ML classification model. I have several .wav files that contain dogs barking and cats &quot;meowing&quot;. The pipeline is the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Load the data&lt;/li&gt;&#xA;&lt;li&gt;Convert the data to FFT for desired window&lt;/li&gt;&#xA;&lt;li&gt;Apply MFCC filtering&lt;/li&gt;&#xA;&lt;li&gt;Back-transform using DCT&lt;/li&gt;&#xA;&lt;li&gt;Creating a &quot;spectrogram&quot; for the window&lt;/li&gt;&#xA;&lt;li&gt;Training a model?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;if I have .wav files of different sizes, lets say 1 second and 0.8 seconds, I will end up with a different number of windows, if window size is 0.1 seconds then the first file will have 10 windows and the second will have 8 windows, so how can I feed this information to the learning algorithm consistently. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Does the algorithm learn from the entire .wav file or window by window?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If the algorithm learns from each window, will each window have a different prediction value?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;thank you.&lt;/p&gt;&#xA;" OwnerUserId="43714" LastEditorUserId="13023" LastEditDate="2018-04-24T16:07:41.153" LastActivityDate="2020-02-17T22:00:55.030" Title="How to get a feature from sound/audio data learning using machine learning supervised classification?" Tags="|deep-learning|time-series|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30761" PostTypeId="2" ParentId="30752" CreationDate="2018-04-24T14:32:22.710" Score="1" Body="&lt;p&gt;As you say @Alexis, cross-validation is the standard way of selecting a model (or its parameters) in offline learning (especially when previous knowledge in not available). For online training, one usually incorporates some sort of (automatic) compensation for the model parameters that adjust them at the same time that training takes place (the alternative being to cross-validate on some gather data or equivalently to pre-optimize your parameters). Check also &lt;a href=&quot;https://stats.stackexchange.com/questions/64224/regularization-and-feature-scaling-in-online-learning&quot;&gt;this&lt;/a&gt; answer as well as, for example, &lt;a href=&quot;https://arxiv.org/pdf/1406.3816v1.pdf%22&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper (it shows that this issue is a very active research area)!&lt;/p&gt;&#xA;" OwnerUserId="50993" LastActivityDate="2018-04-24T14:32:22.710" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30762" PostTypeId="1" CreationDate="2018-04-24T15:24:15.407" Score="14" ViewCount="14933" Body="&lt;p&gt;I just built this &lt;strong&gt;LSTM neural network&lt;/strong&gt; with Keras &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    import numpy as np&#xA;    import pandas as pd &#xA;    from sklearn import preprocessing&#xA;    from keras.layers.core import Dense, Dropout, Activation&#xA;    from keras.activations import linear&#xA;    from keras.layers.recurrent import LSTM&#xA;    from keras.models import Sequential&#xA;    from matplotlib import pyplot&#xA;&#xA;    #read and prepare data from datafile&#xA;    data_file_name = &quot;DailyDemand.csv&quot;&#xA;    data_csv = pd.read_csv(data_file_name, delimiter = ';',header=None, usecols=[1,2,3,4,5])&#xA;    yt = data_csv[1:]&#xA;    data = yt&#xA;    data.columns = ['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction','Demand']&#xA;    # print (data.head(10))&#xA;    pd.options.display.float_format = '{:,.0f}'.format&#xA;    data = data.dropna ()&#xA;    y=data['Demand'].astype(int)&#xA;    cols=['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction']&#xA;    x=data[cols].astype(int)&#xA;&#xA;    #scaling data&#xA;    scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))&#xA;    x = np.array(x).reshape ((len(x),4 ))&#xA;    x = scaler_x.fit_transform(x)&#xA;    scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))&#xA;    y = np.array(y).reshape ((len(y), 1))&#xA;    y = scaler_y.fit_transform(y)&#xA;    print(&quot;longeur de y&quot;,len(y))&#xA;    # Split train and test data&#xA;    train_end = 80&#xA;    x_train=x[0: train_end ,]&#xA;    x_test=x[train_end +1: ,]&#xA;    y_train=y[0: train_end]&#xA;    y_test=y[train_end +1:] &#xA;    x_train=x_train.reshape(x_train.shape +(1,))&#xA;    x_test=x_test.reshape(x_test.shape + (1,))&#xA;&#xA;    print(&quot;Data well prepared&quot;)&#xA;    print ('x_train shape ', x_train.shape)&#xA;    print ('y_train', y_train.shape)&#xA;&#xA;    #Design the model - LSTM Network&#xA;    seed = 2016&#xA;    np.random.seed(seed)&#xA;    fit1 = Sequential ()&#xA;    fit1.add(LSTM(&#xA;        output_dim = 4,&#xA;        activation='tanh',&#xA;        input_shape =(4, 1)))&#xA;    fit1.add(Dense(output_dim =1))&#xA;    fit1.add(Activation(linear))&#xA;    #rmsprop or sgd&#xA;    batchsize = 1&#xA;    fit1.compile(loss=&quot;mean_squared_error&quot;,optimizer=&quot;rmsprop&quot;)&#xA;    #train the model&#xA;    fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =20, shuffle=True)&#xA;&#xA;    print(fit1.summary ())&#xA;&#xA;    #Model error&#xA;    score_train = fit1.evaluate(x_train ,y_train ,batch_size =batchsize)&#xA;    score_test = fit1.evaluate(x_test , y_test ,batch_size =batchsize)&#xA;    print(&quot;in  train  MSE = &quot;,round(score_train,4))&#xA;    print(&quot;in test  MSE = &quot;,round(score_test ,4))&#xA;&#xA;    #Make prediction&#xA;    pred1=fit1.predict(x_test)&#xA;    pred1 = scaler_y.inverse_transform(np.array(pred1).reshape ((len(pred1), 1)))&#xA;    real_test = scaler_y.inverse_transform(np.array(y_test).reshape ((len(y_test), 1))).astype(int)&#xA;&#xA;    #save prediction&#xA;    testData = pd.DataFrame(real_test)&#xA;    preddData = pd.DataFrame(pred1)&#xA;    dataF = pd.concat([testData,preddData], axis=1)&#xA;    dataF.columns =['Real demand','Predicted Demand']&#xA;    dataF.to_csv('Demandprediction.csv')&#xA;&#xA;    pyplot.plot(pred1, label='Forecast')&#xA;    pyplot.plot(real_test,label='Actual')&#xA;    pyplot.legend()&#xA;    pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then it generates this result: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/la8gj.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/la8gj.png&quot; alt=&quot;Prediction on the test data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After building and training a good model on the historical data, I don't know how I can generate the prediction for future values? For example the demand of the next 10 days. Data are daily.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZTKMA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZTKMA.png&quot; alt=&quot;this is an example of how the data is shaped&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NB: this is an example of how the data is shaped, the green is the label and the yellow one are the features.&lt;br&gt;&#xA;after &lt;code&gt;dropna()&lt;/code&gt; (delete null values) it remains 100 data rows, I've used 80 in the training and the 20 in the test.&lt;/p&gt;&#xA;" OwnerUserId="50622" LastEditorUserId="45264" LastEditDate="2023-05-28T14:17:00.220" LastActivityDate="2023-05-28T14:17:00.220" Title="How to Predict the future values of time horizon with Keras?" Tags="|machine-learning|python|keras|pandas|prediction|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 3.0" />
  <row Id="30763" PostTypeId="2" ParentId="30714" CreationDate="2018-04-24T15:56:59.027" Score="8" Body="&lt;p&gt;I will try to answer this question conceptually and not technically so you get a grasp of the mechanisms in RL.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bootstrapping:&lt;/strong&gt; When you estimate something based on another estimation. In the case of Q-learning for example this is what is happening when you modify your current reward estimation $r_t$ by adding the correction term $\max_a' Q(s',a')$ which is the maximum of the action value over all actions of the next state. Essentially you are estimating your current action value Q by using an estimation of the future Q. Neil has answered that in detail &lt;a href=&quot;https://datascience.stackexchange.com/questions/26938/what-exactly-is-bootstrapping-in-reinforcement-learning&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Sampling:&lt;/strong&gt; Imagine samples as realizations (different values) of a function. Many times it is really difficult to estimate, or come up with an analytical expression, of the underlying process that generated your observations. However sampling values can help you determine lots of characteristics of the underlying generative mechanism and even make assumptions of its properties. Sampling can come in many forms in RL. For example, Q learning is a sampled-based point estimation of the optimal action value function (Bellman equation). In a world that your agent knows nothing you cannot use Dynamic Programming to determine the expected reward form every state. Thus, you need to sample &quot;experience&quot; from your world and estimate the expected reward from any state. In Policy Gradient methods in order to determine the gradient of your expected reward you need to sample trajectories over states and actions from a probability distribution as you cannot determine it analytically.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-04-24T15:56:59.027" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30764" PostTypeId="2" ParentId="30716" CreationDate="2018-04-24T16:20:27.233" Score="0" Body="&lt;pre&gt;&lt;code&gt;# Do Keras' binary cross entropy&#xA;x = Input(shape=(3,))&#xA;x_decoded = Input(shape=(3,))  &#xA;bce = metrics.binary_crossentropy(x,x_decoded)&#xA;sess = K.get_session()&#xA;with sess.as_default():&#xA;    print(bce.eval(feed_dict={x: np.array([[1,1,0]]),&#xA;                              x_decoded: np.array([[0.2393,0.7484,-1.1399]])}))&#xA;&#xA;# Do the same thing in numpy directly&#xA;epsilon = 1e-7&#xA;x = np.array([1,1,0]) &#xA;x_decoded = np.array([0.2393,0.7484,-1.1399])&#xA;&#xA;output = np.clip(x_decoded, epsilon, 1-epsilon)&#xA;output = -x * np.log(output) - (1 - x) * np.log(1 - output)&#xA;print(np.mean(output, axis=-1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0.57328504]&#xA;0.5732850228833389&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So the problem is the &lt;code&gt;mean&lt;/code&gt;. It should be a &lt;code&gt;sum&lt;/code&gt;. And also I guess they do some stuff with &lt;code&gt;epsilon&lt;/code&gt; to prevent weird edge cases that can make it explode. &lt;/p&gt;&#xA;" OwnerUserId="51024" LastActivityDate="2018-04-24T16:20:27.233" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30765" PostTypeId="1" AcceptedAnswerId="30769" CreationDate="2018-04-24T17:19:39.350" Score="0" ViewCount="195" Body="&lt;p&gt;I'm new to regression and we are doing a very simple exercise in a course. I'm taking to get a basic understanding of GD and SGD for a linear regression.&lt;/p&gt;&#xA;&lt;p&gt;From my understanding, the only difference between GD and SGD is that instead of performing the algorithm on dataset size m as is processing in GD, SGD performs the operation on subsets of m.&lt;/p&gt;&#xA;&lt;p&gt;My question is, for SGD does one simply perform the algorithm on the mini-batch, or is there some sort of summation of the results to come out with a final answer? Apologies if I'm not asking in the correct terms, I'm newer to some of the mathematical concepts involved.&lt;/p&gt;&#xA;" OwnerUserId="51083" LastEditorUserId="104237" LastEditDate="2020-09-14T12:10:32.637" LastActivityDate="2020-09-14T12:10:32.637" Title="Stochastic Gradient Descent Batching" Tags="|machine-learning|deep-learning|linear-regression|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30766" PostTypeId="1" CreationDate="2018-04-24T17:22:03.303" Score="1" ViewCount="182" Body="&lt;p&gt;I have read many blog articles, research papers and watched many youtube videos, but it seems it is hard to find why experience replay is efficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that the experience replay stores (state, action, reward, newstate) and sample minibatch and update weights, however, what I don't understand is how come the bellman equation can be the target value and how the learning can be done with supervised learning the value of (reward + gamma * (maximum next action)) being target?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great if you can suggest any articles or videos? &#xA;or explain it.&lt;/p&gt;&#xA;" OwnerUserId="51085" LastActivityDate="2018-04-24T17:22:03.303" Title="Experience Replay Explain" Tags="|deep-learning|supervised-learning|q-learning|dqn|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30767" PostTypeId="1" AcceptedAnswerId="30822" CreationDate="2018-04-24T17:26:56.273" Score="1" ViewCount="810" Body="&lt;p&gt;I'm trying to train the neural network to predict the movement of a particular security on the market.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I teach on historical data collected for the year. At the entrance of the neural network candlesticks are served: close price and value&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before submitting, these data are normalized separately for each dataset. This happens with the z Score algorithm. Then the question immediately arises... output can not be obtained in the limit [0;1] or [-1;1] and can reach up to 10 or more (in both directions). Is that okay?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All sets are first shuffled, and then divided into two parts:: train and test (80:20)&#xA;The output is a class: the price will go up or down.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For train, the error is reduced to zero and the accuracy to 100%. For test, the error reaches a minimum of 0.28 and the accuracy reaches 90%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to test this neural network to predict the next two months, which were not used in this neural network. They went through exactly the same normalization. However, the accuracy of this forecast is 0.5289256204750912. And the error - 1.8223290671986982&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know I'm doing something wrong, but I don't know what... I hope someone can help me figure this out.&#xA;PS: ZScore was used To the usual min, max. However, this particular success was not observed.&lt;/p&gt;&#xA;" OwnerUserId="51084" LastEditorUserId="51084" LastEditDate="2018-04-25T15:03:31.227" LastActivityDate="2018-04-25T15:03:31.227" Title="Very low accuracy of new data compared to validation data" Tags="|machine-learning|classification|accuracy|market-basket-analysis|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30768" PostTypeId="2" ParentId="30757" CreationDate="2018-04-24T17:43:24.397" Score="2" Body="&lt;p&gt;We can answer this overarching question by exploring a couple sub-questions:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;What are the properties of popular averaging formulae?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Geometric:&lt;/strong&gt;&#xA;$ \space $&#xA;$ \space $&#xA;$ \begin{equation}&#xA;\bigg(\displaystyle \prod^N_{i=1} x_{i}\bigg)^{\frac{1}{N}}&#xA;\end{equation} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see that any set that contains a zero has a geometric mean of zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, processing negative numbers is awkward as they can become imaginary.  Even if negative numbers do not result in an imaginary mean (e.g. the set has an odd number of elements), the result is often not what we expect an &quot;average&quot; to be.  For instance, given a set, let's take some averages:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \{-1,2,4\} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arithmetic mean: $ 1.\overline{6} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Geometric mean: $ -2 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The geometric mean is clearly not what we would expect for an &quot;average&quot;.  This is because the geometric mean is primarily used for understanding the multiplicative differences in a set, which is useful for averaging data on differing scales.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we only consider the absolute value of the elements.  Then, given a set below, we still get undesirable results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \{2^0,2^1,...2^{10}\} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arithmetic mean: $ 186.\overline{09} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Geometric mean: $ 32 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this example we can see that where the arithmetic mean is linear, the geometric mean is logarithmic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Harmonic:&lt;/strong&gt;&#xA;$ \space $&#xA;$ \space $&#xA;$ \begin{equation}&#xA;N \cdot \bigg({\displaystyle \sum^N_{i=1} x^{-1}_{i}}\bigg)^{-1}&#xA;\end{equation} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like the geometric mean, the harmonic mean of a set containing zero is uninformative.  In fact, it is undefined.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A more interesting property of the harmonic mean is that it discriminates against outliers.  For instance, given a set, let's again take some averages:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \{ 1e^{-5}, 1e^{-5}, 1e^{10}\}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Arithmetic mean: $ 3.\overline{3}e^9 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Harmonic mean: $ 1.5e^{-5} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An easier way of thinking about the averages is like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where arithmetic is linear, geometric is logarithmic, and harmonic is reciprocal.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Should every learner have an equal representation in the final vote?&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If you are using the arithmetic mean, the weak learners are treated equally.  This is generally desired as you do not know which learner is closer to the correct answer.  Using the geometric or harmonic mean would imply that you want to &quot;weight&quot; the set of weak learner outputs according to their respective additive structures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a side-note, you can (and probably should) use machine learning algorithms, such as a MLP or SVM, to combine the outputs of your ensemble, which would probably yield superior results to simple averaging.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-24T17:57:49.703" LastActivityDate="2018-04-24T17:57:49.703" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30769" PostTypeId="2" ParentId="30765" CreationDate="2018-04-24T17:45:05.600" Score="1" Body="&lt;p&gt;In SGD you just feed an example to your model, compute the gradient of the loss function of that example and update the weights according to the gradient of the loss of that example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In mini-batch gradient descent you feed a batch to your model, compute the gradient of the loss of that batch and update the weights according to the gradient of the loss of that batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, SGD is mini-batch gradient descent with batch size equal to 1.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-24T17:45:05.600" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30770" PostTypeId="1" AcceptedAnswerId="30784" CreationDate="2018-04-24T17:45:41.023" Score="3" ViewCount="1363" Body="&lt;p&gt;I'm working with this dataset &lt;a href=&quot;https://www.kaggle.com/c/sf-crime&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/sf-crime&lt;/a&gt; to predict the crime incident using keras. I've encoded the category with pd.get_dummies and then use it as the validation data. At first I try to use categorical_crossentropy and adam for the loss function but result returns with very bad acc which nearly 0.3 and about 2.7 of loss. So I try to experiment with different entropy by using binary_crossentropy. The result seem to be exaggerate the acc is higher than 0.97 and the loss is less than 0.1 at first epoch. I also provide 0.3 of drop out for each layer. So which crossentropy is accurate or better use for this scenario and which kind of this classification is multi-label or multi-class?&lt;/p&gt;&#xA;" OwnerUserId="51082" LastActivityDate="2018-04-24T22:56:08.480" Title="Which Loss cross-entropy do I've to use?" Tags="|deep-learning|keras|multiclass-classification|multilabel-classification|accuracy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30772" PostTypeId="1" CreationDate="2018-04-24T18:28:32.307" Score="2" ViewCount="201" Body="&lt;p&gt;Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have this input text, and we would like to do Named Entity Extraction:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;GOP Sen. Rand Paul was assaulted in his home in Bowling Green,&#xA;  Kentucky, on Friday, according to Kentucky State Police. State&#xA;  troopers responded to a call to the senator\'s residence at 3:21 p.m.&#xA;  Friday. Police arrested a man named Rene Albert Boucher, who they&#xA;  allege &quot;intentionally assaulted&quot; Paul, causing him &quot;minor injury&quot;.&#xA;  Boucher, 59, of Bowling Green was charged with one count of&#xA;  fourth-degree assault. As of Saturday afternoon, he was being held in&#xA;  the Warren County Regional Jail on a $5,000 bond.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Paragraph level: we can take it as one record and each token is marked by the entity label. Model has &lt;strong&gt;ONE&lt;/strong&gt; record with &lt;strong&gt;LONG&lt;/strong&gt; sequence.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Sentence level: we first intelligently split the paragraph into 5 correct sentences, and each token in each sentence is marked by the entity label. Model has &lt;strong&gt;FIVE&lt;/strong&gt; records with &lt;strong&gt;shorter&lt;/strong&gt; sequences:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0) GOP Sen. Rand Paul was assaulted in his home in Bowling Green,&#xA;  Kentucky, on Friday, according to Kentucky State Police. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;1) State&#xA;  troopers responded to a call to the senator's residence at 3:21 p.m.&#xA;  Friday.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) Police arrested a man named Rene Albert Boucher, who they allege&#xA;  &quot;intentionally assaulted&quot; Paul, causing him &quot;minor injury&quot;.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;3) Boucher, 59, of Bowling Green was charged with one count of&#xA;  fourth-degree assault.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;4) As of Saturday afternoon, he was being held in the Warren County&#xA;  Regional Jail on a $5,000 bond.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Which one gives the NER modeling a better NER performance?&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to think sentence level is better, however, shouldn't LSTM memory cells be trained to remember or forget states automatically if given long paragraphs? Especially when the sentence splitting can also potentially make mistakes, for instance:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) State troopers responded to a call to the senator's residence at 3:21 p.m. Friday.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;could have been &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1) State troopers responded to a call to the senator's residence at 3:21 p.m. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;2) Friday.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="51089" LastActivityDate="2018-04-24T18:28:32.307" Title="NER at sentence level or document level?" Tags="|nlp|lstm|word-embeddings|named-entity-recognition|stanford-nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30773" PostTypeId="1" AcceptedAnswerId="30777" CreationDate="2018-04-24T18:32:49.727" Score="1" ViewCount="109" Body="&lt;p&gt;I am very new to data science / ML and I have what I think is a very basic question - when to 'clean' the data?&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Do I clean data before using it to train a classifier (a binary classifier in my experiments)?&lt;/li&gt;&#xA;&lt;li&gt;Do I clean data that I try to classify using this classifer?&lt;/li&gt;&#xA;&lt;li&gt;Both?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The data in my case is just a series of Tweets.&lt;/p&gt;&#xA;" OwnerUserId="51090" LastActivityDate="2018-04-24T20:01:01.673" Title="When to clean data?" Tags="|data-cleaning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30774" PostTypeId="2" ParentId="30773" CreationDate="2018-04-24T19:17:06.530" Score="1" Body="&lt;p&gt;You would want to clean your data before training a classifier. To give you a rough and abstract look on ML, think of your classifiers as a giant mathematical matrix. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You want to use the classifiers to give you information on the data you have. But in order to do that, you would want to clean, parse or even encode your data to a format that the ML would be able to understand and one where you can gain the most knowledge from the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't think of any case where you would want to clean data with a classifier. You could use &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction&quot; rel=&quot;nofollow noreferrer&quot;&gt;dimensionality reduction&lt;/a&gt; but that is not exactly data cleaning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that answers your question&lt;/p&gt;&#xA;" OwnerUserId="49985" LastActivityDate="2018-04-24T19:17:06.530" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30775" PostTypeId="2" ParentId="12679" CreationDate="2018-04-24T19:42:35.720" Score="1" Body="&lt;p&gt;The easiest thing would be to download a Docker Image. I recently created a Docker Image with CUDA 9.0, cudnn, tensorlflow and keras. I have a gtx 1070 and it worked without any issue.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://hub.docker.com/r/deejay217/cuda9_python3_tensorflow17/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://hub.docker.com/r/deejay217/cuda9_python3_tensorflow17/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51091" LastActivityDate="2018-04-24T19:42:35.720" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30776" PostTypeId="2" ParentId="30773" CreationDate="2018-04-24T19:47:23.973" Score="1" Body="&lt;p&gt;Cleaning is usually done in the pre-processing or data preparation phase of data mining.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore you might want to clean before you train or use your classifier. The cleaning algorithm should be the same for training and applying the classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would apply your cleaning algorithm to only one of the training and test datasets, the prediction accuracy must not necessarily be worse than if applied to both. This may be the case when your classifier does not depend on cleaned features. In this case you would probably might want to check if these features are relevant for the classification at all. &lt;/p&gt;&#xA;" OwnerUserId="16213" LastActivityDate="2018-04-24T19:47:23.973" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30777" PostTypeId="2" ParentId="30773" CreationDate="2018-04-24T20:01:01.673" Score="1" Body="&lt;p&gt;Data Cleaning or Data Munging as it is referred in most cases, is the process of transforming the data from the raw form that they exist after their collection into another format with the intent of making it more appropriate for their future process e.g. training models etc.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This process is taking place at the beginning of the whole procedure and before the training and validation of the models. In text mining problems, you have also to treat the punctuation marks, remove the stopwords (it depends on the data representation that you will choose, for unigrams it is fine, but for bigrams it is not recommended at all) and also do the stemming or lemmatization processes.  &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-04-24T20:01:01.673" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30778" PostTypeId="1" AcceptedAnswerId="75807" CreationDate="2018-04-24T20:19:22.623" Score="9" ViewCount="1254" Body="&lt;p&gt;I would like to create a content recommendation system based on binary click data that also takes views into account. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What content a user has been exposed to, and therefore has the chance to click on, is currently biased by a rule based system that is not always documented. I do have view data (if a user saw the content on their screen, regardless of whether it was clicked.), and am wondering how to take this into account with a traditional matrix factorization recommendation system such as &lt;a href=&quot;https://medium.com/radon-dev/item-item-collaborative-filtering-with-binary-or-unary-data-e8f0b465b2c3&quot; rel=&quot;noreferrer&quot;&gt;this&lt;/a&gt; item-item approach, or if there are other other better options.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions for implementation in Python are a bonus!&lt;/p&gt;&#xA;" OwnerUserId="42852" LastEditorUserId="42852" LastEditDate="2018-04-25T14:04:17.277" LastActivityDate="2020-06-16T10:59:30.823" Title="Binary (Unary) Recommendation System with Biased Views" Tags="|python|recommender-system|binary|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30779" PostTypeId="1" AcceptedAnswerId="30793" CreationDate="2018-04-24T20:34:21.357" Score="1" ViewCount="2876" Body="&lt;p&gt;Suppose we have a list of products categorized into 10 categories. We also have customers order details such as&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;order_id, product_name, quantity, order_date&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We want to know for a particular &lt;code&gt;month&lt;/code&gt; what are the most probable &lt;code&gt;categories&lt;/code&gt; from which customer might buy products. How can we approach this problem, Any suggestions.&lt;/p&gt;&#xA;" OwnerUserId="50603" LastActivityDate="2018-04-25T03:19:21.313" Title="Predicting customers purchase" Tags="|machine-learning|data-mining|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30781" PostTypeId="2" ParentId="30779" CreationDate="2018-04-24T21:40:40.710" Score="1" Body="&lt;p&gt;From what you are describing, there is an independent variable (month, you can extract this information from order date) and the response variable which is the probability of purchasing a product of a particular category. You can use a simple Bayesian (Naive) algorithm for fitting a model and making a prediction for a specified month. You can not do much more than this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But if you have available historical data for specific customer then you can build a recommendation system trying to make a profile for each customer and predict what is most probable to buy in a particular month. (Similarities among user or similarities among items) &lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="50292" LastEditDate="2018-04-25T02:53:42.397" LastActivityDate="2018-04-25T02:53:42.397" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30782" PostTypeId="2" ParentId="30370" CreationDate="2018-04-24T21:56:36.073" Score="0" Body="&lt;p&gt;You can try to measure the similarity of the products that a user has bought so far with other user's purhcases (user based recommendation) or you can try perform associate rule among the items that the user has bought and other items (item based recommendation). You can also perform some clustering techiniques to find group of similar items or users.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another approach could be the following, if you have available the information for a user's purchase then you can try to predict the user's next purchase. This approach can be a Markov Model. In a Markov model the most recent state is predicted based on a fixed number of the previous states, and this fixed number of previous states is called the order of the Markov model. At your case, each state could be a different purchase. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-04-24T21:56:36.073" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30783" PostTypeId="1" CreationDate="2018-04-24T22:06:47.727" Score="1" ViewCount="102" Body="&lt;p&gt;I came across the below paragraphs, which I believe are the answers to the question &lt;em&gt;Why infinite sampling is not realistic assumption in most real applications&lt;/em&gt;. Still i dont get the below explanation ?. When we draw more samples from the environment, MC brings the approximate value function close to the true value function isn't it ? Then why infinite sampling is not considered as a realistic assumption.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We made two unlikely assumptions above in order to easily obtain this guarantee of convergence for the Monte Carlo method. One was that the episodes have exploring starts, and the other was that policy evaluation could be done with an infinite number of episodes. To obtain a practical algorithm we will have to remove both assumptions. We postpone consideration of the first assumption until later in this chapter.&lt;/p&gt;&#xA;&lt;p&gt;The second approach to avoiding the infinite number of episodes nominally required for policy evaluation is to forgo trying to complete policy evaluation before returning to policy improvement. On each evaluation step we move the value function toward , but we do not expect to actually get close except over many steps. We used this idea when we first introduced the idea of GPI in Section 4.6. One extreme form of the idea is value iteration, in which only one iteration of iterative policy evaluation is performed between each step of policy improvement. The in-place version of value iteration is even more extreme; there we alternate between improvement and evaluation steps for single states.&lt;/p&gt;&#xA;&lt;p&gt;For now we focus on the assumption that policy evaluation operates on an infinite number of episodes. This assumption is relatively easy to remove. In fact, the same issue arises even in classical DP methods such as iterative policy evaluation, which also converge only asymptotically to the true value function. In both DP and Monte Carlo cases there are two ways to solve the problem. One is to hold firm to the idea of approximating in each policy evaluation. Measurements and assumptions are made to obtain bounds on the magnitude and probability of error in the estimates, and then sufficient steps are taken during each policy evaluation to assure that these bounds are sufficiently small. This approach can probably be made completely satisfactory in the sense of guaranteeing correct convergence up to some level of approximation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2893" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2023-01-01T12:07:40.977" Title="Why infinite sampling is not realisitc assumpition in most real applications" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30784" PostTypeId="2" ParentId="30770" CreationDate="2018-04-24T22:56:08.480" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;which kind of this classification is multi-label or multi-class?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If we take a look at the &lt;a href=&quot;https://www.kaggle.com/c/sf-crime/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;data&lt;/a&gt; panel at the competition's page in kaggle:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Category - category of the crime incident (only in train.csv). This is the target variable you are going to predict.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The target we are trying to predict is the category of the crime (which can take multiple values) so this answers your question: the problem is &lt;strong&gt;multi-class&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So which crossentropy is accurate or better use for this scenario?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;There are three main types of cross-entopy in keras:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://keras.io/losses/#categorical_crossentropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;categorical cross-entropy&lt;/a&gt; is used for &lt;strong&gt;multi-class classification&lt;/strong&gt;, where the target variable is &lt;strong&gt;one-hot encoded&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://keras.io/losses/#sparse_categorical_crossentropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;sparse categorical cross-entropy&lt;/a&gt; is used for &lt;strong&gt;multi-class classification&lt;/strong&gt;, where the target variable is &lt;strong&gt;not&lt;/strong&gt; one-hot encoded.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://keras.io/losses/#binary_crossentropy&quot; rel=&quot;nofollow noreferrer&quot;&gt;binary cross-entropy&lt;/a&gt; is used for &lt;strong&gt;binary classification&lt;/strong&gt; problems. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;From your question I assume that you think the different types of crossentropy are interchangeable. That is not the case. You should select the cross-entropy based on how your problem and how you encoded your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, since you have a multi-class problem and you state that you have used &lt;code&gt;pandas.get_dummies()&lt;/code&gt;, which ont-hot encodes your data you should use &lt;strong&gt;categorical cross-entropy&lt;/strong&gt; and nothing else!&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-24T22:56:08.480" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30785" PostTypeId="2" ParentId="22313" CreationDate="2018-04-24T22:59:49.830" Score="1" Body="&lt;p&gt;The same thing was happening to me with a deep Q Network on the cart-pole  problem. Having a &quot;memory&quot; with past (S,A,R,S) sequences and sampling it to form mini batches with the new observations helped a lot to reduce catastrophic forgetting. Reducing the step size once the agent has improved a certain amount also helped.&lt;/p&gt;&#xA;" OwnerUserId="51095" LastActivityDate="2018-04-24T22:59:49.830" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30786" PostTypeId="1" AcceptedAnswerId="30791" CreationDate="2018-04-24T23:57:03.723" Score="9" ViewCount="13303" Body="&lt;p&gt;I am attempting to implement a basic &lt;code&gt;Stochastic Gradient Descent&lt;/code&gt; algorithm for a &lt;code&gt;2-d linear regression&lt;/code&gt; in Python. I was given some boilerplate code for &lt;code&gt;vanilla GD&lt;/code&gt;, and I have attempted to convert it to work for &lt;code&gt;SGD&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Specifically -- I am a little unsure as to if I correctly implemented the loss function and partial derivatives, since I am new to regressions in general.&lt;/p&gt;&#xA;&lt;p&gt;I do see that the errors tend to &amp;quot;zig zag&amp;quot; as expected. Does the following look like a correct implementation or have I made any mistakes?&lt;/p&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;    #sample data&#xA;    data  = [(1,1),(2,3),(4,3),(3,2),(5,5)]    &#xA;    &#xA;    def compute_error_for_line_given_points(b, m, points):&#xA;        totalError = 0&#xA;        x = points[0]&#xA;        y = points[1]&#xA;        return float(totalError + (y - (m * x + b)) ** 2)&#xA;    &#xA;    def step_gradient(b_current, m_current, points, learningRate):&#xA;        N = float(1)&#xA;        for i in range(0, 1):&#xA;            x = points[0]&#xA;            y = points[1]&#xA;            b_gradient = -(2/N) * (y - ((m_current * x) + b_current)) #this is the part I am unsure&#xA;            m_gradient = -(2/N) * x * (y - ((m_current * x) + b_current)) #here as well&#xA;        new_b = b_current - (learningRate * b_gradient)&#xA;        new_m = m_current - (learningRate * m_gradient)&#xA;        return [new_b, new_m]&#xA;    &#xA;    err_log = []&#xA;    coef_log = []&#xA;    b = 0 #initial intercept&#xA;    m = 0 #initial slope&#xA;    &#xA;    iterations = 4&#xA;    for i in range(iterations): #epochs&#xA;        for point in data: #one point at a time for SGD&#xA;            err = compute_error_for_line_given_points(b,m, point)&#xA;            err_log.append(err)&#xA;            b,m = step_gradient(b,m,point,.01)&#xA;            coef_log.append((b,m))&#xA;&lt;span class=&quot;math-container&quot;&gt;```&lt;/span&gt;  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51083" LastEditorUserId="104237" LastEditDate="2020-09-14T09:12:35.727" LastActivityDate="2020-09-14T09:12:35.727" Title="Implementation of Stochastic Gradient Descent in Python" Tags="|linear-regression|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30787" PostTypeId="1" AcceptedAnswerId="30801" CreationDate="2018-04-25T00:02:00.713" Score="3" ViewCount="66" Body="&lt;p&gt;I am running &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/htA0l/gradient-checking&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient Checking&lt;/a&gt;, to spot any discrepancy between my mathematically-computed gradient and the actual sampled gradient - to reassure my backprop was implemented correctly. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When computing such a discrepancy, can I sum-up squares of differences, then take their average? I could then use this average as my estimate of how correctly the network computes the gradient:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{1}{m}\sum_{i=0}^{i=m}(g_i-n_i)^2$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or even:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\sqrt{\sum_{i=0}^{i=m}(g_i-n_i)^2}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$g$&lt;/span&gt; is a gradient from backpropagation, and &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; is gradient from gradient checking.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;However, &lt;a href=&quot;https://www.coursera.org/learn/deep-neural-network/lecture/htA0l/gradient-checking&quot; rel=&quot;nofollow noreferrer&quot;&gt;Andrew Ng instead recommends:&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\frac{\vert \vert (g-n) \vert \vert _2 }{&#xA;\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;span class=&quot;math-container&quot;&gt;$\vert \vert . \vert \vert _2$&lt;/span&gt; is the length of the vector.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Another post post also recommends an slightly different approach: &lt;a href=&quot;https://stats.stackexchange.com/a/188724/187816&quot;&gt;https://stats.stackexchange.com/a/188724/187816&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Why would their approaches be better than mine?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-09-28T09:25:18.110" LastActivityDate="2018-09-28T09:25:18.110" Title="Grad Checking, verify by average?" Tags="|machine-learning|optimization|gradient-descent|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30788" PostTypeId="2" ParentId="30783" CreationDate="2018-04-25T00:18:25.547" Score="1" Body="&lt;p&gt;It is not a realistic assumption because you don't have infinite time or decimal precision to find the absolutely correct value function, but you don't need that anyway since a rough estimate of it will be enough to improve the policy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your question is why you can't let the agent learn indefinitely in real applications, I'm guessing it is because it may be potentially expensive or dangerous to let it explore randomly in a real scenario, so you want to deploy it with an optimal or near-optimal, deterministic, policy.&lt;/p&gt;&#xA;" OwnerUserId="51096" LastActivityDate="2018-04-25T00:18:25.547" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30789" PostTypeId="1" AcceptedAnswerId="30796" CreationDate="2018-04-25T01:35:11.637" Score="1" ViewCount="1431" Body="&lt;p&gt;In a basic linear regression, can I use the weights of each explanatory variable to describe their relative effects on the predicted value? If parameter A has a weight of 100, and parameter B has a weight of 10, can I say that parameter A has 10 times more effect on the outcome than parameter B?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example: From a weather data set, I used two parameters, humidity and pressure, to make a prediction on temperature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I plot humidity vs. temperature, there's a very clear inverse relationship between the two. That is, the scatter plot tends down and to the right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I plot pressure vs. temperature, there's no relationship at all. The pressure in this report has very little variance, and the scatter plot is nearly vertical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on those two plots, I intuit that changes in humidity have an effect on the high temperature, while changes in pressure have almost no effect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I built a linear regression model (gradient descent, by hand, in Octave). Humidity had a weight of -.32386. Pressure had a weight of -.02219. Humidity's weight was 14.6 times larger than pressure's weight. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on that, can I state that parameter Humidity has nearly 15 times more effect on Temperature than parameter Pressure? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Forgive what must seem like a very amateur question. If you can give a relatively simple answer and point me toward resources where I can learn more, I'd really appreciate it. Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="51099" LastEditorUserId="50727" LastEditDate="2018-05-28T15:34:24.797" LastActivityDate="2018-05-28T15:34:24.797" Title="Linear Regression. Interpret weights as an effect on a prediction?" Tags="|machine-learning|statistics|linear-regression|" AnswerCount="2" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30790" PostTypeId="1" CreationDate="2018-04-25T02:13:26.050" Score="2" ViewCount="140" Body="&lt;p&gt;During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When done, should we return the minibatch back to the memory bank, or should we throw it out?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing the former seems to grow the memory indefinitely, and might result in some state being sampled a few extra times in the future, if the state is lucky and ends up in future mini batches.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-04-26T02:42:36.280" LastActivityDate="2019-05-17T09:03:23.007" Title="Experience Replay, must return minibatch back to Memory Bank?" Tags="|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30791" PostTypeId="2" ParentId="30786" CreationDate="2018-04-25T02:30:05.490" Score="6" Body="&lt;p&gt;There is only one small difference between gradient descent and stochastic gradient descent. Gradient descent calculates the gradient based on the loss function calculated across all training instances, whereas stochastic gradient descent calculates the gradient based on the loss in batches. Both of these techniques are used to find optimal parameters for a model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us try to implement SGD on this 2D dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XA37k.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XA37k.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The algorithm&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The dataset has 2 features, however we will want to add a bias term so we append a column of ones to the end of the data matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;shape = x.shape &#xA;x = np.insert(x, 0, 1, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we initialize our weights, there are many strategies to do this. For simplicity I will set them all to 1 however setting the initial weights randomly is probably better in order to be able to use multiple restarts.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w = np.ones((shape[1]+1,))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Our initial line looks like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/W1SdS.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W1SdS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we will iteratively update the weights of the model if it mistakenly classifies an example.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for ix, i in enumerate(x):&#xA;   pred = np.dot(i,w)&#xA;   if pred &amp;gt; 0: pred = 1&#xA;   elif pred &amp;lt; 0: pred = -1&#xA;   if pred != y[ix]:&#xA;      w = w - learning_rate * pred * i&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This line is the weight update &lt;code&gt;w = w - learning_rate * pred * i&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see that doing this process continuously will lead to convergence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 10 epochs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XPYUb.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XPYUb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 20 epochs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UjRpM.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UjRpM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 50 epochs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/COsSh.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/COsSh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After 100 epochs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AzYB2.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AzYB2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And finally,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/S6M56.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/S6M56.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;The code&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The dataset for this code can be found &lt;a href=&quot;https://github.com/kelhallaoui/tutorials/blob/master/Machine%20Learning/Linear%20Models/Perceptron%20Learning/perceptron.csv&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The function which will train the weights takes in the feature matrix $x$ and the targets $y$. It returns the trained weights $w$ and a list of historical weights encountered throughout the training process.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;%matplotlib inline&#xA;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;&#xA;def get_weights(x, y, verbose = 0):&#xA;    shape = x.shape&#xA;    x = np.insert(x, 0, 1, axis=1)&#xA;    w = np.ones((shape[1]+1,))&#xA;    weights = []&#xA;&#xA;    learning_rate = 10&#xA;    iteration = 0&#xA;    loss = None&#xA;    while iteration &amp;lt;= 1000 and loss != 0:&#xA;        for ix, i in enumerate(x):&#xA;            pred = np.dot(i,w)&#xA;            if pred &amp;gt; 0: pred = 1&#xA;            elif pred &amp;lt; 0: pred = -1&#xA;            if pred != y[ix]:&#xA;                w = w - learning_rate * pred * i&#xA;            weights.append(w)    &#xA;            if verbose == 1:&#xA;                print('X_i = ', i, '    y = ', y[ix])&#xA;                print('Pred: ', pred )&#xA;                print('Weights', w)&#xA;                print('------------------------------------------')&#xA;&#xA;&#xA;        loss = np.dot(x, w)&#xA;        loss[loss&amp;lt;0] = -1&#xA;        loss[loss&amp;gt;0] = 1&#xA;        loss = np.sum(loss - y )&#xA;&#xA;        if verbose == 1:&#xA;            print('------------------------------------------')&#xA;            print(np.sum(loss - y ))&#xA;            print('------------------------------------------')&#xA;        if iteration%10 == 0: learning_rate = learning_rate / 2&#xA;        iteration += 1    &#xA;    print('Weights: ', w)&#xA;    print('Loss: ', loss)&#xA;    return w, weights&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;We will apply this SGD to our data in &lt;strong&gt;perceptron.csv&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = np.loadtxt(&quot;perceptron.csv&quot;, delimiter = ',')&#xA;x = df[:,0:-1]&#xA;y = df[:,-1]&#xA;&#xA;print('Dataset')&#xA;print(df, '\n')&#xA;&#xA;w, all_weights = get_weights(x, y)&#xA;x = np.insert(x, 0, 1, axis=1)&#xA;&#xA;pred = np.dot(x, w)&#xA;pred[pred &amp;gt; 0] =  1&#xA;pred[pred &amp;lt; 0] = -1&#xA;print('Predictions', pred)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's plot the decision boundary&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)&#xA;x2 = np.zeros((2,))&#xA;for ix, i in enumerate(x1):&#xA;    x2[ix] = (-w[0] - w[1]*i) / w[2]&#xA;&#xA;plt.scatter(x[y&amp;gt;0][:,1], x[y&amp;gt;0][:,2], marker = 'x')&#xA;plt.scatter(x[y&amp;lt;0][:,1], x[y&amp;lt;0][:,2], marker = 'o')&#xA;plt.plot(x1,x2)&#xA;plt.title('Perceptron Seperator', fontsize=20)&#xA;plt.xlabel('Feature 1 ($x_1$)', fontsize=16)&#xA;plt.ylabel('Feature 2 ($x_2$)', fontsize=16)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To see the training process you can print the weights as they changed through the epochs.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for ix, w in enumerate(all_weights):&#xA;    if ix % 10 == 0:&#xA;        print('Weights:', w)&#xA;        x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)&#xA;        x2 = np.zeros((2,))&#xA;        for ix, i in enumerate(x1):&#xA;            x2[ix] = (-w[0] - w[1]*i) / w[2]&#xA;        print('$0 = ' + str(-w[0]) + ' - ' + str(w[1]) + 'x_1'+ ' - ' + str(w[2]) + 'x_2$')&#xA;&#xA;        plt.scatter(x[y&amp;gt;0][:,1], x[y&amp;gt;0][:,2], marker = 'x')&#xA;        plt.scatter(x[y&amp;lt;0][:,1], x[y&amp;lt;0][:,2], marker = 'o')&#xA;        plt.plot(x1,x2)&#xA;        plt.title('Perceptron Seperator', fontsize=20)&#xA;        plt.xlabel('Feature 1 ($x_1$)', fontsize=16)&#xA;        plt.ylabel('Feature 2 ($x_2$)', fontsize=16)&#xA;        plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-04-25T02:30:05.490" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30793" PostTypeId="2" ParentId="30779" CreationDate="2018-04-25T03:19:21.313" Score="2" Body="&lt;p&gt;I recommend using LSTM RNN or CNN algorithm to pick up the most popular product of on-going month based on the past purchasing history. I have created a working product ranking model for online shopping site &lt;a href=&quot;https://landsend.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;lands'end&lt;/a&gt; in the United States.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to archieve the goal of this solution, you need to follow several steps.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;collecting the dataset&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think you already gathered the dataset for this part.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Processing of the dataset for supervised learning.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Cleaning up the dataset - please remove rows that contains NaN columns&lt;/li&gt;&#xA;&lt;li&gt;mapping the products into eyes - you need to map the categories and products into row matrix with the element of 1 and 0.&lt;/li&gt;&#xA;&lt;li&gt;Normalization - you need to rescale all real numbers into [0, 1]&lt;/li&gt;&#xA;&lt;li&gt;split the dataset into the training, valid and test data.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Build Deep learning model like LSTM RNN&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;select the deep learning framework. I recommend &lt;a href=&quot;https://www.tensorflow.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow&lt;/a&gt;, &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;set the dimension of model. i.e. the number of layers, the number of neurons per layer.&lt;/li&gt;&#xA;&lt;li&gt;optimizer, metrics&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Training the model and testing&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Get the predicted ranking of the category by monthly.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Protip: the most important thing to get the better result is which column would be set the output for ranking. You should sum up the total purchasing count per product or per category by month or week.&lt;/p&gt;&#xA;" OwnerUserId="50292" LastActivityDate="2018-04-25T03:19:21.313" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30794" PostTypeId="1" CreationDate="2018-04-25T04:47:49.473" Score="1" ViewCount="170" Body="&lt;p&gt;I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow maximize the separation between the regular and irregular data points. I also have seen this in my codes. However, someone has asked me to prove it! by a simple detailed example (e.g, a simple 2*2 matrix)? Can anyone help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance. &lt;/p&gt;&#xA;" OwnerUserId="12345" LastEditorUserId="12345" LastEditDate="2018-04-25T04:55:03.520" LastActivityDate="2018-04-25T10:10:39.960" Title="Principal Component Analysis and abnormal data" Tags="|machine-learning|dimensionality-reduction|pca|anomaly-detection|outlier|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30796" PostTypeId="2" ParentId="30789" CreationDate="2018-04-25T06:12:19.133" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;In a basic linear regression, can I use the weights of each&#xA;  explanatory variable to describe their relative effects on the&#xA;  predicted value? If parameter A has a weight of 100, and parameter B&#xA;  has a weight of 10, can I say that parameter A has 10 times more&#xA;  effect on the outcome than parameter B?&#xA;  Blockquote&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Somewhat yes. Precisely it means if A changes by one unit the outcome will change 100 times, on average (while B is frozen because it is a multiple linear regression -&gt; you have more than one independent variable).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are tons of examples and blogs explaining it, see for example &lt;a href=&quot;https://www.theanalysisfactor.com/interpreting-regression-coefficients/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; .&#xA;Also you can analyze your regression model more by doing some statistical tests like identifying how variables are correlated using Variance Inflation Factors and estimate the statistical significance (p-value, F-test), here is a very nice &lt;a href=&quot;https://onlinecourses.science.psu.edu/stat501/node/347&quot; rel=&quot;nofollow noreferrer&quot;&gt;intro&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;An example: From a weather data set, I used two parameters, humidity&#xA;  and pressure, to make a prediction on temperature.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I do not see any problem with your analysis. As you said you see an inverse correlation between humidity vs. temperature in your plot as well as via your -.32386 weight. While the weight of -.02219 for pressure is nearly zero and negligible and obviously you won't see such relationship between pressure-temperature.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-04-25T06:12:19.133" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30799" PostTypeId="1" AcceptedAnswerId="30800" CreationDate="2018-04-25T06:58:24.773" Score="1" ViewCount="302" Body="&lt;p&gt;I have the below data, I am looking for a effective visualization or graphical method (in Python) to understand how the err is distributed with respect to min and max.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Data_X_err.head()&#xA;Out[137]: &#xA;     min    max   err&#xA;0  35435  35933  1.40&#xA;1  35155  36382  3.43&#xA;2  35305  36042  2.07&#xA;3  35216  36225  2.82&#xA;4  35196  36259  2.98&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="40853" LastEditDate="2018-04-25T07:06:42.190" LastActivityDate="2018-04-25T07:20:43.047" Title="Data Visualisation for Dataframe having 3 columns" Tags="|python|plotting|matplotlib|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30800" PostTypeId="2" ParentId="30799" CreationDate="2018-04-25T07:06:24.803" Score="2" Body="&lt;p&gt;use matplotlib:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;f, (ax1, ax2) = plt.subplots(1, 2)&#xA;ax1.plt(Data_X_err.min, Data_X_err.err)&#xA;ax2.plt(Data_X_err.max, Data_X_err.err)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Alternatively you could use seaborn's pairplot&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;%matplotlib inline&#xA;sns.set(style=&quot;ticks&quot;)&#xA;&#xA;sns.pairplot(df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/70Rbv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/70Rbv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44945" LastEditorUserId="44945" LastEditDate="2018-04-25T07:20:43.047" LastActivityDate="2018-04-25T07:20:43.047" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30801" PostTypeId="2" ParentId="30787" CreationDate="2018-04-25T07:39:38.907" Score="1" Body="&lt;p&gt;Let me give you an example where Andrew's recommendation works better than yours:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say that the real gradient is $(0, 0, 0)$ and the gradient you have computed is $(10^{-4}, 10^{-4}, 10^{-4})$. Then your average would return $10^{-8}$, and Andrew's recommendation would return $1$. Your metric could fool you into thinking that your gradient is computed propperly and the error is just due to a numeric issue, while Andrew's cannot fool you into that, due to the fact that it considers the fact that the gradient can be very small.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To wrap up, if your gradient doesn't have norm close to zero, it wouldn't really matter. However, when the gradient is close to zero you can be fooled into thinking that your gradient is right when it is not.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-25T07:39:38.907" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30802" PostTypeId="2" ParentId="30789" CreationDate="2018-04-25T07:55:54.273" Score="1" Body="&lt;p&gt;I think there is a bit of oversimplification in your analysis. Comparing the values of two parameters doesn't really make sense, as different parameters might have different &lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensional_analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;dimensions&lt;/a&gt;, and by comparing parameters with different dimensions you are saying the equivalent to: &lt;em&gt;this height of this building is two times higher than the weight of this car&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/30302/linear-regression-and-scaling-of-data/30485#30485&quot;&gt;This question&lt;/a&gt; asks a similar thing, and the most reasonable choice to me is to do it with the scaled variables (that are dimensionless) instead of the measured variables, that have dimensions.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-25T07:55:54.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30805" PostTypeId="2" ParentId="30703" CreationDate="2018-04-25T09:24:36.140" Score="3" Body="&lt;p&gt;Several options:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Locally-linear_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;Locally Linear Embedding (LLE)&lt;/a&gt;: This method construct a set of local geometric patches on each of which a data point is reconstructed through the weighted sum of its K nearest neighbor and maps these patches into a lower dimensional space. &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#sphx-glr-auto-examples-manifold-plot-compare-methods-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Find the code here&lt;/a&gt; and I strongly advice to use both LLE and &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Modified_Locally-Linear_Embedding_(MLLE)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modified LLE&lt;/a&gt; and use the better one (visually).&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;t-SNE&lt;/a&gt;: Maps the similarity of points in high-dimension, into a low dimensional manifold extracted by their distance in t-distribution. &lt;a href=&quot;https://distill.pub/2016/misread-tsne/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Be careful to tune parameters properly&lt;/a&gt;. &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.manifold.SpectralEmbedding.html#sklearn.manifold.SpectralEmbedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spectral Embedding&lt;/a&gt;: &lt;a href=&quot;https://en.wikipedia.org/wiki/Spectral_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;Spectral Clustering&lt;/a&gt; in fact (or to be precise, Spectral Clustering is indeed a simple clustering on Spectrally Embedded version of data). It projects the data on eigenvectors of its &lt;a href=&quot;https://en.wikipedia.org/wiki/Laplacian_matrix&quot; rel=&quot;nofollow noreferrer&quot;&gt;Laplacian&lt;/a&gt; according to the magnitude of corresponding eigenvalues.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;an many more ...&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-04-25T09:24:36.140" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30806" PostTypeId="2" ParentId="30794" CreationDate="2018-04-25T09:37:35.577" Score="1" Body="&lt;p&gt;The first eigenvector of the PCA have the direction of the maximum variance of the data. Then the second follows with the direction of the next highest variance and so it continues....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The easiest way to think about this, by my experience, is with 2D data and transform it to 1D. Visualization can be found here: &lt;a href=&quot;http://setosa.io/ev/principal-component-analysis/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA Visualization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It does not maximize the separation between the regular and irregular data points. It does dimension reduction, by trying to explain as much of the variance as possible by creating a new basis, eigenvectors(directions).&#xA;However, it is not very robust to outliers. Imagine all your datapoints are basically on a line, now suddenly you receive an abnormal value. As this abnormal value will have an impact on the variance, the eigenvalue directions will change, and with that the values of the eigenvectors!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For anomaly detection:&#xA;You can use the PCA eigenvectors of &quot;normal data&quot; against thos of new data. Simply do distance calculations between the eigenvectors. If you have a big distance =&gt; Anomaly&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are methods such as robust PCA that better cope with noisy data and can find the outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastEditorUserId="31086" LastEditDate="2018-04-25T10:10:39.960" LastActivityDate="2018-04-25T10:10:39.960" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30808" PostTypeId="2" ParentId="30654" CreationDate="2018-04-25T11:11:13.963" Score="1" Body="&lt;p&gt;One thing you could try is to change the softmax activation on your final layer to sigmoid. This will give a value between zero and one for each class, the sum of which may not equal one. This means that for a given example there may be more than one output neuron activated, or no output neurons activated, in which case you can treat the example as 'unsure' and manually check it out. In the case that there is exactly one output neuron activated (should be most of the time), that should be the classification given. &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-25T11:11:13.963" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30809" PostTypeId="1" AcceptedAnswerId="30811" CreationDate="2018-04-25T11:26:13.420" Score="0" ViewCount="3368" Body="&lt;p&gt;I have been looking into the site &lt;a href=&quot;http://www.saedsayad.com/unsupervised_binning.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.saedsayad.com/unsupervised_binning.htm&lt;/a&gt; and there it shows range values to the right under equal frequency binning ....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have so much looked to find how the ranges are formed but has been in vain. So , can somebody explain how those ranges are formed for the given dataset or explain the freq binning procedure in a good manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Great thanx for any help on this!&lt;/p&gt;&#xA;" OwnerUserId="41801" LastActivityDate="2020-07-04T17:26:57.740" Title="How to find the ranges in Equal frequency/depth binning?" Tags="|data-mining|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30810" PostTypeId="1" AcceptedAnswerId="31287" CreationDate="2018-04-25T11:38:37.293" Score="1" ViewCount="957" Body="&lt;p&gt;I want to use TensorFlow to create a GAN. I have managed to create these parts of code. My images are of shape [299, 299, 3] because I took some images and resized them using TensorFlow and saves them so that all my images are of the same shape.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I am trying to generate more such images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The error now is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Traceback (most recent call last):&#xA;File &quot;D:/Development_Avector/PycharmProjects/TensorFlow/gan.py&quot;, line 41, in &amp;lt;module&amp;gt;&#xA;DgL = discriminator(G_sample, reuse=True)&#xA;File &quot;D:/Development_Avector/PycharmProjects/TensorFlow/gan.py&quot;, line 31, in discriminator&#xA;Z1 = tf.layers.conv2d(x, kernel_size=5,filters=64, strides=2, padding='SAME')&#xA;File &quot;D:\Development_Avecto\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\layers\convolutional.py&quot;, line 551, in conv2d&#xA;return layer.apply(inputs)&#xA;File &quot;D:\Development_Avecto\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\layers\base.py&quot;, line 492, in apply&#xA;return self.__call__(inputs, *args, **kwargs)&#xA;File &quot;D:\Development_Avecto\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\layers\base.py&quot;, line 428, in __call__&#xA;self._assert_input_compatibility(inputs)&#xA;File &quot;D:\Development_Avecto\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\layers\base.py&quot;, line 540, in _assert_input_compat&#xA;ibility&#xA;str(x.get_shape().as_list()))&#xA;ValueError: Input 0 of layer conv2d_5 is incompatible with the layer: expected ndim=4, found ndim=2. Full shape received: [None, 6]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# The part that discriminates&#xA;X = tf.placeholder(tf.float32, shape=[None, 299, 299, 3], name='X')&#xA;&#xA;&#xA;# The part that generates&#xA;Z = tf.placeholder(tf.float32, shape=[None, 299, 299, 3], name='Z')&#xA;&#xA;&#xA;&#xA;def generator(z,reuse=False):&#xA;    with tf.variable_scope('generator',reuse=reuse):&#xA;        #z = tf.reshape(z, shape=[-1, 299, 299, 64])&#xA;        Z1 = tf.layers.conv2d(z,kernel_size=5,filters=64, strides=2, padding='SAME')&#xA;        A1 = tf.nn.relu(Z1)&#xA;        Z2 = tf.layers.conv2d(A1,kernel_size=5, filters=64, strides=1, padding='SAME')&#xA;        A2 = tf.nn.relu(Z2)&#xA;        P2 = tf.contrib.layers.flatten(A2)&#xA;        Z3 = tf.contrib.layers.fully_connected(P2, 6, activation_fn=None)&#xA;        return Z3&#xA;&#xA;&#xA;def discriminator(x,reuse=False):&#xA;    with tf.variable_scope('discriminator',reuse=reuse):&#xA;        #x = tf.reshape(x, shape=[-1, 299, 299, 3])&#xA;        Z1 = tf.layers.conv2d(x, kernel_size=5,filters=64, strides=2, padding='SAME')&#xA;        A1 = tf.nn.relu(Z1)&#xA;        Z2 = tf.layers.conv2d(A1, kernel_size=5,filters=64, strides=1, padding='SAME')&#xA;        A2 = tf.nn.relu(Z2)&#xA;        P2 = tf.contrib.layers.flatten(A2)&#xA;        Z3 = tf.contrib.layers.fully_connected(P2, 6, activation_fn=None)&#xA;        return Z3&#xA;&#xA;G_sample = generator(Z)&#xA;DxL = discriminator(X)&#xA;DgL = discriminator(G_sample, reuse=True)&#xA;print (DxL)&#xA;D_Disc_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = DxL, labels = tf.ones_like(DxL)))&#xA;D_Disc_loss1 = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = DgL, labels = tf.ones_like(DgL)))&#xA;D_MainLoss = D_Disc_loss + D_Disc_loss1&#xA;G_Generate_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = DgL, labels = tf.ones_like(DgL)))&#xA;&#xA;D_loss = tf.summary.scalar(&quot;Discriminator Loss&quot;, D_MainLoss)&#xA;G_loss = tf.summary.scalar(&quot;Generator Loss&quot;, G_Generate_loss)&#xA;merge = tf.summary.merge_all()&#xA;&#xA;variables = tf.trainable_variables()&#xA;dvariables = [var for var in variables if var.name.startswith(&quot;discriminator&quot;)]&#xA;print (dvariables)&#xA;gvariables = [var for var in variables if var.name.startswith(&quot;generator&quot;)]&#xA;print (gvariables)&#xA;&#xA;D_optimizer = tf.train.AdamOptimizer().minimize(D_Disc_loss, var_list=dvariables)&#xA;G_optimizer = tf.train.AdamOptimizer().minimize(G_Generate_loss, var_list=gvariables)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;The part that trains&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train():&#xA;filenames = tf.train.string_input_producer(&#xA;    tf.train.match_filenames_once(&quot;D:/Development_Avecto/TensorFlow/resizedimages/*.png&quot;))&#xA;reader = tf.WholeFileReader()&#xA;_, input = reader.read(filenames)&#xA;input = tf.Print(input,[input,tf.shape(input),&quot;Input shape&quot;])&#xA;input = tf.image.decode_png(input, channels=3)&#xA;input.set_shape([299, 299, 3])&#xA;&#xA;batch = tf.train.batch([input],&#xA;                       batch_size=2)&#xA;&#xA;init = (tf.global_variables_initializer(), tf.local_variables_initializer())&#xA;&#xA;with tf.Session() as sess:&#xA;    sess.run(init)&#xA;    coord = tf.train.Coordinator()&#xA;    threads = tf.train.start_queue_runners(coord=coord)&#xA;    train_writer = tf.summary.FileWriter('D:/Development_Avecto/TensorFlow/logs/1/train', sess.graph)&#xA;&#xA;&#xA;    for it in range(50):&#xA;        _, X_batch =  sess.run([input,batch])&#xA;        summary,_, DiscriminatorLoss = sess.run([merge,D_optimizer, D_Disc_loss], feed_dict={X: X_batch})&#xA;        summary,_, GeneratorLoss = sess.run([merge,G_optimizer, G_Generate_loss])&#xA;&#xA;        train_writer.add_summary(summary, it)&#xA;        train_writer.flush()&#xA;&#xA;train_writer.close()&#xA;coord.request_stop()&#xA;coord.join(threads)&#xA;sess.close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a problem with the shape of &lt;code&gt;X&lt;/code&gt; and &lt;code&gt;Z&lt;/code&gt;. The first commented line in the functions 'discriminator' and 'generator'. How should the shape be ? Can anyone explain ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Update : I have refactored the question.&lt;/p&gt;&#xA;" OwnerUserId="51126" LastEditorUserId="28175" LastEditDate="2018-04-26T15:03:49.567" LastActivityDate="2018-05-06T14:26:48.387" Title="GAN with Conv2D using TensorFlow - Shape error" Tags="|machine-learning|neural-network|deep-learning|tensorflow|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30811" PostTypeId="2" ParentId="30809" CreationDate="2018-04-25T11:41:47.720" Score="0" Body="&lt;p&gt;As far as I can see the choice of the bin size /frequency is arbitrary in those examples. Frequency binning is simple choosing you bin boundaries in a way that the bin content size is the same. For the frequency approach it looks like the order the elements by size and calculate the bin edges in the middle between the highest element of bin A and the lowest of bin B.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to be really fancy you can use methods like &lt;a href=&quot;http://www.astroml.org/examples/algorithms/plot_bayesian_blocks.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;bayesian blocks&lt;/a&gt; that actually &lt;strong&gt;&lt;em&gt;choose bin size dynamically&lt;/em&gt;&lt;/strong&gt; with respect of information content. &lt;/p&gt;&#xA;" OwnerUserId="29217" LastActivityDate="2018-04-25T11:41:47.720" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30812" PostTypeId="1" AcceptedAnswerId="30814" CreationDate="2018-04-25T11:42:10.643" Score="4" ViewCount="13543" Body="&lt;p&gt;Im using the keras LSTM model to make prediction, and the code above is to scale the data:&#xA;inputs are shaped like  (n, 11, 1) and the label is 1D &lt;br&gt;&#xA;&lt;strong&gt;DailyDemand.py&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#scaling data&#xA;scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))&#xA;x = np.array(x).reshape ((len(x),11 ))&#xA;x = scaler_x.fit_transform(x)&#xA;scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))&#xA;y = np.array(y).reshape ((len(y), 1))&#xA;y = scaler_y.fit_transform(y)&#xA;&#xA;# Split train and test data&#xA;x_train=x[0: train_end ,]&#xA;x_test=x[train_end +1: ,]&#xA;y_train=y[0: train_end]&#xA;y_test=y[train_end +1:] &#xA;x_train=x_train.reshape(x_train.shape +(1,))&#xA;x_test=x_test.reshape(x_test.shape + (1,))&#xA;# Train and save the Model named fit1 in a json and h5 files &#xA;[....]&#xA;# serialize model to JSON&#xA;model_json = fit1.to_json()&#xA;with open(&quot;model.json&quot;, &quot;w&quot;) as json_file:&#xA;    json_file.write(model_json)&#xA;# serialize weights to HDF5&#xA;fit1.save_weights(&quot;model.h5&quot;)&#xA;print(&quot;&amp;gt;&amp;gt;&amp;gt;&amp;gt; Model saved to model.h5 in the disk&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and Now im trying to predict a new values of a new data with this trained model. so i loaded the model from the files:&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;predict.py&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from DailyDemand import scaler_y&#xA;from DailyDemand import scaler_x&#xA;[...]&#xA;# load json and create model&#xA;json_file = open('model.json', 'r')&#xA;loaded_model_json = json_file.read()&#xA;json_file.close()&#xA;loaded_model = model_from_json(loaded_model_json)&#xA;# load weights into new model&#xA;loaded_model.load_weights(&quot;model.h5&quot;)&#xA;print(&quot;Loaded model from disk&quot;)&#xA;&#xA;########################################&#xA;# make prediction with the loaded model&#xA;&#xA;FeaturesTest = [267,61200,695,677,70600,116700,130200,768,659,741,419300]&#xA;xaa = np.array(FeaturesTest).reshape ((1,11 )).astype(float)&#xA;print(xaa)&#xA;xaa = scaler_x.fit_transform(xaa) &#xA;xaa = xaa.reshape(xaa.shape +(1,))&#xA;print(&quot;print FeaturesTest scalled: &quot;)&#xA;print(xaa) # incorrect scalled value, always returns -1 ones &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;xaa = [[[-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]&#xA;  [-1.]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tomorrowDemand = loaded_model.predict(xaa)&#xA;print(&quot;tomorrowDemand scalled: &quot;, tomorrowDemand)&#xA;prediction = scaler_y.inverse_transform(np.array(tomorrowDemand).reshape ((len(tomorrowDemand), 1))).astype(int)&#xA;print (&quot;the real demand is 95900 and the prediction is: &quot;, prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The problem is how i can use the same scaler used in the training on the new data?&#xA;i want to know if i made a mistake in this code to use the same scaller on the new data? &lt;/p&gt;&#xA;" OwnerUserId="50622" LastEditorUserId="50622" LastEditDate="2018-04-25T16:37:19.093" LastActivityDate="2019-05-21T22:55:33.620" Title="How to use the same minmaxscaler used on the training data with new data?" Tags="|python|scikit-learn|time-series|data|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30814" PostTypeId="2" ParentId="30812" CreationDate="2018-04-25T12:31:57.287" Score="5" Body="&lt;p&gt;You are refitting &lt;code&gt;scaler_x&lt;/code&gt; on your test set, which you don't want.  Change this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xaa = scaler_x.fit_transform(xaa)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xaa = scaler_x.transform(xaa)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You are getting &lt;code&gt;[-1, -1, ..., -1]&lt;/code&gt; because with one sample, each feature is equal to the minimum.&lt;/p&gt;&#xA;" OwnerUserId="38751" LastEditorUserId="38751" LastEditDate="2018-04-25T14:32:52.037" LastActivityDate="2018-04-25T14:32:52.037" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30815" PostTypeId="2" ParentId="30790" CreationDate="2018-04-25T12:40:34.530" Score="1" Body="&lt;p&gt;If you throw it out, training might suffer in the beginning as you would have even less data at that time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do throw it out, then you have to be aware of your fit interval, your expected episode length, and batch size so you aren't throwing out data faster than you generate it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the memory growing indefinitely, I've used a deque to limit its size.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from collections import deque&#xA;memory = deque(maxlen=10000000)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38751" LastEditorUserId="38751" LastEditDate="2018-04-25T14:34:47.953" LastActivityDate="2018-04-25T14:34:47.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30816" PostTypeId="2" ParentId="24052" CreationDate="2018-04-25T12:41:15.933" Score="17" Body="&lt;p&gt;A colleague and I have conducted some preliminary studies on the performance differences between pandas and data.table. You can find the study (which was split into two parts) on our &lt;a href=&quot;https://www.statworx.com/de/blog/pandas-vs-data-table-a-study-of-data-frames/&quot; rel=&quot;noreferrer&quot;&gt;Blog&lt;/a&gt; (You can find part two &lt;a href=&quot;https://www.statworx.com/de/blog/pandas-vs-data-table-a-study-of-data-frames-part-2/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We figured that there are some tasks where pandas clearly outperforms data.table, but also cases in which data.table is much faster. You can check it out yourself and let us know what you think of the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&lt;br&gt;&#xA;If you don't want to read the blogs in detail, here is a short summary of our setup and our findings:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Setup&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We compared &lt;code&gt;pandas&lt;/code&gt; and &lt;code&gt;data.table&lt;/code&gt; on 12 different simulated data sets on the following operations (so far), which we called scenarios.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Data retrieval with a select-like operation&lt;/li&gt;&#xA;&lt;li&gt;Data filtering with a conditional select operation&lt;/li&gt;&#xA;&lt;li&gt;Data sort operations&lt;/li&gt;&#xA;&lt;li&gt;Data aggregation operations&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The computations were performed on a machine with an Intel i7 2.2GHz with 4 physical cores, 16GB RAM and a SSD hard drive. Software Versions were OS X 10.13.3, Python 3.6.4 and R 3.4.2. The respective library versions used were 0.22 for pandas and 1.10.4-3 for data.table&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Results in a nutshell&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;data.table&lt;/code&gt;seems to be faster when selecting columns (&lt;code&gt;pandas&lt;/code&gt;on average takes 50% more time)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;pandas&lt;/code&gt; is faster at filtering rows (roughly 50% on average)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;data.table&lt;/code&gt; seems to be considerably faster at sorting (&lt;code&gt;pandas&lt;/code&gt; was sometimes 100 times slower)&lt;/li&gt;&#xA;&lt;li&gt;adding a new column appears faster with &lt;code&gt;pandas&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;aggregating results are completely mixed&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Please note that I tried to simplify the results as much as possible to not bore you to death. For a more complete visualization read the studies. If you cannot access our webpage, please send me a message and I will forward you our content. You can find the code for the complete study on &lt;a href=&quot;https://github.com/STATWORX/blog/tree/master/pandas_vs_datatable&quot; rel=&quot;noreferrer&quot;&gt;GitHub&lt;/a&gt;. If you have ideas how to improve our study, please shoot us an e-mail. You can find our contacts on GitHub.&lt;/p&gt;&#xA;" OwnerUserId="51131" LastEditorUserId="51131" LastEditDate="2018-04-26T07:45:02.687" LastActivityDate="2018-04-26T07:45:02.687" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30817" PostTypeId="1" CreationDate="2018-04-25T13:09:33.290" Score="2" ViewCount="85" Body="&lt;p&gt;I've built a fully-connected feed-forward neural network to recognize handwritten digits. I used MNIST and another very similar dataset (containing Arabic digits - same training set and test set count as those of MNIST). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a network with exactly the same architecture (1 hidden layer, same learning rate, same weight sampling), I tested on a sub-sample that used 100 images for training. The test accuracy was very low, but what I noticed was that for the Arabic digits, the network took significantly longer to train (double the amount of time than for English digits - 33ms and 15ms, respectively). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What could be the reason for this?&lt;/p&gt;&#xA;" OwnerUserId="50532" LastActivityDate="2020-04-28T07:03:00.297" Title="Why would 2 sets of similar training samples take significantly longer to train?" Tags="|neural-network|deep-learning|dataset|image-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30819" PostTypeId="1" AcceptedAnswerId="30825" CreationDate="2018-04-25T13:46:47.773" Score="21" ViewCount="45313" Body="&lt;p&gt;I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized images (for example 32x32) to input size, the content of the image is stretched horizontally too much, but for some medium size images it looks okay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the proper method for resizing images while avoiding the content being destroyed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am thinking about padding images with 0s to complete size after resizing them to some degree keeping ratio of width and height. Would it be okay with this method?)&lt;/p&gt;&#xA;" OwnerUserId="27369" LastEditorUserId="50727" LastEditDate="2020-06-18T10:12:57.900" LastActivityDate="2022-04-12T07:18:56.697" Title="Image resizing and padding for CNN" Tags="|machine-learning|deep-learning|image-classification|preprocessing|image-recognition|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30820" PostTypeId="1" AcceptedAnswerId="30831" CreationDate="2018-04-25T13:48:16.000" Score="0" ViewCount="3113" Body="&lt;p&gt;It is argued that the default value of mtry for random forests is square root of total number of features (for classification) and number of features divided by 3 for regression. Can someone tell me the literature where it is specifically mentioned?&lt;/p&gt;&#xA;" OwnerUserId="50257" LastActivityDate="2018-04-25T15:50:29.293" Title="Default value of mtry for random forests" Tags="|random-forest|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30821" PostTypeId="2" ParentId="30697" CreationDate="2018-04-25T13:52:43.057" Score="3" Body="&lt;p&gt;You may need to take a look at this work submitted and accepted for CVPR 2018 : &lt;strong&gt;&lt;a href=&quot;https://arxiv.org/abs/1711.10370&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning to Segment Every Thing&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this work, they try to segment everything, even objects not known to the network. Mask R-CNN has been used, combined with a transfer learning sub-network, they get very good results in segmenting almost everything.&lt;/p&gt;&#xA;" OwnerUserId="51139" LastActivityDate="2018-04-25T13:52:43.057" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30822" PostTypeId="2" ParentId="30767" CreationDate="2018-04-25T13:54:54.230" Score="0" Body="&lt;p&gt;You are experiencing Data Leakage. In a comment, you explained that you shuffle your data before splitting into train/validation. For each validation point, you likely are showing the model data that is nearby temporally both before and after the time of the validation point. This is information that the model can not possibly hope to have when running in real time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To alleviate this, I would keep the data in the correct time order, and instead take the validation data as a contiguous chunk of time. If you want to be the most careful, you could throw data away in a small window around the validation data, so that the training data won’t contain information about the edges of the validation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This appears to be a good resource, but I have not read it too thoroughly: &lt;a href=&quot;https://www.kaggle.com/dansbecker/data-leakage&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/dansbecker/data-leakage&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-04-25T13:54:54.230" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30823" PostTypeId="2" ParentId="30819" CreationDate="2018-04-25T14:17:15.000" Score="10" Body="&lt;p&gt;You have a few options:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Small Images:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;upsample through interpolation&lt;/li&gt;&#xA;&lt;li&gt;pad the image using zeros&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the excess pixels in the largest dimension.  Of course this would result in losing data, but you can repeatedly shift the center of your crop.  This would help your model be more robust.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For Large Images:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;downsample&lt;/li&gt;&#xA;&lt;li&gt;crop down to your input size&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Lastly, if you are using a Fully Convolutional Network (FCN), you do not need to resize your images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;yes, padding with zeros is a valid option.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-25T14:17:15.000" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30824" PostTypeId="1" AcceptedAnswerId="30835" CreationDate="2018-04-25T14:23:51.090" Score="2" ViewCount="397" Body="&lt;p&gt;I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using the inverse of the class probabilities (1/(Proportion of class)). This means that the loss gets multiplied a great lot for the minority classes. This does help in counteracting the class imbalance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, for this weighted loss I use the proportions in the &lt;em&gt;training&lt;/em&gt; dataset. In the validation dataset, the distribution is somewhat different (96.66%, 2.59%, 0.75% in val set as opposed to 98.28%, 0.98%, 0.73% in the training set). This means that if I use the weights based on the training set, mainly misclassified class 1 images gets punished a lot more than they should according to the distribution in the validation set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason for this distribution discrepancy is that I can't just do a random split on my images, since they are taken by some 10 cameras all taking pictures of the same respective static scenery (highways). This means that I have to hold out all images of about 2/3 cameras as validation and use the other cameras for training, otherwise the model will be too optimistic, since validation images come from the same camera as the train images. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This makes the weighted loss uninterpretable for the validation data. Is there a way to counteract this? For instance, would giving separate weights for the train/validation sets (based on their respective distributions) help?&lt;/p&gt;&#xA;" OwnerUserId="51141" LastActivityDate="2018-04-25T16:56:53.787" Title="Class distribution discrepancy training/validation. Loss now uninterpretable?" Tags="|cnn|class-imbalance|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30825" PostTypeId="2" ParentId="30819" CreationDate="2018-04-25T14:28:14.110" Score="18" Body="&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/questions/47697622/cnn-image-resizing-vs-padding-keeping-aspect-ratio-or-not/49882055#49882055&quot;&gt;This question&lt;/a&gt; on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.&lt;/p&gt;&#xA;&lt;p&gt;[Update April.11 2022]&#xA;Well, based on the newer research[2019], it seems like zero-padding would not affect to the CNN model at all, since &lt;strong&gt;zeros&lt;/strong&gt; would not change the synaptic weights during forward propagation or back propagation during convolution.&lt;/p&gt;&#xA;&lt;p&gt;paper reference: &lt;a href=&quot;https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0263-7&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0263-7&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="134443" LastEditDate="2022-04-12T07:18:56.697" LastActivityDate="2022-04-12T07:18:56.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30826" PostTypeId="1" AcceptedAnswerId="43903" CreationDate="2018-04-25T15:19:28.300" Score="0" ViewCount="70" Body="&lt;p&gt;I want to do KDE on data that are not necessarily normal using Gaussian kernels. In &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation&quot; rel=&quot;nofollow noreferrer&quot;&gt;KDE in wikipedia&lt;/a&gt; an expression for the bandwidth is given when the underlying distribution of the data is gaussian. I am looking for a similar bandwidth selection for non-normal data.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2019-01-13T06:34:05.830" Title="Bandwidth selection Kernel Density Estimation" Tags="|unsupervised-learning|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30828" PostTypeId="1" CreationDate="2018-04-25T15:26:44.983" Score="1" ViewCount="37" Body="&lt;p&gt;Given that I have a word that does not occur in any of my documents: &lt;code&gt;newword&lt;/code&gt;, and given that I have two classes: &lt;code&gt;class1&lt;/code&gt; and &lt;code&gt;class2&lt;/code&gt;. For instance, the total number of words in &lt;code&gt;class1&lt;/code&gt; is 3, the total number of words in &lt;code&gt;class2&lt;/code&gt; is 6, and the number of unique words in all documents is 8.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, given the Naive Bayes formula, the word &lt;code&gt;newword&lt;/code&gt; will have a higher probability of belonging to &lt;code&gt;class1&lt;/code&gt; (because the denominator will be lower while the numerator stays the same). Is there any statistical/logical theoretical explanation (that drives the algorithm) about this behavior?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell, I just want to know what is the &lt;strong&gt;motivation&lt;/strong&gt; of giving higher probability to classes with fewer words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ps.: I am using Laplace Smoothing. Therefore:&lt;br&gt;&#xA; P(newword|class1) = 0 + 1 / 3 + 8 = 0.09 &gt;&lt;br&gt;&#xA;P(newword|class2) = 0 + 1 / 6 + 8 = 0.07&lt;/p&gt;&#xA;" OwnerUserId="51147" LastEditorUserId="51147" LastEditDate="2018-04-26T01:43:59.647" LastActivityDate="2018-04-26T01:43:59.647" Title="Why do classes with fewer words have higher probability?" Tags="|classification|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30829" PostTypeId="1" CreationDate="2018-04-25T15:29:31.747" Score="11" ViewCount="1856" Body="&lt;p&gt;I'm trying to build a gesture recognition system for classifying &lt;strong&gt;ASL (American Sign Language)&lt;/strong&gt; Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it detects the sequence and maps it to it's corresponding class (sleep, help, eat, run, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is I've already built a similar system but for static images (no motion included), it was useful for translating alphabets only in which building a &lt;strong&gt;CNN&lt;/strong&gt; was a straight forward task, as the hand doesn't move so much and the data set structure was also manageable as I was using &lt;strong&gt;keras&lt;/strong&gt; and maybe still intending to do so (every folder contained a set of images for a particular sign and the name of the folder is the class name of this sign ex: A, B, C,..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question here, how can I organize my data set to be able to input it into a &lt;strong&gt;RNN&lt;/strong&gt; in keras and what certain functions should I use to effectively train my model and any necessary parameters, some people suggested using &lt;strong&gt;TimeDistributed&lt;/strong&gt; class but I don't have a clear idea on how to use it for my favor, and take in account the input shape of every layer in the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;also considering that my data set would consist of images, I'll probably need a convolutional layer, how would it be feasible to combine the &lt;strong&gt;conv&lt;/strong&gt; layer into the &lt;strong&gt;LSTM&lt;/strong&gt; one (I mean in terms of code).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example I imagine my data set to be something like this&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Is2x0.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Is2x0.jpg&quot; alt=&quot;data set structure&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Folder named 'Run' contains 3 folders 1, 2 and 3, each folder corresponds to it's frame in the sequence&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IOFNg.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IOFNg.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/kAqve.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kAqve.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/ntCPs.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ntCPs.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So &lt;strong&gt;Run_1&lt;/strong&gt; will contain some set of images for the first frame, &lt;strong&gt;Run_2&lt;/strong&gt; for the second frame and &lt;strong&gt;Run_3&lt;/strong&gt; for the third, my model's objective is to be trained with this sequence to output the word &lt;strong&gt;Run&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="51143" LastEditorUserId="51143" LastEditDate="2018-04-25T20:42:34.047" LastActivityDate="2018-08-24T16:51:14.763" Title="Using RNN (LSTM) for Gesture Recognition System" Tags="|neural-network|deep-learning|keras|rnn|lstm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30830" PostTypeId="1" AcceptedAnswerId="30832" CreationDate="2018-04-25T15:43:03.713" Score="2" ViewCount="2182" Body="&lt;p&gt;What is the difference between using CNN with handcrafted features and CNN without handcrafted features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="29169" LastEditDate="2021-02-13T04:23:35.950" LastActivityDate="2021-02-13T04:23:35.950" Title="Using handcrafted features in CNN" Tags="|deep-learning|tensorflow|cnn|computer-vision|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30831" PostTypeId="2" ParentId="30820" CreationDate="2018-04-25T15:50:29.293" Score="1" Body="&lt;p&gt;The textbook 'The Elements of Statistical Learning' by authors Trevor Hastie, Robert Tibshirani, and Jerome Friedman discuss how to tune the number of variables to sample as you build the trees in Chapter 15, Section 3. The pdf of the text is freely available and can be found &lt;a href=&quot;https://web.stanford.edu/~hastie/Papers/ESLII.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the values you have listed are in fact default values, as you have mentioned. In practice, it is best to tune the value of 'mtry', as the best value will depend on the specific problem and dataset you are working with.&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-04-25T15:50:29.293" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30832" PostTypeId="2" ParentId="30830" CreationDate="2018-04-25T16:20:27.153" Score="2" Body="&lt;p&gt;A CNN automatically extracts features, so hand-crafting features has become unnecessary for most applications.  It learns what features to extract via backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can introduce handcrafted features in a few ways to boost the performance of a CNN:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;inject hand-crafted features into the fully connected layer&lt;/li&gt;&#xA;&lt;li&gt;feed the CNN output into another classifier alongside the hand-crafted features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.researchgate.net/publication/317340604_Handcrafted_vs_Non-Handcrafted_Features_for_computer_vision_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper&lt;/a&gt; explores such concepts.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-25T16:20:27.153" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30834" PostTypeId="1" AcceptedAnswerId="30838" CreationDate="2018-04-25T16:33:50.000" Score="1" ViewCount="1558" Body="&lt;p&gt;I'm trying to use Tensorflow for signal classification. The signals are either normal or high-risk signals. For this purpose, I used convolutional neural networks. The length of signals are 685 and the architecture is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Convolution layer with 27 channels and 1 by 16 window size and stride 1.&lt;/li&gt;&#xA;&lt;li&gt;Max pooling layer with 1 by 2 window size and stride 2.&lt;/li&gt;&#xA;&lt;li&gt;Convolution layer with 14 channels and 1 by 32 window size and stride 1.&lt;/li&gt;&#xA;&lt;li&gt;Max pooling layer with 1 by 2 window size and stride 2.&lt;/li&gt;&#xA;&lt;li&gt;Convolution layer with 4 channels and 1 by 32 window size and stride 1.&lt;/li&gt;&#xA;&lt;li&gt;Max pooling layer with 1 by 2 window size and stride 2.&lt;/li&gt;&#xA;&lt;li&gt;Convolution layer with 3 channels and 1 by 10 window size and stride 1.&lt;/li&gt;&#xA;&lt;li&gt;Max pooling layer with 1 by 2 window size and stride 2.&lt;/li&gt;&#xA;&lt;li&gt;Fully connected layer with 20 neurons and dropout layer.&lt;/li&gt;&#xA;&lt;li&gt;Fully connected layer with 10 neurons and dropout layer.&lt;/li&gt;&#xA;&lt;li&gt;And finally Soft max layer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;After training the network using AdamOptimizer with learning rate 0.001 with 150000 signals the training accuracy is near 95 percent (batch training with 16 batch size is used), however testing accuracy using 20000 new signals is almost 50 percent. Since there is just 2 classes, this accuracy is no better than random guess.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=&quot;conv1d-27-16-1,maxpool-2,conv1d-14-32-1,maxpool-2,conv1d-4-32-1,maxpool-2,conv1d-3-10-1,maxpool-2,full-20,full-10,softmax&quot;&#xA;&#xA;cnn=CNN_1D(model,input_size=685,n_classes=2,num_epochs=20,batch_size=16,dropout=0.75)&#xA;cnn.read_data('train_input','train_targets','test_input','test_targets')&#xA;cnn.build_model()&#xA;cnn.training(validation_set='all')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can I improve the testing accuracy in my network?&lt;/p&gt;&#xA;" OwnerUserId="43959" LastEditorUserId="1330" LastEditDate="2018-04-25T20:14:32.307" LastActivityDate="2018-04-25T20:14:32.307" Title="Improve test accuracy for TensorFlow CNN" Tags="|machine-learning|python|neural-network|deep-learning|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30835" PostTypeId="2" ParentId="30824" CreationDate="2018-04-25T16:56:53.787" Score="0" Body="&lt;p&gt;The validation set is used to estimate how well the model is fitting to your solution space.  For instance, if your training loss continues to decrease but your validation loss remains constant or increases, then you should employ early stopping as to avoid overfitting.  This does not require a weighted loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you wanted to compare the training loss to the validation loss, you cannot use the same weights if the distributions differ.  Instead, you can simply compare the average unweighted loss per set.  The weighted loss is for penalization.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-25T16:56:53.787" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30836" PostTypeId="2" ParentId="30701" CreationDate="2018-04-25T17:49:55.617" Score="4" Body="&lt;p&gt;It's probably this way..(not sure , give it a try)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Iputs needs to be reshaped to be [samples, time steps, features]&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TrainX= np.reshape(TrainX,(TrainX.shape[0], 1, TrainX.shape[1]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You need to passin the actual vector also?&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-04-25T17:49:55.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30838" PostTypeId="2" ParentId="30834" CreationDate="2018-04-25T20:04:27.037" Score="2" Body="&lt;p&gt;This is clearly a case of overfitting, since your validation error is way bigger than your training error. Having understood that overfitting is your problem, there are many ways you can address this problem:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network#Dropout&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dropout&lt;/a&gt; (&lt;a href=&quot;https://keras.io/layers/core/#dropout&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;li&gt;Weight decay (L2 regularization)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Early stopping&lt;/a&gt; (&lt;a href=&quot;https://keras.io/regularizers/&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I can see that you are already doing dropout, so setting the dropout higher should help. Moreover, you can try using dropout in more layers, not just the two last ones. If setting the dropout rate higher and setting more layers with dropout does not work, then this probably means that your training and validation data do not come from the same distribution. You can study statistical properties of your training data and validation and study if they come from the same distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Extra comment: if there are just two classes, why don't you use a sigmoid activation function in the last layer?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-25T20:04:27.037" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="30839" PostTypeId="2" ParentId="30754" CreationDate="2018-04-25T20:10:02.263" Score="1" Body="&lt;p&gt;&lt;em&gt;Disclaimer, I have not tried any of these ideas.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Predict the CI directly&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: This method will require very large batch sizes, which may not be possible due to memory constraints.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Set the batch size to something large. You'll also need to decide what you want your confidence interval to be, 95% matches a lot of expectations but may be less numerically stable than something like 90% of 80%. We'll call this value $0 &amp;lt; c &amp;lt; 1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition to outputting a prediction $\hat{y_i}$ of the true value $y$, have your model also output a lower bound $l_i(c)$ and an upper bound $u_i(c)$ for each individual estimate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, define&#xA;$$&#xA;z = &#xA;\begin{align}&#xA;&amp;amp; \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i &amp;lt; l_i(c))\ \text{ else }\ 0)}{n} \right)^2 \\&#xA;+ &amp;amp; \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i &amp;gt; u_i(c))\ \text{ else }\ 0)}{n} \right)^2&#xA;\end{align}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In essence, $z$ is the (squared) measure of how many times the predicted lower/upper bounds are exceeded vs. how many times we expect them to be exceeded. In practice, some tweaks to exactly how $z$ is computed may be necessary, but the main idea of penalizing the model for having too lax or too restricted in its estimations of the lower/upper bounds should be preserved.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Add $z$ to your MSE loss function, you'll likely need to balance these two losses, i.e. add an additional hyperparameter $\alpha$ to your model such that $\text{TOTAL LOSS} = \text{MSE} + \alpha z$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, if you set your batch size/time-step length so that there are 500 &#xA;distinct estimates (e.g. $n=500$), and set $c$ to 0.9, in theory this training procedure should encourage the model to estimate $l_i(c)$ and $u_i(c)$ such that exactly 25 actual $y_i$ values are below $l_i(c)$ and exactly 25 $y_i$ are above $u_i(c)$.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Predict the error directly&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;In addition to getting a prediction $\hat{y}$ of the true value $y$ and training with the MSE loss function, add another output of your model which attempts to predict the squared error $\epsilon = (\hat{y} - y)^2$ directly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can create the MSE of the estimate $\hat{\epsilon}$ of $\epsilon$ and combine that with your existing MSE to get the total loss for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then use this value to normalize your errors to try and get an anomaly score that is usable across different sub-populations:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\text{anomaly score} = \frac{(y - \hat{y})^2}{\hat{\epsilon}}&#xA;$$&#xA;where higher scores indicate that the true error is larger than expected.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="33202" LastEditDate="2018-04-25T20:40:05.950" LastActivityDate="2018-04-25T20:40:05.950" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30840" PostTypeId="2" ParentId="30697" CreationDate="2018-04-25T20:36:52.970" Score="5" Body="&lt;h1&gt;Fast answear&lt;/h1&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/beckgael/Mean-Shift-LSH&quot; rel=&quot;nofollow noreferrer&quot;&gt;Mean Shift LSH&lt;/a&gt; which is an upgrade in &lt;strong&gt;&lt;span class=&quot;math-container&quot;&gt;$O(n)$&lt;/span&gt;&lt;/strong&gt; of the famous Mean Shift algorithm in &lt;span class=&quot;math-container&quot;&gt;$O(n^2)$&lt;/span&gt; well know for its image segmentation ability&lt;/p&gt;&#xA;&lt;h1&gt;Some explanations&lt;/h1&gt;&#xA;&lt;p&gt;If you desire a true &lt;strong&gt;unsupervised&lt;/strong&gt; approach to segment images, use &lt;strong&gt;clustering algorithms&lt;/strong&gt;. The fact is that there is a lot of algorithms with different &lt;strong&gt;time complexity&lt;/strong&gt; and &lt;strong&gt;specificity&lt;/strong&gt;. Take the most famous one, the &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt;-Means, it is in &lt;span class=&quot;math-container&quot;&gt;$O(n)$&lt;/span&gt; so pretty fast but you have to specify how many cluster you want which is not what you intend by exploring an unknown image without any information about how many shapes are presents in it. Moreover even if you suppose that you know how many shape are present, we may suppose that there shapes are random which is another point where the &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt;-Means fail because it is design to find elliptic clusters and &lt;strong&gt;NOT&lt;/strong&gt; random shape ones.&lt;/p&gt;&#xA;&lt;p&gt;At the opposite we have the Mean Shift that is able to find &lt;strong&gt;automatically&lt;/strong&gt; the number of cluster -- which is useful when you don't know what you're looking for -- with &lt;strong&gt;random shapes&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Of course you replace the &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt; parameter of &lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt;-Means by others Mean Shift parameters which can be tricky to fine tuned but it doesn't exist a tool that allow you to do magic if you're not exercising to do magic.&lt;/p&gt;&#xA;&lt;h1&gt;An advice to image segmentation clustering&lt;/h1&gt;&#xA;&lt;p&gt;Transform your color space from RGB to LUV which is better for euclidean distance.&lt;/p&gt;&#xA;&lt;h1&gt;&lt;span class=&quot;math-container&quot;&gt;$K$&lt;/span&gt;-Means vs Mean Shift LSH time complexity&lt;/h1&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Mean Shift : &lt;span class=&quot;math-container&quot;&gt;$O(\alpha.n)$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;K-Means : &lt;span class=&quot;math-container&quot;&gt;$O(\beta.n)$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;span class=&quot;math-container&quot;&gt;$\alpha \gt \beta$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Mean Shift LSH is slower but it fits better with your needs. It stay still linear and is also &lt;strong&gt;scalable&lt;/strong&gt; with the mentioned implementation.&lt;/p&gt;&#xA;&lt;p&gt;PS : My profile picture is an application of the Mean Shift LSH on myself if it can help to figure out how it works.&lt;/p&gt;&#xA;" OwnerUserId="9257" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-04-25T20:36:52.970" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30841" PostTypeId="1" CreationDate="2018-04-25T20:54:51.277" Score="1" ViewCount="68" Body="&lt;p&gt;I am trying to build out a customer sensitivity model to price increase. My hypothesis is that as &quot;Annual Fee&quot; to a product subscription increases so will the probability of a customer to churn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am planning to initially build out a classifier model with output as 0 - (Does not churn) and 1 (Churn). For each percent increase in annual fee the probability of churn should also increase. Can I use a classifiers proba function for this kind of analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;For eg: If a customer is given 5% increase and the output of classifier is 1 and probability is 60% then for my next data point, keeping everything constant and just changing the increase from 5% to 10%, I want to see that output of the classifier is again 1 but has a higher probability (higher than 60%)&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will really appreciate any help input on this!&lt;/p&gt;&#xA;" OwnerUserId="51007" LastActivityDate="2018-04-25T20:54:51.277" Title="Using Random Forest Probabilities for customer sensitivity" Tags="|machine-learning|python|classification|data-mining|probability|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30843" PostTypeId="2" ParentId="30741" CreationDate="2018-04-25T21:47:15.797" Score="1" Body="&lt;p&gt;You can trivially modify DBSCAN (see GDBSCAN) to take time into account rather than dropping the precious time information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. neighbors are those points that are within a time difference of dt, and that have a minimum similarity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, classic &quot;change in mean&quot; analysis techniques for time series come to my mind. Maybe also changepoint detection.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-25T21:47:15.797" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30844" PostTypeId="2" ParentId="30560" CreationDate="2018-04-25T21:56:04.527" Score="-2" Body="&lt;p&gt;On such data, naive bayes (and maybe non-naive Bayes variants) should perform extremely well. Because all your inputs are binary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's also incredibly cheap to train, evaluate, and explain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can combine it with frequent itemset to make it less naive, but I wouldn't be surprised if that does not improve the accuracy much. You would use FIM to find dependencies worth modeling, and build an optimal Bayes for non-overlapping frequent subsets (e.g. engineering subjects vs. languages). Then combine these partitions assuming independence.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-25T21:56:04.527" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30845" PostTypeId="2" ParentId="26308" CreationDate="2018-04-25T22:03:11.460" Score="29" Body="&lt;p&gt;In case this can help anyone else. Here is a solution that is more computationally efficient.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;TL;DR version&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If each row already has a unique index, then do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df.loc[df.groupby('A')['C'].idxmin()]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you've already indexed by 'A', then convert 'A' back into a column first.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df2 = df.reset_index()&#xA;&amp;gt;&amp;gt;&amp;gt; df2.loc[df2.groupby('A')['C'].idxmin()]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Step by Step explanation:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 1.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;First, make sure each row in your dataframe is uniquely indexed. This is the default when importing csv data. e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    &amp;gt;&amp;gt;&amp;gt; df = pd.read_csv('questionData.csv'); df&#xA;        A       B       C&#xA;    0   196512  1325    12.901051&#xA;    1   196512  114569  12.926770&#xA;    2   196512  118910  12.898335&#xA;    3   196512  100688  12.950509&#xA;    4   196795  28978   12.780517&#xA;    5   196795  34591   12.899411&#xA;    6   196795  13078   12.913575&#xA;    7   196795  24173   12.876965&#xA;    8   196341  118910  12.898335&#xA;    9   196341  100688  12.950509&#xA;    10  196641  28972   12.780517&#xA;    11  196641  34591   12.899411&#xA;    12  196346  118910  12.898335&#xA;    13  196346  100688  12.950509&#xA;    14  196646  28980   12.780517&#xA;    15  196646  34591   12.899411&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Aside: If you already converted column 'A' into an index, then you can turn the index back into a column (&lt;a href=&quot;https://stackoverflow.com/questions/20461165/how-to-convert-pandas-index-in-a-dataframe-to-a-column&quot;&gt;https://stackoverflow.com/questions/20461165/how-to-convert-pandas-index-in-a-dataframe-to-a-column&lt;/a&gt;) by doing: &lt;code&gt;df.reset_index()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Step 2.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Use the &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.idxmin.html&quot; rel=&quot;noreferrer&quot;&gt;pandas.DataFrame.idxmin&lt;/a&gt; function to retrieve the indices of the minimum of each group.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The semantics of the example below is this: &quot;group by 'A', then just look at the 'C' column of each group, and finally return the index corresponding to the minimum 'C' in each group.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; indices = df.groupby('A')['C'].idxmin; indices&#xA;A&#xA;196341     8&#xA;196346    12&#xA;196512     2&#xA;196641    10&#xA;196646    14&#xA;196795     4&#xA;Name: C, dtype: int64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;Step 3.&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Finally, use the retrieved indices in the original dataframe using &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html&quot; rel=&quot;noreferrer&quot;&gt;pandas.DataFrame.loc&lt;/a&gt; to get the rows of the original dataframe correponding to the minimum values of 'C' in each group that was grouped by 'A'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df.loc[indices]&#xA;    A       B       C&#xA;8   196341  118910  12.898335&#xA;12  196346  118910  12.898335&#xA;2   196512  118910  12.898335&#xA;10  196641  28972   12.780517&#xA;14  196646  28980   12.780517&#xA;4   196795  28978   12.780517&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: The groupby('A') operation returns groups sorted by A. Thus 'indices' is sorted by A. If we want the original order, we just have to do&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df.loc[indices].sort_index()&#xA;    A       B       C&#xA;2   196512  118910  12.898335&#xA;4   196795  28978   12.780517&#xA;8   196341  118910  12.898335&#xA;10  196641  28972   12.780517&#xA;12  196346  118910  12.898335&#xA;14  196646  28980   12.780517&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51153" LastEditorUserId="51153" LastEditDate="2018-04-25T22:09:47.863" LastActivityDate="2018-04-25T22:09:47.863" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30847" PostTypeId="2" ParentId="30828" CreationDate="2018-04-25T23:24:24.387" Score="2" Body="&lt;p&gt;The Naive Bayes formula doesn't do what you say. For classes $C_k$ and words $x_1, \ldots, x_n$, the formula is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(C_k | x_1, \ldots, x_n) \propto P(C_k) \prod_{i=1}^N P(x_i | C_k)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the word &lt;code&gt;newword&lt;/code&gt; doesn't occur, we have no prior information about $P(x_i | C_k)$, so all these conditional probabilities should be equal. So the right-hand side is just proportional to $P(C_k)$, which is independent of the number of words in $C_k$.&lt;/p&gt;&#xA;" OwnerUserId="39007" LastActivityDate="2018-04-25T23:24:24.387" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30849" PostTypeId="1" AcceptedAnswerId="30854" CreationDate="2018-04-26T04:54:23.490" Score="0" ViewCount="258" Body="&lt;p&gt;I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and edges in the hidden layers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example Data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float)&#xA;Y = np.array([[0],[1],[1],[0]], dtype=np.float)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Example Graph:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/hxY8I.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hxY8I.png&quot; alt=&quot;Partially Connected Neural Network&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using the &lt;i&gt;sum product&lt;/i&gt; aggregation function to calculate each layer's values:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ &#xA;\sum_{i=0}^n w_{ij}x_i&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The synapse weights are denoted by two subscripts $ij$. Subscript $i$ represents the previous neuron and subscript $j$ represents the current neuron under consideration. The sum product for current neuron $j$ is computed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;s_j = w_{0j}x_0 + w_{1j}x_1 + ... + w_{nj}x_n = \sum_{i=0}^n w_{ij}x_i&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is &lt;i&gt;sum product&lt;/i&gt; an ideal aggregation function for PCNNs? Dot product won't work as the tensor sizes are almost always incompatible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: As &lt;code&gt;Bence Mélykúti&lt;/code&gt; suggested, my solution was to use a weight vector $w$ that is padded with zeros and reshaped for dot product. Here is how I accomplished it in my Python library:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from functools import reduce&#xA;import numpy as np&#xA;import math&#xA;&#xA;# An example of some input/sensor data&#xA;X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float16)&#xA;# From the graph, two inbound synapses in layer 1, here randomly generated&#xA;W = np.random.random((2,))&#xA;&#xA;# Enumerate the element count in each array&#xA;xElements = reduce(lambda x, y: x*y, list(X.shape))&#xA;wElements = reduce(lambda x, y: x*y, list(W.shape))&#xA;&#xA;# If previous layer has more elements than the weights in the current layer&#xA;if xElements &amp;gt; wElements:&#xA;    # Create a list of zeros of shape X.shape&#xA;    A = np.zeros(X.shape).flatten().tolist()&#xA;else:&#xA;    # Otherwise, create a list with the larger number of elements in W&#xA;    A = np.zeros(W.shape).flatten().tolist()&#xA;&#xA;# If the list is odd, add one element to ensure it is even&#xA;# Done so the array can be reshaped to fit the data and dot product&#xA;if len(A) % 2 != 0:&#xA;    A.extend([np.float16(0.0)])&#xA;&#xA;# Fill the list with the elements from W&#xA;for ix,ele in enumerate(W):&#xA;    A[ix] = ele&#xA;A = np.array(A)&#xA;&#xA;# Convert list to an array and reshape to be compatible with dot product&#xA;A = A.reshape(X.shape[-1],(math.floor(A.shape[0]/2)))&#xA;&#xA;result = np.dot(X,A)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The result of the dot product looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(result)&#xA;array([[0.        , 0.        , 0.        , 0.        ],&#xA;       [0.        , 0.        , 0.        , 0.        ],&#xA;       [0.45510922, 0.79058734, 0.        , 0.        ],&#xA;       [0.45510922, 0.79058734, 0.        , 0.        ]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thanks Bence!&lt;/p&gt;&#xA;" OwnerUserId="37125" LastEditorUserId="37125" LastEditDate="2018-04-28T15:10:09.780" LastActivityDate="2018-04-28T15:10:09.780" Title="Ideal aggregation function for Partially Connected Neural Network (PCNN)" Tags="|neural-network|aggregation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30850" PostTypeId="1" AcceptedAnswerId="30853" CreationDate="2018-04-26T05:32:31.020" Score="3" ViewCount="234" Body="&lt;p&gt;I am trying to GradientCheck my c++ LSTM. The structure is as follows:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;output vector (5D)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Dense Layer with softmax (5 neurons)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;LSTM layer (5 neurons)&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;input vector (5D)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My gradient check uses this formula&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$d = \frac{\vert \vert (g-n) \vert \vert _2 }{&#xA;\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;Dense Layer&lt;/strong&gt; returns descrepancy = 2.843e-05  with an epsilon of 5e-3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With 5e-5 or lower the network doesn't see any changes in Cost at all, anything greater than 5e-2 results in low precision. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is only for top layer, and I am &lt;strong&gt;using 'float' 32 bit values&lt;/strong&gt; for everything.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For the &lt;strong&gt;LSTM layer&lt;/strong&gt;, I am using 5e-3 as well, but descrepancy is 0.95&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The LSTM seems to converge fairly quickly on any task&lt;/p&gt;&#xA;&#xA;&lt;p&gt;0.95 bothered me, so I manually compared NumericalGradient array against the BackpropGradient array, side by side. All of the gradients match sign, and only differ in size of each entry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;numerical:    backpropped:&#xA;&#xA;-0.015223     -0.000385&#xA; 0.000000      0.000000&#xA;-0.058794     -0.001509&#xA;-0.000381     -9.238e-06&#xA; 9.537e-05     2.473e-06&#xA; 0.000215      6.266e-0.6&#xA;-0.015223     -0.000385&#xA; ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As you can see, the signs do indeed match, and the numerical gradient is always larger than the back propped gradient&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would you say it is acceptable and I can blame float precision?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somewhat solved, - I simply forgot to turn-off &quot;average the initial gradients by the number of timesteps&quot; during backprop of my LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That's why my gradients were always smaller in the &quot;backpropped&quot; column.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now getting descrepancy of 0.00025 for LSTM&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; setting epsilon to 0.02 (lol) seems like a sweet-spot, as it results in descrepancy of 6.5e-05. Anything larger or smaller makes it deviate from 6.5e-05, so it seems like a numerical issue ...Only 2 layers deep though, weird af&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone had this precision before?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-11-22T13:10:50.157" LastActivityDate="2018-11-22T13:10:50.157" Title="GradientChecking, can I blame float precision?" Tags="|machine-learning|gradient-descent|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30853" PostTypeId="2" ParentId="30850" CreationDate="2018-04-26T07:43:38.963" Score="3" Body="&lt;p&gt;The fact that there is a sweet spot is a common issue in &lt;a href=&quot;https://en.wikipedia.org/wiki/Numerical_differentiation#Practical_considerations_using_floating_point_arithmetic&quot; rel=&quot;nofollow noreferrer&quot;&gt;numerical differentiation&lt;/a&gt;. The issue is that using a high epsilon value will give you high error due to the fact that the numerical derivatives have error $O(\epsilon)$ (or $O(\epsilon^2)$ if you use centered difference), and using a very low epsilon will result in cancelling errors (see &lt;a href=&quot;https://math.stackexchange.com/questions/1920525/why-is-catastrophic-cancellation-called-so&quot;&gt;catastrophic cancellation question&lt;/a&gt;) due to the fact that you are substracting two numbers that are very close. It seems that $\epsilon = 0.01$ is a fair trade-off between these two extreme situations in your case.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T07:43:38.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30854" PostTypeId="2" ParentId="30849" CreationDate="2018-04-26T08:55:07.493" Score="0" Body="&lt;p&gt;What you call sum product is the same as a dot product of two vectors of equal length, $(w_{0j},w_{1j},\dots,w_{nj})$ and $(x_0,x_1,\dots,x_n)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on your diagram, your difficulty is that you want to be able to define connections between neurons that are not in consecutive layers (in other words, the connections might skip layers). I assume that there are no directed edges (synapses) that would go in the opposite direction (from layer $n$ to layer $m$ for some $m&amp;lt;n$). In this setting, you have to forward propagate information to layer $n$ from input $X$ and layers $1,2,\ldots,n-1$. I don't know when and how often you want to 'update' your network topology. Do you want to change it during training?! (Otherwise you'd fix a partially connected NN and you wouldn't write update in your question.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For forward propagation in a fixed network topology, I would associate to neuron $j$ a vector of weights $(w_{ij})_{i\in I_j}$, where $I_j$ is the set of neurons that bring information to $j$ (those $i$ where there exists a directed edge from $i$ to $j$). Then you could &lt;strong&gt;either&lt;/strong&gt; cherry-pick the neurons $i$ from the set of all neurons (by maintaining a list of indexes to find them in the vector of all neurons) and put them into a vector $x$, which is now as long as $(w_{ij})_{i\in I_j}$, and you can use &lt;em&gt;dot product&lt;/em&gt; on the two vectors. (With &lt;code&gt;numpy&lt;/code&gt; for Python, &lt;code&gt;numpy.dot&lt;/code&gt; would work.) You would need to do this separately for all $j$ in the given layer. As these are receiving information from differing numbers of inputs and neurons, they indeed cannot be arranged into a matrix/tensor, not even receiving neurons of the same layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Or&lt;/strong&gt; you could force all potential preceding neurons into one vector. Neuron $j$ of layer $n$ can only receive information from neurons $x=(&amp;lt;$input nodes$&amp;gt;$, $&amp;lt;$neurons of layer $1&amp;gt;$, $\ldots$, $&amp;lt;$neurons of layer $n-1&amp;gt;)$. This is a vector, and you can define a vector $w$ of equal length for $j$, which would indeed have many zeros, and $x$ and $w$ can be multiplied. For all neurons of layer $n$, you could stack these $w$ as row vectors on top of each other into a matrix/tensor of size (the number of neurons in layer $n$) times (the size of input $+$ the number of neurons in layer $1$ $+\ldots+$ the number of neurons in layer $n-1$) and use &lt;code&gt;np.dot&lt;/code&gt; between this matrix and the $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to do training (backpropagation) with such topologies is something you have to consider a bit. I suspect it will work without much adjustment compared to a fully connected neural network without layer skipping. In the second option, you would only need to zero out the weights which must be kept zero after each backpropagation update. This indeed carries overhead if you're not careful. But if you maintained a binary ($0$ or $1$) 'mask' matrix of which weights are live (which correspond to existing connections), then you would only ever update these weights in the backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Additionally, you may want to introduce a bias term $w_{-1j}$ (also known as intercept term, a constant offset) as if $x$ had an extra dimension that is kept constant $1$. After the aggregation with the sum product, you should also pass the result through a non-linear &lt;a href=&quot;https://en.wikipedia.org/wiki/Activation_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;activation function&lt;/a&gt; before entering the value into the neuron in the next layer.&lt;/p&gt;&#xA;" OwnerUserId="47856" LastEditorUserId="47856" LastEditDate="2018-04-26T20:50:59.493" LastActivityDate="2018-04-26T20:50:59.493" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30855" PostTypeId="2" ParentId="9852" CreationDate="2018-04-26T09:09:14.643" Score="0" Body="&lt;p&gt;As pointed out by Jacob Panikulam, Universal Approximation Theorem gives you the answer. So...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Yes. But please do note that anyway this is valid for GLMs, SVMs... So...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Would the Neural Network not tent towards being the same as the Bayesian Network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, it does, as does with any other model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this does not happen the other way around, Bayesian Networks cannot model certain types of dependencies, e.g. circular dependencies. Also, all Bayesian Network structure learning algorithms assume faithfulness, so they perform pretty badly with certain types of independent and conditionally dependent variables: see &lt;a href=&quot;https://stats.stackexchange.com/questions/342624/is-there-a-probabilistic-graphical-model-for-this-situation&quot;&gt;Is there a Probabilistic Graphical Model for this Situation?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caution Note: &lt;strong&gt;Please do not mistake this with the fact that you should only learn about and use neural networks&lt;/strong&gt;. &lt;a href=&quot;http://www-bcf.usc.edu/~gareth/ISL/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Introduction to Statistical Learning&lt;/a&gt; thoroughly explains why.&lt;/p&gt;&#xA;" OwnerUserId="39294" LastEditorUserId="39294" LastEditDate="2018-07-19T10:12:39.317" LastActivityDate="2018-07-19T10:12:39.317" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30856" PostTypeId="1" CreationDate="2018-04-26T09:11:07.067" Score="1" ViewCount="1471" Body="&lt;p&gt;I have an e-commerce dataset for a month consisting of five columns where the fifth column is &quot;Revenue&quot;. The other four columns are different factors for the revenue such as &quot;Page-Views&quot;,&quot;Visitors&quot;,&quot;Bounces&quot; and &quot;Click Through Rates&quot;. Each row is basically a particular day of the month(hence 30 rows in total).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if I would like to find out the contribution/importance of each factor for the revenue, will correlation values between the variables be sufficient or are there better metrics ?&lt;/p&gt;&#xA;" OwnerUserId="50967" LastEditorUserId="50967" LastEditDate="2018-04-26T12:48:38.213" LastActivityDate="2018-04-27T19:42:53.667" Title="How to find out the percentage of contribution of a variable for another variable/feature?" Tags="|machine-learning|data-mining|dataset|statistics|data|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30858" PostTypeId="2" ParentId="26907" CreationDate="2018-04-26T09:16:49.143" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Does it mean that the best network is always the empty network?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The empty DAG (i.e. the DAG with no arcs) implies the factorization is simply $$P(X_1, \dots, X_n) = P(X_1) \dots P(X_n),$$ which is telling you all variables are independent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This will be the best network (DAG) if all the variables are actually independent (Remember that independence for a dataset entails a certain tolerance), because the penalization coefficient will make the score lower as any arc is added.&lt;/p&gt;&#xA;" OwnerUserId="39294" LastActivityDate="2018-04-26T09:16:49.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30859" PostTypeId="1" CreationDate="2018-04-26T09:16:51.497" Score="7" ViewCount="5823" Body="&lt;p&gt;Suppose I know that I want to use a ResNet-101 architecture for my specific problem. There are ReseNet-101 models trained on ImageNet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any disadvantage of using those pre-trained models and just resetting the last (few) layers to match the specific number of classes or should that be the default option?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please don't simply post your gut feeling... I have a gut feeling as well, but I want to know.&lt;/p&gt;&#xA;" OwnerUserId="8820" LastEditorUserId="8820" LastEditDate="2018-04-26T20:03:26.777" LastActivityDate="2019-08-06T02:17:24.503" Title="Is there any proven disadvantage of transfer learning for CNNs?" Tags="|machine-learning|neural-network|cnn|reference-request|transfer-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30860" PostTypeId="1" CreationDate="2018-04-26T09:21:30.230" Score="12" ViewCount="6522" Body="&lt;p&gt;I was wondering if there exists techniques to cluster data according to a target. For example, suppose we want to find groups of customers likely to churn: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Target is churn.&lt;/li&gt;&#xA;&lt;li&gt;We want to find clusters exhibiting the same behaviour according to the fact that they are likely to churn (or not). Therefore, variables not explaining churn behaviour should not influence how clusters are built.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I have done this analysis the following way: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predict target (e.g. using a Random Forest) and retrieve &quot;most important features&quot; (from feature importance analysis).&lt;/li&gt;&#xA;&lt;li&gt;Cluster samples with selected features (e.g. using k-means).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;However, I am afraid the clustering technique used in the 2nd step might not catch behaviours found in the 1st step which might explain churn (suppose there is a complex interaction in some trees in the RF, this interaction might not be cought in the k-means algorithm).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of another way of doing this by using a neural network:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Predict target using a neural network with several layers, and for each sample retrieve activations from a given layer.&lt;/li&gt;&#xA;&lt;li&gt;Cluster samples with their activations.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;If the performance of the neural network is good and if the layer from which activations are retrieved is carefully chosen (not too close to the input or the output layer), I suppose the clusters could show customers displaying the same behaviour explaining the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I did not find any articles having this approach. Did anyone deal with the same issue or have other ideas?&lt;/p&gt;&#xA;" OwnerUserId="41888" LastEditorUserId="41888" LastEditDate="2018-04-26T13:01:42.217" LastActivityDate="2021-12-01T12:10:33.987" Title="Is it possible to cluster data according to a target?" Tags="|clustering|predictive-modeling|predictor-importance|" AnswerCount="6" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30862" PostTypeId="2" ParentId="19889" CreationDate="2018-04-26T09:46:59.507" Score="1" Body="&lt;h3&gt;train.py code:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;builder = tf.saved_model.builder.SavedModelBuilder(&quot;/home/datam/cnn-text-classification-tf/model/20180423&quot;)&#xA;builder.add_meta_graph_and_variables(&#xA;    sess,&#xA;    [tf.saved_model.tag_constants.SERVING],&#xA;    clear_devices=True&#xA;)&#xA;builder.save()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h3&gt;Java Coding:&lt;/h3&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;SavedModelBundle model = SavedModelBundle.load(SAVED_MODEL_PATH+&quot;/20180423&quot;, &quot;serve&quot;);&#xA;Session s = model.session();&#xA;Graph g = model.graph();&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51188" LastEditorUserId="29575" LastEditDate="2018-04-26T12:58:10.390" LastActivityDate="2018-04-26T12:58:10.390" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30863" PostTypeId="2" ParentId="30856" CreationDate="2018-04-26T09:47:17.837" Score="1" Body="&lt;p&gt;You could try implementing a predictive model, like linear regression or a decision tree and then measure feature importance using the model. However, measuring feature importance is a slippery concept. &lt;a href=&quot;https://datascience.stackexchange.com/questions/30302/linear-regression-and-scaling-of-data/30485#30485&quot;&gt;This question&lt;/a&gt; illustrates that feature importance is not  trivially measured in regression models. &lt;a href=&quot;http://www.cs.cmu.edu/~atalwalk/dstump_nips17.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper&lt;/a&gt; shows how to measure importance of features in a decision tree. To me, these are richer measures of just linear correlation. &lt;a href=&quot;https://en.wikipedia.org/wiki/Distance_correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Distance correlation&lt;/a&gt; is another measure that does not need predictive models, and more meaningful than just linear correlation.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T09:47:17.837" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30864" PostTypeId="2" ParentId="12215" CreationDate="2018-04-26T10:31:20.787" Score="1" Body="&lt;p&gt;I have worked on a similar condition where I needed to separate each digit.&#xA;I have done this using Image segmentation, segmenting only the white pixel column wise.&lt;/p&gt;&#xA;" OwnerUserId="51192" LastEditorUserId="29575" LastEditDate="2018-04-26T13:01:31.623" LastActivityDate="2018-04-26T13:01:31.623" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30866" PostTypeId="2" ParentId="30859" CreationDate="2018-04-26T10:55:55.707" Score="1" Body="&lt;p&gt;I would not do that if your data is very different from the data in Imagenet. This is not typically the case, since Imagenet has lots of images representing many different things. However, let's say that your data comes from pictures taken by telescopes. In that case, even the most basic features (the very first layers) of a model trained on Imagenet will not be useful to your model, and in this case it might be more convenient to train the model from scratch using random initialization. One thing you can do, and that it might work better, is use the Imagenet model as your initial parameters and update &lt;em&gt;all the parameters&lt;/em&gt; of the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in most cases your images look like a subset of Imagenet, being transfer learning a very powerful technique.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T10:55:55.707" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30867" PostTypeId="1" CreationDate="2018-04-26T11:06:35.333" Score="1" ViewCount="66" Body="&lt;p&gt;I'm sure this is a standard problem in machine learning so links to pages or books that discuss the problem are also welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is: We have some input data points, $x_i$, and some output labels, $y_i$, that are drawn from a probability distribution whose parameters are somehow encoded in the input data. Now we want to learn a model, $f(x_i, \theta): x_i \rightarrow \lambda_i$ with some fit variables $\theta$, that outputs $\lambda_i$, which are the parameters of the distribution where $y_i$ are drawn from. The type of the distribution (e.g. normal, exponential, etc.) and number of its parameters is known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first idea would be to draw a new random variable from the distribution produced by $f$ for each sample, do this many times in batches and for each batch minimize $\sum_i (y^*_i - y_i)^2$. I can't really prove it, but my intuition tells me that this way the distribution parameters $\lambda_i$ should converge to those of the actual distribution.&lt;/p&gt;&#xA;" OwnerUserId="51198" LastActivityDate="2018-04-26T11:06:35.333" Title="Machine learning with stochastic labels" Tags="|optimization|supervised-learning|probability|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30868" PostTypeId="1" CreationDate="2018-04-26T11:19:22.560" Score="1" ViewCount="83" Body="&lt;p&gt;I have two different (regression) models spitting out predictions on a daily basis for the same dependent variable. My intention is to assign weights to those two predictions and calculate a weighted average. To this end, I have developed a simple system where the MSEs of the models are calculated and used as weights everyday. As a result, the higher the MSE a model has, the lower the weight it is assigned to the model. However, this is a very lame approach and I haven't observed any improvement compared to taking a simple average of the predictions. So what are the ways that I can use to assign weights to those predictions dynamically. That is, I want to update the weights everyday. From where should I start?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am aware that with ensemble models, I can get the weights automatically (i.e. H2O stacked ensemble). However, I am not allowed to retrain the predictions everyday. So applying an ensemble method and retrain it everyday is not applicable in my case.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="33335" LastEditorUserId="33335" LastEditDate="2018-04-26T11:26:23.847" LastActivityDate="2019-12-26T10:00:39.533" Title="How to construct a self learning process for an ensemble model?" Tags="|machine-learning|python|ensemble-modeling|ensemble-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30870" PostTypeId="1" CreationDate="2018-04-26T11:40:11.063" Score="2" ViewCount="40" Body="&lt;p&gt;I am new to jupyter notebook and i can't find my saved .ipynb code after logging out of jupyter notebook accidentally.&#xA;Thanks in anticipation&lt;/p&gt;&#xA;" OwnerUserId="51204" LastActivityDate="2018-04-26T14:48:31.017" Title="Unable to access saved code" Tags="|python|jupyter|scraping|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30871" PostTypeId="2" ParentId="30859" CreationDate="2018-04-26T12:11:31.237" Score="11" Body="&lt;p&gt;Based on my experience, not just for &lt;em&gt;ImageNet&lt;/em&gt;, if you have enough data it's better to train your network from scratch. There are numerous reasons that I can explain why.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;First of all, I don't know whether you have had this experience or not but I've trained complicated &lt;code&gt;CNNs&lt;/code&gt; neworks with over 25 million parameters. After reaching &lt;code&gt;95%&lt;/code&gt; accuracy, after convergence, I changed the learning rate to a bit bigger number to find another probable local minimum. I've not found any answer till today but whenever I did so, my accuracy decreased signifincantly all the time and it never imporoved even after more than thosands of epochs. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The other problem is that whenver you use transfer learning, your training data should have two options. First of all, the distribution of the training data which your pre-trianed model has used should be like the data that you are going to face during test time or at least don't vary too much. Second, the number of training data for transfer learning should be in a way that you don't overfit the model. If you have a few number of labeled training data, and your typical pre-trained model, like &lt;code&gt;ResNet&lt;/code&gt; has millions of parameters, you have to be aware of overfitting by choosing appropriate metrics for evaluation and good testing data which represent the real distribution of the real population.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Next problem is that you can not remove layers with confidence to reduce the number of parameters. Basically the number of layers, as you yourself know, is a hyper-parameter that there is no consensus on how to be chosen. If you remove the convolutional layers from the first layers, again based on experience, you won't have good learning because of the nature of the architecture which finds low level features. Furthermore, if you remove first layers you will have problem for your dense layers, because the number of trainable parameters changes in this way. Densely connected layers and deep convolutional layers can be good points for reduction but it may take time to find how many layers and neurons to be diminished inorder not to overfit.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you don't have enough data and there is already a pre-trained model, you can do something that can help you. I divide my answer to two parts for this situation:&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;The pre-trained model does not have common labels for most of its classes&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If this is the case, you can forget all the fully connected layers and replace new  ones. Basically what they do is just classifying the features that the network has found to reduce the error rate. About the convolutional layers you have to consider two major points:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Pooling layers try to sum up the information in a local neighbourhood and make a higher representation of the inputs. Suppose that the inputs of a pooling layer have nose, eyes, eybrows and so on. Your pooling layers somehow attempt to check whether they exist in a neighbourhood or not. Consequently, convolutional layers after pooling layers usually keep information which may be irrelevant to your task. There is a downside for this interpretation. The information may be distributed among different activation maps as Jason Yosinski &lt;em&gt;et al.&lt;/em&gt; have investigated this nature at &lt;a href=&quot;http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf&quot; rel=&quot;noreferrer&quot;&gt;Understanding neural networks through deep visualization&lt;/a&gt; where you can read &lt;em&gt;One of the most interesting conclusions so far has been that representations on &lt;strong&gt;some&lt;/strong&gt; layers seem to be surprisingly local. Instead of finding distributed representations on all layers, we see, for example, detectors for text, flowers, fruit, and faces on conv4 and conv5. These conclusions can be drawn either from the live visualization or the optimized images (or, best, by using both in concert) and suggest several directions for&#xA;future research&lt;/em&gt; and &lt;em&gt;These visualizations suggest that further study into the exact nature of learned representations—whether they are local to a single channel or distributed across several ...&lt;/em&gt;. A partial solution can be keeping the first layers which find low level features that are usually shared among different data distributions and removing deeper convolutional layers which find higher abstractions. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As stated, the main problem of convolutional layers is that the information they find may be distributed among different activation maps. Consequently, you can not be sure by removing a layer you can have better performance or not. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;The pre-trained model has some common classes with your labels&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;If this is the case you can visualize the activations using the techniques described &lt;a href=&quot;http://yosinski.com/deepvis&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. They have shown that although human face is not a label in &lt;code&gt;ImageNet&lt;/code&gt;, some of internal activation maps get activated by face. This observation can be seen for other labels too. For instance, netwroks for deciding a scene contains a car or not are usually sensetive to roads and trees. The image below shows that which parts of ouputs get activated for which parts of images. This can help you when you don't have enough data and you have to use transfer learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cUBJb.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cUBJb.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on the answer &lt;a href=&quot;https://stats.stackexchange.com/a/267242/179078&quot;&gt;here&lt;/a&gt; &lt;em&gt;The standard classification setting is an input distribution $p(X)$ and a label distribution $p(Y|X)$. Domain adaptation: when $p(X)$ changes between training and test. Transfer learning: when $p(Y|X)$ changes between training and test. In other words, in DA the input distribution changes but the labels remain the same; in TL, the input distributions stay the same, but the labels change.&lt;/em&gt; Consequently, domain adaption problems also can be considered for the mentioned solutions.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-14T13:29:58.830" LastActivityDate="2018-05-14T13:29:58.830" CommentCount="14" ContentLicense="CC BY-SA 4.0" />
  <row Id="30872" PostTypeId="1" CreationDate="2018-04-26T12:14:59.587" Score="1" ViewCount="3622" Body="&lt;p&gt;&lt;strong&gt;Contextualization&lt;/strong&gt;&lt;br&gt;&#xA;I am working on a multi_label classification problem with images. I am trying to predict 39 labels. In other words, I am trying to identifying which one of the 39 characteristics is present in a given image( many characteristics can be found in one image that's why I am on a multi label classification situation.&lt;br&gt;&#xA;&lt;strong&gt;Data&lt;/strong&gt;&lt;br&gt;&#xA;My input data are (X,Y): X is of shape  (1814,204,204,3) and Y is of shape (1814,39). So basically X are the set of images and Y are the labels associated to each images which will be used for the supervised learning process.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model&lt;/strong&gt;&lt;br&gt;&#xA;I am building a Convolutional Neural Network in order to make predictions. For this task, I am using Keras in order to create my model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have done&lt;/strong&gt;&lt;br&gt;&#xA;In order to validate my model, I need to choose a &lt;strong&gt;metric&lt;/strong&gt;. However, metrics available in Keras are irrelevant in my case and won't help me validate my model since I am in multi-label classification situation. That's why I decided to create my custom metric. I created recall and precision metrics applied to columns of Y and Y_predict . In other words, I will calculate recall and precision for each class of the 39 classes. So here is the code of my metrics:&#xA;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def recall(y_true, y_pred):&#xA;    #Recall metric.&#xA;&#xA;    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)),axis=0)&#xA;    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)),axis=0)&#xA;    recall = true_positives / (possible_positives + K.epsilon())&#xA;    return recall&#xA;&#xA;def precision(y_true, y_pred):&#xA;    #Precision metric.&#xA;&#xA;    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)),axis=0)&#xA;    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)),axis=1)&#xA;    precision = true_positives / (predicted_positives + K.epsilon())&#xA;    return precision&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My vector Y is of shape (n,39) that's why I am doing operations over axis=0. In other words, for each label, I am calculating precision and recall.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next step, I called these two metrics by precising it to Keras fit function. In other words I used this line of  code:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=[precision,recall])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code for building, compiling and fitting model:&lt;/strong&gt;&lt;br&gt;&#xA;Here is the code that I use for building the model an training it + its result . ( I am not putting the part of code where I split data into train and validation: &lt;em&gt;Train on 1269 samples, validate on 545 samples&lt;/em&gt;)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Model: CNN&#xA;model = Sequential()&#xA;model.add(Conv2D(32, (3, 3), input_shape=(204, 204, 3), padding='same', activation='relu', kernel_constraint=maxnorm(3)))&#xA;model.add(Dropout(0.2))&#xA;model.add(Conv2D(32, (3, 3), activation='relu', padding='same', kernel_constraint=maxnorm(3)))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Flatten())&#xA;model.add(Dense(512, activation='relu', kernel_constraint=maxnorm(3)))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(39))&#xA;model.add(Activation('sigmoid'))&#xA;# Compile model&#xA;epochs = 5&#xA;lrate = 0.001&#xA;decay = lrate/epochs&#xA;sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)&#xA;model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=[precision,recall])&#xA;&#xA;# fitting the model &#xA;model.fit(X_train, Y_train, epochs=epochs, batch_size=32,validation_data=(X_valid,Y_valid))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Results&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train on 1269 samples, validate on 545 samples&#xA;Epoch 1/5&#xA;  96/1269 [=&amp;gt;............................] - ETA: 6:40 - loss: 0.6668 - precision: 0.1031 - recall: 0.2493&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Issues/Questions&lt;/strong&gt;&lt;br&gt;&#xA;&lt;strong&gt;&lt;em&gt;Question 1:&lt;/em&gt;&lt;/strong&gt; In the log of the results section , there are precision and recall values. I don't know why I got real values instead of vector of values. The way I constructed my two metrics should give me an array of shape (1,39) for precision and (1,39) for recall, which should contain precision and recall for each class, still The output is only a number?&lt;br&gt;&#xA;&lt;strong&gt;&lt;em&gt;Question 2:&lt;/em&gt;&lt;/strong&gt;  these values of precision and recall given by the log, they represent  metric calculation for a data of size= batch? How can I calculate the metric over an epoch ( which is more useful as an information than just a calculation over a batch? Some may say, just calculate the average over all batches? Sure , that's what I am thinking of but I don't know how to do it since Keras is kinda of a black box to me and I don't exactly what is happening ''behind the scenes'' in order to follow/modify the adequate part of code?&lt;/p&gt;&#xA;" OwnerUserId="51208" LastEditorUserId="29575" LastEditDate="2018-04-26T13:12:58.207" LastActivityDate="2018-04-26T13:12:58.207" Title="Multi-label classifciation: keras custom metrics" Tags="|machine-learning|deep-learning|keras|tensorflow|convolution|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30873" PostTypeId="2" ParentId="30870" CreationDate="2018-04-26T12:16:22.747" Score="1" Body="&lt;p&gt;I guess you have made a file in your default document folder. A probable location for your notebook is where it starts if you have not changed the default path or maybe the location in which your files is located. Try to rerun your jupyter from your terminal or command prompt. It will open the contents of the folder you are looking for your notebook.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-26T12:16:22.747" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30875" PostTypeId="2" ParentId="12215" CreationDate="2018-04-26T12:53:02.680" Score="2" Body="&lt;p&gt;Actually this is called &lt;code&gt;text line extraction&lt;/code&gt;. What I'm going to tell you is inspired from the lectures of &lt;a href=&quot;https://scholar.google.com/citations?user=i9zCOJEAAAAJ&amp;amp;hl=en&amp;amp;oi=ao&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; scientist. For finding digits you don't need to design a network. You should extract them and then feed them to a network in turn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all you have to read your image.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;import numpy as np&#xA;from matplotlib import pyplot as plt&#xA;%matplotlib inline&#xA;img = cv2.imread('./doc1.png')&#xA;plt.imshow(img)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Rj7Wl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Rj7Wl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you have to make your image as a binary array.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;img.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(2360, 1649, 3)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;img = cv2.imread('./doc1.png',0)&#xA;img.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(2360, 1649)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.imshow(img)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rvdzP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rvdzP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.imshow(img,cmap = 'gray')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8fKmW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8fKmW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following code shows some of the lines of the document:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.imshow(img[900:1020,500:900],cmap = 'gray')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ppaLK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ppaLK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;bimg=cv2.cvtColor(img[900:1020,500:900],cv2.COLOR_GRAY2RGB)&#xA;bimg.shape&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(120, 400, 3)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Next, you have to find the lines in the image, then you have to find the characters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZYrag.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZYrag.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/upjyh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/upjyh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IIWk4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IIWk4.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ss47c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ss47c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6IF1c.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6IF1c.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BBD8e.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BBD8e.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kw09f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kw09f.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fmj9K.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fmj9K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JBL3y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JBL3y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WHdbv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WHdbv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/dKhmC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dKhmC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SPcHV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SPcHV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GlfYH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GlfYH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LRlk6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LRlk6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically the code was completely clear to me. If you don't understand it, let me know.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-26T12:53:02.680" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30876" PostTypeId="1" CreationDate="2018-04-26T12:57:36.527" Score="5" ViewCount="1582" Body="&lt;p&gt;I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;, &lt;code&gt;RandomForestClassifier&lt;/code&gt; and a &lt;code&gt;GradientBoostingClassifier&lt;/code&gt; using a randomized search and a bayesian search.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I used just &lt;code&gt;accuracy&lt;/code&gt; for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that for example &lt;code&gt;recall_micro&lt;/code&gt; and &lt;code&gt;recall_weighted&lt;/code&gt; yield the same results as &lt;code&gt;accuracy&lt;/code&gt;. This should be the same for other metrics like &lt;code&gt;f1_micro&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;So my question is&lt;/strong&gt;: Is the scoring relevant for tuning? I see that &lt;code&gt;recall_macro&lt;/code&gt; leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?&lt;/p&gt;&#xA;" OwnerUserId="50402" LastEditorUserId="83275" LastEditDate="2019-11-20T12:16:20.793" LastActivityDate="2022-03-19T21:23:48.753" Title="Hyperparameter tuning in multiclass classification problem: which scoring metric?" Tags="|machine-learning|classification|model-evaluations|hyperparameter-tuning|" AnswerCount="4" CommentCount="11" ContentLicense="CC BY-SA 4.0" />
  <row Id="30877" PostTypeId="1" CreationDate="2018-04-26T13:31:47.240" Score="2" ViewCount="263" Body="&lt;p&gt;A sentence (Segmented from a document) as below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;This Amendment dated 26th of April 2018 modifies the Agreement dated 20th April 2017.&#xA;or&#xA;This Amendment depends on the agreement dated 20th April 2017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to extract the following as a CSV&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Document_type | Date_of_document | relation | related_doc | related_doc_date&#xA;----------------------------------------------------------------------------&#xA;Amendment     | 26-04-2018       | modifies | Agreement   | 20-04-2017&#xA;Amendment     | Null             | depends  | agreement   | 20-04-2017&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What have I done till now&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Found and extracted dates using Stanford NER. Got them in a normalized form. Can extract Noun Phrases using TextBlob. But cannot link which Noun Phrase is linked to which date. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I doing it wrong? Any tutorials/examples/advice?&lt;/p&gt;&#xA;" OwnerUserId="7725" LastActivityDate="2018-04-26T13:31:47.240" Title="Extracting date, relation and noun phrase from text" Tags="|nlp|text-mining|information-retrieval|nltk|stanford-nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30878" PostTypeId="2" ParentId="30381" CreationDate="2018-04-26T13:44:44.240" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://web.stanford.edu/class/cs124/lec/naivebayes.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; is almost always my first go to for a problem with text data. This is primarily due to the independent distributions that integrate so well with the &lt;em&gt;document by feature &lt;/em&gt; independent style matrix by nature. I would extend your idea of &lt;em&gt;Ridge&lt;/em&gt; with &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lasso&lt;/a&gt; or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;ElasticNet&lt;/a&gt;. I would not recommend normalize your data if it is sparse, this has been known to cause problems, but &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler&quot; rel=&quot;nofollow noreferrer&quot;&gt;StandardScaler&lt;/a&gt; is a rather conventional way to get a centered dataset without producing disharmony among the variables in the data. I would consider your error metric of precision or fmeasure under each of the above models as my personal next step, then do some visualizations to argue your best fit.&lt;/p&gt;&#xA;" OwnerUserId="51210" LastActivityDate="2018-04-26T13:44:44.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30879" PostTypeId="1" AcceptedAnswerId="30902" CreationDate="2018-04-26T14:11:26.540" Score="1" ViewCount="813" Body="&lt;p&gt;If I'm not mistaken, in this paper here &lt;a href=&quot;http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf&lt;/a&gt; it appears that topics with larger number of topics will inherently have larger coherence measures because of the summation over all the topics. Is there any flaw in dividing this coherence calculation by the number of topics to get an average for a model of 5 topics and then for a model of 10 topics and comparing the two? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In gensim specifically, I'm using c_v coherence calculation.&lt;/p&gt;&#xA;" OwnerUserId="51211" LastActivityDate="2018-04-26T19:59:44.280" Title="How to compare the topic coherence between models of different number of topics?" Tags="|topic-model|lda|gensim|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30880" PostTypeId="2" ParentId="27703" CreationDate="2018-04-26T14:22:29.510" Score="1" Body="&lt;p&gt;&lt;strong&gt;Using a multi-class network would probably be the most efficient approach.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training two networks (as suggested in the comments: one network to detect the presence of a person and one to detect the state of the knee) is unnecessary and overly complex.  You can train the network on three classes: bent knee, straight knee, absent knee.  For the absent knee, &lt;strong&gt;you do not need to train on &lt;em&gt;all&lt;/em&gt; possible images, just images you expect to see.&lt;/strong&gt;  For instance, these negative data could be the background of your intended images; take a picture with the knee and one without.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If speed is a major requirement, the model architecture becomes critical.  I suggest exploring &lt;a href=&quot;https://arxiv.org/abs/1704.04861&quot; rel=&quot;nofollow noreferrer&quot;&gt;MobileNets&lt;/a&gt; as they are quite fast and do not sacrifice too much accuracy.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-04-26T14:22:29.510" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30881" PostTypeId="1" AcceptedAnswerId="30882" CreationDate="2018-04-26T14:31:01.753" Score="46" ViewCount="89085" Body="&lt;p&gt;Can anyone give me some examples where precision is important and some examples where recall is important?&lt;/p&gt;&#xA;" OwnerUserId="50247" LastEditorUserId="29169" LastEditDate="2021-02-08T02:13:30.417" LastActivityDate="2021-11-17T18:53:17.903" Title="When is precision more important over recall?" Tags="|machine-learning|model-evaluations|" AnswerCount="10" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30882" PostTypeId="2" ParentId="30881" CreationDate="2018-04-26T14:40:51.927" Score="66" Body="&lt;ul&gt;&#xA;&lt;li&gt;For rare cancer data modeling, anything that doesn't account for false-negatives is a crime. &lt;strong&gt;Recall&lt;/strong&gt; is a better measure than precision.&lt;/li&gt;&#xA;&lt;li&gt;For YouTube recommendations, false-negatives is less of a concern. &lt;strong&gt;Precision&lt;/strong&gt; is better here.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="9123" LastEditorUserId="50247" LastEditDate="2018-04-26T14:54:00.163" LastActivityDate="2018-04-26T14:54:00.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30883" PostTypeId="2" ParentId="30870" CreationDate="2018-04-26T14:41:54.497" Score="0" Body="&lt;p&gt;The document will be at the place you started jupyter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you’re on Linux and don’t remember where you started jupyter, you can try &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sudo updatedb &#xA;locate ‘.ipynb’&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It will search for ipynb files.&#xA;Otherwise try to search for ipynb files from the root folder of your OS.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastEditorUserId="41728" LastEditDate="2018-04-26T14:48:31.017" LastActivityDate="2018-04-26T14:48:31.017" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30884" PostTypeId="2" ParentId="30881" CreationDate="2018-04-26T14:56:31.647" Score="21" Body="&lt;p&gt;I can give you my real case when recall is more important:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We have thousands of free customers registering in our website every week. The call center team wants to call them all, but it is imposible, so they ask me to select those with good chances to be a buyer (with high temperature is how we refer to them). We don't care to call a guy that is not going to buy (so precision is not important) but for us is very important that all of them with high temperature are always in my selection, so they don't go without buying. That means that my model needs to have a &lt;strong&gt;high recall&lt;/strong&gt;, no matter if the precision goes to hell.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope it helps!&#xA;Miguel.&lt;/p&gt;&#xA;" OwnerUserId="51216" LastActivityDate="2018-04-26T14:56:31.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30885" PostTypeId="2" ParentId="30810" CreationDate="2018-04-26T15:03:03.463" Score="0" Body="&lt;pre&gt;&lt;code&gt;#x = tf.reshape(x, shape=[-1, 299, 299, 3])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;change the above line to the following code snippet.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dim = x.get_shape().as_list()&#xA;x = tf.reshape(x, shape = [-1, *dim[1:]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The same behavior applies for &lt;code&gt;z&lt;/code&gt; and &lt;code&gt;Z1&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-04-26T15:20:01.400" LastActivityDate="2018-04-26T15:20:01.400" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30886" PostTypeId="2" ParentId="30881" CreationDate="2018-04-26T15:08:10.853" Score="9" Body="&lt;p&gt;Although in some situations recall may be more important than precision (or vice versa), &lt;strong&gt;you need both to get a more interpretable assessment.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, as noted by @SmallChess, in the medical community, a false negative is usually more disastrous than a false positive for preliminary diagnoses.  Therefore, one might consider recall to be a more important measurement.  However, you could have 100% recall yet have a useless model: if your model always outputs a positive prediction, it would have 100% recall but be completely uninformative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why we look at multiple metrics:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.quora.com/What-is-Precision-Recall-PR-curve&quot; rel=&quot;noreferrer&quot;&gt;precision-recall curve&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;noreferrer&quot;&gt;AUROC&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;and sometimes metrics such as the &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;noreferrer&quot;&gt;F score&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-26T15:15:42.783" LastActivityDate="2018-04-26T15:15:42.783" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30888" PostTypeId="2" ParentId="24310" CreationDate="2018-04-26T15:23:59.217" Score="2" Body="&lt;p&gt;Great question, I've been checking back to see if anyone responds but nothing yet, so I'll do my best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like me, you have also found out that &lt;strong&gt;simply adding more and more training data does not increase the model’s accuracy&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to make sure your model &lt;strong&gt;contains balanced, representative samples for each intent&lt;/strong&gt;. Simply picking utterances at random means they may not be represent all of the types of language a user may use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, let’s say you had two intents - “buy_food” for buying some food,  and “list_menu”  for finding out what is available for order. If you were to heavily populate buy_food with lots of utterances that begin with “I want to”, the NLP engine will associate the phrase “I want to” with buying food, even though someone may quite rightly say “I want to see what’s on the menu”. To overcome this, make sure you balance out each intent’s utterances and don’t focus on phrases that aren’t specific to the intent. In this case, you could counterbalance by having some utterances in your “list_menu” intent such as “I want to see what’s on the menu”.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a rule, for ea&lt;strong&gt;ch intent I would try and include up to 5 or 6 ways to say something&lt;/strong&gt;, and then for each of these, &lt;strong&gt;include 4 or 5 variations of this&lt;/strong&gt;, leaving you with &lt;strong&gt;about 25 samples for an intent&lt;/strong&gt; in total. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is also the question around how you plan your intents. For example, do you have a “buy_food” intent and a “buy_drinks” intent, or should you simply have a “buy” intent, and then have the thing they want to buy as an entity? (think of your intent as a method, and your entity as a parameter) My suggestion would be to &lt;strong&gt;keep your intents broader and have the specifics handled by entities&lt;/strong&gt;. This will help avoid confusion between intents.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend checking out a tool called QBOX. It connects to LUIS/Watson/DialogFlow and lets you score and visualise intents so you can see any possible confusion between them. This should help you to plan your intent structure and visualise the confusion between your intents, and know if you're causing regressions as you expand your model. &#xA;&lt;em&gt;(Disclaimer: this is a product I am working on, and built precisely because of the type of problem you're having, so I've not linked to it, but if you Google the name along with the term 'chatbot', you should find it!)&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="49524" LastActivityDate="2018-04-26T15:23:59.217" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30889" PostTypeId="1" AcceptedAnswerId="30899" CreationDate="2018-04-26T15:44:01.290" Score="2" ViewCount="62" Body="&lt;p&gt;When i print this in the module without printing to file, it prints all values. But when I print it to the designated file, it only prints the first value. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Create the model base&#xA;from sklearn.feature_selection import chi2&#xA;import numpy as np&#xA;N = 2&#xA;for UNSPSC, category_id in sorted(category_to_id.items()):&#xA;features_chi2 = chi2(features, labels == category_id)&#xA;indices = np.argsort(features_chi2[0])&#xA;feature_names = np.array(tfidf.get_feature_names())[indices]&#xA;unigrams = [v for v in feature_names if len(v.split(' ')) == 1]&#xA;bigrams = [v for v in feature_names if len(v.split(' ')) == 2]&#xA;&#xA;with open('results.csv', 'w') as f:&#xA; print(&quot;# '{}':&quot;.format(UNSPSC), file=f)&#xA; print(&quot;  . Most correlated unigrams:\n. {}&quot;.format('\n. '.join(unigrams[-N:])), file=f)&#xA; print(&quot;  . Most correlated bigrams:\n. {}&quot;.format('\n. '.join(bigrams[-N:])), file=f)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How can i adjust this so that all values get printed onto the file? &lt;/p&gt;&#xA;" OwnerUserId="50899" LastEditorUserId="381" LastEditDate="2018-04-26T16:21:20.230" LastActivityDate="2018-04-27T00:39:44.953" Title="Do I need a next for loop in order to get all values?" Tags="|python|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30890" PostTypeId="2" ParentId="25124" CreationDate="2018-04-26T15:44:37.843" Score="2" Body="&lt;p&gt;There is a Python library called &lt;a href=&quot;https://dateparser.readthedocs.io/en/latest/&quot; rel=&quot;nofollow noreferrer&quot;&gt;dateparser&lt;/a&gt; that will accept a wide variety of formats, including relative dates like &quot;next Tuesday&quot;, and return exact &lt;code&gt;datetime&lt;/code&gt; representation. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-26T15:44:37.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30891" PostTypeId="1" CreationDate="2018-04-26T15:55:27.297" Score="1" ViewCount="44" Body="&lt;p&gt;i have a data set with personal properties (age, education etc.), and each entry in the dataset is associated with a list of websites and daily visit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;my goal is to predict for a given person the list of websites he will visit, and to predict, for each website, how many daily visit it will have from that person.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the number of websites each person visits is very limited, usually between 3 and 20 (same websites for the whole dataset).&#xA;the number of different persons is limited (a few hundreds) but the number of entries per person is large (a few thousands, each representing a day).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;currently my thoughts are to have two stage method - &#xA;first stage is for each possible website a person might visit (like mentioned before, usually 3-20), classify that website as &quot;visited&quot; or &quot;not visited&quot; (using binary classifiers for each website).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;stage two is for each website that was classified as &quot;visited&quot;, use a linear regression to predict the number of daily visits.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am very new to the field of ML so dont spare your words! your comments and tips are very appreciated :)&#xA;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="41025" LastActivityDate="2018-04-26T15:55:27.297" Title="Choosing the right prediction method" Tags="|classification|regression|prediction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30893" PostTypeId="2" ParentId="23423" CreationDate="2018-04-26T15:56:45.027" Score="2" Body="&lt;p&gt;Give a shot to &lt;a href=&quot;https://dataturks.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataTurks&lt;/a&gt; &#xA;It gives you simply slick UI + web based collaborative framework to work with your colleagues/mates to build the dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image from a sample Project which classifies images to celberity&#xA;&lt;a href=&quot;https://i.stack.imgur.com/5zzzR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5zzzR.png&quot; alt=&quot;Image from a sample Project which classifies images to celberity&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some more sample projects:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dataturks.com/projects/trending&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://dataturks.com/projects/trending&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="26581" LastActivityDate="2018-04-26T15:56:45.027" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30894" PostTypeId="2" ParentId="30312" CreationDate="2018-04-26T16:26:14.120" Score="4" Body="&lt;p&gt;The main difference between Linear Regression and Tree-based methods is that Linear Regression is parametric: it can be writen with a mathematical closed expression depending on some parameters. Therefore, the coefficients are the parameters of the model, and should not be taken as any kind of importances unless the data is normalized. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, non-parametric methods have very different ways of measure the importance. In layman terms, measuring the importance of variables on a tree can be done by checking how close they appear to the root node. In ensembling methods, like bagging, one can compute the importance of a variable as the average among the ensemble, like in this &lt;a href=&quot;https://stackoverflow.com/questions/44333573/feature-importances-bagging-scikit-learn&quot;&gt;stackoverflow answer.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main difference is, then, the fact that parametric models have, through their parameters, a way of showing the importance of the variables, while non parametric models need some extra work.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T16:26:14.120" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30896" PostTypeId="2" ParentId="30868" CreationDate="2018-04-26T17:30:21.407" Score="1" Body="&lt;p&gt;It is not much more sofisticated than what you thought, but you can try training a linear regression (in the online setting, by stochastic gradient descent) whose inputs are the predictions spit by your models and the output is the real value of the predicted variable. In this case, you wouldn't need to retrain with the whole dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, I think that all &lt;a href=&quot;https://en.wikipedia.org/wiki/Online_machine_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;online machine learning&lt;/a&gt; techniques can be applied in this setting.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T17:30:21.407" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30898" PostTypeId="1" CreationDate="2018-04-26T17:57:21.873" Score="0" ViewCount="104" Body="&lt;p&gt;I'm following this &lt;a href=&quot;https://medium.com/@awjuliani/super-simple-reinforcement-learning-tutorial-part-2-ded33892c724&quot; rel=&quot;nofollow noreferrer&quot;&gt;Reinforcement Learning Tutorial&lt;/a&gt;. There, training data is collected during an episode. When the episode is done, the data is used to do backpropagation. However, instead of applying the calculated gradients to the model, they are stored in a buffer. The gradients of multiple episodes get summed up before they are applied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder if that is a reasonable thing to do. Usually, if the model is very good after training with the first episodes data, the gradients of the other episodes will be close to 0 (no change). If you used the tutorials code, the gradients of all episodes would be large, and the model would move away from the optimum. Am I missing something?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: I tried &lt;a href=&quot;https://pastebin.com/buhjgsGt&quot; rel=&quot;nofollow noreferrer&quot;&gt;buffering gradients with Q-Learning&lt;/a&gt; and compared it to directly updating the weights. It is working, but buffering is definitely worse.&lt;/p&gt;&#xA;" OwnerUserId="39281" LastEditorUserId="39281" LastEditDate="2018-04-28T01:27:02.850" LastActivityDate="2018-04-28T01:27:02.850" Title="Can you sum up gradients and apply in batch?" Tags="|machine-learning|tensorflow|backpropagation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30899" PostTypeId="2" ParentId="30889" CreationDate="2018-04-26T18:23:09.863" Score="3" Body="&lt;p&gt;&lt;strong&gt;Don't forget to open the file first..&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Py 2.6&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;For item in list:&#xA;    the_opened_file.write(&quot;%s\n&quot; % item)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Py 3.x&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;with open(filepath,'w') as fileptr:&#xA;    for item in list:&#xA;        fileptr.write(&quot;{}\n&quot;.format(item))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to save your list itself,&#xA;Convert it to numpy format using &lt;code&gt;np.assarray(list)&lt;/code&gt; and then use &lt;code&gt;np.save(file,array,....(other opts)&lt;/code&gt; and reload it whenever required using &lt;code&gt;np.load()&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-04-27T00:39:44.953" LastActivityDate="2018-04-27T00:39:44.953" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30900" PostTypeId="1" CreationDate="2018-04-26T18:54:59.617" Score="1" ViewCount="64" Body="&lt;p&gt;Is correct to get a bit lower AUC with bagged algorithms than no bagged algorithms? The first figure shows the ROC with bagged algorithms and the second figure shows the ROC without bagging. The curves represents the mean function in a 10-fold cross validation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3GU1D.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3GU1D.png&quot; alt=&quot;ROC bagged algorithms&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/n8oEq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/n8oEq.png&quot; alt=&quot;ROC algorithms&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As expected the mean accuracy of a 10-fold cross validation with bagged algorithms (1st Figure below) is higher than without bagging (2nd Figure below):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AoAcV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AoAcV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UGeA8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UGeA8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought that bagging algorithms performs better than without bagging. Am I wrong?&lt;/p&gt;&#xA;" OwnerUserId="47064" LastActivityDate="2018-04-26T18:54:59.617" Title="AUC for basic models is higher than bagged models" Tags="|python|scikit-learn|random-forest|svm|cross-validation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30901" PostTypeId="1" AcceptedAnswerId="30903" CreationDate="2018-04-26T19:01:34.050" Score="1" ViewCount="157" Body="&lt;p&gt;Is there any possibility that cost function might end up in local minima rather than global minima while implementing the Neural network using Tensorflow ?&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="50727" LastEditDate="2020-07-14T06:54:40.133" LastActivityDate="2020-07-14T06:54:40.133" Title="Neural network using Tensorflow" Tags="|neural-network|deep-learning|tensorflow|optimization|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30902" PostTypeId="2" ParentId="30879" CreationDate="2018-04-26T19:59:44.280" Score="0" Body="&lt;p&gt;After doing some more digging -- it appears that gensim's functions output difference calculations &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This below actually does what I am describing above:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cm = CoherenceModel(model=model, texts=docs, dictionary=dictionary, coherence='c_v')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;While this below is actually only the summation across the topics of the model not divided by the total number of topics in the model. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;top_topics = model.top_topics(texts=docs,dictionary=dictionary,coherence='c_v')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51211" LastActivityDate="2018-04-26T19:59:44.280" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30903" PostTypeId="2" ParentId="30901" CreationDate="2018-04-26T20:20:30.473" Score="1" Body="&lt;p&gt;Most of the critical points in a neural network are not local minima, as it can be seen in &lt;a href=&quot;https://datascience.stackexchange.com/q/22853/50727&quot;&gt;this question&lt;/a&gt;. Although it is not impossible to fall into a local minimum, the probability of it happening is so low that in practice it does not happen, except from very special cases as a single-layer perceptron. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Being local minima so hard to find, this means that it is highly unlikely to come across the global minimum in your optimization method. All that we do in deep learning is decrease the loss function to find fairly good parameters, but finding local and global minima is extremely unlikely.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-26T20:20:30.473" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30904" PostTypeId="1" CreationDate="2018-04-26T20:42:53.217" Score="2" ViewCount="3054" Body="&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/13746/how-to-define-a-custom-performance-metric-in-keras&quot;&gt;How to define a custom performance metric in Keras?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to use it but I can not see the metrics values on each epoch.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clf51.compile(optimizer=sgd51, loss='binary_crossentropy', metrics=[&quot;accuracy&quot;])&#xA;&#xA;clf51.fit(X_train, Y_train, batch_size=384, epochs=5, callbacks=[metrics], validation_split=0.30, verbose=2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Train on 139554 samples, validate on 59810 samples&#xA;&#xA;Epoch 1/5 - 4s - loss: 0.3576 - acc: 0.9885 - val_loss: 0.0531 - val_acc: 0.9989&#xA;&#xA;Epoch 2/5 - 3s - loss: 0.0261 - acc: 0.9987 - val_loss: 0.0135 - val_acc: 0.9987&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Am I doing something wrong? I would like to make a earlystop using f1s.&lt;/p&gt;&#xA;" OwnerUserId="51167" LastEditorUserId="26562" LastEditDate="2018-04-26T20:45:44.043" LastActivityDate="2020-02-15T06:01:30.343" Title="Custom Metrics with Keras" Tags="|python|keras|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30905" PostTypeId="1" CreationDate="2018-04-26T20:57:46.517" Score="5" ViewCount="2862" Body="&lt;p&gt;I have a few basic questions about tracking losses during training. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If I am using mini-batch training, should I validate after each&#xA;batch update or after I have seen the entire dataset?&lt;/li&gt;&#xA;&lt;li&gt;What should be the condition to stop the training to prevent overfitting? Do you save the model at that point?&lt;/li&gt;&#xA;&lt;li&gt;In case I use mini-batch training losses fluctuate a lot, depending on the random choice of training data, and sometimes validation loss is less than training loss. Is this normal? I think my confusion about this point will be answered in 1 by itself. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1165" LastEditorUserId="50727" LastEditDate="2018-05-22T14:21:07.373" LastActivityDate="2018-05-22T14:21:07.373" Title="Train loss vs validation loss" Tags="|machine-learning|cross-validation|training|loss-function|mini-batch-gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30907" PostTypeId="2" ParentId="30905" CreationDate="2018-04-26T21:15:05.627" Score="2" Body="&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Both approaches can be done: I recommend to validate after every batch when your are just playing around with your gradient method, to see if the validation accuracy goes up or down and to figure out how everything is going.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;In this setting, you can adopt &lt;a href=&quot;https://en.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;early stopping&lt;/a&gt;, although there are many ways to prevent from overfitting. Early stopping checks, at the end of an epoch, your validation accuracy, and saves the model if it is the best so far. If the validation accuracy does not increase in the next $n$ epochs (and here $n$ is a parameter that you can decide), then you keep the last model you saved and stop your gradient method.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Validation loss can be lower than training loss, this happens sometimes. In this case,  you can state that you are not overfitting.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-04-27T06:49:57.907" LastActivityDate="2018-04-27T06:49:57.907" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="30908" PostTypeId="2" ParentId="30762" CreationDate="2018-04-26T22:56:50.853" Score="8" Body="This answer goes a little bit in a different direction, but I hope it still answers your question. It uses the idea of a rolling forecast/prediction.&#xA;&#xA;&lt;p&gt;Because you use the word &lt;em&gt;horizon&lt;/em&gt;, I will assume you mean that you would like to predict 10 days into the future at a given time step. There are a few ways of doing this. With this kind of time-series problem, it is common to make the assumption that only a certain history will influence the next few time steps (neglecting seasonal effects). &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example in words:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;So in your case, you might use e.g. the previous 60 days, and predict the next 10.&#xA;Taking your 100 rows of data as an example, this means you can actually make &lt;code&gt;(100 - 60 - 9) = 31&lt;/code&gt; predictions, each prediction of 10 time steps ahead (we will need these 31 &lt;strong&gt;predictive_blocks&lt;/strong&gt; later). From 100 rows we lose the first 60 to fit the first model. Of the remaining 40 rows of data, we can predict 10 steps ahead (rows 61-70), then we shift the whole thing one row further and repeat. The last prediction of 10 future points would be for rows 91-100. After that we cannot predict 10 steps anymore, so we stop - and this is why we have to subtract that extra 9. [There are of course ways to continue making prediction, as to use all the data]&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Example with a thousand words:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Let me paint the picture; to help explain the idea of a shifting window prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For each train set (e.g. from &lt;code&gt;t=0&lt;/code&gt; to &lt;code&gt;t=5&lt;/code&gt; in red - train set 1), you want to predict the following H time steps (corresponding to t=6 in orange - test set 1). In this, your horizon is simply one i.e. &lt;code&gt;H=1&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Iz3ij.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Iz3ij.png&quot; alt=&quot;Basic sketch of a rolling out-of-sample forecast&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From what I understand, you would like to predict the next 10 days, meaning you need &lt;code&gt;H=10&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to try this with your example, I think you will need to make two changes.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Change #1&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;The shape of your train and test sets will need to match the new horizon. Each sample of your model input (the &lt;code&gt;x_train&lt;/code&gt; and &lt;code&gt;x_test&lt;/code&gt; can stay the same as before. However, each sample in your test set will have to contain the next &lt;code&gt;H=10&lt;/code&gt; values of the label, not just a single value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a rough example of how you might do this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Define our horizon&#xA;H = 10&#xA;&#xA;# Create data split, using values from my example above&#xA;window_size = 60&#xA;num_pred_blocks = 31    # as computed above&#xA;&#xA;# Loop over the train and test samples to create the sliding window sets&#xA;x_train = []&#xA;y_train = []&#xA;for i in range(num_pred_blocks):&#xA;    x_train_block = x_train[i:(i + window_size)]    # 31 blocks of 60 * num-columns&#xA;    x_train.append(x_train_block)&#xA;    y_train_block = y_train[(i + window_size):(i + window_size + H)]    # 31 blocks of 10 * 1&#xA;    y_train.append(y_train_block)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because you are doing out-of-sample testing, your predictions are already interesting to look analyse. Once this runs, you can then create the equivalent test datasets with the new data you mentioned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing your data too well, I don't know if your should be predicting the y-values of the same row as the input, or of the following row. Additionally, depending on your data, you could be including the past values of &lt;code&gt;y&lt;/code&gt; in each of the &lt;code&gt;x_train&lt;/code&gt; blocks. In this case you'd simply swap &lt;code&gt;x&lt;/code&gt; for the whole table i.e. &lt;code&gt;data[cols]&lt;/code&gt;, where &lt;code&gt;new_cols = ['Demand'] + cols&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Change #2&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;You will need to make the model reflect this horizon, by forcing it to output &lt;code&gt;H&lt;/code&gt; values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is an example of how to specify the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Define our horizon&#xA;H = 10&#xA;&#xA;# Create the model using the parameterised horizon&#xA;fit1 = Sequential ()&#xA;fit1.add(LSTM(output_dim = 4, activation='tanh', input_shape =(4, 1)))&#xA;fit1.add(Dense(output_dim=30, activation='sigmoid')&#xA;fit1.add(Dense(output_dim=H))    # our horizon is produced!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt;&#xA;In your model specification, you don't need to add the final linear &lt;code&gt;Activation&lt;/code&gt;, as the preceding Dense layer by default includes a linear activation. See &lt;a href=&quot;https://keras.io/layers/core/#dense&quot; rel=&quot;noreferrer&quot;&gt;the excellent documentation here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a big topic and there are many things that you could try out. I agree with the comments on your question, that you will need a lot more data to allow an RNN to make a meaning representation of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are not just doing this to learn about LSTMs etc., another practical approach might be to look into simpler time-series models such as an &lt;a href=&quot;https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average&quot; rel=&quot;noreferrer&quot;&gt;ARIMA model&lt;/a&gt; (do not be intimidated by the complicated name - it is much simpler than an LSTM). Such models can be constructed quite easily with Python, using &lt;a href=&quot;http://www.statsmodels.org/devel/index.html&quot; rel=&quot;noreferrer&quot;&gt;the statsmodels package&lt;/a&gt;, which has &lt;a href=&quot;http://www.statsmodels.org/devel/generated/statsmodels.tsa.arima_model.ARIMA.html#statsmodels.tsa.arima_model.ARIMA&quot; rel=&quot;noreferrer&quot;&gt;a nice implementation&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="-1" LastEditDate="2018-11-11T16:57:03.917" LastActivityDate="2018-11-11T16:57:03.917" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30909" PostTypeId="2" ParentId="30904" CreationDate="2018-04-26T23:03:50.493" Score="1" Body="&lt;p&gt;The callback you are using isn't for displaying the desired metrics, just recording them. For example if you want to access the F1-score you need to type: &lt;code&gt;metrics.f1s&lt;/code&gt;. This is useful, let's say, if you want to make a graph on how the F1-score reduced during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use the &lt;a href=&quot;https://keras.io/callbacks/#earlystopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;EarlyStopping&lt;/a&gt; callback, however, f1-score needs to be a &lt;strong&gt;metric&lt;/strong&gt; not a &lt;strong&gt;callback&lt;/strong&gt; like you have it! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to write (or find) a function that calculates the F1-score through keras' &lt;a href=&quot;https://keras.io/backend/&quot; rel=&quot;nofollow noreferrer&quot;&gt;backend&lt;/a&gt; functions. You might want to check if &lt;a href=&quot;https://stackoverflow.com/questions/43547402/how-to-calculate-f1-macro-in-keras&quot;&gt;this&lt;/a&gt; works for you.&lt;/p&gt;&#xA;" OwnerUserId="34269" LastActivityDate="2018-04-26T23:03:50.493" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30910" PostTypeId="2" ParentId="937" CreationDate="2018-04-26T23:05:27.697" Score="3" Body="&lt;p&gt;In fact there is a nice algorithm called &quot;Forward_Select&quot; that uses Statsmodels and allows you to set your own metric (AIC, BIC, Adjusted-R-Squared, or whatever you like) to progressively add a variable to the model. The algorithm can be found in the comments section of this page - scroll down and you'll see it near the bottom of the page.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://planspace.org/20150423-forward_selection_with_statsmodels/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://planspace.org/20150423-forward_selection_with_statsmodels/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would add that the algorithm also has one nice feature: you can apply it to either classification or regression problems! You just have to tell it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try it and see for yourself.&lt;/p&gt;&#xA;" OwnerUserId="51241" LastEditorUserId="51241" LastEditDate="2018-09-21T21:09:26.220" LastActivityDate="2018-09-21T21:09:26.220" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30911" PostTypeId="2" ParentId="30898" CreationDate="2018-04-26T23:15:49.730" Score="1" Body="&lt;p&gt;Reading the relevant paragrpah, we can gain a little more insight. It seems to come down to the idea of &lt;strong&gt;delayed reward&lt;/strong&gt;. By only updating the model (i.e. the &lt;em&gt;agent&lt;/em&gt;) every once in a while, we are inherently inducing the agent to make moves which are benficial over longer periods of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the paragraphs from &lt;a href=&quot;https://medium.com/@awjuliani/super-simple-reinforcement-learning-tutorial-part-2-ded33892c724&quot; rel=&quot;nofollow noreferrer&quot;&gt;the post&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Delayed reward — Keeping the pole in the air as long as possible means moving in ways that will be advantageous for both the present&#xA;  and the future. To accomplish this we will adjust the reward value for&#xA;  each observation-action pair using a function that weighs actions over&#xA;  time.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;To take reward over time into account, the form of Policy Gradient we&#xA;  used in the previous tutorials will need a few adjustments. The first&#xA;  of which is that we now need to update our agent with more than one&#xA;  experience at a time. To accomplish this, we will collect experiences&#xA;  in a buffer, and then occasionally use them to update the agent all at&#xA;  once. These sequences of experience are sometimes referred to as&#xA;  rollouts, or experience traces. We can’t just apply these rollouts by&#xA;  themselves however, we will need to ensure that the rewards are&#xA;  properly adjusted by a discount factor&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Intuitively this allows each action to be a little bit responsible for&#xA;  not only the immediate reward, but all the rewards that followed. We&#xA;  now use this modified reward as an estimation of the advantage in our&#xA;  loss equation. With those changes, we are ready to solve CartPole!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The agent might otherwise want to just do the optimal move to keep the pole upright for the next milllisecond, but with this &lt;strong&gt;buffer&lt;/strong&gt;, we are essentially only allowing the agent to consider larger chunks of time.&#xA;This will take longer to train, but should in theory (well, the assumption behind these rollouts...) lend itself to creating a more thoughtful agent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other contexts, such as a standard CNN used to classify images, this approach would make less sense as there isn't a temporal dimension to the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Saving up the errors and making one big update can have several other explanations in general:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I can save compute, as we could formulate the graph to then only perform backprop after our buffer is full. For buffer size not close to 1, this would potentially halve the training time (making the gross simplication of an assumption that learning occurs at the same rate)&lt;/li&gt;&#xA;&lt;li&gt;It may be equivalent to using batches, in that it could help smooth the learning process. If you have a batch size of 1 sample, the error that is measures and used to update weights may be quite erratic. Using a batch has a nice averaging/smoothin effect. Saving up errors in a buffer, one could argue, may have a similar effect.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I should add, I haven't personally seen this in practice outside of reinforcement learning.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-04-26T23:21:36.240" LastActivityDate="2018-04-26T23:21:36.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30912" PostTypeId="1" AcceptedAnswerId="30914" CreationDate="2018-04-26T23:17:16.687" Score="28" ViewCount="69955" Body="&lt;p&gt;What does &quot;baseline&quot; mean in the context of machine learning and data science?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Someone wrote me:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Hint: An appropriate baseline will give an RMSE of approximately 200.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I don't get this. Does he mean that if my predictive model on the training data has a RMSE below 500, it's good? &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And what could be a &quot;baseline approach&quot;?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="41968" LastEditorDisplayName="user10640" LastEditDate="2019-03-02T12:24:10.477" LastActivityDate="2019-08-07T14:19:32.030" Title="What does &quot;baseline&quot; mean in the context of machine learning?" Tags="|machine-learning|regression|predictive-modeling|terminology|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30913" PostTypeId="1" CreationDate="2018-04-26T23:18:57.213" Score="4" ViewCount="18870" Body="&lt;p&gt;What tools can I use to make a visualization similar to this one? I want to have the mean be bolded and the standard deviation be shaded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://cdn.pbrd.co/images/Hit3Xn6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://cdn.pbrd.co/images/Hit3Xn6.png&quot; alt=&quot;image&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51242" LastEditorUserId="8878" LastEditDate="2018-04-27T09:21:47.557" LastActivityDate="2019-06-16T01:31:00.070" Title="Plots with shaded standard deviation" Tags="|text-mining|visualization|k-means|gradient-descent|plotting|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30914" PostTypeId="2" ParentId="30912" CreationDate="2018-04-27T00:12:53.873" Score="31" Body="&lt;p&gt;A baseline is the result of a very basic model/solution. You generally create a baseline and then try to make more complex solutions in order to get a better result.&#xA;If you achieve a better score than the baseline, it is good.&lt;/p&gt;&#xA;" OwnerUserId="31086" LastActivityDate="2018-04-27T00:12:53.873" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30915" PostTypeId="2" ParentId="30912" CreationDate="2018-04-27T00:13:18.507" Score="23" Body="&lt;p&gt;A baseline is a method that uses heuristics, simple summary statistics, randomness, or machine learning to create predictions for a dataset. You can use these predictions to measure the baseline's performance (e.g., accuracy)-- this metric will then become what you compare any other machine learning algorithm against.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In more detail:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A machine learning algorithm tries to learn a function that models the relationship between the input (feature) data and the target variable (or label). When you test it, you will typically measure performance in one way or another. For example, your algorithm may be 75% accurate. But what does this mean? You can infer this meaning by comparing with a baseline's performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typical baselines include those supported by &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation&quot; rel=&quot;noreferrer&quot;&gt;scikit-learn's &quot;dummy&quot; estimators&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html&quot; rel=&quot;noreferrer&quot;&gt;Classification&lt;/a&gt; baselines:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;“stratified”: generates predictions by respecting the training set’s class distribution.&lt;/li&gt;&#xA;&lt;li&gt;“most_frequent”: always predicts the most frequent label in the training set.&lt;/li&gt;&#xA;&lt;li&gt;“prior”: always predicts the class that maximizes the class prior.&lt;/li&gt;&#xA;&lt;li&gt;“uniform”: generates predictions uniformly at random.&lt;/li&gt;&#xA;&lt;li&gt;“constant”: always predicts a constant label that is provided by the user. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is useful for metrics that evaluate a non-majority class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html&quot; rel=&quot;noreferrer&quot;&gt;Regression&lt;/a&gt; baselines:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;“median”: always predicts the median of the training set&lt;/li&gt;&#xA;&lt;li&gt;“quantile”: always predicts a specified quantile of the training set,provided with the quantile parameter.&lt;/li&gt;&#xA;&lt;li&gt;“constant”: always predicts a constant value that is provided by the user.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, you will want your approach to outperform the baselines you have selected. In the example above, you would want your 75% accuracy to be higher than any baseline you have run on the same data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, if you are dealing with a specific domain of machine learning (such as recommender systems), then you will typically pick baselines that are current state-of-the-art(SoTA) approaches - since you will usually want to demonstrate that your approach does better than these. For example, while you evaluate a new collaborative filtering algorithm, you may want to compare it to matrix factorization -- which itself is a learning algorithm, but is now a popular baseline since it has been so successful in recommender system research.&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-04-27T00:13:18.507" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30916" PostTypeId="2" ParentId="30913" CreationDate="2018-04-27T00:22:42.233" Score="4" Body="&lt;p&gt;The below piece of code will generate the following Image(your's is Subplotting Three of them, so you will get 3 different axe's and per axes you have to use fill-between) (Kindly ignore the Axis Label's..)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.style.use('ggplot') #Change/Remove This If you Want&#xA;&#xA;fig, ax = plt.subplots(figsize=(8, 4))&#xA;ax.plot(trees_grid, train_acc.mean(axis=1), alpha=0.5, color='blue', label='train', linewidth = 4.0)&#xA;ax.plot(trees_grid, test_acc.mean(axis=1), alpha=0.5, color='red', label='cv', linewidth = 1.0)&#xA;ax.fill_between(trees_grid, test_acc.mean(axis=1) - test_acc.std(axis=1), test_acc.mean(axis=1) + test_acc.std(axis=1), color='#888888', alpha=0.4)&#xA;ax.fill_between(trees_grid, test_acc.mean(axis=1) - 2*test_acc.std(axis=1), test_acc.mean(axis=1) + 2*test_acc.std(axis=1), color='#888888', alpha=0.2)&#xA;ax.legend(loc='best')&#xA;ax.set_ylim([0.88,1.02])&#xA;ax.set_ylabel(&quot;Accuracy&quot;)&#xA;ax.set_xlabel(&quot;N_estimators&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7a9MQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7a9MQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-09-25T09:18:00.557" LastActivityDate="2018-09-25T09:18:00.557" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30917" PostTypeId="1" CreationDate="2018-04-27T00:38:13.767" Score="8" ViewCount="7650" Body="&lt;p&gt;I'm trying to cluster words based on pre trained embeddings. I ran a simple experiment where I obtained around 100 words relating to &quot;food taste&quot;, obtained word embeddings from a pre-trained set, and tried to run k-means on the result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do get sensible results on certain clusters, but very strange results on others. E.g: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Cluster 1: [fatty, oily, greasy] -- (good)&#xA;Cluster 2: [crumbly, powdery, grainy, flakey, chalky] -- (good)&#xA;Cluster 3: [flavorful, hearty, unflavored, savory, full-bodied] -(bad)&#xA;Cluster 4: [seasoned, unseasoned] -- (bad)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone suggest as to why seemingly opposite words like (seasoned, unseasoned) and (flavorful, unflavored) would be clustered together? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I tried: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Using fasttext embeddings and Glove embeddings. The latest results are from concatenating fasttext wikipedia and common crawl embeddings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Normalizing the vectors to same length before doing k-means (using eucledian distnaces). I think this is somewhat similar to having cosine distances. Tried not normalizing s well, but normalizing gave better results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Tried some other clustring methods like DBSCAN but k-means seems better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Tried PCA to reduce the dimensionality of the word vectors - didnt change the results much. Tried PCA for the selected embeddings in my vocab that needs to be clustered- not the whole word set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions to improve my results are welcome. If someone came across research articles discussing similar issues please post them as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&#xA;TNC&lt;/p&gt;&#xA;" OwnerUserId="43768" LastActivityDate="2018-04-27T06:20:27.727" Title="K-means clustering of word embedding gives strange results" Tags="|nlp|clustering|word-embeddings|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30918" PostTypeId="1" CreationDate="2018-04-27T02:26:24.220" Score="1" ViewCount="978" Body="&lt;p&gt;I'm working on a very small scale pet project in which inputs are essentially sets of (x, y) pairs, and are to be classified into categories, using deep learning, specifically using Keras (I know this may not be the best for this, but it's more of a proof of concept / I want to try it out).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not sure how to go about representing the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm starting with a simple classification problem (i.e. if (a, b) is a feature of a sample, and a and b are both within 5% of a certain (c, d), then they are a positive example, and not otherwise), but I'm not sure how to represent the data such that the network can learn this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was thinking of doing one-hot encoding, but then the dimensionality of the data may grow immensely (x and y both take on values in a continuous interval), and I'm worried that it would not generalize well to data augmentation (I would augment the data by adding noise to each x, y).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="51248" LastActivityDate="2023-12-28T00:07:55.887" Title="Neural Nets: unordered sets of ordered tuples as features of data" Tags="|neural-network|deep-learning|keras|representation|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30919" PostTypeId="1" AcceptedAnswerId="41565" CreationDate="2018-04-27T04:42:14.163" Score="5" ViewCount="229" Body="&lt;p&gt;I am performing gradient check for my LSTM which has 4 timesteps. The LSTM looks as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   01       01       01       01&#xA;   ^        ^        ^        ^&#xA;  LSTM --&amp;gt; LSTM --&amp;gt; LSTM --&amp;gt; LSTM&#xA;   ^        ^        ^        ^&#xA;   11       11       11       11&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, at every timestep we are feeding in vector {1,1} and expect {0,1} at the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume I perturb the weight inside LSTM, then perform 4 forward props - one for each timestep - how do I now get delta of the cost function that this single perturbation has caused?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I allowed to simply add-up the change in Cost from all 4 timesteps to treat it as derivative estimate?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Also, should I perform it as follows for LSTM:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;perturb a single weight upwards&lt;/li&gt;&#xA;&lt;li&gt;forward prop 4 timesteps&lt;/li&gt;&#xA;&lt;li&gt;perturb the weight downwards&lt;/li&gt;&#xA;&lt;li&gt;forward prop 4 timesteps&lt;/li&gt;&#xA;&lt;li&gt;get 4 deltas&lt;/li&gt;&#xA;&lt;li&gt;sum the 4 deltas to get a total change in Cost&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Set N=0&lt;/li&gt;&#xA;&lt;li&gt;perturb the weight upwards&lt;/li&gt;&#xA;&lt;li&gt;foward prop at a particular timestep N&lt;/li&gt;&#xA;&lt;li&gt;perturb the weight downwards&lt;/li&gt;&#xA;&lt;li&gt;forward prop at a particular timestep N&lt;/li&gt;&#xA;&lt;li&gt;get single delta, store it away&lt;/li&gt;&#xA;&lt;li&gt;increment N&lt;/li&gt;&#xA;&lt;li&gt;until N not equal 4 return to step 2)&lt;/li&gt;&#xA;&lt;li&gt;sum the 4 deltas to get a total change in Cost&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The second approach somehow seems more correct, because LSTM will have a hidden state ..Is this correct intuition or it won't matter?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-04-28T20:01:48.387" LastActivityDate="2019-02-26T07:25:35.807" Title="Gradient Checking LSTM - how to get change in Cost across timesteps?" Tags="|gradient-descent|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30920" PostTypeId="2" ParentId="30302" CreationDate="2018-04-27T05:32:36.573" Score="6" Body="&lt;p&gt;You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Rescaling variables should not change the significance of results at all.&lt;/strong&gt; Indeed, when I reran the regression (with the variables as is, and normalized by subtracting off the mean and dividing by the standard errors), each coefficient estimate (except the constant) had exactly the same t-stat as before scaling, and the F-test of overall significance remained exactly the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, even when all of the variables have been scaled to have a mean of zero and a variance of 1, there is no one size of standard error for each of the regression coefficients, so just looking at the magnitude of each coefficient in the standardized regression is still misleading about significance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As David Masip explained, the apparent size of the coefficients has an inverse relationship with the magnitude of the data points. But even when the coefficients on disp and hp are huge, they are still not significantly different from zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, hp and disp are highly correlated with each other, r=.79, so the standard errors on those coefficients are especially high relative to the coefficient magnitude because they are so collinear. In this regression, they're doing a weird counterbalancing, which is why one has a positive coefficient and one has a negative coefficient; it seems like a case of overfitting and does not seem meaningful.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way to see which variables explain the most variation in mpg is the (adjusted) R-squared. It is literally the percentage of the variation in y that is explained by the variation in the x variables. (Adjusted R-squared includes a slight penalty for each additional x variable in the equation, to counterbalance overfitting.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good way to see what's important--in light of the other variables--is to look at the change in adjusted R-squared when you leave out that variable from the regression. That change is the percentage of variance in the dependent variable that that factor explains, after holding constant the other variables. (Formally, you can &lt;a href=&quot;https://en.wikipedia.org/wiki/F-test#Regression_problems&quot; rel=&quot;nofollow noreferrer&quot;&gt;test whether the left-out variables matter with an F-test&lt;/a&gt;; this is how stepwise regressions for variable selection work.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate this, I ran single linear regressions for each of the variables separately, predicting mpg. The variable wt alone explains 75.3% of the variation in mpg, and no single variable explains more. However, many of other variables are correlated with wt and explain some of that same variation. (I used robust standard errors, which might lead to slight differences in standard error and significance calculations but will not affect coefficients or R-squared.)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+------+-----------+---------+----------+---------+----------+-------+&#xA;|      |   coeff   |   se    | constant |   se    | adj R-sq | R-sq  |&#xA;+------+-----------+---------+----------+---------+----------+-------+&#xA;| cyl  | -0.852*** | [0.110] |        0 | [0.094] |    0.717 | 0.726 |&#xA;| disp | -0.848*** | [0.105] |        0 | [0.095] |    0.709 | 0.718 |&#xA;| hp   | -0.776*** | [0.154] |        0 | [0.113] |    0.589 | 0.602 |&#xA;| drat |  0.681*** | [0.123] |        0 | [0.132] |    0.446 | 0.464 |&#xA;| wt   | -0.868*** | [0.106] |        0 | [0.089] |    0.745 | 0.753 |&#xA;| qsec |  0.419**  | [0.136] |        0 | [0.163] |    0.148 | 0.175 |&#xA;| vs   |  0.664*** | [0.142] |        0 | [0.134] |    0.422 | 0.441 |&#xA;| am   |  0.600*** | [0.158] |        0 | [0.144] |    0.338 | 0.360 |&#xA;| gear |  0.480*   | [0.178] |        0 | [0.158] |    0.205 | 0.231 |&#xA;| carb | -0.551**  | [0.168] |        0 | [0.150] |    0.280 | 0.304 |&#xA;+------+-----------+---------+----------+---------+----------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When all the variables are in there together, the R-squared is 0.869, and the adjusted R-squared is 0.807. So, throwing in 9 more variables to join wt just explains another 11% of the variation (or merely 5% more, if we correct for overfitting). (Many of the variables explained some of the same variation in mpg that wt does.) And in that full model, the only coefficient with a p-value under 20% is wt, at p= 0.089. &lt;/p&gt;&#xA;" OwnerUserId="51239" LastEditorUserId="51239" LastEditDate="2018-04-27T19:35:11.113" LastActivityDate="2018-04-27T19:35:11.113" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30921" PostTypeId="1" AcceptedAnswerId="96937" CreationDate="2018-04-27T05:35:05.490" Score="1" ViewCount="4839" Body="&lt;p&gt;Similar to &lt;a href=&quot;https://datascience.stackexchange.com/questions/18647/is-it-possible-to-customize-the-activation-function-in-scikit-learns-mlpclassif&quot;&gt;this question&lt;/a&gt; about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to change the activation function of the output layer in an &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLPRegressor&lt;/a&gt; neural network in scikit-learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to use it for function approximation.  I.e.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;y = f(x)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where x is a vector of no more than 10 variables and y is a single continuous variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I would like to change the output activation to linear or tanh.  Right now it looks like sigmoid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, I fail to see how you can use scikit-learn for anything other than classification which would be a shame.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, I realise I could use tensorflow or PyTorch but my application is so basic I think scikit learn would be perfect fit (pardon the pun there).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to build a more customized network with &lt;a href=&quot;http://scikit-neuralnetwork.readthedocs.io/en/latest/module_mlp.html#multilayerperceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;MultiLayerPerceptron&lt;/a&gt; or perhaps from individual layers (&lt;a href=&quot;http://scikit-neuralnetwork.readthedocs.io/en/latest/module_mlp.html#multilayerperceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;sknn.mlp&lt;/a&gt;)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;UPDATE:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the documentation for MultiLayerPerceptron it does say:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For output layers, you can use the following layer types: Linear or&#xA;  Softmax.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But then further down it says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;When using the multi-layer perceptron, you should initialize a&#xA;  Regressor or a Classifier directly.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And there is no example of how to instantiate a MultiLayerPerceptron object.&lt;/p&gt;&#xA;" OwnerUserId="50811" LastEditorUserId="50811" LastEditDate="2018-04-27T05:41:18.100" LastActivityDate="2021-06-22T19:14:14.970" Title="Is it possible to customize the activation function in scikit-learn's MLPRegressor?" Tags="|scikit-learn|regression|activation-function|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30923" PostTypeId="2" ParentId="26755" CreationDate="2018-04-27T06:15:40.063" Score="10" Body="&lt;p&gt;I'm not sure if you can change accepted answers, but since the only answer to your question on back propagation is one about forward propagation, I decided to give it a go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Essentially, you treat the weight delta ($\frac{\delta R}{\delta h_j^l}$) the same as you would a weight delta for a linear neuron, but train it once for each time you overlaid your filter (kernel) on the input, all in a single backprop pass. The result is the sum of the deltas for all overlays of your filter. I think in your notation, this would be $\frac{\delta R}{\delta h_j^l} = \sum_{i=1}^nx_i^{l-1}\frac{\delta R}{\delta x_j^{l+1}}$ where $x^{l-1}$ is an input that was multiplied with $h_j^l$ for one of your overlays during forward propagation, and $x^{l+1}$ is the output that resulted from that overlay.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Between the two results of backprop through a convolutional layer (parameter deltas, and input deltas) it sounds like you're more interested in parameter deltas, or more specifically, the weight matrix deltas ($\frac{\delta R}{\delta h_j^l}$). For the sake of completeness, I'll go over both, with the following as our example layer:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a 1D set of inputs $[1, 2, 3]$, and a filter $[0.3, 0.5]$ that is applied with stride 1, no padding, zero bias, and no activation function, then the activation of your filter would have looked like $[1*0.3+2*0.5, 2*0.3+3*0.5] = [1.3, 2.1]$. When you come back through this layer on your backprop pass, let's say the activation deltas you use for your calculations are $[-0.1, 0.2]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Weight deltas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you came through in the forward pass, you cached your inputs $[1, 2, 3]$, let's call that A_prev, since it's likely the activation of your previous layer. For each potential overlay of your filter (in this case, you can only overlay it on the input in two places [&lt;strong&gt;1,2&lt;/strong&gt;,3] and [1,&lt;strong&gt;2,3&lt;/strong&gt;]), take that slice of the input A_slice, multiply each element by the associated output delta dZ, and add it to your weight delta dW for this pass. In this example, you would add $[1*-0.1, 2*-0.1]$ to dW for the first overlay, then add $[2*0.2, 3*0.2]$ for the second overlay. All told, your dW for this convolutional  layer on this backprop pass is $[0.3, 0.4]$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bias deltas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Same as for weight deltas, but just add your output delta without multiplying by the input matrix.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Input deltas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reconstruct the shape of the input for this layer, call it dA_prev, initialize it to zeros, and walk through the configurations in which you overlaid your filter on the input. For each overlay, multiply your weight matrix by the output delta associated with this overlay, and add that to the slice of dA_prev associated with this overlay. That is to say, overlay 1 will add $[0.3 * -0.1, 0.5 * -0.1] = [-0.03, -0.05]$ to dA_prev resulting in $[-0.03, -0.05, 0]$, then overlay 2 will add $[0.3 * 0.2, 0.5 * 0.2] = [0.06, 0.1]$, resulting in $[-0.03, 0.01, 0.1]$ for dA_prev.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a pretty good source if you want to read the same answer in different terms: &lt;a href=&quot;http://neural.vision/blog/deep-learning/backpropagation-with-shared-weights/&quot; rel=&quot;noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51244" LastEditorUserId="40853" LastEditDate="2018-04-27T08:46:30.163" LastActivityDate="2018-04-27T08:46:30.163" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30924" PostTypeId="2" ParentId="30917" CreationDate="2018-04-27T06:20:27.727" Score="17" Body="&lt;p&gt;Word embeddings are trained by &lt;strong&gt;substitutability, not similarity&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you consider a sentence like &quot;This food is unflavored.&quot; Then a good substitute word would be &quot;flavored&quot;, and the sentence will still be &quot;correct&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In many cases, substitutability arises from similarity (crunchy, crispy) but it &lt;em&gt;does&lt;/em&gt; also arise from opposites. You may consider &quot;king&quot; and &quot;queen&quot; to be opposites, too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You probably should use a supervised approach then.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-27T06:20:27.727" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30926" PostTypeId="1" CreationDate="2018-04-27T07:32:16.810" Score="1" ViewCount="48" Body="&lt;p&gt;Would a company like OpenAi, it the &quot;average&quot; person be able to make a bot, using machine learning, that can outperform the best e-sport players in complex games, like PubG, with the technology we have available today? And if so: what kind of models and approach would most likely be used?&lt;/p&gt;&#xA;" OwnerUserId="51255" LastActivityDate="2022-06-27T10:04:55.437" Title="Is it possible to make an machine learning algorithm, that can outperform humans in complex games with current technology?" Tags="|machine-learning|machine-learning-model|state-of-the-art|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30927" PostTypeId="2" ParentId="30860" CreationDate="2018-04-27T07:58:02.483" Score="1" Body="&lt;p&gt;First of all, your approaches are smart and creative but some remarks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The question is not defined well. You talk about clustering using target which is actually paradoxical however I understand your point. The problem is that, not caring about this paradox may hurt your analysis which comes in second point.&lt;/li&gt;&#xA;&lt;li&gt;According to &lt;strong&gt;1&lt;/strong&gt;, you classify your points based on target and try to find dense subgroups there. That will not work as you are including the target in your analysis. This is the confusion made from paradoxical definition I mentioned above. &lt;/li&gt;&#xA;&lt;li&gt;You have your targets, so divide your data according to them and analyze each subset of data separately, having an eye on their interaction.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Second the suggestions:&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Subgroup Discovery&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is the right, solid and difficult way to find these interesting subgroups. To adopt it to your use-case you need to modify the algorithm which might make it even more difficult but is certainly worth, at least, to &lt;a href=&quot;https://www.kde.cs.uni-kassel.de/atzmueller/paper/2005-SDSchlagwortKI_AtzmuellerM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;have a look at&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Creative Way&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Partition your data according to target. You will end up with several disjoint subsets of data. Then start a statistical analysis on the association of variables within and between subsets of data. For instance the distribution of values within a variable should be significantly different among different classes, if that variable really contributes to that target (&lt;a href=&quot;https://en.wikipedia.org/wiki/Analysis_of_variance&quot; rel=&quot;nofollow noreferrer&quot;&gt;ANOVA&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This also helps to remove variables which do not contribute to the target (by doing this &lt;strong&gt;1.&lt;/strong&gt; you reduce the complexity of data and analysis and improve interpretability by removing them, &lt;strong&gt;2.&lt;/strong&gt; you already found which target IS NOT contributing which is a part of your answer)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;PS:&lt;/strong&gt; I just improvised. Please try it and let me know if it works or not so we can think of another solution :)&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-04-27T07:58:02.483" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30928" PostTypeId="1" CreationDate="2018-04-27T08:28:13.610" Score="2" ViewCount="40" Body="&lt;p&gt;I am working on the GPS dataset of the football players with the energy expenditure being my output variable and other 15 variables being input/independent variables. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My objective is to forecast the energy progress of the players in the future matches if they are going to play for the full 90 minutes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Say now I have 50 player's 150 matches data and the problem I am facing is that each player has played for different duration. Few players have played for 90 minutes in few of their matches and few have played only for 20-25 minutes in the match, so it is difficult for me to understand how to solve this problem. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample data: &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OtpWj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OtpWj.png&quot; alt=&quot;Dataset&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The highlighted columns in the dataset are the columns I generated. Predicted is my prediction using Multi Linear Regression for each player, which has an accuracy of 98%. I tried to add energy/time * 90 mins, but the result is not efficient. I have done feature analysis and selection to filter the important 15 variables out of the 90 variables in the actual dataset.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Solution Needed:&#xA;&#xA;Day     Player     Actual_Time player    Actual_Energy_Spent    Energy_if played_for_90mins&#xA;1        A            20                        15000              XXXXX&#xA;&#xA;2        A            35                        16000              XXXXX&#xA;&#xA;3        A            30                        14000              XXXXX&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, If I am able to predict that value for 90 minutes from the past matches using the 15 input variables, I would be able to forecast easily for the future matches as well. I am not sure of how to achieve this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other input variables such as distance are also dependant on the time_played.Can this be done using any statistical model or should I be using any system dynamics in resolving this problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LrJzg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LrJzg.png&quot; alt=&quot;Variable importance&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon doing some research, I came to know about transfer function(Activation function) and there is a function in python SciPy to do it, but I haven't learned anything about it before and don't know if that would solve this problem. Being the only Data Analyst Intern with no seniors to work with or to guide, I am helpless. Any idea or suggestion or reference material for such solving this problem would be highly helpful and much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;thanks. &lt;/p&gt;&#xA;" OwnerUserId="47265" LastEditorUserId="47265" LastEditDate="2018-04-27T08:37:42.097" LastActivityDate="2018-04-27T08:37:42.097" Title="Normalising time(minutes) related data with n other input variables(also dependant on time)" Tags="|python|time-series|normalization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30930" PostTypeId="1" AcceptedAnswerId="30931" CreationDate="2018-04-27T09:19:40.337" Score="5" ViewCount="19498" Body="&lt;p&gt;My task is to perform classify news articles as Interesting [1] or Uninteresting [0].&#xA;My training set has &lt;strong&gt;4053&lt;/strong&gt; articles out of which &lt;strong&gt;179&lt;/strong&gt; are &lt;em&gt;Interesting&lt;/em&gt;. The validation set has &lt;strong&gt;664&lt;/strong&gt; articles out of which &lt;strong&gt;17&lt;/strong&gt; are &lt;em&gt;Interesting&lt;/em&gt;.&#xA;I have preprocessed the articles and converted to vectors using word2vec.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The CNN architecture is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sentence_length, vector_length = 500, 100&#xA;def create_convnet(img_path='../new_out/cnn_model_word2vec.png'):&#xA;    input_shape = Input(shape=(sentence_length, vector_length, 1))&#xA;&#xA;    tower_1 = Conv2D(8, (vector_length, 3), padding='same', activation='relu')(input_shape)&#xA;    tower_1 = MaxPooling2D((1,vector_length-3+1), strides=(1, 1), padding='same')(tower_1)&#xA;    tower_1 = Dropout(0.25)(tower_1)&#xA;&#xA;    tower_2 = Conv2D(8, (vector_length, 4), padding='same', activation='relu')(input_shape)&#xA;    tower_2 = MaxPooling2D((1,vector_length-4+1), strides=(1, 1), padding='same')(tower_2)&#xA;    tower_2 = Dropout(0.25)(tower_2)&#xA;&#xA;    tower_3 = Conv2D(8, (vector_length, 5), padding='same', activation='relu')(input_shape)&#xA;    tower_3 = MaxPooling2D((1, vector_length-5+1), strides=(1, 1), padding='same')(tower_3)&#xA;    tower_3 = Dropout(0.25)(tower_3)&#xA;&#xA;    merged = concatenate([tower_1, tower_2, tower_3], axis=1)&#xA;    merged = Flatten()(merged)&#xA;    dropout1 = Dropout(0.5)(merged)&#xA;    out = Dense(1, activation='sigmoid')(dropout1)&#xA;&#xA;    model = Model(input_shape, out)&#xA;    plot_model(model, to_file=img_path)&#xA;    return model&#xA;&#xA;some_model = create_convnet()&#xA;some_model.compile(loss=keras.losses.binary_crossentropy,&#xA;              optimizer='adam',&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The model predicts &lt;strong&gt;all&lt;/strong&gt; articles in the validation set as &lt;em&gt;Uninteresting&lt;/em&gt; [0]. The accuracy is 97.44% which is same as the ratio of Uninteresting articles in the validation set. I have tried variations of this architecture but still, the issue exists.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For experimentation, I predicted on the training data itself, for that too, it predicts &lt;strong&gt;all&lt;/strong&gt; as &lt;em&gt;Uninteresting&lt;/em&gt; [0].&#xA;Here are the logs for 10 epochs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;some_model.fit_generator(train_gen, train_steps, epochs=num_epoch, verbose=1, callbacks=callbacks_list, validation_data=val_gen, validation_steps=val_steps)&#xA;Epoch 1/10&#xA;254/253 [==============================] - 447s 2s/step - loss: 0.7119 - acc: 0.9555 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00001: val_loss improved from inf to 0.41266, saving model to ../new_out/cnn_model_word2vec&#xA;Epoch 2/10&#xA;254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00002: val_loss did not improve&#xA;Epoch 3/10&#xA;254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00003: val_loss did not improve&#xA;&#xA;Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.&#xA;Epoch 4/10&#xA;254/253 [==============================] - 448s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00004: val_loss did not improve&#xA;&#xA;Epoch 00004: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.&#xA;Epoch 5/10&#xA;254/253 [==============================] - 444s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00005: val_loss did not improve&#xA;&#xA;Epoch 00005: ReduceLROnPlateau reducing learning rate to 1.0000000656873453e-06.&#xA;Epoch 6/10&#xA;254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00006: val_loss did not improve&#xA;&#xA;Epoch 00006: ReduceLROnPlateau reducing learning rate to 1.0000001111620805e-07.&#xA;Epoch 7/10&#xA;254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00007: val_loss did not improve&#xA;&#xA;Epoch 00007: ReduceLROnPlateau reducing learning rate to 1e-07.&#xA;Epoch 8/10&#xA;254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00008: val_loss did not improve&#xA;&#xA;Epoch 00008: ReduceLROnPlateau reducing learning rate to 1e-07.&#xA;Epoch 9/10&#xA;254/253 [==============================] - 444s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00009: val_loss did not improve&#xA;&#xA;Epoch 00009: ReduceLROnPlateau reducing learning rate to 1e-07.&#xA;Epoch 10/10&#xA;254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744&#xA;&#xA;Epoch 00010: val_loss did not improve&#xA;&#xA;Epoch 00010: ReduceLROnPlateau reducing learning rate to 1e-07.&#xA;Out[3]: &amp;lt;keras.callbacks.History at 0x7f19898b90f0&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="37585" LastEditorUserId="50727" LastEditDate="2020-07-13T11:40:17.110" LastActivityDate="2020-07-13T11:40:17.110" Title="Accuracy and loss don't change in CNN. Is it over-fitting?" Tags="|machine-learning|deep-learning|cnn|word2vec|overfitting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30931" PostTypeId="2" ParentId="30930" CreationDate="2018-04-27T09:35:49.330" Score="7" Body="&lt;p&gt;Your dataset is highly imbalanced. Your optimization process is just minimizing the loss function, and cannot do better than a model that predicts uninteresting regardless of the input, due to the fact that your training set is very imbalanced. Moreover, you are not overfitting, since your training accuracy is lower than your validation accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to have a model that learns something less dummy than your model (and you might have to pay the price of having a lower accuracy), I would do the following: when providing a mini-batch to your optimizer, generate a mini-batch that is more balanced, that is, bias the elements you select towards the interesting articles. For instance, if your batch size is 64, ensure that it has 32 interesting elements and 32 uninteresting elements. Using this your network might start learning some features regarding the words in it, and in principle it should help you achieve a not so dummy predictor.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-27T09:35:49.330" CommentCount="7" ContentLicense="CC BY-SA 3.0" />
  <row Id="30934" PostTypeId="2" ParentId="19381" CreationDate="2018-04-27T09:54:52.317" Score="1" Body="&lt;p&gt;To &lt;code&gt;train&lt;/code&gt; a model you need two things, your training data(a matrix of variables and observations) and your labels (the known classification of each observation in the training data). Perhaps the last column in your data is actually the label? If so you will to vertically split your &lt;code&gt;dtrain&lt;/code&gt; so that parameter &lt;code&gt;data=&lt;/code&gt; all columns but the last, and a parameter &lt;code&gt;label=&lt;/code&gt; just the last column. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have the model, you can use it to &lt;code&gt;predict&lt;/code&gt; on data that doesn’t have labels and the output will be the labels.&lt;/p&gt;&#xA;" OwnerUserId="32402" LastActivityDate="2018-04-27T09:54:52.317" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30935" PostTypeId="1" AcceptedAnswerId="30938" CreationDate="2018-04-27T10:44:33.317" Score="0" ViewCount="1969" Body="&lt;p&gt;I'm new to data mining using WEKA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was trying out datasets with a large dataset (2000+ attributes with 90 instances) and left the default parameters as it is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is Multilayer Perceptron running long on a dataset with 2000+ attributes? &#xA;K-Nearest Neighbour does a better job in terms of speed given the same dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does the hiddenLayer in MLP affect the speed and accuracy of the training set? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the most recommended way in running such large dataset, or is there none? &lt;/p&gt;&#xA;" OwnerUserId="51264" LastActivityDate="2018-04-27T11:52:39.670" Title="WEKA Multilayer Perceptron with large dataset" Tags="|weka|" AnswerCount="1" CommentCount="0" ClosedDate="2018-04-28T06:26:28.903" ContentLicense="CC BY-SA 3.0" />
  <row Id="30937" PostTypeId="1" CreationDate="2018-04-27T11:16:24.043" Score="4" ViewCount="174" Body="&lt;p&gt;I have some experience in machine learning, mainly clustering and classifiers.  However, I am somewhat of a newbie when it comes to NLP.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said I am aware of all the various issues and difficulties involved in processing &lt;strong&gt;natural&lt;/strong&gt; language eg part-of-speech, ambiguity, negation detection etc.  I am also aware of certain models to represent text such as bag-of-words and Word vector representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Whilst the specific problem I have is a textual problem, it is not to my mind a natural language problem.  Instead I need to compare 1000s of files of programming code to determine how similar they are to one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike natural language processing where we remove stop-words and we care very much about issues such as ambiguity and the scope of negation etc, with programming code there is no ambiguity, at least not with reserved key words such as &quot;function&quot;, &quot;if&quot;, &quot;else, &quot;while&quot;, &quot;end&quot;, nor is there any ambiguity with core mathematical functions and other operators eg &quot;log&quot;, &quot;sin&quot;, &quot;tanh&quot; etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For my problem, we can assume all files are in the same programming language (let's say R) and let's also assume that each programming file contains at most 250 lines of code.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The initial objective is to produce a correlation matrix that shows how similar the code files are to one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are to my mind 2 ways of approaching this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Quick and simplistic&lt;/strong&gt;.  I propose to remove from each text file:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;All comments as I'm not interested in each developer's individual comments&lt;/li&gt;&#xA;&lt;li&gt;All excessive white space, ie remove all formatting ensuring there is &#xA;only one space between reserved words and blocks of code&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;This is the only pre-processing I would do in a simplistic solution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then propose to use the Doc2Vec model to reduce each file to a single vector and then compare the various vectors to one another by calculating the correlation coefficient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this seem reasonable to people for a very quick and simplistic solution?  If not, please advise what other pre-processing you would do and/or what model you would use to represent the files.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;More advanced&lt;/strong&gt;.  An immediate flaw with the simplistic solution is that it is too naive.  For instance:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;If Code A and Code B have exactly the code (ie same functions that perform exactly the same task and all variable and object names are the same) except that Code A's functions are listed in a slightly different order to those listed in Code B's file, then a line-by-line comparison of the two code files will obviously show differences.  However, if the code in file B was ``reordered'' so that the functions were in exactly the same order as file A, then there wouldn't be any differences.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Code A and Code B could in logical terms be exactly the same (ie same number of functions that perform logically the same operations and output) except that object, variable and function names are different (after all that is a subjective choice by each coder) then again we would have significant differences.  For example, in Code A's file there is a function called MyFunction(a,b) whereas in Code B's file there is a function called CopyMyFunction(x,y).  Now if both functions perform exactly the same operations (eg the sum of the two input values) then logically the code is the same - it's just the variable names that are different.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Finally a more complicated version.  Suppose there is a function in Code A's file called MyAverage(a,b) that performs the operations, returning as output d:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c = a + b&lt;/p&gt;&#xA;&#xA;&lt;p&gt;d = c/2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose in Code B's file there is also a function called MyAverage(a,b) that performs the same calculation but in only one step, ie&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c = (a + b)/2&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Logically the two functions produce exactly the same output, albeit Code A does it in 2 steps whereas Code B does it in 1 step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The above are clearly only some examples of the types of issues that could be encountered.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The objective is not to determine whether there has been plagiarism or whether sections of code have been copied from one code file to another.  Instead, if a serious error (a bug or logical error) is subsequently found in one code file, then I need a means of identifying how likely that same error occurs in other code files, ie has that bug propagated itself by being copied from file to file?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How would people approach the more advanced problem? For instance, to solve the last problem (ie determining whether two sections of code are logically equivalent) I could see two ways of approaching this:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Renaming of variables and substitution.  So in Code A, as variable c is defined as &quot;a + b&quot;, it could be substituted into the expression for variable d with the addition of the brackets.  After the substitution, variable d could be renamed as variable c.  In this case, Code B would then be the same as Code A.&lt;/li&gt;&#xA;&lt;li&gt;Simulation of output.  In more complicated function, the above technique may not always work.  Instead, if the outputs of the two functions are the same when given the same inputs, can we deduce that the logic of the two must be the same within a certain level of confidence?  The problem with this is that simulating millions of inputs would be very time consuming and expensive surely.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other than the above I've no idea how to approach this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any thoughts on the approach and/or resources to use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am happy to be directed to a good reference or resource on the issue.&lt;/p&gt;&#xA;" OwnerUserId="51230" LastActivityDate="2018-04-28T18:05:39.603" Title="Automatic code checking" Tags="|machine-learning|r|nlp|word2vec|similar-documents|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30938" PostTypeId="2" ParentId="30935" CreationDate="2018-04-27T11:46:55.907" Score="0" Body="&lt;p&gt;That is a lot of attributes.  Moreover, WEKA's default MLP size of the hidden layer is &quot;a&quot;, where the following preset sizes are given:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a = (features + classes) / 2&lt;/li&gt;&#xA;&lt;li&gt;i = features&lt;/li&gt;&#xA;&lt;li&gt;o = classes &lt;/li&gt;&#xA;&lt;li&gt;t = features + classes.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;It is expected that the MLP will take longer.  Some things you can try:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use dimensionality reduction such as PCA to reduce the number of features&lt;/li&gt;&#xA;&lt;li&gt;reduce the number of folds if you are using k-fold cross validation&lt;/li&gt;&#xA;&lt;li&gt;reduce hidden layer size (which will probably negatively affect accuracy)&lt;/li&gt;&#xA;&lt;li&gt;use another model such as a SVM, which would be a &lt;em&gt;lot&lt;/em&gt; faster&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I recommend going through this &lt;a href=&quot;https://towardsdatascience.com/a-gentle-introduction-to-neural-networks-series-part-1-2b90b87795bc&quot; rel=&quot;nofollow noreferrer&quot;&gt;high level introduction&lt;/a&gt; to neural networks.  It is important to gain some intuition about the models you are using and the options you have available before tackling your problem.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-27T11:52:39.670" LastActivityDate="2018-04-27T11:52:39.670" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30939" PostTypeId="2" ParentId="30921" CreationDate="2018-04-27T11:51:49.133" Score="3" Body="&lt;p&gt;I see in the code for the MLPRegressor, that the final activation comes from a general initialisation function in the parent class: &lt;code&gt;BaseMultiLayerPerceptron&lt;/code&gt;, and the logic for what you want is shown around &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/neural_network/multilayer_perceptron.py#L271&quot; rel=&quot;nofollow noreferrer&quot;&gt;Line 271&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Output for regression&#xA;if not is_classifier(self):&#xA;    self.out_activation_ = 'identity'&#xA;# Output for multi class&#xA;...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then during a foward pass this &lt;code&gt;self.out_activation_&lt;/code&gt; is called (&lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/neural_network/multilayer_perceptron.py#L112&quot; rel=&quot;nofollow noreferrer&quot;&gt;defined here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# For the last layer&#xA;output_activation = ACTIVATIONS[self.out_activation_]&#xA;activations[i + 1] = output_activation(activations[i + 1])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That ominous looking variable &lt;code&gt;ACTIVATIONS&lt;/code&gt; is simply a dictionary ,with the keywords being the descriptions you can choose as a parameter in your MLP, each mapping an actual function. Here is &lt;a href=&quot;https://github.com/scikit-learn/scikit-learn/blob/a24c8b464d094d2c468a16ea9f8bf8d42d949f84/sklearn/neural_network/_base.py#L98&quot; rel=&quot;nofollow noreferrer&quot;&gt;the dictionary&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,&#xA;               'relu': relu, 'softmax': softmax}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With all of this information, you might be able to come up with a few ways of putting in your custom function. Off the top of my head, I can't see a quick way to simply provide a function. You could for example:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;define your function where all the other activation functions are defined&lt;/li&gt;&#xA;&lt;li&gt;add it to that &lt;code&gt;ACTIVATIONS&lt;/code&gt; dictionary&lt;/li&gt;&#xA;&lt;li&gt;make &lt;code&gt;self.out_activation_&lt;/code&gt; equal to your custom function (or even a new parameter in &lt;code&gt;MLPRegressor&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;cross your fingers it doesn't break something somewhere else&lt;/li&gt;&#xA;&lt;li&gt;run it and solve the inevitable small adaptations that will be necessary in a few places&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm, afraid I have never looked at the source code of that library before, so cannot give more nuanced advice. Perhaps there is a beautifully elegant way to do it that we have both overlooked.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-04-27T11:51:49.133" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30940" PostTypeId="2" ParentId="28446" CreationDate="2018-04-27T11:57:47.867" Score="0" Body="&lt;p&gt;A good thing that you could try on top of Euclidian distance and DTW would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;DBA which is a sidegrade to DTW, here is an &lt;a href=&quot;https://github.com/fpetitjean/DBA&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt; &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Telescopic distance (as suggested by horaceT) &lt;a href=&quot;http://www.grappa.univ-lille3.fr/~mary/cours/stats/telescopic/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here is a link&lt;/a&gt; that one of the authors gave me, this should prove more concrete than the math only.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="51269" LastEditorUserId="51269" LastEditDate="2018-04-27T15:02:04.597" LastActivityDate="2018-04-27T15:02:04.597" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30941" PostTypeId="2" ParentId="30918" CreationDate="2018-04-27T12:11:15.627" Score="0" Body="&lt;p&gt;Your features are already numerical, so you can simply pass your array of numbers into a MLP.  For instance, if you have 5 sets of (x,y) coordinates per sample, your MLP will have 10 inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One-hot encoding is used to convert categorical features into numerical values.  This image from Kaggle does a great job of illustrating how you might use one-hot encoding:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rvWtQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rvWtQ.png&quot; alt=&quot;one-hot encoding&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were to directly convert categorical values to numerical values, that would imply that one color is greater than another, so we need to use a boolean array instead.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As for the variable length feature vector, it is difficult to give an exact answer without knowing more details about your problem, but here are general possible solutions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;use a recurrent neural network&lt;/li&gt;&#xA;&lt;li&gt;use distance-based regression/classification algorithms such as k nearest neighbors&lt;/li&gt;&#xA;&lt;li&gt;transform your data into feature vectors of fixed length (which may not be reasonable depending on your data and what it represents)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;A quick &lt;a href=&quot;https://towardsdatascience.com/a-gentle-introduction-to-neural-networks-series-part-1-2b90b87795bc&quot; rel=&quot;nofollow noreferrer&quot;&gt;high level introduction&lt;/a&gt; to neural networks might be helpful.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-04-28T14:06:51.577" LastActivityDate="2018-04-28T14:06:51.577" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30943" PostTypeId="1" CreationDate="2018-04-27T13:45:55.817" Score="1" ViewCount="335" Body="&lt;p&gt;I don't have a background in maths so I sometimes get confused by basic definitions. Let's for instance consider Bayes Theorem in Bayes Data Analysis:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(\theta|\textbf{y}) \propto P(\theta) P(\textbf{y}|\theta)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should this equation be understood? What does it mean that the RHS is &lt;em&gt;proportional to&lt;/em&gt; LHS?&lt;/p&gt;&#xA;" OwnerUserId="51275" LastEditorUserId="50727" LastEditDate="2018-05-16T14:10:29.053" LastActivityDate="2018-05-16T14:10:29.053" Title="What does the term &quot;proportional to&quot; mean in Bayes Equation?" Tags="|statistics|definitions|mathematics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="30944" PostTypeId="1" AcceptedAnswerId="30946" CreationDate="2018-04-27T13:48:14.700" Score="3" ViewCount="1278" Body="&lt;p&gt;I would like to ask what is the goal of fine-tuning a VGGNet on my dataset.&#xA;What does fine-tuning mean? Does it mean to change the weights or keep the weight values?&lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="50727" LastEditDate="2018-05-19T05:26:36.253" LastActivityDate="2018-05-19T05:26:36.253" Title="The goal of fine tuning" Tags="|machine-learning|neural-network|deep-learning|cnn|computer-vision|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30945" PostTypeId="2" ParentId="30943" CreationDate="2018-04-27T13:54:06.410" Score="3" Body="&lt;p&gt;It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}{P(y)}$. This proportion has to be added in order that $\sum_{\theta}P(\theta | y) = 1$ is satisfied (that is, $P(\theta | y)$ has to be a probability).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-27T13:54:06.410" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30946" PostTypeId="2" ParentId="30944" CreationDate="2018-04-27T13:57:47.680" Score="6" Body="&lt;p&gt;Fine tuning means changing the weights such that the VGGNet can perform the task you want in your dataset. The reason why fine-tuning is not called training (which is what you are doing) is because it implies that you already use a network that has been trained on a dataset. However, the concept is the same as training, but you just happen to it do with a convenient set of initial weights.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-04-27T15:12:04.700" LastActivityDate="2018-04-27T15:12:04.700" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30947" PostTypeId="2" ParentId="30913" CreationDate="2018-04-27T13:58:32.990" Score="2" Body="&lt;p&gt;Use the seaborn plotting library for python, specifically &lt;code&gt;seaborn.tsplot&lt;/code&gt;: &lt;/p&gt;&#xA;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;import seaborn as sns&#xA;&#xA;gammas = sns.load_dataset(&quot;gammas&quot;)&#xA;ax = sns.tsplot(time=&quot;timepoint&quot;, value=&quot;BOLD signal&quot;,&#xA;                 unit=&quot;subject&quot;, condition=&quot;ROI&quot;,&#xA;                 data=gammas)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/M9v8o.jpg&quot; alt=&quot;error plot&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt;  &lt;code&gt;tsplot()&lt;/code&gt; &lt;a href=&quot;https://seaborn.pydata.org/whatsnew.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;is deprecated&lt;/a&gt; as of seaborn version 0.9. It thus might be good to use some other way to plot the data.&lt;/p&gt;&#xA;" OwnerUserId="33202" LastEditorUserId="29575" LastEditDate="2019-06-16T01:31:00.070" LastActivityDate="2019-06-16T01:31:00.070" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30949" PostTypeId="2" ParentId="30937" CreationDate="2018-04-27T15:04:07.530" Score="2" Body="&lt;p&gt;You can start with &lt;a href=&quot;https://theory.stanford.edu/~aiken/moss/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MOSS (Measure Of Software Similarity)&lt;/a&gt;. It can find similar software documents in a set of software documents. It has several nice properties - whitespace insensitivity, noise suppression, and position independence. It is based on the more general idea of document fingerprinting where a hash is constructed of the document. The document hash values are compared. There is a more technical explanation in the &lt;a href=&quot;http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Winnowing paper&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://A%20Comparison%20of%20Similarity%20Techniques%20for%20Detecting%20Source%20Code%20Plagiarism&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a review paper covering additional techniques, such as Levenshtein String Edit Distance, RTED—Robust Algorithm for Tree Edit Distance, and Graph Edit Distance.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-04-28T18:05:39.603" LastActivityDate="2018-04-28T18:05:39.603" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30950" PostTypeId="2" ParentId="28311" CreationDate="2018-04-27T15:41:20.617" Score="1" Body="&lt;p&gt;Assuming you have betting history for multiple players, this is a collaborative filtering problem. I’d treat each bet as a separate transaction that has the characteristics price, class, and outcome. You can then predict the next class. &lt;/p&gt;&#xA;" OwnerUserId="51282" LastActivityDate="2018-04-27T15:41:20.617" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30951" PostTypeId="1" CreationDate="2018-04-27T19:38:21.887" Score="0" ViewCount="250" Body="&lt;p&gt;I am working on a project with a sample size of 30. I have 7 features predicting a continuous variable where I am aiming to optimize the r-value. If I change the random seed of my train_test_split() 100 times, I get correlation coefficients that range between .6 and .9 with an occasional negative r-value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I interpret this? Is this variability normal for such a small data set and a random shuffling of my training data? Or is something else going on? How should I decided which seed to use? &lt;/p&gt;&#xA;" OwnerUserId="38047" LastActivityDate="2018-07-19T09:07:42.597" Title="Interpreting random seed effect on correlation in regression with small sample" Tags="|scikit-learn|regression|linear-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30953" PostTypeId="2" ParentId="30856" CreationDate="2018-04-27T19:42:53.667" Score="1" Body="&lt;p&gt;To explicitly get the percentage of contribution for one variable to another feature, you can take their (pairwise Pearson) correlation and square it. That is the percentage of variation in one explained by variation in the other. (This is the R-squared of a linear regression, if you just regressed one variable on the other.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, if you want to control for other factors (e.g. you want to control for day of the week), you can run a linear regression with just the control variables, then one with those control variables and another one of your predictors. The difference in adjusted R-squared between those models will be how much of the variation in Revenue is explained by the other predictor, beyond what's explained by your controls, and with a slight overfitting penalty for adding another variable.&lt;/p&gt;&#xA;" OwnerUserId="51239" LastActivityDate="2018-04-27T19:42:53.667" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30954" PostTypeId="2" ParentId="24494" CreationDate="2018-04-27T19:46:10.440" Score="1" Body="&lt;p&gt;Based on the answer &lt;a href=&quot;https://datascience.stackexchange.com/a/17708/28175&quot;&gt;here&lt;/a&gt; and the blog post &lt;a href=&quot;https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; there are two variants for using bias in convolutional layers. &lt;strong&gt;&lt;em&gt;Tied biases&lt;/strong&gt; if you use one bias per convolutional filter/kernel  and &lt;strong&gt;untied biases&lt;/strong&gt; if you use one bias per kernel and output location.&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-27T19:46:10.440" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30955" PostTypeId="2" ParentId="19378" CreationDate="2018-04-27T19:51:14.553" Score="0" Body="&lt;p&gt;This might have more of what you are looking for...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://selbydavid.com/2018/01/09/neural-network/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Building a neural network from scratch in R&#xA;Tea &amp;amp; Stats with David Selby&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="51297" LastEditorUserId="29575" LastEditDate="2018-04-27T20:27:13.447" LastActivityDate="2018-04-27T20:27:13.447" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30957" PostTypeId="1" CreationDate="2018-04-27T22:13:53.323" Score="0" ViewCount="797" Body="&lt;p&gt;If I am trying to build a classification model where Values for columns have different ranges: For Example: Column A in training set ranges from 0 - 30 but for testing set it ranges from 0 - 80. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would a classification model still work?&lt;/p&gt;&#xA;" OwnerUserId="51007" LastActivityDate="2018-05-01T21:54:40.327" Title="Different Values in Training and Testing set" Tags="|classification|machine-learning-model|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30958" PostTypeId="1" CreationDate="2018-04-27T22:36:33.177" Score="1" ViewCount="3501" Body="&lt;p&gt;I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. &lt;code&gt;df&lt;/code&gt; has 5000 rows and 30 columns.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.DataFrame({'col1':list('mno'),&#xA;                    'col2':[['a','b'],['d'],['a','b','c','d']],&#xA;                    'col3':[1,2,3],&#xA;                    'col4':[[1,3,4],[5,6,7],[8,3]]})&#xA;df&#xA;&#xA;   col1     col2              col3        col4&#xA;0   m    ['a', 'b']            1        [1, 3, 4]&#xA;1   n    ['d']                 2        [5, 6, 7]&#xA;2   o    ['a', 'b', 'c', 'd']  3        [8, 3]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I am hoping to use &lt;a href=&quot;https://github.com/nicodv/kmodes&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-prototypes&lt;/a&gt; to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. &lt;code&gt;col2&lt;/code&gt; and &lt;code&gt;col4&lt;/code&gt;. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. &lt;code&gt;pd.get_dummies&lt;/code&gt;, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?&lt;/p&gt;&#xA;" OwnerUserId="51301" LastEditorUserId="98307" LastEditDate="2020-08-11T12:06:39.727" LastActivityDate="2020-08-11T12:06:39.727" Title="Segmenting pandas dataframe with lists as elements" Tags="|python|clustering|pandas|dataframe|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30959" PostTypeId="1" AcceptedAnswerId="30961" CreationDate="2018-04-27T22:46:44.850" Score="2" ViewCount="166" Body="&lt;p&gt;Why does &lt;a href=&quot;https://en.wikipedia.org/wiki/Ensemble_averaging_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;ensemble averaging&lt;/a&gt; work for neural networks? This is the main idea behind things like dropout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider an example of a hypersurface defined by the following image (white means lowest Cost). We have two networks: yellow and red, each network has 2 weights, and adjusts them to end up in the white portion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pI4vc.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pI4vc.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, if we average them after they were trained we will end up in the middle of the space, where the error is very high.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="50727" LastEditDate="2018-05-09T15:18:57.977" LastActivityDate="2018-05-09T15:18:57.977" Title="Why does Ensemble Averaging actually improve results?" Tags="|gradient-descent|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30960" PostTypeId="1" CreationDate="2018-04-27T23:56:58.383" Score="3" ViewCount="3550" Body="&lt;p&gt;While I'm reading the &lt;strong&gt;deep learning&lt;/strong&gt; book, I came across something as &quot;top-5  error rate&quot;. I want to know what is it really about and how it is evaluated in   ILSVRC (ImageNet Large Scale Visual Recognition challenge) ?&lt;/p&gt;&#xA;" OwnerUserId="51305" LastEditorUserId="26562" LastEditDate="2018-04-28T01:31:12.693" LastActivityDate="2019-01-05T14:48:52.920" Title="What is the top-5 error rate in ILSVRC challenge?" Tags="|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30961" PostTypeId="2" ParentId="30959" CreationDate="2018-04-28T00:40:36.320" Score="1" Body="&lt;p&gt;I think there is a missunderstanding in your question. In your question you imply that you take the average of the weights of the networks, and you should not do this. Instead, what you average are the predictions of different networks. For this reason, if you average two correct predictions the result will be a correct prediction, and the problem you were thinking about does not exist.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-28T00:40:36.320" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30962" PostTypeId="2" ParentId="30960" CreationDate="2018-04-28T01:43:22.633" Score="5" Body="&lt;p&gt;The Top-5 error rate is the percentage of test examples for which the correct class was not in the top 5 predicted classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, for example, if a test image is a picture of a &lt;code&gt;Persian cat&lt;/code&gt;, and the top 5 predicted classes in order are &lt;code&gt;[Pomeranian (0.4), mongoose (0.25), dingo (0.15), Persian cat (0.1), tabby cat (0.02)]&lt;/code&gt;, then it is still treated as being 'correct' because the actual class is in the top 5 predicted classes for this test image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Remember to keep in mind that ImageNet has 1000 classes, so a low top-5 error rate is still not extremely easy to achieve. &lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-04-28T01:43:22.633" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30963" PostTypeId="1" AcceptedAnswerId="30970" CreationDate="2018-04-28T03:42:01.003" Score="2" ViewCount="164" Body="&lt;p&gt;I am currently looking at a &lt;a href=&quot;https://papers.nips.cc/paper/6573-binarized-neural-networks.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; by Hubara et al  on binarized neural network. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am stuck in understanding algorithm 2 of the paper. The algorithm uses shift-based (bit-shifting) AdaMax, where AdaMax is an extension of the Adam optimizer. In particular, they are using &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$m_{t} = \beta_{1}.m_{t-1} + (1 - \beta_{1}).g_{t} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$v_{t} = \max(\beta_{2}.v_{t-1}, |g_{t}|) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\theta_{t} = \theta_{t−1} − (\alpha\oslash(1−β_{1}^t)).(m_{t} \oslash v_{t}^{-1} ) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $g_{t}$ is the gradient, $\theta_{t-1}$ is the previous parameter, $\alpha$, $\beta_{1}$, $\beta_{2}$ are learning rate, and the betas of Adam optimizer. They stated that $\oslash$ stands for &lt;b&gt;both&lt;/b&gt; left and right  bit shift. I know the left shift and the right shift on its own, but I am not sure how do we have both at the same time? Help will be much appreciated. Thank you. &lt;/p&gt;&#xA;" OwnerUserId="51308" LastEditorUserId="38892" LastEditDate="2018-04-28T15:41:18.227" LastActivityDate="2018-04-28T15:41:18.227" Title="Binarized neural network" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30964" PostTypeId="2" ParentId="30937" CreationDate="2018-04-28T04:01:29.570" Score="1" Body="&lt;p&gt;An alternative approach which is probably not as advanced yet for this is some combination of a deep learning system with and modification of output to provide a 'similarity likelyhood.' &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This paper &lt;a href=&quot;https://arxiv.org/abs/1803.09473&quot; rel=&quot;nofollow noreferrer&quot;&gt;Code2vec&lt;/a&gt;, and &lt;a href=&quot;https://arxiv.org/abs/1711.03543&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt; have a neural network training on code to encode it as a vector in a scalable way in order to predict its properties, and then perform some task depending on the paper. It could be conceivable that if you have 'labeled' code that fits the criteria that you want to train on, that a NN could be something to look into (atleast from a distance). The Code2vec intro seems to fit the problem for more complex examples, in terms of being able to predict outputs and compare code in a way that is similar to language (as there are plenty of NN architectures which do that in terms of summarization and sentiment analysis).&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-28T04:01:29.570" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30965" PostTypeId="2" ParentId="30944" CreationDate="2018-04-28T05:14:49.150" Score="2" Body="&lt;p&gt;Basically fine-tuning or transfer learning is used for situations where you don't have so much data or computation time or maybe computation capacity to train a whole network from scratch. In this way, you usually replace the last layers of a pre-trained network with new ones. The first layers have already found the needed good features and last layers try to classify, you replace them because you have your own labels. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/30871/28175&quot;&gt;here&lt;/a&gt; for considering how it should be applied and &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/4THzO/transfer-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; for more details. You can keep weights, also called freezing, or let the gradient-based algorithms change them, but usually freezing first layers and training last layers is faster.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-04-28T05:14:49.150" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30966" PostTypeId="1" CreationDate="2018-04-28T06:50:34.967" Score="2" ViewCount="215" Body="&lt;p&gt;I am starting with Machine Learning and many people suggested that I should first start with Descriptive and Inferential Statistics on Udacity. I started with it. But I have a doubt that is it absolutely necessary to do the courses to learn ML or I can directly start ML?&lt;/p&gt;&#xA;" OwnerUserId="51311" LastActivityDate="2018-04-28T06:50:34.967" Title="Is Inferential and Descriptive Statistics for Machine Learning?" Tags="|machine-learning|statistics|descriptive-statistics|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-02T03:09:37.503" ContentLicense="CC BY-SA 3.0" />
  <row Id="30967" PostTypeId="2" ParentId="30958" CreationDate="2018-04-28T06:58:24.643" Score="1" Body="&lt;p&gt;Are you concerned that a column might have too many values and that one hot encoding will produce features with less predictive power? In that case you could select a handful of the features for more prevalent class values and keep them as is, while combining the other features for rarer class values into one feature.&lt;/p&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2018-04-28T06:58:24.643" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30968" PostTypeId="2" ParentId="30860" CreationDate="2018-04-28T07:10:52.650" Score="2" Body="&lt;p&gt;Use &lt;strong&gt;constrained clustering&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This allows you to set up &quot;must link&quot; and &quot;cannot link&quot; constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can cluster your data such that no cluster contains both 'churn' and 'non churn' entries bybsettingn&quot;cannot link&quot; constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm just not aware of any good implementations.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-04-28T07:10:52.650" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30969" PostTypeId="1" AcceptedAnswerId="30971" CreationDate="2018-04-28T07:22:09.543" Score="4" ViewCount="182" Body="&lt;p&gt;What are the best ideas or approaches to trade off between bias and variance in Machine Learning models.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-04-29T00:14:01.773" Title="Trade off between Bias and Variance" Tags="|machine-learning|variance|bias|" AnswerCount="2" CommentCount="0" ClosedDate="2018-04-30T07:42:14.217" ContentLicense="CC BY-SA 3.0" />
  <row Id="30970" PostTypeId="2" ParentId="30963" CreationDate="2018-04-28T07:52:10.657" Score="3" Body="&lt;p&gt;I take a look at their &lt;a href=&quot;https://github.com/itayhubara/BinaryNet/blob/master/adaMax_binary_clip_shift.lua&quot; rel=&quot;nofollow noreferrer&quot;&gt;github&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the relevant parts:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;local stepSize = lr/biasCorrection1 --math.sqrt(biasCorrection2)/biasCorrection1&#xA;&#xA;stepSize=math.pow(2,torch.round(math.log(stepSize)/(math.log(2))))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and also&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;state.v:copy(torch.pow(2,torch.round(torch.log(state.v):div(math.log(2)))))&#xA;state.v:add(epsilon)&#xA;tmp:addcdiv(1, state.m, state.v)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems that for $\alpha \oslash (1-\beta_1^t)$, what is being done is $$2^{\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is a left shift if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor &amp;gt; 0$ and a right shift  if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor &amp;lt; 0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, for $m_t \oslash v_t^{-1}$, what is being done is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{m_t}{2^{\lfloor \log_2 v_t\rfloor}}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is a right shift if $\lfloor \log_2 v_t\rfloor&amp;gt; 0$ and a left shift if $\lfloor \log_2 v_t\rfloor &amp;lt; 0$.&lt;/p&gt;&#xA;" OwnerUserId="38892" LastEditorUserId="38892" LastEditDate="2018-04-28T08:02:41.570" LastActivityDate="2018-04-28T08:02:41.570" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30971" PostTypeId="2" ParentId="30969" CreationDate="2018-04-28T08:51:26.450" Score="4" Body="&lt;p&gt;You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If your model is overfitting, you want to regularize the model and/or feed it more training data, decreasing variance and increasing bias.&lt;/p&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2018-04-28T08:51:26.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30972" PostTypeId="2" ParentId="27881" CreationDate="2018-04-28T08:54:34.320" Score="3" Body="&lt;p&gt;Have you tried a &lt;em&gt;correlation test&lt;/em&gt; between the output and the features used to model it?&lt;/p&gt;&#xA;&lt;p&gt;I usually use &lt;em&gt;Pearson's Correlation&lt;/em&gt; to determine the features with low correlation factors. Then I remove them from the training.&lt;/p&gt;&#xA;&lt;p&gt;Here, I have &lt;code&gt;DOC_DS&lt;/code&gt; used to model &lt;code&gt;DOC_US&lt;/code&gt;; I have them both in a data frame &lt;code&gt;df&lt;/code&gt; with my other features. From the trials I did on my models, the features with less correlation affected the model accuracy later on.&lt;/p&gt;&#xA;&lt;h3&gt;In Python Environment:&lt;/h3&gt;&#xA;&lt;pre class=&quot;lang-py prettyprint-override&quot;&gt;&lt;code&gt;print(np.array_str(np.corrcoef(df.DOC_DS, df.DOC_US)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51314" LastEditorUserId="69263" LastEditDate="2023-02-18T18:35:18.987" LastActivityDate="2023-02-18T18:35:18.987" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="30973" PostTypeId="1" CreationDate="2018-04-28T08:58:22.020" Score="6" ViewCount="1848" Body="&lt;p&gt;I have noticed that most of the deep learning developers use TensorFlow. So why choose TensorFlow? What is the advantage of TensorFlow over Theano and CNTK?&lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="29169" LastEditDate="2021-02-13T04:23:26.887" LastActivityDate="2021-02-13T04:23:26.887" Title="Why choose TensorFlow?" Tags="|deep-learning|keras|tensorflow|computer-vision|convolutional-neural-network|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30974" PostTypeId="2" ParentId="30973" CreationDate="2018-04-28T09:12:55.107" Score="0" Body="&lt;p&gt;Tensorflow has much more flexibility to do the things that you want. If you just want to train a standard CNN, then perhaps Theano will do the Job. If you want to do more funny stuff, like creating a new architecture, using a new learning rate schedule or test time augmentation, Theano is too much restrictive. I have also heard that Tensorflow is a bit more performant so it is better to use in production.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-28T09:12:55.107" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30975" PostTypeId="2" ParentId="30944" CreationDate="2018-04-28T09:27:20.433" Score="2" Body="&lt;p&gt;Networks like VGGNet have huge numbers of parameters (see &lt;a href=&quot;https://arxiv.org/pdf/1707.09725.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Appendix D for details&lt;/a&gt;, but it's something like 135 million). Training such a big network takes a lot of data and a lot of time. There is ImageNet, which has 1000 classes and many thousand images. For ImageNet, some people already trained VGGNet and provided the parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way of finetuning is to replace only the last layer (about 4 million parameters) and add a new layer which has the number of classes you need. Then you can freeze all other weights and train on your data. Hence you will have &lt;strong&gt;lower training time&lt;/strong&gt; and you will likely get &lt;strong&gt;better results&lt;/strong&gt; (at least on similar datasets ... for datasets which look very different, &lt;a href=&quot;https://datascience.stackexchange.com/q/30859/8820&quot;&gt;there seems not to be documented conclusive evidence&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-04-28T09:27:20.433" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30977" PostTypeId="2" ParentId="30973" CreationDate="2018-04-28T12:52:41.530" Score="7" Body="&lt;p&gt;I will outline some points about the libraries and point you to some good comparisons that I have read. The GitHub star counts are just another reference point to help compare popularity. While I don't condone simply following the masses, those stars do help you see what a lot of other people think about the frameworks.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Tensorflow&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is very well documented&lt;/li&gt;&#xA;&lt;li&gt;scales up into production, being able to use many GPUs or &lt;a href=&quot;https://en.wikipedia.org/wiki/Tensor_processing_unit&quot; rel=&quot;nofollow noreferrer&quot;&gt;Google TPU&lt;/a&gt;s&lt;/li&gt;&#xA;&lt;li&gt;allows flexible creation of DL architectures, using basic building blocks&lt;/li&gt;&#xA;&lt;li&gt;is backed by Google&lt;/li&gt;&#xA;&lt;li&gt;has a very large following - just look at how many stars it has received over on GitHub!&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9wM46.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9wM46.png&quot; alt=&quot;Tensorflow GitHub stars as of April 2018&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For reference, the awesome &lt;a href=&quot;http://www.numpy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NumPy&lt;/a&gt; only has &lt;a href=&quot;https://github.com/numpy/numpy&quot; rel=&quot;nofollow noreferrer&quot;&gt;7059 stars&lt;/a&gt;!&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;CNTK&lt;/h3&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;is gaining popularity (they started a bit later than Google)&lt;/li&gt;&#xA;&lt;li&gt;has very efficient implementations for specific use cases, such as the usage of CNNs and LSTMs in the text domain&lt;/li&gt;&#xA;&lt;li&gt;is nicely linked to other Micosoft products, like their &lt;a href=&quot;https://azure.microsoft.com/en-gb/overview/what-is-azure/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Azure&lt;/a&gt; based toolkits&lt;/li&gt;&#xA;&lt;li&gt;scales well in production&lt;/li&gt;&#xA;&lt;li&gt;is backed by Microsoft&lt;/li&gt;&#xA;&lt;li&gt;has a good following on Github:&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9b4ZM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9b4ZM.png&quot; alt=&quot;CNTK GitHub stars as of April 2018&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Theano&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;While still open source (and so able to be further developed by the community), the team behind Theano &lt;a href=&quot;https://groups.google.com/forum/#!msg/theano-users/7Poq8BZutbY/rNCIfvAEAwAJ&quot; rel=&quot;nofollow noreferrer&quot;&gt;announced&lt;/a&gt; that they will no longer actively develop it. This means it will now likely fall even further behind other leading frameworks, and new functionality coming from ongoing research are not likely to make it into the library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can see that many people admire Theano, but given that it is basically the oldest DL framework, the star count tells a tale:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KMALj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KMALj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Additional points&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Have a look for some overviews and comparisons of the deep learning frameworks in general to get a better understanding of when you might use one over another. Here are some I found useful: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://deeplearning4j.org/compare-dl4j-tensorflow-pytorch&quot; rel=&quot;nofollow noreferrer&quot;&gt;Comparing top deep learning frameworks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://towardsdatascience.com/battle-of-the-deep-learning-frameworks-part-i-cff0e3841750&quot; rel=&quot;nofollow noreferrer&quot;&gt;Battle of the deep learning frameworks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://agi.io/2018/02/09/survey-machine-learning-frameworks/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Choosing a machine learning framework&lt;/a&gt;   - has a good conclusion&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I would suggest reading about the difference between static and dynamic computational graphs. Tensorflow e.g. builds and compiles a static graph of your model and then pushes data through. PyTorch on the other hand allows you to create your model dynamically, giving more freedom during the development of new architectures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A common workflow would be to develop and do research with PyTorch, then try to write the final production code in Tensorflow for deployment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One last point - you should be aware of the library Keras. This is a wrapper around the base libraries, such as Tensorflow, Theano and CNTK - maybe more in the future). It is highger-level and easier to use that the others, but behind the curtains it is really just using one of the libraries you are asking about. It is just change on flag in a config file to swap between them!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for completeness, the Keras GitHub star banner:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KAgTZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KAgTZ.png&quot; alt=&quot;Keras GitHub star count as of April 2018&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;[All gitHub star counts as of April 2018]&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;I realise some of these points may become outdated and untrue over time. In which case, feel free to leave a comment and I can revise my post.&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-06-11T14:46:51.933" LastActivityDate="2018-06-11T14:46:51.933" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="30978" PostTypeId="1" CreationDate="2018-04-28T14:29:23.317" Score="2" ViewCount="125" Body="&lt;p&gt;I am classifying text using &lt;a href=&quot;https://fasttext.cc&quot; rel=&quot;nofollow noreferrer&quot;&gt;fastText&lt;/a&gt; which is a word2vec library that can also create vectors for character level n-grams and I have successfully trained a binary classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I’d like to see what words or subword n-grams are the most predictive of a class for the two classes (e.g. if classifier sees a word &lt;code&gt;forest&lt;/code&gt; or a subword &lt;code&gt;res&lt;/code&gt; then that might be a strong indication that the document has label &lt;code&gt;Nature&lt;/code&gt;, but if it sees word “and” then that is probably not very informative for this classification task).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I guess the question could be phrased as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Given vectors representing words and subwords and a trained fastText classifier, what would be the best way to get a list of e.g. top 10 most informative words and subwords for deciding which class a sample belongs to?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though I’d be glad if you could make specific suggestions that consider my current setup with fastText, I’m also open to the more general solution suggestions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="51321" LastActivityDate="2019-12-28T12:01:00.093" Title="Identifying most informative (sub)words/vectors that help classify a sample" Tags="|neural-network|nlp|text-mining|feature-selection|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30979" PostTypeId="1" AcceptedAnswerId="30984" CreationDate="2018-04-28T15:37:09.850" Score="3" ViewCount="244" Body="&lt;p&gt;I am analyzing a time-series dataset using (supervised) tensorflow deep learning. The tensorflow code is given a series of inputs, and based on each input, the NN has to predict output value in near future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For training there are lots of pre-labeled input/output of past. Each input is an array of state per time step, and each input is labeled with a result value. So far this kind of task is common at deep learning, and I used multiple GRU layers to solve this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But problem is, I found that the pattern for finding appropriate output actually changes as time goes. For every 10 hours (approximately), the pattern changes so at every start of each pattern period (like first 1-2hours of total 10 hours) the NN is bad at predicting, but after that, prediction rate enhances. Plus, I assume the data also has some noise too.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far my implementation with GRUs do its job, but I want to find better way to build NN if possible. I currently know some basic supervised learning techniques and some advanced like LSTM, and for unsupervised learning I know DQN and policy-gradient.&lt;/p&gt;&#xA;" OwnerUserId="51322" LastActivityDate="2018-04-29T05:59:35.733" Title="which NN should I use for Time-series dataset, whose pattern change as time goes" Tags="|tensorflow|rnn|lstm|unsupervised-learning|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="30980" PostTypeId="2" ParentId="30978" CreationDate="2018-04-28T16:11:44.787" Score="1" Body="&lt;p&gt;Note 1: I believe the technical term for what you are looking for is '(sub)word salience'. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First thought (which is not the best method, but could be worth a quick try if your examples are smallish) - Run the text with all words through fastText, and get the baseline most likely labels with probabilities (predict-prob function). Then remove each word that you are interested in testing, (maybe removing common words like 'a' 'the' etc), and compare the predicted probabilities of each obtained set of predicted_probs. The ones that give the greatest difference between baseline and missing wordX can be interpreted as the most informative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A permutation of this idea is to get the embedding for the interesting words, and then get a distance measure (I assume cosine distance) to the labels you have. Then you have a problem, as they may be distributed evenly across labels in terms of distance, but if there is a group of words that are close to a particular label, that can be interpreted as informative. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am sure there is a better technique to do this, so I am curious as well :). You may want to check these out (but they use additional modeling to get salience)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1709.01186&quot; rel=&quot;nofollow noreferrer&quot;&gt;Learning Neural Word Salience Scores&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://openreview.net/pdf?id=BkSDMA36Z&quot; rel=&quot;nofollow noreferrer&quot;&gt;A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION&lt;/a&gt; - this deals with the exact problems of how to identify the useful bits of an input text &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-28T16:11:44.787" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30981" PostTypeId="1" CreationDate="2018-04-28T17:19:25.887" Score="1" ViewCount="1073" Body="&lt;p&gt;I am trying to do multi-label classification via LSTM in Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a simple example, suppose I have sequences of length 4. Each sequence element is a 3x3 numpy array with values 0, 1, or -1, like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-1  1 -1&#xA; 0 -1  0&#xA; 1  0 -1 &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The -1 values indicate that this position of the numpy array should be ignored (no weight calculation, no associated gradient update, no contribution to the cost function, etc). The positions of the -1 values are the same in each 3x3 numpy array. The 0/1 values are the labels for each position.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I tell Keras to ignore the values -1 in my input? I have tried one-hot encoding each 0/1/-1 value and using a mask that way (since masking does not accept 1D input). I also saw that masking is applied to dimension 1 of the (num_samples, num_timesteps, num_features) input shape, which is not what I want: I want it applied to the features (the 3x3 numpy arrays). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the below code do what I want it to do?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Input, Masking, LSTM, Dense, Flatten&#xA;from keras.models import Model&#xA;import numpy as np&#xA;import tensorflow as tf&#xA;from keras import backend as K&#xA;from keras.utils import to_categorical&#xA;from keras.optimizers import adam&#xA;&#xA;#Creating some sample data&#xA;&#xA;#Matrix has size 3*3, values -1, 0, 1&#xA;X = np.random.rand(3, 3).flatten()&#xA;X[X &amp;lt; 0.2] = 0.&#xA;X[(X &amp;gt;= 0.2) &amp;amp; (X &amp;lt; 0.4)] = 1.&#xA;X[X &amp;gt;= 0.4] = -1&#xA;&#xA;X2 = np.random.rand(4, 3*3)&#xA;for i in range(X2.shape[0]):&#xA;    X2[i,:][(X==-1.)] = -1.&#xA;    X2[i,:][(X !=-1.)] = 0.&#xA;    tobeone = int(len(np.where(X2[i,:] == 0.)[0])*0.5)&#xA;    selected_ones = np.random.choice(np.where(X2[i,:] == 0.)[0], tobeone)&#xA;    X2[i,selected_ones] = 1.&#xA;&#xA;X = np.reshape(X, ((1, 3*3)))&#xA;X_new = np.concatenate((X, X2), axis=0)&#xA;Y = X_new[4,:]&#xA;X = X_new[:4,:]&#xA;&#xA;#Categories are 0, 1, -1 one-hot encoded&#xA;X = to_categorical(X, num_classes=3)&#xA;X = np.reshape(X, (4, 3, 3*3)) #X needs to be shape (num_samples, num_timesteps, num_features), not sure if I have this right&#xA;&#xA;Y = to_categorical(Y, num_classes=3)&#xA;y_true = np.reshape(Y, (1, 3, 3*3)) #predicting a single timestep&#xA;&#xA;#Building the model&#xA;&#xA;mask_val = np.tile([0,0,1], 3*3).reshape((3, 3*3))&#xA;input_tensor = Input(shape=(3, 3*3))&#xA;masking_input = Masking(mask_value=mask_val)(input_tensor)&#xA;lstm = LSTM(1, return_sequences=True)(masking_input)&#xA;output = Dense(3*3, activation='sigmoid')(lstm)&#xA;&#xA;model = Model(input_tensor, output)&#xA;model.compile(loss='categorical_crossentropy', optimizer='adam')&#xA;print(model.summary())&#xA;&#xA;y_pred = model.predict(X)&#xA;&#xA;#See if the model's loss is the same as the unmasked loss (shouldn't be)&#xA;print(model.evaluate(np.expand_dims(X[0,:,:], axis=0), y_true))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="31434" LastActivityDate="2018-04-28T17:19:25.887" Title="Keras: apply masking to non-sequential data" Tags="|keras|time-series|lstm|multilabel-classification|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="30983" PostTypeId="1" AcceptedAnswerId="31010" CreationDate="2018-04-28T18:51:11.863" Score="2" ViewCount="69" Body="&lt;p&gt;For Spark's RDD operations, data must be in shape of RDD or be parallelized using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ParallelizedData = sc.parallelize(data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is that if I store data in HDFS, does it get parallelized automatically or I should use code above for using data in Spark? Does storing data in HDFS makes it in shape of RDD?&lt;/p&gt;&#xA;" OwnerUserId="40473" LastActivityDate="2021-08-29T03:58:58.923" Title="does storing file in hdfs parallelize it for Spark?" Tags="|bigdata|apache-spark|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30984" PostTypeId="2" ParentId="30979" CreationDate="2018-04-28T19:05:43.460" Score="3" Body="&lt;p&gt;Do you know when does the new pattern start? You could reset the hidden state of the RNN each time this happens during training. A more detailed explanation in this paper: &lt;a href=&quot;https://arxiv.org/pdf/1706.04148.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1706.04148.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In that paper, they try to predict the next item in a user's session. They use a GRU to create a user-level representation (which in your case could be the pattern representation) and another GRU to predict the next item based on the current session information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do that, they reset the hidden state of the second GRU every time the session changes and reset the hidden state of the first GRU every time the user changes, and I think you could reset the hidden state every time the pattern changes. It is difficult to explain it all right here, I recommend you to read the paper I mentioned before.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To know where are the pattern change points you could train another network, maybe using as labels points where your GRU's accuracy decreases substantially, and then use this network's predictions to reset the hidden state on these points at training time as I mentioned before.&lt;/p&gt;&#xA;" OwnerUserId="51330" LastEditorUserId="51330" LastEditDate="2018-04-28T22:29:20.107" LastActivityDate="2018-04-28T22:29:20.107" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="30986" PostTypeId="1" AcceptedAnswerId="33740" CreationDate="2018-04-28T19:30:03.340" Score="3" ViewCount="1111" Body="&lt;p&gt;I have a trained sequence-to-sequence LSTM that predicts a person's velocity trace (Δx, Δy) given their position (x, y) as they walk along a path. I trained the LSTM using 200 examples of different people walking along the path. Now, I would like to use multi-step forecasting to predict the future velocity of several new people after they have walked 1/3 of the path. Is this possible? If so, how can I implement this in Python or MATLAB? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Train = 200 sequence observations, 2 input features (x, y), time lengths of observations vary&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Test = 5 partial sequence observations (1/3 of the sequence), I would like to predict the last 2/3 of the sequence for each person.&lt;/p&gt;&#xA;" OwnerUserId="51334" LastEditorUserId="51334" LastEditDate="2018-04-28T19:50:29.037" LastActivityDate="2018-06-28T03:49:39.827" Title="How do I perform multi-step forecasting on LSTM trained with multiple observations of a sequence?" Tags="|rnn|lstm|forecast|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30987" PostTypeId="2" ParentId="30986" CreationDate="2018-04-28T20:24:37.443" Score="0" Body="&lt;p&gt;One technique is optical flow [1], [2], which is popular with people doing modeling of action videos. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is a github implementation of [3] &lt;a href=&quot;https://github.com/viorik/ConvLSTM.&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, which she calls ConvLSTM, and is coded in Lua.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[1] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. Deepflow: Large displacement optical flow with deep matching. In Computer Vision (ICCV), 2013 IEEE International Conference on, pp. 1385–1392, Dec 2013. doi: 10.1109/ICCV.2013.175&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[2] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. CoRR, abs/1504.06852, 2015&lt;/p&gt;&#xA;&#xA;&lt;p&gt;[3] Patraucean, V., Handa, A., &amp;amp; Cipolla, R. (2015). Spatio-temporal video autoencoder with differentiable memory. arXiv preprint arXiv:1511.06309.&lt;/p&gt;&#xA;" OwnerUserId="27250" LastActivityDate="2018-04-28T20:24:37.443" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30988" PostTypeId="2" ParentId="30969" CreationDate="2018-04-29T00:14:01.773" Score="4" Body="&lt;p&gt;In my opinion you should do the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Try to &lt;strong&gt;minimize the bias&lt;/strong&gt; as much as possible. This can usually be achieved by choosing a more complex model. This step is done to ensure that your model has enough capacity to solve the problem. This will, however, cause your model to overfit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;Regularize&lt;/strong&gt; the above model to &lt;strong&gt;reduce its variance&lt;/strong&gt;. This can be done through norm penalties (L1, L2, etc.), early stopping, or other model specific techniques (dropout for neural networks, pruning for decision trees, etc.).&lt;/p&gt;&#xA;" OwnerDisplayName="user50386" LastActivityDate="2018-04-29T00:14:01.773" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30989" PostTypeId="1" CreationDate="2018-04-29T01:10:06.770" Score="7" ViewCount="8647" Body="&lt;p&gt;I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, breaking symmetry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there were other cases where I saw people initializing using zero weights. Unfortunately, I can't remember what those were. I think it might have been non-neural-network cases, like a simple linear or logistic regression model (simple weights only on the inputs, leading directly to an output).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are those cases safe for zero initialization? Alternatively, could we use random initialization in those cases too, just to stay consistent?&lt;/p&gt;&#xA;" OwnerUserId="44394" LastEditorUserId="28175" LastEditDate="2018-04-29T06:00:55.930" LastActivityDate="2018-04-29T07:40:17.347" Title="What are the cases where it is fine to initialize all weights to zero" Tags="|machine-learning|neural-network|deep-learning|cost-function|weight-initialization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30990" PostTypeId="1" CreationDate="2018-04-29T02:33:00.680" Score="2" ViewCount="945" Body="&lt;p&gt;I am classifying aerial imagery that is tiled into 256x256 tiles using Keras and TensorFlow. The model splits the training data (i.e. the 256x256 image tiles making up the study area) into 70% training data and 30% validation data. A sequential model is used followed by an image data generator. Lastly, a fit generator is used to fit the model to the data. The model is then saved to h5 format to be used to predict classes with other imagery in different study areas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I run the model using the 70%/30% training/validation split, the predictions on the validation images work great with increasingly higher accuracies and steadily decreasing loss per epoch. Additionally, when I visualize the predictions (i.e. probability arrays) by joining the probability arrays to vector polygons representing the tile boundaries, the classified results look very good. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is when I use the saved &lt;code&gt;h5&lt;/code&gt; model to make predictions on new imagery--the results are nonsensical and appear random for each tile. It is as if the probability arrays are being shuffled randomly such that when I join the results to the vector image boundary tiles, the results look totally random. How can I resolve this issue?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is relevant portions of the code used to train the model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows, img_cols, img_channel))&#xA;&#xA;add_model = Sequential()&#xA;add_model.add(Flatten(input_shape=base_model.output_shape[1:]))&#xA;add_model.add(Dense(256, activation='relu'))&#xA;add_model.add(Dense(n_classes, activation='sigmoid')) # n classes&#xA;&#xA;model = Model(inputs=base_model.input, outputs=add_model(base_model.output))&#xA;model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),&#xA;              metrics=['accuracy'])&#xA;&#xA;######################&#xA;&#xA;batch_size = 32&#xA;epochs = 50&#xA;&#xA;print('Running the image data generator...')&#xA;train_datagen = ImageDataGenerator(&#xA;        rotation_range=30, &#xA;        width_shift_range=0.1,&#xA;        height_shift_range=0.1, &#xA;        horizontal_flip=True)&#xA;train_datagen.fit(x_train)&#xA;&#xA;print('Fitting the model...')&#xA;history = model.fit_generator(&#xA;    train_datagen.flow(x_train, y_train, batch_size=batch_size),&#xA;    steps_per_epoch=x_train.shape[0] // batch_size,&#xA;    epochs=epochs,&#xA;    #validation_data=(x_valid, y_valid),&#xA;    #callbacks=[ModelCheckpoint(model_checkpoint, monitor='val_acc', save_best_only=True)]&#xA;)&#xA;&#xA;######################&#xA;&#xA;## Predict&#xA;#print('Predicting...')&#xA;#p_valid = model.predict(x_valid, batch_size=128)&#xA;&#xA;## Write predictions to csv&#xA;#print('Saving predictions to CSV...')&#xA;#df = pd.DataFrame(p_valid)&#xA;#df['image'] = split + 1 + df.index &#xA;#df.to_csv(out_csv, index=False, header=False)&#xA;&#xA;&quot;&quot;&quot; &#xA;Save model, including these details:&#xA;-the architecture of the model, allowing to re-create the model&#xA;-the weights of the model&#xA;-the training configuration (loss, optimizer)&#xA;-the state of the optimizer, allowing to resume training exactly where you left off.&#xA;&quot;&quot;&quot;&#xA;print(&quot;Saving model&quot;)&#xA;model.save(&quot;/vgg16-model-50epochs.h5&quot;)&#xA;&#xA;print('Processing complete.')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;And the following script uses the saved model from above to make predictions on test imagery from a different study area. Note, there is no 70/30 training/validation split in the final training run above--I simply use 100% of the tiles to train the model, which I then save and reuse in the following script:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import glob, os, time&#xA;import cv2&#xA;import numpy as np&#xA;import pandas as pd&#xA;&#xA;from keras.models import load_model&#xA;#from keras.models import model_from_json&#xA;&#xA;# Path to the input tiles which will be used to predict classes&#xA;inws = '/image-directory-for-another-study-area'&#xA;tiles = glob.glob(os.path.join(inws, '*.tif'))&#xA;&#xA;# h5 file from trained model&#xA;in_h5 = &quot;/vgg16-model-50epochs.h5&quot;&#xA;&#xA;# Output model predictions in csv format&#xA;out_csv = '/new-predictions.csv'&#xA;&#xA;# Read images and convert to numpy array&#xA;x_test = np.array([cv2.imread(tile) for tile in tiles], np.float16) / 255.&#xA;&#xA;print('Loading existing model...')&#xA;loaded_model = load_model(in_h5)&#xA;&#xA;print(&quot;Predicting on image tiles...&quot;)&#xA;predictions = loaded_model.predict(x_test, batch_size=128)&#xA;&#xA;# Save to csv&#xA;df = pd.DataFrame(predictions)&#xA;df['image'] = df.index + 1&#xA;df.to_csv(out_csv, index=False, header=False)&#xA;print(&quot;Predictions saved to disk: {0}&quot;.format(out_csv))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="17539" LastEditorUserId="17539" LastEditDate="2018-04-29T03:38:06.187" LastActivityDate="2018-04-29T03:38:06.187" Title="Test data predictions yield random results when making predictions from a saved model" Tags="|machine-learning|deep-learning|keras|tensorflow|prediction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30991" PostTypeId="1" CreationDate="2018-04-29T02:54:41.350" Score="2" ViewCount="97" Body="&lt;p&gt;I am working to create a Risk score on data where I have variables -&#xA;Invested_amount, Profit Amount, Age of Account in days, Total Trading Transactions, Profit per Transaction &amp;amp; Investment per transaction. Basically, I want to derive a method to calculate the Risk score, where a person's profit is more (most of the time he is winning). Here the score should be higher so that I can classify them as High Risk (Mostly wins and create a big profit), medium Risk, and Low-Risk customers (Always makes Loss).&lt;/p&gt;&#xA;&lt;p&gt;Basically, my problem is to assign a Risk score to each customer, the higher the score more the customer is Risky. Once this score is derived then using this score we will segment the customers into three classes.&lt;/p&gt;&#xA;&lt;p&gt;Below are the variables currently I am using…&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;|Variable|               Definition|&#xA;|username|               UserId to identify the records|&#xA;|Total_Freq_transaction| Total Number of trades done|&#xA;|Total_Freq_win|         Total number of trade won|&#xA;|Account_age_days|       Account activated date - Last trade date |&#xA;|Invest_Amount|          Total Amount invested in trade|&#xA;|total_profit|           Profit made on trade (pay out amount - Investment Amount)|&#xA;|Trade_per_day|          Total Number of trades done / Account age|&#xA;|Win_prob%|              Wining probability from total number of trades|&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="51344" LastEditorUserId="97640" LastEditDate="2021-05-29T05:29:12.173" LastActivityDate="2024-03-27T08:08:46.030" Title="Riskscore creation on Numerical Data" Tags="|machine-learning|neural-network|r|data-mining|data-science-model|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="30992" PostTypeId="1" CreationDate="2018-04-29T03:40:34.107" Score="3" ViewCount="41" Body="&lt;p&gt;Lets say a paper is published which describes a data science algorithm, and the paper is made available on arxiv (no patent or anything else mentioned in paper). The paper is by a university researcher, and the algorithm is not something well known.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A. Is it considered ethical to use my own implementation of that algorithm for:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;work/own profit&lt;/li&gt;&#xA;&lt;li&gt;own learning (I assume yes)&lt;/li&gt;&#xA;&lt;li&gt;Competition (with monetary gain). &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;B. On doing so, is it correct that it is the right thing to do to cite the paper (lets say I am required to hand over competition code). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now lets say the author published some code to go along with the paper under an MIT/other open source license on github, and I mess around with it to fit my needs. What if the paper was by a company (like facebook or google, but does not cite any patent or limitation on use of the code/method) Do the same rules apply or are there other considerations? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I guess I have not found a good blog covering this and any surrounding issues I may not be aware of as self-taught professional human running into this for the first time.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-29T03:40:34.107" Title="Ethics of using Existing Algorithms in Competitions" Tags="|machine-learning|algorithms|methodology|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30993" PostTypeId="2" ParentId="28608" CreationDate="2018-04-29T05:24:08.720" Score="8" Body="&lt;p&gt;You simply need to do:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern)&#xA;df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2018-04-29T05:24:08.720" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="30994" PostTypeId="2" ParentId="30989" CreationDate="2018-04-29T05:28:06.053" Score="9" Body="&lt;p&gt;Whenever you have a convex cost function you are allowed to initialize your weights to zeros. The cost function of logistic regression and linear regression have convex cost function if you use MSE for, also &lt;a href=&quot;https://stats.stackexchange.com/a/90556/179078&quot;&gt;RSS&lt;/a&gt;, linear regression and cross-entropy for logistic regression. The main idea is that for convex cost function you'll have just a single optimal point and it does not matter where you start, the starting point just changes the number of epochs to reach to that optimal point whilst for neural networks the cost function does not have just one optimal point. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/q/18802/28175&quot;&gt;here&lt;/a&gt;. About random initialization, you have to consider that you are not allowed to choose random weights which are too small or too big although the former was a more significant problem. If you choose random small weights you may have vanishing gradient problem which may lead to a network that does not learn. Consequently, you have to use standard initialization methods like &lt;code&gt;He&lt;/code&gt; or &lt;code&gt;Glorot&lt;/code&gt;, take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/10930/28175&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://scholar.google.com/scholar?cluster=17889055433985220047&amp;amp;hl=en&amp;amp;as_sdt=0,22&quot; rel=&quot;noreferrer&quot;&gt;Understanding the difficulty of training deep feedforward neural networks&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, take a look at the following question.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/a/18145/28175&quot;&gt;How should the bias be initialized and regularized&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://datascience.stackexchange.com/q/13061/28175&quot;&gt;When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-04-29T05:35:31.187" LastActivityDate="2018-04-29T05:35:31.187" CommentCount="6" ContentLicense="CC BY-SA 3.0" />
  <row Id="30995" PostTypeId="2" ParentId="30979" CreationDate="2018-04-29T05:59:35.733" Score="1" Body="&lt;p&gt;Going off the discussion in the @Andres Espinosa's answer, here is an option that might work for this without any extra gating voodoo. The &lt;a href=&quot;https://arxiv.org/pdf/1801.10308&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nested LSTMs&lt;/a&gt; paper, &lt;a href=&quot;https://github.com/hannw/nlstm&quot; rel=&quot;nofollow noreferrer&quot;&gt;Implementation here&lt;/a&gt;. The nested LSTM idea is to build in the idea of a temporal abstraction hierarchy - which in your case could possible capture the current pattern at the first level, and the need to switch between patterns at a deeper level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition there is the untouched idea about why your LSTM is not working in the first place. Before doing extra work, maybe it is worth trying &lt;a href=&quot;https://arxiv.org/pdf/1506.02078&quot; rel=&quot;nofollow noreferrer&quot;&gt;visualizing&lt;/a&gt; the LSTM &lt;a href=&quot;https://github.com/HendrikStrobelt/LSTMVis&quot; rel=&quot;nofollow noreferrer&quot;&gt;activations&lt;/a&gt;. In theory, if the LSTM has learned different phases in your input, there would be a different distribution of activations depending on the 'phase' or more significant changes in the hidden state between phases vs within the phase, since it would pay attention to different things depending on the phase, it could be a matter of just more training time. If it is consistently not picking up something that can be visually diagnosed, then it would suggest trying one of the other ideas. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-04-29T05:59:35.733" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30996" PostTypeId="2" ParentId="30989" CreationDate="2018-04-29T07:40:17.347" Score="2" Body="&lt;p&gt;Zeroing weights disables them. Yes, there are various &lt;em&gt;applications&lt;/em&gt; of zero tensors (such as convex cost functions as you mention). Let's take the case of Neural Nets (NNs) and see if the math gives us more intuition:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;tensor\div0 = undefined\\&#xA;tensor * 0 = 0\\&#xA;tensor \cdot 0 = 0\\&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example Graph #1&lt;/strong&gt;: How would one disable a single synapse connected to the output layer? &lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&#xA;&lt;img src=&quot;https://i.stack.imgur.com/kXYtU.png&quot; width=&quot;200&quot;/&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Math Example&lt;/strong&gt;: Let $X$ be an input tensor of shape (1,2). Let $W$ be a weight tensor of shape (2,1). The &lt;a href=&quot;https://www.mathsisfun.com/algebra/matrix-multiplying.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;dot product&lt;/a&gt; is represented here by the $\cdot$ symbol.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If &lt;em&gt;all&lt;/em&gt; elements in tensor $W$ are zero:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align*} &#xA;X = \begin{matrix}[1&amp;amp;1]\\ \end{matrix}\ \ \ &#xA;W = \begin{matrix}[0]\\ [0]\end{matrix}\&#xA;\end{align*}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;X \cdot W = [0]\\&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If all elements in tensor $W$ are randomly initialized (between -1 and 1):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align*} &#xA;X = \begin{matrix}[1&amp;amp;1]\\ \end{matrix}\ \ \ &#xA;W = \begin{matrix}[0.24660266]\\ [0.05121049]\end{matrix}\&#xA;\end{align*}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;X \cdot W = [0.29781315]\\&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If one element in tensor $W$ is randomly set to zero:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\begin{align*} &#xA;X = \begin{matrix}[1&amp;amp;1]\\ \end{matrix}\ \ \ &#xA;W = \begin{matrix}[0]\\ [0.05121049]\end{matrix}\&#xA;\end{align*}&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;X \cdot W = [0.05121049]\\&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Aha, some intuition!&lt;/em&gt; Set an element of your weight vector $W$ to zero to disable it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Example Graph #2&lt;/strong&gt;: As complexity scales, so too does our potential loss of control over architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/MpEwB.png&quot; width=&quot;800&quot; /&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you need to tweak the details, it is important to have tools to change nodes and edges on a per unit basis. Zero weighting gives you that ability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This idea generalizes to CNNs, GANs, RNNs, etc. Look at the particular algorithm and go layer by layer. What are the designers trying to accomplish?&lt;/p&gt;&#xA;" OwnerUserId="37125" LastActivityDate="2018-04-29T07:40:17.347" CommentCount="8" ContentLicense="CC BY-SA 3.0" />
  <row Id="30997" PostTypeId="1" CreationDate="2018-04-29T07:42:48.047" Score="2" ViewCount="222" Body="&lt;p&gt;The gist is me wanting to separate system faults from sensor faults given some dataset from a wireless sensor network using a machine learning algorithm. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, if I have some temperature sensors in a given area and their corresponding readings from every sort of time interval, I would like to know whether an abnormal value is due to an actual fault, or due to a faulty sensor. Of course, it would be a given that the training set would have such entries tagged with either sensor fault or system fault.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have thought of just using something like linear regression but I would like it to work even if the system could not be modeled like that. Decision tree seemed to me like a more appropriate algorithm for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, there is also some consideration for the time it takes for training and classification as I wish to see if it can be used for systems which respond really quickly to such anomalies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sorry if it's a bit wordy but I wasn't sure how much information I should put since this is my first time posting (I'm not even sure if this is the right stack exchange to post this). Anyway, thanks in advance for the answers!&lt;/p&gt;&#xA;" OwnerUserId="51349" LastActivityDate="2024-02-05T07:05:00.643" Title="Is it feasible to use decision tree algorithms for sensor fault detection?" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="30998" PostTypeId="1" CreationDate="2018-04-29T11:50:18.610" Score="2" ViewCount="189" Body="&lt;p&gt;I am building a neural network to predict if a video is pornographic or not by analysing the bytes of upload and download at every 0.1 seconds for a total of 25.6 seconds. So, I have 512 input variables, 256 of download and 256 of upload and I realised that for the same second, the upload and download variables are highly correlated. Does this mean that I am just adding complexity by distinguishing between upload and download variables?&lt;/p&gt;&#xA;" OwnerUserId="49678" LastActivityDate="2022-02-25T23:00:25.077" Title="High Correlation between inputs of neural network" Tags="|machine-learning|neural-network|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="30999" PostTypeId="1" CreationDate="2018-04-29T11:56:28.657" Score="-2" ViewCount="197" Body="&lt;p&gt;I have a set of features including text field(3-10 sentences) and about 10 scalar fields. I need to predict another scalar field (between 0 and 1). I have this field in my training/validation data.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's my first experience with a text feature. What is the correct way of creating a model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I treat text field as embedding, flatten them to 1 dimension matrix of X size and combine with other features? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks. &lt;/p&gt;&#xA;" OwnerUserId="51315" LastEditorUserId="51315" LastEditDate="2018-04-29T17:24:06.440" LastActivityDate="2018-04-29T17:24:06.440" Title="Text + scalar features in one model" Tags="|nlp|word-embeddings|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31002" PostTypeId="1" AcceptedAnswerId="31004" CreationDate="2018-04-29T13:25:55.620" Score="0" ViewCount="347" Body="&lt;p&gt;I have a dataset with a column named &lt;em&gt;Package Size&lt;/em&gt; and it has data in two units inches and meters mentioned in &lt;strong&gt;words&lt;/strong&gt;. How do I preprocess the data such that I remove the unnecessary word and also convert them into a single unit?&#xA;&lt;br/&gt;&#xA;Example:&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Package Size &lt;br/&gt;13 inches&lt;br/&gt;4 meters&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;br/&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Package Size(in meters)&lt;br/&gt; 0.3302 &lt;br/&gt;4&lt;/p&gt;&#xA;" OwnerUserId="45713" LastActivityDate="2018-04-29T13:59:29.440" Title="Removing irrelevant content in columns" Tags="|dataset|preprocessing|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31003" PostTypeId="2" ParentId="30999" CreationDate="2018-04-29T13:40:09.273" Score="1" Body="&lt;p&gt;Your approach seems fair enough. Create a low dimensional vector of text features or if your corpus of text is small (like in comments) then make a &lt;code&gt;bag of words&lt;/code&gt; representation or &lt;code&gt;tf-idf&lt;/code&gt; based word scoring (after cleaning) and use them as normal features by flattening them with other features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing is if you go with word embeddings and when you want to un-blackbox the model you had created based on text, it can be hard as each individual dimension in &lt;code&gt;word2vec&lt;/code&gt; does not hold any meaning. So start with bad of words and move to embeddings and see the metrics.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-04-29T13:40:09.273" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31004" PostTypeId="2" ParentId="31002" CreationDate="2018-04-29T13:59:29.440" Score="0" Body="&lt;p&gt;Since you had not provided the tag on which language you are using. I am providing a sample solution in python with pandas for loading the data and transforming.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt; import pandas as pd&#xA;&amp;gt;&amp;gt; sample = pd.DataFrame({'a': ['1 inch', '2 meters', '1 metre', '2.5 inches', '2.8 meters']})&#xA;            a&#xA;0      1 inch&#xA;1    2 meters&#xA;2     1 metre&#xA;3  2.5 inches&#xA;4  2.8 meters&#xA;&amp;gt;&amp;gt; def get_number(a):&#xA;       for _ in a.split(' '):&#xA;           try:&#xA;               return float(_.strip())&#xA;           except ValueError:&#xA;               pass&#xA;&amp;gt;&amp;gt; m['b'] = m['a'].apply(lambda a: get_number(a) * 0.0254 if 'inch' in a else get_number(a))&#xA;&amp;gt;&amp;gt; m&#xA;              a       b&#xA;  0      1 inch  0.0254&#xA;  1    2 meters  2.0000&#xA;  2     1 metre  1.0000&#xA;  3  2.5 inches  0.0635&#xA;  4  2.8 meters  2.8000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this gives you a general idea of how to do it.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-04-29T13:59:29.440" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31005" PostTypeId="1" CreationDate="2018-04-29T14:22:48.580" Score="3" ViewCount="2123" Body="&lt;p&gt;This question is on an implementation aspect of scikit-learn's &lt;code&gt;DecisionTreeClassifier()&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;How do I get the feature names ranked in descending order, from the &lt;code&gt;feature_importances_&lt;/code&gt; returned by the scikit-learn &lt;code&gt;DecisionTreeClassifier()&lt;/code&gt;?&lt;/p&gt;&#xA;&lt;p&gt;The problem is that the input features to the classifier are not the original ones - they are numerically encoded ones from pandas DataFrame get_dummies.&lt;/p&gt;&#xA;&lt;p&gt;For example, I take the mushroom dataset from the UCI repository.&#xA;Features in the dataset include - &lt;code&gt;cap_shape&lt;/code&gt;, &lt;code&gt;cap_surface&lt;/code&gt;, &lt;code&gt;cap_color&lt;/code&gt;, &lt;code&gt;odor&lt;/code&gt;, etc.&lt;/p&gt;&#xA;&lt;p&gt;pandas dataframe getdummies encodes these into multiple features based on values of the original features.&#xA;Say &lt;code&gt;cap_shape&lt;/code&gt; has values b,c,f,k...after encoding new columns are &lt;code&gt;cap_shape_b&lt;/code&gt;, &lt;code&gt;cap_shape_c&lt;/code&gt;, &lt;code&gt;cap_shape_f&lt;/code&gt;. Similar transformations happen for other features.&lt;/p&gt;&#xA;&lt;p&gt;After training, the classifier tells me that the top two features are:&#xA;&lt;code&gt;cap_shape_b&lt;/code&gt;, &lt;code&gt;cap_shape_c&lt;/code&gt;, &lt;code&gt;cap_shape_f&lt;/code&gt;, &lt;code&gt;odor_a,odor_c&lt;/code&gt;, &lt;code&gt;odor_f,odor_l&lt;/code&gt;.&#xA;From this result thrown by the classifier, I would like my function to return the original features, that is, &lt;code&gt;cap_shape&lt;/code&gt; and &lt;code&gt;odor&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="51223" LastEditorUserId="29169" LastEditDate="2021-08-28T04:46:05.103" LastActivityDate="2021-08-29T03:59:07.093" Title="How to obtain original feature names after using one-hot encoding" Tags="|feature-selection|decision-trees|one-hot-encoding|dummy-variables|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31006" PostTypeId="2" ParentId="30973" CreationDate="2018-04-29T14:24:49.143" Score="4" Body="&lt;p&gt;Here is some, probably exaggerated criticism of TensorFlow:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;a href=&quot;http://nicodjimenez.github.io/2017/10/08/tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow sucks&lt;/a&gt; by Nico Jimenez, 8 Oct 2017&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The article was discussed on &lt;a href=&quot;https://news.ycombinator.com/item?id=15429824&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hacker News&lt;/a&gt;. I highlight some quotes that I find interesting to consider. First, about its technical content.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Despite its shortcomings, I share the same vision as this article. Here are my reasons:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Tensorflow has a way too large API surface area: parsing command lines arguments handling, unit test runners, logging, help formatting strings... most of those are not as good as available counterparts in python.&lt;/li&gt;&#xA;  &lt;li&gt;The C++ and Go versions are radically different from the Python version. Limited code reuse, different APIs, not maintained or documented with the same attention.&lt;/li&gt;&#xA;  &lt;li&gt;The technical debt in the source code is huge. For instance, There are 3 redundant implementations in the source code of a safe division (_safe_div), with slightly different interfaces (sometimes with default params, sometimes not). It's technical debt.&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;  &#xA;  &lt;p&gt;In every way, it reminds me of Angular.io project. A failed promise to be true multi-language, failing to use the expressiveness of python, with a super large API that tries to do things we didn't ask it to do and a lack of a general sounding architecture. (by batmansmk)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Some &lt;em&gt;speculation&lt;/em&gt; about the philosophy and politics behind TensorFlow:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I think the author raises a good point about Google envy. TensorFlow is not the most intuitive or flexible library out there, and it is very over-engineered if you're not doing large-scale distributed training. The main reason why everyone talks it up so much is because Google heavily marketed it from the outset, and everyone automatically assumes Google == Virtuoso Software Design because they couldn't make it through the interview. Really it's just modern enterprise software which has five different ways to implement batch norm that they push on the community so they don't have to train new hires on how to use it. (by zo7)&lt;/p&gt;&#xA;  &#xA;  &lt;blockquote&gt;&#xA;    &lt;p&gt;Or maybe it is built by a company that is doing large-scale distributed training, and they open sourced it not to cater to every need, but to help others trying to do the same thing they are. Companies are under no obligation to make sure their open source is well suited for others use cases. (by dsl)&lt;/p&gt;&#xA;    &#xA;    &lt;blockquote&gt;&#xA;      &lt;p&gt;That was kinda my point, it's not the be-all deep learning library because they made it for their own use case, but its towering popularity (as in 10x the number of stars of other popular libraries) is not genuine.&lt;/p&gt;&#xA;      &#xA;      &lt;p&gt;Also I highly doubt that the main reason Google open sourced it was to be charitable. (by zo7)&lt;/p&gt;&#xA;    &lt;/blockquote&gt;&#xA;  &lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="47856" LastActivityDate="2018-04-29T14:24:49.143" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31007" PostTypeId="1" CreationDate="2018-04-29T16:23:09.433" Score="3" ViewCount="23449" Body="&lt;p&gt;My data looks like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;date, cardio_time, muscles, muscle_time, stretch_time&#xA;2018-01-01, 0, &quot;biceps / lats&quot;, 40, 5&#xA;2018-01-02, 30, &quot;&quot;, 0, 10&#xA;2018-01-03, 0, &quot;lats / calf&quot;, 41, 6&#xA;2018-01-03, 30, &quot;hamstring&quot;, 4, 5&#xA;2018-01-04, 0, &quot;biceps / lats&quot;, 42, 8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like to merge those rows with the same date, and save the info from both rows. My data would look like this after transformation, note the 3rd of January has been changed:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;2018-01-01, 0, &quot;biceps / lats&quot;, 40, 5&#xA;2018-01-02, 30, &quot;&quot;, 0, 10&#xA;2018-01-03, 30, &quot;lats / calf / hamstring&quot;, 45, 11&#xA;2018-01-04, 0, &quot;biceps / lats&quot;, 42, 8&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I think I could use a for loop that checks if the date on row i is the same as on row i-1, and if it is not, check the next row, but if the rows do have the same date, merge the rows together by doing something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# set default value to 1 exercise per row &#xA;df['nr_excercises'] = 1 &#xA;# for loop &#xA;for i in range(1, T):&#xA;    if df.index[i] == df.index[i-1]:&#xA;      # set nr of nr_excercises to 2&#xA;      df.iloc[i, nr_excercises] = 2&#xA;      # create temp variables that hold info from both rows &#xA;      cardiotimetot = df[i, cardio_time] +  df[i-1, cardio_time]&#xA;      stretchtimetot = df[i, stretch_time] +  df[i-1, stretch_time]&#xA;      etc...&#xA;      # save temp variables to i&#xA;      df.iloc[i, cardio_time] = cardiotimetot&#xA;      # drop row i-1&#xA;      df = df.drop[df.index[i-1]] # I think this is correct &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question: Is this a good approach? Is there a better way to do it?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe, the code would be faster if I first use .groupby(df.index).size() to find out which days have multiple entries, and then only apply the for loop to this subset of df.&lt;/p&gt;&#xA;" OwnerUserId="45191" LastEditorDisplayName="user86768" LastEditDate="2019-12-11T13:05:28.607" LastActivityDate="2019-12-12T02:32:15.470" Title="Merging information of rows with the same date" Tags="|python|pandas|" AnswerCount="2" CommentCount="1" ClosedDate="2019-12-12T03:23:19.313" ContentLicense="CC BY-SA 4.0" />
  <row Id="31008" PostTypeId="2" ParentId="31007" CreationDate="2018-04-29T17:19:25.510" Score="1" Body="&lt;p&gt;As @Emre has pointed out in comments, you need a pandas custom aggregator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So since you need a &lt;code&gt;string&lt;/code&gt; custom  &lt;code&gt;join&lt;/code&gt; by &lt;code&gt;/&lt;/code&gt;. Create a custom aggregator as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foo = lambda a: &quot;/&quot;.join(a) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(or if you need spaces around the join)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;foo = lambda a: &quot; / &quot;.join(a) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then make a pandas &lt;code&gt;groupby&lt;/code&gt; as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data_.groupby(by='date').agg({'muscle_time': 'sum',&#xA;                              'stretch_time': 'sum',&#xA;                              'cardio_time': 'sum',&#xA;                              'muscles': foo}).reset_index()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should get you the aggregated dataframe.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastEditorUserId="-1" LastEditDate="2019-05-20T21:25:19.733" LastActivityDate="2019-05-20T21:25:19.733" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31009" PostTypeId="2" ParentId="31005" CreationDate="2018-04-29T17:35:52.060" Score="0" Body="&lt;p&gt;As shown in these &lt;a href=&quot;http://scikit-learn.org/stable/modules/tree.html#tips-on-practical-use&quot; rel=&quot;nofollow noreferrer&quot;&gt;docs&lt;/a&gt; at the section &amp;quot;Classification&amp;quot;, you can export your tree using graphviz (it states that you have to install the graphviz package, too). And this way you're able to visualize the tree built by the algorithm.&#xA;About the problem of the input features being transformed from the original ones, it's a problem the algorithm can't help you with but you should be able to manage that by yourself if you've made the transformations yourself.&lt;/p&gt;&#xA;&lt;p&gt;Any further doubt, comment.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="85045" LastEditDate="2021-08-29T03:59:07.093" LastActivityDate="2021-08-29T03:59:07.093" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31010" PostTypeId="2" ParentId="30983" CreationDate="2018-04-29T17:39:02.107" Score="2" Body="&lt;p&gt;As you can see for the following examples presented &lt;a href=&quot;https://spark.apache.org/examples.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; at the documentation, you'd be able to read directly from HDFS without much trouble. And spark will parallelize the data for you properly.&lt;/p&gt;&#xA;&lt;p&gt;We only use parallelize when you're coming data structures that were built in Scala themselves (like a val assignment or something like that).&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="85045" LastEditDate="2021-08-29T03:58:58.923" LastActivityDate="2021-08-29T03:58:58.923" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31011" PostTypeId="2" ParentId="29725" CreationDate="2018-04-29T17:51:46.557" Score="1" Body="&lt;p&gt;Take a look at association rule learning (&lt;a href=&quot;https://en.wikipedia.org/wiki/Association_rule_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Association_rule_learning&lt;/a&gt;). A really common algorithm is the Apriori agorithm. You could use the package apyori, it works great: &lt;a href=&quot;https://pypi.python.org/pypi/apyori/1.1.1&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pypi.python.org/pypi/apyori/1.1.1&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generate association rules over all items bought by a user, not in a single transaction/purchase.&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-29T17:51:46.557" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31013" PostTypeId="1" AcceptedAnswerId="31033" CreationDate="2018-04-29T18:54:46.300" Score="-1" ViewCount="1239" Body="&lt;p&gt;I have a column in my dataset by name 'production_output' whose values are integers in the range of 2.300000e+01 to 1.110055e+08.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to arbitrarily split the values in this column into different buckets based on say, percentile ranges like say [0, 25, 50, 75, 100] and get count of the length of each of theses buckets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I do this using python data-science packages?&lt;/p&gt;&#xA;" OwnerUserId="51369" LastActivityDate="2018-04-30T12:36:01.273" Title="How do I get a count of values based on custom bucket-ranges I create for a select column in dataframe?" Tags="|pandas|numpy|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="31014" PostTypeId="2" ParentId="27377" CreationDate="2018-04-29T19:58:10.833" Score="1" Body="&lt;p&gt;If the reference to 7 chunks of information for humans is a reference to the &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two&quot; rel=&quot;nofollow noreferrer&quot;&gt;'Magical Number Seven' Paper&lt;/a&gt;, which is the human working memory, then:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the high level operation of DNC at a timestep. An input comes in, and is processed by an LSTM to generate a 'Interface vector' which interacts with memory with Write, Erase and Read operations. The operations are then executed, and a final layer produces an output based on the retrieved information and the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is the analogy to human 'working memory'?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1). If you consider the working memory is what is retrieved on read, then there is a hyperparameter for number of memory reads. This applies generally to Write and Erase as well as those are all hyperparameters which determine the size of the Interface Vector. In the paper they used a write size of 1 in all cases for purposes of controlling the experiments. Note that the 'size' of what is read/written in a single operation is a separate parameter, which just depends on how the problem is encoded (see the methods section of the paper).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2). If you consider working memory as external memory, then this is another hyperparameter. I think it is a conceptual mistake to view the external memory as short term memory, as the addition of this and all the gating was specifically developed to store information long term and then recall when needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3). There is a part of the DNC's operation which queries a link matrix (storing what was written to long term memory and when) during the read phase. This can be seen as being related to human working memory, as numerous behavioral economics studies showed that what we recall and how we process it is effected by what we have just seen. This is the only part in the paper where the computation is O(N^2) w.r.t to the size of the memory, (though they approximate with an O(NlogN) method). This portion of retrieval/bias of what is retrieved to most recently accessed information can then be said to have a limit, since for large memories it will not scale with rest of network, but this is still not constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4). Probably the best candidate for the working memory analogy IMO is the hidden state of the controller LSTM, since this part of the algorithm is responsible for storing what is needed over time for retrieval, and on its own is used for storing a state over time. The reason that this is short term memory in this analogy is just that the external memory is an explicit external memory, and this is the only other place where a state is maintained. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So in short no, there is no Constant, as just about every conceivable analogy to 'working memory' and the entire architecture is controlled by parameters. It is worth checking out the &lt;a href=&quot;https://github.com/deepmind/dnc&quot; rel=&quot;nofollow noreferrer&quot;&gt;code&lt;/a&gt; and &lt;a href=&quot;https://pdfs.semanticscholar.org/7635/78fa9003f6c0f735bc3250fc2116f6100463.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; for more details on exactly how the parameters are set for different experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let me know if this makes sense.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastEditorUserId="47488" LastEditDate="2018-04-29T20:27:03.647" LastActivityDate="2018-04-29T20:27:03.647" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31015" PostTypeId="2" ParentId="25024" CreationDate="2018-04-29T20:57:14.823" Score="1" Body="&lt;p&gt;The reason is exactly as mentioned in the other answer with a great suggestion to use a smaller learning rate to avoid this problem around small gradients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a couple of approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;You can clip the gradients with an upper/lower bound but this does not guarantee convergence and may result in training freeze by getting trapped in some local minima and never get out of it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Train with a higher batch size, more epochs and with a decayed learning rate. Now I do not have any practical proof that increasing a batch size results in better gradients but from what I had observed by facing problems similar to yours, doing so has almost always helped. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am sure there are other methods (like &lt;a href=&quot;https://github.com/bckenstler/CLR&quot; rel=&quot;nofollow noreferrer&quot;&gt;cyclical learning rate&lt;/a&gt; etc) that try to find an optimal learning rate based on statistics.&lt;/p&gt;&#xA;" OwnerUserId="51375" LastActivityDate="2018-04-29T20:57:14.823" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31016" PostTypeId="2" ParentId="30919" CreationDate="2018-04-29T21:13:31.630" Score="2" Body="&lt;p&gt;Interesting question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Like standard backpropagation, [backpropagation through time] consists of a repeated application of the chain rule. The subtlety is that, for recurrent networks, the loss function depends on the activation of the hidden layer not only through its influence on the output layer, but also through its influence on the hidden layer at the next timestep.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It looks like both approaches would have similar results but at a different granularity (adding noise at different levels) this is because the backprop is not really disturbed by this addition as it is still chained from last time step to the first. Hence it boils down to why you actually want to add noise in the first place, as mentioned in &lt;a href=&quot;https://arxiv.org/pdf/1604.08275.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; paper , sections IIIC and IV.&lt;/p&gt;&#xA;" OwnerUserId="51375" LastActivityDate="2018-04-29T21:13:31.630" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31018" PostTypeId="1" CreationDate="2018-04-30T05:00:50.103" Score="1" ViewCount="308" Body="&lt;p&gt;I am quite new to data science and I have been given the task to find relationship between two entities for a given data.This is the scatter plot graph I made&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/yQtM0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/yQtM0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a time series data.Here x axis is the mean and y axis is the mse (mean square error).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I can infer from this graph is the mean and mse don't really go along with each other.In other words when mean and mse doesn't share any strong relationship?Is my understanding of the graph correct?Is there anything else that can be found? &lt;/p&gt;&#xA;" OwnerUserId="51385" LastActivityDate="2018-04-30T07:14:53.740" Title="What to infer from a given scatter plot?" Tags="|bigdata|visualization|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31021" PostTypeId="2" ParentId="30973" CreationDate="2018-04-30T07:06:10.450" Score="3" Body="&lt;p&gt;Reasons to chose Tensorflow (IMHO):&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Large community of users: for every problem that you run into there are 100s of solutions and sample codes.&lt;/li&gt;&#xA;&lt;li&gt;Keras: it makes it easy to create some advanced models on top of Tensorflow. &lt;/li&gt;&#xA;&lt;li&gt;Support: everyone knows that Tesnorflow is backed by Google and they regularly update that. That is true for some other frameworks too, except Theano; Mila (the lab that created Theano) stopped this project.&lt;/li&gt;&#xA;&lt;li&gt;Tensorboard: it allows you to easily visualize the metrics of your model and is straight forward to work with.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Given all that, I really like PyTorch too. It is easy and natural to work with, it is backed by some big names, and most importantly, if you have worked with Matlab or numpy, you feel more comfortable using it, compared to Tensorflow for example.&lt;/p&gt;&#xA;" OwnerUserId="31464" LastActivityDate="2018-04-30T07:06:10.450" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31022" PostTypeId="2" ParentId="31018" CreationDate="2018-04-30T07:14:53.740" Score="3" Body="&lt;p&gt;It is not easy to say based on this graph. Maybe if you could apply a transformation, it was easier to see a trend. I would have at least plotted them on log-log, log-linear, etc. to get better idea.&lt;/p&gt;&#xA;" OwnerUserId="31464" LastActivityDate="2018-04-30T07:14:53.740" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31023" PostTypeId="1" CreationDate="2018-04-30T07:48:08.660" Score="1" ViewCount="42" Body="&lt;p&gt;I am new to Machine Learning, and I am trying to solve &lt;a href=&quot;https://github.com/openai/gym/wiki/MountainCar-v0&quot; rel=&quot;nofollow noreferrer&quot;&gt;MountainCar-v0&lt;/a&gt; using Q-learning. I can solve the problem now, but I am still confused.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the &lt;a href=&quot;https://github.com/openai/gym/wiki/MountainCar-v0&quot; rel=&quot;nofollow noreferrer&quot;&gt;MountainCar-v0's Wiki&lt;/a&gt;, the reward remains -1 for every step, even if the car has reached the destination. How does the invariant reward help the agent learn? If every step gives the same reward, how can the agent tell if it is a good move or a bad move?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="51395" LastActivityDate="2018-06-29T11:16:39.910" Title="Why the invariant reward helps training?" Tags="|machine-learning|openai-gym|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31025" PostTypeId="1" CreationDate="2018-04-30T08:16:32.680" Score="1" ViewCount="65" Body="&lt;p&gt;I need to use both NVIDIA Digits and tensorflow &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/research/object_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Object Detection API&lt;/a&gt; for different deep learning purposes. I am a bit lost with the compatibility issues, especially for what concerns CUDA. So which version should I choose for DIGITS, Tensorflow-gpu, CUDA and Object Detection API in order to make everything work?&lt;/p&gt;&#xA;" OwnerUserId="45420" LastActivityDate="2018-05-30T12:59:51.073" Title="How to setup my Ubuntu PC for these Deep Learning frameworks?" Tags="|tensorflow|gpu|object-detection|nvidia|" AnswerCount="1" CommentCount="2" ClosedDate="2018-06-02T00:37:17.140" ContentLicense="CC BY-SA 3.0" />
  <row Id="31026" PostTypeId="1" CreationDate="2018-04-30T09:12:00.100" Score="2" ViewCount="82" Body="&lt;p&gt;I'm trying to solve &lt;a href=&quot;https://www.kaggle.com/c/quora-question-pairs&quot; rel=&quot;nofollow noreferrer&quot;&gt;Quora Question Pairs&lt;/a&gt; with model stacking.&lt;/p&gt;&#xA;&lt;p&gt;My first layers are:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;CNN trained to predict the same target as whole model should&lt;/li&gt;&#xA;&lt;li&gt;&amp;quot;Magic features&amp;quot; like question frequency in whole dataset&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;And the second is gradient boosting.&lt;/p&gt;&#xA;&lt;p&gt;My first attempt was to train CNN on whole(with 10% to dev) train dataset and than train gradient boosting (GB) on the same split.&lt;/p&gt;&#xA;&lt;p&gt;As expected it was terrible because GB learned that CNN almost always right.&lt;/p&gt;&#xA;&lt;p&gt;So I split train data into 10 parts and trained 10 different CNN with 1 part as dev.&#xA;Than I trained GB with CNN predictions on dev parts(so it's almost unseen data). And used average of 10 CNNs for same feature on test dataset.&lt;/p&gt;&#xA;&lt;p&gt;The result was worse than gradient boosting applied to &amp;quot;magic features&amp;quot; alone.&lt;/p&gt;&#xA;&lt;p&gt;Can you help me to identify my mistakes?&lt;/p&gt;&#xA;" OwnerUserId="50583" LastEditorUserId="1330" LastEditDate="2020-12-08T17:39:47.720" LastActivityDate="2020-12-08T17:39:47.720" Title="Averaging CNN perform worse than boosting" Tags="|xgboost|cnn|ensemble-modeling|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31027" PostTypeId="1" AcceptedAnswerId="31036" CreationDate="2018-04-30T10:19:41.153" Score="5" ViewCount="19240" Body="&lt;p&gt;I have a insurance dataset as given below. For which I need to build a model to calculate the charges.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   age     sex     bmi  children smoker     region      charges&#xA;0   19  female  27.900         0    yes  southwest  16884.92400&#xA;1   18    male  33.770         1     no  southeast   1725.55230&#xA;2   28    male  33.000         3     no  southeast   4449.46200&#xA;3   33    male  22.705         0     no  northwest  21984.47061&#xA;4   32    male  28.880         0     no  northwest   3866.85520&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I am not sure whether the column &quot;region&quot; can be dropped. Is there any test can be performed to consider only significant variables?&lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2021-08-29T13:50:57.857" Title="Feature Selection in Linear Regression" Tags="|feature-selection|linear-regression|feature-engineering|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31028" PostTypeId="2" ParentId="31023" CreationDate="2018-04-30T10:31:56.713" Score="1" Body="&lt;p&gt;Well, some inefficient agents will need more steps to reach the goal. Others will have a more target-oriented, efficient way to reach the goal. The efficient agents will need less steps and have a better/bigger score.&lt;/p&gt;&#xA;" OwnerUserId="50471" LastActivityDate="2018-04-30T10:31:56.713" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31029" PostTypeId="2" ParentId="31027" CreationDate="2018-04-30T10:40:31.117" Score="1" Body="&lt;p&gt;The pacakge &lt;code&gt;leaps&lt;/code&gt; in R has functions, like &lt;code&gt;regsubsets&lt;/code&gt;, that do this. Essentially, what this functions do is they compare different models with criteria that you can choose and take the model that does the best according to the criterium chosen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please note that keeping the model with the highest $R^2$ is not a very good criterium, as models with more variables will always have higher $R^2$. The criteria that &lt;code&gt;regsubsets&lt;/code&gt; can use are &lt;a href=&quot;https://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2&quot; rel=&quot;nofollow noreferrer&quot;&gt;adjusted R^2&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow noreferrer&quot;&gt;Bayesian Information Criterion&lt;/a&gt;, among others.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-04-30T10:40:31.117" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="31030" PostTypeId="2" ParentId="31025" CreationDate="2018-04-30T12:15:57.917" Score="0" Body="&lt;p&gt;An easy way to run different versions of multiple frameworks alongside each other is to use &lt;a href=&quot;https://www.docker.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Docker&lt;/a&gt;. With Docker, you can create a new &quot;Container&quot;, kind of a light-weight virtual machine, for each project. Docker handles all hardware abstraction and separates the different Frameworks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: To use the GPU within a Docker container, you need to install &lt;a href=&quot;https://github.com/NVIDIA/nvidia-docker&quot; rel=&quot;nofollow noreferrer&quot;&gt;Nvidia-Docker&lt;/a&gt;, as a &quot;normal&quot; Docker installation can not share the GPU with a container. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your example, you could have a Docker container for Digits, and a separate one for TensorFlow - then the two frameworks will be completely isolated from each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This approach comes with multiple advantages:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can run the same container on a different machine without installing e.g. TensorFlow - all you need is (Nvidia-)Docker.&lt;/li&gt;&#xA;&lt;li&gt;You can easily use different (TensorFlow) versions for different projects, as each project gets it's own container which is separated from the system.&lt;/li&gt;&#xA;&lt;li&gt;Containers can use different CUDA versions, e.g. CUDA 8 for Digits and CUDA 9 for TensorFlow.&lt;/li&gt;&#xA;&lt;li&gt;There are pre-built containers for almost all existing deep learning frameworks. Nvidia even provides some GPU-optimized containers with their free &lt;a href=&quot;https://www.nvidia.com/en-us/gpu-cloud/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NGC (Nvidia GPU Cloud)&lt;/a&gt; service.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2018-04-30T12:15:57.917" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31031" PostTypeId="1" CreationDate="2018-04-30T12:22:09.793" Score="-1" ViewCount="1156" Body="&lt;p&gt;Suppose I have two columns namely Goods, and Quality which are to be one encoded.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Goods  &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; Quality  &lt;br/&gt;Processed Food&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;A+ grade&lt;br/&gt; Canned Food &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;B grade &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;br/&gt;&#xA;Does encoding two columns of categorical variables advisable? If so, is placing these columns at the beginning of table advisable or what is the suitable method in case of a prediction model for machine learning.&lt;/p&gt;&#xA;" OwnerUserId="45713" LastEditorUserId="45713" LastEditDate="2018-04-30T12:37:46.203" LastActivityDate="2018-04-30T12:46:25.163" Title="Encoding categorical data 2 different columns" Tags="|python|categorical-data|encoding|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="31032" PostTypeId="2" ParentId="30754" CreationDate="2018-04-30T12:25:06.823" Score="0" Body="&lt;p&gt;Right now instead of predicting a confidence interval I'm using a different approach, as described in section 3.1.2 of the following papers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.diva-portal.org/smash/get/diva2:1149130/FULLTEXT01.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.diva-portal.org/smash/get/diva2:1149130/FULLTEXT01.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50832" LastActivityDate="2018-04-30T12:25:06.823" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31033" PostTypeId="2" ParentId="31013" CreationDate="2018-04-30T12:36:01.273" Score="1" Body="&lt;p&gt;&lt;code&gt;numpy.histogram&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.histogram.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.histogram.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Use &lt;code&gt;numpy.percentile&lt;/code&gt; to get the bin edges you desire. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.percentile.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.percentile.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-04-30T12:36:01.273" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31034" PostTypeId="2" ParentId="31031" CreationDate="2018-04-30T12:46:25.163" Score="1" Body="&lt;p&gt;If you are planning to use machine-learning algorithms from &lt;code&gt;scikit-learn&lt;/code&gt; library, then it is not only recommended, but mandatory to convert your data into dummy variables (aka one-hot encoding).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It can be done quite easily using &lt;code&gt;pandas&lt;/code&gt; library. Just use the function &lt;code&gt;get_dummies&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;mydf = pd.DataFrame({'temperature':[30,10,27,24,-6],&#xA;                     'feeling':['Hot', 'Cold', 'Hot', 'Hot', 'Cold'],&#xA;                     'weather':['Sunny', 'Rainy', 'Rainy', 'Sunny', 'Snowy']})&#xA;dummy_df = pd.get_dummies(mydf[['feeling', 'weather']])&#xA;pd.concat([mydf[['temperature']],dummy_df], axis = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Whether the columns are at the beginning or the end does not change anything.&lt;/p&gt;&#xA;" OwnerUserId="42829" LastActivityDate="2018-04-30T12:46:25.163" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31035" PostTypeId="2" ParentId="31027" CreationDate="2018-04-30T13:12:58.003" Score="4" Body="&lt;p&gt;Why don't you consider &lt;strong&gt;Gradient Boosting Decision Trees (GBDT) for Regression&lt;/strong&gt; which you will find many Python implementation for (XGboost, LightGBM and CatBoost). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The good things about GBDTs (more relevant to your problem) are: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;They have an intrinsic way to calculate feature importance (due to the way trees splits work .e.g Gini score and so on). &lt;/li&gt;&#xA;&lt;li&gt;They can deal with categorical variables that you have (sex, smoke, region) &lt;/li&gt;&#xA;&lt;li&gt;Also account for any possible correlations among your variables. Simple linear models fail to capture any correlations which could lead to overfitting.&lt;/li&gt;&#xA;&lt;li&gt;There are many ways to regularize GBDTs, which may come very handy! &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With GBDTs you only have to be careful with continuous variable, in your case the bmi variable, not to be artificially overruling your trees (trees have hard time dealing very continuous data). You can easily overcome this challenge by rounding up/down or binning your continuous variable or other methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have strong reasons to stick to linear regressions, maybe you could use LASSO which is a regularized linear regression that harshly penalizes (=0) the less important variables. People actually use LASSO for feature selection as well.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-04-30T13:12:58.003" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31036" PostTypeId="2" ParentId="31027" CreationDate="2018-04-30T13:52:38.750" Score="6" Body="&lt;p&gt;First, transform your columns, then apply linear regression, but do you want to know about the influence of your features on your selected dependent variable?&lt;/p&gt;&#xA;&lt;p&gt;Read this &lt;a href=&quot;http://www.ritchieng.com/machine-learning-evaluate-linear-regression-model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;, as it provides great insight at how interpret the coefficients given by the algorithm can be interpreted and further discussed or tweaked.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastEditorUserId="29575" LastEditDate="2021-08-29T13:50:57.857" LastActivityDate="2021-08-29T13:50:57.857" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31039" PostTypeId="2" ParentId="29164" CreationDate="2018-04-30T14:14:31.800" Score="-1" Body="&lt;p&gt;You are right it is probably due to new features in the unseen test data. This is data science 101, you should have same features in the test/train datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that it is due to new levels of a categorical variable not present in the train data, but present in the unseen test data and this comes up as an error in the &lt;code&gt;randomforest&lt;/code&gt; scoring. To take care of it you should select a randomized sample for test/train from your population and then harmonize the levels of the categorical variable using the techniques given here: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://stats.stackexchange.com/questions/29446/random-forest-how-to-handle-new-factor-levels-in-test-set&quot;&gt;Harmonize levels of a categorical variable in test/train&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="15790" LastActivityDate="2018-04-30T14:14:31.800" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31040" PostTypeId="1" CreationDate="2018-04-30T14:47:21.570" Score="3" ViewCount="2337" Body="&lt;p&gt;I am looking for a good book about unsupervised learning that goes beyond the typical k-means and hierarchical clustering algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Practical implementations in R or Python will be a plus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I am working in retail, so the typical use cases I am interested are customer segmentation, products segmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas?&lt;/p&gt;&#xA;" OwnerUserId="51408" LastActivityDate="2018-04-30T15:04:35.640" Title="Good books on unsupervised learning" Tags="|clustering|unsupervised-learning|books|" AnswerCount="1" CommentCount="1" ClosedDate="2018-05-02T03:13:36.017" ContentLicense="CC BY-SA 3.0" />
  <row Id="31041" PostTypeId="1" AcceptedAnswerId="31045" CreationDate="2018-04-30T14:55:54.370" Score="51" ViewCount="56039" Body="&lt;p&gt;&quot;One common mistake that I would make is adding a non-linearity to my logits output.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What does the term &quot;logit&quot; means here or what does it represent ?&lt;/p&gt;&#xA;" OwnerUserId="50247" LastActivityDate="2023-06-05T07:26:49.290" Title="What does Logits in machine learning mean?" Tags="|machine-learning|deep-learning|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31042" PostTypeId="2" ParentId="29164" CreationDate="2018-04-30T14:59:06.013" Score="2" Body="&lt;p&gt;It's perfectly fine to have different features in your training and testing partitions. If the two groups are randomly selected from the same population, one would expect, if there are many possible features, as is frequently the case in text classification, that differences would be observed. To handle this, the standard practice in the field is to train your model on the training partition, and then evaluate on the testing partition with any new features in that grouping being discarded. If you encounter the edge case of a test observation with no features after this procedure, it is common practice to use a heuristic to classify it, bypassing the model entirely (e.g., an observation with no features is classified as neutral).&lt;/p&gt;&#xA;" OwnerUserId="13413" LastActivityDate="2018-04-30T14:59:06.013" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31043" PostTypeId="2" ParentId="31040" CreationDate="2018-04-30T15:04:35.640" Score="2" Body="&lt;p&gt;There might not be an entire book about unsupervised techniques, given the greater effectiveness of supervised learning techniques. However, there are small books and selected chapters from longer books that could be relevant:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&quot;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/B01G1HH5T4&quot; rel=&quot;nofollow noreferrer&quot;&gt;Unsupervised Machine Learning in Python&lt;/a&gt;&quot; which includes Gaussian Mixture Models&lt;/li&gt;&#xA;&lt;li&gt;&quot;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/1787125939&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Machine Learning&lt;/a&gt;&quot; which has a section on Latent Dirichlet allocation (LDA)&lt;/li&gt;&#xA;&lt;li&gt;&quot;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/0133892069&quot; rel=&quot;nofollow noreferrer&quot;&gt;Modeling Techniques in Predictive Analytics with Python and R: A Guide to Data Science&lt;/a&gt;&quot; which has a chapter on Market Basket Analysis&lt;/li&gt;&#xA;&lt;li&gt;&quot;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/3319144359&quot; rel=&quot;nofollow noreferrer&quot;&gt;R for Marketing Research and Analytics&lt;/a&gt;&quot; which covers Confirmatory Factor Analysis and Structural Equation Modeling&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-04-30T15:04:35.640" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31044" PostTypeId="2" ParentId="31041" CreationDate="2018-04-30T15:46:22.843" Score="27" Body="&lt;p&gt;Logits is the unnormalized final scores of your model. You apply softmax to it to get a probability distribution over your classes. &lt;/p&gt;&#xA;" OwnerUserId="21608" LastActivityDate="2018-04-30T15:46:22.843" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31045" PostTypeId="2" ParentId="31041" CreationDate="2018-04-30T16:06:04.120" Score="72" Body="&lt;p&gt;&lt;strong&gt;Logits interpreted to be the unnormalised&lt;/strong&gt; (or not-yet normalised) &lt;strong&gt;predictions&lt;/strong&gt; (or outputs) &lt;strong&gt;of a model. These can give results, but we don't normally stop with logits, because interpreting their raw values is not easy.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Logit&quot; rel=&quot;noreferrer&quot;&gt;their definition&lt;/a&gt; to help understand how &lt;em&gt;logits&lt;/em&gt; are produced.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Let me explain with an example:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;We want to train a model that learns how to classify cats and dogs, using photos that each contain either one cat or one dog. You build a model give it some of the data you have to approximate a mapping between images and predictions. You then give the model some of the unseen photos in order to test its predictive accuracy on new data. As we have a classification problem (we are trying to put each photo into one of two classes), the model will give us two scores for each input image. A score for how likely it believes the image contains a cat, and then a score for its belief that the image contains a dog.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps for the first new image, you get logit values out of &lt;code&gt;16.917&lt;/code&gt; for a cat and then &lt;code&gt;0.772&lt;/code&gt; for a dog. Higher means better, or ('more likely'), so you'd say that a cat is the answer. The correct answer is a cat, so the model worked!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the second image, the model may say the logit values are 1.004 for a cat and 0.709 for a dog. So once again, our model says we the image contains a cat. The correct answer is once again a cat, so the model worked again!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we want to compare the two result. One way to do this is to normalise the scores. That is, &lt;strong&gt;we normalise the logits&lt;/strong&gt;! Doing this we gain some insight into the confidence of our model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's using &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function#Softmax_Normalization&quot; rel=&quot;noreferrer&quot;&gt;the softmax&lt;/a&gt;, where all results sum to &lt;code&gt;1&lt;/code&gt; and so allow us to think of them as probabilities:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} \hspace{20mm}   for \hspace{5mm} j = 1, …, K.$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the first test image, we get&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$prob(cat) = \frac{exp(16.917)}{exp(16.917) + exp(0.772)} = 0.9999$$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$prob(dog) = \frac{exp(0.772)}{exp(16.917) + exp(0.772)} = 0.0001$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we do the same for the second image, we get the results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$prob(cat) = \frac{exp(1.004)}{exp(1.004) + exp(0.709)} = 0.5732$$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$prob(dog) = \frac{exp(0.709)}{exp(1.004) + exp(0.709)} = 0.4268$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model was not really sure about the second image, as it was very close to 50-50 - a guess!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The last part of the quote from your question likely refers to a neural network as the model. The layers of a neural network commonly take input data, multiply that by some parameters (weights) that we want to learn, then apply a &lt;strong&gt;non-linearity&lt;/strong&gt; function, which provides the model with the power to learn non-linear relationships. Without this non-linearity, a neural network would simply be a list of linear operations, performed on some input data, which means it would only be able to learn linear relationships. This would be a massive constraint, meaning the model could always be reduced to a basic linear model.&#xA;That being said, it is not considered helpful to apply a non-linearity to the logit outputs of a model, as you are generally going to be cutting out some information, right before a final prediction is made. Have a look for &lt;a href=&quot;https://stats.stackexchange.com/questions/163695/non-linearity-before-final-softmax-layer-in-a-convolutional-neural-network&quot;&gt;related comments in this thread&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2020-04-28T17:41:45.830" LastActivityDate="2020-04-28T17:41:45.830" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31046" PostTypeId="1" AcceptedAnswerId="31960" CreationDate="2018-04-30T16:14:09.050" Score="1" ViewCount="1852" Body="&lt;p&gt;I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log probability of an action times the reward (or discounted reward). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a state $s$, if I take action $a$ and arrive in state $s'$ I will always see reward $r$. So if I have keep all these values, run an $s$ from the past through my current actor then why can't I update my actor with the new log probability of $a$ and the known reward $r$. I don't need to actually have played and seen a new result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone correct my understanding? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="51411" LastActivityDate="2021-06-01T19:27:04.993" Title="Why are policy gradients on-policy?" Tags="|machine-learning|deep-learning|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31048" PostTypeId="1" AcceptedAnswerId="31055" CreationDate="2018-04-30T17:14:54.773" Score="8" ViewCount="9832" Body="&lt;p&gt;What are the pros / cons of removing stop words from text in the context of a text classification problem, I'm wondering what the best approach is (i.e. to remove or not to remove)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've read somewhere (but can't locate the reference) that it may be detrimental the the performance of a model in the case of sentiment analysis to remove stop words.&lt;/p&gt;&#xA;" OwnerUserId="51090" LastActivityDate="2018-05-01T02:11:47.547" Title="Pros/Cons of stop word removal?" Tags="|nlp|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31049" PostTypeId="1" AcceptedAnswerId="31075" CreationDate="2018-04-30T17:44:19.990" Score="1" ViewCount="139" Body="&lt;p&gt;What is the difference between &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(mtcars, aes(mpg)) +&#xA;  geom_histogram(aes(y = ..density..))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(mtcars, aes(mpg), aes(y = ..density..)) +&#xA;  geom_histogram()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know that aes() i the geom layer overrides the aes() in the data layer. But are one of the code snippets above preferable?&lt;/p&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-05-01T13:49:50.240" Title="ggplot aes() choice" Tags="|r|visualization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31050" PostTypeId="1" CreationDate="2018-04-30T18:01:30.077" Score="5" ViewCount="237" Body="&lt;p&gt;I am working on a text classification problem, and plan on using Naive Bayes based model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In which cases should I consider using Multinomial Naive Bayes?&lt;/p&gt;&#xA;" OwnerUserId="51090" LastActivityDate="2020-04-15T23:00:55.287" Title="When to use Multinomial Naive Bayes?" Tags="|naive-bayes-classifier|machine-learning-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31051" PostTypeId="2" ParentId="17704" CreationDate="2018-04-30T18:12:09.283" Score="0" Body="&lt;p&gt;For classifying a document into Main_Category and Sub_Category, you can use a hierarchical approach. First classify into main category (Sports, Politics, etc.), then classify into sub-category based on the main category (if Sports, then one of Football, Cricket, Tennis, etc.) This gives you the flexibility of choosing the classifier that works best at a given level of hierarchy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sentiment, it is likely that the way sentiment is expressed is common to several categories, so in a sense this is &quot;orthogonal&quot; to category. You could learn a classifier on Positive vs Negative sentiment, using data from all categories together.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you will have a chain of two classifiers -&gt; Main_Category, Sub_Category, and a single classifier -&gt; Sentiment, that you will combine to get the final result.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-04-30T18:12:09.283" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31052" PostTypeId="1" CreationDate="2018-04-30T18:19:15.517" Score="0" ViewCount="2029" Body="&lt;p&gt;So I have a set of 3000 features from which I would like to generate clusters. I passed my features through the T-SNE algorithm to reduce dimensionality to 2 features, and clusters are really visible in this reduced space. How would I go about figuring out a deterministic way to create this same T-SNE output to use for clustering? &lt;/p&gt;&#xA;" OwnerUserId="51416" LastActivityDate="2018-05-01T06:41:23.903" Title="How to recreate T-SNE dimensions deterministically?" Tags="|clustering|unsupervised-learning|tsne|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31053" PostTypeId="2" ParentId="30991" CreationDate="2018-04-30T18:20:36.223" Score="0" Body="&lt;p&gt;There are traditional ways of creating a risk scorecard using linear regression techniques. It's a whole topic in itself. A good book for beginners to read in-depth about it would be &lt;a href=&quot;https://support.sas.com/content/dam/SAS/support/en/books/cedit-risk-scorecards/59376_excerpt.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For sure, you can also tackle it as three-class classification problem if you have labelled data for that. Or consider it as binary classification problem and create three buckets out of class probability output by the classifier.&lt;/p&gt;&#xA;" OwnerUserId="10904" LastActivityDate="2018-04-30T18:20:36.223" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="31054" PostTypeId="1" CreationDate="2018-04-30T18:40:42.090" Score="4" ViewCount="8280" Body="&lt;p&gt;I'm trying to do some PairPlots using seaborn, so I can compare a bunch of features against the label I'm trying to model. There are a bunch of features though (~50). So I'm basically doing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.pairplot(train_df,x_vars=[&quot;MoSold&quot;,&quot;GarageArea&quot;,&quot;ScreenPorch&quot;,&quot;OverallQual&quot;],y_vars=&quot;SalePrice&quot;,size=4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where &quot;SalePrice&quot; is the label that I want to compare the others against. For now I don't care about comparing the features to each other.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I'm having, though, is that when I plot this in my Jupyter notebook, it formats it as one long row. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/SoBdh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SoBdh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm only doing it with 4 features here, but when I add as many as I want to, it basically makes the row thinner and longer, and impossible to read (because it's trying to make them all share the same y axis).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'd like is for it to automatically wrap like panda's hist does:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DL1AN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DL1AN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of a few little hacks. I could do several individual PairPlot commands to manually do it, using a for loop or something, but that would be kind of a pain. Is there some more elegant way to do this? Something where I'd specify the number of columns it'll display with, and then it'll make a grid, where the y axis is always SalePrice?&lt;/p&gt;&#xA;" OwnerUserId="50805" LastActivityDate="2018-05-01T03:09:23.347" Title="How can I make seaborn do PairPlot such that it wraps around?" Tags="|pandas|plotting|seaborn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31055" PostTypeId="2" ParentId="31048" CreationDate="2018-04-30T19:03:53.000" Score="9" Body="&lt;p&gt;In the context of sentiment analysis, removing stop words can be problematic if context is affected. For example suppose your stop word corpus includes ‘not’, which is a negation that can alter the valence of the passage. So you have to be cautious of exactly what is being dropped and what consequences it can have.  &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-04-30T19:03:53.000" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31056" PostTypeId="2" ParentId="31054" CreationDate="2018-04-30T19:11:31.440" Score="1" Body="&lt;p&gt;You should not use &lt;code&gt;sns.pairplot&lt;/code&gt; which is for plotting all-by-all comparisons. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should use &lt;a href=&quot;https://seaborn.pydata.org/generated/seaborn.regplot.html?highlight=regplot#seaborn.regplot&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;sns.regplot&lt;/code&gt;&lt;/a&gt; which can specify particular comparisons to make. Then use &lt;a href=&quot;https://jakevdp.github.io/PythonDataScienceHandbook/04.08-multiple-subplots.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;subplots&lt;/a&gt; can control the tiling.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-01T03:09:23.347" LastActivityDate="2018-05-01T03:09:23.347" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31059" PostTypeId="1" AcceptedAnswerId="31127" CreationDate="2018-04-30T21:58:04.087" Score="1" ViewCount="580" Body="&lt;p&gt;Was trying my hand at the Titanic dataset, when I wanted to One Hot Encode a categorical feature, after which I wanted to combine the original data with the new one hot vectors. The datatypes are as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;data&lt;/code&gt; : Pandas Dataframe&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Titles_ohe&lt;/code&gt; : Numpy sparse matrix (float64)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to merge them into a dataframe using &lt;code&gt;np.c_&lt;/code&gt; :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;columns = (list(data))+list(Titles.values)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;data = pd.DataFrame(np.c_[data.values, Titles_ohe.toarray()], columns=columns)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However on checking the data type of the resulting Dataframe, all the attributes have been changed to the &lt;code&gt;object&lt;/code&gt; datatype. Is there any way I can prevent this while using &lt;code&gt;np.c_&lt;/code&gt;, or is there an alternative solution? Thanks in advance for any help!&lt;/p&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2018-05-02T17:25:11.807" Title="np.c_ converts data type to object. Can I prevent that?" Tags="|machine-learning|python|pandas|data-cleaning|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31063" PostTypeId="1" AcceptedAnswerId="31066" CreationDate="2018-05-01T01:07:32.197" Score="0" ViewCount="1639" Body="&lt;p&gt;This is a rather conceptual question. From what I've read I gather that one shot learning is useful for use cases in which you don't have datasets of millions of images of employees etc. By a one shot operation, the network or networks will do a similarity measure to determine whether the given image belongs to a bunch of images or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, some of these siamese networks are trained on millions or hundreds of millions of images. So doesn't that defeat the purpose of one shot learning ? Why can't plain image classification + transfer learning be used instead ? For instance, in face recognition systems, for every image - classify, doesn't fall into any of the classes - reject. If this is a legitimate new employee image, add the image to the database and retrain the network using transfer learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does one shot learning remove the need for this retraining ? Can someone help me with these basic intuitions please&lt;/p&gt;&#xA;" OwnerUserId="39274" LastActivityDate="2018-05-01T05:16:10.500" Title="Advantages of one shot learning over image classification" Tags="|deep-learning|image-classification|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31064" PostTypeId="1" AcceptedAnswerId="31094" CreationDate="2018-05-01T01:21:50.067" Score="0" ViewCount="156" Body="&lt;p&gt;&lt;strong&gt;What I have so far&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of images that I am trying to classify. I can also extract different feature descriptors from the images using algorithms such as hu moments, color histogram, and SIFT. I can the build a vocabularies from each of these algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I do not understand&lt;/strong&gt; is how can I combine these vocabularies together. From my understanding these feature descriptors can be of different size depending on what the features are. What would be the proper way to build this vocabulary from these different algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What is confusing me&lt;/strong&gt; is how would we be combining feature descriptors from different algorithms which have different sizes? SIFT will give a m x 128 array where m is the number of feature descriptors of set size 128. A color histogram may give a count of each color in the range of 0-255. How would I combine these two in a meaningful way?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the paper I was reading which gave me the idea of combining the vocabularies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.robots.ox.ac.uk/~vgg/publications/papers/nilsback06.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Visual Vocabulary for Flower Classification&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is what I was reading before: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://kushalvyas.github.io/BOV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Implementing Bag of Visual words for Object Recognition&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If anyone can point me in the right direction I would greatly appreciate it.&lt;/p&gt;&#xA;" OwnerUserId="51431" LastEditorUserId="51431" LastEditDate="2018-05-01T02:09:12.720" LastActivityDate="2018-05-01T19:19:41.430" Title="BOVW - Combine vocabularies" Tags="|classification|feature-extraction|image-classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31065" PostTypeId="2" ParentId="31048" CreationDate="2018-05-01T02:11:47.547" Score="6" Body="&lt;p&gt;If you are using some bag of words based methods, i.e, countVectorizer or tfidf that works on counts and frequency of the words, removing stopwords is great as it lowers the dimensional space and also a few stop words won't drive your analysis. On the other hand, when you are exploiting the semantics of the given text, say in a seq2seq model, removing stopwords will omit the context and you will end up with ambiguous results. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-05-01T02:11:47.547" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31066" PostTypeId="2" ParentId="31063" CreationDate="2018-05-01T05:16:10.500" Score="1" Body="&lt;p&gt;In the case of face detection in particular it can get very use case specific:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1a) Let's say I am facebook, and I have a billion possible faces of users to tag.  With the one-shot approach, I am storing a representation of each new face as you suggest, I assume it is something like &lt;a href=&quot;https://arxiv.org/pdf/1605.06065.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;One-shot Learning with Memory-Augmented Neural Networks&lt;/a&gt;. Now I have a problem with a 'memory' that there are billion faces, and that is a lot of computation to do for lookup. The representations learned must increase in size w.r.t to the dataset in order to be able to distinguish between instances, so I get hit again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1b) With &lt;a href=&quot;https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf?&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deepface&lt;/a&gt;, which uses the siamese architecture you mention I have the advantage that to classify a new user, I just have to run it on maybe 1k images of friends around the network where that image comes from. I do not have to retrain the network if a new human is found, since after the initial training, there is not such a problem with doing a comparison between images, as the network is doing a one-vs-one, not a one-vs-all query. The network is not learning classes, but representations and the differences between representations that are most indicative of two examples being different classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2a) Now let's say I am a modeling agency with 800 employees, and I have 1000 examples per person. In that case, the one shot learning approach is feasible, especially if  you have downloaded a pretrained network, and do the fine-tuning as you suggested. There is close to no cost for any projected growth (the one-shot learning paper trained on a 4million example dataset). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2b) In this case it may be feasible to finetune a network with just a final layer with 1000 logits (800 for employees, and a 200 for growth). This would be totally infeasible for the billion or even million classes (individual humans) - but still probably not advisable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, it depends on how many examples per class you have and the number of classes you have. Many architectures out there only work when you have millions of images for relatively few classes (this applies to all the imagenet entries). Others are designed to learn from 10k examples or less, with 50 examples per class, and yet others are designed for not explicitly using the notion of classes, when even that becomes too big. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a comprehensive review of face detection check out &lt;a href=&quot;https://arxiv.org/abs/1804.06655v1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Face Recognition: A Survey&lt;/a&gt;. New on arxiv 4/18/2018&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-01T05:16:10.500" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31067" PostTypeId="2" ParentId="13405" CreationDate="2018-05-01T06:26:34.763" Score="2" Body="&lt;p&gt;If $x$ is your input vector, an affine transformation over $x$ will have this form:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;y = Ax+b&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where the coefficients of the matrix $A$ and the vector $b$ are the parameters of the transformation.&#xA;So it's like a linear function over $x$ but it's not a linear mapping in terms of vector spaces, although you can take it to the form of a linear mapping using homogeneous coordinates.&lt;/p&gt;&#xA;" OwnerUserId="51439" LastActivityDate="2018-05-01T06:26:34.763" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31068" PostTypeId="1" CreationDate="2018-05-01T06:32:44.457" Score="3" ViewCount="312" Body="&lt;p&gt;I want to import a TensorFlow pretrained model in OpenCV dnn C++ module.&lt;/p&gt;&#xA;&lt;p&gt;How can this be done? Are there any examples available?&lt;/p&gt;&#xA;" OwnerUserId="51440" LastEditorUserId="85045" LastEditDate="2020-11-15T10:56:28.660" LastActivityDate="2020-11-15T10:56:28.660" Title="Importing TensorFlow trained model in OpenCV" Tags="|tensorflow|opencv|c++|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31069" PostTypeId="2" ParentId="31052" CreationDate="2018-05-01T06:41:23.903" Score="2" Body="&lt;p&gt;Do not use tSNE visualizations for clustering. The results are misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See this great answer: &lt;a href=&quot;https://stats.stackexchange.com/a/264647/7828&quot;&gt;https://stats.stackexchange.com/a/264647/7828&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from that, you just need to fix the initial positions. For examples by fixing the random generator seed. But the fact thar it does &lt;em&gt;not&lt;/em&gt; work every time should already warn you that it is not too reliable...&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-05-01T06:41:23.903" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31070" PostTypeId="2" ParentId="30490" CreationDate="2018-05-01T06:43:03.400" Score="1" Body="&lt;p&gt;Yes, calculating $\sum_{h} e^{-E(v, h)}$ does include summing over $2^N$ possible configurations of the hidden vector $h$. However you don't need to actually calculate it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we have the negative log likelihood function&#xA;$$ \mathcal{L}(W, c, b) = - \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right) $$&#xA;where $p(v^{(t)}) = \frac 1Z \sum_h e^{-E(v^{(t)},h)}$,&#xA;we don't need to calculate $\mathcal{L}(W, c, b)$ directly, instead we want to minimize it using gradient descent. Using $\theta$ as the full parameter vector instead of $W, c, b$ for simplicity,  we get:&#xA;$$\begin{array}{ll}&#xA;    \nabla\mathcal{L}(\theta)&#xA;    &amp;amp;\displaystyle = \frac{\partial}{\partial\theta} \left(- \frac 1T \sum_{t=1}^T \log\left(p(v^{(t)})\right)\right) \\&#xA;    &amp;amp;\displaystyle= \frac 1T \sum_{t=1}^T \left( &#xA;        \sum_h p(h \mid v^{(t)}) \frac{\partial E(v^{(t)},h)}{\partial\theta} - \sum_h\sum_vp(v,h)\frac{\partial E(v,h)}{\partial\theta}&#xA;    \right)&#xA;\end{array}$$&#xA;The first term of the difference above is the expected value of $\frac{\partial E(v^{(t)},h)}{\partial\theta}$, given a specific visible vector (i.e. a training example) $v^{(t)}$. This can easily be calculated for each parameter $W, c, b$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second term is the expected value of the energy, calculated over the full distribution of the RBM. Unfortunately, this is again intractable. Here, the contrastive divergence algorithm by Hinton comes in useful: we run Gibbs sampling for a number of steps and estimate the second term using the resulting states $\tilde{v}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;tl;dr:&lt;/strong&gt; yes, the problem is intractable, which is why we use the contrastive divergence algorithm by Hinton, which is only an approximation but works well in practice.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastActivityDate="2018-05-01T06:43:03.400" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="31071" PostTypeId="1" CreationDate="2018-05-01T07:00:47.673" Score="3" ViewCount="267" Body="&lt;p&gt;What are the issues of dealing with highly skewed variable in a supervised problem? What are the machine learning algorithms that suffer more from skewness in the data and what are the solutions to this problem? &lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-01T07:00:47.673" Title="Skewed distributions in predictive models" Tags="|statistics|feature-selection|feature-engineering|" AnswerCount="0" CommentCount="4" ClosedDate="2020-10-03T21:04:43.410" ContentLicense="CC BY-SA 3.0" />
  <row Id="31073" PostTypeId="1" CreationDate="2018-05-01T11:07:26.580" Score="1" ViewCount="32" Body="&lt;p&gt;I am trying to implement feature selection using &lt;code&gt;regsubsets&lt;/code&gt; function. &#xA;I would like to know which algorithm is used in the background in order to build the model when we use &lt;code&gt;regsubsets&lt;/code&gt;?&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="26562" LastEditDate="2018-05-01T11:26:23.230" LastActivityDate="2018-05-01T11:26:23.230" Title="regsubsets function in R for feature selection" Tags="|machine-learning|r|feature-selection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31075" PostTypeId="2" ParentId="31049" CreationDate="2018-05-01T13:49:50.240" Score="2" Body="&lt;p&gt;The difference is that when the aesthetics are set in the ggplot function, they are inherited by any other geom's that build on top of it. If you specify the aesthetics only in a geom, it will only be used in that geom. And, as you mentioned, any aesthetics used in the geom override the settings in the ggplot function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as which is preferable, I think it depends on you goal with the ggplot objects. For example, if you are only creating a single plot then it doesn't really matter which method you use. However, if you plan to present multiple different visualizations of the same data, you could reuse the ggplot object and simply add different geom layers to it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Create a reusable ggplot object:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p &amp;lt;- ggplot(mtcars)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Add histogram geom:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p + geom_histogram( aes(mpg, ..density..))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Reuse ggplot object with different geom:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;p + geom_point(aes(cyl, mpg))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a simple example but you can understand that when creating more complicated visualizations, the ability to reuse plot objects comes in handy.&lt;/p&gt;&#xA;" OwnerUserId="51453" LastActivityDate="2018-05-01T13:49:50.240" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31077" PostTypeId="1" AcceptedAnswerId="32446" CreationDate="2018-05-01T14:54:48.717" Score="3" ViewCount="1801" Body="&lt;p&gt;I have implemented a vanilla GAN which gave good results very fast but it had a lot of mode collapse issue, because of this I learned about WGAN which suppose to fix this, in fact they claim they have never encountered mode collapse problem which is great. My main issue is how slow WGAN takes to converge to good results. With a vanilla GAN by epoch 10 I was getting good looking generations, with WGAN I am still seeing noise images on epoch 50! Besides bumping up the learning rate is there anything else I can do? The network is a modified smaller DCGAN that has 1 less output layer than the original. So instead of the extra 1024 output layer mine stops at 512. I knew WGAN was going to be slow from all the research I have done but I didn't except to be this slow. Can some one point me to the right direction on how to better optimize WGAN's speed? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have googled around but most results are just vague comment chains or agreements that there is a speed problem with no further leads on what you should do next to counter this issue.&lt;/p&gt;&#xA;" OwnerUserId="46645" LastActivityDate="2018-05-31T10:38:11.943" Title="WGAN is too slow what are some ways to tweak for speed ups?" Tags="|training|gan|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31078" PostTypeId="1" CreationDate="2018-05-01T15:08:20.737" Score="1" ViewCount="392" Body="&lt;p&gt;On an eCommerce website we want to create some personalization for visitors who are more likely to make a purchase.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume we only have one single item for sale.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The likelihood should be based on data that we have, for example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Price of the item when the customer visited (it may vary)&lt;/li&gt;&#xA;&lt;li&gt;Average customer reviews rating of the item&lt;/li&gt;&#xA;&lt;li&gt;If a discount coupon was offered or not &lt;/li&gt;&#xA;&lt;li&gt;If certain words were present in the item title or description&lt;/li&gt;&#xA;&lt;li&gt;etc.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Our plan is to create a DB that will contain each visitor's visit data (the above data), and indication if eventually he made a purchase or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once we have at least a few purchases, we need to somehow look for a pattern among all these cases, and based on this pattern we can later forecast the likelihood.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if at least in &lt;code&gt;50%&lt;/code&gt; of the cases that the item was purchased, its price was no more than &lt;code&gt;$100&lt;/code&gt;, than we will make personalization only on cases where the item is being sold at a max price of &lt;code&gt;$100&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should we do it technically? From both DB structure concerns, and website speed concern - &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should we collect each variable in its own DB column, or is it better to create some kind of long string in the browser session like &lt;code&gt;p:99-r:4&lt;/code&gt;,  (which means: &lt;code&gt;price $99&lt;/code&gt;, &lt;code&gt;rating: 4&lt;/code&gt;) and then later break this data down using another script?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there some kind of library that can analyze this data and show it maybe in graph, like some fancy fronted, or does this project need to be made from scratch?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What kind of technical skills does the person who do it need to have? Is this related to machine-learning?&lt;/p&gt;&#xA;" OwnerUserId="51446" LastActivityDate="2018-05-01T22:25:38.863" Title="Visitor's probability to purchase on eCommerce site, based on aggregate historic data" Tags="|machine-learning|deep-learning|predictive-modeling|dataset|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31080" PostTypeId="1" AcceptedAnswerId="31084" CreationDate="2018-05-01T15:28:02.260" Score="1" ViewCount="986" Body="&lt;p&gt;Can someone explain to me how the predict method of the perceptron algorithm works? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def predict(self, pattern):&#xA;    activation = np.dot(self.w, pattern) + self.b&#xA;    if activation &amp;gt; 0:&#xA;        return 1&#xA;    else:&#xA;        return -1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this example b stands for bias. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def train(self, pattern, label, step = 0.01):&#xA;    out_label = self.predict(pattern)&#xA;    if out_label == label:&#xA;        return True&#xA;    self.w += step * (label - out_label) * pattern&#xA;    self.b += step * (label - out_label)&#xA;    return False&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the train method. But as far as I understand in out_label there will only be either 1 or -1. How could this be possibly equal to label? If we would return activation I (think) I would understand it.&lt;/p&gt;&#xA;" OwnerUserId="51462" LastActivityDate="2018-05-01T17:21:02.400" Title="Predict method of the perceptron algorithm" Tags="|python|perceptron|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 3.0" />
  <row Id="31082" PostTypeId="2" ParentId="30957" CreationDate="2018-05-01T16:21:59.823" Score="1" Body="&lt;p&gt;On principle, you should try to achieve a fairly balanced distribution between your training and testing sets. Otherwise your model might not make appropriate generalizations.  Additionally, you should use some form of &lt;a href=&quot;https://en.wikipedia.org/wiki/Normalization_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;normalization&lt;/a&gt; such as &lt;a href=&quot;https://en.wikipedia.org/wiki/Feature_scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;feature scaling&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is difficult (or even impossible) to understand your data distribution by simply looking at the range.  It would be better to look at the mean/median or even a histogram/normal curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say, for example, that our ordered column A in your training/testing sets look as such (before normalization):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{|c|}&#xA;\hline&#xA;&amp;amp;column \ A \ split\\&#xA;\hline&#xA;training &amp;amp;&amp;amp; testing \\&#xA;0 &amp;amp;&amp;amp; 0\\&#xA;4 &amp;amp;&amp;amp; 2\\&#xA;5 &amp;amp;&amp;amp; 3\\&#xA;7 &amp;amp;&amp;amp; 3\\&#xA;9 &amp;amp;&amp;amp; 8\\&#xA;15 &amp;amp;&amp;amp; 11\\&#xA;23 &amp;amp;&amp;amp; 15\\&#xA;30 &amp;amp;&amp;amp; 80\\&#xA;\hline&#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{|c|}&#xA;\hline&#xA;&amp;amp;set\ metrics\\&#xA;\hline&#xA;&amp;amp; training &amp;amp; testing\\&#xA;range &amp;amp; [0,30] &amp;amp; [0,80]\\&#xA;mean &amp;amp; 11.625 &amp;amp; 15.250\\&#xA;median &amp;amp; 8.0 &amp;amp; 5.5\\&#xA;stddev &amp;amp; 10.309 &amp;amp; 26.650\\&#xA;\hline&#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align}&#xA;\large\frac{1}{\sqrt{2\pi}\cdot\sigma}e^{\frac{-(x-\mu)^2}{2\sigma^2}}&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PMRxo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PMRxo.png&quot; alt=&quot;training/testing curves&quot;&gt;&lt;/a&gt;&#xA;(red=training, blue=testing)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we can see that &lt;strong&gt;despite the large range discrepancy, our sets are relatively balanced.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can observe a problematic training/testing split in this example where we shift the training mean from 11.625 to 100:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mlGM9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mlGM9.png&quot; alt=&quot;bad training/testing split&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, we could generate another hypothetical dataset with a bad training/testing split where our set ranges are identical.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do not evaluate the set distribution by range; instead use mean/median or histogram/normal curve.  Also &lt;a href=&quot;https://en.wikipedia.org/wiki/Normalization_(statistics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;normalize&lt;/a&gt; your data.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-05-01T17:28:24.963" LastActivityDate="2018-05-01T17:28:24.963" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31084" PostTypeId="2" ParentId="31080" CreationDate="2018-05-01T17:21:02.400" Score="0" Body="&lt;p&gt;Here is how the code appears to break down:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;method parameters:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;train(...)&lt;/code&gt; method takes an input vector called &lt;code&gt;pattern&lt;/code&gt; along with the target output called &lt;code&gt;label&lt;/code&gt; and a weight update coefficient (learning rate) called &lt;code&gt;step&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;method execution:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;train(...)&lt;/code&gt; first passes the input vector through our neuron using the &lt;code&gt;predict(...)&lt;/code&gt; method.&lt;/li&gt;&#xA;&lt;li&gt;The result of &lt;code&gt;predict(...)&lt;/code&gt; is stored in a variable named &lt;code&gt;out_label&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;code&gt;out_label&lt;/code&gt; variable is either &lt;code&gt;1&lt;/code&gt; or &lt;code&gt;-1&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Depending on the pattern that we provide we want the target output (&lt;code&gt;label&lt;/code&gt;) to be either &lt;code&gt;1&lt;/code&gt; or &lt;code&gt;-1&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If &lt;code&gt;out_label&lt;/code&gt; is equal to &lt;code&gt;label&lt;/code&gt; (&lt;code&gt;1 == 1&lt;/code&gt; or &lt;code&gt;-1 == -1&lt;/code&gt;), then we return &lt;code&gt;True&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Otherwise, the neuron outputted a value we do not want (&lt;code&gt;-1 != 1&lt;/code&gt; or &lt;code&gt;1 != -1&lt;/code&gt;), so we update the weights accordingly, then return &lt;code&gt;False&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;example where the neuron is &lt;em&gt;correct&lt;/em&gt;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have the following parameters:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;input vector (&lt;code&gt;pattern&lt;/code&gt;) = &lt;code&gt;[-5,2,8]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;target output (&lt;code&gt;label&lt;/code&gt;) = &lt;code&gt;1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;train(...)&lt;/code&gt; method executes as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;we call the &lt;code&gt;predict(...)&lt;/code&gt; which does the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;passes our vector &lt;code&gt;[-5,2,8]&lt;/code&gt; through the neuron using the function &lt;code&gt;np.dot(self.w, pattern) + self.b&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;stores the result of the above function in a variable called &lt;code&gt;activation&lt;/code&gt; (let's say &lt;code&gt;activation = 0.23&lt;/code&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;returns &lt;code&gt;1&lt;/code&gt; because &lt;code&gt;activation &amp;gt; 0&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;we store the result of &lt;code&gt;predict(...)&lt;/code&gt; in the variable &lt;code&gt;out_label&lt;/code&gt;, which is &lt;code&gt;1&lt;/code&gt; in this example&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;because our target output (&lt;code&gt;label = 1&lt;/code&gt;), is equal to the actual output (&lt;code&gt;out_label = 1&lt;/code&gt;) of the neuron, we return &lt;code&gt;True&lt;/code&gt; and consider the neuron output correct.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;example where the neuron is &lt;em&gt;incorrect&lt;/em&gt;:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say we have the following parameters:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;input vector (&lt;code&gt;pattern&lt;/code&gt;) = &lt;code&gt;[-1,-5,3]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;target output (&lt;code&gt;label&lt;/code&gt;) = &lt;code&gt;1&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;code&gt;train(...)&lt;/code&gt; method executes as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;we call the &lt;code&gt;predict(...)&lt;/code&gt; which does the following:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;passes our vector &lt;code&gt;[-1,-5,3]&lt;/code&gt; through the neuron using the function &lt;code&gt;np.dot(self.w, pattern) + self.b&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;stores the result of the above function in a variable called &lt;code&gt;activation&lt;/code&gt; (let's say &lt;code&gt;activation = -0.65&lt;/code&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;returns &lt;code&gt;-1&lt;/code&gt; because &lt;code&gt;activation &amp;lt;= 0&lt;/code&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;we store the result of &lt;code&gt;predict(...)&lt;/code&gt; in the variable &lt;code&gt;out_label&lt;/code&gt;, which is &lt;code&gt;-1&lt;/code&gt; in this example&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;because our target output (&lt;code&gt;label = 1&lt;/code&gt;), is &lt;em&gt;not&lt;/em&gt; equal to the actual output (&lt;code&gt;out_label = -1&lt;/code&gt;) of the neuron, we update the weights using:&lt;br&gt;&#xA;&lt;code&gt;self.w += step * (label - out_label) * pattern&#xA;self.b += step * (label - out_label)&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;we then return &lt;code&gt;False&lt;/code&gt; because the neuron was incorrect&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-05-01T17:21:02.400" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31085" PostTypeId="2" ParentId="30881" CreationDate="2018-05-01T17:21:53.333" Score="11" Body="&lt;p&gt;Which is more important simply depends on what the costs of each error are.&lt;/p&gt;&#xA;&lt;p&gt;Precision tends to involve direct costs; the more false positives you have, the more cost per true positive you have. If your costs are low, then precision does not matter as much. For instance, if you have 1M email addresses, and it will cost $10 to send an email to all of them, it is probably not worth your time to try to identify the people most likely to respond, rather than just spamming all of them.&lt;/p&gt;&#xA;&lt;p&gt;Recall, on the other hand, tends to involve opportunity costs; you are giving up opportunities every time you have a false negative. So recall is least important when the marginal value of additional correct identification is small, e.g. there are multiple opportunities, there is little difference between them, and only a limited number can be pursued. For instance, suppose you would like to buy an apple. There are 100 apples at the store, and 10 of them are bad. If you have a method of distinguishing bad apples that misses 80% of good ones, then you will identify about 18 good apples. Normally, a recall of 20% would be terrible, but if you only want 5 apples, then missing those other 72 apples does not really matter.&lt;/p&gt;&#xA;&lt;p&gt;So recall is most important when:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The number of opportunities is small (if there were only 10 good apples, then you would be unlikely to find 5 good ones with a recall rate of only 20%)&lt;BR&gt;&lt;/li&gt;&#xA;&lt;li&gt;There are significant differences between opportunities (if some apples are better than others, then a recall rate of 20% is enough to get 5 good apples, but they are not necessarily going to be the &lt;em&gt;best&lt;/em&gt; apples)&lt;br&gt;&#xA;OR&lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;The marginal benefit of opportunities remains high, even for a large number of opportunities. For instance, while most shoppers would not have much benefit from more than 18 good apples, the &lt;em&gt;store&lt;/em&gt; would like to have more than 18 apples to sell.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Thus, precision will be more important than recall when the cost of acting is high, but the cost of not acting is low. Note that this is the cost of acting/not acting per candidate, not the &amp;quot;cost of having any action at all&amp;quot; versus the &amp;quot;cost of not having any action at all&amp;quot;. In the apple example, it is the cost of buying/not buying a particular apple, not the cost of buying some apples versus the cost of not buying any apples; the cost of not buying a particular apple is low because there are lots of other apples. Since the cost of buying a bad apple is high, but the cost of passing up a particular good apple is low, precision is more important in that example. Another example would be hiring when there are a lot of similar candidates.&lt;/p&gt;&#xA;&lt;p&gt;Recall is more important than precision when the cost of acting is low, but the opportunity cost of passing up on a candidate is high. There is the spam example I gave earlier (the cost of missing out on an email address is not high, but the cost of sending out an email to someone who does not respond is even lower), and another example would be identifying candidates for the flu shot: give the flu shot to someone who does not need it, and it costs a few dollars, do not give it to someone who does need it, and they could die. Because of this, health care plans will generally offer the flu shot to everyone, disregarding precision entirely.&lt;/p&gt;&#xA;" OwnerUserId="51467" LastEditorUserId="29169" LastEditDate="2021-11-17T18:52:30.613" LastActivityDate="2021-11-17T18:52:30.613" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31086" PostTypeId="2" ParentId="18323" CreationDate="2018-05-01T17:56:49.967" Score="2" Body="&lt;p&gt;For your situation, I would recommend using an object detection network to both locate and classify your components.  Although you do not need the location or classification data, you can still count the number of objects the network detects.  Moreover, this approach does not require any data science background because there are many tutorials on this subject and tons of out-of-the-box tools, such as &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/research/object_detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow's object detection API&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will need to annotate your data, but there are tons of programs to help you with that too.  &lt;a href=&quot;https://github.com/tzutalin/labelImg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's one I found quickly&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest using faster-RCNN with the NAS feature extractor because, although it is slower than other CNN architectures, it is the most accurate.  Also, the network would probably perform better if you did not combine all the components into a single class.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The steps necessary to accomplish this task are essentially the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Install and setup &lt;a href=&quot;https://www.tensorflow.org/install/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;Download faster RCNN NAS from the &lt;a href=&quot;https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow detection model zoo&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/tzutalin/labelImg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Annotate&lt;/a&gt; your data by drawing bounding boxes around the components and labeling each bounding box.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://becominghuman.ai/tensorflow-object-detection-api-tutorial-training-and-evaluating-custom-object-detector-ed2594afcf73&quot; rel=&quot;nofollow noreferrer&quot;&gt;Train your model&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;After saving the trained model, you just need to setup a pipeline that takes an input image, feeds it to your network, and returns the total number of detections ignoring the bounding box and classification data.  &lt;a href=&quot;https://medium.com/@WuStangDan/step-by-step-tensorflow-object-detection-api-tutorial-part-5-saving-and-deploying-a-model-8d51f56dbcf1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a great tutorial for deploying a trained model&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-05-01T17:56:49.967" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31088" PostTypeId="1" CreationDate="2018-05-01T18:23:14.867" Score="1" ViewCount="113" Body="&lt;p&gt;I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using &lt;code&gt;doc2vec&lt;/code&gt; to create document vectors and the &lt;code&gt;SVC&lt;/code&gt; classifier from &lt;code&gt;sklearn&lt;/code&gt; to make predictions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my learning curve and the mean scores:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GsC2K.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GsC2K.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/k0YFl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/k0YFl.png&quot; alt=&quot;Learning curve after adjusting gamme&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.&lt;/p&gt;&#xA;" OwnerUserId="50668" LastEditorUserId="50668" LastEditDate="2018-05-01T22:40:54.533" LastActivityDate="2018-05-01T22:40:54.533" Title="How much more data does my classifier need?" Tags="|scikit-learn|multiclass-classification|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 3.0" />
  <row Id="31089" PostTypeId="1" CreationDate="2018-05-01T18:27:02.020" Score="2" ViewCount="818" Body="&lt;p&gt;I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.&lt;/p&gt;&#xA;" OwnerUserId="51472" LastActivityDate="2018-11-23T12:25:54.713" Title="How can I measure the similarity between 2 IP addresses? Is there any code to re-use?" Tags="|preprocessing|similarity|distance|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 3.0" />
  <row Id="31090" PostTypeId="1" CreationDate="2018-05-01T18:49:01.800" Score="2" ViewCount="190" Body="&lt;p&gt;I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly classify the type of failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Background:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is collected at random time intervals of about 5 to 10 minutes in 1-minute bursts. Each 1-minute burst contains 4 groups of distinct sensors for about 15 seconds each. These groups are organized by sensors commonly used together for manual diagnosis by a technician. The idea is to hopefully be able to capture important relationships between these sensors and the 'degradation' over time. There are many engines that are missing data because of unreliable communication interfaces during collection. Another layer of complexity is that there is not a feature for defining the type of each engine. EX: Manufacture or different engine stats &#xA;There are not a lot of data on engines with a failure as there are only 800 engines, in 8 possible classes of failures, out of the 20,000 engines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I might have gotten ahead of myself, but here is what I have done so far:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;resampled the data into days by getting the min, max, mean, std, 25%, 50%, 75%&lt;/li&gt;&#xA;&lt;li&gt;graphed a correlation matrix&lt;/li&gt;&#xA;&lt;li&gt;graphed some sensor data to verify the data shows signs of 'degrading' over time&lt;/li&gt;&#xA;&lt;li&gt;ran some simple ML classifiers and regressors to get a baseline auc and accuracy&lt;/li&gt;&#xA;&lt;li&gt;used some unsupervised learning techniques to create new features&#xA;EX: used dynamic time warping as a distance metric for DBscan clustering &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Right now, I don't see much correlation between the engineered features and didn't get many good results with any supervised ML. I have not yet tried LSTMs or variations on the OneVsRest classifier, but from what I have read those are popular techniques for these types of problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas how to explore the data further?&#xA;Any ideas how to create new features, resample, or extract the relationships of the sensor of time series data?&lt;/p&gt;&#xA;" OwnerUserId="51426" LastEditorUserId="51426" LastEditDate="2018-05-01T22:15:33.773" LastActivityDate="2018-05-01T22:15:33.773" Title="Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines" Tags="|time-series|lstm|unsupervised-learning|feature-engineering|supervised-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31091" PostTypeId="2" ParentId="31088" CreationDate="2018-05-01T18:56:13.713" Score="2" Body="&lt;p&gt;I think you already have enough data. Your problem seems to be one of generalization, which is in general a tricky one to solve.&#xA;For SVM, depending on what kind you have, you could try tinkering with the parameters and kernels.&#xA;Have a look here. -- &lt;a href=&quot;https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&quot;&gt;https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51474" LastActivityDate="2018-05-01T18:56:13.713" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31092" PostTypeId="1" CreationDate="2018-05-01T19:10:04.023" Score="2" ViewCount="205" Body="&lt;p&gt;Wasserstein distance between two gaussians has a closed form solution. Does the same hold for the distance between a Gaussian with a fixed variance(say 1) and the empirical data distribution? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Empirical data distibution defined as:&#xA;$$ p(x) = \frac{\sum_i \delta (x - x_i)}{n} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the 1-d Gaussian with $\sigma^2 = 1$ and some unknown mean $\mu$&#xA;$$ q(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$&lt;/p&gt;&#xA;" OwnerUserId="51474" LastActivityDate="2018-05-01T19:10:04.023" Title="Wasserstein distance between Gaussian and the empirical distribution" Tags="|distribution|gaussian|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31093" PostTypeId="2" ParentId="26186" CreationDate="2018-05-01T19:13:50.527" Score="0" Body="&lt;p&gt;You can model the data in a Bayesian way to capture the robustness of parameter estimates.  &lt;a href=&quot;https://www.chrisstucchio.com/blog/2013/bayesian_bandit.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Construct a probability distribution&lt;/a&gt; for each estimated CTR. Each distribution will have the same mean but the page with fewer samples will have greater variance (i.e., less confidence that the population parameter is near the observed data) than the page with more samples.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-01T19:13:50.527" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31094" PostTypeId="2" ParentId="31064" CreationDate="2018-05-01T19:19:41.430" Score="0" Body="&lt;p&gt;I also found an interesting paper (with code) where they merged local and global descriptors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/flytxtds/scene-recognition/blob/master/scene_recognition_local_global_approach.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scene Recognition by Combining Local and Global Image Descriptors&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the paper: They take the daisy features (local features), cluster the features into k clusters, then create a histogram with k bins. They normalize the histogram, once they do this they concatenate this vector (the histogram) with the normalized HOG vector (global features). So in the end we just cluster the local features and create histogram, then just concat with global vector and there we have a training sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also there may be an alternate method where you just use local features to cluster: &lt;a href=&quot;https://www.quora.com/How-do-I-merge-features-from-different-feature-extractors-i-e-color-histogram-and-SIFT-for-bag-of-visual-words#&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-do-I-merge-features-from-different-feature-extractors-i-e-color-histogram-and-SIFT-for-bag-of-visual-words#&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51431" LastActivityDate="2018-05-01T19:19:41.430" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31096" PostTypeId="2" ParentId="30957" CreationDate="2018-05-01T21:54:40.327" Score="0" Body="&lt;p&gt;The range of the dataset values can be influenced by extreme values (outliers), so it is not the right way to check if the training set and the test set follow the same distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all you need to make a comparison using the five-number summary (min-Q1-median-Q3-max) and make a first conclusion for the distribution equality.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A second option could be to perform a statistical test as the Kolmogorov-Smirnov test to check for distribution equality and see if there is a significant difference or not between the two samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the is a significant difference, then the classifier trained using a dataset with different distribution from the test set will have a poor performance on the test set. A solution could be to use Transfer Learning/Domain Adaptation Methods to tackle this difference and adapt the classifier trained on a different distribution for the test set instances. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-01T21:54:40.327" CommentCount="0" ContentLicense="CC BY-SA 3.0" />
  <row Id="31097" PostTypeId="2" ParentId="31078" CreationDate="2018-05-01T22:25:38.863" Score="1" Body="&lt;p&gt;This is a big topic known as Recommender Systems or Recommendation Engines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common method is use what is known as &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;collaborative filtering&lt;/a&gt;. Most methods on the internet try to guess what the user will rank an item, but I think this can be extended to if the user will make a purchase or not, given that if a user were to rank something highly they would also be more likely to buy that item.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that each item has a n-dimensional vector, and each user has an n-dimensional vector. You can predict a user's predicted rating by some operation between the two vectors. In this example lets use the dot product between the two vectors as a means to predict the user's rating for an item they haven't yet seen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While taking the 2 vectors and getting a prediction is not a machine learning problem, finding suitable values for the vectors is most definitely a machine learning problem. I won't get too deep into how you can get these vectors because many different algorithms apply and it seems that you are asking more of a system design question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have trained 1 vector for each user and 1 vector for each item, you have 2 more problems to solve:&#xA;1. Estimate the top N candidate items to recommend&#xA;2. Take the dot product between each of these N items and recommend the few with the highest score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number 1 will likely involve some use of spatial indexing in your database (example PostGIS for PSQL) so you can find some vectors that would give reasonably good dot-products. You may want to consider using a ball tree data structure to search for a good set of candidate items.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Number 2 will be pretty simple by just dotting the vectors and picking the best few.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will be a big challenge for you to undertake but I'm certain you'll learn a lot. Since it was such a big system you want to create I wasn't able to go into detail but hopefully, I pointed you in the right direction.&lt;/p&gt;&#xA;" OwnerUserId="45289" LastActivityDate="2018-05-01T22:25:38.863" CommentCount="2" ContentLicense="CC BY-SA 3.0" />
  <row Id="31098" PostTypeId="1" CreationDate="2018-05-01T22:38:38.473" Score="1" ViewCount="105" Body="&lt;p&gt;Let's say I want to find the probabilities of winning the best movie category in the Oscars. I know to following rules:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;There is only 1 winner per year. &lt;/li&gt;&#xA;&lt;li&gt;Logically, the sum of the predicted probabilities for each year should add up to be 1. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I can have a year feature in my classifier but that does not mean that 1 and 2 are enforced. If I use a simple classifier like logistic regression I also don't see how having the year as a feature is going to help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to enforce those rules in a model? Does it matter? &#xA;Should I just use a model without those rules and normalize the probabilities to 1 when I try to predict for a given year?&lt;/p&gt;&#xA;" OwnerUserId="51478" LastActivityDate="2024-02-04T00:04:35.537" Title="Classifier with a single 1 value per year" Tags="|classification|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 3.0" />
  <row Id="31100" PostTypeId="1" AcceptedAnswerId="31106" CreationDate="2018-05-02T02:09:59.287" Score="1" ViewCount="133" Body="&lt;p&gt;I am testing the pretraining example in Chapter 15 of Aurélien Géron's book &quot;Hands-On Machine Learning with Scikit-Learn and TensorFlow&quot;. The code is on his github page: &lt;a href=&quot;https://github.com/ageron/handson-ml/blob/master/15_autoencoders.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; - see the example in section &quot;Unsupervised pretraining&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Pretraining the network with the weights from the previously trained encoder should assist in training the network. To check this I slightly modified Aurelien's code so it outputs the error after every batch, and also reduced the batch size. I did this so I could see the error at the start of the training, where the effect of the pretrained weights should have been most obvious. I expected the pretrained network would start with a lower error (compared to the network not using pretraining) because it was starting with pretrained weights. However the pretraining seems to make training slower.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Has anyone any idea why this could be?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first few lines of output (when using pretraining) is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 Train accuracy after each mini-batch: 0.08&#xA;0 Train accuracy after each mini-batch: 0.24&#xA;0 Train accuracy after each mini-batch: 0.32&#xA;0 Train accuracy after each mini-batch: 0.2&#xA;0 Train accuracy after each mini-batch: 0.32&#xA;0 Train accuracy after each mini-batch: 0.26&#xA;0 Train accuracy after each mini-batch: 0.32&#xA;0 Train accuracy after each mini-batch: 0.5&#xA;0 Train accuracy after each mini-batch: 0.58&#xA;0 Train accuracy after each mini-batch: 0.48&#xA;0 Train accuracy after each mini-batch: 0.54&#xA;0 Train accuracy after each mini-batch: 0.48&#xA;0 Train accuracy after each mini-batch: 0.5&#xA;0 Train accuracy after each mini-batch: 0.56&#xA;0 Train accuracy after each mini-batch: 0.64&#xA;0 Train accuracy after each mini-batch: 0.56&#xA;0 Train accuracy after each mini-batch: 0.68&#xA;0 Train accuracy after each mini-batch: 0.62&#xA;0 Train accuracy after each mini-batch: 0.74&#xA;0 Train accuracy after each mini-batch: 0.78&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you can see the accuracy is initially low. In contrast, when using He-initialized weights (i.e. not using pretraining), the initial accuracy is actually higher:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0 Train accuracy after each mini-batch: 0.62&#xA;0 Train accuracy after each mini-batch: 0.5&#xA;0 Train accuracy after each mini-batch: 0.52&#xA;0 Train accuracy after each mini-batch: 0.38&#xA;0 Train accuracy after each mini-batch: 0.56&#xA;0 Train accuracy after each mini-batch: 0.56&#xA;0 Train accuracy after each mini-batch: 0.6&#xA;0 Train accuracy after each mini-batch: 0.7&#xA;0 Train accuracy after each mini-batch: 0.72&#xA;0 Train accuracy after each mini-batch: 0.86&#xA;0 Train accuracy after each mini-batch: 0.86&#xA;0 Train accuracy after each mini-batch: 0.8&#xA;0 Train accuracy after each mini-batch: 0.82&#xA;0 Train accuracy after each mini-batch: 0.84&#xA;0 Train accuracy after each mini-batch: 0.88&#xA;0 Train accuracy after each mini-batch: 0.9&#xA;0 Train accuracy after each mini-batch: 0.82&#xA;0 Train accuracy after each mini-batch: 0.9&#xA;0 Train accuracy after each mini-batch: 0.84&#xA;0 Train accuracy after each mini-batch: 0.98&#xA;0 Train accuracy after each mini-batch: 0.96&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In other words, the pretraining seems to slow down training, the opposite of what it should be doing! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My modified code is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import sys&#xA;import tensorflow as tf&#xA;from tensorflow.examples.tutorials.mnist import input_data&#xA;&#xA;&#xA;def reset_graph(seed=42):&#xA;    tf.reset_default_graph()&#xA;    tf.set_random_seed(seed)&#xA;    np.random.seed(seed)&#xA;&#xA;&#xA;def train_stacked_autoencoder():&#xA;    reset_graph()&#xA;&#xA;    # Load the dataset to use&#xA;    mnist = input_data.read_data_sets(&quot;/tmp/data/&quot;)&#xA;&#xA;    n_inputs = 28 * 28&#xA;    n_hidden1 = 300&#xA;    n_hidden2 = 150  # codings&#xA;    n_hidden3 = n_hidden1&#xA;    n_outputs = n_inputs&#xA;&#xA;    learning_rate = 0.01&#xA;    l2_reg = 0.0001&#xA;&#xA;    activation = tf.nn.elu&#xA;    regularizer = tf.contrib.layers.l2_regularizer(l2_reg)&#xA;    initializer = tf.contrib.layers.variance_scaling_initializer()&#xA;&#xA;    X = tf.placeholder(tf.float32, shape=[None, n_inputs])&#xA;&#xA;    weights1_init = initializer([n_inputs, n_hidden1])&#xA;    weights2_init = initializer([n_hidden1, n_hidden2])&#xA;    weights3_init = initializer([n_hidden2, n_hidden3])&#xA;    weights4_init = initializer([n_hidden3, n_outputs])&#xA;&#xA;    weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=&quot;weights1&quot;)&#xA;    weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=&quot;weights2&quot;)&#xA;    weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=&quot;weights3&quot;)&#xA;    weights4 = tf.Variable(weights4_init, dtype=tf.float32, name=&quot;weights4&quot;)&#xA;&#xA;    biases1 = tf.Variable(tf.zeros(n_hidden1), name=&quot;biases1&quot;)&#xA;    biases2 = tf.Variable(tf.zeros(n_hidden2), name=&quot;biases2&quot;)&#xA;    biases3 = tf.Variable(tf.zeros(n_hidden3), name=&quot;biases3&quot;)&#xA;    biases4 = tf.Variable(tf.zeros(n_outputs), name=&quot;biases4&quot;)&#xA;&#xA;    hidden1 = activation(tf.matmul(X, weights1) + biases1)&#xA;    hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)&#xA;    hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)&#xA;    outputs = tf.matmul(hidden3, weights4) + biases4&#xA;&#xA;    reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))&#xA;&#xA;    optimizer = tf.train.AdamOptimizer(learning_rate)&#xA;&#xA;    with tf.name_scope(&quot;phase1&quot;):&#xA;        phase1_outputs = tf.matmul(hidden1, weights4) + biases4  # bypass hidden2 and hidden3&#xA;        phase1_reconstruction_loss = tf.reduce_mean(tf.square(phase1_outputs - X))&#xA;        phase1_reg_loss = regularizer(weights1) + regularizer(weights4)&#xA;        phase1_loss = phase1_reconstruction_loss + phase1_reg_loss&#xA;        phase1_training_op = optimizer.minimize(phase1_loss)&#xA;&#xA;    with tf.name_scope(&quot;phase2&quot;):&#xA;        phase2_reconstruction_loss = tf.reduce_mean(tf.square(hidden3 - hidden1))&#xA;        phase2_reg_loss = regularizer(weights2) + regularizer(weights3)&#xA;        phase2_loss = phase2_reconstruction_loss + phase2_reg_loss&#xA;        train_vars = [weights2, biases2, weights3, biases3]&#xA;        phase2_training_op = optimizer.minimize(phase2_loss, var_list=train_vars) # freeze hidden1&#xA;&#xA;    init = tf.global_variables_initializer()&#xA;    saver = tf.train.Saver()&#xA;&#xA;    training_ops = [phase1_training_op, phase2_training_op]&#xA;    reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]&#xA;    n_epochs = [4, 4]&#xA;    batch_sizes = [150, 150]&#xA;&#xA;    use_cached_results = True&#xA;&#xA;    # Train both phases&#xA;    if not use_cached_results:&#xA;        with tf.Session() as sess:&#xA;            init.run()&#xA;            for phase in range(2):&#xA;                print(&quot;Training phase #{}&quot;.format(phase + 1))&#xA;                for epoch in range(n_epochs[phase]):&#xA;                    n_batches = mnist.train.num_examples // batch_sizes[phase]&#xA;                    for iteration in range(n_batches):&#xA;                        print(&quot;\r{}%&quot;.format(100 * iteration // n_batches), end=&quot;&quot;)&#xA;                        sys.stdout.flush()&#xA;                        X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])&#xA;                        sess.run(training_ops[phase], feed_dict={X: X_batch})&#xA;                    loss_train = reconstruction_losses[phase].eval(feed_dict={X: X_batch})&#xA;                    print(&quot;\r{}&quot;.format(epoch), &quot;Train MSE:&quot;, loss_train)&#xA;                    saver.save(sess, &quot;./my_model_one_at_a_time.ckpt&quot;)&#xA;            loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})&#xA;            print(&quot;Test MSE (uncached method):&quot;, loss_test)&#xA;&#xA;    # Train both phases, but in this case we cache the frozen layer outputs&#xA;    if use_cached_results:&#xA;        with tf.Session() as sess:&#xA;            init.run()&#xA;            for phase in range(2):&#xA;                print(&quot;Training phase #{}&quot;.format(phase + 1))&#xA;                if phase == 1:&#xA;                    hidden1_cache = hidden1.eval(feed_dict={X: mnist.train.images})&#xA;                for epoch in range(n_epochs[phase]):&#xA;                    n_batches = mnist.train.num_examples // batch_sizes[phase]&#xA;                    for iteration in range(n_batches):&#xA;                        print(&quot;\r{}%&quot;.format(100 * iteration // n_batches), end=&quot;&quot;)&#xA;                        sys.stdout.flush()&#xA;                        if phase == 1:&#xA;                # Phase 2 - use the cached output from hidden layer 1&#xA;                            indices = np.random.permutation(mnist.train.num_examples)&#xA;                            hidden1_batch = hidden1_cache[indices[:batch_sizes[phase]]]&#xA;                            feed_dict = {hidden1: hidden1_batch}&#xA;                            sess.run(training_ops[phase], feed_dict=feed_dict)&#xA;                        else:&#xA;                # Phase 1&#xA;                            X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])&#xA;                            feed_dict = {X: X_batch}&#xA;                            sess.run(training_ops[phase], feed_dict=feed_dict)&#xA;                    loss_train = reconstruction_losses[phase].eval(feed_dict=feed_dict)&#xA;                    print(&quot;\r{}&quot;.format(epoch), &quot;Train MSE:&quot;, loss_train)&#xA;                    saver.save(sess, &quot;./my_model_cache_frozen.ckpt&quot;)&#xA;            loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})&#xA;            print(&quot;Test MSE (cached method):&quot;, loss_test)&#xA;&#xA;&#xA;def unsupervised_pretraining():&#xA;    reset_graph()&#xA;&#xA;    # Load the dataset to use&#xA;    mnist = input_data.read_data_sets(&quot;/tmp/data/&quot;)&#xA;&#xA;    n_inputs = 28 * 28&#xA;    n_hidden1 = 300&#xA;    n_hidden2 = 150&#xA;    n_outputs = 10&#xA;&#xA;    learning_rate = 0.01&#xA;    l2_reg = 0.0005&#xA;&#xA;    activation = tf.nn.elu&#xA;    regularizer = tf.contrib.layers.l2_regularizer(l2_reg)&#xA;    initializer = tf.contrib.layers.variance_scaling_initializer()&#xA;&#xA;    X = tf.placeholder(tf.float32, shape=[None, n_inputs])&#xA;    y = tf.placeholder(tf.int32, shape=[None])&#xA;&#xA;    weights1_init = initializer([n_inputs, n_hidden1])&#xA;    weights2_init = initializer([n_hidden1, n_hidden2])&#xA;    weights3_init = initializer([n_hidden2, n_outputs])&#xA;&#xA;    weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=&quot;weights1&quot;)&#xA;    weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=&quot;weights2&quot;)&#xA;    weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=&quot;weights3&quot;)&#xA;&#xA;    biases1 = tf.Variable(tf.zeros(n_hidden1), name=&quot;biases1&quot;)&#xA;    biases2 = tf.Variable(tf.zeros(n_hidden2), name=&quot;biases2&quot;)&#xA;    biases3 = tf.Variable(tf.zeros(n_outputs), name=&quot;biases3&quot;)&#xA;&#xA;    hidden1 = activation(tf.matmul(X, weights1) + biases1)&#xA;    hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)&#xA;    logits = tf.matmul(hidden2, weights3) + biases3&#xA;&#xA;    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)&#xA;    reg_loss = regularizer(weights1) + regularizer(weights2) + regularizer(weights3)&#xA;    loss = cross_entropy + reg_loss&#xA;    optimizer = tf.train.AdamOptimizer(learning_rate)&#xA;    training_op = optimizer.minimize(loss)&#xA;&#xA;    correct = tf.nn.in_top_k(logits, y, 1)&#xA;    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))&#xA;&#xA;    init = tf.global_variables_initializer()&#xA;    pretrain_saver = tf.train.Saver([weights1, weights2, biases1, biases2])&#xA;    saver = tf.train.Saver()&#xA;&#xA;    n_epochs = 4&#xA;    batch_size = 50&#xA;    n_labeled_instances = 2000&#xA;&#xA;    pretraining = True&#xA;&#xA;    # Regular training (without pretraining):&#xA;    if not pretraining:&#xA;        with tf.Session() as sess:&#xA;            init.run()&#xA;            for epoch in range(n_epochs):&#xA;                n_batches = n_labeled_instances // batch_size&#xA;                for iteration in range(n_batches):&#xA;                    #print(&quot;\r{}%&quot;.format(100 * iteration // n_batches), end=&quot;&quot;)&#xA;                    #sys.stdout.flush()&#xA;                    indices = np.random.permutation(n_labeled_instances)[:batch_size]&#xA;                    X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]&#xA;                    sess.run(training_op, feed_dict={X: X_batch, y: y_batch})&#xA;                    accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})&#xA;                    print(&quot;\r{}&quot;.format(epoch), &quot;Train accuracy after each mini-batch:&quot;, accuracy_val)&#xA;                    sys.stdout.flush()&#xA;                accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})&#xA;                print(&quot;\r{}&quot;.format(epoch), &quot;Train accuracy after all batched:&quot;, accuracy_val, end=&quot; &quot;)&#xA;                saver.save(sess, &quot;./my_model_supervised.ckpt&quot;)&#xA;                accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})&#xA;                print(&quot;Test accuracy (without pretraining):&quot;, accuracy_val)&#xA;&#xA;    # Now reuse the first two layers of the autoencoder we pretrained:&#xA;    if pretraining:&#xA;        training_op = optimizer.minimize(loss, var_list=[weights3, biases3])  # Freeze layers 1 and 2 (optional)&#xA;        with tf.Session() as sess:&#xA;            init.run()&#xA;            pretrain_saver.restore(sess, &quot;./my_model_cache_frozen.ckpt&quot;)&#xA;            for epoch in range(n_epochs):&#xA;                n_batches = n_labeled_instances // batch_size&#xA;                for iteration in range(n_batches):&#xA;                    #print(&quot;\r{}%&quot;.format(100 * iteration // n_batches), end=&quot;&quot;)&#xA;            #sys.stdout.flush()&#xA;                    indices = np.random.permutation(n_labeled_instances)[:batch_size]&#xA;                    X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]&#xA;                    sess.run(training_op, feed_dict={X: X_batch, y: y_batch})&#xA;                    accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})&#xA;                    print(&quot;\r{}&quot;.format(epoch), &quot;Train accuracy after each mini-batch:&quot;, accuracy_val)&#xA;                    sys.stdout.flush()&#xA;                accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})&#xA;                print(&quot;\r{}&quot;.format(epoch), &quot;Train accuracy after all batched:&quot;, accuracy_val, end=&quot; &quot;)&#xA;                saver.save(sess, &quot;./my_model_supervised_pretrained.ckpt&quot;)&#xA;                accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})&#xA;                print(&quot;Test accuracy (with pretraining):&quot;, accuracy_val)&#xA;&#xA;&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;    # Seed the random number generator&#xA;    np.random.seed(42)&#xA;    tf.set_random_seed(42)&#xA;&#xA;    # Fit a multi-layer autoencoder and save the weights&#xA;    # - this part is from Aurelien Geron's Ch 15, &quot;Training one Autoencoder at a time in a single graph&quot; example&#xA;    train_stacked_autoencoder()&#xA;&#xA;    # Fit a network, using the weights previously saved for pretraining&#xA;    # - this part is from Aurelien Geron's Ch 15, &quot;Unsupervised pretraining&quot; example&#xA;    unsupervised_pretraining()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47380" LastActivityDate="2018-05-02T09:08:34.067" Title="Pretraining neural net example in Aurelien Geron's book" Tags="|machine-learning|python|neural-network|tensorflow|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31102" PostTypeId="1" CreationDate="2018-05-02T02:58:26.700" Score="0" ViewCount="39" Body="&lt;p&gt;Lots of machine learning tutorials always mention about the identification of hand written numbers or cats or dogs, and there is always a prebuilt dataset for training. &lt;strong&gt;But no one said that how to build that dataset&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we want to build our own model about human poses identification. We know how to train, how to inference, and how to integrate it to our project sources. But the most difficult task we seem is how to obtain &lt;strong&gt;LOTS&lt;/strong&gt; human pose images and mark them with correct tags to train.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would someone kindly show us a way to achieve that? Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="51487" LastEditorUserId="51487" LastEditDate="2018-05-03T11:47:35.127" LastActivityDate="2018-05-03T11:47:35.127" Title="How can I obtain datasets about human poses?" Tags="|dataset|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31106" PostTypeId="2" ParentId="31100" CreationDate="2018-05-02T09:08:34.067" Score="1" Body="&lt;p&gt;&lt;em&gt;[NOTE: I have not myself worked through Aurélien Geron's tutorials, but I have read the book]&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;On an intuitive level, I can persuade myself that the training would actually be slower for a pretrained model. In other words, it could make sense that the &lt;strong&gt;rate&lt;/strong&gt; at which a error decreases (or accuracy increases) might be lower. The fact that training accuracy is lower is (for me at least) a little more complex and, perhaps, case specific.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Rate of learning&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However the pretraining seems to make training slower.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using a pretrained model, we have essentially taken a set of weights, which are already (at least partially) optimised for one problem. They are geared towards solving that problem based on the dataset they received, which means they expect the input to correspond to a certain distribution. You have frozen the first two layers with this line:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;if pretraining:&#xA;    training_op = optimizer.minimize(loss, var_list=[weights3, biases3])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Freezing two layers (in your case, out of three), intuitively kind of restricts the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a somewhat contrived analogy that I might use to explain such cases to myself. Imagine we had a clown who could juggle with three balls, but now we want them to learn to use a fourth ball. At the same time, we ask an amateur to learn how to juggle, also with four balls. Before measuring their rate of learning, we decide to tie one of the clown's hands behind their back. So the clown already knows some tricks, but is also constrained in some way during the learning process. In my mind, the amateur would most likely learn a lot faster (relatively), as there is more to learn -  but also because they have more freedom to explore the parameter space i.e. they can move more freely using both arms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the setting of optimisation, one might imagine that position of the pretrained model on a loss curve is already in a place where gradients are very small in certain dimensions (don't forget, we have a high-dimensional search space). This ends up meaning that it cannot as quickly make changes to the output of the weights whilst backpropagating errors, as the weight updates are multiples of these potentially small optimised weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...&lt;em&gt;Ok - might sounds plausible, but this only addresses the problem of slow learning - what about the fact that the actual training accuracy is lower that that of the model with random initialisation??&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Intial training accuracy&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I expected the pretrained network would start with a lower error (compared to the network not using pretraining)...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here I tend to agree with you. In the optimal case, we could take a pretrained model, use the initial layers as they are and just fine-tune the final layers. There are, however, some cases in which this might not work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Looking into related literature, there is a possible explanation from the abstract of the paper: &lt;a href=&quot;https://arxiv.org/abs/1411.1792&quot; rel=&quot;nofollow noreferrer&quot;&gt;How transferable are features in deep neural networks? (Yosinski et al.)&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I find the second reason to be particularly interesting and relevant to your setup. This is because you actually only have three layers. You are therefore not allowing must freedom to fine-tune, and the final layer was likely very dependent on its relationship to the preceding layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you might expect to see as a result of using a pretrained model, is rather that the final model exhibits better generalisation. This may indeed come at the cost of a lower test accuracy on the hold-out set of the specific dataset you train on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are &lt;a href=&quot;http://cs231n.github.io/transfer-learning/#tf&quot; rel=&quot;nofollow noreferrer&quot;&gt;another thoughts&lt;/a&gt;, summarised well by the amazing (and free) &lt;a href=&quot;http://cs231n.stanford.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford CS231n&lt;/a&gt; course:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;em&gt;Learning rates&lt;/em&gt;. It’s common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In your code, the learning rate seems to be fixed for all learning phases at &lt;code&gt;0.01&lt;/code&gt;. This is something you could experiment with; making it smaller for the pretrained layers, or just starting with a lower learning rate globally.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/06/transfer-learning-the-art-of-fine-tuning-a-pre-trained-model/&quot; rel=&quot;nofollow noreferrer&quot;&gt;comprehensive introduction to tranfer learning&lt;/a&gt; that might give you some more ideas about why/where you might make some different modelling decisions.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-02T09:08:34.067" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31107" PostTypeId="1" AcceptedAnswerId="31108" CreationDate="2018-05-02T09:09:06.717" Score="3" ViewCount="381" Body="&lt;p&gt;I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, but I just can't understand what is the importance of finding the roots or using second derivatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I know the idea of Newton Raphson, I am wondering why do we need to use this method and find the zero's or roots of function? What does it want to tell us for logistic regression for instance?&lt;/p&gt;&#xA;" OwnerUserId="51495" LastEditorUserId="50727" LastEditDate="2018-05-25T07:02:06.790" LastActivityDate="2018-05-25T07:02:06.790" Title="Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)" Tags="|machine-learning|logistic-regression|optimization|gradient-descent|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31108" PostTypeId="2" ParentId="31107" CreationDate="2018-05-02T09:24:00.960" Score="6" Body="&lt;p&gt;&lt;a href=&quot;http://thelaziestprogrammer.com/sharrington/math-of-machine-learning/solving-logreg-newtons-method&quot; rel=&quot;noreferrer&quot;&gt;This blogpost&lt;/a&gt; gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are either in a maximum, a minimum, or a saddle point (the three of them are critical points). When using the cross-entropy loss function in logistic regression, it can be proved that this loss function has only one critical point, and this critical point corresponds to the global minimum. For this reason, finding a root of the gradient is equivalent to finding the only critical point, therefore the global minimum of the cross-entropy loss function.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-02T09:24:00.960" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="31109" PostTypeId="1" AcceptedAnswerId="31126" CreationDate="2018-05-02T09:43:47.713" Score="8" ViewCount="16533" Body="&lt;p&gt;Using &lt;code&gt;Embedding&lt;/code&gt; layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension vector.  I think that the embedded vector need to have a minimum length to be able to map a given vocabulary.&lt;/p&gt;&#xA;" OwnerUserId="9461" LastEditorUserId="9461" LastEditDate="2018-05-02T18:43:44.173" LastActivityDate="2019-03-29T07:36:22.527" Title="Ratio between embedded vector dimensions and vocabulary size" Tags="|nlp|word-embeddings|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31110" PostTypeId="1" CreationDate="2018-05-02T10:22:21.237" Score="2" ViewCount="149" Body="&lt;p&gt;I am using SGD matrix factorisation (python) using the movielens dataset to make recommendations. I have a website which allows users to give feedback which is positive or negative to whether an item is a good recommendation for a particular movie. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if I could use this feedback in my matrix factorisation. I wasn't 100% sure how I would proceed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So for example I would have a vector like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  m1 m2 m3&#xA;m1 0 0  0&#xA;m2 5 0 -4&#xA;m3 0 0  0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where m2-m1 is a score of 5 so with an example case where noone rated it negative, 5 people think its good. With the other for m2-m3 being -4 so a poor recommendation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be greatly appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit: response from answer&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using a sparse matrix for user item ratings and am using bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been trying to add additional input sources and am using the following to create a attribute matrix containing a genre representation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for genre in item_genres:&#xA;       genres[genre] = 1&#xA;       self.attribute_item_matrix[i] = sum(list(genres.values()))&#xA;&#xA;&#xA; pred = self.global_mean + self.bias_user[u] + self.bias_item[i] +np.dot(self.P[u, :],(self.Q[:, i]+self.attribute_item_matrix[i]))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, I don't think this is correct as it eventually fails and causes a nan error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been following this paper &lt;a href=&quot;https://datajobs.com/data-science-repo/Recommender-Systems-[Netflix].pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Matrix Factorisation Techniques for recommender systems&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="51499" LastEditorUserId="51499" LastEditDate="2018-05-04T14:41:46.367" LastActivityDate="2024-02-28T10:03:55.553" Title="Matrix Factorisation Improvement" Tags="|python|recommender-system|matrix-factorisation|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31111" PostTypeId="2" ParentId="31089" CreationDate="2018-05-02T11:04:46.667" Score="1" Body="&lt;p&gt;In Python there is &lt;a href=&quot;https://docs.python.org/3/howto/ipaddress.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the ipaddress module&lt;/a&gt; which can convert an IP address to an integer, then simply subtract to find the difference.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from ipaddress import ip_address&#xA;int(ip_address(‘10.0.0.1’)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="32402" LastEditorUserId="32402" LastEditDate="2018-05-02T11:12:07.577" LastActivityDate="2018-05-02T11:12:07.577" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31112" PostTypeId="1" AcceptedAnswerId="31116" CreationDate="2018-05-02T11:06:21.790" Score="1" ViewCount="2384" Body="&lt;p&gt;I am using Boruta feature selection (with Random forest) to decide the important features in the below data set.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;   Gender Married Dependents     Education Self_Employed  ApplicantIncome  \&#xA;0   Male      No          0      Graduate            No             5849   &#xA;1   Male     Yes          1      Graduate            No             4583   &#xA;2   Male     Yes          0      Graduate           Yes             3000   &#xA;3   Male     Yes          0  Not Graduate            No             2583   &#xA;4   Male      No          0      Graduate            No             6000   &#xA;&#xA; CoapplicantIncome  LoanAmount  Loan_Amount_Term  Credit_History  \&#xA;0                0.0         NaN             360.0             1.0   &#xA;1             1508.0       128.0             360.0             1.0   &#xA;2                0.0        66.0             360.0             1.0   &#xA;3             2358.0       120.0             360.0             1.0   &#xA;4                0.0       141.0             360.0             1.0   &#xA;&#xA;Property_Area Loan_Status  &#xA;0         Urban           Y  &#xA;1         Rural           N  &#xA;2         Urban           Y  &#xA;3         Urban           Y  &#xA;4         Urban           Y  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please help me in clarifying the below doubts &#xA;1) whether I need to convert the all categorical variable into numeric variable (using one hot encoding) before applying Boruta?&#xA;2) Whether the NA values would be taken care by Boruta or do we need to remove NA values before feeding into Boruta ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of Regression problem, whether the Boruta approach (Random forest classifier -&gt; Boruta) remains the same as in classification problem ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you. &lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="50146" LastEditDate="2018-05-02T12:36:19.940" LastActivityDate="2018-05-02T12:36:19.940" Title="Boruta Feature Selection package" Tags="|machine-learning|feature-selection|preprocessing|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31113" PostTypeId="1" AcceptedAnswerId="31115" CreationDate="2018-05-02T11:25:11.463" Score="10" ViewCount="3501" Body="&lt;p&gt;I'm training a CNN for a 3-class image classification problem. My training loss decreased smoothly, which is the expected behaviour. However, my validation loss shows a lot of fluctuation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/XnCS8.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XnCS8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional info:&lt;/strong&gt; I'm fine-tuning the last layer of a Resnet-18 that was pre-trained on ImageNet data in PyTorch. I must note that I'm using a weighted loss function for the training phase since my data is highly unbalanced. To plot the losses, however, I use the unweighted loss as to be able to compare the validation and training loss. I would use the unweighted loss, were it not that the distributions of the training dataset and validation dataset are somewhat different (they are both highly unbalanced however).&lt;/p&gt;&#xA;" OwnerUserId="51141" LastActivityDate="2018-05-02T13:26:09.113" Title="Validation showing huge fluctuations. What could be the cause?" Tags="|image-classification|cnn|loss-function|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31115" PostTypeId="2" ParentId="31113" CreationDate="2018-05-02T12:11:50.090" Score="5" Body="&lt;p&gt;From simple inspection of your plot, I could make a few conclusions and list things to try. (This is without knowing any more about your setup: training parameters and model hyperparameters).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It looks like the loss is decreasing (put a line of best fit through the validation loss). It also looks like you might be able to train for longer to improve results, as the curve is still headed downwards.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;First I will try answer your title question:&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;what is the cause of the fluctation in the validation loss?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I can think of three possibilities:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Regularisation&lt;/strong&gt; - to help smoothing the learning process and make the model weights more robust. Adding/increasing your regularisation will prevent large updates to weights being introduced.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Batch size&lt;/strong&gt; - is it relatively small (e.g. &amp;lt; 20?). This would mean that the measured mean error at the end of the network is computed using only a few samples. With a batch size of, say &lt;code&gt;8&lt;/code&gt;, then getting &lt;code&gt;4/8&lt;/code&gt; correct and compared to getting &lt;code&gt;6/8&lt;/code&gt; correct has a large &lt;em&gt;relative&lt;/em&gt; difference when looking at the loss. Taking the mean of the errors with such small batches will lead to a not-so-smooth loss curve. If you have enough GPU memory/RAM, try increasing batch size.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Learning Rate&lt;/strong&gt; - might be too large. This is similar to the first point regarding regularisation. To make smoother improvements, you might need to slow down the pace of learning as you approach a loss-minimum. You can make this perhaps run on a schedule, whereby is is reduce by some factor (e.g. multiply it by 0.5) every time the validation loss has not improved after, say &lt;code&gt;6&lt;/code&gt; epochs. This will prevent you from taking big steps and then maybe overshooting a minumum and just bouncing around it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Specific to your task, I would also suggest trying to perhaps &lt;strong&gt;&lt;em&gt;unfreeze&lt;/em&gt; another layer&lt;/strong&gt;, to increase the scope of your fine-tuning. This will give the Resnet-18 a little more freedom to learn, based on your data.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Regarding your last question:&lt;/h3&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Should you be worried? In short, no. A validation loss curve like yours can be perfectly fine and deliver reasonable results; however, I would try some of the steps I mentioned above before settling for it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should you just pick the best performing model? If you mean taking the model at its point with best validation loss (validation accuracy), then I would say to be more careful. On your plot above, this might equate to around epoch 30, but I would personally take a point that has trained a little more, where the curve gets a little less volatile. Again, after having tried some of the steps outlined above.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-02T12:11:50.090" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31116" PostTypeId="2" ParentId="31112" CreationDate="2018-05-02T12:11:54.593" Score="3" Body="&lt;p&gt;In &lt;code&gt;R&lt;/code&gt;, &lt;code&gt;Boruta&lt;/code&gt; relies on the &lt;code&gt;ranger&lt;/code&gt; implementation of random forest. So:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Converting input variables from categorical to numeric is not necessary.&lt;/li&gt;&#xA;&lt;li&gt;You will need to address &lt;code&gt;NA&lt;/code&gt; values prior to running the algorithm.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Be aware that &lt;code&gt;Boruta&lt;/code&gt; can be very slow!&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-05-02T12:11:54.593" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31117" PostTypeId="2" ParentId="31098" CreationDate="2018-05-02T12:15:50.113" Score="0" Body="&lt;p&gt;As @Emre noted in the comments, you will want to use the &lt;a href=&quot;https://en.wikipedia.org/wiki/Softmax_function&quot; rel=&quot;nofollow noreferrer&quot;&gt;softmax function&lt;/a&gt;. After acquiring a set of scores per film, the function will squash the scores into the range [0,1] and the scores will add up to 1.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Here's a &lt;strong&gt;possible procedure:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Generate a feature set per image&lt;/li&gt;&#xA;&lt;li&gt;Train a model on your feature set to output a score for a movie.  The score is a number in an arbitrary domain.&lt;/li&gt;&#xA;&lt;li&gt;Repeat step 2 for all of your candidate movies to create a score vector.&lt;/li&gt;&#xA;&lt;li&gt;Feed the score vector into the softmax function to squash the scores into the domain [0,1] and make them add to 1&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For reference, the softmax function is defined:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \sigma(z_i) = \frac{e^{z_i}}{\displaystyle\sum^N_{n=1}e^{z_n}} $&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;And &lt;strong&gt;an example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our feature set will include the following: critic/audience ratings, revenue, cost, total tickets sold, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's say you have historical data of the Oscars that includes the feature sets and a score that we will use for the ground truth metric.  You define the score.  For instance, if a movie was not even nominated for the Oscar award, it might have a score of zero, whereas an Oscar winning movie might have a score of 1.  A movie that was nominated but did not receive many votes might have a score of 0.50.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You train a model on your historical data so that, given a movie's feature set, it will output a score, similar to your training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you are considering three movies for prediction: Avengers Infinity War, Deadpool 2, and Venom.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You acquire their feature sets (the same categories that you used for training: critic/audience ratings, revenue, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you pass each feature set through your model and acquire a vector of scores:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{|l|c|}&#xA;\hline&#xA;Movie &amp;amp; Score\\&#xA;\hline&#xA;Avengers\ Infinity\ War &amp;amp; 0.98\\&#xA;Deadpool\ 2 &amp;amp; 0.82\\&#xA;Venom &amp;amp; 0.24\\&#xA;\hline&#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can interpret the score results as a probability by using the softmax function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The denominator of the function is given:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \displaystyle\sum^N_{n=1} e^{z_n} = e^{0.98} + e^{0.82} + e^{0.24} \ \ \ where: z=\{0.98,0.82,0.24\} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We calculate the softmax of a given score $ x $ as such:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ \Large\frac{e^x}{e^{0.98} + e^{0.82} + e^{0.24}} $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore the softmax scores are:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{array}{|l|c|c|}&#xA;\hline&#xA;Movie &amp;amp; Score &amp;amp; Softmax\\&#xA;\hline&#xA;Avengers\ Infinity\ War &amp;amp; 0.98 &amp;amp; 0.42932\\&#xA;Deadpool\ 2 &amp;amp; 0.82 &amp;amp; 0.36584\\&#xA;Venom &amp;amp; 0.24 &amp;amp; 0.20484\\&#xA;\hline&#xA;\end{array}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and we can see that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ 0.42932 + 0.36584 + 0.20484 = 1 $&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-05-02T17:06:37.743" LastActivityDate="2018-05-02T17:06:37.743" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31118" PostTypeId="1" AcceptedAnswerId="31124" CreationDate="2018-05-02T13:06:59.440" Score="0" ViewCount="230" Body="&lt;p&gt;I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 observations that I use for a final validation. All 10,200 observations were obtained in similar fashion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I carried out a grid search with 5-fold cross-validation in order to obtain a suitable set of hyper parameters. The results for my test set are good and very stable. However, there is a big drop off in performance when use my final validation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Results for the test set &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-&amp;gt;  Precision: 0.836 / Recall: 0.629 / Accuracy 0.874&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results for the final validation set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;-&amp;gt;  Precision: 0.149 / Recall: 0.725 / Accuracy 0.484&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;At first I thought this could be the caused by &lt;a href=&quot;https://machinelearningmastery.com/data-leakage-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;data leakage&lt;/a&gt;, but even after removing &quot;suspicious&quot; features, there is still a big drop off when comparing the test results with the final validation results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Surely I'm doing something wrong, but I'm at a loss as to what exactly. Here are the relevant lines of code (nothing fancy):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; X = pd.read_csv('train_test.csv')&#xA;&amp;gt; y = X.pop('Target')&#xA;&amp;gt; X_final = pd.read_csv('final_validation.csv')&#xA;&amp;gt; y_final = X_final.pop('Target')&#xA;&#xA;&amp;gt; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)&#xA;&#xA;&amp;gt; gb = GradientBoostingClassifier(n_estimators=300, max_depth=5, learning_rate=0.2)&#xA;&amp;gt; gb_model = gb.fit(X_train, y_train)&#xA;&#xA;&amp;gt; # test set&#xA;&amp;gt; y_pred = gb_model.predict(X_test)&#xA;&amp;gt; precision, recall, fscore, support = score(y_test, y_pred, average='binary')&#xA;&#xA;&amp;gt; # final validation set&#xA;&amp;gt; y_hat = gb_model.predict(X_final)&#xA;&amp;gt; precision, recall, fscore, support = score(y_final, y_hat, average='binary')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="51504" LastActivityDate="2018-05-02T16:03:36.087" Title="Why is there a big drop off in performance in my GBM?" Tags="|gbm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31119" PostTypeId="2" ParentId="31113" CreationDate="2018-05-02T13:26:09.113" Score="4" Body="&lt;p&gt;I would also recommend that you use methods for data augmentation and oversampling to compensate the unbalance of the classes. This standford paper explains some of the ideas that you can implement &lt;a href=&quot;http://cs231n.stanford.edu/reports/2017/pdfs/300.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Effectiveness of Data Augmentation in Image Classification using Deep&#xA;Learning&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope this helps! &lt;/p&gt;&#xA;" OwnerUserId="43459" LastActivityDate="2018-05-02T13:26:09.113" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31120" PostTypeId="1" CreationDate="2018-05-02T13:55:06.507" Score="0" ViewCount="1805" Body="&lt;p&gt;how can I calculate the y_score for a &lt;code&gt;roc_auc_score&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a classifier, for classes {0,1}, say &lt;code&gt;RandomForestClassifier&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, when I apply it to my test data, I will get a list of {0,1}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;roc_auc_score&lt;/code&gt;&lt;/a&gt; expects &lt;code&gt;y_true&lt;/code&gt; and &lt;code&gt;y_score&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As dummy as it might look, after fitting the model, I was making the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def find_prob(df):&#xA;    def prob(token):&#xA;        # the mean is the probability of label {1}&#xA;        return df[df.token == token].predicted.mean()&#xA;    df['predicted_score'] = df.token.map(prob)&#xA;    return df&#xA;&#xA;X_val['predicted'] = model.predict(X_val).astype(int)&#xA;X_val = find_prob(tokenize(X_val))&#xA;&#xA;roc_auc_score(y_val, X_val.predicted_score)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where &lt;code&gt;tokenize&lt;/code&gt; is only a function which concatenate all the values in a row to a &quot;unique&quot; token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this the correct approach? Thank you very much.&lt;/p&gt;&#xA;" OwnerUserId="51459" LastEditorUserId="51459" LastEditDate="2018-05-02T14:12:46.623" LastActivityDate="2019-03-18T11:15:50.287" Title="How to calculate y_score for ROC AUC?" Tags="|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31121" PostTypeId="1" CreationDate="2018-05-02T14:30:12.520" Score="5" ViewCount="527" Body="&lt;p&gt;Say you had a set of users, tens of thousands.  You have time series of each of their behaviors using this app.  How might you use these time series to predict future behavior of new users?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The intuitive solution is to feature engineer behavior of users and use that to train a model, such as average weekly minutes in app and things like that.  My issue with this approach is that you lose a great deal of information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I'm wondering is if there's a better technique for aggregating a large set of time series data to build a predictive model.  Perhaps an LSTM would work but that seems like it wouldn't capture the nuances of the dataset and I don't believe they're typically used to aggregate predictions on a variety of individual samples.&lt;/p&gt;&#xA;" OwnerUserId="51506" LastActivityDate="2020-10-26T09:41:53.237" Title="Predicting with multiple time series" Tags="|predictive-modeling|time-series|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31123" PostTypeId="1" CreationDate="2018-05-02T15:52:05.553" Score="1" ViewCount="2808" Body="&lt;p&gt;I'm currently working on a bigger project.&#xA;&lt;strong&gt;The goal is to automatically find split points in time series which splits the series into elementary patterns.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a lot of training data in form of time series with &lt;strong&gt;different lengths&lt;/strong&gt; and split points manually recorded on useful positions.&#xA;Basically, I have the &lt;code&gt;x&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;z&lt;/code&gt; positions and the distances between those points over a centered window of the lengths 2, 25 and 50 (&lt;code&gt;dist2&lt;/code&gt;, &lt;code&gt;dist25&lt;/code&gt;, &lt;code&gt;dist50&lt;/code&gt;). I also divided the room into a 3D grid and figured out in which cell the position lies (&lt;code&gt;cell_x&lt;/code&gt;, &lt;code&gt;cell_y&lt;/code&gt;, &lt;code&gt;cell_z&lt;/code&gt;). I think this could be useful because elementary operations will more likely be in one or two cells. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ycXYX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ycXYX.png&quot; alt=&quot;Fig1&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/igrRX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/igrRX.png&quot; alt=&quot;Fig2&quot;&gt;&lt;/a&gt;&#xA;cell_x, cell_y, cell_z based on size 50. The vertical red lines are the split points of the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Now I want to know at every point in the time series if it is a split point or not.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my opinion, these are enough features to start with but I think my Keras model isn't correct because the result is always and only 1 at the 0's time series index. It's basically, a binary classification problem based on past and future values. This was the reason I've tried to solve it with an &lt;strong&gt;LSTM&lt;/strong&gt;. It looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(LSTM(20, input_shape = (None, input_dim), return_sequences = True))&#xA;model.add(Dropout(0.5))&#xA;model.add(LSTM(20))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(1, activation = 'sigmoid'))&#xA;&#xA;model.compile(loss='binary_crossentropy',&#xA;            optimizer='adam',&#xA;            metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Because I'm at the very beginning of the whole Machine Learning topic and the mass of materials is a bit overwhelming I'm here to ask for some help. I know that there are a lot of other things (bad training data, wrong features, wrong parameters, ...) which can cause to this results but I want to know how I can build a Keras model for this problem correctly. I already found a lot of Keras examples but I'm not sure if they fit the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example of data: Features and label of one record: &lt;a href=&quot;http://gist.github.com/chryb/2b3312b0ab640a1f28c8cb2d3dbbd86e&quot; rel=&quot;nofollow noreferrer&quot;&gt;gist.github.com/chryb/2b3312b0ab640a1f28c8cb2d3dbbd86e&lt;/a&gt; | The column is_keyframe represents the labels for each index. Based on a Stackoverflow question I transformed the data with the following to the right LSTM input format: &lt;a href=&quot;http://gist.github.com/chryb/3de34ba44488012bf79ff4c51a2a43ae&quot; rel=&quot;nofollow noreferrer&quot;&gt;gist.github.com/chryb/3de34ba44488012bf79ff4c51a2a43ae&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="32543" LastActivityDate="2018-05-02T15:52:05.553" Title="Keras LSTM model for binary classification with sequences" Tags="|classification|keras|tensorflow|time-series|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31124" PostTypeId="2" ParentId="31118" CreationDate="2018-05-02T16:03:36.087" Score="0" Body="&lt;p&gt;I would first suggest trying to plot the results during training. How do your metrics (or at least the loss) vary over the training process for training and cross-validation datasets?&#xA;The loss at each iteration is appended to your GBC object &lt;code&gt;gb_model&lt;/code&gt; in the &lt;code&gt;train_score_&lt;/code&gt; attribute.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Normally, when there is such a big gap between training and test data, it indicates that you are overfitting to your training data, and that the model does not generalise well to unseen data. You could think about doing  shuffling your data, in order to balance the training/validation/test datasets - [if you are looking at a time-series problem, you should be careful as to how you do this].&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-02T16:03:36.087" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31125" PostTypeId="1" CreationDate="2018-05-02T16:38:18.423" Score="0" ViewCount="99" Body="&lt;p&gt;For example, financial data has in some cases strong positive and negative correlations between different stocks (feature space) &lt;strong&gt;and&lt;/strong&gt; also strong positive and negative correlations between specific dates (sample space), if surprising positive or negative news was released on those dates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are other sources of continuous data, such as audio, visual, or data obtained by some measurement instrument or process, that display the above characteristics? See also &quot;transposable data&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Importantly, it seems that in the sample space often negative correlations are non-sensical or non-causal, which limits the types of data sources with positive and negative correlations in both sample and feature space.  &lt;/p&gt;&#xA;" OwnerUserId="51511" LastEditorUserId="51511" LastEditDate="2018-05-03T09:32:10.597" LastActivityDate="2018-05-05T07:27:57.750" Title="What are some data sources with strong positive and negative correlations in feature and sample space?" Tags="|data|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31126" PostTypeId="2" ParentId="31109" CreationDate="2018-05-02T17:01:55.517" Score="9" Body="&lt;p&gt;The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding vectors don't add enough information and smaller ones don't represent the semantics well enough. What matters more is the network architecture, the algorithm(s) and the dataset size. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A simple way to understand this concept is that a bidirectional LSTM model with 50 neurons (nodes) followed by a fully connected layer of 70 neurons will outperform a simple MLP of 1000 neurons (nodes) connected to a embedding layer simply due to its architecture. Adding dropout will improve performance as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, even if the vocabulary is just 300 words, using pre-trained embeddings will probably yield better results than training the embeddings directly on the dataset. The same applies to data size, a dataset with more samples will make a better classifier than a dataset with just a couple thousand samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, it is preferable to try many architectures and cross-validate them (and/or ensemble them depending if you have a large enough dataset) with the smallest number of neurons possible and then start building up in size, depending on what computational resources you have and the speed of development you need. Large models slow down development speed whereas small models speed it up. This goes whether your vocabulary is the size of common crawl or just 300. As usual, try feature engineering (sentence length, special characters, etc.) and increase the dataset size as doing so often helps in whatever task you're trying to predict. &lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2018-05-02T17:01:55.517" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31127" PostTypeId="2" ParentId="31059" CreationDate="2018-05-02T17:25:11.807" Score="2" Body="&lt;p&gt;I'd use DataFrame.join() in this case:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;data = data.join(pd.SparseDataFrame(Titles_ohe, index=data.index, columns=Titles))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-05-02T17:25:11.807" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31128" PostTypeId="2" ParentId="27048" CreationDate="2018-05-02T18:02:19.717" Score="5" Body="&lt;p&gt;You can add a second Input layer to your architecture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Look at this link &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models&quot; rel=&quot;noreferrer&quot;&gt;Keras Guide Multiple Inputs&lt;/a&gt; for a more detailed explanation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output of the embedding layer can be combined with the secondary input and passed to the next layer.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;auxiliary_input = Input(shape=(5,), name='aux_input')&#xA;x = keras.layers.concatenate([lstm_out, auxiliary_input])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8206" LastActivityDate="2018-05-02T18:02:19.717" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31129" PostTypeId="1" AcceptedAnswerId="31358" CreationDate="2018-05-02T18:04:38.293" Score="19" ViewCount="31552" Body="&lt;p&gt;How do you add more importance to some samples than others (sample weights) in Keras? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not looking for &lt;code&gt;class_weight&lt;/code&gt;which is a fix for unbalanced datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I currently have is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;trainingWeights&lt;/code&gt; which is the desired importance I want to give to each sample.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs = 30&#xA;batchSize = 512&#xA;&#xA;# Fit model with selected data&#xA;model.fit(trainingMatrix, trainingTargets,&#xA;          batch_size=batchSize, epochs=epochs, &#xA;          sample_weight=trainingWeights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However the training error is much lower than before, and according to &lt;a href=&quot;https://keras.io/models/sequential/&quot; rel=&quot;noreferrer&quot;&gt;Keras' documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sample_weight: Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PD. This is a similar question &lt;a href=&quot;https://datascience.stackexchange.com/questions/9488/xgboost-give-more-importance-to-recent-samples/9493#9493&quot;&gt;xgboost: give more importance to recent samples&lt;/a&gt; but I would like an applicable answer to Keras.&lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2018-05-11T17:59:28.863" Title="Sample Importance (Training Weights) in Keras" Tags="|neural-network|keras|weighted-data|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31130" PostTypeId="2" ParentId="15056" CreationDate="2018-05-02T18:38:29.723" Score="2" Body="&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;&#xA;sess = tf.InteractiveSession()&#xA;&#xA;my_list = tf.Variable(initial_value=[1,2,3,4,5])&#xA;&#xA;init = tf.global_variables_initializer()&#xA;&#xA;sess.run(init)&#xA;&#xA;sess.run(my_list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result: array([1, 2, 3, 4, 5])&lt;/p&gt;&#xA;" OwnerUserId="51516" LastEditorUserId="51516" LastEditDate="2018-05-02T18:45:49.943" LastActivityDate="2018-05-02T18:45:49.943" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31131" PostTypeId="1" CreationDate="2018-05-02T18:53:00.760" Score="1" ViewCount="45" Body="&lt;p&gt;I would appreciate your input on which predictive ML model(s) could fit our dataset the best.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The primary features of the dataset are x,y (continous) and c (factor with around L=500 unordered levels). The secondary features (~20) are both continous and factors.&#xA;We know that the lowest-order correlation is x~y.&#xA;We also know that modeling x~y for each level in c improves the fit significantly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We could of course create L independent linear models, but would like to explore a single model where these sub-linear relationships are partially correlated. The reason being that some levels of c have relatively low statistics and would benefit from the global x~y relationship. Also, we would like a single model when extending to secondary features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions are welcome. (In case of specific libraries, R is preferred.)&lt;/p&gt;&#xA;" OwnerUserId="39907" LastActivityDate="2018-05-04T12:18:22.487" Title="Model selection for a &quot;set of linear models&quot;?" Tags="|machine-learning|r|algorithms|model-selection|machine-learning-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31134" PostTypeId="1" AcceptedAnswerId="31311" CreationDate="2018-05-02T20:49:21.877" Score="4" ViewCount="3981" Body="&lt;p&gt;Using Python's sklearn module,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; from sklearn.metrics import classification_report&#xA; y1_predict = [0, 1, 1, 0]&#xA; y1_dev = [0, 1, 1, 0]&#xA; report_1 = classification_report(y1_dev, y1_predict)&#xA; y2_predict = [1, 0, 1, 0]&#xA; y2_dev = [1, 1, 0, 0]&#xA; report_2 = classification_report(y2_dev, y2_predict)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Is there a way to combine (maybe just an average) &lt;code&gt;report_1&lt;/code&gt; and &lt;code&gt;report_2&lt;/code&gt; I'm looking for an implementation like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt; report_average = average(report_1,report_2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Or does this have to be done manually? I was hoping that printing the &lt;code&gt;report_average&lt;/code&gt; would have average values between the two reports.&lt;/p&gt;&#xA;" OwnerUserId="49162" LastEditorUserId="43000" LastEditDate="2022-09-28T13:51:47.617" LastActivityDate="2022-11-10T13:36:42.497" Title="Python sklearn - average classification reports" Tags="|python|scikit-learn|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31139" PostTypeId="2" ParentId="31110" CreationDate="2018-05-02T22:26:55.570" Score="0" Body="&lt;p&gt;In my view, you should better go for &lt;code&gt;user&lt;/code&gt;-&lt;code&gt;movie&lt;/code&gt; matrix, to create a &lt;code&gt;sparse matrix&lt;/code&gt; that can be more meaningful. By doing that, if you already know &lt;code&gt;polarity&lt;/code&gt; of choices i.e. &lt;code&gt;positive&lt;/code&gt;, &lt;code&gt;negative&lt;/code&gt; or &lt;code&gt;neutral&lt;/code&gt;, you may also have some &lt;code&gt;intensity&lt;/code&gt; score of &lt;code&gt;like&lt;/code&gt; or &lt;code&gt;dislike&lt;/code&gt; like in case of &lt;code&gt;rating&lt;/code&gt;. Now, if you want to just classify them as &lt;code&gt;likes&lt;/code&gt;, &lt;code&gt;neutral&lt;/code&gt;, and &lt;code&gt;dislike&lt;/code&gt;, you may assign &lt;code&gt;1&lt;/code&gt;,&lt;code&gt;0&lt;/code&gt; and &lt;code&gt;-1&lt;/code&gt; respectively. This method can have more meaning when you have many movies and many users. Further this can help you establish &lt;code&gt;user-user&lt;/code&gt;, &lt;code&gt;movie-user&lt;/code&gt; and &lt;code&gt;movie-movie&lt;/code&gt; relationship as well. These relationship can potentially help you understand and explore more about &lt;code&gt;recommendation engines&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     m1 m2 m3 m4&#xA;u1 -1   0  -1  1&#xA;u2  1   1  -1  0&#xA;u3  0   0   1  0&#xA;u4  -1  1   1   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If you want to add &lt;code&gt;rating&lt;/code&gt; instead of just &lt;code&gt;polarity&lt;/code&gt;, just add numbers scaled &lt;code&gt;0-1&lt;/code&gt;. I personally worked on them to write &lt;code&gt;hybrid recommendation engines&lt;/code&gt; which have both &lt;code&gt;collaborative filtering&lt;/code&gt; and &lt;code&gt;context based filtering&lt;/code&gt; models embedded. I would suggest you to work more on relationships and try to explore more about next steps like possibility of &lt;code&gt;like&lt;/code&gt; or &lt;code&gt;dislike&lt;/code&gt; bias, association of movie info/ actors-directors influence and similarity in movies etc. I hope this explanation helps.&lt;/p&gt;&#xA;" OwnerUserId="42648" LastEditorUserId="42648" LastEditDate="2018-05-03T02:46:37.627" LastActivityDate="2018-05-03T02:46:37.627" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31140" PostTypeId="1" CreationDate="2018-05-02T22:30:35.267" Score="8" ViewCount="3287" Body="&lt;p&gt;&lt;strong&gt;What I am trying to do:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to classify some images using local and global features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I have done so far:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have extracted sift descriptors for each image and I am using this as my input for k-means to create my vocabulary from all of the features of every image. From here I create a histogram from my clusters for each image by passing the image sift features to the predict method in k-means giving me the labels of the clusters. From here I create the histogram by counting the number of labels for each bin. Now I have a nxm matrix where n is the number of images, and m being the number of clusters (features/words) of each image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will feed this matrix to a classifier to get my image classifications.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Steps in a nutshell:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Extract sift feature descriptors giving us a nx128 matrix for each image&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Stack all feature descriptors into one large list&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Fit all of these features into the kmeans algorithm setting k=100&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For every image use its sift features to predict the labels of the clusters using the same trained kmeans model&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a histogram from the clusters using k as the number of bins, adding 1 to the bin for each label in the model. (if an image has 10 features from sift, it will give us 10 labels, these 10 labels will be in the range of k, so for each label we will add it to the corresponding bin for our histogram).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We now have a nxk matrix, where n is the number of images, and k being the number of clusters.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;We now feed in the histograms to a classifier and ask it to predict on the testing data.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I correctly performing bag of visual words?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def extract_features(df):&#xA;    IF = imageFeatures()&#xA;    global_features = []&#xA;    sift_features = []&#xA;    labels = []&#xA;    for i, (index, sample) in enumerate(df.iterrows()):&#xA;        image = cv2.imread(sample[&quot;location&quot;])&#xA;        image = cv2.resize(image, shape)&#xA;        hist = IF.fd_histogram(image)&#xA;        haralick = IF.fd_haralick(image)&#xA;        hu = IF.fd_hu_moments(image)&#xA;        lbp = IF.LocalBinaryPatterns(image, 24, 8)&#xA;        kp, des = IF.SIFT(image)&#xA;        if len(kp) == 0:&#xA;            #print (i)&#xA;            #print (index)&#xA;            #print (sample)&#xA;            #return 0&#xA;            des = np.zeros(128)&#xA;        sift_features.append(des)&#xA;        global_feature = np.hstack([hist, haralick, hu, lbp])&#xA;        global_features.append(global_feature)&#xA;        labels.append(sample[&quot;class_id&quot;])&#xA;    scaler = MinMaxScaler(feature_range=(0, 1))&#xA;    rescaled = scaler.fit_transform(global_features)&#xA;    return sift_features, rescaled, labels&#xA;&#xA;def BOVW(feature_descriptors, n_clusters = 100):&#xA;    print(&quot;Bag of visual words with {} clusters&quot;.format(n_clusters))&#xA;    #take all features and put it into a giant list&#xA;    combined_features = np.vstack(np.array(feature_descriptors))&#xA;    #train kmeans on giant list&#xA;    print(&quot;Starting K-means training&quot;)&#xA;    kmeans = MiniBatchKMeans(n_clusters=n_clusters, random_state=0).fit(combined_features)&#xA;    print(&quot;Finished K-means training, moving on to prediction&quot;)&#xA;    bovw_vector = np.zeros([len(feature_descriptors), n_clusters])#number of images x number of clusters. initiate matrix of histograms&#xA;    for index, features in enumerate(feature_descriptors):#sift descriptors in each image&#xA;        try:&#xA;            for i in kmeans.predict(features):#get label for each centroid&#xA;                bovw_vector[index, i] += 1#create individual histogram vector&#xA;        except:&#xA;            pass&#xA;    return bovw_vector#this should be our histogram&#xA;&#xA;if __name__ == '__main__':&#xA;    n_clusters = 100&#xA;    #set model&#xA;    model = GaussianNB()&#xA;    image_list = pd.read_csv(&quot;image_list.csv&quot;)&#xA;    image_list_subset = image_list.groupby('class_id').head(80)#image_list.loc[(image_list[&quot;class_id&quot;] == 0) | (image_list[&quot;class_id&quot;] == 19)]&#xA;    shape = (330,230)&#xA;    train, test = train_test_split(image_list_subset, test_size=0.1, random_state=42)&#xA;&#xA;    train_sift_features, train_global_features, y_train = extract_features(train)&#xA;    train_histogram = BOVW(train_sift_features, n_clusters)&#xA;    import matplotlib.pyplot as plt&#xA;    plt.plot(train_histogram[100], 'o')&#xA;    plt.ylabel('frequency');&#xA;    plt.xlabel('features');&#xA;&#xA;    test_sift_features, test_global_features, y_test = extract_features(test)&#xA;    test_histogram = BOVW(test_sift_features, n_clusters)&#xA;&#xA;    '''Naive Bays'''&#xA;    y_hat = model.fit(train_histogram, y_train).predict(test_histogram)&#xA;    print(&quot;Number of correctly labeled points out of a total {} points : {}. An accuracy of {}&quot;&#xA;          .format(len(y_hat), sum(np.equal(y_hat,np.array(y_test))), &#xA;                  sum(np.equal(y_hat,np.array(y_test)))/len(y_hat)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51431" LastEditorUserId="25218" LastEditDate="2018-05-03T06:37:38.367" LastActivityDate="2018-12-03T14:26:21.923" Title="Bag of Visual Words" Tags="|python|clustering|image-classification|k-means|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31141" PostTypeId="1" AcceptedAnswerId="31155" CreationDate="2018-05-02T23:29:23.750" Score="2" ViewCount="5026" Body="&lt;p&gt;I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;Note that Keras has built-in utilities for doing one-hot encoding of text at the word level or character&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked into Keras metods and I cannot find which function he is referring to. &lt;code&gt;keras.utils.to_categorical&lt;/code&gt; does not seem to be applicable directly here, as it requires &lt;code&gt;int&lt;/code&gt; inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to efficiently encode some text at character level for a small RNN project: what can I use there?&lt;/p&gt;&#xA;" OwnerUserId="39830" LastActivityDate="2018-05-03T09:48:40.953" Title="One hot encoding at character level with Keras" Tags="|keras|text|encoding|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31143" PostTypeId="2" ParentId="11928" CreationDate="2018-05-03T03:24:08.900" Score="6" Body="&lt;p&gt;Don't forget&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col_mask=df.isnull().any(axis=0) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which returns a boolean mask indicating np.nan values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;row_mask=df.isnull().any(axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which return the rows where np.nan appeared. Then by simple indexing you can flag all of your points that are np.nan.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.loc[row_mask,col_mask]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51210" LastEditorUserId="82136" LastEditDate="2019-09-14T14:03:51.563" LastActivityDate="2019-09-14T14:03:51.563" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31144" PostTypeId="1" CreationDate="2018-05-03T04:34:25.040" Score="0" ViewCount="61" Body="&lt;p&gt;So I was training a fairly shallow convnet, because my deepnet based on vgg19 wasn't working. 2 conv layers and 2 flat layers, the second flat layer was the output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It converged quickly to all zeros in the second conv layer. First conv layer wasn't all zeros. First flat layer appears to have learned the ditribution of classes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it appears that my network's strategy os to ignore the inputs and just predict the class distribution. I tried class weights, same result just a more uniform distribution gets predicted.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I changed the learning rate, optimizer, and even tried gradient clipping. I augmented the data, and introduced regularization on the data, and network via dropout layers. No luck. Same result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why would my network exhibit this behavior and what can be done about it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a code snippet that defines the network that I am using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def discriminator_model():&#xA;&#xA;&#xA;model = Sequential()&#xA;model.add(Conv2D(32, &#xA;                 (10, 10),&#xA;                 strides=(2,2),&#xA;                 input_shape=( 256, 256, 3),&#xA;                 kernel_initializer='random_uniform',&#xA;                 bias_initializer='zeros'))&#xA;#model.add(Activation('relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Conv2D(64, (5, 5), activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Flatten())&#xA;model.add(Dense(228, activation='sigmoid'))&#xA;return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After training I was trying to see what the layers had actually learned with the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras import backend as K&#xA;&#xA;inp = d.input                                           # input placeholder&#xA;outputs = d.layers[3].output&#xA;functors = K.function([inp], [outputs])&#xA;print(functors([x])[0].shape)&#xA;arr = functors([x])[0].reshape(64,29,29,1)&#xA;for i in range(32):&#xA;     #p = (255./(np.max(arr[i])-np.min(arr[i])))*(arr[i]-np.min(arr[i]))&#xA;     p=255.0*arr[i]&#xA;     plt.imshow(p.reshape(29,29))&#xA;     plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is an example of what I am getting as output from the &#xA;&lt;a href=&quot;https://i.stack.imgur.com/aEzZr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aEzZr.png&quot; alt=&quot;output from 2nd conv layer&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And here is example output from the first layer for comparison, which I generated with similar code:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/fpcAG.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fpcAG.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also tried changing loss functions, removing pooling layers, etc. I know it is weird. I've never come across anything quite like this before.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastEditorUserId="29169" LastEditDate="2021-02-13T04:23:17.223" LastActivityDate="2021-02-13T04:23:17.223" Title="My weight matrix converged to zeros" Tags="|convolutional-neural-network|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31145" PostTypeId="2" ParentId="30819" CreationDate="2018-05-03T04:46:04.307" Score="3" Body="&lt;p&gt;You can do the following&#xA;First resize the images up to certain extent and then pad the image from all sides ,which could help in maintaining the features in the image.&lt;/p&gt;&#xA;" OwnerUserId="51192" LastActivityDate="2018-05-03T04:46:04.307" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31146" PostTypeId="2" ParentId="30097" CreationDate="2018-05-03T04:53:35.880" Score="1" Body="&lt;p&gt;You can use dropout which will help in controlling the model to over train.&#xA;While using CNN,data-set plays an important role.The more data-set the more model can learn features from it.&#xA;You can divide your data-set in 3 parts.&#xA;Training ,testing and validation.&lt;/p&gt;&#xA;" OwnerUserId="51192" LastActivityDate="2018-05-03T04:53:35.880" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31149" PostTypeId="1" CreationDate="2018-05-03T07:03:19.573" Score="2" ViewCount="93" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9TSL0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9TSL0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good day. Taking into account the picture shown, using tensor product is computationally expensive considering the fact that it has higher dimensions. I am just thinking why it is compared both additive and tensor products here (having the same training error, test error, bayes error values)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing I want to clarify is how did the formula for degrees of freedom for additive arrive as total df = 1 + (4-1) + (4-1) = 7? (Tensor product is a simple multiplication of 4x4).  &lt;/p&gt;&#xA;" OwnerUserId="51495" LastActivityDate="2018-05-03T07:03:19.573" Title="What Is The Difference Between Additive Natural Cubic Splines and Tensor Product Natural Cubic Splines?" Tags="|logistic-regression|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31153" PostTypeId="1" CreationDate="2018-05-03T08:17:13.427" Score="2" ViewCount="1874" Body="&lt;p&gt;Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like:&lt;br&gt;&#xA;X = very long sequence of varying length. &#xA;Y = class (assume for simplicity y=0/1).&lt;br&gt;&#xA;The class depends on the whole sequence - it is impossible to determine the class without seeing all of it.&lt;br&gt;&#xA;How NN handle this? I have read various methods, such as truncating the sequence (which is a problem - there is no class for truncated part of the sequence, so I cannot label it), using encoder/decoder lstm (again, couldn't completely understand how it solves the problem), etc. &lt;/p&gt;&#xA;" OwnerUserId="51540" LastActivityDate="2018-05-03T13:25:47.553" Title="Very long sequence in neural networks" Tags="|deep-learning|lstm|sequence|sequence-to-sequence|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31154" PostTypeId="2" ParentId="31153" CreationDate="2018-05-03T09:16:09.757" Score="2" Body="&lt;p&gt;You can indeed use the ability of recurrent network like LSTM to handle the varying length problem.&#xA;But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the Input layer : the 0s will not be taken into account for training the model. Bellow an example using the Keras framework (not complete, not tested) :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Model&#xA;from keras.layers import Input, LSTM, Dense&#xA;input = Input(shape=[None, nb_of_features_per_timestep])&#xA;layer = Masking(mask_value=0)(input)&#xA;layer = LSTM(50, return_sequences=False)(layer)&#xA;output = Dense(nb_classe)(layer)&#xA;model = Model(input,output)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can also use the bucketing trick : you group your dataset into &quot;buckets&quot; of same size (approximately). You can then create batches of same size. You don't need the Masking layer anymore.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;You can also use fully convolutional network. Those networks can also take Input with different size. I let you search if you want to use that.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Concerning encoder/decoder : those model can maybe help you but I think that you will have to use one of the tricks that I mentioned.&#xA;If you don't understand how RNNs work, I recommend you to dig a little into the subject before.&lt;/p&gt;&#xA;" OwnerUserId="51541" LastEditorUserId="51541" LastEditDate="2018-05-03T13:25:47.553" LastActivityDate="2018-05-03T13:25:47.553" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31155" PostTypeId="2" ParentId="31141" CreationDate="2018-05-03T09:48:40.953" Score="5" Body="&lt;p&gt;I think that you are looking for the keras Tokenizer with the char_level=True flag:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.preprocessing.text import Tokenizer&#xA;tokenizer = Tokenizer(char_level=True)&#xA;tokenizer.fit_on_texts(your_dataset_train)&#xA;sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now that you have sequences of Integer, you can use &lt;code&gt;keras.utils.to_categorical&lt;/code&gt; =)&lt;/p&gt;&#xA;" OwnerUserId="51541" LastActivityDate="2018-05-03T09:48:40.953" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31156" PostTypeId="1" CreationDate="2018-05-03T09:59:12.253" Score="2" ViewCount="193" Body="&lt;p&gt;For example, something along the lines of the professionalism of:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51511" LastActivityDate="2018-05-15T20:36:36.040" Title="What are the online platforms that cater to freelance data scientists looking for work?" Tags="|data-mining|data-cleaning|" AnswerCount="2" CommentCount="0" ClosedDate="2018-05-15T22:55:07.787" ContentLicense="CC BY-SA 4.0" />
  <row Id="31157" PostTypeId="2" ParentId="29672" CreationDate="2018-05-03T10:08:43.913" Score="1" Body="&lt;p&gt;The cause of your huge loss is probably a lack of normalization. As an example, in the &lt;a href=&quot;https://blog.keras.io/building-autoencoders-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras-blog&lt;/a&gt; post you linked, they conduct such normalization by dividing the input by 255, which is the maximum value an element in the input can be of. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for which loss function to use, if you have a look at &lt;a href=&quot;https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this keras-blog post&lt;/a&gt;, they construct a LSTM-based auto encoder for text which uses categorical crossentropy as a loss function.&lt;/p&gt;&#xA;" OwnerUserId="51545" LastActivityDate="2018-05-03T10:08:43.913" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31158" PostTypeId="1" CreationDate="2018-05-03T10:43:13.053" Score="0" ViewCount="885" Body="&lt;p&gt;I am trying to plot results from document (data source: twitter) clustering using python's sklearn. Below is 10-words of words belonging to each cluster:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;donated     0.221646&#xA;cross       0.034354&#xA;red         0.033817&#xA;relief      0.030248&#xA;every       0.026620&#xA;victims     0.023084&#xA;efforts     0.021768&#xA;redcross    0.021427&#xA;wade        0.019810&#xA;million     0.018602&#xA;Name: 0, dtype: float64&#xA;&#xA;&#xA;victims      0.030352&#xA;help         0.026362&#xA;relief       0.020772&#xA;donate       0.020332&#xA;donations    0.012362&#xA;please       0.012352&#xA;want         0.012348&#xA;sandy        0.010720&#xA;new          0.010089&#xA;like         0.008957&#xA;Name: 1, dtype: float64&#xA;&#xA;&#xA;affected     0.179101&#xA;help         0.054969&#xA;please       0.027684&#xA;donate       0.026976&#xA;red          0.026417&#xA;people       0.026100&#xA;cross        0.025541&#xA;sandyhelp    0.025537&#xA;thoughts     0.023171&#xA;like         0.021116&#xA;Name: 2, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am able to generate wordcloud by using &lt;a href=&quot;https://github.com/amueller/word_cloud&quot; rel=&quot;nofollow noreferrer&quot;&gt;Amueller's wordcloud&lt;/a&gt; . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wanted to create a scatter plot of these clusters for an instance in above example, indicate the words are in one cluster have red color, other cluster has green, etc. I want those in one plot. &lt;/p&gt;&#xA;" OwnerUserId="9273" LastActivityDate="2018-05-03T10:43:13.053" Title="plot the results from kmeans" Tags="|python|visualization|k-means|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31159" PostTypeId="1" AcceptedAnswerId="31169" CreationDate="2018-05-03T11:47:55.400" Score="0" ViewCount="656" Body="&lt;p&gt;&lt;a href=&quot;https://www.datacamp.com/community/tutorials/learn-data-science-infographic&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.datacamp.com/community/tutorials/learn-data-science-infographic&lt;/a&gt;&#xA;&lt;a href=&quot;https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These links contain almost everything but not the difference between data science and data analytics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is data analytics a part of the data science workflow? Is data analytics a subset of data science?&lt;/p&gt;&#xA;" OwnerUserId="51546" LastActivityDate="2022-09-14T08:25:31.997" Title="What is difference between a Data Scientist and a Data Analyst?" Tags="|data|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31164" PostTypeId="1" CreationDate="2018-05-03T14:46:03.003" Score="2" ViewCount="62" Body="&lt;p&gt;I work with two datasets. The first dataset contains fluor values measured every minute. The second dataset contains certain events and their time. We know that these events cause peaks in fluor values shortly before and shortly after the event time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A simplified reproducible example in R:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I provide a simplified version of the R code I use to relate the fluor values to events. I have a series of fluor values measured every minute. Next I have a second dataset with three events A, B and C that occur at three different times. It seems there are three large peaks in fluor value around each event.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The objective is to classify each value as corresponding to an event whenever the time of that value falls within 15 minutes of that time of an event. Fluor values that fall outside of any events are categorized as NA.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Set seed&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Generate fluor values dataset and events dataset&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values &amp;lt;- data.table(date.time.value=seq(ymd_hms(&quot;2018-01-01 00:00:00&quot;), by= &quot;min&quot;, length.out = 200),fluor=5*sin(seq(0,20,length.out = 200)))+(abs(rnorm(200,6,1))+2)&#xA;events &amp;lt;- data.table(event=c(&quot;A&quot;,&quot;B&quot;,&quot;C&quot;),date.time.event=c(ymd_hms(&quot;2018-01-01 00:20:00&quot;),ymd_hms(&quot;2018-01-01 01:20:00&quot;),ymd_hms(&quot;2018-01-01 02:20:00&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Using the data.table package: I add an &quot;event&quot; variable to the values dataset. That variable takes on the name of an event (in this case A, B or C) whenever the date.time.value is within 15 minutes of a date.time.event.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;values[, event:=events[.SD[, .(d_dn=date.time.value-15*60, d_up=date.time.value+15*60)], on=.(date.time.event&amp;gt;=d_dn, date.time.event&amp;lt;=d_up), event]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Here's what the result looks like. We see that we have one fluor value peak for each event A, B and C and that the other fluor values have NA as event:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ggplot(values, aes(date.time.value, fluor,col=event)) + geom_point()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PYZpr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PYZpr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;A real example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I managed to relate the two datasets by coloring the fluor values that occur several minutes before or several minutes after an event. This is what the result looks like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FsK3S.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FsK3S.png&quot; alt=&quot;Note how the fluor is higher whenever there is an event as opposed to when the event is &amp;quot;none&amp;quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I'm missing:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I would like to define &quot;peaks&quot; so that in my simplified example all the green-colored fluor values of event B belong to a specific &quot;peak number x&quot;. Currently it's impossible to distinguish between fluor values of the same event but from different peaks because I only have observations of fluor with a corresponding variable &quot;event&quot;.&lt;/li&gt;&#xA;&lt;li&gt;In the simplified example you can see (exagerated) that quite a lot of high fluor values are considered to be NA even though their high value is most likely are caused by an event. Hence I'm looking for a better way to relate fluor values to events than simply saying &quot;15 minutes before or 15 minutes after&quot;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm almost certain that there must be some kind of R-package that's specifically built for my problem here but I just can't figure out which one...&lt;/p&gt;&#xA;" OwnerUserId="51558" LastActivityDate="2024-01-18T19:00:35.073" Title="Relating changes of a value in time to known events" Tags="|r|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31166" PostTypeId="1" AcceptedAnswerId="31187" CreationDate="2018-05-03T15:53:24.130" Score="0" ViewCount="63" Body="&lt;p&gt;I'm developing a model for unsupervised anomaly detection. I have a dataset representing communications between users (each example represents a communication): there are many features (time, duration, ...) and the ids of sender and receiver. My question is: how to represent the link between those two users?&#xA;I have several ideas, but each of them seems to have serious drawbacks:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Use id as is. Drawback: even if ids are integers, they have no &lt;em&gt;numerical sense&lt;/em&gt; (id 15 is not 3 times id 5) and I think this may mislead the system&lt;/li&gt;&#xA;&lt;li&gt;Use sort of vectors: for example, with 3 users: user1 = (0 0 1), user2 = (0 1 0), user3 = (1 0 0). Drawback : the number of users may vary over time, thus the number of features would vary as well and I would have to re-train my model.&lt;/li&gt;&#xA;&lt;li&gt;Graph theory: I've heard of that way of representing data, which could fit perfectly my data model. Drawback: I've absolutely no knowledge in graph analysis&lt;/li&gt;&#xA;&lt;li&gt;Assign each user a id which is a prime number. That way a communication could be represented in an unique way as the product of the 2 ids. Drawback: as for point 1, ids do not have a &quot;numerical sense&quot;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;What do you think may be the better way to represent these relations?&lt;/p&gt;&#xA;" OwnerUserId="50973" LastActivityDate="2018-05-04T00:06:28.187" Title="How to represent relation between users as a feature?" Tags="|feature-extraction|anomaly-detection|feature-construction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31167" PostTypeId="1" CreationDate="2018-05-03T16:03:30.750" Score="0" ViewCount="16994" Body="&lt;pre&gt;&lt;code&gt;from keras import optimizers&#xA;from keras.models import load_model&#xA;from keras.preprocessing import image&#xA;import numpy as np&#xA;import cv2&#xA;import scipy.misc&#xA;from keras.wrappers.scikit_learn import KerasClassifier&#xA;# dimensions of our images&#xA;img_width, img_height = 313, 220&#xA;# load the model we saved&#xA;model = load_model('model.h5')&#xA;sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)&#xA;model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy','mse'])&#xA;test_image= image.load_img('/Images/1.jpg',target_size = (img_width, img_height))&#xA;#x= scipy.misc.imread('/Images/1.jpg').shape&#xA;test_image = image.img_to_array(test_image)&#xA;print test_image&#xA;test_image = np.expand_dims(test_image, axis = 0)&#xA;test_image = test_image.reshape(img_width, img_height*3)&#xA;result = model.predict(test_image)&#xA;print result&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I ran the above code I get this error: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: Error when checking : expected dense_1_input to have shape (36,) but got array with shape (660,)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="51570" LastEditorUserId="28175" LastEditDate="2018-05-03T18:42:05.910" LastActivityDate="2018-05-04T00:53:14.820" Title="how to predict an image using saved model" Tags="|machine-learning|python|keras|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31168" PostTypeId="1" CreationDate="2018-05-03T16:51:49.397" Score="0" ViewCount="294" Body="&lt;p&gt;I am new to R and usually rely on stata. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. The variables have not been standardised. The data is at an Individual level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Questions I have are the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I aggregate the surveys?&#xA;How can Identify the number of surveys which have provided information on a particular variable i'm interested in like education?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ML0fI.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ML0fI.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;raj&lt;/p&gt;&#xA;" OwnerUserId="51573" LastActivityDate="2023-07-11T14:40:03.447" Title="R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?" Tags="|r|time-series|data-cleaning|aggregation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31169" PostTypeId="2" ParentId="31159" CreationDate="2018-05-03T17:07:10.767" Score="1" Body="&lt;p&gt;Please visit &lt;a href=&quot;https://www.projectpro.io/article/difference-between-data-analyst-and-data-scientist/332&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and read the difference between Data analyst and Data scientist.&lt;/p&gt;&#xA;&lt;p&gt;You will find the above link very interesting that fulfills your need as you will also find the words by Data scientist of LinkedIn.&lt;/p&gt;&#xA;&lt;p&gt;Important lines from the above link:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Data analysts are masters in SQL and use regular expression to slice and dice the data. With some level of scientific curiosity data analysts can tell a story from data. A data scientist on the other hand possess all the skills of a data analysts with strong foundation in modelling, analytics, math, statistics and computer science. What differentiates a data scientist from a data analyst is the strong acumen along with the ability to communicate the findings in the form of a story to both IT leaders and business stakeholders in such a way that it can influence the manner in which a company approaches a business challenge.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="49655" LastEditorUserId="140409" LastEditDate="2022-09-14T08:25:31.997" LastActivityDate="2022-09-14T08:25:31.997" CommentCount="0" CommunityOwnedDate="2018-05-03T17:07:10.767" ContentLicense="CC BY-SA 4.0" />
  <row Id="31170" PostTypeId="1" AcceptedAnswerId="31378" CreationDate="2018-05-03T09:29:05.003" Score="1" ViewCount="80" Body="&lt;p&gt;I am trying to make my neural network as dynamic as possible. For example, I want to be able to increase hidden_layers number. It works so far with one hidden layer, but if I increase it to two, it instantly breaks. There is a problem when the backpropagation is reaching the layer between &lt;strong&gt;Inputs&lt;/strong&gt; &lt;strong&gt;&amp;lt;-&lt;/strong&gt; &lt;strong&gt;1st_hidden&lt;/strong&gt;. Here is &lt;a href=&quot;https://pastebin.com/AEmf1rfm&quot; rel=&quot;nofollow noreferrer&quot;&gt;my code.&lt;/a&gt; I am trying to solve XOR problem by providing this to my NN:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs  = np.array([[1, 0], [0, 1], [1, 1], [0, 0]]) &#xA;targets = np.array([[1], [1], [0], [0]])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone help me? I can not seem to solve it.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;nn = NeuralNetwork(input_nodes = 2, hidden_layers = 1, hidden_nodes = 3, output_nodes = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I initialize my neural network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class NeuralNetwork:&#xA;def __init__(self, input_nodes, hidden_layers, hidden_nodes, output_nodes):&#xA;&#xA;    self.input_nodes   = input_nodes&#xA;    self.hidden_layers = hidden_layers&#xA;    self.hidden_nodes  = hidden_nodes&#xA;    self.output_nodes  = output_nodes&#xA;    self.layers_count  = self.hidden_layers + 1&#xA;&#xA;    self.weights       = []&#xA;    self.bias          = []&#xA;&#xA;    min_value = -2&#xA;    max_value =  2&#xA;&#xA;    # Initialize weights and bias matrices and place them in a list&#xA;    # Matrix shape within lists&#xA;    # [(2,2), (2,2), (1,2)]&#xA;&#xA;    for i in range(self.layers_count):&#xA;        if i == 0:&#xA;            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.input_nodes)))&#xA;            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))&#xA;        elif i == self.hidden_layers:&#xA;            self.weights.append(np.random.uniform(min_value, max_value, (self.output_nodes, self.hidden_nodes)))&#xA;            self.bias.append(np.random.uniform(min_value, max_value, (self.output_nodes,)))&#xA;        else:&#xA;            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.hidden_nodes)))&#xA;            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))&#xA;&#xA;    self.learning_rate = 0.1&#xA;&#xA;def train(self, input_array, targets):&#xA;&#xA;    inputs = np.transpose(input_array)&#xA;&#xA;    applied_activation = nn.feedforward(inputs)&#xA;&#xA;    output_errors = np.subtract(targets, applied_activation[-1])&#xA;&#xA;    # Backpropogation&#xA;    for i in reversed(range(self.layers_count)):&#xA;        delta    = np.ndarray((self.input_nodes,))&#xA;        gradient = np.ndarray((self.hidden_layers,))&#xA;&#xA;        if i == self.hidden_layers:&#xA;            errors = output_errors&#xA;            delta, gradient = calculate_weights_deltas(applied_activation[i], output_errors, applied_activation[i-1])&#xA;        elif i == 0:     &#xA;            # IT BREAKS HERE&#xA;            errors          = calculate_errors(self.weights[i+1], output_errors)&#xA;            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, inputs)&#xA;        else:&#xA;            errors          = calculate_errors(self.weights[i+1], output_errors)&#xA;            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])&#xA;&#xA;        delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])&#xA;        # Adjust weights by deltas&#xA;        self.weights[i] = np.add(self.weights[i], delta)&#xA;        # Adjust the bias by its deltas &#xA;        self.bias[i] = np.add(self.bias[i], gradient)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The error I get is the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;            elif i == 0:       &#xA;                errors          = calculate_errors(self.weights[i+1], output_errors)&#xA;&#xA;            ValueError: shapes (3,3) and (1,) not aligned: 3 (dim 1) != 1 (dim 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51778" OwnerDisplayName="Gabriele" LastActivityDate="2018-05-08T08:38:53.840" Title="Neural network breaks if hidden layers are increased" Tags="|python|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31173" PostTypeId="1" CreationDate="2018-05-03T18:18:20.480" Score="6" ViewCount="466" Body="&lt;p&gt;I'm working on machine learning problem where I'm only interested in getting high accuracy within a narrow band of my predicted likelihoods. Specifically, I want an algorithm that will score very accurately when it predicts a likelihood above a specific threshold. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A motivating example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm presented with 1 million closed boxes, 10 percent of which contain a gold coin. The likelihood of a box containing a gold coin is associated with physical characteristics of the boxes. I'm allowed to open 500K boxes, chosen at random, to see which contain the coins.  After this, I'm allowed to choose 100 of the remaining 500K boxes and keep any gold that I find.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm looking to build a classifier to solve this problem.  Specifically, I'm looking for the correct way to incorporate the constraint that I am only able to open 100 boxes in the test sample. Put another way, my measure of performance is limited to a subsample of my test data, so I need a classifier that is highly accurate when it predicts the highest likelihoods. Prediction errors outside of these 100 boxes are of zero concern.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm aware of cost-weighted loss functions that penalize false positives/negatives differently, but I'm not familiar with any that solve this specific problem of optimizing for a localized accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I'm trying to answer two questions:&lt;br&gt;&#xA;1) Are there specific classification algorithms well suited for this type of problem? &#xA;2) Can I modify the loss function used on my training set to produce a high localized accuracy within these specific 100 boxes?  &lt;/p&gt;&#xA;" OwnerUserId="51578" LastActivityDate="2020-04-30T02:44:38.970" Title="Classifier that optimizes performance on only a subset of the data?" Tags="|machine-learning|classification|loss-function|classifier|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31174" PostTypeId="2" ParentId="28582" CreationDate="2018-05-03T18:31:19.903" Score="1" Body="&lt;p&gt;In your method, you would build a model which maps the input features (ad type, customer descriptors) to the output (purchase made = 0/1). You can use a model such as a logistic regression, or a decision tree (which can model the interaction effects between the ad type and the customer descriptors.) This seems to be reasonable and is one of the approaches commonly used for ad selection.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few other options based on customer similarity:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Near-neighbour based approach: Based on the customer descriptors, identify similar neighbours of the current customer. Compute the probability of conversion for each ad type for all the neighbours put together, and pick the ad type which has the highest probability of conversion.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Cluster based approach: Form clusters of customers based on the customer descriptors. Map the current customer into one of the clusters, and pick the ad type which maximizes conversion probability for that cluster.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;With respect to designing a system of ad selection using such a model, it is also important for the system to explore new regions of the space, so some small randomization of predictions will help here.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-03T18:31:19.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31175" PostTypeId="1" AcceptedAnswerId="31213" CreationDate="2018-05-03T19:07:53.610" Score="4" ViewCount="7784" Body="&lt;p&gt;I have a question targeting some basics of CNN. I came across various CNN networks like AlexNet, GoogLeNet and LeNet.&lt;/p&gt;&#xA;&lt;p&gt;I read at a lot of places that AlexNet has 3 Fully Connected layers with 4096, 4096, 1000 layers each. The layer containing 1000 nodes is the classification layer and each neuron represents the each class.&lt;/p&gt;&#xA;&lt;p&gt;Now I came across GoogLeNet. I read about its architecture &lt;a href=&quot;http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;It says that GoogLeNet has 0 FC layers. However, you do need the 1000 node layer in the end with Softmax activation for the classification task. So the final layer isn't treated as FC in this?&lt;/p&gt;&#xA;&lt;p&gt;Also then, what is the number of FC layers in LeNet-5?&lt;/p&gt;&#xA;&lt;p&gt;A bit confused. Any help or leads would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="51581" LastEditorUserId="29169" LastEditDate="2021-02-13T04:23:04.533" LastActivityDate="2021-02-13T04:23:04.533" Title="Number of Fully connected layers in standard CNNs" Tags="|deep-learning|neural-network|cnn|convolutional-neural-network|alex-net|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31176" PostTypeId="2" ParentId="31173" CreationDate="2018-05-03T19:26:06.337" Score="0" Body="&lt;p&gt;There are at least two approaches:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train on a subsample of data. 100 positive examples and only 100 negative examples (randomly sampled from all negative samples).&lt;/li&gt;&#xA;&lt;li&gt;Create a 2 pass system. 1st pass uses a high recall algorithm, selects reasonable options (eliminate options that are not reasonable). The 2nd pass is a high precision algorithm. It only looks possible options and selects the most likely options. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-03T19:26:06.337" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31177" PostTypeId="2" ParentId="31175" CreationDate="2018-05-03T19:33:02.003" Score="0" Body="&lt;p&gt;There are 2 fully connected layers in LeNet-5. It is shown in Figure 2 of &quot;&lt;a href=&quot;http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gradient-Based Learning Applied to Document Recognition&lt;/a&gt;&quot;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/sKH4d.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/sKH4d.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-03T19:33:02.003" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31178" PostTypeId="2" ParentId="27247" CreationDate="2018-05-03T19:42:22.337" Score="0" Body="&lt;p&gt;You can use something like &lt;a href=&quot;https://imagecolorpicker.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Color Picker&lt;/a&gt; to manually find the HTML or RGB or HSV values.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-03T19:42:22.337" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31179" PostTypeId="1" AcceptedAnswerId="31214" CreationDate="2018-05-03T19:45:29.013" Score="2" ViewCount="96" Body="&lt;p&gt;Let's assume that we have this magical candy machine which takes candies as input and delivers again candies as output.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;For any given time t, it picks a random function which is strictly&#xA;increasing up to a point such as f(2) = 6 here, and then it strictly&#xA;decreases. It likes to be challenging but if you become greedy it&#xA;punishes you like most of the stuff in life.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;f(1) = 5&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;f(2) = 6&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;f(3) = 4&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;f(4) = 2&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;f(100) = 0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The tricky point is that this function is changing all the time, but&#xA;still highly correlated with time. So f() will be similar between&#xA;t(1) and t(2), but very different between t(1) and t(100).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I want to write a program to maximize my candies using this magical candy machine. I know the fundamentals of ML but I'm not sure which approach would fit best here. Any ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: You can only play only once every minute.&lt;/p&gt;&#xA;" OwnerUserId="51582" LastEditorUserId="40853" LastEditDate="2018-05-04T10:57:39.877" LastActivityDate="2018-05-18T08:01:19.933" Title="How to optimize for time correlated hidden function - the magical candy machine" Tags="|regression|optimization|game|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31180" PostTypeId="1" CreationDate="2018-05-03T21:09:45.843" Score="1" ViewCount="188" Body="&lt;p&gt;I'm a computer science student and one of my professors ask me if I can use cnn to make a python application which detect tumors &#xA;can anyone please guide me if that is possible, to understand cnn and apply It on my project&lt;/p&gt;&#xA;" OwnerUserId="50477" LastActivityDate="2018-05-03T21:48:56.550" Title="Understanding cnn" Tags="|deep-learning|cnn|" AnswerCount="1" CommentCount="1" ClosedDate="2018-05-04T03:44:57.983" ContentLicense="CC BY-SA 4.0" />
  <row Id="31181" PostTypeId="2" ParentId="31175" CreationDate="2018-05-03T21:43:18.203" Score="0" Body="&lt;p&gt;I just checked the original inception network paper (GoogLeNet) &lt;a href=&quot;https://arxiv.org/pdf/1409.4842.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; and it does have a fully connected layer before the softmax.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/scaoF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/scaoF.png&quot; alt=&quot;Part of the network from the original paper.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51375" LastActivityDate="2018-05-03T21:43:18.203" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31182" PostTypeId="2" ParentId="31180" CreationDate="2018-05-03T21:48:56.550" Score="1" Body="&lt;p&gt;First review the basics of CNNs:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad5df8&quot; rel=&quot;nofollow noreferrer&quot;&gt;high level introduction&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs231n.stanford.edu/&quot; rel=&quot;nofollow noreferrer&quot;&gt;a more detailed overview&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;After you gain some intuition, look into &lt;a href=&quot;https://www.tensorflow.org/tutorials/deep_cnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;tensorflow&lt;/a&gt; for python implementations.  Depending on what your professor asked of you, explore:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review&quot; rel=&quot;nofollow noreferrer&quot;&gt;semantic segmentation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9&quot; rel=&quot;nofollow noreferrer&quot;&gt;object detection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;or the basic &lt;a href=&quot;https://towardsdatascience.com/deep-learning-for-image-classification-why-its-challenging-where-we-ve-been-and-what-s-next-93b56948fcef&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification&lt;/a&gt; task&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There is a plethora of information on this topic, especially for your particular situation:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S2314728817300636&quot; rel=&quot;nofollow noreferrer&quot;&gt;Classification using deep learning neural networks for brain tumors&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1703.07047.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;High-Resolution Breast Cancer Screening with&#xA;Multi-View Deep Convolutional Neural Networks&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs231n.stanford.edu/reports/2017/pdfs/518.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Convolutional Neural Networks for Lung Cancer Detection&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Google is your friend!&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-05-03T21:48:56.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31183" PostTypeId="2" ParentId="31179" CreationDate="2018-05-03T21:54:08.137" Score="2" Body="&lt;h3&gt;Thought #1&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;From the short description you give, and assuming you have some data (or can synthesise it), I would have a go at training a Hidden Markov Model. Have a look here for &lt;a href=&quot;http://setosa.io/ev/markov-chains/&quot; rel=&quot;nofollow noreferrer&quot;&gt;an awesome visual primer&lt;/a&gt;. Intuitively, HMMs do what you describe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some observables:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;how much candy you put in the machine,&lt;/li&gt;&#xA;&lt;li&gt;how much comes out, and&lt;/li&gt;&#xA;&lt;li&gt;which timestep we are in&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are certain states (here just discrete states):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;amount of dispensed candy goes up (strictly increasing)&lt;/li&gt;&#xA;&lt;li&gt;amount of dispensed candy goes down (strictly decreasing)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Lastly, there is a &lt;em&gt;transition&lt;/em&gt;: between these state, given the inputs. These are the observables, as far as you are concerned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With HMMs, we are however assuming that there is more than meets the eye; there are some hiddens states (latent states), which are unobservable to use. The model keeps track of these and uses them in conjunction with the observables list above to decide how to act - it has the full mapping function. Below is a schematic diagram of such a model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ydQE8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ydQE8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This just shows several states, along with the probabilities of transitioning to the other states, or stating in the same state. Check out &lt;a href=&quot;http://www.blackarbs.com/blog/introduction-hidden-markov-models-python-networkx-sklearn/2/9/2017&quot; rel=&quot;nofollow noreferrer&quot;&gt;the source&lt;/a&gt; for a detailed description of the diagram. One can already imagine sketching this drawing for your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more on the theory of HMMs, I recommend this &lt;a href=&quot;http://melodi.ee.washington.edu/people/bilmes/mypapers/em.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;text based walkthrough by Jeff Bilmes&lt;/a&gt;, and &lt;a href=&quot;https://www.youtube.com/watch?v=7KGdE2AK_MQ&amp;amp;list=PLD0F06AA0D2E8FFBA&amp;amp;index=94&quot; rel=&quot;nofollow noreferrer&quot;&gt;this YouTube series by &lt;em&gt;mathematicalmonk&lt;/em&gt; (chapter 14 of the playlist)&lt;/a&gt;. If you go this route, you might consider trying out &lt;a href=&quot;http://pomegranate.readthedocs.io/en/latest/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Pomegranate library (in Python)&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final logic of deciding what to do, what action to take, is something you could either hard-code or use a model to take it for you. For example, a hard-coded approach would be something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if current_state == 'normal':&#xA;    take_lots_of_candy()&#xA;elif current_state == 'unsure':&#xA;    take_some_candy()&#xA;elif current_State == 'dangerous':&#xA;    pause_taking_candy()&#xA;else:&#xA;    catch_other_states()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For a model-based approach, you could expand further on the ideas outlines below.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Thought #2&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Of course, you are in essence just mapping some input to some outputs. Almost any machine learning algorithm will have a good go at it and - depending on the complexity of the internal workings of your magical machine - will be better or worse. Algorithms such as boosting (e.g. via gradient descent) generally require you to put some kind of information into the model e.g. by specifying a regression equation, mapping inputs to outputs. A good implementation is the &lt;a href=&quot;https://cran.r-project.org/web/packages/mboost/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;mboost&lt;/code&gt; package&lt;/a&gt; in &lt;strong&gt;R&lt;/strong&gt;. There is &lt;a href=&quot;https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;a great tutorial&lt;/a&gt; to help decide what might be important.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Thought #3&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Coming back to the point of just mapping inputs to outputs: a simple &lt;a href=&quot;http://scikit-learn.org/stable/modules/neural_networks_supervised.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-Layer Perceptron (MLP)&lt;/a&gt; from Scikit-Learn - a.k.a. feed-forward neural network - should theoretically offer the flexibility to approximate any function you might throw at it (with a few assumptions).&#xA;You could also implement this using one of the various deep learning frameworks, should your MLP just not &lt;a href=&quot;https://www.phrases.org.uk/meanings/cut-the-mustard.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cut the mustard&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Final thoughts&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I would lastly like to highlight that your final model will only likely be as good as the data you feed it, and that while it may work well on your synthesised training data, there are no guarantees of generalisation, without putting more thought into the specific problem and specific distributions from which your data are sampled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you go the way of Markov models, you could perhaps then use their state predictions as inputs to further models; thus creating an ensemble.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get really fancy and more towards current state-of-the-art models in sequence analysis, you could venture into the world of Recurrent Neural Networks, e.g. utilising LSTM cells, which maintain an internal state representation, including new information and selectively discarding old information. This might be a good way to take all your points into consideration in one large model. The catch here would be that you generally require vast amounts of data to train such a model.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-04T01:11:17.620" LastActivityDate="2018-05-04T01:11:17.620" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31184" PostTypeId="2" ParentId="29699" CreationDate="2018-05-03T22:21:11.253" Score="2" Body="&lt;p&gt;&lt;a href=&quot;https://medium.com/@u39kun/titan-v-vs-1080-ti-head-to-head-battle-of-the-best-desktop-gpus-on-cnns-d55a19866b7c&quot; rel=&quot;nofollow noreferrer&quot;&gt;This link&lt;/a&gt; has more realistic benchmarks of 1080ti vs Titan V (same as the V100 but less RAM), using pytorch. Its not nearly the best case scenario of 10x speedups, and seems to be a more modest 20% to 50% boost. This is fairly expected, nVidias grandiose performance claims tend to be only applicable to fairly restricted benchmarks that don't scale well to real world usage. You'd probably get a lot more out of 4 1080ti cards than a single V100 unless you REALLY need a large amount of RAM in a single card.&lt;/p&gt;&#xA;" OwnerUserId="51587" LastActivityDate="2018-05-03T22:21:11.253" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31186" PostTypeId="1" AcceptedAnswerId="31192" CreationDate="2018-05-03T23:46:44.320" Score="2" ViewCount="228" Body="&lt;p&gt;I am using encoder-decoder model to predict binary images from grayscale images. Here is the model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs = Input(shape=(height, width, 1))&#xA;&#xA;conv1 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)&#xA;&#xA;conv1 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)&#xA;&#xA;pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)&#xA;&#xA;conv2 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)&#xA;&#xA;conv2 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)&#xA;&#xA;drop2 = Dropout(0.2)(conv2)&#xA;&#xA;pool2 = MaxPooling2D(pool_size=(2, 2))(drop2)&#xA;&#xA;conv4 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)&#xA;&#xA;conv4 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)&#xA;&#xA;drop4 = Dropout(0.2)(conv4)&#xA;&#xA;pool4 = MaxPooling2D(pool_size=(2, 2))(drop4)&#xA;&#xA;conv5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)&#xA;&#xA;conv5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)&#xA;&#xA;drop5 = Dropout(0.2)(conv5)&#xA;&#xA;up6 = Conv2D(16, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop5))&#xA;&#xA;merge6 = concatenate([drop4,up6], axis = 3)&#xA;&#xA;conv6 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)&#xA;&#xA;conv6 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)&#xA;&#xA;up7 = Conv2D(8, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))&#xA;&#xA;merge7 = concatenate([drop2,up7], axis = 3)&#xA;&#xA;conv7 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)&#xA;&#xA;conv7 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)&#xA;&#xA;&#xA;up9 = Conv2D(4, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))&#xA;&#xA;merge9 = concatenate([conv1,up9], axis = 3)&#xA;&#xA;conv9 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)&#xA;&#xA;conv9 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)&#xA;&#xA;conv9 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)&#xA;&#xA;conv10 = Conv2D(1, 1, activation = 'sigmoid')(conv9)&#xA;&#xA;model = Model(inputs=inputs, outputs=conv10)&#xA;nadam = optimizers.Nadam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=None,  schedule_decay=0.0004)&#xA;model.compile(loss='mean_squared_error', optimizer=nadam, metrics=['accuracy', norm_mse])&#xA;&#xA;nb_epoch = 30&#xA;batch_size = 8&#xA;history = unet.fit(training, trainingLabel,&#xA;      validation_data=(validation, validationLabel),&#xA;      batch_size=batch_size, epochs=nb_epoch, shuffle=&quot;batch&quot;, callbacks=[checkpointer],  verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I also have training dataset size is: 6912, validation dataset size is: 1728&#xA;Each time I start training my encoder-decoder in the very beginning I get different training accuracy and normalized MSE. It significantly affects the testing each time. I understand that weights in the beginning are randomly chosen and so it will affect the performance on the first epoch. My concern is that the difference is too large. Here are some examples:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/30&#xA;6912/6912 [==============================] - 27s 4ms/step - loss: 0.0612 -    acc: 0.9252 - normalized_mse: 0.3661 - val_loss: 0.0367 - val_acc: 0.9559 -    val_normalized_mse: 0.2920&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/30&#xA;6912/6912 [==============================] - 28s 4ms/step - loss: 0.1251 -     acc: 0.8982 - normalized_mse: 0.5686 - val_loss: 0.1077 - val_acc: 0.9564 -     val_normalized_mse: 0.5302&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The last one:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Epoch 1/30&#xA;6912/6912 [==============================] - 26s 4ms/step - loss: 0.1721 -    acc: 0.9400 - normalized_mse: 0.6751 - val_loss: 0.1582 - val_acc: 0.9582 -   val_normalized_mse: 0.6473&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;if I run my model for more than 30 epochs it starts overestimating. &#xA;The size of the Input images is 256x256 each has quite a bit of features and it is grayscale - the output is binary image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can some one please help me to understand if I am doing something wrong? And how can I make my model more stable? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[SOLVED] For those who face the same problem just feed the kernel_initializer some random seed. Example kernel_initializer=he_normal(seed=seed_val). That will solve the issue.&lt;/strong&gt; &lt;/p&gt;&#xA;" OwnerUserId="51589" LastEditorUserId="51589" LastEditDate="2018-05-04T05:50:46.620" LastActivityDate="2018-05-09T12:32:59.537" Title="why is mse training drastically different from the begining of each training with Encoder-Decoder" Tags="|machine-learning|neural-network|deep-learning|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31187" PostTypeId="2" ParentId="31166" CreationDate="2018-05-04T00:06:28.187" Score="0" Body="&lt;p&gt;There's a couple of approaches you can take depending on the nature of your data. It sounds like you're trying to detect social anomalies in your data so you need to model the communication boundaries between them which leads to some sort of graph representation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have too many users in the system (say $n$) then you can create an $n\times n$ matrix, $M$ over a time period that represents the communications between users. The component $M_{ij}$ could be either $1$ or $0$ if user $i$ and $j$ communicated or the number of times that they communicated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have more data then you would want to represent the data in terms of nodes and edges. Nodes would be the users and edges would be the presence of a communication. This can be done manually or by using a library such as &lt;a href=&quot;https://networkx.github.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;NetworkX&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/crailtap/basic-network-analysis-tutorial&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here's a Python tutorial on getting started with graph network analysis&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're doing this at a large scale then you might want to use a graph database such as &lt;a href=&quot;https://neo4j.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Neo4J&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="51591" LastActivityDate="2018-05-04T00:06:28.187" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31188" PostTypeId="1" AcceptedAnswerId="31190" CreationDate="2018-05-04T00:13:39.000" Score="0" ViewCount="46" Body="&lt;p&gt;I'm trying to understand the theoretical reasoning behind the method, but I can't understand a particular step in the middle of this &lt;a href=&quot;https://www.itl.nist.gov/div898/handbook/pmc/section5/pmc551.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;page&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;The constraint on the numbers in $v_1$ is that the sum of the squares of the coefficients equals $1$. Expressed mathematically, we wish to maximize&#xA;$$\frac1N\sum_{i=1}^NY_{1i}^2$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where&#xA;$$y_{1i}=v_1'z_i,$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and $$v_1'v_1=1$$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(this is called &quot;normalizing&quot; $v_1$).&#xA;Computation of first principal component from $R$ and $v_1$. Substituting the middle equation in the first yields&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac1N\sum_{i=1}^NY_{1i}^2=v_1'Rv_1.&quot;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't understand how $R$ suddenly appeared in this equation. The right hand side &quot;$v_1' R v_1$&quot; seems to have appeared out of nowhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help&lt;/p&gt;&#xA;" OwnerUserId="51507" LastEditorUserId="38892" LastEditDate="2018-05-04T01:34:42.500" LastActivityDate="2018-05-04T01:34:42.500" Title="how to derive the steps of principal component analysis?" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31189" PostTypeId="2" ParentId="31167" CreationDate="2018-05-04T00:53:14.820" Score="2" Body="&lt;p&gt;Having problems with dimensions is very common. As others have mentioned, the method &lt;code&gt;predict&lt;/code&gt; expects to get a batch of images. You should run &lt;code&gt;model.summary()&lt;/code&gt; to see what the expected dimensions of the input are. The batch size itself might have been designed to be flexible during training, by specifying &lt;code&gt;None&lt;/code&gt; in the first dimension on the model &lt;code&gt;input_shape&lt;/code&gt; parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing more of the dimensions, I cannot say exactly where your problems lies. You could include that information by editing your question.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It will likely help to have a look at the &lt;a href=&quot;https://keras.io/models/sequential/#predict&quot; rel=&quot;nofollow noreferrer&quot;&gt;documentation&lt;/a&gt; for predict function of the model objects.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is an argument: &lt;code&gt;batch_size&lt;/code&gt;, which defaults to &lt;strong&gt;32&lt;/strong&gt; if not fixed by the model itself, which you can see from the &lt;code&gt;model.summary()&lt;/code&gt;. If you set this equal to 1, perhaps you will get a prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a modified version of your code that I would expect to return a prediction.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import load_model&#xA;from keras.preprocessing import image&#xA;import numpy as np&#xA;&#xA;# dimensions of our images    -----   are these then grayscale (black and white)?&#xA;img_width, img_height = 313, 220&#xA;&#xA;# load the model we saved&#xA;model = load_model('model.h5')&#xA;&#xA;# Get test image ready&#xA;test_image = image.load_img('/Images/1.jpg', target_size=(img_width, img_height))&#xA;test_image = image.img_to_array(test_image)&#xA;test_image = np.expand_dims(test_image, axis=0)&#xA;&#xA;test_image = test_image.reshape(img_width, img_height*3)    # Ambiguity!&#xA;# Should this instead be: test_image.reshape(img_width, img_height, 3) ??&#xA;&#xA;result = model.predict(test_image, batch_size=1)&#xA;print result&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;: there is ambiguity in the dimensions where I highlight it. A colour image will have three dimensions: (height, width, 3). Black and white will have only two dimensions: (height, width).&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-04T00:53:14.820" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31190" PostTypeId="2" ParentId="31188" CreationDate="2018-05-04T00:56:41.470" Score="2" Body="&lt;p&gt;\begin{align}&#xA;\frac1N\sum_{i=1}^N Y_{1i}^2 &amp;amp;= \frac1N\sum_{i=1}^N (v_1'z_i)(z_i'v_1)\\&#xA;&amp;amp;=v_1'\left(\frac1N \sum_{i=1}^N z_iz_i' \right) v_1 \\&#xA;&amp;amp;= v_1'Rv_1&#xA;\end{align}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $R=\frac1N \sum_{i=1}^N z_iz_i'.$&lt;/p&gt;&#xA;" OwnerUserId="38892" LastActivityDate="2018-05-04T00:56:41.470" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="31192" PostTypeId="2" ParentId="31186" CreationDate="2018-05-04T01:30:36.937" Score="0" Body="&lt;p&gt;Just a couple of ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Batch size&lt;/strong&gt;: 8 is quite a small batch, meaning the average loss that is computed might have high volatility. If you have enough memory, you could increase this.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Diversity of input&lt;/strong&gt;: try adding batch normalisation layers in the encoder part, to try smoother the input for the conv layers. You said there are quite a few features, so perhaps this makes for &lt;em&gt;noisy&lt;/em&gt; input, which would benefit from being normalised.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You could trying running the same experiment for 15 epochs, then plotting the training and validation losses (as they evolve perhaps, using the &lt;a href=&quot;https://keras.io/callbacks/#tensorboard&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorBoard callback&lt;/a&gt; alongside your others). Do they follow any patterns or converge after some time?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could try using different initilisation methods, or even gradient clipping, in order to make training a little smoother - constraining the size of the updates to weights during backpropagation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, another &lt;a href=&quot;https://arxiv.org/abs/1710.10196&quot; rel=&quot;nofollow noreferrer&quot;&gt;(brand-new!) result from research&lt;/a&gt; into GAN models, shows that progressively increasing the size of the inputs to your models might help to smooth learning and also extract a more robust set of features, which generalise better. Have a read of &lt;a href=&quot;http://www.fast.ai/2018/04/30/dawnbench-fastai/#imagenet&quot; rel=&quot;nofollow noreferrer&quot;&gt;this section of an article&lt;/a&gt; from FastAI on their experience.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;EDIT: information regarding the ELU activation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Using this activation my help learning, as it has a little more &lt;em&gt;flexibility&lt;/em&gt; than e.g. the ELU, because it may also assume negative values. Here is an image from &lt;a href=&quot;https://arxiv.org/pdf/1511.07289v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;the original paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mpaGf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mpaGf.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the official definition (might slightly differ in the implementation of your framework):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UKZZr.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UKZZr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The authors mentions that this activation assists in pushing the mean activation closer to zero, just as batch normalisation does. In your case this might mean simply getting past the bumpy initial epochs and converging a little more quickly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final major point that the authors highlight is that the models they trained also generalised better - so you might enjoy an improved performance with your model using ELUs versus a model using ReLUs (assuming both are trained for similar time).&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-09T12:32:59.537" LastActivityDate="2018-05-09T12:32:59.537" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31193" PostTypeId="2" ParentId="31170" CreationDate="2018-05-04T02:18:24.650" Score="0" Body="&lt;p&gt;I suggest you to hand run a computation graph for the given NN. Then Try to derive backpropogation equations and check the dimensions of each gradients.&lt;/p&gt;&#xA;" OwnerUserId="51595" LastActivityDate="2018-05-04T02:18:24.650" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31194" PostTypeId="1" AcceptedAnswerId="31348" CreationDate="2018-05-04T02:31:32.747" Score="1" ViewCount="3718" Body="&lt;p&gt;I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My training datasets are&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;X: 2-D Cartesian coordinate as input data&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y: radio signal strength (RSS) at the 2-D coordinates points as observed output&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I've done so far&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've installed python and Scikit-learn software. I've successfully tested the sample codes. I'm able to predict RSS at test points using training data. I use squared exponential Kernel.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I want to do&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to train the Kernel parameters (hyper-parameter) with different optimizing algorithms like gradient descent, swarm intelligence, and trust-region-reflective algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What I learned and What help I am asking for&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've learned that, in the GaussianProcessRegressor class of scikit, the optimizer is an argument where I can use my own optimizing algorithm. Since it is callable, I need to write my own function/method for it. Can I use any inbuilt library (library of optimization algorithm) in GaussianProcessRegressor class? Are there such libraries available for python? Could anybody provide any sample code for using kernel parameter optimization algorithm in GaussianProcessRegressor? I've learned that we use the training datasets for optimizing the hyper-parameters. Could anybody provide any insight about relating the training datasets with the optimization algorithm, please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="50881" LastActivityDate="2018-05-08T03:05:42.923" Title="Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn" Tags="|scikit-learn|regression|supervised-learning|hyperparameter|hyperparameter-tuning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31195" PostTypeId="1" CreationDate="2018-05-04T03:54:10.130" Score="1" ViewCount="40" Body="&lt;p&gt;Good day. I know how thin plate spline is used for smoothing splines since I've been using it personally in my research field. I am just totally confused why lattice of knots (red and blue) are used in this figure with a contour plot. What this image really wants to expound to us? Any ideas folks?&lt;a href=&quot;https://i.stack.imgur.com/iq4wN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iq4wN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51495" LastActivityDate="2018-05-04T03:54:10.130" Title="What This Systolic Blood Pressure Wants to Explain To Us?" Tags="|machine-learning|" AnswerCount="0" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="31196" PostTypeId="1" CreationDate="2018-05-04T05:13:53.950" Score="0" ViewCount="20" Body="&lt;p&gt;I have a dataset of computer systems with system characteristics as parameters. I want to rank best systems based on these parameters. The system works on a recurrent neural network model with the inputs of all previous transactions considered while choosing the current output. To my understanding, LSTM is something that is commonly used for such usecases but I heard it has limitations when I want to dynamically add new systems.  Are there any better alternatives?&lt;/p&gt;&#xA;" OwnerUserId="51601" LastActivityDate="2018-05-04T05:13:53.950" Title="ANN algorithm for system selection" Tags="|neural-network|rnn|lstm|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31198" PostTypeId="1" AcceptedAnswerId="31202" CreationDate="2018-05-04T06:35:23.377" Score="0" ViewCount="118" Body="&lt;p&gt;Why is it a problem that your response variable is skewed in regression? Is taking logarithms the only way to solve it?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-04T07:16:17.817" Title="Skewness in response variable" Tags="|regression|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31199" PostTypeId="2" ParentId="24614" CreationDate="2018-05-04T06:46:35.453" Score="0" Body="&lt;p&gt;There is no direct relationship between LSTM_nums and timesteps. LSTM_nums would be the number of hidden neurons (output dimensionality/vector) whereas timesteps is how far back the model could remember its state (unfold).&lt;/p&gt;&#xA;" OwnerUserId="51604" LastActivityDate="2018-05-04T06:46:35.453" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31200" PostTypeId="2" ParentId="26172" CreationDate="2018-05-04T06:52:29.283" Score="1" Body="&lt;p&gt;In Keras, the first argument in LSTM gives the dimensionality of the cell state.&lt;/p&gt;&#xA;" OwnerUserId="51604" LastActivityDate="2018-05-04T06:52:29.283" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31201" PostTypeId="1" AcceptedAnswerId="31204" CreationDate="2018-05-04T07:02:07.820" Score="1" ViewCount="2431" Body="&lt;p&gt;I have a text file with information that needs to classified based on keywords. The text file contains many number of paragraphs. And the paragraph contains keywords that we want (lets say salary amount, interest rate and so on..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to write a model which will extract the paragraph (or 3 to 4 lines of text) containing the keyword i want. How do i create a label in this case? All i have is a raw text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am new to NLP. Any suggestions how i can approach this?&lt;/p&gt;&#xA;" OwnerUserId="32544" LastActivityDate="2018-05-04T08:48:01.100" Title="Creating labels for Text classification using keras" Tags="|python|deep-learning|nlp|keras|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31202" PostTypeId="2" ParentId="31198" CreationDate="2018-05-04T07:16:17.817" Score="1" Body="&lt;p&gt;Taking logarithms is not a one-size-fits-all approach - what if your response is negatively skewed?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Simple&quot; linear regression assumes that the response variable is normally distributed. If you'd like to perform this kind of regression, then you should probably transform your response variable to be as normal as possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are other approaches / model types to deal with skewed responses - check out the Tweedie family.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-05-04T07:16:17.817" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31204" PostTypeId="2" ParentId="31201" CreationDate="2018-05-04T08:48:01.100" Score="1" Body="&lt;p&gt;You can build the text classification application with CNN algorithm by Keras library.&#xA;Please take a look at this git repository. &lt;a href=&quot;https://github.com/leeyamkeng/CNN-text-classification-keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see, you need to create training and testing data by loading polarity data from files, splitting the data into words, generating labels and returning split sentences and labels.&#xA;And you can create the convolutional neural network with Dense, Embedding, Conv2D, MaxPool2D of keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the final model training snippet.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.layers import Input, Dense, Embedding, Conv2D, MaxPool2D&#xA;from keras.layers import Reshape, Flatten, Dropout, Concatenate&#xA;from keras.callbacks import ModelCheckpoint&#xA;from keras.optimizers import Adam&#xA;from keras.models import Model&#xA;from sklearn.model_selection import train_test_split&#xA;from data_helpers import load_data&#xA;&#xA;print('Loading data')&#xA;x, y, vocabulary, vocabulary_inv = load_data()&#xA;&#xA;# x.shape -&amp;gt; (10662, 56)&#xA;# y.shape -&amp;gt; (10662, 2)&#xA;# len(vocabulary) -&amp;gt; 18765&#xA;# len(vocabulary_inv) -&amp;gt; 18765&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split( x, y,     test_size=0.2, random_state=42)&#xA;&#xA;# X_train.shape -&amp;gt; (8529, 56)&#xA;# y_train.shape -&amp;gt; (8529, 2)&#xA;# X_test.shape -&amp;gt; (2133, 56)&#xA;# y_test.shape -&amp;gt; (2133, 2)&#xA;&#xA;&#xA;sequence_length = x.shape[1] # 56&#xA;vocabulary_size = len(vocabulary_inv) # 18765&#xA;embedding_dim = 256&#xA;filter_sizes = [3,4,5]&#xA;num_filters = 512&#xA;drop = 0.5&#xA;&#xA;epochs = 100&#xA;batch_size = 30&#xA;&#xA;# this returns a tensor&#xA;print(&quot;Creating Model...&quot;)&#xA;inputs = Input(shape=(sequence_length,), dtype='int32')&#xA;embedding = Embedding(input_dim=vocabulary_size,     output_dim=embedding_dim, input_length=sequence_length)(inputs)&#xA;reshape = Reshape((sequence_length,embedding_dim,1))(embedding)&#xA;&#xA;conv_0 = Conv2D(num_filters, kernel_size=(filter_sizes[0],     embedding_dim), padding='valid', kernel_initializer='normal',     activation='relu')(reshape)&#xA;conv_1 = Conv2D(num_filters, kernel_size=(filter_sizes[1],     embedding_dim), padding='valid', kernel_initializer='normal',     activation='relu')(reshape)&#xA;conv_2 = Conv2D(num_filters, kernel_size=(filter_sizes[2],     embedding_dim), padding='valid', kernel_initializer='normal', activation='relu')(reshape)&#xA;&#xA;maxpool_0 = MaxPool2D(pool_size=(sequence_length - filter_sizes[0] + 1, 1), strides=(1,1), padding='valid')(conv_0)&#xA;maxpool_1 = MaxPool2D(pool_size=(sequence_length - filter_sizes[1] + 1, 1), strides=(1,1), padding='valid')(conv_1)&#xA;maxpool_2 = MaxPool2D(pool_size=(sequence_length - filter_sizes[2] + 1, 1), strides=(1,1), padding='valid')(conv_2)&#xA;&#xA;concatenated_tensor = Concatenate(axis=1)([maxpool_0, maxpool_1, maxpool_2])&#xA;flatten = Flatten()(concatenated_tensor)&#xA;dropout = Dropout(drop)(flatten)&#xA;output = Dense(units=2, activation='softmax')(dropout)&#xA;&#xA;# this creates a model that includes&#xA;model = Model(inputs=inputs, outputs=output)&#xA;&#xA;checkpoint = ModelCheckpoint('weights.{epoch:03d}-{val_acc:.4f}.hdf5',     monitor='val_acc', verbose=1, save_best_only=True, mode='auto')&#xA;adam = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)&#xA;&#xA;model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])&#xA;print(&quot;Traning Model...&quot;)&#xA;model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, callbacks=[checkpoint], validation_data=(X_test, y_test))  #     starts training&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;By running this code, you will get the trained model with the format of hd5. Finally, you can use your model for prediction.&lt;/p&gt;&#xA;" OwnerUserId="50292" LastActivityDate="2018-05-04T08:48:01.100" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31205" PostTypeId="2" ParentId="30513" CreationDate="2018-05-04T09:29:37.640" Score="0" Body="&lt;p&gt;Use &lt;strong&gt;LSA (Latent Semantic Analysis) Algorithm&lt;/strong&gt; for semantic meaning similarity.. It will be useful for your requirement&lt;/p&gt;&#xA;" OwnerUserId="32776" LastActivityDate="2018-05-04T09:29:37.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31206" PostTypeId="1" CreationDate="2018-05-04T09:31:25.070" Score="0" ViewCount="310" Body="&lt;p&gt;I test the &lt;code&gt;Sagemaker AWS&lt;/code&gt; solution for RNN: &lt;a href=&quot;https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;deepAR&lt;/a&gt;.&#xA;Previously I used sklearn for this and obviously I cleaned the data to avoid highly correlated time series (KBest, PCA, etc.)&#xA;Do you think it is needed here, or RNN will handle internally correlated series.&#xA;PS. This is typically the case with blackbox commercial solutions. They look fantastic, but when it comes to details you have nowhere to consult.&lt;/p&gt;&#xA;" OwnerUserId="47522" LastActivityDate="2018-07-03T12:49:23.223" Title="deepAR RNN from AWS Sagemaker - should I clean the data first?" Tags="|rnn|pca|aws|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31207" PostTypeId="2" ParentId="18982" CreationDate="2018-05-04T10:13:28.420" Score="1" Body="&lt;p&gt;As noted by @Emre in the comments:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;Same thing. Just call it classification; it's the more common name for the task.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although there is no difference with respect to machine learning, &lt;a href=&quot;https://pdfs.semanticscholar.org/774e/ab27b22aa92dfaa9aeeeafbe845058e85f58.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt; may provide another perspective in detail.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastActivityDate="2018-05-04T10:13:28.420" CommentCount="0" CommunityOwnedDate="2018-05-04T10:13:28.420" ContentLicense="CC BY-SA 4.0" />
  <row Id="31208" PostTypeId="2" ParentId="31206" CreationDate="2018-05-04T10:23:57.523" Score="0" Body="&lt;p&gt;I am not a specialist but I would say that you only have to use the right format for your data. Since NNs have the ability to handle high dimensional features it will not harm your model if those features are correlated.&lt;/p&gt;&#xA;" OwnerUserId="51541" LastActivityDate="2018-05-04T10:23:57.523" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31210" PostTypeId="2" ParentId="31131" CreationDate="2018-05-04T12:18:22.487" Score="1" Body="&lt;p&gt;A generalized linear model (glm) appeared to be work reasonable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;glm(y ~ x * c, data=df) and extensions hereof. &lt;/p&gt;&#xA;" OwnerUserId="39907" LastActivityDate="2018-05-04T12:18:22.487" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31211" PostTypeId="1" CreationDate="2018-05-04T13:08:49.697" Score="1" ViewCount="1228" Body="&lt;p&gt;So I have google Team drive set-up with all the permissions.&#xA;1. How I can access google team drive in google colaboratory? I think i'm missing something... it's always showing me output of my drive&#xA;2. I want to download all the &quot;data&quot; from google team drive to my google colab notebook is it possible to make it automatically?&lt;/p&gt;&#xA;" OwnerUserId="44574" LastActivityDate="2018-05-04T13:08:49.697" Title="Google team drive and google colaboratory uploading/downloading files" Tags="|colab|" AnswerCount="0" CommentCount="3" ClosedDate="2018-05-06T21:07:52.857" ContentLicense="CC BY-SA 4.0" />
  <row Id="31212" PostTypeId="1" CreationDate="2018-05-04T13:15:12.693" Score="3" ViewCount="780" Body="&lt;p&gt;Let's make an example: I want to build a neural network which should predict if a person is obese. It gets the following features:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Weight&lt;/li&gt;&#xA;&lt;li&gt;Height&lt;/li&gt;&#xA;&lt;li&gt;Number of hours the person makes sports&lt;/li&gt;&#xA;&lt;li&gt;superspecial health index&lt;/li&gt;&#xA;&lt;li&gt;foobar&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The output of the model is a probability that the given person is obese. The higher, the more likely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I know, if everything else stays the same, that a higher weight should always be the same or a higher output probability. But I don't know the exact relationship, only that it changes monothonically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I include this knowledge into the network and force it to show the desired behaviour?&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-05-04T14:52:44.440" Title="How can I include prior knowledge about monotonic features into a neural network?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31213" PostTypeId="2" ParentId="31175" CreationDate="2018-05-04T13:43:15.903" Score="1" Body="&lt;p&gt;I think the confusion with the Inception module is the somewhat complicated structure. The point on the relevant &lt;a href=&quot;http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture9.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;CS231n slide (#37)&lt;/a&gt;, saying there are no FC layers is partially correct. (Remember this is only a summary of the model to get the main points across!).  In the actual part of the model being explained on that slide, they are referring only to the Inception modules:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BVDcs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BVDcs.png&quot; alt=&quot;Inception module&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No FC layers!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Definitions will, however, play a big role in deciding whether or not there are FC layers in the model.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;In the bigger scheme of things (beyond a single Inception module), we have first to distinguish between the train and test time architectures. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At train time there are auxilliary branches, which do indeed have a few fully connected layers. These are used to force intermediate layers (or inception modules) to be more aggressive in their quest for a final answer, or in the words of the authors, to be more &lt;em&gt;discriminate&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the paper (page 6 &lt;a href=&quot;https://arxiv.org/pdf/1409.4842.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;[Szegedy et al., 2014]&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;One interesting insight is that the strong performance&#xA;  of relatively shallower networks on this task suggests that the features produced by the layers in the&#xA;  middle of the network should be very discriminative. By adding auxiliary classifiers connected to&#xA;  these intermediate layers, we would expect to encourage discrimination in the lower stages in the&#xA;  classifier, increase the gradient signal that gets propagated back, and provide additional regularization.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The slice of the model shown below displays one of the auxilliary classifiers (branches) on the right of the inception module:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/G5ItL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/G5ItL.png&quot; alt=&quot;Inception block with auxilliary classifier&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This branch clearly has a few FC layers, the first of which is likely followed by a non-linearity such as a &lt;code&gt;ReLU&lt;/code&gt; or &lt;code&gt;tanh&lt;/code&gt;. The second one simply squishes the 1000 input weights into whatever number of classes are to be predicted (coincidentally or not, this is a 1000 here for ImageNet).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, at test time, these branches are not active. They were used simply to train the weights of the modules, but do not contribute to the final classification probabilities produces at the end of the entire model architecture.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This all leaves us with just the suspsicious looking block right at the end of the model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/IzOkd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/IzOkd.png&quot; alt=&quot;Final layers after last Inception module, before class probabilities are produced.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is clearly a big blue FC layer there!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is where definitions come into play. It is somewhat subjective.&#xA;Is a fully connected layer one in which each $m$ weight is connected to each of $n$ nodes? Is it a layer in which representation are learned, and if so, does the layer require a non-linearity?  We know that neural networks requires the non-linearities, such as ReLU and tanh functions to be applied to the outputs of a layer (thinking in forward flow). Without these, neural networks would simply be a combinations of linear functions, and so going deeper wouldn't theoretically add any power as we essentially would just be performing a huge linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this spirit, we can look at the final piece of the puzzle, and point out that tis final FC layer is noted to simply be linear! That is, it takes all the weights resulting from the preceding &lt;code&gt;Average Pooling&lt;/code&gt; layer, and combines them into a &lt;em&gt;linear combination&lt;/em&gt; of only 1000 values - ready for the softmax. This can all be understood from the tabular overview of the network architecture:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FukvF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FukvF.png&quot; alt=&quot;Start and end sections of the tabular model overview&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, do you agree with the stanford guys or not? I do!&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-04T13:43:15.903" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31214" PostTypeId="2" ParentId="31179" CreationDate="2018-05-04T13:59:57.173" Score="2" Body="&lt;p&gt;It sounds like a use case for stochastic optimization algorithms: since your reward function is not observed but highly correlated in time, I would expect stochastic optimization algorithms to converge fast to the maximum of the reward function and follow it closely as it slowly changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Because of this, I expect that you would obtain pretty good results with simple stochastic optimization algorithms without going to complex models such as Reinforcement Learning. Also, it's always good to start easy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: whether stochastic optimization algorithms can converge to the maximum of the reward function depends on how fast the reward function changes.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edit, after request for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would start with &lt;a href=&quot;https://en.wikipedia.org/wiki/Random_search&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Search&lt;/a&gt;.&#xA;Basically the idea is to&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;pick a position at random&lt;/li&gt;&#xA;&lt;li&gt;pick one new position at random&lt;/li&gt;&#xA;&lt;li&gt;select the position with highest reward&lt;/li&gt;&#xA;&lt;li&gt;repeat steps 2-3&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The trick is in generating good candidate points (step 2) in order to speed up convergence to the maximum. I have a few basic ideas that can help:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;pick a random position by drawing from a random distribution (say gaussian) with mean equal to the current position and standard deviation somehow inversely proportional to the current reward $F_t$ (something like $\alpha e^{(-\beta F_t)}$) so that if we are close to the maximum of the reward function we pick a new candidate close to the current position, and if we have a low reward we can explore farther away from the current position. These two strategies are usually called &quot;exploitation&quot; and &quot;exploration&quot;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;the idea above has the disadvantage of being &quot;memory-less&quot;, which makes it so that it doesn't take into account the gradient. In facts, drawing from a gaussian distribution can either push you closer or away from the maximum with equal chance, being the gaussian symmetric around the mean. A possible workaround would be to draw the new random candidate position from a &lt;strong&gt;skew normal&lt;/strong&gt; distribution, with skewness parameter somehow dependent on the sign of the gradient of the reward function at the previous step.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think that in your case you don't need much fancier algorithms since - if I understood correctly - your reward function is not multimodal thus we do not risk getting stuck in a local maximum.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2018-05-18T08:01:19.933" LastActivityDate="2018-05-18T08:01:19.933" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31215" PostTypeId="2" ParentId="31212" CreationDate="2018-05-04T14:22:26.233" Score="1" Body="&lt;p&gt;If you plug this kind of data into a standard network, e.g. an MLP, you will usually hope that the model actually extracts this information itself. You could introduce a dummy variable that encodes this information, but you run the risk that the model learns to just follow the dummy variable and doesn't learn its own powerful abstractions and feastures from the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;[EDITED: ]&lt;/strong&gt; An example could be to create a weight based variable that is normalised to other physical characteristics, such as &lt;code&gt;DV = weight / (height + waist circumfrence)&lt;/code&gt;. This should then, according to your assumptions, scale nicely with the output obesity.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Another way that people might include &lt;em&gt;prior&lt;/em&gt; information into a model, is to use a Probabilistic modelling, which incorporates ideas from Bayesian statistics. You can do things such as define a prior distribution for your outputs, conditioned on your inputs - essentially allowing you to provide information to the model (such as weight being correlated with obesity, &lt;em&gt;ceteris paribus&lt;/em&gt;) - this then nudges the model to go along these lines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you would like to get into it, there are already some great libraries to make it really easy:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://mc-stan.org/users/interfaces/index.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stan&lt;/a&gt; - which has interfaces to many languages: Python, R, State etc.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/blei-lab/edward&quot; rel=&quot;nofollow noreferrer&quot;&gt;Edward&lt;/a&gt; - probabilistic programming with modern GPU accelration and deep learning integration (Tensorflow and Keras).&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://edwardlib.org/tutorials/supervised-regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Linear regression example&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://edwardlib.org/tutorials/bayesian-neural-network&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Bayesian Neural Network examples&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1801.07710.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This&lt;/a&gt; seems to be a nice overview of some of the methods and tools, but I haven't read through it all.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Another way that I could suggest is to play with the architecture of your model and the idea of &lt;strong&gt;auxilliary models&lt;/strong&gt;. Have a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/31213/45264&quot;&gt;my recent answer&lt;/a&gt; on a question talking about the Inception model from &lt;a href=&quot;https://arxiv.org/pdf/1409.4842.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Szegedy et al.&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that you have branches coming off the model at train time only, which also make predictions and produce error to be backpropagated through the preceding weights.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could make a side model that predicts the obesity, based on e.g. the input weights and perhaps some related features extracted from the first layer of your neural network. This would make the idea or importance of this relationship more prominent during training and the weights would subsequently be tuned to take that into account.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At test time, you simply ignore these auxilliary branches.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-04T14:52:44.440" LastActivityDate="2018-05-04T14:52:44.440" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31216" PostTypeId="1" AcceptedAnswerId="31241" CreationDate="2018-05-04T14:54:27.380" Score="4" ViewCount="115" Body="&lt;p&gt;I intend on monetising some large datasets. These datasets are anonymised and released to (paying) clients via a web api. Are there any standard algorithms such that if the datasets are intentionally leaked publicly, the data can be altered such that the responsible party can be identified, while at the same time the data remains practically useful?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are certain approaches which come to mind, such as every client's data being very slightly different with known changes. For example in spatial data, every lon/lat pair is altered by the same very small vector. My worry is that if the data is anonymised again by the client before being leaked, a naive attempt might easily be circumvented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(I am not a data scientist so I'm not really sure what the correct jargon is for what I am looking for)&lt;/p&gt;&#xA;" OwnerUserId="51621" LastActivityDate="2018-05-05T06:49:35.513" Title="How to release datasets with fingerprinting" Tags="|data|anonymization|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31218" PostTypeId="2" ParentId="30559" CreationDate="2018-05-04T15:24:30.247" Score="2" Body="&lt;p&gt;I think what you are trying to ask is whether your variables are important or not. There is no thing as the variables are overfitting to the model, however you can overfit to the model by tuning and minimizing bias to the very end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In models such as Decision Trees and Multiple Decision Trees Ensembles (Random Forest), you can calculate variable importance by using entropy measure of the variables. This will lead you to understand which variables are actually important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, in models such as random forest the intuition is to have multiple decision trees where at each splitting node having different variables across many trees. Most of the time you would not even use all the variables you have in your data set in a single tree. This enable to decrease your model's variance while not affecting your bias.&lt;/p&gt;&#xA;" OwnerUserId="44827" LastActivityDate="2018-05-04T15:24:30.247" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31219" PostTypeId="1" CreationDate="2018-05-04T15:41:06.930" Score="3" ViewCount="78" Body="&lt;p&gt;Please forgive the potential ignorance of this question. I am not a complete virgin regarding NLP:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to assign parts of speech to individual words. I realize that many words can serve as multiple parts of speech, but I am looking for the POS that a human would be most likely to interpret the word as, seeing it out of context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any database that has this info in it for a reasonably large set of words? Or, is there any large corpus with POS tags where I could feed it  words, and get frequency distributions of the POS that each word is playing in the corpus.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="51623" LastActivityDate="2022-10-21T12:05:00.607" Title="Identifying parts of speech for individual words (not texts)" Tags="|nlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31220" PostTypeId="1" CreationDate="2018-05-04T16:11:45.023" Score="1" ViewCount="212" Body="&lt;p&gt;A have a data frame with shops with a number of independent variables (all categorical), I'm trying to find a decent model to predict a dependent variable that is ordered and categorical (ie volume of foot traffic: &quot;low&quot;, &quot;medium&quot;, &quot;high&quot;). The dependent variable is defined for about 20% of the observations, and I want to predict for the missing 80%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What models/techniques would you suggest?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For now, I started by learning some &lt;strong&gt;association rules&lt;/strong&gt; with &lt;code&gt;arules&lt;/code&gt; (&lt;a href=&quot;https://cran.r-project.org/web/packages/arules/vignettes/arules.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;manual here&lt;/a&gt;):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;require(arules)&#xA;train # my data frame with the complete observations&#xA;target_field = 'foot.traffic'&#xA;# filter rules to observe only the relevant ones&#xA;appearance_list = list(rhs=paste0(target_field,&quot;=&quot;, unique(train[,target_field])))&#xA;&#xA;&#xA;rules &amp;lt;- arules::apriori(train,&#xA;            parameter = list( support=0.1, confidence=0.2 ),&#xA;            appearance = appearance_list)&#xA;rules &amp;lt;- sort(rules, by=&quot;lift&quot;)&#xA;inspect(rules)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I can inspect these rules, and they generally make sense. They mostly refer to &lt;code&gt;low&lt;/code&gt; and &lt;code&gt;high&lt;/code&gt; cases, indicating that rules for &lt;code&gt;medium&lt;/code&gt; are harder to identify.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I apply them back to the training set to see how well they predict the &lt;code&gt;foot.traffic&lt;/code&gt; variable? Is there a good off-the-shelf strategy to apply the rules?&lt;/p&gt;&#xA;" OwnerUserId="51624" LastActivityDate="2018-05-04T16:11:45.023" Title="Applying association rules for variable prediction in R" Tags="|r|association-rules|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31223" PostTypeId="1" CreationDate="2018-05-04T17:22:38.973" Score="0" ViewCount="211" Body="&lt;p&gt;I am training object detection using CNN&lt;/p&gt;&#xA;&lt;p&gt;I am using MSE as loss functions, at best it throws loss around 1800 on testing and 1850 on training data, Which in my opinion is quite acceptable, considering that if the bounding box is &amp;quot;dislocated&amp;quot; by 20px on every axis, its already 1600 ( before mean ).&lt;/p&gt;&#xA;&lt;p&gt;However, when the network on the exact data i have trained it on, it result in extremely wrong results, almost like random no way it has around 1800 mean difference.&lt;/p&gt;&#xA;&lt;p&gt;What would general approach be for dealing with things like this? Thanks for help.&lt;/p&gt;&#xA;" OwnerUserId="40162" LastEditorUserId="1330" LastEditDate="2021-09-19T22:18:22.643" LastActivityDate="2021-09-19T22:18:22.643" Title="Low MSE loss for object detection but really bad results" Tags="|neural-network|convolution|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31225" PostTypeId="2" ParentId="13438" CreationDate="2018-05-04T17:47:43.187" Score="1" Body="&lt;p&gt;The labels for small ImageNet dataset are provided in the same manner as the labels for the original (large) dataset. See here: &lt;a href=&quot;https://stackoverflow.com/questions/40744700/how-can-i-find-imagenet-data-labels&quot;&gt;https://stackoverflow.com/questions/40744700/how-can-i-find-imagenet-data-labels&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="24724" LastActivityDate="2018-05-04T17:47:43.187" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31227" PostTypeId="2" ParentId="31219" CreationDate="2018-05-04T18:55:13.767" Score="0" Body="&lt;p&gt;A POS tagged dataset that might help: &lt;a href=&quot;https://www.clips.uantwerpen.be/conll2000/chunking/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-04T18:55:13.767" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31229" PostTypeId="1" CreationDate="2018-05-04T20:23:37.040" Score="1" ViewCount="113" Body="&lt;p&gt;I have a dataset containing a set of normal user sessions. Each session contains a suite of ordered user requests on N system resources {R1, ..., RN}.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0wY9A.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0wY9A.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to design a continuous authentication algorithm, by confirming the user identity at each request command. More precisely, I don’t to let the user complete the whole session (all commands sequence) to authenticate him, but I want to do this at each resource requested command based on his previous normal sequences in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I build a model over normal sequences to detect anomalous ones in Resource-by-Resource fashion ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a lot of key concepts (Sequential Pattern Mining, Sequence Anomaly Detection, …), but I am completely confused about these concepts&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know how to begin and which ML algorithm(s) I should use.&lt;/p&gt;&#xA;" OwnerUserId="46174" LastEditorUserId="46174" LastEditDate="2018-05-05T15:31:31.840" LastActivityDate="2021-09-19T21:54:11.653" Title="Detection of Anomalous Sequential User Behavior" Tags="|anomaly-detection|sequential-pattern-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31230" PostTypeId="1" CreationDate="2018-05-04T20:35:13.367" Score="1" ViewCount="17" Body="&lt;p&gt;I've created a hybrid model by taking an existing decision engine (&lt;code&gt;TRUE&lt;/code&gt;/&lt;code&gt;FALSE&lt;/code&gt; output) and supplementing it with a random forest classifier (&lt;code&gt;TRUE&lt;/code&gt;/&lt;code&gt;FALSE&lt;/code&gt;) model. The output of the hybrid model is produced by computing an &lt;code&gt;OR&lt;/code&gt; from the predictions of the two models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When evaluating the performance on historical data, I can produce a ROC for the random forest by computing the class probabilities based on the votes from all of the random trees combined. I cannot do this for the existing decision engine because it does not output probabilities. For the purpose of this analysis, I need to treat it as a black box.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;How can I compute an ROC for the hybrid model?&lt;/strong&gt; I would like to do this with &lt;code&gt;R&lt;/code&gt;. I'm imagining that for each point on the ROC of the first model, I would need to compute the lift generated by adding the existing decision engine model.&lt;/p&gt;&#xA;" OwnerUserId="39233" LastActivityDate="2018-05-04T20:35:13.367" Title="Compute an ROC for a hybrid model where only one of the model components computes class probabilities" Tags="|random-forest|confusion-matrix|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31231" PostTypeId="2" ParentId="31156" CreationDate="2018-05-04T20:40:56.313" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Stackoverflow has a jobs page. &lt;a href=&quot;https://stackoverflow.com/jobs?sort=i&amp;amp;q=data&amp;amp;r=true&quot;&gt;This link&lt;/a&gt; is to a search with keyword &lt;code&gt;data&lt;/code&gt; and option &lt;code&gt;Remote&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://freelandr.com/?search=data&amp;amp;city=&quot; rel=&quot;nofollow noreferrer&quot;&gt;freelandr&lt;/a&gt; offers you to `Land your next Freelance Job'.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Here are two specialised job agencies, although I don't know if they relay freelance opportunities.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;We are well aware of the data scientist being the sexiest job of the 21st century. &lt;a href=&quot;https://jamieai.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;JamieAi&lt;/a&gt; harnessing this fact connects technical talent with data-oriented jobs organizations of all types and sizes. The start-up firm has combined AI insights and human oversight to reduce hiring costs and eliminate bias.  Also, third party recruitment agencies are removed from the process to boost transparency and efficiency in the path to employment. Another example is &lt;a href=&quot;https://woo.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Woo.io&lt;/a&gt;, a marketplace for matching tech professionals and companies.&#xA;  &lt;a href=&quot;https://hub.packtpub.com/25-startups-machine-learning-differently-2018/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://hub.packtpub.com/25-startups-machine-learning-differently-2018/&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.r-users.com/job-type/freelance/&quot; rel=&quot;nofollow noreferrer&quot;&gt;R-users&lt;/a&gt; specialises in the language R.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="47856" LastEditorUserId="47856" LastEditDate="2018-05-15T08:11:34.987" LastActivityDate="2018-05-15T08:11:34.987" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31232" PostTypeId="1" AcceptedAnswerId="31235" CreationDate="2018-05-04T20:47:17.730" Score="2" ViewCount="3249" Body="&lt;p&gt;In sklearn I'm normalizing the data with MinMaxScaler. The example I'm following uses &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.preprocessing import MinMaxScaler&#xA;scaler = MinMaxScaler()&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X_crime, y_crime,random_state = 0)&#xA;&#xA;X_train_scaled = scaler.fit_transform(X_train)&#xA;X_test_scaled = scaler.transform(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I wonder why this is done separately on the &lt;code&gt;train&lt;/code&gt; and &lt;code&gt;test&lt;/code&gt; set, and not on the X_crime dataframe like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_crime_scaled = scaler.fit_transform(X_crime)&#xA;&#xA;X_train_scaled, X_test_scaled, y_train, y_test = train_test_split(X_crime_scaled, y_crime, random_state = 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the R-squared score is higher and with this option I know all my values are normalized between 0 and 1.&lt;/p&gt;&#xA;" OwnerUserId="47906" LastActivityDate="2018-05-04T21:25:04.540" Title="Why not use Scaler.fit_transform on total dataframe?" Tags="|normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31233" PostTypeId="2" ParentId="31232" CreationDate="2018-05-04T21:00:27.393" Score="3" Body="&lt;p&gt;The main reason for that is the assumption that we make that our model has never seen our test data or have any information about it. So if you run your scaler on the whole dataset, they are probably going to have different max and mins, can you see that? Then, our model &lt;strong&gt;will indeed&lt;/strong&gt; have information about our test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have further doubts, comment, I'm available to help you again.&lt;/p&gt;&#xA;" OwnerUserId="13370" LastActivityDate="2018-05-04T21:00:27.393" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31235" PostTypeId="2" ParentId="31232" CreationDate="2018-05-04T21:20:04.083" Score="5" Body="&lt;p&gt;If you would use the &lt;code&gt;scaler&lt;/code&gt; on the full dataset you would provide the algorithm with some information about the values in the test set that it would not have otherwise. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This additional information (&lt;a href=&quot;https://machinelearningmastery.com/data-leakage-machine-learning/&quot; rel=&quot;noreferrer&quot;&gt;&quot;data leakage&quot;&lt;/a&gt;) from your test set to your training set is problematic because it leads to unreliable performance estimates. It is therefore not surprising that you achieve a higher R-squared. Due to the data leakage, this R-squared might be overly optimistic because it could depend on the additional information that you introduced into the training set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is particularly true for the &lt;code&gt;MinMaxScaler&lt;/code&gt; because it is by definition very sensitive to outliers. The effect will probably less problematic if you use the &lt;code&gt;RobustScaler&lt;/code&gt; (or even the &lt;code&gt;StandardScaler&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same holds for other preprocessing steps like outlier removal, feature selection etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are worried that your training data does not adequately reflect the true distribution, you can fall back to a cross-validation approach with multiple folds so you can estimate the effect across multiple splits of the data. Again, remember to fit the &lt;code&gt;scaler&lt;/code&gt; on all the training folds and apply it to the test fold &lt;strong&gt;for every iteration&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastEditorUserId="23305" LastEditDate="2018-05-04T21:25:04.540" LastActivityDate="2018-05-04T21:25:04.540" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31237" PostTypeId="1" AcceptedAnswerId="31239" CreationDate="2018-05-05T04:19:52.633" Score="2" ViewCount="522" Body="&lt;p&gt;Trying to get my feet wet with machine learning on text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most common dataset I've seen in this space is the sms dataset with classes ham and spam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the most common and successful approach seems to be to model this as a binary classification problem and to use a multinomial naïve Bayes to solve it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I'm trying to understand why this is a binary classification problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that the spam category has some common features associated with it across the class - such as ads, offers , free discounts and so on.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But there's no definition for what is a ham class is there? The definition of ham is - everything other than spam.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So why is this a binary classification task? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more context - I'm trying to solve the problem of whether a news article belongs to the politics class or to the non-political class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose I have a labelled dataset of around 3000 samples in each class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The non political class is a mix of classes like sports , religion , science and technology and miscellaneous. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Will a binary classifier work better than an algorithm such as oneclassSVM where anything other than political news is an outlier ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some of the other algorithms that I can use to solve this problem? I have heard about PU learning but I haven't seen any implementations of algorithms in any machine learning libraries ( I'm working with python) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If any of you have experience doing class modelling on text. Please share your comments and insights&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="33428" LastActivityDate="2023-01-24T05:01:11.460" Title="Why is spam detection a classification problem and not a class modelling problem" Tags="|classification|text-mining|svm|naive-bayes-classifier|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31239" PostTypeId="2" ParentId="31237" CreationDate="2018-05-05T06:16:39.753" Score="0" Body="&lt;p&gt;Regarding the case of spam vs ham, you are right that the spam category has common features (words), whereas the ham category could have multiple sub-categories, each with distinct feature sets. However, these distinct features can also be used to label an instance as &quot;ham&quot;. Eg: If spam messages don't usually talk about sports scores, then the occurrence of the word &quot;score&quot;, could be used to classify a message as &quot;ham&quot;, even when it co-occurs with typical spam words such as &quot;ad&quot;, &quot;offer&quot;, etc. However, the one-class classifier cannot make use of such a mechanism.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a similar manner, specific features will help to identify a news article as belonging to &quot;sports&quot;, &quot;religion&quot;, etc, thereby making them easier to label as &quot;non-political&quot;. The miscellaneous class would be more diverse, however.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One-class SVMs are useful in cases where you want to detect novel instances, the kind of which you have not seen before, and hence cannot characterize in advance. Binary classification can be used when the understanding is that the available labeled dataset covers the typical kind of examples you would see while using the model for prediction.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastEditorUserId="26562" LastEditDate="2018-05-05T08:04:57.107" LastActivityDate="2018-05-05T08:04:57.107" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31241" PostTypeId="2" ParentId="31216" CreationDate="2018-05-05T06:49:35.513" Score="0" Body="&lt;p&gt;&quot;Digital watermarking&quot; is a set of techniques that might be useful in this context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the wikipedia &lt;a href=&quot;https://en.wikipedia.org/wiki/Digital_watermarking&quot; rel=&quot;nofollow noreferrer&quot;&gt;page&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Watermarking&quot; is the process of hiding digital information in a carrier signal... Digital watermarks may be used to verify the authenticity or integrity of the carrier signal or to &lt;strong&gt;show the identity of its owners&lt;/strong&gt;. [Emphasis added]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;To address your requirement, you would insert a unique watermark for each client that receives your data. Watermarking techniques address requirements such as robustness to modification and imperceptibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an example, this paper talks about watermarking numerical data: &quot;Watermarking Numerical Data in the Presence of Noise&quot;, &lt;a href=&quot;https://www.researchgate.net/profile/Jordi_Castella-Roca/publication/220354602_Watermarking_Numerical_Data_in_the_Presence_of_Noise/links/0912f508805fb61713000000.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-05T06:49:35.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31242" PostTypeId="2" ParentId="31125" CreationDate="2018-05-05T07:27:57.750" Score="1" Body="&lt;p&gt;A couple of spatio-temporal datasets are likely to exhibit this characteristic:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Transportation demand across different locations of a city over different hours of a day. While residential locations have high demand originating there in the morning hours, and low demand in the evening, the reverse is true of work locations. The demand pattern of these two location types will be negatively correlated, whereas that of two different residential locations (or two locations of the same type) will be positively correlated. From the point of view of hours, correlation between two morning hours will be positive, whereas between a morning and an evening hour will be negative.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Temperature pattern across days, for different cities across the world. Here, data for cities in the northern and southern hemisphere will be negatively correlated.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-05T07:27:57.750" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31243" PostTypeId="1" AcceptedAnswerId="31247" CreationDate="2018-05-05T08:39:13.257" Score="3" ViewCount="1605" Body="&lt;p&gt;I am looking for single-number evaluation method that can be used in multi-class classification tasks that take into account imbalanced data-sets. For instance, &lt;code&gt;ROC-AUC&lt;/code&gt; defined by binary classifiers, is a single-number and takes into account imbalanced data-sets. On the other hand, accuracy is single-number, defined for multi-class classifiers and does not take into account imbalanced data-sets. Finally, the &lt;em&gt;confusion matrix&lt;/em&gt; is defined for multi-class, takes that into account but is not single-number. Is there any evaluation method that satisfies the three conditions? &lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="28175" LastEditDate="2018-05-05T12:31:42.283" LastActivityDate="2018-05-08T07:04:20.983" Title="Evaluation methods for multi-class classification" Tags="|machine-learning|neural-network|deep-learning|multiclass-classification|model-evaluations|" AnswerCount="3" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31244" PostTypeId="1" AcceptedAnswerId="31245" CreationDate="2018-05-05T09:01:28.533" Score="2" ViewCount="3164" Body="&lt;p&gt;I do understand the concept of normalizing &amp;amp; scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;I train and validate my model with the normalized data (MinMaxScaler) and save my model.&lt;/li&gt;&#xA;&lt;li&gt;A new input data comes in and I want to use my saved model to make predictions.&lt;/li&gt;&#xA;&lt;li&gt;I no longer have access to the training/test data at this point. All I have is the new single row of input data.&lt;/li&gt;&#xA;&lt;li&gt;How will I normalize this single vector of input data so that it can be fed to my model?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The only normalization that I can think of simply linear transformation of data (e.g. simply map values from [40, 70] range to [-1, 1]). I need a normalization technique that doesn't depend on the full range of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="47144" LastActivityDate="2019-06-08T09:22:47.200" Title="How to Normalize &amp; Scale a Single Data Point" Tags="|machine-learning|normalization|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31245" PostTypeId="2" ParentId="31244" CreationDate="2018-05-05T09:48:26.843" Score="4" Body="&lt;p&gt;You need to normalise the input in the same way that the training data was normalised -- however, you don't need access to this training data during predictions of new data. If you have used a &lt;code&gt;MinMaxScaler&lt;/code&gt; for example, then you can re-use this to transform your new data point:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = MinMaxScaler()&#xA;X_normalised = scaler.fit_transform(X)&#xA;&#xA;# do other stuff here like train &amp;amp; validate your model&#xA;&#xA;# now we have a new test data point from somewhere, we scale&#xA;# it using the scaler we fitted on the training data&#xA;new_test_point_normalised = scaler.transform(new_test_point)&#xA;&#xA;# now we can classify it with our model!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can serialise the scaler to disk using &lt;code&gt;pickle&lt;/code&gt; and re-load it when predicting new data points if necessary (i.e., the model training and testing happen in two different scripts).&lt;/p&gt;&#xA;" OwnerUserId="26562" LastActivityDate="2018-05-05T09:48:26.843" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31247" PostTypeId="2" ParentId="31243" CreationDate="2018-05-05T10:36:53.460" Score="2" Body="&lt;p&gt;How about a weighted log-loss?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say we have $m$ classes $c_1, \dots, c_m$. We can give each class $c_i$ a weight $w_i$ which is inversely proportional to the percentage of the dataset that belongs to $c_i$. Then, the loss for some data set with actual classes $y = y_1, \dots, y_n$ and predictions $\hat{y} = \hat{y}_1, \dots, \hat{y}_n$ can be defined as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;\text{loss}(y, \hat{y}) = \frac{1}{mn} \sum_{j=1}^n\sum_{i=1}^m w_i {I}_{(y_j == i)}\text{log}(\hat{y}_j)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where ${I}_{(y_j == i)}$ is an indicator function which evaluates to 1 if $y_j == i$ and 0 otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One disadvantage is that it's not immediately obvious, given some value of the loss function, how good a particular value of the loss function is. However, it &lt;em&gt;is&lt;/em&gt; easy to compare two values (lower is better).&lt;/p&gt;&#xA;" OwnerUserId="26562" LastEditorUserId="26562" LastEditDate="2018-05-05T23:26:05.607" LastActivityDate="2018-05-05T23:26:05.607" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31248" PostTypeId="1" AcceptedAnswerId="31269" CreationDate="2018-05-05T10:38:48.163" Score="6" ViewCount="22870" Body="&lt;p&gt;I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the parameter initialisation.&#xA;   Batch size = 100&#xA;   Epochs = 30&#xA;   Number of hidden layers = 3&#xA;   Nodes in each hidden layer = 100&#xA;   epochs = 30&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; Epoch  1 completed out of  30 loss :  22611.10902404785&#xA; Epoch  2 completed out of  30 loss :  12377.467597961426&#xA; Epoch  3 completed out of  30 loss :  8659.753067016602&#xA; Epoch  4 completed out of  30 loss :  6678.618850708008&#xA; Epoch  5 completed out of  30 loss :  5391.995906829834&#xA; Epoch  6 completed out of  30 loss :  4476.406986236572&#xA; Epoch  7 completed out of  30 loss :  3776.497922897339&#xA; -------------------------------------------------------&#xA; Epoch  25 completed out of  30 loss :  478.93606185913086&#xA; Epoch  26 completed out of  30 loss :  450.8017848730087&#xA; Epoch  27 completed out of  30 loss :  435.0913710594177&#xA; Epoch  28 completed out of  30 loss :  452.10553523898125&#xA; Epoch  29 completed out of  30 loss :  539.5199084281921&#xA; Epoch  30 completed out of  30 loss :  685.9198244810104&#xA; Accuracy of Train : 0.88155556&#xA; Accuracy of Test : 0.524&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="28175" LastEditDate="2018-05-05T12:38:30.300" LastActivityDate="2019-05-02T05:20:39.083" Title="How to improve accuracy of deep neural networks" Tags="|machine-learning|neural-network|deep-learning|tensorflow|accuracy|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31251" PostTypeId="1" AcceptedAnswerId="31254" CreationDate="2018-05-05T12:11:55.107" Score="2" ViewCount="726" Body="&lt;p&gt;I've been a Data Scientist for a few years now, but I've only recently started to do most of my work in Python (boy, do I miss &lt;code&gt;ggplot2&lt;/code&gt;! But &lt;code&gt;altair&lt;/code&gt; is coming to the rescue). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to improve my Python skills and, since most of my work is related to developing Data Science &amp;amp; Analytics applications, I'd rather learn from a book about these topics than from a book on, say, Application Server frameworks. Also, since I mostly develop Deep Learning models, I was looking for a PyTorch book. However I could only find two, of which one is really crappy, and the other one is from authors I really respect, but on a topic I don't work with (NLP):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/1491978236&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.amazon.com/Natural-Language-Processing-PyTorch-Applications/dp/1491978236&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I've done a bit of research and found about the following books:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/1491962291&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/149190142X&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Science from Scratch: First Principles with Python&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/1491957662&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython&lt;/a&gt;(second edition recently came out)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://rads.stackoverflow.com/amzn/click/1491912057&quot; rel=&quot;nofollow noreferrer&quot;&gt;Python Data Science Handbook: Essential Tools for Working with Data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which book would you choose? You're free to suggest other titles. Requirements, in order of importance:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;It should be a good book about Python, so the code should be Pythonic&lt;/li&gt;&#xA;&lt;li&gt;It should be about about PyTorch, but not only about NLP (this requirement will probably be unfulfilled, for reasons above). Not about Keras, or mainly about Keras: I already have a reference for that.&lt;/li&gt;&#xA;&lt;li&gt;It should be about Data Science&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Price is not an issue.&lt;/p&gt;&#xA;" OwnerUserId="27526" LastActivityDate="2018-05-06T18:44:07.737" Title="Help me choose a Data Science book in Python" Tags="|python|deep-learning|reference-request|pytorch|books|" AnswerCount="2" CommentCount="1" ClosedDate="2018-05-09T06:14:55.470" ContentLicense="CC BY-SA 4.0" />
  <row Id="31252" PostTypeId="2" ParentId="31248" CreationDate="2018-05-05T12:14:17.983" Score="2" Body="&lt;p&gt;For increasng your accuracy the simplest thing to do in tensorflow is using &lt;code&gt;Dropout&lt;/code&gt; technique. Try to use &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/dropout&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;tf.nn.dropout&lt;/code&gt;&lt;/a&gt;. between your hidden layers. Do not use it for your first and last layers. For applying that, you can take a look at &lt;a href=&quot;https://stackoverflow.com/questions/40879504/how-to-apply-drop-out-in-tensorflow-to-improve-the-accuracy-of-neural-network&quot;&gt;How to apply Drop Out in Tensorflow to improve the accuracy of neural network&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-05T12:14:17.983" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31253" PostTypeId="2" ParentId="31243" CreationDate="2018-05-05T12:24:56.747" Score="2" Body="&lt;p&gt;You have to use &lt;code&gt;F1&lt;/code&gt; score. A simple solution for that is to use &lt;em&gt;confusion matrix&lt;/em&gt;. The way you can find &lt;code&gt;F1&lt;/code&gt; score for each class is simple. your true labels for each class can be considered as true predictions and the rest which are classified wrongly as the other classes should be added to specify the number of false predictions. For each class, you can find the &lt;code&gt;F1&lt;/code&gt; score. For more details take a look at &lt;a href=&quot;https://stackoverflow.com/q/37615544/5120235&quot;&gt;F1-score per class for multi-class classification&lt;/a&gt;. You can take a look at &lt;a href=&quot;https://github.com/keras-team/keras/issues/6507&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; implementation.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-05T12:24:56.747" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31254" PostTypeId="2" ParentId="31251" CreationDate="2018-05-05T13:09:20.317" Score="2" Body="&lt;p&gt;I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow, its a great book (1/2 way thru).  However it uses Scikit as per the title and some of the mechanics inside Sckit are a black box (not explained in depth).  The general concepts however are still quite well written.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 2nd half of the book is about TensorFlow so again, perhaps not what you want if you're focused on PyTorch ?&lt;/p&gt;&#xA;" OwnerUserId="50951" LastActivityDate="2018-05-05T13:09:20.317" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31257" PostTypeId="1" CreationDate="2018-05-03T21:52:25.323" Score="2" ViewCount="3722" Body="&lt;p&gt;I have a model in keras with a custom loss. this loss is calculated using actual and predicted labels(or values) and &lt;strong&gt;is also based on some input value&lt;/strong&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(an example would be to define loss based on reward or advantage as in a policy gradient method in reinforcement learning context )&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example code:&lt;br&gt;&#xA;note: the input value used in loss calculation is named 'in2' and variable is &lt;code&gt;input_2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input_1=keras.layers.Input(...,name=&quot;in1&quot;)&#xA;output= keras.layers.Dense(...)(input_1)&#xA;&#xA;input_2=keras.layers.Input(...,name=&quot;in2&quot;)&#xA;def loss_func(y_true,y_pred):&#xA;    # some custom loss i define based on input_2&#xA;    loss = keras.layers.Dense(..)(input_2) &#xA;    return loss&#xA;&#xA;my_model=keras.Model(inputs=[input_1,input_2],outputs=output)&#xA;my_model.compile(...,loss=loss_func,)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now the problem is that when I want to just predict an output, I have to give a dummy input for the unused &lt;code&gt;input_2&lt;/code&gt;. The &lt;code&gt;input_2&lt;/code&gt; is only used for computing the loss and it is not used for computing the model output. but when i use this code, an error ocures for not providing 'in2':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# this throws error for not providing input value for 'in2'&#xA;my_model.predict({'in1':some_value}) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;so I have to provide some dummy value for &lt;code&gt;in2&lt;/code&gt; so the model outputs its value, although it does not use the &lt;code&gt;in2&lt;/code&gt; value provided. note that providing None value does not work either. I thought this functionality is implemented inside keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or &lt;strong&gt;maybe I am doing it the wrong way? if yes, what is the appropriate way?&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;idea:&lt;br&gt;&#xA;An idea came to my mind was that maybe we can define two models. one model for training and another for predicting: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_model_for_training = keras.Model(inputs=[input_1,input_2],outputs=output)&#xA;my_model_for_predicting = keras.Model(inputs=[input_1],outputs=output)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and when I want to predict, I just use the second model which does not require the input_2 (or &lt;code&gt;in2&lt;/code&gt;) as in example. both models share the same layers so when train the first model, the weights are updated for both models as explained in docs &lt;a href=&quot;https://keras.io/getting-started/functional-api-guide/&quot; rel=&quot;nofollow noreferrer&quot;&gt;(section: Shared layers)&lt;/a&gt;. is there a better solution?&lt;/p&gt;&#xA;" OwnerDisplayName="Soroush" LastActivityDate="2018-05-05T15:34:47.433" Title="Keras custom loss using multiple input" Tags="|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31258" PostTypeId="2" ParentId="31257" CreationDate="2018-05-04T12:08:20.163" Score="1" Body="&lt;p&gt;When you are in the inference phase, there is no loss to be calculated unless you are using a validation set to fine tune your model on.&lt;br&gt;&#xA;Using two models with shared layers in between is fine yet I suggest to start training the model and don't waste your time writing perfect code.&lt;/p&gt;&#xA;" OwnerUserId="44039" OwnerDisplayName="Fadi Bakoura" LastActivityDate="2018-05-04T12:08:20.163" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31259" PostTypeId="1" CreationDate="2018-05-05T16:26:11.527" Score="1" ViewCount="3543" Body="&lt;p&gt;I see one paradox here. If we use Train/Test split and evaluate our Test data, we might get a good score, but any further prediction will not be credible, because model didn't train the Test data and include it's sequences in memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other side, we can train the data on Train and Test sequence as train data, but then we can not evaluate our predictions, because we have no testing reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do you properly predict LSTM models?&lt;/p&gt;&#xA;" OwnerUserId="51652" LastActivityDate="2023-07-24T15:32:02.200" Title="How to properly predict data with LSTM using train/test?" Tags="|neural-network|lstm|" AnswerCount="5" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31263" PostTypeId="1" CreationDate="2018-05-05T23:27:52.760" Score="1" ViewCount="918" Body="&lt;p&gt;I am trying to code a two class classification DT problem that I used SAS EM before. But trying to do it in Sklearn. The target variable is a two class categorical variable. But there are a few continuous independent variables. In SAS I could specify the &quot;Maximum Number of Branches&quot; for each split. So when it is set to 4, some leaf will split into 2 and some in 4 (especially for continuous variables). I could not find an equivalent parameter in sklearn. Looked at &quot;max_leaf-nodes&quot;. But that controls the total number of &quot;leaf&quot; nodes of the entire tree. I am sure some of you probably has faced the same situation and already found a solution. Please help/share. I will really appreciate it.&lt;/p&gt;&#xA;" OwnerUserId="51658" LastActivityDate="2019-05-06T22:00:27.567" Title="How to control number of branches for each split in sklearn DT classifier?" Tags="|python|classification|scikit-learn|decision-trees|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31264" PostTypeId="1" CreationDate="2018-05-06T00:46:42.300" Score="3" ViewCount="12877" Body="&lt;p&gt;I know that keras provides a &lt;code&gt;.class_indicies&lt;/code&gt; dictionary containing the mapping from class names to class as part of &lt;code&gt;.flow_from_directory()&lt;/code&gt; from its &lt;code&gt;ImageDataGenerator&lt;/code&gt; class  (&lt;a href=&quot;https://keras.io/preprocessing/image/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://keras.io/preprocessing/image/&lt;/a&gt;).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;HOWEVER, is there a way to access the corresponding class labels from an existing &lt;em&gt;saved&lt;/em&gt; model (a model saved in as an .h5 file)?  This seems important when putting my model into production and serving predictions since the classes are not known upfront and therefore the images are not separated in pre-labeled directories.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, are there any good examples of how a keras model should be deployed into production (especially for mobile)?  Thanks! &lt;/p&gt;&#xA;" OwnerUserId="10548" LastActivityDate="2019-02-08T14:45:15.057" Title="In Keras how to get the `class_indices` or prediction labels for an existing model" Tags="|keras|cnn|prediction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31265" PostTypeId="2" ParentId="31263" CreationDate="2018-05-06T01:10:57.987" Score="1" Body="&lt;p&gt;Scikit-learn uses an optimized version of the CART algorithm, which follows binary splits. What you are looking for is a multi-way split on nodes for information gain. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should look for a C4.5 or C5.0 implementation of Decision tress in python.&lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-06T16:21:08.683" LastActivityDate="2018-05-06T16:21:08.683" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31266" PostTypeId="1" CreationDate="2018-05-06T03:55:38.330" Score="1" ViewCount="183" Body="&lt;p&gt;Generally speaking, when training a deep learning model, like MLP,  what kind of data pre-processing operation has to be conducted when the input is a numerical sequence.&lt;/p&gt;&#xA;" OwnerUserId="21767" LastEditorUserId="29169" LastEditDate="2021-02-13T04:22:53.073" LastActivityDate="2021-02-13T04:22:53.073" Title="Data pre-processing before feeding into a deep learning model" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|mlp|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31267" PostTypeId="1" CreationDate="2018-05-06T04:17:57.897" Score="3" ViewCount="4923" Body="&lt;p&gt;I'm currently working on a demand forecasting task, with data on tens of thousands of products across a couple thousand stores. More specifically,I have 3 years' worth of daily sales data per product in each store, and my goal is to forecast the future sales of each item in each store, one day ahead; then two days ahead, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I've considered breaking down each product-store pair into a single time series, and doing a forecast for each time series as was done in Neal Wagner's paper, &lt;a href=&quot;https://cs.adelaide.edu.au/~zbyszek/Papers/p51.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Intelligent techniques for forecasting multiple time series in real-world systems&lt;/a&gt;. In other words, I will use only the historical information of a particular store's sales of the product to forecast the future sales of that product in that store. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I've been browsing Kaggle and competitions like &lt;a href=&quot;https://www.kaggle.com/c/favorita-grocery-sales-forecasting&quot; rel=&quot;nofollow noreferrer&quot;&gt;Corporación Favorita Grocery Sales Forecasting&lt;/a&gt; suggest a different approach, which is to use the information from all stores and all products to predict future sales. As I understand it, historical sales information of all products in all stores are dumped into the training set, from which the model will learn to forecasts future sales. It's very different from traditional time series methods, but apparently, based on the results of the competition, it works.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter method seems promising and more robust. However, there's the problem of having to process hundreds of millions of data points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which method is more appropriate for my task? For those who have worked on similar problems, which methodology would you recommend?&lt;/p&gt;&#xA;" OwnerUserId="41254" LastEditorUserId="41254" LastEditDate="2018-05-06T04:30:15.007" LastActivityDate="2018-05-24T09:33:30.413" Title="Demand Forecasting for Multiple Products Across Thousands of Stores" Tags="|predictive-modeling|time-series|forecast|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31268" PostTypeId="2" ParentId="31259" CreationDate="2018-05-06T05:08:00.373" Score="1" Body="&lt;p&gt;This situation is common in a generic modeling setting, not only for LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the development phase, the model is built using training data, and test data is used to estimate the model quality. Post this, the model is trained on the entire data, and this updated model is used for prediction on new data points.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-06T05:08:00.373" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31269" PostTypeId="2" ParentId="31248" CreationDate="2018-05-06T07:41:11.267" Score="5" Body="&lt;p&gt;You need much more data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Deep NN shines when you have excessive amounts of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-05-06T07:41:11.267" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31270" PostTypeId="2" ParentId="31251" CreationDate="2018-05-06T07:43:43.033" Score="1" Body="&lt;p&gt;Use the &lt;code&gt;random.randint&lt;/code&gt; function to have Python recommend a book for you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Probably none is objectively best. And I don't see any training data to predict this.&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2018-05-06T18:44:07.737" LastActivityDate="2018-05-06T18:44:07.737" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31271" PostTypeId="1" AcceptedAnswerId="31272" CreationDate="2018-05-06T07:45:12.430" Score="0" ViewCount="8131" Body="&lt;p&gt;I've a &lt;code&gt;dat&lt;/code&gt; file in the following format which I'm trying to load using &lt;code&gt;pandas&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;28::Persuasion (1995)::Romance&#xA;29::City of Lost Children, The (1995)::Adventure|Sci-Fi&#xA;30::Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)::Drama&#xA;31::Dangerous Minds (1995)::Drama&#xA;32::Twelve Monkeys (1995)::Drama|Sci-Fi&#xA;33::Wings of Courage (1995)::Adventure|Romance&#xA;34::Babe (1995)::Children's|Comedy|Drama&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;movies = pd.read_csv('movies.dat')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm getting this error &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ParserError: Error tokenizing data. C error: Expected 1 fields in line 11, saw 2&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Line 11: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;11::American President, The (1995)::Comedy|Drama|Romance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand that it's because of the comma. One option I found is to skip these lines, which I don't want to. Is it possible to set newline as the delimiter here? Any help will be truly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="51670" LastEditorUserId="51670" LastEditDate="2018-05-06T07:52:57.713" LastActivityDate="2020-05-11T17:49:53.727" Title="custom delimiter for dat file" Tags="|python|pandas|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31272" PostTypeId="2" ParentId="31271" CreationDate="2018-05-06T08:00:33.357" Score="2" Body="&lt;p&gt;You have to tell &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.read_csv &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;the separator that you want to use by doing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pd.read_csv('movies.dat', sep = '::')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should work.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-06T08:00:33.357" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31274" PostTypeId="1" CreationDate="2018-05-06T08:50:05.670" Score="1" ViewCount="196" Body="&lt;p&gt;I am a sophomore student who's interested in deep learning and its method layering up some linear/non-linear operations and constructs up the complex function through the network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to comprise a network that finds the word upon the given explanations :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.1 input: a spherically shaped fruit which tastes sour and sweet and gets red when matured well output: apple&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g.2, input: to make up the boundary between multiple options for the base of one's behavior or decision output: determine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically it's a classification problem but the class might be more than 20,000.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to name this task &quot;rephrase neural network&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any good reference or starting point, or, any network recommendation for comprising this neural network?&lt;/p&gt;&#xA;" OwnerUserId="51672" LastActivityDate="2022-06-09T08:01:00.070" Title="Rephrase Neural Network : Find the Fittest Word for Given Meaning/Explanation" Tags="|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31275" PostTypeId="2" ParentId="30652" CreationDate="2018-05-06T09:53:38.913" Score="1" Body="&lt;p&gt;You could solve the problem in an unsupervised approach.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;build your data distribution with all food Portuguese words&lt;/li&gt;&#xA;&lt;li&gt;formulate your data points as a sequence of characters, &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;rnn-effectiveness&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;transform those words into its phonemic form, see &lt;a href=&quot;https://en.wikipedia.org/wiki/ARPABET&quot; rel=&quot;nofollow noreferrer&quot;&gt;text to phonemes converter&#xA;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;use denoising auto-encoder, instead of using random gaussian as your corruption process.&lt;br&gt;&#xA;Since the problem is discrete, you could use some heuristics to corrupt each word by replacing random phonemes with others which is most likely to be similar.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You have to find some automated way to do that corruption, get some insights from customers history data and see their words spellings error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the architecture, vanilla RNN is fine. see more &lt;a href=&quot;https://towardsdatascience.com/creating-a-spell-checker-with-tensorflow-d35b23939f60&quot; rel=&quot;nofollow noreferrer&quot;&gt;Creating a Spell Checker with TensorFlow&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a nutshell: it's formulated as a seq2seq problem, the source is the misspelled word, while the target is the correct one.&#xA;DAE will be trained on input that is corrupted stochastically, thus it must learn to guess the distribution of the missing information (reconstruct the clean original input)&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T09:53:38.913" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31277" PostTypeId="2" ParentId="31259" CreationDate="2018-05-06T10:33:54.803" Score="0" Body="&lt;p&gt;In general, effective learning is all about making the training error small and the gap between training and test error small.&lt;br&gt;&#xA;By test data, we mean examples that your model has never seen before. so you need development (validation) set, to fine-tune your hyperparameters such hidden cells, the number of layers, learning rate, etc.&lt;br&gt;&#xA;Split the training data into train/dev sets, be careful test set must always be generated from the same data distribution that generates your train/dev sets.&lt;br&gt;&#xA;LSTM might overfit your dataset, start with vanilla RNN, or small GRU.&lt;br&gt;&#xA;Use early stopping to stop training when the loss of the validation examples stop decreasing.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T10:33:54.803" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31278" PostTypeId="2" ParentId="31248" CreationDate="2018-05-06T10:39:39.693" Score="2" Body="&lt;p&gt;Whatever regularization technique you're using, if you keep training long enough, you will eventually overfit the training data, you need to keep track of the validation loss each epoch. at least use early stopping to stop the training process when the validation loss stops decreasing.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T10:39:39.693" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31279" PostTypeId="2" ParentId="31274" CreationDate="2018-05-06T11:13:43.673" Score="1" Body="&lt;p&gt;In a perfect setting, where you have infinite sentences (definitions).&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/EMSrN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/EMSrN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could use some kind of attention models by chunking sentences into smaller pieces.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;[a spherically shaped] [fruit] which [tastes sour] and [sweet] and gets [red] when matured well.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a distributed representation framework, every little piece of information is simply like a binary indicator (the feature exists or not).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe each piece is vague by itself and it could not tell you anything about the target yet by combining them together they define useful information, see &lt;a href=&quot;http://www.cs.toronto.edu/~fritz/absps/tr00-004.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;products of experts&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To learn p(Y|X) where x consists of the sentence, y is the embedding of some class, so may similar classes share similar sentences.&lt;br&gt;&#xA;If you have a model trained on the sentences and learned to project similar sentences near to each other, and assume you learned the word classes by some word2vec like algorithm, you could generalize to new pair even if the sentence and that class target were never paired (never seen in your training data), their respective feature vectors have been related to each other by the mapping function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end of the day if you have a smooth function and your input is continuous then any small change of the input will introduce small change at the output, yet how could we truly extract the knowledge exists in the data. see &lt;a href=&quot;https://arxiv.org/pdf/1511.06349.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generating Sentences from a Continuous Space&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T11:13:43.673" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31280" PostTypeId="1" CreationDate="2018-05-06T11:20:57.537" Score="0" ViewCount="108" Body="&lt;p&gt;I have a hard time intuitively understand the Bayer error in the context of supervised learning. We have an input X and an output Y. We want to find the function f(X) = Y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I feel like I don't understand why we model X as a random variable in the first place. I think we construct the function f not based on the distribution of X but rather on actual values of X. Why should we care if X is stochastic or not?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example I:&#xA;Let's say X = [1,2,3], Y = [1,2,3] and f(x) = x.&#xA;When X=1 =&gt; Y=1, X=2 =&gt; Y=2, X=3 =&gt; Y=3, I will never make an error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example II: Why should it be different if X = [picture of dog, picture of cat, picture of house] and Y = [dog, cat, house]. I can still find a function which does the mapping. It is obviously more complex but doable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where did the Bayes error get lost in my examples?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking for an intuitive explanation of the Bayes error preferably in the context of image classification.&lt;/p&gt;&#xA;" OwnerUserId="50021" LastActivityDate="2018-09-04T05:00:59.950" Title="image based example for Bayes error/irreducible error" Tags="|machine-learning|bayes-error|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31282" PostTypeId="2" ParentId="31280" CreationDate="2018-05-06T11:50:08.560" Score="1" Body="&lt;p&gt;As an MNIST dataset, see the mispredicted images on this figure below,  you notice for some samples, we as human don't know for sure what is the true label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hcCD2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hcCD2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe there are other factors that are not visible to us to predict the correct output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume I want to predict customer will leave our company, maybe there are tons of useful features to start with, yet he may simply have some other problems that we are not aware of, and it may affect his decision in leaving the company.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T11:50:08.560" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31283" PostTypeId="2" ParentId="31266" CreationDate="2018-05-06T12:27:54.977" Score="0" Body="&lt;p&gt;If you simply transform the data to have zero mean unit variance Gaussian,&#xA;be careful to not include your test samples when calculating the actual mean and variance, by doing so you are doing some kind of data snooping thus test error is not reliable, doesn't reflect the true error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see &lt;a href=&quot;http://www.visiondummy.com/2014/03/divide-variance-n-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Why divide the sample variance by N-1?&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T12:27:54.977" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31284" PostTypeId="2" ParentId="23538" CreationDate="2018-05-06T12:49:32.133" Score="1" Body="&lt;p&gt;Use the same model you trained for the classification task, and append it with 10 logistic units which indicates respectively whether if the magnitude is k or less, as previously suggested.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the imbalance problem, you could pick your batch examples to have equal probability for all the classes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;An intuitive explanation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Class A contains 10 examples while class B contains 90 examples.&#xA;duplicate each example of the class A eight times, and then shuffle both classes examples, then during training, keep sampling from a uniform distribution.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Of course, you don't need to duplicate the example data explicitly, just keep a list of indices mapped to your input vectors and start duplicating from there.  &lt;/p&gt;&#xA;" OwnerUserId="44039" LastEditorUserId="29575" LastEditDate="2018-05-06T14:54:51.847" LastActivityDate="2018-05-06T14:54:51.847" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31285" PostTypeId="2" ParentId="27466" CreationDate="2018-05-06T13:04:07.417" Score="3" Body="&lt;p&gt;If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that best suits the problem).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each step will make the loss decrease until you reach a point, you will have to search over the hyperparameters to squeeze every percent of accuracy, which is heavily a computational problem.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T13:04:07.417" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31286" PostTypeId="2" ParentId="27407" CreationDate="2018-05-06T13:57:28.197" Score="0" Body="&lt;p&gt;I suggest trying to learn distributed representation of HTML code, build an embedding matrix for tags, use some RNN auto-encoder like &lt;a href=&quot;https://arxiv.org/abs/1511.06349&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; to learn elements. thus you end up with some low-level dimensionality, clustering in that latent space will be more efficient and effective. &lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T13:57:28.197" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31287" PostTypeId="2" ParentId="30810" CreationDate="2018-05-06T14:26:48.387" Score="0" Body="&lt;p&gt;I want to post the working code here as an answer. I had to read and understand many concepts to make the code work. The is working but my training process isn't thoroughly vetted but that is a different subject.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I had to revise my understanding of these subjects.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Convolutions&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Deconvolutions(&lt;a href=&quot;https://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers/9975#9975&quot;&gt;What are deconvolutional layers?&lt;/a&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Basic TensorFlow API and shapes&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is the main part that tripped me. So I am posting that alone.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = tf.placeholder(tf.float32, shape=[None, 299, 299, 3], name='X')&#xA;&#xA;&#xA;Z = tf.placeholder(dtype=tf.float32,&#xA;                              shape=(None, 100),&#xA;                              name='Z')&#xA;&#xA;filters = [2, 2, 3, 32]&#xA;weights = tf.Variable(tf.truncated_normal(filters, stddev=0.03))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is just basic code but it helped me answer my original question.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generator(z,reuse=False):&#xA;    with tf.variable_scope('generator',reuse=reuse):&#xA;        linear = tf.layers.dense(z, 1024 * 4 * 4)&#xA;        conv = tf.reshape(linear, (-1, 4, 4, 1024))&#xA;        out = tf.layers.conv2d_transpose(conv, 3,kernel_size=4,strides=[2,2], padding='SAME')&#xA;        out = tf.nn.relu(out)&#xA;&#xA;        tf.nn.tanh(out)&#xA;        return out&#xA;&#xA;&#xA;def discriminator(x,reuse=False):&#xA;    with tf.variable_scope('discriminator',reuse=reuse):&#xA;        out = tf.nn.conv2d(x, weights, [1, 1, 1, 1], padding='SAME')&#xA;        out = tf.nn.relu(out)&#xA;        out = tf.nn.max_pool(out, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],padding='SAME')&#xA;        out = tf.layers.dense(out, units=128, activation=tf.nn.relu)&#xA;        out = tf.layers.dense(out, units=1, activation=tf.nn.sigmoid)&#xA;        return out&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51126" LastActivityDate="2018-05-06T14:26:48.387" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31288" PostTypeId="1" CreationDate="2018-05-06T15:01:38.167" Score="2" ViewCount="88" Body="&lt;p&gt;This is my general understanding on the affect of Multi-Col linearity on Linear and Logistic regression. However I am still not able to understand whether multi-col linearity is a problem in decision tree (Classification and regression).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Linear Regression - It has a huge impact as the co-efficient might vary drastically if there are 2 variables with high correlation. And regularisation would overcome this issue.&#xA;Logistic Regression - Same as Linear regression.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anybody please give me some insight on how it affects in decision tree classifier and decision tree regressor. And how to overcome this problem if it affects. &lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-05-06T15:01:38.167" Title="Affect of multi-Col linearity in Machine Learning Algorithms" Tags="|machine-learning|classification|regression|decision-trees|" AnswerCount="0" CommentCount="2" ClosedDate="2020-01-20T04:19:43.077" ContentLicense="CC BY-SA 4.0" />
  <row Id="31289" PostTypeId="1" CreationDate="2018-05-06T15:17:04.507" Score="1" ViewCount="408" Body="&lt;p&gt;I am very new to machine learning, and it's hard for me to know in what direction I should go. Thanks in advance for your help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of data showing what customers bought:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A customer may have bought 1 or more items&lt;/li&gt;&#xA;&lt;li&gt;Each item is discribed by :&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A quality level which is an integer between 1 and 7&lt;/li&gt;&#xA;&lt;li&gt;Its content from a first point of view (the item has x1% of content 1, x2% of content 2, ...)&lt;/li&gt;&#xA;&lt;li&gt;Its content from a second point of view (the item has xA% of content A, xB% of content B, ...)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;A customer says that he belongs to the group G1, G2 or G3&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I would like to be able to guess the group a new customer belongs to, given the items he bought.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What algorithm should I ideally use ?&#xA;What algorithm would you recommend if I can only use numpy, without scipy or scikit (it's on a server which only has numpy) ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you very much&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit : I'm adding the appropriate words to the title and to the tags&lt;/p&gt;&#xA;" OwnerUserId="41726" LastEditorUserId="41726" LastEditDate="2018-05-06T16:31:17.100" LastActivityDate="2019-01-05T00:01:38.603" Title="Which supervised learning algorithm to classify customers buying items" Tags="|classification|supervised-learning|numpy|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31290" PostTypeId="2" ParentId="31289" CreationDate="2018-05-06T15:45:06.457" Score="1" Body="&lt;p&gt;As per the problem statement statement it does look like 3 class classification problem. &#xA;But I don't think you can use any ML algorithms using Python without sklearn package in it. Try getting it installed and you can implement the problem using Logistic regression, Decision Tree, there are still more classification algorithms.&#xA;However if you are planning to extract a pattern from the dataset and making into cluster, you can use K-means clustering which is a un-supervised algorithm. &lt;/p&gt;&#xA;" OwnerUserId="50146" LastActivityDate="2018-05-06T15:45:06.457" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31291" PostTypeId="1" AcceptedAnswerId="31312" CreationDate="2018-05-06T16:04:13.700" Score="1" ViewCount="7518" Body="&lt;p&gt;I am working with scikit learn and GridSearch in order to find the best parameters in my classifiers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a map of different hyperparameters and I want to print out GridSearch results, but I do not understand one thing - what is the difference between &lt;code&gt;mean_test_score&lt;/code&gt; and &lt;code&gt;mean_train_score&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I understand, GridSearch performs cross-validation in order to find the best classifier, but how do these 2 params differ from one another? I always thought that cross-validation gives only one mean, which is a mean of the performance from trained models using N subsets of given data. For example, if I perform a cross-validation with X subsets, I will have X different accuracy scores and then I will have only one mean value.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how can I interpret these 2 parameters and what is the difference between them?&lt;/p&gt;&#xA;" OwnerUserId="41740" LastEditorUserId="50727" LastEditDate="2018-05-22T15:33:40.543" LastActivityDate="2018-05-22T15:33:40.543" Title="GridSearch mean_test_score vs mean_train_score" Tags="|machine-learning|scikit-learn|cross-validation|grid-search|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31292" PostTypeId="1" AcceptedAnswerId="31295" CreationDate="2018-05-06T16:44:04.660" Score="8" ViewCount="3274" Body="&lt;p&gt;I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def normalize(array):&#xA;    return (array - array.mean()) / array.std()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also please explain what this &lt;code&gt;array - array.mean()&lt;/code&gt; does?&lt;/p&gt;&#xA;" OwnerUserId="24665" LastEditorUserId="28175" LastEditDate="2018-05-06T17:41:00.317" LastActivityDate="2018-05-08T00:18:30.763" Title="What is normalization for?" Tags="|machine-learning|python|deep-learning|dataset|normalization|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31293" PostTypeId="2" ParentId="31292" CreationDate="2018-05-06T16:55:40.557" Score="1" Body="&lt;p&gt;It is a standard approach is to scale the inputs to have zero mean unit variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Mathematically:&lt;/strong&gt; &#xA;if you have some observed data (your training examples), called the empirical distribution, which is a proxy for the true data distribution that is generated from the unknown data distribution.&lt;br&gt;&#xA;you could assume that your input features, each dimension is drawn from a uni-variate Gaussian distribution, which you could estimate using maximum likelihood estimation in closed form solution see &lt;a href=&quot;http://www.visiondummy.com/2014/03/divide-variance-n-1/&quot; rel=&quot;nofollow noreferrer&quot;&gt;MLE for Gaussian&lt;/a&gt; and you could obtain unbiased mean and biased variance could be fixed to be unbiased. following that, it's straightforward to just subtract all your data points from the original mean, you end up with zero mean, and scale the variance to be one by dividing by the fixed unbiased variance.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T16:55:40.557" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31294" PostTypeId="2" ParentId="27329" CreationDate="2018-05-06T17:01:04.710" Score="2" Body="&lt;p&gt;The most important thing here is to model your problem as time series, as you know, each example consists of one customer that has many orders over time. see &lt;a href=&quot;http://proceedings.mlr.press/v56/Choi16.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Predicting Clinical Events via Recurrent Neural Networks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-06T17:01:04.710" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31295" PostTypeId="2" ParentId="31292" CreationDate="2018-05-06T17:22:20.677" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Also please explain what this array - array.mean() do?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Basically, it is doing memberwise subtraction operation after &lt;a href=&quot;https://docs.scipy.org/doc/numpy-1.14.0/user/basics.broadcasting.html&quot; rel=&quot;noreferrer&quot;&gt;broadcasting&lt;/a&gt;. &lt;code&gt;np.mean&lt;/code&gt; function finds the mean in your array and its result will be a scalar, a single number. Your array is a &lt;code&gt;numpy&lt;/code&gt; array and the result of the latter term is a single value as mentioned. Consequently, the single value gets extended to the shape of the former term. then a memberwise subtraction will be performed for each entry of the array and the result will have the same shape as the former term.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Can you please explain what does this normalize function do?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Normalizing data is done for accelerating optimization. If you have features with different scales, it will take too much time for your optimizer function to find optimal points. Suppose you have age feature which can change between 0 to 150 (!) and salary which can be changed from 0 to whatever, like 500,000,000 $. your optimization algorithm used in your ML model will take too much time, if possible, to find appropriate weights for each feature. Moreover, if you don't scale your data, your ML algorithm may take too much care to features with large scales.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-06T17:22:20.677" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31296" PostTypeId="2" ParentId="30652" CreationDate="2018-05-06T17:37:28.693" Score="6" Body="&lt;p&gt;There are many ways to build a spell corrector. One of the simplest is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Detect an incorrect word&lt;/li&gt;&#xA;&lt;li&gt;Generate candidate suggestions&lt;/li&gt;&#xA;&lt;li&gt;Score and rank the candidate replacements&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For detecting an incorrect word, a simplifying assumption is that any word not a dictionary is a spelling error. Otherwise, you have to build a separate model to detect if a word is a potential spelling mistake based the current context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To generate candidate words, you need to find dictionary words that are similar to the incorrect word. That requires defining &quot;similar&quot;, typically similar is measured by &lt;a href=&quot;https://en.wikipedia.org/wiki/Edit_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;edit distance&lt;/a&gt; (e.g., deletes, transposes, replaces, or inserts of individual characters). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For scoring and ranking, probabilities of candidate words are &lt;a href=&quot;https://towardsdatascience.com/language-models-spellchecking-and-autocorrection-dd10f739443c&quot; rel=&quot;nofollow noreferrer&quot;&gt;weighted by a language model and channel model&lt;/a&gt;. A &lt;a href=&quot;https://en.wikipedia.org/wiki/Language_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;language model&lt;/a&gt; weights how likely a word will appear in the current context. A &lt;a href=&quot;https://en.wikipedia.org/wiki/Communication_channel#Channel_models&quot; rel=&quot;nofollow noreferrer&quot;&gt;channel model&lt;/a&gt; reflects if an error happens depending on how the word is transmitted (e.g., &lt;a href=&quot;http://nbviewer.jupyter.org/url/norvig.com/ipython/Gesture%20Typing.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;full computer keyboard errors are different from mobile phone errors&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your example, you need to construct a custom dictionary of all possible words and a corpus reflecting the frequency of occurrences for the custom dictionary words. If possible, construct an error model of common mistakes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Peter Norvig goes into greater detail &lt;a href=&quot;https://norvig.com/spell-correct.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Later you can add Deep Learning methods, but it is better to start with a simpler approach that requires less data.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-09T18:48:47.397" LastActivityDate="2018-05-09T18:48:47.397" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31297" PostTypeId="1" CreationDate="2018-05-06T17:41:48.863" Score="3" ViewCount="254" Body="&lt;p&gt;What is &lt;strong&gt;the main goal&lt;/strong&gt; of using an activation function in CNN?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know the activation functions types and the purpose of each one. But here I am asking why to use them. &lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="28175" LastEditDate="2018-05-06T18:25:25.293" LastActivityDate="2018-05-06T18:25:25.293" Title="What is the main goal of using activation function in CNN?" Tags="|machine-learning|neural-network|deep-learning|tensorflow|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31298" PostTypeId="1" CreationDate="2018-05-06T18:07:27.180" Score="3" ViewCount="398" Body="&lt;p&gt;I have trained a simple neural net, to make predictions based on three inputs. For examples sake lets say I m trying to work out &quot;how many days does it take to complete an accademic study&quot;, where the data could be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f1 = institution&#xA;f2 = student age&#xA;f3 = study topic&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and the label is the total number of days to complete. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you imagine the following data structure:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;f1, f2, f3, label&#xA;&#xA;A,  D,  G,   v&#xA;B,  E,  H,   w&#xA;C,  F,  I,   x&#xA;A,  E,  H,   y&#xA;A,  D,  G,   z&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there anyway that I can work out how long each feature adds to the total number of days to complete, so for example I want to be able to say that if you choose &lt;code&gt;f1 = A&lt;/code&gt; that 4 days will be added to the total days to complete the study. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can work out how much difference there is between each choice, eg choosing &lt;code&gt;f1 = A&lt;/code&gt; might be 7 days quicker than if you chose &lt;code&gt;f1 = B&lt;/code&gt;, but I do not know how to roughly ( maybe average ) identify how long each choice represents.&lt;/p&gt;&#xA;" OwnerUserId="51656" LastEditorUserId="8501" LastEditDate="2018-05-07T05:27:19.657" LastActivityDate="2018-07-07T00:43:10.240" Title="How to quantify the numerical influence of categorical variables?" Tags="|machine-learning|statistics|mathematics|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31299" PostTypeId="2" ParentId="31297" CreationDate="2018-05-06T18:24:51.610" Score="1" Body="&lt;p&gt;The idea of convolutional layers is that we need same weights to be applied to different regions of the input. It lets you identify same patterns that occur in different regions of the input. You can consider each convolution operation which is carried out by each window as a single neuron which just transforms a local region with a non-linear transformation. The linear and non-linear operations that are done by these so-called local neurons help the convolutional layers learn non-linear and complex features. Another aspect of this operation is that it reduces the number of weights significantly. The final interpretation is that if you don't use non-linear activations, you would have linear local features extracted from the input. Those linear features can easily be found with fewer number of parameters.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-06T18:24:51.610" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31300" PostTypeId="1" CreationDate="2018-05-06T19:01:09.103" Score="1" ViewCount="495" Body="&lt;p&gt;Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I know that I can use TruncatedSVD to do the dimensionality reduction. However, my problem is about choosing the correct amount of &lt;code&gt;n_components&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach suggested doing TruncatedSVD to the maximum amount of variables first and then calculate on this, how many variables on needs to reach a &quot;explained variance&quot; goal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But doing the initial TSVD on 99999 variables seems to 1) take a long time (it hasn't finished it) and it 2) slowly seems to occupy all my memory (which I have 120GB).&lt;/p&gt;&#xA;" OwnerUserId="51687" LastEditorUserId="51687" LastEditDate="2018-05-06T20:25:08.617" LastActivityDate="2018-05-06T20:25:08.617" Title="Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables" Tags="|bigdata|scipy|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31302" PostTypeId="1" AcceptedAnswerId="31303" CreationDate="2018-05-06T21:01:24.580" Score="0" ViewCount="107" Body="&lt;p&gt;When training my LSTM, is there any incentive to pre-pass its inputs through an auto-encoder, or should I always supply as raw data as possible?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have is a small amount of training data, and it contains only a few very rare significant events (spikes, spaced out in time)&lt;/p&gt;&#xA;" OwnerUserId="43077" LastActivityDate="2018-05-06T21:36:52.097" Title="autoEncoder as LSTM input, any benefit?" Tags="|lstm|autoencoder|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31303" PostTypeId="2" ParentId="31302" CreationDate="2018-05-06T21:36:52.097" Score="2" Body="&lt;p&gt;The purpose of the autencoder would be similar to doing an embedding of text to pass latent space features of the input to the LSTM, as opposed to raw inputs. In the case of text, this is usually done to reduce the very large and sparse dictionary (word subword, whatever) space to a   space which is less sparse, significantly smaller, and therefore more computationally efficient. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, it looks like the data is not high dimensional to start with (I am assuming this because of the significant events over time), so you would not get the benefit of mapping the input to a latent space. If your trend over time is high dimensional, then it may be worth it, as the embedding/autencoding may pick up and reduce correlations between trends in a few dimensions.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-06T21:36:52.097" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31304" PostTypeId="2" ParentId="786" CreationDate="2018-05-06T22:22:04.473" Score="2" Body="&lt;p&gt;There are multiple platforms (like quandl, iex, yahoo) that provide such data and pandas_datareader gives you even better interface to collect data from these platforms. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out this blog for a quick &lt;a href=&quot;https://medium.com/@kyle.jinhai.li/collect-trading-data-with-pandas-library-8904659f2122&quot; rel=&quot;nofollow noreferrer&quot;&gt;example&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="51694" LastEditorUserId="40853" LastEditDate="2018-05-07T01:44:05.917" LastActivityDate="2018-05-07T01:44:05.917" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31305" PostTypeId="2" ParentId="25722" CreationDate="2018-05-06T22:55:19.020" Score="4" Body="&lt;p&gt;So far this part hasn't been answered: &quot;should it be used after pooling or before pooling and after applying activation?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One team did some interesting experiments and found that, at least for residual networks, batch norm and activation should come before the weights layer (So it goes BN --&gt; ReLU --&gt; Weights). They found pooling placement could vary without detriment. If you try to replicate that paper note very carefully the paragraph on the exceptions in placement for the beginning and end of the residual stacks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Publication: &lt;a href=&quot;https://arxiv.org/abs/1603.05027&quot; rel=&quot;nofollow noreferrer&quot;&gt;He et al. - 2016 - Identity mappings in deep residual networks&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51696" LastActivityDate="2018-05-06T22:55:19.020" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31307" PostTypeId="1" CreationDate="2018-05-07T02:41:33.810" Score="0" ViewCount="503" Body="&lt;p&gt;I want to train a model to detect wrong word using in sentence.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is wrong or correct .&lt;/li&gt;&#xA;&lt;li&gt;If a sentence length is  10,  I want LSTM learn 2 words surrounding current word ( so each word become 5 word) . So I need transform the sentence to [10, 5] . &lt;/li&gt;&#xA;&lt;li&gt;But the first word(doesn't have left words) and  right word (doesn't have right words) , so I need pad them.&lt;/li&gt;&#xA;&lt;li&gt;Then use word2vec transform this sentence to  [10, 5, embedding_size] , same as other sentences. (i.e each word transform to 5 word embeddings)&lt;/li&gt;&#xA;&lt;li&gt;trainning (feed model by [5, embedding_size] * bacth_size )&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I saw a lot LSTM input format and finally work out above flow.&#xA;Is my steps right ?&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Actually I am still confusing with LSTM input format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Some pad all sentence to same length &lt;/li&gt;&#xA;&lt;li&gt;Some split sentence into word blocks with same size (like I do above).  &lt;/li&gt;&#xA;&lt;li&gt;others ....&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am trying to use code to generate error samples, and mix with correct samples for lstm training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;corrects  = [&#xA;    ['How', 'are', 'you', '!'],          # [1, 1, 1, 1]&#xA;    ['Fine', ',', 'thank', 'you', '.'],   #  [1, 1, 1, 1, 1]&#xA;    ['Do', 'you', 'have', 'meal', '?'],   #  [1, 1, 1, 1, 1]&#xA;    ...&#xA;]&#xA;&#xA;&#xA;wrongs  = [&#xA;    ['How', 'were', 'you', '!'],            # [1, 0, 1, 1]&#xA;    ['Find', ',', 'thank', 'you', '.'],    # [0, 1, 1, 1, 1]&#xA;    ['Did', 'you', 'have', 'meal', '.'],    #  [0, 1, 1, 1, 0]&#xA;    ...&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I come up with several thoughts&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Because lstm can learn current word's network weight with pre words. So directly pass sentence to lstm model, don't transform from &lt;code&gt;[sentence_len, embedding_size]&lt;/code&gt; to &lt;code&gt;[sentence_len, step_len, embedding_size]&lt;/code&gt; (&lt;code&gt;step_len&lt;/code&gt; means present a word with it surround words). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Previous sentence's last word may be affacted by latter sentence's first few word, so It better chain all sentence . (How to BiLSTM in this situation ?) &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use feature like &lt;code&gt;[sentence_len, step_len, embedding_size]&lt;/code&gt;, doesn't need &lt;code&gt;BiLSTM&lt;/code&gt; any more ? I think it still help here. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are too many thoughts in my brain, hard to make it clear. So I post question here. &lt;/p&gt;&#xA;" OwnerUserId="15325" LastActivityDate="2018-05-07T03:59:54.960" Title="What is the proper train data format in LSTM?" Tags="|neural-network|keras|text-mining|lstm|word2vec|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31308" PostTypeId="2" ParentId="11960" CreationDate="2018-05-07T03:33:56.050" Score="0" Body="&lt;p&gt;Your problem is suitable for machine learning technique.&lt;br&gt;&#xA;You are modeling the problem as a non-parametric model, i.e. parameters are dependent on your data, as your data grows, parameters also grow as well.&lt;br&gt;&#xA;Simple non-parametric models will be something like k-nearest neighbors.  each time you want to predict some new data point, you have to calculate all the training examples distances with that data point.&lt;br&gt;&#xA;So keeping track of email addresses is not a good idea, what if you let a machine read the content, then automatically decide with a fixed set of parameters what the output should be?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could access large dataset of spam/ham emails. and train large deep models on, following that fine-tune it on your custom dataset.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for the accuracy of the model, after you train a classifier, the output of the model will be a simple real value between 0 and 1, telling you how much the model believes it's ham or spam respectively, usually, we set a threshold of 0.5 to give definite answers. yet if you want to optimize the false negatives, you could increase the threshold.&lt;br&gt;&#xA;e.g. if you set it to be 0.8, you are saying I don't want the machine to classify it as ham if it's not 80% sure.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-07T03:33:56.050" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31309" PostTypeId="2" ParentId="31298" CreationDate="2018-05-07T03:35:27.250" Score="1" Body="&lt;p&gt;Yes you're talking about training a model, then doing &lt;strong&gt;sensitivity analysis&lt;/strong&gt; to determine the numerical influence of each explanatory variable in isolation:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some considerations/tips:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;you said the three features are categorical. Presumably &lt;code&gt;student_age&lt;/code&gt; is ordered categorical, and the other two are unordered. But you didn't say how many levels they each have (cardinality), or how many bins. You don't necessarily want to (say) bin &lt;code&gt;student_age&lt;/code&gt; in bins of 1 year each. You might bin in say 2/3-year increments, or quantiles (see the R &lt;code&gt;cut&lt;/code&gt; command), e.g. &lt;code&gt;student_age: [0-17, 18-20, 21-23, 24-28, 29-34, 35+]&lt;/code&gt;. Similarly, if you have a long tail of smaller institutions with few exemplars, you might bin them together, or bin by country/ region/ size/ focus/ ranking.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;what is the product of cardinality of each variable, vs the size of the dataset? Do you have 1 exemplar for each possible combination? &gt;10? &amp;lt;0.1? If you don't have that much data, you might prefer to use Linear Regression rather than NN, or at least a shallower NN rather than deep. With Linear Regression you can allow quadratic interaction terms, nonlinear terms etc - again &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;(with Linear Regression you can directly see the influence just by directly reading the model coefficients e.g. &lt;code&gt;Princeton: -1.4, Princeton.Age22_24: -3.6&lt;/code&gt;)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;if you specifically want to use NN, see the &lt;strong&gt;&lt;a href=&quot;https://github.com/marcotcr/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;lime&lt;/a&gt; package&lt;/strong&gt;. Also see the many articles/tutorials on sensitivity analysis with NNs, and &lt;a href=&quot;https://stats.stackexchange.com/search?q=sensitivity+analysis&quot;&gt;CrossValidated answers on sensitivity analysis&lt;/a&gt;. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;deep NNs suffer from a lack of &lt;a href=&quot;https://datascience.stackexchange.com/questions/19112/explainability-of-algorithms&quot;&gt;explainability&lt;/a&gt;; you can view the many individual layer weights but it's near-impossible to picture how that translates into effect on output.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="8501" LastEditorUserId="8501" LastEditDate="2018-05-07T23:03:39.920" LastActivityDate="2018-05-07T23:03:39.920" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31310" PostTypeId="2" ParentId="31307" CreationDate="2018-05-07T03:59:54.960" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;[sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Use step_len=1, just the current word. after squeezing the tensor, its shape will be  [batch_size,sentence_len,embedding_size].&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Previous sentence's last word may be affected by latter sentence's first few words.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Use padding, append the last 3 to 5 words of previous sentences at the start of the current sentences, do not backpropagate for the errors of those padded words, by masking their loss. as for the case of first sentences of paragraphs, padded with a simple &lt;code&gt;&amp;lt;pad&amp;gt;&lt;/code&gt; token.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;doesn't need BiLSTM anymore? I think it still helps here.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Using BiLSTM is a must. if you want to limit step_len to be 1, by the way, you are computing the conditional probability of Y being true (correct/not correct) conditioned on the whole context (sentence). as you are feeding the sentence at once, the loss will be the summation over all single losses. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Side note:&lt;br&gt;&#xA;To batch the training examples, you also need to pad the end of sentences with the three to five words of the next ones. use &lt;code&gt;&amp;lt;pad&amp;gt;&lt;/code&gt; for sentences ending the paragraph, don't permit their contributions to the model (adjusting the weight), mask your padded examples to have zero loss. because their labels are just noise.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-07T03:59:54.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31311" PostTypeId="2" ParentId="31134" CreationDate="2018-05-07T05:16:07.227" Score="1" Body="&lt;p&gt;It maybe a little bit complicated, since I convert the reports to pandas.DataFrame for calculation. But I think it's worth it, because it works well with two or more report as well. Try below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from functools import reduce&#xA;def report_average(*args):&#xA;    report_list = list()&#xA;    for report in args:&#xA;        splited = [' '.join(x.split()) for x in report.split('\n\n')]&#xA;        header = [x for x in splited[0].split(' ')]&#xA;        data = np.array(splited[1].split(' ')).reshape(-1, len(header) + 1)&#xA;        data = np.delete(data, 0, 1).astype(float)&#xA;        avg_total = np.array([x for x in splited[2].split(' ')][3:]).astype(float).reshape(-1, len(header))&#xA;        df = pd.DataFrame(np.concatenate((data, avg_total)), columns=header)&#xA;        report_list.append(df)&#xA;    res = reduce(lambda x, y: x.add(y, fill_value=0), report_list) / len(report_list)&#xA;    return res.rename(index={res.index[-1]: 'avg / total'})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;output:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;report_average  = report_average(report_1, report_2)&#xA;print(report_average)&#xA;             precision  recall  f1-score  support&#xA;0                 0.75    0.75      0.75      2.0&#xA;1                 0.75    0.75      0.75      2.0&#xA;avg / total       0.75    0.75      0.75      4.0&#xA;&#xA;report_3 = report_2&#xA;report_average  = report_average(report_1, report_2,report_3)&#xA;print(report_average)&#xA;             precision    recall  f1-score  support&#xA;0             0.666667  0.666667  0.666667      2.0&#xA;1             0.666667  0.666667  0.666667      2.0&#xA;avg / total   0.666667  0.666667  0.666667      4.0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51685" LastActivityDate="2018-05-07T05:16:07.227" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31312" PostTypeId="2" ParentId="31291" CreationDate="2018-05-07T05:16:45.497" Score="4" Body="&lt;p&gt;When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. The train error will be the error on the data selected to train the model, and the validation error will be the data left out of the training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this reason, you have k training errors and k validation/test errors, and computing their averages will give you the quantities you are talking about.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-07T05:16:45.497" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31313" PostTypeId="2" ParentId="14412" CreationDate="2018-05-07T05:17:08.667" Score="0" Body="&lt;p&gt;It's mentioned in the paper:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;A local normalization layer normalizes the feature map&#xA;  around a local neighborhood to have unit norm and zero&#xA;  mean. It leads to feature maps that are robust to the differences&#xA;  in illumination and contrast.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They take each part of the model and normalize it separately. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for combining them, as you commented, to capture the most salient features, with under-complete representation no needs for the non-linearity.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-07T05:17:08.667" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31314" PostTypeId="1" CreationDate="2018-05-07T05:17:09.613" Score="1" ViewCount="47" Body="&lt;p&gt;I am currently learning DeepLearning and wanted to ask a few questions in relation to my current project &lt;a href=&quot;https://github.com/deepcollege/deeplearning/blob/master/030-chatbot/chatbot_simple.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/deepcollege/deeplearning/blob/master/030-chatbot/chatbot_simple.py&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First of all, how do you generate an useful Tensorboard model graph? As you can see from the code, current session.graph isn't outputing anything helpful or comprehensible. My model is basically a chatbot model based on Seq2Seq, how do you generate more simplified and useful Tensorboard model graph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My second question is, in NLP situation (more specifically a chatbot), what other things can you do to analyse the model in Tensorboard? And how do you enable them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="51562" LastActivityDate="2018-05-07T05:17:09.613" Title="How do you use Tensorboard to visualise my Chatbot? What can I learn from it?" Tags="|machine-learning|deep-learning|nlp|tensorflow|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31315" PostTypeId="1" CreationDate="2018-05-07T05:33:13.923" Score="8" ViewCount="13737" Body="&lt;p&gt;I am trying to solve a multiclass classification problem. The dataset is balanced. I have been using accuracy as a performace metric till now. Are there any other good performance metrics for this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already know about precision and recall but as far as I know they are used when the dataset is imbalanced.&lt;/p&gt;&#xA;" OwnerUserId="37063" LastActivityDate="2018-12-19T15:47:12.627" Title="Good performance metrics for multiclass classification problem besides accuracy?" Tags="|multiclass-classification|metric|" AnswerCount="3" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31316" PostTypeId="2" ParentId="31315" CreationDate="2018-05-07T05:52:02.253" Score="2" Body="&lt;p&gt;Take a look at multi-class confusion matrix. maybe the model has some difficulty on a subset of the classes.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-07T05:52:02.253" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31317" PostTypeId="1" CreationDate="2018-05-07T06:00:12.483" Score="3" ViewCount="826" Body="&lt;p&gt;It was my understanding that when trying to create a tensorflow session, if the amount of available GPU memory is not enough for what the program asks for, the program should raise an error and exit. But it seems to be not this case, and this behavior is very annoying and hard to detect especially when using Jupyter Notebook.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was trying to run the following program (supposedly debug.py)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;a1 = tf.constant([0.0,0.0])&#xA;b1 = tf.constant([2.0,2.0])&#xA;&#xA;mse1, update1 = tf.metrics.mean_squared_error(a1,b1)&#xA;mse2 = tf.losses.mean_squared_error(a1,b1)&#xA;with tf.Session() as sess:&#xA;    sess.run(tf.local_variables_initializer())&#xA;    update1.eval() &#xA;    print(mse1.eval())  &#xA;    print(mse2.eval())  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This program simply calculate the mean_squared_error (MSE) between the two vectors a1 and b1, and the correct output (from the two print() lines) should be 4.0 and 4.0. If no other python programs are using my GPU, this is indeed the output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Unexpected behavior&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If another program is using the GPU (say, another jupyter notebook running something with tensorflow without limiting its GPU usage by &lt;code&gt;gpu_options.per_process_gpu_memory_fraction&lt;/code&gt;), then the above code would output something different. The first print line (&lt;code&gt;mse1&lt;/code&gt;) would output some random number, e.g. &quot;0.0&quot; (most common), &quot;3e-45&quot;, &quot;nan&quot;, etc, which is obviously wrong. The second print line (&lt;code&gt;mse2&lt;/code&gt;) always print the correct answer: 4.0. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Running this code by &lt;code&gt;python debug.py&lt;/code&gt; SOMETIMES generate an error message&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;E tensorflow/stream_executor/cuda/cuda_driver.cc:924] failed to allocate 176.44M (185008128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;But the program does NOT exit, and still prints the results, although incorrect.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other cases, running this code by &lt;code&gt;python debug.py&lt;/code&gt;  generate another error message&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;E tensorflow/core/common_runtime/direct_session.cc:170] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 8502771712&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and the program quits without printing any output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most annoying thing is, running the same code in Jupyte Notebook ALWAYS generates no error or warning messages, though the output is incorrect. That's why I always thought I misunderstood about how to call &lt;code&gt;tf.metrics.mean_squared_error&lt;/code&gt; without realizing there could be a memory error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Tb1Cn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Tb1Cn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;System&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I verified the problem with my two computers running Tensorflow 1.3 and 1.4 respectively, both with CUDA 8, Ubuntu 16.04 and Nvidia GPUs (1080 and 770M). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;My Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(0) Is there anything wrong with my code?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(1) If you also use Tensorflow with Jupyter Notebook, do you ever get the wrong output (printing anything other than 4.0, 4.0) when running the above code? If so, does Jupyter alerts you about any error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(2) What is the nature of this problem of returning the wrong &lt;code&gt;mse1&lt;/code&gt;? Is it a GPU memory leak?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(3) Why could there be such inconsistent behavior when the problem occurs? To be specific, why the &lt;code&gt;debug.py&lt;/code&gt; sometimes quits, sometimes not? Why Jupyter Notebook never quits or raises an error?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(4) Why the &lt;code&gt;mse2&lt;/code&gt; never goes wrong when &lt;code&gt;mse1&lt;/code&gt; goes wrong? My understanding is that &lt;code&gt;tf.metrics.mean_squared_error&lt;/code&gt; is a streaming operation and therefore need to store some variable in a different place.&lt;/p&gt;&#xA;" OwnerUserId="14006" LastActivityDate="2018-05-07T06:00:12.483" Title="Why Tensorflow does NOT quit when CUDA_ERROR_OUT_OF_MEMORY" Tags="|tensorflow|gpu|jupyter|" AnswerCount="0" CommentCount="0" ClosedDate="2020-12-27T20:24:25.540" ContentLicense="CC BY-SA 4.0" />
  <row Id="31319" PostTypeId="2" ParentId="31315" CreationDate="2018-05-07T06:53:08.240" Score="2" Body="&lt;p&gt;&lt;a href=&quot;http://wiki.fast.ai/index.php/Log_Loss#Multi-class_Classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi class log-loss&lt;/a&gt; is used in many data science competitions. Although not so easily interpretable as accuracy, it penalizes based on your confidence on your predictions. If the models you are using output probabilities, it may be a better way than accuracy to compare and select different models on your validation data, as it takes into account the probabilities and not only the amount of correct predictions.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-07T06:53:08.240" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31320" PostTypeId="1" AcceptedAnswerId="31339" CreationDate="2018-05-07T09:08:23.440" Score="3" ViewCount="3038" Body="&lt;p&gt;I want to have a global IoU metric for each class in a segmentation model with a neural net. The idea is, once the net is trained, doing the forward pass over all training examples an calculate the IoU, I'm thinking in two approaches (for each class):&#xA;1) Calculate IoU for each training instance, and finally, calculate the mean IoU (per class)&#xA;2) Accumulate the intersections and unions along all the training instances, (per class) and finally taking the ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To illustrate the problem, let's take two training instances in which for class=0, intersection_1 = 2, intersection_2 = 3, union_1=7, union_2=6. The mean IoU (approach 1) wil be 0.3929 and the second approach will be 5/13 = 0.3846. What method do you think will give better/unbiased result?&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2020-11-10T19:56:20.783" Title="Calculate average Intersection over Union" Tags="|model-evaluations|accuracy|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31322" PostTypeId="1" CreationDate="2018-05-07T09:33:46.267" Score="2" ViewCount="763" Body="&lt;p&gt;In R,using dplyr package, I tried the function &quot;summarise&quot; and I expect the result to show along with the groupby field. However, all of a sudden I see summarized output but without the groupby filed which makes the results meaningless. Any one any idea?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if data has ages and salaries for members belonging to a team and I try to view the Average Age and Salary by Team the code below returns just the averages without the team names&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Code: &#xA;df_summ = df %&amp;gt;% group_by(Team)%&amp;gt;% summarise(AvgAge= mean(Age),AvgSalary= mean(Salary)) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So without the groupby column &quot;Team&quot; the result is meaningless&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What Am I missing here?&lt;/p&gt;&#xA;" OwnerUserId="51716" LastEditorUserId="40853" LastEditDate="2018-05-07T09:43:32.970" LastActivityDate="2018-05-07T09:50:19.947" Title="Group_by field is not showing in the summarise output in R" Tags="|r|dplyr|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31323" PostTypeId="1" CreationDate="2018-05-07T09:38:09.647" Score="2" ViewCount="109" Body="&lt;p&gt;I am in the research phase of a long project and am willing to get some useful feedback from your side about the most appropriate project path to take.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Current situation:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A large team of so called production editors (PEs) is semi-automatically processing MS word files in order to do the copy-editing and layout.&lt;/li&gt;&#xA;&lt;li&gt;This is currently semi-automatized with word macros where the PEs will apply different macros on different text paragraphs in order to set the appropriate style, position, font etc.&lt;/li&gt;&#xA;&lt;li&gt;Still many actions are manual and depending on the human eye checks, resulting at the end many mistakes and an average quality&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Plan&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Start using machine learning, later maybe neuronal networks, statistics probability and artificial intelligence&lt;/li&gt;&#xA;&lt;li&gt;The usage of machine learning (to start with) could be in the way that the system will learn the manual tasks done by the user (meaning the tool was not able to do those correctly in the automatized way) and then try to apply those in next similar situation&lt;/li&gt;&#xA;&lt;li&gt;The term of similar situations is important here as this will imply natural language processing (NLP). The staff is dealing with word documents that could contain anything (within a similar layout)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The dilemma of the day is :) :&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where to start from in order to do some use case analyses? What I currently have:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;More than 100k word files that are correctly done (except the user mistakes that may still lay around)&lt;/li&gt;&#xA;&lt;li&gt;A list of rules that apply to the word documents describing the exact styles that needs to be used (non exhaustive)&lt;/li&gt;&#xA;&lt;li&gt;Basic Machine Learning knowledge (starting with AI studying)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="51717" LastEditorUserId="50146" LastEditDate="2018-05-11T04:07:19.213" LastActivityDate="2018-05-11T04:07:19.213" Title="Machine Learning and Natural Language Processing : Project Initiation" Tags="|machine-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31325" PostTypeId="1" CreationDate="2018-05-07T10:05:56.213" Score="1" ViewCount="466" Body="&lt;p&gt;I am using tibco spotfire and I have columns with data like this &quot;0001V&quot;, &quot;0002V&quot;, &quot;0003N&quot;. I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I should use &quot;limit data using expression&quot; but I don't know how to write the expression. So it should be something like &quot;if X ends in 'N' delete X&quot;... Could you help me please?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="41991" LastActivityDate="2018-05-07T10:05:56.213" Title="Delete strings with a specific last character in tibco spotfire" Tags="|dataset|data-cleaning|data|preprocessing|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-11T11:04:18.207" ContentLicense="CC BY-SA 4.0" />
  <row Id="31326" PostTypeId="2" ParentId="31266" CreationDate="2018-05-07T10:18:26.960" Score="1" Body="&lt;ol&gt;&#xA;&lt;li&gt;It depends on the context of your problem.&#xA;If it is an sequence of images with not great quality then we need to perform some morphological operation on it e.g.grayscale,binary image etc.&#xA;Also if the digits are distorted,then we could translate it up-to certain extent for model to learn easily.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="51192" LastActivityDate="2018-05-07T10:18:26.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31328" PostTypeId="1" CreationDate="2018-05-07T11:20:20.793" Score="1" ViewCount="90" Body="&lt;p&gt;The company I work for produces packaging paper on a mass scale; my task is to see if machine learning would be able to help identify paper breaks on the machines before they actually happen.  We have sensors and high speed camera systems on the machine for real-time analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know which machine learning algorithms would be applicable for this problem?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the paper making process, a continuous sheet of paper is formed and fed through a series of machines.  If the paper breaks, the process must be restarted.  We have machines for detecting a paper break after one has already occurred, but my goal is to use machine learning to detect a break before it happens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could an algorithm be created that could constantly analyze many different points on the machine and based on that data be able to predict a sheet break or not?&lt;/p&gt;&#xA;" OwnerUserId="51721" LastEditorUserId="33551" LastEditDate="2018-05-10T21:39:00.043" LastActivityDate="2018-05-10T21:39:00.043" Title="Machine Learning in Paper making machines" Tags="|machine-learning|" AnswerCount="0" CommentCount="3" ClosedDate="2018-05-08T11:35:51.937" ContentLicense="CC BY-SA 4.0" />
  <row Id="31329" PostTypeId="1" AcceptedAnswerId="31334" CreationDate="2018-05-07T11:51:59.870" Score="1" ViewCount="303" Body="&lt;p&gt;I have a dataset of about 1k samples and I want to apply some unsuspervised techniques in order to clustering and visualization of these data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data can be interpreted as a table of a spreadsheet and unfortunately it doensn't have a very defined pattern of structure. The number of table lines varies, but not the columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data is structured like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sample 1:&#xA;&#xA;{&#xA;  &quot;table1&quot;: {&#xA;    &quot;column1&quot;: [&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;&#xA;    ],&#xA;    &quot;column2&quot;: [&#xA;      &quot;2017-04-16 10:00&quot;,&#xA;      &quot;2017-04-16 10:00&quot;,&#xA;      &quot;2017-04-16 10:00&quot;&#xA;    ],&#xA;    &quot;column3&quot;: [&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;&#xA;    ],&#xA;    &quot;column4&quot;: [&#xA;      &quot;name X&quot;,&#xA;      &quot;name Y&quot;,&#xA;      &quot;name Z&quot;&#xA;    ],&#xA;    &quot;column5&quot;: [&#xA;      &quot;0&quot;,&#xA;      &quot;0&quot;,&#xA;      &quot;0&quot;&#xA;    ],&#xA;  }&#xA;}&#xA;&#xA;&#xA;sample 2:&#xA;&#xA;{&#xA;  &quot;table1&quot;: {&#xA;    &quot;column1&quot;: [&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;&#xA;    ],&#xA;    &quot;column2&quot;: [&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;,&#xA;      &quot;2017-04-10 22:00&quot;&#xA;    ],&#xA;    &quot;column3&quot;: [&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;,&#xA;      &quot;-&quot;&#xA;    ],&#xA;    &quot;column4&quot;: [&#xA;      &quot;name A&quot;,&#xA;      &quot;name Z&quot;,&#xA;      &quot;name B&quot;,&#xA;      &quot;name X&quot;,&#xA;      &quot;name C&quot;,&#xA;      &quot;name D&quot;,&#xA;      &quot;name E&quot;,&#xA;      &quot;name F&quot;&#xA;    ],&#xA;    &quot;coumn5&quot;: [&#xA;      &quot;&quot;,&#xA;      &quot;&quot;,&#xA;      &quot;3&quot;,&#xA;      &quot;1&quot;,&#xA;      &quot;0&quot;,&#xA;      &quot;3&quot;,&#xA;      &quot;0&quot;,&#xA;      &quot;0&quot;&#xA;    ]&#xA;  }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;These samples comes from alarms generated by a systems that collects informations from a lot of nodes (these nodes are named as &quot;name A&quot;, &quot;name B&quot;...). My objective is to transform these data into a matrix (n_samples x n_features) to apply clustering and visualization algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I work with these data for unsupervised training? Is padding a way forward for this problem? If so, how can I apply the padding on this case?&lt;/p&gt;&#xA;" OwnerUserId="51719" LastEditorUserId="51719" LastEditDate="2018-05-07T19:04:43.110" LastActivityDate="2018-05-07T19:04:43.110" Title="Training with data of different shapes. Is padding an alternative?" Tags="|python|visualization|data-cleaning|data|unsupervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31330" PostTypeId="1" AcceptedAnswerId="31331" CreationDate="2018-05-07T13:00:02.343" Score="3" ViewCount="52084" Body="&lt;p&gt;I was trying the Keras CNN Stater Code on Ubuntu 16.04, from the below link:&#xA;&lt;a href=&quot;https://www.hackerearth.com/challenge/competitive/deep-learning-3/machine-learning/predict-the-energy-used-612632a9/#c144537&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.hackerearth.com/challenge/competitive/deep-learning-3/machine-learning/predict-the-energy-used-612632a9/#c144537&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get “MemoryError:” for&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train = np.array(train_img, np.float32) / 255.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any idea, what should I be doing?&lt;/p&gt;&#xA;" OwnerUserId="51724" LastActivityDate="2018-05-07T13:30:22.180" Title="MemoryError for np.array" Tags="|python|jupyter|numpy|" AnswerCount="1" CommentCount="3" ClosedDate="2018-05-09T06:14:46.513" ContentLicense="CC BY-SA 4.0" />
  <row Id="31331" PostTypeId="2" ParentId="31330" CreationDate="2018-05-07T13:30:22.180" Score="3" Body="&lt;p&gt;MemoryError is exactly what it means, you have run out of memory in your RAM for your code to execute. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When this error occurs it is likely because you have loaded the entire data into memory. For large datasets you will want to use &lt;strong&gt;batch processing&lt;/strong&gt;. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access it in batches. If you are using Keras there is a helper class with a very efficient implementation of batch processing. Take a look at this &lt;a href=&quot;https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt;. This is a good starting point for avoiding MemoryError. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As a short term fix you can train your model using a subset of the data available to you and discard the rest. Doing this really is a shame however.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-07T13:30:22.180" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31332" PostTypeId="1" CreationDate="2018-05-07T14:53:22.000" Score="2" ViewCount="55" Body="&lt;p&gt;What algorithms should I research to sort pie charts by similarity?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The real-world problem I'm working with is a collection of race data for a few dozen cities — each race listed as percent of the city's population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm interested in selecting a single city and finding other cities with similar racial proportions.&lt;/p&gt;&#xA;" OwnerUserId="49102" LastEditorUserId="49102" LastEditDate="2018-05-07T15:04:46.157" LastActivityDate="2018-05-07T15:04:46.157" Title="How to find similar pie charts?" Tags="|algorithms|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31333" PostTypeId="1" CreationDate="2018-05-07T14:56:50.580" Score="1" ViewCount="482" Body="&lt;p&gt;I have a script which I wrote using python and tflearn. I created a regression neural network model which takes in chemical analysis of wine as input and predicts a score out of 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset: &lt;a href=&quot;http://archive.ics.uci.edu/ml/datasets/Wine+Quality&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://archive.ics.uci.edu/ml/datasets/Wine+Quality&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem I have is that the prediction of my model is very bad. I'm also new to tflearn, I have only coded classification neural networks(twice). So, I'm a complete beginner in coding regression and in using tflearn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;import tflearn&#xA;from tflearn.layers.core import input_data, fully_connected, dropout&#xA;from tflearn.layers.estimator import regression&#xA;from sklearn.model_selection import train_test_split&#xA;from sklearn.preprocessing import StandardScaler&#xA;import matplotlib.pyplot as plt&#xA;&#xA;&#xA;def preprocess():&#xA;&#xA;    data_source_red = 'F:\Gautam\...\winequality-red.csv'&#xA;    data_source_white = 'F:\Gautam\...\winequality-white.csv'&#xA;&#xA;    data_red = pd.read_csv(data_source_red, index_col=False, sep=';')&#xA;    data_white = pd.read_csv(data_source_white, index_col=False, sep=';')&#xA;&#xA;    data = pd.concat([data_red, data_white])&#xA;    data = data.dropna(inplace=False)&#xA;&#xA;    x = data[data.columns[0:11]].values&#xA;&#xA;    y = data[data.columns[11]].values&#xA;&#xA;    sc = StandardScaler()&#xA;    x = sc.fit_transform(x)&#xA;&#xA;    y = np.expand_dims(y, -1)&#xA;&#xA;    x = np.float32(x)&#xA;    y = np.float32(y)&#xA;&#xA;    return (x, y)&#xA;&#xA;&#xA;x, y = preprocess()&#xA;&#xA;train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2)&#xA;&#xA;network = input_data(shape=[None, 11], name='Input_layer')&#xA;&#xA;network = fully_connected(network, 5, activation='relu', name='Hidden_layer_1')&#xA;&#xA;network = fully_connected(network, 1, activation='linear', name='Output_layer')&#xA;&#xA;network = regression(network, batch_size=64, optimizer='sgd', learning_rate=0.2, loss='mean_square', metric='R2')&#xA;&#xA;model = tflearn.DNN(network)&#xA;&#xA;model.fit(train_x, train_y, show_metric=True, run_id='wine_regression', validation_set=0.1, n_epoch=10)&#xA;&#xA;result = model.evaluate(test_x, test_y)&#xA;print('Accuracy is %0.2f%%' % (result[0] * 100))&#xA;&#xA;pred_y = model.predict(test_x)&#xA;&#xA;plt.plot(test_y, color = 'red', label = 'Real data')&#xA;plt.plot(pred_y, color = 'blue', label = 'Predicted data')&#xA;plt.title('Prediction')&#xA;plt.legend()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Again, the prediction is very bad. Also, the loss and R2 values go hectic. Sometimes, the loss is low(6.45) and sometimes very high(23445.45). The same goes for R2 value too. Sometimes, R2 value goes above 1.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if the loss is minimum(0.1) and R2 as 0.95, the graph shows that the actual value varies lot from the predicted values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What mistake am I doing? Why is my prediction very bad? And why are the values R2 and loss too high sometimes and too low sometimes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Am I missing something here? This is my first regression neural network, so I don't know much about this. I hope that my question is clear. Thanks.&lt;/p&gt;&#xA;" OwnerUserId="51733" LastActivityDate="2018-05-07T14:56:50.580" Title="Regression Neural Network using tflearn" Tags="|machine-learning|neural-network|deep-learning|regression|tflearn|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31334" PostTypeId="2" ParentId="31329" CreationDate="2018-05-07T15:43:35.667" Score="1" Body="&lt;p&gt;Whether or not padding is ppropriate really depends on the entire structure of your dataset, how relevant the different variables/columns are and also the type of model you want to run at the end.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Padding would be used, whereby you would have to fix the length of each sample (either to the length of the longest sample, or to a fixed length - longer samples would be trimmed or filtered somehow to fit into that length). Variables that are strings can be padded with  empty strings, variables with number can be padded with zeros. There are however many other ways to pad, e.g. using the average of a numerical variable, or even model-based padding, padding with values that &quot;make most sense&quot; for filling gaps in that specific sample. Getting deep into it like that might more generally be called imputation, instead of padding - it is common in time series data, where gaps aren't always at one end of a sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below I outline one approach to padding or standardizing the length of each sample. It it not specifically padding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you did not mention a programming language, I will give and code snippet in &lt;strong&gt;Python&lt;/strong&gt;, but the same is easily achievable in other languages such as &lt;strong&gt;R&lt;/strong&gt; and &lt;strong&gt;Julia&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;The Approach&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Based on the two examples you provide, it seems each example would be a calendar day, on which there are a variable number of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are also columns that are strings, and others are strings of numbers (e.g. column 5 in sample 2).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In time-series analysis in general, it is desirable to have a continuous frequency of data. That means have one day give one input. So my approach would be to make your data into a form that resembles a single input for each of the variables (i.e. columns) of each sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is general approach, and you will have to try things out or do more research on how this would look in reality for your specific data at large.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Timestamps&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I would use these as a kind of index, like the index in a &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pandas DataFrame&lt;/a&gt;. One row = one timestamp. Multiple variables are then different columns.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Dealing with strings&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;I will assume that your dataset has a finite number of possible strings in each column. For example, that column 4 (holding &lt;em&gt;names&lt;/em&gt;), will always hold a name from a given set. One could perform &lt;code&gt;set(table2['column 4'])&lt;/code&gt; to see which values there are (removing duplicates). Or even:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Gather a single list containing all strings in column 4 of all samples&#xA;all_names = []&#xA;[]    # list comprehension to loop&#xA;&#xA;# Check how many strings there are&#xA;from collections import Counter&#xA;counter = Counter(table2['column4'])&#xA;print(len(counter))                 # see how many unique values exist&#xA;print(counter)                      # see how many times each string appears&#xA;print(counter.most_common(5))       # see the most common 5 strings&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now assuming this shows there is a finite number (more than likely the case), You could look into using &lt;a href=&quot;https://machinelearningmastery.com/sparse-matrices-for-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;a sparse representation&lt;/a&gt; of each sample (that means for each day). For example, if all the words in the entire dataset were: &lt;code&gt;['hi', 'hello', 'wasup', 'yo', 'bonjour']&lt;/code&gt; (duplicates removed), then for one single sample with column 4 holding e.g. &lt;code&gt;['hi', 'hello', 'yo', 'hi']&lt;/code&gt;, your sparse representation for this sample would be: &lt;code&gt;[2, 1, 0, 1, 0]&lt;/code&gt;, because the sample has two 'hi', one 'hello', zero 'wasup' and so on. This sparse representation would then be your single input for column 4 for the timestamp (that single sample).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It might be worth looking into something like the &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DictVectorizer&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CountVectorizer&lt;/a&gt; from Scikit-Learn.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Dealing with number columns&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;As I mentioned right at the beginning, you could pad these to a chosen length, perhaps matching the length of the string based representation above (or not!), depending on your final model.&#xA;You can then pad the inputs with a value that makes sense for your model (see the kind of options I mentioned at the beginning of my answer).&#xA;This should land you with, once again, a single vector for the given day, containing the information in the numerical column.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-07T15:43:35.667" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31335" PostTypeId="2" ParentId="20339" CreationDate="2018-05-07T16:01:56.960" Score="2" Body="&lt;p&gt;The &lt;strong&gt;sampling frequency&lt;/strong&gt;, or sample rate, is the number of equal-spaced samples per unit of time. For instance, if you have 96 equally spaced observation per day, then you sampling rate is 96/day, or 96/24/3600=0.0011 Hz. Hz, which means per second, is widely used for sample rate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;strong&gt;frequency of your data&lt;/strong&gt; is different. I you signal is periodic or approximately periodic (say, the similar energy consumption patterns repeat everyday), the the frequency is the number of occurrence of that repeating pattern per unit of time. More generally, if your data does not look like periodic, it would help to conduct &lt;strong&gt;Fourier Analysis&lt;/strong&gt; to understand the frequency components in your data. The Fourier Analysis (Fourier Transform) is the operation to transform time-domain data into the frequency domain.&lt;/p&gt;&#xA;" OwnerUserId="14006" LastActivityDate="2018-05-07T16:01:56.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31336" PostTypeId="1" CreationDate="2018-05-07T17:06:04.197" Score="1" ViewCount="556" Body="&lt;p&gt;I am using &lt;code&gt;GridSearchCV&lt;/code&gt; in order to find best estimator with best hiperparameters. I also want to check it once against a small portion of data used in cross validation again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print 'Load data'&#xA;iris = load_iris()&#xA;X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target)&#xA;&#xA;tree_param_grid = {&#xA;    'criterion': ['gini', 'entropy'],&#xA;    'splitter': ['best', 'random'],&#xA;    'max_depth': np.arange(2, 8),&#xA;}&#xA;&#xA;tree = GridSearchCV(DecisionTreeClassifier(), tree_param_grid)&#xA;&#xA;tree.fit(X_train, y_train)&#xA;tree_preds = tree.predict(X_test)&#xA;tree_performance = accuracy_score(y_test, tree_preds)&#xA;&#xA;print '\nDecision tree:'&#xA;print 'Best params: ', tree.best_params_&#xA;print 'Best scores: ', tree.best_score_&#xA;print 'My score: ', tree_performance&#xA;&#xA;best = tree.best_estimator_&#xA;best.fit(X_train, y_train)&#xA;best_preds = best.predict(X_test)&#xA;best_performance = accuracy_score(y_test, best_preds)&#xA;&#xA;print 'Best score', best_performance&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What I do not understand is this: best_performance and tree_performance have two different values. Why is that?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought that when I perform &lt;code&gt;fit&lt;/code&gt; method on an object returned from &lt;code&gt;GridSearch&lt;/code&gt;, I will automatically perform fitting on best estimator found in Grid Search. It turns out, it's not true - I have to manually get the best estimator from Grid Search. So, where does &lt;code&gt;fit&lt;/code&gt; method in first usage come from? What is returned from &lt;code&gt;GridSearchCV&lt;/code&gt; if not the best estimator itself?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;I added a result from printing these two objects into console:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tree GridSearchCV(cv=None, error_score='raise',&#xA;       estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,&#xA;            max_features=None, max_leaf_nodes=None,&#xA;            min_impurity_decrease=0.0, min_impurity_split=None,&#xA;            min_samples_leaf=1, min_samples_split=2,&#xA;            min_weight_fraction_leaf=0.0, presort=False, random_state=None,&#xA;            splitter='best'),&#xA;       fit_params=None, iid=True, n_jobs=1,&#xA;       param_grid={'splitter': ['best', 'random'], 'criterion': ['gini', 'entropy'], 'max_depth': array([2, 3, 4, 5, 6, 7])},&#xA;       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',&#xA;       scoring=None, verbose=0)&#xA;best DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,&#xA;            max_features=None, max_leaf_nodes=None,&#xA;            min_impurity_decrease=0.0, min_impurity_split=None,&#xA;            min_samples_leaf=1, min_samples_split=2,&#xA;            min_weight_fraction_leaf=0.0, presort=False, random_state=None,&#xA;            splitter='best')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41740" LastEditorUserId="41740" LastEditDate="2018-05-07T19:08:24.997" LastActivityDate="2018-05-07T19:08:24.997" Title="Different values from GridSearch estimation" Tags="|scikit-learn|grid-search|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31337" PostTypeId="1" CreationDate="2018-05-07T17:10:50.240" Score="1" ViewCount="65" Body="&lt;p&gt;this code works with a VGG caffe model (VGG_ILSVRC_19_layers.caffemodel).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I tried to change the model to a LeNet caffemodel (below the prototxt file), I got the following error. &#xA;In 6 module of nn.Sequential:&#xA;xx/.luarocks/share/lua/5.1/nn/Linear.lua:66: size mismatch, m1: [1 x 594050], m2: [140450 x 500] at xx/torch/pkg/torch/lib/TH/generic/THTensorMath.c:1293 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions? Many thanks!  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;===========================&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;  Code&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;net = loadcaffe.load(opt.cnn_proto, opt.cnn_model, opt.backend);&#xA;--for i = 1, 9 do  -- Lines removed when using LeNet&#xA;--    net:remove()&#xA;--end&#xA;print(net)&#xA;&#xA;if opt.gpuid &amp;gt;= 0 then&#xA;    require 'cutorch'&#xA;    require 'cunn'&#xA;    cutorch.setDevice(opt.gpuid+1)&#xA;    net = net:cuda()&#xA;end&#xA;net:evaluate()&#xA;&#xA;function loadim(imname)&#xA;    im = image.load(imname)&#xA;    im = image.scale(im,448,448)&#xA;    if im:size(1) == 1 then&#xA;        im2=torch.cat(im,im,1)&#xA;        im2=torch.cat(im2,im,1)&#xA;        im=im2&#xA;    elseif im:size(1) == 4 then&#xA;        im=im[{{1,3},{},{}}]&#xA;    end&#xA;    im = im * 255;&#xA;    im2 = im:clone()&#xA;    im2[{{3},{},{}}] = im[{{1},{},{}}]-123.68&#xA;    im2[{{2},{},{}}] = im[{{2},{},{}}]-116.779&#xA;    im2[{{1},{},{}}] = im[{{3},{},{}}]-103.939&#xA;    return im2&#xA;end&#xA;&#xA;local image_root = opt.image_root&#xA;&#xA;local file = io.open(opt.input_json, 'r')&#xA;local text = file:read()&#xA;file:close()&#xA;json_file = cjson.decode(text)&#xA;&#xA;print(json_file['unique_img_train'])&#xA;&#xA;local train_list={}&#xA;for i,imname in pairs(json_file['unique_img_train']) do&#xA;    table.insert(train_list, image_root .. imname)&#xA;end&#xA;&#xA;local test_list={}&#xA;for i,imname in pairs(json_file['unique_img_test']) do&#xA;    table.insert(test_list, image_root .. imname)&#xA;end&#xA;&#xA;local batch_size = opt.batch_size&#xA;local sz=#train_list&#xA;local feat_train=torch.FloatTensor(sz, 14, 14, 512)&#xA;print(string.format('processing %d images...',sz))&#xA;for i = 1, sz, batch_size do&#xA;    xlua.progress(i, sz)&#xA;    r = math.min(sz, i+batch_size-1)&#xA;    ims = torch.DoubleTensor(r-i+1, 3, 448,448)&#xA;    for j = 1, r-i+1 do&#xA;        ims[j] = loadim(train_list[i+j-1])&#xA;    end&#xA;    if opt.gpuid &amp;gt;= 0 then&#xA;        ims = ims:cuda()&#xA;    end&#xA;    net:forward(ims)&#xA;    feat_train[{{i,r}, {}}] = net.output:permute(1,3,4,2):contiguous():float()&#xA;    collectgarbage()&#xA;end&#xA;&#xA;local train_h5_file = hdf5.open(opt.out_name_train, 'w')&#xA;train_h5_file:write('/images_train', feat_train)&#xA;train_h5_file:close()&#xA;&#xA;print('DataLoader loading h5 file: ', 'data_train')&#xA;local sz = #test_list&#xA;local feat_test = torch.FloatTensor(sz, 14, 14, 512)&#xA;print(string.format('processing %d images...',sz))&#xA;for i = 1, sz, batch_size do&#xA;    xlua.progress(i, sz)&#xA;    r = math.min(sz, i + batch_size-1)&#xA;    ims = torch.DoubleTensor(r-i+1, 3, 448, 448)&#xA;    for j = 1, r-i+1 do&#xA;        ims[j] = loadim(test_list[i+j-1])&#xA;    end&#xA;    if opt.gpuid &amp;gt;= 0 then&#xA;        ims = ims:cuda()&#xA;    end&#xA;    net:forward(ims)&#xA;    feat_test[{{i,r}, {}}] = net.output:permute(1,3,4,2):contiguous():float()&#xA;    collectgarbage()&#xA;end&#xA;&#xA;local test_h5_file = hdf5.open(opt.out_name_test, 'w')&#xA;test_h5_file:write('/images_test', feat_test)&#xA;test_h5_file:close()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;========================&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;LeNet-deploy.prototxt&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input: &quot;data&quot;&#xA;input_shape {&#xA;  dim: 1&#xA;  dim: 3&#xA;  dim: 224&#xA;  dim: 224&#xA;}&#xA;layer {&#xA;  name: &quot;scale&quot;&#xA;  type: &quot;Power&quot;&#xA;  bottom: &quot;data&quot;&#xA;  top: &quot;scaled&quot;&#xA;  power_param {&#xA;    scale: 0.0125000001863&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;conv1&quot;&#xA;  type: &quot;Convolution&quot;&#xA;  bottom: &quot;scaled&quot;&#xA;  top: &quot;conv1&quot;&#xA;  param {&#xA;    lr_mult: 1.0&#xA;  }&#xA;  param {&#xA;    lr_mult: 2.0&#xA;  }&#xA;  convolution_param {&#xA;    num_output: 20&#xA;    kernel_size: 5&#xA;    stride: 1&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;pool1&quot;&#xA;  type: &quot;Pooling&quot;&#xA;  bottom: &quot;conv1&quot;&#xA;  top: &quot;pool1&quot;&#xA;  pooling_param {&#xA;    pool: MAX&#xA;    kernel_size: 2&#xA;    stride: 2&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;conv2&quot;&#xA;  type: &quot;Convolution&quot;&#xA;  bottom: &quot;pool1&quot;&#xA;  top: &quot;conv2&quot;&#xA;  param {&#xA;    lr_mult: 1.0&#xA;  }&#xA;  param {&#xA;    lr_mult: 2.0&#xA;  }&#xA;  convolution_param {&#xA;    num_output: 50&#xA;    kernel_size: 5&#xA;    stride: 1&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;pool2&quot;&#xA;  type: &quot;Pooling&quot;&#xA;  bottom: &quot;conv2&quot;&#xA;  top: &quot;pool2&quot;&#xA;  pooling_param {&#xA;    pool: MAX&#xA;    kernel_size: 2&#xA;    stride: 2&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;ip1&quot;&#xA;  type: &quot;InnerProduct&quot;&#xA;  bottom: &quot;pool2&quot;&#xA;  top: &quot;ip1&quot;&#xA;  param {&#xA;    lr_mult: 1.0&#xA;  }&#xA;  param {&#xA;    lr_mult: 2.0&#xA;  }&#xA;  inner_product_param {&#xA;    num_output: 500&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;relu1&quot;&#xA;  type: &quot;ReLU&quot;&#xA;  bottom: &quot;ip1&quot;&#xA;  top: &quot;ip1&quot;&#xA;}&#xA;layer {&#xA;  name: &quot;ip2&quot;&#xA;  type: &quot;InnerProduct&quot;&#xA;  bottom: &quot;ip1&quot;&#xA;  top: &quot;ip2&quot;&#xA;  param {&#xA;    lr_mult: 1.0&#xA;  }&#xA;  param {&#xA;    lr_mult: 2.0&#xA;  }&#xA;  inner_product_param {&#xA;    num_output: 5&#xA;    weight_filler {&#xA;      type: &quot;xavier&quot;&#xA;    }&#xA;    bias_filler {&#xA;      type: &quot;constant&quot;&#xA;    }&#xA;  }&#xA;}&#xA;layer {&#xA;  name: &quot;softmax&quot;&#xA;  type: &quot;Softmax&quot;&#xA;  bottom: &quot;ip2&quot;&#xA;  top: &quot;softmax&quot;&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51740" LastEditorUserId="47488" LastEditDate="2018-05-07T19:35:30.530" LastActivityDate="2018-05-07T19:35:30.530" Title="Replacing VGG by LeNet Caffe model" Tags="|caffe|torch|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31338" PostTypeId="1" AcceptedAnswerId="31376" CreationDate="2018-05-07T17:31:01.273" Score="6" ViewCount="4399" Body="&lt;p&gt;I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;          Q1        Q2        Q3&#xA;Q1  1.000000  0.707568  0.014746&#xA;Q2  0.707568  1.000000 -0.039130&#xA;Q3  0.014746 -0.039130  1.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Why is it symmetric? Why is Q1:Q2, the same as Q2:Q1? Shouldn't they be inverses of each other? How do I read this and understand the relation? &lt;/p&gt;&#xA;" OwnerUserId="51743" LastEditorUserId="381" LastEditDate="2018-05-07T20:46:51.763" LastActivityDate="2020-06-27T11:04:45.983" Title="Why is a correlation matrix symmetric?" Tags="|dataset|visualization|data|correlation|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31339" PostTypeId="2" ParentId="31320" CreationDate="2018-05-07T17:56:22.327" Score="1" Body="&lt;p&gt;The two approaches won't usually make a big difference if all your images and objects are of reasonable size. By reasonable I mean you don't are not working on some objects that are only a few pixels big.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would usually prefer the second approach. One particular reason is you don't have to worry about instances where both I and U are 0, which could happen frequently at the beginning of your training stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From my experience most machine learning software adopt the second approach. For instance, the &lt;code&gt;mean_iou&lt;/code&gt; in Tensorflow simply flatten the input tensor into a vector before calculating the IoU for each class.&lt;/p&gt;&#xA;" OwnerUserId="14006" LastActivityDate="2018-05-07T17:56:22.327" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31340" PostTypeId="2" ParentId="31338" CreationDate="2018-05-07T18:33:38.180" Score="3" Body="&lt;p&gt;Intuitively, the correlation matrix is symmetric because every variable pair has to have the same relationship (correlation) whether their correlation is in the upper right or lower left triangle. It doesn’t make sense to say the correlation between variables $X_1$ and $X_2$ is $\rho$, but the correlation between $X_2$ and $X_1$ is $\rho’\neq \rho$ if calculating a Pearson correlation (so correlation is symmetric).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mathematically, correlation between two variables, $X$ and $Y$, is commutative: $Corr(X,Y)=Corr(Y,X)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In OP’s case, the correlation between Q1 and Q2 is the same calculation and therefore the same result as the correlation between Q2 and Q1. Therefore the correlation matrix will be symmetric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are more mathematical reasons and proofs why a correlation matrix of real valued variables has to be symmetric and positive semi-definite, but I’ve excluded them from this answer.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastEditorUserId="41929" LastEditDate="2018-05-07T20:26:48.650" LastActivityDate="2018-05-07T20:26:48.650" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31341" PostTypeId="1" CreationDate="2018-05-07T18:55:00.683" Score="0" ViewCount="98" Body="&lt;p&gt;I have two encoder-decoder models. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;*First model:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/15sXe.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/15sXe.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*Second model: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/brwnN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/brwnN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I check the performance of the models I get approximately the same performance time (First model ~ 42 sec, Second model ~ 40 sec). I train my model on GPU and check performance on CPU. I test it only on one large image where the size is 12348x12348.&#xA;I was expecting the larger model that has more parameters to train (second model) to give me longer run time. Anyone can help me to understand why it is not the case here? Am I doing something wrong? &lt;/p&gt;&#xA;" OwnerUserId="51589" LastEditorUserId="89090" LastEditDate="2020-11-28T10:58:10.697" LastActivityDate="2024-01-13T09:07:50.987" Title="Encoder-Decoder performance time" Tags="|neural-network|deep-learning|lstm|rnn|autoencoder|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31343" PostTypeId="1" CreationDate="2018-05-07T19:53:53.103" Score="3" ViewCount="763" Body="&lt;p&gt;I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now moving on to MLP results,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exp 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Network Architecture:&lt;/em&gt; [67 40 5]&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Optimizer:&lt;/em&gt; Adam&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Learning Rate:&lt;/em&gt; exponential decay of base learning rate&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Validation Accuracy:&lt;/em&gt; ~45%&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Observation:&lt;/em&gt; Both training accuracy and validation accuracy stops improving.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exp 2:&lt;/strong&gt;&#xA;Repeated &lt;strong&gt;Exp 1&lt;/strong&gt; with batchnorm layer&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Validation Accuracy:&lt;/em&gt; ~50%&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Observation:&lt;/em&gt; Got 5% increase in accuracy.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Exp 3:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To overfit, increased the depth of MLP. A deeper version of &lt;strong&gt;Exp 1&lt;/strong&gt; network&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;em&gt;Network Architecture:&lt;/em&gt; [67 40 40 40 40 40 40 5]&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Optimizer:&lt;/em&gt; Adam&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Learning Rate:&lt;/em&gt; exponential decay of base learning rate&lt;/li&gt;&#xA;&lt;li&gt;&lt;em&gt;Validation Accuracy:&lt;/em&gt; ~55%&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Thoughts on what might be happening? &lt;/p&gt;&#xA;" OwnerUserId="51748" LastActivityDate="2021-01-27T19:02:06.337" Title="Unable to overfit using MLP" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31344" PostTypeId="1" CreationDate="2018-05-07T20:01:01.310" Score="6" ViewCount="1752" Body="&lt;p&gt;When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any theoretical considerations for selecting a threshold for min_value? Depending on the contexts the words are seen in (and the potential variety of them), it seems that it could take more than 5 words for the model to learn a good vector representation of the word. Are there any papers that specify when the learned vectors become more static? Alternatively, can you train the model on all words, but only utilize the vectors of words that appeared in the corpus more than a specified number of times? Is frequency a better cutoff than min_count?&lt;/p&gt;&#xA;" OwnerUserId="47257" LastActivityDate="2024-01-29T22:04:20.490" Title="What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?" Tags="|deep-learning|word2vec|text|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31345" PostTypeId="2" ParentId="31343" CreationDate="2018-05-07T20:24:53.557" Score="1" Body="&lt;p&gt;Some things that could be happening: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You do not have enough parameters in your hidden Layer to learn. Try something like [67, 512, 5] or expand deeper with something like [67, 1024, 256, 5]. The idea is that your hidden layer may be too small to capture interactions between the attributes given the amount of training data you have.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;A rule of thumb that I &lt;a href=&quot;http://hagan.okstate.edu/NNDesign.pdf#page=469&quot; rel=&quot;nofollow noreferrer&quot;&gt;have seen&lt;/a&gt; is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Nh=Ns/(α∗(Ni+No))&#xA;&#xA;Nh = number of hidden neurons.&#xA;Ni = number of input neurons.&#xA;No = number of output neurons.&#xA;Ns = number of samples in training data set.&#xA;α = an arbitrary scaling factor usually 2-10.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which in your case gets you to like 5000 neurons, but since you are oversampling your classes, I would also suggest &lt;a href=&quot;https://stats.stackexchange.com/questions/234016/opinions-about-oversampling-in-general-and-the-smote-algorithm-in-particular&quot;&gt;under-sampling&lt;/a&gt; instead, and that gets you maybe to around 1024, which is fairly standard. The [67, 512, 5] configuration is pure gut reaction. &lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;Learning rate is too high. Try starting with lower LR (like 10e-5). Sometimes this works with Adam.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Weight Initialization : double check you are using Xavier. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-07T20:24:53.557" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31346" PostTypeId="1" AcceptedAnswerId="31355" CreationDate="2018-05-07T21:22:03.843" Score="5" ViewCount="13331" Body="&lt;p&gt;Question relating to the caret package 'rpart' method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the method='rpart' automatically prune the tree?  If so, what rules does it follow?  If not, how does one go about directing caret to do this?&lt;/p&gt;&#xA;" OwnerUserId="41219" LastActivityDate="2022-01-22T16:00:06.480" Title="Caret and rpart - does caret automatically prune rpart trees" Tags="|machine-learning|r|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31348" PostTypeId="2" ParentId="31194" CreationDate="2018-05-07T22:26:54.340" Score="3" Body="&lt;p&gt;Given scikit-learn's API, you create a separate instance for each optimizer and compare the results to see which optimizer makes better predictions. It would looks something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.gaussian_process import GaussianProcessRegressor&#xA;from sklearn.gaussian_process.kernels import RBF&#xA;&#xA;gp1 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer1)&#xA;gp1.fit(X, y)&#xA;y_pred1 = gp1.predict(x)&#xA;&#xA;gp2 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer2)&#xA;gp2.fit(X, y)&#xA;y_pred2 = gp2.predict(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.dominodatalab.com/fitting-gaussian-process-models-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;This blog post&lt;/a&gt; gives a general overview of fitting Gaussian Processes, including scikit-learn's GaussianProcessRegressor. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-08T03:05:42.923" LastActivityDate="2018-05-08T03:05:42.923" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31349" PostTypeId="1" CreationDate="2018-05-07T23:11:06.923" Score="1" ViewCount="182" Body="&lt;p&gt;Why training deep learning models is more likely to suffer from plateaus than local minima? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone illustrate it.&lt;/p&gt;&#xA;" OwnerUserId="51760" LastActivityDate="2018-05-07T23:11:06.923" Title="plateaus and local minima in deep learning" Tags="|machine-learning|deep-learning|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-08T13:37:48.603" ContentLicense="CC BY-SA 4.0" />
  <row Id="31350" PostTypeId="1" AcceptedAnswerId="31353" CreationDate="2018-05-08T00:02:15.600" Score="0" ViewCount="78" Body="&lt;p&gt;I was watching this &lt;a href=&quot;https://youtu.be/040CfFRJ9Rs?t=3m25&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt;  and Sentdex mentioned he had to switch around 1024 by 3 model. What does he mean by 1024 by 3 model and what did he change around? &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Edited: Youtube link with the timestamp where he talks about this topic&lt;/p&gt;&#xA;" OwnerUserId="51562" LastEditorUserId="40853" LastEditDate="2018-05-08T02:12:05.567" LastActivityDate="2018-05-08T02:12:05.567" Title="What does 1024 by 3 model mean?" Tags="|deep-learning|nlp|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31351" PostTypeId="2" ParentId="31292" CreationDate="2018-05-08T00:18:30.763" Score="3" Body="&lt;p&gt;The black box answer is you can’t train models when your features have different ranges (1-5 vs 1-5000).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I really recommend writing a simple gradient descent solver (plenty of samples online) and train a simple linear model (y=mX+b) where the target solution is something like y=5x+500.  With a fixed learning rate for both m and b - the training goes super slow or doesn’t converge and goes backwards.  The solution is normalizing or scaling.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I said the best way to learn this is code it up and experiment.&lt;/p&gt;&#xA;" OwnerUserId="50951" LastActivityDate="2018-05-08T00:18:30.763" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31352" PostTypeId="1" AcceptedAnswerId="31356" CreationDate="2018-05-08T00:20:45.883" Score="11" ViewCount="9052" Body="&lt;p&gt;I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually I have to add rows and cols. Trying to select the best for my problem, I want to understand which are the best cases to use each of this types: lil_matrix, coo_matrix, csr_matrix, csc_matrix, dok_matrix. Can someone explain me? Its not necessary to show examples of all the types in the same answer. &lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-09-17T05:06:49.840" Title="Understanding scipy sparse matrix types" Tags="|python|scipy|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31353" PostTypeId="2" ParentId="31350" CreationDate="2018-05-08T01:23:42.420" Score="2" Body="&lt;p&gt;At 3:30 in video, he mentions this switching. What he is referring to by 'switch around' is that he is specifying the size of the LSTM being used. The 1024 refers to number of cells (or size) of the LSTM, and the 3 is the number of layers. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the tutorial page being referenced I the snippet he is referencing is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;python translate.py&#xA;  --data_dir [your_data_directory] --train_dir [checkpoints_directory]&#xA;  --size=256 --num_layers=2 --steps_per_checkpoint=50&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;For details on why LSTMs are useful for this task, check out the &lt;a href=&quot;https://arxiv.org/abs/1409.3215&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt;. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-08T01:23:42.420" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31355" PostTypeId="2" ParentId="31346" CreationDate="2018-05-08T01:48:16.410" Score="5" Body="&lt;p&gt;&lt;em&gt;To give a proper background for &lt;code&gt;rpart&lt;/code&gt; package and  &lt;code&gt;rpart&lt;/code&gt; method with &lt;code&gt;caret&lt;/code&gt; package:&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1.&lt;/strong&gt; If you use the &lt;code&gt;rpart&lt;/code&gt; package directly, it will construct the complete tree by default. &#xA;If you want to prune the tree, you need to provide the optional parameter &lt;code&gt;rpart.control&lt;/code&gt; which controls the fit of the tree. R documentation below, eg.:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    rpart(formula, data, method, control = prune.control)&#xA;&#xA;    prune.control = rpart.control(minsplit = 20, &#xA;                    minbucket =  round(minsplit/3), cp = 0.01, &#xA;                    maxcompete = 4, maxsurrogate = 5, usesurrogate = 2, &#xA;                    xval = 10, surrogatestyle = 0, maxdepth = 30 )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;these are the hyper parameters you can tune to obtain a pruned tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One followed way is to not provide the &lt;strong&gt;&lt;em&gt;cp&lt;/em&gt;&lt;/strong&gt; i.e complexity parameter and perform cross validation (xval), something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    rpart.control(minsplit = 20, minbucket = round(minsplit/3), xval = 10)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;complexity parameter &lt;em&gt;(cp)&lt;/em&gt; can be thought of as a measure of &lt;em&gt;complexity/ no of splits&lt;/em&gt; of your model and you want to increase complexity until your model generalizes to new observations. i.e. &lt;em&gt;regularization&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore evaluate the &lt;em&gt;cross validated error vs cp&lt;/em&gt; and choose the &lt;em&gt;cp&lt;/em&gt; that gives the good value (cp_good). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, add it as your control parameter i.e. &lt;em&gt;rpart.control(cp = cp_good)&lt;/em&gt; or use the prune function i.e. &lt;em&gt;prune(fit, cp = cp_good)&lt;/em&gt; to get the desired tree.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2.&lt;/strong&gt; &lt;code&gt;caret&lt;/code&gt; package on the other hand already implements the &lt;code&gt;rpart&lt;/code&gt; method with &lt;code&gt;cp&lt;/code&gt; as the tuning parameter. &lt;code&gt;caret&lt;/code&gt; by default will prune your tree based on a default run it makes on a default parameter grid (even if you don't supply any &lt;code&gt;tuneGrid&lt;/code&gt; and &lt;code&gt;trControl&lt;/code&gt; while training your model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    model &amp;lt;- train(data,&#xA;                   labels,&#xA;                   method = &quot;rpart&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2019-01-01T07:07:45.607" LastActivityDate="2019-01-01T07:07:45.607" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31356" PostTypeId="2" ParentId="31352" CreationDate="2018-05-08T02:46:17.140" Score="17" Body="&lt;p&gt;Ok, I was looking for an answer and now I have it clearer:&#xA;Scipy documentation does not elaborate too much on the explanation, but &lt;a href=&quot;https://en.wikipedia.org/wiki/Sparse_matrix&quot; rel=&quot;noreferrer&quot;&gt;wikipedia&lt;/a&gt; article is much more clear. For those who are looking for an answer, there are two major groups of sparse matrices:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a) Sparse types used to construct the matrices:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;DOK (Dictionary Of Keys): a dictionary that maps (row, column) to the value of the elements. It uses a hash table so it's efficient to set elements.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;LIL (LIst of Lists): LIL stores one list per row. The lil_matrix format is row-based, so if we want to use it then in other operations, conversion to CSR is efficient, whereas conversion to CSC is less so.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;COO (COOrdinate list): stores a list of (row, column, value) tuples.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;b) Sparse types that support efficient access, arithmetic operations, column or row slicing, and matrix-vector products:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;CSR (Compressed Sparse Row): similar to COO, but compresses the row indices.  Holds all the nonzero entries of M in left-to-right top-to-bottom (&quot;row-major&quot;) order (all elements in the first row, all elements in the second row, and so). More efficient in &lt;a href=&quot;https://stackoverflow.com/questions/41998147/what-is-row-slicing-vs-what-is-column-slicing&quot;&gt;&lt;em&gt;row&lt;/em&gt; indexing and &lt;em&gt;row&lt;/em&gt; slicing&lt;/a&gt;, because elements in the same row are stored contiguously in the memory.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;CSC (Compressed Sparse Column): similar to CSR except that values are read first by column. More efficient in a &lt;em&gt;column&lt;/em&gt; indexing and &lt;em&gt;column&lt;/em&gt; slicing.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Once the matrices are build using one of the a) types, to perform manipulations such as multiplication or inversion, we should convert the matrix to either CSC or CSR format.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="59027" LastEditDate="2018-09-13T13:29:05.403" LastActivityDate="2018-09-13T13:29:05.403" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31358" PostTypeId="2" ParentId="31129" CreationDate="2018-05-08T04:20:51.083" Score="7" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;when the loss function is calculated differently that means the backprop will behave differently (more emphasis to important samples).  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;However the training error is much lower than before  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I understand by error you mean the average loss calculated for each batch.&lt;br&gt;&#xA;the cross-entropy loss function is a proxy (differentiable) to train the model, while the zero-one loss is the one to fine-tune it.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is a similar question xgboost: give more importance to recent samples but I would like an applicable answer to Keras&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Penalizing neural networks for specific examples means give them a high probability to be included in the batches. there is no correspondence with XGboost here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;side note: maybe you could try &lt;strong&gt;Training with a Curriculum&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Gradually transforming&#xA;  the training task, from an easy one (maybe convex) where&#xA;  examples illustrate the simpler concepts, to the target one (with more&#xA;  difficult examples)&lt;br&gt;&#xA;  The basic idea is to start small, learn easier aspects of the&#xA;  task or easier sub-tasks, and then gradually increase the difficulty&#xA;  level. From the point of view of building representations, advocated&#xA;  here, the idea is to learn representations that capture low-level abstractions&lt;br&gt;&#xA;  first, and then exploit them and compose them to learn slightly&#xA;  higher-level abstractions necessary to explain more complex structure&#xA;  in the data. By choosing which examples to present and in which&#xA;  order to present them to the learning system, one can guide training&#xA;  and remarkably increase the speed at which learning can occur.&#xA;  &lt;a href=&quot;https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf&quot; rel=&quot;noreferrer&quot;&gt;learning deep architectures for ai by yoshua bengio&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-08T04:20:51.083" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31359" PostTypeId="1" AcceptedAnswerId="31372" CreationDate="2018-05-08T04:23:42.637" Score="0" ViewCount="385" Body="&lt;p&gt;Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression. &#xA;My doubts are given below:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is the penalty the same (by same proportion) for all the coefficients or is it based on variable importance?&#xA;If it is the latter I believe we can directly apply regularization rather than spending time in feature selection.&lt;/li&gt;&#xA;&lt;li&gt;Whether the Multi-collinearity is taken care by ridge and lasso regularization? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="50727" LastEditDate="2018-05-25T07:17:31.320" LastActivityDate="2018-05-25T07:17:31.320" Title="Ridge and Lasso Regularization" Tags="|machine-learning|regression|regularization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31360" PostTypeId="1" CreationDate="2018-05-08T04:50:44.310" Score="-1" ViewCount="170" Body="&lt;p&gt;I have a dataset and I have tried plotting it in a box plot after loading the csv in usings pandas. The plot itself is given below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/C9Njl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/C9Njl.png&quot; alt=&quot;Box Plot with 6 features and 1 output y&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to understand why the &lt;code&gt;x1&lt;/code&gt; feature doesn't have a lower whisker and what is represented by solid blue bar after the top whisker of the &lt;code&gt;x1&lt;/code&gt; feature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help with this so that I may decide whether I need to look into outliers removal for feature &lt;code&gt;x1&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="51648" LastEditorUserId="9123" LastEditDate="2018-05-08T05:02:11.360" LastActivityDate="2018-05-08T06:06:38.940" Title="Understanding a Box plot" Tags="|visualization|outlier|" AnswerCount="1" CommentCount="1" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31361" PostTypeId="1" CreationDate="2018-05-08T05:15:28.973" Score="0" ViewCount="7214" Body="&lt;p&gt;I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose that tweet is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I was wid Ali.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to convert:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wid -&amp;gt; with&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Similarly&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;wud -&amp;gt; would&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;i have 6000 tweets in which there are lots of short words. How i can replace them ? is there any library available in python for this task? or any dictionary of shorts words available online?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;i read answer of &lt;a href=&quot;https://stackoverflow.com/questions/43018030/replace-appostrophe-short-words-in-python&quot;&gt;Replace appostrophe/short words in python&lt;/a&gt; Question but it provides dictionary of appostrophe only.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently i am using NLTK but this task is not possible with NLTK.&lt;/p&gt;&#xA;" OwnerUserId="51744" LastActivityDate="2018-06-07T06:47:20.413" Title="How to replace short words into full words from tweets using python" Tags="|machine-learning|python|nlp|nltk|twitter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31362" PostTypeId="1" AcceptedAnswerId="31373" CreationDate="2018-05-08T05:19:05.493" Score="0" ViewCount="579" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uofly.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uofly.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is to find thetha which will give minimum cost function. Why is the x0 column required? why cant we assign size as x0? why do we need the feature count to be n+1?&lt;/p&gt;&#xA;" OwnerUserId="30653" LastActivityDate="2018-05-08T07:47:29.850" Title="Linear Regression - finding thetha using Normal equation" Tags="|machine-learning|linear-regression|normal-equation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31363" PostTypeId="1" AcceptedAnswerId="31375" CreationDate="2018-05-08T05:40:38.940" Score="1" ViewCount="885" Body="&lt;p&gt;While training the neural network (or any other supervised learning algorithms), we supply input variables and corresponding outputs. The input variables can be continuous or discrete (binary in many cases). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What happens if after training with a binary input data, we supply a continuous value for the same input at the time of evaluation? Does the algorithm internally treat all variables as continuous variables? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, suppose one of the inputs is Young/Old encoded in the form of 0/1 in the training dataset. What happens if we supply a value of, say, 0.2 at the prediction stage? Does it/&lt;strong&gt;should it&lt;/strong&gt; make any sense to the network? &lt;/p&gt;&#xA;" OwnerUserId="50828" LastEditorUserId="50727" LastEditDate="2018-05-25T07:17:57.947" LastActivityDate="2018-05-25T07:17:57.947" Title="What is the input space of a neural network (or other supervised learning algorithms)?" Tags="|machine-learning|neural-network|deep-learning|categorical-data|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31364" PostTypeId="2" ParentId="31362" CreationDate="2018-05-08T05:43:26.330" Score="1" Body="&lt;p&gt;Adding extra column x0 is a design choice.&lt;br&gt;&#xA;mathematically: if you multiply w by x you get a single value (simply dot product), thus you need to encode another learnable parameter independent of x like a prior (belief), you end up doing w.x + 1*(bias)&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-08T05:43:26.330" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31365" PostTypeId="2" ParentId="31361" CreationDate="2018-05-08T05:53:18.593" Score="2" Body="&lt;p&gt;I have worked on lexicon level token normalization on twitter data, which seems to be very similar to what you are trying to do. In our task, we were not only normalizing &lt;em&gt;short word&lt;/em&gt;, but also &quot;welllll&quot;-&gt;&quot;well&quot;, &quot;lysm&quot;-&gt;&quot;love you so much&quot;, &quot;2&quot;-&gt; to, etc. Refer to the task description &lt;a href=&quot;https://noisy-text.github.io/2015/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. At that time we did not find a good library directly for this job. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In our approach, we resort to buidling a lookup table to replace &lt;em&gt;suspicious&lt;/em&gt; words. The lookup table is compiled from the following resources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.webopedia.com/quick_ref/textmessageabbreviations.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;Huge List of Texting and Online Chat Abbreviations&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.webopedia.com/quick_ref/Twitter_Dictionary_Guide.asp&quot; rel=&quot;nofollow noreferrer&quot;&gt;Twitter Dictionary: A Guide to Understanding Twitter Lingo&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dl.acm.org/citation.cfm?id=2002520&quot; rel=&quot;nofollow noreferrer&quot;&gt;Lexical normalisation of short text messages: makn sens a #twitter&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.5702&quot; rel=&quot;nofollow noreferrer&quot;&gt;Insertion, Deletion, or Substitution? Normalizing Text Messages without Pre-categorization nor Supervision&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter two links are two relevant papers. They both include a dataset that we used in our lookup table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;suspicious&lt;/em&gt; words that need to be replaced can be simply identified by whether they are OOV, or using a more complicated machine learning approach to determine based on context.&lt;/p&gt;&#xA;" OwnerUserId="14006" LastActivityDate="2018-05-08T05:53:18.593" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31366" PostTypeId="2" ParentId="31360" CreationDate="2018-05-08T06:06:38.940" Score="2" Body="&lt;p&gt;The solid blue bar above the top whisker are your outliers. If you dont want them, you could pass showfliers=False as an argument.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And for the lower whisker, you may want to adjust the y-scaling of your boxplt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps.&lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-08T06:06:38.940" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31367" PostTypeId="2" ParentId="31362" CreationDate="2018-05-08T06:11:14.150" Score="1" Body="&lt;p&gt;The column x0 is a constant that is used to fit the intercept. Excluding that will give you the coefficients but without an intercept.&lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-08T06:11:14.150" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31368" PostTypeId="1" CreationDate="2018-05-08T06:53:38.240" Score="0" ViewCount="111" Body="&lt;p&gt;In a supervised learning problem (like a neural network) with binary output, we supply input data for which the output values are defined 0 and 1 (indicating probabilities of occurrence of an event).  But the input data not necessarily cover the entire input space. How does the algorithm treat the rest of the input space? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, for an annular dataset in the figure, there are two sets of data in the 2-D input space annotated as 0s and 1s. But there are a lot of input space which is not defined (neither 0 nor 1). What output can be expected at the prediction stage for an input that does not lie around the input data clouds?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/bCpFy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bCpFy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50828" LastActivityDate="2018-05-08T17:24:47.457" Title="How to treat undefined input space in supervised learning algorithms?" Tags="|dataset|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31369" PostTypeId="2" ParentId="31243" CreationDate="2018-05-08T07:04:20.983" Score="3" Body="&lt;p&gt;In &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub&lt;/a&gt;, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In &lt;a href=&quot;https://ieeexplore.ieee.org/document/6940273/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ieeexplore.ieee.org/document/6940273/&lt;/a&gt; and &lt;a href=&quot;https://ieeexplore.ieee.org/abstract/document/5428802/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ieeexplore.ieee.org/abstract/document/5428802/&lt;/a&gt; you can find these alternatives.&#xA;Good luck.&#xA;Rafael.&lt;/p&gt;&#xA;" OwnerUserId="16560" LastActivityDate="2018-05-08T07:04:20.983" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31371" PostTypeId="1" AcceptedAnswerId="31391" CreationDate="2018-05-08T07:39:19.873" Score="1" ViewCount="1542" Body="&lt;p&gt;The Gaussian process regression can be computed in scikit learn using an object of class GaussianProcessRegressor as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; gp= GaussianProcessRegressor(alpha=1e-10, copy_X_train=True,&#xA;    kernel=1**2 + Matern(length_scale=2, nu=1.5) + WhiteKernel(noise_level=1),&#xA;    n_restarts_optimizer=0, normalize_y=False,&#xA;    optimizer='fmin_l_bfgs_b', random_state=None)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, we see that the L-BFGS-B algorithm has been used to optimize the hyperparameters. Is there any alternative (for example trust-region-reflective algorithm) to this algorithm available in sklearn? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;: &lt;a href=&quot;https://scipy.github.io/devdocs/optimize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;It&lt;/a&gt; provides some Constrained multivariate methods for optimization. Is there such functions available for other methods like trust-region-reflective algorithm and swarm intelligence (eg. Firefly algorithm)?&lt;/p&gt;&#xA;" OwnerUserId="50881" LastEditorUserId="50881" LastEditDate="2018-05-08T08:08:12.307" LastActivityDate="2018-05-08T14:08:12.870" Title="Is there any alternative to L-BFGS-B algorithm for hyperparameter optimization in Scikit learn?" Tags="|python|scikit-learn|regression|hyperparameter|hyperparameter-tuning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31372" PostTypeId="2" ParentId="31359" CreationDate="2018-05-08T07:41:39.767" Score="1" Body="&lt;p&gt;The penalty of both Lasso and Ridge is proportional to the magnitude of the weight. That is, the penalization added to the cost function is $\lambda ||\omega||_2$ or $\lambda ||\omega||_1$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Wether it is more convenient to apply regularization or feature selection, Lasso already does some feature selection for you, as the estimated weights for Lasso are sparse (there will be many coefficients equal to 0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;About multi-colinearity, Ridge tends to eliminate variables that are colinear, while Lasso doesn't.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All that I have said is taken from An Introduction to Statistical Learning book, from James, Witten, Hastie and Tibshirani. &lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-08T07:41:39.767" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31373" PostTypeId="2" ParentId="31362" CreationDate="2018-05-08T07:47:29.850" Score="2" Body="&lt;p&gt;Consider the 1-D equivalent of the table you have provided. In this case, you have one input x and one output y (see figure). If you try to fit the data points with y=mx equation, you can not fit the data successfully. You need an equation like y=mx+c to have a good fit. y=mx will be a good fit only if the line goes through the origin.&lt;/p&gt;&#xA;&lt;p&gt;Now you can look at&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;y=mx+c&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;as&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;y = m * x + c * 1&lt;/p&gt;&#xA;&lt;p&gt;y = m * X1 + c * X0&lt;/p&gt;&#xA;&lt;p&gt;y = W1 * X1 + W0 * X0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I hope the figures below will clarify why we need the intercept term X0 and its weight W0=c.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/0a0y5.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0a0y5.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/uEBO8.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/uEBO8.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50828" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-08T07:47:29.850" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31374" PostTypeId="1" AcceptedAnswerId="31397" CreationDate="2018-05-08T07:50:00.873" Score="2" ViewCount="1621" Body="&lt;p&gt;I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I guess I'm right in saying the feature data then needs to be scaled (using min-max?), after calculating z-score?&lt;/p&gt;&#xA;" OwnerUserId="51125" LastActivityDate="2018-05-08T16:12:40.123" Title="Using z-score for neural network normalization" Tags="|normalization|feature-scaling|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31375" PostTypeId="2" ParentId="31363" CreationDate="2018-05-08T07:58:29.790" Score="1" Body="&lt;p&gt;I guess it depends on the algorithm, but linear models, as well as neural networks, treat all variables as continuous. The algorithm will not explode or anything if you supply 0.2 at prediction stage. &lt;em&gt;However&lt;/em&gt;, your algorithm is trained on data. The algorithm can at best do what it has learnt from the training data. For this reason, &lt;strong&gt;do not&lt;/strong&gt; expect anything meaningful when you feed an example with a value that has not been seen in the whole training set, or that it does not follow the training set distribution.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-08T08:21:29.927" LastActivityDate="2018-05-08T08:21:29.927" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31376" PostTypeId="2" ParentId="31338" CreationDate="2018-05-08T08:04:52.410" Score="2" Body="&lt;p&gt;The correlation matrix is a measure of linearity. It does not express how two variables are dependent on each other. If the relationship is approximately linear, the absolute value of correlation will be closer to 1. If there is no linear relationship, the value is zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider two sets of variables (x1,y1) and (x2,y2). &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;y1 = 2 * x1&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;y2 = 1000 * x2&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In both these cases, the correlation is 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The exact relationship between x1 and y1 cannot be understood by looking only at the correlation matrix.&lt;/p&gt;&#xA;" OwnerUserId="50828" LastActivityDate="2018-05-08T08:04:52.410" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31377" PostTypeId="1" AcceptedAnswerId="31379" CreationDate="2018-05-08T08:34:33.680" Score="2" ViewCount="370" Body="&lt;p&gt;In a machine learning procedure, suppose we've chosen k=10 for the &quot;K-Fold Cross Validation&quot;.&#xA;After we do the k steps of &quot;K-Fold Cross Validation&quot;, how do we choose the final model for the classifier ? (The one will we use in order to predict new data)&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-05-08T08:41:09.890" Title="Question about K-Fold Cross Validation" Tags="|machine-learning|classification|bigdata|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31378" PostTypeId="2" ParentId="31170" CreationDate="2018-05-08T08:38:53.840" Score="1" Body="&lt;p&gt;I found my error. My early calculations only involved one hidden layer, however, I did not include the derivatives of previous weights when calculating the hidden_error. This means that in the scenario of a neural network of:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input -&amp;gt; H1 -&amp;gt; H2 -&amp;gt; Output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;to calculate the backpropagation of the last step (Input &amp;lt;- H1), I did not calculate derivatives of weights_hidden_output (H2 -&gt; Output) and weights_hidden_to_hidden (H1 -&gt; H2).&lt;/p&gt;&#xA;" OwnerUserId="51778" LastActivityDate="2018-05-08T08:38:53.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31379" PostTypeId="2" ParentId="31377" CreationDate="2018-05-08T08:41:09.890" Score="3" Body="&lt;p&gt;K-fold cross validation only helps you giving an estimation of the error you are going to make with a given model. K-fold can be used in order to tune hyperparameters: we have a model that depends on some hyperparameters, we compute the K-fold validation error using different values of the hyperparameters, and we take the model with the lowest cross-validation error. Finally, we train the model with the chosen hyperparameters with all the training data and we build a classifier based on that model.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-08T08:41:09.890" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31380" PostTypeId="2" ParentId="22632" CreationDate="2018-05-08T09:04:55.100" Score="0" Body="&lt;p&gt;One thing you can do is to replace the softmax layer by some other type of classifier, I'd recommend SVM. Thus, you feed the SVM with your handcrafted features concatenated with the output of the dense layer.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-08T09:04:55.100" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31381" PostTypeId="1" CreationDate="2018-05-08T09:10:06.467" Score="1" ViewCount="291" Body="&lt;p&gt;I'm training a CNN with images which have lots of horizontal black lines (due to the nature of the sensor). I'm thinking in removing this artifacts by some kind of preprocessing (interpolation, median filters...). The thing is: does it make sense, since the CNN tries to apply optimal filtering? (if some 2D filtering is intended to be done just before the CNN, is just adding a deterministic layer at the beginning of the net...)&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2024-03-17T16:02:05.307" Title="Does it make sense to interpolate image just before a CNN?" Tags="|cnn|interpolation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31382" PostTypeId="1" CreationDate="2018-05-08T09:23:04.467" Score="9" ViewCount="2225" Body="&lt;p&gt;I'm currently facing a Machine Learning problem and I've reached a point where I need some help to proceed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have various time series of positional (&lt;code&gt;x&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;z&lt;/code&gt;) data tracked by sensors. I've developed some more features. For example, I rasterized the whole 3D space and calculated a &lt;code&gt;cell_x&lt;/code&gt;, &lt;code&gt;cell_y&lt;/code&gt; and &lt;code&gt;cell_z&lt;/code&gt; for every time step. The time series itself have variable lengths.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to build a model which classifies every time step with the labels &lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt; (binary classification based on past and future values). Therefore I have a lot of training time series where the labels are already set. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One thing which could be very problematic is that there are very few &lt;code&gt;1&lt;/code&gt;'s labels in the data (for example only 3 of 800 samples are labeled with &lt;code&gt;1&lt;/code&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would be great if someone can help me in the right direction because there are too many possible problems:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Wrong hyperparameters&lt;/li&gt;&#xA;&lt;li&gt;Incorrect model&lt;/li&gt;&#xA;&lt;li&gt;Too few &lt;code&gt;1&lt;/code&gt;'s labels, but I think that's not a big problem because I only need the model to suggests the right time steps. So I would only use the peaks of the output.&lt;/li&gt;&#xA;&lt;li&gt;Bad or too less training data&lt;/li&gt;&#xA;&lt;li&gt;Bad features&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I appreciate any help and tips.&lt;/p&gt;&#xA;" OwnerUserId="32543" LastEditorUserId="32543" LastEditDate="2018-05-17T10:40:19.743" LastActivityDate="2020-04-29T02:27:29.170" Title="Binary classification of every time series step based on past and future values" Tags="|classification|keras|time-series|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31384" PostTypeId="1" AcceptedAnswerId="31428" CreationDate="2018-05-08T11:23:42.613" Score="2" ViewCount="248" Body="&lt;p&gt;I've written a simple neural network that can predict XOR gate function. I think I've used the math correctly, but the loss doesn't go down and remains near 0.6. Can anyone help me find the reason why?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib as plt&#xA;&#xA;train_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T&#xA;train_Y = np.array([[0,1,1,0]])&#xA;test_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T&#xA;test_Y = np.array([[0,1,1,0]])&#xA;&#xA;learning_rate = 0.1&#xA;S = 5&#xA;&#xA;def sigmoid(z):&#xA;    return 1/(1+np.exp(-z))&#xA;&#xA;def sigmoid_derivative(z):&#xA;    return sigmoid(z)*(1-sigmoid(z))&#xA;&#xA;S0, S1, S2 = 2, 5, 1&#xA;m = 4&#xA;&#xA;w1 = np.random.randn(S1, S0) * 0.01&#xA;b1 = np.zeros((S1, 1))&#xA;w2 = np.random.randn(S2, S1) * 0.01&#xA;b2 = np.zeros((S2, 1))&#xA;&#xA;for i in range(1000000):&#xA;    Z1 = np.dot(w1, train_X) + b1&#xA;    A1 = sigmoid(Z1)&#xA;    Z2 = np.dot(w2, A1) + b2&#xA;    A2 = sigmoid(Z2)&#xA;&#xA;    J = np.sum(-train_Y * np.log(A2) + (train_Y-1) * np.log(1-A2)) / m&#xA;&#xA;    dZ2 = A2 - train_Y&#xA;    dW2 = np.dot(dZ2, A1.T) / m&#xA;    dB2 = np.sum(dZ2, axis = 1, keepdims = True) / m&#xA;    dZ1 = np.dot(w2.T, dZ2) * sigmoid_derivative(Z1)&#xA;    dW1 = np.dot(dZ1, train_X.T) / m&#xA;    dB1 = np.sum(dZ1, axis = 1, keepdims = True) / m&#xA;&#xA;    w1 = w1 - dW1 * 0.03&#xA;    w2 = w2 - dW2 * 0.03&#xA;    b1 = b1 - dB1 * 0.03&#xA;    b2 = b2 - dB2 * 0.03&#xA;&#xA;    print(J)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerDisplayName="user51784" LastEditorDisplayName="user51784" LastEditDate="2018-05-08T11:32:14.037" LastActivityDate="2018-05-09T17:38:21.587" Title="neural network xor gate classification" Tags="|neural-network|backpropagation|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31385" PostTypeId="2" ParentId="31382" CreationDate="2018-05-08T11:32:09.970" Score="10" Body="&lt;p&gt;You are facing a very common problem: handling imbalanced data. For neural networks, typical procedures are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Having the proper metrics: global accuracy should not be used.&lt;/li&gt;&#xA;&lt;li&gt;Oversampling the minority class: randomly generate replicas of the minority class until the imbalance disappears.You can also perform data augmentation on the minority class. Synthetic data can be generated from the feature space, using SMOTE algorithm, but I don't know how it applies to neural networks.&lt;/li&gt;&#xA;&lt;li&gt;Under-sampling the majority class: randomly remove instances of the majority class. It can deteriorate the performance on the majority class&lt;/li&gt;&#xA;&lt;li&gt;Include class-weights in the loss function: the idea is to penalize the misclassification of the minority class. The weights are usually inversely proportional to the occurrence frequency of each class.&lt;/li&gt;&#xA;&lt;li&gt;Using different learning rates per class: you can use a bigger learning rate for the majority class, thus the net stops learning  earlier from the majority class than from the minority class&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'd recommend a combination of 1 (always), 2 and 4. For a higher insight in this topic, which is of very importance, I recommend reading:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://dl.acm.org/citation.cfm?id=1592322&quot; rel=&quot;noreferrer&quot;&gt;https://dl.acm.org/citation.cfm?id=1592322&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1710.05381.pdf&quot; rel=&quot;noreferrer&quot;&gt;https://arxiv.org/pdf/1710.05381.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-08T11:32:09.970" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31386" PostTypeId="1" CreationDate="2018-05-08T12:06:23.593" Score="4" ViewCount="463" Body="&lt;p&gt;I just want to know if is it possible to use &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf.estimator.DNNClassifier&lt;/a&gt; with multiple different activation functions. I mean, could I use a DNNClassifier estimator which use different activation functions for different layers?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I have a three layers model, could I use for the first layer a &lt;em&gt;sigmoid&lt;/em&gt; function, for the second one a &lt;em&gt;ReLu&lt;/em&gt; function and finally for the last one a &lt;em&gt;tanh&lt;/em&gt; function?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to know if it isn't possible to do it with &lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier&quot; rel=&quot;nofollow noreferrer&quot;&gt;DNNClassifier&lt;/a&gt; how can I do it by a easy way.&lt;/p&gt;&#xA;" OwnerUserId="51787" LastEditorUserId="51787" LastEditDate="2018-05-23T10:37:28.827" LastActivityDate="2024-02-04T19:03:40.457" Title="Multiple activation functions with TensorFlow estimator DNNClassifier" Tags="|machine-learning|python|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31387" PostTypeId="1" CreationDate="2018-05-08T12:09:28.343" Score="2" ViewCount="193" Body="&lt;p&gt;My final project is number plate recognition.i need a data set of A-Z characters and 0-9 letters. i donot find it on any website give me a data set or send me a link. i have to make a neural network in which i put that data set to train and test my data.&lt;/p&gt;&#xA;" OwnerUserId="51788" LastActivityDate="2019-02-14T15:26:06.140" Title="i need a characters /letters dataset for matlab" Tags="|neural-network|matlab|image-recognition|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31388" PostTypeId="1" AcceptedAnswerId="31394" CreationDate="2018-05-08T12:13:16.647" Score="23" ViewCount="6619" Body="&lt;p&gt;I'm training a convolutional neural network to classify images on fog conditions (3 classes). However, for each of about 150.000 images I also have four meteorological variables available that might help in predicting the classes of the images. I was wondering how I could add the meteorological variables (e.g. temperature, wind speed) to the existing CNN structure so that it can help in the classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I can already think of is creating another (small) feedforward neural net alongside the CNN and then concatenating the outputs of the CNN layers and the hidden layers of the non-image neural net to each other at the dense layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The second way I could think of is just contacting these features to the dense layer. However, in this case, the non-image variables will (I think) only be able to make linear predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any other (better) ways that the non-image features could be included in the model? And what would be the advisable method considering the amount of data I have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another question I have is whether or not I should unfreeze the convolutional layers while training with these non-image features? These layers of a Resnet-18 (which were initialized as pre-trained on ImageNet) have already been fine-tuned using the images. My guess is that I should keep them frozen and only unfreeze the dense layer since it is only here that the non-image features come into 'contact' with the image features (not earlier in the CNN). If I'm wrong on this, please say so!&lt;/p&gt;&#xA;" OwnerUserId="51141" LastEditorUserId="28175" LastEditDate="2019-07-04T13:48:51.123" LastActivityDate="2019-09-28T05:37:37.977" Title="How to add non-image features along side images as the input of CNNs" Tags="|machine-learning|neural-network|deep-learning|tensorflow|cnn|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31389" PostTypeId="2" ParentId="31387" CreationDate="2018-05-08T13:03:05.180" Score="1" Body="&lt;p&gt;You can find car license plat &lt;a href=&quot;https://github.com/Sardhendu/License-Plate-Detection&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt;  or you can collect them from this &lt;a href=&quot;http://www.worldlicenseplates.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;web site&lt;/a&gt; but it will take you  a while.&lt;/p&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-05-08T13:03:05.180" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31390" PostTypeId="2" ParentId="812" CreationDate="2018-05-08T13:36:02.133" Score="2" Body="&lt;p&gt;I would recommend to use Graphexp. Gephi is highly dependent on the RAM of your computer which is obviously limited. Graphexp on the other hand  display's only a limited number of Nodes, through which you can navigate to other nodes. I have visualized a graph with 700 Million vertices using Graphexp as a UI and Janusgraph with HBase as a back-end storage. &lt;a href=&quot;https://github.com/bricaud/graphexp&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/bricaud/graphexp&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43590" LastActivityDate="2018-05-08T13:36:02.133" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31391" PostTypeId="2" ParentId="31371" CreationDate="2018-05-08T13:36:56.160" Score="2" Body="&lt;p&gt;Scikit-learn package has a limited selection of optimizers. The &lt;a href=&quot;https://www.scipy.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;scipy&lt;/a&gt; package has many more &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.optimize.minimize.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;optimitizers&lt;/a&gt;, including &lt;a href=&quot;https://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.optimize.least_squares.html#scipy.optimize.least_squares&quot; rel=&quot;nofollow noreferrer&quot;&gt;trust-region-reflective algorithm&lt;/a&gt;. You would have to use another &lt;a href=&quot;https://github.com/7ossam81/EvoloPy&quot; rel=&quot;nofollow noreferrer&quot;&gt;third party package for Firefly algorithm&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-08T14:08:12.870" LastActivityDate="2018-05-08T14:08:12.870" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31392" PostTypeId="1" CreationDate="2018-05-08T13:52:48.107" Score="4" ViewCount="4498" Body="&lt;p&gt;Hello everyone I'm having a weird problem.&lt;/p&gt;&#xA;&lt;p&gt;I having problem with one of two models I've been using. Models take an image data as input and outputs joystick and keyboard information.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/keras-team/keras/blob/9c689775a22f9e8647207eeae012978048d44bed/examples/mnist_cnn.py#L47&quot; rel=&quot;nofollow noreferrer&quot;&gt;A simple CNN&lt;/a&gt;, implemented using Keras to use with MNIST data, in which I changed the input size;&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a href=&quot;https://github.com/Sentdex/pygta5/blob/efdd66841ff909a027a42ebb48facbd63670e281/models.py#L219&quot; rel=&quot;nofollow noreferrer&quot;&gt;inception_v3&lt;/a&gt; (which is way more complex than the first one).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;While training the first model, it throws the following error:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; 2018-05-07 23:03:22.080446: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:101] Allocation of 1037238272 exceeds 10% of system memory.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have tried changing the batch_size parameter to a smaller batch size without any results.&lt;/p&gt;&#xA;" OwnerUserId="51798" LastEditorUserId="29169" LastEditDate="2021-02-13T04:21:38.817" LastActivityDate="2021-02-13T04:21:38.817" Title="Memory problems with smaller CNN" Tags="|keras|numpy|convolutional-neural-network|tflearn|" AnswerCount="0" CommentCount="7" ClosedDate="2020-08-05T02:37:02.573" ContentLicense="CC BY-SA 4.0" />
  <row Id="31393" PostTypeId="1" CreationDate="2018-05-08T14:54:44.450" Score="2" ViewCount="109" Body="&lt;p&gt;I have a neural net (BEGAN Generative Adversarial Network) where I need to apply this formula:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/MdNHH.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/MdNHH.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And this one:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4YSCp.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4YSCp.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I made an operator that does each operation and outputs a value for each losses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I need to compute the gradients. I guess my gradient takes 4 inputs (Ld, Lg, kT+1, and MGlobal) and outputs 3 ((grad kT, grad Lx, grad L(G(zd))). Can somebody explain to me what would they be? Those are constants dependent on each others, I am not sure what to put.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks for who could explain this to me.&lt;/p&gt;&#xA;" OwnerUserId="51803" LastEditorUserId="1330" LastEditDate="2021-08-29T19:57:58.927" LastActivityDate="2021-08-29T19:57:58.927" Title="Gradient of a sum" Tags="|deep-learning|mathematics|generative-models|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31394" PostTypeId="2" ParentId="31388" CreationDate="2018-05-08T15:17:04.010" Score="15" Body="&lt;p&gt;My solution is like your first recommendation, but with slight changes.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Construct your convolutional layers and stack them till the flatten-layer. This network should be fed with the image data.&lt;/li&gt;&#xA;&lt;li&gt;Flat your activation maps&lt;/li&gt;&#xA;&lt;li&gt;Construct a fully connected network with the desired number of neurons and layers. &lt;/li&gt;&#xA;&lt;li&gt;Concatenate the outputs of the flattened layer of the convolutional net and the fully connected net. &lt;/li&gt;&#xA;&lt;li&gt;add some dense layers and connect them to the last layer which represents your classes.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You can use customary cost functions for this architecture.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-08T15:17:04.010" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31395" PostTypeId="1" CreationDate="2018-05-08T15:56:29.513" Score="1" ViewCount="61" Body="&lt;p&gt;The Conditional Variational Autoencoders (CVAE) (and other classification networks) I have come across use a &lt;strong&gt;one hot vector encoding&lt;/strong&gt; for labeling categorial data sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, I do not have a categorial data set. I have 2 vectors &lt;code&gt;states&lt;/code&gt; and &lt;code&gt;costs&lt;/code&gt; which both contain floating point values. &lt;code&gt;states&lt;/code&gt; has dim 4 and &lt;code&gt;costs&lt;/code&gt; has dim 3. I want to train my CVAE in such a way that at test time I feed the decoder a &lt;code&gt;states&lt;/code&gt; vector and it should output an approximate &lt;code&gt;costs&lt;/code&gt; vector. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words my decoder should return: &lt;code&gt;p(costs|states,z)&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How should I train and organise my CVAE in this case? Should I simply concat both vectors?&lt;/p&gt;&#xA;" OwnerUserId="47328" LastActivityDate="2018-05-08T15:56:29.513" Title="Using vector of floating points as labels for input data in a CVAE" Tags="|python|neural-network|classification|keras|autoencoder|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31396" PostTypeId="1" CreationDate="2018-05-08T16:04:02.183" Score="1" ViewCount="137" Body="&lt;p&gt;I'm looking to build a predictive model for hockey players individual statistics. My goal is to predict how many points a player can be expected to have at the end of the season.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To do so, I thought I could base myself on the progression of the player's statistics over the year. I currently have 10 years worth of data and, for example, a player could have played in 5 of those 10 years. In this case, I'm able to see if this particular player is on an increasing or decreasing curve regarding his statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to resume, for some thousands of players the data goes like this:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;h3&gt;Player #1&lt;/h3&gt;&#xA;  &#xA;  &lt;p&gt;Season 2005-2006, Points : X, Game played : y, ...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Season 2006-2007, Points : X, Game played: Y, ...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Season 2007-2008, Points : X, Game played: Y, ...&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;...&lt;/p&gt;&#xA;  &#xA;  &lt;h3&gt;Player #2&lt;/h3&gt;&#xA;  &#xA;  &lt;p&gt;...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My main problem is that I don't know what &lt;strong&gt;kind of data this is&lt;/strong&gt; so I can't find more information on how to build my model. In my title I've written &quot;multiple small multivariate time series&quot; for a lack of better name. I've obviously looked into multivariate time series to fit them in an RNN architecture, but it doesn't seem to fit because of the multiple time series that aren't really related.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've found &lt;a href=&quot;https://stats.stackexchange.com/questions/305197/multiple-short-multivariate-time-series-forecasting&quot;&gt;this question&lt;/a&gt; which kind of relates but there were no answer and the related comments didn't help me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All in all, I'm trying to find some guidance as to what kind of model I could use to work with my data. I'd also like to know if there's a name for this kind of data, so I can inform myself better.&lt;/p&gt;&#xA;" OwnerUserId="44446" LastEditorUserId="1330" LastEditDate="2021-09-21T18:00:07.480" LastActivityDate="2024-03-20T15:55:33.437" Title="Build a predictive model for number of hockey goals in a season for an individual player" Tags="|machine-learning|time-series|machine-learning-model|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31397" PostTypeId="2" ParentId="31374" CreationDate="2018-05-08T16:12:40.123" Score="3" Body="&lt;p&gt;One of the reasons z-scores are useful in neural nets is that they can allow gradient descent to converge faster. I don't think taking these z-scores and scaling them further to be between -1 and 1 will add any extra benefit, as you already have your features expressed on a common scale.&lt;/p&gt;&#xA;" OwnerUserId="30034" LastActivityDate="2018-05-08T16:12:40.123" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31398" PostTypeId="2" ParentId="14088" CreationDate="2018-05-08T17:00:53.547" Score="1" Body="&lt;p&gt;There is a difference between accuracy and validation accuracy. Add in a term &quot;validation_split = .2&quot; into the model.fit statement and see if the validation accuracy is still 99%. If you are dealing with truly uncorrelated data and are getting 99% accuracy then you are over fitting the dataset and need more Ys in your sample as compared to the number of X variables.&lt;/p&gt;&#xA;" OwnerUserId="51808" LastActivityDate="2018-05-08T17:00:53.547" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31399" PostTypeId="1" AcceptedAnswerId="31401" CreationDate="2018-05-08T17:12:35.940" Score="3" ViewCount="62" Body="&lt;p&gt;I have a series of scripts. Some are in R, some in Python, and others in SAS. I have built them in such a way that one code outputs a .csv file that the next code obtains and then that code outputs a .csv file, and so on... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to create a script that will automatically run each script in order so that the final output can be generated automatically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method would be best for this and can anyone point me to any examples of the procedure?&lt;/p&gt;&#xA;" OwnerUserId="46886" LastActivityDate="2018-05-08T17:54:41.670" Title="How to build a data analysis pipeline procedure" Tags="|management|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31400" PostTypeId="2" ParentId="31368" CreationDate="2018-05-08T17:24:47.457" Score="2" Body="&lt;p&gt;It depends on the machine learning technique. The following link gives a really good example comparing many classifiers and their segregation area. The same happens with clustering methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Classifier comparison&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51809" LastActivityDate="2018-05-08T17:24:47.457" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31401" PostTypeId="2" ParentId="31399" CreationDate="2018-05-08T17:54:41.670" Score="3" Body="&lt;p&gt;Nomially, I would simply just try to write a bash script (or powershell in windows) and just string the commands together.  However, this approach is rather fragile as in things get overwritten, and if it's an end to end process that has long batches.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tend to use a workflow packages like luigi or airflow when stringing dependent task together.  The idea for Luigi is that you can break each action into a task.  Each task has three needed functions.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Requirement - what needs to exist before this task runs?&lt;/li&gt;&#xA;&lt;li&gt;Output - where is the output going?&lt;/li&gt;&#xA;&lt;li&gt;Run - what is the task?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;So essentially you just chain a bunch of task and define your run function to call your previously built scripts using something like subprocess.  In requirement, you would reference the last step, and for output you would point towards where the file is being written.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pro for doing this is that if your process breaks in task 50 out of 100, you don't have to rerun all 50 task, luigi will go down the dependency tree until it finds a requirement not fulfilled.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.r-bloggers.com/integrating-python-and-r-part-ii-executing-r-from-python-and-vice-versa/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Calling R from Pytho&lt;/a&gt; &#xA;&lt;a href=&quot;https://github.com/spotify/luigi&quot; rel=&quot;nofollow noreferrer&quot;&gt;Luigi&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-05-08T17:54:41.670" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31402" PostTypeId="1" AcceptedAnswerId="33565" CreationDate="2018-05-08T18:43:49.513" Score="10" ViewCount="19366" Body="&lt;p&gt;Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression).&#xA;I have done some searching but was not able to find the right answer as some say it affects it and others say it doesn't.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="23305" LastEditDate="2018-05-08T19:29:50.037" LastActivityDate="2023-08-21T13:50:20.113" Title="Multicollinearity in Decision Tree" Tags="|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31404" PostTypeId="2" ParentId="565" CreationDate="2018-05-08T20:53:49.750" Score="16" Body="&lt;p&gt;Remember that the &lt;code&gt;GradientBoostingRegressor&lt;/code&gt; (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now if the tree in stage i predicts a value larger than the target variable for a particular training example, the residual of stage i for that example is going to be negative, and so the regression tree at stage i+1 will face negative target values (which are the residuals from stage i).&#xA;As the boosting algorithm adds up all these trees to make the final prediction, I believe this can explain why you may end up with negative predictions, even though all the target values in the training set were positive, especially as you mentioned that this happens more often when you increase the number of trees.&lt;/p&gt;&#xA;" OwnerUserId="51481" LastEditorUserId="51481" LastEditDate="2018-09-06T05:36:59.257" LastActivityDate="2018-09-06T05:36:59.257" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31405" PostTypeId="1" CreationDate="2018-05-08T22:38:28.837" Score="7" ViewCount="118" Body="&lt;p&gt;I'm looking into a problem where the data points have unequal features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each instance represents the progression of an item throughout the system. A number of them have progressed to their end point, others are still at intermediary stages. The number of stages is known (10 in total). I have the time when it enters each stage. The point is to make predictions as to when they will reach their final stage.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of ways to handle this. It seems that a decision tree might be able to handle this properly. Even if a stage's entry time isn't known, it should still be able to make a reasonable prediction from the data it does have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="51820" LastActivityDate="2018-11-03T15:20:55.870" Title="Prediction from data series with varying features" Tags="|machine-learning|feature-scaling|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31406" PostTypeId="1" AcceptedAnswerId="31416" CreationDate="2018-05-08T22:54:13.667" Score="3" ViewCount="23739" Body="&lt;pre&gt;&lt;code&gt;clf = tree.DecisionTreeClassifier(random_state = 0)&#xA;clf = clf.fit(X_train, y_train)&#xA;importances = clf.feature_importances_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;importances&lt;/code&gt; variable is an array consisting of numbers that represent the importance of the variables. I wonder what order is this? Is the order of variable &lt;code&gt;importances&lt;/code&gt; is the same as &lt;code&gt;X_train&lt;/code&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to make a plot from this. So order matters.&lt;/p&gt;&#xA;" OwnerUserId="51668" LastEditorUserId="26562" LastEditDate="2018-05-08T22:56:48.797" LastActivityDate="2018-05-09T06:21:13.963" Title="tree.DecisionTree.feature_importances_ Numbers correspond to how features?" Tags="|python|scikit-learn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31407" PostTypeId="1" CreationDate="2018-05-08T23:26:01.773" Score="1" ViewCount="621" Body="&lt;p&gt;I am trying to perform sentiment classification task where I have some text and some information about whether the user smiled or not. &#xA;Now when I use count-vectorizer to convert my text to feature set (Bag of Words) approach it results in more than 5000 features. When I add smile to these 5000 features, total number of features become 5001. However, I do not see any improvement in the classification accuracy. I think the 5000 sparse features dominate the one and only smile feature. Hence no effect on the classification. Can anyone recommend, what is the good way to merge textual feature with smile, prosody or other features?&lt;/p&gt;&#xA;" OwnerUserId="51823" LastActivityDate="2018-05-09T01:15:41.043" Title="How to combine sparse text features with user smile for sentiment classification?" Tags="|classification|sentiment-analysis|nlp|" AnswerCount="1" CommentCount="0" ClosedDate="2018-05-10T11:06:11.023" ContentLicense="CC BY-SA 4.0" />
  <row Id="31408" PostTypeId="1" CreationDate="2018-05-09T01:03:35.763" Score="3" ViewCount="562" Body="&lt;p&gt;My understanding of towers in inception architecture and in tensorflow terminology is that they are part of a neural network model for which separate computation can happen on forward phase and gradient computation phase of back-propagation, independently. My present understanding is based on another datascience.stackoverflow post &lt;a href=&quot;https://datascience.stackexchange.com/questions/16361/what-is-a-tower&quot;&gt;What is a tower?&lt;/a&gt;, tensorflow documentation at &lt;a href=&quot;https://www.tensorflow.org/tutorials/deep_cnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/tutorials/deep_cnn&lt;/a&gt; and blog post on inception architecture &lt;a href=&quot;https://pseudoprofound.wordpress.com/2016/08/28/notes-on-the-tensorflow-implementation-of-inception-v3/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pseudoprofound.wordpress.com/2016/08/28/notes-on-the-tensorflow-implementation-of-inception-v3/&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I read Xception paper (&lt;a href=&quot;https://arxiv.org/abs/1610.02357&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/abs/1610.02357&lt;/a&gt;) in introduction section &quot;The Inception Hypothesis&quot;, it talks about average pooling tower. Does this mean it is referring to branches within a inception module as towers, even though independent backpropagation computation will not be performed on these branches? &lt;/p&gt;&#xA;" OwnerUserId="51811" LastActivityDate="2018-05-09T01:03:35.763" Title="What are towers in inception architecture and tensorflow?" Tags="|deep-learning|tensorflow|terminology|inception|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31409" PostTypeId="2" ParentId="31407" CreationDate="2018-05-09T01:15:41.043" Score="1" Body="&lt;p&gt;If you are using a RNN and the 5000 features refers to size of your dictionary, I would recommend you consider an embedding of your words, eg &lt;a href=&quot;https://www.tensorflow.org/tutorials/word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt;. Each word would then result in a maybe 200-300 - sized vector, to which you can concatenate the single smile feature, and then run that through your RNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If that is not doing much, you can run the embedded text through an RNN, and then take the result of that (lets say 10-20 features per time-step) and concatenate the smile feature, and run those through a second RNN. The idea is the first one is a further compression over time, and the second is your 'final' RNN which works at a higher level of abstraction. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, this is a classic case of combining a high-dimensional feature set with a low dimensional one, to which there are many approaches, the question being, when do you join the features. Take a look at &lt;a href=&quot;https://arxiv.org/abs/1708.02099&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;, which covers a few ideas on multi-modal features aka when and how to merge them (diagram 3.2), though I think it is more instructive than useful since their features are more rich than what you have with the single additional feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another thing to try is to soften the 'smile' feature if it is not so already, aka if it is a 0 or 1, make it a random (0, 0.1) and (0.9, 1) respectively. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-09T01:15:41.043" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31410" PostTypeId="2" ParentId="27575" CreationDate="2018-05-09T02:34:41.303" Score="1" Body="&lt;p&gt;While an RNN using one-hot encoded moves is possible, I would suggest that your model needs to understand chess (or similar complex games) at a deeper level to be able to associate comments to positions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would encode the position itself (eg a layered representation like in &lt;a href=&quot;https://arxiv.org/pdf/1712.01815&quot; rel=&quot;nofollow noreferrer&quot;&gt;Alpha-Zero&lt;/a&gt; paper), and pass those through a conv-RNN to model the temporal relationship between positions corresponding to annotations. (Maybe you can find a pretrained model for evaluation as a baseline).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read a fair amount of chess commentary, and one thing that is common is that there are cases of single move as well as comments about several moves at a time (think opening lines, or forced continuations), which are fairly discrete (my 'move 10 sacrifice' and my 'move 15 blunder' are not related by a hidden state). This leads me to think that instead of an RNN model, you are really trying to first &lt;strong&gt;Partition&lt;/strong&gt; then &lt;strong&gt;Embed&lt;/strong&gt; moves in a comment space - which then leads to a model that is first an RNN which may learn likely partitions of moves to be commented on, then another model, perhaps conditional autoencoder, is trained jointly on (position | comment) subsequences (you should still have plenty of features to work with at this point, so no worry about a weak signal). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The 'search' functionality here points not at a pure deep learning model, since you want to find specific instances of positions, rather than a typical position with that comment (ie sampling a latent space). This leads me to think that you create an explicit data-structure to map points in your final embedding space to their positions. So if you wanted to search 'Nice Sacrifice', model would find closest points in 'comment space' and return those games using something like cosine distance, or this last step is clustering of the (position | comment) space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, interesting problem, try the simpler thing first.&lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-09T02:34:41.303" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31413" PostTypeId="1" CreationDate="2018-05-09T04:52:58.173" Score="2" ViewCount="3722" Body="&lt;p&gt;I am solving a problem that address this question &quot;What are the Actions that lead to high or low score?&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have the following Data that consist of text and score , I want to derive the words or Actions from text that lead to high/low score&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have huge data , and text length can be a paragraph, sample data like below&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;+----------------------+-------+&#xA;|         Text         | Score |&#xA;+----------------------+-------+&#xA;| Support Team Goal    |    90 |&#xA;| Generate Lead        |    80 |&#xA;| Contact 30 customers |    30 |&#xA;| Support Team Goal    |    30 |&#xA;+----------------------+-------+&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Approach followed:&#xA;I followed Naive Bayes way.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;First I classified my Data into High (score(75 and above)) and Low (score below 75)&lt;/li&gt;&#xA;&lt;li&gt;I converted the High to Term Document Matrix and the Low as well&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;I found words appearing only in low&lt;/strong&gt; and judge they lead to low score (include word if frequency greater than 7 lets say)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;I found words appearing only in high&lt;/strong&gt; and judge they lead to high score (include word if frequency greater than 7 lets say in high)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;I found probability of common words,&lt;/strong&gt; For words appearing in both high or low, I calculated the probability of the word occurrence to happen in low/high (include word if probability greater than 75%)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Note: I worked with bigrams. Not sure if this is the right approach to analyze my problem, or there are better techniques. Please advise&lt;/p&gt;&#xA;" OwnerUserId="32220" LastActivityDate="2018-05-09T06:08:34.020" Title="How to improve Naive Bayes?" Tags="|classification|naive-bayes-classifier|ngrams|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31414" PostTypeId="1" CreationDate="2018-05-09T06:02:12.103" Score="1" ViewCount="606" Body="&lt;p&gt;I want to sell a day-pass and month-pass ticket, the price is segmented based on customers' residential community or other factor. I would like to know the pricing strategy by which I can get profit or how much loss I would get.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone share some ideas or reference?&lt;/p&gt;&#xA;" OwnerUserId="51829" LastActivityDate="2018-05-09T07:03:30.163" Title="machine learning application in ticket pricing" Tags="|machine-learning|deep-learning|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31415" PostTypeId="2" ParentId="31413" CreationDate="2018-05-09T06:05:56.357" Score="1" Body="&lt;p&gt;Without looking at the actual data, all we can really do is second guess and suggest best practices. Here are some pointers you can pursue - &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Gather more data&lt;/strong&gt; - If it can be done, nothing like it. &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Improve data quality&lt;/strong&gt; - &lt;em&gt;The algorithm will always be as good as the data&lt;/em&gt;. Try extensive cleaning methods like - &lt;/li&gt;&#xA;&lt;li&gt;Lowercase (basic) so all your data is standardized&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stemming / Lemmatization&lt;/a&gt; - These techniques reduces a word to its root form&lt;/li&gt;&#xA;&lt;li&gt;Try &lt;a href=&quot;https://machinelearningblogs.com/2016/12/01/opinion-mining-beyond-sentiment-analysis/&quot; rel=&quot;nofollow noreferrer&quot;&gt;parts of speech tagging (POS)&lt;/a&gt; and retain important parts of speech like nouns and look at their overall importance&lt;/li&gt;&#xA;&lt;li&gt;Stopword removal - Just like you tried doing word frequencies, there are popular word sets like the word &lt;em&gt;the&lt;/em&gt; that don't help much. They are filtered out&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://norvig.com/spell-correct.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Correcting Spelling mistakes&lt;/a&gt; - If you are expecting there will be spelling mistakes, might be a good idea to correct them&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Converting the unstructured data&lt;/strong&gt; - I see that you have used bigrams. Try using unigrams and trigrams as well, or in combinations, run your algorithm and see which one works better. Try &lt;code&gt;CountVectorizer&lt;/code&gt;, &lt;code&gt;TfidfVectorizer&lt;/code&gt; and other techniques like word embeddings as well&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Algorithms&lt;/strong&gt; - Finally focus on the algorithm itself. For Naive Bayes, focus on &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;MultinomialNB&lt;/a&gt;. Try &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;RandomForestClassifier&lt;/a&gt;and other ensemble family algorithms. Try Deep Learning techniques with &lt;a href=&quot;https://machinelearningmastery.com/prepare-text-data-deep-learning-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;. Fine tune hyperparameters based on the validation results&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Of course, there are other best practices like splitting your data into &lt;code&gt;train&lt;/code&gt;, &lt;code&gt;test&lt;/code&gt; and &lt;code&gt;cross validation&lt;/code&gt; sets. Evaluating the right metrics such as &lt;code&gt;accuracy&lt;/code&gt;, &lt;code&gt;precision&lt;/code&gt;, &lt;code&gt;recall&lt;/code&gt; and the &lt;code&gt;confusion matrix&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="30578" LastEditorUserId="40853" LastEditDate="2018-05-09T06:08:34.020" LastActivityDate="2018-05-09T06:08:34.020" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31416" PostTypeId="2" ParentId="31406" CreationDate="2018-05-09T06:21:13.963" Score="5" Body="&lt;p&gt;You can take the column names from &lt;code&gt;X&lt;/code&gt; and tie it up with the &lt;code&gt;feature_importances_&lt;/code&gt; to understand them better. Here is an example - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_iris&#xA;from sklearn.tree import DecisionTreeClassifier&#xA;import pandas as pd&#xA;&#xA;clf = DecisionTreeClassifier(random_state=0)&#xA;iris = load_iris()&#xA;iris_pd = pd.DataFrame(iris.data, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])&#xA;clf = clf.fit(iris_pd, iris.target)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am taking the iris example, converting to a &lt;code&gt;pandas.DataFrame()&lt;/code&gt; and fitting a simple &lt;code&gt;DecisionTreeClassifier&lt;/code&gt;. Once the training is done, you can take the &lt;code&gt;columns&lt;/code&gt; attribute of a pandas &lt;code&gt;df&lt;/code&gt; and make a &lt;code&gt;dict&lt;/code&gt; with the &lt;code&gt;feature_importances_&lt;/code&gt; output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;print(dict(zip(iris_pd.columns, clf.feature_importances_)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give you what you want -&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'sepal_length': 0.0, 'sepal_width': 0.013333333333333329, 'petal_length': 0.064055958132045052, 'petal_width': 0.92261070853462157}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="30578" LastActivityDate="2018-05-09T06:21:13.963" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31417" PostTypeId="2" ParentId="31414" CreationDate="2018-05-09T06:57:12.723" Score="5" Body="&lt;p&gt;This sounds a lot like a linear optimisation problem: &lt;em&gt;given certain resources and constraints, maximise profit&lt;/em&gt;. If you are new to this, check out some of these resources:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=o9YRFziAXUA&quot; rel=&quot;nofollow noreferrer&quot;&gt;walkthrough of an example&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.rand.org/pubs/reports/R366.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;in depth, this book&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/09/a-beginners-guide-to-shelf-space-optimization-using-linear-programming/&quot; rel=&quot;nofollow noreferrer&quot;&gt;practical example using Python #1&lt;/a&gt; - and &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/10/linear-optimization-in-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here #2&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Machine learning is generally an approach to model and understand data, hopefully in such a way as to allow us to make predictions in the future, given some data that is representational of past observations. It more often than not considers a dynamic world and non-linear functions: given the state of the world, what should the ticket price be? Practically speaking, selling tickets doesn't work that way - it would be very confusing for customers if ticket prices always change! One more point that would concern me with your specific problem, is that your variables (ticket price, frequency of trains etc.) are not mutually exclusive from the number of tickets sold. If you just make each ticket cost $1000 dollars each, profit would be zero, because nobody would by a ticket. What I want to say is: we're not classifying cats and dogs here  :-)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With your problem and using machine learning, I would be more inclined to try answering questions such as:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How many passengers can we expect next Saturday?&lt;/li&gt;&#xA;&lt;li&gt;Which are peak travel times in a certain region?&lt;/li&gt;&#xA;&lt;li&gt;which factors should we improve to increase customer feedback scores?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Maybe you can try looking at one of these approached to help in the optimisation, i.e to help find your linear constraints.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;One can of course attempt to frame it as a pure &lt;strong&gt;machine learning model&lt;/strong&gt;, and there are many ways of doing this; it is optimisation, after all. I think the first step, whether using some machine learning algorithm or a linear optimisation construction, is to understand your data, the effects of each factor and their relationship to your target variable to be maximised/minimised (profit/loss, respectively).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as I was told back when learning about this myself: &quot;The first step is to translate words into linear inequalities&quot;. So you could think about the factors you have and understand which ones effect price. If there are constraints/limitations, these should also be considered. For example, if you know that you are not allowed to sell more than 10,000 tickets on one day due to capacity constraints of your system.&#xA;There are some interesting points made in &lt;a href=&quot;https://stats.stackexchange.com/questions/326970/function-maximisation-using-non-linear-regression&quot;&gt;this thread over on Cross-Validated&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now you understand the data a little better, you could think about which models to try out. Without knowing more about your data, I can't really offer more guidance as to which models might be worth trying, but hopefully this answer helps you along that path.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-09T07:03:30.163" LastActivityDate="2018-05-09T07:03:30.163" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31419" PostTypeId="1" AcceptedAnswerId="31420" CreationDate="2018-05-09T07:52:16.183" Score="0" ViewCount="1227" Body="&lt;p&gt;So I'm working on a document processing AI and I already have a character recognition model which performs decently well. Now the problem is, how do I feed each character to the model in order to make predictions. Sliding window is one technique to segment arbitrary sequences of the image but it might not work in many cases as the words may get cut off. Can anyone suggest a robust way to detect text regions in an image? I've been researching about MSER but they claim to work well on grayscale images and not RGB. I know that we can convert RGB to grayscale but that is not performant enough.&lt;/p&gt;&#xA;" OwnerUserId="50640" LastActivityDate="2018-05-09T07:58:00.557" Title="Detecting text region from an image" Tags="|computer-vision|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31420" PostTypeId="2" ParentId="31419" CreationDate="2018-05-09T07:58:00.557" Score="2" Body="&lt;p&gt;You can use OpenCV's Scene Text Detection &lt;a href=&quot;https://docs.opencv.org/3.3.1/d4/d61/group__text.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.opencv.org/3.3.1/d4/d61/group__text.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-09T07:58:00.557" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31421" PostTypeId="2" ParentId="10522" CreationDate="2018-05-09T08:15:21.933" Score="0" Body="&lt;p&gt;This question has already been answered in StackOverflow. &lt;a href=&quot;https://datascience.stackexchange.com/questions/812/visualizing-a-graph-with-a-million-vertices?rq=1&quot;&gt;Visualizing a graph with a million vertices&lt;/a&gt;&#xA;. I would recommend use Graphexp for visualization with JanusGraph  as a back-end storage &lt;/p&gt;&#xA;" OwnerUserId="43590" LastActivityDate="2018-05-09T08:15:21.933" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31422" PostTypeId="1" CreationDate="2018-05-09T08:23:14.320" Score="1" ViewCount="167" Body="&lt;p&gt;I'm trying to learn a Gaussian Process Regressor in SKLearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried it both with and without feature (and output) normalization, and even though results seem similar-ish, the reported log marginal likelihood differs a lot (-5000 vs 9000)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My intuition would be that the log likelihood would not be very dependent on feature normalization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is: does feature scaling have an effect on the reported log-likelihood?&lt;/p&gt;&#xA;" OwnerUserId="42681" LastActivityDate="2018-05-09T08:23:14.320" Title="Gaussian Process Regression: does feature normalization affect final log-likelihood of model?" Tags="|python|scikit-learn|regression|gaussian|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31424" PostTypeId="1" AcceptedAnswerId="31425" CreationDate="2018-05-09T08:44:30.437" Score="8" ViewCount="4950" Body="&lt;p&gt;From my experiences the R Decision tree returns more accurate results than the python decision tree. Can anymore confirm this assumption and maybe knows the reason?&lt;/p&gt;&#xA;" OwnerUserId="42238" LastActivityDate="2022-09-30T13:28:15.077" Title="R vs. Python Decision Tree" Tags="|machine-learning|python|r|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31425" PostTypeId="2" ParentId="31424" CreationDate="2018-05-09T08:51:10.457" Score="7" Body="&lt;p&gt;Decision trees involve a lot of hyperparameters -&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;min / max samples in each leaf/leaves&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;size&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;depth of tree&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;criteria for splitting (gini/entropy) etc&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Now different packages may have different default settings. Even within &lt;code&gt;R&lt;/code&gt; or &lt;code&gt;python&lt;/code&gt; if you use multiple packages and compare results, chances are they will be different.&lt;/p&gt;&#xA;&lt;p&gt;There is nothing which suggests &lt;code&gt;R&lt;/code&gt; is &lt;em&gt;&amp;quot;better&amp;quot;&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;If you want to get the same results, you need to make sure the implicit defaults are similar. For instance, try running the following:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;fit &amp;lt;- rpart(y_train ~ ., data = x_train,method=&amp;quot;class&amp;quot;,&#xA;             parms = list(split = &amp;quot;gini&amp;quot;), &#xA;             control = rpart.control(minsplit = 2, minbucket = 1, xval=0, maxdepth = 30))&#xA;&#xA;(predicted5= predict(fit,x_test))&#xA;    setosa versicolor virginica&#xA;149      0  0.3333333 0.6666667&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Here, the parameters &lt;code&gt;minsplit = 2, minbucket = 1, xval=0&lt;/code&gt; and &lt;code&gt;maxdepth = 30&lt;/code&gt; are chosen so as to be identical to the &lt;code&gt;sklearn&lt;/code&gt;-options, see &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;code&gt;maxdepth = 30&lt;/code&gt; is the largest value &lt;code&gt;rpart&lt;/code&gt; will let you have;&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;sklearn&lt;/code&gt; on the other hand has no bound here. If you want &lt;code&gt;probabilities&lt;/code&gt; to be identical, you probably want to play around with the &lt;code&gt;cp&lt;/code&gt; parameter as well.&lt;/p&gt;&#xA;&lt;p&gt;Similarly, with&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;model = tree.DecisionTreeClassifier(criterion='gini', &#xA;                                    min_samples_split=20, &#xA;                                    min_samples_leaf=round(20.0/3.0), max_depth=30)&#xA;model.fit(iris.data, iris.target)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I get&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;print model.predict([iris.data[49]])&#xA;print model.predict([iris.data[99]])&#xA;print model.predict([iris.data[100]])&#xA;print model.predict([iris.data[149]])&#xA;print model.predict([[6.3,2.8,6,1.3]])&#xA;&#xA;[0]&#xA;[1]&#xA;[2]&#xA;[2]&#xA;[1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;which looks similar to your initial &lt;code&gt;R&lt;/code&gt; output.&lt;/p&gt;&#xA;&lt;p&gt;All in all, I believe the defaults in &lt;code&gt;R&lt;/code&gt; are better suited for the dataset that you are working on, hence the &lt;em&gt;&amp;quot;better&amp;quot;&lt;/em&gt; results. But rest assured, they are similar given the parameters are explicit and equal.&lt;/p&gt;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="30578" LastEditorUserId="30578" LastEditDate="2022-09-30T13:28:15.077" LastActivityDate="2022-09-30T13:28:15.077" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31427" PostTypeId="1" CreationDate="2018-05-09T09:48:45.023" Score="0" ViewCount="1682" Body="&lt;p&gt;I am trying to insert a column of values from one dataframe to another. The total count of values of both the indiviual dataframes is &lt;code&gt;91744&lt;/code&gt; but when I use &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df1.insert(loc=0, column='x1', value=ts)&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;to insert the values of ts as the first column in df1, the value count after insertion turns out to be &lt;code&gt;84175&lt;/code&gt; instead of actual &lt;code&gt;91744&lt;/code&gt;. Please tell me what is wrong with this.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df1.insert(loc=0, column='x1', value=ts)&#xA;&amp;gt;&amp;gt;&amp;gt; df1.x1.count()&#xA;84175&#xA;&amp;gt;&amp;gt;&amp;gt; ts.count()&#xA;0    91744&#xA;dtype: int64&#xA;&#xA;&amp;gt;&amp;gt;&amp;gt; df1.describe()&#xA;                 x1            x2&#xA;count  84175.000000  91744.000000&#xA;mean       0.249044     11.744387&#xA;std        0.191239     13.294185&#xA;min        0.000000      1.000000&#xA;25%        0.108383      3.000000&#xA;50%        0.212760     12.000000&#xA;75%        0.311860     15.000000&#xA;max        1.000000    551.000000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sample data is given below.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df1.head()&#xA;   x2  x3  x4   x5                   x6  y&#xA;0  12   1  13  497  2017-11-07 09:30:38  0&#xA;1  25   1  17  259  2017-11-07 13:40:27  0&#xA;2  12   1  19  212  2017-11-07 18:05:24  0&#xA;3  13   1  13  477  2017-11-07 04:58:08  0&#xA;4  12   1   1  178  2017-11-09 09:00:09  0&#xA;&amp;gt;&amp;gt;&amp;gt; ts.head()&#xA;         x1&#xA;0  0.218561&#xA;1  0.269357&#xA;2  0.257698&#xA;3  0.238417&#xA;4  0.164645&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My objective was to add the &lt;code&gt;x1&lt;/code&gt; column from of &lt;code&gt;ts&lt;/code&gt; dataframe as the first column of &lt;code&gt;df1&lt;/code&gt; data frame.&lt;/p&gt;&#xA;" OwnerUserId="51648" LastEditorUserId="51648" LastEditDate="2018-05-09T10:40:41.490" LastActivityDate="2018-05-09T10:40:41.490" Title="Pandas column count mismatch after insert into dataframe" Tags="|python|pandas|dataframe|" AnswerCount="1" CommentCount="2" ClosedDate="2018-05-11T11:04:47.803" ContentLicense="CC BY-SA 4.0" />
  <row Id="31428" PostTypeId="2" ParentId="31384" CreationDate="2018-05-09T10:17:36.763" Score="2" Body="&lt;p&gt;I've solved the problem with three changes:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;The weights are too small at the beginning, remove the scaling&lt;/li&gt;&#xA;&lt;li&gt;I think the biases should also be randomly initialized:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w1 = np.random.randn(S1, S0) &#xA;b1 = np.random.randn(S1, 1)&#xA;w2 = np.random.randn(S2, S1) &#xA;b2 = np.random.randn(S2, 1)&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Your are using a fixed learning rate of 0.03, change it to use the learning rate, and you can also increase it:&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;    learning_rate = 0.1 ...&#xA;&#xA;    w1 = w1 - dW1 * learning_rate&#xA;    w2 = w2 - dW2 * learning_rate&#xA;    b1 = b1 - dB1 * learning_rate&#xA;    b2 = b2 - dB2 * learning_rate&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The image below shpws what I get for the progression of J:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/7RMzW.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/7RMzW.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just for curiosity, if we change the sigmoid function with the ReLU, take a look to what happens to J (in the code I didn't change the name of the defs...). The learning with ReLU is much faster.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iYEPX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iYEPX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;    def sigmoid(z):&#xA;    return np.maximum(0,z)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&#xA;    def sigmoid_derivative(x):&#xA;    x[x &amp;lt; = 0] = 0&#xA;    x[x &amp;gt; 0]  = 1&#xA;    return x&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51714" LastEditorUserId="51714" LastEditDate="2018-05-09T10:38:29.680" LastActivityDate="2018-05-09T10:38:29.680" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31429" PostTypeId="2" ParentId="31427" CreationDate="2018-05-09T10:29:52.903" Score="1" Body="&lt;p&gt;Can you share some of your data frame i would recommend using merge instead of insert if you have a common column let's say &lt;code&gt;ID&lt;/code&gt; for example your code will look like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mergedStuff = pd.merge(df1, df2, on=['id'], how='outer')&#xA;mergedStuff.head()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;as simple as that&lt;/p&gt;&#xA;" OwnerUserId="46527" LastActivityDate="2018-05-09T10:29:52.903" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31430" PostTypeId="1" CreationDate="2018-05-09T10:41:21.967" Score="3" ViewCount="4648" Body="&lt;p&gt;I have set of night images which I will be using for self driving. But I want to convert those images into day images. I have developed algorithm based on day image but it is not good for night images , so I want to convert night images to day images then feed into the network. As far as I have explored image colourization techniques of grey scale image ( converting night image to black and white and then coloring it) ànd night to day style transfer using GANs. Is it possible to achieve the goal with the above two mentioned techniques or you can also suggest any other way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example link. :-  &lt;a href=&quot;https://goo.gl/images/kTwnHB&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://goo.gl/images/kTwnHB&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37502" LastEditorUserId="37502" LastEditDate="2018-05-09T10:52:04.180" LastActivityDate="2018-05-09T21:50:45.183" Title="How to convert night image to day image?" Tags="|neural-network|computer-vision|generative-models|neural-style-transfer|self-driving|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31433" PostTypeId="1" AcceptedAnswerId="31440" CreationDate="2018-05-09T12:03:21.603" Score="0" ViewCount="212" Body="&lt;h2&gt;Right now&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I've read that:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hadoop&lt;/strong&gt;: allows you to perform any task you want, if it fits the Map-Reduce paradigm. It can add concurrence using a cluster, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;SciPy&lt;/strong&gt;: python based ecosystem for (in fact, a lot of things)...&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;So my question is&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;If I've data collected from an environment, and want to correlate the data, calculate means, extract conclusions, etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Is SciPy as suitable as Hadoop (or more) to do so?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm basically trying to choose between them in order to build a little example, so I wonder which one fits better, or may I be misunderstanding the uses of each one.&lt;/p&gt;&#xA;" OwnerUserId="51851" LastEditorUserId="51851" LastEditDate="2018-05-09T13:40:59.817" LastActivityDate="2018-05-14T12:58:07.330" Title="Are Hadoop and Python SciPy used for the same?" Tags="|apache-hadoop|scipy|data-analysis|" AnswerCount="2" CommentCount="0" ClosedDate="2018-05-11T11:04:32.413" ContentLicense="CC BY-SA 4.0" />
  <row Id="31434" PostTypeId="1" AcceptedAnswerId="31439" CreationDate="2018-05-09T12:18:54.210" Score="6" ViewCount="10187" Body="&lt;p&gt;Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example?&#xA;(What if I have a features ranging from 0 to 9 but there is an outlier of which value is 10000?)&#xA;Whether it creates a separate leaf for that outlier sample or would it be merged with some other tree's leaves?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;NULL Values : Do we need to replace the null values before building the model using Decision tree or it would be taken care automatically by the decision tree technique?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="50727" LastEditDate="2018-05-14T10:28:34.063" LastActivityDate="2021-05-18T02:49:49.020" Title="Handling outliers and Null values in Decision tree" Tags="|classification|regression|decision-trees|outlier|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31435" PostTypeId="1" CreationDate="2018-05-09T12:25:37.143" Score="4" ViewCount="8972" Body="&lt;p&gt;The &lt;code&gt;keras.wrappers.scikit_learn&lt;/code&gt; module can be used to build &lt;code&gt;KerasClassifier&lt;/code&gt; model.&lt;/p&gt;&#xA;&lt;p&gt;Can &lt;code&gt;Keras&lt;/code&gt; be used to build clustering models?&lt;/p&gt;&#xA;&lt;p&gt;If it can be, are there any examples for that?&lt;/p&gt;&#xA;" OwnerUserId="51856" LastEditorUserId="-1" LastEditDate="2020-12-22T16:42:16.890" LastActivityDate="2020-12-22T16:42:16.890" Title="Can Keras be used to build clustering models" Tags="|scikit-learn|clustering|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31436" PostTypeId="2" ParentId="31433" CreationDate="2018-05-09T12:26:21.930" Score="1" Body="&lt;p&gt;If you have the data on your local machine, I guess, although quite large, you can perform the operations you want avoiding the Map-Reduce philosophy, thus go for python and scipy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first thing you should bare is wether you want to perform Map-Reduce. Map-Reduce is a way of manipulating very large amounts of data. Taking it very simply, Map-Reduce divides the data into smaller chunks and the analysis is done over each chunk in parallel (Mapping stage). When all the chunks have been analyzed, the partial results have to be collected and summarized (Reducing-stage). Typically, Map-reduce/Hadoop is performed in both distributed data and computing systems (Amazon S3, etc... One important task of the mapping stage is to orchestrate how the different chunks of data are shared over the nodes that conform the distributed system). Hadoop is a software framework written in Java for doing Map-Reduce. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you decided your needs fit the Map-Reduce, go for Hadoop-Java. It is possible, though, to run Hadoop operations also in python.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-09T12:26:21.930" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31437" PostTypeId="2" ParentId="31435" CreationDate="2018-05-09T12:45:23.860" Score="2" Body="&lt;p&gt;Yes, you can use Keras to build clustering algorithms. Recalling that Keras is a high-level api for Tensorflow or Theano, and that you can build a K-means algorithm (very well known clustering technique) in Tensorflow, you should be able to build it in Keras. Example (in tensorflow):&lt;a href=&quot;https://learningtensorflow.com/lesson6/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://learningtensorflow.com/lesson6/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nevertheless, sklearn has in-built clustering algorithms like K-means or Gaussian Mixture Models, so at first, I think a better approach is using these sklearn funcionalities.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastEditorUserId="51714" LastEditDate="2018-05-10T06:34:36.793" LastActivityDate="2018-05-10T06:34:36.793" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31439" PostTypeId="2" ParentId="31434" CreationDate="2018-05-09T13:02:58.673" Score="7" Body="&lt;p&gt;Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there are few outliers (which should be the case: if not, you cannot use any model), then they will not be relevant to these proportions. For this reason, decision trees are robust to outliers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Null values: You have to replace them (unless the software you use already does that for you, which is not generally the case).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit about outliers:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I have said in outliers is only about classification trees. However, it is certainly not true in regression tress. Regression tree split criterium depends on the averages of the two groups that are splitted, and, as the average is severly affected  by outliers, then the regression tree will suffer from outliers. There are two main approaches to solve this problem: either remove the outliers or build your own decision tree algorithm that makes splits based on the median instead of the average, as the median is not affected by outliers. However, basing the tree algorithm on the median will be very slow, as computing the median is way slower than computing the average.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-09T15:53:47.060" LastActivityDate="2018-05-09T15:53:47.060" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31440" PostTypeId="2" ParentId="31433" CreationDate="2018-05-09T13:03:42.300" Score="2" Body="&lt;p&gt;I think you're quite confused.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster.  You can access file stores as you would in a local file store (with some modification) and modify things via Java API.  Furthermore, ON TOP OF the file system there exist a MapReduce engine that allows for distributive workflow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Python on the other hand is a generic programming language that can be made to do a myriad of task such as build a web applciation, to generating reports and even peforming analytics.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SciPy is a package that can be used in conjunction with Python (and often numpy) as a way to perform common scientific task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Truthfully, they focus on different paradigms.  If you have LARGE DATA (ie terabytes worth of it), it might be worth wild to setup a hadoop cluster (ie multiple servers and racks) and use Java MapReduce, Hive, Pig or Spark (of which there is a python version) to do analytics.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your data is small or you only have one computer, then it probably makes sense to just use python instead of adding the overhead of setting up hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:  Made correction via comment.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastEditorUserId="43294" LastEditDate="2018-05-14T12:58:07.330" LastActivityDate="2018-05-14T12:58:07.330" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31441" PostTypeId="2" ParentId="31434" CreationDate="2018-05-09T13:14:20.637" Score="6" Body="&lt;p&gt;Generally speaking, &lt;strong&gt;decision trees are able to handle outliers&lt;/strong&gt; because their leafs are constructed under metrics which aim to discriminate as much as possible the resulting subsets. Whether you are using Gini Impurity, Information Gain or Variance Reduction to construct your decision tree does not change the outcome : all of these models aim to create as large (and homogeneous) buckets as possible. Under this approach, what matters is to understand the &lt;strong&gt;general behavior of your features&lt;/strong&gt;. Individual behaviors (outliers) are disregarded because you gain very little information from them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that your outliers represent a tiny proportion of your dataset, it is very unlikely that a leaf will be created to isolate them (at least in the first steps of creation of your decision tree), because you will gain very little information on the complimentary subset. However, if you have a large amount of outliers (but then are they really outliers ?), and that they tend to have the same outcome, chances are that you may get a leaf to isolate them. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;strong&gt;null values should be treated&lt;/strong&gt; whether it is through replacement, transformation or deletion from your observations. This would depend on your dataset. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the amount of null values is quite significant in your dataset, you should consider creating an additional feature stating whether the value is missing or absent. Some implementations handle that directly by creating a boolean, or replace the missing values by an &quot;outlier&quot; value.&lt;/li&gt;&#xA;&lt;li&gt;If the amount of null values is quite insignificant, and your dataset is large enough, you should consider deleting them, because it is the simpler and safer approach.&lt;/li&gt;&#xA;&lt;li&gt;Else, you might try to replace them by an imputed value, whether it is mean, median, modal, or another value that you may calculate from your features.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="42829" LastActivityDate="2018-05-09T13:14:20.637" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31443" PostTypeId="1" AcceptedAnswerId="31445" CreationDate="2018-05-09T14:07:27.343" Score="3" ViewCount="1934" Body="&lt;p&gt;I am trying to USE Lag features AND Concat() and Shift() function, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    seies = Series.from_csv('sugar_price_2.csv', header=0, sep=';')&#xA;&#xA;    In [25]: prices = DataFrame(series.values)&#xA;&#xA;    In [26]: dataframe = concat([prices.shift(3), prices.shift(2), prices.shift(1), prices], axis=1)&#xA;&#xA;    In [27]: dataframe.coloumns = ['t-2', 't-1', 't', 't+1']&#xA;&#xA;    In [28]: print(dataframe.head(20))&#xA;&#xA;       0       0       0     0      &#xA;0      NaN     NaN     NaN  2800&#xA;1      NaN     NaN  2800.0  2800&#xA;2      NaN  2800.0  2800.0  2800&#xA;3   2800.0  2800.0  2800.0  2800&#xA;4   2800.0  2800.0  2800.0  2800&#xA;5   2800.0  2800.0  2800.0  2800&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But The 't-2', 't-1', 't' Coloumn names aren't showing up . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone say what's wrong in my code... &lt;/p&gt;&#xA;" OwnerUserId="51859" LastEditorUserId="51859" LastEditDate="2018-05-12T20:08:46.637" LastActivityDate="2018-05-12T20:08:46.637" Title="Handling Concat and Shift Feature in Pandas for Data Science" Tags="|python|neural-network|pandas|dataframe|" AnswerCount="1" CommentCount="2" ClosedDate="2018-05-11T11:04:55.667" ContentLicense="CC BY-SA 4.0" />
  <row Id="31444" PostTypeId="1" AcceptedAnswerId="31464" CreationDate="2018-05-09T15:34:56.877" Score="12" ViewCount="3752" Body="&lt;p&gt;I am reading about &lt;code&gt;SVM&lt;/code&gt; and I've faced to the point that non-kernelized &lt;code&gt;SVMs&lt;/code&gt; are nothing more than linear separators. Therefore, is the only difference between an &lt;code&gt;SVM&lt;/code&gt; and logistic regression the criterium to choose the boundary?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apparently, &lt;code&gt;SVM&lt;/code&gt; chooses the maximum margin classifier and logistic regression is the one that minimizes the &lt;code&gt;cross-entropy&lt;/code&gt; loss. Are there situations where &lt;code&gt;SVM&lt;/code&gt;  performs better than logistic regression or vice-versa?&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-14T10:04:12.547" LastActivityDate="2018-05-14T10:04:12.547" Title="The differences between SVM and Logistic Regression" Tags="|machine-learning|classification|svm|logistic-regression|generalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31445" PostTypeId="2" ParentId="31443" CreationDate="2018-05-09T15:50:33.487" Score="1" Body="&lt;p&gt;As was pointed out by &lt;strong&gt;@Stephan Rauch&lt;/strong&gt; in his comment, the names of the columns are stored in &lt;code&gt;dataframe.columns&lt;/code&gt; - the OP had a typo.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is a working example with dummy data, getting the same output as the user - using instead a little loop to compute the shifted values.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame&#xA;&#xA;prices = dict(&#xA;    col1=[0, 1, 2, 3, 4, 5, 6],&#xA;    col2=[2, 3, 4, 5, 6, 7, 8],&#xA;    col3=[5, 6, 7, 8, 9, 10, 11],&#xA;    col4=[12, 13, 14, 15, 16, 17, 18])&#xA;&#xA;dataframe = DataFrame.from_dict(prices)&#xA;print(dataframe)&#xA;new_col_names = ['t-2', 't-1', 't', 't+1']&#xA;dataframe.columns = new_col_names&#xA;print(dataframe)&#xA;&#xA;# Number of columns we have&#xA;N = len(dataframe.columns)&#xA;&#xA;for n, col in enumerate(dataframe.columns):&#xA;    shift_by = N - n - 1  # don't shift the final column&#xA;    dataframe[col] = dataframe[col].shift(periods=shift_by, axis=0)&#xA;&#xA;print(dataframe)&#xA;&#xA;# If desired, remove the new NaNs that appear in the first&#xA;final_dataframe = dataframe.drop(labels=dataframe.index[:N - 1], axis='index')&#xA;print(final_dataframe)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-09T15:50:33.487" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31446" PostTypeId="1" CreationDate="2018-05-09T16:06:05.853" Score="2" ViewCount="125" Body="&lt;p&gt;I have been self-learning data science from different sources. I have a dataset which was sent to me by my friend from one of her college courses. The work is to be done in R preferably.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Description: It is the response of survey and results regarding usage of services for financial transactions using traditional methods or mobiles as a medium in a developing country. The study was conducted to understand the usage of technology for financial services by men and women in developing countries to find out how comfortable they are to use technology for this purpose. A dictionary is provided with the dataset which you may refer to column details. The dataset is big and contains many missing values. In the dataset, for male they have used 0 and for female are using 1. The purpose of this problem is that you will be able to understand how to process real data for valuable information extraction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset dictionary: &lt;a href=&quot;https://1drv.ms/x/s!ArGTGzC7esWNa25KnJACZTXuGPw&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://1drv.ms/x/s!ArGTGzC7esWNa25KnJACZTXuGPw&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset: &lt;a href=&quot;https://1drv.ms/u/s!ArGTGzC7esWNbCQevyUWRWhgrl8&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://1drv.ms/u/s!ArGTGzC7esWNbCQevyUWRWhgrl8&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The tasks given to my friend were:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Preprocess the data &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Which feature engineering techniques you used?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) predict the gender of the survey respondent. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) Predict that respondent is female with a certain probability. 5) Find interesting patterns that are associated with a specific gender&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I want to know what preprocess can I do in the data. I can remove missing and duplicate values, but for missing values there is a problem. If the survey respondent chose other in the religion column (96 value in column DG3A), they specified the name in the next column. But if they gave a response such as 1 (which is for Christianity) in the DG3A column then the value in the next column would be empty. How would I cater for these missing values (which aren't really missing instead needed not to be filled)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I am not sure which ones should I use and how.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which anyone can also guide me which columns to use and how for the remaining parts it would be great. :)&lt;/p&gt;&#xA;" OwnerUserId="51870" LastEditorUserId="1330" LastEditDate="2021-08-29T19:57:35.470" LastActivityDate="2021-08-29T19:57:35.470" Title="Prediction questions related to the dataset" Tags="|r|dataset|data-mining|data-cleaning|prediction|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31447" PostTypeId="2" ParentId="27927" CreationDate="2018-05-09T16:29:06.750" Score="0" Body="&lt;p&gt;use pandas to efficient handle tables in python. Pandas has a &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.corr.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;tool&lt;/a&gt; to calculate correlation between two Series, or between to columns of a Dataframe. Assuming you have your data in a csv file, you can read it and calculate the correlation this way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;data = pd.read_csv(&quot;my_file.csv&quot;)&#xA;correlation = data[&quot;col1&quot;].corr(data[&quot;col2&quot;], method=&quot;pearson&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also choose the method used to calculate the correlation between this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-pearson&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-kendall&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-spearman&lt;/p&gt;&#xA;" OwnerUserId="49914" LastActivityDate="2018-05-09T16:29:06.750" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31448" PostTypeId="1" CreationDate="2018-05-09T17:27:06.427" Score="4" ViewCount="320" Body="&lt;p&gt;I have a pipeline built which at the end outputs a bunch (thousands to tens of thousands or more) of named entities. I'd like to do aggregates on those named entities (to see, e.g. how many times a specific named entity is mentioned in my corpus). A problem that I am arriving at; however, is that the named entities often don't match up with each other even though they are the same entity. For example, one instance of the named entity might be &quot;Dr. John Smith&quot; while another instance is &quot;John Smith&quot; or one instance might be &quot;Google&quot; while another might be &quot;Google Inc.&quot;. This makes aggregating quite hard to do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to deal with this issue and set &quot;Dr. John Smith&quot; to be the same entity as &quot;John Smith&quot;, I was thinking of doing word matching between my named entities. I.e. I would check if named entity A has a word in common with named entity B and if they do set them to be the same entity. This approach is obviously seriously flawed. I will be equating &quot;John Nguyen&quot; and &quot;John Smith&quot; as the same entity even though they are obviously not. What's potentially even worse with this method though is I might run into similarity chains where I have &quot;John Smith&quot; linked with &quot;Richard Smith&quot; linked with &quot;Richard Sporting Goods Inc.&quot; linked with &quot;Google Inc.&quot; etc etc... While I may be willing to allow issues arising from former problem through, the latter problem appears to be catastrophic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there any accepted techniques in the NLP community for dealing with this issue? &lt;/p&gt;&#xA;" OwnerUserId="41306" LastActivityDate="2019-11-21T15:17:09.843" Title="How to filter Named Entity Recognition results" Tags="|nlp|named-entity-recognition|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31449" PostTypeId="1" CreationDate="2018-05-09T17:28:06.770" Score="4" ViewCount="71" Body="&lt;p&gt;My questions follow the below page 4 excerpt from &lt;a href=&quot;http://isle.illinois.edu/sst/meetings/2015/hochreiter-lstm.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Hochreiter's LSTM paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If $f_{l_{m}}$ is the logistic sigmoid function, then the maximal&#xA;  value of $f^\prime_{l_{m}}$ is 0.25. If $y^{l_{m-1}}$ is constant and&#xA;  not equal to zero, then&#xA;  $|f^\prime_{l_{m}}(net_{l_{m}})w_{l_{m}l_{m-1}}|$ takes on maximal&#xA;  values where&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;$w_{l_{m}l_{m-1}} = {1 \over y^{l_{m-1}}} \coth \left( {1 \over 2}net_{l_{m}} \right)$,&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;goes to zero for $|w_{l_{m}l_{m-1}}| \rightarrow \infty$, and is less&#xA;  than 1.0 for $|w_{l_{m}l_{m-1}}| &amp;lt; 4.0$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;The derivative of the sigmoid $f_{l_{m}} =  f^\prime_{l_{m}} = \sigma$, is $\sigma(1-\sigma)$, so of course its maximum value is 0.25. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I don't understand the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Where does the $\coth$ come from?&lt;/li&gt;&#xA;&lt;li&gt;Why would $y^{l_{m-1}}$ be static? My understanding is that it is the activation of the unit, which changes each timestep.&lt;/li&gt;&#xA;&lt;li&gt;Multiplying a non-zero positive number, which $f^\prime_{l_{m}}$ could be, by $|w_{l_{m}l_{m-1}}|$ as it approaches infinity would make it approach infinity, not zero. What am I missing here?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance for your help.&lt;/p&gt;&#xA;" OwnerUserId="51871" LastActivityDate="2023-12-08T19:01:45.200" Title="Hochreiter LSTM (p. 4): Maximal values of logistic sigmoid derivative times weight" Tags="|machine-learning|neural-network|lstm|backpropagation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31450" PostTypeId="2" ParentId="31384" CreationDate="2018-05-09T17:38:21.587" Score="0" Body="&lt;p&gt;Here's the same in tf &lt;a href=&quot;https://github.com/AdityaSoni19031997/Machine-Learning/blob/master/__TensorFlow__/xor_in_tf.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;From Github Repo Mine..&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf    &#xA;&#xA;input_data = [[0., 0.], [0., 1.], [1., 0.], [1., 1.]]  # XOR input&#xA;output_data = [[0.], [1.], [1.], [0.]]  # XOR output&#xA;&#xA;n_input = tf.placeholder(tf.float32, shape=[None, 2], name=&quot;n_input&quot;)&#xA;n_output = tf.placeholder(tf.float32, shape=[None, 1], name=&quot;n_output&quot;)&#xA;&#xA;hidden_nodes = 5&#xA;&#xA;b_hidden_1 = tf.Variable(tf.random_normal([hidden_nodes]), name=&quot;hidden_bias&quot;)&#xA;W_hidden_1 = tf.Variable(tf.random_normal([2, hidden_nodes]), name=&quot;hidden_weights&quot;)&#xA;hidden_1 = tf.sigmoid(tf.matmul(n_input, W_hidden) + b_hidden)&#xA;&#xA;W_output = tf.Variable(tf.random_normal([hidden_nodes, 1]), name=&quot;output_weights&quot;)  # output layer's weight matrix&#xA;output = tf.sigmoid(tf.matmul(hidden, W_output))  # calc output layer's activation&#xA;&#xA;cross_entropy = tf.square(n_output - output)  # simpler, but also works&#xA;&#xA;loss = tf.reduce_mean(cross_entropy)  # mean the cross_entropy&#xA;optimizer = tf.train.AdamOptimizer(0.01)  # take a Adam Optimizer for optimizing with a &quot;stepsize&quot; of 0.01&#xA;train = optimizer.minimize(loss)  # let the optimizer train&#xA;&#xA;init = tf.initialize_all_variables()&#xA;&#xA;sess = tf.Session()  # create the session and therefore the graph&#xA;sess.run(init)  # initialize all variables  &#xA;&#xA;for epoch in range(0, 1000):&#xA;    # run the training operation&#xA;    cvalues = sess.run([train, loss, W_hidden, b_hidden, W_output],&#xA;                       feed_dict={n_input: input_data, n_output: output_data})&#xA;&#xA;    if epoch % 200 == 0:&#xA;        print(&quot;&quot;)&#xA;        print(&quot;step: {:&amp;gt;3}&quot;.format(epoch))&#xA;        print(&quot;loss: {}&quot;.format(cvalues[1]))&#xA;&#xA;print(&quot;&quot;)&#xA;print(&quot;input: {} | output: {}&quot;.format(input_data[0], sess.run(output, feed_dict={n_input: [input_data[0]]})))&#xA;print(&quot;input: {} | output: {}&quot;.format(input_data[1], sess.run(output, feed_dict={n_input: [input_data[1]]})))&#xA;print(&quot;input: {} | output: {}&quot;.format(input_data[2], sess.run(output, feed_dict={n_input: [input_data[2]]})))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-05-09T17:38:21.587" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31451" PostTypeId="1" CreationDate="2018-05-09T17:38:56.750" Score="2" ViewCount="14" Body="&lt;p&gt;for my master thesis I am analyzing stock outs. I want to find out what factors played key roles that finally led to a stock out or (will) lead to a high stockout probability, for example late production, too long quality checks, late supplier etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What method is best to use here? I was thinking about random forest or logistic regression, but maybe there is even something better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="51872" LastActivityDate="2018-05-09T17:38:56.750" Title="Analysis of factors that have impact on explanatory variable" Tags="|random-forest|logistic-regression|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-11T11:04:00.253" ContentLicense="CC BY-SA 4.0" />
  <row Id="31452" PostTypeId="1" CreationDate="2018-05-09T17:42:09.470" Score="4" ViewCount="4895" Body="&lt;p&gt;I'm using Python with Keras to make a convolutional neural network (CNN) for an image classifier. I took about 50 images of documents and 150 images of non-documents for training. I shrunk the resolution, using &lt;code&gt;scipy.misc.imresize(small, (32,32))&lt;/code&gt; to standardize the images, when looking through these pixelated images I thought that I could still tell the difference between photos and documents, so I figured that the ML algorithm should be able to as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is in regards to the number of epochs and batch size. I'm currently using &lt;code&gt;model.fit(X_train_temp, y_train_temp, epochs=N_epochs, batch_size=batch_size, verbose=verbose)&lt;/code&gt; . I've seen this question regarding the topic here:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://datascience.stackexchange.com/questions/29719/how-to-set-batch-size-steps-per-epoch-and-validation-steps?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&quot;&gt;How to set batch_size, steps_per epoch and validation steps&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But the answer was very definition-based; I'm looking for intuition. I would like to know if there are general guidelines as to &lt;em&gt;what values&lt;/em&gt; to set the number of epochs and batch size to for a given problem. I performed a crude parameter sweep across the number of epochs and batch size. Here is the CNN model:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))&#xA;model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape, activation='sigmoid'))&#xA;model.add(Flatten())&#xA;model.add(Dense(num_classes, activation='sigmoid'))&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And here are the results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f03K0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f03K0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some observations that I've made so far. The circled region seems to be pretty good for training since high accuracy is achieved relatively early on and it does not seem to oscillate much as further epochs pass. It seems that &lt;code&gt;batch_size = 1&lt;/code&gt; is generally a bad idea since training does not seem to improve the model. Additionally, it seems that &lt;code&gt;batch_size &amp;gt;~ N_epochs&lt;/code&gt; seems to be desirable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 1:&lt;/strong&gt; is &lt;code&gt;batch_size = 1&lt;/code&gt; generally a bad? Or is this just for my particular parameter sweep?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 2:&lt;/strong&gt; Is it generally the case that &lt;code&gt;N_epochs &amp;gt;~ batch_size&lt;/code&gt; is recommended?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question 3:&lt;/strong&gt; Can anyone speak to the robustness of the generalizations in questions 2?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="49162" LastEditorUserId="29169" LastEditDate="2021-02-13T04:20:46.577" LastActivityDate="2021-02-13T04:20:46.577" Title="Keras intuition/guidelines for setting epochs and batch size" Tags="|neural-network|keras|convolutional-neural-network|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31453" PostTypeId="1" CreationDate="2018-05-09T18:35:27.333" Score="5" ViewCount="13015" Body="&lt;p&gt;How to decide what threshold to use for removing low-variance features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly, I have 100000 features and the variances look like:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Mfqks.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Mfqks.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could I e.g. take the average and use it to split this to ~half?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or some other method of grouping?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastEditorUserId="51687" LastEditDate="2018-05-09T18:45:17.083" LastActivityDate="2023-01-30T15:57:59.573" Title="How to decide what threshold to use for removing low-variance features?" Tags="|variance|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31456" PostTypeId="1" CreationDate="2018-05-09T21:21:46.867" Score="4" ViewCount="14964" Body="&lt;p&gt;I want to convert timedelta64[ns] to int&lt;/p&gt;&#xA;&#xA;&lt;p&gt;timedelta64[ns] example :&#xA;01:00:00 , 02:00:00 , 03:00:00 , 04:00:00&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to convert as : 1, 2 , 3 , 4&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Help me please T^T&lt;/p&gt;&#xA;" OwnerUserId="51880" LastActivityDate="2018-05-09T22:26:10.580" Title="how to convert timedelta64[ns] to int?" Tags="|python|pandas|" AnswerCount="2" CommentCount="1" ClosedDate="2018-05-11T11:05:26.073" ContentLicense="CC BY-SA 4.0" />
  <row Id="31457" PostTypeId="2" ParentId="31444" CreationDate="2018-05-09T21:46:00.027" Score="2" Body="&lt;p&gt;Logistic regression isn’t trying to find a class boundary per se as linear SVMs do. LR attempts to model the logit-transformed y scores using predictors. To use a silly analogy , LR tries to put the function ‘through the points’ while SVMs attempt to put support vectors ‘between the points’&lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-05-09T21:46:00.027" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31458" PostTypeId="2" ParentId="31430" CreationDate="2018-05-09T21:50:45.183" Score="2" Body="&lt;p&gt;One problem with this is that dark images simply contain less information. Anyone with a background in photography will tell you it’s easier to decrease exposure on a bright image than increase exposure on a dark one (you can’t create what’s not there but you can throw away information you have). If you want all images on the same playing field, perhaps do the reverse. Or, try to extract only edges or high contrast regions from all images. Or, model them differently. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-05-09T21:50:45.183" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31459" PostTypeId="2" ParentId="31456" CreationDate="2018-05-09T21:54:29.643" Score="2" Body="&lt;p&gt;You can do: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; t, _ , _ = &quot;01:00:00&quot;.split(&quot;:&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and then:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;int(t)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;That will return you: 1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Putting this together as a function would be:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def time_to_integer(time):&#xA;    t, _,_ = time.split(&quot;:&quot;)&#xA;    return int(t)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Please note that you should pass the &lt;em&gt;time&lt;/em&gt; variable as string. For example: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;time_to_integer(&quot;02:00:00&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Would return: 2&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-09T21:54:29.643" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31460" PostTypeId="1" AcceptedAnswerId="31465" CreationDate="2018-05-09T22:14:45.917" Score="2" ViewCount="52" Body="&lt;p&gt;I'm debating on using a DNN or CNN for this classification, but I have 10,000 spectrograms of songs. I was wondering if I need more because the images have a low variants.&lt;/p&gt;&#xA;" OwnerUserId="51639" LastActivityDate="2018-05-10T02:13:43.413" Title="Is 10,000 images for one class of spectrograms good enough for music classification model?" Tags="|machine-learning|image-classification|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31461" PostTypeId="2" ParentId="31456" CreationDate="2018-05-09T22:26:10.580" Score="2" Body="&lt;pre&gt;&lt;code&gt;In [106]: t = pd.to_timedelta(['01:12:34', '02:00:00', '03:00:00', '04:00:00'])&#xA;&#xA;In [107]: t.seconds // 3600&#xA;Out[107]: Int64Index([1, 2, 3, 4], dtype='int64')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-05-09T22:26:10.580" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31462" PostTypeId="2" ParentId="31453" CreationDate="2018-05-09T23:53:41.443" Score="2" Body="&lt;p&gt;You can do this a few ways, which I can list in ascending order of effort:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Pick a value that seems ok for you and your dataset by eye-balling it then simply cut variables below the theshold from the dataset&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Create a function, which given a threshold, tells you how many variables would be removed, if you used that threshold. Then create a simple plot and see if there is a certain level that seems appealing (this depends on your target model once data is ready).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Use some smarter functions that d a little more for you, e.g. &lt;a href=&quot;http://topepo.github.io/caret/pre-processing.html#zero--and-near-zero-variance-predictors&quot; rel=&quot;nofollow noreferrer&quot;&gt;the NearZeroVar function in the Caret package in R&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;There are, however, some arguments as to whether these approaches in general are optimal. Have a look at some of the discussion on &lt;a href=&quot;https://stats.stackexchange.com/questions/145602/justification-for-feature-selection-by-removing-predictors-with-near-zero-varian&quot;&gt;this thread over at Cross-Validated&lt;/a&gt;. The quote from the OP in that thread if from the book of the guy who wrote the above mentioned Caret package - Max Kuhn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The arguments against this approach say that you may be moving variables that, although they have low variance, might actually be extremely powerful in explaining your target (dependent) variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A final approach I can suggest goes into the realm of &lt;strong&gt;covariance&lt;/strong&gt;, that is to look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Collinearity&quot; rel=&quot;nofollow noreferrer&quot;&gt;the &lt;em&gt;collinearity&lt;/em&gt;&lt;/a&gt; between pairs of variables. I have done this in the past and it worked out well for me. The basic algorithm would look something like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Compute the covariance matrix across all your variables&lt;/li&gt;&#xA;&lt;li&gt;Find the pair with the highest covariance&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;correlation if you prefer that measure&lt;/li&gt;&#xA;&lt;li&gt;the plan is to remove one of these two variables&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;using your covariance matrix from step 1, compute which of these two variables in step 2 has the highest sum of covariances with the rest of the variables&lt;/li&gt;&#xA;&lt;li&gt;remove the variable with the highest sum of covariances&lt;/li&gt;&#xA;&lt;li&gt;Repeat steps 1-4 until you arrive at your desired number of variables, or a threshold is reached in terms of covariances or individual variable variances.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;h3&gt;EDIT:&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Here &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;is the Scikit-Learn class&lt;/a&gt; which can do basic variance thresholding for you - there is also &lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html#variance-threshold&quot; rel=&quot;nofollow noreferrer&quot;&gt;a short tutorial&lt;/a&gt;. They also present some ways to do recursive feature selection, similar in nature to my final approach outlined above.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-10T13:22:32.233" LastActivityDate="2018-05-10T13:22:32.233" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31463" PostTypeId="1" AcceptedAnswerId="31467" CreationDate="2018-05-10T01:52:35.937" Score="3" ViewCount="7110" Body="&lt;p&gt;I have started working on the Decision Tree Regressor and KNN Regressor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have built the model and not sure what are the metrics needs to be considered for evaluation. As of now I have considered Root mean squared error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we use $R^2$ values for Decision Tree and KNN or is it applicable only to Linear Regression Model&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have also pasted the $R^2$ and MSE values.&#xA;As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;R2 Score for train - KNN regression 0.8215942683102192&#xA;R2 Score for test - KNN regression 0.7160388084850589&#xA;Mean squared error for train - KNN regression 49.92162362176166&#xA;Mean squared error for test - KNN regression 78.30907381395349&#xA;&#xA;R2 Score for train - linear regression 0.6141419744748021&#xA;R2 Score for test - linear regression 0.6117893766210736&#xA;Mean squared error for train - linear regression 107.97107771851036&#xA;Mean squared error for test - linear regression 107.0583420197463&#xA;&#xA;R2 Score for train - Decision Tree regression 0.9962039204515297&#xA;R2 Score for test - Decision Tree regression 0.7866182225490949&#xA;Mean squared error for train - Decision tree regression 1.0622217832469776&#xA;Mean squared error for test  - Decision tree regression 58.84511637596899&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Updating the AIC and BIC values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;AIC value - Test - KNN       :   -3.8272461328797505&#xA;BIC value - Test - KNN       :   1169.4748549110836&#xA;AIC value - Test - Linear Reg:   -4.452667046616746&#xA;BIC value - Test - Linear Reg:   1250.154152783156&#xA;AIC value - Test - Decision T:   -3.2766787253336602&#xA;BIC value - Test - Decision T:   1098.4516593376377&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="40853" LastEditDate="2018-05-10T07:57:23.740" LastActivityDate="2018-05-10T07:57:23.740" Title="Evaluation metrics for Decision Tree regressor and KNN regressor" Tags="|regression|decision-trees|linear-regression|k-nn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31464" PostTypeId="2" ParentId="31444" CreationDate="2018-05-10T02:00:18.173" Score="11" Body="&lt;p&gt;If you use logistic regression and the &lt;code&gt;cross-entropy&lt;/code&gt; cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on &lt;a href=&quot;https://stats.stackexchange.com/a/29326/179078&quot;&gt;statistical methods&lt;/a&gt;. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fAEnA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fAEnA.png&quot; alt=&quot;logistic regression decision boundaries&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What &lt;code&gt;SVM&lt;/code&gt; does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. &lt;code&gt;SVM&lt;/code&gt; is &lt;a href=&quot;https://stats.stackexchange.com/a/95348/179078&quot;&gt;geometrically motivated rather than statistically&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/K5Flr.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K5Flr.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an &lt;code&gt;SVM&lt;/code&gt; for diminishing the risk of error on test data.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, as stated &lt;code&gt;SVM&lt;/code&gt; is based on geometrical properties of the data whilst &lt;code&gt;logistic regression&lt;/code&gt; is based on statistical approaches.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ostensibly, their results are not very different, but they are. &lt;code&gt;SVM&lt;/code&gt;s are better for generalization &lt;a href=&quot;http://tka4.org/materials/lib/Articles-Books/Speech%20Recognition/from%20Nickolas/SVM%20and%20Generalization.pdf&quot; rel=&quot;noreferrer&quot;&gt;1&lt;/a&gt;, &lt;a href=&quot;https://stats.stackexchange.com/q/259788/179078&quot;&gt;2&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-10T02:07:59.460" LastActivityDate="2018-05-10T02:07:59.460" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31465" PostTypeId="2" ParentId="31460" CreationDate="2018-05-10T02:13:43.413" Score="1" Body="&lt;p&gt;Agreed with Emre. One thing that can be helpful when asking the question is to look at comparable datasets since there's usually some around. For spectrograms for example, there are &lt;a href=&quot;https://www.kaggle.com/ollmer/labels-spectrograms-exploration/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;150,000 samples w/ 12 labels here&lt;/a&gt;, &lt;a href=&quot;http://deepsound.io/dcgan_spectrograms.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;2890 samples here&lt;/a&gt;, and &lt;a href=&quot;http://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset&quot; rel=&quot;nofollow noreferrer&quot;&gt;1 million samples with various labels here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Find a few examples and gauge a rough order of magnitude of samples per class given how close the tasks are to yours. That should give you at-least a starting point. Then, if lets say one set is very large, usually someone will have &lt;a href=&quot;https://arxiv.org/pdf/1703.01789&quot; rel=&quot;nofollow noreferrer&quot;&gt;written a paper&lt;/a&gt; using that set, and that can help start both the architecture and size jumping off point (and maybe a baseline for transfer learning) of your task. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-10T02:13:43.413" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31466" PostTypeId="2" ParentId="24577" CreationDate="2018-05-10T03:41:50.283" Score="1" Body="&lt;p&gt;You should use a Seq2Seq (which uses LSTM/GRU/RNNs) architecture for the task of cleaning text. The encoder network would take in the &quot;noisy&quot; sequence and the decoder would generate the &quot;cleaned&quot; sequence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Seq2Seq models is often used for Neural Machine Translation (NMT). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next thing is you will need a very large data set. Fortunately this shouldn't be too hard to generate noise injections (of real words, symbols, fake words, letter omissions etc) into sequences of text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A good guide for a similar task is in the following post. The implementation uses Keras and has all of the code. &lt;a href=&quot;https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30982" LastActivityDate="2018-05-10T03:41:50.283" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31467" PostTypeId="2" ParentId="31463" CreationDate="2018-05-10T05:44:58.267" Score="4" Body="&lt;p&gt;Generally when ever we are trying to compare between models and to choose the best one, we go for other metrics like &lt;a href=&quot;https://en.wikipedia.org/wiki/Akaike_information_criterion&quot; rel=&quot;nofollow noreferrer&quot;&gt;AIC&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Bayesian_information_criterion&quot; rel=&quot;nofollow noreferrer&quot;&gt;BIC&lt;/a&gt;, &lt;a href=&quot;https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it&quot;&gt;AUC&lt;/a&gt;(this is not applicable as it is used for classification algorithm) etc along with &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Now why are they important criteria because AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered. On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.&lt;/p&gt;&#xA;&lt;p&gt;Generally we use both AIC and BIC together.&lt;/p&gt;&#xA;&lt;p&gt;you can got through this &lt;a href=&quot;https://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other&quot;&gt;Link&lt;/a&gt; to understand better on AIC, BIC and which is better but conclusion is both are important.&lt;/p&gt;&#xA;&lt;p&gt;Now when we compare between &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; and AIC value:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; and AIC are answering two different questions. I want to keep this breezy and non-mathematical, so my statements are non-mathematical.  &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; is saying something to the effect of how well your model explains the observed data.  If the model is regression and non-adjusted &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; is used, then this is correct on the nose.&lt;br /&gt;&#xA;AIC, on the other hand, is trying to explain how well the model will predict on new data. That is, AIC is a measure of how well the  model will fit new data, not the existing data.  Lower AIC means that a model should have improved prediction.&lt;/p&gt;&#xA;&lt;p&gt;Frequently, adding more variables decreases predictive accuracy and in that case the model with higher &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; will have a higher (worse) AIC. A nice example of this is in &amp;quot;Introduction to Statistical Learning with R&amp;quot; in the chapter on regression models including 'best subset' and regularization. They do a pretty thorough analysis of the 'hitters' data set.  One can also do a thought experiment.  Imagine one is trying to predict output on the basis of some known variables.   Adding noise variables to the fit will increase &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt;, but it will also decrease predictive power of the model.  Thus the model with noise variables will have higher &lt;span class=&quot;math-container&quot;&gt;$R^2$&lt;/span&gt; and higher AIC.&lt;/p&gt;&#xA;&lt;p&gt;To understand more with respect to the above explanation, you can go through this &lt;a href=&quot;https://stats.stackexchange.com/questions/140965/when-aic-and-adjusted-r2-lead-to-different-conclusions/342719#342719&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;I have also pasted the R2 and MSE values. As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;To answer this I think you need to derive AIC and BIC values and finally you can decide on which model to choose.&lt;/p&gt;&#xA;&lt;p&gt;To derive the AIC value for Linear Regression can be done by using &lt;a href=&quot;https://stackoverflow.com/questions/45033980/how-compute-aic-for-linear-regression-model-in-python&quot;&gt;OLS&lt;/a&gt;, as the value of AIC is directly available.&lt;/p&gt;&#xA;&lt;p&gt;If you want to derive, you can go through this &lt;a href=&quot;https://www.reddit.com/r/statistics/comments/5h2rbw/does_python_have_a_package_for_aicbic/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt;&lt;/p&gt;&#xA;&lt;h1&gt;AIC:&lt;/h1&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$AIC= 2k - 2ln(sse)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;where k= number of variables&lt;/p&gt;&#xA;&lt;h1&gt;BIC:&lt;/h1&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$BIC = n * ln(sse/n) + k * ln(n)$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Do let me know if have any additional questions.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-10T06:02:35.847" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="31470" PostTypeId="1" CreationDate="2018-05-10T09:47:51.883" Score="1" ViewCount="483" Body="&lt;p&gt;I use deepAR RNN on AWS via python. I run the &lt;code&gt;estimator.fit(inputs=data_channels)&lt;/code&gt; and want to get insight on the algorithm performance and the fit that the model obtained, i.e. epoch to epoch convergence, quantile loss, RMSE, etc..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Instead I get a log dump like this:   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; [05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.5]): 0.0768988&#xA;[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.9]): 0.037788&#xA;[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, RMSE): 4433.98308522&#xA;#metrics {&quot;Metrics&quot;: {&quot;totaltime&quot;: {&quot;count&quot;: 1, &quot;max&quot;: 125569.84615325928, &quot;sum&quot;: 125569.84615325928, &quot;min&quot;: 125569.84615325928}, &quot;setuptime&quot;: {&quot;count&quot;: 1, &quot;max&quot;: 14.33110237121582, &quot;sum&quot;: 14.33110237121582, &quot;min&quot;: 14.33110237121582}}, &quot;EndTime&quot;: 1525945442.991945, &quot;Dimensions&quot;: {&quot;Host&quot;: &quot;algo-1&quot;, &quot;Operation&quot;: &quot;training&quot;, &quot;Algorithm&quot;: &quot;AWS/DeepAR&quot;}, &quot;StartTime&quot;: 1525945442.891469}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;which is completely unreadable and unprocessable. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a way to get those metrics back to python? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS. The folder in &lt;code&gt;output_path&lt;/code&gt; of estimator contains some files, but they do not seem to have the results I am looking for &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I managed to handle this way, yet horribly inefficient:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;First I capture the console output with this trick:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;f = io.StringIO()&#xA;with redirect_stdout(f):&#xA;    estimator.fit(inputs=data_channels)&#xA;dr_log = f.getvalue()&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then I manually process the long multiline string (blaaah:/)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;def dr_metrics(log, r):&#xA;rr = list()&#xA;final_res = dict()&#xA;for line in log:&#xA;    if &quot;Epoch&quot; in line:&#xA;        l = line.split(&quot;]&quot;)&#xA;        print(l)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    e = l[-3].split(&quot;[&quot;)[-1]&#xA;    b = l[-2].split(&quot;[&quot;)[-1]&#xA;    val = l[-1][:-2].split(&quot;=&quot;)[1]&#xA;    rr.append([e, b, val, &quot;Speed&quot; in line])&#xA;    print([e, b, val])&#xA;if &quot;Final loss&quot; in line:&#xA;    final_res['final_loss'] = line.split(&quot;:&quot;)[-1].split(&quot; &quot;)[1]&#xA;if &quot;#quality_metric&quot; in line:&#xA;    if &quot;mean_wQuantileLoss&quot; in line:&#xA;        final_res['mean_wQuantileLoss'] = line.split(&quot;=&quot;)[-1]&#xA;    if &quot;RMSE&quot; in line:&#xA;        final_res['RMSE'] = line.split(&quot;=&quot;)[-1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;res_df = pd.DataFrame(rr, columns=[&quot;Epoch&quot;, &quot;Batch&quot;, &quot;AvgLoss&quot;, &quot;EpochEnd&quot;])&#xA;res_df.head(10)&#xA;print(final_res)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="47522" LastEditorUserId="47522" LastEditDate="2018-05-24T11:59:24.053" LastActivityDate="2018-05-24T11:59:24.053" Title="Get deepAR fit() metrics to python" Tags="|python|rnn|aws|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31472" PostTypeId="1" AcceptedAnswerId="31565" CreationDate="2018-05-10T10:14:04.247" Score="3" ViewCount="154" Body="&lt;p&gt;I'm currently reading Hands on Machine Learning with Scikit-Learn &amp;amp; Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On page 465, it states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;As we will see, the training algorithm we will use requires two DQNs&#xA;  with the same archicture (but different parameters): one will be used&#xA;  to drive Ms. Pac-Man during training (the actor), and the other will&#xA;  watch the actor and learn from its trials and errors (the critic).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Is this a typical Q-learning implementation? If not, what is?&lt;/p&gt;&#xA;" OwnerUserId="50158" LastEditorUserId="50158" LastEditDate="2018-05-12T04:55:07.317" LastActivityDate="2018-05-12T07:45:03.117" Title="Why does Q-learning use an actor model and critic model?" Tags="|deep-learning|q-learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31475" PostTypeId="1" CreationDate="2018-05-10T11:50:43.797" Score="1" ViewCount="1908" Body="&lt;p&gt;Reading definition of interpolation below how are the O terms defined? Is this a value that is set manually?&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;h1&gt;&lt;a href=&quot;https://gist.github.com/ttezel/4138642#example-1&quot; rel=&quot;nofollow noreferrer&quot;&gt;Example&lt;/a&gt;&lt;/h1&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;P( Sam | I am ) = count( Sam I am ) / count(I am) = 1 / 2&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h3&gt;Interpolation using N-grams&lt;/h3&gt;&#xA;&lt;p&gt;We can combine knowledge from each of our n-grams by using interpolation.&lt;/p&gt;&#xA;&lt;p&gt;E.g. assuming we have calculated unigram, bigram, and trigram probabilities, we can do:&lt;/p&gt;&#xA;&lt;p&gt;P ( Sam | I am ) = Θ&lt;sub&gt;1&lt;/sub&gt; x P( Sam ) + Θ&lt;sub&gt;2&lt;/sub&gt; x P( Sam | am ) + Θ&lt;sub&gt;3&lt;/sub&gt; x P( Sam | I am )&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;sub&gt;(&lt;a href=&quot;https://i.stack.imgur.com/uIZUW.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;original problem statement image&lt;/a&gt;)&lt;/sub&gt;&lt;/p&gt;&#xA;" OwnerUserId="51908" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-04-18T15:45:26.153" Title="Interpolation in nlp - definition of O term" Tags="|nlp|interpolation|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31476" PostTypeId="1" CreationDate="2018-05-10T12:09:59.287" Score="1" ViewCount="110" Body="&lt;p&gt;I implemented a CNN image classifier with three classes. The samples belonging to the first class have a low variance (patterns and colors are very similar from image to image). The samples from the second class have a higher variance. Finally, the samples from the third class have been obtained by taking samples from the second class and adding a specific pattern in a random location.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What features should I add to a traditional CNN classifier in order to improve the performances, considering that I have a very large dataset (&gt;15K images per class)? Could it be useful to pre-train the network with the two similar classes and then adding the other one?&lt;/p&gt;&#xA;" OwnerUserId="45420" LastEditorUserId="45420" LastEditDate="2018-05-12T10:47:06.803" LastActivityDate="2018-05-12T10:47:06.803" Title="What is the best deep learning architecture for image classification with different intra-class variances?" Tags="|deep-learning|image-classification|cnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31477" PostTypeId="1" CreationDate="2018-05-10T12:14:17.083" Score="3" ViewCount="1111" Body="&lt;p&gt;I am working with Python. I have 3000 thousands images of front-faced watches like the following ones: &lt;a href=&quot;https://i.stack.imgur.com/WP6Op.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Watch1&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/vcUH9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Watch2&lt;/a&gt;, &lt;a href=&quot;https://i.stack.imgur.com/9M0Cj.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;Watch3&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to find an API which receives this collection of photos or even others taken under less ideal conditions (different background colour, darker lightning etc) and which finds/matches the most similar watches with one each other. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;By similarity I mean that I expect that a round in shape, brown watch with thin lace will be only matched with watches of round shape, of dark colour and with thin lace (from the same collection of photos).&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of APIs of this kind from &lt;code&gt;Google&lt;/code&gt;, &lt;code&gt;Amazon&lt;/code&gt;, &lt;code&gt;Microsoft&lt;/code&gt;, &lt;code&gt;TinEye&lt;/code&gt;, &lt;code&gt;Clarifai&lt;/code&gt;, &lt;code&gt;Indico&lt;/code&gt; etc but I am not sure that they will perform so well in a so specialised application. For example, these APIs are useful for matching car images with car images and not with food images but matching among  the same kind of objects (e.g. watches) based on a very high level of detail (shape, colour, thickness etc) is significantly more demanding.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, this is an application on a specific kind of objects like clothes with Indico: &lt;a href=&quot;https://indico.io/blog/clothing-similarity-how-a-program-is-more-fashionable-than-me/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://indico.io/blog/clothing-similarity-how-a-program-is-more-fashionable-than-me/&lt;/a&gt;. However, if you notice it, the results are not that good and essentially they could be retrieved to a great extent even by simply applying PCA and KNN to these images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Therefore, my question is: Is there any API which can match similar images based on a high level of detail?&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="45610" LastEditorUserId="45610" LastEditDate="2018-05-10T16:37:53.450" LastActivityDate="2018-05-10T16:37:53.450" Title="Are there any very good APIs for matching similar images?" Tags="|machine-learning|python|similarity|software-recommendation|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31478" PostTypeId="1" CreationDate="2018-05-10T14:30:16.393" Score="1" ViewCount="22" Body="&lt;p&gt;I have large text files on HDFS... I would like to label some text in those files to improve text analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you know of any tool like that?&lt;/p&gt;&#xA;" OwnerUserId="51914" LastActivityDate="2018-05-10T14:30:16.393" Title="Any Hadoop Tool to Annotate Text" Tags="|nlp|text-mining|tools|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31479" PostTypeId="1" CreationDate="2018-05-10T14:35:32.763" Score="1" ViewCount="16" Body="&lt;p&gt;Doing feature selection based on non-diagonalized covariance matrix vs on eigenvalues (that equal the diagonal of diagonalized covariance matrix)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the values differ in non-diagonalized covariance matrix, then can it be possible that the feature selection done in these two cases offers different outcomes? Is one of them better than another? Do both approaches retain the differences/ratios between variables/values or do they differ even in that sense?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-10T14:35:32.763" Title="Doing feature selection based on non-diagonalized covariance matrix vs on eigenvalues?" Tags="|feature-selection|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31480" PostTypeId="2" ParentId="12851" CreationDate="2018-05-10T14:41:03.450" Score="92" Body="&lt;p&gt;I recently created a tool for drawing NN architectures and exporting SVG, called &lt;a href=&quot;http://alexlenail.me/NN-SVG/index.html&quot; rel=&quot;noreferrer&quot;&gt;NN-SVG&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f96kw.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f96kw.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OpS8l.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OpS8l.png&quot; alt=&quot;second screenshot&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50013" LastEditorUserId="139757" LastEditDate="2022-08-29T01:27:44.843" LastActivityDate="2022-08-29T01:27:44.843" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="31481" PostTypeId="1" CreationDate="2018-05-10T15:23:52.593" Score="7" ViewCount="659" Body="&lt;p&gt;Can one build linear models on &quot;chunks&quot; of the data set, if one can't build them on the entire data set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly, I still have over 88k variables (features) left and one cannot do much with them without massive amounts of memory. But does doing models on &quot;blocks&quot; lose the interactions occuring between blocks or are there some technique for &quot;aggregating&quot; these?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastEditorUserId="51687" LastEditDate="2018-05-10T15:59:55.113" LastActivityDate="2018-05-10T15:59:55.113" Title="Can one build linear models on &quot;chunks&quot; of the data set, if one can't build them on the entire data set?" Tags="|bigdata|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31483" PostTypeId="2" ParentId="31481" CreationDate="2018-05-10T15:34:51.943" Score="6" Body="&lt;h2&gt;If &amp;quot;variables&amp;quot; refers to training examples:&lt;/h2&gt;&#xA;&lt;p&gt;You can use &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Stochastic_gradient_descent&quot; rel=&quot;noreferrer&quot;&gt;Stochastic Gradient Descent (SGD)&lt;/a&gt;&lt;/strong&gt; where each iteration uses one training example.&lt;/p&gt;&#xA;&lt;p&gt;Or you could use &lt;strong&gt;Mini-Batch Gradient Descent&lt;/strong&gt; where each iteration uses a partition of the training set.  SGD is Mini-Batch Gradient Descent where the partition size is one training example.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2&gt;If &amp;quot;variables&amp;quot; refers to features:&lt;/h2&gt;&#xA;&lt;p&gt;You should use &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Dimensionality_reduction&quot; rel=&quot;noreferrer&quot;&gt;dimensionality reduction&lt;/a&gt;&lt;/strong&gt; to reduce your number of features.  For instance, you can use &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Principal_component_analysis&quot; rel=&quot;noreferrer&quot;&gt;Principal Component Analysis (PCA)&lt;/a&gt;&lt;/strong&gt; to reduce your feature vector size while maintaining high variance.  This would also help your models train significantly faster.&lt;/p&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-10T15:57:14.430" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31484" PostTypeId="2" ParentId="31477" CreationDate="2018-05-10T16:06:12.527" Score="1" Body="&lt;p&gt;Never used this, but &lt;a href=&quot;http://matelabs.in&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://matelabs.in&lt;/a&gt; has something close... pretrained models on jewelry. You can probably extend those models. The new &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2018/05/google-released-latest-open-images-dataset/&quot; rel=&quot;nofollow noreferrer&quot;&gt;google dataset&lt;/a&gt; may also have labeled versions of the stuff you need. I cannot find the link I saw origionally, but it has bounding boxes for 20K classes down to 'teacup' 'teacup handle', 'tea', so I would not write google off before testing either.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since this is DataScience SE, I will give the more DS answer as well. The thing you are looking for sounds like a multi-category classifier. You can probably find good resources and pretrained models &lt;a href=&quot;https://github.com/topics/multi-label&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, and on the end, &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; has apis for &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2017/08/introduction-to-multi-label-classification/&quot; rel=&quot;nofollow noreferrer&quot;&gt;common classifiers&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The general idea is run your image through a Convnet, minus the classification part. You then have 1K-8K 'features' which a second algorithm set of classifiers, [decision forest](&lt;a href=&quot;https://pdfs.semanticscholar.org/.../99dff11dcb3a48dee07a19052b07fdd2e7fe.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pdfs.semanticscholar.org/.../99dff11dcb3a48dee07a19052b07fdd2e7fe.pdf&lt;/a&gt;&#xA;), &lt;a href=&quot;https://arxiv.org/abs/1801.08676&quot; rel=&quot;nofollow noreferrer&quot;&gt;logical compisitional model&lt;/a&gt;, &lt;a href=&quot;https://arxiv.org/pdf/1611.05088&quot; rel=&quot;nofollow noreferrer&quot;&gt;embedding&lt;/a&gt;, etc... gives a score for each binary attribute. &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-10T16:06:12.527" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31485" PostTypeId="1" CreationDate="2018-05-10T16:46:37.390" Score="2" ViewCount="44" Body="&lt;p&gt;Why don't all feature selection methods in sklearn allow specifying desired variance explained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;sklearn.decomposition.PCA does allow inputting a percentage of variance that one wants to be explained in place of &lt;code&gt;n_components&lt;/code&gt;. However other methods such as SparsePCA expect the user to &quot;know&quot; how many n_components he/she wants to take.&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-15T15:09:34.073" Title="Why don't all feature selection methods in sklearn allow specifying desired variance explained?" Tags="|scikit-learn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31486" PostTypeId="2" ParentId="28922" CreationDate="2018-05-10T16:54:41.593" Score="0" Body="&lt;p&gt;IMO, the goal of cleaning/preprocessing is to remove the &quot;randomness&quot; of the impact of missing/dirty values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine learning is just mathematics. And this specific set of mathematics require numbers to work with. An empty value doesn't represent a number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if I take a simple linear regression example where : &lt;/p&gt;&#xA;&#xA;&lt;p&gt;${y=a*x+b}$ where ${x=\emptyset}$, what is the algorithm supposed to output?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By cleaning the data, eg. adding the mode value, you minimize the impact of the missing data on your algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, don't forget that &lt;strong&gt;missing data is often just that, missing&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the variable &lt;code&gt;height&lt;/code&gt; of ${x_1}$ in my... tree classifier (example) is &lt;code&gt;null&lt;/code&gt;, it's not because the tree doesn't actually have an height, it's just because it is absent in the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, to answer your question, we need to generalize to deal with real world problems (missing data) while having the smallest impact possible on our prediction.&lt;/p&gt;&#xA;" OwnerUserId="44446" LastActivityDate="2018-05-10T16:54:41.593" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31487" PostTypeId="1" AcceptedAnswerId="31488" CreationDate="2018-05-10T17:19:16.613" Score="2" ViewCount="973" Body="&lt;pre&gt;&lt;code&gt;confusion_matrix(y_test1, pred)&#xA;That is my codes &#xA;&#xA;Confusion matrix, without normalization&#xA;True     [[724258    438]&#xA;value    [ 25396    302]]&#xA;          Predicted value&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I understand that is how this is the confusion matrix. But I do not know the order of classification result.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0   [[724258    438]                1 [[724258    438]&#xA;1   [ 25396    302]]                0 [ 25396    302]]&#xA;       0         1                        1        0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which order generally? I check the documentation, it does not tell me the specific result.&lt;/p&gt;&#xA;" OwnerUserId="51668" LastActivityDate="2018-05-10T18:03:46.900" Title="Confusion-matrix clarification from Python" Tags="|scikit-learn|confusion-matrix|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31488" PostTypeId="2" ParentId="31487" CreationDate="2018-05-10T18:03:46.900" Score="1" Body="&lt;p&gt;it's the first one  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;0   [[724258    438]       &#xA;1   [ 25396    302]]      &#xA;       0         1  &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;According to the documentation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sklearn.metrics.confusion_matrix(y_true, y_pred, labels=None, sample_weight=None) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;labels : array, shape = [n_classes], optional List of labels to index&#xA;  the matrix. This may be used to reorder or select a subset of labels.&#xA;  If none is given, those that appear at least once in y_true or y_pred&#xA;  are used in sorted order.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-10T18:03:46.900" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31489" PostTypeId="2" ParentId="15113" CreationDate="2018-05-10T18:23:34.240" Score="1" Body="&lt;p&gt;You may wish to do some data aggregation.&lt;br&gt;&#xA;for each week and for each item unit, aggregate all of its records, such that you could define how many are returned, also you need to expose transactions,in such a way it could infer those numbers of returns, following that formulate the problem as time series, you have time steps defined as weeks, preprocess the data by computing the relative value of return units instead of absolute ones, apply z-score normalization, then auto-encode the data by feeding it to RNN like models to predict the value at timestep t+1 given previous t timesteps.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-10T18:23:34.240" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31490" PostTypeId="2" ParentId="31485" CreationDate="2018-05-10T18:49:01.387" Score="3" Body="&lt;p&gt;Because PCA is just one of the feature selection methods, don't expect others to have a tractable computable formula to specify ahead of time the optimal number of new features.&lt;br&gt;&#xA;I suggest to start reading on Representation learning, and what we mean by latent variables.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-10T18:49:01.387" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31491" PostTypeId="1" CreationDate="2018-05-10T18:53:27.477" Score="1" ViewCount="435" Body="&lt;p&gt;I read in a blog that the decision tree has this disadvantage:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Not fit for continuous variables&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;If this is true, then why?&lt;/p&gt;&#xA;" OwnerUserId="51926" LastEditorUserId="29575" LastEditDate="2020-12-03T12:44:56.413" LastActivityDate="2023-11-28T17:08:56.010" Title="Disadvantage of decision tree" Tags="|decision-trees|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31492" PostTypeId="2" ParentId="31491" CreationDate="2018-05-10T19:00:39.467" Score="0" Body="&lt;p&gt;Decision trees work well with categorical variables because of the node structure of a tree. A categorical variable can be easily split at a node. For example, yes or no or 0 or 1. A continuous variable is just that, continuous along a range which cannot be easily split at a node. For example, a floating point number between 0 and 100. &lt;/p&gt;&#xA;" OwnerUserId="40044" LastActivityDate="2018-05-10T19:00:39.467" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31493" PostTypeId="2" ParentId="23969" CreationDate="2018-05-10T19:04:45.420" Score="2" Body="&lt;p&gt;There is some recent work based on Variational Auto-Encoder in RNN models.&lt;a href=&quot;https://arxiv.org/abs/1511.06349&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generating Sentences from a Continuous Space&lt;/a&gt;, with pytorch implementations: &lt;a href=&quot;https://github.com/timbmg/Sentence-VAE&quot; rel=&quot;nofollow noreferrer&quot;&gt;github code&lt;/a&gt;.&lt;br&gt;&#xA;they managed to compress the semantic, syntactic global feature of a sentence into some latent space expressed maybe with some finite 10 to 30 independent random variables (factorized distribution).&lt;br&gt;&#xA;the novel idea in this work, they interpolate between two sentences. and the results were quite amazing.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-10T19:04:45.420" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31494" PostTypeId="1" AcceptedAnswerId="31506" CreationDate="2018-05-10T19:08:27.673" Score="0" ViewCount="1312" Body="&lt;p&gt;I am using decision tree classifier to predict some block selected based on the below data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iJt0x.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iJt0x.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am able to predict the &quot;Block selected&quot; column based on the data. How to get the second best, third best prediction and so on(I need a ordered list)? Can I get this using decision tree? Or should I be using a different model?&#xA;Any thoughts on how to do this using python scikit-learn?&lt;/p&gt;&#xA;" OwnerUserId="51905" LastActivityDate="2018-05-13T21:32:03.843" Title="Next best predictions in decision tree" Tags="|python|classification|scikit-learn|decision-trees|supervised-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31495" PostTypeId="2" ParentId="31494" CreationDate="2018-05-10T19:10:50.180" Score="1" Body="&lt;p&gt;Remove the best block and run again in batches, removing the best selected at each round and placing is as the &lt;em&gt;n&lt;/em&gt; best prediction.&lt;/p&gt;&#xA;" OwnerUserId="40044" LastActivityDate="2018-05-10T19:10:50.180" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31496" PostTypeId="2" ParentId="31164" CreationDate="2018-05-10T19:21:01.897" Score="0" Body="&lt;p&gt;You seem to be looking for a clustering mechanism for your event sequence data based on a dissimilarity/ distance measure. &#xA;I would recommend you to explore TraMineR package. &lt;/p&gt;&#xA;" OwnerUserId="51659" LastActivityDate="2018-05-10T19:21:01.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31497" PostTypeId="2" ParentId="23969" CreationDate="2018-05-10T19:44:19.440" Score="10" Body="&lt;p&gt;&lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;noreferrer&quot;&gt;Word Mover’s Distance (WMD)&lt;/a&gt; is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to &quot;travel&quot; to reach the embedded words of another document.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DjJW1.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DjJW1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;Source: &lt;a href=&quot;http://proceedings.mlr.press/v37/kusnerb15.pdf&quot; rel=&quot;noreferrer&quot;&gt;&quot;From Word Embeddings To Document Distances&quot; Paper&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://github.com/RaRe-Technologies/gensim&quot; rel=&quot;noreferrer&quot;&gt;gensim package&lt;/a&gt; has a &lt;a href=&quot;https://radimrehurek.com/gensim/models/keyedvectors.html&quot; rel=&quot;noreferrer&quot;&gt;WMD implementation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your problem, you would compare the inputted sentence to all other sentences and return the sentence that has lowest WMD.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-10T19:44:19.440" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31498" PostTypeId="2" ParentId="23969" CreationDate="2018-05-10T19:45:40.950" Score="5" Body="&lt;p&gt;You can try an easy solution using &lt;strong&gt;sklearn&lt;/strong&gt; and it's going to work fine.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Use &lt;strong&gt;tfidfvectorizer&lt;/strong&gt; to get a vector representation of each text&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Fit&lt;/strong&gt; the vectorizer with your data, removing &lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;noreferrer&quot;&gt;stop-words.&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Transform&lt;/strong&gt; the new entry with the vectorizer previously trained&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Compute the &lt;strong&gt;cosine similarity&lt;/strong&gt; between this representation and each representation of the elements in your data set.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If you have a hugh dataset you can cluster it (for example using KMeans from scikit learn) after obtaining the representation, and before predicting on new data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This code perform all these steps. You can check it on my github &lt;a href=&quot;https://github.com/fedecaccia/text_clustering/blob/master/simple_text_recognition.py&quot; rel=&quot;noreferrer&quot;&gt;repo&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.feature_extraction.text import TfidfVectorizer&#xA;from sklearn.cluster import KMeans&#xA;from sklearn.metrics import adjusted_rand_score&#xA;import numpy&#xA;&#xA;texts = [&quot;This first text talks about houses and dogs&quot;,&#xA;        &quot;This is about airplanes and airlines&quot;,&#xA;        &quot;This is about dogs and houses too, but also about trees&quot;,&#xA;        &quot;Trees and dogs are main characters in this story&quot;,&#xA;        &quot;This story is about batman and superman fighting each other&quot;, &#xA;        &quot;Nothing better than another story talking about airplanes, airlines and birds&quot;,&#xA;        &quot;Superman defeats batman in the last round&quot;]&#xA;&#xA;# vectorization of the texts&#xA;vectorizer = TfidfVectorizer(stop_words=&quot;english&quot;)&#xA;X = vectorizer.fit_transform(texts)&#xA;# used words (axis in our multi-dimensional space)&#xA;words = vectorizer.get_feature_names()&#xA;print(&quot;words&quot;, words)&#xA;&#xA;&#xA;n_clusters=3&#xA;number_of_seeds_to_try=10&#xA;max_iter = 300&#xA;number_of_process=2 # seads are distributed&#xA;model = KMeans(n_clusters=n_clusters, max_iter=max_iter, n_init=number_of_seeds_to_try, n_jobs=number_of_process).fit(X)&#xA;&#xA;labels = model.labels_&#xA;# indices of preferible words in each cluster&#xA;ordered_words = model.cluster_centers_.argsort()[:, ::-1]&#xA;&#xA;print(&quot;centers:&quot;, model.cluster_centers_)&#xA;print(&quot;labels&quot;, labels)&#xA;print(&quot;intertia:&quot;, model.inertia_)&#xA;&#xA;texts_per_cluster = numpy.zeros(n_clusters)&#xA;for i_cluster in range(n_clusters):&#xA;    for label in labels:&#xA;        if label==i_cluster:&#xA;            texts_per_cluster[i_cluster] +=1 &#xA;&#xA;print(&quot;Top words per cluster:&quot;)&#xA;for i_cluster in range(n_clusters):&#xA;    print(&quot;Cluster:&quot;, i_cluster, &quot;texts:&quot;, int(texts_per_cluster[i_cluster])),&#xA;    for term in ordered_words[i_cluster, :10]:&#xA;        print(&quot;\t&quot;+words[term])&#xA;&#xA;print(&quot;\n&quot;)&#xA;print(&quot;Prediction&quot;)&#xA;&#xA;text_to_predict = &quot;Why batman was defeated  by superman so easy?&quot;&#xA;Y = vectorizer.transform([text_to_predict])&#xA;predicted_cluster = model.predict(Y)[0]&#xA;texts_per_cluster[predicted_cluster]+=1&#xA;&#xA;print(text_to_predict)&#xA;print(&quot;Cluster:&quot;, predicted_cluster, &quot;texts:&quot;, int(texts_per_cluster[predicted_cluster])),&#xA;for term in ordered_words[predicted_cluster, :10]:&#xA;print(&quot;\t&quot;+words[term])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-05-10T19:51:50.120" LastActivityDate="2018-05-10T19:51:50.120" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31499" PostTypeId="1" AcceptedAnswerId="58851" CreationDate="2018-05-10T21:41:23.843" Score="2" ViewCount="114" Body="&lt;p&gt;I am new to tensorflow and am learning the basics at the moment so please bear with me. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem concerns strange non-convergent behaviour of neural networks when presented with the supposedly simple task of finding a regression function for a small training set consisting only of &lt;span class=&quot;math-container&quot;&gt;$m = 100$&lt;/span&gt; data points &lt;span class=&quot;math-container&quot;&gt;$\{(x_1, y_1), (x_2, y_2),...,(x_{100}, y_{100})\}$&lt;/span&gt;, where &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt; are real numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I first constructed a function that automatically generates a computational graph corresponding to a classical fully connected feedforward neural network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import tensorflow as tf&#xA;import matplotlib.pyplot as plt&#xA;import math&#xA;&#xA;def neural_network_constructor(arch_list = [1,3,3,1], &#xA;                               act_func = tf.nn.sigmoid, &#xA;                               w_initializer = tf.contrib.layers.xavier_initializer(), &#xA;                               b_initializer = tf.zeros_initializer(),&#xA;                               loss_function = tf.losses.mean_squared_error,&#xA;                               training_method = tf.train.GradientDescentOptimizer(0.5)):&#xA;&#xA;    n_input = arch_list[0]&#xA;    n_output = arch_list[-1]&#xA;&#xA;    X = tf.placeholder(dtype = tf.float32, shape = [None, n_input])&#xA;&#xA;    layer = tf.contrib.layers.fully_connected(&#xA;            inputs = X,&#xA;            num_outputs = arch_list[1],&#xA;            activation_fn = act_func,&#xA;            weights_initializer = w_initializer,&#xA;            biases_initializer = b_initializer)&#xA;&#xA;    for N in arch_list[2:-1]:&#xA;        layer = tf.contrib.layers.fully_connected(&#xA;                inputs = layer,&#xA;                num_outputs = N,&#xA;                activation_fn = act_func,&#xA;                weights_initializer = w_initializer,&#xA;                biases_initializer = b_initializer)&#xA;&#xA;    Phi = tf.contrib.layers.fully_connected(&#xA;            inputs = layer,&#xA;            num_outputs = n_output,&#xA;            activation_fn = tf.identity,&#xA;            weights_initializer = w_initializer,&#xA;            biases_initializer = b_initializer)&#xA;&#xA;&#xA;    Y = tf.placeholder(tf.float32, [None, n_output])&#xA;&#xA;    loss = loss_function(Y, Phi)&#xA;    train_step = training_method.minimize(loss)&#xA;&#xA;    return [X, Phi, Y, train_step]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With the above default values for the arguments, this function would construct a computational graph corresponding to a neural network with 1 input neuron, 2 hidden layers with 3 neurons each and 1 output neuron. The activation function is per default the sigmoid function. X corresponds to the input tensor, Y to the labels of the training data and Phi to the feedforward output of the neural network. The operation train_step performs one gradient-descent step when executed in the session environment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, so good. If I now test a particular neural network (constructed with this function and the exact default values for the arguments given above) by making it learn a simple regression function for artificial data extracted from a sinewave, strange things happen:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4LrFP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4LrFP.png&quot; alt=&quot;before training&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Y0DY5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Y0DY5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before training, the network seems to be a flat line. After 100.000 training iterations, it manages to partially learn the function, but only the part which is closer to 0. After this, it becomes flat again. Further training does not decrease the loss function anymore.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This get even stranger, when I take the exact same data set, but shift all x-values by adding 500:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nPzhJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nPzhJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/DP78W.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DP78W.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here, the network completely refuses to learn. I cannot understand why this is happening. I have tried changing the architecture of the network and its learning rate, but have observed similar effects: the closer the x-values of the data cloud are to the origin, the easier the network can learn. After a certain distance to the origin, learning stops completely. Changing the activation function from sigmoid to ReLu has only made things worse; here, the network tends to just converge to the average, no matter what position the data cloud is in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there something wrong with my implementation of the neural-network-constructor? Or does this have something do do with initialization values? I have tried to get a deeper understanding of this problem now for quite a while and would greatly appreciate some advice. What could be the cause of this? All thoughts on why this behaviour is occurring are very much welcome!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks,&#xA;Joker&lt;/p&gt;&#xA;" OwnerUserId="51931" LastEditorUserId="55122" LastEditDate="2019-09-08T03:59:14.737" LastActivityDate="2019-09-08T03:59:14.737" Title="Neural network only converges when data cloud is close to 0" Tags="|machine-learning|neural-network|regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31500" PostTypeId="2" ParentId="30496" CreationDate="2018-05-10T22:22:47.007" Score="3" Body="&lt;p&gt;It depends on the ensemble model technique. If you are going to use a bagging approach then the term individual models or alternative models is the appropriate. But in the case of boosting methods the appropriate term is weak learner (classifier). Boosting algorithms is a family of machine learning algorithms that convert weak learners to strong ones. A weak learner is defined to be a classifier that is only slightly correlated with the true classification (it can label examples better than random guessing).&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-10T22:22:47.007" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31502" PostTypeId="2" ParentId="31491" CreationDate="2018-05-10T23:10:21.017" Score="0" Body="&lt;p&gt;It can't be used for micro trend detection as each node looks for values greater than or smaller than the threshold value. &lt;/p&gt;&#xA;" OwnerUserId="26444" LastActivityDate="2018-05-10T23:10:21.017" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31503" PostTypeId="1" CreationDate="2018-05-10T23:27:52.500" Score="1" ViewCount="2907" Body="&lt;p&gt;I want to create sparse feed-forward networks in Pytorch and Tensorflow, i.e., say each node is only connected to k number of neurons of the next layer where k is strictly less than the total number of nodes in the next layer. But all the tutorials/examples I have seen so far are for fully connected feed-forward networks. Is there any way to construct such sparse networks in these tools?&lt;/p&gt;&#xA;" OwnerUserId="51939" LastEditorUserId="29575" LastEditDate="2018-05-10T23:36:22.050" LastActivityDate="2019-04-08T15:03:03.793" Title="Sparse connections in feedforward network tensorflow or pytorch?" Tags="|neural-network|deep-learning|tensorflow|pytorch|" AnswerCount="4" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31504" PostTypeId="1" CreationDate="2018-05-11T01:22:38.053" Score="1" ViewCount="2763" Body="&lt;p&gt;I was trying to create a design document for a machine learning project. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a flow perspective, I know that I would have a training flow, a validation/test flow followed by a production flow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, from a design document perspective, I was no able to find any sample template that details out the different sections / subsections that a design document for a Machine Learning project should contain. I believe that the design doc for a ML project should be different from a normal project.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, what are the different documents that a ML project should have, from a process perspective?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas in this direction would be helpful.&lt;/p&gt;&#xA;" OwnerUserId="12312" LastActivityDate="2018-05-11T04:28:05.413" Title="Machine Learning Design Document" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31505" PostTypeId="2" ParentId="31323" CreationDate="2018-05-11T01:29:15.233" Score="2" Body="&lt;p&gt;Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then slowly increase your training batch size and see the impact on the results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you believe that the training set is a partially good representation of the dataset, go ahead do a test run. Then keep growing from there.&lt;/p&gt;&#xA;" OwnerUserId="12312" LastActivityDate="2018-05-11T01:29:15.233" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31506" PostTypeId="2" ParentId="31494" CreationDate="2018-05-11T02:29:10.607" Score="2" Body="&lt;p&gt;What you are looking for is a probabilistic classification for 2+ classes (multi class) i.e. evaluating a probability of being associated to a list of &lt;strong&gt;&lt;em&gt;k&lt;/em&gt;&lt;/strong&gt; classes given the observation. &lt;em&gt;k = 4 blocks/classes&lt;/em&gt; for your example. Its a &lt;em&gt;one vs many&lt;/em&gt; situation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Binary classifiers (like logistic regression) have been extended to multi class problems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the strategy used is to train/test your model &lt;em&gt;k&lt;/em&gt; times, each time keeping one class in predicted block intact and turning other three predicted class to a single dummy class. Then use a binary probabilistic classifier(eg. logistic regression) to predict class probabilities. The order of class preference will be in decreasing order of class probabilities. &#xA;For logistic regression and for other classifiers, this is inherently built in by scikit and applied by default:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;lr = linear_model.LogisticRegression(multi_class = 'ovr')&#xA;# ovr is &quot;one vs rest&quot;&#xA;lr.predict_proba(test_features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;refer to &lt;a href=&quot;http://scikit-learn.org/stable/modules/multiclass.html#multilabel-classification-format&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multi-class capability built&#xA; in scikit&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yes, you can even use a pruned decision tree to get the class probabilities. But most probably you will not be able to get 2nd, 3rd... best predictions for most of your observations from a single tree due to the underlying splitting mechanism of algorithm.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; dt = tree.DecisionTreeClassifier(min_samples_split=25)&#xA; dt.predict_proba(test_features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Therefore, rather then relying on a single decision three, better option would be to use a &lt;em&gt;randomforest&lt;/em&gt; classifier to obtain proportion of votes going to each class while making a prediction for an observation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, any multi-class classifier (or ensemble) which spits out a likelihood over multiple classes will do. Another eg. &lt;em&gt;XGBoost&lt;/em&gt; as pointed out in below comment by bradS. &lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-13T21:32:03.843" LastActivityDate="2018-05-13T21:32:03.843" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31508" PostTypeId="1" AcceptedAnswerId="31514" CreationDate="2018-05-11T03:18:03.280" Score="3" ViewCount="911" Body="&lt;p&gt;I am trying to do a variance-based uncertainty sampling like in the book &quot;Active Learning&quot; by Burr Settles, page 18, Figure 2.6. Link to the book: &lt;a href=&quot;http://active-learning.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://active-learning.net/&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use Gaussian process regressor from scikit-learn to fit a 1-D Gaussian function. The parameters of the regressor are the same as in this example (a noise-free case) &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;kernel = C(1.0, (1e-3, 1e3)) * RBF(10, (1e-2, 1e2))&#xA;gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=9)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Initially, I randomly choose two training data points and fit the curve based on them. The Figure below shows the fit (top) and the variance (bottom)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Q7j2U.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Q7j2U.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then I add one more training data point taken at the place where the variance is the largest, and re-fit the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/bk1ZC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bk1ZC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I repeat this process for a few iterations and the interpolation improves with each iteration, as expected. For example, this is the iteration 7:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Cgjpg.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Cgjpg.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, at the next iteration, the interpolation is worse. The variance suddenly increases:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/f2zIi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f2zIi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The next few iterations are fine again. For example, iteration 9:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pRg1W.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pRg1W.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But at the iteration 11 I get a large variance again:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VGG6s.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VGG6s.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Iterations 12 through 15 show very small variance again - the curve is interpolated very well at those iterations. I stopped at 15.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why does this sudden increase of variance happen at certain steps? How to avoid it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I change the parameter n_restarts_optimizer, this behavior is still observed but at different iteration steps. For example, when n_restarts_optimizer=8, the increase in variance is at iteration 9. When n_restarts_optimizer=7, the increase in variance is at iteration 4, 10, 11. When n_restarts_optimizer=4, the increase in variance is at iteration 7, 14. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I change the parameter length_scale in the RBF kernel from 10 to 1.0 I don't see this increase in variance anymore. However, when I make it even smaller (length_scale = 0.1) it appears again.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-11T08:04:07.037" Title="Gaussian process regression: sudden increase of the prediction's variance" Tags="|machine-learning|python|scikit-learn|regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31510" PostTypeId="2" ParentId="19362" CreationDate="2018-05-11T04:08:10.020" Score="3" Body="&lt;p&gt;The key is to first do &lt;code&gt;.get_layer&lt;/code&gt; on the Model object, then do another &lt;code&gt;.get_layer&lt;/code&gt; on that specifying the specific layer, &lt;strong&gt;THEN&lt;/strong&gt; do .output: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;layer_output = model.get_layer('Your-Model-Object').get_layer('the-layer-contained-in-your-Model-object').output&lt;/code&gt; &lt;/p&gt;&#xA;" OwnerUserId="10548" LastActivityDate="2018-05-11T04:08:10.020" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31511" PostTypeId="2" ParentId="31504" CreationDate="2018-05-11T04:28:05.413" Score="1" Body="&lt;p&gt;By ‘design document’ I assume you are authoring a technical doc in an industry job. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I treat this similarly to an academic journal publication but with less formatting and page restrictions. Ultimately, you want to provide sufficient information such that if Apple hired you tomorrow your replacement could reproduce the work. Further, your colleagues should be able to grasp the problem attacked, solution proposed, and evidence of success. I usually put the yawn-inducing deep stuff in appendix,  but suggested sections might include:&#xA;0) executive summary&#xA;1) background/motivation&#xA;2) data sources&#xA;3) analysis/algorithm/model&#xA;4) validation performance &#xA;5) [optional] real world test results/pilot&#xA;6) [optional] estimated return on investment &#xA;7) [appendix] SQL source code, modeling heuristic, outlier treatment, etc. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-05-11T04:28:05.413" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31513" PostTypeId="1" CreationDate="2018-05-11T07:22:28.283" Score="3" ViewCount="29" Body="&lt;p&gt;I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need guidance on how can I combine those node categories to get the overall category of the tree. What classification techniques could be used ?&#xA;As of now I know about usage of markov chains for this task from this research paper, &lt;a href=&quot;http://www.dbs.informatik.uni-muenchen.de/Publikationen/Papers/kdd-MatthiasSchubert.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Web Site Mining&lt;/a&gt;. I would be grateful if i could get some more ideas about how to perform the task. &lt;/p&gt;&#xA;" OwnerUserId="33509" LastActivityDate="2018-05-11T08:49:25.660" Title="What could be some Classification techniques to classify a tree of webpages given the category of each webpage" Tags="|multiclass-classification|markov-process|machine-learning-model|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31514" PostTypeId="2" ParentId="31508" CreationDate="2018-05-11T08:04:07.037" Score="1" Body="&lt;p&gt;Hmm, I've seen this before. It seemed to be some kind of overfitting on the time component where only very nearby samples were taken into account, before defaulting to the prior (which has zero mean). That would explain why increasing the length scale would help. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you see large changes in the hyper parameters of the kernel between iterations (&lt;code&gt;gp.kernel_.get_params()&lt;/code&gt;)? You could consider using the hyper parameters from a previous iteration as initial guess for the next iteration.&lt;/p&gt;&#xA;" OwnerUserId="42681" LastActivityDate="2018-05-11T08:04:07.037" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31515" PostTypeId="2" ParentId="31513" CreationDate="2018-05-11T08:19:36.747" Score="1" Body="&lt;p&gt;There are of course a number of ways this can be done, such as majority voting or some other rule-based algorithm, however it can also be done through supervised learning since you have some labels for the trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would make the input space of my model the normalized frequency of the categories for a tree. This means you will need a dictionary of possible categories for the nodes, usually obtained from your training set. Then you can tabulate the frequency of instances. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if we have a website with the following node classes:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;News: 5&lt;/li&gt;&#xA;&lt;li&gt;Opinions: 9 &lt;/li&gt;&#xA;&lt;li&gt;About: 1&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then we can formulate our input vector as $[0.33, 0.6, 0.067]$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can then train this model using your already labeled trees. The model will then be capable of classifying future trees in this same way. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To determine the top $K$ classes for a tree you will need a model which can do this (most can). If you use K-NN (different K) then you can pick the $K$ closest neighbourhoods. With Random Forests or Naive Bayes you can pick the $K$ classes with the highest probabilities.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;To consider the fact that you have a list of $K$ categories for each node you can add a &lt;strong&gt;weighting&lt;/strong&gt; when calculating the normalized frequencies. For example let's say we have 3 classifications and the following webpages (nodes).&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Page 1: News, Opinion, Commentary&lt;/li&gt;&#xA;&lt;li&gt;Page 2: News, Advertisement, Opinion&lt;/li&gt;&#xA;&lt;li&gt;Page 3: Commentary, News, Adverisement&lt;/li&gt;&#xA;&lt;li&gt;Page 4: News, Opinion, Advertisement&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Then the input vector can be calculated by awarding 3 points to the first category, 2 to the next and 1 to the last. This results in $[0.49, 0.21,0.17, 0.17]$. Alternatively, if you have a probability for these classifications you can use that as the weighting factor.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-05-11T08:49:25.660" LastActivityDate="2018-05-11T08:49:25.660" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31517" PostTypeId="1" CreationDate="2018-05-11T09:29:25.140" Score="1" ViewCount="79" Body="&lt;p&gt;In noise contrastive estimation, we learn a binary classifier to classify noise from the true distribution.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem I'm trying to solve&lt;/strong&gt; has discrete input variables yet it's intractable to sum over. I want to approximate the true distribution by using NCE.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The Question is:&lt;/strong&gt; is it okay to draw difficult examples (noise samples)&lt;br&gt;&#xA; the model is likely to misclassify, instead of sampling naive noise examples.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;From a manifold perspective&lt;/strong&gt; I want to learn the structure of the manifold, I know for each data point a group of data points that are for sure out of the manifold.&lt;br&gt;&#xA;So I directly replace the noise distribution by this (difficult examples distribution) is still valid? Am I still doing NCE?&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-11T09:29:25.140" Title="On Noise Contrastive Estimation, replace noise distribution with difficult examples" Tags="|distribution|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31520" PostTypeId="2" ParentId="31503" CreationDate="2018-05-11T11:53:37.780" Score="0" Body="&lt;p&gt;We don't use sparse connections in feedforward to make use of an efficient matrix product operations.&lt;br&gt;&#xA;But If you want to model that aspect, see &lt;a href=&quot;http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;dropout&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-11T11:53:37.780" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31522" PostTypeId="2" ParentId="31499" CreationDate="2018-05-11T12:09:49.993" Score="0" Body="&lt;p&gt;the input is simply not enough to correctly predict the output, the model can not learn the output conditional distribution P(y|x).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either you have to add more features to the Naive NN model, e.g concat the previous x to the current x to predict the current y or use RNN like models to model the problem as&lt;br&gt;&#xA; p(y_t|x_t,x_t-1,x_t-2,....x_t0)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;see &lt;a href=&quot;https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Time Series Prediction with LSTM&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-11T12:09:49.993" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31524" PostTypeId="1" CreationDate="2018-05-11T13:23:20.443" Score="1" ViewCount="63" Body="&lt;p&gt;I have a terror attack (tabular) data set. Each row is one attack and there are columns like: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Date of the attack (daily resolution)&lt;/li&gt;&#xA;&lt;li&gt;Location of the attack (long/lat, as well as city/country)&lt;/li&gt;&#xA;&lt;li&gt;Number of casualties&lt;/li&gt;&#xA;&lt;li&gt;Attack/weapon type&lt;/li&gt;&#xA;&lt;li&gt;A few boolean columns like whether it was a suicide attack&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Furthermore, I have a text column that holds a 2-3 sentence description of the attack. This is the main column I want to use for training/predicting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are several target columns of the form &quot;is_left_wing&quot;, &quot;is_right_wing&quot;, etc. The values are 0, 1, and -1. Here 0 means the attack didn't have the respective motive, 1 means it had the motive and -1 means it is unknown.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, my goal is to build a model that is trained on the 0 and 1 values in the target columns and makes predictions about the -1s.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The main thing I'm stuck on is how to extract features from the text column with the attack description. I have limited NLP experience and I want to use something more sophisticated than a simple bag of words model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would appreciate suggestions about the general approach to this problem (also some good readings on the topic).&lt;/p&gt;&#xA;" OwnerUserId="51973" LastActivityDate="2018-05-11T14:01:43.570" Title="Supervised learning model for extracting terror attack motives from attack summary (NLP)" Tags="|python|nlp|supervised-learning|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31526" PostTypeId="2" ParentId="31341" CreationDate="2018-05-11T13:30:54.763" Score="0" Body="&lt;p&gt;Could you tell us a little bit more about those 2 models ?&#xA;If your small network is recurrent, the computation graph of the model is not very parallelizable. It could explain the latency.&lt;/p&gt;&#xA;" OwnerUserId="51541" LastActivityDate="2018-05-11T13:30:54.763" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31527" PostTypeId="2" ParentId="31524" CreationDate="2018-05-11T14:01:43.570" Score="1" Body="&lt;p&gt;Use word embedding and encode the entire sentence into one fixed feature vector by using vanilla RNN or more sophisticated model like attentional LSTM.&lt;br&gt;&#xA;see &lt;a href=&quot;https://towardsdatascience.com/sentiment-analysis-using-rnns-lstm-60871fa6aeba&quot; rel=&quot;nofollow noreferrer&quot;&gt;Sentiment analysis using RNNs(LSTM)&lt;/a&gt;&lt;br&gt;&#xA;Concat the other features with the fixed length representation of the sentence and append on top of them either dense layer.&lt;br&gt;&#xA;the architecture &#xA; feedforward(concat(other_features,RNN(sentence)))&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-11T14:01:43.570" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31529" PostTypeId="1" CreationDate="2018-05-11T15:45:47.927" Score="2" ViewCount="672" Body="&lt;p&gt;I've been trying to find a simple tutorial to take me through using TensorFlow.js to do predictions on a series of values, but I haven't been able to find something simple and based in JS.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.  I have over a million values in this set, and they're just simple values in sequence.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any direction or helpful and simple tutorials would be really appreciated.&lt;/p&gt;&#xA;" OwnerUserId="51987" LastActivityDate="2018-07-11T06:19:15.420" Title="How to do time-series prediction in the browser" Tags="|tensorflow|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31530" PostTypeId="2" ParentId="31503" CreationDate="2018-05-11T15:49:10.787" Score="1" Body="&lt;p&gt;Agreed with Fadi. It is just not very efficient. You have to split up the input and weights tensor, into tensor.size()/k chunks and then do a separate mult,add operation on each pair of chunks. Even if you had an efficient indexing scheme of concatenating the chunks to mul-add in one go, you would then have to un-index output to a flat batched tensor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A few things to think about would be &lt;a href=&quot;https://arxiv.org/abs/1701.08734&quot; rel=&quot;nofollow noreferrer&quot;&gt;pathnet&lt;/a&gt; - &lt;a href=&quot;https://github.com/kimhc6028/pathnet-pytorch&quot; rel=&quot;nofollow noreferrer&quot;&gt;pytorch&lt;/a&gt; and &lt;a href=&quot;https://arxiv.org/abs/1605.07648&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; (I have a half-working implementation somewhere if you are interested), but both are variations on the efficient version of what you are talking about, which is some form of routing.  &lt;/p&gt;&#xA;" OwnerUserId="47488" LastActivityDate="2018-05-11T15:49:10.787" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31531" PostTypeId="1" CreationDate="2018-05-11T15:55:38.513" Score="1" ViewCount="185" Body="&lt;p&gt;I'm quite new to coding and even newer to machine learning, so please excuse if this is a stupid question to ask.&#xA;The project I am thinking about building is a program that helps correct pitch accent (the changes in high and low in between syllables) by analyzing speech from user input.&#xA;So let's say we have a random word I want to analyze, what I give to the Program is the representation of the audio (see further down) and a count for how many syllables the audio contains. As an output I want to know for each Syllable if it is either High or Low.&#xA;Let's do an example for the English word Animal . Pitch is the blue line&#xA;As an input I give the number of syllables (3) and a representation of the audio. As an output I want to see the Syllables mapped onto either High or Low, for the example &quot;animal&quot; [High, Low, Low/ 1,0,0] for &quot;malaria&quot; (4 syllables) I'd want [LLLH/0,0,0,1].&#xA;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to do this I plan to train a Neural Network (Since with those the difference in speaking pace apperently doesn't need to be controlled for) and preprocess my audio with mel frequency cepstral coefficients in order to represent the features present in human speech. (I also thought about preprocessing to represent just the pitch but i am not sure if I might take away too many features then).&#xA;As mentioned earlier as an output I want the representation of where in the word the changes in hight occur.&#xA;My training data consists of 50000 recordings of various pitch-accent-patterns over words, the recordings themselves are labeled with the pattern and the word they represent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this seem doable with a Neural Network or machine learning in general?&#xA;How do I get the Syllable Count to be weighted so much that the output is directly related to the high and low of each syllable?&#xA;Do you have any ideas/tips/suggestions in general?&lt;/p&gt;&#xA;" OwnerUserId="51837" LastActivityDate="2018-05-11T15:55:38.513" Title="Is machine learning a viable tool to map accent from speech onto text/syllables?" Tags="|machine-learning|python|neural-network|feature-selection|audio-recognition|" AnswerCount="0" CommentCount="5" ClosedDate="2018-05-14T02:31:41.227" ContentLicense="CC BY-SA 4.0" />
  <row Id="31532" PostTypeId="2" ParentId="27420" CreationDate="2018-05-11T15:59:23.383" Score="1" Body="&lt;p&gt;since the problem is to combine several runs different clustering algorithms to get a common partition of the original dataset, aiming for consolidation of results from a portfolio of individual clustering results. the solution is simple for there is no correct way to answer it we formally define the CDC problem as an optimization problem from the viewpoint of CE, and apply CE approach for clustering categorical data. Experimental results on real datasets show that CE based clustering method is competitive with existing CDC algorithms with respect to clustering accuracy.&lt;/p&gt;&#xA;" OwnerUserId="51989" LastActivityDate="2018-05-11T15:59:23.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31533" PostTypeId="2" ParentId="26099" CreationDate="2018-05-11T16:39:55.023" Score="1" Body="&lt;p&gt;The &quot;parts-based&quot; is explicitly stated in the text to be a synonym to &quot;non-subtractive&quot;. It is a natural feature of NMF. Its underlying assumption is that all elements of a source matrix are non-negative, as well as that the source matrix is a product of two other matrices, all elements of which are non-negative. Thus the source matrix is a sum of independent &quot;parts&quot;. If the parts were allowed to be negative, the problem would lose capacity for unique solutions, since then, for example one could generate out of thin air fake components that are equal and opposite.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;local&quot; refers to spatial locality. Especially in image processing, frequently each component is expected to be concentrated within a small region, such that all related pixels are not too far from each other. This is not achieved by naive implementation of NMF, which results in components &quot;smeared&quot; across the entire domain, being obviously non-local.&lt;/p&gt;&#xA;" OwnerUserId="51993" LastActivityDate="2018-05-11T16:39:55.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31534" PostTypeId="1" CreationDate="2018-05-11T16:51:12.863" Score="4" ViewCount="277" Body="&lt;p&gt;I have a fairly simple dataset of energy consumption values generated every half hour. I want to train a model to predict the energy consumption at a particular time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/y2Lbq.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y2Lbq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I model time values?&lt;/p&gt;&#xA;" OwnerUserId="42587" LastEditorUserId="29169" LastEditDate="2021-01-14T18:54:32.950" LastActivityDate="2021-01-14T18:54:32.950" Title="Predicting household energy consumption?" Tags="|time-series|predictive-modeling|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31535" PostTypeId="1" CreationDate="2018-05-11T17:15:14.263" Score="2" ViewCount="3023" Body="&lt;p&gt;I am reading the gini index definition for decision tree:  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset. &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This seems to be the same as misclassification. Is Gini index just a fancy name for misclassification? Or is there really some subtle difference? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2018-05-12T08:20:08.967" Title="Difference between impurity and misclassificaton" Tags="|decision-trees|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31536" PostTypeId="1" CreationDate="2018-05-11T17:16:45.167" Score="5" ViewCount="649" Body="&lt;p&gt;I have a large dataset for the activities performed by multiple staff in a factory over a long period of time - 01/01/2017 - present.  The activities performed by the different staff are recorded at each point in time (since they interact with software).  I have tabulated these to record the number of activities performed by each operator for each day.&lt;/p&gt;&#xA;&lt;p&gt;My table looks something like this:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Date         Name    Activity  UnitsProcessed   Shift      Team&#xA;01/10/2017   MMouse  Soldering   1000           Shift A   Team A&#xA;01/10/2017   MMouse  Welding     3000           Shift A   Team A&#xA;01/10/2017   DDuck   Soldering   4000           Shift A   Team B&#xA;01/10/2017   DDuck   Welding     1000           Shift A   Team B&#xA;...&#xA;02/11/2017   MMouse  Soldering   1100           Shift A   Team A&#xA;02/11/2017   MMouse  Welding     2900           Shift A   Team A&#xA;02/11/2017   DDuck   Soldering   4010           Shift A   Team B&#xA;02/11/2017   DDuck   Welding     900            Shift A   Team B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Each line represents the activities performed by each operator each day they worked, and the number of units processed during each activity, along with a classifier for the shift and team that the operator was a member of (for simplicity, assume that this does not change over time).&lt;/p&gt;&#xA;&lt;p&gt;For brevity, I have omitted values from Shift B, but essentially, the team members of Shift B will (usually) work different days from Shift A, but the proportion of tasks performed will usually be similar to the corresponding Team on Shift A - Team A or Team B.&lt;/p&gt;&#xA;&lt;p&gt;The problem I want to solve is:  I have the date, activity, and units processed - as well as various other variables - for another operator, &amp;quot;GOofy&amp;quot;.  But I don't know which shift/team Mr. GOofy works.  So I am using the data for Mr. Mouse and Mr. Duck (obviously, in reality, scores of people) to see if I can try to train kNN to correctly guess the Shift and Team pattern for each Name, given the Date, Activity, and Units Processed fields, among others.&lt;/p&gt;&#xA;&lt;p&gt;My thinking is:&lt;/p&gt;&#xA;&lt;p&gt;a) There is no particular value in keeping Team and Shift separate, so I should create a joined column that combines both&lt;/p&gt;&#xA;&lt;p&gt;B) If I create separate columns for &amp;quot;Soldering&amp;quot;, &amp;quot;Welding&amp;quot; and the other activities, I can have the number of units processed for each day for each activity by date.  Given that the kind of activity performed is (I hypothesize) a strong predictor of the shift/team combination, I should be able then to train to recognize shift/team based on both the date of that line (perhaps converted to a numeric date) and the relative values for each line of soldering, welding, etc.&lt;/p&gt;&#xA;&lt;p&gt;The problem is that, when extended to the test set, this will only give me predictors for each individual day.  But I suspect that the days worked are also a significant predictor, so I would like to include this as well.  But I'm not sure how to approach this.&lt;/p&gt;&#xA;&lt;p&gt;The two approaches I can think of are:&lt;/p&gt;&#xA;&lt;p&gt;Option 1:  To use kNN to generate estimators for shift/team for each operator/day combination, with the x variables being the date, units soldered and units welded per day, and then calculate, for each operator, the proportion of times &amp;quot;Shift A, Team A&amp;quot;, &amp;quot;Shift A, Team B&amp;quot;, etc were assigned to that operator - so if DDuck is assigned &amp;quot;Shift A, Team A&amp;quot;, on 90% of days, it seems most likely that DDuck is indeed &amp;quot;Shift A, Team A&amp;quot;.&lt;/p&gt;&#xA;&lt;p&gt;Option 2:  To somehow create a variable that represents all of the days worked for each operator - although I'm not sure how to do this - and thus create a  set that contains one row for each operator.&lt;/p&gt;&#xA;&lt;p&gt;Is either of these approaches valid?  Is there a widely-used alternative approach for this?&lt;/p&gt;&#xA;&lt;p&gt;Note that this is not a duplicate of previous questions relating to kNN and time-series data, since those were concerned with forecasting, whereas I am trying to analyze retrospective data.&lt;/p&gt;&#xA;" OwnerUserId="51988" LastEditorUserId="97640" LastEditDate="2021-06-20T19:21:56.423" LastActivityDate="2023-12-20T18:07:05.863" Title="k-Nearest Neighbours with time series data - how to obtain whole-time-period estimators" Tags="|r|time-series|k-nn|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31539" PostTypeId="2" ParentId="31534" CreationDate="2018-05-11T17:27:30.007" Score="1" Body="&lt;p&gt;At first sight, the total acumulated energy consumption seems to have a linear relation with time, so I suggest to try a &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;linear regression&lt;/strong&gt;&lt;/a&gt; at first. There are several libraries you can use to code it. I recommend you do it with &lt;a href=&quot;https://pandas.pydata.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt;, here is an answer related to this: &lt;a href=&quot;https://stackoverflow.com/questions/29934083/linear-regression-on-pandas-dataframe-using-sci-kit-learn&quot;&gt;answer&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the relation is not linear, so I could try with a more complex model (but I suggest to keep simplicity at first). Since you are trying to predict a temporal serie, I would try with an &lt;a href=&quot;https://en.wikipedia.org/wiki/Long_short-term_memory&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;LSTM&lt;/strong&gt;&lt;/a&gt; model. &lt;a href=&quot;http://adventuresinmachinelearning.com/keras-lstm-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a tutorial to implement an LSTM neural network with &lt;a href=&quot;https://keras.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;keras&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-05-11T17:34:30.130" LastActivityDate="2018-05-11T17:34:30.130" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="31540" PostTypeId="1" CreationDate="2018-05-11T17:43:57.880" Score="1" ViewCount="31" Body="&lt;p&gt;I'm confused,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it the job of the modelling approach to consider or not consider independence/dependence of features? Or should it be known before-hand?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I know nothing about independence/dependence, but my data could have either, then how should this affect the modelling approach that I take?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-11T17:43:57.880" Title="Is it the job of the modelling approach to consider or not consider independence/dependence of features? Or should it be known before-hand?" Tags="|regression|correlation|" AnswerCount="0" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31541" PostTypeId="1" CreationDate="2018-05-11T17:51:04.967" Score="2" ViewCount="43" Body="&lt;p&gt;How to evaluate the &quot;betterness&quot; of competitive good models?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say I could get good models (&gt; 90% prediction rate) with e.g.:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;LinearSVC&lt;/li&gt;&#xA;&lt;li&gt;F-test based sklearn.feature_selection.f_regression&lt;/li&gt;&#xA;&lt;li&gt;mutual info -based sklearn.feature_selection.mutual_info_regression&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;But since these treat e.g. the independence/dependence of features differently, particularly e.g. LinearSVC assumes &quot;relatively independent&quot; features, where as mutual info particularly measure dependence between variables, then&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I compare these models to each other?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tests? Knowledge of the data a priori? Something else?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-12-08T22:00:08.580" Title="How to evaluate the &quot;betterness&quot; of competitive good models?" Tags="|model-selection|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31542" PostTypeId="2" ParentId="31129" CreationDate="2018-05-11T17:59:28.863" Score="9" Body="&lt;p&gt;I'm not sure I completely understand what you want but it looks like you're trying to find an implementation for &lt;code&gt;sample_weight&lt;/code&gt;. Well, I had tried something similar. Before I go into it, I want to mention that my error actually went up - I haven't had the time to go into why that happened. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For simplicity, lets say you know the weights you need for each class and can pass it as a dictionary. In my example I have my &lt;code&gt;y_train&lt;/code&gt; as a one hot encoded vector. I'm using that fact to reverse engineer which class each row is pointing to and adding a weight for it. You essentially need to pass an array of weights mapping to each label (so the &lt;strong&gt;same length as your training data&lt;/strong&gt;) when you fit the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def generate_sample_weights(training_data, class_weight_dictionary): &#xA;    sample_weights = [class_weight_dictionary[np.where(one_hot_row==1)[0][0]] for one_hot_row in training_data]&#xA;    return np.asarray(sample_weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is passed when you call &lt;code&gt;fit&lt;/code&gt; on the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(x=X_train, &#xA;    y=y_train, &#xA;    batch_size = 64,&#xA;    validation_data=(X_val, y_val),&#xA;    shuffle=True,&#xA;    epochs=20,&#xA;    sample_weight = generate_sample_weights(y_train, class_weights_dict)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="8206" LastActivityDate="2018-05-11T17:59:28.863" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31543" PostTypeId="1" CreationDate="2018-05-11T18:01:32.907" Score="1" ViewCount="655" Body="&lt;p&gt;I have three different inputs I would like to send into my LSTM - the sequence of &quot;words&quot;, extra temporal information, as well as extra nontemporal information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Following Adam Sypniewski excellent answer to this question (&lt;a href=&quot;https://datascience.stackexchange.com/questions/17099/adding-features-to-time-series-model-lstm&quot;&gt;Adding Features To Time Series Model LSTM&lt;/a&gt;) I came with this architecture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word_input = Input(shape=(sequence_length,))&#xA;temporal_metadata = Input(shape=(sequence_length, 1))&#xA;nontemporal_metadata = Input(shape=(nontemporal_num_features,))&#xA;&#xA;embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)&#xA;temporal_info = concatenate(inputs=[embedding, temporal_metadata])&#xA;rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)&#xA;all_info = concatenate(inputs=[rnn, nontemporal_metadata])&#xA;dense_1 = Dense(10, activation='relu')(all_info)&#xA;dropout_1 = Dropout(.5)(dense_1)&#xA;dense_2 = Dense(1, activation='sigmoid')(dropout_1)&#xA;&#xA;model = Model([word_input, temporal_metadata, nontemporal_metadata], dense_2)&#xA;model.compile(loss='binary_crossentropy', optimizer='adam'&#xA;model.fit(x=[my_words, my_temporal, my_nontemporal], y = my_y, &#xA;     epochs = 30, &#xA;     validation =([my_words_test, my_temporal_test, my_nontemporal_test], y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This does not learn, and gets an AUC of 0.5&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if I don't add the nontemporal data and change the above code to&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;word_input = Input(shape=(sequence_length,))&#xA;temporal_metadata = Input(shape=(sequence_length, 1))&#xA;&#xA;embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)&#xA;temporal_info = concatenate(inputs=[embedding, temporal_metadata])&#xA;rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)&#xA;dense_1 = Dense(10, activation='relu')(rnn)&#xA;dropout_1 = Dropout(.5)(dense_1)&#xA;dense_2 = Dense(1, activation='sigmoid')(dropout_1)&#xA;&#xA;model = Model([word_input, temporal_metadata], dense_2)&#xA;model.compile(loss='binary_crossentropy', optimizer='adam'&#xA;model.fit(x=[my_words, my_temporal], y = my_y, &#xA;     epochs = 30, &#xA;     validation =([my_words_test, my_temporal_test], y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It works great. Can anyone please help me find what is going wrong in the first code sample that would make the model unable to learn. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="51996" LastActivityDate="2018-05-11T21:28:02.383" Title="LSTM not learning with extra nontemporal data added after LSTM layer - Keras" Tags="|deep-learning|nlp|keras|rnn|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31544" PostTypeId="2" ParentId="31541" CreationDate="2018-05-11T18:22:51.907" Score="1" Body="&lt;p&gt;The typical approach would be to compare (cross-)validation performance of the models, at least if accuracy is the selection criterion. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other criteria could be&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;simplicity/ease to interpret &lt;/li&gt;&#xA;&lt;li&gt;how easy the models could be implemented and/or updated in a productive environment&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="36977" LastActivityDate="2018-05-11T18:22:51.907" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31545" PostTypeId="1" AcceptedAnswerId="31654" CreationDate="2018-05-11T18:23:22.447" Score="3" ViewCount="245" Body="&lt;p&gt;I have extracted &lt;strong&gt;~550 video scripts (subtitles)&lt;/strong&gt; from 11 free courses on the Coursera platform. I have pre-processed them in terms of &lt;strong&gt;punctuation removal, stop words removal, tokenization, stemming and lemmatization&lt;/strong&gt;. Now, I've been advised that for my task I can attempt to use a simple &lt;strong&gt;Bag of Words&lt;/strong&gt;. However I am not sure how exactly would that help me towards classifying my text into &lt;strong&gt;one out of six categories&lt;/strong&gt;. The categories are related to the intent a video material was created with and more precisely, which part explains a concept, which part discusses an example, which part gives practical advice etc. Below are my categories:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;ConceptDescription&lt;/code&gt;-&gt; Explanation of the Main concept(s)&lt;br&gt;&#xA;&lt;code&gt;ConceptMention&lt;/code&gt;-&gt; Mentioning of a concept, related to the main concept&lt;br&gt;&#xA;&lt;code&gt;Methodology / Technique&lt;/code&gt;-&gt; To achieve something, what should one do&lt;br&gt;&#xA;&lt;code&gt;Summary&lt;/code&gt;-&gt; Summary of the discussed material or of the whole course&lt;br&gt;&#xA;&lt;code&gt;Application&lt;/code&gt;-&gt; Practical advise for the concept&lt;br&gt;&#xA;&lt;code&gt;Example&lt;/code&gt;-&gt; Concept example&lt;br&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By manually reading several files from 3 of the courses, I created a dictionary, containing spoken language words, that may help me identify which class a specific sentence/paragraph falls into. However &lt;strong&gt;I do NOT have a train dataset&lt;/strong&gt; for a classifier. So my idea was to use that dictionary to label my data, e.g. sentence 1 as &lt;code&gt;Summary&lt;/code&gt;, sentence 4 as &lt;code&gt;ConceptDescription&lt;/code&gt; and sentence 12 as &lt;code&gt;Example&lt;/code&gt; and then marking sentences 2 and 3 the same as 1, sentence 5-11 like sentence 4 etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, is this idea too lame? And &lt;strong&gt;is there a way to create at least an average quality training dataset in a way that is not manual?&lt;/strong&gt; Or &lt;strong&gt;if manual check is the only option, is there an option where I would need to do manual labeling on only a small fraction of the files&lt;/strong&gt;, say 50 out of 550 and classification would still produce average to good result? I don't aim at perfect result, but I aim at something less time-consuming due to limited time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also played with tf-idf which outputs terms, but of course, not really what I need, so that was a bit random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for your help. &lt;strong&gt;Any specific ideas and algorithms would be very welcome.&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="50625" LastActivityDate="2018-05-14T18:56:47.197" Title="Build train data set for natural language text classification?" Tags="|python|classification|nlp|text-mining|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31546" PostTypeId="1" CreationDate="2018-05-11T18:51:24.343" Score="1" ViewCount="82" Body="&lt;p&gt;I am very intrigued and I decided to do a little project in my free time to get a hold of it.The goal of my project is to make a NN that can play a turn-based game.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have searched for a bit about this subject and all I could find was NNs that use images of the game to make decisions, but what if I already have the game's data directly? What type of NN would work best in this situation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am hoping you can point me a bit to the right direction if possible.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="51999" LastActivityDate="2018-05-11T23:05:15.397" Title="What type of NN should be used for a turn-based game (with direct access to data)?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ClosedDate="2018-05-14T02:36:01.740" ContentLicense="CC BY-SA 4.0" />
  <row Id="31547" PostTypeId="1" CreationDate="2018-05-11T19:12:07.243" Score="1" ViewCount="129" Body="&lt;p&gt;How do different models take in account independence/dependence of features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've only found very rough &quot;pointers&quot; as to what assumptions different models make about the independence/dependence of features, but I've not found nothing holistic yet.&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-11T20:53:07.527" Title="How do different models take in account independence/dependence of features?" Tags="|correlation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31548" PostTypeId="1" CreationDate="2018-05-11T19:13:47.380" Score="3" ViewCount="10601" Body="&lt;p&gt;I'm doing a Kaggle challenge, and a lot of entries in the data are NA. However, according to the data description, this doesn't actually mean &quot;missing data&quot;, it means something like &quot;Not applicable&quot;, in the sense of it just not having that quality (for example, not having a basement).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, when I use pandas to import the data using read_csv(), and then use head() to look at it, it shows NaN for all those things that should be NA (comparing with the spreadsheet in LibreOffice).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know how to just replace one value with another for a given column, but there's still a problem. NaN is usually what it should enter if there's just a missing value. So, if it's importing both &quot;NA&quot; and a blank cell as NaN, I won't know which it is. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's probably not a huge issue, because I'll probably want to replace actual missing values with something common anyway, but it's good to know. I know I could also go into the csv with regular python and rename those actual &quot;NA&quot;s, but that's cumbersome. Is there a way I can just import certain columns as strings?&lt;/p&gt;&#xA;" OwnerUserId="50805" LastActivityDate="2018-05-13T00:17:09.090" Title="Is there a way in pandas to import NA fields as a string rather than NaN?" Tags="|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31549" PostTypeId="2" ParentId="31548" CreationDate="2018-05-11T19:48:58.450" Score="2" Body="&lt;p&gt;I think you can do something like this which might help in terms of specifying the data type.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = pandas.read_csv('foo.csv', dtype={'ColumnName': 'S10'})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, i think it is a wiser idea to &quot;Find and Replace&quot; all such values to something like blank cell or &quot;Not Available&quot; and then parse the file as this kind of data cleaning is better done before importing rather than after importing and processing&lt;/p&gt;&#xA;" OwnerUserId="52003" LastActivityDate="2018-05-11T19:48:58.450" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31550" PostTypeId="2" ParentId="31535" CreationDate="2018-05-11T20:03:58.703" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Is Gini index just a fancy name for misclassification?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that Gini index definition doesn't involve predicted values, and also it involves some probabilities, which are not dependent on classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in context of decision trees, Gini impurity corresponds to &lt;strong&gt;each region&lt;/strong&gt;, and is not a single value, such as missclassification rate (technically you could also count missclassification rate per region, but then you'd also ).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://github.com/lambdaofgod/stackexchange/blob/master/datascience/Difference%20between%20impurity%20and%20misclassificaton.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this notebook&lt;/a&gt; for a concrete example.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-11T20:03:58.703" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31552" PostTypeId="2" ParentId="31547" CreationDate="2018-05-11T20:53:07.527" Score="1" Body="&lt;p&gt;It depends on the different families of the learning algorithms. For example, Naive Bayes and all the semi-Naive Bayes models assume that the features are indpendent, which is not true in most real world problems. The latter is a disadvantage of Naive Bayes learning algorithm, although in some problems as in natural language process these models usually perform well. Most of the machine learning algorithms do not assume such thing (feature independence). Finally, there is no holistic rule for that. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-11T20:53:07.527" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31553" PostTypeId="2" ParentId="31543" CreationDate="2018-05-11T21:28:02.383" Score="2" Body="&lt;p&gt;if anyone stumbles upon this - it seemed like my dimension of nontemporal data was too high and sparse. After filtering only for the most common/important variables in the nontemporal data, the model learned.&lt;/p&gt;&#xA;" OwnerUserId="52009" LastActivityDate="2018-05-11T21:28:02.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31554" PostTypeId="2" ParentId="31546" CreationDate="2018-05-11T22:53:27.487" Score="2" Body="&lt;p&gt;Your question is quite broad so it is not easy to give a direct answer.  However, a simple approach and a good introduction to machine learning would be to use a &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Multilayer_perceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;Multilayer Perceptron (MLP)&lt;/a&gt;&lt;/strong&gt;.  This is one of the most basic and fundamental neural network architectures.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will also want to read about &lt;strong&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Backpropagation&quot; rel=&quot;nofollow noreferrer&quot;&gt;Backpropagation&lt;/a&gt;&lt;/strong&gt;, which is the standard method of training such algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some &lt;strong&gt;code examples/libraries&lt;/strong&gt; (in no particular order) that can get you started:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/keras-team/keras/blob/master/examples/mnist_mlp.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras - (Python)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://deeplearning.net/tutorial/mlp.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Theano - (Python)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://deeplearning4j.org/multilayerperceptron&quot; rel=&quot;nofollow noreferrer&quot;&gt;DeepLearning4J - (Java)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://caffe.berkeleyvision.org/tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Caffe  - (C++/Python/Matlab)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.tensorflow.org/tutorials/&quot; rel=&quot;nofollow noreferrer&quot;&gt;TensorFlow - (C++/Python)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://weka.sourceforge.net/doc.dev/weka/classifiers/functions/MultilayerPerceptron.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Weka - (Java)&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-05-11T23:05:15.397" LastActivityDate="2018-05-11T23:05:15.397" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31555" PostTypeId="1" CreationDate="2018-05-11T23:17:21.407" Score="4" ViewCount="279" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5xL6s.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5xL6s.png&quot; alt=&quot;Accuracy vs Epochs&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The orange curve is train accuracy and blue is validation accuracy. Is this clear overfitting or should I let it run for more epochs?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With custom dataset (1D data with 70 features) I trained a 2 layer MLP. &#xA;Network Architecture: [70-200-200-4].&#xA;I'm only able to reach ~50+% accuracy. Any suggestions on what steps I can take to improve accuracy? (Obtaining more data isn't an option)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="51748" LastEditorUserId="51748" LastEditDate="2018-05-11T23:27:23.977" LastActivityDate="2018-05-15T04:20:39.907" Title="Is this clear overfitting?" Tags="|machine-learning|neural-network|overfitting|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31557" PostTypeId="1" CreationDate="2018-05-12T01:21:22.340" Score="1" ViewCount="112" Body="&lt;p&gt;I am working on the Webspam UK 2006 dataset.I have a dataset with around 98 different features. I am planning to apply Decision Tree C4.5 classifier on it. I have done feature selection using random forest and I am able to get the MeanDecreaseGini for each feature. Now my question is out of these 98 features how do I decide the total number of features to drop?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; importance(model_rf)&#xA;       MeanDecreaseGini&#xA;hostid        16.470372&#xA;HST_1          8.243852&#xA;HST_2          7.269966&#xA;HST_3          6.867545&#xA;HST_4         12.726805&#xA;HST_5         11.496738&#xA;HST_6         16.547676&#xA;HST_7         11.990072&#xA;HST_8         12.133017&#xA;HST_9         13.215381&#xA;HST_10        15.987938&#xA;HST_11         7.452778&#xA;HST_12         6.426637&#xA;HST_13         6.989566&#xA;HST_14         7.523594&#xA;HST_15         9.180248&#xA;HST_16        17.578608&#xA;HST_17        13.813119&#xA;HST_18        12.469869&#xA;HST_19         8.205692&#xA;HST_20         9.928898&#xA;HST_21         8.204766&#xA;HST_22        10.453883&#xA;HST_23         7.250751&#xA;HST_24         7.768237&#xA;HMG_25         6.888539&#xA;HMG_26         5.202057&#xA;HMG_27         7.272760&#xA;HMG_28         9.496095&#xA;HMG_29        22.814004&#xA;HMG_30         8.385849&#xA;HMG_31         8.962740&#xA;HMG_32         8.788728&#xA;HMG_33         8.500586&#xA;HMG_34         9.605438&#xA;HMG_35         6.916501&#xA;HMG_36         5.499630&#xA;HMG_37         7.707432&#xA;HMG_38         7.170538&#xA;HMG_39         7.519015&#xA;HMG_40        20.864126&#xA;HMG_41        13.891407&#xA;HMG_42        17.097263&#xA;HMG_43         5.907863&#xA;HMG_44         8.961656&#xA;HMG_45         6.505953&#xA;HMG_46         8.198271&#xA;HMG_47         6.683629&#xA;HMG_48         7.521221&#xA;AVG_49         6.978512&#xA;AVG_50        10.625453&#xA;AVG_51        11.787007&#xA;AVG_52        10.501896&#xA;AVG_53         8.998260&#xA;AVG_54         7.169787&#xA;AVG_55        19.950379&#xA;AVG_56        10.941224&#xA;AVG_57        11.911999&#xA;AVG_58        11.145681&#xA;AVG_59        23.529763&#xA;AVG_60         8.726543&#xA;AVG_61        11.881046&#xA;AVG_62         6.889028&#xA;AVG_63        12.834768&#xA;AVG_64         6.642953&#xA;AVG_65        16.349392&#xA;AVG_66         6.885495&#xA;AVG_67        15.492990&#xA;AVG_68         7.982041&#xA;AVG_69         7.274046&#xA;AVG_70         6.883014&#xA;AVG_71         6.684649&#xA;AVG_72         7.105827&#xA;STD_73         6.440758&#xA;STD_74         6.952049&#xA;STD_75         7.192637&#xA;STD_76         8.917629&#xA;STD_77        23.578747&#xA;STD_78         9.402747&#xA;STD_79        39.137215&#xA;STD_80        11.908751&#xA;STD_81        34.713739&#xA;STD_82         9.006674&#xA;STD_83        42.645293&#xA;STD_84         9.555861&#xA;STD_85        13.003373&#xA;STD_86         6.609194&#xA;STD_87        13.641984&#xA;STD_88         6.627306&#xA;STD_89         8.565469&#xA;STD_90         6.497847&#xA;STD_91         8.985288&#xA;STD_92         6.702219&#xA;STD_93         8.187475&#xA;STD_94        10.954584&#xA;STD_95         8.111179&#xA;STD_96         8.080897&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51605" LastActivityDate="2018-05-12T01:21:22.340" Title="How to choose the number of features to select and number of features to drop?" Tags="|machine-learning|r|data-mining|feature-selection|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-15T01:40:55.300" ContentLicense="CC BY-SA 4.0" />
  <row Id="31558" PostTypeId="1" CreationDate="2018-05-12T01:47:02.247" Score="0" ViewCount="95" Body="&lt;p&gt;I would like to make predictions about how crowded a location (postal code mapped to lat/lng coords) would be. I would like to provide predictions to questions like: what’s the crowd density going to be like for the area surrounding postal code 12345 for 50metre next Monday between 6pm to 8pm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a start I’m working with historical transaction data (as a proxy for human activity) which contain time stamps and postal codes. I would also like to incorporate new transaction data into my model as they occur. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this make sense?&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Filter the historical transactions based on requested time (between 6pm to 8pm).&lt;/li&gt;&#xA;&lt;li&gt;Cluster the transactions based on the desired proximity to get a daily transaction totals for the area.&lt;/li&gt;&#xA;&lt;li&gt;Use the daily totals as training data for my regression model.&lt;/li&gt;&#xA;&lt;li&gt;Use the model to predict &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Would I need to rerun the entire process to account for new data which arrives in real time (e.g every 5mins)? Are there any incremental ML models which could be suitable?&lt;/p&gt;&#xA;" OwnerUserId="52012" LastActivityDate="2018-09-09T06:01:13.303" Title="Predicting crowd density using real time streaming data" Tags="|clustering|regression|online-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31559" PostTypeId="2" ParentId="31558" CreationDate="2018-05-12T02:46:45.360" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Are there any incremental ML models which could be suitable?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You have to be able to access the model when it's in production mode and actually be able to update it with a new version. all ML models should be adjusted when new data comes because of the so-called data shifting. you could periodically evaluate how much the model is performing.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest modeling the problem as a sequence model, use multiple output at each day to predict densities for each postal code.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-12T02:46:45.360" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31561" PostTypeId="2" ParentId="22952" CreationDate="2018-05-12T04:09:44.017" Score="2" Body="&lt;p&gt;A more pythonic (or pandorable) way to change a list of columns to different data types&lt;/p&gt;&#xA;" OwnerUserId="52017" LastActivityDate="2018-05-12T04:09:44.017" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31562" PostTypeId="2" ParentId="31529" CreationDate="2018-05-12T04:53:58.627" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The first thing comes to mind is sequence models, I think this will be useful for you, see &lt;a href=&quot;https://magenta.tensorflow.org/performance-rnn-browser&quot; rel=&quot;nofollow noreferrer&quot;&gt;Real-time Performance RNN in the Browser&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-12T04:53:58.627" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31565" PostTypeId="2" ParentId="31472" CreationDate="2018-05-12T07:39:05.727" Score="1" Body="&lt;p&gt;The book you are reading is being somewhat lax with terms. It uses the terms &quot;actor&quot; and &quot;critic&quot;, but there is another algorithm called &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2&quot; rel=&quot;nofollow noreferrer&quot;&gt;actor-critic&lt;/a&gt; which is very popular recently and is quite different from Q learning. Actor-critic does have two function estimators with the roles suggested in the quote. Q-learning has one such estimator*.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have looked at the chapter in more detail, and where it says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;one will be used to drive Ms. Pac-Man during training (the actor), and the other will watch the actor and learn from its trials and errors (the critic).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I would substitute:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;one will be used learn from current actions, and the other will remember results from some time steps ago in order to estimate the values for next actions.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is not something that's inherently part of Q-learning, but it is part of DQN's adjustments when combining Q-learning with neural networks. Both experience replay and having two copies of the learning network (one a temporarily &quot;frozen&quot; version of the other) are important for stabilising the learning algorithm. Without them it can become numerically unstable.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is this a typical Q-learning implementation? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's a typical implementation of basic DQN, which is how many people nowadays would implement Q-learning with neural networks. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can ignore the references to &quot;actor&quot; and &quot;critic&quot;. Instead it is easier to consider that there is just one &quot;action value&quot; network, and you keep an old copy of it around to help with stability.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;* Generally in RL, the term &quot;model&quot; is reserved for a model of the environment - which neither Q-learning nor actor-critic provide. So you will also read that Q-learning is a &quot;model free&quot; algorithm. For the rest of the book, you will have seen &quot;model&quot; to refer to any statistical learning algorithm (or the architecture and learned parameters) . . . what you will see in RL texts is the careful use of &quot;function estimator&quot; or other terms for networks which learn something else other than how the environment behaves.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-05-12T07:45:03.117" LastActivityDate="2018-05-12T07:45:03.117" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31566" PostTypeId="2" ParentId="31535" CreationDate="2018-05-12T08:20:08.967" Score="0" Body="&lt;p&gt;To compute misclassification rate, you should specify what the method of classification is.&lt;/p&gt;&#xA;&lt;p&gt;Gini impurity uses a random classification with the same distribution of labels as in the set. i.e., if a set had 70 positive and 30 negative examples, each example would be randomly labeled: 70% of the time as positive and 30% of the time as negative. The misclassification rate for this classifier will be:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;= Pr[Positive] * Pr[Label is Negative] + Pr[Negative] * Pr[Label is Positive]&lt;/p&gt;&#xA;&lt;p&gt;= 0.7 * 0.3 + 0.3 * 0.7 = 0.42&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;We can also compute misclassification rate using a different classifier method: a majority rule. In the above example, we would always predict positive. Misclassification rate will be:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;= Pr[Positive] * Pr[Label is Negative] + Pr[Negative] * Pr[Label is Positive]&lt;/p&gt;&#xA;&lt;p&gt;= 0.7 * 0 + 0.3 * 1 = 0.3&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;We see that Gini impurity is one specific type of misclassification rate.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-12T08:20:08.967" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31567" PostTypeId="1" CreationDate="2018-05-12T09:05:45.527" Score="2" ViewCount="440" Body="&lt;p&gt;I've stumbled upon &lt;a href=&quot;https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=WRL12345USEN&quot; rel=&quot;nofollow noreferrer&quot;&gt;this IBM text&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It states that:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Every day, we create 2.5 quintillion bytes of data&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It's very interesting statistic, but I couldn't find any source for this. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone know how they calculated this? Or maybe someone knows some papers showing similar/different results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://downloads.cloudsecurityalliance.org/initiatives/bdwg/Big_Data_Top_Ten_v1.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;This paper from CSA&lt;/a&gt; has the same statistic, but again without any source. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: The quote from IBM's text refers to data created by humanity, not by IBM alone.&lt;/p&gt;&#xA;" OwnerUserId="52026" LastEditorUserId="29575" LastEditDate="2018-05-12T14:23:52.127" LastActivityDate="2018-05-12T14:23:52.127" Title="How much data is created daily (source of IBM estimation)" Tags="|statistics|data|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31570" PostTypeId="2" ParentId="31567" CreationDate="2018-05-12T11:16:18.847" Score="1" Body="&lt;p&gt;The McKinsey Global Institute published an interesting report on the topic of Big Data.&#xA;You can find it &lt;a href=&quot;https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/Big%20data%20The%20next%20frontier%20for%20innovation/MGI_big_data_full_report.ashx&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don’t know where IBM go the numbers from, but this report is a good source.&lt;/p&gt;&#xA;" OwnerUserId="50471" LastEditorUserId="29575" LastEditDate="2018-05-12T14:22:09.457" LastActivityDate="2018-05-12T14:22:09.457" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31572" PostTypeId="1" AcceptedAnswerId="31573" CreationDate="2018-05-12T13:12:38.147" Score="1" ViewCount="333" Body="&lt;p&gt;In GAN architecture, during training, what keeps the generator's output dependant on the input noise? Why don't the weights of the noise input become zero (plus a bias)?&#xA;I would expect the generator to converge to outputting a single picture which is extremely real and non-distinguishable from a real picture, and ignore the noise all together, since this is a &quot;cheaper&quot; way (in convergence time and in number of parameters used) to decrease the generator's loss.&lt;/p&gt;&#xA;" OwnerUserId="38813" LastActivityDate="2019-02-04T15:17:52.510" Title="GAN - why doesn't the generator nullify the noise input?" Tags="|neural-network|gan|generative-models|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31573" PostTypeId="2" ParentId="31572" CreationDate="2018-05-12T15:20:49.840" Score="1" Body="&lt;p&gt;If the generator always outputs the same image, then it's easy for the discriminator to win the game and tell apart the output of the generator from random images in the training set: if the input to the discriminator is that one image, then it outputs &quot;came from the generator&quot;, otherwise outputs &quot;came from the training step&quot;.  The game is set up so that the generator is rewarded for fooling the discriminator.  Always outputting the same image isn't going to fool the discriminator.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-05-12T15:20:49.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31574" PostTypeId="1" AcceptedAnswerId="31589" CreationDate="2018-05-12T15:49:17.380" Score="1" ViewCount="851" Body="&lt;p&gt;My data is:&lt;br&gt;&#xA;userID, gameID, rating (1.0 through 10.0)&#xA;First, I normalize the values the ratings of each row.&#xA;I use cosine similarity to create a similarity matrix where each cell represents similarity between a pair of userIDs (value 0.0 through 1.0).&#xA;For each unrated gameID, I find the 10 most similar users that have rated the gameID. The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems fine for finding the best predictions, just take the top N rating values, but for actually predicting the values it doesn't perform so well. Intuitively, the average of a group of similar userID's ratings would be an accurate prediction. When each rating is made smaller by multiplying by a value that is always less than 1.0, the predicted value is consistently smaller than expected. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seemed to follow formulas that I found, but I feel like I missed something. I am using Python, Pandas, and Numpy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Edit&lt;/em&gt;&#xA;I am seeing that this final value is sometimes referred to as a &quot;weighted predicted rating&quot;. I am wondering how to &quot;de-weight&quot;.&lt;/p&gt;&#xA;" OwnerUserId="46715" LastEditorUserId="46715" LastEditDate="2018-05-12T18:43:59.593" LastActivityDate="2018-05-13T05:04:32.430" Title="Is this the correct way to apply a recommender system based on KNN and cosine similarity to predict continuous values?" Tags="|machine-learning|python|pandas|recommender-system|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31575" PostTypeId="1" CreationDate="2018-05-12T16:01:54.077" Score="1" ViewCount="142" Body="&lt;p&gt;While determining the house price I don't want to throw away geo location as feature. At least I want to see if there is correlation. Geo-location is more like categorical variable. Is there a way to make it useful feature&lt;/p&gt;&#xA;" OwnerUserId="52035" LastActivityDate="2018-05-12T21:30:44.617" Title="How can I use geo location for predictive modelling" Tags="|regression|feature-selection|linear-regression|geospatial|" AnswerCount="1" CommentCount="2" ClosedDate="2018-05-15T01:45:40.790" ContentLicense="CC BY-SA 4.0" />
  <row Id="31576" PostTypeId="1" AcceptedAnswerId="31588" CreationDate="2018-05-12T17:48:54.073" Score="1" ViewCount="117" Body="&lt;p&gt;I have a big database with 40k recors and 2 classification classes. In this big database the 76% of records belong to the first class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've used a 70-30 split partition with stratified sampling, and the K-nn gives the best accuracy on k = 20.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Is it too big value for k ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Is it possible that this big value for k is due to the disproportion of the 2 classes in the database , even if i used a stratified sampling ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-05-13T18:21:25.570" Title="Question about Knn and split validation" Tags="|classification|data-mining|dataset|k-nn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31577" PostTypeId="2" ParentId="24905" CreationDate="2018-05-12T18:03:30.110" Score="0" Body="&lt;p&gt;You can do following the two approaches.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;As others mentioned already, you can change the distribution to 50-50 in the training dataset.&lt;/li&gt;&#xA;&lt;li&gt;You can use weighted random forest algorithm for creating a balance between the two classes. In this algorithm, Random Forest itself adds weight to the two classes in order to achieve 50-50 percent weight.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52037" LastEditorUserId="29575" LastEditDate="2018-05-12T18:20:45.170" LastActivityDate="2018-05-12T18:20:45.170" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31578" PostTypeId="1" CreationDate="2018-05-12T18:03:31.053" Score="3" ViewCount="126" Body="&lt;p&gt;I'm trying to use stacking when predicting for the infamous Iris dataset. Also, I'd like to build to stacked classifier by myself which means I don't want to use mlxtend because it's too &quot;easy&quot; and out of the box. I'm using python 3 and working on windows 10.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not having any issue choosing algorithms or during cross-validation but when I try to stack the base learners ( gaussian and multinomial naive bayes, logistic regression) which gave me a string answer : the species of iris ; with the meta-learner (gradient boosting classifier), I can't seem to have a string as an answer, only a probability and it my accuracy score is much lower than any of the individual base learner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could anyone direct me at some code and/or explanations as to what I should be doing instead?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks a lot!&lt;/p&gt;&#xA;" OwnerUserId="49278" LastActivityDate="2018-05-12T18:03:31.053" Title="Stacking when the the target variable is categorical?" Tags="|machine-learning|python|supervised-learning|ensemble-learning|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31579" PostTypeId="2" ParentId="29028" CreationDate="2018-05-12T19:22:25.357" Score="1" Body="&lt;p&gt;One hot encoding is only a symptom. The cause of the problem is that your factor valible has not the same leveles in the test and train data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here you should distinct. Is it only a problem of sampling? You created your test data as (say) 20% sample of the original data. Some levels with small cardinalty could fail to get in the sample. If it is the case you must take care to sample &lt;strong&gt;all levels&lt;/strong&gt; and get 20% of data for each level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other problem is if your factor valible is not static and through the time new lavels can emerge. Here is it realy possible to encounter new levels in &quot;unseen data&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible approach to handle this is to train an explicit &lt;em&gt;unknown&lt;/em&gt; level based on some prepared average data. In the preprocessing phase all new levels are recognised and mapped to this &lt;em&gt;unknown&lt;/em&gt; level.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Periodlcally refresh the model to include the recent appeared levels.&lt;/p&gt;&#xA;" OwnerUserId="10620" LastActivityDate="2018-05-12T19:22:25.357" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31580" PostTypeId="1" CreationDate="2018-05-12T19:54:08.427" Score="0" ViewCount="93" Body="&lt;p&gt;In a dataset of &lt;strong&gt;longitude&lt;/strong&gt;, &lt;strong&gt;latitude&lt;/strong&gt; and &lt;strong&gt;price&lt;/strong&gt; (of houses) I'm using sklearn's KNearestRegressor to get the 5 nearest neighbors &lt;strong&gt;mean price&lt;/strong&gt; for each point. The problem is I want to do this for the whole dataset and each point is taking its own price into consideration since KNearestRegressor is a supervised algorithm, and I'm predicting the training set. How can I get the 5 nearest neighbors mean price for each point, not taking into consideration that point's price?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried sklearn's unsupervised NearestNeighbours but it only outputs the indexes of the nearest neighbors, not the mean price.&lt;/p&gt;&#xA;" OwnerUserId="9988" LastActivityDate="2018-05-14T16:11:17.310" Title="How to get nearest 5 points mean with Nearest Neighbours?" Tags="|scikit-learn|k-nn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31581" PostTypeId="1" CreationDate="2018-05-12T20:13:10.043" Score="2" ViewCount="697" Body="&lt;p&gt;I have a fine-tuned network that I created which uses vgg16 as it's base.  I am following section 5.4.2 Visualizing CovNet Filters in &lt;a href=&quot;https://www.manning.com/books/deep-learning-with-python&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Learning With Python&lt;/a&gt; (which is very similar to the guide on the Keras blog to visualize convnet filters &lt;a href=&quot;https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The guide simply uses the vgg16 network.  My fine tuned model uses the vgg16 model as the base, for example: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model.summary()&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Layer (type)                 Output Shape              Param #&lt;br&gt;&#xA;======================================================================= vgg16 (Model)                (None, 4, 4, 512)         14714688&lt;br&gt;&#xA;________________________________________________________________________ flatten_1 (Flatten)          (None, 8192)              0&lt;br&gt;&#xA;________________________________________________________________________ dense_7 (Dense)              (None, 256)               2097408&lt;br&gt;&#xA;________________________________________________________________________ dense_8 (Dense)              (None, 3)                 771&lt;br&gt;&#xA;======================================================================== Total params: 16,812,867 Trainable params: 16,812,867 Non-trainable params: 0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm running into an issue when I run this line: &#xA;&lt;code&gt;grads = K.gradients(loss, model.input)[0]&lt;/code&gt; where when I use my fine tuned network I get a result that's a &quot;NoneType&quot; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code from the guide: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; from keras.applications import VGG16 from keras import backend as K&#xA;&amp;gt; &#xA;&amp;gt; model = VGG16(weights='imagenet',&#xA;&amp;gt;               include_top=False)&#xA;&amp;gt; &#xA;&amp;gt; layer_name = 'block3_conv1' filter_index = 0&#xA;&amp;gt; &#xA;&amp;gt; layer_output = model.get_layer(layer_name).output loss =&#xA;&amp;gt; K.mean(layer_output[:, :, :, filter_index])&#xA;&amp;gt; &#xA;&amp;gt; grads = K.gradients(loss, model.input)[0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To reproduce the on my fine tuned model, I've used the &lt;strong&gt;exact&lt;/strong&gt; same code, &lt;strong&gt;except&lt;/strong&gt; I obviously changed the model that I imported: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;model = keras.models.load_model(trained_models_dir + 'fine_tuned_model.h5')&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;...and I also had to index into the nested Model object (my first layer is a Model object as is shown above) to get the 'block2_con1' layer: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;my_Model_object = 'vgg16'&#xA;layer_name = 'block3_conv1'&#xA;filter_index = 0&#xA;&#xA;layer_output = &#xA;model.get_layer(my_Model_object).get_layer(layer_name).output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;any idea why running &lt;code&gt;grads = K.gradients(loss, model.input)[0]&lt;/code&gt; on my fine tuned network would result in a &quot;NoneType&quot;??  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="10548" LastEditorUserId="29169" LastEditDate="2021-02-13T00:57:53.780" LastActivityDate="2021-02-13T00:57:53.780" Title="Visualizing ConvNet filters using my own fine-tuned network resulting in a &quot;NoneType&quot; when running: K.gradients(loss, model.input)[0]" Tags="|keras|tensorflow|cnn|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31582" PostTypeId="2" ParentId="31581" CreationDate="2018-05-12T20:37:45.283" Score="1" Body="&lt;p&gt;&lt;strong&gt;SOLVED:&lt;/strong&gt;  I had to use: &lt;code&gt;grads = K.gradients(loss, model.get_layer(my_Model_object).get_layer('input_1').input)[0]&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;instead of just &lt;code&gt;grads = K.gradients(loss, model.input)[0]&lt;/code&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which is confusing because both &lt;code&gt;model.get_layer(my_Model_object).get_layer('input_1').input)[0]&lt;/code&gt; &#xA;and &#xA;&lt;code&gt;model.input[0]&lt;/code&gt; print the same thing and are of the same type. &lt;/p&gt;&#xA;" OwnerUserId="10548" LastActivityDate="2018-05-12T20:37:45.283" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31583" PostTypeId="2" ParentId="31575" CreationDate="2018-05-12T21:30:44.617" Score="0" Body="&lt;p&gt;Why is it categorical? You can use the longitude / latitude numeric coordinates as two features.&lt;/p&gt;&#xA;" OwnerUserId="50765" LastActivityDate="2018-05-12T21:30:44.617" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31584" PostTypeId="1" CreationDate="2018-05-12T22:30:11.453" Score="4" ViewCount="1308" Body="&lt;p&gt;If I want to build a named entity linking system for resumes using an ontology of occupations and skills about how many annotations would I need? The ontology has about 20,000 entities. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a lower bound I'm guessing I would need about 10 examples per entity and maybe 3 different annotators to label each mention so ~600K annotations. Does that make sense?&lt;/p&gt;&#xA;" OwnerUserId="10654" LastActivityDate="2018-05-18T15:46:25.687" Title="How many examples needed for named entity disambiguation?" Tags="|nlp|named-entity-recognition|" AnswerCount="2" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31585" PostTypeId="1" CreationDate="2018-05-12T23:57:40.347" Score="2" ViewCount="60" Body="&lt;p&gt;I am currently trying to optimize the learning rate of a neural network built in tensorflow. The network has 3 hidden layers, with 500, 250 and 100 neurons respectively.&lt;/p&gt;&#xA;&lt;p&gt;I have adapted code from an example provided by the author of hyper-engine, a python package for hyperparameter optimization that can be used with tensorflow.&lt;/p&gt;&#xA;&lt;p&gt;The example may be found &lt;a href=&quot;https://github.com/maxim5/hyper-engine/blob/master/hyperengine/examples/1_2_getting_started_with_tuning.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Essentially all that has been altered is the definition of the model provided for hyperparameter optimization. The range over which the learning rate will be evaluated has also been narrowed by an order of magnitude. The current search space is from 0.1 to 0.01.&lt;/p&gt;&#xA;&lt;p&gt;The output from this python script is an xjson file. The xjson consists of multiple entries of a single key value pair, {'points' : 'values'}. 'values' quite clearly represents the accuracy of the classifier on the test data. The key, 'points' seems like it should represent the learning rate corresponding to the respective accuracy for that key value pair. However, the values taken on by points range from 0.000002 - 1.&lt;/p&gt;&#xA;&lt;p&gt;For this reason, I'm finding quite difficult to interpret the meaning of 'points'. I also failed to find any information about interpretation of the output on the github for this package.&lt;/p&gt;&#xA;&lt;p&gt;If anybody could help me understand the output from this code, it would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="52048" LastEditorUserId="1330" LastEditDate="2021-08-14T14:40:00.720" LastActivityDate="2021-08-14T14:40:00.720" Title="Interpreting hyper-engine output for hyperparameter optimization of a neural network?" Tags="|machine-learning|python|neural-network|tensorflow|hyperparameter-tuning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31586" PostTypeId="2" ParentId="31548" CreationDate="2018-05-13T00:17:09.090" Score="5" Body="&lt;p&gt;Use the pandas.read_csv() options:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d = pandas.read_csv('foo.csv', keep_default_na=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;na_values : scalar, str, list-like, or dict, default None&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Additional strings to recognize as NA/NaN. If dict passed, specific &gt;per-&gt;column NA values. By default the following values are interpreted as &gt;NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, &gt;‘1.#IND’, ‘1.#QNAN’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;keep_default_na : bool, default True&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;If na_values are specified and keep_default_na is False the default NaN &gt;values are overridden, otherwise they’re appended to.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;from &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29180" LastActivityDate="2018-05-13T00:17:09.090" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31587" PostTypeId="2" ParentId="31580" CreationDate="2018-05-13T01:58:10.617" Score="1" Body="&lt;p&gt;First of all you train/ fit a model on the training set, may be cross validate using a training set. But it is not meant to predict itself. That's what test set is for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can achieve your objective by predicting each observation in a Leave One Out manner (also used for cross validation):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For every observation (i) in your dataset, fit a KNN regressor using all data except (i), evaluate your prediction for (i). &lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-14T16:11:17.310" LastActivityDate="2018-05-14T16:11:17.310" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31588" PostTypeId="2" ParentId="31576" CreationDate="2018-05-13T03:36:13.943" Score="1" Body="&lt;p&gt;There is generally a trade of in k-NN:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;k should be large enough to cancel out any noise.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;k shouldn't be too large to create big biased boundaries.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;One rule of thumb is to select an odd k value to avoid ties in binary class problems.&#xA;Small and big value of k is subjective to the underlying structure of the data set itself, that's why we check over a range of k values. &#xA;Both of the above gets settled if done a good job with cross validation.&#xA;So the most important question you need to focus is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Have you chosen the right metric for model selection, keeping the imbalance of classes into consideration?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would recommend you to first identify the class that is more important for you to predict correctly. Then, look at the confusion matrix and ponder upon different metrics (eg. recall, precision etc.) before finalizing a model/ k value.&lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-13T18:21:25.570" LastActivityDate="2018-05-13T18:21:25.570" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31589" PostTypeId="2" ParentId="31574" CreationDate="2018-05-13T05:04:32.430" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You want to obtain a weighted-average of ratings, where the weight is the similarity score. Instead of dividing by 10 above, you should divide by the sum of similarities, to get a correct normalized score. Dividing by 10 is the likely reason for the predictions to be smaller than expected.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The formula you should use is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Final score = Sum_over_neighbours(Rating * Similarity) / Sum_over_neighbours(Similarity)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-13T05:04:32.430" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31590" PostTypeId="1" CreationDate="2018-05-13T07:37:47.743" Score="1" ViewCount="30" Body="&lt;p&gt;I need to learn which item will be (most probably) picked from a group. The data I have consists of groups by 10 items and which one was picked(or if neither was picked).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current approach is to ignore the groups and to use flat list with all items classified as picked or not picked and use some classification algorithm, but I wanted to know if there is a way to use the knowledge that the item was picked from that specific group.&lt;/p&gt;&#xA;" OwnerUserId="37967" LastActivityDate="2018-05-13T09:11:41.257" Title="Correct learning algorithm for grouped examples" Tags="|machine-learning|classification|algorithms|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31591" PostTypeId="1" CreationDate="2018-05-13T07:47:19.007" Score="2" ViewCount="72" Body="&lt;p&gt;I need a network with the following organization:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;the input text, it is constant for training, for example &quot;mama soap frame sponge&quot; or the formulas of physics, math, chemistry and others&lt;/li&gt;&#xA;&lt;li&gt;a series of questions (who, where, when, what, whom, what, for what), for example &quot;who washed the frame?&quot;&lt;/li&gt;&#xA;&lt;li&gt;A series of answers (labels), for example, the answer &quot;Mom&quot;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;How to do it with TensorFlow?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you&lt;/p&gt;&#xA;" OwnerUserId="50152" LastActivityDate="2018-05-13T07:47:19.007" Title="NLP: how to organize this research into Tensorflow" Tags="|nlp|tensorflow|" AnswerCount="0" CommentCount="5" ClosedDate="2018-05-15T01:45:04.333" ContentLicense="CC BY-SA 4.0" />
  <row Id="31593" PostTypeId="2" ParentId="31590" CreationDate="2018-05-13T09:11:41.257" Score="1" Body="&lt;p&gt;Let $M$ be the size of each group. You want to learn the function $f$ that takes $M$ vectors $v_{i}$, $i \in [1,M]$  and yields a vector $\hat{v}$ that can be 0 or one of the $v_{i}$.&#xA;You can train a DNN to learn $f$ by organizing your training data as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Input              Label&#xA;[batchSize, M, 1]  [batchSize,1]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Optimize on the reduced RMS error between the predicted vectors and the label vectors. The group information will then be learned by your DNN.&lt;/p&gt;&#xA;" OwnerUserId="31106" LastActivityDate="2018-05-13T09:11:41.257" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31594" PostTypeId="1" CreationDate="2018-05-13T10:49:47.407" Score="1" ViewCount="53" Body="&lt;p&gt;Having preprocessed data using weka supervised cfssubseteval forward and backward selection, the decision tree model builds and makes prediction well, but naive Bayes do not recognize the same Arff file of preprocessed data with 11 nominal attributes and 74 instances&lt;/p&gt;&#xA;" OwnerUserId="49003" LastActivityDate="2018-05-13T10:49:47.407" Title="Processed data cannot build Naive Bayes Classifier model" Tags="|naive-bayes-classifier|java|weka|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31595" PostTypeId="1" CreationDate="2018-05-13T12:18:37.447" Score="1" ViewCount="54" Body="&lt;p&gt;I try to understand my time series data. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. So I have a vector x with length 10.000. This I repeat with certain time intervals. Since the vectors are big, I want to create subsequences of them for my model later to predict. But now I do not understand if I have to see the whole time series data with a shape (number of samples, vector x length) or (number of samples*vector x length, 1) to move on. &lt;/p&gt;&#xA;" OwnerUserId="52068" LastEditorUserId="52068" LastEditDate="2018-05-14T16:57:36.780" LastActivityDate="2018-05-14T16:57:36.780" Title="Features for Neural Networks" Tags="|neural-network|time-series|data|" AnswerCount="0" CommentCount="5" ClosedDate="2018-05-15T01:48:31.300" ContentLicense="CC BY-SA 4.0" />
  <row Id="31596" PostTypeId="1" AcceptedAnswerId="31624" CreationDate="2018-05-13T13:58:58.903" Score="2" ViewCount="165" Body="&lt;p&gt;Is there a way to determine the number of forward and backward passes in the training of a neural network using python? &lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="29169" LastEditDate="2021-02-13T00:57:37.697" LastActivityDate="2021-02-13T00:57:37.697" Title="How to determine the number of forward and backward passes in deep learning (CNN)?" Tags="|deep-learning|training|convolution|convolutional-neural-network|epochs|" AnswerCount="1" CommentCount="0" ClosedDate="2018-05-15T01:46:21.940" ContentLicense="CC BY-SA 4.0" />
  <row Id="31597" PostTypeId="1" CreationDate="2018-05-13T14:48:04.617" Score="2" ViewCount="3908" Body="&lt;p&gt;Is there any set of machine learning algorithms that do not require training and directly gives answers if a set of labeled and unlabeled data gave at once?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is skipping explicit training of model can be skipped either in supervised or unsupervised machine learning algorithms? [I do not talk of using pre-trained model]&lt;/p&gt;&#xA;" OwnerUserId="29033" LastActivityDate="2020-05-27T16:14:45.680" Title="Machine learning without explicit training" Tags="|machine-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31599" PostTypeId="1" CreationDate="2018-05-13T16:25:19.313" Score="1" ViewCount="35" Body="&lt;p&gt;I am new to Machine learning. While reading SparkMLLib java code, I found Binary_classification &lt;a href=&quot;https://github.com/apache/spark/blob/master/data/mllib/sample_binary_classification_data.txt&quot; rel=&quot;nofollow noreferrer&quot;&gt;dataset&lt;/a&gt;. But I am not able to understand how this data is modeled and if I want to model same type of data, what I have to do?&lt;/p&gt;&#xA;" OwnerUserId="52077" LastEditorUserId="1330" LastEditDate="2021-09-22T16:32:17.690" LastActivityDate="2021-09-22T16:32:17.690" Title="modeling binary classification data" Tags="|machine-learning|machine-learning-model|apache-spark|java|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31600" PostTypeId="1" CreationDate="2018-05-13T16:34:49.520" Score="1" ViewCount="16" Body="&lt;p&gt;How to create a model that will generate a similar pattern/image?&#xA;And how sould I address it, if I only have 400 exmples after the augmentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;This is the kind of data I have (16X16)&lt;/strong&gt;&lt;br&gt;&#xA;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,1,5,2,2,1,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,1,2,3,3,1,0,1,1,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,1,2,3,2,1,0,1,7,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,1,1,1,2,1,1,1,7,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,1,1,2,2,2,2,7,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,1,2,2,2,1,2,2,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,1,2,2,2,1,1,1,1,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&lt;br&gt;&#xA;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to go for DL but I can't get my hand on an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cheers :))&lt;/p&gt;&#xA;" OwnerUserId="26525" LastActivityDate="2018-05-13T16:34:49.520" Title="Model for 'Pattern to Pattern'?" Tags="|deep-learning|scikit-learn|keras|lstm|sequential-pattern-mining|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31601" PostTypeId="1" AcceptedAnswerId="31894" CreationDate="2018-05-13T17:19:32.263" Score="0" ViewCount="139" Body="&lt;p&gt;I try to understand my time series data for my ANN for a regression problem. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. This I repeat with certain time intervals. Do I have than 10.000 different features or only one feature as input dimension for my lstm neural network ? I want to shape my data for a lstm encoder decoder. &lt;/p&gt;&#xA;" OwnerUserId="52068" LastActivityDate="2018-05-20T21:26:56.410" Title="define number of features" Tags="|keras|regression|lstm|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31602" PostTypeId="1" AcceptedAnswerId="31645" CreationDate="2018-05-13T17:57:04.537" Score="1" ViewCount="295" Body="&lt;p&gt;I'm trying to implement a simple text classifier wherein the data is split into training (70%) and testing (30%) sets, but cross validation (k=10) to be performed on the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My main concern here is the basis used for transforming a given set. I've seen tutorials where the whole data set was used to fit a Count/TFIDF vectorizer, but wouldn't this introduce bias when transforming the validation and testing sets since the previously mentioned sets were included in the whole data set? Or is the bias so small that it is acceptable to do so?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Within a fold, &lt;strong&gt;would it be better if the training set was used to fit a vectorizer and transform the validation set&lt;/strong&gt;? And for testing, &lt;strong&gt;should the training + validation sets be used to fit a vectorizer to transform the testing set&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And on this note, should the validation set also be treated as &quot;unseen&quot; data similar to the test set?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of the tutorials and notes online show ready made datasets, but there are certain cases with text data where values are dependent on a given set of documents (e.g. like how IDF is computed based on some group of documents) and that there are a lot more nitty gritty details when extracting features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think I'm just confused and am seeking a little clarification regarding this manner. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="52049" LastActivityDate="2018-05-14T13:53:49.250" Title="Fitting and transforming text data in training, testing, and validation sets" Tags="|classification|feature-extraction|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31604" PostTypeId="1" CreationDate="2018-05-13T18:37:03.593" Score="2" ViewCount="467" Body="&lt;p&gt;This question is about tuning the hyper-parameters of a LSTM-RNN on tensorflow. I would like to know about an approach to finding the best hyper-parameters for your RNN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on a dataset to predict monsoon patterns in central India.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My time-series plot looks like this.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/iI2B8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iI2B8.png&quot; alt=&quot;Time series plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My goal is to predict the value of rainfall of a day 30 days in advance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently I have the following hyper-parameters.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;num_days = 30 #number of days used to make next prediction&#xA;lstm_size=256&#xA;num_layers=2&#xA;keep_prob=0.8&#xA;batch_size = 50&#xA;init_learning_rate = 0.03&#xA;learning_rate_decay = 0.99&#xA;init_epoch = 5&#xA;max_epoch = 1000&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Based on the time-series plot what is a good choice for my given hyper-parameters?&lt;/p&gt;&#xA;" OwnerUserId="52082" LastEditorUserId="52082" LastEditDate="2018-05-13T18:45:39.763" LastActivityDate="2018-05-13T18:45:39.763" Title="Hyper-parameter search for time series LSTM-RNN using tensorflow" Tags="|tensorflow|time-series|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31606" PostTypeId="1" CreationDate="2018-05-13T18:46:55.370" Score="1" ViewCount="91" Body="&lt;p&gt;I have been allocated the OTB (open to Buy) budget at the start of the month for purchasing products in apparel category for online retail. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The inventory purchased on the budget should serve the following  objective:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Meet the demand forecast for the month&lt;/li&gt;&#xA;&lt;li&gt;Ensure a certain target sell through of the inventory&lt;/li&gt;&#xA;&lt;li&gt;Minimize the chance of mark downs of the inventory&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Problem statement :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How do I allocate the budget across brands to buy products (both old and new season)? &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example output: Allocate $10k for brand A, buy 10 styles each with quantity 20 units for new season merchandise, where as buy 15 styles with 30 units for old season merchandise. Please note that I can't allocate all the budget to the best brand. It would result in lower variety available to the customers that would lead to sales loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This has similarities to the merchandising planning problem. I have the forecast at overall level, not at style or brand level. I also have historical purchase, product attributes , margins, customer browse and transaction data. Is there any analytical framework to solve this problem ? Any help would be greatly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="52083" LastEditorUserId="52083" LastEditDate="2018-05-25T06:27:35.940" LastActivityDate="2018-05-25T06:27:35.940" Title="OTB planning for apparel category" Tags="|optimization|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-15T01:47:59.793" ContentLicense="CC BY-SA 4.0" />
  <row Id="31607" PostTypeId="1" AcceptedAnswerId="31608" CreationDate="2018-05-13T19:20:03.500" Score="1" ViewCount="152" Body="&lt;p&gt;Can I install Tensorflow in Anaconda without using Keras?&#xA;If I can what is the difference between using Keras with Tensorflow and only Tensorflow?&lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="83275" LastEditDate="2022-07-22T16:33:48.627" LastActivityDate="2022-07-22T16:33:48.627" Title="Can I install Tensorflow in Anaconda without using Keras?" Tags="|deep-learning|keras|tensorflow|cnn|convolution|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31608" PostTypeId="2" ParentId="31607" CreationDate="2018-05-13T19:47:56.117" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can I install Tensorflow in Anaconda without using Keras?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Absolutely.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I can what is the difference between using Keras with Tensorflow&#xA;  and only Tensorflow?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;While Tensorflow  is a super powerful numerical computation and optimization library with lots of features for building neural networks, YET it is a bit tedious and nontrivial especially for beginners to use. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Here Keras comes very handy&lt;/strong&gt;. In short, Keras is Tensorflow abstraction. It allows to quickly and easily define a neural net even complex ones with a few lines of codes. Also note that Keras not only runs on top of TensorFlow but also on top of Microsoft Cognitive Toolkit, Theano, or MXNet. Look at at this &lt;a href=&quot;https://www.beyondthelines.net/machine-learning/keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; where the blogger compares defining a typical neural network via directly in Tensorflow and Keras. You can see that with Tensorflow it is minimum 17 lines of code, whereas Keras reduce it to 10 lines of code.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-13T19:47:56.117" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31609" PostTypeId="1" CreationDate="2018-05-13T20:25:59.203" Score="10" ViewCount="6807" Body="&lt;p&gt;I have found mentions of two advantages in using gradients instead of actual residuals:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I don't understand these points, namely:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.&lt;/p&gt;&#xA;" OwnerUserId="52087" LastActivityDate="2023-04-03T02:55:36.667" Title="Why do we use gradients instead of residuals in Gradient Boosting?" Tags="|machine-learning|xgboost|optimization|gradient-descent|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31610" PostTypeId="1" AcceptedAnswerId="31631" CreationDate="2018-05-13T20:52:07.210" Score="1" ViewCount="259" Body="&lt;p&gt;I trained a neural network model, a MLP type of network, where the first several layers are 1-D convolution for processing sequence type of input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, the training process looks like as follows, where the orange line represents the validation loss and the blue line represents the training loss. The validation loss is large compared to the training loss and the training loss also  stops decreasing after the first several iterations. Are there any generic guidance to improve the performance? I have about 1 million training traces, and the number of parameters of the network is about 140K.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pvcWN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pvcWN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="20978" LastEditorUserId="50727" LastEditDate="2018-05-15T10:16:23.977" LastActivityDate="2018-05-15T10:16:23.977" Title="problems during training a MLP type of network" Tags="|machine-learning|neural-network|deep-learning|training|overfitting|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31611" PostTypeId="2" ParentId="22488" CreationDate="2018-05-13T21:05:18.800" Score="9" Body="&lt;p&gt;The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.&lt;/p&gt;&#xA;&lt;p&gt;First of all, you have to understand what is performed by the call&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;reframed = series_to_supervised(scaled, 1, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;in line 50.&lt;/p&gt;&#xA;&lt;p&gt;Shortly, let us assume that the input has &lt;code&gt;n&lt;/code&gt; samples of &lt;code&gt;m&lt;/code&gt; features (variables). This means that the shape of the &lt;em&gt;scaled&lt;/em&gt; dataframe is &lt;code&gt;(n) x (m)&lt;/code&gt;. The particular call to &lt;code&gt;series_to_supervised()&lt;/code&gt; in line 50 implies that the &lt;em&gt;scaled&lt;/em&gt; dataframe will be reshaped into a &lt;code&gt;(n-1) x (2m)&lt;/code&gt; dataframe, to include the current and the 1-step look-back values, after dropping the row of &lt;code&gt;NaN&lt;/code&gt; values due to the &lt;em&gt;shift&lt;/em&gt; operation inside the body of &lt;code&gt;series_to_supervised()&lt;/code&gt;. Most likely, your error is caused by line 52:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;reframed.drop(reframed.columns[[9,10,11,12,13,14,15]], axis=1, inplace=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;because you are probably dropping an arbitrary number of columns. The code is structured so that it forecasts &lt;strong&gt;one&lt;/strong&gt; time series based on the past values of &lt;strong&gt;multiple&lt;/strong&gt; time series (including the signal of interest). Therefore, you &lt;strong&gt;have to&lt;/strong&gt; drop &lt;code&gt;(m-1)&lt;/code&gt; columns, so that in the end all &lt;code&gt;m&lt;/code&gt; past values persist, along with the current value of the signal of interest that will be used as ground truth. This means that the shape of &lt;code&gt;reframed&lt;/code&gt; should be &lt;code&gt;(n-1) x (m+1)&lt;/code&gt; after line 52. If not, you probably get that error with the dimension mismatch when reverting the scaling.&lt;/p&gt;&#xA;&lt;p&gt;At this point I would like to emphasize on the fact that you should be &lt;strong&gt;very careful&lt;/strong&gt; with logical errors when dropping columns, because you want to drop &lt;strong&gt;all but&lt;/strong&gt; the signal of interest in line 52. If you keep any other column the code will still run, but you will certainly get logical artifacts.&lt;/p&gt;&#xA;&lt;p&gt;Moreover, if you drop any column other than the first one (at index 0), you have to modify the concatenation in lines 85 &amp;amp; 90 accordingly, otherwise you will also get logical artifacts.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2021-01-15T21:58:41.147" LastActivityDate="2021-01-15T21:58:41.147" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31613" PostTypeId="1" CreationDate="2018-05-14T00:39:29.990" Score="0" ViewCount="418" Body="&lt;p&gt;I am working on a traffic sign recognition code in MATLAB using Belgian Traffic Sign Dataset. The dataset consists of training data and test data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I resized the given images and extracted HOG features using the &lt;code&gt;VL_HOG&lt;/code&gt; function from VL_feat library and then trained a multi class SVM using all of the signs inside the training dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am particularly interested detecting 8 signs. &lt;strong&gt;Question 1:&lt;/strong&gt; Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Upon training the multi-class SVM, I want to test the classifier performance using the test data. &lt;strong&gt;Question 2:&lt;/strong&gt; Can someone guide me through how to do this? I found something similar to what I am looking for available &lt;a href=&quot;https://www.mathworks.com/help/vision/examples/digit-classification-using-hog-features.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="41626" LastActivityDate="2018-12-19T15:43:26.160" Title="How to quantify the performance of the classifier (multi-class SVM) using the test data?" Tags="|svm|matlab|confusion-matrix|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31614" PostTypeId="1" AcceptedAnswerId="31618" CreationDate="2018-05-14T01:15:53.603" Score="6" ViewCount="40390" Body="&lt;p&gt;Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?&lt;/p&gt;&#xA;" OwnerUserId="52096" LastActivityDate="2019-09-22T02:18:52.507" Title="Which parameters are hyper parameters in a linear regression?" Tags="|linear-regression|hyperparameter|hyperparameter-tuning|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31615" PostTypeId="2" ParentId="31614" CreationDate="2018-05-14T01:57:56.857" Score="6" Body="&lt;p&gt;&lt;em&gt;Hyper-parameters&lt;/em&gt; by definition are input parameters which are necessarily required by an algorithm to learn from data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For standard linear regression i.e &lt;em&gt;OLS&lt;/em&gt;, there is none. The number/ choice of features is not a hyperparameter, but can be viewed as a post processing or iterative tuning process. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, &lt;em&gt;Lasso&lt;/em&gt;  takes care of number/choice of features in its formulation of the loss function itself, so only hyper-parameter for it would be the shrinkage factor i.e &lt;em&gt;lambda&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-14T21:08:08.653" LastActivityDate="2018-05-14T21:08:08.653" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31617" PostTypeId="1" AcceptedAnswerId="31713" CreationDate="2018-05-14T04:23:08.600" Score="26" ViewCount="36042" Body="&lt;p&gt;I have been working on developing a system &quot;Converting Natural Language to SQL Query&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read the answers from the similar questions, but was not able to get the information that I was looking for.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below is the flowchart for such system which I have got from &lt;strong&gt;&lt;a href=&quot;http://www.iaees.org/publications/journals/selforganizology/articles/2016-3(3)/algorithm-to-transform-natural-language-into-SQL-queries.pdf&quot; rel=&quot;noreferrer&quot;&gt;An Algorithm to Transform Natural Language into SQL Queries for Relational Databases by Garima Singh, Arun Solanki&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9Ekcd.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9Ekcd.png&quot; alt=&quot;Flowchart&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have understood till part of speech tagging step. But how do I approach the remaining steps.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Do I need to train all the possible SQL queries?&lt;/li&gt;&#xA;&lt;li&gt;Or, once part of speech tagging is done, I have to play with the words and form a SQL query?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Edit: I have successfully implemented the from step &quot;user query&quot; to &quot;Part of speech tagging&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50146" LastEditorUserId="23546" LastEditDate="2019-02-21T02:39:36.953" LastActivityDate="2021-09-22T12:57:40.023" Title="Natural Language to SQL query" Tags="|machine-learning|sql|nlp|" AnswerCount="5" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31618" PostTypeId="2" ParentId="31614" CreationDate="2018-05-14T05:58:14.543" Score="9" Body="&lt;p&gt;I like the way Wikipedia generally defines it:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In machine learning, a hyperparameter is a parameter whose value is&#xA;  set before the learning process begins. By contrast, the values of&#xA;  other parameters are derived via training.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;On top of what Wikipedia says I would add:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hyperparameter&lt;/strong&gt; is a parameter that concerns the numerical optimization problem at hand. The hyperparameter won't appear in the machine learning model you build at the end. Simply put it is to control the process of defining your model. For example like in many machine learning algorithms we have learning rate in gradient descent (that need to be set before the learning process begins as Wikipedia defines it), that is a value that concerns how fast we want the gradient descent to take the next step during the optimization. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly as in &lt;strong&gt;Linear Regression&lt;/strong&gt;, hyperparameter is for instance the learning rate. If it is a &lt;strong&gt;regularized Regression&lt;/strong&gt; like LASSO or Ridge, the regularization term is the hyperparameter as well. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Number of features&lt;/strong&gt;: I would not regard &quot;Number of features&quot; as hyperparameter. You may ask yourself whether it is a parameter you can simply define during the model optimization? How you set the Number of features beforehand? To me &quot;Number of features&quot; is part of feature selection i.e. feature engineering that goes before you run your optimization! Think of image preprocessing before building a deep neural network. Whatever image preprocessing is done is never considered hyperparameter, it is rather a feature engineering step before feeding it to your model.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-05-14T07:41:03.977" LastActivityDate="2018-05-14T07:41:03.977" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31619" PostTypeId="2" ParentId="31617" CreationDate="2018-05-14T06:13:45.200" Score="6" Body="&lt;p&gt;&lt;a href=&quot;https://www.nltk.org/book/ch10.html&quot; rel=&quot;noreferrer&quot;&gt;NLTK&lt;/a&gt; has an excellent step by step guide on everything you need to convert human language to an SQL query using the nltk package in python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It’s rudimentary, but it answers your question.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-05-14T06:13:45.200" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31620" PostTypeId="1" AcceptedAnswerId="31621" CreationDate="2018-05-14T06:21:52.050" Score="1" ViewCount="90" Body="&lt;p&gt;I'm working in python. Would like to practice some machine learning, and I've always been curious about an analog to the problem below...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A collection of 3 letters are drawn randomly from the 26 letters of the alphabet. None, any, or all of the letters can be discarded and replaced with an equal number of not-yet-drawn letters. This discarding and replacing happens 3 times (or more generally n times). After all the replacements have taken place a point value is awarded to the resulting 3-letter-word. Some words have high point values, and others less so. I'm interested in the optimum replacement strategy. For example, say only the words &quot;FOX&quot; and &quot;THE&quot; reward any value. All other 3 letter combos are worthless. I want the machine to learn the correct replacement strategy while holding &quot;FOW&quot; with one replacement remaining. In this simple case, the strategy is to replace the W only, and attempt to draw the X. This strategy is superior to replacing all 3 letters in an attempt to make &quot;THE,&quot; since drawing &quot;THE&quot; only gets points 1/('count of remaining letters' choose 3) times, whereas replacing the W gets points 1/('count of remaining letters'); the instance where an X is drawn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone point me in the right direction?&lt;/p&gt;&#xA;" OwnerUserId="50175" LastActivityDate="2018-05-14T06:49:47.120" Title="Which machine learning model should I learn for this problem?" Tags="|machine-learning|python|probability|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31621" PostTypeId="2" ParentId="31620" CreationDate="2018-05-14T06:49:47.120" Score="0" Body="&lt;p&gt;You should use a &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_reward_model&quot; rel=&quot;nofollow noreferrer&quot;&gt;Markov reward model&lt;/a&gt; to model your problem. All the possible words are the different states of your chain. The replacement process corresponds to the transitions of your Markov chain.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After defining all the properties of your chain (states, transitions, rewards, ...), you can train your model and get the best strategy for each current word.&lt;/p&gt;&#xA;" OwnerUserId="27003" LastActivityDate="2018-05-14T06:49:47.120" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31623" PostTypeId="2" ParentId="31610" CreationDate="2018-05-14T07:33:12.383" Score="0" Body="&lt;p&gt;This is clearly a case of overfitting, as your validation loss is much higher than your training loss. I would proceed by performing &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network#Dropout&quot; rel=&quot;nofollow noreferrer&quot;&gt;dropout&lt;/a&gt; or weight decay: both of them are regularizers. The regularization technique aim is to set the orange line closer to the blue line. If this works, then you have to ask yourself: is the blue line where I aim to? Or do I want a smaller loss? If you want a smaller loss, you can try using another optimizer, another learning rate or training through more epochs (your training error is still decreasing). However, your first move should be using a regularization technique.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-14T07:33:12.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31624" PostTypeId="2" ParentId="31596" CreationDate="2018-05-14T07:47:10.977" Score="2" Body="&lt;p&gt;Forward and backward passes of the whole dataset are called epochs. The number of epochs is a parameter of the training procedure that cannot be estimated a priori, and it depends on how low you want your training/validation errors. One approach is to train until your validation error is small enough, and as long as the number of epochs is smaller than a threshold. Another approach is early stopping: when the validation error achieves a minimum, stop training. For a broader explanation on early stopping, see &lt;a href=&quot;https://deeplearning4j.org/earlystopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;this blog&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up: the number of epochs is particular of every network and dataset, as well as optimization technique, and you should set them in order to have a low enough loss, but not overfit (early stopping is done in order to not overfit).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-14T07:47:10.977" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31625" PostTypeId="1" CreationDate="2018-05-14T08:20:36.730" Score="1" ViewCount="461" Body="&lt;p&gt;I have implemented the algorithm to train self organizing maps in Python and it seems to be working well. I checked with some labeled data and the maps are learning the topology well. Here are some images: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/J3CG2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/J3CG2.png&quot; alt=&quot;Iris Dataset&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hd42M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hd42M.png&quot; alt=&quot;A text dataset&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am calculating a Unified Distance Matrix for visualization which does not seem to be working well as seen in the 2nd image. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe that similarity coloring as seen in the world poverty map developed by Prof. Kaski and Prof. Kohonen would be apt for my application but I am unable to find a clear explanation of how it is to be calculated. The procedure has been outlined in the paper &quot;Tips for Processing and Color-coding of Self-Organizing Maps &quot;, however, it omits some technical details which are necessary to implement the similarity coloring. Here is the image that I am referring to: &#xA;&lt;a href=&quot;https://i.stack.imgur.com/U3N6y.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U3N6y.gif&quot; alt=&quot;World Poverty Map&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Could someone help me with the exact steps to calculate the similarity coloring map? I have searched everywhere for a month and I am unable to find an explanation.&lt;/p&gt;&#xA;" OwnerUserId="45603" LastEditorUserId="45603" LastEditDate="2018-05-16T11:26:53.527" LastActivityDate="2024-01-11T16:01:15.207" Title="Similarity coloring of a self organizing map" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31626" PostTypeId="2" ParentId="31613" CreationDate="2018-05-14T08:32:13.587" Score="1" Body="&lt;p&gt;Machine Learning process usually follows three steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Training (on training data): this data is used in the learning stage. The model optimally adjusts the weights in order to minimize the loss function over this dataset. &lt;/li&gt;&#xA;&lt;li&gt;Evaluation (on evaluation data): it serves to fine-tune the model, for instance you can decide if your model overfits &lt;/li&gt;&#xA;&lt;li&gt;Testing (on test data): tells you about the inference power of your model&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The intersection of these three datasets has to be zero, and they should come from the same disttibution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Recalling that, the first thing is the naming convention for your two datasets:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If training and test datasets come from the same distribution (I guess yes, since they are all Belgian sign traffics) you can name it Training and Validation datasets.&lt;/li&gt;&#xA;&lt;li&gt;If the same condition holds, you can perform cross-validation over your Training dataset, and carry on naming it Training and Tets datasets.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage? &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It depends on whether the practical application you are designing the algorithm for will encounter those remaining signs. If so, I will give these signs a label, for examle 'Don't care'. The algorithm should have the capability of recognizing your 8 signs and to know that the others are of non importance, but do not fall on your desired targets. On the contrary, if the algorithm will be always fed with these 8 signs, you can use only those images. There would be other approaches: you can train a previous SVM in which you set two groups of signs (binary classifier or &lt;a href=&quot;https://link.springer.com/article/10.1023/B:MACH.0000008084.60811.49&quot; rel=&quot;nofollow noreferrer&quot;&gt;Support Vector Data Description&lt;/a&gt;), one containing the target 8, and other with the remaining ones. Then, another SVM can recognize the sign only if it has been previously classified among the 8 desired one (this SVM is only trained with the 8 signs).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Taking it into account, I guess you have to name your datasets as Training and Evaluation. You first train the model with the training data, and then you evaluate how it generalizes with the evaluation data. First you have to decide which metrics you will use, I recommend &lt;strong&gt;precision&lt;/strong&gt; and &lt;strong&gt;recall&lt;/strong&gt;, per class. Once you have your trained model, you perform predictions over the evaluation test and you'll get the metrics for this dataset. For instance, if your results over the evaluation data are worse than on over the training data, your model is overfitting the training dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I strongly recommend you to use cross-validation on the Training-dataset, visit this &lt;a href=&quot;https://es.mathworks.com/help/stats/classificationsvm.crossval.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-14T08:32:13.587" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31628" PostTypeId="1" CreationDate="2018-05-14T08:51:00.017" Score="1" ViewCount="43" Body="&lt;p&gt;I am trying to forecast energy data generated by &lt;a href=&quot;http://www.loadprofilegenerator.de/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LoadProfileGenerater&lt;/a&gt;. This data is generated every half hour for 2 years. I am following this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hEF6L.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hEF6L.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have checked it for stationarity using Dickey Fuller Test&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Results of Dickey-Fuller Test:&#xA;Test Statistic                -1.684705e+01&#xA;p-value                        1.126331e-29&#xA;#Lags Used                     5.200000e+01&#xA;Number of Observations Used    3.503500e+04&#xA;Critical Value (5%)           -2.861623e+00&#xA;Critical Value (1%)           -3.430537e+00&#xA;Critical Value (10%)          -2.566814e+00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results indicate that the time series is very close to stationarity (&lt;strong&gt;Test Statistic &amp;lt; Critical Value (1%)&lt;/strong&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have therefore used &lt;code&gt;ARMA&lt;/code&gt; for forecasting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gANTh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gANTh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However I am not being able to determine the values of &lt;code&gt;p&lt;/code&gt; and &lt;code&gt;q&lt;/code&gt;. Both, PACF and ACF charts indicate intersection with the upper confidence line at &lt;strong&gt;0.01&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xLd8f.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xLd8f.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What values for &lt;strong&gt;p&lt;/strong&gt; and &lt;strong&gt;q&lt;/strong&gt; should I use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried differencing as suggested in the comments. Rolling mean has dropped to nearly 0 but rolling std has increased to 0.75. ACF and PACF have generated similar charts as without differencing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source used for Differencing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Differencing&#xA;ts_log = np.log(ts)&#xA;ts_log_diff = ts_log - ts_log.shift()&#xA;ts_log_diff.dropna(inplace=True)&#xA;stationarity.test_stationarity(ts_log_diff)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zc6nH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zc6nH.png&quot; alt=&quot;After Differencing&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Results of Dickey-Fuller Test:&#xA;Test Statistic                   -64.965417&#xA;p-value                            0.000000&#xA;#Lags Used                        50.000000&#xA;Number of Observations Used    35036.000000&#xA;Critical Value (5%)               -2.861622&#xA;Critical Value (1%)               -3.430537&#xA;Critical Value (10%)              -2.566814&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42587" LastEditorUserId="42587" LastEditDate="2018-05-14T09:28:32.663" LastActivityDate="2018-05-14T09:28:32.663" Title="What values should I choose for P and Q?" Tags="|machine-learning|time-series|forecasting|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31629" PostTypeId="2" ParentId="24403" CreationDate="2018-05-14T08:54:05.960" Score="2" Body="&lt;p&gt;&lt;strong&gt;Conformal Prediction&lt;/strong&gt; as a buzz word might be interesting for you because it works under many conditions - in particular it does not need normal distributed error and it works for almost any machine learning model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Two nice introductions are given by &#xA;&lt;a href=&quot;https://scottlocklin.wordpress.com/2016/12/05/predicting-with-confidence-the-best-machine-learning-idea-you-never-heard-of/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Scott Locklin&lt;/a&gt; and &lt;a href=&quot;http://clrc.rhul.ac.uk/copa2017/presentations/CP_Tutorial_2017.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Henrik Linusson&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="52105" LastActivityDate="2018-05-14T08:54:05.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31630" PostTypeId="2" ParentId="31597" CreationDate="2018-05-14T08:54:43.073" Score="2" Body="&lt;p&gt;Nearest neighbour algorithms (kNN and variants) do not have a training phase. They work by storing all the labelled examples, and using them directly for inference on new data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are some caveats:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&quot;Training&quot; is fast as it just stores each example once, but inference is slow, as typically it involves searching for relevant examples. This can be improved by indexing routines (which are not part of the algorithm, but might be part of a specific implementation).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Although there are no parameters to train, there may still by hyper-parameters to select, which will involve running a cross-validation exercise with hold-out data and varying the choices, e.g. of number of nearest neighbours to consider, or best distance metric to use to determine closest neighbours.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Data preparation (normalisation) and feature engineering are often required to get the best results out of kNN, and also require multiple attempts and tests.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;In general, approaches that store and query labelled data without a training phase are &lt;a href=&quot;https://en.wikipedia.org/wiki/Instance-based_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;instance based learning&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-05-14T10:00:35.727" LastActivityDate="2018-05-14T10:00:35.727" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31631" PostTypeId="2" ParentId="31610" CreationDate="2018-05-14T09:54:04.763" Score="1" Body="&lt;p&gt;When the training loss is lower than the validation loss, the model is said to overfit the training data, i.e. it has learned so much from the training data that it only adjusts well to it and it can't generalize to new data. This phenomena is regarded as the &lt;strong&gt;variance of the model&lt;/strong&gt;. The &lt;strong&gt;bias&lt;/strong&gt; of the model is the difference between the training loss and and the loss you've previoussly selected as the minimum loss reachable, or the desired one. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this analysts is usually done over other well known metrics, such as &lt;strong&gt;precision&lt;/strong&gt; and &lt;strong&gt;recall&lt;/strong&gt;. You first calculate these metrics on your training data, and then, on the evaluation data. Then you perform the analysis taking the same considerations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to reduce the variance/overfitting, there are common techniques:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Increse the training data by adding more instances, if available&lt;/li&gt;&#xA;&lt;li&gt;If no more instances are available, perform data augmentation to increase the training dataset&lt;/li&gt;&#xA;&lt;li&gt;Use regularization, for example, dropout.&lt;/li&gt;&#xA;&lt;li&gt;Shorten the network. If there are lots of layers, the network may be learning too specific features from the training data &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;As mentioned before, I'd perform the analysis using other metrics rather than the loss.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-14T09:54:04.763" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31633" PostTypeId="1" CreationDate="2018-05-14T11:11:05.837" Score="1" ViewCount="428" Body="&lt;p&gt;I am working with Orange for my thesis with logs and core data; however, since I am a beginner, I am a little bit stuck with the feature construction widget. Ultimately, I would like to combine different features to compare them.&lt;/p&gt;&#xA;&lt;p&gt;What kind of information should I put in &amp;quot;Values&amp;quot; field with a categorical feature?&lt;/p&gt;&#xA;&lt;p&gt;If you have any examples on this it would be really appreciated (the ones from Orange did not help me).&lt;/p&gt;&#xA;" OwnerUserId="52115" LastEditorUserId="29169" LastEditDate="2021-08-13T20:07:49.300" LastActivityDate="2024-02-16T10:04:40.597" Title="Feature construction widget on Orange 3.13" Tags="|orange|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31634" PostTypeId="1" CreationDate="2018-05-14T11:18:33.517" Score="0" ViewCount="2433" Body="&lt;p&gt;What is the most common order of data cleaning, data transformation and exploratory data analysis?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For me it seems most logical to do data cleaning, then EDA and finally data transformation (encoding of categorical variables, and feature scaling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing data transformation &lt;em&gt;before&lt;/em&gt; the EDA, seems to make the EDA not that useful, as you cant ex. check for stuff like:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Passengers in the age interval 0-18 has a higher chance of survival&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;(if feature scaling has been applied to &lt;code&gt;age&lt;/code&gt; feature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then again, doing data transformation &lt;em&gt;after&lt;/em&gt; the EDA, also miss out on chance of encoding categorical variables and thereby visualize correlations of those with the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the order of the mentioned processes? And is there even an order?&lt;/p&gt;&#xA;" OwnerUserId="52118" LastActivityDate="2018-12-12T05:00:40.270" Title="Data cleaning and data transformation before EDA?" Tags="|visualization|data-cleaning|data-analysis|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31636" PostTypeId="1" CreationDate="2018-05-14T12:08:19.400" Score="-3" ViewCount="78" Body="&lt;p&gt;I am not able to understand what is asked in the question. Can somebody please explain it to me?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cmMyA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cmMyA.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The given dataset is this, with 15 countries, 5 continent and countries population, renewable (continuous values) etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wmKGi.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wmKGi.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52121" LastEditorUserId="29575" LastEditDate="2018-05-14T13:10:54.107" LastActivityDate="2018-05-14T13:10:54.107" Title="Unable to figure out the groupby question" Tags="|python|pandas|data-cleaning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31637" PostTypeId="2" ParentId="31636" CreationDate="2018-05-14T12:12:41.973" Score="2" Body="&lt;p&gt;Sorry for posting an answer (rather than commenting), I just don't have enough reputation to leave comments yet.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recognize this question from Coursera's Applied Data Science Specialization.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should be asking for clarification on the course's discussion forums (where those running the course will decide how much information it's appropriate to give you), not here.&lt;/p&gt;&#xA;" OwnerUserId="52109" LastActivityDate="2018-05-14T12:12:41.973" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31638" PostTypeId="2" ParentId="31555" CreationDate="2018-05-14T12:16:18.230" Score="1" Body="&lt;p&gt;When training accuracy increases while validation accuracy remains constant or decreases, then the model is most likely overfitting, or it may be saturated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can try the following methods to increase accuracy:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum&quot; rel=&quot;nofollow noreferrer&quot;&gt;Momentum&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Regularization_(mathematics)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Regularization techniques&lt;/a&gt;: &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Early_stopping&quot; rel=&quot;nofollow noreferrer&quot;&gt;Early Stopping&lt;/a&gt;, &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Dropout_(neural_networks)&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dropout&lt;/a&gt;, &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Tikhonov_regularization&quot; rel=&quot;nofollow noreferrer&quot;&gt;Weight Decay&lt;/a&gt;, etc&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Synthetic_data&quot; rel=&quot;nofollow noreferrer&quot;&gt;Synthetic Data&lt;/a&gt; and related techniques such as &lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume16/chawla02a-html/chawla2002.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;SMOTE&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.m.wikipedia.org/wiki/Ensemble_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;Ensembles&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="33551" LastEditorUserId="33551" LastEditDate="2018-05-14T12:38:32.667" LastActivityDate="2018-05-14T12:38:32.667" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31639" PostTypeId="2" ParentId="31634" CreationDate="2018-05-14T12:21:18.747" Score="2" Body="&lt;p&gt;Although not very helpful, the answer is probably &quot;it depends&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends on the techniques you plan to use.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-05-14T12:21:18.747" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31640" PostTypeId="1" CreationDate="2018-05-14T12:22:20.297" Score="2" ViewCount="68" Body="&lt;p&gt;I have a feature array of around 4000 elements, extracted from one source. On this array I've extracted 7 more feature from other source and now I basically have a 4007 feature array from each data point. I am trying to classify these data points based on this feature array, basically doing a 1NN with manhattan distance. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However since I am really bad at maths I'm not sure how to weigh this so my 7 elements can actually help in regards to this. It feels like those 4000 elements are way more important and on distance calculation the other 7 are insignificant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've also given this input to a Neural Network and i'm wondering if I need to preprocess something there aswell or can I just give it the 4007 elements to the input neurons?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've done feature-scaling but I guess that is irrelevant to the problem I'm asking.&lt;/p&gt;&#xA;" OwnerUserId="52122" LastActivityDate="2024-03-27T07:01:00.213" Title="How to weigh feature array" Tags="|neural-network|feature-scaling|k-nn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31641" PostTypeId="1" CreationDate="2018-05-14T12:51:10.300" Score="13" ViewCount="8431" Body="&lt;p&gt;What is the difference between &lt;code&gt;Fully Connected&lt;/code&gt; layers and &lt;code&gt;Bilinear&lt;/code&gt; layers in deep learning?&lt;/p&gt;&#xA;" OwnerUserId="51129" LastEditorUserId="28175" LastEditDate="2018-05-14T13:20:53.113" LastActivityDate="2020-12-22T05:55:56.690" Title="What is difference between Fully Connected layer and Bilinear layer in CNN?" Tags="|machine-learning|neural-network|deep-learning|cnn|computer-vision|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31642" PostTypeId="2" ParentId="28899" CreationDate="2018-05-14T12:55:30.433" Score="0" Body="&lt;p&gt;As others have already said, you force the model to choose one or another. If you do not want this, then you can use sigmoid instead of softmax as the activation function in your final layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But beware, in addition to assigning low probabilities to both classes, the model could assign high probabilities to both classes. You will need to think about how you will handle and interpret that. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-05-14T12:55:30.433" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31643" PostTypeId="2" ParentId="31641" CreationDate="2018-05-14T13:19:39.087" Score="8" Body="&lt;p&gt;I quote the answers from &lt;a href=&quot;https://www.quora.com/What-is-a-bilinear-tensor-layer-in-contrast-to-a-standard-linear-neural-network-layer-or-how-can-I-imagine-it&quot; rel=&quot;nofollow noreferrer&quot;&gt;What is a bilinear tensor layer (in contrast to a standard linear neural network layer) or how can I imagine it?&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;A bilinear function is a function of two inputs &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt; that is linear in each input separately. Simple bilinear functions on vectors are the dot product or the element-wise product.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Let &lt;span class=&quot;math-container&quot;&gt;$M$&lt;/span&gt; be a matrix. The function &lt;span class=&quot;math-container&quot;&gt;$f(x,y)=x^TMy=\sum_iM_{ij}x_iy_j$&lt;/span&gt; is bilinear in &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$y$&lt;/span&gt;. In fact, any scalar bilinear function on two vectors takes this form. Note that a bilinear function is a linear combination of &lt;span class=&quot;math-container&quot;&gt;$x_iy_j$&lt;/span&gt; whereas a linear function such as &lt;span class=&quot;math-container&quot;&gt;$g(x,y)=Ax+By$&lt;/span&gt; can only have &lt;span class=&quot;math-container&quot;&gt;$x_i$&lt;/span&gt; or &lt;span class=&quot;math-container&quot;&gt;$y_i$&lt;/span&gt;. For neural nets, that means a bilinear function allows for richer interactions between inputs.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;Now what if you want a bilinear function that outputs a vector? Well, you simply define a matrix &lt;span class=&quot;math-container&quot;&gt;$M_k$&lt;/span&gt; for each coordinate of the output and you end up with a stack of matrices. That stack of matrices is called a tensor (3-mode tensor to be exact). You can imagine the bilinear tensor product with two vectors as &lt;span class=&quot;math-container&quot;&gt;$x^⊤M_ky$&lt;/span&gt; computed on each “slice” of the tensor.&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;Bilinear Models &lt;em&gt;consists of two feature extractors whose outputs&#xA;are multiplied using an outer product at each location of the image&#xA;and pooled to obtain an image descriptor. &lt;a href=&quot;http://www.ics.uci.edu/%7Eskong2/img/CVgourp_20160120.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/pfvFk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/pfvFk.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Its advantage is that &lt;em&gt;it can model pairwise feature interactions in a translationally invariant manner, which is particularly useful for fine-grained categorization. It also allows end-to-end training using image labels only, and achieves state-of-the-art performance on fine-grained classification.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="63844" LastEditDate="2020-12-22T05:55:56.690" LastActivityDate="2020-12-22T05:55:56.690" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31644" PostTypeId="1" CreationDate="2018-05-14T13:41:29.340" Score="1" ViewCount="67" Body="&lt;p&gt;I have to predict the call traffic for a telecom company.&lt;/p&gt;&#xA;" OwnerUserId="52129" LastActivityDate="2018-05-14T20:10:10.760" Title="How can I predict call traffic for the telecom company" Tags="|machine-learning|random-forest|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-15T07:28:37.543" ContentLicense="CC BY-SA 4.0" />
  <row Id="31645" PostTypeId="2" ParentId="31602" CreationDate="2018-05-14T13:53:49.250" Score="2" Body="&lt;p&gt;It's best to leave the test data unseen. You can create a data transformation pipeline for your test data so that your test data will be transformed into a TFIDF vector after being passed through the pipeline. This way you get to know how well your model performs on unseen data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;And it's OK to transform the entire training data into a TFIDF vector and run the k-fold validation. &lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-14T13:53:49.250" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31647" PostTypeId="2" ParentId="30652" CreationDate="2018-05-14T15:49:39.887" Score="0" Body="&lt;p&gt;You could use menus in other languages as part of your corpus along with your Portuguese words. I would focus on having text that is correctly spelled for the corpus. Then you can use something like Levenshtein distance to give candidates for words never seen in your corpus (implying incorrectly spelled or missing).  &lt;/p&gt;&#xA;" OwnerUserId="3430" LastActivityDate="2018-05-14T15:49:39.887" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31649" PostTypeId="2" ParentId="31555" CreationDate="2018-05-14T16:21:17.680" Score="0" Body="&lt;p&gt;There are a few approaches you might take to establish how and whether accuracy can be improved. Some options sketched out here in brief are to compare network results to a baseline estimator, diagnose misclassifications, apply dimensionality reduction, and network architecture troubleshooting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Without knowing much about the data, I think you could try to establish whether it is even possible to increase the accuracy of your network by comparing network performance with a baseline estimator like a decision tree or support vector machine. The bonus of using a decision tree is that an algorithm such as &lt;a href=&quot;https://www.cise.ufl.edu/~ddd/cap6635/Fall-97/Short-papers/2.htm&quot; rel=&quot;nofollow noreferrer&quot;&gt;ID3&lt;/a&gt; uses information gain to make the splits and this might give you some intuition about your data and whether it's possible to improve accuracy before sinking time into it. If you haven't already, you could do some exploratory analysis to understand how noisy the data/classes are.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may also be useful to take a closer look at the errors your network is making. A confusion matrix or &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;classification report&lt;/a&gt; can help you to diagnose whether your network is struggling with (for example) one particular class or a specific area of the decision boundary. If your classes are unbalanced, you could look at re-weighting your training examples and approaches such as SMOTE etc that Benji mentions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With 70 features, the network may benefit from some dimensionality reduction. If you apply &lt;a href=&quot;http://setosa.io/ev/principal-component-analysis/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PCA&lt;/a&gt; to the data you can establish how many components you need to explain a reasonable amount of variance in the data, or if it is more complicated. It is possible that your network may respond better if taking the transformed data as input rather than the raw data (on that topic, have you pre-processed your data?). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, a broader architecture search might yield improved results - I'm assuming you may have done this already, but have you tried increasing the units in each layer or cutting out that second hidden layer? You might also try using different activation functions or a different learning rate schedule. Bengio's '&lt;a href=&quot;https://arxiv.org/abs/1206.5533&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical recommendations for gradient-based training of deep architectures&lt;/a&gt;' is worth a skim (the article is aimed mainly at deeper networks, but elements of sections 3.1, 3.2 and 4 are relevant). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This answer has only touched briefly on each areas but I'm hoping there are some solid leads you can chase up. &lt;/p&gt;&#xA;" OwnerUserId="32697" LastActivityDate="2018-05-14T16:21:17.680" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31650" PostTypeId="1" CreationDate="2018-05-14T16:28:32.633" Score="-1" ViewCount="545" Body="&lt;p&gt;i have a big dataset with 2 classes. The 70% of records belong to the first class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It if create a training and test set with the operator &quot;&lt;strong&gt;split validation&lt;/strong&gt;&quot; with &lt;strong&gt;stratified sampling&lt;/strong&gt; i get a test set with 70% records of first class .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What can i do in order to create a test set with 50% or records belonging to the first class ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2020-08-06T00:02:55.217" Title="RapidMiner create test partition" Tags="|classification|data-mining|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31651" PostTypeId="1" CreationDate="2018-05-14T17:29:34.157" Score="1" ViewCount="600" Body="&lt;p&gt;I have a function that visualizes 2D convolutional kernels. The weights come as 4D-tensors [filter_height, filter_width, in_channels, out_channels], as specified for conv2d. Now I try to sort the kernels by their variance before visualizing them, highest variance first.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current approach would be something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#get variance along 2D kernel dims, returns tensor [in_channels, out_channels]&#xA;mean, variance = tf.nn.moments(kernel, [0,1])&#xA;&#xA;#get sorted indices of output kernels by their variance&#xA;indices = tf.nn.top_k(variance, k=(variance.shape[1]), sorted=True).indices&#xA;&#xA;#kernel to [in_channels, out_channels,filter_height, filter_width ]&#xA;kernel= tf.transpose(kernel, (2, 3, 0, 1))&#xA;&#xA;#get indices to same dimension as kernel&#xA;indices=tf.expand_dims(indices,axis=2)&#xA;indices=tf.expand_dims(indices,axis=2)&#xA;&#xA;#sort kernel according to indices&#xA;kernel=tf.gather_nd(kernel,indices)&#xA;&#xA;#kernel back to [filter_height, filter_width, in_channels, out_channels]&#xA;kernel= tf.transpose(kernel, (2, 3, 0, 1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But after trying endless approaches, I can't even get tf.gather_nd() to output a tensor with the correct  amount of dimensions.&#xA;I think I'm not entirely sure about the functions inner workings though.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone lead me into the right direction? Any help is highly appreciated!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; to clarify, i would like a tensor  [filter_height, filter_width, in_channels, out_channels] where the first of the filter_heightXfilter_width kernels is the one with the highest variance, the second the one with the next highest and so on up to out_channels&lt;/p&gt;&#xA;" OwnerUserId="47405" LastEditorUserId="1330" LastEditDate="2018-05-14T18:29:36.447" LastActivityDate="2018-05-14T18:29:36.447" Title="How to sort tensor along two axis by variance in TensorFlow?" Tags="|neural-network|tensorflow|variance|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31652" PostTypeId="1" AcceptedAnswerId="31653" CreationDate="2018-05-14T17:54:58.557" Score="30" ViewCount="17626" Body="&lt;p&gt;In the case of having a combination of categorical and numerical Attributes, I usually convert the categorical attributes to one hot vectors. My question is do I leave those vectors as is and scale the numerical attributes through standardization/normalization, or should I scale the one hot vectors along with the numerical attributes?&lt;/p&gt;&#xA;" OwnerUserId="41312" LastActivityDate="2018-05-14T18:11:29.380" Title="Should one hot vectors be scaled with numerical attributes" Tags="|feature-engineering|feature-scaling|data-science-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31653" PostTypeId="2" ParentId="31652" CreationDate="2018-05-14T18:11:29.380" Score="25" Body="&lt;p&gt;Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The effect of doing so will depend on the model class, and type of normalisation you apply, but I have noticed some (small) improvements when scaling to mean 0, std 1 for one-hot-encoded categorical data, when training neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It may make a difference too for model classes based on distance metrics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, like most of these kind of choices, often you have to try both approaches and take the one with the best metric.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-05-14T18:11:29.380" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31654" PostTypeId="2" ParentId="31545" CreationDate="2018-05-14T18:26:22.283" Score="1" Body="&lt;p&gt;It would help to do some analysis of the scripts to identify aspects that distinguish the various categories. Once you do this manually for some examples, you could consider writing some rules based on the observations. The rest of the examples can be labeled using the rules. For a model-based approach, if you label a small set of examples (~50), then a simple model (Naive Bayes, etc.) can potentially be trained on these.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-14T18:26:22.283" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31655" PostTypeId="2" ParentId="31545" CreationDate="2018-05-14T18:46:23.883" Score="1" Body="&lt;p&gt;Since you know the quantity of labels (6) you can use &lt;a href=&quot;https://es.wikipedia.org/wiki/K-means&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means&lt;/a&gt; algorithm to cluster your data into 6 groups. I recommend you to represent each using the tfidf method. You can implement your code using &lt;a href=&quot;http://scikit-learn.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; functions.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-05-14T18:56:47.197" LastActivityDate="2018-05-14T18:56:47.197" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31657" PostTypeId="1" AcceptedAnswerId="31658" CreationDate="2018-05-14T19:26:59.803" Score="2" ViewCount="1353" Body="&lt;p&gt;I have been using different machine learning algorithms throughout various projects at university, and attended some inspirational lectures where industrial companies show and present how they use machine learning, data mining, etc. in their work. I myself mostly use Python, and have previously used libraries such as sklearn. My problem is, that I have a huge difficulty understanding the role of built in algorithms vs making them completely from scratch with pure coding and math - i.e. using theoretical machine learning tools to actually do the work yourself. I understand doing everything yourself can be constrained by time/money/resources. Also sometimes it doesn't make sense to reinvent stuff that has been vastly optimized by others before you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I keep feeling that using sklearn's built in random forest classifier or using xgboost in python is kind of cheating. I am only preparing the data, cleaning it to get the right formats, maybe do some feature engineering og initial plottings and statistical analysis. The problem is that when all that is done, we simply feed the data to this pre-made algorithm and it does everything behind the scene, and just spits out predictions. I feel that i am not doing anything, and not using all the knowledge i have learned in the data exploration analysis. Neither am i using any of the patterns that i found in the data. Still i hear from big companies that they use xgboost and sklearn - and I can see it actively being used in Kaggle competitions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Almost every website i find only provides examples using these built in libraries, and do not go through any deeper math or statistics at all. I really enjoy working with machine learning - but i have this strong feeling that im completely missing the &quot;professional&quot; approach of doing things. I know there is a lot of books on theoretical machine learning - but still almost everyone online seems to just use pre-made algorithms. I have been struggling with this understanding for about a year now. The validity of these pre-made algorithms in serious industrial/business/academic use is still not clear to me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; To be more specific. My question is: How are these libraries/tools viewed in professional/industrial/academical context in comparison to actually building a model yourself. Are they just a &quot;quick and easy&quot; alternative way to start learning machine learning and data mining for students and amateurs, or are they in fact more powerful (than i at least know) and should not be seen as an alternative, but a viable solution for professionals? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The motivation for my single question above, can be ellaborated by explaining the questions i ask myself. The very questions that began this confusion for me. Is it cheating to use these models? In which situations would you use a pre-built library, and when to avoid it? How do i merge (or use) the knowledge gained from the scientific data analysis i did before modelling, and these pre-built classifiers.&lt;/p&gt;&#xA;" OwnerUserId="50174" LastEditorUserId="50174" LastEditDate="2018-05-16T00:53:17.163" LastActivityDate="2018-05-16T00:53:17.163" Title="Using packages such as sklearn vs building ML algorithm from scratch" Tags="|machine-learning|methodology|" AnswerCount="2" CommentCount="5" ClosedDate="2018-05-15T22:54:53.227" ContentLicense="CC BY-SA 4.0" />
  <row Id="31658" PostTypeId="2" ParentId="31657" CreationDate="2018-05-14T19:52:52.213" Score="4" Body="&lt;p&gt;It depends entirely on your goal.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Student Phase&lt;/strong&gt;  When you're learning about machine learning algorithms, I think it is a really good idea to implement toy examples.  I find this process helps with finding what you understood well and what you did not understand as well as you thought.  It's doing this work where you'll find deeper understanding of how the algorithm really works and different internal choices you have to make.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Professional Phase&lt;/strong&gt;  When you have a project to deliver, you don't necessary want to rewrite a random forest implementation from scratch.  Even if you could build one in a reasonable amount of time, there's value in having something like sklearn that is well vetted and robust enough to handle edge cases that you wouldn't even consider.  That's the advantage of using pre-built libraries.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I need more Phase&lt;/strong&gt;  Eventually, you get the point where you understand the math and know how to use the packages well and you realize that there's a feature lacking.  That's when you rip open a framework like xgboost or sklearn and modify existing code or even create you own implementation.  The reason you would do this is because methods are cutting edge so there just isn't anything out there or the implementation of the framework is actually a handicap in production (as I tend to find with sklearn).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue you seem to be facing is lack of accountability for your output.  If all you do is clean data push it through a model and get good results, make a chart showing results, I would say you are forgetting the &quot;science&quot; part in data science.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The challenging part isn't using the model it is knowing what moves your model and the potential hurdles your model may face in the real world.  I've seen this play out often in my career, where a junior member will make an awesome model on training and test dataset, and suddenly production comes along and performance tanks.  Why would that happen?  Well, because test and training were similar (and often from the same source), but the junior member failed to question the lack of variety in the data source and question if the real world behaved the same way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am trying to say is, being a data scientist, a small part of the job is cleaning data, running model and making pretty pictures.  The real challenge is asking the why question.  Why does this work?  Why does the model perform poorly?  Why does the model perform well?  Why is this feature important?  &lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-05-14T19:52:52.213" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31661" PostTypeId="2" ParentId="24249" CreationDate="2018-05-14T20:26:34.787" Score="1" Body="&lt;p&gt;Let's first begin with a primer to reinforcement learning. Reinforcement Learning is a method of learning that predicts the action to perform at a given state or predicts the value associated with a given state and action pair.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, Neil is correct that this problem is not explicitly a reinforcement learning problem. This begs the question, how does one define a reinforcement learning problem. It is defined as a 5-tuple containing the following elements. $S$ is the set of states, which must be finite. $A$ is the set of actions available at a given state, it is also sometimes written as $A_s$. $P_a(s, s') = \Pr(s_{t+1}=s' \mid s_t = s, a_t=a)$ is the probability that action $a$ at state $s$ will lead to state $s'$. $R_a(s, s')$ is the reward received when using action $a$ to transition from state $s$ to $s'$. $\gamma$ which is the discount factor which is a value between $0, 1$ inclusive to discount rewards at certain steps.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, we have to frame the question in the form of a reinforcement learning problem. You need to establish a state, in your case this would be the information about the patient that you believe has a high correlation with their happiness (or is a causal factor for their emotions). Now, you have a state of your patient with these features you've engineered. Now you must decide how to frame your question appropriately. Here you have two choices, either, you can choose what their emotion is and update your weights (your model weights, or the weights you want to output) as necessary (based on the true labels) or you can use your model to choose an action (in this case it would be a class of 1-7). These approaches are labelled as $Q$ learning or $\pi$ Policy methods. These are the primary methods present in RL, one chooses the value to assign to a given state and action $Q(a, s)$ and the other takes in a state $\pi(s)$ and maps it to an action to perform. You can decide which one you'd like to use based on convenience/experience.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once this is done, you'd have to decide on an architecture you'd like to use to update the gradients, this is totally up to you. You can do a literature review to see what's going on in this domain and use what you think is appropriate. Just as an example, let's say you'd want to use a Policy method that is updated by a vanilla CNN. You would pass in the state to your CNN in some form that would be interpretable by the CNN and then choose an action based on the class labels. Now, you would have a reward function which would choose how far you are from the true class (you can do this heuristically, like Happiness is 10 units from Sadness). The reward function would be propagated to the network and it would update the weights as required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PERSONAL COMMENTS: I think this is completely unnecessary for the current task. In my opinion a simple vision classifier should do perfectly well. There are an abundance of datasets that are available which takes in facial features and classifies sentiment/happiness. Also, a lot of statisticians believe that numeric values for happiness/sadness is a bad metric because questions like, &quot;How different is very happy from happy&quot; are hard questions to answer in the context of associating a reward with them. Do take Neil's comments into consideration.&lt;/p&gt;&#xA;" OwnerUserId="27114" LastActivityDate="2018-05-14T20:26:34.787" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31662" PostTypeId="1" AcceptedAnswerId="31664" CreationDate="2018-05-14T20:32:13.697" Score="3" ViewCount="1469" Body="&lt;p&gt;Say I've divided the data into 3 parts: training, validation and test. I know for example, that in Neural Networks, the number of hidden layers is a hyper parameter. Why can't I train numerous NN architectures in the training set and then test their accuracy in the test set; thereby allowing me to choose one final model? What is the purpose of the validation set in this instance?&lt;/p&gt;&#xA;" OwnerUserId="52096" LastActivityDate="2018-05-15T12:52:41.087" Title="Why can't I choose my hyper-parameter in the training set?" Tags="|cross-validation|training|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31663" PostTypeId="2" ParentId="31662" CreationDate="2018-05-14T20:36:41.723" Score="1" Body="&lt;p&gt;This is exactly why. If you use your &lt;em&gt;test&lt;/em&gt; dataset to select the best architecture that can predict, then you are not using it as a &lt;em&gt;test&lt;/em&gt; data, you are using your test data as a &lt;em&gt;validation&lt;/em&gt; data. And, in the end, you run out of data to test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it clear:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Train&lt;/strong&gt; dataset: Use your &lt;em&gt;train&lt;/em&gt; dataset to learn the parameters of your model (for example, weight of connections in your NN).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Validation&lt;/strong&gt; dataset: Use your &lt;em&gt;validation&lt;/em&gt; dataset to see how they perform and tune these hyperparameters (for example, prune your nets, etc). &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Test&lt;/strong&gt; dataset: Your &lt;em&gt;test&lt;/em&gt; dataset should only be used just in the last instance to test the model. Otherwise, you are not be able to test on data that you haven't use to train your model.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;You can read more about it on this &lt;a href=&quot;https://en.wikipedia.org/wiki/Training,_test,_and_validation_sets&quot; rel=&quot;nofollow noreferrer&quot;&gt;article&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="49914" LastEditorUserId="49914" LastEditDate="2018-05-15T12:52:41.087" LastActivityDate="2018-05-15T12:52:41.087" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31664" PostTypeId="2" ParentId="31662" CreationDate="2018-05-14T22:17:46.663" Score="3" Body="&lt;p&gt;I wouldn't say you can not tune the hyper-parameters in the trainig dataset, but the purpose of doing so is different than in the validation set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, what it is intended with a ML algorithm is how to optimally classify or perform regression given some training data. Once the model is trained, it will be used with new data to obtain predictions, thus:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;We need some training data. With this dataset, the model will look for the optimal weights and biases that minimizes the selected loss function.&lt;/li&gt;&#xA;&lt;li&gt;We need some upper bound to our performance metric: for example, we can decide that our model has to perform like a human, which has showed an average metric of 95% for some specific task. The upper bound can be the best score in a certan benchmark, etc.&lt;/li&gt;&#xA;&lt;li&gt;We start training the model on the &lt;strong&gt;training&lt;/strong&gt; data and we evaluate its metric, lets take the accuracy. If the accuracy is too low, we can tune the hyper-paramters until the accuracy increases in the training data (no evaluation dataset is used here). The difference between our model's accuracy and the upper bound is taken as the &lt;strong&gt;bias&lt;/strong&gt; of the model (this bias has not to be confused with the biases of the neurons). So, the accuracy on the training data gives us the bias of our model.&lt;/li&gt;&#xA;&lt;li&gt;Once we decided our bias is reasonable (optimally 0) arises the question of how the model will perform when it is fed with data that has not be used for training (the real application). The difference between the accuracy on this unseen data and the accuracy on the training data is the &lt;strong&gt;variance&lt;/strong&gt; of the model. This unseen data is the &lt;strong&gt;validation set&lt;/strong&gt; and give us an idea of how the model generalizes to new data. If the variance is high, then the model poorly generalizes to new data. Then, we perform hyper-paramater tunning and evaluating the accuracy on the validation data until the variance is low enough, trying to not worsening the bias (variance-bias trade-off).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Example: desired accuracy = 95%. Training accuracy = 93%. Validation accuracy = 82% -&gt; Bias = 2%, Variance = 11%&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Summarizing:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Tunning in the training data = decreasing bias&lt;/li&gt;&#xA;&lt;li&gt;Tunning in the validation data = decreasing variance&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;What's more unclear is the role of the test dataset. I usually use it when I decided that the model will be no longer changed and I need the final metric that describes the model. The test set can also be used to have an idea about how the model performs with data which have not be intended to work with. In general, the test set gives the power of the model for the inference task.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastEditorUserId="51714" LastEditDate="2018-05-15T06:38:41.717" LastActivityDate="2018-05-15T06:38:41.717" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31665" PostTypeId="1" CreationDate="2018-05-15T00:01:49.893" Score="2" ViewCount="93" Body="&lt;p&gt;For adaptive learning optimizers such as Adam and RMSProb, the &lt;em&gt;effective&lt;/em&gt; learning rate is not the same for all weight parameters. This means that we are not really following the direction of the steepest decent vector in the weights space, correct? If so, why then should we bother with computing the gradient (via backprop) in the first place?&lt;/p&gt;&#xA;" OwnerUserId="52148" LastActivityDate="2018-05-15T07:13:02.693" Title="Do adaptive learning optimizers follow the steepest decent?" Tags="|neural-network|optimization|gradient-descent|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31666" PostTypeId="2" ParentId="31555" CreationDate="2018-05-15T04:20:39.907" Score="0" Body="&lt;p&gt;It is hard to determine if this is in fact overfitting because your validation accuracy has yet to start decreasing. Overfitting is characterized by a set of model parameters which perform very well on the training set but do not generalize to different subsets of the data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I assume you are using Keras based on these plots. To avoid overfitting when training you can save the weights only when the model gets better on your validation set and ignore those which do no better or worse. This will ensure that at the end of the training session you have the weights which generalized the best to the validation set saved to file which can then be loaded into the model. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Saving only the best set of weights in Keras&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Use a callback which will run between epochs&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name = 'my_model'&#xA;&#xA;if not os.path.exists('weights//'+name):&#xA;    os.makedirs('weights//'+name)&#xA;&#xA;# Save the weights using a checkpoint.&#xA;filepath='weights//' + name + '//weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5'&#xA;checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')&#xA;callbacks_list = [checkpoint]&#xA;&#xA;epochs = 100&#xA;# Fit the model weights.&#xA;history = model.fit_generator(generator=training_generator,&#xA;                              steps_per_epoch=training_generator.num_batches,&#xA;                              epochs=epochs,&#xA;                              verbose=1,&#xA;                              callbacks=callbacks_list,&#xA;                              validation_data=validation_generator,&#xA;                              validation_steps=validation_generator.num_batches)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Loading your best set of weights&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Then when the training is done you can retrieve the best set of weights using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;weight_filename = '...'&#xA;model.load_weights('weights//' + name + weight_filename)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-15T04:20:39.907" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31669" PostTypeId="2" ParentId="31665" CreationDate="2018-05-15T07:13:02.693" Score="4" Body="&lt;p&gt;Adam and RMSprop are both optimization algorithms which make the vanilla gradient descent more robust. You still need to calculate the gradient however, these algorithms will facilitate the descent in certain directions and punish the descent in others based on some criteria. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before delving into the details consider this analogy; imagine you are on a mountain and you want to get back to the village. To do this you will follow the downhill slope until you get to the bottom of the mountain. However imagine a mountain that looks like the following&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/g90W4.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/g90W4.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Evidently the best path down the mountain should just be straight down the snowy path. However, if we carefully calculate the gradient we will notice that it will always have a slight left and right component. This will cause us to oscilate from left to right while going down the mountain. This is a huge waste of energy, but it's not catastrophic. However, when we start reaching the base of the mountain the downward slope will start to subside and the oscillations will become more prominent. This may cause us to never truly reach the minimum due to the oscillations becoming dominant, or cause the learning process to be dragged on excessively thus wasting resources. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The optimization techniques such as Adam and RMSprop simply make adjustments to the gradient descent step such that we can more directly find the minimum of the objective function without distraction from oscillating gradients. This is achieved in RMSprop and Adam by maintaining statistics relevant to the past computed gradients. RMSprop keeps track of the mean and Adam keeps track of both the mean and the moment of the past gradients.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;An amazing reference for the inner-workings of the gradient descent variants can be found &lt;a href=&quot;https://arxiv.org/pdf/1609.04747.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Gradient descent&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Vanilla gradient descent is defined as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\theta$ is the set of parameters which define our objective function $J(\theta)$. The learning rate is $\eta$ and $\nabla$ defines the gradient operator. We will extend this function.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Adding Momentum&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;Firstly, if we throw a ball down our cliff it will get some momentum as it falls. This will cause it to tend towards the steepest part of the gradient and left to right oscillations would be minimized. We can do the same to the gradient descent algorithm by introducing momentum $\gamma$ as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nu_{t} = \gamma \nu_{t-1} + \eta \nabla_\theta J(\theta)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1} = \theta_{t} - \nu_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is a good result but we can still do better.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Adagrad&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;It is preferable to have a distinct learning rate for each parameter such that we can adjust their impact on the gradient based on their past values. This is achieved by weighing the gradients by a sum of squares term of their past gradients up until the current time $t$. The matrix $G_t$ is a diagonal matrix where its entries are precisely this sum of squares of past gradients for each parameter. The Adagrad update is thus,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1,i} = \theta_{t,i} - \frac{\eta}{\sqrt{G_{t,ii} + \epsilon}}\nabla_{\theta_t}J(\theta_{t,i})$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this method suffers from diminishing gradients when the sum of squares matrix makes it such that the past values are all too small and cause no more updates to be possible. Or for parameters which may be impactfull in the future to lose all relevance due to their low gradients at the start.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;RMSprop&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This is a solution to the diminishing gradients problem. We will use a running average of the past gradients until a time $t$ as defined by $E[g^2]_t$. The update rule is thus&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g^2_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}}g_t$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $g$ is the gradient for a single parameter.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Adam&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;In addition to keeping track of the running average of the past gradients we will also keep track of their second order moment. $m$ is the mean and $\nu$ is the moment. Thus we keep track of them as follows&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\nu_t = \beta_2 \nu_{t-1} + (1-\beta_2)g^2_t$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will correct these terms to remove their bias that tends towards a zero vector as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{m}_t = \frac{m_t}{1-\beta^t_1}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{\nu}_t = \frac{\nu_t}{1-\beta^t_2}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the update rule is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{\nu}} + \epsilon}\hat{m}_t$&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-15T07:13:02.693" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31670" PostTypeId="1" AcceptedAnswerId="31674" CreationDate="2018-05-15T08:10:03.107" Score="0" ViewCount="483" Body="&lt;p&gt;I'm handling a high imbalanced dataset, thus, I'm weighing the loss function in order to penalize the misclassification of the minority classes, I set the weights in each batch as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;w[i] = num_total_instances / num_instances_of_class[i]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Where 'i' goes from 0 to N-1 (for N classes), 'num_total_instances' is the total number of instances in the batch, and 'num_instances_of_class[i]' is the number of instances of class 'i' in the batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, as I'm doing it on each batch, It may occur that, in a certain batch there is no instance of class 'j', thus w[j] = inf. ¿What weigh should I set for class 'j'? At this moment, I'm setting this w[j] to 1E6&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know that there are other approaches to fight against imbalanced datasets and I also know that I can pre-compute the weights for the whole dataset and use them (fixed) cross all batches, but, I'd like the specific answer for this question, whereas it makes sense. For sure, it will be well received other suggestions for the calculation of the weights.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-15T08:51:31.680" Title="Setting class-weights in a batch in where a certain class is not present" Tags="|dataset|class-imbalance|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31671" PostTypeId="1" CreationDate="2018-05-15T08:22:36.473" Score="2" ViewCount="1470" Body="&lt;p&gt;My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after prediction, I mulitiplied this scale value back.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But since I used a DNN, wouldn't the nonlinear change the scale so make the multiply not valid? e.g.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;given input data: X, label: y;&#xA;y' = y/scale&#xA;X' = X/scale&#xA;predicted = f(X')&#xA;predicted_update = predicted * scale&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Anyone could provide some advice on whether I could do this or it's actually not correct? How do we handle this kind of problem?&lt;/p&gt;&#xA;" OwnerUserId="51708" LastEditorUserId="40853" LastEditDate="2018-05-15T08:26:20.973" LastActivityDate="2021-09-09T14:04:30.433" Title="normalization/denormalization for linear regression problem" Tags="|machine-learning|linear-regression|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31672" PostTypeId="1" AcceptedAnswerId="31673" CreationDate="2018-05-15T08:30:15.803" Score="2" ViewCount="212" Body="&lt;p&gt;Assuming that we have a dataset with the respondents on each row (&lt;code&gt;N&lt;/code&gt; respondents) and their respective characteristics as columns (&lt;code&gt;C&lt;/code&gt; characteristics). Each respondent has also a &lt;code&gt;weight&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In case of high number of respondents, is it a good idea to remove the duplicate respondents and sum their &lt;code&gt;weight&lt;/code&gt;s ? Will this lead to different results ? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my initial data would look like this &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dt&#xA;   id weight v1 v2&#xA;1:  1     10  2  4&#xA;2:  2     11  2  4&#xA;3:  3     12  2  4&#xA;4:  4     13  3  5&#xA;5:  5     14  3  5&#xA;6:  6     15  3  5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And since respondents &lt;code&gt;1,2,3&lt;/code&gt; are the same, and respondents &lt;code&gt;4,5,6&lt;/code&gt; are the same i would end up with this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; dt&#xA;   id weight v1 v2&#xA;1:  1     33  2  4&#xA;2:  2     42  3  5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="19212" LastEditorUserId="19212" LastEditDate="2018-05-15T08:40:40.253" LastActivityDate="2018-05-15T08:45:22.643" Title="do the results change if you remove duplicate rows and you sum their weights?" Tags="|machine-learning|regression|data-cleaning|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31673" PostTypeId="2" ParentId="31672" CreationDate="2018-05-15T08:45:22.643" Score="1" Body="&lt;p&gt;With weighted linear regression, it is exactly the same, as the &lt;a href=&quot;https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#Weighted_linear_least_squares&quot; rel=&quot;nofollow noreferrer&quot;&gt;expression for the loss function&lt;/a&gt; is a sum of weights multiplied by errors in the prediction. This works, of course, for other methods with loss functions, such as logistic regression and neural networks. This is due to the fact that the loss function is linear with respect to the weights. As you save memory, it is totally recommendable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With other methods, you should check if the criteria for choosing parameters or method is linear or not with respect to the weights. If not, you should not do it (to me it doesn't make sense that methods are not linear with respect to weights, but there might be a case where this happens).&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-15T08:45:22.643" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31674" PostTypeId="2" ParentId="31670" CreationDate="2018-05-15T08:51:31.680" Score="2" Body="&lt;p&gt;I don't think that the weight you use matters, whether you set it to $10^6$ or $10^8$. This is because, as examples of that class are not in the batch, the loss function for that batch will not have any of those examples contributing, so that weight won't appear. If something does not appear in the loss function, it is not used at all. For that reason, it doesn't really matter what weight to use.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-15T08:51:31.680" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31675" PostTypeId="2" ParentId="31671" CreationDate="2018-05-15T08:59:23.733" Score="1" Body="&lt;p&gt;I think it is ok, as long as your training and test data have the same maximum values for every feature, approximately. The idea is that the scaling has to be done with the training set (remember that using the test set for anything that is not testing is illegal, not even for scaling).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, you actually fit $y'$ as a function of $X'$, and you have a model that maps properly $y' = f(X')$. When you get your test data, you just obtain the predictions by doing $f(X_{test}')$. As the paragraph before states, if you have that $scale \approx scale_{test}$, then you can just recover $y_{test}$ by doing $y_{test} = scale \cdot f(X_{test}')$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: Don't worry about nonlinearities&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even if the function $f$ is highly nonlinear, it is a function capable of mapping $X'$ to $y'$. If you trust this function and trust the fact that $y = y' \cdot scale$, then there is no need to worry about the way $f$ acts, as function composition makes sense for all kind of functions, both linear and nonlinear.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-15T10:58:26.877" LastActivityDate="2018-05-15T10:58:26.877" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31678" PostTypeId="1" AcceptedAnswerId="31679" CreationDate="2018-05-15T10:17:58.053" Score="0" ViewCount="545" Body="&lt;p&gt;I want to build a fast binary classifier that decides if an image belongs to a given class (e.g. if it is a picture of a person). I want to do this by training a network on the RGB of pixels at a predetermined set of coordinates (e.g. 4 points, one near each corner of the image) and I want to achieve at least 75% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How many points and what architecture should I use? Or if this is a very bad method what is another way to build a classifier that makes training and classification as fast as possible maybe on the account of lowering accuracy?&lt;/p&gt;&#xA;" OwnerUserId="52172" LastActivityDate="2018-05-15T10:32:56.347" Title="Lightweight binary image classifier" Tags="|machine-learning|python|classification|image-classification|" AnswerCount="1" CommentCount="8" ClosedDate="2018-05-16T19:13:47.033" ContentLicense="CC BY-SA 4.0" />
  <row Id="31679" PostTypeId="2" ParentId="31678" CreationDate="2018-05-15T10:32:56.347" Score="3" Body="&lt;p&gt;If I wanted to build a classifier that takes short time to be trained, I would rather simplify the classifier than the data. In this case, I would rather train a logistic regression model (probably with Ridge or Lasso regularization), than a very complex architecture with few of the data I have.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I had to simplify the data, I would take the average of the three channels, thus having a black and white image as input. I would not simplify it further (I am very confident that 4 points won't give decent accuracies).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't really know your circumstances, but Lasso in logistic regression is a method that might allow you to select some pixels. As the coefficients obtained by Lasso are sparse (&lt;a href=&quot;https://stats.stackexchange.com/questions/151954/sparsity-in-lasso-and-advantage-over-ridge-statistical-learning&quot;&gt;see this answer&lt;/a&gt;), Lasso will select the pixels that are relevant towards the prediction of the class. You can then train a bigger network with those pixels that allows you to capture nonlinearities and more complexity.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-15T10:32:56.347" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31680" PostTypeId="1" AcceptedAnswerId="31682" CreationDate="2018-05-15T11:28:37.817" Score="-1" ViewCount="166" Body="&lt;p&gt;I've got a problem with understanding the CV parameter in cross_validate. Could you check if I understand it correctly?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm running ML algorithms in big set of data (train 37M rows), therefore I would like to run  a big validation procedure to choose the best model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html#sklearn.model_selection.ShuffleSplit&quot; rel=&quot;nofollow noreferrer&quot;&gt;ShuffleSplit&lt;/a&gt;, I want to build 100 different ways of splitting data in random way:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cv_split = model_selection.ShuffleSplit(n_splits = 100, test_size = .1, train_size = .9, random_state = 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I want to use it as CV hyperparameter in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;cross_validate&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cv_results = model_selection.cross_validate(model, X, Y, cv  = cv_split)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does it mean that my Train set (X &amp;amp; Y) is divided into 100 random samples (each is then divided into: train (90% of sample), test (10 % of sample)) and during cross_validation model is built for each sample separetly (fitted on 90% of particular one/10 sample and tested on remaining 10% if this sample) and the mean prediction of those 100 models is the result? Also, if I am using Shaffle, does it mean that particular row can be in multiple samples and other will not be in any of them?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, 37M set is devided:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;First Sample 370k XY1, 90% *3.7 = 333k rows as XY1_1(train), 37k as XY1_2 (test); model fitted on .fit(X1_1, Y1_1), predition is build on .predict(X1_2) and validated against Y1_2&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Second Sample 370k, 333k rows as XY2_1 and 37k rows as XY2_2; model fitted on .fit(X2_1, Y2_1), prediction built on .predict(X2_1) and validated against Y2_2&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;etc&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I am not sure If the second explanation is more clear. But this is how I structure it in my head.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also read scikit.learn guide: &lt;a href=&quot;http://3.1.%20Cross-validation:%20evaluating%20estimator%20performance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cross-validation: evaluating estimator performance&lt;/a&gt; but I am still not sure &lt;/p&gt;&#xA;" OwnerUserId="22074" LastActivityDate="2018-05-15T12:29:37.403" Title="CV hyperparameter in sklearn.model_selection.cross_validate" Tags="|machine-learning|scikit-learn|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31682" PostTypeId="2" ParentId="31680" CreationDate="2018-05-15T12:21:05.303" Score="2" Body="&lt;p&gt;I'll answer this first:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;if I am using Shaffle, does it mean that particular row can be in&#xA;  multiple samples and other will not be in any of them&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If I've understood the question correctly, then yes, that is possible. See below:&#xA;from sklearn.model_selection import ShuffleSplit&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn.model_selection import ShuffleSplit&#xA;X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])&#xA;y = np.array([1, 2, 1, 2])&#xA;rs = ShuffleSplit(n_splits=3, test_size=.25, random_state=8)&#xA;rs.get_n_splits(X)&#xA;train = []&#xA;for train_index, test_index in rs.split(X):&#xA;    print(&quot;TRAIN:&quot;, train_index, &quot;TEST:&quot;, test_index)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TRAIN: [1 0 3] TEST: [2]&#xA;TRAIN: [0 3 1] TEST: [2]&#xA;TRAIN: [1 3 0] TEST: [2]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Row &lt;code&gt;2&lt;/code&gt; doesn't appear in any of the training sets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next, to answer your first question:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the mean prediction of those 100 models is the result&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Not quite. From the documentation (which you linked to):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Returns: scores : dict of float arrays of shape=(n_splits,) Array of&#xA;  scores of the estimator for each run of the cross validation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Let's see this in action, on the above example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn linear_model&#xA;from sklearn.model_selection import cross_validate&#xA;lasso = linear_model.Lasso()&#xA;cross_validate(lasso, X, y, cv = rs)['test_score']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returns&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Out[36]: array([ 0.,  0.,  0.])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So you see, it returns an array with the score on each cross-validation fold.&lt;/p&gt;&#xA;" OwnerUserId="52109" LastEditorUserId="52109" LastEditDate="2018-05-15T12:29:37.403" LastActivityDate="2018-05-15T12:29:37.403" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31685" PostTypeId="1" CreationDate="2018-05-15T12:35:13.917" Score="10" ViewCount="26644" Body="&lt;p&gt;I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any standard way of handling this sort of imbalance? &lt;/p&gt;&#xA;" OwnerUserId="49617" LastActivityDate="2019-01-07T12:00:08.737" Title="weighted cross entropy for imbalanced dataset - multiclass classification" Tags="|deep-learning|multiclass-classification|class-imbalance|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31688" PostTypeId="1" CreationDate="2018-05-15T13:49:36.260" Score="-2" ViewCount="49" Body="&lt;p&gt;A little bit of background info,I recently started a new job as a data analyst where I have to use data to provide insights on the IT operations and devops process of my company.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’m fresh out of college, I have a bachelors in engineering, which covered the basics of statistics such as discrete/continuous probability distributions, naïve bayes probability and some hypothesis testing. Regarding data science, I used Splunk to analyze machine data and applied a few basic machine learning models during an internship and taught myself how to use some python libraries such as pandas and numpy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But now I’ve hit a brick wall, I don’t really know how to apply the statistics that I’ve learnt, and how probability distributions and hypothesis testing all gel together to provide insight. My immediate task involves providing some insights, machine learning will come at a later stage and although I am interested, I intend to work on it later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, where do I go from here? Are there any books that I can read that provide examples on how to apply the statistical theories that I’ve learnt? Most ebooks that I’ve found don’t really touch much on this, and most internet articles give examples in a business context, which I don’t find helpful or applicable.&lt;/p&gt;&#xA;" OwnerUserId="47796" LastActivityDate="2018-05-15T14:28:41.967" Title="How to improve when starting out in data science" Tags="|bigdata|data-science-model|data-analysis|" AnswerCount="1" CommentCount="2" ClosedDate="2018-05-16T19:14:13.663" ContentLicense="CC BY-SA 4.0" />
  <row Id="31689" PostTypeId="2" ParentId="31688" CreationDate="2018-05-15T14:28:41.967" Score="1" Body="&lt;p&gt;I tend to recommend &lt;a href=&quot;http://appliedpredictivemodeling.com/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Applied Predictive Modeling&lt;/a&gt;.  It serves as a nice bridge between theory and application.  The organization of the book, which is essentially data prep, pick model, and validate is what one does in the real world (at least at a high level.)  Plus there's plenty of R examples and datasets you can try things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as statistics, I think you just have to go out and learn more of it.  Basic statistics is often insufficient to do anything meaningful in the real world.  The only real way that it comes together is by learning something and then applying it to something.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A comment suggested kaggle.  I personally wouldn't do kaggle for the competitions but rather for the data.  I like to find relevant datasets and use models i'm toying around with as a way to learn more about them.  For example, when I have interns, I give them the zillow housing dataset and show how different generalized linear models work and how each change can improve or decrease model performance.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short, I suggest reading Applied Predictive Modeling, take some courses in Coursera and apply that knowledge to real datasets.  By learning and then doing, you'll be amazed at how quickly you learn and how much easier it becomes to grasp more advance techniques and applications.&lt;/p&gt;&#xA;" OwnerUserId="43294" LastActivityDate="2018-05-15T14:28:41.967" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31691" PostTypeId="1" CreationDate="2018-05-15T14:30:27.563" Score="-2" ViewCount="1123" Body="&lt;p&gt;I have 10k records of data, each record represents a unique product(10k class labels) and its description. For example, &quot;Coffee Maker, this product takes coffee beans and brew it, to make tasty cofe&quot;. Description can be little bigger and smaller. Most of the data also mentions acronyms in the description to refer the product or some other related product and yeah there spelling mistakes too. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So what is the best approach to solve this problem, im open for using Machine Learning/Deep Learning. Please help me on how to build a model, that takes a small description like three or four words or more as a input and suggest a list of products that closely represent those words. &lt;/p&gt;&#xA;" OwnerUserId="52187" LastActivityDate="2018-05-15T15:45:23.533" Title="Best way to extract information from text description and match it with set of words" Tags="|machine-learning|deep-learning|nlp|text-mining|recommender-system|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="31693" PostTypeId="2" ParentId="31657" CreationDate="2018-05-15T14:49:30.303" Score="4" Body="&lt;p&gt;If your goal is to learn how to create new or improved data science methods, then working from scratch may be a good idea. The advantage is that you will learn the detailed inner workings of the algorithms, and you will learn enough to think about new methods or features. Other people may come to know you as an innovator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your goal is to apply data science tools to solve problems, then I encourage you to use published/shared tools such as scikit-learn and others. The advantage is that the code has been examined, tested, debugged, and updates by the community, so the likelihood of hidden errors is decreased. Nothing is perfect, but community code is often closer to perfect than code that is newly made from scratch. Other people may come to know you as an analyst who uses proven, quality methods, and whose results are worthy of trust.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Good luck!&lt;/p&gt;&#xA;" OwnerUserId="52191" LastActivityDate="2018-05-15T14:49:30.303" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31694" PostTypeId="2" ParentId="31634" CreationDate="2018-05-15T14:56:23.347" Score="0" Body="&lt;p&gt;I agree about &quot;it depends&quot; - upon your goal, and upon the nature of the data, and upon how much you and your team know about the data. For text data, the cleaning methods are pretty clear, so I would probably do data cleaning first. For high-volume image data (cancer screening, seismology), there are major trade-offs between data-reduction and feature-detection. In this high-volume domain, I think your pathway would be less clear - UNLESS you or your team know what methods of data-reduction and cleaning have been useful in the past. Domain knowledge can be very important in selecting and formatting the data for your subsequent analyses.&lt;/p&gt;&#xA;" OwnerUserId="52191" LastActivityDate="2018-05-15T14:56:23.347" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31695" PostTypeId="1" AcceptedAnswerId="31697" CreationDate="2018-05-15T15:01:48.643" Score="4" ViewCount="490" Body="&lt;p&gt;I use genetic / evolutionary algorithms in python's &lt;a href=&quot;https://github.com/EpistasisLab/tpot&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;TPOT&quot;&gt;TPOT&lt;/a&gt; package to find the overall best model (GBM, RF, SVM, elastic net, etc) and its tuning parameters. Now I need a way to measure each variable's contribution to the chosen model's predictive performance. How can I do this in a model-agonistic way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current approach is to retrain the best model architecture after holding out each of the variables. For example, if my variables are &lt;code&gt;[a,b,c]&lt;/code&gt; I'll retrain on &lt;code&gt;[a,b]&lt;/code&gt;, &lt;code&gt;[a,c]&lt;/code&gt;,  and &lt;code&gt;[b,c]&lt;/code&gt;. I define the removed variable associated with the worst performing model as the most important variable and I define the variable's predictive contribution as the decrease in predictive performance. I measure all variable's predictive performance this way. Is there anything obviously wrong with this approach? Is there a better approach? I'm familiar with variable importance in decision trees, and p-values in linear models, but I need a model agnostic approach.&lt;/p&gt;&#xA;" OwnerUserId="12515" LastActivityDate="2018-05-15T15:23:29.790" Title="Model-agnostic variable importance metric" Tags="|machine-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31696" PostTypeId="2" ParentId="31485" CreationDate="2018-05-15T15:09:34.073" Score="3" Body="&lt;p&gt;The theory behind PCA involves variance as an important concept. That's not necessarily true for other methods. Note that different methods also have different &quot;figures of merit&quot; for the quality of the outcome. There are many different methods, and you might want to (a) understand what outcomes and quality-metrics are associated with the methods of interest, and (b) develop familiarity with how other people have used and interpreted these quality-metrics. I've made some big mistakes by not understanding properly the interpretive conventions about certain metrics (e.g., stress in multi-dimensional scaling). The community has more wisdom than any one of us - use the community (and contribute back to the community)!&lt;/p&gt;&#xA;" OwnerUserId="52191" LastActivityDate="2018-05-15T15:09:34.073" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31697" PostTypeId="2" ParentId="31695" CreationDate="2018-05-15T15:23:29.790" Score="3" Body="&lt;p&gt;Have you looked at the permutation importance approach in the &lt;a href=&quot;http://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;eli5&lt;/a&gt; package?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is that instead of retraining the model without the feature, which is computationally expensive, they replace each feature in turn with random noise in the test set. To get random noise that is drawn from the same distribution as the original feature, they just randomly shuffle that feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that, as with feature importance in decision trees, this measure is biased against categorical variables with low cardinality. &lt;/p&gt;&#xA;" OwnerUserId="2532" LastActivityDate="2018-05-15T15:23:29.790" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31698" PostTypeId="2" ParentId="31691" CreationDate="2018-05-15T15:45:23.533" Score="0" Body="&lt;p&gt;You can solve this problem using traditional recommendation system algorithms for text. I will show you two ways how this can be implemented, one using a simple linear search and the other using some clustering approach. First, you will need to vectorize your descriptions.&lt;/p&gt;&#xA;&lt;h1&gt;Feature extraction&lt;/h1&gt;&#xA;&lt;p&gt;You will build a dictionary of existing words or character sequences and then you will fill this vector with the number of occurances in your short description. There are two techniques, I would recommend for this, the first is bag-of-words, don't forget to only use the stem of words since coffee should be the same as coffees. The second is n_grams which is sequences of characters.&lt;/p&gt;&#xA;&lt;h2&gt;n-grams&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;n-grams&lt;/strong&gt; is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...&lt;/p&gt;&#xA;&lt;p&gt;The segmentation works as follows:&lt;/p&gt;&#xA;&lt;p&gt;The String: Hello World&lt;/p&gt;&#xA;&lt;p&gt;2-gram: &amp;quot;He&amp;quot;, &amp;quot;el&amp;quot;, &amp;quot;ll&amp;quot;, &amp;quot;lo&amp;quot;, &amp;quot;o &amp;quot;, &amp;quot; W&amp;quot;, &amp;quot;Wo&amp;quot;, &amp;quot;or&amp;quot;, &amp;quot;rl&amp;quot;, &amp;quot;ld&amp;quot;&#xA;3-gram: &amp;quot;Hel&amp;quot;, &amp;quot;ell&amp;quot;, &amp;quot;llo&amp;quot;, &amp;quot;lo &amp;quot;, &amp;quot;o W&amp;quot;, &amp;quot; Wo&amp;quot;, &amp;quot;Wor&amp;quot;, &amp;quot;orl&amp;quot;, &amp;quot;rld&amp;quot;&#xA;4-gram: &amp;quot;Hell&amp;quot;, &amp;quot;ello&amp;quot;, &amp;quot;llo &amp;quot;, &amp;quot;lo W&amp;quot;, &amp;quot;o Wo&amp;quot;, &amp;quot; Wor&amp;quot;, &amp;quot;Worl&amp;quot;, &amp;quot;orld&amp;quot;&lt;/p&gt;&#xA;&lt;p&gt;Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.&lt;/p&gt;&#xA;&lt;h2&gt;Bag-of-words&lt;/h2&gt;&#xA;&lt;p&gt;This builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;For example: If the dictionary includes the words {car, coffee, motel, hotel, world, van, soup} and we have the description &amp;quot;the soup in this motel is amazing!&amp;quot;. Then the resulting vector would be &lt;span class=&quot;math-container&quot;&gt;$[0, 0, 1, 0, 0, 0, 1]$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h1&gt;Applying your search&lt;/h1&gt;&#xA;&lt;p&gt;You will now have a vector representation of each of your descriptions. Then for a search term like &amp;quot;yummy food motel&amp;quot;, you can vectorize it as &lt;span class=&quot;math-container&quot;&gt;$[0, 0, 1, 0, 0, 0, 0]$&lt;/span&gt;, then find the most similar instance in your set to this vector. You can use the Euclidean distance as a metric of similarity. This method can be considered a k-nearest neighbors (k-NN) approach.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-15T15:45:23.533" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31700" PostTypeId="1" CreationDate="2018-05-15T16:53:52.527" Score="2" ViewCount="11295" Body="&lt;p&gt;Data-set has 3 features. The number of clusters are two.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;I am figuring out how to print clusters using scatter plot for the data having 3 feature column and clustered into 2 clusters using kmeans.&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The train data is in dataframe format and data is of activity data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X =  pd.concat([train_data['start'], train_data['end'], train_data  ['duration']], axis=1)  &#xA;kmeans.fit(X)  &#xA;Y_pred = kmeans.predict(X)  &#xA;plt.scatter(X.iloc[:,0], X.iloc[:,1], c=Y_pred, cmap=plt.cm.Paired)  &#xA;plt.legend()  &#xA;plt.title('train data')  &#xA;plt.show()  &#xA;getting following output:&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xUe3n.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xUe3n.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50829" LastEditorUserId="35644" LastEditDate="2018-05-15T17:53:44.823" LastActivityDate="2018-05-15T17:53:44.823" Title="how to print kmeans cluster python" Tags="|python|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31701" PostTypeId="2" ParentId="31700" CreationDate="2018-05-15T17:02:32.400" Score="3" Body="&lt;p&gt;&lt;strong&gt;Code Is Self Explanatory...&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    from sklearn.cluster import KMeans&#xA;    from sklearn.datasets.samples_generator import make_blobs&#xA;    np.random.seed(0)&#xA;    centers = [[1, 1], [-1, -1]]&#xA;    n_clusters = len(centers)&#xA;    X, labels_true = make_blobs(n_samples=3000, &#xA;                                centers=centers, &#xA;                                cluster_std=0.5)&#xA;&#xA;def plot_cluster_data(X, c=[1]*X.shape[0], mu=None):&#xA;    fig = plt.figure(figsize=(8, 8))&#xA;    ax = fig.add_subplot(1, 1, 1)&#xA;    if len(np.unique(c)) == 1:&#xA;        ax.plot(X[:,0], X[:,1], 'o')&#xA;    else:&#xA;        ix = np.where(c==1)&#xA;        ax.plot(X[ix,0], X[ix,1], 'o', &#xA;                markerfacecolor='red')&#xA;        ax.plot(mu[0,0], mu[0,1], 'o', &#xA;                markerfacecolor='red', &#xA;                markersize=12)&#xA;        ix = np.where(c==0)&#xA;        ax.plot(X[ix,0], X[ix,1], 'o', &#xA;                markerfacecolor='green')&#xA;        ax.plot(mu[1,0], mu[1,1], 'o', &#xA;                markerfacecolor='green', &#xA;                markersize=12)&#xA;    if not mu is None:&#xA;        ax.plot(mu[0,0], mu[0,1], 'o', &#xA;                markerfacecolor='red', &#xA;                markersize=12)&#xA;        ax.plot(mu[1,0], mu[1,1], 'o', &#xA;                markerfacecolor='green', &#xA;                markersize=12)        &#xA;    plt.show()&#xA;&#xA;plot_cluster_data(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ut3yC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ut3yC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;clst = KMeans(n_clusters=2, random_state=2342)&#xA;clst.fit(X)&#xA;mu = clst.cluster_centers_&#xA;plot_cluster_data(X, mu = mu)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Od7mk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Od7mk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-05-15T17:02:32.400" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31702" PostTypeId="2" ParentId="31700" CreationDate="2018-05-15T17:07:53.413" Score="2" Body="&lt;h1&gt;3D plot&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;3 features indicates that your data is 3 dimensional. Thus you can use a 3D plot. The following code will plot 3 dimensional data. $x$ is a numpy matrix with the 3 features as columns, and the rows are the instances. Then $y$ is the cluster label that you obtain from k-means.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import matplotlib.pyplot as plt&#xA;from mpl_toolkits.mplot3d import Axes3D&#xA;import numpy as np&#xA;from numpy import where&#xA;&#xA;# Plots 2 features, with an output, shows the decision boundary&#xA;def plot3D(x, y):&#xA;    fig = plt.figure()&#xA;    ax = fig.add_subplot(111, projection='3d')&#xA;    pos = where(y == 1)&#xA;    neg = where(y == 0)&#xA;&#xA;    color = ['r', 'b', 'y', 'k', 'g', 'c', 'm']&#xA;&#xA;    for i in range(30):&#xA;        ax.scatter(x[i, 0], x[i, 1],x[i, 2], marker='o', c=color[int(y[i])-1])&#xA;    #ax.scatter(x[:,1], x[:,2], y)&#xA;    ax.set_xlabel('X Label')&#xA;    ax.set_ylabel('Y Label')&#xA;    ax.set_zlabel('Z Label')&#xA;&#xA;    axes = plt.axis()&#xA;    plt.show()&#xA;&#xA;plot3D(X, cluster_labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This will give you the following plot&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/2G7f7.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/2G7f7.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;2D plot&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Alternatively you can project the data into two dimensions. You can do this naively by collapsing any of the three dimensions. For example this will only show the first 2 features, the third would be projected onto the plane of the first and second feature.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(X[:,0], X[:,1], c=cluster_labels)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You can also plot the 2nd and 3rd features, where the first feature is projected as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(X[:,1], X[:,2], c=cluster_labels)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Projecting data naively can lead to problems so instead you can use a feature embedding method. Here I will give an example for 4 different methods: Isomap, MDS, spectral embedding and TSNE (my favorite).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is continuous data that I have access to but you can easily do the same for clustered data. Just set the labels $y$ as your determined clusters.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_boston&#xA;from sklearn.linear_model import LinearRegression&#xA;from sklearn.model_selection import train_test_split&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;from sklearn.manifold import TSNE, SpectralEmbedding, Isomap, MDS&#xA;&#xA;boston = load_boston()&#xA;X = boston.data&#xA;Y = boston.target&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)&#xA;&#xA;# Embed the features into 2 features using TSNE&#xA;X_embedded_iso  = Isomap(n_components=2).fit_transform(X)&#xA;X_embedded_mds  = MDS(n_components=2, max_iter=100, n_init=1).fit_transform(X)&#xA;X_embedded_tsne = TSNE(n_components=2).fit_transform(X)&#xA;X_embedded_spec = SpectralEmbedding(n_components=2).fit_transform(X)&#xA;&#xA;print('Description of the dataset: \n')&#xA;&#xA;print('Input shape : ', X_train.shape)&#xA;print('Target shape: ', y_train.shape)&#xA;&#xA;print('Embed the features into 2 features using Spectral Embedding: ', X_embedded_spec.shape)&#xA;print('Embed the features into 2 features using TSNE: ', X_embedded_tsne.shape)&#xA;&#xA;fig = plt.figure(figsize=(12,5),facecolor='w')&#xA;plt.subplot(1, 2, 1)&#xA;plt.scatter(X_embedded_iso[:,0], X_embedded_iso[:,1], c = Y, cmap = 'hot')&#xA;plt.title('2D embedding using Isomap \n The color of the points is the price')&#xA;plt.xlabel('Feature 1')&#xA;plt.ylabel('Feature 2')&#xA;plt.colorbar()&#xA;plt.tight_layout()&#xA;&#xA;plt.subplot(1, 2, 2)&#xA;plt.scatter(X_embedded_mds[:,0], X_embedded_mds[:,1], c = Y, cmap = 'hot')&#xA;plt.title('2D embedding using MDS \n The color of the points is the price')&#xA;plt.xlabel('Feature 1')&#xA;plt.ylabel('Feature 2')&#xA;plt.colorbar()&#xA;plt.show()&#xA;plt.tight_layout()&#xA;&#xA;fig = plt.figure(figsize=(12,5),facecolor='w')&#xA;plt.subplot(1, 2, 1)&#xA;plt.scatter(X_embedded_spec[:,0], X_embedded_spec[:,1], c = Y, cmap = 'hot')&#xA;plt.title('2D embedding using Spectral Embedding \n The color of the points is the price')&#xA;plt.xlabel('Feature 1')&#xA;plt.ylabel('Feature 2')&#xA;plt.colorbar()&#xA;plt.tight_layout()&#xA;&#xA;plt.subplot(1, 2, 2)&#xA;plt.scatter(X_embedded_tsne[:,0], X_embedded_tsne[:,1], c = Y, cmap = 'hot')&#xA;plt.title('2D embedding using TSNE \n The color of the points is the price')&#xA;plt.xlabel('Feature 1')&#xA;plt.ylabel('Feature 2')&#xA;plt.colorbar()&#xA;plt.show()&#xA;plt.tight_layout()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wypl0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wypl0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/FGaak.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/FGaak.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-15T17:07:53.413" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31703" PostTypeId="1" AcceptedAnswerId="31714" CreationDate="2018-05-15T17:47:51.327" Score="0" ViewCount="194" Body="&lt;p&gt;Question RE this research paper if anyone has experience with CNN's, pooling &amp;amp; skip connections: &lt;a href=&quot;https://arxiv.org/pdf/1606.01781.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1606.01781.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;In figure 1, the input to the first convolutional block has shape (batch_size, 64, s)&lt;/li&gt;&#xA;&lt;li&gt;The output from block 1 must be (batch_size, 64, s)&lt;/li&gt;&#xA;&lt;li&gt;The output from block 2 must be (batch_size, 64, s)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However the output from the pooling step has shape (batch_size, 128, s/2). How can a pooling step increase the number of parameters on axis 1 from 64 to 128?!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My guess is that the input to the pooling layer is actually a concatenation of 2 of the previous layer's outputs. In this case it would have input shape (batch_size, 128, s). However, the paper does not appear to clearly specify at what outputs are concatenated...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone clarify how this is the case?&lt;/p&gt;&#xA;" OwnerUserId="50025" LastActivityDate="2018-05-16T06:45:21.263" Title="Very Deep Convolutional Networks for Text Classification: Clarifying skip connections" Tags="|deep-learning|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31705" PostTypeId="2" ParentId="31156" CreationDate="2018-05-15T20:36:36.040" Score="1" Body="&lt;p&gt;Kaggle is more contest platform with a jobs board more than a freelance platform. Try upwork and freelancer.com. They are the ebay of freelancing and they support a wide range of payments although some of their fees have been going up for a while and search is favoring fees over matching to best results. Still you will be able to sort by skills like visualization, machine learning, data cleaning. It will depend on what skills you have to offer. I'm sure there are others better out there and probably others without too much traction. AngelList also offer contracts for data scientists if you're looking for something more structured. &lt;/p&gt;&#xA;" OwnerUserId="13023" LastActivityDate="2018-05-15T20:36:36.040" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31706" PostTypeId="2" ParentId="31685" CreationDate="2018-05-15T21:43:52.750" Score="3" Body="&lt;p&gt;To handle class imbalance, do nothing -- use the ordinary cross-entropy loss, which handles class imbalance about as well as can be done.  Make sure you have enough instances of each class in the training set, otherwise the neural network might not be able to learn: neural networks often need a lot of data.  Assuming you care about global accuracy (rather than the average of the accuracy on each individual class, say), I wouldn't bother with a weighted cross-entropy loss or duplicating images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your training sounds rather small.  To deal with that, you might try starting from an existing pre-trained model and fine-tune the last few layers.  Also use image augmentation.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastEditorUserId="8560" LastEditDate="2018-05-16T15:07:38.850" LastActivityDate="2018-05-16T15:07:38.850" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="31707" PostTypeId="1" CreationDate="2018-05-15T21:58:48.927" Score="2" ViewCount="658" Body="&lt;p&gt;I am trying to use DBSMOTE(Density-Based Synthetic Oversampling TEqnique) to on a data set of short text--tweets to be specific. This will be used to train a classifier model in a multiclass classification model. This will be done at feature level augmentation, using TF-IDF as features. I have read though, that to use SMOTE on NLP, the feature vectors must be reduced in dimension. What is an optimal size of a feature vector to use in a SMOTE family algorithm? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similar Question: &lt;a href=&quot;https://datascience.stackexchange.com/questions/27671/how-do-you-apply-smote-on-text-classification?rq=1&quot;&gt;How do you apply SMOTE on text classification?&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DBSMOTE Code: &lt;a href=&quot;https://rdrr.io/cran/smotefamily/man/DBSMOTE.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://rdrr.io/cran/smotefamily/man/DBSMOTE.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DBSMOTE Paper: &lt;a href=&quot;https://link.springer.com/article/10.1007/s10489-011-0287-y&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://link.springer.com/article/10.1007/s10489-011-0287-y&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52207" LastEditorUserId="52207" LastEditDate="2018-05-15T22:57:37.327" LastActivityDate="2018-05-15T22:57:37.327" Title="DBSMOTE on Short Text Classification" Tags="|nlp|feature-extraction|tfidf|smote|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31708" PostTypeId="2" ParentId="26411" CreationDate="2018-05-15T22:26:11.417" Score="2" Body="&lt;p&gt;You should check if there are any among the features, after adding the new ones. After that, you have to check if each feature is informative taking into account the target variable of the problem. For the latter you can perform a statistical test and check the p-value to determine if there is statistical significance, otherwise you could compute the Mutual Information between each new feature and the target variable. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="29575" LastEditDate="2018-07-14T17:54:53.310" LastActivityDate="2018-07-14T17:54:53.310" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31709" PostTypeId="1" AcceptedAnswerId="93190" CreationDate="2018-05-15T23:02:28.800" Score="7" ViewCount="2715" Body="&lt;p&gt;The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc.  The Adam optimizer is an improvement on gradient descent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a situation where I want to use &lt;a href=&quot;https://www.stats.ox.ac.uk/~lienart/blog_opti_pgd.html&quot; rel=&quot;noreferrer&quot;&gt;projected gradient descent&lt;/a&gt; (see also &lt;a href=&quot;https://math.stackexchange.com/q/571068/14578&quot;&gt;here&lt;/a&gt;).  Basically, instead of trying to minimize a function $f(x)$, I want to minimize $f(x)$ subject to the requirement that $x \ge 0$.  Projected gradient descent works by clipping the value of $x$ after each iteration of gradient descent: each negative entry is replaced with 0, after each step.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unfortunately, projected gradient descent seems to interact poorly with the Adam optimizer.  I'm guessing that Adam's exponential moving average of the gradients gets messed up by the clipping.  And plain projected gradient descent has hyperparameters that can be tuned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a version of Adam that can be used with projected gradient descent?  I'm looking for a method that is an improvement on projected gradient descent, in the same way that Adam is an improvement on ordinary gradient descent (e.g., doesn't require hyperparameter tuning).  Is there any such algorithm?&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2021-04-22T11:40:59.063" Title="Adam optimizer for projected gradient descent" Tags="|neural-network|optimization|gradient-descent|momentum|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31711" PostTypeId="2" ParentId="31584" CreationDate="2018-05-16T00:46:40.843" Score="1" Body="&lt;p&gt;It depends on the signal-to-noise in the dataset. The amount of data to perform named entity disambiguation will depend on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Tf%E2%80%93idf&quot; rel=&quot;nofollow noreferrer&quot;&gt;tf-idf score&lt;/a&gt; of the occupation and skills, rare occupations and skills will require less data to build a performant model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, that the sentence &quot;I am a cook that multitasks well.&quot; &quot;Cook&quot; is an occupation and &quot;multitask&quot; is a relevant skill. In a similar sentence, &quot;I multitasked while I cooked.&quot; &quot;Cook&quot; is no longer an occupation and &quot;multitask&quot; is no longer a relevant skill. However, the phrase &quot;saturation diver&quot; is less frequent than &quot;cook&quot;, thus much easier to build a model to identify as an occupation and find relevant skills.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Annotator performance is easier to measure. &lt;a href=&quot;https://en.wikipedia.org/wiki/Cohen%27s_kappa&quot; rel=&quot;nofollow noreferrer&quot;&gt;Cohen's kappa&lt;/a&gt; is a common method of judging &lt;a href=&quot;https://en.wikipedia.org/wiki/Inter-rater_reliability&quot; rel=&quot;nofollow noreferrer&quot;&gt;inter-rater reliability&lt;/a&gt;. Again, the number of needed raters depends on their agreement on the task. If task performance is easy, the number of raters and the number of items per rater can be lower. It is best to benchmark your system and then decide how much data you need to raise the benchmark scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way to automatically create ontologies from a text is the &lt;a href=&quot;http://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;TextRank&lt;/a&gt; algorithm.  &lt;/p&gt;&#xA;" OwnerUserId="1330" LastEditorUserId="1330" LastEditDate="2018-05-16T04:36:27.657" LastActivityDate="2018-05-16T04:36:27.657" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31712" PostTypeId="2" ParentId="9663" CreationDate="2018-05-16T01:02:33.560" Score="0" Body="&lt;p&gt;You might try using word vectors to average the top N words in a topic and then using the cosine similarity to find the closest word in the corpus?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just a quick and dirty an idea...&lt;/p&gt;&#xA;" OwnerUserId="52213" LastActivityDate="2018-05-16T01:02:33.560" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31713" PostTypeId="2" ParentId="31617" CreationDate="2018-05-16T06:06:18.620" Score="21" Body="&lt;p&gt;If you want to tackle the problem from another perspective, with &lt;em&gt;an end to end learning&lt;/em&gt;, such that you don't specify ahead of time this large pipeline you've mentioned earlier, all you care about is the mapping between sentences and their corresponding SQL queries. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Tutorials:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.einstein.ai/how-to-talk-to-your-database&quot; rel=&quot;noreferrer&quot;&gt;How to talk to your database&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Papers:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1709.00103&quot; rel=&quot;noreferrer&quot;&gt;Seq2SQL: Generating Structured Queries from&#xA;Natural Language using Reinforcement Learning&lt;/a&gt;  salesforce&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.ijcai.org/Proceedings/16/Papers/329.pdf&quot; rel=&quot;noreferrer&quot;&gt;Neural Enquirer: Learning to Query Tables in Natural Language&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Dataset:&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://github.com/salesforce/WikiSQL&quot; rel=&quot;noreferrer&quot;&gt;A large annotated semantic parsing corpus for developing natural language interfaces.&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Github code:&lt;/strong&gt;   &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/dadashkarimi/seq2sql&quot; rel=&quot;noreferrer&quot;&gt;seq2sql&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://github.com/xiaojunxu/SQLNet&quot; rel=&quot;noreferrer&quot;&gt;SQLNet&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Also, there are commercial solutions like &lt;a href=&quot;https://www.nlsql.com/&quot; rel=&quot;noreferrer&quot;&gt;nlsql&lt;/a&gt;   &lt;/p&gt;&#xA;" OwnerUserId="44039" LastEditorUserId="41484" LastEditDate="2020-01-06T17:14:17.487" LastActivityDate="2020-01-06T17:14:17.487" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31714" PostTypeId="2" ParentId="31703" CreationDate="2018-05-16T06:45:21.263" Score="0" Body="&lt;p&gt;As you mentioned, the paper doesn't clarify. However, my guess is that this is not due to concatenating 2 previous layers (I don't really see an specific reason to do this here) but because of concatenating the &lt;em&gt;ResNet shorcut&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Generally, all conv layers have a number of filters, thus determining the output size (&lt;em&gt;num_filters&lt;/em&gt;, &lt;em&gt;size&lt;/em&gt;) regardless the inputs. On the other hand, MaxPooling does keep the input &lt;em&gt;num_filters&lt;/em&gt; (though in this case reducing the &lt;em&gt;size&lt;/em&gt;). In the paper, note that the &lt;em&gt;num_filters&lt;/em&gt; is doubled at the output of all convlayer except for the one that does not keep the &lt;em&gt;ResNet shorcut&lt;/em&gt; (last 512 conv layer). So my guess is that they are concatenating the output of the conv layer and the shorcut which would explain the output size.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps!&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-05-16T06:45:21.263" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31715" PostTypeId="1" CreationDate="2018-05-16T07:50:03.820" Score="3" ViewCount="2153" Body="&lt;p&gt;I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing,&#xA;I would like to predict errors or failures but till now they have not been recorded yet. So I thought to using these time series to understand if there are abnormal behaviours (which could lead to a faillure), maybe clustering time series. Beacuse I suppose that the machine, in normal condition, has the same parameters for each work.&#xA;Is there a procedure/documentation/ best practises in order to do anomaly detecion with time series?&#xA;Thanks a lot!&lt;/p&gt;&#xA;" OwnerUserId="50074" LastActivityDate="2020-01-11T21:01:00.373" Title="Anomaly detection with time series" Tags="|time-series|anomaly-detection|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31719" PostTypeId="1" CreationDate="2018-05-16T08:38:32.650" Score="0" ViewCount="101" Body="&lt;p&gt;In RapidMiner I want to create a k-NN model in order to create a classifier. To generate the test sets and the training sets I use the the cross-validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I choose 10 as the number of folds, the cross-validation algorithm will do 10 iterations with different test sets and training sets. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the end, what training set will be used to create my k-NN model ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="26562" LastEditDate="2018-05-16T09:04:18.117" LastActivityDate="2018-05-16T12:35:05.067" Title="Cross Validation and training set" Tags="|classification|data-mining|cross-validation|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31724" PostTypeId="1" CreationDate="2018-05-16T09:38:43.770" Score="1" ViewCount="73" Body="&lt;p&gt;How to read factor vs. factor plots?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Particularly, what do negative values mean? Also, why is it possible for features to have negative effect on the factors?&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-16T09:38:43.770" Title="How to read factor vs. factor plots?" Tags="|pca|plotting|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31725" PostTypeId="1" AcceptedAnswerId="31727" CreationDate="2018-05-16T10:06:21.123" Score="0" ViewCount="819" Body="&lt;p&gt;Before I was learning about gradient descent, but now I understand this. Now, I have a problem with the backpropagation algorithm. I know the idea - minimalize error in multilayer neural network using chain rule. However, I don't understand the role of the derivative of the sigmoid function. This derivative is described in the algorithm. What is the point of this? Can you explain this step by step using simple language?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H5usA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H5usA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49067" LastEditorUserId="29169" LastEditDate="2019-06-09T15:17:32.803" LastActivityDate="2019-06-09T15:17:32.803" Title="How to understand backpropagation using derivative" Tags="|machine-learning|neural-network|backpropagation|" AnswerCount="1" CommentCount="2" ClosedDate="2018-05-16T19:14:06.553" ContentLicense="CC BY-SA 4.0" />
  <row Id="31727" PostTypeId="2" ParentId="31725" CreationDate="2018-05-16T10:30:41.617" Score="3" Body="&lt;p&gt;This is a continuation to &lt;a href=&quot;https://datascience.stackexchange.com/questions/30676/role-derivative-of-sigmoid-function-in-neural-networks/30679#30679&quot;&gt;this answer&lt;/a&gt;. The first part of that answer covered gradient descent. In short this algorithm finds a set of parameters which results in a minimum of a function. From the description of the algorithm you can see where the derivative of the function is needed.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How this applies to a neural network&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Let's first consider a single neuron network. This is called the perceptron. We will use the sigmoid activation function. It has inputs $x$ and an associated output &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\hat{y} = \frac{1}{1+e^{w^Tx+b}}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The model is parametrized by the weights $w$ and the biases. These are the model parameters we need to tune. So this single neuron takes a vector $x$ and using the weights vector and biases will output some value $y$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the weights are randomly chosen then the output will also be random. This is of no use to us. We need to determine a way of calculating a measure of correctness or wrongness in order to know how we should change our weights. I will use a measure of how wrong we are and call this the &lt;strong&gt;cost&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is takes the squared difference between the predicted output $\hat{y}$ and the ground truth $y$. We will take the sum of this wrongness for all the instances that we try to predict, hence the summation. And we divide by $N$ to normalize the error. The additional division by 2 is simply to make the derivative easier to compute you will see soon.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we want to &lt;strong&gt;minimize&lt;/strong&gt; the cost function using gradient decent. Thus we need to take the derivative of the cost function with respect to the parameters we can tune, these being the weights $w$. We must thus compute $\frac{\partial C}{\partial w}$. Using calculus we can use chain rule in order to simplify our derivative as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our gradient descent calculation thus becomes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$w^{new} = w^{old} - \nu \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\nu$ is the learning rate.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Let's explicitly spell out this derivative&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and, the second is the derivative of the activation function which we are using which is the sigmoid function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-16T10:30:41.617" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31729" PostTypeId="1" AcceptedAnswerId="31959" CreationDate="2018-05-16T11:20:22.857" Score="2" ViewCount="5557" Body="&lt;p&gt;I have studied &lt;a href=&quot;https://link.springer.com/content/pdf/10.1023/A:1010933404324.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Random Forest&lt;/a&gt; and &lt;a href=&quot;http://www.dsi.unive.it/~dm/gehrke98rainforest.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;RainForest&lt;/a&gt; papers, but they are a bit confusing! In summary, I understand following steps for these algorithms. Could you help me to find out if I am right or not? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate your help.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Random forest first:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;define number of trees&lt;/li&gt;&#xA;&lt;li&gt;partition data by bootstrapping&lt;/li&gt;&#xA;&lt;li&gt;on each partition construct trees (in each node a sub sample of features is selected)&lt;/li&gt;&#xA;&lt;li&gt;label leaf nodes&lt;/li&gt;&#xA;&lt;li&gt;for classifying a  new instance vote over all trees.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In RainForest:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Partition dataset &lt;/li&gt;&#xA;&lt;li&gt;Build AVC-set of a partition&lt;/li&gt;&#xA;&lt;li&gt;Build tree over the partition by computing a purity criterion (like gini-index) over AVC-sets&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23857" LastActivityDate="2018-06-18T10:00:04.913" Title="Random Forest vs. RainForest" Tags="|classification|random-forest|decision-trees|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31730" PostTypeId="1" AcceptedAnswerId="34242" CreationDate="2018-05-16T11:25:47.463" Score="3" ViewCount="2604" Body="&lt;p&gt;I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. It contains only numbers. I want to generate the same input using AutoEncoders but I didn't find any useful link that I can use for that. The ones that I used had some dimension problems when I use my data. Does Anyone recommend any helpful tutorial ? &lt;/p&gt;&#xA;" OwnerUserId="44240" LastActivityDate="2018-07-12T07:08:19.413" Title="Is there any clear tutorial for how to use AutoEncoders with text as input" Tags="|keras|pandas|lstm|autoencoder|text|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31731" PostTypeId="1" CreationDate="2018-05-16T12:15:02.730" Score="1" ViewCount="162" Body="&lt;p&gt;Let this, be a dataset labeling the head pose for each photo:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;image_path, head_pose&#xA;---------------------&#xA;michael/01-right.png, right&#xA;donie/01-right.png, right&#xA;lea/01-left.png, left&#xA;jack/01-up.png, up&#xA;(...)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The dataset is too big to be load at once but a data frame containing the above information is manageable (ie, not loading the images). Using Keras, I was thinking of creating my own &lt;code&gt;DataGenerator&lt;/code&gt; in order to load the images batch by batch.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then using &lt;code&gt;ImageDataGenerator&lt;/code&gt; I can augment each batch. But I'd like to know when a horizontal flip is done because I want to change the associated prediction. Like if the person was facing her left, the expected class should be right and left if the image is flipped.  Maybe &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/preprocessing/image.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;extending&lt;/a&gt; the &lt;code&gt;ImageDataGenerator&lt;/code&gt; ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the best way to handle that with Keras ?&lt;/p&gt;&#xA;" OwnerUserId="9461" LastEditorUserId="9461" LastEditDate="2018-05-17T07:34:39.987" LastActivityDate="2018-05-17T07:34:39.987" Title="Data augmentation from directory and oriented features with Keras" Tags="|keras|data-augmentation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31732" PostTypeId="1" CreationDate="2018-05-16T12:20:08.490" Score="2" ViewCount="1081" Body="&lt;p&gt;I'm a bit confused about how to go about plotting a 3-axis bar chart:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Bugds.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Bugds.jpg&quot; alt=&quot;3 Axis Chart&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my jupyter notebook reads in an excel/sheet and I have a table:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;     2001  2002   2003   2004&#xA;Mar  15    16     14     18&#xA;Jun  23    25     28     24&#xA;Jul  24    23     22     26&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm a bit confused about how to go about building the visualization and having the axes/labels set up correctly.&lt;/p&gt;&#xA;" OwnerUserId="52237" LastActivityDate="2021-09-13T02:04:10.340" Title="How to plot a 3-axis bar chart with matplotlib (and pandas + jupyter)" Tags="|pandas|jupyter|matplotlib|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31733" PostTypeId="2" ParentId="31719" CreationDate="2018-05-16T12:35:05.067" Score="1" Body="&lt;p&gt;Apart from having an average accuracy of the trained model, also you can use cross-validation to approximate the optimal value for the k parameter and the optimal metric.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-16T12:35:05.067" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31734" PostTypeId="2" ParentId="31633" CreationDate="2018-05-16T12:46:56.947" Score="0" Body="&lt;p&gt;As per &lt;a href=&quot;https://docs.orange.biolab.si/3/visual-programming/widgets/data/featureconstructor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Constructor widget documentation&lt;/a&gt;, the expression field should return values from $[0, n]$, and the Values field should contain $n+1$ comma-separated labels.&lt;/p&gt;&#xA;" OwnerUserId="15527" LastActivityDate="2018-05-16T12:46:56.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31736" PostTypeId="1" CreationDate="2018-05-16T13:34:28.793" Score="3" ViewCount="18681" Body="&lt;p&gt;Can you please help me add error bars to my graph?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the csv:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;run,testcase,algorithm,group,avg_weightedcost,std_weight&#xA;1,1,QI,0,20007037.36,0&#xA;2,1,Q2,0,60000000,3.76E-09&#xA;3,1,Q4,0,181801581.2,13353630.74&#xA;4,1,Q3,0,585605657.3,54852458.59&#xA;6,1,QI,1,10003518.68,0&#xA;7,1,Q2,1,292802828.7,2.00E+01&#xA;8,1,Q4,1,90900790.6,13353630.74&#xA;9,1,Q3,1,292802828.7,27426229.3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the code and graph generated:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.barplot(y=&quot;algorithm&quot;, x=&quot;avg_weightedcost&quot;, hue=&quot;group&quot;, &#xA;capsize=.2, data=df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ch3qJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ch3qJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how I tried to add the error bar:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.barplot(y=&quot;algorithm&quot;, x=&quot;avg_weightedcost&quot;, hue=&quot;group&quot;, &#xA;xerr=&quot;std_weight&quot;, capsize=.2, data=df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: err must be [ scalar | N, Nx1 or 2xN array-like ]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Modified:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;csv file:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;run,testcase,algorithm,group,avg_weightedcost,std_weight,avg,err&#xA;1,1,QI,0.00,20007037.36,0.00,100.00,5.00&#xA;2,1,Q2,0.00,60000000.00,0.00,50.00,20.00&#xA;3,1,Q4,0.00,181801581.20,13353630.74,50.00,10.00&#xA;4,1,Q3,0.00,585605657.30,54852458.59,20.00,1.00&#xA;6,1,QI,1.00,10003518.68,0.00,20.00,20.00&#xA;7,1,Q2,1.00,292802828.65,20.00,30.00,10.00&#xA;8,1,Q4,1.00,90900790.60,13353630.74,10.00,10.00&#xA;9,1,Q3,1.00,292802828.65,27426229.30,50.00,20.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.barplot(x=data2['avg_weightedcost'], y=data2['algorithm'], &#xA;hue=data2['group']) &#xA;&#xA;g.errorbar(x=data2['avg_weightedcost'], y=data2['algorithm'], &#xA;xerr=data2['std_weight'], ecolor='red', linewidth=0, capsize=15) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ValueError: could not convert string to float: 'Q3'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52241" LastEditorUserId="52241" LastEditDate="2018-05-16T16:49:33.297" LastActivityDate="2020-01-17T08:04:20.753" Title="Unable to generate error bars with seaborn" Tags="|python|graphs|seaborn|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31737" PostTypeId="2" ParentId="12721" CreationDate="2018-05-16T13:54:54.143" Score="4" Body="&lt;p&gt;As an extreme case, I had a chance to study on Forex (Foreign Exchange Rate) forecast and intensively compared performances of LSTM, windowed-MLP and ARIMA. As many articles say, Forex time series is close to the random walk series (it is completely non-stationary). None of these algorithms can predict next day's spot rate. For example, if there is no (or little) change, then it will maintain current value and it looks fit. However, if there is a sudden (substantial) change in tomorrow's spot rate, then it always fails to predict. The problem is that there is nothing to learn from the training data (say, past 10 years' spot rates history) whether it is large enough or not, because it is completely random (perhaps, due to exchange dealers' sentiments ...).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In fact, LSTM with just '1' lag, or ARIMA with (1,0,0) will perform the best since storing data with more than 1 time lag doesn't help at all. For this kind of time series, I would say, in terms of tracking performance, 1) ARIMA, 2) LSTM, 3) MLP. Last comment... MLP with macro/micro economic parameters (say, GDP difference, bond rates, crude oil price, dollar index, etc) as additional features didn't work at all also.&lt;/p&gt;&#xA;" OwnerUserId="52243" LastActivityDate="2018-05-16T13:54:54.143" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31738" PostTypeId="2" ParentId="31736" CreationDate="2018-05-16T14:05:56.463" Score="4" Body="&lt;p&gt;Try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;g = sns.barplot(y=&quot;algorithm&quot;, x=&quot;avg_weightedcost&quot;, hue=&quot;group&quot;, &#xA;    xerr=df[std_weight]*1, capsize=.2, data=df)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope it helps&lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-16T14:05:56.463" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31740" PostTypeId="1" AcceptedAnswerId="31743" CreationDate="2018-05-16T14:36:22.320" Score="1" ViewCount="1000" Body="&lt;p&gt;I'm struggling to write some tensor manipulation code for a custom loss function I'm using in Keras.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, I'm trying to modify a binary_crossentropy loss by adding a weight that is calculated from a particular feature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First thing I do is pass in my extra feature data to the custom loss by appending it to the y_true like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_trainModded = numpy.append(y_train, MyExtraData, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which is then passed to the fit function like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit(X_train, y_trainModded, epochs=2500, .....)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then extracted to make it usable like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def myCustomLoss(data, y_pred):&#xA;    y_true = data[:,:2]&#xA;    MyExtraData = data[:,2]&#xA;    ...&#xA;    ...&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So far, that all works fine.  However, I'm struggling with a section where I want to only select the MyExtraData where I predicted '1'.  Intuitively, this would simply be something like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ExtraDataWherePredicted1 = MyExtraData[y_pred &amp;gt; 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;However, we're dealing with tensors, not numpy arrays. I tried casting to numpy arrays using eval(), but that didn't work. I also tried various approaches using keras.backend operations such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;WherePredicted1 = K.greater( y_pred,0)&#xA;ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which I could then use to weight my loss such as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return K.mean(K.binary_crossentropy(y_pred,y_true), axis=-1)-(K.mean(ExtraDataWherePredicted1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But anything I try throws out various errors...I just can't figure out how to calculate &lt;em&gt;ExtraDataWherePredicted1&lt;/em&gt;. I'm also finding it super hard to debug the loss function because I can't print() anything inside it, so it's very hard to double check to see if the arrays/tensors are what I expect them to be.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciated!&lt;/p&gt;&#xA;" OwnerUserId="51852" LastActivityDate="2018-05-16T15:52:35.383" Title="Keras/Theano custom loss calculation - working with tensors" Tags="|machine-learning|python|deep-learning|keras|theano|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31741" PostTypeId="1" AcceptedAnswerId="31742" CreationDate="2018-05-16T15:19:24.977" Score="3" ViewCount="5545" Body="&lt;p&gt;Is there someway to create a bubble plot with seaborn? I already know how to do it with Matplot lib, as in &lt;a href=&quot;https://python-graph-gallery.com/270-basic-bubble-plot/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this tutorial&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# libraries&#xA;import matplotlib.pyplot as plt&#xA;import numpy as np&#xA; &#xA;# create data&#xA;x = np.random.rand(40)&#xA;y = np.random.rand(40)&#xA;z = np.random.rand(40)&#xA; &#xA;# use the scatter function&#xA;plt.scatter(x, y, s=z*1000, alpha=0.5)&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I would like to replicate this with Seaborn. I am able to plot, but cannot control the size of marker.&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;import numpy as np&#xA;%matplotlib inline&#xA;data=pd.DataFrame({&amp;quot;x&amp;quot;:x, &amp;quot;y&amp;quot;:y,&amp;quot;z&amp;quot;:z})&#xA;sns.lmplot(x=&amp;quot;x&amp;quot;, y=&amp;quot;y&amp;quot;,data=data,  fit_reg=False)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52250" LastEditorUserId="98307" LastEditDate="2020-08-01T10:30:42.277" LastActivityDate="2020-08-01T10:30:42.277" Title="Bubbleplot with seaborn" Tags="|python|plotting|seaborn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31742" PostTypeId="2" ParentId="31741" CreationDate="2018-05-16T15:52:08.800" Score="1" Body="&lt;p&gt;A duplicate of this &lt;a href=&quot;https://stackoverflow.com/questions/34796451/changing-the-marker-size-in-python-seaborn-lmplot&quot;&gt;question&lt;/a&gt; in stackoverflow. Anyhow just to recap quickly, you can do it with:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scatter_kws={&quot;s&quot;: 10}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And in your case it is simply:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sns.lmplot(x=&quot;x&quot;, y=&quot;y&quot;,data=data,  fit_reg=False,scatter_kws={&quot;s&quot;: z*1000})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-16T15:52:08.800" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31743" PostTypeId="2" ParentId="31740" CreationDate="2018-05-16T15:52:35.383" Score="0" Body="&lt;p&gt;I think I might have finally just solved this myself. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) I changed my Keras backend to use TensorFlow instead of Theano, so that I could use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tf.boolean_mask&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This command was not available under the Theano backend and thus giving me errors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I had to change my code slightly to work with the correct dimensions. It now reads:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;WherePredicted1 = K.greater( y_pred[:,1],0.5) &#xA;ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Still finding it hard to debug/test a custom loss function, but it's looking like feasible values to might well be correct.&lt;/p&gt;&#xA;" OwnerUserId="51852" LastActivityDate="2018-05-16T15:52:35.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31745" PostTypeId="1" AcceptedAnswerId="31747" CreationDate="2018-05-16T17:02:25.770" Score="2" ViewCount="732" Body="&lt;p&gt;I am having a problem understanding the cost function in a neural network. I have read many books and blog posts, but all of them describe that point in neural networks is to minimize the cost function (like sum squared error):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WQ70T.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WQ70T.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to look at code for solving a problem with a multi layer neural network and back propagation. My question is: where in the code can I find the cost function? How can I plot the error surface? &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;X_XOR = np.array([[0,0,1], [0,1,1], [1,0,1],[1,1,1]])&#xA;y_truth = np.array([[0],[1],[1],[0]])&#xA;&#xA;def sigmoid(x):&#xA;    return 1 / (1 + np.exp(-x))&#xA;def sigmoid_der(output):&#xA;    return output * (1 - output)&#xA;&#xA;np.random.seed(1)&#xA;syn_0 = 2*np.random.random((3,4)) - 1&#xA;syn_1 = 2*np.random.random((4,1)) - 1&#xA;&#xA;for i in range(60000):&#xA;    layer_1 = sigmoid(X_XOR.dot(syn_0))&#xA;    layer_2 = sigmoid(layer_1.dot(syn_1))&#xA;    error = 0.5 * ((layer_2 - y_truth) ** 2)&#xA;    layer_2_delta = error * sigmoid_der(layer_2)&#xA;    layer_1_error = layer_2_delta.dot(syn_1.T)&#xA;    layer_1_delta = layer_1_error * sigmoid_der(layer_1)&#xA;    syn_1 -= layer_1.T.dot(layer_2_delta)&#xA;    syn_0 -= X_XOR.T.dot(layer_1_delta)&#xA;    if i % 10000 == 1:&#xA;        print(layer_2)&#xA;&#xA;print(layer_2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="49067" LastEditorUserId="29169" LastEditDate="2019-06-07T05:53:51.453" LastActivityDate="2019-06-07T05:53:51.453" Title="XOR problem with neural network, cost function" Tags="|neural-network|deep-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31746" PostTypeId="1" CreationDate="2018-05-16T17:04:47.440" Score="20" ViewCount="157938" Body="&lt;p&gt;I got this matrix &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    120     100     80      40      20      10      5       0&#xA;120 64.21   58.20   51.20   56.37   47.00   45.61   46.86   2.16&#xA;100 62.84   57.80   50.60   51.32   39.43   39.30   42.80   0.89&#xA;80  62.62   56.20   51.20   51.61   46.23   37.20   42.20   5.32&#xA;40  62.05   52.10   44.20   48.79   42.22   35.16   41.80   1.81&#xA;20  61.65   50.90   42.30   46.23   44.83   32.70   41.50   6.24&#xA;10  59.69   50.20   40.10   40.20   44.28   32.80   39.90   12.31&#xA;5   59.05   49.20   40.60   38.90   44.10   30.80   32.80   9.91&#xA;0   56.20   49.10   40.50   38.60   36.20   32.20   31.50   0.00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I know how to plot heatmap for the values inside by specifying it as numpy array and then using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ax = sns.heatmap(nd, annot=True, fmt='g')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But can someone help me how do I include the column and row labels? The column labels and row labels are given (120,100,80,42,etc.)&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2022-02-01T17:40:15.893" Title="How to include labels in sns heatmap" Tags="|python|visualization|numpy|seaborn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31747" PostTypeId="2" ParentId="31745" CreationDate="2018-05-16T17:15:03.130" Score="2" Body="&lt;p&gt;The cost function can be found in the &lt;strong&gt;delta rule&lt;/strong&gt;, meaning the way you calculate your deltas. This delta is nothing more than the &lt;strong&gt;derivative of your error function after the weights&lt;/strong&gt;: $\frac{\partial E}{\partial w_{ij}}$. So, if you are just interested in where the cost is encoded, this is the answer you are looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you, on the other hand want to know &lt;strong&gt;why&lt;/strong&gt; this formula works, I can suggest you to read the &lt;a href=&quot;https://en.wikipedia.org/wiki/Delta_rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;derivation on wikipedia&lt;/a&gt;. The maths behind it is quite uncomplicated, you only compute the derivative in each layer and propagate this derivative through the layers. This is, by the way, how &lt;strong&gt;backpropagation&lt;/strong&gt; got its name. &lt;/p&gt;&#xA;" OwnerUserId="52257" LastActivityDate="2018-05-16T17:15:03.130" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31748" PostTypeId="2" ParentId="8681" CreationDate="2018-05-16T19:25:27.870" Score="1" Body="&lt;p&gt;The distance metric implemented by @gregorymatchado has one bug.&#xA;For Numeric attributes, range will give NaN for same values throughout.&#xA;For that we need a change use &lt;code&gt;max(np.ptp(feature.values),1)&lt;/code&gt; instead of &lt;code&gt;np.ptp(feature.values)&lt;/code&gt;.&#xA;Complete code below :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.neighbors import DistanceMetric&#xA;&#xA;&#xA;def gower_distance(X):&#xA;&quot;&quot;&quot;&#xA;This function expects a pandas dataframe as input&#xA;The data frame is to contain the features along the columns. Based on these features a&#xA;distance matrix will be returned which will contain the pairwise gower distance between the rows&#xA;All variables of object type will be treated as nominal variables and the others will be treated as &#xA;numeric variables.&#xA;Distance metrics used for:&#xA;Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)&#xA;Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)&#xA;&quot;&quot;&quot;&#xA;individual_variable_distances = []&#xA;&#xA;for i in range(X.shape[1]):&#xA;    feature = X.iloc[:,[i]]&#xA;    if feature.dtypes[0] == np.object:&#xA;        feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))&#xA;    else:&#xA;        feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / max(np.ptp(feature.values),1)&#xA;&#xA;    individual_variable_distances.append(feature_dist)&#xA;&#xA;return np.array(individual_variable_distances).mean(0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29335" LastActivityDate="2018-05-16T19:25:27.870" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31749" PostTypeId="1" CreationDate="2018-05-16T19:32:15.713" Score="1" ViewCount="110" Body="&lt;p&gt;I had an assignment in which we had to classify the cuisine and also give back the top-5 recipes based on given input. I did a count vectorization (countVectorize.transformer()) for the following data and then used Jaccard's distance to calculate the closest matches. Is this approach right or are there better distance metrics for my purpose?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset : &lt;a href=&quot;https://www.kaggle.com/c/whats-cooking/data&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/c/whats-cooking/data&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;{&#xA;  &quot;id&quot;: 24717,&#xA;   &quot;cuisine&quot;: &quot;indian&quot;,&#xA;   &quot;ingredients&quot;: [&#xA;       &quot;tumeric&quot;,&#xA;       &quot;vegetable stock&quot;,&#xA;       &quot;tomatoes&quot;,&#xA;       &quot;garam masala&quot;,&#xA;       &quot;naan&quot;,&#xA;       &quot;red lentils&quot;,&#xA;       &quot;red chili peppers&quot;,&#xA;       &quot;onions&quot;,&#xA;       &quot;spinach&quot;,&#xA;       &quot;sweet potatoes&quot;&#xA;   ]&#xA;   },&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="49967" LastActivityDate="2021-08-08T02:05:40.570" Title="Suitability of Jaccard's distance for count vectorized features" Tags="|distance|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31750" PostTypeId="1" AcceptedAnswerId="33631" CreationDate="2018-05-16T22:56:57.700" Score="1" ViewCount="895" Body="&lt;p&gt;Gaussian kernel&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;has a hyperparameter $\sigma$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know &lt;strong&gt;grid search cross validation&lt;/strong&gt;, but this would require a lot of computation since computational cost of kernel method scales with the number of samples to the power of 2. &lt;/p&gt;&#xA;" OwnerUserId="25837" LastActivityDate="2018-06-25T17:51:47.963" Title="How to tune bandwidth in machine learning kernel model?" Tags="|machine-learning|hyperparameter|gaussian|kernel|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31752" PostTypeId="1" AcceptedAnswerId="31755" CreationDate="2018-05-17T02:41:22.487" Score="1" ViewCount="8606" Body="&lt;p&gt;My dependent variable is binary. Most of my independent variables are not. I am at the exploratory stage right now.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Y     X1     X2&#xA;0     23    0&#xA;1     29    1&#xA;0     15    1&#xA;1     40    0&#xA;1     25    1&#xA;0     22    1&#xA;This is just a portion of my data.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I was thinking scatter plot to find out the relationship between Y X1 and X2.&#xA;What other plots I can do to see the relationship more clearly.&lt;/p&gt;&#xA;" OwnerUserId="51668" LastActivityDate="2018-05-17T03:27:02.053" Title="plot show relationship between independent variable and dependent variable(Binary)" Tags="|classification|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31754" PostTypeId="2" ParentId="31752" CreationDate="2018-05-17T03:15:59.063" Score="1" Body="&lt;p&gt;There is package named &lt;a href=&quot;https://cran.r-project.org/web/packages/Boruta/Boruta.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Boruta&lt;/a&gt; in R/Python. This is also called as predictor importance test. Used to pick the important variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Uses of this package:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boruta is a feature selection algorithm. Precisely, it works as a wrapper algorithm around Random Forest. This package derive its name from a demon in Slavic mythology who dwelled in pine forests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We know that feature selection is a crucial step in predictive modeling. This technique achieves supreme importance when a data set comprised of several variables is given for model building.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Boruta can be your algorithm of choice to deal with such data sets. Particularly when one is interested in understanding the mechanisms related to the variable of interest, rather than just building a black box predictive model with good prediction accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outcome would be a box whisker plot with their importance with the target variable. You need not worry about categorical and continuous variables. you just need to cast them correctly before passing the data to the algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can go through this &lt;a href=&quot;https://www.analyticsvidhya.com/blog/2016/03/select-important-variables-boruta-package/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Link&lt;/a&gt; for better understanding on the package and implementation of the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the sample plot:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/WRukE.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WRukE.png&quot; alt=&quot;sample plot&quot;&gt;&lt;/a&gt;&#xA;Let me know if you need any additional information.&lt;/p&gt;&#xA;" OwnerUserId="40853" LastEditorUserId="40853" LastEditDate="2018-05-17T03:27:02.053" LastActivityDate="2018-05-17T03:27:02.053" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31755" PostTypeId="2" ParentId="31752" CreationDate="2018-05-17T03:18:39.897" Score="1" Body="&lt;p&gt;First we will load this data into a Pandas DataFrame&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;df = pd.DataFrame(data = {'Y': [0,1,0,1,1,0], &#xA;                          'X1':[23,29,15,40,25,22],&#xA;                          'X2':[0,1,1,0,1,1]})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You want to see which variables best describe your output $Y$. First step plot your features against your output to see how they are distributed.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(14,5))&#xA;&#xA;plt.subplot(1,2,1)&#xA;plt.scatter(df['X1'], df['Y'])&#xA;plt.ylabel('Feature Y')&#xA;plt.xlabel('Feature X1')&#xA;&#xA;plt.subplot(1,2,2)&#xA;plt.scatter(df['X2'], df['Y'])&#xA;plt.ylabel('Feature Y')&#xA;plt.xlabel('Feature X2')&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6z8TJ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6z8TJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can split your data based on the output and see the output distributions separability. This will give you information regarding the importance of each feature for building a classifier.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.figure(figsize=(14,5))&#xA;&#xA;plt.subplot(1,2,1)&#xA;plt.hist(df['X1'][df['Y'] == 0], bins=3, alpha = 0.7, label = 'Y = 0')&#xA;plt.hist(df['X1'][df['Y'] == 1], bins=3, alpha = 0.7, label = 'Y = 1')&#xA;plt.ylabel('Distribution')&#xA;plt.xlabel('Feature X1')&#xA;plt.legend()&#xA;&#xA;plt.subplot(1,2,2)&#xA;plt.hist(df['X2'][df['Y'] == 0], bins=3, alpha = 0.7, label = 'Y = 0')&#xA;plt.hist(df['X2'][df['Y'] == 1], bins=3, alpha = 0.7, label = 'Y = 1')&#xA;plt.ylabel('Distribution')&#xA;plt.xlabel('Feature X2')&#xA;plt.legend()&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4jrLu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4jrLu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this point we can see that feature X2 alone is useless in classifying Y. However, when we consider both X1 and X2 on the output we can see that X2 can help us make a better prediction of Y. We can see a linear separator would do the trick for this data.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plt.scatter(df['X1'], df['X2'], c = df['Y'], cmap = 'autumn')&#xA;plt.ylabel('Feature X2')&#xA;plt.xlabel('Feature X1')&#xA;plt.colorbar()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QsabS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QsabS.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-17T03:18:39.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31756" PostTypeId="1" CreationDate="2018-05-17T04:40:05.983" Score="0" ViewCount="256" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/gutfeeling/word_forms&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/gutfeeling/word_forms&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the URL, I find out how to convert word into all the forms(verb, noun, adjective). I need that after conversion, list format.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;e.g:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from word_forms.word_forms import get_word_forms&#xA;&#xA;a=get_word_forms(&quot;review&quot;)&#xA;print(a)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'n': {'reviewer', 'reviewers', 'reviews', 'review'}, 'a': set(), 'v': {'reviewed', 'reviews', 'reviewing', 'review'}, 'r': set()}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;my expected output is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['reviewer', 'reviewers', 'reviews', 'review','reviewed', 'reviews', 'reviewing', 'review']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;how this is possible?&lt;/p&gt;&#xA;" OwnerUserId="45741" LastEditorUserId="45264" LastEditDate="2018-08-15T23:03:33.690" LastActivityDate="2018-08-16T05:09:20.783" Title="How to combine dictionary of word-forms into a single list without duplicates" Tags="|python|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31759" PostTypeId="2" ParentId="31756" CreationDate="2018-05-17T05:55:36.013" Score="0" Body="&lt;pre&gt;&lt;code&gt;from word_forms.word_forms import get_word_forms&#xA;&#xA;a=get_word_forms(&quot;review&quot;)&#xA;&#xA;list_item=[];&#xA;list_item1=[];&#xA;list_item2=[];&#xA;list_item3=[];&#xA;for key,val in a.items():&#xA;    if(key=='n' ):&#xA;        for n in val:&#xA;            list_item.append(n)&#xA;    if (key == 'v'):&#xA;        for v in val:&#xA;            list_item1.append(v)&#xA;    if (key == 'r'):&#xA;        for r in val:&#xA;            list_item2.append(r)&#xA;    if (key == 'a'):&#xA;        for a in val:&#xA;            list_item3.append(a)&#xA;&#xA;&#xA;list=list_item+list_item1+list_item2+list_item3&#xA;print(list)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;output:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['reviews', 'review', 'reviewers', 'reviewer', 'reviews', 'review', 'reviewing', 'reviewed']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45741" LastEditorUserId="45264" LastEditDate="2018-07-16T07:57:12.080" LastActivityDate="2018-07-16T07:57:12.080" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31761" PostTypeId="1" CreationDate="2018-05-17T06:41:11.893" Score="1" ViewCount="73" Body="&lt;p&gt;I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text.&#xA;I use a simple naive Bayes classifier (sklearn) to try to predict a rating given a review which works reasonably well (~65% accuracy - I hope that's decent given that there are 5 classes and I'm only using a naive Bayes?)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;classifier = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])&#xA;classifier.fit(x_train, y_train)&#xA;predicted = classifier.predict(x_test)&#xA;accuracy = accuracy_score(y_test, predicted) # ~65%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Anyway, what I would like to do is build a classifier that is really good at predicting if a review has the rating 4 or 5 for example. I don't really care how (bad) the classifier is doing for ratings other than 4 or 5.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My ideas were&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;modify the dataset and change all labels to 1 where the label is not 4 or 5, then fit the classifier again&lt;/li&gt;&#xA;&lt;li&gt;only use the training examples that have label 4 or 5&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, both approaches do not help at all, the F-Score for classes 4 and 5 do not improve. What can I do to make a classifier better for these 2 classes only?&lt;/p&gt;&#xA;" OwnerUserId="50028" LastActivityDate="2024-03-20T21:05:55.760" Title="Optimize F-Score only for certain classes, disregard other classes" Tags="|scikit-learn|multiclass-classification|naive-bayes-classifier|accuracy|sentiment-analysis|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31762" PostTypeId="1" CreationDate="2018-05-17T07:33:58.860" Score="1" ViewCount="20" Body="&lt;p&gt;Within the recent years, leveraging crowdsourcing for performing, mainly intelligent-intensive, tasks has become prevalent. An important question in this regard, that I could not find any specific answer for, is how many participants are adequate for running such a project? Is there any standard for that?  &lt;/p&gt;&#xA;" OwnerUserId="6641" LastActivityDate="2021-09-22T14:04:12.620" Title="Numbers of participants in crowdsourcing tasks" Tags="|dataset|data-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31763" PostTypeId="2" ParentId="31750" CreationDate="2018-05-17T07:44:26.477" Score="1" Body="&lt;p&gt;As shown in &lt;a href=&quot;https://en.wikipedia.org/wiki/Kernel_density_estimation#A_rule-of-thumb_bandwidth_estimator&quot; rel=&quot;nofollow noreferrer&quot;&gt;wikipedia for KDE&lt;/a&gt;, a rule-of-thumb bandwidth estimator can be given if the underlying density for your data is Gaussian. This estimator is given by: $h = (\frac{4\hat{\sigma}}{3n})^{1/5}$, where $h$ is the bandwidth of your KDE estimation, $n$ the number of data and $\hat{\sigma}$ the estimation of the standard deviation of your sample.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the underlying distribution for your data is not gaussian, you can still try with this bandwidth, but it might smooth everything too much. In that case, you should go for cross-validation with smaller bandwidths.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-17T07:44:26.477" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31764" PostTypeId="1" CreationDate="2018-05-17T07:59:31.270" Score="1" ViewCount="369" Body="&lt;p&gt;I would like to render rapid reports using the Markdown files with links to csv files [csv](path to csv file). This part is working, I can dowload the csv file from the markdown preview, but I can't open the file in the Jupyter Lab &quot;Table&quot;. Is there is something to do about this? &lt;/p&gt;&#xA;" OwnerUserId="52283" LastActivityDate="2018-05-17T07:59:31.270" Title="How to link csv file within a markdown file when using Jupyter Lab?" Tags="|jupyter|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31765" PostTypeId="2" ParentId="30374" CreationDate="2018-05-17T08:02:08.310" Score="1" Body="&lt;p&gt;If you have a date variable (or something similar), you can create a weight using this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you're using XGBoost, there is an option to specify a &lt;code&gt;weight&lt;/code&gt; for each instance when creating the &lt;code&gt;DMatrix&lt;/code&gt; - feed your observation weighting in here.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-05-17T08:02:08.310" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31766" PostTypeId="1" CreationDate="2018-05-17T08:06:06.797" Score="1" ViewCount="32" Body="&lt;p&gt;Is it a good idea majoring in data science for undergraduate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Auckland University offers an undergraduate major in data science, which has courses in computer science and statistics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I read somewhere that it's a bad idea to to undergraduate in data science. &lt;/p&gt;&#xA;" OwnerUserId="52285" LastActivityDate="2018-05-17T08:06:06.797" Title="Is it a good idea majoring in data science for undergraduate?" Tags="|data|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-18T00:34:14.287" ContentLicense="CC BY-SA 4.0" />
  <row Id="31767" PostTypeId="1" AcceptedAnswerId="31769" CreationDate="2018-05-17T08:24:35.950" Score="2" ViewCount="2609" Body="&lt;p&gt;I'm studying the knn classification algorithm.&#xA;Why can the euclidean distance be considered a nice measure of affinity between examples ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In one dimension (1 attribute) this seems correct, but if I add dimensions, can the euclidean distance still be considerd a good measure of affinity? Why?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="50727" LastEditDate="2018-05-24T09:18:57.773" LastActivityDate="2018-05-24T09:18:57.773" Title="Knn and euclidean distance" Tags="|machine-learning|classification|distance|k-nn|mathematics|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31768" PostTypeId="1" CreationDate="2018-05-17T08:38:57.477" Score="2" ViewCount="463" Body="&lt;p&gt;I have a lot of time series with &lt;strong&gt;different lengths&lt;/strong&gt;. I would like to know what are the best practices to fit them to a Bidirectional LSTM model. The problem is a Binary Classification of Sequence to Sequence. So for every time step, I want to predict the binary class.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently, I create a tensor for each data frame with the shape of &lt;code&gt;(1, None, #Features)&lt;/code&gt;. Then I fit every tensor separately to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it better to combine every data frame to a tensor of the shape &lt;code&gt;(#Time Series, None, #Features)&lt;/code&gt; and fit them all at once? Does this make even a difference?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or could it be better to go with the sliding window approach and split one time-series into smaller windows?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can't specify a max-length of the time-series so I think I cannot use &lt;code&gt;pad-sequence&lt;/code&gt; from Keras.&lt;/p&gt;&#xA;" OwnerUserId="32543" LastEditorUserId="44039" LastEditDate="2018-05-17T16:22:43.973" LastActivityDate="2024-01-20T01:03:50.400" Title="Batch processing with variable length sequences" Tags="|classification|keras|tensorflow|time-series|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31769" PostTypeId="2" ParentId="31767" CreationDate="2018-05-17T08:40:53.313" Score="3" Body="&lt;p&gt;You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine distance or $L^p$ metrics. In fact, if $p$ is very high, then Euclidean distance is not a good measure, as it tends to make the distances too uniform (see &lt;a href=&quot;https://bib.dbvis.de/uploadedFiles/155.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: When $p$ is very high:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions&quot;&gt;this magnificient answer&lt;/a&gt; to the issues that very high $p$ may have.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-17T09:12:39.097" LastActivityDate="2018-05-17T09:12:39.097" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="31770" PostTypeId="1" CreationDate="2018-05-17T10:19:57.520" Score="1" ViewCount="55" Body="&lt;p&gt;For SVM, it is better to solve the problem in the primal for very large data-sets. However, the non-linear mapping $\phi$ for a RBF kernel is not explicit. Approximation methods for $\phi$ like the Nyström methods are then used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working on a classification problem where there is no strong duality (duality gap between the primal and the dual). I have already tested a linear kernel (there $\phi$ is completely explicit: $\phi$(x) = x). I would however be very interested to also test a RBF kernel and then solve the problem in the primal.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;=&gt; Can a method like the Nyström method still be applied in my case?&lt;/p&gt;&#xA;" OwnerUserId="52290" LastActivityDate="2018-05-17T10:19:57.520" Title="Nyström approximation of the non-linear mapping $\phi$ for a RBF kernel - what is the impact of weak duality?" Tags="|svm|kernel|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31771" PostTypeId="2" ParentId="9453" CreationDate="2018-05-17T10:29:06.630" Score="2" Body="&lt;p&gt;You can still use a linear SVM classifier in a non-separable case. However, you should then tolerate misclassifications. You should then tune the hyperparameter $C$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are cases where a linear SVM in a non-separable case gives better results than a RBF kernel. For instance, the optimal decision boundary for two classes with equal covariance matrices is linear.&lt;/p&gt;&#xA;" OwnerUserId="52290" LastActivityDate="2018-05-17T10:29:06.630" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31773" PostTypeId="1" AcceptedAnswerId="31775" CreationDate="2018-05-17T12:05:38.457" Score="9" ViewCount="5316" Body="&lt;p&gt;Every classifier in scikit-learn has a method &lt;code&gt;predict_proba(x)&lt;/code&gt; that predicts class probabilities for &lt;code&gt;x&lt;/code&gt;. How to do the same thing for regressors? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only regressor for which I know how to estimate the variance of the predictions is Gaussian process regression, for which I can do the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;y_pred, sigma = gp.predict(x, return_std=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In one dimension, I can even plot, how confident the Gaussian process regressor is about its prediction of different data points&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/gPHav.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/gPHav.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to estimate the variance of predictions for other regressors? For example, for kernel ridge regressor, multi-layer perceptron, ensemble regressors?&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2019-08-07T13:00:40.977" Title="How to estimate the variance of regressors in scikit-learn?" Tags="|python|scikit-learn|regression|variance|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31774" PostTypeId="1" CreationDate="2018-05-17T12:28:20.493" Score="2" ViewCount="35" Body="&lt;p&gt;Currently, i am using 1-D acceleration by dividing data in 10 seconds segments and then making a test train split into these segments. But i am not sure how to use 3-D data, especially how to make test train split out this data because for each label there will 3 channel input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;My dataset looks lie this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Sd6GI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Sd6GI.png&quot; alt=&quot;Data Set&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now I am only using &lt;code&gt;X-Axis&lt;/code&gt;. I am segmenting &lt;code&gt;X-acc&lt;/code&gt; in 200 samples and for each sample, I have an activity label. So now I have a number of segments and a label for each of these segments. After that, I am splitting these segments and their corresponding labels in &lt;strong&gt;test train split&lt;/strong&gt;. This all works fine but no i want to use &lt;code&gt;Y-axis&lt;/code&gt; and &lt;code&gt;x-axis&lt;/code&gt; also. so now i will have three arrays each segmented in 200 samples, now i am not sure how toperform test train split.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt; Activity labels are not shown here, but there are total 6 labels and are one hot encoded.&lt;/p&gt;&#xA;" OwnerUserId="51904" LastEditorUserId="51904" LastEditDate="2018-05-17T13:56:12.350" LastActivityDate="2018-05-17T13:56:12.350" Title="How to use 3-Axial acceleration data for CNN?" Tags="|machine-learning|neural-network|cnn|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31775" PostTypeId="2" ParentId="31773" CreationDate="2018-05-17T12:37:49.680" Score="5" Body="&lt;p&gt;I believe it is the &lt;em&gt;probabilistic nature of a model&lt;/em&gt; that allows you to get the variance of predictions, or more generally defined as &lt;strong&gt;the uncertainty of predictions&lt;/strong&gt;, like the Gaussian process you mentioned. This is not simply avaialble in standard regressors.&lt;/p&gt;&#xA;&lt;p&gt;I think you should be looking at &lt;strong&gt;Probabilistic regressors&lt;/strong&gt; like &lt;em&gt;BayesianRidge&lt;/em&gt; if you would like to estimate the uncertainty of your model. An implementation  is also avaialble in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html#sklearn.linear_model.BayesianRidge&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn&lt;/a&gt;, also &lt;a href=&quot;https://github.com/markdregan/Bayesian-Modelling-in-Python&quot; rel=&quot;nofollow noreferrer&quot;&gt;this nice python package&lt;/a&gt; based on PyMC3 or directly via &lt;a href=&quot;https://docs.pymc.io/notebooks/GLM-linear.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyMC3&lt;/a&gt; itself for instance. In the latter there are examples like for Bayesian regression in Jupyter Notebook with a good explanation.&lt;/p&gt;&#xA;&lt;p&gt;In principle, &lt;strong&gt;Bayesian Models&lt;/strong&gt; do not return a a single estimate for the model parameters, but a distribution that make it possible to make inferences about new observations as well as to examine our uncertainty in the model. You may find this &lt;a href=&quot;https://towardsdatascience.com/bayesian-linear-regression-in-python-using-machine-learning-to-predict-student-grades-part-2-b72059a8ac7e&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt; useful.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Adding a normal prior on the weights as it is done in Bayesian regression, one turn the Least-Squares problem to regularized L2 regression under the hood as well (see the full math. derivation &lt;a href=&quot;https://wiseodd.github.io/techblog/2017/01/05/bayesian-regression/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Updated Answer&lt;/strong&gt;: I totally forgot the classical yet simple and powerful &lt;strong&gt;Bootstrap Sampling&lt;/strong&gt; method to calculate confidence intervals for machine learning algorithms. A textbook definition says:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Bootstrapping is a nonparametric approach to statistical inference&#xA;that substitutes computation for more traditional distributional&#xA;assumptions and asymptotic results. A number of advantages:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The bootstrap is quite general, although there are some cases in which it fails.&lt;/li&gt;&#xA;&lt;li&gt;Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate&#xA;inferences when the data are not well behaved or when the sample size&#xA;is small.&lt;/li&gt;&#xA;&lt;li&gt;It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.&lt;/li&gt;&#xA;&lt;li&gt;It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Reference: Fox, John. Applied regression analysis and generalized linear&#xA;models. Sage Publications, 2015.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Please note you do not need a model with probabilistic nature. See this &lt;a href=&quot;https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/&quot; rel=&quot;nofollow noreferrer&quot;&gt;post&lt;/a&gt;, or this &lt;a href=&quot;https://stackoverflow.com/questions/16707141/python-estimating-regression-parameter-confidence-intervals-with-scikits-boots&quot;&gt;answer&lt;/a&gt; or this &lt;a href=&quot;https://stats.stackexchange.com/questions/183230/bootstrapping-confidence-interval-from-a-regression-prediction&quot;&gt;one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-06-01T11:23:30.047" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31776" PostTypeId="1" CreationDate="2018-05-17T13:26:03.670" Score="1" ViewCount="38" Body="&lt;p&gt;I am an undergraduate that is currently working on a project in which I need to collect reviews and comments about mobile phones across several platforms and analyse the data to see if the sentiments and reviews have direct impact on the sales of the mobile. My problem is the best method to get this data from sites like GSM arena and the rest of them. &lt;/p&gt;&#xA;" OwnerUserId="52301" LastActivityDate="2018-05-17T16:19:21.793" Title="Method of data collection" Tags="|sentiment-analysis|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-18T01:44:37.503" ContentLicense="CC BY-SA 4.0" />
  <row Id="31777" PostTypeId="2" ParentId="31650" CreationDate="2018-05-17T14:14:04.237" Score="0" Body="&lt;p&gt;you have two options. You can extract the number of records belonging to the smaller class and use that number to sample that many examples from the larger class to get an exact 50/50 ratio. In RapidMiner you can achieve this using the &quot;Filter Examples&quot; operator combined with &quot;Extract Macro&quot; and &quot;Sample&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The other option is using only the &quot;Sample&quot; Operator and check the &lt;em&gt;balance data&lt;/em&gt; option. With &quot;absolute sampling&quot; you can define the exact number of examples used from each class. With ratio sampling you can also reduce the number of records from one class, but hitting an exact 50/50 ratio might be more difficult.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See the attached RapidMiner process xml (just c&amp;amp;p it into the process view) showing an example process.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&amp;gt;&amp;lt;process version=&quot;8.2.000&quot;&amp;gt;&#xA;&amp;lt;context&amp;gt;&#xA;&amp;lt;input/&amp;gt;&#xA;&amp;lt;output/&amp;gt;&#xA;&amp;lt;macros/&amp;gt;&#xA; &amp;lt;/context&amp;gt;&#xA; &amp;lt;operator activated=&quot;true&quot; class=&quot;process&quot; compatibility=&quot;8.2.000&quot; &#xA;  expanded=&quot;true&quot; name=&quot;Process&quot;&amp;gt;&#xA;&amp;lt;process expanded=&quot;true&quot;&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;retrieve&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;68&quot; name=&quot;Retrieve Iris&quot; width=&quot;90&quot; x=&quot;39&quot; y=&quot;136&quot;&amp;gt;&#xA;     &amp;lt;parameter key=&quot;repository_entry&quot; value=&quot;//Samples/data/Iris&quot;/&amp;gt;&#xA;   &amp;lt;/operator&amp;gt;&#xA;   &amp;lt;operator activated=&quot;true&quot; class=&quot;generate_attributes&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;82&quot; name=&quot;Generate Attributes&quot; width=&quot;90&quot; x=&quot;179&quot; y=&quot;136&quot;&amp;gt;&#xA;    &amp;lt;list key=&quot;function_descriptions&quot;&amp;gt;&#xA;      &amp;lt;parameter key=&quot;label&quot; value=&quot;if(label !=&amp;amp;quot;Iris-setosa&amp;amp;quot;, TRUE, FALSE)&quot;/&amp;gt;&#xA;    &amp;lt;/list&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;Generate a binary class with a 1/3 to 2/3 ratio&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;multiply&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;103&quot; name=&quot;Multiply&quot; width=&quot;90&quot; x=&quot;313&quot; y=&quot;136&quot;/&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;filter_examples&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;103&quot; name=&quot;Filter Examples&quot; width=&quot;90&quot; x=&quot;447&quot; y=&quot;136&quot;&amp;gt;&#xA;    &amp;lt;list key=&quot;filters_list&quot;&amp;gt;&#xA;      &amp;lt;parameter key=&quot;filters_entry_key&quot; value=&quot;label.equals.false&quot;/&amp;gt;&#xA;    &amp;lt;/list&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;Split between the two classes&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;extract_macro&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;68&quot; name=&quot;Extract Macro&quot; width=&quot;90&quot; x=&quot;648&quot; y=&quot;136&quot;&amp;gt;&#xA;    &amp;lt;parameter key=&quot;macro&quot; value=&quot;size_smaller_class&quot;/&amp;gt;&#xA;    &amp;lt;list key=&quot;additional_macros&quot;/&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;1) Extract the number of examples from the smaller class&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;sample&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;82&quot; name=&quot;Sample&quot; width=&quot;90&quot; x=&quot;648&quot; y=&quot;340&quot;&amp;gt;&#xA;    &amp;lt;parameter key=&quot;sample_size&quot; value=&quot;%{size_smaller_class}&quot;/&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_size_per_class&quot;&amp;gt;&#xA;      &amp;lt;parameter key=&quot;true&quot; value=&quot;50&quot;/&amp;gt;&#xA;      &amp;lt;parameter key=&quot;false&quot; value=&quot;50&quot;/&amp;gt;&#xA;    &amp;lt;/list&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_ratio_per_class&quot;&amp;gt;&#xA;      &amp;lt;parameter key=&quot;true&quot; value=&quot;0.5&quot;/&amp;gt;&#xA;    &amp;lt;/list&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_probability_per_class&quot;/&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;2) Sample as many examples from the bigger class as there are in the other class.&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;append&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;103&quot; name=&quot;Append&quot; width=&quot;90&quot; x=&quot;849&quot; y=&quot;136&quot;&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;Merge the two classes back together.&amp;amp;lt;br/&amp;amp;gt;&amp;amp;lt;br/&amp;amp;gt;Now both have the same number of examples&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;operator activated=&quot;true&quot; class=&quot;sample&quot; compatibility=&quot;8.2.000&quot; expanded=&quot;true&quot; height=&quot;82&quot; name=&quot;Sample (2)&quot; width=&quot;90&quot; x=&quot;447&quot; y=&quot;544&quot;&amp;gt;&#xA;    &amp;lt;parameter key=&quot;balance_data&quot; value=&quot;true&quot;/&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_size_per_class&quot;&amp;gt;&#xA;      &amp;lt;parameter key=&quot;true&quot; value=&quot;50&quot;/&amp;gt;&#xA;      &amp;lt;parameter key=&quot;false&quot; value=&quot;50&quot;/&amp;gt;&#xA;    &amp;lt;/list&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_ratio_per_class&quot;/&amp;gt;&#xA;    &amp;lt;list key=&quot;sample_probability_per_class&quot;/&amp;gt;&#xA;    &amp;lt;description align=&quot;center&quot; color=&quot;yellow&quot; colored=&quot;true&quot; width=&quot;126&quot;&amp;gt;By using &amp;amp;amp;quot;absolute sampling&amp;amp;amp;quot; with the &amp;amp;amp;quot;balanced data&amp;amp;amp;quot; option, you can get an equally balanced result set.&amp;amp;lt;br&amp;amp;gt;&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Retrieve Iris&quot; from_port=&quot;output&quot; to_op=&quot;Generate Attributes&quot; to_port=&quot;example set input&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Generate Attributes&quot; from_port=&quot;example set output&quot; to_op=&quot;Multiply&quot; to_port=&quot;input&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Multiply&quot; from_port=&quot;output 1&quot; to_op=&quot;Filter Examples&quot; to_port=&quot;example set input&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Multiply&quot; from_port=&quot;output 2&quot; to_op=&quot;Sample (2)&quot; to_port=&quot;example set input&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Filter Examples&quot; from_port=&quot;example set output&quot; to_op=&quot;Extract Macro&quot; to_port=&quot;example set&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Filter Examples&quot; from_port=&quot;unmatched example set&quot; to_op=&quot;Sample&quot; to_port=&quot;example set input&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Extract Macro&quot; from_port=&quot;example set&quot; to_op=&quot;Append&quot; to_port=&quot;example set 1&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Sample&quot; from_port=&quot;example set output&quot; to_op=&quot;Append&quot; to_port=&quot;example set 2&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Append&quot; from_port=&quot;merged set&quot; to_port=&quot;result 1&quot;/&amp;gt;&#xA;  &amp;lt;connect from_op=&quot;Sample (2)&quot; from_port=&quot;example set output&quot; to_port=&quot;result 2&quot;/&amp;gt;&#xA;  &amp;lt;portSpacing port=&quot;source_input 1&quot; spacing=&quot;0&quot;/&amp;gt;&#xA;  &amp;lt;portSpacing port=&quot;sink_result 1&quot; spacing=&quot;105&quot;/&amp;gt;&#xA;  &amp;lt;portSpacing port=&quot;sink_result 2&quot; spacing=&quot;378&quot;/&amp;gt;&#xA;  &amp;lt;portSpacing port=&quot;sink_result 3&quot; spacing=&quot;189&quot;/&amp;gt;&#xA;  &amp;lt;description align=&quot;center&quot; color=&quot;green&quot; colored=&quot;true&quot; height=&quot;118&quot; resized=&quot;true&quot; width=&quot;683&quot; x=&quot;39&quot; y=&quot;10&quot;&amp;gt;&amp;amp;lt;br&amp;amp;gt; First solution is extracting the number of examples from the smaller class and sampling that many from the larger class. This gives an exact 50/50 ratio&amp;lt;/description&amp;gt;&#xA;  &amp;lt;description align=&quot;center&quot; color=&quot;green&quot; colored=&quot;true&quot; height=&quot;122&quot; resized=&quot;true&quot; width=&quot;335&quot; x=&quot;39&quot; y=&quot;488&quot;&amp;gt;Second solution is using the&amp;amp;lt;br&amp;amp;gt;&amp;amp;amp;quot;balance data&amp;amp;amp;quot;&amp;amp;lt;br&amp;amp;gt;parameter of the sample operator.&amp;amp;lt;br&amp;amp;gt;With that you can define the ratio, or exact number of examples to be samples from each class.&amp;amp;lt;br&amp;amp;gt;&amp;lt;/description&amp;gt;&#xA;  &amp;lt;/process&amp;gt;&#xA;  &amp;lt;/operator&amp;gt;&#xA; &amp;lt;/process&amp;gt;&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Also feel free to ask further, or re-post, questions in the &lt;a href=&quot;http://community.rapidminer.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;RapidMiner community forum&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="52287" LastActivityDate="2018-05-17T14:14:04.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31778" PostTypeId="1" CreationDate="2018-05-17T14:51:00.160" Score="1" ViewCount="131" Body="&lt;p&gt;What tools are available that provide an interface to present text classification results? I need to keep and compare the results of different classification algorithms for unsupervised data. The tool should allow users to give input (like the good match or not) for the presented examples of the test data.&lt;/p&gt;&#xA;" OwnerUserId="52303" LastActivityDate="2018-07-18T15:55:39.380" Title="Annotation tool for classification experiments" Tags="|classification|unsupervised-learning|software-recommendation|experiments|annotation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31779" PostTypeId="1" CreationDate="2018-05-17T15:23:30.363" Score="3" ViewCount="691" Body="&lt;p&gt;I'm figuring out how to manipulate convolutional neural networks (CNN) in python and I want to apply this kind of NN to an agent player that plays tic tac toe. I know that's weird and the problem doesn't need some elaborate and complex solutions like CNN, but I'm using this environment just to learn better and compare with the training of a MLP that I implemented.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I'm trying to think about how to represent the board correctly as an input to the CNN. Unlike the chess board which we can think the input as 8x8x6 (8x8 as the 2d array representing the board and 6 channels which one representing the different pieces of the game), the tic tac toe is a bit more complicated for me because there are just one kind of piece. Is it possible and correct to represent the X and O as different pieces? Has anyone implemented something like this?&lt;/p&gt;&#xA;" OwnerUserId="52206" LastActivityDate="2019-11-18T19:26:20.573" Title="How to properly represent a tic tac toe board to a CNN?" Tags="|machine-learning|neural-network|deep-learning|convolution|machine-learning-model|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31782" PostTypeId="1" CreationDate="2018-05-17T17:42:21.630" Score="3" ViewCount="168" Body="&lt;p&gt;I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue I’m facing is that some of the factors come in and out of the time series. For example I may have 12 years in total of data points, some factors may exist for the entire 12 years but some may dip in and out (active for the first two years, inactive for three years, active for the rest for example). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can use iterative PCA to fill in some of the usual gaps in data but I am unsure of what to use for the large chunks (years) of data gaps. For example if a factor only exists for the final three years should I even incorporate it into the analysis? &lt;/p&gt;&#xA;" OwnerUserId="47869" LastActivityDate="2022-05-11T20:06:50.887" Title="PCA huge parts of missing data filling" Tags="|data-cleaning|k-means|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31784" PostTypeId="1" AcceptedAnswerId="31789" CreationDate="2018-05-17T19:10:33.307" Score="3" ViewCount="83" Body="&lt;p&gt;From my understanding of unsupervised DNNs for image classification:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The input layer is a 4,096 dimension vector (for 64 x 64 images)&lt;/li&gt;&#xA;&lt;li&gt;The hidden layers represent much lower &quot;features&quot; as identified by the back propagation&lt;/li&gt;&#xA;&lt;li&gt;As the model is generative, the output layer is also a 64 x 64 image&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Therefore, how do can we make a prediction that a new unseen image contains a specific image class (e.g. cat) if we lack labelled data?&lt;/p&gt;&#xA;" OwnerUserId="52310" LastEditorUserId="52310" LastEditDate="2018-05-17T21:18:39.693" LastActivityDate="2018-05-17T22:23:28.827" Title="How do we predict what is in an image using unsupervised deep neural networks?" Tags="|machine-learning|image-classification|computer-vision|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31785" PostTypeId="1" AcceptedAnswerId="32231" CreationDate="2018-05-17T19:21:16.973" Score="1" ViewCount="715" Body="&lt;p&gt;I'm trying to train a fully convolutional network to generate binary image showing roads from satellite photos. For a long time, I struggled to get the network to output anything but black - then I figured out the problem. I have plenty of training data, but of course most of the output pixels &#xA;will be black, since roads take only small percentage of space.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created a toy example showing the problem:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D&#xA;import numpy as np&#xA;import random&#xA;import sklearn.model_selection&#xA;&#xA;THRESHOLD = 3&#xA;BALANCED = False&#xA;&#xA;if BALANCED:&#xA;    X = [[&#xA;            random.randint(THRESHOLD + 1, 255) if&#xA;                random.randint(0, 1) else&#xA;            random.randint(0, THRESHOLD)&#xA;            for _ in range(10)] for _ in range(10**4)]&#xA;    Y = [[0] if x[0] &amp;lt;= THRESHOLD else [1] for x in X]&#xA;else:&#xA;    X = [[random.randint(0, 255) for _ in range(10)] for _ in range(10**4)]&#xA;    Y = [[0] if x[0] &amp;lt;= THRESHOLD else [1] for x in X]&#xA;&#xA;X, Y = np.array(X), np.array(Y)&#xA;X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(&#xA;        X, Y, test_size = 0.2)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(8, activation = &quot;sigmoid&quot;, input_shape = (10, )))&#xA;model.add(Dense(1, activation = &quot;sigmoid&quot;))&#xA;&#xA;model.compile(loss = &quot;binary_crossentropy&quot;, optimizer = &quot;adam&quot;)&#xA;&#xA;model.summary()&#xA;model.fit(X_train, Y_train, &#xA;        batch_size = 10, epochs = 10, &#xA;        verbose = 2, &#xA;        validation_data = (X_test, Y_test))&#xA;&#xA;Y_res = model.predict(X_test)&#xA;Ys0 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 0]&#xA;Ys1 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 1]&#xA;&#xA;print(np.mean(Ys0))&#xA;print(np.mean(Ys1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here, I artificially create data, with &lt;code&gt;X&lt;/code&gt; being ten random bytes, and &lt;code&gt;Y&lt;/code&gt; being boolean thresholding first byte on the value of three. That means normally only around 1/64 of all samples will be generated with &lt;code&gt;y&lt;/code&gt; equal to 0, similarly to my original problem. There's also a second dataset, with the same decision boundary, but more evenly (class-wise) distributed data - set by changing &lt;code&gt;BALANCED&lt;/code&gt; toggle in the code. The results confirm my suspicions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For balanced set, actual 0's are given, on average, value of 0.025. 1's - 0.97. That's reasonable. On the unbalanced, raw dataset, 0's have mean of 0.87, and 1's - 0.98. That means the zeros are pretty much always classified as one anyway. On my real data, the effect is even worse, and the returned value is pretty much equal to percentage of black, regardless of real classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried adding &lt;code&gt;class_weights = [100, 1]&lt;/code&gt; to &lt;code&gt;fit&lt;/code&gt; call, but it didn't change much. Is there a good strategy for dealing with such unbalanced data? In this toy example I could probably sample the x-y pairs (e.g. keep only 5% of 1-classified data), but I don't think this would work for my  full problem - since I'm using FCN, I'm putting a whole image into the network at once. There are no images with 50% roads, so I cannot possibly sample images to make roads take enough space. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also tried, quite randomly, changing loss functions, including using one for categorical data and one-hot encoding expected outputs (i.e. [1, 0] and [0, 1] instead of 0 and 1). This did not fix anything either.&lt;/p&gt;&#xA;" OwnerUserId="52311" LastActivityDate="2018-05-27T20:24:47.043" Title="How to deal with unbalanced data in pixelwise classification?" Tags="|classification|keras|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31787" PostTypeId="1" CreationDate="2018-05-17T20:07:40.927" Score="1" ViewCount="24" Body="&lt;p&gt;I want to use phone type (e.g., &quot;iPhone 6s&quot;, &quot;Lenovo K8 Note&quot;, etc.) as a feature in a model.  Does anyone know of a way to obtain release date, price, etc of a phone (besides scraping)?  That seems like it would be more useful than one-hot encoding.  &lt;/p&gt;&#xA;" OwnerUserId="9122" LastEditorUserId="26562" LastEditDate="2018-05-18T04:06:20.073" LastActivityDate="2018-05-18T04:06:20.073" Title="Feature building - phone device type" Tags="|dataset|data|feature-engineering|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31788" PostTypeId="2" ParentId="31609" CreationDate="2018-05-17T21:37:05.153" Score="8" Body="&lt;p&gt;Hmmm, I am little perplexed by your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In gradient boosting, we &lt;strong&gt;do&lt;/strong&gt; use the residuals. The residuals are the gradients.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can check &lt;a href=&quot;https://github.com/rpmcruz/machine-learning/blob/master/ensemble/boosting/gboost.py&quot; rel=&quot;noreferrer&quot;&gt;my simple implementation&lt;/a&gt; of gradient boosting. This is where the magic happens:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fit(self, X, y):&#xA;    self.fs = [self.first_estimator]&#xA;    # step 0&#xA;    f = self.first_estimator.fit(X, y)&#xA;    # step 1&#xA;    for m in range(self.M):&#xA;        f = self.predict(X)&#xA;        # step 2&#xA;        U = self.loss(y, f)&#xA;        # step 3&#xA;        g = clone(self.base_estimator).fit(X, U)&#xA;        # step 4&#xA;        self.fs.append(g)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You start by a dummy model $f$. Then you create a new model $g$ based on the errors of the existing ensemble $L(f(x),y)$. The code should be pretty straight-forward. Check &lt;a href=&quot;https://en.wikipedia.org/wiki/Gradient_boosting&quot; rel=&quot;noreferrer&quot;&gt;the algorithm in the wikipedia page&lt;/a&gt; for a more formal presentation.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Why am I using residuals and gradients as interchangeable words?&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have &lt;strong&gt;discrete&lt;/strong&gt; mathematics and &lt;strong&gt;continuous&lt;/strong&gt; mathematics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In neural networks, we use the definition of gradient from continuous mathematics. You have a loss function that is $C_1$ continuous (meaning that the loss must be differentiable at least once).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In gradient boosting, we use the definition of gradient from discrete mathematics. It's usually called &lt;a href=&quot;https://en.wikipedia.org/wiki/Finite_difference&quot; rel=&quot;noreferrer&quot;&gt;fininite differences&lt;/a&gt;, instead of derivative or gradient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$f'(x) = f(x+1)-f(x)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Personally, I think gradient is a misnomer. But it's marketing. Gradient boosting sounds more mathematical and sophisticated than &quot;differences boosting&quot; or &quot;residuals boosting&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By the way, the term boosting already existed when gradient boosting was invented. It was used for AdaBoost, which changes the weights of the observations based on the loss, rather than training in the residuals. AdaBoost only works for &lt;em&gt;weak&lt;/em&gt; models (each model must commit errors), while gradient boosting can do resample and work for &lt;em&gt;strong&lt;/em&gt; models.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Important point on gradient boosting&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One important difference between gradient boosting (discrete optimization) and neural networks (continuous optimization) is that gradient boosting allows you to work with functions whose derivative is constant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In gradient boosting, you can use &quot;weird&quot; functions like MAE or the &lt;a href=&quot;https://www.lokad.com/pinball-loss-function-definition&quot; rel=&quot;noreferrer&quot;&gt;Pinball function&lt;/a&gt;. In my code, you can see that you can choose between MSE, MAE and quantile (which is the pinball loss). These latter two do not work very well for neural networks.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The pinball loss is used for quantile predictions, like wind forecasts in energy systems. This is the reason why gradient boosting is much used in these systems.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Small point: many people think that the reason why MAE and Pinball functions work bad for neural networks is because they are not continuous. While it is true they are not continuous when $y=\hat y$, it's very easy to work-around that. This is not the reason why results are so poor with those losses. The reason why they don't work well is because the optimization steps use the gradient (in gradient descent), and the gradient is constant for the entire function $L'(y,\hat y)=1$ or $L'(y,\hat y)=-1$, so the magnitude of the gradient is always 1. The optimization will depend solely on the learning rate heuristic that you are using, which usually will not work out very well.&lt;/p&gt;&#xA;" OwnerUserId="16853" LastEditorUserId="16853" LastEditDate="2018-05-17T22:48:14.490" LastActivityDate="2018-05-17T22:48:14.490" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31789" PostTypeId="2" ParentId="31784" CreationDate="2018-05-17T22:23:28.827" Score="3" Body="&lt;p&gt;After a lot of reading, I think I now understand. We really need to build 2 models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unsupervised&lt;/li&gt;&#xA;&lt;li&gt;Lots of unlabelled images&lt;/li&gt;&#xA;&lt;li&gt;Used to 'learn features' (i.e. better that we have done manually through years of research e.g. edge detection, colour features etc).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model 2&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Supervised&lt;/li&gt;&#xA;&lt;li&gt;Few labelled images&lt;/li&gt;&#xA;&lt;li&gt;Use model 1 as the 'feature extractor'. i.e. pass a training image through model 1 and use the output layer as the feature vector.&lt;/li&gt;&#xA;&lt;li&gt;Use the same approach to test images e.g. model 1 to extract features, then use the second model to output label predictions&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="52310" LastActivityDate="2018-05-17T22:23:28.827" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31791" PostTypeId="2" ParentId="9832" CreationDate="2018-05-18T00:15:29.150" Score="58" Body="&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$V^\pi(s)$&lt;/span&gt; is the &amp;quot;state&amp;quot; value function of an MDP (Markov Decision Process). It's the expected return starting from state &lt;span class=&quot;math-container&quot;&gt;$s$&lt;/span&gt; following policy &lt;span class=&quot;math-container&quot;&gt;$\pi$&lt;/span&gt;:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$V^\pi(s) = E_{\pi} \{G_t \vert s_t = s\} $$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$G_t$&lt;/span&gt; is the total DISCOUNTED reward from time step &lt;span class=&quot;math-container&quot;&gt;$t$&lt;/span&gt;, as opposed to &lt;span class=&quot;math-container&quot;&gt;$R_t$&lt;/span&gt; which is an immediate return. Here you are taking the expectation over ALL actions according to the policy &lt;span class=&quot;math-container&quot;&gt;$\pi$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$Q^\pi(s, a)$&lt;/span&gt; is the &amp;quot;state action&amp;quot; value function, also known as the quality function. It is the expected return starting from state &lt;span class=&quot;math-container&quot;&gt;$s$&lt;/span&gt;, taking action &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt;, then following policy &lt;span class=&quot;math-container&quot;&gt;$\pi$&lt;/span&gt;. It's focusing on the particular action at the particular state.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$Q^\pi(s, a) = E_\pi \{G_t | s_t = s, a_t = a\}$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;The relationship between &lt;span class=&quot;math-container&quot;&gt;$Q^\pi$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$V^\pi$&lt;/span&gt; (the value of being in that state) is:&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$V^\pi(s) = \sum_{a ∈ A} \pi (a|s) * Q^\pi(s,a)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;You sum every state action-value multiplied by the probability of taking that action (given by the policy &lt;span class=&quot;math-container&quot;&gt;$\pi(a|s)$&lt;/span&gt;).&lt;/p&gt;&#xA;&lt;p&gt;If you think of the grid world example, you multiply the probability of (up/down/right/left) with the one step ahead of the state value of (up/down/right/left).&lt;/p&gt;&#xA;" OwnerUserId="37932" LastEditorUserId="85045" LastEditDate="2023-02-09T02:47:02.507" LastActivityDate="2023-02-09T02:47:02.507" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31792" PostTypeId="2" ParentId="19997" CreationDate="2018-05-18T02:38:32.573" Score="4" Body="&lt;p&gt;Using the notation from the wikipedia page, the convolution in a CNN is going to be the kernel $g$ of which we will learn some weights in order to extract the information we need and then perhaps apply an activation function.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Discrete convolutions&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;From the wikipedia page the convolution is described as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$(f * g)[n] = \sum_{m=-\inf}^{\inf} f[m]g[n-m]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example assuming $a$ is the function $f$ and $b$ is the convolution function $g$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/0RTyh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/0RTyh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To solve this we can use the equation first we flip the function $b$ vertically, due to the $-m$ that appears in the equation. Then we will calculate the summation for each value of $n$. Whilst changing $n$, the original function does not move, however the convolution function is shifted accordingly. Starting at $n=0$,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[0] = \sum_m a[m]b[-m] = 0 * 0.25 + 0 * 0.5 + 1 * 1 + 0.5 * 0 + 1 * 0 + 1 * 0 = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[1] = \sum_m a[m]b[-m] = 0 * 0.25 + 1 * 0.5 + 0.5 * 1 + 1 * 0 + 1 * 0 = 1$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[2] = \sum_m a[m]b[-m] = 1 * 0.25 + 0.5 * 0.5 + 1 * 1 + 1 * 0 + 1 * 0 = 1.5$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[3] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0.25 + 1 * 0.5 + 1 * 1 = 1.625$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[4] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0.25 + 1 * 0.5 + 0 * 1 = 0.75$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$c[5] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0 + 1 * 0.25 + 0 * 0.5 * 0 * 1 = 0.25$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see that is exactly what we get on the plot $c[n]$. So we shifted around the function $b[n]$ over the function $a[n]$.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;2D Discrete Convolution&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For example, if we have the matrix in green &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3rkab.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3rkab.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;with the convolution filter &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/6Bhkp.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6Bhkp.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. Very much like the wikipedia page shows, this kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/s47gB.gif&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s47gB.gif&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;taken from the &lt;a href=&quot;https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; that @Hobbes reference. You will notice that there is no flip of the kernel $g$ like we did for the explicit computation of the convolution above. This is a matter of notation as @Media points out. This should be called cross-correlation. However, computationally this difference does not affect the performance of the algorithm because the kernel is being trained such that its weights are best suited for the operation, thus adding the flip operation would simply make the algorithm learn the weights in different cells of the kernel to accommodate the flip. So we can omit the flip. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-18T02:38:32.573" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31793" PostTypeId="1" CreationDate="2018-05-18T03:39:58.810" Score="2" ViewCount="174" Body="&lt;p&gt;Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer in each CNN. Likewise, I can also extract features using conventional hand-crafted techniques, such as: HOG, SIFT, LBP, LTP, Local Phase Quantization, Rotation Invariant Co-occurrence Local Binary Patterns, etc. Thus, I can obtain a very high-dimensional feature vector of an image that concatenates the individual features vectors outputted by these individual algorithms. Given these features, and given a data set of images over which I want to perform similar image retrieval (i.e. finding the &lt;strong&gt;top-k&lt;/strong&gt; most similar images to a query image X), what would be the most appropriate ways to implement this task?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One possible idea I have in mind is to learn an image similarity embedding in euclidean space by training a neural network that would receive as input the aforementioned feature vectors, and perhaps down-sampled versions of the image as well, and output a lower dimensional embedding vector that ideally should place similar images close to each other and dissimilar images far apart. And I could train this network using for example Siamese Loss or Triplet Loss. The challenge of this approach though is generating the labels for the (supervised) training itself. For example, in the case of the Triplet Loss I would need to sample triplets (Q,X,Y) and &lt;em&gt;somehow&lt;/em&gt; determine which one between X and Y is most similar to Q, in order to generate the label for the triplet (i.e., in order to &quot;teach&quot; the network I need to know the answers myself beforehand, but how? I guess this is domain dependent, but think of challenging cases where you have very heterogeneous images, such as photography galleries, artwork galleries, etc).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Anyways, this is just an idea and by no means I pretend to mean this is the right approach. I'm open to new suggestions and insights about how to solve this task.&lt;/p&gt;&#xA;" OwnerUserId="46305" LastActivityDate="2018-05-18T03:39:58.810" Title="How to combine heterogeneous image features extracted with different algorithms for similar image retrieval?" Tags="|deep-learning|feature-engineering|computer-vision|information-retrieval|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31794" PostTypeId="1" CreationDate="2018-05-18T04:42:06.370" Score="1" ViewCount="34" Body="&lt;p&gt;After executing this code: &lt;a href=&quot;https://www.kaggle.com/naveenc131/cnn-with-accuracy-of-98/code&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/naveenc131/cnn-with-accuracy-of-98/code&lt;/a&gt;, an error occurred:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ValueError: error while checking input: it is expected that conv2d_1_input has 4 dimensions, but got an array with the form (0, 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Dataset: &lt;a href=&quot;https://www.kaggle.com/moltean/fruits&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/moltean/fruits&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;What&lt;/strong&gt; is the cause of the error, and &lt;strong&gt;how&lt;/strong&gt; to fix the code?&lt;/p&gt;&#xA;" OwnerUserId="50234" LastActivityDate="2018-05-18T04:42:06.370" Title="How to reform array" Tags="|machine-learning|python|keras|dataset|tensorflow|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-18T20:05:17.323" ContentLicense="CC BY-SA 4.0" />
  <row Id="31795" PostTypeId="1" CreationDate="2018-05-18T05:44:36.947" Score="3" ViewCount="6410" Body="&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import preprocessing, cross_validation, neighbors&#xA;import pandas as pd&#xA;from sklearn.linear_model import LinearRegression&#xA;&#xA;df = pd.read_csv('Downloads/breast-cancer-wisconsin.data.txt',skiprows=1)&#xA;df.replace('?', -99999, inplace=True)&#xA;df.drop('id', 1, inplace=True )&#xA;&#xA;X= np.array(df.drop(['class'],1))&#xA;y= np.array(df['class'])&#xA;&#xA;X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)&#xA;&#xA;#clf = neighbors.KNeighborsClassifier()&#xA;clf = LinearRegression(normalize=True)&#xA;clf.fit(X_train, y_train)&#xA;&#xA;accuracy= clf.score(X_test, y_test)&#xA;print(accuracy)&#xA;&#xA;example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])&#xA;example_measures = example_measures.reshape(1,-1)&#xA;&#xA;prediction = clf.predict(example_measures)     ##(example_measures)&#xA;&#xA;print(prediction)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Problem arises when  I run the above command line at Ubuntu or Anaconda:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;ValueError: query data dimension must match training data dimension&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How to solve that problem ? I am sure that by method of isolating individual commandline-- and find it appears Error at :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prediction = clf.predict(example_measures)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I try to use : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prediction = clf.predict(X_test).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It is ok.&#xA;I really want to predict the example I create. How can I change the code?&lt;/p&gt;&#xA;" OwnerUserId="52326" LastEditorUserId="29575" LastEditDate="2018-05-18T12:25:52.553" LastActivityDate="2018-05-18T12:38:36.853" Title="Query data dimension" Tags="|machine-learning-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31796" PostTypeId="1" CreationDate="2018-05-18T06:28:35.560" Score="1" ViewCount="58" Body="&lt;p&gt;My question is in the title. I'm currently looking for a paper or academic reference to that algorithm, even in a novel framework.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you for all possible replies,&lt;/p&gt;&#xA;" OwnerUserId="50106" LastEditorUserId="1330" LastEditDate="2021-09-23T15:52:45.933" LastActivityDate="2024-03-30T14:01:55.213" Title="Is there mathematical verification for TBPTT (truncated back propagation through time)" Tags="|rnn|gradient-descent|mathematics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31797" PostTypeId="1" CreationDate="2018-05-18T06:58:19.073" Score="2" ViewCount="292" Body="&lt;p&gt;while learning about ROC curves I got confused by the how these are made.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am considering here the Iris flower classification problem.&#xA;To calculate TPR we can use $$TPR=\frac{True \ positive}{True\ positive + \ False \ Negative }$$&#xA;Now in this case first we feed in the test data and find out the TPR and FPR by using the formulas above.But in this case how we get different array size.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt;fpr = dict()&#xA;&amp;gt;&amp;gt;&amp;gt;tpr = dict()&#xA;&amp;gt;&amp;gt;&amp;gt;roc_auc = dict()&#xA;&amp;gt;&amp;gt;&amp;gt;for i in range(n_classes):&#xA;       fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])&#xA;       roc_auc[i] = auc(fpr[i], tpr[i])&#xA;&#xA;&#xA;tpr(true positive rate)&#xA;0: array([ 0.04761905,  1.        ,  1.        ]),  1: array([&#xA;0.03333333,  0.03333333,  0.1       ,  0.1       ,  0.2       ,&#xA;         0.2       ,  0.23333333,  0.23333333,  0.36666667,  0.36666667,&#xA;         0.4       ,  0.4       ,  0.7       ,  0.7       ,  0.73333333,&#xA;         0.73333333,  0.76666667,  0.76666667,  0.8       ,  0.8       ,&#xA;         0.83333333,  0.83333333,  0.86666667,  0.86666667,  0.9       ,&#xA;         0.9       ,  0.93333333,  0.93333333,  0.96666667,  0.96666667,&#xA;         1.        ,  1.        ]),  2: array([ 0.04166667,  0.79166667,  0.79166667,  0.95833333,  0.95833333,&#xA;         1.        ,  1.        ]),&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-05-18T13:06:59.967" Title="how I get different array size in calculating tpr?" Tags="|classification|scikit-learn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31798" PostTypeId="1" CreationDate="2018-05-18T07:12:19.217" Score="3" ViewCount="482" Body="&lt;p&gt;I am working on a problem with 4 inputs and 1 continuous output variable. The sum of all values of the 4 input variables is always 1.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;a1+a2+a3+a4=1&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, they are correlated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: should I use all 4 variables for neural network training? Or, should I use any 3 of them to get rid of correlation? Is there any problem if I use all 4?&lt;/p&gt;&#xA;" OwnerUserId="50828" LastActivityDate="2018-05-18T19:43:42.243" Title="Discarding correlation among inputs in a neural network" Tags="|neural-network|correlation|dimensionality-reduction|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31801" PostTypeId="1" CreationDate="2018-05-18T08:21:43.133" Score="1" ViewCount="545" Body="&lt;p&gt;I'm studying the decision trees in Data Mining.&#xA;A weak point of the &lt;strong&gt;information gain&lt;/strong&gt; criterion is that it can lead to an overfitting, a solution can be the use of the &lt;strong&gt;gain ratio&lt;/strong&gt; criterion.&lt;/p&gt;&#xA;&lt;p&gt;May this last criterion suffer from underfitting ? If yes, when ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="1330" LastEditDate="2021-07-09T01:49:51.940" LastActivityDate="2021-07-09T01:49:51.940" Title="Information Gain vs Gain Ratio in decision trees" Tags="|classification|data-mining|decision-trees|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31802" PostTypeId="1" CreationDate="2018-05-18T08:30:05.307" Score="1" ViewCount="846" Body="&lt;p&gt;Since decision tree algorithm splits the training dataset one feature at a time, how the heck is possibly that it suffers from curse of dimensionality ? &lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2024-01-01T13:46:15.607" Title="Decision trees and Curse of Dimensionality" Tags="|classification|decision-trees|dimensionality-reduction|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31803" PostTypeId="1" CreationDate="2018-05-18T09:17:17.370" Score="1" ViewCount="588" Body="&lt;pre&gt;&lt;code&gt;print(model.predict(image))&#xA;print(model.predict_proba(image))&#xA;print(model.predict_classes(image))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using the above different predict methods I always get predictions like&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[ 1.  0.  0.  0.]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;but I want the probabilities of each classes something like this&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[ 0.8  0.05  0.05  0.1]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This is the model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential([&#xA;    Conv2D(32, (3, 3), input_shape=input_shape, padding='same',activation='relu'),&#xA;    Conv2D(32, (3, 3), activation='relu', padding='same'),&#xA;    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),&#xA;    Conv2D(64, (3, 3), activation='relu', padding='same', ),&#xA;    Conv2D(64, (3, 3), activation='relu', padding='same', ),&#xA;    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),&#xA;    Conv2D(128, (3, 3), activation='relu', padding='same', ),&#xA;    Conv2D(128, (3, 3), activation='relu', padding='same', ),&#xA;    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),&#xA;    Conv2D(256, (3, 3), activation='relu', padding='same', ),&#xA;    Conv2D(256, (3, 3), activation='relu', padding='same', ),&#xA;    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),&#xA;    Conv2D(512, (3, 3), activation='relu', padding='same', ),&#xA;    Conv2D(512, (3, 3), activation='relu', padding='same', ),&#xA;    MaxPooling2D(pool_size=(2, 2), strides=(2, 2)),&#xA;    Flatten(),&#xA;    Dense(512, activation='relu'),&#xA;    Dense(num_classes, activation='softmax')&#xA;])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Compiled as &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;sgd = SGD(lr=0.0001,momentum=0.9)&#xA;model.compile(loss = &quot;categorical_crossentropy&quot;, optimizer =sgd,metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Not sure if this has any effects, but this is how I loaded the data&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;train_datagen = ImageDataGenerator(&#xA;      rescale=1./255,&#xA;  )&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;train_generator = train_datagen.flow_from_directory(folder+'apex/train',&#xA;                                                      target_size=(200, 400),&#xA;                                                      batch_size=batch_size,&#xA;                                                      class_mode='categorical')&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="52335" LastEditorUserId="52335" LastEditDate="2018-05-18T11:27:42.773" LastActivityDate="2018-05-18T11:27:42.773" Title="keras categorical CNN prediction always output [[ 0. 0. 0. 0. 0. 0. 1. 0.]] No probability for other classes?" Tags="|machine-learning|keras|cnn|probability|" AnswerCount="0" CommentCount="2" ClosedDate="2020-10-26T06:04:14.367" ContentLicense="CC BY-SA 4.0" />
  <row Id="31804" PostTypeId="1" AcceptedAnswerId="31813" CreationDate="2018-05-18T11:07:32.130" Score="6" ViewCount="517" Body="&lt;p&gt;I'm simply trying to repeat a benchmark from the sklearn's &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_vs_fa_model_selection.html#sphx-glr-auto-examples-decomposition-plot-pca-vs-fa-model-selection-py&quot; rel=&quot;noreferrer&quot;&gt;docs&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The unclear part is: &lt;code&gt;n_components = np.arange(0, n_features, 5)&lt;/code&gt;.&#xA;They are applying a PCA transform with 0 components! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can somebody, please, explain, what's the mathematical meaning of this transform?&lt;/p&gt;&#xA;" OwnerUserId="29931" LastActivityDate="2018-05-18T15:14:43.320" Title="Sklearn PCA with zero components example" Tags="|scikit-learn|pca|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31805" PostTypeId="2" ParentId="31795" CreationDate="2018-05-18T12:31:23.073" Score="2" Body="&lt;p&gt;How many columns do &lt;code&gt;X_train&lt;/code&gt; and &lt;code&gt;X_test&lt;/code&gt; have? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd imagine (though can't confirm, as I don't have access to your data) that they have fewer (or more) than 18 columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is because your code &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])&#xA;example_measures = example_measures.reshape(1,-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;produces an array of shape &lt;code&gt;(1, 18)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: I've tried matching your dataset, and got the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;from sklearn import preprocessing, cross_validation, neighbors&#xA;import pandas as pd&#xA;from sklearn.linear_model import LinearRegression&#xA;from sklearn.datasets import load_breast_cancer&#xA;&#xA;X= load_breast_cancer().data&#xA;y= load_breast_cancer().target&#xA;&#xA;X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)&#xA;&#xA;clf = LinearRegression(normalize=True)&#xA;clf.fit(X_train, y_train)&#xA;&#xA;accuracy= clf.score(X_test, y_test)&#xA;print(accuracy)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If I call &lt;code&gt;X.shape&lt;/code&gt;, I get &lt;code&gt;(569, 30)&lt;/code&gt;. So, if you want to make your own array to pass to &lt;code&gt;clf&lt;/code&gt;, it needs to have 30 columns (one for each feature).&lt;/p&gt;&#xA;" OwnerUserId="52109" LastEditorUserId="52109" LastEditDate="2018-05-18T12:38:36.853" LastActivityDate="2018-05-18T12:38:36.853" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31806" PostTypeId="1" AcceptedAnswerId="31807" CreationDate="2018-05-18T12:38:51.923" Score="3" ViewCount="1259" Body="&lt;p&gt;Let's say I already have a logistic regression model (or other) with N number of explanatory variables and is 70% accurate. Now if there are other variables available, how would I test if the new variables would improve my accuracy without building new model.&lt;/p&gt;&#xA;" OwnerUserId="28584" LastActivityDate="2022-09-16T03:12:03.287" Title="Adding new variable to model" Tags="|regression|feature-selection|logistic-regression|model-selection|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31807" PostTypeId="2" ParentId="31806" CreationDate="2018-05-18T12:58:54.113" Score="5" Body="&lt;p&gt;I do not think you can estimate the effect of a variable without adding it to the model. This is because the effect of a variable on the model's discriminatory power depends on &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the strength of association between the outcome variable and the new variable&lt;/li&gt;&#xA;&lt;li&gt;whether the new variable are collinear with some of the old variables&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You could in principle estimate both strength of association and collinearity, but it probably is bad practice and would result in overfitting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also in general I think it is best to not use accuracy to evaluate a logistic regression (&lt;a href=&quot;http://www.fharrell.com/post/class-damage/&quot; rel=&quot;noreferrer&quot;&gt;see ref&lt;/a&gt;) but rather a proper scoring rule like Brier Score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Further, when comparing two nested models (i.e. where one model contains a subset of the variables of the other model) I believe that best practice is to compare the AIC or BIC, or perform a likelihood ratio test.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastActivityDate="2018-05-18T12:58:54.113" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31808" PostTypeId="2" ParentId="31797" CreationDate="2018-05-18T13:06:59.967" Score="2" Body="&lt;p&gt;You get different array sizes for your different classes because the number of points in your ROC curve depends on the number of unique predictions. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model only predicts 0, .5, and 1 your ROC curve will have three points. If your model predicts 100 different probabilities, your ROC curve will have ~100 points. (In practice, the length of the ROC curve may have a couple extra points where the points (0,0) and (1,1) are appended. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you look at &lt;code&gt;np.unique(y_score[:,i]).size&lt;/code&gt; for different &lt;code&gt;i&lt;/code&gt; you should see the differences. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-05-18T13:06:59.967" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31810" PostTypeId="2" ParentId="12554" CreationDate="2018-05-18T13:55:28.723" Score="28" Body="&lt;p&gt;I was curious about this and made a few tests.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold.&#xA;Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems that when the correlation between two columns is 1, xgboost removes the extra column before calculating the model, so the importance is not affected.&#xA;However, when you add a column that is partially correlated to another, thus with a lower coefficient, the importance of the original variable x is lowered. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example if I add a variable xy = x + y, the importance of both x and y decrease. Similarly, the importance of x decreases if I add new variables with r=0.4, 0.5 or 0.6, although just by a bit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that collinearity is not a problem for boosting when you calculate the accuracy of the model, because the decision tree doesn’t care which one of the variables is used. However it might affect the importance of the variables, because removing one of the two correlated variables doesn't have a big impact on the accuracy of the model, given that the other contains similar information.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse)&#xA;library(xgboost)&#xA;&#xA;evaluate_model = function(dataset) {&#xA;    print(&quot;Correlation matrix&quot;)&#xA;    dataset %&amp;gt;% select(-cut, -color, -clarity, -price) %&amp;gt;% cor %&amp;gt;% print&#xA;&#xA;    print(&quot;running model&quot;)&#xA;    diamond.model = xgboost(&#xA;        data=dataset %&amp;gt;% select(-cut, -color, -clarity, -price) %&amp;gt;% as.matrix, &#xA;        label=dataset$price &amp;gt; 400, &#xA;        max.depth=15, nrounds=30, nthread=2, objective = &quot;binary:logistic&quot;,&#xA;        verbose=F&#xA;        )&#xA;&#xA;    print(&quot;Importance matrix&quot;)&#xA;    importance_matrix &amp;lt;- xgb.importance(model = diamond.model)&#xA;    importance_matrix %&amp;gt;% print&#xA;    xgb.plot.importance(importance_matrix)&#xA;    }&#xA;&#xA;&amp;gt; diamonds %&amp;gt;% head&#xA;carat   cut color   clarity depth   table   price   x   y   z&#xA;0.23    Ideal   E   SI2 61.5    55  326 3.95    3.98    2.43&#xA;0.21    Premium E   SI1 59.8    61  326 3.89    3.84    2.31&#xA;0.23    Good    E   VS1 56.9    65  327 4.05    4.07    2.31&#xA;0.29    Premium I   VS2 62.4    58  334 4.20    4.23    2.63&#xA;0.31    Good    J   SI2 63.3    58  335 4.34    4.35    2.75&#xA;0.24    Very Good   J   VVS2    62.8    57  336 3.94    3.96    2.48&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Evaluate a model on the diamonds data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We predict whether the price is higher than 400, given all numeric variables available (carat, depth, table, x, y, x)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that x is the most important variable, with an importance gain score of 0.375954.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;evaluate_model(diamonds)&#xA;    [1] &quot;Correlation matrix&quot;&#xA;               carat       depth      table           x           y          z&#xA;    carat 1.00000000  0.02822431  0.1816175  0.97509423  0.95172220 0.95338738&#xA;    depth 0.02822431  1.00000000 -0.2957785 -0.02528925 -0.02934067 0.09492388&#xA;    table 0.18161755 -0.29577852  1.0000000  0.19534428  0.18376015 0.15092869&#xA;    x     0.97509423 -0.02528925  0.1953443  1.00000000  0.97470148 0.97077180&#xA;    y     0.95172220 -0.02934067  0.1837601  0.97470148  1.00000000 0.95200572&#xA;    z     0.95338738  0.09492388  0.1509287  0.97077180  0.95200572 1.00000000&#xA;    [1] &quot;running model&quot;&#xA;    [1] &quot;Importance matrix&quot;&#xA;       Feature       Gain      Cover  Frequency&#xA;    1:       x 0.37595419 0.54788335 0.19607102&#xA;    2:   carat 0.19699839 0.18015576 0.04873442&#xA;    3:   depth 0.15358261 0.08780079 0.27767284&#xA;    4:       y 0.11645929 0.06527969 0.18813751&#xA;    5:   table 0.09447853 0.05037063 0.17151492&#xA;    6:       z 0.06252699 0.06850978 0.11786929&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model trained on Diamonds, adding a variable with r=1 to x&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here we add a new column, which however doesn't add any new information, as it is perfectly correlated to x.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that this new variable is not present in the output. It seems that xgboost automatically removes perfectly correlated variables before starting the calculation. The importance gain of x is the same, 0.3759.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diamonds_xx = diamonds %&amp;gt;%&#xA;    mutate(xx = x + runif(1, -1, 1))&#xA;evaluate_model(diamonds_xx)&#xA;[1] &quot;Correlation matrix&quot;&#xA;           carat       depth      table           x           y          z&#xA;carat 1.00000000  0.02822431  0.1816175  0.97509423  0.95172220 0.95338738&#xA;depth 0.02822431  1.00000000 -0.2957785 -0.02528925 -0.02934067 0.09492388&#xA;table 0.18161755 -0.29577852  1.0000000  0.19534428  0.18376015 0.15092869&#xA;x     0.97509423 -0.02528925  0.1953443  1.00000000  0.97470148 0.97077180&#xA;y     0.95172220 -0.02934067  0.1837601  0.97470148  1.00000000 0.95200572&#xA;z     0.95338738  0.09492388  0.1509287  0.97077180  0.95200572 1.00000000&#xA;xx    0.97509423 -0.02528925  0.1953443  1.00000000  0.97470148 0.97077180&#xA;               xx&#xA;carat  0.97509423&#xA;depth -0.02528925&#xA;table  0.19534428&#xA;x      1.00000000&#xA;y      0.97470148&#xA;z      0.97077180&#xA;xx     1.00000000&#xA;[1] &quot;running model&quot;&#xA;[1] &quot;Importance matrix&quot;&#xA;   Feature       Gain      Cover  Frequency&#xA;1:       x 0.37595419 0.54788335 0.19607102&#xA;2:   carat 0.19699839 0.18015576 0.04873442&#xA;3:   depth 0.15358261 0.08780079 0.27767284&#xA;4:       y 0.11645929 0.06527969 0.18813751&#xA;5:   table 0.09447853 0.05037063 0.17151492&#xA;6:       z 0.06252699 0.06850978 0.11786929&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model trained on Diamonds, adding a column for x + y&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We add a new column xy = x + y. This is partially correlated to both x and y.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the importance of x and y is slightly reduced, going from 0.3759 to 0.3592 for x, and from 0.116 to 0.079 for y.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;diamonds_xy = diamonds %&amp;gt;%&#xA;    mutate(xy=x+y)&#xA;evaluate_model(diamonds_xy)&#xA;&#xA;[1] &quot;Correlation matrix&quot;&#xA;           carat       depth      table           x           y          z&#xA;carat 1.00000000  0.02822431  0.1816175  0.97509423  0.95172220 0.95338738&#xA;depth 0.02822431  1.00000000 -0.2957785 -0.02528925 -0.02934067 0.09492388&#xA;table 0.18161755 -0.29577852  1.0000000  0.19534428  0.18376015 0.15092869&#xA;x     0.97509423 -0.02528925  0.1953443  1.00000000  0.97470148 0.97077180&#xA;y     0.95172220 -0.02934067  0.1837601  0.97470148  1.00000000 0.95200572&#xA;z     0.95338738  0.09492388  0.1509287  0.97077180  0.95200572 1.00000000&#xA;xy    0.96945349 -0.02750770  0.1907100  0.99354016  0.99376929 0.96744200&#xA;              xy&#xA;carat  0.9694535&#xA;depth -0.0275077&#xA;table  0.1907100&#xA;x      0.9935402&#xA;y      0.9937693&#xA;z      0.9674420&#xA;xy     1.0000000&#xA;[1] &quot;running model&quot;&#xA;[1] &quot;Importance matrix&quot;&#xA;   Feature       Gain      Cover  Frequency&#xA;1:       x 0.35927767 0.52924339 0.15952849&#xA;2:   carat 0.17881931 0.18472506 0.04793713&#xA;3:   depth 0.14353540 0.07482622 0.24990177&#xA;4:   table 0.09202059 0.04714548 0.16267191&#xA;5:      xy 0.08203819 0.04706267 0.13555992&#xA;6:       y 0.07956856 0.05284980 0.13595285&#xA;7:       z 0.06474029 0.06414738 0.10844794&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Model trained on Diamonds data, modified adding redundant columns&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We add three new columns that are correlated to x (r = 0.4, 0.5 and 0.6) and see what happens.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that the importance of x gets reduced, dropping from 0.3759 to 0.279.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#' given a vector of values (e.g. diamonds$x), calculate three new vectors correlated to it&#xA;#' &#xA;#' Source: https://stat.ethz.ch/pipermail/r-help/2007-April/128938.html&#xA;calculate_correlated_vars = function(x1) {&#xA;&#xA;    # create the initial x variable&#xA;    #x1 &amp;lt;- diamonds$x&#xA;&#xA;    # x2, x3, and x4 in a matrix, these will be modified to meet the criteria&#xA;    x234 &amp;lt;- scale(matrix( rnorm(nrow(diamonds) * 3), ncol=3 ))&#xA;&#xA;    # put all into 1 matrix for simplicity&#xA;    x1234 &amp;lt;- cbind(scale(x1),x234)&#xA;&#xA;    # find the current correlation matrix&#xA;    c1 &amp;lt;- var(x1234)&#xA;&#xA;    # cholesky decomposition to get independence&#xA;    chol1 &amp;lt;- solve(chol(c1))&#xA;&#xA;    newx &amp;lt;-  x1234 %*% chol1 &#xA;&#xA;    # check that we have independence and x1 unchanged&#xA;    zapsmall(cor(newx))&#xA;    all.equal( x1234[,1], newx[,1] )&#xA;&#xA;    # create new correlation structure (zeros can be replaced with other r vals)&#xA;    newc &amp;lt;- matrix( &#xA;    c(1  , 0.4, 0.5, 0.6, &#xA;      0.4, 1  , 0  , 0  ,&#xA;      0.5, 0  , 1  , 0  ,&#xA;      0.6, 0  , 0  , 1  ), ncol=4 )&#xA;&#xA;    # check that it is positive definite&#xA;    eigen(newc)&#xA;&#xA;    chol2 &amp;lt;- chol(newc)&#xA;&#xA;    finalx &amp;lt;- newx %*% chol2 * sd(x1) + mean(x1)&#xA;&#xA;    # verify success&#xA;    mean(x1)&#xA;    colMeans(finalx)&#xA;&#xA;    sd(x1)&#xA;    apply(finalx, 2, sd)&#xA;&#xA;    zapsmall(cor(finalx))&#xA;    #pairs(finalx)&#xA;&#xA;    all.equal(x1, finalx[,1])&#xA;    finalx&#xA;}&#xA;finalx = calculate_correlated_vars(diamonds$x)&#xA;diamonds_cor = diamonds&#xA;diamonds_cor$x5 = finalx[,2]&#xA;diamonds_cor$x6 = finalx[,3]&#xA;diamonds_cor$x7 = finalx[,4]&#xA;evaluate_model(diamonds_cor)&#xA;[1] &quot;Correlation matrix&quot;&#xA;           carat        depth       table           x           y          z&#xA;carat 1.00000000  0.028224314  0.18161755  0.97509423  0.95172220 0.95338738&#xA;depth 0.02822431  1.000000000 -0.29577852 -0.02528925 -0.02934067 0.09492388&#xA;table 0.18161755 -0.295778522  1.00000000  0.19534428  0.18376015 0.15092869&#xA;x     0.97509423 -0.025289247  0.19534428  1.00000000  0.97470148 0.97077180&#xA;y     0.95172220 -0.029340671  0.18376015  0.97470148  1.00000000 0.95200572&#xA;z     0.95338738  0.094923882  0.15092869  0.97077180  0.95200572 1.00000000&#xA;x5    0.39031255 -0.007507604  0.07338484  0.40000000  0.38959178 0.38734145&#xA;x6    0.48879000 -0.016481580  0.09931705  0.50000000  0.48835896 0.48487442&#xA;x7    0.58412252 -0.013772440  0.11822089  0.60000000  0.58408881 0.58297414&#xA;                 x5            x6            x7&#xA;carat  3.903125e-01  4.887900e-01  5.841225e-01&#xA;depth -7.507604e-03 -1.648158e-02 -1.377244e-02&#xA;table  7.338484e-02  9.931705e-02  1.182209e-01&#xA;x      4.000000e-01  5.000000e-01  6.000000e-01&#xA;y      3.895918e-01  4.883590e-01  5.840888e-01&#xA;z      3.873415e-01  4.848744e-01  5.829741e-01&#xA;x5     1.000000e+00  5.925447e-17  8.529781e-17&#xA;x6     5.925447e-17  1.000000e+00  6.683397e-17&#xA;x7     8.529781e-17  6.683397e-17  1.000000e+00&#xA;[1] &quot;running model&quot;&#xA;[1] &quot;Importance matrix&quot;&#xA;   Feature       Gain      Cover  Frequency&#xA;1:       x 0.27947762 0.51343709 0.09748172&#xA;2:   carat 0.13556427 0.17401365 0.02680747&#xA;3:      x5 0.13369515 0.05267688 0.18155971&#xA;4:      x6 0.12968400 0.04804315 0.19821284&#xA;5:      x7 0.10600238 0.05148826 0.16450041&#xA;6:   depth 0.07087679 0.04485760 0.11251015&#xA;7:       y 0.06050565 0.03896716 0.08245329&#xA;8:   table 0.04577057 0.03135677 0.07554833&#xA;9:       z 0.03842355 0.04515944 0.06092608&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52341" LastEditorUserId="52341" LastEditDate="2018-05-18T14:01:33.610" LastActivityDate="2018-05-18T14:01:33.610" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31811" PostTypeId="1" AcceptedAnswerId="31816" CreationDate="2018-05-18T14:04:24.277" Score="2" ViewCount="261" Body="&lt;p&gt;If, for example, I have a neural network for classifying dog breeds, and I feed it an image of some dog, inherently it shouldn't matter whether I feed it the original image or the image, mirrored horizontally.&lt;br&gt;&#xA;I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But,&#xA;by my research modern networks don't use filters like that. Well... Why? what disadvantages does this architecture have that I can't find a single mention of such idea?&lt;/p&gt;&#xA;" OwnerUserId="52342" LastEditorUserId="29169" LastEditDate="2021-02-13T00:57:26.533" LastActivityDate="2021-02-13T00:57:26.533" Title="Why don't convolutional computer vision networks use horizontally - symmetric filters?" Tags="|computer-vision|convolution|convolutional-neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31812" PostTypeId="1" CreationDate="2018-05-18T14:42:06.383" Score="0" ViewCount="2594" Body="&lt;p&gt;Hi everybody can somebody help me!! I don't know what's wrong in my code and why it gives me this error ???&lt;a href=&quot;https://i.stack.imgur.com/KzN8p.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KzN8p.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KCxNk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KCxNk.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52343" LastEditorUserId="57909" LastEditDate="2018-08-23T18:39:40.573" LastActivityDate="2018-08-23T18:39:40.573" Title="ValueError: Error when checking target: expected dense_1 to have shape (7,) but got array with shape (1,)" Tags="|python|neural-network|keras|rnn|lstm|" AnswerCount="1" CommentCount="1" ClosedDate="2018-08-23T18:44:07.130" ContentLicense="CC BY-SA 4.0" />
  <row Id="31813" PostTypeId="2" ParentId="31804" CreationDate="2018-05-18T14:51:38.210" Score="4" Body="&lt;p&gt;Think of it this way: a PCA &quot;transform&quot; with $k$ components essentially approximates your $n$-dimensional data points by projecting them onto a $k$-dimensional linear subspace, trying not to loose too much data variance along the way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More precisely, what you are doing is representing your original points $y \in R^n$ as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$y \approx \mu + Vx$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $x \in R^k$ are the lower-dimensional &quot;new coordinates&quot;, $\mu \in R^n$ is the mean of your data and $V\in R^{n \times k}$ is the matrix of principal component vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &quot;new coordinates&quot; $x$ tell you how many steps you need to make along the $k$ principal components in order to reach the best possible linear approximation to $y$ by starting your trip from $\mu$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, if $k=0$ the model becomes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ x \approx \mu. $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, you are modeling all of your data as a single, fixed center point. Of course, you do not need to store any &quot;new coordinates&quot; here (because you do not need to move away from the mean), hence it does not make much sense as a &quot;transform&quot;, but it is, none the less, a proper probabilistic model (a maximum-likelihood fit for a Gaussian distribution of errors, to be precise).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In particular, you can speak about the log-likelihood of data under this model (which is, up to an affine transform, equal to the sum of squared errors here, but &lt;a href=&quot;http://www.miketipping.com/papers/met-mppca.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;is not as trivial as you might think in the general case&lt;/a&gt;) and we can compare various models, choosing the one with the best likelihood. This is exactly what is done in the example from the docs you mention in the question.&lt;/p&gt;&#xA;" OwnerUserId="14519" LastEditorUserId="14519" LastEditDate="2018-05-18T15:14:43.320" LastActivityDate="2018-05-18T15:14:43.320" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31814" PostTypeId="1" CreationDate="2018-05-18T15:06:04.353" Score="1" ViewCount="183" Body="&lt;p&gt;I have some user data where each user has a certain pattern of being at different places for some time. I would like to create a model which will cluster/classify these users based on these patterns and the time spent at each place. So suppose user patterns are like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;Place_1(60 min)- Place_2(30 min)- Place_5(45 min)- user 1 -label(1)&#xA;&#xA;Place_1(60 min)- Place_2(60 min)- Place_5(45 min)- user 2 -label(2)&#xA;&#xA;Place_1(60 min)- Place_2(60 min)- Place_5(40 min)- user 3 -label(2)&#xA;&#xA;Place_2(60 min)- Place_1(60 min)- Place_5(45 min)- user 4 -label(3)&#xA;&#xA;Place_2(60 min)- Place_1(60 min)- Place_5(45 min)- user 5 -label(3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;They should be clustered/classified as:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;1- User 1&#xA;&#xA;2- User 2, User 3&#xA;&#xA;3- User 4, User 5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;The time duration is continuous. Also, I already have labels for these patterns so I can do classification as well as clustering. I initially thought of doing kmeans clustering on these patterns, but  introducing the duration of stay at each place is messing the clustering up. I am currently using a random forest classifier, but the results are not as promising. Any help would be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="39102" LastEditorUserId="29169" LastEditDate="2021-07-10T17:03:04.443" LastActivityDate="2024-01-20T03:02:45.300" Title="Clustering/ Classifying users based on sequence of action and time" Tags="|machine-learning|classification|clustering|sequential-pattern-mining|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31815" PostTypeId="1" CreationDate="2018-05-18T15:21:34.033" Score="1" ViewCount="1837" Body="&lt;p&gt;I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my data points and LOF score to have a supervised model [regression /classification].     &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is which should be approach out of&lt;br&gt;&#xA; 1) classification ( taking a cutoff of LOF score and after having binary variable , build classification model) and   use this classification model to predict outliers at run time.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does any clustering/anomaly detection( using clustering)  is even used at run time ?? Any guidance will help.&lt;/p&gt;&#xA;" OwnerUserId="29412" LastActivityDate="2018-06-18T06:22:17.470" Title="Using local outlier factor score to detect outliers at run time" Tags="|clustering|anomaly-detection|outlier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31816" PostTypeId="2" ParentId="31811" CreationDate="2018-05-18T15:39:09.360" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If two mirrored images are in the same class - e.g. they both show a dog or a cat - that is &lt;em&gt;not&lt;/em&gt; the same as having all the components in the image (lines, textures, shapes) responding well to symmetric filters. In general this is not the case. Even for symmetric looking shapes such as faces, it is only true at a certain scale and specific pose.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;what disadvantages does this architecture have that I can't find a single mention of such idea?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It will work poorly, because the lines, textures and shapes being detected as components of the image are rarely horizontally symmetrical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CNNs cannot easily take advantage of mirroring or rotational invariance - where the class or detection does not vary under mirror or rotation transformations. One thing you can do to improve generalisation for transformations is data augmentation, i.e. performing the non-class-changing transformations of your training images, perhaps randomly on demand during the training process. This does work, but has the opposite impact that you were hoping for in terms of efficiency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The recently-published &lt;a href=&quot;https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b&quot; rel=&quot;nofollow noreferrer&quot;&gt;CapsNet by Geoffrey Hinton's group&lt;/a&gt; may be able to take advantage of more types of transformation invariance. However, this is still in early stages of research, and not clear whether it offers a practical advantage in your case. There are implementations in various frameworks &lt;a href=&quot;https://github.com/XifengGuo/CapsNet-Keras&quot; rel=&quot;nofollow noreferrer&quot;&gt;such as Keras&lt;/a&gt;, that you could try if you are interested.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-05-18T15:46:02.137" LastActivityDate="2018-05-18T15:46:02.137" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31817" PostTypeId="2" ParentId="31584" CreationDate="2018-05-18T15:46:25.687" Score="0" Body="&lt;p&gt;You could be describing a variation of &lt;a href=&quot;https://en.wikipedia.org/wiki/Named-entity_recognition&quot; rel=&quot;nofollow noreferrer&quot;&gt;named-entity recognition&lt;/a&gt; (NER). You have labels/ categories for tokens. Given a corpus (resumes), you want a NER tagger to classify tokens as belonging to one of the labels or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to create a training set of ground-truth / &quot;gold&quot; labels of tokens and labels. Since you are only dealing with nouns, you can run a standard Part-Of-Speech (POS) Tagger then only custom tag the noun phrases. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It best to take an &lt;a href=&quot;https://en.wikipedia.org/wiki/Active_learning_(machine_learning)&quot; rel=&quot;nofollow noreferrer&quot;&gt;active learning&lt;/a&gt; approach. Active learning makes tagging the training set part of the entire machine learning pipeline, thus greatly reducing the number of annotations. &quot;&lt;a href=&quot;https://arxiv.org/pdf/1707.05928.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Active learning for named entity recognition&lt;/a&gt;&quot; is the current state-of-the-art.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Once you have a set of labels you can train NER classifier. The common options are &lt;a href=&quot;https://nlp.stanford.edu/software/CRF-NER.shtml&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stanford Named Entity Recognizer&lt;/a&gt; (NER)  and &lt;a href=&quot;https://spacy.io/usage/training#section-ner&quot; rel=&quot;nofollow noreferrer&quot;&gt;spaCy NER&lt;/a&gt;. A detailed example for Stanford Core NLP can be found &lt;a href=&quot;https://medium.com/swlh/stanford-corenlp-training-your-own-custom-ner-tagger-8119cc7dfc06&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-18T15:46:25.687" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31818" PostTypeId="1" AcceptedAnswerId="31824" CreationDate="2018-05-18T16:00:07.773" Score="1" ViewCount="60" Body="&lt;p&gt;I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dataset A: Company name: Facebook&lt;/li&gt;&#xA;&lt;li&gt;Dataset B: Company name: Facebook, Inc&lt;/li&gt;&#xA;&lt;li&gt;Dataset C: Company name: facebook&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Some other signals like &lt;strong&gt;company url&lt;/strong&gt; exists, but in terms of name matching wondering if text similarity is a good approach for this grouping problem.&lt;/p&gt;&#xA;" OwnerUserId="15613" LastEditorUserId="15613" LastEditDate="2018-05-18T19:15:07.077" LastActivityDate="2018-05-18T19:15:07.077" Title="Grouping company information" Tags="|nlp|similarity|text|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31819" PostTypeId="1" CreationDate="2018-05-18T16:24:37.490" Score="3" ViewCount="25" Body="&lt;p&gt;Let's take some specific book to narrow this example. Atlas shrugged by Ayn Rand. When i'll be saying something like &quot;Ayn Rand's ideas&quot;, i'll mean only those, which are clearly stated in the book.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Humans who read this book, can clearly distinguish sentences, that describe Ayn Rand's point of view from sentences, that oppose it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there trained neural networks, that could distinguish sentences with Ayn Rand's ideas vs sentences with opposing ideas?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If not, are there any reasons, why it's impossible to train such network, if each sentence in such book would be marked as pro-, contra- or neutral to Ayn Rand's ideas? Successfully trained network should be able to distinguish author's point of view in books, where humans can clearly do it.&lt;/p&gt;&#xA;" OwnerUserId="52344" LastActivityDate="2018-06-22T14:47:38.930" Title="Are there trained neural networks, that can distinguish a book's author point from what he stands against?" Tags="|text-mining|sentiment-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31820" PostTypeId="1" CreationDate="2018-05-18T17:07:11.083" Score="3" ViewCount="887" Body="&lt;p&gt;For computer vision tasks using deep learning, should I worry about image size (e.g. 256 x 256) or PPI (pixels per inch)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I find that PPI is not discussed in the computer vision/deep learning literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What's the best PPI for images of size 256 x 256?&lt;/p&gt;&#xA;" OwnerUserId="52347" LastActivityDate="2021-07-06T22:38:23.653" Title="Best image PPI for computer vision deep learning algorithms" Tags="|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31822" PostTypeId="2" ParentId="27700" CreationDate="2018-05-18T17:56:54.693" Score="1" Body="&lt;p&gt;Two things that stand out is that you are using sigmoid as activation which is used for binary classification (it just squash values between 0 and 1). The other is that your learning rate is 5x lower than the default value. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also because you are classifying images, I think the title is misleading; semantics is associated with &quot;meaning&quot; i.e. with language. &lt;/p&gt;&#xA;" OwnerUserId="51868" LastActivityDate="2018-05-18T17:56:54.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31823" PostTypeId="2" ParentId="31818" CreationDate="2018-05-18T18:04:01.050" Score="2" Body="&lt;p&gt;You can consider string similarity based on edit distance. eg: &lt;a href=&quot;https://en.wikipedia.org/wiki/Levenshtein_distance&quot; rel=&quot;nofollow noreferrer&quot;&gt;Levenshtein distance&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, you also need to compute this with respect to prefixes of the strings, and not only the full strings. (eg: to compare &quot;Facebook&quot; with &quot;Facebook Inc.&quot;). Also normalize the capitalization of the strings before comparison (make them all lowercase, etc.), and maybe remove punctuation as well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Experiment with a few of the above, see if the results make sense, and fine tune the method accordingly. Also using other metadata will help. URL and context/topic words/entities occurring around the company name will be associated with the company, and can help find the right groupings. This will be more involved though, so you may want to attempt this later on, if the simpler approaches do not suffice.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-18T18:04:01.050" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31824" PostTypeId="2" ParentId="31818" CreationDate="2018-05-18T18:09:00.790" Score="2" Body="&lt;p&gt;It seems like you want to want to first get the datasets together using as shared key which in this case is the company name. That does not sound like a modelling problem but a data wrangling problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that each datasets are in csv, then get them to pandas dataframe:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df_a = pd.read_csv('data_set_a.csv')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then assuming you have the datasets in df_a, df_b, df_c and each have a column 'name':&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df_a.name = df_a.name.str.lower()&#xA;df_a.name = [i.split()[0] for i in df_a.name]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now you will have a dataset where all three are 'facebook'. Then the three dataframes can be merged in to one: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df = pd.merge(df_a, df_b, left_on='name', right_on='name')&#xA;df = pd.merge(df, df_c, left_on='name', right_on='name')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And now you have the three different datasets merged, and ready for analysis. &lt;/p&gt;&#xA;" OwnerUserId="51868" LastActivityDate="2018-05-18T18:09:00.790" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31825" PostTypeId="2" ParentId="31798" CreationDate="2018-05-18T19:43:42.243" Score="2" Body="&lt;p&gt;You could try some feature selection techniques if you want to choose 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That said, have you tested the data for multicollinearity?  Maybe you have but I don't think that &lt;code&gt;a1+a2+a3+a4=1&lt;/code&gt; implies a high degree of correlation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To answer your question though, it really doesn't hurt to try and train a neural net with the four variables.  If one of the inputs is unnecessary, the NN training may well set the weight on one to close to zero without you having to choose one.&lt;/p&gt;&#xA;" OwnerUserId="23159" LastActivityDate="2018-05-18T19:43:42.243" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31828" PostTypeId="2" ParentId="17146" CreationDate="2018-05-18T20:51:10.890" Score="2" Body="&lt;p&gt;The short answer is no.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves.  In some cases the precision and recall was better with the ensemble, but more often, it was not.  That's not to say it's not worth investigating.  But sometimes, there is one model that does a reasonable job of classifying data, but it can get drowned out in an ensemble.  Perhaps a weighted ensemble might improve the results, but its not a clear cut approach to improving the performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In practice, I would try several models and then try an ensemble of the models.  If the ensemble is the best, however you define best, go with it.  But sometimes it is easier to just pick the best base model and then figure out how to tune that model.&lt;/p&gt;&#xA;" OwnerUserId="23159" LastActivityDate="2018-05-18T20:51:10.890" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31829" PostTypeId="2" ParentId="31820" CreationDate="2018-05-18T21:24:04.070" Score="2" Body="&lt;p&gt;PPI is only relevant for physical objects, like representations on screen or paper. It is &lt;em&gt;pixels per inch&lt;/em&gt;, and defines how big an image X by Y pixels appears on that object. So if you have a 256x256 pixel image, and you show it at 256 PPI on a screen it will be 1 inch square if you put a ruler up to it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many physical devices have natural PPI characteristics. Screens are often 72 PPI, 100 PPI, or about 300 PPI for &quot;Retina&quot; devices. Laser printers start at 300 PPI and higher.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So if you show a 256x256 image on a 100 PPI screen as a direct mapping of one image pixel to one screen pixel, you'll get an image 2.56&quot; across.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you zoom into the image you can make it bigger on the screen, and it will now cover more than 256 pixels of the screen. The underlying image data will still be an array of 256x256 values&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Scanners are another area where PPI is important. If you scan a 1&quot; image at 300 PPI, you get a 300x300 pixel image. Set your scanner to 1200 DPI and you get a 1200x1200 pixel image, that is much finer detail. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, PPI is irrelevant to CV or ML algorithms because they only care about the number of pixels. &lt;/p&gt;&#xA;" OwnerUserId="471" LastActivityDate="2018-05-18T21:24:04.070" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31830" PostTypeId="1" CreationDate="2018-05-18T21:37:12.507" Score="6" ViewCount="2263" Body="&lt;p&gt;I am using the imblearn package to resample some data before applying other transformation/prediction techniques.  Specfically, I am using SMOTE in a slightly unconventional way, as a data augmentation technique to upsample all classes rather than just the minority.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to do a pipeline of SMOTE -&gt; PCA -&gt; SVM on data separated into training and test (most likely K-fold).  However, consider the following snippet of code from the imblearn pipeline doccumentation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipeline = Pipeline([('smt', smt), ('pca', pca), ('knn', knn)])&#xA;X_train, X_test, y_train, y_test = tts(X, y, random_state=42)&#xA;pipeline.fit(X_train, y_train) &#xA;y_hat = pipeline.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the last line, it seems the test data will be resampled, which doesn't make sense -- only the training data should be resampled using this technique, or you will be testing on artificially generated data.  The documentation for pipeline.predict() implies samplers are applied during prediction as well.  Is there a way to avoid this, besides simply removing SMOTE from the pipeline?  &lt;/p&gt;&#xA;" OwnerUserId="52356" LastActivityDate="2018-07-23T12:14:59.973" Title="How to avoid resampling part of pipeline on test data (imblearn package, SMOTE)" Tags="|python|scikit-learn|class-imbalance|smote|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31832" PostTypeId="1" AcceptedAnswerId="51507" CreationDate="2018-05-18T23:03:52.933" Score="2" ViewCount="699" Body="&lt;p&gt;I have a seemingly esoteric machine learning problem and I'm not sure where to start looking at literature.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to train a binary classifier that accepts/rejects each object in a collection of N objects. As input, it is given feature vectors describing each object, as well as feature vectors describing pairwise relationships between each object. The list of pairwise relationships is not complete, but not very sparse: approx 1/3 of the possible pairs are described.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The ground truth is the accept/reject label of each object, and I have a set of collections to train on. From domain knowledge I know that the global properties of the entire collection are important to determining the classification of the object, which is why I would like to use the pairwise relationships in classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Currently to classify the objects we are using a greedy algorithm that uses a rule-based score based on the relationship measure. (We start with an object that we trust to be very good, then we greedily add the other objects until no more objects have a positive total relationship score in relation to the already-accepted subset) So we know that the relationship measure is important and will be present in the test data. I am attempting to improve on this by using a machine learning approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My first attempt was to treat each pairwise relationship I have as a data point, and concatenate the two self-descriptor vectors to the relationship vector. Then, I labeled this as &quot;1&quot; if they were an accept-accept pair, or &quot;-1&quot; if it was an accept-reject pair. Then, during testing, I could put these predictions in one affinity matrix for each collection (if the pairwise relationship is unknown, it that matrix entry is left as 0) and use spectral clustering (&quot;accept&quot; and &quot;reject&quot; objects would have opposite signs in the first eigenvector). However, this performed worse than using only the self-descriptors for classification. I suspect this is because there is a considerable deficiency in this training protocol -- unlike the rule-based score we currently use, there is no indicator of how strong the agreement or disagreement between the two objects is.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, I would like to create a classifier that looks at the entire collection of self-descriptors and pairwise relationships, and uses the accept/reject labels directly as ground truth somehow.&lt;/p&gt;&#xA;" OwnerUserId="52360" LastEditorUserId="52360" LastEditDate="2018-05-22T20:42:39.790" LastActivityDate="2019-05-06T17:35:33.410" Title="Binary classification based on pairwise relationships" Tags="|machine-learning|classification|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31834" PostTypeId="1" CreationDate="2018-05-18T23:54:34.657" Score="3" ViewCount="170" Body="&lt;p&gt;I've been looking into the Cox Regression method for Survival Analysis in Churn Prediction. Cox regression will allow us to determine the probability that a subscriber will unsubscribe after a time $t$, defined by the &lt;em&gt;hazard rate&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;h(t \lvert X_i ) = h_0(t)exp\big( \boldsymbol{\beta} ^T\boldsymbol{X}_{i} \big)&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;$h_0(t)$: &lt;em&gt;Baseline Hazard&lt;/em&gt; is a prior Probability that any customer churns at time t when all influencing factors are 0.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\boldsymbol{\beta} \in \mathbb{R}^D$: Exponent of each Coefficient gives us a &lt;em&gt;Hazard ratio&lt;/em&gt;. These should be constant w.r.t time (proportionality assumption).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;$\boldsymbol{X}\in \mathbb{R}^{N\times D}$: Set of $N$ sample customers&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Problem&lt;/strong&gt;: Proportionality Hazard Assumption: Cox regression makes an assumption that the Hazard Ratios should remain constant through time $t$. For example, for a covariate $X_1$ = &quot;gender&quot;, say $\beta_1=1.8$. In english, it means male subscribers tend to leave the service $80\%$ more than females after a time $t$. However, this $80\%$ should hold for any time $t$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is usually an unreasonable constrain for many variables. But there are other methods that can incorporate variables that don’t follow the proportional hazards assumption. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;stratified cox regression&lt;/li&gt;&#xA;&lt;li&gt;pseudo-observations&lt;/li&gt;&#xA;&lt;li&gt;cox regression with time-dependent covariates&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I was just reading up on stratified cox regression. The only apparent downside here is:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The variables that are &lt;em&gt;stratified&lt;/em&gt; need to be converted into categorical variables&lt;/li&gt;&#xA;&lt;li&gt;The stratified categorical variables should not have too many degrees of freedom. This will lead to a LARGE number of models whose parameters need to be estimated.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question&lt;/strong&gt;: Is pseudo-observations similar? Does it have less/more rigid constraints? Even so, how is it's performance considering I have copious amounts of data?&lt;/p&gt;&#xA;" OwnerUserId="30435" LastEditorUserId="30435" LastEditDate="2018-05-19T17:08:43.043" LastActivityDate="2023-12-24T12:08:05.927" Title="Survival Analysis: Pseudo Observation Vs Stratified Cox Regression. Which one is better?" Tags="|machine-learning|statistics|survival-analysis|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31835" PostTypeId="1" AcceptedAnswerId="31847" CreationDate="2018-05-18T23:55:19.953" Score="1" ViewCount="64" Body="&lt;p&gt;I read the AlphaGo Zero paper and I didn't found nothing about it in there. But I would like to know if AlphaGo Zero can adapt to the way the oponent plays (oponent profile) or something like this. Thanks!!&lt;/p&gt;&#xA;" OwnerUserId="52206" LastEditorUserId="52206" LastEditDate="2018-05-19T00:18:46.447" LastActivityDate="2018-05-19T11:02:35.437" Title="Can AlphaGo Zero adapt to oponents skills/profile?" Tags="|machine-learning|neural-network|game|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31836" PostTypeId="1" CreationDate="2018-05-19T00:51:57.350" Score="1" ViewCount="284" Body="&lt;p&gt;How to get sentiment analysis data widget in python script in orange and how this data used for other purpose and apply on this sentiments data  different methods&lt;/p&gt;&#xA;" OwnerUserId="41909" LastActivityDate="2018-05-19T00:51:57.350" Title="get data in python script in orange" Tags="|data-mining|orange|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-28T09:42:31.743" ContentLicense="CC BY-SA 4.0" />
  <row Id="31837" PostTypeId="1" CreationDate="2018-05-19T02:15:09.893" Score="1" ViewCount="116" Body="&lt;p&gt;I'm interested in doing research similar to the work done in this blog, &lt;a href=&quot;http://karpathy.github.io/2015/05/21/rnn-effectiveness/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Unreasonable Effectiveness of Recurrent Neural Networks&lt;/a&gt; by Andrej Karpathy. Specifically, where the author explores the rules that individual neurons have learned by extracting the activations or excitement of random neurons at the prediction step and visualizing them as a heatmap on top of the predicted text. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm having a hard time figuring out how to do this with existing platforms such as TensorFlow. My first strategy was to try a form of transfer learning and cut off a copy of the network at an activation layer and have that layer's activations output side by side with a text prediction. But the weakness of this approach is that I can't actually see the activations in the hidden state (or more than one layer at a time), just the outputs of an LSTM layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone recommend a strategy or starting point where I can reveal the activations of neurons in the hidden states of RNNs (LSTM or GRU), or if there's a way to do this in TensorFlow that I've overlooked? If I build the network from scratch with TensorFlow, is there a way to efficiently set it up to output this data on prediction without significantly adding to the computation expense during training? I couldn't find the answer in the &lt;a href=&quot;https://github.com/karpathy/char-rnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;source code&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone give me a nudge in the right direction? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance! &lt;/p&gt;&#xA;" OwnerUserId="44299" LastActivityDate="2018-05-19T02:15:09.893" Title="Catching hidden state activations of an RNN on prediction." Tags="|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31838" PostTypeId="1" CreationDate="2018-05-19T03:39:49.167" Score="1" ViewCount="2945" Body="&lt;p&gt;I am working on a dataset which has around 2500 observations and 46 variables. I was able to find out why exactly is the error occuring, but just not able to resolve it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the code producing the error the error:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test$rating&amp;lt;-as.factor(test$rating)&#xA;fitTree&amp;lt;-rpart(rating~.,data=train)&#xA;predict(fitTree,data=mode,type=&quot;class&quot;)&#xA;&amp;gt; predict(fitTree,data=mode,type=&quot;class&quot;)&#xA;Error in predict.rpart(fitTree, data = mode, type = &quot;class&quot;) : &#xA;  Invalid prediction for &quot;rpart&quot; object&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The mode dataset in data= mode inside the predict function has 45 variables and was created by using this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mode&amp;lt;-data.frame(functionary=0,re.balanced..paid.back..a.recently.overdrawn.current.acount=1,FI3O.credit.score=1,...till 45 variables)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The mode dataset doesn't have the credit column and so we are predicting it by training with the train dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Works if I remove the part type=&quot;class&quot; from the code but the problem is it doesn't give me right output. The output it gives me looks something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; predict(fitTree,data=hypo_mode)&#xA;&amp;gt; 1         2         3         4         5         6         7        &#xA;&amp;gt; 8         9        10        11 &#xA;&amp;gt; 0.0000000 0.0000000 0.4285714 0.4285714 1.6592179 1.6592179 1.9379562 1.9379562 1.5789474 2.0739437 1.9379562 &#xA;&amp;gt;        12        13        14        15        16        17        18        19        20        21        22 &#xA;&amp;gt; 2.0739437 1.5789474 1.6592179 2.0739437 0.0000000 1.6592179 0.0000000 1.5789474 1.9379562 1.9379562 2.0739437 &#xA;&amp;gt;        23        24        25        26        27        28        29        30        31        32        33 &#xA;&amp;gt; 2.7741935 0.0000000 1.6592179 1.9379562 0.0000000 1.9379562 1.9379562 1.9379562 1.9379562 0.0000000 1.6592179 &#xA;&amp;gt;        34        35        36        37        38        39        40        41        42        43        44 &#xA;&amp;gt; 0.1929825 1.5789474 0.0000000 0.1929825 2.0739437 1.1666667 2.0739437 2.0739437 1.5789474 1.5789474 0.0000000 &#xA;&amp;gt;        45        46        47        48        49        50        51        52        53        54        55 &#xA;&amp;gt; 1.6592179 1.9379562 0.0000000 1.5789474 1.9379562 1.9379562 1.9379562 1.6592179 1.6592179 1.5789474 1.5789474 &#xA;&amp;gt;        56        57        58        59        60        61        62        63        64        65        66 &#xA;&amp;gt; 2.0739437 1.6592179 2.0739437 1.5789474 0.4285714 0.0000000 1.5789474 1.6592179 1.5789474 1.6592179 1.9379562 &#xA;&amp;gt;        67        68        69        70        71        72        73        74        75        76        77 &#xA;&amp;gt; 2.0739437 0.0000000 1.9379562 2.0739437 1.5789474 2.0739437 0.0000000 1.6592179 2.0739437 2.0739437 1.9379562 &#xA;&amp;gt;        78        79        80        81        82        83        84        85        86        87        88 &#xA;&amp;gt; 1.6592179 0.1929825 1.6592179 1.5789474 1.9379562 1.9379562 0.0000000 2.0739437 2.7741935 2.7741935 2.0739437 &#xA;&amp;gt;        89        90        91        92        93        94        95        96        97        98        99 &#xA;&amp;gt; 1.5789474 0.1929825 1.5789474 0.0000000 1.9379562 0.1929825 0.0000000 1.6592179 1.9379562 1.9379562 1.5789474 &#xA;&amp;gt;       100       101       102       103       104       105       106       107       108       109       110 &#xA;&amp;gt; 2.0739437 1.5789474 1.5789474 1.9379562 2.7741935 0.0000000 1.5789474 2.0739437 2.0739437 0.0000000 2.0739437 &#xA;&amp;gt;       111       112       113       114       115       116       117       118       119       120       121 &#xA;&amp;gt; 1.6592179 1.9379562 2.0739437 1.6592179 1.5789474 1.6592179 2.0739437 2.0739437 1.5789474 1.9379562 2.0739437 &#xA;&amp;gt;       122       123       124       125       126       127       128       129       130       131       132 &#xA;&amp;gt; 1.6592179 2.7741935 1.5789474 1.9379562 1.6592179 0.1929825 1.9379562 2.0739437 1.5789474 1.9379562 2.0739437 &#xA;&amp;gt;       133       134       135       136       137       138       139       140       141       142       143 &#xA;&amp;gt; 1.5789474 1.9379562 2.7741935 0.0000000 1.9379562 2.0739437 2.0739437 2.2777778 1.9379562 1.6592179 2.0739437 &#xA;&amp;gt;       144       145       146       147       148       149       150       151       152       153       154 &#xA;&amp;gt; 2.0739437 1.9379562 1.6592179 1.9379562 1.6592179 2.7741935 2.0739437 0.0000000 2.0739437 0.0000000 0.0000000 &#xA;&amp;gt;       155       156       157       158       159       160       161       162       163       164       165 &#xA;&amp;gt; 1.5789474 2.7741935 1.9379562 1.6592179 1.1666667 1.5789474 1.5789474 0.0000000 2.0739437 1.6592179 1.6592179 &#xA;&amp;gt;       166       167       168       169       170       171       172       173       174       175       176 &#xA;&amp;gt; 0.0000000 1.5789474 2.0739437 1.6592179 0.0000000 1.5789474 1.6592179 0.0000000 2.2777778 1.6592179 1.9379562 &#xA;&amp;gt;       177       178       179       180       181       182       183       184       185       186       187 &#xA;&amp;gt; 1.9379562 1.9379562 2.2777778 0.1929825 1.1666667 1.9379562 2.0739437 0.0000000 0.0000000 1.9379562 1.6592179 &#xA;&amp;gt;       188       189       190       191       192       193       194       195       196       197       198 &#xA;&amp;gt; 2.0739437 1.6592179 2.0739437 1.9379562 1.9379562 1.9379562 2.0739437 1.9379562 1.1666667 0.0000000 2.0739437 &#xA;&amp;gt;       199       200       201       202       203       204       205       206       207       208       209 &#xA;&amp;gt; 1.6592179 1.9379562 1.9379562 2.0739437 2.0739437 1.5789474 2.0739437 1.1666667 1.9379562 1.9379562 0.0000000 &#xA;&amp;gt;       210       211       212       213       214       215       216       217       218       219       220 &#xA;&amp;gt; 1.9379562 0.0000000 2.0739437 1.9379562 1.9379562 1.5789474 1.5789474 1.9379562 0.0000000 1.9379562 0.0000000 &#xA;&amp;gt;       221       222       223       224       225       226       227       228       229       230       231 &#xA;&amp;gt; 0.0000000 1.5789474 2.0739437 1.9379562 2.0739437 1.6592179 0.1929825 1.5789474 2.0739437 1.5789474 1.9379562 &#xA;&amp;gt;       232       233       234       235       236       237       238       239       240       241       242 &#xA;&amp;gt; 2.0739437 1.6592179 1.5789474 1.9379562 1.9379562 1.6592179 2.0739437 1.9379562 1.9379562 1.9379562 1.5789474 &#xA;&amp;gt;       243       244       245       246       247       248       249       250       251       252       253 &#xA;&amp;gt; 1.9379562 2.0739437 2.0739437 2.0739437 1.6592179 2.0739437 0.0000000 1.5789474 0.0000000 1.6592179 1.9379562 &#xA;&amp;gt;       254       255       256       257       258       259       260       261       262       263       264 &#xA;&amp;gt; 2.7741935 0.0000000 2.0739437 1.5789474 1.9379562 1.9379562 0.0000000 2.0739437 1.6592179 1.9379562 0.0000000 &#xA;&amp;gt;       265       266       267       268       269       270       271       272       273       274       275 &#xA;&amp;gt; 2.2777778 1.5789474 1.9379562 2.0739437 2.0739437 1.5789474 1.1666667 1.6592179 1.5789474 1.5789474 1.5789474 &#xA;&amp;gt;       276       277       278       279       280       281       282       283       284       285       286 &#xA;&amp;gt; 2.0739437 1.5789474 2.0739437 2.0739437 1.5789474 1.5789474 1.6592179 1.9379562 1.9379562 1.9379562 0.0000000 &#xA;&amp;gt;       287       288       289       290       291       292       293       294       295       296       297 &#xA;&amp;gt; 1.6592179 1.9379562 2.0739437 2.0739437 2.0739437 1.5789474 2.0739437 0.1929825 1.5789474 1.1666667 1.5789474 &#xA;&amp;gt;       298       299       300       301       302       303       304       305       306       307       308 &#xA;&amp;gt; 1.5789474 1.6592179 1.6592179 0.4285714 1.9379562 1.5789474 0.0000000 1.9379562 1.5789474 1.9379562 0.0000000 &#xA;&amp;gt;       309       310       311       312       313       314       315       316       317       318       319 &#xA;&amp;gt; 1.9379562 1.1666667 0.0000000 1.6592179 1.5789474 1.5789474 1.5789474 1.6592179 2.0739437 2.0739437 1.9379562 &#xA;&amp;gt;       320       321       322       323       324       325       326       327       328       329       330 &#xA;&amp;gt; 2.0739437 0.0000000 2.0739437 0.0000000 1.6592179 1.6592179 1.9379562 2.0739437 1.9379562 1.5789474 1.5789474 &#xA;&amp;gt;       331       332       333       334       335       336       337       338       339       340       341 &#xA;&amp;gt; 1.5789474 0.1929825 0.0000000 1.1666667 2.0739437 2.0739437 0.0000000 1.6592179 1.9379562 1.5789474 0.1929825 &#xA;&amp;gt;       342       343       344       345       346       347       348       349       350       351       352 &#xA;&amp;gt; 1.6592179 1.6592179 2.0739437 0.0000000 2.7741935 1.9379562 0.0000000 1.6592179 1.9379562 1.9379562 0.0000000 &#xA;&amp;gt;       353       354       355       356       357       358       359       360       361       362       363 &#xA;&amp;gt; 1.6592179 1.9379562 0.0000000 0.1929825 0.0000000 0.0000000 2.0739437 0.1929825 2.0739437 1.6592179 1.5789474 &#xA;&amp;gt;       364       365       366       367       368       369       370       371       372       373       374 &#xA;&amp;gt; 1.6592179 1.1666667 0.1929825 1.9379562 2.0739437 2.0739437 1.9379562 2.0739437 1.9379562 1.6592179 2.7741935 &#xA;&amp;gt;       375       376       377       378       379       380       381       382       383       384       385 &#xA;&amp;gt; 2.0739437 1.9379562 2.0739437 0.0000000 1.5789474 2.2777778 2.7741935 0.1929825 1.6592179 2.7741935 0.0000000 &#xA;&amp;gt;       386       387       388       389       390       391       392       393       394       395       396 &#xA;&amp;gt; 2.0739437 1.6592179 1.5789474 1.9379562 2.0739437 1.9379562 2.0739437 2.0739437 1.1666667 1.6592179 1.6592179 &#xA;&amp;gt;       397       398       399       400       401       402       403       404       405       406       407 &#xA;&amp;gt; 1.9379562 1.5789474 1.5789474 1.5789474 1.9379562 1.6592179 2.0739437 2.0739437 0.0000000 2.0739437 1.5789474 &#xA;&amp;gt;       408       409       410       411       412       413       414       415       416       417       418 &#xA;&amp;gt; 1.6592179 2.0739437 1.1666667 1.6592179 1.5789474 2.0739437 1.1666667 1.9379562 1.6592179 0.0000000 1.6592179 &#xA;&amp;gt;       419       420       421       422       423       424       425       426       427       428       429 &#xA;&amp;gt; 1.6592179 1.5789474 1.5789474 1.6592179 2.0739437 1.5789474 2.7741935 1.9379562 0.4285714 1.9379562 1.5789474 &#xA;&amp;gt;       430       431       432       433       434       435       436       437       438       439       440 &#xA;&amp;gt; 1.1666667 1.5789474 1.9379562 0.0000000 2.0739437 1.9379562 2.0739437 1.6592179 1.6592179 1.5789474 1.5789474 &#xA;&amp;gt;       441       442       443       444       445       446       447       448       449       450       451 &#xA;&amp;gt; 1.5789474 1.9379562 1.6592179 1.5789474 1.9379562 1.5789474 1.5789474 2.0739437 0.4285714 0.4285714 2.0739437 &#xA;&amp;gt;       452       453       454       455       456       457       458       459       460       461       462 &#xA;&amp;gt; 1.6592179 1.6592179 2.0739437 0.0000000 1.6592179 2.0739437 1.9379562 1.9379562 2.0739437 0.1929825 1.5789474 &#xA;&amp;gt;       463       464       465       466       467       468       469       470       471       472       473 &#xA;&amp;gt; 1.5789474 1.9379562 0.0000000 2.0739437 2.0739437 1.9379562 1.5789474 0.0000000 0.0000000 2.0739437 2.0739437 &#xA;&amp;gt;       474       475       476       477       478       479       480       481       482       483       484 &#xA;&amp;gt; 1.6592179 2.0739437 2.0739437 1.9379562 1.6592179 2.0739437 1.5789474 1.9379562 1.6592179 2.2777778 1.5789474 &#xA;&amp;gt;       485       486       487       488       489       490       491       492       493       494       495 &#xA;&amp;gt; 0.1929825 1.5789474 1.5789474 1.6592179 2.0739437 1.9379562 1.5789474 2.0739437 1.6592179 1.9379562 1.1666667 &#xA;&amp;gt;       496       497       498       499       500       501       502       503       504       505       506 &#xA;&amp;gt; 1.5789474 2.0739437 2.0739437 2.2777778 1.9379562 1.1666667 1.5789474 1.9379562 0.0000000 1.6592179 2.0739437 &#xA;&amp;gt;       507       508       509       510       511       512       513       514       515       516       517 &#xA;&amp;gt; 1.9379562 1.9379562 2.0739437 2.0739437 1.9379562 1.9379562 0.0000000 1.5789474 2.0739437 2.0739437 2.0739437 &#xA;&amp;gt;       518       519       520       521       522       523       524       525       526       527       528 &#xA;&amp;gt; 2.0739437 1.5789474 2.0739437 2.0739437 2.0739437 1.5789474 1.5789474 1.9379562 0.1929825 2.0739437 2.0739437 &#xA;&amp;gt;       529       530       531       532       533       534       535       536       537       538       539 &#xA;&amp;gt; 1.5789474 1.9379562 1.5789474 2.0739437 1.1666667 1.6592179 1.5789474 1.9379562 2.0739437 1.9379562 0.1929825 &#xA;&amp;gt;       540       541       542       543       544       545       546       547       548       549       550 &#xA;&amp;gt; 0.0000000 2.0739437 0.4285714 0.0000000 2.7741935 1.5789474 1.6592179 1.5789474 0.1929825 0.0000000 2.0739437 &#xA;&amp;gt;       551       552       553       554       555       556       557       558       559       560       561 &#xA;&amp;gt; 1.9379562 1.6592179 1.5789474 2.0739437 1.5789474 1.5789474 1.9379562 1.6592179 1.9379562 1.9379562 1.5789474 &#xA;&amp;gt;       562       563       564       565       566       567       568       569       570       571       572 &#xA;&amp;gt; 0.0000000 1.9379562 1.5789474 1.5789474 0.0000000 1.1666667 1.9379562 1.9379562 1.6592179 2.0739437 1.5789474 &#xA;&amp;gt;       573       574       575       576       577       578       579       580       581       582       583 &#xA;&amp;gt; 2.0739437 1.5789474 1.6592179 0.1929825 1.5789474 1.9379562 1.5789474 2.0739437 1.6592179 1.6592179 1.6592179 &#xA;&amp;gt;       584       585       586       587       588       589       590       591       592       593       594 &#xA;&amp;gt; 1.5789474 2.0739437 2.0739437 1.5789474 1.6592179 2.0739437 1.5789474 2.7741935 0.0000000 1.9379562 1.1666667 &#xA;&amp;gt;       595       596       597       598       599       600       601       602       603       604       605 &#xA;&amp;gt; 1.5789474 2.0739437 2.7741935 1.9379562 1.5789474 1.9379562 2.7741935 0.0000000 2.0739437 1.9379562 1.5789474 &#xA;&amp;gt;       606       607       608       609       610       611       612       613       614       615       616 &#xA;&amp;gt; 1.9379562 0.1929825 1.9379562 1.5789474 1.5789474 0.1929825 1.9379562 0.0000000 1.6592179 1.5789474 0.1929825 &#xA;&amp;gt;       617       618       619       620       621       622       623       624       625       626       627 &#xA;&amp;gt; 2.0739437 0.0000000 1.6592179 1.9379562 0.0000000 1.1666667 1.5789474 2.0739437 2.0739437 2.7741935 0.0000000 &#xA;&amp;gt;       628       629       630       631       632       633       634       635       636       637       638 &#xA;&amp;gt; 2.2777778 0.0000000 2.0739437 1.5789474 1.6592179 1.5789474 0.1929825 0.1929825 2.0739437 1.9379562 0.0000000 &#xA;&amp;gt;       639       640       641       642       643       644       645       646       647       648       649 &#xA;&amp;gt; 0.0000000 1.6592179 1.5789474 2.0739437 1.6592179 0.0000000 1.6592179 2.0739437 1.6592179 0.1929825 1.5789474 &#xA;&amp;gt;       650       651       652       653       654       655       656       657       658       659       660 &#xA;&amp;gt; 0.1929825 2.7741935 2.0739437 1.5789474 1.9379562 1.9379562 0.0000000 0.0000000 2.0739437 1.1666667 1.6592179 &#xA;&amp;gt;       661       662       663       664       665       666       667       668       669       670       671 &#xA;&amp;gt; 1.9379562 0.1929825 1.6592179 1.6592179 2.0739437 1.5789474 2.0739437 1.9379562 1.6592179 0.0000000 0.1929825 &#xA;&amp;gt;       672       673       674       675       676       677       678       679       680       681       682 &#xA;&amp;gt; 0.1929825 2.0739437 1.9379562 2.0739437 2.7741935 1.6592179 1.5789474 1.9379562 2.2777778 1.9379562 2.0739437 &#xA;&amp;gt;       683       684       685       686       687       688       689       690       691       692       693 &#xA;&amp;gt; 2.0739437 1.5789474 0.0000000 1.9379562 1.6592179 1.5789474 2.0739437 1.9379562 1.5789474 1.9379562 1.9379562 &#xA;&amp;gt;       694       695       696       697       698       699       700       701       702       703       704 &#xA;&amp;gt; 2.0739437 2.0739437 1.9379562 1.6592179 1.6592179 2.0739437 1.6592179 1.6592179 1.6592179 2.0739437 1.5789474 &#xA;&amp;gt;       705       706       707       708       709       710       711       712       713       714       715 &#xA;&amp;gt; 1.5789474 2.0739437 1.6592179 1.9379562 1.6592179 2.0739437 2.0739437 2.0739437 1.5789474 1.9379562 2.0739437 &#xA;&amp;gt;       716       717       718       719       720       721       722       723       724       725       726 &#xA;&amp;gt; 0.1929825 1.6592179 1.9379562 2.0739437 2.0739437 2.0739437 1.6592179 1.9379562 1.6592179 2.0739437 2.0739437 &#xA;&amp;gt;       727       728       729       730       731       732       733       734       735       736       737 &#xA;&amp;gt; 1.9379562 2.0739437 2.0739437 1.9379562 0.0000000 0.0000000 1.9379562 1.9379562 0.4285714 1.6592179 2.0739437 &#xA;&amp;gt;       738       739       740       741       742       743       744       745       746       747       748 &#xA;&amp;gt; 1.9379562 1.1666667 0.0000000 2.0739437 1.6592179 1.5789474 1.5789474 0.0000000 2.0739437 1.5789474 1.6592179 &#xA;&amp;gt;       749       750       751       752       753       754       755       756       757       758       759 &#xA;&amp;gt; 1.9379562 1.9379562 2.0739437 2.0739437 1.5789474 2.0739437 1.9379562 1.1666667 1.9379562 2.0739437 1.5789474 &#xA;&amp;gt;       760       761       762       763       764       765       766       767       768       769       770 &#xA;&amp;gt; 1.9379562 1.5789474 1.5789474 2.0739437 1.9379562 1.6592179 1.5789474 0.1929825 1.9379562 1.9379562 1.5789474 &#xA;&amp;gt;       771       772       773       774       775       776       777       778       779       780       781 &#xA;&amp;gt; 1.9379562 1.6592179 1.5789474 2.0739437 0.0000000 1.5789474 1.5789474 2.0739437 1.6592179 2.0739437 2.0739437 &#xA;&amp;gt;       782       783       784       785       786       787       788       789       790       791       792 &#xA;&amp;gt; 0.0000000 1.9379562 0.0000000 0.1929825 1.1666667 1.6592179 1.5789474 0.1929825 0.0000000 1.9379562 1.9379562 &#xA;&amp;gt;       793       794       795       796       797       798       799       800       801       802       803 &#xA;&amp;gt; 1.1666667 1.6592179 2.0739437 1.1666667 1.9379562 0.0000000 1.6592179 1.6592179 2.0739437 2.0739437 1.6592179 &#xA;&amp;gt;       804       805       806       807       808       809       810       811       812       813       814 &#xA;&amp;gt; 0.0000000 2.0739437 1.5789474 1.9379562 1.9379562 0.4285714 1.9379562 2.0739437 2.0739437 1.6592179 1.9379562 &#xA;&amp;gt;       815       816       817       818       819       820       821       822       823       824       825 &#xA;&amp;gt; 2.0739437 1.9379562 1.6592179 1.6592179 0.1929825 1.5789474 1.5789474 1.9379562 1.9379562 1.9379562 1.9379562 &#xA;&amp;gt;       826       827       828       829       830       831       832       833       834       835       836 &#xA;&amp;gt; 2.0739437 1.6592179 1.6592179 1.6592179 1.9379562 1.5789474 2.0739437 0.1929825 0.0000000 2.0739437 2.0739437 &#xA;&amp;gt;       837       838       839       840       841       842       843       844       845       846       847 &#xA;&amp;gt; 0.0000000 2.0739437 1.5789474 0.0000000 1.9379562 2.2777778 1.5789474 0.1929825 1.5789474 1.9379562 2.7741935 &#xA;&amp;gt;       848       849       850       851       852       853       854       855       856       857       858 &#xA;&amp;gt; 0.0000000 1.6592179 2.0739437 1.9379562 1.6592179 2.0739437 1.9379562 1.1666667 1.5789474 1.5789474 2.0739437 &#xA;&amp;gt;       859       860       861       862       863       864       865       866       867       868       869 &#xA;&amp;gt; 1.6592179 1.9379562 1.9379562 2.0739437 1.9379562 2.0739437 1.5789474 0.0000000 1.5789474 0.0000000 1.9379562 &#xA;&amp;gt;       870       871       872       873       874       875       876       877       878       879       880 &#xA;&amp;gt; 1.6592179 1.9379562 1.9379562 0.0000000 2.0739437 1.5789474 0.0000000 1.5789474 2.0739437 1.5789474 1.6592179 &#xA;&amp;gt;       881       882       883       884       885       886       887       888       889       890       891 &#xA;&amp;gt; 1.6592179 2.0739437 2.0739437 1.9379562 1.9379562 0.0000000 2.0739437 2.2777778 0.1929825 2.0739437 1.9379562 &#xA;&amp;gt;       892       893       894       895       896       897       898       899       900       901       902 &#xA;&amp;gt; 0.1929825 0.0000000 0.1929825 2.0739437 0.0000000 1.6592179 2.0739437 2.2777778 1.9379562 1.5789474 2.0739437 &#xA;&amp;gt;       903       904       905       906       907       908       909       910       911       912       913 &#xA;&amp;gt; 1.9379562 1.9379562 2.0739437 1.6592179 2.0739437 1.9379562 2.0739437 1.5789474 1.9379562 1.6592179 1.9379562 &#xA;&amp;gt;       914       915       916       917       918       919       920       921       922       923       924 &#xA;&amp;gt; 0.1929825 2.0739437 2.0739437 2.0739437 0.1929825 2.0739437 2.0739437 1.9379562 2.0739437 1.9379562 1.6592179 &#xA;&amp;gt;       925       926       927       928       929       930       931       932       933       934       935 &#xA;&amp;gt; 0.1929825 0.0000000 2.7741935 1.5789474 1.9379562 1.6592179 1.5789474 0.4285714 0.4285714 0.0000000 2.0739437 &#xA;&amp;gt;       936       937       938       939       940       941       942       943       944       945       946 &#xA;&amp;gt; 1.6592179 2.0739437 1.6592179 1.9379562 0.1929825 1.9379562 0.1929825 1.5789474 2.0739437 1.9379562 1.9379562 &#xA;&amp;gt;       947       948       949       950       951       952       953       954       955       956       957 &#xA;&amp;gt; 1.5789474 0.0000000 1.5789474 1.5789474 1.5789474 1.5789474 2.0739437 1.5789474 1.5789474 0.0000000 1.5789474 &#xA;&amp;gt;       958       959       960       961       962       963       964       965       966       967       968 &#xA;&amp;gt; 1.6592179 2.0739437 1.5789474 1.6592179 1.5789474 1.9379562 2.0739437 0.1929825 1.5789474 1.6592179 2.0739437 &#xA;&amp;gt;       969       970       971       972       973       974       975       976       977       978       979 &#xA;&amp;gt; 1.5789474 2.0739437 1.6592179 1.9379562 2.0739437 2.0739437 1.5789474 2.7741935 1.9379562 1.9379562 1.5789474 &#xA;&amp;gt;       980       981       982       983       984       985       986       987       988       989       990 &#xA;&amp;gt; 0.0000000 2.0739437 1.9379562 1.9379562 1.5789474 0.1929825 0.0000000 1.5789474 2.0739437 1.5789474 2.0739437 &#xA;&amp;gt;       991       992       993       994       995       996       997       998       999      1000 &#xA;&amp;gt; 1.5789474 1.9379562 2.0739437 1.6592179 1.9379562 1.9379562 0.0000000 1.6592179 1.9379562 1.6592179&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The correct output should be either if the rating is 0,1,2 or 3 and should look something like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt; predict(fitTree,newdata2,type=&quot;class&quot;)&#xA;        1         2         3         4 &#xA;virginica virginica virginica virginica &#xA;Levels: setosa versicolor virginica&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In place of virginica it should be either 1, 2 or 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I resolve this error? Thanks for any suggestions provided :)&lt;/p&gt;&#xA;" OwnerUserId="51605" LastActivityDate="2018-05-19T03:39:49.167" Title="How to resolve Error in predict.rpart(fitTree, data = hypo_mode, type = “class”) : Invalid prediction for “rpart” object?" Tags="|machine-learning|classification|r|data-mining|decision-trees|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31839" PostTypeId="2" ParentId="31815" CreationDate="2018-05-19T06:24:08.763" Score="2" Body="&lt;p&gt;You should find an appropriate threshold of the LOF score, above which the point will be considered as an outlier. Typically, a point with score below 1 is considered as normal (non-outlier), but the threshold above 1 at which to consider a point as outlier depends on the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have a dataset with outlier points labeled, then you can use that to fine tune the value of the threshold, such that you get an acceptable tradeoff between false-positive rate (out of points labeled as outliers, fraction of normal points), and recall (out of all actual outliers, fraction which are labeled as outliers).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At run time, if new points are getting added to your dataset, then the local densities and LOF scores across points could change, and these should be recomputed. However, the threshold of LOF score that you determined as optimal should continue to apply, and requires revisiting only over a larger time period.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-19T06:24:08.763" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31841" PostTypeId="2" ParentId="31815" CreationDate="2018-05-19T07:08:24.693" Score="1" Body="&lt;p&gt;You can use LOF online.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For all your training data, compute and store the &lt;code&gt;lrd&lt;/code&gt; values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you get a new sample, find it's neighbors. Based on the distances to the neighbors and their kids, you can estimate the LOF of the new data point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you now do the extra mile to also update the lrd values of the existing points, then you would get an updatable LOF. But that likely isn't worth the effort.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Pokrajac, D., Lazarevic, A., &amp;amp; Latecki, L. J. (2007, March). Incremental local outlier detection for data streams. In Computational Intelligence and Data Mining, 2007. CIDM 2007. IEEE Symposium on (pp. 504-515). IEEE.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="924" LastEditorUserId="924" LastEditDate="2018-06-18T06:22:17.470" LastActivityDate="2018-06-18T06:22:17.470" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31842" PostTypeId="1" CreationDate="2018-05-19T07:22:14.180" Score="1" ViewCount="458" Body="&lt;p&gt;I'm trying to add custom lines to seaborn swarmplot (based on &lt;a href=&quot;https://stackoverflow.com/a/36152188/993324&quot;&gt;this&lt;/a&gt; answer) and it seems that x axis is wrongly scaled.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data preparation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn.datasets import load_iris&#xA;&#xA;iris = load_iris()&#xA;&#xA;X_full = pd.DataFrame(data = iris[&quot;data&quot;], columns = iris[&quot;feature_names&quot;])&#xA;y_full = pd.Series(data = iris[&quot;target&quot;])&#xA;&#xA;# selecting two classes and two features&#xA;y = y_full[(y_full == 0) | (y_full == 1)]&#xA;X = X_full[[&quot;sepal width (cm)&quot;, &quot;petal width (cm)&quot;]].loc[y.index]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and actual plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import seaborn as sns&#xA;&#xA;grid = sns.JointGrid(x = X[&quot;sepal width (cm)&quot;], y = X[&quot;petal width (cm)&quot;])&#xA;grid = grid.plot_joint(sns.swarmplot, hue = y)&#xA;&#xA;xmin = X[&quot;sepal width (cm)&quot;].min() &#xA;xmax = X[&quot;sepal width (cm)&quot;].max()&#xA;&#xA;ymin = X[&quot;petal width (cm)&quot;].min()&#xA;ymax = X[&quot;petal width (cm)&quot;].max()&#xA;&#xA;grid.ax_joint.plot([1, 2, 3], [1, 2, 3], 'b-', linewidth = 2)&#xA;&#xA;grid.ax_joint.plot([xmin, xmax], [ymin, ymax] )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Result:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PObKC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PObKC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thin line should go from (xmin, ymin) to (xmax,ymax). On the &lt;strong&gt;y&lt;/strong&gt; axis it looks correct, but &lt;strong&gt;x&lt;/strong&gt; axis is wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to fix plot so coordinates will be plotted correctly?&lt;/p&gt;&#xA;" OwnerUserId="82" LastActivityDate="2018-05-19T07:22:14.180" Title="Problem with x axis on swarmplot and added line with seaborn" Tags="|python|visualization|seaborn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31843" PostTypeId="1" CreationDate="2018-05-19T07:36:17.080" Score="5" ViewCount="54" Body="&lt;p&gt;I'm new to the data science world, and I hope to solve a problem using deep learning methods, I started learning how FNN and CNN work and when I saw how many models and methods are the I got a bit confused and decided to ask for your help in pointing me at the right direction,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;The problem I need to solve:&lt;/strong&gt; I have a data set which unlabeled and consist of actions taken by a multi-agent system, that will probably contain&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Time | Role of Agent | Action taken | Position of agent | outcome ....&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I need to train a model\models to detect a strategic sequences of actions (i.e rows in the data) that means that I need a model that will take into consideration not only the current row in data but also the rows that came before (means actions taken by other agents) and if possible rows that will be inputted later.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the output needs to be a classification on the data into strategic plays, such that a play is a few actions that happend one after the other,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the dataset is still unprepared and we are just in the planning and learning stage, by I hope that ow that you understand the problem that I'm facing a bit you can help me decide on a Deep Learning model(or models) that can meet the requirements and preform good for such task, so I can really dig into that model and learn plus if you know of any articles that can be of assistance it will be great as well&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for the help!&lt;/p&gt;&#xA;" OwnerUserId="52366" LastActivityDate="2018-05-19T07:36:17.080" Title="Choosing the right model to learn" Tags="|machine-learning|deep-learning|machine-learning-model|" AnswerCount="0" CommentCount="2" ClosedDate="2018-05-23T03:11:19.783" ContentLicense="CC BY-SA 4.0" />
  <row Id="31844" PostTypeId="2" ParentId="19365" CreationDate="2018-05-19T09:46:37.540" Score="1" Body="&lt;p&gt;My code...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;import numpy as np&#xA;import matplotlib.pylab as plt&#xA;&#xA;# Create dataset&#xA;x = np.arange(0, np.pi * 2, 0.1)&#xA;y = np.sin(x)&#xA;&#xA;# Some parameters&#xA;ACTIVE_FUN = 'tanh'&#xA;BATCH_SIZE = 1&#xA;VERBOSE=0&#xA;&#xA;# Create the model&#xA;model = Sequential()&#xA;model.add(Dense(5, input_shape=(1,), activation=ACTIVE_FUN))&#xA;model.add(Dense(5, activation=ACTIVE_FUN))&#xA;model.add(Dense(1, activation='linear'))&#xA;&#xA;# Compile the model&#xA;model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['mean_squared_error'])&#xA;&#xA;# Fit the model&#xA;model.fit(x, y, epochs=1000, batch_size=BATCH_SIZE, verbose=VERBOSE)&#xA;&#xA;# Evaluate the model&#xA;scores = model.evaluate(x, y, verbose=VERBOSE)&#xA;print('%s: %.2f%%' % (model.metrics_names[1], scores[1] * 100))&#xA;&#xA;# Make predictions&#xA;y_pred = model.predict(x)&#xA;&#xA;# Plot&#xA;plt.plot(x, y, color='blue', linewidth=1, markersize='1')&#xA;plt.plot(x, y_pred, color='green', linewidth=1, markersize='1')&#xA;plt.xlabel('Angle [rad]')&#xA;plt.ylabel('sin(x)')&#xA;plt.axis('tight')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52370" LastActivityDate="2018-05-19T09:46:37.540" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31845" PostTypeId="1" CreationDate="2018-05-19T10:19:09.777" Score="8" ViewCount="3600" Body="&lt;p&gt;The objective is to build a model that is capable of identifying information on receipts and invoices that can look completely different.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've had a discussion with my brother about the right approach. I have attached an example, here the original and below is the important information in boxes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DEGIX.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DEGIX.jpg&quot; alt=&quot;Original example receipt&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wYcj0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wYcj0.png&quot; alt=&quot;Important information&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The green boxes are the must-have information. The one in purple and green indicates that we need either or. The orange information would be a nice-to-have, but not necessarily required.&#xA;Some of the boxes have context and inter linkage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;From a data set point of view, we have a sample size of 1,000 receipts and all of them have the necessary information extracted. We could increase the sample size further if that was required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach that I would have chosen:&lt;br&gt;&#xA;Treat everyone of the images of the receipts like a game and let the model figure out itself how to arrive at the right conclusion. This will most likely be very computing intensive but I feel like it will be more robust when dealing with new image types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approach my brother has suggested:&lt;br&gt;&#xA;Basically using the boxes that I've provided and let me model learn from that. The model would then learn to identify the important areas on a receipt or invoice and would go from there. He compared the model to one that would identify license plates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&#xA;Just to reiterate why I think OCR is only part of the solution but not the solution itself. Here is the Adobe Acrobat OCR result: &lt;a href=&quot;https://i.stack.imgur.com/4bSLT.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4bSLT.png&quot; alt=&quot;OCR result&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perfect if you ask me. It just doesn't help me figure out what values to use and which to ignore. I don't want to do this manually. I want the model(s) to return for me:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Total amount&lt;/li&gt;&#xA;&lt;li&gt;Sales Tax &amp;amp; Amount&lt;/li&gt;&#xA;&lt;li&gt;Creditor (i.e. the company and ideally the tax identifier &lt;em&gt;CHE-xxxxx MWST&lt;/em&gt;&lt;/li&gt;&#xA;&lt;li&gt;Date and ideally time&lt;/li&gt;&#xA;&lt;li&gt;Payment method&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Does this make more sense now? I just don't see how OCR gets me there. It will only be the method to extract the values.&lt;/p&gt;&#xA;" OwnerUserId="52371" LastEditorUserId="52371" LastEditDate="2018-05-19T23:38:41.500" LastActivityDate="2018-10-11T00:58:42.337" Title="How can you build a model that reads out receipts and invoices?" Tags="|image-recognition|information-retrieval|ocr|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31846" PostTypeId="2" ParentId="31835" CreationDate="2018-05-19T10:42:51.697" Score="1" Body="&lt;p&gt;It's not important to adjust to the opponent, as Go is only about winning or losing. There is no bigger reward the faster / the more obvious it wins.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or to put it different: only the current board situation is important in a min-Max setting (although the value approximation of a state admittedly depends a bit on the opponent)&lt;/p&gt;&#xA;" OwnerUserId="8820" LastActivityDate="2018-05-19T10:42:51.697" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31847" PostTypeId="2" ParentId="31835" CreationDate="2018-05-19T11:02:35.437" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But I would like to know if AlphaGo Zero can adapt to the way the oponent plays (oponent profile) or something like this. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That is not included in the algorithm as written, where the &quot;profile&quot; of the opponent is effectively AlphaGo Zero itself (learned through self play).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is not clear whether adapting play style to a given opponent would offer any advantage. It would be difficult to assess because AlphaGo Zero is such a strong player, that it will win a large percentage of games against human players as-is. Seeking and measuring any improvement, except versus earlier versions of itself, would be quite hard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, there are a likely a few places in the code where learned play style of an opponent &lt;em&gt;could&lt;/em&gt; in theory allow AlphaGo Zero to be more efficient. The most obvious is in the &quot;rollout&quot; policy (I'm not 100% sure if they use the same term), where the algorithm simulates and samples different possible trajectories through the game in order to predict likely outcomes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The current rollout policy in AlphaGo is learned through self play. But it is just a neural network that predicts probability of making plays given board state. It could easily be adjusted in a supervised learning fashion, based on sampled plays from an opponent. If it could be learned accurately, then it should make searches more efficient and accurate - the impossible but ideal situation being that it predicted opponents' move exactly and thus could quickly find the ultimate counter to their actions. In fact the original AlphaGo rollout policy did model human play in this way. It was based on large database of many human master level play moves, not a single player. The Deep Mind team did suggest in their paper that this gave better results &lt;em&gt;at the time&lt;/em&gt; than a self-play policy - they tried both and the human database was better. Since then, AlphaGo Zero has surpassed the performance of original AlphaGo without the database of human moves.&lt;/p&gt;&#xA;" OwnerUserId="836" LastActivityDate="2018-05-19T11:02:35.437" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31849" PostTypeId="1" AcceptedAnswerId="31850" CreationDate="2018-05-19T13:12:10.657" Score="-1" ViewCount="94" Body="&lt;p&gt;I have a dataset containing fraud and non fraudulent data. The system in place is a rule based engine with over 20 rules. If the total score is above a certain threshold the payment is classified as fraudulent. &#xA;What would be an effective way of using machine learning to optimize the assigned score to the different rules(fraud rules).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="52378" LastEditorUserId="8560" LastEditDate="2018-05-20T05:14:12.953" LastActivityDate="2018-05-20T05:14:12.953" Title="Using machine learning to optimize parameter scores" Tags="|machine-learning|data-mining|dataset|association-rules|" AnswerCount="1" CommentCount="1" ClosedDate="2018-05-20T13:34:46.600" ContentLicense="CC BY-SA 4.0" />
  <row Id="31850" PostTypeId="2" ParentId="31849" CreationDate="2018-05-19T13:32:36.440" Score="1" Body="&lt;p&gt;This is a classification problem. You have 20 features (I.e., rules) and output is binary (I.e., fraud or no fraud). It is not clear what the rule features are. It sounds like they might be binary (e.g., pass or fail). Do you have training data on which items are actually fraud and which are not? If so, you can train classification model. For example, you could perform logistic regression to predict probability of fraud with your rules as inputs. The coefficients assigned to each rule could be interpreted as the scores or weights for each rule. If the predicted probability of fraud  is above some probability threshold you would classify as fraud.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you do not have training data you can attempt to cluster similar rule results into two clusters. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT:&#xA;If you can define some loss function (e.g., accuracy of predictions) then you can set up an optimization problem to find the best coefficients (I.e., those that minimize error metric). This is just an optimization problem and suitable solvers are available depending on your coding language. However, your method of weighted rules will likely not perform as well as established classification methods. &lt;/p&gt;&#xA;" OwnerUserId="27702" LastEditorUserId="27702" LastEditDate="2018-05-19T18:47:35.897" LastActivityDate="2018-05-19T18:47:35.897" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31851" PostTypeId="2" ParentId="30516" CreationDate="2018-05-19T13:34:33.083" Score="5" Body="&lt;p&gt;&lt;strong&gt;Feature Extraction&lt;/strong&gt; is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;I do not like the views of @Candidate1 on #Topic1. Too conservative!! I can't stand it!&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;then we can't just feed this into a learning algorithm. We need to convert it into a proper format, so we perform &lt;strong&gt;pre-processing&lt;/strong&gt; on our data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To start, we might want to try to &lt;strong&gt;tokenize&lt;/strong&gt; the tweet. To tokenize is something like segmentation. If you're familiar with Python, we can use ready made libraries (like &lt;a href=&quot;https://www.nltk.org/&quot; rel=&quot;noreferrer&quot;&gt;NLTK&lt;/a&gt;) to aid us. Depending on how your tokenizer is made, you could transform the previous tweet into something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['I', 'do', 'not', 'like', 'the', 'views', 'of', '@', 'Candidate1', 'on', '#', 'Topic1', '.', 'Too', 'conservative', '!', '!', 'I', 'can', &quot;'&quot;, 't', 'stand', 'it', '!']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I manually segmented the tweet with the rule that words separated by a space should be tokens and punctuation should be tokenized separately. The way you build a tokenizer (or adjust the settings of a ready made one) will determine the output from a tweet. Notice how '@' and 'Candidate1' are separated? A tokenizer for regular text might not be able to identify that this is a social media entity -- a user mention. If you can adjust your tokenizer to account for social media identities and contractions (like &quot;can't&quot;), you could produce a list of tokens as such&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['I', 'do', 'not', 'like', 'the', 'views', 'of', '@Candidate1', 'on', '#Topic1', '.', 'Too', 'conservative', '!', '!', 'I', &quot;can't&quot;, 'stand', 'it', '!']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, you mentioned &lt;strong&gt;bigrams&lt;/strong&gt; and &lt;strong&gt;unigrams&lt;/strong&gt;. An n-grams (e.g. 1-gram == unigram) is just a sequence of tokens. So what we produced awhile ago was just a unigram list of tokens. If you want a bigram, then you'd want to take 2 tokens at a time. An example output of a bigram list of tokens would be &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;['I do', 'do not', 'not like', 'like the', 'the views', 'views of', 'of @Candaite1', '@Candidate1 on', 'on #Topic1', '#Topic1 .', '. Too', 'Too conservative', 'conservative !', '! !', '! I', &quot;I can't&quot;, &quot;can't stand&quot;, 'stand it', 'it !',]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Notice how words repeat? Imagine what a 3-gram or 5-gram would look like.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Before anything, why would we use bigrams over unigrams or those of higher n values? Well, the higher the n, the more about of order you're able to capture. Sometimes order is an important factor in learning. Playing around with how you'll represent your data might show you important features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now that we have our text tokenized, we can start extracting features! We can turning our example test, and other text samples, into a &lt;strong&gt;Bag-Of-Words (BOW) model&lt;/strong&gt;. Think of a BOW as a table with column headers as the words/terms and rows as your text samples/tweets. A cell could then contain the number of words/terms for a given sample of text. You could start with counting each term in a sample, so based on the tweet, you'd come up with something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tweet1: {&#xA;   'I': 2,&#xA;   'do': 1,&#xA;   'not': 1,&#xA;   'like': 1,&#xA;   'the': 1,&#xA;   ...&#xA;   '!': 3,&#xA;   ... &#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I don't want to manually write it all, but I hope you get the picture here. See how 'I' is 2 because it was mentioned twice in the sample. '!' was mentioned trice, so its value was 2. You'll find that there will be a lot of 1 values, specially in tweets, because there isn't room for much to be written.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You'd do this for each of your tweets and you'll come up with something like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       | 'I' | '!' | '#Candiate1' | ...&#xA;tweet1 |  2  |  3  |       1&#xA;tweet2 |  0  |  0  |       0&#xA;...    &#xA;tweetn |  1  |  5  |       0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a BOW! Adjust your n-grams and you'll get different features with different values. This representation is actually a simple starting point for you because this is now understandable by a computer. You mentioned you labeled your tweets, so imagine that table, but with your label appended to it.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;       | 'I' | '!' | '#Candiate1' | label&#xA;tweet1 |  2  |  3  |       1      | negative&#xA;tweet2 |  0  |  0  |       0      | negative&#xA;...    &#xA;tweetn |  1  |  5  |       0      | positive&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And with that, you can build a simple model with the appropriate steps after feature extraction.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There is a lot more to feature extraction! So many different combinations of methods to try out, but stick to things simple for now, if you don't feel so adventurous. Here are some things to consider:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Instead of counting terms, you could go with &lt;strong&gt;frequency&lt;/strong&gt; (i.e. number of occurrence of a term divided by total number of terms)&lt;/li&gt;&#xA;&lt;li&gt;Instead of extracting the words themselves, you can use &lt;strong&gt;part-of-speech (POS) tags&lt;/strong&gt; (POS tags are language dependent, so you'll have to look for a POS reference for the specific language you're dealing with)&lt;/li&gt;&#xA;&lt;li&gt;Consider extracting &lt;strong&gt;topic models&lt;/strong&gt; or maybe &lt;strong&gt;meta information&lt;/strong&gt; (avg length of words, avg length of tweets)&lt;/li&gt;&#xA;&lt;li&gt;Is there an &lt;strong&gt;emotion lexicon&lt;/strong&gt; for the language you're dealing with? Consider counting the number of emotion related words&lt;/li&gt;&#xA;&lt;li&gt;Consider reducing the number of unique words or applying weighted schemes (like &lt;strong&gt;TFIDF&lt;/strong&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;There are so many ways to approach this, but it ultimately boils down to a design you'll finalize. I guess the best advice I can give you is to refer to related literature. Look at other papers/articles that process text written by Nigerians. What are the methods they use to process text? Is it language dependent or independent? Did they reduce the number of features? Can I create the language resources? Once you identify a good source, try modifying and experimenting from there.&lt;/p&gt;&#xA;" OwnerUserId="52049" LastActivityDate="2018-05-19T13:34:33.083" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31853" PostTypeId="2" ParentId="23492" CreationDate="2018-05-19T14:50:53.333" Score="3" Body="&lt;p&gt;Yes, you can perturb your data (and targets) in ways that you wish your model to be robust against, for example by adding small amounts of noise (possibly Gaussian) or synthetic anomalies, or by creating meaningful aggregates. Some models also improve just by running the same training data through them more than once, randomly shuffled (“epochs”).&lt;/p&gt;&#xA;" OwnerUserId="52382" LastEditorUserId="52382" LastEditDate="2018-05-19T20:01:10.577" LastActivityDate="2018-05-19T20:01:10.577" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31854" PostTypeId="2" ParentId="31778" CreationDate="2018-05-19T14:59:00.570" Score="1" Body="&lt;p&gt;Try &lt;a href=&quot;https://prodi.gy&quot; rel=&quot;nofollow noreferrer&quot;&gt;prodigy&lt;/a&gt;. An annotation tool powered by active learning.&lt;/p&gt;&#xA;" OwnerUserId="13058" LastActivityDate="2018-05-19T14:59:00.570" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31855" PostTypeId="1" CreationDate="2018-05-19T15:11:06.093" Score="1" ViewCount="36" Body="&lt;p&gt;I remember reading an article comparing predicted probabilities of boosted, stacked encoders, random forests, and neural network algorithms. One important point I got from the article was that gbm pushes predicted probabilities towards 1 and zero. Does anyone know an algorithm that produces predicted probabilities that are close to 0.5?&lt;/p&gt;&#xA;" OwnerUserId="26733" LastEditorUserId="26733" LastEditDate="2018-05-25T18:13:54.910" LastActivityDate="2018-05-25T18:13:54.910" Title="Predicted Values in machine learning" Tags="|neural-network|random-forest|gbm|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-20T13:33:52.133" ContentLicense="CC BY-SA 4.0" />
  <row Id="31856" PostTypeId="2" ParentId="26175" CreationDate="2018-05-19T18:07:51.003" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;understand how the weight matrix in an LSTM cell is used  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;In LSTM you have a cell vector that keeps track of necessary information for the task at hand.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To backpropagate the errors from far away time steps, LSTM by design has simple linear operations (*/+) that update the cell vector, thus it’s very easy for the gradient to just flow.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The key idea is we manipulate the cell vector by linear interactions through &lt;strong&gt;gates&lt;/strong&gt;, where we introduce some non-linearity between the current input and the previous hidden features to obtain new features that range between zero to one, indicating how much we manipulate (add/remove/update) across each element of the new feature vector.   &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;how much information we need to keep track of?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;w_f:&lt;/strong&gt; guess from the interactions between the previous hidden features and the current input by concatenating both vectors and multiplying them with a weight matrix and applying the sigmoid non-linearity to get values between [0,1] (different for each element) to see how much should we forget from the old cell vector.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;modified_old_cell = old_cell * forgetting_some_dimensions&lt;/strong&gt; (forget gate f_t).  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;how much information we need to extract from the current input.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;we introduce two feature vectors, by concatenating both vectors (input,previous hiddens) and applying tanh, sigmoid to get proposed cell features, input gate features respectively.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;proposed_cell = non_linearity(previous_hiddens,input)&lt;/strong&gt;, hence we used tanh to get values between [-1,+1].  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;modified_proposed_cell = proposed_cell * input_gate&lt;/strong&gt;. hence we don't need to add all information from the current time step. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;current_cell = integration (modified_old_cell,modified_proposed_cell)&lt;/strong&gt;. hence the integration operation is simply the plus operation.  &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;how much information do we need to pass to the next time step, hence if the problem is sequence tagging, maybe some of the output features are only important for the current time step.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;the same mechanism as above:&lt;br&gt;&#xA;&lt;strong&gt;current_hidden_features = current_cell * outputting_some_dimensions&lt;/strong&gt; (output gate o_t).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-19T18:07:51.003" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31857" PostTypeId="2" ParentId="31845" CreationDate="2018-05-19T18:12:33.953" Score="1" Body="&lt;p&gt;You should start by trying OCR.  I am not sure why you are rejecting it without trying it. Start with Tesseract (free but not very good) and then try a commercial OCR as well (Abbyy is well regarded, but you could also try Adobe Acrobat).  Also, research document structure extraction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That idea that you are going to re-invent a machine learning model that does OCR better than existing OCR solutions seems ... not very realistic.  Current OCR tools have years or decades of engineering put into them.  There is no way you are going to be able to afford to put that much effort into your own custom tool.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Image quality on those example images is not wonderful, and you might have to accept that accuracy will be less than 100%.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-05-19T18:12:33.953" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31858" PostTypeId="2" ParentId="14843" CreationDate="2018-05-19T18:18:08.207" Score="0" Body="&lt;p&gt;Neural networks are best trained with stochastic gradient descent (with minibatches), not with conjugate gradient descent or other optimization methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I recommend using a framework designed for this, like Keras or Tensorflow.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Separately: I recommend using a convolutional network for this particular task, not a fully connected network.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-05-19T18:18:08.207" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31859" PostTypeId="1" CreationDate="2018-05-19T18:35:06.013" Score="1" ViewCount="66" Body="&lt;p&gt;If I want to suggest a course path for a student who wanted to be a chemical engineering where each degree has to go through certain mandatory courses like math ,physics chemistry . Again to complete these mandatory courses there is various pre-requisite courses . Again I have to suggest a best course path by breaking the course in season like fall,winter and summer etc according credit points rule of university .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So just wanted understand which algorithm or approach i should choose to solve this kind of problem . &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I thought of choosing heuristic approach find the best path but there is various mandatory courses and student has to complete each mandatory courses and heuristic A* path finding algorithm works if you know the source and destination  .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Again I thought of using Travelling Salesman problem approach where student has to travel mandatory course but in this case problem is each mandatory courses has again various path . so how do approach about this .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance , Please guide me friends . appreciate your earl reply.&lt;/p&gt;&#xA;" OwnerUserId="52387" LastEditorUserId="1330" LastEditDate="2021-07-09T01:39:32.173" LastActivityDate="2021-07-09T01:39:32.173" Title="What algorithm to choose order of university courses?" Tags="|algorithms|recommender-system|graphs|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31860" PostTypeId="1" CreationDate="2018-05-19T18:55:53.580" Score="3" ViewCount="1719" Body="&lt;p&gt;I am training a binary classification neural network model using matlab the graph that I got using 20 neurons in hidden layer is given below.&#xA;the confusion matrix and graph between cross entropy vs epochs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;to prevent overfitting in a model the training curve in a loss graph should be similar to the validation curve.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but in the current situation the third graph shows curve where validation curve is similar to training although the overall accuracy is low as compared to the curve where the two curve diverges in the above plot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;WHy this is happening and what I am doing wrong in understanding these curves?&#xA;&lt;a href=&quot;https://i.stack.imgur.com/zRHxY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zRHxY.png&quot; alt=&quot;20 neurons&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/QGqyc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QGqyc.png&quot; alt=&quot;confusin&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/U6dnn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U6dnn.png&quot; alt=&quot;cross&quot;&gt;&lt;/a&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/Pns81.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Pns81.png&quot; alt=&quot;confusion&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44413" LastActivityDate="2021-03-10T17:38:11.857" Title="Why training and validation similar loss curves lead to poor performance" Tags="|machine-learning|classification|matlab|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31861" PostTypeId="1" AcceptedAnswerId="31902" CreationDate="2018-05-19T19:04:38.840" Score="2" ViewCount="777" Body="&lt;p&gt;Okay so I wrote a very simple python code to read a wav file, get the mfcc features and then use kmeans clustering on the features. The hello.wav file has two different people saying hello at the same time, I wanted to output two separate voices&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from python_speech_features import mfcc&#xA;from scipy.io import wavfile&#xA;from sklearn.cluster import KMeans&#xA;sampfreq, data = wavfile.read('hello.wav')&#xA;&#xA;mfcc_feat = mfcc(data,sampfreq)&#xA;&#xA;kmeans = KMeans(n_clusters=2)&#xA;kmeans.fit(mfcc_feat)&#xA;print(kmeans.cluster_centers_)&#xA;print(kmeans.labels_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I got the labels and cluster centers but how do I get back my two separate sound samples using this?&lt;/p&gt;&#xA;" OwnerUserId="52389" LastActivityDate="2018-06-20T08:52:36.877" Title="How to get data back into two separate audio files after successfully applying kmeans clustering on an audio file?" Tags="|clustering|k-means|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31862" PostTypeId="1" CreationDate="2018-05-19T19:07:07.987" Score="1" ViewCount="329" Body="&lt;p&gt;I'd like to change the log level of yarn. For example, when I run &lt;code&gt;yarn node -list&lt;/code&gt;, I see a line that says &lt;code&gt;INFO client.RMProxy: Connecting to Resource Manager...&lt;/code&gt;. I don't want to see that. Is there an easy way?&lt;/p&gt;&#xA;" OwnerUserId="13672" LastActivityDate="2024-02-01T06:02:30.303" Title="Change log level of yarn?" Tags="|apache-hadoop|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31863" PostTypeId="2" ParentId="31860" CreationDate="2018-05-19T19:19:20.767" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;to prevent overfitting in a model the training curve in a loss graph&#xA;  should be similar to the validation curve.   &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's not always the case, maybe the train set has slightly a different distribution from the validation set, therefore, our focus is just when training loss decreases while validation loss slightly or never decreases.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the former graph, you notice there was a slight increase in the validation error while the training error was decreasing, while the latter graph has the same behavior due to a less complex model, the model was not able to fit the training data well. &lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-19T19:19:20.767" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31864" PostTypeId="1" CreationDate="2018-05-19T19:54:07.223" Score="11" ViewCount="5348" Body="&lt;p&gt;I would like to share a Jupyter Notebook with somebody who is not a programmer (via a URL). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that I must publish my work to GitHub and then use &lt;a href=&quot;http://nbviewer.jupyter.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://nbviewer.jupyter.org/&lt;/a&gt; to render the notebook. However, I don't know how to correctly publish my .ipynb file into Github.&lt;/p&gt;&#xA;" OwnerUserId="52390" LastEditorUserId="56354" LastEditDate="2020-05-18T20:13:41.490" LastActivityDate="2020-05-18T20:14:50.547" Title="Share Jupyter Notebook with a non programmer" Tags="|python|jupyter|" AnswerCount="5" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31865" PostTypeId="2" ParentId="31405" CreationDate="2018-05-20T00:40:57.160" Score="0" Body="&lt;p&gt;If I understand your problem, you could formulate the problem as a &lt;strong&gt;Classification task with missing inputs&lt;/strong&gt;&lt;br&gt;&#xA;When some of the inputs may be missing, rather than providing a single classification function, the learning algorithm must learn a set of functions. Each function corresponds to classifying x with a different subset of its inputs missing. &lt;em&gt;&quot;but we only&#xA;need to learn a single function describing the joint probability distribution of all of them&quot;.&lt;/em&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you could formulate it as &lt;strong&gt;Imputation of missing values&lt;/strong&gt;&lt;br&gt;&#xA;An algorithm is given a new example x but with some entries of x&#xA;missing. The algorithm must provide a prediction of the values of the missing entries. &lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-20T00:40:57.160" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31866" PostTypeId="1" AcceptedAnswerId="31871" CreationDate="2018-05-20T02:01:11.763" Score="0" ViewCount="56" Body="&lt;p&gt;I was prototyping a network architecture out on the macbook, and after finding something I was somewhat happy with, I wanted to test it out on a big data set on a system with a Titan V as the macbook was very slow for the bigger dataset (12 hours). I was expecting at least a 50x speed up per epoch over the CPU, if not more. Why might the speedup be only 4x (3 hours)? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Err it turned out to be some very tricky driver issues, and by just cleaning out my vm and starting over, was able to fix it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This line of code in particular was invaluable - some of the other checks on &quot;is tensorflow using the gpu&quot; I found hovering around the web were not adequate to solving this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import tensorflow as tf&#xA;with tf.device('/gpu:0'):&#xA;    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')&#xA;    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')&#xA;    c = tf.matmul(a, b)&#xA;&#xA;with tf.Session() as sess:&#xA;    print (sess.run(c))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52396" LastEditorUserId="52396" LastEditDate="2018-05-20T05:20:44.147" LastActivityDate="2022-07-04T01:07:53.617" Title="Moving from macbook (without GPU) to linux system with Titan V, only getting a 4x speedup, what am I doing wrong?" Tags="|keras|tensorflow|gpu|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31868" PostTypeId="2" ParentId="19882" CreationDate="2018-05-20T02:36:00.097" Score="3" Body="&lt;p&gt;In the past the Scikit-Learn wrapper &lt;code&gt;XGBRegressor&lt;/code&gt; and &lt;code&gt;XGBClassifier&lt;/code&gt; should get the feature importance using &lt;code&gt;model.booster().get_score()&lt;/code&gt;. Not sure from which version but now in &lt;code&gt;xgboost 0.71&lt;/code&gt; we can access it using&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.feature_importances_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="27423" LastActivityDate="2018-05-20T02:36:00.097" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31869" PostTypeId="2" ParentId="31864" CreationDate="2018-05-20T03:08:34.720" Score="7" Body="&lt;p&gt;GitHub has built in support for showing a notebook. You will just need to run the notebook yourself, then upload the file to Github like all other file types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your viewer will be able to see your notebook without any installation.&lt;/p&gt;&#xA;" OwnerUserId="9123" LastActivityDate="2018-05-20T03:08:34.720" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31870" PostTypeId="2" ParentId="31864" CreationDate="2018-05-20T04:31:59.910" Score="8" Body="&lt;p&gt;There is an option to convert the notebook to HTML. If the non programmer just have to view the notebook, do that then upload it in google drive or any website or you can share it even through mail. I do this every time when I want to present using a Jupyter notebook so that it will be supported on any system.&lt;/p&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-05-20T04:31:59.910" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31871" PostTypeId="2" ParentId="31866" CreationDate="2018-05-20T04:37:17.450" Score="2" Body="&lt;pre&gt;&lt;code&gt;# Copyright 2015 The TensorFlow Authors. All Rights Reserved.&#xA;#&#xA;# Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);&#xA;# you may not use this file except in compliance with the License.&#xA;# You may obtain a copy of the License at&#xA;#&#xA;#     http://www.apache.org/licenses/LICENSE-2.0&#xA;#&#xA;# Unless required by applicable law or agreed to in writing, software&#xA;# distributed under the License is distributed on an &quot;AS IS&quot; BASIS,&#xA;# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.&#xA;# See the License for the specific language governing permissions and&#xA;# limitations under the License.&#xA;# ==============================================================================&#xA;&quot;&quot;&quot;A script for testing that TensorFlow is installed correctly on Windows.&#xA;&#xA;The script will attempt to verify your TensorFlow installation, and print&#xA;suggestions for how to fix your installation.&#xA;&quot;&quot;&quot;&#xA;&#xA;import ctypes&#xA;import imp&#xA;import sys&#xA;&#xA;def main():&#xA;  try:&#xA;    import tensorflow as tf&#xA;    print(&quot;TensorFlow successfully installed.&quot;)&#xA;    if tf.test.is_built_with_cuda():&#xA;      print(&quot;The installed version of TensorFlow includes GPU support.&quot;)&#xA;    else:&#xA;      print(&quot;The installed version of TensorFlow does not include GPU support.&quot;)&#xA;    sys.exit(0)&#xA;  except ImportError:&#xA;    print(&quot;ERROR: Failed to import the TensorFlow module.&quot;)&#xA;&#xA;  candidate_explanation = False&#xA;&#xA;  python_version = sys.version_info.major, sys.version_info.minor&#xA;  print(&quot;\n- Python version is %d.%d.&quot; % python_version)&#xA;  if not (python_version == (3, 5) or python_version == (3, 6)):&#xA;    candidate_explanation = True&#xA;    print(&quot;- The official distribution of TensorFlow for Windows requires &quot;&#xA;          &quot;Python version 3.5 or 3.6.&quot;)&#xA;&#xA;  try:&#xA;    _, pathname, _ = imp.find_module(&quot;tensorflow&quot;)&#xA;    print(&quot;\n- TensorFlow is installed at: %s&quot; % pathname)&#xA;  except ImportError:&#xA;    candidate_explanation = False&#xA;    print(&quot;&quot;&quot;&#xA;- No module named TensorFlow is installed in this Python environment. You may&#xA;  install it using the command `pip install tensorflow`.&quot;&quot;&quot;)&#xA;&#xA;  try:&#xA;    msvcp140 = ctypes.WinDLL(&quot;msvcp140.dll&quot;)&#xA;  except OSError:&#xA;    candidate_explanation = True&#xA;    print(&quot;&quot;&quot;&#xA;- Could not load 'msvcp140.dll'. TensorFlow requires that this DLL be&#xA;  installed in a directory that is named in your %PATH% environment&#xA;  variable. You may install this DLL by downloading Microsoft Visual&#xA;  C++ 2015 Redistributable Update 3 from this URL:&#xA;  https://www.microsoft.com/en-us/download/details.aspx?id=53587&quot;&quot;&quot;)&#xA;&#xA;  try:&#xA;    cudart64_80 = ctypes.WinDLL(&quot;cudart64_80.dll&quot;)&#xA;  except OSError:&#xA;    candidate_explanation = True&#xA;    print(&quot;&quot;&quot;&#xA;- Could not load 'cudart64_80.dll'. The GPU version of TensorFlow&#xA;  requires that this DLL be installed in a directory that is named in&#xA;  your %PATH% environment variable. Download and install CUDA 8.0 from&#xA;  this URL: https://developer.nvidia.com/cuda-toolkit&quot;&quot;&quot;)&#xA;&#xA;  try:&#xA;    nvcuda = ctypes.WinDLL(&quot;nvcuda.dll&quot;)&#xA;  except OSError:&#xA;    candidate_explanation = True&#xA;    print(&quot;&quot;&quot;&#xA;- Could not load 'nvcuda.dll'. The GPU version of TensorFlow requires that&#xA;  this DLL be installed in a directory that is named in your %PATH%&#xA;  environment variable. Typically it is installed in 'C:\Windows\System32'.&#xA;  If it is not present, ensure that you have a CUDA-capable GPU with the&#xA;  correct driver installed.&quot;&quot;&quot;)&#xA;&#xA;  cudnn5_found = False&#xA;  try:&#xA;    cudnn5 = ctypes.WinDLL(&quot;cudnn64_5.dll&quot;)&#xA;    cudnn5_found = True&#xA;  except OSError:&#xA;    candidate_explanation = True&#xA;    print(&quot;&quot;&quot;&#xA;- Could not load 'cudnn64_5.dll'. The GPU version of TensorFlow&#xA;  requires that this DLL be installed in a directory that is named in&#xA;  your %PATH% environment variable. Note that installing cuDNN is a&#xA;  separate step from installing CUDA, and it is often found in a&#xA;  different directory from the CUDA DLLs. You may install the&#xA;  necessary DLL by downloading cuDNN 5.1 from this URL:&#xA;  https://developer.nvidia.com/cudnn&quot;&quot;&quot;)&#xA;&#xA;  cudnn6_found = False&#xA;  try:&#xA;    cudnn = ctypes.WinDLL(&quot;cudnn64_6.dll&quot;)&#xA;    cudnn6_found = True&#xA;  except OSError:&#xA;    candidate_explanation = True&#xA;&#xA;  if not cudnn5_found or not cudnn6_found:&#xA;    print()&#xA;    if not cudnn5_found and not cudnn6_found:&#xA;      print(&quot;- Could not find cuDNN.&quot;)&#xA;    elif not cudnn5_found:&#xA;      print(&quot;- Could not find cuDNN 5.1.&quot;)&#xA;    else:&#xA;      print(&quot;- Could not find cuDNN 6.&quot;)&#xA;      print(&quot;&quot;&quot;&#xA;  The GPU version of TensorFlow requires that the correct cuDNN DLL be installed&#xA;  in a directory that is named in your %PATH% environment variable. Note that&#xA;  installing cuDNN is a separate step from installing CUDA, and it is often&#xA;  found in a different directory from the CUDA DLLs. The correct version of&#xA;  cuDNN depends on your version of TensorFlow:&#xA;&#xA;  * TensorFlow 1.2.1 or earlier requires cuDNN 5.1. ('cudnn64_5.dll')&#xA;  * TensorFlow 1.3 or later requires cuDNN 6. ('cudnn64_6.dll')&#xA;&#xA;  You may install the necessary DLL by downloading cuDNN from this URL:&#xA;  https://developer.nvidia.com/cudnn&quot;&quot;&quot;)&#xA;&#xA;  if not candidate_explanation:&#xA;    print(&quot;&quot;&quot;&#xA;- All required DLLs appear to be present. Please open an issue on the&#xA;  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues&quot;&quot;&quot;)&#xA;&#xA;  sys.exit(-1)&#xA;&#xA;if __name__ == &quot;__main__&quot;:&#xA;  main()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Copy paste, run this code. Will tell you if you have tensorflow GPU support.&#xA;For example, on my MacBook Pro it says&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;TensorFlow successfully installed.&#xA;The installed version of TensorFlow does not include GPU support.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="44680" LastActivityDate="2018-05-20T04:37:17.450" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31872" PostTypeId="1" CreationDate="2018-05-20T06:12:02.837" Score="10" ViewCount="11870" Body="&lt;p&gt;Why the area under the ROC Curve for a random classifier is equal to 0.5 and has diagonal shape?&#xA;For me a random classifier would have 25% of TP,TN,FP,FN and therefore it would only be a single point on the ROC Curve.&lt;/p&gt;&#xA;" OwnerUserId="52379" LastActivityDate="2021-02-19T15:51:35.120" Title="AUC-ROC of a random classifier" Tags="|classification|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31873" PostTypeId="2" ParentId="9248" CreationDate="2018-05-20T06:58:49.237" Score="0" Body="&lt;p&gt;You can use Gower distance to obtain a similarity matrix when the data is mixed-type. This function is implemented in R packages &lt;code&gt;cluster&lt;/code&gt;, &lt;code&gt;CluMix&lt;/code&gt; and &lt;code&gt;FD&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In cluster package, this is implemented in the function &lt;code&gt;daisy&lt;/code&gt;. &#xA;An example use would be, &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; diss_mat &amp;lt;- daisy(data, metric = &quot;gower&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the columns of &lt;code&gt;data&lt;/code&gt; contain the variables. You have to assign the correct type (numeric, factor or ordered factor) for the variables.&lt;/p&gt;&#xA;" OwnerUserId="40805" LastActivityDate="2018-05-20T06:58:49.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31874" PostTypeId="1" AcceptedAnswerId="31912" CreationDate="2018-05-20T10:11:36.057" Score="1" ViewCount="41" Body="&lt;p&gt;I am using a recurrent network for time series forecasting.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction from the last cell in the network always seems to be the most accurate. For example if I have 20 cells (so my input samples are of length 20), the prediction from the latter cells (e.g. cells 11-20) seem better than the predictions from the early cells (e.g. cells 1-10).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this normal?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume my network only has 3 cells. At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach? If I want good predictions at all timesteps, do I need to using a &quot;moving window&quot; i.e. should I use a rolling sequence of inputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;For prediction at t=0, input x(t-2, t-1, t=0)&#xA;For prediction at t=1, input x(t-1, t=0, t=1)&#xA;For prediction at t=2, input x(t=0, t+1, t+2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47380" LastActivityDate="2018-05-21T09:29:54.973" Title="Last cell in recurrent network always the most accurate" Tags="|tensorflow|time-series|forecasting|rnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31875" PostTypeId="2" ParentId="31872" CreationDate="2018-05-20T10:21:47.380" Score="2" Body="&lt;p&gt;A couple of definitions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;True positive rate (TPR): Probability that a positive is labeled as positive.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;False positive rate (FPR): Probability that a negative is labeled as positive&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;ROC curves are plotted by varying the threshold of output score of the classifier (above which an instance is classified as positive, and below which, it is negative), computing and plotting the true positive rate (y-axis) and false positive rate (x-axis) for each threshold value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &quot;random&quot; classifier assigns a score sampled from the uniform distribution between 0 and 1, to each instance. If the threshold selected is 'x', then any instance having score above 'x' is positive. For each instance (irrespective of whether it is actually positive or negative), the probability of being labeled positive is 1-x. As x is varied between 1 and 0, both TPR and FPR vary between 0 and 1 (being equal at all points). Hence the line obtained is x=y (diagonal), and the area under this line can be computed as 0.5.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-20T10:21:47.380" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31877" PostTypeId="2" ParentId="31872" CreationDate="2018-05-20T12:47:41.473" Score="21" Body="&lt;p&gt;&lt;strong&gt;Premise&lt;/strong&gt;:&#xA;The confusion matrix that you mention above is only correct if&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;there is as much Positive and Negative cases in the data&lt;/li&gt;&#xA;&lt;li&gt;we use a random classifier that assigns Positive and Negative class both with probability 0.5&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;So you derive the confusion matrix for a very particular random classifier for a very particular dataset. This very particular random classifier is indeed, as you point out, only a point in the ROC plot. &lt;strong&gt;To obtain the whole ROC curve, we have to vary the probability with which we assign the positive class, from 0 to 1.&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;So in effects, the ROC curve is a graphical evaluation of the performance of infinitely many classifiers!&lt;/p&gt;&#xA;&lt;p&gt;Each one of these random classifiers with a different probability will have a different expected confusion matrix.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Derivation&lt;/strong&gt;: Here I derive the AUC of a random classifier on a dataset with an arbitrary class imbalance.&lt;/p&gt;&#xA;&lt;p&gt;Assuming that there is a fraction &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt; of positive cases and a fraction &lt;span class=&quot;math-container&quot;&gt;$1-x$&lt;/span&gt; of negative cases, and that our classifier consists of randomly assigning the positive class with probability &lt;span class=&quot;math-container&quot;&gt;$\rho$&lt;/span&gt; and the negative class with probability &lt;span class=&quot;math-container&quot;&gt;$1-\rho$&lt;/span&gt;. The confusion matrix of a random classifier will have the following expected proportions&#xA;&lt;span class=&quot;math-container&quot;&gt;$$TP = \rho x \\ FP = \rho (1-x) \\ FN = (1-\rho)x \\ TN = (1-\rho)(1-x)$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;Then we calculate the True Positive Rate (sensitivity) and False Positive Rate (1-specificity) of our random classifier&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$TPR=\frac{TP}{TP+FN}=\frac{\rho x}{\rho x + (1-\rho)x}=\rho\\ FPR=\frac{FP}{TN+FP}=\frac{\rho(1-x)}{\rho(1-x)+(1-\rho)(1-x)}=\rho$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;So as you can see TPR and TFR do not depend on the class proportion &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;, which means that the ROC-AUC will also be independent of &lt;span class=&quot;math-container&quot;&gt;$x$&lt;/span&gt;. Moreover TPR=FPR, meaning that the ROC-curve will look like a straight line from (0,0) to (1,1). Now we calculate the AUC as the integral between 0 and 1 of the area under TPR as a function of FPR as we vary the threshold &lt;span class=&quot;math-container&quot;&gt;$\rho$&lt;/span&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$AUC=\int_0^1 \rho d\rho' = \frac{\rho'^2}{2}|_0^1 = 1/2$$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;p&gt;So the area under the ROC curve for a random classifier is 0.5 regardless of the class proportion.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2021-02-19T15:51:35.120" LastActivityDate="2021-02-19T15:51:35.120" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31879" PostTypeId="2" ParentId="22002" CreationDate="2018-05-20T14:07:39.567" Score="1" Body="&lt;p&gt;You should normalize your input data for different purposes. As you can read from &lt;a href=&quot;https://datascience.stackexchange.com/a/31295/28175&quot;&gt;here&lt;/a&gt;, &lt;em&gt;Normalizing data is done for accelerating optimization. If you have features with different scales, it will take too much time for your optimizer function to find optimal points. Suppose you have age feature which can change between 0 to 150 (!) and salary which can be changed from 0 to whatever, like 500,000,000 $. your optimization algorithm used in your ML model will take too much time, if possible, to find appropriate weights for each feature. Moreover, if you don't scale your data, your ML algorithm may take too much care to features with large scales&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You have not explicitly specified your task, if its classification, you may need to turn the labels into one-hot encoded versions or if possible you may need to use &lt;a href=&quot;https://en.wikipedia.org/wiki/Word2vec&quot; rel=&quot;nofollow noreferrer&quot;&gt;word embeding&lt;/a&gt; for adding logical distance to different classes. If your task is a regression one, you may need to &lt;a href=&quot;https://datascience.stackexchange.com/a/26960/28175&quot;&gt;normalize your outputs&lt;/a&gt;. &lt;em&gt;In regression problems it is customary to normalize the output too, because the scale of output and input features may differ.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-20T20:06:30.923" LastActivityDate="2018-05-20T20:06:30.923" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31880" PostTypeId="1" CreationDate="2018-05-20T14:26:16.023" Score="1" ViewCount="31" Body="&lt;p&gt;I'd like to solve the problem that denoted in the below picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Fmf6M.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Fmf6M.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With which network could I try and start to solve this problem?&lt;/p&gt;&#xA;" OwnerUserId="51672" LastActivityDate="2018-05-20T14:26:16.023" Title="Rephrase Neural Network: Where to Start?" Tags="|deep-learning|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-26T19:52:08.357" ContentLicense="CC BY-SA 4.0" />
  <row Id="31881" PostTypeId="1" CreationDate="2018-05-20T16:16:00.043" Score="1" ViewCount="324" Body="&lt;p&gt;In Rapidminer, are the decision tree's weights a measure of the &quot;importance&quot; of attributes in the splitting procedure ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If yes, why is useful to know these weights ?&#xA;Are there better methods to know the most discriminant features in a data set ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="1330" LastEditDate="2021-07-09T01:36:52.273" LastActivityDate="2024-02-21T19:09:21.270" Title="Rapidminer and decision tree weights" Tags="|machine-learning|data-mining|feature-selection|decision-trees|rapidminer|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31882" PostTypeId="2" ParentId="31405" CreationDate="2018-05-20T16:50:54.377" Score="0" Body="&lt;p&gt;The simplest solution is to build 10 models, one per stage. It will enable you to use different features or even different algorithms for each stage. &lt;/p&gt;&#xA;" OwnerUserId="24162" LastActivityDate="2018-05-20T16:50:54.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31883" PostTypeId="2" ParentId="31864" CreationDate="2018-05-20T17:11:48.847" Score="3" Body="&lt;p&gt;Export as HTML is probably the best option. All the graphics are embedded in the HTML file and anyone can view it in a browser pretty easily just by clicking on the file. &lt;/p&gt;&#xA;" OwnerUserId="44299" LastActivityDate="2018-05-20T17:11:48.847" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31885" PostTypeId="1" CreationDate="2018-05-20T18:37:59.807" Score="0" ViewCount="380" Body="&lt;p&gt;I am currently implementing a forecasting model. The whole builds on a regression. The first model is an LSTM network based on Keras and the second a Random-Forest model based on sklearn.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use the same input values and target variables for both models. The forecasts of the LSTM network are closer to the actual observed values (better rmse and r2) (probably due to the memory / lookback), but the Random Forest model provides better altitudes and depths. It seems that the Random-Forest model has found better dependencies than the LSTM model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;LSTM:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NzyhH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NzyhH.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Random-Forest:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/vBpQC.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/vBpQC.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best way for the LSTM model to understand finer dependencies? Should I use an ensemble or stack method from LSTM + RF? Does it make sense?&lt;/p&gt;&#xA;" OwnerUserId="41968" LastActivityDate="2018-05-21T09:15:30.270" Title="LSTM doesnt find finer dependencies than the Random Forest model" Tags="|neural-network|predictive-modeling|regression|random-forest|lstm|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31886" PostTypeId="1" CreationDate="2018-05-20T18:44:47.407" Score="2" ViewCount="71" Body="&lt;p&gt;So I have chosen &quot;Orange&quot; for my research as a main app for clustering ant it's analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Excel 2016 is primary app used for dashboard (via &quot;pivot table&quot;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Problem - how to add graphs from &quot;scatter plot&quot; to excel dashboard and sync it at certain time? Is there a way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Orange provided reporting does not help, maybe there is another - more interactive way? because all I can get now is images via hyperlinks, which is not interactive.&lt;/p&gt;&#xA;" OwnerUserId="52422" LastActivityDate="2018-05-20T18:44:47.407" Title="Orange &quot;Scatter plot&quot; widget to excel, best way" Tags="|orange|excel|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31887" PostTypeId="1" CreationDate="2018-05-20T18:59:44.430" Score="1" ViewCount="2041" Body="&lt;p&gt;What's immediate superset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why is it called immediate?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They appear in the context of Apriori algorithm. Particularly the one enhanced with hashing.&lt;/p&gt;&#xA;" OwnerUserId="51687" LastEditorUserId="40326" LastEditDate="2018-10-18T05:54:56.197" LastActivityDate="2019-04-16T12:02:35.467" Title="What's immediate superset?" Tags="|algorithms|association-rules|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31888" PostTypeId="1" CreationDate="2018-05-20T19:44:58.143" Score="1" ViewCount="32" Body="&lt;p&gt;Any idea to what $\bar{X}$ refers to in association rules? Is it &quot;not-$X$&quot;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's in the context of interestingness measure for rule $X \rightarrow Y$.&lt;/p&gt;&#xA;" OwnerUserId="51687" LastActivityDate="2018-05-20T22:47:19.160" Title="Any idea to what $\bar{X}$ refers to in association rules? Is it &quot;not-$X$&quot;?" Tags="|association-rules|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31889" PostTypeId="2" ParentId="31860" CreationDate="2018-05-20T20:18:22.300" Score="2" Body="&lt;blockquote&gt;&#xA;&lt;p&gt;to prevent overfitting in a model the training curve in a loss graph&#xA;should be similar to the validation curve&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Similar in what way? If your training error keeps decreasing and the validation error started growing, you are overfitting. If you have a binary problem, why is the confusion matrix 3x3? Validation set accuracy after 33 epochs looks better than after 8. I don't see a problem.&lt;/p&gt;&#xA;" OwnerUserId="1426" LastEditorUserId="29169" LastEditDate="2021-03-10T17:38:11.857" LastActivityDate="2021-03-10T17:38:11.857" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31891" PostTypeId="1" CreationDate="2018-05-20T20:42:05.283" Score="2" ViewCount="28" Body="&lt;p&gt;Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) &gt; sim(Q,Y).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;However, examples of trickier cases would be: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) &gt; sim(Q,X) (assuming the car shapes are more or less the same).&lt;/li&gt;&#xA;&lt;li&gt;Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) &gt; sim(Q,Y).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?&lt;/p&gt;&#xA;" OwnerUserId="46305" LastActivityDate="2018-05-20T20:42:05.283" Title="Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?" Tags="|algorithms|computer-vision|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31893" PostTypeId="1" CreationDate="2018-05-20T21:25:55.420" Score="2" ViewCount="31" Body="&lt;p&gt;Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or more specifically:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are 100 people pressing buttons in 10 colors, with more than 200k presses collected altogether.&#xA;I want to test for individual differences in response-time (RT) to the buttons, what's the best way to do this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(remember there are differences in response time in-between the different colors).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I thought of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;ANOVA - calculate a general (weighted) mean RT and compare to individual people.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The distribution level: compare the mean distribution of RT to individual distributions using:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;chi-squared for goodness of fit&lt;/li&gt;&#xA;&lt;li&gt;Kullback–Leibler divergence test&lt;/li&gt;&#xA;&lt;li&gt;Kolmogorov–Smirnov test&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Find a cool bayesian way to do it (which?)&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;Use a mixed-effect regression model, and calculate the marginal contribution of the people (R square test).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;please help me decide - what's the best practice here?&#xA;I'd love to learn about another way I didn't think of that would do the trick.&lt;/p&gt;&#xA;" OwnerUserId="50163" LastActivityDate="2021-08-20T08:01:30.123" Title="Individual differences in response time (RT) experiment - searching for the right test" Tags="|categorical-data|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31894" PostTypeId="2" ParentId="31601" CreationDate="2018-05-20T21:26:56.410" Score="1" Body="&lt;p&gt;How I understand your data is a single feature of temperature. LSTM works with RNN time series like &lt;code&gt;(batch_size, timesteps, feature)&lt;/code&gt;. In your case it is &lt;code&gt;(none, 10000, 1)&lt;/code&gt;. Forget about the first one, it's technical.&lt;/p&gt;&#xA;" OwnerUserId="52426" LastActivityDate="2018-05-20T21:26:56.410" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31895" PostTypeId="1" CreationDate="2018-05-20T21:34:09.857" Score="3" ViewCount="584" Body="&lt;p&gt;if I say that in hadoop cluster system...       A client has 10GB data to send to name node. After analysing the &quot;packet capturing&quot; actually the name node doesn't divide the data for data node, actually all the communication happens only between client and data node , after taking the IPs of data node from name node, client directly connect to data node and after that he decide , in how many partitions  he is sending the data by dividing the blocks.&#xA;And then the data is receives by the data node and during this time name node contact to the data nodes to be prepare of replicating these data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is that the by this process I realized that the data sent by client is receives by data node in serial manner and date node copies that data regularly and hand to hand. Then if I sending a file of 100GB and usually it takes 100 minutes but after using tha Hadoop my speed and time is consuming more than before because the data node receives the data serially from client .&#xA;&quot;IS THIS A MAJOR DISADVANTAGE OF HADOOP??&quot;&lt;/p&gt;&#xA;" OwnerUserId="52429" LastEditorUserId="924" LastEditDate="2018-05-21T00:30:38.847" LastActivityDate="2018-05-21T00:30:38.847" Title="Communication between name node, data node and client in hadoop by analysing Packet Capturing" Tags="|bigdata|apache-hadoop|" AnswerCount="0" CommentCount="1" ClosedDate="2020-06-18T21:52:30.293" ContentLicense="CC BY-SA 4.0" />
  <row Id="31896" PostTypeId="2" ParentId="31888" CreationDate="2018-05-20T22:47:19.160" Score="1" Body="&lt;p&gt;Do you have a link to the actual source you are referring to? I haven't myself seen it before in the context of association rules and interestingness.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;As was pointed out by &lt;em&gt;@Sean Owen&lt;/em&gt; in his comment, it is one of a few standard notations for saying something is not in a &lt;strong&gt;set&lt;/strong&gt; - i.e. the complement. This would make most sense, as we are generally referring to &lt;strong&gt;itemsets&lt;/strong&gt; when computing the associations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Regarding the complement and set notation, &lt;a href=&quot;https://en.wikipedia.org/wiki/Complement_(set_theory)&quot; rel=&quot;nofollow noreferrer&quot;&gt;the Wikipedia article contains this&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The absolute complement of A is usually denoted by $${\displaystyle  A^{\complement }}$$ &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Other notations include $$ {\displaystyle A^{\text{c}}}, \hspace{3mm} \overline A, \hspace{3mm} A',\hspace{3mm} \complement _{U}A \hspace{3mm} and \hspace{3mm} \complement A$$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Of course a &lt;em&gt;bar&lt;/em&gt; can also mean things such as the mean or expectation value of some values (I have even seen people using it to mean the unobserved or predicted value!), but with the information you provide, the complement seems most likely.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here also from the same Wiki page linked above for those who are unsure, a visual definition of the complement:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UzUPV.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UzUPV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-20T22:47:19.160" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31897" PostTypeId="2" ParentId="31887" CreationDate="2018-05-20T23:07:24.300" Score="2" Body="&lt;p&gt;&lt;em&gt;Immediate&lt;/em&gt; often occurs simply to say that there are perhaps many supersets, e.g. when we have a tree like structure. There is also for example, the converse, an &lt;em&gt;immediate subset&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;Example&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Set &lt;strong&gt;A&lt;/strong&gt; has a superset &lt;strong&gt;B&lt;/strong&gt;, and that superset has its own superset &lt;strong&gt;C&lt;/strong&gt;. In this case, &lt;strong&gt;A&lt;/strong&gt; must also be contained within &lt;strong&gt;C&lt;/strong&gt;, but the set &lt;strong&gt;B&lt;/strong&gt; lies within that realm and so is the immedaite superset of &lt;strong&gt;A&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps it is best visualised with a Venn diagram, where &lt;strong&gt;B&lt;/strong&gt; is the immediate superset of &lt;strong&gt;A&lt;/strong&gt; and equally the immediate subset of &lt;strong&gt;C&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ke0F8.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ke0F8.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;There is some discussion &lt;a href=&quot;https://stats.stackexchange.com/questions/77465/maximal-closed-frequent-answer-included&quot;&gt;here on Cross-Validated&lt;/a&gt; too, where the selected answer shows some further usage of the term (albeit in a slightly different context). Have a look at &lt;a href=&quot;http://www.cs.kent.edu/~jin/DM08/FIM.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this source for some usage of the term&lt;/a&gt;, which leads you to my understanding.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-05-20T23:07:24.300" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31898" PostTypeId="2" ParentId="31806" CreationDate="2018-05-21T00:01:48.880" Score="2" Body="&lt;p&gt;Another thing to consider, if you are unable to refit the full model, but are able to access its fitted values (i.e.as score or probability values), you can try setting up a new logistic regression model with both the fitted values and the new variable as covariates for the same response variable. In this way you will see, whether all of the newly provided information was already captured by the existing model, or will the new covariate add further out of sample accuracy. &lt;/p&gt;&#xA;" OwnerUserId="23334" LastActivityDate="2018-05-21T00:01:48.880" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31899" PostTypeId="1" CreationDate="2018-05-21T03:21:59.340" Score="1" ViewCount="751" Body="&lt;p&gt;I was wondering what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially, on different set of data and the model refines itself. Are there any algorithms which support this mathematically, moreover, is there some tradeoff in runtime or accuracy of these models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Iteration here not to be confused with : What does iteration in Machine Learning mean?&lt;/p&gt;&#xA;" OwnerUserId="52439" LastEditorUserId="29575" LastEditDate="2018-05-21T11:17:39.117" LastActivityDate="2018-05-21T11:17:39.117" Title="Iterative Machine Learning" Tags="|machine-learning|machine-learning-model|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31900" PostTypeId="1" AcceptedAnswerId="31906" CreationDate="2018-05-21T04:03:34.020" Score="2" ViewCount="172" Body="&lt;p&gt;Does this mean that as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any additional things i can interpret from below ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/hATIx.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/hATIx.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52440" LastActivityDate="2018-05-28T08:02:58.183" Title="How do i interpret this correlation" Tags="|correlation|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31901" PostTypeId="1" CreationDate="2018-05-21T05:21:52.130" Score="3" ViewCount="760" Body="&lt;p&gt;I'm using a pre-trained ResNet50 model in keras and am trying to see predictions for single samples&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue is that &lt;code&gt;model.predict()&lt;/code&gt; is returning different values, depending on when I input a single sample vs the same sample within a larger array.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My model outputs a 1x6 vector of probabilities&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I run on a single sample:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case A: &lt;code&gt;model.predict(X_test[0:1])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and within an array of multiple samples:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case B: &lt;code&gt;model.predict(X_test[0:2])&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction for &lt;code&gt;X_test[0]&lt;/code&gt; is&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case A: &lt;code&gt;[9.999e-01, 4.228e-06, 8.278e-05, 1.121e-06, 1.061e-06, 2.958e-05]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case B: &lt;code&gt;[1.000e+00, 7.702e-13, 2.969e-13, 6.343e-11, 7.477e-14, 1.304e-10]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why are the predictions different? (predictions here being the probabilities themselves, not &lt;code&gt;argmax&lt;/code&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As requested, the main code is below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import load_model&#xA;from my_utils import load_dataset&#xA;import numpy as np&#xA;&#xA;X_train, Y_train, X_test, Y_test, classes = load_dataset()&#xA;&#xA;model = load_model('model.h5') # ResNet50 with pre-trained weights&#xA;&#xA;model.predict(X_test[0:1]) # Case A&#xA;&#xA;model.predict(X_test[0:2]) # Case B&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The model architecture of ResNet50 can be found &lt;a href=&quot;http://ethereon.github.io/netscope/#/gist/db945b393d40bfa26006&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;; the one difference with the implementation I'm using is that I'm using 6 classes not 1000&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="52442" LastEditorUserId="52442" LastEditDate="2018-05-21T06:45:56.790" LastActivityDate="2024-01-26T14:00:25.067" Title="Keras single sample prediction returns different values" Tags="|keras|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31902" PostTypeId="2" ParentId="31861" CreationDate="2018-05-21T06:58:21.880" Score="2" Body="&lt;p&gt;What you are trying to do is Blind Source Separation, and unfortunately it does not works as easy as that. When you are extracting MFCC from an audio file with a mixture of voices, MFCC clusters won't represent two people. For the separation, you should end up with two sound signals with the same number of samples. The MFCC are used as the feature's space for speakaer recognition tasks, for example, but not for separation in general. For more mathematical background about BSS you can visit: &lt;a href=&quot;https://arxiv.org/pdf/1603.03089.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://arxiv.org/pdf/1603.03089.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The most simple case is the so-called Cocktail Party Problem, in where you have at least as many microphones as persons and the mixture has to be instantaneous (no delay between signals, only attenuations due to path-losses). This very simple case can be solved applying Independent Component Analysis, with the FastICA algorithm: &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For delayed mixtures you can try the DUET algorithm: &lt;a href=&quot;https://pdfs.semanticscholar.org/1413/746141f2871e0f45056a7696e019b8f8a100.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pdfs.semanticscholar.org/1413/746141f2871e0f45056a7696e019b8f8a100.pdf&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For convolved mixtures and undetermined problems (more speakers than microphones) this PARAFAC-based algorithm performs quite well: &lt;a href=&quot;http://people.ece.umn.edu/~nikos/05233821.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://people.ece.umn.edu/~nikos/05233821.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-21T06:58:21.880" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31903" PostTypeId="1" CreationDate="2018-05-21T07:01:47.843" Score="1" ViewCount="35" Body="&lt;p&gt;While designing a search system, which searches in N identifiable categories, how many search queries does one need in each category to validate the target metric (DCG) scores accurately (balanced variance and bias)? does this number depend on N or the corpus size or both? Please add any publications possible.&#xA;I would also like the understand if effect size and and bayesian effective sample sizes play some role here. &lt;br&gt;&lt;br&gt;&#xA;Given a set of search queries &lt;strong&gt;Q&lt;/strong&gt; for retrieving documents from another set &lt;strong&gt;&lt;em&gt;D&lt;/em&gt;&lt;/strong&gt;, where there are &lt;strong&gt;N&lt;/strong&gt; possible categories in the corpus with &lt;strong&gt;k(n)&lt;/strong&gt; documents in each set, the objective is to define test minset(&lt;strong&gt;Q&lt;/strong&gt;) such that DCG obtained is reliable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example: Searching for Fruits &lt;br&gt;&#xA;Search queries: [Red apple, Green apple] &lt;br&gt;&#xA;Category: Apple&lt;br&gt;&#xA;Documents: [Red Apple, Gala Apple, Green Apple,...]k1 documents&lt;br&gt;&#xA;Search queries: [Alphonso, Haden, Keitt]&lt;br&gt;&#xA;Category: Mango&lt;br&gt;&#xA;Documents: [Keitt, Kent, Haden,....]k2 documents&lt;br&gt;&#xA;.&lt;br&gt;&#xA;.&lt;br&gt;&#xA;.&lt;br&gt;&#xA;The search results will be ranked and the accuracy of search is measured with DCG. How many search queries do you need in each set to have a reliable DCG as a function of the categories and the document set?&lt;/p&gt;&#xA;" OwnerUserId="13548" LastEditorUserId="13548" LastEditDate="2018-05-22T11:26:47.017" LastActivityDate="2018-05-22T11:26:47.017" Title="Search Query Sample Size Determination for validation set" Tags="|statistics|sampling|search|search-engine|learning-to-rank|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31904" PostTypeId="1" AcceptedAnswerId="31907" CreationDate="2018-05-21T07:40:11.247" Score="4" ViewCount="6315" Body="&lt;p&gt;I want to make simple predictions with Keras and I'm not really sure if I am doing it right. My data looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;col1,col2&#xA;1.68,237537&#xA;1.69,240104&#xA;1.70,244885&#xA;1.71,246196&#xA;1.72,246527&#xA;1.73,254588&#xA;1.74,255112&#xA;1.75,259035&#xA;1.76,267229&#xA;1.77,267314&#xA;1.78,268931&#xA;1.79,273497&#xA;1.80,273900&#xA;1.81,277132&#xA;1.82,278066&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, I want to predict &lt;code&gt;col2&lt;/code&gt; by &lt;code&gt;col1&lt;/code&gt; and this is how I'm doing it:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;df = pandas.read_csv('data.csv', usecols=[0, 1], header=None)&#xA;X = df.iloc[:, :-1].values.astype(np.float64)&#xA;y = df.iloc[:, -1:].values.astype(np.float64)&#xA;scalarX, scalarY = MinMaxScaler(), MinMaxScaler()&#xA;scalarX.fit(X)&#xA;scalarY.fit(y.reshape(len(y),1))&#xA;X = scalarX.transform(X)&#xA;y = scalarY.transform(y.reshape(len(y),1))&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(4, input_dim=1, activation='relu'))&#xA;model.add(Dense(4, activation='relu'))&#xA;model.add(Dense(1, activation='linear'))&#xA;model.compile(loss='mse', optimizer='adam')&#xA;model.fit(x=X, y=y, epochs=3, verbose=1)&#xA;for num in range(1, 21):&#xA;    Xnew = np.array([[float(Decimal('2.{}'.format(num)))]])&#xA;    ynew = model.predict(Xnew)&#xA;    print(&quot;X=%s, Predicted=%s&quot; % (Xnew[0], ynew[0]))&#xA;&lt;/pre&gt;&#xA;" OwnerUserId="52444" LastEditorUserId="24000" LastEditDate="2018-05-21T21:54:15.877" LastActivityDate="2018-05-21T21:54:15.877" Title="Simple prediction with Keras" Tags="|python|keras|predictive-modeling|prediction|" AnswerCount="1" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="31905" PostTypeId="1" CreationDate="2018-05-21T07:59:39.183" Score="1" ViewCount="168" Body="&lt;p&gt;i am trying to use the dynamic rnn for toy sequence clasification . the code is copied from github link as:&lt;a href=&quot;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py&lt;/a&gt;.&#xA;The errors i am getting during predc function are as: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;ValueError                                Traceback (most recent call last)&#xA;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)&#xA;    509                 as_ref=input_arg.is_ref,&#xA;--&gt; 510                 preferred_dtype=default_dtype)&#xA;    511           except TypeError as err:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)&#xA;   1103     if ret is None:&#xA;-&gt; 1104       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)&#xA;   1105 &lt;/p&gt;&#xA;&#xA;&lt;p&gt;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _TensorTensorConversionFunction(t, dtype, name, as_ref)&#xA;    946         &quot;Tensor conversion requested dtype %s for Tensor with dtype %s: %r&quot; %&#xA;--&gt; 947         (dtype.name, t.dtype.name, str(t)))&#xA;    948   return t&lt;/p&gt;&#xA;&#xA;&lt;p&gt;ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor(&quot;Reshape_4:0&quot;, shape=(?, 1), dtype=float32)'&lt;/p&gt;&#xA;&#xA;&lt;p&gt;During handling of the above exception, another exception occurred:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TypeError                                 Traceback (most recent call last)&#xA; in ()&#xA;----&gt; 1 pred = dynamicRNN(x, seqlen, weights, biases)&lt;/p&gt;&#xA;&#xA;&lt;p&gt; in dynamicRNN(x, seqlen, weights, biases)&#xA;      2     x = tf.transpose(x, [1, 0, 2])&#xA;      3     x = tf.reshape(x, [-1, 1])&#xA;----&gt; 4     x = tf.split(0, seq_max_len, x)&#xA;      5     lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden)&#xA;      6     outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32, sequence_length=seqlen)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in split(value, num_or_size_splits, axis, num, name)&#xA;   1389   if size_splits._rank() == 0 and size_splits.dtype.is_integer:&#xA;   1390     return gen_array_ops.split(&#xA;-&gt; 1391         axis=axis, num_split=num_or_size_splits, value=value, name=name)&#xA;   1392 &#xA;   1393   if num is None:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py in split(axis, value, num_split, name)&#xA;   7707     num_split = _execute.make_int(num_split, &quot;num_split&quot;)&#xA;   7708     _, _, _op = _op_def_lib._apply_op_helper(&#xA;-&gt; 7709         &quot;Split&quot;, split_dim=axis, value=value, num_split=num_split, name=name)&#xA;   7710     _result = _op.outputs[:]&#xA;   7711     _inputs_flat = _op.inputs&lt;/p&gt;&#xA;&#xA;&lt;p&gt;~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)&#xA;    531             if input_arg.type != types_pb2.DT_INVALID:&#xA;    532               raise TypeError(&quot;%s expected type of %s.&quot; %&#xA;--&gt; 533                               (prefix, dtypes.as_dtype(input_arg.type).name))&#xA;    534             else:&#xA;    535               # Update the maps with the default, if needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;TypeError: Input 'split_dim' of 'Split' Op has type float32 that does not match expected type of int32.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;​&lt;/p&gt;&#xA;" OwnerUserId="52447" LastActivityDate="2018-05-21T07:59:39.183" Title="Dynamic rnn for toysequence classification" Tags="|tensorflow|" AnswerCount="0" CommentCount="0" ClosedDate="2020-10-26T06:04:43.337" ContentLicense="CC BY-SA 4.0" />
  <row Id="31906" PostTypeId="2" ParentId="31900" CreationDate="2018-05-21T08:17:08.490" Score="6" Body="&lt;p&gt;No. From this correlation matrix you cannot draw the conclusion that&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The reason is that correlation is a measure of association between &lt;em&gt;single&lt;/em&gt; pairs of variables. The conclusion you draw above - on the contrary - is based on a combination of three different variables plus the outcome variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to get an estimation of the probability that a student will be admitted to college based on her &lt;code&gt;gpa&lt;/code&gt;, &lt;code&gt;gre&lt;/code&gt; and &lt;code&gt;prestige&lt;/code&gt; the right way is to create a logistic regression model. Here's an example in R (provided that &lt;code&gt;admit&lt;/code&gt; is a binary variable, with &lt;code&gt;admit=1&lt;/code&gt; indicating that the student is admitted and &lt;code&gt;admit=0&lt;/code&gt; that she is not admitted)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- glm(admit ~.,family=binomial(link='logit'),data=data)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this fitted model you can then compute the probability that a student is admitted given her particular combination of &lt;code&gt;gpa&lt;/code&gt;, &lt;code&gt;gre&lt;/code&gt; and &lt;code&gt;prestige&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2018-05-28T08:02:58.183" LastActivityDate="2018-05-28T08:02:58.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31907" PostTypeId="2" ParentId="31904" CreationDate="2018-05-21T08:25:52.647" Score="5" Body="&lt;p&gt;What you are trying to do here is &lt;strong&gt;forecast&lt;/strong&gt; the future values of a time series. This is a predictive problem and the future values will depend on a number of latent factors. I will assume all we have access to is historical data from the series as your question indicates. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to predict a future value for the time series, you should not only use the current value as an input, but rather you should use a chunk of the historical data. Since you have 18,000,000 instances, this is a lot, you can make your network quite complex in order to capture some latent trends hidden inside your data which can help predict the future value. To predict a value at time $t$ we will use the $k$ previous values. This hyper-parameter needs to be effectively tuned. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Restructure the data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;We will structure the data such that the features $X$ are the $k$ previous time measurements, and the output target $Y$ is the current time measurement. The one that is being estimated by the model.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k = 3&#xA;X, Y = [], []&#xA;for i in range(len(col1) - k):&#xA;    X.append(col2[i:i+k])&#xA;    Y.append(col2[i+k])&#xA;&#xA;X = np.asarray(X)&#xA;Y = np.asarray(Y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Split your data&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import train_test_split&#xA;x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Using the data in a Keras model&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is a simple Keras model which should work as a first iteration step. However, due to the small amount of data you provided us I cannot get any meaningful results after training.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import keras&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten&#xA;from keras.layers import Conv1D, MaxPooling1D, Reshape&#xA;from keras.callbacks import ModelCheckpoint&#xA;from keras.models import model_from_json&#xA;from keras import backend as K&#xA;&#xA;x_train = x_train.reshape(len(x_train), k, )&#xA;x_test = x_test.reshape(len(x_test), k, )&#xA;&#xA;input_shape  = (k,)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(32, activation='tanh',&#xA;                 input_shape=input_shape))&#xA;model.add(Dense(32, activation='tanh'))&#xA;model.add(Dense(1, activation='linear'))&#xA;&#xA;model.compile(loss=keras.losses.mean_squared_error,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&#xA;model.summary()&#xA;&#xA;epochs = 10&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;history = model.fit(x_train, y_train,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-21T08:25:52.647" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="31908" PostTypeId="1" CreationDate="2018-05-21T08:38:11.950" Score="1" ViewCount="184" Body="&lt;p&gt;I am trying to read kernel ridge regression from this &lt;a href=&quot;https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But , I am unable to get the intution behind the derivation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone please help me ?&lt;/p&gt;&#xA;" OwnerUserId="44477" LastActivityDate="2019-08-25T15:02:35.417" Title="Unable to understand Kernel Ridge regression" Tags="|machine-learning|regression|kernel|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31909" PostTypeId="2" ParentId="31885" CreationDate="2018-05-21T08:46:25.820" Score="1" Body="&lt;p&gt;&lt;strong&gt;Some (possibly stupid) ideas for combining the two models:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Make an average of the two forecasts&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of a naive average you use LSTM for estimating the coarse daily trend and the RF for the fine details, maybe like so:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;a. Smooth (maybe with a loess or a kernel) the LSTM to get the daily trend&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b. For each day, subtract the mean volume call from the prediction of the RF, to get the fine details&lt;/p&gt;&#xA;&#xA;&lt;p&gt;c. Add the forecast of two models &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;General suggestions for improving your model:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Instead of training a model on the whole series of call volume, you might be better off feeding your model one day at a time, e.g. you cut the time series between 8 am and 5 pm every day. After all, you do not want your model to spend parameters on learning that no calls are made at night or during the weekend.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Add the day of the week as a parameter in the model, as well as some information on the call volume in the past (like previous day) if you think that there might be some time correlation between consecutive days.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If you think that the day of the week has a strong effect on the call volume (e.g. if each day has a special, recurring pattern) then you might consider creating a separate model for each day of the week. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2018-05-21T09:15:30.270" LastActivityDate="2018-05-21T09:15:30.270" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31910" PostTypeId="1" CreationDate="2018-05-21T09:16:15.100" Score="2" ViewCount="57" Body="&lt;p&gt;I am trying to build a news tagging system. Given a piece of news article, find 5-6 key terms from the news article that best describe the article. Refer to the image below from google news.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8a7o3.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8a7o3.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What are some approaches I can look at to get human readable tags ?&lt;/p&gt;&#xA;" OwnerUserId="13457" LastActivityDate="2021-06-08T22:08:55.860" Title="How to build News Tagging model(s)" Tags="|topic-model|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31911" PostTypeId="2" ParentId="31899" CreationDate="2018-05-21T09:18:56.273" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;from deep learning book ch5, SGD&lt;/em&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A recurring problem in machine learning is that large training sets are necessary for good generalization, but large training sets are also more computationally expensive.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Prior to the advent of deep learning, the main way to learn nonlinear models was to use the kernel trick in combination with a linear model. Many kernel learning algorithms require constructing an m×m matrix. Constructing this matrix has computational cost O(m^2), which is clearly undesirable for datasets with billions of examples.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient descent, in general, has often been regarded as slow or unreliable. In the past, the application of gradient descent to non-convex optimization problems was regarded as foolhardy or unprincipled, The optimization algorithm may not be guaranteed to arrive at even a local minimum in a reasonable amount of time, but it often finds a very low value of the cost function quickly enough to be useful&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient Descent-based methods, the basic premise is that you estimate the gradient based on a random small set of the training data (batch) and update the weights accordingly, the algorithm will iteratively keep alternating between the two steps until some objective has been satisfied (find a good local minima in the parameter space), the procedure is called stochastic gradient descent (The insight of SGD is that the gradient is an expectation. The expectation may be approximately estimated using a small set of samples).&lt;br&gt;&#xA;In the extreme case, the gradient is estimated based on one single example, (online learning)&lt;br&gt;&#xA;see &lt;a href=&quot;https://arxiv.org/pdf/1206.5533&quot; rel=&quot;nofollow noreferrer&quot;&gt;Practical Recommendations for Gradient-Based Training of Deep&#xA;Architectures&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;is there some tradeoff in runtime or accuracy of these models&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;For a fixed model size, the cost per SGD update does not depend on the&#xA;training set size m. In practice, we often use a larger model as the training set size&#xA;increases, but we are not forced to do so. The number of updates required to reach convergence usually increases with training set size. However, as m approaches infinity, the model will eventually converge to its best possible test error before SGD has sampled every example in the training set. Increasing m further will not extend the amount of training time needed to reach the model’s best possible test error. From this point of view, one can argue that the asymptotic cost of training a model with SGD is O(1) as a function of m.&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-21T09:18:56.273" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31912" PostTypeId="2" ParentId="31874" CreationDate="2018-05-21T09:29:54.973" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I am using a recurrent network for time series forecasting.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Interesting would be especially the type of the network you chose. How many layers? How many neurons per layer? &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The prediction from the last cell in the network always seems to be the most accurate.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I assume by cells you refer to your &lt;strong&gt;samples&lt;/strong&gt;. Always, if you learn in datastreams (online), the network in later iterations &lt;strong&gt;has seen more data&lt;/strong&gt; than the network in early iterations. Thus, with having seen more samples, your predictions become more accurate. Of course, this assumes that you do online learning on your datastream. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This depends on how your RNN is set up. A &lt;a href=&quot;https://en.wikipedia.org/wiki/Recurrent_neural_network#Elman_networks_and_Jordan_networks&quot; rel=&quot;nofollow noreferrer&quot;&gt;Jordan Network&lt;/a&gt; for example has context neurons, which should do exactly what you propose. The reason for recurrency in the network is typically that you want to simulate short term memory - meaning, in a mathematical view, you want to feed information from previous iterations into the front of the network. Thus, since this feature should be taken care of in your RNN, I do not think it is good to extend your input, since you introduce additional, potentially useless weights &lt;strong&gt;that have to be learned&lt;/strong&gt; by the network. &lt;/p&gt;&#xA;" OwnerUserId="52257" LastActivityDate="2018-05-21T09:29:54.973" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31913" PostTypeId="2" ParentId="8120" CreationDate="2018-05-21T09:38:25.917" Score="2" Body="&lt;p&gt;It should be noted that XGBoost makes optimal splits assuming it has access to the entire dataset, so you would likely be losing out on some predictive power by updating rather than retraining (though of course this may be worth the lessened computational cost).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;See an interesting discussion on this &lt;a href=&quot;https://github.com/dmlc/xgboost/issues/3055&quot; rel=&quot;nofollow noreferrer&quot;&gt;XGBoost github issue&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="39286" LastActivityDate="2018-05-21T09:38:25.917" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31914" PostTypeId="1" CreationDate="2018-05-21T09:42:52.953" Score="4" ViewCount="2211" Body="&lt;p&gt;While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the &lt;code&gt;dual&lt;/code&gt; parameter, which is described in the documentation as:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;dual : bool, (default=True)&#xA;&#xA;    Select the algorithm to either solve the dual or primal optimization problem.&#xA;    Prefer dual=False when n_samples &amp;gt; n_features.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone explain it, preferably in ELI5 style?&lt;/p&gt;&#xA;" OwnerUserId="82" LastEditorUserId="14519" LastEditDate="2018-11-27T18:22:57.707" LastActivityDate="2018-12-29T00:03:55.880" Title="What does the &quot;dual&quot; parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?" Tags="|python|scikit-learn|svm|hyperparameter|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31915" PostTypeId="2" ParentId="21911" CreationDate="2018-05-21T10:09:30.297" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;The hidden-layer has as many neurons as there are training samples.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This already sounds a bit fishy to me. The hidden layer of an RBF network resembles the Radial Basis functions. Thus, you have to think about &lt;strong&gt;how many centers&lt;/strong&gt; shall your network have. This should not correspond to the number of training samples your have, since this would force the network to learn by heart. Let's look at the example you gave in the question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/W9fBe.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/W9fBe.jpg&quot; alt=&quot;Centers of an RBF network&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here I propose to use a network with &lt;strong&gt;four hidden neurons&lt;/strong&gt;, corresponding to &lt;strong&gt;four RBF centers&lt;/strong&gt;, marked with stars. This is, in the end, how your network should learn to generalize over the data you feed into it. Note that you do not have to set the class of an RBF network in prior, the network is supposed to learn the classes of each neuron by itself. How this is done is &lt;a href=&quot;https://en.wikipedia.org/wiki/Radial_basis_function_network#Function_approximation&quot; rel=&quot;nofollow noreferrer&quot;&gt;properly explained on the wikipedia page&lt;/a&gt;. A nice thing about RBF networks is that there is an analytical solution (over the Moore-Penrose pseudoinverse) for your weight vector.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $σ_i$?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I suppose you are talking about the centers $\mu_i$ and their width $\sigma_i$. These values, like described in the link above, can be set via a clustering over the data you have. I've also seen applications that adapt the centers online, but this is usually not necessary. A simple unsupervised learning / clustering approach is typically sufficient if you have all training data available. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If your designed network is able to solve your task, the analytical solution will do the job. &lt;a href=&quot;https://en.wikipedia.org/wiki/Radial_basis_function_network#Pseudoinverse_solution_for_the_linear_weights&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia describes how it is calculated.&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52257" LastActivityDate="2018-05-21T10:09:30.297" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31916" PostTypeId="1" AcceptedAnswerId="31917" CreationDate="2018-05-21T10:13:25.087" Score="6" ViewCount="89186" Body="&lt;p&gt;I am trying to clean the data. But I don't know how to remove a function from a column in data frame.&lt;a href=&quot;https://i.stack.imgur.com/iNZq0.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iNZq0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At row number 473 it show column N has a function . How it should be filtered out ?&lt;/p&gt;&#xA;" OwnerUserId="32771" LastActivityDate="2020-07-19T21:04:43.387" Title="TypeError: float() argument must be a string or a number, not 'function'" Tags="|machine-learning|python|predictive-modeling|pandas|data-cleaning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31917" PostTypeId="2" ParentId="31916" CreationDate="2018-05-21T10:17:47.317" Score="6" Body="&lt;p&gt;a generic way to convert pandas column to numeric dtype:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df['col_name'] = pd.to_numeric(df['col_name'], errors='coerce')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;this will replace all values, that couldn't be converted to numeric values with NaN's (Not a Number).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS you may want to analyze how and why do you have a reference to a lambda function in your cells. Fixing the source of that problem would be a more appropriate solution.&lt;/p&gt;&#xA;" OwnerUserId="20492" LastActivityDate="2018-05-21T10:17:47.317" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31918" PostTypeId="2" ParentId="31834" CreationDate="2018-05-21T10:19:06.253" Score="0" Body="&lt;p&gt;I suggest using a model with more relaxed assumptions on proportionality of hazards. In my work I use &lt;a href=&quot;ftp://cran.r-project.org/pub/R/web/packages/pch/pch.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;piecewise constant hazard model&lt;/a&gt;, which works wonderfully. Its assumption is that the hazards are proportional in a time interval. It allows using numerical covariates with splines, and time-dependent covariates. Moreover in my experience the model is usually very well calibrated and does not overfit much.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastActivityDate="2018-05-21T10:19:06.253" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31921" PostTypeId="2" ParentId="29317" CreationDate="2018-05-21T11:25:59.253" Score="2" Body="&lt;p&gt;If you read the package &lt;a href=&quot;https://cran.r-project.org/web/packages/cluster/news.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;News section&lt;/a&gt; you will find:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;pam() now signals an error when there are more than 65536 observational units (whereas it could segfault previously), thanks to a patch from Mikko Korpela, Helsinki.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So it seems there isn't a solution. &lt;/p&gt;&#xA;" OwnerUserId="52458" LastEditorUserId="29575" LastEditDate="2018-05-21T12:25:43.720" LastActivityDate="2018-05-21T12:25:43.720" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31922" PostTypeId="1" CreationDate="2018-05-21T12:02:44.720" Score="1" ViewCount="36" Body="&lt;p&gt;Several traffic networks offer &quot;smart links&quot;, where you can send online traffic, and the traffic will be sent to the most profitable segment. These networks include Monetizer, Glize etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I.e. if you send IOS, mobile, Orange carrier in Spain, the traffic will primarily go to offer Y. If you send Android, Tablet in France it will go to offer X etc. I assume they optimize for earning per view/click. A fraction will be for testing, but the majority of the traffic will go to the optimal offer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wonder what the general workings of such algorithms are, and if there's a place (research papers, or books), I can learn more about this type of problem? I have a hard time finding anything directly related online, as it's not a very common problem in research papers, and a lot of the algorithms are being disclosed -- all though some mention using Bayesian statistics to optimize.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I started looking at multi-armed bandit problems, which I feel are very close to this type of problem. If you imagine the different offers are different &quot;bandits&quot;, which bandit should you choose to maximize return. From this they can calculate a percentage weight per offer. Maybe initially they will all start at an even weight, and once there's enough data in the timespan, the weights will be shuffled (at a timed basis, i.e. every 30 min). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think this is a very interesting problem, as I work in online marketing and also study computer science.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="52460" LastActivityDate="2018-05-21T12:02:44.720" Title="Algorithms behind &quot;smartlinks&quot; for online traffic distribution" Tags="|optimization|bayesian|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31923" PostTypeId="1" CreationDate="2018-05-21T12:28:51.973" Score="4" ViewCount="5269" Body="&lt;p&gt;I'm experimenting (read: playing around) with LSTMs on Keras. &#xA;I want to train an LSTM network so it would &quot;track&quot; sine waves, that is, given sine waves with different wave length, phases and lengths, it would output the rest of wave. &#xA;In a sense, a many-to-many problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I decided that the network would need to &quot;track&quot; 5 sine waves simultaneously. &#xA;I would generate random sine waves and use the 50 last data points as the required output of the NN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My code is a follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from keras.models import Sequential&#xA;from keras.layers import LSTM, Dense, TimeDistributed,Lambda, Dropout&#xA;import numpy as np&#xA;&#xA;model = Sequential()&#xA;&#xA;model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(5, return_sequences=True))&#xA;model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values&#xA;&#xA;&#xA;&#xA;print(model.summary())&#xA;&#xA;model.compile(loss='mean_squared_error',&#xA;              optimizer='adam')&#xA;&#xA;&#xA;def make_line(length):&#xA;    shift= np.random.random()&#xA;    wavelength = 5+10*np.random.random()&#xA;    a=np.arange(length)&#xA;    answer=np.sin(a/wavelength+shift)&#xA;    return answer&#xA;&#xA;&#xA;&#xA;&#xA;def make_data(seq_num,seq_len,dim):&#xA;    data=np.array([]).reshape(0,seq_len,dim)&#xA;    for i in range (seq_num):&#xA;        mini_data=np.array([]).reshape(0,seq_len)&#xA;        for j in range (dim):&#xA;            line = make_line(seq_len)&#xA;            line=line.reshape(1,seq_len)            &#xA;            mini_data=np.append(mini_data,line,axis=0)&#xA;        mini_data=np.swapaxes(mini_data,1,0)&#xA;        mini_data=mini_data.reshape(1,seq_len,dim)      &#xA;        data=np.append(data,mini_data,axis=0)&#xA;    return (data)&#xA;&#xA;&#xA;&#xA;def train_generator():&#xA;    while True:&#xA;        sequence_length = np.random.randint(50, 150)+50     &#xA;        data=make_data(1000,sequence_length,5)&#xA;        x_train = data[:,:-50,:] # all but last 50      &#xA;        y_train = (data[:, -50:, :]) # last 50      &#xA;        yield x_train, y_train&#xA;&#xA;&#xA;model.fit_generator(train_generator(), steps_per_epoch=30, epochs=100, verbose=1,validation_data=train_generator(),validation_steps=30)&#xA;model.save('vi2h_model.h5')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I then try to generate data and use the model to predict the continuation of a sine wave:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def data_generator():&#xA;    while True:&#xA;        sequence_length = np.random.randint(50, 150)+50&#xA;        data=make_data(1000,sequence_length,5)&#xA;        x_train = data[:,:-50,:]    &#xA;        y_train = (data[:, -50:, :])                &#xA;        return x_train, y_train&#xA;&#xA;&#xA;x,y= data_generator()   &#xA;&#xA;model = load_model('vi2h_model.h5')&#xA;pred=model.predict (x)&#xA;np.save ('pred.npy',pred)&#xA;np.save ('y.npy',y)&#xA;np.save ('x.npy',x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When inspecting the ability of the network to predict the continuation of a sine wave (comparing y[0,:,0] to pred[0,:,0] for example, in order to compare the first sine wave of the first generated datum), see that my network performed very badly:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/i4qmK.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/i4qmK.jpg&quot; alt=&quot;The predicted vs actual 50 last points in a sample sine wave, placed after the part of the wave used as input for the NN&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;(Edit: legend: The predicted vs actual 50 last points in a sample sine wave (blue and orange, respectively), placed after the part of the wave used as input for the NN)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What did I do wrong? And how should I my code to correctly track sine waves?&#xA;Many thanks!&lt;/p&gt;&#xA;" OwnerUserId="43472" LastEditorUserId="43472" LastEditDate="2018-05-22T07:30:16.800" LastActivityDate="2018-07-22T21:10:55.160" Title="Training an LSTM to track sine waves" Tags="|python|deep-learning|keras|lstm|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31924" PostTypeId="2" ParentId="31908" CreationDate="2018-05-21T12:32:02.323" Score="1" Body="&lt;p&gt;Equation (4) simply gives us an identity (without proof):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's check that it does indeed hold. The left hand side equals&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{align*}&#xA;(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &amp;amp;= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\&#xA;&amp;amp;= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\&#xA;&amp;amp;= \left(RB^{-T}P^{-1} + B\right)^{-1}.&#xA;\end{align*}&#xA;The right hand side equals&#xA;\begin{align*}&#xA;PB^T (BPB^T + R)^{-1} &amp;amp;= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\&#xA;&amp;amp;= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\&#xA;&amp;amp;= \left(B + RB^{-T}P^{-1} \right)^{-1}.&#xA;\end{align*}&#xA;So indeed, the equality holds.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you take the right hand side of equation (3)&#xA;$$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$&#xA;replace $\mathbf{x}_i$ with $\phi_i$  and rewrite it as&#xA;$$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$&#xA;then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where&#xA;$$P^{-1} = \lambda^{-1}\mathbf{I}$$&#xA;$$B^T = \Phi$$&#xA;$$R = \mathbf{I}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, we can rewrite the right hand side of equation (4) as&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as&#xA;$$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$&lt;/p&gt;&#xA;" OwnerUserId="52109" LastEditorUserId="52109" LastEditDate="2018-05-22T08:29:45.683" LastActivityDate="2018-05-22T08:29:45.683" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31925" PostTypeId="1" CreationDate="2018-05-21T12:43:56.443" Score="1" ViewCount="108" Body="&lt;p&gt;A complete newbie to data science. I have to predict the word using characters given. But the characters given can be in any order, and some might be missing or wrong. The problem is I get too many suggestions that arent right.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Trial 1:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I have tried permutating the given characters and checking if it is a valid english word : &lt;a href=&quot;https://pastebin.com/ZUZDZPPa&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pastebin.com/ZUZDZPPa&lt;/a&gt;&lt;br&gt;&#xA;This does not take into account autocorrect like features.&#xA;With autocorrect: &lt;a href=&quot;https://pastebin.com/mDjVn0Rz&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pastebin.com/mDjVn0Rz&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Trial 2:&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Tried limiting editdistance (levenshtein distance) to 2 and used my custom corpus that I built as I knew the context of the word (Occupations or hobbies).&lt;br&gt;&lt;a href=&quot;https://pastebin.com/iC5TYxzS&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://pastebin.com/iC5TYxzS&lt;/a&gt; &lt;br&gt;&lt;br&gt;The problem is that the predictions I get arent close to the answer. I got pretty good results in Trial 2, but I am wondering if there was a better approach as edit distance of 2 is pretty limiting.&lt;/p&gt;&#xA;" OwnerUserId="52462" LastActivityDate="2018-05-21T12:43:56.443" Title="Predict words in a given corpus from jumbled incomplete characters" Tags="|python|nlp|nltk|" AnswerCount="0" CommentCount="3" ClosedDate="2018-05-27T07:39:46.217" ContentLicense="CC BY-SA 4.0" />
  <row Id="31926" PostTypeId="1" CreationDate="2018-05-21T12:49:56.853" Score="2" ViewCount="1006" Body="&lt;p&gt;I have a dataset that has a large number of categorical features and a few numerical features, and I want to predict the probability that any given input is one of two types of a certain binary output feature. I don't want to solve a classifier problem for reasons that are well outlined &lt;a href=&quot;http://www.fharrell.com/post/class-damage/&quot; rel=&quot;nofollow noreferrer&quot;&gt;in this link&lt;/a&gt; .&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have seen many tutorials about how to handle them independently but am less sure how to handle them together.&lt;/p&gt;&#xA;" OwnerUserId="34158" LastActivityDate="2020-04-14T09:01:10.290" Title="Handling a combined dataset of numerical and categorical features for Regression" Tags="|python|regression|pandas|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31927" PostTypeId="2" ParentId="29408" CreationDate="2018-05-21T13:37:03.040" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I would like to know which metrics can be useful to do such things&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Since you are clustering your data, why not use the same metric that you used during clustering for comparing your data inside your clusters? I assume this should technically give you exactly what you need. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;there is no threshold to tell whether individuals in a cluster share similar pattern or not&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;This hardly depends on your data and what you want to do with your similarity. If you just want to retrieve the most similar point, you can do it pretty straight forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you, on the other hand, want to remove outliers using this measure, you can surely also do so, but you might as well just use an algorithm that deals with outliers in the first place.&lt;/p&gt;&#xA;" OwnerUserId="52257" LastActivityDate="2018-05-21T13:37:03.040" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31928" PostTypeId="2" ParentId="31926" CreationDate="2018-05-21T13:45:25.457" Score="1" Body="&lt;p&gt;One could approach this in two general ways:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) &lt;strong&gt;bottom up:&lt;/strong&gt; thinking about unifying the data somehow to begin with&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) &lt;strong&gt;top down:&lt;/strong&gt; deciding how the data needs to look based on the final model you wish to use&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Do you already know which model you will use? If that is fixed (for whatever reason), you already know you need to get your data into the correct form, be it numerical or categorical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you pinned your question with the tag &lt;code&gt;regression&lt;/code&gt;, I can tell you that you need to make your data all numerical, so regression can work.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;An example of making numerical data categorical would be to put it into bins. Imagine we have values ranging from zero to ten: [0.173, 7.88, 3.91, ...]. You could simply say that values between 0.00 and 0.99 are category A, values between 1.00 and 1.99 are category B, and so on.&lt;/p&gt;&#xA;&#xA;&lt;h3&gt;[Edit:]&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;A slightly more sophisticated way of defining the bins to use would be to define the bins based on some characteristic statistics of your dataset. For example, have a look at the possible ways possible implemented within python's &lt;a href=&quot;https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;Numpy&lt;/code&gt;&lt;/a&gt;. Of the available methods there, I have found the &lt;strong&gt;Doane&lt;/strong&gt; method to work best - it will depend on your data though, so read the descriptions.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Making categorical values numerical in a meaningful way depends a little more on you data. It is easy to make them numberic, but you should focus on doing it in such a way as to retain as much of the information each variable contains &lt;strong&gt;as well as&lt;/strong&gt; the relative relationships between each of the categories that you started with. E.g. converting colours into integers would allow you to perform regression, but if yellow becomes 1 and purple 10, the model needs to be able to learn that purple isn't necessarily 10 times bigger than yellow, and that is difficult in the context of regression!&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-21T14:59:50.923" LastActivityDate="2018-05-21T14:59:50.923" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31929" PostTypeId="2" ParentId="31926" CreationDate="2018-05-21T13:51:05.673" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I don't want to solve a classifier problem for reasons that are well outlined in this link.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I doubt that the link wants to tell you to stop performing classification tasks - the problem you propose is a classical example for classification. How I understand your source, it does not want you to use scoring rules as a heuristic.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the problem you described I would propose a simple &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes approach&lt;/a&gt;. To make your numerical values discrete, you can simply use the mean of two adjacent numeric values as a treshold. E.g. for a list [1, 2] of numeric values, just split them at a threshold of 1.5 and check above and below. &lt;/p&gt;&#xA;" OwnerUserId="52257" LastActivityDate="2018-05-21T13:51:05.673" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31930" PostTypeId="2" ParentId="30362" CreationDate="2018-05-21T14:01:53.117" Score="1" Body="&lt;p&gt;Stationarity is an important factor in determining the model structure for the ARIMA family of models. Some of the reasons for this are discussed in relative depth in &lt;a href=&quot;https://stats.stackexchange.com/questions/19715/why-does-a-time-series-have-to-be-stationary&quot;&gt;this question&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Briefly, the reason that you have to consider the stationarity of the series explicitly when developing a model in the ARIMA family is because the model has to have descriptive power over changing conditions, which you have to make choices to enable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, an LSTM is not constrained on this dimension- i.e. a sufficiently trained LSTM with a sufficient architectural descriptive base can determine the changing nature of the time series without the modeler making explicit choices based on that feature of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You WILL want to understand stationarity in some form regardless of this feature, as it will help inform the useful life period of your model as well as lifecycle concerns - i.e. how often it will need to be retrained, what monitoring is necessary in production, etc.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2018-05-21T14:01:53.117" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31931" PostTypeId="1" AcceptedAnswerId="32133" CreationDate="2018-05-21T14:04:43.087" Score="4" ViewCount="597" Body="&lt;p&gt;Usual decision trees are directed acyclic graphs. Are there generalizations of decision trees that contain cycles analogously to recurrent neural networks? If such trees exist, can they be applied to sequences of variable length? I could not find information about such trees. Or their existence is impossible of infeasible?&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-24T22:22:57.370" Title="Are there cyclic decision trees?" Tags="|decision-trees|sequence|rnn|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31932" PostTypeId="1" CreationDate="2018-05-21T14:34:53.860" Score="2" ViewCount="861" Body="&lt;p&gt;I am not a data scientist but am trying to implement a recommender system for my company. My application runs on PHP but I will use Python to process the data.&lt;/p&gt;&#xA;&lt;p&gt;My company is an online school, with 40 online courses as of now. I have a CSV file with around 30k users preferences and it looks like this:&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/fXx4G.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/fXx4G.png&quot; alt=&quot;Dataframe&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;0 means that user is not subscribed (I consider here that they have no interest), while 1 means subscribed (interested).&lt;/p&gt;&#xA;&lt;p&gt;My idea is to compare one single user array such as [0,1,0,0,0,1,1...] with all this data and return a grade for each course with the probability of interest for this user.&lt;/p&gt;&#xA;&lt;p&gt;I was thinking of using a Multinomial Logistic Regression, but as far as I know (and I do not know much) it would return me a binary result, right?&lt;/p&gt;&#xA;&lt;p&gt;What classification model would you recommend me to use? Ideally, my result should be something like:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;[0.95, 0.1, 0.54, 0.3, 0.87...]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Cheers!&lt;/p&gt;&#xA;" OwnerUserId="52472" LastEditorUserId="29169" LastEditDate="2021-08-12T00:25:52.717" LastActivityDate="2022-11-16T15:00:15.037" Title="What's the best classification model for this recommendation engine?" Tags="|python|recommender-system|multiclass-classification|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31933" PostTypeId="1" CreationDate="2018-05-21T14:39:23.670" Score="1" ViewCount="197" Body="&lt;p&gt;While studying Gaussian mixture models and the expectation maximization algorithm, I also came across a number of studies that used 'Discriminative training' for the models that addressed some limitations of the EM algorithm. In particular, the use of maximum mutual information is quite common.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the basic idea behind maximizing mutual information; however, I am unable to find any source that gives the equations involved in estimating the means, variances, mixing parameters and the predicted labels. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you please state them along with an explanation?&lt;/p&gt;&#xA;" OwnerUserId="23850" LastActivityDate="2018-06-22T12:21:19.640" Title="Understanding GMM-MMI" Tags="|machine-learning|clustering|gaussian|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31934" PostTypeId="2" ParentId="31932" CreationDate="2018-05-21T14:47:26.580" Score="0" Body="&lt;p&gt;Without more information about your dataset, it's impossible to recommend one particular classifier over another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want your classifier to return a vector of probabilities, then if you're using the sklearn library, you could use the &lt;code&gt;predict_proba&lt;/code&gt; method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's an example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.datasets import load_digits&#xA;digits = load_digits(2)&#xA;from sklearn.linear_model import LogisticRegression&#xA;preds = LogisticRegression().fit(digits.data, digits.target).predict_proba(digits.data)&#xA;print([i[1] for i in preds]) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52109" LastActivityDate="2018-05-21T14:47:26.580" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31935" PostTypeId="2" ParentId="31931" CreationDate="2018-05-21T14:50:26.413" Score="2" Body="&lt;p&gt;The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not help accomplish the goal, it would unnecessarily repeat the modeling fitting for a given region. &lt;/p&gt;&#xA;" OwnerUserId="1330" LastActivityDate="2018-05-21T14:50:26.413" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31936" PostTypeId="1" CreationDate="2018-05-21T15:08:38.920" Score="1" ViewCount="51" Body="&lt;p&gt;Below is my r code which I used on this &lt;a href=&quot;https://www.kaggle.com/smeschke/four-shapes/discussion/new&quot; rel=&quot;nofollow noreferrer&quot;&gt;kaggle dataset&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure why the accuracy is so low, but I suspect that I encoded the pixel matrix incorrectly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I transformed each pixel matrix picture into itself as a row of 200*200 columns and then did that for each pixel matrix making the individual rows for the 14970 pictures in the dataset. I'm not sure if the neural network understands the data and since it is taking data batch after batch I wonder how it can make sense of it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A quick explanation and some suggested improvements to my code would be appreciated. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(png) &#xA;library(colorspace) &#xA;library(abind) &#xA;library(reticulate) &#xA;library(keras) &#xA;&#xA;setwd(&quot;~/E-books/Coding/Machine Learning with R/Neural Networks/four-shapes&quot;) &#xA;&#xA;# Here I create a list of links to retrieve each image &#xA;files &amp;lt;- list.files(pattern = &quot;.png$&quot;, recursive = TRUE) &#xA;filePathFunction &amp;lt;- function(x){ &#xA;a &amp;lt;- matrix(nrow=length(x)) &#xA;for(i in 1:length(x)){ &#xA;conc &amp;lt;- c(&quot;~/E-books/Coding/Machine Learning with R/Neural Networks/four-shapes&quot;,x[i]) &#xA;a[i] &amp;lt;- paste(conc, collapse=&quot;/&quot;) &#xA;} &#xA;return(a) &#xA;} &#xA;filePaths &amp;lt;- filePathFunction(files) &#xA;image(readPNG(filePaths[1]), useRaster=TRUE, axes=FALSE) &#xA;&#xA;&#xA;#circle is 0 &#xA;#square is 1 &#xA;#star is 2 &#xA;#triangle is 3 &#xA;&#xA;circle &amp;lt;- rep(0,3720) &#xA;circle_array &amp;lt;- array(circle, dim=c(3720,1)) &#xA;square &amp;lt;- rep(1,3765) &#xA;square_array &amp;lt;- array(square, dim=c(3765,1)) &#xA;star &amp;lt;- rep(2,3765) &#xA;star_array &amp;lt;- array(star, dim=c(3765,1)) &#xA;triangle &amp;lt;- rep(3,3720) &#xA;triangle_array &amp;lt;- array(triangle, dim=c(3720,1)) &#xA;labels &amp;lt;- rbind(circle_array,square_array,star_array, triangle_array) &#xA;&#xA;&#xA;&#xA;set.seed(101) &#xA;ind &amp;lt;- sample(nrow(filePaths),round(0.75*nrow(filePaths)),replace=F) &#xA;train_data &amp;lt;- filePaths[ind,] &#xA;test_data &amp;lt;- filePaths[-ind,] &#xA;&#xA;train_labels &amp;lt;- labels[ind,] &#xA;test_labels &amp;lt;- labels[-ind,] &#xA;&#xA;train_labels &amp;lt;- to_categorical(train_labels) &#xA;test_labels &amp;lt;- to_categorical(test_labels) &#xA;&#xA;#trainPathArray and testPathArray reshape each pixel matrix into a 11228x200*200 array and &#xA;#3742x200*200 array matrix &#xA;&#xA;trainPathArray &amp;lt;- matrix(ncol=200*200, nrow= length(train_data)) &#xA;for(i in 1:length(train_data)){ &#xA;trainPathArray[i,] &amp;lt;- array_reshape(readPNG(train_data[i]), dim= c(1,200*200)) &#xA;} &#xA;testPathArray &amp;lt;- matrix(ncol=200*200, nrow= length(test_data)) &#xA;for(i in 1:length(test_data)){ &#xA;testPathArray[i,] &amp;lt;- array_reshape(readPNG(test_data[i]), dim= c(1,200*200)) &#xA;} &#xA;&#xA;&#xA;network &amp;lt;- keras_model_sequential() %&amp;gt;% &#xA;layer_dense(units = 1000, activation = &quot;relu&quot;, input_shape = c(200*200)) %&amp;gt;% &#xA;layer_dense(units = 1000, activation = &quot;relu&quot;) %&amp;gt;% &#xA;layer_dense(units = 4, activation = &quot;softmax&quot;) &#xA;&#xA;network %&amp;gt;% compile( &#xA;optimizer = &quot;rmsprop&quot;, &#xA;loss = &quot;categorical_crossentropy&quot;, &#xA;metrics = c(&quot;accuracy&quot;) &#xA;) &#xA;network %&amp;gt;% fit(trainPathArray, train_labels, epochs = 5, batch_size = 12smilie &#xA;metrics &amp;lt;- network %&amp;gt;% evaluate(testPathArray, test_labels)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="30940" LastEditorUserId="29575" LastEditDate="2019-07-09T13:00:14.563" LastActivityDate="2019-07-09T13:00:14.563" Title="Why do I get such a low on image classification task with neural networks?" Tags="|neural-network|keras|r|tensorflow|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31937" PostTypeId="1" CreationDate="2018-05-21T15:10:43.303" Score="1" ViewCount="2625" Body="&lt;p&gt;I am looking at how to extract a set of skills from an unstructured resume format using Python. Please note that, though I have a basic idea of NLP, I am completely new to Python.&lt;/p&gt;&#xA;" OwnerUserId="52473" LastActivityDate="2018-05-21T15:10:43.303" Title="How can I extract skills from a resume using python?" Tags="|python|nlp|text-mining|" AnswerCount="0" CommentCount="4" ClosedDate="2018-05-24T15:27:21.387" ContentLicense="CC BY-SA 4.0" />
  <row Id="31938" PostTypeId="1" AcceptedAnswerId="31939" CreationDate="2018-05-21T15:41:36.230" Score="1" ViewCount="11466" Body="&lt;p&gt;I'm trying to set &lt;code&gt;stratify = True&lt;/code&gt;, but somehow jupyter notebook says 'name 'y' is not defined'&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(&#xA;    loan.drop('Loan_Status', axis=1), &#xA;    loan['Loan_Status'], test_size=0.2, random_state=0, stratify=y)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can anyone tell me what is the proper way to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm using Scikit-learn v0.19.1 and have tried to set &lt;code&gt;stratify = True / y / 2&lt;/code&gt; but none of them worked.&lt;/p&gt;&#xA;" OwnerUserId="52475" LastEditorUserId="29575" LastEditDate="2018-05-22T01:40:42.383" LastActivityDate="2019-11-14T04:21:00.730" Title="train_test_split : stratify can not be recognized?" Tags="|machine-learning|python|scikit-learn|" AnswerCount="2" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31939" PostTypeId="2" ParentId="31938" CreationDate="2018-05-21T15:48:35.937" Score="4" Body="&lt;p&gt;You need to pass an &lt;strong&gt;array&lt;/strong&gt; containing the class-labels (or whatever the criterion for stratifying is) as an argument to &lt;code&gt;stratify&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, the answer is probably &lt;code&gt;loan['Loan_Status'].values&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="23305" LastActivityDate="2018-05-21T15:48:35.937" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31940" PostTypeId="1" CreationDate="2018-05-21T15:55:37.387" Score="10" ViewCount="18934" Body="&lt;p&gt;I have to draw a CNN diagram similar to this:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/4yZDR.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4yZDR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried all the tools mentioned in &lt;a href=&quot;https://datascience.stackexchange.com/a/14900&quot;&gt;https://datascience.stackexchange.com/a/14900&lt;/a&gt;, but there is no easy way to do it.&#xA;Is there any automated way to do it? Or do I have to do it manually.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In addition, is it possible to draw this?:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/X2yUL.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/X2yUL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found &lt;a href=&quot;http://alexlenail.me/NN-SVG/LeNet.html&quot; rel=&quot;noreferrer&quot;&gt;http://alexlenail.me/NN-SVG/LeNet.html&lt;/a&gt;, however, in this the input to each layer is assumed to be a perfect square. Thus, I cannot draw above diagram.&lt;/p&gt;&#xA;" OwnerUserId="52476" LastEditorUserId="52476" LastEditDate="2018-05-22T12:22:39.830" LastActivityDate="2018-08-26T07:05:40.673" Title="How to draw convolutional neural network diagrams?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31941" PostTypeId="2" ParentId="31926" CreationDate="2018-05-21T16:06:07.367" Score="1" Body="&lt;p&gt;Adding to the answers of above, there's one more better way to do the same i.e Target Encoding, which naively means you are e encoding your cats according to the target variable via using some aggregate (works out of the box)&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-05-21T16:06:07.367" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31942" PostTypeId="1" CreationDate="2018-05-21T16:08:39.950" Score="1" ViewCount="24" Body="&lt;p&gt;I'm using Keras for various machine learning projects and save my models after every training run. Let's say I want to do a training session for an algorithm - how do you decide whether to (1) load a saved algorithm and train it or (2) start training the algorithm from scratch? Some potential approaches I thought of:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Only load a saved model when the new, labeled, examples differ significantly from what the saved model was trained on&lt;/li&gt;&#xA;&lt;li&gt;Always train a new model since data is data&lt;/li&gt;&#xA;&lt;li&gt;Always train a saved model since even small sampling variations can have an effect on the algorithm&lt;/li&gt;&#xA;&lt;li&gt;Always train a saved model as soon as the input &amp;amp; output shapes are determined; you should only be training a model from scratch once.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="23240" LastActivityDate="2018-05-21T16:08:39.950" Title="Deciding whether to train a saved model or start a new one in Keras?" Tags="|deep-learning|keras|methodology|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31943" PostTypeId="1" CreationDate="2018-05-21T16:30:33.123" Score="2" ViewCount="228" Body="&lt;p&gt;I have 60000 images divided into two classes. I have tried to build transfer learning with pretrained ResNet50 but my new GTX 1080 ti returns -1 after couples of epochs. My guess is that it runs out of memory. Which is weird because with the same script and the same data I could successfully ran it on GTX 1050 ti. Everything is in tensorflow-gpu backend&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have installed CUDA 9.0 with cudnn 7.1.4 on ubuntu for GTX 1080 ti.&#xA;Here is nvidia-smi&#xA;&lt;a href=&quot;https://i.stack.imgur.com/6ovGF.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/6ovGF.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And I see lack of ram:&lt;a href=&quot;https://i.stack.imgur.com/u8aF3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/u8aF3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="50946" LastEditorUserId="50946" LastEditDate="2018-05-22T08:11:50.277" LastActivityDate="2018-05-22T08:11:50.277" Title="GTX 1080t ti rans out of memory" Tags="|deep-learning|classification|computer-vision|transfer-learning|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31944" PostTypeId="2" ParentId="11970" CreationDate="2018-05-21T16:55:04.417" Score="0" Body="&lt;p&gt;A very fine choice, plotly is...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my case, i was trying to plot similar designation based on skills, where skills was a word2vec embedding of 300 dimensions; brought it to a 3 dimension vector space, and using plotly Scatter3D, i was able to plot a 3D scatterplot for the same. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Et Viola!! Got an awesome 3 dimension graph, with hover and enlarge functionality. And best part is it can be exported as an html file, making it a plug and play suitable for any other PC, just drag and drop in a browser (included in the code below). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anything BEE anymore simpler&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from plotly.offline import plot&#xA;from plotly.graph_objs import *&#xA;import numpy as np&#xA;&#xA;# x = np.random.randn(2000)&#xA;# y = np.random.randn(2000)&#xA;&#xA;# Instead of simply calling plot(...), store your plot as a variable and pass it to displayHTML().&#xA;# Make sure to specify output_type='div' as a keyword argument.&#xA;# (Note that if you call displayHTML() multiple times in the same cell, only the last will take effect.)&#xA;&#xA;p = plot(&#xA;  [&#xA;    Scatter3d(x=skills_df[0], y=skills_df[1], z=skills_df[2], text= skills_df['designation'], mode='markers', marker=Marker(color=skills_df['cluster_number'], size=3, opacity=0.5, colorscale='Viridis'))&#xA;  ],&#xA;  output_type='div'&#xA;#   filename='/dbfs/FileStore/tables/lnkdn_jobroles_viridis.html' turn it on to save the file&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24818" LastActivityDate="2018-05-21T16:55:04.417" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31945" PostTypeId="2" ParentId="31931" CreationDate="2018-05-21T17:39:03.930" Score="2" Body="&lt;p&gt;That's an interesting proposition ... What would you hope to gain from adding a cyclic component? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Various quick answers of my own to this question have lead to existing methods like boosting (ex: xgboost) and ensembling (ex: random forest) which help address bias, variance, smoothing, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you could use a cyclical component to create a reinforcement-esque framework?&lt;/p&gt;&#xA;" OwnerUserId="50706" LastActivityDate="2018-05-21T17:39:03.930" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31946" PostTypeId="1" CreationDate="2018-05-21T17:46:24.897" Score="2" ViewCount="222" Body="&lt;p&gt;I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I get anywhere between 10 and 20 clusters, but when I ran the first test in production I get 3500. How can I repeat the AD models dynamically amongst all the clusters?&lt;/p&gt;&#xA;" OwnerUserId="50083" LastActivityDate="2024-01-05T14:02:45.333" Title="Dynamic clustering" Tags="|machine-learning|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31947" PostTypeId="1" AcceptedAnswerId="37793" CreationDate="2018-05-21T18:02:16.610" Score="7" ViewCount="10128" Body="&lt;p&gt;According to the Scikit-Learn documentation for the RBF kernel:&#xA;The length scale of the kernel. If a float, an isotropic kernel is used. If an array, an anisotropic kernel is used where each dimension of l defines the length-scale of the respective feature dimension.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently working on a problem where I am setting the length-scale of each individual feature (which I assume is synonymous with dimension here). My understanding is that a smaller length scale implies a more complex function.&lt;a href=&quot;https://i.stack.imgur.com/w4WPH.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/w4WPH.png&quot; alt=&quot;&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is, can I use this parameter to explain how well a certain feature will help a model generalize to new data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I have a data set which, after optimizing the length-scale value looks like this:&#xA;[Feature_1: length-scale = 20]&#xA;[Feature_2: length-scale = 1]&#xA;[Feature_3: length-scale = 5]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does this mean that, if I had to pick one feature which would help a model generalize to new data, it would be Feature_1? Is Feature_2 potentially causing my model to overfit? Are these fair assumptions to make? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I am using support vector regression with this kernel. &lt;/p&gt;&#xA;" OwnerUserId="38047" LastActivityDate="2018-09-04T12:42:33.947" Title="How do I interpret the length-scale parameter of the RBF kernel?" Tags="|scikit-learn|svm|kernel|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31948" PostTypeId="2" ParentId="30362" CreationDate="2018-05-21T19:10:04.850" Score="10" Body="&lt;p&gt;In principle no you do not need to check for stationarity nor correct for it when you are using an LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing about stationarity is that it makes prediction tasks much more efficient, and stable. Think about stationarity in terms of a target. When what you are trying to predict is not stationary, it is like trying to shoot a moving target. It isn't impossible to do, but there is a better way to do things.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are trying to predict a stationary variable, then what you are trying to do is hit a non-moving, for lack of a better word, stationary target. It is staying in the same spot relative to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Correcting for non-stationarities in an ARIMA model is necessary because it is one of the fundamental assumptions that you make about that particular model. LSTMs do not make that assumption. So you don't have to do it with an LSTM. Seems like the answer to your question is the same as what the people you asked originally. Case closed, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well... you remember shooting at moving targets is much more difficult than shooting at a stationary target right? You may want to force stationarity on a problem that you are going to run against an LSTM, simply because you may be able to eek out a little bit more performance by making it easier for the neural network to learn. So although you don't need to do it, it may still be a good idea and give you a boost in performance.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-05-21T19:10:04.850" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31949" PostTypeId="1" CreationDate="2018-05-21T19:14:32.447" Score="2" ViewCount="180" Body="&lt;p&gt;In lecture notes for cs231n while discussing checking analytical gradient with numerical gradient the paragraph says this:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Check only few dimensions. In practice the gradients can have sizes of million parameters. In these cases it is only practical to check some of the dimensions of the gradient and assume that the others are correct.&lt;/p&gt;&#xA;&lt;p&gt;Be careful: One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter. In some applications, people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector, so it is important to not sample at random but to take this into account and check that all parameters receive the correct gradients.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I understand to check only a few dimensions but what does the part after be careful mean. I fail to understand that warning. can somebody explain it?&lt;/p&gt;&#xA;" OwnerUserId="32169" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-23T05:24:33.873" Title="What does it mean to 'gradient check a few dimensions for every separate parameter'?" Tags="|neural-network|deep-learning|optimization|cs231n|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31950" PostTypeId="1" CreationDate="2018-05-21T20:00:42.800" Score="2" ViewCount="502" Body="&lt;p&gt;I would like to pass a vector that is outside of the training data, but the same length as the training data, to a custom loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vector represents a post-prediction funnel (one or zero) that an observation has to pass through before they can yield (one or zero).  Obviously, I can't use this funnel as a feature, but I would like to use it in a loss function. Something along the lines of cross entropy plus an additional penalty...&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;funnel &amp;lt;- k_constant(fnl)&#xA;loss &amp;lt;- function(y_true,y_pred){&#xA;   k_sum(-((y_true*k_log(y_pred)) +&#xA;   (1-y_true)*k_log(1-y_pred) +&#xA;   k_transpose(funnel)*(1-y_true)*k_log(1-y_pred)))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This code works syntactically, but I'm concerned that when I go to fit the model, &lt;code&gt;batch_size&lt;/code&gt; and &lt;code&gt;shuffle&lt;/code&gt; arguments rearrange the data in a way that &lt;code&gt;funnel&lt;/code&gt; no longer lines up with the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Question...&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I be sure that the &lt;code&gt;funnel&lt;/code&gt; and the (&lt;code&gt;y_true&lt;/code&gt;, &lt;code&gt;y_pred&lt;/code&gt;) pairs are referencing the same observation when calculating the loss?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More info on the 'why' of this problem, as well as additional code that shows my progress thus far, can be found &lt;a href=&quot;http://rpubs.com/burchz/KerasCustomLoss&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance for any help!  I'm relatively new to deep learning, so please bear with me.  &lt;/p&gt;&#xA;" OwnerUserId="52485" LastEditorUserId="52485" LastEditDate="2018-06-18T15:44:53.220" LastActivityDate="2018-06-18T15:44:53.220" Title="Keras/TensorFlow in R - Additional Vector to Custom Loss Function" Tags="|r|keras|loss-function|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31951" PostTypeId="1" CreationDate="2018-05-21T20:18:23.223" Score="1" ViewCount="1100" Body="&lt;p&gt;I'm trying to build a Convolutional Neural Network model to classify and predict a brain tumor based on images. I am looking for a database containing images of brain tumor. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone introduce some medical databases (small one to test or big ones) with links to download?&lt;/p&gt;&#xA;" OwnerUserId="50477" LastEditorUserId="44456" LastEditDate="2018-05-22T11:53:57.443" LastActivityDate="2018-05-22T11:53:57.443" Title="Brain Tumor Database for CNN?" Tags="|dataset|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31952" PostTypeId="1" CreationDate="2018-05-21T20:24:59.503" Score="3" ViewCount="205" Body="&lt;p&gt;I have digit images as below which I would like to identify: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/UGnak.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/UGnak.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some are of slightly worse quality :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/rFjWy.png,&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rFjWy.png,&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The images are not of a fixed resolution but are mostly in the range (80*20 to 130 *40).&#xA;Due to lack of enough labelled data(~3K rows), I had to go for an open source digit dataset. I tried both MNIST and SVHN with not much luck. I've detailed both my approaches below..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note : The only preprocessing that I've used is converting to grayscale ,resizing the images to a fixed size and doing a mean subtraction on the image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1. SINGLE DIGIT MNIST and SLIDING WINDOW FOR TESTING&lt;/strong&gt; &lt;strong&gt;:&lt;/strong&gt; I trained a Convnet  on the MNIST set with ~99% on the validation set.Then I used a sliding window to move over my test image and predict for each. This approach had two problems &lt;strong&gt;:&lt;/strong&gt; First, as my images are not of a fixed resolution often I couldn't get the right sliding window (one which contains just one digit completely). Second, I cropped out &lt;strong&gt;individual digits&lt;/strong&gt; from my test images, and tested my MNIST model on these, the results were 'wildly off', out of the ~15 samples that I had cropped out, my MNIST model couldn't get even one right. Here's a cropped out image for reference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DXSs0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DXSs0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;SINGLE DIGIT SVHN&lt;/strong&gt; I thought that the performance of my MNIST model on my test set maybe bad as these two are different type of images, numbers in the SVHN dataset are more similar to the ones I posted.&#xA;I repeated the same single digit exercise with SVHN, with far better results, on my small cropped out set of 15 numbers,this time I got 10 right but the sliding window problem still remained.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;MULTI DIGIT SVHN&lt;/strong&gt; : As house numbers in SVHN are of smaller lengths(max 5) than my numbers, I created 50k numbers of length 8 by horizontally stacking different numbers from the SVHN multi digits dataset.The best validation loss I've been able to achieve from this model is 6.70, not very great and obviously not very effective at predicting. I used a  ConvNet with the following architecture:&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;model_input = Input(input_shape)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  x = Conv2D(32, (3, 3),data_format='channels_first', activation='relu', name='conv_32_1',padding='same')(model_input)&#xA;  x = Conv2D(32, (3, 3), data_format='channels_first',activation='relu',name='conv_32_2',padding='same')(x)&#xA;  x = MaxPooling2D(pool_size=(2, 2))(x)    &#xA;  x = Dropout(0.35)(x)&#xA;&#xA;  x = Conv2D(64, (3, 3),  data_format='channels_first',activation='relu',name='conv_64_1',padding='same')(x)       &#xA;  x = Conv2D(64, (3, 3), data_format='channels_first',activation='relu',name='conv_64_2',padding='same')(x)    &#xA;  x = MaxPooling2D(pool_size=(2, 2))(x)    &#xA;&#xA;  x = Conv2D(128, (3, 3), data_format='channels_first',activation='relu',name='conv_128_1',padding='same')(x)    &#xA; x = Dropout(0.35)(x)&#xA;&#xA; x = Flatten()(x)&#xA;&#xA; x = Dense(768, activation='relu')(x)    &#xA; x = Dropout(0.50)(x)&#xA; output_list = [Dense(11, activation='softmax',name='digit_'+str(i))(x)    for i in range(1,9)]&#xA; adam = Adam(lr=0.0005)&#xA;&#xA; model = Model(inputs=model_input, outputs=output_list)&#xA;&#xA; model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is the loss curve:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/aszQ6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aszQ6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My questions,going from the last approach to the 1st :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Is my MULTI SVHN model too small for this problem/dataset? If not, how do I improve its performance?&lt;/li&gt;&#xA;&lt;li&gt;Is there a way to generate better candidate sliding windows, as my single digit SVHN seems to be the best model I have among the three.&lt;/li&gt;&#xA;&lt;li&gt;Is my reasoning correct that the MNIST model didn't work well on my images as the digits are somewhat different (handwritten vs computer generated) ?&lt;/li&gt;&#xA;&lt;li&gt;Are there other preprocessing  techniques that I could use to improve my performance espeically for images like the 2nd one (18458882)?&lt;/li&gt;&#xA;&lt;li&gt;Is there any other approach that I could try to solve this?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="30805" LastActivityDate="2018-05-21T20:24:59.503" Title="Identifying computer scanned digits" Tags="|deep-learning|cnn|image-recognition|mnist|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31953" PostTypeId="2" ParentId="700" CreationDate="2018-05-21T20:48:53.453" Score="1" Body="&lt;p&gt;Try using HDBSCAN, even though it is a hierarchical method, it might prove more efficient. I am running it on multidimensional datasets a bit lengthier than 200k, and it takes less than a minute to run. The caveat is the number of clusters that it might produce. If they are too many you might want to stick with a partitioning method. &lt;/p&gt;&#xA;" OwnerUserId="50083" LastActivityDate="2018-05-21T20:48:53.453" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31954" PostTypeId="2" ParentId="31951" CreationDate="2018-05-21T21:23:41.247" Score="2" Body="&lt;p&gt;Some of the followings might be useful:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.virtualskeleton.ch/BRATS/Start2015&quot; rel=&quot;nofollow noreferrer&quot;&gt;Brain Tumor Image Segmentation Challenge&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets/primary+tumor&quot; rel=&quot;nofollow noreferrer&quot;&gt;Primary Tumor Data Set by UC Irvine Machine Learning Repository&lt;/a&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.ncbi.nlm.nih.gov/gds?term=brain%5BSample%20Source%5D&quot; rel=&quot;nofollow noreferrer&quot;&gt;ncbi.nlm.nih.gov&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.medinfo.cs.ucy.ac.cy/index.php/downloads/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;University of Cyprus&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://figshare.com/articles/brain_tumor_dataset/1512427&quot; rel=&quot;nofollow noreferrer&quot;&gt;Brain Tumor Dataset&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-21T21:23:41.247" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31955" PostTypeId="1" CreationDate="2018-05-21T21:43:52.043" Score="2" ViewCount="180" Body="&lt;p&gt;I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under &quot;AppName&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing this in Pyspark:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;display(df.filter(df.AppName==&quot;23954ec32332dfgd&quot;))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;only yields 1 row as opposed to at least 1400 rows.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas ?&lt;/p&gt;&#xA;" OwnerUserId="52493" LastEditorUserId="29575" LastEditDate="2018-05-22T01:42:34.473" LastActivityDate="2018-12-19T07:01:15.233" Title="PySpark Filter shows only 1 row" Tags="|apache-spark|pyspark|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31956" PostTypeId="2" ParentId="31893" CreationDate="2018-05-21T21:54:45.223" Score="0" Body="&lt;p&gt;In most individual differences research, subjects are additionally coded along some dimension, such as young/old, low/high working memory, normal/cognitively impaired. Without this, you’re simply looking at individual variation, which doesn’t seem to address any question (what &lt;em&gt;is&lt;/em&gt; your research question?). If you simply want to look at variation in overall RT on a subject level, I’d construct CDFs of mean RT collapsing over color, or perhaps for each color separately.  This will simply demonstrate that people differ, and you’ll find the distributions adhere to roughly ex-Gaussian. If you mean to test differences between colors, you can use ANOVA or compare CDFs using K-S tests, but again there isn’t a clear research question here. &lt;/p&gt;&#xA;" OwnerUserId="24593" LastActivityDate="2018-05-21T21:54:45.223" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31957" PostTypeId="1" AcceptedAnswerId="31974" CreationDate="2018-05-21T23:28:38.230" Score="2" ViewCount="7467" Body="&lt;p&gt;I am using Scikit's &lt;code&gt;MLPRegressor&lt;/code&gt; for a timeseries prediction task. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My data is scaled between 0 and 1 using the &lt;code&gt;MinMaxScaler&lt;/code&gt; and my model is initialized using the following parameters:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MLPRegressor(solver='lbfgs', hidden_layer_sizes=50,&#xA;                           max_iter=10000, shuffle=False, random_state=9876,&#xA;                           activation='relu')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am expecting output between 0 and 1 but getting values outside the bound (both negative values as well as &gt; 1).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Non-normalized data has the same problem, I get predictions out of range! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea where I could be wrong?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt;&#xA;Based on the answers below I played a bit with modifying the output activation layers and got some interesting results that I thought worth sharing. There are three scenarios, hope the captions convey the message clearly:&#xA;Legend:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Black solid line = Training Epoch&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Red solid line = Test Epoch&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cyan dashed line = Network prediction over the entire data set&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Output when the network is trained using 'relu' activation layer but output_activation_ set to 'logistic&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4Vxbs.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Vxbs.png&quot; alt=&quot;Output when the network is trained using &amp;#39;relu&amp;#39; activation layer but output_activation_ set to &amp;#39;logistic&amp;#39;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Output when the network is trained using 'relu' activation layer and output_activation_ set explicitly to 'relu'&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mXUr2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mXUr2.png&quot; alt=&quot;Output when the network is trained using &amp;#39;relu&amp;#39; activation layer and output_activation_ set explicitly to &amp;#39;relu&amp;#39;&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;3&quot;&gt;&#xA;&lt;li&gt;Output when the network is trained using 'relu' activation layer and output_activation is left alone&#xA;&lt;a href=&quot;https://i.stack.imgur.com/XRk2P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/XRk2P.png&quot; alt=&quot;Output when the network is trained using &amp;#39;relu&amp;#39; activation layer and output_activation is left alone&quot;&gt;&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52494" LastEditorUserId="50727" LastEditDate="2020-07-23T15:12:27.270" LastActivityDate="2020-07-23T15:12:27.270" Title="MLPRegressor Output Range" Tags="|machine-learning|deep-learning|scikit-learn|mlp|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31958" PostTypeId="1" AcceptedAnswerId="32135" CreationDate="2018-05-22T01:24:12.030" Score="3" ViewCount="215" Body="&lt;p&gt;We're working on a ranking texts by degree of the similarity to vacancies. We have 4-year data set (≈1M texts) of custom search feeds from social network. We also have vacancies (≈30K) manually selected from this set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the vacancies are selected in 2 stages:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;We make several specific search queries (i.e. &quot;need translator&quot;) on the social network by API, receiving and combining the search feeds into one.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Then we look through each post in it for real vacancies. Generally, it should contain a request for interpreter services. Often it contains some &lt;em&gt;job details&lt;/em&gt; (such as a place, salary, subject, deadlines, languages, contacts) and &lt;em&gt;stylistic features of speech&lt;/em&gt; (i.e. greetings, exclamations, request to write in private, ask for recommendations, etc.).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Before the point 2, we would like to have first in our final moderation feed the texts, that are most likely to be vacancies. We're going to try ranking by using the Naive Bayes classifier or the Natural Language Processing, but it's not as simple as it seems: there are a lot of various tools, schemes, models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is, &lt;strong&gt;what algorithms and tools could help in our case?&lt;/strong&gt; What we should pay attention to?&lt;/p&gt;&#xA;" OwnerUserId="52433" LastActivityDate="2018-05-29T08:09:37.230" Title="Algorithms and tools for ranking text as a job description" Tags="|neural-network|nlp|naive-bayes-classifier|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31959" PostTypeId="2" ParentId="31729" CreationDate="2018-05-22T04:07:05.180" Score="1" Body="&lt;p&gt;Random forest is a learning algorithm. It is an ensemble learning algorithm that uses decision trees as base learners. You wrote the steps for it correctly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Rain forest is not a learning algorithm. It is an algorithm of constructing a decision tree (how to do splitting) when the dataset is so large that it does not fit the memory. In rain forest, the whole dataset is not required for making a splitting decision. Only some aggregated information (AVC-set for an attribute or AVC-group if you have more memory) is required.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your dataset is large, and memory is small, you can use rain forest to build several different decision trees. Then use random forest algorithm with those trees as base learners.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-22T04:07:05.180" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31960" PostTypeId="2" ParentId="31046" CreationDate="2018-05-22T04:19:56.023" Score="2" Body="&lt;p&gt;The Policy Gradient theorem states that the gradient of the expected reward is equal to the expectation of the log probability of the current policy multiplied by the reward. Notice that to compute the integral of the expectation we can use Monte Carlo method. For this you would need sampled trajectories. Have you looked at a &lt;a href=&quot;http://www.scholarpedia.org/article/Policy_gradient_methods&quot; rel=&quot;nofollow noreferrer&quot;&gt;derivation&lt;/a&gt;? The whole theorem comes from mathematical derivations. Check in that page the &lt;em&gt;Assumptions&lt;/em&gt; and the &lt;em&gt;Likelihood Ratios&lt;/em&gt; (REINFORCE) sections. Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-05-22T04:19:56.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31961" PostTypeId="2" ParentId="31955" CreationDate="2018-05-22T05:07:39.143" Score="2" Body="&lt;p&gt;Can you try this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df1 = df.filter(df.AppName==&quot;23954ec32332dfgd&quot;)&#xA;print df1.count()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2018-05-22T05:07:39.143" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31964" PostTypeId="1" CreationDate="2018-05-22T06:35:48.520" Score="0" ViewCount="170" Body="&lt;p&gt;I have a dataset which is categorical. I did one hot encoding in the train data which is taking much time for each run of my algorithm. I have also performed one hot encoding on the test data which I give at the run time to the algorithm. I want to know how can we avoid the one hot encoding of the dataset for each run, rather I want it to be done only on the new test data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on classification problem using Decision tree classifier.My dataset is categorical so I have done one hot encoding on the dataset as well as on the active user input given at the run time . It takes too much time while doing encoding in dataset. I wish , if once one hot encoding is done on training dataset, I should not do it again and again on each run and to perform encoding only the active test input i.e. on 'a'.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;features = [0,1,2,4,5,6,7,8]&#xA;X = dataset.iloc[:,features]&#xA;y = dataset.iloc[:,3]&#xA;X_ordinal = X.values&#xA;&#xA;encoder = OneHotEncoder(handle_unknown = 'ignore')&#xA;encoder.fit(X_ordinal)&#xA;X_one_hot = enc.transform(X_ordinal)&#xA;end4 = time.clock()&#xA;&#xA;&#xA;print(&quot;enter the inputs in space separated format:&quot; )&#xA;start5 = time.clock()&#xA;a = [int(p) for p in input().split()]&#xA;&#xA;values=array(a)&#xA;X_test_one_hot=encoder.transform(values.reshape(1, -1))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52506" LastEditorUserId="52506" LastEditDate="2018-05-22T09:18:00.963" LastActivityDate="2018-05-22T09:18:00.963" Title="How to avoid one hot encoding again and again on each run" Tags="|machine-learning|python|" AnswerCount="0" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="31966" PostTypeId="2" ParentId="20296" CreationDate="2018-05-22T07:15:39.703" Score="2" Body="&lt;p&gt;Let's see how the gradient of the loss behaves... We have the cross-entropy as a loss function, which is given by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$&#xA;H(p,q) = -\sum_{i=1}^n p(x_i) \log(q(x_i)) = -(p(x_1)\log(q(x_1)) + \ldots + p(x_n)\log(q(x_n)) &#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Going from here.. we would like to know the derivative with respect to some $x_i$:&#xA;$$&#xA;\frac{\partial}{\partial x_i} H(p,q) = -\frac{\partial}{\partial x_i} p(x_i)\log(q(x_i)).&#xA;$$&#xA;Since all the other terms are cancelled due to the differentiation. We can take this equation one step further to&#xA;$$&#xA;\frac{\partial}{\partial x_i} H(p,q)  = -p(x_i)\frac{1}{q(x_i)}\frac{\partial q(x_i)}{\partial x_i}.&#xA;$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From this we can see that we are still only penalizing the true classes (for which there is value for $p(x_i)$). Otherwise we just have a gradient of zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do wonder how to software packages deal with a predicted value of 0, while the true value was larger than zero... Since we are dividing by zero in that case.&lt;/p&gt;&#xA;" OwnerUserId="38670" LastActivityDate="2018-05-22T07:15:39.703" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31968" PostTypeId="2" ParentId="31957" CreationDate="2018-05-22T07:42:24.420" Score="1" Body="&lt;p&gt;Altough your data is between 0 and 1, the predictions can be outside of this range, as with any satistical model. This shows that you are not able to represent your data properly with your model. However, this is not the important point. If you want to ensure the outputs to be between 0 and 1, you have to change the relu activation for another one, such as sigmoid or softmax. These activations ensure that your outputs will be between 0 and 1, although they might lead to other problems, such as vanishing gradients. What is usually advised is to use relu layers until the last one, which has to be a sigmoid or a softmax.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-22T07:42:24.420" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31969" PostTypeId="1" AcceptedAnswerId="31972" CreationDate="2018-05-22T08:04:03.037" Score="4" ViewCount="7463" Body="&lt;p&gt;So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The issue I'm having is that I'm using preprocessing to scale the data to train my classifier. However when I make a prediction, the answer is also scaled. When I comment out all the preprocessing, I get accurate results. Is there a way to scale the prediction back?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The outputs are rounded to 0, 1 or 2 with each number representing one of three species.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can see my code below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn import preprocessing, model_selection&#xA;from sklearn.linear_model import LinearRegression&#xA;&#xA;df = pd.read_csv(&quot;iris.csv&quot;)&#xA;&#xA;# setosa - 0&#xA;# versicolor - 1&#xA;# virginica - 2&#xA;df = df.replace(&quot;setosa&quot;, 0)&#xA;df = df.replace(&quot;versicolor&quot;, 1)&#xA;df = df.replace(&quot;virginica&quot;, 2)&#xA;&#xA;X = np.array(df.drop(['species'], 1))&#xA;&#xA;y = np.array(df['species'])&#xA;&#xA;# Scale features&#xA;# X = preprocessing.scale(X)&#xA;&#xA;X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)&#xA;&#xA;clf = LinearRegression(n_jobs=1)  # Linear regression clf&#xA;&#xA;clf.fit(X_train, y_train)&#xA;&#xA;confidence = clf.score(X_test, y_test)&#xA;&#xA;print(&quot;Confidence: &quot; + confidence)&#xA;&#xA;# Inputs&#xA;sepal_length = float(input(&quot;Enter sepal length: &quot;))&#xA;sepal_width = float(input(&quot;Enter sepal width: &quot;))&#xA;petal_length = float(input(&quot;Enter petal length: &quot;))&#xA;petal_width = float(input(&quot;Enter petal width: &quot;))&#xA;&#xA;# Create panda data frame with inputted data&#xA;index = [0]&#xA;d = {'sepal_length': sepal_length, 'sepal_width': sepal_width, 'petal_length': petal_length, 'petal_width': petal_width}&#xA;predict_df = pd.DataFrame(data=d, index=index)&#xA;&#xA;# Create np array of features&#xA;predict_X = np.array(predict_df)&#xA;&#xA;# Need to scale new X feature values&#xA;# predict_X = preprocessing.scale(predict_X, axis=1)&#xA;&#xA;# Make a prediction against prediction features&#xA;prediction = clf.predict(predict_X)&#xA;&#xA;print(predict_X, prediction)&#xA;&#xA;rounded_prediction = int(round(prediction[0]))&#xA;&#xA;if rounded_prediction == 0:&#xA;    print(&quot;== Predicted as Setosa ==&quot;)&#xA;elif rounded_prediction == 1:&#xA;    print(&quot;== Predicted as Versicolor ==&quot;)&#xA;elif rounded_prediction == 2:&#xA;    print(&quot;== Predicted as Virginica ==&quot;)&#xA;else:&#xA;    print(&quot;== Unable to make a prediction ==&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is an example of my output with preprocessing enabled. I'll be using one of the lines from the CSV as an example (6.4 sepal length, 3.2 sepal width, 4.5 petal length and 1.5 petal width) which should equal the versicolor species (1):&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Confidence: 0.9449475378336242&#xA;Enter sepal length: 6.4&#xA;Enter sepal width: 3.2&#xA;Enter petal length: 4.5&#xA;Enter petal width: 1.5&#xA;[[ 1.39427847 -0.39039797  0.33462683 -1.33850733]] [0.41069281]&#xA;== Predicted as Setosa ==&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now with preprocessing commented out:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Confidence: 0.9132522144785978&#xA;Enter sepal length: 6.4&#xA;Enter sepal width: 3.2&#xA;Enter petal length: 4.5&#xA;Enter petal width: 1.5&#xA;[[6.4 3.2 4.5 1.5]] [1.29119283]&#xA;== Predicted as Versicolor ==&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It seems I'm either doing the preprocessing wrong, or there's an extra step that I've missed out. I'm sorry if I get some of the terminology wrong and thanks in advance for answering.&lt;/p&gt;&#xA;" OwnerUserId="52509" LastActivityDate="2018-05-22T08:27:51.587" Title="How to scale prediction back after preprocessing" Tags="|machine-learning|python|preprocessing|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31971" PostTypeId="2" ParentId="31969" CreationDate="2018-05-22T08:27:20.343" Score="2" Body="&lt;p&gt;When you decided you have to scale your data, you usually have to follow these steps:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For training:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Scale / Standarize the training set&lt;/li&gt;&#xA;&lt;li&gt;Store the scaling / standarization factors of the training set&lt;/li&gt;&#xA;&lt;li&gt;Train the model&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;For predicting:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Scale / standarize the input data, but very important, with the scaling/standarization factors stored during the training process. You dont have to compute the min, max or mean values of the new data.&lt;/li&gt;&#xA;&lt;li&gt;Predict&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;The reason is that you have to map the new data to the same feature-space used for the training process, so you have to scale/std it with the same factors, otherwise, you are changing the feature space.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-22T08:27:20.343" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31972" PostTypeId="2" ParentId="31969" CreationDate="2018-05-22T08:27:51.587" Score="1" Body="&lt;p&gt;I think your methodology is correct, but this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Scale features&#xA;# X = preprocessing.scale(X)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;should be changed to:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Scale features&#xA;# X = preprocessing.scale(X, axis = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As the default for scale is to set axis to 0 (I wonder why!). If the problem persists comment it and I will edit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Although your methodology is not wrong, it is more suitable to use sklearn StandardScaler. See &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the documentation&lt;/a&gt; of this class. Usually, it is better to fit the scaler with the training data and transform the test data according to that fit.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-22T08:27:51.587" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31973" PostTypeId="1" AcceptedAnswerId="31999" CreationDate="2018-05-22T08:40:09.823" Score="4" ViewCount="256" Body="&lt;p&gt;I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, as the colorization process can be completely automated and gives good results. I trained a wide variety of models and architectures (CNNs, Pix2Pix, DCGANs and WGANs with UNet or residual architectures, etc.) and settled with convolutional cGANs &lt;em&gt;(condition: grayscal image ; output : colored image)&lt;/em&gt;, but end up facing the same problem every time. Indeed, when training my network, the output is quite correct but always gives me grayish roofs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think it has something to do with the distribution of pixels values, as roofs are usually orange or black in my geographic area. Many articles state the fact that GANs are prone to mode collapse. In this case, there are two modes (orange and black) which probably results in a bad local equilibrium.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried many different techniques in order to improve training and/or avoid mode collapse :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;L1 loss regularization for the generator&lt;/li&gt;&#xA;&lt;li&gt;Label smoothing&lt;/li&gt;&#xA;&lt;li&gt;Noise taken from a normal distribution and concatenated to every layer in the generator/discriminator&lt;/li&gt;&#xA;&lt;li&gt;Dropout for some layers in the generator&lt;/li&gt;&#xA;&lt;li&gt;Gradient penalty&lt;/li&gt;&#xA;&lt;li&gt;Learning procedure inspired by DRAGANs and BEGANs&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;So far, gradient penalty has given me the best results, despite still having these grayish roofs. I know there are other ways of improving the training procedure (e.g. batch discrimination), but I do not have the pre-requisites in mathematics or computer science nor the time for implementing such techniques.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if someone had some ideas for getting better results, or if someone also tried colorizing satellite imagery and ended up having such issues.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for reading this long message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Have a nice day !&lt;/p&gt;&#xA;" OwnerUserId="52510" LastActivityDate="2018-05-22T21:02:04.807" Title="GANs and grayscale imagery colorization" Tags="|machine-learning|deep-learning|gan|regularization|generative-models|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31974" PostTypeId="2" ParentId="31957" CreationDate="2018-05-22T09:08:19.533" Score="1" Body="&lt;p&gt;The default output activation of the Scikit-Learn &lt;code&gt;MLPRegressor&lt;/code&gt; is 'identity', which actually does nothing to the weights it receives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As was mentioned by @David Masip in his answer, changing the final activation layer would allow this. Doing so in frameworks such as Pytorch, Keras and Tensorflow is fairly straight-forward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doing it in your code with the &lt;code&gt;MLPRegressor&lt;/code&gt; means using an object attribute that isn't a standard parameter, namely &lt;code&gt;output_activation_&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are the built-in options that I can see in &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;the documentation&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;&lt;pre&gt;&lt;code&gt;activation : {‘identity’, ‘logistic’, ‘tanh’, ‘relu’}, default ‘relu’&#xA;&#xA;Activation function for the hidden layer.&#xA;&#xA;‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x&#xA;‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).&#xA;‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).&#xA;‘relu’, the rectified linear unit function, returns f(x) = max(0, x)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Setting it's value to &lt;code&gt;logistic&lt;/code&gt; gives you the property you would like, values between 0 and 1.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&#xA;After comments and update from OP: in their case, using &lt;code&gt;logistic&lt;/code&gt; (sigmoid) as the final activation negatively affected results. So perhaps it is worth trying out all possible activation functions to investigate which activation best suits the model and data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One further remark, at least within the context of deep learning, it is common practice not to use an activation at the final output of a neural network - for some thoughts around that discussion, &lt;a href=&quot;https://stats.stackexchange.com/questions/163695/non-linearity-before-final-softmax-layer-in-a-convolutional-neural-network&quot;&gt;see this thread&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;That being said, below is a simple working example of a model that doesn't set it, and one that does. I use random numbers to make it work, but the take-away is that the predicted values for the altered model are always within the range from 0 to 1. Try changing the random seed and re-running the script.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from sklearn.neural_network import MLPRegressor&#xA;&#xA;# To see an example where output falls outside of the range of y&#xA;np.random.seed(1)&#xA;&#xA;# Create the default NN as you did&#xA;nn = MLPRegressor(&#xA;    solver='lbfgs',&#xA;    hidden_layer_sizes=50,&#xA;    max_iter=10000,&#xA;    shuffle=False,&#xA;    random_state=9876,&#xA;    activation='relu')&#xA;&#xA;# Generate some fake data&#xA;num_train_samples = 50&#xA;num_test_samples = 50&#xA;num_vars = 2&#xA;&#xA;X = np.random.random((num_train_samples, num_vars)) * \&#xA;    100  # random numbers between 0 and 100&#xA;y = np.random.uniform(0, 1, (num_train_samples, 1))  # uniform numbers between 0 and 1&#xA;&#xA;X_test = np.random.random((num_test_samples, num_vars)) * 100&#xA;y_test = np.random.uniform(0, 1, (num_test_samples, 1))&#xA;&#xA;# Fit the network&#xA;nn.fit(X, y)&#xA;&#xA;print('*** Before scaling the output via final activation:\n')&#xA;&#xA;# Now see that the output activation is (by default) simply linear i.e. 'identity'&#xA;print('Output activation by default: {}'.format(nn.out_activation_))&#xA;predictions = nn.predict(X_test)&#xA;&#xA;print('Prediction mean: {:.2f}'.format(predictions.mean()))&#xA;print('Prediction max: {:.2f}'.format(predictions.max()))&#xA;print('Prediction min: {:.2f}'.format(predictions.min()))&#xA;&#xA;&#xA;print('\n*** After scaling the output via final activation:\n')&#xA;&#xA;# Need to recreate the NN&#xA;nn_logistic = MLPRegressor(&#xA;    solver='lbfgs',&#xA;    hidden_layer_sizes=50,&#xA;    max_iter=10000,&#xA;    shuffle=False,&#xA;    random_state=9876,&#xA;    activation='relu')&#xA;&#xA;# Fit the new network&#xA;nn_logistic.fit(X, y)&#xA;&#xA;&#xA;# --------------- #&#xA;#  Crucial step!  #&#xA;# --------------- #&#xA;&#xA;# before making predictions = alter the attribute: &quot;output_activation_&quot;&#xA;nn_logistic.out_activation_ = 'logistic'&#xA;print('New output activation: {}'.format(nn_logistic.out_activation_))&#xA;&#xA;new_predictions = nn_logistic.predict(X_test)&#xA;&#xA;print('Prediction mean: {:.2f}'.format(new_predictions.mean()))&#xA;print('Prediction max: {:.2f}'.format(new_predictions.max()))&#xA;print('Prediction min: {:.2f}'.format(new_predictions.min()))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Tested using Python 3.5.2.&lt;/p&gt;&#xA;" OwnerUserId="45264" LastEditorUserId="45264" LastEditDate="2018-05-23T07:54:33.770" LastActivityDate="2018-05-23T07:54:33.770" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31975" PostTypeId="1" AcceptedAnswerId="31976" CreationDate="2018-05-22T09:20:13.463" Score="3" ViewCount="4135" Body="&lt;p&gt;I have a large labeled dataset with 29 classes. Is is possible to use a clustering algorithm (like k-means) in this dataset, or it's not possible since clustering algorithms are unsupervised ? &lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-05-22T09:36:54.637" Title="Clustering a labeled data set" Tags="|machine-learning|classification|clustering|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31976" PostTypeId="2" ParentId="31975" CreationDate="2018-05-22T09:36:54.637" Score="1" Body="&lt;p&gt;You can do many things:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Forget about the labels: just use the features that are not labels and cluster along those features using the k-means algorithm (or another).&lt;/li&gt;&#xA;&lt;li&gt;Forget about the features: this is the dummiest way of clustering. Cluster the data in 29 clusters according to the labels that they have. If you want less clusters, you can compute the centroids of the classes and use them to join clusters of different labels.&lt;/li&gt;&#xA;&lt;li&gt;Use everything: create a categorical variable refering to the class that every example belongs to. Then, with this new variable and all the features perform a classical clustering algorithm.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The way to proceed depends on if you want to use the labels or not, and how much importance you want them to have.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-22T09:36:54.637" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="31977" PostTypeId="1" CreationDate="2018-05-22T09:43:55.527" Score="1" ViewCount="23" Body="&lt;p&gt;Now I am using the keras model: Inception-ResnetV2 to do image classification using transfer learning. The main code about this model is as following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; def model_InceptionResNetV2(img_dim, num_label):&#xA;    print('begin to get model')&#xA;    input_tensor = Input(shape=img_dim)&#xA;    #InceptionResNetV2&#xA;#     base_model = InceptionV3(include_top=False,&#xA;#                    weights='imagenet',&#xA;#                    input_shape=img_dim)&#xA;    base_model = InceptionResNetV2(include_top=False, input_shape=img_dim, weights='imagenet')&#xA;    bn = BatchNormalization()(input_tensor)&#xA;    x = base_model(bn)&#xA;    x = GlobalAveragePooling2D()(x)&#xA;    x = Dropout(0.5)(x)&#xA;    output = Dense(num_label, activation='softmax')(x)&#xA;    model = Model(input_tensor, output)&#xA;    print('finish getting model')&#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But the model is not good for my image classification, some guys suggest me changing the pooling to adaptpooling before FC layer in model_InceptionResNetV2 so that the mode is not sensitive to image size, I want to use 512*512 size not just 299*299. Could you guys suggest me how to do it. Thanks!&lt;/p&gt;&#xA;" OwnerUserId="35597" LastEditorUserId="35597" LastEditDate="2018-05-22T10:10:55.787" LastActivityDate="2018-05-22T10:10:55.787" Title="How to change the pooling to adaptpooling before FC layer in the Inception-ResnetV2 model in keras" Tags="|python|deep-learning|keras|transfer-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31979" PostTypeId="1" CreationDate="2018-05-22T10:40:11.760" Score="3" ViewCount="516" Body="&lt;p&gt;I am doing a text generation project -- the task is to basically represent the statistical data in a readable way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I decided to go about this is template-based: each data type has a template for how sentence should be formed and what synonyms can be used.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm torn about whether some kind of ML techinque can bolster this template-based approach. Text should be unique -- so I need an algorithm that optimises for uniqueness. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, there are API solutions that can give me uniqueness score at the end of the text (and even in the middle of it) -- so my first instinct was to try out reinforcement learning with sparse rewards. Templates can be represented as a tree, which the algorithm traverses, getting rewards in the end and during its trip. Input is its current options on where to go and the output is its decision on where to go.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem with this approach is that after it succeeded in generating an unique text, it can't generate the same one (I mean it can, but the score would be 0), which might prove difficult for the model to learn. Also many articles on the web indicate that RL is really, really hard to tune properly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am now in the pre-research process, so any feedback on how I should approach this task is appreciated. Maybe there is no need for ML at all?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What do you think? My instinct tells me that such problems should have established solutions and I'm just searching the wrong way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="52517" LastActivityDate="2024-01-13T14:06:45.807" Title="Choosing a right algorithm for template-based text generation" Tags="|machine-learning|decision-trees|reinforcement-learning|text-generation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31980" PostTypeId="1" CreationDate="2018-05-22T12:17:03.627" Score="2" ViewCount="3388" Body="&lt;p&gt;I'm trying to study the effect of curse of dimensionality in classification algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a simple dataset with only 1 feature. So i can represent the set in a 1 dimension.&#xA;If now i want to introduce some noise in this dataset, is it correct to add another feature with random values to my dataset ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-05-22T13:07:30.400" Title="Add noise to an example set" Tags="|machine-learning|classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31981" PostTypeId="2" ParentId="31980" CreationDate="2018-05-22T12:44:03.457" Score="2" Body="&lt;p&gt;The problem with adding an extra feature with random values is that, if it's uninformative (as it likely is, given that its values are all random), it might get ignored by your classifier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.linear_model import LinearRegression&#xA;from numpy import arange, random, array&#xA;&#xA;x1 = arange(100)&#xA;y = 3 + 2*x1&#xA;LinearRegression().fit(array([x1, x2]).T, y).coef_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;returns &lt;code&gt;array([ 2.00000000e+00, -1.30768001e-15])&lt;/code&gt;, meaning that the coefficient of the new feature (the one with random values) was practically set to $0$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What you could do is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = arange(100)&#xA;y = 3 + 2*x + random.randn(100)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then, if you try plotting &lt;code&gt;y&lt;/code&gt; against &lt;code&gt;x&lt;/code&gt;, you'll see that the values don't lie on a perfectly straight line, but rather they deviate from it slightly (and randomly).&lt;/p&gt;&#xA;" OwnerUserId="52109" LastActivityDate="2018-05-22T12:44:03.457" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31982" PostTypeId="1" CreationDate="2018-05-22T12:52:44.597" Score="1" ViewCount="19" Body="&lt;p&gt;I want to label natural text Dokuments in various different categories (arround 400 different categories) using Neural Networks. I have around 50.000 documents already labeled. The problem is around half of them are english only, the others are either chinese or a mix of chinese and english.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would it be possible to train a network with all of the documents and get a network with an good quality or is it more useful to separate the languages and train the network with english documents only (arround 20.000 docs then)? &lt;/p&gt;&#xA;" OwnerUserId="52523" LastEditorUserId="52523" LastEditDate="2018-05-22T13:46:05.137" LastActivityDate="2018-05-22T13:46:05.137" Title="Machine Learning Text Categorization with Data in multiple Languages" Tags="|machine-learning|neural-network|nlp|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31983" PostTypeId="2" ParentId="31980" CreationDate="2018-05-22T13:05:54.317" Score="3" Body="&lt;p&gt;In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily by regularized models), I would like add that you also increase the dimensionally of the feature space synthetically using many simple mathematical expressions as well. Concertedly, let's say your only feature (column) is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;x&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may easily construct other features like (it is common practice actually in physical sciences):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;x$^2$&lt;/strong&gt;, &lt;strong&gt;x$^3$&lt;/strong&gt;, &lt;strong&gt;x$^{0.5}$&lt;/strong&gt;, $sin(x)$, &lt;strong&gt;x$^2$sin(x)&lt;/strong&gt;,...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Till you hit the so-called &lt;em&gt;the curse of dimensionality&lt;/em&gt; for your exercise. Although I am not yet sure what features/num_samples ratio exactly causes the curse of dimensionality! What I have gathered so far about the curse of dimensionality has been very subjective. &lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-22T13:05:54.317" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31984" PostTypeId="2" ParentId="31980" CreationDate="2018-05-22T13:07:30.400" Score="3" Body="&lt;p&gt;It depends on what is understood as noise, since a noise source can be interpreted as any way of corrupting/altering the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Technically, if you want to add noise to your dataset you can proceed as follows:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Add noise to the raw data, i.e, corrupt the raw data with some noise distribution and with certain signal to noise ratio,&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;or&lt;/p&gt;&#xA;&#xA;&lt;ol start=&quot;2&quot;&gt;&#xA;&lt;li&gt;Add noise to the feature space, but keeping its dimension. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Adding noise is not the same as changing the dimension of the feature space. If the data is linearly separable in the original feature space, it will be also separable although you add an extra random feature. Take a look at figure 1 and figure 2. Figure depicts the scatter plot (var1_1 vs var1_1) of a linear separable data in a one dimensional feature space. Figure 2 depicts the scatter plot of the same feature space with an extra random feature, now the dimension is 2, but the data is still linearly separable. You only have to look at the projection of the data in the var1_1 axis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure 1: Scatter plot of separable data in a 1-D feature space&#xA;&lt;a href=&quot;https://i.stack.imgur.com/mmkDz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mmkDz.png&quot; alt=&quot;Scatter plot of separable data in a 1-D feature space&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Figure 2:Scatter plot of separable data in a 2-D feature space, which is the same as the previous space plus an extra random feature&#xA;[&lt;img src=&quot;https://i.stack.imgur.com/mE9dw.png&quot; alt=&quot;Scatter plot of separable data in a 2-D feature space], which is the same as the previous space plus an extra random feature[2]&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to evaluate the robustness of your prediction model against noise, I will take option 1, since it not straightforward to derive what kind of noise to apply in the feature space. If you are working with images, you can blur them or if you are dealing with audio files, you can add white gaussian noise, or another kind of noise source, for example another mixing the original audio files with other sound sources.&lt;/p&gt;&#xA;" OwnerUserId="51714" LastActivityDate="2018-05-22T13:07:30.400" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31985" PostTypeId="1" AcceptedAnswerId="31990" CreationDate="2018-05-22T13:32:06.943" Score="1" ViewCount="349" Body="&lt;p&gt;I'm using a python script to log IO of a grid job.&lt;br&gt;&#xA;the log is formatted like this: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp;fullpath;event;size&#xA;1526994189.49;/tmp/folder/;IN_ISDIR;6&#xA;1526994189.49;/tmp/folder2/File;IN_ACCESS;36&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO.&#xA;But before doing that I need to insert the job ID and the jobname to the column making the file :  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;timestamp;fullpath;event;size;jobid;jobname&#xA;1526994189.49;/tmp/folder/;IN_ISDIR;6;123456;afakejobname&#xA;1526994189.49;/tmp/folder2/File;IN_ACCESS;36;123456;afakejobname&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both &lt;code&gt;jobname&lt;/code&gt; and &lt;code&gt;jobid&lt;/code&gt;&lt;br&gt;&#xA;or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just &lt;code&gt;jobname;jobid&lt;/code&gt; and put only the &lt;code&gt;jobid&lt;/code&gt; in my log.&lt;/p&gt;&#xA;" OwnerUserId="52527" LastActivityDate="2018-05-22T14:41:54.777" Title="parquet format: advise on log content" Tags="|apache-spark|data-formats|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31986" PostTypeId="2" ParentId="31923" CreationDate="2018-05-22T14:02:43.897" Score="3" Body="&lt;p&gt;I have added a dense layer to the network:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;model = Sequential()&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(LSTM(32, return_sequences=True))&#xA;model.add(Dropout(0.2))&#xA;model.add(TimeDistributed(Dense(5)))  # &amp;lt;--- This is new :)&#xA;model.add(Activation('linear')) &#xA;model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This has provided some improvement :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/5EuKl.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5EuKl.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These are two sine waves and the predictions of the last 50 data points of them.&#xA;As can be seen, the results have improved, with some of them being okay approximations of the sine waves. Other times... not so much.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Still a lot better than the previous results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I then tried another approach: Instead of having one (multilayered) LSTM, in which I ignore all but the 50 inputs (in a sense trying to predict from each one of the 50 last values the value located 50 places after it... which might not be an ideal approach) I split the network into 2 LSTMs, with one receiving the input and &quot;injecting&quot; its results via a dense layer into the second LTSM layer which generates the output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;features_num=5 &#xA;&#xA;model.add(LSTM(40, return_sequences=True, input_shape=(None, 5)))&#xA;model.add(LSTM(40, return_sequences=False))&#xA;&#xA;model.add(Dense(80))&#xA;model.add(Activation('tanh')) &#xA;model.add(RepeatVector(50)) # 50 output points&#xA;&#xA;model.add(LSTM(40, return_sequences=True))&#xA;model.add(LSTM(40, return_sequences=True))&#xA;&#xA;model.add(TimeDistributed(Dense(features_num)))&#xA;model.add(Activation('linear')) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Results are, for the most part, better:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/jUkKn.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jUkKn.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="43472" LastEditorUserId="43472" LastEditDate="2018-05-23T13:54:54.300" LastActivityDate="2018-05-23T13:54:54.300" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="31988" PostTypeId="1" CreationDate="2018-05-22T14:12:56.290" Score="2" ViewCount="268" Body="&lt;p&gt;I would like to test (predict) whether camera images are dusty/dirty or not.; a classification task indeed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is about finding the most appropriate Machine Learning approach, perhaps any suggestions for possible Deep Learning methods specifically tailored for this particular problem?&lt;/p&gt;&#xA;" OwnerUserId="52532" LastEditorUserId="44456" LastEditDate="2018-05-23T17:18:22.740" LastActivityDate="2018-05-23T17:18:22.740" Title="How Do I Classify Dirty/Dusty Camera Images?" Tags="|deep-learning|computer-vision|" AnswerCount="1" CommentCount="4" ClosedDate="2018-05-31T05:16:52.390" ContentLicense="CC BY-SA 4.0" />
  <row Id="31989" PostTypeId="1" CreationDate="2018-05-22T14:21:55.400" Score="5" ViewCount="1322" Body="&lt;p&gt;I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_2.append(sum(np.min(cdist(data,&#xA;                                      kmeanModel.cluster_centers_,&#xA;                                      'euclidean'),&#xA;                                axis = 1)) / data.shape[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/6vG2P.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another is to use inertia_ from sklearn.cluster.KMeans:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_3.append(kmeanModel.inertia_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/kGvjo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt; If I replace the normalisation factor / data.shape[0] with squared **2 as suggested below, then I still don't get the same as for the inertia plot:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_2.append(sum(np.min(cdist(data, &#xA;                                      kmeanModel.cluster_centers_, &#xA;                                      'euclidean'), &#xA;                                axis = 1)) ** 2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Using squared just makes the plot a little smoother, but definitely not the same as using intertia_, I'm just not quite sure how inertia_ is calculated and what it means.&lt;/p&gt;&#xA;" OwnerUserId="51845" LastEditorUserId="51845" LastEditDate="2019-01-23T11:23:25.883" LastActivityDate="2019-01-23T11:23:25.883" Title="What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?" Tags="|python|scikit-learn|clustering|k-means|" AnswerCount="1" CommentCount="3" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31990" PostTypeId="2" ParentId="31985" CreationDate="2018-05-22T14:41:54.777" Score="1" Body="&lt;p&gt;Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, you're using the features available in spark to optimize it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reference: &lt;a href=&quot;https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="13370" LastActivityDate="2018-05-22T14:41:54.777" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31991" PostTypeId="1" CreationDate="2018-05-22T14:50:53.630" Score="4" ViewCount="987" Body="&lt;p&gt;The Tensorflow's documentation of &lt;em&gt;softmax_cross_entropy_with_logits&lt;/em&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Measures the probability error in discrete classification tasks in&#xA;  which the classes are mutually exclusive (each entry is in exactly one&#xA;  class). For example, each CIFAR-10 image is labeled with one and only&#xA;  one label: an image can be a dog or a truck, but not both.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;NOTE: While the classes are mutually exclusive, their probabilities&#xA;  need not be. All that is required is that each row of labels is a&#xA;  valid probability distribution. If they are not, the computation of&#xA;  the gradient will be incorrect.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;At first glance it may seen contradictory, but my guess is that, provided these conditions for the classes and for the labels, the probability distribution of the labels have always one maximum. For instance, labels = [0.5 0.5] is a valid probability distribution in a binary case but it does not comply with the exclusivity of the classes. If I'm wrong, I don't get the idea behind the documentation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also pointed out that for mutually elusive probabilities, &lt;em&gt;sparse_categorical_cross_entropy&lt;/em&gt; should be used, but I think that &lt;em&gt;softmax_cross_entropy_with_logits&lt;/em&gt; can be used if the labels are one-hot encoded, which is a valid probability distribution (a deterministic one)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When going around this question again, I'm wondering if in a multi-class exclusive case where the only constraint on the labels are that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1... &lt;/p&gt;&#xA;" OwnerUserId="51714" LastEditorUserId="28175" LastEditDate="2018-05-25T14:44:34.597" LastActivityDate="2018-05-25T14:44:34.597" Title="What does it mean that classes are mutually exlcusive but soft-labels are accepeted?" Tags="|tensorflow|multiclass-classification|multilabel-classification|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="31992" PostTypeId="1" CreationDate="2018-05-22T15:02:21.550" Score="2" ViewCount="95" Body="&lt;p&gt;I'm trying to figure out how to decrease the error in my LSTM. It's an odd use-case because rather than classifying, we are taking in short lists (up to 32 elements long) and outputting a series of real numbers, ranging from -1 to 1 - representing angles. Essentially, we want to reconstruct short protein loops from amino acid inputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the past we had redundant data in our datasets, so the accuracy reported was incorrect. Since removing the redundant data our validation accuracy has gotten much worse, which suggests our network had learned to memorise the most frequent examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our dataset is 10,000 items, split 70/20/10 between train, validation and test. We use a bi-directional, LSTM as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = tf.cast(tf_train_dataset, dtype=tf.float32)&#xA;output_size = FLAGS.max_cdr_length * 4&#xA;dmask = tf.placeholder(tf.float32, [None, output_size], name=&quot;dmask&quot;)&#xA;keep_prob = tf.placeholder(tf.float32, name=&quot;keepprob&quot;)&#xA;sizes = [FLAGS.lstm_size,int(math.floor(FLAGS.lstm_size/2)),int(math.floor(FLAGS.lstm_size/     4))]&#xA;single_rnn_cell_fw = tf.contrib.rnn.MultiRNNCell( [lstm_cell(sizes[i], keep_prob, &quot;cell_fw&quot; + str(i)) for i in range(len(sizes))])&#xA;single_rnn_cell_bw = tf.contrib.rnn.MultiRNNCell( [lstm_cell(sizes[i], keep_prob, &quot;cell_bw&quot; + str(i)) for i in range(len(sizes))])&#xA;&#xA;length = create_length(x)&#xA;initial_state = single_rnn_cell_fw.zero_state(FLAGS.batch_size, dtype=tf.float32)&#xA;initial_state = single_rnn_cell_bw.zero_state(FLAGS.batch_size, dtype=tf.float32)&#xA;&#xA;outputs, states = tf.nn.bidirectional_dynamic_rnn(cell_fw=single_rnn_cell_fw, cell_bw=single_rnn_cell_bw, inputs=x, dtype=tf.float32, sequence_length = length)&#xA;output_fw, output_bw = outputs&#xA;states_fw, states_bw = states&#xA;output_fw = last_relevant(FLAGS, output_fw, length, &quot;last_fw&quot;)&#xA;output_bw = last_relevant(FLAGS, output_bw, length, &quot;last_bw&quot;)&#xA;output = tf.concat((output_fw, output_bw), axis=1, name='bidirectional_concat_outputs')&#xA;&#xA;test = tf.placeholder(tf.float32, [None, output_size], name=&quot;train_test&quot;)&#xA;W_o = weight_variable([sizes[-1]*2, output_size], &quot;weight_output&quot;)&#xA;b_o = bias_variable([output_size],&quot;bias_output&quot;)&#xA;y_conv = tf.tanh( ( tf.matmul(output, W_o)) * dmask, name=&quot;output&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Essentially, we use 3 layers of LSTM, with 256, 128 and 64 units each. We take the last step of both the Forward and Backward passes and concatenate them together. These feed into a final, fully connected layer that presents the data in the way we need it. We use a mask to set these steps we don't need to zero. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our cost function uses a mask again, and takes the mean of the squared difference. We build the mask from the test data. Values to ignore are set to -3.0.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def cost(goutput, gtest, gweights, FLAGS):&#xA;    mask = tf.sign(tf.add(gtest,3.0))&#xA;    basic_error = tf.square(gtest-goutput) * mask&#xA;    basic_error = tf.reduce_sum(basic_error)&#xA;    basic_error /= tf.reduce_sum(mask)&#xA;    return basic_error&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;To train the net I've used a variety of optimizers. The lowest scores have been obtained with the AdamOptimizer. The others, such as Adagrad, Adadelta, RMSProp tend to flatline around 0.3/0.4 error which is not particularly great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Our learning rate is 0.004, batch size of 200. We use a 0.5 probability dropout layer. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've tried adding more layers, changing learning rates, batch sizes, even the representation of the data. I've attempted batch regularisation, L1 and L2 weight regularisation (though perhaps incorrectly) and I've even considered switching to a convnet approach instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Nothing seems to make any difference. What has seemed to work is changing the optimizer. Adam seems noisier as it improves, but it does get closer than the other optimizers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xA44P.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xA44P.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We need to get down to a value much closer to 0.05 or 0.01. Sometimes the training error touches 0.09 but the validation doesn't follow. I've run this network for about 500 epochs so far (about 8 hours) and it tends to settle around 0.2 validation error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not quite sure what to attempt next. Decayed learning rate might help but I suspect there is something more fundamental I need to do. It could be something as simple as a bug in the code - I need to double check the masking, &lt;/p&gt;&#xA;" OwnerUserId="52531" LastActivityDate="2018-05-22T15:02:21.550" Title="Breaking through an accuracy brickwall with my LSTM" Tags="|machine-learning|neural-network|lstm|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31994" PostTypeId="1" CreationDate="2018-05-22T16:37:01.727" Score="4" ViewCount="13874" Body="&lt;p&gt;I trained a faster-rcnn model on the tensorflow object detection API on a custom dataset. I found that the loss is ~2 after 3.5k steps. However, when I ran eval.py, the mAP scores are all almost 0 as shown below.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/CGbEf.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CGbEf.png&quot; alt=&quot;Category-wise mAP scores on Tensorboard&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I do not understand why this is the case. However, when I look at the images at 3.5k steps, the model has captured some of the boxes as shown below&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zMZ7F.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zMZ7F.jpg&quot; alt=&quot;output of the Faster-RCNN model after 3.5k iterations&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Can someone please explain why the mAP scores are close to zero, even though the model has learned to output quite a few boxes?&lt;/p&gt;&#xA;" OwnerUserId="34017" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-10-07T17:00:10.407" Title="mAP scores on tensorboard (Tensorflow Object Detection API) are all 0 even though the loss value is low" Tags="|deep-learning|tensorflow|computer-vision|object-detection|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31995" PostTypeId="1" CreationDate="2018-05-22T17:03:44.643" Score="2" ViewCount="531" Body="&lt;p&gt;I would like to train a network to conjugate verbs and recognize verb conjugations. For example:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Given &quot;To eat, 1st person, past continuous&quot;, the model would return &quot;I was eating&quot;&lt;/li&gt;&#xA;&lt;li&gt;Given &quot;I was eating&quot;, the model would return &quot;To eat, 1st person, past continuous&quot;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The benefit of training would be to have the model conjugate and recognize verbs it hasn't seen previously.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to train the network on standard conjugation tables such as &lt;a href=&quot;http://conjugator.reverso.net/conjugation-english-verb-eat.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://conjugator.reverso.net/conjugation-english-verb-eat.html&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this within the realm of possibilities with our present machine learning technologies? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: If yes (and it seems to be the case), hints for suggested architectures or actual implementations would be most welcome. &lt;/p&gt;&#xA;" OwnerUserId="52538" LastEditorUserId="52538" LastEditDate="2018-05-23T16:08:47.133" LastActivityDate="2018-05-23T16:08:47.133" Title="Learning verb conjugation" Tags="|machine-learning|nlp|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="31996" PostTypeId="2" ParentId="29470" CreationDate="2018-05-22T17:52:38.183" Score="1" Body="&lt;p&gt;That is the real-world! in text analysis you usually end up with a extremely high-dimensional representation. First of all, &lt;a href=&quot;https://datascience.stackexchange.com/a/28516/8878&quot;&gt;I always recommend TF-IDF over original BoW&lt;/a&gt; (CountVectrizer).&#xA;To end up with a low-dimensional space, you may use &lt;a href=&quot;https://github.com/kasramsh/Feature-Engineering/blob/master/NSF%20Awards.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; naive approach which is not written efficient in computational way but the approach gives you an idea where you need to go. It will be more clear when I add explanations to the notebook soon.&lt;/p&gt;&#xA;" OwnerUserId="8878" LastActivityDate="2018-05-22T17:52:38.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="31997" PostTypeId="2" ParentId="5226" CreationDate="2018-05-22T18:09:07.990" Score="8" Body="&lt;p&gt;&lt;strong&gt;2018 Update!&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can create an embedding (dense vector) space for your categorical variables.  Many of you are familiar with word2vec and fastext, which embed words in a meaningful dense vector space.  Same idea here-- your categorical variables will map to a vector with some meaning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the &lt;a href=&quot;https://arxiv.org/abs/1604.06737&quot; rel=&quot;nofollow noreferrer&quot;&gt;Guo/Berkhahn paper&lt;/a&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Entity embedding not only reduces memory usage and speeds up neural&#xA;  networks compared with one-hot encoding, but more importantly by&#xA;  mapping similar values close to each other in the embedding space it&#xA;  reveals the intrinsic properties of the categorical variables. We&#xA;  applied it successfully in a recent Kaggle competition and were able&#xA;  to reach the third position with relative simple features.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The authors found that representing categorical variables this way improved the effectiveness of all machine learning algorithms tested, including random forest.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The best example might be &lt;a href=&quot;https://medium.com/the-graph/applying-deep-learning-to-related-pins-a6fee3c92f5e&quot; rel=&quot;nofollow noreferrer&quot;&gt;Pinterest's application of the technique&lt;/a&gt; to group related Pins:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/365pZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/365pZ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The folks at fastai have implemented categorical embeddings and created a very nice &lt;a href=&quot;http://www.fast.ai/2018/04/29/categorical-embeddings/&quot; rel=&quot;nofollow noreferrer&quot;&gt;blog post&lt;/a&gt; with companion &lt;a href=&quot;https://github.com/fastai/fastai/blob/master/courses/dl1/lesson3-rossman.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;demo notebook&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Additional Details and Explanation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A neural net is used to create the embeddings i.e. assign a vector to each categorical value. Once you have the vectors, you may use them in any model which accepts numerical values. Each component of vector becomes an input variable. For example, if you used 3-D vectors to embed your categorical list of colors, you might get something like: red=(0, 1.5, -2.3), blue=(1, 1, 0) etc. You would use three input variables in your random forest corresponding to the three components. For red things, c1=0, c2=1.5, and c3=-2.3. For blue things, c1=1, c2=1, and c3=0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You don't actually &lt;strong&gt;need&lt;/strong&gt; to use a neural network to create embeddings (although I don't recommend shying away from the technique).  You're free to create your own embeddings by hand or other means, when possible.  Some examples:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Map colors to RGB vectors.&lt;/li&gt;&#xA;&lt;li&gt;Map locations to lat/long vectors.&lt;/li&gt;&#xA;&lt;li&gt;In a U.S. political model, map cities to some vector components representing left/right alignment, tax burden, etc.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="1077" LastEditorUserId="1077" LastEditDate="2019-06-15T00:01:18.657" LastActivityDate="2019-06-15T00:01:18.657" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="31999" PostTypeId="2" ParentId="31973" CreationDate="2018-05-22T21:02:04.807" Score="2" Body="&lt;p&gt;Colorfulness, specifically managing overly gray output, was touched on in the &lt;a href=&quot;https://arxiv.org/abs/1611.07004&quot; rel=&quot;nofollow noreferrer&quot;&gt;pix2pix paper&lt;/a&gt;, down in Section 4.2. It might be worthwhile to try out their formulation of a conditional GAN, as their results indicate that their objective function was useful in generating a more diverse set of colors.&lt;/p&gt;&#xA;" OwnerUserId="50363" LastActivityDate="2018-05-22T21:02:04.807" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32001" PostTypeId="1" CreationDate="2018-05-22T21:25:38.233" Score="3" ViewCount="167" Body="&lt;p&gt;I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which  predictions are made is better than a low one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence:&#xA;$$\text{predicted class}_i = \begin{cases}&#xA; 1 &amp;amp;\text{ if } \hat{Y}_i &amp;gt; 0.5 + a \\ &#xA; 0 &amp;amp;\text{ if } \hat{Y}_i &amp;lt; 0.5 - a \\&#xA; \text{NA} &amp;amp;\text{else}&#xA;\end{cases} \\&#xA;\text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To tune the hyperparameters (including $a$), I have designed a metric that depends positively on: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Test-sample accuracy (only counting predictions different from NA)&lt;/li&gt;&#xA;&lt;li&gt;Percentage of predictions that are different from NA. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.&lt;/p&gt;&#xA;" OwnerUserId="52546" LastEditorUserId="28175" LastEditDate="2018-05-23T12:38:46.777" LastActivityDate="2018-05-23T12:38:46.777" Title="Neural Networks - Strategies for problems with high Bayes error rate" Tags="|machine-learning|neural-network|deep-learning|bayes-error|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32003" PostTypeId="2" ParentId="17484" CreationDate="2018-05-22T23:36:19.270" Score="0" Body="&lt;p&gt;I have an answer now for my question.&#xA;I will share briefly the main steps / technologies I used to deploy the model in production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using Python programming language.&#xA;After training and generating valid models I wrote a restful api using Python programming language and flask.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using flask you can write a restful api. Three important points:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- It is very important to give attention to where you will define the model architecture/initialize the parameters/ define the session.&#xA;Avoid doing this each time you call the restful api. this will be very expensive.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Flask provide a very good mechanism to run servers in production environment.&#xA;Read about flask + wcgi &#xA;Avoid runing the server code (the resful api) directly, in this case you will not have direct and full control. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3- Watch the memory and the cpu usage, make sure to limit the maximum number of instances that can run in parallel. Remember these models can take a lot from the memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;unfortunately, I can not share codes with public. But hopefully my answer can give an idea about how to do it in production.&lt;/p&gt;&#xA;" OwnerUserId="17696" LastEditorUserId="17696" LastEditDate="2018-05-23T00:03:56.547" LastActivityDate="2018-05-23T00:03:56.547" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32004" PostTypeId="2" ParentId="19966" CreationDate="2018-05-22T23:51:42.040" Score="0" Body="&lt;p&gt;I'm doing something similar and found that the best approach has to be comprehensive. My process obtains a ranking of signals based on degree and measure of anomaly, both as a function of 'bads' (if available). If information on 'bads' is unavailable then I perform the same study but as a function of the degree of anomaly. In both cases, I perform hierarchical clusters. These are helpful in grouping signals that have similar behavior.&lt;/p&gt;&#xA;" OwnerUserId="50083" LastActivityDate="2018-05-22T23:51:42.040" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32005" PostTypeId="1" CreationDate="2018-05-23T00:50:02.303" Score="11" ViewCount="177" Body="&lt;p&gt;I would like to train my LSTM with a &quot;synthetic gradients&quot; &lt;a href=&quot;https://arxiv.org/pdf/1608.05343.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Decoupled Neural Interface (DNI).&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to decide on the number of layers and neurons for my DNI?&#xA;  Searching for them by trial end error or what's worse - by Genetic&#xA;  algorithm which would seem to outweigh [sic] defeat the purpose of Synthetic Gradients. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;And, if my DNI is an LSTM itself - it seems it would take even longer&#xA;  to determine its optimal structure&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;SG speed up the training speed, by allowing multiple forward passes (with immediate weight adjustments), since DNI will already predict the future gradient.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, we will lose time &quot;experiencing&quot; a few hundreds of training sessions only to find a optimal structure of DNI with which it will predict gradient the best way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By that time we could have already finished our training with an oldschool Backprop through Time.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="91826" LastEditDate="2020-06-16T07:35:19.113" LastActivityDate="2020-07-17T23:02:05.163" Title="Synthetic Gradients good number of Layers &amp; neurons" Tags="|backpropagation|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32006" PostTypeId="1" AcceptedAnswerId="32021" CreationDate="2018-05-23T02:12:16.930" Score="1" ViewCount="162" Body="&lt;p&gt;I've been working through the Q-Network learning example in this &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0&quot; rel=&quot;nofollow noreferrer&quot;&gt;Arthur Juliani's blog&lt;/a&gt;. It's based on the pretty trivia &lt;a href=&quot;https://gym.openai.com/envs/FrozenLake-v0/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Open Gym Frozen Lake&lt;/a&gt; example. It's base implementation get's about 47% success rate over 3000 iterations. I decided to add a bias to the implementation, and found that it severely harmed the results to no better than random.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is, I added the bias term below:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)&#xA;bias = tf.Variable(tf.zeros(shape=[1,4]))&#xA;W = tf.Variable(tf.random_uniform([16,4],0,0.01))&#xA;Qout = tf.matmul(inputs1,W) + bias&#xA;predict = tf.argmax(Qout,1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The rest of the code is identical to the original solution. Any ideas why this would so negatively affect performance?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;&#xA;It looks like &lt;a href=&quot;https://medium.com/@awjuliani/yep-the-bias-is-the-main-issue-ee19bf24190c&quot; rel=&quot;nofollow noreferrer&quot;&gt;someone else ran into this issue&lt;/a&gt;, and the answer given was that &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Having a bias term with the one-hot encoding prevents each state’s Q&#xA;  values from being independent&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Any ideas why this is the case? The bias is added after the multiplication, so it's in the dimension of the actions, not the inputs. I don't see why this would make learning fail.&lt;/p&gt;&#xA;" OwnerUserId="52553" LastEditorUserId="52553" LastEditDate="2018-05-24T02:14:27.923" LastActivityDate="2018-05-24T06:52:13.893" Title="Adding a bias makes Q-learning algorithm ineffective" Tags="|tensorflow|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32007" PostTypeId="2" ParentId="27412" CreationDate="2018-05-23T02:38:15.393" Score="3" Body="&lt;p&gt;This is in fact a very well suited problem for machine learning and has been explored in the past as published &lt;a href=&quot;https://arxiv.org/ftp/arxiv/papers/1504/1504.02945.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. It is often called the cocktail party problem, because of the way our human brains can focus on a very particular voice pattern despite being drowned by countless other voices all around us. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can think of two ways by which this problem can be approached the first is to transform the spectrogram to only keep the birds chirping. The second, would be to extract the voice patterns from a sample and then substract the extracted voices from the original sample. This second approach would not remove the noise from the truck.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would go with the former technique. I would get a few thousands clips of birds chirping. Then introduce some other sounds into it. The mixed sample would be my input $X$ and the target $Y$ would be the clean bird sample. This was you can simulate many instances. However, you will need enough bird chirping data to cover the wide distribution which exists for birds making noise.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Q1&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;For this to work well, you will need A LOT of data. Yes, you should definitely try to get as many samples from wherever you can. You can then randomly introduce these human voices with the bird clips in order to get the dirty signals, then use these to train your model. &lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Q2&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;You most likely will not achieve 100% accuracy with existing machine learning methods. However, if you get close to 90% that is already a miracle and will truly be great. If you are using spectrograms I would first try &lt;strong&gt;convolutional neural network (CNN)&lt;/strong&gt;. These have been shown to produce very good results &lt;a href=&quot;https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.27096&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. The same paper then proposes an &lt;strong&gt;autoencoder&lt;/strong&gt; to clean up the spectrograms. I think this has a high probability of working, but will introduce smoothing to the spectrogram which may distort the birds chirping. Lastly, I would suggest using a &lt;strong&gt;generative adversarial network (GAN)&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Q3&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I would look into the &lt;strong&gt;cocktail party problem&lt;/strong&gt;. You can also get some insight as to what techniques to use by looking at the extensive work that has been done on spectrograms in MRI spectroscopy (MRSI).&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Q4&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Like I said above I would suggest going from a $n \times n$ sample containing a mixture of sounds to a clean $n \times n$ sample with only the bird chirping. I think this strategy is the one where you will find the most existing literature and many groups have shown it to work.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-23T02:38:15.393" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32008" PostTypeId="2" ParentId="31989" CreationDate="2018-05-23T02:51:12.740" Score="2" Body="&lt;p&gt;You will notice that the inertia is the sum of squared distance between each point and its nearest cluster center. Thus by removing the normalization term &lt;code&gt;/ data.shape[0]&lt;/code&gt; and instead replacing that by a squared &lt;code&gt;**2&lt;/code&gt; these two expressions will be equivalent&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_2.append(sum(np.min(cdist(data&#xA;                              , kmeanModel.cluster_centers_&#xA;                              , 'euclidean')&#xA;                              , axis = 1)) **2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_2.append(kmeanModel.inertia_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;For example using this artificial data source, where we generate 5 Gaussian distributions with 300 points in each.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = [[[ 0,1],  [ 0,1]], &#xA;          [[ 5,1],  [ 5,1]], &#xA;          [[-2,5],  [ 2,5]],&#xA;          [[ 2,1],  [ 2,1]],&#xA;          [[-5,1],  [-5,1]]]&#xA;&#xA;n = 300&#xA;dims = len(params[0])&#xA;&#xA;data = []&#xA;y = []&#xA;for ix, i in enumerate(params):&#xA;    inst = np.random.randn(n, dims)&#xA;    for dim in range(dims):&#xA;        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]&#xA;        label = ix + np.zeros(n)&#xA;&#xA;    if len(data) == 0: data = inst&#xA;    else: data = np.append( data, inst, axis= 0)&#xA;    if len(y) == 0: y = label&#xA;    else: y = np.append(y, label)&#xA;&#xA;num_clusters = len(params)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/4RpTO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will then apply KMeans using different number of clusters&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;k = [1,2,3,4,5,6,7,8,9,10]&#xA;&#xA;inertias = []&#xA;dists = []&#xA;&#xA;for i in k:&#xA;    kmeans = KMeans(i)&#xA;    kmeans.fit(data)&#xA;    inertias.append(kmeans.inertia_)&#xA;    dists.append(sum(np.min(spatial.distance.cdist(data, kmeans.cluster_centers_, 'euclidean'), axis=1)**2))&#xA;&#xA;plt.plot(range(1, len(inertias)+1), inertias, label = 'Inertia')&#xA;plt.plot(range(1, len(dists)+1), dists, label = 'Distance')&#xA;plt.legend()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/9dKLJ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="51845" LastEditDate="2018-05-25T00:50:08.343" LastActivityDate="2018-05-25T00:50:08.343" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32009" PostTypeId="1" AcceptedAnswerId="32019" CreationDate="2018-05-23T03:46:16.723" Score="5" ViewCount="199" Body="&lt;p&gt;I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the &lt;strong&gt;general problem&lt;/strong&gt; of composing by machine in known musical styles. I'm more interested in the &lt;strong&gt;specific problem&lt;/strong&gt; of using a computer to assist myself in composing... to speed up certain repetitive tasks, or to delight me with unexpected ideas. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First&lt;/strong&gt;, I want to design ML that can make a good/bad categorization on a &lt;strong&gt;single short passage.&lt;/strong&gt; (Not looking to generate music at first.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's pretty easier to decompose such a passage into significant features. I can say things like&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;maybe there are particular &quot;intervals&quot; (harmony) present, and others absent&lt;/li&gt;&#xA;&lt;li&gt;maybe notes are spread out evenly from low to high, or maybe not&lt;/li&gt;&#xA;&lt;li&gt;maybe notes are clustered around a certain register rather than spread out&lt;/li&gt;&#xA;&lt;li&gt;maybe the rhythm is particularly uneven&lt;/li&gt;&#xA;&lt;li&gt;maybe a particular note occurs a lot&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I can easily write algorithms that produce a list of such features for any given passage, and I can even be pretty sure that I'm including all of the relevant features. For an &quot;art form,&quot; music is surprisingly amenable to analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The trick is, first of all, there will be hundreds of observations, and second, that no one of these observations is good or bad in itself. Say we have three factors A, B, and C. A is not bad alone, but perhaps when A and B occur together, that's bad. But then maybe if A, B, and C all occur together, it becomes good again! And then maybe a fourth factor D, changes the context so that A, B, and C no longer matter much. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This seems like an ML problem, but I wonder what the best approach is? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can &lt;strong&gt;construct or compose hundreds of examples of music labeled good and bad, and identify hundreds of &quot;features&quot; in each example.&lt;/strong&gt; Some of the features are &quot;present/not present&quot; categories, while others are numbers that can be scaled from 0 to 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a general and simple learning algorithm that even an ML novice such as myself (but I'm an experienced programmer) can implement?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to generate my own training examples and would be limited to around 300 to 1000, as I'm training it on &lt;strong&gt;simple passages&lt;/strong&gt; in &lt;strong&gt;my style.&lt;/strong&gt; This may actually be good problem to have... my job is not that complex compared to general musical intelligence. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will contrast this task with image recognition. An image has millions of pixels and the features are spread out, occurring in endless  permutations. It makes my head hurt to try to figure out what's going on in a deep neural net.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, in music, there will be at most a few hundred features. The patterns will &lt;strong&gt;not&lt;/strong&gt; occur in endless permutations. In fact, in stripped-down situations my wild guess is that you could make an 80% accurate classifier by looking at 10 features (assuming you have the weights and combinations well-trained). I'm aiming somewhat higher than that (I need more features and a bit more accurate).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To give you some idea of my data set:&lt;/strong&gt; I'm in the early stages of writing the software to extract the features, but I have a pretty good mental idea of what I'm aiming for (been composing by hand for years and years). To make this more concrete I created a web page showing what the data will look like. This is very simple and just totally made up. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://theunexpectedpearl.com/dataset.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://theunexpectedpearl.com/dataset.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This shows four rows. Each row is an example passage of music that I will compose or generate automatically. Each row is classified &quot;good&quot; or &quot;bad&quot; &lt;strong&gt;by my own ear.&lt;/strong&gt; The rest of the row is a long list of features, each of which is either &quot;yes/no&quot; or a single number scaled from 0.0 to 1.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't think a Naive Bayes classifier will work for this (if I understand correctly) because features are neither good nor bad in and of themselves... it is only combinations of them (context) that matter. In fact, the very same feature can contribute to &quot;goodness&quot; in one context, and be an immediate strong indication of &quot;bad&quot; in another context.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data could possibly be structured differently. In practice, this spreadsheet would be &quot;sparse&quot; and maybe features should be grouped in some way, or arranged in a tree.&lt;/p&gt;&#xA;" OwnerUserId="52555" LastEditorUserId="52555" LastEditDate="2018-05-23T06:52:19.127" LastActivityDate="2018-05-23T06:52:19.127" Title="Selecting ML algorithm for music composition" Tags="|machine-learning|classification|categorical-data|" AnswerCount="2" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="32010" PostTypeId="2" ParentId="32009" CreationDate="2018-05-23T04:21:00.170" Score="-1" Body="&lt;p&gt;I'll offer up a few suggestions to get the creative juices flowing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First, since you have two tasks, you could consider two models configured as a Generative Adversarial Network where&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;One model generates musical passages&lt;/li&gt;&#xA;&lt;li&gt;One model classifies musical passages&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The output from the generative model is then evaluated by the classifier.  The output from the classifier can then be used to improve (train) the generative model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I imagine these would be trained on the same labeled data set.  I would use a Recurrent Neural Network to train the classifier and tune the snot out of it (so to speak).&#xA;I would use a Generative Neural Network for the generative model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you were looking for more direction.  If so, sorry to disappoint.  I figured you may have been looking for better ways to search for help.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Best of luck.&lt;/p&gt;&#xA;" OwnerUserId="23159" LastActivityDate="2018-05-23T04:21:00.170" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32011" PostTypeId="2" ParentId="24093" CreationDate="2018-05-23T04:45:39.760" Score="13" Body="&lt;p&gt;First export environment configuration of your current conda environment using:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda-env  export -n your_env_name &amp;gt; your_env_name.yml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda-env  export -n base&amp;gt; base.yml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;After running above command their should be yml configuration file in your current directory which contain information of your conda environment&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To create new environment using yml configuration file run:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda-env create -n new_env -f=\path\to\base.yml &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda-env create -n venv -f=base.yml&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In case the above one does not work (due to the various issues of conda itself), it's always worth a try with the following variation:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;conda-env create --name new_env --file \path\to\base.yml &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52557" LastEditorUserId="14379" LastEditDate="2019-06-19T21:16:16.187" LastActivityDate="2019-06-19T21:16:16.187" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32012" PostTypeId="1" CreationDate="2018-05-23T04:49:00.577" Score="1" ViewCount="53" Body="&lt;p&gt;I have some customer (say 10,000 customers) transaction data of many stores, what would be a good approach to rank these stores? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The transaction data is of entire year, and the transaction data including transaction amount, type of store, location of store ... etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And how would one evaluate such ranking algorithm? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2021-10-04T23:25:36.120" Title="business ranking based on transaction data" Tags="|ranking|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32013" PostTypeId="2" ParentId="31949" CreationDate="2018-05-23T05:24:33.873" Score="1" Body="&lt;p&gt;As you might know, biases are used in the CNN only when there is no &lt;a href=&quot;https://www.youtube.com/watch?v=dXB-KQYkzNU&quot; rel=&quot;nofollow noreferrer&quot;&gt;batch normalization&lt;/a&gt; layer is included because this automatically adds a type of bias to the activation outputs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, this is what I understood:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us assume a layer of CNN (without batch norm) with 50 kernels each of size of 5 x 5. So, total weight updates will be 5 x 5 x 50 = 1250 and total biases updates will be 50 (a bias for each kernel).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the part &quot;...people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector...&quot; :-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If you merge all these parameters into one single vector and randomly sample the parameters to check gradients for, you may end up checking very few or none of the gradient updates for biases as they make a very less portion of that vector (only 50 out of 1300).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;For the part &quot;One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter...&quot; :-&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;On the other hand, you can choose some kernels and take all parameters (weights and biases) of only those kernels to check gradient for and so in this case you will definitely be checking gradients of some biases which they say is a good way of doing gradient checking.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2018-05-23T05:24:33.873" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32014" PostTypeId="1" CreationDate="2018-05-23T05:26:41.423" Score="1" ViewCount="16" Body="&lt;p&gt;I want to access social media advertisements and information about how popular they were ( how many likes, comments and shares they received and the comments). Can I get these information using &lt;a href=&quot;https://datascience.stackexchange.com/questions/422/publicly-available-social-network-datasets-apis?newreg=eeb7ba99c0844fd2969a7f49da720bcb&quot;&gt;these APIs&lt;/a&gt; ? Or else where can I get this data? &lt;/p&gt;&#xA;" OwnerUserId="52559" LastActivityDate="2018-05-23T05:26:41.423" Title="Can we access social media advertisements and information like how many likes, comments, shares they received using their APIs?" Tags="|dataset|social-network-analysis|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32015" PostTypeId="1" CreationDate="2018-05-23T05:36:20.633" Score="2" ViewCount="125" Body="&lt;p&gt;What are the possible approaches when we need to train a model, but the training dataset is really small? (Assuming we have a lot of data, just not many data are labeled)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know a library from Stanford: &lt;a href=&quot;https://hazyresearch.github.io/snorkel/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://hazyresearch.github.io/snorkel/&lt;/a&gt; that can generate training labels based on a some pre-determined experts rules. (Side question, anyone happen to know what's the underlining mathematics/statistics of this library?)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am wondering in the scenarios when the snorkel packages can not be used, what are the approaches to label more data for training? Could Maximum Likelihood Estimator be used here? How would one implement such algorithm for labelling the data for training?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW, I am looking for a mathematical approach, not a brute-force one like using Amazon Mechanical Turk.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastEditorUserId="17310" LastEditDate="2018-05-23T06:04:00.530" LastActivityDate="2018-05-23T12:17:19.560" Title="Mathematical algorithms to generate labels for training data" Tags="|machine-learning|training|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32016" PostTypeId="1" CreationDate="2018-05-23T06:04:59.210" Score="6" ViewCount="1116" Body="&lt;p&gt;This is embarrassing but I think I miss understand something.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;In multinomial distribution, &quot;while the trials are independent, their outcomes X are dependent because they must be summed to n.&quot; &lt;a href=&quot;https://en.wikipedia.org/wiki/Multinomial_distribution&quot; rel=&quot;noreferrer&quot;&gt;wiki&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Naive Bayes assumption is that the features are independent (given the class): &quot;assume that each feature $x_{i}$ is conditionally independent of every other feature&quot; &lt;a href=&quot;https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes&quot; rel=&quot;noreferrer&quot;&gt;wiki&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thus, how can 1&amp;amp;2 co-exist in what's known as the Multinomial Naive-Bayes model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, $p(x_i|c)=p_i^{x_i}(1-p_i)^{n-x_i}\binom{n}{k}$ and $p(x_1,...,x_n)=\frac{n!}{\prod_i{x_i}!}\prod{p_i^{x_i}}$ thus $p(x_1,...,x_n) \ne \prod_ip(x_i|c) $&lt;/p&gt;&#xA;" OwnerUserId="21596" LastEditorUserId="21596" LastEditDate="2018-05-23T08:52:41.493" LastActivityDate="2022-02-07T23:35:50.793" Title="Naive Bayes Multinomial, independence assumption misunderstood" Tags="|naive-bayes-classifier|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32017" PostTypeId="2" ParentId="32015" CreationDate="2018-05-23T06:08:07.020" Score="3" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Assuming we have a lot of data, just not many data are labeled.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Try to tackle the problem first by using an unsupervised approach, following that, use the learned features for any downstream task.&lt;br&gt;&#xA;see &lt;a href=&quot;http://www.deeplearningbook.org/contents/generative_models.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deep Generative Models&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;generate training labels based on some pre-determined experts rules  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you have some function that can generate the data distribution then why you bother yourself with such finding yet another function that approximate the former.&lt;br&gt;&#xA;you could use this approach for evaluation purposes, e.g. compare different models regarding the problem being solved.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What are the approaches to label more data for training  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;you could aggregate (average) different models that tackle the same problem but with a different dataset. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Could Maximum Likelihood Estimator be used here.  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As I said, we could maximize the likelihood of the data, giving high probability to samples that are very similar to the training data is very useful at exploiting the internal structure (learning the data manifold) that could potentially disentangle the factors that are responsible for generating the data which presumably part of them could be a direct signal for predicting the target y that you are trying to predict.   &lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-23T06:08:07.020" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32018" PostTypeId="2" ParentId="32016" CreationDate="2018-05-23T06:27:37.460" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Their outcomes X are dependent because they must be summed to n  &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you are really want to model multiple experiments probabilistically, please do not forget the normalization constant that make the unnormalized quantity normalized by integrating/summing them to one.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Multinomial Naive Bayes model, as described by Manning et al (2008), estimates the conditional probability of a particular word/term/token given a class as the relative frequency of term t in documents belonging to class c:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A48SQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A48SQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus this variation takes into account the number of occurrences of term t in training documents from class c, including multiple occurrences &lt;strong&gt;Not multiple experiments.&lt;/strong&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;see &lt;a href=&quot;http://blog.datumbox.com/machine-learning-tutorial-the-naive-bayes-text-classifier/&quot; rel=&quot;nofollow noreferrer&quot;&gt;The Naive Bayes Text Classifier&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="44039" LastActivityDate="2018-05-23T06:27:37.460" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="32019" PostTypeId="2" ParentId="32009" CreationDate="2018-05-23T06:51:02.287" Score="1" Body="&lt;p&gt;Let us formulate this problem in such a way that it can be understood from a machine learning perspective. You have a set of instances $X$ where each instance $x_i \in \mathbb{R}^m$ where $m$ is the dimensionality of the instance. In other words $m$ is the number of features that describe the instance. Your problem intends to go from a set of features to a class label &lt;strong&gt;good&lt;/strong&gt; or &lt;strong&gt;bad&lt;/strong&gt;. Thus, this is a mapping from $\mathbb{R}^m$ to $y \in \{0, 1\}$.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How to achieve this mapping?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is when we will use the machine learning algorithms. We will train a model to effectively approximate the function which gives the output label from a set of inputs. It is evident that sparse features (low information entropy) will complicate the mapping function and will thus provide worse results. This is why feature engineering is of upmost importance for machine learning. It is probably the hardest part of the machine learning pipeline, however it is the lead factor in dictating your results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can use some feature reduction techniques in order to remove features which are uninformative with respect to the output label. Some techniques that I use frequently are principle component analysis (PCA), linear discriminant analysis (LDA). Alternatively, you can use some projection methods to reduce the dimensionality of the data whilst maintaining separation between the classes. Such techniques are Isomap, MDS, Spectral Embeddings and TSNE. You can check to see which is best suited for your type of data.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;How to choose a model?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Firstly, your problem is a supervised classification problem. This already narrows the types of models you can use. Furthermore, model selection is based on some key factors such as: the number of instances you have, the number of features per instance and the number of output nodes. You should also keep in consideration that the separability of the probability distribution between the output classes will impact the performance of the model directly. For example discriminating between cars and oranges is much easier than oranges and clementines.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you have 1,000 instances and around 13 features. This means that deep learning based techniques are possible but discouraged. You do not have enough data. You can then attempt the following popular classification models&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Support Vector Classifier &lt;/li&gt;&#xA;&lt;li&gt;Naive Bayes &lt;/li&gt;&#xA;&lt;li&gt;K-Nearest Neighbors &lt;/li&gt;&#xA;&lt;li&gt;Decision Trees &lt;/li&gt;&#xA;&lt;li&gt;Random Forests&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;To evaluate which model performs the best you will use the accuracy attained with a trained model on a test set. This set should be drawn independently from the training set as to catch overfitting. This is when the model cannot generalize to new data. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;In code&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Assuming matrix $X$ contains the data where rows are the instances and columns are the features, and matrix $Y$ contains the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First we split our data into a training and testing set &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)&#xA;&#xA;from sklearn.svm import SVC&#xA;clf = SVC()&#xA;clf.fit(X_train, y_train)&#xA;print('Score: ', clf.score(X_test, y_test))&#xA;&#xA;from sklearn.neighbors import KNeighborsClassifier&#xA;neigh = KNeighborsClassifier(n_neighbors=3)&#xA;neigh.fit(X_train, y_train)&#xA;print('Score: ', neigh.score(X_test, y_test))&#xA;&#xA;from sklearn import tree&#xA;clf = tree.DecisionTreeClassifier()&#xA;clf.fit(X_train, y_train)&#xA;print('Score: ', clf.score(X_test, y_test))&#xA;&#xA;from sklearn.ensemble import RandomForestClassifier &#xA;forest = RandomForestClassifier(n_estimators = 100)&#xA;forest.fit(X_train, y_train)&#xA;print('Score: ', forest.score(X_test, y_test))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This should be a starting point. Let us know if you fall into any problems, and let us know what accuracy you are getting we can then look deeper into these models and better suit them to your data source. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-23T06:51:02.287" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32021" PostTypeId="2" ParentId="32006" CreationDate="2018-05-23T07:22:17.173" Score="2" Body="&lt;blockquote&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Having a bias term with the one-hot encoding prevents each state’s Q values from being independent&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Any ideas why this is the case?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;The usual point of using linear regression or other function approximation in Q-learning is to generalise, and thus prevent the Q values from being independent deliberately. So this is not a general statement about Q-learning and linear function approximation. In general yes you can add bias terms - also, in general when using linear regression or neural networks to estimate action values, then you &lt;em&gt;should&lt;/em&gt; have bias terms.&lt;/p&gt;&#xA;&lt;p&gt;Having said that, I would expect the bias should make little difference here, since the one hot coding of state makes it redundant - you can already express any mapping of state to Q value of each possible action using just the weights. Although you are using what looks like a linear regression model, essentially this is just the tabular form of Q learning.&lt;/p&gt;&#xA;&lt;p&gt;Q learning with function approximation can be unstable. The combination of off-policy, bootstrap updates and function approximation is known as the &amp;quot;deadly triad&amp;quot;, and you need to add tricks like experience replay to keep it stable. Although in theory the linear regression should learn a correct bias, probably adding bias puts the algorithm into that unstable zone.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-24T06:52:13.893" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32022" PostTypeId="2" ParentId="29480" CreationDate="2018-05-23T07:34:30.930" Score="0" Body="&lt;p&gt;You can upload stuff to Google Drive and then download it from there on Colab. I've written some utils for that - see &lt;a href=&quot;https://github.com/lambdaofgod/api-examples/blob/master/Google_Colab_%26_Drive.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;this notebook&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As to how upload files to Google Drive, Media's suggestion is useful - upload zipped image folder.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-23T07:34:30.930" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32023" PostTypeId="1" CreationDate="2018-05-23T07:46:47.340" Score="2" ViewCount="124" Body="&lt;p&gt;I have this simple neural networks (&quot;Self Organizing Maps&quot;):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/d4XvT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/d4XvT.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the purpose of the lateral connections beetween the perceptrons (neurons) ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2018-06-23T15:53:22.223" Title="About neural network called &quot;Self Organizing Maps&quot;" Tags="|neural-network|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32024" PostTypeId="2" ParentId="32015" CreationDate="2018-05-23T07:53:32.023" Score="2" Body="&lt;p&gt;Some approaches when there is a small amount of labeled data and a large amount of unlabeled data:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Semi-supervised learning&lt;/strong&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Semi-supervised_learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Semi-supervised_learning&lt;/a&gt; - mixtures of supervised algorithms on labeled data and unsupervised algorithms on unlabeled data. One of them (label propagation) is even implemented in scikit-learn &lt;a href=&quot;http://scikit-learn.org/stable/modules/label_propagation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/label_propagation.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Active learning&lt;/strong&gt; &lt;a href=&quot;http://active-learning.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://active-learning.net/&lt;/a&gt; - algorithms that actively choose the data from which they learn, so they can achieve better performance using less labeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two approaches are complementary. Therefore, there are combinations of active + semi-supervised learning algorithms.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-23T07:53:32.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32025" PostTypeId="1" AcceptedAnswerId="35629" CreationDate="2018-05-23T08:08:20.973" Score="3" ViewCount="4307" Body="&lt;p&gt;I have the pandas df in the following format (with NoOFDays being EndDate - StartDate)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;------------------------------------------&#xA;|Item | StartDate | EndDate    | NoOFDays|&#xA;------------------------------------------&#xA;| A |  01-Jan-2018| 04-Jan-2018|    4    |&#xA;| A |  07-Jan-2018| 08-Jan-2018|    2    |&#xA;| B |  03-Jan-2018| 05-Jan-2018|    3    |&#xA;| A |  03-Jan-2018| 05-Jan-2018|    3    |&#xA;------------------------------------------&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And wanted to get the count of each day grouped by item&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;A 01-Jan-2018 1&#xA;A 02-Jan-2018 1&#xA;A 03-Jan-2018 2&#xA;A 04-Jan-2018 2&#xA;A 05-Jan-2018 1&#xA;A 06-Jan-2018 0&#xA;A 07-Jan-2018 1&#xA;A 08-Jan-2018 1&#xA;B 01-Jan-2018 0&#xA;B 02-Jan-2018 0&#xA;B 03-Jan-2018 1&#xA;B 04-Jan-2018 1&#xA;B 05-Jan-2018 1&#xA;B 06-Jan-2018 0&#xA;B 07-Jan-2018 0&#xA;B 08-Jan-2018 0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50641" LastActivityDate="2018-07-18T03:51:12.973" Title="With the Start Date and End Date given , finding date counts in Pandas" Tags="|python|pandas|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32026" PostTypeId="2" ParentId="29488" CreationDate="2018-05-23T08:10:46.010" Score="1" Body="&lt;p&gt;Try to make your explanation simple and brief. Maybe something like &quot;PCA distils a dataset's features into as few attributes as possible; these attributes explain as much of the observed trend as possible while being as different from each other as possible&quot;. If you can do this using visual aids, even better.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as analogies go, maybe use an example of a motor accident. There are many predictors of an accident, but PCA may combine the majority of them to form 2 features - perhaps vehicle attributes and driver attributes. These are different but arguably are both highly predictive of a motor accident. [&lt;em&gt;I realise this is not technically how PCA works, but might help a &lt;strong&gt;layman&lt;/strong&gt; understand&lt;/em&gt;].&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-05-23T08:10:46.010" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32027" PostTypeId="2" ParentId="32016" CreationDate="2018-05-23T08:30:00.943" Score="1" Body="&lt;p&gt;Your confusion stems from the fact that in ML communities people sometimes call categorical distribution multinomial, and also Bernoulli distribution is called binomial.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As for Naive Bayes, see &lt;a href=&quot;http://scikit-learn.org/stable/modules/naive_bayes.html#multinomial-naive-bayes&quot; rel=&quot;nofollow noreferrer&quot;&gt;scikit-learn's documentation&lt;/a&gt; provides a good explanation.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-23T08:30:00.943" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32028" PostTypeId="2" ParentId="32025" CreationDate="2018-05-23T08:38:37.053" Score="0" Body="&lt;p&gt;What you are looking for is groupby with multiple columns.&#xA;Based on this example your pandas groupby would look like this :&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;df.groupby(['Item', 'StartDate']).agg(['count'])&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="1029" LastActivityDate="2018-05-23T08:38:37.053" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32029" PostTypeId="2" ParentId="29470" CreationDate="2018-05-23T08:40:33.047" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I tried using a CountVectorizer but it's giving me a dataframe with&#xA;  way too many columns and the model is having a hard time using it&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What model did you use? There are models that are well suited for such data, and some even can handle sparse data natively - in scikit-learn you can use :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;LASSO&lt;/a&gt; or other sparse linear models. &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/naive_bayes.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Naive Bayes&lt;/a&gt; - Multinomial or Bernoulli one, make sure to try both, sometimes Bernoulli (which is simple, it only uses indicator, and not counts) performs better&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Other than that, Factorization Machines can be also used (FMs inventor, Steffan Rendle did even win a Kaggle competition which had really sparse data using them).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also another approach would be to just extract keywords and run BoW on them - for example Gensim library has methods to do that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For both of the above approaches the suggestion mentioned by Kasra Manshaei is still valid.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet another approach would be to use neural networks - especially Recurrent NNs, but their usage is not that straightforward, and they might need serious computational resources to be trained reasonably (though they can use pretrained word embeddings, which makes it easier).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;BTW if you have problems with keeping all that data in memory, you might try to dump it to disk and then use a library that gets files as input - for example &lt;a href=&quot;https://github.com/JohnLangford/vowpal_wabbit/wiki&quot; rel=&quot;nofollow noreferrer&quot;&gt;Vowpal Wabbit&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastEditorUserId="45064" LastEditDate="2018-05-23T08:46:15.293" LastActivityDate="2018-05-23T08:46:15.293" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32031" PostTypeId="1" CreationDate="2018-05-23T09:00:23.120" Score="5" ViewCount="149" Body="&lt;p&gt;I'm studying the training algorithm for the self-organizing maps.&#xA;The slides i'm studying says that : after i've found a winner neuron ( in reference to a specific example in the example set), i have to apply this formula to its, and its neighbours.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/G8PDt.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/G8PDt.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $d_{t}$ is the specific example, $m_{ij}$ the neuron ij.&#xA;What i've not understood is the meaning of the &lt;strong&gt;dist&lt;/strong&gt; function in the radius . And what this &lt;strong&gt;radius&lt;/strong&gt; means and what is its purpose. Can you help me ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastActivityDate="2021-02-15T02:36:05.650" Title="Neural Network Self-organizing maps" Tags="|neural-network|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32032" PostTypeId="1" AcceptedAnswerId="32034" CreationDate="2018-05-23T09:25:41.180" Score="5" ViewCount="6198" Body="&lt;p&gt;I have to calculate precision and recall for a university project to measure the quality of the classification output (with sklearn).&#xA;Say this would be my results:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;y_true = [0, 1, 2, 1, 1]&lt;br&gt;&#xA;y_pred = [0, 2, 1, 2, 1]&lt;/p&gt;&#xA;&#xA;&lt;p&gt;confusion matrix:&lt;br&gt;&#xA;[1 0 0]&lt;br&gt;&#xA;[0 1 2]&lt;br&gt;&#xA;[0 1 0]   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have read about it and the definition makes sense for me in a binary setting but with 3 labels I find it hard to interpret precision/recall.&lt;br&gt;&#xA;If I use sklearn.metrics.precision/recall_score it gives me 0.4 for both (average = micro)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now for the precision this makes somewhat sense because 2 out of 5 are correctly classified. But I am having problems interpreting the 0.4 result for recall.&lt;/p&gt;&#xA;" OwnerUserId="52572" LastActivityDate="2020-03-15T01:18:24.503" Title="Precision and Recall if not binary" Tags="|machine-learning|confusion-matrix|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32033" PostTypeId="1" CreationDate="2018-05-23T09:26:06.947" Score="1" ViewCount="171" Body="&lt;p&gt;I have tried clustering using kmeans in Orange but it looks like there are certain limitations as listed below,&#xA;- Supports up to 5000 records only&#xA;- no. of clusters can be only 30&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone please confirm on this limitation? And is there a way to overcome the same. &lt;/p&gt;&#xA;" OwnerUserId="52573" LastEditorUserId="52573" LastEditDate="2018-05-24T11:59:19.727" LastActivityDate="2018-05-24T11:59:19.727" Title="Limitations while using orange for clustering" Tags="|clustering|orange|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32034" PostTypeId="2" ParentId="32032" CreationDate="2018-05-23T09:43:43.077" Score="2" Body="&lt;p&gt;&lt;code&gt;from sklearn.metrics import recall_score&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you then call &lt;code&gt;recall_score.__dir__&lt;/code&gt; (or directly read the docs &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) you'll see that recall is&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The recall is the ratio &lt;code&gt;tp / (tp + fn)&lt;/code&gt; where &lt;code&gt;tp&lt;/code&gt; is the number&#xA;  of    true positives and &lt;code&gt;fn&lt;/code&gt; the number of false negatives&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you go down to where they define &lt;code&gt;micro&lt;/code&gt;, it says&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;'micro'&lt;/code&gt;:            Calculate metrics globally by counting the&#xA;  total true positives,             false negatives and false positives&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Here, the true positives are &lt;span class=&quot;math-container&quot;&gt;$2$&lt;/span&gt; (the sum of the terms on the diagonal, also known as the trace), while the sum of the false negatives plus the false positives (the off-diagonal terms) is &lt;span class=&quot;math-container&quot;&gt;$3$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As &lt;span class=&quot;math-container&quot;&gt;$2/5=.4$&lt;/span&gt;, the recall (using the &lt;code&gt;micro&lt;/code&gt; argument for &lt;code&gt;average&lt;/code&gt;) is indeed &lt;span class=&quot;math-container&quot;&gt;$.4$&lt;/span&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that, using &lt;code&gt;micro&lt;/code&gt;, precision and recall are the same. The following, in fact, returns nothing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from numpy import random&#xA;from sklearn.metrics import recall_score, precision_score&#xA;for i in range(100):&#xA;    y_pred = random.randint(0, 3, 5)&#xA;    y_true = random.randint(0, 3, 5)&#xA;    if recall_score(y_pred, y_true, average='micro') != precision_score(y_pred, y_true, average='micro'):&#xA;        print(i)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52109" LastEditorUserId="29575" LastEditDate="2020-03-15T01:18:24.503" LastActivityDate="2020-03-15T01:18:24.503" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32035" PostTypeId="1" AcceptedAnswerId="32489" CreationDate="2018-05-23T09:50:30.123" Score="6" ViewCount="725" Body="&lt;p&gt;We can define the F - measure as follows: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$F_{\alpha}=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}} $&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Now we might be interested in choosen a good $\alpha$. In the article  &lt;em&gt;The truth of the F-measure&lt;/em&gt; the author states that one can choose the conditions: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\beta=R/P$, where $\frac{\partial F_{\alpha}}{\partial P}=\frac{\partial F_{\alpha}}{\partial R}$&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and then we obtain $\alpha=1/(\beta^2+1)$ and &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R} $&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;It is said that &lt;em&gt;The motivation behind this condition is that at the point where the gradients of E w.r.t. P and R are equal, the ratio of R against P should be&#xA;a desired ratio $\beta$.&lt;/em&gt; I understand that the condition will garantify that the user is willing to trade an increment in precision for an equal loss in recall. But I do not get why the equality of both partial derivatives correspond to these hypothesis. I would rather understand when one partial derivative equals the other partial derivative multiplied by minus one. Could anyone explain me why the desired condition (condition in words) correspond to this equality (condition in math terms)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Well, we could do the following: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;$\partial F=\frac{\partial F_{\alpha}}{\partial P}\partial P+\frac{\partial F_{\alpha}}{\partial R}\partial R$.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And since we want for $\partial P=-\partial R$ that $\partial F=0$, we obtain easily the condition. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I have one problem with this: Since $\frac{\partial F_{\alpha}}{\partial P}/\frac{\partial F_{\alpha}}{\partial R}=1$, and the fact that the gradient is perpendicular to each level curve (&lt;a href=&quot;https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/part-b-chain-rule-gradient-and-directional-derivatives/session-36-proof/MIT18_02SC_pb_32_comb.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/part-b-chain-rule-gradient-and-directional-derivatives/session-36-proof/MIT18_02SC_pb_32_comb.pdf&lt;/a&gt;) we would have that the level curve must have $m=-1$. Nonetheless, when I calculate the level curve for some constant $c$ I get the result, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$R(P)=\frac{c(1-\alpha ) P}{P-c\alpha}$, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;which clearly is not a linear funcion with $m=-1$. What am I missing?&lt;/p&gt;&#xA;" OwnerUserId="52518" LastEditorUserId="52518" LastEditDate="2018-05-27T15:08:53.647" LastActivityDate="2018-06-01T07:54:10.023" Title="F - measure derivation (harmonic mean of precision and recall)" Tags="|machine-learning|clustering|metric|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32036" PostTypeId="1" CreationDate="2018-05-23T09:55:22.327" Score="3" ViewCount="43" Body="&lt;p&gt;How would one go about selecting the validation set used to evaluate trained models by an automated system, in order to ensure that each new model is at least as good as, or better than the previous model? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets say that in this case we have a regression problem, and we can depend on a single performance metric such as RMSE.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm working on a program that automatically trains, tests and if the tests passed, deploys a new deep learning model every x new data samples/time.&lt;br&gt;&#xA;Each training is done on the original dataset (labeled data) and the new dataset which is labeled by predictions of the previous model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These were the options I came up with and some pros/cons.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option A:&lt;/strong&gt;&lt;br&gt;&#xA;Create a validation set once, always apply this set&lt;br&gt;&#xA;+ Always have a consistent benchmark&lt;br&gt;&#xA;- Validation set might get 'out of date' in cases where the relevancy of data might change with time&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option B:&lt;/strong&gt;&lt;br&gt;&#xA;Create an initial validation set, add X percentage of new training samples to the original set&lt;br&gt;&#xA;+ Consistent time spread of samples&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Option C:&lt;/strong&gt;&lt;br&gt;&#xA;Create a random validation set every training&lt;br&gt;&#xA;+ Benchmark is most likely to contain a relevant set of samples (That are in data)&lt;br&gt;&#xA;- Test results can vary each training session&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Option D:&lt;/em&gt;&lt;/strong&gt;&lt;br&gt;&#xA;Cross-validation&lt;br&gt;&#xA;+ Benchmark is most likely to contain a relevant set of samples (That are in data)&lt;br&gt;&#xA;+ Likely least variance between each training session&lt;br&gt;&#xA;- Computationally expensive, even more so for deep learning models&lt;/p&gt;&#xA;" OwnerUserId="46868" LastEditorUserId="46868" LastEditDate="2018-05-23T14:07:20.573" LastActivityDate="2018-05-23T14:07:20.573" Title="Validation of a model generated by automated deep learning system" Tags="|deep-learning|regression|cross-validation|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32037" PostTypeId="2" ParentId="26652" CreationDate="2018-05-23T10:42:13.067" Score="2" Body="&lt;p&gt;I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called &lt;code&gt;vectorizer.fit_transform()&lt;/code&gt; and I got similar error. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What solved the issue was calling &lt;code&gt;vectorizer.transform()&lt;/code&gt;. It is because, &lt;code&gt;fit_transform()&lt;/code&gt; will fit the current data in the model, which is not what we are seeking because vectorizer has already been fitted. We just need to transform the new data to model which has been created. So, calling &lt;code&gt;vectorizer.transform()&lt;/code&gt; did the work.&lt;/p&gt;&#xA;" OwnerUserId="52576" LastEditorUserId="29575" LastEditDate="2018-05-23T11:54:02.380" LastActivityDate="2018-05-23T11:54:02.380" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32038" PostTypeId="1" CreationDate="2018-05-23T10:46:46.163" Score="4" ViewCount="1926" Body="&lt;p&gt;I want to train YOLO3 for a custom dataset that has raw labels in JSON format. Each bounding box in JSON is specified as &lt;code&gt;[x1, y1, x2, y2]&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I have converted &lt;code&gt;[x1, y1, x2, y2]&lt;/code&gt; to &lt;code&gt;[cx, cy, pw, ph]&lt;/code&gt;, that is, center x, center y of the bounding box, scaled to image width and height; &lt;code&gt;pw, ph&lt;/code&gt; are the ratios of bounding box's width and height relative to the image's width and height. But I don't think that's complete (or even right).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I understood, YOLO3 assigns N anchor boxes to each grid cell (image is divided into SxS grid cells) and thus, the prediction of a bounding box is relative to a given anchor box from a grid cell (that one anchor box that has the highest IOU with the ground truth). The formulas are below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/c0igd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/c0igd.png&quot; alt=&quot;YOLO3 BB&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, how should I prepare ground truths so that YOLO3 can understand them? Do I have to, somehow, &lt;em&gt;reverse&lt;/em&gt; those formulas? Also how to account for different number of scales and different number of anchor boxes?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For a concrete example: Suppose I have a &lt;code&gt;416 x 416&lt;/code&gt; image and a configuration of &lt;code&gt;13 x 13&lt;/code&gt; grid cells. The ground truth bounding box (from the dataset) is &lt;code&gt;[x1=100, y1=100, x2=200, y2=200]&lt;/code&gt;, class = c. What will be the converted values for YOLO3? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;L.E.: Say we have 2 classes &lt;code&gt;[car, person]&lt;/code&gt; and 2 anchors (1 wide, 1 tall).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would the output be a tensor of shape &lt;code&gt;13 x 13 x (2*(5+2))&lt;/code&gt; where most of the values (that vector of shape &lt;code&gt;2*(5+2)&lt;/code&gt;) for the grid cells are 0 except for one particular cell (the one in which the center of the ground truth bounding box falls)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this case, for that cell (say &lt;code&gt;c[i,j]&lt;/code&gt;), suppose the largest IOU is given for anchor 2 and that the ground truth class is &lt;code&gt;person&lt;/code&gt;. This means that &lt;code&gt;c[i,j,:7]&lt;/code&gt; (anchor 1 prediction) will be ignored and &lt;code&gt;c[i,j,7:]&lt;/code&gt; (anchor 2 prediction) will be &lt;code&gt;[bx, by, bw, bh, conf, 0, 1]&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, how should the ground truth for the person's bounding box should be encoded? Should it be an offset from a particular anchor of a grid cell? This is what it's still unclear for me.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;" OwnerUserId="42357" LastEditorUserId="42357" LastEditDate="2018-05-23T22:52:01.817" LastActivityDate="2021-06-03T08:33:10.790" Title="Preparing ground truth labels for YOLO3" Tags="|dataset|object-detection|yolo|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32039" PostTypeId="2" ParentId="32032" CreationDate="2018-05-23T11:29:33.133" Score="4" Body="&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn.metrics.classification_report&lt;/a&gt; provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes. &lt;/p&gt;&#xA;" OwnerUserId="52428" LastActivityDate="2018-05-23T11:29:33.133" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32040" PostTypeId="1" CreationDate="2018-05-23T11:31:48.613" Score="1" ViewCount="63" Body="&lt;p&gt;R code :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse) # utility functions&#xA;library(rpart) # for regression trees&#xA;library(randomForest) # for random forests&#xA;library(modelr)&#xA;&#xA;split_data = resample_partition(melb_data,c(test=.3,train=.7))&#xA;&#xA;get_mae &amp;lt;- function(maxdepth, target, predictors, training_data, testing_data){&#xA;&#xA;&#xA;predictors &amp;lt;- paste(predictors, collapse=&quot;+&quot;)&#xA;formula &amp;lt;- as.formula(paste(target,&quot;~&quot;,predictors,sep = &quot;&quot;))&#xA;&#xA;&#xA;model &amp;lt;- rpart(formula, data = training_data,&#xA;             control = rpart.control(maxdepth = maxdepth))&#xA;&#xA;mae &amp;lt;- mae(model, testing_data)&#xA;return(mae)&#xA;}&#xA;&#xA;target &amp;lt;- &quot;Price&quot;&#xA;predictors &amp;lt;-  c(&quot;Rooms&quot;,&quot;Bathroom&quot;,&quot;Landsize&quot;,&quot;BuildingArea&quot;,&#xA;             &quot;YearBuilt&quot;,&quot;Lattitude&quot;,&quot;Longtitude&quot;)&#xA;&#xA;for(i in 1:10){&#xA;mae &amp;lt;- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)&#xA;print(glue::glue(&quot;Maxdepth: &quot;,i,&quot;\t MAE: &quot;,mae))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;OUTPUT : &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Maxdepth: 1  MAE: 356628.697268696  &#xA;Maxdepth: 2  MAE: 356628.697268696  &#xA;Maxdepth: 3  MAE: 356628.697268696  &#xA;Maxdepth: 4  MAE: 356628.697268696  &#xA;Maxdepth: 5  MAE: 356628.697268696  &#xA;Maxdepth: 6  MAE: 356628.697268696  &#xA;Maxdepth: 7  MAE: 356628.697268696  &#xA;Maxdepth: 8  MAE: 356628.697268696  &#xA;Maxdepth: 9  MAE: 356628.697268696  &#xA;Maxdepth: 10 MAE: 356628.697268696&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52577" LastEditorUserId="29575" LastEditDate="2018-05-23T11:52:19.333" LastActivityDate="2018-05-23T11:52:19.333" Title="I'm getting same MAE value in decision tree model for maxdepth value from 1 to 10" Tags="|machine-learning|decision-trees|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32041" PostTypeId="1" AcceptedAnswerId="32069" CreationDate="2018-05-23T11:40:57.517" Score="1" ViewCount="60" Body="&lt;p&gt;I am building a mobile app that can predict what apps users may be interested in downloading from the play store, based on what apps the user has already installed on their device and how much time they have spent on these apps. Also, there is the option to scroll through the top apps in the play store and you can &quot;favourite&quot; any apps that you find interesting/want to download. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Based on this data, I would like to make predictions and notify users of any new popular apps that are released in the future, however since the predictions will be user specific, I am unsure if the dataset will be too small for k-means clustering (Group the apps into genres and find most popular genres). 35 is the average number of apps installed on user's smartphones, and if you include any &quot;favourited apps&quot; the total dataset could be around 50. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Perhaps there is another more suitable technique I could apply to make accurate predictions but I am unsure where to go at the moment.&lt;/p&gt;&#xA;" OwnerUserId="52578" LastActivityDate="2018-05-23T18:04:33.440" Title="Predicting which apps users may be interested in" Tags="|clustering|k-means|prediction|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32042" PostTypeId="1" CreationDate="2018-05-23T11:49:29.320" Score="2" ViewCount="28" Body="&lt;p&gt;I trained &lt;a href=&quot;https://github.com/ZFTurbo/ZF_UNET_224_Pretrained_Model/blob/master/zf_unet_224_model.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;segnet&lt;/a&gt; on a dataset of remote sensing imagery. When I run &lt;code&gt;model.evaluate&lt;/code&gt; a set of metrics is returned. When I compile the model again with the same settings and evaulate the model again, the results differ. For as far as I understand, there is nothing stochastic or random in the compiling of the model, am I right? I thought that the order of the training images in the batches could make a difference, so I tried a batch size of 1, but still I get different results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does the initialization of the model result in different results when predicting as well? And if yes, how do I resolve this? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the code I'm using for the initialization of the network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def my_acc(y_true, y_pred):&#xA;    class_id_true = K.argmax(y_true, axis=-1)&#xA;    class_id_preds = K.argmax(y_pred, axis=-1)&#xA;    accuracy_mask = K.cast(K.equal(class_id_preds, 1), 'int32')&#xA;    class_acc_tensor = K.cast(K.equal(class_id_true, class_id_preds), 'int32') * accuracy_mask&#xA;    class_acc = K.sum(class_acc_tensor) / K.maximum(K.sum(accuracy_mask), 1) &#xA;    return class_acc&#xA;&#xA;model = ZF_UNET_224() # This model is from the link in the beginning of the text&#xA;model.compile(optimizer=Adam(lr=0.0000001), &#xA;              loss = class_weighted_pixelwise_crossentropy,&#xA;              metrics=['accuracy',my_acc,'categorical_accuracy'])&#xA;model.evaluate(x=imgs_test,y=imgs_mask_test,batch_size = 1,verbose=1)&#xA;&#xA;&#xA;&#xA;# result1: [77778.927522078811, 0.94418407911839697, 0.0039057209515957264, 0.94418407911839697]&#xA;# result2: [77586.558084239135, 0.077934800141045577, 0.02123359974321911,  0.077934800141045577]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="47339" LastActivityDate="2018-05-23T11:49:29.320" Title="keras evaulate method results vary with equal testset" Tags="|keras|training|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32044" PostTypeId="2" ParentId="32040" CreationDate="2018-05-23T11:51:28.363" Score="1" Body="&lt;p&gt;In this line:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:10){mae &amp;lt;- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You are setting &lt;code&gt;maxdepth&lt;/code&gt; to 3 for every &lt;code&gt;i&lt;/code&gt;. You have to change it for:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for(i in 1:10){mae &amp;lt;- get_mae(maxdepth = i, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-23T11:51:28.363" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32045" PostTypeId="1" AcceptedAnswerId="32049" CreationDate="2018-05-23T12:14:34.643" Score="3" ViewCount="166" Body="&lt;p&gt;I've been given this problem but cannot seem to get an analytical solution. I've tried satisfying the logistic function with several vectors but have difficulty finding ones which are also orthogonal. The problem begins with me being given a neural unit weight vector,&#xA;$$w =\begin{bmatrix}&#xA;           1 \\&#xA;           \frac{1}{4} \\&#xA;           \frac{1}{9} \\&#xA;         \end{bmatrix}$$&#xA;The neuron output u is related to its input pattern v∈R3 by&#xA;$$u = f(w^Tv + 1)$$&#xA;where&#xA;$$f(x) = (1 + e^{-x})^{-1}$$&#xA;To find a pair of orthogonal vectors, v1 and v2, for which the unit output $u_{i} = 0.5, i=1,2.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far I know that $w^TV = 0$ for the function output to be 0.5.&lt;/p&gt;&#xA;" OwnerUserId="52583" LastActivityDate="2018-05-23T12:22:59.490" Title="Finding orthogonal input patterns associated with logistic function output" Tags="|neural-network|mathematics|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32046" PostTypeId="1" CreationDate="2018-05-23T12:15:47.917" Score="2" ViewCount="476" Body="&lt;p&gt;I'm making a multiclassifier model with 5 classes.&#xA;(it is not important in my question whether it has 2 classes or 5 classes, though).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;class distribution is very imbalanced.&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ANCiy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ANCiy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, I did resampling for oversampling to match each number of classes.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/ZfWnY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZfWnY.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;MY question is.&#xA;Is there any statistical error or problem&#xA;if I resample classes for oversampling(bootstrap resampling or SMOTE)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can anyone explain for this approach?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="51646" LastActivityDate="2018-05-30T13:11:53.060" Title="Class Imbalance Problem" Tags="|classification|class-imbalance|classifier|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32048" PostTypeId="2" ParentId="31991" CreationDate="2018-05-23T12:19:00.873" Score="-2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;What does it mean that classes are mutually exclusive but soft-labels are accepted?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As it can be seen from &lt;a href=&quot;https://stackoverflow.com/a/34243720/5120235&quot;&gt;here&lt;/a&gt;, &lt;em&gt;&lt;code&gt;tf.nn.softmax&lt;/code&gt; produces just the result of applying the softmax function to an input tensor. The softmax &quot;squishes&quot; the inputs so that sum(input) = 1; it's a way of normalizing. The shape of the output of a softmax is the same as the input - it just normalizes the values. The outputs of softmax can be interpreted as probabilities. In contrast, &lt;code&gt;tf.nn.softmax_cross_entropy_with_logits&lt;/code&gt; computes the cross-entropy of the result after applying the softmax function (but it does it all together in a more mathematically careful way).&lt;/em&gt; It's mathematically careful due to the fact that $y_i$ in $log(y_i)$ can be zero. As you can read from &lt;a href=&quot;https://datascience.stackexchange.com/a/9311/28175&quot;&gt;here&lt;/a&gt;, &lt;em&gt;A randomly-initialized softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate log(yi) for any y′i=0, because the negative classes always contribute 0 to the error. Second, in the practical code you can limit the value to something like $log( max( y_predict, 1e-15 ) )$ for numerical stability - in many cases, it is not required, but this is sensible defensive programming.&lt;/em&gt; I encourage you to take a look at the answers to &lt;a href=&quot;https://stackoverflow.com/q/37312421/5120235&quot;&gt;this&lt;/a&gt; question.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The first sentence means that your classifier may not be able to classify the labels exactly as they are, one-hot-encoded. What it does is to find the chance that how likely it is that the input belongs to each class. And this won't make a problem if it does not have mutually exclusive output vector as the algorithm implies. It just needs a vector that the sum of its entries is equal to one. &lt;em&gt;If they are not, the computation of the gradient will be incorrect.&lt;/em&gt; I guess this line is added to announce that the outputs of this differentiable component will not be one-hot-encoded and it's because of the nature of these nets. The first layers of convolutional networks are like bases vectors and each instance of classes share these bases. All inputs are made up of these bases. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I'm wondering if, in a multi-class exclusive case where the only constraint on the labels is that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Basically, for multi-label classification, your input may have different labels. Consequently, your classes won't be mutually exclusive anymore. Moreover, in those cases, we don't use softmax as the last layer. You have to have sigmoid for each output and your cross-entropy cost function will be slightly different &lt;a href=&quot;https://datascience.stackexchange.com/q/9302/28175&quot;&gt;1&lt;/a&gt;. Consequently, the output of each entry should be a valid probability, that is why sigmoid is used, and the label vectors for such tasks is a not one-hot-encoded anymore. different classes have different entries and based on the existence of instances of each class, the corresponding entry should be one. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-23T13:09:02.377" LastActivityDate="2018-05-23T13:09:02.377" CommentCount="12" ContentLicense="CC BY-SA 4.0" />
  <row Id="32049" PostTypeId="2" ParentId="32045" CreationDate="2018-05-23T12:22:59.490" Score="0" Body="&lt;p&gt;All you have to do is complete $\omega$ to an orthogonal $\mathbb{R}^3$ basis, as $\omega^T V = 0$ and $v_1$, $v_2$ are orthogonal. You can do this using &lt;a href=&quot;https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gram Schmidt orthogonalization&lt;/a&gt;. The two other vectors of the basis will be $v_1$ and $v_2$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The initial vectors to do the Gram Schmidt process can be any pair of independent vectors, for instance $(1, 0, 0)$ and $(0, 1, 0)$.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-23T12:22:59.490" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32050" PostTypeId="2" ParentId="32001" CreationDate="2018-05-23T12:38:17.513" Score="1" Body="&lt;p&gt;You have a high Bayes error rate and it means that you almost can not learn anything. You have to add extra features and investigate whether your data has a small Bayes error or not. Currently, it is worse than a disaster. This large Bayes error illustrates that you have patterns, input vectors, that have completely same components as each feature but different labels. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/25413/28175&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-23T12:38:17.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32051" PostTypeId="1" CreationDate="2018-05-23T12:48:45.797" Score="4" ViewCount="16666" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1oVfl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1oVfl.png&quot; alt=&quot;Question&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WEecm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WEecm.png&quot; alt=&quot;Formula for silhouette coefficient&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Calculate the silhouette coefficient of point Pi from the above image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To apply the given formula, how to know which is a(i) and b(i)?&lt;/p&gt;&#xA;" OwnerUserId="52586" LastActivityDate="2019-09-12T12:48:44.510" Title="How to calculate the silhouette coefficient?" Tags="|data-mining|clustering|k-means|distance|k-nn|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32052" PostTypeId="2" ParentId="31988" CreationDate="2018-05-23T13:01:09.133" Score="0" Body="&lt;p&gt;What kind of distortion does dirt/dust produce? If it is somewhat regular, for example more or less regularly sized speckles (greyish circles) it might be worthwhile to try to classify the logarithm of the real part of the fourier transform of the picture, instead of the original picture.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, knowing the physical properties of the distortion might guide you towards devising simple heuristics (for example based on some frequencies in the fuorier spectrum) that are as good as a ml-based approach.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastActivityDate="2018-05-23T13:01:09.133" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32053" PostTypeId="2" ParentId="32036" CreationDate="2018-05-23T13:10:44.640" Score="1" Body="&lt;p&gt;I would suggest Option E: Efron-Gong &quot;optimism&quot; bootstrap. It is similar to Cross-validation in spirit, but requires orders of magnitude less repetitions as CV and it exploits the whole dataset. The procedure is described in section 6 of &lt;a href=&quot;https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291097-0258%2819960229%2915%3A4%3C361%3A%3AAID-SIM168%3E3.0.CO%3B2-4&quot; rel=&quot;nofollow noreferrer&quot;&gt;this reference&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="50211" LastEditorUserId="50211" LastEditDate="2018-05-23T13:34:02.070" LastActivityDate="2018-05-23T13:34:02.070" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32054" PostTypeId="1" CreationDate="2018-05-23T13:26:24.137" Score="4" ViewCount="46" Body="&lt;p&gt;We are looking for data sets with 3D images, preferably from the medical field. It is important that they have been segmented by more than one person/expert. An example of this is the BraTS Challenge dataset (&lt;a href=&quot;https://www.med.upenn.edu/sbia/brats2018/data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.med.upenn.edu/sbia/brats2018/data.html&lt;/a&gt;) or the LIDC IDRI Dataset (&lt;a href=&quot;https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI&lt;/a&gt;), but we need more datasets for our research.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Many thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="52589" LastEditorUserId="52589" LastEditDate="2018-05-23T14:53:33.760" LastActivityDate="2018-05-23T15:05:21.313" Title="Searching for a 3D Dataset, segmented by 2 or more Experts" Tags="|machine-learning|dataset|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32055" PostTypeId="1" CreationDate="2018-05-23T13:29:48.737" Score="3" ViewCount="716" Body="&lt;p&gt;I have a dataset that contains weekly sales for stores and categories. It looks like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Kebx6.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Kebx6.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to apply gradient boosting method to predict weekly sales. My question is: Should I create dummy variables for categories(1 to 7 which indicates product type) and stores(1 to 11)?&lt;/p&gt;&#xA;" OwnerUserId="52590" LastEditorUserId="29575" LastEditDate="2018-05-23T14:23:21.513" LastActivityDate="2018-05-24T10:58:06.883" Title="how to decide categorical variables for prediction" Tags="|predictive-modeling|categorical-data|forecast|forecasting|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32056" PostTypeId="1" CreationDate="2018-05-23T13:45:54.967" Score="1" ViewCount="309" Body="&lt;p&gt;I am very much a beginner in ML space. I am learning keras to get hands-on experience. &#xA;I picked the classification of &lt;a href=&quot;https://nlp.stanford.edu/wiki/Software/Classifier/20_Newsgroups&quot; rel=&quot;nofollow noreferrer&quot;&gt;20_newsgroups&lt;/a&gt; data for my task, I used &lt;a href=&quot;http://nlp.stanford.edu/data/glove.6B.zip&quot; rel=&quot;nofollow noreferrer&quot;&gt;glove.6B.50d.txt&lt;/a&gt; for embeddings. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I chose to train a RNN for the task cos it has the capability to learn time series data better. Trained with different combinations of LSTM layers and output dimensions before converging to one combination. Then I tried &lt;a href=&quot;https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CNN&lt;/a&gt; model for the same task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I found was CNN model could learn faster and gave about 65% accuracy at the end of 15 epochs, where as the RNN model took 50 epochs to get the same validation accuracy even after trying several learning rates. But at the end both models were giving about 65-70% validation accuracy after training for some more duration. I stopped my exploration there.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is I assumed/expected RNN to perform better. According to my understanding of RNN, it builds up the memory of time series data. What is it I am missing here ?&lt;/p&gt;&#xA;" OwnerUserId="52579" LastActivityDate="2024-04-03T22:03:05.417" Title="CNN vs RNN: Classification on 20_newsgroup data" Tags="|rnn|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32057" PostTypeId="2" ParentId="32055" CreationDate="2018-05-23T13:48:16.477" Score="1" Body="&lt;p&gt;Short answer : &lt;strong&gt;yes&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Gradient boosting relies on &lt;strong&gt;decision trees&lt;/strong&gt;. The leaves of your decision trees are built in a fashion to discriminate optimally your features. For numeric features, this means finding the &lt;strong&gt;best separation value&lt;/strong&gt; to decline your dataset into two subsets. One contains observations with a value above or equal to this separation value, while the other presents values that are below this separation value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It would not make any sense to have a leaf splitting your data on a criterion such as $Store &amp;gt;= 5$. However, it would make sense to have a separator such as $Store_5 = 1$ (vs $Store_5 = 0$). This is precisely why &lt;strong&gt;dummy variables are created for categorical values in ensemble methods&lt;/strong&gt;, such as gradient boosting.&lt;/p&gt;&#xA;" OwnerUserId="42829" LastActivityDate="2018-05-23T13:48:16.477" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32058" PostTypeId="1" CreationDate="2018-05-23T14:37:19.937" Score="2" ViewCount="1340" Body="&lt;p&gt;From a very high level of possible machine learning methodologies, what method of classification would be good to identify objects or situations based on the color sensor pixel array (i.e., 16x16 array) being fed into Python?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, if I want to identify certain fruits based on the shape and color detected in the array:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;banana is curved and yellow &lt;/li&gt;&#xA;&lt;li&gt;apple is circle and red&lt;/li&gt;&#xA;&lt;li&gt;orange is circle and orange&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;If I am reading in a constant stream of data in this array form, are there any similar machine learning and classification applications that might be best suited for this type of application?&lt;/p&gt;&#xA;" OwnerUserId="12202" LastActivityDate="2018-05-23T15:02:57.357" Title="Classification of Colors using Machine Learning from an Array of Pixels" Tags="|machine-learning|python|computer-vision|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32059" PostTypeId="2" ParentId="31819" CreationDate="2018-05-23T14:47:37.437" Score="1" Body="&lt;p&gt;Absolutely. You've described topic modeling and sentiment analysis. You might want to train separate models for each part of the problem and stack them, but this is a totally solvable problem. &lt;/p&gt;&#xA;" OwnerUserId="44299" LastActivityDate="2018-05-23T14:47:37.437" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32060" PostTypeId="2" ParentId="32058" CreationDate="2018-05-23T15:02:57.357" Score="1" Body="&lt;p&gt;This is a &lt;a href=&quot;http://cv-tricks.com/tensorflow-tutorial/training-convolutional-neural-network-for-image-classification/&quot; rel=&quot;nofollow noreferrer&quot;&gt;classical problem for a convolutional neural network&lt;/a&gt;. Typically, these work with series of layers using 2d-convolution to find features (e.g., &quot;curved&quot; or &quot;yellow&quot;), then pooling (to borrow from adjacent convolutions), then some combination.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the main question is how you are planning to get a good, labeled, dataset (that is, one where a banana is already labeled as &quot;banana&quot;, for training your NN).&lt;/p&gt;&#xA;" OwnerUserId="9518" LastActivityDate="2018-05-23T15:02:57.357" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32061" PostTypeId="2" ParentId="32054" CreationDate="2018-05-23T15:05:21.313" Score="1" Body="&lt;p&gt;You can look into the &lt;a href=&quot;https://docs.google.com/forms/d/e/1FAIpQLSfwkdSt7hWo_tjHUDu2stDsxWTaWyLJIUiS_iapbtKaydEMIw/viewform&quot; rel=&quot;nofollow noreferrer&quot;&gt;MICCAI dataset&lt;/a&gt;. This contains 35 3D MRI scans of brains. The brains have been parcelated into 134 distinct brain regions by professionals. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-23T15:05:21.313" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32062" PostTypeId="1" AcceptedAnswerId="32064" CreationDate="2018-05-23T15:27:37.527" Score="3" ViewCount="71" Body="&lt;p&gt;What if the data that we could use for the training is obsolete. For instance, if I train my model with the computer sales report from the 20th century and try to predict the actual trends, a disaster, right?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another good example is the one that profiles criminals. It will use the historical &quot;mistakes&quot; as fact and could wrongly suspect innocents(based on race or ethnicity).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How could I avoid this kind of situations?&lt;/p&gt;&#xA;" OwnerUserId="52602" LastEditorUserId="52602" LastEditDate="2018-05-25T21:58:46.643" LastActivityDate="2018-05-25T21:58:46.643" Title="Supervised Learning could be biased if we use obsolete data" Tags="|machine-learning|reinforcement-learning|supervised-learning|" AnswerCount="2" CommentCount="0" ClosedDate="2018-05-25T12:36:50.887" ContentLicense="CC BY-SA 4.0" />
  <row Id="32064" PostTypeId="2" ParentId="32062" CreationDate="2018-05-23T15:58:29.750" Score="3" Body="&lt;p&gt;One approach to using knowledge from the example you gave above is to use that information as a prior for your current model. While it is unlikely that the exact trends observed a while ago will predict future trends, some general observations/correlations will likely still be relevant for the present day. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Concerning your example of criminal profiling. Making predictive models in such cases can be highly biased and is controversial. I invite you to read the Outlook section XVIII.D of this ML review on the &lt;a href=&quot;https://arxiv.org/abs/1803.08823&quot; rel=&quot;nofollow noreferrer&quot;&gt;social implications of machine learning&lt;/a&gt; and the references mentioned.&lt;/p&gt;&#xA;" OwnerUserId="29203" LastEditorUserId="29203" LastEditDate="2018-05-24T17:55:22.957" LastActivityDate="2018-05-24T17:55:22.957" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32065" PostTypeId="1" CreationDate="2018-05-23T16:48:01.243" Score="1" ViewCount="30" Body="&lt;p&gt;I need to &lt;strong&gt;classify MOOC video scripts into one of 5 classes which specify the intent of the sentence, e.g. &lt;code&gt;explanation&lt;/code&gt;, &lt;code&gt;example&lt;/code&gt;, &lt;code&gt;summary&lt;/code&gt; etc., sentence-wise.&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 11 courses extracted and selected equal ratio of files from every course based on the number of files I have for each of them. Say course 1 has 100 files and course 2 has 5 times less, then I selected data for labeling such that the labeled daya for course 1 will be 5 times more than the labeled data for course 2.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to lack of time, &lt;strong&gt;I wanted to label 1/5th of my data&lt;/strong&gt;, which is about ~8800 sentences in 111 files. &lt;strong&gt;My question is.. can this be enough to predict the rest of the data&lt;/strong&gt;, or approx ~29600 sentences more.. maybe if i slowly introduce it to the model, not at once.. or i don't know.. any ideas are welcome, since I am not too experiences. Also &lt;strong&gt;specific text classification algorithms&lt;/strong&gt; you think work best, &lt;strong&gt;would be very helpful&lt;/strong&gt; to mention too. I am open to trying several, as long as their implementation is not too much time-consuming. I will of course try the most common ones first such as Naive Bayes and SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Oh also, &lt;strong&gt;another noob question&lt;/strong&gt; - do I need to convert my text into vectors for some reason? Or will the classifier do it's own work based on the textual data?&lt;/p&gt;&#xA;" OwnerUserId="50625" LastEditorUserId="50625" LastEditDate="2018-05-24T12:33:08.787" LastActivityDate="2018-05-24T12:33:08.787" Title="Would it work to label 1/5th of my data and once I learn a classifier to slowly provide the unlabeled data?" Tags="|python|classification|text-mining|" AnswerCount="0" CommentCount="1" ClosedDate="2018-06-23T20:47:15.597" ContentLicense="CC BY-SA 4.0" />
  <row Id="32066" PostTypeId="1" AcceptedAnswerId="32082" CreationDate="2018-05-23T16:57:25.813" Score="2" ViewCount="7539" Body="&lt;p&gt;I'm not very familiar with neural networks, however, I though I understood the concept of back propagation as starting from the error in the output layer. Say, we have 3 neurons in the output layer and their respective values end up being:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[1 0.5 0.3]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And we wished to obtain values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[0 1 0]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So we can compute a vector of errors between the two:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[-1, +0.5, -0.3] (Not necessarily with the - operation, but you get the point)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And back propagate from there. However, in a the generator of a GAN, it seems to me that the output layer has a bunch of neurons (representative to whatever size the entity we want to generate is) but the error is based only on a % of images the discriminator classified wrongfully. So how exactly do we do back-prop for the generator ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only human readable examples of GAN I've found use frameworks and libraries like autoGrad that kind of obfuscate the problem for me :/&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there a simple way to explain how this backprop can be done ? (Like, in code, not with an &quot;intuitive example&quot; or &quot;simple 4 variable equation where each of the 4 variable packs 50 years of mathematics behind it&quot;)&lt;/p&gt;&#xA;" OwnerUserId="52605" LastActivityDate="2018-05-24T03:55:40.950" Title="Could someone explain to me how back-prop is done for the generator in a GAN?" Tags="|machine-learning|neural-network|backpropagation|gan|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32068" PostTypeId="1" CreationDate="2018-05-23T18:03:04.250" Score="6" ViewCount="186" Body="&lt;p&gt;I'm trying to find a way to predict/calculate how a shape (e.g. outline of a glacier) will change in the future—based on its history (previous shape) and additional factors (e.g. Δtemperature).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;img src=&quot;https://i.stack.imgur.com/jisix.png&quot; width=&quot;200&quot;&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In my example: I have the shape/coordinates of a glacier and an average temperature at 1970, 1985, 2000, 2015. How can I give an estimate on how that shape will look like in 2030, based on the previous shapes and a predicted temperature?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The shapes would ideally come in a format similar to this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[&#xA;  [&#xA;    [-113.74926783355818, 48.32440832757374],&#xA;    [-113.74926767330584, 48.32440813255405],&#xA;    [-113.74926748239692, 48.32440458296232],&#xA;    [-113.74926717549286, 48.32439887665663],&#xA;    [-113.74926686858888, 48.32439317035087],&#xA;    ...&#xA;  ],&#xA;  [&#xA;    [-113.75763099733634, 48.32877681033601],&#xA;    [-113.75763955455557, 48.32877660502744],&#xA;    [-113.75764546550784, 48.32878027200765],&#xA;    [-113.75765402272751, 48.32878006669797],&#xA;    [-113.75766003497893, 48.32878563609638],&#xA;    ...&#xA;  ]&#xA;]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But I imagine it could be necessary to convert this to coordinate offsets (&lt;code&gt;Δx&lt;/code&gt;, &lt;code&gt;Δy&lt;/code&gt;) similar to what &lt;a href=&quot;https://github.com/tensorflow/magenta/blob/master/magenta/models/sketch_rnn/README.md#creating-your-own-dataset&quot; rel=&quot;noreferrer&quot;&gt;sketch_rnn&lt;/a&gt; is doing.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any pointers would help. Even that this is too ambitious. Thanks!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;&lt;sup&gt;Disclaimer: I'm a beginner with this and hope the question isn't too naïve :)&lt;/sup&gt;&lt;/p&gt;&#xA;" OwnerUserId="52608" LastActivityDate="2024-02-07T05:05:40.653" Title="Predicting change of shapes/coordinates" Tags="|machine-learning|predictive-modeling|regression|statistics|algorithms|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32069" PostTypeId="2" ParentId="32041" CreationDate="2018-05-23T18:04:33.440" Score="0" Body="&lt;p&gt;Broadly speaking, what you want is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Recommender_system&quot; rel=&quot;nofollow noreferrer&quot;&gt;recommender system&lt;/a&gt;. If you consider only the user-app association data (app installs, likes, etc.), and no user or app metadata, then you should look at &lt;a href=&quot;https://en.wikipedia.org/wiki/Collaborative_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;collaborative filtering&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;More specifically, a simpler technique you can consider is &lt;a href=&quot;https://en.wikipedia.org/wiki/Item-item_collaborative_filtering&quot; rel=&quot;nofollow noreferrer&quot;&gt;item-similarity&lt;/a&gt; based recommendation (users who install app A, also install app B). A slightly more complex method involves factorization of the partial user-app association matrix and then inferring missing points in the matrix. This &lt;a href=&quot;https://endymecy.gitbooks.io/spark-ml-source-analysis/content/%E6%8E%A8%E8%8D%90/papers/Large-scale%20Parallel%20Collaborative%20Filtering%20the%20Netflix%20Prize.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; describes one such factorization technique used in the Netflix Prize movie recommendation challenge.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-23T18:04:33.440" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32070" PostTypeId="1" CreationDate="2018-05-23T18:12:52.457" Score="2" ViewCount="84" Body="&lt;p&gt;I have a series of sequences, each sequence contains multiple mouse click with the following features: [x coordinate on screen, y coordinate on screen, duration of click], and each sequence is labeled with 1 of four classes which represents the user that did the sequence of clicks. I need help determining the right model/approach for this problem. Here's what I've tried so far:&#xA; 1. RNN with LSTM where I had to pad the sequences with zeros since they're not equal in length&#xA; 2. Draw an image with points on the coordinates clicked, and color coding the points based on the duration of the click, generating an image for each sequence, then running the images through a CNN and classifying them depending on the points&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both of these yield ~40% accuracy on submission. I need ideas/different approaches to the problem. Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="52611" LastActivityDate="2018-05-23T18:12:52.457" Title="Classifying sequences of mouse clicks with different number of clicks per sequence" Tags="|machine-learning|deep-learning|classification|sequence|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32071" PostTypeId="1" AcceptedAnswerId="32076" CreationDate="2018-05-23T18:41:40.727" Score="2" ViewCount="13477" Body="&lt;p&gt;Could you please let me know how to set &lt;code&gt;class-weight&lt;/code&gt; for imbalanced classes in &lt;code&gt;KerasClassifier&lt;/code&gt; while it is used inside the &lt;code&gt;GridSearchCV&lt;/code&gt;?&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;# Use scikit-learn to grid search the batch size and epochs&#xA;from collections import Counter&#xA;from sklearn.model_selection import train_test_split,StratifiedKFold,learning_curve,validation_curve,GridSearchCV&#xA;from sklearn.datasets import make_classification&#xA;from sklearn.preprocessing import StandardScaler&#xA;import numpy as np&#xA;from sklearn.model_selection import GridSearchCV&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense&#xA;from keras.wrappers.scikit_learn import KerasClassifier&#xA;from sklearn.metrics import classification_report&#xA;import pandas as pd&#xA;from sklearn.pipeline import Pipeline&#xA;&#xA;# Function to create model, required for KerasClassifier&#xA;def create_model():&#xA;    # create model&#xA;    model = Sequential()&#xA;    model.add(Dense(12, input_dim=20, activation='relu'))&#xA;    model.add(Dense(1, activation='sigmoid'))&#xA;    # Compile model&#xA;    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;    return model&#xA;# fix random seed for reproducibility&#xA;seed = 7&#xA;np.random.seed(seed)&#xA;# load dataset&#xA;X, y = make_classification(n_classes=2, class_sep=2,weights=[0.95, 0.05], n_informative=3, n_redundant=2, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)&#xA;print('Original dataset shape {}'.format(Counter(y)))&#xA;&#xA;ln = X.shape&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)&#xA;st=StandardScaler()&#xA;&#xA;# create model&#xA;model = KerasClassifier(build_fn=create_model, verbose=0)&#xA;pipeline = Pipeline(steps=[('scaler', st),&#xA;                               ('clf', model )])&#xA;# define the grid search parameters&#xA;batch_size = [20, 40, 60, 80, 100]&#xA;epochs = [ 50, 100]&#xA;param_grid = dict(clf__batch_size=batch_size, clf__epochs=epochs)&#xA;cv = StratifiedKFold(n_splits=5, random_state=42)&#xA;&#xA;grid = GridSearchCV(estimator=pipeline, param_grid=param_grid,cv=cv,scoring=&amp;quot;f1&amp;quot;)&#xA;grid_result = grid.fit(X_train, y_train)&#xA;# summarize results&#xA;print(&amp;quot;Best: %f using %s&amp;quot; % (grid_result.best_score_, grid_result.best_params_))&#xA;# Predictions&#xA;ypred = grid_result.predict(X_train)&#xA;print(classification_report(y_train, ypred))&#xA;print('######################')&#xA;ypred2 = grid_result.predict(X_test)&#xA;print(classification_report(y_test, ypred2))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="26019" LastEditorUserId="83275" LastEditDate="2021-03-12T05:47:34.873" LastActivityDate="2021-03-12T05:47:34.873" Title="How to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV?" Tags="|scikit-learn|keras|class-imbalance|weighted-data|gridsearchcv|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32072" PostTypeId="1" AcceptedAnswerId="32425" CreationDate="2018-05-23T21:09:36.090" Score="8" ViewCount="912" Body="&lt;p&gt;I can see two motives to use &lt;a href=&quot;https://arxiv.org/abs/1608.05343&quot; rel=&quot;noreferrer&quot;&gt;Synthetic Gradients&lt;/a&gt; in RNN:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;To speed up training, by imediately correcting each layer with predicted gradient&lt;/li&gt;&#xA;&lt;li&gt;To be able to learn longer sequences&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.&lt;/p&gt;&#xA;&lt;p&gt;I will now show why Point 1 and Point 2 don't seem to be beneficial, &lt;strong&gt;and I need you to correct me, if they are actually beneficial:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;&lt;strong&gt;Point 1:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Synthetic Gradients tell us we can rely on another &amp;quot;mini-helper-network&amp;quot; (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.&lt;/p&gt;&#xA;&lt;p&gt;However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.&lt;/p&gt;&#xA;&lt;p&gt;Adding this Asyncronisity shouldn't not make layers train faster than during the traditional &amp;quot;locked&amp;quot; full fwdprop -&amp;gt; full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;This makes me think Point 1) will not work. Simply adding SG between each layer&#xA;shouldn't improve training speed.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;&lt;strong&gt;Point 2:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;Ok, how about adding SG only &lt;strong&gt;on the last layer&lt;/strong&gt; to predict &amp;quot;gradient from future&amp;quot; &lt;strong&gt;and only if it's the final timestep during forward prop&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Consider several training sessions (session A, session B):&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;fwdprop timestep_1A ---&amp;gt; fwdprop timestep_2A ---&amp;gt; fwdprop timestep_3A&#xA;----&amp;gt; stop and bkprop!&lt;/p&gt;&#xA;&lt;p&gt;fwdprop timestep_1B ---&amp;gt; fwdprop timestep_2B ---&amp;gt; fwdprop timestep_3B&#xA;----&amp;gt; stop and bkprop!&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;We've just forced our network to &amp;quot;parse&amp;quot; 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Notice, we have our DNI sitting at the very end of &lt;strong&gt;&amp;quot;Session A&amp;quot;&lt;/strong&gt; and predicting &amp;quot;what gradient I would get flowing from the beginning of &lt;strong&gt;Session B&lt;/strong&gt; (from future)&amp;quot;.&#xA;Because of that, timestep_3A will be equipped with gradient &amp;quot;that would have come from timestep_1B&amp;quot;, so indeed, corrections done during A will be more reliable.&lt;/p&gt;&#xA;&lt;p&gt;But, hey! These predicted &amp;quot;synthetic gradients&amp;quot; will be very small (negligible) anyway - after all, that's why we start a new backprop &lt;strong&gt;session B&lt;/strong&gt;. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop &amp;quot;session A&amp;quot;.&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last&#xA;timestep of fwdprop allows to effectively train longer sequences, but&#xA;vanishing gradients didn't go anywhere.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;Ok. Maybe we can get the benefit of training &lt;strong&gt;&amp;quot;session A&amp;quot;&lt;/strong&gt;, &lt;strong&gt;&amp;quot;session B&amp;quot;&lt;/strong&gt; &lt;strong&gt;etc&lt;/strong&gt; on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by &lt;strong&gt;sessionA&lt;/strong&gt; predicting gradients which are vanishing anyway.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Question:&#xA;Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2019-05-31T09:30:44.217" Title="Synthetic Gradients - what's the practical benefit?" Tags="|backpropagation|" AnswerCount="2" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32073" PostTypeId="1" CreationDate="2018-05-23T21:21:04.783" Score="2" ViewCount="491" Body="&lt;p&gt;Recently I discovered such thing as GWR neural networks. This is a family of neural nets that can expand dynamically: in breadth and/or in depth. As far as I understood, such networks are declared to give the best results in continuous learning tasks, because their main goal is to mitigate catastrophic interference/forgetting. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Despite that fact that this approach was new to me, it appears that it was first introduced in far 2002 - more than 15 years ago. But the most interesting here is that I can't find implementations of any of those algorithms in popular frameworks! My main tool for development is Deeplearning4j and I'm quite sure that continuous learning is not implemented there in any way. I also googled for something like &quot;tensorflow growing when required&quot; or &quot;keras dynamically growing networks&quot;, but it doesn't give any results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: why there's a situation that this mechanism is not available in popular NN-frameworks out of the box, despite its promising results (at least based on fancy plots from science papers I've read)? Or maybe I'm missing something or googling using wrong terms and it's called somehow differently? &lt;/p&gt;&#xA;" OwnerUserId="52617" LastActivityDate="2018-05-23T21:21:04.783" Title="Implementations of &quot;Growing When Required&quot; approach" Tags="|neural-network|keras|tensorflow|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32074" PostTypeId="1" CreationDate="2018-05-23T21:58:51.407" Score="2" ViewCount="32" Body="&lt;p&gt;I have this data set with consist of ISO3166 Alpha-2 codes for countries.&#xA;Example: DE, AD, AE etc&#xA;They are coded as factor variables in R and there are about 173 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now because there are too many and this would just overwhelm a boxplot, I want to make a contingency table with other variables by condensing the codes and create shorter categories (also coded as factors) with the codes, for example, having &lt;/p&gt;&#xA;&#xA;&lt;p&gt;DE, RE, ED, FR-&gt;  Europe&lt;/p&gt;&#xA;&#xA;&lt;p&gt;CA, US-&gt;          North America&lt;/p&gt;&#xA;&#xA;&lt;p&gt;VF, HG, HY, TY-&gt;  South America&lt;/p&gt;&#xA;&#xA;&lt;p&gt;HG, TY, UT,FT -&gt;    Africa&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I do that because I tried a few things that did not work.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S  These codes are made up codes but it is just to illustrate.&lt;/p&gt;&#xA;" OwnerUserId="52619" LastEditorUserId="29169" LastEditDate="2021-02-01T20:11:25.487" LastActivityDate="2021-02-01T20:11:25.487" Title="How to make smaller categories with factor character variables" Tags="|r|dataset|data-cleaning|rstudio|data-table|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32075" PostTypeId="1" AcceptedAnswerId="56309" CreationDate="2018-05-23T22:40:47.827" Score="1" ViewCount="599" Body="&lt;p&gt;I am having a hard time implementing/using an ordinal classification cost function for my Convolutional Neural Network in Tensorflow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any implementations/documentation/repository on that which you know of?&lt;/p&gt;&#xA;" OwnerUserId="52618" LastActivityDate="2019-07-24T13:31:47.893" Title="Ordinal Classification As Cost Function - In Keras or Tensorflow" Tags="|classification|keras|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32076" PostTypeId="2" ParentId="32071" CreationDate="2018-05-24T00:20:02.300" Score="2" Body="&lt;pre&gt;&lt;code&gt;grid_result = grid.fit(X_train, y_train, clf__class_weight={0:0.95, 1:0.05})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;FYI, per the docs &lt;code&gt;fit_params&lt;/code&gt; should no longer be passed to the &lt;code&gt;GridSearchCV&lt;/code&gt; constructor as a dict, but should be passed directly to &lt;code&gt;fit&lt;/code&gt; as above.    &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="38751" LastEditorUserId="38751" LastEditDate="2018-05-24T11:03:49.610" LastActivityDate="2018-05-24T11:03:49.610" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32077" PostTypeId="1" CreationDate="2018-05-24T01:09:00.130" Score="2" ViewCount="27" Body="&lt;p&gt;I'm doing a longitudinal study and trying to find out patterns before a defined event happens. For instance, we have data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID Date Amount Feature Event_ind&#xA;1  2010 5      M       0&#xA;1  2011 4      M       0&#xA;1  2012 3      M       1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The objective is to 'predict' the time when there would be an event happening, given historical data.&#xA;Currently, we are exploring whether we can convert this long table to a wide table. The table would look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ID Date1 Amount1 Date2 Amount2 Feature&#xA;1  2010  5       2011  4       M&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is, what are the &lt;strong&gt;implications of this conversion?&lt;/strong&gt; (or any good blog/academic paper explaining this)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of the implications that I can think of is, in a wide table each repeated measurements have its own distribution; and this conversion doesn't make sense if it's measured in different time or at different frequency.&lt;/p&gt;&#xA;" OwnerDisplayName="user52621" LastActivityDate="2018-05-24T01:09:00.130" Title="Longitudinal study - conversion long to wide - implications" Tags="|machine-learning|data-mining|dataset|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32078" PostTypeId="2" ParentId="12649" CreationDate="2018-05-24T01:13:19.770" Score="9" Body="&lt;p&gt;@StatsSorceress TL;DR:&lt;/p&gt;&#xA;&lt;p&gt;I'm going through this activity to see if I can calculate the memory required myself:&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Activations:&lt;/strong&gt; 532,752 * &lt;strong&gt;2&lt;/strong&gt; * 4 / (1024^2) = &lt;strong&gt;4.06&lt;/strong&gt; MB&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Parameters:&lt;/strong&gt; 19,072,984 * 4 / (1024^2) * 3 = 218.27 MB&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Miscellaneous:&lt;/strong&gt; 128 * 9,000 * 4 / (1024^2) = 4.39 MB&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Total Memory:&lt;/strong&gt; (4.06 * &lt;strong&gt;128&lt;/strong&gt;) + 218.27 + 4.39 = &lt;strong&gt;742.34 MB&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;(&lt;em&gt;Someone please correct me on this if I'm wrong. FYI, you already multiplied miscellaneous by 128, so that's why I didn't multiply it by 128 above&lt;/em&gt;)&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;p&gt;I would point you to &lt;a href=&quot;http://imatge-upc.github.io/telecombcn-2016-dlcv/slides/D2L1-memory.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;this article&lt;/a&gt; and &lt;a href=&quot;https://www.youtube.com/watch?v=Hqtg7fznlnM&quot; rel=&quot;nofollow noreferrer&quot;&gt;the corresponding video&lt;/a&gt;. They helped me to understand what is going on a lot better.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;NOTE:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;The memory required to use a network for predictions is far less than that required for training for two reasons:&lt;/em&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;When predicting, we only send an image forward through the network and not backward (so we don't multiply memory X 3; see below)&lt;/li&gt;&#xA;&lt;li&gt;There's one prediction per image (so we don't need to multiply the memory required for one image by a batch size because we don't use batches in prediction).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2&gt;Process (Memory to Train)&lt;/h2&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Calculate &lt;strong&gt;&lt;em&gt;the memory required to train on one image&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;Multiply this number by &lt;strong&gt;&lt;em&gt;the number of images in your batch&lt;/em&gt;&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;(&lt;strong&gt;REMEMBER:&lt;/strong&gt; &lt;em&gt;Mini-batching says we take a subset of our data, compute the gradients and errors for each image in the subset, then average these and step forward in the direction of the average. For convnets, weights and biases are shared, but the number of activations is mutliplied by the number of images in the batch.&lt;/em&gt;).&lt;/p&gt;&#xA;&lt;h2&gt;STEP 1: Memory for 1 Image&lt;/h2&gt;&#xA;&lt;p&gt;To train one image, you must reserve memory for:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Model Parameters:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;The &lt;strong&gt;weights&lt;/strong&gt; and &lt;strong&gt;biases&lt;/strong&gt; at each layer, their &lt;strong&gt;gradients&lt;/strong&gt;, and their &lt;strong&gt;momentum variables&lt;/strong&gt; (if Adam, Adagrad, RMSProp, etc., optimizers are used)&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;To approximate the memory for this, calculate the memory required to store the weights and biases and multiply that by 3 (i.e. &amp;quot;by 3&amp;quot; because we're saying the amount of memory needed to store the weights and biases is (roughly) equal to that needed for the gradients and for the momentum variables)&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;EQUATIONS:&lt;/em&gt;&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Convolutions:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;weights(n) = depth(n) * (kernel_width * kernel_height) * depth(n-1)&lt;/p&gt;&#xA;&lt;p&gt;biases(n) = depth(n)&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Fully Connected (Dense) Layers:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;weights(n) = outputs(n) * inputs(n)&lt;/p&gt;&#xA;&lt;p&gt;biases(n) = outputs(n)&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;where &lt;strong&gt;&lt;em&gt;n&lt;/em&gt;&lt;/strong&gt; is the current layer and &lt;strong&gt;&lt;em&gt;n-1&lt;/em&gt;&lt;/strong&gt; is the previous layer, and &lt;strong&gt;&lt;em&gt;outputs&lt;/em&gt;&lt;/strong&gt; are the number of outputs from the FC layer and &lt;strong&gt;&lt;em&gt;inputs&lt;/em&gt;&lt;/strong&gt; are the number of inputs to the FC layer (if the previous layer is not a fully-connected layer, the number of inputs is equal to the size of that layer flattened).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;NOTE:&lt;/em&gt;&lt;/strong&gt; The memory for the weights and biases alone, plus the memory for the activations for one image (see below), is the total amount of memory you need for predictions (excluding some overhead for memory for convolutions and some other things).&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Activations (these are &amp;quot;Blobs&amp;quot; in Caffe):&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;(I'm using terms loosely here, bear with me)&lt;/p&gt;&#xA;&lt;p&gt;Each convolution in a convolution layer produces &amp;quot;&lt;em&gt;number of pixels in image&lt;/em&gt;&amp;quot; activations (i.e. you pass an image through a single convolution, you get a single feature map consisting of &amp;quot;&lt;em&gt;m&lt;/em&gt;&amp;quot; activations, where &amp;quot;&lt;em&gt;m&lt;/em&gt;&amp;quot; is the number of pixels from your image/input).&lt;/p&gt;&#xA;&lt;p&gt;For fully-connected layers, the number of activations you produce is equal to the size of your output.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Convolutions:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;activations(n) = image_width * image_height * image_num_channels&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;Fully Connected (Dense) Layers:&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;p&gt;activations(n) = outputs(n)&lt;/p&gt;&#xA;&lt;p&gt;Note that your input is really only an image at the beginning of the network. After convolutions, it turns into something else (feature maps). So really replace &amp;quot;image_width&amp;quot;, &amp;quot;image_height&amp;quot;, and &amp;quot;image_num_channels&amp;quot; with &amp;quot;input_width&amp;quot;, &amp;quot;input_height&amp;quot;, and &amp;quot;layer_depth&amp;quot; to be more precise. (It's easier for me to think of this concept in terms of images.)&lt;/p&gt;&#xA;&lt;p&gt;Since we also need to store the error for the activations at each layer (used in the backward pass), we multiply the number of activations by 2 to get the total number of entities we need to make room for in our storage space. The number of activations increases with the number of images in the batch, so you multiply this number by the batch size.&lt;/p&gt;&#xA;&lt;h2&gt;STEP 2: Memory to Train Batch&lt;/h2&gt;&#xA;&lt;p&gt;Sum the number of weights and biases (times 3) and the number of activations (times 2 times the batch size). Multiply this by 4, and you get the number of bytes required to train the batch. You can divide by 1024^3 to get the answer in GB.&lt;/p&gt;&#xA;" OwnerUserId="52623" LastEditorUserId="135707" LastEditDate="2023-04-15T07:48:36.233" LastActivityDate="2023-04-15T07:48:36.233" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32079" PostTypeId="2" ParentId="25994" CreationDate="2018-05-24T01:16:04.397" Score="1" Body="&lt;p&gt;This question can have many ways to answer but here is what I think:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume the example of very common, facebook.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Very naive sense&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Number of friends is equal to the 'social-ness' of the person. In graph context, a node with more degree will considered to be strongly connected in the graph. But ofcourse we know this is not at all a good way of defining the social-ness.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Every human point of view&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Ask yourself a question about when do you like or comment in someone's post or photo ? The answer most probably will be when you and that person have good terms or friendship or relationship. So, with this kind of approach you can say that the person with more number of likes/comments in their posts is more social than the one with less likes/comments. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, coming to your question, it depends on how you define 'social' as I can also say a person with similar interest as the community with largest number of members with similar interest is most social. So, lets say we have a facebook graph and you want to measure social-ness of vertices you need to exploit the edge information also which is nothing but likes/comments/messages shared between two people. You can convert this edge data into some kind of number to denote the strength of that edge. Also, in the facebook scenario there is a directed edge between any two nodes (people) as it can happen that one person likes/comment/message the another person but vice-versa is not true. So, there are many more kinds of data/information that you can use in facebook graph.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then as answered in &lt;a href=&quot;https://datascience.stackexchange.com/questions/24081/what-are-graph-embedding&quot;&gt;this post&lt;/a&gt; graph embedding will give you the representation of a graph/subgraph as a whole. So, you need vertex embeddings in this case so you can represent  every vertex separately and compare against one another.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to use &lt;a href=&quot;http://www.perozzi.net/publications/14_kdd_deepwalk.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;deepwalk&lt;/a&gt; and get vector embeddings based on topology (you need to modify it a bit as the original method is only for undirected graphs). Then you can form another vector like [1050, 300, 500, 2, ...] where each element will be some kind of number denoting number of total likes, number of total comments, number of friends, frequency of messages, frequency of likes etc. You can then train a neural network which basically outputs a number between 1-5, 1 being least social and 5 being most social. So, in this way you can take advantage of vector embeddings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is one approach that I can think of. If you have any other thoughts please share them.&lt;/p&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2018-05-24T01:16:04.397" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32080" PostTypeId="2" ParentId="32051" CreationDate="2018-05-24T01:42:28.133" Score="3" Body="&lt;p&gt;a(i) : the average distance between 'i' and all other data within the same cluster (&lt;a href=&quot;https://en.wikipedia.org/wiki/Silhouette_(clustering)&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;b(i) :  the lowest average distance of 'i' to all points in any other cluster, of which 'i' is not a member (&lt;a href=&quot;https://en.wikipedia.org/wiki/Silhouette_(clustering)&quot; rel=&quot;nofollow noreferrer&quot;&gt;source&lt;/a&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, from the question, a(i) will be 24 as point 'Pi' belongs to cluster A and b(i) will be 48 as it is the least average distance that 'Pi' has from any other cluster than A (to which it belongs).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, as a(i) &amp;lt; b(i),  silhouette coefficient s(i) = 1 - 24/48 = 0.5&lt;/p&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2018-05-24T01:42:28.133" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32082" PostTypeId="2" ParentId="32066" CreationDate="2018-05-24T03:26:01.040" Score="4" Body="&lt;p&gt;There is really nothing special about the backpropagation algorithm in a generative adversarial network (GAN). It is the same as that of a convolutional neural network (CNN), as CNNs are usually what the generator and discriminator of the GAN are made of. I will assume the MNIST toy example for the explanation and I will provide code to get a GAN working below.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;The GAN&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;A GAN is composed of a discriminator and a generator. Each of these are held constant while training the other. We will thus alternate between training the discriminator and the generator. This is done separately. Training the discriminator is much easier so lets look at that, then we will look at training the generator as you asked in your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/roBI3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/roBI3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Training the discriminator&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;The discriminator has two output nodes and is used to distinguish real instances from artificial instances. To train the discriminator we will generate $m$ instances using forward passes from the generator, these are the artificial instances, their label will be $y = 0$. To generate these we simply pass a noise vector as the input to the model. We will also use $m$ instances from the real data these will have labels $y = 1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/8v3iu.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/8v3iu.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can then see that the discriminator is trained exactly the same way as a basic classification CNN with 2 output nodes. I will describe the training process of a CNN below.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Training the generator&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;When we train the generator we will keep the discriminator fixed, this is necessary as to not saturate our parameters and make the discriminator too strong to ever beat. So what we essentially have is a CNN (the generator) connected to another CNN (the discriminator). The connecting nodes between these two models is going to be output which will generate the desired images once trained. Do note that the generator wants the instances it will produce in this case to be classified by the discriminator as being from the real distribution, so they will have labels $y = 1$. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/mMo7z.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mMo7z.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All together this is just a CNN and the backpropagation will be computed in the exact same way. First we pass through a noise vector, it goes through the generator, some random image gets produced at its output, that then goes through the discriminator and gets classified as $artificial$. But, we are expecting the discriminator to be fooled in this case, so this is an error, it should have been labeled as $real$. We then use backpropagation to get the error contribution of each model parameter. We will then use gradient descent to update all the parameters associated with the generator. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Backpropagation&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;This is a method used to compute the contribution of each parameter on the error term. We then use gradient descent to update these parameters such that the next pass through should result in a lower loss rate. Picking the right loss function is essential for this process. For classification tasks, as is the case with a GAN, we typically choose binary cross entropy as defined by&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L = - ylog(\hat{y}) - (1-y)log(1-\hat{y})$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and over $N$ instances as is typically the case for stochastic gradient descent the loss function is &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$L(w) = - \frac{1}{N} \sum_{n = 1}^N \big[ y_n log(\hat{y}_n) + (1-y_n)log(1-\hat{y}_n) \big] $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $y$ is the true label and $\hat{y}$ is the predicted label.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Backpropagation in deep neural networks&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/questions/30676/role-derivative-of-sigmoid-function-in-neural-networks/30679#30679&quot;&gt;this answer&lt;/a&gt; here which describes the process of using backpropagation and gradient descent to train a single neuron perceptron, and then a multi-layered network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The only difference is we are using the binary entropy loss function here which has a different derivative with respect to $\hat{y}$. This becomes&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$\frac{\partial L}{\partial \hat{y}} = - \frac{1}{N} \sum_{n = 1}^N \Big[ \frac{y}{\hat{y}} - \frac{1-y}{1-\hat{y}} \Big]$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You will then backpropagate this loss through the network by using the chain rule with the activation functions you selected at each layer.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Backpropagation in a CNN&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Please refer to &lt;a href=&quot;https://datascience.stackexchange.com/questions/27506/back-propagation-in-cnn/27751#27751&quot;&gt;this answer&lt;/a&gt; for extensive details and a derivation of backpropagation for a CNN. The process is very similar to that for a deep neural network. However, the CNN uses the cross-correlation function at each layer, so you need to backpropagate the loss function through the derivative of this function. That question asks about the inner working of a CNN with two outputs very much like our discriminator. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, in order to train the generator, just imagine the same process with more convolutional layers ahead of it. When we have the associated gradient $\nabla L(w)$ for each parameters, only apply the gradient descent algorithm to the parameters that are a part of the generator model. I am sure that through these derivations you will see that the number of intermediate nodes between layers does not affect the backpropagation algorithm. It remains the same process. You should also be convinced that for a non-human the intermediate nodes that are the generated image from the GAN, are no different from any intermediate nodes in the model. We simply train these intermediate nodes in such a way that we can perceive some meaning, such as generating instances of the MNIST dataset. &lt;/p&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-05-24T03:55:40.950" LastActivityDate="2018-05-24T03:55:40.950" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32083" PostTypeId="1" CreationDate="2018-05-24T05:37:48.530" Score="5" ViewCount="1954" Body="&lt;p&gt;After going over the math/stats behind the &lt;strong&gt;Generative model&lt;/strong&gt; &amp;amp;  &lt;strong&gt;Discriminative model&lt;/strong&gt;, I still have no intuition about it. Does any one have a good real-world example or use &#xA; case for &lt;strong&gt;Generative model&lt;/strong&gt; and  &lt;strong&gt;Discriminative model&lt;/strong&gt; ? Thanks!&lt;/p&gt;&#xA;" OwnerUserId="17310" LastActivityDate="2018-05-24T06:07:34.540" Title="Real world example of Generative model &amp; Discriminative model" Tags="|generative-models|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32084" PostTypeId="2" ParentId="32083" CreationDate="2018-05-24T06:07:34.540" Score="4" Body="&lt;p&gt;A generative model is able to generate instances from a given distribution. So let's try to get our model to generate instance from the distribution of all hand written digits. This includes numbers 0 to 9 and for each of these numbers there are a very large set of possible way of writing the same number. Take the number 4 from the MNIST dataset, which are comprised of handwritten numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VyA2Y.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VyA2Y.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can quickly see how many different ways each of these numbers can be written yet still represent the same value. This is the distribution we want to capture using our generative model. The model works by training a generator to effectively try and cheat a discriminator that has been trained to distinguish between real and artificial instances from the desired distribution. When the discriminator can no longer distinguish between the artificial and real instances, we can say the generative model can successfully produce examples which match this distribution. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words, once the generator starts producing realistic digits, we can say that the generator is producing instances which match the instances captured in the MNIST dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ioeuP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ioeuP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This idea of matching a distribution can be applied to  wide range of applications where you want to draw from a distribution. Some for demonstration purposes of the techniques without any real useful application and some very useful applications which are currently being used for a number of tasks. Here is a short lists of some examples:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1706.03227.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generating artificial faces&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1711.10485.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Text to image generation&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1802.04208.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Producing fake voices or noises&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://stanford.edu/class/ee367/Winter2017/yan_wang_ee367_win17_report.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image denoising&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1803.01837.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Changing geometries within an image&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://cs231n.stanford.edu/reports/2017/pdfs/302.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Coloring images&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://arxiv.org/pdf/1709.00753.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;MRI image reconstruction&lt;/a&gt; &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Most importantly, to &lt;a href=&quot;https://www.youtube.com/watch?v=yz6dNf7X7SA&quot; rel=&quot;nofollow noreferrer&quot;&gt;generate new Pokemon&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In theory, given sufficient data, any distribution (anything in the world, including the world itself) can be modeled by a generative model from which we can draw instances. So any time you think of a simulation in the far future some generative model will be able to observe multiple real examples and then produce a large number of simulated examples. For example, simulating car crashes, a generative model can visualize multiple real crashes, then we can use the generative model to produce millions of similar instances for testing purposes without having to buy more cars. Of course GANs are not that powerful, they require a lot of data to learn very constraint distributions. However, through innovation anything is possible, until then we can continue to generate images of handwritten numbers and bedroom scenes.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-24T06:07:34.540" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32085" PostTypeId="2" ParentId="32074" CreationDate="2018-05-24T06:31:49.233" Score="-1" Body="&lt;p&gt;While you are trying this in R, I have a solution in Pyhton. You could use a similar logic in R:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Say this is your list:&#xA;countries = ['IN', 'DE', 'US', 'UK']&#xA;&#xA;# Create a list of countries in different continents like this:&#xA;asia = ['IN', 'NP']&#xA;america = ['US', 'CD']&#xA;europe = ['DE', 'UK', 'FR']&#xA;&#xA;# You can then map your list with these values using a definition:&#xA;for i in range(len(countries)):&#xA;    if countries[i] in asia:&#xA;        countries[i] = 'Asia'&#xA;    elif countries[i] in america:&#xA;        countries[i] = 'America'&#xA;    elif countries[i] in europe:&#xA;        countries[i] = 'Europe'&#xA;    else:&#xA;        countries[i] = 'Others'&#xA;&#xA;# Now check your list&#xA;countries&#xA;['Asia', 'Europe', 'America', 'Europe']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once this list is ready, you could create your boxplot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-24T06:31:49.233" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32087" PostTypeId="2" ParentId="32072" CreationDate="2018-05-24T07:01:36.120" Score="0" Body="&lt;p&gt;Synthetic gradients make training faster, not by reducing the number of epochs needed or by speeding up the convergence of gradient descent, but rather by making each epoch faster to compute.  The synthetic gradient is faster to compute than the real gradient (computing the synthetic gradient is faster than the backpropagation), so each iteration of gradient descent can be computed more rapidly.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-05-24T07:01:36.120" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32091" PostTypeId="1" CreationDate="2018-05-24T07:37:25.493" Score="1" ViewCount="489" Body="&lt;p&gt;I'm not familiar with the ways and tools of data scientists but I have a background in software engineering and mathematics. I'd need to recognize patterns in a two-dimensional data set. I have a lot of measurements of a valve's power consumption over time. The valve is under test. I'd need to tell based on the measurements if the valve is working properly or not. I have sample measurements of properly working and not properly working valves. This is a custom valve so it has no datasheet. I think I could start investigating this on my own but some insight from you would be really great. For example, what are some good tools I can use to analyze the data? Is there any recommended standard way of doing this? Are there any good references I could follow? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="52634" LastActivityDate="2018-05-24T07:37:25.493" Title="Pattern recognition in 2D dataset" Tags="|python|r|data-mining|dataset|algorithms|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-27T17:52:21.657" ContentLicense="CC BY-SA 4.0" />
  <row Id="32095" PostTypeId="2" ParentId="31267" CreationDate="2018-05-24T09:33:30.413" Score="1" Body="&lt;p&gt;I was also in a similiar situation. I had a dataset with number of stores and product categories and I needed to predict weekly sales for each category. You may want to check gradient boosting alghorithm and random forest which give promising results in terms of forecast accuracy. Also, regardless of the method, the best thing to do is to create dummy variables as creating single time series is very time consuming. &lt;/p&gt;&#xA;" OwnerUserId="52594" LastActivityDate="2018-05-24T09:33:30.413" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32096" PostTypeId="2" ParentId="31830" CreationDate="2018-05-24T09:42:38.593" Score="2" Body="&lt;p&gt;Checking &lt;a href=&quot;https://github.com/scikit-learn-contrib/imbalanced-learn/blob/f4271d9fe180ffd93c4cd02481d6ffa0ec35652d/imblearn/pipeline.py#L348&quot; rel=&quot;nofollow noreferrer&quot;&gt;the code on GitHub&lt;/a&gt;, it seems that a 'transform' performing the resampling of data is not applied (cf : 'if hasattr(transform, &quot;fit_sample&quot;'): pass). So, from my understanding only modifications that do not resample the test set will be applied.&lt;/p&gt;&#xA;" OwnerUserId="52644" LastActivityDate="2018-05-24T09:42:38.593" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32098" PostTypeId="2" ParentId="22008" CreationDate="2018-05-24T10:01:35.820" Score="0" Body="&lt;p&gt;In hive if you are doing using &lt;code&gt;drop&lt;/code&gt; to drop a table and if the table is external table then it will only delete the table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you are creating table it creates a directory in hadoop with the path you specified. But if you are not giving any path it will take the default directory (like: &lt;code&gt;/user/hive/warehouse/table_name&lt;/code&gt;)and create a directory with the table name. &#xA;When you load the data into external table from a text file than it will load data into table as well as it will also load the data into the hadoop path. &#xA;If you drop table then it will only delete the table since it is an external table it will not delete the data from hadoop.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Next time after drop when you recreate the table and reload the data it stores the new data and old data in hadoop path. That is why you are getting double counts. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you want to avoid this you have to delete the file from hadoop manually using &lt;code&gt;hadoop fs -rmr /path to directory/&lt;/code&gt;. Or before droping the table &lt;code&gt;truncate&lt;/code&gt; command.&lt;/p&gt;&#xA;" OwnerUserId="8729" LastActivityDate="2018-05-24T10:01:35.820" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32099" PostTypeId="1" CreationDate="2018-05-24T10:01:53.623" Score="3" ViewCount="82" Body="&lt;p&gt;I have data in a bad format that I want to make tidy using an R script. I lack the skills to convert it into the format I want, so I look for an answer that (a) outlines a method, or (b) hands me the entire script. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition of tidy here is that the output has the following columns:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;swap_id&lt;/code&gt; a unique identifier.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;leg&lt;/code&gt; either pay or rec.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;cashflow&lt;/code&gt; the amount that is recieved&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;date&lt;/code&gt; the date the cashflow arrives&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;type&lt;/code&gt; whether the cashflow is of INTEREST of FINAL_EX.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The variable &lt;code&gt;leg&lt;/code&gt; takes on two different values: pay and rec. Currently &lt;code&gt;leg&lt;/code&gt;=pay is in one excel sheet and &lt;code&gt;leg&lt;/code&gt;=rec is in a second excel sheet. The entire excel workbook hence contains two sheets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have uploaded the entire excel file to &lt;a href=&quot;https://we.tl/V9C5iVMH4w&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://we.tl/V9C5iVMH4w&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a sample of the content of sheet &quot;rec&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    id1                                 id2         &#xA;    date        cf      type            date        cf        type  &#xA;    2017-04-04  42961   INTEREST        2015-04-07  33953   INTEREST    &#xA;    2017-07-04  43438   INTEREST        2016-04-04  203161  INTEREST    &#xA;    2017-10-04  43915   INTEREST        2017-04-04  203161  INTEREST    &#xA;    2018-01-04  43915   INTEREST        2018-04-04  203161  INTEREST    &#xA;    2018-04-04  42961   INTEREST        2019-04-04  203161  INTEREST    &#xA;    2018-07-04  43438   INTEREST        2020-04-06  203161  INTEREST    &#xA;    2018-10-04  43915   INTEREST        2021-04-06  203161  INTEREST    &#xA;    2019-01-04  43915   INTEREST        2022-04-04  203161  INTEREST    &#xA;    2019-04-04  42961   INTEREST        2023-04-04  203161  INTEREST    &#xA;    2019-07-04  43438   INTEREST        2023-04-04  5016330 FINAL_EX    &#xA;    2019-10-04  43915   INTEREST                    &#xA;    2020-01-07  43915   INTEREST                    &#xA;    2020-04-06  43438   INTEREST                    &#xA;    2020-07-06  43438   INTEREST                    &#xA;    2020-10-05  43915   INTEREST                    &#xA;    2021-01-04  43915   INTEREST                    &#xA;    2021-04-06  42961   INTEREST                    &#xA;    2021-07-05  43438   INTEREST                    &#xA;    2021-10-04  43915   INTEREST                    &#xA;    2021-10-04  2988563 FINAL_EX&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here is a sample of the content of sheet &quot;pay&quot;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    id1                                 id2     &#xA;    date        cf      type            date        cf      type&#xA;    2017-04-04  5250    INTEREST        2015-04-07  30938   INTEREST&#xA;    2017-07-04  5308    INTEREST        2016-04-04  30938   INTEREST&#xA;    2017-10-04  5367    INTEREST        2017-04-04  30938   INTEREST&#xA;    2018-01-04  5367    INTEREST        2018-04-04  30938   INTEREST&#xA;    2018-04-04  5250    INTEREST        2019-04-04  30938   INTEREST&#xA;    2018-07-04  5308    INTEREST        2020-04-06  30938   INTEREST&#xA;    2018-10-04  5367    INTEREST        2021-04-06  30938   INTEREST&#xA;    2019-01-04  5367    INTEREST        2022-04-04  30938   INTEREST&#xA;    2019-04-04  5250    INTEREST        2023-04-04  30938   INTEREST&#xA;    2019-07-04  5308    INTEREST        2023-04-04  540000  FINAL_EX&#xA;    2019-10-04  5367    INTEREST                &#xA;    2020-01-06  5367    INTEREST                &#xA;    2020-04-06  5308    INTEREST                &#xA;    2020-07-06  5308    INTEREST                &#xA;    2020-10-05  5367    INTEREST                &#xA;    2021-01-04  5367    INTEREST                &#xA;    2021-04-06  5250    INTEREST                &#xA;    2021-07-05  5308    INTEREST                &#xA;    2021-10-04  5367    INTEREST                &#xA;    2021-10-04  315000  FINAL_EX                &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In other words, each id gets itws own little table. An every table is separated using an empty column. &#xA;This is good for eye balling the data, but horrible for working with it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is how I want the output to be structured after the transformation. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    swap_id leg date        cf      type&#xA;    id1     pay 2017-04-04  5250    INTEREST&#xA;    id1     pay 2017-07-04  5308    INTEREST&#xA;    id1     pay 2017-10-04  5367    INTEREST&#xA;    id1     pay 2018-01-04  5367    INTEREST&#xA;    id1     pay 2018-04-04  5250    INTEREST&#xA;    id1     pay 2018-07-04  5308    INTEREST&#xA;    id1     pay 2018-10-04  5367    INTEREST&#xA;    id1     pay 2019-01-04  5367    INTEREST&#xA;    id1     pay 2019-04-04  5250    INTEREST&#xA;    id1     pay 2019-07-04  5308    INTEREST&#xA;    id1     pay 2019-10-04  5367    INTEREST&#xA;    id1     pay 2020-01-06  5367    INTEREST&#xA;    id1     pay 2020-04-06  5308    INTEREST&#xA;    id1     pay 2020-07-06  5308    INTEREST&#xA;    id1     pay 2020-10-05  5367    INTEREST&#xA;    id1     pay 2021-01-04  5367    INTEREST&#xA;    id1     pay 2021-04-06  5250    INTEREST&#xA;    id1     pay 2021-07-05  5308    INTEREST&#xA;    id1     pay 2021-10-04  5367    INTEREST&#xA;    id1     pay 2021-10-04  315000  FINAL_EX&#xA;    id1     rec 2017-04-04  42961   INTEREST&#xA;    id1     rec 2017-07-04  43438   INTEREST&#xA;    id1     rec 2017-10-04  43915   INTEREST&#xA;    id1     rec 2018-01-04  43915   INTEREST&#xA;    id1     rec 2018-04-04  42961   INTEREST&#xA;    id1     rec 2018-07-04  43438   INTEREST&#xA;    id1     rec 2018-10-04  43915   INTEREST&#xA;    id1     rec 2019-01-04  43915   INTEREST&#xA;    id1     rec 2019-04-04  42961   INTEREST&#xA;    id1     rec 2019-07-04  43438   INTEREST&#xA;    id1     rec 2019-10-04  43915   INTEREST&#xA;    id1     rec 2020-01-07  43915   INTEREST&#xA;    id1     rec 2020-04-06  43438   INTEREST&#xA;    id1     rec 2020-07-06  43438   INTEREST&#xA;    id1     rec 2020-10-05  43915   INTEREST&#xA;    id1     rec 2021-01-04  43915   INTEREST&#xA;    id1     rec 2021-04-06  42961   INTEREST&#xA;    id1     rec 2021-07-05  43438   INTEREST&#xA;    id1     rec 2021-10-04  43915   INTEREST&#xA;    id1     rec 2021-10-04  2988563 FINAL_EX&#xA;    id2 …   …   …   … and so on... &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2022-04-22T21:02:30.303" Title="Convert from many sub-tables to a single tidy dataframe" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32100" PostTypeId="1" CreationDate="2018-05-24T10:16:38.010" Score="1" ViewCount="1042" Body="&lt;p&gt;As a self study exercise I am trying to understand the implementation of locally weighted regression (Loess) in python. Alexandre Gramfort (Sklearn developper) provides the following code on his &lt;a href=&quot;https://gist.github.com/agramfort/850437&quot; rel=&quot;nofollow noreferrer&quot;&gt;github page&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def lowess(x, y, f=2. / 3., iter=3):&#xA;    '''....the number of robustifying iterations is given by iter. The&#xA;    function will run faster with a smaller number of iterations'''&#xA;&#xA;    n = len(x)&#xA;    r = int(ceil(f * n))&#xA;    h = [np.sort(np.abs(x - x[i]))[r] for i in range(n)]&#xA;    w = np.clip(np.abs((x[:, None] - x[None, :]) / h), 0.0, 1.0)&#xA;    w = (1 - w ** 3) ** 3&#xA;    yest = np.zeros(n)&#xA;    delta = np.ones(n)&#xA;    for iteration in range(iter):&#xA;        for i in range(n):&#xA;            weights = delta * w[:, i]&#xA;            b = np.array([np.sum(weights * y), np.sum(weights * y * x)])&#xA;            A = np.array([[np.sum(weights), np.sum(weights * x)],&#xA;                          [np.sum(weights * x), np.sum(weights * x * x)]])&#xA;            beta = linalg.solve(A, b)&#xA;            yest[i] = beta[0] + beta[1] * x[i]&#xA;&#xA;        residuals = y - yest&#xA;        s = np.median(np.abs(residuals))&#xA;        delta = np.clip(residuals / (6.0 * s), -1, 1)&#xA;        delta = (1 - delta ** 2) ** 2&#xA;&#xA;    return yest&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h1&gt;Question&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;What I don't understand is the part related to the &lt;strong&gt;'robustifying iterations'&lt;/strong&gt;. How does this work and how could I learn more about this ? &lt;/p&gt;&#xA;&#xA;&lt;h3&gt;Vectorized implementation&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;For those who might be interested here is my attempt at describing the vectorized implementation mathematically.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Consider the 1D case where $\Theta = [\theta_0, \theta_1]$ and $x$ and $y$ are vectors of size $m$. The cost function $J(\theta)$ is a weighted version of the OLS regression, where the weights $w$ are defined by some kernel function&lt;/p&gt;&#xA;&#xA;&lt;p&gt;\begin{aligned}&#xA;	J(\theta) &amp;amp;= \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right)^2&#xA;	\\&#xA;	\frac{\partial J}{\partial \theta_0} &amp;amp;= -2 \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right) &#xA;	\\&#xA;	\frac{\partial J}{\partial \theta_1} &amp;amp;= -2 \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right) x^{(i)} &#xA;\end{aligned}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cancelling the $-2$ terms, equating to zero, expanding and re-arranging the terms: &#xA;\begin{aligned}&#xA;	&amp;amp; \frac{\partial J}{\partial \theta_0} = \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right)  = 0&#xA;	\\&#xA;	&amp;amp; \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  &amp;amp;\text{Eq. (1)}&#xA;	\\&#xA;	\\&#xA;	&amp;amp; \frac{\partial J}{\partial \theta_1} = \sum_{i=1}^m w^{(i)} \left( y^{(i)} - (\theta_0 + \theta_1 x^{(i)}) \right) x^{(i)}  = 0 &#xA;	\\&#xA;	&amp;amp; \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)} x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  x^{(i)} &amp;amp;\text{Eq. (2)}&#xA;\end{aligned}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Writing Eq. (1) and Eq. (2) in matrix form $\mathbf{A \Theta = b}$ allows us to solve for $\Theta$&#xA;\begin{aligned}&#xA;	&amp;amp; \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  &#xA;	\\&#xA;	&amp;amp; \sum_{i=1}^m w^{(i)}  \theta_0 + \sum_{i=1}^m w^{(i)}  \theta_1 x^{(i)} x^{(i)}  = \sum_{i=1}^m w^{(i)} y^{(i)}  x^{(i)}&#xA;	\\&#xA;	&amp;amp; \begin{bmatrix} \sum w^{(i)} &amp;amp; \sum w^{(i)} x^{(i)} \\ \sum w^{(i)} x^{(i)} &amp;amp; \sum w^{(i)} x^{(i)} x^{(i)} \end{bmatrix}  \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}   = \begin{bmatrix}  \sum w^{(i)} y^{(i)} \\  \sum w^{(i)} y^{(i)} x^{(i)} \end{bmatrix}  &#xA;	\\&#xA;	&amp;amp; \mathbf{A} \Theta = \mathbf{b}&#xA;	\\&#xA;	&amp;amp;  \Theta = \mathbf{A}^{-1} \mathbf{b}&#xA;\end{aligned}&lt;/p&gt;&#xA;" OwnerUserId="52603" LastEditorUserId="1330" LastEditDate="2021-10-05T16:21:11.430" LastActivityDate="2021-10-05T16:21:11.430" Title="Locally Weighted Regression (Loess) - Robustifying iterations" Tags="|machine-learning|python|regression|mathematics|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32102" PostTypeId="1" CreationDate="2018-05-24T11:43:13.593" Score="1" ViewCount="19" Body="&lt;p&gt;When working with a 3D laser sensor (LiDAR), the volumetric point density versus distance &lt;span class=&quot;math-container&quot;&gt;$\rho_r$&lt;/span&gt; can be theoretically worked out taking into account the physical properties of the laser (number of layers, TOF, etc.). On the other hand, it can also be computed &lt;span class=&quot;math-container&quot;&gt;$\hat \rho_r$&lt;/span&gt;  from the available training data, yielding into an estimation of this quantity.&lt;/p&gt;&#xA;&lt;p&gt;When this density vs. distance is wanted to be included in a classification problem (in the preprocessing stage as parameter) what value should be regarded? I share my concerns about the two different approaches:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Theoretical values&lt;/strong&gt; &lt;span class=&quot;math-container&quot;&gt;$\rho_r$&lt;/span&gt; : It will give an upper bound, as it is only taking into account the nature of the sensor. Furthermore, if only physical properties from the data-sheet are used and no statistical information about the uncertainty of the measurements are regarded, this value will be deterministic.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;Empirical values&lt;/strong&gt; &lt;span class=&quot;math-container&quot;&gt;$\hat \rho$&lt;/span&gt; : they collect both the nature of the sensor and the scenario. A clear disadvantage is that, it will work well for one scenario and not for others. If the estimator is consistent and asymptotically unbiased, the more data the better the parametrization.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The classification task is approached with CNN. Should the empirical value be used only for specific types of scenarios/applications? If a general classification model is itended, should the theoretical value be used, or having enough variability on the training data will make the empirical value work better?&lt;/p&gt;&#xA;" OwnerUserId="51714" LastEditorUserId="84263" LastEditDate="2021-10-06T03:24:23.810" LastActivityDate="2021-10-06T03:24:23.810" Title="Parametrization of a model with data-sheet's sensor information or with empirical data" Tags="|data|cnn|sensors|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32103" PostTypeId="1" AcceptedAnswerId="49124" CreationDate="2018-05-24T11:44:56.247" Score="2" ViewCount="419" Body="&lt;p&gt;I have a dataset as below and I want to see 3 days high low average, back from for each day in the same plot.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    index   closed  high    low     open&#xA;tarih                   &#xA;2018-05-18  1364    0.00000442  0.00000445  0.00000405  0.00000416&#xA;2018-05-19  1365    0.00000458  0.00000465  0.00000427  0.00000442&#xA;2018-05-20  1366    0.00000482  0.00000489  0.00000448  0.00000455&#xA;2018-05-21  1367    0.00000489  0.00000492  0.00000463  0.00000482&#xA;2018-05-22  1368    0.00000475  0.00000492  0.00000465  0.00000487&#xA;2018-05-23  1369    0.00000461  0.00000486  0.00000450  0.00000475&#xA;2018-05-24  1370    0.00000462  0.00000480  0.00000455  0.00000464&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;for example for 05-24 it will calculate 21,22,23 days high low average and point it to 05-24.&lt;/p&gt;&#xA;" OwnerUserId="52651" LastEditorUserId="201" LastEditDate="2019-04-03T06:09:15.503" LastActivityDate="2019-04-11T13:02:33.997" Title="n days back average for each day" Tags="|python|pandas|matplotlib|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32104" PostTypeId="1" CreationDate="2018-05-24T12:06:19.170" Score="1" ViewCount="101" Body="&lt;p&gt;I'm trying to compare two images (the first one is the ID's image and the second one is a selfie taken by phone) so I am wondering if can we extract features from the two images by using CNN and comparing the two vectors by using the cosine  or the Euclidean distance.&#xA;sift and Gabor didn't show a good result&lt;/p&gt;&#xA;" OwnerUserId="52653" LastEditorUserId="1330" LastEditDate="2021-10-07T15:51:40.560" LastActivityDate="2021-10-07T15:51:40.560" Title="Comparing two images with features extracted from a CNN" Tags="|neural-network|feature-extraction|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32105" PostTypeId="1" AcceptedAnswerId="32107" CreationDate="2018-05-24T12:07:07.207" Score="3" ViewCount="1017" Body="&lt;p&gt;I'm a new programmer and this is my first ever neural network for real world application. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the deal, I'm using a top-less pre-trained VGG-16 with some dense layers on top of it.(for image classification problem) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But no matter what hyper parameters I change I always get plots similar to these.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/OW0nP.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/OW0nP.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZqgdR.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZqgdR.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So my question is at what loss (val_loss) can we consider the model trained?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And is there a problem if the images that I feed the VGG-16 are RGB/255?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for taking time to answer me, and sorry if the questions seems stupid but I didn't find answers when searching else where.&lt;/p&gt;&#xA;" OwnerUserId="52411" LastActivityDate="2018-05-24T18:35:45.390" Title="Is this an over-fitting case?" Tags="|image-classification|overfitting|vgg16|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32107" PostTypeId="2" ParentId="32105" CreationDate="2018-05-24T13:07:15.843" Score="2" Body="&lt;p&gt;Your model is indeed overfitting. There can be a lot of reasons why it could be happening - How many images are you training with?, Aren't you using Regularization, data augmentation using random crops or flips, Dropout to prevent overfitting?&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;So my question is at what loss (val_loss) can we consider the model&#xA;  trained?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We can generally stop training when Validation loss doesn't decrease anymore. This is generally done by defining a variable eg: n = 5 and checking if the validation loss has decreased after running n epochs. If you're solving a well-known problem(MNIST, Fashion-MNIST, CIFAR-10, etc), you can check the best score/loss achievable by the model and tweak your model(add or remove layers, prevent overfitting, etc) to achieve the score. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And is there a problem if the images that I feed the VGG-16 are&#xA;  RGB/255?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You should preprocess the images for faster training and also subtract  the mean RGB pixel values of images used to train pretrained network(if the pretrained net is trained using imagenet data,then subtract imagenet dataset's mean RGB pixel values from your input images) from your input images. &lt;a href=&quot;https://github.com/keras-team/keras/blob/master/keras/applications/imagenet_utils.py&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; you can read keras preprocessing_input &#xA;method.  &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Validation loss measures how generalizable your model is to unseen data. If your training loss and validation loss don't improve during training, then you are underfitting. If you're training loss is low but your validation loss is stagnant or increasing then you're overfitting, which means that your model is learning unwanted noise or patterns which help it learn or memorize training data but are not generalizable on an unseen validation set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is a link explaining data preprocessing, augmentation, transfer learning using pretrained net.&lt;/p&gt;&#xA;" OwnerUserId="39208" LastActivityDate="2018-05-24T13:07:15.843" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32108" PostTypeId="2" ParentId="32099" CreationDate="2018-05-24T13:32:02.900" Score="1" Body="&lt;pre&gt;&lt;code&gt;## read file &#xA;# create path &#xA;file4 &amp;lt;- &quot;GetSwapFlowdb2.xlsm&quot;&#xA;filepath3 &amp;lt;- paste(folder1, file4, sep=&quot;&quot;)&#xA;# read. sheet 2 is recieve leg, and 3 is pay leg&#xA;db_rec &amp;lt;- read_excel(filepath3, col_names = FALSE, sheet=2) %&amp;gt;% as.data.frame()&#xA;db_pay &amp;lt;- read_excel(filepath3, col_names = FALSE, sheet=3) %&amp;gt;% as.data.frame() &#xA;&#xA;## save swap IDs &#xA;(transid_list &amp;lt;- db_rec[1,] %&amp;gt;% t() %&amp;gt;% na.omit() %&amp;gt;% as.vector()) #%&amp;gt;% as.numeric()&#xA;transid_list_pay &amp;lt;- db_pay[1,] %&amp;gt;% t() %&amp;gt;% na.omit() %&amp;gt;% as.vector()&#xA;stopifnot(transid_list_pay == transid_list)&#xA;rm(transid_list_pay)&#xA;(nr_id &amp;lt;- length(transid_list))&#xA;&#xA;## make function that cleans a single swap&#xA;db_clean_oneswap &amp;lt;- function(aswap){&#xA;  # input...&#xA;  # aswap: a single swap in the form fo data.frame&#xA;  # output...&#xA;  # aswpap: the cleaned version&#xA;&#xA;  # format cf &#xA;  aswap$cf &amp;lt;- as.numeric(aswap$cf)&#xA;  # format dates from excel format to yyyy-mm-dd&#xA;  aswap$date &amp;lt;- as.POSIXct(as.Date(as.numeric(aswap$date), origin=&quot;1899-12-30&quot;))&#xA;  aswap$date &amp;lt;- as.Date(aswap$date)&#xA;  # create t &#xA;  aswap$t &amp;lt;- time_length(difftime(aswap$date, date_valuation), &quot;years&quot;)&#xA;  # save the swap to our list &#xA;  return(aswap)&#xA;}&#xA;&#xA;&#xA;## make function so we can save swap cashflows to a list &#xA;db_create_swaplist &amp;lt;- function(data, transids){&#xA;  # input... &#xA;  # data: the dataframe which comes from excel &#xA;  # transids: a list character vector of swap trans IDs&#xA;  # output... &#xA;  # a list with each element containing a dataframe of the swap &#xA;&#xA;  # create empty list&#xA;  swap_list &amp;lt;- NULL&#xA;&#xA;  # for loop to create a list containing each swap. &#xA;  for (id in transids){&#xA;    # create df we will &quot;eat&quot; from &#xA;    data &amp;lt;- db_rec&#xA;    # select three cols, save to a swap&#xA;    one_swap &amp;lt;- data[-1, c(1,2,3)]&#xA;    # remove those cols we saved &#xA;    data &amp;lt;- data[-c(1,2,3)]&#xA;    # rename cols &#xA;    names(one_swap) &amp;lt;- c(&quot;date&quot;, &quot;cf&quot;, &quot;type&quot;)&#xA;    # remove a row, remove na &#xA;    one_swap &amp;lt;- one_swap[-1, ]&#xA;    one_swap &amp;lt;- na.omit(one_swap)&#xA;&#xA;    # clean it&#xA;    one_swap &amp;lt;- db_clean_oneswap(one_swap)&#xA;    # save the swap to our list &#xA;    swap_list[[id]] &amp;lt;- one_swap&#xA;  }&#xA;  return(swap_list)&#xA;}&#xA;&#xA;## create a list for rec and pay&#xA;swaps_rec &amp;lt;- db_create_swaplist(data = db_rec, transids = transid_list)&#xA;swaps_pay &amp;lt;- db_create_swaplist(data = db_pay, transids = transid_list)&#xA;&#xA;## ways of accessing the lists&#xA;swaps_rec[[1]]&#xA;transid_list[1]&#xA;swaps_rec[[ transid_list[2] ]]&#xA;swaps_rec[[as.character(transid)]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="45191" LastActivityDate="2018-05-24T13:32:02.900" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32109" PostTypeId="1" CreationDate="2018-05-24T14:28:21.943" Score="16" ViewCount="24297" Body="&lt;p&gt;I'm studying Data Scaling, and in particular the Standardization method.&#xA;I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit variance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you explain me ?&lt;/p&gt;&#xA;" OwnerUserId="46375" LastEditorUserId="42498" LastEditDate="2018-05-25T14:46:16.913" LastActivityDate="2019-05-25T06:12:24.967" Title="Zero Mean and Unit Variance" Tags="|machine-learning|feature-scaling|normalization|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32110" PostTypeId="1" CreationDate="2018-05-24T14:46:29.343" Score="2" ViewCount="1046" Body="&lt;h1&gt;NLP Analysis for keyword clustering&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I have a set of keywords for search engines and I would like to create a python script to classify and tag them under unknown categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To make it clear I should have an output like this one, without knowing the categories (Product, Colour, Accessory, Brand...):&lt;br&gt;&#xA;+----------------------------+------------+----------+--------------+-----------+&lt;br&gt;&#xA;|.......Keywords............|.Product...|.Colour.|.Accessory.|.Brand...|&lt;br&gt;&#xA;+----------------------------+------------+----------+--------------+-----------+&lt;br&gt;&#xA;|.red shoes with heels.|.shoes......|.red......|.heels.........|..............|&lt;br&gt;&#xA;|.Apple computer.........|.computer.|............|..................|.Apple....|&lt;br&gt;&#xA;|.Armani blue shoes....|.shoes......|.blue.....|..................|.Armani..|&lt;br&gt;&#xA;|.black mouse..............|.mouse.....|.black...|..................|..............|&lt;br&gt;&#xA;|.gaming laptop...........|.computer.|.............|..................|..............|&lt;br&gt;&#xA;+----------------------------+------------+----------+--------------+-----------+  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestions on how I could be able to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently using Word2Vec to find similarities between words and some APIs to recognize Brands and entities in the keywords&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    model2 = models.Word2Vec.load('semantic_clustering/datasets/it/it.bin')&#xA;    with open('tmp/kw_msm.txt') as f:&#xA;        kwlist = f.readlines()&#xA;    kwlist = [x.strip() for x in kwlist&#xA;&#xA;    # unique_words = list(set([word for word in kw.split(' ') for kw in kwlist]))&#xA;    freq_words = defaultdict(int)&#xA;    words = set()&#xA;    for kw in kwlist:&#xA;        for word in kw.split(' '):&#xA;            freq_words[word] += 1&#xA;            words.add(word)&#xA;    sorted_freq_words = sorted(freq_words.items(), key=operator.itemgetter(1), reverse=True)&#xA;&#xA;    # Creating Dataframe with words as columns&#xA;    df = pd.DataFrame(columns=words)&#xA;    df['keyword'] = kwlist&#xA;    for i, row in df.iterrows():&#xA;        for w in row['keyword'].split():&#xA;            df.loc[i, w] = 1&#xA;&#xA;    good_words = [w for w in words if kw_in_vocab(w, model2)]&#xA;&#xA;    KW_MODEL = model2[good_words]&#xA;    NUM_CLUSTERS = 10&#xA;    kclusterer = KMeansClusterer(NUM_CLUSTERS, distance=nltk.cluster.util.cosine_distance, repeats=25)&#xA;    assigned_clusters = kclusterer.cluster(KW_MODEL, assign_clusters=True)&#xA;    clusters = {}&#xA;    for i, w in enumerate(good_words):&#xA;        clusters[w] = assigned_clusters[i]&#xA;    sorted_clusters = sorted(clusters.items(), key=operator.itemgetter(1))&#xA;    for k in sorted_clusters:&#xA;        print(k)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is a snippet of code I am using, creating a sparse matrix of words and clustering the columns with a fixed number of clusters, it's only a first test&lt;/p&gt;&#xA;" OwnerUserId="52664" LastActivityDate="2018-05-24T14:46:29.343" Title="Automatic Semantic Clustering and Tagging of sentences using NLP" Tags="|python|nlp|clustering|word2vec|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32111" PostTypeId="2" ParentId="32109" CreationDate="2018-05-24T15:21:30.377" Score="12" Body="&lt;p&gt;The questions of whether and why it's important, depends on the context.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;For gradient boosted decision trees, for example, it is not important - these ML algorithms &quot;don't care&quot; about monotone transformations to the data; they just look for points to split it.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For linear predictors, for example, scaling can improve interpretability of the results. If you'd like to think of the magnitude of the coefficients as some indication of how much a feature is affecting outcome, then the features has to be scaled somehow to the same area.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;For some predictors, in particular NNs, scaling, and in particular scaling to a particular range, can be important for technical reasons. Some of the layers use functions that effectively change only within some area (similar to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Hyperbolic_function&quot; rel=&quot;noreferrer&quot;&gt;hyperbolic-family of functions&lt;/a&gt;), and if the features are too much out of the range, saturation can occur. If this happens, numerical derivatives will work badly, and the algorithm might not be able to converge to a good point.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/nuZAD.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/nuZAD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="9518" LastActivityDate="2018-05-24T15:21:30.377" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32112" PostTypeId="1" CreationDate="2018-05-24T15:22:19.610" Score="3" ViewCount="1037" Body="&lt;p&gt;So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Premise&lt;/strong&gt;: There is a 2D grid space of which an agent needs to visit all the cells (because it is searching the space), once a cell is visited it is considered &quot;searched&quot; and therefore does not need to be visited again.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;First Take&lt;/strong&gt;: I thought well what if the state representation of the environment to the agent was just the 8 adjacent cells and their searched/unsearched status. In this case the resulting actions would be to move to one of the adjecent cells. Here we have 2^8 different possible states and then 8 actions for our Q(s,a) function. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since I am using a DQN structure to the problem, the Q function would be approximated by a neural network with 8 inputs (status of adjacent cells) and 8 outputs (actions). Obviously the reward function would be simple as it yields a high reward for moving to a cell that has not been searched and a low reward otherwise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, this presents an obvious problem when all the adjecent cells have been searched. The agent would move randomly and possibly away from cells that need to be searched. This is not good for my purposes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Second Take&lt;/strong&gt;: I thought if I wanted to work around this I would need to incorporate information about the entire state-space into my Q function. Meaning that the agent knows the status of all cells and has actions that are basically &quot;waypoints&quot; that bring the agent to that cell. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This would make my inputs and outputs to the Q function A LOT larger as I have a really big state-space. I don't have a specific size yet, but it could easily be 10,000 (100x100 grid) or larger. And obviously my reward function would need to reward the agent more for actions that involve less movement. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, my problem can be formulated in a simulation to train the Q function, so I'm not worried about training time due to physical constraints.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So basically my question is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can this Q function still be approximated by a neural network of reasonable size? Yes, I have a lot of processing power at my disposal (sorry I know this is all very vague), but I'd rather not blow things out of proportion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;tldr: I have Deep Q-Learning problem where I think I need a large state-action space, and am wondering if it is too large to be reasonably approximated by a neural network.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;: As @Constantinos noted, I didn't really define my problem well enough, so here are more assumptions and definitions below.&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;State-Space:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The grid itself is not exactly the state-space, each cell in the grid has a &quot;searched&quot; or &quot;unsearched&quot; status so the total number of unique states that my state-space can take on is really along the lines of 2^10,000 (assuming 100x100 grid). &lt;/li&gt;&#xA;&lt;li&gt;Additionally, the state-space also contains information about the location of the agent.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Action-Space:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;At any given position, the agent may move to any other location on the grid. Therefor the action space is simply the 10,000 moves available.&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Reward Function:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Positive reward for choosing action that brings agent to unsearched location.&lt;/li&gt;&#xA;&lt;li&gt;Negative reward for choosing action that brings agent to location that is far away (to constrain movement).&lt;/li&gt;&#xA;&lt;li&gt;High positive reward for choosing action that brings the agent to a goal state (i.e. where all locations have been searched).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;I/O of Q-Network:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The input will be the a vector of elements (each representing a cell on the grid, can also be seen as an 'image feature' where the image is the grid) that take on values 0 or 1 (unsearched or searched).&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;The input will also have to include some embedded information on the agents location (I am currently thinking of just altering the value of the input element that represents the agents location cell to be -1).&lt;/li&gt;&#xA;&lt;li&gt;Output of the network will be a vector of action utility values, each representing the utility of moving to a certain grid location.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Lastly, I should note that once an agent visits a location, that location is now considered &quot;searched&quot; and the state of the environment is updated accordingly.&lt;/p&gt;&#xA;" OwnerUserId="52670" LastEditorUserId="52670" LastEditDate="2018-05-29T20:38:52.203" LastActivityDate="2018-05-29T20:38:52.203" Title="Defining State Representation in Deep Q-Learning" Tags="|machine-learning|neural-network|reinforcement-learning|" AnswerCount="0" CommentCount="15" ContentLicense="CC BY-SA 4.0" />
  <row Id="32113" PostTypeId="1" CreationDate="2018-05-24T15:29:11.483" Score="3" ViewCount="106" Body="&lt;p&gt;I'm looking for a reference or point in the right direction since I'm not too familiar with machine learning or algorithms. I'd prefer to work in R, but I could also do Python. Any reference to a specific package I could use would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So I have a list of Senate bills and their descriptions that are one or two sentences long. I have a subset of bills that also have accompanied bill types (ie &quot;budget&quot;, &quot;immigration&quot;, etc). I want to create an algorithm that will assign a bill type(s) to the other subset of bills that don't have bill types.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would want to be able to do this by using the underlying relationship between descriptions and bill types for the first subset and apply it to the second so that I can predict bill type from bill description. Does this make sense? Any advice would be welcome!&lt;/p&gt;&#xA;" OwnerUserId="52672" LastActivityDate="2018-07-23T17:17:30.433" Title="Machine Learning/Text Analysis/Algorithm Pointers" Tags="|machine-learning|r|algorithms|learning|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32114" PostTypeId="1" AcceptedAnswerId="32125" CreationDate="2018-05-24T15:40:18.300" Score="1" ViewCount="566" Body="&lt;p&gt;Softmax cross entropy with logits can be defined as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$l={\sum_{\forall i}}y_ilog(a_i)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where $l$ is the actual loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when you look deep inside into &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/bc86da090f2f2e850768bbdfd603c7217aecdb53/tensorflow/compiler/tf2xla/kernels/softmax_op.cc#L79&quot; rel=&quot;nofollow noreferrer&quot;&gt;C++ Tensorflow implementation&lt;/a&gt; of &lt;code&gt;SoftmaxCrossEntropyWithLogits&lt;/code&gt; operation, the exact formula which they use is descibed as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$l={\sum_{\forall j}}y_j ((z_j-max(z))-log({\sum_{\forall i}}e^{z_i-max(z)}))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The part: $z-max(z)$ - is perfectly understood - it is just &lt;strong&gt;normalization&lt;/strong&gt; which helps to avoid under/overflow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;BUT:&lt;/strong&gt; &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Where is the actual &lt;code&gt;Softmax&lt;/code&gt; in their implementation?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Why from each $z_j$ they subtract $log({\sum_{\forall i}}e^{z_i-max(z)})$ before multiply it by $y_j$?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Note: One may argue that the code I provide is just Tensorflow's implementation of &lt;code&gt;CrossEntropyWithLogits&lt;/code&gt; operation, but the actual &lt;a href=&quot;https://github.com/tensorflow/tensorflow/blob/bc86da090f2f2e850768bbdfd603c7217aecdb53/tensorflow/compiler/tf2xla/kernels/softmax_op.cc#L132&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;SoftmaxCrossEntropyWithLogits&lt;/code&gt;&lt;/a&gt; operation - additionaly checks only dimentions and do not perform any more computation.&lt;/em&gt; &lt;/p&gt;&#xA;" OwnerUserId="52669" LastEditorUserId="52669" LastEditDate="2018-05-24T16:00:49.813" LastActivityDate="2018-05-24T18:25:44.210" Title="Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit" Tags="|tensorflow|loss-function|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32115" PostTypeId="2" ParentId="32023" CreationDate="2018-05-24T15:50:01.237" Score="2" Body="&lt;p&gt;Lateral connections exist so that the update of a neuron forces the neighboring neurons also to be updated but to a lesser degree. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of each neuron that has &lt;em&gt;n&lt;/em&gt; inputs as a vector, a point in an &lt;em&gt;n&lt;/em&gt;-dimensional space. It's coordinates are the values of the weights on its inputs. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When the network receives an input - also an n-dimensional vector, each neuron calculates the distance between its weights and the input vector. The neuron whose weights are the closest to the input is the winner, and is allowed to update its weights. It updates them by moving one step towards the input vector. The size of the step is equal to the learning rate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While it moves, it uses the lateral connections to pull its neighbors (or push some of them away, depending on their distance to the winner and on the form of the function that you use for the lateral connections). As a result, the neighbors also move but they move less than the winner. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Only the winner and its neighbors are allowed to update their weights. All other neurons don't move. The neighbors are those who are closer to the winner than a certain radius. This radius is a hyper-parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This type of learning allows to chart the space of the input vectors. The result is the weights of the neurons are distributed more or less uniformly in the set of the input vectors. After training, each neuron can be thought of as a representative of some region of the input space.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-24T15:50:01.237" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32116" PostTypeId="2" ParentId="32113" CreationDate="2018-05-24T15:52:14.790" Score="2" Body="&lt;p&gt;Yes, This absolutely makes sense. This is a common NLP (Natural Language Processing) problem. You should use word embedding models alingside LSTM (Long Short Term Memory) and deep neural networks. Actually, first you should represent every word as a vector of fixed dimension (e.g. 100) using word2vec. Then you should build a deep neural network architecture. The inputs of this network are your word vectors for each bill that are concatenated to form a sequence of vectors. The output of this network is a label that indicate which type of bill you are considering. I highly recommend that you use &lt;strong&gt;&lt;em&gt;Keras&lt;/em&gt;&lt;/strong&gt; which is a great python package for dealing with deep learning and NLP. Also, you can use existing word2vec sets that contain vector representations of most of words of a language (e.g. English). For example you can use GoogleNews word2vec set.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastEditorUserId="51209" LastEditDate="2018-05-24T16:47:58.403" LastActivityDate="2018-05-24T16:47:58.403" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32118" PostTypeId="2" ParentId="32031" CreationDate="2018-05-24T16:14:49.830" Score="1" Body="&lt;p&gt;I gave some explanations &lt;a href=&quot;https://datascience.stackexchange.com/questions/32023/about-neural-network-called-self-organizing-maps/32115#32115&quot;&gt;here&lt;/a&gt; that may help.&lt;/p&gt;&#xA;&lt;p&gt;The radius &lt;span class=&quot;math-container&quot;&gt;$r_{ij}$&lt;/span&gt; is the radius of the neighborhood of the winner. Only those neurons that are closer than &lt;span class=&quot;math-container&quot;&gt;$r_{ij}$&lt;/span&gt; to the winner are allowed to update.&lt;/p&gt;&#xA;&lt;p&gt;The formulas that you show here say that this radius is not constant but decreases with each iteration. Initially you take a large radius, and then make it smaller and smaller with each iteration.&lt;/p&gt;&#xA;&lt;p&gt;Since you probably initialize your weights randomly, you might want to change many of them in the beginning of training but you might want to disturb less and less weights as they become more trained. That's why you decrease the radius.&lt;/p&gt;&#xA;&lt;p&gt;This formula is just a particular policy of change of the radius with each iteration. In this case it is exponential. You can choose another policy or keep the radius constant or decrease it slightly (not exponentially) which may be reasonable if you use some domain knowledge when initializing the weights. Also, you can keep the learning rate constant if you like. The radius of the neighborhood and the learning rate are hyper-parameters. It is up to you how to choose them.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastEditorUserId="51836" LastEditDate="2021-02-15T02:36:05.650" LastActivityDate="2021-02-15T02:36:05.650" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32120" PostTypeId="2" ParentId="32031" CreationDate="2018-05-24T16:26:15.510" Score="0" Body="&lt;p&gt;The radius for a SOM is used to update weights of neighboring nodes as well as the winning node.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically SOM nodes are organized on a 2D lattice and the neighborhood is calculated with Euclidean distance.  Thus the nodes develop a spatial relationship between each other in 2D, with different areas of the lattice representing different clusters.&lt;/p&gt;&#xA;" OwnerUserId="38751" LastActivityDate="2018-05-24T16:26:15.510" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32121" PostTypeId="1" AcceptedAnswerId="32124" CreationDate="2018-05-24T17:00:40.280" Score="2" ViewCount="133" Body="&lt;p&gt;Let's take the MNIST dataset (My application is different) with a lot of noise, I am going to train a deep NN to classify the letters.&lt;/p&gt;&#xA;&lt;p&gt;What's the right way to infer, there's no letter possibility? or a letter not included in the training set?&lt;/p&gt;&#xA;&lt;p&gt;Do I have to add the class &amp;quot;not a letter&amp;quot;  and try to pick or simulate the part of the dataset that has no letters but noise,&lt;/p&gt;&#xA;&lt;p&gt;Or, I can do it without adding the extra label?&lt;/p&gt;&#xA;" OwnerUserId="52678" LastEditorUserId="29169" LastEditDate="2021-02-13T04:20:35.103" LastActivityDate="2021-02-13T04:20:35.103" Title="How can I infer no target in a target classification problem based on deep learning?" Tags="|machine-learning|deep-learning|neural-network|convolutional-neural-network|mnist|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32122" PostTypeId="2" ParentId="32109" CreationDate="2018-05-24T17:11:00.653" Score="3" Body="&lt;p&gt;In case of zero mean, that is because some machine learning models do not include bias term in their representation so we have to move data around origin before feeding it to the algorithm to conpensate for lack of bias term. In case of unit variance, that is because lots of machine learning algorithms use some kind of distance (e.g. Euclidean) to decide or predict. If a particular feature has broad values (i.e. large variance), the distance will be highly affected by that feature and the effect of other features will be ignored. By the way, some optimization algorithms (including gradient descent) have better performance when data is standardized.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-24T17:11:00.653" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32123" PostTypeId="2" ParentId="29470" CreationDate="2018-05-24T17:23:30.183" Score="0" Body="&lt;p&gt;I recommend that you use word2vec to represent a word as a vector. After that you can feed your sequence of word vectors to LSTM to get vector representation of your sentence or paragraph.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-24T17:23:30.183" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32124" PostTypeId="2" ParentId="32121" CreationDate="2018-05-24T18:17:09.320" Score="1" Body="&lt;p&gt;What you are looking for is &lt;em&gt;anomaly detection&lt;/em&gt;, or &lt;em&gt;novelty detection&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This cannot always be solved by labeling some images as &quot;no letter&quot;, since all kind of images having no letters may not be available for labeling, or maybe too costly to exhaustively label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;http://www.inf-cv.uni-jena.de/novelty_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt; has some directions you can look at for a start.&lt;/p&gt;&#xA;" OwnerUserId="33759" LastActivityDate="2018-05-24T18:17:09.320" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32125" PostTypeId="2" ParentId="32114" CreationDate="2018-05-24T18:17:59.747" Score="1" Body="&lt;p&gt;As I understand it, the softmax function for $z_i$ is given by $a_i$. Then just taking the loss you've defined you get back exactly the formula that is implemented. The way it is written down however is, as you mentioned, to avoid underflow/overflow. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, suppose you want to compute the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A=\log(\sum_{i=1}^{4}\exp(z_i))$, with $z_i=(-1000.5,-2000.5,-3000.5,-4000.5)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Clearly, if you just type in the formula directly, you will get an underflow error. Instead if you isolate the main contribution in the exponential by taking the $\max(z_i)$, the same formula can be written as:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$A=\max_i(z_i)+\log(\sum_{i=1}^{4}\exp(z_i-\max_i(z_i)))$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference now is that the expression is &quot;numerically stable&quot; and we see that $A\approx -1000.5$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, let's make the softmax numerically stable:&#xA;\begin{align}&#xA;\log(a_i)&amp;amp;=z_i-\log(\sum_j e^{z_j})\\&#xA;&amp;amp;=z_i-\max_j(z_j)-\log(\sum_je^{z_j-\max_j(z_j)})&#xA;\end{align}&#xA;which is the expression that is implemented for the loss (just multiply by $y_i$ and sum over $i$).&lt;/p&gt;&#xA;" OwnerUserId="29203" LastEditorUserId="29203" LastEditDate="2018-05-24T18:25:44.210" LastActivityDate="2018-05-24T18:25:44.210" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32126" PostTypeId="1" AcceptedAnswerId="39282" CreationDate="2018-05-24T18:19:22.807" Score="24" ViewCount="24506" Body="&lt;p&gt;Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series?&lt;/p&gt;&#xA;&lt;p&gt;There is a one class SVM package in scikit-learn but it is not for the time series data. I’m looking for more sophisticated packages that, for example, use Bayesian networks for anomaly detection.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastEditorUserId="85045" LastEditDate="2020-12-22T03:16:47.327" LastActivityDate="2021-07-01T10:43:14.440" Title="Looking for a good package for anomaly detection in time series" Tags="|python|time-series|anomaly-detection|bayesian-networks|anomaly|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32127" PostTypeId="2" ParentId="32105" CreationDate="2018-05-24T18:35:45.390" Score="0" Body="&lt;p&gt;Indeed, high validation loss can badly effect the accuracy of your model with test data. In this case, absolutely overfitting is happening which in machine learning literature is addressed as high variance problem, too. Overfitting limits the generalization of your model.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-24T18:35:45.390" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32128" PostTypeId="1" AcceptedAnswerId="55304" CreationDate="2018-05-24T18:42:20.837" Score="7" ViewCount="625" Body="&lt;p&gt;I can't seem to convince myself why a GAN model similar to &lt;a href=&quot;https://openreview.net/pdf/90f0e675457dac151517a5ed5c0f15d856366494.pdf&quot; rel=&quot;noreferrer&quot;&gt;regGAN&lt;/a&gt;  couldn't be modified to maximize a regression predictor (see the image below).  By changing the loss function to the difference between the current predicted value and the maximum predicted value generated so far, wouldn't gradient decent converge such that the generator builds the inputs that will maximize the prediction in the Discriminator CNN? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In math terms, the loss calculation would look like: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  yhat = current prediction&#xA;  ymax = best prediction achieved yet&#xA;  Loss = ymax - yhat&#xA;  if Loss &amp;lt; 0 then Loss = 0; ymax = yhat&#xA;  Back-propagate the loss using SGD&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;If the current predicted value is higher than the maximum predicted so far, then the loss is 0 and the loss function is updated.  Essentially, we are changing the objective from generating inputs that look real to generating inputs that optimize the complex function encoded in the CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Qv4Su.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Qv4Su.png&quot; alt=&quot;GAN Network ArchitectureDiscriminator &quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52663" LastActivityDate="2019-07-08T19:54:42.977" Title="Can a GAN-like architecture be used for maximizing the value of a regression predictor?" Tags="|regression|optimization|gan|generative-models|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32129" PostTypeId="1" CreationDate="2018-05-24T19:50:32.517" Score="2" ViewCount="622" Body="&lt;p&gt;I read &lt;a href=&quot;https://www.researchgate.net/post/How_to_compute_impurity_using_Gini_Index&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.researchgate.net/post/How_to_compute_impurity_using_Gini_Index&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand why choosing smallest gini index, but how do I come up with different candidate splits in the first place? How does R come up with the splits? Take the iris data as an instance:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;plot(iris$Sepal.Length, col=iris$Species, pch=20)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZSQsd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZSQsd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I determine where I want to draw the horizontal line to separate each class? I can draw infinitely amount of lines and then compare their gini index, but practically, it will not work.&lt;/p&gt;&#xA;" OwnerUserId="52684" LastActivityDate="2018-05-24T20:05:51.053" Title="How to come up with the splitting point in a decision tree?" Tags="|decision-trees|" AnswerCount="0" CommentCount="1" ClosedDate="2020-05-13T22:24:38.990" ContentLicense="CC BY-SA 4.0" />
  <row Id="32131" PostTypeId="1" CreationDate="2018-05-24T20:21:14.760" Score="1" ViewCount="32" Body="&lt;p&gt;when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about questions that I asked myself back then (with newly acquired math background) and noticed that some things are still unclear to me. I appreciate help and pointers to good - mathematical - literature. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's consider the following learning problem: We are given a training sample $((x_1, y_1), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathbb{R})^n$ and want to learn a functional relationship between $\mathcal{X}$ and $\mathbb{R}$ from this data. Two popular hypothesis spaces are:&#xA;$$\mathcal{H}_{\phi} = \{\langle w, \phi(\cdot)\rangle: w \in \mathcal{F}\},$$ in which $\phi: \mathcal{X} \to \mathcal{F}$ denotes a feature space mapping from the input space to a Hilbert space, and $$\mathcal{H}_{k} = \{\sum_{i=1}^m \alpha_i k(x_i, \cdot): m \in \mathbb{N, x_i \in \mathcal{X}, \alpha_i \in \mathbb{R},\ for\ i \in \{1, \dots, n\}}\},$$&#xA;in which $k$ denotes the positive definite kernel $k(x, x') := \langle\phi(x),\phi(x')\rangle$. &lt;strong&gt;The question is whether those two hypothesis spaces are equivalent&lt;/strong&gt; and it bugs me. The Moore-Aronszajn Theorem states that there is a unique correspondence between reproducing kernel Hilbert space and positive definite kernel. When I studied kernel theory I proofed, that $\mathcal{H}_{\phi}$ is a reproducing kernel Hilbert space. Still, I did not manage to proof that the two mentioned hypothesis spaces are the same without requiring that $\phi$ is surjective (which is rarely required in ML-literature). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank you in advance!&lt;/p&gt;&#xA;" OwnerUserId="52685" LastEditorUserId="1330" LastEditDate="2021-10-07T22:35:09.090" LastActivityDate="2021-10-07T22:35:09.090" Title="Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?" Tags="|mathematics|kernel|theory|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32132" PostTypeId="1" AcceptedAnswerId="32140" CreationDate="2018-05-24T21:00:12.437" Score="1" ViewCount="25" Body="&lt;p&gt;I am building an LSTM to recognize if the person is sad, happy, angry or neutral. This is done by feeding-in his the wave of his voice into the network, as a sequence of bytes (each byte is 0-to-255). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The problem is, my dataset is not large enough, are there efficient ways I could augument my dataset? I am training on short 1.5 second clips and I have 800 of those, which is not enough.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My current augumentation is:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;to add variations in volume&lt;/li&gt;&#xA;&lt;li&gt;to add a bit of white noise, which makes it worse :(&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Reversing the sequences doesn't seem to be applicable, after all, my network will be predicting non-reversed speech when it's fully trained.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-06-16T01:09:44.187" LastActivityDate="2018-06-16T01:09:44.187" Title="how to augument speech sentiment dataset?" Tags="|sentiment-analysis|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32133" PostTypeId="2" ParentId="31931" CreationDate="2018-05-24T22:22:57.370" Score="2" Body="&lt;p&gt;Abstractly, if you've already considered decision trees as decomposable into directed acyclic graphs, then one example of you're looking for is, straightforwardly, a &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Markov_chain&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Markov chain&quot;&gt;Markov Chain&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Markov chains can, indeed, model sequences of arbitrary length. Additionally, markov chains containing cycles are possible-- usually referred to as &lt;strong&gt;hamiltonian-embedded markov chains&lt;/strong&gt;.&lt;/p&gt;&#xA;" OwnerUserId="2932" LastActivityDate="2018-05-24T22:22:57.370" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32134" PostTypeId="2" ParentId="32062" CreationDate="2018-05-24T22:47:20.113" Score="1" Body="&lt;p&gt;In the case that your training data is outdated, you might have a problem if you train a model considering that dataset and then applying it at a test dataset which is newer and not outdated. In the case that you have available some new data at the training time, one solution could be to perform domain adaptation/transfer learning methods to tackle this difference e.g. in the distribution of the features taking into account the small newer dataset and the large outdated dataset. Actually you are going to find out which part of the outdated dataset remains the same or which part could be useful after some transformations. You could check the related bibliography for more information about transfer learning and domain adaptation. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-24T22:47:20.113" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32135" PostTypeId="2" ParentId="31958" CreationDate="2018-05-24T23:07:48.257" Score="2" Body="&lt;p&gt;If I understand the problem correctly, there are several approaches that can be taken.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We'll begin with the least deep learning oriented solution first and slowly move into the spectrum of Deep Learning. This choice is primarily for cost and difficulty reasons. The amount of bugs that you can find with a deep learning solution can be very difficult to understand in certain situations and that results in slower prototyping and ultimately slower time to production.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At the very base, your problem deals with ranking the jobs with respect to how relevant they are to a particular job. The literature for ranking problems is vast and I will suffix my answer with links to these papers or studies. I will also say that instead of dealing with this as a ranking problem there may be another way of going about it. However, this is hinged on the type of data you possess. If you know that each query is certainly looking for a particular type of job. Then, you can break this down into a multi-task problem, where one of the tasks is figuring out whether there is a vacancy or not and the second is figuring out if the vacancy relates in any way to the search query.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Solution 1&lt;/strong&gt;: Take the dataset and handcraft a set of features that you think are relevant to the process of classifying the output. To do this effectively, you should know certain core features that you think relate the input features to the final label (here I use input features to represent the features that you think represent your features well. For example, some features that may be relevant is information about the person or entity posting the title (in stage 2), if they are a well known company in the domain (this can be accesses through some intelligent web scraping and text retrieval methods). Then, it is possibly more likely for it to be a posting, right? Let us assume now that you have constructed this set of features. Now, for each of the set of features, you have a corresponding label to whether there is a vacancy or not. To turn this approach into a ranking problem is very simple. All you have to do is observe the confidence (the softmax probability is good enough) and the one which has a higher probability is ranked higher). This solution is an easy one to accomplish using even simple feed forward neural networks. You would also benefit from testing out simple classifiers like SVMs or Random Forests. It must also be remembered that the actual sentence must be embedded into some numeric space, like a sequence of vectors in $R^n$. This solution does not detail all of the explicit steps but provides a general framework with which you should be able to attack the problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Drawbacks&lt;/strong&gt; The above method requires a lot of time spent in handcrafting the feature set and there is going to be a lot of iteration because the feature set is never going to be perfect. There is always going to be something that you possibly missed out. Secondly training a good word embedding is not a very easy task. You may have to spend a lot of time getting that running, especially if you want the embedding to be somehow related to your words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Solution 2&lt;/strong&gt; This solution begins to touch the fence of Deep Learning. This solution is a comparison of the attention encodings within the source utterance and the target utterance. The source utterance is, a phrase like &quot;need janitor&quot;. Now you should train an attention model over the sequence of words and observe what is the word(s) being given the maximum attention to. Once you know this, the problem becomes a question of consolidating the attentions in some meaningful manner and comparing the distance between the source and target through some metric space. So you should certainly pay attention to attention based models (haha!). Again, there are going to be hiccups in the process of figuring out how much attention actually affects the output and whether it is a good model for your data. Distances between attention will return a vector and you can consider the $\mathcal{L}_2$ norm or any other map from $\mathbb{R^n} \rightarrow \mathbb{R}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Drawbacks&lt;/strong&gt; Attention distances are a good metric, however, they are extremely data centric. For example if the ads are very long, it may be true that the model does not pick up the appropriate words to attend to.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now we get to some resources that may be useful to go through in order to understand exactly how the different components of the answer work.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Learning_to_rank&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/Learning_to_rank&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.shivani-agarwal.net/Events/SDM-10-Tutorial/sdm10-tutorial.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.shivani-agarwal.net/Events/SDM-10-Tutorial/sdm10-tutorial.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://towardsdatascience.com/learning-to-rank-with-python-scikit-learn-327a5cfd81f&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://towardsdatascience.com/learning-to-rank-with-python-scikit-learn-327a5cfd81f&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://times.cs.uiuc.edu/course/598f16/l2r.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://times.cs.uiuc.edu/course/598f16/l2r.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/slides/agarwal.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/slides/agarwal.pdf&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Now for the posts about attention:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://akosiorek.github.io/ml/2017/10/14/visual-attention.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://akosiorek.github.io/ml/2017/10/14/visual-attention.html&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Finally, my last component of the answer will deal with evaluating exactly how well you have done using metrics. I have added some metrics above in my Answer, however there is another metric which in particular represents the similarity, syntactically between the input query and the output query.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/BLEU&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://en.wikipedia.org/wiki/BLEU&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="27114" LastActivityDate="2018-05-24T23:07:48.257" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32137" PostTypeId="1" CreationDate="2018-05-25T04:45:40.817" Score="2" ViewCount="379" Body="&lt;p&gt;For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using &lt;strong&gt;XGboost&lt;/strong&gt; implementation.  Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df2 &amp;lt;- df1[sample(nrow(df1)),]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it because, a large portion of my final test data already learned from the observations also found in my training data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?&lt;/p&gt;&#xA;" OwnerUserId="52695" LastEditorUserId="29575" LastEditDate="2018-09-12T15:36:46.960" LastActivityDate="2022-08-06T15:04:45.453" Title="Can you reuse observations from your train data in your final test data?" Tags="|classification|r|xgboost|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32138" PostTypeId="1" CreationDate="2018-05-25T05:06:15.540" Score="1" ViewCount="172" Body="&lt;p&gt;I am new to keras and machine learning. How can i tell if i am on the right track of setting up keras layer for binary classification, LSTM or convulsion network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example (from keras documentation) &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    # For a single-input model with 2 classes (binary classification):&#xA;    model = Sequential()&#xA;    model.add(Dense(32, activation='relu', input_dim=100))&#xA;    model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;    model.compile(optimizer='rmsprop', loss='binary_crossentropy',&#xA;    metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Multi layer perceptron multimax softmax classification &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;&#xA;model.add(LSTM(32, return_sequences=True, stateful=True,&#xA;batch_input_shape=(batch_size, timesteps, data_dim)))&#xA;&#xA;model.add(LSTM(32, return_sequences=True, stateful=True))&#xA;model.add(LSTM(32, stateful=True))&#xA;model.add(Dense(10, activation='softmax'))&#xA;&#xA;model.compile(loss='categorical_crossentropy',&#xA;              optimizer='rmsprop',&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Sequence classification for 1d convulsion &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv1D(64, 3, activation='relu', input_shape=(seq_length, 100)))&#xA;model.add(Conv1D(64, 3, activation='relu'))&#xA;&#xA;model.add(MaxPooling1D(3))&#xA;model.add(Conv1D(128, 3, activation='relu'))&#xA;model.add(Conv1D(128, 3, activation='relu'))&#xA;model.add(GlobalAveragePooling1D())&#xA;&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="24370" LastActivityDate="2018-05-25T05:06:15.540" Title="Keras - layers configuration" Tags="|machine-learning|python|keras|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32139" PostTypeId="1" CreationDate="2018-05-25T05:35:04.437" Score="0" ViewCount="10808" Body="&lt;p&gt;In a dataset containing temperatures of different years and I want to extract data of particular month from all different years in single liner code what is the syntax? To extract all the data from a single month the syntax is :&lt;br/&gt;&lt;code&gt;df['Temperature'].loc['2010-08']&lt;/code&gt; &lt;br/&gt;It is always year followed by month.What if I want to extract 2010 &amp;amp; 2011 years temperatures in August in single line.&lt;/p&gt;&#xA;" OwnerUserId="45713" LastActivityDate="2020-05-03T23:23:48.800" Title="Extract all the data of a particular month from dataset of different years" Tags="|python|pandas|preprocessing|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32140" PostTypeId="2" ParentId="32132" CreationDate="2018-05-25T05:47:41.787" Score="1" Body="&lt;p&gt;Your problem is to identify whether a person is sad, happy, angry or neutral from his voice which is emotion classification problem. For speech, we use frames of short duration like 10-20 ms, and extract features from the same like MFCC or other frequency domain features. We extract short duration frames because speech is non-stationary over time, and frequency domain features make the features shift-invariant.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I suggest you read some latest research papers on emotion classification to get latest research in emotion classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Emotion in speech is captured by variation in pitch and amplitude across time. So capturing short time energy and instantaneous pitch frequency in speech across time  are basic features for emotion classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Modifying speech by adding variation in volume and white noise will not help at all.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to build a emotion classification system with 4 classes: sad, happy, angry or neutral, using the below steps:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Extract frames of 10-30 ms duration&lt;/li&gt;&#xA;&lt;li&gt;Convert these frames to frequency domain features like STFT or extract instantaneous pitch&lt;/li&gt;&#xA;&lt;li&gt;Use these STFT features along with pitch to train a LSTM &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="42984" LastActivityDate="2018-05-25T05:47:41.787" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32141" PostTypeId="1" CreationDate="2018-05-25T06:04:57.970" Score="3" ViewCount="296" Body="&lt;p&gt;What are some ways that I can generate the negation of a sentence such that the output sentence reflects the negation of original sentence?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the sentence is: &lt;code&gt;That is an apple.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then, the expected negation could be: &lt;code&gt;That is not an apple.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Sentence: Rita is so hot.&#xA;Negation: Rita is boring.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="42498" LastEditorUserId="29169" LastEditDate="2019-06-07T16:21:13.453" LastActivityDate="2019-06-07T16:21:13.453" Title="How can I create a negation of the sentence?" Tags="|deep-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32142" PostTypeId="1" CreationDate="2018-05-25T06:33:14.247" Score="2" ViewCount="81" Body="&lt;p&gt;It is a general question on how to learning representation of one entity but the dataset is mixed with a lot of other entities, which their statis are always waiting to be learnt.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question is best be explained by an example. Let's say, the entity is a football player. How can we assign player stats to them by just learning from the match result. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Dataset: Football match data with startup line of the two teams and the final result score.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Player stats, something like this &lt;a href=&quot;https://www.fifaindex.com/player/158023/lionel-messi/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.fifaindex.com/player/158023/lionel-messi/&lt;/a&gt; but much simpler. We can define a player stats is just 3D vector v = (x , y , z) where x , y , z is between 0 to 1.0.  We don't need to know what actually x mean shooting power or anything.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So after learning, It output each player a v vector assigned.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea or some material that you can suggest?&lt;/p&gt;&#xA;" OwnerUserId="24919" LastActivityDate="2018-05-25T19:05:40.110" Title="Learning Football Player Stats like FIFA's by only the game result" Tags="|machine-learning|unsupervised-learning|representation|" AnswerCount="1" CommentCount="1" ClosedDate="2018-05-30T08:25:50.817" ContentLicense="CC BY-SA 4.0" />
  <row Id="32143" PostTypeId="1" AcceptedAnswerId="32146" CreationDate="2018-05-25T06:39:07.440" Score="3" ViewCount="331" Body="&lt;p&gt;So I'm quickly learning that dealing with missing values for feature(s) in some of your observations is a part of every day life in data.  I get the gist of imputation, when/how it's appropriate and when it's not, and I'll read up on it in the near future.  But, what about this: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose you have predictors $X_1, \dots, X_p$, and you want to model, say, a binary response $Y$ via, say, logistic regression.  Instead of just imputing values to missing predictor values, couldn't you have a separate model built for every possible subset of the predictors, and apply that model for prediction when those predictors happen to be present?  Each of those models would naturally be trained on just the data for which those predictors (and also others) are present.  This seems to me a more reasonable approach than just making up values, but I have no theoretical justification for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do realize that this involves building $2^p$ different models, with $2^p$ different model matrices, etc., but for a moderate $p$ and $n$ it could be feasible, and especially if only a few of your features tend to be missing more often than others.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this ever done?  And if so, is there a standard way to implement this in R?  In the case of logistic regression, you can specify to R's glm function how you want it to handle NA values, but your only options seem to be either tossing out observations altogether, or some kind of imputation scheme.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thoughts?&lt;/p&gt;&#xA;" OwnerUserId="47958" LastEditorUserId="47958" LastEditDate="2018-05-25T06:47:26.297" LastActivityDate="2018-05-25T15:35:16.460" Title="Alternatives to imputation of missing values?" Tags="|predictive-modeling|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32144" PostTypeId="1" AcceptedAnswerId="32242" CreationDate="2018-05-25T06:46:39.803" Score="4" ViewCount="199" Body="&lt;p&gt;I would like to extract some specific information from web pages. Web pages contain person profiles, and I want to extract information such as name, email, research interested-areas. &#xA;Structure of each page is different from one another. How can I extract such information using machine learning? What kind of a method, features I can use?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or can I use NLP for such task?&lt;/p&gt;&#xA;" OwnerUserId="52697" LastActivityDate="2018-05-28T04:52:15.773" Title="Web page data extraction using machine learning" Tags="|machine-learning|python|nlp|" AnswerCount="1" CommentCount="0" ClosedDate="2018-05-30T00:50:09.750" ContentLicense="CC BY-SA 4.0" />
  <row Id="32145" PostTypeId="1" CreationDate="2018-05-25T06:52:20.293" Score="3" ViewCount="790" Body="&lt;p&gt;I am trying to do some analysis about user behaviour when typing (keystroke biometrics). Ideally, it will include traits extracted when people are writing code. Although not technically Natural Language, code also has some structured characteristics as language and I wanted to leverage that. I was wondering if there has been some research about performing language analysis focusing on programming languages instead of traditional spoken languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mainly, I am interested in having a comprehensive list of stopwords for as many languages as possible. For example, stopwords will include: &lt;em&gt;for, while, return, break, string, if, else&lt;/em&gt;, and so on. Although it would be nice to have them separated by languages, I wouldn't mind a list comprising several languages.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I know this could be done for example by getting some sample code and retrieving the most frequent terms, but I also wanted to know if there has been some research towards this direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any ideas, papers, methods would be welcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2020-04-21T14:56:23.887" Title="Stopwords for programming languages (for, while, print,...)" Tags="|nlp|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32146" PostTypeId="2" ParentId="32143" CreationDate="2018-05-25T06:54:18.690" Score="1" Body="&lt;p&gt;These are my thoughts:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This seems to me a more reasonable approach than just making up values, but I have no theoretical justification for this.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Actually, there is not much theoretical justification for classical imputation. I think your method makes sense when just very few of the predictors have missing values. For instance, if there is just one predictor that has missing values, you can build a model to imput those missing values. However, as you said, things scale very badly. Moreover, apart from computational issues, if there are lots of missing values in every predictor your data is not very good, and therefore any predictive model won't be very good anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is done sometimes, but I don't think there's any library in R that implements it, you'll have to code it yourself (I don't think it is difficult). If you do it and your final model is a logistic regression, I don't recommend to fit the missing values with another linear model, as you will suffer from a collinearity problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One other thing that is typically done is that, for every predictor with missing values, create a binary predictor that is $0$ if the other predictor's value is missing and $1$ otherwise.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-25T06:54:18.690" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32147" PostTypeId="2" ParentId="32139" CreationDate="2018-05-25T07:28:33.373" Score="1" Body="&lt;pre&gt;&lt;code&gt;df.loc[df['Temperature'].isin(['2010-08', '2011-08'])]&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;isin() will help in setting filter for multiple values. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;br/&gt;The other way is to use regex to extract data of particular month from different years:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import re&#xA;df['Temperature'].filter(lambda x: re.match('-08$', x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;It will check if the value ends with -08 (in this case, if the month is August) and select only such records. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="52428" LastEditorUserId="52428" LastEditDate="2018-05-25T08:33:09.557" LastActivityDate="2018-05-25T08:33:09.557" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32151" PostTypeId="1" CreationDate="2018-05-25T10:01:08.800" Score="3" ViewCount="1109" Body="&lt;p&gt;I have read about SVM and although I did not understand the math behind it completly, I know that it produces decision plane with maximum margin between examples of different classes and role of support vectors in the process.&#xA;I also know that SVM is a kind of dual learing algorithm(algorithms that operate only using the dot product between examples). It uses kernel functions to calculate dot product(measure of similarity) between training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I want to understand in simple terms is that: Suppose I have a similarity matrix of all training examples specifying amount of similary between any(all) two examples in training sample. How Can I make a classifier or cluster based only on this information?&lt;/p&gt;&#xA;" OwnerUserId="52701" LastEditorUserId="52701" LastEditDate="2018-05-25T17:32:32.303" LastActivityDate="2022-10-17T00:00:30.077" Title="Intuition behind the fact that SVM uses only measure of similarity between examples for classification" Tags="|machine-learning|classification|clustering|svm|kernel|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32152" PostTypeId="2" ParentId="32109" CreationDate="2018-05-25T10:33:45.640" Score="3" Body="&lt;ul&gt;&#xA;&lt;li&gt;Whenever we start with any dataset in machine learning, we often assume that all the data features are equally important with respect to the output and one feature should not dominate over other feature. That’s GENERALLY the reason we choose to bring all the features to same scale.&lt;br&gt;&#xA; However, one may raise a doubt here that even if the features are not normalised then the weights assigned to it while learning may help the data set converge to expected output while training. The problem with this is that it will take really long to train and produce results.&lt;/li&gt;&#xA;&lt;li&gt;To choose specific number 0 as mean and variance 1 is just the ease to visualise and keeping such small numbers would help in faster training. &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Hence, it is suggested to bring all features to same scale smaller enough to train easily.&#xA;Below link also discusses similar concept.&#xA;&lt;a href=&quot;https://stats.stackexchange.com/questions/41704/how-and-why-do-normalization-and-feature-scaling-work&quot;&gt;https://stats.stackexchange.com/questions/41704/how-and-why-do-normalization-and-feature-scaling-work&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="42498" LastActivityDate="2018-05-25T10:33:45.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32155" PostTypeId="1" CreationDate="2018-05-25T11:02:48.537" Score="2" ViewCount="101" Body="&lt;p&gt;I want to choose an unsupervised algorithm which learns to predict $n$ outputs from the data, for eg. 4 coordinates (pixels) in an image. What algorithm should I choose? I think it's a 2-class classification to divide the set of points in the image as belonging to output (1) or not (0), maybe logistic regression to give the probabilities of a point being the output point. But I am confused because classification algorithms are part of supervised algorithms where we have labelled data. Should I use clustering to find 2 groups of points that can be output or not? Maybe anomaly detection to find the 4 odd points out? &lt;/p&gt;&#xA;" OwnerUserId="39387" LastActivityDate="2018-05-25T15:57:05.570" Title="Which machine learning algorithm to choose?" Tags="|algorithms|unsupervised-learning|machine-learning-model|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32157" PostTypeId="2" ParentId="32155" CreationDate="2018-05-25T11:26:35.657" Score="0" Body="&lt;p&gt;Since you don't have any available labeled data, it is not easy to perform a supervised learning algorithm or at least a semi-supervised learning algorithm. The latter could be very useful in case that you could available a small dataset of labeled data. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution could be to perform a clustering algorithm first excluding the target feature, for example k-means for k equal to 2 and then train a model taking into account the cluster pseudo-labels. But you will not still be able to identify which of these two clusters is the class &quot;1&quot; or which one is the class &quot;2&quot;.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can have a reasonable solution only including a few labeled data, otherwise you can just perform some cluster analysis. &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-25T11:26:35.657" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32159" PostTypeId="2" ParentId="11129" CreationDate="2018-05-25T12:52:23.850" Score="0" Body="&lt;p&gt;I suggest that you use categorical HMM (Hidden Markov Model) or LSTM (Long Short Term Memory) networks to predict the next values of time series.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-25T12:52:23.850" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32161" PostTypeId="1" AcceptedAnswerId="32172" CreationDate="2018-05-25T14:09:30.320" Score="2" ViewCount="27" Body="&lt;p&gt;I am new to the field of data science and trying to figure out ways to handle data quality issues before performing any modeling. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am working on a house rental price data set. In this data there is feature called &lt;strong&gt;Total Squarefeet&lt;/strong&gt;. The issue I am facing here is that out of 12000 records 200 have a range, for e.g., 1200 - 1800, OR 850 - 855, OR even and these ranges also have a random difference between them. Rest other are simple numbers. Is there a way to correctly handle this kind of data. Can anyone help me or guide me to a place from where i can learn and use techniques to handle such data. Thanks in advance.&lt;/p&gt;&#xA;" OwnerUserId="44611" LastEditorUserId="44611" LastEditDate="2018-05-27T12:21:51.823" LastActivityDate="2018-05-27T12:21:51.823" Title="Handling binned feature" Tags="|pandas|data-cleaning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32162" PostTypeId="1" CreationDate="2018-05-25T14:18:01.100" Score="3" ViewCount="523" Body="&lt;p&gt;I have an imbalanced dataset (True labels are ~10x than False labels) and thus use the f_beta score as a metric for model performance, as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def fbeta(beta):&#xA;    def f1(y_true, y_pred):&#xA;        def recall(y_true, y_pred):&#xA;            true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))&#xA;            possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))&#xA;            recall = true_positives / (possible_positives + K.epsilon())&#xA;            return recall&#xA;&#xA;        def precision(y_true, y_pred):&#xA;            true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))&#xA;            predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))&#xA;            precision = true_positives / (predicted_positives + K.epsilon())&#xA;            return precision&#xA;        beta_squared = beta ** 2&#xA;        precision = precision(y_true, y_pred)&#xA;        recall = recall(y_true, y_pred)&#xA;        fbeta_score = (beta_squared + 1) * (precision * recall) / (beta_squared * precision + recall + K.epsilon())&#xA;        return fbeta_score&#xA;    return f1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running my model I get the following output.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Epoch 00001: val_f1 improved from -inf to 0.90196, saving model to&#xA;  output\LSTM\weights\floods_weights.h5         Epoch 2/3&lt;br&gt;&#xA;  - 0s - loss: 0.4114 - f1: 0.8522 - val_loss: 0.4193 - val_f1: 0.9020                                                                                                                                                             Epoch 00002: val_f1 did not improve from 0.90196&lt;br&gt;&#xA;  Epoch 3/3&lt;br&gt;&#xA;  - 0s - loss: 0.3386 - f1: 0.8867 - val_loss: 0.3589 - val_f1: 0.9020                                                                                                                                                             Epoch 00003: val_f1 did not improve from 0.90196&lt;br&gt;&#xA;  Evaluating :&lt;br&gt;&#xA;  51/51 [==============================] - 0s 372us/step&lt;br&gt;&#xA;  Final Accuracy :  0.9019607305526733&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, when running the model again with swapped y-labels (True-&gt;False, False-&gt;True) I get the exact same values and do not understand why as the f-measure should be highly unlikely to give the exact same results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When assessing the final model using the following code for precision and recall, I do get different results: 0.6 vs 0.95788:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;prediction = model.predict(data['X_test'], batch_size=64)&#xA;tp = 0&#xA;fp = 0&#xA;fn = 0&#xA;for y, p in zip(data['y_test'], np.argmax(prediction, axis=1)):&#xA;    y = y[0]&#xA;    if y == 1 and p == 1:&#xA;        tp += 1&#xA;    elif p == 1 and y == 0:&#xA;        fp += 1&#xA;    elif p == 0 and y == 1:&#xA;        fn += 1&#xA;&#xA;precision = tp / (tp + fp)&#xA;recall = tp / (tp + fn)&#xA;print(2 * (precision * recall) / (precision + recall))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What's happening?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am compiling the model as such:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;beta = 1&#xA;model.compile(&#xA;    optimizer=Adam(lr=1e-3),&#xA;    loss='categorical_crossentropy',&#xA;    metrics=[fbeta(beta)]&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52640" LastEditorUserId="1330" LastEditDate="2020-10-03T19:51:38.977" LastActivityDate="2020-10-03T19:51:38.977" Title="Validation score (f1) remains the same when swapping labels" Tags="|keras|accuracy|implementation|" AnswerCount="0" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32164" PostTypeId="1" AcceptedAnswerId="32170" CreationDate="2018-05-25T14:31:46.797" Score="2" ViewCount="97" Body="&lt;pre&gt;&lt;code&gt;    batch_size = 128&#xA;size_1 = 1024&#xA;size_2 = 256&#xA;size_3 = 128&#xA;beta = 0.001&#xA;&#xA;graph = tf.Graph()&#xA;with graph.as_default():&#xA;&#xA;    tf_train_dataset = tf.placeholder(&#xA;        tf.float32,shape=(batch_size,image_size*image_size))&#xA;    tf_train_labels = tf.placeholder(&#xA;        tf.float32, shape=(batch_size, num_labels))&#xA;    tf_valid_dataset = tf.constant(valid_dataset)&#xA;    tf_test_dataset = tf.constant(test_dataset)&#xA;&#xA;    # Weights and Biases&#xA;    g_W1 = tf.Variable(&#xA;        tf.truncated_normal([image_size*image_size,size_1]))&#xA;    g_B1 = tf.Variable(&#xA;        tf.zeros([size_1]))&#xA;&#xA;    g_W2 = tf.Variable(&#xA;        tf.truncated_normal([size_1,size_2]))&#xA;    g_B2 = tf.Variable(&#xA;        tf.zeros([size_2]))&#xA;&#xA;    g_W3 = tf.Variable(&#xA;        tf.truncated_normal([size_2,num_labels]))&#xA;    g_B3 = tf.Variable(&#xA;        tf.zeros([num_labels]))&#xA;&#xA;#     g_W4 = tf.Variable(&#xA;#         tf.truncated_normal([size_3,num_labels]))&#xA;#     g_B4 = tf.Variable(&#xA;#         tf.zeros([num_labels]))&#xA;&#xA;&#xA;    L1 = tf.nn.relu(&#xA;        tf.matmul(tf_train_dataset,g_W1) + g_B1)&#xA;    L2 = tf.nn.relu(&#xA;        tf.matmul(L1,g_W2) + g_B2)&#xA;#     L3 = tf.nn.relu(&#xA;#         tf.matmul(L2,g_W3) + g_B3)&#xA;&#xA;    dr_prob = tf.placeholder(&quot;float&quot;)&#xA;&#xA;    ##add dropout here&#xA;    #L1 = tf.nn.dropout(tf.nn.relu(&#xA;     #   tf.matmul(tf_train_dataset,g_W1) + g_B1), 1.0)&#xA;    #L2 = tf.nn.dropout(tf.nn.relu(&#xA;     #   tf.matmul(L1,g_W2) + g_B2), 1.0)&#xA;    #L3 = tf.nn.dropout(tf.nn.relu(&#xA;     #   tf.matmul(L2,g_W3) + g_B3), 1.0)&#xA;&#xA;&#xA;    logits = tf.matmul(L2, g_W3) + g_B3&#xA;&#xA;    loss = tf.reduce_mean(&#xA;        tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits))+\&#xA;        beta*tf.nn.l2_loss(g_W1) +\&#xA;        beta*tf.nn.l2_loss(g_W2)+\&#xA;        beta*tf.nn.l2_loss(g_W3)&#xA;#         beta*tf.nn.l2_loss(g_W4)&#xA;&#xA;        # Optimizer.&#xA;    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)&#xA;&#xA;    # Predictions for the training, validation, and test data.&#xA;    train_prediction = tf.nn.softmax(logits)&#xA;    L1_pred = tf.nn.relu(tf.matmul(tf_valid_dataset, g_W1) + g_B1)&#xA;    L2_pred = tf.nn.relu(tf.matmul(L1_pred, g_W2) + g_B2)&#xA;#     L3_pred = tf.nn.relu(tf.matmul(L2_pred, g_W3) + g_B3)&#xA;    valid_prediction = tf.nn.softmax(tf.matmul(L2_pred, g_W3) + g_B3)&#xA;&#xA;    L1_test = tf.nn.relu(tf.matmul(tf_test_dataset, g_W1) + g_B1)&#xA;    L2_test = tf.nn.relu(tf.matmul(L1_test, g_W2) + g_B2)&#xA;#     L3_test = tf.nn.relu(tf.matmul(L2_test, g_W3) + g_B3)&#xA;    test_prediction = tf.nn.softmax(tf.matmul(L2_test, g_W3) + g_B3)&#xA;&#xA;num_steps = 3001&#xA;&#xA;with tf.Session(graph=graph) as session:&#xA;  tf.global_variables_initializer().run()&#xA;  print(&quot;Initialized&quot;)&#xA;  for step in range(num_steps):&#xA;    # Pick an offset within the training data, which has been randomized.&#xA;    # Note: we could use better randomization across epochs.&#xA;    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)&#xA;    # Generate a minibatch.&#xA;    batch_data = train_dataset[offset:(offset + batch_size), :]&#xA;    batch_labels = train_labels[offset:(offset + batch_size), :]&#xA;    # Prepare a dictionary telling the session where to feed the minibatch.&#xA;    # The key of the dictionary is the placeholder node of the graph to be fed,&#xA;    # and the value is the numpy array to feed to it.&#xA;    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, dr_prob : 0.5}&#xA;    _, l, predictions = session.run(&#xA;      [optimizer, loss, train_prediction], feed_dict=feed_dict)&#xA;    if (step % 500 == 0):&#xA;      print(&quot;Minibatch loss at step %d: %f&quot; % (step, l))&#xA;      print(&quot;Minibatch accuracy: %.1f%%&quot; % accuracy(predictions, batch_labels))&#xA;      print(&quot;Validation accuracy: %.1f%%&quot; % accuracy(&#xA;        valid_prediction.eval(), valid_labels))&#xA;  print(&quot;Test accuracy: %.1f%%&quot; % accuracy(test_prediction.eval(), test_labels))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now it's 2 days trying to know what's wrong with my solution, I hope somebody can spot it,&#xA;the purpose is to train a simple deep NN of two hidden NN, I have checked other's solutions and I still don't get what's wrong with my code&#xA;(it's 4th problem 3rd assignment of Udacity deep learning online course)&#xA;I am getting the following output..&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Initialized&#xA;Minibatch loss at step 0: 3983.812256&#xA;Minibatch accuracy: 8.6%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 500: nan&#xA;Minibatch accuracy: 9.4%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 1000: nan&#xA;Minibatch accuracy: 8.6%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 1500: nan&#xA;Minibatch accuracy: 11.7%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 2000: nan&#xA;Minibatch accuracy: 6.2%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 2500: nan&#xA;Minibatch accuracy: 10.2%&#xA;Validation accuracy: 10.0%&#xA;Minibatch loss at step 3000: nan&#xA;Minibatch accuracy: 7.8%&#xA;Validation accuracy: 10.0%&#xA;Test accuracy: 10.0%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52678" LastActivityDate="2018-12-26T14:30:43.980" Title="What's wrong with my deep NN of two hidden layers?" Tags="|machine-learning|neural-network|deep-learning|tensorflow|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32167" PostTypeId="2" ParentId="32143" CreationDate="2018-05-25T15:35:16.460" Score="2" Body="&lt;p&gt;Supposedly you had these models, divided into subspaces, then what if in your test dataset a point is close proximity to where the missing point is in your training data. What would be the best solution? Have the average output of the two closest models? Is the output binary? And then would you have to weight each model, based on that proximity? Then I am thinking that many submodels could overfit on the subset of the data. Just some thoughts, which may be challenging to overcome.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, you could use other types of Classifiers which deal with missing data, like some decision trees implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, data imputation is a delicate art, and doing it right will leverage your model. For instance, you wouldn't want to impute all the missing data with the average as then you may change your data distribution. Knn techniques or ensemble regressions would be preferred. &lt;/p&gt;&#xA;" OwnerUserId="14560" LastActivityDate="2018-05-25T15:35:16.460" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32168" PostTypeId="2" ParentId="32155" CreationDate="2018-05-25T15:57:05.570" Score="2" Body="&lt;p&gt;If you want to use an unsupervised method i.e if your data is not labelled with classes, then something like &lt;a href=&quot;https://en.m.wikipedia.org/wiki/K-means_clustering&quot; rel=&quot;nofollow noreferrer&quot;&gt;k-means clustering&lt;/a&gt; may be your best bet to find patterns in the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternatively, if you want to do anomaly detection, 2 possible options are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Assuming your data is normally distributed, you could fit a &lt;a href=&quot;https://en.m.wikipedia.org/wiki/Normal_distribution&quot; rel=&quot;nofollow noreferrer&quot;&gt;Gaussian&lt;/a&gt; to your data and calculate the Probability Density Function (PDF). Once you have the PDF, you can set a threshold probability, below which a data point could be classified as an anomaly&lt;/li&gt;&#xA;&lt;li&gt;If you have enough data, use a &lt;a href=&quot;https://youtu.be/9zKuYvjFFS8&quot; rel=&quot;nofollow noreferrer&quot;&gt;Variational Autoencoder&lt;/a&gt; neural network. Very roughly speaking, you train this on all the data you deem to be “normal” (the neural network learns how to reconstruct the input data in the output), and then when anomalous cases are passed to the network, it can’t reconstruct it. If the network can’t reconstruct it accurately, the data is an anomaly. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-05-25T15:57:05.570" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32169" PostTypeId="1" AcceptedAnswerId="38688" CreationDate="2018-05-25T16:05:26.797" Score="6" ViewCount="6591" Body="&lt;p&gt;I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The unfodling process thus introduces two major advantages:&lt;/p&gt;&#xA;  &#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition&#xA;  from one state to another state, rather then specified in terms of&#xA;  variable-length history of states.&lt;/li&gt;&#xA;  &lt;li&gt;It is possible to use same transition function f with same parameters at every time step.&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;  &#xA;  &lt;p&gt;These two factors make it possible to learn a single model f that&#xA;  operates on all time steps and all sequence lengths, rather then&#xA;  needing to learn a separate model for all possible time steps.&#xA;  Learning a single shared model allows generalization to sequence&#xA;  lengths that did not appear in the training set, and enables the model&#xA;  to be estimated with far fewer training examples than would be&#xA;  required without parameter sharing.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I understand the 2nd advantage. Because the computations are recurrent, the input besides the current element in the sequence is the output of the previous hidden state which has the same structure as the current hidden state, thus the shared parameters.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I do not understand the first advantage. I cannot visualize it or mathematically prove it, or at least I dont know how. Can anyone help me with this one? And if anyone has anything else to add on the difference ( advantages ) of RNN over ANN I would really appreciate it. Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="45907" LastActivityDate="2018-09-23T19:33:12.013" Title="Advantages of Recurrent Neural Networks over basic Artificial Neural Networks" Tags="|machine-learning|neural-network|deep-learning|rnn|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32170" PostTypeId="2" ParentId="32164" CreationDate="2018-05-25T16:25:46.607" Score="3" Body="&lt;p&gt;You didn't tell in your question what you tried when debugging, but I'll try to answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Short answer: It looks to me you got to choose a lower learning rate since your loss is exploding after the first iteration.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Explanation: You are using a standard &lt;code&gt;Stochastic Gradient Descent&lt;/code&gt; to perform optimization. Therefore, it's an non-adaptive learning rate algorithms, which means if that latter is poorly chosen, loss can explode if the learning rate is too high. That's why when I'm running in such optimization issues with a new neural network, what I like to do is to set a very low learning rate to ensure convergence at first. Also you could use an adaptive optimizer such as &lt;code&gt;AdaGrad&lt;/code&gt; or &lt;code&gt;Adam&lt;/code&gt; who have both tensorflow implementations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I hope that'll solve your issue.&lt;/p&gt;&#xA;" OwnerUserId="49293" LastActivityDate="2018-05-25T16:25:46.607" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32171" PostTypeId="2" ParentId="32169" CreationDate="2018-05-25T16:39:11.533" Score="3" Body="&lt;p&gt;The main advantage of RNN over ANN is that RNN can model sequence of data (i.e. time series) so that each sample can be assumed to be dependent on previous ones. On the contrary, ANN can not model sequence of data. So, ANN is useful if only each sample is assumed to be independent of previous and next ones (akn as iid assumption).&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-25T16:39:11.533" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32172" PostTypeId="2" ParentId="32161" CreationDate="2018-05-25T17:25:04.767" Score="0" Body="&lt;p&gt;The column you are using must be of string data type. First filter the dataset where it contain &quot;-&quot; and create another dataframe. On this new dataframe perform split string operation and create two different column and convert them to integer. Now create another column which has average or whatever you prefer with two column. Rename column and drop unnecessary columns.  Merge it back to dataframe.&lt;/p&gt;&#xA;" OwnerUserId="20517" LastActivityDate="2018-05-25T17:25:04.767" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32173" PostTypeId="2" ParentId="29488" CreationDate="2018-05-25T17:44:37.963" Score="2" Body="&lt;p&gt;I think one of the main problems with PCA is the PC itself is a very abstract idea that doesn't map well to anything in the real world.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To someone non-technical, it lacks a reference point and it does not generate an insight that is readily actionable (i.e.,&quot;Sales would improve if we change the colour of the font to blue&quot;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As such, I'd suggest that you focus on the storytelling (e.g., the what?, so what?, and now what? framework, etc.), and only elaborate on what PCA is when asked.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance:&#xA;What: We build a prediction model based on web traffic to predict sales.&#xA;So what: The model shows that, if we do xyz xyz and increase xyz by xyz, the sales could potentially increase by xyz.&#xA;Now what:  We should validate such hypothesis by running a controlled test, blah blah blah. If it works, we should adopt the new process.&lt;/p&gt;&#xA;" OwnerUserId="52719" LastEditorUserId="52719" LastEditDate="2018-05-25T18:42:56.570" LastActivityDate="2018-05-25T18:42:56.570" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32174" PostTypeId="2" ParentId="32137" CreationDate="2018-05-25T18:08:41.097" Score="1" Body="&lt;p&gt;The Out of Bag(OOB) error in XG Boost is considered as a Validation error(test error). During Hyper-parameter selection in XG boost, OOB error is measured for different hyperparameters. In XG Boost algorithm, the decision trees are constructed by a random sample of observations and the successive decision trees are built by giving more importance to the observation which the previous trees are misclassified. As this XG boost keeps increasing the number of decisions trees, the validation error decreases and saturates at some point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your case, you can consider this OOB error as test error instead of creating a separate test set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The following plot explains this more accurately &lt;a href=&quot;https://i.stack.imgur.com/K0zXj.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/K0zXj.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="44617" LastActivityDate="2018-05-25T18:08:41.097" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32175" PostTypeId="1" CreationDate="2018-05-25T18:19:54.540" Score="1" ViewCount="648" Body="&lt;p&gt;I have worked with time series data to predict the defect in production lines. I want to extract the feature from time series data as inputs variables for machine learning algorithms (such as support vector machine). My dataset looks like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Timestamp  Pressure&#xA;t0           x0&#xA;t1           x1&#xA;t2           x2&#xA;.             .&#xA;.             .&#xA;.             .&#xA;tn           xn&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;There is a threshold b. If xt&gt;b that means the defect happened. My goals are to extract features from time series dataset above and put it into algorithms to predict the value of time stamp t&lt;code&gt;n+1&lt;/code&gt;, t&lt;code&gt;n+2&lt;/code&gt; (short-term) and long-term t&lt;code&gt;n+10&lt;/code&gt;. However, I have not yet found the way to extract feature and how to bring the threshold into the algorithm. Could anyone suggest me how to deal with it? Thank you.&lt;/p&gt;&#xA;" OwnerUserId="52722" LastActivityDate="2018-06-24T19:53:25.037" Title="Feature extraction in time series data as input variables for machine learning algorithms" Tags="|r|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32176" PostTypeId="2" ParentId="32175" CreationDate="2018-05-25T18:37:10.777" Score="1" Body="&lt;p&gt;You don’t need to extract any features. You should feed the pressure data to a LSTM network. LSTM is a type of recurrent ANN that can be used to predict time series.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-25T18:37:10.777" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32178" PostTypeId="2" ParentId="32142" CreationDate="2018-05-25T19:05:40.110" Score="2" Body="&lt;p&gt;For that, you would probably need something more granular than just the score (e.g., in-game analytics, etc.) to help you assess the attributes of each player.  Even then, it would be difficult to map the outcome to multiple attributes (i.e., defending skills, offensive skills).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you only care about a single score of a player (i.e., say, how effective is a player in helping her team to win, etc.), you could potentially look at something like Corsi which essentially measures puck possession (not without its criticism or flaws). There are many variants out there, but it should give you an idea how to leverage in-game analytics to assess a player.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But then, Corsi is merely a calculation from in-game statistic; you will need more than that to call it a machine learning exercise.&lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-05-25T19:05:40.110" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32179" PostTypeId="1" CreationDate="2018-05-25T21:52:48.527" Score="2" ViewCount="199" Body="&lt;p&gt;We say Bernoulli naive bayes assumes gaussian distribution of all continuous features. What happens if I have categorical features also in the dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of prior transformation in data is suitable before applying Bernoulli naive baeyes to a dataset ?&lt;/p&gt;&#xA;" OwnerUserId="52726" LastActivityDate="2018-05-26T10:22:58.267" Title="Doubt in interpretation of Bernoulli Naive Baeyes Algorithm" Tags="|naive-bayes-classifier|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32180" PostTypeId="2" ParentId="32179" CreationDate="2018-05-25T22:20:11.777" Score="3" Body="&lt;p&gt;Bernoulli Naive bayes does not assume gaussian distribution of all continuous features, because it does not make sense. Gaussian Naive Bayes assumes gaussian distribution for continuous features and it is the appropriate way for using Naive Bayes approach if you have continuous features. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if you have binary categorical data then the appropriate approach is Bernoulli Naive Bayes. If your features are categorical but not binary then you could transform them into binary categorical using dummy boolean variables for each available value of the categorical features. The main point of Naive Bayes algorithm is the assumption of feature independence, which in some real world classification problems does not hold.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You need to specify a conditional probability p(x|y) of the feature value x given the class label y. Since Naive Bayes assumes that all features are conditionally independent given the class, you can mix different likelihood models for each feature considering any prior knowledge about it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example, considering a continuous feature you might assume that p(x|y) is normally distributed, then you can stimate the mean and variance for this feature under each class in the training set and after that you can use the PDF of the Normal Distribution to estimate p(x|y).&#xA;Considering another feature which is categorical, you can estimate p(x|y) using a Bernoulli or multinomial event model and multiply the two conditional probabilities together in the final prediction (since they are assumed to be independent anyway).&lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="35679" LastEditDate="2018-05-26T10:22:58.267" LastActivityDate="2018-05-26T10:22:58.267" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32181" PostTypeId="1" CreationDate="2018-05-25T23:50:48.487" Score="1" ViewCount="33" Body="&lt;p&gt;I'm interested in any research materials on voting patterns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data set of how PMs (members of parliament) voted in my country during last couple of years. Each PM has 3 buttons: Yes, No, Abstain. There are also two special situations: PM can be absent during the voting and he/she can be present but not vote (it's wrong but they do it sometimes). So any particular voting result is represented as a huge vector whose values are yes/no/abstain/present_not_voted/absent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'd like to find some similarities in PMs' voting patterns and cluster them in groups based on these similarities. There are some interesting questions like coalition stability and party loyalty which (in theory) could be answered with the data described above. Unfortunately i deeply lack any references. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While googling i've come across some site which formed an undirected graph based on American senators voting. It looked cool but i didn't find any explanation on how such graph can be formed. Intuitively &quot;similar&quot; is a symmetric binary relation so undirected graph representation sounds quite natural. But how can one define &quot;similarity&quot; for my data? Probably there are some standard ways of doing this. Any help will be highly appreciated.&lt;/p&gt;&#xA;" OwnerUserId="38802" LastActivityDate="2018-05-26T18:59:54.917" Title="Voting patterns similarities" Tags="|clustering|similarity|social-network-analysis|distance|reference-request|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32182" PostTypeId="1" CreationDate="2018-05-26T00:11:57.480" Score="4" ViewCount="61" Body="&lt;p&gt;I was wondering if labels could be visualized below images in the image grid tool in Image Analytics?&lt;/p&gt;&#xA;&lt;p&gt;I know, this feature might be not terribly useful for images in general, hence unlikely to be deployed in Orange. How hard it would be to modify  &lt;code&gt;image_grid.py&lt;/code&gt; in order to display such image names? Any pointers would be useful.&lt;/p&gt;&#xA;" OwnerUserId="52728" LastEditorUserId="29169" LastEditDate="2020-11-18T21:42:45.173" LastActivityDate="2021-05-21T09:57:01.187" Title="Image grid - labels?" Tags="|orange|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32183" PostTypeId="1" CreationDate="2018-05-26T01:31:56.930" Score="4" ViewCount="6504" Body="&lt;p&gt;I need to write a function which identifies and removes the &quot;*&quot; character after some numeric values in a vector. I also need  that the resultant vector is a numeric vector.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;c(&quot;21,34,99*&quot;, &quot;56,90*&quot;, &quot;45*&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to remove &quot;&lt;code&gt;*&lt;/code&gt;&quot; which is unwanted.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to write function so whenever such data cleaning requirement I can use function and pass certain parameters.&lt;/p&gt;&#xA;" OwnerUserId="52729" LastEditorUserId="29575" LastEditDate="2018-05-26T02:02:10.913" LastActivityDate="2018-05-26T20:23:55.730" Title="how to remove unwanted characters from data" Tags="|r|data-cleaning|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="32185" PostTypeId="1" CreationDate="2018-05-26T04:50:08.060" Score="2" ViewCount="337" Body="&lt;p&gt;I am experimenting with a character-level LSTM model doing the standard task of predicting the next character given a sequence of characters. I am training the network using truncated backpropagation through time (BPTT).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to compare two techniques for setting up the training examples for this task. For my diagrams below, let's say the input sequence is the alphabet: &quot;abcde...&quot; and the horizon size for BPTT is 3. From what I understand, the standard way of training the network would be:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 1 ('d' is the target character):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/A6Ene.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/A6Ene.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Go53i.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Go53i.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 3:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TUr7H.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TUr7H.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Technique #2&lt;/strong&gt;:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is the alternative setup that seems reasonable to me:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JvYrm.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JvYrm.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This training example has three separate target characters. The error from 'd' backpropagates through the full three time steps. However, error from 'c' backpropagates through two time steps and 'b' through one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 2:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/9Y8b1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/9Y8b1.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Training example 3:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KO2qL.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KO2qL.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For training a single epoch over N characters, the second technique has a significant time complexity improvement. Increasing the horizon size has no impact on the running time. A larger horizon size will decrease the number of training examples, but each training example will have more target characters (i.e. more meaningful weight updates).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this second technique used at all for training LSTMs? Any relevant links? Why is the first technique more commonly used? Are there alternative training example setups which are also effective? &lt;/p&gt;&#xA;" OwnerUserId="52731" LastEditorUserId="52731" LastEditDate="2018-05-26T17:41:54.653" LastActivityDate="2018-05-26T17:41:54.653" Title="Alternative method for RNN backpropagation through time" Tags="|machine-learning|neural-network|rnn|lstm|backpropagation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32187" PostTypeId="1" AcceptedAnswerId="32191" CreationDate="2018-05-26T07:25:58.637" Score="4" ViewCount="308" Body="&lt;p&gt;When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score normalisation on min-max normalised data? That is if the input data is already normalised to [0, 1], is it a good idea to train on the z-scores of that? &lt;/p&gt;&#xA;" OwnerUserId="47562" LastEditorUserId="28175" LastEditDate="2018-05-26T09:43:15.793" LastActivityDate="2018-12-23T12:02:25.520" Title="Normalising data with multiple methods" Tags="|machine-learning|neural-network|deep-learning|preprocessing|normalization|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32188" PostTypeId="1" AcceptedAnswerId="32190" CreationDate="2018-05-26T08:02:35.413" Score="3" ViewCount="1047" Body="&lt;p&gt;Is dropping out parts of the Input vector better than dropping out parts of the Output vector? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring pieces of the &lt;strong&gt;input&lt;/strong&gt; means some of the further neurons will be able to see this neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is masking the input pretty much the same as masking the weights (aka DropConnect) and thus gives a higher quality regularization? &lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-05-26T15:19:15.550" LastActivityDate="2018-05-26T15:19:15.550" Title="Dropout on inputs instead on outputs = DropConnect?" Tags="|machine-learning|neural-network|deep-learning|dropout|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32190" PostTypeId="2" ParentId="32188" CreationDate="2018-05-26T09:23:20.807" Score="3" Body="&lt;p&gt;It depends on the type of input pattern but to make a decision, I suggest not to. There are different reasons for that. First of all, you are damaging your input signal. I don't know whether you are familiar with the information theory or not but the signal to noise ratio will be too small and if you do so, you will be left with a signal which is far from your real signal. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover, it is also not good to add dropout in the convolutional layers because they are feature extractors and they are significant features for classification problems. If you miss them, means that you are losing information more than usual. Consider the point that your input to the network is already resized to a smaller shape than its original shape, for instance, the input shape of typical &lt;code&gt;CNNs&lt;/code&gt; is &lt;code&gt;224 * 224&lt;/code&gt; while the original shape may be ten times bigger or even more  for each direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You may have seen that in the Lenet-5 the authors have used a data-augmentation technique that changes the colors of the inputs with different distributions. The point there is that they have not changed the locality of the input signal. Moreover, the signal to noise ratio also is not a too small number due to the fact that they have not set the input features to zeros. They just have changed them slightly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, the last layer should not employ drop-out. Because the output size has to have specified characteristics, sum to one. Dense layers due to having a large number of weights, and consequently a large number of activations, are good points for exploiting drop-out.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-26T10:13:38.357" LastActivityDate="2018-05-26T10:13:38.357" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32191" PostTypeId="2" ParentId="32187" CreationDate="2018-05-26T09:33:20.047" Score="0" Body="&lt;p&gt;I've not seen any paper about that but based on what I've faced till now, normalizing data intuitively is just for assigning same importance to different features which their raw values do not have a same range. Take a look at &lt;a href=&quot;https://datascience.stackexchange.com/a/27617/28175&quot;&gt;here&lt;/a&gt;. Also, you can take a look at &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/lecture/xx3Da/gradient-descent-in-practice-i-feature-scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt; that professor says that you just need to employ a technique and it's not really important which technique. Also, take a look at &lt;a href=&quot;https://www.coursera.org/learn/machine-learning/supplement/CTA0D/gradient-descent-in-practice-i-feature-scaling&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-26T09:50:13.550" LastActivityDate="2018-05-26T09:50:13.550" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32194" PostTypeId="1" AcceptedAnswerId="32198" CreationDate="2018-05-26T12:20:13.857" Score="7" ViewCount="14575" Body="&lt;p&gt;While using Keras' &lt;code&gt;flow_from_directory&lt;/code&gt; method to train my model on a multi-class image classification problem, the &lt;code&gt;predict_generator&lt;/code&gt; function gives the class probabilities.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my query is how to get the corresponding class-labels for those class probabilities?&lt;/p&gt;&#xA;" OwnerUserId="52738" LastEditorUserId="58944" LastEditDate="2018-09-11T21:08:53.610" LastActivityDate="2020-08-07T06:47:15.497" Title="How to predict class label from class probability given by predict_generator for testdata?" Tags="|python|deep-learning|keras|image-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32196" PostTypeId="1" CreationDate="2018-05-26T12:46:20.510" Score="3" ViewCount="255" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/i862Q.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/i862Q.jpg&quot; alt=&quot;The above shows the K-Means clustering algorithm applied to the data points.&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I fail to understand how the circled point is assigned to cluster c2 and not c1. From what I have understood, the points are assigned to the (nearest?) centroid in order to minimise the squared distances i.e. to minimise the assignment cost.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Visually too, it appears to me that the circled point belongs to the first cluster. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please correct me if I'm wrong.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Source - &lt;a href=&quot;http://www.cs.yale.edu/homes/el327/papers/OnlineKMeansAlenexEdoLiberty.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.cs.yale.edu/homes/el327/papers/OnlineKMeansAlenexEdoLiberty.pdf&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52740" LastEditorUserId="50727" LastEditDate="2018-06-12T12:03:49.473" LastActivityDate="2018-06-12T12:03:49.473" Title="Clarification on K Means Clustering Algorithm and its Analysis" Tags="|machine-learning|clustering|k-means|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32197" PostTypeId="2" ParentId="32196" CreationDate="2018-05-26T14:23:18.203" Score="0" Body="&lt;p&gt;Yes, I don't think this is the final result of k-means. K-means is an iterative process involving computing centroids and associating points to clusters. I think that the picture shows the state of the process when centroids have been recomputed but association has not been done for the new centroids.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-26T14:23:18.203" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32198" PostTypeId="2" ParentId="32194" CreationDate="2018-05-26T14:24:28.570" Score="2" Body="&lt;p&gt;You just take the class with the maximum probability. This can be done using numpy argmax function.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-26T14:24:28.570" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32199" PostTypeId="2" ParentId="32196" CreationDate="2018-05-26T14:42:51.427" Score="1" Body="&lt;p&gt;K-means is an iterative algorithm. So what you have in the picture may be one of the early iterations and therefore the sum of squared distances (between points and clusters) has not been minimised yet. In further iterations, the circled point would indeed be assigned to cluster centroid C1.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, having said that, and this is purely speculation, but if the diagram in your picture is not simply 2 dimensional as we interpret it to be, but is actually a 2-d projection from a higher dimensional feature space, then indeed the circled point may be closer to centroid C2 in that higher dimensional space (although it doesn’t look like it in 2-d).&lt;/p&gt;&#xA;" OwnerUserId="41929" LastActivityDate="2018-05-26T14:42:51.427" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32200" PostTypeId="1" CreationDate="2018-05-26T15:19:43.243" Score="6" ViewCount="204" Body="&lt;p&gt;I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I include the prediction matrix (ie. df containing columns of prediction results from the various base learners) or should I just include the original features?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried both methods with just the 'n_estimators' tuned with F1 score as the metric for cross-validation. (learning rate =0.1) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Method 1: With pred matrix + original features:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_estimators = 1 (this means only one tree is included in the model, is this abnormal? )&#xA;F1 Score (Train): 0.907975 (suggest overfitting)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Method 2: With original features only:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;n_estimators = 1&#xA;F1 Score (Train): 0.39&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I am getting rather different results for both methods, which makes sense as the feature importance plot for Method 1 shows that one of the first-level predictions is the most important.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that the first-level predictions by the base-learners should be included in the gridsearch. Any thoughts?&lt;/p&gt;&#xA;" OwnerUserId="52742" LastEditorUserId="29169" LastEditDate="2019-04-04T06:48:16.400" LastActivityDate="2024-03-31T02:04:20.513" Title="Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?" Tags="|python|scikit-learn|xgboost|ensemble|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32201" PostTypeId="1" AcceptedAnswerId="32209" CreationDate="2018-05-26T15:23:57.097" Score="5" ViewCount="3765" Body="&lt;p&gt;I am using GTZAN dataset to make a CNN and classify by musical genres.&lt;/p&gt;&#xA;&lt;p&gt;I'm getting very good results except Val. loss (See Image)&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/o05Xv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/o05Xv.png&quot; alt=&quot;Results&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;I am processing the audio files using Librosa, obtaining the spectogram and then using the power_to_db function.&lt;/p&gt;&#xA;&lt;p&gt;This is my CNN Model:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;class CNNModel(object):&#xA;&#xA;def __init__(self, config, X):&#xA;    self.filters = 32 # number of convolutional filters to use&#xA;    self.pool_size = (2, 2)  # size of pooling area for max pooling&#xA;    self.kernel_size = (3, 3)  # convolution kernel size&#xA;    self.nb_layers = 4&#xA;    self.input_shape = (128, 625, 1) # cambiar por x.shape&#xA;    &#xA;    &#xA;def build_model(self, nb_classes):&#xA;&#xA;    model = Sequential()&#xA;    model.add(&#xA;            Conv2D(&#xA;                self.filters,&#xA;                self.kernel_size,&#xA;                padding ='same',&#xA;                input_shape = self.input_shape))&#xA;    model.add(BatchNormalization(axis=1))&#xA;    model.add(Activation('relu'))&#xA;    &#xA;    model.add(&#xA;            Conv2D(&#xA;                self.filters,&#xA;                self.kernel_size))&#xA;    model.add(Activation('relu'))&#xA;    model.add(MaxPooling2D(pool_size = self.pool_size))&#xA;    model.add(Dropout(0.25))&#xA;    &#xA;    model.add(&#xA;        Conv2D(&#xA;            self.filters + 32,&#xA;            self.kernel_size,&#xA;            padding ='same'))&#xA;    model.add(Activation('relu'))&#xA;    &#xA;    model.add(&#xA;        Conv2D(&#xA;            self.filters + 32,&#xA;            self.kernel_size,&#xA;            padding ='same'))&#xA;    model.add(MaxPooling2D(pool_size = self.pool_size))&#xA;    model.add(Dropout(0.25))&#xA;         &#xA;    model.add(Flatten())&#xA;    model.add(Dense(512))&#xA;    model.add(Activation('relu'))&#xA;    model.add(Dropout(0.5))&#xA;    model.add(Dense(nb_classes))&#xA;    model.add(Activation(&amp;quot;softmax&amp;quot;)) #mirar&#xA;    &#xA;    return model&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I leave the link of my github in case you want to see the whole code.&lt;/p&gt;&#xA;&lt;p&gt;Every song is (128, 625) Shape, I used MinMaxScale to Scale the data.&lt;/p&gt;&#xA;&lt;p&gt;This is my loss function and my optimizer&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;loss = losses.categorical_crossentropy,&#xA;optimizer = optimizers.SGD(lr=0.001, momentum=0, decay=1e-5, nesterov=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;I have read about overfitting and that seems to be the cause but I do not know how to solve it at the code level.&lt;/p&gt;&#xA;&lt;p&gt;Update 1: With Dropout(0.9) I get this results:&#xA;&lt;a href=&quot;https://i.stack.imgur.com/dChx1.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/dChx1.png&quot; alt=&quot;Results 2&quot; /&gt;&lt;/a&gt;&#xA;Thank you&lt;/p&gt;&#xA;" OwnerDisplayName="user45581" LastEditorDisplayName="user45581" LastEditDate="2021-03-03T21:20:21.330" LastActivityDate="2021-03-03T21:20:21.330" Title="Really bad value of Val loss" Tags="|machine-learning|python|neural-network|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32202" PostTypeId="1" AcceptedAnswerId="32226" CreationDate="2018-05-26T15:38:53.270" Score="2" ViewCount="528" Body="&lt;p&gt;Putting aside things applicable to neural networks such as dropout, l2 regularization, new opitmizers - what are the cool things I should be adding to my Reinforcement Learning algorithm (Q-Learning and SARSA) to make it stronger, in &lt;strong&gt;2018&lt;/strong&gt;?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I know these:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Experience Replay (gather experiences, then train at once on some of them)&lt;/li&gt;&#xA;&lt;li&gt;Slowly making our network approach the Q-values of a Target Network. The target Network is only cloned sometimes.&lt;/li&gt;&#xA;&lt;li&gt;If our function approximator is RNN and our Memory Bank stores sequences of several timesteps, &lt;a href=&quot;https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-6-partial-observability-and-deep-recurrent-q-68463e9aeefc&quot; rel=&quot;nofollow noreferrer&quot;&gt;after selecting few random sequences from Memory bank, only backpropagate a half of the sequence, without going back to its start&lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-05-26T15:45:09.800" LastActivityDate="2018-05-27T14:27:41.063" Title="Reinforcement Learning in 2018, best tips and tricks?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32203" PostTypeId="2" ParentId="31168" CreationDate="2018-05-26T15:42:48.507" Score="1" Body="&lt;p&gt;If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset.&#xA;This can be obtained with the plyr library&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(plyr)&#xA;df &amp;lt;- data.frame(Study=c('Study_1', 'Study_1', 'Study_2', 'Study_3'), col1=c(NA,NA,5,8), col2=c(8, NA, NA, 7))&#xA;&#xA;column &amp;lt;- 'col1'           # is the column your are interested in&#xA;&#xA;ddply(df, .(Study), function(x){&#xA;  length(which(is.na(x[column])))/nrow(x)*100&#xA;})&#xA;&#xA;Study  V1&#xA;Study_1 100&#xA;Study_2   0&#xA;Study_3   0&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Study_1 has 100% missing values and the other studies none, so except Study_1 all studies contains some information&lt;/p&gt;&#xA;" OwnerUserId="51329" LastEditorUserId="49700" LastEditDate="2018-10-24T06:03:55.530" LastActivityDate="2018-10-24T06:03:55.530" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32204" PostTypeId="2" ParentId="30352" CreationDate="2018-05-26T15:54:03.167" Score="0" Body="&lt;p&gt;A variance inflation factor(VIF) detects multicollinearity in regression analysis. I learned that a VIF above 10 indicates multicollinearity and should be treated carefully.&lt;br&gt;&#xA;Like Stephen mentioned the vif function is a part of the car library&lt;/p&gt;&#xA;" OwnerUserId="51329" LastActivityDate="2018-05-26T15:54:03.167" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32205" PostTypeId="1" AcceptedAnswerId="32207" CreationDate="2018-05-26T17:09:15.193" Score="1" ViewCount="2589" Body="&lt;p&gt;I have a script to download images, but the images are of different resolutions so I have written a script to shrink the image. I have two options:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;size=(224,224)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with cv2&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cv2.resize(img,size,interpolation=cv2.INTER_AREA)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;with PIL&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;img.thumbnail(size,Image.ANTIALIAS)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;after saving them I see cv2 doesn't maintain the original ratio where as PIL maintains the ratio.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question :&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Maintaining aspect ratio is important or not.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;If yes &lt;code&gt;(224,224)&lt;/code&gt; is a good choice or should I set it to higher&#xA;resolution.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Sorry if the question is naive, I am new to image processing.&lt;/p&gt;&#xA;" OwnerUserId="44673" LastEditorUserId="28175" LastEditDate="2018-05-26T18:23:08.137" LastActivityDate="2018-05-26T18:23:08.137" Title="Resizing images for training with Mobilenets" Tags="|machine-learning|python|neural-network|deep-learning|cnn|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32206" PostTypeId="1" CreationDate="2018-05-26T17:15:54.127" Score="3" ViewCount="103" Body="&lt;p&gt;I have somewhat of a general/high level question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class classification). My supervised model performs fairly well on testing data but what I'm trying to do now is find a way to sort out which &lt;strong&gt;documents&lt;/strong&gt; from the production data (not human labeled) if added to the training set would improve accuracy the most on that particular production data set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is once the initial predictions are made on production data, another algorithm would be run to determine the n most important documents (say 500) that if manually labeled by a human and added to the training set would improve accuracy the most, assuming the model does a 2nd round of prediction after the n documents have been added to the training set.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the process would look like this:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Initial predictions on production data are made&lt;/li&gt;&#xA;&lt;li&gt;Algorithm runs to determine the most important records to increase accuracy of the model (based on features most likely).&lt;/li&gt;&#xA;&lt;li&gt;Human reviews these most important records and adds them to the training set&lt;/li&gt;&#xA;&lt;li&gt;A 2nd round of prediction takes place, hopefully with a better accuracy metric&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I'm thinking of using something like K-Means after doing some dimensionality reduction since it's NLP problem with lots of features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Does anyone have any experience or suggestions in regard to this topic? Am I on the right track?&lt;/p&gt;&#xA;" OwnerUserId="52748" LastActivityDate="2024-04-04T14:01:00.083" Title="Determine the most important documents for supervised learning" Tags="|nlp|text-mining|feature-selection|k-means|unsupervised-learning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32207" PostTypeId="2" ParentId="32205" CreationDate="2018-05-26T17:34:28.813" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Maintaining aspect ratio is important or not.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, it's really important in most cases. As you can read from &lt;a href=&quot;https://digital-photography-school.com/aspect-ratio-what-it-is-and-why-it-matters/&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;, &lt;em&gt;Why does aspect ratio matter? It’s all to do with the relationship of the main subject to the sides of the frame, and the amount of empty space you end up with around the subject. An awareness of the characteristics of the aspect ratio of your particular camera can help you compose better images. It also helps you recognise when cropping to a different aspect ratio will improve the composition of your image.&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In deep learning tasks, it depends how you want to feed data to your network. It's better to train your network with real data. Consequently if you are going to face data with standard aspect ratio, you have too keep it during training.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If yes (224,224) is a good choice or should I set it to higher resolution.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Depending on your task &lt;code&gt;224&lt;/code&gt; may be good or not. In typical classification networks, it is an acceptable size, like the inputs of &lt;em&gt;AlexNet&lt;/em&gt; or &lt;em&gt;VGG&lt;/em&gt; and such customary nets. For localization tasks, not really. For instance, the input height and width of YOLO is more than that &lt;a href=&quot;https://datascience.stackexchange.com/a/27385/28175&quot;&gt;1&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-26T17:34:28.813" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32208" PostTypeId="2" ParentId="32181" CreationDate="2018-05-26T18:59:54.917" Score="1" Body="&lt;p&gt;The politicians and Political parties follow a set of principles and characteristics. They always follow these things during voting on legislations and bills. you can use T-distributed Stochastic Neighbor Embedding (t-SNE), machine learning algorithm for visualization developed by Laurens van der Maaten and Geoffrey Hinton. This method helps us to visualize the clusters and similarities of the observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Like-minded MP's and political parties with similar principles cluster together(neighbors). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The t-SNE &lt;a href=&quot;https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;algorithm comprises two main stages. First, t-SNE constructs a probability distribution over pairs of high-dimensional objects in such a way that similar objects have a high probability of being picked, whilst dissimilar points have an extremely small probability of being picked. Second, t-SNE defines a similar probability distribution over the points in the low-dimensional map, and it minimizes the Kullback–Leibler divergence between the two distributions with respect to the locations of the points in the map. Note that whilst the original algorithm uses the Euclidean distance between objects as the base of its similarity metric, this should be changed as appropriate.&lt;/p&gt;&#xA;" OwnerUserId="44617" LastActivityDate="2018-05-26T18:59:54.917" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32209" PostTypeId="2" ParentId="32201" CreationDate="2018-05-26T20:12:45.987" Score="2" Body="&lt;p&gt;You can test with higher values of Dropout (0.5,0.7,0.9) and/or try L1/L2 Regularization to combat overfitting  : &lt;a href=&quot;https://keras.io/regularizers/&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;Keras Regularizers&quot;&gt;Keras Regularizers&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;:&#xA;You can play with a combination of l1/l2 regularization and dropout for your convolutional and FC layers. Start with low values of lambda (0.001) and increase it thereon. A common practice is to have dropout only in the FC layers, see if it helps your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, from the looks of your loss curve, your model probably hasn't converged.You can train it until your validation loss starts going up/becomes stable.&lt;/p&gt;&#xA;" OwnerUserId="30805" LastEditorUserId="30805" LastEditDate="2018-05-27T19:42:50.770" LastActivityDate="2018-05-27T19:42:50.770" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32210" PostTypeId="2" ParentId="32183" CreationDate="2018-05-26T20:23:55.730" Score="1" Body="&lt;p&gt;Should have been posted in SO as others rightly pointed out. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Simple solution would be to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;Split on '*' and select the first part of the split.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Apply &lt;em&gt;(1)&lt;/em&gt; over list items and return a vector.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x &amp;lt;- c(&quot;21,34,99*&quot;, &quot;56,90*&quot;, &quot;45*&quot;)               #test case&#xA;&#xA;data_clean &amp;lt;- function(x) sapply (strsplit(x , '[*]' ), `[` , 1)&#xA;data_clean(x)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="51659" LastActivityDate="2018-05-26T20:23:55.730" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32211" PostTypeId="2" ParentId="32196" CreationDate="2018-05-26T22:33:16.797" Score="1" Body="&lt;p&gt;K-means clusters are always convex.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even when not converged.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This does not hold here, and therefore that slide is not depicting k-means correctly. It's probably a &lt;strong&gt;bad set of slides&lt;/strong&gt;. There are many many really bad slides on clustering.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-05-26T22:33:16.797" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32212" PostTypeId="2" ParentId="32187" CreationDate="2018-05-27T00:27:19.640" Score="1" Body="&lt;p&gt;Depends on the dataset, and what you're looking for. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is possible to do multiple data transformations. Standardization is often the safest choice, but only when the data is normally distributed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you have normalized the data, you can always apply standardisation. Standardization does not mutate the data set, but rescaling does create a new data set. From the standardized data set you can go back to the original data set.&lt;/p&gt;&#xA;" OwnerUserId="44393" LastActivityDate="2018-05-27T00:27:19.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32214" PostTypeId="1" CreationDate="2018-05-27T06:13:11.190" Score="1" ViewCount="1526" Body="&lt;p&gt;What should be taken as a parameter to plot ROC curves , for example in a classification model, I can get predicted labels and predicted probabilities.&#xA;I have softmax layer in the output layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I use predicted labels or predicted probabilities to plot ROC curve in a classification problem.&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-05-27T10:29:16.767" Title="predicted labels or predicted probabilities in a ROC plot?" Tags="|machine-learning|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32215" PostTypeId="2" ParentId="32151" CreationDate="2018-05-27T07:40:31.953" Score="0" Body="&lt;p&gt;You can try running SVM just on this similarity matrix. But you'll then need to provide the sikikaritirs also for new data points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Furthermore, SVMs rely on the similarities bring dot products in &lt;em&gt;some&lt;/em&gt; vector space. If they aren't you may get inconsistencies.&#xA;They &lt;em&gt;may&lt;/em&gt; rely on the triangle inequality being satisfied for the distance function d(x,y)=sqrt(2k(x,y)-k(x,x)-k(y,y)). Although I cannot find a clear reference on whether or not this is needed. If k is a scalar product in some vector space, this obviously is satisfied.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Last but not least, SVMs are good for larger amounts of data, where you cannot afford to keep the entire similarity matrix in memory! By reducing the data set to the support vectors only, the resulting classifier will need much less memory and much less time. Much of the challenge of learning a SVM is to manage memory during training.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-05-27T07:40:31.953" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32216" PostTypeId="1" CreationDate="2018-05-27T08:11:01.457" Score="2" ViewCount="258" Body="&lt;p&gt;I have the following dataset (.csv format) which contains: &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;100 columns: $\textbf{Period}$ (in years, e.g. 2017, 2018, ..., 2028), $\textbf{Gross Revenue}$, $\textbf{Region}$ (e.g. APAC, NEMEA, etc), 97 other columns as $\textbf{factors}$ (e.g. # of customers, # of goods produced, etc).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;1,000,000 rows&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;where for $\textbf{Period}$, only 2017 is actual data, and 2018, ..., 2028 are $\textbf{planned}$, in the sense that data from these years are what the company expects to commit to and receive in these years in the future.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, I want to predict the gross sales for each $\textbf{Region}$ in 2018.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried using regression with stepwise, but the linear model that resulted from the stepwise had NA values for a majority of the predictors, and also the probability arising from the t-test.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've consulted a few of my pals whom are working in the field of data science &amp;amp; analytics, and they all told me that it is impossible to predict for 2018..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it possible to do so given the state of the data? Some insight on this will be deeply appreciated!&lt;/p&gt;&#xA;" OwnerUserId="49782" LastActivityDate="2018-05-28T04:01:21.667" Title="How to predict next year's gross revenue given this year's data?" Tags="|predictive-modeling|bigdata|random-forest|prediction|forecasting|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32217" PostTypeId="1" CreationDate="2018-05-27T09:18:01.627" Score="3" ViewCount="3404" Body="&lt;p&gt;I understand that the LSTM recurrent neural network has a forget gate,  input gate, and output gate. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I do not understand how the equation below calculates 'forget' information (from Chris Olah's &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;LSTM post&lt;/a&gt;) &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/PB51i.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/PB51i.png&quot; alt=&quot;LSTM forget gate layer and equation&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How does this forget gate decide which information is discarded from the previous cell state? In other words, what is forgotten or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It appears to me that this equation just takes &lt;span class=&quot;math-container&quot;&gt;$x_t$&lt;/span&gt; from the current state and &lt;span class=&quot;math-container&quot;&gt;$h_{t-1}$&lt;/span&gt; from the previous state. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone explain how this equation works as a forget gate to retain or discard cell state?&lt;/p&gt;&#xA;" OwnerUserId="51026" LastEditorUserId="32697" LastEditDate="2019-01-11T14:31:34.627" LastActivityDate="2020-04-09T08:24:32.947" Title="How is the LSTM RNN forget gate calculated?" Tags="|deep-learning|lstm|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32218" PostTypeId="2" ParentId="32217" CreationDate="2018-05-27T09:53:40.847" Score="3" Body="&lt;p&gt;The equation and value of &lt;span class=&quot;math-container&quot;&gt;$f_t$&lt;/span&gt; by itself does not fully explain the gate. You need to look at first term of the next step:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$C_t = f_t \odot C_{t-1} + i_t \odot \bar{C}_{t}$&lt;/span&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The vector &lt;span class=&quot;math-container&quot;&gt;$f_t$&lt;/span&gt; that is the output from the forget gate, is used as element-wise multiply against the previous cell state &lt;span class=&quot;math-container&quot;&gt;$C_{t-1}$&lt;/span&gt;. It is this stage where individual elements of &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; are &quot;remembered&quot; or &quot;forgotten&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Due to the sigmoid function, the vector &lt;span class=&quot;math-container&quot;&gt;$f_t$&lt;/span&gt; behaves like a binary classifier for each element, with saturated values tending to settle on either not modifying &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; at all (a value of &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt;) or &quot;forgetting&quot; what the previous value was (a value of &lt;span class=&quot;math-container&quot;&gt;$0$&lt;/span&gt;). Of course intermediate values are also possible, and the analogy there between simply remembering and forgetting values is less direct.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The analogy with forgetting or remembering helps towards understanding the improvements regarding preserving gradients over multiple timesteps. For any step where an element of &lt;span class=&quot;math-container&quot;&gt;$f_t$&lt;/span&gt; is close to &lt;span class=&quot;math-container&quot;&gt;$1$&lt;/span&gt; (and thus the effect of previous timesteps is being &quot;remembered&quot;), then the corresponding gradient elements backpropagating from &lt;span class=&quot;math-container&quot;&gt;$\nabla_{\theta} C_{t}$&lt;/span&gt; to &lt;span class=&quot;math-container&quot;&gt;$\nabla_{\theta} C_{t-1}$&lt;/span&gt; remain the same, avoiding the gradient loss seen in more simple RNN architectures (especially related to saturated values). Once a previous element in &lt;span class=&quot;math-container&quot;&gt;$C$&lt;/span&gt; is &quot;forgotten&quot;, then the error gradient connection is cut off between time steps.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2020-04-09T08:24:32.947" LastActivityDate="2020-04-09T08:24:32.947" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32219" PostTypeId="2" ParentId="32214" CreationDate="2018-05-27T10:29:16.767" Score="1" Body="&lt;p&gt;Use the predicted labels.&lt;br&gt;&#xA;Explanation - The ROC curve show possible classification performance at different setups.&lt;br&gt;&#xA;Where 'classification performance' is a factor of True-positive-rate and False-positive-rate. And 'different setups' are thresholds used to determine a predicted label(class) based on predicted probability.&#xA;The latter is the reason why you should use predicted labels and not predicted probability. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some additional context:&lt;br&gt;&#xA;In my experience, ROC curve can be useful for three things:&lt;br&gt;&#xA;1) Determining optimal setup(probability threshold) for a given model. Based on possible setups and problem needs.&lt;br&gt;&#xA;2) Evaluating overall model performance and different models comparison. This can be achieved using the &lt;a href=&quot;https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it&quot;&gt;AUC&lt;/a&gt;.&lt;br&gt;&#xA;3) Understanding classification model behavior and identifying possible abnormalities. For example, if the curve has a step like pattern its possible your model is too simple.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here is a &lt;a href=&quot;https://www.theanalysisfactor.com/what-is-an-roc-curve/&quot; rel=&quot;nofollow noreferrer&quot;&gt;short nice explanation&lt;/a&gt; on the ROC curve. &lt;/p&gt;&#xA;" OwnerUserId="27983" LastActivityDate="2018-05-27T10:29:16.767" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32220" PostTypeId="1" CreationDate="2018-05-27T11:04:57.507" Score="3" ViewCount="1055" Body="&lt;p&gt;I am trying to use &lt;code&gt;NasNetlarge&lt;/code&gt; in Keras without the top but I cant get rid of the top:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from keras.applications import NASNetLarge&#xA;model = NASNetLarge(input_shape=(224, 224, 3), include_top=False, &#xA;                    weights=&amp;quot;imagenet&amp;quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;If I use any shape other than the (331, 331, 3) shape, I get the error that I must use (331, 331, 3) if I &lt;code&gt;include_top=True&lt;/code&gt; even though I set &lt;code&gt;include_top=False&lt;/code&gt;. If I set the input shape to (331, 331, 3) and &lt;code&gt;include_top&lt;/code&gt; to &lt;code&gt;False&lt;/code&gt;, it downloads the weights with the top. Am I doing something wrong or is there a bug in the Keras module?&lt;/p&gt;&#xA;" OwnerUserId="37300" LastEditorUserId="29169" LastEditDate="2021-02-13T04:20:15.670" LastActivityDate="2024-03-27T06:02:00.207" Title="Keras bug NasNetlarge no top" Tags="|machine-learning|python|keras|convolutional-neural-network|transfer-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32221" PostTypeId="1" CreationDate="2018-05-22T12:33:28.123" Score="0" ViewCount="245" Body="&lt;p&gt;I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One approach is to use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_2.append(sum(np.min(cdist(data&#xA;                              , kmeanModel.cluster_centers_&#xA;                              , 'euclidean')&#xA;                              , axis = 1)) / data.shape[0])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Another is to use inertia_ from sklearn.cluster.KMeans:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;distortions_3.append(kmeanModel.inertia_)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?&lt;/p&gt;&#xA;" OwnerUserId="51845" OwnerDisplayName="lstodd" LastActivityDate="2018-05-27T12:22:53.200" Title="What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?" Tags="|python|scikit-learn|k-means|" AnswerCount="1" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32222" PostTypeId="2" ParentId="32221" CreationDate="2018-05-23T05:02:19.827" Score="1" Body="&lt;p&gt;K-means does not minimize Euclidean distances, but &lt;em&gt;squared&lt;/em&gt; Euclidean distances. This is not the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The nearest center is the same for both, but the &lt;em&gt;mean&lt;/em&gt; only optimizes the squares. You can find the counterexample on my earlier answers here.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The proper value to use for the basic elbow approach is the &quot;inertia&quot;, which is the sum of squares. But some evaluation measures may use other distances. For example the Silhouette index &lt;em&gt;can&lt;/em&gt; be used with other distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget that these approaches are just heuristics for trying some k. Always also consider other values.&lt;/p&gt;&#xA;" OwnerUserId="924" OwnerDisplayName="Anony-Mousse" LastActivityDate="2018-05-23T05:02:19.827" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32223" PostTypeId="1" CreationDate="2018-05-27T12:29:20.160" Score="2" ViewCount="709" Body="&lt;p&gt;I have tried creating a pipeline like this but I am getting error&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;- AttributeError: 'Pipeline' object has no attribute 'compile'&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = StandardScaler()&#xA;model = Sequential()&#xA;&#xA;model.add(Dense(120, input_dim=46,activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(80, activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(40, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;&#xA;pipeline = make_pipeline(scaler,model)&#xA;pipeline.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;&#xA;pipeline.fit(X_train,y_train, epochs=50, batch_size=20, validation_data = (X_test,y_test))&#xA;&#xA;# evaluate the model&#xA;scores = pipeline.evaluate(X_test,y_test)&#xA;print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38596" LastActivityDate="2018-07-27T04:05:13.487" Title="How to create pipeline for Neural Network models?" Tags="|neural-network|deep-learning|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32224" PostTypeId="2" ParentId="32223" CreationDate="2018-05-27T12:33:52.947" Score="0" Body="&lt;p&gt;You should compile your model before you put it into the pipeline.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = StandardScaler()&#xA;model = Sequential()&#xA;&#xA;model.add(Dense(120, input_dim=46,activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(80, activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(40, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;pipeline = make_pipeline(scaler,model)&#xA;pipeline.fit(X_train,y_train, epochs=50, batch_size=20, validation_data = (X_test,y_test))&#xA;&#xA;# evaluate the model&#xA;scores = pipeline.evaluate(X_test,y_test)&#xA;print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-27T12:33:52.947" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32225" PostTypeId="2" ParentId="32223" CreationDate="2018-05-27T12:47:37.847" Score="1" Body="&lt;p&gt;I am able to achieve this with below code &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks  @JahKnows&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;scaler = StandardScaler()&#xA;model = Sequential()&#xA;&#xA;model.add(Dense(120, input_dim=46,activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(80, activation='relu'))&#xA;model.add(Dropout(0.1, noise_shape=None, seed=None))&#xA;&#xA;model.add(Dense(40, activation='relu'))&#xA;model.add(Dense(1, activation='sigmoid'))&#xA;model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])&#xA;&#xA;pipeline = Pipeline([(&quot;scaler&quot;,scaler),(&quot;model&quot;,model)])&#xA;pipeline.fit(X_train,y_train, model__epochs=50, model__batch_size=20, model__validation_data = (X_test,y_test))&#xA;&#xA;# evaluate the model&#xA;nn_pred = grid_search.predict(X_test)&#xA;&#xA;print('Accuracy of NN on test is:',accuracy_score(y_test,nn_pred))&#xA;print(&quot;\n%s: %.2f%%&quot; % (model.metrics_names[1], scores[1]*100))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="38596" LastActivityDate="2018-05-27T12:47:37.847" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32226" PostTypeId="2" ParentId="32202" CreationDate="2018-05-27T14:27:41.063" Score="1" Body="&lt;p&gt;Have you looked at the &lt;a href=&quot;https://arxiv.org/pdf/1710.02298.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Rainbow RL&lt;/a&gt;? It combines all improvements of RL. Apart from structural changes other improvements come from Reward Shaping, Curiosity-driven Learning, RL with Unsupervised Auxiliary tasks and RL with external memory (e.g. Neural Episodic Control, Model-free episodic Control). It's pitty that you leave out from your question Policy Gradients, they are lots of fun :) Happy reading!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastActivityDate="2018-05-27T14:27:41.063" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32230" PostTypeId="2" ParentId="28317" CreationDate="2018-05-27T19:51:15.337" Score="1" Body="&lt;p&gt;Since the problem you are trying to tacle is image classification, then classification accuracy is the appropriate measure of comparison. Also you could consider the Precision, Recall and F1 metrics (for multi-class problems). See below there is description of an extension about these multi-class metrics.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://dl.acm.org/citation.cfm?id=513543&quot; rel=&quot;nofollow noreferrer&quot;&gt;A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems&lt;/a&gt;  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another appropriate metric is AUC/ROC, which has to be extended cosnidering the multi-class case. See the link below &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Receiver Operating Characteristic (ROC)&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="35679" LastEditDate="2018-05-28T14:55:43.687" LastActivityDate="2018-05-28T14:55:43.687" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32231" PostTypeId="2" ParentId="31785" CreationDate="2018-05-27T20:24:47.043" Score="0" Body="&lt;p&gt;Looks like the problem lied just in poorly chosen parameters. In the example given above, simply increasing epoch count significantly improved the results. In my actual image processing problem, I had to decrease learning rate, since with the default one, ReLU's were saturating very quickly, hindering further progress.&lt;/p&gt;&#xA;" OwnerUserId="52311" LastActivityDate="2018-05-27T20:24:47.043" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32232" PostTypeId="1" CreationDate="2018-05-27T21:51:47.203" Score="1" ViewCount="674" Body="&lt;p&gt;I'm just starting off as a data scientist and i need to understand how regression, random forest and much more.. algorithms function under the hood, the maths behind them and when to use them. I started with some MOOC courses but not a fan, it's just not my style of learning as i need practical examples or books i can dive into with detailed examples i can just apply myself after understanding how each algorithm works and some 20 mins vids with scarce maths explanation is not what i aim for, i've done some research and ended up with these books:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-python for data analysis&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-Python Crash Course&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-oreilly hands on machine learning with scikit learn and tensorflow&lt;/p&gt;&#xA;&#xA;&lt;p&gt;-automate the boring stuff with python&lt;/p&gt;&#xA;&#xA;&lt;p&gt;kindly note that i have some python experience(less than a year) and  unfamiliar with tensorflow , keras, scikit-learn etc.. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Kindly also note that i didn't mention any maths books to explain how the machine learning algorithms work so i'm open to suggestions  .&lt;/p&gt;&#xA;" OwnerUserId="52774" LastActivityDate="2018-05-28T17:42:34.083" Title="What is the best references / books to get familiar with machine learning algorithms?" Tags="|machine-learning|python|statistics|machine-learning-model|" AnswerCount="2" CommentCount="0" ClosedDate="2018-06-01T16:03:41.697" ContentLicense="CC BY-SA 4.0" />
  <row Id="32233" PostTypeId="1" AcceptedAnswerId="32253" CreationDate="2018-05-27T22:29:23.580" Score="1" ViewCount="669" Body="&lt;p&gt;I have built a multiclass perceptron, but it has low accuracy (around 80%). I think I'm missing something. One possibility is that I should add a bias, but I'm not sure how to incorporate that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The task is, given 2 dimensions, predict the class, which is between 0 and 8. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I could use some pointers as to where this code is going wrong. We are not given classes for test data, but rather need to populate it and it is checked elsewhere.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def dot_product(weights_list, training_data):&#xA;    return [sum([a * b for a, b in zip(training_data, i)]) for i in weights_list]&#xA;&#xA;def predict(row, weights_list, n_classes):&#xA;    # get dimensions * weight&#xA;    activations = dot_product(weights_list, row[:-1])&#xA;    # get index of argmax&#xA;    predicted_label = activations.index(max(activations))&#xA;    return predicted_label&#xA;&#xA;def train_weights(train, n_epoch, n_classes):&#xA;    # prepopulate just weights&#xA;    weights_list = [[0.0 for x in range(len(train[0])-1)] for x in range(n_classes)]&#xA;    for epoch in range(n_epoch):&#xA;        for row in train:&#xA;            actual_class = int(row[-1])&#xA;            # argmax from predicting each class&#xA;            prediction = int(predict(row, weights_list, n_classes))&#xA;            # if incorrect:&#xA;            #   lower score of wrong answer by this row's values&#xA;            #   raise score of correct answer by this row's values&#xA;            if actual_class != prediction:&#xA;                weights_list[prediction] = [a - b for a, b in zip(weights_list[prediction], row[:-1])]&#xA;                weights_list[actual_class] = [a + b for a, b in zip(weights_list[actual_class], row[:-1])]&#xA;&#xA;    return weights_list&#xA;&#xA;train = [[83.0, -14.0, 6.0],&#xA;        [77.0, 15.0, 6.0],&#xA;        [93.0, 35.0, 3.0],&#xA;        [86.0, -8.0, 6.0],&#xA;        [-51.0, -79.0, 1.0],&#xA;        [62.0, -73.0, 1.0]]&#xA;&#xA;test = [[36.0, 27.0, -1],&#xA;        [6.0, 99.0, -1],&#xA;        [-3.0, 16.0, -1],&#xA;        [-40.0, -61.0, -1],&#xA;        [70.0, 67.0, -1],&#xA;        [86.0, -14.0, -1],&#xA;        [-92.0, 67.0, -1]]&#xA;&#xA;n_classes = 9&#xA;n_epoch = 10000&#xA;weights_list = train_weights(train, n_epoch, n_classes)&#xA;&#xA;for row in test:&#xA;    prediction = predict(row, weights_list, n_classes)&#xA;    row[2] = prediction&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="10799" LastEditorUserId="10799" LastEditDate="2018-05-28T03:35:59.667" LastActivityDate="2018-05-30T03:37:21.450" Title="Trouble with accuracy of multiclass perceptron" Tags="|machine-learning|python|perceptron|" AnswerCount="1" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="32234" PostTypeId="2" ParentId="32232" CreationDate="2018-05-27T22:41:51.113" Score="0" Body="&lt;p&gt;If you want to get more into Machine Learning and the mathematical/theoritical aspects that it includes, the book mentioned below is an option: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Machine Learning by Tom Mitchell, McGraw Hill, 1997.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-27T22:41:51.113" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32235" PostTypeId="1" CreationDate="2018-05-27T22:59:36.220" Score="1" ViewCount="115" Body="&lt;p&gt;I have data that has some features such as latitude, longitude, timestamp, and category of an event. So the data set only has the events that actually took place. The thing is, I am wondering if there are preferred models that can be used to predict the following cases:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;The place (latitude and longitude) of next event given the current timestamp.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The high probable time that event could occur given the current place.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The probability of an event taking place given both time and space data.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="52775" LastEditorUserId="29169" LastEditDate="2021-10-13T17:23:43.550" LastActivityDate="2021-10-13T17:23:43.550" Title="spatio-temporal event prediction" Tags="|machine-learning|python|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32236" PostTypeId="1" CreationDate="2018-05-27T23:41:23.123" Score="1" ViewCount="264" Body="&lt;p&gt;has somebody an idea/approach how to do text mining for a SWOT analysis? (e.g. sentiment analysis) I need to assign categories (Strengths, Weaknesses, Opportunities, Risks) to words in a document and then rank them.&lt;/p&gt;&#xA;" OwnerUserId="52776" LastActivityDate="2018-05-27T23:41:23.123" Title="How to do an automated SWOT analysis with Text Mining" Tags="|text-mining|" AnswerCount="0" CommentCount="1" ClosedDate="2018-05-31T05:18:56.773" ContentLicense="CC BY-SA 4.0" />
  <row Id="32237" PostTypeId="1" CreationDate="2018-05-28T02:26:04.340" Score="2" ViewCount="600" Body="&lt;p&gt;How to plot calibration curve for multi-class problems, for example the available example on python plots it for 2 classes but here in the e-book link it is done for multi-class&lt;a href=&quot;https://books.google.co.in/books?id=tHBZDwAAQBAJ&amp;amp;pg=PA12&amp;amp;lpg=PA12&amp;amp;dq=calibration%20curve%20for%20mnist&amp;amp;source=bl&amp;amp;ots=msQlL5ZF9R&amp;amp;sig=RVPpl-NnEg4OR1KNE60_i5edf50&amp;amp;hl=en&amp;amp;sa=X&amp;amp;ved=0ahUKEwi6oq2NmabbAhUWT48KHdCCAS8Q6AEIbTAK#v=onepage&amp;amp;q=calibration%20curve%20for%20mnist&amp;amp;f=false&quot; rel=&quot;nofollow noreferrer&quot;&gt;book&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;how can I do so in python or R?&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-05-28T02:26:04.340" Title="How to plot calibration curve for multi-class problems?" Tags="|machine-learning|classification|multiclass-classification|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32238" PostTypeId="1" CreationDate="2018-05-28T03:53:17.983" Score="1" ViewCount="1332" Body="&lt;p&gt;I am working on a simple linear regression model,&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is my Python code : &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import matplotlib.pyplot as plt&#xA;import pandas as pd&#xA;&#xA;dataset=pd.read_csv('sample.csv')&#xA;X=dataset.iloc[:,:-1].values&#xA;Y=dataset.iloc[:,1].values&#xA;&#xA;from sklearn.model_selection import train_test_split&#xA;X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=1/3)&#xA;&#xA;from sklearn.linear_model import LinearRegression&#xA;regressor=LinearRegression()&#xA;regressor.fit(X_train,Y_train)&#xA;&#xA;plt.scatter(X_train,Y_train,color='red')&#xA;plt.plot(X_train,regressor.predict(X_train),color='blue')&#xA;plt.title('X vs Y(Training Set)')&#xA;plt.xlabel('X')&#xA;plt.ylabel('Y')&#xA;plt.show()&#xA;&#xA;plt.scatter(X_test,Y_test,color='red')&#xA;plt.plot(X_train,regressor.predict(X_train),color='blue')&#xA;plt.title('X vs Y(Test Set)')&#xA;plt.xlabel('X')[enter image description here][1]&#xA;plt.ylabel('Y')&#xA;plt.show()`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;This is my plot of Training Set &lt;a href=&quot;https://i.stack.imgur.com/dybyT.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Training Set &lt;/a&gt;&#xA;  This is my plot of Test Set &lt;a href=&quot;https://i.stack.imgur.com/LnPT9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;Test Set&lt;/a&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;How can I increase the efficiency of my ML model???&#xA;This is my first ML model, so all suggestions are welcome.&#xA;Thanks in Advance&lt;/p&gt;&#xA;" OwnerUserId="52780" LastActivityDate="2018-09-25T20:01:44.087" Title="Simple Linear Regression-----How to make my model more efficient??" Tags="|python|linear-regression|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32239" PostTypeId="2" ParentId="32216" CreationDate="2018-05-28T04:01:21.667" Score="2" Body="&lt;p&gt;Predicting next year's revenue from previous years' revenues is Time Series Forecasting.&#xA;You will need data from many previous years to do this. One year is clearly not enough.&#xA;Just think about it this way: Can you predict tomorrow stock's price given only today price?&lt;/p&gt;&#xA;" OwnerUserId="45726" LastActivityDate="2018-05-28T04:01:21.667" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32240" PostTypeId="1" CreationDate="2018-05-28T04:02:28.850" Score="2" ViewCount="96" Body="&lt;p&gt;For Deep Learning using Tensorflow, is it necessary to resize the images to a predefined width and height before before training using a model?&#xA;     OR&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In model configuration file , I noticed this snippet? Does this mean all training images are resized automatically by tesorflow ?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  image_resizer {&#xA;      keep_aspect_ratio_resizer {&#xA;        min_dimension: 600&#xA;        max_dimension: 1024&#xA;      }&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My positive training site has varied sizes..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;corners_40N1_0.jpg 323*283&lt;/p&gt;&#xA;&#xA;&lt;p&gt;corners_40N1_1.jpg 318*271&lt;/p&gt;&#xA;&#xA;&lt;p&gt;corners_40N1_2.jpg 341*306&lt;/p&gt;&#xA;&#xA;&lt;p&gt;corners_40N1_3.jpg 318*329&lt;/p&gt;&#xA;&#xA;&lt;p&gt;corners_45O16_0.jpg 250*280&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Negative training data are of equal size.&#xA;56K7NE_80.jpg 360*360&lt;/p&gt;&#xA;&#xA;&lt;p&gt;56K7NE_81.jpg 360*360&lt;/p&gt;&#xA;&#xA;&lt;p&gt;56K7NE_82.jpg 360*360&lt;/p&gt;&#xA;&#xA;&lt;p&gt;56K7NE_83.jpg 360*360&lt;/p&gt;&#xA;&#xA;&lt;p&gt;56K7NE_84.jpg 360*360 &lt;/p&gt;&#xA;" OwnerUserId="35503" LastEditorUserId="35503" LastEditDate="2018-05-30T04:28:36.583" LastActivityDate="2018-05-30T04:28:36.583" Title="Image resizing to be done programatically or taken care by model config file in tensorflow?" Tags="|deep-learning|tensorflow|machine-learning-model|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32241" PostTypeId="1" CreationDate="2018-05-28T04:45:23.517" Score="3" ViewCount="3143" Body="&lt;p&gt;I have trained my model on the very big dataset (approx 1200000 train images, and 15000 classes) using Keras' &lt;code&gt;flow_from_directory&lt;/code&gt; method. I have successfully trained my model using &lt;code&gt;fit_generator&lt;/code&gt;. But, now I am facing the problem for &lt;code&gt;predict_generator&lt;/code&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got the following error while executing &lt;code&gt;predict_generator&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Found 0 images belonging to 0 classes.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;However, I have converted my train image dataset into the subdirectory based on class labels. But, for test dataset, I don't have any labels. &#xA;So how can I overcome this issue?&lt;/p&gt;&#xA;" OwnerUserId="52738" LastEditorUserId="24000" LastEditDate="2018-05-28T10:06:18.920" LastActivityDate="2022-03-29T20:18:55.867" Title="predict_generator gives &quot;Found 0 images belonging to 0 classes.&quot;" Tags="|machine-learning|python|deep-learning|keras|image-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32242" PostTypeId="2" ParentId="32144" CreationDate="2018-05-28T04:52:15.773" Score="1" Body="&lt;p&gt;If I understand your question properly, this seems to be a scraping problem which you can do using Beautifulsoup in python.&lt;/p&gt;&#xA;" OwnerUserId="52784" LastActivityDate="2018-05-28T04:52:15.773" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32244" PostTypeId="1" CreationDate="2018-05-28T06:40:54.240" Score="2" ViewCount="492" Body="&lt;p&gt;In computer vision, if we don't have a large training set, a common method is to start with a pre-trained model for some related task (e.g., ImageNet) and fine-tune that model to solve our problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can something similar be done with natural language processing problems?  I have a boolean classification problem on sentences and don't have a large enough training set to train a RNN from scratch.  In particular, is there a good way to fine-tune a LSTM or 1D CNN or otherwise do transfer learning?  And, if we want to do classification on sentences, is there a reasonable pre-trained model to start from?&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2024-01-24T17:45:32.197" Title="Fine-tuning NLP models" Tags="|neural-network|deep-learning|lstm|nlp|transfer-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32245" PostTypeId="1" CreationDate="2018-05-28T06:58:11.940" Score="1" ViewCount="114" Body="&lt;p&gt;I'm new to learning neural nets and trying to recreate stuff in order to get an understanding of how things work. I'd like to replicate a pointer network to train on the travelling salesman problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I'm able to understand most of what the paper by Vinyals et al 2015. The point of confusion is the use of the special starting token (=&gt;)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand that for text you could create such a token. But how does it translate to co-ordinate data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would be really useful if someone could expand on this or point to a resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="49234" LastActivityDate="2018-05-28T06:58:11.940" Title="What do I use as initial token? TSP using pointer networks" Tags="|python|deep-learning|keras|rnn|lstm|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32246" PostTypeId="1" AcceptedAnswerId="32348" CreationDate="2018-05-28T07:22:33.077" Score="11" ViewCount="7784" Body="&lt;p&gt;I am having a hard time understanding difference between Target Network and Double DQN&lt;/p&gt;&#xA;&lt;p&gt;From &lt;a href=&quot;https://medium.com/@awjuliani/simple-reinforcement-learning-with-tensorflow-part-4-deep-q-networks-and-beyond-8438a3e2b8df&quot; rel=&quot;noreferrer&quot;&gt;this blog:&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;Target Network generates the target-Q values that will be used to compute the loss for every action during training. The target network’s weights are fixed, and are frequently but by small amounts updated towards the primary Q-networks values.&lt;/p&gt;&#xA;&lt;p&gt;Double DQN: instead of taking the max over Q-values when computing the target-Q value for our training step, we use our primary network to chose an action, and our target network to generate the target Q-value for that action&lt;/p&gt;&#xA;&lt;p&gt;That sounds very similar to me, and the equation makes me lost too :(&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;qTarget = r + γQ(  s’,  argmax(Q(s’,a,ϴ),ϴ’)  )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;From whaat I know, &lt;strong&gt;Target Network Approach&lt;/strong&gt; allows us to do the&#xA;following:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;using Online Network, get scores for each action as the response for the current state. Pick the most appropriate action &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; and take a&#xA;note of its score &lt;span class=&quot;math-container&quot;&gt;$q$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;using that action, &amp;quot;travel&amp;quot; to the next state and get scores from &lt;strong&gt;Target Net&lt;/strong&gt; for even further actions.&lt;/li&gt;&#xA;&lt;li&gt;Select the most appropriate action from step 2) and call it &lt;span class=&quot;math-container&quot;&gt;$A$&lt;/span&gt;. also, take a note at its score &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt;&lt;/li&gt;&#xA;&lt;li&gt;Punish OnlineNetwork for any difference between &lt;span class=&quot;math-container&quot;&gt;$q$&lt;/span&gt; and &lt;span class=&quot;math-container&quot;&gt;$Q$&lt;/span&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Note: the gradient flows &lt;strong&gt;only&lt;/strong&gt; through &lt;span class=&quot;math-container&quot;&gt;$a$&lt;/span&gt; component of the &amp;quot;output&#xA;vector&amp;quot; &lt;span class=&quot;math-container&quot;&gt;$\{a, b, c, d ...\}$&lt;/span&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Please correct me if these steps are wrong&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2020-07-16T01:48:36.850" LastActivityDate="2020-07-16T01:48:36.850" Title="Q-Learning: Target Network vs Double DQN" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32247" PostTypeId="2" ParentId="32238" CreationDate="2018-05-28T07:53:39.647" Score="0" Body="&lt;p&gt;Your data does not follow a linear trend. For this reason, your linear model has clear limitations. In order to overcome them, you can build a nonlinear model. With the few data that you have, I advise you to do &lt;a href=&quot;https://en.wikipedia.org/wiki/Local_regression&quot; rel=&quot;nofollow noreferrer&quot;&gt;LOESS&lt;/a&gt;. As your training data seems to be able to follow a cubic trend, you can also try with polynomial regression of degree 3.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Bear in mind that the distribution of your train and test data is very different, so you are very likely to overfit. For this reason, it might not be the best data to get started with ML.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-28T07:53:39.647" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32248" PostTypeId="1" AcceptedAnswerId="32250" CreationDate="2018-05-28T07:54:11.520" Score="3" ViewCount="254" Body="&lt;p&gt;My task is to estimate a person's age based on a face image of that person.&lt;br&gt;&#xA;To that end I'm using a CNN and at first stage I was based on the following article: &lt;a href=&quot;https://www.vision.ee.ethz.ch/en/publications/papers/articles/eth_biwi_01299.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;DeepExpectation&lt;/a&gt; which uses a VGG16 architecture to predict a person apparent age (the age that other people would vote). I'm using ResNet 50 architecture (and I'm using this implementation of it: &lt;a href=&quot;https://github.com/tensorflow/models/tree/master/official/resnet&quot; rel=&quot;nofollow noreferrer&quot;&gt;ResNet Tensorflow&lt;/a&gt; in tensorflow). The dataset I use for learning is taken from the same article above and can be downloaded from here: &lt;a href=&quot;https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/&quot; rel=&quot;nofollow noreferrer&quot;&gt;WIKI-IMDB dataset&lt;/a&gt;. It is composed of 523,051 face images with tagged ages but I found that most of them are garbage (doesn't have real age as the label or has more than one face in it or no face at all). after filtering this dataset (by throwing images that have more than one face or no face at all or the label that represents the age doesn't make sense) I'm left with approximately 150k images in my dataset. in the process of making this dataset I centered all faces in the image and cropped it to be 224*224*3. To increase the size of my dataset I flipped horizontally each image in my dataset to get a total of 300k images in my dataset. I then split the dataset to 240k train, 30k validation and 30k test images.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm loading a pre-trained model (trained on ImageNet) and I hoped to get the results they got in the article above (something close to 3.2 years with MAE as the evaluation metric). I also should mention that I'm using cross-entropy as the loss and an l2 loss for regularization and I have 101 classes as the logits (ages 0-100). The batch size I'm using is 64 (The maximum my GPU can handle with). I tried several combinations of hyperparameters but until now the best MAE I got on the validation set was 5.8 years.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Examples of loss graphs I got and their hyperparameters:&lt;br&gt;&#xA;weight decay = 5e-4,&lt;br&gt;&#xA;momentum optimizer with momentum=0.9,&lt;br&gt;&#xA;first learning rate = 0.01 and reducing it by a factor of 10 each 2 epochs until reaching to 1e-6,&lt;br&gt;&#xA;the above LR is the LR assinged to the last FC layer. for the layer before it I used 0.5 of that LR. for the middle layers I used 1e-2 of that LR and for the first layer I used 1e-3 of that LR.&lt;br&gt;&#xA;I got the following graphs: (orange = train , blue = validation)&lt;br&gt;&#xA;  loss: &lt;a href=&quot;https://i.stack.imgur.com/bVMgR.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bVMgR.jpg&quot; alt=&quot;Loss&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;  LR: &lt;a href=&quot;https://i.stack.imgur.com/aPWct.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aPWct.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;  MAE: &lt;a href=&quot;https://i.stack.imgur.com/rCSno.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rCSno.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;example #2:&lt;br&gt;&#xA;same hyperparameters as in the example above only different ratios between first layers and last FC layer. in this example I used 0.5 of the LR in the last FC layer for the layer before it and 0.1 of the LR in the last FC layer for the other layers.&lt;br&gt;&#xA;I got the following graphs: (orange = train , blue = validation)&lt;br&gt;&#xA;  loss: &lt;a href=&quot;https://i.stack.imgur.com/5fpQh.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5fpQh.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;  LR: &lt;a href=&quot;https://i.stack.imgur.com/tGDIX.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tGDIX.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;br&gt;&#xA;  MAE: &lt;a href=&quot;https://i.stack.imgur.com/CFh7D.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/CFh7D.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see from the last 2 examples, I'm reaching to ~6 pretty fast but it seems as the optimization on the train loss gets stuck and also that the validation loss doesn't approach the train loss from some point.&#xA;Do you have any suggestions?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, I'm having some thoughts on whether it is good to apply some of the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Initalize weights from the pre-trained network until some layer&lt;/li&gt;&#xA;&lt;li&gt;Initialize all weights from the pre-trained network beside the batch-norm&lt;/li&gt;&#xA;&lt;li&gt;Freeze some of the layers after loading them and train only the last layers&lt;/li&gt;&#xA;&lt;li&gt;Apply different LR to different layers (as I did in the examples above)&lt;/li&gt;&#xA;&lt;li&gt;Should I use Momentum optimizer or maybe Adam Optimizer&lt;/li&gt;&#xA;&lt;li&gt;If I'm using Momentum optimizer, what should be the learning rate schedule (learning rate decay)?&lt;/li&gt;&#xA;&lt;li&gt;If I'm using Adam optimizer, does it make sense to use different learning rates to different layers?&lt;/li&gt;&#xA;&lt;li&gt;What should be the weight decay hyperparameter?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Any help would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't know which stackexchange site is more appropriate for this question so I also posted this question on CrossValidated: &lt;a href=&quot;https://stats.stackexchange.com/questions/348569/difficulty-in-choosing-hyperparameters-for-my-cnn&quot;&gt;Difficulty in choosing Hyperparameters for my CNN&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52786" LastEditorUserId="29169" LastEditDate="2021-02-13T04:19:51.893" LastActivityDate="2021-02-13T04:19:51.893" Title="Difficulty in choosing Hyperparameters for my CNN" Tags="|machine-learning|deep-learning|tensorflow|convolutional-neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32249" PostTypeId="2" ParentId="32145" CreationDate="2018-05-28T07:59:14.140" Score="6" Body="&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Stop_words&quot; rel=&quot;noreferrer&quot;&gt;Stop words&lt;/a&gt; are common words in a language and they are usually removed &lt;strong&gt;when there appearance is not indicative to the analysis goal&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Suppose our goal is to text mine and find out if a given text is about sports or politics. If we use a bag of words, &quot;not&quot; is probably a stop word we should remove since it is probably not indicative of either categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand, if we would like to differ between &quot;This article will not discuss politics&quot; and &quot;this article will discuss politics&quot;, we must not remove &quot;not&quot;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The examples you gave look like programming languages &lt;a href=&quot;https://en.wikipedia.org/wiki/Reserved_word&quot; rel=&quot;noreferrer&quot;&gt;reserved words&lt;/a&gt;. In most languages you can find this documentation and skip the learning phase. Looking at the frequent tokens will probably also give you the reserved words (and probably plenty of English stop words that will appear in the remarks). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, before utilising such a list, be sure that neglecting them will serve your goal. In case that typing them doesn't differ from typing other words, you might better leave them. This way you will have more data the represent better the user behaviour. &lt;/p&gt;&#xA;" OwnerUserId="13727" LastActivityDate="2018-05-28T07:59:14.140" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32250" PostTypeId="2" ParentId="32248" CreationDate="2018-05-28T08:06:34.667" Score="2" Body="&lt;p&gt;I am going to comment on your ideas:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.&lt;/li&gt;&#xA;&lt;li&gt;Initialize everything that you can from pre-trained network.&lt;/li&gt;&#xA;&lt;li&gt;I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, &lt;a href=&quot;https://towardsdatascience.com/transfer-learning-using-differential-learning-rates-638455797f00&quot; rel=&quot;nofollow noreferrer&quot;&gt;as this post shows&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Totally, as shown above.&lt;/li&gt;&#xA;&lt;li&gt;Adam is more widely used, I would use that.&lt;/li&gt;&#xA;&lt;li&gt;Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.&lt;/li&gt;&#xA;&lt;li&gt;It does make sense.&lt;/li&gt;&#xA;&lt;li&gt;I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-28T08:06:34.667" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="32251" PostTypeId="2" ParentId="27100" CreationDate="2018-05-28T08:30:14.520" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But I can't feed those data to catboost because of catboost only&#xA;  accept the Categorical variable.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That's not true. Actually CatBoost uses continuous variables by default, and to use categorical ones you'd need to tell your estimators which features are categorical with &lt;a href=&quot;https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostregressor_fit-docpage/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;cat_features&lt;/code&gt;&lt;/a&gt; parameter.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you still want to use this column as categorical, then there are two options: as you didn't specify whether this column contains words or some longer text, you could either one-hot encode words (in the first case) or use tf-idf features on text (in the latter). In the second case you'd have to note though that GBMs don't work well when there are thousands of features (which is common for tf-idf or Bag of Words models).&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-28T08:30:14.520" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32253" PostTypeId="2" ParentId="32233" CreationDate="2018-05-28T09:26:26.510" Score="2" Body="&lt;p&gt;Firstly, in all the linear separator algorithms such as linear regression, logistic regression and the perceptron, adding the bias is as simple as adding a feature column consisting of all 1's. Then the third weight that will be trained will act as the bias $b$.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;I have some working code for a multi-class perceptron&lt;/p&gt;&#xA;&#xA;&lt;p&gt;First let's generate some artificial data with 2 features. Each distribution of points is going to be Gaussian with a given mean and variance described as a list of lists for each dimension under the variable &lt;code&gt;params&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gen_data(params, n):&#xA;    dims = len(params[0])&#xA;    num_classes = len(params)&#xA;&#xA;    x = np.zeros((n*num_classes, dims))&#xA;    y = np.zeros((n*num_classes,))&#xA;    for ix, i in enumerate(range(num_classes)):&#xA;        inst = np.random.randn(n, dims)&#xA;&#xA;        for dim in range(dims): x[ix*n:(ix+1)*n,dim] = np.random.normal(params[ix][dim][0], &#xA;                                                                        params[ix][dim][1], n)&#xA;        y[ix*n:(ix+1)*n] = ix&#xA;    return x, y&#xA;&#xA;params = [[[ 5,1],  [ 5,1]], &#xA;          [[ 0,1],  [ 0,1]],&#xA;          [[2, 1],  [ 2,1]],&#xA;          [[-2, 1],  [ 2,1]]]&#xA;&#xA;n = 300&#xA;x, y = gen_data(params, 300)&#xA;&#xA;plt.scatter(x[:,0], x[:,1])&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/AA1sX.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/AA1sX.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alright so now we have 4 distributions with different labels. Let's split the data for sanctity's sake.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's train the weights using the training data&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def get_weights(x, y, n_epochs, verbose = 0):&#xA;    # Append a ones column to the feature for the bias&#xA;    data = np.ones((x.shape[0], x.shape[1]+1))&#xA;    data[:, 0:x.shape[1]] = x&#xA;    # Set the targets as integers for comparison&#xA;    targets = y.astype(int)&#xA;&#xA;    # Initialize the weights as a matrix &#xA;    # number of classes by number of features&#xA;    weights = np.ones((len(set(y)), x.shape[1]+1))&#xA;    for epoch in range(n_epochs):&#xA;        for i, target in zip(data, targets):&#xA;            temp = np.dot(i, weights.T)&#xA;            pred = np.argmax(temp)&#xA;            # If wrongly predicted update prediction&#xA;            if pred != target:&#xA;                weights[target, :] = weights[target, :] + i&#xA;                weights[pred, :] = weights[pred, :] - i&#xA;&#xA;        if verbose == 1:&#xA;            print('Iteration: ', epoch)&#xA;            print(weights)&#xA;            print('---------------------------------------------')&#xA;    return weights&#xA;&#xA;weights = get_weights(x_train, y_train, n_epochs = 30, verbose = 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This converges to approximately this&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[[  23.62752045   16.03867499 -111.        ] &lt;br/&gt; &#xA;  [  -3.96545848  -8.66924406   47.        ] &lt;br/&gt;[  -0.94290763   -1.84413793   33.        ] &lt;br/&gt; [ -14.71915434   -1.52529301   35.        ]]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We get an accuracy calculated using the score&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def predict(x, weights):&#xA;    data = np.ones((  x.shape[0], x.shape[1]+1  ))&#xA;    data[:, 0:x.shape[1]] = x&#xA;    predictions = np.argmax(np.dot(data, weights.T), axis = 1)&#xA;    return predictions&#xA;&#xA;def score(x, y, weights):&#xA;    pred = predict(x, weights)&#xA;    return sum(pred == y_test)/len(pred)&#xA;&#xA;score(x_test, y_test, weights)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;0.8686868686868687&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We can check our results using a confusion matrix. For the training set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.metrics import confusion_matrix&#xA;&#xA;predictions = predict(x_train, weights)&#xA;plt.imshow(confusion_matrix(y_train, predictions))&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jlLqQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jlLqQ.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And the testing set&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;predictions = predict(x_test, weights)&#xA;plt.imshow(confusion_matrix(y_test, predictions))&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/YqIIh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/YqIIh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So we see that in fact our algorithm is performing quite well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We can then plot our points to see how it is classifying them. I will plot the training points as small circles, and the testing points as larger ones. The dark points are those which are misclassified&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;colors = ['y', 'r', 'b', 'g', 'k']&#xA;&#xA;# Predict training set&#xA;predictions = predict(x_train, weights)&#xA;for i, t, p in zip(x_train, y_train, predictions):&#xA;    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2, s=20)&#xA;    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)&#xA;&#xA;# Predict test set&#xA;predictions = predict(x_test, weights)&#xA;for i, t, p in zip(x_test, y_test, predictions):&#xA;    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2)&#xA;    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)&#xA;&#xA;# Plot the linear separators&#xA;x1 = np.linspace(np.min(x[:,0]),np.max(x[:,1]),2)&#xA;x2 = np.zeros((weights.shape[0], 2))&#xA;for ix_w, weight in enumerate(weights):&#xA;    x2 = 1 * ( - weight[2] - weight[0]*x1) / weight[1]&#xA;    plt.plot(x1, x2, c = colors[ix_w])&#xA;&#xA;plt.xlabel('Feature 1')&#xA;plt.ylabel('Feature 2')&#xA;&#xA;plt.xlim([np.min(x[:,0]), np.max(x[:,0])])&#xA;plt.ylim([np.min(x[:,1]), np.max(x[:,1])])&#xA;&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/H3YMh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/H3YMh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;This code generalizes to the binary classification task as well&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;params = [[[ 5,1],  [ 5,1]], &#xA;          [[ 0,1],  [ 0,1]]]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Re1Mc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Re1Mc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/STMPI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/STMPI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Stop training on convergence&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;If you want to stop the algorithm based on convergence you can use a stop criteria. For example you can stop training once every weight in your matrix changes by less than a very small number. The very small number we usually choose is machine epsilon &lt;code&gt;2.220446049250313e-16&lt;/code&gt;, which is essentially zero. Sometimes this requirement is too stringent so it can be replaced by any number of significant values.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Change the get_weights code to include the break criteria as&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from copy import deepcopy&#xA;&#xA;def get_weights(x, y, n_epochs, verbose = 0):&#xA;    # Append a ones column to the feature for the bias&#xA;    data = np.ones((x.shape[0], x.shape[1]+1))&#xA;    data[:, 0:x.shape[1]] = x&#xA;    # Set the targets as integers for comparison&#xA;    targets = y.astype(int)&#xA;&#xA;    # Initialize the weights as a matrix &#xA;    # number of classes by number of features&#xA;    weights = np.zeros((len(set(y)), x.shape[1]+1))&#xA;    past_weights = np.zeros((len(set(y)), x.shape[1]+1))&#xA;    for epoch in range(n_epochs):&#xA;        for i, target in zip(data, targets):&#xA;            temp = np.dot(i, weights.T)&#xA;            pred = np.argmax(temp)&#xA;            # If wrongly predicted update prediction&#xA;            if pred != target:&#xA;                weights[target, :] = weights[target, :] + i&#xA;                weights[pred, :] = weights[pred, :] - i&#xA;&#xA;        if np.abs(weights - past_weights).all() &amp;lt; np.finfo(float).eps:&#xA;            break&#xA;        past_weights = deepcopy(weights)&#xA;&#xA;        if verbose == 1:&#xA;            print('Iteration: ', epoch)&#xA;            print(weights)&#xA;            print('---------------------------------------------')&#xA;    return weights&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastEditorUserId="29587" LastEditDate="2018-05-30T03:37:21.450" LastActivityDate="2018-05-30T03:37:21.450" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32254" PostTypeId="1" AcceptedAnswerId="32385" CreationDate="2018-05-28T09:34:17.570" Score="5" ViewCount="168" Body="&lt;p&gt;If I have, for example, a classification network which can tell if there is a dog or a cat in a picture, is it possible to adapt the network so it can also learn to detect a mouse? Without making a new one from scratch.&#xA;In this case it doesn't make sense, but I'm wondering if for example Netflix has to retrain its network completely with every new show they add. And if so, when do they do that? The first few days may be the most crucial ones, but also the ones with the least data to train a network from.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The actual problem I do have is that I'm trying to train a network to predict the location of public transport vehicles. That by itself is hard enough, but what do I do if there is a new transport line, which I don't have an input neuron for. - Here I would need to add an input neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or another idea I'm thinking about is a neural network which can help me order my documents by detecting the company and date. I would like to classify an unknown document layout, a few times, so the neural network is able to classify it by itself, but without losing everything it learned so far. - Here I would need to add an output neuron.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It seems like there is a way to do this, since there are some machine learning algorithms out there which seem to pull off stuff like that. Or do they somehow work around this? If so, how?&lt;/p&gt;&#xA;" OwnerUserId="52793" LastEditorUserId="29575" LastEditDate="2018-05-30T11:54:42.797" LastActivityDate="2018-05-30T11:54:42.797" Title="Can neural networks be adapted without recreating them completely?" Tags="|neural-network|deep-learning|classification|transfer-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32255" PostTypeId="1" AcceptedAnswerId="32269" CreationDate="2018-05-28T09:41:38.437" Score="3" ViewCount="640" Body="&lt;p&gt;Assume that we have two small sets of feature vectors (each feature vector representing an item). What is a good way of finding which features have the biggest difference (in distribution) between the two (small, 50 samples) sets? Given that the distribution of the data per feature is not necessarily normal.&lt;/p&gt;&#xA;" OwnerUserId="52796" LastEditorUserId="29169" LastEditDate="2019-06-07T21:50:50.203" LastActivityDate="2019-06-07T21:50:50.203" Title="Find most important features which differentiate two sets" Tags="|machine-learning|data-cleaning|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32256" PostTypeId="1" AcceptedAnswerId="32504" CreationDate="2018-05-28T10:34:56.000" Score="1" ViewCount="91" Body="&lt;p&gt;When I intialize Faster R-CNN in the deployment phase, the number of samples per image (parameter from config file: TEST.RPN_POST_NMS_TOP_N) is set to 300, that's the number of predicted bounding boxes to keep after non-max suppression. However, the network is initialized with the number set to 1: &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;('rpn/output', (1, 512, 14, 14))&#xA;('rpn/output_rpn_relu/3x3_0_split_0', (1, 512, 14, 14))&#xA;('rpn/output_rpn_relu/3x3_0_split_1', (1, 512, 14, 14))&#xA;('rpn_cls_score', (1, 18, 14, 14))&#xA;('rpn_bbox_pred', (1, 36, 14, 14))&#xA;('rpn_cls_score_reshape', (1, 2, 126, 14))&#xA;('rpn_cls_prob', (1, 2, 126, 14))&#xA;('rpn_cls_prob_reshape', (1, 18, 14, 14))&#xA;('rois', (1, 5))&#xA;('pool5', (1, 512, 7, 7))&#xA;('fc6', (1, 4096))&#xA;('fc7', (1, 4096))&#xA;('fc7_relu7_0_split_0', (1, 4096))&#xA;('fc7_relu7_0_split_1', (1, 4096))&#xA;('cls_score', (1, 21))&#xA;('bbox_pred', (1, 84))&#xA;('cls_prob', (1, 21))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The ones I'm particularly interested are &lt;code&gt;fc6/7&lt;/code&gt;, &lt;code&gt;bbox_pred&lt;/code&gt; and &lt;code&gt;cls_prob&lt;/code&gt;. After &lt;code&gt;net.forward(**kwargs)&lt;/code&gt; is run, the first dimension of these layers is changed to 300: &lt;code&gt;(300,4096), (300, 84), (300,21)&lt;/code&gt; to match the number of RoIs. The &lt;code&gt;rois&lt;/code&gt; output is reshaped in the TargetLayer class, but rest are a bit of a problem: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Caffe doesn't implement this out of the box, there should be some wrapper for this, but I can't find it. Any suggestions on where to look? I want to implement something similar for my algorithm.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is a bit confusing because all these four layer (&lt;code&gt;fc6/7, bbox_pred, cls_prob&lt;/code&gt;) are just fully connected layers defined in the config, nothing fancy.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;PS Also I don't think (100,4096) means 100 times more weights, it would cetrainly be undoable for such layer size, so the weights are shared. But how? &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2018-06-01T13:30:54.473" Title="Faster R-CNN wrapper for the number of RPNs in the layer dimensions?" Tags="|cnn|faster-rcnn|" AnswerCount="1" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32257" PostTypeId="1" AcceptedAnswerId="32268" CreationDate="2018-05-28T10:59:18.043" Score="2" ViewCount="49" Body="&lt;p&gt;I'd like to get a recommendation how to attack a problem of predicting multiple numbers.&#xA;Training data contains 4 columns, each says a probability of record being in the bucket. So, for example:&#xA;&lt;code&gt;X = [0.25, 0.5, 0.25, 0.0]&lt;/code&gt;&#xA;and&#xA;corresponding output should be e.g. &lt;code&gt;Y = [0.8, 0.1, 0.0, 0.1]&lt;/code&gt;.&#xA;Each row should sum to 1.0&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What type of approach would you recommend?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I already tried simple neural network with 4 neurons in the last layer and softmax activation but wondering if there is a better solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks!&lt;/p&gt;&#xA;" OwnerUserId="18501" LastActivityDate="2018-05-28T14:24:03.857" Title="Regression - predict n numbers based on another n numbers" Tags="|regression|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32258" PostTypeId="2" ParentId="32145" CreationDate="2018-05-28T11:05:39.970" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://github.com/AnanthaRajuCprojects/Reserved-Key-Words-list-of-various-programming-languages&quot; rel=&quot;nofollow noreferrer&quot;&gt;Here&lt;/a&gt; is an overview of reserved words, which dal mentioned. &lt;/p&gt;&#xA;" OwnerUserId="52589" LastActivityDate="2018-05-28T11:05:39.970" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32260" PostTypeId="1" AcceptedAnswerId="32362" CreationDate="2018-05-28T11:43:56.073" Score="1" ViewCount="1193" Body="&lt;p&gt;I am trying to cluster my datasets using affinity propagation. I followed &lt;a href=&quot;https://www.toptal.com/machine-learning/clustering-algorithms&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; and &lt;a href=&quot;http://scikit-learn.org/stable/modules/clustering.html#affinity-propagation&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; links to grasp the basics of affinity propagation clustering. The sample code available at &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/cluster/plot_affinity_propagation.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn&lt;/a&gt; is as follows:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cluster import AffinityPropagation&#xA;from sklearn import metrics&#xA;from sklearn.datasets.samples_generator import make_blobs&#xA;# Generate sample data&#xA;centers = [[1, 1], [-1, -1], [1, -1]]&#xA;X, labels_true = make_blobs(n_samples=300, centers=centers, cluster_std=0.5,&#xA;                            random_state=0)&#xA;# Compute Affinity Propagation&#xA;af = AffinityPropagation(preference=-50).fit(X)&#xA;cluster_centers_indices = af.cluster_centers_indices_&#xA;labels = af.labels_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;While running it, it works as stated on the website. However, I could not understand it completely. I want to modify this code to use with my datasets. My datasets consist of values from different sensors on a 2-D surface. I want to cluster the values with similar sensors readings at various points on the 2-D surface. How can I do it?&lt;/p&gt;&#xA;&lt;p&gt;Thank you.&lt;/p&gt;&#xA;" OwnerUserId="50881" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-30T06:57:01.640" Title="How to use Scikit-learn's affinity propagation clustering with my own datasets?" Tags="|machine-learning|scikit-learn|clustering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32261" PostTypeId="2" ParentId="32145" CreationDate="2018-05-28T12:53:30.123" Score="1" Body="&lt;p&gt;About what you are talking about for the programming languages the right definition is reserved words. A reserved word is a word that cannot be used as an identifier, such as the name of a variable, function, or label, so it is &quot;reserved from use&quot;. This is a syntactic definition and it is related to the compiling procedure of the source code of a program. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Stop words are not the same as reserved words. Stop words are words which are filtered out before or after processing of natural language data. Stop words usually are the most common words in a language and there is no single universal list of stop words for each different language. Stop words are related to the natural language (text) and not to the source code in any programming laguage.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastEditorUserId="29575" LastEditDate="2020-04-21T14:56:23.887" LastActivityDate="2020-04-21T14:56:23.887" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32262" PostTypeId="1" CreationDate="2018-05-28T12:54:09.540" Score="2" ViewCount="334" Body="&lt;p&gt;How to correctly model a if condition to choose estimator/predictor(linear regression, gbt) to be used in scikit/spark-ml in a single pipeline.   &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if feature_x &amp;lt; constant: &#xA;    result = pipeline1.predict(feature_vector)&#xA;else:&#xA;    result = pipeline2.predict(feature_vector)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Other than modelling it as custom transformer/predictor, is there a alternate way to model it in a pipeline&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;" OwnerUserId="52803" LastActivityDate="2018-05-28T12:54:09.540" Title="Modelling if condition of multiple estimators in a pipeline" Tags="|python|scikit-learn|apache-spark|data-product|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32263" PostTypeId="1" CreationDate="2018-05-28T13:04:03.507" Score="1" ViewCount="83" Body="&lt;p&gt;I have seen &lt;a href=&quot;https://datascience.stackexchange.com/q/16347&quot;&gt;this question&lt;/a&gt; on the site, however it is asking for the techniques that can be used for implementation. The thing I want to know is that, on the other hand, if are there any &lt;strong&gt;APIs or services for financial pattern recognition&lt;/strong&gt;, i.e. capable of finding particular patterns that occur in a given set of data? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;Think like that:&lt;/em&gt; I supply the data and if there are head-and-shoulders patterns detected in it; the API returns the index of the left/right shoulder etc.&lt;/p&gt;&#xA;" OwnerUserId="52804" LastActivityDate="2018-05-28T13:04:03.507" Title="Are there any APIs for financial pattern recognition?" Tags="|machine-learning|finance|" AnswerCount="0" CommentCount="0" ClosedDate="2018-06-02T02:03:27.687" ContentLicense="CC BY-SA 4.0" />
  <row Id="32264" PostTypeId="1" AcceptedAnswerId="32276" CreationDate="2018-05-28T13:16:29.283" Score="56" ViewCount="69890" Body="&lt;p&gt;I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see the main difference between them in terms of performance estimation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I see, bootstrapping is also producing a certain number of random training+testing subsets (albeit in a different way) so what is the point, advantage for using this method over CV? The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this. But this aspect seems to be a very little nuisance.&lt;/p&gt;&#xA;" OwnerUserId="9011" LastActivityDate="2020-05-20T23:49:33.117" Title="What is the difference between bootstrapping and cross-validation?" Tags="|cross-validation|model-evaluations|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32265" PostTypeId="1" AcceptedAnswerId="122858" CreationDate="2018-05-28T13:30:23.343" Score="1" ViewCount="135" Body="&lt;p&gt;I would like to know how is determined the dimension of the context in &lt;code&gt;Gensim&lt;/code&gt; &lt;code&gt;Doc2Vec&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="32429" LastActivityDate="2023-07-21T04:50:59.070" Title="How is determined the context's dimension in Doc2Vec?" Tags="|word2vec|word-embeddings|gensim|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32266" PostTypeId="2" ParentId="32264" CreationDate="2018-05-28T14:13:44.560" Score="7" Body="&lt;p&gt;Bootstrapping is any test or metric that relies on random sampling with replacement.It is a method that helps in many situations like validation of a predictive model performance, ensemble methods, estimation of bias and variance of the parameter of a model etc. It works by performing sampling with replacement from the original dataset, and at the same time assuming that the data points that have not been choses are the test dataset. We can repeat this procedure several times and compute the average score as estimation of our model performance. Also, Bootstrapping is related to the ensemble training methods, because we can build a model using each bootstrap datasets and “bag” these models in an ensemble using the majority voting (for classification) or computing the average (for numerical predictions) for all of these models as our final result.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Cross validation is a procedure for validating a model's performance, and it is done by splitting the training data into k parts. We assume that the k-1 parts is the training set and use the other part is our test set. We can repeat that k times differently holding out a different part of the data every time. Finally, we take the average of the k scores as our performance estimation. Cross validation can suffer from bias or variance. Increasing the number of splits, the variance will increase too and the bias will decrease. On the other hand, if we decrease the number of splits, the bias will increase and the variance will decrease. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, Cross validation splits the available dataset to create multiple datasets, and Bootstrapping method uses the original dataset to create multiple datasets after resampling with replacement. Bootstrapping it is not as strong as Cross validation when it is used for model validation. Bootstrapping is more about building ensemble models or just estimating parameters.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-28T14:13:44.560" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32268" PostTypeId="2" ParentId="32257" CreationDate="2018-05-28T14:24:03.857" Score="2" Body="&lt;p&gt;I would recommend using an &lt;strong&gt;encoder-decoder structure based on LSTM-RNN for sequence-to-sequence learning&lt;/strong&gt;; it will provide you with exactly what you want. A link that I find very helpful is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/&lt;/a&gt; &lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-28T14:24:03.857" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32269" PostTypeId="2" ParentId="32255" CreationDate="2018-05-28T14:42:06.420" Score="2" Body="&lt;p&gt;A reformulation of your question would be:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Which are the features that differentiate the two datasets more accurately, because they differ a lot across them?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A common way to answer this is by checking the &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_gain_in_decision_trees&quot; rel=&quot;nofollow noreferrer&quot;&gt;information gain&lt;/a&gt; of the attributes, based on the reduction of the &lt;a href=&quot;https://en.wikipedia.org/wiki/Entropy_(information_theory)&quot; rel=&quot;nofollow noreferrer&quot;&gt;entropy&lt;/a&gt; of the dataset. The attributes with the highest information gain are the ones that separate the dataset more accurately.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming you have &lt;strong&gt;m&lt;/strong&gt; attributes, you need to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Label your samples based on which dataset they belong (A or B)&lt;/li&gt;&#xA;&lt;li&gt;Attempt to split the dataset based on the values of each attribute. This means you need to perform &lt;strong&gt;m splits&lt;/strong&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The split that maximizes the information gain, thus minimizes the individual sub-entropy of the two splitted groups, shows the attribute that separates the data the best. This attribute is the one that differs the most in both datasets and is the answer to your question.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;FYI this is exactly what is done when training decision trees, to cut out (prune) features that are not informative enough in order to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Reduce the tree's variance&lt;/li&gt;&#xA;&lt;li&gt;Reduce training / inferencing time&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-28T14:42:06.420" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32270" PostTypeId="2" ParentId="32238" CreationDate="2018-05-28T14:54:23.890" Score="1" Body="&lt;p&gt;You cannot really do much: the fit of the regressor is optimum, therefore it is the best that the algorithm can do given these points. What you can do, is change the weights of individual points of the dataset using the &lt;strong&gt;sample weight&lt;/strong&gt; parameter of &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;LinearRegression.fit(X, y, sample_weight)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;method, to &quot;attract&quot; the line towards them and see how this affects the accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would &lt;strong&gt;not expect&lt;/strong&gt; notable difference because your data do not follow a linear pattern.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-28T14:54:23.890" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32271" PostTypeId="2" ParentId="28497" CreationDate="2018-05-28T14:59:06.847" Score="0" Body="&lt;p&gt;Take a look at &lt;a href=&quot;https://stackoverflow.com/questions/20414667/cocktail-party-algorithm-svd-implementation-in-one-line-of-code&quot;&gt;this&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No DNN, but math, if different channels are available.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;DNN were used for single channel input, but have to be &lt;a href=&quot;https://www.technologyreview.com/s/537101/deep-learning-machine-solves-the-cocktail-party-problem/&quot; rel=&quot;nofollow noreferrer&quot;&gt;trained to the signals you want to separate&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="52811" LastEditorUserId="29575" LastEditDate="2018-05-28T15:22:12.733" LastActivityDate="2018-05-28T15:22:12.733" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32272" PostTypeId="1" CreationDate="2018-05-28T15:24:19.363" Score="3" ViewCount="49" Body="&lt;p&gt;I am new into Artificial Intelligence field and I am working on the classic example of Spam detection using classification. I am using Naive Bayes algorithm as well as SVM. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While working on them it came to my mind that having the email addresses of who has sent the emails as an extra feature would improve the accuracy of my algoithm as we will have some extra info besides the content of the email itself. Would it be true? And how could I use this extra feature? Should I just add it to my email content? &lt;/p&gt;&#xA;" OwnerUserId="50405" LastActivityDate="2018-05-28T16:49:02.960" Title="How would knowing spammers email address improve spam detection algorithms?" Tags="|machine-learning|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32273" PostTypeId="1" AcceptedAnswerId="33064" CreationDate="2018-05-28T16:34:09.583" Score="3" ViewCount="2279" Body="&lt;p&gt;Task: Build CNN Model (preferably Keras or TensorFlow) to Predict Labels Associated to Each Image in &lt;a href=&quot;http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;CelebA Dataset&lt;/a&gt; (Multi-label Image Classification)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In past, for majority of multiclass/binary image classification problems, I used to feed images efficiently using &lt;code&gt;ImageDataGenerator&lt;/code&gt; and &lt;code&gt;.flow_from_directory&lt;/code&gt; in Keras after images are properly organized in a separate directory for each class. Therefore, I have never bothered converting images to numpy.arrary prior to feeding to the model, unless I had to and of course datasets were small so that I could do it easily in my local machine.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, CelebA is a &lt;em&gt;Multi-label Image Classification&lt;/em&gt; with each images having 40 labels (attributes like Smiling, Eyeglasses, Young etc.)  meaning that I can not organize them in subclasses as I used to do, so &lt;code&gt;.flow_from_directory&lt;/code&gt; is off the table (as far as I know!). Still I've managed to convert the images to numpy.array by the following simple loop:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import skimage.transform&#xA;&#xA;images_path='../img_align_celeba/'&#xA;&#xA;train_images=[]&#xA;from skimage import data&#xA;&#xA;for filename in train['Images'].tolist()):&#xA;    tmp=np.array(skimage.transform.resize(io.imread(os.path.join(images_path,filename))/255., (64, 64)))&#xA;    train_images.append(tmp)&#xA;&#xA;x_train=np.array(train_images)&#xA;del train_images&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Well it was not an impossible task. CelebA dataset is large, well not super large compared to many other image datasets (&gt;200K RGB images, totally 1.4GB in size, each image ~ 8 KB). YET surprisingly it takes the hell of the time to convert these images to numpy arrays and even stuck during the run of a small CNN model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My computer specs: MacBook Pro (2015), Memory: 8GB, Harddisk: 128 GB.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even with almost more than 4GM free memory, and 20 GB free hard-disk I could not manage this on my local machine. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; It seems quite possible and way more efficient via &lt;code&gt;.flow_from_directory&lt;/code&gt; method of &lt;code&gt;ImageDataGenerator&lt;/code&gt; in Keras. While it is not that option for this multi label classification at hand, I simply made some dummy subclasses and it worked and the model runs much faster!! &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;To My Questions (finally!!):&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;Maybe I am not doing the image-to-numpy conversion efficiently? Please advice how I could construct the arrays in a more efficient approach!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;UPDATE:&lt;/strong&gt; I found a very similar &lt;a href=&quot;https://stackoverflow.com/questions/44078327/fastest-approach-to-read-thousands-of-images-into-one-big-numpy-array&quot;&gt;question in stackoverflow&lt;/a&gt; a year ago, yet answers do not seem to offer any better alternative.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe it is what it is and I only need better hardware to do it locally!? &lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;How then Keras manage to do the conversion efficiently under the hood then?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;At the end (as we speak), I sampled only 20% of the images to have at least a model prototype up and running, although accuracy is not impressive!! &lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-06-13T06:53:32.480" LastActivityDate="2018-06-13T07:01:38.333" Title="Large Numpy.Array for Multi-label Image Classification (CelebA Dataset)" Tags="|machine-learning|image-classification|cnn|multilabel-classification|numpy|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32274" PostTypeId="2" ParentId="32272" CreationDate="2018-05-28T16:49:02.960" Score="2" Body="&lt;p&gt;Please accept my apology as it is not really a data science answer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A lot of spammers are using spoofed emails address as sender, so that field alone might not be very helpful as a dimension. The number of possible values also could make it impractical to be used (i.e., 1 hot-encoding) as a feature unless you preprocess it somehow. Sender domain, sender server reputation scores, etc. would probably be more helpful than the email address by itself.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://lifehacker.com/how-spammers-spoof-your-email-address-and-how-to-prote-1579478914&quot; rel=&quot;nofollow noreferrer&quot;&gt;How spammers spoof your email&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-05-28T16:49:02.960" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32275" PostTypeId="2" ParentId="28317" CreationDate="2018-05-28T16:56:59.903" Score="0" Body="&lt;p&gt;One thing to also consider is the effect of false positive/negative. Without that info, it will be very difficult to help you to use the right metric.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, a model for approving car loans vs. a model for screening cancer would probably need to be assessed quite differently.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;a href=&quot;https://en.wikipedia.org/wiki/Classification_rule&quot; rel=&quot;nofollow noreferrer&quot;&gt;Wikipedia article&lt;/a&gt; is a great starting point.&lt;/p&gt;&#xA;" OwnerUserId="52719" LastEditorUserId="29575" LastEditDate="2018-05-28T17:22:28.393" LastActivityDate="2018-05-28T17:22:28.393" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32276" PostTypeId="2" ParentId="32264" CreationDate="2018-05-28T17:06:06.137" Score="60" Body="&lt;p&gt;Both cross validation and bootstrapping are &lt;em&gt;resampling&lt;/em&gt; methods. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;bootstrap resamples with replacement (and usually produces new &quot;surrogate&quot; data sets with the same number of cases as the original data set). Due to the drawing with replacement, a bootstrapped data set may contain multiple instances of the same original cases, and may completely omit other original cases.&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;cross validation resamples without replacement and thus produces surrogate data sets that are smaller than the original. These data sets are produced in a systematic way so that after a pre-specified number &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt; of surrogate data sets, each of the &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; original cases has been left out exactly once. This is called k-fold cross validation or leave-&lt;em&gt;x&lt;/em&gt;-out cross validation with &lt;span class=&quot;math-container&quot;&gt;$x = \frac{n}{k}$&lt;/span&gt;, e.g. leave-one-out cross validation omits 1 case for each surrogate set, i.e. &lt;span class=&quot;math-container&quot;&gt;$k = n$&lt;/span&gt;.&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;As the name cross &lt;em&gt;validation&lt;/em&gt; suggests, its primary purpose is measuring (generalization) performance of a model. On contrast, bootstrapping is primarily used to establish empirical distribution functions for a widespread range of statistics (widespread as in ranging from, say, the variation of the mean to the variation of models in bagged ensemble models).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The leave-one-out analogue of the bootstrap procedure is called &lt;em&gt;jackknifing&lt;/em&gt; (and is actually older than bootstrapping).&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;The bootstrap analogue to cross validation estimates of generalization error is called &lt;em&gt;out-of-bootstrap&lt;/em&gt; estimate (because the test cases are those that were left out of the bootstrap resampled training set). &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[cross validation vs. out-of-bootstrap validation] However, I cannot see the main difference between them in terms of performance estimation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That intuition is correct: in practice there's often not much of a difference between iterated &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt;-fold cross validation and out-of-bootstrap. With a similar total number of evaluated surrogate models, total error [of the model prediction error measurement] has been found to be similar, although oob typically has more bias and less variance than the corresponding CV estimates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are a number of attempts to reduce &lt;a href=&quot;https://en.wikipedia.org/wiki/Out-of-bag_error&quot; rel=&quot;noreferrer&quot;&gt;oob bias&lt;/a&gt; (.632-bootstrap, .632+-bootstrap) but whether they will actually improve the situation depends on the situation at hand.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Literature:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.cs.iastate.edu/~jtian/cs573/Papers/Kohavi-IJCAI-95.pdf&quot; rel=&quot;noreferrer&quot;&gt;Kohavi, R.: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Mellish, C. S. (ed.) Artificial Intelligence Proceedings 14&lt;span class=&quot;math-container&quot;&gt;$^th$&lt;/span&gt; International Joint Conference, 20 -- 25. August 1995, Montréal, Québec, Canada, Morgan Kaufmann, USA, , 1137 - 1145 (1995).&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://dx.doi.org/10.1016/j.csda.2009.04.009&quot; rel=&quot;noreferrer&quot;&gt; Kim, J.-H. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , Computational Statistics &amp;amp; Data Analysis , 53, 3735 - 3745 (2009). DOI: 10.1016/j.csda.2009.04.009&lt;/a&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0169743905000687&quot; rel=&quot;noreferrer&quot;&gt;Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. &amp;amp; Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005). &lt;/a&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Yes, there are fewer combinations possible for CV than for bootstrapping. But the limit for CV is probably higher than you are aware of. &#xA;For a data set with  &lt;span class=&quot;math-container&quot;&gt;$n$&lt;/span&gt; cases and &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt;-fold cross validation, you have &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;CV  &lt;span class=&quot;math-container&quot;&gt;$\binom{n}{k}$&lt;/span&gt; combinations without replacement (for k &amp;lt; n that are far more than the &lt;span class=&quot;math-container&quot;&gt;$k$&lt;/span&gt; possibilities that are usually evaluated) vs. &lt;/li&gt;&#xA;&lt;li&gt;bootstrap/oob &lt;span class=&quot;math-container&quot;&gt;$\binom{2 n - 1}{n}$&lt;/span&gt; combinations with replacement (which are again far more than the, say, 100 or 1000 surrogate models that are typically evaluated)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="5268" LastEditorUserId="1393" LastEditDate="2020-05-20T23:49:33.117" LastActivityDate="2020-05-20T23:49:33.117" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32277" PostTypeId="2" ParentId="32232" CreationDate="2018-05-28T17:42:34.083" Score="0" Body="&lt;p&gt;Regarding Deep Learning (as sub-part of Machine Learning), the best resource is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://www.deeplearningbook.org/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.deeplearningbook.org/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It also has a very good recap of the mathematic background required to understand the theory.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-28T17:42:34.083" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32278" PostTypeId="2" ParentId="28344" CreationDate="2018-05-28T19:48:28.533" Score="0" Body="&lt;p&gt;I believe that &quot;Sequence Classification using LSTM RNN&quot; is the exact answer to your problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check out this tutorial, it's one of the best:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-28T19:48:28.533" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32279" PostTypeId="1" AcceptedAnswerId="32283" CreationDate="2018-05-28T20:14:53.867" Score="2" ViewCount="4507" Body="&lt;p&gt;I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false-&gt;0 and true-&gt;1 because 0 as an input will deactivate weights of a neuron. Is this correct?&lt;/p&gt;&#xA;" OwnerUserId="52426" LastEditorUserId="28175" LastEditDate="2018-05-28T21:49:34.187" LastActivityDate="2018-11-11T09:26:10.490" Title="How to normalize a boolean feature for neural nets?" Tags="|machine-learning|neural-network|deep-learning|feature-scaling|normalization|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32280" PostTypeId="1" AcceptedAnswerId="32304" CreationDate="2018-05-28T20:38:42.057" Score="3" ViewCount="10198" Body="&lt;p&gt;I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export to csv file so that i can do classification task on that data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data table&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LSycv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LSycv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52820" LastEditorUserId="52820" LastEditDate="2018-05-29T19:16:00.170" LastActivityDate="2019-03-28T13:09:27.293" Title="Time series feature extraction from raw sensor data for classification?" Tags="|time-series|feature-selection|feature-extraction|multiclass-classification|feature-engineering|" AnswerCount="4" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32281" PostTypeId="1" AcceptedAnswerId="32314" CreationDate="2018-05-28T21:04:43.353" Score="3" ViewCount="2396" Body="&lt;p&gt;I need help to create a plot using 3 different columns from a dataframe. my dataframe looks like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;index      CMPGN_NM  COST_SUM    SUMRY_DT&#xA;&#xA;2   GSA_SMB_SMB_Generic_BMM      8985  2018-05-17&#xA;&#xA;3   GSA_SMB_SMB_Generic_BMM      7456  2018-05-18&#xA;&#xA;4   GSA_SMB_SMB_Generic_BMM      5761  2018-05-19&#xA;&#xA;10  GSA_SMB_SMB_Generic_BMM      4251  2018-05-20&#xA;&#xA;5   GSA_SMB_SMB_Generic_BMM     10521  2018-05-21&#xA;&#xA;6   GSA_SMB_SMB_Generic_BMM     10216  2018-05-22&#xA;&#xA;7   GSA_SMB_SMB_Generic_BMM     11023  2018-05-23&#xA;&#xA;9   GSA_SMB_SMB_Generic_BMM     11242  2018-05-24&#xA;&#xA;8   GSA_SMB_SMB_Generic_BMM      8817  2018-05-25&#xA;&#xA;1   GSA_SMB_SMB_Generic_BMM      6937  2018-05-26&#xA;&#xA;0   GSA_SMB_SMB_Generic_BMM      4581  2018-05-27&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would like the output to look like  the graph as below&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4eIB2.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4eIB2.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52823" LastEditorUserId="29217" LastEditDate="2018-05-29T15:41:12.803" LastActivityDate="2018-05-30T00:46:51.080" Title="How can I plot line plots based on an input python dataframe?" Tags="|python|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32283" PostTypeId="2" ParentId="32279" CreationDate="2018-05-28T21:49:07.503" Score="1" Body="&lt;p&gt;Actually, it is not clear what you mean by deactivating but if it means the output of neuron would be zero, it is not correct due to having bias term, also known as intercept. Furthermore, we usually use normalisation for features which are of different scales. Your boolean values do not have a large range. You don't need to scale them. If I want to be more precise, you may need depending on the other features' range, because they may change slightly among different input patterns and vary less than let say 1e-5 for different samples, but most of the time, booleans are not needed to be scaled. &lt;/p&gt;&#xA;" OwnerUserId="28175" LastEditorUserId="28175" LastEditDate="2018-05-29T00:36:25.460" LastActivityDate="2018-05-29T00:36:25.460" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32284" PostTypeId="1" CreationDate="2018-05-28T22:15:29.990" Score="1" ViewCount="44" Body="&lt;p&gt;I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is a valuable input. Time measure is restricted to seconds. By random frequency I mean the time between the signals varies between 0 (delta not measurable in seconds but order is known) and an extreme of 1-2 hours of rough maximum. I struggle with the following questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;How to convert the frequency to an input feature for a neural net? My best idea is to measure the time elapsed since the last signal. This gives me a feature for each new signal in seconds. Approximate range is 0 through 7200 seconds. Is there any better way?&lt;/li&gt;&#xA;&lt;li&gt;Depending on the answer on point 1 how should I normalize the data? I can have anything between 0 seconds and 1-2 hours. Providing my assumption on point 1 is right then I can add: the median of the entire sliding window is sometimes well below 30 seconds, sometimes well over 300 seconds. Problems I have:&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;I think I should normalize by each of the sliding window separately.&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;If I apply Z-score exclusively on each sliding window scope then the overall magnitude is eliminated. It would be interesting to keep an indicator of the overall input signal frequency measured on the entire dataset not only on the given window. Shall I create a second feature for compensating this?&lt;/li&gt;&#xA;&lt;li&gt;If I apply a min-max approach on the given window then the small values will be too close to 0 due to having only a few very high values.&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;li&gt;If I apply a min-max or the Z-score on the entire dataset then the small values will be too close to 0 due to having only a few very high values.&#xA;Any ideas about how to overcome these problems?&lt;/li&gt;&#xA;&lt;/ol&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52426" LastActivityDate="2018-05-28T22:15:29.990" Title="How to pre-process frequency of a series of signals?" Tags="|neural-network|lstm|normalization|encoding|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32285" PostTypeId="1" CreationDate="2018-05-28T22:16:08.920" Score="1" ViewCount="602" Body="&lt;p&gt;I'm implementing a captcha solver in tensorflow (also using ipython/jupyter notebook). After adding more layers to my model it now prints this message in a red box which I don't understand. What does it mean? I'm thinking it has something to do with the loss since I'm using tf.reduce_sum in the loss term (softmax-cross-entropy) which is also mentioned in the message.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/home/user/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: RuntimeWarning: overflow encountered in exp&#xA;  This is separate from the ipykernel package so we can avoid doing imports until&#xA;/home/user/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: RuntimeWarning: invalid value encountered in true_divide&#xA;  This is separate from the ipykernel package so we can avoid doing imports until&#xA;/home/user/anaconda3/lib/python3.6/site-packages/numpy/core/_methods.py:32: RuntimeWarning: overflow encountered in reduce&#xA;  return umr_sum(a, axis, dtype, out, keepdims)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;What does this mean?&lt;/p&gt;&#xA;" OwnerUserId="52767" LastActivityDate="2024-03-22T05:07:41.043" Title="&quot;RuntimeWarning: overflow[...]&quot; in TensorFlow in juypter notebook" Tags="|neural-network|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32286" PostTypeId="2" ParentId="32280" CreationDate="2018-05-28T22:16:12.880" Score="0" Body="&lt;p&gt;I don't have enough reputation to leave a comment, but could you please provide some sample data so that we can help you better?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When you say mean, max, min, are you trying to aggregate multiple rows of data on a date column with these functions? Or, do you have a timespan/ datetime/ timestamp column that you want to use?&lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-05-28T22:16:12.880" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32287" PostTypeId="1" AcceptedAnswerId="32335" CreationDate="2018-05-28T22:40:27.153" Score="1" ViewCount="476" Body="&lt;p&gt;I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My problem is that I do not want to include frames with no sound (or only noise) in my dataset. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;One solution I believe is to use a model for speech recognition or something similar to do classification and see if a frame includes only silence or noise.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I am searching for a solution that will not rely on machine learning but mostly on signal processing techniques or some other pre-processing method.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, how could I exclude these frames with only silence or noise?&lt;/p&gt;&#xA;" OwnerUserId="32390" LastActivityDate="2018-05-29T18:36:11.463" Title="How to check if audio samples have only noise or are silent?" Tags="|preprocessing|audio-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32288" PostTypeId="1" CreationDate="2018-05-28T23:27:31.897" Score="0" ViewCount="583" Body="&lt;p&gt;I have been taking the FastAI course. And if anyone else has taken this course they know that they use their own python library called &lt;code&gt;fastai&lt;/code&gt; that is a wrapper for PyTorch functions. This is effective for having time to explain theory, but I am wondering how these same methods would be implemented directly from PyTorch. Has anyone seen any blog posts or anything where someone has rewritten some fastai lessons in base PyTorch commands? I found one here: &lt;a href=&quot;https://medium.com/@tomgrek/wtf-is-going-on-with-fast-ai-db59741b5da2&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/@tomgrek/wtf-is-going-on-with-fast-ai-db59741b5da2&lt;/a&gt; but that is all I could find, and I'm not even sure which lesson this converts.  Thanks!&lt;/p&gt;&#xA;" OwnerUserId="46886" LastActivityDate="2018-05-28T23:27:31.897" Title="FastAI to PyTorch conversion" Tags="|neural-network|pytorch|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32289" PostTypeId="1" CreationDate="2018-05-29T00:20:27.423" Score="5" ViewCount="1274" Body="&lt;p&gt;I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like &quot;build an image classifier that recognizes classes A, B, and C but if the input is not those classes, don't recognize it&quot;. This features in other questions here from time to time, notably &lt;a href=&quot;https://datascience.stackexchange.com/questions/30032/other-class-in-image-classification&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/27807/fine-tuning-a-cnn-for-recognizing-two-classes-but-also-being-able-to-tell-if-no&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://datascience.stackexchange.com/questions/310/one-class-discriminatory-classification-with-imbalanced-heterogenous-negative-b&quot;&gt;here&lt;/a&gt;; however, I have had difficulty finding a general solution to what seems to be a common problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The approaches I've seen so far seem to fall into roughly two categories:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Treat it as an imbalanced class problem. Usual tricks of dataset balancing, under/oversampling, and class weight all apply here. However, the &quot;other&quot; class tends to be broad and diverse, essentially making the CNN try to encode all the information for &quot;other&quot; in the network. While this works &quot;good enough&quot; for some problems, I have seen it perform poorly and generally it seems like a square peg in a round hole.&lt;/li&gt;&#xA;&lt;li&gt;One class classification/outlier detection based approaches. In particular, the use of siamese and triplet networks are in a similar line of thought by trying to separate image feature vectors more clearly. Most of the research I've seen tends to focus on image similarity search or verification rather than applying to a multi-class + negative class problem directly. However, the notion of a contrastative or triplet loss to separate a more tightly definable class from everything else seems like a much more sound approach. Perhaps something like training the feature vector and then using one class SVM or some such?&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Is there a general strategy for this situation like finetuning CNN's for normal classification? It seems as though some general solution that works much like finetuning plus a contrastative loss of some sort at the feature vector layer likely exists but I'm just not aware of it.&lt;/p&gt;&#xA;" OwnerUserId="52269" LastActivityDate="2018-07-30T17:02:16.400" Title="Multi Class + Negative Class Image Classification Strategies" Tags="|classification|image-classification|cnn|anomaly-detection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32290" PostTypeId="1" AcceptedAnswerId="32417" CreationDate="2018-05-29T01:50:28.743" Score="1" ViewCount="659" Body="&lt;p&gt;As a first introduction to machine learning and keras, I just finished reading Deep Learning with R by François Chollet with J.J. Allaire. I would like to extend the book's IMDB example of two-class classification to a multi-input version using functional API.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(keras)&#xA;&#xA;## Keras IMDB example&#xA;&#xA;#load data&#xA;imdb &amp;lt;- dataset_imdb(num_words = 10000)&#xA;train_data &amp;lt;- imdb$train$x&#xA;train_labels &amp;lt;- imdb$train$y&#xA;test_data &amp;lt;- imdb$test$x&#xA;test_labels &amp;lt;- imdb$test$y&#xA;&#xA;#Encoding the integer sequences into a binary matrix&#xA;vectorize_sequences &amp;lt;- function(sequences, dimension = 10000) {&#xA;results &amp;lt;- matrix(0, nrow = length(sequences), ncol = dimension)&#xA;for (i in 1:length(sequences))&#xA;results[i, sequences[[i]]] &amp;lt;- 1&#xA;results&#xA;}&#xA;&#xA;x_train &amp;lt;- vectorize_sequences(train_data)&#xA;x_test &amp;lt;- vectorize_sequences(test_data)&#xA;y_train &amp;lt;- as.numeric(train_labels)&#xA;y_test &amp;lt;- as.numeric(test_labels)&#xA;&#xA;#Defining the model&#xA;model &amp;lt;- keras_model_sequential() %&amp;gt;%&#xA;layer_dense(units = 16, activation = &quot;relu&quot;, input_shape = c(10000)) %&amp;gt;%&#xA;layer_dense(units = 16, activation = &quot;relu&quot;) %&amp;gt;%&#xA;layer_dense(units = 1, activation = &quot;sigmoid&quot;)&#xA;&#xA;#Compiling the model &#xA;model %&amp;gt;% compile(&#xA;optimizer = &quot;rmsprop&quot;,&#xA;loss = &quot;binary_crossentropy&quot;,&#xA;metrics = c(&quot;accuracy&quot;)&#xA;)&#xA;&#xA;#Training the model&#xA;history &amp;lt;- model %&amp;gt;% fit(&#xA;partial_x_train,&#xA;partial_y_train,&#xA;epochs = 20,&#xA;batch_size = 512,&#xA;validation_data = list(x_val, y_val)&#xA;)&#xA;&#xA;#Defining the model using functional API&#xA;model &amp;lt;- keras_model_sequential() %&amp;gt;%&#xA;layer_embedding(input_dim = 10000, output_dim = 8,&#xA;input_length = maxlen) %&amp;gt;%&#xA;layer_flatten() %&amp;gt;%&#xA;layer_dense(units = 1, activation = &quot;sigmoid&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, I would like to create a similar model where we predict class &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;B&lt;/code&gt; based on 1 continuous and 3 categorical input data. In this dummy example, &lt;code&gt;continuous1&lt;/code&gt;, &lt;code&gt;categorical1&lt;/code&gt; and &lt;code&gt;categorical2&lt;/code&gt; are 1D tensors while similarly to the IMDB example, &lt;code&gt;categorical3&lt;/code&gt; is a 2D tensor of shape &lt;code&gt;(samples, indices)&lt;/code&gt; with length &lt;code&gt;num_index=20&lt;/code&gt; and are one-hot encoded.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V9eIw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V9eIw.png&quot; alt=&quot;Two-class classification multi-input model&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;#Dummy data&#xA;dat &amp;lt;- data.frame(samples=c(rep(paste0(&quot;A_&quot;,1:9800)),rep(paste0(&quot;B_&quot;,9801:10000))))&#xA;dat$label &amp;lt;- c(rep(&quot;A&quot;,9800),rep(&quot;B&quot;,200))&#xA;dat$continuous1 &amp;lt;- c(rnorm(9800, 100, 45),rnorm(200, 0, 25)) #may    normalize to unit variance&#xA;dat$continuous1[dat$continuous1&amp;lt;0] &amp;lt;- 0&#xA;dat$categorical1 &amp;lt;- c(rep(1:100,98),rep(1:100,2))&#xA;dat$categorical2 &amp;lt;- c(rep(1:98,100),rep(99:100,100))&#xA;&#xA;pool_A &amp;lt;- factor(1:15,levels=1:20)&#xA;pool_B &amp;lt;- factor(16:20,levels=1:20)&#xA;dat_categorical3 &amp;lt;- vector(&quot;list&quot;,10000)&#xA;names(dat_categorical3) &amp;lt;- c(as.character(dat[dat$label==&quot;A&quot;,]$samples),as.character(dat[dat$label==&quot;B&quot;,]$samples))&#xA;for(i in 1:9800){&#xA;  dat_categorical3[[i]] &amp;lt;- (as.numeric(table(sample(pool_A, 20, replace=T))) &amp;gt; 0) + 0L&#xA;}&#xA;for(i in 9801:10000){&#xA;  dat_categorical3[[i]] &amp;lt;- (as.numeric(table(sample(pool_B, 20, replace=T))) &amp;gt; 0) + 0L&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As I've never built a Keras model before, I'm a bit confused on how to set up the layers representing the different data inputs using functional API. Any suggestions and feedback are highly appreciated, thanks!&lt;/p&gt;&#xA;" OwnerUserId="52832" LastEditorUserId="52832" LastEditDate="2018-05-29T14:57:40.613" LastActivityDate="2018-05-31T01:49:10.887" Title="Two-class classification model with multi-type input data" Tags="|neural-network|deep-learning|r|keras|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32291" PostTypeId="2" ParentId="25524" CreationDate="2018-05-29T02:09:25.933" Score="2" Body="&lt;p&gt;As the gensim tool cites the very famous paper by Mikolov - &quot;&lt;a href=&quot;https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Distributed Representations of Words and Phrases...&lt;/a&gt;&quot; using which it is implemented. In the paper if you look at the section &quot;4 Learning Phrases&quot; they give a nice explanation of how n-grams are calculated (Equation 6). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KXD7F.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KXD7F.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.&lt;/p&gt;&#xA;" OwnerUserId="41058" LastActivityDate="2018-05-29T02:09:25.933" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32292" PostTypeId="1" CreationDate="2018-05-29T02:33:27.627" Score="2" ViewCount="5111" Body="&lt;pre&gt;&lt;code&gt;import statsmodels.api as sm&#xA;&#xA;y_hat_avg = test.copy()&#xA;fit1 = sm.tsa.statespace.SARIMAX(train.Netsales, order=(2, 1, 4),seasonal_order=(0,1,1,7)).fit()&#xA;y_hat_avg['SARIMA'] = fit1.predict(start=&quot;2017-12-06&quot;, end=&quot;2018-04-06&quot;, dynamic=True)&#xA;plt.figure(figsize=(16,8))&#xA;plt.plot( train['Netsales'], label='Train')&#xA;plt.plot(test['Netsales'], label='Test')&#xA;plt.plot(y_hat_avg['SARIMA'], label='SARIMA')&#xA;plt.legend(loc='best')&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Getting this error when I use this code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:225: ValueWarning: A date index has been provided, but it has no associated frequency information and so will be ignored when e.g. forecasting.&#xA;  ' ignored when e.g. forecasting.', ValueWarning)&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals&#xA;  &quot;Check mle_retvals&quot;, ConvergenceWarning)&#xA;---------------------------------------------------------------------------&#xA;TypeError                                 Traceback (most recent call last)&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;TypeError: an integer is required&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)&#xA;   2524             try:&#xA;-&amp;gt; 2525                 return self._engine.get_loc(key)&#xA;   2526             except KeyError:&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()&#xA;&#xA;KeyError: '2017-12-06'&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;TypeError                                 Traceback (most recent call last)&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;TypeError: an integer is required&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)&#xA;   1434         try:&#xA;-&amp;gt; 1435             return Index.get_loc(self, key, method, tolerance)&#xA;   1436         except (KeyError, ValueError, TypeError):&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)&#xA;   2526             except KeyError:&#xA;-&amp;gt; 2527                 return self._engine.get_loc(self._maybe_cast_indexer(key))&#xA;   2528 &#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine._date_check_type()&#xA;&#xA;KeyError: '2017-12-06'&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;KeyError: 1512518400000000000&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)&#xA;   2524             try:&#xA;-&amp;gt; 2525                 return self._engine.get_loc(key)&#xA;   2526             except KeyError:&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;KeyError: Timestamp('2017-12-06 00:00:00')&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()&#xA;&#xA;KeyError: 1512518400000000000&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)&#xA;   1443                 stamp = Timestamp(key, tz=self.tz)&#xA;-&amp;gt; 1444                 return Index.get_loc(self, stamp, method, tolerance)&#xA;   1445             except KeyError:&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)&#xA;   2526             except KeyError:&#xA;-&amp;gt; 2527                 return self._engine.get_loc(self._maybe_cast_indexer(key))&#xA;   2528 &#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()&#xA;&#xA;KeyError: Timestamp('2017-12-06 00:00:00')&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_index_label_loc(self, key, base_index)&#xA;    415                 if not isinstance(key, (int, long, np.integer)):&#xA;--&amp;gt; 416                     loc = self.data.row_labels.get_loc(key)&#xA;    417                 else:&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)&#xA;   1445             except KeyError:&#xA;-&amp;gt; 1446                 raise KeyError(key)&#xA;   1447             except ValueError as e:&#xA;&#xA;KeyError: '2017-12-06'&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_prediction_index(self, start, end, index, silent)&#xA;    476         try:&#xA;--&amp;gt; 477             start, start_index, start_oos = self._get_index_label_loc(start)&#xA;    478         except KeyError:&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_index_label_loc(self, key, base_index)&#xA;    422             except:&#xA;--&amp;gt; 423                 raise e&#xA;    424         return loc, index, index_was_expanded&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_index_label_loc(self, key, base_index)&#xA;    411             loc, index, index_was_expanded = (&#xA;--&amp;gt; 412                 self._get_index_loc(key, base_index))&#xA;    413         except KeyError as e:&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_index_loc(self, key, base_index)&#xA;    363             except (IndexError, ValueError) as e:&#xA;--&amp;gt; 364                 raise KeyError(str(e))&#xA;    365             loc = key&#xA;&#xA;KeyError: &quot;invalid literal for int() with base 10: '2017-12-06'&quot;&#xA;&#xA;During handling of the above exception, another exception occurred:&#xA;&#xA;KeyError                                  Traceback (most recent call last)&#xA;&amp;lt;ipython-input-58-0249e7d3f9ee&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      3 y_hat_avg = test.copy()&#xA;      4 fit1 = sm.tsa.statespace.SARIMAX(train.Netsales, order=(2, 1, 4),seasonal_order=(0,1,1,7)).fit()&#xA;----&amp;gt; 5 y_hat_avg['SARIMA'] = fit1.predict(start=&quot;2017-12-06&quot;, end=&quot;2018-04-06&quot;, dynamic=True)&#xA;      6 plt.figure(figsize=(16,8))&#xA;      7 plt.plot( train['Netsales'], label='Train')&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/base/wrapper.py in wrapper(self, *args, **kwargs)&#xA;     93             obj = data.wrap_output(func(results, *args, **kwargs), how[0], how[1:])&#xA;     94         elif how:&#xA;---&amp;gt; 95             obj = data.wrap_output(func(results, *args, **kwargs), how)&#xA;     96         return obj&#xA;     97 &#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/statespace/mlemodel.py in predict(self, start, end, dynamic, **kwargs)&#xA;   2432         &quot;&quot;&quot;&#xA;   2433         # Perform the prediction&#xA;-&amp;gt; 2434         prediction_results = self.get_prediction(start, end, dynamic, **kwargs)&#xA;   2435         return prediction_results.predicted_mean&#xA;   2436 &#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/statespace/sarimax.py in get_prediction(self, start, end, dynamic, index, exog, **kwargs)&#xA;   1924         # Handle start, end, dynamic&#xA;   1925         _start, _end, _out_of_sample, prediction_index = (&#xA;-&amp;gt; 1926             self.model._get_prediction_index(start, end, index, silent=True))&#xA;   1927 &#xA;   1928         # Handle exogenous parameters&#xA;&#xA;/anaconda3/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py in _get_prediction_index(self, start, end, index, silent)&#xA;    477             start, start_index, start_oos = self._get_index_label_loc(start)&#xA;    478         except KeyError:&#xA;--&amp;gt; 479             raise KeyError('The `start` argument could not be matched to a'&#xA;    480                            ' location related to the index of the data.')&#xA;    481         if end is None:&#xA;&#xA;KeyError: 'The `start` argument could not be matched to a location related to the index of the data.'&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the data I am trying to train the model with&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;lt;bound method NDFrame.head of               SaleDate   Netsales&#xA;SaleDate                         &#xA;2012-12-01  01-12-2012      83.60&#xA;2012-12-04  04-12-2012   -3515.00&#xA;2012-12-05  05-12-2012   26103.25&#xA;.&#xA;.&#xA;.&#xA;.&#xA;.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Can someone please help on how to fix it, tried changing the start and end parameters but doesn't work.&lt;/p&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="44680" LastEditDate="2018-05-29T02:44:58.637" LastActivityDate="2018-11-04T05:41:24.563" Title="Error when using seasonal arima in python" Tags="|python|forecast|forecasting|" AnswerCount="0" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32293" PostTypeId="2" ParentId="32289" CreationDate="2018-05-29T02:45:56.780" Score="4" Body="&lt;p&gt;There is no general strategy to anything in machine learning. Moreover, research is scattered over multiple domains so it becomes harder to get your head around, &quot;the best strategy for a case&quot;. You can solve a single problem in multiple ways. Coming back to the multi class plus a negative class classification issue, this is one of &quot;many approaches&quot; you can follow:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Hierarchical classification:&lt;/strong&gt; This is still an open problem. As the name suggests you perform classification on multiple levels starting from a root to internal node to leaf nodes (similar to tree or DAG structure). For eg. Given a digits data set ( 0,1,2,3...9) mixed with some letters (a,b,c....z). On a higher level you need to first classify if its a digit or not i.e. meta-class and if its a digit then you perform lower level classification task to identify the right digit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;There are different variants depending on your knowledge of labels/ groups at different level of hierarchy. For eg. if you don't have a pre-defined meta-class then you might have to perform clustering of similar classes at the higher level before final classification. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you may want to tune the weights in your loss function to identify a particular class efficiently at any level of the classification.   &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;P.S. My knowledge is limited in neural nets context. Definitely there would be more strategies specific to them.&lt;/em&gt;&lt;/p&gt;&#xA;" OwnerUserId="51659" LastEditorUserId="51659" LastEditDate="2018-05-31T14:57:48.840" LastActivityDate="2018-05-31T14:57:48.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32294" PostTypeId="1" CreationDate="2018-05-29T04:20:20.667" Score="1" ViewCount="65" Body="&lt;p&gt;I have unstructured data consist of three excel sheet&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/LFpCl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LFpCl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4Acf5.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4Acf5.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/qFaUn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/qFaUn.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Text contents of 3 tables some how related to each other But there is no linear direct relationship between them e.g First row of table 1 is  not related to first row of table 2  and table 3 although they sharing same numeric and letter key but may be there is relationship between First row of table 1, second row of table 2  and first row of table 3.  so table 1 is related to both table 2 and table 3 via sharing common numeric key and letter key but not in orderly way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can consider Table 1 as questions table each row is individual question and Table 2 is Responses from one group of people and table 2 is response from other group of people. But in collected responses some people did not reply the answers orderly way . for example they Gave answer of first question then skip the answer of second, third and wrote answer of fourth one. all of data was collected on paper. so they just place bullet points in front of answers  without specify the the answer belong to what number of question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have to figure out some insight from data, so what kind of data analysis or approach should I use to analysis this unstructured data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am already using Power BI for creating &lt;a href=&quot;https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/tutorials/tutorial-power-bi-key-phrases&quot; rel=&quot;nofollow noreferrer&quot;&gt;word cloud&lt;/a&gt; to visualize the data. What kind of other machine learning approach I can use to figure some insights from this data&lt;/p&gt;&#xA;" OwnerUserId="52836" LastEditorUserId="52836" LastEditDate="2018-05-29T19:12:05.953" LastActivityDate="2018-05-29T19:12:05.953" Title="Data analysis of unstructured data that have non linear Key value position" Tags="|machine-learning|deep-learning|text-mining|azure-ml|" AnswerCount="0" CommentCount="0" ClosedDate="2018-05-30T21:17:20.680" ContentLicense="CC BY-SA 4.0" />
  <row Id="32295" PostTypeId="1" CreationDate="2018-05-29T04:27:21.363" Score="10" ViewCount="1634" Body="&lt;p&gt;In simple terms, what are the assumptions of Linear Regression?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I just want to know that when I can apply a linear regression model to our dataset.&lt;/p&gt;&#xA;" OwnerUserId="52780" LastEditorUserId="29575" LastEditDate="2018-05-29T11:25:36.117" LastActivityDate="2018-06-03T06:40:51.350" Title="Assumptions of linear regression" Tags="|linear-regression|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32296" PostTypeId="1" CreationDate="2018-05-29T05:37:59.623" Score="1" ViewCount="10" Body="&lt;p&gt;I am trying to make a system that can recognize poses that are attained by a humanoid robotic arm; like that of asimo. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am currently looking into the libraries that detect real human poses like openpose, as the visual similarity is of a high degree between the arms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I was wondering if I could retrain a model that already recognizes human arms.&lt;/p&gt;&#xA;" OwnerUserId="52840" LastActivityDate="2018-05-29T05:37:59.623" Title="recognising humanoid arm" Tags="|deep-learning|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32297" PostTypeId="2" ParentId="32295" CreationDate="2018-05-29T06:04:55.187" Score="14" Body="&lt;p&gt;There are three major assumptions (statistically strictly speaking):&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;There is a linear relationship between the dependent variables and the regressors (right figure below), meaning the model you are creating actually fits the data.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/VKgmc.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VKgmc.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;The errors or residuals of the data are normally distributed and independent from each other.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/s31RO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/s31RO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Homoscedasticity. This means the variance around the regression line is the same for all values of the predictor variable.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/bNokv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/bNokv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update 2:&lt;/strong&gt;: Multicollinearity is not an assumption, but it is rather a sanity check especially if interpretability of the model is important (thanks Ricardo Cruz for the comment). Multicollinearity occurs when the independent variables are not independent from each other. Multicollinearity between explanatory variables which can leads to less stable parameter fits (thanks KT. for pointing this out)). There are tests like correlation matrix (Pearson's Bivariate Correlation), Variance Inflation Factor that can be used to verify this.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastEditorUserId="44456" LastEditDate="2018-06-03T06:40:51.350" LastActivityDate="2018-06-03T06:40:51.350" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="32298" PostTypeId="1" AcceptedAnswerId="39983" CreationDate="2018-05-29T06:12:42.167" Score="5" ViewCount="4030" Body="&lt;pre&gt;&lt;code&gt;clf = xgboost(data.matrix(train2), label = train$MACROSOMIA,objective = &quot;binary:logistic&quot;, nrounds = 20)&#xA;pred = predict(clf, data.matrix(test[, -12]))&#xA;&#xA;xgb.plot.shap(data.matrix(test2), model = clf, plot_loess =TRUE,features= &quot;GESTAC&quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is my code, and I get a SHAP feature contribution to prediction dependencies on feature value. I just have a hard time understanding it. I wonder what is it? How do I interpret it? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here's &lt;a href=&quot;https://cran.r-project.org/web/packages/xgboost/xgboost.pdf&quot; rel=&quot;noreferrer&quot;&gt;a link&lt;/a&gt;! Could you help me understand what does this plot mean?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgb.plot.shap()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51668" LastActivityDate="2018-10-20T17:50:34.383" Title="What's SHAP contribution dependency plots from xgboost package in R?" Tags="|machine-learning|r|xgboost|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32299" PostTypeId="1" CreationDate="2018-05-29T06:51:13.437" Score="1" ViewCount="99" Body="&lt;p&gt;I have gone through &lt;a href=&quot;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Liu_DeepFashion_Powering_Robust_CVPR_2016_paper.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Deepfashion&lt;/a&gt; paper.I have query regarding Landmark pooling layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Image has 4,6 or 8 landmark points depending on cloth type. So how do you decide size of last FC layer in landmark branch? This size will depend on landmark points but we have variable landmark points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) I have eight landmark points for an image and feature maps of conv4. I have to do max pooling around landmark l to get local feature map, similar to Fast-RCNN. But which point on feature maps of conv4 corresponds to landmark point l (x,y). Also what is the kernel size for max pooling?&lt;/p&gt;&#xA;" OwnerUserId="37760" LastActivityDate="2018-05-29T06:51:13.437" Title="Query about Landmark pooling layer in Fashionnet paper" Tags="|deep-learning|cnn|computer-vision|faster-rcnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32300" PostTypeId="1" CreationDate="2018-05-29T07:34:37.470" Score="2" ViewCount="562" Body="&lt;p&gt;Given a pair of &lt;code&gt;Variable&lt;/code&gt; &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;j&lt;/code&gt; and a &lt;code&gt;Tensor&lt;/code&gt; &lt;code&gt;A&lt;/code&gt;, how to get a new &lt;code&gt;Tensor&lt;/code&gt; which is &lt;code&gt;A&lt;/code&gt; with &lt;code&gt;i&lt;/code&gt;th element and &lt;code&gt;j&lt;/code&gt;th element swapped? How about &lt;code&gt;N&lt;/code&gt; pairs of &lt;code&gt;i&lt;/code&gt; and &lt;code&gt;j&lt;/code&gt; and we need to swap them all? How to do that in batch training?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found swapping one pair is easy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;if i &amp;lt; j:&#xA;    return tf.concat((A[:i], A[j], A[i+1:j], A[i], A[j+1:]), 0)&#xA;else:&#xA;    return tf.concat((A[:j], A[i], A[j+1:i], A[j], A[i+1:]), 0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Swapping multiple pairs would require explicit loop / reducing:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for i, j in ijpairs:&#xA;    A = swap_one_pair(A, i, j)&#xA;return A&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Swapping multiple samples in a batch would require explicit mapping:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;return tf.map_fn(swap_one_pair, (As, is, js))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I found if I separate the swapping into &quot;read&quot; and &quot;write&quot;, reading can be easily vectorized via &lt;code&gt;gather_nd&lt;/code&gt;:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;indexes = tf.stack((tf.tile(tf.range(batch_size)[:,None], (1,read_size)), all_i), 2)&#xA;index_results = tf.gather_nd(A, indexes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there a &quot;write&quot; version of &lt;code&gt;gather_nd&lt;/code&gt;? Or is there any more elegant way of implementing swapping utilizing vectorization/broadcasting? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;EDIT&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found a &quot;write&quot; version &lt;code&gt;scatter_nd_update&lt;/code&gt;. Unfortunately it only works for &lt;code&gt;Variable&lt;/code&gt;. I tried casting &lt;code&gt;Tensor&lt;/code&gt; to &lt;code&gt;Variable&lt;/code&gt; using &lt;code&gt;tf.Variable(myTensor)&lt;/code&gt; but don't work. I currently use:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def scatter_nd_update(ref, indices, updates):&#xA;    original_part = ref * (1 - tf.scatter_nd(indices, tf.ones(updates.shape), ref.shape))&#xA;    update_part   = tf.scatter_nd(index, update, ref.shape)&#xA;    return original_part + update_part&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This method uses a lot of meaningless computation. Any idea of better one?&lt;/p&gt;&#xA;" OwnerUserId="52842" LastEditorUserId="52842" LastEditDate="2018-06-01T04:50:50.437" LastActivityDate="2018-06-01T04:50:50.437" Title="Swap 2 number in tf.Tensor" Tags="|python|tensorflow|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32301" PostTypeId="1" CreationDate="2018-05-29T07:49:21.363" Score="1" ViewCount="985" Body="&lt;p&gt;I have a particular dataset consisting of 50k elements with 40 features each. I want to try to cluster the data as it is, without any dimensionality reduction. The main algorithm I am considering is the DBSCAN since is the more versatile and I can accept some poits to result as noise. However how can I judge if the clustering is &quot;significant&quot; since I can't plot the clusters in comparison to the data?&#xA;Tring to select the paremeters for the DBSCAN I've done a k-nn analysis, but the results worried me. For example, the following is the third nearest neighbor plot.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/kBScQ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kBScQ.png&quot; alt=&quot;3-NN&quot;&gt;&lt;/a&gt;&#xA;As you can see the distances (y-axis) are pretty much &quot;uniform&quot; along the x-axis (object). Does it mean that the data is somehow uniformly sparse and, in this conditions, clustering is useless?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S. I have tried to cluster the data anyway, in particulary using epsilon around 2-2.5 and for different values of min_sample. The silhouette score however results very low, about 0.11, and the noiseless fraction of points is about 80%.&lt;/p&gt;&#xA;" OwnerUserId="52846" LastActivityDate="2024-01-08T19:03:10.330" Title="Is my data good for (DBSCAN) clustering?" Tags="|clustering|dbscan|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="32302" PostTypeId="2" ParentId="31958" CreationDate="2018-05-29T08:09:37.230" Score="0" Body="&lt;p&gt;My suggestion would be a &lt;strong&gt;sequence classification&lt;/strong&gt; approach using Deep Learning, in particular &lt;strong&gt;Recurrent Neural Networks&lt;/strong&gt;, which are proven to be powerful in text processing. For fast prototyping, you should use the &lt;strong&gt;Keras API&lt;/strong&gt; (with Tensorflow backend).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In short:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Label&lt;/strong&gt; your dataset (2 classes: vacancies or not-vacancies) &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Embed&lt;/strong&gt; every word in your dataset into a vector of integers, using the &lt;code&gt;Embedding&lt;/code&gt; layer of Keras&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Find&lt;/strong&gt; the &quot;X&quot; most common words in your dataset, where &quot;X&quot; depends on the size of your dataset&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Constrain&lt;/strong&gt; your dataset to include only those words and remove the rest &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Pad or truncate&lt;/strong&gt; the sequences to have the same length using &lt;code&gt;keras.preprocessing.sequence.pad_sequences()&lt;/code&gt; &lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Train&lt;/strong&gt; an LSTM-RNN on your labeled dataset (simply by inserting a &lt;code&gt;LSTM&lt;/code&gt; layer in Keras) and then &lt;strong&gt;test&lt;/strong&gt; it&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Extra tips for improvement: use a &lt;code&gt;Dropout&lt;/code&gt; layer to decrease the variance of your model and also try improving it by combining it with a Convolutional Neural Network (CNN) to exploit spatial relations in the data. This can be &lt;strong&gt;easily&lt;/strong&gt; implemented if you insert a &lt;code&gt;Conv1D&lt;/code&gt; layer before the &lt;code&gt;LSTM&lt;/code&gt; layer.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An excellent reference that I have found online is &lt;a href=&quot;https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-29T08:09:37.230" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32303" PostTypeId="2" ParentId="32280" CreationDate="2018-05-29T08:24:03.697" Score="2" Body="&lt;p&gt;Perhaps you need to look at this self-contained blogpost on &lt;a href=&quot;http://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Machine Learning with Signal Processing Techniques&lt;/a&gt; on how to prepare your time series data and extract useful statistical estimate and feature for machine learning models. At the end an example is given for classification. I found it super useful and straightforward.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Somewhere in the middle of the post, this great method for the &lt;a href=&quot;https://github.com/demotu/BMC/blob/master/notebooks/DetectPeaks.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;Detection of peaks in data&lt;/a&gt; is introduced as well.&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-05-29T08:24:03.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32304" PostTypeId="2" ParentId="32280" CreationDate="2018-05-29T08:38:02.637" Score="2" Body="&lt;p&gt;For clarification: mean,max,min,std are &lt;strong&gt;not&lt;/strong&gt; &quot;time series features&quot;, they are data features in general.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assuming that you want to do it in python, you should take a look at &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas.DataFrame&lt;/a&gt; class. Once you initialize a Dataframe object with your tabular data, you can call its methods &lt;code&gt;DataFrame.min()&lt;/code&gt;, &lt;code&gt;DataFrame.max()&lt;/code&gt;, &lt;code&gt;DataFrame.mean()&lt;/code&gt;, &lt;code&gt;DataFrame.std()&lt;/code&gt; for your purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can insert all these calculated characteristics into a &lt;em&gt;new&lt;/em&gt; DataFrame and thereafter call &lt;code&gt;Dataframe.to_csv()&lt;/code&gt; to export them in a &lt;strong&gt;csv&lt;/strong&gt; file.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-29T08:38:02.637" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32306" PostTypeId="1" AcceptedAnswerId="32315" CreationDate="2018-05-29T09:33:48.877" Score="10" ViewCount="16133" Body="&lt;p&gt;I'm working on an age estimation project trying to classify a given face in a predefined age range. For that purpose I'm training a deep NN using the keras library. The accuracy for the training and the validation sets is shown in the graph below:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/GWbtf.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/GWbtf.png&quot; alt=&quot;picture&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As you can see the validation accuracy keeps rising with smaller steps than the training accuracy. Should i stop training at the epoch 280 in which the training and the validation accuracy have the same value or should i proceed the training process as long as the validation accuracy is rising, even thought the training accuracy value is also getting at overfitted values (eg. 93%).&lt;/p&gt;&#xA;" OwnerUserId="52852" LastEditorUserId="50727" LastEditDate="2018-06-05T10:44:11.513" LastActivityDate="2021-01-26T11:43:57.000" Title="In which epoch should i stop the training to avoid overfitting" Tags="|machine-learning|neural-network|deep-learning|accuracy|overfitting|" AnswerCount="5" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32309" PostTypeId="1" AcceptedAnswerId="32311" CreationDate="2018-05-29T10:03:08.917" Score="2" ViewCount="1059" Body="&lt;p&gt;&lt;strong&gt;Goal&lt;/strong&gt;&lt;br&gt;&#xA;I am trying to build a neural network that recognizes multiple label within a given image. I started with a database composed of 1800 images (each image is an array of shape (204,204,3). I trained my model and concluded that data used wasn't enough in order to build a good model ( with respect to chosen metric). So i decided to apply data augmentation technique in order to get more images. I managed to get 25396 images ( all of them are of shape (204,204,3)).&#xA;I stored all of them in arrays . I obtained (X,Y) where X are the training examples (is an array of shape (25396,204,204,3)) and Y are the labels ( an array of shape (25396,39) : the number 39 refers to the possible labels in a given image).&lt;br&gt;&#xA;&lt;strong&gt;Issues&lt;/strong&gt;&lt;br&gt;&#xA;My data (X,Y) weights approximately arround 26 giga bytes. I successfully managed to use them .  However, when i try to do manipulation (like permutations) I encounter memory Error in python.&#xA;&lt;strong&gt;Exemple&lt;/strong&gt;&lt;br&gt;&#xA;&lt;strong&gt;&lt;em&gt;1.&lt;/em&gt;&lt;/strong&gt; I started jupyter and successfully imported my data (X,Y)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x=np.load('x.npy')&#xA;y=np.load('y.npy')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output: x is an np.array of shape (25396,204,204,3) and y is an np.array of shape (25396,39).&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;&lt;em&gt;2.&lt;/em&gt;&lt;/strong&gt; I divide my dataSet in train and test by using sklearn built in function train_test_split&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X_train, X_valid, Y_train, Y_valid= `train_test_split(x_train,y_train_augmented,test_size=0.3, random_state=42)`&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;output&#xA;-------------testing size of different elements et toplogie: &#xA;-------------x size: (25396, 204, 204, 3)&#xA;-------------y size: (25396, 39) &#xA;-------------X_train size:  (17777, 204, 204, 3)&#xA;-------------X_valid size:  (7619, 204, 204, 3)&#xA;-------------Y_train size:  (17777, 39)&#xA;-------------Y_valid size:  (7619, 39)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;3.&lt;/strong&gt; I am creating a list composed of random batches extracted from (X,Y) and then iterate over the batches in order to complete the learning process for a given epoch :'this opperation is done in each epoch of the training part. &#xA;Here is the function used in order to create the list of random batches:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):&#xA;    &quot;&quot;&quot;&#xA;    Creates a list of random minibatches from (X, Y)&#xA;&#xA;    Arguments:&#xA;    X -- input data, of shape (input size, number of examples)&#xA;    Y -- true &quot;label&quot; vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)&#xA;    mini_batch_size -- size of the mini-batches, integer&#xA;&#xA;    Returns:&#xA;    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)&#xA;    &quot;&quot;&quot;&#xA;&#xA;&#xA;    np.random.seed(seed)            &#xA;    m = X.shape[0]                  &#xA;    mini_batches = []&#xA;    # Step 1: Shuffle (X, Y)&#xA;    permutation = list(np.random.permutation(m))&#xA;&#xA;&#xA;&#xA;    shuffled_X = X[permutation,:]&#xA;    shuffled_Y = Y[permutation,:]&#xA;&#xA;&#xA;    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.&#xA;    num_complete_minibatches = floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning&#xA;    for k in range(0, num_complete_minibatches):&#xA;&#xA;&#xA;        mini_batch_X = shuffled_X[k * mini_batch_size : (k + 1) * mini_batch_size, :]&#xA;        mini_batch_Y = shuffled_Y[k * mini_batch_size : (k + 1) * mini_batch_size, :]&#xA;        mini_batch = (mini_batch_X, mini_batch_Y)&#xA;        mini_batches.append(mini_batch)&#xA;        '''&#xA;        mini_batches.append((X[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :], Y[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :]))&#xA;        '''&#xA;    # Handling the end case (last mini-batch &amp;lt; mini_batch_size)&#xA;    if m % mini_batch_size != 0:&#xA;        ### START CODE HERE ### (approx. 2 lines)&#xA;        mini_batch_X = shuffled_X[ num_complete_minibatches * mini_batch_size:, :]&#xA;        mini_batch_Y = shuffled_Y[ num_complete_minibatches * mini_batch_size:, :]&#xA;        ### END CODE HERE ###&#xA;        mini_batch = (mini_batch_X, mini_batch_Y)&#xA;        mini_batches.append(mini_batch)&#xA;        '''&#xA;        mini_batches.append((X[permutation,:][ num_complete_minibatches * mini_batch_size:, :], Y[permutation,:][ num_complete_minibatches * mini_batch_size:, :]))&#xA;        '''&#xA;    shuffled_X=None&#xA;    shuffled_Y=None&#xA;    return mini_batches&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;4. I am creating a loop (of 4 iterations) and i am testing the random_mini_batch function in each iteration. At the end of each iteration I am assigning None values to the list of mini_batches in order to liberate memory and redo the random_mini_batch_function in the next iteration .So these line of codes works fine and I ve got no memory issues:&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minibatch_size=32&#xA;seed=2&#xA;for i in range(4):&#xA;    seed=seed+1&#xA;    minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)&#xA;&#xA;    minibatches=None &#xA;    minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)&#xA;    print(i)&#xA;minibatches_valid=None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;h2&gt;5. If I add iteration over the different batches! then I am getting a memory issue. In other words, if a run this code i get an error:&lt;/h2&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;minibatch_size=32&#xA;seed=2&#xA;for i in range(4):&#xA;    seed=seed+1&#xA;    minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)&#xA;    #added code: iteration over mini_batches&#xA;    for minibatch in minibatches:&#xA;                print('batch training number ')&#xA;    #end of added code        &#xA;    minibatches=None &#xA;    minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)&#xA;    print(i)&#xA;    minibatches_valid=None&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;MemoryError                               Traceback (most recent call last)&#xA;&amp;lt;ipython-input-13-9c1942cdf0bc&amp;gt; in &amp;lt;module&amp;gt;()&#xA;      3 for i in range(4):&#xA;      4     seed=seed+1&#xA;----&amp;gt; 5     minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)&#xA;      6     &#xA;      7     for minibatch in minibatches:&#xA;&#xA;&amp;lt;ipython-input-3-2056fee14def&amp;gt; in random_mini_batches(X, Y, mini_batch_size, seed)&#xA;&#xA;     23 &#xA;---&amp;gt; 24     shuffled_X = X[permutation,:]&#xA;     25     shuffled_Y = Y[permutation,:]&#xA;     26 &#xA;&#xA;MemoryError: &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does any one knows what's the issue with np.arrays ? And why does the simple fact of adding an loop (iterating over the list of batches) result in a memory error.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Questions&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1.Is it a good idea to load the whole dataset and then proceed to training? ( I need to create random batches in each epoch, so I don't see how to do so if the data is not preloaded ? You take random mini-batches from preloaded data, right?)&#xA;2. Are there any possible solutions guys?&lt;/p&gt;&#xA;" OwnerUserId="51208" LastEditorUserId="51208" LastEditDate="2018-05-29T10:09:03.543" LastActivityDate="2018-05-29T11:17:56.793" Title="Deep learning with Tensorflow: training with big data sets" Tags="|python|deep-learning|tensorflow|numpy|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32310" PostTypeId="1" AcceptedAnswerId="32360" CreationDate="2018-05-29T10:22:21.560" Score="1" ViewCount="668" Body="&lt;p&gt;I am trying to develop my own implementation of isolation forest algorithm. However I don't know how to deal with points that have the same value for a given feature. To better understand the problem, consider this example:&#xA;in my dataset I have the following data: (1, 2), (3,5), (3,4)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We may suppose that after one iteration, we get the following split:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      root&#xA;     /     \&#xA;    /       \&#xA;   /         \&#xA;(1,2)    (3,4), (3,5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, how am I supposed to deal with the right branch?&#xA;If (due to randomness) we decide to split according to the second feature, we should not have any problem because they are different (4 and 5). However what about if we decide to split according to the first feature (which is 3 in both cases)? Should I repeat the random feature selection until I can split the remaining data?&lt;/p&gt;&#xA;" OwnerUserId="50973" LastActivityDate="2018-05-30T06:52:10.070" Title="Isolation forest: how to deal with identical values?" Tags="|anomaly-detection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32311" PostTypeId="2" ParentId="32309" CreationDate="2018-05-29T11:17:56.793" Score="0" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Does any one knows what's the issue with np.arrays ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;shuffled_X = X[permutation,:]&lt;/code&gt; makes copies, so it will allocate new array each time you do a permutation and blow up your memory.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you don't have problems with storing whole dataset in memory you should be fine if you create batches just by using random indices, not &lt;em&gt;shuffling the entire data matrix&lt;/em&gt; (&lt;code&gt;np.random.choice&lt;/code&gt; is your friend).&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Is it a good idea to load the whole dataset and then proceed to&#xA;  training?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If your data fits into memory, then yes. You might want to try to learn what to do when that's not the case though - I personally find Keras - stuff from &lt;code&gt;keras.preprocessing.image&lt;/code&gt; useful for that (at least for loading images).&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-29T11:17:56.793" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32312" PostTypeId="1" AcceptedAnswerId="32316" CreationDate="2018-05-29T11:32:20.897" Score="1" ViewCount="2609" Body="&lt;p&gt;I have installed Orange 3 in Ubuntu 18.04 using Anaconda. It runs just fine, but the menus appear as blank.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/R5rCw.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/R5rCw.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I obtain the following error when I execute it:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;QGtkStyle could not resolve GTK. Make sure you have installed the proper libraries.&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been trying to sort it for days without success.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any idea of how to fix this?&lt;/p&gt;&#xA;" OwnerUserId="52860" LastActivityDate="2018-05-29T12:26:52.897" Title="QGtkStyle could not resolve GTK" Tags="|linux|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32314" PostTypeId="2" ParentId="32281" CreationDate="2018-05-29T12:17:06.287" Score="2" Body="&lt;p&gt;Here's a solution:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've created a sample dataframe with some arbitrary values. Here it is:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;import numpy as np&#xA;from datetime import datetime&#xA;&#xA;test = pd.read_csv('/home/sagar/Desktop/test.csv')&#xA;# Convert your date from 'str' to 'datetime' format&#xA;test['SUMRY_DT'] = test['SUMRY_DT'].map(lambda x: datetime.strptime(x, '%Y-%m-%d'))&#xA;# Set it as your dataframe index&#xA;test.set_index('SUMRY_DT', inplace=True)&#xA;test&#xA;            CMPGN_NM                    COST_SUM&#xA;SUMRY_DT        &#xA;2018-05-17  GSA_SMB_SMB_Generic_BMM     8985&#xA;2018-05-18  GSA_SMB_SMB_Generic_BMM     7456&#xA;2018-05-19  GSA_SMB_SMB_Generic_BMM     5761&#xA;2018-05-20  GSA_SMB_SMB_Generic_BMM     4251&#xA;2018-05-21  GSA_SMB_SMB_Generic_BMM     10521&#xA;2018-05-22  GSA_SMB_SMB_Generic_BMM     10216&#xA;2018-05-23  GSA_SMB_SMB_Generic_BMM     11023&#xA;2018-05-24  GSA_SMB_SMB_Spark           11242&#xA;2018-05-25  GSA_SMB_SMB_Generic_BMM     8817&#xA;2018-05-26  GSA_SMB_SMB_Generic_BMM     6937&#xA;2018-05-27  GSA_SMB_SMB_Generic_BMM     4581&#xA;2018-05-10  GSA_SMB_SMB_Spark           7089&#xA;2018-05-13  GSA_SMB_SMB_Spark           2121&#xA;2018-05-11  GSA_SMB_SMB_Spark           234&#xA;2018-05-12  GSA_SMB_SMB_Spark           11077&#xA;&#xA;# Plot your data&#xA;test.groupby('CMPGN_NM')['COST_SUM'].plot(legend=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1INvv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1INvv.png&quot; alt=&quot;This is how your chart will look with given data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the actual data, your chart would resemble the picture you have provided.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="51771" LastEditorUserId="51771" LastEditDate="2018-05-29T14:01:42.263" LastActivityDate="2018-05-29T14:01:42.263" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="32315" PostTypeId="2" ParentId="32306" CreationDate="2018-05-29T12:23:06.713" Score="20" Body="&lt;p&gt;As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep the model that performs the best in your validation set.&lt;/p&gt;&#xA;&lt;p&gt;Although it is right that your model overfits a little since epoch 280, it is not necessarily a bad thing provided that your validation accuracy is high. In general, most machine learning models will have higher training accuracy compared to validation accuracy, but this doesn't have to be bad.&lt;/p&gt;&#xA;&lt;p&gt;In a general case, you expect your accuracy to behave in the following way.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Z2BOz.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Z2BOz.png&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;In your case, you're before the early stopping epoch, so even if your training set accuracy is higher than your test set accuracy, it is not necessarily an issue.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2021-01-26T11:43:57.000" LastActivityDate="2021-01-26T11:43:57.000" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32316" PostTypeId="2" ParentId="32312" CreationDate="2018-05-29T12:26:52.897" Score="0" Body="&lt;p&gt;A quick fix has been published at &lt;a href=&quot;https://stackoverflow.com/a/50583925/3866828&quot;&gt;https://stackoverflow.com/a/50583925/3866828&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works adding&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;export QT_STYLE_OVERRIDE=gtk2&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;at the end of the &lt;code&gt;~/.bashrc&lt;/code&gt; file. Then it's a matter of executing&#xA;&lt;code&gt;source ~/.bashrc&lt;/code&gt; to see the content of the menus the next time that Orange 3 is loaded.&lt;/p&gt;&#xA;" OwnerUserId="52860" LastActivityDate="2018-05-29T12:26:52.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32317" PostTypeId="2" ParentId="32306" CreationDate="2018-05-29T12:28:09.447" Score="4" Body="&lt;p&gt;Keep training until your validation accuracy saturates (or starts dropping). Since the accuracy increases slowly, try to increase your learning rate parameter &lt;code&gt;eta&lt;/code&gt;to force the network to converge faster to the optimum weights. Be aware though, if you increase it too much though, it will become unstable. &lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-29T12:28:09.447" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32318" PostTypeId="2" ParentId="27124" CreationDate="2018-05-29T12:35:35.993" Score="1" Body="&lt;p&gt;You should definitely check an &lt;strong&gt;LSTM-RNN&lt;/strong&gt; or &lt;strong&gt;GRU-RNN&lt;/strong&gt; implementation; the second one is easier to understand and less computationally expensive. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A valuable example is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-29T12:35:35.993" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32320" PostTypeId="1" CreationDate="2018-05-29T13:15:39.057" Score="1" ViewCount="45" Body="&lt;p&gt;I have my dataset in the form RDF triples in various domains such as Movie, music etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data is in the form of RDF triples (Subject, Property,Object (all uris). A sample input in the movie domain is displayed below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/LaNRS.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/LaNRS.png&quot; alt=&quot;RDF sample dataset&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as my understanding, We need data in the form of a matrix for PCA and we need to classify data into different labels for Information gain. I need to automatically fetch relevant features(properties) from all the properties available.say 50 out  150 features.  It would be really great if someone could help me with more insight into pca and information gain and how to perform these machine learning algorithms based my raw rdf triples dataset. &lt;/p&gt;&#xA;" OwnerUserId="52867" LastActivityDate="2018-05-29T13:15:39.057" Title="How to preprocess raw data in the form RDF triples to perform Information Gain and Principal Component Analysis?" Tags="|machine-learning|data-cleaning|recommender-system|" AnswerCount="0" CommentCount="0" ClosedDate="2018-06-01T22:07:37.737" ContentLicense="CC BY-SA 4.0" />
  <row Id="32321" PostTypeId="1" CreationDate="2018-05-29T13:29:27.917" Score="3" ViewCount="615" Body="&lt;p&gt;I have the following data frame:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;      Date.POSIXct       Date      WeekDay DayCategory Hour Holidays   value&#xA;1 2018-05-01 00:00:00 2018-05-01      MA    MA-MI-JU    0        0      30&#xA;2 2018-05-01 01:00:00 2018-05-01      MA    MA-MI-JU    1        0      80&#xA;3 2018-05-01 02:00:00 2018-05-01      MA    MA-MI-JU    2        0      42&#xA;4 2018-05-01 03:00:00 2018-05-01      MA    MA-MI-JU    3        0      90&#xA;5 2018-05-01 04:00:00 2018-05-01      MA    MA-MI-JU    4        0      95&#xA;6 2018-05-01 05:00:00 2018-05-01      MA    MA-MI-JU    5        0       5&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;DayCategory groups days of the week in the following way: Mondays goes to &lt;code&gt;LU&lt;/code&gt; DayCategory. Tuesday, Wednesday and Thursdays go to &lt;code&gt;MA-MI-JU&lt;/code&gt; DayCategory.&#xA;Friday goes to &lt;code&gt;VI&lt;/code&gt;, Saturdays to &lt;code&gt;SA&lt;/code&gt; and Sundays to &lt;code&gt;DO&lt;/code&gt; Categories respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like to find the &lt;code&gt;value&lt;/code&gt; for the same &lt;code&gt;hour&lt;/code&gt; in the previous day (&lt;code&gt;Date&lt;/code&gt;) with the same &lt;code&gt;DayCategory&lt;/code&gt;, while &lt;code&gt;Holidays&lt;/code&gt; remains within the same group (e.g. if one instance has &lt;code&gt;holiday&lt;/code&gt; 0 but previous day with same &lt;code&gt;DayCategory&lt;/code&gt; has 1, we should lookv for the previous one, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As an intermediate step and to understand the process I would like to add a column &lt;code&gt;PreviousDaySameDayCategory&lt;/code&gt; with the &lt;code&gt;Date&lt;/code&gt; of the previous day that has the same &lt;code&gt;DayCategory&lt;/code&gt; that the corresponding row. Some times it will be just the same &lt;code&gt;date&lt;/code&gt; minus seven days (&quot;LU&quot;,&quot;VI&quot;,&quot;SA&quot;,&quot;DO&quot;) but other days it will be just one day.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Reproducible data:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(lubridate)&#xA;Date.POSIXct &amp;lt;- seq(as.POSIXct(&quot;2018-05-01&quot;), as.POSIXct(&quot;2018-05-31&quot;), &quot;hour&quot;)&#xA;mydf &amp;lt;- as.data.frame(Date.POSIXct)&#xA;mydf$Date &amp;lt;- as.Date(substr(as.character(mydf$Date.POSIXct),1,10))&#xA;mydf$WeekDay &amp;lt;- substr(toupper((weekdays(mydf$Date))),1,2)&#xA;mydf$DayCategory &amp;lt;-as.factor(ifelse(mydf$WeekDay == &quot;MA&quot; | mydf$WeekDay == &quot;MI&quot; | mydf$WeekDay == &quot;JU&quot;, &#xA;                                    &quot;MA-MI-JU&quot;, mydf$WeekDay))&#xA;mydf$Hour &amp;lt;- hour(mydf$Date.POSIXct)&#xA;mydf$Holidays &amp;lt;- c(rep(0, 24*7),rep(1, 24*7), rep(0, 24*16+1))&#xA;set.seed(123)&#xA;mydf$myvalue &amp;lt;- sample.int(101,size=nrow(mydf),replace=TRUE)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have manually started the first days and craeted a vector of how the solution should look like:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  a &amp;lt;- rep(NA, 24)&#xA;  b &amp;lt;- mydf$value[1:24]&#xA;  c &amp;lt;- mydf$value[25:48]&#xA;  d &amp;lt;- rep(NA, 24)&#xA;  e &amp;lt;- rep(NA,24)&#xA;  f &amp;lt;- rep(NA,24)&#xA;  g &amp;lt;- rep(NA,24)&#xA;  h &amp;lt;- rep(NA,24)&#xA;  i &amp;lt;- mydf$value[169:192]&#xA;  solution &amp;lt;- c(a,b,c,d,e,f,g,h,i)&#xA;  solution&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I would appreciate any hint in the thinking process to solve this kind of problems that I face with relative frequency.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I also published this question in Stack Over Flow, I guess this kind of question fits more in this community than in StackOverflow, is this correct? I would like to delete the one that less fit the commnunity. &lt;/p&gt;&#xA;" OwnerUserId="52844" LastEditorUserId="29169" LastEditDate="2021-01-21T20:22:25.590" LastActivityDate="2024-02-28T17:03:40.437" Title="Look for previous date in dataframe that has certain column category in R" Tags="|r|dataframe|data-table|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32323" PostTypeId="1" CreationDate="2018-05-29T14:24:26.580" Score="1" ViewCount="117" Body="&lt;p&gt;I'm a newbie of Deep Q Learning. After read some papers and tutorials on the web, I tried to train a DQN to control a game using TensorFlow.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The input is the screenshoot of the game. The output is an float array [x, y](where 0&amp;lt;=x&amp;lt;=1, 0&amp;lt;=y&amp;lt;=1), which contorls the tap location of the screen.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But I found that almost all the DQN I saw before only has a discrete, and they can use the tf.one_hot function to generate all the state table easily.&#xA;And I don't know how to deal with my float output and how to perform my backpropagation with this float output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can train the to DQN output such a float result and perform a backpropagation?&lt;/p&gt;&#xA;" OwnerUserId="52877" LastActivityDate="2018-05-29T14:24:26.580" Title="Can a DQN output a float result?" Tags="|dqn|" AnswerCount="0" CommentCount="1" ClosedDate="2018-06-02T02:03:18.737" ContentLicense="CC BY-SA 4.0" />
  <row Id="32324" PostTypeId="2" ParentId="16850" CreationDate="2018-05-29T14:25:12.447" Score="2" Body="&lt;p&gt;$x$ could be the position of the center of the peak in the Bell-shaped function for defining the weights. Note that the weights depend on the particular point $x$ at which we’re trying to evaluate $x$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Moreover,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if $|x^{(i)} − x|$ is small, then $w(i)$ is large (close to 1)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and&lt;/p&gt;&#xA;&#xA;&lt;p&gt;if $|x^{(i)} − x|$ is large, then $w(i)$ is small (close to 0).&lt;/p&gt;&#xA;" OwnerDisplayName="user52875" LastEditorUserId="29575" LastEditDate="2018-05-29T22:40:11.893" LastActivityDate="2018-05-29T22:40:11.893" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32325" PostTypeId="1" AcceptedAnswerId="32560" CreationDate="2018-05-29T14:30:41.613" Score="2" ViewCount="1094" Body="&lt;p&gt;I am trying to cluster a large set of documents of which I have a DOC2VEC representation. But I want to cluster them with more features, thus resulting in having both a vector (&lt;code&gt;numpy.asarray(doc2vec_for_document)&lt;/code&gt;) and single values as features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this possible? I would like to try clustering with both K-means and DBSCAN of the SKLearn module in Python.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My dataset would look like this:&lt;br&gt;&#xA;| Document | DOC2VEC | extra_ft1 | extra_ft2 | ...  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, what if I had multiple vectors instead of 1 vector and many values?&lt;/p&gt;&#xA;" OwnerUserId="52664" LastEditorUserId="924" LastEditDate="2018-06-03T07:40:41.737" LastActivityDate="2018-06-03T07:51:45.203" Title="Clustering with arrays / vectors as features?" Tags="|clustering|feature-selection|word2vec|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32327" PostTypeId="2" ParentId="28239" CreationDate="2018-05-29T15:39:46.760" Score="1" Body="&lt;p&gt;Sklearn comes with various ways to save and reuse models, see &lt;a href=&quot;http://scikit-learn.org/stable/modules/model_persistence.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Model Persistence&lt;/a&gt; for more information.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it is worth mentioning that not all unsupervised outlier detection models can easily &quot;make the prediction&quot; for the unseen data. predict() method may not be always consistent and visible, such as &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html#sklearn.neighbors.LocalOutlierFactor&quot; rel=&quot;nofollow noreferrer&quot;&gt;LOF&lt;/a&gt;. As you can see, there is no &quot;predict()&quot; method but &quot;fit()&quot; and &quot;fit_predict()&quot; only. Alternatively, you should use an internal method _decision_function() for LOF in Sklearn instead. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def _decision_function(self, X):&#xA;    &quot;&quot;&quot;Opposite of the Local Outlier Factor of X (as bigger is better,&#xA;    i.e. large values correspond to inliers).&#xA;&#xA;    The argument X is supposed to contain *new data*: if X contains a&#xA;    point from training, it consider the later in its own neighborhood.&#xA;    Also, the samples in X are not considered in the neighborhood of any&#xA;    point.&#xA;    The decision function on training data is available by considering the&#xA;    opposite of the negative_outlier_factor_ attribute.&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This inconsistency is due to the &lt;strong&gt;unsupervised nature&lt;/strong&gt; of most outlier detection methods. They are no difference for &quot;training&quot; and &quot;test&quot; since the ground truth is missing anyway. For some unsupervised outlier detection methods, saving the &quot;trained&quot; model does not help. It is case by case though. For efficient algorithms, it may be fine to combine the historical data with the new data and refit the model, given they come from the same distribution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thus, for different outlier detection methods in Sklearn, you should be careful with model persistence and APIs. If you are looking for more unified APIs to try various anomaly detection methods, I would recommend my python outlier detection toolbox &lt;a href=&quot;https://github.com/yzhao062/&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyOD&lt;/a&gt;; the documentation can be found here &lt;a href=&quot;https://pyod.readthedocs.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyOD documentation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52879" LastActivityDate="2018-05-29T15:39:46.760" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32328" PostTypeId="1" AcceptedAnswerId="32331" CreationDate="2018-05-29T15:48:56.150" Score="12" ViewCount="55253" Body="&lt;p&gt;I have a pandas dataframe &lt;code&gt;df&lt;/code&gt; that looks like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;name    value1     value2&#xA;A       123         1&#xA;B       345         5&#xA;C       712         4&#xA;B       768         2&#xA;A       318         9&#xA;C       178         6&#xA;A       321         3&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I want to convert this into a dictionary with &lt;code&gt;name&lt;/code&gt; as a key and list of dictionaries (value1 key and value2 value) for all values that are in &lt;code&gt;name&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, the output would look like this&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{&#xA; 'A': [{'123':1}, {'318':9}, {'321':3}],&#xA; 'B': [{'345':5}, {'768':2}],&#xA; 'C': [{'712':4}, {'178':6}]&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, far I have managed to get a dictionary with &lt;code&gt;name&lt;/code&gt; as key and list of only one of the values as a list by doing &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;df.set_index('name').transpose().to_dict(orient='list')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;How do I get my desired output? Is there a way to aggregate all the values for the same &lt;code&gt;name&lt;/code&gt; column and get them in the form I want?&lt;/p&gt;&#xA;" OwnerUserId="8841" LastActivityDate="2021-03-14T09:41:29.623" Title="Export pandas to dictionary by combining multiple row values" Tags="|python|pandas|data-wrangling|" AnswerCount="4" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32329" PostTypeId="2" ParentId="32328" CreationDate="2018-05-29T16:21:24.453" Score="4" Body="&lt;p&gt;The &lt;a href=&quot;http://pandas.pydata.org/pandas-docs/version/0.21/generated/pandas.DataFrame.to_dict.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;to_dict()&lt;/code&gt;&lt;/a&gt; method sets the column names as dictionary keys so you'll need to reshape your DataFrame slightly. Setting the 'ID' column as the index and then transposing the DataFrame is one way to achieve this.&lt;/p&gt;&#xA;&lt;p&gt;The same can be done with the following line:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; df.set_index('ID').T.to_dict('list')&#xA;{'p': [1, 3, 2], 'q': [4, 3, 2], 'r': [4, 0, 9]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Better to use the groupby,&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;df.groupby('name')[['value1','value2']].apply(lambda g: g.values.tolist()).to_dict()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2021-03-14T09:41:29.623" LastActivityDate="2021-03-14T09:41:29.623" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32330" PostTypeId="2" ParentId="6547" CreationDate="2018-05-29T16:27:28.223" Score="18" Body="&lt;p&gt;I recently developed a toolbox: &lt;strong&gt;Py&lt;/strong&gt;thon &lt;strong&gt;O&lt;/strong&gt;utlier &lt;strong&gt;D&lt;/strong&gt;etection toolbox (&lt;strong&gt;PyOD&lt;/strong&gt;). See &lt;a href=&quot;https://github.com/yzhao062/Pyod&quot; rel=&quot;noreferrer&quot;&gt;GitHub&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is featured for:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Unified APIs, detailed documentation, and interactive examples across&#xA;various algorithms.&lt;/li&gt;&#xA;&lt;li&gt;Advanced models, including Neural Networks/Deep&#xA;Learning and Outlier Ensembles.&lt;/li&gt;&#xA;&lt;li&gt;Optimized performance with JIT and&#xA;parallelization when possible, using numba and joblib.&#xA;Compatible&#xA;with both Python 2 &amp;amp; 3 (scikit-learn compatible as well).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Here are some important links:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://github.com/yzhao062/Pyod&quot; rel=&quot;noreferrer&quot;&gt;Github&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://pypi.org/project/pyod/&quot; rel=&quot;noreferrer&quot;&gt;PyPI&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://pyod.readthedocs.io&quot; rel=&quot;noreferrer&quot;&gt;Documentation&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https://mybinder.org/v2/gh/yzhao062/Pyod/master&quot; rel=&quot;noreferrer&quot;&gt;Interactive Jupyter Notebooks&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;If you use PyOD in a scientific publication, we would appreciate&#xA;citations to the following paper&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;@article{zhao2019pyod,&#xA;  title={PyOD: A Python Toolbox for Scalable Outlier Detection},&#xA;  author={Zhao, Yue and Nasrullah, Zain and Li, Zheng},&#xA;  journal={arXiv preprint arXiv:1901.01588},&#xA;  year={2019},&#xA;  url={https://arxiv.org/abs/1901.01588}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;It is currently under review at &lt;a href=&quot;http://www.jmlr.org/mloss/&quot; rel=&quot;noreferrer&quot;&gt;JMLR&lt;/a&gt;&#xA;(machine learning open-source software track). See &lt;a href=&quot;https://arxiv.org/abs/1901.01588&quot; rel=&quot;noreferrer&quot;&gt;preprint&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr /&gt;&#xA;&lt;h3&gt;Quick Introduction&lt;/h3&gt;&#xA;&lt;p&gt;PyOD toolkit consists of three major groups of functionalities: (i) outlier&#xA;detection algorithms; (ii) outlier ensemble frameworks and (iii) outlier&#xA;detection utility functions.&lt;/p&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Individual Detection Algorithms&lt;/strong&gt;&lt;/em&gt;:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;PCA&lt;/strong&gt;: Principal Component Analysis (the sum of weighted projected distances to the eigenvector hyperplanes)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MCD&lt;/strong&gt;: Minimum Covariance Determinant (use the mahalanobis distances as the outlier scores)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;OCSVM&lt;/strong&gt;: One-Class Support Vector Machines&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LOF&lt;/strong&gt;: Local Outlier Factor&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;CBLOF&lt;/strong&gt;: Clustering-Based Local Outlier Factor&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LOCI&lt;/strong&gt;: LOCI: Fast outlier detection using the local correlation integral&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;HBOS&lt;/strong&gt;: Histogram-based Outlier Score&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;kNN&lt;/strong&gt;: k Nearest Neighbors (use the distance to the kth nearest neighbor as the - **outlier score&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;AvgKNN&lt;/strong&gt;: Average kNN (use the average distance to k nearest neighbors as the outlier score)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MedKNN&lt;/strong&gt;: Median kNN (use the median distance to k nearest neighbors as the outlier score)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;ABOD&lt;/strong&gt;: Angle-Based Outlier Detection&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;FastABOD&lt;/strong&gt;: Fast Angle-Based Outlier Detection using approximation&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;SOS&lt;/strong&gt;: Stochastic Outlier Selection&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;IForest&lt;/strong&gt;: Isolation Forest&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature Bagging&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LSCP&lt;/strong&gt;: LSCP: Locally Selective Combination of Parallel Outlier Ensembles&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;XGBOD&lt;/strong&gt;: Extreme Boosting Based Outlier Detection (Supervised)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;AutoEncoder&lt;/strong&gt;: Fully connected AutoEncoder (use reconstruction error as the outlier score)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;SO_GAAL&lt;/strong&gt;: Single-Objective Generative Adversarial Active Learning&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MO_GAAL&lt;/strong&gt;: Multiple-Objective Generative Adversarial Active Learning&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Outlier Detector/Scores Combination Frameworks&lt;/strong&gt;&lt;/em&gt;:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Feature Bagging&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;LSCP&lt;/strong&gt;: LSCP: Locally Selective Combination of Parallel Outlier Ensembles&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Average&lt;/strong&gt;: Simple combination by averaging the scores&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weighted Average&lt;/strong&gt;: Simple combination by averaging the scores with detector weights&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Maximization&lt;/strong&gt;: Simple combination by taking the maximum scores&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;AOM&lt;/strong&gt;: Average of Maximum&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MOA&lt;/strong&gt;: Maximization of Average&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;em&gt;&lt;strong&gt;Utility Functions for Outlier Detection&lt;/strong&gt;&lt;/em&gt;:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;score_to_lable()&lt;/code&gt;: convert raw outlier scores to binary labels&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;precision_n_scores()&lt;/code&gt;: one of the popular evaluation metrics for outlier&#xA;mining (precision @ rank n)&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;generate_data()&lt;/code&gt;: generate pseudo data for outlier detection experiment&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;wpearsonr()&lt;/code&gt;: weighted pearson is useful in pseudo ground truth generation&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;&lt;strong&gt;Comparison of all implemented models&lt;/strong&gt; are made available below:&#xA;(&lt;a href=&quot;https://github.com/yzhao062/Pyod/blob/master/examples/ALL.png&quot; rel=&quot;noreferrer&quot;&gt;Figure&lt;/a&gt;,&#xA;&lt;a href=&quot;https://github.com/yzhao062/Pyod/blob/master/examples/compare_all_models.py&quot; rel=&quot;noreferrer&quot;&gt;Code&lt;/a&gt;,&#xA;&lt;a href=&quot;https://mybinder.org/v2/gh/yzhao062/Pyod/master&quot; rel=&quot;noreferrer&quot;&gt;Jupyter Notebooks&lt;/a&gt;):&lt;a href=&quot;https://i.stack.imgur.com/HNmMR.jpg&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/HNmMR.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&lt;p&gt;If you are interested, please check &lt;a href=&quot;https://github.com/yzhao062/Pyod&quot; rel=&quot;noreferrer&quot;&gt;Github&lt;/a&gt; for more information.&lt;/p&gt;&#xA;" OwnerUserId="52879" LastEditorUserId="98307" LastEditDate="2020-08-03T09:30:53.200" LastActivityDate="2020-08-03T09:30:53.200" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32331" PostTypeId="2" ParentId="32328" CreationDate="2018-05-29T16:29:00.447" Score="10" Body="&lt;p&gt;Does this do what you want it to?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from pandas import DataFrame&#xA;&#xA;df = DataFrame([['A', 123, 1], ['B', 345, 5], ['C', 712, 4], ['B', 768, 2], ['A', 318, 9], ['C', 178, 6], ['A', 321, 3]], columns=['name', 'value1', 'value2'])&#xA;&#xA;d = {}&#xA;for i in df['name'].unique():&#xA;    d[i] = [{df['value1'][j]: df['value2'][j]} for j in df[df['name']==i].index]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This returns&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  Out[89]: &#xA;{'A': [{123: 1}, {318: 9}, {321: 3}],&#xA; 'B': [{345: 5}, {768: 2}],&#xA; 'C': [{712: 4}, {178: 6}]}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52109" LastActivityDate="2018-05-29T16:29:00.447" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32332" PostTypeId="1" CreationDate="2018-05-29T17:04:55.833" Score="2" ViewCount="219" Body="&lt;p&gt;I'm trying to develop a captcha solver using a simple fully-connected neural network in TensorFlow. All captchas have 5 digits/letters, each character can be a number 0-9 or a letter A-Z. They all look like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WRTtn.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WRTtn.png&quot; alt=&quot;captcha1&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/QaPCH.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/QaPCH.png&quot; alt=&quot;captcha2&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/mDZRU.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/mDZRU.png&quot; alt=&quot;captcha3&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/f5gyZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/f5gyZ.png&quot; alt=&quot;captcha4&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://i.stack.imgur.com/tmWXY.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tmWXY.png&quot; alt=&quot;captcha5&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Each captcha has a label like this for example: &quot;AB4D1&quot;. I take these labels, split them ['A', 'B', '4', 'D', '1'], convert each of the characters to a number between 0 and 35 (26 letters and 10 digits) and convert each of those to one-hot representations [0, 0, 0, 1, 0,..]. Then I concatenate the one-hot arrays for all the letters in a captcha so I end up with a single 5*36-length array for each image. (This array has 5 ones and the rest zeros)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I'm wondering what loss function to use. I tried mean-squared-error but that didn't work at all for some reason. Now I'm trying to use cross-entropy (with softmax first) but as far as I understand it, I would need to apply it to each letter separately and then add up the loss for each letter since softmax requires exclusive categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried splitting the model output and labels:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;split_logits = tf.split(logits, num_or_size_splits=5, axis=1)&#xA;split_train_labels = tf.split(tf_train_labels, num_or_size_splits=5, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This works afaik. Now I just add the cross-entropy losses for each letter together.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;loss = &#xA;tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[0], logits=split_logits[0])) +&#xA;tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[1], logits=split_logits[1])) +&#xA;tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[2], logits=split_logits[2])) +&#xA;tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[3], logits=split_logits[3])) +&#xA;tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[4], logits=split_logits[4])) &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;But this is where I have a problem. As long as I only have the line for the first (0th) digit/letter in there it works really well (But obviously only learns recognizing the first letter). As soon as I add the term for another letter, or or just have a single other letter instead of the first one, the network immediately plateaus after training starts and never improves to above-random performance.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm not sure how to fix this. I'm thinking the problem is either in my loss function or in my network structure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is the problem that all the letters except the first one are varying a lot in their position due to the different font sizes/widths? I understand that convolutional networks would be much better for this but I really want to try it with fully-connected networks first.&lt;/p&gt;&#xA;" OwnerUserId="52767" LastActivityDate="2024-03-14T19:09:16.093" Title="How to use a cross entropy loss function for each letter/digit in a captcha?" Tags="|neural-network|tensorflow|loss-function|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32333" PostTypeId="2" ParentId="6547" CreationDate="2018-05-29T17:10:36.763" Score="1" Body="&lt;p&gt;Since you have multivariate time series, I would go for a LSTM-RNN implementation that models the dynamics of your system based on training data, which are usually semi-supervised (only normal class included). This means that you train your model to learn what is &quot;normal&quot;. During testing, you test both normal and anomalous conditions to see how well the model tells them apart. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An advantage of neural networks is that they &quot;learn&quot; the cross-correlations between input signals by themselves; you do not need to explore them manually. LSTM-RNNs, in particular, are an ideal choice when it comes to time series modelling simply because of their ability to keep memory of previous inputs, similar to a state space model in Control Theory (if you see the analogy).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In Python, it is almost trivial to implement an LSTM-RNN using &lt;a href=&quot;https://keras.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Keras&lt;/a&gt; API (on top of Tensorflow backend). This network learns to estimate the signal(s) of interest given an arbitrary number of inputs, which you thereafter compare with the actual measured value. If there is &quot;big&quot; deviation, you got an anomaly (given that the model is accurate enough)! &lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-29T17:10:36.763" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32334" PostTypeId="2" ParentId="32325" CreationDate="2018-05-29T17:39:44.617" Score="-1" Body="&lt;p&gt;Yes, it is possible. First, you should concatenate the outputs of doc2vec with your extra feature vectors to form an augmented feature vector. Then, you must scale the augmented feature vector, using min-max scaling or standardization (zero mean, unit variance). That is because the different features should be in the same range to have a good behaviour in clustering task. You can find these pre-processing modules in scikit-learn:&#xA;&lt;a href=&quot;http://scikit-learn.org/stable/modules/preprocessing.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://scikit-learn.org/stable/modules/preprocessing.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-29T17:39:44.617" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32335" PostTypeId="2" ParentId="32287" CreationDate="2018-05-29T18:36:11.463" Score="1" Body="&lt;p&gt;First you should apply speech enhancement algorithms to remove noise from speech. Then, you should use VAD (Voice Activity Detector) to remove silence from speech.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-29T18:36:11.463" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32336" PostTypeId="1" CreationDate="2018-05-29T18:57:00.823" Score="3" ViewCount="232" Body="&lt;p&gt;I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;for seller,seller_df in g:&#xA;&#xA;df=df.append({'Seller':seller,'AveragePrice':seller_df.Price.mean() }, ignore_index=True)&#xA;print(seller)&#xA;    df.sort_values(by='AveragePrice',ascending=False,inplace=True)&#xA;plt.figure(figsize=(15,30))&#xA;sns.barplot(data=df.dropna()[:100],y='Seller',x='AveragePrice')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The bar plot is only for the top 100 sellers and shows who has the max average selling price. &lt;a href=&quot;https://i.stack.imgur.com/Mdqov.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Mdqov.png&quot; alt=&quot;Plot&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I believe this method is inaccurate as I also should consider the amount of properties that he has sold in order to get the best seller. For e.g.: if A sold 20 houses with average 5 and B sold 200 with average 5, B should be the winner.So I have two questions:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;p&gt;How can I incorporate this in my code?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;p&gt;Can I optimise the code snippet?&lt;/p&gt;&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I am a newbie and any help is appreciated. Please refer to this link for my Kaggle notebook. It does not contain the code posted here as I am doing the analysis locally but you can have a look at the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/shinydhar/melbourne-housing-analysis&quot; rel=&quot;nofollow noreferrer&quot;&gt;Dataset&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="37997" LastEditorUserId="37997" LastEditDate="2018-05-30T07:58:34.560" LastActivityDate="2024-04-02T17:01:45.187" Title="Pandas: Calculating the best seller to incorporate number of houses sold" Tags="|python|pandas|data-analysis|" AnswerCount="1" CommentCount="12" ContentLicense="CC BY-SA 4.0" />
  <row Id="32338" PostTypeId="1" CreationDate="2018-05-29T19:10:09.897" Score="1" ViewCount="951" Body="&lt;p&gt;I have a tensor that is of the size &lt;code&gt;(19,19,5,80)&lt;/code&gt;. I want to print out all the elements in it but if I use &lt;code&gt;print(XXX.eval())&lt;/code&gt;, most of the elements will end up as ellipsis. How can I print all of them out? &lt;/p&gt;&#xA;" OwnerUserId="46454" LastEditorUserId="28175" LastEditDate="2018-05-29T19:45:32.890" LastActivityDate="2018-05-29T19:45:32.890" Title="TensorFlow: how to print out all elements in a tensor" Tags="|machine-learning|python|deep-learning|tensorflow|jupyter|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32339" PostTypeId="1" CreationDate="2018-05-29T19:12:14.273" Score="7" ViewCount="10189" Body="&lt;p&gt;I have a pandas dataframe with school names as one of the columns. However, there is quite a bit of misspelling in the school names, for example:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;'Abernethy Elem School', 'Abernethy Elementary School', 'BOISE/ELIOT ELEM SCHOOL', 'Boise/Eliot Elementary School', 'Boise-Eliot Elementary School' ...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am trying to clean up the names by doing this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;school_perf_report[&quot;SCHLNM&quot;] = school_perf_report[&quot;SCHLNM&quot;].str.lower().str.replace(r&quot;elementary&quot;,&quot;elem&quot;).str.replace(r&quot;/&quot;,&quot; &quot;).str.replace(&quot;-&quot;,&quot; &quot;).str.replace(r&quot;\s+&quot;,&quot; &quot;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Is there some cleaner way to perform the same operation? Basically&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Change &quot;elementary&quot; to &quot;elem&quot; &lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Remove / or - and replace them with space. &lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Remove multiple spaces &lt;br&gt;&lt;/li&gt;&#xA;&lt;li&gt;Lower case everything&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Thanks&lt;/p&gt;&#xA;" OwnerUserId="52250" LastActivityDate="2018-05-30T05:58:35.320" Title="Multiple search/replace on pandas series" Tags="|python|pandas|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32340" PostTypeId="1" CreationDate="2018-05-29T19:19:11.247" Score="2" ViewCount="54" Body="&lt;p&gt;I saw this &lt;a href=&quot;https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;tutorial on generating text using LSTM&lt;/a&gt;. In this tutorial the author trained the network by taking 100 previous characters as input and the next character as the output label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am interested to try some simple speech recognition using LSTM. I may use mfcc features of the audio signal as input data, but what's confusing me most is how to represent the output label. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The dataset I have is the VCTK corpus which contains sentence level audio recording and its transcription. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the tutorial, next character that comes after the input vector was used as output label. But for speech it's impractical to know which part of speech produced which character without transcribing the audio for every second. So, how would I represent the output labels for this problem?&lt;/p&gt;&#xA;" OwnerUserId="52889" LastEditorUserId="29575" LastEditDate="2018-05-29T22:36:27.010" LastActivityDate="2018-05-29T22:36:27.010" Title="Representing output label for character level speec recognition using RNN" Tags="|machine-learning|rnn|speech-to-text|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32341" PostTypeId="1" CreationDate="2018-05-29T19:39:32.610" Score="11" ViewCount="17931" Body="&lt;p&gt;I am trying to classify raw accelerometer data x,y,z to its corresponding label.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is the best architecture for best results?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or, does anyone have any suggestions on LSTM architectures built on keras with input and output nodes?&lt;/p&gt;&#xA;" OwnerUserId="52820" LastEditorUserId="29169" LastEditDate="2019-06-08T03:30:21.330" LastActivityDate="2021-02-03T09:08:28.937" Title="What is the best method for classification of time series data? Should I use LSTM or a different method?" Tags="|classification|keras|time-series|lstm|multiclass-classification|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32342" PostTypeId="2" ParentId="32338" CreationDate="2018-05-29T19:43:09.187" Score="1" Body="&lt;p&gt;Basically what you want to do is showing the full numpy array in jupyter notebook. Use the following code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy&#xA;numpy.set_printoptions(threshold=numpy.nan)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then run your code and see the entire activation map.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-29T19:43:09.187" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32344" PostTypeId="1" CreationDate="2018-05-29T20:03:16.970" Score="2" ViewCount="72" Body="&lt;p&gt;I have tried a few of the keyword extraction techniques and they work fine. But when I uploaded documents from scientific domains such as medical, genetics, healthcare, many medical or rather technical terms were missed. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;What would I have to do to have the algorithm include those as well, am I missing a dictionary of such terms etc?&lt;/p&gt;&#xA;" OwnerUserId="52890" LastEditorUserId="29575" LastEditDate="2018-05-29T22:45:49.933" LastActivityDate="2018-05-29T22:45:49.933" Title="Extract keywords from text misses some entities" Tags="|named-entity-recognition|" AnswerCount="0" CommentCount="0" ClosedDate="2018-06-02T00:37:35.210" ContentLicense="CC BY-SA 4.0" />
  <row Id="32345" PostTypeId="1" CreationDate="2018-05-29T20:07:26.047" Score="5" ViewCount="6093" Body="&lt;p&gt;Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional &lt;code&gt;&amp;lt;pad&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;unk&amp;gt;&lt;/code&gt; replacements for padding and unknown types. Of course, there are no pretrained vectors for them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Solutions I've come up with are&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Fill them with random values&lt;/li&gt;&#xA;&lt;li&gt;Fill them with zeros&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Which approach is better/wrong? What is the common practice?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note: I use pytorch+torchtext if it matters.&lt;/p&gt;&#xA;" OwnerUserId="32120" LastEditorUserId="32120" LastEditDate="2018-05-31T19:59:02.030" LastActivityDate="2018-05-31T19:59:02.030" Title="Initial embeddings for unknown, padding?" Tags="|deep-learning|nlp|lstm|word-embeddings|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32346" PostTypeId="1" CreationDate="2018-05-29T20:09:18.263" Score="1" ViewCount="328" Body="&lt;p&gt;I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, basically those of stream functions of the atmosphere, obtained as contour plots in MATLAB from data generated out of a PDE solver. So, it is quite difficult to understand the difference between the two labels by just looking at the respective images. The labels are generated through some statistical algorithm that operates on the intensity of the pixels or simply the value of the matrices that make the contour plot. While training the CNN, I see that with a learning rate of .001, the loss decreases gradually and monotonically at all time where it goes down to 0.6 in the first 200 epochs (not suddenly, quite gradually, the slope decreasing as the value goes down) and settles there for the next 500 epochs. However, the training accuracy fluctuates around 50% and my testing accuracy is 50% too. Moreover, I see while looking at the false +ve and -ves that the CNN has got one label right all the time and the other label wrong all the time. I have used dropout of 0.5 and L2 regularization. I am using TensorFlow for all the programming. There is no imbalance in the training set between the TRUE and FALSE labels. I have 4500 TRUE example and 4500 False examples. I have randomly shuffled the training set and normalized them with 255 since I'm using RGB images. Testing is done on 200 TRUE examples and 200 False examples. Kernel size 5x5 in both layers , first one having 32 and second one having 64 filters and FC layer has 1024 neurons. I have played around with these parameters with no real improvement. Any suggestions would be really appreciated.&lt;/p&gt;&#xA;" OwnerUserId="52891" LastActivityDate="2018-07-05T07:12:51.410" Title="Loss for CNN decreases and settles but training accuracy does not improve" Tags="|image-classification|cnn|loss-function|overfitting|regularization|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32347" PostTypeId="1" CreationDate="2018-05-29T21:00:57.370" Score="1" ViewCount="193" Body="&lt;p&gt;Context:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have documents with reviews of articles that have the following structure:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Introduction: a description of the review, dates and metadata that will be discarded. (avg~180 words, std~30 words)&lt;/li&gt;&#xA;&lt;li&gt;Loop with:&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Reference: the reference article to review. A single review can talk about multiple articles. (avg~8 words, std~2 words)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Subject: topic of the following comments. There can be more than one subject for the following comments. (avg~7 words, std~3 words)&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Comment: atomic comment about the article. There can be multiple comments per subjects. (avg~45 words, std~30 words)&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&lt;/li&gt;&#xA;&lt;li&gt;Final: a brief summary of the review and other info that will be discarded. (avg~165 words, std~40 words)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I've to identify orderly the different chunks of references, subjects and comments. I've already designed a pipeline, and have a dataset to test and train my models, I'm using BIO tags for chunking as output, but I can adapt the pipeline to use BMEWO (BILOU).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I'm searching for a similar problem dataset, to compare how my pipeline is performing against the best metrics accomplished in that problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've already searched in:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://archive.ics.uci.edu/ml/datasets.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://archive.ics.uci.edu/ml/datasets.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.kaggle.com/datasets&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.kaggle.com/datasets&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Until now the closest dataset I've found is the CoNLL for named entity recognition or grammar tagging(noun phrases, prepositional phrases, verb phrases).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But that problems seems very different from mine:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Small length per chunk in relation with my problem.&lt;/li&gt;&#xA;&lt;li&gt;The length of every chunk class are similar, In my case the size of the comments are very different than the subjects and references.&lt;/li&gt;&#xA;&lt;li&gt;The outside tag could be everywhere while in my problem only can be at the beginning or the end.&lt;/li&gt;&#xA;&lt;li&gt;These problems seems more focus on short grammatic structure instead of long sequence meaning.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;I'm searching with the following keywords: chunk, long-sequence, text. I'm not sure if the word chunk could be misleading.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where can I keep searching datasets? What keywords can I use to improve my search? There is a similar problem in which I could try to transform mine to test against?&lt;/p&gt;&#xA;" OwnerUserId="52885" LastEditorUserId="52885" LastEditDate="2018-05-30T14:12:18.183" LastActivityDate="2018-05-30T14:12:18.183" Title="Where can I find a dataset for long sequence text chunking?" Tags="|nlp|dataset|text-mining|text|" AnswerCount="0" CommentCount="2" ClosedDate="2018-06-02T00:36:42.790" ContentLicense="CC BY-SA 4.0" />
  <row Id="32348" PostTypeId="2" ParentId="32246" CreationDate="2018-05-29T22:00:37.763" Score="14" Body="&lt;p&gt;Ok, it's simple!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Just the &quot;Target network approach&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Select an item from Memory Bank&lt;/li&gt;&#xA;  &lt;li&gt;Using &lt;strong&gt;Target Network&lt;/strong&gt;, from &lt;span class=&quot;math-container&quot;&gt;$S_{t+1}$&lt;/span&gt; determine the &lt;strong&gt;index&lt;/strong&gt; of the best action &lt;span class=&quot;math-container&quot;&gt;$A_{t+1}$&lt;/span&gt; and its Q-value&lt;/li&gt;&#xA;  &lt;li&gt;Do corrections as usual&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The &quot;double DQN approach&quot;&quot;:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;ol&gt;&#xA;  &lt;li&gt;Select an item from Memory Bank&lt;/li&gt;&#xA;  &lt;li&gt;Using Online Network, from &lt;span class=&quot;math-container&quot;&gt;$S_{t+1}$&lt;/span&gt; determine the &lt;strong&gt;index&lt;/strong&gt; of the best action &lt;span class=&quot;math-container&quot;&gt;$A_{t+1}$&lt;/span&gt;. &lt;/li&gt;&#xA;  &lt;li&gt;Using &lt;strong&gt;Target Network&lt;/strong&gt;, from &lt;span class=&quot;math-container&quot;&gt;$S_{t+1}$&lt;/span&gt; get the Q-value of that action.&lt;/li&gt;&#xA;  &lt;li&gt;Do corrections as usual using that Q-value&lt;/li&gt;&#xA;  &lt;/ol&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;As seen from 3, the Online Network gives us the action that it thinks is highest.  However, we let the &lt;strong&gt;Target Network&lt;/strong&gt; have the final say on the score of such an action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Were we to let &lt;strong&gt;Target Network&lt;/strong&gt; select a best action, it could have very well selected some other index. &lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-01-01T15:37:53.897" LastActivityDate="2019-01-01T15:37:53.897" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32349" PostTypeId="2" ParentId="32321" CreationDate="2018-05-29T22:05:21.710" Score="0" Body="&lt;p&gt;If I understand you correctly, you want to retrieve the date of the previous same day category, same holiday class, of the same hour.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If it is the case, perhaps you can try:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Segment rows within the same hour, day category, and holiday group into the same cluster (or filter, etc.)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) For each cluster, Rank the Date (first day in the group is 0, second = 1, ... nth day = n - 1, etc.) from the lowest, etc.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) For each Rank r, assign the PreviousDaySameDayCategory by looking up the Date for row r - 1 for r - 1 &gt;= 0 within the same cluster&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is rather straightforward to achieve with dplyr. Please let me know if the following makes sense to you.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(For some reasons, I am not able to run your solution)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&#xA;library(dplyr)&#xA;rankedDf &amp;lt;- mydf %&gt;%&#xA;  group_by(DayCategory, Hour, Holidays) %&gt;%&#xA;  arrange(DayCategory, Hour, Holidays, Date) %&gt;%&#xA;  mutate(rowRank = order(Date), previousRowRank = order(Date) - 1) %&gt;%&#xA;  left_join(., ., by = c(&quot;previousRowRank&quot; = &quot;rowRank&quot;, &quot;DayCategory&quot;, &quot;Hour&quot;, &quot;Holidays&quot;)) %&gt;%&#xA;  select(&#xA;    Date.POSIXct = Date.POSIXct.x,&#xA;    Date = Date.x,&#xA;    WeekDay = WeekDay.x,&#xA;    DayCategory,&#xA;    Hour,&#xA;    Holidays,&#xA;    myvalue = myvalue.x,&#xA;    PreviousDaySameDayCategory = Date.y,&#xA;    PreviousValueSameDayCategory = myvalue.y&#xA;  )&#xA;print.data.frame(rankedDf, nrow = 500)&#xA;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Note: I also include DayCategory, etc. in the arrange so that it's easier to understand how it works; the code works just fine with only Date as the parameter.&lt;/p&gt;&#xA;" OwnerUserId="52719" LastEditorUserId="52719" LastEditDate="2018-05-29T23:23:01.677" LastActivityDate="2018-05-29T23:23:01.677" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32350" PostTypeId="1" CreationDate="2018-05-30T00:03:31.503" Score="1" ViewCount="413" Body="&lt;p&gt;I've been reading Google's DeepMind Atari &lt;a href=&quot;http://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf&quot; rel=&quot;nofollow noreferrer&quot; title=&quot;RL&quot;&gt;paper&lt;/a&gt; and I'm trying to understand how to implement &lt;code&gt;experience replay.&lt;/code&gt; My question is that whether we update the parameters $\theta$ of function $Q$ once for all the samples of the minibatch or we do that for each sample of the minibatch separately? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;According to the following code from this paper, it performs the gradient descent on loss term for the $j$-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Ow0HM.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Ow0HM.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52894" LastEditorUserId="1330" LastEditDate="2021-07-20T14:08:44.057" LastActivityDate="2021-07-20T14:08:44.057" Title="Implementing experience replay in reinforcement learning" Tags="|deep-learning|reinforcement-learning|implementation|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32351" PostTypeId="2" ParentId="32281" CreationDate="2018-05-30T00:46:51.080" Score="1" Body="&lt;p&gt;Thank you so much Sagar for your response with code. I am able to put together and got the result as I expected... Here is the code...         #working with 3 columns dataframe&lt;/p&gt;&#xA;&lt;p&gt;import matplotlib.pyplot as plt&#xA;import datetime&#xA;import pandas as pd&lt;/p&gt;&#xA;&lt;h1&gt;spark dataframe into pandas dataframe&lt;/h1&gt;&#xA;&lt;p&gt;tt=df.toPandas()&lt;/p&gt;&#xA;&lt;h1&gt;convert column as int&lt;/h1&gt;&#xA;&lt;p&gt;tt[&amp;quot;COST_SUM&amp;quot;] = tt[&amp;quot;COST_SUM&amp;quot;].astype(int)&lt;/p&gt;&#xA;&lt;h1&gt;date as index&lt;/h1&gt;&#xA;&lt;p&gt;tt[&amp;quot;SUMRY_DT&amp;quot;] = pd.to_datetime(tt.SUMRY_DT)&#xA;tt.set_index('SUMRY_DT', inplace=True)&lt;/p&gt;&#xA;&lt;h1&gt;style&lt;/h1&gt;&#xA;&lt;p&gt;plt.style.use('seaborn-darkgrid')&#xA;fig, ax = plt.subplots()&lt;/p&gt;&#xA;&lt;p&gt;tt.groupby('CMPGN_NM')['COST_SUM'].plot(title='Cost by Campaign',ax=ax, legend=True,marker = 'o')&lt;/p&gt;&#xA;&lt;p&gt;ax.legend(fontsize='small')&#xA;plt.xticks(rotation=45)&lt;/p&gt;&#xA;&lt;h1&gt;output location&lt;/h1&gt;&#xA;&lt;p&gt;plt.savefig('/tmp/kenshoo/GRAPH1.pdf')&lt;/p&gt;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/NQ1OY.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/NQ1OY.jpg&quot; alt=&quot;enter image description here&quot; /&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52823" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-05-30T00:46:51.080" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32352" PostTypeId="2" ParentId="14273" CreationDate="2018-05-30T02:40:26.683" Score="-2" Body="&lt;p&gt;change data type&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;d=as.matrix(d)&#xA;d[is.na(d)] &amp;lt;-&quot;None&quot;&#xA;d=as.data.frame(d)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52898" LastEditorUserId="52898" LastEditDate="2018-05-30T02:53:12.440" LastActivityDate="2018-05-30T02:53:12.440" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32353" PostTypeId="1" AcceptedAnswerId="32363" CreationDate="2018-05-30T04:09:03.803" Score="2" ViewCount="8866" Body="&lt;p&gt;This is my first time training a model in cnn and predicting results but I am getting same value for images I have input. Here is my code&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;import pandas as pd&#xA;import matplotlib.pyplot as plt&#xA;%matplotlib inline&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Activation, Flatten&#xA;from keras.optimizers import Adam&#xA;from keras.layers.normalization import BatchNormalization&#xA;from keras.utils import np_utils&#xA;from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D, &#xA;GlobalAveragePooling2D&#xA;from keras.layers.advanced_activations import LeakyReLU &#xA;from keras.preprocessing.image import ImageDataGenerator&#xA;from keras.models import Sequential&#xA;from keras.layers import Convolution2D&#xA;from keras.layers import MaxPooling2D&#xA;from keras.layers import Flatten&#xA;from keras.layers import Dense&#xA;classifier=Sequential()&#xA;(X_train, y_train), (X_test, y_test) = mnist.load_data()&#xA;X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)&#xA;X_test = X_test.reshape(X_test.shape[0], 28, 28, 1)&#xA;X_train = X_train.astype('float32')&#xA;X_test = X_test.astype('float32')&#xA;X_train/=255&#xA;X_test/=255&#xA;number_of_classes = 10&#xA;Y_train = np_utils.to_categorical(y_train, number_of_classes)&#xA;Y_test = np_utils.to_categorical(y_test, number_of_classes)&#xA;classifier.add(Convolution2D(32,3,3,input_shape= &#xA;(28,28,1),activation='relu'))&#xA;classifier.add(BatchNormalization(axis=-1))&#xA;classifier.add(MaxPooling2D(pool_size=(2,2)))&#xA;&#xA;classifier.add(Convolution2D(32,3,3,activation='relu'))&#xA;classifier.add(BatchNormalization(axis=-1))&#xA;classifier.add(MaxPooling2D(pool_size=(2,2)))&#xA;&#xA;classifier.add(Flatten())&#xA;&#xA;classifier.add(Dense(output_dim=256,activation='relu'))&#xA;classifier.add(BatchNormalization())&#xA;classifier.add(Dense(output_dim=10,activation='softmax'))&#xA;&#xA;classifier.compile(optimizer='adam',loss='categorical_crossentropy',metrics=            &#xA;['accuracy'])&#xA;gen = ImageDataGenerator(rotation_range=8, width_shift_range=0.08,         &#xA;shear_range=0.3, height_shift_range=0.08, zoom_range=0.08)&#xA;&#xA;test_gen = ImageDataGenerator()&#xA;train_generator = gen.flow(X_train, Y_train, batch_size=64)&#xA;test_generator = test_gen.flow(X_test, Y_test, batch_size=64)&#xA;classifier.fit_generator(train_generator, steps_per_epoch=60000, epochs=1, &#xA;                validation_data=test_generator, validation_steps=10000)&#xA;import cv2&#xA;image = cv2.imread(&quot;pitrain.png&quot;)&#xA;gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) # grayscale&#xA;ret,thresh = cv2.threshold(gray,150,255,cv2.THRESH_BINARY_INV) &#xA;#threshold&#xA;kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(3,3))&#xA;dilated = cv2.dilate(thresh,kernel,iterations = 13) # dilate&#xA;im2,contours, hierarchy =             &#xA;cv2.findContours(dilated,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE) &#xA;# get contours&#xA;# for each contour found, draw a rectangle around it on original image&#xA;for contour in contours:&#xA;&#xA;# get rectangle bounding contour&#xA;&#xA;[x,y,w,h] = cv2.boundingRect(contour)&#xA;# discard areas that are too large&#xA;&#xA;if h&amp;gt;300 and w&amp;gt;300:&#xA;&#xA;    continue&#xA;# discard areas that are too small&#xA;&#xA;if h&amp;lt;40 or w&amp;lt;40:&#xA;&#xA;    continue&#xA;# draw rectangle around contour on original image&#xA;&#xA;cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,255),2)&#xA;image.shape&#xA;image=image[:,:,:1]&#xA;newimg = cv2.resize(image,(28,28))&#xA;img.shape&#xA;img = np.reshape(newimg,[1,28,28,1])&#xA;cv2.imshow(&quot;screen&quot;,img)&#xA;classifier.predict(img)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The output I am getting is array of zeros with 1 at third position.&#xA;This is where I copied the contour part from &lt;a href=&quot;https://www.quora.com/How-do-I-extract-a-particular-object-from-images-using-OpenCV&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.quora.com/How-do-I-extract-a-particular-object-from-images-using-OpenCV&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Epoch is equal to 1 because I only wanted to test my model and still I got &lt;strong&gt;accuracy of above 99%&lt;/strong&gt;&lt;/p&gt;&#xA;" OwnerUserId="52902" LastActivityDate="2018-05-30T07:08:10.743" Title="Getting same result for all predictions in cnn" Tags="|python|image-classification|cnn|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="32355" PostTypeId="1" CreationDate="2018-05-30T05:23:58.680" Score="7" ViewCount="1917" Body="&lt;p&gt;In random forest algorithm, m (say) number of decision trees are generated using p ( $log_2 n$ +1, where n= number of features) randomly selected features. The label of any sample from test data is chosen by the number of votes by the result of these decision trees. But these decision trees result in different accuracy ( quite obvious). My question is, why should we give same priority to all the decision trees? If a decision tree has higher accuracy in predicting the labels of the test samples, shouldn't we give it more priority than those with lower prediction accuracy ?&lt;/p&gt;&#xA;" OwnerUserId="46671" LastActivityDate="2018-06-29T15:16:46.823" Title="In a random forest, are all decision trees given same priority?" Tags="|machine-learning|classification|random-forest|decision-trees|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32356" PostTypeId="2" ParentId="32339" CreationDate="2018-05-30T05:58:35.320" Score="2" Body="&lt;p&gt;Create a definition for string replacements:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def change_string(x):&#xA;    return x.replace('|', '_').replace(&quot;elementary&quot;,&quot;elem&quot;).replace('old_value', 'new_value)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then use map command to perform operation on entire column:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# You could create a new column to check whether the output is as expected&#xA;# If it is as expected, you could modify the original column&#xA;school_perf_report[&quot;SCHLNM_MODIFIED&quot;] = school_perf_report[&quot;SCHLNM&quot;].map(lambda x: change_string(x))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="51771" LastActivityDate="2018-05-30T05:58:35.320" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32358" PostTypeId="2" ParentId="32355" CreationDate="2018-05-30T06:07:34.407" Score="1" Body="&lt;p&gt;As you mentioned, each decision tree is trained on p (sometimes sqrt(p) random features). This ensures that each tree is “grown” (trained) differently so that the model 1. Does not overfit the training data (reduces variance) and 2. Generalizes better to new data (reduces bias). Therefore we don’t weigh the trees differently, as this would be similar to having all tress trained on the same features.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can change the voting threshold however from the standard 50% to anything you want (e.g 40%, 70%, 90%) which will change the precision and recall accuracies of the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: changing the voting threshold means changing the number of trees needed to make a classification. For example, in binary classification most standard random forests require 50% or more of the trees to vote for a class for that class to “win” (be predicted to be that class). But if you change this threshold, to say 70%, then 70% or more of the trees need to vote for the same class for that class to win.&lt;/p&gt;&#xA;" OwnerUserId="41929" LastEditorUserId="41929" LastEditDate="2018-05-30T06:25:23.677" LastActivityDate="2018-05-30T06:25:23.677" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32359" PostTypeId="2" ParentId="32306" CreationDate="2018-05-30T06:37:01.687" Score="5" Body="&lt;p&gt;&quot;Early Stopping&quot; is the concept which needs to be used here. &#xA;As mentioned in wikipedia about early stopping, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;In machine learning, early stopping is a form of regularization used to avoid overfitting when training a learner with an iterative method, such as gradient descent. Such methods update the learner so as to make it better fit the training data with each iteration. Up to a point, this improves the learner's performance on data outside of the training set. Past that point, however, improving the learner's fit to the training data comes at the expense of increased generalization error. Early stopping rules provide guidance as to how many iterations can be run before the learner begins to over-fit. Early stopping rules have been employed in many different machine learning methods, with varying amounts of theoretical foundation.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;At epoch &gt; 280 in your graph, validation accuracy becomes lesser than training accuracy and hence it becomes a case of overfitting. In order to avoid overfitting here, training further is not recommended. However you may choose to train the model beyond the epoch where training and validation accuracy matches if the resulting validation accuracy is sufficient for the particular problem you are working on. &lt;/p&gt;&#xA;" OwnerUserId="42498" LastActivityDate="2018-05-30T06:37:01.687" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32360" PostTypeId="2" ParentId="32310" CreationDate="2018-05-30T06:52:10.070" Score="0" Body="&lt;p&gt;No need to use all the features. That is the cool part of having an ensemble. If one tree who is using a small set of data with some set of features is unable to use the feature, some other tree will use it. If the point is a true outlier, the score that is calculated later will offer the due reflection.&lt;/p&gt;&#xA;" OwnerUserId="26394" LastActivityDate="2018-05-30T06:52:10.070" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32361" PostTypeId="1" AcceptedAnswerId="32364" CreationDate="2018-05-30T06:55:32.067" Score="2" ViewCount="1105" Body="&lt;p&gt;I've model with two output layers, age and gender prediction layers. I want to assign different  weight values for each output layer's loss. I've the following line of code to do so.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(loss=[losses.mean_squared_error,losses.categorical_crossentropy], optimizer='sgd',loss_weights=[1,10])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;My question is what is the effect of loss weights on performance of a model? How can I configure the loss weights so that the model can perform better on age prediction?&lt;/p&gt;&#xA;" OwnerUserId="52910" LastActivityDate="2018-05-30T07:13:38.447" Title="Model's loss weights" Tags="|machine-learning|keras|tensorflow|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32362" PostTypeId="2" ParentId="32260" CreationDate="2018-05-30T06:57:01.640" Score="1" Body="&lt;p&gt;&lt;em&gt;Clustering&lt;/em&gt; algorithms usually assume that you have &lt;em&gt;positions&lt;/em&gt; of objects, and want to find dense groups of observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If I understand you correctly, you have a 2d grid of sensor readings, and you want to segment them into regions. That is a slightly different problem. If you'd just put your sensor readings into a clustering, then the clusters will not be spatially coherent: clustering assumes there is no particular order to the points.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So you'll need to look into &lt;em&gt;segmentation&lt;/em&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A naive way would be to use (sensor.x, sensor.y, sensor.value) tuples. Including the sensor positions will cause the results to be somewhat spatially coherent. But that makes it very sensitive to scaling, and there is no &quot;correct&quot; way of scaling this. There is a trade-off between spatial coherence and measurement coherence.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-05-30T06:57:01.640" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32363" PostTypeId="2" ParentId="32353" CreationDate="2018-05-30T07:08:10.743" Score="2" Body="&lt;p&gt;Ok so I changed your model to simplify the training for example's sake. I will go through the example in detail below. When you feed a value from a different distribution to your model you are always at risk of misclassification. For example, your model was trained using handwritten numbers, thus it is not surprising for the model to missclassify numbers that are typewritten. Of course a deeper more complex model could do this. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Getting the data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The MNIST data is of size 28 by 28. We will convert these values to floating point values and then normalize them to the range 0 to 1. We will also determine that we have 10 output classes.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from __future__ import print_function&#xA;import keras&#xA;from keras.datasets import mnist&#xA;from keras.models import Sequential&#xA;from keras.layers import Dense, Dropout, Flatten&#xA;from keras.layers import Conv2D, MaxPooling2D&#xA;from keras.callbacks import ModelCheckpoint&#xA;from keras.models import model_from_json&#xA;from keras import backend as K&#xA;from keras.datasets import mnist&#xA;import numpy as np&#xA;&#xA;(x_train, y_train), (x_test, y_test) = mnist.load_data()&#xA;&#xA;x_train = x_train.astype('float32') / 255.&#xA;x_test = x_test.astype('float32') / 255.&#xA;&#xA;# The known number of output classes.&#xA;num_classes = 10&#xA;# Input image dimensions&#xA;img_rows, img_cols = 28, 28&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Next we need to reshape the data such that it matches with the Tensorflow framework which we will be using under the hood of Keras. This requires that the instances be the first dimension and it also requires a channels dimension as the last one. thus for the MNIST data we need to have $(6000, 28, 28, 1)$.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Channels go last for TensorFlow backend&#xA;x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)&#xA;x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)&#xA;input_shape = (img_rows, img_cols, 1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then we will bin the outputs into one-hot-encoded vectors&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Convert class vectors to binary class matrices. This uses 1 hot encoding.&#xA;y_train_binary = keras.utils.to_categorical(y_train, num_classes)&#xA;y_test_binary = keras.utils.to_categorical(y_test, num_classes)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's build our simple model, you can add more layers to this to make it more robust&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv2D(32, kernel_size=(3, 3),&#xA;                 activation='relu',&#xA;                 input_shape=input_shape))&#xA;model.add(Conv2D(64, (3, 3), activation='relu'))&#xA;model.add(MaxPooling2D(pool_size=(2, 2)))&#xA;model.add(Dropout(0.25))&#xA;model.add(Flatten())&#xA;model.add(Dense(128, activation='relu'))&#xA;model.add(Dropout(0.5))&#xA;model.add(Dense(num_classes, activation='softmax'))&#xA;&#xA;model.compile(loss=keras.losses.categorical_crossentropy,&#xA;              optimizer=keras.optimizers.Adadelta(),&#xA;              metrics=['accuracy'])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Let's train our model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;epochs = 4&#xA;batch_size = 128&#xA;# Fit the model weights.&#xA;model.fit(x_train_reshaped, y_train_binary,&#xA;          batch_size=batch_size,&#xA;          epochs=epochs,&#xA;          verbose=1,&#xA;          validation_data=(x_test_reshaped, y_test_binary))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This should yield about 98.75% on the validation set. This is pretty good and should be enough to test some new data using this set.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Validating results using MNIST&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Let's pass through a few random values from the MNIST dataset to see if they get rightfully classified by our model. Note how I set the dimensionality of the data. The instance dimension must exist even if we only have a single instance to predict as well as the channel dimension&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;ix = 0&#xA;&#xA;plt.imshow(X_test[ix,:,:,0], 'gray')&#xA;plt.show()&#xA;&#xA;temp = np.zeros((1,28,28,1))&#xA;temp[0,:,:,0] = X_test[ix,:,:,0]&#xA;&#xA;model.predict_classes(temp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xrTp0.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xrTp0.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([7], dtype=int64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Ok so that worked! Let's try another one &lt;code&gt;ix = 100&lt;/code&gt;!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WjZeD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WjZeD.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([6], dtype=int64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;That worked as well!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;Validating the model on novel data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;The image you sue has to match stylistically with the ones in the MNIST data set. Ideally they should be of the same distribution. The image you provided luckily gets correctly classified but it should be noted that this may not be the case for other numbers that are typewritten. If you want your algorithm to detect these then you should really train with typewritten numbers as well as handwritten numbers.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from skimage.transform import resize&#xA;&#xA;def rgb2gray(rgb):&#xA;    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])&#xA;&#xA;im = plt.imread(&quot;7 type.jpg&quot;)&#xA;im = rgb2gray(im)&#xA;im = resize(im, (28, 28))&#xA;&#xA;plt.imshow(im[:,:], 'gray')&#xA;plt.show()&#xA;&#xA;temp = np.zeros((1,28,28,1))&#xA;temp[0,:,:,0] = im&#xA;&#xA;model.predict_classes(temp)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/kILus.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/kILus.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([7], dtype=int64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h1&gt;Validating on hand drawn numbers&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;More realistically would be to use hand drawn numbers since that is what we used to train. I just drew this one using Paint.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/1Pnyl.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/1Pnyl.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array([8], dtype=int64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/t8gM3.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/t8gM3.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;array(&lt;a href=&quot;https://i.stack.imgur.com/WjZeD.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;2&lt;/a&gt;, dtype=int64)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T07:08:10.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32364" PostTypeId="2" ParentId="32361" CreationDate="2018-05-30T07:13:38.447" Score="1" Body="&lt;p&gt;This will affect how the backpropagation for each of these outputs will cause the intermediate nodes within the network to be updated. If the output nodes' error were equal then the gradient descent process of the intermediate nodes will favor the resulting error of each of these outputs equally. By more heavily weighting an output node it means that the gradient descent will favor a set of parameters that perform better on that output node.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T07:13:38.447" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32365" PostTypeId="2" ParentId="32341" CreationDate="2018-05-30T07:55:38.090" Score="13" Body="&lt;p&gt;I would not use the word &amp;quot;best&amp;quot; but LSTM-RNN are &lt;strong&gt;very powerful when dealing with timeseries&lt;/strong&gt;, simply because they can store information about previous values and exploit the time dependencies between the samples. That said, it is definitely worth going for it.&lt;/p&gt;&#xA;&lt;p&gt;It has been proven that their performance can be boosted significantly if they are &lt;strong&gt;combined with a Convolutional Neural Network (CNN)&lt;/strong&gt; that can learn the spatial structures in your data, which in this case is one-dimensional.&lt;/p&gt;&#xA;&lt;p&gt;Take a look at this &lt;a href=&quot;https://ieeexplore.ieee.org/document/8141873/&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;strong&gt;state-of-the-art&lt;/strong&gt; method&lt;/a&gt; that combines LSTM and CNN, published in 2019.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2020-11-07T19:12:12.300" LastActivityDate="2020-11-07T19:12:12.300" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32367" PostTypeId="1" CreationDate="2018-05-30T08:42:04.120" Score="1" ViewCount="124" Body="&lt;p&gt;Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any vector in R3 is a linear combination of e1, e2 and e3.&lt;/p&gt;&#xA;" OwnerUserId="52516" LastActivityDate="2018-05-30T08:48:06.423" Title="Are euclidian vectors and unit vectors same thing?" Tags="|linear-algebra|mathematics|" AnswerCount="2" CommentCount="1" ClosedDate="2018-06-02T01:49:25.587" ContentLicense="CC BY-SA 4.0" />
  <row Id="32368" PostTypeId="1" AcceptedAnswerId="32377" CreationDate="2018-05-30T08:42:33.683" Score="4" ViewCount="3462" Body="&lt;p&gt;Suppose I have these elements:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;a = [1, 6, 3, 4, 10, 32, 2, 54]&#xA;b = [20, 5, 14, 25, 18, 1]&#xA;c = [54, 3, 6, 12, 41, 1, 9]&#xA;d = [3, 4, 1]&#xA;e = [19, 20, 25, 5]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each identified by a list of ids (that actually represent a string but we'll use numbers for simplicity).  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I cluster these by finding those who have the most elements in common?&lt;br&gt;&#xA;For example:&lt;br&gt;&#xA;&lt;code&gt;a&lt;/code&gt; and &lt;code&gt;d&lt;/code&gt; have 3 elements in common&lt;br&gt;&#xA;&lt;code&gt;b&lt;/code&gt; and &lt;code&gt;e&lt;/code&gt; have 3 elements in common&lt;br&gt;&#xA;&lt;code&gt;a&lt;/code&gt; and &lt;code&gt;c&lt;/code&gt; have 4 elements in common  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So the clusters I would like to have ar (&lt;code&gt;a&lt;/code&gt;, &lt;code&gt;c&lt;/code&gt;, &lt;code&gt;d&lt;/code&gt;) and (&lt;code&gt;b&lt;/code&gt;, &lt;code&gt;e&lt;/code&gt;).&lt;br&gt;&#xA;I would like to obtain this in a Python script&lt;/p&gt;&#xA;" OwnerUserId="52664" LastEditorUserId="50727" LastEditDate="2018-05-30T10:31:41.580" LastActivityDate="2018-05-30T13:31:58.557" Title="Clustering by common elements in a list" Tags="|machine-learning|python|scikit-learn|clustering|feature-selection|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32369" PostTypeId="2" ParentId="32367" CreationDate="2018-05-30T08:47:51.467" Score="0" Body="&lt;p&gt;A unit vector in a normed vector space is a vector of length 1. A unit vector is an euclidian vector of length 1. Not every euclidean vetor has length 1.&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-30T08:47:51.467" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32371" PostTypeId="2" ParentId="32367" CreationDate="2018-05-30T08:48:06.423" Score="0" Body="&lt;p&gt;A Euclidean vector is any vector with a magnitude and direction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A unit vector, is a vector with a magnitude 1. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So it is fair to say that all unit vectors are Euclidean vectors. However, not all Euclidean vectors are unit vectors. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Given the statement above we can conclude that any Euclidean vector in $\mathbb{R}^3$ can be described by the unit vectors $e_1, e_2, e_3$. These span the space.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T08:48:06.423" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32372" PostTypeId="2" ParentId="32345" CreationDate="2018-05-30T09:07:32.840" Score="2" Body="&lt;p&gt;In my experience, what works well is :&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;For padding, fill a zero vector embedding (as pixel intensity in image data padding) is the only and best solution.&lt;/li&gt;&#xA;&lt;li&gt;For words that don't have a pre-trained embedding, you should try to train them: as you do, fill them with random values when initializing, but set them to trainable.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-30T09:07:32.840" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32373" PostTypeId="1" AcceptedAnswerId="32374" CreationDate="2018-05-30T09:08:53.913" Score="4" ViewCount="1682" Body="&lt;p&gt;Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: &quot;I would just calculate the Pearson correlation coefficient&quot;. That wasn't a good answer since as demonstrated in the figure below, dependency between two variables may not be linear and the Pearson coefficient can be close to $0$ (parabola, circle). I have seen people on Kaggle always starting with correlation matrix and discarding data that are not correlated. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is: is Pearson coefficient always a good measure of correlation between variables and we should always rely on it? Can you give practical example when some variable in your problem was important, but Pearson indicated there is hardly any correlation?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KkGE9.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KkGE9.jpg&quot; alt=&quot;Artificially generated data&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="51969" LastActivityDate="2018-05-30T09:26:52.230" Title="Is Pearson coefficient a good indicator of dependency between variables?" Tags="|predictive-modeling|statistics|feature-selection|correlation|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32374" PostTypeId="2" ParentId="32373" CreationDate="2018-05-30T09:21:43.040" Score="3" Body="&lt;p&gt;As you show, Pearson coefficient is clearly not a good measure of how variables depend on each other. A better measure is &lt;a href=&quot;https://en.wikipedia.org/wiki/Distance_correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;distance correlation&lt;/a&gt;. A nice property of distance correlation is that distance correlation 0 implies independence. An easy example that can happen in real life is that a variable is the square of another variable, as you show in your first picture. In this case, the Pearson correlation will be 0, but the distance correlation will be around 0.5. I think it is a mistake for Kagglers to choose variables in terms of Pearson correlations.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-30T09:21:43.040" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32375" PostTypeId="2" ParentId="32373" CreationDate="2018-05-30T09:25:06.197" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Can you give practical example when some variable in your problem was&#xA;  important, but Pearson indicated there is hardly any correlation?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Sure, when there is underlying &lt;strong&gt;nonlinear relation between two RVs&lt;/strong&gt;. Pearson correlation investigates &lt;strong&gt;linear relations&lt;/strong&gt;, therefore a low value does not exclude nonlinear relations.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-30T09:25:06.197" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32376" PostTypeId="2" ParentId="32373" CreationDate="2018-05-30T09:26:52.230" Score="1" Body="&lt;p&gt;If you are considering time series only, you may have an the option to run a linear regression model, considering one variable as dependent.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you can get a good R² and residual plot, with whatever transformation to make a linear model, then you may be able to assess there is a correlation between both.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-30T09:26:52.230" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32377" PostTypeId="2" ParentId="32368" CreationDate="2018-05-30T09:27:27.367" Score="6" Body="&lt;p&gt;In order to do clustering, you only have to define a distance measure. When you have defined a distance, you can apply K-means, hierarchical clustering or other algorithms. In your case, I would define the following distance function:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$ d(a, b) = 1 - \frac{\text{number of common elements of a and b}}{\text{maximum between lengths of a and b}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As $d(a, a) = 0$ and $d(a, b) &amp;gt; 0$ hold, this can be a distance. I don't think the triangle inequality holds, so this will not be a well-defined distance. However, you can still try to work with this function, it might give good results for your aim.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit: sklearn implementation&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In order to implement this using scikit-learn, the way to make it work is to use &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DBSCAN&lt;/a&gt;, set &lt;code&gt;metric='precomputed'&lt;/code&gt; and pass the distance matrix of our data as an argument. We have to use the precomputed distance matrix because scikit-learn does not allow (now) to use custom distances for clustering.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-05-30T13:31:58.557" LastActivityDate="2018-05-30T13:31:58.557" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32378" PostTypeId="1" CreationDate="2018-05-30T09:57:15.403" Score="10" ViewCount="4860" Body="&lt;p&gt;I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network will be confused by jumps at the points of concatenation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How to overcome?&lt;/p&gt;&#xA;" OwnerUserId="35613" LastActivityDate="2018-05-30T12:00:25.150" Title="How to train the same RNN over multiple series?" Tags="|lstm|rnn|" AnswerCount="2" CommentCount="7" ContentLicense="CC BY-SA 4.0" />
  <row Id="32379" PostTypeId="1" AcceptedAnswerId="32384" CreationDate="2018-05-30T10:00:13.327" Score="4" ViewCount="1076" Body="&lt;p&gt;Okay so i scrape data from the web on movie reviews. I also have already got my own 'dictionary' or 'lexicon' with words and their labels (1-poor, 2-ok, 3-good, 4-very good, 5-excellent). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;SO the input are paragraphs of movie reviews and i use Scikit Learn Naive Bayes to evaluate the sentiment of each comment , which would be a paragraph. I would like to know how it works under the hood. I ASSUME it uses a bag of words concept.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So , I am describing my assumption about how Scikit Learn Naive Bayes works for SA. How close am i to the reality ?  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am neither a data scientist nor a statistician, but this is a summary of what i THINK happens in  Naive Bayes algorithms for Sentiment Analysis, in Scikit Learn. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Training Phase&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lets assume i am using labels like 1,2,3,4,5 for each paragraph in the training set. Each paragraph is one unit for training as well as evaluation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;STEPS :-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Drop unwanted words like THE, BUT, AND and so on&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Read the first word say 'BEACH', pick it's label from it's parent paragraph, say '5'. So attach&#xA;5 to BEACH and put it back in the bag.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) So add up the number of times each word matched a given label.   Same word 'BEACH' could occur with multiple labels across the input paragraphs. So keep count of each label for the given word. So maybe ['BEACH' - label 1 - 10], ['BEACH' - label 2 - 8] and so on. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) After above step for all words, sum up the probability of getting a certain label for a given word. So &quot;BEACH&quot; may have 1/6 probability of label 1, 2/6&#xA;probability of 2, 1/6 probability of 3, 1/6 probability of 4 and 1/6&#xA;probability of 5. So these 5 probabilities for 'BEACH&quot; are put back in the bag of words. Remove the Word-label-count entities from Step 3 , which were used to obtain these probabilities. So now each word can have a maximum of 5 probabilities in the bag. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Evaluating Sentiment Analysis - New Data&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Now when we feed the real data, when it hits a word, it checks if it is&#xA;found in its bag. If not it omits it&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Say it finds BEACH in one comment. It checks highest probability&#xA;for 'BEACH' is 2/6 for label 2. So it picks label 2 for the word&#xA;BEACH. Similarly it picks corresponding labels with highest probability for all the words in bag matching the input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Now it sums up all the labels for all words for the paragraph whose SA needs to be computed. Let us say they add up to 20. It has to&#xA;convert this 20 to somewhere between 0 and 1. So it uses a logarithmic&#xA;conversion which converts 20 to somewhere between 0 and 1. Let us say&#xA;0.75.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;4) So 0.75 is the weight for this comment.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is this how it works ? Thanks for any inputs. &lt;/p&gt;&#xA;" OwnerUserId="31689" LastEditorUserId="31689" LastEditDate="2018-05-30T10:21:11.780" LastActivityDate="2018-05-30T10:56:37.440" Title="Naive Bayes for SA in Scikit Learn - how does it work" Tags="|python|scikit-learn|naive-bayes-classifier|sentiment-analysis|" AnswerCount="2" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="32380" PostTypeId="2" ParentId="28398" CreationDate="2018-05-30T10:11:24.663" Score="1" Body="&lt;p&gt;In general, features are engineered so as to retain optimum relevant information present in the dataset with succinct representation and then features are adapted so that an algorithm can accept it as input.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Feature engineering generally involves methods like binning, PCA, etc. Adapting those features to pass to algorithm is another step which is a very small part of the feature engineering step. Adapting example: for an image, we may have to reshape the image like below so that x[0] points to the image&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;x = image.img_to_array(img)&#xA;normalise(x) // feature engineering&#xA;x = x.reshape((1,) + x.shape) // Adapting for algorithm&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;With this understanding, if features are generated once for a dataset, it may be used for any relevant algorithm with corresponding adaption.&lt;/p&gt;&#xA;" OwnerUserId="42498" LastActivityDate="2018-05-30T10:11:24.663" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32381" PostTypeId="2" ParentId="32379" CreationDate="2018-05-30T10:29:57.790" Score="1" Body="&lt;p&gt;In Naive Bayes, you need two values : &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The prior of each class : $p(c_j)$ which is the proportion of each class in the training set.&lt;/li&gt;&#xA;&lt;li&gt;The conditional probability of each term $i$ from a document regarding to each class : $p(w_i|c_j) = \frac{count(w_i, c_j)}{\sum\limits_{w\in V}count(w, c_j)}$ which corresponds to the fraction of times the word $w_i$ appears among all words in documents of class $c_j$ ($V$ is the vocabulary).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;The classification rule being $\underset{c_j}{argmax} \; p(c_j)p(w|c_j)$ &#xA;with $p(w|c_j) = \prod\limits_{w_i}p(w_i|c_j)$ thanks to the independence assumption.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To prevent underflow errors, we often use the log : $log \;\;p(w|c_j) = \sum\limits_{w_i}log\;\;p(w_i|c_j)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, the final word is : to classify a document (or paragraph or whatever piece of text), you take each word $i$ from it, and for each class $j$, see the fraction of times it appears in it (in documents from class $j$), in the training data (which is a probability), and add up the log. It will give you $j$ probabilities (one for each class), and you take the class corresponding to the maximum.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What is not clear in your question is the relation between labels and sentiment analysis. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;See &lt;a href=&quot;https://sebastianraschka.com/Articles/2014_naive_bayes_1.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;this&lt;/a&gt; for a detailed explanation and examples.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-30T10:29:57.790" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32382" PostTypeId="1" CreationDate="2018-05-30T10:33:02.777" Score="1" ViewCount="280" Body="&lt;p&gt;I have built a simple tree predictor and then applied this to a new set of data. This was giving a very different probability distribution so I tested connecting the same training data and comparing probability scores across the two predictions widgets in the following flow.The pre-processing steps are the same for both models (Group Mosaic + select columns), but the Predictions widgets give different outputs. Any idea why this might be? I'm on version 3.13.0.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/En0Qk.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/En0Qk.png&quot; alt=&quot;Workflow&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52918" LastActivityDate="2018-05-30T10:33:02.777" Title="Orange: Predictions widget on same data gives different results" Tags="|prediction|orange|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32383" PostTypeId="2" ParentId="32254" CreationDate="2018-05-30T10:34:42.077" Score="1" Body="&lt;p&gt;When I comes to adding a new output, you need only to retrain the last layer, it's called transfer learning. And pretty good example of that is retraining the last layer of one of the inception modules of google (Originally trained on the ImageNet dataset) to classify flowers. You can find the example on &lt;a href=&quot;https://www.tensorflow.org/tutorials/image_retraining&quot; rel=&quot;nofollow noreferrer&quot;&gt;the website of Tensorflow&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When it comes to adding and input neuron, i never encountered anything of the sort to be honest. &lt;/p&gt;&#xA;" OwnerUserId="43459" LastActivityDate="2018-05-30T10:34:42.077" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32384" PostTypeId="2" ParentId="32379" CreationDate="2018-05-30T10:56:37.440" Score="4" Body="&lt;h1&gt;Collecting your data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your &lt;strong&gt;training set&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not have access to your data but I have a similar dummy example which uses comments and tries to classify them as being a review of a hotel or a car rental. I think this example would suit this purpose nicely.&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Pre-process the data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;There's many techniques we can use for this purpose. For example some popula rones are &lt;strong&gt;bag-of-words&lt;/strong&gt;, &lt;strong&gt;tf-idf&lt;/strong&gt;, and &lt;strong&gt;n-grams&lt;/strong&gt;. These techniques will take a comment and vectorize it into a set of features. The features which we will use are learned from the training set and then will be the same for new comments coming in. We will use bag-of-words for this example.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;Bag-of-words&lt;/h2&gt;&#xA;&#xA;&lt;p&gt;This method has a dictionary of words which we identify from our training set. We take all the unique words after stemming. We then count the number of times that each word shows up in the comment, that is the vector that represents a given instance. For example if our dictionary looks like&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;['car', 'book', 'test', 'wow']&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;And the comment was &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Wow, this car is AMAZING!!!!!&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;The resulting vector would be&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;[1, 0, 0, 1]&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;We will repeat this vectorization process for each comment. We will then be left with a matrix where the rows represents each comment and the columns represent the frequency at which these words appear in the comment. We will call this matrix $X$ it is our dataset. Each of these instances will also have a label we will call that vector $Y$. We want to map each row in $X$ to it's label $Y$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Assume this is the data you have for 40 instances in a training dataset for the cars/hotel comments. With it's corresponding label on the right. Label 0 is for cars and label 1 is for hotels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/y9EOK.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/y9EOK.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The columns of the matrix $X$ are&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;['car', 'passenger', 'seat', 'drive', 'power', 'highway', 'purchase',&#xA;  'hotel', 'room', 'night', 'staff', 'water', 'location']&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;h1&gt;Split the data&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;We will split the data in order to test the accuracy of our model&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.cross_validation import train_test_split&#xA;&#xA;X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/tGmMO.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/tGmMO.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Apply Naive Bayes&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.naive_bayes import GaussianNB&#xA;&#xA;clf = GaussianNB()&#xA;clf = clf.fit(X_train, y_train)&#xA;clf.score(X_test, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This gives a result of &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;1.0&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Perfect classification!&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;How does it work?&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;Gaussian Naive Bayes assumes that each feature is described by a Gaussian distribution (you can pick other distributions which may be better suited for your data, such as multinomial). It will then assume that each feature is entirely independent of one another. Thus we can say that the probability of &lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y=0|X) = P(X_0|Y=0)P(X_0) * ... * P(X_n|Y=0)P(X_n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$P(Y=1|X) = P(X_0|Y=1)P(X_0) * ... * P(X_n|Y=1)P(X_n)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;for your $n$ features. The the one that is the largest is going to be our label. If $P(Y=1|X)$ is larger then we will say that the comment was that of a car. So to do this we need to train the parameters of our probability distributions $P(X_i|Y)$. To calculate this term we need to know the parameters of our distribution, in our case we are using a Gaussian distribution, thus we need the mean and the variance $\mathcal{N}(\mu, \theta)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;We will build a dictionary of values for each label that contains the mean and variance for each feature. This is the training stage!&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;Homemade Naive Bayes&lt;/h1&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import csv&#xA;import random&#xA;import math&#xA;import pandas as pd&#xA;import numpy as np&#xA;&#xA;class NaiveBayes(object):&#xA;&#xA;    def __init__(self):&#xA;        self.groupClass = None&#xA;        self.stats = None&#xA;&#xA;    def calculateGaussian(self, x, mean, std):    &#xA;        exponent = np.exp(-1*(np.power(x-mean,2)/(2*np.power(std,2))))&#xA;        std[std==0] = 0.00001&#xA;        return (1 / (np.sqrt(2*math.pi) * std)) * exponent&#xA;&#xA;    def predict(self, x):&#xA;        probs = np.ones((len(x), len(self.stats)))&#xA;&#xA;        for ix, instance in enumerate(x):&#xA;            for label_ix, label in enumerate(self.stats):&#xA;                probs[ix, int(label)] = probs[ix, int(label)] * \&#xA;                                      np.prod(self.calculateGaussian(instance, self.stats[label][0], self.stats[label][1]))&#xA;        return np.argmax(probs, 1)&#xA;&#xA;    def score(self, x, y):&#xA;        pred = self.predict(x)&#xA;        return np.sum(1-np.abs(y - pred))/len(x)&#xA;&#xA;    def train(self, x, y):&#xA;        self.splitClasses(x, y)&#xA;        self.getStats()&#xA;        pass&#xA;&#xA;    def splitClasses(self, x, y):&#xA;        groupClass = {}&#xA;&#xA;        for instance, label in zip(x, y):&#xA;            if not label in groupClass:&#xA;                groupClass.update({label: [instance]})&#xA;            else:&#xA;                groupClass[label].append(instance)&#xA;        self.groupClass = groupClass&#xA;&#xA;    def getStats(self):&#xA;        stats = {}&#xA;&#xA;        for label in self.groupClass:&#xA;            mean = np.mean(np.asarray(self.groupClass[label]), 0)&#xA;            std = np.std(np.asarray(self.groupClass[label]), 0)&#xA;            stats.update({label: [mean, std]})&#xA;        self.stats = stats&#xA;&#xA;clf = NaiveBayes()&#xA;clf.train(X_train, y_train)&#xA;clf.score(X_test, y_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Once again we get an accuracy of 1.0 on the testing set!&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T10:56:37.440" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32385" PostTypeId="2" ParentId="32254" CreationDate="2018-05-30T11:04:11.107" Score="2" Body="&lt;p&gt;The answer to your question is &quot;Transfer Learning&quot;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Since the datasets &quot;cat and dog&quot; and &quot;mouse&quot; are quite similar as both are images. In DeepNet for recognising &quot;cats and dogs&quot;, any deep learning network in its early layers learn to identify low-level features like edges, etc. It learns high-level feature like eyes, ears, etc in few further layers and in very later layers of DeepNet, it starts to recognise the intended object. In DeepNet for recognising &quot;dogs&quot;, similar pattern will follow. On comparing these two, one may find that first few layers of DeepNet in both cases produce similar low-level features like edges, etc. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, first few layers of the DeepNet learned from &quot;cats&quot; can be used as base layers for the DeepNet for &quot;mouse&quot; detection. This technique is called transfer learning. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To apply transfer learning, dataset on which DeepNet has been trained on and dataset on which this technique can be applied must be similar. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This &lt;a href=&quot;https://www.coursera.org/learn/convolutional-neural-networks/lecture/4THzO/transfer-learning&quot; rel=&quot;nofollow noreferrer&quot;&gt;transfer learning video by Andrew Ng&lt;/a&gt; would also be helpful in understanding the concept &lt;/p&gt;&#xA;" OwnerUserId="42498" LastActivityDate="2018-05-30T11:04:11.107" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32386" PostTypeId="1" CreationDate="2018-05-30T11:18:55.337" Score="1" ViewCount="126" Body="&lt;p&gt;The question is similar to the one asked in &lt;a href=&quot;https://stackoverflow.com/questions/46173661/how-to-evaluate-the-importance-of-input-in-lstm&quot;&gt;SO&lt;/a&gt;, but I think it is more appropriate to be here. It remains anyway unanswered. &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Assuming &lt;strong&gt;multivariate time series&lt;/strong&gt;, how to evaluate the importance of&#xA;  all inputs of an &lt;strong&gt;already trained LSTM-RNN&lt;/strong&gt;?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Thus, which are the inputs that affect the output the most?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am aware of Pearson correlation (for linear relations) and Information Gain based on Entropy reduction to explore input/output relations in general, but this is not what I need. How can you infer from an &lt;strong&gt;already trained LSTM&lt;/strong&gt; network, which inputs affect the output the most? Can you tell by the weights? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any suggestion or literature would be welcome.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2018-05-30T11:40:35.520" LastActivityDate="2018-05-30T11:40:35.520" Title="Find most important inputs of LSTM-RNN for multivariate time series modeling" Tags="|neural-network|keras|lstm|rnn|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32387" PostTypeId="1" AcceptedAnswerId="32391" CreationDate="2018-05-30T11:38:00.393" Score="2" ViewCount="1216" Body="&lt;p&gt;I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given product description. Since the dataset was large I was not able to able TF-IDF on that data it throws MemoryError. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I found that Hashingvector was desirable when dealing with large data. But when Hashingvector was applied I found it produced data with 1048576 features. It took around 1 hour to train and SGD model and produced 78% accuracy.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import pandas as pd&#xA;from sklearn.feature_extraction.text import HashingVectorizer&#xA;from sklearn.model_selection import train_test_split&#xA;from sklearn.metrics import accuracy_score&#xA;from sklearn.decomposition import TruncatedSVD&#xA;from sklearn.linear_model import SGDClassifier&#xA;from sklearn.calibration import CalibratedClassifierCV&#xA;&#xA;data_file = &quot;Product_description.txt&quot;&#xA;#Reading the input/ dataset&#xA;data = pd.read_csv(data_file, header = 0, delimiter= &quot;\t&quot;, quoting = 3, encoding = &quot;utf8&quot;)&#xA;data = data.dropna()&#xA;&#xA;train_data, test_data, train_label,  test_label = train_test_split(data.Product_description, data.Category, &#xA;                                                                   test_size=0.3, random_state=100, stratify=data.Category)&#xA;&#xA;sgd_model = SGDClassifier(loss='hinge', n_iter=20, class_weight=&quot;balanced&quot;, n_jobs=-1,&#xA;                          random_state=42, alpha=1e-06, verbose=1)&#xA;vectorizer = HashingVectorizer(ngram_range=(1,3))&#xA;data_features = vectorizer.fit_transform(train_data.Product_description)&#xA;sgd_model.fit(data_features, train_label)&#xA;test_data_feature = vectorizer.transform(test_data.Product_Combined_Cleansed)&#xA;Output_predict = sgd_model.predict(test_data_feature)&#xA;print(accuracy_score(test_label, Output_predict))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Accuracy 77.01%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Since the dimension was high I thought reducing the dimension would increase the accuracy and reduce the training time. I used TrancatedSVD to reduce the dimension but this drastically reduced the prediction accuracy but reduced the training time to 10 minutes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Code2:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn.decomposition import TruncatedSVD&#xA;clf = TruncatedSVD(100)&#xA;clf.fit(data_features)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Output:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Accuracy 14%&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When I tried TruncatedSVD with 1000 as limit it throws memory error so only I opted to use 100 as the limit.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is stated that reducing n_features on HashingVector will cause collisions &lt;code&gt;there can be collisions: distinct tokens can be mapped to the same feature index. However, in practice, this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems)&lt;/code&gt; in scikit site.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I got optimum accuracy when I used ngram between 1 to 3 so only used that.&lt;/p&gt;&#xA;" OwnerUserId="15902" LastEditorUserId="15902" LastEditDate="2018-05-30T13:25:28.493" LastActivityDate="2018-05-30T13:25:28.493" Title="Accuracy reduces drastically when using TruncatedSVD with hashingvector" Tags="|classification|scikit-learn|dimensionality-reduction|text|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32388" PostTypeId="2" ParentId="32378" CreationDate="2018-05-30T11:46:15.463" Score="5" Body="&lt;p&gt;just re-use &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&lt;code&gt;model.fit()&lt;/code&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;on the fresh datasets using the &lt;strong&gt;already trained&lt;/strong&gt; model, simple as that :) !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(given that you do it in Keras)&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2018-05-30T11:52:09.573" LastActivityDate="2018-05-30T11:52:09.573" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32389" PostTypeId="2" ParentId="32378" CreationDate="2018-05-30T12:00:25.150" Score="5" Body="&lt;p&gt;If I understand your question correctly, the reason you think cannot concatenate your time series into a one dataset is because of their different length. Depending on your problem, you can handle this issue in multiple manners in preprocessing. But the more common way is to use sequence padding. Preprocessing methods are natively implemented in keras: &lt;a href=&quot;https://keras.io/preprocessing/sequence/&quot; rel=&quot;noreferrer&quot;&gt;https://keras.io/preprocessing/sequence/&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope that answers your question.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT 1:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The answer of @pcko1 anwser would work but'll force you into using a batch size of 1, which might give you a much higher convergence time.&lt;/p&gt;&#xA;" OwnerUserId="49293" LastActivityDate="2018-05-30T12:00:25.150" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32390" PostTypeId="2" ParentId="32387" CreationDate="2018-05-30T12:49:19.133" Score="1" Body="&lt;p&gt;Reducing from 1048576 to 100 features makes more than a 99.99% reduction of the input features dimension, knowing that SVD does not focuses on finding interesting features for classification, but creates an uncorrelated vector space from the initial one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should try to increase the dimension resulting from the SVD (have you looked at the retained variance for 100 components ?) but more specifically I suggest you to use a feature selection pipeline using a metric such as chi2 score, which gives better results for a classification goal (it may however be longer to compute).&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-30T12:49:19.133" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32391" PostTypeId="2" ParentId="32387" CreationDate="2018-05-30T12:49:30.897" Score="2" Body="&lt;p&gt;Some possible directions:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;when you're unsure how your BoW model works check different n-gram ranges (did you even check how it works with only unigrams?)&lt;/li&gt;&#xA;&lt;li&gt;TruncatedSVD by default only iterates 5 times (&lt;code&gt;n_iter&lt;/code&gt;), so you could increase that&lt;/li&gt;&#xA;&lt;li&gt;It's not at all surprising to get such drastic change of classification quality for such low dimensionality. Some questions you could ask are why 100 dimensions? Does TSVD work 1000 dimensions or something like that? What is the reconstruction error (how does TSVD's &lt;code&gt;explained_variance_ratio_&lt;/code&gt; look like)?&lt;/li&gt;&#xA;&lt;li&gt;HashingVectorizer also has parameter that can control number of features, namely &lt;code&gt;n_features&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-05-30T12:49:30.897" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32392" PostTypeId="1" AcceptedAnswerId="32403" CreationDate="2018-05-30T13:04:36.797" Score="1" ViewCount="265" Body="&lt;p&gt;I have been training a GAN in the cloud for some time now. I use Google's free credit. My laptop with a CPU doesn't seem to be up to the task.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The image I want to generate is this.&#xA;&lt;a href=&quot;https://i.stack.imgur.com/5V9io.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/5V9io.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Even though the number of epochs is about 15000 I don't get anything close to the original. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the main code. I don't claim to fully understand the deep layers. It took a few days to even write this code. The rest of the code is boilerplate to train. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There is no compilation error and I look at the images using TensorBoard.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The output from the generator is (1024,1024) images. Should this be the same as my original which is a (299,299) images.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Should I calculate using formulas how each layer transforms the image to understand it better ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How do I fix this ? I have mixed and matched API's just to create a working example assuming that doesn't create any problem.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = tf.placeholder(tf.float32, shape=[None, 299, 299, 1], name='X')&#xA;&#xA;&#xA;Z = tf.placeholder(dtype=tf.float32,&#xA;                              shape=(None, 100),&#xA;                              name='Z')&#xA;is_training = tf.placeholder(dtype=tf.bool,name='is_training')&#xA;&#xA;keep_prob = tf.placeholder(dtype=tf.float32, name='keep_prob')&#xA;keep_prob_value = 0.6&#xA;&#xA;&#xA;def generator(z,reuse=False, keep_prob=keep_prob_value,is_training=is_training):&#xA;    with tf.variable_scope('generator',reuse=reuse):&#xA;        linear = tf.layers.dense(z, 1024 * 8 * 8)&#xA;        linear  = tf.contrib.layers.batch_norm(linear, is_training=is_training,decay=0.88)&#xA;        conv = tf.reshape(linear, (-1, 128, 128, 8))&#xA;        out = tf.layers.conv2d_transpose(conv, 64,kernel_size=4,strides=2, padding='SAME')&#xA;        out = tf.layers.dropout(out, keep_prob)&#xA;        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)&#xA;        out = tf.nn.leaky_relu(out)&#xA;        out = tf.layers.conv2d_transpose(out, 32,kernel_size=4,strides=2, padding='SAME')&#xA;        out = tf.layers.dropout(out, keep_prob)&#xA;        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)&#xA;        out = tf.layers.conv2d_transpose(out, 1,kernel_size=4,strides=2, padding='SAME')&#xA;        out = tf.layers.dropout(out, keep_prob)&#xA;        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)&#xA;        print( out.get_shape())&#xA;        out = tf.nn.leaky_relu(out)&#xA;        tf.nn.tanh(out)&#xA;        return out&#xA;&#xA;&#xA;def discriminator(x,reuse=False, keep_prob=keep_prob_value):&#xA;    with tf.variable_scope('discriminator',reuse=reuse):&#xA;        out = tf.layers.conv2d(x, filters=32, kernel_size=[3, 3], padding='SAME')&#xA;        out = tf.layers.dropout(out, keep_prob)&#xA;        out = tf.nn.leaky_relu(out)&#xA;        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)&#xA;        out = tf.layers.conv2d(out, filters=64, kernel_size=[3, 3], padding='SAME')&#xA;        out = tf.layers.dropout(out, keep_prob)&#xA;        out = tf.nn.leaky_relu(out)&#xA;        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)&#xA;        out = tf.layers.dense(out, units=256, activation=tf.nn.leaky_relu)&#xA;        out = tf.layers.dense(out, units=1, activation=tf.nn.sigmoid)&#xA;        return out&#xA;&#xA;&#xA;GeneratedImage = generator(Z)&#xA;&#xA;DxL = discriminator(X)&#xA;DgL = discriminator(GeneratedImage, reuse=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51126" LastEditorUserId="51126" LastEditDate="2018-05-30T15:12:09.330" LastActivityDate="2018-05-30T15:49:43.380" Title="GAN to generate a custom image does not work" Tags="|tensorflow|gan|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="32393" PostTypeId="2" ParentId="32046" CreationDate="2018-05-30T13:11:53.060" Score="1" Body="&lt;p&gt;This depends on the data in your minority classes. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The data in each class can be considered as a sample of observations from some population. This sample may or may not represent well the whole population of instances from that class. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If the sample represents the population well, then oversampling will introduce only a small error. However, if the sample does not represent the population well, then oversampling will produce data that have statistical properties different from those of the population. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All estimates (like &lt;a href=&quot;https://machinelearningmastery.com/confidence-intervals-for-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;confidence intervals&lt;/a&gt;, &lt;a href=&quot;https://machinelearningmastery.com/prediction-intervals-for-machine-learning/&quot; rel=&quot;nofollow noreferrer&quot;&gt;prediction intervals&lt;/a&gt;) are calculated from the statistical properties of the sample (mean, variance, etc), the exact calculations being different for different distributions and learning algorithms. If statistical properties of your oversampled data are different from the statistical properties of their populations, you will get wrong estimates of the confidence and prediction intervals for your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I will illustrate this with an example.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's assume that you have 2-dimensional data (two features in each observaton) that belong to 2 classes. Data in each class are normally distributed with the standard deviation for each feature = 1. The population mean of the class 1 is (-2, 0). The population mean of the class 2 is (1, 0).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I illustrate a large population, taking 500 points for each class. The classes can be separated by logistic regression line as follows:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/DTeKI.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/DTeKI.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The regression line is almost exactly between the population means and is almost vertical because both ordinates of the population means are zero. If I take a couple of thousand points then it will be vertical.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for this picture:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np &#xA;import matplotlib.pyplot as plt&#xA;from sklearn.linear_model import LogisticRegression&#xA;&#xA;n_points1 = 500&#xA;n_points2 = 500&#xA;mu1 = np.array([-2,0])&#xA;mu2 = np.array([1,0])&#xA;&#xA;sig = 1&#xA;&#xA;cl1 = sig * np.random.randn(n_points1,2) + mu1&#xA;cl2 = sig * np.random.randn(n_points2,2) + mu2&#xA;&#xA;print('Red points:')&#xA;print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f' &#xA;    % (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )&#xA;&#xA;y1 = np.zeros((cl1.shape[0],1))&#xA;y2 = np.ones((cl2.shape[0],1))&#xA;&#xA;X = np.vstack((cl1,cl2))&#xA;y = np.vstack((y1,y2))&#xA;&#xA;logreg = LogisticRegression()&#xA;&#xA;logreg.fit(X, y.ravel())&#xA;w = logreg.coef_[0]&#xA;a = -w[0] / w[1]&#xA;&#xA;xx = np.linspace(-4,4)&#xA;yy = a * xx - (logreg.intercept_[0]) / w[1]&#xA;&#xA;plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')&#xA;plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')&#xA;plt.plot(xx, yy, 'b-')&#xA;plt.xlim((-5, 4))   &#xA;plt.ylim((-5, 5))&#xA;plt.grid()&#xA;plt.show() &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now, let's assume that the red class is under-represented (a minority class). I take 500 points for the green class, and 10 points for the red class. Then I oversample the red data by two methods. One is duplicating them 50 times, like bootstrap resampling (I color them red), and another is something like SMOTE (they are magenta). This is not exactly SMOTE. I simply added data that are in the middle between red data points. I was too lazy to calculate nearest neighbors for each observation in this simple example but it illustrates SMOTE nevertheless because in SMOTE, synthetic examples are generated inside the convex hull of existing minority examples which reduces the variance of the data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is what I get:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/POCyb.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/POCyb.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The blue line is for bootstrap, and the black line is for SMOTE. The sample means and standard deviations for the minority class are as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Red points:&#xA;mu_x = 1.600, mu_y = -0.182, std_x = 0.719, std_y = 0.753&#xA;&#xA;Magenta points:&#xA;mu_x = 1.625, mu_y = -0.174, std_x = 0.513, std_y = 0.550&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The x-component of the sample mean is overestimated (it should be = 1) and standard deviations are underestimated (they should be = 1). As a result, the line separating the classes is different from the true line on the previous picture, so a lot of new data will be classified incorrectly.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let's take a couple of more samples randomly from the same population. Again, the size of the minority class is 10, and I resample them using the same two methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/3uEZo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/3uEZo.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Red points:&#xA;mu_x = 0.944, mu_y = 0.289, std_x = 0.943, std_y = 0.867&#xA;&#xA;Magenta points:&#xA;mu_x = 0.974, mu_y = 0.298, std_x = 0.700, std_y = 0.617&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This time x-component of the sample mean is OK, its y-component is a little overestimated, and the standard deviation is underestimated. The separating line is again incorrect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/zpE7q.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/zpE7q.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;Red points:&#xA;mu_x = 0.922, mu_y = -0.236, std_x = 1.066, std_y = 1.097&#xA;&#xA;Magenta points:&#xA;mu_x = 0.924, mu_y = -0.244, std_x = 0.757, std_y = 0.749&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In the last picture we are lucky to get a sample that has almost the same mean and standard deviation as the population. Therefore, the separating line is very close to the line in the first picture where the data were balanced. Notice that the standard deviation for SMOTE is always smaller because new data are added between the existing data, not outside of them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You might consider undersampling instead of oversampling. Check this &lt;a href=&quot;http://www.svds.com/learning-imbalanced-classes/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The code for the last 3 pictures:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np &#xA;import matplotlib.pyplot as plt&#xA;from sklearn.linear_model import LogisticRegression&#xA;&#xA;n_points1 = 500&#xA;n_points2 = 10&#xA;mu1 = np.array([-2,0])&#xA;mu2 = np.array([1,0])&#xA;&#xA;sig = 1&#xA;&#xA;cl1 = sig * np.random.randn(n_points1,2) + mu1&#xA;cl2 = sig * np.random.randn(n_points2,2) + mu2&#xA;&#xA;cl2 = np.tile(cl2,(n_points1//n_points2,1)) # oversampling &#xA;&#xA;# A kind of SMOTE but not quite&#xA;cl3 = np.zeros_like(cl1)&#xA;for k in range(len(cl3)):&#xA;    i = np.random.randint(low=0, high=n_points2, size=2)&#xA;    cl3[k,:] = (cl2[i[0]]+cl2[i[1]])/2&#xA;&#xA;print('Red points:')&#xA;print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f' &#xA;    % (np.mean(cl2[:,0]), np.mean(cl2[:,1]), np.std(cl2[:,0]), np.std(cl2[:,1])) )&#xA;&#xA;print('Magenta points:')&#xA;print('mu_x = %.3f, mu_y = %.3f, std_x = %.3f, std_y = %.3f' &#xA;    % (np.mean(cl3[:,0]), np.mean(cl3[:,1]), np.std(cl3[:,0]), np.std(cl3[:,1])) )&#xA;&#xA;y1 = np.zeros((cl1.shape[0],1))&#xA;y2 = np.ones((cl2.shape[0],1))&#xA;&#xA;X = np.vstack((cl1,cl2))&#xA;X3 = np.vstack((cl1,cl3))&#xA;&#xA;y = np.vstack((y1,y2))&#xA;&#xA;logreg = LogisticRegression()&#xA;&#xA;logreg.fit(X, y.ravel())&#xA;w = logreg.coef_[0]&#xA;a = -w[0] / w[1]&#xA;&#xA;xx = np.linspace(-4,4)&#xA;yy = a * xx - (logreg.intercept_[0]) / w[1]&#xA;&#xA;plt.scatter(cl1[:,0], cl1[:,1], s = 50, marker='.', color='green')&#xA;plt.scatter(cl2[:,0], cl2[:,1], s = 50, marker='.', color='red')&#xA;plt.plot(xx, yy, 'b-')&#xA;&#xA;logreg.fit(X3, y.ravel())&#xA;w = logreg.coef_[0]&#xA;a = -w[0] / w[1]&#xA;yy = a * xx - (logreg.intercept_[0]) / w[1]&#xA;&#xA;plt.scatter(cl3[:,0], cl3[:,1], s = 10, marker='.', color='magenta')&#xA;plt.plot(xx, yy, 'k-')&#xA;plt.xlim((-5, 4))   &#xA;plt.ylim((-5, 5))&#xA;plt.grid()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-05-30T13:11:53.060" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32395" PostTypeId="2" ParentId="32355" CreationDate="2018-05-30T13:33:49.737" Score="1" Body="&lt;p&gt;The point is that we can not use test data to choose the best model or give weights to our set of models because test data must be used for the finsl evaluation of our machine learning algorithm. Thus, we have two starategies:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1- Keeping a separate validation data and choose the best model or or give weights to our set of models according to their performance on validation data. This is not the best action because our training data set usually does not include the whole data space, so when we see the new test data which is far from the training set, our model fails. So, our model lacks generalization and is overfitted to validation data and has high variance in terms of expected loss.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2- Training multiple models (trees) and average the results of them which is called bagging. This effectively reduces the variance of the model. In case of random forest which includes random selection of features, the variance is further reduced. This techniques lead to better model generalization (performance on test data).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To cut the long story short, if we choose the best model according to its performance on validation data, we run the risk of over-fitting on validation data. To prevent this problem, we should train multiple models and because we do not know anything about the distribution of test data, we should give equal weigth (priority) to these models while averaging them.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-30T13:33:49.737" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32396" PostTypeId="1" CreationDate="2018-05-30T13:36:08.403" Score="1" ViewCount="343" Body="&lt;p&gt;In Orange3 while only using its widgets, without writing Python code, I’ve implemented the following typical machine learning processes&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Train a training set, (1 file)&lt;/li&gt;&#xA;&lt;li&gt;Validating a validation set (e.g. using Naiive Bayes and kNN), and&lt;/li&gt;&#xA;&lt;li&gt;Testing the model with a test set.(another file)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I keep getting:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;test and train datasets have different target variables&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;error message. This is despite using exact same formats and names (with different values of course). What did I do wrong?&lt;/p&gt;&#xA;" OwnerUserId="52926" LastEditorUserId="29575" LastEditDate="2018-06-01T05:08:46.887" LastActivityDate="2018-06-01T05:08:46.887" Title="Classification using Orange 3" Tags="|classification|orange|machine-learning-model|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32397" PostTypeId="1" AcceptedAnswerId="32400" CreationDate="2018-05-30T13:41:39.417" Score="3" ViewCount="391" Body="&lt;p&gt;I am building a model for classification on a dataset which is collected by recording a system's behaviour through 2 years period of time. The model is going to be used in the same system in real time.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Right now I am using the whole dataset (2 years) to build my classifier but I doubt it might not be the right approach. Since I am trying to model the behaviour of a system in real time, it is possible that older data points in the dataset have become irrelevant or uninformative compared to system's current environment (distribution of the system's input changing drastically through time for example).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is how can I determine which parts of the dataset to use for training, taking the last 1.5 years instead of all 2 years for example. Is there a statistical way to help me decide that a specific time period is not helping or possibly hurting the model to correctly classify more recent data points?&lt;/p&gt;&#xA;" OwnerUserId="52925" LastActivityDate="2018-05-30T13:56:51.727" Title="Predictive modeling on time series: How far should I look back?" Tags="|predictive-modeling|time-series|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32399" PostTypeId="1" AcceptedAnswerId="32689" CreationDate="2018-05-30T13:52:33.950" Score="7" ViewCount="2512" Body="&lt;p&gt;I'm training a neural network on &lt;em&gt;'easy'&lt;/em&gt; dataset with ~15k examples. &#xA;Network overfits pretty fast.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The thing I cannot understand that after 5th epoch validation loss is starting to worsen, while precision and recall are continue to improve for 10 more epochs.&#xA;(loss = binary cross-entropy)&#xA;&lt;a href=&quot;https://i.stack.imgur.com/SkoRc.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/SkoRc.png&quot; alt=&quot;validation loss vs precision/recall&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Diving deeper:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I've checked if there is a lot of predictions around probability ~0.5, but it is not:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/wKKom.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/wKKom.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, there is a plot of percent of correct predictions based on prediction probability. There is some pattern here, but the number of elements is quite small to make conclusions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/V89h9.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/V89h9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So, my question is: why can it happen, and what to do about it?&lt;/p&gt;&#xA;" OwnerUserId="28101" LastActivityDate="2018-08-04T19:01:03.787" Title="Why validation loss worsens while precision/recall continue to improve?" Tags="|machine-learning|neural-network|loss-function|accuracy|overfitting|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32400" PostTypeId="2" ParentId="32397" CreationDate="2018-05-30T13:56:51.727" Score="0" Body="&lt;p&gt;Hold out the most recent block of data (maybe something like a month) to use as your validation data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Try multiple models and different ways of training those models. For example, training only on the most recent couple of months vs the whole time period. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take a small number of your top performers and test their performance on the validation set and see what does better. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;There’s nothing special about this answer. Anytime you want to know is X better than Y, testing performance on a held out validation set is almost always the way to go. &lt;/p&gt;&#xA;" OwnerUserId="33202" LastActivityDate="2018-05-30T13:56:51.727" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32401" PostTypeId="2" ParentId="26401" CreationDate="2018-05-30T14:59:24.863" Score="2" Body="&lt;p&gt;The point of using any recurrent layer is to have the output be a result of not only a single item independent of other items, but rather a sequence of items, such that the output of the layer's operation on one item in the sequence is the result of both that item and any item before it in the sequence. The number of timesteps defines how long such a sequence is. That is, how many items that should be handled in a sequence, and affect each other's resulting output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A LSTM layer operates in such a way that it accepts input on the form number_of_timesteps, dimensions_of_each_item. If the parameter return_sequences is set to False, which it is by default, the layer &quot;compounds&quot; the inputs of all the timesteps into a single output. If you consider a sequence of, say 10 items, a LSTM layer with return_sequences set to False will from such a sequence produce a single output item, and the attributes of this single item will be a result of all the items (timesteps) in the sequence. This is what you want in the case of a many-to-one design.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A LSTM layer with return_sequences set to True will for each item (timestep) in an input sequence produce an output. This is done in such a way that at any timestep, the output will depend not only on the item that is currently being operated on, but also the previous items in the sequence. This is what you want in the case of a many-to-many design. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;As a LSTM layer takes a sequence of items as input, any layer before a LSTM layer in your model will need to produce a sequence as an output. In the case of your Type 1 model, the first few layers do not operate on sequences, but rather a single item at a time. This does hence not produce a sequence of items on which to operate for the LSTM.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Using TimeDistributed makes it possible to have a layer operate on every item in a sequence without the items affecting each other. TimeDistributed layers thus operate on sequences of items, but there is no recursion.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of your type 2 model, the first layers will produce a sequence 5 timesteps long, and the operations done on each of the items in the sequence will be independent of each other, since the layers wrapped in TimeDistributed are non-recurrent. As the LSTM layer uses the default settings, return_sequences = False, the LSTM layer will produce a single output for each such sequence of 5 items. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The final number of output nodes in your model depends completely on the use case. A single node is suitable for something like binary classification or for producing some sort of score.&lt;/p&gt;&#xA;" OwnerUserId="51545" LastActivityDate="2018-05-30T14:59:24.863" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32403" PostTypeId="2" ParentId="32392" CreationDate="2018-05-30T15:49:43.380" Score="6" Body="&lt;p&gt;Generating text as an image is extremely difficult and I have never seen a GAN applied in the image space to generate pages of text. The reason this is so hard is because of the way in which text is perceived by humans and the way a GAN works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Humans read arbitrary symbols which are sequenced from left to right along the same line and combined into rows. Moreover, these symbols are combined into groups which represent words. This is extremely complex. The symbols must be intelligible, the words must be real ones as invented by humans. Lastly, the combination of words into sentences need to be logical and follow guidelines of human language. And even FURTHER the sequence of sentences must be coherent to transmit a message.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A GAN operating in image space will try to learn the distribution of the training set in a pixel-wise manner as that is your inputs. The distribution of the pixels will not effectively be able to group characters together in a logical manner, and the words will not be real, and the sentences will all be nonsense. You will most likely end up with blurry lines of random looking symbols, kind of like a zebra print.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Another problem is the amount of data you have. Even if this problem was possible with a GAN you would need tens of thousands of instances to train a GAN effectively. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h1&gt;What I suggest&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;I suggest to read the texts that you are training with and use this data to train a LSTM which has been shown to be very effective for generating language. But, be warned even with the best LSTMs you rarely get text that can fool humans into thinking its real.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The LSTM will provide you with your text.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then you can train a GAN to generate the characters of the alphabet and you can use this generator to print out the text that the LSTM generates.&lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T15:49:43.380" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32404" PostTypeId="2" ParentId="32399" CreationDate="2018-05-30T15:52:02.297" Score="0" Body="&lt;p&gt;Your precision and recall are not really moving significantly, neither is your loss. There are some predictions with probability around $0.5$, and I think that these are the reason for this slight trends.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As I see it, the quantities that you claim to be changing are almost constant, so I wouldn't worry about it. I might be wrong, though, but the changes in loss function are ridiculous.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-30T15:52:02.297" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32405" PostTypeId="1" AcceptedAnswerId="32406" CreationDate="2018-05-30T15:54:05.363" Score="2" ViewCount="154" Body="&lt;p&gt;Can anyone explain why the following code produces &lt;em&gt;input_t&lt;/em&gt; with a shape of (32,) instead of (,32), given the fact that &lt;em&gt;inputs&lt;/em&gt; has a shape (100, 32)? Shouldn't input_t produce a vector with 32 attributes/columns?&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np&#xA;&#xA;timesteps = 100&#xA;input_features = 32&#xA;output_features = 64&#xA;&#xA;inputs = np.random.random((timesteps, input_features))&#xA;&#xA;state_t = np.zeros((output_features,))&#xA;&#xA;W = np.random.random((output_features, input_features))&#xA;U = np.random.random((output_features, output_features))&#xA;b = np.random.random((output_features,))&#xA;&#xA;successive_outputs = [ ]&#xA;&#xA;for input_t in inputs:&#xA;    output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b)&#xA;    successive_outputs.append(output_t)&#xA;    state_t = output_t &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52713" LastActivityDate="2018-05-30T15:59:02.223" Title="Dimensions For Matrix Multiplication" Tags="|python|numpy|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32406" PostTypeId="2" ParentId="32405" CreationDate="2018-05-30T15:59:02.223" Score="0" Body="&lt;p&gt;Imagine the matrix &lt;code&gt;inputs&lt;/code&gt; as a 2D table. You have 100 rows and 32 columns. Then the for loop acts as an iterator which will return values along the first dimensional axis. This dimension thus disappears and returns the remaining dimensions. When there is a single dimension the default in Python is $(n,)$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A matrix $(100, 32)$ iterates through $(32,)$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A matrix $(100,28,28)$ iterated through $(28,28)$ &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A matrix $(100,2,2,2)$ iterated through $(2,2,2)$ &lt;/p&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-05-30T15:59:02.223" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32407" PostTypeId="1" AcceptedAnswerId="32408" CreationDate="2018-05-30T15:59:42.917" Score="3" ViewCount="10974" Body="&lt;p&gt;I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;em&gt;What quantitative metrics can I use to determine that a feature will be useful to the learning model?&lt;/em&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To analyze the feature X, I take the target mean for each group within feature X.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;mean (X=0) = 0.07&#xA;mean (X=1) = 1.15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;In this way, I can observe the effect of a feature on the target.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;I know that there must be some stronger metrics which people use to determine the strength of a feature.&lt;/strong&gt; In school I used &lt;em&gt;p&lt;/em&gt; tests to determine the statistical significance of a variable. Is there an analog in DS/ML?&lt;/p&gt;&#xA;" OwnerUserId="52935" LastEditorUserId="29575" LastEditDate="2018-05-31T04:41:18.113" LastActivityDate="2019-10-14T06:50:32.650" Title="Determine useful features for machine learning model" Tags="|feature-selection|training|beginner|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32408" PostTypeId="2" ParentId="32407" CreationDate="2018-05-30T16:41:58.947" Score="3" Body="&lt;p&gt;I suggest taking a look at this page for some more ideas:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://scikit-learn.org/stable/modules/feature_selection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature Selection&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;That being said a couple of ideas that come to mind quickly, is to:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.&lt;/li&gt;&#xA;&lt;li&gt;Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.&lt;/li&gt;&#xA;&lt;li&gt;Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;Generally, these methods will be &quot;expensive&quot; as you are fitting multiple models to get you where you need to go.&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-05-30T16:41:58.947" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32411" PostTypeId="1" CreationDate="2018-05-30T17:05:46.560" Score="1" ViewCount="42" Body="&lt;p&gt;Our team is sharing Orange &quot;work bench&quot; files across team members. Unfortunately, we find the Feature Construction widget does not display code consistently (nor is there any &quot;x&quot; to indicate an error).&#xA;1. In 2 versions of 3.4.1, we could share the Feature Construction widget&#xA;2. However, Feature nodes created in version 3.4.1 could not be viewed in 3.13 or the standalone Orange.&#xA;An image of the original coder's screen - followed by the recipient's screen - is shown. How do we package the program appropriately to share across users who may have multiple versions of Orange? &#xA;Generally, do you recommend that data manipulation like feature engineering be done in Python instead of this widget?&#xA;Thanks, Layne&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/Xm4vq.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/Xm4vq.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="30345" LastActivityDate="2018-05-30T17:05:46.560" Title="Orange Feature Construction widget: Inability to see formulas while file sharing" Tags="|orange|feature-construction|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32413" PostTypeId="1" AcceptedAnswerId="41010" CreationDate="2018-05-30T18:16:01.063" Score="7" ViewCount="358" Body="&lt;p&gt;The customary objective function for multi-label (e.g. M labels) classification is binary cross-entropy. The problem is, if we use binary cross-entropy, we are assuming that the output labels are independent of each other, turning the problem to M independent binary classification problems. Is there any suitable objective function that makes the output labels to be dependent on each other?&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2019-04-10T23:02:20.513" Title="Objective function for multi-label classification" Tags="|machine-learning|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32414" PostTypeId="1" AcceptedAnswerId="32424" CreationDate="2018-05-30T18:47:43.783" Score="1" ViewCount="118" Body="&lt;p&gt;I have some data in raw csv files which I would like to store in a MySQL database.  The problem is there are constant feature engineering done on this dataset so coming up with one schema to fit all the needs is not possible.  The approach I thought of was to have one main table where the original data is held, and for each new feature that is created, a new table is created.  Then, the user can join the original table with other tables which includes the features they want and use it for their own purposes.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;With the above approach, I'm worried about having too many joins when a user needs numerous features.  Please advise on alternative approaches to this problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;" OwnerUserId="7739" LastActivityDate="2018-05-30T23:43:18.373" Title="Storing engineered features in a database" Tags="|data-cleaning|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32415" PostTypeId="1" CreationDate="2018-05-30T18:53:29.650" Score="2" ViewCount="468" Body="&lt;p&gt;I want to separate numbers in suppose 7638 into different images which can be predicted individually using cnn.&lt;br&gt;&#xA;By finding contours how can I divide each contour into separate image in python.&lt;br&gt;&#xA;To be more clear: How can I divide &lt;a href=&quot;https://www.letter2word.com/products/americana-numbers&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.letter2word.com/products/americana-numbers&lt;/a&gt; into 0,1,2,3..9 into individual images using the concept of contours in opencv in python so that the resulting images can be predicted by cnn classifier and later all the results written together to give the number in image(initial) as output.&lt;br&gt;&#xA;Also, is there any other method to do so.&lt;br&gt;&#xA;PS.-I tried searching in google but everywhere it is given to extract 1 particular part of the image while I need more than 1 parts of the image for correct prediction&lt;/p&gt;&#xA;" OwnerUserId="52902" LastActivityDate="2024-02-28T23:00:30.097" Title="Division of numbers from contours in opencv in python for cnn" Tags="|python|cnn|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32416" PostTypeId="1" AcceptedAnswerId="32418" CreationDate="2018-05-30T19:03:19.533" Score="7" ViewCount="165" Body="&lt;p&gt;I have a data set like this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;postID  Sentence                                         drugYesOrNo&#xA;1       He went out with his friends    &#xA;2       He behaved nicely while talking with me &#xA;3       He stopped using drugs after a while                 1&#xA;4       He did not meet any friend during last week &#xA;1       He slowly cut usage of drugs                         1&#xA;2       He smiled like he is good   &#xA;3       He did not seem happy with his situation    &#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;As you see there is two features. the first feature is our sentence and the second feature shows is this sentence is a sign that patient has stopped his drug or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the first column shows that sentence which are part of a paragraph. for example HERE sentence 1-4 are one paragraph in which we have splited them to see which sentence exactly &lt;strong&gt;show the stopping of drugs&lt;/strong&gt;. &#xA;so sentence 3 in first paragraph shows this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the second case, sentence 1-3 is part of a paragraph. here sentence one shows that this person &lt;strong&gt;stopped using drugs&lt;/strong&gt;(which is not good the person should continue)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so my goal is to apply a deep learning text classifier on my text data and make a model and so when I received  A NEW PARAGRAPH, I will be able to predict if the person has stopped his drugs or not.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;first question, with this case study, which deep learning text classifier may work best?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Secondly, as you see we have cut the paragraph in to series of the sentences. but in reality we will give a paragraph to test the model. in your idea what will be the best approach to deal with this?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the thing that came to my mind is that while testing and receiving a paragraph, we again split the paragraph to sentences and give those sentences to the model but I am not sure it is a good approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;we have 900 of these sentences, again Im not sure with this much of data it will be goof to apply deep learning classifier on it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I appreciate it if you give me your point of view:)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Update after reading comments&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I asked a couple of guys to make such a dataset for me. I mean looking at paragraph, spliting, then saying which sentence has that meaning(stopping drugs or not). what if I did not ask them to explicitly say which sentence does have that meaning and just point which paragraph does have that meaning(stopping drugs or not). do you think labeling exactly which sentence does have that meaning has been a good idea rather than which paragraph does have that meaning? I hope I am clear enough :)&lt;/p&gt;&#xA;" OwnerUserId="37510" LastEditorUserId="37510" LastEditDate="2018-05-31T21:59:45.987" LastActivityDate="2018-05-31T21:59:45.987" Title="which deep learning text classifier is good for health data" Tags="|machine-learning|deep-learning|classification|text-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32417" PostTypeId="2" ParentId="32290" CreationDate="2018-05-30T19:36:41.680" Score="0" Body="&lt;p&gt;With the help from a colleague, we came up with this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;set.seed(666)&#xA;&#xA;#Dummy data&#xA;dat &amp;lt;- data.frame(samples=c(rep(paste0(&quot;A_&quot;,1:9800)),rep(paste0(&quot;B_&quot;,9801:10000))))&#xA;dat$label &amp;lt;- c(rep(&quot;A&quot;,9800),rep(&quot;B&quot;,200))&#xA;dat$continuous1 &amp;lt;- c(rnorm(9800, 100, 45),rnorm(200, 0, 25))&#xA;dat$continuous1[dat$continuous1&amp;lt;0] &amp;lt;- 0&#xA;dat$categorical1 &amp;lt;- c(rep(1:100,98),rep(1:100,2))&#xA;dat$categorical2 &amp;lt;- c(rep(1:98,100),rep(99:100,100))&#xA;&#xA;pool_A &amp;lt;- factor(1:15,levels=1:20)&#xA;pool_B &amp;lt;- factor(16:20,levels=1:20)&#xA;dat_categorical3 &amp;lt;- vector(&quot;list&quot;,10000)&#xA;names(dat_categorical3) &amp;lt;- c(as.character(dat[dat$label==&quot;A&quot;,]$samples),as.character(dat[dat$label==&quot;B&quot;,]$samples))&#xA;for(i in 1:9800){&#xA;  dat_categorical3[[i]] &amp;lt;- (as.numeric(table(sample(pool_A, 20, replace=T))) &amp;gt; 0) + 0L&#xA;}&#xA;for(i in 9801:10000){&#xA;  dat_categorical3[[i]] &amp;lt;- (as.numeric(table(sample(pool_B, 20, replace=T))) &amp;gt; 0) + 0L&#xA;}&#xA;dat_categorical3_tensor &amp;lt;- do.call(rbind,dat_categorical3)&#xA;&#xA;&#xA;## Split data for training-validating-testing&#xA;&#xA;# As we have much fewer Bs than As, each partition has to have a more or less equal number of Bs(?)&#xA;training_Bs &amp;lt;- sample(x=as.character(dat[dat$label==&quot;B&quot;,]$samples), size=67, replace=F)&#xA;validating_Bs &amp;lt;- sample(x=setdiff(as.character(dat[dat$label==&quot;B&quot;,]$samples),training_Bs), size=67, replace=F)&#xA;testing_Bs &amp;lt;- setdiff(as.character(dat[dat$label==&quot;B&quot;,]$samples),c(training_Bs,validating_Bs))&#xA;&#xA;# We also parition the data containing As equally, though this could probably be done in e.g., 80-10-10&#xA;training_As &amp;lt;- sample(x=as.character(dat[dat$label==&quot;A&quot;,]$samples), size=3267, replace=F)&#xA;validating_As &amp;lt;- sample(x=setdiff(as.character(dat[dat$label==&quot;A&quot;,]$samples),training_As), size=3267, replace=F)&#xA;testing_As &amp;lt;- setdiff(as.character(dat[dat$label==&quot;A&quot;,]$samples),c(training_As,validating_As))&#xA;&#xA;# Put together&#xA;training_dat &amp;lt;- dat[dat$samples %in% c(training_As,training_Bs),]&#xA;training_categorical3_tensor &amp;lt;- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(training_As,training_Bs),]&#xA;training_labels &amp;lt;- ifelse(training_dat$label==&quot;A&quot;,0,1)&#xA;training_dat2 &amp;lt;- as.matrix(training_dat[,c(3:5)]) #use this&#xA;&#xA;validating_dat &amp;lt;- dat[dat$samples %in% c(validating_As,validating_Bs),]&#xA;validating_categorical3_tensor &amp;lt;- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(validating_As,validating_Bs),]&#xA;validating_labels &amp;lt;- ifelse(validating_dat$label==&quot;A&quot;,0,1) &#xA;validating_dat2 &amp;lt;- as.matrix(validating_dat[,c(3:5)]) #use this&#xA;&#xA;testing_dat &amp;lt;- dat[dat$samples %in% c(testing_As,testing_Bs),]&#xA;testing_categorical3_tensor &amp;lt;- dat_categorical3_tensor[rownames(dat_categorical3_tensor) %in% c(testing_As,testing_Bs),]&#xA;testing_labels &amp;lt;- ifelse(testing_dat$label==&quot;A&quot;,0,1)&#xA;testing_dat2 &amp;lt;- as.matrix(testing_dat[,c(3:5)]) #use this&#xA;&#xA;&#xA;## Keras model&#xA;&#xA;# Input layers&#xA;all_dat_input &amp;lt;- layer_input(shape = 3, dtype = 'float32', name = 'all_dat_input')&#xA;categorical3_indices_input &amp;lt;- layer_input(shape = 20, dtype = 'float32', name = 'categorical3_input')&#xA;input_tensor &amp;lt;- c(all_dat_input, categorical3_indices_input)&#xA;&#xA;# Output layers&#xA;all_dat_out &amp;lt;- all_dat_input %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5) %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5)&#xA;&#xA;categorical3_indices_out &amp;lt;- categorical3_indices_input %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5) %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5)&#xA;&#xA;output_tensor &amp;lt;- layer_concatenate(c(all_dat_out, categorical3_indices_out)) %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5) %&amp;gt;%&#xA;  layer_dense(units = 128, activation = 'relu') %&amp;gt;%&#xA;  layer_dropout(rate = 0.5) %&amp;gt;%&#xA;  layer_dense(units=1, activation=&quot;sigmoid&quot;)&#xA;&#xA;model &amp;lt;- keras_model(inputs=input_tensor, outputs=output_tensor)&#xA;&#xA;# Compile&#xA;model %&amp;gt;% compile(&#xA;  optimizer = &quot;rmsprop&quot;,&#xA;  loss = &quot;binary_crossentropy&quot;,&#xA;  metrics = &quot;accuracy&quot;&#xA;)&#xA;&#xA;# Fit&#xA;history &amp;lt;- model %&amp;gt;% fit(&#xA;  x=list(training_dat2,training_categorical3_tensor),&#xA;  y=training_labels,&#xA;  batch_size=256,&#xA;  epochs=20,&#xA;  validation_data=list(list(validating_dat2,validating_categorical3_tensor),validating_labels)&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/WaxPy.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/WaxPy.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Compare with put aside testing data&#xA;results &amp;lt;- model %&amp;gt;% evaluate(list(testing_dat2,testing_categorical3_tensor),testing_labels)&#xA;&#xA;results&#xA;$loss&#xA;[1] 0.0000001584833&#xA;$acc&#xA;[1] 1&#xA;&#xA;# Using the trained network to generate predictions on new data&#xA;predictions &amp;lt;- model %&amp;gt;% predict(list(testing_dat2,testing_categorical3_tensor))&#xA;&#xA;head(predictions[3267:3332])&#xA;[1] 0.9999921 0.9999992 0.9999999 0.9999996 0.9999999 0.9999542    &#xA;# the network correctly identifies Bs as Bs with a confidence between &amp;gt;99%&#xA;&#xA;(Remember that this is dummy data!)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I'm currently looking into &lt;code&gt;feature importance&lt;/code&gt; to see which data inputs contributed more than others to the prediction. &lt;a href=&quot;https://github.com/thomasp85/lime&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;lime&lt;/code&gt;&lt;/a&gt; seems to be made for this!&lt;/p&gt;&#xA;" OwnerUserId="52832" LastEditorUserId="52832" LastEditDate="2018-05-31T01:49:10.887" LastActivityDate="2018-05-31T01:49:10.887" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32418" PostTypeId="2" ParentId="32416" CreationDate="2018-05-30T19:43:47.557" Score="5" Body="&lt;p&gt;Yes, you should split the paragraph to sentences and give those sentences to the model. Your deep structure should be like this:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the first layer, you must put a word embedding layer to represent a sentence as a sequence of vectors. In the second layer, you must put LSTM to be able to model your sequence vector as a single vector. Now, you can add successive layers with linear, relu or sigmoid activation functions to make your model deeper. In the last layer, you must use sigmoid activation function to do binary classification.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastActivityDate="2018-05-30T19:43:47.557" CommentCount="10" ContentLicense="CC BY-SA 4.0" />
  <row Id="32419" PostTypeId="1" CreationDate="2018-05-30T20:22:30.683" Score="1" ViewCount="63" Body="&lt;p&gt;i was trying to find a closed domain conversational agent/chatbot paper in Question and answering so not long conversation, and i don't think i see any. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;All the paper i can find are related to an open domain conversational agent or a persona based neural conversational model, etc. I really need this kind of paper since i want to implement a chatbot with a quite niche topic and i can't seem to find any that can help me out.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I search most of the paper from arxiv and this &lt;a href=&quot;https://github.com/snakeztc/NeuralDialogPapers#task-bots&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/snakeztc/NeuralDialogPapers#task-bots&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any kind of help will be greatly appreciated&lt;/p&gt;&#xA;" OwnerUserId="52946" LastActivityDate="2018-05-30T20:22:30.683" Title="Are there any paper for a closed domain conversational agent" Tags="|deep-learning|nlp|word2vec|word-embeddings|" AnswerCount="0" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32420" PostTypeId="2" ParentId="32345" CreationDate="2018-05-30T20:42:30.853" Score="1" Body="&lt;p&gt;That will vary depending on how and what framewoirk are you using, for example using tensorflow with a tf.nn.dynamic_rnn, if you pass the length sizes:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;sequence_length: (optional) An int32/int64 vector sized [batch_size]. Used to copy-through state and zero-out outputs when past a batch element's sequence length. So it's more for correctness than performance.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Hence it will not train over those sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;However if you need to pass the padded tokens, it will depend mostly on how you manage your LSTM initial state. If you have a zero initial state and want to maintain it that way you will pad with zeroes, equally if you have a random initial state you will pad with random numbers.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Unless you senunknownd the sequence_length in general you will need to use different tokens for padding and unknown words.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here are some words on pading:&#xA;(Note that when it uses tf.nn.dynamic_rnn it feeds the sequence_length):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://r2rt.com/recurrent-neural-networks-in-tensorflow-iii-variable-length-sequences.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://r2rt.com/recurrent-neural-networks-in-tensorflow-iii-variable-length-sequences.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is an analysis of different ways of manage the LSTM initial state&#xA;(Note that when it uses tf.nn.dynamic_rnn it does not feed the sequence_length):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://r2rt.com/non-zero-initial-states-for-recurrent-neural-networks.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://r2rt.com/non-zero-initial-states-for-recurrent-neural-networks.html&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Lastly, you need to train the unknown token. One thing that could improve your embedding is to categorize your unknown tokens for example with regex (If they are like verbs, adjectives, nouns, numbers, etc) . The end of the following article  under the title &quot;Unknown Words&quot; gives some ideas.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://stathwang.github.io/part-of-speech-tagging-with-trigram-hidden-markov-models-and-the-viterbi-algorithm.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://stathwang.github.io/part-of-speech-tagging-with-trigram-hidden-markov-models-and-the-viterbi-algorithm.html&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52885" LastActivityDate="2018-05-30T20:42:30.853" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32421" PostTypeId="1" CreationDate="2018-05-30T23:04:07.587" Score="1" ViewCount="76" Body="&lt;p&gt;I normalize time series data using a sliding window to be fed into a Neural Net. I normalize each window independently using Z-score. (Because on the entire dataset proven to be useless. Because on later on batch level is random due to shuffling.)&lt;br&gt;&#xA;Z-score normalize data as a pattern, as a deviation from the mean measured in the portion of the standard deviation. I apply the function on every time window (and not on batch). It would return the same for:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Window: [1, 5, 2] Mean: 2.66 STDev: 2.081 Result[-0.800, 1.120, -0.320]&lt;/li&gt;&#xA;&lt;li&gt;Window: [10, 50, 20] Mean 26.66 STDev: 20.81 Result[-0.800, 1.120, -0320]&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This is not desired. The magnitude of the input carries information that is useful and I need to preserve it while applying normalization benefits. My best idea is:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Z-Score(i) / (1 / Mean(Window(n)))&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Where (i) is the item in Window(n)&lt;br&gt;&#xA;and (n) represents the Nth sliding window  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;This results an output that keeps the magnitude: &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;[-2.135, 2.890, -0.854]&lt;/li&gt;&#xA;&lt;li&gt;[-21.35, 29.89, -8.540]&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;This output keeps the Z-score benefits while preserves the magnitude. Can you help me out if this a fair approach?&lt;/p&gt;&#xA;" OwnerUserId="52426" LastActivityDate="2018-05-30T23:04:07.587" Title="How to preserve levels with Z-score?" Tags="|neural-network|normalization|" AnswerCount="0" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32422" PostTypeId="2" ParentId="32407" CreationDate="2018-05-30T23:06:06.447" Score="1" Body="&lt;p&gt;You could stepwise (backwards or forward) remove or add features to your feature subset. For the Feature Selection procedure, you need a metric to measure which features should be included in the reduced data set of your available data. One important entropy measure is &lt;strong&gt;Mutual Information&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Mutual information is a measure between two (possibly multi-dimensional) random variables X and Y, that quantifies the amount of information obtained about one random variable, through the other random variable. The mutual information is given by &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I(X;Y)=∫∫p(x,y)logp(x,y)/p(x)p(y)dxdy,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where p(x,y) is the joint probability density function of X and Y, and where p(x) and p(y) are the marginal density functions. The mutual information determines how similar the joint distribution p(x,y) is to the products of the factored marginal distributions. If X and Y are completely unrelated (and therefore independent), then p(x,y) would equal p(x)p(y), and this integral would be zero.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If we assume that X is one Feature and Y is the target variable then we could measure their Mutual Information. We would like to keep the features with the highest mutual information between them and the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the stepwise algorithms for selecting the appropriate features there are also some greedy methods also trying to maximize the Mutual Information between the joint distribution and the target variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Below you can find some indicative links&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;http://repositorio.uchile.cl/bitstream/handle/2250/126533/A-review-of-feature-selection-methods-based-on-mutual-information.pdf?sequence=1&quot; rel=&quot;nofollow noreferrer&quot;&gt;A review of feature selection methods based&#xA;on mutual information&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0957417415004674&quot; rel=&quot;nofollow noreferrer&quot;&gt;Feature selection using Joint Mutual Information Maximisation&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-30T23:06:06.447" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32423" PostTypeId="1" AcceptedAnswerId="32891" CreationDate="2018-05-30T23:32:24.470" Score="4" ViewCount="846" Body="&lt;p&gt;I am reading about &lt;a href=&quot;https://arxiv.org/pdf/1511.05952.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Prioritized Experience Replay&lt;/a&gt;, and can't understand the following:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On page 4, every transition can be selected from the table with its own probability.&#xA;Here is the &lt;strong&gt;cumulative density&lt;/strong&gt; function (if I understood correctly):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$P(i) = \frac{p_i}{\sum_k{p_k}} $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$p_i = \frac{1}{index In Table}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Aterwards, the paper says: &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For the rank-based variant, we can approximate the &lt;strong&gt;cumulative density&lt;/strong&gt; function with a piecewise&#xA;  linear function with k segments of equal probability. The segment boundaries can be precomputed&#xA;  (they change only when N or α change). At runtime, we sample a segment, and then sample uniformly&#xA;  among the transitions within it.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;My question is, why do we have to approximate the density if it can be achieved with the following:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;roll a dice between 1 and N (with a dice that is exponentially more likely to roll a '1' rather than a '2', etc)&lt;/li&gt;&#xA;&lt;li&gt;select an item from index according to the dice.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;In c++ we have &lt;code&gt;std::exponential_distribution&lt;/code&gt; &lt;a href=&quot;http://www.cplusplus.com/reference/random/exponential_distribution/&quot; rel=&quot;nofollow noreferrer&quot;&gt;[source]&lt;/a&gt; so there is no need to approximate anything. ...If we maintain our table sorted in a descending order.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-06-01T09:59:13.087" LastActivityDate="2019-12-13T11:56:56.573" Title="Prioritized Experience Replay - why to approximate the Density Function?" Tags="|reinforcement-learning|q-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32424" PostTypeId="2" ParentId="32414" CreationDate="2018-05-30T23:43:18.373" Score="2" Body="&lt;p&gt;Depending on the technical level of your users, the frequency of the update, the complexity of the transformation, the need to share these features among the users, etc. would a custom VIEW for each user be a feasible solution?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Alternately, would you consider some ETL tools where you can create/ modify calculated columns, customizing the data pipeline as needed?&lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-05-30T23:43:18.373" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32425" PostTypeId="2" ParentId="32072" CreationDate="2018-05-31T01:01:28.867" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;But, hey! These predicted &quot;synthetic gradients&quot; will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop &quot;session A&quot;.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;-That's not necessarily correct. We typically truncate and start a new backprop because of hardware constraints, such as memory or computational speed. Vanishing-gradient can be improved by other means, such as Gradient Normalization - scaling-up the gradient vector if it gets too small beyond certain layers, or scaling-down if it's about to explode. Or even using &lt;a href=&quot;https://www.quora.com/Why-does-Batch-Normalization-for-deep-Neural-Networks-fix-the-vanishing-gradient-problem&quot; rel=&quot;nofollow noreferrer&quot;&gt;Batch Normalization&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;It's important to understand how to update any DNI module. To clear things up, consider an example of network with several layers and 3 DNI modules:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; input&#xA;   |&#xA;   V&#xA;Layer_0 &amp;amp; DNI_0&#xA;Layer_1&#xA;Layer_2&#xA;Layer_3 &amp;amp; DNI_3&#xA;Layer_4&#xA;Layer_5 &amp;amp; DNI_5&#xA;Layer_6&#xA;Layer_7&#xA;   | &#xA;   V&#xA;output&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The &lt;strong&gt;DNI_0&lt;/strong&gt; is always trained with a &lt;em&gt;synthetic&lt;/em&gt; gradient arriving from &lt;strong&gt;DNI_3&lt;/strong&gt; (flowing through Layer_2, and Layer_1 of course), siting several layers further. &lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;Likewise, the &lt;strong&gt;DNI_3&lt;/strong&gt; is always trained with a &lt;em&gt;synthetic&lt;/em&gt; gradient arriving from &lt;strong&gt;DNI_5&lt;/strong&gt;&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;&lt;strong&gt;DNI_0&lt;/strong&gt; or &lt;strong&gt;DNI_3&lt;/strong&gt; will never get to see the true gradient, because &lt;strong&gt;true&lt;/strong&gt; grad is only delivered to &lt;strong&gt;DNI_5&lt;/strong&gt;, and not earlier.&lt;/p&gt;&#xA;  &#xA;  &lt;p&gt;For anyone still struggling to understand them, read &lt;a href=&quot;https://iamtrask.github.io/2017/03/21/synthetic-gradients/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this awesome blogpost&lt;/a&gt;, part 3&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Earlier layers will have to be content with synthetic gradients, because they or their DNI will never witness the &quot;true gradient&quot;.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Regarding training in parallel with minibatches instead of Parallelizing via Synthetic Grads:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Longer sequences are more precise than minibatches, however &lt;a href=&quot;https://datascience.stackexchange.com/a/16818/43077&quot;&gt;minibatches add a regulization effect&lt;/a&gt;. But, given some technique to prevent gradient from exploding or vanishing, training longer sequences can provide a lot better insight into context of the problem. That's because network infers output after considering a longer sequence of input, so the outcome is more rational.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For the comparison of benefits granted by SG refer to the diagrams &lt;a href=&quot;https://arxiv.org/pdf/1608.05343.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;page 6 of the Paper&lt;/a&gt;, mainly being able to solve longer sequences, which I feel is most beneficial (we can already parallelize via Minibatches anyway, and thus SG shouldn't speed up the process when performed on the same machine - even if we indeed only propagate up to the next DNI). &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;However, the more DNI modules we have, the noisier the signal should&#xA; be. So it might be worth-while to train the layers and DNI all by the legacy backprop, and only after some epochs have elapsed we start using the DNI-bootstrapping discussed above. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That way, the earliest DNI will acquire at least some sense of what to expect at the start of the training.  That's because the following DNIs are themselves unsure of what the true gradient actually looks like, when the training begins, so initially, they will be advising &quot;garbage&quot; gradient to anyone sitting earlier than them.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Don't forget that authors also experimented with predicting the actual  inputs for every layer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;If your layers have expensive backprop (perhaps you have Batch-Normalization or some fancy activation functions), correcting with DNI might be a lot cheaper, once it's sufficiently well-trained.  Just remember that DNI is not free - it requires matrix multiplication and probably won't provide much speed-up on a simple dense layer.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Minibatches gives us speedup (via parallelization) and also give us the  regularization.&#xA;Synthetic Gradients allow us to infer better by working with longer sequences and (potentially) less-expensive gradient. All together this is a very powerful system.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2019-05-31T09:30:44.217" LastActivityDate="2019-05-31T09:30:44.217" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32427" PostTypeId="1" CreationDate="2018-05-31T03:34:06.440" Score="2" ViewCount="431" Body="&lt;p&gt;I am trying to estimate propensity scores in R. By this I mean that I am trying to estimate probability that an individual selects into treatment, where the selection into treatment is a binary variable. Typically people use logit or probit to estimate these. I was thinking of using regression trees.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is it okay to use rpart &quot;anova&quot; method to estimate propensities? I think I cannot use classification tree because that gives a binary outcome whereas I want the continuous propensity score function i.e. how does probability of treatment changes as the continuous explanatory variables changes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am using the following command where x and z are continuous explanatory variables and d is a binary treatment variable and all 3 are observed for each individual and I want to estimate P(D=1|X,Z).&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  cart = rpart(d~x+z, data=data, method=&quot;anova&quot;)&#xA;&#xA;  data[,pz_c:=predict(cart, type = &quot;vector&quot;)]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;where the first command estimates the propensity and the second adds the predicted propensities to the data set for each individual.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I originally posted this on stackoverflow and was told to post thee question here instead.&lt;/p&gt;&#xA;" OwnerUserId="52952" LastActivityDate="2021-01-27T12:57:20.143" Title="Estimating Propensity Score via Regression Trees (in R Using rpart)" Tags="|machine-learning|classification|r|regression|logistic-regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32428" PostTypeId="1" CreationDate="2018-05-31T03:43:00.543" Score="2" ViewCount="810" Body="&lt;p&gt;I'm developing a pipeline to fit parameters for a gradient boosting classifier while also fitting the optimum number of features in a PCA model. This is the current setup:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;pipe = Pipeline([&#xA;    ('reduce_dim', PCA()),&#xA;    ('classify', GradientBoostingClassifier())&#xA;])&#xA;score = {'f1': 'f1', 'accuracy': 'accuracy'}&#xA;N_FEATURES_OPTIONS = [12,13,14,15]&#xA;max_dep = [3,4,5,6]&#xA;n_est = [50,80,100, 120, 150]&#xA;min_samp = [4,5,6,10]&#xA;param_grid = [&#xA;    {&#xA;        'reduce_dim': [PCA()],&#xA;        'reduce_dim__n_components': N_FEATURES_OPTIONS,&#xA;        'classify__n_estimators': n_est,&#xA;        'classify__max_depth': max_dep,&#xA;        'classify__min_samples_split':min_samp&#xA;    }]&#xA;reducer_labels = ['PCA']&#xA;&#xA;grid_adc = GridSearchCV(pipe, cv=5, n_jobs=-1, param_grid=param_grid, scoring=score, refit='accuracy')&#xA;grid_adc.fit(X_train, y_train)&#xA;grid_adc.best_params_&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Which outputs:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;{'classify__max_depth': 3,&#xA; 'classify__min_samples_split': 4,&#xA; 'classify__n_estimators': 50,&#xA; 'reduce_dim': PCA(copy=True, iterated_power='auto', n_components=12, random_state=None,&#xA;   svd_solver='auto', tol=0.0, whiten=False),&#xA; 'reduce_dim__n_components': 12}&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now I'd like to validate and score the model with cross validation. If I run the following:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cross_val_score(grid, X_train, y_train, cv=5, n_jobs=-1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Does the PCA I fit from the pipeline carry over to the cross_val_score function? If so, does the cross_val_score function transform the data with the PCA every time it generates a new train/test split?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or do I need to create a new PCA after the pipeline to fit into the cross_val_score function?&lt;/p&gt;&#xA;" OwnerUserId="52940" LastActivityDate="2018-05-31T10:32:09.593" Title="Sklearn Pipelines - How to carry over PCA?" Tags="|pca|gridsearchcv|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32429" PostTypeId="1" CreationDate="2018-05-31T04:04:29.447" Score="5" ViewCount="4355" Body="&lt;p&gt;I am using the following augmentations on dataset of size 9 GB:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;datagen = ImageDataGenerator(&#xA;    featurewise_center=False,  # set input mean to 0 over the dataset&#xA;    samplewise_center=False,  # set each sample mean to 0&#xA;    featurewise_std_normalization=False,  # divide inputs by std of the dataset&#xA;    samplewise_std_normalization=False,  # divide each input by its std&#xA;    zca_whitening=True,  # apply ZCA whitening&#xA;    rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)&#xA;    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)&#xA;    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)&#xA;    horizontal_flip=True,  # randomly flip images&#xA;    vertical_flip=True)  # randomly flip images&#xA;datagen.fit(a)&#xA;&#xA;model.fit_generator(datagen.flow(a,b, batch_size=32),&#xA;                    steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;When the code comes to the &lt;code&gt;datagen.fit&lt;/code&gt;, I get into memory error ( the code doesn't even go into training)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It works fine when I comment all the augmentations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can someone tell me where I am going wrong?&lt;/p&gt;&#xA;" OwnerUserId="44680" LastEditorUserId="55122" LastEditDate="2019-06-27T11:42:28.317" LastActivityDate="2020-01-14T23:02:45.360" Title="Memory error on using data generator in keras" Tags="|python|keras|tensorflow|dataset|data-augmentation|" AnswerCount="1" CommentCount="8" ContentLicense="CC BY-SA 4.0" />
  <row Id="32431" PostTypeId="1" AcceptedAnswerId="32437" CreationDate="2018-05-31T05:37:26.533" Score="1" ViewCount="68" Body="&lt;p&gt;The typical setting for least squares regression (or over-determined linear system) for $Ax=b$ is to solve $x$ given $A$ and $b$. &#xA;In other words, $A$ and $b$ are fixed when we solve the problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is that is there any application that $b$ is changed by users (i.e., $b$ is interpreted as a query) while $A$ is fixed?&lt;/p&gt;&#xA;" OwnerUserId="52958" LastEditorUserId="50727" LastEditDate="2018-06-06T07:54:54.167" LastActivityDate="2018-06-06T07:54:54.167" Title="Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied" Tags="|machine-learning|data-mining|linear-regression|linear-algebra|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32433" PostTypeId="2" ParentId="10695" CreationDate="2018-05-31T06:28:59.277" Score="23" Body="&lt;p&gt;Thank Abhishek. I've figure it out! Here are my experiments.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;1). we plot a easy example:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from gensim.models import Word2Vec&#xA;from sklearn.decomposition import PCA&#xA;from matplotlib import pyplot&#xA;# define training data&#xA;sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],&#xA;            ['this', 'is', 'the', 'second', 'sentence'],&#xA;            ['yet', 'another', 'sentence'],&#xA;            ['one', 'more', 'sentence'],&#xA;            ['and', 'the', 'final', 'sentence']]&#xA;# train model&#xA;model_1 = Word2Vec(sentences, size=300, min_count=1)&#xA;&#xA;# fit a 2d PCA model to the vectors&#xA;X = model_1[model_1.wv.vocab]&#xA;pca = PCA(n_components=2)&#xA;result = pca.fit_transform(X)&#xA;# create a scatter plot of the projection&#xA;pyplot.scatter(result[:, 0], result[:, 1])&#xA;words = list(model_1.wv.vocab)&#xA;for i, word in enumerate(words):&#xA;    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))&#xA;pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ytYGP.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ytYGP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the above plots, we can see that easy sentences cannot distinguish different words' meaning by distances.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;2). Load pre-trained word embedding:&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from gensim.models import KeyedVectors&#xA;&#xA;model_2 = Word2Vec(size=300, min_count=1)&#xA;model_2.build_vocab(sentences)&#xA;total_examples = model_2.corpus_count&#xA;model = KeyedVectors.load_word2vec_format(&quot;glove.6B.300d.txt&quot;, binary=False)&#xA;model_2.build_vocab([list(model.vocab.keys())], update=True)&#xA;model_2.intersect_word2vec_format(&quot;glove.6B.300d.txt&quot;, binary=False, lockf=1.0)&#xA;model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)&#xA;&#xA;# fit a 2d PCA model to the vectors&#xA;X = model_2[model_1.wv.vocab]&#xA;pca = PCA(n_components=2)&#xA;result = pca.fit_transform(X)&#xA;# create a scatter plot of the projection&#xA;pyplot.scatter(result[:, 0], result[:, 1])&#xA;words = list(model_1.wv.vocab)&#xA;for i, word in enumerate(words):&#xA;    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))&#xA;pyplot.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/KA5xE.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/KA5xE.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;From the above figure, we can see that word embeddings are more meaningful.&lt;br&gt;&#xA;Hope this answer will be helpful.&lt;/p&gt;&#xA;" OwnerUserId="52960" LastEditorUserId="52960" LastEditDate="2018-05-31T06:58:59.260" LastActivityDate="2018-05-31T06:58:59.260" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="32436" PostTypeId="1" CreationDate="2018-05-31T07:18:55.117" Score="2" ViewCount="1054" Body="&lt;p&gt;I came across this article on how to create a Custom R Model and use it in Azure ML studio&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/create-r-model&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/create-r-model&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can we create a Python model instead ? While I have seen articles of python scripts for evaluation or data processing , I haven't not found enough documents on creating / using custom model built using Python ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All I have to do is , expose my custom model as web / rest service&lt;/p&gt;&#xA;" OwnerUserId="50641" LastActivityDate="2019-06-08T14:00:07.133" Title="Create Custom Python Model in Azure ML Studio?" Tags="|python|machine-learning-model|azure-ml|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32437" PostTypeId="2" ParentId="32431" CreationDate="2018-05-31T07:44:39.523" Score="1" Body="&lt;p&gt;In least squares regression, the problem that you solve is:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$X^T X \beta = X^T y$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where $\beta$ are the parameters, $X$ the regressors and $y$ the predicted variable. If you want, in your notation $A$ to be constant and $b$ to change, you need to have constant regressors and change the predicted variable $y$. I don't think there are applications that the regressors are not changed but the predictors do change, but if you want to find one it will probably be in online learning.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-31T07:44:39.523" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32438" PostTypeId="1" AcceptedAnswerId="32479" CreationDate="2018-05-31T08:00:03.103" Score="0" ViewCount="699" Body="&lt;p&gt;I want to get into machine learning. I've been in information security for the last 10 years so i have an IT background.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where is the best place to start:&#xA;Can anyone recommend a good book? And also a platform i can use to practice (preferably free)&#xA;Also if there are any online courses someone could recommend that would be great.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I looked into AWS's offering of machine learning but that is not included in the free tier.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help/advice would be much appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thank You.&lt;/p&gt;&#xA;" OwnerUserId="52966" LastActivityDate="2018-05-31T22:51:42.260" Title="How to learn Machine Learning" Tags="|machine-learning|" AnswerCount="2" CommentCount="2" ClosedDate="2018-06-02T02:03:23.417" ContentLicense="CC BY-SA 4.0" />
  <row Id="32440" PostTypeId="2" ParentId="32438" CreationDate="2018-05-31T08:13:13.470" Score="1" Body="&lt;p&gt;Courses: Andrew Ng Machine Learning Course from Coursera is what introduced me to machine learning, and I cannot think of a better way to do it. The course focuses on the theory rather than practice, but it offers solid theoretical foundations. &lt;a href=&quot;http://www.fast.ai&quot; rel=&quot;nofollow noreferrer&quot;&gt;fast.ai&lt;/a&gt; Machine Learning course is more programming oriented, I haven't taken it but it looks good.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Books: Hands on machine learning with Scikit-learn and Tensorflow by Aurélien Géron has a decent amount of theory explained in a very simple way, and it complements it with lots of code examples. If you want to dig deeper in statistics and theory, The Elements of Statistical Learning might be a good resource.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Practice: &lt;a href=&quot;http://www.kaggle.com&quot; rel=&quot;nofollow noreferrer&quot;&gt;Kaggle&lt;/a&gt; is for sure the best place to practice and learn. After reading something or doing some course I advise to download the Titanic dataset and try to submit predictions.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-31T08:13:13.470" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32441" PostTypeId="2" ParentId="32427" CreationDate="2018-05-31T08:49:02.390" Score="1" Body="&lt;p&gt;You can use logistic regression itself, since your target variable is binary. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While predicting, use &lt;code&gt;type = &quot;prob&quot;&lt;/code&gt; &#xA;This will provide the propensities which is basically just the probability of being a 1 or 0. &lt;/p&gt;&#xA;" OwnerUserId="52428" LastActivityDate="2018-05-31T08:49:02.390" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32442" PostTypeId="1" CreationDate="2018-05-31T09:02:01.423" Score="3" ViewCount="1694" Body="&lt;p&gt;I've been reading the &lt;a href=&quot;http://scikit-learn.org/stable/modules/outlier_detection.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;sklearn documentation on outlier detection&lt;/a&gt;, and even the &lt;a href=&quot;http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html#sphx-glr-auto-examples-ensemble-plot-isolation-forest-py&quot; rel=&quot;nofollow noreferrer&quot;&gt;examples provided by the documentation&lt;/a&gt;. Once I have fitted my dataset, all I can do is apply the predict function to the estimator in order to find outliers. However, I would like to get the probabilities that the point is an outlier. Can this be done in sklearn? Is there an R package to do it?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I don't even know if there is a theoretical foundation of the outlier detection methods used by sklearn that allows you to give probabilities. If not, what is the criterium that tells you what is an outlier and what is not? Does it consider probabilities or some kind of non-probabilistic scores?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any piece of information will be appreciated.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Edit&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I would like the outlier method to consider the multivariate distribution of the data. I think that univariate detection methods are rather poor.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-06-01T08:56:38.400" LastActivityDate="2019-05-01T10:02:40.707" Title="Outlier detection with sklearn" Tags="|machine-learning|scikit-learn|anomaly-detection|outlier|" AnswerCount="2" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32443" PostTypeId="1" AcceptedAnswerId="46354" CreationDate="2018-05-31T09:09:01.420" Score="5" ViewCount="566" Body="&lt;p&gt;This question is about placing the classes of neural networks in perspective to other models.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In &quot;An Introduction to Conditional Random Fields&quot; by Sutton and McCallum, the following figure is presented:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/BNspA.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/BNspA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It shows that Naive Bayes and Logistic Regression form a generative/discriminative pair and that linear-chain CRFs are a natural extension of logistic regression to sequences.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question: is it possible to extend this figure to also contain (certain kinds) of neural networks? For example, a plain feedforward neural network can be seen as multiple stacked layers of logistic regressions with activation functions. Can we then say that linear-chain CRF's in this class are a specific kind of Recurrent Neural Networks (RNN's)? &lt;/p&gt;&#xA;" OwnerUserId="52971" LastActivityDate="2019-02-27T21:21:41.837" Title="How are ANN's, RNN's related to logistic regression and CRF's?" Tags="|neural-network|logistic-regression|naive-bayes-classifier|rnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32444" PostTypeId="1" AcceptedAnswerId="32505" CreationDate="2018-05-31T10:31:25.777" Score="2" ViewCount="86" Body="&lt;p&gt;When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who is planning to start learning machine learning, I was curious how it accomplishes this? Lastly, and more specifically, I was especially wondering how and/or if this can be applied to problems such as image classification, when there is a limited amount of data (limited number of images to train the algorithm on)?&lt;/p&gt;&#xA;&lt;p&gt;Any textbook, research, or other references with regards to the latter would also be greatly appreciated.&lt;/p&gt;&#xA;&lt;p&gt;Thanks.&lt;/p&gt;&#xA;" OwnerUserId="49134" LastEditorUserId="49134" LastEditDate="2021-04-22T21:26:51.287" LastActivityDate="2021-04-22T21:26:51.287" Title="How do Bayesian methods in machine learning help with the problem of limited data? Can this be used for image classification?" Tags="|machine-learning|data|image-classification|bayesian|" AnswerCount="1" CommentCount="2" ClosedDate="2018-06-02T02:03:36.657" ContentLicense="CC BY-SA 4.0" />
  <row Id="32445" PostTypeId="2" ParentId="32428" CreationDate="2018-05-31T10:32:09.593" Score="2" Body="&lt;p&gt;When doing GridSearchCv, the best model is already scored. You can access it with the attribute &lt;code&gt;best_score_&lt;/code&gt; and get the model with &lt;code&gt;best_estimator_&lt;/code&gt;. You do not need to re-score it in a cross validation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, yes, the pipeline is entirely fitted when doing each split during the cv.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-31T10:32:09.593" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32446" PostTypeId="2" ParentId="31077" CreationDate="2018-05-31T10:38:11.943" Score="3" Body="&lt;p&gt;If you are using WGAN with Gradient Penalty, I think the Framework you are using is the limited factor since computing all the gradients will take time. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you are using WGAN with Gradient Penalty one way to get faster results is to omit the gradient penalty and just do weight clipping as mentioned in original &lt;a href=&quot;https://arxiv.org/pdf/1701.07875.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;WGAN Paper&lt;/a&gt;. But be careful in &lt;a href=&quot;https://arxiv.org/pdf/1704.00028.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Improved WGAN&lt;/a&gt; (with gradient penalty) they showed, weight clipping can lead to bias in the discriminator.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But for my experience with GANs WGAN GP gave the overall best results and I will spend the time training for it. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But also your training routine could be part of the problem since lot of reference implementations try to get an rate of which the generator updated, e.g. 1 generator update for 10-100 discriminator updates (which in some scenarios totally makes sense). So there are several factors for slow training but most important is that gradient penalty is computational expensive.&lt;/p&gt;&#xA;" OwnerUserId="52851" LastActivityDate="2018-05-31T10:38:11.943" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32449" PostTypeId="1" CreationDate="2018-05-31T12:27:45.220" Score="5" ViewCount="3958" Body="&lt;p&gt;I'm using two different functions to calculate the accuracy of my deep learning model and I am confused which one is which. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The first one is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    loss, accuracy = model.evaluate(x_train, y_train, verbose=0)&#xA;    print('Accuracy: %f' % (accuracy))&#xA;    print('Loss: %f' % (loss))&#xA;&#xA;    Accuracy: 0.731495&#xA;    Loss: 1.136258&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And the second way is to obtain the accuracy:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;test_accuracy = history2.history['val_acc']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Accuracy is 0.731 and test_accuracy is around 0.21&#xA;Which is one is my models' accuracy &lt;/p&gt;&#xA;" OwnerUserId="50353" LastEditorUserId="50727" LastEditDate="2018-06-05T08:26:05.563" LastActivityDate="2019-05-05T02:03:10.703" Title="Classification Accuracy in Keras" Tags="|machine-learning|neural-network|deep-learning|keras|accuracy|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32450" PostTypeId="1" CreationDate="2018-05-31T12:30:08.040" Score="2" ViewCount="193" Body="&lt;p&gt;The IMageNEt paper &lt;a href=&quot;https://www.researchgate.net/profile/Li_Jia_Li/publication/221361415_ImageNet_a_Large-Scale_Hierarchical_Image_Database/links/00b495388120dbc339000000/ImageNet-a-Large-Scale-Hierarchical-Image-Database.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Image Net.&lt;/a&gt; presents the Average ROC curve for the 16 classes in the imagenet data, visit image figure. 8 in the paper. what is the known function to compute this ROC plot. As ROC plot is for a binary classification problem. Is this average ROC plot made in the picture average of ROC's for all the 16 categories. Any Help would be appreciated. &lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2023-12-28T03:02:45.313" Title="calculation of average ROC in IMageNet paper?" Tags="|machine-learning|image-classification|image-recognition|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32451" PostTypeId="2" ParentId="32450" CreationDate="2018-05-31T13:20:02.543" Score="0" Body="&lt;p&gt;I would guess it is using the one vs all approach. You measure the auc of the ROC on each class adding the probabilities of all other classes to create the other class. Then you average among classes the obtained areas under the curve.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-31T13:20:02.543" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32452" PostTypeId="2" ParentId="32449" CreationDate="2018-05-31T13:23:22.620" Score="1" Body="&lt;p&gt;The accuracy given by Keras is the training accuracy. This is not a proper measure of the performance of your classifier, as it is not fair to measure accuracy with the data that has been fed to the NN. On the other hand, the test accuracy is a more fair measure of the real performance. As there is a big gap between them, you are overfitting very badly, and you should regularize your model. Consider using dropout or weight decay.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-05-31T13:23:22.620" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32453" PostTypeId="1" CreationDate="2018-05-31T13:35:57.950" Score="-3" ViewCount="6472" Body="&lt;p&gt;I have a dataset that I have run a K-means algorithm on (scikit-learn), and I want to build a decision tree on each cluster. I can recuperate the values from the cluster, but not the &quot;class&quot; values (I'm doing supervised learning, each element can belong to one of two classes and I need the value associated with the data to build my trees)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Ex: unfiltered data set:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;[val1 val2 class]&#xA;X_train=[val1 val2]&#xA;y_train=[class]&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The clustering code is this:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;X = clusterDF[clusterDF.columns[clusterDF.columns.str.contains('\'AB\'')]]&#xA;y = clusterDF['Class']&#xA;(X_train, X_test, y_train, y_test) = train_test_split(X, y,&#xA;        test_size=0.30)&#xA;&#xA;kmeans = KMeans(n_clusters=3, n_init=5, max_iter=3000, random_state=1)&#xA;kmeans.fit(X_train, y_train)&#xA;y_pred = kmeans.predict(X_test)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;And this is my (unbelievably clunky!) code for extracting the values to build the tree. The issue is the Y values; they aren't consistent with the X values&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;cl={i: np.where(kmeans.labels_ == i)[0] for i in range(kmeans.n_clusters)}&#xA;for j in range(0,len(k_means_labels_unique)):&#xA;    X=None&#xA;    Y=None&#xA;    #for i in range(0,len(k_means_labels_unique)):&#xA;    indexes = cl.get(j,0)&#xA;    for i, row in X_train.iterrows():&#xA;        if i in indexes:&#xA;            if X is not None:&#xA;                X = np.vstack([X, [row['first occurrence of \'AB\''],row['similarity to \'AB\'']]])&#xA;            else:&#xA;                X = np.array([row['first occurrence of \'AB\''],row['similarity to \'AB\'']])&#xA;    X = pd.DataFrame(data=X, index=range(0, len(X)),&#xA;                     columns=['first occurrence of \'AB\'',&#xA;        'similarity to \'AB\''])  # 1st row as the column names&#xA;    for i in range(len(y_train)):&#xA;        if i in indexes and i in list(y_train.index.values):&#xA;            if Y is not None:&#xA;                Y = np.vstack([Y, y_train[i]])&#xA;            else:&#xA;                Y = np.array(y_train[i])&#xA;&#xA;    Y = pd.DataFrame(data=Y, index=range(0, len(Y)),columns=['Class'])&#xA;&#xA;    print(&quot;\n\t-----Classifier &quot;, j + 1,&quot;----&quot;)&#xA;&#xA;    (X_train, X_test, y_train, y_test) = train_test_split(X, Y,&#xA;        test_size=0.30)&#xA;&#xA;    classifier = DecisionTreeClassifier(criterion='entropy',max_depth = 2)&#xA;    classifier = getResults(&#xA;        X_train,&#xA;        y_train,&#xA;        X_test,&#xA;        y_test,&#xA;        classifier,&#xA;        filename='classif'+str(3 + i),&#xA;        )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Any ideas (or downright more efficient ways) of taking the clustered data to make a decision tree from?&lt;/p&gt;&#xA;" OwnerUserId="52980" LastActivityDate="2018-06-01T21:41:04.597" Title="Get values from k-means cluster after clustering" Tags="|python|scikit-learn|clustering|decision-trees|supervised-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32454" PostTypeId="2" ParentId="32449" CreationDate="2018-05-31T13:59:43.103" Score="1" Body="&lt;pre&gt;&lt;code&gt;test_accuracy = history2.history['val_acc']&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;shows your model's accuracy.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-31T13:59:43.103" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32455" PostTypeId="1" AcceptedAnswerId="32462" CreationDate="2018-05-31T14:00:36.890" Score="7" ViewCount="21018" Body="&lt;p&gt;I've dataset which contains dlib landmark points of the faces. I'm using keras to train a model.&lt;/p&gt;&#xA;&lt;p&gt;The dataset shape is &lt;code&gt;(length_of_dataset,68,2)&lt;/code&gt;. I know that I've two options.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first is using &lt;code&gt;conv1d&lt;/code&gt; with &lt;code&gt;input_shape = (68,2)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The second is using &lt;code&gt;conv2d&lt;/code&gt; with &lt;code&gt;input_shape = (1,68,2)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;My question is which one is better? And why?&lt;/p&gt;&#xA;" OwnerUserId="52910" LastEditorUserId="98307" LastEditDate="2020-08-02T11:20:10.363" LastActivityDate="2020-08-02T12:23:45.540" Title="Which convolution should I use? Conv2d or Conv1d" Tags="|python|keras|tensorflow|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32456" PostTypeId="2" ParentId="25596" CreationDate="2018-05-31T14:32:46.937" Score="20" Body="&lt;p&gt;Feeding your column names into the y values argument as a list works for me like so:&lt;/p&gt;&#xA;&lt;pre&gt;&lt;code&gt;total_year[-15:].plot(x='year', y=['action', 'comedy'], figsize=(10,5), grid=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&lt;p&gt;Using something like the answer at this link is better and gives you way more control over the labels and whatnot:&#xA;&lt;a href=&quot;https://stackoverflow.com/a/21254745&quot;&gt;adding lines with plt.plot()&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52981" LastEditorUserId="29169" LastEditDate="2021-03-13T20:07:45.597" LastActivityDate="2021-03-13T20:07:45.597" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32457" PostTypeId="1" CreationDate="2018-05-31T14:48:32.400" Score="3" ViewCount="3870" Body="&lt;p&gt;I run a Support Vector Machines model on part of my train set with following result:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;alg = sk.svm.SVC(probability=True, gamma='auto')&#xA;cv_results = model_selection.cross_validate(alg, X_pca, labels, cv =4)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/cC7g9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cC7g9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;but when I am trying to tune the parameters, with following method:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model=sk.svm.SVC()&#xA;params = {'C' : [0.01, 0.1, 1, 10],&#xA;      'gamma' : [0.1, 1, 'auto'],&#xA;      'probability' : [True] &#xA;     }&#xA;clf =  GridSearchCV(model, params, cv=2, return_train_score=False).fit(X_pca, labels)&#xA;pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score', 'params']].sort_values(by='rank_test_score')&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/aDeKP.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/aDeKP.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So not only all results looks scetchy because they are the same. but also in one of the rows I have C:1, gamma:auto and probability: True which is the same parameters as in first table.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to also say, that the same logic I am using for the rest of my 15 ML algorithms and only SVM showed this kind of weird behavior. Wondering that maybe I have some stupid mistake in how I create X_pca and labels data table, I copied code from other algorithm and just replaced second code but it gave the same results. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you spot something wrong?&lt;/p&gt;&#xA;" OwnerUserId="22074" LastActivityDate="2018-09-02T18:01:54.260" Title="GridSearchCV results are different to directly applied default model (SVM)" Tags="|python|scikit-learn|svm|cross-validation|gridsearchcv|" AnswerCount="1" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32458" PostTypeId="1" CreationDate="2018-05-31T15:08:48.600" Score="2" ViewCount="383" Body="&lt;p&gt;I have a question about RNN's based on LSTM cells. Currently i'm trying to predict anomalies in time series data, based on the prediction error. Is it reasonable to run RNN's distributed in production? For example to scale it with Google Cloud ML Engine.&#xA;I want to be able to scale the model in case it has to compute to many requests during inference. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;But when I distribute the model what will happen to the memory cell? The data is split and distributed over multiple nodes, will it still recognize the pattern of the time series data?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thnx!&lt;/p&gt;&#xA;" OwnerUserId="50832" LastActivityDate="2018-05-31T21:31:11.697" Title="Can Recurrent Neural Networks (LSTM) run distributed in production?" Tags="|machine-learning|deep-learning|time-series|rnn|lstm|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32460" PostTypeId="1" CreationDate="2018-05-31T15:30:55.910" Score="10" ViewCount="4773" Body="&lt;p&gt;Several neural network libraries such as tensorflow and pytorch offer an &lt;em&gt;Embedding&lt;/em&gt; layer. Having implemented word2vec in the past, I understand the reasoning behind wanting a lower dimensional representation. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, it would seem the embedding layer is just a linear layer. All other things being equal, would an embedding layer not just learn the same weights as the equivalent linear layer? If so, then what are the advantages of using an embedding layer? &lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of word2vec, the lower dimensional representation can be used for other tasks (the infamous king/queen example). However, if your embedding layer will never be used for alternative tasks, what would be its purpose?&lt;/p&gt;&#xA;" OwnerUserId="28668" LastEditorUserId="50727" LastEditDate="2020-06-26T07:44:35.830" LastActivityDate="2020-06-26T07:44:35.830" Title="NN embedding layer" Tags="|machine-learning|deep-learning|tensorflow|word-embeddings|vector-space-models|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32461" PostTypeId="2" ParentId="32460" CreationDate="2018-05-31T15:42:17.953" Score="4" Body="&lt;p&gt;The embedding layer maps your vocabulary index input to a dense vector, so it acts as lookup layer and (if set to trainable) will be influenced on some weights only, by the words occurring in a batch of training data. Having a linear layer, it would be sequentially trained by all the data batches and would not provide a lookup functionality (each word given to the input would share the same weights).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, you're right considering word2vec differently. When using a custom trainable embedding layer, the dense vectors will be optimized (by SGD) for the task you are considering whereas models like word2vec act like language modeling and find a semantic optimum representation in the embeddings.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, depending on data size, the representation found during training might be better for your task than the one found by a &lt;em&gt;neutral&lt;/em&gt; word2vec or other model.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-31T15:42:17.953" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32462" PostTypeId="2" ParentId="32455" CreationDate="2018-05-31T15:57:17.710" Score="7" Body="&lt;p&gt;When using Conv2D , the input_shape does not have to be (1,68,2). The number of samples does not have anything to do with the convolution, one sample is given to the layer at each time anyway.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What changes is the number of spatial dimensions of your input that is convolved:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;With Conv1D, one dimension only is used, so the convolution operates on the first axis (size &lt;code&gt;68&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;With Conv2D, two dimensions are used, so the convolution operates on the two axis defining the data (size &lt;code&gt;(68,2)&lt;/code&gt;)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Therefore you have to carefully chose the filter size. For instance, if you chose a Conv2D with a filter size (4,2), it will produce the same results as a Conv1D with size (4) as it will operate fully on the second axis of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, there is no response to what is the best method. Generally, Conv2D work well on images and Conv1D on text. Given the size of your second dimension of data, Conv2D does not seem to make a lot of sense hence Conv1D should work well.&lt;/p&gt;&#xA;" OwnerUserId="41728" LastActivityDate="2018-05-31T15:57:17.710" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32463" PostTypeId="1" AcceptedAnswerId="32466" CreationDate="2018-05-31T16:14:40.237" Score="0" ViewCount="5134" Body="&lt;p&gt;How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.&lt;/p&gt;&#xA;" OwnerUserId="52987" LastActivityDate="2018-05-31T18:37:15.777" Title="Implement Sliding Window Algorithm to Detect Spikes" Tags="|python|outlier|dataframe|" AnswerCount="1" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32464" PostTypeId="1" CreationDate="2018-05-31T18:00:46.210" Score="1" ViewCount="37" Body="&lt;p&gt;I have a trained mnist-based cnn classifier which is working fine when there is only one number in the image.For more than 1 number in the image I am taking contours and separating the numbers using ROI.&lt;br&gt;&#xA;But my model is failing on the separated images&lt;br&gt;&#xA;Original image&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/rfYH9.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/rfYH9.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Contoured image&lt;br&gt;&#xA;&lt;a href=&quot;https://i.stack.imgur.com/cO1Xv.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/cO1Xv.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Here 5 is being classified as 3.&lt;br&gt;&#xA;Here is my code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import cv2&#xA;image = cv2.imread(&quot;im1.png&quot;)&#xA;gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) # grayscale&#xA;ret,thresh1 = cv2.threshold(gray,127,255,cv2.THRESH_BINARY)&#xA;gray = cv2.bilateralFilter(thresh1, 11, 17, 17)&#xA;kernel = np.ones((5,5), np.uint8)&#xA;img_dilation = cv2.dilate(gray, kernel, iterations=1)&#xA;img_erosion = cv2.erode(img_dilation, kernel, iterations=1)&#xA;#edged = cv2.Canny(gray, 30, 200)&#xA;plt.imshow(img_erosion[:,:], 'gray')&#xA;im2,contours, hierarchy = cv2.findContours(img_erosion.copy(),         cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)&#xA;contours = sorted(contours, key = cv2.contourArea, reverse = True)&#xA;# get contours&#xA;# for each contour found, draw a rectangle around it on original image&#xA;roi=[]&#xA;i=0&#xA;for contour in contours:&#xA;&#xA;    # get rectangle bounding contour&#xA;&#xA;    [x,y,w,h] = cv2.boundingRect(contour)&#xA;# discard areas that are too large&#xA;&#xA;    if h&amp;gt;300 and w&amp;gt;300:&#xA;&#xA;        continue&#xA;# discard areas that are too small&#xA;&#xA;    if h&amp;lt;40 or w&amp;lt;40:&#xA;&#xA;        continue&#xA;# draw rectangle around contour on original image&#xA;    i=i+1&#xA;    roi.append(image[y:y+h, x:x+w])&#xA;#cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,255),2)&#xA;# write original image with added contours to disk  &#xA;im_names=[&quot;im1.png&quot;,&quot;im2.png&quot;,&quot;im3.png&quot;]&#xA;i=0&#xA;&quot;&quot;&quot;for roi in roi:&#xA;    cv2.imwrite(im_names[i], roi)&#xA;    i=i+1&quot;&quot;&quot;&#xA;str1=''&#xA;for roi in roi:&#xA;&#xA;   #im = cv2.bitwise_not(image)&#xA;    #cv2.imwrite(&quot;contoured.png&quot;, image)&#xA;    im=roi&#xA;    from skimage.transform import resize&#xA;&#xA;    def rgb2gray(rgb):&#xA;        return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])&#xA;&#xA;    #im = plt.imread(&quot;contoured.png&quot;)&#xA;    im = rgb2gray(im)&#xA;    im = resize(im, (28, 28))&#xA;&#xA;    plt.imshow(im[:,:], 'gray')&#xA;    #plt.show()&#xA;&#xA;    temp = np.zeros((1,28,28,1))&#xA;    temp[0,:,:,0] = im&#xA;    num=int(classifier.predict_classes(temp))&#xA;    str1+=str((num))&#xA;print(str1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Here classifier is the trained cnn model&lt;/p&gt;&#xA;" OwnerUserId="52902" LastActivityDate="2018-05-31T18:00:46.210" Title="cnn classifier failing on image after contour selecting" Tags="|image-classification|cnn|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32465" PostTypeId="1" AcceptedAnswerId="32484" CreationDate="2018-05-31T18:05:34.900" Score="1" ViewCount="82" Body="&lt;p&gt;I have a folder which has a number of files which have a format like these &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;madvise&#xA;write&#xA;write&#xA;write&#xA;write&#xA;read&#xA;read&#xA;madvise&#xA;ioctl&#xA;ioctl&#xA;getuid&#xA;epoll_pwait&#xA;read&#xA;recvfrom&#xA;sendto&#xA;getuid&#xA;epoll_pwait&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that is it is a set of words which repeat.This is how all the files are like.&#xA;Now I have created a feature vector table using unigram that is each word becomes a feature and each file becomes a row where I put the frequency of that word occuring in the respective columns.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now I want to create a similar FVT using bigrams.I was wondering how to do that in this case.&lt;/p&gt;&#xA;" OwnerUserId="52994" LastEditorUserId="52994" LastEditDate="2018-06-01T06:57:39.190" LastActivityDate="2018-06-01T06:57:39.190" Title="Making bigram features from a particular dataset" Tags="|machine-learning|dataset|data|feature-construction|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32466" PostTypeId="2" ParentId="32463" CreationDate="2018-05-31T18:37:15.777" Score="0" Body="&lt;p&gt;welcome to DS-SE and to Data Science in general! :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Your problem can be solved really easily in Python. Please, take a look at Pandas &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;DataFrame&lt;/a&gt; class to represent your data, it makes it really convenient because of all the pre-built methods that includes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One of its methods is &lt;code&gt;pandas.DataFrame.rolling()&lt;/code&gt; (see &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rolling.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;) that does exactly what you asked for: rolling (sliding) window calculations, please take a look!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-31T18:37:15.777" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32467" PostTypeId="1" CreationDate="2018-05-31T18:45:37.197" Score="2" ViewCount="149" Body="&lt;p&gt;I have a training dataset with all the images and no sub -directories , also the images are named as random long ids and the target labels are stored in another file labels.txt with ids and the corresponding  labels. &#xA;How should I implement a generator using keras to pass these images to the model without loading the dataset in ram.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;edit:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One way I see it as &lt;/p&gt;&#xA;&#xA;&lt;p&gt;sorting the image names and labels so that i can use ImageDataGenerator.flow&lt;/p&gt;&#xA;" OwnerUserId="52997" LastEditorUserId="52997" LastEditDate="2018-05-31T19:14:17.393" LastActivityDate="2018-08-30T04:03:09.927" Title="Optimum way to train a keras nn by a dataset" Tags="|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32468" PostTypeId="1" AcceptedAnswerId="32488" CreationDate="2018-05-31T18:49:40.147" Score="1" ViewCount="181" Body="&lt;p&gt;TL:DR - Do you know of good automated tools to explore a dataset?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/scrips that could automatically round up the following answers for me:&#xA;Things I would be looking for:&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;datatype of a column?&lt;/li&gt;&#xA;&lt;li&gt;dimension or measure?&lt;/li&gt;&#xA;&lt;li&gt;histogram or count of values in a column&lt;/li&gt;&#xA;&lt;li&gt;correlation between measures&lt;/li&gt;&#xA;&lt;li&gt;unique key auto-find&lt;/li&gt;&#xA;&lt;li&gt;import into a strongly typed data structure&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Obviously I'm looking for something more than basic summary(cars). Something that goes through each column and spits out detailed analysis report&lt;/p&gt;&#xA;" OwnerUserId="52995" LastActivityDate="2018-06-22T06:59:46.530" Title="Tools to explore various datasets" Tags="|dataset|data-cleaning|data-analysis|" AnswerCount="4" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32469" PostTypeId="1" CreationDate="2018-05-31T18:53:45.440" Score="2" ViewCount="4914" Body="&lt;p&gt;I have 1000, 28*28 resolution images. I converted those 1000 images into &lt;code&gt;numpy&lt;/code&gt; array and formed a new array with size of (1000,28,28). So, while&#xA; creating convolution layer using &lt;code&gt;Keras&lt;/code&gt;, input shape(X value) is specified as (1000,28,28) and output shape(Y value) as (1000,10). Because I have 1000 examples as inputs and 10 categories of output.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model = Sequential()&#xA;model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal',input_shape=(1000,28,28)))&#xA;.&#xA;.&#xA;.&#xA;model.fit(train_x,train_y,batch_size=32,epochs=10,verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;So, while using &lt;code&gt;fit&lt;/code&gt; function, it shows &lt;code&gt;ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (1000, 28, 28)&lt;/code&gt; as an error. How can I provide proper input and output dimension for &lt;code&gt;CNN&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; model = Sequential()&#xA; model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal',input_shape=(4132,28,28)))&#xA; model.add(MaxPooling2D((2, 2)))&#xA; model.add(Dropout(0.25))&#xA;&#xA; model.add(Conv2D(64, (3, 3), activation='relu'))&#xA; model.add(MaxPooling2D(pool_size=(2, 2)))&#xA; model.add(Dropout(0.25))&#xA;&#xA; model.add(Conv2D(128, (3, 3), activation='relu'))&#xA; model.add(Dropout(0.4))&#xA;&#xA; model.add(Flatten())&#xA; model.add(Dense(128, activation='relu'))&#xA; model.add(Dropout(0.3))&#xA; model.add(Dense(10, activation='softmax'))&#xA;&#xA; model.compile(loss=keras.losses.categorical_crossentropy,optimizer=keras.optimizers.Adam(),metrics=['accuracy'])&#xA; model.summary()&#xA;&#xA; train_x  = numpy.array([train_x])&#xA;&#xA; model.fit(train_x,train_y,batch_size=32,epochs=10,verbose=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="52998" LastEditorUserId="28175" LastEditDate="2018-05-31T20:04:37.717" LastActivityDate="2019-08-29T15:00:46.387" Title="Error in Convolutional Neural network for input shape" Tags="|machine-learning|neural-network|deep-learning|keras|cnn|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32471" PostTypeId="2" ParentId="32468" CreationDate="2018-05-31T20:02:44.340" Score="1" Body="&lt;p&gt;You can probably google and find tons of similar tools, and many of them with a Freemium model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Most tools would have most of the features you want maybe except:&#xA;1) Dimension vs. Measures This one would be a bit tricky; An integer will most likely be a measure, but it could go either way&#xA;2) Unique key auto-find: again, it could be a bit tricky; It should be too much work for you to specify as needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These two should be relatively easy for a human to label accordingly as needed.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have tried the following tool and it looks quite promising. (note: I am not associated with the tool in any way)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://exploratory.io&quot; rel=&quot;nofollow noreferrer&quot;&gt;Exploratory.io&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-05-31T20:02:44.340" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32472" PostTypeId="2" ParentId="32469" CreationDate="2018-05-31T20:02:54.547" Score="1" Body="&lt;p&gt;The convolution operation used in &lt;code&gt;CNNs&lt;/code&gt; are batch operations. The customary inputs for these networks, at least for &lt;code&gt;Keras&lt;/code&gt; and &lt;code&gt;Tensorflow&lt;/code&gt;, are channels last and channels first. The first dimension specifies the number of data in your batch, the second and third can be height and width if you choose &lt;em&gt;channels-last&lt;/em&gt; and the last one will be the number of channels of your input, for instance the number of channels for grey-scaled images is one whilst for coloured images is three. If you use channels first, the first dimension would be the number of data in your batch, the second would be the number of channels and finally, the third and fourth dimensions would specify the number of height and width respectively.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For your case, it seems you are using &lt;code&gt;MNIST&lt;/code&gt; data-set. What you need to do is just reshaping your input images to one of the standard mentioned manners. You can resize your input to &lt;code&gt;(1000,28,28, 1)&lt;/code&gt;.&lt;/p&gt;&#xA;" OwnerUserId="28175" LastActivityDate="2018-05-31T20:02:54.547" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32473" PostTypeId="2" ParentId="28548" CreationDate="2018-05-31T20:20:25.483" Score="2" Body="&lt;p&gt;From my understanding (and I am no expert!)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Point 1&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let $ \hat f_k $ be a weak learner or one tree in the model, then we would make a prediction as the $$\hat y_i  = \sum_{i=0}^k \hat f_k(x) $$ with $ (x) $ being a particular feature The algorithm learns in an additive way so $$f_0(x) + f_1(x) + f_2(x) .... f_k(x) = \hat y_i $$ So adding a shinkage parameter $ \delta $ will penalise the model for each additional tree (or weak learner) you add to it, which looks like $ \delta f_k $. So having a low shinkage $ \delta $ would penalise the overall prediction score quite a lot.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Having a shrinkage score of 0.3 vs 0.01 with some $ (x) $ variable scores from each tree $f_k$;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1: $ 0.3(2.1) + 0.3(2.9) + 0.3(3.4) + 0.3(3.5) = 3.57$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 2: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;After four trees, so according to my understanding Model 2 is going to need many more trees to get to the same result as Model 1. However having a low number of rounds will likely miss important structural information in the model. I understand it this way.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Shrinkage extremely high and low, 1 and 0.01&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 1x: $ 1(2.1) + 1(2.9) + 1(3.4) + 1(3.5) = 11.9 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Model 2x: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The difference here being that Model 2x can capture 3 decimal places in its increments, whereas Model 1x goes up in increments of 0.1. So if we choose Model 1x then its possible to overshoot the optimal number of trees. If we were to increase the number of trees for Model 2x to 500 or 1000 then its possible that we would overfit the data, alternatively if we were to reduce the number of trees for Model 1x we would underfit the data. So this is where the cross validation and search for optimal parameters comes in.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So to answer point 1. Theoretically I understand yes, the numbers will converge, however this would lead to potential overfitting issues and a balance is necessary.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Point 2)&lt;/strong&gt; nrounds to me, means the number of additional trees or weak learners added to the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;At $ nround = 0 $ we have;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat y_i^0 = 0$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;at $nround = 1$ we have;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat y_i^1 = \hat y_t^0 + \delta f_1(x)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;so $ \hat y_i $ adds the previous weak learners to its new prediction&lt;/p&gt;&#xA;&#xA;&lt;p&gt;then $ nround = 2 $ we have&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$ \hat y_i^2 = \hat y_i^1 + \delta f_2(x) $$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at each $nround$ a new weak learner is added to the prediction.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Point 3)&lt;/strong&gt; Here for me the mean evaluation metric is the mean score over k-fold cross validation. If you set k-folds to 10 then it would run the model 10 times and produce just the mean of these scores, the same with standard deviation scores.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Point 4)&lt;/strong&gt; Theres many places to read about xgboost tuning, I have visited many of these websites countless times &lt;a href=&quot;https://www.google.es/search?q=xgboost%20tuning&amp;amp;oq=xgboost%20tuning&amp;amp;aqs=chrome..69i57j69i60l3j0l2.1888j0j4&amp;amp;sourceid=chrome&amp;amp;ie=UTF-8&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;One really cool piece of code I am using from &lt;a href=&quot;https://datascience.stackexchange.com/questions/9364/hypertuning-xgboost-parameters?utm_medium=organic&amp;amp;utm_source=google_rich_qa&amp;amp;utm_campaign=google_rich_qa&quot;&gt;here&lt;/a&gt;. Although my code now has expanded this for most of the parameters of XGBoost and for an AUC evaluation metric not RMSE. I can post it if you are using AUC for evaluation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am still no expert but I am trying to answer some of the questions that you have myself, so I post my understanding of what I know, sure somebody will correct me :)&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;searchGridSubCol &amp;lt;- expand.grid(subsample = c(0.5, 0.75, 1), &#xA;                                colsample_bytree = c(0.6, 0.8, 1))&#xA;ntrees &amp;lt;- 100&#xA;&#xA;#Build a xgb.DMatrix object&#xA;DMMatrixTrain &amp;lt;- xgb.DMatrix(data = yourMatrix, label = yourTarget)&#xA;&#xA;rmseErrorsHyperparameters &amp;lt;- apply(searchGridSubCol, 1, function(parameterList){&#xA;&#xA;    #Extract Parameters to test&#xA;    currentSubsampleRate &amp;lt;- parameterList[[&quot;subsample&quot;]]&#xA;    currentColsampleRate &amp;lt;- parameterList[[&quot;colsample_bytree&quot;]]&#xA;&#xA;    xgboostModelCV &amp;lt;- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE, &#xA;                           metrics = &quot;rmse&quot;, verbose = TRUE, &quot;eval_metric&quot; = &quot;rmse&quot;,&#xA;                           &quot;objective&quot; = &quot;reg:linear&quot;, &quot;max.depth&quot; = 15, &quot;eta&quot; = 2/ntrees,                               &#xA;                           &quot;subsample&quot; = currentSubsampleRate, &quot;colsample_bytree&quot; = currentColsampleRate)&#xA;&#xA;    xvalidationScores &amp;lt;- as.data.frame(xgboostModelCV)&#xA;    #Save rmse of the last iteration&#xA;    rmse &amp;lt;- tail(xvalidationScores$test.rmse.mean, 1)&#xA;&#xA;    return(c(rmse, currentSubsampleRate, currentColsampleRate))&#xA;&#xA;})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="53001" LastEditorUserId="53001" LastEditDate="2018-05-31T22:46:21.180" LastActivityDate="2018-05-31T22:46:21.180" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32474" PostTypeId="2" ParentId="32468" CreationDate="2018-05-31T21:09:42.590" Score="1" Body="&lt;p&gt;Welcome to DS-SE :) !&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A &lt;strong&gt;very&lt;/strong&gt; convenient way of doing this is by utilizing the &lt;code&gt;pandas.DataFrame&lt;/code&gt; &lt;a href=&quot;https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;class&lt;/a&gt; in Python. It includes methods (and attributes) that perform what you asked:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;Something that goes through each column and spits out detailed&#xA;  analysis report&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;such as:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;code&gt;DataFrame.corr()&lt;/code&gt; for correlation analysis&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;DataFrame.data.shape&lt;/code&gt; for dimensions&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;DataFrame.drop_duplicates()&lt;/code&gt; to keep the unique elements&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;DataFrame.hist()&lt;/code&gt; for histogram representation of the data&lt;/li&gt;&#xA;&lt;li&gt;etc..&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-05-31T21:09:42.590" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32477" PostTypeId="1" AcceptedAnswerId="33032" CreationDate="2018-05-31T22:09:30.473" Score="1" ViewCount="44" Body="&lt;p&gt;I am new in this world, I am going to start working on my final degree project about data mining. I need to look for an &lt;strong&gt;ontology related with water supplier&lt;/strong&gt; companies.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I mean, I have words like: SuppliedWater, RegisteredWater, LengthSupplyingNetwork...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am looking at &lt;a href=&quot;http://lod-cloud.net/&quot; rel=&quot;nofollow noreferrer&quot;&gt;lod-cloud&lt;/a&gt;, but I don't know where to start looking for this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any help would be appreciate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks for your help.&lt;/p&gt;&#xA;" OwnerUserId="53003" LastActivityDate="2018-06-12T15:05:13.257" Title="Water supplier ontology" Tags="|data-mining|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32478" PostTypeId="1" AcceptedAnswerId="32516" CreationDate="2018-05-31T22:43:18.123" Score="3" ViewCount="258" Body="&lt;p&gt;Inertial measurement units (IMU), usually composed of accelerometers and gyroscopes; are well known to have inherent errors in their data, originating from bias, random walk noise, temperature dependence etc. creating a highly non-linear dependence. Typically, extended Kalman filters are used to estimate and remove these errors for stable measurement of orientations and angular velocities: but even this is not entirely accurate, as some higher order errors are ignored or approximated, and the fact that the Markov assumption ignores the effect of values older than the immediately previous value while predicting a future value. The highest levels of accuracy in IMUs are usually obtained after a rigorous factory calibration (which, in turn, makes the good ones very expensive).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In a scenario like this, how applicable would recurrent neural networks be in modeling these errors? Assuming my 'training data' consists of the accelerometer and gyroscope values, which can be fused to obtain a noisy orientation estimate, and at the same time, a much more precise orientation estimate coming from another sensor (for example, a very accurate GPS): would it be possible to replace the functionality of the Kalman filter by an RNN for error estimation and sensor output prediction?&lt;/p&gt;&#xA;" OwnerUserId="50451" LastActivityDate="2018-11-05T02:50:53.870" Title="Using recurrent neural networks for modeling errors in IMUs" Tags="|rnn|lstm|estimators|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32479" PostTypeId="2" ParentId="32438" CreationDate="2018-05-31T22:51:42.260" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;Online Course: Andrew Ng, Machine Learning Course from Coursera.&lt;/li&gt;&#xA;&lt;li&gt;Book: Tom Mitchell, Machine Learning, McGraw-Hill, 1997.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-05-31T22:51:42.260" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32480" PostTypeId="1" AcceptedAnswerId="32577" CreationDate="2018-06-01T02:33:57.877" Score="13" ViewCount="11418" Body="&lt;p&gt;I've been trying to add &lt;a href=&quot;https://arxiv.org/abs/1506.02438&quot; rel=&quot;noreferrer&quot;&gt;GAE&lt;/a&gt; to my A2C implementation for a while now, but I' can't quite seem to grok how it works.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My understanding of it, is that it reduces the variance of the advantage estimation function by kind of 'averaging out' (or generalising) the advantages based off the values in the rollout. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I tried to run through the maths on my own, and in the end I just had one advantage for the whole rollout, is this right? Normally, we'd have one advantage for each timestep in the rollout.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can anyone provide an explanation on the intuition of GAE?&lt;/p&gt;&#xA;" OwnerUserId="53006" LastActivityDate="2019-02-25T22:49:42.797" Title="How does generalised advantage estimation work?" Tags="|machine-learning|reinforcement-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32481" PostTypeId="2" ParentId="32467" CreationDate="2018-06-01T02:58:30.477" Score="1" Body="&lt;p&gt;I would recommend running a script before even using Keras just to clean up your data into a more standard format such as &lt;code&gt;.h5&lt;/code&gt;. You can then bin this data into batches with a given index in the key id. Then writing the &lt;code&gt;DataGenerator&lt;/code&gt; in Keras becomes extremely easy. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The way I do it is to have the following type of keys:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;dataset_identifier_ix&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where dataset is either &lt;strong&gt;train&lt;/strong&gt;, &lt;strong&gt;validation&lt;/strong&gt; or &lt;strong&gt;test&lt;/strong&gt;, depending on the set you wish to use. Identifier in your case will be either &lt;strong&gt;image&lt;/strong&gt; and &lt;strong&gt;label&lt;/strong&gt;. Lastly, &lt;strong&gt;ix&lt;/strong&gt; is the batch index. Inside each of these batches you have $n$ instances with which to train your model. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If this is the structure of your &lt;code&gt;.h5&lt;/code&gt;, then the &lt;code&gt;DataGenerator&lt;/code&gt; code can be like&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;class DataGenerator(keras.utils.Sequence):&#xA;&#xA;    'Generates data for Keras'&#xA;    def __init__(self, dataset, filename, X_identifier, Y_identifier, &#xA;                 batch_size, percent_data_use = 1, shuffle=True):&#xA;        super(DataGenerator,self).__init__()&#xA;&#xA;        with h5py.File(filename+'.h5', 'r') as hf:&#xA;            keys = list(hf.keys())&#xA;            # Get dimensions of the input space&#xA;            temp = [i for i in keys if dataset+'_'+X_identifier in i]&#xA;            x_dims = hf[temp[0]].shape[1::]&#xA;            num_files = len(temp)&#xA;            # Get dimensions of the output space&#xA;            temp = [i for i in keys if dataset+'_'+Y_identifier in i]&#xA;            y_dims = hf[temp[0]].shape[1::]&#xA;&#xA;        # The batch size&#xA;        self.batch_size = batch_size&#xA;        self.num_files = num_files&#xA;        # Assumes the file ids are always from 0 to num_files&#xA;        self.file_ids = list(range(int(percent_data_use*num_files)))&#xA;        # Calculate the number of batches&#xA;        self.num_batches = int(percent_data_use*num_files)&#xA;&#xA;        self.filename = filename&#xA;        self.dataset = dataset &#xA;        self.X_identifier = X_identifier&#xA;        self.Y_identifier = Y_identifier&#xA;&#xA;        # Dimensions of the input and the output&#xA;        self.x_dims = x_dims&#xA;        self.y_dims = y_dims&#xA;        self.input_shape = (int(x_dims[0]*sampling_percent),) + x_dims[1:]&#xA;&#xA;        self.sampling_percent = sampling_percent&#xA;        self.shuffle = shuffle&#xA;        if shuffle is True: self.on_epoch_end()&#xA;&#xA;    def __len__(self):&#xA;        &quot;&quot;&quot;Number of batch in the Sequence.&#xA;&#xA;        Returns&#xA;            The number of batches in the Sequence.&#xA;        &quot;&quot;&quot;&#xA;        return self.num_batches&#xA;&#xA;    def __getitem__(self, index):&#xA;        &quot;&quot;&quot;Gets batch at position `index`.&#xA;        Arguments&#xA;            index: position of the batch in the Sequence.&#xA;        Returns&#xA;            A batch&#xA;        &quot;&quot;&quot;        &#xA;        X = np.zeros((self.batch_size,) + self.x_dims)&#xA;        Y = np.zeros((self.batch_size,) + self.y_dims)&#xA;&#xA;        with h5py.File(self.filename+'.h5', 'r') as hf:&#xA;&#xA;            # Input and output identifiers&#xA;            x_id = self.dataset + '_' + \&#xA;                    self.X_identifier + '_' + str(self.file_ids[index])&#xA;            y_id = self.dataset + '_' + \&#xA;                    self.Y_identifier + '_' + str(self.file_ids[index])&#xA;&#xA;            X = np.asarray(hf[x_id])&#xA;            Y = np.asarray(hf[y_id]) &#xA;&#xA;        if len(X) == 0:&#xA;            return None&#xA;&#xA;        ## Restructure X, and Y for use in the Keras network&#xA;        return X, Y&#xA;&#xA;    def on_epoch_end(self):&#xA;        'Updates indexes after each epoch'&#xA;        if self.shuffle == True:&#xA;            np.random.shuffle(self.file_ids)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="29587" LastActivityDate="2018-06-01T02:58:30.477" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32482" PostTypeId="2" ParentId="32415" CreationDate="2018-06-01T04:10:43.287" Score="0" Body="&lt;p&gt;After browsing for long hours I have found out that taking the contours as ROI is a considerable option&lt;/p&gt;&#xA;" OwnerUserId="52902" LastActivityDate="2018-06-01T04:10:43.287" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32484" PostTypeId="2" ParentId="32465" CreationDate="2018-06-01T04:43:56.067" Score="1" Body="&lt;p&gt;Bigram is better to be used with sentences. In your case, files contain a list of words, as I could understand. Therefore using bigrams in your project might not yield expecting results. However, if you are still willing to do that, this is how you calculate bigrams:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Take the list of words and count the frequencies of adjacent words. &#xA;Ex:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(madvise, write) - 1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(write, write) -  3&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(write, read) -  1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(read, read) -  1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(read, madvise) -  1&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(sendto, epoll_pwait) -  1&lt;/p&gt;&#xA;" OwnerUserId="33331" LastActivityDate="2018-06-01T04:43:56.067" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32485" PostTypeId="2" ParentId="32442" CreationDate="2018-06-01T05:11:14.917" Score="1" Body="&lt;p&gt;One of the easiest and quick way would be to plot a 'boxplot' where the &quot;dots&quot; above the upper whisker(T shaoed line) of the boxplot denotes an outlier(refer to the attached image). I hope this was what you are searching for. Do let me know if otherwise, may be I can help.&lt;img src=&quot;https://i.stack.imgur.com/jJqqV.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/p&gt;&#xA;" OwnerUserId="45356" LastActivityDate="2018-06-01T05:11:14.917" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32486" PostTypeId="2" ParentId="31773" CreationDate="2018-06-01T05:58:11.353" Score="1" Body="&lt;p&gt;I have an idea but I am not sure if it is correct. Please feel free to express whatever opinion or emotions you might have about the following solution.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Classification and regression tasks are very similar. If done, for example, via neural networks, then a network for regression will differ from the corresponding network for classification only in activation function of the output neuron and the loss function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The idea is to bin the target variable for the regression task, make a classification on the binned labels, and then use &lt;code&gt;predict_proba&lt;/code&gt; to get the probability of the predicted values to be in a certain interval.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The prediction probability for the initial regression task can be estimated based on the results of &lt;code&gt;predict_proba&lt;/code&gt; for the corresponding classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is how it can be done for the same toy problem as shown on the picture in the question. The task is to learn a 1-D gaussian function&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;def gaussian(x, mu, sig):&#xA;    return np.exp(-np.square((x-mu)/sig)/2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;given some training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I build the following neural network in Keras:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/jqkPB.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/jqkPB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The network is trained simultaneously for both classification and regression. It splits only in the last layer. The input is one-dimensional. &#xA;The hidden layer has 10 neurons. The output layer for regression is one neuron with the linear activation. The output layer for classification has several softmax neurons. Their amount depends on how many bins are filled with target variables.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In this toy example, I have 6 training data points:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# training data&#xA;x_train = np.atleast_2d([3.2487, -1.2235, -10.0, 10.0, -5.7789, 6.6834]).T&#xA;y_train = gaussian(x_train, mu, sig)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I divide the whole range were the target variable changes (0 to 1) into 10 bins. Each bin is 0.1 wide. The amount of bins can be thought of as a hyper-parameter. The more bins the closer the classification problem to the corresponding regression problem. But too many bins is probably not good.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Binning the target variable&#xA;hist, bin_edges = np.histogram(y_train, bins=np.linspace(0, 1, 11))&#xA;y_c = np.digitize(y_train, bin_edges)&#xA;n_classes = len(np.unique(y_c))&#xA;&#xA;# Binarize targets for classification&#xA;lb = LabelBinarizer()&#xA;y_b = lb.fit_transform(y_c)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;The training data fall into three bins. You can see in the picture why. The four points far on each side (two on the left side, and two on the right side) are all in one bin, and each of the remaining points in the middle is in a separate bin. The remaining seven bins are empty. So, the output layer for the classification task has 3 softmax neurons. I use 1-hot encoding for the labels.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is the network:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# NNet with one input and two outputs: one for reg, another for clf&#xA;main_input = Input(shape=(1,), dtype='float32', name='main_input')&#xA;hidden = Dense(10, input_dim=1, activation='tanh')(main_input)&#xA;reg_output = Dense(1, activation='linear', name='reg_output')(hidden)&#xA;clf_output = Dense(n_classes, activation='softmax', name='clf_output')(hidden)&#xA;model = Model(inputs=[main_input], outputs=[reg_output, clf_output])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Different loss functions are used for classification and regression. I also assign different loss weights which can be thought of as another hyper-parameter.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(optimizer='adam',&#xA;              loss={'reg_output': 'mse', 'clf_output': 'categorical_crossentropy'},&#xA;              loss_weights={'reg_output': 1., 'clf_output': 0.2})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Training:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.fit({'main_input': x_train},&#xA;          {'reg_output': y_train, 'clf_output': y_b},&#xA;          epochs=1000, verbose=0)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Running &lt;code&gt;model.predict&lt;/code&gt; gives prediction for the regression and prediction probabilities for classification.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# Prediction for both classification and regression &#xA;y_pred, pred_proba_c = model.predict({'main_input': x})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Each row of the array &lt;code&gt;pred_proba_c&lt;/code&gt; contains probabilities of putting a test point to one of three classes. I estimate a regression's analogue of &lt;code&gt;predict_proba&lt;/code&gt; by taking the maximum of these three probabilities. &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;# This is a regression's analogue of predict_proba &#xA;r_pred_proba = np.max(pred_proba_c, axis=1)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;This is the result. The prediction probability is shown in the bottom half of the picture. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZQdvd.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZQdvd.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Intuitively, the probability is high where there are training data, and it decreases in the regions between the training data. The model becomes less sure about its predictions far from the training data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The maxima of the prediction probability are not exactly at the training points. This might be because there is no exact correspondence between the underlying classification and regression problems. They are related but they are not the same, and the relationship between them depends on the values of the hyper-parameters, and the learning algorithm. For example, if I change the loss weights,&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model.compile(optimizer='adam',&#xA;              loss={'reg_output': 'mse', 'clf_output': 'categorical_crossentropy'},&#xA;              loss_weights={'reg_output': 1., 'clf_output': 1})&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I get the following picture:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/ZuuVN.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/ZuuVN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now the prediction probability values are different but the qualitative behavior is the same.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The complete code is as follows:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;import numpy as np &#xA;import matplotlib.pyplot as plt&#xA;from keras.models import Model&#xA;from keras.layers import Input, Dense&#xA;from sklearn.preprocessing import LabelBinarizer&#xA;&#xA;np.random.seed(1)&#xA;&#xA;x = np.atleast_2d(np.linspace(-10, 10, 200)).T&#xA;&#xA;mu = 0&#xA;sig = 2&#xA;&#xA;def gaussian(x, mu, sig):&#xA;    return np.exp(-np.square((x-mu)/sig)/2)&#xA;&#xA;# training data&#xA;x_train = np.atleast_2d([3.2487, -1.2235, -10.0, 10.0, -5.7789, 6.6834]).T&#xA;y_train = gaussian(x_train, mu, sig)&#xA;&#xA;# Binning the target variable&#xA;hist, bin_edges = np.histogram(y_train, bins=np.linspace(0, 1, 11))&#xA;y_c = np.digitize(y_train, bin_edges)&#xA;n_classes = len(np.unique(y_c))&#xA;&#xA;# Binarize targets for classification&#xA;lb = LabelBinarizer()&#xA;y_b = lb.fit_transform(y_c)&#xA;&#xA;# NNet with one input and two outputs: one for reg, another for clf&#xA;main_input = Input(shape=(1,), dtype='float32', name='main_input')&#xA;hidden = Dense(10, input_dim=1, activation='tanh')(main_input)&#xA;reg_output = Dense(1, activation='linear', name='reg_output')(hidden)&#xA;clf_output = Dense(n_classes, activation='softmax', name='clf_output')(hidden)&#xA;model = Model(inputs=[main_input], outputs=[reg_output, clf_output])&#xA;&#xA;model.compile(optimizer='adam',&#xA;              loss={'reg_output': 'mse', 'clf_output': 'categorical_crossentropy'},&#xA;              loss_weights={'reg_output': 1., 'clf_output': 0.2})&#xA;&#xA;model.fit({'main_input': x_train},&#xA;          {'reg_output': y_train, 'clf_output': y_b},&#xA;          epochs=1000, verbose=0)&#xA;&#xA;&#xA;# Prediction for both classification and regression &#xA;y_pred, pred_proba_c = model.predict({'main_input': x})&#xA;&#xA;# This is a regression's analogue of predict_proba &#xA;r_pred_proba = np.max(pred_proba_c, axis=1)&#xA;&#xA;f, ax = plt.subplots(2, sharex=True)&#xA;ax[0].plot(x, gaussian(x, mu, sig), color=&quot;red&quot;, label=&quot;ground truth&quot;)&#xA;ax[0].scatter(x_train, y_train, color='navy', s=30, marker='o', label=&quot;training data&quot;)&#xA;ax[0].plot(x, y_pred, 'b-', color=&quot;blue&quot;, label=&quot;prediction&quot;)&#xA;ax[0].legend(loc='best')&#xA;ax[0].grid()&#xA;ax[1].plot(x, r_pred_proba, color=&quot;navy&quot;, label=&quot;prediction probability&quot;)&#xA;ax[1].legend(loc='best')&#xA;ax[1].grid()&#xA;plt.show()&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="51836" LastEditorUserId="51836" LastEditDate="2018-06-01T06:10:58.233" LastActivityDate="2018-06-01T06:10:58.233" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32487" PostTypeId="2" ParentId="32460" CreationDate="2018-06-01T06:10:50.417" Score="10" Body="&lt;p&gt;At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However,  in practice, if you are building a deep learning software, you have to make a difference among them. This is because it does not make sense to apply an embedding layer using traditional matrix multiplication, as the input matrix is very sparse. For this reason, it is faster to do a look-up, although in terms of theory it is equivalent to doing a matrix multiplication.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-01T06:10:50.417" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32488" PostTypeId="2" ParentId="32468" CreationDate="2018-06-01T06:22:14.857" Score="2" Body="&lt;p&gt;There is a very cool active Python package called &lt;a href=&quot;https://github.com/pandas-profiling/pandas-profiling&quot; rel=&quot;nofollow noreferrer&quot;&gt;pandas-profiling&lt;/a&gt;, is exactly what you want. With a simple &lt;code&gt;pandas_profiling.ProfileReport(df)&lt;/code&gt; it returns a lot of important statistical information about your data, the official documentation says:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;For each column the following statistics - if relevant for the column&#xA;  type - are presented in an interactive HTML report:&lt;/p&gt;&#xA;  &#xA;  &lt;ul&gt;&#xA;  &lt;li&gt;Essentials: type, unique values, missing values &lt;/li&gt;&#xA;  &lt;li&gt;Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile    range &lt;/li&gt;&#xA;  &lt;li&gt;Descriptive statistics like mean, mode, standard deviation,    sum, median absolute deviation, coefficient of variation, kurtosis,&lt;br&gt;&#xA;  skewness Most frequent values &lt;/li&gt;&#xA;  &lt;li&gt;Histogram &lt;/li&gt;&#xA;  &lt;li&gt;Correlations highlighting of highly correlated variables, Spearman and Pearson matrixes&lt;/li&gt;&#xA;  &lt;/ul&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Look at their demo &lt;a href=&quot;http://nbviewer.jupyter.org/github/JosPolfliet/pandas-profiling/blob/master/examples/meteorites.ipynb&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;.&#xA;I personally used it few times and it quite nice, BUT it dataset is large (mostly I mean in variable space) it takes a long time to give the statistics. I think there should be an option to even return all these statistics for a subset of features (columns) in case you do not want for all and it is very high dimensional!&lt;/p&gt;&#xA;" OwnerUserId="44456" LastActivityDate="2018-06-01T06:22:14.857" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32489" PostTypeId="2" ParentId="32035" CreationDate="2018-06-01T07:54:10.023" Score="4" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;why the equality of both partial derivatives correspond to these&#xA;  hypothesis. I would rather understand when one partial derivative&#xA;  equals the other partial derivative multiplied by minus one.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Your intuition of &quot;trading off&quot; by &quot;subtracting&quot; a value is correct when you speak in terms of $\Delta R$ and $\Delta P$ (as you yourself noticed in the edit), but is not valid if you speak in terms of derivatives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of it this way: a partial derivative with respect to a parameter tells you by how much your metric changes if you &lt;em&gt;improve&lt;/em&gt; that variable. By the very design, $F$ will &lt;em&gt;always increase&lt;/em&gt; if you improve either $R$ or $P$ - its partial derivatives are positive everywhere.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;They are not equal everywhere, however. At some configurations $(R,P)$ you would gain more by improving one parameter rather than the other. Say, if your $R$ is much lower than $P$, your $F$-score would increase more if you improved the $R$ a bit rather than $P$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In the case of an $F_1$ metric, $R$ and $P$ enter the formula symmetrically and it is kind-of easy to see that whenever $R=P$, either of the components contributes equally to the increase of $F$. In fact, the set of points where $R$ and $P$ are &quot;equal in importance&quot; is exactly the line $R=P$ (which can also be written as $R/P=1$, if you prefer).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now consider the equation for $F_\beta$:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\frac{\text{constant}}{\beta^2\frac{1}{R} + \frac{1}{P}}.$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let us see how much it changes in response to increasing $R$ or $P$. As both parameters are in the denominator, the change in value of $F_\beta$ is proportional to the change in the value of its denominator, hence we may simplify a bit and only consider the change in&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$\beta^2\frac{1}{R} + \frac{1}{P}$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can now compute that a $dx$ increase in $R$ would decrease the value of the expression above by $\beta^2\frac{dx}{R^2}$. A similar increase in $P$ would decrease the value of the above expression by $\frac{dx}{P^2}$. There is a set of points $(R,P)$ where these effects are exactly equal. These are the points where &#xA;$$\beta^2\frac{dx}{R^2} = \frac{dx}{P^2} \quad\Rightarrow\quad \frac{R}{P}=\beta$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When $\beta &amp;gt; 1$ you are effectively putting more &lt;em&gt;marginal importance&lt;/em&gt; on recall, in the sense that until your recall is at least $\beta$ times the precision, it is always beneficial to work on it, rather than improving precision.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;What am I missing?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;You are missing the fact that the condition $\frac{\partial F_\alpha}{\partial P} = \frac{\partial F_\alpha}{\partial R}$ does not hold &lt;em&gt;everywhere&lt;/em&gt;, but just on the set of points $R=\beta P = \sqrt{\frac{1-\alpha}{\alpha}}P$. This line intersects the level curve at just a single point.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The line with slope -1 would be the tangent to the level line at that point - the green dashed line on the illustration below.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/TLzhZ.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/TLzhZ.png&quot; alt=&quot;R vs P&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="14519" LastActivityDate="2018-06-01T07:54:10.023" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32490" PostTypeId="1" AcceptedAnswerId="32502" CreationDate="2018-06-01T09:01:20.977" Score="3" ViewCount="492" Body="&lt;p&gt;I would like to generate a regression based on 6 different inputs and 4 output. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I use python and I compare sklearn and scipy. But in both cases, regression models are essentially focused on 1 output parameter. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is : can I considerer that all of my outputs are unlinked, and if yes, is it true to imagine that I can perform 4 regressions in parallel (one for each output) to make an equivalent 4 ouput model ?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Tank you for your tips !&lt;/p&gt;&#xA;" OwnerUserId="52972" LastActivityDate="2018-06-08T09:28:57.390" Title="is a 4 output regression equivalent to 4 single output regression superposition?" Tags="|python|neural-network|regression|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32491" PostTypeId="1" CreationDate="2018-06-01T09:10:34.593" Score="-1" ViewCount="30" Body="&lt;p&gt;If I have generated features using state of the art feature engineering methods of a dataset, can I use it for any kind of algorithm to build the model apart from few modifications in the features so as to plug in different algorithm?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any dependency of algorithm while building features from dataset?&lt;/p&gt;&#xA;" OwnerUserId="42498" LastActivityDate="2018-06-01T09:34:36.277" Title="Is it safe to say if features are generated once for a dataset, it may be used for any relevant algorithm?" Tags="|feature-engineering|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32492" PostTypeId="2" ParentId="24534" CreationDate="2018-06-01T09:27:52.920" Score="7" Body="&lt;p&gt;Conjugate gradient is not guaranteed to reach a global optimum or a local optimum! There are points where the gradient is very small, that are not optima (inflection points, saddle points). Gradient Descent could converge to a point $x = 0$ for the function $f(x) = x^3$.&lt;/p&gt;&#xA;" OwnerUserId="53019" LastEditorUserId="29575" LastEditDate="2018-06-01T12:47:36.447" LastActivityDate="2018-06-01T12:47:36.447" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32493" PostTypeId="2" ParentId="32491" CreationDate="2018-06-01T09:34:36.277" Score="0" Body="&lt;p&gt;No.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;An example: feature engineering for Gradient Boosting algorithms.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;XGBoost can't handle categorical variables - you need to use one-hot encoding, target encoding, or something like that if you want to use it.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the other hand more recent GBM libraries like CatBoost or LightGBM can handle categorical data and have reasonable defaults for doing that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I wouldn't say encoding categorical variables would be 'few modifications' because depending on what you do (one-hot encode vs target encoding) your model's behavior can change significantly.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastActivityDate="2018-06-01T09:34:36.277" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32494" PostTypeId="1" AcceptedAnswerId="32495" CreationDate="2018-06-01T09:39:33.917" Score="2" ViewCount="103" Body="&lt;p&gt;Let us consider a case where $1000$ data is given, i.e., the data set $U=\{x_1, \ldots, x_{1000}\}.$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When we want to use $k$-fold validation scheme, we first divide the data set into $k$ groups.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;With out loss of generality, the parameter $k$ is assumed to be $10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, we have $S_1=\{x_1, \ldots, x_{100}\}$, $S_2=\{x_{101}, \ldots, x_{200}\}$, $\ldots$, $S_{10}=\{x_{901}, \ldots, x_{1000}\}$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can obtain models, $f_k$, by learning a data set $U \setminus S_k$ for $k=1, 2, \ldots, 10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I can obtain error rates, $r_k$, by testing a data set $S_k$ with $f_k$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hence, I can obtain the error rates, $r$, by averaging $r_k$'s, i.e., $\sum_{k=1}^{10}r_k/10$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I understand the $k$-fold cross validation so far.&lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;p&gt;Most materials I've seen just say the error rate averaged by $k$ scenarios in $k$-fold cross validation. However, they do no t say about $f_k$'s.&lt;/p&gt;&#xA;&#xA;&lt;h2&gt;However, in this case, which model do I have to use among all $f_k$'s?&lt;/h2&gt;&#xA;" OwnerUserId="53021" LastEditorUserId="50727" LastEditDate="2018-06-01T17:11:02.223" LastActivityDate="2018-06-01T17:11:02.223" Title="How to select the learned model using $k$-fold cross validation?" Tags="|machine-learning|cross-validation|model-selection|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32495" PostTypeId="2" ParentId="32494" CreationDate="2018-06-01T09:44:33.900" Score="2" Body="&lt;p&gt;$k$-fold is just performed to obtain a measure of your accuracy, as using the training accuracy is generally a too optimistic measure of the accuracy. If you want to deploy a final model, what is recommended is to train a last model with all the data. In fact, when you compare two models, $f$ and $g$, what you do is obtain two error rates $r_f$ and $r_g$ by cross-validation, and you keep the model with the lowest error rate. After that, if the model with the lowest error rate is $f$, you retrain $f$ with all your data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To sum up, $k$-fold cross-validation is a method to give measures of the performance of the model, if you want to obtain the best model just train it with all your data.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-01T09:44:33.900" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32499" PostTypeId="2" ParentId="32468" CreationDate="2018-06-01T11:44:26.583" Score="0" Body="&lt;p&gt;I know Facets &lt;a href=&quot;https://github.com/PAIR-code/facets&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/PAIR-code/facets&lt;/a&gt; - which should be nice for a preliminary in-browser preview.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U0RHh.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U0RHh.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="6550" LastEditorUserId="6550" LastEditDate="2018-06-22T06:59:46.530" LastActivityDate="2018-06-22T06:59:46.530" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32501" PostTypeId="2" ParentId="32442" CreationDate="2018-06-01T13:12:26.607" Score="0" Body="&lt;p&gt;A simple trick to do outlier detection is to use the output probability of your model. If you are using a neural network for instance, you can use a softmax output which will give you a probability for each labels:&#xA;\begin{equation}&#xA;p(y=y_i) = \frac{e^{W_i^Tx+b_i}}{\sum_j e^{W_j^Tx+b_j}} &#xA;\end{equation}&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your model is accurate, for most points in your dataset, the probability should be peaked on the true label. However, if you have an outlier, then the model should get confused and then return probabilities that are more spread out over the labels. You could measure this spread by measuring the entropy of the output softmax probability distribution p(y):&#xA;\begin{equation}&#xA;H[p] = - \sum_y p(y)\log p(y),&#xA;\end{equation}&#xA;where $y$ takes one the different output categories (labels). Entropy is a measure of uncertainty, so if you have $H(p)=0$, the model is confident of the output, whereas if $H[p] = \log(N_c)$ ($N_c$ the number of labels), the model has no idea what to predict. Thus, using a threshold on the entropy (you might have to tune that threshold depending on your specific problem), you categorize a point as being an outlier or not. I think this may be a good starting point for high-dimensional data. For low-dimensional data, you can always do some density estimation and use the density as a threshold.&lt;/p&gt;&#xA;" OwnerUserId="29203" LastActivityDate="2018-06-01T13:12:26.607" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32502" PostTypeId="2" ParentId="32490" CreationDate="2018-06-01T13:15:41.597" Score="2" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;My question is : can I consider that all of my outputs are unlinked&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;No&lt;/strong&gt;, in general you can't, unless you have prior knowledge indicating so. There is a high chance that your outputs are correlated. To explore linear correlations, perform a &lt;a href=&quot;https://en.wikipedia.org/wiki/Correlation_and_dependence&quot; rel=&quot;nofollow noreferrer&quot;&gt;correlation&lt;/a&gt; analysis (calculate correlation matrix). To investigate nonlinear relations, calculate the &lt;a href=&quot;https://en.wikipedia.org/wiki/Distance_correlation&quot; rel=&quot;nofollow noreferrer&quot;&gt;distance correlation&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;and if yes, is it true to imagine that I can perform 4 regressions in&#xA;  parallel (one for each output) to make an equivalent 4 ouput model ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If your analysis proves that there is low correlation between the outputs, &lt;strong&gt;only then&lt;/strong&gt; you can implement 4 independent regression models in parallel with meaningful results.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My suggestion would be to implement a MIMO machine learning model (multiple inputs / multiple outputs), such as a &lt;a href=&quot;https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;multivariate Recurrent Neural Network&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope it helps :)&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-01T13:15:41.597" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32504" PostTypeId="2" ParentId="32256" CreationDate="2018-06-01T13:30:54.473" Score="0" Body="&lt;p&gt;OK so the trick was in the RoI Pooling Layer: it is initialized with dimensions &lt;code&gt;(1,512,7,7)&lt;/code&gt; and outputs the tensor &lt;code&gt;(#RoIs,512,7,7)&lt;/code&gt;, which in turn is an input in a fully-connected layer &lt;code&gt;fc6&lt;/code&gt; that treats the input as batch size &lt;code&gt;#RoIs&lt;/code&gt; and hence processes them in the normal way, like it would have processed any batch. &lt;/p&gt;&#xA;" OwnerUserId="1426" LastActivityDate="2018-06-01T13:30:54.473" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32505" PostTypeId="2" ParentId="32444" CreationDate="2018-06-01T14:10:36.193" Score="1" Body="&lt;p&gt;The discussion of Bayesian reasoning and its advantages over frequentist reasoning is very wide. I refer you to the book “Pattern recognition and machine learning” by Cristopher Bishop which is a great book on Bayesian reasoning.&lt;/p&gt;&#xA;" OwnerUserId="51209" LastEditorUserId="51209" LastEditDate="2018-06-01T15:25:31.443" LastActivityDate="2018-06-01T15:25:31.443" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32507" PostTypeId="1" CreationDate="2018-06-01T14:52:50.997" Score="1" ViewCount="127" Body="&lt;p&gt;As Kernel Ridge Regression computes inverse (Not a pseudo inverse) of a square matrix, so whether obtained solution would always be global optimal or not?&lt;/p&gt;&#xA;" OwnerUserId="18818" LastActivityDate="2018-08-03T17:01:37.440" Title="Whether solution of Kernel ridge regression through inverse calculation yields global optimal solution or not?" Tags="|machine-learning|neural-network|svm|" AnswerCount="1" CommentCount="0" ClosedDate="2018-09-01T23:06:03.183" ContentLicense="CC BY-SA 4.0" />
  <row Id="32508" PostTypeId="1" CreationDate="2018-06-01T16:05:09.563" Score="2" ViewCount="593" Body="&lt;p&gt;For a dataset I want to use xgboost for the optimal ensembling of $n$ forecasts instead of just using their arithmetic mean for combination. I found that xgboost generates forecasts that are worse than many of the $n$ individual forecasts the moedl could choose for combination. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I do not know why this can be the case. For the illustration of my observation I created the toy dataset below. The artificial target variable is generated by $$y = \frac{x_1+x_2}{2} \, \,\mbox{with } x_1, x_2 \sim N(0,1) $$ &#xA;Given the deterministic relationship between $y$ and the two explanatory variables $x_1$ and $x_2$, xgboost could make perfect forecasts, but it does not. The linear model easily does. Since this is the most simple multivariate linear regression model I can think of and xgboost fails I wondering about the implications. &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Why is this the case? What are the limitations of tree models for regression?   &lt;/li&gt;&#xA;&lt;li&gt;Why is then xgboost used for stacking and ensembling of forecasts if it cannot reproduce the MSE minimizing arithmetic mean as optimal combination mechanism?&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Note that the parameters of xgboost do not affect that. I tried many parameter settings and the results are never perfect. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Data Generation &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;library(tidyverse)&#xA;library(xgboost)&#xA;n &amp;lt;- 1000&#xA;param0 &amp;lt;- list(&quot;objective&quot;  = &quot;reg:linear&quot;, &quot;eval_metric&quot; = &quot;rmse&quot;)&#xA;set.seed(1)&#xA;df &amp;lt;- tibble(x1 = rnorm(n), x2 = rnorm(n), y = (x1+x2)/2)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;xgboost &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;xgtrain &amp;lt;- xgb.DMatrix(as.matrix(df[1:900,c(&quot;x1&quot;,&quot;x2&quot;)]), label = df$y[1:900], missing = NA)&#xA;xgtest &amp;lt;- xgb.DMatrix(as.matrix(df[901:1000,c(&quot;x1&quot;,&quot;x2&quot;)]), missing = NA)&#xA;#Crossvalidation just to illustrate that the algorithm &#xA;#learns something that is not correct since the test data &#xA;#cannot be forecasted with 0 error. &#xA;#xgb.cv(nrounds = 100,nfold = 10, params = param0, data = xgtrain)  &#xA;#nrounds and other parameters do not not get you to the prefect forecast&#xA;model &amp;lt;- xgb.train(nrounds = 100, params = param0, data = xgtrain)  &#xA;preds_xgb &amp;lt;- predict(model, xgtest)&#xA;#no perfect forecasts&#xA;sqrt(mean((preds_xgb-df$y[901:1000])^2))&#xA;0.04654448&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Linear regression&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;model &amp;lt;- lm(y ~ x1+x2, data = df[1:900,])&#xA;#0.5 and 0.5 for x1 and x2 as expected&#xA;model$coefficients &#xA;preds_lm &amp;lt;- predict(model, df[901:1000,c(&quot;x1&quot;,&quot;x2&quot;)])&#xA;#perfect forecasts&#xA;sqrt(mean((preds_lm-df$y[901:1000])^2))&#xA;1.389314e-15&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="53038" LastActivityDate="2018-06-03T01:42:32.783" Title="xgboost cannot identify perfectly fitting regression line" Tags="|r|regression|decision-trees|xgboost|ensemble|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32509" PostTypeId="1" CreationDate="2018-06-01T16:15:01.197" Score="2" ViewCount="83" Body="&lt;p&gt;How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by:&#xA;-(ylog(actual_output)+(1-y)*(1-actual_output)&lt;/p&gt;&#xA;" OwnerUserId="53040" LastEditorUserId="53040" LastEditDate="2018-06-02T02:23:36.597" LastActivityDate="2018-06-05T15:25:26.393" Title="Cross error loss function cause division by zero error" Tags="|machine-learning|deep-learning|classification|loss-function|activation-function|" AnswerCount="0" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32510" PostTypeId="1" AcceptedAnswerId="32511" CreationDate="2018-05-26T09:09:23.300" Score="0" ViewCount="1906" Body="&lt;p&gt;I am trying to perform a very simple experiment, predict the input number. The concept is same as an auto-encoder. But with just one layer, which can handle the task of encoding and decoding-&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also, wanted to observe how the network learns, with different training examples.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Initially, I took 10000 training samples, of integers, in range[1, 10000]. So, example if I pass 767676 as one of the test sample it should predict the output as 767676.0&#xA;This test passed very well.&#xA;The activation function used here was 'softplus', which keeps the values in range [0, inf).&#xA;the network - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_d = numpy.array([i for i in range(10000)], dtype=np.int)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(1, input_shape=(1, ), activation='softplus'))&#xA;&#xA;model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Now when I gave it 10000 training sample of decimal values in the range[-1, 1].  E.g. 0.3456 the expected output will be 0.3456 but rather, its giving me 0.5721. &#xA;The network - &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;train_d = numpy.array[round(random.uniform(-1, 1), 4) for i in range(10000)], dtype=np.float)&#xA;&#xA;model = Sequential()&#xA;model.add(Dense(1, input_shape=(1,), activation='softsign'))&#xA;model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="53169" OwnerDisplayName="Shivam Srivastava" LastActivityDate="2018-06-01T18:30:35.007" Title="Why my neural network does not predict decimal values in range [-1,1]? When it is able to predict the integer values" Tags="|deep-learning|keras|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32511" PostTypeId="2" ParentId="32510" CreationDate="2018-05-28T13:38:43.650" Score="3" Body="&lt;p&gt;First of all, during the 10000 integer test, did you use &lt;strong&gt;all&lt;/strong&gt; the integers from 0 to 9999 in training? If yes, then you have &lt;strong&gt;fully covered the whole input range&lt;/strong&gt;. This means that while testing, you actually feed the network with data that are identical to the training data, therefore the accuracy is very high. What is the result of the network if you train it with 10000 randomly sampled integers in range of 10,000 and then test it with ALL integers in range of 10000? This test will reveal if your encoder generalizes well enough. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also keep in mind that the smaller the number, the more difficult it is for the network to train due to vanishing gradient. Therefore, decimal numbers can saturate the learning process and lead to lower accuracy. Try changing the batch size (use smaller batches) and see if you decrease your output error.&lt;/p&gt;&#xA;" OwnerUserId="52089" OwnerDisplayName="pcko1" LastActivityDate="2018-05-28T13:38:43.650" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32512" PostTypeId="2" ParentId="32298" CreationDate="2018-06-01T18:46:10.657" Score="1" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/03Tu4.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/03Tu4.png&quot; alt=&quot;This is the output from xgboost package&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;On the y axis. That represents the log odd ratios. The formula is log(p/1-p)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P/1-P  P represents probability of success. 1- P represents probability of failure.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;When log odd ratio is 0. It means odd is 1. Probability of success is the same as the probability of failure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When log odd ratio is below 0, it means that the probability of sucess is less than probability of failure.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;When log odd ratio is above 0, it means that the probability of sucess is higher than probability of failure. &lt;/p&gt;&#xA;" OwnerUserId="51668" LastActivityDate="2018-06-01T18:46:10.657" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32515" PostTypeId="1" AcceptedAnswerId="32519" CreationDate="2018-06-01T19:52:45.043" Score="3" ViewCount="681" Body="&lt;p&gt;Vector Autoregressive models are exploited at Economics faculties all around the world. They are just another statistical model that solves problem of forecasting, although in a deeply complexity-uncovering manner.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Yet to my surprise, there is no evidence it has been used outside pure economics domain, namely, to solve business problems like we all - Data Scientists - do.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Can you share either your experience with application of VAR to solve business problem, a scenario in which it could hypothetically be used or state why it cannot be easily applied in business environment (if this is an objective, mathematically/empirically grounded reason, I am &lt;em&gt;not&lt;/em&gt; looking for &lt;em&gt;opinions&lt;/em&gt; on why this method has slow adoption)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The question can apply as well to VECM models of course.&lt;/p&gt;&#xA;" OwnerUserId="15430" LastEditorUserId="15430" LastEditDate="2018-06-01T20:44:57.253" LastActivityDate="2018-06-01T22:03:19.950" Title="Can you provide examples of business application of vector autoregressive model?" Tags="|time-series|methodology|methods|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32516" PostTypeId="2" ParentId="32478" CreationDate="2018-06-01T20:05:06.630" Score="3" Body="&lt;p&gt;The only way to find out is to try it, but I doubt it.  A Kalman makes a specific assumption about the &quot;priors&quot;, i.e., about the nature of how the signals are related over time.  This is based on knowledge about the physics of movement and how this affects the sensor values, combined with some assumptions about the likelihood of different kinds of movements.  Assuming the model and assumptions are accurate, this helps provide a more accurate estimate of the orientation.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In contrast, a recurrent neural network has no knowledge built in, and no useful &quot;priors&quot;.  It has no model about the physics of movements and no assumptions.  In principle, this makes it more general.  But in practice, it means that, if you have a limited amount of data, it is likely to be less accurate.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general, a strong prior and useful model can be very helpful.  In the limit, as you have an unlimited amount of data, you might not need a model.  But with a limited amount of data, the model can be helpful.  The Kalman filter provides a pretty reasonable model, and when trying to work with data from IMUs, we typically have only a limited amount of data.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The place where RNNs could be more effective would be if the assumptions made by the Kalman filter's model do not correspond to reality.  Then, you could imagine that RNNs could possibly learn the dynamics, because they don't assume a particular model, whereas Kalman filter is locked into a particular model.  I don't particularly expect this to happen in practice, but it's a possibility.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Overall, from first principles, I would expect/predict RNNs to be less effective than Kalman filtering.  But that's just speculation/guessing.  If you really want to know, the way to find out for sure is to try some experiments.&lt;/p&gt;&#xA;" OwnerUserId="8560" LastActivityDate="2018-06-01T20:05:06.630" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32517" PostTypeId="1" CreationDate="2018-06-01T20:39:08.487" Score="2" ViewCount="25" Body="&lt;p&gt;An example of what I'd like to do is identify the price of a product on a product page. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;While I can train a CNN to identify prices, it's likely that it would recognise every instance of a price on a given page and end up picking up the prices of recommended products in addition to the price of the primary product.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Are there examples of visual neural architectures which take the context of the object into account? I'd imagine that would be something like a combination of an RNN and a CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT: &#xA;Maybe a simpler approach would be to find a way to apply multiple labels to the same bounding box?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The reason I say this is that it if there are 10 prices on the page, it would be easier for the network to identify them first as prices, and secondarily apply a label for the primary price&lt;/p&gt;&#xA;" OwnerUserId="53046" LastEditorUserId="29169" LastEditDate="2021-02-13T04:19:32.897" LastActivityDate="2021-02-13T04:19:32.897" Title="Contextual Object Detection" Tags="|rnn|object-detection|convolutional-neural-network|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32518" PostTypeId="2" ParentId="32453" CreationDate="2018-06-01T21:21:05.370" Score="1" Body="&lt;p&gt;From the sklearn manual on kmeans:&lt;/p&gt;&#xA;&lt;blockquote&gt;&#xA;&lt;p&gt;fit(X, y=None)&lt;/p&gt;&#xA;&lt;p&gt;Compute k-means clustering.&lt;/p&gt;&#xA;&lt;p&gt;Parameters:&lt;/p&gt;&#xA;&lt;p&gt;X : array-like or sparse matrix, shape=(n_samples, n_features)&#xA;Training instances to cluster.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;y : ignored&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&lt;p&gt;Clustering is not classification. It is not even trying to &lt;code&gt;predict&lt;/code&gt; the Y you provided. So it's not obvious to me what you are trying to achieve. You have some unused variables in there (&lt;code&gt;y_pred&lt;/code&gt;). Why would you want to use k-means here: you are doing supervised learning!&lt;/p&gt;&#xA;&lt;p&gt;If you want the labels of the points in the first cluster, that probably is just a &lt;code&gt;y_train[kmeans.labels_==0]&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="924" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-06-01T21:41:04.597" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32519" PostTypeId="2" ParentId="32515" CreationDate="2018-06-01T22:00:09.260" Score="1" Body="&lt;p&gt;Caveat: I have a doctorate in economics and that is why I knew how, and where, and when to apply this type of model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sure, I used a vecm model last year to figure out how many credit cards get compromised per month given how much activity we see. (I work for a financial institution).  We were looking for a long-run relationship and a short-run relationship. From this model, a survival analysis, and a couple of other techniques we were able to determine the optimal time to set the expiration date on a card. I applied this technique because I was afraid I was getting the wrong answer, namely because the number of transactions and number of fraud transactions were cointegrated.(All of the models I tried gave similar results which was comforting). Given that we are talking about fraud strategies I don't feel comfortable giving more details than this, hopefully, that gives you a flavor of what I was trying to accomplish. Basically, I was trying to do a robustness check.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It isn't easily applied in a business context because this model isn't intuitive, it isn't easily explained, and it takes great skill to interpret it. The &quot;mathematical&quot; answer probably has something to go like this, ask an average data scientist what the impulse-response function for their model looks like and you will get a blank stare. Ask them what the FEVD looks like for their model, and you will get a blank stare. Data scientists tend to be trained in non-linear models (read machine learning), not cointegrated time-series models. It isn't that they aren't capable, they've just never been trained to think in those terms. I know technically not math, and technically that is an opinion but it is probably the truth.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also doing the math, I can probably get a better fit out of a non-linear model like an LSTM network(?). So if I only care about pure prediction, it wouldn't be worth my time. Where VAR and VECM shine is that although complicated, they are essentially linear, and therefore, fairly interpretable. So perhaps what you are looking for is a business setting that requires causally valid, interpretable, multivariate time-series models. If you look hard enough, I'm sure you can come up with a couple of interelated KPIs being tracked in your business where it would make sense.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But in my experience, if you need a fast dirty answer, forgive me econometrics teachers everywhere, you can just plug the differenced series into a linear regression. (Warning not best practices, but it does alright).&lt;/p&gt;&#xA;" OwnerUserId="16538" LastActivityDate="2018-06-01T22:00:09.260" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32520" PostTypeId="2" ParentId="32515" CreationDate="2018-06-01T22:03:19.950" Score="1" Body="&lt;p&gt;Vector autoregression, just like plain autoregression, applies to linear problems. Linearity is an assumption that barely applies to everyday scenarios, especially with disturbance sources, such as human sentiment, being nonlinear.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Models that capture temporal nonlinearities, such as LSTM-RNNs, are much more useful in such problems. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;A business application of VAR that I can think of would be to forecast the sales of a product based on its past values.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-01T22:03:19.950" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32521" PostTypeId="2" ParentId="32478" CreationDate="2018-06-01T22:27:11.963" Score="1" Body="&lt;p&gt;&lt;strong&gt;Yes&lt;/strong&gt; you can do sensor fusion, given that you have access to ground truth during training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You should not try to model the errors with your RNN. Instead you should directly build a MISO RNN model (multiple inputs -&gt; your sensors | single output -&gt; the estimation of ground truth).&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2018-06-01T23:16:43.153" LastActivityDate="2018-06-01T23:16:43.153" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32525" PostTypeId="2" ParentId="28000" CreationDate="2018-06-02T04:53:32.840" Score="1" Body="&lt;p&gt;I think most likely the return of your model wouldn’t be worth it given the amount of effort to generalize it enough. &lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;You may have millions of users, but each users need will probably too unique for generalization. I.e., everyone’s commute is so different that what you have learned from other users are probably not applicable to other users. (Unless during rush hours where most people are heading to the core business areas. You don’t need a model for that. )&lt;/li&gt;&#xA;&lt;li&gt;The trained model will probably be marginally better than logging the users usage history. It probably isn’t worth the effort to train and process these data for what you can gain. &lt;/li&gt;&#xA;&lt;li&gt;Recent locations is probably good enough for most users and super easy to implement. Your model will probably have a hard time to predict the odd unusual trips anyway. &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;You are probably better off storing the users current location and query the most likely destination. Or simply look at all the popular trip destinations from that location. A data base with proper index should be able to handle that &lt;/p&gt;&#xA;" OwnerUserId="52719" LastActivityDate="2018-06-02T04:53:32.840" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32527" PostTypeId="2" ParentId="32508" CreationDate="2018-06-02T06:40:27.743" Score="2" Body="&lt;p&gt;I think that the reason for this to happen is that tree-based methods have problems with linear problems. This is because tree-based methods do partitions of the variables, and not on combinations of the variables.  To fit a linear regression, a tree-based method will have to do a lot of partitions to obtain low error. However, in principle, using enough deep trees you should be able to overfit your training data, although it might take many trees. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your concern is to make a perfect forecast, no tree based method is able to do a perfect forecast, and this happens with most kind of data. As your data is linear, you happen to be able to do a perfect forecast with linear regression, but this won't happen in real life.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-02T06:40:27.743" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32531" PostTypeId="1" CreationDate="2018-06-02T09:50:00.190" Score="3" ViewCount="2934" Body="&lt;p&gt;Are there any reasons to turn regression into classification - binning continuous targets into classes and then learning model on those instead? (Let's say the algorithm isn't a problem.)  &lt;/p&gt;&#xA;" OwnerUserId="20923" LastEditorUserId="20923" LastEditDate="2018-06-02T10:31:02.267" LastActivityDate="2018-06-03T13:20:45.720" Title="Regression turned into classiffication" Tags="|machine-learning|classification|regression|statistics|" AnswerCount="2" CommentCount="0" ClosedDate="2018-06-05T15:22:40.593" ContentLicense="CC BY-SA 4.0" />
  <row Id="32532" PostTypeId="1" AcceptedAnswerId="32542" CreationDate="2018-06-02T10:19:41.617" Score="1" ViewCount="155" Body="&lt;p&gt;I will evaluate classification models I made. That's logistic regression and decision tress. &lt;p&gt;&#xA;1. What I use standards for comparison?&lt;br&gt;&#xA;2. Suppose model selection's standard is ASE. One is high ASE of training data, low ASE of test data, and the other is ASE of training data is low and ASE of test data is high.. If you select a model, which models do you choose?&lt;/p&gt;&#xA;" OwnerUserId="53062" LastActivityDate="2019-03-13T07:57:10.073" Title="How can I evaluate data mining model?" Tags="|data-mining|regression|decision-trees|" AnswerCount="3" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32533" PostTypeId="2" ParentId="32531" CreationDate="2018-06-02T11:13:31.807" Score="2" Body="&lt;p&gt;You actually convert the output of your algorithm from &lt;strong&gt;continuous&lt;/strong&gt; to &lt;strong&gt;categorical&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I see many reasons of why you would want to do that. A simple case for this would be when you have &lt;strong&gt;very long time-series data&lt;/strong&gt; that take a lot of space to be saved. In this case, it is convenient to convert the timeseries in a &lt;strong&gt;histogram representation&lt;/strong&gt;, which actually is a number of predefined bins where all the values fall into. This will greatly reduce the memory requirements of your algorithm because you just increase the count of a bin (integer number) when a measurement is taken that falls in it, instead of storing the measurement itself as a float number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This of course has pitfalls, such as losing the &quot;time&quot; dependencies between the measurements as well as reduction in resolution of your dataset. BUT, it can be really useful when forecasting measurements because instead of having the continuous R space as output of a predictor, you actually have (e.g.) 10 bins (classes) that the next measurement is predicted to fall into, making it easier for your model to be trained.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;FYI, this is exactly what is done in the automotive industry.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;EDIT - (Added sources)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Sources:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://people.dsv.su.se/~tony/papers/dmin_2015.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://people.dsv.su.se/~tony/papers/dmin_2015.pdf&lt;/a&gt; (check the algorithm)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.phmsociety.org/node/2284&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.phmsociety.org/node/2284&lt;/a&gt; (the previous algorithm is applied here for trucks)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://link.springer.com/article/10.1007/s10618-017-0538-6&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://link.springer.com/article/10.1007/s10618-017-0538-6&lt;/a&gt; (another algorithm based on histograms of truck data)&lt;/p&gt;&#xA;" OwnerUserId="52089" LastEditorUserId="52089" LastEditDate="2018-06-02T11:36:52.303" LastActivityDate="2018-06-02T11:36:52.303" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32534" PostTypeId="2" ParentId="32532" CreationDate="2018-06-02T11:20:21.203" Score="0" Body="&lt;p&gt;The standards for evaluation of classification accuracy are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;F-score (usually &lt;a href=&quot;https://en.wikipedia.org/wiki/F1_score&quot; rel=&quot;nofollow noreferrer&quot;&gt;F1&lt;/a&gt; score, giving same importance to precision and recall)&lt;/li&gt;&#xA;&lt;li&gt;AUC - &lt;a href=&quot;https://en.wikipedia.org/wiki/Receiver_operating_characteristic&quot; rel=&quot;nofollow noreferrer&quot;&gt;ROC&lt;/a&gt; for binary classification (check &lt;a href=&quot;https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it&quot;&gt;this&lt;/a&gt;)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-02T11:20:21.203" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32537" PostTypeId="2" ParentId="32508" CreationDate="2018-06-02T13:53:41.910" Score="1" Body="&lt;p&gt;(adding to what's said above by @David),&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The short answer is that,&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;You can't expect the tree based models to &lt;strong&gt;Extrapolate&lt;/strong&gt;...&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Had asked on Slack (quoting miguel_perez)and this was the reply, realize that in your example you are aproximating a line with an staircase. Even discarding other errors suspect number one would be not enough data points. Trees are just not the proper tool to approximate lines especially with not enough data... &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or you don't have enough features to do the same...(cpmp)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also if you want to do regression only, then we have different regressors, Vowpal Wabbit, KNN etc...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-06-03T01:42:32.783" LastActivityDate="2018-06-03T01:42:32.783" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32541" PostTypeId="1" AcceptedAnswerId="32595" CreationDate="2018-06-02T15:53:00.123" Score="1" ViewCount="4465" Body="&lt;p&gt;Do we have an equivalence for the following question, but with tensorflow : &lt;a href=&quot;https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras&quot;&gt;How to set class weights for imbalanced classes in Keras?&lt;/a&gt; ??&lt;/p&gt;&#xA;" OwnerUserId="53068" LastActivityDate="2018-06-04T10:06:14.693" Title="How to set class weights for imbalanced classes in Tensorflow?" Tags="|machine-learning|tensorflow|" AnswerCount="1" CommentCount="1" ClosedDate="2018-06-05T06:40:29.890" ContentLicense="CC BY-SA 4.0" />
  <row Id="32542" PostTypeId="2" ParentId="32532" CreationDate="2018-06-02T16:06:57.927" Score="1" Body="&lt;ul&gt;&#xA;&lt;li&gt;Accuracy (for classification problems)&lt;/li&gt;&#xA;&lt;li&gt;Precision&lt;/li&gt;&#xA;&lt;li&gt;Recall&lt;/li&gt;&#xA;&lt;li&gt;F1 Score&lt;/li&gt;&#xA;&lt;li&gt;AUC-ROC, particularly for imbalanced datasets &lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Good performance on the training set and bad performance on the test set is due to overfitting. So you should try to find ways to tackle overfitting, such as regularization of parameters, parameter tuning using cross-validation etc... &lt;/p&gt;&#xA;" OwnerUserId="35679" LastActivityDate="2018-06-02T16:06:57.927" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32544" PostTypeId="1" CreationDate="2018-06-02T16:53:15.933" Score="0" ViewCount="152" Body="&lt;p&gt;I have a dataset having text documents as each entry , the documents are of various ethical languages which are not specified. How should be my strategy if I want to summarise the content of the dataset.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For eg. Doc1 : &quot;Ruby is a dog lover.Ruby loves her dogs so much. Ruby has a pet whose name is a dog&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now summarising it is easy as &quot;Ruby loves dog.Ruby has a dog&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But How should be the approach , when the dataset also has documents in french, chinese, hindi etc...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Some of the problems are: there are cases of code switch, and I don't know what are the languages in the dataset before hand.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What I am guessing at this is to may be change all the dataset to english language and then do the tasks? In case if converting to english is a good choice then which library would be better?&lt;/p&gt;&#xA;" OwnerUserId="52997" LastActivityDate="2018-12-31T09:01:40.320" Title="Multilingual data handling for text summarisation" Tags="|deep-learning|nlp|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32546" PostTypeId="1" AcceptedAnswerId="32554" CreationDate="2018-06-02T17:27:11.933" Score="1" ViewCount="80" Body="&lt;p&gt;I fail to understand as to how learning rate is used in XGBoost?&#xA;Can anyone explain using a numerical example?&lt;/p&gt;&#xA;" OwnerUserId="52726" LastActivityDate="2018-07-04T22:36:29.187" Title="Numerical example of Confusion in understanding learning rate in xgboost" Tags="|xgboost|" AnswerCount="2" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32548" PostTypeId="2" ParentId="20157" CreationDate="2018-06-02T18:30:27.863" Score="3" Body="&lt;p&gt;if &lt;code&gt;yHat&lt;/code&gt; are your predictions and &lt;code&gt;yval&lt;/code&gt; are your y true then&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;tp = sum((yHaT == 1) &amp;amp; (yval == 1));&#xA;fp = sum((yHaT == 1) &amp;amp; (yval == 0));&#xA;fn = sum((yHaT == 0) &amp;amp; (yval == 1));&#xA;&#xA;precision = tp / (tp + fp);&#xA;recall = tp / (tp + fn);&#xA;F1 = (2 * precision * recall) / (precision + recall);&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;" OwnerUserId="53075" LastActivityDate="2018-06-02T18:30:27.863" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32549" PostTypeId="2" ParentId="32544" CreationDate="2018-06-02T18:37:04.347" Score="1" Body="&lt;p&gt;There are two basic approaches to summarization: &lt;strong&gt;abstractive&lt;/strong&gt; and &lt;strong&gt;extractive&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Extractive&lt;/strong&gt; summarization is simpler and consists of selecting informative pieces/sentences from text. It is commonly performed using algorithms like TextRank (a variation of PageRank). An example implementation can be found in &lt;a href=&quot;https://rare-technologies.com/text-summarization-with-gensim/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt;. There are other approaches, for example check out &lt;a href=&quot;https://github.com/miso-belica/sumy&quot; rel=&quot;nofollow noreferrer&quot;&gt;sumy&lt;/a&gt; library.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Abstractive&lt;/strong&gt; summarization on the other hand consists of summarizing documents with shorter sentences which do not have to come from the input text (an abstract may paraphrase some fragments of text). It is commonly performed using Deep Learning Seq2Seq methods.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Which approach is better for start? &lt;strong&gt;Extractive&lt;/strong&gt; wins by far. First of all, you don't have to train any model, as these methods rely on information found solely in summarized text. Second, deep learning models, even if they exist and can be easily found *, are trained on specific texts, so they might not generalize well across application domains.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;But How should be the approach , when the dataset also has documents&#xA;  in french, chinese, hindi etc...&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;If you have good tokenizers and stemmers for these languages you should be fine with TextRank, as it only relies on word counts (I'm not sure about Chinese though). BTW language detection is by far easier than summarization - just count number of stopwords of different languages that document contains.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;*They're not, for example Google published a &lt;a href=&quot;https://ai.googleblog.com/2016/08/text-summarization-with-tensorflow.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;paper&lt;/a&gt; sometime ago on the topic and didn't even disclose model, since the training dataset was proprietary. I was trying to find something for abstractive summarization sometime ago but I couldn't find anything that had good pretraining and worked out of the box.&lt;/p&gt;&#xA;" OwnerUserId="45064" LastEditorUserId="45064" LastEditDate="2018-06-02T18:47:11.753" LastActivityDate="2018-06-02T18:47:11.753" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32550" PostTypeId="1" CreationDate="2018-06-02T19:54:21.740" Score="1" ViewCount="29" Body="&lt;p&gt;For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In other words:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\vec q=W\cdot \vec x$$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$\vec h = activate(drop(\vec q))$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;or:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;span class=&quot;math-container&quot;&gt;$$\vec q=W\cdot (drop(\vec x)) $$&lt;/span&gt;&#xA;&lt;span class=&quot;math-container&quot;&gt;$$ \vec h = activate(\vec q)$$&lt;/span&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think the second variant is smoother (none of our current vector is fully dropped out, but is assembled from a mix of the dropped-out &lt;em&gt;input&lt;/em&gt;) and is therefore softer.&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="1330" LastEditDate="2021-07-20T14:08:25.687" LastActivityDate="2021-07-20T14:08:25.687" Title="Dropout onto pre-weighted vs onto pre-activated vector?" Tags="|mathematics|dropout|" AnswerCount="0" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32551" PostTypeId="1" AcceptedAnswerId="32561" CreationDate="2018-06-02T21:25:42.540" Score="1" ViewCount="82" Body="&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/oQmYo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/oQmYo.png&quot; alt=&quot;Median income Histogram&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As per the figure &lt;a href=&quot;https://i.stack.imgur.com/oQmYo.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000.  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;I didn't understand the explanation behind why housing['median_income'] has to be divide by 1.5&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; housing['income_cat'] = np.ceil(housing['median_income'] / 1.5)&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Explanation - It is important to have a sufficient number of instances in your dataset for each stratum, or else the estimate of the stratum's importance may be biased.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Can someone help me to understand the explanation and also why is it only 1.5 ?. As per the explanation, why the estimate of stratum's importance will be biased when there is no adequate instances for each category ? &lt;/p&gt;&#xA;" OwnerUserId="2893" LastActivityDate="2018-06-03T08:45:22.770" Title="Why is it important to have sufficient number of instances in your dataset for each stratum?" Tags="|python|pandas|preprocessing|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32553" PostTypeId="2" ParentId="31173" CreationDate="2018-06-02T22:58:17.177" Score="1" Body="&lt;p&gt;I want to clarify that the fact that you can pick only 100 boxes from the test dataset is NOT a limitation (like you describe it). You just need to see it from this perspective:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How to pick the 100 boxes (out of 500K) that MOST LIKELY contain gold?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;A very suitable algorithm would be a Decision Tree classifier. By using the Information Gain (based on Entropy Reduction), it can learn which features can split the dataset more accurately into two classes, i.e. the one that contains gold and the one that doesn't, and neglect (prune) the weakest attributes. After that, you will not only have a trained classifier but also gain insight on which features are the most important to make the decision (very important information).&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-02T22:58:17.177" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32554" PostTypeId="2" ParentId="32546" CreationDate="2018-06-03T01:37:25.383" Score="1" Body="&lt;p&gt;Each iteration is supposed to provide an improvement to the training loss.&#xA;Such improvement is multiplied with the learning rate in order to perform smaller updates.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Smaller updates allow to overfit slower the data, but requires more iterations for training.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For instance, doing 5 iteations at a learning rate of 0.1 approximately would require doing 5000 iterations at a learning rate of 0.001, which might be obnoxious for large datasets.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Typically, we use a learning rate of 0.05 or lower for training, while a learning rate of 0.10 or larger is used for tinkering the hyperparameters...&lt;/p&gt;&#xA;" OwnerUserId="35644" LastActivityDate="2018-06-03T01:37:25.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32556" PostTypeId="1" AcceptedAnswerId="32558" CreationDate="2018-06-03T04:51:35.743" Score="0" ViewCount="100" Body="&lt;p&gt;I'm doing research where a part of the collected data is of Ordinal type. I will implement ANN with Logistic Regression function in the Activation function. What I have learnt from documents of other websites as well as an answer in &lt;a href=&quot;https://datascience.stackexchange.com/&quot;&gt;https://datascience.stackexchange.com/&lt;/a&gt;, the target value is of ordinal type while the independent variables are of ratio or interval type. But my independent data is of Ordinal type and the target data will be label (say Like or Unlike).&#xA;How should I build a function for ANN if I'm not wrong in my understanding? &lt;/p&gt;&#xA;" OwnerUserId="40137" LastActivityDate="2018-06-04T12:07:51.080" Title="Regression equation for ordinal data" Tags="|machine-learning|logistic-regression|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32557" PostTypeId="1" AcceptedAnswerId="32563" CreationDate="2018-06-03T05:13:09.600" Score="28" ViewCount="35772" Body="&lt;p&gt;I'm studying machine learning from Andrew Ng Stanford lectures and just came across the theory of VC dimensions. According to the lectures and what I understood, the definition of VC dimension can be given as, &lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify all possible $2^n$ labeling correctly) and you cannot find any set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not separate all points correctly), then the VC dimension is $n$. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Also Professor took an example and explained this nicely. Which is: &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$H=\{{set\ of\ linear\ classifiers\ in\ 2\ Dimensions \}}$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then any 3 points can be classified by $H$ correctly with separating hyper plane as shown in the following figure. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/iPJBN.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/iPJBN.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;And that's why the VC dimension of $H$ is 3. Because for any 4 points in 2D plane, a linear classifier can not shatter all the combinations of the points. For example, &lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/U7xVB.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/U7xVB.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this set of points, &lt;strong&gt;there is no separating hyper plane can be drawn to classify this set. So the VC dimension is 3.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I get the idea till here. But what if we've following type of pattern?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/4nLut.png&quot; rel=&quot;noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/4nLut.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Or the pattern where a three points coincides on each other, Here also we can not draw separating hyper plane between 3 points. But still this pattern is not considered in the definition of the VC dimension. Why? The same point is also discussed the lectures I'm watching &lt;a href=&quot;https://www.youtube.com/watch?v=0kWZoyNRxTY&amp;amp;index=10&amp;amp;list=PLA89DCFA6ADACE599&quot; rel=&quot;noreferrer&quot;&gt;Here at 16:24&lt;/a&gt; but professor does not mention the exact reason behind this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Any intuitive example of explanation will be appreciated. Thanks&lt;/p&gt;&#xA;" OwnerUserId="53083" LastActivityDate="2021-02-20T19:06:59.970" Title="What is the exact definition of VC dimension?" Tags="|machine-learning|classification|vc-theory|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32558" PostTypeId="2" ParentId="32556" CreationDate="2018-06-03T07:15:03.807" Score="0" Body="&lt;p&gt;I think that what is influenced by the type of your independent variables is not the structure of your neural network but the encoding of your data. If your independent variables are ordinal, then you can map them to integer numbers or other ordered sets (and maybe normalize them to make them between 0 and 1). &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If your target variable is a label, it is a classification problem. Therefore, the neurons in the last layer in your neural network should have an activation function optimized for classification problems, like sigmoid. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;You also need a specific loss function, like binary crossentropy. There was a &lt;a href=&quot;https://link.springer.com/chapter/10.1007/978-3-642-01818-3_25&quot; rel=&quot;nofollow noreferrer&quot;&gt;case study&lt;/a&gt;, however, that showed that for ordinal classification (when your target labels are ordinal), mean squared error works well.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You can take any activation functions for all hidden layers in your network as long as they are non-linear. The choice of a particular activation function will influence the rate of convergence of your learning algorithm but I don't see how a particular activation function in hidden layers may be preferable for a certain type of input variables, like if they are ordinal or not.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastEditorUserId="51836" LastEditDate="2018-06-04T12:07:51.080" LastActivityDate="2018-06-04T12:07:51.080" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32559" PostTypeId="1" AcceptedAnswerId="32566" CreationDate="2018-06-03T07:24:29.577" Score="0" ViewCount="89" Body="&lt;p&gt;I'm planning on training a CNN on CT scans for classification. The problem is CT scans are taken slice by slice, and in a typical scan, there could be more than 200 slices. The number of slices in a scan isn't uniform and depend on the scanning machine and age of the person(for whom the scan is taken).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;1)How should I make the number of slices uniform for feeding to a deep learning network?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This sorta problem is handled in NLP by padding a chosen vector( or something similar) to sentences which have lengths less than the predefined length and truncating sentences which have lengths greater than the preset length.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2)Can a similar approach be used to make slices(timesteps) uniform or is there a better way?&lt;/p&gt;&#xA;" OwnerUserId="39208" LastEditorUserId="39208" LastEditDate="2018-06-03T11:24:17.730" LastActivityDate="2022-03-24T10:55:47.870" Title="Computer Vision: Handling dataset(3D data or scan) with different timesteps" Tags="|deep-learning|cnn|lstm|preprocessing|computer-vision|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32560" PostTypeId="2" ParentId="32325" CreationDate="2018-06-03T07:51:45.203" Score="1" Body="&lt;p&gt;You need to define a distance function that yields the desirable output.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Usually it will be enough if you can construct it such that d(a,b)for your purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Doc2vec vectors are a bit tricky because they have so many dimensions, and a very strange geometry. It doesn't even seem to be clear whether cosine or Euclidean is to be used on these vectors...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Either way, you have to &lt;em&gt;carefully balance&lt;/em&gt; the different features. In the other answer, minmax or stddev scaling was proposed. On one hand, that likely ruins the doc2vec properties, on the other hand this will but much more weight on the word vectors than on the other attributes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For DBSCAN, you can also follow the &quot;Generalized DBSCAN&quot; approach. Here, the idea is just to define different thresholds for different features. Then neighbors must satisfy all thresholds. I.e. have doc2vec cosine less than A, and other features' distance less than B.&#xA;This is likely easier than balancing them as factors inside a single distance function.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;But nothing saves you from weighting different feature (sets} carefully.&lt;/p&gt;&#xA;" OwnerUserId="924" LastActivityDate="2018-06-03T07:51:45.203" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32561" PostTypeId="2" ParentId="32551" CreationDate="2018-06-03T08:45:22.770" Score="1" Body="&lt;p&gt;Lets take an example that you are trying to evaluate average income of a company. Assume company has 100 employees in total and they belong to two stratum (70 management and 30 technical executives) and management executives on average are paid higher compared to technical executives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You are the surveyor:&#xA;Due to resource constraints, you have access to survey of any 20 individuals chosen at random. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Case 1: you did a random sampling from 100 employees and end up with 10 management and 10 technical executives. you will get an estimate for average income that is biased towards technical executives. In other words, you are giving more importance/ representation to income of technical executives than&#xA;actual representation in the population.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Similarly, in other attempt:&#xA;Case 2: you did a random sampling from 100 employees and end up with 17 management and 3 technical executives. you will get an estimate for average income that is biased towards management executives. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Both the above sampled cases have bias and are not representative of the population. Therefore, when we have the stratum identified before the sampling, stratified sampling should be done. For eg. random sample for 14 out of 70 management executives and  6 out of 30 technical executives.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In your given example of the housing and income data, income scale has been shrunk by a factor of 1.5, so as to create less no of income categories for the same bin width. Any other factor &gt; 1 could also be used to ensure each stratum has considerable number of instances. &lt;/p&gt;&#xA;" OwnerUserId="51659" LastActivityDate="2018-06-03T08:45:22.770" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32563" PostTypeId="2" ParentId="32557" CreationDate="2018-06-03T08:50:46.693" Score="27" Body="&lt;p&gt;The definition of VC dimension is: if &lt;strong&gt;there exists&lt;/strong&gt; a set of n points that can be shattered by the classifier and &lt;strong&gt;there is no&lt;/strong&gt; set of n+1 points that can be shattered by the classifier, then the VC dimension of the classifier is n. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The definition does not say: if &lt;strong&gt;any set&lt;/strong&gt; of n points can be shattered by the classifier...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If a classifier's VC dimension is 3, it does not have to shatter &lt;strong&gt;all possible&lt;/strong&gt; arrangements of 3 points. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If of all arrangements of 3 points you can find &lt;strong&gt;at least one&lt;/strong&gt; such arrangement that can be shattered by the classifier, and cannot find 4 points that can be shattered, then VC dimension is 3.&lt;/p&gt;&#xA;" OwnerUserId="51836" LastActivityDate="2018-06-03T08:50:46.693" CommentCount="9" ContentLicense="CC BY-SA 4.0" />
  <row Id="32565" PostTypeId="1" CreationDate="2018-06-03T09:35:39.720" Score="1" ViewCount="110" Body="&lt;p&gt;I want to clarify point in random forest literature. In random forest we select random features from a subset of data, i.e it is subset of features from subset of data for a single tree. But when we test the data, let's take a test sample 'v' from the test data. The data has say '5' features.&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Tree one was constructed using features 3 and 2.&lt;/li&gt;&#xA;&lt;li&gt;Tree two was constructed using features 2,3, and 5.&lt;/li&gt;&#xA;&lt;li&gt;Tree three was constructed using features 1,2 and 3.&lt;/li&gt;&#xA;&lt;li&gt;Tree four was constructed using features 1,4,and 5.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;at leaf node we will have probability showing to which class does the 'v' point belongs.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My question is : Will each forest choose same features from the test data sample 'v' as it used when the tree was constructed. ?&lt;br&gt;&#xA;for example will tree one will use features 3 and 2. and tree two use feature 2, 3 and 5. respectively&lt;/p&gt;&#xA;" OwnerUserId="38506" LastActivityDate="2018-06-04T07:50:41.243" Title="testing of data in random forest?" Tags="|machine-learning|random-forest|algorithms|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32566" PostTypeId="2" ParentId="32559" CreationDate="2018-06-03T11:58:55.940" Score="1" Body="&lt;p&gt;I would use an LSTM-RNN to &lt;strong&gt;encode&lt;/strong&gt; sequences of &lt;strong&gt;arbitrary&lt;/strong&gt; length (length meaning the number of slices) into &lt;strong&gt;fixed-size&lt;/strong&gt; output. It should be very easy to implement it in Keras :)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Then feed the sequences in your CNN.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Check sequence-to-sequence encoder-decoder scheme, like &lt;a href=&quot;https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The same concept is applied with state-of-the-art algorithms that require fixed size inputs, like &lt;a href=&quot;https://arxiv.org/abs/1710.09207&quot; rel=&quot;nofollow noreferrer&quot;&gt;this one&lt;/a&gt;.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-03T11:58:55.940" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32567" PostTypeId="1" AcceptedAnswerId="32596" CreationDate="2018-06-03T12:53:55.550" Score="0" ViewCount="548" Body="&lt;p&gt;I would like to construct system that would suggest user ingredients once he/she inputs title of the recipe.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that this is the task of machine learning or AI, but on the other hand I am pretty new to ML and generally in AI development and I feel kinda lost, since I don't know where to start and with what to start.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So far, I've managed to crawl data from web page with recipe title and ingredients which are used for this recipes.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;My thoughts are that, this problem may be solved with suggester system, but on the other hand I think classification / clustering algorithms may be used to divide recipes in clusters / categories and once input title is associated to cluster ingredients may be generated from that cluster, but I don't know which is best solution and I wonder if there may be other ones.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Thanks in advance!&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;h3&gt;I'm posting my temporary solution&lt;/h3&gt; which may help others solve similar problems and I think this is pretty basic, but works nice:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;import all data to database (postgres in my case), where Recipe table has only name and Ingredient table has name and ForeignKey to recipe&lt;/li&gt;&#xA;&lt;li&gt;once user inputs recipe name (&lt;code&gt;rname&lt;/code&gt;), I run query using postgres' Trigram Similarity module to detect similarity between &lt;code&gt;rname&lt;/code&gt; and recipe table names. Then I filter out recipes which similarity is greater than .1 (for example)&lt;/li&gt;&#xA;&lt;li&gt;After that, I get all ingredients which are linked to filtered recipes, annotate ingredient usage for individual ingredient (calculate ingredient usage ratio). And order by usage ratio &lt;code&gt;DESC&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;Finally, I return ingredients (10 mostly used ones)&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;I think, looking through proposed solutions and also posting them. &lt;/p&gt;&#xA;" OwnerUserId="53088" LastEditorUserId="53088" LastEditDate="2018-06-05T17:16:30.007" LastActivityDate="2018-06-05T17:16:30.007" Title="suggest ingredients based on recipe title" Tags="|machine-learning|python|deep-learning|supervised-learning|semi-supervised-learning|" AnswerCount="1" CommentCount="2" ClosedDate="2018-06-05T15:26:01.697" ContentLicense="CC BY-SA 4.0" />
  <row Id="32568" PostTypeId="2" ParentId="32531" CreationDate="2018-06-03T13:20:45.720" Score="0" Body="&lt;p&gt;While doing this might simplify your analysis, it is not a recommended approach.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Let’s use an example. Suppose that you are using a series of explanatory variables (&lt;strong&gt;X1, X2, X3&lt;/strong&gt;) to estimate &lt;strong&gt;car sales (in US$)&lt;/strong&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Now, suppose that our variable for car sales is interval, i.e. we have a series of car sales figures (25000, 37500, 3000, 71000…), etc.&#xA;Let’s say you were to convert your dependent variable to a categorical one. e.g. &lt;strong&gt;&gt;25000 = 1, &amp;lt;25000 = 0.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By doing this, you would lose a lot of information from your dependent variable and your model would have significantly less capacity to quantify unit effects of each explanatory variable on the dependent one.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;This is why when binary logistic regressions are run, it is often recommended that a minimum of &lt;strong&gt;500&lt;/strong&gt; observations should be used to induce a significant level of variation in the dependent variable to analyse the effects of such variation (Studenmund, 2010). Additionally, it is for this reason that traditional measures of fit in a regression model such as R-Squared become invalid when analysing a dataset with a categorical dependent variable.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;strong&gt;Categorical variables – or classes&lt;/strong&gt; – are used in situations where it is not possible to use an interval variable to quantify a particular condition. e.g. suppose you are a medical researcher constructing a model to determine the presence of diabetes in a particular patient. Now, either a person has diabetes or he does not (diabetes = 1, no diabetes = 0). Therefore, your model is not “losing” information because the dependent variable is providing all the information necessary for processing the model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;By discarding information, you risk increasing the possibility of a &lt;strong&gt;Type 1&lt;/strong&gt; error – where you reject a true null hypothesis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In summary, it depends on the data you are working with. In some instances, the dependent variable might make more sense if it were expressed categorically. However, there are many other instances where it is not a recommended approach.&lt;/p&gt;&#xA;" OwnerUserId="32926" LastActivityDate="2018-06-03T13:20:45.720" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32569" PostTypeId="1" AcceptedAnswerId="32592" CreationDate="2018-06-03T16:10:15.793" Score="1" ViewCount="238" Body="&lt;p&gt;Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook &quot;Hands on machine learning&quot;.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt;&amp;gt;&amp;gt; housing['income_cat'].value_counts() &#xA; &amp;gt;&amp;gt;&amp;gt; 3.0    7236&#xA;     2.0    6581&#xA;     4.0    3639&#xA;     5.0    2362&#xA;     1.0     822&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;If I am not wrong, the above step is to get the counts for each class. For example, for class '3' there are 7236 instances. Likewise, for class '2' there are 6581 instances. &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt;  housing['income_cat'].value_counts / len(housing) &#xA;&amp;gt;&amp;gt;&amp;gt;  3.0    0.350581&#xA;     2.0    0.318847&#xA;     4.0    0.176308&#xA;     5.0    0.114438&#xA;     1.0    0.039826&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Next, I was not clear, what was the intention behind the above step. By doing the above step, what am I suppose to learn ?. and&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt; &amp;gt;&amp;gt;&amp;gt; from sklearn.model_selection import StratifiedShuffleSplit&#xA;&#xA; &amp;gt;&amp;gt;&amp;gt; split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)&#xA; &amp;gt;&amp;gt;&amp;gt; for train_index, test_index in split.split(housing, housing[&quot;income_cat&quot;]):&#xA;         strat_train_set = housing.loc[train_index]&#xA;         strat_test_set = housing.loc[test_index]&#xA;&#xA; &amp;gt;&amp;gt;&amp;gt;  strat_test_set['income_cat'].value_counts() / len(strat_test_set)&#xA; &amp;gt;&amp;gt;&amp;gt;  3.0    0.350533&#xA;      2.0    0.318798&#xA;      4.0    0.176357&#xA;      5.0    0.114583&#xA;      1.0    0.039729&#xA;      Name: income_cat, dtype: float64&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;How come &lt;code&gt;strat_test_set['income_cat'].value_counts() / len(strat_test_set)&lt;/code&gt; results are almost same to the results of &lt;code&gt;housing['income_cat'].value_counts / len(housing)&lt;/code&gt; ?&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="2893" LastEditorUserId="2893" LastEditDate="2018-06-03T16:54:35.267" LastActivityDate="2018-06-04T16:48:07.743" Title="Need help in understanding a small example" Tags="|python|pandas|preprocessing|" AnswerCount="3" CommentCount="0" FavoriteCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32570" PostTypeId="2" ParentId="32569" CreationDate="2018-06-03T16:23:40.513" Score="1" Body="&lt;p&gt;The answer is pretty clear as you answered your question yourself,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Also check out the normalise Param in value_counts..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;What stratified sampling does is it keeps the actual percentage of each same in different folds as that in the original dataset...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(That's why they both are same..)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Why's it's done that way?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It's done so that you can reduce the overall bias in each of your sample and no one dominates over another..&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Think of a worst case scenario where you have extracted few rows and all have the same value in a particular column...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It will be disastrous for the model as it hasn't seen anything else except that value in that fold...&lt;/p&gt;&#xA;&#xA;&lt;p&gt;(You can refer to the sklearn docs for &lt;a href=&quot;http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;Stratified Sampling..&lt;/a&gt;)&lt;/p&gt;&#xA;" OwnerUserId="35644" LastEditorUserId="35644" LastEditDate="2018-06-03T17:46:47.367" LastActivityDate="2018-06-03T17:46:47.367" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32572" PostTypeId="1" AcceptedAnswerId="32574" CreationDate="2018-06-03T18:25:00.723" Score="1" ViewCount="153" Body="&lt;p&gt;I build model that on input have correct word. On output there is possible word written by human (it contain some errors). My training dataset looks that:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input - output  &#xA;hello - helo  &#xA;hello - heelo  &#xA;hello - hellou  &#xA;between - betwen  &#xA;between - beetween  &#xA;between - beetwen  &#xA;between - bettwen  &#xA;between - bitween&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;etc.&#xA;During preprocessing I add a measure of the distortion of a word. Then I hardcoding letters for numbers.&#xA;My current model's using CNN. The number of neurons of input is the same as the longest word in training dataset and the number of neurons of output is the same as the longest word in traning dataset. &#xA;This model doesn't work as I excepted. Word on the output is not look as I except.&#xA;eg.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;input - output&#xA;house - gjrtdd&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Question:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;How can I build/improve model for this task? Is CNN a good idea? What other methods can I use for this task?&lt;/p&gt;&#xA;" OwnerUserId="39590" LastEditorUserId="29169" LastEditDate="2021-02-13T04:19:22.440" LastActivityDate="2021-02-13T04:19:22.440" Title="ML model to transform words" Tags="|neural-network|nlp|convolutional-neural-network|" AnswerCount="2" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="32574" PostTypeId="2" ParentId="32572" CreationDate="2018-06-03T19:47:31.977" Score="1" Body="&lt;p&gt;Try a totally different approach, using &lt;a href=&quot;https://en.wikipedia.org/wiki/Generative_adversarial_network&quot; rel=&quot;nofollow noreferrer&quot;&gt;Generative Adversarial Networks&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For this purpose you need:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;A Generator &lt;/li&gt;&#xA;&lt;li&gt;A Discriminator &lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;See the scheme (credit O'Reilly):&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/xRxPA.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/xRxPA.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The &lt;em&gt;&quot;Real Images&quot;&lt;/em&gt; block in the scheme should be your &lt;em&gt;training dataset&lt;/em&gt; (or ground truth). The Generator should generate the distorted words and the Discriminator should verify if the word is &quot;adequately&quot; distorted, based on a criterion of your choice, which can be any similarity measure between known words (database) and the generated one. Both the Generator and the Discriminator get trained &lt;strong&gt;on-the-go&lt;/strong&gt; while in the training phase and in the end you will have two trained networks, of which the Generator would be very useful for your purpose.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Helpful source: &lt;a href=&quot;https://deeplearning4j.org/generative-adversarial-network&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://deeplearning4j.org/generative-adversarial-network&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-03T19:47:31.977" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32576" PostTypeId="2" ParentId="14187" CreationDate="2018-06-04T04:11:17.447" Score="17" Body="&lt;p&gt;&lt;strong&gt;Hyper-parameters&lt;/strong&gt; are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while &lt;strong&gt;Parameters&lt;/strong&gt; are those which would be learned by the machine like Weights and Biases.&lt;/p&gt;&#xA;" OwnerUserId="53106" LastActivityDate="2018-06-04T04:11:17.447" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32577" PostTypeId="2" ParentId="32480" CreationDate="2018-06-04T04:43:27.900" Score="13" Body="&lt;p&gt;I found very intuitive the explanation of the GAE in the Supplementary material of this paper: &lt;a href=&quot;https://arxiv.org/pdf/1804.02717.pdf&quot; rel=&quot;noreferrer&quot;&gt;DeepMimic&lt;/a&gt;. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!&lt;/p&gt;&#xA;" OwnerUserId="18626" LastEditorUserId="44965" LastEditDate="2019-02-25T22:49:42.797" LastActivityDate="2019-02-25T22:49:42.797" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32578" PostTypeId="1" CreationDate="2018-06-04T05:54:29.177" Score="1" ViewCount="1236" Body="&lt;p&gt;I am looking for a machine learning algorithm for my problem.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a set of sentences like,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[&quot;The cat in the hat disabled&quot;, &quot;A cat is a fine pet ponies.&quot;, &quot;Dogs and cats make good pets.&quot;,&quot;I haven't got a hat.&quot;]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;and the search-words like,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[&quot;cat&quot;,&quot;hat&quot;]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I want to convert my sentence list and search-words to a vector space and find matching score between my sentence list and search-word list.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;the type of output I am expecting is,&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;code&gt;[(&quot;The cat in the hat disabled&quot;,0.9), (&quot;A cat is a fine pet ponies.&quot;,0.5), &quot;(Dogs and cats make good pets.&quot;,0.6),(&quot;I haven't got a hat.&quot;,0.49)]&lt;/code&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please suggest a machine learning algorithm for my task, if possible please share a reference link.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;let me know if you have any queries or need more information.&#xA;I am currently using semanticpy for this &lt;a href=&quot;https://github.com/josephwilk/semanticpy&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/josephwilk/semanticpy&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I am getting key-error for many search-words. its not performing stemming and lemmatization for the sentence list but only performing for the search-words list. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Please help on this.&lt;/p&gt;&#xA;" OwnerUserId="42377" LastEditorUserId="64506" LastEditDate="2019-01-31T19:20:40.703" LastActivityDate="2019-01-31T19:20:40.703" Title="Machine learning - Algorithm suggestion for my problem using NLP" Tags="|machine-learning|python|nlp|text-mining|vector-space-models|" AnswerCount="3" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32579" PostTypeId="2" ParentId="32578" CreationDate="2018-06-04T06:01:11.333" Score="2" Body="&lt;p&gt;There is a really good &lt;a href=&quot;https://www.youtube.com/watch?v=ZiKMIuYidY0&amp;amp;index=5&amp;amp;list=PLd1QJ09bWBCIL_e1W-GqfT0OG18KReGWp&amp;amp;t=8244s&quot; rel=&quot;nofollow noreferrer&quot;&gt;video&lt;/a&gt; about this topic from a PyCon in 2016. There is a pretty in-depth description on how to vectorize your sentences as well as make predictions based on those vectors.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I think that would greatly help you out. That is what I used when I was learning about how to perform sentiment analysis. &lt;/p&gt;&#xA;" OwnerUserId="47767" LastActivityDate="2018-06-04T06:01:11.333" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32581" PostTypeId="2" ParentId="26663" CreationDate="2018-06-04T07:45:03.350" Score="11" Body="&lt;p&gt;As for stateful LSTM and its understanding, refer to &lt;a href=&quot;http://philipperemy.github.io/keras-stateful-lstm/&quot; rel=&quot;noreferrer&quot;&gt;here&lt;/a&gt;. Quoting an answer from there:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&quot;&lt;em&gt;I’m given a big sequence (e.g. Time Series) and I split it into smaller sequences to construct my input matrix X. Is it possible that the LSTM may find dependencies between the sequences?&lt;/em&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;No it’s not possible unless you go for the stateful LSTM. Most of the problems can be solved with stateless LSTM so if you go for the stateful mode, make sure you really need it. In stateless mode, long term memory does not mean that the LSTM will remember the content of the previous batches.&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Therefore, stateful is useful &lt;strong&gt;if you wish to save the state&lt;/strong&gt; of the neurons &lt;strong&gt;for the next training&lt;/strong&gt; session instead of resetting it.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;My goal here is to generate sequences that looks like something that&#xA;  the model have seen already.&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Then maybe &lt;a href=&quot;https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/&quot; rel=&quot;noreferrer&quot;&gt;this sequence2sequence (seq2seq)&lt;/a&gt; LSTM encoder-decoder is exactly what you need.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-04T07:45:03.350" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32582" PostTypeId="1" AcceptedAnswerId="32586" CreationDate="2018-06-04T07:45:17.557" Score="0" ViewCount="1352" Body="&lt;p&gt;I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into the scipy.stats.chi2_contingency module. This module returns the chi-squared value and the p-value.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have acheived reasonable results with boolean variables, but I am suspicious of the results for categorical variables with more than 2 categories.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Specifically, I am fairly sure that one continuous feature, age, is correlated with the target, to some level of significance. From plotting histograms and KDEs, I know that the probability distribution of the feature for (target = 0) is quite different from the probability distribution for (target = 1). However, when I bin the age feature into 2-7 bins, the chi-squared test yields a p-value of ~1e-39.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there anything that I am missing with regards to the chi-squared test and categorical variables? Does this test only work for monotonic relationships?&lt;/p&gt;&#xA;" OwnerUserId="52935" LastActivityDate="2018-06-04T08:16:08.217" Title="Chi-squared for continuous variables" Tags="|feature-selection|correlation|beginner|" AnswerCount="1" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32583" PostTypeId="2" ParentId="32565" CreationDate="2018-06-04T07:50:41.243" Score="1" Body="&lt;p&gt;Yes, it &lt;em&gt;has&lt;/em&gt; to use the same featues. If not, the training algorithm does not make any sense.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-04T07:50:41.243" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32584" PostTypeId="2" ParentId="27459" CreationDate="2018-06-04T07:52:42.383" Score="1" Body="&lt;p&gt;I recommend you to read &lt;a href=&quot;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://colah.github.io/posts/2015-08-Understanding-LSTMs/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Basically, the long-term memory and short-term memory are reflected in the structure of the LSTM Network. Please take a look at the diagram from the website I've mentioned. &lt;a href=&quot;https://i.stack.imgur.com/RUC6U.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/RUC6U.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;The long-term memory is constructed in the &quot;bus&quot; on the top, where the previous value can be passed, depends on the $\sigma$ value. The $\sigma$ is between $[0,1]$ and $0$ means- forget the last value, $1$ means- get all (exact) the value from the last state. Sigmas for each cell are established during the training phase.&lt;/p&gt;&#xA;" OwnerUserId="50866" LastActivityDate="2018-06-04T07:52:42.383" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32585" PostTypeId="1" CreationDate="2018-06-04T07:57:48.557" Score="1" ViewCount="272" Body="&lt;p&gt;I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line as a vector of one-hot encoded vectors. I've also considered using a dictionary and one-hot encoding each word in the sentence based on its alphabetical position. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;However, I'm not sure about any of this, as I am new to natural language processing in machine learning. What would be the best way to format my input (and my output) for this problem?&lt;/p&gt;&#xA;" OwnerUserId="53110" LastEditorUserId="38887" LastEditDate="2020-05-08T10:42:28.173" LastActivityDate="2020-05-08T10:42:28.173" Title="How should I format input and output for text generation with LSTMs" Tags="|lstm|nlp|text-generation|" AnswerCount="2" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32586" PostTypeId="2" ParentId="32582" CreationDate="2018-06-04T08:16:08.217" Score="1" Body="&lt;p&gt;It sounds like the chi-squared test confirms your suspicions about age being correlated with the response.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;As far as I am aware, the null-hypothesis for the chi-squared test is &quot;there is no relationship&quot; between the two variables. The test statistic is calculated based on the assumption that all observations are evenly distributed amongst the cells in the contingency table, so the test should work for most kinds of relationships.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;A word of warning - the test is sensitive to data imbalances.&lt;/p&gt;&#xA;" OwnerUserId="45374" LastActivityDate="2018-06-04T08:16:08.217" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32587" PostTypeId="1" AcceptedAnswerId="32594" CreationDate="2018-06-04T08:41:28.733" Score="1" ViewCount="70" Body="&lt;p&gt;I've noticed that in the Andrew Ng Deep Learning course that for image analysis he always has &lt;code&gt;X_train&lt;/code&gt; matrices in the shape of &lt;code&gt;[height, width, 3, num_inputs]&lt;/code&gt;, or, if flattened, &lt;code&gt;[height X width X 3, num_inputs]&lt;/code&gt;. He also has his &lt;code&gt;y_train&lt;/code&gt; as &lt;code&gt;[1, num_inputs]&lt;/code&gt;. To me, it is more intuitive to flip these so that &lt;code&gt;X_train&lt;/code&gt; is &lt;code&gt;[num_inputs, height X width X 3]&lt;/code&gt; and &lt;code&gt;y_train&lt;/code&gt; is &lt;code&gt;[num_inputs, 1]&lt;/code&gt;. Is there any motivating reason or justification that it has to be the way he does it or is it just preference? Is this a standard or does it vary?&lt;/p&gt;&#xA;" OwnerUserId="41203" LastEditorUserId="41203" LastEditDate="2021-01-07T16:33:40.297" LastActivityDate="2021-01-07T16:33:40.297" Title="Does matrix shape for training/testing sets have to be in a particular order?" Tags="|machine-learning|neural-network|deep-learning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32588" PostTypeId="1" AcceptedAnswerId="32749" CreationDate="2018-06-04T08:51:19.360" Score="4" ViewCount="505" Body="&lt;p&gt;There is a quite famous article by H. Wickham, &lt;a href=&quot;https://www.jstatsoft.org/article/view/v059i10/v59i10.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tidy data,&lt;/a&gt; where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he compares his own definition of dataframe-tidy to other possible ways of achieving tidiness and mentions array-tidy in the following paragraph (but does not give any further explanations or examples):&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Fortunately, because there are many efficient tools for working with high-dimensional arrays, even sparse ones, such an array-tidy&#xA;  format is not only likely to be quite compact and efficient, it should&#xA;  also be able to easily connect with the mathematical basis of&#xA;  statistics. This, in fact, is the approach taken by the pandas Python&#xA;  data analysis library (McKinney 2010).&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;What does array-tidy data mean and why does he imply that this is somehow the default for the pandas library in Python?&lt;/p&gt;&#xA;" OwnerUserId="53114" LastActivityDate="2018-06-06T17:33:08.980" Title="Tidy data in panda dataframe" Tags="|r|pandas|data-cleaning|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32589" PostTypeId="2" ParentId="32585" CreationDate="2018-06-04T09:09:23.860" Score="1" Body="&lt;p&gt;You can use &lt;strong&gt;word embedding&lt;/strong&gt;, to encode words as vectors of real numbers. Then all calculations, such as comparison of words (to find similarity), are performed in that high-dimensional space.&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;&quot;What would be the best way to format my input (and my output) for&#xA;  this problem?&quot;&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;I cannot tell which is the best approach (depends on your problem) but this one is very common. It's also very easy to implement in Keras, using the &lt;a href=&quot;https://keras.io/layers/embeddings/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Embedding&lt;/a&gt; layer. &lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-04T09:09:23.860" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32590" PostTypeId="1" AcceptedAnswerId="32597" CreationDate="2018-06-04T09:22:52.780" Score="3" ViewCount="332" Body="&lt;p&gt;what does $a'$ mean in the &quot;combining&quot; equation in Dueling DQN? &lt;a href=&quot;https://arxiv.org/pdf/1511.06581.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(top of the page 5)&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Where there are $N$ actions to choose from;&lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$s$ is the incoming state (the input vector)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;$a$ is the action taken? (the chosen action)&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;$a'$ I don't know what it represents in this context&lt;/strong&gt;&lt;/li&gt;&#xA;&lt;li&gt;$\theta$ represents the weights of the convolutional layers&lt;/li&gt;&#xA;&lt;li&gt;$\alpha$ are the weights of the &quot;Advantage stream&quot; which outputs a vector&lt;/li&gt;&#xA;&lt;li&gt;$\beta$ are the weights of the Value stream (which outputs a scalar)&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Why not to simply use $a$ everywhere, why is $a'$ used in the average?&lt;/p&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-06-04T09:38:52.863" LastActivityDate="2018-06-04T10:18:12.490" Title="Dueling DQN what does a' mean?" Tags="|reinforcement-learning|q-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32591" PostTypeId="1" CreationDate="2018-06-04T09:23:26.330" Score="1" ViewCount="1430" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;from keras.models import Sequential&#xA;  from keras.layers import LSTM, Dense, Dropout&#xA;  from sklearn.metrics import precision_recall_fscore_support, roc_auc_score&#xA;  from keras.utils import np_utils&#xA;  from keras.callbacks import EarlyStopping&#xA;  data_dim = 41&#xA;  timesteps = 20&#xA;  num_classes = 10&lt;/p&gt;&#xA;  &#xA;  &lt;h1&gt;expected input data shape: (batch_size, timesteps, data_dim)&lt;/h1&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;model = Sequential()&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;returns a sequence of vectors of dimension 256&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;model.add(LSTM(256, return_sequences=True, input_shape=(timesteps, data_dim)))&lt;br&gt;&#xA;model.add(Dropout(0.2))&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;return a single vector of dimension 128&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;model.add(LSTM(128))&lt;br&gt;&#xA;model.add(Dropout(0.2))&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;apply softmax to output&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;model.add(Dense(num_classes, activation='softmax'))&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;compile the model for multi-class classification&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;load fold1 for testing&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;train_x, train_y = load_folds([1,2,3,4,5,6])&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;load fold2 for validation&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;valid_x, valid_y = load_folds([9])&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;load fold3 for testing&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;test_x, test_y = load_folds([10])&lt;/p&gt;&#xA;&#xA;&lt;h1&gt;a stopping function to stop training before we excessively overfit to the training set&lt;/h1&gt;&#xA;&#xA;&lt;p&gt;earlystop = EarlyStopping(monitor='val_loss', patience=0, verbose=1, mode='auto')&#xA;model.fit(train_x, train_y, batch_size=128, nb_epoch=10, callbacks=[earlystop], validation_data=(valid_x, valid_y))&lt;/p&gt;&#xA;&#xA;&lt;p&gt;this is my code when i run it i got error &quot;ValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (0,)&quot;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;can someone help me to understand this and how can i solve it ?&lt;/p&gt;&#xA;" OwnerUserId="53115" LastActivityDate="2018-08-02T14:16:45.160" Title="ValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (0,)" Tags="|machine-learning|neural-network|" AnswerCount="0" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32592" PostTypeId="2" ParentId="32569" CreationDate="2018-06-04T09:25:35.600" Score="3" Body="&lt;p&gt;Regarding your first question, by dividing by the length, you get the percentage of each category, and you can see that one category has a very low percentage. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you use the regular &lt;code&gt;train_test_split&lt;/code&gt;, the proportion of the categories will be different in each set, as the split is made randomly, and this can introduce a bias.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Imagine categories with a very low number of observations, you could even have categories missing entirely in either of the sets, which will cause trouble for your model.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The use of stratified sampling allows you to have the same proportion for the categories. That is why you see the same results: it is the goal of this sampling. &lt;/p&gt;&#xA;" OwnerUserId="39673" LastActivityDate="2018-06-04T09:25:35.600" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32593" PostTypeId="1" CreationDate="2018-06-04T09:42:36.590" Score="3" ViewCount="351" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure &lt;a href=&quot;https://i.stack.imgur.com/JsKoV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;1&lt;/a&gt;. Why is it so ?. Even the shapes differ for second and third line ?. Can someone help me to gain an understanding of below picture&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/JsKoV.jpg&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/JsKoV.jpg&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;The second question is related to the correlation between attributes. The intention behind asking this question it to gain more understanding from StackOverflow data science users. The very one thing I learnt from the below picture is &lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;if you plot attribute against the same attribute, you will end up with lines, which is of no use. For instance, if you notice median_house_value in y-axis against median_house_value in the x-axis, they are basically histograms.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;p&gt;Is there any valuable feedback that you can comment on the below picture ?. If so, please let me know.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://i.stack.imgur.com/VJoqR.png&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;img src=&quot;https://i.stack.imgur.com/VJoqR.png&quot; alt=&quot;enter image description here&quot;&gt;&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="2893" LastEditorUserId="2893" LastEditDate="2018-06-04T10:34:54.963" LastActivityDate="2019-01-01T19:02:00.503" Title="Standard correlation coefficient of various datasets" Tags="|python|preprocessing|correlation|" AnswerCount="1" CommentCount="6" ContentLicense="CC BY-SA 4.0" />
  <row Id="32594" PostTypeId="2" ParentId="32587" CreationDate="2018-06-04T09:56:37.000" Score="1" Body="&lt;p&gt;It depends on the deep learning framework that you use, and you have to use the shape that the functions of the framework use. I think it is different in Tensorflow and Pytorch. The recommendation is to check before doing anything in the documentation of the framework.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-04T09:56:37.000" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32595" PostTypeId="2" ParentId="32541" CreationDate="2018-06-04T09:58:42.923" Score="2" Body="&lt;p&gt;There's a function that does it automatically: &lt;code&gt;tf.contrib.losses.softmax_cross_entropy(logits, onehot_labels, weight=weight)&lt;/code&gt;&lt;/p&gt;&#xA;" OwnerUserId="50727" LastEditorUserId="50727" LastEditDate="2018-06-04T10:06:14.693" LastActivityDate="2018-06-04T10:06:14.693" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32596" PostTypeId="2" ParentId="32567" CreationDate="2018-06-04T10:02:31.400" Score="3" Body="&lt;p&gt;If the user inputs a title, then you could construct a system which finds the most similar titles in the corpus, and outputs the ingredients in the retrieved recipes. Some ideas below:  &lt;/p&gt;&#xA;&#xA;&lt;p&gt;1) Represent the titles in a common vector space using a vocabulary-based vectorization and use Jaccard or Cosine similarity to find the most similar titles.  &lt;a href=&quot;http://text2vec.org/similarity.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;examples of similarity measures&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;2) Convert the words in the title to their respective word-embeddings using FastText, GloVe, or Word2Vec (publicly available), and then use cosine distance or  word-movers distance to find the most similar titles. &lt;a href=&quot;https://nlp.stanford.edu/projects/glove/&quot; rel=&quot;nofollow noreferrer&quot;&gt;example of publicly available GloVe word embeddings&lt;/a&gt; &lt;/p&gt;&#xA;&#xA;&lt;p&gt;3) Create sentence embeddings for the title by taking the average of the word embedding vectors or using InferSent embeddings, and then use a distance function to find the most similar titles. &lt;a href=&quot;https://github.com/facebookresearch/InferSent&quot; rel=&quot;nofollow noreferrer&quot;&gt;InferSent repository link&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;spaCy is a good Python package for performing these type of NLP tasks &lt;a href=&quot;https://spacy.io/&quot; rel=&quot;nofollow noreferrer&quot;&gt;link&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="49852" LastActivityDate="2018-06-04T10:02:31.400" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32597" PostTypeId="2" ParentId="32590" CreationDate="2018-06-04T10:11:52.220" Score="2" Body="&lt;p&gt;It is just a type of namespacing, because $a$ is already assigned the chosen action. There are two contexts of action being considered in the equation, so there needs to be a symbol for each context. Using $a'$ is an obvious choice as the letter $a$ is implicitly linked to representing an action already.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;The sum over $a'$ is a sum over all possible actions in state $s$, irrespective of the chosen action $a$.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So both $a$ and $a'$ represent actions. $a$ is the current action, supplied on the LHS of the equation. $a'$ represents the iterator of a sum over all actions $[\forall a' \in \mathcal{A}(s)]$, only used in the calculation on the RHS. Sometimes you will see a completely different letter chosen, or some subscripting or other way to show these represent &lt;em&gt;different&lt;/em&gt; actions.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;It is also quite common to see $a$ representing current action, and $a'$ representing the &lt;em&gt;next&lt;/em&gt; action (taken when in state $s'$). But that is not what is happening here.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-06-04T10:18:12.490" LastActivityDate="2018-06-04T10:18:12.490" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32598" PostTypeId="2" ParentId="32572" CreationDate="2018-06-04T10:43:38.410" Score="0" Body="&lt;p&gt;You can use a sequence-to-sequence model based on different RNN (LSTM for example) types. As far as I know, seq2seq models has been applied to spell checking. You should take a look to the existing work on spell checking and figure out how to use them in your problem. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Maybe you can treat correct word - misspelled word pairs as paraphrases and solve the problem as a paraphrase generation task. &lt;/p&gt;&#xA;" OwnerUserId="52925" LastActivityDate="2018-06-04T10:43:38.410" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32599" PostTypeId="1" AcceptedAnswerId="32656" CreationDate="2018-06-04T10:49:07.787" Score="1" ViewCount="258" Body="&lt;p&gt;In the self-organizing map method, each of the output units is called neurons. But after reading the details about the algorithm, I could not find any logistic regression unit which is the neuron in a neural network architecture. I find it closer to the clustering algorithms like K-NN. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Is there any explanation why SOM units are called neurons? And why is it linked to neural networks (while ANN does a completely different job i.e., classification)&lt;/p&gt;&#xA;&#xA;&lt;p&gt;P.S.: In this paper (&lt;a href=&quot;https://ieeexplore.ieee.org/document/6007170/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://ieeexplore.ieee.org/document/6007170/&lt;/a&gt;), the SOM is called as an ANN. Here is a sentence from the abstract:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;the SOM is a type of artificial neural network that is trained using unsupervised data to produce a low-dimensional, discretized representation of an input space of training samples in a cell map.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="50828" LastActivityDate="2018-06-05T10:44:24.093" Title="Why Self-Organizing Map (SOM) units are called neurons?" Tags="|neural-network|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32600" PostTypeId="2" ParentId="32593" CreationDate="2018-06-04T11:46:19.043" Score="1" Body="&lt;h2&gt;First part:&lt;/h2&gt;&#xA;&lt;p&gt;The question you have to ask yourself is &amp;quot;Given I know the x-value, what knowledge do I gain about the y-value&amp;quot;. And a small side note: you may only consider &lt;em&gt;linear&lt;/em&gt; relationships.&lt;/p&gt;&#xA;&lt;p&gt;Consider the first row. In the first image, if you know the x-value, then the y-value is &lt;code&gt;y = x&lt;/code&gt;, i.e. you know the y-value exactly. In the following two images, if the x-value is high, then the y-value is high too, but there is a random component too. Thus, the correlation coefficient is positive (i.e. knowing x helps in estimating y), but not 1 (i.e. there is no exact equation &lt;code&gt;y = a*x&lt;/code&gt;. In the middle image, there is no relationship between the x- and y-coordinate - they are purely random, so no correlation. In the right three images, it is the same story as in the left three images, but the sign is flipped.&lt;/p&gt;&#xA;&lt;p&gt;Now consider the second row: in the first three cases, given you know the x-value, you can always infer the y-value, e.g. &lt;code&gt;y = x&lt;/code&gt;, &lt;code&gt;y = 0.5 x&lt;/code&gt;, and &lt;code&gt;y = 0.1 x&lt;/code&gt;. There is no random component in y, so if you know x, you also know the exact value of y, thus the correlation coefficient is 1. It doesn't matter if the formula is &lt;code&gt;y = 1 x&lt;/code&gt; or &lt;code&gt;y = 0.1 x&lt;/code&gt; (i.e. the slope of the line doesn't matter!) all that matters for the correlation coefficient is that there &lt;em&gt;is&lt;/em&gt; such a linear coefficient which leads to an exact match.&lt;/p&gt;&#xA;&lt;p&gt;In the third row, knowing x &lt;em&gt;does&lt;/em&gt; give some knowledge on y. For example in the middle plot, you have a relationship of &lt;code&gt;y = x^2 + random&lt;/code&gt;, so there is indeed some relationship between x and y, but it is &lt;em&gt;non-linear&lt;/em&gt;. Thus the correlation coefficient is zero. There is no way for you to say &amp;quot;a high x-value also leads to a high y-value&amp;quot; or &amp;quot;a high x-value leads to a low y-value&amp;quot;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h2&gt;Second part:&lt;/h2&gt;&#xA;&lt;p&gt;I assume this plot is generated e.g. with Seaborn's &lt;a href=&quot;https://seaborn.pydata.org/generated/seaborn.pairplot.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;code&gt;pairplot()&lt;/code&gt;&lt;/a&gt; function, though of course different functions exist for creating this kind of plot. It is important to note that the plots on the diagonal are &lt;em&gt;fundamentally&lt;/em&gt; different plots than the other elements.&lt;/p&gt;&#xA;&lt;p&gt;The off-diagonal plots are two-dimensional scatter plots, i.e. you draw a point for each data sample.&#xA;The diagonal plots are histograms of each feature, and not scatter plots. This is because a scatter plot of twice the same variable would always give you a straight line &lt;code&gt;y = x&lt;/code&gt;.&#xA;The histograms are actually quite useful as you can get a feeling for the data and make a guess about the underlying distribution.&lt;/p&gt;&#xA;&lt;p&gt;For example, your &lt;code&gt;total_rooms&lt;/code&gt; variable is heavily skewed, most houses have very few rooms, but there are a few outliers with huge numbers of rooms. I thus wouldn't trust the mean value of &lt;code&gt;total_rooms&lt;/code&gt; much and rather use the median - of course this is highly dependent on what kind of analyse you are doing.&lt;/p&gt;&#xA;" OwnerUserId="14906" LastEditorUserId="-1" LastEditDate="2020-06-16T11:08:43.077" LastActivityDate="2018-06-04T11:46:19.043" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32601" PostTypeId="1" AcceptedAnswerId="34068" CreationDate="2018-06-04T11:58:13.757" Score="1" ViewCount="498" Body="&lt;p&gt;For Dueling DQN &lt;a href=&quot;https://arxiv.org/pdf/1511.06581.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;(page 5)&lt;/a&gt;, why do authors use an average for Advantage stream, and don't simply &quot;activate&quot; the Advantage stream (with a $tanh$ for example)?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Would &quot;activating&quot; work in theory, and is it a similar idea to what the authors intended to achieve, or am I missing the point?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;To remind, this is the equation, which Produces a Q-value for the taken action $a$ by combining the Value stream with the Advantage stream:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$&lt;/p&gt;&#xA;&#xA;&lt;p&gt;where &lt;/p&gt;&#xA;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;$s$ is the current state we are in&lt;/li&gt;&#xA;&lt;li&gt;$a$ is the action we've decided to take&lt;/li&gt;&#xA;&lt;li&gt;$a'$ is one of any action we could have taken (including the action we've taken)&lt;/li&gt;&#xA;&lt;li&gt;$\theta$ are parameters (weights) of the network before the &quot;splitting&quot; into two separate streams&lt;/li&gt;&#xA;&lt;li&gt;$\alpha$ are the parameters of Advantage stream&lt;/li&gt;&#xA;&lt;li&gt;$\beta$ are the parameters of the Value stream&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;" OwnerUserId="43077" LastEditorUserId="43077" LastEditDate="2018-07-13T14:31:00.433" LastActivityDate="2018-07-13T14:31:00.433" Title="Dueling DQN - Advantage Stream, why use average and not the tanh?" Tags="|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32603" PostTypeId="2" ParentId="32457" CreationDate="2018-06-04T12:18:13.767" Score="1" Body="&lt;p&gt;I think the problem might be due to the data, as this code:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;from sklearn import svm&#xA;from sklearn.model_selection import GridSearchCV&#xA;import pandas as pd&#xA;import numpy as np&#xA;&#xA;X_pca = np.random.rand(100, 2)&#xA;labels = X_pca[:, 0] + X_pca[:, 1] &amp;gt; 0.5&#xA;&#xA;model = svm.SVC()&#xA;params = {'C' : [0.01, 0.1, 1, 10],&#xA;      'gamma' : [0., 0.1, 1, 'auto'],&#xA;      'probability' : [True] &#xA;     }&#xA;clf =  GridSearchCV(model, params, cv=2, return_train_score=False)&#xA;clf.fit(X_pca, labels)&#xA;&#xA;print(pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score']] \&#xA;        .sort_values(by='rank_test_score'))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;gives better output:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;    mean_test_score  rank_test_score&#xA;10             1.00                1&#xA;13             1.00                1&#xA;14             1.00                1&#xA;15             1.00                1&#xA;11             0.95                5&#xA;0              0.83                6&#xA;1              0.83                6&#xA;2              0.83                6&#xA;3              0.83                6&#xA;4              0.83                6&#xA;5              0.83                6&#xA;6              0.83                6&#xA;7              0.83                6&#xA;8              0.83                6&#xA;9              0.83                6&#xA;12             0.83                6&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;(consider the fact that the results will depend on the seed of the &lt;code&gt;rand&lt;/code&gt; function, but with other seeds they are similar and the mean test score changes using different values of the parameters.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-04T12:18:13.767" CommentCount="5" ContentLicense="CC BY-SA 4.0" />
  <row Id="32604" PostTypeId="2" ParentId="32507" CreationDate="2018-06-04T12:21:09.907" Score="0" Body="&lt;p&gt;In the case that the inverse exists, it is the same as the pseudo inverse. Unless the method crashes due to the fact that the inverse does not exist, the results will be the same.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-04T12:21:09.907" CommentCount="3" ContentLicense="CC BY-SA 4.0" />
  <row Id="32605" PostTypeId="2" ParentId="32587" CreationDate="2018-06-04T12:31:06.803" Score="2" Body="&lt;h3&gt;In practice&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;Andrew Ng seems to be using the convention on the Theano framework. If you had 10 colour images, each 100 pixels high and 200 pixels wide, Theano models would expect input of the form:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(batch size, input channels, input rows, input columns)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;and so dimensions: &lt;code&gt;(10, 3, 100, 200)&lt;/code&gt;. The three is because of the three RGb dimensions in a colour image.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://www.tensorflow.org/programmers_guide/tensors#higher_ranks&quot; rel=&quot;nofollow noreferrer&quot;&gt;Tensorflow&lt;/a&gt; on the other hand reverse this order to instead use:&#xA;(num_obs, height, width, channels) - for the same example as above this becomes:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;(batch size, input rows, input columns, input channels)&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;meaning the dimensions of the input should be &lt;code&gt;(10, 100, 200, 3)&lt;/code&gt;.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Keras works with Tensorflow and Theano and supports both conventions by simply allowing the user to set the position in which the number of channels are given. This can be set in a config file or using a specific environment variable upon setup. Have a look at &lt;a href=&quot;https://keras.io/backend/#what-is-a-backend&quot; rel=&quot;nofollow noreferrer&quot;&gt;the relevant documentation&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;hr&gt;&#xA;&#xA;&lt;h3&gt;(Possible) justifications&lt;/h3&gt;&#xA;&#xA;&lt;p&gt;There are different points of view on what is more natural. If you come from a computer vision arena (or image processing in general), then libraries like &lt;a href=&quot;https://docs.opencv.org/2.4/doc/tutorials/core/mat_the_basic_image_container/mat_the_basic_image_container.html&quot; rel=&quot;nofollow noreferrer&quot;&gt;OpenCV&lt;/a&gt; use a coordinate system that has &lt;code&gt;(0,0)&lt;/code&gt; in the top left of an image, and you specify a single pixel by given the vertical movement, then the horizontal movement from the origin. This means you give a &lt;em&gt;height&lt;/em&gt;, then a &lt;em&gt;width&lt;/em&gt;. One justification for this might be that many computer vision algorithms (e.g. colour filters) focus on difference across the channels of an image, but it is just convention.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;In general graphing and mathematics, it is common to give the &lt;strong&gt;X&lt;/strong&gt; coordinates first, then the &lt;strong&gt;Y&lt;/strong&gt;, which means talking about horizontal movements, then vertical. Linear algebra in general is a good example of this.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;All this being said, the best advice is to make sure you use the correct dimensions according to the documentation of the software you use. If you make your own software - you get to choose! (But make sure to document it!)&lt;/p&gt;&#xA;" OwnerUserId="45264" LastActivityDate="2018-06-04T12:31:06.803" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32606" PostTypeId="2" ParentId="24363" CreationDate="2018-06-04T12:46:53.107" Score="0" Body="&lt;p&gt;One approach is to use an artificial neural network to extract features representing the images. This can be done either by using a pre-configured network with pre-trained weights and extracting the output of one of the hidden layers, or by constructing and training your own network for this purpose. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;To use a pre-configured, pre-trained module can be accomplished easily with Keras and TensorFlow, where you can import InceptionV3 or MobileNet with weights pre-trained on ImageNet, which would net you 2048 or 1024 features per image, respectively. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;An article discussing such an approach can be found &lt;a href=&quot;https://medium.com/@franky07724_57962/using-keras-pre-trained-models-for-feature-extraction-in-image-clustering-a142c6cdf5b1&quot; rel=&quot;nofollow noreferrer&quot;&gt;here&lt;/a&gt;. This could hopefully give you better performance than using something like PCA to conduct dimensionality reduction.&lt;/p&gt;&#xA;" OwnerUserId="51545" LastEditorUserId="29575" LastEditDate="2018-06-04T13:16:45.283" LastActivityDate="2018-06-04T13:16:45.283" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32607" PostTypeId="2" ParentId="32578" CreationDate="2018-06-04T13:24:05.653" Score="2" Body="&lt;p&gt;This looks like a proper task for &lt;a href=&quot;https://cs.stanford.edu/~quocle/paragraph_vector.pdf&quot; rel=&quot;nofollow noreferrer&quot;&gt;Doc2Vec&lt;/a&gt;, which is an algorithm to build paragraph embeddings. For a nice implementation with usage examples you can try out &lt;a href=&quot;https://rare-technologies.com/doc2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;gensim&lt;/a&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;Other options can be using &lt;a href=&quot;https://rare-technologies.com/word2vec-tutorial/&quot; rel=&quot;nofollow noreferrer&quot;&gt;word2vec&lt;/a&gt; and using vector averaging or sum to build sentence vectors (look at &lt;a href=&quot;https://stackoverflow.com/questions/45234310/doc2vec-worse-than-mean-or-sum-of-word2vec-vectors&quot;&gt;this&lt;/a&gt;).&lt;/p&gt;&#xA;&#xA;&lt;p&gt;For more approaches have a look at these two tutorials where you can see how to implement LSA, LDA, TFIDF:&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://nlpforhackers.io/topic-modeling/&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://nlpforhackers.io/topic-modeling/&lt;/a&gt;&lt;/p&gt;&#xA;&#xA;&lt;p&gt;&lt;a href=&quot;https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730&lt;/a&gt;&lt;/p&gt;&#xA;" OwnerUserId="45166" LastActivityDate="2018-06-04T13:24:05.653" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32608" PostTypeId="1" AcceptedAnswerId="32611" CreationDate="2018-06-04T13:54:28.843" Score="2" ViewCount="466" Body="&lt;p&gt;Given the significant advancements in reinforcement learning, I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.&lt;/p&gt;&#xA;" OwnerUserId="53127" LastActivityDate="2018-06-04T15:13:33.107" Title="Reframing action recognition as a reinforcement learning problem" Tags="|machine-learning|deep-learning|classification|image-classification|reinforcement-learning|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32609" PostTypeId="1" CreationDate="2018-06-04T14:45:57.047" Score="1" ViewCount="391" Body="&lt;p&gt;I have a multi-label classification problem-- millions of records that potentially could hold more than one label. I'm running into issues related to lack of research/examples online, and am unable to create a strong model. Would it be advised against to build a multi-class classification model, and take any label with a predicted probability over 0 to be considered a potential label? Even with problem transformations, like binary relevance and classifier chains, the accuracy is terribly low (~12%) as opposed to using Naive Bayes and getting ~57% accuracy.&lt;/p&gt;&#xA;" OwnerUserId="50746" LastActivityDate="2018-06-04T14:58:35.513" Title="Is it bad practice to use multi-class over multi-label classification?" Tags="|python|classification|multiclass-classification|multilabel-classification|" AnswerCount="1" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32610" PostTypeId="2" ParentId="32609" CreationDate="2018-06-04T14:58:35.513" Score="1" Body="&lt;p&gt;Perhaps I understood the question wrong, but &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-label_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-label&lt;/a&gt; and &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiclass_classification&quot; rel=&quot;nofollow noreferrer&quot;&gt;multi-class&lt;/a&gt; problems are fundamentally different, as the multi-class is exclusive (no instance can be classified into two classes a the same time), while the multi-label relaxes this condition. As Wikipedia on multi-label states:&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;in the multi-label problem there is no constraint on how many of the classes the instance can be assigned to.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;So, using multi-class or multi-label depends on the problem itself and not on how you solve it.&lt;/p&gt;&#xA;" OwnerUserId="50727" LastActivityDate="2018-06-04T14:58:35.513" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32611" PostTypeId="2" ParentId="32608" CreationDate="2018-06-04T15:01:02.463" Score="1" Body="&lt;blockquote&gt;&#xA;  &lt;p&gt;Given the significant advancements in reinforcement learning&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Worth noting that many of the recent advancements are due to improvements in neural networks used as function approximators, and understanding how to integrate them with reinforcement learning (RL) to help solve RL challenges involving vision or other complex non-linear mapping from state to best action. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;So at least some of current improvements in RL were due to researchers asking the opposite question &quot;Given the significant advancements in neural networks . . .&quot;&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;Generally, at the top level, the answer is &lt;em&gt;Yes, but it offers no benefit, and could perform a lot worse&lt;/em&gt;. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;That is because in typical supervised learning scenarios there is nothing to match the concept of actions that modify state. In the example classifiers you can have the equivalent of a state (the input to be classified), an action (the choice of category) and the reward (whether of not the choice matches the label). But taking an action does not lead to another state, rewards are not sparse or cumulative across multiple actions within a specific environment or &quot;episode&quot;. There are no time steps. &lt;/p&gt;&#xA;&#xA;&lt;p&gt;RL algorithms are generic &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_decision_process&quot; rel=&quot;nofollow noreferrer&quot;&gt;MDP solvers&lt;/a&gt; - they can learn about relationships between state, action and likely next state, and optimise long-term goals over may time steps when given a current state. That also makes them less efficient learners when those relationships are not valid or important in the problem you want to solve. If you trained e.g. Q-learning on a typical image classification dataset, and added time steps, it would spend a lot of time/resources establishing that its choice of action had no influence on which images it was subsequently presented with, or how easy it was to get a reward from later images as opposed to earlier ones depending on how the state varied. If you really did allow the choice of action to determine the next image, then you would be training the RL to do something else other than classification.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;You could frame the classifiers as &lt;a href=&quot;https://en.wikipedia.org/wiki/Multi-armed_bandit#Contextual_bandit&quot; rel=&quot;nofollow noreferrer&quot;&gt;contextual bandits&lt;/a&gt;, which is perhaps closer match. However, that still throws away knowledge that you have about the classification problem, replacing it with a generic reward system. For instance, a contextual bandit solver would deliberately guess wrong classes to check whether sometimes there was a small chance of a high reward for doing that.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you were very careful about how you represented actions and rewards, and set other hyper-parameters, then you might be able to re-create a similar gradient setup to normal supervised learning, and only lose a little bit of efficiency through using RL or contextual bandit framing of your problem. However, you would still have added some non-necessary complexity.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;If you search, you may find some ways to combine RL with supervised learning, for instance in &lt;a href=&quot;https://arxiv.org/pdf/1611.02796v3.pdf]&quot; rel=&quot;nofollow noreferrer&quot;&gt;this paper, the authors propose using RL to refine a generative RNN&lt;/a&gt;. However, these currently seem niche and are not aimed to improve upon or replace supervised learning.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Finally, in theory you could allow RL to control a video camera pan/zoom as part of activity recognition (or any other video or multi-image classification task). That &lt;em&gt;would&lt;/em&gt; be a full RL problem, because the agent's actions really would be influencing the later states and hopefully improving the accuracy of recognition. For learning efficiency, you would likely want to combine this initially with a network that had already been trained to recognise actions on a supervised data set. You would need to experiment with how much the recognition part was trained compared to the RL part (as it will start to collect data outside of your normal datasets). And of course setup and training of the combined system could be a &lt;em&gt;major&lt;/em&gt; project. You may be able to simulate it in a game engine perhaps in the early stages.&lt;/p&gt;&#xA;" OwnerUserId="836" LastEditorUserId="836" LastEditDate="2018-06-04T15:13:33.107" LastActivityDate="2018-06-04T15:13:33.107" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32612" PostTypeId="1" CreationDate="2018-06-04T15:14:18.823" Score="0" ViewCount="93" Body="&lt;p&gt;So I want to compare strings and find out there relation to one another. For this I figured out I also have to encode the positional info. For eg:&#xA;&lt;code&gt;aaaaassssssbbbbbb&lt;/code&gt; should be more relevant to &lt;code&gt;aaaasssssbbbb&lt;/code&gt;rather than &lt;code&gt;bbbbaaasssss&lt;/code&gt;. The output will be a relative measure and 1 and 2 should match more compared to 1 and 3. Also I will represent each character by a number.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;So how can I re-design my feature vector (feature transformation) to be input in a NN to capture this information effectively?&lt;/p&gt;&#xA;" OwnerUserId="47832" LastActivityDate="2018-06-04T16:45:29.670" Title="How to encode positional information of a string in a NN effectively" Tags="|machine-learning|neural-network|" AnswerCount="2" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32613" PostTypeId="1" AcceptedAnswerId="32615" CreationDate="2018-06-04T15:27:20.900" Score="0" ViewCount="116" Body="&lt;p&gt;How can I subset/count rows in one data frame that correspond to rows in another data frame?&lt;/p&gt;&#xA;&#xA;&lt;p&gt;I have a data frame DF1 with dates, categories and time instances for each of the date and category combinations. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF1&amp;lt;-data.frame(&quot;DATE&quot;=c(as.Date(&quot;2018-12-05&quot;),as.Date(&quot;2018-12-06&quot;),as.Date(&quot;2018-12-07&quot;)),&#xA;            &quot;CATEGORY&quot;=c(&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;),&#xA;            &quot;TIME&quot;=c(as.POSIXct(&quot;2018-12-05 10:05&quot;),as.POSIXct(&quot;2018-12-06 10:20&quot;),as.POSIXct(&quot;2018-12-07 10:40&quot;)))&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        DATE CATEGORY                TIME&#xA;1 2018-12-05     cat1 2018-12-05 10:05:00&#xA;2 2018-12-06     cat2 2018-12-06 10:20:00&#xA;3 2018-12-07     cat3 2018-12-07 10:40:00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Then I have another data frame DF2 with objects, categories and a time interval. For example:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF2&amp;lt;-data.frame(&quot;OBJECT_ID&quot;=1:9,&#xA;            &quot;CATEGORY&quot;=c(&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;,&quot;cat1&quot;,&quot;cat3&quot;,&quot;cat2&quot;,&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;),&#xA;            &quot;START&quot;=c(as.POSIXct(&quot;2018-12-05 09:00&quot;),as.POSIXct(&quot;2018-12-06 10:00&quot;),as.POSIXct(&quot;2018-12-07 10:00&quot;),&#xA;                      as.POSIXct(&quot;2018-12-05 09:30&quot;),as.POSIXct(&quot;2018-12-06 08:30&quot;),as.POSIXct(&quot;2018-12-07 10:30&quot;),&#xA;                      as.POSIXct(&quot;2018-12-05 08:30&quot;),as.POSIXct(&quot;2018-12-06 08:30&quot;),as.POSIXct(&quot;2018-12-07 08:30&quot;)),&#xA;            &quot;END&quot;=c(as.POSIXct(&quot;2018-12-05 10:00&quot;),as.POSIXct(&quot;2018-12-06 11:00&quot;),as.POSIXct(&quot;2018-12-07 10:00&quot;),&#xA;                    as.POSIXct(&quot;2018-12-05 11:30&quot;),as.POSIXct(&quot;2018-12-06 10:30&quot;),as.POSIXct(&quot;2018-12-07 10:30&quot;),&#xA;                    as.POSIXct(&quot;2018-12-05 11:30&quot;),as.POSIXct(&quot;2018-12-06 12:30&quot;),as.POSIXct(&quot;2018-12-07 13:30&quot;))&#xA;              )&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;that is &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;  OBJECT_ID CATEGORY               START                 END&#xA;1         1     cat1 2018-12-05 09:00:00 2018-12-05 10:00:00&#xA;2         2     cat2 2018-12-06 10:00:00 2018-12-06 11:00:00&#xA;3         3     cat3 2018-12-07 10:00:00 2018-12-07 10:00:00&#xA;4         4     cat1 2018-12-05 09:30:00 2018-12-05 11:30:00&#xA;5         5     cat3 2018-12-06 08:30:00 2018-12-06 10:30:00&#xA;6         6     cat2 2018-12-07 10:30:00 2018-12-07 10:30:00&#xA;7         7     cat1 2018-12-05 08:30:00 2018-12-05 11:30:00&#xA;8         8     cat2 2018-12-06 08:30:00 2018-12-06 12:30:00&#xA;9         9     cat3 2018-12-07 08:30:00 2018-12-07 13:30:00&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I need to count the number of rows in DF2 that contain the time instance given in DF1 for each date and category. Meaning:&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;        DATE CATEGORY                TIME NO_OF_OBJECTS&#xA;1 2018-12-05     cat1 2018-12-05 10:05:00             2&#xA;2 2018-12-06     cat2 2018-12-06 10:20:00             2&#xA;3 2018-12-07     cat3 2018-12-07 10:40:00             1&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;I have a feeling that the apply family should be able to do something here, but I cannot quite grasp how this could be achieved.&lt;/p&gt;&#xA;" OwnerUserId="53107" LastActivityDate="2018-06-04T18:40:46.590" Title="R: for each row in DF1 subset/count corresponding rows in DF2" Tags="|r|" AnswerCount="3" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32614" PostTypeId="1" CreationDate="2018-06-04T15:58:38.887" Score="2" ViewCount="2060" Body="&lt;p&gt;I think in writing we often refer to the part of the neural networks that tell us about the data are the parameters when we really mean the weights, is this correct? Would the sentence...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Better understanding &lt;strong&gt;parameters&lt;/strong&gt; of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;&#xA;&lt;p&gt;...be better phrased as...&lt;/p&gt;&#xA;&#xA;&lt;blockquote&gt;&#xA;  &lt;p&gt;Better understanding the &lt;strong&gt;weights&lt;/strong&gt; of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.&lt;/p&gt;&#xA;&lt;/blockquote&gt;&#xA;" OwnerUserId="46886" LastActivityDate="2018-06-04T16:18:45.697" Title="Difference between parameters and weights in ANN" Tags="|neural-network|" AnswerCount="1" CommentCount="4" ContentLicense="CC BY-SA 4.0" />
  <row Id="32615" PostTypeId="2" ParentId="32613" CreationDate="2018-06-04T16:06:29.760" Score="0" Body="&lt;p&gt;Here is a solution using &lt;code&gt;data.table&lt;/code&gt; package.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF3 &amp;lt;- data.table(merge(DF1, DF2, by = &quot;CATEGORY&quot;)) # merge tables by category&#xA;DF3$TIME_OK &amp;lt;- (DF3$TIME &amp;gt;= DF3$START &amp;amp; DF3$TIME &amp;lt;= DF3$END) # create a column to check time&#xA;DF3 &amp;lt;- DF3[DF3$TIME_OK == T,] # subset table&#xA;DF3 &amp;lt;- DF3[,.(NO_OF_OBJECT = .N), by = c(&quot;CATEGORY&quot;, &quot;DATE&quot;, &quot;TIME&quot;)] # aggregate results&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;Basically, it merges both data frames and subsets only relevant time period before aggregating results.&lt;/p&gt;&#xA;" OwnerUserId="42829" LastActivityDate="2018-06-04T16:06:29.760" CommentCount="1" ContentLicense="CC BY-SA 4.0" />
  <row Id="32616" PostTypeId="2" ParentId="32614" CreationDate="2018-06-04T16:18:45.697" Score="2" Body="&lt;p&gt;That's partially right.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Apart from the weights (and bias) you also have the so called &lt;strong&gt;hyper-parameters&lt;/strong&gt;. Depending on your architecture, you may have different ones. For a simple feedforward neural network, the hyperparameters are:&lt;/p&gt;&#xA;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;number of neurons&lt;/li&gt;&#xA;&lt;li&gt;number of layers&lt;/li&gt;&#xA;&lt;li&gt;learning rate eta (η)&lt;/li&gt;&#xA;&lt;li&gt;regularization penalty lambda (λ)&lt;/li&gt;&#xA;&lt;li&gt;momentum&lt;/li&gt;&#xA;&lt;li&gt;number of epochs&lt;/li&gt;&#xA;&lt;li&gt;batch size&lt;/li&gt;&#xA;&lt;li&gt;dropout&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&#xA;&lt;p&gt;etc&lt;/p&gt;&#xA;&#xA;&lt;p&gt;These hyperparameters are (usually) user-tuned. The weights (and bias) are self-trained by the network, based on optimization. Keep in mind that hyperparameters like the number of neurons or layers are immediately connected with the underlying model of your data, therefore they are an index of your dataset's complexity.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-04T16:18:45.697" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32617" PostTypeId="2" ParentId="32613" CreationDate="2018-06-04T16:22:44.440" Score="0" Body="&lt;p&gt;One way you could do this is by replicating VLOOKUP in R, i.e. cross-check the times of DF2 against DF1 to see which ones match.&lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;DF1&amp;lt;-data.frame(&quot;DATE&quot;=c(as.Date(&quot;2018-12-05&quot;),as.Date(&quot;2018-12-06&quot;),as.Date(&quot;2018-12-07&quot;)),&#xA;            &quot;CATEGORY&quot;=c(&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;),&#xA;            &quot;TIME&quot;=c(as.POSIXct(&quot;2018-12-05 10:05&quot;),as.POSIXct(&quot;2018-12-06 10:20&quot;),as.POSIXct(&quot;2018-12-07 10:40&quot;)))&#xA;&#xA;DF2&amp;lt;-data.frame(&quot;OBJECT_ID&quot;=1:9,&#xA;                &quot;CATEGORY&quot;=c(&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;,&quot;cat1&quot;,&quot;cat3&quot;,&quot;cat2&quot;,&quot;cat1&quot;,&quot;cat2&quot;,&quot;cat3&quot;),&#xA;                &quot;TIME&quot;=c(as.POSIXct(&quot;2018-12-05 09:00&quot;),as.POSIXct(&quot;2018-12-06 10:00&quot;),as.POSIXct(&quot;2018-12-07 10:00&quot;),&#xA;                          as.POSIXct(&quot;2018-12-05 09:30&quot;),as.POSIXct(&quot;2018-12-06 08:30&quot;),as.POSIXct(&quot;2018-12-07 10:30&quot;),&#xA;                          as.POSIXct(&quot;2018-12-05 08:30&quot;),as.POSIXct(&quot;2018-12-06 08:30&quot;),as.POSIXct(&quot;2018-12-07 08:30&quot;)),&#xA;                &quot;END&quot;=c(as.POSIXct(&quot;2018-12-05 10:00&quot;),as.POSIXct(&quot;2018-12-06 11:00&quot;),as.POSIXct(&quot;2018-12-07 10:00&quot;),&#xA;                        as.POSIXct(&quot;2018-12-05 11:30&quot;),as.POSIXct(&quot;2018-12-06 10:30&quot;),as.POSIXct(&quot;2018-12-07 10:30&quot;),&#xA;                        as.POSIXct(&quot;2018-12-05 11:30&quot;),as.POSIXct(&quot;2018-12-06 12:30&quot;),as.POSIXct(&quot;2018-12-07 13:30&quot;))&#xA;)&#xA;&#xA;mergeinfo&amp;lt;-merge(DF1[, c(&quot;TIME&quot;, &quot;CATEGORY&quot;)],DF2[, c(&quot;TIME&quot;, &quot;CATEGORY&quot;)])&#xA;&lt;/code&gt;&lt;/pre&gt;&#xA;&#xA;&lt;p&gt;You will see that we are using the &lt;strong&gt;merge&lt;/strong&gt; function in R to identify which dates and times correspond. In this case, none of them do, so upon running the above you will see that the data frame (which I have named mergeinfo) contains 0 observations.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Note that for using this method, you would need to give your TIME variable the same name across both data frames. e.g. in the below example, I renamed &quot;START&quot; in DF2 to &quot;TIME&quot;, so that they are being compared. If you wished to check this across &quot;END&quot; times, you could rename this variable to &quot;TIME&quot; instead.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Hope this helps.&lt;/p&gt;&#xA;" OwnerUserId="32926" LastActivityDate="2018-06-04T16:22:44.440" CommentCount="0" ContentLicense="CC BY-SA 4.0" />
  <row Id="32618" PostTypeId="2" ParentId="32612" CreationDate="2018-06-04T16:23:11.467" Score="1" Body="&lt;p&gt;Assuming you are using &lt;strong&gt;Keras&lt;/strong&gt;, you &lt;strong&gt;should&lt;/strong&gt; use the &lt;a href=&quot;https://keras.io/layers/embeddings/&quot; rel=&quot;nofollow noreferrer&quot;&gt;Embedding&lt;/a&gt; Layer to convert your strings into vectors of real numbers and compare their distance in a high-dimensional space.&lt;/p&gt;&#xA;" OwnerUserId="52089" LastActivityDate="2018-06-04T16:23:11.467" CommentCount="2" ContentLicense="CC BY-SA 4.0" />
  <row Id="32619" PostTypeId="2" ParentId="28493" CreationDate="2018-06-04T16:24:05.270" Score="2" Body="&lt;p&gt;I suggest &lt;a href=&quot;https://github.com/sepandhaghighi/pycm&quot; rel=&quot;nofollow noreferrer&quot;&gt;PyCM&lt;/a&gt; lib for confusion matrix analysis.&lt;/p&gt;&#xA;&#xA;&lt;p&gt;Example :  &lt;/p&gt;&#xA;&#xA;&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; from pycm import *&#xA;&amp;gt;&amp;gt;&amp;gt; y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])&#xA;&amp;gt;&amp;gt;&amp;gt; y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])&#xA;&amp;gt;&amp;gt;&amp;gt; cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred) # Create CM From Data&#xA;&amp;gt;&amp;gt;&amp;gt; cm.classes&#xA;[0, 1, 2]&#xA;&amp;gt;&amp;gt;&amp;gt; cm.table&#xA;{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}&#xA;&amp;gt;&amp;gt;&amp;gt; print(cm)&#xA;Predict          0        1        2        &#xA;Actual&#xA;0                3        0        0        &#xA;1                0        1        2        &#xA;2                2        1        3        &#xA;&#xA;&#xA;&#xA;&#xA;Overall Statistics : &#xA;&#xA;95% CI                                                           (0.30439,0.86228)&#xA;Bennett_S                                                        0.375&#xA;Chi-Squared                                                      6.6&#xA;Chi-Squared DF                                                   4&#xA;Conditional Entropy                                              0.95915&#xA;Cramer_V                                                         0.5244&#xA;Cross Entropy                                                    1.59352&#xA;Gwet_AC1                                                         0.38931&#xA;Joint Entropy                                                    2.45915&#xA;KL Divergence                                                    0.09352&#xA;Kappa                                                            0.35484&#xA;Kappa 95% CI                                                     (-0.07708,0.78675)&#xA;Kappa No Prevalence                                              0.16667&#xA;Kappa Standard Error                                             0.22036&#xA;Kappa Unbiased                                                   0.34426&#xA;Lambda A                                                         0.16667&#xA;Lambda B                                                         0.42857&#xA;Mutual Information                                               0.52421&#xA;Overall_ACC                                                      0.58333&#xA;Overall_RACC                                                     0.35417&#xA;Overall_RACCU                                                    0.36458&#xA;PPV_Macro                                                        0.56667&#xA;PPV_Micro                                                        0.58333&#xA;Phi-Squared                                                      0.55&#xA;Reference Entropy                                                1.5&#xA;Response Entropy                                                 1.48336&#xA;Scott_PI                                                         0.34426&#xA;Standard Error                                                   0.14232&#xA;Strength_Of_Agreement(Altman)                                    Fair&#xA;Strength_Of_Agreement(Cicchetti)                                 Poor&#xA;Strength_Of_Agreement(Fleiss)                                    Poor&#xA;Strength_Of_Agreement(Landis and Koch)                           Fair&#xA;TPR_Macro                                                        0.61111&#xA;TPR_Micro                                                        0.58333&#xA;&#xA;Class Statistics :&#xA;&#xA;Classes                                                          0                       1                       2                       &#xA;ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 &#xA;BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667                 &#xA;DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     &#xA;ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 &#xA;F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692                 &#xA;F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 &#xA;F2(F2 score)                                                     0.88235                 0.35714                 0.51724                 &#xA;FDR(False discovery rate)                                        0.4                     0.5                     0.4                     &#xA;FN(False negative/miss/type 2 error)                             0                       2                       3                       &#xA;FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     &#xA;FOR(False omission rate)                                         0.0                     0.2                     0.42857                 &#xA;FP(False positive/type 1 error/false alarm)                      2                       1                       2                       &#xA;FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333                 &#xA;G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 &#xA;LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5                     &#xA;LR-(Negative likelihood ratio)            